2024-03-20T08:14:52.738 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-20T08:14:52.738 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2024-03-19_00:09:45-rados-wip-yuri5-testing-2024-03-18-1144-distro-default-smithi --verbose --description rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/quincy backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/radosbench} --archive /home/teuthworker/archive/yuriw-2024-03-19_00:09:45-rados-wip-yuri5-testing-2024-03-18-1144-distro-default-smithi/7610089 --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2024-03-19_00:09:45-rados-wip-yuri5-testing-2024-03-18-1144-distro-default-smithi/7610089/orig.config.yaml 2024-03-20T08:14:52.784 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-20T08:14:52.874 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-03-19_00:09:45-rados-wip-yuri5-testing-2024-03-18-1144-distro-default-smithi/7610089 branch: wip-yuri5-testing-2024-03-18-1144 description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/quincy backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/radosbench} email: yweinste@redhat.com first_in_suite: false job_id: '7610089' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-03-19_00:09:45-rados-wip-yuri5-testing-2024-03-18-1144-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 30 - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri5-testing-2024-03-18-1144 ceph: conf: client.2: debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms inject socket failures: 2500 ms tcp read timeout: 5 osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd debug verify cached snaps: true osd debug verify missing on start: true osd max backfills: 3 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(MON_DOWN\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED sha1: e7e80a6ac9849f0f23432d0d9832f096730308ae ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: e7e80a6ac9849f0f23432d0d9832f096730308ae selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: min_in: 4 workunit: branch: wip-yuri5-testing-2024-03-18-1144 sha1: e7e80a6ac9849f0f23432d0d9832f096730308ae owner: scheduled_yuriw@teuthology priority: 100 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - - mon.c - osd.8 - osd.9 - osd.10 - osd.11 - client.2 seed: 7763 sha1: e7e80a6ac9849f0f23432d0d9832f096730308ae sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri5-testing-2024-03-18-1144 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_e7e80a6ac9849f0f23432d0d9832f096730308ae/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: e7e80a6ac9849f0f23432d0d9832f096730308ae targets: smithi026.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFbLfsyKtlfbM5hSsEszpvyZMDjMdWI9xNUVFfef50zLnCGyt0rpJ7JUTvAFjDcBw8134z1fXKU4dYKPnmdSf1M= smithi115.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLj6MPU4ULnRmUWzb51y6RVPXdfj2wc0JekcyMhoibK975aiTV3lLi0sYxM41j/jczfyoyeOitSz1Q5k9vstByY= smithi122.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG6HnHBgjbRlOWERJX5IeyadJ4AUYQe3C1Oxb4ZYCrhO2nni/x68UjAN3rPgLS4Az4+DjCJ1tR+rHrWIrd5wwgg= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: branch: quincy exclude_packages: - ceph-mgr-dashboard - ceph-mgr-diskprediction-local - ceph-mgr-rook - ceph-mgr-cephadm - ceph-base-debuginfo - ceph-common-debuginfo - ceph-immutable-object-cache-debuginfo - ceph-radosgw-debuginfo - ceph-test-debuginfo - ceph-base-debuginfo - ceph-mgr-debuginfo - ceph-mds-debuginfo - ceph-mon-debuginfo - ceph-osd-debuginfo - ceph-fuse-debuginfo - librados-devel-debuginfo - libcephfs2-debuginfo - librados2-debuginfo - librbd1-debuginfo - python3-cephfs-debuginfo - python3-rados-debuginfo - python3-rbd-debuginfo - python3-rgw-debuginfo - rbd-fuse-debuginfo - rbd-mirror-debuginfo - rbd-nbd-debuginfo - ceph-volume - cephadm: conf: mon: auth allow insecure global id reclaim: true - thrashosds: chance_pgnum_grow: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-19_00:09:45 tube: smithi user: yuriw verbose: true verify_ceph_hash: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-20T08:14:52.875 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_e7e80a6ac9849f0f23432d0d9832f096730308ae/qa; will attempt to use it 2024-03-20T08:14:52.876 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_e7e80a6ac9849f0f23432d0d9832f096730308ae/qa/tasks 2024-03-20T08:14:52.876 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-20T08:14:52.877 INFO:teuthology.task.internal:Saving configuration 2024-03-20T08:14:52.898 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-20T08:14:52.905 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-20T08:14:52.931 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi026.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-19_00:09:45-rados-wip-yuri5-testing-2024-03-18-1144-distro-default-smithi/7610089', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-20 08:09:39.970341', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFbLfsyKtlfbM5hSsEszpvyZMDjMdWI9xNUVFfef50zLnCGyt0rpJ7JUTvAFjDcBw8134z1fXKU4dYKPnmdSf1M='} 2024-03-20T08:14:52.953 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi115.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-19_00:09:45-rados-wip-yuri5-testing-2024-03-18-1144-distro-default-smithi/7610089', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-20 08:09:39.972023', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLj6MPU4ULnRmUWzb51y6RVPXdfj2wc0JekcyMhoibK975aiTV3lLi0sYxM41j/jczfyoyeOitSz1Q5k9vstByY='} 2024-03-20T08:14:52.974 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi122.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-19_00:09:45-rados-wip-yuri5-testing-2024-03-18-1144-distro-default-smithi/7610089', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-20 08:09:39.968332', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG6HnHBgjbRlOWERJX5IeyadJ4AUYQe3C1Oxb4ZYCrhO2nni/x68UjAN3rPgLS4Az4+DjCJ1tR+rHrWIrd5wwgg='} 2024-03-20T08:14:52.974 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-20T08:14:52.980 INFO:teuthology.task.internal:roles: ubuntu@smithi026.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-03-20T08:14:52.980 INFO:teuthology.task.internal:roles: ubuntu@smithi115.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-03-20T08:14:52.980 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2'] 2024-03-20T08:14:52.981 INFO:teuthology.run_tasks:Running task console_log... 2024-03-20T08:14:53.103 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f60ea306f70>, signals=[15]) 2024-03-20T08:14:53.103 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-20T08:14:53.110 INFO:teuthology.task.internal:Opening connections... 2024-03-20T08:14:53.110 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi026.front.sepia.ceph.com 2024-03-20T08:14:53.112 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-20T08:14:53.198 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi115.front.sepia.ceph.com 2024-03-20T08:14:53.199 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-20T08:14:53.274 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi122.front.sepia.ceph.com 2024-03-20T08:14:53.275 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-20T08:14:53.350 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-20T08:14:53.357 DEBUG:teuthology.orchestra.run.smithi026:> uname -m 2024-03-20T08:14:53.381 INFO:teuthology.orchestra.run.smithi026.stdout:x86_64 2024-03-20T08:14:53.381 DEBUG:teuthology.orchestra.run.smithi026:> cat /etc/os-release 2024-03-20T08:14:53.443 INFO:teuthology.orchestra.run.smithi026.stdout:NAME="CentOS Stream" 2024-03-20T08:14:53.443 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION="8" 2024-03-20T08:14:53.443 INFO:teuthology.orchestra.run.smithi026.stdout:ID="centos" 2024-03-20T08:14:53.443 INFO:teuthology.orchestra.run.smithi026.stdout:ID_LIKE="rhel fedora" 2024-03-20T08:14:53.443 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION_ID="8" 2024-03-20T08:14:53.443 INFO:teuthology.orchestra.run.smithi026.stdout:PLATFORM_ID="platform:el8" 2024-03-20T08:14:53.444 INFO:teuthology.orchestra.run.smithi026.stdout:PRETTY_NAME="CentOS Stream 8" 2024-03-20T08:14:53.444 INFO:teuthology.orchestra.run.smithi026.stdout:ANSI_COLOR="0;31" 2024-03-20T08:14:53.444 INFO:teuthology.orchestra.run.smithi026.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-03-20T08:14:53.444 INFO:teuthology.orchestra.run.smithi026.stdout:HOME_URL="https://centos.org/" 2024-03-20T08:14:53.444 INFO:teuthology.orchestra.run.smithi026.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-20T08:14:53.444 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-20T08:14:53.444 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-20T08:14:53.445 INFO:teuthology.lock.ops:Updating smithi026.front.sepia.ceph.com on lock server 2024-03-20T08:14:53.469 DEBUG:teuthology.orchestra.run.smithi115:> uname -m 2024-03-20T08:14:53.491 INFO:teuthology.orchestra.run.smithi115.stdout:x86_64 2024-03-20T08:14:53.492 DEBUG:teuthology.orchestra.run.smithi115:> cat /etc/os-release 2024-03-20T08:14:53.553 INFO:teuthology.orchestra.run.smithi115.stdout:NAME="CentOS Stream" 2024-03-20T08:14:53.553 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION="8" 2024-03-20T08:14:53.553 INFO:teuthology.orchestra.run.smithi115.stdout:ID="centos" 2024-03-20T08:14:53.554 INFO:teuthology.orchestra.run.smithi115.stdout:ID_LIKE="rhel fedora" 2024-03-20T08:14:53.554 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION_ID="8" 2024-03-20T08:14:53.554 INFO:teuthology.orchestra.run.smithi115.stdout:PLATFORM_ID="platform:el8" 2024-03-20T08:14:53.554 INFO:teuthology.orchestra.run.smithi115.stdout:PRETTY_NAME="CentOS Stream 8" 2024-03-20T08:14:53.554 INFO:teuthology.orchestra.run.smithi115.stdout:ANSI_COLOR="0;31" 2024-03-20T08:14:53.554 INFO:teuthology.orchestra.run.smithi115.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-03-20T08:14:53.554 INFO:teuthology.orchestra.run.smithi115.stdout:HOME_URL="https://centos.org/" 2024-03-20T08:14:53.554 INFO:teuthology.orchestra.run.smithi115.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-20T08:14:53.554 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-20T08:14:53.554 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-20T08:14:53.554 INFO:teuthology.lock.ops:Updating smithi115.front.sepia.ceph.com on lock server 2024-03-20T08:14:53.577 DEBUG:teuthology.orchestra.run.smithi122:> uname -m 2024-03-20T08:14:53.604 INFO:teuthology.orchestra.run.smithi122.stdout:x86_64 2024-03-20T08:14:53.604 DEBUG:teuthology.orchestra.run.smithi122:> cat /etc/os-release 2024-03-20T08:14:53.667 INFO:teuthology.orchestra.run.smithi122.stdout:NAME="CentOS Stream" 2024-03-20T08:14:53.667 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION="8" 2024-03-20T08:14:53.667 INFO:teuthology.orchestra.run.smithi122.stdout:ID="centos" 2024-03-20T08:14:53.667 INFO:teuthology.orchestra.run.smithi122.stdout:ID_LIKE="rhel fedora" 2024-03-20T08:14:53.667 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_ID="8" 2024-03-20T08:14:53.667 INFO:teuthology.orchestra.run.smithi122.stdout:PLATFORM_ID="platform:el8" 2024-03-20T08:14:53.667 INFO:teuthology.orchestra.run.smithi122.stdout:PRETTY_NAME="CentOS Stream 8" 2024-03-20T08:14:53.668 INFO:teuthology.orchestra.run.smithi122.stdout:ANSI_COLOR="0;31" 2024-03-20T08:14:53.668 INFO:teuthology.orchestra.run.smithi122.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-03-20T08:14:53.668 INFO:teuthology.orchestra.run.smithi122.stdout:HOME_URL="https://centos.org/" 2024-03-20T08:14:53.668 INFO:teuthology.orchestra.run.smithi122.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-20T08:14:53.668 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-20T08:14:53.668 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-20T08:14:53.668 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server 2024-03-20T08:14:53.685 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-20T08:14:53.695 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-20T08:14:53.713 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-20T08:14:53.714 DEBUG:teuthology.orchestra.run.smithi026:> test '!' -e /home/ubuntu/cephtest 2024-03-20T08:14:53.716 DEBUG:teuthology.orchestra.run.smithi115:> test '!' -e /home/ubuntu/cephtest 2024-03-20T08:14:53.718 DEBUG:teuthology.orchestra.run.smithi122:> test '!' -e /home/ubuntu/cephtest 2024-03-20T08:14:53.740 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-20T08:14:53.746 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-20T08:14:53.746 DEBUG:teuthology.orchestra.run.smithi026:> test -z $(ls -A /var/lib/ceph) 2024-03-20T08:14:53.779 DEBUG:teuthology.orchestra.run.smithi115:> test -z $(ls -A /var/lib/ceph) 2024-03-20T08:14:53.782 DEBUG:teuthology.orchestra.run.smithi122:> test -z $(ls -A /var/lib/ceph) 2024-03-20T08:14:53.814 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-20T08:14:53.893 INFO:teuthology.run_tasks:Running task kernel... 2024-03-20T08:14:53.909 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-20T08:14:53.909 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-20T08:14:53.909 DEBUG:teuthology.orchestra.run.smithi026:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-20T08:14:53.910 DEBUG:teuthology.orchestra.run.smithi115:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-20T08:14:53.910 DEBUG:teuthology.orchestra.run.smithi122:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-20T08:14:53.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-20T08:14:53.931 DEBUG:teuthology.orchestra.run.smithi115:> uname -r 2024-03-20T08:14:53.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-20T08:14:53.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-20T08:14:53.933 DEBUG:teuthology.orchestra.run.smithi026:> uname -r 2024-03-20T08:14:53.933 DEBUG:teuthology.orchestra.run.smithi122:> uname -r 2024-03-20T08:14:53.992 INFO:teuthology.orchestra.run.smithi115.stdout:4.18.0-489.el8.x86_64 2024-03-20T08:14:53.993 INFO:teuthology.task.kernel:Running kernel on smithi115: 4.18.0-489.el8.x86_64 2024-03-20T08:14:53.993 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum install -y kernel 2024-03-20T08:14:53.994 INFO:teuthology.orchestra.run.smithi122.stdout:4.18.0-489.el8.x86_64 2024-03-20T08:14:53.994 INFO:teuthology.task.kernel:Running kernel on smithi122: 4.18.0-489.el8.x86_64 2024-03-20T08:14:53.995 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum install -y kernel 2024-03-20T08:14:53.996 INFO:teuthology.orchestra.run.smithi026.stdout:4.18.0-489.el8.x86_64 2024-03-20T08:14:53.996 INFO:teuthology.task.kernel:Running kernel on smithi026: 4.18.0-489.el8.x86_64 2024-03-20T08:14:53.996 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum install -y kernel 2024-03-20T08:34:38.044 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 8 - AppStream 24 kB/s | 28 MB 19:41 2024-03-20T08:34:43.560 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 8 - BaseOS 18 MB/s | 10 MB 00:00 2024-03-20T08:34:45.611 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 8 - Extras 43 kB/s | 18 kB 00:00 2024-03-20T08:34:45.962 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 8 - Extras common packages 31 kB/s | 7.5 kB 00:00 2024-03-20T08:34:46.230 INFO:teuthology.orchestra.run.smithi122.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2024-03-20T08:34:47.392 INFO:teuthology.orchestra.run.smithi122.stdout:Extra Packages for Enterprise Linux 15 MB/s | 16 MB 00:01 2024-03-20T08:34:51.115 INFO:teuthology.orchestra.run.smithi122.stdout:lab-extras 336 kB/s | 24 kB 00:00 2024-03-20T08:34:53.650 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - AppStream 24 kB/s | 28 MB 19:57 2024-03-20T08:34:54.179 INFO:teuthology.orchestra.run.smithi122.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-03-20T08:34:54.179 INFO:teuthology.orchestra.run.smithi122.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-03-20T08:34:54.289 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-20T08:34:54.290 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:34:54.290 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2024-03-20T08:34:54.290 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:34:54.290 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-20T08:34:54.290 INFO:teuthology.orchestra.run.smithi122.stdout: kernel x86_64 4.18.0-546.el8 baseos 10 M 2024-03-20T08:34:54.290 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-03-20T08:34:54.290 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-core x86_64 4.18.0-546.el8 baseos 44 M 2024-03-20T08:34:54.290 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules x86_64 4.18.0-546.el8 baseos 36 M 2024-03-20T08:34:54.290 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:34:54.290 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-20T08:34:54.290 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:34:54.291 INFO:teuthology.orchestra.run.smithi122.stdout:Install 3 Packages 2024-03-20T08:34:54.291 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:34:54.291 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 90 M 2024-03-20T08:34:54.291 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 97 M 2024-03-20T08:34:54.291 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-20T08:34:54.808 INFO:teuthology.orchestra.run.smithi122.stdout:(1/3): kernel-4.18.0-546.el8.x86_64.rpm 22 MB/s | 10 MB 00:00 2024-03-20T08:34:55.384 INFO:teuthology.orchestra.run.smithi122.stdout:(2/3): kernel-core-4.18.0-546.el8.x86_64.rpm 42 MB/s | 44 MB 00:01 2024-03-20T08:34:59.508 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - BaseOS 11 MB/s | 10 MB 00:00 2024-03-20T08:34:59.676 INFO:teuthology.orchestra.run.smithi122.stdout:(3/3): kernel-modules-4.18.0-546.el8.x86_64.rpm 6.8 MB/s | 36 MB 00:05 2024-03-20T08:34:59.676 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:34:59.676 INFO:teuthology.orchestra.run.smithi122.stdout:Total 17 MB/s | 90 MB 00:05 2024-03-20T08:35:00.361 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-20T08:35:00.774 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-20T08:35:00.775 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-20T08:35:01.450 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - Extras 56 kB/s | 18 kB 00:00 2024-03-20T08:35:01.767 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - Extras common packages 36 kB/s | 7.5 kB 00:00 2024-03-20T08:35:02.022 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-20T08:35:02.044 INFO:teuthology.orchestra.run.smithi115.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2024-03-20T08:35:02.055 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-20T08:35:03.263 INFO:teuthology.orchestra.run.smithi115.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2024-03-20T08:35:04.820 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-20T08:35:05.339 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-core-4.18.0-546.el8.x86_64 1/3 2024-03-20T08:35:07.044 INFO:teuthology.orchestra.run.smithi115.stdout:lab-extras 405 kB/s | 24 kB 00:00 2024-03-20T08:35:08.906 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-core-4.18.0-546.el8.x86_64 1/3 2024-03-20T08:35:09.151 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 8 - AppStream 23 kB/s | 28 MB 20:12 2024-03-20T08:35:09.313 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-modules-4.18.0-546.el8.x86_64 2/3 2024-03-20T08:35:10.072 INFO:teuthology.orchestra.run.smithi115.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-03-20T08:35:10.072 INFO:teuthology.orchestra.run.smithi115.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-03-20T08:35:10.178 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-20T08:35:10.179 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:35:10.179 INFO:teuthology.orchestra.run.smithi115.stdout: Package Architecture Version Repository Size 2024-03-20T08:35:10.179 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:35:10.179 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-20T08:35:10.179 INFO:teuthology.orchestra.run.smithi115.stdout: kernel x86_64 4.18.0-546.el8 baseos 10 M 2024-03-20T08:35:10.179 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-03-20T08:35:10.180 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-core x86_64 4.18.0-546.el8 baseos 44 M 2024-03-20T08:35:10.180 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-modules x86_64 4.18.0-546.el8 baseos 36 M 2024-03-20T08:35:10.180 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:35:10.180 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-20T08:35:10.180 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:35:10.180 INFO:teuthology.orchestra.run.smithi115.stdout:Install 3 Packages 2024-03-20T08:35:10.180 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:35:10.181 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 90 M 2024-03-20T08:35:10.181 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 97 M 2024-03-20T08:35:10.181 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-20T08:35:11.262 INFO:teuthology.orchestra.run.smithi115.stdout:(1/3): kernel-4.18.0-546.el8.x86_64.rpm 10 MB/s | 10 MB 00:01 2024-03-20T08:35:11.812 INFO:teuthology.orchestra.run.smithi115.stdout:(2/3): kernel-modules-4.18.0-546.el8.x86_64.rpm 23 MB/s | 36 MB 00:01 2024-03-20T08:35:12.304 INFO:teuthology.orchestra.run.smithi115.stdout:(3/3): kernel-core-4.18.0-546.el8.x86_64.rpm 21 MB/s | 44 MB 00:02 2024-03-20T08:35:12.304 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:35:12.304 INFO:teuthology.orchestra.run.smithi115.stdout:Total 43 MB/s | 90 MB 00:02 2024-03-20T08:35:12.748 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-modules-4.18.0-546.el8.x86_64 2/3 2024-03-20T08:35:12.985 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-20T08:35:13.022 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-4.18.0-546.el8.x86_64 3/3 2024-03-20T08:35:13.404 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-20T08:35:13.404 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-20T08:35:14.654 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-20T08:35:14.701 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-20T08:35:15.242 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 8 - BaseOS 12 MB/s | 10 MB 00:00 2024-03-20T08:35:17.293 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 8 - Extras 49 kB/s | 18 kB 00:00 2024-03-20T08:35:17.459 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-20T08:35:17.752 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 8 - Extras common packages 22 kB/s | 7.5 kB 00:00 2024-03-20T08:35:17.986 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : kernel-core-4.18.0-546.el8.x86_64 1/3 2024-03-20T08:35:18.012 INFO:teuthology.orchestra.run.smithi026.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2024-03-20T08:35:21.531 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-core-4.18.0-546.el8.x86_64 1/3 2024-03-20T08:35:21.728 INFO:teuthology.orchestra.run.smithi026.stdout:Extra Packages for Enterprise Linux 4.5 MB/s | 16 MB 00:03 2024-03-20T08:35:21.960 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : kernel-modules-4.18.0-546.el8.x86_64 2/3 2024-03-20T08:35:25.406 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-modules-4.18.0-546.el8.x86_64 2/3 2024-03-20T08:35:25.684 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : kernel-4.18.0-546.el8.x86_64 3/3 2024-03-20T08:35:25.721 INFO:teuthology.orchestra.run.smithi026.stdout:lab-extras 521 kB/s | 24 kB 00:00 2024-03-20T08:35:28.943 INFO:teuthology.orchestra.run.smithi026.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-03-20T08:35:28.943 INFO:teuthology.orchestra.run.smithi026.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-03-20T08:35:29.091 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-20T08:35:29.092 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:35:29.092 INFO:teuthology.orchestra.run.smithi026.stdout: Package Architecture Version Repository Size 2024-03-20T08:35:29.092 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:35:29.092 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-20T08:35:29.092 INFO:teuthology.orchestra.run.smithi026.stdout: kernel x86_64 4.18.0-546.el8 baseos 10 M 2024-03-20T08:35:29.092 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-03-20T08:35:29.092 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-core x86_64 4.18.0-546.el8 baseos 44 M 2024-03-20T08:35:29.092 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-modules x86_64 4.18.0-546.el8 baseos 36 M 2024-03-20T08:35:29.092 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:35:29.093 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-20T08:35:29.093 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:35:29.093 INFO:teuthology.orchestra.run.smithi026.stdout:Install 3 Packages 2024-03-20T08:35:29.093 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:35:29.093 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 90 M 2024-03-20T08:35:29.093 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 97 M 2024-03-20T08:35:29.093 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-20T08:35:29.789 INFO:teuthology.orchestra.run.smithi026.stdout:(1/3): kernel-4.18.0-546.el8.x86_64.rpm 20 MB/s | 10 MB 00:00 2024-03-20T08:35:30.455 INFO:teuthology.orchestra.run.smithi026.stdout:(2/3): kernel-modules-4.18.0-546.el8.x86_64.rpm 31 MB/s | 36 MB 00:01 2024-03-20T08:35:31.022 INFO:teuthology.orchestra.run.smithi026.stdout:(3/3): kernel-core-4.18.0-546.el8.x86_64.rpm 25 MB/s | 44 MB 00:01 2024-03-20T08:35:31.023 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:35:31.023 INFO:teuthology.orchestra.run.smithi026.stdout:Total 47 MB/s | 90 MB 00:01 2024-03-20T08:35:31.739 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-20T08:35:32.184 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-20T08:35:32.184 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-20T08:35:33.489 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-20T08:35:33.524 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-20T08:35:35.006 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-core-4.18.0-546.el8.x86_64 3/3 2024-03-20T08:35:35.028 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-modules-4.18.0-546.el8.x86_64 3/3 2024-03-20T08:35:35.373 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-4.18.0-546.el8.x86_64 3/3 2024-03-20T08:35:35.373 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-4.18.0-546.el8.x86_64 1/3 2024-03-20T08:35:35.373 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-core-4.18.0-546.el8.x86_64 2/3 2024-03-20T08:35:35.923 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-modules-4.18.0-546.el8.x86_64 3/3 2024-03-20T08:35:35.923 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:35:35.923 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-20T08:35:35.923 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-4.18.0-546.el8.x86_64 kernel-core-4.18.0-546.el8.x86_64 2024-03-20T08:35:35.923 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules-4.18.0-546.el8.x86_64 2024-03-20T08:35:35.923 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:35:35.923 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-20T08:35:36.059 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q kernel | sort -rV | head -n 1 2024-03-20T08:35:36.274 INFO:teuthology.orchestra.run.smithi122.stdout:kernel-4.18.0-546.el8.x86_64 2024-03-20T08:35:36.274 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-546.el8.x86_64 2024-03-20T08:35:36.274 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-546.el8.x86_64 2024-03-20T08:35:36.274 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-20T08:35:36.275 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-20T08:35:36.275 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-03-20T08:35:36.275 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi122.front.sepia.ceph.com, path=None, version=distro) 2024-03-20T08:35:36.275 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q kernel | sort -rV | head -n 1 2024-03-20T08:35:36.464 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-20T08:35:36.485 INFO:teuthology.orchestra.run.smithi122.stdout:kernel-4.18.0-546.el8.x86_64 2024-03-20T08:35:36.485 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-546.el8.x86_64 2024-03-20T08:35:36.486 DEBUG:teuthology.orchestra.run.smithi122:> sudo rpm -qi grub2-tools 2024-03-20T08:35:36.519 INFO:teuthology.orchestra.run.smithi122.stdout:Name : grub2-tools 2024-03-20T08:35:36.520 INFO:teuthology.orchestra.run.smithi122.stdout:Epoch : 1 2024-03-20T08:35:36.520 INFO:teuthology.orchestra.run.smithi122.stdout:Version : 2.02 2024-03-20T08:35:36.520 INFO:teuthology.orchestra.run.smithi122.stdout:Release : 129.el8 2024-03-20T08:35:36.520 INFO:teuthology.orchestra.run.smithi122.stdout:Architecture: x86_64 2024-03-20T08:35:36.520 INFO:teuthology.orchestra.run.smithi122.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-03-20T08:35:36.520 INFO:teuthology.orchestra.run.smithi122.stdout:Group : System Environment/Base 2024-03-20T08:35:36.520 INFO:teuthology.orchestra.run.smithi122.stdout:Size : 9265586 2024-03-20T08:35:36.520 INFO:teuthology.orchestra.run.smithi122.stdout:License : GPLv3+ 2024-03-20T08:35:36.520 INFO:teuthology.orchestra.run.smithi122.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-03-20T08:35:36.520 INFO:teuthology.orchestra.run.smithi122.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-03-20T08:35:36.520 INFO:teuthology.orchestra.run.smithi122.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-03-20T08:35:36.520 INFO:teuthology.orchestra.run.smithi122.stdout:Build Host : kbuilder.bsys.centos.org 2024-03-20T08:35:36.521 INFO:teuthology.orchestra.run.smithi122.stdout:Relocations : (not relocatable) 2024-03-20T08:35:36.521 INFO:teuthology.orchestra.run.smithi122.stdout:Packager : CentOS BuildSystem 2024-03-20T08:35:36.521 INFO:teuthology.orchestra.run.smithi122.stdout:Vendor : CentOS 2024-03-20T08:35:36.521 INFO:teuthology.orchestra.run.smithi122.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-20T08:35:36.521 INFO:teuthology.orchestra.run.smithi122.stdout:Summary : Support tools for GRUB. 2024-03-20T08:35:36.521 INFO:teuthology.orchestra.run.smithi122.stdout:Description : 2024-03-20T08:35:36.521 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:35:36.521 INFO:teuthology.orchestra.run.smithi122.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-20T08:35:36.521 INFO:teuthology.orchestra.run.smithi122.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-20T08:35:36.521 INFO:teuthology.orchestra.run.smithi122.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-20T08:35:36.521 INFO:teuthology.orchestra.run.smithi122.stdout:hardware devices. 2024-03-20T08:35:36.521 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:35:36.521 INFO:teuthology.orchestra.run.smithi122.stdout:This subpackage provides tools for support of all platforms. 2024-03-20T08:35:36.522 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-20T08:35:36.522 INFO:teuthology.task.kernel:Updating grub on smithi122 to boot 4.18.0-546.el8.x86_64 2024-03-20T08:35:36.522 DEBUG:teuthology.orchestra.run.smithi122:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-20T08:35:37.061 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : kernel-core-4.18.0-546.el8.x86_64 1/3 2024-03-20T08:35:38.241 INFO:teuthology.orchestra.run.smithi122.stderr:Generating grub configuration file ... 2024-03-20T08:35:40.437 INFO:teuthology.orchestra.run.smithi122.stderr:done 2024-03-20T08:35:40.438 DEBUG:teuthology.orchestra.run.smithi122:> mktemp 2024-03-20T08:35:40.473 INFO:teuthology.orchestra.run.smithi122.stdout:/tmp/tmp.icXN0R4B5b 2024-03-20T08:35:40.474 DEBUG:teuthology.orchestra.run.smithi122:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.icXN0R4B5b 2024-03-20T08:35:40.550 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0666 /tmp/tmp.icXN0R4B5b 2024-03-20T08:35:40.709 DEBUG:teuthology.orchestra.remote:smithi122:/tmp/tmp.icXN0R4B5b is 6KB 2024-03-20T08:35:40.757 DEBUG:teuthology.orchestra.run.smithi122:> rm -fr /tmp/tmp.icXN0R4B5b 2024-03-20T08:35:40.775 DEBUG:teuthology.orchestra.run.smithi122:> sudo /bin/ls /boot/loader/entries || true 2024-03-20T08:35:40.850 INFO:teuthology.orchestra.run.smithi122.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-03-20T08:35:40.850 INFO:teuthology.orchestra.run.smithi122.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-03-20T08:35:40.850 INFO:teuthology.orchestra.run.smithi122.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-03-20T08:35:40.850 INFO:teuthology.orchestra.run.smithi122.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-546.el8.x86_64.conf 2024-03-20T08:35:40.851 DEBUG:teuthology.orchestra.run.smithi122:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-546.el8.x86_64 2024-03-20T08:35:40.866 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-core-4.18.0-546.el8.x86_64 1/3 2024-03-20T08:35:40.991 DEBUG:teuthology.orchestra.run.smithi122:> sudo shutdown -r now 2024-03-20T08:35:41.293 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : kernel-modules-4.18.0-546.el8.x86_64 2/3 2024-03-20T08:35:44.979 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-modules-4.18.0-546.el8.x86_64 2/3 2024-03-20T08:35:45.313 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : kernel-4.18.0-546.el8.x86_64 3/3 2024-03-20T08:35:47.661 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-core-4.18.0-546.el8.x86_64 3/3 2024-03-20T08:35:47.683 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-modules-4.18.0-546.el8.x86_64 3/3 2024-03-20T08:35:48.334 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: kernel-4.18.0-546.el8.x86_64 3/3 2024-03-20T08:35:48.334 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : kernel-4.18.0-546.el8.x86_64 1/3 2024-03-20T08:35:48.334 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : kernel-core-4.18.0-546.el8.x86_64 2/3 2024-03-20T08:35:48.838 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : kernel-modules-4.18.0-546.el8.x86_64 3/3 2024-03-20T08:35:48.839 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:35:48.839 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-20T08:35:48.839 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-4.18.0-546.el8.x86_64 kernel-core-4.18.0-546.el8.x86_64 2024-03-20T08:35:48.839 INFO:teuthology.orchestra.run.smithi115.stdout: kernel-modules-4.18.0-546.el8.x86_64 2024-03-20T08:35:48.839 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:35:48.839 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-20T08:35:48.986 DEBUG:teuthology.orchestra.run.smithi115:> rpm -q kernel | sort -rV | head -n 1 2024-03-20T08:35:49.203 INFO:teuthology.orchestra.run.smithi115.stdout:kernel-4.18.0-546.el8.x86_64 2024-03-20T08:35:49.204 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-546.el8.x86_64 2024-03-20T08:35:49.204 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-546.el8.x86_64 2024-03-20T08:35:49.204 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-20T08:35:49.204 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-20T08:35:49.204 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-20T08:35:49.204 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi115.front.sepia.ceph.com, path=None, version=distro) 2024-03-20T08:35:49.204 DEBUG:teuthology.orchestra.run.smithi115:> rpm -q kernel | sort -rV | head -n 1 2024-03-20T08:35:49.415 INFO:teuthology.orchestra.run.smithi115.stdout:kernel-4.18.0-546.el8.x86_64 2024-03-20T08:35:49.415 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-546.el8.x86_64 2024-03-20T08:35:49.415 DEBUG:teuthology.orchestra.run.smithi115:> sudo rpm -qi grub2-tools 2024-03-20T08:35:49.449 INFO:teuthology.orchestra.run.smithi115.stdout:Name : grub2-tools 2024-03-20T08:35:49.449 INFO:teuthology.orchestra.run.smithi115.stdout:Epoch : 1 2024-03-20T08:35:49.449 INFO:teuthology.orchestra.run.smithi115.stdout:Version : 2.02 2024-03-20T08:35:49.450 INFO:teuthology.orchestra.run.smithi115.stdout:Release : 129.el8 2024-03-20T08:35:49.450 INFO:teuthology.orchestra.run.smithi115.stdout:Architecture: x86_64 2024-03-20T08:35:49.450 INFO:teuthology.orchestra.run.smithi115.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-03-20T08:35:49.450 INFO:teuthology.orchestra.run.smithi115.stdout:Group : System Environment/Base 2024-03-20T08:35:49.450 INFO:teuthology.orchestra.run.smithi115.stdout:Size : 9265586 2024-03-20T08:35:49.450 INFO:teuthology.orchestra.run.smithi115.stdout:License : GPLv3+ 2024-03-20T08:35:49.450 INFO:teuthology.orchestra.run.smithi115.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-03-20T08:35:49.450 INFO:teuthology.orchestra.run.smithi115.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-03-20T08:35:49.450 INFO:teuthology.orchestra.run.smithi115.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-03-20T08:35:49.450 INFO:teuthology.orchestra.run.smithi115.stdout:Build Host : kbuilder.bsys.centos.org 2024-03-20T08:35:49.450 INFO:teuthology.orchestra.run.smithi115.stdout:Relocations : (not relocatable) 2024-03-20T08:35:49.450 INFO:teuthology.orchestra.run.smithi115.stdout:Packager : CentOS BuildSystem 2024-03-20T08:35:49.450 INFO:teuthology.orchestra.run.smithi115.stdout:Vendor : CentOS 2024-03-20T08:35:49.451 INFO:teuthology.orchestra.run.smithi115.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-20T08:35:49.451 INFO:teuthology.orchestra.run.smithi115.stdout:Summary : Support tools for GRUB. 2024-03-20T08:35:49.451 INFO:teuthology.orchestra.run.smithi115.stdout:Description : 2024-03-20T08:35:49.451 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:35:49.451 INFO:teuthology.orchestra.run.smithi115.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-20T08:35:49.451 INFO:teuthology.orchestra.run.smithi115.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-20T08:35:49.451 INFO:teuthology.orchestra.run.smithi115.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-20T08:35:49.451 INFO:teuthology.orchestra.run.smithi115.stdout:hardware devices. 2024-03-20T08:35:49.451 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:35:49.451 INFO:teuthology.orchestra.run.smithi115.stdout:This subpackage provides tools for support of all platforms. 2024-03-20T08:35:49.452 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-20T08:35:49.452 INFO:teuthology.task.kernel:Updating grub on smithi115 to boot 4.18.0-546.el8.x86_64 2024-03-20T08:35:49.452 DEBUG:teuthology.orchestra.run.smithi115:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-20T08:35:50.887 INFO:teuthology.orchestra.run.smithi115.stderr:Generating grub configuration file ... 2024-03-20T08:35:52.623 INFO:teuthology.orchestra.run.smithi115.stderr:done 2024-03-20T08:35:52.625 DEBUG:teuthology.orchestra.run.smithi115:> mktemp 2024-03-20T08:35:52.697 INFO:teuthology.orchestra.run.smithi115.stdout:/tmp/tmp.lfnwnDkrSm 2024-03-20T08:35:52.698 DEBUG:teuthology.orchestra.run.smithi115:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.lfnwnDkrSm 2024-03-20T08:35:52.771 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 0666 /tmp/tmp.lfnwnDkrSm 2024-03-20T08:35:52.928 DEBUG:teuthology.orchestra.remote:smithi115:/tmp/tmp.lfnwnDkrSm is 6KB 2024-03-20T08:35:52.976 DEBUG:teuthology.orchestra.run.smithi115:> rm -fr /tmp/tmp.lfnwnDkrSm 2024-03-20T08:35:52.994 DEBUG:teuthology.orchestra.run.smithi115:> sudo /bin/ls /boot/loader/entries || true 2024-03-20T08:35:53.092 INFO:teuthology.orchestra.run.smithi115.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-03-20T08:35:53.092 INFO:teuthology.orchestra.run.smithi115.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-03-20T08:35:53.093 INFO:teuthology.orchestra.run.smithi115.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-03-20T08:35:53.093 INFO:teuthology.orchestra.run.smithi115.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-546.el8.x86_64.conf 2024-03-20T08:35:53.094 DEBUG:teuthology.orchestra.run.smithi115:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-546.el8.x86_64 2024-03-20T08:35:53.204 DEBUG:teuthology.orchestra.run.smithi115:> sudo shutdown -r now 2024-03-20T08:36:08.735 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-core-4.18.0-546.el8.x86_64 3/3 2024-03-20T08:36:08.758 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-modules-4.18.0-546.el8.x86_64 3/3 2024-03-20T08:36:09.117 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-4.18.0-546.el8.x86_64 3/3 2024-03-20T08:36:09.117 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : kernel-4.18.0-546.el8.x86_64 1/3 2024-03-20T08:36:09.117 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : kernel-core-4.18.0-546.el8.x86_64 2/3 2024-03-20T08:36:09.690 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : kernel-modules-4.18.0-546.el8.x86_64 3/3 2024-03-20T08:36:09.690 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:36:09.690 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-20T08:36:09.690 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-4.18.0-546.el8.x86_64 kernel-core-4.18.0-546.el8.x86_64 2024-03-20T08:36:09.690 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-modules-4.18.0-546.el8.x86_64 2024-03-20T08:36:09.690 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:36:09.690 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-20T08:36:09.841 DEBUG:teuthology.orchestra.run.smithi026:> rpm -q kernel | sort -rV | head -n 1 2024-03-20T08:36:10.112 INFO:teuthology.orchestra.run.smithi026.stdout:kernel-4.18.0-546.el8.x86_64 2024-03-20T08:36:10.113 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-546.el8.x86_64 2024-03-20T08:36:10.113 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-546.el8.x86_64 2024-03-20T08:36:10.113 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-20T08:36:10.113 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-20T08:36:10.113 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-20T08:36:10.113 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi026.front.sepia.ceph.com, path=None, version=distro) 2024-03-20T08:36:10.113 DEBUG:teuthology.orchestra.run.smithi026:> rpm -q kernel | sort -rV | head -n 1 2024-03-20T08:36:10.335 INFO:teuthology.orchestra.run.smithi026.stdout:kernel-4.18.0-546.el8.x86_64 2024-03-20T08:36:10.335 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-546.el8.x86_64 2024-03-20T08:36:10.335 DEBUG:teuthology.orchestra.run.smithi026:> sudo rpm -qi grub2-tools 2024-03-20T08:36:10.370 INFO:teuthology.orchestra.run.smithi026.stdout:Name : grub2-tools 2024-03-20T08:36:10.370 INFO:teuthology.orchestra.run.smithi026.stdout:Epoch : 1 2024-03-20T08:36:10.370 INFO:teuthology.orchestra.run.smithi026.stdout:Version : 2.02 2024-03-20T08:36:10.370 INFO:teuthology.orchestra.run.smithi026.stdout:Release : 129.el8 2024-03-20T08:36:10.370 INFO:teuthology.orchestra.run.smithi026.stdout:Architecture: x86_64 2024-03-20T08:36:10.370 INFO:teuthology.orchestra.run.smithi026.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-03-20T08:36:10.370 INFO:teuthology.orchestra.run.smithi026.stdout:Group : System Environment/Base 2024-03-20T08:36:10.370 INFO:teuthology.orchestra.run.smithi026.stdout:Size : 9265586 2024-03-20T08:36:10.371 INFO:teuthology.orchestra.run.smithi026.stdout:License : GPLv3+ 2024-03-20T08:36:10.371 INFO:teuthology.orchestra.run.smithi026.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-03-20T08:36:10.371 INFO:teuthology.orchestra.run.smithi026.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-03-20T08:36:10.371 INFO:teuthology.orchestra.run.smithi026.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-03-20T08:36:10.371 INFO:teuthology.orchestra.run.smithi026.stdout:Build Host : kbuilder.bsys.centos.org 2024-03-20T08:36:10.371 INFO:teuthology.orchestra.run.smithi026.stdout:Relocations : (not relocatable) 2024-03-20T08:36:10.371 INFO:teuthology.orchestra.run.smithi026.stdout:Packager : CentOS BuildSystem 2024-03-20T08:36:10.371 INFO:teuthology.orchestra.run.smithi026.stdout:Vendor : CentOS 2024-03-20T08:36:10.371 INFO:teuthology.orchestra.run.smithi026.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-20T08:36:10.371 INFO:teuthology.orchestra.run.smithi026.stdout:Summary : Support tools for GRUB. 2024-03-20T08:36:10.371 INFO:teuthology.orchestra.run.smithi026.stdout:Description : 2024-03-20T08:36:10.371 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:36:10.371 INFO:teuthology.orchestra.run.smithi026.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-20T08:36:10.371 INFO:teuthology.orchestra.run.smithi026.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-20T08:36:10.372 INFO:teuthology.orchestra.run.smithi026.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-20T08:36:10.372 INFO:teuthology.orchestra.run.smithi026.stdout:hardware devices. 2024-03-20T08:36:10.372 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:36:10.372 INFO:teuthology.orchestra.run.smithi026.stdout:This subpackage provides tools for support of all platforms. 2024-03-20T08:36:10.373 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-20T08:36:10.373 INFO:teuthology.task.kernel:Updating grub on smithi026 to boot 4.18.0-546.el8.x86_64 2024-03-20T08:36:10.373 DEBUG:teuthology.orchestra.run.smithi026:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-20T08:36:10.993 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-03-20T08:36:10.993 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-03-20T08:36:10.995 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-20T08:36:11.670 INFO:teuthology.orchestra.run.smithi026.stderr:Generating grub configuration file ... 2024-03-20T08:36:13.692 INFO:teuthology.orchestra.run.smithi026.stderr:done 2024-03-20T08:36:13.693 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2024-03-20T08:36:13.725 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.x6lnlGKCAT 2024-03-20T08:36:13.725 DEBUG:teuthology.orchestra.run.smithi026:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.x6lnlGKCAT 2024-03-20T08:36:13.801 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.x6lnlGKCAT 2024-03-20T08:36:13.948 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.x6lnlGKCAT is 6KB 2024-03-20T08:36:13.997 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.x6lnlGKCAT 2024-03-20T08:36:14.015 DEBUG:teuthology.orchestra.run.smithi026:> sudo /bin/ls /boot/loader/entries || true 2024-03-20T08:36:14.082 INFO:teuthology.orchestra.run.smithi026.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-03-20T08:36:14.082 INFO:teuthology.orchestra.run.smithi026.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-03-20T08:36:14.083 INFO:teuthology.orchestra.run.smithi026.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-03-20T08:36:14.083 INFO:teuthology.orchestra.run.smithi026.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-546.el8.x86_64.conf 2024-03-20T08:36:14.084 DEBUG:teuthology.orchestra.run.smithi026:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-546.el8.x86_64 2024-03-20T08:36:14.251 DEBUG:teuthology.orchestra.run.smithi026:> sudo shutdown -r now 2024-03-20T08:36:23.214 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-20T08:36:23.215 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-03-20T08:36:23.215 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-20T08:36:41.631 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.115 2024-03-20T08:36:44.254 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-20T08:36:44.254 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-03-20T08:36:44.255 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-20T08:36:50.636 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-03-20T08:36:50.636 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-20T08:36:53.694 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.115 2024-03-20T08:37:05.706 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-03-20T08:37:05.707 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-20T08:37:11.002 DEBUG:teuthology.orchestra.remote:timed out 2024-03-20T08:37:20.010 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-03-20T08:37:20.011 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-20T08:37:20.373 DEBUG:teuthology.orchestra.run.smithi122:> true 2024-03-20T08:37:20.695 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-03-20T08:37:20.695 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "4.18.0-546.el8.x86_64"... 2024-03-20T08:37:20.695 DEBUG:teuthology.orchestra.run.smithi122:> uname -r 2024-03-20T08:37:20.751 INFO:teuthology.orchestra.run.smithi122.stdout:4.18.0-546.el8.x86_64 2024-03-20T08:37:20.751 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-546.el8.x86_64 vs 4.18.0-546.el8.x86_64 2024-03-20T08:37:20.751 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-20T08:37:20.751 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-20T08:37:21.752 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-03-20T08:37:21.752 DEBUG:teuthology.orchestra.run.smithi122:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-20T08:37:21.831 INFO:teuthology.orchestra.run.smithi122.stdout:ttyS1 2024-03-20T08:37:21.847 DEBUG:teuthology.parallel:result is None 2024-03-20T08:37:24.126 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.115 2024-03-20T08:37:39.127 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-03-20T08:37:39.128 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-20T08:37:39.513 DEBUG:teuthology.orchestra.run.smithi115:> true 2024-03-20T08:37:39.806 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-03-20T08:37:39.806 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-546.el8.x86_64"... 2024-03-20T08:37:39.806 DEBUG:teuthology.orchestra.run.smithi115:> uname -r 2024-03-20T08:37:39.823 INFO:teuthology.orchestra.run.smithi115.stdout:4.18.0-546.el8.x86_64 2024-03-20T08:37:39.823 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-546.el8.x86_64 vs 4.18.0-546.el8.x86_64 2024-03-20T08:37:39.823 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-20T08:37:39.824 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-20T08:37:40.824 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-20T08:37:40.825 DEBUG:teuthology.orchestra.run.smithi115:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-20T08:37:40.938 INFO:teuthology.orchestra.run.smithi115.stdout:ttyS1 2024-03-20T08:37:40.953 DEBUG:teuthology.parallel:result is None 2024-03-20T08:37:44.258 DEBUG:teuthology.orchestra.remote:timed out 2024-03-20T08:37:53.259 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-03-20T08:37:53.259 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-20T08:37:53.638 DEBUG:teuthology.orchestra.run.smithi026:> true 2024-03-20T08:37:53.924 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-03-20T08:37:53.924 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-546.el8.x86_64"... 2024-03-20T08:37:53.924 DEBUG:teuthology.orchestra.run.smithi026:> uname -r 2024-03-20T08:37:53.945 INFO:teuthology.orchestra.run.smithi026.stdout:4.18.0-546.el8.x86_64 2024-03-20T08:37:53.945 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-546.el8.x86_64 vs 4.18.0-546.el8.x86_64 2024-03-20T08:37:53.945 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-20T08:37:53.945 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-20T08:37:54.946 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-20T08:37:54.946 DEBUG:teuthology.orchestra.run.smithi026:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-20T08:37:55.032 INFO:teuthology.orchestra.run.smithi026.stdout:ttyS1 2024-03-20T08:37:55.044 DEBUG:teuthology.parallel:result is None 2024-03-20T08:37:55.045 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-20T08:37:55.053 INFO:teuthology.task.internal:Creating test directory... 2024-03-20T08:37:55.054 DEBUG:teuthology.orchestra.run.smithi026:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-20T08:37:55.056 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-20T08:37:55.059 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-20T08:37:55.083 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-20T08:37:55.103 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-20T08:37:55.138 INFO:teuthology.task.internal:Creating archive directory... 2024-03-20T08:37:55.138 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-20T08:37:55.141 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-20T08:37:55.143 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-20T08:37:55.215 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-20T08:37:55.222 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-20T08:37:55.222 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-20T08:37:55.224 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-20T08:37:55.227 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-20T08:37:55.263 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-20T08:37:55.267 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-20T08:37:55.268 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-20T08:37:55.275 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-20T08:37:55.278 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-20T08:37:55.280 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-20T08:37:55.281 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-20T08:37:55.293 INFO:teuthology.task.internal:Configuring sudo... 2024-03-20T08:37:55.293 DEBUG:teuthology.orchestra.run.smithi026:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-20T08:37:55.319 DEBUG:teuthology.orchestra.run.smithi115:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-20T08:37:55.322 DEBUG:teuthology.orchestra.run.smithi122:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-20T08:37:55.352 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-20T08:37:55.361 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-20T08:37:55.362 DEBUG:teuthology.orchestra.run.smithi026:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-20T08:37:55.391 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-20T08:37:55.393 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-20T08:37:55.411 DEBUG:teuthology.orchestra.run.smithi026:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-20T08:37:55.496 DEBUG:teuthology.orchestra.run.smithi026:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-20T08:37:55.557 DEBUG:teuthology.orchestra.run.smithi026:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-20T08:37:55.645 DEBUG:teuthology.orchestra.run.smithi026:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-20T08:37:55.690 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-20T08:37:55.690 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-20T08:37:55.760 DEBUG:teuthology.orchestra.run.smithi115:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-20T08:37:55.809 DEBUG:teuthology.orchestra.run.smithi115:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-20T08:37:55.867 DEBUG:teuthology.orchestra.run.smithi115:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-20T08:37:55.912 DEBUG:teuthology.orchestra.run.smithi115:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-20T08:37:55.956 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-20T08:37:55.956 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-20T08:37:56.023 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-20T08:37:56.074 DEBUG:teuthology.orchestra.run.smithi122:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-20T08:37:56.139 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-20T08:37:56.186 DEBUG:teuthology.orchestra.run.smithi122:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-20T08:37:56.230 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-20T08:37:56.230 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-20T08:37:56.299 DEBUG:teuthology.orchestra.run.smithi026:> sudo service rsyslog restart 2024-03-20T08:37:56.301 DEBUG:teuthology.orchestra.run.smithi115:> sudo service rsyslog restart 2024-03-20T08:37:56.306 DEBUG:teuthology.orchestra.run.smithi122:> sudo service rsyslog restart 2024-03-20T08:37:56.370 INFO:teuthology.orchestra.run.smithi026.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-20T08:37:56.379 INFO:teuthology.orchestra.run.smithi115.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-20T08:37:56.413 INFO:teuthology.orchestra.run.smithi122.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-20T08:37:56.732 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-20T08:37:56.740 INFO:teuthology.task.internal:Starting timer... 2024-03-20T08:37:56.740 INFO:teuthology.run_tasks:Running task pcp... 2024-03-20T08:37:56.784 INFO:teuthology.run_tasks:Running task selinux... 2024-03-20T08:37:56.814 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-20T08:37:56.814 DEBUG:teuthology.orchestra.run.smithi026:> sudo service auditd rotate 2024-03-20T08:37:56.895 INFO:teuthology.orchestra.run.smithi026.stdout:Rotating logs: 2024-03-20T08:37:56.898 DEBUG:teuthology.orchestra.run.smithi115:> sudo service auditd rotate 2024-03-20T08:37:56.981 INFO:teuthology.orchestra.run.smithi115.stdout:Rotating logs: 2024-03-20T08:37:56.982 DEBUG:teuthology.orchestra.run.smithi122:> sudo service auditd rotate 2024-03-20T08:37:57.061 INFO:teuthology.orchestra.run.smithi122.stdout:Rotating logs: 2024-03-20T08:37:57.063 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-20T08:37:57.063 DEBUG:teuthology.orchestra.run.smithi026:> /usr/sbin/getenforce 2024-03-20T08:37:57.085 INFO:teuthology.orchestra.run.smithi026.stdout:Permissive 2024-03-20T08:37:57.085 DEBUG:teuthology.orchestra.run.smithi115:> /usr/sbin/getenforce 2024-03-20T08:37:57.117 INFO:teuthology.orchestra.run.smithi115.stdout:Permissive 2024-03-20T08:37:57.117 DEBUG:teuthology.orchestra.run.smithi122:> /usr/sbin/getenforce 2024-03-20T08:37:57.218 INFO:teuthology.orchestra.run.smithi122.stdout:Permissive 2024-03-20T08:37:57.218 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi026.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi115.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi122.front.sepia.ceph.com': 'permissive'} 2024-03-20T08:37:57.218 DEBUG:teuthology.orchestra.run.smithi026:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-20T08:37:57.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-20T08:37:57.249 DEBUG:teuthology.orchestra.run.smithi115:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-20T08:37:57.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-20T08:37:57.277 DEBUG:teuthology.orchestra.run.smithi122:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-20T08:37:57.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-20T08:37:57.304 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-20T08:37:57.304 DEBUG:teuthology.orchestra.run.smithi026:> sudo /usr/sbin/setenforce permissive 2024-03-20T08:37:57.334 DEBUG:teuthology.orchestra.run.smithi115:> sudo /usr/sbin/setenforce permissive 2024-03-20T08:37:57.361 DEBUG:teuthology.orchestra.run.smithi122:> sudo /usr/sbin/setenforce permissive 2024-03-20T08:37:57.389 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-20T08:37:57.399 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-20T08:37:57.405 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-20T08:37:57.542 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-20T08:37:57.560 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-20T08:37:57.562 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi026.front.sepia.ceph.com,smithi115.front.sepia.ceph.com,smithi122.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-20T08:44:14.913 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi026.front.sepia.ceph.com'), Remote(name='ubuntu@smithi115.front.sepia.ceph.com'), Remote(name='ubuntu@smithi122.front.sepia.ceph.com')] 2024-03-20T08:44:14.914 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-03-20T08:44:14.916 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-20T08:44:14.992 DEBUG:teuthology.orchestra.run.smithi026:> true 2024-03-20T08:44:15.063 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-03-20T08:44:15.064 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-03-20T08:44:15.064 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-20T08:44:15.137 DEBUG:teuthology.orchestra.run.smithi115:> true 2024-03-20T08:44:15.210 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi115.front.sepia.ceph.com' 2024-03-20T08:44:15.211 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-03-20T08:44:15.211 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-20T08:44:15.284 DEBUG:teuthology.orchestra.run.smithi122:> true 2024-03-20T08:44:15.353 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-03-20T08:44:15.354 INFO:teuthology.run_tasks:Running task clock... 2024-03-20T08:44:15.364 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-20T08:44:15.364 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-20T08:44:15.365 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-20T08:44:15.368 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-20T08:44:15.368 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-20T08:44:15.370 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-20T08:44:15.370 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-20T08:44:15.405 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-20T08:44:15.405 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-20T08:44:15.419 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-20T08:44:15.420 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-20T08:44:15.426 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-20T08:44:15.440 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-20T08:44:15.483 INFO:teuthology.orchestra.run.smithi115.stderr:sudo: ntpd: command not found 2024-03-20T08:44:15.491 INFO:teuthology.orchestra.run.smithi026.stderr:sudo: ntpd: command not found 2024-03-20T08:44:15.494 INFO:teuthology.orchestra.run.smithi115.stdout:506 Cannot talk to daemon 2024-03-20T08:44:15.503 INFO:teuthology.orchestra.run.smithi026.stdout:506 Cannot talk to daemon 2024-03-20T08:44:15.508 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-20T08:44:15.510 INFO:teuthology.orchestra.run.smithi122.stderr:sudo: ntpd: command not found 2024-03-20T08:44:15.518 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-20T08:44:15.521 INFO:teuthology.orchestra.run.smithi122.stdout:506 Cannot talk to daemon 2024-03-20T08:44:15.522 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-20T08:44:15.533 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-20T08:44:15.534 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-20T08:44:15.547 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-20T08:44:15.584 INFO:teuthology.orchestra.run.smithi115.stderr:bash: ntpq: command not found 2024-03-20T08:44:15.588 INFO:teuthology.orchestra.run.smithi115.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-20T08:44:15.588 INFO:teuthology.orchestra.run.smithi115.stdout:=============================================================================== 2024-03-20T08:44:15.588 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-20T08:44:15.588 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-20T08:44:15.588 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-20T08:44:15.588 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-20T08:44:15.595 INFO:teuthology.orchestra.run.smithi026.stderr:bash: ntpq: command not found 2024-03-20T08:44:15.599 INFO:teuthology.orchestra.run.smithi026.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-20T08:44:15.600 INFO:teuthology.orchestra.run.smithi026.stdout:=============================================================================== 2024-03-20T08:44:15.600 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-20T08:44:15.600 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-20T08:44:15.600 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-20T08:44:15.600 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-20T08:44:15.618 INFO:teuthology.orchestra.run.smithi122.stderr:bash: ntpq: command not found 2024-03-20T08:44:15.623 INFO:teuthology.orchestra.run.smithi122.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-20T08:44:15.623 INFO:teuthology.orchestra.run.smithi122.stdout:=============================================================================== 2024-03-20T08:44:15.623 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-20T08:44:15.623 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-20T08:44:15.623 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-20T08:44:15.623 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-20T08:44:15.624 INFO:teuthology.run_tasks:Running task pexec... 2024-03-20T08:44:15.634 INFO:teuthology.task.pexec:Executing custom commands... 2024-03-20T08:44:15.634 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi026.front.sepia.ceph.com 2024-03-20T08:44:15.634 DEBUG:teuthology.orchestra.run.smithi026:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-20T08:44:15.635 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi115.front.sepia.ceph.com 2024-03-20T08:44:15.635 DEBUG:teuthology.orchestra.run.smithi115:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-20T08:44:15.635 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi122.front.sepia.ceph.com 2024-03-20T08:44:15.636 DEBUG:teuthology.orchestra.run.smithi122:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-20T08:44:16.394 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:43 ago on Wed 20 Mar 2024 08:42:33 AM UTC. 2024-03-20T08:44:16.425 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:43 ago on Wed 20 Mar 2024 08:42:33 AM UTC. 2024-03-20T08:44:16.433 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:36 ago on Wed 20 Mar 2024 08:42:40 AM UTC. 2024-03-20T08:44:18.492 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-20T08:44:18.493 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:44:18.493 INFO:teuthology.orchestra.run.smithi115.stdout: Package Architecture Version Repository Size 2024-03-20T08:44:18.494 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:44:18.494 INFO:teuthology.orchestra.run.smithi115.stdout:Resetting modules: 2024-03-20T08:44:18.494 INFO:teuthology.orchestra.run.smithi115.stdout: container-tools 2024-03-20T08:44:18.494 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:44:18.494 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-20T08:44:18.494 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:44:18.495 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:44:18.542 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-20T08:44:18.543 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:44:18.543 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2024-03-20T08:44:18.544 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:44:18.544 INFO:teuthology.orchestra.run.smithi122.stdout:Resetting modules: 2024-03-20T08:44:18.544 INFO:teuthology.orchestra.run.smithi122.stdout: container-tools 2024-03-20T08:44:18.544 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:44:18.544 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-20T08:44:18.544 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:44:18.544 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:44:18.713 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-20T08:44:18.714 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:44:18.714 INFO:teuthology.orchestra.run.smithi026.stdout: Package Architecture Version Repository Size 2024-03-20T08:44:18.714 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:44:18.714 INFO:teuthology.orchestra.run.smithi026.stdout:Resetting modules: 2024-03-20T08:44:18.714 INFO:teuthology.orchestra.run.smithi026.stdout: container-tools 2024-03-20T08:44:18.714 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:44:18.715 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-20T08:44:18.715 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:44:18.715 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:44:18.764 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-20T08:44:18.788 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-20T08:44:19.132 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-20T08:44:19.575 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:46 ago on Wed 20 Mar 2024 08:42:33 AM UTC. 2024-03-20T08:44:19.627 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:46 ago on Wed 20 Mar 2024 08:42:33 AM UTC. 2024-03-20T08:44:20.285 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:40 ago on Wed 20 Mar 2024 08:42:40 AM UTC. 2024-03-20T08:44:21.695 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-20T08:44:21.732 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:44:21.732 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2024-03-20T08:44:21.733 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:44:21.733 INFO:teuthology.orchestra.run.smithi115.stdout:Installing group/module packages: 2024-03-20T08:44:21.733 INFO:teuthology.orchestra.run.smithi115.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-03-20T08:44:21.733 INFO:teuthology.orchestra.run.smithi115.stdout: buildah x86_64 2:1.33.6-2.module_el8+976+11360731 appstream 9.6 M 2024-03-20T08:44:21.733 INFO:teuthology.orchestra.run.smithi115.stdout: cockpit-podman noarch 84.1-1.module_el8+977+8ab54275 appstream 689 k 2024-03-20T08:44:21.733 INFO:teuthology.orchestra.run.smithi115.stdout: crun x86_64 1.14.3-2.module_el8+968+fbb249c7 appstream 258 k 2024-03-20T08:44:21.733 INFO:teuthology.orchestra.run.smithi115.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-03-20T08:44:21.733 INFO:teuthology.orchestra.run.smithi115.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-03-20T08:44:21.733 INFO:teuthology.orchestra.run.smithi115.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-03-20T08:44:21.733 INFO:teuthology.orchestra.run.smithi115.stdout: toolbox x86_64 0.0.99.5-2.module_el8+971+3d3df00d appstream 2.5 M 2024-03-20T08:44:21.733 INFO:teuthology.orchestra.run.smithi115.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-03-20T08:44:21.733 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-03-20T08:44:21.733 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-03-20T08:44:21.733 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-03-20T08:44:21.734 INFO:teuthology.orchestra.run.smithi115.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-03-20T08:44:21.734 INFO:teuthology.orchestra.run.smithi115.stdout:Installing weak dependencies: 2024-03-20T08:44:21.734 INFO:teuthology.orchestra.run.smithi115.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-03-20T08:44:21.734 INFO:teuthology.orchestra.run.smithi115.stdout:Installing module profiles: 2024-03-20T08:44:21.734 INFO:teuthology.orchestra.run.smithi115.stdout: container-tools/common 2024-03-20T08:44:21.734 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:44:21.734 INFO:teuthology.orchestra.run.smithi115.stdout:Enabling module streams: 2024-03-20T08:44:21.734 INFO:teuthology.orchestra.run.smithi115.stdout: container-tools rhel8 2024-03-20T08:44:21.734 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:44:21.735 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-20T08:44:21.735 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:44:21.735 INFO:teuthology.orchestra.run.smithi115.stdout:Install 13 Packages 2024-03-20T08:44:21.735 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:44:21.735 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 27 M 2024-03-20T08:44:21.735 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 91 M 2024-03-20T08:44:21.735 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-20T08:44:21.745 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-20T08:44:21.782 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:44:21.782 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-03-20T08:44:21.782 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:44:21.782 INFO:teuthology.orchestra.run.smithi122.stdout:Installing group/module packages: 2024-03-20T08:44:21.783 INFO:teuthology.orchestra.run.smithi122.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-03-20T08:44:21.783 INFO:teuthology.orchestra.run.smithi122.stdout: buildah x86_64 2:1.33.6-2.module_el8+976+11360731 appstream 9.6 M 2024-03-20T08:44:21.783 INFO:teuthology.orchestra.run.smithi122.stdout: cockpit-podman noarch 84.1-1.module_el8+977+8ab54275 appstream 689 k 2024-03-20T08:44:21.783 INFO:teuthology.orchestra.run.smithi122.stdout: crun x86_64 1.14.3-2.module_el8+968+fbb249c7 appstream 258 k 2024-03-20T08:44:21.783 INFO:teuthology.orchestra.run.smithi122.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-03-20T08:44:21.783 INFO:teuthology.orchestra.run.smithi122.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-03-20T08:44:21.783 INFO:teuthology.orchestra.run.smithi122.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-03-20T08:44:21.783 INFO:teuthology.orchestra.run.smithi122.stdout: toolbox x86_64 0.0.99.5-2.module_el8+971+3d3df00d appstream 2.5 M 2024-03-20T08:44:21.783 INFO:teuthology.orchestra.run.smithi122.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-03-20T08:44:21.783 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-03-20T08:44:21.783 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-03-20T08:44:21.783 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-03-20T08:44:21.783 INFO:teuthology.orchestra.run.smithi122.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-03-20T08:44:21.783 INFO:teuthology.orchestra.run.smithi122.stdout:Installing weak dependencies: 2024-03-20T08:44:21.784 INFO:teuthology.orchestra.run.smithi122.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-03-20T08:44:21.784 INFO:teuthology.orchestra.run.smithi122.stdout:Installing module profiles: 2024-03-20T08:44:21.784 INFO:teuthology.orchestra.run.smithi122.stdout: container-tools/common 2024-03-20T08:44:21.784 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:44:21.784 INFO:teuthology.orchestra.run.smithi122.stdout:Enabling module streams: 2024-03-20T08:44:21.784 INFO:teuthology.orchestra.run.smithi122.stdout: container-tools rhel8 2024-03-20T08:44:21.784 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:44:21.784 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-20T08:44:21.784 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:44:21.784 INFO:teuthology.orchestra.run.smithi122.stdout:Install 13 Packages 2024-03-20T08:44:21.784 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:44:21.785 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 27 M 2024-03-20T08:44:21.785 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 91 M 2024-03-20T08:44:21.785 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-20T08:44:22.088 INFO:teuthology.orchestra.run.smithi115.stdout:(1/13): cockpit-podman-84.1-1.module_el8+977+8a 2.2 MB/s | 689 kB 00:00 2024-03-20T08:44:22.121 INFO:teuthology.orchestra.run.smithi115.stdout:(2/13): aardvark-dns-1.10.0-1.module_el8+871+55 3.0 MB/s | 1.0 MB 00:00 2024-03-20T08:44:22.146 INFO:teuthology.orchestra.run.smithi115.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 647 kB/s | 38 kB 00:00 2024-03-20T08:44:22.180 INFO:teuthology.orchestra.run.smithi115.stdout:(4/13): crun-1.14.3-2.module_el8+968+fbb249c7.x 4.3 MB/s | 258 kB 00:00 2024-03-20T08:44:22.239 INFO:teuthology.orchestra.run.smithi115.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 3.0 MB/s | 180 kB 00:00 2024-03-20T08:44:22.315 INFO:teuthology.orchestra.run.smithi122.stdout:(1/13): cockpit-podman-84.1-1.module_el8+977+8a 2.1 MB/s | 689 kB 00:00 2024-03-20T08:44:22.329 INFO:teuthology.orchestra.run.smithi115.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 275 kB/s | 25 kB 00:00 2024-03-20T08:44:22.347 INFO:teuthology.orchestra.run.smithi122.stdout:(2/13): aardvark-dns-1.10.0-1.module_el8+871+55 2.9 MB/s | 1.0 MB 00:00 2024-03-20T08:44:22.372 INFO:teuthology.orchestra.run.smithi122.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 673 kB/s | 38 kB 00:00 2024-03-20T08:44:22.389 INFO:teuthology.orchestra.run.smithi115.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.5 MB/s | 94 kB 00:00 2024-03-20T08:44:22.405 INFO:teuthology.orchestra.run.smithi122.stdout:(4/13): crun-1.14.3-2.module_el8+968+fbb249c7.x 4.3 MB/s | 258 kB 00:00 2024-03-20T08:44:22.472 INFO:teuthology.orchestra.run.smithi122.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 2.6 MB/s | 180 kB 00:00 2024-03-20T08:44:22.523 INFO:teuthology.orchestra.run.smithi122.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 494 kB/s | 25 kB 00:00 2024-03-20T08:44:22.560 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-20T08:44:22.600 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:44:22.600 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-03-20T08:44:22.600 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:44:22.601 INFO:teuthology.orchestra.run.smithi026.stdout:Installing group/module packages: 2024-03-20T08:44:22.601 INFO:teuthology.orchestra.run.smithi026.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-03-20T08:44:22.601 INFO:teuthology.orchestra.run.smithi026.stdout: buildah x86_64 2:1.33.6-2.module_el8+976+11360731 appstream 9.6 M 2024-03-20T08:44:22.601 INFO:teuthology.orchestra.run.smithi026.stdout: cockpit-podman noarch 84.1-1.module_el8+977+8ab54275 appstream 689 k 2024-03-20T08:44:22.601 INFO:teuthology.orchestra.run.smithi026.stdout: crun x86_64 1.14.3-2.module_el8+968+fbb249c7 appstream 258 k 2024-03-20T08:44:22.601 INFO:teuthology.orchestra.run.smithi026.stdout: netavark x86_64 2:1.10.3-1.module_el8+951+32019cde appstream 4.1 M 2024-03-20T08:44:22.601 INFO:teuthology.orchestra.run.smithi026.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-03-20T08:44:22.601 INFO:teuthology.orchestra.run.smithi026.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-03-20T08:44:22.601 INFO:teuthology.orchestra.run.smithi026.stdout: toolbox x86_64 0.0.99.5-2.module_el8+971+3d3df00d appstream 2.5 M 2024-03-20T08:44:22.601 INFO:teuthology.orchestra.run.smithi026.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-03-20T08:44:22.601 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-03-20T08:44:22.601 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-03-20T08:44:22.601 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-03-20T08:44:22.601 INFO:teuthology.orchestra.run.smithi026.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-03-20T08:44:22.602 INFO:teuthology.orchestra.run.smithi026.stdout:Installing weak dependencies: 2024-03-20T08:44:22.602 INFO:teuthology.orchestra.run.smithi026.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-03-20T08:44:22.602 INFO:teuthology.orchestra.run.smithi026.stdout:Installing module profiles: 2024-03-20T08:44:22.602 INFO:teuthology.orchestra.run.smithi026.stdout: container-tools/common 2024-03-20T08:44:22.602 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:44:22.602 INFO:teuthology.orchestra.run.smithi026.stdout:Enabling module streams: 2024-03-20T08:44:22.602 INFO:teuthology.orchestra.run.smithi026.stdout: container-tools rhel8 2024-03-20T08:44:22.602 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:44:22.602 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-20T08:44:22.602 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:44:22.602 INFO:teuthology.orchestra.run.smithi026.stdout:Install 13 Packages 2024-03-20T08:44:22.602 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:44:22.603 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 27 M 2024-03-20T08:44:22.603 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 91 M 2024-03-20T08:44:22.603 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-20T08:44:22.606 INFO:teuthology.orchestra.run.smithi122.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.1 MB/s | 94 kB 00:00 2024-03-20T08:44:22.656 INFO:teuthology.orchestra.run.smithi115.stdout:(8/13): netavark-1.10.3-1.module_el8+951+32019c 8.1 MB/s | 4.1 MB 00:00 2024-03-20T08:44:22.765 INFO:teuthology.orchestra.run.smithi115.stdout:(9/13): buildah-1.33.6-2.module_el8+976+1136073 9.9 MB/s | 9.6 MB 00:00 2024-03-20T08:44:22.807 INFO:teuthology.orchestra.run.smithi122.stdout:(8/13): netavark-1.10.3-1.module_el8+951+32019c 9.4 MB/s | 4.1 MB 00:00 2024-03-20T08:44:22.840 INFO:teuthology.orchestra.run.smithi115.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 702 kB/s | 53 kB 00:00 2024-03-20T08:44:22.907 INFO:teuthology.orchestra.run.smithi115.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 616 kB/s | 41 kB 00:00 2024-03-20T08:44:22.957 INFO:teuthology.orchestra.run.smithi122.stdout:(9/13): buildah-1.33.6-2.module_el8+976+1136073 10 MB/s | 9.6 MB 00:00 2024-03-20T08:44:22.982 INFO:teuthology.orchestra.run.smithi115.stdout:(12/13): toolbox-0.0.99.5-2.module_el8+971+3d3d 7.8 MB/s | 2.5 MB 00:00 2024-03-20T08:44:23.007 INFO:teuthology.orchestra.run.smithi122.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 1.0 MB/s | 53 kB 00:00 2024-03-20T08:44:23.066 INFO:teuthology.orchestra.run.smithi122.stdout:(11/13): toolbox-0.0.99.5-2.module_el8+971+3d3d 9.8 MB/s | 2.5 MB 00:00 2024-03-20T08:44:23.083 INFO:teuthology.orchestra.run.smithi122.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 542 kB/s | 41 kB 00:00 2024-03-20T08:44:23.125 INFO:teuthology.orchestra.run.smithi026.stdout:(1/13): cockpit-podman-84.1-1.module_el8+977+8a 1.4 MB/s | 689 kB 00:00 2024-03-20T08:44:23.175 INFO:teuthology.orchestra.run.smithi026.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 760 kB/s | 38 kB 00:00 2024-03-20T08:44:23.182 INFO:teuthology.orchestra.run.smithi115.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 11 MB/s | 8.8 MB 00:00 2024-03-20T08:44:23.183 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:44:23.183 INFO:teuthology.orchestra.run.smithi115.stdout:Total 19 MB/s | 27 MB 00:01 2024-03-20T08:44:23.217 INFO:teuthology.orchestra.run.smithi026.stdout:(3/13): aardvark-dns-1.10.0-1.module_el8+871+55 1.8 MB/s | 1.0 MB 00:00 2024-03-20T08:44:23.301 INFO:teuthology.orchestra.run.smithi026.stdout:(4/13): crun-1.14.3-2.module_el8+968+fbb249c7.x 2.0 MB/s | 258 kB 00:00 2024-03-20T08:44:23.308 INFO:teuthology.orchestra.run.smithi122.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 13 MB/s | 8.8 MB 00:00 2024-03-20T08:44:23.309 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:44:23.309 INFO:teuthology.orchestra.run.smithi122.stdout:Total 18 MB/s | 27 MB 00:01 2024-03-20T08:44:23.412 INFO:teuthology.orchestra.run.smithi026.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 1.6 MB/s | 180 kB 00:00 2024-03-20T08:44:23.452 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-20T08:44:23.468 INFO:teuthology.orchestra.run.smithi026.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 446 kB/s | 25 kB 00:00 2024-03-20T08:44:23.474 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-20T08:44:23.474 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-20T08:44:23.527 INFO:teuthology.orchestra.run.smithi026.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.6 MB/s | 94 kB 00:00 2024-03-20T08:44:23.577 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-20T08:44:23.598 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-20T08:44:23.599 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-20T08:44:23.733 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-20T08:44:23.736 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-20T08:44:23.861 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-20T08:44:23.865 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-20T08:44:23.894 INFO:teuthology.orchestra.run.smithi026.stdout:(8/13): netavark-1.10.3-1.module_el8+951+32019c 6.1 MB/s | 4.1 MB 00:00 2024-03-20T08:44:24.144 INFO:teuthology.orchestra.run.smithi026.stdout:(9/13): toolbox-0.0.99.5-2.module_el8+971+3d3df 10 MB/s | 2.5 MB 00:00 2024-03-20T08:44:24.244 INFO:teuthology.orchestra.run.smithi026.stdout:(10/13): buildah-1.33.6-2.module_el8+976+113607 6.1 MB/s | 9.6 MB 00:01 2024-03-20T08:44:24.270 INFO:teuthology.orchestra.run.smithi026.stdout:(11/13): udica-0.2.6-20.module_el8+804+f131391c 420 kB/s | 53 kB 00:00 2024-03-20T08:44:24.303 INFO:teuthology.orchestra.run.smithi026.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 702 kB/s | 41 kB 00:00 2024-03-20T08:44:24.415 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-20T08:44:24.587 INFO:teuthology.orchestra.run.smithi026.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 8.3 MB/s | 8.8 MB 00:01 2024-03-20T08:44:24.587 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:44:24.587 INFO:teuthology.orchestra.run.smithi026.stdout:Total 14 MB/s | 27 MB 00:01 2024-03-20T08:44:24.595 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-20T08:44:24.865 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-20T08:44:24.888 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-20T08:44:24.888 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-20T08:44:25.177 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-20T08:44:25.181 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-20T08:44:25.338 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-03-20T08:44:25.523 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-03-20T08:44:25.543 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-03-20T08:44:25.726 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-03-20T08:44:25.746 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-03-20T08:44:25.897 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-03-20T08:44:25.950 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-03-20T08:44:25.959 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-20T08:44:26.119 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-03-20T08:44:26.383 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-03-20T08:44:26.624 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-03-20T08:44:26.677 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-03-20T08:44:26.950 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-03-20T08:44:27.156 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-03-20T08:44:27.211 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-03-20T08:44:27.367 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-03-20T08:44:27.597 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-03-20T08:44:27.649 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-03-20T08:44:27.843 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-03-20T08:44:27.856 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 8/13 2024-03-20T08:44:28.118 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 9/13 2024-03-20T08:44:28.202 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-03-20T08:44:28.434 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-03-20T08:44:28.436 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 8/13 2024-03-20T08:44:28.632 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 9/13 2024-03-20T08:44:28.904 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-03-20T08:44:29.098 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-03-20T08:44:29.139 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-03-20T08:44:29.277 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-03-20T08:44:29.355 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-03-20T08:44:29.499 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-03-20T08:44:29.500 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 7/13 2024-03-20T08:44:29.583 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-03-20T08:44:29.714 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-03-20T08:44:29.819 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 8/13 2024-03-20T08:44:30.011 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-03-20T08:44:30.011 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-03-20T08:44:30.011 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2/13 2024-03-20T08:44:30.011 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 3/13 2024-03-20T08:44:30.011 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-03-20T08:44:30.011 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 5/13 2024-03-20T08:44:30.011 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-03-20T08:44:30.012 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-03-20T08:44:30.012 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-03-20T08:44:30.012 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-03-20T08:44:30.012 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-03-20T08:44:30.012 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-03-20T08:44:30.012 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-03-20T08:44:30.035 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 9/13 2024-03-20T08:44:30.139 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-03-20T08:44:30.139 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-03-20T08:44:30.139 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2/13 2024-03-20T08:44:30.139 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 3/13 2024-03-20T08:44:30.139 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-03-20T08:44:30.139 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 5/13 2024-03-20T08:44:30.139 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-03-20T08:44:30.139 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-03-20T08:44:30.139 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-03-20T08:44:30.139 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-03-20T08:44:30.140 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-03-20T08:44:30.140 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-03-20T08:44:30.140 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-03-20T08:44:30.496 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-03-20T08:44:30.551 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-03-20T08:44:30.551 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:44:30.551 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-20T08:44:30.551 INFO:teuthology.orchestra.run.smithi115.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-03-20T08:44:30.551 INFO:teuthology.orchestra.run.smithi115.stdout: buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2024-03-20T08:44:30.551 INFO:teuthology.orchestra.run.smithi115.stdout: cockpit-podman-84.1-1.module_el8+977+8ab54275.noarch 2024-03-20T08:44:30.551 INFO:teuthology.orchestra.run.smithi115.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-03-20T08:44:30.552 INFO:teuthology.orchestra.run.smithi115.stdout: crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 2024-03-20T08:44:30.552 INFO:teuthology.orchestra.run.smithi115.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-03-20T08:44:30.552 INFO:teuthology.orchestra.run.smithi115.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-03-20T08:44:30.552 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-03-20T08:44:30.552 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-03-20T08:44:30.552 INFO:teuthology.orchestra.run.smithi115.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-03-20T08:44:30.552 INFO:teuthology.orchestra.run.smithi115.stdout: toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 2024-03-20T08:44:30.552 INFO:teuthology.orchestra.run.smithi115.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-03-20T08:44:30.552 INFO:teuthology.orchestra.run.smithi115.stdout: yajl-2.1.0-12.el8.x86_64 2024-03-20T08:44:30.552 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:44:30.552 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-20T08:44:30.644 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-03-20T08:44:30.645 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:44:30.645 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-20T08:44:30.645 INFO:teuthology.orchestra.run.smithi122.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-03-20T08:44:30.645 INFO:teuthology.orchestra.run.smithi122.stdout: buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2024-03-20T08:44:30.645 INFO:teuthology.orchestra.run.smithi122.stdout: cockpit-podman-84.1-1.module_el8+977+8ab54275.noarch 2024-03-20T08:44:30.645 INFO:teuthology.orchestra.run.smithi122.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-03-20T08:44:30.645 INFO:teuthology.orchestra.run.smithi122.stdout: crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 2024-03-20T08:44:30.645 INFO:teuthology.orchestra.run.smithi122.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-03-20T08:44:30.645 INFO:teuthology.orchestra.run.smithi122.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-03-20T08:44:30.645 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-03-20T08:44:30.645 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-03-20T08:44:30.645 INFO:teuthology.orchestra.run.smithi122.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-03-20T08:44:30.645 INFO:teuthology.orchestra.run.smithi122.stdout: toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 2024-03-20T08:44:30.645 INFO:teuthology.orchestra.run.smithi122.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-03-20T08:44:30.646 INFO:teuthology.orchestra.run.smithi122.stdout: yajl-2.1.0-12.el8.x86_64 2024-03-20T08:44:30.646 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:44:30.646 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-20T08:44:30.743 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-03-20T08:44:30.804 DEBUG:teuthology.parallel:result is None 2024-03-20T08:44:30.899 DEBUG:teuthology.parallel:result is None 2024-03-20T08:44:30.960 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-03-20T08:44:31.191 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-03-20T08:44:31.642 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 13/13 2024-03-20T08:44:31.643 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-03-20T08:44:31.643 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2/13 2024-03-20T08:44:31.643 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : cockpit-podman-84.1-1.module_el8+977+8ab54275.noar 3/13 2024-03-20T08:44:31.643 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-03-20T08:44:31.643 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 5/13 2024-03-20T08:44:31.643 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 6/13 2024-03-20T08:44:31.643 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-03-20T08:44:31.643 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-03-20T08:44:31.643 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-03-20T08:44:31.643 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-03-20T08:44:31.643 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 11/13 2024-03-20T08:44:31.643 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-03-20T08:44:32.242 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-03-20T08:44:32.242 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:44:32.243 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-20T08:44:32.243 INFO:teuthology.orchestra.run.smithi026.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-03-20T08:44:32.243 INFO:teuthology.orchestra.run.smithi026.stdout: buildah-2:1.33.6-2.module_el8+976+11360731.x86_64 2024-03-20T08:44:32.243 INFO:teuthology.orchestra.run.smithi026.stdout: cockpit-podman-84.1-1.module_el8+977+8ab54275.noarch 2024-03-20T08:44:32.243 INFO:teuthology.orchestra.run.smithi026.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-03-20T08:44:32.243 INFO:teuthology.orchestra.run.smithi026.stdout: crun-1.14.3-2.module_el8+968+fbb249c7.x86_64 2024-03-20T08:44:32.243 INFO:teuthology.orchestra.run.smithi026.stdout: netavark-2:1.10.3-1.module_el8+951+32019cde.x86_64 2024-03-20T08:44:32.243 INFO:teuthology.orchestra.run.smithi026.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-03-20T08:44:32.243 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-03-20T08:44:32.243 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-03-20T08:44:32.243 INFO:teuthology.orchestra.run.smithi026.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-03-20T08:44:32.243 INFO:teuthology.orchestra.run.smithi026.stdout: toolbox-0.0.99.5-2.module_el8+971+3d3df00d.x86_64 2024-03-20T08:44:32.243 INFO:teuthology.orchestra.run.smithi026.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-03-20T08:44:32.244 INFO:teuthology.orchestra.run.smithi026.stdout: yajl-2.1.0-12.el8.x86_64 2024-03-20T08:44:32.244 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:44:32.244 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-20T08:44:32.505 DEBUG:teuthology.parallel:result is None 2024-03-20T08:44:32.505 INFO:teuthology.run_tasks:Running task install... 2024-03-20T08:44:32.516 DEBUG:teuthology.task.install:project ceph 2024-03-20T08:44:32.516 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'e7e80a6ac9849f0f23432d0d9832f096730308ae'}} 2024-03-20T08:44:32.516 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'flavor': 'default', 'sha1': 'e7e80a6ac9849f0f23432d0d9832f096730308ae'} 2024-03-20T08:44:32.516 INFO:teuthology.task.install:Using flavor: default 2024-03-20T08:44:32.523 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-20T08:44:32.523 INFO:teuthology.task.install:extra packages: [] 2024-03-20T08:44:32.523 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e7e80a6ac9849f0f23432d0d9832f096730308ae', 'tag': None, 'wait_for_package': False} 2024-03-20T08:44:32.524 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-20T08:44:32.524 INFO:teuthology.packaging:ref: None 2024-03-20T08:44:32.524 INFO:teuthology.packaging:tag: None 2024-03-20T08:44:32.524 INFO:teuthology.packaging:branch: quincy 2024-03-20T08:44:32.524 INFO:teuthology.packaging:sha1: e7e80a6ac9849f0f23432d0d9832f096730308ae 2024-03-20T08:44:32.524 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-03-20T08:44:32.527 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e7e80a6ac9849f0f23432d0d9832f096730308ae', 'tag': None, 'wait_for_package': False} 2024-03-20T08:44:32.527 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-20T08:44:32.527 INFO:teuthology.packaging:ref: None 2024-03-20T08:44:32.527 INFO:teuthology.packaging:tag: None 2024-03-20T08:44:32.527 INFO:teuthology.packaging:branch: quincy 2024-03-20T08:44:32.527 INFO:teuthology.packaging:sha1: e7e80a6ac9849f0f23432d0d9832f096730308ae 2024-03-20T08:44:32.528 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-03-20T08:44:32.529 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e7e80a6ac9849f0f23432d0d9832f096730308ae', 'tag': None, 'wait_for_package': False} 2024-03-20T08:44:32.529 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-20T08:44:32.530 INFO:teuthology.packaging:ref: None 2024-03-20T08:44:32.530 INFO:teuthology.packaging:tag: None 2024-03-20T08:44:32.530 INFO:teuthology.packaging:branch: quincy 2024-03-20T08:44:32.530 INFO:teuthology.packaging:sha1: e7e80a6ac9849f0f23432d0d9832f096730308ae 2024-03-20T08:44:32.530 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-03-20T08:44:32.794 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/quincy/0192297efae4bac0733401bcd9f7897b92ebe40a/centos/8/flavors/default/ 2024-03-20T08:44:32.794 INFO:teuthology.task.install.rpm:Package version is 17.2.7-717.g0192297e 2024-03-20T08:44:32.853 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/quincy/0192297efae4bac0733401bcd9f7897b92ebe40a/centos/8/flavors/default/ 2024-03-20T08:44:32.854 INFO:teuthology.task.install.rpm:Package version is 17.2.7-717.g0192297e 2024-03-20T08:44:32.899 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/quincy/0192297efae4bac0733401bcd9f7897b92ebe40a/centos/8/flavors/default/ 2024-03-20T08:44:32.900 INFO:teuthology.task.install.rpm:Package version is 17.2.7-717.g0192297e 2024-03-20T08:44:32.908 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/quincy/0192297efae4bac0733401bcd9f7897b92ebe40a/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/quincy/0192297efae4bac0733401bcd9f7897b92ebe40a/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/quincy/0192297efae4bac0733401bcd9f7897b92ebe40a/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-20T08:44:32.908 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-20T08:44:32.908 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-20T08:44:32.943 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-20T08:44:32.943 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-20T08:44:32.944 INFO:teuthology.packaging:ref: None 2024-03-20T08:44:32.944 INFO:teuthology.packaging:tag: None 2024-03-20T08:44:32.944 INFO:teuthology.packaging:branch: quincy 2024-03-20T08:44:32.944 INFO:teuthology.packaging:sha1: e7e80a6ac9849f0f23432d0d9832f096730308ae 2024-03-20T08:44:32.944 DEBUG:teuthology.orchestra.run.smithi115:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-20T08:44:32.970 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/quincy/0192297efae4bac0733401bcd9f7897b92ebe40a/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/quincy/0192297efae4bac0733401bcd9f7897b92ebe40a/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/quincy/0192297efae4bac0733401bcd9f7897b92ebe40a/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-20T08:44:32.970 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-20T08:44:32.970 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-20T08:44:33.001 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-20T08:44:33.001 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-20T08:44:33.001 INFO:teuthology.packaging:ref: None 2024-03-20T08:44:33.001 INFO:teuthology.packaging:tag: None 2024-03-20T08:44:33.001 INFO:teuthology.packaging:branch: quincy 2024-03-20T08:44:33.001 INFO:teuthology.packaging:sha1: e7e80a6ac9849f0f23432d0d9832f096730308ae 2024-03-20T08:44:33.001 DEBUG:teuthology.orchestra.run.smithi026:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-20T08:44:33.015 DEBUG:teuthology.orchestra.run.smithi115:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-20T08:44:33.074 DEBUG:teuthology.orchestra.run.smithi026:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-20T08:44:33.076 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/quincy/0192297efae4bac0733401bcd9f7897b92ebe40a/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/quincy/0192297efae4bac0733401bcd9f7897b92ebe40a/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/quincy/0192297efae4bac0733401bcd9f7897b92ebe40a/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-20T08:44:33.077 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-20T08:44:33.077 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-20T08:44:33.096 DEBUG:teuthology.orchestra.run.smithi115:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-20T08:44:33.110 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-20T08:44:33.110 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-20T08:44:33.111 INFO:teuthology.packaging:ref: None 2024-03-20T08:44:33.111 INFO:teuthology.packaging:tag: None 2024-03-20T08:44:33.111 INFO:teuthology.packaging:branch: quincy 2024-03-20T08:44:33.111 INFO:teuthology.packaging:sha1: e7e80a6ac9849f0f23432d0d9832f096730308ae 2024-03-20T08:44:33.111 DEBUG:teuthology.orchestra.run.smithi122:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-20T08:44:33.154 DEBUG:teuthology.orchestra.run.smithi026:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-20T08:44:33.166 INFO:teuthology.orchestra.run.smithi115.stdout:check_obsoletes = 1 2024-03-20T08:44:33.167 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum clean all 2024-03-20T08:44:33.184 DEBUG:teuthology.orchestra.run.smithi122:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-20T08:44:33.223 INFO:teuthology.orchestra.run.smithi026.stdout:check_obsoletes = 1 2024-03-20T08:44:33.224 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum clean all 2024-03-20T08:44:33.269 DEBUG:teuthology.orchestra.run.smithi122:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-20T08:44:33.301 INFO:teuthology.orchestra.run.smithi122.stdout:check_obsoletes = 1 2024-03-20T08:44:33.303 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum clean all 2024-03-20T08:44:33.437 INFO:teuthology.orchestra.run.smithi115.stdout:76 files removed 2024-03-20T08:44:33.464 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-radosgw 2024-03-20T08:44:33.515 INFO:teuthology.orchestra.run.smithi026.stdout:76 files removed 2024-03-20T08:44:33.544 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-radosgw 2024-03-20T08:44:33.590 INFO:teuthology.orchestra.run.smithi122.stdout:76 files removed 2024-03-20T08:44:33.617 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-radosgw 2024-03-20T08:44:34.131 INFO:teuthology.orchestra.run.smithi115.stdout:ceph packages for x86_64 295 kB/s | 78 kB 00:00 2024-03-20T08:44:34.279 INFO:teuthology.orchestra.run.smithi026.stdout:ceph packages for x86_64 254 kB/s | 78 kB 00:00 2024-03-20T08:44:34.326 INFO:teuthology.orchestra.run.smithi122.stdout:ceph packages for x86_64 256 kB/s | 78 kB 00:00 2024-03-20T08:44:34.466 INFO:teuthology.orchestra.run.smithi115.stdout:ceph noarch packages 31 kB/s | 9.1 kB 00:00 2024-03-20T08:44:34.597 INFO:teuthology.orchestra.run.smithi026.stdout:ceph noarch packages 33 kB/s | 9.1 kB 00:00 2024-03-20T08:44:34.635 INFO:teuthology.orchestra.run.smithi122.stdout:ceph noarch packages 34 kB/s | 9.1 kB 00:00 2024-03-20T08:44:34.816 INFO:teuthology.orchestra.run.smithi115.stdout:ceph source packages 4.5 kB/s | 1.4 kB 00:00 2024-03-20T08:44:34.965 INFO:teuthology.orchestra.run.smithi026.stdout:ceph source packages 4.3 kB/s | 1.4 kB 00:00 2024-03-20T08:44:34.994 INFO:teuthology.orchestra.run.smithi122.stdout:ceph source packages 4.4 kB/s | 1.4 kB 00:00 2024-03-20T08:44:36.353 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 8 - AppStream 21 MB/s | 28 MB 00:01 2024-03-20T08:44:37.924 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - AppStream 9.0 MB/s | 28 MB 00:03 2024-03-20T08:44:40.690 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 8 - AppStream 4.9 MB/s | 28 MB 00:05 2024-03-20T08:44:44.484 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - BaseOS 6.7 MB/s | 10 MB 00:01 2024-03-20T08:44:46.283 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - Extras 80 kB/s | 18 kB 00:00 2024-03-20T08:44:46.586 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 8 - BaseOS 1.9 MB/s | 10 MB 00:05 2024-03-20T08:44:46.800 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - Extras common packages 18 kB/s | 7.5 kB 00:00 2024-03-20T08:44:48.493 INFO:teuthology.orchestra.run.smithi115.stdout:CentOS Stream 8 - PowerTools 3.1 MB/s | 4.9 MB 00:01 2024-03-20T08:44:48.503 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 8 - Extras 54 kB/s | 18 kB 00:00 2024-03-20T08:44:48.915 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 8 - Extras common packages 25 kB/s | 7.5 kB 00:00 2024-03-20T08:44:49.645 INFO:teuthology.orchestra.run.smithi115.stdout:Copr repo for python3-asyncssh owned by ceph 16 kB/s | 3.5 kB 00:00 2024-03-20T08:44:49.845 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 8 - BaseOS 2.7 MB/s | 10 MB 00:03 2024-03-20T08:44:50.416 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 8 - PowerTools 3.5 MB/s | 4.9 MB 00:01 2024-03-20T08:44:50.837 INFO:teuthology.orchestra.run.smithi115.stdout:Extra Packages for Enterprise Linux 15 MB/s | 16 MB 00:01 2024-03-20T08:44:51.651 INFO:teuthology.orchestra.run.smithi122.stdout:Copr repo for python3-asyncssh owned by ceph 11 kB/s | 3.5 kB 00:00 2024-03-20T08:44:51.923 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 8 - Extras 55 kB/s | 18 kB 00:00 2024-03-20T08:44:52.407 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 8 - Extras common packages 20 kB/s | 7.5 kB 00:00 2024-03-20T08:44:53.193 INFO:teuthology.orchestra.run.smithi122.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2024-03-20T08:44:53.967 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 8 - PowerTools 3.4 MB/s | 4.9 MB 00:01 2024-03-20T08:44:54.600 INFO:teuthology.orchestra.run.smithi115.stdout:lab-extras 333 kB/s | 24 kB 00:00 2024-03-20T08:44:55.344 INFO:teuthology.orchestra.run.smithi026.stdout:Copr repo for python3-asyncssh owned by ceph 12 kB/s | 3.5 kB 00:00 2024-03-20T08:44:56.943 INFO:teuthology.orchestra.run.smithi122.stdout:lab-extras 531 kB/s | 24 kB 00:00 2024-03-20T08:44:58.202 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-20T08:44:58.204 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:44:58.205 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2024-03-20T08:44:58.205 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:44:58.205 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-20T08:44:58.205 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-radosgw x86_64 2:17.2.7-717.g0192297e.el8 ceph 17 M 2024-03-20T08:44:58.205 INFO:teuthology.orchestra.run.smithi115.stdout:Upgrading: 2024-03-20T08:44:58.205 INFO:teuthology.orchestra.run.smithi115.stdout: librados2 x86_64 2:17.2.7-717.g0192297e.el8 ceph 3.5 M 2024-03-20T08:44:58.205 INFO:teuthology.orchestra.run.smithi115.stdout: librbd1 x86_64 2:17.2.7-717.g0192297e.el8 ceph 3.6 M 2024-03-20T08:44:58.205 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-03-20T08:44:58.205 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-base x86_64 2:17.2.7-717.g0192297e.el8 ceph 5.2 M 2024-03-20T08:44:58.205 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-common x86_64 2:17.2.7-717.g0192297e.el8 ceph 23 M 2024-03-20T08:44:58.205 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-selinux x86_64 2:17.2.7-717.g0192297e.el8 ceph 24 k 2024-03-20T08:44:58.205 INFO:teuthology.orchestra.run.smithi115.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-03-20T08:44:58.206 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs2 x86_64 2:17.2.7-717.g0192297e.el8 ceph 798 k 2024-03-20T08:44:58.206 INFO:teuthology.orchestra.run.smithi115.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-03-20T08:44:58.206 INFO:teuthology.orchestra.run.smithi115.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-03-20T08:44:58.206 INFO:teuthology.orchestra.run.smithi115.stdout: libradosstriper1 x86_64 2:17.2.7-717.g0192297e.el8 ceph 505 k 2024-03-20T08:44:58.206 INFO:teuthology.orchestra.run.smithi115.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-03-20T08:44:58.206 INFO:teuthology.orchestra.run.smithi115.stdout: librgw2 x86_64 2:17.2.7-717.g0192297e.el8 ceph 5.6 M 2024-03-20T08:44:58.206 INFO:teuthology.orchestra.run.smithi115.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-03-20T08:44:58.206 INFO:teuthology.orchestra.run.smithi115.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-03-20T08:44:58.206 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-argparse x86_64 2:17.2.7-717.g0192297e.el8 ceph 45 k 2024-03-20T08:44:58.206 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-common x86_64 2:17.2.7-717.g0192297e.el8 ceph 117 k 2024-03-20T08:44:58.206 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cephfs x86_64 2:17.2.7-717.g0192297e.el8 ceph 190 k 2024-03-20T08:44:58.206 INFO:teuthology.orchestra.run.smithi115.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-03-20T08:44:58.207 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rados x86_64 2:17.2.7-717.g0192297e.el8 ceph 370 k 2024-03-20T08:44:58.207 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rbd x86_64 2:17.2.7-717.g0192297e.el8 ceph 359 k 2024-03-20T08:44:58.207 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rgw x86_64 2:17.2.7-717.g0192297e.el8 ceph 107 k 2024-03-20T08:44:58.207 INFO:teuthology.orchestra.run.smithi115.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-03-20T08:44:58.207 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:44:58.207 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-20T08:44:58.207 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:44:58.207 INFO:teuthology.orchestra.run.smithi115.stdout:Install 21 Packages 2024-03-20T08:44:58.207 INFO:teuthology.orchestra.run.smithi115.stdout:Upgrade 2 Packages 2024-03-20T08:44:58.207 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:44:58.208 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 64 M 2024-03-20T08:44:58.208 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-20T08:44:59.113 INFO:teuthology.orchestra.run.smithi115.stdout:(1/23): ceph-base-17.2.7-717.g0192297e.el8.x86_ 7.0 MB/s | 5.2 MB 00:00 2024-03-20T08:44:59.179 INFO:teuthology.orchestra.run.smithi115.stdout:(2/23): ceph-selinux-17.2.7-717.g0192297e.el8.x 369 kB/s | 24 kB 00:00 2024-03-20T08:44:59.279 INFO:teuthology.orchestra.run.smithi115.stdout:(3/23): libcephfs2-17.2.7-717.g0192297e.el8.x86 7.8 MB/s | 798 kB 00:00 2024-03-20T08:44:59.757 INFO:teuthology.orchestra.run.smithi115.stdout:(4/23): ceph-radosgw-17.2.7-717.g0192297e.el8.x 12 MB/s | 17 MB 00:01 2024-03-20T08:44:59.973 INFO:teuthology.orchestra.run.smithi115.stdout:(5/23): libradosstriper1-17.2.7-717.g0192297e.e 729 kB/s | 505 kB 00:00 2024-03-20T08:45:00.223 INFO:teuthology.orchestra.run.smithi115.stdout:(6/23): python3-ceph-argparse-17.2.7-717.g01922 179 kB/s | 45 kB 00:00 2024-03-20T08:45:00.531 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-20T08:45:00.533 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:45:00.533 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-03-20T08:45:00.534 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:45:00.534 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-20T08:45:00.534 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-radosgw x86_64 2:17.2.7-717.g0192297e.el8 ceph 17 M 2024-03-20T08:45:00.534 INFO:teuthology.orchestra.run.smithi122.stdout:Upgrading: 2024-03-20T08:45:00.534 INFO:teuthology.orchestra.run.smithi122.stdout: librados2 x86_64 2:17.2.7-717.g0192297e.el8 ceph 3.5 M 2024-03-20T08:45:00.534 INFO:teuthology.orchestra.run.smithi122.stdout: librbd1 x86_64 2:17.2.7-717.g0192297e.el8 ceph 3.6 M 2024-03-20T08:45:00.534 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-03-20T08:45:00.534 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-base x86_64 2:17.2.7-717.g0192297e.el8 ceph 5.2 M 2024-03-20T08:45:00.534 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-common x86_64 2:17.2.7-717.g0192297e.el8 ceph 23 M 2024-03-20T08:45:00.534 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-selinux x86_64 2:17.2.7-717.g0192297e.el8 ceph 24 k 2024-03-20T08:45:00.534 INFO:teuthology.orchestra.run.smithi122.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-03-20T08:45:00.535 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs2 x86_64 2:17.2.7-717.g0192297e.el8 ceph 798 k 2024-03-20T08:45:00.535 INFO:teuthology.orchestra.run.smithi122.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-03-20T08:45:00.535 INFO:teuthology.orchestra.run.smithi122.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-03-20T08:45:00.535 INFO:teuthology.orchestra.run.smithi122.stdout: libradosstriper1 x86_64 2:17.2.7-717.g0192297e.el8 ceph 505 k 2024-03-20T08:45:00.535 INFO:teuthology.orchestra.run.smithi122.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-03-20T08:45:00.535 INFO:teuthology.orchestra.run.smithi122.stdout: librgw2 x86_64 2:17.2.7-717.g0192297e.el8 ceph 5.6 M 2024-03-20T08:45:00.535 INFO:teuthology.orchestra.run.smithi122.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-03-20T08:45:00.535 INFO:teuthology.orchestra.run.smithi122.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-03-20T08:45:00.535 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-argparse x86_64 2:17.2.7-717.g0192297e.el8 ceph 45 k 2024-03-20T08:45:00.535 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-common x86_64 2:17.2.7-717.g0192297e.el8 ceph 117 k 2024-03-20T08:45:00.535 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cephfs x86_64 2:17.2.7-717.g0192297e.el8 ceph 190 k 2024-03-20T08:45:00.535 INFO:teuthology.orchestra.run.smithi122.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-03-20T08:45:00.535 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rados x86_64 2:17.2.7-717.g0192297e.el8 ceph 370 k 2024-03-20T08:45:00.535 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rbd x86_64 2:17.2.7-717.g0192297e.el8 ceph 359 k 2024-03-20T08:45:00.536 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rgw x86_64 2:17.2.7-717.g0192297e.el8 ceph 107 k 2024-03-20T08:45:00.536 INFO:teuthology.orchestra.run.smithi122.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-03-20T08:45:00.536 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:45:00.536 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-20T08:45:00.536 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:45:00.536 INFO:teuthology.orchestra.run.smithi122.stdout:Install 21 Packages 2024-03-20T08:45:00.536 INFO:teuthology.orchestra.run.smithi122.stdout:Upgrade 2 Packages 2024-03-20T08:45:00.536 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:45:00.537 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 64 M 2024-03-20T08:45:00.537 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-20T08:45:00.690 INFO:teuthology.orchestra.run.smithi115.stdout:(7/23): librgw2-17.2.7-717.g0192297e.el8.x86_64 6.0 MB/s | 5.6 MB 00:00 2024-03-20T08:45:00.815 INFO:teuthology.orchestra.run.smithi115.stdout:(8/23): python3-ceph-common-17.2.7-717.g0192297 197 kB/s | 117 kB 00:00 2024-03-20T08:45:00.840 INFO:teuthology.orchestra.run.smithi115.stdout:(9/23): python3-cephfs-17.2.7-717.g0192297e.el8 1.2 MB/s | 190 kB 00:00 2024-03-20T08:45:00.941 INFO:teuthology.orchestra.run.smithi115.stdout:(10/23): python3-rbd-17.2.7-717.g0192297e.el8.x 3.5 MB/s | 359 kB 00:00 2024-03-20T08:45:00.989 INFO:teuthology.orchestra.run.smithi115.stdout:(11/23): python3-rados-17.2.7-717.g0192297e.el8 2.1 MB/s | 370 kB 00:00 2024-03-20T08:45:01.017 INFO:teuthology.orchestra.run.smithi115.stdout:(12/23): python3-rgw-17.2.7-717.g0192297e.el8.x 1.4 MB/s | 107 kB 00:00 2024-03-20T08:45:01.317 INFO:teuthology.orchestra.run.smithi115.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 865 kB/s | 259 kB 00:00 2024-03-20T08:45:01.375 INFO:teuthology.orchestra.run.smithi115.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 754 kB/s | 44 kB 00:00 2024-03-20T08:45:01.401 INFO:teuthology.orchestra.run.smithi115.stdout:(15/23): librdkafka-1.6.1-1.el8.x86_64.rpm 1.6 MB/s | 677 kB 00:00 2024-03-20T08:45:01.703 INFO:teuthology.orchestra.run.smithi122.stdout:(1/23): ceph-base-17.2.7-717.g0192297e.el8.x86_ 8.5 MB/s | 5.2 MB 00:00 2024-03-20T08:45:01.762 INFO:teuthology.orchestra.run.smithi122.stdout:(2/23): ceph-selinux-17.2.7-717.g0192297e.el8.x 420 kB/s | 24 kB 00:00 2024-03-20T08:45:01.776 INFO:teuthology.orchestra.run.smithi115.stdout:(16/23): gperftools-libs-2.7-9.el8.x86_64.rpm 816 kB/s | 306 kB 00:00 2024-03-20T08:45:01.801 INFO:teuthology.orchestra.run.smithi115.stdout:(17/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 111 kB/s | 47 kB 00:00 2024-03-20T08:45:01.843 INFO:teuthology.orchestra.run.smithi115.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 879 kB/s | 59 kB 00:00 2024-03-20T08:45:01.878 INFO:teuthology.orchestra.run.smithi122.stdout:(3/23): libcephfs2-17.2.7-717.g0192297e.el8.x86 6.7 MB/s | 798 kB 00:00 2024-03-20T08:45:01.962 INFO:teuthology.orchestra.run.smithi122.stdout:(4/23): libradosstriper1-17.2.7-717.g0192297e.e 5.9 MB/s | 505 kB 00:00 2024-03-20T08:45:02.027 INFO:teuthology.orchestra.run.smithi115.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 333 kB/s | 75 kB 00:00 2024-03-20T08:45:02.285 INFO:teuthology.orchestra.run.smithi115.stdout:(20/23): thrift-0.13.0-2.el8.x86_64.rpm 3.9 MB/s | 1.7 MB 00:00 2024-03-20T08:45:02.454 INFO:teuthology.orchestra.run.smithi122.stdout:(5/23): ceph-radosgw-17.2.7-717.g0192297e.el8.x 12 MB/s | 17 MB 00:01 2024-03-20T08:45:02.588 INFO:teuthology.orchestra.run.smithi122.stdout:(6/23): librgw2-17.2.7-717.g0192297e.el8.x86_64 8.9 MB/s | 5.6 MB 00:00 2024-03-20T08:45:02.613 INFO:teuthology.orchestra.run.smithi122.stdout:(7/23): python3-ceph-argparse-17.2.7-717.g01922 282 kB/s | 45 kB 00:00 2024-03-20T08:45:02.619 INFO:teuthology.orchestra.run.smithi115.stdout:(21/23): librbd1-17.2.7-717.g0192297e.el8.x86_6 11 MB/s | 3.6 MB 00:00 2024-03-20T08:45:02.647 INFO:teuthology.orchestra.run.smithi122.stdout:(8/23): python3-ceph-common-17.2.7-717.g0192297 2.0 MB/s | 117 kB 00:00 2024-03-20T08:45:02.672 INFO:teuthology.orchestra.run.smithi122.stdout:(9/23): python3-cephfs-17.2.7-717.g0192297e.el8 3.2 MB/s | 190 kB 00:00 2024-03-20T08:45:02.698 INFO:teuthology.orchestra.run.smithi122.stdout:(10/23): python3-rados-17.2.7-717.g0192297e.el8 7.2 MB/s | 370 kB 00:00 2024-03-20T08:45:02.723 INFO:teuthology.orchestra.run.smithi122.stdout:(11/23): python3-rbd-17.2.7-717.g0192297e.el8.x 6.9 MB/s | 359 kB 00:00 2024-03-20T08:45:02.748 INFO:teuthology.orchestra.run.smithi122.stdout:(12/23): python3-rgw-17.2.7-717.g0192297e.el8.x 2.1 MB/s | 107 kB 00:00 2024-03-20T08:45:03.182 INFO:teuthology.orchestra.run.smithi122.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 599 kB/s | 259 kB 00:00 2024-03-20T08:45:03.232 INFO:teuthology.orchestra.run.smithi122.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 880 kB/s | 44 kB 00:00 2024-03-20T08:45:03.286 INFO:teuthology.orchestra.run.smithi115.stdout:(22/23): librados2-17.2.7-717.g0192297e.el8.x86 2.8 MB/s | 3.5 MB 00:01 2024-03-20T08:45:03.575 INFO:teuthology.orchestra.run.smithi122.stdout:(15/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 137 kB/s | 47 kB 00:00 2024-03-20T08:45:03.825 INFO:teuthology.orchestra.run.smithi122.stdout:(16/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.2 MB/s | 306 kB 00:00 2024-03-20T08:45:03.911 INFO:teuthology.orchestra.run.smithi122.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 683 kB/s | 59 kB 00:00 2024-03-20T08:45:03.950 INFO:teuthology.orchestra.run.smithi122.stdout:(18/23): librdkafka-1.6.1-1.el8.x86_64.rpm 551 kB/s | 677 kB 00:01 2024-03-20T08:45:03.976 INFO:teuthology.orchestra.run.smithi122.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.1 MB/s | 75 kB 00:00 2024-03-20T08:45:04.259 INFO:teuthology.orchestra.run.smithi122.stdout:(20/23): librados2-17.2.7-717.g0192297e.el8.x86 12 MB/s | 3.5 MB 00:00 2024-03-20T08:45:04.310 INFO:teuthology.orchestra.run.smithi122.stdout:(21/23): thrift-0.13.0-2.el8.x86_64.rpm 4.8 MB/s | 1.7 MB 00:00 2024-03-20T08:45:04.493 INFO:teuthology.orchestra.run.smithi122.stdout:(22/23): librbd1-17.2.7-717.g0192297e.el8.x86_6 15 MB/s | 3.6 MB 00:00 2024-03-20T08:45:15.526 INFO:teuthology.orchestra.run.smithi026.stdout:Extra Packages for Enterprise Linux 827 kB/s | 16 MB 00:20 2024-03-20T08:45:17.118 INFO:teuthology.orchestra.run.smithi122.stdout:(23/23): ceph-common-17.2.7-717.g0192297e.el8.x 1.5 MB/s | 23 MB 00:16 2024-03-20T08:45:17.120 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:45:17.120 INFO:teuthology.orchestra.run.smithi122.stdout:Total 3.8 MB/s | 64 MB 00:16 2024-03-20T08:45:17.162 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-20T08:45:17.208 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-20T08:45:17.208 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-20T08:45:17.582 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-20T08:45:17.585 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-20T08:45:18.059 INFO:teuthology.orchestra.run.smithi115.stdout:(23/23): ceph-common-17.2.7-717.g0192297e.el8.x 1.2 MB/s | 23 MB 00:19 2024-03-20T08:45:18.061 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:45:18.061 INFO:teuthology.orchestra.run.smithi115.stdout:Total 3.2 MB/s | 64 MB 00:19 2024-03-20T08:45:18.107 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-20T08:45:18.153 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-20T08:45:18.153 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-20T08:45:18.527 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-20T08:45:18.530 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-20T08:45:18.649 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-20T08:45:18.966 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-03-20T08:45:19.150 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-03-20T08:45:19.555 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-20T08:45:19.568 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-03-20T08:45:19.593 INFO:teuthology.orchestra.run.smithi026.stdout:lab-extras 404 kB/s | 24 kB 00:00 2024-03-20T08:45:19.755 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-03-20T08:45:19.800 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : librados2-2:17.2.7-717.g0192297e.el8.x86_64 3/25 2024-03-20T08:45:19.908 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados2-2:17.2.7-717.g0192297e.el8.x86_64 3/25 2024-03-20T08:45:19.948 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-03-20T08:45:20.165 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rados-2:17.2.7-717.g0192297e.el8.x86_64 4/25 2024-03-20T08:45:20.331 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-03-20T08:45:20.383 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephfs2-2:17.2.7-717.g0192297e.el8.x86_64 5/25 2024-03-20T08:45:20.548 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : librados2-2:17.2.7-717.g0192297e.el8.x86_64 3/25 2024-03-20T08:45:20.656 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librados2-2:17.2.7-717.g0192297e.el8.x86_64 3/25 2024-03-20T08:45:20.778 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephfs2-2:17.2.7-717.g0192297e.el8.x86_64 5/25 2024-03-20T08:45:20.889 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rados-2:17.2.7-717.g0192297e.el8.x86_64 4/25 2024-03-20T08:45:20.967 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : librbd1-2:17.2.7-717.g0192297e.el8.x86_64 6/25 2024-03-20T08:45:21.072 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libcephfs2-2:17.2.7-717.g0192297e.el8.x86_64 5/25 2024-03-20T08:45:21.224 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librbd1-2:17.2.7-717.g0192297e.el8.x86_64 6/25 2024-03-20T08:45:21.420 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2024-03-20T08:45:21.467 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libcephfs2-2:17.2.7-717.g0192297e.el8.x86_64 5/25 2024-03-20T08:45:21.648 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : librbd1-2:17.2.7-717.g0192297e.el8.x86_64 6/25 2024-03-20T08:45:21.659 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 8/25 2024-03-20T08:45:21.860 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librbd1-2:17.2.7-717.g0192297e.el8.x86_64 6/25 2024-03-20T08:45:22.171 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2024-03-20T08:45:22.349 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 9/25 2024-03-20T08:45:22.407 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 8/25 2024-03-20T08:45:22.542 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librgw2-2:17.2.7-717.g0192297e.el8.x86_64 10/25 2024-03-20T08:45:22.611 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librgw2-2:17.2.7-717.g0192297e.el8.x86_64 10/25 2024-03-20T08:45:22.785 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ceph-argparse-2:17.2.7-717.g0192297e.el8.x 11/25 2024-03-20T08:45:23.011 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cephfs-2:17.2.7-717.g0192297e.el8.x86_64 12/25 2024-03-20T08:45:23.075 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 9/25 2024-03-20T08:45:23.235 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rgw-2:17.2.7-717.g0192297e.el8.x86_64 13/25 2024-03-20T08:45:23.264 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librgw2-2:17.2.7-717.g0192297e.el8.x86_64 10/25 2024-03-20T08:45:23.318 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librgw2-2:17.2.7-717.g0192297e.el8.x86_64 10/25 2024-03-20T08:45:23.466 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-ceph-argparse-2:17.2.7-717.g0192297e.el8.x 11/25 2024-03-20T08:45:23.491 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-20T08:45:23.494 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:45:23.494 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-03-20T08:45:23.494 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:45:23.495 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-20T08:45:23.495 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-radosgw x86_64 2:17.2.7-717.g0192297e.el8 ceph 17 M 2024-03-20T08:45:23.495 INFO:teuthology.orchestra.run.smithi026.stdout:Upgrading: 2024-03-20T08:45:23.495 INFO:teuthology.orchestra.run.smithi026.stdout: librados2 x86_64 2:17.2.7-717.g0192297e.el8 ceph 3.5 M 2024-03-20T08:45:23.495 INFO:teuthology.orchestra.run.smithi026.stdout: librbd1 x86_64 2:17.2.7-717.g0192297e.el8 ceph 3.6 M 2024-03-20T08:45:23.495 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-03-20T08:45:23.495 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-base x86_64 2:17.2.7-717.g0192297e.el8 ceph 5.2 M 2024-03-20T08:45:23.495 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-common x86_64 2:17.2.7-717.g0192297e.el8 ceph 23 M 2024-03-20T08:45:23.495 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-selinux x86_64 2:17.2.7-717.g0192297e.el8 ceph 24 k 2024-03-20T08:45:23.495 INFO:teuthology.orchestra.run.smithi026.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-03-20T08:45:23.495 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs2 x86_64 2:17.2.7-717.g0192297e.el8 ceph 798 k 2024-03-20T08:45:23.495 INFO:teuthology.orchestra.run.smithi026.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-03-20T08:45:23.496 INFO:teuthology.orchestra.run.smithi026.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-03-20T08:45:23.496 INFO:teuthology.orchestra.run.smithi026.stdout: libradosstriper1 x86_64 2:17.2.7-717.g0192297e.el8 ceph 505 k 2024-03-20T08:45:23.496 INFO:teuthology.orchestra.run.smithi026.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-03-20T08:45:23.496 INFO:teuthology.orchestra.run.smithi026.stdout: librgw2 x86_64 2:17.2.7-717.g0192297e.el8 ceph 5.6 M 2024-03-20T08:45:23.496 INFO:teuthology.orchestra.run.smithi026.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-03-20T08:45:23.496 INFO:teuthology.orchestra.run.smithi026.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-03-20T08:45:23.496 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-argparse x86_64 2:17.2.7-717.g0192297e.el8 ceph 45 k 2024-03-20T08:45:23.496 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-common x86_64 2:17.2.7-717.g0192297e.el8 ceph 117 k 2024-03-20T08:45:23.496 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cephfs x86_64 2:17.2.7-717.g0192297e.el8 ceph 190 k 2024-03-20T08:45:23.496 INFO:teuthology.orchestra.run.smithi026.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-03-20T08:45:23.496 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rados x86_64 2:17.2.7-717.g0192297e.el8 ceph 370 k 2024-03-20T08:45:23.496 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rbd x86_64 2:17.2.7-717.g0192297e.el8 ceph 359 k 2024-03-20T08:45:23.496 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rgw x86_64 2:17.2.7-717.g0192297e.el8 ceph 107 k 2024-03-20T08:45:23.496 INFO:teuthology.orchestra.run.smithi026.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-03-20T08:45:23.497 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:45:23.497 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-20T08:45:23.497 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:45:23.497 INFO:teuthology.orchestra.run.smithi026.stdout:Install 21 Packages 2024-03-20T08:45:23.497 INFO:teuthology.orchestra.run.smithi026.stdout:Upgrade 2 Packages 2024-03-20T08:45:23.497 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:45:23.497 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 64 M 2024-03-20T08:45:23.497 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-20T08:45:23.503 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rbd-2:17.2.7-717.g0192297e.el8.x86_64 14/25 2024-03-20T08:45:23.642 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cephfs-2:17.2.7-717.g0192297e.el8.x86_64 12/25 2024-03-20T08:45:23.658 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libradosstriper1-2:17.2.7-717.g0192297e.el8.x86_64 15/25 2024-03-20T08:45:23.709 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libradosstriper1-2:17.2.7-717.g0192297e.el8.x86_64 15/25 2024-03-20T08:45:23.882 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rgw-2:17.2.7-717.g0192297e.el8.x86_64 13/25 2024-03-20T08:45:23.964 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-03-20T08:45:24.092 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rbd-2:17.2.7-717.g0192297e.el8.x86_64 14/25 2024-03-20T08:45:24.272 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libradosstriper1-2:17.2.7-717.g0192297e.el8.x86_64 15/25 2024-03-20T08:45:24.340 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libradosstriper1-2:17.2.7-717.g0192297e.el8.x86_64 15/25 2024-03-20T08:45:24.410 INFO:teuthology.orchestra.run.smithi026.stdout:(1/23): ceph-base-17.2.7-717.g0192297e.el8.x86_ 11 MB/s | 5.2 MB 00:00 2024-03-20T08:45:24.477 INFO:teuthology.orchestra.run.smithi026.stdout:(2/23): ceph-selinux-17.2.7-717.g0192297e.el8.x 368 kB/s | 24 kB 00:00 2024-03-20T08:45:24.483 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-03-20T08:45:24.539 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-03-20T08:45:24.569 INFO:teuthology.orchestra.run.smithi026.stdout:(3/23): libcephfs2-17.2.7-717.g0192297e.el8.x86 8.5 MB/s | 798 kB 00:00 2024-03-20T08:45:24.628 INFO:teuthology.orchestra.run.smithi026.stdout:(4/23): libradosstriper1-17.2.7-717.g0192297e.e 8.5 MB/s | 505 kB 00:00 2024-03-20T08:45:24.668 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-03-20T08:45:24.738 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-03-20T08:45:24.892 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ceph-common-2:17.2.7-717.g0192297e.el8.x86 19/25 2024-03-20T08:45:24.930 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-03-20T08:45:24.962 INFO:teuthology.orchestra.run.smithi026.stdout:(5/23): librgw2-17.2.7-717.g0192297e.el8.x86_64 17 MB/s | 5.6 MB 00:00 2024-03-20T08:45:25.003 INFO:teuthology.orchestra.run.smithi026.stdout:(6/23): python3-ceph-argparse-17.2.7-717.g01922 1.1 MB/s | 45 kB 00:00 2024-03-20T08:45:25.101 INFO:teuthology.orchestra.run.smithi026.stdout:(7/23): python3-ceph-common-17.2.7-717.g0192297 1.2 MB/s | 117 kB 00:00 2024-03-20T08:45:25.139 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-ceph-common-2:17.2.7-717.g0192297e.el8.x86 19/25 2024-03-20T08:45:25.154 INFO:teuthology.orchestra.run.smithi026.stdout:(8/23): python3-cephfs-17.2.7-717.g0192297e.el8 3.5 MB/s | 190 kB 00:00 2024-03-20T08:45:25.221 INFO:teuthology.orchestra.run.smithi026.stdout:(9/23): python3-rados-17.2.7-717.g0192297e.el8. 5.4 MB/s | 370 kB 00:00 2024-03-20T08:45:25.271 INFO:teuthology.orchestra.run.smithi026.stdout:(10/23): python3-rbd-17.2.7-717.g0192297e.el8.x 7.0 MB/s | 359 kB 00:00 2024-03-20T08:45:25.321 INFO:teuthology.orchestra.run.smithi026.stdout:(11/23): python3-rgw-17.2.7-717.g0192297e.el8.x 2.1 MB/s | 107 kB 00:00 2024-03-20T08:45:25.789 INFO:teuthology.orchestra.run.smithi026.stdout:(12/23): librdkafka-1.6.1-1.el8.x86_64.rpm 1.4 MB/s | 677 kB 00:00 2024-03-20T08:45:26.056 INFO:teuthology.orchestra.run.smithi026.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 973 kB/s | 259 kB 00:00 2024-03-20T08:45:26.131 INFO:teuthology.orchestra.run.smithi026.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 589 kB/s | 44 kB 00:00 2024-03-20T08:45:26.473 INFO:teuthology.orchestra.run.smithi026.stdout:(15/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 138 kB/s | 47 kB 00:00 2024-03-20T08:45:26.706 INFO:teuthology.orchestra.run.smithi026.stdout:(16/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.3 MB/s | 306 kB 00:00 2024-03-20T08:45:26.773 INFO:teuthology.orchestra.run.smithi026.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 886 kB/s | 59 kB 00:00 2024-03-20T08:45:27.065 INFO:teuthology.orchestra.run.smithi026.stdout:(18/23): ceph-common-17.2.7-717.g0192297e.el8.x 7.6 MB/s | 23 MB 00:03 2024-03-20T08:45:27.166 INFO:teuthology.orchestra.run.smithi026.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 191 kB/s | 75 kB 00:00 2024-03-20T08:45:27.367 INFO:teuthology.orchestra.run.smithi026.stdout:(20/23): ceph-radosgw-17.2.7-717.g0192297e.el8. 4.9 MB/s | 17 MB 00:03 2024-03-20T08:45:27.584 INFO:teuthology.orchestra.run.smithi026.stdout:(21/23): librados2-17.2.7-717.g0192297e.el8.x86 8.4 MB/s | 3.5 MB 00:00 2024-03-20T08:45:27.621 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-common-2:17.2.7-717.g0192297e.el8.x86_64 20/25 2024-03-20T08:45:27.735 INFO:teuthology.orchestra.run.smithi026.stdout:(22/23): librbd1-17.2.7-717.g0192297e.el8.x86_6 9.8 MB/s | 3.6 MB 00:00 2024-03-20T08:45:27.842 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-common-2:17.2.7-717.g0192297e.el8.x86_64 20/25 2024-03-20T08:45:27.870 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-common-2:17.2.7-717.g0192297e.el8.x86_64 20/25 2024-03-20T08:45:28.067 INFO:teuthology.orchestra.run.smithi026.stdout:(23/23): thrift-0.13.0-2.el8.x86_64.rpm 1.7 MB/s | 1.7 MB 00:01 2024-03-20T08:45:28.071 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:45:28.071 INFO:teuthology.orchestra.run.smithi026.stdout:Total 14 MB/s | 64 MB 00:04 2024-03-20T08:45:28.116 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-20T08:45:28.130 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-common-2:17.2.7-717.g0192297e.el8.x86_64 20/25 2024-03-20T08:45:28.166 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-20T08:45:28.166 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-20T08:45:28.423 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-common-2:17.2.7-717.g0192297e.el8.x86_64 20/25 2024-03-20T08:45:28.557 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-20T08:45:28.561 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-20T08:45:28.683 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-base-2:17.2.7-717.g0192297e.el8.x86_64 21/25 2024-03-20T08:45:28.702 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-common-2:17.2.7-717.g0192297e.el8.x86_64 20/25 2024-03-20T08:45:28.819 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-base-2:17.2.7-717.g0192297e.el8.x86_64 21/25 2024-03-20T08:45:28.983 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-selinux-2:17.2.7-717.g0192297e.el8.x86_64 22/25 2024-03-20T08:45:29.005 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-base-2:17.2.7-717.g0192297e.el8.x86_64 21/25 2024-03-20T08:45:29.111 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-base-2:17.2.7-717.g0192297e.el8.x86_64 21/25 2024-03-20T08:45:29.313 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-selinux-2:17.2.7-717.g0192297e.el8.x86_64 22/25 2024-03-20T08:45:29.710 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-20T08:45:29.961 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-03-20T08:45:30.123 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-03-20T08:45:30.563 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-03-20T08:45:30.756 INFO:teuthology.orchestra.run.smithi026.stdout: Upgrading : librados2-2:17.2.7-717.g0192297e.el8.x86_64 3/25 2024-03-20T08:45:30.860 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librados2-2:17.2.7-717.g0192297e.el8.x86_64 3/25 2024-03-20T08:45:31.111 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rados-2:17.2.7-717.g0192297e.el8.x86_64 4/25 2024-03-20T08:45:31.348 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libcephfs2-2:17.2.7-717.g0192297e.el8.x86_64 5/25 2024-03-20T08:45:31.765 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libcephfs2-2:17.2.7-717.g0192297e.el8.x86_64 5/25 2024-03-20T08:45:31.990 INFO:teuthology.orchestra.run.smithi026.stdout: Upgrading : librbd1-2:17.2.7-717.g0192297e.el8.x86_64 6/25 2024-03-20T08:45:32.274 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librbd1-2:17.2.7-717.g0192297e.el8.x86_64 6/25 2024-03-20T08:45:32.514 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2024-03-20T08:45:32.720 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 8/25 2024-03-20T08:45:33.448 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 9/25 2024-03-20T08:45:33.598 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librgw2-2:17.2.7-717.g0192297e.el8.x86_64 10/25 2024-03-20T08:45:33.652 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librgw2-2:17.2.7-717.g0192297e.el8.x86_64 10/25 2024-03-20T08:45:33.859 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-ceph-argparse-2:17.2.7-717.g0192297e.el8.x 11/25 2024-03-20T08:45:34.076 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cephfs-2:17.2.7-717.g0192297e.el8.x86_64 12/25 2024-03-20T08:45:34.269 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rgw-2:17.2.7-717.g0192297e.el8.x86_64 13/25 2024-03-20T08:45:34.521 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rbd-2:17.2.7-717.g0192297e.el8.x86_64 14/25 2024-03-20T08:45:34.697 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libradosstriper1-2:17.2.7-717.g0192297e.el8.x86_64 15/25 2024-03-20T08:45:34.759 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libradosstriper1-2:17.2.7-717.g0192297e.el8.x86_64 15/25 2024-03-20T08:45:34.960 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-03-20T08:45:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-03-20T08:45:35.422 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-03-20T08:45:35.665 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-ceph-common-2:17.2.7-717.g0192297e.el8.x86 19/25 2024-03-20T08:45:38.488 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-common-2:17.2.7-717.g0192297e.el8.x86_64 20/25 2024-03-20T08:45:38.739 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-common-2:17.2.7-717.g0192297e.el8.x86_64 20/25 2024-03-20T08:45:39.381 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-common-2:17.2.7-717.g0192297e.el8.x86_64 20/25 2024-03-20T08:45:39.725 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-base-2:17.2.7-717.g0192297e.el8.x86_64 21/25 2024-03-20T08:45:39.847 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-base-2:17.2.7-717.g0192297e.el8.x86_64 21/25 2024-03-20T08:45:40.089 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-selinux-2:17.2.7-717.g0192297e.el8.x86_64 22/25 2024-03-20T08:45:48.411 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-selinux-2:17.2.7-717.g0192297e.el8.x86_64 22/25 2024-03-20T08:45:48.411 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /sys 2024-03-20T08:45:48.411 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /proc 2024-03-20T08:45:48.412 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /dev 2024-03-20T08:45:48.412 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /run 2024-03-20T08:45:48.412 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /mnt 2024-03-20T08:45:48.412 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /var/tmp 2024-03-20T08:45:48.412 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /home 2024-03-20T08:45:48.412 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /tmp 2024-03-20T08:45:48.412 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /dev 2024-03-20T08:45:48.412 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:45:48.635 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-selinux-2:17.2.7-717.g0192297e.el8.x86_64 22/25 2024-03-20T08:45:48.635 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /sys 2024-03-20T08:45:48.635 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /proc 2024-03-20T08:45:48.635 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /dev 2024-03-20T08:45:48.636 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /run 2024-03-20T08:45:48.636 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /mnt 2024-03-20T08:45:48.636 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /var/tmp 2024-03-20T08:45:48.636 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /home 2024-03-20T08:45:48.636 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /tmp 2024-03-20T08:45:48.636 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /dev 2024-03-20T08:45:48.636 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:45:50.131 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-radosgw-2:17.2.7-717.g0192297e.el8.x86_64 23/25 2024-03-20T08:45:50.192 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-717.g0192297e.el8.x86_64 23/25 2024-03-20T08:45:50.192 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-03-20T08:45:50.350 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-radosgw-2:17.2.7-717.g0192297e.el8.x86_64 23/25 2024-03-20T08:45:50.390 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-03-20T08:45:50.390 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-03-20T08:45:50.436 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-717.g0192297e.el8.x86_64 23/25 2024-03-20T08:45:50.437 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-03-20T08:45:50.676 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-03-20T08:45:50.677 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-03-20T08:45:51.112 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-03-20T08:45:51.112 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-base-2:17.2.7-717.g0192297e.el8.x86_64 1/25 2024-03-20T08:45:51.112 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-common-2:17.2.7-717.g0192297e.el8.x86_64 2/25 2024-03-20T08:45:51.112 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-radosgw-2:17.2.7-717.g0192297e.el8.x86_64 3/25 2024-03-20T08:45:51.113 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-selinux-2:17.2.7-717.g0192297e.el8.x86_64 4/25 2024-03-20T08:45:51.113 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephfs2-2:17.2.7-717.g0192297e.el8.x86_64 5/25 2024-03-20T08:45:51.113 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libradosstriper1-2:17.2.7-717.g0192297e.el8.x86_64 6/25 2024-03-20T08:45:51.113 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librgw2-2:17.2.7-717.g0192297e.el8.x86_64 7/25 2024-03-20T08:45:51.113 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ceph-argparse-2:17.2.7-717.g0192297e.el8.x 8/25 2024-03-20T08:45:51.113 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ceph-common-2:17.2.7-717.g0192297e.el8.x86 9/25 2024-03-20T08:45:51.113 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cephfs-2:17.2.7-717.g0192297e.el8.x86_64 10/25 2024-03-20T08:45:51.113 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rados-2:17.2.7-717.g0192297e.el8.x86_64 11/25 2024-03-20T08:45:51.113 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rbd-2:17.2.7-717.g0192297e.el8.x86_64 12/25 2024-03-20T08:45:51.113 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rgw-2:17.2.7-717.g0192297e.el8.x86_64 13/25 2024-03-20T08:45:51.113 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-03-20T08:45:51.114 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-03-20T08:45:51.114 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-03-20T08:45:51.114 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-03-20T08:45:51.114 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-03-20T08:45:51.114 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2024-03-20T08:45:51.114 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2024-03-20T08:45:51.114 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2024-03-20T08:45:51.115 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados2-2:17.2.7-717.g0192297e.el8.x86_64 22/25 2024-03-20T08:45:51.115 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-03-20T08:45:51.115 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librbd1-2:17.2.7-717.g0192297e.el8.x86_64 24/25 2024-03-20T08:45:51.358 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-03-20T08:45:51.369 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-base-2:17.2.7-717.g0192297e.el8.x86_64 1/25 2024-03-20T08:45:51.369 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-common-2:17.2.7-717.g0192297e.el8.x86_64 2/25 2024-03-20T08:45:51.369 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-radosgw-2:17.2.7-717.g0192297e.el8.x86_64 3/25 2024-03-20T08:45:51.369 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-selinux-2:17.2.7-717.g0192297e.el8.x86_64 4/25 2024-03-20T08:45:51.369 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libcephfs2-2:17.2.7-717.g0192297e.el8.x86_64 5/25 2024-03-20T08:45:51.369 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libradosstriper1-2:17.2.7-717.g0192297e.el8.x86_64 6/25 2024-03-20T08:45:51.370 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librgw2-2:17.2.7-717.g0192297e.el8.x86_64 7/25 2024-03-20T08:45:51.370 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-ceph-argparse-2:17.2.7-717.g0192297e.el8.x 8/25 2024-03-20T08:45:51.370 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-ceph-common-2:17.2.7-717.g0192297e.el8.x86 9/25 2024-03-20T08:45:51.370 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cephfs-2:17.2.7-717.g0192297e.el8.x86_64 10/25 2024-03-20T08:45:51.370 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rados-2:17.2.7-717.g0192297e.el8.x86_64 11/25 2024-03-20T08:45:51.370 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rbd-2:17.2.7-717.g0192297e.el8.x86_64 12/25 2024-03-20T08:45:51.370 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rgw-2:17.2.7-717.g0192297e.el8.x86_64 13/25 2024-03-20T08:45:51.370 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-03-20T08:45:51.370 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-03-20T08:45:51.370 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-03-20T08:45:51.370 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-03-20T08:45:51.371 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-03-20T08:45:51.371 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2024-03-20T08:45:51.371 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2024-03-20T08:45:51.371 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2024-03-20T08:45:51.371 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librados2-2:17.2.7-717.g0192297e.el8.x86_64 22/25 2024-03-20T08:45:51.371 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-03-20T08:45:51.371 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librbd1-2:17.2.7-717.g0192297e.el8.x86_64 24/25 2024-03-20T08:45:52.058 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-03-20T08:45:52.058 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:45:52.058 INFO:teuthology.orchestra.run.smithi122.stdout:Upgraded: 2024-03-20T08:45:52.058 INFO:teuthology.orchestra.run.smithi122.stdout: librados2-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:45:52.058 INFO:teuthology.orchestra.run.smithi122.stdout: librbd1-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:45:52.058 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-20T08:45:52.058 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-base-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:45:52.058 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-common-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:45:52.058 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-radosgw-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:45:52.059 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-selinux-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:45:52.059 INFO:teuthology.orchestra.run.smithi122.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-03-20T08:45:52.059 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs2-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:45:52.059 INFO:teuthology.orchestra.run.smithi122.stdout: liboath-2.6.2-3.el8.x86_64 2024-03-20T08:45:52.059 INFO:teuthology.orchestra.run.smithi122.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-03-20T08:45:52.059 INFO:teuthology.orchestra.run.smithi122.stdout: libradosstriper1-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:45:52.059 INFO:teuthology.orchestra.run.smithi122.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-03-20T08:45:52.059 INFO:teuthology.orchestra.run.smithi122.stdout: librgw2-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:45:52.059 INFO:teuthology.orchestra.run.smithi122.stdout: libunwind-1.3.1-3.el8.x86_64 2024-03-20T08:45:52.059 INFO:teuthology.orchestra.run.smithi122.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-03-20T08:45:52.059 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-argparse-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:45:52.059 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-common-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:45:52.059 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cephfs-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:45:52.060 INFO:teuthology.orchestra.run.smithi122.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-03-20T08:45:52.060 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rados-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:45:52.060 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rbd-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:45:52.060 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rgw-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:45:52.060 INFO:teuthology.orchestra.run.smithi122.stdout: thrift-0.13.0-2.el8.x86_64 2024-03-20T08:45:52.060 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:45:52.060 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-20T08:45:52.352 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-test 2024-03-20T08:45:52.372 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-03-20T08:45:52.372 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:45:52.372 INFO:teuthology.orchestra.run.smithi115.stdout:Upgraded: 2024-03-20T08:45:52.372 INFO:teuthology.orchestra.run.smithi115.stdout: librados2-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:45:52.372 INFO:teuthology.orchestra.run.smithi115.stdout: librbd1-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:45:52.372 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-20T08:45:52.372 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-base-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:45:52.372 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-common-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:45:52.372 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-radosgw-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:45:52.372 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-selinux-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:45:52.373 INFO:teuthology.orchestra.run.smithi115.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-03-20T08:45:52.373 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs2-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:45:52.373 INFO:teuthology.orchestra.run.smithi115.stdout: liboath-2.6.2-3.el8.x86_64 2024-03-20T08:45:52.373 INFO:teuthology.orchestra.run.smithi115.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-03-20T08:45:52.373 INFO:teuthology.orchestra.run.smithi115.stdout: libradosstriper1-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:45:52.373 INFO:teuthology.orchestra.run.smithi115.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-03-20T08:45:52.373 INFO:teuthology.orchestra.run.smithi115.stdout: librgw2-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:45:52.373 INFO:teuthology.orchestra.run.smithi115.stdout: libunwind-1.3.1-3.el8.x86_64 2024-03-20T08:45:52.373 INFO:teuthology.orchestra.run.smithi115.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-03-20T08:45:52.373 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-argparse-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:45:52.373 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-common-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:45:52.373 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cephfs-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:45:52.373 INFO:teuthology.orchestra.run.smithi115.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-03-20T08:45:52.373 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rados-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:45:52.374 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rbd-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:45:52.374 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rgw-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:45:52.374 INFO:teuthology.orchestra.run.smithi115.stdout: thrift-0.13.0-2.el8.x86_64 2024-03-20T08:45:52.374 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:45:52.374 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-20T08:45:52.714 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-test 2024-03-20T08:45:53.106 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:00:57 ago on Wed 20 Mar 2024 08:44:56 AM UTC. 2024-03-20T08:45:53.463 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:00:59 ago on Wed 20 Mar 2024 08:44:54 AM UTC. 2024-03-20T08:45:54.721 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-20T08:45:54.722 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:45:54.723 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-03-20T08:45:54.723 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:45:54.723 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-20T08:45:54.723 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-test x86_64 2:17.2.7-717.g0192297e.el8 ceph 52 M 2024-03-20T08:45:54.723 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-03-20T08:45:54.723 INFO:teuthology.orchestra.run.smithi122.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-03-20T08:45:54.723 INFO:teuthology.orchestra.run.smithi122.stdout: libcephsqlite x86_64 2:17.2.7-717.g0192297e.el8 ceph 179 k 2024-03-20T08:45:54.723 INFO:teuthology.orchestra.run.smithi122.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-03-20T08:45:54.723 INFO:teuthology.orchestra.run.smithi122.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-03-20T08:45:54.723 INFO:teuthology.orchestra.run.smithi122.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-03-20T08:45:54.723 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:45:54.723 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-20T08:45:54.723 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:45:54.724 INFO:teuthology.orchestra.run.smithi122.stdout:Install 6 Packages 2024-03-20T08:45:54.724 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:45:54.724 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 53 M 2024-03-20T08:45:54.724 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 234 M 2024-03-20T08:45:54.724 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-20T08:45:54.975 INFO:teuthology.orchestra.run.smithi122.stdout:(1/6): libcephsqlite-17.2.7-717.g0192297e.el8.x 923 kB/s | 179 kB 00:00 2024-03-20T08:45:55.001 INFO:teuthology.orchestra.run.smithi122.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 925 kB/s | 203 kB 00:00 2024-03-20T08:45:55.085 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-20T08:45:55.086 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:45:55.086 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2024-03-20T08:45:55.086 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:45:55.086 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-20T08:45:55.086 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-test x86_64 2:17.2.7-717.g0192297e.el8 ceph 52 M 2024-03-20T08:45:55.086 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-03-20T08:45:55.086 INFO:teuthology.orchestra.run.smithi115.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-03-20T08:45:55.086 INFO:teuthology.orchestra.run.smithi115.stdout: libcephsqlite x86_64 2:17.2.7-717.g0192297e.el8 ceph 179 k 2024-03-20T08:45:55.086 INFO:teuthology.orchestra.run.smithi115.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-03-20T08:45:55.087 INFO:teuthology.orchestra.run.smithi115.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-03-20T08:45:55.087 INFO:teuthology.orchestra.run.smithi115.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-03-20T08:45:55.087 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:45:55.087 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-20T08:45:55.087 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:45:55.087 INFO:teuthology.orchestra.run.smithi115.stdout:Install 6 Packages 2024-03-20T08:45:55.087 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:45:55.088 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 53 M 2024-03-20T08:45:55.088 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 234 M 2024-03-20T08:45:55.088 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-20T08:45:55.126 INFO:teuthology.orchestra.run.smithi122.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.5 MB/s | 323 kB 00:00 2024-03-20T08:45:55.184 INFO:teuthology.orchestra.run.smithi122.stdout:(4/6): oniguruma-6.8.2-3.el8.x86_64.rpm 901 kB/s | 188 kB 00:00 2024-03-20T08:45:55.210 INFO:teuthology.orchestra.run.smithi122.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 845 kB/s | 71 kB 00:00 2024-03-20T08:45:55.570 INFO:teuthology.orchestra.run.smithi115.stdout:(1/6): jq-1.6-8.el8.x86_64.rpm 733 kB/s | 203 kB 00:00 2024-03-20T08:45:55.679 INFO:teuthology.orchestra.run.smithi115.stdout:(2/6): libcephsqlite-17.2.7-717.g0192297e.el8.x 464 kB/s | 179 kB 00:00 2024-03-20T08:45:55.746 INFO:teuthology.orchestra.run.smithi115.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 1.0 MB/s | 188 kB 00:00 2024-03-20T08:45:55.846 INFO:teuthology.orchestra.run.smithi115.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 706 kB/s | 71 kB 00:00 2024-03-20T08:45:55.930 INFO:teuthology.orchestra.run.smithi115.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.3 MB/s | 323 kB 00:00 2024-03-20T08:45:58.335 INFO:teuthology.orchestra.run.smithi122.stdout:(6/6): ceph-test-17.2.7-717.g0192297e.el8.x86_6 15 MB/s | 52 MB 00:03 2024-03-20T08:45:58.337 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:45:58.337 INFO:teuthology.orchestra.run.smithi122.stdout:Total 15 MB/s | 53 MB 00:03 2024-03-20T08:45:58.378 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-20T08:45:58.398 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-20T08:45:58.399 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-20T08:45:58.686 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-20T08:45:58.689 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-20T08:45:59.168 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-20T08:45:59.399 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-03-20T08:45:59.631 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-03-20T08:45:59.770 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-03-20T08:45:59.831 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-03-20T08:45:59.904 INFO:teuthology.orchestra.run.smithi115.stdout:(6/6): ceph-test-17.2.7-717.g0192297e.el8.x86_6 11 MB/s | 52 MB 00:04 2024-03-20T08:45:59.906 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:45:59.907 INFO:teuthology.orchestra.run.smithi115.stdout:Total 11 MB/s | 53 MB 00:04 2024-03-20T08:45:59.949 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-20T08:45:59.970 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-20T08:45:59.971 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-20T08:46:00.069 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-03-20T08:46:00.245 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephsqlite-2:17.2.7-717.g0192297e.el8.x86_64 5/6 2024-03-20T08:46:00.256 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-20T08:46:00.259 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-20T08:46:00.583 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-selinux-2:17.2.7-717.g0192297e.el8.x86_64 22/25 2024-03-20T08:46:00.583 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /sys 2024-03-20T08:46:00.583 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /proc 2024-03-20T08:46:00.583 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /dev 2024-03-20T08:46:00.583 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /run 2024-03-20T08:46:00.583 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /mnt 2024-03-20T08:46:00.583 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /var/tmp 2024-03-20T08:46:00.583 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /home 2024-03-20T08:46:00.583 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /tmp 2024-03-20T08:46:00.583 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /dev 2024-03-20T08:46:00.583 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:46:00.729 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-20T08:46:00.968 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-03-20T08:46:01.182 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-03-20T08:46:01.330 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-03-20T08:46:01.392 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-03-20T08:46:01.638 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-03-20T08:46:01.788 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libcephsqlite-2:17.2.7-717.g0192297e.el8.x86_64 5/6 2024-03-20T08:46:02.415 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-radosgw-2:17.2.7-717.g0192297e.el8.x86_64 23/25 2024-03-20T08:46:02.537 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-717.g0192297e.el8.x86_64 23/25 2024-03-20T08:46:02.537 INFO:teuthology.orchestra.run.smithi026.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-03-20T08:46:02.871 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-03-20T08:46:02.871 INFO:teuthology.orchestra.run.smithi026.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-03-20T08:46:03.593 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-03-20T08:46:03.593 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-base-2:17.2.7-717.g0192297e.el8.x86_64 1/25 2024-03-20T08:46:03.593 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-common-2:17.2.7-717.g0192297e.el8.x86_64 2/25 2024-03-20T08:46:03.594 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-radosgw-2:17.2.7-717.g0192297e.el8.x86_64 3/25 2024-03-20T08:46:03.594 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-selinux-2:17.2.7-717.g0192297e.el8.x86_64 4/25 2024-03-20T08:46:03.594 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libcephfs2-2:17.2.7-717.g0192297e.el8.x86_64 5/25 2024-03-20T08:46:03.594 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libradosstriper1-2:17.2.7-717.g0192297e.el8.x86_64 6/25 2024-03-20T08:46:03.594 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librgw2-2:17.2.7-717.g0192297e.el8.x86_64 7/25 2024-03-20T08:46:03.594 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-ceph-argparse-2:17.2.7-717.g0192297e.el8.x 8/25 2024-03-20T08:46:03.594 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-ceph-common-2:17.2.7-717.g0192297e.el8.x86 9/25 2024-03-20T08:46:03.594 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cephfs-2:17.2.7-717.g0192297e.el8.x86_64 10/25 2024-03-20T08:46:03.594 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rados-2:17.2.7-717.g0192297e.el8.x86_64 11/25 2024-03-20T08:46:03.594 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rbd-2:17.2.7-717.g0192297e.el8.x86_64 12/25 2024-03-20T08:46:03.595 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rgw-2:17.2.7-717.g0192297e.el8.x86_64 13/25 2024-03-20T08:46:03.595 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-03-20T08:46:03.595 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-03-20T08:46:03.595 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-03-20T08:46:03.595 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-03-20T08:46:03.595 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-03-20T08:46:03.595 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2024-03-20T08:46:03.595 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2024-03-20T08:46:03.595 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2024-03-20T08:46:03.595 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librados2-2:17.2.7-717.g0192297e.el8.x86_64 22/25 2024-03-20T08:46:03.595 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-03-20T08:46:03.596 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librbd1-2:17.2.7-717.g0192297e.el8.x86_64 24/25 2024-03-20T08:46:04.463 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-03-20T08:46:04.463 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:46:04.463 INFO:teuthology.orchestra.run.smithi026.stdout:Upgraded: 2024-03-20T08:46:04.463 INFO:teuthology.orchestra.run.smithi026.stdout: librados2-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:04.463 INFO:teuthology.orchestra.run.smithi026.stdout: librbd1-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:04.463 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-20T08:46:04.463 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-base-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:04.463 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-common-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:04.463 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-radosgw-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:04.464 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-selinux-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:04.464 INFO:teuthology.orchestra.run.smithi026.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-03-20T08:46:04.464 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs2-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:04.464 INFO:teuthology.orchestra.run.smithi026.stdout: liboath-2.6.2-3.el8.x86_64 2024-03-20T08:46:04.464 INFO:teuthology.orchestra.run.smithi026.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-03-20T08:46:04.464 INFO:teuthology.orchestra.run.smithi026.stdout: libradosstriper1-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:04.464 INFO:teuthology.orchestra.run.smithi026.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-03-20T08:46:04.464 INFO:teuthology.orchestra.run.smithi026.stdout: librgw2-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:04.464 INFO:teuthology.orchestra.run.smithi026.stdout: libunwind-1.3.1-3.el8.x86_64 2024-03-20T08:46:04.464 INFO:teuthology.orchestra.run.smithi026.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-03-20T08:46:04.464 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-argparse-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:04.464 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-common-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:04.464 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cephfs-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:04.464 INFO:teuthology.orchestra.run.smithi026.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-03-20T08:46:04.465 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rados-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:04.465 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rbd-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:04.465 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rgw-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:04.465 INFO:teuthology.orchestra.run.smithi026.stdout: thrift-0.13.0-2.el8.x86_64 2024-03-20T08:46:04.465 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:46:04.465 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-20T08:46:04.854 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-test 2024-03-20T08:46:05.227 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephsqlite-2:17.2.7-717.g0192297e.el8.x86_64 5/6 2024-03-20T08:46:05.485 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-test-2:17.2.7-717.g0192297e.el8.x86_64 6/6 2024-03-20T08:46:05.642 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:00:46 ago on Wed 20 Mar 2024 08:45:19 AM UTC. 2024-03-20T08:46:05.768 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-test-2:17.2.7-717.g0192297e.el8.x86_64 6/6 2024-03-20T08:46:05.768 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-test-2:17.2.7-717.g0192297e.el8.x86_64 1/6 2024-03-20T08:46:05.769 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephsqlite-2:17.2.7-717.g0192297e.el8.x86_64 2/6 2024-03-20T08:46:05.769 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-03-20T08:46:05.769 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-03-20T08:46:05.769 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-03-20T08:46:06.189 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-03-20T08:46:06.190 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:46:06.190 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-20T08:46:06.190 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-test-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:06.190 INFO:teuthology.orchestra.run.smithi122.stdout: jq-1.6-8.el8.x86_64 2024-03-20T08:46:06.190 INFO:teuthology.orchestra.run.smithi122.stdout: libcephsqlite-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:06.190 INFO:teuthology.orchestra.run.smithi122.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-03-20T08:46:06.190 INFO:teuthology.orchestra.run.smithi122.stdout: socat-1.7.4.1-1.el8.x86_64 2024-03-20T08:46:06.190 INFO:teuthology.orchestra.run.smithi122.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-03-20T08:46:06.190 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:46:06.190 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-20T08:46:06.385 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph 2024-03-20T08:46:06.761 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libcephsqlite-2:17.2.7-717.g0192297e.el8.x86_64 5/6 2024-03-20T08:46:07.003 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-test-2:17.2.7-717.g0192297e.el8.x86_64 6/6 2024-03-20T08:46:07.146 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:11 ago on Wed 20 Mar 2024 08:44:56 AM UTC. 2024-03-20T08:46:07.272 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-test-2:17.2.7-717.g0192297e.el8.x86_64 6/6 2024-03-20T08:46:07.272 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-test-2:17.2.7-717.g0192297e.el8.x86_64 1/6 2024-03-20T08:46:07.273 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libcephsqlite-2:17.2.7-717.g0192297e.el8.x86_64 2/6 2024-03-20T08:46:07.273 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-03-20T08:46:07.273 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-03-20T08:46:07.273 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-03-20T08:46:07.369 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-20T08:46:07.370 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:46:07.370 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-03-20T08:46:07.370 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:46:07.370 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-20T08:46:07.370 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-test x86_64 2:17.2.7-717.g0192297e.el8 ceph 52 M 2024-03-20T08:46:07.371 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-03-20T08:46:07.371 INFO:teuthology.orchestra.run.smithi026.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-03-20T08:46:07.371 INFO:teuthology.orchestra.run.smithi026.stdout: libcephsqlite x86_64 2:17.2.7-717.g0192297e.el8 ceph 179 k 2024-03-20T08:46:07.371 INFO:teuthology.orchestra.run.smithi026.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-03-20T08:46:07.371 INFO:teuthology.orchestra.run.smithi026.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-03-20T08:46:07.371 INFO:teuthology.orchestra.run.smithi026.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-03-20T08:46:07.371 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:46:07.371 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-20T08:46:07.371 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:46:07.371 INFO:teuthology.orchestra.run.smithi026.stdout:Install 6 Packages 2024-03-20T08:46:07.371 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:46:07.372 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 53 M 2024-03-20T08:46:07.372 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 234 M 2024-03-20T08:46:07.372 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-20T08:46:07.681 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-03-20T08:46:07.681 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:46:07.681 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-20T08:46:07.681 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-test-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:07.681 INFO:teuthology.orchestra.run.smithi115.stdout: jq-1.6-8.el8.x86_64 2024-03-20T08:46:07.681 INFO:teuthology.orchestra.run.smithi115.stdout: libcephsqlite-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:07.681 INFO:teuthology.orchestra.run.smithi115.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-03-20T08:46:07.681 INFO:teuthology.orchestra.run.smithi115.stdout: socat-1.7.4.1-1.el8.x86_64 2024-03-20T08:46:07.681 INFO:teuthology.orchestra.run.smithi115.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-03-20T08:46:07.682 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:46:07.682 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-20T08:46:07.705 INFO:teuthology.orchestra.run.smithi026.stdout:(1/6): jq-1.6-8.el8.x86_64.rpm 728 kB/s | 203 kB 00:00 2024-03-20T08:46:07.780 INFO:teuthology.orchestra.run.smithi026.stdout:(2/6): oniguruma-6.8.2-3.el8.x86_64.rpm 2.5 MB/s | 188 kB 00:00 2024-03-20T08:46:07.947 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph 2024-03-20T08:46:07.955 INFO:teuthology.orchestra.run.smithi026.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.8 MB/s | 323 kB 00:00 2024-03-20T08:46:07.980 INFO:teuthology.orchestra.run.smithi026.stdout:(4/6): libcephsqlite-17.2.7-717.g0192297e.el8.x 323 kB/s | 179 kB 00:00 2024-03-20T08:46:08.022 INFO:teuthology.orchestra.run.smithi026.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.0 MB/s | 71 kB 00:00 2024-03-20T08:46:08.700 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:14 ago on Wed 20 Mar 2024 08:44:54 AM UTC. 2024-03-20T08:46:08.732 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-20T08:46:08.736 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:46:08.736 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-03-20T08:46:08.736 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:46:08.736 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-20T08:46:08.736 INFO:teuthology.orchestra.run.smithi122.stdout: ceph x86_64 2:17.2.7-717.g0192297e.el8 ceph 6.4 k 2024-03-20T08:46:08.736 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-03-20T08:46:08.736 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mds x86_64 2:17.2.7-717.g0192297e.el8 ceph 2.3 M 2024-03-20T08:46:08.736 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr x86_64 2:17.2.7-717.g0192297e.el8 ceph 1.6 M 2024-03-20T08:46:08.736 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-modules-core noarch 2:17.2.7-717.g0192297e.el8 ceph-noarch 240 k 2024-03-20T08:46:08.736 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mon x86_64 2:17.2.7-717.g0192297e.el8 ceph 3.9 M 2024-03-20T08:46:08.737 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-osd x86_64 2:17.2.7-717.g0192297e.el8 ceph 17 M 2024-03-20T08:46:08.737 INFO:teuthology.orchestra.run.smithi122.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-03-20T08:46:08.737 INFO:teuthology.orchestra.run.smithi122.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-03-20T08:46:08.737 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-03-20T08:46:08.737 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-03-20T08:46:08.737 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-03-20T08:46:08.737 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-03-20T08:46:08.737 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-03-20T08:46:08.737 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-03-20T08:46:08.737 INFO:teuthology.orchestra.run.smithi122.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-03-20T08:46:08.737 INFO:teuthology.orchestra.run.smithi122.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-03-20T08:46:08.738 INFO:teuthology.orchestra.run.smithi122.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-03-20T08:46:08.738 INFO:teuthology.orchestra.run.smithi122.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-03-20T08:46:08.738 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-03-20T08:46:08.738 INFO:teuthology.orchestra.run.smithi122.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-03-20T08:46:08.738 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-03-20T08:46:08.738 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-03-20T08:46:08.738 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-03-20T08:46:08.738 INFO:teuthology.orchestra.run.smithi122.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-03-20T08:46:08.738 INFO:teuthology.orchestra.run.smithi122.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-03-20T08:46:08.738 INFO:teuthology.orchestra.run.smithi122.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-03-20T08:46:08.738 INFO:teuthology.orchestra.run.smithi122.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-03-20T08:46:08.738 INFO:teuthology.orchestra.run.smithi122.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-03-20T08:46:08.738 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-03-20T08:46:08.739 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-03-20T08:46:08.739 INFO:teuthology.orchestra.run.smithi122.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-03-20T08:46:08.739 INFO:teuthology.orchestra.run.smithi122.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-03-20T08:46:08.739 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:46:08.739 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-20T08:46:08.739 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:46:08.739 INFO:teuthology.orchestra.run.smithi122.stdout:Install 32 Packages 2024-03-20T08:46:08.739 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:46:08.740 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 28 M 2024-03-20T08:46:08.740 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 110 M 2024-03-20T08:46:08.740 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-20T08:46:09.804 INFO:teuthology.orchestra.run.smithi122.stdout:(1/32): ceph-17.2.7-717.g0192297e.el8.x86_64.rp 26 kB/s | 6.4 kB 00:00 2024-03-20T08:46:10.081 INFO:teuthology.orchestra.run.smithi026.stdout:(6/6): ceph-test-17.2.7-717.g0192297e.el8.x86_6 20 MB/s | 52 MB 00:02 2024-03-20T08:46:10.083 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:46:10.083 INFO:teuthology.orchestra.run.smithi026.stdout:Total 19 MB/s | 53 MB 00:02 2024-03-20T08:46:10.126 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-20T08:46:10.149 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-20T08:46:10.149 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-20T08:46:10.320 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-20T08:46:10.323 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:46:10.324 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2024-03-20T08:46:10.324 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:46:10.324 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-20T08:46:10.324 INFO:teuthology.orchestra.run.smithi115.stdout: ceph x86_64 2:17.2.7-717.g0192297e.el8 ceph 6.4 k 2024-03-20T08:46:10.324 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-03-20T08:46:10.325 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mds x86_64 2:17.2.7-717.g0192297e.el8 ceph 2.3 M 2024-03-20T08:46:10.325 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr x86_64 2:17.2.7-717.g0192297e.el8 ceph 1.6 M 2024-03-20T08:46:10.325 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-modules-core noarch 2:17.2.7-717.g0192297e.el8 ceph-noarch 240 k 2024-03-20T08:46:10.325 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mon x86_64 2:17.2.7-717.g0192297e.el8 ceph 3.9 M 2024-03-20T08:46:10.325 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-osd x86_64 2:17.2.7-717.g0192297e.el8 ceph 17 M 2024-03-20T08:46:10.325 INFO:teuthology.orchestra.run.smithi115.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-03-20T08:46:10.326 INFO:teuthology.orchestra.run.smithi115.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-03-20T08:46:10.326 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-03-20T08:46:10.326 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-03-20T08:46:10.326 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-03-20T08:46:10.326 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-03-20T08:46:10.326 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-03-20T08:46:10.327 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-03-20T08:46:10.327 INFO:teuthology.orchestra.run.smithi115.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-03-20T08:46:10.327 INFO:teuthology.orchestra.run.smithi115.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-03-20T08:46:10.327 INFO:teuthology.orchestra.run.smithi115.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-03-20T08:46:10.327 INFO:teuthology.orchestra.run.smithi115.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-03-20T08:46:10.328 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-03-20T08:46:10.328 INFO:teuthology.orchestra.run.smithi115.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-03-20T08:46:10.328 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-03-20T08:46:10.328 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-03-20T08:46:10.328 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-03-20T08:46:10.328 INFO:teuthology.orchestra.run.smithi115.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-03-20T08:46:10.329 INFO:teuthology.orchestra.run.smithi115.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-03-20T08:46:10.329 INFO:teuthology.orchestra.run.smithi115.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-03-20T08:46:10.329 INFO:teuthology.orchestra.run.smithi115.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-03-20T08:46:10.329 INFO:teuthology.orchestra.run.smithi115.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-03-20T08:46:10.329 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-03-20T08:46:10.329 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-03-20T08:46:10.330 INFO:teuthology.orchestra.run.smithi115.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-03-20T08:46:10.330 INFO:teuthology.orchestra.run.smithi115.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-03-20T08:46:10.330 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:46:10.330 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-20T08:46:10.330 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:46:10.331 INFO:teuthology.orchestra.run.smithi115.stdout:Install 32 Packages 2024-03-20T08:46:10.331 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:46:10.332 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 28 M 2024-03-20T08:46:10.332 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 110 M 2024-03-20T08:46:10.332 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-20T08:46:10.451 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-20T08:46:10.455 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-20T08:46:10.864 INFO:teuthology.orchestra.run.smithi115.stdout:(1/32): ceph-17.2.7-717.g0192297e.el8.x86_64.rp 18 kB/s | 6.4 kB 00:00 2024-03-20T08:46:10.943 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-20T08:46:10.971 INFO:teuthology.orchestra.run.smithi122.stdout:(2/32): ceph-mgr-17.2.7-717.g0192297e.el8.x86_6 1.1 MB/s | 1.6 MB 00:01 2024-03-20T08:46:11.209 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-03-20T08:46:11.437 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-03-20T08:46:11.558 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-03-20T08:46:11.636 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-03-20T08:46:11.654 INFO:teuthology.orchestra.run.smithi122.stdout:(3/32): ceph-mds-17.2.7-717.g0192297e.el8.x86_6 1.1 MB/s | 2.3 MB 00:02 2024-03-20T08:46:11.838 INFO:teuthology.orchestra.run.smithi122.stdout:(4/32): ceph-mgr-modules-core-17.2.7-717.g01922 1.3 MB/s | 240 kB 00:00 2024-03-20T08:46:11.889 INFO:teuthology.orchestra.run.smithi115.stdout:(2/32): ceph-mds-17.2.7-717.g0192297e.el8.x86_6 1.7 MB/s | 2.3 MB 00:01 2024-03-20T08:46:11.985 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-03-20T08:46:12.013 INFO:teuthology.orchestra.run.smithi122.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 898 kB/s | 157 kB 00:00 2024-03-20T08:46:12.072 INFO:teuthology.orchestra.run.smithi122.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 666 kB/s | 39 kB 00:00 2024-03-20T08:46:12.130 INFO:teuthology.orchestra.run.smithi122.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.7 MB/s | 103 kB 00:00 2024-03-20T08:46:12.166 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libcephsqlite-2:17.2.7-717.g0192297e.el8.x86_64 5/6 2024-03-20T08:46:12.189 INFO:teuthology.orchestra.run.smithi122.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 979 kB/s | 57 kB 00:00 2024-03-20T08:46:12.281 INFO:teuthology.orchestra.run.smithi122.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 4.9 MB/s | 457 kB 00:00 2024-03-20T08:46:12.306 INFO:teuthology.orchestra.run.smithi115.stdout:(3/32): ceph-mgr-17.2.7-717.g0192297e.el8.x86_6 893 kB/s | 1.6 MB 00:01 2024-03-20T08:46:12.540 INFO:teuthology.orchestra.run.smithi115.stdout:(4/32): ceph-mgr-modules-core-17.2.7-717.g01922 1.0 MB/s | 240 kB 00:00 2024-03-20T08:46:12.715 INFO:teuthology.orchestra.run.smithi115.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 898 kB/s | 157 kB 00:00 2024-03-20T08:46:12.773 INFO:teuthology.orchestra.run.smithi115.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 668 kB/s | 39 kB 00:00 2024-03-20T08:46:12.824 INFO:teuthology.orchestra.run.smithi115.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 2.0 MB/s | 103 kB 00:00 2024-03-20T08:46:12.882 INFO:teuthology.orchestra.run.smithi115.stdout:(8/32): ceph-mon-17.2.7-717.g0192297e.el8.x86_6 2.0 MB/s | 3.9 MB 00:02 2024-03-20T08:46:12.931 INFO:teuthology.orchestra.run.smithi122.stdout:(10/32): ceph-mon-17.2.7-717.g0192297e.el8.x86_ 1.3 MB/s | 3.9 MB 00:03 2024-03-20T08:46:12.940 INFO:teuthology.orchestra.run.smithi115.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 490 kB/s | 57 kB 00:00 2024-03-20T08:46:13.140 INFO:teuthology.orchestra.run.smithi122.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 290 kB/s | 249 kB 00:00 2024-03-20T08:46:13.216 INFO:teuthology.orchestra.run.smithi115.stdout:(10/32): ceph-osd-17.2.7-717.g0192297e.el8.x86_ 13 MB/s | 17 MB 00:01 2024-03-20T08:46:13.316 INFO:teuthology.orchestra.run.smithi115.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.0 MB/s | 457 kB 00:00 2024-03-20T08:46:13.450 INFO:teuthology.orchestra.run.smithi115.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 488 kB/s | 249 kB 00:00 2024-03-20T08:46:13.599 INFO:teuthology.orchestra.run.smithi122.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 238 kB/s | 109 kB 00:00 2024-03-20T08:46:13.624 INFO:teuthology.orchestra.run.smithi122.stdout:(13/32): python3-cryptography-3.2.1-7.el8.x86_6 926 kB/s | 641 kB 00:00 2024-03-20T08:46:13.767 INFO:teuthology.orchestra.run.smithi115.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 242 kB/s | 109 kB 00:00 2024-03-20T08:46:13.793 INFO:teuthology.orchestra.run.smithi115.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 127 kB/s | 44 kB 00:00 2024-03-20T08:46:13.799 INFO:teuthology.orchestra.run.smithi122.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 218 kB/s | 44 kB 00:00 2024-03-20T08:46:13.841 INFO:teuthology.orchestra.run.smithi122.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 854 kB/s | 185 kB 00:00 2024-03-20T08:46:13.851 INFO:teuthology.orchestra.run.smithi115.stdout:(15/32): python3-cryptography-3.2.1-7.el8.x86_6 1.0 MB/s | 641 kB 00:00 2024-03-20T08:46:13.900 INFO:teuthology.orchestra.run.smithi122.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.7 MB/s | 173 kB 00:00 2024-03-20T08:46:13.925 INFO:teuthology.orchestra.run.smithi122.stdout:(17/32): python3-cherrypy-18.4.0-1.el8.noarch.r 4.5 MB/s | 384 kB 00:00 2024-03-20T08:46:13.967 INFO:teuthology.orchestra.run.smithi122.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 165 kB/s | 11 kB 00:00 2024-03-20T08:46:14.029 INFO:teuthology.orchestra.run.smithi122.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 169 kB/s | 18 kB 00:00 2024-03-20T08:46:14.059 INFO:teuthology.orchestra.run.smithi122.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 530 kB/s | 49 kB 00:00 2024-03-20T08:46:14.093 INFO:teuthology.orchestra.run.smithi122.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 921 kB/s | 59 kB 00:00 2024-03-20T08:46:14.160 INFO:teuthology.orchestra.run.smithi122.stdout:(22/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.8 MB/s | 283 kB 00:00 2024-03-20T08:46:14.186 INFO:teuthology.orchestra.run.smithi122.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 169 kB/s | 16 kB 00:00 2024-03-20T08:46:14.201 INFO:teuthology.orchestra.run.smithi115.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 427 kB/s | 185 kB 00:00 2024-03-20T08:46:14.218 INFO:teuthology.orchestra.run.smithi115.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 406 kB/s | 173 kB 00:00 2024-03-20T08:46:14.227 INFO:teuthology.orchestra.run.smithi122.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 280 kB/s | 19 kB 00:00 2024-03-20T08:46:14.253 INFO:teuthology.orchestra.run.smithi122.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 357 kB/s | 24 kB 00:00 2024-03-20T08:46:14.287 INFO:teuthology.orchestra.run.smithi122.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 500 kB/s | 29 kB 00:00 2024-03-20T08:46:14.320 INFO:teuthology.orchestra.run.smithi122.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 411 kB/s | 27 kB 00:00 2024-03-20T08:46:14.370 INFO:teuthology.orchestra.run.smithi122.stdout:(28/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.0 MB/s | 255 kB 00:00 2024-03-20T08:46:14.385 INFO:teuthology.orchestra.run.smithi115.stdout:(18/32): python3-cherrypy-18.4.0-1.el8.noarch.r 721 kB/s | 384 kB 00:00 2024-03-20T08:46:14.410 INFO:teuthology.orchestra.run.smithi115.stdout:(19/32): python3-jaraco-6.2-6.el8.noarch.rpm 53 kB/s | 11 kB 00:00 2024-03-20T08:46:14.421 INFO:teuthology.orchestra.run.smithi122.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.5 MB/s | 251 kB 00:00 2024-03-20T08:46:14.483 INFO:teuthology.orchestra.run.smithi122.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 760 kB/s | 85 kB 00:00 2024-03-20T08:46:14.486 INFO:teuthology.orchestra.run.smithi115.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 66 kB/s | 18 kB 00:00 2024-03-20T08:46:14.513 INFO:teuthology.orchestra.run.smithi122.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 250 kB/s | 23 kB 00:00 2024-03-20T08:46:14.753 INFO:teuthology.orchestra.run.smithi115.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 133 kB/s | 49 kB 00:00 2024-03-20T08:46:14.770 INFO:teuthology.orchestra.run.smithi115.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 164 kB/s | 59 kB 00:00 2024-03-20T08:46:14.987 INFO:teuthology.orchestra.run.smithi115.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 565 kB/s | 283 kB 00:00 2024-03-20T08:46:15.012 INFO:teuthology.orchestra.run.smithi115.stdout:(24/32): python3-portend-2.6-1.el8.noarch.rpm 60 kB/s | 16 kB 00:00 2024-03-20T08:46:15.029 INFO:teuthology.orchestra.run.smithi115.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 72 kB/s | 19 kB 00:00 2024-03-20T08:46:15.254 INFO:teuthology.orchestra.run.smithi115.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 89 kB/s | 24 kB 00:00 2024-03-20T08:46:15.304 INFO:teuthology.orchestra.run.smithi115.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 100 kB/s | 29 kB 00:00 2024-03-20T08:46:15.330 INFO:teuthology.orchestra.run.smithi115.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 91 kB/s | 27 kB 00:00 2024-03-20T08:46:15.680 INFO:teuthology.orchestra.run.smithi115.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 244 kB/s | 85 kB 00:00 2024-03-20T08:46:15.797 INFO:teuthology.orchestra.run.smithi115.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 471 kB/s | 255 kB 00:00 2024-03-20T08:46:15.822 INFO:teuthology.orchestra.run.smithi115.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 485 kB/s | 251 kB 00:00 2024-03-20T08:46:15.931 INFO:teuthology.orchestra.run.smithi115.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 92 kB/s | 23 kB 00:00 2024-03-20T08:46:15.934 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:46:15.934 INFO:teuthology.orchestra.run.smithi115.stdout:Total 5.1 MB/s | 28 MB 00:05 2024-03-20T08:46:16.015 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-20T08:46:16.051 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-20T08:46:16.051 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-20T08:46:16.383 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-20T08:46:16.386 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-20T08:46:17.484 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libcephsqlite-2:17.2.7-717.g0192297e.el8.x86_64 5/6 2024-03-20T08:46:17.496 INFO:teuthology.orchestra.run.smithi122.stdout:(32/32): ceph-osd-17.2.7-717.g0192297e.el8.x86_ 2.6 MB/s | 17 MB 00:06 2024-03-20T08:46:17.500 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:46:17.501 INFO:teuthology.orchestra.run.smithi122.stdout:Total 3.2 MB/s | 28 MB 00:08 2024-03-20T08:46:17.538 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-20T08:46:17.581 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-20T08:46:17.618 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-20T08:46:17.618 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-20T08:46:17.740 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-test-2:17.2.7-717.g0192297e.el8.x86_64 6/6 2024-03-20T08:46:17.790 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-03-20T08:46:17.918 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-20T08:46:17.921 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-20T08:46:18.026 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-test-2:17.2.7-717.g0192297e.el8.x86_64 6/6 2024-03-20T08:46:18.027 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-test-2:17.2.7-717.g0192297e.el8.x86_64 1/6 2024-03-20T08:46:18.027 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libcephsqlite-2:17.2.7-717.g0192297e.el8.x86_64 2/6 2024-03-20T08:46:18.027 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-03-20T08:46:18.027 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-03-20T08:46:18.027 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-03-20T08:46:18.057 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-03-20T08:46:18.334 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-03-20T08:46:18.489 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-03-20T08:46:18.489 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:46:18.489 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-20T08:46:18.489 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-test-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:18.489 INFO:teuthology.orchestra.run.smithi026.stdout: jq-1.6-8.el8.x86_64 2024-03-20T08:46:18.490 INFO:teuthology.orchestra.run.smithi026.stdout: libcephsqlite-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:18.490 INFO:teuthology.orchestra.run.smithi026.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-03-20T08:46:18.490 INFO:teuthology.orchestra.run.smithi026.stdout: socat-1.7.4.1-1.el8.x86_64 2024-03-20T08:46:18.490 INFO:teuthology.orchestra.run.smithi026.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-03-20T08:46:18.490 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:46:18.490 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-20T08:46:18.558 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-03-20T08:46:18.782 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph 2024-03-20T08:46:18.789 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-03-20T08:46:18.985 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-03-20T08:46:19.067 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-20T08:46:19.188 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-03-20T08:46:19.290 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-03-20T08:46:19.357 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-03-20T08:46:19.523 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-03-20T08:46:19.548 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-03-20T08:46:19.569 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:00 ago on Wed 20 Mar 2024 08:45:19 AM UTC. 2024-03-20T08:46:19.726 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-03-20T08:46:19.795 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-03-20T08:46:19.999 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-03-20T08:46:19.999 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-03-20T08:46:20.189 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-03-20T08:46:20.290 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-03-20T08:46:20.412 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-03-20T08:46:20.621 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-03-20T08:46:20.664 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-03-20T08:46:20.832 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-03-20T08:46:20.920 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-03-20T08:46:21.032 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-03-20T08:46:21.142 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-03-20T08:46:21.262 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-03-20T08:46:21.307 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-20T08:46:21.310 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:46:21.311 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-03-20T08:46:21.311 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:46:21.311 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-20T08:46:21.311 INFO:teuthology.orchestra.run.smithi026.stdout: ceph x86_64 2:17.2.7-717.g0192297e.el8 ceph 6.4 k 2024-03-20T08:46:21.311 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-03-20T08:46:21.311 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mds x86_64 2:17.2.7-717.g0192297e.el8 ceph 2.3 M 2024-03-20T08:46:21.311 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr x86_64 2:17.2.7-717.g0192297e.el8 ceph 1.6 M 2024-03-20T08:46:21.311 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-modules-core noarch 2:17.2.7-717.g0192297e.el8 ceph-noarch 240 k 2024-03-20T08:46:21.312 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mon x86_64 2:17.2.7-717.g0192297e.el8 ceph 3.9 M 2024-03-20T08:46:21.312 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-osd x86_64 2:17.2.7-717.g0192297e.el8 ceph 17 M 2024-03-20T08:46:21.312 INFO:teuthology.orchestra.run.smithi026.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-03-20T08:46:21.312 INFO:teuthology.orchestra.run.smithi026.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-03-20T08:46:21.312 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-03-20T08:46:21.312 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-03-20T08:46:21.312 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-03-20T08:46:21.312 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-03-20T08:46:21.312 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-03-20T08:46:21.312 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-03-20T08:46:21.312 INFO:teuthology.orchestra.run.smithi026.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-03-20T08:46:21.312 INFO:teuthology.orchestra.run.smithi026.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-03-20T08:46:21.312 INFO:teuthology.orchestra.run.smithi026.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-03-20T08:46:21.313 INFO:teuthology.orchestra.run.smithi026.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-03-20T08:46:21.313 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-03-20T08:46:21.313 INFO:teuthology.orchestra.run.smithi026.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-03-20T08:46:21.313 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-03-20T08:46:21.313 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-03-20T08:46:21.313 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-03-20T08:46:21.313 INFO:teuthology.orchestra.run.smithi026.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-03-20T08:46:21.313 INFO:teuthology.orchestra.run.smithi026.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-03-20T08:46:21.313 INFO:teuthology.orchestra.run.smithi026.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-03-20T08:46:21.313 INFO:teuthology.orchestra.run.smithi026.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-03-20T08:46:21.313 INFO:teuthology.orchestra.run.smithi026.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-03-20T08:46:21.313 INFO:teuthology.orchestra.run.smithi026.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-03-20T08:46:21.313 INFO:teuthology.orchestra.run.smithi026.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-03-20T08:46:21.314 INFO:teuthology.orchestra.run.smithi026.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-03-20T08:46:21.314 INFO:teuthology.orchestra.run.smithi026.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-03-20T08:46:21.314 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:46:21.314 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-20T08:46:21.314 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:46:21.314 INFO:teuthology.orchestra.run.smithi026.stdout:Install 32 Packages 2024-03-20T08:46:21.314 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:46:21.315 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 28 M 2024-03-20T08:46:21.315 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 110 M 2024-03-20T08:46:21.315 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-20T08:46:21.418 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-03-20T08:46:21.474 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-03-20T08:46:21.652 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-03-20T08:46:21.731 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-03-20T08:46:21.984 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-03-20T08:46:22.058 INFO:teuthology.orchestra.run.smithi026.stdout:(1/32): ceph-17.2.7-717.g0192297e.el8.x86_64.rp 35 kB/s | 6.4 kB 00:00 2024-03-20T08:46:22.090 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-03-20T08:46:22.223 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-03-20T08:46:22.250 INFO:teuthology.orchestra.run.smithi026.stdout:(2/32): ceph-mds-17.2.7-717.g0192297e.el8.x86_6 6.1 MB/s | 2.3 MB 00:00 2024-03-20T08:46:22.362 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-03-20T08:46:22.387 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-03-20T08:46:22.392 INFO:teuthology.orchestra.run.smithi026.stdout:(3/32): ceph-mgr-17.2.7-717.g0192297e.el8.x86_6 3.0 MB/s | 1.6 MB 00:00 2024-03-20T08:46:22.492 INFO:teuthology.orchestra.run.smithi026.stdout:(4/32): ceph-mgr-modules-core-17.2.7-717.g01922 2.3 MB/s | 240 kB 00:00 2024-03-20T08:46:22.589 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-03-20T08:46:22.592 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-03-20T08:46:22.659 INFO:teuthology.orchestra.run.smithi026.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 942 kB/s | 157 kB 00:00 2024-03-20T08:46:22.709 INFO:teuthology.orchestra.run.smithi026.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 783 kB/s | 39 kB 00:00 2024-03-20T08:46:22.793 INFO:teuthology.orchestra.run.smithi026.stdout:(7/32): ceph-mon-17.2.7-717.g0192297e.el8.x86_6 5.4 MB/s | 3.9 MB 00:00 2024-03-20T08:46:22.819 INFO:teuthology.orchestra.run.smithi026.stdout:(8/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 950 kB/s | 103 kB 00:00 2024-03-20T08:46:22.844 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-03-20T08:46:22.866 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-03-20T08:46:22.902 INFO:teuthology.orchestra.run.smithi026.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 5.4 MB/s | 457 kB 00:00 2024-03-20T08:46:22.928 INFO:teuthology.orchestra.run.smithi026.stdout:(10/32): python3-pytz-2017.2-11.el8.noarch.rpm 425 kB/s | 57 kB 00:00 2024-03-20T08:46:23.097 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-03-20T08:46:23.110 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-03-20T08:46:23.228 INFO:teuthology.orchestra.run.smithi026.stdout:(11/32): ceph-osd-17.2.7-717.g0192297e.el8.x86_ 17 MB/s | 17 MB 00:00 2024-03-20T08:46:23.323 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-03-20T08:46:23.437 INFO:teuthology.orchestra.run.smithi026.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 466 kB/s | 249 kB 00:00 2024-03-20T08:46:23.529 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-03-20T08:46:23.553 INFO:teuthology.orchestra.run.smithi026.stdout:(13/32): python3-cryptography-3.2.1-7.el8.x86_6 1.0 MB/s | 641 kB 00:00 2024-03-20T08:46:23.654 INFO:teuthology.orchestra.run.smithi026.stdout:(14/32): python3-pycparser-2.14-14.el8.noarch.r 256 kB/s | 109 kB 00:00 2024-03-20T08:46:23.729 INFO:teuthology.orchestra.run.smithi026.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 149 kB/s | 44 kB 00:00 2024-03-20T08:46:23.896 INFO:teuthology.orchestra.run.smithi026.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 543 kB/s | 185 kB 00:00 2024-03-20T08:46:23.954 INFO:teuthology.orchestra.run.smithi026.stdout:(17/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.7 MB/s | 384 kB 00:00 2024-03-20T08:46:24.006 INFO:teuthology.orchestra.run.smithi026.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 100 kB/s | 11 kB 00:00 2024-03-20T08:46:24.038 INFO:teuthology.orchestra.run.smithi026.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 210 kB/s | 18 kB 00:00 2024-03-20T08:46:24.064 INFO:teuthology.orchestra.run.smithi026.stdout:(20/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 422 kB/s | 173 kB 00:00 2024-03-20T08:46:24.089 INFO:teuthology.orchestra.run.smithi026.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 594 kB/s | 49 kB 00:00 2024-03-20T08:46:24.114 INFO:teuthology.orchestra.run.smithi026.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 777 kB/s | 59 kB 00:00 2024-03-20T08:46:24.156 INFO:teuthology.orchestra.run.smithi026.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 233 kB/s | 16 kB 00:00 2024-03-20T08:46:24.191 INFO:teuthology.orchestra.run.smithi026.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 248 kB/s | 19 kB 00:00 2024-03-20T08:46:24.215 INFO:teuthology.orchestra.run.smithi026.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.8 MB/s | 283 kB 00:00 2024-03-20T08:46:24.233 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-03-20T08:46:24.240 INFO:teuthology.orchestra.run.smithi026.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 285 kB/s | 24 kB 00:00 2024-03-20T08:46:24.266 INFO:teuthology.orchestra.run.smithi026.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 387 kB/s | 29 kB 00:00 2024-03-20T08:46:24.291 INFO:teuthology.orchestra.run.smithi026.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 364 kB/s | 27 kB 00:00 2024-03-20T08:46:24.351 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-03-20T08:46:24.366 INFO:teuthology.orchestra.run.smithi026.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.5 MB/s | 251 kB 00:00 2024-03-20T08:46:24.392 INFO:teuthology.orchestra.run.smithi026.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.7 MB/s | 255 kB 00:00 2024-03-20T08:46:24.405 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-03-20T08:46:24.408 INFO:teuthology.orchestra.run.smithi026.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 730 kB/s | 85 kB 00:00 2024-03-20T08:46:24.442 INFO:teuthology.orchestra.run.smithi026.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 306 kB/s | 23 kB 00:00 2024-03-20T08:46:24.447 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:46:24.447 INFO:teuthology.orchestra.run.smithi026.stdout:Total 9.1 MB/s | 28 MB 00:03 2024-03-20T08:46:24.533 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-20T08:46:24.573 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-20T08:46:24.573 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-20T08:46:24.623 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-03-20T08:46:24.690 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-03-20T08:46:24.853 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-03-20T08:46:24.904 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-20T08:46:24.907 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-20T08:46:25.056 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-717.g0192297e.el8.n 27/32 2024-03-20T08:46:25.105 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-03-20T08:46:25.259 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-2:17.2.7-717.g0192297e.el8.x86_64 28/32 2024-03-20T08:46:25.329 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-03-20T08:46:25.693 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-03-20T08:46:26.005 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-03-20T08:46:26.105 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-20T08:46:26.347 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-717.g0192297e.el8.n 27/32 2024-03-20T08:46:26.382 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-03-20T08:46:26.534 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-2:17.2.7-717.g0192297e.el8.x86_64 28/32 2024-03-20T08:46:26.655 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-03-20T08:46:26.752 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-2:17.2.7-717.g0192297e.el8.x86_64 28/32 2024-03-20T08:46:26.944 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-03-20T08:46:26.967 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-osd-2:17.2.7-717.g0192297e.el8.x86_64 29/32 2024-03-20T08:46:27.246 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-03-20T08:46:27.360 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-osd-2:17.2.7-717.g0192297e.el8.x86_64 29/32 2024-03-20T08:46:27.420 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-03-20T08:46:27.558 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mon-2:17.2.7-717.g0192297e.el8.x86_64 30/32 2024-03-20T08:46:27.653 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-03-20T08:46:27.774 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mon-2:17.2.7-717.g0192297e.el8.x86_64 30/32 2024-03-20T08:46:27.867 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-03-20T08:46:27.975 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mds-2:17.2.7-717.g0192297e.el8.x86_64 31/32 2024-03-20T08:46:27.993 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mds-2:17.2.7-717.g0192297e.el8.x86_64 31/32 2024-03-20T08:46:28.019 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-2:17.2.7-717.g0192297e.el8.x86_64 28/32 2024-03-20T08:46:28.062 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-03-20T08:46:28.132 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-2:17.2.7-717.g0192297e.el8.x86_64 32/32 2024-03-20T08:46:28.226 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-osd-2:17.2.7-717.g0192297e.el8.x86_64 29/32 2024-03-20T08:46:28.308 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-03-20T08:46:28.544 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-2:17.2.7-717.g0192297e.el8.x86_64 32/32 2024-03-20T08:46:28.544 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-2:17.2.7-717.g0192297e.el8.x86_64 1/32 2024-03-20T08:46:28.544 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mds-2:17.2.7-717.g0192297e.el8.x86_64 2/32 2024-03-20T08:46:28.544 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-2:17.2.7-717.g0192297e.el8.x86_64 3/32 2024-03-20T08:46:28.544 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mon-2:17.2.7-717.g0192297e.el8.x86_64 4/32 2024-03-20T08:46:28.544 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-osd-2:17.2.7-717.g0192297e.el8.x86_64 5/32 2024-03-20T08:46:28.544 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-717.g0192297e.el8.n 6/32 2024-03-20T08:46:28.544 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-03-20T08:46:28.544 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-03-20T08:46:28.545 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-03-20T08:46:28.545 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-03-20T08:46:28.545 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-03-20T08:46:28.545 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-03-20T08:46:28.545 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-03-20T08:46:28.545 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-03-20T08:46:28.545 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-03-20T08:46:28.545 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-03-20T08:46:28.545 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-03-20T08:46:28.545 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-03-20T08:46:28.545 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-03-20T08:46:28.545 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-03-20T08:46:28.545 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-03-20T08:46:28.546 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-03-20T08:46:28.546 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-03-20T08:46:28.546 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-03-20T08:46:28.546 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-03-20T08:46:28.546 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-03-20T08:46:28.546 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-03-20T08:46:28.546 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-03-20T08:46:28.546 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-03-20T08:46:28.546 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-03-20T08:46:28.546 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-03-20T08:46:28.568 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-03-20T08:46:28.618 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-osd-2:17.2.7-717.g0192297e.el8.x86_64 29/32 2024-03-20T08:46:28.790 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-03-20T08:46:28.817 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mon-2:17.2.7-717.g0192297e.el8.x86_64 30/32 2024-03-20T08:46:29.034 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mon-2:17.2.7-717.g0192297e.el8.x86_64 30/32 2024-03-20T08:46:29.056 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-03-20T08:46:29.417 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mds-2:17.2.7-717.g0192297e.el8.x86_64 31/32 2024-03-20T08:46:29.435 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mds-2:17.2.7-717.g0192297e.el8.x86_64 31/32 2024-03-20T08:46:29.463 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-03-20T08:46:29.463 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:46:29.463 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-20T08:46:29.463 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:29.463 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mds-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:29.463 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:29.463 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-modules-core-2:17.2.7-717.g0192297e.el8.noarch 2024-03-20T08:46:29.463 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mon-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:29.463 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-osd-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:29.463 INFO:teuthology.orchestra.run.smithi115.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-03-20T08:46:29.463 INFO:teuthology.orchestra.run.smithi115.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-03-20T08:46:29.463 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-03-20T08:46:29.464 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-03-20T08:46:29.464 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-03-20T08:46:29.464 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-03-20T08:46:29.464 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-6.2-6.el8.noarch 2024-03-20T08:46:29.464 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-03-20T08:46:29.464 INFO:teuthology.orchestra.run.smithi115.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-03-20T08:46:29.464 INFO:teuthology.orchestra.run.smithi115.stdout: python3-mako-1.0.6-14.el8.noarch 2024-03-20T08:46:29.464 INFO:teuthology.orchestra.run.smithi115.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-03-20T08:46:29.464 INFO:teuthology.orchestra.run.smithi115.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-03-20T08:46:29.464 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-03-20T08:46:29.464 INFO:teuthology.orchestra.run.smithi115.stdout: python3-portend-2.6-1.el8.noarch 2024-03-20T08:46:29.464 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-03-20T08:46:29.464 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pycparser-2.14-14.el8.noarch 2024-03-20T08:46:29.464 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pytz-2017.2-11.el8.noarch 2024-03-20T08:46:29.464 INFO:teuthology.orchestra.run.smithi115.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-03-20T08:46:29.465 INFO:teuthology.orchestra.run.smithi115.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-03-20T08:46:29.465 INFO:teuthology.orchestra.run.smithi115.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-03-20T08:46:29.465 INFO:teuthology.orchestra.run.smithi115.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-03-20T08:46:29.465 INFO:teuthology.orchestra.run.smithi115.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-03-20T08:46:29.465 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-03-20T08:46:29.465 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-03-20T08:46:29.465 INFO:teuthology.orchestra.run.smithi115.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-03-20T08:46:29.465 INFO:teuthology.orchestra.run.smithi115.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-03-20T08:46:29.465 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:46:29.465 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-20T08:46:29.498 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-03-20T08:46:29.608 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-2:17.2.7-717.g0192297e.el8.x86_64 32/32 2024-03-20T08:46:29.720 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-base 2024-03-20T08:46:29.743 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-03-20T08:46:29.964 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-03-20T08:46:30.018 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-2:17.2.7-717.g0192297e.el8.x86_64 32/32 2024-03-20T08:46:30.018 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-2:17.2.7-717.g0192297e.el8.x86_64 1/32 2024-03-20T08:46:30.018 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mds-2:17.2.7-717.g0192297e.el8.x86_64 2/32 2024-03-20T08:46:30.018 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-2:17.2.7-717.g0192297e.el8.x86_64 3/32 2024-03-20T08:46:30.018 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mon-2:17.2.7-717.g0192297e.el8.x86_64 4/32 2024-03-20T08:46:30.018 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-osd-2:17.2.7-717.g0192297e.el8.x86_64 5/32 2024-03-20T08:46:30.018 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-717.g0192297e.el8.n 6/32 2024-03-20T08:46:30.019 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-03-20T08:46:30.019 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-03-20T08:46:30.019 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-03-20T08:46:30.019 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-03-20T08:46:30.019 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-03-20T08:46:30.019 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-03-20T08:46:30.019 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-03-20T08:46:30.019 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-03-20T08:46:30.019 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-03-20T08:46:30.019 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-03-20T08:46:30.019 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-03-20T08:46:30.019 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-03-20T08:46:30.019 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-03-20T08:46:30.019 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-03-20T08:46:30.019 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-03-20T08:46:30.020 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-03-20T08:46:30.020 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-03-20T08:46:30.020 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-03-20T08:46:30.020 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-03-20T08:46:30.020 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-03-20T08:46:30.020 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-03-20T08:46:30.020 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-03-20T08:46:30.021 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-03-20T08:46:30.021 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-03-20T08:46:30.021 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-03-20T08:46:30.286 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-03-20T08:46:30.472 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:36 ago on Wed 20 Mar 2024 08:44:54 AM UTC. 2024-03-20T08:46:30.532 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-03-20T08:46:30.822 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-03-20T08:46:30.822 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:46:30.822 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-20T08:46:30.822 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:30.822 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mds-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:30.822 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:30.822 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-modules-core-2:17.2.7-717.g0192297e.el8.noarch 2024-03-20T08:46:30.822 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mon-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:30.823 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-osd-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:30.823 INFO:teuthology.orchestra.run.smithi122.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-03-20T08:46:30.823 INFO:teuthology.orchestra.run.smithi122.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-03-20T08:46:30.823 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-03-20T08:46:30.823 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-03-20T08:46:30.823 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-03-20T08:46:30.823 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-03-20T08:46:30.823 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-6.2-6.el8.noarch 2024-03-20T08:46:30.823 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-03-20T08:46:30.823 INFO:teuthology.orchestra.run.smithi122.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-03-20T08:46:30.823 INFO:teuthology.orchestra.run.smithi122.stdout: python3-mako-1.0.6-14.el8.noarch 2024-03-20T08:46:30.823 INFO:teuthology.orchestra.run.smithi122.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-03-20T08:46:30.823 INFO:teuthology.orchestra.run.smithi122.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-03-20T08:46:30.824 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-03-20T08:46:30.824 INFO:teuthology.orchestra.run.smithi122.stdout: python3-portend-2.6-1.el8.noarch 2024-03-20T08:46:30.824 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-03-20T08:46:30.824 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pycparser-2.14-14.el8.noarch 2024-03-20T08:46:30.824 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytz-2017.2-11.el8.noarch 2024-03-20T08:46:30.824 INFO:teuthology.orchestra.run.smithi122.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-03-20T08:46:30.824 INFO:teuthology.orchestra.run.smithi122.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-03-20T08:46:30.824 INFO:teuthology.orchestra.run.smithi122.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-03-20T08:46:30.824 INFO:teuthology.orchestra.run.smithi122.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-03-20T08:46:30.824 INFO:teuthology.orchestra.run.smithi122.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-03-20T08:46:30.824 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-03-20T08:46:30.824 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-03-20T08:46:30.824 INFO:teuthology.orchestra.run.smithi122.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-03-20T08:46:30.825 INFO:teuthology.orchestra.run.smithi122.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-03-20T08:46:30.825 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:46:30.825 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-20T08:46:30.906 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-03-20T08:46:31.130 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-base 2024-03-20T08:46:31.171 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-03-20T08:46:31.335 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-03-20T08:46:31.561 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-03-20T08:46:31.839 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-03-20T08:46:31.881 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:35 ago on Wed 20 Mar 2024 08:44:56 AM UTC. 2024-03-20T08:46:32.045 INFO:teuthology.orchestra.run.smithi115.stdout:Package ceph-base-2:17.2.7-717.g0192297e.el8.x86_64 is already installed. 2024-03-20T08:46:32.094 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-20T08:46:32.095 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-03-20T08:46:32.095 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-20T08:46:32.111 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-03-20T08:46:32.199 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install cephadm 2024-03-20T08:46:32.317 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-03-20T08:46:32.652 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-03-20T08:46:32.942 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:38 ago on Wed 20 Mar 2024 08:44:54 AM UTC. 2024-03-20T08:46:32.943 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-03-20T08:46:33.323 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-717.g0192297e.el8.n 27/32 2024-03-20T08:46:33.433 INFO:teuthology.orchestra.run.smithi122.stdout:Package ceph-base-2:17.2.7-717.g0192297e.el8.x86_64 is already installed. 2024-03-20T08:46:33.484 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-20T08:46:33.485 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-03-20T08:46:33.485 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-20T08:46:33.518 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-2:17.2.7-717.g0192297e.el8.x86_64 28/32 2024-03-20T08:46:33.612 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install cephadm 2024-03-20T08:46:34.361 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:38 ago on Wed 20 Mar 2024 08:44:56 AM UTC. 2024-03-20T08:46:34.555 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-20T08:46:34.556 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:46:34.556 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2024-03-20T08:46:34.556 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:46:34.556 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-20T08:46:34.556 INFO:teuthology.orchestra.run.smithi115.stdout: cephadm noarch 2:17.2.7-717.g0192297e.el8 ceph-noarch 82 k 2024-03-20T08:46:34.556 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:46:34.556 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-20T08:46:34.556 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:46:34.556 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-03-20T08:46:34.557 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:46:34.557 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 82 k 2024-03-20T08:46:34.557 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 372 k 2024-03-20T08:46:34.557 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-20T08:46:35.095 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-2:17.2.7-717.g0192297e.el8.x86_64 28/32 2024-03-20T08:46:36.001 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-20T08:46:36.001 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:46:36.001 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-03-20T08:46:36.002 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:46:36.002 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-20T08:46:36.002 INFO:teuthology.orchestra.run.smithi122.stdout: cephadm noarch 2:17.2.7-717.g0192297e.el8 ceph-noarch 82 k 2024-03-20T08:46:36.002 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:46:36.002 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-20T08:46:36.002 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:46:36.002 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-20T08:46:36.002 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:46:36.003 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 82 k 2024-03-20T08:46:36.003 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 372 k 2024-03-20T08:46:36.003 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-20T08:46:36.340 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-osd-2:17.2.7-717.g0192297e.el8.x86_64 29/32 2024-03-20T08:46:36.655 INFO:teuthology.orchestra.run.smithi115.stdout:cephadm-17.2.7-717.g0192297e.el8.noarch.rpm 39 kB/s | 82 kB 00:02 2024-03-20T08:46:36.656 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:46:36.656 INFO:teuthology.orchestra.run.smithi115.stdout:Total 39 kB/s | 82 kB 00:02 2024-03-20T08:46:36.657 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-20T08:46:36.666 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-20T08:46:36.666 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-20T08:46:36.668 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-20T08:46:36.671 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-20T08:46:36.720 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-20T08:46:36.763 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-osd-2:17.2.7-717.g0192297e.el8.x86_64 29/32 2024-03-20T08:46:36.931 INFO:teuthology.orchestra.run.smithi122.stdout:cephadm-17.2.7-717.g0192297e.el8.noarch.rpm 88 kB/s | 82 kB 00:00 2024-03-20T08:46:36.932 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:46:36.932 INFO:teuthology.orchestra.run.smithi122.stdout:Total 88 kB/s | 82 kB 00:00 2024-03-20T08:46:36.933 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-20T08:46:36.941 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-20T08:46:36.941 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-20T08:46:36.943 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-20T08:46:36.947 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-20T08:46:36.988 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mon-2:17.2.7-717.g0192297e.el8.x86_64 30/32 2024-03-20T08:46:37.012 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-20T08:46:37.218 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mon-2:17.2.7-717.g0192297e.el8.x86_64 30/32 2024-03-20T08:46:37.253 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: cephadm-2:17.2.7-717.g0192297e.el8.noarch 1/1 2024-03-20T08:46:37.405 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mds-2:17.2.7-717.g0192297e.el8.x86_64 31/32 2024-03-20T08:46:37.423 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mds-2:17.2.7-717.g0192297e.el8.x86_64 31/32 2024-03-20T08:46:37.452 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : cephadm-2:17.2.7-717.g0192297e.el8.noarch 1/1 2024-03-20T08:46:37.509 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: cephadm-2:17.2.7-717.g0192297e.el8.noarch 1/1 2024-03-20T08:46:37.604 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-2:17.2.7-717.g0192297e.el8.x86_64 32/32 2024-03-20T08:46:37.654 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : cephadm-2:17.2.7-717.g0192297e.el8.noarch 1/1 2024-03-20T08:46:37.679 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: cephadm-2:17.2.7-717.g0192297e.el8.noarch 1/1 2024-03-20T08:46:37.887 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: cephadm-2:17.2.7-717.g0192297e.el8.noarch 1/1 2024-03-20T08:46:38.036 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-2:17.2.7-717.g0192297e.el8.x86_64 32/32 2024-03-20T08:46:38.036 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-2:17.2.7-717.g0192297e.el8.x86_64 1/32 2024-03-20T08:46:38.036 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mds-2:17.2.7-717.g0192297e.el8.x86_64 2/32 2024-03-20T08:46:38.036 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-2:17.2.7-717.g0192297e.el8.x86_64 3/32 2024-03-20T08:46:38.036 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mon-2:17.2.7-717.g0192297e.el8.x86_64 4/32 2024-03-20T08:46:38.036 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-osd-2:17.2.7-717.g0192297e.el8.x86_64 5/32 2024-03-20T08:46:38.036 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-717.g0192297e.el8.n 6/32 2024-03-20T08:46:38.036 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-03-20T08:46:38.036 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-03-20T08:46:38.037 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-03-20T08:46:38.037 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-03-20T08:46:38.037 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-03-20T08:46:38.037 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-03-20T08:46:38.037 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-03-20T08:46:38.037 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-03-20T08:46:38.037 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-03-20T08:46:38.037 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-03-20T08:46:38.037 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-03-20T08:46:38.037 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-03-20T08:46:38.037 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-03-20T08:46:38.037 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-03-20T08:46:38.037 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-03-20T08:46:38.037 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-03-20T08:46:38.038 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-03-20T08:46:38.038 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-03-20T08:46:38.038 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-03-20T08:46:38.038 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-03-20T08:46:38.038 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-03-20T08:46:38.038 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-03-20T08:46:38.038 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-03-20T08:46:38.038 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-03-20T08:46:38.038 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-03-20T08:46:38.047 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : cephadm-2:17.2.7-717.g0192297e.el8.noarch 1/1 2024-03-20T08:46:38.047 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:46:38.047 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-20T08:46:38.047 INFO:teuthology.orchestra.run.smithi115.stdout: cephadm-2:17.2.7-717.g0192297e.el8.noarch 2024-03-20T08:46:38.047 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:46:38.048 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-20T08:46:38.255 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-immutable-object-cache 2024-03-20T08:46:38.327 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : cephadm-2:17.2.7-717.g0192297e.el8.noarch 1/1 2024-03-20T08:46:38.327 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:46:38.327 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-20T08:46:38.327 INFO:teuthology.orchestra.run.smithi122.stdout: cephadm-2:17.2.7-717.g0192297e.el8.noarch 2024-03-20T08:46:38.327 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:46:38.328 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-20T08:46:38.573 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-immutable-object-cache 2024-03-20T08:46:38.998 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-03-20T08:46:38.998 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:46:38.998 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-20T08:46:38.998 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:38.998 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mds-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:38.998 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:38.998 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-modules-core-2:17.2.7-717.g0192297e.el8.noarch 2024-03-20T08:46:38.998 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mon-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:38.998 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-osd-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:38.998 INFO:teuthology.orchestra.run.smithi026.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-03-20T08:46:38.999 INFO:teuthology.orchestra.run.smithi026.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-03-20T08:46:38.999 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-03-20T08:46:38.999 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-03-20T08:46:38.999 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-03-20T08:46:38.999 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-03-20T08:46:38.999 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-6.2-6.el8.noarch 2024-03-20T08:46:38.999 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-03-20T08:46:38.999 INFO:teuthology.orchestra.run.smithi026.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-03-20T08:46:38.999 INFO:teuthology.orchestra.run.smithi026.stdout: python3-mako-1.0.6-14.el8.noarch 2024-03-20T08:46:38.999 INFO:teuthology.orchestra.run.smithi026.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-03-20T08:46:38.999 INFO:teuthology.orchestra.run.smithi026.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-03-20T08:46:38.999 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-03-20T08:46:38.999 INFO:teuthology.orchestra.run.smithi026.stdout: python3-portend-2.6-1.el8.noarch 2024-03-20T08:46:38.999 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-03-20T08:46:39.000 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pycparser-2.14-14.el8.noarch 2024-03-20T08:46:39.000 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pytz-2017.2-11.el8.noarch 2024-03-20T08:46:39.000 INFO:teuthology.orchestra.run.smithi026.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-03-20T08:46:39.000 INFO:teuthology.orchestra.run.smithi026.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-03-20T08:46:39.000 INFO:teuthology.orchestra.run.smithi026.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-03-20T08:46:39.000 INFO:teuthology.orchestra.run.smithi026.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-03-20T08:46:39.000 INFO:teuthology.orchestra.run.smithi026.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-03-20T08:46:39.000 INFO:teuthology.orchestra.run.smithi026.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-03-20T08:46:39.000 INFO:teuthology.orchestra.run.smithi026.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-03-20T08:46:39.000 INFO:teuthology.orchestra.run.smithi026.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-03-20T08:46:39.000 INFO:teuthology.orchestra.run.smithi026.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-03-20T08:46:39.000 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:46:39.000 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-20T08:46:39.009 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:45 ago on Wed 20 Mar 2024 08:44:54 AM UTC. 2024-03-20T08:46:39.309 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-base 2024-03-20T08:46:39.326 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:43 ago on Wed 20 Mar 2024 08:44:56 AM UTC. 2024-03-20T08:46:40.089 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:21 ago on Wed 20 Mar 2024 08:45:19 AM UTC. 2024-03-20T08:46:40.623 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-20T08:46:40.623 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:46:40.624 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2024-03-20T08:46:40.624 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:46:40.624 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-20T08:46:40.624 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-717.g0192297e.el8 ceph 162 k 2024-03-20T08:46:40.624 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:46:40.624 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-20T08:46:40.624 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:46:40.624 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-03-20T08:46:40.624 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:46:40.625 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 162 k 2024-03-20T08:46:40.625 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 494 k 2024-03-20T08:46:40.625 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-20T08:46:40.810 INFO:teuthology.orchestra.run.smithi115.stdout:ceph-immutable-object-cache-17.2.7-717.g0192297 874 kB/s | 162 kB 00:00 2024-03-20T08:46:40.811 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:46:40.811 INFO:teuthology.orchestra.run.smithi115.stdout:Total 868 kB/s | 162 kB 00:00 2024-03-20T08:46:40.811 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-20T08:46:40.822 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-20T08:46:40.822 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-20T08:46:40.869 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-20T08:46:40.871 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-20T08:46:40.933 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-20T08:46:40.934 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:46:40.934 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-03-20T08:46:40.934 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:46:40.934 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-20T08:46:40.934 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-717.g0192297e.el8 ceph 162 k 2024-03-20T08:46:40.934 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:46:40.934 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-20T08:46:40.934 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:46:40.935 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-20T08:46:40.935 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:46:40.935 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 162 k 2024-03-20T08:46:40.935 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 494 k 2024-03-20T08:46:40.935 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-20T08:46:40.982 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-20T08:46:41.213 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-717.g0192297e.e 1/1 2024-03-20T08:46:41.628 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-717.g0192297e.e 1/1 2024-03-20T08:46:41.786 INFO:teuthology.orchestra.run.smithi026.stdout:Package ceph-base-2:17.2.7-717.g0192297e.el8.x86_64 is already installed. 2024-03-20T08:46:41.791 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-immutable-object-cache-17.2.7-717.g0192297 189 kB/s | 162 kB 00:00 2024-03-20T08:46:41.792 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:46:41.792 INFO:teuthology.orchestra.run.smithi122.stdout:Total 189 kB/s | 162 kB 00:00 2024-03-20T08:46:41.793 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-20T08:46:41.805 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-20T08:46:41.805 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-20T08:46:41.838 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-20T08:46:41.839 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-03-20T08:46:41.839 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-20T08:46:41.856 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-20T08:46:41.859 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-20T08:46:41.961 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install cephadm 2024-03-20T08:46:41.976 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-20T08:46:42.174 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-717.g0192297e.e 1/1 2024-03-20T08:46:42.227 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-717.g0192297e.e 1/1 2024-03-20T08:46:42.228 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:46:42.228 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-20T08:46:42.228 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-immutable-object-cache-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:42.228 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:46:42.228 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-20T08:46:42.408 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-mgr 2024-03-20T08:46:42.605 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-717.g0192297e.e 1/1 2024-03-20T08:46:42.740 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:23 ago on Wed 20 Mar 2024 08:45:19 AM UTC. 2024-03-20T08:46:42.950 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-717.g0192297e.e 1/1 2024-03-20T08:46:42.950 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:46:42.950 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-20T08:46:42.950 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-immutable-object-cache-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:42.950 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:46:42.950 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-20T08:46:43.169 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:49 ago on Wed 20 Mar 2024 08:44:54 AM UTC. 2024-03-20T08:46:43.228 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr 2024-03-20T08:46:43.989 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:47 ago on Wed 20 Mar 2024 08:44:56 AM UTC. 2024-03-20T08:46:44.470 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-20T08:46:44.471 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:46:44.471 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-03-20T08:46:44.471 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:46:44.471 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-20T08:46:44.471 INFO:teuthology.orchestra.run.smithi026.stdout: cephadm noarch 2:17.2.7-717.g0192297e.el8 ceph-noarch 82 k 2024-03-20T08:46:44.472 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:46:44.472 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-20T08:46:44.472 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:46:44.472 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-20T08:46:44.472 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:46:44.472 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 82 k 2024-03-20T08:46:44.472 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 372 k 2024-03-20T08:46:44.472 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-20T08:46:44.718 INFO:teuthology.orchestra.run.smithi115.stdout:Package ceph-mgr-2:17.2.7-717.g0192297e.el8.x86_64 is already installed. 2024-03-20T08:46:44.766 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-20T08:46:44.767 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-03-20T08:46:44.767 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-20T08:46:44.867 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-fuse 2024-03-20T08:46:45.178 INFO:teuthology.orchestra.run.smithi026.stdout:cephadm-17.2.7-717.g0192297e.el8.noarch.rpm 116 kB/s | 82 kB 00:00 2024-03-20T08:46:45.178 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:46:45.179 INFO:teuthology.orchestra.run.smithi026.stdout:Total 116 kB/s | 82 kB 00:00 2024-03-20T08:46:45.179 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-20T08:46:45.188 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-20T08:46:45.188 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-20T08:46:45.191 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-20T08:46:45.195 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-20T08:46:45.236 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-20T08:46:45.556 INFO:teuthology.orchestra.run.smithi122.stdout:Package ceph-mgr-2:17.2.7-717.g0192297e.el8.x86_64 is already installed. 2024-03-20T08:46:45.605 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-20T08:46:45.606 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-03-20T08:46:45.606 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-20T08:46:45.619 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:51 ago on Wed 20 Mar 2024 08:44:54 AM UTC. 2024-03-20T08:46:45.729 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-fuse 2024-03-20T08:46:45.929 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: cephadm-2:17.2.7-717.g0192297e.el8.noarch 1/1 2024-03-20T08:46:46.130 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : cephadm-2:17.2.7-717.g0192297e.el8.noarch 1/1 2024-03-20T08:46:46.381 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: cephadm-2:17.2.7-717.g0192297e.el8.noarch 1/1 2024-03-20T08:46:46.478 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:50 ago on Wed 20 Mar 2024 08:44:56 AM UTC. 2024-03-20T08:46:47.233 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-20T08:46:47.234 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:46:47.234 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2024-03-20T08:46:47.234 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:46:47.234 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-20T08:46:47.234 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-fuse x86_64 2:17.2.7-717.g0192297e.el8 ceph 876 k 2024-03-20T08:46:47.234 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2024-03-20T08:46:47.234 INFO:teuthology.orchestra.run.smithi115.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-03-20T08:46:47.234 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:46:47.235 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-20T08:46:47.235 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:46:47.235 INFO:teuthology.orchestra.run.smithi115.stdout:Install 2 Packages 2024-03-20T08:46:47.235 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:46:47.235 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 959 k 2024-03-20T08:46:47.235 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 3.1 M 2024-03-20T08:46:47.236 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-20T08:46:47.446 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : cephadm-2:17.2.7-717.g0192297e.el8.noarch 1/1 2024-03-20T08:46:47.446 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:46:47.446 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-20T08:46:47.447 INFO:teuthology.orchestra.run.smithi026.stdout: cephadm-2:17.2.7-717.g0192297e.el8.noarch 2024-03-20T08:46:47.447 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:46:47.447 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-20T08:46:47.888 INFO:teuthology.orchestra.run.smithi115.stdout:(1/2): fuse-2.9.7-19.el8.x86_64.rpm 216 kB/s | 83 kB 00:00 2024-03-20T08:46:48.092 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-20T08:46:48.093 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:46:48.093 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-03-20T08:46:48.093 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:46:48.093 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-20T08:46:48.093 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-fuse x86_64 2:17.2.7-717.g0192297e.el8 ceph 876 k 2024-03-20T08:46:48.093 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-03-20T08:46:48.094 INFO:teuthology.orchestra.run.smithi122.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-03-20T08:46:48.094 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:46:48.094 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-20T08:46:48.094 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:46:48.094 INFO:teuthology.orchestra.run.smithi122.stdout:Install 2 Packages 2024-03-20T08:46:48.094 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:46:48.095 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 959 k 2024-03-20T08:46:48.095 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 3.1 M 2024-03-20T08:46:48.095 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-20T08:46:48.189 INFO:teuthology.orchestra.run.smithi115.stdout:(2/2): ceph-fuse-17.2.7-717.g0192297e.el8.x86_6 1.2 MB/s | 876 kB 00:00 2024-03-20T08:46:48.189 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:46:48.190 INFO:teuthology.orchestra.run.smithi115.stdout:Total 1.0 MB/s | 959 kB 00:00 2024-03-20T08:46:48.201 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-20T08:46:48.223 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-20T08:46:48.223 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-20T08:46:48.308 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-20T08:46:48.311 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-20T08:46:48.525 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-20T08:46:48.531 INFO:teuthology.orchestra.run.smithi122.stdout:(1/2): fuse-2.9.7-19.el8.x86_64.rpm 217 kB/s | 83 kB 00:00 2024-03-20T08:46:48.813 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-03-20T08:46:49.061 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-fuse-2:17.2.7-717.g0192297e.el8.x86_64 2/2 2024-03-20T08:46:49.249 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-immutable-object-cache 2024-03-20T08:46:49.429 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-fuse-2:17.2.7-717.g0192297e.el8.x86_64 2/2 2024-03-20T08:46:49.430 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-fuse-2:17.2.7-717.g0192297e.el8.x86_64 1/2 2024-03-20T08:46:49.457 INFO:teuthology.orchestra.run.smithi122.stdout:(2/2): ceph-fuse-17.2.7-717.g0192297e.el8.x86_6 669 kB/s | 876 kB 00:01 2024-03-20T08:46:49.458 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:46:49.458 INFO:teuthology.orchestra.run.smithi122.stdout:Total 703 kB/s | 959 kB 00:01 2024-03-20T08:46:49.469 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-20T08:46:49.491 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-20T08:46:49.491 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-20T08:46:49.579 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-20T08:46:49.582 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-20T08:46:49.761 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-20T08:46:49.827 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-03-20T08:46:49.827 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:46:49.827 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-20T08:46:49.827 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-fuse-2:17.2.7-717.g0192297e.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-03-20T08:46:49.827 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:46:49.828 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-20T08:46:50.022 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install librados-devel 2024-03-20T08:46:50.042 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:31 ago on Wed 20 Mar 2024 08:45:19 AM UTC. 2024-03-20T08:46:50.056 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-03-20T08:46:50.247 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-fuse-2:17.2.7-717.g0192297e.el8.x86_64 2/2 2024-03-20T08:46:50.681 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-fuse-2:17.2.7-717.g0192297e.el8.x86_64 2/2 2024-03-20T08:46:50.681 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-fuse-2:17.2.7-717.g0192297e.el8.x86_64 1/2 2024-03-20T08:46:50.778 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:56 ago on Wed 20 Mar 2024 08:44:54 AM UTC. 2024-03-20T08:46:51.134 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-03-20T08:46:51.134 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:46:51.134 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-20T08:46:51.134 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-fuse-2:17.2.7-717.g0192297e.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-03-20T08:46:51.134 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:46:51.134 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-20T08:46:51.421 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install librados-devel 2024-03-20T08:46:51.778 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-20T08:46:51.778 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:46:51.778 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-03-20T08:46:51.778 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:46:51.778 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-20T08:46:51.778 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-717.g0192297e.el8 ceph 162 k 2024-03-20T08:46:51.778 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:46:51.779 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-20T08:46:51.779 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:46:51.779 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-20T08:46:51.779 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:46:51.779 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 162 k 2024-03-20T08:46:51.779 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 494 k 2024-03-20T08:46:51.779 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-20T08:46:52.188 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:56 ago on Wed 20 Mar 2024 08:44:56 AM UTC. 2024-03-20T08:46:52.226 INFO:teuthology.orchestra.run.smithi026.stdout:ceph-immutable-object-cache-17.2.7-717.g0192297 362 kB/s | 162 kB 00:00 2024-03-20T08:46:52.226 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:46:52.226 INFO:teuthology.orchestra.run.smithi026.stdout:Total 361 kB/s | 162 kB 00:00 2024-03-20T08:46:52.227 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-20T08:46:52.239 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-20T08:46:52.239 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-20T08:46:52.290 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-20T08:46:52.293 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-20T08:46:52.404 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-20T08:46:52.405 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:46:52.405 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2024-03-20T08:46:52.405 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:46:52.405 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-20T08:46:52.405 INFO:teuthology.orchestra.run.smithi115.stdout: librados-devel x86_64 2:17.2.7-717.g0192297e.el8 ceph 130 k 2024-03-20T08:46:52.405 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:46:52.405 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-20T08:46:52.405 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:46:52.405 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-03-20T08:46:52.406 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:46:52.406 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 130 k 2024-03-20T08:46:52.406 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 464 k 2024-03-20T08:46:52.406 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-20T08:46:52.419 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-20T08:46:52.660 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-717.g0192297e.e 1/1 2024-03-20T08:46:52.795 INFO:teuthology.orchestra.run.smithi115.stdout:librados-devel-17.2.7-717.g0192297e.el8.x86_64. 334 kB/s | 130 kB 00:00 2024-03-20T08:46:52.796 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:46:52.796 INFO:teuthology.orchestra.run.smithi115.stdout:Total 333 kB/s | 130 kB 00:00 2024-03-20T08:46:52.796 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-20T08:46:52.802 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-20T08:46:52.802 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-20T08:46:52.851 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-20T08:46:52.854 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-20T08:46:52.962 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-20T08:46:53.088 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-717.g0192297e.e 1/1 2024-03-20T08:46:53.184 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librados-devel-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:46:53.469 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librados-devel-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:46:53.495 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-717.g0192297e.e 1/1 2024-03-20T08:46:53.495 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:46:53.495 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-20T08:46:53.496 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-immutable-object-cache-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:53.496 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:46:53.496 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-20T08:46:53.760 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr 2024-03-20T08:46:53.798 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-20T08:46:53.799 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:46:53.799 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-03-20T08:46:53.799 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:46:53.799 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-20T08:46:53.799 INFO:teuthology.orchestra.run.smithi122.stdout: librados-devel x86_64 2:17.2.7-717.g0192297e.el8 ceph 130 k 2024-03-20T08:46:53.799 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:46:53.799 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-20T08:46:53.799 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:46:53.800 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-20T08:46:53.800 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:46:53.800 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 130 k 2024-03-20T08:46:53.800 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 464 k 2024-03-20T08:46:53.800 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-20T08:46:53.827 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librados-devel-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:46:53.827 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:46:53.827 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-20T08:46:53.828 INFO:teuthology.orchestra.run.smithi115.stdout: librados-devel-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:53.828 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:46:53.828 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-20T08:46:54.063 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install libcephfs2 2024-03-20T08:46:54.220 INFO:teuthology.orchestra.run.smithi122.stdout:librados-devel-17.2.7-717.g0192297e.el8.x86_64. 308 kB/s | 130 kB 00:00 2024-03-20T08:46:54.220 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:46:54.221 INFO:teuthology.orchestra.run.smithi122.stdout:Total 307 kB/s | 130 kB 00:00 2024-03-20T08:46:54.221 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-20T08:46:54.226 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-20T08:46:54.226 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-20T08:46:54.274 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-20T08:46:54.277 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-20T08:46:54.379 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-20T08:46:54.551 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:35 ago on Wed 20 Mar 2024 08:45:19 AM UTC. 2024-03-20T08:46:54.595 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librados-devel-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:46:54.824 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:00 ago on Wed 20 Mar 2024 08:44:54 AM UTC. 2024-03-20T08:46:54.884 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados-devel-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:46:55.294 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados-devel-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:46:55.295 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:46:55.295 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-20T08:46:55.295 INFO:teuthology.orchestra.run.smithi122.stdout: librados-devel-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:46:55.295 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:46:55.295 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-20T08:46:55.529 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install libcephfs2 2024-03-20T08:46:56.232 INFO:teuthology.orchestra.run.smithi026.stdout:Package ceph-mgr-2:17.2.7-717.g0192297e.el8.x86_64 is already installed. 2024-03-20T08:46:56.285 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-20T08:46:56.286 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-03-20T08:46:56.286 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-20T08:46:56.287 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:00 ago on Wed 20 Mar 2024 08:44:56 AM UTC. 2024-03-20T08:46:56.384 INFO:teuthology.orchestra.run.smithi115.stdout:Package libcephfs2-2:17.2.7-717.g0192297e.el8.x86_64 is already installed. 2024-03-20T08:46:56.406 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-fuse 2024-03-20T08:46:56.432 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-20T08:46:56.433 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-03-20T08:46:56.433 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-20T08:46:56.537 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install libcephfs-devel 2024-03-20T08:46:57.197 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:38 ago on Wed 20 Mar 2024 08:45:19 AM UTC. 2024-03-20T08:46:57.282 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:03 ago on Wed 20 Mar 2024 08:44:54 AM UTC. 2024-03-20T08:46:57.870 INFO:teuthology.orchestra.run.smithi122.stdout:Package libcephfs2-2:17.2.7-717.g0192297e.el8.x86_64 is already installed. 2024-03-20T08:46:57.918 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-20T08:46:57.919 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-03-20T08:46:57.919 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-20T08:46:58.042 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install libcephfs-devel 2024-03-20T08:46:58.785 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:02 ago on Wed 20 Mar 2024 08:44:56 AM UTC. 2024-03-20T08:46:58.903 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-20T08:46:58.904 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:46:58.904 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2024-03-20T08:46:58.904 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:46:58.904 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-20T08:46:58.904 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs-devel x86_64 2:17.2.7-717.g0192297e.el8 ceph 25 k 2024-03-20T08:46:58.904 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:46:58.904 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-20T08:46:58.905 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:46:58.905 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-03-20T08:46:58.905 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:46:58.905 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 25 k 2024-03-20T08:46:58.905 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 109 k 2024-03-20T08:46:58.905 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-20T08:46:58.932 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-20T08:46:58.933 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:46:58.933 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-03-20T08:46:58.933 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:46:58.933 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-20T08:46:58.933 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-fuse x86_64 2:17.2.7-717.g0192297e.el8 ceph 876 k 2024-03-20T08:46:58.933 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-03-20T08:46:58.933 INFO:teuthology.orchestra.run.smithi026.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-03-20T08:46:58.933 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:46:58.933 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-20T08:46:58.933 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:46:58.934 INFO:teuthology.orchestra.run.smithi026.stdout:Install 2 Packages 2024-03-20T08:46:58.934 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:46:58.934 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 959 k 2024-03-20T08:46:58.934 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 3.1 M 2024-03-20T08:46:58.934 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-20T08:46:59.257 INFO:teuthology.orchestra.run.smithi115.stdout:libcephfs-devel-17.2.7-717.g0192297e.el8.x86_64 70 kB/s | 25 kB 00:00 2024-03-20T08:46:59.258 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:46:59.258 INFO:teuthology.orchestra.run.smithi115.stdout:Total 70 kB/s | 25 kB 00:00 2024-03-20T08:46:59.258 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-20T08:46:59.262 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-20T08:46:59.262 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-20T08:46:59.264 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-20T08:46:59.268 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-20T08:46:59.355 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-20T08:46:59.447 INFO:teuthology.orchestra.run.smithi026.stdout:(1/2): fuse-2.9.7-19.el8.x86_64.rpm 181 kB/s | 83 kB 00:00 2024-03-20T08:46:59.549 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libcephfs-devel-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:46:59.672 INFO:teuthology.orchestra.run.smithi026.stdout:(2/2): ceph-fuse-17.2.7-717.g0192297e.el8.x86_6 1.2 MB/s | 876 kB 00:00 2024-03-20T08:46:59.673 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:46:59.673 INFO:teuthology.orchestra.run.smithi026.stdout:Total 1.3 MB/s | 959 kB 00:00 2024-03-20T08:46:59.685 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-20T08:46:59.709 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-20T08:46:59.755 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-20T08:46:59.800 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:46:59.808 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-20T08:46:59.812 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-20T08:47:00.002 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-20T08:47:00.138 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libcephfs-devel-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:47:00.138 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:47:00.138 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-20T08:47:00.138 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs-devel-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:47:00.138 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:47:00.138 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-20T08:47:00.259 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install librados2 2024-03-20T08:47:00.265 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-03-20T08:47:00.397 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-20T08:47:00.397 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:47:00.397 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-03-20T08:47:00.397 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:47:00.397 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-20T08:47:00.397 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs-devel x86_64 2:17.2.7-717.g0192297e.el8 ceph 25 k 2024-03-20T08:47:00.397 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:47:00.397 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-20T08:47:00.398 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:47:00.398 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-20T08:47:00.398 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:47:00.398 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 25 k 2024-03-20T08:47:00.398 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 109 k 2024-03-20T08:47:00.398 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-20T08:47:00.542 INFO:teuthology.orchestra.run.smithi122.stdout:libcephfs-devel-17.2.7-717.g0192297e.el8.x86_64 172 kB/s | 25 kB 00:00 2024-03-20T08:47:00.543 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:47:00.543 INFO:teuthology.orchestra.run.smithi122.stdout:Total 171 kB/s | 25 kB 00:00 2024-03-20T08:47:00.543 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-20T08:47:00.550 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-20T08:47:00.550 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-20T08:47:00.550 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-20T08:47:00.552 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-20T08:47:00.599 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-20T08:47:00.617 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-fuse-2:17.2.7-717.g0192297e.el8.x86_64 2/2 2024-03-20T08:47:00.819 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephfs-devel-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:47:00.999 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:06 ago on Wed 20 Mar 2024 08:44:54 AM UTC. 2024-03-20T08:47:01.016 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-fuse-2:17.2.7-717.g0192297e.el8.x86_64 2/2 2024-03-20T08:47:01.016 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-fuse-2:17.2.7-717.g0192297e.el8.x86_64 1/2 2024-03-20T08:47:01.080 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:47:01.430 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephfs-devel-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:47:01.431 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:47:01.431 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-20T08:47:01.431 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs-devel-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:47:01.431 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:47:01.431 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-20T08:47:01.443 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-03-20T08:47:01.443 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:47:01.443 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-20T08:47:01.443 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-fuse-2:17.2.7-717.g0192297e.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-03-20T08:47:01.443 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:47:01.444 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-20T08:47:01.550 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install librados2 2024-03-20T08:47:01.710 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install librados-devel 2024-03-20T08:47:02.293 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:06 ago on Wed 20 Mar 2024 08:44:56 AM UTC. 2024-03-20T08:47:02.493 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:43 ago on Wed 20 Mar 2024 08:45:19 AM UTC. 2024-03-20T08:47:02.574 INFO:teuthology.orchestra.run.smithi115.stdout:Package librados2-2:17.2.7-717.g0192297e.el8.x86_64 is already installed. 2024-03-20T08:47:02.622 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-20T08:47:02.623 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-03-20T08:47:02.624 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-20T08:47:02.745 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install librbd1 2024-03-20T08:47:03.485 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:09 ago on Wed 20 Mar 2024 08:44:54 AM UTC. 2024-03-20T08:47:03.847 INFO:teuthology.orchestra.run.smithi122.stdout:Package librados2-2:17.2.7-717.g0192297e.el8.x86_64 is already installed. 2024-03-20T08:47:03.895 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-20T08:47:03.896 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-03-20T08:47:03.896 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-20T08:47:03.994 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install librbd1 2024-03-20T08:47:04.219 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-20T08:47:04.220 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:47:04.220 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-03-20T08:47:04.220 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:47:04.220 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-20T08:47:04.221 INFO:teuthology.orchestra.run.smithi026.stdout: librados-devel x86_64 2:17.2.7-717.g0192297e.el8 ceph 130 k 2024-03-20T08:47:04.221 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:47:04.221 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-20T08:47:04.221 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:47:04.221 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-20T08:47:04.221 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:47:04.221 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 130 k 2024-03-20T08:47:04.221 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 464 k 2024-03-20T08:47:04.221 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-20T08:47:04.738 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:08 ago on Wed 20 Mar 2024 08:44:56 AM UTC. 2024-03-20T08:47:05.030 INFO:teuthology.orchestra.run.smithi115.stdout:Package librbd1-2:17.2.7-717.g0192297e.el8.x86_64 is already installed. 2024-03-20T08:47:05.080 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-20T08:47:05.081 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-03-20T08:47:05.081 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-20T08:47:05.177 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install python3-rados 2024-03-20T08:47:05.762 INFO:teuthology.orchestra.run.smithi026.stdout:librados-devel-17.2.7-717.g0192297e.el8.x86_64. 84 kB/s | 130 kB 00:01 2024-03-20T08:47:05.762 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:47:05.763 INFO:teuthology.orchestra.run.smithi026.stdout:Total 84 kB/s | 130 kB 00:01 2024-03-20T08:47:05.763 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-20T08:47:05.770 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-20T08:47:05.770 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-20T08:47:05.827 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-20T08:47:05.830 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-20T08:47:05.922 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:11 ago on Wed 20 Mar 2024 08:44:54 AM UTC. 2024-03-20T08:47:05.943 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-20T08:47:06.174 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librados-devel-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:47:06.296 INFO:teuthology.orchestra.run.smithi122.stdout:Package librbd1-2:17.2.7-717.g0192297e.el8.x86_64 is already installed. 2024-03-20T08:47:06.346 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-20T08:47:06.347 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-03-20T08:47:06.347 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-20T08:47:06.479 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-rados 2024-03-20T08:47:06.481 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librados-devel-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:47:06.889 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librados-devel-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:47:06.889 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:47:06.889 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-20T08:47:06.889 INFO:teuthology.orchestra.run.smithi026.stdout: librados-devel-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:47:06.889 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:47:06.890 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-20T08:47:07.174 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install libcephfs2 2024-03-20T08:47:07.222 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:11 ago on Wed 20 Mar 2024 08:44:56 AM UTC. 2024-03-20T08:47:07.470 INFO:teuthology.orchestra.run.smithi115.stdout:Package python3-rados-2:17.2.7-717.g0192297e.el8.x86_64 is already installed. 2024-03-20T08:47:07.519 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-20T08:47:07.520 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-03-20T08:47:07.520 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-20T08:47:07.617 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install python3-rgw 2024-03-20T08:47:07.958 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:48 ago on Wed 20 Mar 2024 08:45:19 AM UTC. 2024-03-20T08:47:08.359 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:14 ago on Wed 20 Mar 2024 08:44:54 AM UTC. 2024-03-20T08:47:08.773 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-rados-2:17.2.7-717.g0192297e.el8.x86_64 is already installed. 2024-03-20T08:47:08.821 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-20T08:47:08.822 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-03-20T08:47:08.822 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-20T08:47:08.939 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-rgw 2024-03-20T08:47:09.644 INFO:teuthology.orchestra.run.smithi026.stdout:Package libcephfs2-2:17.2.7-717.g0192297e.el8.x86_64 is already installed. 2024-03-20T08:47:09.680 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:13 ago on Wed 20 Mar 2024 08:44:56 AM UTC. 2024-03-20T08:47:09.699 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-20T08:47:09.700 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-03-20T08:47:09.700 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-20T08:47:09.820 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install libcephfs-devel 2024-03-20T08:47:09.928 INFO:teuthology.orchestra.run.smithi115.stdout:Package python3-rgw-2:17.2.7-717.g0192297e.el8.x86_64 is already installed. 2024-03-20T08:47:09.977 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-20T08:47:09.978 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-03-20T08:47:09.978 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-20T08:47:10.079 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install python3-cephfs 2024-03-20T08:47:10.609 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:51 ago on Wed 20 Mar 2024 08:45:19 AM UTC. 2024-03-20T08:47:10.841 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:16 ago on Wed 20 Mar 2024 08:44:54 AM UTC. 2024-03-20T08:47:11.240 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-rgw-2:17.2.7-717.g0192297e.el8.x86_64 is already installed. 2024-03-20T08:47:11.288 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-20T08:47:11.290 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-03-20T08:47:11.290 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-20T08:47:11.413 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-cephfs 2024-03-20T08:47:12.158 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:16 ago on Wed 20 Mar 2024 08:44:56 AM UTC. 2024-03-20T08:47:12.357 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-20T08:47:12.358 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:47:12.358 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-03-20T08:47:12.358 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:47:12.358 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-20T08:47:12.358 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs-devel x86_64 2:17.2.7-717.g0192297e.el8 ceph 25 k 2024-03-20T08:47:12.358 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:47:12.358 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-20T08:47:12.358 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:47:12.359 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-20T08:47:12.359 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:47:12.359 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 25 k 2024-03-20T08:47:12.359 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 109 k 2024-03-20T08:47:12.359 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-20T08:47:12.402 INFO:teuthology.orchestra.run.smithi115.stdout:Package python3-cephfs-2:17.2.7-717.g0192297e.el8.x86_64 is already installed. 2024-03-20T08:47:12.453 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-20T08:47:12.454 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-03-20T08:47:12.454 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-20T08:47:12.552 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install python3-rbd 2024-03-20T08:47:12.616 INFO:teuthology.orchestra.run.smithi026.stdout:libcephfs-devel-17.2.7-717.g0192297e.el8.x86_64 97 kB/s | 25 kB 00:00 2024-03-20T08:47:12.616 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:47:12.617 INFO:teuthology.orchestra.run.smithi026.stdout:Total 96 kB/s | 25 kB 00:00 2024-03-20T08:47:12.617 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-20T08:47:12.620 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-20T08:47:12.620 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-20T08:47:12.622 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-20T08:47:12.625 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-20T08:47:12.714 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-20T08:47:12.881 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libcephfs-devel-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:47:13.148 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:47:13.295 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:19 ago on Wed 20 Mar 2024 08:44:54 AM UTC. 2024-03-20T08:47:13.521 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libcephfs-devel-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:47:13.521 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:47:13.521 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-20T08:47:13.521 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs-devel-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:47:13.522 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:47:13.522 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-20T08:47:13.664 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install librados2 2024-03-20T08:47:13.718 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-cephfs-2:17.2.7-717.g0192297e.el8.x86_64 is already installed. 2024-03-20T08:47:13.766 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-20T08:47:13.767 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-03-20T08:47:13.768 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-20T08:47:13.870 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-rbd 2024-03-20T08:47:14.443 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:55 ago on Wed 20 Mar 2024 08:45:19 AM UTC. 2024-03-20T08:47:14.614 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:18 ago on Wed 20 Mar 2024 08:44:56 AM UTC. 2024-03-20T08:47:14.859 INFO:teuthology.orchestra.run.smithi115.stdout:Package python3-rbd-2:17.2.7-717.g0192297e.el8.x86_64 is already installed. 2024-03-20T08:47:14.903 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-20T08:47:14.904 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2024-03-20T08:47:14.904 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-20T08:47:15.037 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install rbd-fuse 2024-03-20T08:47:15.786 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:21 ago on Wed 20 Mar 2024 08:44:54 AM UTC. 2024-03-20T08:47:16.114 INFO:teuthology.orchestra.run.smithi026.stdout:Package librados2-2:17.2.7-717.g0192297e.el8.x86_64 is already installed. 2024-03-20T08:47:16.168 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-20T08:47:16.169 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-03-20T08:47:16.169 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-20T08:47:16.179 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-rbd-2:17.2.7-717.g0192297e.el8.x86_64 is already installed. 2024-03-20T08:47:16.228 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-20T08:47:16.230 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-03-20T08:47:16.230 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-20T08:47:16.296 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install librbd1 2024-03-20T08:47:16.342 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-fuse 2024-03-20T08:47:17.077 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:58 ago on Wed 20 Mar 2024 08:45:19 AM UTC. 2024-03-20T08:47:17.082 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:21 ago on Wed 20 Mar 2024 08:44:56 AM UTC. 2024-03-20T08:47:17.408 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-20T08:47:17.409 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:47:17.410 INFO:teuthology.orchestra.run.smithi115.stdout: Package Architecture Version Repository Size 2024-03-20T08:47:17.410 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:47:17.410 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-20T08:47:17.410 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-fuse x86_64 2:17.2.7-717.g0192297e.el8 ceph 91 k 2024-03-20T08:47:17.410 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:47:17.410 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-20T08:47:17.410 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:47:17.410 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-03-20T08:47:17.410 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:47:17.410 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 91 k 2024-03-20T08:47:17.410 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 268 k 2024-03-20T08:47:17.410 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-20T08:47:17.600 INFO:teuthology.orchestra.run.smithi115.stdout:rbd-fuse-17.2.7-717.g0192297e.el8.x86_64.rpm 476 kB/s | 91 kB 00:00 2024-03-20T08:47:17.601 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:47:17.601 INFO:teuthology.orchestra.run.smithi115.stdout:Total 473 kB/s | 91 kB 00:00 2024-03-20T08:47:17.601 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-20T08:47:17.611 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-20T08:47:17.611 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-20T08:47:17.657 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-20T08:47:17.660 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-20T08:47:17.759 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-20T08:47:17.962 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : rbd-fuse-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:47:18.241 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: rbd-fuse-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:47:18.612 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : rbd-fuse-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:47:18.612 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:47:18.612 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-20T08:47:18.612 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-fuse-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:47:18.612 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:47:18.613 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-20T08:47:18.688 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-20T08:47:18.689 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:47:18.689 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2024-03-20T08:47:18.689 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:47:18.689 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-20T08:47:18.689 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-fuse x86_64 2:17.2.7-717.g0192297e.el8 ceph 91 k 2024-03-20T08:47:18.689 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:47:18.689 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-20T08:47:18.690 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:47:18.690 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-20T08:47:18.690 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:47:18.690 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 91 k 2024-03-20T08:47:18.690 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 268 k 2024-03-20T08:47:18.690 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-20T08:47:18.755 INFO:teuthology.orchestra.run.smithi026.stdout:Package librbd1-2:17.2.7-717.g0192297e.el8.x86_64 is already installed. 2024-03-20T08:47:18.808 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-20T08:47:18.809 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-03-20T08:47:18.809 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-20T08:47:18.854 INFO:teuthology.orchestra.run.smithi122.stdout:rbd-fuse-17.2.7-717.g0192297e.el8.x86_64.rpm 555 kB/s | 91 kB 00:00 2024-03-20T08:47:18.854 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:47:18.855 INFO:teuthology.orchestra.run.smithi122.stdout:Total 551 kB/s | 91 kB 00:00 2024-03-20T08:47:18.855 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-20T08:47:18.864 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-20T08:47:18.864 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-20T08:47:18.885 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install rbd-mirror 2024-03-20T08:47:18.910 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-20T08:47:18.913 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-20T08:47:18.929 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-rados 2024-03-20T08:47:19.011 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-20T08:47:19.217 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-fuse-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:47:19.514 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-fuse-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:47:19.638 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:25 ago on Wed 20 Mar 2024 08:44:54 AM UTC. 2024-03-20T08:47:19.710 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:02:00 ago on Wed 20 Mar 2024 08:45:19 AM UTC. 2024-03-20T08:47:19.904 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-fuse-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:47:19.904 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:47:19.905 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-20T08:47:19.905 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-fuse-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:47:19.905 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:47:19.905 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-20T08:47:20.119 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-mirror 2024-03-20T08:47:20.869 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:24 ago on Wed 20 Mar 2024 08:44:56 AM UTC. 2024-03-20T08:47:21.255 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-20T08:47:21.255 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:47:21.255 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repo Size 2024-03-20T08:47:21.255 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:47:21.256 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-20T08:47:21.256 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-mirror x86_64 2:17.2.7-717.g0192297e.el8 ceph 3.5 M 2024-03-20T08:47:21.256 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:47:21.256 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-20T08:47:21.256 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:47:21.256 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-03-20T08:47:21.256 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:47:21.257 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 3.5 M 2024-03-20T08:47:21.257 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 16 M 2024-03-20T08:47:21.257 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-20T08:47:21.379 INFO:teuthology.orchestra.run.smithi026.stdout:Package python3-rados-2:17.2.7-717.g0192297e.el8.x86_64 is already installed. 2024-03-20T08:47:21.431 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-20T08:47:21.432 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-03-20T08:47:21.433 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-20T08:47:21.603 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-rgw 2024-03-20T08:47:22.386 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:02:03 ago on Wed 20 Mar 2024 08:45:19 AM UTC. 2024-03-20T08:47:22.476 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-20T08:47:22.477 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:47:22.477 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-03-20T08:47:22.477 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:47:22.477 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-20T08:47:22.477 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-mirror x86_64 2:17.2.7-717.g0192297e.el8 ceph 3.5 M 2024-03-20T08:47:22.477 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:47:22.477 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-20T08:47:22.477 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:47:22.478 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-20T08:47:22.478 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:47:22.478 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 3.5 M 2024-03-20T08:47:22.478 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 16 M 2024-03-20T08:47:22.478 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-20T08:47:23.416 INFO:teuthology.orchestra.run.smithi122.stdout:rbd-mirror-17.2.7-717.g0192297e.el8.x86_64.rpm 3.7 MB/s | 3.5 MB 00:00 2024-03-20T08:47:23.417 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:47:23.417 INFO:teuthology.orchestra.run.smithi122.stdout:Total 3.7 MB/s | 3.5 MB 00:00 2024-03-20T08:47:23.417 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-20T08:47:23.432 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-20T08:47:23.432 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-20T08:47:23.492 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-20T08:47:23.495 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-20T08:47:23.644 INFO:teuthology.orchestra.run.smithi115.stdout:rbd-mirror-17.2.7-717.g0192297e.el8.x86_64.rpm 1.4 MB/s | 3.5 MB 00:02 2024-03-20T08:47:23.645 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:47:23.646 INFO:teuthology.orchestra.run.smithi115.stdout:Total 1.4 MB/s | 3.5 MB 00:02 2024-03-20T08:47:23.646 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-20T08:47:23.660 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-20T08:47:23.660 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-20T08:47:23.724 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-20T08:47:23.727 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-20T08:47:23.943 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-20T08:47:24.061 INFO:teuthology.orchestra.run.smithi026.stdout:Package python3-rgw-2:17.2.7-717.g0192297e.el8.x86_64 is already installed. 2024-03-20T08:47:24.114 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-20T08:47:24.115 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-03-20T08:47:24.115 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-20T08:47:24.127 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-mirror-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:47:24.163 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-20T08:47:24.232 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-cephfs 2024-03-20T08:47:24.354 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : rbd-mirror-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:47:24.529 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-mirror-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:47:24.796 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: rbd-mirror-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:47:24.915 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-mirror-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:47:24.915 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:47:24.915 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-20T08:47:24.915 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-mirror-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:47:24.915 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:47:24.915 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-20T08:47:25.010 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:02:06 ago on Wed 20 Mar 2024 08:45:19 AM UTC. 2024-03-20T08:47:25.119 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-nbd 2024-03-20T08:47:25.170 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : rbd-mirror-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:47:25.170 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:47:25.170 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-20T08:47:25.170 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-mirror-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:47:25.170 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:47:25.170 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-20T08:47:25.385 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install rbd-nbd 2024-03-20T08:47:25.882 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:29 ago on Wed 20 Mar 2024 08:44:56 AM UTC. 2024-03-20T08:47:26.150 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:02:32 ago on Wed 20 Mar 2024 08:44:54 AM UTC. 2024-03-20T08:47:26.727 INFO:teuthology.orchestra.run.smithi026.stdout:Package python3-cephfs-2:17.2.7-717.g0192297e.el8.x86_64 is already installed. 2024-03-20T08:47:26.779 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-20T08:47:26.781 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-03-20T08:47:26.781 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-20T08:47:26.919 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-rbd 2024-03-20T08:47:27.490 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-20T08:47:27.491 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:47:27.491 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2024-03-20T08:47:27.491 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:47:27.491 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-20T08:47:27.491 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-nbd x86_64 2:17.2.7-717.g0192297e.el8 ceph 175 k 2024-03-20T08:47:27.491 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:47:27.492 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-20T08:47:27.492 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-20T08:47:27.492 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-20T08:47:27.492 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:47:27.492 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 175 k 2024-03-20T08:47:27.492 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 556 k 2024-03-20T08:47:27.492 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-20T08:47:27.677 INFO:teuthology.orchestra.run.smithi122.stdout:rbd-nbd-17.2.7-717.g0192297e.el8.x86_64.rpm 946 kB/s | 175 kB 00:00 2024-03-20T08:47:27.678 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:47:27.678 INFO:teuthology.orchestra.run.smithi122.stdout:Total 940 kB/s | 175 kB 00:00 2024-03-20T08:47:27.678 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-20T08:47:27.688 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-20T08:47:27.688 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-20T08:47:27.706 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:02:08 ago on Wed 20 Mar 2024 08:45:19 AM UTC. 2024-03-20T08:47:27.736 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-20T08:47:27.739 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-20T08:47:27.768 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2024-03-20T08:47:27.768 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:47:27.768 INFO:teuthology.orchestra.run.smithi115.stdout: Package Architecture Version Repository Size 2024-03-20T08:47:27.768 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:47:27.769 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2024-03-20T08:47:27.769 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-nbd x86_64 2:17.2.7-717.g0192297e.el8 ceph 175 k 2024-03-20T08:47:27.769 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:47:27.769 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2024-03-20T08:47:27.769 INFO:teuthology.orchestra.run.smithi115.stdout:================================================================================ 2024-03-20T08:47:27.769 INFO:teuthology.orchestra.run.smithi115.stdout:Install 1 Package 2024-03-20T08:47:27.769 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:47:27.770 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 175 k 2024-03-20T08:47:27.770 INFO:teuthology.orchestra.run.smithi115.stdout:Installed size: 556 k 2024-03-20T08:47:27.770 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2024-03-20T08:47:27.846 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-20T08:47:28.033 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-nbd-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:47:28.231 INFO:teuthology.orchestra.run.smithi115.stdout:rbd-nbd-17.2.7-717.g0192297e.el8.x86_64.rpm 379 kB/s | 175 kB 00:00 2024-03-20T08:47:28.232 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:47:28.232 INFO:teuthology.orchestra.run.smithi115.stdout:Total 378 kB/s | 175 kB 00:00 2024-03-20T08:47:28.232 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2024-03-20T08:47:28.243 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2024-03-20T08:47:28.243 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2024-03-20T08:47:28.290 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2024-03-20T08:47:28.293 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2024-03-20T08:47:28.303 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-nbd-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:47:28.399 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2024-03-20T08:47:28.568 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : rbd-nbd-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:47:28.689 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-nbd-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:47:28.689 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:47:28.689 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-20T08:47:28.689 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-nbd-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:47:28.689 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:47:28.690 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-20T08:47:28.847 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: rbd-nbd-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:47:28.922 DEBUG:teuthology.parallel:result is None 2024-03-20T08:47:29.263 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : rbd-nbd-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:47:29.263 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:47:29.263 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2024-03-20T08:47:29.264 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-nbd-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:47:29.264 INFO:teuthology.orchestra.run.smithi115.stdout: 2024-03-20T08:47:29.264 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2024-03-20T08:47:29.380 INFO:teuthology.orchestra.run.smithi026.stdout:Package python3-rbd-2:17.2.7-717.g0192297e.el8.x86_64 is already installed. 2024-03-20T08:47:29.435 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-20T08:47:29.437 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-03-20T08:47:29.437 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-20T08:47:29.486 DEBUG:teuthology.parallel:result is None 2024-03-20T08:47:29.555 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install rbd-fuse 2024-03-20T08:47:30.333 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:02:11 ago on Wed 20 Mar 2024 08:45:19 AM UTC. 2024-03-20T08:47:32.066 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-20T08:47:32.067 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:47:32.067 INFO:teuthology.orchestra.run.smithi026.stdout: Package Architecture Version Repository Size 2024-03-20T08:47:32.067 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:47:32.067 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-20T08:47:32.067 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-fuse x86_64 2:17.2.7-717.g0192297e.el8 ceph 91 k 2024-03-20T08:47:32.067 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:47:32.067 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-20T08:47:32.067 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:47:32.067 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-20T08:47:32.067 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:47:32.068 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 91 k 2024-03-20T08:47:32.068 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 268 k 2024-03-20T08:47:32.068 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-20T08:47:32.359 INFO:teuthology.orchestra.run.smithi026.stdout:rbd-fuse-17.2.7-717.g0192297e.el8.x86_64.rpm 312 kB/s | 91 kB 00:00 2024-03-20T08:47:32.359 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:47:32.359 INFO:teuthology.orchestra.run.smithi026.stdout:Total 310 kB/s | 91 kB 00:00 2024-03-20T08:47:32.360 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-20T08:47:32.370 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-20T08:47:32.370 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-20T08:47:32.422 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-20T08:47:32.425 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-20T08:47:32.532 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-20T08:47:32.727 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : rbd-fuse-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:47:33.017 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: rbd-fuse-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:47:33.397 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : rbd-fuse-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:47:33.397 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:47:33.397 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-20T08:47:33.398 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-fuse-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:47:33.398 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:47:33.398 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-20T08:47:33.666 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install rbd-mirror 2024-03-20T08:47:34.448 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:02:15 ago on Wed 20 Mar 2024 08:45:19 AM UTC. 2024-03-20T08:47:36.184 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-20T08:47:36.185 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:47:36.185 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-03-20T08:47:36.185 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:47:36.185 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-20T08:47:36.185 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-mirror x86_64 2:17.2.7-717.g0192297e.el8 ceph 3.5 M 2024-03-20T08:47:36.185 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:47:36.185 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-20T08:47:36.185 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:47:36.185 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-20T08:47:36.185 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:47:36.186 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 3.5 M 2024-03-20T08:47:36.186 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 16 M 2024-03-20T08:47:36.186 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-20T08:47:36.553 INFO:teuthology.orchestra.run.smithi026.stdout:rbd-mirror-17.2.7-717.g0192297e.el8.x86_64.rpm 9.4 MB/s | 3.5 MB 00:00 2024-03-20T08:47:36.554 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:47:36.554 INFO:teuthology.orchestra.run.smithi026.stdout:Total 9.4 MB/s | 3.5 MB 00:00 2024-03-20T08:47:36.555 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-20T08:47:36.567 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-20T08:47:36.568 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-20T08:47:36.632 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-20T08:47:36.635 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-20T08:47:37.124 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-20T08:47:37.320 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : rbd-mirror-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:47:37.817 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: rbd-mirror-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:47:38.183 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : rbd-mirror-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:47:38.184 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:47:38.184 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-20T08:47:38.184 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-mirror-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:47:38.184 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:47:38.184 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-20T08:47:38.468 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install rbd-nbd 2024-03-20T08:47:39.261 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:02:20 ago on Wed 20 Mar 2024 08:45:19 AM UTC. 2024-03-20T08:47:40.986 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-20T08:47:40.988 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:47:40.988 INFO:teuthology.orchestra.run.smithi026.stdout: Package Architecture Version Repository Size 2024-03-20T08:47:40.988 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:47:40.988 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-20T08:47:40.988 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-nbd x86_64 2:17.2.7-717.g0192297e.el8 ceph 175 k 2024-03-20T08:47:40.989 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:47:40.989 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-20T08:47:40.989 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-20T08:47:40.989 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-20T08:47:40.989 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:47:40.990 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 175 k 2024-03-20T08:47:40.990 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 556 k 2024-03-20T08:47:40.990 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-20T08:47:41.190 INFO:teuthology.orchestra.run.smithi026.stdout:rbd-nbd-17.2.7-717.g0192297e.el8.x86_64.rpm 866 kB/s | 175 kB 00:00 2024-03-20T08:47:41.190 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:47:41.191 INFO:teuthology.orchestra.run.smithi026.stdout:Total 861 kB/s | 175 kB 00:00 2024-03-20T08:47:41.191 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-20T08:47:41.204 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-20T08:47:41.204 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-20T08:47:41.255 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-20T08:47:41.258 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-20T08:47:41.372 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-20T08:47:41.593 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : rbd-nbd-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:47:41.896 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: rbd-nbd-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:47:42.315 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : rbd-nbd-2:17.2.7-717.g0192297e.el8.x86_64 1/1 2024-03-20T08:47:42.316 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:47:42.316 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-20T08:47:42.316 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-nbd-2:17.2.7-717.g0192297e.el8.x86_64 2024-03-20T08:47:42.316 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:47:42.316 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-20T08:47:42.569 DEBUG:teuthology.parallel:result is None 2024-03-20T08:47:42.570 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-20T08:47:42.570 INFO:teuthology.packaging:ref: None 2024-03-20T08:47:42.570 INFO:teuthology.packaging:tag: None 2024-03-20T08:47:42.570 INFO:teuthology.packaging:branch: quincy 2024-03-20T08:47:42.570 INFO:teuthology.packaging:sha1: e7e80a6ac9849f0f23432d0d9832f096730308ae 2024-03-20T08:47:42.571 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-03-20T08:47:42.825 DEBUG:teuthology.orchestra.run.smithi026:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-20T08:47:42.848 INFO:teuthology.orchestra.run.smithi026.stdout:17.2.7-717.g0192297e.el8 2024-03-20T08:47:42.849 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-717.g0192297e.el8 2024-03-20T08:47:42.849 INFO:teuthology.task.install:The correct ceph version 17.2.7-717.g0192297e is installed. 2024-03-20T08:47:42.851 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-20T08:47:42.851 INFO:teuthology.packaging:ref: None 2024-03-20T08:47:42.851 INFO:teuthology.packaging:tag: None 2024-03-20T08:47:42.851 INFO:teuthology.packaging:branch: quincy 2024-03-20T08:47:42.851 INFO:teuthology.packaging:sha1: e7e80a6ac9849f0f23432d0d9832f096730308ae 2024-03-20T08:47:42.851 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-03-20T08:47:43.216 DEBUG:teuthology.orchestra.run.smithi115:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-20T08:47:43.245 INFO:teuthology.orchestra.run.smithi115.stdout:17.2.7-717.g0192297e.el8 2024-03-20T08:47:43.245 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-717.g0192297e.el8 2024-03-20T08:47:43.246 INFO:teuthology.task.install:The correct ceph version 17.2.7-717.g0192297e is installed. 2024-03-20T08:47:43.248 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-20T08:47:43.249 INFO:teuthology.packaging:ref: None 2024-03-20T08:47:43.249 INFO:teuthology.packaging:tag: None 2024-03-20T08:47:43.249 INFO:teuthology.packaging:branch: quincy 2024-03-20T08:47:43.249 INFO:teuthology.packaging:sha1: e7e80a6ac9849f0f23432d0d9832f096730308ae 2024-03-20T08:47:43.249 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=quincy 2024-03-20T08:47:43.675 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-20T08:47:43.703 INFO:teuthology.orchestra.run.smithi122.stdout:17.2.7-717.g0192297e.el8 2024-03-20T08:47:43.704 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-717.g0192297e.el8 2024-03-20T08:47:43.704 INFO:teuthology.task.install:The correct ceph version 17.2.7-717.g0192297e is installed. 2024-03-20T08:47:43.706 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-20T08:47:43.706 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-20T08:47:43.706 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-20T08:47:43.742 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-20T08:47:43.742 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-20T08:47:43.776 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-20T08:47:43.776 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-20T08:47:43.809 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-20T08:47:43.809 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-20T08:47:43.809 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/usr/bin/daemon-helper 2024-03-20T08:47:43.842 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-20T08:47:43.913 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-20T08:47:43.914 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/usr/bin/daemon-helper 2024-03-20T08:47:43.945 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-20T08:47:44.016 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-20T08:47:44.065 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/daemon-helper 2024-03-20T08:47:44.103 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-20T08:47:44.179 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-20T08:47:44.180 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-20T08:47:44.180 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-20T08:47:44.216 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-20T08:47:44.287 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-20T08:47:44.287 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-20T08:47:44.320 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-20T08:47:44.390 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-20T08:47:44.391 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-20T08:47:44.424 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-20T08:47:44.494 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-20T08:47:44.494 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-20T08:47:44.494 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/usr/bin/stdin-killer 2024-03-20T08:47:44.527 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-20T08:47:44.601 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-20T08:47:44.601 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/usr/bin/stdin-killer 2024-03-20T08:47:44.630 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-20T08:47:44.699 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-20T08:47:44.699 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/stdin-killer 2024-03-20T08:47:44.730 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-20T08:47:44.801 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-20T08:47:44.903 INFO:tasks.cephadm:Config: {'conf': {'mon': {'auth allow insecure global id reclaim': True, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'client.2': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MON_DOWN\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'sha1': 'e7e80a6ac9849f0f23432d0d9832f096730308ae'} 2024-03-20T08:47:44.903 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae 2024-03-20T08:47:44.904 INFO:tasks.cephadm:Cluster fsid is 852fbb68-e696-11ee-95c9-87774f69a715 2024-03-20T08:47:44.904 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-20T08:47:44.905 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.26', 'mon.b': '172.21.15.115', 'mon.c': '172.21.15.122'} 2024-03-20T08:47:44.905 INFO:tasks.cephadm:First mon is mon.a on smithi026 2024-03-20T08:47:44.905 INFO:tasks.cephadm:First mgr is y 2024-03-20T08:47:44.905 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-20T08:47:44.905 DEBUG:teuthology.orchestra.run.smithi026:> sudo hostname $(hostname -s) 2024-03-20T08:47:44.939 DEBUG:teuthology.orchestra.run.smithi115:> sudo hostname $(hostname -s) 2024-03-20T08:47:44.970 DEBUG:teuthology.orchestra.run.smithi122:> sudo hostname $(hostname -s) 2024-03-20T08:47:45.003 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-20T08:47:45.004 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=e7e80a6ac9849f0f23432d0d9832f096730308ae 2024-03-20T08:47:45.244 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'e7e80a6ac9849f0f23432d0d9832f096730308ae', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/77739/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2153-ge7e80a6a', 'node_name': '172.21.2.8+braggi08', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2153.ge7e80a6a'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-03-18-1144/e7e80a6ac9849f0f23432d0d9832f096730308ae/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2024-03-18 20:40:29.324849', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-yuri5-testing-2024-03-18-1144', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-yuri5-testing-2024-03-18-1144/e7e80a6ac9849f0f23432d0d9832f096730308ae/centos/8/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-20T08:47:45.380 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-yuri5-testing-2024-03-18-1144, sha1 e7e80a6ac9849f0f23432d0d9832f096730308ae from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=e7e80a6ac9849f0f23432d0d9832f096730308ae 2024-03-20T08:47:45.382 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-yuri5-testing-2024-03-18-1144/e7e80a6ac9849f0f23432d0d9832f096730308ae/centos/8/x86_64/flavors/default/cephadm 2024-03-20T08:47:45.382 DEBUG:teuthology.orchestra.run.smithi026:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-yuri5-testing-2024-03-18-1144/e7e80a6ac9849f0f23432d0d9832f096730308ae/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-20T08:47:45.861 INFO:teuthology.orchestra.run.smithi026.stdout:-rw-rw-r--. 1 ubuntu ubuntu 2251844 Mar 20 08:47 /home/ubuntu/cephtest/cephadm 2024-03-20T08:47:45.861 DEBUG:teuthology.orchestra.run.smithi115:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-yuri5-testing-2024-03-18-1144/e7e80a6ac9849f0f23432d0d9832f096730308ae/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-20T08:47:46.367 INFO:teuthology.orchestra.run.smithi115.stdout:-rw-rw-r--. 1 ubuntu ubuntu 2251844 Mar 20 08:47 /home/ubuntu/cephtest/cephadm 2024-03-20T08:47:46.368 DEBUG:teuthology.orchestra.run.smithi122:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-yuri5-testing-2024-03-18-1144/e7e80a6ac9849f0f23432d0d9832f096730308ae/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-20T08:47:47.090 INFO:teuthology.orchestra.run.smithi122.stdout:-rw-rw-r--. 1 ubuntu ubuntu 2251844 Mar 20 08:47 /home/ubuntu/cephtest/cephadm 2024-03-20T08:47:47.090 DEBUG:teuthology.orchestra.run.smithi026:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-20T08:47:47.114 DEBUG:teuthology.orchestra.run.smithi115:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-20T08:47:47.137 DEBUG:teuthology.orchestra.run.smithi122:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-20T08:47:47.188 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae on all hosts... 2024-03-20T08:47:47.189 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae pull 2024-03-20T08:47:47.191 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae pull 2024-03-20T08:47:47.194 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae pull 2024-03-20T08:47:48.439 INFO:teuthology.orchestra.run.smithi115.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae... 2024-03-20T08:47:48.454 INFO:teuthology.orchestra.run.smithi122.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae... 2024-03-20T08:47:48.543 INFO:teuthology.orchestra.run.smithi026.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae... 2024-03-20T08:48:16.626 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T08:48:16.627 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph_version": "ceph version 19.0.0-2153-ge7e80a6a (e7e80a6ac9849f0f23432d0d9832f096730308ae) squid (dev)", 2024-03-20T08:48:16.627 INFO:teuthology.orchestra.run.smithi026.stdout: "image_id": "9ceba81fb2c8d8682cb159527017d591637b153b3aa4a390ffa8586db6dcda00", 2024-03-20T08:48:16.627 INFO:teuthology.orchestra.run.smithi026.stdout: "repo_digests": [ 2024-03-20T08:48:16.627 INFO:teuthology.orchestra.run.smithi026.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912" 2024-03-20T08:48:16.627 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:48:16.627 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T08:48:18.176 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T08:48:18.177 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_version": "ceph version 19.0.0-2153-ge7e80a6a (e7e80a6ac9849f0f23432d0d9832f096730308ae) squid (dev)", 2024-03-20T08:48:18.177 INFO:teuthology.orchestra.run.smithi122.stdout: "image_id": "9ceba81fb2c8d8682cb159527017d591637b153b3aa4a390ffa8586db6dcda00", 2024-03-20T08:48:18.177 INFO:teuthology.orchestra.run.smithi122.stdout: "repo_digests": [ 2024-03-20T08:48:18.177 INFO:teuthology.orchestra.run.smithi122.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912" 2024-03-20T08:48:18.177 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T08:48:18.177 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T08:48:21.530 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T08:48:21.530 INFO:teuthology.orchestra.run.smithi115.stdout: "ceph_version": "ceph version 19.0.0-2153-ge7e80a6a (e7e80a6ac9849f0f23432d0d9832f096730308ae) squid (dev)", 2024-03-20T08:48:21.531 INFO:teuthology.orchestra.run.smithi115.stdout: "image_id": "9ceba81fb2c8d8682cb159527017d591637b153b3aa4a390ffa8586db6dcda00", 2024-03-20T08:48:21.531 INFO:teuthology.orchestra.run.smithi115.stdout: "repo_digests": [ 2024-03-20T08:48:21.531 INFO:teuthology.orchestra.run.smithi115.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912" 2024-03-20T08:48:21.531 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:48:21.531 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T08:48:21.557 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /etc/ceph 2024-03-20T08:48:21.595 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /etc/ceph 2024-03-20T08:48:21.625 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /etc/ceph 2024-03-20T08:48:21.663 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 777 /etc/ceph 2024-03-20T08:48:21.697 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 777 /etc/ceph 2024-03-20T08:48:21.723 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 777 /etc/ceph 2024-03-20T08:48:21.755 INFO:tasks.cephadm:Writing seed config... 2024-03-20T08:48:21.756 INFO:tasks.cephadm: override: [mon] auth allow insecure global id reclaim = True 2024-03-20T08:48:21.757 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-20T08:48:21.757 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-20T08:48:21.757 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-20T08:48:21.757 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-03-20T08:48:21.757 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-03-20T08:48:21.757 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-03-20T08:48:21.757 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-03-20T08:48:21.757 INFO:tasks.cephadm: override: [mon] mon scrub interval = 300 2024-03-20T08:48:21.757 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-03-20T08:48:21.757 INFO:tasks.cephadm: override: [client.2] debug ms = 1 2024-03-20T08:48:21.757 INFO:tasks.cephadm: override: [client.2] debug objecter = 20 2024-03-20T08:48:21.757 INFO:tasks.cephadm: override: [client.2] debug rados = 20 2024-03-20T08:48:21.758 INFO:tasks.cephadm: override: [global] mon client directed command retry = 5 2024-03-20T08:48:21.758 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-03-20T08:48:21.758 INFO:tasks.cephadm: override: [global] ms inject socket failures = 2500 2024-03-20T08:48:21.758 INFO:tasks.cephadm: override: [global] ms tcp read timeout = 5 2024-03-20T08:48:21.758 INFO:tasks.cephadm: override: [global] osd_pool_default_min_size = 2 2024-03-20T08:48:21.758 INFO:tasks.cephadm: override: [global] osd_pool_default_size = 3 2024-03-20T08:48:21.758 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-20T08:48:21.758 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-20T08:48:21.758 INFO:tasks.cephadm: override: [osd] bluestore zero block detection = True 2024-03-20T08:48:21.758 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-20T08:48:21.758 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-20T08:48:21.759 INFO:tasks.cephadm: override: [osd] osd blocked scrub grace period = 3600 2024-03-20T08:48:21.759 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-03-20T08:48:21.759 INFO:tasks.cephadm: override: [osd] osd debug verify cached snaps = True 2024-03-20T08:48:21.759 INFO:tasks.cephadm: override: [osd] osd debug verify missing on start = True 2024-03-20T08:48:21.759 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-03-20T08:48:21.759 INFO:tasks.cephadm: override: [osd] osd max markdown count = 1000 2024-03-20T08:48:21.759 INFO:tasks.cephadm: override: [osd] osd mclock override recovery settings = True 2024-03-20T08:48:21.759 INFO:tasks.cephadm: override: [osd] osd mclock profile = high_recovery_ops 2024-03-20T08:48:21.759 INFO:tasks.cephadm: override: [osd] osd op queue = debug_random 2024-03-20T08:48:21.759 INFO:tasks.cephadm: override: [osd] osd op queue cut off = debug_random 2024-03-20T08:48:21.759 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-03-20T08:48:21.759 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-03-20T08:48:21.759 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-20T08:48:21.760 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-03-20T08:48:21.760 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-20T08:48:21.760 DEBUG:teuthology.orchestra.run.smithi026:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-20T08:48:21.783 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 852fbb68-e696-11ee-95c9-87774f69a715 mon client directed command retry = 5 mon election default strategy = 3 ms inject socket failures = 2500 ms tcp read timeout = 5 osd_pool_default_min_size = 2 osd_pool_default_size = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = True osd debug verify missing on start = True osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore zero block detection = True debug ms = 1 debug osd = 20 osd blocked scrub grace period = 3600 osd debug reject backfill probability = 0.3 osd max backfills = 3 osd max markdown count = 1000 osd mclock override recovery settings = True osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false auth allow insecure global id reclaim = True debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 mon scrub interval = 300 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.2] debug ms = 1 debug objecter = 20 debug rados = 20 2024-03-20T08:48:21.783 DEBUG:teuthology.orchestra.run.smithi026:mon.a> sudo journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@mon.a.service 2024-03-20T08:48:21.825 DEBUG:teuthology.orchestra.run.smithi026:mgr.y> sudo journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@mgr.y.service 2024-03-20T08:48:21.858 INFO:journalctl@ceph.mon.a.smithi026.stdout:-- Logs begin at Wed 2024-03-20 08:37:24 UTC. -- 2024-03-20T08:48:21.859 INFO:tasks.cephadm:Bootstrapping... 2024-03-20T08:48:21.859 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae -v bootstrap --fsid 852fbb68-e696-11ee-95c9-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.26 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-20T08:48:21.889 INFO:journalctl@ceph.mgr.y.smithi026.stdout:-- Logs begin at Wed 2024-03-20 08:37:24 UTC. -- 2024-03-20T08:48:22.078 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-20T08:48:22.078 INFO:teuthology.orchestra.run.smithi026.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae', '-v', 'bootstrap', '--fsid', '852fbb68-e696-11ee-95c9-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.26', '--skip-admin-label'] 2024-03-20T08:48:22.116 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-20T08:48:22.116 INFO:teuthology.orchestra.run.smithi026.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-20T08:48:22.116 INFO:teuthology.orchestra.run.smithi026.stdout:Verifying podman|docker is present... 2024-03-20T08:48:22.148 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-20T08:48:22.148 INFO:teuthology.orchestra.run.smithi026.stdout:Verifying lvm2 is present... 2024-03-20T08:48:22.148 INFO:teuthology.orchestra.run.smithi026.stdout:Verifying time synchronization is in place... 2024-03-20T08:48:22.156 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-20T08:48:22.156 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-20T08:48:22.162 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-20T08:48:22.162 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout inactive 2024-03-20T08:48:22.168 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout enabled 2024-03-20T08:48:22.174 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout active 2024-03-20T08:48:22.174 INFO:teuthology.orchestra.run.smithi026.stdout:Unit chronyd.service is enabled and running 2024-03-20T08:48:22.174 INFO:teuthology.orchestra.run.smithi026.stdout:Repeating the final host check... 2024-03-20T08:48:22.209 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-20T08:48:22.209 INFO:teuthology.orchestra.run.smithi026.stdout:podman (/bin/podman) version 4.9.4 is present 2024-03-20T08:48:22.209 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl is present 2024-03-20T08:48:22.209 INFO:teuthology.orchestra.run.smithi026.stdout:lvcreate is present 2024-03-20T08:48:22.215 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-20T08:48:22.215 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-20T08:48:22.221 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-20T08:48:22.221 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout inactive 2024-03-20T08:48:22.227 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout enabled 2024-03-20T08:48:22.233 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout active 2024-03-20T08:48:22.233 INFO:teuthology.orchestra.run.smithi026.stdout:Unit chronyd.service is enabled and running 2024-03-20T08:48:22.233 INFO:teuthology.orchestra.run.smithi026.stdout:Host looks OK 2024-03-20T08:48:22.234 INFO:teuthology.orchestra.run.smithi026.stdout:Cluster fsid: 852fbb68-e696-11ee-95c9-87774f69a715 2024-03-20T08:48:22.234 INFO:teuthology.orchestra.run.smithi026.stdout:Acquiring lock 140056795088376 on /run/cephadm/852fbb68-e696-11ee-95c9-87774f69a715.lock 2024-03-20T08:48:22.234 INFO:teuthology.orchestra.run.smithi026.stdout:Lock 140056795088376 acquired on /run/cephadm/852fbb68-e696-11ee-95c9-87774f69a715.lock 2024-03-20T08:48:22.234 INFO:teuthology.orchestra.run.smithi026.stdout:Verifying IP 172.21.15.26 port 3300 ... 2024-03-20T08:48:22.235 INFO:teuthology.orchestra.run.smithi026.stdout:Verifying IP 172.21.15.26 port 6789 ... 2024-03-20T08:48:22.235 INFO:teuthology.orchestra.run.smithi026.stdout:Base mon IP(s) is [172.21.15.26:3300, 172.21.15.26:6789], mon addrv is [v2:172.21.15.26:3300,v1:172.21.15.26:6789] 2024-03-20T08:48:22.238 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.26 metric 100 2024-03-20T08:48:22.238 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.26 metric 100 2024-03-20T08:48:22.241 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-20T08:48:22.241 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-03-20T08:48:22.241 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1791sec hoplimit 64 pref medium 2024-03-20T08:48:22.244 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-20T08:48:22.244 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-20T08:48:22.244 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-20T08:48:22.244 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-20T08:48:22.244 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fea5:cac0/64 scope link 2024-03-20T08:48:22.244 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-20T08:48:22.245 INFO:teuthology.orchestra.run.smithi026.stdout:Mon IP `172.21.15.26` is in CIDR network `172.21.0.0/20` 2024-03-20T08:48:22.245 INFO:teuthology.orchestra.run.smithi026.stdout:Mon IP `172.21.15.26` is in CIDR network `172.21.0.0/20` 2024-03-20T08:48:22.245 INFO:teuthology.orchestra.run.smithi026.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-20T08:48:22.246 INFO:teuthology.orchestra.run.smithi026.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-20T08:48:22.247 INFO:teuthology.orchestra.run.smithi026.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae... 2024-03-20T08:48:22.770 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stdout 9ceba81fb2c8d8682cb159527017d591637b153b3aa4a390ffa8586db6dcda00 2024-03-20T08:48:22.770 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae... 2024-03-20T08:48:22.770 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stderr Getting image source signatures 2024-03-20T08:48:22.770 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stderr Copying blob sha256:29f2579974f82cd19c8183e36ca34a7e5a6e57e8b4789a8e115604a69a0d38b4 2024-03-20T08:48:22.770 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-20T08:48:22.770 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stderr Copying config sha256:9ceba81fb2c8d8682cb159527017d591637b153b3aa4a390ffa8586db6dcda00 2024-03-20T08:48:22.770 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-20T08:48:23.978 INFO:teuthology.orchestra.run.smithi026.stdout:ceph: stdout ceph version 19.0.0-2153-ge7e80a6a (e7e80a6ac9849f0f23432d0d9832f096730308ae) squid (dev) 2024-03-20T08:48:23.978 INFO:teuthology.orchestra.run.smithi026.stdout:Ceph version: ceph version 19.0.0-2153-ge7e80a6a (e7e80a6ac9849f0f23432d0d9832f096730308ae) squid (dev) 2024-03-20T08:48:23.979 INFO:teuthology.orchestra.run.smithi026.stdout:Extracting ceph user uid/gid from container image... 2024-03-20T08:48:27.899 INFO:teuthology.orchestra.run.smithi026.stdout:stat: stdout 167 167 2024-03-20T08:48:27.900 INFO:teuthology.orchestra.run.smithi026.stdout:Creating initial keys... 2024-03-20T08:48:29.140 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph-authtool: stdout AQDcovplbwYGFRAAJcGTI8d8dSArkfbt2BrDhQ== 2024-03-20T08:48:30.421 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph-authtool: stdout AQDdovpliavKIxAAVB6CpihinYxGRPjZVLxUyg== 2024-03-20T08:48:31.557 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph-authtool: stdout AQDeovpl41D+NxAAurJOxy63FUriC1OxpeO1qA== 2024-03-20T08:48:31.559 INFO:teuthology.orchestra.run.smithi026.stdout:Creating initial monmap... 2024-03-20T08:48:32.815 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-20T08:48:32.815 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-20T08:48:32.816 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 852fbb68-e696-11ee-95c9-87774f69a715 2024-03-20T08:48:32.816 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-20T08:48:32.816 INFO:teuthology.orchestra.run.smithi026.stdout:monmaptool for a [v2:172.21.15.26:3300,v1:172.21.15.26:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-20T08:48:32.816 INFO:teuthology.orchestra.run.smithi026.stdout:setting min_mon_release = quincy 2024-03-20T08:48:32.816 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/monmaptool: set fsid to 852fbb68-e696-11ee-95c9-87774f69a715 2024-03-20T08:48:32.816 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-20T08:48:32.816 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:48:32.816 INFO:teuthology.orchestra.run.smithi026.stdout:Creating mon... 2024-03-20T08:48:33.923 INFO:teuthology.orchestra.run.smithi026.stdout:create mon.a on 2024-03-20T08:48:34.055 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-03-20T08:48:34.151 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-20T08:48:34.405 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-852fbb68-e696-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-852fbb68-e696-11ee-95c9-87774f69a715.target. 2024-03-20T08:48:34.406 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-852fbb68-e696-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-852fbb68-e696-11ee-95c9-87774f69a715.target. 2024-03-20T08:48:34.590 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-852fbb68-e696-11ee-95c9-87774f69a715@mon.a 2024-03-20T08:48:34.590 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Failed to reset failed state of unit ceph-852fbb68-e696-11ee-95c9-87774f69a715@mon.a.service: Unit ceph-852fbb68-e696-11ee-95c9-87774f69a715@mon.a.service not loaded. 2024-03-20T08:48:34.688 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-852fbb68-e696-11ee-95c9-87774f69a715.target.wants/ceph-852fbb68-e696-11ee-95c9-87774f69a715@mon.a.service → /etc/systemd/system/ceph-852fbb68-e696-11ee-95c9-87774f69a715@.service. 2024-03-20T08:48:35.325 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-20T08:48:35.325 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout disabled 2024-03-20T08:48:35.333 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-20T08:48:35.333 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout inactive 2024-03-20T08:48:35.334 INFO:teuthology.orchestra.run.smithi026.stdout:firewalld.service is not enabled 2024-03-20T08:48:35.334 INFO:teuthology.orchestra.run.smithi026.stdout:Not possible to enable service . firewalld.service is not available 2024-03-20T08:48:35.334 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for mon to start... 2024-03-20T08:48:35.334 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for mon... 2024-03-20T08:48:35.544 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:35 smithi026 bash[67788]: c51300a4cab31cdbd0280124bf68af8296c04453ddb04516219e35707a1695f4 2024-03-20T08:48:35.544 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:35 smithi026 systemd[1]: Started Ceph mon.a for 852fbb68-e696-11ee-95c9-87774f69a715. 2024-03-20T08:48:36.539 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout cluster: 2024-03-20T08:48:36.539 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout id: 852fbb68-e696-11ee-95c9-87774f69a715 2024-03-20T08:48:36.539 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-20T08:48:36.539 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-03-20T08:48:36.539 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout services: 2024-03-20T08:48:36.539 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.679016s) 2024-03-20T08:48:36.539 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-20T08:48:36.539 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-20T08:48:36.539 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-03-20T08:48:36.539 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout data: 2024-03-20T08:48:36.539 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-20T08:48:36.539 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-20T08:48:36.540 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-20T08:48:36.540 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout pgs: 2024-03-20T08:48:36.540 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-03-20T08:48:36.540 INFO:teuthology.orchestra.run.smithi026.stdout:mon is available 2024-03-20T08:48:36.540 INFO:teuthology.orchestra.run.smithi026.stdout:Assimilating anything we can from ceph.conf... 2024-03-20T08:48:36.540 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:36 smithi026 ceph-mon[67827]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-20T08:48:36.540 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:36 smithi026 ceph-mon[67827]: monmap e1: 1 mons at {a=[v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-20T08:48:36.540 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:36 smithi026 ceph-mon[67827]: fsmap 2024-03-20T08:48:36.540 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:36 smithi026 ceph-mon[67827]: osdmap e1: 0 total, 0 up, 0 in 2024-03-20T08:48:36.541 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:36 smithi026 ceph-mon[67827]: mgrmap e1: no daemons active 2024-03-20T08:48:36.541 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:36 smithi026 ceph-mon[67827]: from='client.? 172.21.15.26:0/1001160886' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-20T08:48:37.649 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:37 smithi026 ceph-mon[67827]: from='client.? 172.21.15.26:0/885737696' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-20T08:48:37.649 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:37 smithi026 ceph-mon[67827]: from='client.? 172.21.15.26:0/885737696' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-20T08:48:37.895 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-03-20T08:48:37.896 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout [global] 2024-03-20T08:48:37.896 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout fsid = 852fbb68-e696-11ee-95c9-87774f69a715 2024-03-20T08:48:37.897 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.26:3300,v1:172.21.15.26:6789] 2024-03-20T08:48:37.897 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-20T08:48:37.897 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-20T08:48:37.897 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-20T08:48:37.897 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ms_tcp_read_timeout = 5 2024-03-20T08:48:37.898 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-20T08:48:37.898 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-03-20T08:48:37.898 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-20T08:48:37.898 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-20T08:48:37.898 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-03-20T08:48:37.899 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout [osd] 2024-03-20T08:48:37.899 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-20T08:48:37.899 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-20T08:48:37.899 INFO:teuthology.orchestra.run.smithi026.stdout:Generating new minimal ceph.conf... 2024-03-20T08:48:39.926 INFO:teuthology.orchestra.run.smithi026.stdout:Restarting the monitor... 2024-03-20T08:48:39.928 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:39 smithi026 ceph-mon[67827]: from='client.? 172.21.15.26:0/2040724349' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:48:40.184 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:39 smithi026 systemd[1]: Stopping Ceph mon.a for 852fbb68-e696-11ee-95c9-87774f69a715... 2024-03-20T08:48:40.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:40 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mon-a[67803]: 2024-03-20T08:48:40.182+0000 7fb7cfea5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-20T08:48:40.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:40 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mon-a[67803]: 2024-03-20T08:48:40.182+0000 7fb7cfea5700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-20T08:48:40.911 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:40 smithi026 bash[68152]: ceph-852fbb68-e696-11ee-95c9-87774f69a715-mon-a 2024-03-20T08:48:41.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 systemd[1]: ceph-852fbb68-e696-11ee-95c9-87774f69a715@mon.a.service: Succeeded. 2024-03-20T08:48:41.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 systemd[1]: Stopped Ceph mon.a for 852fbb68-e696-11ee-95c9-87774f69a715. 2024-03-20T08:48:41.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 systemd[1]: Starting Ceph mon.a for 852fbb68-e696-11ee-95c9-87774f69a715... 2024-03-20T08:48:41.660 INFO:teuthology.orchestra.run.smithi026.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-20T08:48:41.862 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: set uid:gid to 167:167 (ceph:ceph) 2024-03-20T08:48:41.862 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: ceph version 19.0.0-2153-ge7e80a6a (e7e80a6ac9849f0f23432d0d9832f096730308ae) squid (dev), process ceph-mon, pid 7 2024-03-20T08:48:41.862 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: pidfile_write: ignore empty --pid-file 2024-03-20T08:48:41.863 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: load: jerasure load: lrc 2024-03-20T08:48:41.863 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: RocksDB version: 7.9.2 2024-03-20T08:48:41.863 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Git sha 0 2024-03-20T08:48:41.863 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Compile date 2024-03-18 19:18:37 2024-03-20T08:48:41.863 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: DB SUMMARY 2024-03-20T08:48:41.863 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: DB Session ID: 28MMTO05I5CC5VYCTK1I 2024-03-20T08:48:41.863 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: CURRENT file: CURRENT 2024-03-20T08:48:41.863 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: IDENTITY file: IDENTITY 2024-03-20T08:48:41.863 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-20T08:48:41.863 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-20T08:48:41.864 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 95513 ; 2024-03-20T08:48:41.864 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.error_if_exists: 0 2024-03-20T08:48:41.864 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.create_if_missing: 0 2024-03-20T08:48:41.864 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.paranoid_checks: 1 2024-03-20T08:48:41.864 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-20T08:48:41.864 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-20T08:48:41.864 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-20T08:48:41.864 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.env: 0x563de5e99b20 2024-03-20T08:48:41.864 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.fs: PosixFileSystem 2024-03-20T08:48:41.864 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.info_log: 0x563de850f920 2024-03-20T08:48:41.864 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.max_file_opening_threads: 16 2024-03-20T08:48:41.864 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.statistics: (nil) 2024-03-20T08:48:41.864 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.use_fsync: 0 2024-03-20T08:48:41.865 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.max_log_file_size: 0 2024-03-20T08:48:41.865 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-20T08:48:41.865 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-20T08:48:41.865 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.keep_log_file_num: 1000 2024-03-20T08:48:41.865 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.recycle_log_file_num: 0 2024-03-20T08:48:41.865 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.allow_fallocate: 1 2024-03-20T08:48:41.865 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.allow_mmap_reads: 0 2024-03-20T08:48:41.865 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.allow_mmap_writes: 0 2024-03-20T08:48:41.865 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.use_direct_reads: 0 2024-03-20T08:48:41.865 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-20T08:48:41.865 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.create_missing_column_families: 0 2024-03-20T08:48:41.865 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.db_log_dir: 2024-03-20T08:48:41.865 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.wal_dir: 2024-03-20T08:48:41.865 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-20T08:48:41.865 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-20T08:48:41.866 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-20T08:48:41.866 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-20T08:48:41.866 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-20T08:48:41.866 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-20T08:48:41.866 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.advise_random_on_open: 1 2024-03-20T08:48:41.866 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.db_write_buffer_size: 0 2024-03-20T08:48:41.866 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.write_buffer_manager: 0x563de84e21e0 2024-03-20T08:48:41.866 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-20T08:48:41.866 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-20T08:48:41.866 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-20T08:48:41.866 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.rate_limiter: (nil) 2024-03-20T08:48:41.867 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-20T08:48:41.867 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.wal_recovery_mode: 2 2024-03-20T08:48:41.867 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.enable_thread_tracking: 0 2024-03-20T08:48:41.867 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.enable_pipelined_write: 0 2024-03-20T08:48:41.867 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.unordered_write: 0 2024-03-20T08:48:41.867 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-20T08:48:41.867 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-20T08:48:41.867 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-20T08:48:41.867 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-20T08:48:41.867 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.row_cache: None 2024-03-20T08:48:41.867 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.wal_filter: None 2024-03-20T08:48:41.867 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-20T08:48:41.867 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.allow_ingest_behind: 0 2024-03-20T08:48:41.868 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.two_write_queues: 0 2024-03-20T08:48:41.868 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.manual_wal_flush: 0 2024-03-20T08:48:41.868 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.wal_compression: 0 2024-03-20T08:48:41.868 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.atomic_flush: 0 2024-03-20T08:48:41.868 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-20T08:48:41.868 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-20T08:48:41.868 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-20T08:48:41.868 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.log_readahead_size: 0 2024-03-20T08:48:41.868 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-20T08:48:41.868 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.best_efforts_recovery: 0 2024-03-20T08:48:41.868 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-20T08:48:41.868 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-20T08:48:41.868 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.allow_data_in_errors: 0 2024-03-20T08:48:41.868 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.db_host_id: __hostname__ 2024-03-20T08:48:41.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-20T08:48:41.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.max_background_jobs: 2 2024-03-20T08:48:41.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.max_background_compactions: -1 2024-03-20T08:48:41.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.max_subcompactions: 1 2024-03-20T08:48:41.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-20T08:48:41.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-20T08:48:41.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-20T08:48:41.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.max_total_wal_size: 0 2024-03-20T08:48:41.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-20T08:48:41.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-20T08:48:41.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-20T08:48:41.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-20T08:48:41.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.max_open_files: -1 2024-03-20T08:48:41.870 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.bytes_per_sync: 0 2024-03-20T08:48:41.870 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-20T08:48:41.870 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-20T08:48:41.870 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.compaction_readahead_size: 0 2024-03-20T08:48:41.870 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.max_background_flushes: -1 2024-03-20T08:48:41.870 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Compression algorithms supported: 2024-03-20T08:48:41.870 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-20T08:48:41.870 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: kZSTD supported: 0 2024-03-20T08:48:41.870 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: kXpressCompression supported: 0 2024-03-20T08:48:41.870 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: kLZ4HCCompression supported: 1 2024-03-20T08:48:41.870 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: kZlibCompression supported: 1 2024-03-20T08:48:41.870 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: kSnappyCompression supported: 1 2024-03-20T08:48:41.871 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: kLZ4Compression supported: 1 2024-03-20T08:48:41.871 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: kBZip2Compression supported: 0 2024-03-20T08:48:41.871 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-20T08:48:41.871 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-20T08:48:41.871 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-20T08:48:41.871 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-20T08:48:41.871 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-20T08:48:41.871 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.merge_operator: 2024-03-20T08:48:41.871 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.compaction_filter: None 2024-03-20T08:48:41.871 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.compaction_filter_factory: None 2024-03-20T08:48:41.871 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.sst_partitioner_factory: None 2024-03-20T08:48:41.871 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-20T08:48:41.871 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-20T08:48:41.871 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563de850fa20) 2024-03-20T08:48:41.872 INFO:journalctl@ceph.mon.a.smithi026.stdout: cache_index_and_filter_blocks: 1 2024-03-20T08:48:41.872 INFO:journalctl@ceph.mon.a.smithi026.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-20T08:48:41.872 INFO:journalctl@ceph.mon.a.smithi026.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-20T08:48:41.872 INFO:journalctl@ceph.mon.a.smithi026.stdout: pin_top_level_index_and_filter: 1 2024-03-20T08:48:41.872 INFO:journalctl@ceph.mon.a.smithi026.stdout: index_type: 0 2024-03-20T08:48:41.872 INFO:journalctl@ceph.mon.a.smithi026.stdout: data_block_index_type: 0 2024-03-20T08:48:41.872 INFO:journalctl@ceph.mon.a.smithi026.stdout: index_shortening: 1 2024-03-20T08:48:41.872 INFO:journalctl@ceph.mon.a.smithi026.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-20T08:48:41.872 INFO:journalctl@ceph.mon.a.smithi026.stdout: checksum: 4 2024-03-20T08:48:41.872 INFO:journalctl@ceph.mon.a.smithi026.stdout: no_block_cache: 0 2024-03-20T08:48:41.872 INFO:journalctl@ceph.mon.a.smithi026.stdout: block_cache: 0x563de8515090 2024-03-20T08:48:41.872 INFO:journalctl@ceph.mon.a.smithi026.stdout: block_cache_name: BinnedLRUCache 2024-03-20T08:48:41.872 INFO:journalctl@ceph.mon.a.smithi026.stdout: block_cache_options: 2024-03-20T08:48:41.872 INFO:journalctl@ceph.mon.a.smithi026.stdout: capacity : 536870912 2024-03-20T08:48:41.873 INFO:journalctl@ceph.mon.a.smithi026.stdout: num_shard_bits : 4 2024-03-20T08:48:41.873 INFO:journalctl@ceph.mon.a.smithi026.stdout: strict_capacity_limit : 0 2024-03-20T08:48:41.873 INFO:journalctl@ceph.mon.a.smithi026.stdout: high_pri_pool_ratio: 0.000 2024-03-20T08:48:41.873 INFO:journalctl@ceph.mon.a.smithi026.stdout: block_cache_compressed: (nil) 2024-03-20T08:48:41.873 INFO:journalctl@ceph.mon.a.smithi026.stdout: persistent_cache: (nil) 2024-03-20T08:48:41.873 INFO:journalctl@ceph.mon.a.smithi026.stdout: block_size: 4096 2024-03-20T08:48:41.873 INFO:journalctl@ceph.mon.a.smithi026.stdout: block_size_deviation: 10 2024-03-20T08:48:41.873 INFO:journalctl@ceph.mon.a.smithi026.stdout: block_restart_interval: 16 2024-03-20T08:48:41.873 INFO:journalctl@ceph.mon.a.smithi026.stdout: index_block_restart_interval: 1 2024-03-20T08:48:41.873 INFO:journalctl@ceph.mon.a.smithi026.stdout: metadata_block_size: 4096 2024-03-20T08:48:41.873 INFO:journalctl@ceph.mon.a.smithi026.stdout: partition_filters: 0 2024-03-20T08:48:41.873 INFO:journalctl@ceph.mon.a.smithi026.stdout: use_delta_encoding: 1 2024-03-20T08:48:41.873 INFO:journalctl@ceph.mon.a.smithi026.stdout: filter_policy: bloomfilter 2024-03-20T08:48:41.873 INFO:journalctl@ceph.mon.a.smithi026.stdout: whole_key_filtering: 1 2024-03-20T08:48:41.873 INFO:journalctl@ceph.mon.a.smithi026.stdout: verify_compression: 0 2024-03-20T08:48:41.874 INFO:journalctl@ceph.mon.a.smithi026.stdout: read_amp_bytes_per_bit: 0 2024-03-20T08:48:41.874 INFO:journalctl@ceph.mon.a.smithi026.stdout: format_version: 5 2024-03-20T08:48:41.874 INFO:journalctl@ceph.mon.a.smithi026.stdout: enable_index_compression: 1 2024-03-20T08:48:41.874 INFO:journalctl@ceph.mon.a.smithi026.stdout: block_align: 0 2024-03-20T08:48:41.874 INFO:journalctl@ceph.mon.a.smithi026.stdout: max_auto_readahead_size: 262144 2024-03-20T08:48:41.874 INFO:journalctl@ceph.mon.a.smithi026.stdout: prepopulate_block_cache: 0 2024-03-20T08:48:41.874 INFO:journalctl@ceph.mon.a.smithi026.stdout: initial_auto_readahead_size: 8192 2024-03-20T08:48:41.874 INFO:journalctl@ceph.mon.a.smithi026.stdout: num_file_reads_for_auto_readahead: 2 2024-03-20T08:48:41.874 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.write_buffer_size: 33554432 2024-03-20T08:48:41.874 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.max_write_buffer_number: 2 2024-03-20T08:48:41.874 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.compression: NoCompression 2024-03-20T08:48:41.875 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.bottommost_compression: Disabled 2024-03-20T08:48:41.875 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.prefix_extractor: nullptr 2024-03-20T08:48:41.875 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-20T08:48:41.875 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.num_levels: 7 2024-03-20T08:48:41.875 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-20T08:48:41.875 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-20T08:48:41.875 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-20T08:48:41.875 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-20T08:48:41.875 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-20T08:48:41.875 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-20T08:48:41.875 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-20T08:48:41.875 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-20T08:48:41.876 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-20T08:48:41.876 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-20T08:48:41.876 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-20T08:48:41.876 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-20T08:48:41.876 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-20T08:48:41.876 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.compression_opts.level: 32767 2024-03-20T08:48:41.876 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.compression_opts.strategy: 0 2024-03-20T08:48:41.876 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-20T08:48:41.876 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-20T08:48:41.876 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-20T08:48:41.876 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-20T08:48:41.876 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.compression_opts.enabled: false 2024-03-20T08:48:41.876 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-20T08:48:41.876 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-20T08:48:41.877 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-20T08:48:41.877 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-20T08:48:41.877 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.target_file_size_base: 67108864 2024-03-20T08:48:41.877 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-20T08:48:41.877 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-20T08:48:41.877 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-20T08:48:41.877 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-20T08:48:41.877 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-20T08:48:41.877 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-20T08:48:41.877 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-20T08:48:41.877 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-20T08:48:41.877 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-20T08:48:41.877 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-20T08:48:41.877 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-20T08:48:41.878 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-20T08:48:41.878 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-20T08:48:41.878 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-20T08:48:41.878 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.arena_block_size: 1048576 2024-03-20T08:48:41.878 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-20T08:48:41.878 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-20T08:48:41.878 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.disable_auto_compactions: 0 2024-03-20T08:48:41.878 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-20T08:48:41.878 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-20T08:48:41.878 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-20T08:48:41.878 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-20T08:48:41.878 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-20T08:48:41.878 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-20T08:48:41.879 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-20T08:48:41.879 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-20T08:48:41.879 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-20T08:48:41.879 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-20T08:48:41.879 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.table_properties_collectors: 2024-03-20T08:48:41.879 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.inplace_update_support: 0 2024-03-20T08:48:41.879 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-20T08:48:41.879 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-20T08:48:41.879 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-20T08:48:41.879 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-20T08:48:41.879 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.bloom_locality: 0 2024-03-20T08:48:41.879 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.max_successive_merges: 0 2024-03-20T08:48:41.879 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-20T08:48:41.879 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.paranoid_file_checks: 0 2024-03-20T08:48:41.879 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.force_consistency_checks: 1 2024-03-20T08:48:41.880 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.report_bg_io_stats: 0 2024-03-20T08:48:41.880 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.ttl: 2592000 2024-03-20T08:48:41.880 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-20T08:48:41.880 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-20T08:48:41.880 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-20T08:48:41.880 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.enable_blob_files: false 2024-03-20T08:48:41.880 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.min_blob_size: 0 2024-03-20T08:48:41.880 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.blob_file_size: 268435456 2024-03-20T08:48:41.880 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-20T08:48:41.880 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-20T08:48:41.880 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-20T08:48:41.880 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-20T08:48:41.880 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-20T08:48:41.880 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.blob_file_starting_level: 0 2024-03-20T08:48:41.881 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-20T08:48:41.881 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-20T08:48:41.881 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-20T08:48:41.881 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 6a91ec43-00ab-4c02-b120-327670bfff79 2024-03-20T08:48:41.881 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710924521612978, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-20T08:48:41.881 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-20T08:48:41.881 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710924521614434, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 90987, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 294, "table_properties": {"data_size": 89073, "index_size": 260, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 12730, "raw_average_key_size": 50, "raw_value_size": 82141, "raw_average_value_size": 324, "num_data_blocks": 11, "num_entries": 253, "num_filter_entries": 253, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710924521, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "6a91ec43-00ab-4c02-b120-327670bfff79", "db_session_id": "28MMTO05I5CC5VYCTK1I", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-20T08:48:41.881 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710924521614547, "job": 1, "event": "recovery_finished"} 2024-03-20T08:48:41.881 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-20T08:48:41.881 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-20T08:48:41.881 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x563de8608000 2024-03-20T08:48:41.881 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: DB pointer 0x563de85f2000 2024-03-20T08:48:41.881 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-20T08:48:41.881 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-20T08:48:41.882 INFO:journalctl@ceph.mon.a.smithi026.stdout: ** DB Stats ** 2024-03-20T08:48:41.882 INFO:journalctl@ceph.mon.a.smithi026.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-20T08:48:41.882 INFO:journalctl@ceph.mon.a.smithi026.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-20T08:48:41.882 INFO:journalctl@ceph.mon.a.smithi026.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-20T08:48:41.882 INFO:journalctl@ceph.mon.a.smithi026.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-20T08:48:41.882 INFO:journalctl@ceph.mon.a.smithi026.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-20T08:48:41.882 INFO:journalctl@ceph.mon.a.smithi026.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-20T08:48:41.882 INFO:journalctl@ceph.mon.a.smithi026.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-20T08:48:41.882 INFO:journalctl@ceph.mon.a.smithi026.stdout: 2024-03-20T08:48:41.882 INFO:journalctl@ceph.mon.a.smithi026.stdout: ** Compaction Stats [default] ** 2024-03-20T08:48:41.882 INFO:journalctl@ceph.mon.a.smithi026.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-20T08:48:41.882 INFO:journalctl@ceph.mon.a.smithi026.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-20T08:48:41.882 INFO:journalctl@ceph.mon.a.smithi026.stdout: L0 2/0 90.69 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 103.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-20T08:48:41.883 INFO:journalctl@ceph.mon.a.smithi026.stdout: Sum 2/0 90.69 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 103.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-20T08:48:41.883 INFO:journalctl@ceph.mon.a.smithi026.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 103.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-20T08:48:41.883 INFO:journalctl@ceph.mon.a.smithi026.stdout: 2024-03-20T08:48:41.883 INFO:journalctl@ceph.mon.a.smithi026.stdout: ** Compaction Stats [default] ** 2024-03-20T08:48:41.883 INFO:journalctl@ceph.mon.a.smithi026.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-20T08:48:41.883 INFO:journalctl@ceph.mon.a.smithi026.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-20T08:48:41.883 INFO:journalctl@ceph.mon.a.smithi026.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 103.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-20T08:48:41.883 INFO:journalctl@ceph.mon.a.smithi026.stdout: 2024-03-20T08:48:41.883 INFO:journalctl@ceph.mon.a.smithi026.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-20T08:48:41.883 INFO:journalctl@ceph.mon.a.smithi026.stdout: 2024-03-20T08:48:41.883 INFO:journalctl@ceph.mon.a.smithi026.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-20T08:48:41.883 INFO:journalctl@ceph.mon.a.smithi026.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-20T08:48:41.883 INFO:journalctl@ceph.mon.a.smithi026.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-20T08:48:41.883 INFO:journalctl@ceph.mon.a.smithi026.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-20T08:48:41.883 INFO:journalctl@ceph.mon.a.smithi026.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-20T08:48:41.884 INFO:journalctl@ceph.mon.a.smithi026.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-20T08:48:41.884 INFO:journalctl@ceph.mon.a.smithi026.stdout: Cumulative compaction: 0.00 GB write, 19.18 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-20T08:48:41.884 INFO:journalctl@ceph.mon.a.smithi026.stdout: Interval compaction: 0.00 GB write, 19.18 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-20T08:48:41.884 INFO:journalctl@ceph.mon.a.smithi026.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-20T08:48:41.884 INFO:journalctl@ceph.mon.a.smithi026.stdout: Block cache BinnedLRUCache@0x563de8515090#7 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-20T08:48:41.884 INFO:journalctl@ceph.mon.a.smithi026.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-03-20T08:48:41.884 INFO:journalctl@ceph.mon.a.smithi026.stdout: 2024-03-20T08:48:41.884 INFO:journalctl@ceph.mon.a.smithi026.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-20T08:48:41.884 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: starting mon.a rank 0 at public addrs [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] at bind addrs [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 852fbb68-e696-11ee-95c9-87774f69a715 2024-03-20T08:48:41.884 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: mon.a@-1(???) e1 preinit fsid 852fbb68-e696-11ee-95c9-87774f69a715 2024-03-20T08:48:41.884 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: mon.a@-1(???).mds e1 new map 2024-03-20T08:48:41.884 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: mon.a@-1(???).mds e1 print_map 2024-03-20T08:48:41.884 INFO:journalctl@ceph.mon.a.smithi026.stdout: e1 2024-03-20T08:48:41.884 INFO:journalctl@ceph.mon.a.smithi026.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-20T08:48:41.885 INFO:journalctl@ceph.mon.a.smithi026.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-20T08:48:41.885 INFO:journalctl@ceph.mon.a.smithi026.stdout: legacy client fscid: -1 2024-03-20T08:48:41.885 INFO:journalctl@ceph.mon.a.smithi026.stdout: 2024-03-20T08:48:41.885 INFO:journalctl@ceph.mon.a.smithi026.stdout: No filesystems configured 2024-03-20T08:48:41.885 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-20T08:48:41.885 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-20T08:48:41.885 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-20T08:48:41.885 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-20T08:48:41.885 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-20T08:48:41.885 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 bash[68271]: 8e31b6c72afb55f7430be76b011083ff99c6351fdba4b2c23a2bc0fcca8ea034 2024-03-20T08:48:41.885 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 systemd[1]: Started Ceph mon.a for 852fbb68-e696-11ee-95c9-87774f69a715. 2024-03-20T08:48:41.885 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-20T08:48:41.885 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: monmap e1: 1 mons at {a=[v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-20T08:48:41.885 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: fsmap 2024-03-20T08:48:41.886 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: osdmap e1: 0 total, 0 up, 0 in 2024-03-20T08:48:41.886 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:41 smithi026 ceph-mon[68309]: mgrmap e1: no daemons active 2024-03-20T08:48:42.723 INFO:teuthology.orchestra.run.smithi026.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-20T08:48:42.745 INFO:teuthology.orchestra.run.smithi026.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-20T08:48:42.745 INFO:teuthology.orchestra.run.smithi026.stdout:Creating mgr... 2024-03-20T08:48:42.746 INFO:teuthology.orchestra.run.smithi026.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-20T08:48:42.746 INFO:teuthology.orchestra.run.smithi026.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-20T08:48:42.964 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-852fbb68-e696-11ee-95c9-87774f69a715@mgr.y 2024-03-20T08:48:42.964 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Failed to reset failed state of unit ceph-852fbb68-e696-11ee-95c9-87774f69a715@mgr.y.service: Unit ceph-852fbb68-e696-11ee-95c9-87774f69a715@mgr.y.service not loaded. 2024-03-20T08:48:43.058 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-852fbb68-e696-11ee-95c9-87774f69a715.target.wants/ceph-852fbb68-e696-11ee-95c9-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-852fbb68-e696-11ee-95c9-87774f69a715@.service. 2024-03-20T08:48:43.428 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:43 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/259547812' entity='client.admin' 2024-03-20T08:48:43.429 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:48:43 smithi026 systemd[1]: Starting Ceph mgr.y for 852fbb68-e696-11ee-95c9-87774f69a715... 2024-03-20T08:48:43.665 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-20T08:48:43.665 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout disabled 2024-03-20T08:48:43.675 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-20T08:48:43.675 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout inactive 2024-03-20T08:48:43.675 INFO:teuthology.orchestra.run.smithi026.stdout:firewalld.service is not enabled 2024-03-20T08:48:43.675 INFO:teuthology.orchestra.run.smithi026.stdout:Not possible to enable service . firewalld.service is not available 2024-03-20T08:48:43.682 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-20T08:48:43.682 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout disabled 2024-03-20T08:48:43.691 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-20T08:48:43.691 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout inactive 2024-03-20T08:48:43.691 INFO:teuthology.orchestra.run.smithi026.stdout:firewalld.service is not enabled 2024-03-20T08:48:43.691 INFO:teuthology.orchestra.run.smithi026.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-20T08:48:43.691 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for mgr to start... 2024-03-20T08:48:43.691 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for mgr... 2024-03-20T08:48:43.928 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:48:43 smithi026 bash[68548]: 4ab0f6e36531bcb494b645a08aae68273643bdc9a10b1eca556ab77533e76d61 2024-03-20T08:48:43.928 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:48:43 smithi026 systemd[1]: Started Ceph mgr.y for 852fbb68-e696-11ee-95c9-87774f69a715. 2024-03-20T08:48:43.928 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:48:43 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:48:43.741+0000 7f5811878200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-20T08:48:44.220 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:48:43 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:48:43.928+0000 7f5811878200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-20T08:48:44.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:44 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2562510026' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-20T08:48:44.720 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:48:44 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:48:44.280+0000 7f5811878200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-20T08:48:45.064 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:48:44 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:48:44.811+0000 7f5811878200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-20T08:48:45.970 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:48:45 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:48:45.063+0000 7f5811878200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-20T08:48:45.970 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:48:45 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:48:45.514+0000 7f5811878200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-20T08:48:46.470 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:48:46 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:48:46.267+0000 7f5811878200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-20T08:48:46.470 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:48:46 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:48:46.356+0000 7f5811878200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-20T08:48:47.394 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:48:46 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:48:46.981+0000 7f5811878200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-20T08:48:47.394 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:48:47 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:48:47.069+0000 7f5811878200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-20T08:48:47.394 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:48:47 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:48:47.159+0000 7f5811878200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-20T08:48:47.720 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:48:47 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:48:47.394+0000 7f5811878200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-20T08:48:47.720 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:48:47 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:48:47.489+0000 7f5811878200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-20T08:48:47.720 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:48:47 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:48:47.585+0000 7f5811878200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-20T08:48:48.118 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:48:47 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:48:47.826+0000 7f5811878200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-20T08:48:48.118 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:48:47 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:48:47.925+0000 7f5811878200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-20T08:48:48.118 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:48:48 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:48:48.117+0000 7f5811878200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-20T08:48:48.720 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:48:48 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:48:48.378+0000 7f5811878200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-20T08:48:48.756 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-03-20T08:48:48.756 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout { 2024-03-20T08:48:48.756 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "fsid": "852fbb68-e696-11ee-95c9-87774f69a715", 2024-03-20T08:48:48.756 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "health": { 2024-03-20T08:48:48.756 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-20T08:48:48.756 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-20T08:48:48.756 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-20T08:48:48.756 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:48:48.756 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-20T08:48:48.756 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-20T08:48:48.757 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 0 2024-03-20T08:48:48.757 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-03-20T08:48:48.757 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-20T08:48:48.757 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "a" 2024-03-20T08:48:48.757 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-03-20T08:48:48.757 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-20T08:48:48.757 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-20T08:48:48.757 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-20T08:48:48.757 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-20T08:48:48.757 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-20T08:48:48.757 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:48:48.757 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-20T08:48:48.757 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-20T08:48:48.757 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-20T08:48:48.757 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-20T08:48:48.758 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-20T08:48:48.758 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-20T08:48:48.758 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-20T08:48:48.758 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-20T08:48:48.758 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:48:48.758 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-20T08:48:48.758 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-20T08:48:48.758 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-20T08:48:48.758 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-20T08:48:48.758 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-20T08:48:48.758 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-20T08:48:48.758 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-20T08:48:48.758 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-20T08:48:48.758 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-20T08:48:48.759 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:48:48.759 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-20T08:48:48.759 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-20T08:48:48.759 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-20T08:48:48.759 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-20T08:48:48.759 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:48:48.759 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-20T08:48:48.759 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-20T08:48:48.759 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-20T08:48:48.759 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-20T08:48:48.759 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-20T08:48:48.759 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-20T08:48:48.759 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "restful" 2024-03-20T08:48:48.759 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-03-20T08:48:48.759 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-20T08:48:48.760 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:48:48.760 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-20T08:48:48.760 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-20T08:48:48.760 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "modified": "2024-03-20T08:48:35.281565+0000", 2024-03-20T08:48:48.760 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-20T08:48:48.760 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:48:48.760 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-20T08:48:48.760 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout } 2024-03-20T08:48:48.760 INFO:teuthology.orchestra.run.smithi026.stdout:mgr not available, waiting (1/15)... 2024-03-20T08:48:49.211 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:48:48 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:48:48.990+0000 7f5811878200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-20T08:48:49.470 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:48:49 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:48:49.211+0000 7f5811878200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-20T08:48:49.470 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:48:49 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:48:49.415+0000 7f5811878200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-20T08:48:50.470 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:48:50 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:48:50.106+0000 7f5811878200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-20T08:48:50.470 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:48:50 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:48:50.311+0000 7f5811878200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-20T08:48:50.970 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:48:50 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:48:50.499+0000 7f5811878200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-20T08:48:50.970 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:48:50 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:48:50.618+0000 7f5811878200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-20T08:48:50.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:50 smithi026 ceph-mon[68309]: Activating manager daemon y 2024-03-20T08:48:50.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:50 smithi026 ceph-mon[68309]: mgrmap e2: y(active, starting, since 0.0031343s) 2024-03-20T08:48:50.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:50 smithi026 ceph-mon[68309]: from='mgr.14100 172.21.15.26:0/2743293749' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-20T08:48:50.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:50 smithi026 ceph-mon[68309]: from='mgr.14100 172.21.15.26:0/2743293749' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-20T08:48:50.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:50 smithi026 ceph-mon[68309]: from='mgr.14100 172.21.15.26:0/2743293749' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-20T08:48:50.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:50 smithi026 ceph-mon[68309]: from='mgr.14100 172.21.15.26:0/2743293749' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:48:50.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:50 smithi026 ceph-mon[68309]: from='mgr.14100 172.21.15.26:0/2743293749' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-20T08:48:50.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:50 smithi026 ceph-mon[68309]: Manager daemon y is now available 2024-03-20T08:48:50.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:50 smithi026 ceph-mon[68309]: from='mgr.14100 172.21.15.26:0/2743293749' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-20T08:48:50.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:50 smithi026 ceph-mon[68309]: from='mgr.14100 172.21.15.26:0/2743293749' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-20T08:48:50.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:50 smithi026 ceph-mon[68309]: from='mgr.14100 172.21.15.26:0/2743293749' entity='mgr.y' 2024-03-20T08:48:50.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:50 smithi026 ceph-mon[68309]: from='mgr.14100 172.21.15.26:0/2743293749' entity='mgr.y' 2024-03-20T08:48:50.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:50 smithi026 ceph-mon[68309]: from='mgr.14100 172.21.15.26:0/2743293749' entity='mgr.y' 2024-03-20T08:48:52.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:52 smithi026 ceph-mon[68309]: mgrmap e3: y(active, since 1.00754s) 2024-03-20T08:48:54.945 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-03-20T08:48:54.948 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout { 2024-03-20T08:48:54.948 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "fsid": "852fbb68-e696-11ee-95c9-87774f69a715", 2024-03-20T08:48:54.948 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "health": { 2024-03-20T08:48:54.948 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-20T08:48:54.948 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-20T08:48:54.948 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-20T08:48:54.948 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:48:54.948 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-20T08:48:54.949 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-20T08:48:54.949 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 0 2024-03-20T08:48:54.949 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-03-20T08:48:54.949 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-20T08:48:54.949 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "a" 2024-03-20T08:48:54.949 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-03-20T08:48:54.949 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-03-20T08:48:54.949 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-20T08:48:54.949 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-20T08:48:54.949 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-20T08:48:54.949 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-20T08:48:54.949 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:48:54.949 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-20T08:48:54.949 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-20T08:48:54.950 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-20T08:48:54.950 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-20T08:48:54.950 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-20T08:48:54.950 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-20T08:48:54.950 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-20T08:48:54.950 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-20T08:48:54.951 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:48:54.952 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-20T08:48:54.952 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-20T08:48:54.952 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-20T08:48:54.952 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-20T08:48:54.952 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-20T08:48:54.952 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-20T08:48:54.952 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-20T08:48:54.952 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-20T08:48:54.952 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-20T08:48:54.952 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:48:54.952 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-20T08:48:54.952 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-20T08:48:54.952 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-20T08:48:54.953 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:54 smithi026 ceph-mon[68309]: mgrmap e4: y(active, since 3s) 2024-03-20T08:48:54.953 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:54 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3103000541' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-20T08:48:54.955 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-20T08:48:54.955 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:48:54.955 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-20T08:48:54.955 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-20T08:48:54.955 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-20T08:48:54.955 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-20T08:48:54.955 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-20T08:48:54.955 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-20T08:48:54.955 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "restful" 2024-03-20T08:48:54.955 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-03-20T08:48:54.955 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-20T08:48:54.955 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:48:54.956 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-20T08:48:54.956 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-20T08:48:54.956 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "modified": "2024-03-20T08:48:35.281565+0000", 2024-03-20T08:48:54.956 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-20T08:48:54.956 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-20T08:48:54.956 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-20T08:48:54.956 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout } 2024-03-20T08:48:54.956 INFO:teuthology.orchestra.run.smithi026.stdout:mgr is available 2024-03-20T08:48:56.156 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:55 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3650636622' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-20T08:48:56.461 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-03-20T08:48:56.461 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout [global] 2024-03-20T08:48:56.461 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout fsid = 852fbb68-e696-11ee-95c9-87774f69a715 2024-03-20T08:48:56.461 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-20T08:48:56.461 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-20T08:48:56.461 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-20T08:48:56.462 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ms_tcp_read_timeout = 5 2024-03-20T08:48:56.462 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-20T08:48:56.462 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-03-20T08:48:56.462 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-20T08:48:56.462 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-20T08:48:56.462 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-03-20T08:48:56.462 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout [osd] 2024-03-20T08:48:56.462 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-20T08:48:56.462 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-20T08:48:56.462 INFO:teuthology.orchestra.run.smithi026.stdout:Enabling cephadm module... 2024-03-20T08:48:57.850 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:57 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4290837534' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-20T08:48:57.850 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:48:57 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: ignoring --setuser ceph since I am not root 2024-03-20T08:48:57.850 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:48:57 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: ignoring --setgroup ceph since I am not root 2024-03-20T08:48:57.850 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:48:57 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:48:57.666+0000 7f5a3b460200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-20T08:48:58.189 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:48:57 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:48:57.849+0000 7f5a3b460200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-20T08:48:58.217 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr 2024-03-20T08:48:57.167+0000 7f7b46305700 0 --2- 172.21.15.26:0/4290837534 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f7b40105a40 0x7f7b4019aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-03-20T08:48:58.457 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:48:58 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:48:58.189+0000 7f5a3b460200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-20T08:48:58.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:58 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4290837534' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-20T08:48:58.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:58 smithi026 ceph-mon[68309]: mgrmap e5: y(active, since 6s) 2024-03-20T08:48:58.970 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:48:58 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:48:58.714+0000 7f5a3b460200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-20T08:48:59.431 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:48:58 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:48:58.972+0000 7f5a3b460200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-20T08:48:59.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:48:59 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4175161212' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-20T08:48:59.720 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:48:59 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:48:59.430+0000 7f5a3b460200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-20T08:48:59.799 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout { 2024-03-20T08:48:59.799 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-20T08:48:59.799 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-20T08:48:59.799 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-20T08:48:59.799 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-20T08:48:59.799 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout } 2024-03-20T08:48:59.799 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for the mgr to restart... 2024-03-20T08:48:59.799 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for mgr epoch 5... 2024-03-20T08:49:00.470 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:00 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:00.189+0000 7f5a3b460200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-20T08:49:00.470 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:00 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:00.280+0000 7f5a3b460200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-20T08:49:01.220 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:00 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:00.910+0000 7f5a3b460200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-20T08:49:01.220 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:00 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:00.998+0000 7f5a3b460200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-20T08:49:01.220 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:01 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:01.091+0000 7f5a3b460200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-20T08:49:01.720 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:01 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:01.328+0000 7f5a3b460200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-20T08:49:01.721 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:01 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:01.424+0000 7f5a3b460200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-20T08:49:01.721 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:01 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:01.523+0000 7f5a3b460200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-20T08:49:02.061 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:01 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:01.765+0000 7f5a3b460200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-20T08:49:02.062 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:01 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:01.865+0000 7f5a3b460200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-20T08:49:02.324 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:02 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:02.061+0000 7f5a3b460200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-20T08:49:02.720 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:02 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:02.323+0000 7f5a3b460200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-20T08:49:03.220 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:02 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:02.935+0000 7f5a3b460200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-20T08:49:03.220 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:03 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:03.156+0000 7f5a3b460200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-20T08:49:03.720 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:03 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:03.358+0000 7f5a3b460200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-20T08:49:04.464 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:04 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:04.062+0000 7f5a3b460200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-20T08:49:04.464 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:04 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:04.271+0000 7f5a3b460200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-20T08:49:04.720 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:04 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:04.463+0000 7f5a3b460200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-20T08:49:04.720 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:04 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:04.582+0000 7f5a3b460200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-20T08:49:04.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:04 smithi026 ceph-mon[68309]: Active manager daemon y restarted 2024-03-20T08:49:04.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:04 smithi026 ceph-mon[68309]: Activating manager daemon y 2024-03-20T08:49:04.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:04 smithi026 ceph-mon[68309]: osdmap e2: 0 total, 0 up, 0 in 2024-03-20T08:49:04.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:04 smithi026 ceph-mon[68309]: mgrmap e6: y(active, starting, since 0.00304549s) 2024-03-20T08:49:04.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:04 smithi026 ceph-mon[68309]: from='mgr.14116 172.21.15.26:0/2785097880' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:49:04.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:04 smithi026 ceph-mon[68309]: from='mgr.14116 172.21.15.26:0/2785097880' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-20T08:49:04.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:04 smithi026 ceph-mon[68309]: from='mgr.14116 172.21.15.26:0/2785097880' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-20T08:49:04.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:04 smithi026 ceph-mon[68309]: from='mgr.14116 172.21.15.26:0/2785097880' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-20T08:49:04.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:04 smithi026 ceph-mon[68309]: from='mgr.14116 172.21.15.26:0/2785097880' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-20T08:49:04.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:04 smithi026 ceph-mon[68309]: Manager daemon y is now available 2024-03-20T08:49:04.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:04 smithi026 ceph-mon[68309]: from='mgr.14116 172.21.15.26:0/2785097880' entity='mgr.y' 2024-03-20T08:49:04.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:04 smithi026 ceph-mon[68309]: from='mgr.14116 172.21.15.26:0/2785097880' entity='mgr.y' 2024-03-20T08:49:04.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:04 smithi026 ceph-mon[68309]: from='mgr.14116 172.21.15.26:0/2785097880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:49:04.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:04 smithi026 ceph-mon[68309]: from='mgr.14116 172.21.15.26:0/2785097880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:49:05.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:05 smithi026 ceph-mon[68309]: Found migration_current of "None". Setting to last migration. 2024-03-20T08:49:05.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:05 smithi026 ceph-mon[68309]: from='mgr.14116 172.21.15.26:0/2785097880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:49:05.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:05 smithi026 ceph-mon[68309]: from='mgr.14116 172.21.15.26:0/2785097880' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-20T08:49:05.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:05 smithi026 ceph-mon[68309]: from='mgr.14116 172.21.15.26:0/2785097880' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-20T08:49:05.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:05 smithi026 ceph-mon[68309]: from='mgr.14116 172.21.15.26:0/2785097880' entity='mgr.y' 2024-03-20T08:49:05.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:05 smithi026 ceph-mon[68309]: from='mgr.14116 172.21.15.26:0/2785097880' entity='mgr.y' 2024-03-20T08:49:05.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:05 smithi026 ceph-mon[68309]: mgrmap e7: y(active, since 1.00511s) 2024-03-20T08:49:06.244 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout { 2024-03-20T08:49:06.244 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-20T08:49:06.244 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-20T08:49:06.244 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout } 2024-03-20T08:49:06.244 INFO:teuthology.orchestra.run.smithi026.stdout:mgr epoch 5 is available 2024-03-20T08:49:06.244 INFO:teuthology.orchestra.run.smithi026.stdout:Setting orchestrator backend to cephadm... 2024-03-20T08:49:06.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:06 smithi026 ceph-mon[68309]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-20T08:49:06.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:06 smithi026 ceph-mon[68309]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-20T08:49:06.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:06 smithi026 ceph-mon[68309]: [20/Mar/2024:08:49:05] ENGINE Bus STARTING 2024-03-20T08:49:06.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:06 smithi026 ceph-mon[68309]: [20/Mar/2024:08:49:05] ENGINE Serving on https://172.21.15.26:7150 2024-03-20T08:49:06.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:06 smithi026 ceph-mon[68309]: [20/Mar/2024:08:49:05] ENGINE Serving on http://172.21.15.26:8765 2024-03-20T08:49:06.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:06 smithi026 ceph-mon[68309]: [20/Mar/2024:08:49:05] ENGINE Bus STARTED 2024-03-20T08:49:06.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:06 smithi026 ceph-mon[68309]: from='mgr.14116 172.21.15.26:0/2785097880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:49:07.929 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:07 smithi026 ceph-mon[68309]: mgrmap e8: y(active, since 2s) 2024-03-20T08:49:07.929 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:07 smithi026 ceph-mon[68309]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:49:07.929 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:07 smithi026 ceph-mon[68309]: from='mgr.14116 172.21.15.26:0/2785097880' entity='mgr.y' 2024-03-20T08:49:07.929 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:07 smithi026 ceph-mon[68309]: from='mgr.14116 172.21.15.26:0/2785097880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:49:08.890 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:08 smithi026 ceph-mon[68309]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:49:09.228 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-20T08:49:09.228 INFO:teuthology.orchestra.run.smithi026.stdout:Generating ssh key... 2024-03-20T08:49:11.689 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:11 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: Generating public/private rsa key pair. 2024-03-20T08:49:11.689 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:11 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: Your identification has been saved in /tmp/tmpnbupnv6w/key. 2024-03-20T08:49:11.689 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:11 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: Your public key has been saved in /tmp/tmpnbupnv6w/key.pub. 2024-03-20T08:49:11.689 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:11 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: The key fingerprint is: 2024-03-20T08:49:11.689 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:11 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: SHA256:/1/o6hdRHIrCCZg3SUr/MVkDxTd7xPF7cwdNpU9MXC4 ceph-852fbb68-e696-11ee-95c9-87774f69a715 2024-03-20T08:49:11.689 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:11 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: The key's randomart image is: 2024-03-20T08:49:11.689 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:11 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: +---[RSA 3072]----+ 2024-03-20T08:49:11.689 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:11 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: | .+o..++ .=X| 2024-03-20T08:49:11.689 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:11 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: | .oo+o +.oo.X=| 2024-03-20T08:49:11.689 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:11 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: | ....B ...E.*| 2024-03-20T08:49:11.689 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:11 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: | . + ..*.| 2024-03-20T08:49:11.690 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:11 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: | S. .+=| 2024-03-20T08:49:11.690 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:11 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: | . ..=| 2024-03-20T08:49:11.690 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:11 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: | . ...| 2024-03-20T08:49:11.690 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:11 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: | . ... | 2024-03-20T08:49:11.690 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:11 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: | .++o | 2024-03-20T08:49:11.690 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:11 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: +----[SHA256]-----+ 2024-03-20T08:49:12.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:12 smithi026 ceph-mon[68309]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:49:12.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:12 smithi026 ceph-mon[68309]: Generating ssh key... 2024-03-20T08:49:12.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:12 smithi026 ceph-mon[68309]: from='mgr.14116 172.21.15.26:0/2785097880' entity='mgr.y' 2024-03-20T08:49:12.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:12 smithi026 ceph-mon[68309]: from='mgr.14116 172.21.15.26:0/2785097880' entity='mgr.y' 2024-03-20T08:49:13.641 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDA8zSpottQ6AiyqAvkmbWX/qxjFNgBLmajItT+FHPtX7Y4UhjyHX0Xa3qZ/y2O7J4hjCPUloU7neab+OdEv50GOXMPnMzFMzC9AQadHjQmmYV2CAIXW40CXfAjGcXVR/M1u1pM0t60kr8gUqz4oq+46+mu9ma4rBb2LNL7z/hHShqveERiQUyGnC2I8Pgtpaw41hWim2Qs9Zb3grkNfo5Csj265S4dKbW8pzcVOD+iFOK5QeneUyt+Sj/Y1dptNt3jin7VkWrfv3d7RW9R+X8vMQdvm4uEk0sCBnP0zftk3RCK6odNT8pnr9zBODQgKrGh9dxu0B7N/T0DnHfYxTgaaKr1bnBP7PZUMYc04/tGznu2N55v/7XT+haAUaw3yzYbOQ4nCH81y5RURtE16paK0DjNxPOMljTR9fmEhj3JO5Ysn8DnBxqtuuGmr/0AcuUgztbyuUJ9lGthNIwLUDkIenHLg8pj+Y4gEftMyCCeJDbeSiAV1B3Fq5LFrqHPvqE= ceph-852fbb68-e696-11ee-95c9-87774f69a715 2024-03-20T08:49:13.642 INFO:teuthology.orchestra.run.smithi026.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-20T08:49:13.642 INFO:teuthology.orchestra.run.smithi026.stdout:Adding key to root@localhost authorized_keys... 2024-03-20T08:49:13.643 INFO:teuthology.orchestra.run.smithi026.stdout:Adding host smithi026... 2024-03-20T08:49:13.916 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:13 smithi026 ceph-mon[68309]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:49:14.850 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:14 smithi026 ceph-mon[68309]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi026", "addr": "172.21.15.26", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:49:15.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:15 smithi026 ceph-mon[68309]: Deploying cephadm binary to smithi026 2024-03-20T08:49:19.246 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout Added host 'smithi026' with addr '172.21.15.26' 2024-03-20T08:49:19.246 INFO:teuthology.orchestra.run.smithi026.stdout:Deploying unmanaged mon service... 2024-03-20T08:49:19.612 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:19 smithi026 ceph-mon[68309]: from='mgr.14116 172.21.15.26:0/2785097880' entity='mgr.y' 2024-03-20T08:49:19.612 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:19 smithi026 ceph-mon[68309]: Added host smithi026 2024-03-20T08:49:19.612 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:19 smithi026 ceph-mon[68309]: from='mgr.14116 172.21.15.26:0/2785097880' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:49:21.164 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-20T08:49:21.164 INFO:teuthology.orchestra.run.smithi026.stdout:Deploying unmanaged mgr service... 2024-03-20T08:49:21.423 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:21 smithi026 ceph-mon[68309]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:49:21.423 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:21 smithi026 ceph-mon[68309]: Saving service mon spec with placement count:5 2024-03-20T08:49:21.424 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:21 smithi026 ceph-mon[68309]: from='mgr.14116 172.21.15.26:0/2785097880' entity='mgr.y' 2024-03-20T08:49:23.165 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-20T08:49:23.168 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:22 smithi026 ceph-mon[68309]: from='mgr.14116 172.21.15.26:0/2785097880' entity='mgr.y' 2024-03-20T08:49:23.168 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:22 smithi026 ceph-mon[68309]: from='mgr.14116 172.21.15.26:0/2785097880' entity='mgr.y' 2024-03-20T08:49:23.168 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:22 smithi026 ceph-mon[68309]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:49:23.168 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:22 smithi026 ceph-mon[68309]: Saving service mgr spec with placement count:2 2024-03-20T08:49:23.168 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:22 smithi026 ceph-mon[68309]: from='mgr.14116 172.21.15.26:0/2785097880' entity='mgr.y' 2024-03-20T08:49:25.374 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:25 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1140592433' entity='client.admin' 2024-03-20T08:49:25.374 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:25 smithi026 ceph-mon[68309]: from='mgr.14116 172.21.15.26:0/2785097880' entity='mgr.y' 2024-03-20T08:49:26.295 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:26 smithi026 ceph-mon[68309]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:49:26.295 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:26 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3138336940' entity='client.admin' 2024-03-20T08:49:26.295 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:26 smithi026 ceph-mon[68309]: from='mgr.14116 172.21.15.26:0/2785097880' entity='mgr.y' 2024-03-20T08:49:26.573 INFO:teuthology.orchestra.run.smithi026.stdout:Enabling the dashboard module... 2024-03-20T08:49:28.412 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:28 smithi026 ceph-mon[68309]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:49:28.412 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:28 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/332877334' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-20T08:49:28.412 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:28 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: ignoring --setuser ceph since I am not root 2024-03-20T08:49:28.412 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:28 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: ignoring --setgroup ceph since I am not root 2024-03-20T08:49:28.412 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:28 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:28.411+0000 7fe6399b4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-20T08:49:28.708 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:28 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:28.595+0000 7fe6399b4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-20T08:49:28.970 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:28 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:28.934+0000 7fe6399b4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-20T08:49:29.253 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:29 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/332877334' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-20T08:49:29.253 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:29 smithi026 ceph-mon[68309]: mgrmap e9: y(active, since 23s) 2024-03-20T08:49:29.717 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:29 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:29.458+0000 7fe6399b4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-20T08:49:29.970 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:29 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:29.717+0000 7fe6399b4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-20T08:49:30.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:30 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1793834987' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-20T08:49:30.470 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:30 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:30.175+0000 7fe6399b4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-20T08:49:30.568 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout { 2024-03-20T08:49:30.569 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-20T08:49:30.569 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-20T08:49:30.569 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-20T08:49:30.569 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-20T08:49:30.569 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout } 2024-03-20T08:49:30.569 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for the mgr to restart... 2024-03-20T08:49:30.569 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for mgr epoch 9... 2024-03-20T08:49:31.220 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:30 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:30.939+0000 7fe6399b4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-20T08:49:31.220 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:31 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:31.031+0000 7fe6399b4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-20T08:49:31.970 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:31 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:31.661+0000 7fe6399b4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-20T08:49:31.970 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:31 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:31.750+0000 7fe6399b4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-20T08:49:31.970 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:31 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:31.843+0000 7fe6399b4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-20T08:49:32.470 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:32 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:32.078+0000 7fe6399b4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-20T08:49:32.470 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:32 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:32.175+0000 7fe6399b4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-20T08:49:32.470 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:32 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:32.273+0000 7fe6399b4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-20T08:49:32.813 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:32 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:32.518+0000 7fe6399b4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-20T08:49:32.813 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:32 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:32.618+0000 7fe6399b4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-20T08:49:33.073 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:32 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:32.813+0000 7fe6399b4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-20T08:49:33.470 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:33 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:33.073+0000 7fe6399b4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-20T08:49:33.970 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:33 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:33.686+0000 7fe6399b4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-20T08:49:33.970 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:33 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:33.906+0000 7fe6399b4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-20T08:49:34.470 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:34 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:34.110+0000 7fe6399b4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-20T08:49:35.203 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:34 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:34.804+0000 7fe6399b4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-20T08:49:35.203 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:35 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:35.012+0000 7fe6399b4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-20T08:49:35.203 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:35 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:35.203+0000 7fe6399b4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-20T08:49:35.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:35 smithi026 ceph-mon[68309]: Active manager daemon y restarted 2024-03-20T08:49:35.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:35 smithi026 ceph-mon[68309]: Activating manager daemon y 2024-03-20T08:49:35.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:35 smithi026 ceph-mon[68309]: osdmap e3: 0 total, 0 up, 0 in 2024-03-20T08:49:35.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:35 smithi026 ceph-mon[68309]: mgrmap e10: y(active, starting, since 0.00309201s) 2024-03-20T08:49:35.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:35 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:49:35.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:35 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-20T08:49:35.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:35 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-20T08:49:35.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:35 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-20T08:49:35.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:35 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-20T08:49:35.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:35 smithi026 ceph-mon[68309]: Manager daemon y is now available 2024-03-20T08:49:35.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:35 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:49:35.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:35 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:49:35.459 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:49:35 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:49:35.322+0000 7fe6399b4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-20T08:49:36.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:36 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-20T08:49:36.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:36 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-20T08:49:36.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:36 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-20T08:49:36.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:36 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:49:36.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:36 smithi026 ceph-mon[68309]: [20/Mar/2024:08:49:35] ENGINE Bus STARTING 2024-03-20T08:49:36.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:36 smithi026 ceph-mon[68309]: [20/Mar/2024:08:49:36] ENGINE Serving on https://172.21.15.26:7150 2024-03-20T08:49:36.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:36 smithi026 ceph-mon[68309]: [20/Mar/2024:08:49:36] ENGINE Serving on http://172.21.15.26:8765 2024-03-20T08:49:36.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:36 smithi026 ceph-mon[68309]: [20/Mar/2024:08:49:36] ENGINE Bus STARTED 2024-03-20T08:49:36.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:36 smithi026 ceph-mon[68309]: mgrmap e11: y(active, since 1.00593s) 2024-03-20T08:49:37.346 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout { 2024-03-20T08:49:37.346 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-20T08:49:37.346 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-20T08:49:37.346 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout } 2024-03-20T08:49:37.346 INFO:teuthology.orchestra.run.smithi026.stdout:mgr epoch 9 is available 2024-03-20T08:49:37.346 INFO:teuthology.orchestra.run.smithi026.stdout:Generating a dashboard self-signed certificate... 2024-03-20T08:49:38.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:37 smithi026 ceph-mon[68309]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-20T08:49:38.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:37 smithi026 ceph-mon[68309]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-20T08:49:38.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:37 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:49:38.876 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-20T08:49:38.877 INFO:teuthology.orchestra.run.smithi026.stdout:Creating initial admin user... 2024-03-20T08:49:38.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:38 smithi026 ceph-mon[68309]: mgrmap e12: y(active, since 2s) 2024-03-20T08:49:38.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:38 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:49:38.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:38 smithi026 ceph-mon[68309]: from='client.14160 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:49:38.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:38 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:49:38.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:38 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:49:40.854 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$PCCEiX5VjVvk5vVdAoflauE2HN.QJbyMFRg4iIu38HLfXldFPg4/2", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710924580, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-20T08:49:40.854 INFO:teuthology.orchestra.run.smithi026.stdout:Fetching dashboard port number... 2024-03-20T08:49:41.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:41 smithi026 ceph-mon[68309]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:49:41.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:41 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:49:42.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:42 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3755752091' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-20T08:49:42.496 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 8443 2024-03-20T08:49:42.507 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-20T08:49:42.507 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout disabled 2024-03-20T08:49:42.517 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-20T08:49:42.517 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout inactive 2024-03-20T08:49:42.517 INFO:teuthology.orchestra.run.smithi026.stdout:firewalld.service is not enabled 2024-03-20T08:49:42.517 INFO:teuthology.orchestra.run.smithi026.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-20T08:49:42.519 INFO:teuthology.orchestra.run.smithi026.stdout:Ceph Dashboard is now available at: 2024-03-20T08:49:42.519 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:49:42.519 INFO:teuthology.orchestra.run.smithi026.stdout: URL: https://smithi026.front.sepia.ceph.com:8443/ 2024-03-20T08:49:42.519 INFO:teuthology.orchestra.run.smithi026.stdout: User: admin 2024-03-20T08:49:42.519 INFO:teuthology.orchestra.run.smithi026.stdout: Password: d403pu1yyr 2024-03-20T08:49:42.519 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:49:42.519 INFO:teuthology.orchestra.run.smithi026.stdout:Saving cluster configuration to /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/config directory 2024-03-20T08:49:42.520 INFO:teuthology.orchestra.run.smithi026.stdout:Enabling autotune for osd_memory_target 2024-03-20T08:49:43.348 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:43 smithi026 ceph-mon[68309]: mgrmap e13: y(active, since 6s) 2024-03-20T08:49:45.560 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-20T08:49:45.560 INFO:teuthology.orchestra.run.smithi026.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-20T08:49:45.560 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:49:45.560 INFO:teuthology.orchestra.run.smithi026.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-20T08:49:45.560 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:49:45.561 INFO:teuthology.orchestra.run.smithi026.stdout:Or, if you are only running a single cluster on this host: 2024-03-20T08:49:45.561 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:49:45.561 INFO:teuthology.orchestra.run.smithi026.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-20T08:49:45.561 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:49:45.561 INFO:teuthology.orchestra.run.smithi026.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-20T08:49:45.561 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:49:45.561 INFO:teuthology.orchestra.run.smithi026.stdout: ceph telemetry on 2024-03-20T08:49:45.561 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:49:45.561 INFO:teuthology.orchestra.run.smithi026.stdout:For more information see: 2024-03-20T08:49:45.561 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:49:45.561 INFO:teuthology.orchestra.run.smithi026.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-20T08:49:45.561 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:49:45.561 INFO:teuthology.orchestra.run.smithi026.stdout:Bootstrap complete. 2024-03-20T08:49:45.615 INFO:tasks.cephadm:Fetching config... 2024-03-20T08:49:45.615 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-20T08:49:45.615 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-20T08:49:45.636 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-20T08:49:45.636 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-20T08:49:45.636 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-20T08:49:45.695 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-20T08:49:45.695 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-20T08:49:45.695 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-20T08:49:45.764 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-20T08:49:45.764 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-20T08:49:45.764 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-20T08:49:45.823 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-20T08:49:45.823 DEBUG:teuthology.orchestra.run.smithi026:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDA8zSpottQ6AiyqAvkmbWX/qxjFNgBLmajItT+FHPtX7Y4UhjyHX0Xa3qZ/y2O7J4hjCPUloU7neab+OdEv50GOXMPnMzFMzC9AQadHjQmmYV2CAIXW40CXfAjGcXVR/M1u1pM0t60kr8gUqz4oq+46+mu9ma4rBb2LNL7z/hHShqveERiQUyGnC2I8Pgtpaw41hWim2Qs9Zb3grkNfo5Csj265S4dKbW8pzcVOD+iFOK5QeneUyt+Sj/Y1dptNt3jin7VkWrfv3d7RW9R+X8vMQdvm4uEk0sCBnP0zftk3RCK6odNT8pnr9zBODQgKrGh9dxu0B7N/T0DnHfYxTgaaKr1bnBP7PZUMYc04/tGznu2N55v/7XT+haAUaw3yzYbOQ4nCH81y5RURtE16paK0DjNxPOMljTR9fmEhj3JO5Ysn8DnBxqtuuGmr/0AcuUgztbyuUJ9lGthNIwLUDkIenHLg8pj+Y4gEftMyCCeJDbeSiAV1B3Fq5LFrqHPvqE= ceph-852fbb68-e696-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-20T08:49:45.901 INFO:teuthology.orchestra.run.smithi026.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDA8zSpottQ6AiyqAvkmbWX/qxjFNgBLmajItT+FHPtX7Y4UhjyHX0Xa3qZ/y2O7J4hjCPUloU7neab+OdEv50GOXMPnMzFMzC9AQadHjQmmYV2CAIXW40CXfAjGcXVR/M1u1pM0t60kr8gUqz4oq+46+mu9ma4rBb2LNL7z/hHShqveERiQUyGnC2I8Pgtpaw41hWim2Qs9Zb3grkNfo5Csj265S4dKbW8pzcVOD+iFOK5QeneUyt+Sj/Y1dptNt3jin7VkWrfv3d7RW9R+X8vMQdvm4uEk0sCBnP0zftk3RCK6odNT8pnr9zBODQgKrGh9dxu0B7N/T0DnHfYxTgaaKr1bnBP7PZUMYc04/tGznu2N55v/7XT+haAUaw3yzYbOQ4nCH81y5RURtE16paK0DjNxPOMljTR9fmEhj3JO5Ysn8DnBxqtuuGmr/0AcuUgztbyuUJ9lGthNIwLUDkIenHLg8pj+Y4gEftMyCCeJDbeSiAV1B3Fq5LFrqHPvqE= ceph-852fbb68-e696-11ee-95c9-87774f69a715 2024-03-20T08:49:45.912 DEBUG:teuthology.orchestra.run.smithi115:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDA8zSpottQ6AiyqAvkmbWX/qxjFNgBLmajItT+FHPtX7Y4UhjyHX0Xa3qZ/y2O7J4hjCPUloU7neab+OdEv50GOXMPnMzFMzC9AQadHjQmmYV2CAIXW40CXfAjGcXVR/M1u1pM0t60kr8gUqz4oq+46+mu9ma4rBb2LNL7z/hHShqveERiQUyGnC2I8Pgtpaw41hWim2Qs9Zb3grkNfo5Csj265S4dKbW8pzcVOD+iFOK5QeneUyt+Sj/Y1dptNt3jin7VkWrfv3d7RW9R+X8vMQdvm4uEk0sCBnP0zftk3RCK6odNT8pnr9zBODQgKrGh9dxu0B7N/T0DnHfYxTgaaKr1bnBP7PZUMYc04/tGznu2N55v/7XT+haAUaw3yzYbOQ4nCH81y5RURtE16paK0DjNxPOMljTR9fmEhj3JO5Ysn8DnBxqtuuGmr/0AcuUgztbyuUJ9lGthNIwLUDkIenHLg8pj+Y4gEftMyCCeJDbeSiAV1B3Fq5LFrqHPvqE= ceph-852fbb68-e696-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-20T08:49:45.956 INFO:teuthology.orchestra.run.smithi115.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDA8zSpottQ6AiyqAvkmbWX/qxjFNgBLmajItT+FHPtX7Y4UhjyHX0Xa3qZ/y2O7J4hjCPUloU7neab+OdEv50GOXMPnMzFMzC9AQadHjQmmYV2CAIXW40CXfAjGcXVR/M1u1pM0t60kr8gUqz4oq+46+mu9ma4rBb2LNL7z/hHShqveERiQUyGnC2I8Pgtpaw41hWim2Qs9Zb3grkNfo5Csj265S4dKbW8pzcVOD+iFOK5QeneUyt+Sj/Y1dptNt3jin7VkWrfv3d7RW9R+X8vMQdvm4uEk0sCBnP0zftk3RCK6odNT8pnr9zBODQgKrGh9dxu0B7N/T0DnHfYxTgaaKr1bnBP7PZUMYc04/tGznu2N55v/7XT+haAUaw3yzYbOQ4nCH81y5RURtE16paK0DjNxPOMljTR9fmEhj3JO5Ysn8DnBxqtuuGmr/0AcuUgztbyuUJ9lGthNIwLUDkIenHLg8pj+Y4gEftMyCCeJDbeSiAV1B3Fq5LFrqHPvqE= ceph-852fbb68-e696-11ee-95c9-87774f69a715 2024-03-20T08:49:45.968 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDA8zSpottQ6AiyqAvkmbWX/qxjFNgBLmajItT+FHPtX7Y4UhjyHX0Xa3qZ/y2O7J4hjCPUloU7neab+OdEv50GOXMPnMzFMzC9AQadHjQmmYV2CAIXW40CXfAjGcXVR/M1u1pM0t60kr8gUqz4oq+46+mu9ma4rBb2LNL7z/hHShqveERiQUyGnC2I8Pgtpaw41hWim2Qs9Zb3grkNfo5Csj265S4dKbW8pzcVOD+iFOK5QeneUyt+Sj/Y1dptNt3jin7VkWrfv3d7RW9R+X8vMQdvm4uEk0sCBnP0zftk3RCK6odNT8pnr9zBODQgKrGh9dxu0B7N/T0DnHfYxTgaaKr1bnBP7PZUMYc04/tGznu2N55v/7XT+haAUaw3yzYbOQ4nCH81y5RURtE16paK0DjNxPOMljTR9fmEhj3JO5Ysn8DnBxqtuuGmr/0AcuUgztbyuUJ9lGthNIwLUDkIenHLg8pj+Y4gEftMyCCeJDbeSiAV1B3Fq5LFrqHPvqE= ceph-852fbb68-e696-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-20T08:49:46.012 INFO:teuthology.orchestra.run.smithi122.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDA8zSpottQ6AiyqAvkmbWX/qxjFNgBLmajItT+FHPtX7Y4UhjyHX0Xa3qZ/y2O7J4hjCPUloU7neab+OdEv50GOXMPnMzFMzC9AQadHjQmmYV2CAIXW40CXfAjGcXVR/M1u1pM0t60kr8gUqz4oq+46+mu9ma4rBb2LNL7z/hHShqveERiQUyGnC2I8Pgtpaw41hWim2Qs9Zb3grkNfo5Csj265S4dKbW8pzcVOD+iFOK5QeneUyt+Sj/Y1dptNt3jin7VkWrfv3d7RW9R+X8vMQdvm4uEk0sCBnP0zftk3RCK6odNT8pnr9zBODQgKrGh9dxu0B7N/T0DnHfYxTgaaKr1bnBP7PZUMYc04/tGznu2N55v/7XT+haAUaw3yzYbOQ4nCH81y5RURtE16paK0DjNxPOMljTR9fmEhj3JO5Ysn8DnBxqtuuGmr/0AcuUgztbyuUJ9lGthNIwLUDkIenHLg8pj+Y4gEftMyCCeJDbeSiAV1B3Fq5LFrqHPvqE= ceph-852fbb68-e696-11ee-95c9-87774f69a715 2024-03-20T08:49:46.023 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-20T08:49:46.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:45 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2947066303' entity='client.admin' 2024-03-20T08:49:46.398 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:49:49.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:49 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:49:49.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:49 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:49:49.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:49 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi026", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:49:49.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:49 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:49:49.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:49 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:49:49.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:49 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:49:49.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:49 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2813902441' entity='client.admin' 2024-03-20T08:49:50.538 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-20T08:49:50.538 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-20T08:49:50.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:50 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:49:50.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:50 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:49:51.038 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:49:54.139 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi115 2024-03-20T08:49:54.139 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-20T08:49:54.139 DEBUG:teuthology.orchestra.run.smithi115:> dd of=/etc/ceph/ceph.conf 2024-03-20T08:49:54.161 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-20T08:49:54.161 DEBUG:teuthology.orchestra.run.smithi115:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-20T08:49:54.222 INFO:tasks.cephadm:Adding host smithi115 to orchestrator... 2024-03-20T08:49:54.222 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph orch host add smithi115 2024-03-20T08:49:54.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:54 smithi026 ceph-mon[68309]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:49:54.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:54 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:49:54.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:54 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:49:54.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:54 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:49:54.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:54 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:49:54.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:54 smithi026 ceph-mon[68309]: Updating smithi026:/etc/ceph/ceph.conf 2024-03-20T08:49:54.492 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:49:55.620 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:55 smithi026 ceph-mon[68309]: Updating smithi026:/var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-20T08:49:55.620 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:55 smithi026 ceph-mon[68309]: Updating smithi026:/etc/ceph/ceph.client.admin.keyring 2024-03-20T08:49:55.620 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:55 smithi026 ceph-mon[68309]: Updating smithi026:/var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-20T08:49:55.620 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:55 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:49:55.620 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:55 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:49:55.620 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:55 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:49:57.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:57 smithi026 ceph-mon[68309]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:49:58.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:58 smithi026 ceph-mon[68309]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi115", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:49:58.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:58 smithi026 ceph-mon[68309]: Deploying cephadm binary to smithi115 2024-03-20T08:49:59.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:49:59 smithi026 ceph-mon[68309]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:00.402 INFO:teuthology.orchestra.run.smithi026.stdout:Added host 'smithi115' with addr '172.21.15.115' 2024-03-20T08:50:01.166 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-20T08:50:01.441 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:50:01.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:01 smithi026 ceph-mon[68309]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:01.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:01 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:01.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:01 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:50:01.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:01 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:01.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:01 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:02.682 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:02 smithi026 ceph-mon[68309]: Added host smithi115 2024-03-20T08:50:02.683 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:02 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:03.455 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:50:03.455 INFO:teuthology.orchestra.run.smithi026.stdout:[{"addr": "172.21.15.26", "hostname": "smithi026", "labels": [], "status": ""}, {"addr": "172.21.15.115", "hostname": "smithi115", "labels": [], "status": ""}] 2024-03-20T08:50:03.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:03 smithi026 ceph-mon[68309]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:04.026 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi122 2024-03-20T08:50:04.026 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-20T08:50:04.026 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/etc/ceph/ceph.conf 2024-03-20T08:50:04.048 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-20T08:50:04.048 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-20T08:50:04.109 INFO:tasks.cephadm:Adding host smithi122 to orchestrator... 2024-03-20T08:50:04.109 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph orch host add smithi122 2024-03-20T08:50:04.382 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:50:05.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:05 smithi026 ceph-mon[68309]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:05.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:05 smithi026 ceph-mon[68309]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T08:50:05.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:05 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:05.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:05 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:05.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:05 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:05.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:05 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:05.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:05 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi115", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:50:05.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:05 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:05.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:05 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:50:06.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:06 smithi026 ceph-mon[68309]: Updating smithi115:/etc/ceph/ceph.conf 2024-03-20T08:50:06.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:06 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:06.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:06 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:06.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:06 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:07.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:07 smithi026 ceph-mon[68309]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:07.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:07 smithi026 ceph-mon[68309]: Updating smithi115:/var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-20T08:50:07.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:07 smithi026 ceph-mon[68309]: Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2024-03-20T08:50:07.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:07 smithi026 ceph-mon[68309]: Updating smithi115:/var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-20T08:50:08.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:08 smithi026 ceph-mon[68309]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi122", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:50:08.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:08 smithi026 ceph-mon[68309]: Deploying cephadm binary to smithi122 2024-03-20T08:50:09.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:09 smithi026 ceph-mon[68309]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:10.383 INFO:teuthology.orchestra.run.smithi026.stdout:Added host 'smithi122' with addr '172.21.15.122' 2024-03-20T08:50:11.051 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-20T08:50:11.334 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:50:11.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:11 smithi026 ceph-mon[68309]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:11.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:11 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:11.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:11 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:50:11.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:11 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:11.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:11 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:12.670 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:12 smithi026 ceph-mon[68309]: Added host smithi122 2024-03-20T08:50:12.670 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:12 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:13.424 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:50:13.424 INFO:teuthology.orchestra.run.smithi026.stdout:[{"addr": "172.21.15.26", "hostname": "smithi026", "labels": [], "status": ""}, {"addr": "172.21.15.115", "hostname": "smithi115", "labels": [], "status": ""}, {"addr": "172.21.15.122", "hostname": "smithi122", "labels": [], "status": ""}] 2024-03-20T08:50:13.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:13 smithi026 ceph-mon[68309]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:14.166 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-20T08:50:14.166 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd crush tunables default 2024-03-20T08:50:14.441 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:50:15.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:15 smithi026 ceph-mon[68309]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:15.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:15 smithi026 ceph-mon[68309]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T08:50:16.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:16 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:16.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:16 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:16.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:16 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:16.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:16 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:16.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:16 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi122", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:50:16.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:16 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:16.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:16 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:50:16.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:16 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:50:16.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:16 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:50:16.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:16 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:16.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:16 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:16.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:16 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:17.640 INFO:teuthology.orchestra.run.smithi026.stderr:adjusted tunables profile to default 2024-03-20T08:50:17.662 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:17 smithi026 ceph-mon[68309]: Updating smithi122:/etc/ceph/ceph.conf 2024-03-20T08:50:17.662 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:17 smithi026 ceph-mon[68309]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:17.662 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:17 smithi026 ceph-mon[68309]: Updating smithi122:/var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-20T08:50:17.663 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:17 smithi026 ceph-mon[68309]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-03-20T08:50:17.663 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:17 smithi026 ceph-mon[68309]: Updating smithi122:/var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-20T08:50:17.663 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:17 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:50:17.663 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:17 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:17.663 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:17 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:50:17.663 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:17 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:17.663 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:17 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/969314353' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-20T08:50:18.454 INFO:tasks.cephadm:Adding mon.a on smithi026 2024-03-20T08:50:18.454 INFO:tasks.cephadm:Adding mon.b on smithi115 2024-03-20T08:50:18.454 INFO:tasks.cephadm:Adding mon.c on smithi122 2024-03-20T08:50:18.454 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph orch apply mon '3;smithi026:172.21.15.26=a;smithi115:172.21.15.115=b;smithi122:172.21.15.122=c' 2024-03-20T08:50:18.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:18 smithi026 ceph-mon[68309]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:18.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:18 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/969314353' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-20T08:50:18.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:18 smithi026 ceph-mon[68309]: osdmap e4: 0 total, 0 up, 0 in 2024-03-20T08:50:18.747 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-20T08:50:20.545 INFO:teuthology.orchestra.run.smithi122.stdout:Scheduled mon update... 2024-03-20T08:50:20.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:20 smithi026 ceph-mon[68309]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:20.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:20 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:20.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:20 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:50:20.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:20 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:20.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:20 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:50:20.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:20 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:20.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:20 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-20T08:50:20.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:20 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:21.207 DEBUG:teuthology.orchestra.run.smithi115:mon.b> sudo journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@mon.b.service 2024-03-20T08:50:21.209 DEBUG:teuthology.orchestra.run.smithi122:mon.c> sudo journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@mon.c.service 2024-03-20T08:50:21.211 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-20T08:50:21.212 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-20T08:50:21.241 INFO:journalctl@ceph.mon.c.smithi122.stdout:-- Logs begin at Wed 2024-03-20 08:37:02 UTC. -- 2024-03-20T08:50:21.246 INFO:journalctl@ceph.mon.b.smithi115.stdout:-- Logs begin at Wed 2024-03-20 08:37:21 UTC. -- 2024-03-20T08:50:21.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:21 smithi026 ceph-mon[68309]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi026:172.21.15.26=a;smithi115:172.21.15.115=b;smithi122:172.21.15.122=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:50:21.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:21 smithi026 ceph-mon[68309]: Saving service mon spec with placement smithi026:172.21.15.26=a;smithi115:172.21.15.115=b;smithi122:172.21.15.122=c;count:3 2024-03-20T08:50:21.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:21 smithi026 ceph-mon[68309]: Deploying daemon mon.c on smithi122 2024-03-20T08:50:22.010 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-20T08:50:22.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:22 smithi026 ceph-mon[68309]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:24.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:24 smithi026 ceph-mon[68309]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:25.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:25 smithi122 systemd[1]: Starting Ceph mon.c for 852fbb68-e696-11ee-95c9-87774f69a715... 2024-03-20T08:50:25.499 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:50:25.499 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"852fbb68-e696-11ee-95c9-87774f69a715","modified":"2024-03-20T08:48:32.007170Z","created":"2024-03-20T08:48:32.007170Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-20T08:50:25.501 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-03-20T08:50:25.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:25 smithi026 ceph-mon[68309]: from='client.? 172.21.15.122:0/1839444606' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-20T08:50:26.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: set uid:gid to 167:167 (ceph:ceph) 2024-03-20T08:50:26.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: ceph version 19.0.0-2153-ge7e80a6a (e7e80a6ac9849f0f23432d0d9832f096730308ae) squid (dev), process ceph-mon, pid 7 2024-03-20T08:50:26.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: pidfile_write: ignore empty --pid-file 2024-03-20T08:50:26.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: load: jerasure load: lrc 2024-03-20T08:50:26.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: RocksDB version: 7.9.2 2024-03-20T08:50:26.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Git sha 0 2024-03-20T08:50:26.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Compile date 2024-03-18 19:18:37 2024-03-20T08:50:26.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: DB SUMMARY 2024-03-20T08:50:26.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: DB Session ID: MW4KDLQEPHAV2ZEFI9D1 2024-03-20T08:50:26.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: CURRENT file: CURRENT 2024-03-20T08:50:26.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: IDENTITY file: IDENTITY 2024-03-20T08:50:26.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-20T08:50:26.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-20T08:50:26.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-20T08:50:26.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.error_if_exists: 0 2024-03-20T08:50:26.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.create_if_missing: 0 2024-03-20T08:50:26.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.paranoid_checks: 1 2024-03-20T08:50:26.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-20T08:50:26.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-20T08:50:26.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-20T08:50:26.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.env: 0x55faab1ccb20 2024-03-20T08:50:26.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.fs: PosixFileSystem 2024-03-20T08:50:26.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.info_log: 0x55faac1d9920 2024-03-20T08:50:26.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.max_file_opening_threads: 16 2024-03-20T08:50:26.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.statistics: (nil) 2024-03-20T08:50:26.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.use_fsync: 0 2024-03-20T08:50:26.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.max_log_file_size: 0 2024-03-20T08:50:26.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-20T08:50:26.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-20T08:50:26.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.keep_log_file_num: 1000 2024-03-20T08:50:26.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.recycle_log_file_num: 0 2024-03-20T08:50:26.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.allow_fallocate: 1 2024-03-20T08:50:26.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.allow_mmap_reads: 0 2024-03-20T08:50:26.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.allow_mmap_writes: 0 2024-03-20T08:50:26.452 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.use_direct_reads: 0 2024-03-20T08:50:26.452 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-20T08:50:26.452 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.create_missing_column_families: 0 2024-03-20T08:50:26.452 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.db_log_dir: 2024-03-20T08:50:26.452 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.wal_dir: 2024-03-20T08:50:26.452 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-20T08:50:26.452 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-20T08:50:26.452 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-20T08:50:26.452 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-20T08:50:26.452 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-20T08:50:26.452 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-20T08:50:26.452 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.advise_random_on_open: 1 2024-03-20T08:50:26.452 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.db_write_buffer_size: 0 2024-03-20T08:50:26.453 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.write_buffer_manager: 0x55faac1ac1e0 2024-03-20T08:50:26.453 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-20T08:50:26.453 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-20T08:50:26.453 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-20T08:50:26.453 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.rate_limiter: (nil) 2024-03-20T08:50:26.453 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-20T08:50:26.453 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.wal_recovery_mode: 2 2024-03-20T08:50:26.453 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.enable_thread_tracking: 0 2024-03-20T08:50:26.453 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.enable_pipelined_write: 0 2024-03-20T08:50:26.453 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.unordered_write: 0 2024-03-20T08:50:26.453 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-20T08:50:26.453 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-20T08:50:26.453 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-20T08:50:26.453 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-20T08:50:26.454 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.row_cache: None 2024-03-20T08:50:26.454 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.wal_filter: None 2024-03-20T08:50:26.454 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-20T08:50:26.454 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.allow_ingest_behind: 0 2024-03-20T08:50:26.454 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.two_write_queues: 0 2024-03-20T08:50:26.454 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.manual_wal_flush: 0 2024-03-20T08:50:26.454 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.wal_compression: 0 2024-03-20T08:50:26.454 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.atomic_flush: 0 2024-03-20T08:50:26.454 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-20T08:50:26.454 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-20T08:50:26.454 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-20T08:50:26.454 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.log_readahead_size: 0 2024-03-20T08:50:26.454 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-20T08:50:26.455 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.best_efforts_recovery: 0 2024-03-20T08:50:26.455 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-20T08:50:26.455 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-20T08:50:26.455 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.allow_data_in_errors: 0 2024-03-20T08:50:26.455 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.db_host_id: __hostname__ 2024-03-20T08:50:26.455 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-20T08:50:26.455 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.max_background_jobs: 2 2024-03-20T08:50:26.455 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.max_background_compactions: -1 2024-03-20T08:50:26.455 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.max_subcompactions: 1 2024-03-20T08:50:26.455 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-20T08:50:26.455 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-20T08:50:26.455 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-20T08:50:26.455 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.max_total_wal_size: 0 2024-03-20T08:50:26.455 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-20T08:50:26.456 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-20T08:50:26.456 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-20T08:50:26.456 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-20T08:50:26.456 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.max_open_files: -1 2024-03-20T08:50:26.456 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.bytes_per_sync: 0 2024-03-20T08:50:26.456 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-20T08:50:26.456 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-20T08:50:26.456 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.compaction_readahead_size: 0 2024-03-20T08:50:26.456 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.max_background_flushes: -1 2024-03-20T08:50:26.456 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Compression algorithms supported: 2024-03-20T08:50:26.456 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-20T08:50:26.456 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: kZSTD supported: 0 2024-03-20T08:50:26.456 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: kXpressCompression supported: 0 2024-03-20T08:50:26.456 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: kLZ4HCCompression supported: 1 2024-03-20T08:50:26.457 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: kZlibCompression supported: 1 2024-03-20T08:50:26.457 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: kSnappyCompression supported: 1 2024-03-20T08:50:26.457 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: kLZ4Compression supported: 1 2024-03-20T08:50:26.457 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: kBZip2Compression supported: 0 2024-03-20T08:50:26.457 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-20T08:50:26.457 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-20T08:50:26.457 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-20T08:50:26.457 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-20T08:50:26.457 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-20T08:50:26.457 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.merge_operator: 2024-03-20T08:50:26.457 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.compaction_filter: None 2024-03-20T08:50:26.457 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.compaction_filter_factory: None 2024-03-20T08:50:26.457 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.sst_partitioner_factory: None 2024-03-20T08:50:26.457 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-20T08:50:26.458 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-20T08:50:26.458 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55faac1d9a20) 2024-03-20T08:50:26.458 INFO:journalctl@ceph.mon.c.smithi122.stdout: cache_index_and_filter_blocks: 1 2024-03-20T08:50:26.458 INFO:journalctl@ceph.mon.c.smithi122.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-20T08:50:26.458 INFO:journalctl@ceph.mon.c.smithi122.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-20T08:50:26.458 INFO:journalctl@ceph.mon.c.smithi122.stdout: pin_top_level_index_and_filter: 1 2024-03-20T08:50:26.458 INFO:journalctl@ceph.mon.c.smithi122.stdout: index_type: 0 2024-03-20T08:50:26.458 INFO:journalctl@ceph.mon.c.smithi122.stdout: data_block_index_type: 0 2024-03-20T08:50:26.458 INFO:journalctl@ceph.mon.c.smithi122.stdout: index_shortening: 1 2024-03-20T08:50:26.458 INFO:journalctl@ceph.mon.c.smithi122.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-20T08:50:26.458 INFO:journalctl@ceph.mon.c.smithi122.stdout: checksum: 4 2024-03-20T08:50:26.458 INFO:journalctl@ceph.mon.c.smithi122.stdout: no_block_cache: 0 2024-03-20T08:50:26.458 INFO:journalctl@ceph.mon.c.smithi122.stdout: block_cache: 0x55faac1df090 2024-03-20T08:50:26.458 INFO:journalctl@ceph.mon.c.smithi122.stdout: block_cache_name: BinnedLRUCache 2024-03-20T08:50:26.459 INFO:journalctl@ceph.mon.c.smithi122.stdout: block_cache_options: 2024-03-20T08:50:26.459 INFO:journalctl@ceph.mon.c.smithi122.stdout: capacity : 536870912 2024-03-20T08:50:26.459 INFO:journalctl@ceph.mon.c.smithi122.stdout: num_shard_bits : 4 2024-03-20T08:50:26.459 INFO:journalctl@ceph.mon.c.smithi122.stdout: strict_capacity_limit : 0 2024-03-20T08:50:26.459 INFO:journalctl@ceph.mon.c.smithi122.stdout: high_pri_pool_ratio: 0.000 2024-03-20T08:50:26.459 INFO:journalctl@ceph.mon.c.smithi122.stdout: block_cache_compressed: (nil) 2024-03-20T08:50:26.459 INFO:journalctl@ceph.mon.c.smithi122.stdout: persistent_cache: (nil) 2024-03-20T08:50:26.459 INFO:journalctl@ceph.mon.c.smithi122.stdout: block_size: 4096 2024-03-20T08:50:26.459 INFO:journalctl@ceph.mon.c.smithi122.stdout: block_size_deviation: 10 2024-03-20T08:50:26.459 INFO:journalctl@ceph.mon.c.smithi122.stdout: block_restart_interval: 16 2024-03-20T08:50:26.459 INFO:journalctl@ceph.mon.c.smithi122.stdout: index_block_restart_interval: 1 2024-03-20T08:50:26.459 INFO:journalctl@ceph.mon.c.smithi122.stdout: metadata_block_size: 4096 2024-03-20T08:50:26.459 INFO:journalctl@ceph.mon.c.smithi122.stdout: partition_filters: 0 2024-03-20T08:50:26.459 INFO:journalctl@ceph.mon.c.smithi122.stdout: use_delta_encoding: 1 2024-03-20T08:50:26.460 INFO:journalctl@ceph.mon.c.smithi122.stdout: filter_policy: bloomfilter 2024-03-20T08:50:26.460 INFO:journalctl@ceph.mon.c.smithi122.stdout: whole_key_filtering: 1 2024-03-20T08:50:26.460 INFO:journalctl@ceph.mon.c.smithi122.stdout: verify_compression: 0 2024-03-20T08:50:26.460 INFO:journalctl@ceph.mon.c.smithi122.stdout: read_amp_bytes_per_bit: 0 2024-03-20T08:50:26.460 INFO:journalctl@ceph.mon.c.smithi122.stdout: format_version: 5 2024-03-20T08:50:26.460 INFO:journalctl@ceph.mon.c.smithi122.stdout: enable_index_compression: 1 2024-03-20T08:50:26.460 INFO:journalctl@ceph.mon.c.smithi122.stdout: block_align: 0 2024-03-20T08:50:26.460 INFO:journalctl@ceph.mon.c.smithi122.stdout: max_auto_readahead_size: 262144 2024-03-20T08:50:26.460 INFO:journalctl@ceph.mon.c.smithi122.stdout: prepopulate_block_cache: 0 2024-03-20T08:50:26.460 INFO:journalctl@ceph.mon.c.smithi122.stdout: initial_auto_readahead_size: 8192 2024-03-20T08:50:26.460 INFO:journalctl@ceph.mon.c.smithi122.stdout: num_file_reads_for_auto_readahead: 2 2024-03-20T08:50:26.460 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.write_buffer_size: 33554432 2024-03-20T08:50:26.460 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.max_write_buffer_number: 2 2024-03-20T08:50:26.460 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.compression: NoCompression 2024-03-20T08:50:26.461 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.bottommost_compression: Disabled 2024-03-20T08:50:26.461 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.prefix_extractor: nullptr 2024-03-20T08:50:26.461 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-20T08:50:26.461 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.num_levels: 7 2024-03-20T08:50:26.461 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-20T08:50:26.461 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-20T08:50:26.461 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-20T08:50:26.461 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-20T08:50:26.461 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-20T08:50:26.461 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-20T08:50:26.461 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-20T08:50:26.461 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-20T08:50:26.461 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-20T08:50:26.461 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-20T08:50:26.462 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-20T08:50:26.462 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-20T08:50:26.462 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-20T08:50:26.462 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.compression_opts.level: 32767 2024-03-20T08:50:26.462 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.compression_opts.strategy: 0 2024-03-20T08:50:26.462 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-20T08:50:26.462 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-20T08:50:26.462 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-20T08:50:26.462 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-20T08:50:26.462 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.compression_opts.enabled: false 2024-03-20T08:50:26.462 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-20T08:50:26.462 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-20T08:50:26.462 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-20T08:50:26.462 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-20T08:50:26.463 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.target_file_size_base: 67108864 2024-03-20T08:50:26.463 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-20T08:50:26.463 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-20T08:50:26.463 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-20T08:50:26.463 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-20T08:50:26.463 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-20T08:50:26.463 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-20T08:50:26.463 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-20T08:50:26.463 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-20T08:50:26.463 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-20T08:50:26.463 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-20T08:50:26.463 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-20T08:50:26.463 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-20T08:50:26.463 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-20T08:50:26.464 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-20T08:50:26.464 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.arena_block_size: 1048576 2024-03-20T08:50:26.464 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-20T08:50:26.464 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-20T08:50:26.464 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.disable_auto_compactions: 0 2024-03-20T08:50:26.464 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-20T08:50:26.464 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-20T08:50:26.464 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-20T08:50:26.464 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-20T08:50:26.464 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-20T08:50:26.464 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-20T08:50:26.464 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-20T08:50:26.464 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-20T08:50:26.464 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-20T08:50:26.465 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-20T08:50:26.465 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.table_properties_collectors: 2024-03-20T08:50:26.465 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.inplace_update_support: 0 2024-03-20T08:50:26.465 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-20T08:50:26.465 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-20T08:50:26.465 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-20T08:50:26.465 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-20T08:50:26.465 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.bloom_locality: 0 2024-03-20T08:50:26.465 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.max_successive_merges: 0 2024-03-20T08:50:26.465 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-20T08:50:26.465 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.paranoid_file_checks: 0 2024-03-20T08:50:26.465 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.force_consistency_checks: 1 2024-03-20T08:50:26.465 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.report_bg_io_stats: 0 2024-03-20T08:50:26.465 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.ttl: 2592000 2024-03-20T08:50:26.465 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-20T08:50:26.466 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-20T08:50:26.466 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-20T08:50:26.466 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.enable_blob_files: false 2024-03-20T08:50:26.466 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.min_blob_size: 0 2024-03-20T08:50:26.466 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.blob_file_size: 268435456 2024-03-20T08:50:26.466 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-20T08:50:26.466 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-20T08:50:26.466 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-20T08:50:26.466 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-20T08:50:26.466 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-20T08:50:26.466 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.blob_file_starting_level: 0 2024-03-20T08:50:26.466 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-20T08:50:26.466 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-20T08:50:26.467 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-20T08:50:26.467 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 8b3abfba-b5c8-4c06-babe-3f8dc63c1f60 2024-03-20T08:50:26.467 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710924626378853, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-20T08:50:26.467 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-20T08:50:26.467 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710924626379272, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710924626, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "8b3abfba-b5c8-4c06-babe-3f8dc63c1f60", "db_session_id": "MW4KDLQEPHAV2ZEFI9D1", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-20T08:50:26.467 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710924626379357, "job": 1, "event": "recovery_finished"} 2024-03-20T08:50:26.467 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-20T08:50:26.467 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-20T08:50:26.467 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55faac2d2000 2024-03-20T08:50:26.467 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: DB pointer 0x55faac2bc000 2024-03-20T08:50:26.467 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-20T08:50:26.467 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-20T08:50:26.467 INFO:journalctl@ceph.mon.c.smithi122.stdout: ** DB Stats ** 2024-03-20T08:50:26.467 INFO:journalctl@ceph.mon.c.smithi122.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-20T08:50:26.468 INFO:journalctl@ceph.mon.c.smithi122.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-20T08:50:26.468 INFO:journalctl@ceph.mon.c.smithi122.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-20T08:50:26.468 INFO:journalctl@ceph.mon.c.smithi122.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-20T08:50:26.468 INFO:journalctl@ceph.mon.c.smithi122.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-20T08:50:26.468 INFO:journalctl@ceph.mon.c.smithi122.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-20T08:50:26.468 INFO:journalctl@ceph.mon.c.smithi122.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-20T08:50:26.468 INFO:journalctl@ceph.mon.c.smithi122.stdout: 2024-03-20T08:50:26.468 INFO:journalctl@ceph.mon.c.smithi122.stdout: ** Compaction Stats [default] ** 2024-03-20T08:50:26.468 INFO:journalctl@ceph.mon.c.smithi122.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-20T08:50:26.468 INFO:journalctl@ceph.mon.c.smithi122.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-20T08:50:26.468 INFO:journalctl@ceph.mon.c.smithi122.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-20T08:50:26.468 INFO:journalctl@ceph.mon.c.smithi122.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-20T08:50:26.468 INFO:journalctl@ceph.mon.c.smithi122.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-20T08:50:26.468 INFO:journalctl@ceph.mon.c.smithi122.stdout: 2024-03-20T08:50:26.469 INFO:journalctl@ceph.mon.c.smithi122.stdout: ** Compaction Stats [default] ** 2024-03-20T08:50:26.469 INFO:journalctl@ceph.mon.c.smithi122.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-20T08:50:26.469 INFO:journalctl@ceph.mon.c.smithi122.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-20T08:50:26.469 INFO:journalctl@ceph.mon.c.smithi122.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-20T08:50:26.469 INFO:journalctl@ceph.mon.c.smithi122.stdout: 2024-03-20T08:50:26.469 INFO:journalctl@ceph.mon.c.smithi122.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-20T08:50:26.469 INFO:journalctl@ceph.mon.c.smithi122.stdout: 2024-03-20T08:50:26.469 INFO:journalctl@ceph.mon.c.smithi122.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-20T08:50:26.469 INFO:journalctl@ceph.mon.c.smithi122.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-20T08:50:26.469 INFO:journalctl@ceph.mon.c.smithi122.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-20T08:50:26.469 INFO:journalctl@ceph.mon.c.smithi122.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-20T08:50:26.469 INFO:journalctl@ceph.mon.c.smithi122.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-20T08:50:26.469 INFO:journalctl@ceph.mon.c.smithi122.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-20T08:50:26.469 INFO:journalctl@ceph.mon.c.smithi122.stdout: Cumulative compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-20T08:50:26.470 INFO:journalctl@ceph.mon.c.smithi122.stdout: Interval compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-20T08:50:26.470 INFO:journalctl@ceph.mon.c.smithi122.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-20T08:50:26.470 INFO:journalctl@ceph.mon.c.smithi122.stdout: Block cache BinnedLRUCache@0x55faac1df090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-20T08:50:26.470 INFO:journalctl@ceph.mon.c.smithi122.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-20T08:50:26.470 INFO:journalctl@ceph.mon.c.smithi122.stdout: 2024-03-20T08:50:26.470 INFO:journalctl@ceph.mon.c.smithi122.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-20T08:50:26.470 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-20T08:50:26.470 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: using public_addr v2:172.21.15.122:0/0 -> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-03-20T08:50:26.470 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: starting mon.c rank -1 at public addrs [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] at bind addrs [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 852fbb68-e696-11ee-95c9-87774f69a715 2024-03-20T08:50:26.470 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: mon.c@-1(???) e0 preinit fsid 852fbb68-e696-11ee-95c9-87774f69a715 2024-03-20T08:50:26.470 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #13. Immutable memtables: 0. 2024-03-20T08:50:26.470 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: (Original Log Time 2024/03/20-08:50:26.397176) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2024-03-20T08:50:26.470 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: [db/flush_job.cc:856] [default] [JOB 3] Flushing memtable with next log file: 13 2024-03-20T08:50:26.471 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710924626397294, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 1117, "num_deletes": 20, "total_data_size": 2270937, "memory_usage": 2292592, "flush_reason": "Manual Compaction"} 2024-03-20T08:50:26.471 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: [db/flush_job.cc:885] [default] [JOB 3] Level-0 flush table #14: started 2024-03-20T08:50:26.471 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: mon.c@-1(synchronizing).mds e1 new map 2024-03-20T08:50:26.471 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-20T08:50:26.471 INFO:journalctl@ceph.mon.c.smithi122.stdout: e1 2024-03-20T08:50:26.471 INFO:journalctl@ceph.mon.c.smithi122.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-20T08:50:26.471 INFO:journalctl@ceph.mon.c.smithi122.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-20T08:50:26.471 INFO:journalctl@ceph.mon.c.smithi122.stdout: legacy client fscid: -1 2024-03-20T08:50:26.471 INFO:journalctl@ceph.mon.c.smithi122.stdout: 2024-03-20T08:50:26.471 INFO:journalctl@ceph.mon.c.smithi122.stdout: No filesystems configured 2024-03-20T08:50:26.471 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-20T08:50:26.471 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-20T08:50:26.471 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-20T08:50:26.471 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-20T08:50:26.472 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-20T08:50:26.472 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-20T08:50:26.472 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-20T08:50:26.472 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-20T08:50:26.472 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-20T08:50:26.472 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-20T08:50:26.472 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: Updating smithi115:/etc/ceph/ceph.conf 2024-03-20T08:50:26.472 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:26.472 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:26.472 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:26.472 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:26.472 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: Updating smithi115:/var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-20T08:50:26.472 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2024-03-20T08:50:26.472 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: Updating smithi115:/var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-20T08:50:26.472 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi122", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:50:26.473 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: Deploying cephadm binary to smithi122 2024-03-20T08:50:26.473 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:26.473 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:26.473 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:26.473 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:50:26.473 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:26.473 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:26.473 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: Added host smithi122 2024-03-20T08:50:26.473 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:26.473 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:26.473 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:26.473 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T08:50:26.473 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:26.473 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:26.474 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:26.474 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:26.474 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi122", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:50:26.474 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:26.474 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:50:26.474 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:50:26.474 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:50:26.474 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:26.474 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:26.474 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:26.474 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: Updating smithi122:/etc/ceph/ceph.conf 2024-03-20T08:50:26.474 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:26.474 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: Updating smithi122:/var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-20T08:50:26.474 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-03-20T08:50:26.475 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: Updating smithi122:/var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-20T08:50:26.475 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:50:26.475 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:26.475 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:50:26.475 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:26.475 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/969314353' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-20T08:50:26.475 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:26.475 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/969314353' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-20T08:50:26.475 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: osdmap e4: 0 total, 0 up, 0 in 2024-03-20T08:50:26.475 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:26.475 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:26.475 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:50:26.475 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:26.475 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:50:26.476 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:26.476 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-20T08:50:26.476 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:26.476 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi026:172.21.15.26=a;smithi115:172.21.15.115=b;smithi122:172.21.15.122=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:50:26.476 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: Saving service mon spec with placement smithi026:172.21.15.26=a;smithi115:172.21.15.115=b;smithi122:172.21.15.122=c;count:3 2024-03-20T08:50:26.476 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: Deploying daemon mon.c on smithi122 2024-03-20T08:50:26.476 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:26.476 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:26.476 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: from='client.? 172.21.15.122:0/1839444606' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-20T08:50:26.476 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-20T08:50:26.477 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710924626400816, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 14, "file_size": 1999227, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 6, "largest_seqno": 1122, "table_properties": {"data_size": 1993475, "index_size": 2598, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 2117, "raw_key_size": 26035, "raw_average_key_size": 31, "raw_value_size": 1975757, "raw_average_value_size": 2369, "num_data_blocks": 122, "num_entries": 834, "num_filter_entries": 834, "num_deletions": 20, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710924626, "oldest_key_time": 1710924626, "file_creation_time": 1710924626, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "8b3abfba-b5c8-4c06-babe-3f8dc63c1f60", "db_session_id": "MW4KDLQEPHAV2ZEFI9D1", "orig_file_number": 14, "seqno_to_time_mapping": "N/A"}} 2024-03-20T08:50:26.477 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: [db/flush_job.cc:1019] [default] [JOB 3] Flush lasted 3660 microseconds, and 2746 cpu microseconds. 2024-03-20T08:50:26.477 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: (Original Log Time 2024/03/20-08:50:26.400838) [db/flush_job.cc:967] [default] [JOB 3] Level-0 flush table #14: 1999227 bytes OK 2024-03-20T08:50:26.477 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: (Original Log Time 2024/03/20-08:50:26.400852) [db/memtable_list.cc:519] [default] Level-0 commit table #14 started 2024-03-20T08:50:26.477 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: (Original Log Time 2024/03/20-08:50:26.401231) [db/memtable_list.cc:722] [default] Level-0 commit table #14: memtable #1 done 2024-03-20T08:50:26.477 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: (Original Log Time 2024/03/20-08:50:26.401243) EVENT_LOG_v1 {"time_micros": 1710924626401239, "job": 3, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [2, 0, 0, 0, 0, 0, 0], "immutable_memtables": 0} 2024-03-20T08:50:26.477 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: (Original Log Time 2024/03/20-08:50:26.401250) [db/db_impl/db_impl_compaction_flush.cc:299] [default] Level summary: files[2 0 0 0 0 0 0] max score 0.50 2024-03-20T08:50:26.477 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 3] Try to delete WAL files size 2263146, prev total WAL file size 2263369, number of live WAL files 2. 2024-03-20T08:50:26.477 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-20T08:50:26.477 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: (Original Log Time 2024/03/20-08:50:26.401567) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '6B760030' seq:72057594037927935, type:22 .. '6B76003131' seq:0, type:0; will stop at (end) 2024-03-20T08:50:26.477 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 4] Compacting 2@0 files to L6, score -1.00 2024-03-20T08:50:26.477 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 3 Base level 0, inputs: [14(1952KB) 8(1622B)] 2024-03-20T08:50:26.477 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 ceph-mon[73495]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710924626401612, "job": 4, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [14, 8], "score": -1, "input_data_size": 2000849, "oldest_snapshot_seqno": -1} 2024-03-20T08:50:26.477 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 bash[73433]: 9f30bd8bc39c7d84c977c8bca5834c2b5166333fdaf61eb2d00ffc5c7e6d6a88 2024-03-20T08:50:26.478 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:26 smithi122 systemd[1]: Started Ceph mon.c for 852fbb68-e696-11ee-95c9-87774f69a715. 2024-03-20T08:50:27.224 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-20T08:50:27.224 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-20T08:50:27.497 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T08:50:31.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:31 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:50:31.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:31 smithi122 ceph-mon[73495]: mon.a calling monitor election 2024-03-20T08:50:31.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:31 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:50:31.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:31 smithi122 ceph-mon[73495]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:31.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:31 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:50:31.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:31 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:50:31.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:31 smithi122 ceph-mon[73495]: mon.c calling monitor election 2024-03-20T08:50:31.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:31 smithi122 ceph-mon[73495]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:31.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:31 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:50:31.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:31 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:50:31.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:31 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:50:31.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:31 smithi122 ceph-mon[73495]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-20T08:50:31.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:31 smithi122 ceph-mon[73495]: monmap e2: 2 mons at {a=[v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0],c=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-20T08:50:31.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:31 smithi122 ceph-mon[73495]: fsmap 2024-03-20T08:50:31.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:31 smithi122 ceph-mon[73495]: osdmap e4: 0 total, 0 up, 0 in 2024-03-20T08:50:31.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:31 smithi122 ceph-mon[73495]: mgrmap e13: y(active, since 56s) 2024-03-20T08:50:31.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:31 smithi122 ceph-mon[73495]: overall HEALTH_OK 2024-03-20T08:50:31.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:31 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:31.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:31 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:31.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:31 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:31.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:31 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-20T08:50:31.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:31 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:31.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:31 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:50:31.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:31 smithi026 ceph-mon[68309]: mon.a calling monitor election 2024-03-20T08:50:31.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:31 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:50:31.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:31 smithi026 ceph-mon[68309]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:31.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:31 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:50:31.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:31 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:50:31.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:31 smithi026 ceph-mon[68309]: mon.c calling monitor election 2024-03-20T08:50:31.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:31 smithi026 ceph-mon[68309]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:31.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:31 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:50:31.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:31 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:50:31.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:31 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:50:31.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:31 smithi026 ceph-mon[68309]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-20T08:50:31.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:31 smithi026 ceph-mon[68309]: monmap e2: 2 mons at {a=[v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0],c=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-20T08:50:31.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:31 smithi026 ceph-mon[68309]: fsmap 2024-03-20T08:50:31.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:31 smithi026 ceph-mon[68309]: osdmap e4: 0 total, 0 up, 0 in 2024-03-20T08:50:31.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:31 smithi026 ceph-mon[68309]: mgrmap e13: y(active, since 56s) 2024-03-20T08:50:31.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:31 smithi026 ceph-mon[68309]: overall HEALTH_OK 2024-03-20T08:50:31.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:31 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:31.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:31 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:31.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:31 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:31.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:31 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-20T08:50:31.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:31 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:32.624 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:50:32.625 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":2,"fsid":"852fbb68-e696-11ee-95c9-87774f69a715","modified":"2024-03-20T08:50:26.403835Z","created":"2024-03-20T08:48:32.007170Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-20T08:50:32.627 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 2 2024-03-20T08:50:32.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:32 smithi122 ceph-mon[73495]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:32.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:32 smithi122 ceph-mon[73495]: Deploying daemon mon.b on smithi115 2024-03-20T08:50:32.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:32 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:50:32.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:32 smithi026 ceph-mon[68309]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:32.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:32 smithi026 ceph-mon[68309]: Deploying daemon mon.b on smithi115 2024-03-20T08:50:32.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:32 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:50:33.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:33 smithi026 ceph-mon[68309]: from='client.? 172.21.15.122:0/734152903' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-20T08:50:33.720 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 08:50:33 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T08:50:33.403+0000 7fe5fc38b700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-20T08:50:33.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:33 smithi122 ceph-mon[73495]: from='client.? 172.21.15.122:0/734152903' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-20T08:50:34.279 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-20T08:50:34.279 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-20T08:50:34.550 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T08:50:34.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:34 smithi122 ceph-mon[73495]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:34.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:34 smithi115 systemd[1]: Starting Ceph mon.b for 852fbb68-e696-11ee-95c9-87774f69a715... 2024-03-20T08:50:34.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:34 smithi026 ceph-mon[68309]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:35.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: set uid:gid to 167:167 (ceph:ceph) 2024-03-20T08:50:35.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: ceph version 19.0.0-2153-ge7e80a6a (e7e80a6ac9849f0f23432d0d9832f096730308ae) squid (dev), process ceph-mon, pid 8 2024-03-20T08:50:35.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: pidfile_write: ignore empty --pid-file 2024-03-20T08:50:35.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: load: jerasure load: lrc 2024-03-20T08:50:35.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: RocksDB version: 7.9.2 2024-03-20T08:50:35.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Git sha 0 2024-03-20T08:50:35.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Compile date 2024-03-18 19:18:37 2024-03-20T08:50:35.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: DB SUMMARY 2024-03-20T08:50:35.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: DB Session ID: E6PM4XZ1I71S9YQTGWJS 2024-03-20T08:50:35.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: CURRENT file: CURRENT 2024-03-20T08:50:35.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: IDENTITY file: IDENTITY 2024-03-20T08:50:35.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-20T08:50:35.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-20T08:50:35.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-03-20T08:50:35.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.error_if_exists: 0 2024-03-20T08:50:35.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.create_if_missing: 0 2024-03-20T08:50:35.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.paranoid_checks: 1 2024-03-20T08:50:35.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-20T08:50:35.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-20T08:50:35.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-20T08:50:35.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.env: 0x563a6682ab20 2024-03-20T08:50:35.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.fs: PosixFileSystem 2024-03-20T08:50:35.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.info_log: 0x563a67f87920 2024-03-20T08:50:35.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.max_file_opening_threads: 16 2024-03-20T08:50:35.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.statistics: (nil) 2024-03-20T08:50:35.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.use_fsync: 0 2024-03-20T08:50:35.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.max_log_file_size: 0 2024-03-20T08:50:35.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-20T08:50:35.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-20T08:50:35.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.keep_log_file_num: 1000 2024-03-20T08:50:35.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.recycle_log_file_num: 0 2024-03-20T08:50:35.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.allow_fallocate: 1 2024-03-20T08:50:35.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.allow_mmap_reads: 0 2024-03-20T08:50:35.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.allow_mmap_writes: 0 2024-03-20T08:50:35.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.use_direct_reads: 0 2024-03-20T08:50:35.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-20T08:50:35.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.create_missing_column_families: 0 2024-03-20T08:50:35.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.db_log_dir: 2024-03-20T08:50:35.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.wal_dir: 2024-03-20T08:50:35.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-20T08:50:35.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-20T08:50:35.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-20T08:50:35.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-20T08:50:35.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-20T08:50:35.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-20T08:50:35.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.advise_random_on_open: 1 2024-03-20T08:50:35.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.db_write_buffer_size: 0 2024-03-20T08:50:35.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.write_buffer_manager: 0x563a67f5a1e0 2024-03-20T08:50:35.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-20T08:50:35.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-20T08:50:35.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-20T08:50:35.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.rate_limiter: (nil) 2024-03-20T08:50:35.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-20T08:50:35.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.wal_recovery_mode: 2 2024-03-20T08:50:35.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.enable_thread_tracking: 0 2024-03-20T08:50:35.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.enable_pipelined_write: 0 2024-03-20T08:50:35.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.unordered_write: 0 2024-03-20T08:50:35.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-20T08:50:35.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-20T08:50:35.412 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-20T08:50:35.412 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-20T08:50:35.412 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.row_cache: None 2024-03-20T08:50:35.412 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.wal_filter: None 2024-03-20T08:50:35.412 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-20T08:50:35.412 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.allow_ingest_behind: 0 2024-03-20T08:50:35.412 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.two_write_queues: 0 2024-03-20T08:50:35.412 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.manual_wal_flush: 0 2024-03-20T08:50:35.412 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.wal_compression: 0 2024-03-20T08:50:35.412 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.atomic_flush: 0 2024-03-20T08:50:35.412 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-20T08:50:35.413 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-20T08:50:35.413 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-20T08:50:35.413 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.log_readahead_size: 0 2024-03-20T08:50:35.413 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-20T08:50:35.413 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.best_efforts_recovery: 0 2024-03-20T08:50:35.413 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-20T08:50:35.413 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-20T08:50:35.413 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.allow_data_in_errors: 0 2024-03-20T08:50:35.413 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.db_host_id: __hostname__ 2024-03-20T08:50:35.413 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-20T08:50:35.413 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.max_background_jobs: 2 2024-03-20T08:50:35.414 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.max_background_compactions: -1 2024-03-20T08:50:35.414 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.max_subcompactions: 1 2024-03-20T08:50:35.414 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-20T08:50:35.414 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-20T08:50:35.414 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-20T08:50:35.414 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.max_total_wal_size: 0 2024-03-20T08:50:35.414 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-20T08:50:35.414 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-20T08:50:35.414 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-20T08:50:35.414 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-20T08:50:35.415 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.max_open_files: -1 2024-03-20T08:50:35.415 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.bytes_per_sync: 0 2024-03-20T08:50:35.415 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-20T08:50:35.415 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-20T08:50:35.415 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.compaction_readahead_size: 0 2024-03-20T08:50:35.415 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.max_background_flushes: -1 2024-03-20T08:50:35.415 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Compression algorithms supported: 2024-03-20T08:50:35.415 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-20T08:50:35.416 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: kZSTD supported: 0 2024-03-20T08:50:35.416 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: kXpressCompression supported: 0 2024-03-20T08:50:35.416 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: kLZ4HCCompression supported: 1 2024-03-20T08:50:35.416 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: kZlibCompression supported: 1 2024-03-20T08:50:35.416 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: kSnappyCompression supported: 1 2024-03-20T08:50:35.416 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: kLZ4Compression supported: 1 2024-03-20T08:50:35.416 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: kBZip2Compression supported: 0 2024-03-20T08:50:35.416 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-20T08:50:35.416 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-20T08:50:35.416 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-20T08:50:35.416 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-20T08:50:35.416 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-20T08:50:35.416 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.merge_operator: 2024-03-20T08:50:35.417 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.compaction_filter: None 2024-03-20T08:50:35.417 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.compaction_filter_factory: None 2024-03-20T08:50:35.417 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.sst_partitioner_factory: None 2024-03-20T08:50:35.417 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-20T08:50:35.417 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-20T08:50:35.417 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563a67f87a20) 2024-03-20T08:50:35.417 INFO:journalctl@ceph.mon.b.smithi115.stdout: cache_index_and_filter_blocks: 1 2024-03-20T08:50:35.417 INFO:journalctl@ceph.mon.b.smithi115.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-20T08:50:35.417 INFO:journalctl@ceph.mon.b.smithi115.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-20T08:50:35.417 INFO:journalctl@ceph.mon.b.smithi115.stdout: pin_top_level_index_and_filter: 1 2024-03-20T08:50:35.417 INFO:journalctl@ceph.mon.b.smithi115.stdout: index_type: 0 2024-03-20T08:50:35.417 INFO:journalctl@ceph.mon.b.smithi115.stdout: data_block_index_type: 0 2024-03-20T08:50:35.417 INFO:journalctl@ceph.mon.b.smithi115.stdout: index_shortening: 1 2024-03-20T08:50:35.417 INFO:journalctl@ceph.mon.b.smithi115.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-20T08:50:35.418 INFO:journalctl@ceph.mon.b.smithi115.stdout: checksum: 4 2024-03-20T08:50:35.418 INFO:journalctl@ceph.mon.b.smithi115.stdout: no_block_cache: 0 2024-03-20T08:50:35.418 INFO:journalctl@ceph.mon.b.smithi115.stdout: block_cache: 0x563a67f8d090 2024-03-20T08:50:35.418 INFO:journalctl@ceph.mon.b.smithi115.stdout: block_cache_name: BinnedLRUCache 2024-03-20T08:50:35.418 INFO:journalctl@ceph.mon.b.smithi115.stdout: block_cache_options: 2024-03-20T08:50:35.418 INFO:journalctl@ceph.mon.b.smithi115.stdout: capacity : 536870912 2024-03-20T08:50:35.418 INFO:journalctl@ceph.mon.b.smithi115.stdout: num_shard_bits : 4 2024-03-20T08:50:35.418 INFO:journalctl@ceph.mon.b.smithi115.stdout: strict_capacity_limit : 0 2024-03-20T08:50:35.418 INFO:journalctl@ceph.mon.b.smithi115.stdout: high_pri_pool_ratio: 0.000 2024-03-20T08:50:35.418 INFO:journalctl@ceph.mon.b.smithi115.stdout: block_cache_compressed: (nil) 2024-03-20T08:50:35.418 INFO:journalctl@ceph.mon.b.smithi115.stdout: persistent_cache: (nil) 2024-03-20T08:50:35.418 INFO:journalctl@ceph.mon.b.smithi115.stdout: block_size: 4096 2024-03-20T08:50:35.418 INFO:journalctl@ceph.mon.b.smithi115.stdout: block_size_deviation: 10 2024-03-20T08:50:35.418 INFO:journalctl@ceph.mon.b.smithi115.stdout: block_restart_interval: 16 2024-03-20T08:50:35.419 INFO:journalctl@ceph.mon.b.smithi115.stdout: index_block_restart_interval: 1 2024-03-20T08:50:35.419 INFO:journalctl@ceph.mon.b.smithi115.stdout: metadata_block_size: 4096 2024-03-20T08:50:35.419 INFO:journalctl@ceph.mon.b.smithi115.stdout: partition_filters: 0 2024-03-20T08:50:35.419 INFO:journalctl@ceph.mon.b.smithi115.stdout: use_delta_encoding: 1 2024-03-20T08:50:35.419 INFO:journalctl@ceph.mon.b.smithi115.stdout: filter_policy: bloomfilter 2024-03-20T08:50:35.419 INFO:journalctl@ceph.mon.b.smithi115.stdout: whole_key_filtering: 1 2024-03-20T08:50:35.419 INFO:journalctl@ceph.mon.b.smithi115.stdout: verify_compression: 0 2024-03-20T08:50:35.419 INFO:journalctl@ceph.mon.b.smithi115.stdout: read_amp_bytes_per_bit: 0 2024-03-20T08:50:35.419 INFO:journalctl@ceph.mon.b.smithi115.stdout: format_version: 5 2024-03-20T08:50:35.419 INFO:journalctl@ceph.mon.b.smithi115.stdout: enable_index_compression: 1 2024-03-20T08:50:35.419 INFO:journalctl@ceph.mon.b.smithi115.stdout: block_align: 0 2024-03-20T08:50:35.419 INFO:journalctl@ceph.mon.b.smithi115.stdout: max_auto_readahead_size: 262144 2024-03-20T08:50:35.420 INFO:journalctl@ceph.mon.b.smithi115.stdout: prepopulate_block_cache: 0 2024-03-20T08:50:35.420 INFO:journalctl@ceph.mon.b.smithi115.stdout: initial_auto_readahead_size: 8192 2024-03-20T08:50:35.420 INFO:journalctl@ceph.mon.b.smithi115.stdout: num_file_reads_for_auto_readahead: 2 2024-03-20T08:50:35.420 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.write_buffer_size: 33554432 2024-03-20T08:50:35.420 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.max_write_buffer_number: 2 2024-03-20T08:50:35.420 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.compression: NoCompression 2024-03-20T08:50:35.420 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.bottommost_compression: Disabled 2024-03-20T08:50:35.420 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.prefix_extractor: nullptr 2024-03-20T08:50:35.421 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-20T08:50:35.421 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.num_levels: 7 2024-03-20T08:50:35.421 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-20T08:50:35.421 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-20T08:50:35.421 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-20T08:50:35.421 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-20T08:50:35.421 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-20T08:50:35.421 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-20T08:50:35.421 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-20T08:50:35.421 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-20T08:50:35.421 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-20T08:50:35.421 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-20T08:50:35.421 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-20T08:50:35.421 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-20T08:50:35.422 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-20T08:50:35.422 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.compression_opts.level: 32767 2024-03-20T08:50:35.422 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.compression_opts.strategy: 0 2024-03-20T08:50:35.422 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-20T08:50:35.422 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-20T08:50:35.422 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-20T08:50:35.422 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-20T08:50:35.422 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.compression_opts.enabled: false 2024-03-20T08:50:35.422 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-20T08:50:35.423 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-20T08:50:35.423 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-20T08:50:35.423 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-20T08:50:35.423 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.target_file_size_base: 67108864 2024-03-20T08:50:35.423 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-20T08:50:35.423 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-20T08:50:35.423 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-20T08:50:35.423 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-20T08:50:35.423 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-20T08:50:35.423 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-20T08:50:35.423 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-20T08:50:35.423 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-20T08:50:35.423 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-20T08:50:35.424 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-20T08:50:35.424 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-20T08:50:35.424 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-20T08:50:35.424 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-20T08:50:35.424 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-20T08:50:35.424 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.arena_block_size: 1048576 2024-03-20T08:50:35.424 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-20T08:50:35.424 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-20T08:50:35.424 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.disable_auto_compactions: 0 2024-03-20T08:50:35.424 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-20T08:50:35.425 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-20T08:50:35.425 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-20T08:50:35.426 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-20T08:50:35.426 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-20T08:50:35.426 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-20T08:50:35.426 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-20T08:50:35.427 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-20T08:50:35.427 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-20T08:50:35.427 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-20T08:50:35.427 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.table_properties_collectors: 2024-03-20T08:50:35.427 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.inplace_update_support: 0 2024-03-20T08:50:35.427 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-20T08:50:35.427 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-20T08:50:35.427 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-20T08:50:35.427 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-20T08:50:35.427 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.bloom_locality: 0 2024-03-20T08:50:35.427 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.max_successive_merges: 0 2024-03-20T08:50:35.428 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-20T08:50:35.428 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.paranoid_file_checks: 0 2024-03-20T08:50:35.428 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.force_consistency_checks: 1 2024-03-20T08:50:35.428 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.report_bg_io_stats: 0 2024-03-20T08:50:35.428 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.ttl: 2592000 2024-03-20T08:50:35.428 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-20T08:50:35.428 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-20T08:50:35.428 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-20T08:50:35.429 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.enable_blob_files: false 2024-03-20T08:50:35.429 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.min_blob_size: 0 2024-03-20T08:50:35.429 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.blob_file_size: 268435456 2024-03-20T08:50:35.429 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-20T08:50:35.429 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-20T08:50:35.429 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-20T08:50:35.429 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-20T08:50:35.429 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-20T08:50:35.429 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.blob_file_starting_level: 0 2024-03-20T08:50:35.429 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-20T08:50:35.429 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-20T08:50:35.429 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-20T08:50:35.429 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 91dc59ff-c51c-4eb2-b55d-b517d9928716 2024-03-20T08:50:35.430 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710924635100297, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-20T08:50:35.430 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-20T08:50:35.430 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710924635100845, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710924635, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "91dc59ff-c51c-4eb2-b55d-b517d9928716", "db_session_id": "E6PM4XZ1I71S9YQTGWJS", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-20T08:50:35.430 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710924635100946, "job": 1, "event": "recovery_finished"} 2024-03-20T08:50:35.430 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-20T08:50:35.430 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-20T08:50:35.430 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x563a68080000 2024-03-20T08:50:35.430 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: DB pointer 0x563a6806a000 2024-03-20T08:50:35.430 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-20T08:50:35.430 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-20T08:50:35.430 INFO:journalctl@ceph.mon.b.smithi115.stdout: ** DB Stats ** 2024-03-20T08:50:35.430 INFO:journalctl@ceph.mon.b.smithi115.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-20T08:50:35.431 INFO:journalctl@ceph.mon.b.smithi115.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-20T08:50:35.431 INFO:journalctl@ceph.mon.b.smithi115.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-20T08:50:35.431 INFO:journalctl@ceph.mon.b.smithi115.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-20T08:50:35.431 INFO:journalctl@ceph.mon.b.smithi115.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-20T08:50:35.431 INFO:journalctl@ceph.mon.b.smithi115.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-20T08:50:35.431 INFO:journalctl@ceph.mon.b.smithi115.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-20T08:50:35.431 INFO:journalctl@ceph.mon.b.smithi115.stdout: 2024-03-20T08:50:35.431 INFO:journalctl@ceph.mon.b.smithi115.stdout: ** Compaction Stats [default] ** 2024-03-20T08:50:35.431 INFO:journalctl@ceph.mon.b.smithi115.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-20T08:50:35.431 INFO:journalctl@ceph.mon.b.smithi115.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-20T08:50:35.431 INFO:journalctl@ceph.mon.b.smithi115.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-20T08:50:35.431 INFO:journalctl@ceph.mon.b.smithi115.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-20T08:50:35.431 INFO:journalctl@ceph.mon.b.smithi115.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-20T08:50:35.431 INFO:journalctl@ceph.mon.b.smithi115.stdout: 2024-03-20T08:50:35.432 INFO:journalctl@ceph.mon.b.smithi115.stdout: ** Compaction Stats [default] ** 2024-03-20T08:50:35.432 INFO:journalctl@ceph.mon.b.smithi115.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-20T08:50:35.432 INFO:journalctl@ceph.mon.b.smithi115.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-20T08:50:35.432 INFO:journalctl@ceph.mon.b.smithi115.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-20T08:50:35.432 INFO:journalctl@ceph.mon.b.smithi115.stdout: 2024-03-20T08:50:35.432 INFO:journalctl@ceph.mon.b.smithi115.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-20T08:50:35.432 INFO:journalctl@ceph.mon.b.smithi115.stdout: 2024-03-20T08:50:35.432 INFO:journalctl@ceph.mon.b.smithi115.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-20T08:50:35.432 INFO:journalctl@ceph.mon.b.smithi115.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-20T08:50:35.432 INFO:journalctl@ceph.mon.b.smithi115.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-20T08:50:35.432 INFO:journalctl@ceph.mon.b.smithi115.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-20T08:50:35.432 INFO:journalctl@ceph.mon.b.smithi115.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-20T08:50:35.432 INFO:journalctl@ceph.mon.b.smithi115.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-20T08:50:35.432 INFO:journalctl@ceph.mon.b.smithi115.stdout: Cumulative compaction: 0.00 GB write, 0.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-20T08:50:35.433 INFO:journalctl@ceph.mon.b.smithi115.stdout: Interval compaction: 0.00 GB write, 0.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-20T08:50:35.433 INFO:journalctl@ceph.mon.b.smithi115.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-20T08:50:35.433 INFO:journalctl@ceph.mon.b.smithi115.stdout: Block cache BinnedLRUCache@0x563a67f8d090#8 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-20T08:50:35.433 INFO:journalctl@ceph.mon.b.smithi115.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-20T08:50:35.433 INFO:journalctl@ceph.mon.b.smithi115.stdout: 2024-03-20T08:50:35.433 INFO:journalctl@ceph.mon.b.smithi115.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-20T08:50:35.433 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-20T08:50:35.433 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: using public_addr v2:172.21.15.115:0/0 -> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] 2024-03-20T08:50:35.433 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: starting mon.b rank -1 at public addrs [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] at bind addrs [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 852fbb68-e696-11ee-95c9-87774f69a715 2024-03-20T08:50:35.433 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: mon.b@-1(???) e0 preinit fsid 852fbb68-e696-11ee-95c9-87774f69a715 2024-03-20T08:50:35.433 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #13. Immutable memtables: 0. 2024-03-20T08:50:35.433 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: (Original Log Time 2024/03/20-08:50:35.120507) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2024-03-20T08:50:35.433 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: [db/flush_job.cc:856] [default] [JOB 3] Flushing memtable with next log file: 13 2024-03-20T08:50:35.434 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710924635120633, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 1181, "num_deletes": 20, "total_data_size": 2318648, "memory_usage": 2341216, "flush_reason": "Manual Compaction"} 2024-03-20T08:50:35.434 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: [db/flush_job.cc:885] [default] [JOB 3] Level-0 flush table #14: started 2024-03-20T08:50:35.434 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: mon.b@-1(synchronizing).mds e1 new map 2024-03-20T08:50:35.434 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-20T08:50:35.434 INFO:journalctl@ceph.mon.b.smithi115.stdout: e1 2024-03-20T08:50:35.434 INFO:journalctl@ceph.mon.b.smithi115.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-20T08:50:35.434 INFO:journalctl@ceph.mon.b.smithi115.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-20T08:50:35.434 INFO:journalctl@ceph.mon.b.smithi115.stdout: legacy client fscid: -1 2024-03-20T08:50:35.434 INFO:journalctl@ceph.mon.b.smithi115.stdout: 2024-03-20T08:50:35.434 INFO:journalctl@ceph.mon.b.smithi115.stdout: No filesystems configured 2024-03-20T08:50:35.434 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-20T08:50:35.434 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-20T08:50:35.434 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-20T08:50:35.435 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-20T08:50:35.435 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-20T08:50:35.435 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-20T08:50:35.435 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-20T08:50:35.435 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-20T08:50:35.435 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-20T08:50:35.435 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-20T08:50:35.435 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: Updating smithi115:/etc/ceph/ceph.conf 2024-03-20T08:50:35.435 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:35.435 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:35.435 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:35.435 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:35.435 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: Updating smithi115:/var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-20T08:50:35.435 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: Updating smithi115:/etc/ceph/ceph.client.admin.keyring 2024-03-20T08:50:35.436 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: Updating smithi115:/var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-20T08:50:35.436 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi122", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:50:35.436 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710924635124166, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 14, "file_size": 2042658, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 6, "largest_seqno": 1186, "table_properties": {"data_size": 2036554, "index_size": 2821, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 2245, "raw_key_size": 27111, "raw_average_key_size": 30, "raw_value_size": 2018021, "raw_average_value_size": 2290, "num_data_blocks": 131, "num_entries": 881, "num_filter_entries": 881, "num_deletions": 20, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710924635, "oldest_key_time": 1710924635, "file_creation_time": 1710924635, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "91dc59ff-c51c-4eb2-b55d-b517d9928716", "db_session_id": "E6PM4XZ1I71S9YQTGWJS", "orig_file_number": 14, "seqno_to_time_mapping": "N/A"}} 2024-03-20T08:50:35.436 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: Deploying cephadm binary to smithi122 2024-03-20T08:50:35.436 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:35.436 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:35.436 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: [db/flush_job.cc:1019] [default] [JOB 3] Flush lasted 3704 microseconds, and 2280 cpu microseconds. 2024-03-20T08:50:35.436 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:35.436 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:50:35.436 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:35.436 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:35.436 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: Added host smithi122 2024-03-20T08:50:35.436 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:35.437 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:35.437 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:35.437 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T08:50:35.437 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:35.437 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:35.437 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:35.437 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:35.437 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi122", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:50:35.437 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:35.437 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:50:35.437 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:50:35.437 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:50:35.437 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:35.437 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:35.438 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:35.438 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: Updating smithi122:/etc/ceph/ceph.conf 2024-03-20T08:50:35.438 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:35.438 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: Updating smithi122:/var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-20T08:50:35.438 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-03-20T08:50:35.438 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: Updating smithi122:/var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-20T08:50:35.438 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:50:35.438 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:35.438 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:50:35.438 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:35.438 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/969314353' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-20T08:50:35.438 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: (Original Log Time 2024/03/20-08:50:35.124211) [db/flush_job.cc:967] [default] [JOB 3] Level-0 flush table #14: 2042658 bytes OK 2024-03-20T08:50:35.438 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: (Original Log Time 2024/03/20-08:50:35.124226) [db/memtable_list.cc:519] [default] Level-0 commit table #14 started 2024-03-20T08:50:35.439 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: (Original Log Time 2024/03/20-08:50:35.124346) [db/memtable_list.cc:722] [default] Level-0 commit table #14: memtable #1 done 2024-03-20T08:50:35.439 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: (Original Log Time 2024/03/20-08:50:35.124360) EVENT_LOG_v1 {"time_micros": 1710924635124355, "job": 3, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [2, 0, 0, 0, 0, 0, 0], "immutable_memtables": 0} 2024-03-20T08:50:35.439 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:35.439 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: (Original Log Time 2024/03/20-08:50:35.124372) [db/db_impl/db_impl_compaction_flush.cc:299] [default] Level summary: files[2 0 0 0 0 0 0] max score 0.50 2024-03-20T08:50:35.439 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 3] Try to delete WAL files size 2310409, prev total WAL file size 2310632, number of live WAL files 2. 2024-03-20T08:50:35.439 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/969314353' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-20T08:50:35.439 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: osdmap e4: 0 total, 0 up, 0 in 2024-03-20T08:50:35.439 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:35.439 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:35.439 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:50:35.439 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:35.439 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:50:35.439 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:35.439 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-20T08:50:35.440 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:35.440 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi026:172.21.15.26=a;smithi115:172.21.15.115=b;smithi122:172.21.15.122=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:50:35.440 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: Saving service mon spec with placement smithi026:172.21.15.26=a;smithi115:172.21.15.115=b;smithi122:172.21.15.122=c;count:3 2024-03-20T08:50:35.440 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: Deploying daemon mon.c on smithi122 2024-03-20T08:50:35.440 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:35.440 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:35.440 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='client.? 172.21.15.122:0/1839444606' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-20T08:50:35.440 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:50:35.440 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: mon.a calling monitor election 2024-03-20T08:50:35.440 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:50:35.440 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:35.441 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:50:35.441 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:50:35.441 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: mon.c calling monitor election 2024-03-20T08:50:35.441 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:35.441 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:50:35.441 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:50:35.441 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:50:35.441 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-20T08:50:35.441 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: monmap e2: 2 mons at {a=[v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0],c=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-20T08:50:35.441 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: fsmap 2024-03-20T08:50:35.441 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: osdmap e4: 0 total, 0 up, 0 in 2024-03-20T08:50:35.441 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: mgrmap e13: y(active, since 56s) 2024-03-20T08:50:35.441 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: overall HEALTH_OK 2024-03-20T08:50:35.441 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:35.442 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:35.442 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:35.442 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-20T08:50:35.442 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:35.442 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:35.442 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: Deploying daemon mon.b on smithi115 2024-03-20T08:50:35.442 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:50:35.442 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: from='client.? 172.21.15.122:0/734152903' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-20T08:50:35.442 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:35.442 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-20T08:50:35.442 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: (Original Log Time 2024/03/20-08:50:35.124851) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '6B760030' seq:72057594037927935, type:22 .. '6B76003131' seq:0, type:0; will stop at (end) 2024-03-20T08:50:35.442 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 4] Compacting 2@0 files to L6, score -1.00 2024-03-20T08:50:35.442 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 3 Base level 0, inputs: [14(1994KB) 8(1747B)] 2024-03-20T08:50:35.443 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-20T08:50:35.443 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 ceph-mon[72476]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710924635124908, "job": 4, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [14, 8], "score": -1, "input_data_size": 2044405, "oldest_snapshot_seqno": -1} 2024-03-20T08:50:35.443 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 bash[72437]: 001a6e570112b8d15082a09b5c811d5dfe2e199540811b2083428944bb761fb0 2024-03-20T08:50:35.443 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:35 smithi115 systemd[1]: Started Ceph mon.b for 852fbb68-e696-11ee-95c9-87774f69a715. 2024-03-20T08:50:40.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:40 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:50:40.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:40 smithi026 ceph-mon[68309]: mon.c calling monitor election 2024-03-20T08:50:40.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:40 smithi026 ceph-mon[68309]: mon.a calling monitor election 2024-03-20T08:50:40.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:40 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:50:40.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:40 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:50:40.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:40 smithi026 ceph-mon[68309]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:40.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:40 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:50:40.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:40 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:50:40.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:40 smithi026 ceph-mon[68309]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:40.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:40 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:50:40.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:40 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:50:40.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:40 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:50:40.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:40 smithi026 ceph-mon[68309]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-20T08:50:40.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:40 smithi026 ceph-mon[68309]: monmap e3: 3 mons at {a=[v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0],b=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0],c=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-20T08:50:40.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:40 smithi026 ceph-mon[68309]: fsmap 2024-03-20T08:50:40.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:40 smithi026 ceph-mon[68309]: osdmap e4: 0 total, 0 up, 0 in 2024-03-20T08:50:40.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:40 smithi026 ceph-mon[68309]: mgrmap e13: y(active, since 64s) 2024-03-20T08:50:40.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:40 smithi026 ceph-mon[68309]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-20T08:50:40.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:40 smithi026 ceph-mon[68309]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-20T08:50:40.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:40 smithi026 ceph-mon[68309]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-20T08:50:40.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:40 smithi026 ceph-mon[68309]: mon.b (rank 2) addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] is down (out of quorum) 2024-03-20T08:50:40.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:40 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:40.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:40 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:40.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:40 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:50:40.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:40 smithi122 ceph-mon[73495]: mon.c calling monitor election 2024-03-20T08:50:40.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:40 smithi122 ceph-mon[73495]: mon.a calling monitor election 2024-03-20T08:50:40.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:40 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:50:40.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:40 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:50:40.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:40 smithi122 ceph-mon[73495]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:40.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:40 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:50:40.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:40 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:50:40.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:40 smithi122 ceph-mon[73495]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:40.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:40 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:50:40.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:40 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:50:40.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:40 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:50:40.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:40 smithi122 ceph-mon[73495]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-20T08:50:40.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:40 smithi122 ceph-mon[73495]: monmap e3: 3 mons at {a=[v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0],b=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0],c=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-20T08:50:40.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:40 smithi122 ceph-mon[73495]: fsmap 2024-03-20T08:50:40.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:40 smithi122 ceph-mon[73495]: osdmap e4: 0 total, 0 up, 0 in 2024-03-20T08:50:40.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:40 smithi122 ceph-mon[73495]: mgrmap e13: y(active, since 64s) 2024-03-20T08:50:40.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:40 smithi122 ceph-mon[73495]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-20T08:50:40.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:40 smithi122 ceph-mon[73495]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-20T08:50:40.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:40 smithi122 ceph-mon[73495]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-20T08:50:40.699 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:40 smithi122 ceph-mon[73495]: mon.b (rank 2) addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] is down (out of quorum) 2024-03-20T08:50:40.699 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:40 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:40.699 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:40 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:40.880 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-20T08:50:40.880 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":3,"fsid":"852fbb68-e696-11ee-95c9-87774f69a715","modified":"2024-03-20T08:50:35.128848Z","created":"2024-03-20T08:48:32.007170Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:3300","nonce":0},{"type":"v1","addr":"172.21.15.115:6789","nonce":0}]},"addr":"172.21.15.115:6789/0","public_addr":"172.21.15.115:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-20T08:50:40.882 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 3 2024-03-20T08:50:41.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:41 smithi122 ceph-mon[73495]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:41.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:41 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:41.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:41 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:41.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:41 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:41.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:41 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:50:41.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:41 smithi122 ceph-mon[73495]: from='client.? 172.21.15.122:0/2462837481' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-20T08:50:41.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:41 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:50:41.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:41 smithi026 ceph-mon[68309]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:41.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:41 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:41.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:41 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:41.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:41 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:41.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:41 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:50:41.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:41 smithi026 ceph-mon[68309]: from='client.? 172.21.15.122:0/2462837481' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-20T08:50:41.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:41 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:50:42.289 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-20T08:50:42.289 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph config generate-minimal-conf 2024-03-20T08:50:42.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:50:42.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: mon.c calling monitor election 2024-03-20T08:50:42.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: mon.a calling monitor election 2024-03-20T08:50:42.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:50:42.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:50:42.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:42.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:50:42.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:50:42.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:42.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:50:42.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:50:42.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:50:42.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-20T08:50:42.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: monmap e3: 3 mons at {a=[v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0],b=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0],c=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-20T08:50:42.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: fsmap 2024-03-20T08:50:42.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: osdmap e4: 0 total, 0 up, 0 in 2024-03-20T08:50:42.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: mgrmap e13: y(active, since 64s) 2024-03-20T08:50:42.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-20T08:50:42.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-20T08:50:42.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-20T08:50:42.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: mon.b (rank 2) addr [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] is down (out of quorum) 2024-03-20T08:50:42.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:42.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:42.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:42.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:42.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:42.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:42.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:50:42.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: from='client.? 172.21.15.122:0/2462837481' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-20T08:50:42.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:50:42.562 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:50:43.102 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:42 smithi026 ceph-mon[68309]: mon.b calling monitor election 2024-03-20T08:50:43.102 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:42 smithi026 ceph-mon[68309]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:43.102 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:42 smithi026 ceph-mon[68309]: Updating smithi026:/etc/ceph/ceph.conf 2024-03-20T08:50:43.103 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:42 smithi026 ceph-mon[68309]: Updating smithi115:/etc/ceph/ceph.conf 2024-03-20T08:50:43.103 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:42 smithi026 ceph-mon[68309]: Updating smithi122:/etc/ceph/ceph.conf 2024-03-20T08:50:43.103 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:42 smithi026 ceph-mon[68309]: Updating smithi026:/var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-20T08:50:43.103 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:42 smithi026 ceph-mon[68309]: Updating smithi115:/var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-20T08:50:43.103 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:42 smithi026 ceph-mon[68309]: Updating smithi122:/var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-20T08:50:43.103 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:42 smithi026 ceph-mon[68309]: mon.b calling monitor election 2024-03-20T08:50:43.103 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:42 smithi026 ceph-mon[68309]: mon.a calling monitor election 2024-03-20T08:50:43.103 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:42 smithi026 ceph-mon[68309]: mon.c calling monitor election 2024-03-20T08:50:43.103 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:42 smithi026 ceph-mon[68309]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-20T08:50:43.104 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:42 smithi026 ceph-mon[68309]: monmap e3: 3 mons at {a=[v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0],b=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0],c=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-20T08:50:43.104 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:42 smithi026 ceph-mon[68309]: fsmap 2024-03-20T08:50:43.104 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:42 smithi026 ceph-mon[68309]: osdmap e4: 0 total, 0 up, 0 in 2024-03-20T08:50:43.104 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:42 smithi026 ceph-mon[68309]: mgrmap e13: y(active, since 66s) 2024-03-20T08:50:43.104 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:42 smithi026 ceph-mon[68309]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-20T08:50:43.104 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:42 smithi026 ceph-mon[68309]: Cluster is now healthy 2024-03-20T08:50:43.104 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:42 smithi026 ceph-mon[68309]: overall HEALTH_OK 2024-03-20T08:50:43.104 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:42 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:43.104 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:42 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:43.104 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:42 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:43.104 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:42 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:43.105 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:42 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:43.105 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:42 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:43.105 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:42 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:43.105 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:42 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:43.105 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:42 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:43.105 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:42 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:43.105 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:42 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:43.105 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:42 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-20T08:50:43.105 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:42 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-20T08:50:43.105 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:42 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:43.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: mon.b calling monitor election 2024-03-20T08:50:43.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:43.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: Updating smithi026:/etc/ceph/ceph.conf 2024-03-20T08:50:43.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: Updating smithi115:/etc/ceph/ceph.conf 2024-03-20T08:50:43.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: Updating smithi122:/etc/ceph/ceph.conf 2024-03-20T08:50:43.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: Updating smithi026:/var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-20T08:50:43.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: Updating smithi115:/var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-20T08:50:43.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: Updating smithi122:/var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-20T08:50:43.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: mon.b calling monitor election 2024-03-20T08:50:43.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: mon.a calling monitor election 2024-03-20T08:50:43.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: mon.c calling monitor election 2024-03-20T08:50:43.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-20T08:50:43.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: monmap e3: 3 mons at {a=[v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0],b=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0],c=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-20T08:50:43.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: fsmap 2024-03-20T08:50:43.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: osdmap e4: 0 total, 0 up, 0 in 2024-03-20T08:50:43.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: mgrmap e13: y(active, since 66s) 2024-03-20T08:50:43.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-20T08:50:43.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: Cluster is now healthy 2024-03-20T08:50:43.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: overall HEALTH_OK 2024-03-20T08:50:43.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:43.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:43.158 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:43.158 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:43.158 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:43.158 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:43.158 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:43.158 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:43.159 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:43.159 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:43.159 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:43.159 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-20T08:50:43.159 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-20T08:50:43.159 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:43.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:42 smithi122 ceph-mon[73495]: mon.b calling monitor election 2024-03-20T08:50:43.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:42 smithi122 ceph-mon[73495]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:43.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:42 smithi122 ceph-mon[73495]: Updating smithi026:/etc/ceph/ceph.conf 2024-03-20T08:50:43.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:42 smithi122 ceph-mon[73495]: Updating smithi115:/etc/ceph/ceph.conf 2024-03-20T08:50:43.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:42 smithi122 ceph-mon[73495]: Updating smithi122:/etc/ceph/ceph.conf 2024-03-20T08:50:43.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:42 smithi122 ceph-mon[73495]: Updating smithi026:/var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-20T08:50:43.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:42 smithi122 ceph-mon[73495]: Updating smithi115:/var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-20T08:50:43.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:42 smithi122 ceph-mon[73495]: Updating smithi122:/var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-20T08:50:43.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:42 smithi122 ceph-mon[73495]: mon.b calling monitor election 2024-03-20T08:50:43.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:42 smithi122 ceph-mon[73495]: mon.a calling monitor election 2024-03-20T08:50:43.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:42 smithi122 ceph-mon[73495]: mon.c calling monitor election 2024-03-20T08:50:43.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:42 smithi122 ceph-mon[73495]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-20T08:50:43.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:42 smithi122 ceph-mon[73495]: monmap e3: 3 mons at {a=[v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0],b=[v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0],c=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-20T08:50:43.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:42 smithi122 ceph-mon[73495]: fsmap 2024-03-20T08:50:43.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:42 smithi122 ceph-mon[73495]: osdmap e4: 0 total, 0 up, 0 in 2024-03-20T08:50:43.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:42 smithi122 ceph-mon[73495]: mgrmap e13: y(active, since 66s) 2024-03-20T08:50:43.199 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:42 smithi122 ceph-mon[73495]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-20T08:50:43.199 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:42 smithi122 ceph-mon[73495]: Cluster is now healthy 2024-03-20T08:50:43.199 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:42 smithi122 ceph-mon[73495]: overall HEALTH_OK 2024-03-20T08:50:43.199 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:42 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:43.199 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:42 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:43.199 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:42 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:43.199 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:42 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:43.199 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:42 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:43.199 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:42 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:43.199 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:42 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:43.199 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:42 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:43.199 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:42 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:43.199 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:42 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:43.200 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:42 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:43.200 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:42 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-20T08:50:43.200 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:42 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-20T08:50:43.200 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:42 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:44.101 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:43 smithi026 ceph-mon[68309]: Reconfiguring mon.a (unknown last config time)... 2024-03-20T08:50:44.101 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:43 smithi026 ceph-mon[68309]: Reconfiguring daemon mon.a on smithi026 2024-03-20T08:50:44.101 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:43 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:50:44.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:43 smithi115 ceph-mon[72476]: Reconfiguring mon.a (unknown last config time)... 2024-03-20T08:50:44.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:43 smithi115 ceph-mon[72476]: Reconfiguring daemon mon.a on smithi026 2024-03-20T08:50:44.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:43 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:50:44.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:43 smithi122 ceph-mon[73495]: Reconfiguring mon.a (unknown last config time)... 2024-03-20T08:50:44.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:43 smithi122 ceph-mon[73495]: Reconfiguring daemon mon.a on smithi026 2024-03-20T08:50:44.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:43 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:50:45.010 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:44 smithi026 ceph-mon[68309]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:45.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:44 smithi115 ceph-mon[72476]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:45.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:44 smithi122 ceph-mon[73495]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:45.674 INFO:teuthology.orchestra.run.smithi026.stdout:# minimal ceph.conf for 852fbb68-e696-11ee-95c9-87774f69a715 2024-03-20T08:50:45.674 INFO:teuthology.orchestra.run.smithi026.stdout:[global] 2024-03-20T08:50:45.674 INFO:teuthology.orchestra.run.smithi026.stdout: fsid = 852fbb68-e696-11ee-95c9-87774f69a715 2024-03-20T08:50:45.674 INFO:teuthology.orchestra.run.smithi026.stdout: mon_host = [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-03-20T08:50:46.325 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:46.325 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:46.325 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-20T08:50:46.325 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-20T08:50:46.325 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:46.325 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:46 smithi026 ceph-mon[68309]: Reconfiguring daemon mon.b on smithi115 2024-03-20T08:50:46.325 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:50:46.326 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:50:46.326 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:50:46.326 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:46 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1543863267' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:46.358 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-20T08:50:46.359 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-20T08:50:46.359 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/ceph/ceph.conf 2024-03-20T08:50:46.393 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-20T08:50:46.393 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-20T08:50:46.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:46.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:46.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-20T08:50:46.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-20T08:50:46.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:46.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:46 smithi115 ceph-mon[72476]: Reconfiguring daemon mon.b on smithi115 2024-03-20T08:50:46.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:50:46.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:50:46.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:50:46.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:46 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1543863267' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:46.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:46.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:46.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-20T08:50:46.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-20T08:50:46.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:46.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:46 smithi122 ceph-mon[73495]: Reconfiguring daemon mon.b on smithi115 2024-03-20T08:50:46.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:50:46.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:50:46.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:50:46.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:46 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1543863267' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:46.463 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-20T08:50:46.463 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/ceph/ceph.conf 2024-03-20T08:50:46.497 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-20T08:50:46.497 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-20T08:50:46.575 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-20T08:50:46.575 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/ceph.conf 2024-03-20T08:50:46.612 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-20T08:50:46.612 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-20T08:50:46.686 INFO:tasks.cephadm:Adding mgr.y on smithi026 2024-03-20T08:50:46.686 INFO:tasks.cephadm:Adding mgr.x on smithi115 2024-03-20T08:50:46.686 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph orch apply mgr '2;smithi026=y;smithi115=x' 2024-03-20T08:50:47.000 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T08:50:47.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:47 smithi115 ceph-mon[72476]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:47.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:47 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:47.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:47 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:47.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:47 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-20T08:50:47.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:47 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-20T08:50:47.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:47 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:47.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:47 smithi122 ceph-mon[73495]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:47 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:47 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:47 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-20T08:50:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:47 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-20T08:50:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:47 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:47.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:47 smithi026 ceph-mon[68309]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:47.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:47 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:47.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:47 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:47.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:47 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-20T08:50:47.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:47 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-20T08:50:47.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:47 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:48.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:48 smithi115 ceph-mon[72476]: Reconfiguring mon.c (monmap changed)... 2024-03-20T08:50:48.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:48 smithi115 ceph-mon[72476]: Reconfiguring daemon mon.c on smithi122 2024-03-20T08:50:48.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:48 smithi122 ceph-mon[73495]: Reconfiguring mon.c (monmap changed)... 2024-03-20T08:50:48.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:48 smithi122 ceph-mon[73495]: Reconfiguring daemon mon.c on smithi122 2024-03-20T08:50:48.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:48 smithi026 ceph-mon[68309]: Reconfiguring mon.c (monmap changed)... 2024-03-20T08:50:48.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:48 smithi026 ceph-mon[68309]: Reconfiguring daemon mon.c on smithi122 2024-03-20T08:50:49.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:49 smithi122 ceph-mon[73495]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:49.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:49 smithi115 ceph-mon[72476]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:49.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:49 smithi026 ceph-mon[68309]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:49.810 INFO:teuthology.orchestra.run.smithi122.stdout:Scheduled mgr update... 2024-03-20T08:50:50.481 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:50 smithi122 ceph-mon[73495]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:50.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:50 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:50.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:50 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:50.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:50 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:50:50.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:50 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:50.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:50 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:50:50.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:50 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:50.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:50 smithi122 ceph-mon[73495]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi026=y;smithi115=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:50:50.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:50 smithi122 ceph-mon[73495]: Saving service mgr spec with placement smithi026=y;smithi115=x;count:2 2024-03-20T08:50:50.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:50 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:50.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:50 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:50:50.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:50 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:50.483 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:50 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:50:50.483 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:50 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:50.483 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:50 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-20T08:50:50.483 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:50 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-20T08:50:50.483 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:50 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-20T08:50:50.483 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:50 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:50.483 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:50 smithi122 ceph-mon[73495]: Deploying daemon mgr.x on smithi115 2024-03-20T08:50:50.507 DEBUG:teuthology.orchestra.run.smithi115:mgr.x> sudo journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@mgr.x.service 2024-03-20T08:50:50.509 INFO:tasks.cephadm:Deploying OSDs... 2024-03-20T08:50:50.510 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-20T08:50:50.510 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/scratch_devs of=/dev/stdout 2024-03-20T08:50:50.533 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-20T08:50:50.533 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_1 2024-03-20T08:50:50.537 INFO:journalctl@ceph.mgr.x.smithi115.stdout:-- Logs begin at Wed 2024-03-20 08:37:21 UTC. -- 2024-03-20T08:50:50.596 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-20T08:50:50.596 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-20T08:50:50.596 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 6h/6d Inode: 196230 Links: 1 2024-03-20T08:50:50.596 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-20T08:50:50.596 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-03-20T08:50:50.597 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-03-20 08:49:49.801770058 +0000 2024-03-20T08:50:50.597 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-03-20 08:43:31.857248782 +0000 2024-03-20T08:50:50.597 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-03-20 08:43:31.857248782 +0000 2024-03-20T08:50:50.597 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: - 2024-03-20T08:50:50.597 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-20T08:50:50.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:50 smithi115 ceph-mon[72476]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:50.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:50 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:50.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:50 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:50.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:50 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:50:50.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:50 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:50.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:50 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:50:50.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:50 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:50.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:50 smithi115 ceph-mon[72476]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi026=y;smithi115=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:50:50.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:50 smithi115 ceph-mon[72476]: Saving service mgr spec with placement smithi026=y;smithi115=x;count:2 2024-03-20T08:50:50.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:50 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:50.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:50 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:50:50.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:50 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:50.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:50 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:50:50.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:50 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:50.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:50 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-20T08:50:50.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:50 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-20T08:50:50.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:50 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-20T08:50:50.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:50 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:50.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:50 smithi115 ceph-mon[72476]: Deploying daemon mgr.x on smithi115 2024-03-20T08:50:50.666 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:50 smithi026 ceph-mon[68309]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:50.666 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:50 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:50.666 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:50 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:50.667 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:50 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:50:50.667 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:50 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:50.667 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:50 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:50:50.667 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:50 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:50.667 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:50 smithi026 ceph-mon[68309]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi026=y;smithi115=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:50:50.667 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:50 smithi026 ceph-mon[68309]: Saving service mgr spec with placement smithi026=y;smithi115=x;count:2 2024-03-20T08:50:50.667 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:50 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:50.667 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:50 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:50:50.667 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:50 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:50.667 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:50 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:50:50.667 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:50 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:50.667 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:50 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-20T08:50:50.667 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:50 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-20T08:50:50.667 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:50 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-20T08:50:50.668 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:50 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:50.668 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:50 smithi026 ceph-mon[68309]: Deploying daemon mgr.x on smithi115 2024-03-20T08:50:50.668 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-03-20T08:50:50.668 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-03-20T08:50:50.668 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.000122047 s, 4.2 MB/s 2024-03-20T08:50:50.670 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-20T08:50:50.732 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_2 2024-03-20T08:50:50.792 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-20T08:50:50.793 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-20T08:50:50.793 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 6h/6d Inode: 196869 Links: 1 2024-03-20T08:50:50.793 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-20T08:50:50.793 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-03-20T08:50:50.793 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-03-20 08:49:49.848769326 +0000 2024-03-20T08:50:50.793 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-03-20 08:43:32.315241423 +0000 2024-03-20T08:50:50.793 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-03-20 08:43:32.315241423 +0000 2024-03-20T08:50:50.793 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: - 2024-03-20T08:50:50.793 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-20T08:50:50.862 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-03-20T08:50:50.862 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-03-20T08:50:50.863 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.000107287 s, 4.8 MB/s 2024-03-20T08:50:50.864 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-20T08:50:50.925 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_3 2024-03-20T08:50:50.985 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-20T08:50:50.985 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-20T08:50:50.985 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 6h/6d Inode: 197824 Links: 1 2024-03-20T08:50:50.985 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-20T08:50:50.986 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-03-20T08:50:50.986 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-03-20 08:49:49.897768563 +0000 2024-03-20T08:50:50.986 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-03-20 08:43:32.679235575 +0000 2024-03-20T08:50:50.986 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-03-20 08:43:32.679235575 +0000 2024-03-20T08:50:50.986 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: - 2024-03-20T08:50:50.986 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-20T08:50:51.054 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-03-20T08:50:51.054 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-03-20T08:50:51.054 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.000113505 s, 4.5 MB/s 2024-03-20T08:50:51.055 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-20T08:50:51.116 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_4 2024-03-20T08:50:51.175 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-20T08:50:51.176 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-20T08:50:51.176 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 6h/6d Inode: 197942 Links: 1 2024-03-20T08:50:51.176 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-20T08:50:51.176 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-03-20T08:50:51.176 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-03-20 08:49:49.942767862 +0000 2024-03-20T08:50:51.176 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-03-20 08:43:33.009230273 +0000 2024-03-20T08:50:51.176 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-03-20 08:43:33.009230273 +0000 2024-03-20T08:50:51.176 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: - 2024-03-20T08:50:51.176 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-20T08:50:51.244 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-03-20T08:50:51.244 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-03-20T08:50:51.244 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 9.9746e-05 s, 5.1 MB/s 2024-03-20T08:50:51.245 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-20T08:50:51.305 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-20T08:50:51.305 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/scratch_devs of=/dev/stdout 2024-03-20T08:50:51.325 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-20T08:50:51.325 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_1 2024-03-20T08:50:51.383 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-20T08:50:51.384 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-20T08:50:51.384 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 6h/6d Inode: 194929 Links: 1 2024-03-20T08:50:51.384 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-20T08:50:51.384 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2024-03-20T08:50:51.384 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2024-03-20 08:50:34.520700334 +0000 2024-03-20T08:50:51.384 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2024-03-20 08:43:26.975677557 +0000 2024-03-20T08:50:51.384 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2024-03-20 08:43:26.975677557 +0000 2024-03-20T08:50:51.384 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: - 2024-03-20T08:50:51.384 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-20T08:50:51.544 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2024-03-20T08:50:51.544 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2024-03-20T08:50:51.545 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000180427 s, 2.8 MB/s 2024-03-20T08:50:51.545 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-20T08:50:51.564 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_2 2024-03-20T08:50:51.620 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-20T08:50:51.621 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-20T08:50:51.621 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 6h/6d Inode: 192310 Links: 1 2024-03-20T08:50:51.621 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-20T08:50:51.621 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2024-03-20T08:50:51.621 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2024-03-20 08:50:51.494378166 +0000 2024-03-20T08:50:51.621 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2024-03-20 08:43:27.324670930 +0000 2024-03-20T08:50:51.621 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2024-03-20 08:43:27.324670930 +0000 2024-03-20T08:50:51.621 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: - 2024-03-20T08:50:51.621 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-20T08:50:51.688 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2024-03-20T08:50:51.688 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2024-03-20T08:50:51.688 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000235413 s, 2.2 MB/s 2024-03-20T08:50:51.690 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-20T08:50:51.751 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_3 2024-03-20T08:50:51.810 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-20T08:50:51.811 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-20T08:50:51.811 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 6h/6d Inode: 198167 Links: 1 2024-03-20T08:50:51.811 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-20T08:50:51.811 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2024-03-20T08:50:51.811 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2024-03-20 08:50:51.625375709 +0000 2024-03-20T08:50:51.811 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2024-03-20 08:43:27.699663809 +0000 2024-03-20T08:50:51.811 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2024-03-20 08:43:27.699663809 +0000 2024-03-20T08:50:51.811 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: - 2024-03-20T08:50:51.811 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-20T08:50:51.876 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2024-03-20T08:50:51.876 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2024-03-20T08:50:51.877 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000165521 s, 3.1 MB/s 2024-03-20T08:50:51.877 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-20T08:50:51.936 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_4 2024-03-20T08:50:51.995 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-20T08:50:51.995 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-20T08:50:51.995 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 6h/6d Inode: 196429 Links: 1 2024-03-20T08:50:51.995 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-20T08:50:51.995 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2024-03-20T08:50:51.995 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2024-03-20 08:50:51.625375709 +0000 2024-03-20T08:50:51.995 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2024-03-20 08:43:28.034657448 +0000 2024-03-20T08:50:51.995 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2024-03-20 08:43:28.034657448 +0000 2024-03-20T08:50:51.995 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: - 2024-03-20T08:50:51.996 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-20T08:50:52.061 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 20 08:50:51 smithi115 systemd[1]: Starting Ceph mgr.x for 852fbb68-e696-11ee-95c9-87774f69a715... 2024-03-20T08:50:52.063 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2024-03-20T08:50:52.063 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2024-03-20T08:50:52.063 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000226125 s, 2.3 MB/s 2024-03-20T08:50:52.064 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-20T08:50:52.124 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-20T08:50:52.124 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/scratch_devs of=/dev/stdout 2024-03-20T08:50:52.147 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-20T08:50:52.148 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_1 2024-03-20T08:50:52.210 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-20T08:50:52.210 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-20T08:50:52.210 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 6h/6d Inode: 195971 Links: 1 2024-03-20T08:50:52.210 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-20T08:50:52.210 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-03-20T08:50:52.210 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-03-20 08:50:25.207030955 +0000 2024-03-20T08:50:52.210 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-03-20 08:43:24.795093073 +0000 2024-03-20T08:50:52.210 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-03-20 08:43:24.795093073 +0000 2024-03-20T08:50:52.210 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: - 2024-03-20T08:50:52.211 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-20T08:50:52.285 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-03-20T08:50:52.285 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-03-20T08:50:52.285 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000240462 s, 2.1 MB/s 2024-03-20T08:50:52.286 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-20T08:50:52.349 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_2 2024-03-20T08:50:52.409 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-20T08:50:52.410 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-20T08:50:52.410 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 6h/6d Inode: 190324 Links: 1 2024-03-20T08:50:52.410 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-20T08:50:52.410 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-03-20T08:50:52.410 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-03-20 08:50:25.207030955 +0000 2024-03-20T08:50:52.410 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-03-20 08:43:25.136085076 +0000 2024-03-20T08:50:52.411 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-03-20 08:43:25.136085076 +0000 2024-03-20T08:50:52.411 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: - 2024-03-20T08:50:52.411 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-20T08:50:52.478 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-03-20T08:50:52.478 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-03-20T08:50:52.478 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000215382 s, 2.4 MB/s 2024-03-20T08:50:52.479 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-20T08:50:52.540 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_3 2024-03-20T08:50:52.599 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-20T08:50:52.599 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-20T08:50:52.599 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 6h/6d Inode: 190411 Links: 1 2024-03-20T08:50:52.599 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-20T08:50:52.599 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-03-20T08:50:52.599 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-03-20 08:50:25.208030932 +0000 2024-03-20T08:50:52.600 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-03-20 08:43:25.518076118 +0000 2024-03-20T08:50:52.600 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-03-20 08:43:25.518076118 +0000 2024-03-20T08:50:52.600 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: - 2024-03-20T08:50:52.600 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-20T08:50:52.621 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 20 08:50:52 smithi115 ceph-mgr[74158]: -- 172.21.15.115:0/4175370230 <== mon.2 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558191a405a0 con 0x558191a14800 2024-03-20T08:50:52.622 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 20 08:50:52 smithi115 bash[73978]: 761c176dc8e7081f5ff1918d3c2106117a88052ea7f44183af078188b3a7662f 2024-03-20T08:50:52.622 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 20 08:50:52 smithi115 systemd[1]: Started Ceph mgr.x for 852fbb68-e696-11ee-95c9-87774f69a715. 2024-03-20T08:50:52.622 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 20 08:50:52 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-x[74086]: 2024-03-20T08:50:52.449+0000 7f20f47de200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-20T08:50:52.622 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 20 08:50:52 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-x[74086]: 2024-03-20T08:50:52.620+0000 7f20f47de200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-20T08:50:52.664 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-03-20T08:50:52.664 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-03-20T08:50:52.665 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000206457 s, 2.5 MB/s 2024-03-20T08:50:52.665 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-20T08:50:52.724 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_4 2024-03-20T08:50:52.782 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-20T08:50:52.782 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-20T08:50:52.783 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 6h/6d Inode: 198045 Links: 1 2024-03-20T08:50:52.783 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-20T08:50:52.783 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-03-20T08:50:52.783 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-03-20 08:50:25.208030932 +0000 2024-03-20T08:50:52.783 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-03-20 08:43:25.861068074 +0000 2024-03-20T08:50:52.783 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-03-20 08:43:25.861068074 +0000 2024-03-20T08:50:52.783 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: - 2024-03-20T08:50:52.783 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-20T08:50:52.848 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-03-20T08:50:52.848 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-03-20T08:50:52.848 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000227604 s, 2.2 MB/s 2024-03-20T08:50:52.849 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-20T08:50:52.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:52 smithi115 ceph-mon[72476]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:52.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:52 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:52.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:52 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:52.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:52 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:52.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:52 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:52.908 INFO:tasks.cephadm:Deploying osd.0 on smithi026 with /dev/vg_nvme/lv_4... 2024-03-20T08:50:52.908 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-20T08:50:52.939 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:52 smithi026 ceph-mon[68309]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:52.939 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:52 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:52.939 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:52 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:52.939 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:52 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:52.939 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:52 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:52.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:52 smithi122 ceph-mon[73495]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:52.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:52 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:52.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:52 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:52.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:52 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:52.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:52 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:53.190 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:50:53.405 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 20 08:50:52 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-x[74086]: 2024-03-20T08:50:52.976+0000 7f20f47de200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-20T08:50:53.755 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:53 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:50:53.755 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 20 08:50:53 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-x[74086]: 2024-03-20T08:50:53.511+0000 7f20f47de200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-20T08:50:53.862 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:53 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:50:53.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:53 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:50:54.155 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 20 08:50:53 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-x[74086]: 2024-03-20T08:50:53.754+0000 7f20f47de200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-20T08:50:54.655 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 20 08:50:54 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-x[74086]: 2024-03-20T08:50:54.200+0000 7f20f47de200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-20T08:50:54.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:54 smithi026 ceph-mon[68309]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:54.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:54 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:54.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:54 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:54.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:54 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:54.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:54 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:50:54.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:54 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:54.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:54 smithi026 ceph-mon[68309]: Reconfiguring mgr.y (unknown last config time)... 2024-03-20T08:50:54.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:54 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-20T08:50:54.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:54 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-20T08:50:54.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:54 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:54.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:54 smithi026 ceph-mon[68309]: Reconfiguring daemon mgr.y on smithi026 2024-03-20T08:50:55.155 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 20 08:50:54 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-x[74086]: 2024-03-20T08:50:54.907+0000 7f20f47de200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-20T08:50:55.156 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 20 08:50:54 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-x[74086]: 2024-03-20T08:50:54.992+0000 7f20f47de200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-20T08:50:55.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:54 smithi115 ceph-mon[72476]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:55.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:54 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:55.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:54 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:55.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:54 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:55.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:54 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:50:55.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:54 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:55.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:54 smithi115 ceph-mon[72476]: Reconfiguring mgr.y (unknown last config time)... 2024-03-20T08:50:55.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:54 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-20T08:50:55.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:54 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-20T08:50:55.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:54 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:55.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:54 smithi115 ceph-mon[72476]: Reconfiguring daemon mgr.y on smithi026 2024-03-20T08:50:55.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:54 smithi122 ceph-mon[73495]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:55.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:54 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:55.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:54 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:55.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:54 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:55.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:54 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:50:55.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:54 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:55.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:54 smithi122 ceph-mon[73495]: Reconfiguring mgr.y (unknown last config time)... 2024-03-20T08:50:55.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:54 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-20T08:50:55.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:54 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-20T08:50:55.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:54 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:55.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:54 smithi122 ceph-mon[73495]: Reconfiguring daemon mgr.y on smithi026 2024-03-20T08:50:55.905 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 20 08:50:55 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-x[74086]: 2024-03-20T08:50:55.583+0000 7f20f47de200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-20T08:50:55.906 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 20 08:50:55 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-x[74086]: 2024-03-20T08:50:55.668+0000 7f20f47de200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-20T08:50:55.906 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 20 08:50:55 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-x[74086]: 2024-03-20T08:50:55.757+0000 7f20f47de200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-20T08:50:56.209 INFO:teuthology.orchestra.run.smithi026.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-20T08:50:56.210 INFO:teuthology.orchestra.run.smithi026.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-20T08:50:56.210 INFO:teuthology.orchestra.run.smithi026.stderr: stderr: 10+0 records in 2024-03-20T08:50:56.210 INFO:teuthology.orchestra.run.smithi026.stderr:10+0 records out 2024-03-20T08:50:56.210 INFO:teuthology.orchestra.run.smithi026.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00804869 s, 1.3 GB/s 2024-03-20T08:50:56.210 INFO:teuthology.orchestra.run.smithi026.stderr:--> Zapping successful for: 2024-03-20T08:50:56.404 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 20 08:50:55 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-x[74086]: 2024-03-20T08:50:55.986+0000 7f20f47de200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-20T08:50:56.404 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 20 08:50:56 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-x[74086]: 2024-03-20T08:50:56.080+0000 7f20f47de200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-20T08:50:56.404 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 20 08:50:56 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-x[74086]: 2024-03-20T08:50:56.172+0000 7f20f47de200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-20T08:50:56.404 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 20 08:50:56 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-x[74086]: 2024-03-20T08:50:56.403+0000 7f20f47de200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-20T08:50:56.501 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:56 smithi026 ceph-mon[68309]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:56.501 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:56 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:56.655 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 20 08:50:56 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-x[74086]: 2024-03-20T08:50:56.498+0000 7f20f47de200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-20T08:50:56.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:56 smithi115 ceph-mon[72476]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:56.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:56 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:56.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:56 smithi122 ceph-mon[73495]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:56.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:56 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:57.071 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi026:vg_nvme/lv_4 2024-03-20T08:50:57.155 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 20 08:50:56 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-x[74086]: 2024-03-20T08:50:56.681+0000 7f20f47de200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-20T08:50:57.156 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 20 08:50:56 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-x[74086]: 2024-03-20T08:50:56.929+0000 7f20f47de200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-20T08:50:57.403 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:50:57.905 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 20 08:50:57 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-x[74086]: 2024-03-20T08:50:57.510+0000 7f20f47de200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-20T08:50:57.905 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 20 08:50:57 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-x[74086]: 2024-03-20T08:50:57.722+0000 7f20f47de200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-20T08:50:58.400 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 20 08:50:57 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-x[74086]: 2024-03-20T08:50:57.914+0000 7f20f47de200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-20T08:50:58.655 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 20 08:50:58 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-x[74086]: 2024-03-20T08:50:58.573+0000 7f20f47de200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-20T08:50:58.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:58 smithi115 ceph-mon[72476]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:58.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:58 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:58.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:58 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:58.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:58 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:50:58.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:58 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:58.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:58 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:50:58.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:58 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:58.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:58 smithi122 ceph-mon[73495]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:58.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:58 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:58.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:58 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:58.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:58 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:50:58.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:58 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:58.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:58 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:50:58.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:58 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:58.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:58 smithi026 ceph-mon[68309]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:50:58.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:58 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:58.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:58 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:58.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:58 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:50:58.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:58 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:50:58.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:58 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:50:58.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:58 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:50:59.061 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 20 08:50:58 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-x[74086]: 2024-03-20T08:50:58.768+0000 7f20f47de200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-20T08:50:59.061 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 20 08:50:58 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-x[74086]: 2024-03-20T08:50:58.947+0000 7f20f47de200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-20T08:50:59.402 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 20 08:50:59 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-x[74086]: 2024-03-20T08:50:59.060+0000 7f20f47de200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-20T08:50:59.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:59 smithi115 ceph-mon[72476]: Standby manager daemon x started 2024-03-20T08:50:59.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:59 smithi115 ceph-mon[72476]: from='mgr.? 172.21.15.115:0/3555983866' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-20T08:50:59.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:59 smithi115 ceph-mon[72476]: from='mgr.? 172.21.15.115:0/3555983866' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-20T08:50:59.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:59 smithi115 ceph-mon[72476]: from='mgr.? 172.21.15.115:0/3555983866' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-20T08:50:59.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:50:59 smithi115 ceph-mon[72476]: from='mgr.? 172.21.15.115:0/3555983866' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-20T08:50:59.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:59 smithi122 ceph-mon[73495]: Standby manager daemon x started 2024-03-20T08:50:59.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:59 smithi122 ceph-mon[73495]: from='mgr.? 172.21.15.115:0/3555983866' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-20T08:50:59.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:59 smithi122 ceph-mon[73495]: from='mgr.? 172.21.15.115:0/3555983866' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-20T08:50:59.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:59 smithi122 ceph-mon[73495]: from='mgr.? 172.21.15.115:0/3555983866' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-20T08:50:59.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:50:59 smithi122 ceph-mon[73495]: from='mgr.? 172.21.15.115:0/3555983866' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-20T08:50:59.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:59 smithi026 ceph-mon[68309]: Standby manager daemon x started 2024-03-20T08:50:59.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:59 smithi026 ceph-mon[68309]: from='mgr.? 172.21.15.115:0/3555983866' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-20T08:50:59.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:59 smithi026 ceph-mon[68309]: from='mgr.? 172.21.15.115:0/3555983866' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-20T08:50:59.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:59 smithi026 ceph-mon[68309]: from='mgr.? 172.21.15.115:0/3555983866' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-20T08:50:59.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:50:59 smithi026 ceph-mon[68309]: from='mgr.? 172.21.15.115:0/3555983866' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-20T08:51:00.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:00 smithi122 ceph-mon[73495]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:51:00.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:00 smithi122 ceph-mon[73495]: mgrmap e14: y(active, since 84s), standbys: x 2024-03-20T08:51:00.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:00 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-20T08:51:00.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:00 smithi122 ceph-mon[73495]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:51:00.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:00 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:51:00.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:00 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:51:00.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:00 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:51:00.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:00 smithi026 ceph-mon[68309]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:51:00.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:00 smithi026 ceph-mon[68309]: mgrmap e14: y(active, since 84s), standbys: x 2024-03-20T08:51:00.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:00 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-20T08:51:00.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:00 smithi026 ceph-mon[68309]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:51:00.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:00 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:51:00.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:00 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:51:00.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:00 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:51:00.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:00 smithi115 ceph-mon[72476]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:51:00.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:00 smithi115 ceph-mon[72476]: mgrmap e14: y(active, since 84s), standbys: x 2024-03-20T08:51:00.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:00 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-20T08:51:00.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:00 smithi115 ceph-mon[72476]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:51:00.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:00 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:51:00.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:00 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:51:00.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:00 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:51:02.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:02 smithi122 ceph-mon[73495]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:51:02.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:02 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2523068932' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad3fd9c0-316f-4fe5-a63b-d480087c1729"}]: dispatch 2024-03-20T08:51:02.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:02 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2523068932' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad3fd9c0-316f-4fe5-a63b-d480087c1729"}]': finished 2024-03-20T08:51:02.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:02 smithi122 ceph-mon[73495]: osdmap e5: 1 total, 0 up, 1 in 2024-03-20T08:51:02.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:02 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-20T08:51:02.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:02 smithi026 ceph-mon[68309]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:51:02.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:02 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2523068932' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad3fd9c0-316f-4fe5-a63b-d480087c1729"}]: dispatch 2024-03-20T08:51:02.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:02 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2523068932' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad3fd9c0-316f-4fe5-a63b-d480087c1729"}]': finished 2024-03-20T08:51:02.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:02 smithi026 ceph-mon[68309]: osdmap e5: 1 total, 0 up, 1 in 2024-03-20T08:51:02.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:02 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-20T08:51:02.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:02 smithi115 ceph-mon[72476]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:51:02.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:02 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2523068932' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad3fd9c0-316f-4fe5-a63b-d480087c1729"}]: dispatch 2024-03-20T08:51:02.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:02 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2523068932' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad3fd9c0-316f-4fe5-a63b-d480087c1729"}]': finished 2024-03-20T08:51:02.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:02 smithi115 ceph-mon[72476]: osdmap e5: 1 total, 0 up, 1 in 2024-03-20T08:51:02.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:02 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-20T08:51:03.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:03 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3603696201' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:51:03.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:03 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3603696201' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:51:03.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:03 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3603696201' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:51:04.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:04 smithi122 ceph-mon[73495]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:51:04.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:04 smithi026 ceph-mon[68309]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:51:04.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:04 smithi115 ceph-mon[72476]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:51:06.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:06 smithi122 ceph-mon[73495]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:51:06.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:06 smithi026 ceph-mon[68309]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:51:06.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:06 smithi115 ceph-mon[72476]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:51:08.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:08 smithi122 ceph-mon[73495]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:51:08.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:08 smithi026 ceph-mon[68309]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:51:08.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:08 smithi115 ceph-mon[72476]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:51:09.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:09 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-20T08:51:09.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:09 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:51:09.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:09 smithi122 ceph-mon[73495]: Deploying daemon osd.0 on smithi026 2024-03-20T08:51:09.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:09 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-20T08:51:09.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:09 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:51:09.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:09 smithi026 ceph-mon[68309]: Deploying daemon osd.0 on smithi026 2024-03-20T08:51:09.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:09 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-20T08:51:09.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:09 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:51:09.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:09 smithi115 ceph-mon[72476]: Deploying daemon osd.0 on smithi026 2024-03-20T08:51:10.600 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:10 smithi026 ceph-mon[68309]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:51:10.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:10 smithi122 ceph-mon[73495]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:51:10.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:10 smithi115 ceph-mon[72476]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:51:12.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:12 smithi115 ceph-mon[72476]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:51:12.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:12 smithi122 ceph-mon[73495]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:51:12.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:12 smithi026 ceph-mon[68309]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:51:14.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:14 smithi115 ceph-mon[72476]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:51:14.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:14 smithi122 ceph-mon[73495]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:51:14.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:14 smithi026 ceph-mon[68309]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:51:16.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:15 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:16.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:15 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:16.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:15 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:51:16.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:15 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:16.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:15 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:16.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:15 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:51:16.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:15 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:16.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:15 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:16.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:15 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:51:17.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:16 smithi115 ceph-mon[72476]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:51:17.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:16 smithi122 ceph-mon[73495]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:51:17.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:16 smithi026 ceph-mon[68309]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:51:18.481 INFO:teuthology.orchestra.run.smithi026.stdout:Created osd(s) 0 on host 'smithi026' 2024-03-20T08:51:19.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:18 smithi122 ceph-mon[73495]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:51:19.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:18 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:19.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:18 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:19.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:18 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:51:19.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:18 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:51:19.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:18 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:19.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:18 smithi122 ceph-mon[73495]: from='osd.0 [v2:172.21.15.26:6802/2139805293,v1:172.21.15.26:6803/2139805293]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-20T08:51:19.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:18 smithi122 ceph-mon[73495]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-20T08:51:19.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:18 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:19.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:18 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:19.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:18 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:51:19.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:18 smithi026 ceph-mon[68309]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:51:19.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:18 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:19.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:18 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:19.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:18 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:51:19.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:18 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:51:19.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:18 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:19.215 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:18 smithi026 ceph-mon[68309]: from='osd.0 [v2:172.21.15.26:6802/2139805293,v1:172.21.15.26:6803/2139805293]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-20T08:51:19.215 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:18 smithi026 ceph-mon[68309]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-20T08:51:19.215 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:18 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:19.215 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:18 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:19.215 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:18 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:51:19.238 DEBUG:teuthology.orchestra.run.smithi026:osd.0> sudo journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.0.service 2024-03-20T08:51:19.240 INFO:tasks.cephadm:Deploying osd.1 on smithi026 with /dev/vg_nvme/lv_3... 2024-03-20T08:51:19.240 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-20T08:51:19.273 INFO:journalctl@ceph.osd.0.smithi026.stdout:-- Logs begin at Wed 2024-03-20 08:37:24 UTC. -- 2024-03-20T08:51:19.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:18 smithi115 ceph-mon[72476]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:51:19.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:18 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:19.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:18 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:19.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:18 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:51:19.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:18 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:51:19.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:18 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:19.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:18 smithi115 ceph-mon[72476]: from='osd.0 [v2:172.21.15.26:6802/2139805293,v1:172.21.15.26:6803/2139805293]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-20T08:51:19.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:18 smithi115 ceph-mon[72476]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-20T08:51:19.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:18 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:19.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:18 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:19.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:18 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:51:20.062 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:51:20.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:19 smithi026 ceph-mon[68309]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-20T08:51:20.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:19 smithi026 ceph-mon[68309]: osdmap e6: 1 total, 0 up, 1 in 2024-03-20T08:51:20.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:19 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-20T08:51:20.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:19 smithi026 ceph-mon[68309]: from='osd.0 [v2:172.21.15.26:6802/2139805293,v1:172.21.15.26:6803/2139805293]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-20T08:51:20.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:19 smithi026 ceph-mon[68309]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-20T08:51:20.221 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 20 08:51:19 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-0[86340]: 2024-03-20T08:51:19.955+0000 7f97d7d5a700 -1 osd.0 0 waiting for initial osdmap 2024-03-20T08:51:20.221 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 20 08:51:19 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-0[86340]: 2024-03-20T08:51:19.957+0000 7f97d3b4d700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-20T08:51:20.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:19 smithi115 ceph-mon[72476]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-20T08:51:20.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:19 smithi115 ceph-mon[72476]: osdmap e6: 1 total, 0 up, 1 in 2024-03-20T08:51:20.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:19 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-20T08:51:20.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:19 smithi115 ceph-mon[72476]: from='osd.0 [v2:172.21.15.26:6802/2139805293,v1:172.21.15.26:6803/2139805293]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-20T08:51:20.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:19 smithi115 ceph-mon[72476]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-20T08:51:20.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:19 smithi122 ceph-mon[73495]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-20T08:51:20.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:19 smithi122 ceph-mon[73495]: osdmap e6: 1 total, 0 up, 1 in 2024-03-20T08:51:20.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:19 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-20T08:51:20.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:19 smithi122 ceph-mon[73495]: from='osd.0 [v2:172.21.15.26:6802/2139805293,v1:172.21.15.26:6803/2139805293]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-20T08:51:20.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:19 smithi122 ceph-mon[73495]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-20T08:51:21.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:20 smithi026 ceph-mon[68309]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:51:21.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:20 smithi026 ceph-mon[68309]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-03-20T08:51:21.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:20 smithi026 ceph-mon[68309]: osdmap e7: 1 total, 0 up, 1 in 2024-03-20T08:51:21.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:20 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-20T08:51:21.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:20 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-20T08:51:21.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:20 smithi115 ceph-mon[72476]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:51:21.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:20 smithi115 ceph-mon[72476]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-03-20T08:51:21.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:20 smithi115 ceph-mon[72476]: osdmap e7: 1 total, 0 up, 1 in 2024-03-20T08:51:21.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:20 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-20T08:51:21.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:20 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-20T08:51:21.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:20 smithi122 ceph-mon[73495]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:51:21.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:20 smithi122 ceph-mon[73495]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-03-20T08:51:21.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:20 smithi122 ceph-mon[73495]: osdmap e7: 1 total, 0 up, 1 in 2024-03-20T08:51:21.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:20 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-20T08:51:21.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:20 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-20T08:51:22.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:21 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:51:22.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:21 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:51:22.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:21 smithi026 ceph-mon[68309]: osd.0 [v2:172.21.15.26:6802/2139805293,v1:172.21.15.26:6803/2139805293] boot 2024-03-20T08:51:22.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:21 smithi026 ceph-mon[68309]: osdmap e8: 1 total, 1 up, 1 in 2024-03-20T08:51:22.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:21 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-20T08:51:22.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:21 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:51:22.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:21 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:51:22.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:21 smithi115 ceph-mon[72476]: osd.0 [v2:172.21.15.26:6802/2139805293,v1:172.21.15.26:6803/2139805293] boot 2024-03-20T08:51:22.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:21 smithi115 ceph-mon[72476]: osdmap e8: 1 total, 1 up, 1 in 2024-03-20T08:51:22.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:21 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-20T08:51:22.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:21 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:51:22.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:21 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:51:22.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:21 smithi122 ceph-mon[73495]: osd.0 [v2:172.21.15.26:6802/2139805293,v1:172.21.15.26:6803/2139805293] boot 2024-03-20T08:51:22.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:21 smithi122 ceph-mon[73495]: osdmap e8: 1 total, 1 up, 1 in 2024-03-20T08:51:22.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:21 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-20T08:51:23.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:22 smithi026 ceph-mon[68309]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:51:23.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:22 smithi026 ceph-mon[68309]: osdmap e9: 1 total, 1 up, 1 in 2024-03-20T08:51:23.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:22 smithi115 ceph-mon[72476]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:51:23.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:22 smithi115 ceph-mon[72476]: osdmap e9: 1 total, 1 up, 1 in 2024-03-20T08:51:23.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:22 smithi122 ceph-mon[73495]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-20T08:51:23.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:22 smithi122 ceph-mon[73495]: osdmap e9: 1 total, 1 up, 1 in 2024-03-20T08:51:23.567 INFO:teuthology.orchestra.run.smithi026.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-20T08:51:23.567 INFO:teuthology.orchestra.run.smithi026.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-20T08:51:23.567 INFO:teuthology.orchestra.run.smithi026.stderr: stderr: 10+0 records in 2024-03-20T08:51:23.567 INFO:teuthology.orchestra.run.smithi026.stderr:10+0 records out 2024-03-20T08:51:23.567 INFO:teuthology.orchestra.run.smithi026.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00818344 s, 1.3 GB/s 2024-03-20T08:51:23.568 INFO:teuthology.orchestra.run.smithi026.stderr:--> Zapping successful for: 2024-03-20T08:51:24.468 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi026:vg_nvme/lv_3 2024-03-20T08:51:24.744 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:51:25.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:24 smithi122 ceph-mon[73495]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:25.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:24 smithi122 ceph-mon[73495]: Detected new or changed devices on smithi026 2024-03-20T08:51:25.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:24 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:25.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:24 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:25.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:24 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:51:25.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:24 smithi122 ceph-mon[73495]: Adjusting osd_memory_target on smithi026 to 17126M 2024-03-20T08:51:25.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:24 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:25.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:24 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:51:25.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:24 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:51:25.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:24 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:25.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:24 smithi026 ceph-mon[68309]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:25.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:24 smithi026 ceph-mon[68309]: Detected new or changed devices on smithi026 2024-03-20T08:51:25.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:24 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:25.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:24 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:25.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:24 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:51:25.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:24 smithi026 ceph-mon[68309]: Adjusting osd_memory_target on smithi026 to 17126M 2024-03-20T08:51:25.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:24 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:25.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:24 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:51:25.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:24 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:51:25.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:24 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:25.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:24 smithi115 ceph-mon[72476]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:25.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:24 smithi115 ceph-mon[72476]: Detected new or changed devices on smithi026 2024-03-20T08:51:25.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:24 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:25.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:24 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:25.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:24 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:51:25.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:24 smithi115 ceph-mon[72476]: Adjusting osd_memory_target on smithi026 to 17126M 2024-03-20T08:51:25.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:24 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:25.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:24 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:51:25.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:24 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:51:25.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:24 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:27.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:26 smithi122 ceph-mon[73495]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:27.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:51:27.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:51:27.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:51:27.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:26 smithi026 ceph-mon[68309]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:27.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:26 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:51:27.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:26 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:51:27.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:26 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:51:27.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:26 smithi115 ceph-mon[72476]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:27.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:26 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:51:27.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:26 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:51:27.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:26 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:51:27.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:27 smithi026 ceph-mon[68309]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:51:28.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:27 smithi122 ceph-mon[73495]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:51:28.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:27 smithi115 ceph-mon[72476]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:51:29.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:28 smithi122 ceph-mon[73495]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:29.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:28 smithi026 ceph-mon[68309]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:29.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:28 smithi115 ceph-mon[72476]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:30.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:29 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4119598631' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e85b94f5-2954-47a1-9c20-096f2606eb04"}]: dispatch 2024-03-20T08:51:30.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:29 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4119598631' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e85b94f5-2954-47a1-9c20-096f2606eb04"}]': finished 2024-03-20T08:51:30.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:29 smithi026 ceph-mon[68309]: osdmap e10: 2 total, 1 up, 2 in 2024-03-20T08:51:30.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:29 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-20T08:51:30.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:29 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4119598631' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e85b94f5-2954-47a1-9c20-096f2606eb04"}]: dispatch 2024-03-20T08:51:30.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:29 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4119598631' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e85b94f5-2954-47a1-9c20-096f2606eb04"}]': finished 2024-03-20T08:51:30.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:29 smithi115 ceph-mon[72476]: osdmap e10: 2 total, 1 up, 2 in 2024-03-20T08:51:30.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:29 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-20T08:51:30.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:29 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4119598631' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e85b94f5-2954-47a1-9c20-096f2606eb04"}]: dispatch 2024-03-20T08:51:30.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:29 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4119598631' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e85b94f5-2954-47a1-9c20-096f2606eb04"}]': finished 2024-03-20T08:51:30.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:29 smithi122 ceph-mon[73495]: osdmap e10: 2 total, 1 up, 2 in 2024-03-20T08:51:30.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:29 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-20T08:51:31.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:30 smithi026 ceph-mon[68309]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:31.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:30 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2855352872' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:51:31.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:30 smithi115 ceph-mon[72476]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:31.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:30 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2855352872' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:51:31.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:30 smithi122 ceph-mon[73495]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:31.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:30 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2855352872' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:51:33.087 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:32 smithi026 ceph-mon[68309]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:33.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:32 smithi115 ceph-mon[72476]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:33.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:32 smithi122 ceph-mon[73495]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:35.059 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:34 smithi026 ceph-mon[68309]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:35.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:34 smithi115 ceph-mon[72476]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:35.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:34 smithi122 ceph-mon[73495]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:37.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:36 smithi026 ceph-mon[68309]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:37.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:36 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-20T08:51:37.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:36 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:51:37.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:36 smithi115 ceph-mon[72476]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:37.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:36 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-20T08:51:37.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:36 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:51:37.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:36 smithi122 ceph-mon[73495]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:37.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:36 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-20T08:51:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:36 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:51:38.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:37 smithi026 ceph-mon[68309]: Deploying daemon osd.1 on smithi026 2024-03-20T08:51:38.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:37 smithi115 ceph-mon[72476]: Deploying daemon osd.1 on smithi026 2024-03-20T08:51:38.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:37 smithi122 ceph-mon[73495]: Deploying daemon osd.1 on smithi026 2024-03-20T08:51:39.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:38 smithi026 ceph-mon[68309]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:39.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:38 smithi115 ceph-mon[72476]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:39.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:38 smithi122 ceph-mon[73495]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:41.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:40 smithi026 ceph-mon[68309]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:41.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:40 smithi115 ceph-mon[72476]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:41.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:40 smithi122 ceph-mon[73495]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:43.062 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:42 smithi026 ceph-mon[68309]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:43.062 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:42 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:43.063 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:42 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:43.063 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:42 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:51:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:42 smithi115 ceph-mon[72476]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:51:43.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:42 smithi122 ceph-mon[73495]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:43.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:42 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:43.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:42 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:42 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:51:45.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:44 smithi026 ceph-mon[68309]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:45.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:44 smithi115 ceph-mon[72476]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:45.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:44 smithi122 ceph-mon[73495]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:46.512 INFO:teuthology.orchestra.run.smithi026.stdout:Created osd(s) 1 on host 'smithi026' 2024-03-20T08:51:46.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:46 smithi026 ceph-mon[68309]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:46.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:46.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:46.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:46 smithi026 ceph-mon[68309]: from='osd.1 [v2:172.21.15.26:6810/3853355134,v1:172.21.15.26:6811/3853355134]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-20T08:51:46.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:46 smithi026 ceph-mon[68309]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-20T08:51:46.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:51:46.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:51:46.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:46.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:46.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:46.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:51:46.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:46 smithi115 ceph-mon[72476]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:46.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:46.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:46.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:46 smithi115 ceph-mon[72476]: from='osd.1 [v2:172.21.15.26:6810/3853355134,v1:172.21.15.26:6811/3853355134]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-20T08:51:46.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:46 smithi115 ceph-mon[72476]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-20T08:51:46.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:51:46.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:51:46.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:46.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:46.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:46.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:51:46.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:46 smithi122 ceph-mon[73495]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:46.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:46.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:46.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:46 smithi122 ceph-mon[73495]: from='osd.1 [v2:172.21.15.26:6810/3853355134,v1:172.21.15.26:6811/3853355134]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-20T08:51:46.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:46 smithi122 ceph-mon[73495]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-20T08:51:46.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:51:46.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:51:46.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:46.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:46.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:46.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:51:47.234 DEBUG:teuthology.orchestra.run.smithi026:osd.1> sudo journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.1.service 2024-03-20T08:51:47.238 INFO:tasks.cephadm:Deploying osd.2 on smithi026 with /dev/vg_nvme/lv_2... 2024-03-20T08:51:47.239 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-20T08:51:47.270 INFO:journalctl@ceph.osd.1.smithi026.stdout:-- Logs begin at Wed 2024-03-20 08:37:24 UTC. -- 2024-03-20T08:51:47.827 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:47 smithi026 ceph-mon[68309]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-20T08:51:47.827 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:47 smithi026 ceph-mon[68309]: from='osd.1 [v2:172.21.15.26:6810/3853355134,v1:172.21.15.26:6811/3853355134]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-20T08:51:47.827 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:47 smithi026 ceph-mon[68309]: osdmap e11: 2 total, 1 up, 2 in 2024-03-20T08:51:47.827 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:47 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-20T08:51:47.827 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:47 smithi026 ceph-mon[68309]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-20T08:51:47.828 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 20 08:51:47 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-1[94087]: 2024-03-20T08:51:47.534+0000 7efe17790700 -1 osd.1 0 waiting for initial osdmap 2024-03-20T08:51:47.828 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 20 08:51:47 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-1[94087]: 2024-03-20T08:51:47.537+0000 7efe13583700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-20T08:51:47.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:47 smithi115 ceph-mon[72476]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-20T08:51:47.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:47 smithi115 ceph-mon[72476]: from='osd.1 [v2:172.21.15.26:6810/3853355134,v1:172.21.15.26:6811/3853355134]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-20T08:51:47.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:47 smithi115 ceph-mon[72476]: osdmap e11: 2 total, 1 up, 2 in 2024-03-20T08:51:47.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:47 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-20T08:51:47.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:47 smithi115 ceph-mon[72476]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-20T08:51:47.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:47 smithi122 ceph-mon[73495]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-20T08:51:47.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:47 smithi122 ceph-mon[73495]: from='osd.1 [v2:172.21.15.26:6810/3853355134,v1:172.21.15.26:6811/3853355134]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-20T08:51:47.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:47 smithi122 ceph-mon[73495]: osdmap e11: 2 total, 1 up, 2 in 2024-03-20T08:51:47.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:47 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-20T08:51:47.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:47 smithi122 ceph-mon[73495]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-20T08:51:48.157 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:51:48.636 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:48 smithi026 ceph-mon[68309]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:48.636 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:48 smithi026 ceph-mon[68309]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-03-20T08:51:48.636 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:48 smithi026 ceph-mon[68309]: osdmap e12: 2 total, 1 up, 2 in 2024-03-20T08:51:48.636 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:48 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-20T08:51:48.636 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:48 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-20T08:51:48.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:48 smithi115 ceph-mon[72476]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:48.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:48 smithi115 ceph-mon[72476]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-03-20T08:51:48.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:48 smithi115 ceph-mon[72476]: osdmap e12: 2 total, 1 up, 2 in 2024-03-20T08:51:48.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:48 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-20T08:51:48.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:48 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-20T08:51:48.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:48 smithi122 ceph-mon[73495]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:48.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:48 smithi122 ceph-mon[73495]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-03-20T08:51:48.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:48 smithi122 ceph-mon[73495]: osdmap e12: 2 total, 1 up, 2 in 2024-03-20T08:51:48.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:48 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-20T08:51:48.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:48 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-20T08:51:49.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:49 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:51:49.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:49 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:51:49.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:49 smithi026 ceph-mon[68309]: osd.1 [v2:172.21.15.26:6810/3853355134,v1:172.21.15.26:6811/3853355134] boot 2024-03-20T08:51:49.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:49 smithi026 ceph-mon[68309]: osdmap e13: 2 total, 2 up, 2 in 2024-03-20T08:51:49.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:49 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-20T08:51:49.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:49 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:51:49.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:49 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:51:49.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:49 smithi115 ceph-mon[72476]: osd.1 [v2:172.21.15.26:6810/3853355134,v1:172.21.15.26:6811/3853355134] boot 2024-03-20T08:51:49.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:49 smithi115 ceph-mon[72476]: osdmap e13: 2 total, 2 up, 2 in 2024-03-20T08:51:49.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:49 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-20T08:51:49.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:49 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:51:49.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:49 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:51:49.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:49 smithi122 ceph-mon[73495]: osd.1 [v2:172.21.15.26:6810/3853355134,v1:172.21.15.26:6811/3853355134] boot 2024-03-20T08:51:49.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:49 smithi122 ceph-mon[73495]: osdmap e13: 2 total, 2 up, 2 in 2024-03-20T08:51:49.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:49 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-20T08:51:50.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:50 smithi115 ceph-mon[72476]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:50.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:50 smithi115 ceph-mon[72476]: osdmap e14: 2 total, 2 up, 2 in 2024-03-20T08:51:50.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:50 smithi122 ceph-mon[73495]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:50.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:50 smithi122 ceph-mon[73495]: osdmap e14: 2 total, 2 up, 2 in 2024-03-20T08:51:50.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:50 smithi026 ceph-mon[68309]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-20T08:51:50.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:50 smithi026 ceph-mon[68309]: osdmap e14: 2 total, 2 up, 2 in 2024-03-20T08:51:51.158 INFO:teuthology.orchestra.run.smithi026.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-20T08:51:51.159 INFO:teuthology.orchestra.run.smithi026.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-20T08:51:51.159 INFO:teuthology.orchestra.run.smithi026.stderr: stderr: 10+0 records in 2024-03-20T08:51:51.159 INFO:teuthology.orchestra.run.smithi026.stderr:10+0 records out 2024-03-20T08:51:51.159 INFO:teuthology.orchestra.run.smithi026.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00888656 s, 1.2 GB/s 2024-03-20T08:51:51.159 INFO:teuthology.orchestra.run.smithi026.stderr:--> Zapping successful for: 2024-03-20T08:51:52.650 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi026:vg_nvme/lv_2 2024-03-20T08:51:52.927 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:51:53.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:53 smithi115 ceph-mon[72476]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:51:53.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:53 smithi115 ceph-mon[72476]: Detected new or changed devices on smithi026 2024-03-20T08:51:53.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:53 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:53.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:53 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:53.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:53 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:51:53.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:53 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:51:53.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:53 smithi115 ceph-mon[72476]: Adjusting osd_memory_target on smithi026 to 8563M 2024-03-20T08:51:53.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:53 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:53.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:53 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:51:53.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:53 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:51:53.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:53 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:53.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:53 smithi122 ceph-mon[73495]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:51:53.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:53 smithi122 ceph-mon[73495]: Detected new or changed devices on smithi026 2024-03-20T08:51:53.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:53 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:53.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:53 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:53.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:53 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:51:53.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:53 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:51:53.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:53 smithi122 ceph-mon[73495]: Adjusting osd_memory_target on smithi026 to 8563M 2024-03-20T08:51:53.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:53 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:53.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:53 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:51:53.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:53 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:51:53.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:53 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:53.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:53 smithi026 ceph-mon[68309]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:51:53.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:53 smithi026 ceph-mon[68309]: Detected new or changed devices on smithi026 2024-03-20T08:51:53.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:53 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:53.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:53 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:53.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:53 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:51:53.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:53 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:51:53.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:53 smithi026 ceph-mon[68309]: Adjusting osd_memory_target on smithi026 to 8563M 2024-03-20T08:51:53.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:53 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:53.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:53 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:51:53.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:53 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:51:53.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:53 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:55.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:55 smithi115 ceph-mon[72476]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:51:55.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:55 smithi115 ceph-mon[72476]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:51:55.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:55 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:51:55.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:55 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:51:55.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:55 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:51:55.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:55 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:51:55.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:55 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:51:55.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:55 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:51:55.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:55 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:51:55.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:55 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:51:55.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:55 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:51:55.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:55 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:55.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:55 smithi122 ceph-mon[73495]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:51:55.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:55 smithi122 ceph-mon[73495]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:51:55.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:55 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:51:55.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:55 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:51:55.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:55 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:51:55.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:55 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:51:55.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:55 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:51:55.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:55 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:51:55.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:55 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:51:55.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:55 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:51:55.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:55 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:51:55.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:55 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:55.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:55 smithi026 ceph-mon[68309]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:51:55.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:55 smithi026 ceph-mon[68309]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:51:55.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:55 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:51:55.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:55 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:51:55.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:55 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:51:55.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:55 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:51:55.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:55 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:51:55.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:55 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:51:55.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:55 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:51:55.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:55 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:51:55.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:55 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:51:55.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:55 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:51:57.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:57 smithi026 ceph-mon[68309]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:51:57.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:57 smithi115 ceph-mon[72476]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:51:57.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:57 smithi122 ceph-mon[73495]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:51:58.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:58 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/559674543' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd2b10f0-741c-47a4-b224-ac6bfe2c14ef"}]: dispatch 2024-03-20T08:51:58.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:58 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/559674543' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dd2b10f0-741c-47a4-b224-ac6bfe2c14ef"}]': finished 2024-03-20T08:51:58.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:58 smithi115 ceph-mon[72476]: osdmap e15: 3 total, 2 up, 3 in 2024-03-20T08:51:58.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:58 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-20T08:51:58.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:58 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/559674543' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd2b10f0-741c-47a4-b224-ac6bfe2c14ef"}]: dispatch 2024-03-20T08:51:58.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:58 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/559674543' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dd2b10f0-741c-47a4-b224-ac6bfe2c14ef"}]': finished 2024-03-20T08:51:58.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:58 smithi122 ceph-mon[73495]: osdmap e15: 3 total, 2 up, 3 in 2024-03-20T08:51:58.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:58 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-20T08:51:58.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:58 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/559674543' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd2b10f0-741c-47a4-b224-ac6bfe2c14ef"}]: dispatch 2024-03-20T08:51:58.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:58 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/559674543' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dd2b10f0-741c-47a4-b224-ac6bfe2c14ef"}]': finished 2024-03-20T08:51:58.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:58 smithi026 ceph-mon[68309]: osdmap e15: 3 total, 2 up, 3 in 2024-03-20T08:51:58.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:58 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-20T08:51:59.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:59 smithi115 ceph-mon[72476]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:51:59.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:59 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1696092045' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:51:59.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:51:59 smithi115 ceph-mon[72476]: from='client.14280 172.21.15.26:0/1696092045' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:51:59.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:59 smithi122 ceph-mon[73495]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:51:59.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:59 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1696092045' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:51:59.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:51:59 smithi122 ceph-mon[73495]: from='client.14280 172.21.15.26:0/1696092045' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:51:59.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:59 smithi026 ceph-mon[68309]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:51:59.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:59 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1696092045' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:51:59.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:51:59 smithi026 ceph-mon[68309]: from='client.14280 172.21.15.26:0/1696092045' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:52:01.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:01 smithi115 ceph-mon[72476]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:52:01.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:01 smithi122 ceph-mon[73495]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:52:01.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:01 smithi026 ceph-mon[68309]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:52:03.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:03 smithi115 ceph-mon[72476]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:52:03.417 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:03 smithi026 ceph-mon[68309]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:52:03.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:03 smithi122 ceph-mon[73495]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:52:05.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:05 smithi122 ceph-mon[73495]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:52:05.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:05 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-20T08:52:05.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:05 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:52:05.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:05 smithi026 ceph-mon[68309]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:52:05.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:05 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-20T08:52:05.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:05 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:52:05.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:05 smithi115 ceph-mon[72476]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:52:05.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:05 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-20T08:52:05.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:05 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:52:06.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:06 smithi122 ceph-mon[73495]: Deploying daemon osd.2 on smithi026 2024-03-20T08:52:06.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:06 smithi026 ceph-mon[68309]: Deploying daemon osd.2 on smithi026 2024-03-20T08:52:06.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:06 smithi115 ceph-mon[72476]: Deploying daemon osd.2 on smithi026 2024-03-20T08:52:07.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:07 smithi122 ceph-mon[73495]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:52:07.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:07 smithi026 ceph-mon[68309]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:52:07.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:07 smithi115 ceph-mon[72476]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:52:09.436 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:09 smithi026 ceph-mon[68309]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:52:09.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:09 smithi122 ceph-mon[73495]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:52:09.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:09 smithi115 ceph-mon[72476]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:52:11.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:11 smithi122 ceph-mon[73495]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:52:11.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:11 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:11.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:11 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:11.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:11 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:52:11.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:11 smithi026 ceph-mon[68309]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:52:11.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:11 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:11.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:11 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:11.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:11 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:52:11.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:11 smithi115 ceph-mon[72476]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:52:11.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:11 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:11.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:11 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:11.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:11 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:52:13.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:13 smithi122 ceph-mon[73495]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:52:13.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:13 smithi026 ceph-mon[68309]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:52:13.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:13 smithi115 ceph-mon[72476]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:52:14.791 INFO:teuthology.orchestra.run.smithi026.stdout:Created osd(s) 2 on host 'smithi026' 2024-03-20T08:52:15.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:14 smithi122 ceph-mon[73495]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:52:15.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:14 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:15.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:14 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:15.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:14 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:52:15.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:14 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:52:15.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:14 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:15.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:14 smithi122 ceph-mon[73495]: from='osd.2 [v2:172.21.15.26:6818/3993357967,v1:172.21.15.26:6819/3993357967]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-20T08:52:15.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:14 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:15.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:14 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:15.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:14 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:52:15.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:14 smithi026 ceph-mon[68309]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:52:15.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:14 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:15.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:14 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:15.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:14 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:52:15.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:14 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:52:15.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:14 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:15.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:14 smithi026 ceph-mon[68309]: from='osd.2 [v2:172.21.15.26:6818/3993357967,v1:172.21.15.26:6819/3993357967]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-20T08:52:15.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:14 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:15.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:14 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:15.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:14 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:52:15.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:14 smithi115 ceph-mon[72476]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:52:15.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:14 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:15.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:14 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:15.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:14 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:52:15.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:14 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:52:15.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:14 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:15.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:14 smithi115 ceph-mon[72476]: from='osd.2 [v2:172.21.15.26:6818/3993357967,v1:172.21.15.26:6819/3993357967]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-20T08:52:15.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:14 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:15.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:14 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:15.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:14 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:52:15.616 DEBUG:teuthology.orchestra.run.smithi026:osd.2> sudo journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.2.service 2024-03-20T08:52:15.618 INFO:tasks.cephadm:Deploying osd.3 on smithi026 with /dev/vg_nvme/lv_1... 2024-03-20T08:52:15.618 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-20T08:52:15.656 INFO:journalctl@ceph.osd.2.smithi026.stdout:-- Logs begin at Wed 2024-03-20 08:37:24 UTC. -- 2024-03-20T08:52:16.216 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:15 smithi026 ceph-mon[68309]: from='osd.2 [v2:172.21.15.26:6818/3993357967,v1:172.21.15.26:6819/3993357967]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-20T08:52:16.216 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:15 smithi026 ceph-mon[68309]: osdmap e16: 3 total, 2 up, 3 in 2024-03-20T08:52:16.217 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:15 smithi026 ceph-mon[68309]: from='osd.2 [v2:172.21.15.26:6818/3993357967,v1:172.21.15.26:6819/3993357967]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-20T08:52:16.217 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:15 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-20T08:52:16.217 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 20 08:52:15 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-2[101308]: 2024-03-20T08:52:15.955+0000 7fb67366b700 -1 osd.2 0 waiting for initial osdmap 2024-03-20T08:52:16.217 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 20 08:52:15 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-2[101308]: 2024-03-20T08:52:15.959+0000 7fb66f45e700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-20T08:52:16.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:15 smithi115 ceph-mon[72476]: from='osd.2 [v2:172.21.15.26:6818/3993357967,v1:172.21.15.26:6819/3993357967]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-20T08:52:16.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:15 smithi115 ceph-mon[72476]: osdmap e16: 3 total, 2 up, 3 in 2024-03-20T08:52:16.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:15 smithi115 ceph-mon[72476]: from='osd.2 [v2:172.21.15.26:6818/3993357967,v1:172.21.15.26:6819/3993357967]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-20T08:52:16.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:15 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-20T08:52:16.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:15 smithi122 ceph-mon[73495]: from='osd.2 [v2:172.21.15.26:6818/3993357967,v1:172.21.15.26:6819/3993357967]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-20T08:52:16.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:15 smithi122 ceph-mon[73495]: osdmap e16: 3 total, 2 up, 3 in 2024-03-20T08:52:16.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:15 smithi122 ceph-mon[73495]: from='osd.2 [v2:172.21.15.26:6818/3993357967,v1:172.21.15.26:6819/3993357967]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-20T08:52:16.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:15 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-20T08:52:16.566 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:52:17.078 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:16 smithi026 ceph-mon[68309]: pgmap v86: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:52:17.078 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:16 smithi026 ceph-mon[68309]: from='osd.2 [v2:172.21.15.26:6818/3993357967,v1:172.21.15.26:6819/3993357967]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-03-20T08:52:17.078 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:16 smithi026 ceph-mon[68309]: osdmap e17: 3 total, 2 up, 3 in 2024-03-20T08:52:17.078 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:16 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-20T08:52:17.078 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:16 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-20T08:52:17.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:16 smithi115 ceph-mon[72476]: pgmap v86: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:52:17.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:16 smithi115 ceph-mon[72476]: from='osd.2 [v2:172.21.15.26:6818/3993357967,v1:172.21.15.26:6819/3993357967]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-03-20T08:52:17.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:16 smithi115 ceph-mon[72476]: osdmap e17: 3 total, 2 up, 3 in 2024-03-20T08:52:17.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:16 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-20T08:52:17.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:16 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-20T08:52:17.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:16 smithi122 ceph-mon[73495]: pgmap v86: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:52:17.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:16 smithi122 ceph-mon[73495]: from='osd.2 [v2:172.21.15.26:6818/3993357967,v1:172.21.15.26:6819/3993357967]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-03-20T08:52:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:16 smithi122 ceph-mon[73495]: osdmap e17: 3 total, 2 up, 3 in 2024-03-20T08:52:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:16 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-20T08:52:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:16 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-20T08:52:18.086 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:17 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:52:18.086 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:17 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:52:18.086 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:17 smithi026 ceph-mon[68309]: osd.2 [v2:172.21.15.26:6818/3993357967,v1:172.21.15.26:6819/3993357967] boot 2024-03-20T08:52:18.086 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:17 smithi026 ceph-mon[68309]: osdmap e18: 3 total, 3 up, 3 in 2024-03-20T08:52:18.087 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:17 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-20T08:52:18.087 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:17 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-20T08:52:18.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:17 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:52:18.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:17 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:52:18.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:17 smithi115 ceph-mon[72476]: osd.2 [v2:172.21.15.26:6818/3993357967,v1:172.21.15.26:6819/3993357967] boot 2024-03-20T08:52:18.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:17 smithi115 ceph-mon[72476]: osdmap e18: 3 total, 3 up, 3 in 2024-03-20T08:52:18.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:17 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-20T08:52:18.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:17 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-20T08:52:18.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:17 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:52:18.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:17 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:52:18.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:17 smithi122 ceph-mon[73495]: osd.2 [v2:172.21.15.26:6818/3993357967,v1:172.21.15.26:6819/3993357967] boot 2024-03-20T08:52:18.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:17 smithi122 ceph-mon[73495]: osdmap e18: 3 total, 3 up, 3 in 2024-03-20T08:52:18.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:17 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-20T08:52:18.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:17 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-20T08:52:19.093 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:18 smithi026 ceph-mon[68309]: pgmap v89: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:52:19.093 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:18 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-20T08:52:19.093 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:18 smithi026 ceph-mon[68309]: osdmap e19: 3 total, 3 up, 3 in 2024-03-20T08:52:19.093 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:18 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-20T08:52:19.093 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 20 08:52:19 smithi026 sudo[105102]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-20T08:52:19.094 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 20 08:52:19 smithi026 sudo[105102]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-20T08:52:19.094 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 20 08:52:19 smithi026 sudo[105102]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-20T08:52:19.094 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 20 08:52:19 smithi026 sudo[105102]: pam_unix(sudo:session): session closed for user root 2024-03-20T08:52:19.094 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 20 08:52:19 smithi026 sudo[105343]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-20T08:52:19.254 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:18 smithi122 ceph-mon[73495]: pgmap v89: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:52:19.254 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:18 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-20T08:52:19.254 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:18 smithi122 ceph-mon[73495]: osdmap e19: 3 total, 3 up, 3 in 2024-03-20T08:52:19.254 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:18 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-20T08:52:19.318 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:18 smithi115 ceph-mon[72476]: pgmap v89: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-20T08:52:19.318 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:18 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-20T08:52:19.319 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:18 smithi115 ceph-mon[72476]: osdmap e19: 3 total, 3 up, 3 in 2024-03-20T08:52:19.319 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:18 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-20T08:52:19.470 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 20 08:52:19 smithi026 sudo[105343]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-20T08:52:19.470 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 20 08:52:19 smithi026 sudo[105343]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-20T08:52:19.470 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 20 08:52:19 smithi026 sudo[105343]: pam_unix(sudo:session): session closed for user root 2024-03-20T08:52:19.471 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 20 08:52:19 smithi026 sudo[105888]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-20T08:52:19.471 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 20 08:52:19 smithi026 sudo[105888]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-20T08:52:19.471 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 20 08:52:19 smithi026 sudo[105888]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-20T08:52:19.471 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 20 08:52:19 smithi026 sudo[105888]: pam_unix(sudo:session): session closed for user root 2024-03-20T08:52:19.471 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 20 08:52:19 smithi026 sudo[106090]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-20T08:52:19.471 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 20 08:52:19 smithi026 sudo[106090]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-20T08:52:19.471 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 20 08:52:19 smithi026 sudo[106090]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-20T08:52:19.471 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 20 08:52:19 smithi026 sudo[106090]: pam_unix(sudo:session): session closed for user root 2024-03-20T08:52:19.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:19 smithi026 sudo[106205]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-20T08:52:19.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:19 smithi026 sudo[106205]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-20T08:52:19.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:19 smithi026 sudo[106205]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-20T08:52:19.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:19 smithi026 sudo[106205]: pam_unix(sudo:session): session closed for user root 2024-03-20T08:52:19.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:19 smithi026 sudo[106424]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-20T08:52:19.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:19 smithi026 sudo[106424]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-20T08:52:19.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:19 smithi026 sudo[106424]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-20T08:52:19.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:19 smithi026 sudo[106424]: pam_unix(sudo:session): session closed for user root 2024-03-20T08:52:19.472 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 20 08:52:19 smithi026 sudo[105567]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-20T08:52:19.472 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 20 08:52:19 smithi026 sudo[105567]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-20T08:52:19.473 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 20 08:52:19 smithi026 sudo[105567]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-20T08:52:19.473 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 20 08:52:19 smithi026 sudo[105567]: pam_unix(sudo:session): session closed for user root 2024-03-20T08:52:19.473 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 20 08:52:19 smithi026 sudo[105773]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-20T08:52:19.473 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 20 08:52:19 smithi026 sudo[105773]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-20T08:52:19.473 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 20 08:52:19 smithi026 sudo[105773]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-20T08:52:19.473 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 20 08:52:19 smithi026 sudo[105773]: pam_unix(sudo:session): session closed for user root 2024-03-20T08:52:19.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:19 smithi115 sudo[74931]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-20T08:52:19.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:19 smithi115 sudo[74931]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-20T08:52:19.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:19 smithi115 sudo[74931]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-20T08:52:19.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:19 smithi115 sudo[74931]: pam_unix(sudo:session): session closed for user root 2024-03-20T08:52:19.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:19 smithi115 sudo[74934]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-20T08:52:19.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:19 smithi115 sudo[74934]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-20T08:52:19.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:19 smithi115 sudo[74934]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-20T08:52:19.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:19 smithi115 sudo[74934]: pam_unix(sudo:session): session closed for user root 2024-03-20T08:52:19.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:19 smithi122 sudo[75931]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-20T08:52:19.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:19 smithi122 sudo[75931]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-20T08:52:19.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:19 smithi122 sudo[75931]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-20T08:52:19.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:19 smithi122 sudo[75931]: pam_unix(sudo:session): session closed for user root 2024-03-20T08:52:19.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:19 smithi122 sudo[75934]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-20T08:52:19.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:19 smithi122 sudo[75934]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-20T08:52:19.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:19 smithi122 sudo[75934]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-20T08:52:19.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:19 smithi122 sudo[75934]: pam_unix(sudo:session): session closed for user root 2024-03-20T08:52:19.841 INFO:teuthology.orchestra.run.smithi026.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-20T08:52:19.841 INFO:teuthology.orchestra.run.smithi026.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-20T08:52:19.841 INFO:teuthology.orchestra.run.smithi026.stderr: stderr: 10+0 records in 2024-03-20T08:52:19.841 INFO:teuthology.orchestra.run.smithi026.stderr:10+0 records out 2024-03-20T08:52:19.841 INFO:teuthology.orchestra.run.smithi026.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00852293 s, 1.2 GB/s 2024-03-20T08:52:19.841 INFO:teuthology.orchestra.run.smithi026.stderr:--> Zapping successful for: 2024-03-20T08:52:20.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:19 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-20T08:52:20.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:19 smithi026 ceph-mon[68309]: osdmap e20: 3 total, 3 up, 3 in 2024-03-20T08:52:20.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:19 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:20.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:19 smithi026 ceph-mon[68309]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-20T08:52:20.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:19 smithi026 ceph-mon[68309]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-20T08:52:20.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:19 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:52:20.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:19 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:52:20.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:19 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:52:20.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:19 smithi026 ceph-mon[68309]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-20T08:52:20.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:19 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:52:20.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:19 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:52:20.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:19 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:52:20.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:19 smithi026 ceph-mon[68309]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-20T08:52:20.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:19 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:20.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:19 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:52:20.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:19 smithi026 ceph-mon[68309]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-20T08:52:20.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:19 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:52:20.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:19 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:52:20.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:19 smithi026 ceph-mon[68309]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-20T08:52:20.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:19 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:20.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:19 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-20T08:52:20.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:19 smithi115 ceph-mon[72476]: osdmap e20: 3 total, 3 up, 3 in 2024-03-20T08:52:20.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:19 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:20.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:19 smithi115 ceph-mon[72476]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-20T08:52:20.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:19 smithi115 ceph-mon[72476]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-20T08:52:20.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:19 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:52:20.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:19 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:52:20.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:19 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:52:20.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:19 smithi115 ceph-mon[72476]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-20T08:52:20.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:19 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:52:20.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:19 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:52:20.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:19 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:52:20.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:19 smithi115 ceph-mon[72476]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-20T08:52:20.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:19 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:20.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:19 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:52:20.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:19 smithi115 ceph-mon[72476]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-20T08:52:20.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:19 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:52:20.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:19 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:52:20.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:19 smithi115 ceph-mon[72476]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-20T08:52:20.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:19 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:20.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:19 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-20T08:52:20.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:19 smithi122 ceph-mon[73495]: osdmap e20: 3 total, 3 up, 3 in 2024-03-20T08:52:20.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:19 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:20.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:19 smithi122 ceph-mon[73495]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-20T08:52:20.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:19 smithi122 ceph-mon[73495]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-20T08:52:20.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:19 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:52:20.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:19 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:52:20.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:19 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:52:20.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:19 smithi122 ceph-mon[73495]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-20T08:52:20.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:19 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:52:20.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:19 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:52:20.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:19 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:52:20.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:19 smithi122 ceph-mon[73495]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-20T08:52:20.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:19 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:20.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:19 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:52:20.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:19 smithi122 ceph-mon[73495]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-20T08:52:20.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:19 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:52:20.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:19 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:52:20.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:19 smithi122 ceph-mon[73495]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-20T08:52:20.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:19 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:21.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:20 smithi026 ceph-mon[68309]: pgmap v92: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:21.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:20 smithi026 ceph-mon[68309]: osdmap e21: 3 total, 3 up, 3 in 2024-03-20T08:52:21.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:20 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:21.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:20 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:21.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:20 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:52:21.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:20 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:52:21.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:20 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:52:21.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:20 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:21.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:20 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:52:21.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:20 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:52:21.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:20 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:21.253 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi026:vg_nvme/lv_1 2024-03-20T08:52:21.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:20 smithi115 ceph-mon[72476]: pgmap v92: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:21.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:20 smithi115 ceph-mon[72476]: osdmap e21: 3 total, 3 up, 3 in 2024-03-20T08:52:21.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:20 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:21.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:20 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:21.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:20 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:52:21.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:20 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:52:21.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:20 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:52:21.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:20 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:21.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:20 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:52:21.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:20 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:52:21.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:20 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:21.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:20 smithi122 ceph-mon[73495]: pgmap v92: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:21.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:20 smithi122 ceph-mon[73495]: osdmap e21: 3 total, 3 up, 3 in 2024-03-20T08:52:21.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:20 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:21.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:20 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:21.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:20 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:52:21.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:20 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:52:21.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:20 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:52:21.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:20 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:21.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:20 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:52:21.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:20 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:52:21.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:20 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:21.534 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:52:22.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:21 smithi026 ceph-mon[68309]: Detected new or changed devices on smithi026 2024-03-20T08:52:22.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:21 smithi026 ceph-mon[68309]: Adjusting osd_memory_target on smithi026 to 5708M 2024-03-20T08:52:22.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:21 smithi026 ceph-mon[68309]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-20T08:52:22.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:21 smithi115 ceph-mon[72476]: Detected new or changed devices on smithi026 2024-03-20T08:52:22.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:21 smithi115 ceph-mon[72476]: Adjusting osd_memory_target on smithi026 to 5708M 2024-03-20T08:52:22.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:21 smithi115 ceph-mon[72476]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-20T08:52:22.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:21 smithi122 ceph-mon[73495]: Detected new or changed devices on smithi026 2024-03-20T08:52:22.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:21 smithi122 ceph-mon[73495]: Adjusting osd_memory_target on smithi026 to 5708M 2024-03-20T08:52:22.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:21 smithi122 ceph-mon[73495]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-20T08:52:23.179 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:22 smithi026 ceph-mon[68309]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:23.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:22 smithi115 ceph-mon[72476]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:23.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:22 smithi122 ceph-mon[73495]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:24.258 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:23 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:52:24.259 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:23 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:52:24.259 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:23 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:52:24.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:23 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:52:24.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:23 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:52:24.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:23 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:52:24.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:23 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:52:24.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:23 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:52:24.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:23 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:52:25.363 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:24 smithi026 ceph-mon[68309]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:25.364 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:24 smithi026 ceph-mon[68309]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:52:25.364 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:24 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:52:25.364 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:24 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:52:25.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:24 smithi115 ceph-mon[72476]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:25.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:24 smithi115 ceph-mon[72476]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:52:25.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:24 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:52:25.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:24 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:52:25.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:24 smithi122 ceph-mon[73495]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:25.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:24 smithi122 ceph-mon[73495]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:52:25.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:24 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:52:25.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:24 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:52:26.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:26 smithi115 ceph-mon[72476]: osdmap e22: 3 total, 3 up, 3 in 2024-03-20T08:52:26.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:26 smithi122 ceph-mon[73495]: osdmap e22: 3 total, 3 up, 3 in 2024-03-20T08:52:26.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:26 smithi026 ceph-mon[68309]: osdmap e22: 3 total, 3 up, 3 in 2024-03-20T08:52:27.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:27 smithi115 ceph-mon[72476]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:27.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:27 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2535927266' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1"}]: dispatch 2024-03-20T08:52:27.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:27 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2535927266' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1"}]': finished 2024-03-20T08:52:27.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:27 smithi115 ceph-mon[72476]: osdmap e23: 4 total, 3 up, 4 in 2024-03-20T08:52:27.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:27 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:52:27.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:27 smithi122 ceph-mon[73495]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:27.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:27 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2535927266' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1"}]: dispatch 2024-03-20T08:52:27.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:27 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2535927266' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1"}]': finished 2024-03-20T08:52:27.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:27 smithi122 ceph-mon[73495]: osdmap e23: 4 total, 3 up, 4 in 2024-03-20T08:52:27.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:27 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:52:27.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:27 smithi026 ceph-mon[68309]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:27.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:27 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2535927266' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1"}]: dispatch 2024-03-20T08:52:27.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:27 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2535927266' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1"}]': finished 2024-03-20T08:52:27.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:27 smithi026 ceph-mon[68309]: osdmap e23: 4 total, 3 up, 4 in 2024-03-20T08:52:27.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:27 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:52:28.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:28 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/225853042' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:52:28.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:28 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/225853042' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:52:28.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:28 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/225853042' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:52:29.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:29 smithi115 ceph-mon[72476]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:29.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:29 smithi122 ceph-mon[73495]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:29.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:29 smithi026 ceph-mon[68309]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:31.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:31 smithi026 ceph-mon[68309]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:31.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:31 smithi115 ceph-mon[72476]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:31.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:31 smithi122 ceph-mon[73495]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:33.396 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:33 smithi026 ceph-mon[68309]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:33.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:33 smithi115 ceph-mon[72476]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:33.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:33 smithi122 ceph-mon[73495]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:34.382 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:34 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-20T08:52:34.382 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:34 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:52:34.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:34 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-20T08:52:34.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:34 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:52:34.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:34 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-20T08:52:34.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:34 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:52:35.384 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:35 smithi026 ceph-mon[68309]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:35.384 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:35 smithi026 ceph-mon[68309]: Deploying daemon osd.3 on smithi026 2024-03-20T08:52:35.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:35 smithi115 ceph-mon[72476]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:35.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:35 smithi115 ceph-mon[72476]: Deploying daemon osd.3 on smithi026 2024-03-20T08:52:35.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:35 smithi122 ceph-mon[73495]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:35.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:35 smithi122 ceph-mon[73495]: Deploying daemon osd.3 on smithi026 2024-03-20T08:52:37.225 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:37 smithi026 ceph-mon[68309]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:37.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:37 smithi115 ceph-mon[72476]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:37.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:37 smithi122 ceph-mon[73495]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:39.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:39 smithi026 ceph-mon[68309]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:39.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:39 smithi115 ceph-mon[72476]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:39.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:39 smithi122 ceph-mon[73495]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:41.104 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:40 smithi026 ceph-mon[68309]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:41.104 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:40 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:41.104 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:40 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:41.104 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:40 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:52:41.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:40 smithi115 ceph-mon[72476]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:41.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:40 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:41.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:40 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:41.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:40 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:52:41.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:40 smithi122 ceph-mon[73495]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:41.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:40 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:41.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:40 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:41.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:40 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:52:42.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:42 smithi026 ceph-mon[68309]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:42.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:42 smithi026 ceph-mon[68309]: from='osd.3 [v2:172.21.15.26:6826/4110225509,v1:172.21.15.26:6827/4110225509]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-20T08:52:42.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:42 smithi026 ceph-mon[68309]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-20T08:52:43.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:42 smithi115 ceph-mon[72476]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:43.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:42 smithi115 ceph-mon[72476]: from='osd.3 [v2:172.21.15.26:6826/4110225509,v1:172.21.15.26:6827/4110225509]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-20T08:52:43.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:42 smithi115 ceph-mon[72476]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-20T08:52:43.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:42 smithi122 ceph-mon[73495]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:43.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:42 smithi122 ceph-mon[73495]: from='osd.3 [v2:172.21.15.26:6826/4110225509,v1:172.21.15.26:6827/4110225509]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-20T08:52:43.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:42 smithi122 ceph-mon[73495]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-20T08:52:44.016 INFO:teuthology.orchestra.run.smithi026.stdout:Created osd(s) 3 on host 'smithi026' 2024-03-20T08:52:44.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:43 smithi115 ceph-mon[72476]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-20T08:52:44.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:43 smithi115 ceph-mon[72476]: osdmap e24: 4 total, 3 up, 4 in 2024-03-20T08:52:44.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:43 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:52:44.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:43 smithi115 ceph-mon[72476]: from='osd.3 [v2:172.21.15.26:6826/4110225509,v1:172.21.15.26:6827/4110225509]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-20T08:52:44.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:43 smithi115 ceph-mon[72476]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-20T08:52:44.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:43 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:44.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:43 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:44.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:43 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:52:44.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:43 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:52:44.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:43 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:44.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:43 smithi122 ceph-mon[73495]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-20T08:52:44.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:43 smithi122 ceph-mon[73495]: osdmap e24: 4 total, 3 up, 4 in 2024-03-20T08:52:44.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:43 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:52:44.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:43 smithi122 ceph-mon[73495]: from='osd.3 [v2:172.21.15.26:6826/4110225509,v1:172.21.15.26:6827/4110225509]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-20T08:52:44.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:43 smithi122 ceph-mon[73495]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-20T08:52:44.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:43 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:44.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:43 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:44.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:43 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:52:44.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:43 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:52:44.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:43 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:44.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:43 smithi026 ceph-mon[68309]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-20T08:52:44.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:43 smithi026 ceph-mon[68309]: osdmap e24: 4 total, 3 up, 4 in 2024-03-20T08:52:44.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:43 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:52:44.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:43 smithi026 ceph-mon[68309]: from='osd.3 [v2:172.21.15.26:6826/4110225509,v1:172.21.15.26:6827/4110225509]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-20T08:52:44.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:43 smithi026 ceph-mon[68309]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-20T08:52:44.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:43 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:44.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:43 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:44.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:43 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:52:44.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:43 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:52:44.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:43 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:44.883 DEBUG:teuthology.orchestra.run.smithi026:osd.3> sudo journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.3.service 2024-03-20T08:52:44.886 INFO:tasks.cephadm:Deploying osd.4 on smithi115 with /dev/vg_nvme/lv_4... 2024-03-20T08:52:44.886 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-20T08:52:44.917 INFO:journalctl@ceph.osd.3.smithi026.stdout:-- Logs begin at Wed 2024-03-20 08:37:24 UTC. -- 2024-03-20T08:52:45.106 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:44 smithi026 ceph-mon[68309]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:45.106 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:44 smithi026 ceph-mon[68309]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-03-20T08:52:45.106 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:44 smithi026 ceph-mon[68309]: osdmap e25: 4 total, 3 up, 4 in 2024-03-20T08:52:45.106 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:44 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:52:45.106 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:44 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:52:45.106 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:44 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:45.107 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:44 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:45.107 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:44 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:52:45.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:44 smithi115 ceph-mon[72476]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:45.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:44 smithi115 ceph-mon[72476]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-03-20T08:52:45.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:44 smithi115 ceph-mon[72476]: osdmap e25: 4 total, 3 up, 4 in 2024-03-20T08:52:45.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:44 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:52:45.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:44 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:52:45.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:44 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:45.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:44 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:45.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:44 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:52:45.156 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:52:45.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:44 smithi122 ceph-mon[73495]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-20T08:52:45.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:44 smithi122 ceph-mon[73495]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-03-20T08:52:45.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:44 smithi122 ceph-mon[73495]: osdmap e25: 4 total, 3 up, 4 in 2024-03-20T08:52:45.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:44 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:52:45.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:44 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:52:45.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:44 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:45.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:44 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:45.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:44 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:52:46.124 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:45 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:52:46.124 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:45 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:52:46.124 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:45 smithi115 ceph-mon[72476]: osd.3 [v2:172.21.15.26:6826/4110225509,v1:172.21.15.26:6827/4110225509] boot 2024-03-20T08:52:46.124 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:45 smithi115 ceph-mon[72476]: osdmap e26: 4 total, 4 up, 4 in 2024-03-20T08:52:46.124 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:45 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:52:46.124 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:45 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:52:46.124 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:45 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:52:46.124 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:45 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:52:46.124 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:45 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:52:46.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:45 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:52:46.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:45 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:52:46.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:45 smithi122 ceph-mon[73495]: osd.3 [v2:172.21.15.26:6826/4110225509,v1:172.21.15.26:6827/4110225509] boot 2024-03-20T08:52:46.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:45 smithi122 ceph-mon[73495]: osdmap e26: 4 total, 4 up, 4 in 2024-03-20T08:52:46.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:45 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:52:46.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:45 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:52:46.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:45 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:52:46.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:45 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:52:46.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:45 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:52:46.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:45 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:52:46.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:45 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:52:46.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:45 smithi026 ceph-mon[68309]: osd.3 [v2:172.21.15.26:6826/4110225509,v1:172.21.15.26:6827/4110225509] boot 2024-03-20T08:52:46.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:45 smithi026 ceph-mon[68309]: osdmap e26: 4 total, 4 up, 4 in 2024-03-20T08:52:46.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:45 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:52:46.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:45 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-20T08:52:46.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:45 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:52:46.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:45 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:52:46.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:45 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:52:47.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:46 smithi115 ceph-mon[72476]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:52:47.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:46 smithi115 ceph-mon[72476]: osdmap e27: 4 total, 4 up, 4 in 2024-03-20T08:52:47.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:46 smithi122 ceph-mon[73495]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:52:47.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:46 smithi122 ceph-mon[73495]: osdmap e27: 4 total, 4 up, 4 in 2024-03-20T08:52:47.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:46 smithi026 ceph-mon[68309]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:52:47.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:46 smithi026 ceph-mon[68309]: osdmap e27: 4 total, 4 up, 4 in 2024-03-20T08:52:47.292 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-20T08:52:47.292 INFO:teuthology.orchestra.run.smithi115.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-20T08:52:47.292 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10+0 records in 2024-03-20T08:52:47.292 INFO:teuthology.orchestra.run.smithi115.stderr:10+0 records out 2024-03-20T08:52:47.292 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111389 s, 941 MB/s 2024-03-20T08:52:47.292 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping successful for: 2024-03-20T08:52:47.930 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi115:vg_nvme/lv_4 2024-03-20T08:52:48.190 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:52:49.359 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:49 smithi115 ceph-mon[72476]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:52:49.359 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:49 smithi115 ceph-mon[72476]: Detected new or changed devices on smithi026 2024-03-20T08:52:49.359 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:49 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:49.359 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:49 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:49.359 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:49 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:52:49.359 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:49 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:52:49.359 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:49 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:52:49.359 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:49 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:52:49.359 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:49 smithi115 ceph-mon[72476]: Adjusting osd_memory_target on smithi026 to 4281M 2024-03-20T08:52:49.359 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:49 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:49.359 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:49 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:52:49.360 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:49 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:52:49.360 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:49 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:49.360 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:49 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:52:49.360 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:49 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:52:49.360 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:49 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:52:49.360 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:49 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:49.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:49 smithi122 ceph-mon[73495]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:52:49.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:49 smithi122 ceph-mon[73495]: Detected new or changed devices on smithi026 2024-03-20T08:52:49.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:49 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:49.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:49 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:49.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:49 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:52:49.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:49 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:52:49.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:49 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:52:49.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:49 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:52:49.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:49 smithi122 ceph-mon[73495]: Adjusting osd_memory_target on smithi026 to 4281M 2024-03-20T08:52:49.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:49 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:49.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:49 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:52:49.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:49 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:52:49.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:49 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:49.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:49 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:52:49.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:49 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:52:49.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:49 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:52:49.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:49 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:49.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:49 smithi026 ceph-mon[68309]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:52:49.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:49 smithi026 ceph-mon[68309]: Detected new or changed devices on smithi026 2024-03-20T08:52:49.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:49 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:49.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:49 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:49.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:49 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:52:49.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:49 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:52:49.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:49 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:52:49.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:49 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:52:49.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:49 smithi026 ceph-mon[68309]: Adjusting osd_memory_target on smithi026 to 4281M 2024-03-20T08:52:49.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:49 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:49.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:49 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:52:49.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:49 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:52:49.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:49 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:49.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:49 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:52:49.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:49 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:52:49.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:49 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:52:49.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:49 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:52:51.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:51 smithi115 ceph-mon[72476]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:52:51.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:51 smithi115 ceph-mon[72476]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:52:51.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:51 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:52:51.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:51 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:52:51.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:51 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:52:51.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:51 smithi122 ceph-mon[73495]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:52:51.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:51 smithi122 ceph-mon[73495]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:52:51.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:51 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:52:51.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:51 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:52:51.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:51 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:52:51.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:51 smithi026 ceph-mon[68309]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:52:51.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:51 smithi026 ceph-mon[68309]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:52:51.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:51 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:52:51.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:51 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:52:51.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:51 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:52:53.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:53 smithi115 ceph-mon[72476]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:52:53.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:53 smithi115 ceph-mon[72476]: from='client.? 172.21.15.115:0/3971139018' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f14e34d5-9829-42a3-9b25-31a0976c5ef7"}]: dispatch 2024-03-20T08:52:53.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:53 smithi115 ceph-mon[72476]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f14e34d5-9829-42a3-9b25-31a0976c5ef7"}]: dispatch 2024-03-20T08:52:53.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:53 smithi115 ceph-mon[72476]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f14e34d5-9829-42a3-9b25-31a0976c5ef7"}]': finished 2024-03-20T08:52:53.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:53 smithi115 ceph-mon[72476]: osdmap e28: 5 total, 4 up, 5 in 2024-03-20T08:52:53.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:53 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-20T08:52:53.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:53 smithi122 ceph-mon[73495]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:52:53.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:53 smithi122 ceph-mon[73495]: from='client.? 172.21.15.115:0/3971139018' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f14e34d5-9829-42a3-9b25-31a0976c5ef7"}]: dispatch 2024-03-20T08:52:53.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:53 smithi122 ceph-mon[73495]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f14e34d5-9829-42a3-9b25-31a0976c5ef7"}]: dispatch 2024-03-20T08:52:53.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:53 smithi122 ceph-mon[73495]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f14e34d5-9829-42a3-9b25-31a0976c5ef7"}]': finished 2024-03-20T08:52:53.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:53 smithi122 ceph-mon[73495]: osdmap e28: 5 total, 4 up, 5 in 2024-03-20T08:52:53.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:53 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-20T08:52:53.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:53 smithi026 ceph-mon[68309]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:52:53.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:53 smithi026 ceph-mon[68309]: from='client.? 172.21.15.115:0/3971139018' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f14e34d5-9829-42a3-9b25-31a0976c5ef7"}]: dispatch 2024-03-20T08:52:53.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:53 smithi026 ceph-mon[68309]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f14e34d5-9829-42a3-9b25-31a0976c5ef7"}]: dispatch 2024-03-20T08:52:53.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:53 smithi026 ceph-mon[68309]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f14e34d5-9829-42a3-9b25-31a0976c5ef7"}]': finished 2024-03-20T08:52:53.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:53 smithi026 ceph-mon[68309]: osdmap e28: 5 total, 4 up, 5 in 2024-03-20T08:52:53.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:53 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-20T08:52:54.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:54 smithi115 ceph-mon[72476]: from='client.? 172.21.15.115:0/1731918177' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:52:54.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:54 smithi122 ceph-mon[73495]: from='client.? 172.21.15.115:0/1731918177' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:52:54.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:54 smithi026 ceph-mon[68309]: from='client.? 172.21.15.115:0/1731918177' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:52:55.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:55 smithi115 ceph-mon[72476]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:52:55.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:55 smithi122 ceph-mon[73495]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:52:55.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:55 smithi026 ceph-mon[68309]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:52:57.385 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:57 smithi115 ceph-mon[72476]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:52:57.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:57 smithi122 ceph-mon[73495]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:52:57.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:57 smithi026 ceph-mon[68309]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:52:59.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:59 smithi115 ceph-mon[72476]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:52:59.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:59 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:52:59.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:59 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:52:59.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:59 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-20T08:52:59.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:52:59 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:52:59.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:59 smithi122 ceph-mon[73495]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:52:59.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:59 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:52:59.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:59 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:52:59.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:59 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-20T08:52:59.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:52:59 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:52:59.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:59 smithi026 ceph-mon[68309]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:52:59.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:59 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:52:59.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:59 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:52:59.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:59 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-20T08:52:59.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:52:59 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:53:00.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:00 smithi115 ceph-mon[72476]: Deploying daemon osd.4 on smithi115 2024-03-20T08:53:00.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:00 smithi115 ceph-mon[72476]: osdmap e29: 5 total, 4 up, 5 in 2024-03-20T08:53:00.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:00 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-20T08:53:00.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:00 smithi122 ceph-mon[73495]: Deploying daemon osd.4 on smithi115 2024-03-20T08:53:00.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:00 smithi122 ceph-mon[73495]: osdmap e29: 5 total, 4 up, 5 in 2024-03-20T08:53:00.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:00 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-20T08:53:00.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:00 smithi026 ceph-mon[68309]: Deploying daemon osd.4 on smithi115 2024-03-20T08:53:00.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:00 smithi026 ceph-mon[68309]: osdmap e29: 5 total, 4 up, 5 in 2024-03-20T08:53:00.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:00 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-20T08:53:01.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:01 smithi115 ceph-mon[72476]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:53:01.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:01 smithi122 ceph-mon[73495]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:53:01.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:01 smithi026 ceph-mon[68309]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:53:03.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:03 smithi115 ceph-mon[72476]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:53:03.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:03 smithi122 ceph-mon[73495]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:53:03.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:03 smithi026 ceph-mon[68309]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:53:05.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:05 smithi115 ceph-mon[72476]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:53:05.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:05 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:05.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:05 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:05.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:05 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:53:05.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:05 smithi122 ceph-mon[73495]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:53:05.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:05 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:05.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:05 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:05.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:05 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:53:05.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:05 smithi026 ceph-mon[68309]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:53:05.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:05 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:05.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:05 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:05.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:05 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:53:07.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:07 smithi115 ceph-mon[72476]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:53:07.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:07 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:07.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:07 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:07.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:07 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:53:07.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:07 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:53:07.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:07 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:07.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:07 smithi122 ceph-mon[73495]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:53:07.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:07 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:07.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:07 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:07.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:07 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:53:07.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:07 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:53:07.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:07 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:07.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:07 smithi026 ceph-mon[68309]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:53:07.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:07 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:07.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:07 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:07.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:07 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:53:07.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:07 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:53:07.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:07 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:08.238 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:08 smithi115 ceph-mon[72476]: from='osd.4 [v2:172.21.15.115:6800/477923058,v1:172.21.15.115:6801/477923058]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-20T08:53:08.238 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:08 smithi115 ceph-mon[72476]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-20T08:53:08.304 INFO:teuthology.orchestra.run.smithi115.stdout:Created osd(s) 4 on host 'smithi115' 2024-03-20T08:53:08.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:08 smithi122 ceph-mon[73495]: from='osd.4 [v2:172.21.15.115:6800/477923058,v1:172.21.15.115:6801/477923058]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-20T08:53:08.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:08 smithi122 ceph-mon[73495]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-20T08:53:08.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:08 smithi026 ceph-mon[68309]: from='osd.4 [v2:172.21.15.115:6800/477923058,v1:172.21.15.115:6801/477923058]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-20T08:53:08.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:08 smithi026 ceph-mon[68309]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-20T08:53:09.002 DEBUG:teuthology.orchestra.run.smithi115:osd.4> sudo journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.4.service 2024-03-20T08:53:09.005 INFO:tasks.cephadm:Deploying osd.5 on smithi115 with /dev/vg_nvme/lv_3... 2024-03-20T08:53:09.005 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-20T08:53:09.036 INFO:journalctl@ceph.osd.4.smithi115.stdout:-- Logs begin at Wed 2024-03-20 08:37:21 UTC. -- 2024-03-20T08:53:09.289 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:09 smithi115 ceph-mon[72476]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:53:09.289 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:09 smithi115 ceph-mon[72476]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-20T08:53:09.289 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:09 smithi115 ceph-mon[72476]: osdmap e30: 5 total, 4 up, 5 in 2024-03-20T08:53:09.289 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:09 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-20T08:53:09.289 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:09 smithi115 ceph-mon[72476]: from='osd.4 [v2:172.21.15.115:6800/477923058,v1:172.21.15.115:6801/477923058]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-03-20T08:53:09.289 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:09 smithi115 ceph-mon[72476]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-03-20T08:53:09.290 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:09 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:09.290 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:09 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:09.290 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:09 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:53:09.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:09 smithi122 ceph-mon[73495]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:53:09.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:09 smithi122 ceph-mon[73495]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-20T08:53:09.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:09 smithi122 ceph-mon[73495]: osdmap e30: 5 total, 4 up, 5 in 2024-03-20T08:53:09.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:09 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-20T08:53:09.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:09 smithi122 ceph-mon[73495]: from='osd.4 [v2:172.21.15.115:6800/477923058,v1:172.21.15.115:6801/477923058]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-03-20T08:53:09.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:09 smithi122 ceph-mon[73495]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-03-20T08:53:09.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:09 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:09.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:09 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:09.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:09 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:53:09.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:09 smithi026 ceph-mon[68309]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:53:09.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:09 smithi026 ceph-mon[68309]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-20T08:53:09.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:09 smithi026 ceph-mon[68309]: osdmap e30: 5 total, 4 up, 5 in 2024-03-20T08:53:09.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:09 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-20T08:53:09.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:09 smithi026 ceph-mon[68309]: from='osd.4 [v2:172.21.15.115:6800/477923058,v1:172.21.15.115:6801/477923058]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-03-20T08:53:09.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:09 smithi026 ceph-mon[68309]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-03-20T08:53:09.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:09 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:09.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:09 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:09.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:09 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:53:09.905 INFO:journalctl@ceph.osd.4.smithi115.stdout:Mar 20 08:53:09 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-4[77898]: 2024-03-20T08:53:09.572+0000 7f4880622700 -1 osd.4 0 waiting for initial osdmap 2024-03-20T08:53:09.906 INFO:journalctl@ceph.osd.4.smithi115.stdout:Mar 20 08:53:09 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-4[77898]: 2024-03-20T08:53:09.576+0000 7f487840d700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-20T08:53:10.280 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:53:10.281 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:10 smithi115 ceph-mon[72476]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-03-20T08:53:10.281 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:10 smithi115 ceph-mon[72476]: osdmap e31: 5 total, 4 up, 5 in 2024-03-20T08:53:10.281 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:10 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-20T08:53:10.281 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:10 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-20T08:53:10.281 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:10 smithi115 ceph-mon[72476]: from='osd.4 ' entity='osd.4' 2024-03-20T08:53:10.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:10 smithi122 ceph-mon[73495]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-03-20T08:53:10.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:10 smithi122 ceph-mon[73495]: osdmap e31: 5 total, 4 up, 5 in 2024-03-20T08:53:10.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:10 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-20T08:53:10.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:10 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-20T08:53:10.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:10 smithi122 ceph-mon[73495]: from='osd.4 ' entity='osd.4' 2024-03-20T08:53:10.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:10 smithi026 ceph-mon[68309]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-03-20T08:53:10.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:10 smithi026 ceph-mon[68309]: osdmap e31: 5 total, 4 up, 5 in 2024-03-20T08:53:10.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:10 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-20T08:53:10.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:10 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-20T08:53:10.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:10 smithi026 ceph-mon[68309]: from='osd.4 ' entity='osd.4' 2024-03-20T08:53:11.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:11 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:53:11.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:11 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:53:11.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:11 smithi115 ceph-mon[72476]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:53:11.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:11 smithi115 ceph-mon[72476]: osd.4 [v2:172.21.15.115:6800/477923058,v1:172.21.15.115:6801/477923058] boot 2024-03-20T08:53:11.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:11 smithi115 ceph-mon[72476]: osdmap e32: 5 total, 5 up, 5 in 2024-03-20T08:53:11.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:11 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-20T08:53:11.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:11 smithi115 ceph-mon[72476]: osdmap e33: 5 total, 5 up, 5 in 2024-03-20T08:53:11.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:11 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:53:11.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:11 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:53:11.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:11 smithi122 ceph-mon[73495]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:53:11.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:11 smithi122 ceph-mon[73495]: osd.4 [v2:172.21.15.115:6800/477923058,v1:172.21.15.115:6801/477923058] boot 2024-03-20T08:53:11.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:11 smithi122 ceph-mon[73495]: osdmap e32: 5 total, 5 up, 5 in 2024-03-20T08:53:11.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:11 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-20T08:53:11.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:11 smithi122 ceph-mon[73495]: osdmap e33: 5 total, 5 up, 5 in 2024-03-20T08:53:11.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:11 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:53:11.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:11 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:53:11.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:11 smithi026 ceph-mon[68309]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-20T08:53:11.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:11 smithi026 ceph-mon[68309]: osd.4 [v2:172.21.15.115:6800/477923058,v1:172.21.15.115:6801/477923058] boot 2024-03-20T08:53:11.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:11 smithi026 ceph-mon[68309]: osdmap e32: 5 total, 5 up, 5 in 2024-03-20T08:53:11.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:11 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-20T08:53:11.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:11 smithi026 ceph-mon[68309]: osdmap e33: 5 total, 5 up, 5 in 2024-03-20T08:53:13.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:13 smithi115 ceph-mon[72476]: pgmap v131: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:53:13.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:13 smithi115 ceph-mon[72476]: osdmap e34: 5 total, 5 up, 5 in 2024-03-20T08:53:13.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:13 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:13.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:13 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:13.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:13 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:53:13.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:13 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:13.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:13 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:53:13.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:13 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:53:13.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:13 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:13.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:13 smithi122 ceph-mon[73495]: pgmap v131: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:53:13.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:13 smithi122 ceph-mon[73495]: osdmap e34: 5 total, 5 up, 5 in 2024-03-20T08:53:13.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:13 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:13.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:13 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:13.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:13 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:53:13.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:13 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:13.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:13 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:53:13.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:13 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:53:13.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:13 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:13.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:13 smithi026 ceph-mon[68309]: pgmap v131: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:53:13.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:13 smithi026 ceph-mon[68309]: osdmap e34: 5 total, 5 up, 5 in 2024-03-20T08:53:13.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:13 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:13.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:13 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:13.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:13 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:53:13.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:13 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:13.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:13 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:53:13.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:13 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:53:13.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:13 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:13.595 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-20T08:53:13.595 INFO:teuthology.orchestra.run.smithi115.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-20T08:53:13.596 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10+0 records in 2024-03-20T08:53:13.596 INFO:teuthology.orchestra.run.smithi115.stderr:10+0 records out 2024-03-20T08:53:13.596 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110351 s, 950 MB/s 2024-03-20T08:53:13.596 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping successful for: 2024-03-20T08:53:14.251 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:14 smithi115 ceph-mon[72476]: Detected new or changed devices on smithi115 2024-03-20T08:53:14.252 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:14 smithi115 ceph-mon[72476]: Adjusting osd_memory_target on smithi115 to 17122M 2024-03-20T08:53:14.252 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:14 smithi115 ceph-mon[72476]: osdmap e35: 5 total, 5 up, 5 in 2024-03-20T08:53:14.278 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi115:vg_nvme/lv_3 2024-03-20T08:53:14.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:14 smithi122 ceph-mon[73495]: Detected new or changed devices on smithi115 2024-03-20T08:53:14.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:14 smithi122 ceph-mon[73495]: Adjusting osd_memory_target on smithi115 to 17122M 2024-03-20T08:53:14.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:14 smithi122 ceph-mon[73495]: osdmap e35: 5 total, 5 up, 5 in 2024-03-20T08:53:14.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:14 smithi026 ceph-mon[68309]: Detected new or changed devices on smithi115 2024-03-20T08:53:14.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:14 smithi026 ceph-mon[68309]: Adjusting osd_memory_target on smithi115 to 17122M 2024-03-20T08:53:14.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:14 smithi026 ceph-mon[68309]: osdmap e35: 5 total, 5 up, 5 in 2024-03-20T08:53:14.537 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:53:15.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:15 smithi115 ceph-mon[72476]: pgmap v134: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:53:15.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:15 smithi115 ceph-mon[72476]: osdmap e36: 5 total, 5 up, 5 in 2024-03-20T08:53:15.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:15 smithi122 ceph-mon[73495]: pgmap v134: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:53:15.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:15 smithi122 ceph-mon[73495]: osdmap e36: 5 total, 5 up, 5 in 2024-03-20T08:53:15.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:15 smithi026 ceph-mon[68309]: pgmap v134: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:53:15.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:15 smithi026 ceph-mon[68309]: osdmap e36: 5 total, 5 up, 5 in 2024-03-20T08:53:17.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:17 smithi115 ceph-mon[72476]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-03-20T08:53:17.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:17 smithi115 ceph-mon[72476]: from='client.24245 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:53:17.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:17 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:53:17.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:17 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:53:17.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:17 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:53:17.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:17 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:53:17.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:17 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:53:17.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:17 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:53:17.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:17 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:53:17.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:17 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:53:17.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:17 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:53:17.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:17 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:17.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:17 smithi122 ceph-mon[73495]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-03-20T08:53:17.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:17 smithi122 ceph-mon[73495]: from='client.24245 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:53:17.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:17 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:53:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:17 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:53:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:17 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:53:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:17 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:53:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:17 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:53:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:17 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:53:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:17 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:53:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:17 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:53:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:17 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:53:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:17 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:17.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:17 smithi026 ceph-mon[68309]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-03-20T08:53:17.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:17 smithi026 ceph-mon[68309]: from='client.24245 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:53:17.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:17 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:53:17.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:17 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:53:17.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:17 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:53:17.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:17 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:53:17.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:17 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:53:17.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:17 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:53:17.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:17 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:53:17.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:17 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:53:17.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:17 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:53:17.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:17 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:19.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:19 smithi115 ceph-mon[72476]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-20T08:53:19.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:19 smithi122 ceph-mon[73495]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-20T08:53:19.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:19 smithi026 ceph-mon[68309]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-20T08:53:20.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:20 smithi115 ceph-mon[72476]: from='client.? 172.21.15.115:0/2559017984' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cb431b6a-540d-4318-ae35-d379e1df0185"}]: dispatch 2024-03-20T08:53:20.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:20 smithi115 ceph-mon[72476]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cb431b6a-540d-4318-ae35-d379e1df0185"}]: dispatch 2024-03-20T08:53:20.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:20 smithi115 ceph-mon[72476]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cb431b6a-540d-4318-ae35-d379e1df0185"}]': finished 2024-03-20T08:53:20.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:20 smithi115 ceph-mon[72476]: osdmap e37: 6 total, 5 up, 6 in 2024-03-20T08:53:20.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:20 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T08:53:20.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:20 smithi122 ceph-mon[73495]: from='client.? 172.21.15.115:0/2559017984' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cb431b6a-540d-4318-ae35-d379e1df0185"}]: dispatch 2024-03-20T08:53:20.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:20 smithi122 ceph-mon[73495]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cb431b6a-540d-4318-ae35-d379e1df0185"}]: dispatch 2024-03-20T08:53:20.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:20 smithi122 ceph-mon[73495]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cb431b6a-540d-4318-ae35-d379e1df0185"}]': finished 2024-03-20T08:53:20.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:20 smithi122 ceph-mon[73495]: osdmap e37: 6 total, 5 up, 6 in 2024-03-20T08:53:20.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:20 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T08:53:20.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:20 smithi026 ceph-mon[68309]: from='client.? 172.21.15.115:0/2559017984' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cb431b6a-540d-4318-ae35-d379e1df0185"}]: dispatch 2024-03-20T08:53:20.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:20 smithi026 ceph-mon[68309]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cb431b6a-540d-4318-ae35-d379e1df0185"}]: dispatch 2024-03-20T08:53:20.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:20 smithi026 ceph-mon[68309]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cb431b6a-540d-4318-ae35-d379e1df0185"}]': finished 2024-03-20T08:53:20.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:20 smithi026 ceph-mon[68309]: osdmap e37: 6 total, 5 up, 6 in 2024-03-20T08:53:20.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:20 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T08:53:21.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:21 smithi115 ceph-mon[72476]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-20T08:53:21.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:21 smithi115 ceph-mon[72476]: from='client.? 172.21.15.115:0/4060238526' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:53:21.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:21 smithi115 ceph-mon[72476]: 1.0 scrub starts 2024-03-20T08:53:21.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:21 smithi115 ceph-mon[72476]: 1.0 scrub ok 2024-03-20T08:53:21.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:21 smithi122 ceph-mon[73495]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-20T08:53:21.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:21 smithi122 ceph-mon[73495]: from='client.? 172.21.15.115:0/4060238526' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:53:21.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:21 smithi122 ceph-mon[73495]: 1.0 scrub starts 2024-03-20T08:53:21.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:21 smithi122 ceph-mon[73495]: 1.0 scrub ok 2024-03-20T08:53:21.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:21 smithi026 ceph-mon[68309]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-20T08:53:21.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:21 smithi026 ceph-mon[68309]: from='client.? 172.21.15.115:0/4060238526' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:53:21.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:21 smithi026 ceph-mon[68309]: 1.0 scrub starts 2024-03-20T08:53:21.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:21 smithi026 ceph-mon[68309]: 1.0 scrub ok 2024-03-20T08:53:23.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:23 smithi115 ceph-mon[72476]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-20T08:53:23.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:23 smithi122 ceph-mon[73495]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-20T08:53:23.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:23 smithi026 ceph-mon[68309]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-20T08:53:25.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:25 smithi122 ceph-mon[73495]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-03-20T08:53:25.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:25 smithi026 ceph-mon[68309]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-03-20T08:53:25.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:25 smithi115 ceph-mon[72476]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-03-20T08:53:26.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:26 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-20T08:53:26.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:26 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:53:26.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-20T08:53:26.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:53:26.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:26 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-20T08:53:26.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:26 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:53:27.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:27 smithi122 ceph-mon[73495]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:53:27.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:27 smithi122 ceph-mon[73495]: Deploying daemon osd.5 on smithi115 2024-03-20T08:53:27.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:27 smithi026 ceph-mon[68309]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:53:27.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:27 smithi026 ceph-mon[68309]: Deploying daemon osd.5 on smithi115 2024-03-20T08:53:27.534 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:27 smithi115 ceph-mon[72476]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:53:27.534 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:27 smithi115 ceph-mon[72476]: Deploying daemon osd.5 on smithi115 2024-03-20T08:53:29.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:29 smithi115 ceph-mon[72476]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:53:29.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:29 smithi122 ceph-mon[73495]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:53:29.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:29 smithi026 ceph-mon[68309]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:53:30.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:30 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:53:30.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:30 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:53:30.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:30 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:53:30.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:30 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:53:30.583 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:30 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:53:30.583 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:30 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:53:31.440 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:31 smithi115 ceph-mon[72476]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:53:31.440 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:31 smithi115 ceph-mon[72476]: osdmap e38: 6 total, 5 up, 6 in 2024-03-20T08:53:31.440 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:31 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T08:53:31.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:31 smithi122 ceph-mon[73495]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:53:31.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:31 smithi122 ceph-mon[73495]: osdmap e38: 6 total, 5 up, 6 in 2024-03-20T08:53:31.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:31 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T08:53:31.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:31 smithi026 ceph-mon[68309]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:53:31.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:31 smithi026 ceph-mon[68309]: osdmap e38: 6 total, 5 up, 6 in 2024-03-20T08:53:31.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:31 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T08:53:32.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:32 smithi115 ceph-mon[72476]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:53:32.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:32 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:32.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:32 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:32.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:32 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:53:32.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:32 smithi122 ceph-mon[73495]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:53:32.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:32 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:32.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:32 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:32.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:32 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:53:32.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:32 smithi026 ceph-mon[68309]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:53:32.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:32 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:32.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:32 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:32.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:32 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:53:35.141 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:35 smithi115 ceph-mon[72476]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:53:35.141 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:35.141 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:35.141 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:53:35.141 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:53:35.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:35.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:35 smithi115 ceph-mon[72476]: from='osd.5 [v2:172.21.15.115:6808/1339268003,v1:172.21.15.115:6809/1339268003]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-20T08:53:35.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:35 smithi115 ceph-mon[72476]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-20T08:53:35.224 INFO:teuthology.orchestra.run.smithi115.stdout:Created osd(s) 5 on host 'smithi115' 2024-03-20T08:53:35.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:35 smithi122 ceph-mon[73495]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:53:35.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:35 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:35.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:35 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:35.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:35 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:53:35.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:35 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:53:35.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:35 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:35.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:35 smithi122 ceph-mon[73495]: from='osd.5 [v2:172.21.15.115:6808/1339268003,v1:172.21.15.115:6809/1339268003]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-20T08:53:35.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:35 smithi122 ceph-mon[73495]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-20T08:53:35.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:35 smithi026 ceph-mon[68309]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:53:35.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:35 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:35.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:35 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:35.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:35 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:53:35.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:35 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:53:35.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:35 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:35.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:35 smithi026 ceph-mon[68309]: from='osd.5 [v2:172.21.15.115:6808/1339268003,v1:172.21.15.115:6809/1339268003]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-20T08:53:35.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:35 smithi026 ceph-mon[68309]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-20T08:53:35.913 DEBUG:teuthology.orchestra.run.smithi115:osd.5> sudo journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.5.service 2024-03-20T08:53:35.915 INFO:tasks.cephadm:Deploying osd.6 on smithi115 with /dev/vg_nvme/lv_2... 2024-03-20T08:53:35.915 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-20T08:53:35.947 INFO:journalctl@ceph.osd.5.smithi115.stdout:-- Logs begin at Wed 2024-03-20 08:37:21 UTC. -- 2024-03-20T08:53:36.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:36 smithi115 ceph-mon[72476]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-20T08:53:36.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:36 smithi115 ceph-mon[72476]: from='osd.5 [v2:172.21.15.115:6808/1339268003,v1:172.21.15.115:6809/1339268003]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-03-20T08:53:36.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:36 smithi115 ceph-mon[72476]: osdmap e39: 6 total, 5 up, 6 in 2024-03-20T08:53:36.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:36 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T08:53:36.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:36 smithi115 ceph-mon[72476]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-03-20T08:53:36.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:36 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:36.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:36 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:36.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:36 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:53:36.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:36 smithi122 ceph-mon[73495]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-20T08:53:36.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:36 smithi122 ceph-mon[73495]: from='osd.5 [v2:172.21.15.115:6808/1339268003,v1:172.21.15.115:6809/1339268003]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-03-20T08:53:36.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:36 smithi122 ceph-mon[73495]: osdmap e39: 6 total, 5 up, 6 in 2024-03-20T08:53:36.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:36 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T08:53:36.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:36 smithi122 ceph-mon[73495]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-03-20T08:53:36.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:36 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:36.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:36 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:36.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:36 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:53:36.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:36 smithi026 ceph-mon[68309]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-20T08:53:36.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:36 smithi026 ceph-mon[68309]: from='osd.5 [v2:172.21.15.115:6808/1339268003,v1:172.21.15.115:6809/1339268003]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-03-20T08:53:36.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:36 smithi026 ceph-mon[68309]: osdmap e39: 6 total, 5 up, 6 in 2024-03-20T08:53:36.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:36 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T08:53:36.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:36 smithi026 ceph-mon[68309]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-03-20T08:53:36.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:36 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:36.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:36 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:36.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:36 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:53:36.819 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:53:36.905 INFO:journalctl@ceph.osd.5.smithi115.stdout:Mar 20 08:53:36 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-5[85004]: 2024-03-20T08:53:36.458+0000 7fa7536eb700 -1 osd.5 0 waiting for initial osdmap 2024-03-20T08:53:36.906 INFO:journalctl@ceph.osd.5.smithi115.stdout:Mar 20 08:53:36 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-5[85004]: 2024-03-20T08:53:36.463+0000 7fa74b4d6700 -1 osd.5 40 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-20T08:53:37.292 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:37 smithi115 ceph-mon[72476]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:53:37.292 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:37 smithi115 ceph-mon[72476]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-03-20T08:53:37.292 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:37 smithi115 ceph-mon[72476]: osdmap e40: 6 total, 5 up, 6 in 2024-03-20T08:53:37.293 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:37 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T08:53:37.293 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:37 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T08:53:37.293 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:37 smithi115 ceph-mon[72476]: from='osd.5 ' entity='osd.5' 2024-03-20T08:53:37.293 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:37 smithi115 ceph-mon[72476]: osd.5 [v2:172.21.15.115:6808/1339268003,v1:172.21.15.115:6809/1339268003] boot 2024-03-20T08:53:37.293 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:37 smithi115 ceph-mon[72476]: osdmap e41: 6 total, 6 up, 6 in 2024-03-20T08:53:37.293 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:37 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T08:53:37.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:37 smithi122 ceph-mon[73495]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:53:37.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:37 smithi122 ceph-mon[73495]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-03-20T08:53:37.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:37 smithi122 ceph-mon[73495]: osdmap e40: 6 total, 5 up, 6 in 2024-03-20T08:53:37.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:37 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T08:53:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:37 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T08:53:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:37 smithi122 ceph-mon[73495]: from='osd.5 ' entity='osd.5' 2024-03-20T08:53:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:37 smithi122 ceph-mon[73495]: osd.5 [v2:172.21.15.115:6808/1339268003,v1:172.21.15.115:6809/1339268003] boot 2024-03-20T08:53:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:37 smithi122 ceph-mon[73495]: osdmap e41: 6 total, 6 up, 6 in 2024-03-20T08:53:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:37 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T08:53:37.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:37 smithi026 ceph-mon[68309]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-20T08:53:37.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:37 smithi026 ceph-mon[68309]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-03-20T08:53:37.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:37 smithi026 ceph-mon[68309]: osdmap e40: 6 total, 5 up, 6 in 2024-03-20T08:53:37.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:37 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T08:53:37.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:37 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T08:53:37.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:37 smithi026 ceph-mon[68309]: from='osd.5 ' entity='osd.5' 2024-03-20T08:53:37.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:37 smithi026 ceph-mon[68309]: osd.5 [v2:172.21.15.115:6808/1339268003,v1:172.21.15.115:6809/1339268003] boot 2024-03-20T08:53:37.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:37 smithi026 ceph-mon[68309]: osdmap e41: 6 total, 6 up, 6 in 2024-03-20T08:53:37.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:37 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T08:53:38.324 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:38 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:53:38.324 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:38 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:53:38.324 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:38 smithi115 ceph-mon[72476]: osdmap e42: 6 total, 6 up, 6 in 2024-03-20T08:53:38.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:38 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:53:38.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:38 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:53:38.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:38 smithi122 ceph-mon[73495]: osdmap e42: 6 total, 6 up, 6 in 2024-03-20T08:53:38.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:38 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:53:38.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:38 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:53:38.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:38 smithi026 ceph-mon[68309]: osdmap e42: 6 total, 6 up, 6 in 2024-03-20T08:53:39.280 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-20T08:53:39.281 INFO:teuthology.orchestra.run.smithi115.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-20T08:53:39.281 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10+0 records in 2024-03-20T08:53:39.281 INFO:teuthology.orchestra.run.smithi115.stderr:10+0 records out 2024-03-20T08:53:39.281 INFO:teuthology.orchestra.run.smithi115.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113025 s, 928 MB/s 2024-03-20T08:53:39.281 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping successful for: 2024-03-20T08:53:39.316 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:39 smithi115 ceph-mon[72476]: pgmap v152: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:53:39.316 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:39 smithi115 ceph-mon[72476]: osdmap e43: 6 total, 6 up, 6 in 2024-03-20T08:53:39.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:39 smithi122 ceph-mon[73495]: pgmap v152: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:53:39.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:39 smithi122 ceph-mon[73495]: osdmap e43: 6 total, 6 up, 6 in 2024-03-20T08:53:39.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:39 smithi026 ceph-mon[68309]: pgmap v152: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:53:39.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:39 smithi026 ceph-mon[68309]: osdmap e43: 6 total, 6 up, 6 in 2024-03-20T08:53:40.152 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi115:vg_nvme/lv_2 2024-03-20T08:53:40.954 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:53:41.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:41 smithi115 ceph-mon[72476]: pgmap v155: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:53:41.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:41 smithi115 ceph-mon[72476]: osdmap e44: 6 total, 6 up, 6 in 2024-03-20T08:53:41.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:41 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:41.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:41 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:41.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:41 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:53:41.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:41 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:53:41.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:41 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:41.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:41 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:53:41.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:41 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:53:41.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:41 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:41.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:41 smithi122 ceph-mon[73495]: pgmap v155: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:53:41.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:41 smithi122 ceph-mon[73495]: osdmap e44: 6 total, 6 up, 6 in 2024-03-20T08:53:41.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:41 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:41.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:41 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:41.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:41 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:53:41.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:41 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:53:41.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:41 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:41.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:41 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:53:41.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:41 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:53:41.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:41 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:41.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:41 smithi026 ceph-mon[68309]: pgmap v155: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:53:41.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:41 smithi026 ceph-mon[68309]: osdmap e44: 6 total, 6 up, 6 in 2024-03-20T08:53:41.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:41 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:41.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:41 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:41.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:41 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:53:41.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:41 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:53:41.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:41 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:41.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:41 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:53:41.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:41 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:53:41.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:41 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:42.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:42 smithi115 ceph-mon[72476]: Detected new or changed devices on smithi115 2024-03-20T08:53:42.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:42 smithi115 ceph-mon[72476]: Adjusting osd_memory_target on smithi115 to 8561M 2024-03-20T08:53:42.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:42 smithi115 ceph-mon[72476]: osdmap e45: 6 total, 6 up, 6 in 2024-03-20T08:53:42.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:42 smithi122 ceph-mon[73495]: Detected new or changed devices on smithi115 2024-03-20T08:53:42.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:42 smithi122 ceph-mon[73495]: Adjusting osd_memory_target on smithi115 to 8561M 2024-03-20T08:53:42.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:42 smithi122 ceph-mon[73495]: osdmap e45: 6 total, 6 up, 6 in 2024-03-20T08:53:42.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:42 smithi026 ceph-mon[68309]: Detected new or changed devices on smithi115 2024-03-20T08:53:42.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:42 smithi026 ceph-mon[68309]: Adjusting osd_memory_target on smithi115 to 8561M 2024-03-20T08:53:42.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:42 smithi026 ceph-mon[68309]: osdmap e45: 6 total, 6 up, 6 in 2024-03-20T08:53:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:43 smithi115 ceph-mon[72476]: pgmap v158: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:53:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:43 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:53:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:43 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:53:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:43 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:53:43.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:43 smithi122 ceph-mon[73495]: pgmap v158: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:53:43.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:43 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:53:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:43 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:53:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:43 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:53:43.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:43 smithi026 ceph-mon[68309]: pgmap v158: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:53:43.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:43 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:53:43.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:43 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:53:43.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:43 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:53:44.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:44 smithi115 ceph-mon[72476]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:53:44.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:44 smithi122 ceph-mon[73495]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:53:44.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:44 smithi026 ceph-mon[68309]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:53:45.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:45 smithi115 ceph-mon[72476]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-20T08:53:45.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:45 smithi122 ceph-mon[73495]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-20T08:53:45.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:45 smithi026 ceph-mon[68309]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-20T08:53:46.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:46 smithi115 ceph-mon[72476]: from='client.? 172.21.15.115:0/3655982444' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "753ed830-e764-4ff3-9d9b-87e7fa748b81"}]: dispatch 2024-03-20T08:53:46.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:46 smithi115 ceph-mon[72476]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "753ed830-e764-4ff3-9d9b-87e7fa748b81"}]: dispatch 2024-03-20T08:53:46.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:46 smithi115 ceph-mon[72476]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "753ed830-e764-4ff3-9d9b-87e7fa748b81"}]': finished 2024-03-20T08:53:46.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:46 smithi115 ceph-mon[72476]: osdmap e46: 7 total, 6 up, 7 in 2024-03-20T08:53:46.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:53:46.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:46 smithi122 ceph-mon[73495]: from='client.? 172.21.15.115:0/3655982444' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "753ed830-e764-4ff3-9d9b-87e7fa748b81"}]: dispatch 2024-03-20T08:53:46.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:46 smithi122 ceph-mon[73495]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "753ed830-e764-4ff3-9d9b-87e7fa748b81"}]: dispatch 2024-03-20T08:53:46.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:46 smithi122 ceph-mon[73495]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "753ed830-e764-4ff3-9d9b-87e7fa748b81"}]': finished 2024-03-20T08:53:46.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:46 smithi122 ceph-mon[73495]: osdmap e46: 7 total, 6 up, 7 in 2024-03-20T08:53:46.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:53:46.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:46 smithi026 ceph-mon[68309]: from='client.? 172.21.15.115:0/3655982444' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "753ed830-e764-4ff3-9d9b-87e7fa748b81"}]: dispatch 2024-03-20T08:53:46.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:46 smithi026 ceph-mon[68309]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "753ed830-e764-4ff3-9d9b-87e7fa748b81"}]: dispatch 2024-03-20T08:53:46.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:46 smithi026 ceph-mon[68309]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "753ed830-e764-4ff3-9d9b-87e7fa748b81"}]': finished 2024-03-20T08:53:46.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:46 smithi026 ceph-mon[68309]: osdmap e46: 7 total, 6 up, 7 in 2024-03-20T08:53:46.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:53:47.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:47 smithi115 ceph-mon[72476]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-20T08:53:47.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:47 smithi115 ceph-mon[72476]: from='client.? 172.21.15.115:0/1566293862' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:53:47.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:47 smithi122 ceph-mon[73495]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-20T08:53:47.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:47 smithi122 ceph-mon[73495]: from='client.? 172.21.15.115:0/1566293862' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:53:47.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:47 smithi026 ceph-mon[68309]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-20T08:53:47.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:47 smithi026 ceph-mon[68309]: from='client.? 172.21.15.115:0/1566293862' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:53:49.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:49 smithi115 ceph-mon[72476]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-20T08:53:49.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:49 smithi122 ceph-mon[73495]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-20T08:53:49.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:49 smithi026 ceph-mon[68309]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-20T08:53:51.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:51 smithi115 ceph-mon[72476]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-20T08:53:51.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:51 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:53:51.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:51 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:53:51.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:51 smithi122 ceph-mon[73495]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-20T08:53:51.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:51 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:53:51.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:51 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:53:51.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:51 smithi026 ceph-mon[68309]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-20T08:53:51.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:51 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:53:51.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:51 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:53:52.140 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:52 smithi115 ceph-mon[72476]: osdmap e47: 7 total, 6 up, 7 in 2024-03-20T08:53:52.141 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:52 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:53:52.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:52 smithi122 ceph-mon[73495]: osdmap e47: 7 total, 6 up, 7 in 2024-03-20T08:53:52.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:52 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:53:52.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:52 smithi026 ceph-mon[68309]: osdmap e47: 7 total, 6 up, 7 in 2024-03-20T08:53:52.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:52 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:53:53.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:53 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-20T08:53:53.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:53 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:53:53.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:53 smithi115 ceph-mon[72476]: Deploying daemon osd.6 on smithi115 2024-03-20T08:53:53.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:53 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:53:53.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:53 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:53:53.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:53 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:53:53.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:53 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:53:53.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:53 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:53:53.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:53 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:53:53.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:53 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:53.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:53 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-20T08:53:53.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:53 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:53:53.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:53 smithi122 ceph-mon[73495]: Deploying daemon osd.6 on smithi115 2024-03-20T08:53:53.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:53 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:53:53.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:53 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:53:53.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:53 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:53:53.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:53 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:53:53.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:53 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:53:53.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:53 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:53:53.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:53 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:53.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:53 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-20T08:53:53.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:53 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:53:53.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:53 smithi026 ceph-mon[68309]: Deploying daemon osd.6 on smithi115 2024-03-20T08:53:53.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:53 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:53:53.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:53 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:53:53.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:53 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:53:53.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:53 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:53:53.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:53 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:53:53.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:53 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:53:53.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:53 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:55.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:55 smithi115 ceph-mon[72476]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:53:55.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:55 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:53:55.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:55 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:53:55.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:55 smithi115 ceph-mon[72476]: osdmap e48: 7 total, 6 up, 7 in 2024-03-20T08:53:55.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:55 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:53:55.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:55 smithi122 ceph-mon[73495]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:53:55.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:55 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:53:55.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:55 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:53:55.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:55 smithi122 ceph-mon[73495]: osdmap e48: 7 total, 6 up, 7 in 2024-03-20T08:53:55.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:55 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:53:55.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:55 smithi026 ceph-mon[68309]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:53:55.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:55 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:53:55.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:55 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:53:55.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:55 smithi026 ceph-mon[68309]: osdmap e48: 7 total, 6 up, 7 in 2024-03-20T08:53:55.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:55 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:53:57.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:57 smithi115 ceph-mon[72476]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:53:57.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:57 smithi122 ceph-mon[73495]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:53:57.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:57 smithi026 ceph-mon[68309]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:53:59.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:58 smithi115 ceph-mon[72476]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:53:59.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:58 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:59.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:58 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:59.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:53:58 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:53:59.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:58 smithi122 ceph-mon[73495]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:53:59.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:58 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:59.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:58 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:59.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:53:58 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:53:59.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:58 smithi026 ceph-mon[68309]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:53:59.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:58 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:59.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:58 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:53:59.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:53:58 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:54:01.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:00 smithi115 ceph-mon[72476]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:54:01.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:00 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:01.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:00 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:01.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:00 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:54:01.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:00 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:54:01.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:00 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:01.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:00 smithi122 ceph-mon[73495]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:54:01.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:00 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:01.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:00 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:01.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:00 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:54:01.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:00 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:54:01.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:00 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:01.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:00 smithi026 ceph-mon[68309]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:54:01.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:00 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:01.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:00 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:01.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:00 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:54:01.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:00 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:54:01.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:00 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:01.637 INFO:teuthology.orchestra.run.smithi115.stdout:Created osd(s) 6 on host 'smithi115' 2024-03-20T08:54:01.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:01 smithi115 ceph-mon[72476]: from='osd.6 [v2:172.21.15.115:6816/3278301843,v1:172.21.15.115:6817/3278301843]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-20T08:54:01.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:01 smithi115 ceph-mon[72476]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-20T08:54:01.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:01 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:54:01.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:01 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:54:01.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:01 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:01.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:01 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:54:01.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:01 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:02.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:01 smithi122 ceph-mon[73495]: from='osd.6 [v2:172.21.15.115:6816/3278301843,v1:172.21.15.115:6817/3278301843]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-20T08:54:02.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:01 smithi122 ceph-mon[73495]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-20T08:54:02.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:01 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:54:02.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:01 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:54:02.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:01 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:02.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:01 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:54:02.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:01 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:02.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:01 smithi026 ceph-mon[68309]: from='osd.6 [v2:172.21.15.115:6816/3278301843,v1:172.21.15.115:6817/3278301843]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-20T08:54:02.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:01 smithi026 ceph-mon[68309]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-20T08:54:02.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:01 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:54:02.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:01 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:54:02.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:01 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:02.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:01 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:54:02.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:01 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:02.468 DEBUG:teuthology.orchestra.run.smithi115:osd.6> sudo journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.6.service 2024-03-20T08:54:02.470 INFO:tasks.cephadm:Deploying osd.7 on smithi115 with /dev/vg_nvme/lv_1... 2024-03-20T08:54:02.470 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-20T08:54:02.502 INFO:journalctl@ceph.osd.6.smithi115.stdout:-- Logs begin at Wed 2024-03-20 08:37:21 UTC. -- 2024-03-20T08:54:02.965 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:02 smithi115 ceph-mon[72476]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:54:02.966 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:02 smithi115 ceph-mon[72476]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-20T08:54:02.966 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:02 smithi115 ceph-mon[72476]: osdmap e49: 7 total, 6 up, 7 in 2024-03-20T08:54:02.966 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:02 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:54:02.966 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:02 smithi115 ceph-mon[72476]: from='osd.6 [v2:172.21.15.115:6816/3278301843,v1:172.21.15.115:6817/3278301843]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-03-20T08:54:02.966 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:02 smithi115 ceph-mon[72476]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-03-20T08:54:03.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:02 smithi122 ceph-mon[73495]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:54:03.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:02 smithi122 ceph-mon[73495]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-20T08:54:03.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:02 smithi122 ceph-mon[73495]: osdmap e49: 7 total, 6 up, 7 in 2024-03-20T08:54:03.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:02 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:54:03.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:02 smithi122 ceph-mon[73495]: from='osd.6 [v2:172.21.15.115:6816/3278301843,v1:172.21.15.115:6817/3278301843]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-03-20T08:54:03.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:02 smithi122 ceph-mon[73495]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-03-20T08:54:03.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:02 smithi026 ceph-mon[68309]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:54:03.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:02 smithi026 ceph-mon[68309]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-20T08:54:03.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:02 smithi026 ceph-mon[68309]: osdmap e49: 7 total, 6 up, 7 in 2024-03-20T08:54:03.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:02 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:54:03.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:02 smithi026 ceph-mon[68309]: from='osd.6 [v2:172.21.15.115:6816/3278301843,v1:172.21.15.115:6817/3278301843]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-03-20T08:54:03.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:02 smithi026 ceph-mon[68309]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-03-20T08:54:03.640 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:54:03.642 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 08:54:03 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T08:54:03.251+0000 7ff2be551700 -1 osd.6 0 waiting for initial osdmap 2024-03-20T08:54:03.642 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 08:54:03 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T08:54:03.258+0000 7ff2b633c700 -1 osd.6 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-20T08:54:03.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:03 smithi115 ceph-mon[72476]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-03-20T08:54:03.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:03 smithi115 ceph-mon[72476]: osdmap e50: 7 total, 6 up, 7 in 2024-03-20T08:54:03.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:03 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:54:03.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:03 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:54:03.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:03 smithi115 ceph-mon[72476]: from='osd.6 ' entity='osd.6' 2024-03-20T08:54:04.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:03 smithi122 ceph-mon[73495]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-03-20T08:54:04.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:03 smithi122 ceph-mon[73495]: osdmap e50: 7 total, 6 up, 7 in 2024-03-20T08:54:04.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:03 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:54:04.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:03 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:54:04.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:03 smithi122 ceph-mon[73495]: from='osd.6 ' entity='osd.6' 2024-03-20T08:54:04.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:03 smithi026 ceph-mon[68309]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-03-20T08:54:04.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:03 smithi026 ceph-mon[68309]: osdmap e50: 7 total, 6 up, 7 in 2024-03-20T08:54:04.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:03 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:54:04.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:03 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:54:04.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:03 smithi026 ceph-mon[68309]: from='osd.6 ' entity='osd.6' 2024-03-20T08:54:05.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:04 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:54:05.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:04 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:54:05.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:04 smithi115 ceph-mon[72476]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:54:05.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:04 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:54:05.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:04 smithi115 ceph-mon[72476]: osd.6 [v2:172.21.15.115:6816/3278301843,v1:172.21.15.115:6817/3278301843] boot 2024-03-20T08:54:05.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:04 smithi115 ceph-mon[72476]: osdmap e51: 7 total, 7 up, 7 in 2024-03-20T08:54:05.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:04 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:54:05.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:04 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:54:05.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:04 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:54:05.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:04 smithi122 ceph-mon[73495]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:54:05.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:04 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:54:05.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:04 smithi122 ceph-mon[73495]: osd.6 [v2:172.21.15.115:6816/3278301843,v1:172.21.15.115:6817/3278301843] boot 2024-03-20T08:54:05.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:04 smithi122 ceph-mon[73495]: osdmap e51: 7 total, 7 up, 7 in 2024-03-20T08:54:05.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:04 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:54:05.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:04 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:54:05.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:04 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:54:05.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:04 smithi026 ceph-mon[68309]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-20T08:54:05.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:04 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:54:05.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:04 smithi026 ceph-mon[68309]: osd.6 [v2:172.21.15.115:6816/3278301843,v1:172.21.15.115:6817/3278301843] boot 2024-03-20T08:54:05.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:04 smithi026 ceph-mon[68309]: osdmap e51: 7 total, 7 up, 7 in 2024-03-20T08:54:05.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:04 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-20T08:54:06.003 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:05 smithi115 ceph-mon[72476]: osdmap e52: 7 total, 7 up, 7 in 2024-03-20T08:54:06.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:05 smithi122 ceph-mon[73495]: osdmap e52: 7 total, 7 up, 7 in 2024-03-20T08:54:06.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:05 smithi026 ceph-mon[68309]: osdmap e52: 7 total, 7 up, 7 in 2024-03-20T08:54:06.638 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-20T08:54:06.638 INFO:teuthology.orchestra.run.smithi115.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-20T08:54:06.638 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10+0 records in 2024-03-20T08:54:06.638 INFO:teuthology.orchestra.run.smithi115.stderr:10+0 records out 2024-03-20T08:54:06.638 INFO:teuthology.orchestra.run.smithi115.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110292 s, 951 MB/s 2024-03-20T08:54:06.638 INFO:teuthology.orchestra.run.smithi115.stderr:--> Zapping successful for: 2024-03-20T08:54:07.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:06 smithi115 ceph-mon[72476]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:54:07.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:06 smithi115 ceph-mon[72476]: osdmap e53: 7 total, 7 up, 7 in 2024-03-20T08:54:07.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:06 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:07.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:06 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:07.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:06 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:54:07.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:06 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:54:07.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:06 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:54:07.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:06 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:07.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:06 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:54:07.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:06 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:54:07.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:06 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:07.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:06 smithi115 ceph-mon[72476]: osdmap e54: 7 total, 7 up, 7 in 2024-03-20T08:54:07.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:06 smithi122 ceph-mon[73495]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:54:07.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:06 smithi122 ceph-mon[73495]: osdmap e53: 7 total, 7 up, 7 in 2024-03-20T08:54:07.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:06 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:07.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:06 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:07.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:06 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:54:07.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:06 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:54:07.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:06 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:54:07.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:06 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:07.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:06 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:54:07.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:06 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:54:07.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:06 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:07.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:06 smithi122 ceph-mon[73495]: osdmap e54: 7 total, 7 up, 7 in 2024-03-20T08:54:07.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:06 smithi026 ceph-mon[68309]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:54:07.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:06 smithi026 ceph-mon[68309]: osdmap e53: 7 total, 7 up, 7 in 2024-03-20T08:54:07.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:06 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:07.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:06 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:07.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:06 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:54:07.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:06 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:54:07.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:06 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:54:07.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:06 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:07.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:06 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:54:07.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:06 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:54:07.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:06 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:07.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:06 smithi026 ceph-mon[68309]: osdmap e54: 7 total, 7 up, 7 in 2024-03-20T08:54:07.297 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi115:vg_nvme/lv_1 2024-03-20T08:54:07.558 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:54:08.081 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:07 smithi115 ceph-mon[72476]: Detected new or changed devices on smithi115 2024-03-20T08:54:08.081 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:07 smithi115 ceph-mon[72476]: Adjusting osd_memory_target on smithi115 to 5707M 2024-03-20T08:54:08.081 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:07 smithi115 ceph-mon[72476]: osdmap e55: 7 total, 7 up, 7 in 2024-03-20T08:54:08.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:07 smithi122 ceph-mon[73495]: Detected new or changed devices on smithi115 2024-03-20T08:54:08.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:07 smithi122 ceph-mon[73495]: Adjusting osd_memory_target on smithi115 to 5707M 2024-03-20T08:54:08.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:07 smithi122 ceph-mon[73495]: osdmap e55: 7 total, 7 up, 7 in 2024-03-20T08:54:08.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:07 smithi026 ceph-mon[68309]: Detected new or changed devices on smithi115 2024-03-20T08:54:08.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:07 smithi026 ceph-mon[68309]: Adjusting osd_memory_target on smithi115 to 5707M 2024-03-20T08:54:08.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:07 smithi026 ceph-mon[68309]: osdmap e55: 7 total, 7 up, 7 in 2024-03-20T08:54:08.853 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:08 smithi115 ceph-mon[72476]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:54:09.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:08 smithi122 ceph-mon[73495]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:54:09.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:08 smithi026 ceph-mon[68309]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:54:10.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:09 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:54:10.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:09 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:54:10.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:09 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:54:10.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:09 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:54:10.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:09 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:54:10.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:09 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:54:10.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:09 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:54:10.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:09 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:54:10.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:09 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:54:10.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:09 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:54:10.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:09 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:54:10.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:09 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:54:10.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:09 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:54:10.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:09 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:54:10.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:09 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:54:11.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:10 smithi115 ceph-mon[72476]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:54:11.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:10 smithi115 ceph-mon[72476]: from='client.24299 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:54:11.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:10 smithi115 ceph-mon[72476]: osdmap e56: 7 total, 7 up, 7 in 2024-03-20T08:54:11.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:10 smithi122 ceph-mon[73495]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:54:11.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:10 smithi122 ceph-mon[73495]: from='client.24299 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:54:11.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:10 smithi122 ceph-mon[73495]: osdmap e56: 7 total, 7 up, 7 in 2024-03-20T08:54:11.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:10 smithi026 ceph-mon[68309]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:54:11.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:10 smithi026 ceph-mon[68309]: from='client.24299 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi115:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:54:11.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:10 smithi026 ceph-mon[68309]: osdmap e56: 7 total, 7 up, 7 in 2024-03-20T08:54:13.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:12 smithi115 ceph-mon[72476]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-03-20T08:54:13.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:12 smithi115 ceph-mon[72476]: from='client.? 172.21.15.115:0/1659096563' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55bcbd37-7a6f-4ab1-b5a9-91a951444d86"}]: dispatch 2024-03-20T08:54:13.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:12 smithi115 ceph-mon[72476]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55bcbd37-7a6f-4ab1-b5a9-91a951444d86"}]: dispatch 2024-03-20T08:54:13.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:12 smithi115 ceph-mon[72476]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "55bcbd37-7a6f-4ab1-b5a9-91a951444d86"}]': finished 2024-03-20T08:54:13.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:12 smithi115 ceph-mon[72476]: osdmap e57: 8 total, 7 up, 8 in 2024-03-20T08:54:13.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:12 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-20T08:54:13.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:12 smithi122 ceph-mon[73495]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-03-20T08:54:13.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:12 smithi122 ceph-mon[73495]: from='client.? 172.21.15.115:0/1659096563' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55bcbd37-7a6f-4ab1-b5a9-91a951444d86"}]: dispatch 2024-03-20T08:54:13.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:12 smithi122 ceph-mon[73495]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55bcbd37-7a6f-4ab1-b5a9-91a951444d86"}]: dispatch 2024-03-20T08:54:13.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:12 smithi122 ceph-mon[73495]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "55bcbd37-7a6f-4ab1-b5a9-91a951444d86"}]': finished 2024-03-20T08:54:13.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:12 smithi122 ceph-mon[73495]: osdmap e57: 8 total, 7 up, 8 in 2024-03-20T08:54:13.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:12 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-20T08:54:13.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:12 smithi026 ceph-mon[68309]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-03-20T08:54:13.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:12 smithi026 ceph-mon[68309]: from='client.? 172.21.15.115:0/1659096563' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55bcbd37-7a6f-4ab1-b5a9-91a951444d86"}]: dispatch 2024-03-20T08:54:13.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:12 smithi026 ceph-mon[68309]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55bcbd37-7a6f-4ab1-b5a9-91a951444d86"}]: dispatch 2024-03-20T08:54:13.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:12 smithi026 ceph-mon[68309]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "55bcbd37-7a6f-4ab1-b5a9-91a951444d86"}]': finished 2024-03-20T08:54:13.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:12 smithi026 ceph-mon[68309]: osdmap e57: 8 total, 7 up, 8 in 2024-03-20T08:54:13.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:12 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-20T08:54:14.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:13 smithi115 ceph-mon[72476]: from='client.? 172.21.15.115:0/3494307617' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:54:14.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:13 smithi122 ceph-mon[73495]: from='client.? 172.21.15.115:0/3494307617' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:54:14.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:13 smithi026 ceph-mon[68309]: from='client.? 172.21.15.115:0/3494307617' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:54:15.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:14 smithi115 ceph-mon[72476]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-20T08:54:15.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:14 smithi122 ceph-mon[73495]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-20T08:54:15.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:14 smithi026 ceph-mon[68309]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-20T08:54:17.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:16 smithi115 ceph-mon[72476]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-20T08:54:17.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:16 smithi122 ceph-mon[73495]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-20T08:54:17.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:16 smithi026 ceph-mon[68309]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-20T08:54:18.946 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:18 smithi115 ceph-mon[72476]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-20T08:54:19.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:18 smithi122 ceph-mon[73495]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-20T08:54:19.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:18 smithi026 ceph-mon[68309]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-20T08:54:20.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:19 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-20T08:54:20.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:19 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:54:20.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:19 smithi115 ceph-mon[72476]: Deploying daemon osd.7 on smithi115 2024-03-20T08:54:20.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:19 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-20T08:54:20.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:19 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:54:20.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:19 smithi122 ceph-mon[73495]: Deploying daemon osd.7 on smithi115 2024-03-20T08:54:20.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:19 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-20T08:54:20.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:19 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:54:20.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:19 smithi026 ceph-mon[68309]: Deploying daemon osd.7 on smithi115 2024-03-20T08:54:20.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:20 smithi115 ceph-mon[72476]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:54:21.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:20 smithi122 ceph-mon[73495]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:54:21.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:20 smithi026 ceph-mon[68309]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:54:22.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:22 smithi115 ceph-mon[72476]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:54:23.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:22 smithi122 ceph-mon[73495]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:54:23.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:22 smithi026 ceph-mon[68309]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:54:24.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:24 smithi115 ceph-mon[72476]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:54:24.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:24 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:24.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:24 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:24.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:24 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:54:25.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:24 smithi122 ceph-mon[73495]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:54:25.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:24 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:25.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:24 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:25.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:24 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:54:25.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:24 smithi026 ceph-mon[68309]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:54:25.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:24 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:25.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:24 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:25.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:24 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:54:26.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:25 smithi115 ceph-mon[72476]: 1.0 scrub starts 2024-03-20T08:54:26.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:25 smithi115 ceph-mon[72476]: 1.0 scrub ok 2024-03-20T08:54:26.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:25 smithi122 ceph-mon[73495]: 1.0 scrub starts 2024-03-20T08:54:26.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:25 smithi122 ceph-mon[73495]: 1.0 scrub ok 2024-03-20T08:54:26.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:25 smithi026 ceph-mon[68309]: 1.0 scrub starts 2024-03-20T08:54:26.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:25 smithi026 ceph-mon[68309]: 1.0 scrub ok 2024-03-20T08:54:27.119 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:26 smithi115 ceph-mon[72476]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:54:27.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:26 smithi122 ceph-mon[73495]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:54:27.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:26 smithi026 ceph-mon[68309]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:54:28.720 INFO:teuthology.orchestra.run.smithi115.stdout:Created osd(s) 7 on host 'smithi115' 2024-03-20T08:54:29.133 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:28 smithi115 ceph-mon[72476]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:54:29.133 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:28 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:29.133 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:28 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:29.133 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:28 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:54:29.133 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:28 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:54:29.133 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:28 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:29.133 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:28 smithi115 ceph-mon[72476]: from='osd.7 [v2:172.21.15.115:6824/3152424001,v1:172.21.15.115:6825/3152424001]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-20T08:54:29.133 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:28 smithi115 ceph-mon[72476]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-20T08:54:29.133 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:28 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:29.133 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:28 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:29.134 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:28 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:54:29.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:28 smithi122 ceph-mon[73495]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:54:29.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:28 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:29.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:28 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:29.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:28 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:54:29.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:28 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:54:29.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:28 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:29.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:28 smithi122 ceph-mon[73495]: from='osd.7 [v2:172.21.15.115:6824/3152424001,v1:172.21.15.115:6825/3152424001]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-20T08:54:29.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:28 smithi122 ceph-mon[73495]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-20T08:54:29.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:28 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:29.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:28 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:29.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:28 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:54:29.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:28 smithi026 ceph-mon[68309]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:54:29.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:28 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:29.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:28 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:29.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:28 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:54:29.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:28 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:54:29.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:28 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:29.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:28 smithi026 ceph-mon[68309]: from='osd.7 [v2:172.21.15.115:6824/3152424001,v1:172.21.15.115:6825/3152424001]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-20T08:54:29.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:28 smithi026 ceph-mon[68309]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-20T08:54:29.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:28 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:29.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:28 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:29.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:28 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:54:29.533 DEBUG:teuthology.orchestra.run.smithi115:osd.7> sudo journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.7.service 2024-03-20T08:54:29.535 INFO:tasks.cephadm:Deploying osd.8 on smithi122 with /dev/vg_nvme/lv_4... 2024-03-20T08:54:29.536 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-20T08:54:29.567 INFO:journalctl@ceph.osd.7.smithi115.stdout:-- Logs begin at Wed 2024-03-20 08:37:21 UTC. -- 2024-03-20T08:54:29.804 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T08:54:30.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:29 smithi115 ceph-mon[72476]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-20T08:54:30.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:29 smithi115 ceph-mon[72476]: osdmap e58: 8 total, 7 up, 8 in 2024-03-20T08:54:30.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:29 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-20T08:54:30.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:29 smithi115 ceph-mon[72476]: from='osd.7 [v2:172.21.15.115:6824/3152424001,v1:172.21.15.115:6825/3152424001]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-03-20T08:54:30.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:29 smithi115 ceph-mon[72476]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-03-20T08:54:30.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:29 smithi122 ceph-mon[73495]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-20T08:54:30.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:29 smithi122 ceph-mon[73495]: osdmap e58: 8 total, 7 up, 8 in 2024-03-20T08:54:30.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:29 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-20T08:54:30.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:29 smithi122 ceph-mon[73495]: from='osd.7 [v2:172.21.15.115:6824/3152424001,v1:172.21.15.115:6825/3152424001]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-03-20T08:54:30.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:29 smithi122 ceph-mon[73495]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-03-20T08:54:30.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:29 smithi026 ceph-mon[68309]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-20T08:54:30.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:29 smithi026 ceph-mon[68309]: osdmap e58: 8 total, 7 up, 8 in 2024-03-20T08:54:30.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:29 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-20T08:54:30.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:29 smithi026 ceph-mon[68309]: from='osd.7 [v2:172.21.15.115:6824/3152424001,v1:172.21.15.115:6825/3152424001]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-03-20T08:54:30.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:29 smithi026 ceph-mon[68309]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-03-20T08:54:30.655 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 08:54:30 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T08:54:30.326+0000 7ff186ec6700 -1 osd.7 0 waiting for initial osdmap 2024-03-20T08:54:30.656 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 08:54:30 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T08:54:30.334+0000 7ff17ecb1700 -1 osd.7 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-20T08:54:30.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:30 smithi122 ceph-mon[73495]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:54:30.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:30 smithi122 ceph-mon[73495]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-03-20T08:54:30.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:30 smithi122 ceph-mon[73495]: osdmap e59: 8 total, 7 up, 8 in 2024-03-20T08:54:30.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:30 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-20T08:54:30.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:30 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-20T08:54:30.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:30 smithi122 ceph-mon[73495]: from='osd.7 ' entity='osd.7' 2024-03-20T08:54:31.118 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:30 smithi115 ceph-mon[72476]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:54:31.118 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:30 smithi115 ceph-mon[72476]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-03-20T08:54:31.118 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:30 smithi115 ceph-mon[72476]: osdmap e59: 8 total, 7 up, 8 in 2024-03-20T08:54:31.119 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:30 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-20T08:54:31.119 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:30 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-20T08:54:31.119 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:30 smithi115 ceph-mon[72476]: from='osd.7 ' entity='osd.7' 2024-03-20T08:54:31.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:30 smithi026 ceph-mon[68309]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-20T08:54:31.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:30 smithi026 ceph-mon[68309]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]': finished 2024-03-20T08:54:31.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:30 smithi026 ceph-mon[68309]: osdmap e59: 8 total, 7 up, 8 in 2024-03-20T08:54:31.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:30 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-20T08:54:31.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:30 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-20T08:54:31.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:30 smithi026 ceph-mon[68309]: from='osd.7 ' entity='osd.7' 2024-03-20T08:54:31.951 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-20T08:54:31.951 INFO:teuthology.orchestra.run.smithi122.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-20T08:54:31.951 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10+0 records in 2024-03-20T08:54:31.951 INFO:teuthology.orchestra.run.smithi122.stderr:10+0 records out 2024-03-20T08:54:31.952 INFO:teuthology.orchestra.run.smithi122.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0107132 s, 979 MB/s 2024-03-20T08:54:31.952 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping successful for: 2024-03-20T08:54:32.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:31 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:54:32.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:31 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:54:32.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:31 smithi115 ceph-mon[72476]: osd.7 [v2:172.21.15.115:6824/3152424001,v1:172.21.15.115:6825/3152424001] boot 2024-03-20T08:54:32.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:31 smithi115 ceph-mon[72476]: osdmap e60: 8 total, 8 up, 8 in 2024-03-20T08:54:32.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:31 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-20T08:54:32.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:31 smithi115 ceph-mon[72476]: osdmap e61: 8 total, 8 up, 8 in 2024-03-20T08:54:32.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:31 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:54:32.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:31 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:54:32.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:31 smithi122 ceph-mon[73495]: osd.7 [v2:172.21.15.115:6824/3152424001,v1:172.21.15.115:6825/3152424001] boot 2024-03-20T08:54:32.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:31 smithi122 ceph-mon[73495]: osdmap e60: 8 total, 8 up, 8 in 2024-03-20T08:54:32.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:31 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-20T08:54:32.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:31 smithi122 ceph-mon[73495]: osdmap e61: 8 total, 8 up, 8 in 2024-03-20T08:54:32.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:31 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:54:32.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:31 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:54:32.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:31 smithi026 ceph-mon[68309]: osd.7 [v2:172.21.15.115:6824/3152424001,v1:172.21.15.115:6825/3152424001] boot 2024-03-20T08:54:32.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:31 smithi026 ceph-mon[68309]: osdmap e60: 8 total, 8 up, 8 in 2024-03-20T08:54:32.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:31 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-20T08:54:32.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:31 smithi026 ceph-mon[68309]: osdmap e61: 8 total, 8 up, 8 in 2024-03-20T08:54:32.594 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi122:vg_nvme/lv_4 2024-03-20T08:54:32.854 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T08:54:32.876 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:32 smithi122 ceph-mon[73495]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:54:32.876 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:32 smithi122 ceph-mon[73495]: osdmap e62: 8 total, 8 up, 8 in 2024-03-20T08:54:33.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:32 smithi115 ceph-mon[72476]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:54:33.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:32 smithi115 ceph-mon[72476]: osdmap e62: 8 total, 8 up, 8 in 2024-03-20T08:54:33.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:32 smithi026 ceph-mon[68309]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:54:33.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:32 smithi026 ceph-mon[68309]: osdmap e62: 8 total, 8 up, 8 in 2024-03-20T08:54:34.023 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:33 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:34.023 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:33 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:34.023 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:33 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:54:34.023 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:33 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:54:34.023 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:33 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:54:34.023 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:33 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:54:34.023 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:33 smithi122 ceph-mon[73495]: Adjusting osd_memory_target on smithi115 to 4280M 2024-03-20T08:54:34.024 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:33 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:34.024 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:33 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:54:34.024 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:33 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:54:34.024 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:33 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:54:34.024 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:33 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:54:34.024 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:33 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:54:34.024 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:33 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:34.024 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:33 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:54:34.024 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:33 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:54:34.024 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:33 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:54:34.024 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:33 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:34.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:33 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:34.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:33 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:34.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:33 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:54:34.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:33 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:54:34.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:33 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:54:34.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:33 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:54:34.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:33 smithi115 ceph-mon[72476]: Adjusting osd_memory_target on smithi115 to 4280M 2024-03-20T08:54:34.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:33 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:34.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:33 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:54:34.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:33 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:54:34.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:33 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:54:34.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:33 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:54:34.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:33 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:54:34.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:33 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:34.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:33 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:54:34.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:33 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:54:34.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:33 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:54:34.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:33 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:34.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:33 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:34.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:33 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:34.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:33 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:54:34.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:33 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:54:34.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:33 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:54:34.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:33 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:54:34.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:33 smithi026 ceph-mon[68309]: Adjusting osd_memory_target on smithi115 to 4280M 2024-03-20T08:54:34.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:33 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:34.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:33 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:54:34.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:33 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:54:34.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:33 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:54:34.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:33 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:54:34.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:33 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:54:34.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:33 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:34.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:33 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:54:34.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:33 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:54:34.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:33 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:54:34.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:33 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:35.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:34 smithi122 ceph-mon[73495]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:54:35.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:34 smithi122 ceph-mon[73495]: osdmap e63: 8 total, 8 up, 8 in 2024-03-20T08:54:35.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:34 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:54:35.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:34 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:54:35.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:34 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:54:35.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:34 smithi026 ceph-mon[68309]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:54:35.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:34 smithi026 ceph-mon[68309]: osdmap e63: 8 total, 8 up, 8 in 2024-03-20T08:54:35.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:34 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:54:35.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:34 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:54:35.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:34 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:54:35.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:34 smithi115 ceph-mon[72476]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:54:35.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:34 smithi115 ceph-mon[72476]: osdmap e63: 8 total, 8 up, 8 in 2024-03-20T08:54:35.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:34 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:54:35.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:34 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:54:35.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:34 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:54:36.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:35 smithi122 ceph-mon[73495]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:54:36.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:35 smithi122 ceph-mon[73495]: osdmap e64: 8 total, 8 up, 8 in 2024-03-20T08:54:36.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:35 smithi026 ceph-mon[68309]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:54:36.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:35 smithi026 ceph-mon[68309]: osdmap e64: 8 total, 8 up, 8 in 2024-03-20T08:54:36.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:35 smithi115 ceph-mon[72476]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:54:36.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:35 smithi115 ceph-mon[72476]: osdmap e64: 8 total, 8 up, 8 in 2024-03-20T08:54:37.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:36 smithi026 ceph-mon[68309]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:54:37.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:36 smithi115 ceph-mon[72476]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:54:37.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:36 smithi122 ceph-mon[73495]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:54:38.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:37 smithi026 ceph-mon[68309]: from='client.? 172.21.15.122:0/3735477189' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc86248e-6bd0-4369-b8a2-e593bc2a610d"}]: dispatch 2024-03-20T08:54:38.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:37 smithi026 ceph-mon[68309]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc86248e-6bd0-4369-b8a2-e593bc2a610d"}]: dispatch 2024-03-20T08:54:38.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:37 smithi026 ceph-mon[68309]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc86248e-6bd0-4369-b8a2-e593bc2a610d"}]': finished 2024-03-20T08:54:38.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:37 smithi026 ceph-mon[68309]: osdmap e65: 9 total, 8 up, 9 in 2024-03-20T08:54:38.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:37 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-20T08:54:38.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:37 smithi115 ceph-mon[72476]: from='client.? 172.21.15.122:0/3735477189' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc86248e-6bd0-4369-b8a2-e593bc2a610d"}]: dispatch 2024-03-20T08:54:38.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:37 smithi115 ceph-mon[72476]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc86248e-6bd0-4369-b8a2-e593bc2a610d"}]: dispatch 2024-03-20T08:54:38.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:37 smithi115 ceph-mon[72476]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc86248e-6bd0-4369-b8a2-e593bc2a610d"}]': finished 2024-03-20T08:54:38.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:37 smithi115 ceph-mon[72476]: osdmap e65: 9 total, 8 up, 9 in 2024-03-20T08:54:38.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:37 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-20T08:54:38.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:37 smithi122 ceph-mon[73495]: from='client.? 172.21.15.122:0/3735477189' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc86248e-6bd0-4369-b8a2-e593bc2a610d"}]: dispatch 2024-03-20T08:54:38.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:37 smithi122 ceph-mon[73495]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc86248e-6bd0-4369-b8a2-e593bc2a610d"}]: dispatch 2024-03-20T08:54:38.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:37 smithi122 ceph-mon[73495]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc86248e-6bd0-4369-b8a2-e593bc2a610d"}]': finished 2024-03-20T08:54:38.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:37 smithi122 ceph-mon[73495]: osdmap e65: 9 total, 8 up, 9 in 2024-03-20T08:54:38.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:37 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-20T08:54:39.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:38 smithi026 ceph-mon[68309]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-20T08:54:39.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:38 smithi026 ceph-mon[68309]: from='client.? 172.21.15.122:0/83896129' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:54:39.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:38 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:54:39.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:38 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:54:39.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:38 smithi115 ceph-mon[72476]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-20T08:54:39.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:38 smithi115 ceph-mon[72476]: from='client.? 172.21.15.122:0/83896129' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:54:39.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:38 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:54:39.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:38 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:54:39.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:38 smithi122 ceph-mon[73495]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-20T08:54:39.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:38 smithi122 ceph-mon[73495]: from='client.? 172.21.15.122:0/83896129' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:54:39.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:38 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:54:39.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:38 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:54:40.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:39 smithi026 ceph-mon[68309]: osdmap e66: 9 total, 8 up, 9 in 2024-03-20T08:54:40.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:39 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-20T08:54:40.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:39 smithi115 ceph-mon[72476]: osdmap e66: 9 total, 8 up, 9 in 2024-03-20T08:54:40.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:39 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-20T08:54:40.414 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:39 smithi122 ceph-mon[73495]: osdmap e66: 9 total, 8 up, 9 in 2024-03-20T08:54:40.415 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:39 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-20T08:54:41.108 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:40 smithi122 ceph-mon[73495]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-03-20T08:54:41.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:40 smithi026 ceph-mon[68309]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-03-20T08:54:41.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:40 smithi115 ceph-mon[72476]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-03-20T08:54:43.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:42 smithi122 ceph-mon[73495]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-20T08:54:43.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:42 smithi026 ceph-mon[68309]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-20T08:54:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:42 smithi115 ceph-mon[72476]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-20T08:54:45.393 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:44 smithi122 ceph-mon[73495]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-20T08:54:45.393 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:44 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-20T08:54:45.393 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:44 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:54:45.394 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:44 smithi122 ceph-mon[73495]: Deploying daemon osd.8 on smithi122 2024-03-20T08:54:45.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:44 smithi115 ceph-mon[72476]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-20T08:54:45.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:44 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-20T08:54:45.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:44 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:54:45.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:44 smithi115 ceph-mon[72476]: Deploying daemon osd.8 on smithi122 2024-03-20T08:54:45.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:44 smithi026 ceph-mon[68309]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-20T08:54:45.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:44 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-20T08:54:45.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:44 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:54:45.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:44 smithi026 ceph-mon[68309]: Deploying daemon osd.8 on smithi122 2024-03-20T08:54:47.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:46 smithi115 ceph-mon[72476]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:54:47.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:46 smithi122 ceph-mon[73495]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:54:47.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:46 smithi026 ceph-mon[68309]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:54:49.301 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:48 smithi122 ceph-mon[73495]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:54:49.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:48 smithi115 ceph-mon[72476]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:54:49.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:48 smithi026 ceph-mon[68309]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:54:50.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:50 smithi122 ceph-mon[73495]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:54:50.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:50 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:50.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:50 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:54:50.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:50 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:50.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:50 smithi026 ceph-mon[68309]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:54:50.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:50 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:50.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:50 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:54:50.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:50 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:50.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:50 smithi115 ceph-mon[72476]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:54:50.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:50 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:50.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:50 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:54:50.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:50 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:52.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:52 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:54:52.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:52 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:54:52.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:52 smithi122 ceph-mon[73495]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:54:52.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:52 smithi122 ceph-mon[73495]: osdmap e67: 9 total, 8 up, 9 in 2024-03-20T08:54:52.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:52 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-20T08:54:52.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:52 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:52.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:52 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:52.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:52 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:54:52.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:52 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:54:52.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:52 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:52.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:52 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:54:52.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:52 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:54:52.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:52 smithi026 ceph-mon[68309]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:54:52.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:52 smithi026 ceph-mon[68309]: osdmap e67: 9 total, 8 up, 9 in 2024-03-20T08:54:52.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:52 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-20T08:54:52.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:52 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:52.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:52 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:52.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:52 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:54:52.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:52 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:54:52.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:52 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:52.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:52 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:54:52.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:52 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:54:52.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:52 smithi115 ceph-mon[72476]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:54:52.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:52 smithi115 ceph-mon[72476]: osdmap e67: 9 total, 8 up, 9 in 2024-03-20T08:54:52.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:52 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-20T08:54:52.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:52 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:52.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:52 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:52.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:52 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:54:52.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:52 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:54:52.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:52 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:53.291 INFO:teuthology.orchestra.run.smithi122.stdout:Created osd(s) 8 on host 'smithi122' 2024-03-20T08:54:53.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:53 smithi122 ceph-mon[73495]: from='osd.8 [v2:172.21.15.122:6800/1309426740,v1:172.21.15.122:6801/1309426740]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-20T08:54:53.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:53 smithi122 ceph-mon[73495]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-20T08:54:53.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:53 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:53.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:53 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:53.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:53 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:54:53.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:53 smithi026 ceph-mon[68309]: from='osd.8 [v2:172.21.15.122:6800/1309426740,v1:172.21.15.122:6801/1309426740]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-20T08:54:53.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:53 smithi026 ceph-mon[68309]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-20T08:54:53.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:53 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:53.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:53 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:53.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:53 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:54:53.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:53 smithi115 ceph-mon[72476]: from='osd.8 [v2:172.21.15.122:6800/1309426740,v1:172.21.15.122:6801/1309426740]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-20T08:54:53.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:53 smithi115 ceph-mon[72476]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-20T08:54:53.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:53 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:53.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:53 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:53.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:53 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:54:54.049 DEBUG:teuthology.orchestra.run.smithi122:osd.8> sudo journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.8.service 2024-03-20T08:54:54.051 INFO:tasks.cephadm:Deploying osd.9 on smithi122 with /dev/vg_nvme/lv_3... 2024-03-20T08:54:54.052 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-20T08:54:54.082 INFO:journalctl@ceph.osd.8.smithi122.stdout:-- Logs begin at Wed 2024-03-20 08:37:02 UTC. -- 2024-03-20T08:54:54.540 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:54 smithi122 ceph-mon[73495]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:54:54.541 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:54 smithi122 ceph-mon[73495]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-20T08:54:54.541 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:54 smithi122 ceph-mon[73495]: osdmap e68: 9 total, 8 up, 9 in 2024-03-20T08:54:54.541 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:54 smithi122 ceph-mon[73495]: from='osd.8 [v2:172.21.15.122:6800/1309426740,v1:172.21.15.122:6801/1309426740]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-20T08:54:54.541 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:54 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-20T08:54:54.541 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:54 smithi122 ceph-mon[73495]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-20T08:54:54.541 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 08:54:54 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T08:54:54.451+0000 7f96e9bd4700 -1 osd.8 0 waiting for initial osdmap 2024-03-20T08:54:54.541 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 08:54:54 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T08:54:54.451+0000 7f96e1367700 -1 osd.8 0 failed to load OSD map for epoch 13, got 0 bytes 2024-03-20T08:54:54.541 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 08:54:54 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T08:54:54.457+0000 7f96e59c7700 -1 osd.8 69 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-20T08:54:54.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:54 smithi026 ceph-mon[68309]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:54:54.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:54 smithi026 ceph-mon[68309]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-20T08:54:54.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:54 smithi026 ceph-mon[68309]: osdmap e68: 9 total, 8 up, 9 in 2024-03-20T08:54:54.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:54 smithi026 ceph-mon[68309]: from='osd.8 [v2:172.21.15.122:6800/1309426740,v1:172.21.15.122:6801/1309426740]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-20T08:54:54.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:54 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-20T08:54:54.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:54 smithi026 ceph-mon[68309]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-20T08:54:54.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:54 smithi115 ceph-mon[72476]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:54:54.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:54 smithi115 ceph-mon[72476]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-20T08:54:54.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:54 smithi115 ceph-mon[72476]: osdmap e68: 9 total, 8 up, 9 in 2024-03-20T08:54:54.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:54 smithi115 ceph-mon[72476]: from='osd.8 [v2:172.21.15.122:6800/1309426740,v1:172.21.15.122:6801/1309426740]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-20T08:54:54.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:54 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-20T08:54:54.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:54 smithi115 ceph-mon[72476]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-20T08:54:55.229 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T08:54:55.512 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:55 smithi122 ceph-mon[73495]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-03-20T08:54:55.512 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:55 smithi122 ceph-mon[73495]: osdmap e69: 9 total, 8 up, 9 in 2024-03-20T08:54:55.512 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:55 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-20T08:54:55.512 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:55 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-20T08:54:55.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:55 smithi026 ceph-mon[68309]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-03-20T08:54:55.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:55 smithi026 ceph-mon[68309]: osdmap e69: 9 total, 8 up, 9 in 2024-03-20T08:54:55.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:55 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-20T08:54:55.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:55 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-20T08:54:55.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:55 smithi115 ceph-mon[72476]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-03-20T08:54:55.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:55 smithi115 ceph-mon[72476]: osdmap e69: 9 total, 8 up, 9 in 2024-03-20T08:54:55.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:55 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-20T08:54:55.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:55 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-20T08:54:56.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:56 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:54:56.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:56 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:54:56.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:56 smithi026 ceph-mon[68309]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:54:56.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:56 smithi026 ceph-mon[68309]: osd.8 [v2:172.21.15.122:6800/1309426740,v1:172.21.15.122:6801/1309426740] boot 2024-03-20T08:54:56.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:56 smithi026 ceph-mon[68309]: osdmap e70: 9 total, 9 up, 9 in 2024-03-20T08:54:56.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:56 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-20T08:54:56.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:56 smithi026 ceph-mon[68309]: osdmap e71: 9 total, 9 up, 9 in 2024-03-20T08:54:56.817 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:56 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:54:56.818 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:56 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:54:56.818 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:56 smithi122 ceph-mon[73495]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:54:56.818 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:56 smithi122 ceph-mon[73495]: osd.8 [v2:172.21.15.122:6800/1309426740,v1:172.21.15.122:6801/1309426740] boot 2024-03-20T08:54:56.818 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:56 smithi122 ceph-mon[73495]: osdmap e70: 9 total, 9 up, 9 in 2024-03-20T08:54:56.818 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:56 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-20T08:54:56.818 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:56 smithi122 ceph-mon[73495]: osdmap e71: 9 total, 9 up, 9 in 2024-03-20T08:54:56.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:56 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:54:56.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:56 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:54:56.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:56 smithi115 ceph-mon[72476]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-03-20T08:54:56.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:56 smithi115 ceph-mon[72476]: osd.8 [v2:172.21.15.122:6800/1309426740,v1:172.21.15.122:6801/1309426740] boot 2024-03-20T08:54:56.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:56 smithi115 ceph-mon[72476]: osdmap e70: 9 total, 9 up, 9 in 2024-03-20T08:54:56.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:56 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-20T08:54:56.909 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:56 smithi115 ceph-mon[72476]: osdmap e71: 9 total, 9 up, 9 in 2024-03-20T08:54:58.280 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-20T08:54:58.280 INFO:teuthology.orchestra.run.smithi122.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-20T08:54:58.280 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10+0 records in 2024-03-20T08:54:58.280 INFO:teuthology.orchestra.run.smithi122.stderr:10+0 records out 2024-03-20T08:54:58.280 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0119915 s, 874 MB/s 2024-03-20T08:54:58.280 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping successful for: 2024-03-20T08:54:58.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:58 smithi122 ceph-mon[73495]: pgmap v222: 1 pgs: 1 remapped+peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-03-20T08:54:58.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:58 smithi122 ceph-mon[73495]: osdmap e72: 9 total, 9 up, 9 in 2024-03-20T08:54:58.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:58 smithi122 ceph-mon[73495]: Detected new or changed devices on smithi122 2024-03-20T08:54:58.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:58 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:58.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:58 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:58.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:58 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:54:58.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:58 smithi122 ceph-mon[73495]: Adjusting osd_memory_target on smithi122 to 21218M 2024-03-20T08:54:58.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:58 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:58.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:58 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:54:58.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:58 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:54:58.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:58 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:58.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:58 smithi026 ceph-mon[68309]: pgmap v222: 1 pgs: 1 remapped+peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-03-20T08:54:58.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:58 smithi026 ceph-mon[68309]: osdmap e72: 9 total, 9 up, 9 in 2024-03-20T08:54:58.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:58 smithi026 ceph-mon[68309]: Detected new or changed devices on smithi122 2024-03-20T08:54:58.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:58 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:58.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:58 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:58.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:58 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:54:58.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:58 smithi026 ceph-mon[68309]: Adjusting osd_memory_target on smithi122 to 21218M 2024-03-20T08:54:58.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:58 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:58.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:58 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:54:58.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:58 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:54:58.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:58 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:58.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:58 smithi115 ceph-mon[72476]: pgmap v222: 1 pgs: 1 remapped+peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-03-20T08:54:58.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:58 smithi115 ceph-mon[72476]: osdmap e72: 9 total, 9 up, 9 in 2024-03-20T08:54:58.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:58 smithi115 ceph-mon[72476]: Detected new or changed devices on smithi122 2024-03-20T08:54:58.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:58 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:58.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:58 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:58.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:58 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:54:58.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:58 smithi115 ceph-mon[72476]: Adjusting osd_memory_target on smithi122 to 21218M 2024-03-20T08:54:58.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:58 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:58.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:58 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:54:58.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:58 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:54:58.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:58 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:54:59.011 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi122:vg_nvme/lv_3 2024-03-20T08:54:59.267 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T08:54:59.718 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:54:59 smithi122 ceph-mon[73495]: osdmap e73: 9 total, 9 up, 9 in 2024-03-20T08:54:59.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:54:59 smithi026 ceph-mon[68309]: osdmap e73: 9 total, 9 up, 9 in 2024-03-20T08:54:59.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:54:59 smithi115 ceph-mon[72476]: osdmap e73: 9 total, 9 up, 9 in 2024-03-20T08:55:00.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:00 smithi122 ceph-mon[73495]: pgmap v225: 1 pgs: 1 remapped+peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-03-20T08:55:00.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:00 smithi122 ceph-mon[73495]: osdmap e74: 9 total, 9 up, 9 in 2024-03-20T08:55:00.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:00 smithi026 ceph-mon[68309]: pgmap v225: 1 pgs: 1 remapped+peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-03-20T08:55:00.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:00 smithi026 ceph-mon[68309]: osdmap e74: 9 total, 9 up, 9 in 2024-03-20T08:55:00.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:00 smithi115 ceph-mon[72476]: pgmap v225: 1 pgs: 1 remapped+peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-03-20T08:55:00.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:00 smithi115 ceph-mon[72476]: osdmap e74: 9 total, 9 up, 9 in 2024-03-20T08:55:01.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:01 smithi026 ceph-mon[68309]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:55:01.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:01 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:55:01.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:01 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:55:01.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:01 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:55:01.738 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:01 smithi122 ceph-mon[73495]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:55:01.738 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:01 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:55:01.738 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:01 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:55:01.738 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:01 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:55:01.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:01 smithi115 ceph-mon[72476]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:55:01.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:01 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:55:01.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:01 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:55:01.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:01 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:55:02.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:02 smithi122 ceph-mon[73495]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-03-20T08:55:02.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:02 smithi026 ceph-mon[68309]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-03-20T08:55:03.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:02 smithi115 ceph-mon[72476]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-03-20T08:55:04.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:04 smithi026 ceph-mon[68309]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-20T08:55:04.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:04 smithi026 ceph-mon[68309]: from='client.? 172.21.15.122:0/1969728271' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b20a4aa-032a-4c09-ac75-153c554424e7"}]: dispatch 2024-03-20T08:55:04.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:04 smithi026 ceph-mon[68309]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b20a4aa-032a-4c09-ac75-153c554424e7"}]: dispatch 2024-03-20T08:55:04.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:04 smithi026 ceph-mon[68309]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9b20a4aa-032a-4c09-ac75-153c554424e7"}]': finished 2024-03-20T08:55:04.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:04 smithi026 ceph-mon[68309]: osdmap e75: 10 total, 9 up, 10 in 2024-03-20T08:55:04.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:04 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-20T08:55:04.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:04 smithi026 ceph-mon[68309]: from='client.? 172.21.15.122:0/2004863268' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:55:05.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:04 smithi115 ceph-mon[72476]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-20T08:55:05.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:04 smithi115 ceph-mon[72476]: from='client.? 172.21.15.122:0/1969728271' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b20a4aa-032a-4c09-ac75-153c554424e7"}]: dispatch 2024-03-20T08:55:05.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:04 smithi115 ceph-mon[72476]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b20a4aa-032a-4c09-ac75-153c554424e7"}]: dispatch 2024-03-20T08:55:05.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:04 smithi115 ceph-mon[72476]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9b20a4aa-032a-4c09-ac75-153c554424e7"}]': finished 2024-03-20T08:55:05.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:04 smithi115 ceph-mon[72476]: osdmap e75: 10 total, 9 up, 10 in 2024-03-20T08:55:05.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:04 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-20T08:55:05.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:04 smithi115 ceph-mon[72476]: from='client.? 172.21.15.122:0/2004863268' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:55:05.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:04 smithi122 ceph-mon[73495]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-20T08:55:05.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:04 smithi122 ceph-mon[73495]: from='client.? 172.21.15.122:0/1969728271' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b20a4aa-032a-4c09-ac75-153c554424e7"}]: dispatch 2024-03-20T08:55:05.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:04 smithi122 ceph-mon[73495]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b20a4aa-032a-4c09-ac75-153c554424e7"}]: dispatch 2024-03-20T08:55:05.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:04 smithi122 ceph-mon[73495]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9b20a4aa-032a-4c09-ac75-153c554424e7"}]': finished 2024-03-20T08:55:05.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:04 smithi122 ceph-mon[73495]: osdmap e75: 10 total, 9 up, 10 in 2024-03-20T08:55:05.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:04 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-20T08:55:05.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:04 smithi122 ceph-mon[73495]: from='client.? 172.21.15.122:0/2004863268' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:55:06.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:06 smithi026 ceph-mon[68309]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-20T08:55:07.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:06 smithi115 ceph-mon[72476]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-20T08:55:07.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:06 smithi122 ceph-mon[73495]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-20T08:55:08.924 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:08 smithi122 ceph-mon[73495]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-20T08:55:08.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:08 smithi026 ceph-mon[68309]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-20T08:55:09.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:08 smithi115 ceph-mon[72476]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-20T08:55:09.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:09 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:55:09.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:09 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:55:09.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:09 smithi026 ceph-mon[68309]: osdmap e76: 10 total, 9 up, 10 in 2024-03-20T08:55:09.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:09 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-20T08:55:09.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:09 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:55:09.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:09 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:55:10.101 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:09 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:55:10.102 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:09 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:55:10.102 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:09 smithi122 ceph-mon[73495]: osdmap e76: 10 total, 9 up, 10 in 2024-03-20T08:55:10.102 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:09 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-20T08:55:10.102 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:09 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:55:10.102 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:09 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:55:10.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:09 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:55:10.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:09 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:55:10.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:09 smithi115 ceph-mon[72476]: osdmap e76: 10 total, 9 up, 10 in 2024-03-20T08:55:10.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:09 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-20T08:55:10.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:09 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:55:10.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:09 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:55:10.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:10 smithi026 ceph-mon[68309]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-03-20T08:55:10.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:10 smithi026 ceph-mon[68309]: osdmap e77: 10 total, 9 up, 10 in 2024-03-20T08:55:10.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:10 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-20T08:55:10.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:10 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-20T08:55:10.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:10 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:55:11.004 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:10 smithi122 ceph-mon[73495]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-03-20T08:55:11.004 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:10 smithi122 ceph-mon[73495]: osdmap e77: 10 total, 9 up, 10 in 2024-03-20T08:55:11.004 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:10 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-20T08:55:11.004 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:10 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-20T08:55:11.004 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:10 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:55:11.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:10 smithi115 ceph-mon[72476]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-03-20T08:55:11.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:10 smithi115 ceph-mon[72476]: osdmap e77: 10 total, 9 up, 10 in 2024-03-20T08:55:11.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:10 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-20T08:55:11.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:10 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-20T08:55:11.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:10 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:55:11.930 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:11 smithi122 ceph-mon[73495]: Deploying daemon osd.9 on smithi122 2024-03-20T08:55:11.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:11 smithi026 ceph-mon[68309]: Deploying daemon osd.9 on smithi122 2024-03-20T08:55:12.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:11 smithi115 ceph-mon[72476]: Deploying daemon osd.9 on smithi122 2024-03-20T08:55:13.023 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:12 smithi122 ceph-mon[73495]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-03-20T08:55:13.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:12 smithi115 ceph-mon[72476]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-03-20T08:55:13.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:12 smithi026 ceph-mon[68309]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2024-03-20T08:55:14.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:14 smithi122 ceph-mon[73495]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-03-20T08:55:15.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:14 smithi115 ceph-mon[72476]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-03-20T08:55:15.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:14 smithi026 ceph-mon[68309]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-03-20T08:55:16.750 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:16 smithi122 ceph-mon[73495]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-03-20T08:55:16.750 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:16 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:55:16.750 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:16 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:16.750 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:16 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:17.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:16 smithi115 ceph-mon[72476]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-03-20T08:55:17.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:16 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:55:17.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:16 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:17.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:16 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:17.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:16 smithi026 ceph-mon[68309]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-03-20T08:55:17.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:16 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:55:17.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:16 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:17.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:16 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:19.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:18 smithi115 ceph-mon[72476]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-03-20T08:55:19.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:18 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:19.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:18 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:19.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:18 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:55:19.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:18 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:55:19.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:18 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:19.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:18 smithi122 ceph-mon[73495]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-03-20T08:55:19.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:18 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:19.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:18 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:19.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:18 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:55:19.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:18 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:55:19.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:18 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:19.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:18 smithi026 ceph-mon[68309]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-03-20T08:55:19.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:18 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:19.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:18 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:19.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:18 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:55:19.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:18 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:55:19.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:18 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:19.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:19 smithi122 ceph-mon[73495]: from='osd.9 [v2:172.21.15.122:6808/2953210076,v1:172.21.15.122:6809/2953210076]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-20T08:55:19.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:19 smithi122 ceph-mon[73495]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-20T08:55:20.123 INFO:teuthology.orchestra.run.smithi122.stdout:Created osd(s) 9 on host 'smithi122' 2024-03-20T08:55:20.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:19 smithi115 ceph-mon[72476]: from='osd.9 [v2:172.21.15.122:6808/2953210076,v1:172.21.15.122:6809/2953210076]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-20T08:55:20.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:19 smithi115 ceph-mon[72476]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-20T08:55:20.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:19 smithi026 ceph-mon[68309]: from='osd.9 [v2:172.21.15.122:6808/2953210076,v1:172.21.15.122:6809/2953210076]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-20T08:55:20.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:19 smithi026 ceph-mon[68309]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-20T08:55:21.075 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:20 smithi122 ceph-mon[73495]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-03-20T08:55:21.075 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:20 smithi122 ceph-mon[73495]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-20T08:55:21.075 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:20 smithi122 ceph-mon[73495]: from='osd.9 [v2:172.21.15.122:6808/2953210076,v1:172.21.15.122:6809/2953210076]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-20T08:55:21.076 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:20 smithi122 ceph-mon[73495]: osdmap e78: 10 total, 9 up, 10 in 2024-03-20T08:55:21.076 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:20 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-20T08:55:21.076 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:20 smithi122 ceph-mon[73495]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-20T08:55:21.076 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:20 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:21.076 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:20 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:21.076 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:20 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:55:21.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:20 smithi115 ceph-mon[72476]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-03-20T08:55:21.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:20 smithi115 ceph-mon[72476]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-20T08:55:21.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:20 smithi115 ceph-mon[72476]: from='osd.9 [v2:172.21.15.122:6808/2953210076,v1:172.21.15.122:6809/2953210076]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-20T08:55:21.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:20 smithi115 ceph-mon[72476]: osdmap e78: 10 total, 9 up, 10 in 2024-03-20T08:55:21.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:20 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-20T08:55:21.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:20 smithi115 ceph-mon[72476]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-20T08:55:21.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:20 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:21.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:20 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:21.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:20 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:55:21.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:20 smithi026 ceph-mon[68309]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-03-20T08:55:21.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:20 smithi026 ceph-mon[68309]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-20T08:55:21.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:20 smithi026 ceph-mon[68309]: from='osd.9 [v2:172.21.15.122:6808/2953210076,v1:172.21.15.122:6809/2953210076]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-20T08:55:21.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:20 smithi026 ceph-mon[68309]: osdmap e78: 10 total, 9 up, 10 in 2024-03-20T08:55:21.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:20 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-20T08:55:21.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:20 smithi026 ceph-mon[68309]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-20T08:55:21.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:20 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:21.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:20 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:21.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:20 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:55:22.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:21 smithi115 ceph-mon[72476]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-03-20T08:55:22.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:21 smithi115 ceph-mon[72476]: osdmap e79: 10 total, 9 up, 10 in 2024-03-20T08:55:22.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:21 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-20T08:55:22.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:21 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-20T08:55:22.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:21 smithi115 ceph-mon[72476]: from='osd.9 ' entity='osd.9' 2024-03-20T08:55:22.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:21 smithi122 ceph-mon[73495]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-03-20T08:55:22.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:21 smithi122 ceph-mon[73495]: osdmap e79: 10 total, 9 up, 10 in 2024-03-20T08:55:22.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:21 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-20T08:55:22.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:21 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-20T08:55:22.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:21 smithi122 ceph-mon[73495]: from='osd.9 ' entity='osd.9' 2024-03-20T08:55:22.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:21 smithi026 ceph-mon[68309]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-03-20T08:55:22.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:21 smithi026 ceph-mon[68309]: osdmap e79: 10 total, 9 up, 10 in 2024-03-20T08:55:22.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:21 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-20T08:55:22.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:21 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-20T08:55:22.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:21 smithi026 ceph-mon[68309]: from='osd.9 ' entity='osd.9' 2024-03-20T08:55:22.267 DEBUG:teuthology.orchestra.run.smithi122:osd.9> sudo journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.9.service 2024-03-20T08:55:22.269 INFO:tasks.cephadm:Deploying osd.10 on smithi122 with /dev/vg_nvme/lv_2... 2024-03-20T08:55:22.269 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-20T08:55:22.301 INFO:journalctl@ceph.osd.9.smithi122.stdout:-- Logs begin at Wed 2024-03-20 08:37:02 UTC. -- 2024-03-20T08:55:22.834 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T08:55:22.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:22 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:55:22.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:22 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:55:22.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:22 smithi122 ceph-mon[73495]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-03-20T08:55:22.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:22 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-20T08:55:22.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:22 smithi122 ceph-mon[73495]: osd.9 [v2:172.21.15.122:6808/2953210076,v1:172.21.15.122:6809/2953210076] boot 2024-03-20T08:55:22.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:22 smithi122 ceph-mon[73495]: osdmap e80: 10 total, 10 up, 10 in 2024-03-20T08:55:22.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:22 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-20T08:55:23.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:22 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:55:23.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:22 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:55:23.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:22 smithi115 ceph-mon[72476]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-03-20T08:55:23.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:22 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-20T08:55:23.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:22 smithi115 ceph-mon[72476]: osd.9 [v2:172.21.15.122:6808/2953210076,v1:172.21.15.122:6809/2953210076] boot 2024-03-20T08:55:23.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:22 smithi115 ceph-mon[72476]: osdmap e80: 10 total, 10 up, 10 in 2024-03-20T08:55:23.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:22 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-20T08:55:23.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:22 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:55:23.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:22 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:55:23.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:22 smithi026 ceph-mon[68309]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-03-20T08:55:23.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:22 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-20T08:55:23.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:22 smithi026 ceph-mon[68309]: osd.9 [v2:172.21.15.122:6808/2953210076,v1:172.21.15.122:6809/2953210076] boot 2024-03-20T08:55:23.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:22 smithi026 ceph-mon[68309]: osdmap e80: 10 total, 10 up, 10 in 2024-03-20T08:55:23.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:22 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-20T08:55:24.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:24 smithi122 ceph-mon[73495]: osdmap e81: 10 total, 10 up, 10 in 2024-03-20T08:55:24.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:24 smithi026 ceph-mon[68309]: osdmap e81: 10 total, 10 up, 10 in 2024-03-20T08:55:24.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:24 smithi115 ceph-mon[72476]: osdmap e81: 10 total, 10 up, 10 in 2024-03-20T08:55:25.436 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-20T08:55:25.436 INFO:teuthology.orchestra.run.smithi122.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-20T08:55:25.436 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10+0 records in 2024-03-20T08:55:25.436 INFO:teuthology.orchestra.run.smithi122.stderr:10+0 records out 2024-03-20T08:55:25.436 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0106318 s, 986 MB/s 2024-03-20T08:55:25.436 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping successful for: 2024-03-20T08:55:25.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:25 smithi026 ceph-mon[68309]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:25.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:25 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:55:25.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:25 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:55:25.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:25 smithi026 ceph-mon[68309]: osdmap e82: 10 total, 10 up, 10 in 2024-03-20T08:55:25.475 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:25 smithi122 ceph-mon[73495]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:25.475 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:25 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:55:25.475 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:25 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:55:25.475 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:25 smithi122 ceph-mon[73495]: osdmap e82: 10 total, 10 up, 10 in 2024-03-20T08:55:25.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:25 smithi115 ceph-mon[72476]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:25.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:25 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:55:25.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:25 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:55:25.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:25 smithi115 ceph-mon[72476]: osdmap e82: 10 total, 10 up, 10 in 2024-03-20T08:55:26.327 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi122:vg_nvme/lv_2 2024-03-20T08:55:26.586 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T08:55:27.103 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:26 smithi122 ceph-mon[73495]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:27.104 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:26 smithi122 ceph-mon[73495]: Detected new or changed devices on smithi122 2024-03-20T08:55:27.104 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:27.104 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:27.104 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:55:27.104 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:55:27.104 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:26 smithi122 ceph-mon[73495]: Adjusting osd_memory_target on smithi122 to 10609M 2024-03-20T08:55:27.104 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:27.104 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:55:27.104 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:55:27.104 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:27.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:26 smithi115 ceph-mon[72476]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:27.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:26 smithi115 ceph-mon[72476]: Detected new or changed devices on smithi122 2024-03-20T08:55:27.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:26 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:27.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:26 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:27.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:26 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:55:27.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:26 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:55:27.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:26 smithi115 ceph-mon[72476]: Adjusting osd_memory_target on smithi122 to 10609M 2024-03-20T08:55:27.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:26 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:27.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:26 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:55:27.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:26 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:55:27.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:26 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:27.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:26 smithi026 ceph-mon[68309]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:27.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:26 smithi026 ceph-mon[68309]: Detected new or changed devices on smithi122 2024-03-20T08:55:27.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:26 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:27.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:26 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:27.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:26 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:55:27.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:26 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:55:27.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:26 smithi026 ceph-mon[68309]: Adjusting osd_memory_target on smithi122 to 10609M 2024-03-20T08:55:27.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:26 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:27.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:26 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:55:27.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:26 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:55:27.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:26 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:29.150 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:28 smithi122 ceph-mon[73495]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:29.150 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:28 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:55:29.150 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:28 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:55:29.150 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:28 smithi122 ceph-mon[73495]: osdmap e83: 10 total, 10 up, 10 in 2024-03-20T08:55:29.151 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:28 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:55:29.151 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:28 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:55:29.151 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:28 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:55:29.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:28 smithi115 ceph-mon[72476]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:29.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:28 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:55:29.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:28 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:55:29.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:28 smithi115 ceph-mon[72476]: osdmap e83: 10 total, 10 up, 10 in 2024-03-20T08:55:29.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:28 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:55:29.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:28 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:55:29.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:28 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:55:29.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:28 smithi026 ceph-mon[68309]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:29.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:28 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:55:29.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:28 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:55:29.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:28 smithi026 ceph-mon[68309]: osdmap e83: 10 total, 10 up, 10 in 2024-03-20T08:55:29.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:28 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:55:29.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:28 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:55:29.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:28 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:55:30.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:29 smithi115 ceph-mon[72476]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:55:30.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:29 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:55:30.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:29 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:55:30.171 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:29 smithi122 ceph-mon[73495]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:55:30.171 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:29 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:55:30.171 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:29 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:55:30.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:29 smithi026 ceph-mon[68309]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:55:30.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:29 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:55:30.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:29 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:55:31.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:30 smithi115 ceph-mon[72476]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:31.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:30 smithi115 ceph-mon[72476]: osdmap e84: 10 total, 10 up, 10 in 2024-03-20T08:55:31.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:30 smithi122 ceph-mon[73495]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:31.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:30 smithi122 ceph-mon[73495]: osdmap e84: 10 total, 10 up, 10 in 2024-03-20T08:55:31.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:30 smithi026 ceph-mon[68309]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:31.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:30 smithi026 ceph-mon[68309]: osdmap e84: 10 total, 10 up, 10 in 2024-03-20T08:55:32.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:31 smithi115 ceph-mon[72476]: from='client.? 172.21.15.122:0/3245109415' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e62c1fb3-5dd1-4d0d-997c-d20633d19361"}]: dispatch 2024-03-20T08:55:32.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:31 smithi115 ceph-mon[72476]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e62c1fb3-5dd1-4d0d-997c-d20633d19361"}]: dispatch 2024-03-20T08:55:32.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:31 smithi115 ceph-mon[72476]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e62c1fb3-5dd1-4d0d-997c-d20633d19361"}]': finished 2024-03-20T08:55:32.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:31 smithi115 ceph-mon[72476]: osdmap e85: 11 total, 10 up, 11 in 2024-03-20T08:55:32.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:31 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-20T08:55:32.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:31 smithi122 ceph-mon[73495]: from='client.? 172.21.15.122:0/3245109415' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e62c1fb3-5dd1-4d0d-997c-d20633d19361"}]: dispatch 2024-03-20T08:55:32.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:31 smithi122 ceph-mon[73495]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e62c1fb3-5dd1-4d0d-997c-d20633d19361"}]: dispatch 2024-03-20T08:55:32.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:31 smithi122 ceph-mon[73495]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e62c1fb3-5dd1-4d0d-997c-d20633d19361"}]': finished 2024-03-20T08:55:32.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:31 smithi122 ceph-mon[73495]: osdmap e85: 11 total, 10 up, 11 in 2024-03-20T08:55:32.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:31 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-20T08:55:32.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:31 smithi026 ceph-mon[68309]: from='client.? 172.21.15.122:0/3245109415' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e62c1fb3-5dd1-4d0d-997c-d20633d19361"}]: dispatch 2024-03-20T08:55:32.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:31 smithi026 ceph-mon[68309]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e62c1fb3-5dd1-4d0d-997c-d20633d19361"}]: dispatch 2024-03-20T08:55:32.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:31 smithi026 ceph-mon[68309]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e62c1fb3-5dd1-4d0d-997c-d20633d19361"}]': finished 2024-03-20T08:55:32.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:31 smithi026 ceph-mon[68309]: osdmap e85: 11 total, 10 up, 11 in 2024-03-20T08:55:32.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:31 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-20T08:55:33.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:32 smithi115 ceph-mon[72476]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:33.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:32 smithi115 ceph-mon[72476]: from='client.? 172.21.15.122:0/3747287484' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:55:33.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:32 smithi122 ceph-mon[73495]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:33.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:32 smithi122 ceph-mon[73495]: from='client.? 172.21.15.122:0/3747287484' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:55:33.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:32 smithi026 ceph-mon[68309]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:33.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:32 smithi026 ceph-mon[68309]: from='client.? 172.21.15.122:0/3747287484' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:55:34.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:34 smithi122 ceph-mon[73495]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:35.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:34 smithi115 ceph-mon[72476]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:35.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:34 smithi026 ceph-mon[68309]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:36.989 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:36 smithi122 ceph-mon[73495]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:37.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:36 smithi115 ceph-mon[72476]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:37.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:36 smithi026 ceph-mon[68309]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:37.892 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:37 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:55:37.893 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:37 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:55:37.893 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:37 smithi122 ceph-mon[73495]: osdmap e86: 11 total, 10 up, 11 in 2024-03-20T08:55:37.893 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:37 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-20T08:55:38.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:37 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:55:38.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:37 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:55:38.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:37 smithi115 ceph-mon[72476]: osdmap e86: 11 total, 10 up, 11 in 2024-03-20T08:55:38.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:37 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-20T08:55:38.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:37 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:55:38.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:37 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:55:38.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:37 smithi026 ceph-mon[68309]: osdmap e86: 11 total, 10 up, 11 in 2024-03-20T08:55:38.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:37 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-20T08:55:39.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:38 smithi115 ceph-mon[72476]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:39.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:38 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-20T08:55:39.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:38 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:55:39.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:38 smithi115 ceph-mon[72476]: Deploying daemon osd.10 on smithi122 2024-03-20T08:55:39.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:38 smithi122 ceph-mon[73495]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:39.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:38 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-20T08:55:39.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:38 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:55:39.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:38 smithi122 ceph-mon[73495]: Deploying daemon osd.10 on smithi122 2024-03-20T08:55:39.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:38 smithi026 ceph-mon[68309]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:39.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:38 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-20T08:55:39.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:38 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:55:39.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:38 smithi026 ceph-mon[68309]: Deploying daemon osd.10 on smithi122 2024-03-20T08:55:41.133 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:40 smithi122 ceph-mon[73495]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:41.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:40 smithi115 ceph-mon[72476]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:41.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:40 smithi026 ceph-mon[68309]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:42.927 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:42 smithi122 ceph-mon[73495]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:42.928 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {auth=21,config=2,health=12,kv=57,logm=8} crc {auth=3138925955,config=1548763834,health=3897446751,kv=3363716062,logm=3288615554}) 2024-03-20T08:55:42.928 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=678707387}) 2024-03-20T08:55:42.928 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=206640121}) 2024-03-20T08:55:42.928 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2422205613}) 2024-03-20T08:55:42.928 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2012962027}) 2024-03-20T08:55:42.929 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2727022502}) 2024-03-20T08:55:42.929 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4182644244}) 2024-03-20T08:55:42.929 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3969287512}) 2024-03-20T08:55:42.929 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3220888589}) 2024-03-20T08:55:42.929 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3806549655}) 2024-03-20T08:55:42.929 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3795639727}) 2024-03-20T08:55:42.930 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1817548020}) 2024-03-20T08:55:42.930 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=59357095}) 2024-03-20T08:55:42.930 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2294438299}) 2024-03-20T08:55:42.930 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3529971150}) 2024-03-20T08:55:42.930 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=91,mdsmap=3,mgr=6} crc {logm=1942842062,mdsmap=3984425268,mgr=3909676273}) 2024-03-20T08:55:42.930 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {mgr=12,mgr_command_descs=1,mgr_metadata=2,mgrstat=11,mon_config_key=74} crc {mgr=2141790333,mgr_command_descs=3831600018,mgr_metadata=2702834764,mgrstat=3528644150,mon_config_key=3794721988}) 2024-03-20T08:55:42.931 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=448865388}) 2024-03-20T08:55:42.931 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=92,monmap=5,osd_metadata=3} crc {mon_config_key=1541426591,monmap=3481182261,osd_metadata=1870820697}) 2024-03-20T08:55:42.931 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=7,osd_pg_creating=1,osdmap=92} crc {osd_metadata=3927344524,osd_pg_creating=3440251246,osdmap=1117850939}) 2024-03-20T08:55:42.931 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=15} crc {osdmap=3269042135}) 2024-03-20T08:55:43.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:42 smithi115 ceph-mon[72476]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:43.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {auth=21,config=2,health=12,kv=57,logm=8} crc {auth=3138925955,config=1548763834,health=3897446751,kv=3363716062,logm=3288615554}) 2024-03-20T08:55:43.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=678707387}) 2024-03-20T08:55:43.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=206640121}) 2024-03-20T08:55:43.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2422205613}) 2024-03-20T08:55:43.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2012962027}) 2024-03-20T08:55:43.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2727022502}) 2024-03-20T08:55:43.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4182644244}) 2024-03-20T08:55:43.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3969287512}) 2024-03-20T08:55:43.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3220888589}) 2024-03-20T08:55:43.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3806549655}) 2024-03-20T08:55:43.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3795639727}) 2024-03-20T08:55:43.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1817548020}) 2024-03-20T08:55:43.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=59357095}) 2024-03-20T08:55:43.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2294438299}) 2024-03-20T08:55:43.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3529971150}) 2024-03-20T08:55:43.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=91,mdsmap=3,mgr=6} crc {logm=1942842062,mdsmap=3984425268,mgr=3909676273}) 2024-03-20T08:55:43.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {mgr=12,mgr_command_descs=1,mgr_metadata=2,mgrstat=11,mon_config_key=74} crc {mgr=2141790333,mgr_command_descs=3831600018,mgr_metadata=2702834764,mgrstat=3528644150,mon_config_key=3794721988}) 2024-03-20T08:55:43.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=448865388}) 2024-03-20T08:55:43.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=92,monmap=5,osd_metadata=3} crc {mon_config_key=1541426591,monmap=3481182261,osd_metadata=1870820697}) 2024-03-20T08:55:43.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=7,osd_pg_creating=1,osdmap=92} crc {osd_metadata=3927344524,osd_pg_creating=3440251246,osdmap=1117850939}) 2024-03-20T08:55:43.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=15} crc {osdmap=3269042135}) 2024-03-20T08:55:43.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:42 smithi026 ceph-mon[68309]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:43.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {auth=21,config=2,health=12,kv=57,logm=8} crc {auth=3138925955,config=1548763834,health=3897446751,kv=3363716062,logm=3288615554}) 2024-03-20T08:55:43.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=678707387}) 2024-03-20T08:55:43.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=206640121}) 2024-03-20T08:55:43.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2422205613}) 2024-03-20T08:55:43.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2012962027}) 2024-03-20T08:55:43.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2727022502}) 2024-03-20T08:55:43.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4182644244}) 2024-03-20T08:55:43.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3969287512}) 2024-03-20T08:55:43.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3220888589}) 2024-03-20T08:55:43.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3806549655}) 2024-03-20T08:55:43.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3795639727}) 2024-03-20T08:55:43.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1817548020}) 2024-03-20T08:55:43.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=59357095}) 2024-03-20T08:55:43.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2294438299}) 2024-03-20T08:55:43.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3529971150}) 2024-03-20T08:55:43.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=91,mdsmap=3,mgr=6} crc {logm=1942842062,mdsmap=3984425268,mgr=3909676273}) 2024-03-20T08:55:43.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {mgr=12,mgr_command_descs=1,mgr_metadata=2,mgrstat=11,mon_config_key=74} crc {mgr=2141790333,mgr_command_descs=3831600018,mgr_metadata=2702834764,mgrstat=3528644150,mon_config_key=3794721988}) 2024-03-20T08:55:43.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=448865388}) 2024-03-20T08:55:43.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=92,monmap=5,osd_metadata=3} crc {mon_config_key=1541426591,monmap=3481182261,osd_metadata=1870820697}) 2024-03-20T08:55:43.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=7,osd_pg_creating=1,osdmap=92} crc {osd_metadata=3927344524,osd_pg_creating=3440251246,osdmap=1117850939}) 2024-03-20T08:55:43.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=15} crc {osdmap=3269042135}) 2024-03-20T08:55:45.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:45 smithi115 ceph-mon[72476]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:45.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:45 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:45.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:45 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:55:45.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:45 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:45.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:45 smithi122 ceph-mon[73495]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:45.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:45 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:45.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:45 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:55:45.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:45 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:45.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:45 smithi026 ceph-mon[68309]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:45.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:45 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:45.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:45 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:55:45.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:45 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:46.316 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:46 smithi122 ceph-mon[73495]: 1.0 scrub starts 2024-03-20T08:55:46.316 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:46 smithi122 ceph-mon[73495]: 1.0 scrub ok 2024-03-20T08:55:46.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:46 smithi115 ceph-mon[72476]: 1.0 scrub starts 2024-03-20T08:55:46.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:46 smithi115 ceph-mon[72476]: 1.0 scrub ok 2024-03-20T08:55:46.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:46 smithi026 ceph-mon[68309]: 1.0 scrub starts 2024-03-20T08:55:46.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:46 smithi026 ceph-mon[68309]: 1.0 scrub ok 2024-03-20T08:55:47.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:47 smithi115 ceph-mon[72476]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:47.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:47 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:47.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:47 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:47.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:47 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:55:47.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:47 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:55:47.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:47 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:47.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:47 smithi122 ceph-mon[73495]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:47 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:47 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:47 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:55:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:47 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:55:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:47 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:47.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:47 smithi026 ceph-mon[68309]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:47.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:47 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:47.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:47 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:47.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:47 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:55:47.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:47 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:55:47.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:47 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:48.099 INFO:teuthology.orchestra.run.smithi122.stdout:Created osd(s) 10 on host 'smithi122' 2024-03-20T08:55:48.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:48 smithi115 ceph-mon[72476]: from='osd.10 [v2:172.21.15.122:6816/409394340,v1:172.21.15.122:6817/409394340]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-20T08:55:48.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:48 smithi115 ceph-mon[72476]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-20T08:55:48.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:48 smithi122 ceph-mon[73495]: from='osd.10 [v2:172.21.15.122:6816/409394340,v1:172.21.15.122:6817/409394340]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-20T08:55:48.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:48 smithi122 ceph-mon[73495]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-20T08:55:48.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:48 smithi026 ceph-mon[68309]: from='osd.10 [v2:172.21.15.122:6816/409394340,v1:172.21.15.122:6817/409394340]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-20T08:55:48.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:48 smithi026 ceph-mon[68309]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-20T08:55:49.037 DEBUG:teuthology.orchestra.run.smithi122:osd.10> sudo journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.10.service 2024-03-20T08:55:49.039 INFO:tasks.cephadm:Deploying osd.11 on smithi122 with /dev/vg_nvme/lv_1... 2024-03-20T08:55:49.039 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-20T08:55:49.070 INFO:journalctl@ceph.osd.10.smithi122.stdout:-- Logs begin at Wed 2024-03-20 08:37:02 UTC. -- 2024-03-20T08:55:49.394 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:49 smithi122 ceph-mon[73495]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:49.395 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:49 smithi122 ceph-mon[73495]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-20T08:55:49.395 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:49 smithi122 ceph-mon[73495]: osdmap e87: 11 total, 10 up, 11 in 2024-03-20T08:55:49.395 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:49 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-20T08:55:49.395 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:49 smithi122 ceph-mon[73495]: from='osd.10 [v2:172.21.15.122:6816/409394340,v1:172.21.15.122:6817/409394340]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-20T08:55:49.395 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:49 smithi122 ceph-mon[73495]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-20T08:55:49.395 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:49 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:49.395 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:49 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:49.395 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:49 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:55:49.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:49 smithi115 ceph-mon[72476]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:49.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:49 smithi115 ceph-mon[72476]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-20T08:55:49.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:49 smithi115 ceph-mon[72476]: osdmap e87: 11 total, 10 up, 11 in 2024-03-20T08:55:49.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:49 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-20T08:55:49.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:49 smithi115 ceph-mon[72476]: from='osd.10 [v2:172.21.15.122:6816/409394340,v1:172.21.15.122:6817/409394340]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-20T08:55:49.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:49 smithi115 ceph-mon[72476]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-20T08:55:49.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:49 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:49.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:49 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:49.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:49 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:55:49.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:49 smithi026 ceph-mon[68309]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:49.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:49 smithi026 ceph-mon[68309]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-20T08:55:49.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:49 smithi026 ceph-mon[68309]: osdmap e87: 11 total, 10 up, 11 in 2024-03-20T08:55:49.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:49 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-20T08:55:49.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:49 smithi026 ceph-mon[68309]: from='osd.10 [v2:172.21.15.122:6816/409394340,v1:172.21.15.122:6817/409394340]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-20T08:55:49.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:49 smithi026 ceph-mon[68309]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-20T08:55:49.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:49 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:49.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:49 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:49.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:49 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:55:49.953 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T08:55:50.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:50 smithi115 ceph-mon[72476]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-03-20T08:55:50.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:50 smithi115 ceph-mon[72476]: osdmap e88: 11 total, 10 up, 11 in 2024-03-20T08:55:50.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:50 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-20T08:55:50.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:50 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-20T08:55:50.425 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:50 smithi122 ceph-mon[73495]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-03-20T08:55:50.425 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:50 smithi122 ceph-mon[73495]: osdmap e88: 11 total, 10 up, 11 in 2024-03-20T08:55:50.425 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:50 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-20T08:55:50.425 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:50 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-20T08:55:50.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:50 smithi026 ceph-mon[68309]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-03-20T08:55:50.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:50 smithi026 ceph-mon[68309]: osdmap e88: 11 total, 10 up, 11 in 2024-03-20T08:55:50.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:50 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-20T08:55:50.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:50 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-20T08:55:51.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:51 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:55:51.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:51 smithi122 ceph-mon[73495]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:51.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:51 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-20T08:55:51.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:51 smithi122 ceph-mon[73495]: osd.10 [v2:172.21.15.122:6816/409394340,v1:172.21.15.122:6817/409394340] boot 2024-03-20T08:55:51.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:51 smithi122 ceph-mon[73495]: osdmap e89: 11 total, 11 up, 11 in 2024-03-20T08:55:51.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:51 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-20T08:55:51.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:51 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:55:51.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:51 smithi115 ceph-mon[72476]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:51.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:51 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-20T08:55:51.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:51 smithi115 ceph-mon[72476]: osd.10 [v2:172.21.15.122:6816/409394340,v1:172.21.15.122:6817/409394340] boot 2024-03-20T08:55:51.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:51 smithi115 ceph-mon[72476]: osdmap e89: 11 total, 11 up, 11 in 2024-03-20T08:55:51.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:51 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-20T08:55:51.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:51 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:55:51.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:51 smithi026 ceph-mon[68309]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2024-03-20T08:55:51.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:51 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-20T08:55:51.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:51 smithi026 ceph-mon[68309]: osd.10 [v2:172.21.15.122:6816/409394340,v1:172.21.15.122:6817/409394340] boot 2024-03-20T08:55:51.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:51 smithi026 ceph-mon[68309]: osdmap e89: 11 total, 11 up, 11 in 2024-03-20T08:55:51.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:51 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-20T08:55:52.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:52 smithi115 ceph-mon[72476]: osdmap e90: 11 total, 11 up, 11 in 2024-03-20T08:55:52.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:52 smithi122 ceph-mon[73495]: osdmap e90: 11 total, 11 up, 11 in 2024-03-20T08:55:52.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:52 smithi026 ceph-mon[68309]: osdmap e90: 11 total, 11 up, 11 in 2024-03-20T08:55:53.295 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-20T08:55:53.295 INFO:teuthology.orchestra.run.smithi122.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-20T08:55:53.296 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10+0 records in 2024-03-20T08:55:53.296 INFO:teuthology.orchestra.run.smithi122.stderr:10+0 records out 2024-03-20T08:55:53.296 INFO:teuthology.orchestra.run.smithi122.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0106949 s, 980 MB/s 2024-03-20T08:55:53.296 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 2024-03-20T08:55:53.296 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping successful for: 2024-03-20T08:55:53.331 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:53 smithi122 ceph-mon[73495]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:55:53.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:53 smithi115 ceph-mon[72476]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:55:53.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:53 smithi026 ceph-mon[68309]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:55:54.085 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi122:vg_nvme/lv_1 2024-03-20T08:55:54.683 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T08:55:55.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:55 smithi122 ceph-mon[73495]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:55:55.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:55 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:55.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:55 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:55.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:55 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:55:55.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:55 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:55:55.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:55 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:55:55.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:55 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:55.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:55 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:55:55.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:55 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:55:55.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:55 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:55.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:55 smithi115 ceph-mon[72476]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:55:55.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:55 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:55.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:55 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:55.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:55 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:55:55.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:55 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:55:55.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:55 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:55:55.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:55 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:55.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:55 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:55:55.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:55 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:55:55.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:55 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:55.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:55 smithi026 ceph-mon[68309]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:55:55.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:55 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:55.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:55 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:55.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:55 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:55:55.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:55 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:55:55.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:55 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:55:55.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:55 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:55.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:55 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:55:55.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:55 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:55:55.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:55 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:55:56.319 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:56 smithi122 ceph-mon[73495]: Detected new or changed devices on smithi122 2024-03-20T08:55:56.319 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:56 smithi122 ceph-mon[73495]: Adjusting osd_memory_target on smithi122 to 7072M 2024-03-20T08:55:56.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:56 smithi115 ceph-mon[72476]: Detected new or changed devices on smithi122 2024-03-20T08:55:56.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:56 smithi115 ceph-mon[72476]: Adjusting osd_memory_target on smithi122 to 7072M 2024-03-20T08:55:56.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:56 smithi026 ceph-mon[68309]: Detected new or changed devices on smithi122 2024-03-20T08:55:56.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:56 smithi026 ceph-mon[68309]: Adjusting osd_memory_target on smithi122 to 7072M 2024-03-20T08:55:57.322 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:57 smithi122 ceph-mon[73495]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:55:57.322 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:57 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:55:57.322 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:57 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:55:57.322 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:57 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:55:57.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:57 smithi115 ceph-mon[72476]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:55:57.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:57 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:55:57.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:57 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:55:57.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:57 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:55:57.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:57 smithi026 ceph-mon[68309]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:55:57.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:57 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-20T08:55:57.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:57 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-20T08:55:57.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:57 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:55:58.280 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:58 smithi122 ceph-mon[73495]: from='client.24334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:55:58.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:58 smithi115 ceph-mon[72476]: from='client.24334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:55:58.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:58 smithi026 ceph-mon[68309]: from='client.24334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T08:55:59.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:55:59 smithi115 ceph-mon[72476]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:55:59.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:55:59 smithi122 ceph-mon[73495]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:55:59.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:55:59 smithi026 ceph-mon[68309]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:56:00.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:00 smithi115 ceph-mon[72476]: from='client.? 172.21.15.122:0/88498492' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b83994d5-e5ef-48b0-9758-c26843cec3b6"}]: dispatch 2024-03-20T08:56:00.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:00 smithi115 ceph-mon[72476]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b83994d5-e5ef-48b0-9758-c26843cec3b6"}]: dispatch 2024-03-20T08:56:00.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:00 smithi115 ceph-mon[72476]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b83994d5-e5ef-48b0-9758-c26843cec3b6"}]': finished 2024-03-20T08:56:00.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:00 smithi115 ceph-mon[72476]: osdmap e91: 12 total, 11 up, 12 in 2024-03-20T08:56:00.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:00 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-20T08:56:00.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:00 smithi122 ceph-mon[73495]: from='client.? 172.21.15.122:0/88498492' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b83994d5-e5ef-48b0-9758-c26843cec3b6"}]: dispatch 2024-03-20T08:56:00.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:00 smithi122 ceph-mon[73495]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b83994d5-e5ef-48b0-9758-c26843cec3b6"}]: dispatch 2024-03-20T08:56:00.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:00 smithi122 ceph-mon[73495]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b83994d5-e5ef-48b0-9758-c26843cec3b6"}]': finished 2024-03-20T08:56:00.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:00 smithi122 ceph-mon[73495]: osdmap e91: 12 total, 11 up, 12 in 2024-03-20T08:56:00.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:00 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-20T08:56:00.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:00 smithi026 ceph-mon[68309]: from='client.? 172.21.15.122:0/88498492' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b83994d5-e5ef-48b0-9758-c26843cec3b6"}]: dispatch 2024-03-20T08:56:00.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:00 smithi026 ceph-mon[68309]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b83994d5-e5ef-48b0-9758-c26843cec3b6"}]: dispatch 2024-03-20T08:56:00.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:00 smithi026 ceph-mon[68309]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b83994d5-e5ef-48b0-9758-c26843cec3b6"}]': finished 2024-03-20T08:56:00.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:00 smithi026 ceph-mon[68309]: osdmap e91: 12 total, 11 up, 12 in 2024-03-20T08:56:00.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:00 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-20T08:56:01.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:01 smithi115 ceph-mon[72476]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:56:01.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:01 smithi115 ceph-mon[72476]: from='client.? 172.21.15.122:0/1779675786' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:56:01.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:01 smithi122 ceph-mon[73495]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:56:01.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:01 smithi122 ceph-mon[73495]: from='client.? 172.21.15.122:0/1779675786' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:56:01.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:01 smithi026 ceph-mon[68309]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:56:01.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:01 smithi026 ceph-mon[68309]: from='client.? 172.21.15.122:0/1779675786' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-20T08:56:02.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:02 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:56:02.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:02 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:56:02.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:02 smithi115 ceph-mon[72476]: osdmap e92: 12 total, 11 up, 12 in 2024-03-20T08:56:02.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:02 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-20T08:56:02.427 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:02 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:56:02.427 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:02 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:56:02.427 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:02 smithi122 ceph-mon[73495]: osdmap e92: 12 total, 11 up, 12 in 2024-03-20T08:56:02.427 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:02 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-20T08:56:02.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:02 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:56:02.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:02 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:56:02.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:02 smithi026 ceph-mon[68309]: osdmap e92: 12 total, 11 up, 12 in 2024-03-20T08:56:02.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:02 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-20T08:56:03.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:03 smithi115 ceph-mon[72476]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:56:03.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:03 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:56:03.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:03 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:56:03.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:03 smithi122 ceph-mon[73495]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:56:03.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:03 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:56:03.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:03 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:56:03.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:03 smithi026 ceph-mon[68309]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:56:03.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:03 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:56:03.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:03 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:56:04.376 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:04 smithi122 ceph-mon[73495]: osdmap e93: 12 total, 11 up, 12 in 2024-03-20T08:56:04.376 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:04 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-20T08:56:04.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:04 smithi115 ceph-mon[72476]: osdmap e93: 12 total, 11 up, 12 in 2024-03-20T08:56:04.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:04 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-20T08:56:04.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:04 smithi026 ceph-mon[68309]: osdmap e93: 12 total, 11 up, 12 in 2024-03-20T08:56:04.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:04 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-20T08:56:05.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:05 smithi115 ceph-mon[72476]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:56:05.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:05 smithi122 ceph-mon[73495]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:56:05.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:05 smithi026 ceph-mon[68309]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:56:07.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:07 smithi115 ceph-mon[72476]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:56:07.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:07 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-20T08:56:07.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:07 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:56:07.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:07 smithi115 ceph-mon[72476]: Deploying daemon osd.11 on smithi122 2024-03-20T08:56:07.409 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:07 smithi122 ceph-mon[73495]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:56:07.409 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:07 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-20T08:56:07.409 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:07 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:56:07.409 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:07 smithi122 ceph-mon[73495]: Deploying daemon osd.11 on smithi122 2024-03-20T08:56:07.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:07 smithi026 ceph-mon[68309]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:56:07.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:07 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-20T08:56:07.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:07 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:56:07.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:07 smithi026 ceph-mon[68309]: Deploying daemon osd.11 on smithi122 2024-03-20T08:56:09.148 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:09 smithi122 ceph-mon[73495]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:56:09.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:09 smithi115 ceph-mon[72476]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:56:09.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:09 smithi026 ceph-mon[68309]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:56:11.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:11 smithi122 ceph-mon[73495]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:56:11.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:11 smithi115 ceph-mon[72476]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:56:11.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:11 smithi026 ceph-mon[68309]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:56:13.006 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:12 smithi122 ceph-mon[73495]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:56:13.006 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:12 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:56:13.006 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:12 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:56:13.007 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:12 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:56:13.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:12 smithi115 ceph-mon[72476]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:56:13.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:12 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:56:13.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:12 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:56:13.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:12 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:56:13.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:12 smithi026 ceph-mon[68309]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:56:13.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:12 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:56:13.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:12 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:56:13.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:12 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:56:15.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:14 smithi115 ceph-mon[72476]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:56:15.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:14 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:56:15.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:14 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:56:15.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:14 smithi122 ceph-mon[73495]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:56:15.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:14 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:56:15.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:14 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:56:15.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:14 smithi026 ceph-mon[68309]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:56:15.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:14 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:56:15.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:14 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:56:15.750 INFO:teuthology.orchestra.run.smithi122.stdout:Created osd(s) 11 on host 'smithi122' 2024-03-20T08:56:16.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:15 smithi115 ceph-mon[72476]: from='osd.11 [v2:172.21.15.122:6824/2704069354,v1:172.21.15.122:6825/2704069354]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-20T08:56:16.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:15 smithi115 ceph-mon[72476]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-20T08:56:16.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:15 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:56:16.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:15 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:56:16.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:15 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:56:16.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:15 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:56:16.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:15 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:56:16.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:15 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:56:16.165 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:15 smithi122 ceph-mon[73495]: from='osd.11 [v2:172.21.15.122:6824/2704069354,v1:172.21.15.122:6825/2704069354]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-20T08:56:16.165 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:15 smithi122 ceph-mon[73495]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-20T08:56:16.165 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:15 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:56:16.165 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:15 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:56:16.165 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:15 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:56:16.165 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:15 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:56:16.165 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:15 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:56:16.166 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:15 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:56:16.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:15 smithi026 ceph-mon[68309]: from='osd.11 [v2:172.21.15.122:6824/2704069354,v1:172.21.15.122:6825/2704069354]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-20T08:56:16.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:15 smithi026 ceph-mon[68309]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-20T08:56:16.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:15 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:56:16.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:15 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:56:16.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:15 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:56:16.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:15 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:56:16.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:15 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:56:16.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:15 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:56:16.546 DEBUG:teuthology.orchestra.run.smithi122:osd.11> sudo journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.11.service 2024-03-20T08:56:16.548 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-20T08:56:16.549 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-20T08:56:16.578 INFO:journalctl@ceph.osd.11.smithi122.stdout:-- Logs begin at Wed 2024-03-20 08:37:02 UTC. -- 2024-03-20T08:56:16.835 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:56:17.112 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:16 smithi122 ceph-mon[73495]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:56:17.112 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:16 smithi122 ceph-mon[73495]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-20T08:56:17.112 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:16 smithi122 ceph-mon[73495]: osdmap e94: 12 total, 11 up, 12 in 2024-03-20T08:56:17.112 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:16 smithi122 ceph-mon[73495]: from='osd.11 [v2:172.21.15.122:6824/2704069354,v1:172.21.15.122:6825/2704069354]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-20T08:56:17.112 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:16 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-20T08:56:17.112 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:16 smithi122 ceph-mon[73495]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-20T08:56:17.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:16 smithi115 ceph-mon[72476]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:56:17.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:16 smithi115 ceph-mon[72476]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-20T08:56:17.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:16 smithi115 ceph-mon[72476]: osdmap e94: 12 total, 11 up, 12 in 2024-03-20T08:56:17.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:16 smithi115 ceph-mon[72476]: from='osd.11 [v2:172.21.15.122:6824/2704069354,v1:172.21.15.122:6825/2704069354]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-20T08:56:17.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:16 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-20T08:56:17.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:16 smithi115 ceph-mon[72476]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-20T08:56:17.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:16 smithi026 ceph-mon[68309]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:56:17.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:16 smithi026 ceph-mon[68309]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-20T08:56:17.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:16 smithi026 ceph-mon[68309]: osdmap e94: 12 total, 11 up, 12 in 2024-03-20T08:56:17.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:16 smithi026 ceph-mon[68309]: from='osd.11 [v2:172.21.15.122:6824/2704069354,v1:172.21.15.122:6825/2704069354]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-20T08:56:17.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:16 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-20T08:56:17.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:16 smithi026 ceph-mon[68309]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-20T08:56:17.446 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 08:56:17 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T08:56:17.285+0000 7f2b28813700 -1 osd.11 0 waiting for initial osdmap 2024-03-20T08:56:17.446 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 08:56:17 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T08:56:17.286+0000 7f2b1b79d700 -1 osd.11 0 failed to load OSD map for epoch 34, got 0 bytes 2024-03-20T08:56:17.446 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 08:56:17 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T08:56:17.293+0000 7f2b1fdfd700 -1 osd.11 95 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-20T08:56:18.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:17 smithi115 ceph-mon[72476]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-03-20T08:56:18.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:17 smithi115 ceph-mon[72476]: osdmap e95: 12 total, 11 up, 12 in 2024-03-20T08:56:18.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:17 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-20T08:56:18.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:17 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-20T08:56:18.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:17 smithi115 ceph-mon[72476]: from='osd.11 ' entity='osd.11' 2024-03-20T08:56:18.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:17 smithi122 ceph-mon[73495]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-03-20T08:56:18.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:17 smithi122 ceph-mon[73495]: osdmap e95: 12 total, 11 up, 12 in 2024-03-20T08:56:18.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:17 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-20T08:56:18.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:17 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-20T08:56:18.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:17 smithi122 ceph-mon[73495]: from='osd.11 ' entity='osd.11' 2024-03-20T08:56:18.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:17 smithi026 ceph-mon[68309]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-03-20T08:56:18.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:17 smithi026 ceph-mon[68309]: osdmap e95: 12 total, 11 up, 12 in 2024-03-20T08:56:18.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:17 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-20T08:56:18.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:17 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-20T08:56:18.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:17 smithi026 ceph-mon[68309]: from='osd.11 ' entity='osd.11' 2024-03-20T08:56:18.945 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:56:19.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:18 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:56:19.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:18 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:56:19.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:18 smithi115 ceph-mon[72476]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:56:19.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:18 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-20T08:56:19.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:18 smithi115 ceph-mon[72476]: osd.11 [v2:172.21.15.122:6824/2704069354,v1:172.21.15.122:6825/2704069354] boot 2024-03-20T08:56:19.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:18 smithi115 ceph-mon[72476]: osdmap e96: 12 total, 12 up, 12 in 2024-03-20T08:56:19.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:18 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-20T08:56:19.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:18 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:56:19.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:18 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:56:19.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:18 smithi122 ceph-mon[73495]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:56:19.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:18 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-20T08:56:19.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:18 smithi122 ceph-mon[73495]: osd.11 [v2:172.21.15.122:6824/2704069354,v1:172.21.15.122:6825/2704069354] boot 2024-03-20T08:56:19.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:18 smithi122 ceph-mon[73495]: osdmap e96: 12 total, 12 up, 12 in 2024-03-20T08:56:19.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:18 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-20T08:56:19.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:18 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:56:19.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:18 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:56:19.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:18 smithi026 ceph-mon[68309]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-03-20T08:56:19.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:18 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-20T08:56:19.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:18 smithi026 ceph-mon[68309]: osd.11 [v2:172.21.15.122:6824/2704069354,v1:172.21.15.122:6825/2704069354] boot 2024-03-20T08:56:19.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:18 smithi026 ceph-mon[68309]: osdmap e96: 12 total, 12 up, 12 in 2024-03-20T08:56:19.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:18 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-20T08:56:19.750 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":97,"num_osds":12,"num_up_osds":12,"osd_up_since":1710924977,"num_in_osds":12,"osd_in_since":1710924959,"num_remapped_pgs":1} 2024-03-20T08:56:19.751 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-20T08:56:19.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:19 smithi026 ceph-mon[68309]: osdmap e97: 12 total, 12 up, 12 in 2024-03-20T08:56:19.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:19 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2125508683' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-20T08:56:19.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:19 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:56:19.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:19 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:56:19.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:19 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:56:19.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:19 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:56:19.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:19 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:56:19.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:19 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:56:19.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:19 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:56:19.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:19 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:56:19.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:19 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:56:19.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:19 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:56:20.033 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:56:20.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:19 smithi115 ceph-mon[72476]: osdmap e97: 12 total, 12 up, 12 in 2024-03-20T08:56:20.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:19 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2125508683' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-20T08:56:20.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:19 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:56:20.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:19 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:56:20.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:19 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:56:20.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:19 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:56:20.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:19 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:56:20.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:19 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:56:20.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:19 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:56:20.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:19 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:56:20.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:19 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:56:20.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:19 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:56:20.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:19 smithi122 ceph-mon[73495]: osdmap e97: 12 total, 12 up, 12 in 2024-03-20T08:56:20.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:19 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2125508683' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-20T08:56:20.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:19 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:56:20.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:19 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:56:20.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:19 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:56:20.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:19 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:56:20.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:19 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:56:20.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:19 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-20T08:56:20.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:19 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:56:20.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:19 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:56:20.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:19 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:56:20.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:19 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:56:21.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:20 smithi115 ceph-mon[72476]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:21.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:20 smithi115 ceph-mon[72476]: Detected new or changed devices on smithi122 2024-03-20T08:56:21.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:20 smithi115 ceph-mon[72476]: Adjusting osd_memory_target on smithi122 to 5304M 2024-03-20T08:56:21.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:20 smithi115 ceph-mon[72476]: osdmap e98: 12 total, 12 up, 12 in 2024-03-20T08:56:21.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:20 smithi122 ceph-mon[73495]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:21.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:20 smithi122 ceph-mon[73495]: Detected new or changed devices on smithi122 2024-03-20T08:56:21.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:20 smithi122 ceph-mon[73495]: Adjusting osd_memory_target on smithi122 to 5304M 2024-03-20T08:56:21.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:20 smithi122 ceph-mon[73495]: osdmap e98: 12 total, 12 up, 12 in 2024-03-20T08:56:21.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:20 smithi026 ceph-mon[68309]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:21.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:20 smithi026 ceph-mon[68309]: Detected new or changed devices on smithi122 2024-03-20T08:56:21.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:20 smithi026 ceph-mon[68309]: Adjusting osd_memory_target on smithi122 to 5304M 2024-03-20T08:56:21.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:20 smithi026 ceph-mon[68309]: osdmap e98: 12 total, 12 up, 12 in 2024-03-20T08:56:22.076 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:56:22.076 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":100,"fsid":"852fbb68-e696-11ee-95c9-87774f69a715","created":"2024-03-20T08:48:35.281088+0000","modified":"2024-03-20T08:56:21.896013+0000","last_up_change":"2024-03-20T08:56:17.873738+0000","last_in_change":"2024-03-20T08:55:59.486577+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-20T08:52:17.442324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad3fd9c0-316f-4fe5-a63b-d480087c1729","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":2139805293},{"type":"v1","addr":"172.21.15.26:6803","nonce":2139805293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":2139805293},{"type":"v1","addr":"172.21.15.26:6805","nonce":2139805293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":2139805293},{"type":"v1","addr":"172.21.15.26:6809","nonce":2139805293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":2139805293},{"type":"v1","addr":"172.21.15.26:6807","nonce":2139805293}]},"public_addr":"172.21.15.26:6803/2139805293","cluster_addr":"172.21.15.26:6805/2139805293","heartbeat_back_addr":"172.21.15.26:6809/2139805293","heartbeat_front_addr":"172.21.15.26:6807/2139805293","state":["exists","up"]},{"osd":1,"uuid":"e85b94f5-2954-47a1-9c20-096f2606eb04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":3853355134},{"type":"v1","addr":"172.21.15.26:6811","nonce":3853355134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":3853355134},{"type":"v1","addr":"172.21.15.26:6813","nonce":3853355134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":3853355134},{"type":"v1","addr":"172.21.15.26:6817","nonce":3853355134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":3853355134},{"type":"v1","addr":"172.21.15.26:6815","nonce":3853355134}]},"public_addr":"172.21.15.26:6811/3853355134","cluster_addr":"172.21.15.26:6813/3853355134","heartbeat_back_addr":"172.21.15.26:6817/3853355134","heartbeat_front_addr":"172.21.15.26:6815/3853355134","state":["exists","up"]},{"osd":2,"uuid":"dd2b10f0-741c-47a4-b224-ac6bfe2c14ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":3993357967},{"type":"v1","addr":"172.21.15.26:6819","nonce":3993357967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":3993357967},{"type":"v1","addr":"172.21.15.26:6821","nonce":3993357967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":3993357967},{"type":"v1","addr":"172.21.15.26:6825","nonce":3993357967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":3993357967},{"type":"v1","addr":"172.21.15.26:6823","nonce":3993357967}]},"public_addr":"172.21.15.26:6819/3993357967","cluster_addr":"172.21.15.26:6821/3993357967","heartbeat_back_addr":"172.21.15.26:6825/3993357967","heartbeat_front_addr":"172.21.15.26:6823/3993357967","state":["exists","up"]},{"osd":3,"uuid":"9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":4110225509},{"type":"v1","addr":"172.21.15.26:6827","nonce":4110225509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":4110225509},{"type":"v1","addr":"172.21.15.26:6829","nonce":4110225509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":4110225509},{"type":"v1","addr":"172.21.15.26:6833","nonce":4110225509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":4110225509},{"type":"v1","addr":"172.21.15.26:6831","nonce":4110225509}]},"public_addr":"172.21.15.26:6827/4110225509","cluster_addr":"172.21.15.26:6829/4110225509","heartbeat_back_addr":"172.21.15.26:6833/4110225509","heartbeat_front_addr":"172.21.15.26:6831/4110225509","state":["exists","up"]},{"osd":4,"uuid":"f14e34d5-9829-42a3-9b25-31a0976c5ef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":477923058},{"type":"v1","addr":"172.21.15.115:6801","nonce":477923058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":477923058},{"type":"v1","addr":"172.21.15.115:6803","nonce":477923058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":477923058},{"type":"v1","addr":"172.21.15.115:6807","nonce":477923058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":477923058},{"type":"v1","addr":"172.21.15.115:6805","nonce":477923058}]},"public_addr":"172.21.15.115:6801/477923058","cluster_addr":"172.21.15.115:6803/477923058","heartbeat_back_addr":"172.21.15.115:6807/477923058","heartbeat_front_addr":"172.21.15.115:6805/477923058","state":["exists","up"]},{"osd":5,"uuid":"cb431b6a-540d-4318-ae35-d379e1df0185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1339268003},{"type":"v1","addr":"172.21.15.115:6809","nonce":1339268003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":1339268003},{"type":"v1","addr":"172.21.15.115:6811","nonce":1339268003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":1339268003},{"type":"v1","addr":"172.21.15.115:6815","nonce":1339268003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":1339268003},{"type":"v1","addr":"172.21.15.115:6813","nonce":1339268003}]},"public_addr":"172.21.15.115:6809/1339268003","cluster_addr":"172.21.15.115:6811/1339268003","heartbeat_back_addr":"172.21.15.115:6815/1339268003","heartbeat_front_addr":"172.21.15.115:6813/1339268003","state":["exists","up"]},{"osd":6,"uuid":"753ed830-e764-4ff3-9d9b-87e7fa748b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3278301843},{"type":"v1","addr":"172.21.15.115:6817","nonce":3278301843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":3278301843},{"type":"v1","addr":"172.21.15.115:6819","nonce":3278301843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":3278301843},{"type":"v1","addr":"172.21.15.115:6823","nonce":3278301843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":3278301843},{"type":"v1","addr":"172.21.15.115:6821","nonce":3278301843}]},"public_addr":"172.21.15.115:6817/3278301843","cluster_addr":"172.21.15.115:6819/3278301843","heartbeat_back_addr":"172.21.15.115:6823/3278301843","heartbeat_front_addr":"172.21.15.115:6821/3278301843","state":["exists","up"]},{"osd":7,"uuid":"55bcbd37-7a6f-4ab1-b5a9-91a951444d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":3152424001},{"type":"v1","addr":"172.21.15.115:6825","nonce":3152424001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3152424001},{"type":"v1","addr":"172.21.15.115:6827","nonce":3152424001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3152424001},{"type":"v1","addr":"172.21.15.115:6831","nonce":3152424001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3152424001},{"type":"v1","addr":"172.21.15.115:6829","nonce":3152424001}]},"public_addr":"172.21.15.115:6825/3152424001","cluster_addr":"172.21.15.115:6827/3152424001","heartbeat_back_addr":"172.21.15.115:6831/3152424001","heartbeat_front_addr":"172.21.15.115:6829/3152424001","state":["exists","up"]},{"osd":8,"uuid":"bc86248e-6bd0-4369-b8a2-e593bc2a610d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1309426740},{"type":"v1","addr":"172.21.15.122:6801","nonce":1309426740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1309426740},{"type":"v1","addr":"172.21.15.122:6803","nonce":1309426740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1309426740},{"type":"v1","addr":"172.21.15.122:6807","nonce":1309426740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1309426740},{"type":"v1","addr":"172.21.15.122:6805","nonce":1309426740}]},"public_addr":"172.21.15.122:6801/1309426740","cluster_addr":"172.21.15.122:6803/1309426740","heartbeat_back_addr":"172.21.15.122:6807/1309426740","heartbeat_front_addr":"172.21.15.122:6805/1309426740","state":["exists","up"]},{"osd":9,"uuid":"9b20a4aa-032a-4c09-ac75-153c554424e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":2953210076},{"type":"v1","addr":"172.21.15.122:6809","nonce":2953210076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":2953210076},{"type":"v1","addr":"172.21.15.122:6811","nonce":2953210076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2953210076},{"type":"v1","addr":"172.21.15.122:6815","nonce":2953210076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2953210076},{"type":"v1","addr":"172.21.15.122:6813","nonce":2953210076}]},"public_addr":"172.21.15.122:6809/2953210076","cluster_addr":"172.21.15.122:6811/2953210076","heartbeat_back_addr":"172.21.15.122:6815/2953210076","heartbeat_front_addr":"172.21.15.122:6813/2953210076","state":["exists","up"]},{"osd":10,"uuid":"e62c1fb3-5dd1-4d0d-997c-d20633d19361","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":89,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":409394340},{"type":"v1","addr":"172.21.15.122:6817","nonce":409394340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":409394340},{"type":"v1","addr":"172.21.15.122:6819","nonce":409394340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":409394340},{"type":"v1","addr":"172.21.15.122:6823","nonce":409394340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":409394340},{"type":"v1","addr":"172.21.15.122:6821","nonce":409394340}]},"public_addr":"172.21.15.122:6817/409394340","cluster_addr":"172.21.15.122:6819/409394340","heartbeat_back_addr":"172.21.15.122:6823/409394340","heartbeat_front_addr":"172.21.15.122:6821/409394340","state":["exists","up"]},{"osd":11,"uuid":"b83994d5-e5ef-48b0-9758-c26843cec3b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":96,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":2704069354},{"type":"v1","addr":"172.21.15.122:6825","nonce":2704069354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":2704069354},{"type":"v1","addr":"172.21.15.122:6827","nonce":2704069354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":2704069354},{"type":"v1","addr":"172.21.15.122:6831","nonce":2704069354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":2704069354},{"type":"v1","addr":"172.21.15.122:6829","nonce":2704069354}]},"public_addr":"172.21.15.122:6825/2704069354","cluster_addr":"172.21.15.122:6827/2704069354","heartbeat_back_addr":"172.21.15.122:6831/2704069354","heartbeat_front_addr":"172.21.15.122:6829/2704069354","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:55:08.205405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:55:29.017333+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:54:51.253909+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:55:09.223194+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:55:23.835369+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:56:02.401511+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:55:27.457509+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:55:36.320737+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:56:00.648916+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:55:20.203922+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:55:48.088091+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:56:16.011630+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:0/4291404191":"2024-03-21T08:49:35.336921+0000","172.21.15.26:0/642012566":"2024-03-21T08:49:35.336921+0000","172.21.15.26:6801/1181459999":"2024-03-21T08:49:35.336921+0000","172.21.15.26:0/3984513952":"2024-03-21T08:49:04.596754+0000","172.21.15.26:6800/1654945498":"2024-03-21T08:49:04.596754+0000","172.21.15.26:6800/1181459999":"2024-03-21T08:49:35.336921+0000","172.21.15.26:0/137770981":"2024-03-21T08:49:04.596754+0000","172.21.15.26:0/1451237762":"2024-03-21T08:49:35.336921+0000","172.21.15.26:0/3938386631":"2024-03-21T08:49:04.596754+0000","172.21.15.26:6801/1654945498":"2024-03-21T08:49:04.596754+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-20T08:56:22.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:21 smithi115 ceph-mon[72476]: osdmap e99: 12 total, 12 up, 12 in 2024-03-20T08:56:22.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:21 smithi122 ceph-mon[73495]: osdmap e99: 12 total, 12 up, 12 in 2024-03-20T08:56:22.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:21 smithi026 ceph-mon[68309]: osdmap e99: 12 total, 12 up, 12 in 2024-03-20T08:56:22.852 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-20T08:52:17.442324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-20T08:56:22.852 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-20T08:56:23.128 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:56:23.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:22 smithi122 ceph-mon[73495]: pgmap v292: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-20T08:56:23.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:22 smithi122 ceph-mon[73495]: osdmap e100: 12 total, 12 up, 12 in 2024-03-20T08:56:23.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:22 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:56:23.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:22 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:56:23.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:22 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1713618507' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-20T08:56:23.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:22 smithi026 ceph-mon[68309]: pgmap v292: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-20T08:56:23.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:22 smithi026 ceph-mon[68309]: osdmap e100: 12 total, 12 up, 12 in 2024-03-20T08:56:23.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:22 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:56:23.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:22 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:56:23.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:22 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1713618507' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-20T08:56:23.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:22 smithi115 ceph-mon[72476]: pgmap v292: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-20T08:56:23.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:22 smithi115 ceph-mon[72476]: osdmap e100: 12 total, 12 up, 12 in 2024-03-20T08:56:23.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:22 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:56:23.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:22 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:56:23.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:22 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1713618507' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-20T08:56:24.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:23 smithi122 ceph-mon[73495]: osdmap e101: 12 total, 12 up, 12 in 2024-03-20T08:56:24.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:23 smithi026 ceph-mon[68309]: osdmap e101: 12 total, 12 up, 12 in 2024-03-20T08:56:24.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:23 smithi115 ceph-mon[72476]: osdmap e101: 12 total, 12 up, 12 in 2024-03-20T08:56:25.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:24 smithi122 ceph-mon[73495]: pgmap v295: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-20T08:56:25.218 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2024-03-20T08:56:25.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:24 smithi026 ceph-mon[68309]: pgmap v295: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-20T08:56:25.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:24 smithi115 ceph-mon[72476]: pgmap v295: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-20T08:56:26.012 INFO:tasks.cephadm:Setting up client nodes... 2024-03-20T08:56:26.012 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-20T08:56:26.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:25 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:56:26.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:25 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:56:26.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:25 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/21775801' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-20T08:56:26.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:25 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:56:26.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:25 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:56:26.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:25 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/21775801' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-20T08:56:26.289 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:56:26.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:25 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:56:26.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:25 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:56:26.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:25 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/21775801' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-20T08:56:27.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:26 smithi122 ceph-mon[73495]: pgmap v296: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s, 0 objects/s recovering 2024-03-20T08:56:27.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:26 smithi122 ceph-mon[73495]: osdmap e102: 12 total, 12 up, 12 in 2024-03-20T08:56:27.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:26 smithi026 ceph-mon[68309]: pgmap v296: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s, 0 objects/s recovering 2024-03-20T08:56:27.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:26 smithi026 ceph-mon[68309]: osdmap e102: 12 total, 12 up, 12 in 2024-03-20T08:56:27.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:26 smithi115 ceph-mon[72476]: pgmap v296: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s, 0 objects/s recovering 2024-03-20T08:56:27.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:26 smithi115 ceph-mon[72476]: osdmap e102: 12 total, 12 up, 12 in 2024-03-20T08:56:28.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:27 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:56:28.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:27 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:56:28.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:27 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:56:28.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:27 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:56:28.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:27 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:56:28.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:27 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:56:28.461 INFO:teuthology.orchestra.run.smithi026.stdout:[client.0] 2024-03-20T08:56:28.461 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQC8pPpltF1mGxAAV375w8pb5lvHlPW4l9eKyw== 2024-03-20T08:56:29.070 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:28 smithi026 ceph-mon[68309]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:29.071 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:28 smithi026 ceph-mon[68309]: osdmap e103: 12 total, 12 up, 12 in 2024-03-20T08:56:29.071 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:28 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1610937036' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-20T08:56:29.071 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:28 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1610937036' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-20T08:56:29.098 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-20T08:56:29.098 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-20T08:56:29.098 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-20T08:56:29.140 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-20T08:56:29.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:28 smithi122 ceph-mon[73495]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:29.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:28 smithi122 ceph-mon[73495]: osdmap e103: 12 total, 12 up, 12 in 2024-03-20T08:56:29.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:28 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1610937036' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-20T08:56:29.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:28 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1610937036' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-20T08:56:29.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:28 smithi115 ceph-mon[72476]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:29.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:28 smithi115 ceph-mon[72476]: osdmap e103: 12 total, 12 up, 12 in 2024-03-20T08:56:29.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:28 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1610937036' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-20T08:56:29.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:28 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1610937036' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-20T08:56:29.413 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:56:30.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:29 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:56:30.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:29 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:56:30.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:29 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:56:30.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:29 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:56:30.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:29 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:56:30.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:29 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:56:31.108 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:30 smithi115 ceph-mon[72476]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:31.108 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:30 smithi115 ceph-mon[72476]: osdmap e104: 12 total, 12 up, 12 in 2024-03-20T08:56:31.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:30 smithi122 ceph-mon[73495]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:31.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:30 smithi122 ceph-mon[73495]: osdmap e104: 12 total, 12 up, 12 in 2024-03-20T08:56:31.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:30 smithi026 ceph-mon[68309]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:31.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:30 smithi026 ceph-mon[68309]: osdmap e104: 12 total, 12 up, 12 in 2024-03-20T08:56:31.657 INFO:teuthology.orchestra.run.smithi115.stdout:[client.1] 2024-03-20T08:56:31.658 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQC/pPplsVv7JhAAI0ryU8B7vQkWWVnW5EcFPQ== 2024-03-20T08:56:32.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:31 smithi122 ceph-mon[73495]: from='client.? 172.21.15.115:0/1995739106' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-20T08:56:32.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:31 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-20T08:56:32.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:31 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-20T08:56:32.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:31 smithi026 ceph-mon[68309]: from='client.? 172.21.15.115:0/1995739106' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-20T08:56:32.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:31 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-20T08:56:32.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:31 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-20T08:56:32.378 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:31 smithi115 ceph-mon[72476]: from='client.? 172.21.15.115:0/1995739106' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-20T08:56:32.378 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:31 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-20T08:56:32.378 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:31 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-20T08:56:32.405 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2024-03-20T08:56:32.406 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-20T08:56:32.406 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-20T08:56:32.447 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-20T08:56:32.723 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T08:56:33.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:32 smithi122 ceph-mon[73495]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:33.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:32 smithi026 ceph-mon[68309]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:33.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:32 smithi115 ceph-mon[72476]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:34.790 INFO:teuthology.orchestra.run.smithi122.stdout:[client.2] 2024-03-20T08:56:34.790 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQDCpPplH3LkLhAAVRTwXxyvzS+iT4ZXBBKVfQ== 2024-03-20T08:56:35.178 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:34 smithi122 ceph-mon[73495]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:35.178 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:34 smithi122 ceph-mon[73495]: from='client.? 172.21.15.122:0/3206439329' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-20T08:56:35.178 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:34 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-20T08:56:35.178 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:34 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-20T08:56:35.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:34 smithi026 ceph-mon[68309]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:35.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:34 smithi026 ceph-mon[68309]: from='client.? 172.21.15.122:0/3206439329' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-20T08:56:35.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:34 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-20T08:56:35.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:34 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-20T08:56:35.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:34 smithi115 ceph-mon[72476]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:35.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:34 smithi115 ceph-mon[72476]: from='client.? 172.21.15.122:0/3206439329' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-20T08:56:35.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:34 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-20T08:56:35.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:34 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-20T08:56:35.536 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-20T08:56:35.536 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-03-20T08:56:35.536 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-20T08:56:35.576 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-20T08:56:35.576 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-20T08:56:35.576 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph mgr dump --format=json 2024-03-20T08:56:35.862 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:56:36.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:36 smithi026 ceph-mon[68309]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:37.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:36 smithi115 ceph-mon[72476]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:37.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:36 smithi122 ceph-mon[73495]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:37.940 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:56:38.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:37 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1543625409' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-20T08:56:38.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:37 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1543625409' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-20T08:56:38.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:37 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1543625409' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-20T08:56:38.548 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":15,"flags":0,"active_gid":14148,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6800","nonce":1717869069},{"type":"v1","addr":"172.21.15.26:6801","nonce":1717869069}]},"active_addr":"172.21.15.26:6801/1717869069","active_change":"2024-03-20T08:49:35.337033+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24107,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.26:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.26:0","nonce":3746860518}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.26:0","nonce":1075901101}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.26:0","nonce":1806389690}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.26:0","nonce":859200146}]}]} 2024-03-20T08:56:38.555 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-20T08:56:38.555 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-20T08:56:38.555 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-20T08:56:38.831 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:56:39.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:38 smithi026 ceph-mon[68309]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:39.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:38 smithi115 ceph-mon[72476]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:39.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:38 smithi122 ceph-mon[73495]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:40.793 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:56:40.793 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":104,"fsid":"852fbb68-e696-11ee-95c9-87774f69a715","created":"2024-03-20T08:48:35.281088+0000","modified":"2024-03-20T08:56:29.928485+0000","last_up_change":"2024-03-20T08:56:17.873738+0000","last_in_change":"2024-03-20T08:55:59.486577+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-20T08:52:17.442324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad3fd9c0-316f-4fe5-a63b-d480087c1729","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":2139805293},{"type":"v1","addr":"172.21.15.26:6803","nonce":2139805293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":2139805293},{"type":"v1","addr":"172.21.15.26:6805","nonce":2139805293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":2139805293},{"type":"v1","addr":"172.21.15.26:6809","nonce":2139805293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":2139805293},{"type":"v1","addr":"172.21.15.26:6807","nonce":2139805293}]},"public_addr":"172.21.15.26:6803/2139805293","cluster_addr":"172.21.15.26:6805/2139805293","heartbeat_back_addr":"172.21.15.26:6809/2139805293","heartbeat_front_addr":"172.21.15.26:6807/2139805293","state":["exists","up"]},{"osd":1,"uuid":"e85b94f5-2954-47a1-9c20-096f2606eb04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":3853355134},{"type":"v1","addr":"172.21.15.26:6811","nonce":3853355134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":3853355134},{"type":"v1","addr":"172.21.15.26:6813","nonce":3853355134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":3853355134},{"type":"v1","addr":"172.21.15.26:6817","nonce":3853355134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":3853355134},{"type":"v1","addr":"172.21.15.26:6815","nonce":3853355134}]},"public_addr":"172.21.15.26:6811/3853355134","cluster_addr":"172.21.15.26:6813/3853355134","heartbeat_back_addr":"172.21.15.26:6817/3853355134","heartbeat_front_addr":"172.21.15.26:6815/3853355134","state":["exists","up"]},{"osd":2,"uuid":"dd2b10f0-741c-47a4-b224-ac6bfe2c14ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":3993357967},{"type":"v1","addr":"172.21.15.26:6819","nonce":3993357967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":3993357967},{"type":"v1","addr":"172.21.15.26:6821","nonce":3993357967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":3993357967},{"type":"v1","addr":"172.21.15.26:6825","nonce":3993357967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":3993357967},{"type":"v1","addr":"172.21.15.26:6823","nonce":3993357967}]},"public_addr":"172.21.15.26:6819/3993357967","cluster_addr":"172.21.15.26:6821/3993357967","heartbeat_back_addr":"172.21.15.26:6825/3993357967","heartbeat_front_addr":"172.21.15.26:6823/3993357967","state":["exists","up"]},{"osd":3,"uuid":"9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":4110225509},{"type":"v1","addr":"172.21.15.26:6827","nonce":4110225509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":4110225509},{"type":"v1","addr":"172.21.15.26:6829","nonce":4110225509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":4110225509},{"type":"v1","addr":"172.21.15.26:6833","nonce":4110225509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":4110225509},{"type":"v1","addr":"172.21.15.26:6831","nonce":4110225509}]},"public_addr":"172.21.15.26:6827/4110225509","cluster_addr":"172.21.15.26:6829/4110225509","heartbeat_back_addr":"172.21.15.26:6833/4110225509","heartbeat_front_addr":"172.21.15.26:6831/4110225509","state":["exists","up"]},{"osd":4,"uuid":"f14e34d5-9829-42a3-9b25-31a0976c5ef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":477923058},{"type":"v1","addr":"172.21.15.115:6801","nonce":477923058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":477923058},{"type":"v1","addr":"172.21.15.115:6803","nonce":477923058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":477923058},{"type":"v1","addr":"172.21.15.115:6807","nonce":477923058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":477923058},{"type":"v1","addr":"172.21.15.115:6805","nonce":477923058}]},"public_addr":"172.21.15.115:6801/477923058","cluster_addr":"172.21.15.115:6803/477923058","heartbeat_back_addr":"172.21.15.115:6807/477923058","heartbeat_front_addr":"172.21.15.115:6805/477923058","state":["exists","up"]},{"osd":5,"uuid":"cb431b6a-540d-4318-ae35-d379e1df0185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1339268003},{"type":"v1","addr":"172.21.15.115:6809","nonce":1339268003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":1339268003},{"type":"v1","addr":"172.21.15.115:6811","nonce":1339268003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":1339268003},{"type":"v1","addr":"172.21.15.115:6815","nonce":1339268003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":1339268003},{"type":"v1","addr":"172.21.15.115:6813","nonce":1339268003}]},"public_addr":"172.21.15.115:6809/1339268003","cluster_addr":"172.21.15.115:6811/1339268003","heartbeat_back_addr":"172.21.15.115:6815/1339268003","heartbeat_front_addr":"172.21.15.115:6813/1339268003","state":["exists","up"]},{"osd":6,"uuid":"753ed830-e764-4ff3-9d9b-87e7fa748b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3278301843},{"type":"v1","addr":"172.21.15.115:6817","nonce":3278301843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":3278301843},{"type":"v1","addr":"172.21.15.115:6819","nonce":3278301843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":3278301843},{"type":"v1","addr":"172.21.15.115:6823","nonce":3278301843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":3278301843},{"type":"v1","addr":"172.21.15.115:6821","nonce":3278301843}]},"public_addr":"172.21.15.115:6817/3278301843","cluster_addr":"172.21.15.115:6819/3278301843","heartbeat_back_addr":"172.21.15.115:6823/3278301843","heartbeat_front_addr":"172.21.15.115:6821/3278301843","state":["exists","up"]},{"osd":7,"uuid":"55bcbd37-7a6f-4ab1-b5a9-91a951444d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":3152424001},{"type":"v1","addr":"172.21.15.115:6825","nonce":3152424001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3152424001},{"type":"v1","addr":"172.21.15.115:6827","nonce":3152424001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3152424001},{"type":"v1","addr":"172.21.15.115:6831","nonce":3152424001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3152424001},{"type":"v1","addr":"172.21.15.115:6829","nonce":3152424001}]},"public_addr":"172.21.15.115:6825/3152424001","cluster_addr":"172.21.15.115:6827/3152424001","heartbeat_back_addr":"172.21.15.115:6831/3152424001","heartbeat_front_addr":"172.21.15.115:6829/3152424001","state":["exists","up"]},{"osd":8,"uuid":"bc86248e-6bd0-4369-b8a2-e593bc2a610d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1309426740},{"type":"v1","addr":"172.21.15.122:6801","nonce":1309426740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1309426740},{"type":"v1","addr":"172.21.15.122:6803","nonce":1309426740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1309426740},{"type":"v1","addr":"172.21.15.122:6807","nonce":1309426740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1309426740},{"type":"v1","addr":"172.21.15.122:6805","nonce":1309426740}]},"public_addr":"172.21.15.122:6801/1309426740","cluster_addr":"172.21.15.122:6803/1309426740","heartbeat_back_addr":"172.21.15.122:6807/1309426740","heartbeat_front_addr":"172.21.15.122:6805/1309426740","state":["exists","up"]},{"osd":9,"uuid":"9b20a4aa-032a-4c09-ac75-153c554424e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":2953210076},{"type":"v1","addr":"172.21.15.122:6809","nonce":2953210076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":2953210076},{"type":"v1","addr":"172.21.15.122:6811","nonce":2953210076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2953210076},{"type":"v1","addr":"172.21.15.122:6815","nonce":2953210076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2953210076},{"type":"v1","addr":"172.21.15.122:6813","nonce":2953210076}]},"public_addr":"172.21.15.122:6809/2953210076","cluster_addr":"172.21.15.122:6811/2953210076","heartbeat_back_addr":"172.21.15.122:6815/2953210076","heartbeat_front_addr":"172.21.15.122:6813/2953210076","state":["exists","up"]},{"osd":10,"uuid":"e62c1fb3-5dd1-4d0d-997c-d20633d19361","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":89,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":409394340},{"type":"v1","addr":"172.21.15.122:6817","nonce":409394340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":409394340},{"type":"v1","addr":"172.21.15.122:6819","nonce":409394340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":409394340},{"type":"v1","addr":"172.21.15.122:6823","nonce":409394340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":409394340},{"type":"v1","addr":"172.21.15.122:6821","nonce":409394340}]},"public_addr":"172.21.15.122:6817/409394340","cluster_addr":"172.21.15.122:6819/409394340","heartbeat_back_addr":"172.21.15.122:6823/409394340","heartbeat_front_addr":"172.21.15.122:6821/409394340","state":["exists","up"]},{"osd":11,"uuid":"b83994d5-e5ef-48b0-9758-c26843cec3b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":96,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":2704069354},{"type":"v1","addr":"172.21.15.122:6825","nonce":2704069354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":2704069354},{"type":"v1","addr":"172.21.15.122:6827","nonce":2704069354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":2704069354},{"type":"v1","addr":"172.21.15.122:6831","nonce":2704069354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":2704069354},{"type":"v1","addr":"172.21.15.122:6829","nonce":2704069354}]},"public_addr":"172.21.15.122:6825/2704069354","cluster_addr":"172.21.15.122:6827/2704069354","heartbeat_back_addr":"172.21.15.122:6831/2704069354","heartbeat_front_addr":"172.21.15.122:6829/2704069354","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:56:26.947684+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:55:29.017333+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:56:21.912328+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:56:25.197670+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:56:29.596231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:56:02.401511+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:55:27.457509+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:55:36.320737+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:56:00.648916+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:55:20.203922+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:55:48.088091+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:56:16.011630+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:0/4291404191":"2024-03-21T08:49:35.336921+0000","172.21.15.26:0/642012566":"2024-03-21T08:49:35.336921+0000","172.21.15.26:6801/1181459999":"2024-03-21T08:49:35.336921+0000","172.21.15.26:0/3984513952":"2024-03-21T08:49:04.596754+0000","172.21.15.26:6800/1654945498":"2024-03-21T08:49:04.596754+0000","172.21.15.26:6800/1181459999":"2024-03-21T08:49:35.336921+0000","172.21.15.26:0/137770981":"2024-03-21T08:49:04.596754+0000","172.21.15.26:0/1451237762":"2024-03-21T08:49:35.336921+0000","172.21.15.26:0/3938386631":"2024-03-21T08:49:04.596754+0000","172.21.15.26:6801/1654945498":"2024-03-21T08:49:04.596754+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-20T08:56:41.120 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:40 smithi026 ceph-mon[68309]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:41.120 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:40 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1902754060' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-20T08:56:41.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:40 smithi115 ceph-mon[72476]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:41.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:40 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1902754060' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-20T08:56:41.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:40 smithi122 ceph-mon[73495]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:41.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:40 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1902754060' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-20T08:56:41.486 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-20T08:56:41.487 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-20T08:56:41.761 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:56:42.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:41 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:56:42.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:41 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:56:42.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:41 smithi026 ceph-mon[68309]: osdmap e105: 12 total, 12 up, 12 in 2024-03-20T08:56:42.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:41 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:56:42.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:41 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:56:42.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:41 smithi115 ceph-mon[72476]: osdmap e105: 12 total, 12 up, 12 in 2024-03-20T08:56:42.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:41 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:56:42.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:41 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:56:42.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:41 smithi122 ceph-mon[73495]: osdmap e105: 12 total, 12 up, 12 in 2024-03-20T08:56:43.173 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:42 smithi026 ceph-mon[68309]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:43.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:42 smithi115 ceph-mon[72476]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:43.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:42 smithi122 ceph-mon[73495]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:43.677 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:56:43.677 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":105,"fsid":"852fbb68-e696-11ee-95c9-87774f69a715","created":"2024-03-20T08:48:35.281088+0000","modified":"2024-03-20T08:56:40.953556+0000","last_up_change":"2024-03-20T08:56:17.873738+0000","last_in_change":"2024-03-20T08:55:59.486577+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-20T08:52:17.442324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad3fd9c0-316f-4fe5-a63b-d480087c1729","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":2139805293},{"type":"v1","addr":"172.21.15.26:6803","nonce":2139805293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":2139805293},{"type":"v1","addr":"172.21.15.26:6805","nonce":2139805293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":2139805293},{"type":"v1","addr":"172.21.15.26:6809","nonce":2139805293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":2139805293},{"type":"v1","addr":"172.21.15.26:6807","nonce":2139805293}]},"public_addr":"172.21.15.26:6803/2139805293","cluster_addr":"172.21.15.26:6805/2139805293","heartbeat_back_addr":"172.21.15.26:6809/2139805293","heartbeat_front_addr":"172.21.15.26:6807/2139805293","state":["exists","up"]},{"osd":1,"uuid":"e85b94f5-2954-47a1-9c20-096f2606eb04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":3853355134},{"type":"v1","addr":"172.21.15.26:6811","nonce":3853355134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":3853355134},{"type":"v1","addr":"172.21.15.26:6813","nonce":3853355134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":3853355134},{"type":"v1","addr":"172.21.15.26:6817","nonce":3853355134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":3853355134},{"type":"v1","addr":"172.21.15.26:6815","nonce":3853355134}]},"public_addr":"172.21.15.26:6811/3853355134","cluster_addr":"172.21.15.26:6813/3853355134","heartbeat_back_addr":"172.21.15.26:6817/3853355134","heartbeat_front_addr":"172.21.15.26:6815/3853355134","state":["exists","up"]},{"osd":2,"uuid":"dd2b10f0-741c-47a4-b224-ac6bfe2c14ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":3993357967},{"type":"v1","addr":"172.21.15.26:6819","nonce":3993357967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":3993357967},{"type":"v1","addr":"172.21.15.26:6821","nonce":3993357967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":3993357967},{"type":"v1","addr":"172.21.15.26:6825","nonce":3993357967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":3993357967},{"type":"v1","addr":"172.21.15.26:6823","nonce":3993357967}]},"public_addr":"172.21.15.26:6819/3993357967","cluster_addr":"172.21.15.26:6821/3993357967","heartbeat_back_addr":"172.21.15.26:6825/3993357967","heartbeat_front_addr":"172.21.15.26:6823/3993357967","state":["exists","up"]},{"osd":3,"uuid":"9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":4110225509},{"type":"v1","addr":"172.21.15.26:6827","nonce":4110225509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":4110225509},{"type":"v1","addr":"172.21.15.26:6829","nonce":4110225509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":4110225509},{"type":"v1","addr":"172.21.15.26:6833","nonce":4110225509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":4110225509},{"type":"v1","addr":"172.21.15.26:6831","nonce":4110225509}]},"public_addr":"172.21.15.26:6827/4110225509","cluster_addr":"172.21.15.26:6829/4110225509","heartbeat_back_addr":"172.21.15.26:6833/4110225509","heartbeat_front_addr":"172.21.15.26:6831/4110225509","state":["exists","up"]},{"osd":4,"uuid":"f14e34d5-9829-42a3-9b25-31a0976c5ef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":477923058},{"type":"v1","addr":"172.21.15.115:6801","nonce":477923058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":477923058},{"type":"v1","addr":"172.21.15.115:6803","nonce":477923058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":477923058},{"type":"v1","addr":"172.21.15.115:6807","nonce":477923058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":477923058},{"type":"v1","addr":"172.21.15.115:6805","nonce":477923058}]},"public_addr":"172.21.15.115:6801/477923058","cluster_addr":"172.21.15.115:6803/477923058","heartbeat_back_addr":"172.21.15.115:6807/477923058","heartbeat_front_addr":"172.21.15.115:6805/477923058","state":["exists","up"]},{"osd":5,"uuid":"cb431b6a-540d-4318-ae35-d379e1df0185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":1339268003},{"type":"v1","addr":"172.21.15.115:6809","nonce":1339268003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":1339268003},{"type":"v1","addr":"172.21.15.115:6811","nonce":1339268003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":1339268003},{"type":"v1","addr":"172.21.15.115:6815","nonce":1339268003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":1339268003},{"type":"v1","addr":"172.21.15.115:6813","nonce":1339268003}]},"public_addr":"172.21.15.115:6809/1339268003","cluster_addr":"172.21.15.115:6811/1339268003","heartbeat_back_addr":"172.21.15.115:6815/1339268003","heartbeat_front_addr":"172.21.15.115:6813/1339268003","state":["exists","up"]},{"osd":6,"uuid":"753ed830-e764-4ff3-9d9b-87e7fa748b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3278301843},{"type":"v1","addr":"172.21.15.115:6817","nonce":3278301843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":3278301843},{"type":"v1","addr":"172.21.15.115:6819","nonce":3278301843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":3278301843},{"type":"v1","addr":"172.21.15.115:6823","nonce":3278301843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":3278301843},{"type":"v1","addr":"172.21.15.115:6821","nonce":3278301843}]},"public_addr":"172.21.15.115:6817/3278301843","cluster_addr":"172.21.15.115:6819/3278301843","heartbeat_back_addr":"172.21.15.115:6823/3278301843","heartbeat_front_addr":"172.21.15.115:6821/3278301843","state":["exists","up"]},{"osd":7,"uuid":"55bcbd37-7a6f-4ab1-b5a9-91a951444d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":3152424001},{"type":"v1","addr":"172.21.15.115:6825","nonce":3152424001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3152424001},{"type":"v1","addr":"172.21.15.115:6827","nonce":3152424001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3152424001},{"type":"v1","addr":"172.21.15.115:6831","nonce":3152424001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3152424001},{"type":"v1","addr":"172.21.15.115:6829","nonce":3152424001}]},"public_addr":"172.21.15.115:6825/3152424001","cluster_addr":"172.21.15.115:6827/3152424001","heartbeat_back_addr":"172.21.15.115:6831/3152424001","heartbeat_front_addr":"172.21.15.115:6829/3152424001","state":["exists","up"]},{"osd":8,"uuid":"bc86248e-6bd0-4369-b8a2-e593bc2a610d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1309426740},{"type":"v1","addr":"172.21.15.122:6801","nonce":1309426740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1309426740},{"type":"v1","addr":"172.21.15.122:6803","nonce":1309426740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1309426740},{"type":"v1","addr":"172.21.15.122:6807","nonce":1309426740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1309426740},{"type":"v1","addr":"172.21.15.122:6805","nonce":1309426740}]},"public_addr":"172.21.15.122:6801/1309426740","cluster_addr":"172.21.15.122:6803/1309426740","heartbeat_back_addr":"172.21.15.122:6807/1309426740","heartbeat_front_addr":"172.21.15.122:6805/1309426740","state":["exists","up"]},{"osd":9,"uuid":"9b20a4aa-032a-4c09-ac75-153c554424e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":2953210076},{"type":"v1","addr":"172.21.15.122:6809","nonce":2953210076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":2953210076},{"type":"v1","addr":"172.21.15.122:6811","nonce":2953210076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2953210076},{"type":"v1","addr":"172.21.15.122:6815","nonce":2953210076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2953210076},{"type":"v1","addr":"172.21.15.122:6813","nonce":2953210076}]},"public_addr":"172.21.15.122:6809/2953210076","cluster_addr":"172.21.15.122:6811/2953210076","heartbeat_back_addr":"172.21.15.122:6815/2953210076","heartbeat_front_addr":"172.21.15.122:6813/2953210076","state":["exists","up"]},{"osd":10,"uuid":"e62c1fb3-5dd1-4d0d-997c-d20633d19361","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":89,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":409394340},{"type":"v1","addr":"172.21.15.122:6817","nonce":409394340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":409394340},{"type":"v1","addr":"172.21.15.122:6819","nonce":409394340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":409394340},{"type":"v1","addr":"172.21.15.122:6823","nonce":409394340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":409394340},{"type":"v1","addr":"172.21.15.122:6821","nonce":409394340}]},"public_addr":"172.21.15.122:6817/409394340","cluster_addr":"172.21.15.122:6819/409394340","heartbeat_back_addr":"172.21.15.122:6823/409394340","heartbeat_front_addr":"172.21.15.122:6821/409394340","state":["exists","up"]},{"osd":11,"uuid":"b83994d5-e5ef-48b0-9758-c26843cec3b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":96,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":2704069354},{"type":"v1","addr":"172.21.15.122:6825","nonce":2704069354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":2704069354},{"type":"v1","addr":"172.21.15.122:6827","nonce":2704069354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":2704069354},{"type":"v1","addr":"172.21.15.122:6831","nonce":2704069354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":2704069354},{"type":"v1","addr":"172.21.15.122:6829","nonce":2704069354}]},"public_addr":"172.21.15.122:6825/2704069354","cluster_addr":"172.21.15.122:6827/2704069354","heartbeat_back_addr":"172.21.15.122:6831/2704069354","heartbeat_front_addr":"172.21.15.122:6829/2704069354","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:56:26.947684+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:55:29.017333+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:56:21.912328+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:56:25.197670+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:56:29.596231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:56:02.401511+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:55:27.457509+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:55:36.320737+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:56:00.648916+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:56:40.758355+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:55:48.088091+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T08:56:16.011630+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:0/4291404191":"2024-03-21T08:49:35.336921+0000","172.21.15.26:0/642012566":"2024-03-21T08:49:35.336921+0000","172.21.15.26:6801/1181459999":"2024-03-21T08:49:35.336921+0000","172.21.15.26:0/3984513952":"2024-03-21T08:49:04.596754+0000","172.21.15.26:6800/1654945498":"2024-03-21T08:49:04.596754+0000","172.21.15.26:6800/1181459999":"2024-03-21T08:49:35.336921+0000","172.21.15.26:0/137770981":"2024-03-21T08:49:04.596754+0000","172.21.15.26:0/1451237762":"2024-03-21T08:49:35.336921+0000","172.21.15.26:0/3938386631":"2024-03-21T08:49:04.596754+0000","172.21.15.26:6801/1654945498":"2024-03-21T08:49:04.596754+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-20T08:56:44.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:43 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/751488689' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-20T08:56:44.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:43 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/751488689' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-20T08:56:44.420 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-20T08:56:44.420 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-20T08:56:44.421 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-20T08:56:44.421 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-20T08:56:44.421 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-20T08:56:44.422 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-20T08:56:44.422 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-20T08:56:44.423 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-20T08:56:44.423 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-20T08:56:44.424 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-20T08:56:44.424 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-20T08:56:44.424 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-20T08:56:44.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:43 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/751488689' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-20T08:56:45.134 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:56:45.175 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:56:45.185 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:56:45.210 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:56:45.344 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:56:45.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:44 smithi115 ceph-mon[72476]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:45.421 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:44 smithi026 ceph-mon[68309]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:45.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:44 smithi122 ceph-mon[73495]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:45.447 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:56:45.447 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:56:46.480 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:56:46.481 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:56:47.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:46 smithi026 ceph-mon[68309]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:47.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:46 smithi115 ceph-mon[72476]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:47.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:46 smithi122 ceph-mon[73495]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:48.274 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:56:49.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:48 smithi026 ceph-mon[68309]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:49.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:48 smithi115 ceph-mon[72476]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:49.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:48 smithi122 ceph-mon[73495]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:51.373 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:50 smithi026 ceph-mon[68309]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:51.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:50 smithi115 ceph-mon[72476]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:51.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:50 smithi122 ceph-mon[73495]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:52.112 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:51 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:56:52.112 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:51 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:56:52.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:51 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:56:52.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:51 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:56:52.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:51 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:56:52.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:51 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:56:53.329 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:53 smithi026 ceph-mon[68309]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:53.329 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:53 smithi026 ceph-mon[68309]: osdmap e106: 12 total, 12 up, 12 in 2024-03-20T08:56:53.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:53 smithi115 ceph-mon[72476]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:53.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:53 smithi115 ceph-mon[72476]: osdmap e106: 12 total, 12 up, 12 in 2024-03-20T08:56:53.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:53 smithi122 ceph-mon[73495]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:53.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:53 smithi122 ceph-mon[73495]: osdmap e106: 12 total, 12 up, 12 in 2024-03-20T08:56:54.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:54 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:56:54.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:54 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:56:54.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:54 smithi115 ceph-mon[72476]: osdmap e107: 12 total, 12 up, 12 in 2024-03-20T08:56:54.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:54 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:56:54.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:54 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:56:54.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:54 smithi122 ceph-mon[73495]: osdmap e107: 12 total, 12 up, 12 in 2024-03-20T08:56:54.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:54 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:56:54.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:54 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:56:54.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:54 smithi026 ceph-mon[68309]: osdmap e107: 12 total, 12 up, 12 in 2024-03-20T08:56:54.529 INFO:teuthology.orchestra.run.smithi026.stdout:382252089359 2024-03-20T08:56:54.529 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-20T08:56:55.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:55 smithi026 ceph-mon[68309]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:55.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:55 smithi115 ceph-mon[72476]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:55.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:55 smithi122 ceph-mon[73495]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:57.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:57 smithi026 ceph-mon[68309]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:57.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:57 smithi115 ceph-mon[72476]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:57.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:57 smithi122 ceph-mon[73495]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:57.691 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:56:58.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:58 smithi026 ceph-mon[68309]: 1.0 scrub starts 2024-03-20T08:56:58.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:58 smithi026 ceph-mon[68309]: 1.0 scrub ok 2024-03-20T08:56:58.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:58 smithi115 ceph-mon[72476]: 1.0 scrub starts 2024-03-20T08:56:58.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:58 smithi115 ceph-mon[72476]: 1.0 scrub ok 2024-03-20T08:56:58.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:58 smithi122 ceph-mon[73495]: 1.0 scrub starts 2024-03-20T08:56:58.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:58 smithi122 ceph-mon[73495]: 1.0 scrub ok 2024-03-20T08:56:59.319 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:56:59 smithi026 ceph-mon[68309]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:59.358 INFO:teuthology.orchestra.run.smithi026.stdout:111669149749 2024-03-20T08:56:59.359 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-20T08:56:59.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:56:59 smithi115 ceph-mon[72476]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:59.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:56:59 smithi122 ceph-mon[73495]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:56:59.909 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:56:59.946 INFO:teuthology.orchestra.run.smithi026.stdout:257698037792 2024-03-20T08:56:59.946 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-20T08:57:01.092 INFO:teuthology.orchestra.run.smithi026.stdout:176093659179 2024-03-20T08:57:01.093 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-20T08:57:01.399 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:01 smithi026 ceph-mon[68309]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:01.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:01 smithi115 ceph-mon[72476]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:01.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:01 smithi122 ceph-mon[73495]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:01.800 INFO:teuthology.orchestra.run.smithi026.stdout:55834574911 2024-03-20T08:57:01.801 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-20T08:57:02.911 INFO:teuthology.orchestra.run.smithi026.stdout:34359738439 2024-03-20T08:57:02.911 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-20T08:57:03.140 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:03 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:57:03.140 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:03 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:57:03.140 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:03 smithi026 ceph-mon[68309]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:03.140 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:03 smithi026 ceph-mon[68309]: osdmap e108: 12 total, 12 up, 12 in 2024-03-20T08:57:03.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:03 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:57:03.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:03 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:57:03.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:03 smithi115 ceph-mon[72476]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:03.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:03 smithi115 ceph-mon[72476]: osdmap e108: 12 total, 12 up, 12 in 2024-03-20T08:57:03.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:03 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:57:03.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:03 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:57:03.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:03 smithi122 ceph-mon[73495]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:03.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:03 smithi122 ceph-mon[73495]: osdmap e108: 12 total, 12 up, 12 in 2024-03-20T08:57:03.552 INFO:teuthology.orchestra.run.smithi026.stdout:343597383703 2024-03-20T08:57:03.553 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-20T08:57:04.150 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:04 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:57:04.150 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:04 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:57:04.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:04 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:57:04.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:04 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:57:04.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:04 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:57:04.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:04 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:57:04.452 INFO:teuthology.orchestra.run.smithi026.stdout:77309411386 2024-03-20T08:57:04.453 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-20T08:57:05.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:05 smithi026 ceph-mon[68309]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:05.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:05 smithi026 ceph-mon[68309]: osdmap e109: 12 total, 12 up, 12 in 2024-03-20T08:57:05.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:05 smithi115 ceph-mon[72476]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:05.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:05 smithi115 ceph-mon[72476]: osdmap e109: 12 total, 12 up, 12 in 2024-03-20T08:57:05.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:05 smithi122 ceph-mon[73495]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:05.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:05 smithi122 ceph-mon[73495]: osdmap e109: 12 total, 12 up, 12 in 2024-03-20T08:57:06.837 INFO:teuthology.orchestra.run.smithi026.stdout:412316860429 2024-03-20T08:57:06.837 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-20T08:57:07.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:07 smithi026 ceph-mon[68309]: pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:07.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:07 smithi115 ceph-mon[72476]: pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:07.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:07 smithi122 ceph-mon[73495]: pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:08.358 INFO:teuthology.orchestra.run.smithi026.stdout:219043332136 2024-03-20T08:57:08.358 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-20T08:57:09.195 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:57:09.196 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:57:09.197 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:57:09.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:09 smithi026 ceph-mon[68309]: pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:09.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:09 smithi115 ceph-mon[72476]: pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:09.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:09 smithi122 ceph-mon[73495]: pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:10.582 INFO:teuthology.orchestra.run.smithi026.stdout:137438953522 2024-03-20T08:57:10.583 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-20T08:57:11.146 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:11 smithi026 ceph-mon[68309]: pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:11.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:11 smithi115 ceph-mon[72476]: pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:11.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:11 smithi122 ceph-mon[73495]: pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:12.145 INFO:teuthology.orchestra.run.smithi026.stdout:111669149752 2024-03-20T08:57:12.290 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:57:12.291 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:57:12.291 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:57:12.824 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:57:12.826 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:57:12.826 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:57:12.846 INFO:teuthology.orchestra.run.smithi026.stdout:300647710750 2024-03-20T08:57:12.847 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-20T08:57:13.065 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:13 smithi026 ceph-mon[68309]: pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:13.065 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:13 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4266487536' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-20T08:57:13.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:13 smithi115 ceph-mon[72476]: pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:13.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:13 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4266487536' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-20T08:57:13.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:13 smithi122 ceph-mon[73495]: pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:13.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:13 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4266487536' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-20T08:57:15.343 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:15 smithi026 ceph-mon[68309]: pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:15.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:15 smithi115 ceph-mon[72476]: pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:15.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:15 smithi122 ceph-mon[73495]: pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:16.446 INFO:teuthology.orchestra.run.smithi026.stdout:382252089363 2024-03-20T08:57:17.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:17 smithi115 ceph-mon[72476]: pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:17.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:17 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/583291635' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-20T08:57:17.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:17 smithi122 ceph-mon[73495]: pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:17.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:17 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/583291635' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-20T08:57:17.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:17 smithi026 ceph-mon[68309]: pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:17.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:17 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/583291635' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-20T08:57:19.249 INFO:teuthology.orchestra.run.smithi026.stdout:343597383706 2024-03-20T08:57:19.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:19 smithi115 ceph-mon[72476]: pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:19.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:19 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/8034444' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-20T08:57:19.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:19 smithi122 ceph-mon[73495]: pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:19.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:19 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/8034444' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-20T08:57:19.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:19 smithi026 ceph-mon[68309]: pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:19.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:19 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/8034444' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-20T08:57:20.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:20 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:57:20.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:20 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:57:20.451 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:20 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:57:21.089 INFO:teuthology.orchestra.run.smithi026.stdout:257698037796 2024-03-20T08:57:21.164 INFO:teuthology.orchestra.run.smithi026.stdout:55834574916 2024-03-20T08:57:21.351 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:21 smithi026 ceph-mon[68309]: pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:21.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:21 smithi115 ceph-mon[72476]: pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:21.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:21 smithi122 ceph-mon[73495]: pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:21.500 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:57:21.531 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149749 got 111669149752 for osd.3 2024-03-20T08:57:21.531 DEBUG:teuthology.parallel:result is None 2024-03-20T08:57:22.236 INFO:teuthology.orchestra.run.smithi026.stdout:219043332139 2024-03-20T08:57:22.351 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:22 smithi026 ceph-mon[68309]: from='client.14607 172.21.15.26:0/3959182198' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-20T08:57:22.352 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:22 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1210799929' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-20T08:57:22.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:22 smithi115 ceph-mon[72476]: from='client.14607 172.21.15.26:0/3959182198' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-20T08:57:22.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:22 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1210799929' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-20T08:57:22.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:22 smithi122 ceph-mon[73495]: from='client.14607 172.21.15.26:0/3959182198' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-20T08:57:22.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:22 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1210799929' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-20T08:57:22.727 INFO:teuthology.orchestra.run.smithi026.stdout:77309411390 2024-03-20T08:57:22.948 INFO:tasks.cephadm.ceph_manager.ceph:need seq 382252089359 got 382252089363 for osd.10 2024-03-20T08:57:22.948 DEBUG:teuthology.parallel:result is None 2024-03-20T08:57:23.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:23 smithi115 ceph-mon[72476]: pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:23.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:23 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3090381502' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-20T08:57:23.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:23 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2206161078' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-20T08:57:23.422 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:23 smithi026 ceph-mon[68309]: pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:23.422 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:23 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3090381502' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-20T08:57:23.422 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:23 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2206161078' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-20T08:57:23.422 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:57:23.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:23 smithi122 ceph-mon[73495]: pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:23.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:23 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3090381502' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-20T08:57:23.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:23 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2206161078' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-20T08:57:24.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:24 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:57:24.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:24 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:57:24.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:24 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:57:24.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:24 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:57:24.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:24 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:57:24.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:24 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:57:24.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:24 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:57:24.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:24 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:57:24.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:24 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:57:24.531 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411386 got 77309411390 for osd.2 2024-03-20T08:57:24.531 DEBUG:teuthology.parallel:result is None 2024-03-20T08:57:25.097 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332136 got 219043332139 for osd.6 2024-03-20T08:57:25.097 DEBUG:teuthology.parallel:result is None 2024-03-20T08:57:25.326 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:25 smithi026 ceph-mon[68309]: pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:25.356 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383703 got 343597383706 for osd.9 2024-03-20T08:57:25.356 DEBUG:teuthology.parallel:result is None 2024-03-20T08:57:25.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:25 smithi115 ceph-mon[72476]: pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:25.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:25 smithi122 ceph-mon[73495]: pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:25.914 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574911 got 55834574916 for osd.1 2024-03-20T08:57:25.914 DEBUG:teuthology.parallel:result is None 2024-03-20T08:57:27.242 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:27 smithi026 ceph-mon[68309]: pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:27.242 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:27 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:57:27.242 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:27 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:57:27.242 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:27 smithi026 ceph-mon[68309]: osdmap e110: 12 total, 12 up, 12 in 2024-03-20T08:57:27.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:27 smithi115 ceph-mon[72476]: pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:27.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:27 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:57:27.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:27 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:57:27.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:27 smithi115 ceph-mon[72476]: osdmap e110: 12 total, 12 up, 12 in 2024-03-20T08:57:27.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:27 smithi122 ceph-mon[73495]: pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:27.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:27 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:57:27.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:27 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:57:27.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:27 smithi122 ceph-mon[73495]: osdmap e110: 12 total, 12 up, 12 in 2024-03-20T08:57:27.718 INFO:teuthology.orchestra.run.smithi026.stdout:176093659184 2024-03-20T08:57:28.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:28 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:57:28.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:28 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:57:28.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:28 smithi115 ceph-mon[72476]: osdmap e111: 12 total, 12 up, 12 in 2024-03-20T08:57:28.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:28 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3815309588' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-20T08:57:28.423 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:28 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:57:28.424 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:28 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:57:28.424 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:28 smithi026 ceph-mon[68309]: osdmap e111: 12 total, 12 up, 12 in 2024-03-20T08:57:28.424 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:28 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3815309588' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-20T08:57:28.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:28 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:57:28.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:28 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:57:28.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:28 smithi122 ceph-mon[73495]: osdmap e111: 12 total, 12 up, 12 in 2024-03-20T08:57:28.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:28 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3815309588' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-20T08:57:28.449 INFO:tasks.cephadm.ceph_manager.ceph:need seq 176093659179 got 176093659184 for osd.5 2024-03-20T08:57:28.449 DEBUG:teuthology.parallel:result is None 2024-03-20T08:57:29.062 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:57:29.094 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037792 got 257698037796 for osd.7 2024-03-20T08:57:29.094 DEBUG:teuthology.parallel:result is None 2024-03-20T08:57:29.293 INFO:teuthology.orchestra.run.smithi026.stdout:412316860433 2024-03-20T08:57:29.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:29 smithi115 ceph-mon[72476]: pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:29.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:29 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2784566231' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-20T08:57:29.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:29 smithi122 ceph-mon[73495]: pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:29.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:29 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2784566231' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-20T08:57:29.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:29 smithi026 ceph-mon[68309]: pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:29.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:29 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2784566231' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-20T08:57:30.005 INFO:tasks.cephadm.ceph_manager.ceph:need seq 412316860429 got 412316860433 for osd.11 2024-03-20T08:57:30.005 DEBUG:teuthology.parallel:result is None 2024-03-20T08:57:30.989 INFO:teuthology.orchestra.run.smithi026.stdout:137438953526 2024-03-20T08:57:31.321 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:31 smithi026 ceph-mon[68309]: pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:31.321 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:31 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1947393448' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-20T08:57:31.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:31 smithi115 ceph-mon[72476]: pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:31.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:31 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1947393448' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-20T08:57:31.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:31 smithi122 ceph-mon[73495]: pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:31.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:31 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1947393448' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-20T08:57:32.258 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:32 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:57:32.258 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:32 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:57:32.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:32 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:57:32.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:32 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:57:32.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:32 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:57:32.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:32 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:57:32.635 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953522 got 137438953526 for osd.4 2024-03-20T08:57:32.636 DEBUG:teuthology.parallel:result is None 2024-03-20T08:57:33.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:33 smithi026 ceph-mon[68309]: pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:33.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:33 smithi026 ceph-mon[68309]: osdmap e112: 12 total, 12 up, 12 in 2024-03-20T08:57:33.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:33 smithi115 ceph-mon[72476]: pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:33.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:33 smithi115 ceph-mon[72476]: osdmap e112: 12 total, 12 up, 12 in 2024-03-20T08:57:33.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:33 smithi122 ceph-mon[73495]: pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:33.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:33 smithi122 ceph-mon[73495]: osdmap e112: 12 total, 12 up, 12 in 2024-03-20T08:57:33.477 INFO:teuthology.orchestra.run.smithi026.stdout:300647710754 2024-03-20T08:57:33.676 INFO:teuthology.orchestra.run.smithi026.stdout:34359738445 2024-03-20T08:57:34.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:34 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1986004896' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-20T08:57:34.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:34 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1050240841' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-20T08:57:34.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:34 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1986004896' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-20T08:57:34.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:34 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1050240841' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-20T08:57:34.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:34 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1986004896' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-20T08:57:34.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:34 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1050240841' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-20T08:57:34.482 INFO:tasks.cephadm.ceph_manager.ceph:need seq 300647710750 got 300647710754 for osd.8 2024-03-20T08:57:34.482 DEBUG:teuthology.parallel:result is None 2024-03-20T08:57:34.822 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738439 got 34359738445 for osd.0 2024-03-20T08:57:34.822 DEBUG:teuthology.parallel:result is None 2024-03-20T08:57:34.822 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-20T08:57:34.822 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T08:57:35.102 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:57:35.106 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:35 smithi026 ceph-mon[68309]: pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:35.107 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:35 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:57:35.107 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:35 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:57:35.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:35 smithi115 ceph-mon[72476]: pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:35.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:35 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:57:35.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:35 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:57:35.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:35 smithi122 ceph-mon[73495]: pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:35.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:35 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:57:35.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:35 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:57:36.181 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:36 smithi026 ceph-mon[68309]: osdmap e113: 12 total, 12 up, 12 in 2024-03-20T08:57:36.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:36 smithi115 ceph-mon[72476]: osdmap e113: 12 total, 12 up, 12 in 2024-03-20T08:57:36.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:36 smithi122 ceph-mon[73495]: osdmap e113: 12 total, 12 up, 12 in 2024-03-20T08:57:36.920 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:57:36.922 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T08:57:37.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:37 smithi026 ceph-mon[68309]: pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:37.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:37 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:57:37.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:37 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:57:37.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:37 smithi115 ceph-mon[72476]: pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:37.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:37 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:57:37.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:37 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:57:37.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:37 smithi122 ceph-mon[73495]: pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:37 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:57:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:37 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:57:37.609 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2024-03-20T08:57:35.431072+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":338244,"kb_used_data":9348,"kb_used_omap":18,"kb_used_meta":328237,"kb_avail":1124603580,"statfs":{"total":1151940427776,"available":1151594065920,"internally_reserved":0,"allocated":9572352,"data_stored":5957601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":336115003},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001574"},"pg_stats":[{"pgid":"1.0","version":"20'137","reported_seq":367,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-03-20T08:57:27.419545+0000","last_change":"2024-03-20T08:56:57.309733+0000","last_active":"2024-03-20T08:57:27.419545+0000","last_peered":"2024-03-20T08:57:27.419545+0000","last_clean":"2024-03-20T08:57:27.419545+0000","last_became_active":"2024-03-20T08:56:21.910528+0000","last_became_peered":"2024-03-20T08:56:21.910528+0000","last_unstale":"2024-03-20T08:57:27.419545+0000","last_undegraded":"2024-03-20T08:57:27.419545+0000","last_fullsized":"2024-03-20T08:57:27.419545+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T08:56:57.309610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T08:56:57.309610+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:58:12.082766+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860435,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28196,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716956,"statfs":{"total":95995035648,"available":95966162944,"internally_reserved":0,"allocated":1204224,"data_stored":897451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":89,"seq":382252089367,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717488,"statfs":{"total":95995035648,"available":95966707712,"internally_reserved":0,"allocated":659456,"data_stored":361146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":80,"seq":343597383710,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717488,"statfs":{"total":95995035648,"available":95966707712,"internally_reserved":0,"allocated":659456,"data_stored":361146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":70,"seq":300647710755,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27788,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717364,"statfs":{"total":95995035648,"available":95966580736,"internally_reserved":0,"allocated":659456,"data_stored":361166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":60,"seq":257698037800,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32468,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712684,"statfs":{"total":95995035648,"available":95961788416,"internally_reserved":0,"allocated":1191936,"data_stored":890094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332142,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28196,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716956,"statfs":{"total":95995035648,"available":95966162944,"internally_reserved":0,"allocated":1204224,"data_stored":897471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574920,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":671744,"data_stored":368543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738446,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27788,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717364,"statfs":{"total":95995035648,"available":95966580736,"internally_reserved":0,"allocated":659456,"data_stored":361166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27664,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717488,"statfs":{"total":95995035648,"available":95966707712,"internally_reserved":0,"allocated":659456,"data_stored":361166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149757,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":671744,"data_stored":368543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953528,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":671744,"data_stored":368543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659186,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27788,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717364,"statfs":{"total":95995035648,"available":95966580736,"internally_reserved":0,"allocated":659456,"data_stored":361166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T08:57:37.610 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T08:57:37.883 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:57:38.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:38 smithi115 ceph-mon[72476]: from='client.14661 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T08:57:38.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:38 smithi115 ceph-mon[72476]: osdmap e114: 12 total, 12 up, 12 in 2024-03-20T08:57:38.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:38 smithi122 ceph-mon[73495]: from='client.14661 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T08:57:38.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:38 smithi122 ceph-mon[73495]: osdmap e114: 12 total, 12 up, 12 in 2024-03-20T08:57:38.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:38 smithi026 ceph-mon[68309]: from='client.14661 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T08:57:38.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:38 smithi026 ceph-mon[68309]: osdmap e114: 12 total, 12 up, 12 in 2024-03-20T08:57:39.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:39 smithi115 ceph-mon[72476]: pgmap v345: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:39.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:39 smithi122 ceph-mon[73495]: pgmap v345: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:39.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:39 smithi026 ceph-mon[68309]: pgmap v345: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:39.978 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:57:39.980 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T08:57:40.583 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2024-03-20T08:57:39.431730+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":337932,"kb_used_data":9528,"kb_used_omap":18,"kb_used_meta":328237,"kb_avail":1124603892,"statfs":{"total":1151940427776,"available":1151594385408,"internally_reserved":0,"allocated":9756672,"data_stored":6068256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":336115003},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.332435"},"pg_stats":[{"pgid":"1.0","version":"20'137","reported_seq":371,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-03-20T08:57:35.637549+0000","last_change":"2024-03-20T08:56:57.309733+0000","last_active":"2024-03-20T08:57:35.637549+0000","last_peered":"2024-03-20T08:57:35.637549+0000","last_clean":"2024-03-20T08:57:35.637549+0000","last_became_active":"2024-03-20T08:56:21.910528+0000","last_became_peered":"2024-03-20T08:56:21.910528+0000","last_unstale":"2024-03-20T08:57:35.637549+0000","last_undegraded":"2024-03-20T08:57:35.637549+0000","last_fullsized":"2024-03-20T08:57:35.637549+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T08:56:57.309610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T08:56:57.309610+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:58:12.082766+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860436,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28208,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716944,"statfs":{"total":95995035648,"available":95966150656,"internally_reserved":0,"allocated":1216512,"data_stored":904828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":89,"seq":382252089368,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":684032,"data_stored":375900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":80,"seq":343597383711,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":684032,"data_stored":375900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":70,"seq":300647710756,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":684032,"data_stored":375920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":60,"seq":257698037801,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32356,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712796,"statfs":{"total":95995035648,"available":95961903104,"internally_reserved":0,"allocated":1204224,"data_stored":897471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332142,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28196,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716956,"statfs":{"total":95995035648,"available":95966162944,"internally_reserved":0,"allocated":1204224,"data_stored":897471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574920,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":671744,"data_stored":368543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738447,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27692,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717460,"statfs":{"total":95995035648,"available":95966679040,"internally_reserved":0,"allocated":684032,"data_stored":375920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411394,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717452,"statfs":{"total":95995035648,"available":95966670848,"internally_reserved":0,"allocated":696320,"data_stored":383297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149757,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":671744,"data_stored":368543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953528,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717476,"statfs":{"total":95995035648,"available":95966695424,"internally_reserved":0,"allocated":671744,"data_stored":368543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659187,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":684032,"data_stored":375920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T08:57:40.584 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-20T08:57:40.584 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-20T08:57:40.584 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-20T08:57:40.585 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph health --format=json 2024-03-20T08:57:40.861 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:57:41.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:41 smithi115 ceph-mon[72476]: pgmap v346: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:41.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:41 smithi115 ceph-mon[72476]: from='client.14667 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T08:57:41.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:41 smithi122 ceph-mon[73495]: pgmap v346: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:41.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:41 smithi122 ceph-mon[73495]: from='client.14667 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T08:57:41.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:41 smithi026 ceph-mon[68309]: pgmap v346: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:41.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:41 smithi026 ceph-mon[68309]: from='client.14667 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T08:57:42.868 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:57:42.869 INFO:teuthology.orchestra.run.smithi026.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-20T08:57:43.219 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:43 smithi026 ceph-mon[68309]: pgmap v347: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:43.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:43 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/91481119' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-20T08:57:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:43 smithi115 ceph-mon[72476]: pgmap v347: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:43 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/91481119' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-20T08:57:43.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:43 smithi122 ceph-mon[73495]: pgmap v347: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:43 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/91481119' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-20T08:57:43.674 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-20T08:57:43.674 INFO:tasks.cephadm:Setup complete, yielding 2024-03-20T08:57:43.674 INFO:teuthology.run_tasks:Running task thrashosds... 2024-03-20T08:57:43.687 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-20T08:57:43.687 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-yuri5-testing-2024-03-18-1144'}, 'ceph': {'conf': {'client.2': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MON_DOWN\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'sha1': 'e7e80a6ac9849f0f23432d0d9832f096730308ae'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'e7e80a6ac9849f0f23432d0d9832f096730308ae'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'wip-yuri5-testing-2024-03-18-1144', 'sha1': 'e7e80a6ac9849f0f23432d0d9832f096730308ae'}} 2024-03-20T08:57:43.687 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'min_in': 4} 2024-03-20T08:57:43.688 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-03-20T08:57:43.968 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:57:45.359 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:45 smithi026 ceph-mon[68309]: pgmap v348: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:45.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:45 smithi115 ceph-mon[72476]: pgmap v348: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:45.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:45 smithi122 ceph-mon[73495]: pgmap v348: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:46.519 INFO:tasks.thrashosds:Beginning thrashosds... 2024-03-20T08:57:46.519 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-20T08:57:46.519 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T08:57:46.793 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:57:47.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:46 smithi115 ceph-mon[72476]: pgmap v349: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:47.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:46 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3863328848' entity='client.admin' 2024-03-20T08:57:47.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:57:47.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:57:47.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:57:47.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:57:47.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:46 smithi122 ceph-mon[73495]: pgmap v349: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:47.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:46 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3863328848' entity='client.admin' 2024-03-20T08:57:47.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:57:47.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:57:47.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:57:47.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:57:47.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:46 smithi026 ceph-mon[68309]: pgmap v349: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:47.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:46 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3863328848' entity='client.admin' 2024-03-20T08:57:47.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:57:47.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:57:47.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:57:47.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:57:48.647 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:57:48.649 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T08:57:48.932 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:48 smithi026 ceph-mon[68309]: pgmap v350: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:49.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:48 smithi115 ceph-mon[72476]: pgmap v350: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:49.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:48 smithi122 ceph-mon[73495]: pgmap v350: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:49.236 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":350,"stamp":"2024-03-20T08:57:47.433239+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":338128,"kb_used_data":9720,"kb_used_omap":18,"kb_used_meta":328237,"kb_avail":1124603696,"statfs":{"total":1151940427776,"available":1151594184704,"internally_reserved":0,"allocated":9953280,"data_stored":6186288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":336115003},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.311141"},"pg_stats":[{"pgid":"1.0","version":"20'137","reported_seq":373,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-03-20T08:57:37.594389+0000","last_change":"2024-03-20T08:56:57.309733+0000","last_active":"2024-03-20T08:57:37.594389+0000","last_peered":"2024-03-20T08:57:37.594389+0000","last_clean":"2024-03-20T08:57:37.594389+0000","last_became_active":"2024-03-20T08:56:21.910528+0000","last_became_peered":"2024-03-20T08:56:21.910528+0000","last_unstale":"2024-03-20T08:57:37.594389+0000","last_undegraded":"2024-03-20T08:57:37.594389+0000","last_fullsized":"2024-03-20T08:57:37.594389+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T08:56:57.309610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T08:56:57.309610+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:58:12.082766+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860437,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28220,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716932,"statfs":{"total":95995035648,"available":95966138368,"internally_reserved":0,"allocated":1228800,"data_stored":912205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":89,"seq":382252089370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717452,"statfs":{"total":95995035648,"available":95966670848,"internally_reserved":0,"allocated":696320,"data_stored":383277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":80,"seq":343597383712,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717452,"statfs":{"total":95995035648,"available":95966670848,"internally_reserved":0,"allocated":696320,"data_stored":383277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":70,"seq":300647710758,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27704,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717448,"statfs":{"total":95995035648,"available":95966666752,"internally_reserved":0,"allocated":696320,"data_stored":383297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":60,"seq":257698037803,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32380,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712772,"statfs":{"total":95995035648,"available":95961878528,"internally_reserved":0,"allocated":1228800,"data_stored":912225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332144,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28220,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716932,"statfs":{"total":95995035648,"available":95966138368,"internally_reserved":0,"allocated":1228800,"data_stored":912225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574922,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27704,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717448,"statfs":{"total":95995035648,"available":95966666752,"internally_reserved":0,"allocated":696320,"data_stored":383297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738449,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717452,"statfs":{"total":95995035648,"available":95966670848,"internally_reserved":0,"allocated":696320,"data_stored":383297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411395,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717452,"statfs":{"total":95995035648,"available":95966670848,"internally_reserved":0,"allocated":696320,"data_stored":383297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149759,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717452,"statfs":{"total":95995035648,"available":95966670848,"internally_reserved":0,"allocated":696320,"data_stored":383297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953530,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717452,"statfs":{"total":95995035648,"available":95966670848,"internally_reserved":0,"allocated":696320,"data_stored":383297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659189,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717452,"statfs":{"total":95995035648,"available":95966670848,"internally_reserved":0,"allocated":696320,"data_stored":383297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T08:57:49.236 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T08:57:49.512 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:57:50.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:49 smithi115 ceph-mon[72476]: from='client.14685 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T08:57:50.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:49 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:57:50.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:49 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:57:50.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:49 smithi122 ceph-mon[73495]: from='client.14685 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T08:57:50.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:49 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:57:50.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:49 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:57:50.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:49 smithi026 ceph-mon[68309]: from='client.14685 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T08:57:50.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:49 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:57:50.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:49 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:57:51.045 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:50 smithi026 ceph-mon[68309]: pgmap v351: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:51.045 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:50 smithi026 ceph-mon[68309]: osdmap e115: 12 total, 12 up, 12 in 2024-03-20T08:57:51.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:50 smithi115 ceph-mon[72476]: pgmap v351: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:51.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:50 smithi115 ceph-mon[72476]: osdmap e115: 12 total, 12 up, 12 in 2024-03-20T08:57:51.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:50 smithi122 ceph-mon[73495]: pgmap v351: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:51.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:50 smithi122 ceph-mon[73495]: osdmap e115: 12 total, 12 up, 12 in 2024-03-20T08:57:51.508 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:57:51.511 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T08:57:52.139 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":353,"stamp":"2024-03-20T08:57:51.433853+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":338140,"kb_used_data":9732,"kb_used_omap":18,"kb_used_meta":328237,"kb_avail":1124603684,"statfs":{"total":1151940427776,"available":1151594172416,"internally_reserved":0,"allocated":9965568,"data_stored":6193665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":336115003},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001703"},"pg_stats":[{"pgid":"1.0","version":"20'137","reported_seq":375,"reported_epoch":115,"state":"active+clean","last_fresh":"2024-03-20T08:57:50.546338+0000","last_change":"2024-03-20T08:56:57.309733+0000","last_active":"2024-03-20T08:57:50.546338+0000","last_peered":"2024-03-20T08:57:50.546338+0000","last_clean":"2024-03-20T08:57:50.546338+0000","last_became_active":"2024-03-20T08:56:21.910528+0000","last_became_peered":"2024-03-20T08:56:21.910528+0000","last_unstale":"2024-03-20T08:57:50.546338+0000","last_undegraded":"2024-03-20T08:57:50.546338+0000","last_fullsized":"2024-03-20T08:57:50.546338+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T08:56:57.309610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T08:56:57.309610+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:58:12.082766+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860438,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28220,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716932,"statfs":{"total":95995035648,"available":95966138368,"internally_reserved":0,"allocated":1228800,"data_stored":912205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":89,"seq":382252089371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717440,"statfs":{"total":95995035648,"available":95966658560,"internally_reserved":0,"allocated":708608,"data_stored":390654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":80,"seq":343597383713,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717452,"statfs":{"total":95995035648,"available":95966670848,"internally_reserved":0,"allocated":696320,"data_stored":383277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":70,"seq":300647710758,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27704,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717448,"statfs":{"total":95995035648,"available":95966666752,"internally_reserved":0,"allocated":696320,"data_stored":383297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":60,"seq":257698037804,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32380,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712772,"statfs":{"total":95995035648,"available":95961878528,"internally_reserved":0,"allocated":1228800,"data_stored":912225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332145,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28220,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716932,"statfs":{"total":95995035648,"available":95966138368,"internally_reserved":0,"allocated":1228800,"data_stored":912225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574923,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27704,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717448,"statfs":{"total":95995035648,"available":95966666752,"internally_reserved":0,"allocated":696320,"data_stored":383297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738449,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717452,"statfs":{"total":95995035648,"available":95966670848,"internally_reserved":0,"allocated":696320,"data_stored":383297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411396,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717452,"statfs":{"total":95995035648,"available":95966670848,"internally_reserved":0,"allocated":696320,"data_stored":383297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149760,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717452,"statfs":{"total":95995035648,"available":95966670848,"internally_reserved":0,"allocated":696320,"data_stored":383297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953531,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717452,"statfs":{"total":95995035648,"available":95966670848,"internally_reserved":0,"allocated":696320,"data_stored":383297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659189,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717452,"statfs":{"total":95995035648,"available":95966670848,"internally_reserved":0,"allocated":696320,"data_stored":383297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T08:57:52.139 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-20T08:57:52.140 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd dump 2024-03-20T08:57:52.413 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:57:52.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:52 smithi026 ceph-mon[68309]: pgmap v353: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:52.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:52 smithi026 ceph-mon[68309]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T08:57:53.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:52 smithi115 ceph-mon[72476]: pgmap v353: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:53.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:52 smithi115 ceph-mon[72476]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T08:57:53.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:52 smithi122 ceph-mon[73495]: pgmap v353: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:53.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:52 smithi122 ceph-mon[73495]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T08:57:54.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:53 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:57:54.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:53 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:57:54.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:53 smithi115 ceph-mon[72476]: osdmap e116: 12 total, 12 up, 12 in 2024-03-20T08:57:54.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:53 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:57:54.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:53 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:57:54.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:53 smithi122 ceph-mon[73495]: osdmap e116: 12 total, 12 up, 12 in 2024-03-20T08:57:54.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:53 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:57:54.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:53 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:57:54.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:53 smithi026 ceph-mon[68309]: osdmap e116: 12 total, 12 up, 12 in 2024-03-20T08:57:54.238 INFO:teuthology.orchestra.run.smithi026.stdout:epoch 116 2024-03-20T08:57:54.239 INFO:teuthology.orchestra.run.smithi026.stdout:fsid 852fbb68-e696-11ee-95c9-87774f69a715 2024-03-20T08:57:54.239 INFO:teuthology.orchestra.run.smithi026.stdout:created 2024-03-20T08:48:35.281088+0000 2024-03-20T08:57:54.239 INFO:teuthology.orchestra.run.smithi026.stdout:modified 2024-03-20T08:57:52.856981+0000 2024-03-20T08:57:54.239 INFO:teuthology.orchestra.run.smithi026.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-20T08:57:54.239 INFO:teuthology.orchestra.run.smithi026.stdout:crush_version 26 2024-03-20T08:57:54.239 INFO:teuthology.orchestra.run.smithi026.stdout:full_ratio 0.95 2024-03-20T08:57:54.239 INFO:teuthology.orchestra.run.smithi026.stdout:backfillfull_ratio 0.9 2024-03-20T08:57:54.239 INFO:teuthology.orchestra.run.smithi026.stdout:nearfull_ratio 0.85 2024-03-20T08:57:54.239 INFO:teuthology.orchestra.run.smithi026.stdout:require_min_compat_client luminous 2024-03-20T08:57:54.239 INFO:teuthology.orchestra.run.smithi026.stdout:min_compat_client jewel 2024-03-20T08:57:54.239 INFO:teuthology.orchestra.run.smithi026.stdout:require_osd_release squid 2024-03-20T08:57:54.239 INFO:teuthology.orchestra.run.smithi026.stdout:stretch_mode_enabled false 2024-03-20T08:57:54.240 INFO:teuthology.orchestra.run.smithi026.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-20T08:57:54.240 INFO:teuthology.orchestra.run.smithi026.stdout:max_osd 12 2024-03-20T08:57:54.240 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0 up in weight 1 up_from 8 up_thru 62 down_at 0 last_clean_interval [0,0) [v2:172.21.15.26:6802/2139805293,v1:172.21.15.26:6803/2139805293] [v2:172.21.15.26:6804/2139805293,v1:172.21.15.26:6805/2139805293] exists,up ad3fd9c0-316f-4fe5-a63b-d480087c1729 2024-03-20T08:57:54.240 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1 up in weight 1 up_from 13 up_thru 35 down_at 0 last_clean_interval [0,0) [v2:172.21.15.26:6810/3853355134,v1:172.21.15.26:6811/3853355134] [v2:172.21.15.26:6812/3853355134,v1:172.21.15.26:6813/3853355134] exists,up e85b94f5-2954-47a1-9c20-096f2606eb04 2024-03-20T08:57:54.240 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.26:6818/3993357967,v1:172.21.15.26:6819/3993357967] [v2:172.21.15.26:6820/3993357967,v1:172.21.15.26:6821/3993357967] exists,up dd2b10f0-741c-47a4-b224-ac6bfe2c14ef 2024-03-20T08:57:54.240 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3 up in weight 1 up_from 26 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.26:6826/4110225509,v1:172.21.15.26:6827/4110225509] [v2:172.21.15.26:6828/4110225509,v1:172.21.15.26:6829/4110225509] exists,up 9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1 2024-03-20T08:57:54.240 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4 up in weight 1 up_from 32 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.115:6800/477923058,v1:172.21.15.115:6801/477923058] [v2:172.21.15.115:6802/477923058,v1:172.21.15.115:6803/477923058] exists,up f14e34d5-9829-42a3-9b25-31a0976c5ef7 2024-03-20T08:57:54.240 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5 up in weight 1 up_from 41 up_thru 44 down_at 0 last_clean_interval [0,0) [v2:172.21.15.115:6808/1339268003,v1:172.21.15.115:6809/1339268003] [v2:172.21.15.115:6810/1339268003,v1:172.21.15.115:6811/1339268003] exists,up cb431b6a-540d-4318-ae35-d379e1df0185 2024-03-20T08:57:54.240 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6 up in weight 1 up_from 51 up_thru 54 down_at 0 last_clean_interval [0,0) [v2:172.21.15.115:6816/3278301843,v1:172.21.15.115:6817/3278301843] [v2:172.21.15.115:6818/3278301843,v1:172.21.15.115:6819/3278301843] exists,up 753ed830-e764-4ff3-9d9b-87e7fa748b81 2024-03-20T08:57:54.240 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7 up in weight 1 up_from 60 up_thru 99 down_at 0 last_clean_interval [0,0) [v2:172.21.15.115:6824/3152424001,v1:172.21.15.115:6825/3152424001] [v2:172.21.15.115:6826/3152424001,v1:172.21.15.115:6827/3152424001] exists,up 55bcbd37-7a6f-4ab1-b5a9-91a951444d86 2024-03-20T08:57:54.240 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8 up in weight 1 up_from 70 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.122:6800/1309426740,v1:172.21.15.122:6801/1309426740] [v2:172.21.15.122:6802/1309426740,v1:172.21.15.122:6803/1309426740] exists,up bc86248e-6bd0-4369-b8a2-e593bc2a610d 2024-03-20T08:57:54.240 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9 up in weight 1 up_from 80 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.122:6808/2953210076,v1:172.21.15.122:6809/2953210076] [v2:172.21.15.122:6810/2953210076,v1:172.21.15.122:6811/2953210076] exists,up 9b20a4aa-032a-4c09-ac75-153c554424e7 2024-03-20T08:57:54.240 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10 up in weight 1 up_from 89 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.122:6816/409394340,v1:172.21.15.122:6817/409394340] [v2:172.21.15.122:6818/409394340,v1:172.21.15.122:6819/409394340] exists,up e62c1fb3-5dd1-4d0d-997c-d20633d19361 2024-03-20T08:57:54.241 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11 up in weight 1 up_from 96 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.122:6824/2704069354,v1:172.21.15.122:6825/2704069354] [v2:172.21.15.122:6826/2704069354,v1:172.21.15.122:6827/2704069354] exists,up b83994d5-e5ef-48b0-9758-c26843cec3b6 2024-03-20T08:57:54.241 INFO:teuthology.orchestra.run.smithi026.stdout:blocklist 172.21.15.26:0/4291404191 expires 2024-03-21T08:49:35.336921+0000 2024-03-20T08:57:54.241 INFO:teuthology.orchestra.run.smithi026.stdout:blocklist 172.21.15.26:0/642012566 expires 2024-03-21T08:49:35.336921+0000 2024-03-20T08:57:54.241 INFO:teuthology.orchestra.run.smithi026.stdout:blocklist 172.21.15.26:6801/1181459999 expires 2024-03-21T08:49:35.336921+0000 2024-03-20T08:57:54.241 INFO:teuthology.orchestra.run.smithi026.stdout:blocklist 172.21.15.26:0/3984513952 expires 2024-03-21T08:49:04.596754+0000 2024-03-20T08:57:54.241 INFO:teuthology.orchestra.run.smithi026.stdout:blocklist 172.21.15.26:6800/1654945498 expires 2024-03-21T08:49:04.596754+0000 2024-03-20T08:57:54.241 INFO:teuthology.orchestra.run.smithi026.stdout:blocklist 172.21.15.26:6800/1181459999 expires 2024-03-21T08:49:35.336921+0000 2024-03-20T08:57:54.241 INFO:teuthology.orchestra.run.smithi026.stdout:blocklist 172.21.15.26:0/137770981 expires 2024-03-21T08:49:04.596754+0000 2024-03-20T08:57:54.241 INFO:teuthology.orchestra.run.smithi026.stdout:blocklist 172.21.15.26:0/1451237762 expires 2024-03-21T08:49:35.336921+0000 2024-03-20T08:57:54.241 INFO:teuthology.orchestra.run.smithi026.stdout:blocklist 172.21.15.26:0/3938386631 expires 2024-03-21T08:49:04.596754+0000 2024-03-20T08:57:54.241 INFO:teuthology.orchestra.run.smithi026.stdout:blocklist 172.21.15.26:6801/1654945498 expires 2024-03-21T08:49:04.596754+0000 2024-03-20T08:57:54.877 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 62 down_at 0 last_clean_interval [0,0) [v2:172.21.15.26:6802/2139805293,v1:172.21.15.26:6803/2139805293] [v2:172.21.15.26:6804/2139805293,v1:172.21.15.26:6805/2139805293] exists,up ad3fd9c0-316f-4fe5-a63b-d480087c1729', 'osd.1 up in weight 1 up_from 13 up_thru 35 down_at 0 last_clean_interval [0,0) [v2:172.21.15.26:6810/3853355134,v1:172.21.15.26:6811/3853355134] [v2:172.21.15.26:6812/3853355134,v1:172.21.15.26:6813/3853355134] exists,up e85b94f5-2954-47a1-9c20-096f2606eb04', 'osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.26:6818/3993357967,v1:172.21.15.26:6819/3993357967] [v2:172.21.15.26:6820/3993357967,v1:172.21.15.26:6821/3993357967] exists,up dd2b10f0-741c-47a4-b224-ac6bfe2c14ef', 'osd.3 up in weight 1 up_from 26 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.26:6826/4110225509,v1:172.21.15.26:6827/4110225509] [v2:172.21.15.26:6828/4110225509,v1:172.21.15.26:6829/4110225509] exists,up 9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1', 'osd.4 up in weight 1 up_from 32 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.115:6800/477923058,v1:172.21.15.115:6801/477923058] [v2:172.21.15.115:6802/477923058,v1:172.21.15.115:6803/477923058] exists,up f14e34d5-9829-42a3-9b25-31a0976c5ef7', 'osd.5 up in weight 1 up_from 41 up_thru 44 down_at 0 last_clean_interval [0,0) [v2:172.21.15.115:6808/1339268003,v1:172.21.15.115:6809/1339268003] [v2:172.21.15.115:6810/1339268003,v1:172.21.15.115:6811/1339268003] exists,up cb431b6a-540d-4318-ae35-d379e1df0185', 'osd.6 up in weight 1 up_from 51 up_thru 54 down_at 0 last_clean_interval [0,0) [v2:172.21.15.115:6816/3278301843,v1:172.21.15.115:6817/3278301843] [v2:172.21.15.115:6818/3278301843,v1:172.21.15.115:6819/3278301843] exists,up 753ed830-e764-4ff3-9d9b-87e7fa748b81', 'osd.7 up in weight 1 up_from 60 up_thru 99 down_at 0 last_clean_interval [0,0) [v2:172.21.15.115:6824/3152424001,v1:172.21.15.115:6825/3152424001] [v2:172.21.15.115:6826/3152424001,v1:172.21.15.115:6827/3152424001] exists,up 55bcbd37-7a6f-4ab1-b5a9-91a951444d86', 'osd.8 up in weight 1 up_from 70 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.122:6800/1309426740,v1:172.21.15.122:6801/1309426740] [v2:172.21.15.122:6802/1309426740,v1:172.21.15.122:6803/1309426740] exists,up bc86248e-6bd0-4369-b8a2-e593bc2a610d', 'osd.9 up in weight 1 up_from 80 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.122:6808/2953210076,v1:172.21.15.122:6809/2953210076] [v2:172.21.15.122:6810/2953210076,v1:172.21.15.122:6811/2953210076] exists,up 9b20a4aa-032a-4c09-ac75-153c554424e7', 'osd.10 up in weight 1 up_from 89 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.122:6816/409394340,v1:172.21.15.122:6817/409394340] [v2:172.21.15.122:6818/409394340,v1:172.21.15.122:6819/409394340] exists,up e62c1fb3-5dd1-4d0d-997c-d20633d19361', 'osd.11 up in weight 1 up_from 96 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.122:6824/2704069354,v1:172.21.15.122:6825/2704069354] [v2:172.21.15.122:6826/2704069354,v1:172.21.15.122:6827/2704069354] exists,up b83994d5-e5ef-48b0-9758-c26843cec3b6'] 2024-03-20T08:57:54.877 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon mon.a config show 2024-03-20T08:57:55.153 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:57:55.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:54 smithi115 ceph-mon[72476]: pgmap v355: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:55.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:54 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1623245531' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-20T08:57:55.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:54 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:57:55.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:54 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:57:55.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:54 smithi122 ceph-mon[73495]: pgmap v355: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:55.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:54 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1623245531' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-20T08:57:55.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:54 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:57:55.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:54 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:57:55.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:54 smithi026 ceph-mon[68309]: pgmap v355: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:55.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:54 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1623245531' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-20T08:57:55.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:54 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:57:55.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:54 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:57:56.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:55 smithi115 ceph-mon[72476]: osdmap e117: 12 total, 12 up, 12 in 2024-03-20T08:57:56.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:55 smithi122 ceph-mon[73495]: osdmap e117: 12 total, 12 up, 12 in 2024-03-20T08:57:56.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:55 smithi026 ceph-mon[68309]: osdmap e117: 12 total, 12 up, 12 in 2024-03-20T08:57:56.870 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T08:57:56.870 INFO:teuthology.orchestra.run.smithi026.stdout: "name": "mon.a", 2024-03-20T08:57:56.870 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster": "ceph", 2024-03-20T08:57:56.870 INFO:teuthology.orchestra.run.smithi026.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-03-20T08:57:56.870 INFO:teuthology.orchestra.run.smithi026.stdout: "admin_socket_mode": "", 2024-03-20T08:57:56.870 INFO:teuthology.orchestra.run.smithi026.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-03-20T08:57:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: "auth_client_required": "cephx, none", 2024-03-20T08:57:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: "auth_cluster_required": "cephx", 2024-03-20T08:57:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: "auth_debug": "true", 2024-03-20T08:57:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-03-20T08:57:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-03-20T08:57:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: "auth_service_required": "cephx", 2024-03-20T08:57:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: "auth_service_ticket_ttl": "240.000000", 2024-03-20T08:57:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: "auth_supported": "", 2024-03-20T08:57:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_aio": "true", 2024-03-20T08:57:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_aio_max_queue_depth": "1024", 2024-03-20T08:57:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_aio_poll_ms": "250", 2024-03-20T08:57:56.871 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_aio_reap_max": "16", 2024-03-20T08:57:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_async_discard_threads": "0", 2024-03-20T08:57:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_block_size": "4096", 2024-03-20T08:57:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_debug_aio": "false", 2024-03-20T08:57:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-03-20T08:57:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-03-20T08:57:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_debug_inflight_ios": "false", 2024-03-20T08:57:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_enable_discard": "false", 2024-03-20T08:57:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_flock_retry": "3", 2024-03-20T08:57:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_flock_retry_interval": "0.100000", 2024-03-20T08:57:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_inject_crash": "0", 2024-03-20T08:57:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_inject_crash_flush_delay": "2", 2024-03-20T08:57:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_ioring": "false", 2024-03-20T08:57:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_ioring_hipri": "false", 2024-03-20T08:57:56.872 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_ioring_sqthread_poll": "false", 2024-03-20T08:57:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-03-20T08:57:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_read_buffer_alignment": "4096", 2024-03-20T08:57:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-03-20T08:57:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: "bdev_type": "", 2024-03-20T08:57:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_alloc_size": "1048576", 2024-03-20T08:57:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_allocator": "hybrid", 2024-03-20T08:57:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_buffered_io": "true", 2024-03-20T08:57:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_check_for_zeros": "false", 2024-03-20T08:57:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_check_volume_selector_often": "false", 2024-03-20T08:57:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-03-20T08:57:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_compact_log_sync": "false", 2024-03-20T08:57:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-03-20T08:57:56.873 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-03-20T08:57:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_log_compact_min_size": "16777216", 2024-03-20T08:57:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_log_replay_check_allocations": "true", 2024-03-20T08:57:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_max_log_runway": "4194304", 2024-03-20T08:57:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_max_prefetch": "1048576", 2024-03-20T08:57:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_min_flush_size": "524288", 2024-03-20T08:57:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_min_log_runway": "1048576", 2024-03-20T08:57:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_replay_recovery": "false", 2024-03-20T08:57:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-03-20T08:57:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_shared_alloc_size": "65536", 2024-03-20T08:57:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: "bluefs_sync_write": "false", 2024-03-20T08:57:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-03-20T08:57:56.874 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-03-20T08:57:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-03-20T08:57:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_allocation_from_file": "true", 2024-03-20T08:57:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_allocator": "hybrid", 2024-03-20T08:57:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-03-20T08:57:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-03-20T08:57:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-03-20T08:57:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-03-20T08:57:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-03-20T08:57:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-03-20T08:57:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_blobid_prealloc": "10240", 2024-03-20T08:57:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_block_create": "true", 2024-03-20T08:57:56.875 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_block_db_create": "false", 2024-03-20T08:57:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_block_db_path": "", 2024-03-20T08:57:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_block_db_size": "0", 2024-03-20T08:57:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_block_path": "", 2024-03-20T08:57:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_block_preallocate_file": "false", 2024-03-20T08:57:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_block_size": "107374182400", 2024-03-20T08:57:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_block_wal_create": "false", 2024-03-20T08:57:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_block_wal_path": "", 2024-03-20T08:57:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_block_wal_size": "100663296", 2024-03-20T08:57:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_bluefs": "true", 2024-03-20T08:57:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-03-20T08:57:56.876 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_bluefs_env_mirror": "false", 2024-03-20T08:57:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-03-20T08:57:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-03-20T08:57:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-03-20T08:57:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-03-20T08:57:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-03-20T08:57:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-03-20T08:57:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_autotune": "true", 2024-03-20T08:57:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-03-20T08:57:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-03-20T08:57:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-03-20T08:57:56.877 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-03-20T08:57:56.878 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_size": "0", 2024-03-20T08:57:56.878 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-03-20T08:57:56.878 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-03-20T08:57:56.878 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-03-20T08:57:56.878 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-03-20T08:57:56.878 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cache_type": "2q", 2024-03-20T08:57:56.878 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-03-20T08:57:56.878 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_clone_cow": "true", 2024-03-20T08:57:56.879 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_compression_algorithm": "snappy", 2024-03-20T08:57:56.879 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_compression_max_blob_size": "0", 2024-03-20T08:57:56.879 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-03-20T08:57:56.879 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-03-20T08:57:56.879 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_compression_min_blob_size": "0", 2024-03-20T08:57:56.879 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-03-20T08:57:56.879 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-03-20T08:57:56.879 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_compression_mode": "none", 2024-03-20T08:57:56.879 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-03-20T08:57:56.879 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_csum_type": "crc32c", 2024-03-20T08:57:56.879 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-03-20T08:57:56.880 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_debug_enforce_settings": "default", 2024-03-20T08:57:56.880 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_debug_freelist": "false", 2024-03-20T08:57:56.880 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_debug_fsck_abort": "false", 2024-03-20T08:57:56.880 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-03-20T08:57:56.880 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-03-20T08:57:56.880 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_debug_inject_read_err": "false", 2024-03-20T08:57:56.880 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_debug_legacy_omap": "false", 2024-03-20T08:57:56.880 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-03-20T08:57:56.880 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-03-20T08:57:56.880 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-03-20T08:57:56.880 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-03-20T08:57:56.880 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_debug_prefill": "0.000000", 2024-03-20T08:57:56.880 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-03-20T08:57:56.881 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-03-20T08:57:56.881 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-03-20T08:57:56.881 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_debug_small_allocations": "0", 2024-03-20T08:57:56.881 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-03-20T08:57:56.881 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_default_buffered_read": "true", 2024-03-20T08:57:56.881 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_default_buffered_write": "false", 2024-03-20T08:57:56.881 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_deferred_batch_ops": "0", 2024-03-20T08:57:56.881 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-03-20T08:57:56.881 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-03-20T08:57:56.881 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_elastic_shared_blobs": "true", 2024-03-20T08:57:56.881 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-03-20T08:57:56.881 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-03-20T08:57:56.882 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-03-20T08:57:56.882 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-03-20T08:57:56.882 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-03-20T08:57:56.882 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_fail_eio": "false", 2024-03-20T08:57:56.882 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-03-20T08:57:56.883 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-03-20T08:57:56.883 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-03-20T08:57:56.884 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-03-20T08:57:56.884 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_fsck_on_mkfs": "true", 2024-03-20T08:57:56.884 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-03-20T08:57:56.884 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_fsck_on_mount": "false", 2024-03-20T08:57:56.884 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-03-20T08:57:56.884 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_fsck_on_umount": "false", 2024-03-20T08:57:56.884 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-03-20T08:57:56.885 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-03-20T08:57:56.885 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-03-20T08:57:56.885 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-03-20T08:57:56.885 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-03-20T08:57:56.885 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-03-20T08:57:56.885 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-03-20T08:57:56.885 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-03-20T08:57:56.885 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_ignore_data_csum": "false", 2024-03-20T08:57:56.885 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-03-20T08:57:56.885 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_kvbackend": "rocksdb", 2024-03-20T08:57:56.886 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-03-20T08:57:56.886 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-03-20T08:57:56.886 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_log_op_age": "5.000000", 2024-03-20T08:57:56.886 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_max_alloc_size": "0", 2024-03-20T08:57:56.886 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_max_blob_size": "0", 2024-03-20T08:57:56.886 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-03-20T08:57:56.886 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-03-20T08:57:56.886 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_max_defer_interval": "3.000000", 2024-03-20T08:57:56.886 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_max_deferred_txc": "32", 2024-03-20T08:57:56.886 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_min_alloc_size": "0", 2024-03-20T08:57:56.887 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-03-20T08:57:56.887 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-03-20T08:57:56.887 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_nid_prealloc": "1024", 2024-03-20T08:57:56.887 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_prefer_deferred_size": "0", 2024-03-20T08:57:56.887 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-03-20T08:57:56.889 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-03-20T08:57:56.889 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_qfsck_on_mount": "true", 2024-03-20T08:57:56.889 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_retry_disk_reads": "3", 2024-03-20T08:57:56.889 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_rocksdb_cf": "true", 2024-03-20T08:57:56.889 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-03-20T08:57:56.889 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-03-20T08:57:56.890 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_rocksdb_options_annex": "", 2024-03-20T08:57:56.890 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_spdk_coremask": "0x1", 2024-03-20T08:57:56.890 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_spdk_io_sleep": "5", 2024-03-20T08:57:56.890 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_spdk_max_io_completion": "0", 2024-03-20T08:57:56.890 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_spdk_mem": "512", 2024-03-20T08:57:56.890 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_sync_submit_transaction": "false", 2024-03-20T08:57:56.890 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_throttle_bytes": "67108864", 2024-03-20T08:57:56.890 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_throttle_cost_per_io": "0", 2024-03-20T08:57:56.890 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-03-20T08:57:56.890 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-03-20T08:57:56.890 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-03-20T08:57:56.890 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-03-20T08:57:56.890 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_tracing": "false", 2024-03-20T08:57:56.890 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-03-20T08:57:56.891 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-03-20T08:57:56.891 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_volume_selection_reserved": "0", 2024-03-20T08:57:56.891 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-03-20T08:57:56.891 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-03-20T08:57:56.891 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-03-20T08:57:56.891 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-03-20T08:57:56.891 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-03-20T08:57:56.891 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-03-20T08:57:56.891 INFO:teuthology.orchestra.run.smithi026.stdout: "bluestore_zero_block_detection": "false", 2024-03-20T08:57:56.891 INFO:teuthology.orchestra.run.smithi026.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-03-20T08:57:56.891 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_mirror_action_update_interval": "2", 2024-03-20T08:57:56.891 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-03-20T08:57:56.891 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-03-20T08:57:56.891 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-03-20T08:57:56.892 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-03-20T08:57:56.892 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_mirror_mount_timeout": "10", 2024-03-20T08:57:56.892 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-03-20T08:57:56.892 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-03-20T08:57:56.892 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-03-20T08:57:56.892 INFO:teuthology.orchestra.run.smithi026.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-03-20T08:57:56.892 INFO:teuthology.orchestra.run.smithi026.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-03-20T08:57:56.892 INFO:teuthology.orchestra.run.smithi026.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-03-20T08:57:56.892 INFO:teuthology.orchestra.run.smithi026.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-03-20T08:57:56.892 INFO:teuthology.orchestra.run.smithi026.stdout: "cephx_cluster_require_signatures": "false", 2024-03-20T08:57:56.892 INFO:teuthology.orchestra.run.smithi026.stdout: "cephx_cluster_require_version": "2", 2024-03-20T08:57:56.892 INFO:teuthology.orchestra.run.smithi026.stdout: "cephx_require_signatures": "false", 2024-03-20T08:57:56.893 INFO:teuthology.orchestra.run.smithi026.stdout: "cephx_require_version": "2", 2024-03-20T08:57:56.893 INFO:teuthology.orchestra.run.smithi026.stdout: "cephx_service_require_signatures": "false", 2024-03-20T08:57:56.893 INFO:teuthology.orchestra.run.smithi026.stdout: "cephx_service_require_version": "2", 2024-03-20T08:57:56.893 INFO:teuthology.orchestra.run.smithi026.stdout: "cephx_sign_messages": "true", 2024-03-20T08:57:56.893 INFO:teuthology.orchestra.run.smithi026.stdout: "chdir": "", 2024-03-20T08:57:56.893 INFO:teuthology.orchestra.run.smithi026.stdout: "client_acl_type": "", 2024-03-20T08:57:56.893 INFO:teuthology.orchestra.run.smithi026.stdout: "client_asio_thread_count": "2", 2024-03-20T08:57:56.893 INFO:teuthology.orchestra.run.smithi026.stdout: "client_cache_mid": "0.750000", 2024-03-20T08:57:56.893 INFO:teuthology.orchestra.run.smithi026.stdout: "client_cache_size": "16384", 2024-03-20T08:57:56.893 INFO:teuthology.orchestra.run.smithi026.stdout: "client_caps_release_delay": "5", 2024-03-20T08:57:56.893 INFO:teuthology.orchestra.run.smithi026.stdout: "client_check_pool_perm": "true", 2024-03-20T08:57:56.893 INFO:teuthology.orchestra.run.smithi026.stdout: "client_collect_and_send_global_metrics": "false", 2024-03-20T08:57:56.893 INFO:teuthology.orchestra.run.smithi026.stdout: "client_debug_force_sync_read": "false", 2024-03-20T08:57:56.894 INFO:teuthology.orchestra.run.smithi026.stdout: "client_debug_getattr_caps": "false", 2024-03-20T08:57:56.894 INFO:teuthology.orchestra.run.smithi026.stdout: "client_debug_inject_tick_delay": "0", 2024-03-20T08:57:56.894 INFO:teuthology.orchestra.run.smithi026.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-03-20T08:57:56.894 INFO:teuthology.orchestra.run.smithi026.stdout: "client_die_on_failed_remount": "false", 2024-03-20T08:57:56.894 INFO:teuthology.orchestra.run.smithi026.stdout: "client_dirsize_rbytes": "true", 2024-03-20T08:57:56.894 INFO:teuthology.orchestra.run.smithi026.stdout: "client_force_lazyio": "false", 2024-03-20T08:57:56.894 INFO:teuthology.orchestra.run.smithi026.stdout: "client_fs": "", 2024-03-20T08:57:56.894 INFO:teuthology.orchestra.run.smithi026.stdout: "client_inject_fixed_oldest_tid": "false", 2024-03-20T08:57:56.894 INFO:teuthology.orchestra.run.smithi026.stdout: "client_inject_release_failure": "false", 2024-03-20T08:57:56.894 INFO:teuthology.orchestra.run.smithi026.stdout: "client_max_inline_size": "4096", 2024-03-20T08:57:56.894 INFO:teuthology.orchestra.run.smithi026.stdout: "client_max_retries_on_remount_failure": "5", 2024-03-20T08:57:56.894 INFO:teuthology.orchestra.run.smithi026.stdout: "client_mds_namespace": "", 2024-03-20T08:57:56.894 INFO:teuthology.orchestra.run.smithi026.stdout: "client_metadata": "", 2024-03-20T08:57:56.895 INFO:teuthology.orchestra.run.smithi026.stdout: "client_mount_gid": "-1", 2024-03-20T08:57:56.895 INFO:teuthology.orchestra.run.smithi026.stdout: "client_mount_timeout": "300", 2024-03-20T08:57:56.895 INFO:teuthology.orchestra.run.smithi026.stdout: "client_mount_uid": "-1", 2024-03-20T08:57:56.895 INFO:teuthology.orchestra.run.smithi026.stdout: "client_mountpoint": "/", 2024-03-20T08:57:56.895 INFO:teuthology.orchestra.run.smithi026.stdout: "client_notify_timeout": "10", 2024-03-20T08:57:56.895 INFO:teuthology.orchestra.run.smithi026.stdout: "client_oc": "true", 2024-03-20T08:57:56.895 INFO:teuthology.orchestra.run.smithi026.stdout: "client_oc_max_dirty": "104857600", 2024-03-20T08:57:56.895 INFO:teuthology.orchestra.run.smithi026.stdout: "client_oc_max_dirty_age": "5.000000", 2024-03-20T08:57:56.895 INFO:teuthology.orchestra.run.smithi026.stdout: "client_oc_max_objects": "1000", 2024-03-20T08:57:56.895 INFO:teuthology.orchestra.run.smithi026.stdout: "client_oc_size": "209715200", 2024-03-20T08:57:56.895 INFO:teuthology.orchestra.run.smithi026.stdout: "client_oc_target_dirty": "8388608", 2024-03-20T08:57:56.895 INFO:teuthology.orchestra.run.smithi026.stdout: "client_permissions": "true", 2024-03-20T08:57:56.896 INFO:teuthology.orchestra.run.smithi026.stdout: "client_quota": "true", 2024-03-20T08:57:56.896 INFO:teuthology.orchestra.run.smithi026.stdout: "client_quota_df": "true", 2024-03-20T08:57:56.896 INFO:teuthology.orchestra.run.smithi026.stdout: "client_readahead_max_bytes": "0", 2024-03-20T08:57:56.896 INFO:teuthology.orchestra.run.smithi026.stdout: "client_readahead_max_periods": "4", 2024-03-20T08:57:56.896 INFO:teuthology.orchestra.run.smithi026.stdout: "client_readahead_min": "131072", 2024-03-20T08:57:56.896 INFO:teuthology.orchestra.run.smithi026.stdout: "client_reconnect_stale": "false", 2024-03-20T08:57:56.896 INFO:teuthology.orchestra.run.smithi026.stdout: "client_shutdown_timeout": "30", 2024-03-20T08:57:56.896 INFO:teuthology.orchestra.run.smithi026.stdout: "client_snapdir": ".snap", 2024-03-20T08:57:56.896 INFO:teuthology.orchestra.run.smithi026.stdout: "client_tick_interval": "1", 2024-03-20T08:57:56.896 INFO:teuthology.orchestra.run.smithi026.stdout: "client_trace": "", 2024-03-20T08:57:56.896 INFO:teuthology.orchestra.run.smithi026.stdout: "client_try_dentry_invalidate": "false", 2024-03-20T08:57:56.896 INFO:teuthology.orchestra.run.smithi026.stdout: "client_use_faked_inos": "false", 2024-03-20T08:57:56.896 INFO:teuthology.orchestra.run.smithi026.stdout: "client_use_random_mds": "false", 2024-03-20T08:57:56.896 INFO:teuthology.orchestra.run.smithi026.stdout: "clog_to_graylog": "false", 2024-03-20T08:57:56.897 INFO:teuthology.orchestra.run.smithi026.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-03-20T08:57:56.897 INFO:teuthology.orchestra.run.smithi026.stdout: "clog_to_graylog_port": "12201", 2024-03-20T08:57:56.897 INFO:teuthology.orchestra.run.smithi026.stdout: "clog_to_monitors": "default=true", 2024-03-20T08:57:56.897 INFO:teuthology.orchestra.run.smithi026.stdout: "clog_to_syslog": "false", 2024-03-20T08:57:56.897 INFO:teuthology.orchestra.run.smithi026.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-03-20T08:57:56.897 INFO:teuthology.orchestra.run.smithi026.stdout: "clog_to_syslog_level": "info", 2024-03-20T08:57:56.897 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "-", 2024-03-20T08:57:56.897 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_network": "", 2024-03-20T08:57:56.897 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_network_interface": "", 2024-03-20T08:57:56.897 INFO:teuthology.orchestra.run.smithi026.stdout: "compressor_zlib_isal": "false", 2024-03-20T08:57:56.897 INFO:teuthology.orchestra.run.smithi026.stdout: "compressor_zlib_level": "5", 2024-03-20T08:57:56.897 INFO:teuthology.orchestra.run.smithi026.stdout: "compressor_zlib_winsize": "-15", 2024-03-20T08:57:56.897 INFO:teuthology.orchestra.run.smithi026.stdout: "compressor_zstd_level": "1", 2024-03-20T08:57:56.898 INFO:teuthology.orchestra.run.smithi026.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:976a8b2d06e3c6bfd12488297e8584dd704cc06c3e976c7e0f718c13e238e912", 2024-03-20T08:57:56.898 INFO:teuthology.orchestra.run.smithi026.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-03-20T08:57:56.898 INFO:teuthology.orchestra.run.smithi026.stdout: "crimson_alien_op_num_threads": "6", 2024-03-20T08:57:56.898 INFO:teuthology.orchestra.run.smithi026.stdout: "crimson_alien_thread_cpu_cores": "", 2024-03-20T08:57:56.898 INFO:teuthology.orchestra.run.smithi026.stdout: "crimson_osd_obc_lru_size": "512", 2024-03-20T08:57:56.898 INFO:teuthology.orchestra.run.smithi026.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-03-20T08:57:56.898 INFO:teuthology.orchestra.run.smithi026.stdout: "crimson_seastar_cpu_cores": "", 2024-03-20T08:57:56.898 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location": "", 2024-03-20T08:57:56.898 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location_hook": "", 2024-03-20T08:57:56.898 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_location_hook_timeout": "10", 2024-03-20T08:57:56.898 INFO:teuthology.orchestra.run.smithi026.stdout: "daemonize": "false", 2024-03-20T08:57:56.898 INFO:teuthology.orchestra.run.smithi026.stdout: "daos_pool": "tank", 2024-03-20T08:57:56.898 INFO:teuthology.orchestra.run.smithi026.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-03-20T08:57:56.899 INFO:teuthology.orchestra.run.smithi026.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-03-20T08:57:56.899 INFO:teuthology.orchestra.run.smithi026.stdout: "dbstore_db_name_prefix": "dbstore", 2024-03-20T08:57:56.899 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_alienstore": "0/5", 2024-03-20T08:57:56.899 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_allow_any_pool_priority": "false", 2024-03-20T08:57:56.899 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_asok": "1/5", 2024-03-20T08:57:56.899 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_asok_assert_abort": "false", 2024-03-20T08:57:56.899 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_asserts_on_shutdown": "true", 2024-03-20T08:57:56.899 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_auth": "1/5", 2024-03-20T08:57:56.899 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_bdev": "1/3", 2024-03-20T08:57:56.899 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_bluefs": "1/5", 2024-03-20T08:57:56.899 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_bluestore": "1/5", 2024-03-20T08:57:56.899 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_buffer": "0/1", 2024-03-20T08:57:56.899 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_ceph_exporter": "1/5", 2024-03-20T08:57:56.900 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_cephfs_mirror": "0/5", 2024-03-20T08:57:56.900 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_cephsqlite": "0/5", 2024-03-20T08:57:56.900 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_client": "0/5", 2024-03-20T08:57:56.900 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_compressor": "1/5", 2024-03-20T08:57:56.900 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_context": "0/1", 2024-03-20T08:57:56.900 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_crimson_interrupt": "0/5", 2024-03-20T08:57:56.900 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_crush": "1/1", 2024-03-20T08:57:56.900 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_crypto": "1/5", 2024-03-20T08:57:56.900 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_cyanstore": "0/5", 2024-03-20T08:57:56.900 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_deliberately_leak_memory": "false", 2024-03-20T08:57:56.900 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_disable_randomized_ping": "false", 2024-03-20T08:57:56.900 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_dpdk": "1/5", 2024-03-20T08:57:56.900 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_eventtrace": "1/5", 2024-03-20T08:57:56.900 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_filer": "0/1", 2024-03-20T08:57:56.901 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_filestore": "1/3", 2024-03-20T08:57:56.901 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_finisher": "1/1", 2024-03-20T08:57:56.901 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_fuse": "1/5", 2024-03-20T08:57:56.901 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_heartbeat_testing_span": "0", 2024-03-20T08:57:56.901 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_heartbeatmap": "1/5", 2024-03-20T08:57:56.901 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_immutable_obj_cache": "0/5", 2024-03-20T08:57:56.901 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_javaclient": "1/5", 2024-03-20T08:57:56.901 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_journal": "1/3", 2024-03-20T08:57:56.901 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_journaler": "0/5", 2024-03-20T08:57:56.901 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_kstore": "1/5", 2024-03-20T08:57:56.901 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_lockdep": "0/1", 2024-03-20T08:57:56.901 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_mclock": "1/5", 2024-03-20T08:57:56.901 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_mds": "1/5", 2024-03-20T08:57:56.901 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_mds_balancer": "1/5", 2024-03-20T08:57:56.902 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_mds_locker": "1/5", 2024-03-20T08:57:56.902 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_mds_log": "1/5", 2024-03-20T08:57:56.902 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_mds_log_expire": "1/5", 2024-03-20T08:57:56.902 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_mds_migrator": "1/5", 2024-03-20T08:57:56.902 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_mds_quiesce": "3/5", 2024-03-20T08:57:56.902 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_memstore": "1/5", 2024-03-20T08:57:56.902 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_mgr": "2/5", 2024-03-20T08:57:56.902 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_mgrc": "1/5", 2024-03-20T08:57:56.902 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_mon": "20/20", 2024-03-20T08:57:56.902 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_monc": "0/10", 2024-03-20T08:57:56.902 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_ms": "1/1", 2024-03-20T08:57:56.902 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_none": "0/5", 2024-03-20T08:57:56.902 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_objclass": "0/5", 2024-03-20T08:57:56.902 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_objectcacher": "0/5", 2024-03-20T08:57:56.903 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_objecter": "0/1", 2024-03-20T08:57:56.903 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_optracker": "0/5", 2024-03-20T08:57:56.903 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_osd": "1/5", 2024-03-20T08:57:56.903 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_paxos": "20/20", 2024-03-20T08:57:56.903 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_perfcounter": "1/5", 2024-03-20T08:57:56.903 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_prioritycache": "1/5", 2024-03-20T08:57:56.903 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_rados": "0/5", 2024-03-20T08:57:56.903 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_rbd": "0/5", 2024-03-20T08:57:56.903 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_rbd_mirror": "0/5", 2024-03-20T08:57:56.903 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_rbd_pwl": "0/5", 2024-03-20T08:57:56.903 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_rbd_replay": "0/5", 2024-03-20T08:57:56.903 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_refs": "0/0", 2024-03-20T08:57:56.903 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_reserver": "1/1", 2024-03-20T08:57:56.903 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_rgw": "1/5", 2024-03-20T08:57:56.904 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_rgw_access": "1/5", 2024-03-20T08:57:56.904 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_rgw_datacache": "1/5", 2024-03-20T08:57:56.904 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_rgw_dbstore": "1/5", 2024-03-20T08:57:56.904 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_rgw_flight": "1/5", 2024-03-20T08:57:56.904 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_rgw_lifecycle": "1/5", 2024-03-20T08:57:56.904 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_rgw_sync": "1/5", 2024-03-20T08:57:56.904 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_rocksdb": "4/5", 2024-03-20T08:57:56.904 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_seastore": "0/5", 2024-03-20T08:57:56.904 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_seastore_backref": "0/5", 2024-03-20T08:57:56.904 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_seastore_cache": "0/5", 2024-03-20T08:57:56.904 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_seastore_cleaner": "0/5", 2024-03-20T08:57:56.904 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_seastore_device": "0/5", 2024-03-20T08:57:56.904 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_seastore_epm": "0/5", 2024-03-20T08:57:56.904 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-03-20T08:57:56.905 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_seastore_journal": "0/5", 2024-03-20T08:57:56.905 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_seastore_lba": "0/5", 2024-03-20T08:57:56.905 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_seastore_odata": "0/5", 2024-03-20T08:57:56.905 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_seastore_omap": "0/5", 2024-03-20T08:57:56.905 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_seastore_onode": "0/5", 2024-03-20T08:57:56.905 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_seastore_t": "0/5", 2024-03-20T08:57:56.905 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_seastore_tm": "0/5", 2024-03-20T08:57:56.905 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_striper": "0/1", 2024-03-20T08:57:56.905 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_test": "0/5", 2024-03-20T08:57:56.905 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_throttle": "1/1", 2024-03-20T08:57:56.905 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_timer": "0/1", 2024-03-20T08:57:56.905 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_tp": "0/5", 2024-03-20T08:57:56.905 INFO:teuthology.orchestra.run.smithi026.stdout: "debug_trace": "1/5", 2024-03-20T08:57:56.905 INFO:teuthology.orchestra.run.smithi026.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-03-20T08:57:56.906 INFO:teuthology.orchestra.run.smithi026.stdout: "device_failure_prediction_mode": "none", 2024-03-20T08:57:56.906 INFO:teuthology.orchestra.run.smithi026.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-03-20T08:57:56.906 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-03-20T08:57:56.906 INFO:teuthology.orchestra.run.smithi026.stdout: "err_to_graylog": "false", 2024-03-20T08:57:56.906 INFO:teuthology.orchestra.run.smithi026.stdout: "err_to_journald": "false", 2024-03-20T08:57:56.906 INFO:teuthology.orchestra.run.smithi026.stdout: "err_to_stderr": "true", 2024-03-20T08:57:56.906 INFO:teuthology.orchestra.run.smithi026.stdout: "err_to_syslog": "false", 2024-03-20T08:57:56.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event_tracing": "false", 2024-03-20T08:57:56.906 INFO:teuthology.orchestra.run.smithi026.stdout: "exporter_addr": "0.0.0.0", 2024-03-20T08:57:56.906 INFO:teuthology.orchestra.run.smithi026.stdout: "exporter_http_port": "9926", 2024-03-20T08:57:56.906 INFO:teuthology.orchestra.run.smithi026.stdout: "exporter_prio_limit": "5", 2024-03-20T08:57:56.906 INFO:teuthology.orchestra.run.smithi026.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-03-20T08:57:56.906 INFO:teuthology.orchestra.run.smithi026.stdout: "exporter_sort_metrics": "true", 2024-03-20T08:57:56.906 INFO:teuthology.orchestra.run.smithi026.stdout: "exporter_stats_period": "5", 2024-03-20T08:57:56.907 INFO:teuthology.orchestra.run.smithi026.stdout: "fake_statfs_for_testing": "0", 2024-03-20T08:57:56.907 INFO:teuthology.orchestra.run.smithi026.stdout: "fatal_signal_handlers": "true", 2024-03-20T08:57:56.907 INFO:teuthology.orchestra.run.smithi026.stdout: "filer_max_purge_ops": "10", 2024-03-20T08:57:56.907 INFO:teuthology.orchestra.run.smithi026.stdout: "filer_max_truncate_ops": "128", 2024-03-20T08:57:56.907 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_apply_finisher_threads": "1", 2024-03-20T08:57:56.907 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_blackhole": "false", 2024-03-20T08:57:56.907 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_btrfs_clone_range": "true", 2024-03-20T08:57:56.907 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_btrfs_snap": "true", 2024-03-20T08:57:56.907 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_caller_concurrency": "10", 2024-03-20T08:57:56.907 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_collect_device_partition_information": "true", 2024-03-20T08:57:56.907 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_commit_timeout": "600.000000", 2024-03-20T08:57:56.907 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_debug_inject_read_err": "false", 2024-03-20T08:57:56.907 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_debug_omap_check": "false", 2024-03-20T08:57:56.907 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_debug_random_read_err": "0.000000", 2024-03-20T08:57:56.908 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_debug_verify_split": "false", 2024-03-20T08:57:56.908 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_dump_file": "", 2024-03-20T08:57:56.908 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-03-20T08:57:56.908 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-03-20T08:57:56.908 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_fadvise": "true", 2024-03-20T08:57:56.908 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_fail_eio": "true", 2024-03-20T08:57:56.908 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_fd_cache_shards": "16", 2024-03-20T08:57:56.908 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_fd_cache_size": "128", 2024-03-20T08:57:56.908 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_fiemap": "false", 2024-03-20T08:57:56.908 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_fiemap_threshold": "4096", 2024-03-20T08:57:56.908 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-03-20T08:57:56.909 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_index_retry_probability": "0.000000", 2024-03-20T08:57:56.909 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_inject_stall": "0", 2024-03-20T08:57:56.909 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_journal_parallel": "false", 2024-03-20T08:57:56.909 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_journal_trailing": "false", 2024-03-20T08:57:56.909 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_journal_writeahead": "false", 2024-03-20T08:57:56.909 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_kill_at": "0", 2024-03-20T08:57:56.909 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-03-20T08:57:56.909 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_max_inline_xattr_size": "0", 2024-03-20T08:57:56.909 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-03-20T08:57:56.909 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-03-20T08:57:56.909 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-03-20T08:57:56.910 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_max_inline_xattrs": "0", 2024-03-20T08:57:56.910 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-03-20T08:57:56.910 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_max_inline_xattrs_other": "2", 2024-03-20T08:57:56.910 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-03-20T08:57:56.910 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_max_sync_interval": "5.000000", 2024-03-20T08:57:56.910 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_max_xattr_value_size": "0", 2024-03-20T08:57:56.910 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-03-20T08:57:56.910 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-03-20T08:57:56.910 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-03-20T08:57:56.910 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_merge_threshold": "-10", 2024-03-20T08:57:56.910 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_min_sync_interval": "0.010000", 2024-03-20T08:57:56.910 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_odsync_write": "false", 2024-03-20T08:57:56.910 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_omap_backend": "rocksdb", 2024-03-20T08:57:56.911 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_omap_backend_path": "", 2024-03-20T08:57:56.911 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_omap_header_cache_size": "1024", 2024-03-20T08:57:56.911 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_ondisk_finisher_threads": "1", 2024-03-20T08:57:56.911 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-03-20T08:57:56.911 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_op_thread_timeout": "60", 2024-03-20T08:57:56.911 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_op_threads": "2", 2024-03-20T08:57:56.911 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_punch_hole": "false", 2024-03-20T08:57:56.911 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-03-20T08:57:56.911 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-03-20T08:57:56.911 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-03-20T08:57:56.911 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-03-20T08:57:56.911 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-03-20T08:57:56.911 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_queue_max_bytes": "104857600", 2024-03-20T08:57:56.912 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-03-20T08:57:56.912 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-03-20T08:57:56.912 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-03-20T08:57:56.912 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_queue_max_ops": "50", 2024-03-20T08:57:56.912 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-03-20T08:57:56.912 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_seek_data_hole": "false", 2024-03-20T08:57:56.912 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_sloppy_crc": "false", 2024-03-20T08:57:56.912 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-03-20T08:57:56.912 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_splice": "false", 2024-03-20T08:57:56.912 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_split_multiple": "2", 2024-03-20T08:57:56.912 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_split_rand_factor": "20", 2024-03-20T08:57:56.912 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_update_to": "1000", 2024-03-20T08:57:56.912 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-03-20T08:57:56.912 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-03-20T08:57:56.913 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-03-20T08:57:56.913 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-03-20T08:57:56.913 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-03-20T08:57:56.913 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-03-20T08:57:56.913 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_wbthrottle_enable": "true", 2024-03-20T08:57:56.913 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-03-20T08:57:56.913 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-03-20T08:57:56.913 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-03-20T08:57:56.913 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-03-20T08:57:56.913 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-03-20T08:57:56.913 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-03-20T08:57:56.913 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_xfs_extsize": "false", 2024-03-20T08:57:56.913 INFO:teuthology.orchestra.run.smithi026.stdout: "filestore_zfs_snap": "false", 2024-03-20T08:57:56.913 INFO:teuthology.orchestra.run.smithi026.stdout: "fio_dir": "/tmp/fio", 2024-03-20T08:57:56.914 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "852fbb68-e696-11ee-95c9-87774f69a715", 2024-03-20T08:57:56.914 INFO:teuthology.orchestra.run.smithi026.stdout: "fuse_allow_other": "true", 2024-03-20T08:57:56.914 INFO:teuthology.orchestra.run.smithi026.stdout: "fuse_atomic_o_trunc": "true", 2024-03-20T08:57:56.914 INFO:teuthology.orchestra.run.smithi026.stdout: "fuse_big_writes": "true", 2024-03-20T08:57:56.914 INFO:teuthology.orchestra.run.smithi026.stdout: "fuse_debug": "false", 2024-03-20T08:57:56.914 INFO:teuthology.orchestra.run.smithi026.stdout: "fuse_default_permissions": "false", 2024-03-20T08:57:56.914 INFO:teuthology.orchestra.run.smithi026.stdout: "fuse_disable_pagecache": "false", 2024-03-20T08:57:56.914 INFO:teuthology.orchestra.run.smithi026.stdout: "fuse_max_write": "0", 2024-03-20T08:57:56.914 INFO:teuthology.orchestra.run.smithi026.stdout: "fuse_multithreaded": "true", 2024-03-20T08:57:56.914 INFO:teuthology.orchestra.run.smithi026.stdout: "fuse_require_active_mds": "true", 2024-03-20T08:57:56.914 INFO:teuthology.orchestra.run.smithi026.stdout: "fuse_set_user_groups": "true", 2024-03-20T08:57:56.914 INFO:teuthology.orchestra.run.smithi026.stdout: "fuse_splice_move": "true", 2024-03-20T08:57:56.914 INFO:teuthology.orchestra.run.smithi026.stdout: "fuse_splice_read": "true", 2024-03-20T08:57:56.914 INFO:teuthology.orchestra.run.smithi026.stdout: "fuse_splice_write": "true", 2024-03-20T08:57:56.915 INFO:teuthology.orchestra.run.smithi026.stdout: "fuse_syncfs_on_mksnap": "true", 2024-03-20T08:57:56.915 INFO:teuthology.orchestra.run.smithi026.stdout: "fuse_use_invalidate_cb": "true", 2024-03-20T08:57:56.915 INFO:teuthology.orchestra.run.smithi026.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-03-20T08:57:56.915 INFO:teuthology.orchestra.run.smithi026.stdout: "gss_target_name": "ceph", 2024-03-20T08:57:56.915 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_file": "", 2024-03-20T08:57:56.915 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_inject_failure": "0", 2024-03-20T08:57:56.915 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_interval": "5", 2024-03-20T08:57:56.915 INFO:teuthology.orchestra.run.smithi026.stdout: "host": "smithi026", 2024-03-20T08:57:56.915 INFO:teuthology.orchestra.run.smithi026.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-03-20T08:57:56.915 INFO:teuthology.orchestra.run.smithi026.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-03-20T08:57:56.915 INFO:teuthology.orchestra.run.smithi026.stdout: "immutable_object_cache_max_size": "1073741824", 2024-03-20T08:57:56.915 INFO:teuthology.orchestra.run.smithi026.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-03-20T08:57:56.915 INFO:teuthology.orchestra.run.smithi026.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-03-20T08:57:56.916 INFO:teuthology.orchestra.run.smithi026.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-03-20T08:57:56.916 INFO:teuthology.orchestra.run.smithi026.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-03-20T08:57:56.916 INFO:teuthology.orchestra.run.smithi026.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-03-20T08:57:56.916 INFO:teuthology.orchestra.run.smithi026.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-03-20T08:57:56.916 INFO:teuthology.orchestra.run.smithi026.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-03-20T08:57:56.916 INFO:teuthology.orchestra.run.smithi026.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-03-20T08:57:56.916 INFO:teuthology.orchestra.run.smithi026.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-03-20T08:57:56.916 INFO:teuthology.orchestra.run.smithi026.stdout: "immutable_object_cache_watermark": "0.900000", 2024-03-20T08:57:56.916 INFO:teuthology.orchestra.run.smithi026.stdout: "inject_early_sigterm": "false", 2024-03-20T08:57:56.916 INFO:teuthology.orchestra.run.smithi026.stdout: "jaeger_agent_port": "6799", 2024-03-20T08:57:56.916 INFO:teuthology.orchestra.run.smithi026.stdout: "jaeger_tracing_enable": "false", 2024-03-20T08:57:56.916 INFO:teuthology.orchestra.run.smithi026.stdout: "journal_aio": "true", 2024-03-20T08:57:56.916 INFO:teuthology.orchestra.run.smithi026.stdout: "journal_align_min_size": "65536", 2024-03-20T08:57:56.917 INFO:teuthology.orchestra.run.smithi026.stdout: "journal_block_align": "true", 2024-03-20T08:57:56.917 INFO:teuthology.orchestra.run.smithi026.stdout: "journal_block_size": "4096", 2024-03-20T08:57:56.917 INFO:teuthology.orchestra.run.smithi026.stdout: "journal_dio": "true", 2024-03-20T08:57:56.917 INFO:teuthology.orchestra.run.smithi026.stdout: "journal_discard": "false", 2024-03-20T08:57:56.917 INFO:teuthology.orchestra.run.smithi026.stdout: "journal_force_aio": "false", 2024-03-20T08:57:56.917 INFO:teuthology.orchestra.run.smithi026.stdout: "journal_ignore_corruption": "false", 2024-03-20T08:57:56.917 INFO:teuthology.orchestra.run.smithi026.stdout: "journal_max_write_bytes": "10485760", 2024-03-20T08:57:56.917 INFO:teuthology.orchestra.run.smithi026.stdout: "journal_max_write_entries": "100", 2024-03-20T08:57:56.917 INFO:teuthology.orchestra.run.smithi026.stdout: "journal_replay_from": "0", 2024-03-20T08:57:56.917 INFO:teuthology.orchestra.run.smithi026.stdout: "journal_throttle_high_multiple": "0.000000", 2024-03-20T08:57:56.917 INFO:teuthology.orchestra.run.smithi026.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-03-20T08:57:56.917 INFO:teuthology.orchestra.run.smithi026.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-03-20T08:57:56.917 INFO:teuthology.orchestra.run.smithi026.stdout: "journal_throttle_max_multiple": "0.000000", 2024-03-20T08:57:56.918 INFO:teuthology.orchestra.run.smithi026.stdout: "journal_write_header_frequency": "0", 2024-03-20T08:57:56.918 INFO:teuthology.orchestra.run.smithi026.stdout: "journal_zero_on_create": "false", 2024-03-20T08:57:56.918 INFO:teuthology.orchestra.run.smithi026.stdout: "journaler_prefetch_periods": "10", 2024-03-20T08:57:56.918 INFO:teuthology.orchestra.run.smithi026.stdout: "journaler_prezero_periods": "5", 2024-03-20T08:57:56.918 INFO:teuthology.orchestra.run.smithi026.stdout: "journaler_write_head_interval": "15", 2024-03-20T08:57:56.918 INFO:teuthology.orchestra.run.smithi026.stdout: "key": "", 2024-03-20T08:57:56.918 INFO:teuthology.orchestra.run.smithi026.stdout: "keyfile": "", 2024-03-20T08:57:56.918 INFO:teuthology.orchestra.run.smithi026.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-03-20T08:57:56.918 INFO:teuthology.orchestra.run.smithi026.stdout: "kstore_backend": "rocksdb", 2024-03-20T08:57:56.918 INFO:teuthology.orchestra.run.smithi026.stdout: "kstore_default_stripe_size": "65536", 2024-03-20T08:57:56.918 INFO:teuthology.orchestra.run.smithi026.stdout: "kstore_fsck_on_mount": "false", 2024-03-20T08:57:56.918 INFO:teuthology.orchestra.run.smithi026.stdout: "kstore_fsck_on_mount_deep": "true", 2024-03-20T08:57:56.919 INFO:teuthology.orchestra.run.smithi026.stdout: "kstore_max_bytes": "67108864", 2024-03-20T08:57:56.919 INFO:teuthology.orchestra.run.smithi026.stdout: "kstore_max_ops": "512", 2024-03-20T08:57:56.919 INFO:teuthology.orchestra.run.smithi026.stdout: "kstore_nid_prealloc": "1024", 2024-03-20T08:57:56.919 INFO:teuthology.orchestra.run.smithi026.stdout: "kstore_onode_map_size": "1024", 2024-03-20T08:57:56.919 INFO:teuthology.orchestra.run.smithi026.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-03-20T08:57:56.919 INFO:teuthology.orchestra.run.smithi026.stdout: "kstore_sync_submit_transaction": "false", 2024-03-20T08:57:56.919 INFO:teuthology.orchestra.run.smithi026.stdout: "kstore_sync_transaction": "false", 2024-03-20T08:57:56.919 INFO:teuthology.orchestra.run.smithi026.stdout: "librados_thread_count": "2", 2024-03-20T08:57:56.919 INFO:teuthology.orchestra.run.smithi026.stdout: "lockdep": "false", 2024-03-20T08:57:56.919 INFO:teuthology.orchestra.run.smithi026.stdout: "lockdep_force_backtrace": "false", 2024-03-20T08:57:56.919 INFO:teuthology.orchestra.run.smithi026.stdout: "log_coarse_timestamps": "true", 2024-03-20T08:57:56.919 INFO:teuthology.orchestra.run.smithi026.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-03-20T08:57:56.919 INFO:teuthology.orchestra.run.smithi026.stdout: "log_flush_on_exit": "false", 2024-03-20T08:57:56.919 INFO:teuthology.orchestra.run.smithi026.stdout: "log_graylog_host": "127.0.0.1", 2024-03-20T08:57:56.920 INFO:teuthology.orchestra.run.smithi026.stdout: "log_graylog_port": "12201", 2024-03-20T08:57:56.920 INFO:teuthology.orchestra.run.smithi026.stdout: "log_max_new": "1000", 2024-03-20T08:57:56.920 INFO:teuthology.orchestra.run.smithi026.stdout: "log_max_recent": "10000", 2024-03-20T08:57:56.920 INFO:teuthology.orchestra.run.smithi026.stdout: "log_stderr_prefix": "", 2024-03-20T08:57:56.920 INFO:teuthology.orchestra.run.smithi026.stdout: "log_stop_at_utilization": "0.970000", 2024-03-20T08:57:56.920 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_file": "true", 2024-03-20T08:57:56.920 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_graylog": "false", 2024-03-20T08:57:56.920 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_journald": "false", 2024-03-20T08:57:56.920 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_stderr": "false", 2024-03-20T08:57:56.920 INFO:teuthology.orchestra.run.smithi026.stdout: "log_to_syslog": "false", 2024-03-20T08:57:56.920 INFO:teuthology.orchestra.run.smithi026.stdout: "max_rotating_auth_attempts": "10", 2024-03-20T08:57:56.920 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-03-20T08:57:56.920 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_action_on_write_error": "1", 2024-03-20T08:57:56.921 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_alternate_name_max": "8192", 2024-03-20T08:57:56.921 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_asio_thread_count": "2", 2024-03-20T08:57:56.921 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_export_pin": "true", 2024-03-20T08:57:56.921 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_fragment_dirs": "true", 2024-03-20T08:57:56.921 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-03-20T08:57:56.921 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_fragment_interval": "5", 2024-03-20T08:57:56.921 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_fragment_size_max": "100000", 2024-03-20T08:57:56.921 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_idle_threshold": "0.000000", 2024-03-20T08:57:56.921 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_interval": "10", 2024-03-20T08:57:56.921 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_max": "-1", 2024-03-20T08:57:56.921 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_max_until": "-1", 2024-03-20T08:57:56.921 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_merge_size": "50", 2024-03-20T08:57:56.921 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_midchunk": "0.300000", 2024-03-20T08:57:56.922 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_min_rebalance": "0.100000", 2024-03-20T08:57:56.922 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_min_start": "0.200000", 2024-03-20T08:57:56.922 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_minchunk": "0.001000", 2024-03-20T08:57:56.922 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_mode": "0", 2024-03-20T08:57:56.922 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_need_max": "1.200000", 2024-03-20T08:57:56.922 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_need_min": "0.800000", 2024-03-20T08:57:56.922 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_overload_epochs": "2", 2024-03-20T08:57:56.922 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-03-20T08:57:56.922 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_sample_interval": "3.000000", 2024-03-20T08:57:56.922 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_split_bits": "3", 2024-03-20T08:57:56.922 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_split_rd": "25000.000000", 2024-03-20T08:57:56.922 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_split_size": "10000", 2024-03-20T08:57:56.922 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_split_wr": "10000.000000", 2024-03-20T08:57:56.922 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_target_decay": "10.000000", 2024-03-20T08:57:56.923 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-03-20T08:57:56.923 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_beacon_grace": "15.000000", 2024-03-20T08:57:56.923 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_beacon_interval": "4.000000", 2024-03-20T08:57:56.923 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_beacon_mon_down_grace": "60", 2024-03-20T08:57:56.923 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_cache_memory_limit": "4294967296", 2024-03-20T08:57:56.923 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_cache_mid": "0.700000", 2024-03-20T08:57:56.923 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_cache_release_free_interval": "10", 2024-03-20T08:57:56.923 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_cache_reservation": "0.050000", 2024-03-20T08:57:56.923 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-03-20T08:57:56.923 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_cache_trim_interval": "1", 2024-03-20T08:57:56.923 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_cache_trim_threshold": "262144", 2024-03-20T08:57:56.923 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-03-20T08:57:56.923 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-03-20T08:57:56.924 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_client_delegate_inos_pct": "50", 2024-03-20T08:57:56.924 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_client_prealloc_inos": "1000", 2024-03-20T08:57:56.924 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-03-20T08:57:56.924 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_connect_bootstrapping": "false", 2024-03-20T08:57:56.924 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_damage_table_max_entries": "10000", 2024-03-20T08:57:56.924 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-03-20T08:57:56.924 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_debug_auth_pins": "false", 2024-03-20T08:57:56.924 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_debug_frag": "false", 2024-03-20T08:57:56.924 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_debug_scatterstat": "false", 2024-03-20T08:57:56.924 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_debug_subtrees": "false", 2024-03-20T08:57:56.924 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_decay_halflife": "5.000000", 2024-03-20T08:57:56.924 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_default_dir_hash": "2", 2024-03-20T08:57:56.924 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_defer_session_stale": "true", 2024-03-20T08:57:56.924 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_deny_all_reconnect": "false", 2024-03-20T08:57:56.925 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_dir_keys_per_op": "16384", 2024-03-20T08:57:56.925 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_dir_max_commit_size": "10", 2024-03-20T08:57:56.925 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_dir_max_entries": "0", 2024-03-20T08:57:56.926 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_dir_prefetch": "true", 2024-03-20T08:57:56.926 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_dirstat_min_interval": "1.000000", 2024-03-20T08:57:56.926 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_dump_cache_after_rejoin": "false", 2024-03-20T08:57:56.926 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_dump_cache_on_map": "false", 2024-03-20T08:57:56.926 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_dump_cache_threshold_file": "0", 2024-03-20T08:57:56.927 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-03-20T08:57:56.927 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_early_reply": "true", 2024-03-20T08:57:56.927 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_enable_op_tracker": "true", 2024-03-20T08:57:56.927 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_enforce_unique_name": "true", 2024-03-20T08:57:56.927 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_export_ephemeral_distributed": "true", 2024-03-20T08:57:56.927 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-03-20T08:57:56.927 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_export_ephemeral_random": "true", 2024-03-20T08:57:56.927 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-03-20T08:57:56.927 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-03-20T08:57:56.927 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_forward_all_requests_to_auth": "false", 2024-03-20T08:57:56.927 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-03-20T08:57:56.927 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-03-20T08:57:56.928 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-03-20T08:57:56.928 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-03-20T08:57:56.928 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_health_cache_threshold": "1.500000", 2024-03-20T08:57:56.928 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_health_summarize_threshold": "10", 2024-03-20T08:57:56.928 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_heartbeat_grace": "15.000000", 2024-03-20T08:57:56.928 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_heartbeat_reset_grace": "1000", 2024-03-20T08:57:56.928 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_inject_health_dummy": "false", 2024-03-20T08:57:56.928 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-03-20T08:57:56.928 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_inject_migrator_session_race": "false", 2024-03-20T08:57:56.928 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-03-20T08:57:56.928 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-03-20T08:57:56.928 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-03-20T08:57:56.928 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_join_fs": "", 2024-03-20T08:57:56.929 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_journal_format": "1", 2024-03-20T08:57:56.929 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_kill_create_at": "0", 2024-03-20T08:57:56.929 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_kill_export_at": "0", 2024-03-20T08:57:56.929 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_kill_import_at": "0", 2024-03-20T08:57:56.929 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_kill_journal_at": "0", 2024-03-20T08:57:56.929 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_kill_journal_expire_at": "0", 2024-03-20T08:57:56.929 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_kill_journal_replay_at": "0", 2024-03-20T08:57:56.929 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_kill_link_at": "0", 2024-03-20T08:57:56.929 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_kill_mdstable_at": "0", 2024-03-20T08:57:56.929 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_kill_openc_at": "0", 2024-03-20T08:57:56.929 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_kill_rename_at": "0", 2024-03-20T08:57:56.929 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_kill_shutdown_at": "0", 2024-03-20T08:57:56.929 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-03-20T08:57:56.929 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_log_event_large_threshold": "524288", 2024-03-20T08:57:56.930 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_log_events_per_segment": "1024", 2024-03-20T08:57:56.930 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_log_major_segment_event_ratio": "12", 2024-03-20T08:57:56.930 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_log_max_events": "-1", 2024-03-20T08:57:56.930 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_log_max_segments": "128", 2024-03-20T08:57:56.930 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_log_pause": "false", 2024-03-20T08:57:56.930 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_log_segment_size": "0", 2024-03-20T08:57:56.930 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_log_skip_corrupt_events": "false", 2024-03-20T08:57:56.930 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_log_skip_unbounded_events": "false", 2024-03-20T08:57:56.930 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-03-20T08:57:56.930 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_log_trim_threshold": "128", 2024-03-20T08:57:56.930 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-03-20T08:57:56.930 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_log_warn_factor": "2.000000", 2024-03-20T08:57:56.930 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_max_caps_per_client": "1048576", 2024-03-20T08:57:56.931 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_max_completed_flushes": "100000", 2024-03-20T08:57:56.931 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_max_completed_requests": "100000", 2024-03-20T08:57:56.931 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_max_export_size": "20971520", 2024-03-20T08:57:56.931 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_max_file_recover": "32", 2024-03-20T08:57:56.931 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_max_purge_files": "64", 2024-03-20T08:57:56.931 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_max_purge_ops": "8192", 2024-03-20T08:57:56.931 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-03-20T08:57:56.931 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-03-20T08:57:56.931 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_max_snaps_per_dir": "100", 2024-03-20T08:57:56.931 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_metrics_update_interval": "2", 2024-03-20T08:57:56.931 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_min_caps_per_client": "100", 2024-03-20T08:57:56.931 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_min_caps_working_set": "10000", 2024-03-20T08:57:56.931 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-03-20T08:57:56.932 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_numa_node": "-1", 2024-03-20T08:57:56.932 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-03-20T08:57:56.932 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_op_complaint_time": "30.000000", 2024-03-20T08:57:56.932 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_op_history_duration": "600", 2024-03-20T08:57:56.932 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_op_history_size": "20", 2024-03-20T08:57:56.932 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_op_history_slow_op_size": "20", 2024-03-20T08:57:56.932 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_op_history_slow_op_threshold": "10", 2024-03-20T08:57:56.932 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_op_log_threshold": "5", 2024-03-20T08:57:56.932 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_ping_grace": "15", 2024-03-20T08:57:56.932 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_ping_interval": "5", 2024-03-20T08:57:56.932 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-03-20T08:57:56.932 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-03-20T08:57:56.932 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_recall_max_caps": "30000", 2024-03-20T08:57:56.933 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-03-20T08:57:56.933 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_recall_max_decay_threshold": "131072", 2024-03-20T08:57:56.933 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-03-20T08:57:56.933 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_recall_warning_threshold": "262144", 2024-03-20T08:57:56.933 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_reconnect_timeout": "45.000000", 2024-03-20T08:57:56.933 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_replay_interval": "1.000000", 2024-03-20T08:57:56.933 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-03-20T08:57:56.933 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-03-20T08:57:56.933 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_root_ino_gid": "0", 2024-03-20T08:57:56.933 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_root_ino_uid": "0", 2024-03-20T08:57:56.933 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-03-20T08:57:56.933 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_session_blocklist_on_evict": "true", 2024-03-20T08:57:56.933 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_session_blocklist_on_timeout": "true", 2024-03-20T08:57:56.933 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-03-20T08:57:56.934 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-03-20T08:57:56.934 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-03-20T08:57:56.934 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-03-20T08:57:56.934 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-03-20T08:57:56.934 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_session_metadata_threshold": "16777216", 2024-03-20T08:57:56.934 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-03-20T08:57:56.934 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_shutdown_check": "0", 2024-03-20T08:57:56.934 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_skip_ino": "0", 2024-03-20T08:57:56.934 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_sleep_rank_change": "0.000000", 2024-03-20T08:57:56.934 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_snap_max_uid": "4294967294", 2024-03-20T08:57:56.934 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_snap_min_uid": "0", 2024-03-20T08:57:56.934 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_snap_rstat": "false", 2024-03-20T08:57:56.934 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_standby_replay_damaged": "false", 2024-03-20T08:57:56.934 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_symlink_recovery": "true", 2024-03-20T08:57:56.935 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_task_status_update_interval": "2.000000", 2024-03-20T08:57:56.935 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_thrash_exports": "0", 2024-03-20T08:57:56.935 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_thrash_fragments": "0", 2024-03-20T08:57:56.935 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_tick_interval": "5.000000", 2024-03-20T08:57:56.935 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_valgrind_exit": "false", 2024-03-20T08:57:56.935 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_verify_backtrace": "1", 2024-03-20T08:57:56.935 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_verify_scatter": "false", 2024-03-20T08:57:56.935 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_wipe_ino_prealloc": "false", 2024-03-20T08:57:56.935 INFO:teuthology.orchestra.run.smithi026.stdout: "mds_wipe_sessions": "false", 2024-03-20T08:57:56.935 INFO:teuthology.orchestra.run.smithi026.stdout: "mempool_debug": "false", 2024-03-20T08:57:56.935 INFO:teuthology.orchestra.run.smithi026.stdout: "memstore_debug_omit_block_device_write": "false", 2024-03-20T08:57:56.935 INFO:teuthology.orchestra.run.smithi026.stdout: "memstore_device_bytes": "1073741824", 2024-03-20T08:57:56.935 INFO:teuthology.orchestra.run.smithi026.stdout: "memstore_page_set": "false", 2024-03-20T08:57:56.935 INFO:teuthology.orchestra.run.smithi026.stdout: "memstore_page_size": "65536", 2024-03-20T08:57:56.936 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_client_bytes": "134217728", 2024-03-20T08:57:56.936 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_client_messages": "512", 2024-03-20T08:57:56.936 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-03-20T08:57:56.936 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_connect_retry_interval": "1.000000", 2024-03-20T08:57:56.936 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-03-20T08:57:56.936 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-03-20T08:57:56.936 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_disabled_modules": "", 2024-03-20T08:57:56.936 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-03-20T08:57:56.936 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_max_pg_num_change": "128", 2024-03-20T08:57:56.936 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_mds_bytes": "134217728", 2024-03-20T08:57:56.936 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_mds_messages": "128", 2024-03-20T08:57:56.936 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-03-20T08:57:56.936 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_mon_bytes": "134217728", 2024-03-20T08:57:56.937 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_mon_messages": "128", 2024-03-20T08:57:56.937 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_osd_bytes": "536870912", 2024-03-20T08:57:56.937 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_osd_messages": "8192", 2024-03-20T08:57:56.937 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_pool": "true", 2024-03-20T08:57:56.937 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_service_beacon_grace": "60.000000", 2024-03-20T08:57:56.937 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_standby_modules": "true", 2024-03-20T08:57:56.937 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_stats_period": "5", 2024-03-20T08:57:56.937 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_stats_threshold": "5", 2024-03-20T08:57:56.937 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_tick_period": "2", 2024-03-20T08:57:56.937 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-03-20T08:57:56.937 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_accept_timeout_factor": "2.000000", 2024-03-20T08:57:56.937 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_allow_pool_delete": "true", 2024-03-20T08:57:56.937 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_allow_pool_size_one": "false", 2024-03-20T08:57:56.937 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_auth_validate_all_caps": "true", 2024-03-20T08:57:56.938 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-03-20T08:57:56.938 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-03-20T08:57:56.938 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_client_bytes": "104857600", 2024-03-20T08:57:56.938 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_client_directed_command_retry": "5", 2024-03-20T08:57:56.938 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_client_hunt_interval": "3.000000", 2024-03-20T08:57:56.938 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-03-20T08:57:56.938 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-03-20T08:57:56.938 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-03-20T08:57:56.938 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_client_hunt_parallel": "3", 2024-03-20T08:57:56.938 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_client_log_interval": "1.000000", 2024-03-20T08:57:56.938 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-03-20T08:57:56.938 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_client_ping_interval": "10.000000", 2024-03-20T08:57:56.938 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_client_ping_timeout": "30.000000", 2024-03-20T08:57:56.939 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_client_target_rank": "-1", 2024-03-20T08:57:56.939 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_clock_drift_allowed": "1.000000", 2024-03-20T08:57:56.939 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-03-20T08:57:56.939 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-03-20T08:57:56.939 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_cluster_log_file_level": "debug", 2024-03-20T08:57:56.939 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_cluster_log_to_file": "true", 2024-03-20T08:57:56.939 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_cluster_log_to_graylog": "false", 2024-03-20T08:57:56.939 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-03-20T08:57:56.939 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-03-20T08:57:56.939 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_cluster_log_to_journald": "true", 2024-03-20T08:57:56.939 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_cluster_log_to_stderr": "false", 2024-03-20T08:57:56.939 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-03-20T08:57:56.939 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-03-20T08:57:56.939 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-03-20T08:57:56.940 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_compact_on_bootstrap": "false", 2024-03-20T08:57:56.940 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_compact_on_start": "false", 2024-03-20T08:57:56.940 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_compact_on_trim": "true", 2024-03-20T08:57:56.940 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_con_tracker_persist_interval": "10", 2024-03-20T08:57:56.940 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_con_tracker_score_halflife": "43200", 2024-03-20T08:57:56.940 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_config_key_max_entry_size": "65536", 2024-03-20T08:57:56.940 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_cpu_threads": "4", 2024-03-20T08:57:56.940 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_crush_min_required_version": "hammer", 2024-03-20T08:57:56.940 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_daemon_bytes": "419430400", 2024-03-20T08:57:56.940 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-03-20T08:57:56.940 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_data_avail_crit": "5", 2024-03-20T08:57:56.940 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_data_avail_warn": "5", 2024-03-20T08:57:56.940 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_data_size_warn": "16106127360", 2024-03-20T08:57:56.940 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_debug_block_osdmap_trim": "false", 2024-03-20T08:57:56.941 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-03-20T08:57:56.941 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_debug_dump_json": "false", 2024-03-20T08:57:56.941 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-03-20T08:57:56.941 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_debug_dump_transactions": "false", 2024-03-20T08:57:56.941 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_debug_extra_checks": "false", 2024-03-20T08:57:56.941 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-03-20T08:57:56.941 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-03-20T08:57:56.941 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_debug_no_require_reef": "false", 2024-03-20T08:57:56.941 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_debug_no_require_squid": "false", 2024-03-20T08:57:56.941 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-03-20T08:57:56.941 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_delta_reset_interval": "10.000000", 2024-03-20T08:57:56.941 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_dns_srv_name": "ceph-mon", 2024-03-20T08:57:56.941 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_down_mkfs_grace": "60", 2024-03-20T08:57:56.941 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_election_default_strategy": "3", 2024-03-20T08:57:56.942 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_election_timeout": "5.000000", 2024-03-20T08:57:56.942 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-03-20T08:57:56.942 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_elector_ping_divisor": "2", 2024-03-20T08:57:56.942 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_elector_ping_timeout": "2.000000", 2024-03-20T08:57:56.942 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_enable_op_tracker": "true", 2024-03-20T08:57:56.942 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_fake_pool_delete": "false", 2024-03-20T08:57:56.942 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_force_quorum_join": "false", 2024-03-20T08:57:56.942 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_globalid_prealloc": "10000", 2024-03-20T08:57:56.942 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_health_detail_to_clog": "true", 2024-03-20T08:57:56.942 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_health_log_update_period": "5", 2024-03-20T08:57:56.942 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_health_max_detail": "50", 2024-03-20T08:57:56.942 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_health_to_clog": "true", 2024-03-20T08:57:56.942 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_health_to_clog_interval": "600", 2024-03-20T08:57:56.942 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-03-20T08:57:56.943 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_host": "[v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0]", 2024-03-20T08:57:56.943 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_host_override": "", 2024-03-20T08:57:56.943 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_initial_members": "", 2024-03-20T08:57:56.943 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-03-20T08:57:56.943 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-03-20T08:57:56.943 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-03-20T08:57:56.943 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-03-20T08:57:56.943 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_keyvaluedb": "rocksdb", 2024-03-20T08:57:56.943 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_lease": "5.000000", 2024-03-20T08:57:56.943 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-03-20T08:57:56.943 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-03-20T08:57:56.943 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_log_full_interval": "50", 2024-03-20T08:57:56.943 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_log_max": "10000", 2024-03-20T08:57:56.943 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_log_max_summary": "50", 2024-03-20T08:57:56.944 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_max_log_entries_per_event": "4096", 2024-03-20T08:57:56.944 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_max_log_epochs": "500", 2024-03-20T08:57:56.944 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_max_mdsmap_epochs": "500", 2024-03-20T08:57:56.944 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_max_mgrmap_epochs": "500", 2024-03-20T08:57:56.944 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_max_osd": "10000", 2024-03-20T08:57:56.944 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_max_pg_per_osd": "10000", 2024-03-20T08:57:56.944 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_max_pool_pg_num": "65536", 2024-03-20T08:57:56.944 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-03-20T08:57:56.944 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-03-20T08:57:56.944 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_mds_force_trim_to": "0", 2024-03-20T08:57:56.944 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_mds_skip_sanity": "false", 2024-03-20T08:57:56.944 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_memory_autotune": "true", 2024-03-20T08:57:56.944 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_memory_target": "2147483648", 2024-03-20T08:57:56.944 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_mgr_beacon_grace": "30", 2024-03-20T08:57:56.945 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-03-20T08:57:56.945 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_mgr_digest_period": "5", 2024-03-20T08:57:56.945 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_mgr_inactive_grace": "60", 2024-03-20T08:57:56.945 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_mgr_mkfs_grace": "240", 2024-03-20T08:57:56.945 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-03-20T08:57:56.945 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_min_osdmap_epochs": "50", 2024-03-20T08:57:56.945 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_op_complaint_time": "30", 2024-03-20T08:57:56.945 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_op_history_duration": "600", 2024-03-20T08:57:56.945 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_op_history_size": "20", 2024-03-20T08:57:56.945 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_op_history_slow_op_size": "20", 2024-03-20T08:57:56.945 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_op_history_slow_op_threshold": "10", 2024-03-20T08:57:56.945 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_op_log_threshold": "5", 2024-03-20T08:57:56.945 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-03-20T08:57:56.946 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-03-20T08:57:56.946 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-03-20T08:57:56.946 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_auto_mark_in": "false", 2024-03-20T08:57:56.946 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_auto_mark_new_in": "true", 2024-03-20T08:57:56.946 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-03-20T08:57:56.946 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-03-20T08:57:56.946 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_cache_size": "500", 2024-03-20T08:57:56.946 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_cache_size_min": "134217728", 2024-03-20T08:57:56.946 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_crush_smoke_test": "true", 2024-03-20T08:57:56.946 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_destroyed_out_interval": "600", 2024-03-20T08:57:56.946 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_down_out_interval": "600", 2024-03-20T08:57:56.946 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-03-20T08:57:56.946 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-03-20T08:57:56.946 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_force_trim_to": "0", 2024-03-20T08:57:56.947 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_full_ratio": "0.950000", 2024-03-20T08:57:56.947 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-03-20T08:57:56.947 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_laggy_halflife": "3600", 2024-03-20T08:57:56.947 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_laggy_max_interval": "300", 2024-03-20T08:57:56.947 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_laggy_weight": "0.300000", 2024-03-20T08:57:56.947 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-03-20T08:57:56.947 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_max_creating_pgs": "1024", 2024-03-20T08:57:56.947 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_max_initial_pgs": "1024", 2024-03-20T08:57:56.947 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_min_down_reporters": "2", 2024-03-20T08:57:56.947 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-03-20T08:57:56.947 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-03-20T08:57:56.947 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-03-20T08:57:56.947 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_prime_pg_temp": "true", 2024-03-20T08:57:56.947 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-03-20T08:57:56.948 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-03-20T08:57:56.948 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_report_timeout": "900", 2024-03-20T08:57:56.948 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-03-20T08:57:56.948 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-03-20T08:57:56.948 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_warn_num_repaired": "10", 2024-03-20T08:57:56.948 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osd_warn_op_age": "32.000000", 2024-03-20T08:57:56.948 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-03-20T08:57:56.948 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osdmap_full_prune_interval": "2", 2024-03-20T08:57:56.948 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osdmap_full_prune_min": "15", 2024-03-20T08:57:56.948 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-03-20T08:57:56.948 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-03-20T08:57:56.948 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_pg_stuck_threshold": "60", 2024-03-20T08:57:56.948 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-03-20T08:57:56.948 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_pg_warn_min_objects": "10000", 2024-03-20T08:57:56.949 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_pg_warn_min_per_osd": "0", 2024-03-20T08:57:56.949 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-03-20T08:57:56.949 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_pool_quota_crit_threshold": "0", 2024-03-20T08:57:56.949 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_pool_quota_warn_threshold": "0", 2024-03-20T08:57:56.949 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_probe_timeout": "2.000000", 2024-03-20T08:57:56.949 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_reweight_max_change": "0.050000", 2024-03-20T08:57:56.949 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_reweight_max_osds": "4", 2024-03-20T08:57:56.949 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-03-20T08:57:56.949 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-03-20T08:57:56.949 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-03-20T08:57:56.949 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-03-20T08:57:56.949 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-03-20T08:57:56.949 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_scrub_interval": "300", 2024-03-20T08:57:56.949 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_scrub_max_keys": "100", 2024-03-20T08:57:56.950 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_scrub_timeout": "300", 2024-03-20T08:57:56.950 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_session_timeout": "300", 2024-03-20T08:57:56.950 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_smart_report_timeout": "5", 2024-03-20T08:57:56.950 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_stat_smooth_intervals": "6", 2024-03-20T08:57:56.950 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-03-20T08:57:56.950 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_stretch_pool_min_size": "2", 2024-03-20T08:57:56.950 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_stretch_pool_size": "4", 2024-03-20T08:57:56.950 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-03-20T08:57:56.950 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_subscribe_interval": "86400.000000", 2024-03-20T08:57:56.950 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_sync_debug": "false", 2024-03-20T08:57:56.950 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_sync_max_payload_keys": "2000", 2024-03-20T08:57:56.950 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_sync_max_payload_size": "1048576", 2024-03-20T08:57:56.950 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_sync_provider_kill_at": "0", 2024-03-20T08:57:56.950 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_sync_requester_kill_at": "0", 2024-03-20T08:57:56.951 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_sync_timeout": "60.000000", 2024-03-20T08:57:56.951 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_target_pg_per_osd": "100", 2024-03-20T08:57:56.951 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_tick_interval": "5", 2024-03-20T08:57:56.951 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_timecheck_interval": "300.000000", 2024-03-20T08:57:56.951 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-03-20T08:57:56.951 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_use_min_delay_socket": "false", 2024-03-20T08:57:56.951 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_older_version_delay": "604800", 2024-03-20T08:57:56.951 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-03-20T08:57:56.951 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-03-20T08:57:56.951 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-03-20T08:57:56.951 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_on_filestore_osds": "true", 2024-03-20T08:57:56.951 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-03-20T08:57:56.951 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-03-20T08:57:56.952 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-03-20T08:57:56.952 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_on_misplaced": "false", 2024-03-20T08:57:56.952 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-03-20T08:57:56.952 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_on_older_version": "true", 2024-03-20T08:57:56.952 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-03-20T08:57:56.952 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_on_pool_no_app": "true", 2024-03-20T08:57:56.952 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-03-20T08:57:56.952 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-03-20T08:57:56.952 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-03-20T08:57:56.952 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-03-20T08:57:56.952 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_on_too_few_osds": "false", 2024-03-20T08:57:56.952 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-03-20T08:57:56.952 INFO:teuthology.orchestra.run.smithi026.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-03-20T08:57:56.952 INFO:teuthology.orchestra.run.smithi026.stdout: "monmap": "", 2024-03-20T08:57:56.953 INFO:teuthology.orchestra.run.smithi026.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-20T08:57:56.953 INFO:teuthology.orchestra.run.smithi026.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-03-20T08:57:56.953 INFO:teuthology.orchestra.run.smithi026.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-03-20T08:57:56.953 INFO:teuthology.orchestra.run.smithi026.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-20T08:57:56.953 INFO:teuthology.orchestra.run.smithi026.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-03-20T08:57:56.953 INFO:teuthology.orchestra.run.smithi026.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-03-20T08:57:56.953 INFO:teuthology.orchestra.run.smithi026.stdout: "motr_tracing_enabled": "false", 2024-03-20T08:57:56.953 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_async_op_threads": "3", 2024-03-20T08:57:56.953 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_async_rdma_buffer_size": "131072", 2024-03-20T08:57:56.953 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_async_rdma_cm": "false", 2024-03-20T08:57:56.953 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_async_rdma_device_name": "", 2024-03-20T08:57:56.953 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_async_rdma_dscp": "96", 2024-03-20T08:57:56.953 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-03-20T08:57:56.953 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_async_rdma_gid_idx": "0", 2024-03-20T08:57:56.954 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_async_rdma_local_gid": "", 2024-03-20T08:57:56.954 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_async_rdma_polling_us": "1000", 2024-03-20T08:57:56.954 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_async_rdma_port_num": "1", 2024-03-20T08:57:56.954 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-03-20T08:57:56.954 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-03-20T08:57:56.954 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_async_rdma_roce_ver": "1", 2024-03-20T08:57:56.954 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_async_rdma_send_buffers": "1024", 2024-03-20T08:57:56.954 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_async_rdma_sl": "3", 2024-03-20T08:57:56.954 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_async_rdma_support_srq": "true", 2024-03-20T08:57:56.954 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_async_rdma_type": "ib", 2024-03-20T08:57:56.954 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_async_reap_threshold": "5", 2024-03-20T08:57:56.954 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_bind_before_connect": "false", 2024-03-20T08:57:56.954 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_bind_ipv4": "true", 2024-03-20T08:57:56.954 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_bind_ipv6": "false", 2024-03-20T08:57:56.955 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_bind_msgr1": "true", 2024-03-20T08:57:56.955 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_bind_msgr2": "true", 2024-03-20T08:57:56.955 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_bind_port_max": "7568", 2024-03-20T08:57:56.955 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_bind_port_min": "6800", 2024-03-20T08:57:56.955 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_bind_prefer_ipv4": "false", 2024-03-20T08:57:56.955 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_bind_retry_count": "3", 2024-03-20T08:57:56.955 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_bind_retry_delay": "5", 2024-03-20T08:57:56.955 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_blackhole_client": "false", 2024-03-20T08:57:56.955 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_blackhole_mds": "false", 2024-03-20T08:57:56.955 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_blackhole_mgr": "false", 2024-03-20T08:57:56.955 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_blackhole_mon": "false", 2024-03-20T08:57:56.955 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_blackhole_osd": "false", 2024-03-20T08:57:56.955 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_client_mode": "crc secure", 2024-03-20T08:57:56.955 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_cluster_mode": "crc secure", 2024-03-20T08:57:56.955 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_cluster_type": "", 2024-03-20T08:57:56.956 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_compress_secure": "false", 2024-03-20T08:57:56.956 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_connection_idle_timeout": "900", 2024-03-20T08:57:56.956 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_connection_ready_timeout": "10", 2024-03-20T08:57:56.956 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_crc_data": "true", 2024-03-20T08:57:56.956 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_crc_header": "true", 2024-03-20T08:57:56.956 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_die_on_bad_msg": "false", 2024-03-20T08:57:56.956 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_die_on_bug": "true", 2024-03-20T08:57:56.956 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_die_on_old_message": "true", 2024-03-20T08:57:56.956 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_die_on_skipped_message": "false", 2024-03-20T08:57:56.956 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_die_on_unhandled_msg": "false", 2024-03-20T08:57:56.956 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-03-20T08:57:56.956 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_dpdk_coremask": "0xF", 2024-03-20T08:57:56.956 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-03-20T08:57:56.956 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_dpdk_devs_allowlist": "", 2024-03-20T08:57:56.957 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_dpdk_enable_tso": "true", 2024-03-20T08:57:56.957 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-03-20T08:57:56.957 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-03-20T08:57:56.957 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_dpdk_hugepages": "", 2024-03-20T08:57:56.957 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_dpdk_hw_flow_control": "true", 2024-03-20T08:57:56.957 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-03-20T08:57:56.957 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_dpdk_lro": "true", 2024-03-20T08:57:56.957 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_dpdk_memory_channel": "4", 2024-03-20T08:57:56.957 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-03-20T08:57:56.957 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_dpdk_pmd": "", 2024-03-20T08:57:56.957 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_dpdk_port_id": "0", 2024-03-20T08:57:56.957 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-03-20T08:57:56.957 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_dump_corrupt_message_level": "1", 2024-03-20T08:57:56.957 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_dump_on_send": "false", 2024-03-20T08:57:56.958 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_initial_backoff": "0.200000", 2024-03-20T08:57:56.958 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_inject_delay_max": "1.000000", 2024-03-20T08:57:56.958 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_inject_delay_probability": "0.000000", 2024-03-20T08:57:56.958 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_inject_delay_type": "", 2024-03-20T08:57:56.958 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_inject_internal_delays": "0.000000", 2024-03-20T08:57:56.958 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_inject_network_congestion": "0", 2024-03-20T08:57:56.958 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_inject_socket_failures": "2500", 2024-03-20T08:57:56.958 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_learn_addr_from_peer": "true", 2024-03-20T08:57:56.958 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_max_accept_failures": "4", 2024-03-20T08:57:56.958 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_max_backoff": "15.000000", 2024-03-20T08:57:56.958 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_mon_client_mode": "secure crc", 2024-03-20T08:57:56.958 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_mon_cluster_mode": "secure crc", 2024-03-20T08:57:56.958 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_mon_service_mode": "secure crc", 2024-03-20T08:57:56.958 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_osd_compress_min_size": "1024", 2024-03-20T08:57:56.959 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_osd_compress_mode": "none", 2024-03-20T08:57:56.959 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_osd_compression_algorithm": "snappy", 2024-03-20T08:57:56.959 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-03-20T08:57:56.959 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_pq_min_cost": "65536", 2024-03-20T08:57:56.959 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_public_type": "", 2024-03-20T08:57:56.959 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_service_mode": "crc secure", 2024-03-20T08:57:56.959 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_tcp_listen_backlog": "512", 2024-03-20T08:57:56.959 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_tcp_nodelay": "true", 2024-03-20T08:57:56.959 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-03-20T08:57:56.959 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_tcp_rcvbuf": "0", 2024-03-20T08:57:56.959 INFO:teuthology.orchestra.run.smithi026.stdout: "ms_type": "async+posix", 2024-03-20T08:57:56.959 INFO:teuthology.orchestra.run.smithi026.stdout: "no_config_file": "false", 2024-03-20T08:57:56.959 INFO:teuthology.orchestra.run.smithi026.stdout: "objecter_completion_locks_per_session": "32", 2024-03-20T08:57:56.959 INFO:teuthology.orchestra.run.smithi026.stdout: "objecter_debug_inject_relock_delay": "false", 2024-03-20T08:57:56.960 INFO:teuthology.orchestra.run.smithi026.stdout: "objecter_inflight_op_bytes": "104857600", 2024-03-20T08:57:56.960 INFO:teuthology.orchestra.run.smithi026.stdout: "objecter_inflight_ops": "1024", 2024-03-20T08:57:56.960 INFO:teuthology.orchestra.run.smithi026.stdout: "objecter_inject_no_watch_ping": "false", 2024-03-20T08:57:56.960 INFO:teuthology.orchestra.run.smithi026.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-03-20T08:57:56.960 INFO:teuthology.orchestra.run.smithi026.stdout: "objecter_tick_interval": "5.000000", 2024-03-20T08:57:56.960 INFO:teuthology.orchestra.run.smithi026.stdout: "objecter_timeout": "10.000000", 2024-03-20T08:57:56.960 INFO:teuthology.orchestra.run.smithi026.stdout: "objectstore_blackhole": "false", 2024-03-20T08:57:56.960 INFO:teuthology.orchestra.run.smithi026.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-03-20T08:57:56.960 INFO:teuthology.orchestra.run.smithi026.stdout: "openssl_engine_opts": "", 2024-03-20T08:57:56.960 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_agent_delay_time": "5.000000", 2024-03-20T08:57:56.960 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_agent_hist_halflife": "1000", 2024-03-20T08:57:56.960 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_agent_max_low_ops": "2", 2024-03-20T08:57:56.960 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_agent_max_ops": "4", 2024-03-20T08:57:56.960 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-03-20T08:57:56.961 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_agent_quantize_effort": "0.100000", 2024-03-20T08:57:56.961 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_agent_slop": "0.020000", 2024-03-20T08:57:56.961 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-03-20T08:57:56.961 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_allow_recovery_below_min_size": "true", 2024-03-20T08:57:56.961 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_asio_thread_count": "2", 2024-03-20T08:57:56.961 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_async_recovery_min_cost": "100", 2024-03-20T08:57:56.961 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_auto_mark_unfound_lost": "false", 2024-03-20T08:57:56.961 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_backfill_retry_interval": "30.000000", 2024-03-20T08:57:56.961 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_backfill_scan_max": "512", 2024-03-20T08:57:56.961 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_backfill_scan_min": "64", 2024-03-20T08:57:56.961 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_backoff_on_degraded": "false", 2024-03-20T08:57:56.961 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_backoff_on_peering": "false", 2024-03-20T08:57:56.961 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_backoff_on_unfound": "true", 2024-03-20T08:57:56.961 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_beacon_report_interval": "300", 2024-03-20T08:57:56.962 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_bench_duration": "30", 2024-03-20T08:57:56.962 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-03-20T08:57:56.962 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_bench_max_block_size": "67108864", 2024-03-20T08:57:56.962 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_bench_small_size_max_iops": "100", 2024-03-20T08:57:56.962 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_blkin_trace_all": "false", 2024-03-20T08:57:56.962 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_blocked_scrub_grace_period": "120", 2024-03-20T08:57:56.962 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-03-20T08:57:56.962 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-03-20T08:57:56.962 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-03-20T08:57:56.962 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_check_for_log_corruption": "false", 2024-03-20T08:57:56.962 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-03-20T08:57:56.962 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-20T08:57:56.962 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-03-20T08:57:56.962 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-20T08:57:56.963 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_class_update_on_start": "true", 2024-03-20T08:57:56.963 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_client_message_cap": "256", 2024-03-20T08:57:56.963 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_client_message_size_cap": "524288000", 2024-03-20T08:57:56.963 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_client_op_priority": "63", 2024-03-20T08:57:56.963 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_client_watch_timeout": "30", 2024-03-20T08:57:56.963 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_command_max_records": "256", 2024-03-20T08:57:56.963 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_compact_on_start": "false", 2024-03-20T08:57:56.963 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-03-20T08:57:56.963 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_crush_chooseleaf_type": "1", 2024-03-20T08:57:56.963 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_crush_initial_weight": "-1.000000", 2024-03-20T08:57:56.963 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_crush_update_on_start": "true", 2024-03-20T08:57:56.963 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_crush_update_weight_set": "true", 2024-03-20T08:57:56.963 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-03-20T08:57:56.963 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-03-20T08:57:56.964 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-03-20T08:57:56.964 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_drop_ping_duration": "0", 2024-03-20T08:57:56.964 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-03-20T08:57:56.964 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_feed_pullee": "-1", 2024-03-20T08:57:56.964 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-03-20T08:57:56.964 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-03-20T08:57:56.964 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-03-20T08:57:56.964 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_misdirected_ops": "false", 2024-03-20T08:57:56.964 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_no_acting_change": "false", 2024-03-20T08:57:56.964 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_no_purge_strays": "false", 2024-03-20T08:57:56.964 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_op_order": "false", 2024-03-20T08:57:56.964 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_pg_log_writeout": "false", 2024-03-20T08:57:56.964 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_pretend_recovery_active": "false", 2024-03-20T08:57:56.965 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-03-20T08:57:56.965 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-03-20T08:57:56.965 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_shutdown": "false", 2024-03-20T08:57:56.965 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-03-20T08:57:56.965 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-03-20T08:57:56.965 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_trim_objects": "false", 2024-03-20T08:57:56.965 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_verify_cached_snaps": "false", 2024-03-20T08:57:56.965 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_verify_missing_on_start": "false", 2024-03-20T08:57:56.965 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_verify_snaps": "false", 2024-03-20T08:57:56.965 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-03-20T08:57:56.965 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-03-20T08:57:56.965 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_deep_scrub_keys": "1024", 2024-03-20T08:57:56.965 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-03-20T08:57:56.965 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-03-20T08:57:56.966 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-03-20T08:57:56.966 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_deep_scrub_stride": "524288", 2024-03-20T08:57:56.966 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-03-20T08:57:56.966 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_default_data_pool_replay_window": "45", 2024-03-20T08:57:56.966 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_default_notify_timeout": "30", 2024-03-20T08:57:56.966 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_delete_sleep": "0.000000", 2024-03-20T08:57:56.966 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-03-20T08:57:56.966 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-03-20T08:57:56.966 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-03-20T08:57:56.966 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_discard_disconnected_ops": "true", 2024-03-20T08:57:56.966 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_enable_op_tracker": "true", 2024-03-20T08:57:56.966 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-03-20T08:57:56.966 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_extblkdev_plugins": "vdo", 2024-03-20T08:57:56.966 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-03-20T08:57:56.967 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-03-20T08:57:56.967 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_fast_info": "true", 2024-03-20T08:57:56.967 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_fast_shutdown": "true", 2024-03-20T08:57:56.967 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-03-20T08:57:56.967 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_fast_shutdown_timeout": "15", 2024-03-20T08:57:56.967 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-03-20T08:57:56.969 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-03-20T08:57:56.969 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-03-20T08:57:56.969 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_function_tracing": "false", 2024-03-20T08:57:56.969 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_heartbeat_grace": "20", 2024-03-20T08:57:56.970 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_heartbeat_interval": "6", 2024-03-20T08:57:56.970 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-03-20T08:57:56.970 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_heartbeat_min_peers": "10", 2024-03-20T08:57:56.970 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_heartbeat_min_size": "2000", 2024-03-20T08:57:56.970 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_heartbeat_stale": "600", 2024-03-20T08:57:56.970 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-03-20T08:57:56.970 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_hit_set_max_size": "100000", 2024-03-20T08:57:56.970 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_hit_set_min_size": "1000", 2024-03-20T08:57:56.970 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-03-20T08:57:56.970 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-03-20T08:57:56.970 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-03-20T08:57:56.970 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-03-20T08:57:56.970 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-03-20T08:57:56.970 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_journal_flush_on_shutdown": "true", 2024-03-20T08:57:56.971 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_journal_size": "5120", 2024-03-20T08:57:56.971 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_kill_backfill_at": "0", 2024-03-20T08:57:56.971 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_loop_before_reset_tphandle": "64", 2024-03-20T08:57:56.971 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_map_cache_size": "50", 2024-03-20T08:57:56.971 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_map_dedup": "true", 2024-03-20T08:57:56.971 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_map_message_max": "40", 2024-03-20T08:57:56.971 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_map_message_max_bytes": "10485760", 2024-03-20T08:57:56.971 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_map_share_max_epochs": "40", 2024-03-20T08:57:56.971 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_attr_name_len": "100", 2024-03-20T08:57:56.971 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_attr_size": "0", 2024-03-20T08:57:56.971 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_backfills": "1", 2024-03-20T08:57:56.971 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_markdown_count": "5", 2024-03-20T08:57:56.971 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_markdown_period": "600", 2024-03-20T08:57:56.971 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_object_name_len": "2048", 2024-03-20T08:57:56.972 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_object_namespace_len": "256", 2024-03-20T08:57:56.972 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_object_size": "134217728", 2024-03-20T08:57:56.972 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-03-20T08:57:56.972 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_omap_entries_per_request": "1024", 2024-03-20T08:57:56.972 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_pg_blocked_by": "16", 2024-03-20T08:57:56.972 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_pg_log_entries": "10000", 2024-03-20T08:57:56.972 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-03-20T08:57:56.972 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_pgls": "1024", 2024-03-20T08:57:56.972 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_push_cost": "8388608", 2024-03-20T08:57:56.972 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_push_objects": "10", 2024-03-20T08:57:56.972 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_scrubs": "3", 2024-03-20T08:57:56.972 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-03-20T08:57:56.972 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_trimming_pgs": "2", 2024-03-20T08:57:56.972 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_write_op_reply_len": "64", 2024-03-20T08:57:56.973 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_max_write_size": "90", 2024-03-20T08:57:56.973 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-03-20T08:57:56.973 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-03-20T08:57:56.973 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-03-20T08:57:56.973 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-03-20T08:57:56.973 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-03-20T08:57:56.973 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-03-20T08:57:56.973 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-03-20T08:57:56.973 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_override_recovery_settings": "false", 2024-03-20T08:57:56.973 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_profile": "balanced", 2024-03-20T08:57:56.973 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-03-20T08:57:56.973 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-03-20T08:57:56.973 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-03-20T08:57:56.974 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-03-20T08:57:56.974 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-03-20T08:57:56.974 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-03-20T08:57:56.974 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-03-20T08:57:56.974 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-03-20T08:57:56.974 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-03-20T08:57:56.974 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-03-20T08:57:56.974 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mclock_skip_benchmark": "false", 2024-03-20T08:57:56.974 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_memory_base": "805306368", 2024-03-20T08:57:56.974 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_memory_cache_min": "134217728", 2024-03-20T08:57:56.974 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-03-20T08:57:56.974 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-03-20T08:57:56.975 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_memory_target": "4294967296", 2024-03-20T08:57:56.975 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_memory_target_autotune": "false", 2024-03-20T08:57:56.975 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-03-20T08:57:56.975 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_min_pg_log_entries": "250", 2024-03-20T08:57:56.975 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_min_recovery_priority": "0", 2024-03-20T08:57:56.975 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mon_heartbeat_interval": "30", 2024-03-20T08:57:56.975 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-03-20T08:57:56.975 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mon_report_interval": "5", 2024-03-20T08:57:56.975 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mon_report_max_in_flight": "2", 2024-03-20T08:57:56.975 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-03-20T08:57:56.975 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_num_cache_shards": "32", 2024-03-20T08:57:56.975 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_num_op_tracker_shard": "32", 2024-03-20T08:57:56.975 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_numa_auto_affinity": "true", 2024-03-20T08:57:56.975 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_numa_node": "-1", 2024-03-20T08:57:56.976 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_numa_prefer_iface": "true", 2024-03-20T08:57:56.976 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-03-20T08:57:56.976 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_objecter_finishers": "1", 2024-03-20T08:57:56.976 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_objectstore": "bluestore", 2024-03-20T08:57:56.976 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_objectstore_fuse": "false", 2024-03-20T08:57:56.976 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_objectstore_tracing": "false", 2024-03-20T08:57:56.976 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_op_complaint_time": "30.000000", 2024-03-20T08:57:56.976 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_op_history_duration": "600", 2024-03-20T08:57:56.976 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_op_history_size": "20", 2024-03-20T08:57:56.976 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_op_history_slow_op_size": "20", 2024-03-20T08:57:56.976 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-03-20T08:57:56.976 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_op_log_threshold": "5", 2024-03-20T08:57:56.976 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_op_num_shards": "0", 2024-03-20T08:57:56.977 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_op_num_shards_hdd": "5", 2024-03-20T08:57:56.977 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_op_num_shards_ssd": "8", 2024-03-20T08:57:56.977 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_op_num_threads_per_shard": "0", 2024-03-20T08:57:56.977 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-03-20T08:57:56.977 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-03-20T08:57:56.977 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-03-20T08:57:56.977 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_op_pq_min_cost": "65536", 2024-03-20T08:57:56.977 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_op_queue": "mclock_scheduler", 2024-03-20T08:57:56.977 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_op_queue_cut_off": "high", 2024-03-20T08:57:56.977 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_op_thread_suicide_timeout": "150", 2024-03-20T08:57:56.977 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_op_thread_timeout": "15", 2024-03-20T08:57:56.977 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_open_classes_on_start": "true", 2024-03-20T08:57:56.977 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_os_flags": "0", 2024-03-20T08:57:56.977 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_peering_op_priority": "255", 2024-03-20T08:57:56.978 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pg_delete_cost": "1048576", 2024-03-20T08:57:56.978 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pg_delete_priority": "5", 2024-03-20T08:57:56.978 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-03-20T08:57:56.978 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-03-20T08:57:56.978 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pg_log_dups_tracked": "3000", 2024-03-20T08:57:56.978 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pg_log_trim_max": "10000", 2024-03-20T08:57:56.978 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pg_log_trim_min": "100", 2024-03-20T08:57:56.978 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-03-20T08:57:56.978 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pg_object_context_cache_count": "64", 2024-03-20T08:57:56.978 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-03-20T08:57:56.978 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-03-20T08:57:56.978 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-03-20T08:57:56.978 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-03-20T08:57:56.978 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-03-20T08:57:56.979 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-03-20T08:57:56.979 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-03-20T08:57:56.979 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-03-20T08:57:56.979 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_crimson": "false", 2024-03-20T08:57:56.979 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_crush_rule": "-1", 2024-03-20T08:57:56.979 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_ec_fast_read": "false", 2024-03-20T08:57:56.979 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-03-20T08:57:56.979 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_flag_bulk": "false", 2024-03-20T08:57:56.979 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-03-20T08:57:56.979 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_flag_nodelete": "false", 2024-03-20T08:57:56.979 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-03-20T08:57:56.979 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-03-20T08:57:56.979 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_flags": "0", 2024-03-20T08:57:56.979 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-03-20T08:57:56.980 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_min_size": "2", 2024-03-20T08:57:56.980 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-03-20T08:57:56.980 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_pg_num": "32", 2024-03-20T08:57:56.980 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_pgp_num": "0", 2024-03-20T08:57:56.980 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-03-20T08:57:56.980 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_read_ratio": "70", 2024-03-20T08:57:56.980 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_size": "3", 2024-03-20T08:57:56.980 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_default_type": "replicated", 2024-03-20T08:57:56.980 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-03-20T08:57:56.980 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_pool_use_gmt_hitset": "true", 2024-03-20T08:57:56.980 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_push_per_object_cost": "1000", 2024-03-20T08:57:56.980 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_read_ec_check_for_errors": "false", 2024-03-20T08:57:56.980 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_recover_clone_overlap": "true", 2024-03-20T08:57:56.980 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_recover_clone_overlap_limit": "10", 2024-03-20T08:57:56.981 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_recovery_cost": "20971520", 2024-03-20T08:57:56.981 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_recovery_delay_start": "0.000000", 2024-03-20T08:57:56.981 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_recovery_max_active": "0", 2024-03-20T08:57:56.981 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_recovery_max_active_hdd": "3", 2024-03-20T08:57:56.981 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_recovery_max_active_ssd": "10", 2024-03-20T08:57:56.981 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_recovery_max_chunk": "8388608", 2024-03-20T08:57:56.981 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-03-20T08:57:56.981 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_recovery_max_single_start": "1", 2024-03-20T08:57:56.981 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_recovery_op_priority": "3", 2024-03-20T08:57:56.981 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_recovery_op_warn_multiple": "16", 2024-03-20T08:57:56.981 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_recovery_priority": "5", 2024-03-20T08:57:56.981 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_recovery_retry_interval": "30.000000", 2024-03-20T08:57:56.981 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_recovery_sleep": "0.000000", 2024-03-20T08:57:56.981 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-03-20T08:57:56.982 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-03-20T08:57:56.982 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-03-20T08:57:56.982 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_repair_during_recovery": "false", 2024-03-20T08:57:56.982 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_requested_scrub_priority": "120", 2024-03-20T08:57:56.982 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-03-20T08:57:56.982 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_rollback_to_cluster_snap": "", 2024-03-20T08:57:56.982 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_auto_repair": "false", 2024-03-20T08:57:56.982 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-03-20T08:57:56.982 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-03-20T08:57:56.982 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_begin_hour": "0", 2024-03-20T08:57:56.982 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_begin_week_day": "0", 2024-03-20T08:57:56.982 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_chunk_max": "25", 2024-03-20T08:57:56.983 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_chunk_min": "5", 2024-03-20T08:57:56.983 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_cost": "52428800", 2024-03-20T08:57:56.983 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-03-20T08:57:56.983 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_during_recovery": "false", 2024-03-20T08:57:56.983 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_end_hour": "0", 2024-03-20T08:57:56.983 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_end_week_day": "0", 2024-03-20T08:57:56.983 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_event_cost": "4096", 2024-03-20T08:57:56.983 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-03-20T08:57:56.983 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-03-20T08:57:56.983 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_invalid_stats": "true", 2024-03-20T08:57:56.983 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_load_threshold": "0.500000", 2024-03-20T08:57:56.983 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_max_interval": "604800.000000", 2024-03-20T08:57:56.983 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_max_preemptions": "5", 2024-03-20T08:57:56.983 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_min_interval": "86400.000000", 2024-03-20T08:57:56.984 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_priority": "5", 2024-03-20T08:57:56.984 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_reservation_timeout": "300000", 2024-03-20T08:57:56.984 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_sleep": "0.000000", 2024-03-20T08:57:56.984 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_scrub_slow_reservation_response": "30000", 2024-03-20T08:57:56.984 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-03-20T08:57:56.984 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-03-20T08:57:56.984 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_shutdown_pgref_assert": "false", 2024-03-20T08:57:56.984 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-03-20T08:57:56.984 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_skip_data_digest": "false", 2024-03-20T08:57:56.984 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_smart_report_timeout": "5", 2024-03-20T08:57:56.984 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_snap_trim_cost": "1048576", 2024-03-20T08:57:56.984 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_snap_trim_priority": "5", 2024-03-20T08:57:56.984 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_snap_trim_sleep": "0.000000", 2024-03-20T08:57:56.984 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-03-20T08:57:56.985 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-03-20T08:57:56.985 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-03-20T08:57:56.985 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-03-20T08:57:56.985 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_stats_update_period_scrubbing": "15", 2024-03-20T08:57:56.985 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-03-20T08:57:56.985 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_target_transaction_size": "30", 2024-03-20T08:57:56.985 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-03-20T08:57:56.985 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-03-20T08:57:56.985 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-03-20T08:57:56.985 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-03-20T08:57:56.985 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-03-20T08:57:56.985 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-03-20T08:57:56.985 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-03-20T08:57:56.986 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_tier_default_cache_mode": "writeback", 2024-03-20T08:57:56.986 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-03-20T08:57:56.986 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-03-20T08:57:56.986 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_tracing": "false", 2024-03-20T08:57:56.986 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_use_stale_snap": "false", 2024-03-20T08:57:56.986 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-03-20T08:57:56.986 INFO:teuthology.orchestra.run.smithi026.stdout: "osdc_blkin_trace_all": "false", 2024-03-20T08:57:56.986 INFO:teuthology.orchestra.run.smithi026.stdout: "paxos_kill_at": "0", 2024-03-20T08:57:56.986 INFO:teuthology.orchestra.run.smithi026.stdout: "paxos_max_join_drift": "10", 2024-03-20T08:57:56.986 INFO:teuthology.orchestra.run.smithi026.stdout: "paxos_min": "500", 2024-03-20T08:57:56.987 INFO:teuthology.orchestra.run.smithi026.stdout: "paxos_min_wait": "0.050000", 2024-03-20T08:57:56.987 INFO:teuthology.orchestra.run.smithi026.stdout: "paxos_propose_interval": "1.000000", 2024-03-20T08:57:56.987 INFO:teuthology.orchestra.run.smithi026.stdout: "paxos_service_trim_max": "500", 2024-03-20T08:57:56.987 INFO:teuthology.orchestra.run.smithi026.stdout: "paxos_service_trim_max_multiplier": "20", 2024-03-20T08:57:56.987 INFO:teuthology.orchestra.run.smithi026.stdout: "paxos_service_trim_min": "10", 2024-03-20T08:57:56.987 INFO:teuthology.orchestra.run.smithi026.stdout: "paxos_stash_full_interval": "25", 2024-03-20T08:57:56.987 INFO:teuthology.orchestra.run.smithi026.stdout: "paxos_trim_max": "500", 2024-03-20T08:57:56.987 INFO:teuthology.orchestra.run.smithi026.stdout: "paxos_trim_min": "250", 2024-03-20T08:57:56.987 INFO:teuthology.orchestra.run.smithi026.stdout: "perf": "true", 2024-03-20T08:57:56.987 INFO:teuthology.orchestra.run.smithi026.stdout: "pid_file": "", 2024-03-20T08:57:56.988 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-03-20T08:57:56.988 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-03-20T08:57:56.988 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "-", 2024-03-20T08:57:56.988 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrv": "", 2024-03-20T08:57:56.988 INFO:teuthology.orchestra.run.smithi026.stdout: "public_bind_addr": "-", 2024-03-20T08:57:56.988 INFO:teuthology.orchestra.run.smithi026.stdout: "public_network": "", 2024-03-20T08:57:56.988 INFO:teuthology.orchestra.run.smithi026.stdout: "public_network_interface": "", 2024-03-20T08:57:56.988 INFO:teuthology.orchestra.run.smithi026.stdout: "qat_compressor_enabled": "false", 2024-03-20T08:57:56.988 INFO:teuthology.orchestra.run.smithi026.stdout: "qat_compressor_session_max_number": "256", 2024-03-20T08:57:56.988 INFO:teuthology.orchestra.run.smithi026.stdout: "rados_mon_op_timeout": "0", 2024-03-20T08:57:56.988 INFO:teuthology.orchestra.run.smithi026.stdout: "rados_osd_op_timeout": "0", 2024-03-20T08:57:56.988 INFO:teuthology.orchestra.run.smithi026.stdout: "rados_tracing": "false", 2024-03-20T08:57:56.989 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_atime_update_interval": "60", 2024-03-20T08:57:56.989 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-03-20T08:57:56.989 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_balance_parent_reads": "false", 2024-03-20T08:57:56.989 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_balance_snap_reads": "false", 2024-03-20T08:57:56.989 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_blkin_trace_all": "false", 2024-03-20T08:57:56.989 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_blocklist_expire_seconds": "0", 2024-03-20T08:57:56.989 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_blocklist_on_break_lock": "true", 2024-03-20T08:57:56.989 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_cache": "true", 2024-03-20T08:57:56.989 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_cache_block_writes_upfront": "false", 2024-03-20T08:57:56.989 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_cache_max_dirty": "25165824", 2024-03-20T08:57:56.989 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-03-20T08:57:56.989 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_cache_max_dirty_object": "0", 2024-03-20T08:57:56.989 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_cache_policy": "writearound", 2024-03-20T08:57:56.989 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_cache_size": "33554432", 2024-03-20T08:57:56.990 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_cache_target_dirty": "16777216", 2024-03-20T08:57:56.990 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-03-20T08:57:56.990 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_clone_copy_on_read": "false", 2024-03-20T08:57:56.990 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_compression_hint": "none", 2024-03-20T08:57:56.990 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_concurrent_management_ops": "10", 2024-03-20T08:57:56.990 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-03-20T08:57:56.990 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_default_clone_format": "auto", 2024-03-20T08:57:56.990 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_default_data_pool": "", 2024-03-20T08:57:56.990 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_default_features": "61", 2024-03-20T08:57:56.990 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_default_format": "2", 2024-03-20T08:57:56.990 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_default_map_options": "", 2024-03-20T08:57:56.990 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_default_order": "22", 2024-03-20T08:57:56.990 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_default_pool": "rbd", 2024-03-20T08:57:56.991 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-03-20T08:57:56.991 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_default_stripe_count": "0", 2024-03-20T08:57:56.991 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_default_stripe_unit": "0", 2024-03-20T08:57:56.991 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_disable_zero_copy_writes": "true", 2024-03-20T08:57:56.991 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_discard_granularity_bytes": "65536", 2024-03-20T08:57:56.991 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-03-20T08:57:56.991 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_enable_alloc_hint": "true", 2024-03-20T08:57:56.991 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-03-20T08:57:56.991 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_io_scheduler": "simple", 2024-03-20T08:57:56.991 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-03-20T08:57:56.991 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_journal_commit_age": "5.000000", 2024-03-20T08:57:56.991 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-03-20T08:57:56.991 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-03-20T08:57:56.991 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-03-20T08:57:56.992 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-03-20T08:57:56.992 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_journal_object_flush_interval": "0", 2024-03-20T08:57:56.992 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-03-20T08:57:56.992 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-03-20T08:57:56.992 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_journal_order": "24", 2024-03-20T08:57:56.992 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_journal_pool": "", 2024-03-20T08:57:56.992 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_journal_splay_width": "4", 2024-03-20T08:57:56.992 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_localize_parent_reads": "false", 2024-03-20T08:57:56.992 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_localize_snap_reads": "false", 2024-03-20T08:57:56.992 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-03-20T08:57:56.992 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-03-20T08:57:56.992 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-03-20T08:57:56.992 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-03-20T08:57:56.992 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-03-20T08:57:56.992 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-03-20T08:57:56.993 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_image_policy_type": "simple", 2024-03-20T08:57:56.993 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-03-20T08:57:56.993 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-03-20T08:57:56.993 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-03-20T08:57:56.993 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-03-20T08:57:56.993 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-03-20T08:57:56.993 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-03-20T08:57:56.993 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-03-20T08:57:56.993 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_memory_autotune": "true", 2024-03-20T08:57:56.993 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_memory_base": "805306368", 2024-03-20T08:57:56.993 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-03-20T08:57:56.993 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-03-20T08:57:56.993 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-03-20T08:57:56.993 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-03-20T08:57:56.994 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_memory_target": "4294967296", 2024-03-20T08:57:56.994 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-03-20T08:57:56.994 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-03-20T08:57:56.994 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-03-20T08:57:56.994 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirroring_delete_delay": "0", 2024-03-20T08:57:56.994 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-03-20T08:57:56.994 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirroring_replay_delay": "0", 2024-03-20T08:57:56.994 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-03-20T08:57:56.994 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-03-20T08:57:56.994 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_move_to_trash_on_remove": "false", 2024-03-20T08:57:56.994 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-03-20T08:57:56.994 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_mtime_update_interval": "60", 2024-03-20T08:57:56.994 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_non_blocking_aio": "true", 2024-03-20T08:57:56.995 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_op_thread_timeout": "60", 2024-03-20T08:57:56.995 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_op_threads": "1", 2024-03-20T08:57:56.995 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_parent_cache_enabled": "false", 2024-03-20T08:57:56.995 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_persistent_cache_mode": "disabled", 2024-03-20T08:57:56.995 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_persistent_cache_path": "/tmp", 2024-03-20T08:57:56.995 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_persistent_cache_size": "1073741824", 2024-03-20T08:57:56.995 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_plugins": "", 2024-03-20T08:57:56.995 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_bps_burst": "0", 2024-03-20T08:57:56.995 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-03-20T08:57:56.995 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_bps_limit": "0", 2024-03-20T08:57:56.995 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_exclude_ops": "0", 2024-03-20T08:57:56.995 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_iops_burst": "0", 2024-03-20T08:57:56.995 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-03-20T08:57:56.996 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_iops_limit": "0", 2024-03-20T08:57:56.996 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_read_bps_burst": "0", 2024-03-20T08:57:56.996 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-03-20T08:57:56.996 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_read_bps_limit": "0", 2024-03-20T08:57:56.996 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_read_iops_burst": "0", 2024-03-20T08:57:56.996 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-03-20T08:57:56.996 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_read_iops_limit": "0", 2024-03-20T08:57:56.996 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_schedule_tick_min": "50", 2024-03-20T08:57:56.996 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_write_bps_burst": "0", 2024-03-20T08:57:56.996 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-03-20T08:57:56.996 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_write_bps_limit": "0", 2024-03-20T08:57:56.996 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_write_iops_burst": "0", 2024-03-20T08:57:56.996 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-03-20T08:57:56.997 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_qos_write_iops_limit": "0", 2024-03-20T08:57:56.997 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_quiesce_notification_attempts": "10", 2024-03-20T08:57:56.997 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_read_from_replica_policy": "default", 2024-03-20T08:57:56.997 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-03-20T08:57:56.997 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_readahead_max_bytes": "524288", 2024-03-20T08:57:56.997 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_readahead_trigger_requests": "10", 2024-03-20T08:57:56.997 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_request_timed_out_seconds": "30", 2024-03-20T08:57:56.997 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_skip_partial_discard": "true", 2024-03-20T08:57:56.997 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-03-20T08:57:56.997 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_tracing": "false", 2024-03-20T08:57:56.997 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_validate_names": "true", 2024-03-20T08:57:56.997 INFO:teuthology.orchestra.run.smithi026.stdout: "rbd_validate_pool": "true", 2024-03-20T08:57:56.997 INFO:teuthology.orchestra.run.smithi026.stdout: "restapi_base_url": "", 2024-03-20T08:57:56.997 INFO:teuthology.orchestra.run.smithi026.stdout: "restapi_log_level": "", 2024-03-20T08:57:56.998 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_acl_grants_max_num": "100", 2024-03-20T08:57:56.998 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_admin_entry": "admin", 2024-03-20T08:57:56.998 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-03-20T08:57:56.998 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_backend_store": "rados", 2024-03-20T08:57:56.998 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_barbican_url": "", 2024-03-20T08:57:56.998 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_beast_enable_async": "true", 2024-03-20T08:57:56.998 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_bucket_counters_cache": "false", 2024-03-20T08:57:56.998 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-03-20T08:57:56.998 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-03-20T08:57:56.998 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-03-20T08:57:56.998 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_bucket_index_max_aio": "128", 2024-03-20T08:57:56.998 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-03-20T08:57:56.998 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-03-20T08:57:56.999 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_bucket_quota_ttl": "600", 2024-03-20T08:57:56.999 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-03-20T08:57:56.999 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_cache_enabled": "true", 2024-03-20T08:57:56.999 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_cache_expiry_interval": "900", 2024-03-20T08:57:56.999 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_cache_lru_size": "10000", 2024-03-20T08:57:56.999 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_config_store": "rados", 2024-03-20T08:57:56.999 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_content_length_compat": "false", 2024-03-20T08:57:56.999 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_copy_obj_progress": "true", 2024-03-20T08:57:56.999 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-03-20T08:57:56.999 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_cors_rules_max_num": "100", 2024-03-20T08:57:56.999 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_cross_domain_policy": "", 2024-03-20T08:57:56.999 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_default_encryption_key": "", 2024-03-20T08:57:56.999 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_kmip_addr": "", 2024-03-20T08:57:57.000 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_kmip_ca_path": "", 2024-03-20T08:57:57.000 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_kmip_client_cert": "", 2024-03-20T08:57:57.000 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_kmip_client_key": "", 2024-03-20T08:57:57.000 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-03-20T08:57:57.000 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_kmip_password": "", 2024-03-20T08:57:57.000 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-03-20T08:57:57.000 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_kmip_username": "", 2024-03-20T08:57:57.000 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_require_ssl": "true", 2024-03-20T08:57:57.000 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-03-20T08:57:57.000 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-03-20T08:57:57.000 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-03-20T08:57:57.000 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-03-20T08:57:57.000 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-03-20T08:57:57.001 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-03-20T08:57:57.001 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-03-20T08:57:57.001 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-03-20T08:57:57.001 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-03-20T08:57:57.001 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-03-20T08:57:57.001 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-03-20T08:57:57.001 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-03-20T08:57:57.001 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-03-20T08:57:57.001 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-03-20T08:57:57.001 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_suppress_logs": "true", 2024-03-20T08:57:57.001 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_vault_addr": "", 2024-03-20T08:57:57.001 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_vault_auth": "token", 2024-03-20T08:57:57.001 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_vault_namespace": "", 2024-03-20T08:57:57.001 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_vault_prefix": "", 2024-03-20T08:57:57.001 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-03-20T08:57:57.002 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-03-20T08:57:57.002 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-03-20T08:57:57.002 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-03-20T08:57:57.002 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_vault_token_file": "", 2024-03-20T08:57:57.002 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-03-20T08:57:57.002 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_curl_buffersize": "524288", 2024-03-20T08:57:57.002 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_curl_low_speed_limit": "1024", 2024-03-20T08:57:57.002 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_curl_low_speed_time": "300", 2024-03-20T08:57:57.002 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_curl_tcp_keepalive": "0", 2024-03-20T08:57:57.002 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-03-20T08:57:57.002 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-03-20T08:57:57.002 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-03-20T08:57:57.002 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-03-20T08:57:57.002 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-03-20T08:57:57.003 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_d3n_l1_fadvise": "4", 2024-03-20T08:57:57.003 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-03-20T08:57:57.003 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-03-20T08:57:57.003 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-03-20T08:57:57.003 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_d4n_host": "127.0.0.1", 2024-03-20T08:57:57.003 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_d4n_port": "6379", 2024-03-20T08:57:57.003 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-03-20T08:57:57.003 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_data_log_changes_size": "1000", 2024-03-20T08:57:57.003 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_data_log_num_shards": "128", 2024-03-20T08:57:57.003 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-03-20T08:57:57.003 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_data_log_window": "30", 2024-03-20T08:57:57.003 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_data_notify_interval_msec": "0", 2024-03-20T08:57:57.003 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_data_sync_poll_interval": "20", 2024-03-20T08:57:57.003 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_data_sync_spawn_window": "20", 2024-03-20T08:57:57.004 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-03-20T08:57:57.004 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-03-20T08:57:57.004 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-03-20T08:57:57.004 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_default_data_log_backing": "fifo", 2024-03-20T08:57:57.004 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-03-20T08:57:57.004 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_default_region_info_oid": "default.region", 2024-03-20T08:57:57.004 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-03-20T08:57:57.004 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-03-20T08:57:57.004 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_defer_to_bucket_acls": "", 2024-03-20T08:57:57.004 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-03-20T08:57:57.004 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-03-20T08:57:57.004 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-03-20T08:57:57.004 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-03-20T08:57:57.005 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-03-20T08:57:57.005 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-03-20T08:57:57.005 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-03-20T08:57:57.005 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-03-20T08:57:57.005 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_dmclock_data_res": "500.000000", 2024-03-20T08:57:57.005 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-03-20T08:57:57.005 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-03-20T08:57:57.005 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-03-20T08:57:57.005 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-03-20T08:57:57.005 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_dns_name": "", 2024-03-20T08:57:57.005 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_dns_s3website_name": "", 2024-03-20T08:57:57.005 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_dynamic_resharding": "true", 2024-03-20T08:57:57.005 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-03-20T08:57:57.005 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_enable_gc_threads": "true", 2024-03-20T08:57:57.005 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_enable_lc_threads": "true", 2024-03-20T08:57:57.006 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_enable_ops_log": "false", 2024-03-20T08:57:57.006 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_enable_quota_threads": "true", 2024-03-20T08:57:57.006 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_enable_static_website": "false", 2024-03-20T08:57:57.006 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_enable_usage_log": "false", 2024-03-20T08:57:57.006 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_enforce_swift_acls": "true", 2024-03-20T08:57:57.006 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_exit_timeout_secs": "120", 2024-03-20T08:57:57.006 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_expose_bucket": "false", 2024-03-20T08:57:57.006 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_extended_http_attrs": "", 2024-03-20T08:57:57.006 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_filter": "none", 2024-03-20T08:57:57.006 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-03-20T08:57:57.006 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_frontends": "beast port=7480", 2024-03-20T08:57:57.006 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_gc_max_concurrent_io": "10", 2024-03-20T08:57:57.006 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_gc_max_deferred": "50", 2024-03-20T08:57:57.007 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-03-20T08:57:57.007 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_gc_max_objs": "32", 2024-03-20T08:57:57.007 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_gc_max_queue_size": "134213632", 2024-03-20T08:57:57.007 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_gc_max_trim_chunk": "16", 2024-03-20T08:57:57.007 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_gc_obj_min_wait": "7200", 2024-03-20T08:57:57.007 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_gc_processor_max_time": "3600", 2024-03-20T08:57:57.007 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_gc_processor_period": "3600", 2024-03-20T08:57:57.007 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-03-20T08:57:57.007 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_get_obj_window_size": "16777216", 2024-03-20T08:57:57.007 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_healthcheck_disabling_path": "", 2024-03-20T08:57:57.007 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_ignore_get_invalid_range": "false", 2024-03-20T08:57:57.007 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_init_timeout": "300", 2024-03-20T08:57:57.007 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-03-20T08:57:57.007 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-03-20T08:57:57.007 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_kafka_connection_idle": "300", 2024-03-20T08:57:57.008 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_kafka_message_timeout": "5000", 2024-03-20T08:57:57.008 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_kafka_sleep_timeout": "10", 2024-03-20T08:57:57.008 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-03-20T08:57:57.008 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-03-20T08:57:57.008 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-03-20T08:57:57.008 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_admin_domain": "", 2024-03-20T08:57:57.008 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_admin_password": "", 2024-03-20T08:57:57.008 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_admin_password_path": "", 2024-03-20T08:57:57.008 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_admin_project": "", 2024-03-20T08:57:57.008 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_admin_tenant": "", 2024-03-20T08:57:57.008 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_admin_token": "", 2024-03-20T08:57:57.008 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_admin_token_path": "", 2024-03-20T08:57:57.008 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_admin_user": "", 2024-03-20T08:57:57.008 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_api_version": "2", 2024-03-20T08:57:57.009 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_barbican_domain": "", 2024-03-20T08:57:57.009 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_barbican_password": "", 2024-03-20T08:57:57.009 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_barbican_project": "", 2024-03-20T08:57:57.009 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_barbican_tenant": "", 2024-03-20T08:57:57.009 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_barbican_user": "", 2024-03-20T08:57:57.009 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-03-20T08:57:57.009 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_implicit_tenants": "false", 2024-03-20T08:57:57.009 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-03-20T08:57:57.009 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_service_token_enabled": "false", 2024-03-20T08:57:57.009 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_token_cache_size": "10000", 2024-03-20T08:57:57.009 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_url": "", 2024-03-20T08:57:57.009 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_keystone_verify_ssl": "true", 2024-03-20T08:57:57.009 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_lc_debug_interval": "-1", 2024-03-20T08:57:57.009 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_lc_lock_max_time": "90", 2024-03-20T08:57:57.010 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_lc_max_objs": "32", 2024-03-20T08:57:57.010 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_lc_max_rules": "1000", 2024-03-20T08:57:57.010 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_lc_max_worker": "3", 2024-03-20T08:57:57.010 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_lc_max_wp_worker": "3", 2024-03-20T08:57:57.010 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_lc_thread_delay": "0", 2024-03-20T08:57:57.010 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-03-20T08:57:57.010 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_ldap_dnattr": "uid", 2024-03-20T08:57:57.010 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-03-20T08:57:57.010 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_ldap_searchfilter": "", 2024-03-20T08:57:57.010 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-03-20T08:57:57.010 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_ldap_uri": "ldaps://", 2024-03-20T08:57:57.010 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-03-20T08:57:57.010 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-03-20T08:57:57.011 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-03-20T08:57:57.011 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_log_http_headers": "", 2024-03-20T08:57:57.011 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_log_nonexistent_bucket": "false", 2024-03-20T08:57:57.011 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-03-20T08:57:57.011 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_log_object_name_utc": "false", 2024-03-20T08:57:57.011 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-03-20T08:57:57.011 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-03-20T08:57:57.011 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_max_attr_name_len": "0", 2024-03-20T08:57:57.011 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_max_attr_size": "0", 2024-03-20T08:57:57.011 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_max_attrs_num_in_req": "0", 2024-03-20T08:57:57.011 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_max_chunk_size": "4194304", 2024-03-20T08:57:57.011 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_max_concurrent_requests": "1024", 2024-03-20T08:57:57.011 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-03-20T08:57:57.012 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_max_dynamic_shards": "1999", 2024-03-20T08:57:57.012 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_max_listing_results": "1000", 2024-03-20T08:57:57.012 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_max_notify_retries": "10", 2024-03-20T08:57:57.012 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_max_objs_per_shard": "100000", 2024-03-20T08:57:57.012 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_max_put_param_size": "1048576", 2024-03-20T08:57:57.012 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_max_put_size": "5368709120", 2024-03-20T08:57:57.012 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_max_slo_entries": "1000", 2024-03-20T08:57:57.012 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_md_log_max_shards": "64", 2024-03-20T08:57:57.012 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_md_notify_interval_msec": "200", 2024-03-20T08:57:57.012 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_meta_sync_poll_interval": "20", 2024-03-20T08:57:57.012 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_meta_sync_spawn_window": "20", 2024-03-20T08:57:57.012 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-03-20T08:57:57.012 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_mp_lock_max_time": "600", 2024-03-20T08:57:57.012 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-03-20T08:57:57.013 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_multipart_min_part_size": "5242880", 2024-03-20T08:57:57.013 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-03-20T08:57:57.013 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-03-20T08:57:57.013 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_nfs_fhcache_size": "2017", 2024-03-20T08:57:57.013 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-03-20T08:57:57.013 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-03-20T08:57:57.013 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_nfs_lru_lanes": "5", 2024-03-20T08:57:57.013 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_nfs_max_gc": "300", 2024-03-20T08:57:57.013 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-03-20T08:57:57.013 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_nfs_run_gc_threads": "false", 2024-03-20T08:57:57.013 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_nfs_run_lc_threads": "false", 2024-03-20T08:57:57.013 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_nfs_run_quota_threads": "false", 2024-03-20T08:57:57.013 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_nfs_run_sync_thread": "false", 2024-03-20T08:57:57.014 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-03-20T08:57:57.014 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-03-20T08:57:57.014 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_num_async_rados_threads": "32", 2024-03-20T08:57:57.014 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_num_control_oids": "8", 2024-03-20T08:57:57.014 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_numa_node": "-1", 2024-03-20T08:57:57.014 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_obj_stripe_size": "4194304", 2024-03-20T08:57:57.014 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-03-20T08:57:57.014 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_objexp_chunk_size": "100", 2024-03-20T08:57:57.014 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_objexp_gc_interval": "600", 2024-03-20T08:57:57.014 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_objexp_hints_num_shards": "127", 2024-03-20T08:57:57.014 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-03-20T08:57:57.014 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-03-20T08:57:57.014 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_op_thread_timeout": "600", 2024-03-20T08:57:57.014 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_op_tracing": "false", 2024-03-20T08:57:57.015 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_opa_token": "", 2024-03-20T08:57:57.015 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_opa_url": "", 2024-03-20T08:57:57.015 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_opa_verify_ssl": "true", 2024-03-20T08:57:57.015 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-03-20T08:57:57.015 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-03-20T08:57:57.015 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_ops_log_rados": "false", 2024-03-20T08:57:57.015 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_ops_log_socket_path": "", 2024-03-20T08:57:57.015 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-03-20T08:57:57.015 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-03-20T08:57:57.015 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-03-20T08:57:57.015 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_period_push_interval": "2.000000", 2024-03-20T08:57:57.015 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_period_push_interval_max": "30.000000", 2024-03-20T08:57:57.015 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_period_root_pool": ".rgw.root", 2024-03-20T08:57:57.016 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-03-20T08:57:57.016 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-03-20T08:57:57.016 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_posix_cache_lanes": "3", 2024-03-20T08:57:57.016 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-03-20T08:57:57.016 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_posix_cache_max_buckets": "100", 2024-03-20T08:57:57.016 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_posix_cache_partitions": "3", 2024-03-20T08:57:57.016 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-03-20T08:57:57.016 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_print_continue": "true", 2024-03-20T08:57:57.016 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_print_prohibited_content_length": "false", 2024-03-20T08:57:57.016 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-03-20T08:57:57.016 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-03-20T08:57:57.016 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-03-20T08:57:57.016 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-03-20T08:57:57.016 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_rados_tracing": "false", 2024-03-20T08:57:57.017 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_realm": "", 2024-03-20T08:57:57.017 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_realm_id": "", 2024-03-20T08:57:57.017 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-03-20T08:57:57.017 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_region": "", 2024-03-20T08:57:57.017 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_region_root_pool": ".rgw.root", 2024-03-20T08:57:57.017 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_relaxed_region_enforcement": "false", 2024-03-20T08:57:57.017 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-03-20T08:57:57.017 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-03-20T08:57:57.017 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_request_uri": "", 2024-03-20T08:57:57.017 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_reshard_batch_size": "64", 2024-03-20T08:57:57.017 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-03-20T08:57:57.017 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_reshard_max_aio": "128", 2024-03-20T08:57:57.017 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_reshard_num_logs": "16", 2024-03-20T08:57:57.017 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_reshard_thread_interval": "600", 2024-03-20T08:57:57.018 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_resolve_cname": "false", 2024-03-20T08:57:57.018 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_rest_getusage_op_compat": "false", 2024-03-20T08:57:57.018 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_run_sync_thread": "true", 2024-03-20T08:57:57.018 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-03-20T08:57:57.018 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_s3_auth_use_keystone": "false", 2024-03-20T08:57:57.018 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_s3_auth_use_ldap": "false", 2024-03-20T08:57:57.018 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_s3_auth_use_rados": "true", 2024-03-20T08:57:57.018 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_s3_auth_use_sts": "false", 2024-03-20T08:57:57.018 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-03-20T08:57:57.018 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_s3_success_create_obj_status": "0", 2024-03-20T08:57:57.018 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-03-20T08:57:57.018 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_scheduler_type": "throttler", 2024-03-20T08:57:57.018 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_script_uri": "", 2024-03-20T08:57:57.019 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_service_provider_name": "", 2024-03-20T08:57:57.019 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-03-20T08:57:57.019 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_sts_client_id": "", 2024-03-20T08:57:57.019 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_sts_client_secret": "", 2024-03-20T08:57:57.019 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_sts_entry": "sts", 2024-03-20T08:57:57.019 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_sts_key": "sts", 2024-03-20T08:57:57.019 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_sts_max_session_duration": "43200", 2024-03-20T08:57:57.019 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_sts_min_session_duration": "900", 2024-03-20T08:57:57.019 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_sts_token_introspection_url": "", 2024-03-20T08:57:57.019 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_swift_account_in_url": "false", 2024-03-20T08:57:57.019 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_swift_auth_entry": "auth", 2024-03-20T08:57:57.019 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_swift_auth_url": "", 2024-03-20T08:57:57.019 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_swift_custom_header": "", 2024-03-20T08:57:57.019 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_swift_enforce_content_length": "false", 2024-03-20T08:57:57.020 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_swift_need_stats": "true", 2024-03-20T08:57:57.020 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_swift_tenant_name": "", 2024-03-20T08:57:57.020 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_swift_token_expiration": "86400", 2024-03-20T08:57:57.020 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_swift_url": "", 2024-03-20T08:57:57.020 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_swift_url_prefix": "swift", 2024-03-20T08:57:57.020 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_swift_versioning_enabled": "false", 2024-03-20T08:57:57.020 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-03-20T08:57:57.020 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-03-20T08:57:57.020 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_sync_lease_period": "120", 2024-03-20T08:57:57.020 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-03-20T08:57:57.020 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_sync_log_trim_interval": "1200", 2024-03-20T08:57:57.020 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-03-20T08:57:57.020 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-03-20T08:57:57.021 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-03-20T08:57:57.021 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_sync_obj_etag_verify": "false", 2024-03-20T08:57:57.021 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_sync_trace_history_size": "4096", 2024-03-20T08:57:57.021 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-03-20T08:57:57.021 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-03-20T08:57:57.021 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_thread_pool_size": "512", 2024-03-20T08:57:57.021 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_topic_persistency_max_retries": "0", 2024-03-20T08:57:57.021 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-03-20T08:57:57.021 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-03-20T08:57:57.021 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_topic_require_publish_policy": "false", 2024-03-20T08:57:57.021 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_torrent_comment": "", 2024-03-20T08:57:57.021 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_torrent_createby": "", 2024-03-20T08:57:57.021 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_torrent_encoding": "", 2024-03-20T08:57:57.021 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_torrent_flag": "false", 2024-03-20T08:57:57.021 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_torrent_max_size": "5368709120", 2024-03-20T08:57:57.022 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_torrent_origin": "", 2024-03-20T08:57:57.022 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_torrent_sha_unit": "524288", 2024-03-20T08:57:57.022 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_torrent_tracker": "", 2024-03-20T08:57:57.022 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_trust_forwarded_https": "false", 2024-03-20T08:57:57.022 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-03-20T08:57:57.022 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_usage_log_tick_interval": "30", 2024-03-20T08:57:57.022 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_usage_max_shards": "32", 2024-03-20T08:57:57.022 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_usage_max_user_shards": "1", 2024-03-20T08:57:57.022 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_use_opa_authz": "false", 2024-03-20T08:57:57.022 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_user_counters_cache": "false", 2024-03-20T08:57:57.022 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_user_counters_cache_size": "10000", 2024-03-20T08:57:57.022 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-03-20T08:57:57.022 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_user_default_quota_max_size": "-1", 2024-03-20T08:57:57.023 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_user_max_buckets": "1000", 2024-03-20T08:57:57.023 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_user_policies_max_num": "100", 2024-03-20T08:57:57.023 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-03-20T08:57:57.023 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-03-20T08:57:57.023 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_user_quota_sync_interval": "86400", 2024-03-20T08:57:57.023 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-03-20T08:57:57.023 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_user_unique_email": "true", 2024-03-20T08:57:57.023 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_verify_ssl": "true", 2024-03-20T08:57:57.023 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_website_routing_rules_max_num": "50", 2024-03-20T08:57:57.023 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_zone": "", 2024-03-20T08:57:57.023 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_zone_id": "", 2024-03-20T08:57:57.023 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-03-20T08:57:57.023 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_zonegroup": "", 2024-03-20T08:57:57.023 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_zonegroup_id": "", 2024-03-20T08:57:57.024 INFO:teuthology.orchestra.run.smithi026.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-03-20T08:57:57.024 INFO:teuthology.orchestra.run.smithi026.stdout: "rgwlc_auto_session_clear": "true", 2024-03-20T08:57:57.024 INFO:teuthology.orchestra.run.smithi026.stdout: "rgwlc_skip_bucket_step": "false", 2024-03-20T08:57:57.024 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_block_size": "4096", 2024-03-20T08:57:57.024 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-03-20T08:57:57.024 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-03-20T08:57:57.024 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-03-20T08:57:57.024 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-03-20T08:57:57.024 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_cache_shard_bits": "4", 2024-03-20T08:57:57.024 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_cache_size": "536870912", 2024-03-20T08:57:57.024 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_cache_type": "binned_lru", 2024-03-20T08:57:57.024 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-03-20T08:57:57.024 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-03-20T08:57:57.025 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-03-20T08:57:57.025 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_collect_compaction_stats": "false", 2024-03-20T08:57:57.025 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_collect_extended_stats": "false", 2024-03-20T08:57:57.025 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_collect_memory_stats": "false", 2024-03-20T08:57:57.025 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-03-20T08:57:57.025 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_index_type": "binary_search", 2024-03-20T08:57:57.025 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_log_to_ceph_log": "true", 2024-03-20T08:57:57.025 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_metadata_block_size": "4096", 2024-03-20T08:57:57.025 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_partition_filters": "false", 2024-03-20T08:57:57.025 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_perf": "false", 2024-03-20T08:57:57.025 INFO:teuthology.orchestra.run.smithi026.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-03-20T08:57:57.025 INFO:teuthology.orchestra.run.smithi026.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-03-20T08:57:57.025 INFO:teuthology.orchestra.run.smithi026.stdout: "rotating_keys_renewal_timeout": "10", 2024-03-20T08:57:57.026 INFO:teuthology.orchestra.run.smithi026.stdout: "run_dir": "/var/run/ceph", 2024-03-20T08:57:57.026 INFO:teuthology.orchestra.run.smithi026.stdout: "seastore_block_create": "true", 2024-03-20T08:57:57.026 INFO:teuthology.orchestra.run.smithi026.stdout: "seastore_cache_lru_size": "67108864", 2024-03-20T08:57:57.026 INFO:teuthology.orchestra.run.smithi026.stdout: "seastore_cbjournal_size": "5368709120", 2024-03-20T08:57:57.026 INFO:teuthology.orchestra.run.smithi026.stdout: "seastore_data_delta_based_overwrite": "0", 2024-03-20T08:57:57.026 INFO:teuthology.orchestra.run.smithi026.stdout: "seastore_default_max_object_size": "16777216", 2024-03-20T08:57:57.026 INFO:teuthology.orchestra.run.smithi026.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-03-20T08:57:57.026 INFO:teuthology.orchestra.run.smithi026.stdout: "seastore_device_size": "53687091200", 2024-03-20T08:57:57.026 INFO:teuthology.orchestra.run.smithi026.stdout: "seastore_journal_batch_capacity": "16", 2024-03-20T08:57:57.026 INFO:teuthology.orchestra.run.smithi026.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-03-20T08:57:57.026 INFO:teuthology.orchestra.run.smithi026.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-03-20T08:57:57.026 INFO:teuthology.orchestra.run.smithi026.stdout: "seastore_journal_iodepth_limit": "5", 2024-03-20T08:57:57.026 INFO:teuthology.orchestra.run.smithi026.stdout: "seastore_main_device_type": "SSD", 2024-03-20T08:57:57.026 INFO:teuthology.orchestra.run.smithi026.stdout: "seastore_max_concurrent_transactions": "8", 2024-03-20T08:57:57.027 INFO:teuthology.orchestra.run.smithi026.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-03-20T08:57:57.027 INFO:teuthology.orchestra.run.smithi026.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-03-20T08:57:57.027 INFO:teuthology.orchestra.run.smithi026.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-03-20T08:57:57.027 INFO:teuthology.orchestra.run.smithi026.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-03-20T08:57:57.027 INFO:teuthology.orchestra.run.smithi026.stdout: "seastore_segment_size": "67108864", 2024-03-20T08:57:57.027 INFO:teuthology.orchestra.run.smithi026.stdout: "set_keepcaps": "false", 2024-03-20T08:57:57.027 INFO:teuthology.orchestra.run.smithi026.stdout: "setgroup": "ceph", 2024-03-20T08:57:57.027 INFO:teuthology.orchestra.run.smithi026.stdout: "setuser": "ceph", 2024-03-20T08:57:57.027 INFO:teuthology.orchestra.run.smithi026.stdout: "setuser_match_path": "", 2024-03-20T08:57:57.027 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_misplaced_ratio": "0.050000", 2024-03-20T08:57:57.027 INFO:teuthology.orchestra.run.smithi026.stdout: "thp": "false", 2024-03-20T08:57:57.027 INFO:teuthology.orchestra.run.smithi026.stdout: "threadpool_default_timeout": "60", 2024-03-20T08:57:57.027 INFO:teuthology.orchestra.run.smithi026.stdout: "threadpool_empty_queue_max_wait": "2", 2024-03-20T08:57:57.027 INFO:teuthology.orchestra.run.smithi026.stdout: "throttler_perf_counter": "true" 2024-03-20T08:57:57.028 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T08:57:57.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:56 smithi115 ceph-mon[72476]: pgmap v357: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:57.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:56 smithi122 ceph-mon[73495]: pgmap v357: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:57.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:56 smithi026 ceph-mon[68309]: pgmap v357: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:57.546 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-03-20T08:57:57.823 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:57:59.012 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:58 smithi026 ceph-mon[68309]: pgmap v358: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:59.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:58 smithi115 ceph-mon[72476]: pgmap v358: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:59.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:58 smithi122 ceph-mon[73495]: pgmap v358: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:57:59.594 INFO:teuthology.orchestra.run.smithi026.stdout:mon.a: {} 2024-03-20T08:57:59.605 INFO:teuthology.orchestra.run.smithi026.stdout:mon.c: {} 2024-03-20T08:57:59.616 INFO:teuthology.orchestra.run.smithi026.stdout:mon.b: {} 2024-03-20T08:57:59.617 INFO:teuthology.orchestra.run.smithi026.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-20T08:57:59.617 INFO:teuthology.orchestra.run.smithi026.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-20T08:57:59.617 INFO:teuthology.orchestra.run.smithi026.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-20T08:57:59.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:59 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:57:59.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:59 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:57:59.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:57:59 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/998364007' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-20T08:58:00.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:59 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:58:00.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:59 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:58:00.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:57:59 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/998364007' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-20T08:58:00.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:59 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:58:00.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:59 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:58:00.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:57:59 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/998364007' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-20T08:58:00.301 INFO:teuthology.run_tasks:Running task full_sequential... 2024-03-20T08:58:00.325 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-20T08:58:00.328 INFO:tasks.radosbench:Beginning radosbench... 2024-03-20T08:58:00.328 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-03-20T08:58:00.328 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pool create unique_pool_0 16 2024-03-20T08:58:00.329 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-03-20T08:58:00.329 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-03-20T08:58:00.329 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T08:58:00.329 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-03-20T08:58:00.330 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T08:58:00.330 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-03-20T08:58:00.330 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T08:58:00.331 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-03-20T08:58:00.331 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-20T08:58:00.332 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-03-20T08:58:00.332 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T08:58:00.368 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:00.389 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T08:58:00.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:00.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T08:58:00.493 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T08:58:00.524 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:00.544 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T08:58:00.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:00.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T08:58:00.648 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T08:58:00.684 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:00.704 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T08:58:00.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:00.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T08:58:00.714 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:58:00.714 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:58:00.714 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:58:00.718 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:58:00.809 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T08:58:00.839 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:00.859 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T08:58:00.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:00.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T08:58:00.963 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T08:58:00.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:00 smithi026 ceph-mon[68309]: pgmap v359: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:00.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:00 smithi026 ceph-mon[68309]: osdmap e118: 12 total, 12 up, 12 in 2024-03-20T08:58:00.991 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:01.011 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T08:58:01.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:01.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T08:58:01.115 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T08:58:01.140 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:00 smithi115 ceph-mon[72476]: pgmap v359: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:01.140 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:00 smithi115 ceph-mon[72476]: osdmap e118: 12 total, 12 up, 12 in 2024-03-20T08:58:01.142 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:01.160 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T08:58:01.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:01.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T08:58:01.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:00 smithi122 ceph-mon[73495]: pgmap v359: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:01.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:00 smithi122 ceph-mon[73495]: osdmap e118: 12 total, 12 up, 12 in 2024-03-20T08:58:01.264 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T08:58:01.290 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:01.309 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T08:58:01.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:01.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T08:58:01.413 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T08:58:01.445 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:01.694 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T08:58:01.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:01.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T08:58:01.798 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T08:58:01.834 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:01.856 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T08:58:01.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:01.859 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T08:58:01.960 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T08:58:01.988 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:02.007 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T08:58:02.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:02.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T08:58:02.111 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T08:58:02.140 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:02.160 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T08:58:02.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:02.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T08:58:02.263 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T08:58:02.290 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:02.309 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T08:58:02.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:02.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T08:58:02.412 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T08:58:02.439 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:02.458 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T08:58:02.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:02.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T08:58:02.561 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T08:58:02.595 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:02.617 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T08:58:02.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:02.620 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T08:58:02.722 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T08:58:02.748 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:02.767 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T08:58:02.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:02.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T08:58:02.870 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T08:58:02.896 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:02.915 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T08:58:02.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:02.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T08:58:03.018 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T08:58:03.049 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:03.087 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T08:58:03.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:03.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T08:58:03.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:02 smithi115 ceph-mon[72476]: pgmap v361: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:03.191 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T08:58:03.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:02 smithi122 ceph-mon[73495]: pgmap v361: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:03.218 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:03.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:02 smithi026 ceph-mon[68309]: pgmap v361: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:03.236 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T08:58:03.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:03.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T08:58:03.340 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T08:58:03.366 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:03.384 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T08:58:03.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:03.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T08:58:03.488 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T08:58:03.521 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:03.542 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T08:58:03.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:03.545 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T08:58:03.646 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T08:58:03.673 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:03.691 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T08:58:03.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:03.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T08:58:03.794 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T08:58:03.825 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:03.846 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T08:58:03.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:03.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T08:58:03.903 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T08:58:03.950 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T08:58:03.983 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:04.005 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T08:58:04.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:04.008 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T08:58:04.109 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T08:58:04.134 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:03 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3950593703' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T08:58:04.136 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:04.154 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T08:58:04.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:04.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T08:58:04.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:03 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3950593703' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T08:58:04.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:03 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3950593703' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T08:58:04.258 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T08:58:04.287 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:04.413 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T08:58:04.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:04.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T08:58:04.517 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T08:58:04.546 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:05.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:04 smithi115 ceph-mon[72476]: pgmap v362: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:05.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:04 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T08:58:05.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:04 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3950593703' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T08:58:05.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:04 smithi115 ceph-mon[72476]: osdmap e119: 12 total, 12 up, 12 in 2024-03-20T08:58:05.195 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T08:58:05.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:04 smithi122 ceph-mon[73495]: pgmap v362: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:05.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:04 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T08:58:05.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:04 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3950593703' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T08:58:05.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:04 smithi122 ceph-mon[73495]: osdmap e119: 12 total, 12 up, 12 in 2024-03-20T08:58:05.197 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:04 smithi026 ceph-mon[68309]: pgmap v362: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:05.198 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:04 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T08:58:05.198 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:04 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3950593703' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T08:58:05.198 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:04 smithi026 ceph-mon[68309]: osdmap e119: 12 total, 12 up, 12 in 2024-03-20T08:58:05.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:05.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T08:58:05.301 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T08:58:05.334 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:05.355 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T08:58:05.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:05.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T08:58:05.460 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T08:58:05.492 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:05.568 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T08:58:05.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:05.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T08:58:05.672 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T08:58:05.706 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:05.805 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T08:58:05.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:05.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T08:58:05.908 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T08:58:05.940 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:05.960 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T08:58:05.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:05.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T08:58:06.064 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T08:58:06.097 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:06.118 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T08:58:06.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:06.121 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T08:58:06.222 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T08:58:06.254 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:06.274 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T08:58:06.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:06.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T08:58:06.307 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T08:58:06.307 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T08:58:06.307 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-03-20T08:58:06.307 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T08:58:06.378 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T08:58:06.411 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:06.425 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T08:58:06.425 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T08:58:06.425 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T08:58:06.425 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T08:58:06.432 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T08:58:06.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:06.436 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T08:58:06.445 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T08:58:06.465 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T08:58:06.486 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T08:58:06.506 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T08:58:06.527 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T08:58:06.536 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T08:58:06.547 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T08:58:06.567 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T08:58:06.568 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:06.587 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T08:58:06.588 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T08:58:06.588 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T08:58:06.588 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T08:58:06.588 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T08:58:06.588 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T08:58:06.588 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T08:58:06.589 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T08:58:06.589 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T08:58:06.589 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T08:58:06.589 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T08:58:06.589 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T08:58:06.589 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T08:58:06.589 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T08:58:06.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:06.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T08:58:06.694 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T08:58:06.724 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:06.749 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T08:58:06.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:06.752 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T08:58:06.844 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T08:58:06.853 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T08:58:06.918 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:06.956 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T08:58:06.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:06.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T08:58:07.069 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T08:58:07.097 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:07.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:06 smithi122 ceph-mon[73495]: pgmap v364: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:07.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:06 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1124055172' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T08:58:07.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:06 smithi026 ceph-mon[68309]: pgmap v364: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:07.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:06 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1124055172' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T08:58:07.246 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T08:58:07.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:07.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T08:58:07.350 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T08:58:07.383 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:07.404 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T08:58:07.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:06 smithi115 ceph-mon[72476]: pgmap v364: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:07.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:06 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1124055172' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T08:58:07.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:07.407 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T08:58:07.508 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T08:58:07.540 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:07.561 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T08:58:07.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:07.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T08:58:07.666 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T08:58:07.694 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:08.011 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-20T08:58:08.135 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:58:08.143 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T08:58:08.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:08.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T08:58:08.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:07 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:58:08.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:07 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:58:08.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:07 smithi122 ceph-mon[73495]: osdmap e120: 12 total, 12 up, 12 in 2024-03-20T08:58:08.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:07 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:58:08.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:07 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:58:08.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:07 smithi026 ceph-mon[68309]: osdmap e120: 12 total, 12 up, 12 in 2024-03-20T08:58:08.247 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T08:58:08.280 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:08.301 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T08:58:08.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:08.304 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T08:58:08.405 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T08:58:08.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:07 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:58:08.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:07 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:58:08.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:07 smithi115 ceph-mon[72476]: osdmap e120: 12 total, 12 up, 12 in 2024-03-20T08:58:08.437 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:08.459 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T08:58:08.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:08.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T08:58:08.562 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T08:58:08.594 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:08.615 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T08:58:08.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:08.619 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T08:58:08.720 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T08:58:08.749 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:08.768 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T08:58:08.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:08.771 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T08:58:08.861 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:58:08.872 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T08:58:08.905 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:08.922 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'unique_pool_0' created 2024-03-20T08:58:08.928 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T08:58:08.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:08.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T08:58:09.032 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T08:58:09.061 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:09.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:08 smithi115 ceph-mon[72476]: pgmap v366: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:09.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:08 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1066796022' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-20T08:58:09.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:08 smithi122 ceph-mon[73495]: pgmap v366: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:09.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:08 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1066796022' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-20T08:58:09.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:08 smithi026 ceph-mon[68309]: pgmap v366: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:09.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:08 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1066796022' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-20T08:58:09.279 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T08:58:09.410 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T08:58:09.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:09.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T08:58:09.515 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T08:58:09.544 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:10.194 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T08:58:10.195 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:09 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1066796022' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-20T08:58:10.195 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:09 smithi026 ceph-mon[68309]: osdmap e121: 12 total, 12 up, 12 in 2024-03-20T08:58:10.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:10.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T08:58:10.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:09 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1066796022' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-20T08:58:10.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:09 smithi122 ceph-mon[73495]: osdmap e121: 12 total, 12 up, 12 in 2024-03-20T08:58:10.297 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T08:58:10.327 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:10.401 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T08:58:10.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:10.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T08:58:10.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:09 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1066796022' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-20T08:58:10.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:09 smithi115 ceph-mon[72476]: osdmap e121: 12 total, 12 up, 12 in 2024-03-20T08:58:10.504 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T08:58:10.535 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:10.734 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T08:58:10.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:10.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T08:58:10.839 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T08:58:10.868 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:10.928 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T08:58:11.070 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T08:58:11.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:11.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T08:58:11.168 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:10 smithi026 ceph-mon[68309]: pgmap v368: 17 pgs: 2 creating+peering, 1 active+clean, 14 unknown; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:11.169 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:10 smithi026 ceph-mon[68309]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-20T08:58:11.169 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:10 smithi026 ceph-mon[68309]: osdmap e122: 12 total, 12 up, 12 in 2024-03-20T08:58:11.169 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:10 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1591555989' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T08:58:11.174 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T08:58:11.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:10 smithi122 ceph-mon[73495]: pgmap v368: 17 pgs: 2 creating+peering, 1 active+clean, 14 unknown; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:11.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:10 smithi122 ceph-mon[73495]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-20T08:58:11.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:10 smithi122 ceph-mon[73495]: osdmap e122: 12 total, 12 up, 12 in 2024-03-20T08:58:11.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:10 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1591555989' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T08:58:11.203 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:11.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:10 smithi115 ceph-mon[72476]: pgmap v368: 17 pgs: 2 creating+peering, 1 active+clean, 14 unknown; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:11.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:10 smithi115 ceph-mon[72476]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-20T08:58:11.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:10 smithi115 ceph-mon[72476]: osdmap e122: 12 total, 12 up, 12 in 2024-03-20T08:58:11.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:10 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1591555989' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T08:58:11.506 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-03-20T08:58:12.152 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T08:58:12.152 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T08:58:12.167 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:11 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T08:58:12.167 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:11 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1591555989' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T08:58:12.167 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:11 smithi026 ceph-mon[68309]: osdmap e123: 12 total, 12 up, 12 in 2024-03-20T08:58:12.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:11 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T08:58:12.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:11 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1591555989' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T08:58:12.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:11 smithi122 ceph-mon[73495]: osdmap e123: 12 total, 12 up, 12 in 2024-03-20T08:58:12.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:11 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T08:58:12.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:11 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1591555989' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T08:58:12.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:11 smithi115 ceph-mon[72476]: osdmap e123: 12 total, 12 up, 12 in 2024-03-20T08:58:12.652 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T08:58:12.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:12.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T08:58:12.756 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T08:58:12.789 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:13.061 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:12 smithi026 ceph-mon[68309]: pgmap v371: 17 pgs: 3 active+clean, 6 creating+peering, 8 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:13.061 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:12 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:58:13.061 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:12 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:58:13.062 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:58:13.068 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T08:58:13.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:13.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T08:58:13.098 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-20T08:58:13.174 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T08:58:13.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:12 smithi122 ceph-mon[73495]: pgmap v371: 17 pgs: 3 active+clean, 6 creating+peering, 8 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:13.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:12 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:58:13.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:12 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:58:13.206 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:13.227 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T08:58:13.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:13.231 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T08:58:13.332 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T08:58:13.362 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:13.382 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T08:58:13.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:13.385 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T08:58:13.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:12 smithi115 ceph-mon[72476]: pgmap v371: 17 pgs: 3 active+clean, 6 creating+peering, 8 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:13.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:12 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:58:13.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:12 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:58:13.486 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T08:58:13.518 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:13.539 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T08:58:13.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:13.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T08:58:13.643 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T08:58:13.671 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:13.691 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T08:58:13.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:13.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T08:58:13.795 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T08:58:13.824 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:13.843 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T08:58:13.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:13.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T08:58:13.952 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T08:58:13.985 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:14.006 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T08:58:14.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:14.009 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T08:58:14.110 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T08:58:14.136 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:13 smithi026 ceph-mon[68309]: osdmap e124: 12 total, 12 up, 12 in 2024-03-20T08:58:14.137 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:14.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:13 smithi122 ceph-mon[73495]: osdmap e124: 12 total, 12 up, 12 in 2024-03-20T08:58:14.226 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:58:14.226 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:58:14.236 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T08:58:14.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:14.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T08:58:14.347 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T08:58:14.377 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:13 smithi115 ceph-mon[72476]: osdmap e124: 12 total, 12 up, 12 in 2024-03-20T08:58:14.380 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:14.401 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T08:58:14.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:14.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T08:58:14.473 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T08:58:14.505 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T08:58:14.535 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:14.556 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T08:58:14.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:14.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T08:58:14.660 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T08:58:14.688 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:14.708 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T08:58:14.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:14.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T08:58:14.812 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T08:58:14.840 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:15.038 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T08:58:15.040 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:14 smithi026 ceph-mon[68309]: pgmap v373: 17 pgs: 11 active+clean, 6 creating+peering; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:15.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:15.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T08:58:15.142 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T08:58:15.173 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:15.194 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T08:58:15.197 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:14 smithi115 ceph-mon[72476]: pgmap v373: 17 pgs: 11 active+clean, 6 creating+peering; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:15.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:15.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T08:58:15.199 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:14 smithi122 ceph-mon[73495]: pgmap v373: 17 pgs: 11 active+clean, 6 creating+peering; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:15.299 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T08:58:15.329 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:15.348 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T08:58:15.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:15.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T08:58:15.452 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T08:58:15.486 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:15.603 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T08:58:15.622 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T08:58:15.642 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T08:58:15.661 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T08:58:15.682 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T08:58:15.702 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T08:58:15.722 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T08:58:15.742 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T08:58:15.764 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T08:58:15.784 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T08:58:15.805 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T08:58:15.825 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T08:58:15.826 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T08:58:15.826 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T08:58:15.826 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T08:58:15.827 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T08:58:15.827 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T08:58:15.827 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T08:58:15.827 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T08:58:15.827 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T08:58:15.827 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T08:58:15.827 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T08:58:15.827 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T08:58:16.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:15 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1677235936' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T08:58:16.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:15 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1677235936' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T08:58:16.242 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T08:58:16.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:16.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T08:58:16.347 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T08:58:16.377 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:16.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:15 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1677235936' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T08:58:16.849 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T08:58:16.849 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-03-20T08:58:16.849 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-03-20T08:58:16.849 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T08:58:16.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.849 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.14154.0:20 1.0 e20/19 1:30306672:devicehealth::main.db.0000000000000000:head v 20'10, mlcod=20'8)", 2024-03-20T08:58:16.849 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:52:18.984460+0000", 2024-03-20T08:58:16.849 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 357.58321572400001, 2024-03-20T08:58:16.849 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082967100000000003, 2024-03-20T08:58:16.850 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:16.850 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:58:16.850 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:16.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:16.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:18.984460+0000", 2024-03-20T08:58:16.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:16.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:18.984460+0000", 2024-03-20T08:58:16.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:16.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:18.984461+0000", 2024-03-20T08:58:16.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.23e-07 2024-03-20T08:58:16.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:16.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:18.984501+0000", 2024-03-20T08:58:16.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0438999999999997e-05 2024-03-20T08:58:16.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:16.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:18.984512+0000", 2024-03-20T08:58:16.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0667e-05 2024-03-20T08:58:16.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:16.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:18.984524+0000", 2024-03-20T08:58:16.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2251e-05 2024-03-20T08:58:16.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:16.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:18.984578+0000", 2024-03-20T08:58:16.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3599e-05 2024-03-20T08:58:16.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:16.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:18.984600+0000", 2024-03-20T08:58:16.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2070999999999999e-05 2024-03-20T08:58:16.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.853 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:58:16.853 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:18.985247+0000", 2024-03-20T08:58:16.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00064681299999999999 2024-03-20T08:58:16.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.853 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:16.853 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:18.985290+0000", 2024-03-20T08:58:16.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3108000000000001e-05 2024-03-20T08:58:16.853 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.853 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:16.853 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.854 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.14154.0:21 1.0 e20/19 1:30306672:devicehealth::main.db.0000000000000000:head v 20'11, mlcod=20'8)", 2024-03-20T08:58:16.854 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:52:18.984895+0000", 2024-03-20T08:58:16.854 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 357.58278121000001, 2024-03-20T08:58:16.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085451099999999996, 2024-03-20T08:58:16.854 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:16.854 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:58:16.854 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:16.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.854 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:16.854 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:18.984895+0000", 2024-03-20T08:58:16.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.854 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:16.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:18.984895+0000", 2024-03-20T08:58:16.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:16.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:18.984895+0000", 2024-03-20T08:58:16.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3499999999999996e-07 2024-03-20T08:58:16.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:16.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:18.984942+0000", 2024-03-20T08:58:16.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6587e-05 2024-03-20T08:58:16.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:16.856 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:18.984952+0000", 2024-03-20T08:58:16.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0121e-05 2024-03-20T08:58:16.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:16.856 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:18.984959+0000", 2024-03-20T08:58:16.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6660000000000002e-06 2024-03-20T08:58:16.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:16.856 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:18.985018+0000", 2024-03-20T08:58:16.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8850000000000001e-05 2024-03-20T08:58:16.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.857 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:16.857 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:18.985044+0000", 2024-03-20T08:58:16.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6222999999999999e-05 2024-03-20T08:58:16.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.857 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:58:16.857 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:18.985708+0000", 2024-03-20T08:58:16.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00066421799999999995 2024-03-20T08:58:16.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.857 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:16.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:18.985749+0000", 2024-03-20T08:58:16.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1211000000000002e-05 2024-03-20T08:58:16.858 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.858 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:16.858 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.858 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.14154.0:38 1.0 e20/19 1:30306672:devicehealth::main.db.0000000000000000:head v 20'24, mlcod=20'22)", 2024-03-20T08:58:16.858 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:52:18.996496+0000", 2024-03-20T08:58:16.858 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 357.57118012299998, 2024-03-20T08:58:16.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00082799299999999996, 2024-03-20T08:58:16.858 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:16.858 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:58:16.858 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:16.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:16.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:18.996496+0000", 2024-03-20T08:58:16.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:16.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:18.996496+0000", 2024-03-20T08:58:16.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:16.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:18.996496+0000", 2024-03-20T08:58:16.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2900000000000004e-07 2024-03-20T08:58:16.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.860 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:16.860 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:18.996541+0000", 2024-03-20T08:58:16.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4326000000000002e-05 2024-03-20T08:58:16.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.860 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:16.860 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:18.996556+0000", 2024-03-20T08:58:16.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5639999999999999e-05 2024-03-20T08:58:16.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.860 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:16.860 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:18.996569+0000", 2024-03-20T08:58:16.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2860999999999999e-05 2024-03-20T08:58:16.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.861 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:16.861 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:18.996615+0000", 2024-03-20T08:58:16.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5816999999999999e-05 2024-03-20T08:58:16.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.861 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:16.861 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:18.996635+0000", 2024-03-20T08:58:16.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9842000000000001e-05 2024-03-20T08:58:16.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.861 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:58:16.861 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:18.997294+0000", 2024-03-20T08:58:16.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00065896800000000001 2024-03-20T08:58:16.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.862 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:16.862 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:18.997324+0000", 2024-03-20T08:58:16.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0009999999999999e-05 2024-03-20T08:58:16.862 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.862 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:16.862 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.862 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.14154.0:107 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'69, mlcod=20'65)", 2024-03-20T08:58:16.862 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:52:19.102732+0000", 2024-03-20T08:58:16.862 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 357.46494357799997, 2024-03-20T08:58:16.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085708300000000002, 2024-03-20T08:58:16.862 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:16.863 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:58:16.863 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:16.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:16.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.102732+0000", 2024-03-20T08:58:16.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:16.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.102732+0000", 2024-03-20T08:58:16.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:16.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.102733+0000", 2024-03-20T08:58:16.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5899999999999995e-07 2024-03-20T08:58:16.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:16.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.102742+0000", 2024-03-20T08:58:16.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8480000000000007e-06 2024-03-20T08:58:16.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:16.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.102755+0000", 2024-03-20T08:58:16.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2670999999999999e-05 2024-03-20T08:58:16.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:16.865 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.102760+0000", 2024-03-20T08:58:16.865 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7230000000000001e-06 2024-03-20T08:58:16.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.865 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:16.865 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.102817+0000", 2024-03-20T08:58:16.865 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6623000000000003e-05 2024-03-20T08:58:16.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.865 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:16.865 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.102830+0000", 2024-03-20T08:58:16.865 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2796000000000001e-05 2024-03-20T08:58:16.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.865 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:58:16.866 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.103549+0000", 2024-03-20T08:58:16.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071870799999999998 2024-03-20T08:58:16.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.866 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:16.866 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.103590+0000", 2024-03-20T08:58:16.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0954999999999997e-05 2024-03-20T08:58:16.866 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.866 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:16.866 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.866 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.14154.0:109 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'71, mlcod=20'67)", 2024-03-20T08:58:16.866 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:52:19.104018+0000", 2024-03-20T08:58:16.867 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 357.463657829, 2024-03-20T08:58:16.867 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00181131, 2024-03-20T08:58:16.867 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:16.867 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:58:16.867 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:16.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.867 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:16.867 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.104018+0000", 2024-03-20T08:58:16.867 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.867 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:16.867 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.104018+0000", 2024-03-20T08:58:16.867 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:16.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.104019+0000", 2024-03-20T08:58:16.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7899999999999998e-07 2024-03-20T08:58:16.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:16.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.104057+0000", 2024-03-20T08:58:16.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8111999999999997e-05 2024-03-20T08:58:16.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:16.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.104070+0000", 2024-03-20T08:58:16.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2707999999999999e-05 2024-03-20T08:58:16.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:16.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.104077+0000", 2024-03-20T08:58:16.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4749999999999996e-06 2024-03-20T08:58:16.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:16.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.104192+0000", 2024-03-20T08:58:16.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000114962 2024-03-20T08:58:16.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:16.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.104213+0000", 2024-03-20T08:58:16.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1027e-05 2024-03-20T08:58:16.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.870 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:58:16.870 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.105755+0000", 2024-03-20T08:58:16.870 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001542082 2024-03-20T08:58:16.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.870 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:16.870 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.105829+0000", 2024-03-20T08:58:16.870 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4264999999999995e-05 2024-03-20T08:58:16.870 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.870 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:16.870 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.871 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.14154.0:110 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'72, mlcod=20'67)", 2024-03-20T08:58:16.871 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:52:19.104495+0000", 2024-03-20T08:58:16.871 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 357.463180846, 2024-03-20T08:58:16.871 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015354990000000001, 2024-03-20T08:58:16.871 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:16.871 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:58:16.871 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:16.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.871 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:16.871 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.104495+0000", 2024-03-20T08:58:16.871 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.871 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:16.872 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.104495+0000", 2024-03-20T08:58:16.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.872 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:16.872 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.104496+0000", 2024-03-20T08:58:16.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4600000000000004e-07 2024-03-20T08:58:16.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.872 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:16.872 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.104511+0000", 2024-03-20T08:58:16.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5605000000000001e-05 2024-03-20T08:58:16.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:16.873 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.104523+0000", 2024-03-20T08:58:16.873 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1209999999999999e-05 2024-03-20T08:58:16.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:16.873 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.104530+0000", 2024-03-20T08:58:16.873 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2219999999999996e-06 2024-03-20T08:58:16.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:16.873 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.105562+0000", 2024-03-20T08:58:16.873 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010323019999999999 2024-03-20T08:58:16.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.874 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:16.874 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.105581+0000", 2024-03-20T08:58:16.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9160999999999998e-05 2024-03-20T08:58:16.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.874 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:58:16.874 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.105992+0000", 2024-03-20T08:58:16.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041099700000000002 2024-03-20T08:58:16.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.874 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:16.874 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.106031+0000", 2024-03-20T08:58:16.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8355999999999997e-05 2024-03-20T08:58:16.874 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.874 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:16.875 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.875 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.14154.0:111 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'73, mlcod=20'67)", 2024-03-20T08:58:16.875 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:52:19.105532+0000", 2024-03-20T08:58:16.875 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 357.46214413500002, 2024-03-20T08:58:16.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00074798599999999996, 2024-03-20T08:58:16.875 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:16.875 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:58:16.875 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:16.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:16.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.105532+0000", 2024-03-20T08:58:16.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:16.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.105532+0000", 2024-03-20T08:58:16.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:16.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.105532+0000", 2024-03-20T08:58:16.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7400000000000003e-07 2024-03-20T08:58:16.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:16.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.105541+0000", 2024-03-20T08:58:16.877 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4940000000000003e-06 2024-03-20T08:58:16.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.877 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:16.877 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.105552+0000", 2024-03-20T08:58:16.877 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0975000000000001e-05 2024-03-20T08:58:16.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.877 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:16.877 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.105559+0000", 2024-03-20T08:58:16.877 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0269999999999998e-06 2024-03-20T08:58:16.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.877 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:16.878 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.105861+0000", 2024-03-20T08:58:16.878 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00030166200000000001 2024-03-20T08:58:16.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.878 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:16.878 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.105874+0000", 2024-03-20T08:58:16.878 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3346999999999999e-05 2024-03-20T08:58:16.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.878 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:58:16.878 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.106238+0000", 2024-03-20T08:58:16.878 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00036366799999999999 2024-03-20T08:58:16.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.879 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:16.879 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.106280+0000", 2024-03-20T08:58:16.879 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2239e-05 2024-03-20T08:58:16.879 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.879 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:16.879 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.879 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.14154.0:112 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'74, mlcod=20'67)", 2024-03-20T08:58:16.879 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:52:19.105571+0000", 2024-03-20T08:58:16.879 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 357.46210455800002, 2024-03-20T08:58:16.879 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000959124, 2024-03-20T08:58:16.879 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:16.879 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:58:16.879 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:16.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:16.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.105571+0000", 2024-03-20T08:58:16.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:16.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.105571+0000", 2024-03-20T08:58:16.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:16.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.105572+0000", 2024-03-20T08:58:16.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4299999999999999e-07 2024-03-20T08:58:16.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:16.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.105622+0000", 2024-03-20T08:58:16.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0445999999999998e-05 2024-03-20T08:58:16.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:16.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.105631+0000", 2024-03-20T08:58:16.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2369999999999998e-06 2024-03-20T08:58:16.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:16.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.105637+0000", 2024-03-20T08:58:16.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.896e-06 2024-03-20T08:58:16.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:16.882 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.106090+0000", 2024-03-20T08:58:16.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00045275900000000001 2024-03-20T08:58:16.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.882 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:16.882 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.106108+0000", 2024-03-20T08:58:16.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7629999999999999e-05 2024-03-20T08:58:16.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.882 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:58:16.882 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.106495+0000", 2024-03-20T08:58:16.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00038736199999999998 2024-03-20T08:58:16.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.883 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:16.883 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.106531+0000", 2024-03-20T08:58:16.883 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5451000000000003e-05 2024-03-20T08:58:16.883 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.883 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:16.883 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.883 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.14154.0:118 1.0 e20/19 1:30306672:devicehealth::main.db.0000000000000000:head v 20'80, mlcod=20'78)", 2024-03-20T08:58:16.883 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:52:19.110382+0000", 2024-03-20T08:58:16.883 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 357.45729414300001, 2024-03-20T08:58:16.883 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015262369999999999, 2024-03-20T08:58:16.883 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:16.883 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:58:16.883 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:16.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.884 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:16.884 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.110382+0000", 2024-03-20T08:58:16.884 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.884 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:16.884 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.110382+0000", 2024-03-20T08:58:16.884 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.884 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:16.884 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.110383+0000", 2024-03-20T08:58:16.884 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7700000000000004e-07 2024-03-20T08:58:16.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.885 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:16.885 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.110422+0000", 2024-03-20T08:58:16.885 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9687999999999997e-05 2024-03-20T08:58:16.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.885 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:16.885 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.110436+0000", 2024-03-20T08:58:16.885 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3662999999999999e-05 2024-03-20T08:58:16.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.885 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:16.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.110444+0000", 2024-03-20T08:58:16.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7870000000000002e-06 2024-03-20T08:58:16.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:16.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.110496+0000", 2024-03-20T08:58:16.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1876000000000003e-05 2024-03-20T08:58:16.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:16.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.110512+0000", 2024-03-20T08:58:16.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6719e-05 2024-03-20T08:58:16.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.887 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:58:16.887 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.111866+0000", 2024-03-20T08:58:16.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013535699999999999 2024-03-20T08:58:16.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.887 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:16.887 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.111908+0000", 2024-03-20T08:58:16.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2256999999999999e-05 2024-03-20T08:58:16.887 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.887 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:16.887 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.887 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.14154.0:119 1.0 e20/19 1:30306672:devicehealth::main.db.0000000000000000:head v 20'81, mlcod=20'78)", 2024-03-20T08:58:16.888 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:52:19.110903+0000", 2024-03-20T08:58:16.888 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 357.456772935, 2024-03-20T08:58:16.888 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010327579999999999, 2024-03-20T08:58:16.888 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:16.888 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:58:16.888 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:16.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.888 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:16.888 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.110903+0000", 2024-03-20T08:58:16.888 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.888 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:16.888 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.110903+0000", 2024-03-20T08:58:16.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.889 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:16.889 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.110904+0000", 2024-03-20T08:58:16.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0200000000000002e-07 2024-03-20T08:58:16.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.889 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:16.889 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.110933+0000", 2024-03-20T08:58:16.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9564999999999999e-05 2024-03-20T08:58:16.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.889 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:16.890 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.110946+0000", 2024-03-20T08:58:16.890 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2984e-05 2024-03-20T08:58:16.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.890 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:16.890 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.110953+0000", 2024-03-20T08:58:16.890 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9999999999999999e-06 2024-03-20T08:58:16.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.890 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:16.890 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.111003+0000", 2024-03-20T08:58:16.890 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0229000000000003e-05 2024-03-20T08:58:16.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.891 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:16.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.111022+0000", 2024-03-20T08:58:16.891 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8360999999999999e-05 2024-03-20T08:58:16.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.891 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:58:16.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.111910+0000", 2024-03-20T08:58:16.891 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088820100000000003 2024-03-20T08:58:16.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.891 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:16.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.111936+0000", 2024-03-20T08:58:16.892 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5816000000000002e-05 2024-03-20T08:58:16.892 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.892 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:16.892 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.892 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.14154.0:120 1.0 e20/19 1:30306672:devicehealth::main.db.0000000000000000:head v 20'82, mlcod=20'78)", 2024-03-20T08:58:16.892 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:52:19.111366+0000", 2024-03-20T08:58:16.892 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 357.45631013899998, 2024-03-20T08:58:16.892 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084710199999999997, 2024-03-20T08:58:16.892 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:16.892 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:58:16.892 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:16.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.893 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:16.893 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.111366+0000", 2024-03-20T08:58:16.893 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.893 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:16.893 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.111366+0000", 2024-03-20T08:58:16.893 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.893 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:16.893 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.111367+0000", 2024-03-20T08:58:16.893 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0200000000000001e-07 2024-03-20T08:58:16.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.894 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:16.894 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.111395+0000", 2024-03-20T08:58:16.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8751999999999998e-05 2024-03-20T08:58:16.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.894 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:16.894 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.111408+0000", 2024-03-20T08:58:16.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2319e-05 2024-03-20T08:58:16.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.894 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:16.894 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.111413+0000", 2024-03-20T08:58:16.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0989999999999998e-06 2024-03-20T08:58:16.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:16.895 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.111486+0000", 2024-03-20T08:58:16.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3079999999999998e-05 2024-03-20T08:58:16.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:16.895 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.111505+0000", 2024-03-20T08:58:16.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9196999999999999e-05 2024-03-20T08:58:16.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:58:16.896 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.112167+0000", 2024-03-20T08:58:16.896 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00066166100000000002 2024-03-20T08:58:16.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.896 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:16.896 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.112213+0000", 2024-03-20T08:58:16.896 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6292000000000003e-05 2024-03-20T08:58:16.896 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.896 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:16.896 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.896 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.14154.0:129 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'87, mlcod=20'85)", 2024-03-20T08:58:16.896 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:52:19.288104+0000", 2024-03-20T08:58:16.896 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 357.27957206600001, 2024-03-20T08:58:16.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00092977900000000004, 2024-03-20T08:58:16.897 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:16.897 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:58:16.897 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:16.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.897 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:16.897 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.288104+0000", 2024-03-20T08:58:16.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.897 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:16.897 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.288104+0000", 2024-03-20T08:58:16.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:16.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.288105+0000", 2024-03-20T08:58:16.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1800000000000005e-07 2024-03-20T08:58:16.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:16.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.288111+0000", 2024-03-20T08:58:16.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.64e-06 2024-03-20T08:58:16.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:16.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.288122+0000", 2024-03-20T08:58:16.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0644e-05 2024-03-20T08:58:16.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:16.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.288132+0000", 2024-03-20T08:58:16.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8120000000000002e-06 2024-03-20T08:58:16.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:16.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.288181+0000", 2024-03-20T08:58:16.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9527999999999997e-05 2024-03-20T08:58:16.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:16.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.288202+0000", 2024-03-20T08:58:16.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0729000000000001e-05 2024-03-20T08:58:16.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:58:16.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.288990+0000", 2024-03-20T08:58:16.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078749600000000005 2024-03-20T08:58:16.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:16.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.289034+0000", 2024-03-20T08:58:16.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4212000000000003e-05 2024-03-20T08:58:16.900 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.900 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:16.900 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.901 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.14154.0:153 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'107, mlcod=20'105)", 2024-03-20T08:58:16.901 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:52:19.353134+0000", 2024-03-20T08:58:16.901 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 357.21454176399999, 2024-03-20T08:58:16.901 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080359699999999997, 2024-03-20T08:58:16.901 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:16.901 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:58:16.901 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:16.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.901 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:16.901 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.353134+0000", 2024-03-20T08:58:16.901 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.902 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:16.902 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.353134+0000", 2024-03-20T08:58:16.902 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.902 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:16.902 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.353135+0000", 2024-03-20T08:58:16.902 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4600000000000004e-07 2024-03-20T08:58:16.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.902 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:16.902 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.353141+0000", 2024-03-20T08:58:16.903 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4949999999999996e-06 2024-03-20T08:58:16.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.903 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:16.903 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.353154+0000", 2024-03-20T08:58:16.903 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2245999999999999e-05 2024-03-20T08:58:16.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.903 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:16.903 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.353160+0000", 2024-03-20T08:58:16.903 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0549999999999996e-06 2024-03-20T08:58:16.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:16.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.353207+0000", 2024-03-20T08:58:16.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7565999999999998e-05 2024-03-20T08:58:16.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:16.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.353232+0000", 2024-03-20T08:58:16.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4773000000000002e-05 2024-03-20T08:58:16.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:58:16.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.353906+0000", 2024-03-20T08:58:16.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00067361500000000002 2024-03-20T08:58:16.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.905 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:16.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.353938+0000", 2024-03-20T08:58:16.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2101000000000003e-05 2024-03-20T08:58:16.905 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.905 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:16.905 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.905 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.14154.0:161 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'115, mlcod=20'113)", 2024-03-20T08:58:16.905 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:52:19.358281+0000", 2024-03-20T08:58:16.905 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 357.20939536399999, 2024-03-20T08:58:16.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000740779, 2024-03-20T08:58:16.905 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:16.906 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:58:16.906 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:16.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:16.906 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.358281+0000", 2024-03-20T08:58:16.906 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:16.906 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.358281+0000", 2024-03-20T08:58:16.906 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:16.907 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.358281+0000", 2024-03-20T08:58:16.907 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4700000000000001e-07 2024-03-20T08:58:16.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.907 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:16.907 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.358305+0000", 2024-03-20T08:58:16.907 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3952999999999999e-05 2024-03-20T08:58:16.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.907 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:16.907 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.358318+0000", 2024-03-20T08:58:16.907 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3227000000000001e-05 2024-03-20T08:58:16.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.907 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:16.908 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.358327+0000", 2024-03-20T08:58:16.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6109999999999994e-06 2024-03-20T08:58:16.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.908 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:16.908 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.358371+0000", 2024-03-20T08:58:16.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4450000000000003e-05 2024-03-20T08:58:16.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.908 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:16.908 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.358393+0000", 2024-03-20T08:58:16.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1648999999999999e-05 2024-03-20T08:58:16.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.909 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:58:16.909 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.358979+0000", 2024-03-20T08:58:16.909 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000585646 2024-03-20T08:58:16.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.909 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:16.909 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.359021+0000", 2024-03-20T08:58:16.909 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2695999999999999e-05 2024-03-20T08:58:16.909 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.909 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:16.909 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.909 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.14154.0:162 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'116, mlcod=20'113)", 2024-03-20T08:58:16.910 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:52:19.358723+0000", 2024-03-20T08:58:16.910 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 357.208952958, 2024-03-20T08:58:16.910 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00068081000000000001, 2024-03-20T08:58:16.910 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:16.910 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:58:16.910 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:16.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:16.910 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.358723+0000", 2024-03-20T08:58:16.910 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:16.910 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.358723+0000", 2024-03-20T08:58:16.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:16.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.358724+0000", 2024-03-20T08:58:16.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7899999999999998e-07 2024-03-20T08:58:16.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:16.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.358729+0000", 2024-03-20T08:58:16.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6319999999999996e-06 2024-03-20T08:58:16.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:16.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.358741+0000", 2024-03-20T08:58:16.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1970000000000001e-05 2024-03-20T08:58:16.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.912 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:16.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.358746+0000", 2024-03-20T08:58:16.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.916e-06 2024-03-20T08:58:16.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.912 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:16.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.358803+0000", 2024-03-20T08:58:16.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6437999999999997e-05 2024-03-20T08:58:16.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.912 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:16.913 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.358823+0000", 2024-03-20T08:58:16.913 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0259999999999999e-05 2024-03-20T08:58:16.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.913 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:58:16.913 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.359371+0000", 2024-03-20T08:58:16.913 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00054804400000000003 2024-03-20T08:58:16.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.913 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:16.913 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.359404+0000", 2024-03-20T08:58:16.913 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2971e-05 2024-03-20T08:58:16.913 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.913 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:16.914 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.914 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.14154.0:166 1.0 e20/19 1:30306672:devicehealth::main.db.0000000000000000:head v 20'120, mlcod=20'118)", 2024-03-20T08:58:16.914 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:52:19.361568+0000", 2024-03-20T08:58:16.914 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 357.20610814100002, 2024-03-20T08:58:16.914 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00074460100000000001, 2024-03-20T08:58:16.914 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:16.914 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:58:16.914 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:16.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.914 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:16.914 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.361568+0000", 2024-03-20T08:58:16.914 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.915 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:16.915 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.361568+0000", 2024-03-20T08:58:16.915 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.915 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:16.915 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.361568+0000", 2024-03-20T08:58:16.915 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3099999999999998e-07 2024-03-20T08:58:16.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.915 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:16.915 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.361586+0000", 2024-03-20T08:58:16.915 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7303e-05 2024-03-20T08:58:16.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.916 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:16.916 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.361595+0000", 2024-03-20T08:58:16.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0170000000000002e-06 2024-03-20T08:58:16.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.916 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:16.916 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.361601+0000", 2024-03-20T08:58:16.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7680000000000003e-06 2024-03-20T08:58:16.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.916 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:16.916 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.361637+0000", 2024-03-20T08:58:16.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5728000000000001e-05 2024-03-20T08:58:16.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:16.917 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.361655+0000", 2024-03-20T08:58:16.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7595e-05 2024-03-20T08:58:16.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:58:16.917 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.362266+0000", 2024-03-20T08:58:16.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00061164699999999999 2024-03-20T08:58:16.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:16.917 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.362312+0000", 2024-03-20T08:58:16.918 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6011999999999999e-05 2024-03-20T08:58:16.918 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.918 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:16.918 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.918 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.14154.0:168 1.0 e20/19 1:30306672:devicehealth::main.db.0000000000000000:head v 20'122, mlcod=20'118)", 2024-03-20T08:58:16.918 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:52:19.362421+0000", 2024-03-20T08:58:16.918 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 357.205254513, 2024-03-20T08:58:16.918 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00070765400000000005, 2024-03-20T08:58:16.918 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:16.918 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:58:16.918 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:16.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.919 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:16.919 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.362421+0000", 2024-03-20T08:58:16.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.919 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:16.919 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.362421+0000", 2024-03-20T08:58:16.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.919 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:16.919 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.362422+0000", 2024-03-20T08:58:16.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2900000000000002e-07 2024-03-20T08:58:16.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:16.920 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.362446+0000", 2024-03-20T08:58:16.920 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4071e-05 2024-03-20T08:58:16.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:16.920 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.362458+0000", 2024-03-20T08:58:16.920 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1151e-05 2024-03-20T08:58:16.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:16.920 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.362465+0000", 2024-03-20T08:58:16.920 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6709999999999999e-06 2024-03-20T08:58:16.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.921 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:16.921 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.362520+0000", 2024-03-20T08:58:16.921 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4542000000000001e-05 2024-03-20T08:58:16.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.921 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:16.921 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.362540+0000", 2024-03-20T08:58:16.921 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0406000000000001e-05 2024-03-20T08:58:16.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.921 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:58:16.921 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.363098+0000", 2024-03-20T08:58:16.921 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000557655 2024-03-20T08:58:16.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.922 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:16.922 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.363129+0000", 2024-03-20T08:58:16.922 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1328999999999998e-05 2024-03-20T08:58:16.922 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.922 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:16.922 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.922 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.14154.0:181 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'131, mlcod=20'128)", 2024-03-20T08:58:16.922 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:52:19.374660+0000", 2024-03-20T08:58:16.922 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 357.19301629500001, 2024-03-20T08:58:16.922 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014282469999999999, 2024-03-20T08:58:16.923 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:16.923 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:58:16.923 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:16.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:16.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.374660+0000", 2024-03-20T08:58:16.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:16.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.374660+0000", 2024-03-20T08:58:16.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.924 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:16.924 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.374660+0000", 2024-03-20T08:58:16.924 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2799999999999995e-07 2024-03-20T08:58:16.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.924 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:16.924 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.374667+0000", 2024-03-20T08:58:16.924 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0979999999999999e-06 2024-03-20T08:58:16.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.924 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:16.924 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.374680+0000", 2024-03-20T08:58:16.924 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3144e-05 2024-03-20T08:58:16.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.925 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:16.925 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.374687+0000", 2024-03-20T08:58:16.925 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6190000000000002e-06 2024-03-20T08:58:16.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.925 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:16.925 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.374742+0000", 2024-03-20T08:58:16.925 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4555000000000002e-05 2024-03-20T08:58:16.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.925 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:16.925 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.374757+0000", 2024-03-20T08:58:16.926 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5444e-05 2024-03-20T08:58:16.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.926 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:58:16.926 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.376034+0000", 2024-03-20T08:58:16.926 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001276419 2024-03-20T08:58:16.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.926 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:16.926 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.376088+0000", 2024-03-20T08:58:16.926 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4240000000000002e-05 2024-03-20T08:58:16.926 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.926 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:16.926 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.927 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.14154.0:182 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'132, mlcod=20'128)", 2024-03-20T08:58:16.927 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:52:19.375163+0000", 2024-03-20T08:58:16.927 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 357.192513394, 2024-03-20T08:58:16.927 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011315850000000001, 2024-03-20T08:58:16.927 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:16.927 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:58:16.927 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:16.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.927 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:16.927 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.375163+0000", 2024-03-20T08:58:16.927 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.928 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:16.928 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.375163+0000", 2024-03-20T08:58:16.928 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.928 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:16.928 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.375163+0000", 2024-03-20T08:58:16.928 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0699999999999996e-07 2024-03-20T08:58:16.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.928 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:16.928 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.375185+0000", 2024-03-20T08:58:16.928 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1509999999999999e-05 2024-03-20T08:58:16.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.929 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:16.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.375194+0000", 2024-03-20T08:58:16.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5189999999999998e-06 2024-03-20T08:58:16.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.929 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:16.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.375199+0000", 2024-03-20T08:58:16.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8300000000000003e-06 2024-03-20T08:58:16.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.929 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:16.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.375791+0000", 2024-03-20T08:58:16.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00059160300000000001 2024-03-20T08:58:16.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:16.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.375815+0000", 2024-03-20T08:58:16.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4060999999999999e-05 2024-03-20T08:58:16.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:58:16.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.376260+0000", 2024-03-20T08:58:16.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00044518200000000002 2024-03-20T08:58:16.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:16.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.376294+0000", 2024-03-20T08:58:16.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4173e-05 2024-03-20T08:58:16.931 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.931 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:16.931 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.931 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.14154.0:183 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'133, mlcod=20'129)", 2024-03-20T08:58:16.931 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:52:19.377076+0000", 2024-03-20T08:58:16.931 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 357.190600211, 2024-03-20T08:58:16.931 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0017473879999999999, 2024-03-20T08:58:16.931 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:16.931 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:58:16.931 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:16.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.377076+0000", 2024-03-20T08:58:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.377076+0000", 2024-03-20T08:58:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.377077+0000", 2024-03-20T08:58:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0669999999999999e-06 2024-03-20T08:58:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.932 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.377085+0000", 2024-03-20T08:58:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0290000000000005e-06 2024-03-20T08:58:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.377099+0000", 2024-03-20T08:58:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4374e-05 2024-03-20T08:58:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.377108+0000", 2024-03-20T08:58:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7399999999999993e-06 2024-03-20T08:58:16.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.377171+0000", 2024-03-20T08:58:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2570000000000006e-05 2024-03-20T08:58:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.377194+0000", 2024-03-20T08:58:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3391e-05 2024-03-20T08:58:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:58:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.378770+0000", 2024-03-20T08:58:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001576397 2024-03-20T08:58:16.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:16.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:16.935 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:16.935 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:52:19.378823+0000", 2024-03-20T08:58:16.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2819999999999999e-05 2024-03-20T08:58:16.935 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.935 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:16.935 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.935 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:16.935 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:16.935 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T08:58:17.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:16 smithi122 ceph-mon[73495]: pgmap v374: 17 pgs: 17 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:17.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:16 smithi026 ceph-mon[68309]: pgmap v374: 17 pgs: 17 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:17.377 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-20T08:58:17.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:16 smithi115 ceph-mon[72476]: pgmap v374: 17 pgs: 17 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:18.129 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T08:58:18.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:18.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T08:58:18.169 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:58:18.173 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:58:18.234 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T08:58:18.267 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:18.288 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T08:58:18.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:18.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T08:58:18.392 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T08:58:18.422 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:18.769 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T08:58:19.063 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:18 smithi026 ceph-mon[68309]: pgmap v375: 17 pgs: 17 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:19.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:18 smithi115 ceph-mon[72476]: pgmap v375: 17 pgs: 17 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:19.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:18 smithi122 ceph-mon[73495]: pgmap v375: 17 pgs: 17 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:20.578 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T08:58:20.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:20.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T08:58:20.682 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T08:58:20.712 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:20.732 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T08:58:20.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:20.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T08:58:20.839 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T08:58:20.868 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:20.942 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T08:58:20.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:20.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T08:58:20.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:20 smithi026 ceph-mon[68309]: pgmap v376: 17 pgs: 17 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:21.046 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T08:58:21.075 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:21.152 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T08:58:21.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:21.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T08:58:21.193 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T08:58:21.193 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T08:58:21.193 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-03-20T08:58:21.193 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T08:58:21.256 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T08:58:21.285 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:20 smithi115 ceph-mon[72476]: pgmap v376: 17 pgs: 17 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:21.288 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:21.309 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T08:58:21.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:21.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T08:58:21.413 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T08:58:21.443 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:21.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:20 smithi122 ceph-mon[73495]: pgmap v376: 17 pgs: 17 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:21.462 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T08:58:21.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:21.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T08:58:21.567 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T08:58:21.600 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:21.621 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T08:58:21.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:21.624 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T08:58:21.725 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T08:58:21.758 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:21.777 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T08:58:21.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:21.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T08:58:21.881 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T08:58:21.909 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:21.929 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T08:58:21.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:21.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T08:58:21.958 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T08:58:21.958 INFO:teuthology.orchestra.run.smithi026.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-03-20T08:58:22.033 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T08:58:22.065 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:21 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3333817029' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T08:58:22.065 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:21 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/130092334' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-20T08:58:22.066 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:22.087 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T08:58:22.089 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-20T08:58:22.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:22.090 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T08:58:22.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:21 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3333817029' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T08:58:22.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:21 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/130092334' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-20T08:58:22.191 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T08:58:22.221 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:22.241 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T08:58:22.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:22.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T08:58:22.345 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T08:58:22.374 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:22.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:21 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3333817029' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T08:58:22.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:21 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/130092334' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-20T08:58:22.824 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T08:58:22.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:22.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T08:58:22.928 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T08:58:22.955 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:22.976 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T08:58:22.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:22.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T08:58:23.091 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T08:58:23.123 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:23.144 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T08:58:23.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:23.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T08:58:23.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:22 smithi026 ceph-mon[68309]: pgmap v377: 17 pgs: 17 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:23.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:22 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T08:58:23.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:22 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3333817029' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T08:58:23.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:22 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/130092334' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-20T08:58:23.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:22 smithi026 ceph-mon[68309]: osdmap e125: 12 total, 12 up, 12 in 2024-03-20T08:58:23.247 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T08:58:23.275 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:23.295 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T08:58:23.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:23.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T08:58:23.306 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:58:23.399 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T08:58:23.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:22 smithi115 ceph-mon[72476]: pgmap v377: 17 pgs: 17 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:23.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:22 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T08:58:23.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:22 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3333817029' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T08:58:23.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:22 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/130092334' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-20T08:58:23.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:22 smithi115 ceph-mon[72476]: osdmap e125: 12 total, 12 up, 12 in 2024-03-20T08:58:23.431 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:23.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:22 smithi122 ceph-mon[73495]: pgmap v377: 17 pgs: 17 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:23.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:22 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T08:58:23.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:22 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3333817029' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T08:58:23.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:22 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/130092334' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-20T08:58:23.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:22 smithi122 ceph-mon[73495]: osdmap e125: 12 total, 12 up, 12 in 2024-03-20T08:58:23.451 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T08:58:23.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:23.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T08:58:23.555 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T08:58:23.588 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:23.609 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T08:58:23.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:23.612 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T08:58:23.713 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T08:58:23.745 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:23.765 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T08:58:23.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:23.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T08:58:23.870 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T08:58:23.899 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:23.993 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:58:23.996 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T08:58:23.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:23.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T08:58:24.100 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T08:58:24.133 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:24.154 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T08:58:24.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:24.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T08:58:24.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:24 smithi026 ceph-mon[68309]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-20T08:58:24.258 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T08:58:24.288 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:24 smithi122 ceph-mon[73495]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-20T08:58:24.291 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:24.312 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T08:58:24.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:24.315 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T08:58:24.337 DEBUG:teuthology.orchestra.run.smithi122:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-03-20T08:58:24.338 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-20T08:58:24.338 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T08:58:24.357 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-20T08:58:24.357 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-20T08:58:24.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:24 smithi115 ceph-mon[72476]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-20T08:58:24.418 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T08:58:24.428 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:hints = 1 2024-03-20T08:58:24.428 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-20T08:58:24.428 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Object prefix: benchmark_data_smithi122_107654 2024-03-20T08:58:24.429 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T08:58:24.429 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 0 0 0 0 0 0 - 0 2024-03-20T08:58:24.462 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:24.847 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T08:58:25.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:25 smithi026 ceph-mon[68309]: pgmap v379: 17 pgs: 17 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:25.342 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T08:58:25.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:25.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T08:58:25.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:25 smithi115 ceph-mon[72476]: pgmap v379: 17 pgs: 17 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:25.429 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 1 16 3767 3751 234.406 234.438 0.00138484 0.0038467 2024-03-20T08:58:25.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:25 smithi122 ceph-mon[73495]: pgmap v379: 17 pgs: 17 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T08:58:25.448 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T08:58:25.477 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:25.605 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T08:58:25.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:25.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T08:58:25.688 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T08:58:25.711 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T08:58:25.726 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T08:58:25.746 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:25.758 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T08:58:25.773 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T08:58:25.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:25.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T08:58:25.790 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T08:58:25.826 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T08:58:25.863 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T08:58:25.876 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T08:58:25.900 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T08:58:25.910 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:25.934 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T08:58:25.937 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T08:58:25.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:25.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T08:58:25.968 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T08:58:26.011 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T08:58:26.041 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T08:58:26.054 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T08:58:26.074 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:58:26.074 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:26.098 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T08:58:26.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:26.101 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T08:58:26.108 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T08:58:26.129 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T08:58:26.129 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T08:58:26.129 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T08:58:26.129 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T08:58:26.129 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T08:58:26.130 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T08:58:26.130 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T08:58:26.130 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T08:58:26.130 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T08:58:26.130 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T08:58:26.130 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T08:58:26.130 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T08:58:26.136 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:58:26.201 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T08:58:26.235 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:26.260 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T08:58:26.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:26.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T08:58:26.365 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T08:58:26.405 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:26 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/150883223' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T08:58:26.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:26 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/150883223' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T08:58:26.410 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:26.429 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 2 16 7307 7291 227.818 221.25 0.00234346 0.00437668 2024-03-20T08:58:26.436 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T08:58:26.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:26.439 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T08:58:26.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:26 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/150883223' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T08:58:26.539 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T08:58:26.565 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:26.583 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T08:58:26.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:26.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T08:58:26.689 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T08:58:26.739 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:26.765 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T08:58:26.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:26.769 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T08:58:26.869 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T08:58:26.910 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:26.937 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T08:58:26.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:26.940 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T08:58:27.017 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:27 smithi122 ceph-mon[73495]: pgmap v380: 17 pgs: 17 active+clean; 15 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s wr, 22 op/s 2024-03-20T08:58:27.041 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T08:58:27.085 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:27.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:27 smithi026 ceph-mon[68309]: pgmap v380: 17 pgs: 17 active+clean; 15 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s wr, 22 op/s 2024-03-20T08:58:27.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:27 smithi115 ceph-mon[72476]: pgmap v380: 17 pgs: 17 active+clean; 15 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s wr, 22 op/s 2024-03-20T08:58:27.429 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 3 16 10684 10668 222.227 211.062 0.00230466 0.00448879 2024-03-20T08:58:28.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:28 smithi026 ceph-mon[68309]: 1.0 scrub starts 2024-03-20T08:58:28.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:28 smithi026 ceph-mon[68309]: 1.0 scrub ok 2024-03-20T08:58:28.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:28 smithi115 ceph-mon[72476]: 1.0 scrub starts 2024-03-20T08:58:28.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:28 smithi115 ceph-mon[72476]: 1.0 scrub ok 2024-03-20T08:58:28.429 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 4 16 13680 13664 213.479 187.25 0.00452696 0.00467747 2024-03-20T08:58:28.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:28 smithi122 ceph-mon[73495]: 1.0 scrub starts 2024-03-20T08:58:28.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:28 smithi122 ceph-mon[73495]: 1.0 scrub ok 2024-03-20T08:58:28.817 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T08:58:28.817 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T08:58:28.900 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T08:58:29.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:29 smithi026 ceph-mon[68309]: pgmap v381: 17 pgs: 17 active+clean; 215 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s wr, 343 op/s 2024-03-20T08:58:29.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:29 smithi115 ceph-mon[72476]: pgmap v381: 17 pgs: 17 active+clean; 215 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s wr, 343 op/s 2024-03-20T08:58:29.430 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 5 15 17245 17230 215.339 222.875 0.00538193 0.00463786 2024-03-20T08:58:29.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:29 smithi122 ceph-mon[73495]: pgmap v381: 17 pgs: 17 active+clean; 215 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s wr, 343 op/s 2024-03-20T08:58:29.543 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-20T08:58:29.553 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T08:58:29.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:29.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T08:58:29.660 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T08:58:29.694 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:29.718 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T08:58:29.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:29.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T08:58:29.823 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T08:58:29.857 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:29.882 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T08:58:29.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:29.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T08:58:29.986 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T08:58:30.034 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:30.392 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T08:58:30.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:30.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T08:58:30.430 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 6 16 20277 20261 211.02 189.438 0.00142349 0.00470063 2024-03-20T08:58:30.497 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T08:58:30.523 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:30.542 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T08:58:30.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:30.544 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T08:58:30.646 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T08:58:30.693 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:30.714 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T08:58:30.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:30.718 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T08:58:30.720 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T08:58:30.740 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T08:58:30.764 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:58:30.785 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:58:30.818 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T08:58:30.853 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:30.878 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T08:58:30.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:30.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T08:58:30.984 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T08:58:31.024 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:31.054 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T08:58:31.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:31.059 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T08:58:31.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:31 smithi115 ceph-mon[72476]: pgmap v382: 17 pgs: 17 active+clean; 483 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 771 op/s 2024-03-20T08:58:31.159 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T08:58:31.198 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:31.222 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T08:58:31.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:31.225 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T08:58:31.326 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T08:58:31.354 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:31 smithi026 ceph-mon[68309]: pgmap v382: 17 pgs: 17 active+clean; 483 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 771 op/s 2024-03-20T08:58:31.357 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:31.388 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T08:58:31.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:31.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T08:58:31.430 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 7 15 22357 22342 199.453 130.062 0.00345516 0.00500914 2024-03-20T08:58:31.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:31 smithi122 ceph-mon[73495]: pgmap v382: 17 pgs: 17 active+clean; 483 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 771 op/s 2024-03-20T08:58:31.499 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T08:58:31.530 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:31.551 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T08:58:31.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:31.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T08:58:31.655 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T08:58:31.706 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:31.820 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T08:58:31.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:31.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T08:58:31.932 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T08:58:31.978 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:32.194 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T08:58:32.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:32.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T08:58:32.227 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":382,"stamp":"2024-03-20T08:58:29.441486+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":506069536,"num_objects":7715,"num_object_clones":0,"num_object_copies":23145,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7715,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":7894,"num_write_kb":496570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7850,"ondisk_log_size":7850,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":1396660,"kb_used_data":1010056,"kb_used_omap":18,"kb_used_meta":386477,"kb_avail":1123545164,"statfs":{"total":1151940427776,"available":1150510247936,"internally_reserved":0,"allocated":1034297344,"data_stored":1030694824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":395752763},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":505479168,"num_objects":7713,"num_object_clones":0,"num_object_copies":23139,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7713,"num_write_kb":493632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002240"},"pg_stats":[{"pgid":"2.f","version":"125'464","reported_seq":482,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-20T08:58:26.466173+0000","last_change":"2024-03-20T08:58:10.028191+0000","last_active":"2024-03-20T08:58:26.466173+0000","last_peered":"2024-03-20T08:58:26.466173+0000","last_clean":"2024-03-20T08:58:26.466173+0000","last_became_active":"2024-03-20T08:58:10.027677+0000","last_became_peered":"2024-03-20T08:58:10.027677+0000","last_unstale":"2024-03-20T08:58:26.466173+0000","last_undegraded":"2024-03-20T08:58:26.466173+0000","last_fullsized":"2024-03-20T08:58:26.466173+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T08:58:08.911564+0000","objects_scrubbed":0,"log_size":464,"log_dups_size":0,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:59:26.856134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30408704,"num_objects":464,"num_object_clones":0,"num_object_copies":1392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":464,"num_write_kb":29696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"125'48","reported_seq":64,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-20T08:58:24.568949+0000","last_change":"2024-03-20T08:58:10.016305+0000","last_active":"2024-03-20T08:58:24.568949+0000","last_peered":"2024-03-20T08:58:24.568949+0000","last_clean":"2024-03-20T08:58:24.568949+0000","last_became_active":"2024-03-20T08:58:10.015946+0000","last_became_peered":"2024-03-20T08:58:10.015946+0000","last_unstale":"2024-03-20T08:58:24.568949+0000","last_undegraded":"2024-03-20T08:58:24.568949+0000","last_fullsized":"2024-03-20T08:58:24.568949+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T08:58:08.911564+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:59:17.960135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3145728,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"125'589","reported_seq":607,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-20T08:58:27.055625+0000","last_change":"2024-03-20T08:58:09.933450+0000","last_active":"2024-03-20T08:58:27.055625+0000","last_peered":"2024-03-20T08:58:27.055625+0000","last_clean":"2024-03-20T08:58:27.055625+0000","last_became_active":"2024-03-20T08:58:09.933090+0000","last_became_peered":"2024-03-20T08:58:09.933090+0000","last_unstale":"2024-03-20T08:58:27.055625+0000","last_undegraded":"2024-03-20T08:58:27.055625+0000","last_fullsized":"2024-03-20T08:58:27.055625+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T08:58:08.911564+0000","objects_scrubbed":0,"log_size":589,"log_dups_size":0,"ondisk_log_size":589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:59:28.585191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38600704,"num_objects":589,"num_object_clones":0,"num_object_copies":1767,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":589,"num_write_kb":37696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"125'789","reported_seq":807,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-20T08:58:28.182276+0000","last_change":"2024-03-20T08:58:10.027823+0000","last_active":"2024-03-20T08:58:28.182276+0000","last_peered":"2024-03-20T08:58:28.182276+0000","last_clean":"2024-03-20T08:58:28.182276+0000","last_became_active":"2024-03-20T08:58:10.027595+0000","last_became_peered":"2024-03-20T08:58:10.027595+0000","last_unstale":"2024-03-20T08:58:28.182276+0000","last_undegraded":"2024-03-20T08:58:28.182276+0000","last_fullsized":"2024-03-20T08:58:28.182276+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T08:58:08.911564+0000","objects_scrubbed":0,"log_size":789,"log_dups_size":0,"ondisk_log_size":789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:59:09.907915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51707904,"num_objects":789,"num_object_clones":0,"num_object_copies":2367,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":789,"num_write_kb":50496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"125'599","reported_seq":617,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-20T08:58:27.049386+0000","last_change":"2024-03-20T08:58:09.929429+0000","last_active":"2024-03-20T08:58:27.049386+0000","last_peered":"2024-03-20T08:58:27.049386+0000","last_clean":"2024-03-20T08:58:27.049386+0000","last_became_active":"2024-03-20T08:58:09.929029+0000","last_became_peered":"2024-03-20T08:58:09.929029+0000","last_unstale":"2024-03-20T08:58:27.049386+0000","last_undegraded":"2024-03-20T08:58:27.049386+0000","last_fullsized":"2024-03-20T08:58:27.049386+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T08:58:08.911564+0000","objects_scrubbed":0,"log_size":599,"log_dups_size":0,"ondisk_log_size":599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:59:09.425707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39256064,"num_objects":599,"num_object_clones":0,"num_object_copies":1797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":599,"num_write_kb":38336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":395,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-20T08:58:22.353791+0000","last_change":"2024-03-20T08:56:57.309733+0000","last_active":"2024-03-20T08:58:22.353791+0000","last_peered":"2024-03-20T08:58:22.353791+0000","last_clean":"2024-03-20T08:58:22.353791+0000","last_became_active":"2024-03-20T08:56:21.910528+0000","last_became_peered":"2024-03-20T08:56:21.910528+0000","last_unstale":"2024-03-20T08:58:22.353791+0000","last_undegraded":"2024-03-20T08:58:22.353791+0000","last_fullsized":"2024-03-20T08:58:22.353791+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T08:56:57.309610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T08:56:57.309610+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"125'309","reported_seq":327,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-20T08:58:25.884802+0000","last_change":"2024-03-20T08:58:09.934319+0000","last_active":"2024-03-20T08:58:25.884802+0000","last_peered":"2024-03-20T08:58:25.884802+0000","last_clean":"2024-03-20T08:58:25.884802+0000","last_became_active":"2024-03-20T08:58:09.933985+0000","last_became_peered":"2024-03-20T08:58:09.933985+0000","last_unstale":"2024-03-20T08:58:25.884802+0000","last_undegraded":"2024-03-20T08:58:25.884802+0000","last_fullsized":"2024-03-20T08:58:25.884802+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T08:58:08.911564+0000","objects_scrubbed":0,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:59:09.611487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20250624,"num_objects":309,"num_object_clones":0,"num_object_copies":927,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":309,"num_write_kb":19776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"125'474","reported_seq":492,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-20T08:58:26.466146+0000","last_change":"2024-03-20T08:58:10.028082+0000","last_active":"2024-03-20T08:58:26.466146+0000","last_peered":"2024-03-20T08:58:26.466146+0000","last_clean":"2024-03-20T08:58:26.466146+0000","last_became_active":"2024-03-20T08:58:10.027686+0000","last_became_peered":"2024-03-20T08:58:10.027686+0000","last_unstale":"2024-03-20T08:58:26.466146+0000","last_undegraded":"2024-03-20T08:58:26.466146+0000","last_fullsized":"2024-03-20T08:58:26.466146+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T08:58:08.911564+0000","objects_scrubbed":0,"log_size":474,"log_dups_size":0,"ondisk_log_size":474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:59:25.737877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31064064,"num_objects":474,"num_object_clones":0,"num_object_copies":1422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":474,"num_write_kb":30336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"125'42","reported_seq":60,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-20T08:58:24.565710+0000","last_change":"2024-03-20T08:58:10.027969+0000","last_active":"2024-03-20T08:58:24.565710+0000","last_peered":"2024-03-20T08:58:24.565710+0000","last_clean":"2024-03-20T08:58:24.565710+0000","last_became_active":"2024-03-20T08:58:10.027648+0000","last_became_peered":"2024-03-20T08:58:10.027648+0000","last_unstale":"2024-03-20T08:58:24.565710+0000","last_undegraded":"2024-03-20T08:58:24.565710+0000","last_fullsized":"2024-03-20T08:58:24.565710+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T08:58:08.911564+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:59:19.108020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2752512,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42,"num_write_kb":2688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"125'104","reported_seq":121,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-20T08:58:24.817036+0000","last_change":"2024-03-20T08:58:09.932135+0000","last_active":"2024-03-20T08:58:24.817036+0000","last_peered":"2024-03-20T08:58:24.817036+0000","last_clean":"2024-03-20T08:58:24.817036+0000","last_became_active":"2024-03-20T08:58:09.931796+0000","last_became_peered":"2024-03-20T08:58:09.931796+0000","last_unstale":"2024-03-20T08:58:24.817036+0000","last_undegraded":"2024-03-20T08:58:24.817036+0000","last_fullsized":"2024-03-20T08:58:24.817036+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T08:58:08.911564+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:59:32.864060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6815744,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":104,"num_write_kb":6656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"125'338","reported_seq":356,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-20T08:58:25.894265+0000","last_change":"2024-03-20T08:58:10.016434+0000","last_active":"2024-03-20T08:58:25.894265+0000","last_peered":"2024-03-20T08:58:25.894265+0000","last_clean":"2024-03-20T08:58:25.894265+0000","last_became_active":"2024-03-20T08:58:10.016062+0000","last_became_peered":"2024-03-20T08:58:10.016062+0000","last_unstale":"2024-03-20T08:58:25.894265+0000","last_undegraded":"2024-03-20T08:58:25.894265+0000","last_fullsized":"2024-03-20T08:58:25.894265+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T08:58:08.911564+0000","objects_scrubbed":0,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:59:16.889915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22151168,"num_objects":338,"num_object_clones":0,"num_object_copies":1014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":338,"num_write_kb":21632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"125'31","reported_seq":49,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-20T08:58:24.556131+0000","last_change":"2024-03-20T08:58:10.027960+0000","last_active":"2024-03-20T08:58:24.556131+0000","last_peered":"2024-03-20T08:58:24.556131+0000","last_clean":"2024-03-20T08:58:24.556131+0000","last_became_active":"2024-03-20T08:58:10.027614+0000","last_became_peered":"2024-03-20T08:58:10.027614+0000","last_unstale":"2024-03-20T08:58:24.556131+0000","last_undegraded":"2024-03-20T08:58:24.556131+0000","last_fullsized":"2024-03-20T08:58:24.556131+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T08:58:08.911564+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:59:17.423605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2031616,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":1984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"125'1040","reported_seq":1058,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-20T08:58:29.073447+0000","last_change":"2024-03-20T08:58:10.028188+0000","last_active":"2024-03-20T08:58:29.073447+0000","last_peered":"2024-03-20T08:58:29.073447+0000","last_clean":"2024-03-20T08:58:29.073447+0000","last_became_active":"2024-03-20T08:58:10.027712+0000","last_became_peered":"2024-03-20T08:58:10.027712+0000","last_unstale":"2024-03-20T08:58:29.073447+0000","last_undegraded":"2024-03-20T08:58:29.073447+0000","last_fullsized":"2024-03-20T08:58:29.073447+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T08:58:08.911564+0000","objects_scrubbed":0,"log_size":1040,"log_dups_size":0,"ondisk_log_size":1040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:59:28.670773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68157440,"num_objects":1040,"num_object_clones":0,"num_object_copies":3120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1040,"num_write_kb":66560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"125'434","reported_seq":451,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-20T08:58:26.402749+0000","last_change":"2024-03-20T08:58:09.935460+0000","last_active":"2024-03-20T08:58:26.402749+0000","last_peered":"2024-03-20T08:58:26.402749+0000","last_clean":"2024-03-20T08:58:26.402749+0000","last_became_active":"2024-03-20T08:58:09.935063+0000","last_became_peered":"2024-03-20T08:58:09.935063+0000","last_unstale":"2024-03-20T08:58:26.402749+0000","last_undegraded":"2024-03-20T08:58:26.402749+0000","last_fullsized":"2024-03-20T08:58:26.402749+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T08:58:08.911564+0000","objects_scrubbed":0,"log_size":434,"log_dups_size":0,"ondisk_log_size":434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:59:26.694222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28442624,"num_objects":434,"num_object_clones":0,"num_object_copies":1302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":434,"num_write_kb":27776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"125'935","reported_seq":953,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-20T08:58:29.080353+0000","last_change":"2024-03-20T08:58:10.016729+0000","last_active":"2024-03-20T08:58:29.080353+0000","last_peered":"2024-03-20T08:58:29.080353+0000","last_clean":"2024-03-20T08:58:29.080353+0000","last_became_active":"2024-03-20T08:58:10.016313+0000","last_became_peered":"2024-03-20T08:58:10.016313+0000","last_unstale":"2024-03-20T08:58:29.080353+0000","last_undegraded":"2024-03-20T08:58:29.080353+0000","last_fullsized":"2024-03-20T08:58:29.080353+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T08:58:08.911564+0000","objects_scrubbed":0,"log_size":935,"log_dups_size":0,"ondisk_log_size":935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:59:35.809641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61276160,"num_objects":935,"num_object_clones":0,"num_object_copies":2805,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":935,"num_write_kb":59840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"125'736","reported_seq":754,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-20T08:58:27.619075+0000","last_change":"2024-03-20T08:58:10.028230+0000","last_active":"2024-03-20T08:58:27.619075+0000","last_peered":"2024-03-20T08:58:27.619075+0000","last_clean":"2024-03-20T08:58:27.619075+0000","last_became_active":"2024-03-20T08:58:10.027826+0000","last_became_peered":"2024-03-20T08:58:10.027826+0000","last_unstale":"2024-03-20T08:58:27.619075+0000","last_undegraded":"2024-03-20T08:58:27.619075+0000","last_fullsized":"2024-03-20T08:58:27.619075+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T08:58:08.911564+0000","objects_scrubbed":0,"log_size":736,"log_dups_size":0,"ondisk_log_size":736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:59:22.929706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48234496,"num_objects":736,"num_object_clones":0,"num_object_copies":2208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":736,"num_write_kb":47104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"125'781","reported_seq":799,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-20T08:58:28.172589+0000","last_change":"2024-03-20T08:58:10.027851+0000","last_active":"2024-03-20T08:58:28.172589+0000","last_peered":"2024-03-20T08:58:28.172589+0000","last_clean":"2024-03-20T08:58:28.172589+0000","last_became_active":"2024-03-20T08:58:10.027586+0000","last_became_peered":"2024-03-20T08:58:10.027586+0000","last_unstale":"2024-03-20T08:58:28.172589+0000","last_undegraded":"2024-03-20T08:58:28.172589+0000","last_fullsized":"2024-03-20T08:58:28.172589+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T08:58:08.911564+0000","objects_scrubbed":0,"log_size":781,"log_dups_size":0,"ondisk_log_size":781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:59:21.054532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51183616,"num_objects":781,"num_object_clones":0,"num_object_copies":2343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":781,"num_write_kb":49984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":505479168,"num_objects":7713,"num_object_clones":0,"num_object_copies":23139,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7713,"num_write_kb":493632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1356136448,"data_stored":1356136448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7713,"ondisk_log_size":7713,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860446,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":229380,"kb_used_data":189884,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93515772,"statfs":{"total":95995035648,"available":95760150528,"internally_reserved":0,"allocated":194441216,"data_stored":194138327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":89,"seq":382252089378,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62524,"kb_used_data":35508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93682628,"statfs":{"total":95995035648,"available":95931011072,"internally_reserved":0,"allocated":36360192,"data_stored":36060855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":80,"seq":343597383722,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":274364,"kb_used_data":234868,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93470788,"statfs":{"total":95995035648,"available":95714086912,"internally_reserved":0,"allocated":240504832,"data_stored":240205495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":70,"seq":300647710766,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":130684,"kb_used_data":99508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93614468,"statfs":{"total":95995035648,"available":95861215232,"internally_reserved":0,"allocated":101896192,"data_stored":101596875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":60,"seq":257698037811,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":87300,"kb_used_data":56124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93657852,"statfs":{"total":95995035648,"available":95905640448,"internally_reserved":0,"allocated":57470976,"data_stored":57168107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332152,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":34696,"kb_used_data":3516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93710456,"statfs":{"total":95995035648,"available":95959506944,"internally_reserved":0,"allocated":3600384,"data_stored":3297515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574930,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717440,"statfs":{"total":95995035648,"available":95966658560,"internally_reserved":0,"allocated":708608,"data_stored":409291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":157180,"kb_used_data":121844,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93587972,"statfs":{"total":95995035648,"available":95834083328,"internally_reserved":0,"allocated":124768256,"data_stored":124468939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411404,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":218236,"kb_used_data":182900,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93526916,"statfs":{"total":95995035648,"available":95771561984,"internally_reserved":0,"allocated":187289600,"data_stored":186990283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149767,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717440,"statfs":{"total":95995035648,"available":95966658560,"internally_reserved":0,"allocated":708608,"data_stored":409291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953538,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":50556,"kb_used_data":19380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93694596,"statfs":{"total":95995035648,"available":95943266304,"internally_reserved":0,"allocated":19845120,"data_stored":19545803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659197,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":96316,"kb_used_data":65140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93648836,"statfs":{"total":95995035648,"available":95896408064,"internally_reserved":0,"allocated":66703360,"data_stored":66404043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":183304192,"data_stored":183304192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12189696,"data_stored":12189696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":208404480,"data_stored":208404480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":45940736,"data_stored":45940736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36438016,"data_stored":36438016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":77922304,"data_stored":77922304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":13828096,"data_stored":13828096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":107806720,"data_stored":107806720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":121307136,"data_stored":121307136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":281346048,"data_stored":281346048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":72876032,"data_stored":72876032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":194772992,"data_stored":194772992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T08:58:32.229 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pool set .mgr pgp_num 1 2024-03-20T08:58:32.230 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:32 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:58:32.230 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:32 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:58:32.230 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:32 smithi026 ceph-mon[68309]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T08:58:32.230 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:32 smithi026 ceph-mon[68309]: osdmap e126: 12 total, 12 up, 12 in 2024-03-20T08:58:32.299 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T08:58:32.365 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:32.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:32 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:58:32.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:32 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:58:32.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:32 smithi115 ceph-mon[72476]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T08:58:32.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:32 smithi115 ceph-mon[72476]: osdmap e126: 12 total, 12 up, 12 in 2024-03-20T08:58:32.413 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T08:58:32.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:32.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T08:58:32.430 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 8 16 24508 24492 191.316 134.375 0.00203011 0.00518632 2024-03-20T08:58:32.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:32 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:58:32.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:32 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:58:32.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:32 smithi122 ceph-mon[73495]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T08:58:32.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:32 smithi122 ceph-mon[73495]: osdmap e126: 12 total, 12 up, 12 in 2024-03-20T08:58:32.519 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T08:58:32.584 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:33.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:33 smithi115 ceph-mon[72476]: pgmap v384: 17 pgs: 17 active+clean; 871 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s 2024-03-20T08:58:33.430 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 9 16 27473 27457 190.647 185.312 0.00579792 0.00524095 2024-03-20T08:58:33.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:33 smithi122 ceph-mon[73495]: pgmap v384: 17 pgs: 17 active+clean; 871 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s 2024-03-20T08:58:33.460 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T08:58:33.465 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:33 smithi026 ceph-mon[68309]: pgmap v384: 17 pgs: 17 active+clean; 871 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s 2024-03-20T08:58:33.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:33.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T08:58:33.568 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T08:58:33.607 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:33.635 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T08:58:33.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:33.639 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T08:58:33.740 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T08:58:33.776 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:33.807 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T08:58:33.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:33.809 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T08:58:33.910 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T08:58:33.960 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:34.022 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T08:58:34.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:34.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T08:58:34.126 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T08:58:34.159 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:34.160 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T08:58:34.189 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T08:58:34.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:34.192 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T08:58:34.200 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T08:58:34.238 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T08:58:34.283 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T08:58:34.293 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T08:58:34.324 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T08:58:34.359 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:34.366 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T08:58:34.410 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T08:58:34.430 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 10 16 29961 29945 187.132 155.5 0.00322886 0.005337 2024-03-20T08:58:34.440 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T08:58:34.477 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T08:58:34.522 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T08:58:34.557 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T08:58:34.601 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T08:58:34.615 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T08:58:34.615 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T08:58:34.615 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T08:58:34.615 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T08:58:34.615 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T08:58:34.616 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T08:58:34.616 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T08:58:34.616 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T08:58:34.616 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T08:58:34.616 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T08:58:34.616 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T08:58:34.616 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T08:58:35.088 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T08:58:35.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:35 smithi115 ceph-mon[72476]: pgmap v385: 17 pgs: 17 active+clean; 1.3 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 132 MiB/s wr, 2.11k op/s 2024-03-20T08:58:35.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:35 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1384047889' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T08:58:35.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:35 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3606340970' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T08:58:35.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:35 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T08:58:35.430 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 11 16 32263 32247 183.198 143.875 0.00410713 0.00545411 2024-03-20T08:58:35.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:35 smithi122 ceph-mon[73495]: pgmap v385: 17 pgs: 17 active+clean; 1.3 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 132 MiB/s wr, 2.11k op/s 2024-03-20T08:58:35.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:35 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1384047889' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T08:58:35.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:35 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3606340970' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T08:58:35.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:35 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T08:58:35.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:35 smithi026 ceph-mon[68309]: pgmap v385: 17 pgs: 17 active+clean; 1.3 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 132 MiB/s wr, 2.11k op/s 2024-03-20T08:58:35.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:35 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1384047889' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T08:58:35.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:35 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3606340970' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T08:58:35.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:35 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T08:58:36.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:36 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T08:58:36.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:36 smithi115 ceph-mon[72476]: Cluster is now healthy 2024-03-20T08:58:36.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:36 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T08:58:36.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:36 smithi115 ceph-mon[72476]: osdmap e127: 12 total, 12 up, 12 in 2024-03-20T08:58:36.431 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 12 16 34282 34266 178.435 126.188 0.0020649 0.0055992 2024-03-20T08:58:36.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:36 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T08:58:36.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:36 smithi122 ceph-mon[73495]: Cluster is now healthy 2024-03-20T08:58:36.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:36 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T08:58:36.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:36 smithi122 ceph-mon[73495]: osdmap e127: 12 total, 12 up, 12 in 2024-03-20T08:58:36.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:36 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T08:58:36.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:36 smithi026 ceph-mon[68309]: Cluster is now healthy 2024-03-20T08:58:36.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:36 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T08:58:36.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:36 smithi026 ceph-mon[68309]: osdmap e127: 12 total, 12 up, 12 in 2024-03-20T08:58:37.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:37 smithi115 ceph-mon[72476]: pgmap v387: 17 pgs: 17 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 176 MiB/s wr, 2.82k op/s 2024-03-20T08:58:37.431 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 13 16 36531 36515 175.522 140.562 0.00843395 0.00569196 2024-03-20T08:58:37.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:37 smithi122 ceph-mon[73495]: pgmap v387: 17 pgs: 17 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 176 MiB/s wr, 2.82k op/s 2024-03-20T08:58:37.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:37 smithi026 ceph-mon[68309]: pgmap v387: 17 pgs: 17 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 176 MiB/s wr, 2.82k op/s 2024-03-20T08:58:38.431 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 14 16 38885 38869 173.493 147.125 0.00276281 0.00575865 2024-03-20T08:58:39.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:39 smithi115 ceph-mon[72476]: pgmap v388: 17 pgs: 17 active+clean; 1.9 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 183 MiB/s wr, 2.93k op/s 2024-03-20T08:58:39.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:39 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:58:39.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:39 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:58:39.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:39 smithi115 ceph-mon[72476]: osdmap e128: 12 total, 12 up, 12 in 2024-03-20T08:58:39.431 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 15 15 40932 40917 170.459 128 0.00873168 0.00586183 2024-03-20T08:58:39.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:39 smithi122 ceph-mon[73495]: pgmap v388: 17 pgs: 17 active+clean; 1.9 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 183 MiB/s wr, 2.93k op/s 2024-03-20T08:58:39.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:39 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:58:39.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:39 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:58:39.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:39 smithi122 ceph-mon[73495]: osdmap e128: 12 total, 12 up, 12 in 2024-03-20T08:58:39.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:39 smithi026 ceph-mon[68309]: pgmap v388: 17 pgs: 17 active+clean; 1.9 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 183 MiB/s wr, 2.93k op/s 2024-03-20T08:58:39.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:39 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:58:39.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:39 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:58:39.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:39 smithi026 ceph-mon[68309]: osdmap e128: 12 total, 12 up, 12 in 2024-03-20T08:58:40.431 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 16 16 43259 43243 168.891 145.375 0.00578694 0.00591676 2024-03-20T08:58:40.681 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:58:41.254 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:41 smithi026 ceph-mon[68309]: pgmap v390: 17 pgs: 17 active+clean; 2.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 162 MiB/s wr, 2.59k op/s 2024-03-20T08:58:41.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:41 smithi115 ceph-mon[72476]: pgmap v390: 17 pgs: 17 active+clean; 2.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 162 MiB/s wr, 2.59k op/s 2024-03-20T08:58:41.431 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 17 16 45329 45313 166.566 129.375 0.00781029 0.00599817 2024-03-20T08:58:41.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:41 smithi122 ceph-mon[73495]: pgmap v390: 17 pgs: 17 active+clean; 2.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 162 MiB/s wr, 2.59k op/s 2024-03-20T08:58:42.229 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T08:58:42.229 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-03-20T08:58:42.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-03-20T08:58:42.230 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T08:58:42.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.230 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24550.0:35180 2.9 2:943c62f5:::benchmark_data_smithi122_107654_object35179:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-20T08:58:42.230 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:36.801805+0000", 2024-03-20T08:58:42.230 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.7890459539999997, 2024-03-20T08:58:42.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.091291168000000006, 2024-03-20T08:58:42.230 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:42.230 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:58:42.230 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T08:58:42.230 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24550", 2024-03-20T08:58:42.230 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:58:42.230 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 35180 2024-03-20T08:58:42.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.231 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:42.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:42.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.801805+0000", 2024-03-20T08:58:42.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:42.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.801805+0000", 2024-03-20T08:58:42.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:42.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.801806+0000", 2024-03-20T08:58:42.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8000000000000004e-07 2024-03-20T08:58:42.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:42.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.801906+0000", 2024-03-20T08:58:42.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000100208 2024-03-20T08:58:42.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:42.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.801918+0000", 2024-03-20T08:58:42.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.224e-05 2024-03-20T08:58:42.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:42.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.801930+0000", 2024-03-20T08:58:42.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1708e-05 2024-03-20T08:58:42.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:42.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.801980+0000", 2024-03-20T08:58:42.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0589e-05 2024-03-20T08:58:42.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:42.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.802102+0000", 2024-03-20T08:58:42.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012184300000000001 2024-03-20T08:58:42.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,7", 2024-03-20T08:58:42.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.802234+0000", 2024-03-20T08:58:42.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013141199999999999 2024-03-20T08:58:42.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:58:42.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.803751+0000", 2024-03-20T08:58:42.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015174889999999999 2024-03-20T08:58:42.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T08:58:42.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.892971+0000", 2024-03-20T08:58:42.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.089220115000000003 2024-03-20T08:58:42.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:58:42.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.893011+0000", 2024-03-20T08:58:42.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9777999999999998e-05 2024-03-20T08:58:42.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:58:42.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.893058+0000", 2024-03-20T08:58:42.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7386999999999999e-05 2024-03-20T08:58:42.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:42.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.893096+0000", 2024-03-20T08:58:42.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7518999999999998e-05 2024-03-20T08:58:42.236 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.236 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:42.236 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.236 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24550.0:35182 2.9 2:91e55e46:::benchmark_data_smithi122_107654_object35181:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-20T08:58:42.236 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:36.804143+0000", 2024-03-20T08:58:42.236 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.7867079180000003, 2024-03-20T08:58:42.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.089136269000000004, 2024-03-20T08:58:42.237 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:42.237 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:58:42.237 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T08:58:42.237 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24550", 2024-03-20T08:58:42.237 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:58:42.237 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 35182 2024-03-20T08:58:42.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.237 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:42.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:42.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.804143+0000", 2024-03-20T08:58:42.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:42.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.804143+0000", 2024-03-20T08:58:42.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:42.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.804144+0000", 2024-03-20T08:58:42.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.19e-06 2024-03-20T08:58:42.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:42.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.804198+0000", 2024-03-20T08:58:42.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4119000000000001e-05 2024-03-20T08:58:42.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:42.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.804209+0000", 2024-03-20T08:58:42.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1072e-05 2024-03-20T08:58:42.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:42.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.804216+0000", 2024-03-20T08:58:42.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3180000000000001e-06 2024-03-20T08:58:42.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:42.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.804277+0000", 2024-03-20T08:58:42.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0909999999999998e-05 2024-03-20T08:58:42.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:42.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.804474+0000", 2024-03-20T08:58:42.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019707500000000001 2024-03-20T08:58:42.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,7", 2024-03-20T08:58:42.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.804659+0000", 2024-03-20T08:58:42.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000184878 2024-03-20T08:58:42.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T08:58:42.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.893106+0000", 2024-03-20T08:58:42.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.088446569000000003 2024-03-20T08:58:42.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:58:42.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.893140+0000", 2024-03-20T08:58:42.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4051000000000003e-05 2024-03-20T08:58:42.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:58:42.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.893193+0000", 2024-03-20T08:58:42.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2753000000000001e-05 2024-03-20T08:58:42.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:58:42.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.893245+0000", 2024-03-20T08:58:42.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2398999999999998e-05 2024-03-20T08:58:42.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:42.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.893279+0000", 2024-03-20T08:58:42.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3935e-05 2024-03-20T08:58:42.243 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.243 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:42.243 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.243 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24550.0:35183 2.6 2:643fef38:::benchmark_data_smithi122_107654_object35182:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-20T08:58:42.243 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:36.804238+0000", 2024-03-20T08:58:42.243 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.7866121980000003, 2024-03-20T08:58:42.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.088714492000000006, 2024-03-20T08:58:42.243 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:42.244 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:58:42.244 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T08:58:42.244 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24550", 2024-03-20T08:58:42.244 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:58:42.244 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 35183 2024-03-20T08:58:42.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.244 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:42.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:42.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.804238+0000", 2024-03-20T08:58:42.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:42.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.804238+0000", 2024-03-20T08:58:42.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:42.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.804239+0000", 2024-03-20T08:58:42.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4000000000000002e-07 2024-03-20T08:58:42.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:42.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.804285+0000", 2024-03-20T08:58:42.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5859000000000003e-05 2024-03-20T08:58:42.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:42.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.804293+0000", 2024-03-20T08:58:42.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5769999999999994e-06 2024-03-20T08:58:42.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:42.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.804298+0000", 2024-03-20T08:58:42.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2010000000000001e-06 2024-03-20T08:58:42.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:42.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.804368+0000", 2024-03-20T08:58:42.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0241999999999996e-05 2024-03-20T08:58:42.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:42.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.804484+0000", 2024-03-20T08:58:42.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000115655 2024-03-20T08:58:42.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,8", 2024-03-20T08:58:42.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.804621+0000", 2024-03-20T08:58:42.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000136947 2024-03-20T08:58:42.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T08:58:42.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.806123+0000", 2024-03-20T08:58:42.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015025290000000001 2024-03-20T08:58:42.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:58:42.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.892806+0000", 2024-03-20T08:58:42.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.086683456000000006 2024-03-20T08:58:42.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:58:42.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.892852+0000", 2024-03-20T08:58:42.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5108000000000002e-05 2024-03-20T08:58:42.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:58:42.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.892905+0000", 2024-03-20T08:58:42.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3241000000000001e-05 2024-03-20T08:58:42.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:42.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.892953+0000", 2024-03-20T08:58:42.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8136999999999997e-05 2024-03-20T08:58:42.254 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.254 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:42.254 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.255 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24550.0:35184 2.9 2:9637df51:::benchmark_data_smithi122_107654_object35183:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-20T08:58:42.255 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:36.804308+0000", 2024-03-20T08:58:42.255 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.7865426879999999, 2024-03-20T08:58:42.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.095475447000000005, 2024-03-20T08:58:42.255 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:42.256 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:58:42.256 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T08:58:42.256 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24550", 2024-03-20T08:58:42.256 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:58:42.256 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 35184 2024-03-20T08:58:42.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.256 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:42.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:42.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.804308+0000", 2024-03-20T08:58:42.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:42.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.804308+0000", 2024-03-20T08:58:42.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:42.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.804308+0000", 2024-03-20T08:58:42.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3500000000000002e-07 2024-03-20T08:58:42.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:42.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.804404+0000", 2024-03-20T08:58:42.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5253000000000004e-05 2024-03-20T08:58:42.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:42.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.804410+0000", 2024-03-20T08:58:42.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7299999999999999e-06 2024-03-20T08:58:42.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:42.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.804415+0000", 2024-03-20T08:58:42.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7979999999999996e-06 2024-03-20T08:58:42.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:42.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.805984+0000", 2024-03-20T08:58:42.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015685880000000001 2024-03-20T08:58:42.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:42.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.806325+0000", 2024-03-20T08:58:42.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00034088899999999998 2024-03-20T08:58:42.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,7", 2024-03-20T08:58:42.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.891770+0000", 2024-03-20T08:58:42.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.085445307999999998 2024-03-20T08:58:42.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T08:58:42.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.896348+0000", 2024-03-20T08:58:42.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0045781839999999999 2024-03-20T08:58:42.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:58:42.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.896390+0000", 2024-03-20T08:58:42.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2274000000000002e-05 2024-03-20T08:58:42.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:58:42.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.899677+0000", 2024-03-20T08:58:42.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0032866089999999998 2024-03-20T08:58:42.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:58:42.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.899741+0000", 2024-03-20T08:58:42.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3959000000000002e-05 2024-03-20T08:58:42.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:42.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.899783+0000", 2024-03-20T08:58:42.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2419999999999997e-05 2024-03-20T08:58:42.267 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.267 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:42.267 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.268 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop_reply(client.24550.0:35180 2.9 e127/121 ondisk, result = 0)", 2024-03-20T08:58:42.268 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:36.804672+0000", 2024-03-20T08:58:42.268 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.786178778, 2024-03-20T08:58:42.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.088434805000000005, 2024-03-20T08:58:42.268 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:42.269 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T08:58:42.269 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:42.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:42.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.804672+0000", 2024-03-20T08:58:42.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:42.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.804672+0000", 2024-03-20T08:58:42.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:42.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.804673+0000", 2024-03-20T08:58:42.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.06e-07 2024-03-20T08:58:42.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:42.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.804677+0000", 2024-03-20T08:58:42.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2969999999999997e-06 2024-03-20T08:58:42.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:42.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.804690+0000", 2024-03-20T08:58:42.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3171e-05 2024-03-20T08:58:42.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:42.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.804707+0000", 2024-03-20T08:58:42.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.713e-05 2024-03-20T08:58:42.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:42.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.892999+0000", 2024-03-20T08:58:42.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.088292209999999996 2024-03-20T08:58:42.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:42.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.893006+0000", 2024-03-20T08:58:42.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6599999999999998e-06 2024-03-20T08:58:42.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:42.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.893107+0000", 2024-03-20T08:58:42.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010073099999999999 2024-03-20T08:58:42.275 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.275 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:42.275 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.275 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24550.0:35188 2.9 2:9a01d2ae:::benchmark_data_smithi122_107654_object35187:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-20T08:58:42.275 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:36.805067+0000", 2024-03-20T08:58:42.275 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.7857832800000004, 2024-03-20T08:58:42.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.094850124999999993, 2024-03-20T08:58:42.275 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:42.275 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:58:42.275 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T08:58:42.276 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24550", 2024-03-20T08:58:42.276 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:58:42.276 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 35188 2024-03-20T08:58:42.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.276 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:42.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:42.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.805067+0000", 2024-03-20T08:58:42.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:42.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.805067+0000", 2024-03-20T08:58:42.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:42.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.805068+0000", 2024-03-20T08:58:42.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5899999999999995e-07 2024-03-20T08:58:42.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:42.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.805135+0000", 2024-03-20T08:58:42.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6438000000000003e-05 2024-03-20T08:58:42.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:42.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.805143+0000", 2024-03-20T08:58:42.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4689999999999997e-06 2024-03-20T08:58:42.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:42.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.805152+0000", 2024-03-20T08:58:42.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5399999999999996e-06 2024-03-20T08:58:42.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:42.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.893310+0000", 2024-03-20T08:58:42.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.088158142999999994 2024-03-20T08:58:42.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:42.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.893433+0000", 2024-03-20T08:58:42.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000122994 2024-03-20T08:58:42.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,7", 2024-03-20T08:58:42.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.893578+0000", 2024-03-20T08:58:42.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014513199999999999 2024-03-20T08:58:42.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:58:42.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.896437+0000", 2024-03-20T08:58:42.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028590289999999999 2024-03-20T08:58:42.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T08:58:42.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.899506+0000", 2024-03-20T08:58:42.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0030690700000000001 2024-03-20T08:58:42.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:58:42.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.899835+0000", 2024-03-20T08:58:42.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00032922499999999999 2024-03-20T08:58:42.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:58:42.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.899881+0000", 2024-03-20T08:58:42.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5488999999999998e-05 2024-03-20T08:58:42.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:42.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.899918+0000", 2024-03-20T08:58:42.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6837e-05 2024-03-20T08:58:42.282 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.282 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:42.282 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.282 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.24550.0:35187 2.c e127/121 2:3c48990a:::benchmark_data_smithi122_107654_object35186:head v 127'2134, mlcod=127'2132)", 2024-03-20T08:58:42.282 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:36.806031+0000", 2024-03-20T08:58:42.283 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.7848199579999999, 2024-03-20T08:58:42.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.085244709000000002, 2024-03-20T08:58:42.283 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:42.283 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:58:42.283 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:42.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:42.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.806031+0000", 2024-03-20T08:58:42.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:42.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.806031+0000", 2024-03-20T08:58:42.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:42.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.806032+0000", 2024-03-20T08:58:42.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4200000000000005e-07 2024-03-20T08:58:42.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:42.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.806094+0000", 2024-03-20T08:58:42.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2700000000000006e-05 2024-03-20T08:58:42.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:42.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.806104+0000", 2024-03-20T08:58:42.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0193e-05 2024-03-20T08:58:42.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:42.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.806118+0000", 2024-03-20T08:58:42.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3067999999999999e-05 2024-03-20T08:58:42.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:42.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.806182+0000", 2024-03-20T08:58:42.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3998000000000006e-05 2024-03-20T08:58:42.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:42.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.806210+0000", 2024-03-20T08:58:42.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8242000000000002e-05 2024-03-20T08:58:42.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:58:42.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.891222+0000", 2024-03-20T08:58:42.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.085011887999999994 2024-03-20T08:58:42.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:42.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.891275+0000", 2024-03-20T08:58:42.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3777999999999999e-05 2024-03-20T08:58:42.286 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.286 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:42.286 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.287 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24550.0:35191 2.6 2:644224b9:::benchmark_data_smithi122_107654_object35190:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-20T08:58:42.287 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:36.806197+0000", 2024-03-20T08:58:42.287 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.7846539259999998, 2024-03-20T08:58:42.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.090422602000000005, 2024-03-20T08:58:42.287 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:42.287 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:58:42.287 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T08:58:42.287 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24550", 2024-03-20T08:58:42.287 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:58:42.287 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 35191 2024-03-20T08:58:42.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.287 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:42.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:42.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.806197+0000", 2024-03-20T08:58:42.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:42.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.806197+0000", 2024-03-20T08:58:42.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:42.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.806198+0000", 2024-03-20T08:58:42.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.569999999999999e-07 2024-03-20T08:58:42.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:42.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.806786+0000", 2024-03-20T08:58:42.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00058860499999999997 2024-03-20T08:58:42.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:42.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.891190+0000", 2024-03-20T08:58:42.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.084403575999999994 2024-03-20T08:58:42.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:42.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.891200+0000", 2024-03-20T08:58:42.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0403e-05 2024-03-20T08:58:42.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:42.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.891262+0000", 2024-03-20T08:58:42.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1246000000000003e-05 2024-03-20T08:58:42.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:42.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.891408+0000", 2024-03-20T08:58:42.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014648100000000001 2024-03-20T08:58:42.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,8", 2024-03-20T08:58:42.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.891567+0000", 2024-03-20T08:58:42.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000159011 2024-03-20T08:58:42.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T08:58:42.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.894962+0000", 2024-03-20T08:58:42.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003394776 2024-03-20T08:58:42.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:58:42.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.896110+0000", 2024-03-20T08:58:42.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011480170000000001 2024-03-20T08:58:42.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:58:42.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.896529+0000", 2024-03-20T08:58:42.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00041942800000000001 2024-03-20T08:58:42.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:58:42.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.896580+0000", 2024-03-20T08:58:42.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0352000000000002e-05 2024-03-20T08:58:42.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:42.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.896619+0000", 2024-03-20T08:58:42.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9749999999999997e-05 2024-03-20T08:58:42.292 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.292 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:42.293 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.293 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop_reply(client.24550.0:35182 2.9 e127/121 ondisk, result = 0)", 2024-03-20T08:58:42.293 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:36.806210+0000", 2024-03-20T08:58:42.293 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.7846410129999999, 2024-03-20T08:58:42.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.086936751000000007, 2024-03-20T08:58:42.293 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:42.293 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T08:58:42.293 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:42.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:42.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.806210+0000", 2024-03-20T08:58:42.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:42.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.806210+0000", 2024-03-20T08:58:42.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:42.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.806210+0000", 2024-03-20T08:58:42.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5599999999999995e-07 2024-03-20T08:58:42.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:42.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.806215+0000", 2024-03-20T08:58:42.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4839999999999998e-06 2024-03-20T08:58:42.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:42.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.806225+0000", 2024-03-20T08:58:42.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0645000000000001e-05 2024-03-20T08:58:42.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:42.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.806232+0000", 2024-03-20T08:58:42.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2079999999999997e-06 2024-03-20T08:58:42.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:42.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.893129+0000", 2024-03-20T08:58:42.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.086897595999999994 2024-03-20T08:58:42.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:42.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.893135+0000", 2024-03-20T08:58:42.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8579999999999996e-06 2024-03-20T08:58:42.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:42.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.893146+0000", 2024-03-20T08:58:42.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1404e-05 2024-03-20T08:58:42.296 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.296 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:42.296 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.297 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop_reply(client.24550.0:35183 2.6 e127/121 ondisk, result = 0)", 2024-03-20T08:58:42.297 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:36.806327+0000", 2024-03-20T08:58:42.297 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.7845235559999999, 2024-03-20T08:58:42.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.086490758000000001, 2024-03-20T08:58:42.297 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:42.297 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T08:58:42.297 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:42.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:42.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.806327+0000", 2024-03-20T08:58:42.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:42.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.806327+0000", 2024-03-20T08:58:42.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:42.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.806328+0000", 2024-03-20T08:58:42.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7000000000000005e-07 2024-03-20T08:58:42.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:42.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.806332+0000", 2024-03-20T08:58:42.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.25e-06 2024-03-20T08:58:42.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:42.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.891298+0000", 2024-03-20T08:58:42.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.084966455999999996 2024-03-20T08:58:42.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:42.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.891307+0000", 2024-03-20T08:58:42.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1319999999999994e-06 2024-03-20T08:58:42.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:42.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.892786+0000", 2024-03-20T08:58:42.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014799380000000001 2024-03-20T08:58:42.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:42.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.892800+0000", 2024-03-20T08:58:42.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3648e-05 2024-03-20T08:58:42.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:42.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.892818+0000", 2024-03-20T08:58:42.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7764000000000001e-05 2024-03-20T08:58:42.300 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.300 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:42.300 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.300 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop_reply(client.24550.0:35182 2.9 e127/121 ondisk, result = 0)", 2024-03-20T08:58:42.300 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:36.806995+0000", 2024-03-20T08:58:42.301 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4.7838557660000003, 2024-03-20T08:58:42.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.086288650999999994, 2024-03-20T08:58:42.301 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:42.301 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T08:58:42.301 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:42.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.301 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:42.301 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.806995+0000", 2024-03-20T08:58:42.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.301 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:42.301 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.806995+0000", 2024-03-20T08:58:42.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:42.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.806996+0000", 2024-03-20T08:58:42.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3600000000000003e-07 2024-03-20T08:58:42.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:42.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.806999+0000", 2024-03-20T08:58:42.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.912e-06 2024-03-20T08:58:42.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:42.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.891183+0000", 2024-03-20T08:58:42.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.084183230999999997 2024-03-20T08:58:42.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:42.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.891199+0000", 2024-03-20T08:58:42.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6220999999999999e-05 2024-03-20T08:58:42.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:42.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.893169+0000", 2024-03-20T08:58:42.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019699079999999998 2024-03-20T08:58:42.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:42.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.893175+0000", 2024-03-20T08:58:42.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3090000000000001e-06 2024-03-20T08:58:42.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:42.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.893284+0000", 2024-03-20T08:58:42.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000108434 2024-03-20T08:58:42.304 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.304 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:42.304 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.304 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24550.0:37406 2.4 2:2001f8a7:::benchmark_data_smithi122_107654_object37405:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-20T08:58:42.304 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:37.792384+0000", 2024-03-20T08:58:42.304 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 3.798466957, 2024-03-20T08:58:42.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.071392886000000003, 2024-03-20T08:58:42.305 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:42.305 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:58:42.305 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T08:58:42.305 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24550", 2024-03-20T08:58:42.305 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:58:42.305 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 37406 2024-03-20T08:58:42.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.305 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:42.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:42.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.792384+0000", 2024-03-20T08:58:42.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:42.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.792384+0000", 2024-03-20T08:58:42.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:42.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.792385+0000", 2024-03-20T08:58:42.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2860000000000001e-06 2024-03-20T08:58:42.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:42.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.792420+0000", 2024-03-20T08:58:42.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5193e-05 2024-03-20T08:58:42.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:42.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.792431+0000", 2024-03-20T08:58:42.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0507e-05 2024-03-20T08:58:42.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:42.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.792436+0000", 2024-03-20T08:58:42.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9189999999999998e-06 2024-03-20T08:58:42.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:42.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.792478+0000", 2024-03-20T08:58:42.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2113000000000002e-05 2024-03-20T08:58:42.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:42.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.792577+0000", 2024-03-20T08:58:42.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9110999999999993e-05 2024-03-20T08:58:42.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,10", 2024-03-20T08:58:42.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.792680+0000", 2024-03-20T08:58:42.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103378 2024-03-20T08:58:42.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T08:58:42.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.794767+0000", 2024-03-20T08:58:42.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0020866499999999998 2024-03-20T08:58:42.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.309 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:58:42.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.795569+0000", 2024-03-20T08:58:42.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00080189499999999995 2024-03-20T08:58:42.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.309 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:58:42.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.863676+0000", 2024-03-20T08:58:42.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.068107382999999994 2024-03-20T08:58:42.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.309 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:58:42.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.863730+0000", 2024-03-20T08:58:42.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3533999999999999e-05 2024-03-20T08:58:42.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:42.310 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.863777+0000", 2024-03-20T08:58:42.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6916999999999998e-05 2024-03-20T08:58:42.310 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.310 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:42.310 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.310 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24550.0:37425 2.9 2:9dc12bef:::benchmark_data_smithi122_107654_object37424:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-20T08:58:42.310 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:37.795946+0000", 2024-03-20T08:58:42.310 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 3.794904812, 2024-03-20T08:58:42.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.070048684, 2024-03-20T08:58:42.311 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:42.311 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:58:42.311 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T08:58:42.311 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24550", 2024-03-20T08:58:42.311 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:58:42.311 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 37425 2024-03-20T08:58:42.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.311 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:42.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:42.311 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.795946+0000", 2024-03-20T08:58:42.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:42.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.795946+0000", 2024-03-20T08:58:42.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:42.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.795947+0000", 2024-03-20T08:58:42.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1500000000000003e-07 2024-03-20T08:58:42.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:42.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.796039+0000", 2024-03-20T08:58:42.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2624e-05 2024-03-20T08:58:42.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:42.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.796051+0000", 2024-03-20T08:58:42.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1161e-05 2024-03-20T08:58:42.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:42.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.796058+0000", 2024-03-20T08:58:42.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6389999999999992e-06 2024-03-20T08:58:42.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:42.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.796107+0000", 2024-03-20T08:58:42.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9205000000000001e-05 2024-03-20T08:58:42.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:42.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.796224+0000", 2024-03-20T08:58:42.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000117057 2024-03-20T08:58:42.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,7", 2024-03-20T08:58:42.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.796347+0000", 2024-03-20T08:58:42.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000122939 2024-03-20T08:58:42.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:58:42.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.864237+0000", 2024-03-20T08:58:42.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.067889770000000002 2024-03-20T08:58:42.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:58:42.315 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.864283+0000", 2024-03-20T08:58:42.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5581000000000003e-05 2024-03-20T08:58:42.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T08:58:42.315 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.865871+0000", 2024-03-20T08:58:42.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001587855 2024-03-20T08:58:42.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:58:42.315 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.865933+0000", 2024-03-20T08:58:42.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2176999999999998e-05 2024-03-20T08:58:42.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.316 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:42.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.865995+0000", 2024-03-20T08:58:42.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1760999999999994e-05 2024-03-20T08:58:42.316 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.316 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:42.316 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.316 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24550.0:37432 2.4 2:25cf42c0:::benchmark_data_smithi122_107654_object37431:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-20T08:58:42.316 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:37.799625+0000", 2024-03-20T08:58:42.316 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 3.7912258890000001, 2024-03-20T08:58:42.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.071273013999999996, 2024-03-20T08:58:42.316 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:42.316 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:58:42.317 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T08:58:42.317 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24550", 2024-03-20T08:58:42.317 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:58:42.317 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 37432 2024-03-20T08:58:42.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.317 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:42.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:42.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.799625+0000", 2024-03-20T08:58:42.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:42.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.799625+0000", 2024-03-20T08:58:42.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:42.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.799626+0000", 2024-03-20T08:58:42.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6279999999999999e-06 2024-03-20T08:58:42.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:42.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.799700+0000", 2024-03-20T08:58:42.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3807999999999995e-05 2024-03-20T08:58:42.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:42.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.862932+0000", 2024-03-20T08:58:42.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.063231870999999995 2024-03-20T08:58:42.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:42.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.862943+0000", 2024-03-20T08:58:42.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0943e-05 2024-03-20T08:58:42.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:42.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.863814+0000", 2024-03-20T08:58:42.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00087065099999999996 2024-03-20T08:58:42.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:42.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.863970+0000", 2024-03-20T08:58:42.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015613199999999999 2024-03-20T08:58:42.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,10", 2024-03-20T08:58:42.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.864110+0000", 2024-03-20T08:58:42.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014053700000000001 2024-03-20T08:58:42.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:58:42.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.870590+0000", 2024-03-20T08:58:42.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0064793660000000003 2024-03-20T08:58:42.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:58:42.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.870738+0000", 2024-03-20T08:58:42.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000148739 2024-03-20T08:58:42.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.321 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T08:58:42.321 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.870807+0000", 2024-03-20T08:58:42.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8466000000000005e-05 2024-03-20T08:58:42.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.321 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:58:42.321 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.870860+0000", 2024-03-20T08:58:42.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2967999999999998e-05 2024-03-20T08:58:42.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.321 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:42.321 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.870898+0000", 2024-03-20T08:58:42.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7904999999999997e-05 2024-03-20T08:58:42.322 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.322 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:42.322 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.322 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24550.0:40692 2.6 2:6db1a368:::benchmark_data_smithi122_107654_object40691:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-20T08:58:42.322 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:39.285650+0000", 2024-03-20T08:58:42.322 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 2.3052010119999999, 2024-03-20T08:58:42.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.074322559999999996, 2024-03-20T08:58:42.322 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:42.322 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:58:42.322 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T08:58:42.323 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24550", 2024-03-20T08:58:42.323 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:58:42.323 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40692 2024-03-20T08:58:42.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.323 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:42.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:42.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.285650+0000", 2024-03-20T08:58:42.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:42.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.285650+0000", 2024-03-20T08:58:42.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:42.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.285650+0000", 2024-03-20T08:58:42.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3399999999999998e-07 2024-03-20T08:58:42.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:42.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.285745+0000", 2024-03-20T08:58:42.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4825000000000001e-05 2024-03-20T08:58:42.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:42.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.285754+0000", 2024-03-20T08:58:42.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1109999999999997e-06 2024-03-20T08:58:42.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:42.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.285760+0000", 2024-03-20T08:58:42.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6520000000000003e-06 2024-03-20T08:58:42.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:42.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.285811+0000", 2024-03-20T08:58:42.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1428999999999998e-05 2024-03-20T08:58:42.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:42.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.285929+0000", 2024-03-20T08:58:42.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000118051 2024-03-20T08:58:42.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,8", 2024-03-20T08:58:42.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.286065+0000", 2024-03-20T08:58:42.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000136017 2024-03-20T08:58:42.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T08:58:42.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.287305+0000", 2024-03-20T08:58:42.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012394999999999999 2024-03-20T08:58:42.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:58:42.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.359724+0000", 2024-03-20T08:58:42.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.072418900999999994 2024-03-20T08:58:42.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:58:42.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.359854+0000", 2024-03-20T08:58:42.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013059399999999999 2024-03-20T08:58:42.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:58:42.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.359918+0000", 2024-03-20T08:58:42.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3139999999999995e-05 2024-03-20T08:58:42.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:42.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.359972+0000", 2024-03-20T08:58:42.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4605999999999999e-05 2024-03-20T08:58:42.328 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.328 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:42.328 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.328 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24550.0:40704 2.9 2:9ea0d108:::benchmark_data_smithi122_107654_object40703:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-20T08:58:42.328 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:39.288147+0000", 2024-03-20T08:58:42.328 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 2.3027040740000002, 2024-03-20T08:58:42.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.071851931999999993, 2024-03-20T08:58:42.328 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:42.328 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:58:42.328 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T08:58:42.328 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24550", 2024-03-20T08:58:42.329 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:58:42.329 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40704 2024-03-20T08:58:42.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.329 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:42.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:42.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.288147+0000", 2024-03-20T08:58:42.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:42.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.288147+0000", 2024-03-20T08:58:42.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:42.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.288148+0000", 2024-03-20T08:58:42.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1230000000000001e-06 2024-03-20T08:58:42.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:42.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.288224+0000", 2024-03-20T08:58:42.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6544000000000004e-05 2024-03-20T08:58:42.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:42.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.288235+0000", 2024-03-20T08:58:42.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0353999999999999e-05 2024-03-20T08:58:42.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:42.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.288242+0000", 2024-03-20T08:58:42.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4880000000000001e-06 2024-03-20T08:58:42.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:42.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.288291+0000", 2024-03-20T08:58:42.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8896000000000001e-05 2024-03-20T08:58:42.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:42.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.288403+0000", 2024-03-20T08:58:42.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000112349 2024-03-20T08:58:42.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,7", 2024-03-20T08:58:42.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.288549+0000", 2024-03-20T08:58:42.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014557799999999999 2024-03-20T08:58:42.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T08:58:42.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.289992+0000", 2024-03-20T08:58:42.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001443459 2024-03-20T08:58:42.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:58:42.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.359859+0000", 2024-03-20T08:58:42.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.069866703000000002 2024-03-20T08:58:42.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:58:42.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.359899+0000", 2024-03-20T08:58:42.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9885000000000002e-05 2024-03-20T08:58:42.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:58:42.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.359956+0000", 2024-03-20T08:58:42.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7309999999999998e-05 2024-03-20T08:58:42.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:42.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.359999+0000", 2024-03-20T08:58:42.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2243000000000002e-05 2024-03-20T08:58:42.333 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.334 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:42.334 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.334 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24550.0:40705 2.6 2:661e3393:::benchmark_data_smithi122_107654_object40704:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-20T08:58:42.334 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:39.290462+0000", 2024-03-20T08:58:42.334 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 2.3003891040000002, 2024-03-20T08:58:42.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.074785744000000001, 2024-03-20T08:58:42.334 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:42.334 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:58:42.334 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T08:58:42.334 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24550", 2024-03-20T08:58:42.334 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:58:42.334 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 40705 2024-03-20T08:58:42.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.335 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:42.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:42.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.290462+0000", 2024-03-20T08:58:42.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:42.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.290462+0000", 2024-03-20T08:58:42.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:42.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.290463+0000", 2024-03-20T08:58:42.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2550000000000001e-06 2024-03-20T08:58:42.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:42.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.290573+0000", 2024-03-20T08:58:42.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000109968 2024-03-20T08:58:42.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:42.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.359593+0000", 2024-03-20T08:58:42.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.069020403999999994 2024-03-20T08:58:42.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:42.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.359626+0000", 2024-03-20T08:58:42.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2703000000000002e-05 2024-03-20T08:58:42.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.337 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:42.337 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.360016+0000", 2024-03-20T08:58:42.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00039003699999999998 2024-03-20T08:58:42.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.337 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:42.337 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.360159+0000", 2024-03-20T08:58:42.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014284399999999999 2024-03-20T08:58:42.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.337 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,8", 2024-03-20T08:58:42.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.360261+0000", 2024-03-20T08:58:42.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000102038 2024-03-20T08:58:42.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:58:42.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.364651+0000", 2024-03-20T08:58:42.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043905690000000004 2024-03-20T08:58:42.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T08:58:42.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.364713+0000", 2024-03-20T08:58:42.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1849999999999999e-05 2024-03-20T08:58:42.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:58:42.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.365171+0000", 2024-03-20T08:58:42.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00045754699999999999 2024-03-20T08:58:42.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:58:42.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.365216+0000", 2024-03-20T08:58:42.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5305000000000001e-05 2024-03-20T08:58:42.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:42.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.365247+0000", 2024-03-20T08:58:42.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1223999999999998e-05 2024-03-20T08:58:42.339 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.339 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:42.340 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.340 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24550.0:43333 2.9 2:90755d9b:::benchmark_data_smithi122_107654_object43332:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-20T08:58:42.340 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:40.465456+0000", 2024-03-20T08:58:42.340 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 1.125394628, 2024-03-20T08:58:42.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.070420483000000006, 2024-03-20T08:58:42.340 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:42.340 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:58:42.340 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T08:58:42.340 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24550", 2024-03-20T08:58:42.340 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:58:42.340 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 43333 2024-03-20T08:58:42.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.341 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:42.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:42.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.465456+0000", 2024-03-20T08:58:42.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:42.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.465456+0000", 2024-03-20T08:58:42.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:42.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.465457+0000", 2024-03-20T08:58:42.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4799999999999997e-07 2024-03-20T08:58:42.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:42.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.465553+0000", 2024-03-20T08:58:42.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.6082000000000006e-05 2024-03-20T08:58:42.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:42.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.465566+0000", 2024-03-20T08:58:42.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3319000000000001e-05 2024-03-20T08:58:42.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:42.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.465575+0000", 2024-03-20T08:58:42.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9609999999999994e-06 2024-03-20T08:58:42.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:42.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.465622+0000", 2024-03-20T08:58:42.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6660999999999999e-05 2024-03-20T08:58:42.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:42.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.465752+0000", 2024-03-20T08:58:42.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013021300000000001 2024-03-20T08:58:42.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,7", 2024-03-20T08:58:42.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.465912+0000", 2024-03-20T08:58:42.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015981200000000001 2024-03-20T08:58:42.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T08:58:42.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.468730+0000", 2024-03-20T08:58:42.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028184249999999998 2024-03-20T08:58:42.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:58:42.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.469049+0000", 2024-03-20T08:58:42.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00031873999999999997 2024-03-20T08:58:42.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:58:42.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.471584+0000", 2024-03-20T08:58:42.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0025344320000000001 2024-03-20T08:58:42.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:58:42.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.471641+0000", 2024-03-20T08:58:42.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7552000000000001e-05 2024-03-20T08:58:42.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:42.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.535877+0000", 2024-03-20T08:58:42.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.064235338000000003 2024-03-20T08:58:42.345 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.345 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:42.345 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.346 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24550.0:43334 2.9 2:91a0515f:::benchmark_data_smithi122_107654_object43333:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-20T08:58:42.346 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:40.465594+0000", 2024-03-20T08:58:42.346 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 1.1252569269999999, 2024-03-20T08:58:42.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.073984373000000006, 2024-03-20T08:58:42.346 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:42.346 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:58:42.346 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T08:58:42.346 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24550", 2024-03-20T08:58:42.346 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:58:42.346 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 43334 2024-03-20T08:58:42.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.346 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:42.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:42.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.465594+0000", 2024-03-20T08:58:42.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:42.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.465594+0000", 2024-03-20T08:58:42.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:42.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.465594+0000", 2024-03-20T08:58:42.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.37e-07 2024-03-20T08:58:42.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.348 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:42.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.465650+0000", 2024-03-20T08:58:42.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5667000000000001e-05 2024-03-20T08:58:42.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.348 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:42.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.465661+0000", 2024-03-20T08:58:42.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0685e-05 2024-03-20T08:58:42.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.348 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:42.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.465667+0000", 2024-03-20T08:58:42.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8279999999999997e-06 2024-03-20T08:58:42.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:42.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.467330+0000", 2024-03-20T08:58:42.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001662815 2024-03-20T08:58:42.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:42.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.467474+0000", 2024-03-20T08:58:42.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000143719 2024-03-20T08:58:42.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,7", 2024-03-20T08:58:42.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.467577+0000", 2024-03-20T08:58:42.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000103146 2024-03-20T08:58:42.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T08:58:42.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.468982+0000", 2024-03-20T08:58:42.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014045990000000001 2024-03-20T08:58:42.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:58:42.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.539419+0000", 2024-03-20T08:58:42.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.070437071000000004 2024-03-20T08:58:42.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:58:42.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.539461+0000", 2024-03-20T08:58:42.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2379999999999997e-05 2024-03-20T08:58:42.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:58:42.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.539520+0000", 2024-03-20T08:58:42.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8615e-05 2024-03-20T08:58:42.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:42.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.539578+0000", 2024-03-20T08:58:42.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8310999999999998e-05 2024-03-20T08:58:42.351 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.351 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:42.351 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.352 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24550.0:43345 2.6 2:6fc6989d:::benchmark_data_smithi122_107654_object43344:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-20T08:58:42.352 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:40.470488+0000", 2024-03-20T08:58:42.352 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 1.12036318, 2024-03-20T08:58:42.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.069863893999999996, 2024-03-20T08:58:42.352 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:58:42.352 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:58:42.352 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T08:58:42.352 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24550", 2024-03-20T08:58:42.352 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:58:42.352 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 43345 2024-03-20T08:58:42.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.352 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:58:42.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:58:42.353 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.470488+0000", 2024-03-20T08:58:42.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:58:42.353 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.470488+0000", 2024-03-20T08:58:42.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:58:42.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:58:42.353 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.470489+0000", 2024-03-20T08:58:42.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.004e-06 2024-03-20T08:58:42.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.354 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:58:42.354 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.470524+0000", 2024-03-20T08:58:42.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5098999999999997e-05 2024-03-20T08:58:42.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.354 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:58:42.354 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.470537+0000", 2024-03-20T08:58:42.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3086e-05 2024-03-20T08:58:42.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.354 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:58:42.354 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.470545+0000", 2024-03-20T08:58:42.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7239999999999998e-06 2024-03-20T08:58:42.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:58:42.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.470610+0000", 2024-03-20T08:58:42.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4395999999999998e-05 2024-03-20T08:58:42.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:58:42.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.470719+0000", 2024-03-20T08:58:42.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000109079 2024-03-20T08:58:42.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,8", 2024-03-20T08:58:42.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.470844+0000", 2024-03-20T08:58:42.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000125456 2024-03-20T08:58:42.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T08:58:42.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.535906+0000", 2024-03-20T08:58:42.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.065061892999999996 2024-03-20T08:58:42.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:58:42.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.536059+0000", 2024-03-20T08:58:42.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015292800000000001 2024-03-20T08:58:42.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:58:42.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.540245+0000", 2024-03-20T08:58:42.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0041853419999999999 2024-03-20T08:58:42.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.357 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:58:42.357 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.540306+0000", 2024-03-20T08:58:42.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1253999999999993e-05 2024-03-20T08:58:42.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:58:42.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:58:42.357 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:58:42.357 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:40.540351+0000", 2024-03-20T08:58:42.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5633000000000002e-05 2024-03-20T08:58:42.357 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.357 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:42.357 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.357 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:58:42.357 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:58:42.358 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T08:58:42.431 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 18 15 47225 47210 163.899 118.562 0.00777776 0.0060972 2024-03-20T08:58:42.729 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T08:58:42.864 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T08:58:42.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:42.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T08:58:42.914 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-20T08:58:42.980 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T08:58:43.082 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:43.123 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:43 smithi026 ceph-mon[68309]: pgmap v391: 17 pgs: 17 active+clean; 2.4 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 139 MiB/s wr, 2.22k op/s 2024-03-20T08:58:43.318 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T08:58:43.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:43.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T08:58:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:43 smithi115 ceph-mon[72476]: pgmap v391: 17 pgs: 17 active+clean; 2.4 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 139 MiB/s wr, 2.22k op/s 2024-03-20T08:58:43.423 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T08:58:43.431 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 19 16 48994 48978 161.088 110.5 0.00228639 0.00616456 2024-03-20T08:58:43.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:43 smithi122 ceph-mon[73495]: pgmap v391: 17 pgs: 17 active+clean; 2.4 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 139 MiB/s wr, 2.22k op/s 2024-03-20T08:58:43.449 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:43.468 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T08:58:43.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:43.470 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T08:58:43.571 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T08:58:43.603 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:43.625 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T08:58:43.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:43.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T08:58:43.729 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T08:58:43.771 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:43.796 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T08:58:43.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:43.801 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T08:58:43.869 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:58:43.901 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T08:58:43.920 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:58:43.936 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:43.961 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T08:58:43.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:43.964 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T08:58:44.066 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T08:58:44.101 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:44.124 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T08:58:44.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:44.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T08:58:44.228 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T08:58:44.290 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:44.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:44 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:58:44.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:44 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:58:44.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:44 smithi115 ceph-mon[72476]: osdmap e129: 12 total, 12 up, 12 in 2024-03-20T08:58:44.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:44 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:58:44.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:44 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:58:44.416 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:44 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:58:44.416 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:44 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:58:44.416 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:44 smithi026 ceph-mon[68309]: osdmap e129: 12 total, 12 up, 12 in 2024-03-20T08:58:44.416 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:44 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:58:44.416 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:44 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:58:44.432 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:2024-03-20T08:58:44.432524+0000 min lat: 0.00103856 max lat: 0.209407 avg lat: 0.00630191 2024-03-20T08:58:44.432 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T08:58:44.432 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 20 16 50753 50737 158.531 109.938 0.0112511 0.00630191 2024-03-20T08:58:44.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:44 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:58:44.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:44 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:58:44.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:44 smithi122 ceph-mon[73495]: osdmap e129: 12 total, 12 up, 12 in 2024-03-20T08:58:44.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:44 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:58:44.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:44 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:58:45.175 INFO:teuthology.orchestra.run.smithi026.stderr:set pool 1 pgp_num to 1 2024-03-20T08:58:45.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:45 smithi026 ceph-mon[68309]: pgmap v393: 17 pgs: 17 active+clean; 2.7 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 143 MiB/s wr, 2.29k op/s 2024-03-20T08:58:45.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:45 smithi026 ceph-mon[68309]: osdmap e130: 12 total, 12 up, 12 in 2024-03-20T08:58:45.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:45 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3731880449' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-20T08:58:45.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:45 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-20T08:58:45.360 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T08:58:45.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:45 smithi115 ceph-mon[72476]: pgmap v393: 17 pgs: 17 active+clean; 2.7 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 143 MiB/s wr, 2.29k op/s 2024-03-20T08:58:45.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:45 smithi115 ceph-mon[72476]: osdmap e130: 12 total, 12 up, 12 in 2024-03-20T08:58:45.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:45 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3731880449' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-20T08:58:45.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:45 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-20T08:58:45.432 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 21 16 52769 52753 156.981 126 0.00870677 0.00636527 2024-03-20T08:58:45.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:45 smithi122 ceph-mon[73495]: pgmap v393: 17 pgs: 17 active+clean; 2.7 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 143 MiB/s wr, 2.29k op/s 2024-03-20T08:58:45.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:45 smithi122 ceph-mon[73495]: osdmap e130: 12 total, 12 up, 12 in 2024-03-20T08:58:45.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:45 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3731880449' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-20T08:58:45.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:45 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-20T08:58:45.453 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T08:58:45.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:45.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T08:58:45.562 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T08:58:45.594 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:45.619 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T08:58:45.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:45.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T08:58:45.723 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T08:58:45.765 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:45.792 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T08:58:45.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:45.796 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T08:58:45.897 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T08:58:45.923 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:45.942 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T08:58:45.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:45.944 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T08:58:46.046 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T08:58:46.086 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:46.116 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T08:58:46.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:46.120 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T08:58:46.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:46 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-20T08:58:46.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:46 smithi026 ceph-mon[68309]: osdmap e131: 12 total, 12 up, 12 in 2024-03-20T08:58:46.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:58:46.220 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T08:58:46.253 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:46.279 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T08:58:46.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:46.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T08:58:46.385 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T08:58:46.419 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:46 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-20T08:58:46.419 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:46 smithi122 ceph-mon[73495]: osdmap e131: 12 total, 12 up, 12 in 2024-03-20T08:58:46.419 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:58:46.423 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:46.432 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 22 16 54447 54431 154.612 104.875 0.0121878 0.00646189 2024-03-20T08:58:46.451 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T08:58:46.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:46.458 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T08:58:46.559 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T08:58:46.585 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:46 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-20T08:58:46.585 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:46 smithi115 ceph-mon[72476]: osdmap e131: 12 total, 12 up, 12 in 2024-03-20T08:58:46.585 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:58:46.587 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:46.609 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T08:58:46.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:46.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T08:58:46.713 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T08:58:46.742 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:46.766 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T08:58:46.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:46.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T08:58:46.871 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T08:58:46.904 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:46.930 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T08:58:46.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:46.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T08:58:47.036 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T08:58:47.109 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:47.204 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T08:58:47.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:47.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T08:58:47.304 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T08:58:47.311 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T08:58:47.344 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T08:58:47.362 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:47 smithi026 ceph-mon[68309]: pgmap v396: 17 pgs: 17 active+clean; 2.9 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 138 MiB/s wr, 2.21k op/s 2024-03-20T08:58:47.362 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:47 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:58:47.363 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:47 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:58:47.363 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:47 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:58:47.368 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:47.376 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T08:58:47.427 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T08:58:47.432 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 23 16 56165 56149 152.558 107.375 0.00862036 0.00655013 2024-03-20T08:58:47.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:47 smithi122 ceph-mon[73495]: pgmap v396: 17 pgs: 17 active+clean; 2.9 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 138 MiB/s wr, 2.21k op/s 2024-03-20T08:58:47.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:47 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:58:47.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:47 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:58:47.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:47 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:58:47.466 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T08:58:47.485 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T08:58:47.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:47.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T08:58:47.509 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T08:58:47.549 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T08:58:47.590 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T08:58:47.601 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T08:58:47.619 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:47 smithi115 ceph-mon[72476]: pgmap v396: 17 pgs: 17 active+clean; 2.9 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 138 MiB/s wr, 2.21k op/s 2024-03-20T08:58:47.619 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:47 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:58:47.619 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:47 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:58:47.619 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:47 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:58:47.620 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:47.643 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T08:58:47.645 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T08:58:47.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:47.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T08:58:47.685 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T08:58:47.731 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T08:58:47.746 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T08:58:47.764 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T08:58:47.779 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:47.785 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T08:58:47.785 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T08:58:47.786 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T08:58:47.786 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T08:58:47.786 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T08:58:47.786 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T08:58:47.786 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T08:58:47.786 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T08:58:47.786 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T08:58:47.786 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T08:58:47.786 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T08:58:47.786 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T08:58:47.795 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:58:47.803 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T08:58:47.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:47.806 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T08:58:47.851 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T08:58:47.851 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T08:58:47.907 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T08:58:47.961 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:48.027 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T08:58:48.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:48.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T08:58:48.133 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T08:58:48.160 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:48.261 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T08:58:48.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:48.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T08:58:48.367 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T08:58:48.410 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:48.432 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 24 15 57607 57592 149.959 90.1875 0.0114854 0.00666423 2024-03-20T08:58:48.436 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T08:58:48.440 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:48 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1727790361' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T08:58:48.440 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:48 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:58:48.440 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:48 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:58:48.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:48.441 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T08:58:48.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:48 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1727790361' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T08:58:48.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:48 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:58:48.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:48 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:58:48.541 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T08:58:48.583 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:48.607 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T08:58:48.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:48.611 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T08:58:48.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:48 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1727790361' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T08:58:48.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:48 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:58:48.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:48 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:58:48.711 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T08:58:48.774 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:48.916 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T08:58:48.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:48.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T08:58:48.955 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-20T08:58:49.022 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T08:58:49.054 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:49.082 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T08:58:49.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:49.085 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T08:58:49.186 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T08:58:49.231 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:49.263 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T08:58:49.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:49.266 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T08:58:49.367 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T08:58:49.401 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:49.427 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T08:58:49.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:49.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T08:58:49.432 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 25 15 59401 59386 148.446 112.125 0.00947575 0.00673263 2024-03-20T08:58:49.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:49 smithi122 ceph-mon[73495]: pgmap v397: 17 pgs: 17 active+clean; 3.2 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 136 MiB/s wr, 2.18k op/s 2024-03-20T08:58:49.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:49 smithi122 ceph-mon[73495]: osdmap e132: 12 total, 12 up, 12 in 2024-03-20T08:58:49.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:49 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:58:49.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:49 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:58:49.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:49 smithi026 ceph-mon[68309]: pgmap v397: 17 pgs: 17 active+clean; 3.2 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 136 MiB/s wr, 2.18k op/s 2024-03-20T08:58:49.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:49 smithi026 ceph-mon[68309]: osdmap e132: 12 total, 12 up, 12 in 2024-03-20T08:58:49.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:49 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:58:49.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:49 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:58:49.532 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T08:58:49.571 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:49.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:49 smithi115 ceph-mon[72476]: pgmap v397: 17 pgs: 17 active+clean; 3.2 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 136 MiB/s wr, 2.18k op/s 2024-03-20T08:58:49.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:49 smithi115 ceph-mon[72476]: osdmap e132: 12 total, 12 up, 12 in 2024-03-20T08:58:49.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:49 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:58:49.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:49 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:58:49.830 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T08:58:49.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:49.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T08:58:49.947 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T08:58:49.983 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:50.010 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T08:58:50.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:50.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T08:58:50.114 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T08:58:50.160 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:50.258 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T08:58:50.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:50.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T08:58:50.319 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:58:50.362 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T08:58:50.389 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:50 smithi115 ceph-mon[72476]: osdmap e133: 12 total, 12 up, 12 in 2024-03-20T08:58:50.391 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:50.418 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T08:58:50.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:50.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T08:58:50.434 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 26 16 60992 60976 146.548 99.375 0.00991296 0.00681913 2024-03-20T08:58:50.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:50 smithi026 ceph-mon[68309]: osdmap e133: 12 total, 12 up, 12 in 2024-03-20T08:58:50.523 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T08:58:50.568 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:50 smithi122 ceph-mon[73495]: osdmap e133: 12 total, 12 up, 12 in 2024-03-20T08:58:50.570 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:50.598 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T08:58:50.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:50.603 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T08:58:50.704 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T08:58:50.741 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:50.763 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T08:58:50.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:50.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T08:58:50.870 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T08:58:50.945 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:51.434 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 27 16 62655 62639 144.969 103.938 0.0141038 0.00689325 2024-03-20T08:58:51.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:51 smithi026 ceph-mon[68309]: pgmap v400: 17 pgs: 17 active+clean; 3.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 108 MiB/s wr, 1.73k op/s 2024-03-20T08:58:51.593 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-03-20T08:58:51.593 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T08:58:51.594 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-20T08:58:51.594 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.0 'trim stale osdmaps' 2024-03-20T08:58:51.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:51 smithi115 ceph-mon[72476]: pgmap v400: 17 pgs: 17 active+clean; 3.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 108 MiB/s wr, 1.73k op/s 2024-03-20T08:58:51.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:51 smithi122 ceph-mon[73495]: pgmap v400: 17 pgs: 17 active+clean; 3.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 108 MiB/s wr, 1.73k op/s 2024-03-20T08:58:51.891 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T08:58:51.920 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T08:58:51.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:51.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T08:58:52.030 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T08:58:52.062 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:52.086 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T08:58:52.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:52.092 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T08:58:52.193 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T08:58:52.216 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:52 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2333795846' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T08:58:52.248 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:52.248 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T08:58:52.287 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T08:58:52.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:52.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T08:58:52.294 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:58:52.393 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T08:58:52.424 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:52 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2333795846' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T08:58:52.429 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:52.434 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 28 16 64207 64191 143.256 97 0.00815794 0.00697623 2024-03-20T08:58:52.454 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T08:58:52.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:52.457 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T08:58:52.558 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T08:58:52.583 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:52.601 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T08:58:52.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:52.604 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T08:58:52.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:52 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2333795846' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T08:58:52.705 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T08:58:52.744 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:52.768 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T08:58:52.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:52.773 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T08:58:52.791 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T08:58:52.791 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T08:58:52.874 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T08:58:52.927 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:53.019 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T08:58:53.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:53.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T08:58:53.124 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T08:58:53.185 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:53.231 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T08:58:53.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:53.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T08:58:53.337 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T08:58:53.368 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:53.391 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T08:58:53.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:53.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T08:58:53.434 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 29 15 65556 65541 141.226 84.375 0.0158942 0.00707653 2024-03-20T08:58:53.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:53 smithi026 ceph-mon[68309]: pgmap v401: 17 pgs: 17 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.56k op/s 2024-03-20T08:58:53.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:53 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T08:58:53.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:53 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2333795846' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T08:58:53.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:53 smithi026 ceph-mon[68309]: osdmap e134: 12 total, 12 up, 12 in 2024-03-20T08:58:53.495 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T08:58:53.543 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:53.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:53 smithi115 ceph-mon[72476]: pgmap v401: 17 pgs: 17 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.56k op/s 2024-03-20T08:58:53.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:53 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T08:58:53.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:53 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2333795846' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T08:58:53.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:53 smithi115 ceph-mon[72476]: osdmap e134: 12 total, 12 up, 12 in 2024-03-20T08:58:53.664 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T08:58:53.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:53.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T08:58:53.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:53 smithi122 ceph-mon[73495]: pgmap v401: 17 pgs: 17 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.56k op/s 2024-03-20T08:58:53.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:53 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T08:58:53.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:53 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2333795846' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T08:58:53.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:53 smithi122 ceph-mon[73495]: osdmap e134: 12 total, 12 up, 12 in 2024-03-20T08:58:53.704 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-20T08:58:53.775 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T08:58:53.804 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:53.829 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T08:58:53.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:53.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T08:58:53.934 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T08:58:53.969 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:53.998 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T08:58:54.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:54.004 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T08:58:54.106 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T08:58:54.138 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:54.167 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T08:58:54.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:54.170 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T08:58:54.271 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T08:58:54.298 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:54.434 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 30 16 66927 66911 139.372 85.625 0.00655031 0.00715258 2024-03-20T08:58:55.009 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:58:55.011 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T08:58:55.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:55.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T08:58:55.120 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T08:58:55.145 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:55.163 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T08:58:55.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:55.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T08:58:55.240 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:55 smithi026 ceph-mon[68309]: pgmap v403: 17 pgs: 17 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.64k op/s 2024-03-20T08:58:55.267 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T08:58:55.301 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:55.340 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T08:58:55.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:55.344 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T08:58:55.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:55 smithi115 ceph-mon[72476]: pgmap v403: 17 pgs: 17 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.64k op/s 2024-03-20T08:58:55.434 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 31 16 68328 68312 137.701 87.5625 0.00401254 0.00725758 2024-03-20T08:58:55.446 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T08:58:55.483 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:55 smithi122 ceph-mon[73495]: pgmap v403: 17 pgs: 17 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 102 MiB/s wr, 1.64k op/s 2024-03-20T08:58:55.485 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:55.512 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T08:58:55.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:55.515 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T08:58:55.616 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T08:58:55.650 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:55.668 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:58:55.673 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T08:58:55.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:55.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T08:58:55.779 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T08:58:55.804 INFO:teuthology.orchestra.run.smithi026.stderr:no valid command found; 10 closest matches: 2024-03-20T08:58:55.804 INFO:teuthology.orchestra.run.smithi026.stderr:0 2024-03-20T08:58:55.805 INFO:teuthology.orchestra.run.smithi026.stderr:1 2024-03-20T08:58:55.805 INFO:teuthology.orchestra.run.smithi026.stderr:2 2024-03-20T08:58:55.805 INFO:teuthology.orchestra.run.smithi026.stderr:abort 2024-03-20T08:58:55.805 INFO:teuthology.orchestra.run.smithi026.stderr:assert 2024-03-20T08:58:55.805 INFO:teuthology.orchestra.run.smithi026.stderr:bench [] [] [] [] 2024-03-20T08:58:55.805 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs debug_inject_read_zeros 2024-03-20T08:58:55.805 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs files list 2024-03-20T08:58:55.805 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs stats 2024-03-20T08:58:55.805 INFO:teuthology.orchestra.run.smithi026.stderr:bluestore allocator dump block 2024-03-20T08:58:55.805 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: invalid command 2024-03-20T08:58:55.856 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:56.434 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 32 16 69924 69908 136.515 99.75 0.00779376 0.00732092 2024-03-20T08:58:56.520 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T08:58:56.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:56.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T08:58:56.627 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T08:58:56.664 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:56.687 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T08:58:56.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:56.690 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T08:58:56.791 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T08:58:56.817 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:56.835 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T08:58:56.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:56.837 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T08:58:56.938 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T08:58:56.973 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:56.997 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T08:58:57.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:57.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T08:58:57.030 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T08:58:57.103 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T08:58:57.137 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:57.163 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T08:58:57.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:57.169 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T08:58:57.270 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T08:58:57.334 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:57 smithi026 ceph-mon[68309]: pgmap v404: 17 pgs: 17 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s wr, 1.68k op/s 2024-03-20T08:58:57.334 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:57 smithi026 ceph-mon[68309]: osdmap e135: 12 total, 12 up, 12 in 2024-03-20T08:58:57.337 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:57.435 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 33 16 71276 71260 134.939 84.5 0.0199149 0.00740569 2024-03-20T08:58:57.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:57 smithi122 ceph-mon[73495]: pgmap v404: 17 pgs: 17 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s wr, 1.68k op/s 2024-03-20T08:58:57.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:57 smithi122 ceph-mon[73495]: osdmap e135: 12 total, 12 up, 12 in 2024-03-20T08:58:57.602 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T08:58:57.603 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-20T08:58:57.603 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.1 'trim stale osdmaps' 2024-03-20T08:58:57.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:57 smithi115 ceph-mon[72476]: pgmap v404: 17 pgs: 17 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s wr, 1.68k op/s 2024-03-20T08:58:57.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:57 smithi115 ceph-mon[72476]: osdmap e135: 12 total, 12 up, 12 in 2024-03-20T08:58:58.325 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:58:58.428 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T08:58:58.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:58.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T08:58:58.434 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 34 15 72835 72820 133.837 97.5 0.0171289 0.0074674 2024-03-20T08:58:58.535 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T08:58:58.571 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:58.604 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T08:58:58.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:58.608 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T08:58:58.708 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T08:58:58.739 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:58.763 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T08:58:58.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:58.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T08:58:58.867 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T08:58:58.900 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:58.920 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T08:58:58.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:58.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T08:58:58.991 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T08:58:59.025 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T08:58:59.038 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T08:58:59.083 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:59.087 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T08:58:59.128 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T08:58:59.139 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T08:58:59.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:59.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T08:58:59.166 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T08:58:59.208 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T08:58:59.246 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T08:58:59.255 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T08:58:59.278 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:59.287 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T08:58:59.304 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T08:58:59.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:59.308 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T08:58:59.325 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T08:58:59.364 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T08:58:59.399 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T08:58:59.409 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T08:58:59.435 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 35 16 74289 74273 132.605 90.8125 0.0167492 0.00753668 2024-03-20T08:58:59.439 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:59.443 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T08:58:59.450 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T08:58:59.450 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T08:58:59.450 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T08:58:59.450 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T08:58:59.450 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T08:58:59.450 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T08:58:59.450 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T08:58:59.451 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T08:58:59.451 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T08:58:59.451 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T08:58:59.451 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T08:58:59.451 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T08:58:59.465 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T08:58:59.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:59.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T08:58:59.569 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T08:58:59.603 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:59 smithi115 ceph-mon[72476]: pgmap v406: 17 pgs: 17 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 103 MiB/s wr, 1.65k op/s 2024-03-20T08:58:59.603 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:58:59 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2191234512' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T08:58:59.606 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:58:59.628 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T08:58:59.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:58:59.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T08:58:59.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:59 smithi122 ceph-mon[73495]: pgmap v406: 17 pgs: 17 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 103 MiB/s wr, 1.65k op/s 2024-03-20T08:58:59.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:58:59 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2191234512' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T08:58:59.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:59 smithi026 ceph-mon[68309]: pgmap v406: 17 pgs: 17 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 103 MiB/s wr, 1.65k op/s 2024-03-20T08:58:59.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:58:59 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2191234512' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T08:58:59.732 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T08:58:59.791 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:00.028 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T08:59:00.028 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-03-20T08:59:00.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-03-20T08:59:00.028 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T08:59:00.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.028 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24550.0:32404 2.1 2:8165d8bb:::benchmark_data_smithi122_107654_object32403:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-20T08:59:00.029 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:35.524539+0000", 2024-03-20T08:59:00.029 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 24.154836466999999, 2024-03-20T08:59:00.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.058451606000000003, 2024-03-20T08:59:00.029 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:59:00.029 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:00.029 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T08:59:00.029 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24550", 2024-03-20T08:59:00.029 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:59:00.029 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32404 2024-03-20T08:59:00.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.029 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:59:00.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:59:00.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.524539+0000", 2024-03-20T08:59:00.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:59:00.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.524539+0000", 2024-03-20T08:59:00.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:59:00.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.524540+0000", 2024-03-20T08:59:00.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3900000000000003e-07 2024-03-20T08:59:00.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:59:00.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.524633+0000", 2024-03-20T08:59:00.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3690000000000006e-05 2024-03-20T08:59:00.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:59:00.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.524644+0000", 2024-03-20T08:59:00.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0865e-05 2024-03-20T08:59:00.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:59:00.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.524652+0000", 2024-03-20T08:59:00.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1790000000000002e-06 2024-03-20T08:59:00.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:59:00.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.526686+0000", 2024-03-20T08:59:00.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002034807 2024-03-20T08:59:00.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:59:00.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.526826+0000", 2024-03-20T08:59:00.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000139438 2024-03-20T08:59:00.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,3", 2024-03-20T08:59:00.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.526994+0000", 2024-03-20T08:59:00.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00016783999999999999 2024-03-20T08:59:00.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T08:59:00.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.529239+0000", 2024-03-20T08:59:00.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0022449449999999999 2024-03-20T08:59:00.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:00.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.582832+0000", 2024-03-20T08:59:00.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.053593742 2024-03-20T08:59:00.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:00.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.582879+0000", 2024-03-20T08:59:00.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6961e-05 2024-03-20T08:59:00.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:59:00.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.582943+0000", 2024-03-20T08:59:00.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4034000000000004e-05 2024-03-20T08:59:00.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:59:00.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.582990+0000", 2024-03-20T08:59:00.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7166000000000002e-05 2024-03-20T08:59:00.034 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.034 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:59:00.034 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.035 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24550.0:32411 2.1 2:808a8121:::benchmark_data_smithi122_107654_object32410:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-20T08:59:00.035 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:35.527282+0000", 2024-03-20T08:59:00.035 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 24.152092862, 2024-03-20T08:59:00.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.056012816, 2024-03-20T08:59:00.035 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:59:00.035 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:00.035 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T08:59:00.036 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24550", 2024-03-20T08:59:00.036 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:59:00.036 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32411 2024-03-20T08:59:00.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.036 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:59:00.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:59:00.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.527282+0000", 2024-03-20T08:59:00.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:59:00.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.527282+0000", 2024-03-20T08:59:00.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:59:00.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.527284+0000", 2024-03-20T08:59:00.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.085e-06 2024-03-20T08:59:00.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:59:00.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.527539+0000", 2024-03-20T08:59:00.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025563099999999998 2024-03-20T08:59:00.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:59:00.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.527551+0000", 2024-03-20T08:59:00.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2181000000000001e-05 2024-03-20T08:59:00.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:59:00.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.527557+0000", 2024-03-20T08:59:00.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1149999999999999e-06 2024-03-20T08:59:00.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:59:00.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.528307+0000", 2024-03-20T08:59:00.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00074947399999999997 2024-03-20T08:59:00.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:59:00.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.528409+0000", 2024-03-20T08:59:00.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010166300000000001 2024-03-20T08:59:00.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,3", 2024-03-20T08:59:00.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.528523+0000", 2024-03-20T08:59:00.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000114011 2024-03-20T08:59:00.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:00.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.583079+0000", 2024-03-20T08:59:00.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.054556693000000003 2024-03-20T08:59:00.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:00.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.583190+0000", 2024-03-20T08:59:00.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000110375 2024-03-20T08:59:00.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T08:59:00.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.583205+0000", 2024-03-20T08:59:00.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5342e-05 2024-03-20T08:59:00.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:59:00.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.583256+0000", 2024-03-20T08:59:00.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1103000000000002e-05 2024-03-20T08:59:00.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:59:00.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.583295+0000", 2024-03-20T08:59:00.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9143000000000001e-05 2024-03-20T08:59:00.041 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.041 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:59:00.041 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.041 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop_reply(client.24550.0:32399 2.1 e127/121 ondisk, result = 0)", 2024-03-20T08:59:00.041 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:35.528492+0000", 2024-03-20T08:59:00.041 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 24.150883133000001, 2024-03-20T08:59:00.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.054282402, 2024-03-20T08:59:00.041 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:59:00.041 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T08:59:00.041 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:59:00.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:59:00.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.528492+0000", 2024-03-20T08:59:00.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:59:00.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.528492+0000", 2024-03-20T08:59:00.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:59:00.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.528493+0000", 2024-03-20T08:59:00.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8100000000000002e-07 2024-03-20T08:59:00.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:59:00.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.528497+0000", 2024-03-20T08:59:00.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0720000000000001e-06 2024-03-20T08:59:00.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:59:00.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.528506+0000", 2024-03-20T08:59:00.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0270000000000001e-06 2024-03-20T08:59:00.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:59:00.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.528516+0000", 2024-03-20T08:59:00.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.042e-05 2024-03-20T08:59:00.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:59:00.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.529285+0000", 2024-03-20T08:59:00.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00076891800000000005 2024-03-20T08:59:00.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:59:00.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.529294+0000", 2024-03-20T08:59:00.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1839999999999999e-06 2024-03-20T08:59:00.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:59:00.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.582775+0000", 2024-03-20T08:59:00.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.053481000000000001 2024-03-20T08:59:00.045 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.045 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:59:00.045 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.045 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop_reply(client.24550.0:32404 2.1 e127/121 ondisk, result = 0)", 2024-03-20T08:59:00.045 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:35.528692+0000", 2024-03-20T08:59:00.045 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 24.150683296, 2024-03-20T08:59:00.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.054151878000000001, 2024-03-20T08:59:00.046 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:59:00.046 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T08:59:00.046 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:59:00.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:59:00.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.528692+0000", 2024-03-20T08:59:00.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:59:00.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.528692+0000", 2024-03-20T08:59:00.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:59:00.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.528693+0000", 2024-03-20T08:59:00.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1500000000000004e-07 2024-03-20T08:59:00.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:59:00.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.528697+0000", 2024-03-20T08:59:00.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0940000000000001e-06 2024-03-20T08:59:00.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:59:00.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.528706+0000", 2024-03-20T08:59:00.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4949999999999997e-06 2024-03-20T08:59:00.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:59:00.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.528713+0000", 2024-03-20T08:59:00.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9070000000000001e-06 2024-03-20T08:59:00.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:59:00.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.582818+0000", 2024-03-20T08:59:00.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.054104474 2024-03-20T08:59:00.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:59:00.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.582826+0000", 2024-03-20T08:59:00.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4610000000000008e-06 2024-03-20T08:59:00.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:59:00.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.582844+0000", 2024-03-20T08:59:00.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7832000000000001e-05 2024-03-20T08:59:00.049 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.049 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:59:00.049 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.049 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop_reply(client.24550.0:32411 2.1 e127/121 ondisk, result = 0)", 2024-03-20T08:59:00.049 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:35.529537+0000", 2024-03-20T08:59:00.050 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 24.149838509999999, 2024-03-20T08:59:00.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.053550659, 2024-03-20T08:59:00.050 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:59:00.050 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T08:59:00.050 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:59:00.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:59:00.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.529537+0000", 2024-03-20T08:59:00.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:59:00.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.529537+0000", 2024-03-20T08:59:00.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:59:00.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.529537+0000", 2024-03-20T08:59:00.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0399999999999996e-07 2024-03-20T08:59:00.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:59:00.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.529541+0000", 2024-03-20T08:59:00.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6890000000000002e-06 2024-03-20T08:59:00.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:59:00.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.582729+0000", 2024-03-20T08:59:00.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.053187917000000001 2024-03-20T08:59:00.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:59:00.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.582736+0000", 2024-03-20T08:59:00.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8199999999999999e-06 2024-03-20T08:59:00.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:59:00.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.583059+0000", 2024-03-20T08:59:00.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00032330500000000002 2024-03-20T08:59:00.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:59:00.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.583074+0000", 2024-03-20T08:59:00.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4565e-05 2024-03-20T08:59:00.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:59:00.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.583087+0000", 2024-03-20T08:59:00.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3859000000000001e-05 2024-03-20T08:59:00.053 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.053 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:59:00.053 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.053 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop_reply(client.24550.0:32404 2.1 e127/121 ondisk, result = 0)", 2024-03-20T08:59:00.053 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:35.529603+0000", 2024-03-20T08:59:00.053 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 24.149772538000001, 2024-03-20T08:59:00.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.053398744999999997, 2024-03-20T08:59:00.054 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:59:00.054 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T08:59:00.054 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:59:00.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:59:00.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.529603+0000", 2024-03-20T08:59:00.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:59:00.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.529603+0000", 2024-03-20T08:59:00.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:59:00.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.529603+0000", 2024-03-20T08:59:00.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0800000000000004e-07 2024-03-20T08:59:00.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:59:00.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.529608+0000", 2024-03-20T08:59:00.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3810000000000001e-06 2024-03-20T08:59:00.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:59:00.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.582681+0000", 2024-03-20T08:59:00.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.053073479999999999 2024-03-20T08:59:00.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:59:00.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.582690+0000", 2024-03-20T08:59:00.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0650000000000005e-06 2024-03-20T08:59:00.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:59:00.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.582868+0000", 2024-03-20T08:59:00.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000177694 2024-03-20T08:59:00.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:59:00.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.582874+0000", 2024-03-20T08:59:00.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4919999999999998e-06 2024-03-20T08:59:00.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:59:00.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.583002+0000", 2024-03-20T08:59:00.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012702500000000001 2024-03-20T08:59:00.057 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.057 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:59:00.057 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.057 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24550.0:32418 2.e 2:735df223:::benchmark_data_smithi122_107654_object32417:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-20T08:59:00.058 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:35.530243+0000", 2024-03-20T08:59:00.058 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 24.149132808000001, 2024-03-20T08:59:00.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.057540401999999997, 2024-03-20T08:59:00.058 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:59:00.058 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:00.058 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T08:59:00.058 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24550", 2024-03-20T08:59:00.058 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:59:00.058 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32418 2024-03-20T08:59:00.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.058 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:59:00.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:59:00.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.530243+0000", 2024-03-20T08:59:00.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:59:00.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.530243+0000", 2024-03-20T08:59:00.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:59:00.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.530244+0000", 2024-03-20T08:59:00.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1829999999999999e-06 2024-03-20T08:59:00.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:59:00.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.530277+0000", 2024-03-20T08:59:00.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.349e-05 2024-03-20T08:59:00.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:59:00.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.582535+0000", 2024-03-20T08:59:00.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.052257783000000002 2024-03-20T08:59:00.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:59:00.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.582548+0000", 2024-03-20T08:59:00.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3059e-05 2024-03-20T08:59:00.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:59:00.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.582617+0000", 2024-03-20T08:59:00.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9344999999999994e-05 2024-03-20T08:59:00.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:59:00.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.582759+0000", 2024-03-20T08:59:00.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014138900000000001 2024-03-20T08:59:00.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,9", 2024-03-20T08:59:00.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.582939+0000", 2024-03-20T08:59:00.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018024800000000001 2024-03-20T08:59:00.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:00.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.585465+0000", 2024-03-20T08:59:00.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002525645 2024-03-20T08:59:00.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:00.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.587464+0000", 2024-03-20T08:59:00.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019993540000000001 2024-03-20T08:59:00.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T08:59:00.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.587680+0000", 2024-03-20T08:59:00.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00021607500000000001 2024-03-20T08:59:00.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:59:00.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.587739+0000", 2024-03-20T08:59:00.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8428999999999999e-05 2024-03-20T08:59:00.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:59:00.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:35.587783+0000", 2024-03-20T08:59:00.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4401999999999999e-05 2024-03-20T08:59:00.063 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.063 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:59:00.063 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.063 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.24550.0:34621 2.3 e127/121 2:c92cdcd7:::benchmark_data_smithi122_107654_object34620:head v 127'2224, mlcod=127'2221)", 2024-03-20T08:59:00.064 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:36.556617+0000", 2024-03-20T08:59:00.064 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 23.122758654999998, 2024-03-20T08:59:00.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.038292886999999998, 2024-03-20T08:59:00.064 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:59:00.064 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:00.064 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:59:00.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:59:00.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.556617+0000", 2024-03-20T08:59:00.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:59:00.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.556617+0000", 2024-03-20T08:59:00.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:59:00.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.556617+0000", 2024-03-20T08:59:00.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8100000000000003e-07 2024-03-20T08:59:00.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:59:00.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.556676+0000", 2024-03-20T08:59:00.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9296000000000003e-05 2024-03-20T08:59:00.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:59:00.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.556685+0000", 2024-03-20T08:59:00.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1210000000000007e-06 2024-03-20T08:59:00.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:59:00.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.556690+0000", 2024-03-20T08:59:00.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9930000000000003e-06 2024-03-20T08:59:00.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:59:00.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.556721+0000", 2024-03-20T08:59:00.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0957999999999997e-05 2024-03-20T08:59:00.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:59:00.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.556732+0000", 2024-03-20T08:59:00.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1708999999999999e-05 2024-03-20T08:59:00.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:59:00.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.594861+0000", 2024-03-20T08:59:00.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.038129042000000002 2024-03-20T08:59:00.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:59:00.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:36.594910+0000", 2024-03-20T08:59:00.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8287e-05 2024-03-20T08:59:00.067 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.067 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:59:00.068 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.068 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24550.0:37431 2.1 2:8a18b49c:::benchmark_data_smithi122_107654_object37430:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-20T08:59:00.068 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:37.799589+0000", 2024-03-20T08:59:00.068 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 21.879786173999999, 2024-03-20T08:59:00.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.062254674000000003, 2024-03-20T08:59:00.068 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:59:00.068 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:00.068 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T08:59:00.068 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24550", 2024-03-20T08:59:00.068 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:59:00.068 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 37431 2024-03-20T08:59:00.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.069 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:59:00.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:59:00.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.799589+0000", 2024-03-20T08:59:00.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:59:00.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.799589+0000", 2024-03-20T08:59:00.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:59:00.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.799590+0000", 2024-03-20T08:59:00.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.113e-06 2024-03-20T08:59:00.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:59:00.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.799693+0000", 2024-03-20T08:59:00.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000102773 2024-03-20T08:59:00.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:59:00.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.799703+0000", 2024-03-20T08:59:00.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4800000000000007e-06 2024-03-20T08:59:00.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:59:00.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.799708+0000", 2024-03-20T08:59:00.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6400000000000002e-06 2024-03-20T08:59:00.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:59:00.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.799743+0000", 2024-03-20T08:59:00.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4851000000000002e-05 2024-03-20T08:59:00.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:59:00.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.799837+0000", 2024-03-20T08:59:00.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3907999999999996e-05 2024-03-20T08:59:00.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,3", 2024-03-20T08:59:00.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.799927+0000", 2024-03-20T08:59:00.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9615000000000002e-05 2024-03-20T08:59:00.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T08:59:00.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.801041+0000", 2024-03-20T08:59:00.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001114865 2024-03-20T08:59:00.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:00.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.801194+0000", 2024-03-20T08:59:00.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000152258 2024-03-20T08:59:00.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:00.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.801371+0000", 2024-03-20T08:59:00.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00017718599999999999 2024-03-20T08:59:00.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:59:00.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.861780+0000", 2024-03-20T08:59:00.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.060409000999999997 2024-03-20T08:59:00.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:59:00.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.861844+0000", 2024-03-20T08:59:00.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3984000000000003e-05 2024-03-20T08:59:00.073 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.074 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:59:00.074 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.074 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop_reply(client.24550.0:37431 2.1 e127/121 ondisk, result = 0)", 2024-03-20T08:59:00.074 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:37.801314+0000", 2024-03-20T08:59:00.074 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 21.878060868999999, 2024-03-20T08:59:00.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.060537321999999998, 2024-03-20T08:59:00.074 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:59:00.074 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T08:59:00.074 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:59:00.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:59:00.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.801314+0000", 2024-03-20T08:59:00.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:59:00.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.801314+0000", 2024-03-20T08:59:00.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:59:00.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.801315+0000", 2024-03-20T08:59:00.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.6800000000000009e-07 2024-03-20T08:59:00.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:59:00.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.801319+0000", 2024-03-20T08:59:00.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0060000000000003e-06 2024-03-20T08:59:00.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:59:00.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.801327+0000", 2024-03-20T08:59:00.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.072e-06 2024-03-20T08:59:00.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:59:00.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.801332+0000", 2024-03-20T08:59:00.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7809999999999996e-06 2024-03-20T08:59:00.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:59:00.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.801363+0000", 2024-03-20T08:59:00.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.095e-05 2024-03-20T08:59:00.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:59:00.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.801368+0000", 2024-03-20T08:59:00.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5460000000000002e-06 2024-03-20T08:59:00.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:59:00.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.861852+0000", 2024-03-20T08:59:00.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.060483998999999997 2024-03-20T08:59:00.077 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.077 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:59:00.077 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.078 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.24550.0:37437 2.3 e127/121 2:cbed25d6:::benchmark_data_smithi122_107654_object37436:head v 127'2408, mlcod=127'2406)", 2024-03-20T08:59:00.078 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:37.801802+0000", 2024-03-20T08:59:00.078 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 21.877573532, 2024-03-20T08:59:00.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.065170981000000003, 2024-03-20T08:59:00.078 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:59:00.078 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:00.078 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:59:00.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:59:00.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.801802+0000", 2024-03-20T08:59:00.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:59:00.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.801802+0000", 2024-03-20T08:59:00.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:59:00.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.801802+0000", 2024-03-20T08:59:00.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3300000000000002e-07 2024-03-20T08:59:00.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:59:00.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.801906+0000", 2024-03-20T08:59:00.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010330199999999999 2024-03-20T08:59:00.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:59:00.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.861782+0000", 2024-03-20T08:59:00.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.059875976999999997 2024-03-20T08:59:00.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:59:00.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.861791+0000", 2024-03-20T08:59:00.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.6050000000000004e-06 2024-03-20T08:59:00.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:59:00.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.861858+0000", 2024-03-20T08:59:00.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7088000000000005e-05 2024-03-20T08:59:00.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:59:00.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.861876+0000", 2024-03-20T08:59:00.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7855000000000001e-05 2024-03-20T08:59:00.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:59:00.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.866933+0000", 2024-03-20T08:59:00.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.005056779 2024-03-20T08:59:00.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:59:00.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.866973+0000", 2024-03-20T08:59:00.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9742e-05 2024-03-20T08:59:00.082 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.082 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:59:00.082 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.082 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24550.0:37443 2.e 2:76cca025:::benchmark_data_smithi122_107654_object37442:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-20T08:59:00.082 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:37.804637+0000", 2024-03-20T08:59:00.082 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 21.874738346000001, 2024-03-20T08:59:00.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.062772670000000003, 2024-03-20T08:59:00.082 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:59:00.082 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:00.082 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T08:59:00.082 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24550", 2024-03-20T08:59:00.082 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:59:00.082 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 37443 2024-03-20T08:59:00.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.083 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:59:00.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:59:00.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.804637+0000", 2024-03-20T08:59:00.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:59:00.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.804637+0000", 2024-03-20T08:59:00.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:59:00.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.804639+0000", 2024-03-20T08:59:00.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2639999999999998e-06 2024-03-20T08:59:00.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:59:00.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.804689+0000", 2024-03-20T08:59:00.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0126000000000001e-05 2024-03-20T08:59:00.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:59:00.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.861786+0000", 2024-03-20T08:59:00.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.057096656000000003 2024-03-20T08:59:00.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:59:00.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.861807+0000", 2024-03-20T08:59:00.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1044999999999999e-05 2024-03-20T08:59:00.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:59:00.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.861899+0000", 2024-03-20T08:59:00.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1927000000000003e-05 2024-03-20T08:59:00.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:59:00.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.862043+0000", 2024-03-20T08:59:00.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014389700000000001 2024-03-20T08:59:00.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,9", 2024-03-20T08:59:00.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.862196+0000", 2024-03-20T08:59:00.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000153115 2024-03-20T08:59:00.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:00.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.864272+0000", 2024-03-20T08:59:00.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0020763919999999998 2024-03-20T08:59:00.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T08:59:00.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.866991+0000", 2024-03-20T08:59:00.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027182230000000001 2024-03-20T08:59:00.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.087 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:00.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.867295+0000", 2024-03-20T08:59:00.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00030427900000000002 2024-03-20T08:59:00.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.087 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:59:00.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.867363+0000", 2024-03-20T08:59:00.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8124e-05 2024-03-20T08:59:00.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.087 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:59:00.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:37.867410+0000", 2024-03-20T08:59:00.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6622000000000002e-05 2024-03-20T08:59:00.088 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.088 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:59:00.088 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.088 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24550.0:41678 2.1 2:834ac045:::benchmark_data_smithi122_107654_object41677:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-20T08:59:00.088 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:39.728818+0000", 2024-03-20T08:59:00.088 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 19.950557453999998, 2024-03-20T08:59:00.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.064460940999999994, 2024-03-20T08:59:00.088 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:59:00.088 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:00.088 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T08:59:00.089 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24550", 2024-03-20T08:59:00.089 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:59:00.089 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 41678 2024-03-20T08:59:00.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.089 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:59:00.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:59:00.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.728818+0000", 2024-03-20T08:59:00.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:59:00.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.728818+0000", 2024-03-20T08:59:00.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:59:00.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.728818+0000", 2024-03-20T08:59:00.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7300000000000001e-07 2024-03-20T08:59:00.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:59:00.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.728943+0000", 2024-03-20T08:59:00.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012449999999999999 2024-03-20T08:59:00.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:59:00.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.728952+0000", 2024-03-20T08:59:00.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9400000000000008e-06 2024-03-20T08:59:00.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:59:00.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.728958+0000", 2024-03-20T08:59:00.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3999999999999997e-06 2024-03-20T08:59:00.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:59:00.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.730497+0000", 2024-03-20T08:59:00.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015389659999999999 2024-03-20T08:59:00.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:59:00.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.730616+0000", 2024-03-20T08:59:00.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000118915 2024-03-20T08:59:00.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,3", 2024-03-20T08:59:00.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.730748+0000", 2024-03-20T08:59:00.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000131963 2024-03-20T08:59:00.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:00.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.733535+0000", 2024-03-20T08:59:00.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002786752 2024-03-20T08:59:00.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T08:59:00.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.734973+0000", 2024-03-20T08:59:00.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001438148 2024-03-20T08:59:00.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:00.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.793171+0000", 2024-03-20T08:59:00.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.058197725999999998 2024-03-20T08:59:00.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:59:00.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.793231+0000", 2024-03-20T08:59:00.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0041999999999999e-05 2024-03-20T08:59:00.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:59:00.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.793279+0000", 2024-03-20T08:59:00.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8115999999999998e-05 2024-03-20T08:59:00.095 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.095 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:59:00.095 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.096 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24550.0:41681 2.1 2:86d5686d:::benchmark_data_smithi122_107654_object41680:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-20T08:59:00.096 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:39.728977+0000", 2024-03-20T08:59:00.096 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 19.950397896999998, 2024-03-20T08:59:00.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.064431140999999997, 2024-03-20T08:59:00.096 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:59:00.096 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:00.096 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T08:59:00.096 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24550", 2024-03-20T08:59:00.096 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:59:00.096 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 41681 2024-03-20T08:59:00.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.096 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:59:00.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:59:00.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.728977+0000", 2024-03-20T08:59:00.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:59:00.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.728977+0000", 2024-03-20T08:59:00.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:59:00.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.728978+0000", 2024-03-20T08:59:00.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5600000000000005e-07 2024-03-20T08:59:00.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:59:00.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.729008+0000", 2024-03-20T08:59:00.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9595e-05 2024-03-20T08:59:00.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:59:00.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.729016+0000", 2024-03-20T08:59:00.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5129999999999996e-06 2024-03-20T08:59:00.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:59:00.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.729023+0000", 2024-03-20T08:59:00.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8249999999999999e-06 2024-03-20T08:59:00.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:59:00.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.731973+0000", 2024-03-20T08:59:00.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0029503509999999999 2024-03-20T08:59:00.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:59:00.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.732094+0000", 2024-03-20T08:59:00.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012082999999999999 2024-03-20T08:59:00.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,3", 2024-03-20T08:59:00.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.732183+0000", 2024-03-20T08:59:00.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8808999999999996e-05 2024-03-20T08:59:00.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:00.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.735008+0000", 2024-03-20T08:59:00.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028245169999999999 2024-03-20T08:59:00.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T08:59:00.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.735018+0000", 2024-03-20T08:59:00.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0505e-05 2024-03-20T08:59:00.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:00.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.793322+0000", 2024-03-20T08:59:00.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.058304360999999999 2024-03-20T08:59:00.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:59:00.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.793371+0000", 2024-03-20T08:59:00.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8711000000000002e-05 2024-03-20T08:59:00.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:59:00.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.793409+0000", 2024-03-20T08:59:00.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7468000000000001e-05 2024-03-20T08:59:00.101 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.101 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:59:00.101 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.102 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24550.0:41682 2.1 2:8ee5a06e:::benchmark_data_smithi122_107654_object41681:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-20T08:59:00.102 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:39.729051+0000", 2024-03-20T08:59:00.102 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 19.950324337000001, 2024-03-20T08:59:00.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.067596247999999998, 2024-03-20T08:59:00.102 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:59:00.102 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:00.102 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T08:59:00.102 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24550", 2024-03-20T08:59:00.102 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:59:00.102 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 41682 2024-03-20T08:59:00.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.102 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:59:00.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:59:00.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.729051+0000", 2024-03-20T08:59:00.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:59:00.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.729051+0000", 2024-03-20T08:59:00.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:59:00.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.729052+0000", 2024-03-20T08:59:00.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.51e-07 2024-03-20T08:59:00.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:59:00.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.729082+0000", 2024-03-20T08:59:00.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0480999999999998e-05 2024-03-20T08:59:00.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:59:00.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.729090+0000", 2024-03-20T08:59:00.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2290000000000003e-06 2024-03-20T08:59:00.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:59:00.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.729096+0000", 2024-03-20T08:59:00.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5439999999999998e-06 2024-03-20T08:59:00.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:59:00.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.733568+0000", 2024-03-20T08:59:00.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00447175 2024-03-20T08:59:00.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:59:00.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.733723+0000", 2024-03-20T08:59:00.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015544800000000001 2024-03-20T08:59:00.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,3", 2024-03-20T08:59:00.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.733860+0000", 2024-03-20T08:59:00.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013741400000000001 2024-03-20T08:59:00.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:00.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.792825+0000", 2024-03-20T08:59:00.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.058964851999999998 2024-03-20T08:59:00.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:00.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.796482+0000", 2024-03-20T08:59:00.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036568579999999998 2024-03-20T08:59:00.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T08:59:00.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.796543+0000", 2024-03-20T08:59:00.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0572000000000002e-05 2024-03-20T08:59:00.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:59:00.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.796599+0000", 2024-03-20T08:59:00.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6336999999999999e-05 2024-03-20T08:59:00.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:59:00.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.796647+0000", 2024-03-20T08:59:00.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8211999999999999e-05 2024-03-20T08:59:00.107 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.107 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:59:00.107 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.108 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.24550.0:41691 2.3 e128/121 2:c4088840:::benchmark_data_smithi122_107654_object41690:head v 128'2704, mlcod=128'2702)", 2024-03-20T08:59:00.108 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:39.733137+0000", 2024-03-20T08:59:00.108 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 19.946238480000002, 2024-03-20T08:59:00.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.057316843999999999, 2024-03-20T08:59:00.108 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:59:00.108 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:00.108 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:59:00.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:59:00.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.733137+0000", 2024-03-20T08:59:00.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:59:00.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.733137+0000", 2024-03-20T08:59:00.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:59:00.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.733138+0000", 2024-03-20T08:59:00.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2799999999999995e-07 2024-03-20T08:59:00.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:59:00.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.733215+0000", 2024-03-20T08:59:00.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7038000000000003e-05 2024-03-20T08:59:00.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:59:00.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.733222+0000", 2024-03-20T08:59:00.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1940000000000001e-06 2024-03-20T08:59:00.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:59:00.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.733227+0000", 2024-03-20T08:59:00.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4550000000000003e-06 2024-03-20T08:59:00.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:59:00.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.733277+0000", 2024-03-20T08:59:00.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9381e-05 2024-03-20T08:59:00.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:59:00.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.733292+0000", 2024-03-20T08:59:00.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5387999999999999e-05 2024-03-20T08:59:00.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:59:00.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.735867+0000", 2024-03-20T08:59:00.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0025750370000000001 2024-03-20T08:59:00.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:59:00.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.790454+0000", 2024-03-20T08:59:00.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.054586623000000001 2024-03-20T08:59:00.112 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.112 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:59:00.112 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.112 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24550.0:41694 2.e 2:7c0d0568:::benchmark_data_smithi122_107654_object41693:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-20T08:59:00.112 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:39.733243+0000", 2024-03-20T08:59:00.112 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 19.946132154000001, 2024-03-20T08:59:00.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.064416000000000001, 2024-03-20T08:59:00.112 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:59:00.112 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:00.112 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T08:59:00.112 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24550", 2024-03-20T08:59:00.112 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:59:00.113 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 41694 2024-03-20T08:59:00.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.113 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:59:00.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:59:00.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.733243+0000", 2024-03-20T08:59:00.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:59:00.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.733243+0000", 2024-03-20T08:59:00.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:59:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.733245+0000", 2024-03-20T08:59:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3999999999999999e-06 2024-03-20T08:59:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:59:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.733276+0000", 2024-03-20T08:59:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1652999999999997e-05 2024-03-20T08:59:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:59:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.733286+0000", 2024-03-20T08:59:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0163000000000001e-05 2024-03-20T08:59:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:59:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.733293+0000", 2024-03-20T08:59:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8530000000000003e-06 2024-03-20T08:59:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:59:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.733345+0000", 2024-03-20T08:59:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2197999999999997e-05 2024-03-20T08:59:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:59:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.733468+0000", 2024-03-20T08:59:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012234799999999999 2024-03-20T08:59:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,9", 2024-03-20T08:59:00.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.733608+0000", 2024-03-20T08:59:00.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000139942 2024-03-20T08:59:00.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T08:59:00.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.792017+0000", 2024-03-20T08:59:00.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.058409335 2024-03-20T08:59:00.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:00.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.792658+0000", 2024-03-20T08:59:00.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000641064 2024-03-20T08:59:00.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:00.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.797565+0000", 2024-03-20T08:59:00.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049066149999999996 2024-03-20T08:59:00.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:59:00.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.797620+0000", 2024-03-20T08:59:00.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5557999999999999e-05 2024-03-20T08:59:00.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:59:00.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.797659+0000", 2024-03-20T08:59:00.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8871000000000001e-05 2024-03-20T08:59:00.117 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.118 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:59:00.118 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.118 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.24550.0:41692 2.7 e128/121 2:e8fd97ed:::benchmark_data_smithi122_107654_object41691:head v 128'2607, mlcod=128'2605)", 2024-03-20T08:59:00.118 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:39.735041+0000", 2024-03-20T08:59:00.118 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 19.944334610999999, 2024-03-20T08:59:00.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.058596809999999999, 2024-03-20T08:59:00.118 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:59:00.118 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:00.118 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:59:00.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:59:00.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.735041+0000", 2024-03-20T08:59:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:59:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.735041+0000", 2024-03-20T08:59:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:59:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.735041+0000", 2024-03-20T08:59:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2500000000000005e-07 2024-03-20T08:59:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:59:00.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.735078+0000", 2024-03-20T08:59:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6674000000000002e-05 2024-03-20T08:59:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:59:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.790501+0000", 2024-03-20T08:59:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.055423166000000003 2024-03-20T08:59:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:59:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.790513+0000", 2024-03-20T08:59:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.17e-05 2024-03-20T08:59:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:59:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.790703+0000", 2024-03-20T08:59:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000190173 2024-03-20T08:59:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:59:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.790726+0000", 2024-03-20T08:59:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2608000000000001e-05 2024-03-20T08:59:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T08:59:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.793587+0000", 2024-03-20T08:59:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002861611 2024-03-20T08:59:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:59:00.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.793638+0000", 2024-03-20T08:59:00.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0152999999999999e-05 2024-03-20T08:59:00.122 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.122 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:59:00.122 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.122 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop_reply(client.24550.0:41678 2.1 e128/121 ondisk, result = 0)", 2024-03-20T08:59:00.122 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:39.735834+0000", 2024-03-20T08:59:00.122 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 19.943541538000002, 2024-03-20T08:59:00.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.057453356999999997, 2024-03-20T08:59:00.122 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:59:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T08:59:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:59:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:59:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.735834+0000", 2024-03-20T08:59:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:59:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.735834+0000", 2024-03-20T08:59:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:59:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.735835+0000", 2024-03-20T08:59:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8000000000000005e-07 2024-03-20T08:59:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:59:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.735840+0000", 2024-03-20T08:59:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1669999999999998e-06 2024-03-20T08:59:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:59:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.791343+0000", 2024-03-20T08:59:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.055503003000000002 2024-03-20T08:59:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:59:00.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.791353+0000", 2024-03-20T08:59:00.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0192e-05 2024-03-20T08:59:00.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:59:00.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.793157+0000", 2024-03-20T08:59:00.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001804422 2024-03-20T08:59:00.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:59:00.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.793165+0000", 2024-03-20T08:59:00.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7819999999999994e-06 2024-03-20T08:59:00.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:59:00.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.793287+0000", 2024-03-20T08:59:00.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012201099999999999 2024-03-20T08:59:00.126 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.126 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:59:00.126 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.126 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop_reply(client.24550.0:41682 2.1 e128/121 ondisk, result = 0)", 2024-03-20T08:59:00.126 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:58:39.736161+0000", 2024-03-20T08:59:00.126 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 19.943214307000002, 2024-03-20T08:59:00.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.056671484000000001, 2024-03-20T08:59:00.126 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:59:00.126 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T08:59:00.127 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:59:00.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:59:00.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.736161+0000", 2024-03-20T08:59:00.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:59:00.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.736161+0000", 2024-03-20T08:59:00.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:00.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:59:00.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.736162+0000", 2024-03-20T08:59:00.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5300000000000004e-07 2024-03-20T08:59:00.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:59:00.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.736166+0000", 2024-03-20T08:59:00.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3390000000000003e-06 2024-03-20T08:59:00.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:59:00.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.790580+0000", 2024-03-20T08:59:00.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.054413907999999997 2024-03-20T08:59:00.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:59:00.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.790589+0000", 2024-03-20T08:59:00.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0690000000000008e-06 2024-03-20T08:59:00.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:59:00.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.792812+0000", 2024-03-20T08:59:00.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0022232020000000001 2024-03-20T08:59:00.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:59:00.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.792819+0000", 2024-03-20T08:59:00.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2559999999999996e-06 2024-03-20T08:59:00.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:00.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:00.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T08:59:00.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:58:39.792832+0000", 2024-03-20T08:59:00.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3057000000000001e-05 2024-03-20T08:59:00.130 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.130 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:59:00.130 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.130 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:00.130 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:59:00.130 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T08:59:00.435 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 36 16 75873 75857 131.672 99 0.0117398 0.0075907 2024-03-20T08:59:00.863 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T08:59:00.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:00.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T08:59:00.899 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-20T08:59:00.972 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T08:59:00.999 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:01.020 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T08:59:01.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:01.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T08:59:01.125 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T08:59:01.172 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:01.192 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T08:59:01.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:01.196 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T08:59:01.297 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T08:59:01.343 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:01.379 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T08:59:01.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:01.385 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T08:59:01.436 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 37 16 77341 77325 130.593 91.75 0.0127683 0.00765394 2024-03-20T08:59:01.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:01 smithi122 ceph-mon[73495]: pgmap v407: 17 pgs: 17 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.54k op/s 2024-03-20T08:59:01.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:01 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:59:01.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:01 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:59:01.486 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T08:59:01.517 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:01.521 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:59:01.539 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T08:59:01.543 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:01 smithi115 ceph-mon[72476]: pgmap v407: 17 pgs: 17 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.54k op/s 2024-03-20T08:59:01.543 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:01 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:59:01.543 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:01 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:59:01.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:01.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T08:59:01.632 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:01 smithi026 ceph-mon[68309]: pgmap v407: 17 pgs: 17 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.54k op/s 2024-03-20T08:59:01.632 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:01 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:59:01.632 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:01 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:59:01.644 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T08:59:01.700 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:01.965 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T08:59:01.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:01.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T08:59:02.072 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T08:59:02.129 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:02.243 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T08:59:02.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:02.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T08:59:02.350 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T08:59:02.384 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:02.406 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T08:59:02.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:02.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T08:59:02.436 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 38 16 78789 78773 129.537 90.5 0.00615191 0.0077156 2024-03-20T08:59:02.510 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T08:59:02.538 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:02.557 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T08:59:02.559 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:02 smithi115 ceph-mon[72476]: osdmap e136: 12 total, 12 up, 12 in 2024-03-20T08:59:02.559 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:02 smithi115 ceph-mon[72476]: osdmap e137: 12 total, 12 up, 12 in 2024-03-20T08:59:02.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:02.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T08:59:02.597 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:02 smithi026 ceph-mon[68309]: osdmap e136: 12 total, 12 up, 12 in 2024-03-20T08:59:02.597 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:02 smithi026 ceph-mon[68309]: osdmap e137: 12 total, 12 up, 12 in 2024-03-20T08:59:02.661 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T08:59:02.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:02 smithi122 ceph-mon[73495]: osdmap e136: 12 total, 12 up, 12 in 2024-03-20T08:59:02.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:02 smithi122 ceph-mon[73495]: osdmap e137: 12 total, 12 up, 12 in 2024-03-20T08:59:02.697 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:59:02.728 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:03.332 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T08:59:03.436 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 39 16 80207 80191 128.488 88.625 0.00305643 0.00777756 2024-03-20T08:59:03.593 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:03 smithi026 ceph-mon[68309]: pgmap v409: 17 pgs: 17 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-03-20T08:59:03.593 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:03 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/728323523' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T08:59:03.593 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:03 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T08:59:03.619 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T08:59:03.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:03.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T08:59:03.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:03 smithi115 ceph-mon[72476]: pgmap v409: 17 pgs: 17 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-03-20T08:59:03.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:03 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/728323523' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T08:59:03.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:03 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T08:59:03.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:03 smithi122 ceph-mon[73495]: pgmap v409: 17 pgs: 17 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-03-20T08:59:03.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:03 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/728323523' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T08:59:03.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:03 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T08:59:03.728 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T08:59:03.761 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:03.784 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T08:59:03.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:03.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T08:59:03.888 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T08:59:03.919 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:03.946 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T08:59:03.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:03.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T08:59:03.987 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T08:59:04.050 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T08:59:04.082 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:04.109 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T08:59:04.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:04.112 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T08:59:04.213 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T08:59:04.291 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:04.436 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:2024-03-20T08:59:04.436715+0000 min lat: 0.00103856 max lat: 0.219093 avg lat: 0.00784478 2024-03-20T08:59:04.436 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T08:59:04.436 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 40 16 81569 81553 127.404 85.125 0.0178796 0.00784478 2024-03-20T08:59:04.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:04 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T08:59:04.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:04 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T08:59:04.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:04 smithi122 ceph-mon[73495]: osdmap e138: 12 total, 12 up, 12 in 2024-03-20T08:59:04.499 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T08:59:04.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:04.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T08:59:04.612 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T08:59:04.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:04 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T08:59:04.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:04 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T08:59:04.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:04 smithi115 ceph-mon[72476]: osdmap e138: 12 total, 12 up, 12 in 2024-03-20T08:59:04.671 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:04 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T08:59:04.672 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:04 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T08:59:04.672 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:04 smithi026 ceph-mon[68309]: osdmap e138: 12 total, 12 up, 12 in 2024-03-20T08:59:04.674 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:04.896 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T08:59:04.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:04.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T08:59:05.016 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T08:59:05.065 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:05.096 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T08:59:05.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:05.100 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T08:59:05.201 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T08:59:05.232 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:05.263 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T08:59:05.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:05.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T08:59:05.368 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T08:59:05.424 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:05.436 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 41 16 82827 82811 126.214 78.625 0.00768522 0.00791835 2024-03-20T08:59:05.563 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T08:59:05.566 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:05 smithi026 ceph-mon[68309]: pgmap v412: 17 pgs: 17 active+clean; 4.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s 2024-03-20T08:59:05.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:05.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T08:59:05.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:05 smithi115 ceph-mon[72476]: pgmap v412: 17 pgs: 17 active+clean; 4.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s 2024-03-20T08:59:05.684 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T08:59:05.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:05 smithi122 ceph-mon[73495]: pgmap v412: 17 pgs: 17 active+clean; 4.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 92 MiB/s wr, 1.47k op/s 2024-03-20T08:59:05.723 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:05.753 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T08:59:05.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:05.757 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T08:59:05.858 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T08:59:05.889 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:05.913 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T08:59:05.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:05.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T08:59:06.018 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T08:59:06.055 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:06.081 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T08:59:06.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:06.087 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T08:59:06.188 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T08:59:06.199 INFO:teuthology.orchestra.run.smithi026.stderr:no valid command found; 10 closest matches: 2024-03-20T08:59:06.200 INFO:teuthology.orchestra.run.smithi026.stderr:0 2024-03-20T08:59:06.200 INFO:teuthology.orchestra.run.smithi026.stderr:1 2024-03-20T08:59:06.200 INFO:teuthology.orchestra.run.smithi026.stderr:2 2024-03-20T08:59:06.200 INFO:teuthology.orchestra.run.smithi026.stderr:abort 2024-03-20T08:59:06.200 INFO:teuthology.orchestra.run.smithi026.stderr:assert 2024-03-20T08:59:06.200 INFO:teuthology.orchestra.run.smithi026.stderr:bench [] [] [] [] 2024-03-20T08:59:06.200 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs debug_inject_read_zeros 2024-03-20T08:59:06.200 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs files list 2024-03-20T08:59:06.200 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs stats 2024-03-20T08:59:06.200 INFO:teuthology.orchestra.run.smithi026.stderr:bluestore allocator dump block 2024-03-20T08:59:06.200 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: invalid command 2024-03-20T08:59:06.235 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T08:59:06.235 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T08:59:06.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:06.236 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.24550.0:83903 2.d e138/121 2:bf4be756:::benchmark_data_smithi122_107654_object83902:head v 138'5173, mlcod=138'5171)", 2024-03-20T08:59:06.236 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:59:06.231243+0000", 2024-03-20T08:59:06.236 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.003208841, 2024-03-20T08:59:06.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0032344349999999999, 2024-03-20T08:59:06.236 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T08:59:06.236 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T08:59:06.236 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T08:59:06.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:06.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T08:59:06.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:06.231243+0000", 2024-03-20T08:59:06.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:06.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:06.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:06.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T08:59:06.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:06.231243+0000", 2024-03-20T08:59:06.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T08:59:06.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:06.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:06.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T08:59:06.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:06.231243+0000", 2024-03-20T08:59:06.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4700000000000002e-07 2024-03-20T08:59:06.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:06.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:06.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T08:59:06.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:06.231270+0000", 2024-03-20T08:59:06.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.671e-05 2024-03-20T08:59:06.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:06.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:06.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T08:59:06.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:06.231279+0000", 2024-03-20T08:59:06.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.422e-06 2024-03-20T08:59:06.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:06.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:06.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T08:59:06.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:06.231284+0000", 2024-03-20T08:59:06.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5280000000000003e-06 2024-03-20T08:59:06.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:06.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:06.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T08:59:06.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:06.232535+0000", 2024-03-20T08:59:06.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012512210000000001 2024-03-20T08:59:06.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T08:59:06.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T08:59:06.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T08:59:06.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:06.232548+0000", 2024-03-20T08:59:06.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.294e-05 2024-03-20T08:59:06.239 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:06.239 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T08:59:06.239 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:06.239 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T08:59:06.239 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T08:59:06.239 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 1 2024-03-20T08:59:06.239 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T08:59:06.278 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:06.437 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 42 16 84091 84075 125.09 79 0.00284001 0.00797646 2024-03-20T08:59:06.631 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:59:06.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:06.704 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T08:59:06.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T08:59:06.805 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T08:59:06.846 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:06.875 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T08:59:06.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:06.878 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T08:59:06.979 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T08:59:07.016 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:07.051 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T08:59:07.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:07.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T08:59:07.155 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T08:59:07.194 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:07.220 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T08:59:07.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:07.224 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T08:59:07.324 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T08:59:07.358 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:07.388 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T08:59:07.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:07.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T08:59:07.436 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 43 16 85352 85336 124.014 78.8125 0.0175984 0.00805928 2024-03-20T08:59:07.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:07 smithi122 ceph-mon[73495]: pgmap v413: 17 pgs: 17 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-03-20T08:59:07.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:07 smithi122 ceph-mon[73495]: osdmap e139: 12 total, 12 up, 12 in 2024-03-20T08:59:07.494 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T08:59:07.535 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:07.612 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T08:59:07.637 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:07 smithi026 ceph-mon[68309]: pgmap v413: 17 pgs: 17 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-03-20T08:59:07.638 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:07 smithi026 ceph-mon[68309]: osdmap e139: 12 total, 12 up, 12 in 2024-03-20T08:59:07.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:07 smithi115 ceph-mon[72476]: pgmap v413: 17 pgs: 17 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-03-20T08:59:07.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:07 smithi115 ceph-mon[72476]: osdmap e139: 12 total, 12 up, 12 in 2024-03-20T08:59:08.436 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 44 16 86524 86508 122.86 73.25 0.00498988 0.00813533 2024-03-20T08:59:08.592 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T08:59:08.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:08.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T08:59:08.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:08 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:59:08.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:08 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:59:08.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:08 smithi115 ceph-mon[72476]: osdmap e140: 12 total, 12 up, 12 in 2024-03-20T08:59:08.660 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:08 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:59:08.660 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:08 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:59:08.660 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:08 smithi026 ceph-mon[68309]: osdmap e140: 12 total, 12 up, 12 in 2024-03-20T08:59:08.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:08 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:59:08.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:08 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:59:08.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:08 smithi122 ceph-mon[73495]: osdmap e140: 12 total, 12 up, 12 in 2024-03-20T08:59:08.698 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T08:59:08.747 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:09.330 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T08:59:09.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:09.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T08:59:09.377 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-20T08:59:09.436 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 45 16 87964 87948 122.13 90 0.0192499 0.00818406 2024-03-20T08:59:09.445 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T08:59:09.477 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:09.510 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T08:59:09.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:09.512 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T08:59:09.613 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T08:59:09.626 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:09 smithi026 ceph-mon[68309]: pgmap v415: 17 pgs: 17 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-03-20T08:59:09.653 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:09 smithi122 ceph-mon[73495]: pgmap v415: 17 pgs: 17 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-03-20T08:59:09.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:09 smithi115 ceph-mon[72476]: pgmap v415: 17 pgs: 17 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-03-20T08:59:09.659 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:09.686 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T08:59:09.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:09.692 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T08:59:09.794 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T08:59:09.823 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:09.846 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T08:59:09.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:09.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T08:59:09.950 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T08:59:09.979 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:10.002 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T08:59:10.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:10.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T08:59:10.108 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T08:59:10.162 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:10.403 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:59:10.427 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T08:59:10.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:10.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T08:59:10.436 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 46 16 89305 89289 121.297 83.8125 0.0187103 0.00824017 2024-03-20T08:59:10.461 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T08:59:10.462 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-20T08:59:10.462 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.2 'trim stale osdmaps' 2024-03-20T08:59:10.537 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T08:59:10.575 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:10.674 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T08:59:10.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:10.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T08:59:10.788 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T08:59:10.821 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:10.847 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T08:59:10.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:10.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T08:59:10.953 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T08:59:11.024 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:11.128 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T08:59:11.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:11.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T08:59:11.244 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T08:59:11.296 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:11.437 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 47 16 90400 90384 120.17 68.4375 0.00942734 0.00831744 2024-03-20T08:59:11.462 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T08:59:11.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:11.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T08:59:11.567 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T08:59:11.612 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:11 smithi026 ceph-mon[68309]: pgmap v417: 17 pgs: 17 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.24k op/s 2024-03-20T08:59:11.620 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:11.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:11 smithi115 ceph-mon[72476]: pgmap v417: 17 pgs: 17 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.24k op/s 2024-03-20T08:59:11.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:11 smithi122 ceph-mon[73495]: pgmap v417: 17 pgs: 17 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.24k op/s 2024-03-20T08:59:11.913 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:59:12.241 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T08:59:12.308 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T08:59:12.354 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T08:59:12.401 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T08:59:12.437 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 48 16 91702 91686 119.362 81.375 0.0128516 0.0083728 2024-03-20T08:59:12.448 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T08:59:12.491 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T08:59:12.528 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T08:59:12.571 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T08:59:12.606 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T08:59:12.651 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T08:59:12.651 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:12 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/988687908' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T08:59:12.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:12 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/988687908' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T08:59:12.685 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T08:59:12.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:12 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/988687908' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T08:59:12.738 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T08:59:12.746 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T08:59:12.746 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T08:59:12.746 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T08:59:12.746 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T08:59:12.746 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T08:59:12.746 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T08:59:12.746 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T08:59:12.746 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T08:59:12.746 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T08:59:12.746 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T08:59:12.746 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T08:59:12.747 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T08:59:13.104 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T08:59:13.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:13.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T08:59:13.210 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T08:59:13.265 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:13.412 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T08:59:13.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:13.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T08:59:13.437 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 49 16 93003 92987 118.586 81.3125 0.0193997 0.00842851 2024-03-20T08:59:13.520 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T08:59:13.533 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:13 smithi026 ceph-mon[68309]: pgmap v418: 17 pgs: 17 active+clean; 5.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-03-20T08:59:13.573 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:13.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:13 smithi115 ceph-mon[72476]: pgmap v418: 17 pgs: 17 active+clean; 5.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-03-20T08:59:13.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:13 smithi122 ceph-mon[73495]: pgmap v418: 17 pgs: 17 active+clean; 5.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-03-20T08:59:13.962 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T08:59:13.962 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T08:59:14.438 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 50 16 94386 94370 117.942 86.4375 0.00796161 0.00847434 2024-03-20T08:59:14.597 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-20T08:59:14.640 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T08:59:14.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:14.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T08:59:14.751 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T08:59:14.781 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:14.802 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T08:59:14.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:14.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T08:59:14.907 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T08:59:15.000 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:15.076 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T08:59:15.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:15.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T08:59:15.180 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T08:59:15.228 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:15.259 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T08:59:15.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:15.264 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T08:59:15.365 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T08:59:15.394 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:15.419 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T08:59:15.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:15.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T08:59:15.438 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 51 16 95650 95634 117.179 79 0.0185996 0.00852967 2024-03-20T08:59:15.523 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T08:59:15.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:15 smithi122 ceph-mon[73495]: pgmap v419: 17 pgs: 17 active+clean; 5.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-03-20T08:59:15.561 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:15.588 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T08:59:15.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:15.592 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T08:59:15.652 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:59:15.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:15 smithi115 ceph-mon[72476]: pgmap v419: 17 pgs: 17 active+clean; 5.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-03-20T08:59:15.693 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T08:59:15.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:15 smithi026 ceph-mon[68309]: pgmap v419: 17 pgs: 17 active+clean; 5.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-03-20T08:59:15.752 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:15.926 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T08:59:15.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:15.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T08:59:15.975 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:59:16.030 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T08:59:16.090 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:16.420 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:16 smithi026 ceph-mon[68309]: pgmap v420: 17 pgs: 17 active+clean; 5.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.24k op/s 2024-03-20T08:59:16.420 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:16 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2498032830' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T08:59:16.420 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:16 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T08:59:16.437 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 52 16 96860 96844 116.38 75.625 0.0134712 0.00858818 2024-03-20T08:59:16.459 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T08:59:16.699 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:16 smithi122 ceph-mon[73495]: pgmap v420: 17 pgs: 17 active+clean; 5.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.24k op/s 2024-03-20T08:59:16.700 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:16 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2498032830' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T08:59:16.700 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:16 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T08:59:16.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:16 smithi115 ceph-mon[72476]: pgmap v420: 17 pgs: 17 active+clean; 5.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.24k op/s 2024-03-20T08:59:16.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:16 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2498032830' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T08:59:16.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:16 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T08:59:17.126 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T08:59:17.437 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 53 16 98112 98096 115.66 78.25 0.00750077 0.0086417 2024-03-20T08:59:17.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:17 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T08:59:17.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:17 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T08:59:17.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:17 smithi026 ceph-mon[68309]: osdmap e141: 12 total, 12 up, 12 in 2024-03-20T08:59:17.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:17 smithi026 ceph-mon[68309]: osdmap e142: 12 total, 12 up, 12 in 2024-03-20T08:59:17.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:17 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T08:59:17.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:17 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T08:59:17.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:17 smithi122 ceph-mon[73495]: osdmap e141: 12 total, 12 up, 12 in 2024-03-20T08:59:17.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:17 smithi122 ceph-mon[73495]: osdmap e142: 12 total, 12 up, 12 in 2024-03-20T08:59:17.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:17 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T08:59:17.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:17 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T08:59:17.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:17 smithi115 ceph-mon[72476]: osdmap e141: 12 total, 12 up, 12 in 2024-03-20T08:59:17.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:17 smithi115 ceph-mon[72476]: osdmap e142: 12 total, 12 up, 12 in 2024-03-20T08:59:18.364 INFO:teuthology.orchestra.run.smithi026.stderr:no valid command found; 10 closest matches: 2024-03-20T08:59:18.364 INFO:teuthology.orchestra.run.smithi026.stderr:0 2024-03-20T08:59:18.364 INFO:teuthology.orchestra.run.smithi026.stderr:1 2024-03-20T08:59:18.364 INFO:teuthology.orchestra.run.smithi026.stderr:2 2024-03-20T08:59:18.364 INFO:teuthology.orchestra.run.smithi026.stderr:abort 2024-03-20T08:59:18.364 INFO:teuthology.orchestra.run.smithi026.stderr:assert 2024-03-20T08:59:18.364 INFO:teuthology.orchestra.run.smithi026.stderr:bench [] [] [] [] 2024-03-20T08:59:18.364 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs debug_inject_read_zeros 2024-03-20T08:59:18.365 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs files list 2024-03-20T08:59:18.365 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs stats 2024-03-20T08:59:18.365 INFO:teuthology.orchestra.run.smithi026.stderr:bluestore allocator dump block 2024-03-20T08:59:18.365 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: invalid command 2024-03-20T08:59:18.438 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 54 16 99197 99181 114.774 67.8125 0.00494244 0.00870781 2024-03-20T08:59:18.674 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:18 smithi026 ceph-mon[68309]: pgmap v423: 17 pgs: 17 active+clean; 5.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.35k op/s 2024-03-20T08:59:18.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:18 smithi122 ceph-mon[73495]: pgmap v423: 17 pgs: 17 active+clean; 5.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.35k op/s 2024-03-20T08:59:18.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:18 smithi115 ceph-mon[72476]: pgmap v423: 17 pgs: 17 active+clean; 5.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.35k op/s 2024-03-20T08:59:19.121 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T08:59:19.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:19.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T08:59:19.228 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T08:59:19.266 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:19.298 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T08:59:19.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:19.301 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T08:59:19.402 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T08:59:19.438 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 55 16 100494 100478 114.161 81.0625 0.029127 0.0087553 2024-03-20T08:59:19.480 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:19.572 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T08:59:19.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:19.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T08:59:19.676 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:19 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:59:19.676 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:19 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:59:19.676 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T08:59:19.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:19 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:59:19.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:19 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:59:19.714 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:19.738 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T08:59:19.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:19.743 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T08:59:19.844 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T08:59:19.897 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:19.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:19 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:59:19.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:19 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:59:20.020 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T08:59:20.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:20.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T08:59:20.073 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T08:59:20.073 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-20T08:59:20.074 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.3 'trim stale osdmaps' 2024-03-20T08:59:20.138 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T08:59:20.226 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:20.318 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T08:59:20.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:20.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T08:59:20.409 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:59:20.424 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T08:59:20.434 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T08:59:20.434 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T08:59:20.437 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 56 16 101678 101662 113.444 74 0.0229897 0.00880918 2024-03-20T08:59:20.452 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:20.473 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T08:59:20.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:20.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T08:59:20.578 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T08:59:20.615 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:20.639 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T08:59:20.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:20.642 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T08:59:20.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:20 smithi026 ceph-mon[68309]: osdmap e143: 12 total, 12 up, 12 in 2024-03-20T08:59:20.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:20 smithi026 ceph-mon[68309]: pgmap v425: 17 pgs: 17 active+clean; 5.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-03-20T08:59:20.743 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T08:59:20.799 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:20.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:20 smithi115 ceph-mon[72476]: osdmap e143: 12 total, 12 up, 12 in 2024-03-20T08:59:20.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:20 smithi115 ceph-mon[72476]: pgmap v425: 17 pgs: 17 active+clean; 5.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-03-20T08:59:20.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:20 smithi122 ceph-mon[73495]: osdmap e143: 12 total, 12 up, 12 in 2024-03-20T08:59:20.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:20 smithi122 ceph-mon[73495]: pgmap v425: 17 pgs: 17 active+clean; 5.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-03-20T08:59:20.977 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T08:59:20.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:20.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T08:59:21.085 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T08:59:21.120 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:21.122 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T08:59:21.141 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T08:59:21.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:21.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T08:59:21.245 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T08:59:21.273 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:21.297 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T08:59:21.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:21.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T08:59:21.402 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T08:59:21.430 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:21.438 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 57 16 102763 102747 112.643 67.8125 0.00639094 0.00887284 2024-03-20T08:59:21.452 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T08:59:21.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:21.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T08:59:21.461 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-20T08:59:21.556 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T08:59:21.589 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:21.612 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T08:59:21.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:21.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T08:59:21.719 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T08:59:21.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:21 smithi026 ceph-mon[68309]: 2.1 scrub starts 2024-03-20T08:59:21.752 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:21 smithi115 ceph-mon[72476]: 2.1 scrub starts 2024-03-20T08:59:21.755 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:21.787 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T08:59:21.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:21.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T08:59:21.841 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:59:21.893 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T08:59:21.921 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:21.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:21 smithi122 ceph-mon[73495]: 2.1 scrub starts 2024-03-20T08:59:22.121 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T08:59:22.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:22.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T08:59:22.227 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T08:59:22.261 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:22.282 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T08:59:22.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:22.289 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T08:59:22.390 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T08:59:22.406 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:59:22.421 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:22.438 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 58 16 103798 103782 111.817 64.6875 0.00790926 0.00893859 2024-03-20T08:59:22.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:22 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:59:22.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:22 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:59:22.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:22 smithi115 ceph-mon[72476]: pgmap v426: 17 pgs: 17 active+clean; 6.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-03-20T08:59:22.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:22 smithi115 ceph-mon[72476]: osdmap e144: 12 total, 12 up, 12 in 2024-03-20T08:59:22.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:22 smithi115 ceph-mon[72476]: 2.e scrub starts 2024-03-20T08:59:22.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:22 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:59:22.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:22 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:59:22.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:22 smithi122 ceph-mon[73495]: pgmap v426: 17 pgs: 17 active+clean; 6.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-03-20T08:59:22.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:22 smithi122 ceph-mon[73495]: osdmap e144: 12 total, 12 up, 12 in 2024-03-20T08:59:22.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:22 smithi122 ceph-mon[73495]: 2.e scrub starts 2024-03-20T08:59:22.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:22 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:59:22.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:22 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:59:22.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:22 smithi026 ceph-mon[68309]: pgmap v426: 17 pgs: 17 active+clean; 6.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-03-20T08:59:22.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:22 smithi026 ceph-mon[68309]: osdmap e144: 12 total, 12 up, 12 in 2024-03-20T08:59:22.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:22 smithi026 ceph-mon[68309]: 2.e scrub starts 2024-03-20T08:59:23.271 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T08:59:23.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:23.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T08:59:23.376 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T08:59:23.413 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:23.438 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 59 16 104883 104867 111.071 67.8125 0.0106354 0.00899967 2024-03-20T08:59:23.445 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T08:59:23.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:23.451 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T08:59:23.552 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T08:59:23.607 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:23.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:23 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:59:23.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:23 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:59:23.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:23 smithi122 ceph-mon[73495]: osdmap e145: 12 total, 12 up, 12 in 2024-03-20T08:59:23.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:23 smithi122 ceph-mon[73495]: 2.1 scrub ok 2024-03-20T08:59:23.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:23 smithi122 ceph-mon[73495]: from='client.24616 172.21.15.26:0/3029972361' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T08:59:23.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:23 smithi122 ceph-mon[73495]: from='client.24616 172.21.15.26:0/3029972361' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T08:59:23.862 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:23 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:59:23.862 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:23 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:59:23.862 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:23 smithi026 ceph-mon[68309]: osdmap e145: 12 total, 12 up, 12 in 2024-03-20T08:59:23.862 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:23 smithi026 ceph-mon[68309]: 2.1 scrub ok 2024-03-20T08:59:23.862 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:23 smithi026 ceph-mon[68309]: from='client.24616 172.21.15.26:0/3029972361' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T08:59:23.863 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:23 smithi026 ceph-mon[68309]: from='client.24616 172.21.15.26:0/3029972361' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T08:59:23.896 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T08:59:23.897 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T08:59:23.897 INFO:teuthology.orchestra.run.smithi115.stdout: "num_ops": 0 2024-03-20T08:59:23.897 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T08:59:23.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:23 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:59:23.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:23 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:59:23.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:23 smithi115 ceph-mon[72476]: osdmap e145: 12 total, 12 up, 12 in 2024-03-20T08:59:23.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:23 smithi115 ceph-mon[72476]: 2.1 scrub ok 2024-03-20T08:59:23.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:23 smithi115 ceph-mon[72476]: from='client.24616 172.21.15.26:0/3029972361' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T08:59:23.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:23 smithi115 ceph-mon[72476]: from='client.24616 172.21.15.26:0/3029972361' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T08:59:23.982 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T08:59:23.982 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T08:59:23.982 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T08:59:23.982 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T08:59:23.982 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T08:59:23.982 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T08:59:23.983 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T08:59:23.983 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T08:59:23.983 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T08:59:23.983 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T08:59:23.983 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T08:59:23.983 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T08:59:23.983 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T08:59:23.983 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T08:59:23.983 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T08:59:23.983 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T08:59:23.983 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T08:59:23.983 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T08:59:23.983 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T08:59:23.984 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T08:59:23.984 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T08:59:23.984 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T08:59:23.984 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T08:59:23.984 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T08:59:24.190 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T08:59:24.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:24.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T08:59:24.300 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T08:59:24.354 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:24.438 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:2024-03-20T08:59:24.438987+0000 min lat: 0.00103856 max lat: 0.241539 avg lat: 0.00905475 2024-03-20T08:59:24.438 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T08:59:24.438 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 60 16 106007 105991 110.39 70.25 0.00731702 0.00905475 2024-03-20T08:59:24.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:24 smithi115 ceph-mon[72476]: pgmap v429: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-03-20T08:59:24.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:24 smithi115 ceph-mon[72476]: 2.e scrub ok 2024-03-20T08:59:24.839 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-20T08:59:24.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:24 smithi122 ceph-mon[73495]: pgmap v429: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-03-20T08:59:24.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:24 smithi122 ceph-mon[73495]: 2.e scrub ok 2024-03-20T08:59:24.947 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:24 smithi026 ceph-mon[68309]: pgmap v429: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-03-20T08:59:24.947 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:24 smithi026 ceph-mon[68309]: 2.e scrub ok 2024-03-20T08:59:25.020 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T08:59:25.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:25.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T08:59:25.126 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T08:59:25.181 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:25.206 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T08:59:25.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:25.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T08:59:25.247 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:59:25.312 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T08:59:25.382 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:25.438 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 61 16 107209 107193 109.812 75.125 0.0138798 0.00910292 2024-03-20T08:59:25.449 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T08:59:25.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:25.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T08:59:25.555 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T08:59:25.562 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:25 smithi115 ceph-mon[72476]: 2.3 scrub starts 2024-03-20T08:59:25.562 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:25 smithi026 ceph-mon[68309]: 2.3 scrub starts 2024-03-20T08:59:25.601 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:59:25.632 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:25.669 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T08:59:25.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:25.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T08:59:25.717 INFO:teuthology.orchestra.run.smithi026.stderr:no valid command found; 10 closest matches: 2024-03-20T08:59:25.717 INFO:teuthology.orchestra.run.smithi026.stderr:0 2024-03-20T08:59:25.717 INFO:teuthology.orchestra.run.smithi026.stderr:1 2024-03-20T08:59:25.717 INFO:teuthology.orchestra.run.smithi026.stderr:2 2024-03-20T08:59:25.717 INFO:teuthology.orchestra.run.smithi026.stderr:abort 2024-03-20T08:59:25.717 INFO:teuthology.orchestra.run.smithi026.stderr:assert 2024-03-20T08:59:25.717 INFO:teuthology.orchestra.run.smithi026.stderr:bench [] [] [] [] 2024-03-20T08:59:25.717 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs debug_inject_read_zeros 2024-03-20T08:59:25.717 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs files list 2024-03-20T08:59:25.718 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs stats 2024-03-20T08:59:25.718 INFO:teuthology.orchestra.run.smithi026.stderr:bluestore allocator dump block 2024-03-20T08:59:25.718 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: invalid command 2024-03-20T08:59:25.783 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T08:59:25.865 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:25.950 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:25 smithi122 ceph-mon[73495]: 2.3 scrub starts 2024-03-20T08:59:25.951 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T08:59:25.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:25.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T08:59:26.058 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T08:59:26.088 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:26.197 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T08:59:26.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:26.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T08:59:26.300 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T08:59:26.342 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:26.438 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 62 16 108297 108281 109.138 68 0.00380332 0.00914032 2024-03-20T08:59:26.584 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T08:59:26.585 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-20T08:59:26.585 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.4 'trim stale osdmaps' 2024-03-20T08:59:26.603 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T08:59:26.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:26.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T08:59:26.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:26 smithi115 ceph-mon[72476]: 2.3 scrub ok 2024-03-20T08:59:26.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:26 smithi115 ceph-mon[72476]: pgmap v430: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-03-20T08:59:26.707 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T08:59:26.734 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:26.808 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T08:59:26.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:26.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T08:59:26.912 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T08:59:26.939 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:26 smithi026 ceph-mon[68309]: 2.3 scrub ok 2024-03-20T08:59:26.939 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:26 smithi026 ceph-mon[68309]: pgmap v430: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-03-20T08:59:26.941 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:26.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:26 smithi122 ceph-mon[73495]: 2.3 scrub ok 2024-03-20T08:59:26.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:26 smithi122 ceph-mon[73495]: pgmap v430: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-03-20T08:59:26.991 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T08:59:27.040 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T08:59:27.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:27.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T08:59:27.145 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T08:59:27.176 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:27.376 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T08:59:27.376 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T08:59:27.376 INFO:teuthology.orchestra.run.smithi115.stdout: "complaint_time": 30, 2024-03-20T08:59:27.377 INFO:teuthology.orchestra.run.smithi115.stdout: "num_blocked_ops": 0 2024-03-20T08:59:27.377 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T08:59:27.438 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 63 16 108304 108288 107.412 0.4375 0.00341595 0.00913997 2024-03-20T08:59:27.518 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:59:27.533 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T08:59:27.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:27.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T08:59:27.539 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:59:27.639 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T08:59:27.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:27 smithi115 ceph-mon[72476]: 2.5 scrub starts 2024-03-20T08:59:27.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:27 smithi115 ceph-mon[72476]: osdmap e146: 12 total, 12 up, 12 in 2024-03-20T08:59:27.672 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:27.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:27 smithi026 ceph-mon[68309]: 2.5 scrub starts 2024-03-20T08:59:27.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:27 smithi026 ceph-mon[68309]: osdmap e146: 12 total, 12 up, 12 in 2024-03-20T08:59:27.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:27 smithi122 ceph-mon[73495]: 2.5 scrub starts 2024-03-20T08:59:27.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:27 smithi122 ceph-mon[73495]: osdmap e146: 12 total, 12 up, 12 in 2024-03-20T08:59:28.174 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-20T08:59:28.221 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T08:59:28.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:28.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T08:59:28.325 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T08:59:28.354 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:28.377 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T08:59:28.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:28.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T08:59:28.438 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 64 16 108304 108288 105.734 0 - 0.00913997 2024-03-20T08:59:28.481 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T08:59:28.509 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:28.531 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T08:59:28.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:28.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T08:59:28.535 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:59:28.636 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T08:59:28.662 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:28.681 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T08:59:28.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:28.684 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T08:59:28.702 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:28 smithi026 ceph-mon[68309]: 2.5 scrub ok 2024-03-20T08:59:28.702 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:28 smithi026 ceph-mon[68309]: 2.9 scrub starts 2024-03-20T08:59:28.702 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:28 smithi026 ceph-mon[68309]: 2.b scrub starts 2024-03-20T08:59:28.702 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:28 smithi026 ceph-mon[68309]: 2.0 scrub starts 2024-03-20T08:59:28.702 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:28 smithi026 ceph-mon[68309]: pgmap v432: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-03-20T08:59:28.702 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:28 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:59:28.702 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:28 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:59:28.702 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:28 smithi026 ceph-mon[68309]: 2.d scrub starts 2024-03-20T08:59:28.702 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:28 smithi026 ceph-mon[68309]: 2.0 scrub ok 2024-03-20T08:59:28.784 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T08:59:28.795 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:28 smithi115 ceph-mon[72476]: 2.5 scrub ok 2024-03-20T08:59:28.795 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:28 smithi115 ceph-mon[72476]: 2.9 scrub starts 2024-03-20T08:59:28.795 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:28 smithi115 ceph-mon[72476]: 2.b scrub starts 2024-03-20T08:59:28.795 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:28 smithi115 ceph-mon[72476]: 2.0 scrub starts 2024-03-20T08:59:28.795 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:28 smithi115 ceph-mon[72476]: pgmap v432: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-03-20T08:59:28.795 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:28 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:59:28.795 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:28 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:59:28.795 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:28 smithi115 ceph-mon[72476]: 2.d scrub starts 2024-03-20T08:59:28.795 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:28 smithi115 ceph-mon[72476]: 2.0 scrub ok 2024-03-20T08:59:28.815 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:28.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:28 smithi122 ceph-mon[73495]: 2.5 scrub ok 2024-03-20T08:59:28.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:28 smithi122 ceph-mon[73495]: 2.9 scrub starts 2024-03-20T08:59:28.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:28 smithi122 ceph-mon[73495]: 2.b scrub starts 2024-03-20T08:59:28.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:28 smithi122 ceph-mon[73495]: 2.0 scrub starts 2024-03-20T08:59:28.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:28 smithi122 ceph-mon[73495]: pgmap v432: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-03-20T08:59:28.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:28 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:59:28.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:28 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:59:28.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:28 smithi122 ceph-mon[73495]: 2.d scrub starts 2024-03-20T08:59:28.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:28 smithi122 ceph-mon[73495]: 2.0 scrub ok 2024-03-20T08:59:29.366 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T08:59:29.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:29.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T08:59:29.379 INFO:teuthology.orchestra.run.smithi026.stderr:no valid command found; 10 closest matches: 2024-03-20T08:59:29.379 INFO:teuthology.orchestra.run.smithi026.stderr:0 2024-03-20T08:59:29.379 INFO:teuthology.orchestra.run.smithi026.stderr:1 2024-03-20T08:59:29.379 INFO:teuthology.orchestra.run.smithi026.stderr:2 2024-03-20T08:59:29.379 INFO:teuthology.orchestra.run.smithi026.stderr:abort 2024-03-20T08:59:29.379 INFO:teuthology.orchestra.run.smithi026.stderr:assert 2024-03-20T08:59:29.379 INFO:teuthology.orchestra.run.smithi026.stderr:bench [] [] [] [] 2024-03-20T08:59:29.379 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs debug_inject_read_zeros 2024-03-20T08:59:29.380 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs files list 2024-03-20T08:59:29.380 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs stats 2024-03-20T08:59:29.380 INFO:teuthology.orchestra.run.smithi026.stderr:bluestore allocator dump block 2024-03-20T08:59:29.380 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: invalid command 2024-03-20T08:59:29.439 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 65 16 108304 108288 104.107 0 - 0.00913997 2024-03-20T08:59:29.470 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T08:59:29.502 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:29.523 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T08:59:29.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:29.526 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T08:59:29.627 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T08:59:29.656 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:29 smithi026 ceph-mon[68309]: 2.9 scrub ok 2024-03-20T08:59:29.656 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:29 smithi026 ceph-mon[68309]: 2.b scrub ok 2024-03-20T08:59:29.657 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:29 smithi026 ceph-mon[68309]: 2.8 scrub starts 2024-03-20T08:59:29.657 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:29 smithi026 ceph-mon[68309]: 2.d scrub ok 2024-03-20T08:59:29.657 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:29 smithi026 ceph-mon[68309]: osdmap e147: 12 total, 12 up, 12 in 2024-03-20T08:59:29.657 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:29 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3928035093' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T08:59:29.659 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:29.696 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T08:59:29.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:29 smithi115 ceph-mon[72476]: 2.9 scrub ok 2024-03-20T08:59:29.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:29 smithi115 ceph-mon[72476]: 2.b scrub ok 2024-03-20T08:59:29.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:29 smithi115 ceph-mon[72476]: 2.8 scrub starts 2024-03-20T08:59:29.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:29 smithi115 ceph-mon[72476]: 2.d scrub ok 2024-03-20T08:59:29.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:29 smithi115 ceph-mon[72476]: osdmap e147: 12 total, 12 up, 12 in 2024-03-20T08:59:29.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:29 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3928035093' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T08:59:29.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:29 smithi122 ceph-mon[73495]: 2.9 scrub ok 2024-03-20T08:59:29.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:29 smithi122 ceph-mon[73495]: 2.b scrub ok 2024-03-20T08:59:29.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:29 smithi122 ceph-mon[73495]: 2.8 scrub starts 2024-03-20T08:59:29.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:29 smithi122 ceph-mon[73495]: 2.d scrub ok 2024-03-20T08:59:29.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:29 smithi122 ceph-mon[73495]: osdmap e147: 12 total, 12 up, 12 in 2024-03-20T08:59:29.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:29 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3928035093' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T08:59:30.439 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 66 16 108304 108288 102.53 0 - 0.00913997 2024-03-20T08:59:30.542 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T08:59:30.542 INFO:teuthology.orchestra.run.smithi115.stdout: "size": 20, 2024-03-20T08:59:30.542 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 600, 2024-03-20T08:59:30.542 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [ 2024-03-20T08:59:30.542 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.542 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:84191 2.d e138/121 2:bced3af9:::benchmark_data_smithi122_107654_object84190:head v 138'5194, mlcod=138'5189)", 2024-03-20T08:59:30.542 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:59:06.620812+0000", 2024-03-20T08:59:30.543 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 23.919320505000002, 2024-03-20T08:59:30.543 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.010564163, 2024-03-20T08:59:30.543 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:30.543 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:30.543 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:30.543 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.543 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:30.543 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:06.620812+0000", 2024-03-20T08:59:30.543 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.543 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.543 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.543 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:30.543 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:06.620812+0000", 2024-03-20T08:59:30.544 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.544 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.544 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.544 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:30.544 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:06.620813+0000", 2024-03-20T08:59:30.544 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.1500000000000004e-07 2024-03-20T08:59:30.544 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.544 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.544 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:30.544 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:06.620910+0000", 2024-03-20T08:59:30.544 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.7456999999999999e-05 2024-03-20T08:59:30.544 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.544 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.544 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:30.545 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:06.620921+0000", 2024-03-20T08:59:30.545 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0336000000000001e-05 2024-03-20T08:59:30.545 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.545 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.545 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:30.545 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:06.620926+0000", 2024-03-20T08:59:30.545 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0810000000000001e-06 2024-03-20T08:59:30.545 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.545 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.545 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:30.545 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:06.620965+0000", 2024-03-20T08:59:30.545 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.8821e-05 2024-03-20T08:59:30.545 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.545 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.546 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:30.546 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:06.620984+0000", 2024-03-20T08:59:30.546 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9589000000000002e-05 2024-03-20T08:59:30.546 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.546 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.546 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:30.546 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:06.631326+0000", 2024-03-20T08:59:30.546 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.01034213 2024-03-20T08:59:30.546 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.546 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.546 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:30.546 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:06.631376+0000", 2024-03-20T08:59:30.546 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0034000000000002e-05 2024-03-20T08:59:30.546 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.547 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:30.547 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.547 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.547 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.547 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:105403 2.f e145/121 2:f8c8f9eb:::benchmark_data_smithi122_107654_object105402:head v 145'6490, mlcod=145'6488)", 2024-03-20T08:59:30.547 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:59:23.943250+0000", 2024-03-20T08:59:30.547 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 6.5968826260000002, 2024-03-20T08:59:30.547 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.012536729999999999, 2024-03-20T08:59:30.547 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:30.547 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:30.547 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:30.547 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.547 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:30.548 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:23.943250+0000", 2024-03-20T08:59:30.548 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.548 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.548 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.548 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:30.548 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:23.943250+0000", 2024-03-20T08:59:30.548 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.548 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.548 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.548 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:30.548 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:23.943251+0000", 2024-03-20T08:59:30.548 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.6400000000000001e-07 2024-03-20T08:59:30.548 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.548 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.549 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:30.549 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:23.943310+0000", 2024-03-20T08:59:30.549 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8631000000000002e-05 2024-03-20T08:59:30.549 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.549 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.549 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:30.549 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:23.943321+0000", 2024-03-20T08:59:30.549 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1157e-05 2024-03-20T08:59:30.549 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.549 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.549 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:30.549 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:23.943328+0000", 2024-03-20T08:59:30.549 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.0559999999999999e-06 2024-03-20T08:59:30.550 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.550 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.550 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:30.550 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:23.943390+0000", 2024-03-20T08:59:30.550 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2713999999999996e-05 2024-03-20T08:59:30.550 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.550 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.551 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:30.551 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:23.943410+0000", 2024-03-20T08:59:30.551 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9758e-05 2024-03-20T08:59:30.551 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.551 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.551 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:30.551 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:23.955731+0000", 2024-03-20T08:59:30.551 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.012321212 2024-03-20T08:59:30.551 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.551 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.551 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:30.551 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:23.955787+0000", 2024-03-20T08:59:30.551 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5438e-05 2024-03-20T08:59:30.551 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.552 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:30.552 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.552 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.552 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.552 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:105738 2.f e145/121 2:f7729223:::benchmark_data_smithi122_107654_object105737:head v 145'6510, mlcod=145'6507)", 2024-03-20T08:59:30.552 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:59:24.211226+0000", 2024-03-20T08:59:30.552 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 6.3289064340000003, 2024-03-20T08:59:30.552 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0098432109999999993, 2024-03-20T08:59:30.552 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:30.552 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:30.552 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:30.552 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.552 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:30.552 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.211226+0000", 2024-03-20T08:59:30.553 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.553 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.553 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.553 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:30.553 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.211226+0000", 2024-03-20T08:59:30.553 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.553 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.553 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.553 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:30.553 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.211228+0000", 2024-03-20T08:59:30.553 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4169999999999999e-06 2024-03-20T08:59:30.553 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.553 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.553 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:30.553 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.211281+0000", 2024-03-20T08:59:30.554 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3499999999999999e-05 2024-03-20T08:59:30.554 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.554 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.554 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:30.554 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.211298+0000", 2024-03-20T08:59:30.554 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6399999999999999e-05 2024-03-20T08:59:30.554 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.554 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.554 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:30.554 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.211308+0000", 2024-03-20T08:59:30.554 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0020000000000001e-05 2024-03-20T08:59:30.554 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.554 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.554 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:30.555 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.212707+0000", 2024-03-20T08:59:30.555 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001399066 2024-03-20T08:59:30.555 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.555 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.555 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:30.555 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.212730+0000", 2024-03-20T08:59:30.555 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3787000000000001e-05 2024-03-20T08:59:30.555 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.555 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.555 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:30.555 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.221020+0000", 2024-03-20T08:59:30.555 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0082893510000000004 2024-03-20T08:59:30.555 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.555 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.555 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:30.556 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.221070+0000", 2024-03-20T08:59:30.556 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9669999999999997e-05 2024-03-20T08:59:30.556 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.556 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:30.556 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.556 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.556 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.556 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:106340 2.d e145/121 2:b2260be5:::benchmark_data_smithi122_107654_object106339:head v 145'6565, mlcod=145'6562)", 2024-03-20T08:59:30.556 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:59:24.713759+0000", 2024-03-20T08:59:30.556 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 5.8263739409999999, 2024-03-20T08:59:30.556 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.012848297, 2024-03-20T08:59:30.556 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:30.556 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:30.556 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:30.557 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.557 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:30.557 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.713759+0000", 2024-03-20T08:59:30.557 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.557 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.557 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.557 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:30.557 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.713759+0000", 2024-03-20T08:59:30.557 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.557 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.557 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.557 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:30.557 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.713760+0000", 2024-03-20T08:59:30.557 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7400000000000002e-07 2024-03-20T08:59:30.558 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.558 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.558 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:30.558 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.713841+0000", 2024-03-20T08:59:30.558 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.0981000000000003e-05 2024-03-20T08:59:30.558 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.558 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.558 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:30.558 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.713854+0000", 2024-03-20T08:59:30.558 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3078000000000001e-05 2024-03-20T08:59:30.558 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.558 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.558 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:30.558 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.713860+0000", 2024-03-20T08:59:30.559 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.5180000000000001e-06 2024-03-20T08:59:30.559 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.559 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.559 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:30.559 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.713918+0000", 2024-03-20T08:59:30.559 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7556000000000003e-05 2024-03-20T08:59:30.559 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.559 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.559 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:30.559 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.713935+0000", 2024-03-20T08:59:30.559 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7756000000000001e-05 2024-03-20T08:59:30.559 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.559 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.559 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:30.560 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.726549+0000", 2024-03-20T08:59:30.560 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.012613167 2024-03-20T08:59:30.560 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.560 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.560 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:30.560 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.726607+0000", 2024-03-20T08:59:30.560 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8467000000000001e-05 2024-03-20T08:59:30.560 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.561 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:30.561 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.561 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.561 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.561 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:106459 2.5 e145/121 2:aca5f4a2:::benchmark_data_smithi122_107654_object106458:head v 145'6611, mlcod=145'6609)", 2024-03-20T08:59:30.561 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:59:24.807142+0000", 2024-03-20T08:59:30.561 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 5.7329912329999999, 2024-03-20T08:59:30.561 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.012967677, 2024-03-20T08:59:30.561 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:30.561 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:30.561 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:30.561 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.561 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:30.561 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.807142+0000", 2024-03-20T08:59:30.562 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.562 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.562 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.562 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:30.562 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.807142+0000", 2024-03-20T08:59:30.562 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.562 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.562 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.562 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:30.562 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.807143+0000", 2024-03-20T08:59:30.562 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0580000000000001e-06 2024-03-20T08:59:30.562 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.563 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.563 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:30.563 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.807237+0000", 2024-03-20T08:59:30.563 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.4016000000000002e-05 2024-03-20T08:59:30.563 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.563 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.563 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:30.563 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.807255+0000", 2024-03-20T08:59:30.563 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8249000000000001e-05 2024-03-20T08:59:30.563 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.563 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.563 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:30.563 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.807266+0000", 2024-03-20T08:59:30.564 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1382999999999999e-05 2024-03-20T08:59:30.564 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.564 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.564 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:30.564 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.807338+0000", 2024-03-20T08:59:30.564 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.1675999999999996e-05 2024-03-20T08:59:30.564 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.564 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.564 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:30.564 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.807363+0000", 2024-03-20T08:59:30.564 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5089e-05 2024-03-20T08:59:30.564 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.564 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.564 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:30.565 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.820070+0000", 2024-03-20T08:59:30.565 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.012706886000000001 2024-03-20T08:59:30.565 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.565 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.565 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:30.565 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.820109+0000", 2024-03-20T08:59:30.565 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.9320000000000003e-05 2024-03-20T08:59:30.565 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.565 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:30.565 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.565 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.565 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.565 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:106460 2.5 e145/121 2:a0090339:::benchmark_data_smithi122_107654_object106459:head v 145'6612, mlcod=145'6609)", 2024-03-20T08:59:30.565 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:59:24.808061+0000", 2024-03-20T08:59:30.566 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 5.7320717720000003, 2024-03-20T08:59:30.566 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.012093225000000001, 2024-03-20T08:59:30.566 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:30.566 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:30.566 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:30.566 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.566 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:30.566 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.808061+0000", 2024-03-20T08:59:30.566 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.566 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.566 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.566 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:30.566 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.808061+0000", 2024-03-20T08:59:30.567 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.567 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.567 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.567 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:30.567 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.808062+0000", 2024-03-20T08:59:30.567 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4199999999999996e-07 2024-03-20T08:59:30.567 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.567 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.567 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:30.567 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.808136+0000", 2024-03-20T08:59:30.567 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.4807000000000004e-05 2024-03-20T08:59:30.567 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.567 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.567 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:30.567 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.808150+0000", 2024-03-20T08:59:30.568 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3730999999999999e-05 2024-03-20T08:59:30.568 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.568 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.568 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:30.568 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.808157+0000", 2024-03-20T08:59:30.568 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.5189999999999998e-06 2024-03-20T08:59:30.568 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.568 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.568 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:30.568 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.813551+0000", 2024-03-20T08:59:30.568 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0053945190000000004 2024-03-20T08:59:30.568 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.568 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.568 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:30.569 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.813572+0000", 2024-03-20T08:59:30.569 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0581000000000002e-05 2024-03-20T08:59:30.569 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.569 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.569 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:30.569 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.820134+0000", 2024-03-20T08:59:30.569 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0065622379999999998 2024-03-20T08:59:30.569 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.569 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.569 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:30.569 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.820154+0000", 2024-03-20T08:59:30.569 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0288e-05 2024-03-20T08:59:30.569 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.569 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:30.570 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.570 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.570 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.570 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:106461 2.d e145/121 2:b559f2ce:::benchmark_data_smithi122_107654_object106460:head v 145'6575, mlcod=145'6573)", 2024-03-20T08:59:30.570 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:59:24.808201+0000", 2024-03-20T08:59:30.570 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 5.7319313149999997, 2024-03-20T08:59:30.570 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0119308, 2024-03-20T08:59:30.570 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:30.570 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:30.570 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:30.571 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.571 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:30.571 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.808201+0000", 2024-03-20T08:59:30.571 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.571 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.571 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.571 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:30.571 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.808201+0000", 2024-03-20T08:59:30.571 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.571 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.571 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.571 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:30.572 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.808202+0000", 2024-03-20T08:59:30.572 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.0800000000000001e-07 2024-03-20T08:59:30.572 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.572 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.572 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:30.572 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.808293+0000", 2024-03-20T08:59:30.572 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.1487e-05 2024-03-20T08:59:30.572 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.572 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.572 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:30.572 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.808303+0000", 2024-03-20T08:59:30.572 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0145e-05 2024-03-20T08:59:30.572 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.572 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.573 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:30.573 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.808310+0000", 2024-03-20T08:59:30.573 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.6649999999999997e-06 2024-03-20T08:59:30.573 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.573 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.573 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:30.573 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.810492+0000", 2024-03-20T08:59:30.573 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.002181685 2024-03-20T08:59:30.573 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.573 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.573 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:30.573 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.810513+0000", 2024-03-20T08:59:30.573 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0948999999999999e-05 2024-03-20T08:59:30.573 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.574 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.574 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:30.574 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.820112+0000", 2024-03-20T08:59:30.574 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.009599092 2024-03-20T08:59:30.574 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.574 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.574 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:30.574 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.820132+0000", 2024-03-20T08:59:30.574 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0469e-05 2024-03-20T08:59:30.574 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.574 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:30.574 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.574 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.574 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.575 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:106552 2.d e145/121 2:b97415a1:::benchmark_data_smithi122_107654_object106551:head v 145'6583, mlcod=145'6581)", 2024-03-20T08:59:30.575 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:59:24.881670+0000", 2024-03-20T08:59:30.575 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 5.6584625480000001, 2024-03-20T08:59:30.575 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.01198071, 2024-03-20T08:59:30.575 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:30.575 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:30.575 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:30.575 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.575 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:30.575 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.881670+0000", 2024-03-20T08:59:30.575 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.575 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.575 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.576 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:30.576 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.881670+0000", 2024-03-20T08:59:30.576 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.576 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.576 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.576 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:30.576 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.881671+0000", 2024-03-20T08:59:30.576 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.6899999999999996e-07 2024-03-20T08:59:30.576 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.576 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.576 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:30.576 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.881751+0000", 2024-03-20T08:59:30.576 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.9991e-05 2024-03-20T08:59:30.576 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.577 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.577 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:30.577 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.881766+0000", 2024-03-20T08:59:30.577 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4355999999999999e-05 2024-03-20T08:59:30.577 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.577 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.577 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:30.577 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.881774+0000", 2024-03-20T08:59:30.577 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.6510000000000007e-06 2024-03-20T08:59:30.577 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.577 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.577 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:30.577 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.881839+0000", 2024-03-20T08:59:30.577 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4455999999999994e-05 2024-03-20T08:59:30.577 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.578 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.578 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:30.578 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.881864+0000", 2024-03-20T08:59:30.578 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5463e-05 2024-03-20T08:59:30.578 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.578 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.578 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:30.578 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.893582+0000", 2024-03-20T08:59:30.578 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.011718414 2024-03-20T08:59:30.578 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.578 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.578 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:30.578 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.893651+0000", 2024-03-20T08:59:30.578 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.8410000000000004e-05 2024-03-20T08:59:30.579 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.579 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:30.579 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.579 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.579 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.579 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:106589 2.5 e145/121 2:a81bf70a:::benchmark_data_smithi122_107654_object106588:head v 145'6619, mlcod=145'6616)", 2024-03-20T08:59:30.579 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:59:24.907119+0000", 2024-03-20T08:59:30.579 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 5.6330135219999997, 2024-03-20T08:59:30.579 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.010705950000000001, 2024-03-20T08:59:30.579 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:30.579 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:30.579 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:30.579 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.580 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:30.580 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.907119+0000", 2024-03-20T08:59:30.580 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.580 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.580 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.580 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:30.580 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.907119+0000", 2024-03-20T08:59:30.580 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.580 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.580 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.580 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:30.580 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.907120+0000", 2024-03-20T08:59:30.580 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0240000000000001e-06 2024-03-20T08:59:30.580 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.580 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.581 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:30.581 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.907203+0000", 2024-03-20T08:59:30.581 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.2625000000000003e-05 2024-03-20T08:59:30.581 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.581 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.581 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:30.581 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.907215+0000", 2024-03-20T08:59:30.581 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2278e-05 2024-03-20T08:59:30.582 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.582 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.582 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:30.582 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.907223+0000", 2024-03-20T08:59:30.582 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.4660000000000002e-06 2024-03-20T08:59:30.582 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.582 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.582 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:30.582 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.908324+0000", 2024-03-20T08:59:30.582 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0011012730000000001 2024-03-20T08:59:30.582 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.582 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.582 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:30.582 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.908343+0000", 2024-03-20T08:59:30.583 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9170000000000001e-05 2024-03-20T08:59:30.583 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.583 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.583 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:30.583 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.917779+0000", 2024-03-20T08:59:30.583 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0094357989999999999 2024-03-20T08:59:30.583 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.583 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.583 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:30.583 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.917825+0000", 2024-03-20T08:59:30.583 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6315e-05 2024-03-20T08:59:30.583 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.583 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:30.583 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.584 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.584 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.584 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:106585 2.d e145/121 2:bfdf5b9d:::benchmark_data_smithi122_107654_object106584:head v 145'6587, mlcod=145'6583)", 2024-03-20T08:59:30.584 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:59:24.907723+0000", 2024-03-20T08:59:30.584 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 5.6324099939999996, 2024-03-20T08:59:30.584 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.015577387, 2024-03-20T08:59:30.584 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:30.584 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:30.584 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:30.584 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.584 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:30.584 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.907723+0000", 2024-03-20T08:59:30.584 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.584 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.584 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.585 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:30.585 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.907723+0000", 2024-03-20T08:59:30.585 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.585 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.585 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.585 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:30.585 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.907723+0000", 2024-03-20T08:59:30.585 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4600000000000005e-07 2024-03-20T08:59:30.585 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.585 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.585 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:30.585 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.907776+0000", 2024-03-20T08:59:30.585 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2719000000000001e-05 2024-03-20T08:59:30.585 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.585 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.586 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:30.586 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.907783+0000", 2024-03-20T08:59:30.586 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.9630000000000001e-06 2024-03-20T08:59:30.586 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.586 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.586 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:30.586 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.907789+0000", 2024-03-20T08:59:30.586 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6099999999999997e-06 2024-03-20T08:59:30.586 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.586 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.586 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:30.586 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.909141+0000", 2024-03-20T08:59:30.586 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001352497 2024-03-20T08:59:30.587 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.587 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.587 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:30.587 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.909158+0000", 2024-03-20T08:59:30.587 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7156e-05 2024-03-20T08:59:30.587 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.587 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.587 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:30.587 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.923249+0000", 2024-03-20T08:59:30.587 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.014090933999999999 2024-03-20T08:59:30.587 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.587 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.587 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:30.587 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.923300+0000", 2024-03-20T08:59:30.588 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0961999999999998e-05 2024-03-20T08:59:30.588 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.588 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:30.588 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.588 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.588 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.588 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:106587 2.d e145/121 2:b58cb91a:::benchmark_data_smithi122_107654_object106586:head v 145'6588, mlcod=145'6583)", 2024-03-20T08:59:30.588 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:59:24.907998+0000", 2024-03-20T08:59:30.588 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 5.6321344, 2024-03-20T08:59:30.588 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.015336436, 2024-03-20T08:59:30.588 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:30.588 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:30.588 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:30.588 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.588 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:30.589 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.907998+0000", 2024-03-20T08:59:30.589 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.589 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.589 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.589 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:30.589 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.907998+0000", 2024-03-20T08:59:30.589 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.589 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.589 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.589 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:30.589 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.907999+0000", 2024-03-20T08:59:30.589 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6300000000000001e-07 2024-03-20T08:59:30.589 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.589 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.590 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:30.590 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.908088+0000", 2024-03-20T08:59:30.590 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.9157e-05 2024-03-20T08:59:30.590 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.590 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.590 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:30.590 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.908093+0000", 2024-03-20T08:59:30.590 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6180000000000003e-06 2024-03-20T08:59:30.590 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.590 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.590 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:30.590 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.908098+0000", 2024-03-20T08:59:30.590 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8210000000000001e-06 2024-03-20T08:59:30.590 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.591 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.591 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:30.591 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.911848+0000", 2024-03-20T08:59:30.591 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0037500099999999998 2024-03-20T08:59:30.591 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.591 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.591 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:30.591 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.911868+0000", 2024-03-20T08:59:30.592 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9706000000000001e-05 2024-03-20T08:59:30.592 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.592 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.592 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:30.592 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.923306+0000", 2024-03-20T08:59:30.592 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.011437575 2024-03-20T08:59:30.592 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.592 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.592 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:30.592 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:24.923335+0000", 2024-03-20T08:59:30.592 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9186000000000001e-05 2024-03-20T08:59:30.592 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.593 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:30.593 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.593 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.593 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.593 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:106811 2.f e145/121 2:f5c9e0fe:::benchmark_data_smithi122_107654_object106810:head v 145'6578, mlcod=145'6575)", 2024-03-20T08:59:30.593 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:59:25.107954+0000", 2024-03-20T08:59:30.593 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 5.432178725, 2024-03-20T08:59:30.593 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.011075964000000001, 2024-03-20T08:59:30.593 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:30.593 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:30.593 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:30.593 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.593 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:30.593 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.107954+0000", 2024-03-20T08:59:30.594 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.594 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.594 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.594 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:30.594 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.107954+0000", 2024-03-20T08:59:30.594 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.594 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.594 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.594 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:30.594 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.107955+0000", 2024-03-20T08:59:30.594 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.7800000000000001e-07 2024-03-20T08:59:30.594 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.594 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.594 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:30.595 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.108038+0000", 2024-03-20T08:59:30.595 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.3594e-05 2024-03-20T08:59:30.595 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.595 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.595 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:30.595 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.108049+0000", 2024-03-20T08:59:30.595 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0345e-05 2024-03-20T08:59:30.595 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.595 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.595 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:30.595 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.108054+0000", 2024-03-20T08:59:30.595 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4319999999999998e-06 2024-03-20T08:59:30.595 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.595 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.596 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:30.596 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.109513+0000", 2024-03-20T08:59:30.596 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0014586219999999999 2024-03-20T08:59:30.596 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.596 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.596 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:30.596 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.109529+0000", 2024-03-20T08:59:30.596 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5945000000000001e-05 2024-03-20T08:59:30.596 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.596 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.596 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:30.596 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.118986+0000", 2024-03-20T08:59:30.596 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.009457577 2024-03-20T08:59:30.596 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.596 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.597 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:30.597 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.119030+0000", 2024-03-20T08:59:30.597 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.3770999999999998e-05 2024-03-20T08:59:30.597 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.597 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:30.597 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.597 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.597 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.597 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:106826 2.f e145/121 2:f1567c7a:::benchmark_data_smithi122_107654_object106825:head v 145'6580, mlcod=145'6576)", 2024-03-20T08:59:30.597 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:59:25.119195+0000", 2024-03-20T08:59:30.597 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 5.4209379030000004, 2024-03-20T08:59:30.597 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.015767461999999999, 2024-03-20T08:59:30.597 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:30.597 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:30.598 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:30.598 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.598 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:30.598 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.119195+0000", 2024-03-20T08:59:30.598 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.598 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.598 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.598 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:30.598 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.119195+0000", 2024-03-20T08:59:30.598 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.598 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.598 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.598 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:30.598 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.119196+0000", 2024-03-20T08:59:30.599 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.6099999999999999e-07 2024-03-20T08:59:30.599 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.599 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.599 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:30.599 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.119221+0000", 2024-03-20T08:59:30.599 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5316999999999998e-05 2024-03-20T08:59:30.599 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.599 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.599 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:30.599 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.119231+0000", 2024-03-20T08:59:30.599 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0127e-05 2024-03-20T08:59:30.599 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.599 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.599 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:30.599 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.119237+0000", 2024-03-20T08:59:30.600 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8669999999999999e-06 2024-03-20T08:59:30.600 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.600 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.600 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:30.600 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.119282+0000", 2024-03-20T08:59:30.600 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5272000000000003e-05 2024-03-20T08:59:30.600 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.600 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.600 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:30.600 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.119310+0000", 2024-03-20T08:59:30.600 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.7364000000000001e-05 2024-03-20T08:59:30.600 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.601 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.601 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:30.601 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.134903+0000", 2024-03-20T08:59:30.601 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.015593057 2024-03-20T08:59:30.601 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.601 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.601 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:30.601 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.134962+0000", 2024-03-20T08:59:30.601 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9497000000000003e-05 2024-03-20T08:59:30.601 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.601 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:30.601 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.601 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.601 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.602 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:106831 2.f e145/121 2:fe715fcf:::benchmark_data_smithi122_107654_object106830:head v 145'6581, mlcod=145'6578)", 2024-03-20T08:59:30.602 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:59:25.123485+0000", 2024-03-20T08:59:30.602 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 5.4166480659999996, 2024-03-20T08:59:30.602 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.011507402, 2024-03-20T08:59:30.602 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:30.602 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:30.602 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:30.602 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.602 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:30.602 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.123485+0000", 2024-03-20T08:59:30.603 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.603 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.603 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.603 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:30.603 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.123485+0000", 2024-03-20T08:59:30.603 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.603 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.603 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.603 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:30.603 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.123486+0000", 2024-03-20T08:59:30.603 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1519999999999999e-06 2024-03-20T08:59:30.603 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.603 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.603 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:30.604 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.123569+0000", 2024-03-20T08:59:30.604 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.3073999999999998e-05 2024-03-20T08:59:30.604 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.604 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.604 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:30.604 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.123588+0000", 2024-03-20T08:59:30.604 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9204000000000001e-05 2024-03-20T08:59:30.604 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.604 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.604 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:30.604 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.123600+0000", 2024-03-20T08:59:30.604 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2303000000000001e-05 2024-03-20T08:59:30.604 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.604 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.605 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:30.605 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.123657+0000", 2024-03-20T08:59:30.605 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6799999999999998e-05 2024-03-20T08:59:30.605 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.605 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.605 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:30.605 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.123680+0000", 2024-03-20T08:59:30.605 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2501e-05 2024-03-20T08:59:30.605 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.605 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.605 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:30.605 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.134966+0000", 2024-03-20T08:59:30.605 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.011286562 2024-03-20T08:59:30.605 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.606 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.606 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:30.606 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.134992+0000", 2024-03-20T08:59:30.606 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5806e-05 2024-03-20T08:59:30.606 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.606 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:30.606 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.606 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.606 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.606 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:106854 2.f e145/121 2:fcd589ec:::benchmark_data_smithi122_107654_object106853:head v 145'6582, mlcod=145'6580)", 2024-03-20T08:59:30.606 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:59:25.143070+0000", 2024-03-20T08:59:30.606 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 5.3970632199999997, 2024-03-20T08:59:30.606 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.010337898999999999, 2024-03-20T08:59:30.606 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:30.607 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:30.607 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:30.607 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.607 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:30.607 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.143070+0000", 2024-03-20T08:59:30.607 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.607 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.607 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.607 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:30.607 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.143070+0000", 2024-03-20T08:59:30.607 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.607 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.607 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.607 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:30.608 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.143071+0000", 2024-03-20T08:59:30.608 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2100000000000001e-06 2024-03-20T08:59:30.608 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.608 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.608 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:30.608 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.143130+0000", 2024-03-20T08:59:30.608 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9536000000000001e-05 2024-03-20T08:59:30.608 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.608 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.608 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:30.608 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.143147+0000", 2024-03-20T08:59:30.608 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6911000000000001e-05 2024-03-20T08:59:30.608 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.608 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.609 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:30.609 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.143157+0000", 2024-03-20T08:59:30.609 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.4040000000000001e-06 2024-03-20T08:59:30.609 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.609 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.609 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:30.609 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.143240+0000", 2024-03-20T08:59:30.609 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.3653999999999996e-05 2024-03-20T08:59:30.609 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.609 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.609 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:30.609 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.143264+0000", 2024-03-20T08:59:30.609 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3300000000000001e-05 2024-03-20T08:59:30.609 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.610 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.610 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:30.610 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.153360+0000", 2024-03-20T08:59:30.610 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.010096846 2024-03-20T08:59:30.610 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.610 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.610 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:30.610 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.153407+0000", 2024-03-20T08:59:30.610 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.7037999999999999e-05 2024-03-20T08:59:30.610 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.610 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:30.610 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.610 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.610 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.611 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:106857 2.d e145/121 2:b62d1b33:::benchmark_data_smithi122_107654_object106856:head v 145'6605, mlcod=145'6603)", 2024-03-20T08:59:30.611 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:59:25.144345+0000", 2024-03-20T08:59:30.611 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 5.3957876469999997, 2024-03-20T08:59:30.611 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.011619061999999999, 2024-03-20T08:59:30.611 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:30.611 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:30.611 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:30.611 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.611 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:30.611 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.144345+0000", 2024-03-20T08:59:30.611 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.611 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.611 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.611 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:30.612 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.144345+0000", 2024-03-20T08:59:30.612 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.612 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.612 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.612 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:30.612 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.144354+0000", 2024-03-20T08:59:30.612 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.2660000000000007e-06 2024-03-20T08:59:30.612 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.612 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.612 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:30.612 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.144426+0000", 2024-03-20T08:59:30.612 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.2105000000000002e-05 2024-03-20T08:59:30.612 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.612 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.613 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:30.613 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.144441+0000", 2024-03-20T08:59:30.613 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4243000000000001e-05 2024-03-20T08:59:30.613 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.613 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.613 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:30.613 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.144447+0000", 2024-03-20T08:59:30.613 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4910000000000002e-06 2024-03-20T08:59:30.613 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.613 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.613 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:30.613 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.144504+0000", 2024-03-20T08:59:30.613 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6975000000000003e-05 2024-03-20T08:59:30.613 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.614 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.614 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:30.614 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.144518+0000", 2024-03-20T08:59:30.614 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4004e-05 2024-03-20T08:59:30.614 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.614 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.614 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:30.614 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.155915+0000", 2024-03-20T08:59:30.614 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.011397087 2024-03-20T08:59:30.614 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.614 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.614 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:30.614 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.155964+0000", 2024-03-20T08:59:30.615 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8891000000000003e-05 2024-03-20T08:59:30.615 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.615 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:30.615 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.615 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.615 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.615 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:107324 2.5 e145/121 2:a2c63674:::benchmark_data_smithi122_107654_object107323:head v 145'6659, mlcod=145'6657)", 2024-03-20T08:59:30.615 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:59:25.543768+0000", 2024-03-20T08:59:30.615 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 4.9963646969999997, 2024-03-20T08:59:30.615 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.011501257000000001, 2024-03-20T08:59:30.615 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:30.615 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:30.615 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:30.615 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.616 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:30.616 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.543768+0000", 2024-03-20T08:59:30.616 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.616 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.616 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.616 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:30.616 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.543768+0000", 2024-03-20T08:59:30.616 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.616 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.616 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.616 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:30.616 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.543769+0000", 2024-03-20T08:59:30.616 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.3099999999999996e-07 2024-03-20T08:59:30.616 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.617 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.617 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:30.617 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.543810+0000", 2024-03-20T08:59:30.617 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.1325999999999997e-05 2024-03-20T08:59:30.617 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.617 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.617 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:30.617 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.543822+0000", 2024-03-20T08:59:30.617 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2036e-05 2024-03-20T08:59:30.617 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.617 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.617 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:30.617 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.543830+0000", 2024-03-20T08:59:30.617 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.6260000000000003e-06 2024-03-20T08:59:30.617 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.618 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.618 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:30.618 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.543888+0000", 2024-03-20T08:59:30.618 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8303000000000001e-05 2024-03-20T08:59:30.618 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.618 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.618 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:30.618 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.543923+0000", 2024-03-20T08:59:30.618 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.4783000000000002e-05 2024-03-20T08:59:30.618 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.618 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.618 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:30.618 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.555220+0000", 2024-03-20T08:59:30.619 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.011296572 2024-03-20T08:59:30.619 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.619 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.619 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:30.619 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:25.555269+0000", 2024-03-20T08:59:30.619 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9679999999999999e-05 2024-03-20T08:59:30.619 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.619 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:30.619 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.619 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.619 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.619 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:107903 2.5 e145/121 2:a4d6cec3:::benchmark_data_smithi122_107654_object107902:head v 145'6691, mlcod=145'6689)", 2024-03-20T08:59:30.619 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:59:26.059215+0000", 2024-03-20T08:59:30.619 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 4.4809181709999999, 2024-03-20T08:59:30.620 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.013475747, 2024-03-20T08:59:30.620 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:30.620 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:30.620 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:30.620 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.620 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:30.620 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:26.059215+0000", 2024-03-20T08:59:30.620 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.620 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.620 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.620 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:30.620 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:26.059215+0000", 2024-03-20T08:59:30.620 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.620 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.621 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.621 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:30.621 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:26.059215+0000", 2024-03-20T08:59:30.621 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4600000000000004e-07 2024-03-20T08:59:30.621 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.621 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.621 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:30.621 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:26.059285+0000", 2024-03-20T08:59:30.621 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.9976999999999998e-05 2024-03-20T08:59:30.621 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.621 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.621 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:30.621 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:26.059299+0000", 2024-03-20T08:59:30.621 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4011e-05 2024-03-20T08:59:30.622 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.622 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.622 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:30.622 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:26.059307+0000", 2024-03-20T08:59:30.622 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3370000000000002e-06 2024-03-20T08:59:30.622 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.622 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.622 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:30.622 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:26.059357+0000", 2024-03-20T08:59:30.622 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0176999999999997e-05 2024-03-20T08:59:30.622 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.622 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.622 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:30.622 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:26.059377+0000", 2024-03-20T08:59:30.623 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0687999999999999e-05 2024-03-20T08:59:30.623 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.623 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.623 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:30.623 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:26.072653+0000", 2024-03-20T08:59:30.623 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.01327594 2024-03-20T08:59:30.623 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.623 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.623 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:30.623 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:26.072690+0000", 2024-03-20T08:59:30.623 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6971000000000002e-05 2024-03-20T08:59:30.623 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.623 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:30.623 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.624 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.624 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.624 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:107907 2.5 e145/121 2:a0a0bec0:::benchmark_data_smithi122_107654_object107906:head v 145'6692, mlcod=145'6689)", 2024-03-20T08:59:30.624 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:59:26.062662+0000", 2024-03-20T08:59:30.624 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 4.4774712340000002, 2024-03-20T08:59:30.624 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.010054419, 2024-03-20T08:59:30.624 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:30.624 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:30.624 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:30.624 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.624 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:30.624 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:26.062662+0000", 2024-03-20T08:59:30.624 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.624 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.625 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.625 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:30.625 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:26.062662+0000", 2024-03-20T08:59:30.625 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.625 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.625 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.625 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:30.625 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:26.062662+0000", 2024-03-20T08:59:30.625 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.9099999999999991e-07 2024-03-20T08:59:30.625 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.625 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.625 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:30.625 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:26.062697+0000", 2024-03-20T08:59:30.625 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.4272e-05 2024-03-20T08:59:30.625 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.626 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.626 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:30.626 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:26.062710+0000", 2024-03-20T08:59:30.626 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3101e-05 2024-03-20T08:59:30.626 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.626 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.626 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:30.626 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:26.062718+0000", 2024-03-20T08:59:30.626 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.8939999999999993e-06 2024-03-20T08:59:30.626 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.626 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.626 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:30.627 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:26.062768+0000", 2024-03-20T08:59:30.627 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0535000000000003e-05 2024-03-20T08:59:30.627 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.627 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.627 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:30.627 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:26.062788+0000", 2024-03-20T08:59:30.627 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9979e-05 2024-03-20T08:59:30.627 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.627 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.627 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:30.627 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:26.072695+0000", 2024-03-20T08:59:30.627 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0099065750000000008 2024-03-20T08:59:30.627 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.627 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.628 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:30.628 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:26.072716+0000", 2024-03-20T08:59:30.628 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1072000000000001e-05 2024-03-20T08:59:30.628 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.628 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:30.628 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.628 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.628 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.628 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:107942 2.f e145/121 2:fecaf6c8:::benchmark_data_smithi122_107654_object107941:head v 145'6656, mlcod=145'6650)", 2024-03-20T08:59:30.628 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:59:26.088913+0000", 2024-03-20T08:59:30.628 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 4.4512198200000004, 2024-03-20T08:59:30.628 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.01019257, 2024-03-20T08:59:30.628 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:30.629 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:30.629 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:30.629 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.629 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:30.629 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:26.088913+0000", 2024-03-20T08:59:30.629 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.629 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.629 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.629 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:30.629 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:26.088913+0000", 2024-03-20T08:59:30.629 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:30.629 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.629 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.629 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:30.630 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:26.088914+0000", 2024-03-20T08:59:30.630 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8699999999999995e-07 2024-03-20T08:59:30.630 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.630 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.630 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:30.630 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:26.088993+0000", 2024-03-20T08:59:30.630 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.9239999999999993e-05 2024-03-20T08:59:30.630 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.630 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.630 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:30.630 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:26.089004+0000", 2024-03-20T08:59:30.630 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1073000000000001e-05 2024-03-20T08:59:30.630 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.631 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.631 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:30.631 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:26.089011+0000", 2024-03-20T08:59:30.631 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.1189999999999999e-06 2024-03-20T08:59:30.631 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.631 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.631 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:30.631 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:26.093209+0000", 2024-03-20T08:59:30.631 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0041977799999999999 2024-03-20T08:59:30.631 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.631 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.631 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:30.631 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:26.093234+0000", 2024-03-20T08:59:30.631 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.4973e-05 2024-03-20T08:59:30.632 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.632 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.632 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:30.632 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:26.099053+0000", 2024-03-20T08:59:30.632 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0058190569999999999 2024-03-20T08:59:30.632 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:30.632 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:30.632 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:30.632 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:26.099105+0000", 2024-03-20T08:59:30.632 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2741000000000002e-05 2024-03-20T08:59:30.632 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.632 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:30.632 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.632 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:30.633 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:30.633 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T08:59:30.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:30 smithi115 ceph-mon[72476]: 2.8 scrub ok 2024-03-20T08:59:30.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:30 smithi115 ceph-mon[72476]: pgmap v434: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 940 op/s 2024-03-20T08:59:30.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:30 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T08:59:30.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:30 smithi115 ceph-mon[72476]: Cluster is now healthy 2024-03-20T08:59:30.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:30 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3928035093' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T08:59:30.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:30 smithi115 ceph-mon[72476]: osdmap e148: 12 total, 12 up, 12 in 2024-03-20T08:59:30.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:30 smithi115 ceph-mon[72476]: 2.a scrub starts 2024-03-20T08:59:30.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:30 smithi122 ceph-mon[73495]: 2.8 scrub ok 2024-03-20T08:59:30.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:30 smithi122 ceph-mon[73495]: pgmap v434: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 940 op/s 2024-03-20T08:59:30.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:30 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T08:59:30.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:30 smithi122 ceph-mon[73495]: Cluster is now healthy 2024-03-20T08:59:30.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:30 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3928035093' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T08:59:30.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:30 smithi122 ceph-mon[73495]: osdmap e148: 12 total, 12 up, 12 in 2024-03-20T08:59:30.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:30 smithi122 ceph-mon[73495]: 2.a scrub starts 2024-03-20T08:59:30.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:30 smithi026 ceph-mon[68309]: 2.8 scrub ok 2024-03-20T08:59:30.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:30 smithi026 ceph-mon[68309]: pgmap v434: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 940 op/s 2024-03-20T08:59:30.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:30 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T08:59:30.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:30 smithi026 ceph-mon[68309]: Cluster is now healthy 2024-03-20T08:59:30.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:30 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3928035093' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T08:59:30.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:30 smithi026 ceph-mon[68309]: osdmap e148: 12 total, 12 up, 12 in 2024-03-20T08:59:30.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:30 smithi026 ceph-mon[68309]: 2.a scrub starts 2024-03-20T08:59:31.334 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-20T08:59:31.439 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 67 16 108304 108288 101 0 - 0.00913997 2024-03-20T08:59:31.599 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:59:31.884 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:31 smithi115 ceph-mon[72476]: 2.a scrub ok 2024-03-20T08:59:31.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:31 smithi122 ceph-mon[73495]: 2.a scrub ok 2024-03-20T08:59:31.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:31 smithi026 ceph-mon[68309]: 2.a scrub ok 2024-03-20T08:59:32.439 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 68 16 108304 108288 99.5146 0 - 0.00913997 2024-03-20T08:59:33.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:32 smithi115 ceph-mon[72476]: pgmap v436: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 657 op/s 2024-03-20T08:59:33.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:32 smithi115 ceph-mon[72476]: 2.2 scrub starts 2024-03-20T08:59:33.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:32 smithi115 ceph-mon[72476]: 2.2 scrub ok 2024-03-20T08:59:33.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:32 smithi122 ceph-mon[73495]: pgmap v436: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 657 op/s 2024-03-20T08:59:33.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:32 smithi122 ceph-mon[73495]: 2.2 scrub starts 2024-03-20T08:59:33.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:32 smithi122 ceph-mon[73495]: 2.2 scrub ok 2024-03-20T08:59:33.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:32 smithi026 ceph-mon[68309]: pgmap v436: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 657 op/s 2024-03-20T08:59:33.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:32 smithi026 ceph-mon[68309]: 2.2 scrub starts 2024-03-20T08:59:33.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:32 smithi026 ceph-mon[68309]: 2.2 scrub ok 2024-03-20T08:59:33.319 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T08:59:33.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:33.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T08:59:33.424 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T08:59:33.439 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 69 16 108304 108288 98.0724 0 - 0.00913997 2024-03-20T08:59:33.456 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:33.478 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T08:59:33.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:33.482 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T08:59:33.583 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T08:59:33.619 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:33.862 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T08:59:33.863 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T08:59:33.863 INFO:teuthology.orchestra.run.smithi115.stdout: "num_ops": 0 2024-03-20T08:59:33.863 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T08:59:33.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:33 smithi026 ceph-mon[68309]: 2.f scrub starts 2024-03-20T08:59:34.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:33 smithi115 ceph-mon[72476]: 2.f scrub starts 2024-03-20T08:59:34.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:33 smithi122 ceph-mon[73495]: 2.f scrub starts 2024-03-20T08:59:34.439 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 70 16 108304 108288 96.6714 0 - 0.00913997 2024-03-20T08:59:34.659 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-20T08:59:34.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:34 smithi115 ceph-mon[72476]: pgmap v437: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s wr, 329 op/s 2024-03-20T08:59:34.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:34 smithi115 ceph-mon[72476]: 2.7 scrub starts 2024-03-20T08:59:34.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:34 smithi115 ceph-mon[72476]: 2.f scrub ok 2024-03-20T08:59:34.920 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:59:35.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:34 smithi122 ceph-mon[73495]: pgmap v437: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s wr, 329 op/s 2024-03-20T08:59:35.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:34 smithi122 ceph-mon[73495]: 2.7 scrub starts 2024-03-20T08:59:35.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:34 smithi122 ceph-mon[73495]: 2.f scrub ok 2024-03-20T08:59:35.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:34 smithi026 ceph-mon[68309]: pgmap v437: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s wr, 329 op/s 2024-03-20T08:59:35.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:34 smithi026 ceph-mon[68309]: 2.7 scrub starts 2024-03-20T08:59:35.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:34 smithi026 ceph-mon[68309]: 2.f scrub ok 2024-03-20T08:59:35.439 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 71 16 108304 108288 95.3099 0 - 0.00913997 2024-03-20T08:59:36.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:35 smithi115 ceph-mon[72476]: 2.7 scrub ok 2024-03-20T08:59:36.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:35 smithi122 ceph-mon[73495]: 2.7 scrub ok 2024-03-20T08:59:36.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:35 smithi026 ceph-mon[68309]: 2.7 scrub ok 2024-03-20T08:59:36.439 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 72 16 108304 108288 93.9862 0 - 0.00913997 2024-03-20T08:59:36.857 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T08:59:36.858 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T08:59:36.858 INFO:teuthology.orchestra.run.smithi115.stdout: "complaint_time": 30, 2024-03-20T08:59:36.858 INFO:teuthology.orchestra.run.smithi115.stdout: "num_blocked_ops": 0 2024-03-20T08:59:36.858 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T08:59:36.884 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:36 smithi115 ceph-mon[72476]: pgmap v438: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 200 op/s 2024-03-20T08:59:36.884 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:36 smithi115 ceph-mon[72476]: 2.c deep-scrub starts 2024-03-20T08:59:37.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:36 smithi122 ceph-mon[73495]: pgmap v438: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 200 op/s 2024-03-20T08:59:37.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:36 smithi122 ceph-mon[73495]: 2.c deep-scrub starts 2024-03-20T08:59:37.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:36 smithi026 ceph-mon[68309]: pgmap v438: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 6.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 200 op/s 2024-03-20T08:59:37.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:36 smithi026 ceph-mon[68309]: 2.c deep-scrub starts 2024-03-20T08:59:37.442 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 73 16 108304 108288 92.6988 0 - 0.00913997 2024-03-20T08:59:37.979 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-20T08:59:38.246 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:59:38.439 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 74 16 108304 108288 91.4461 0 - 0.00913997 2024-03-20T08:59:38.975 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T08:59:38.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:38.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T08:59:38.985 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T08:59:38.986 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-20T08:59:38.986 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.5 'trim stale osdmaps' 2024-03-20T08:59:39.054 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:38 smithi026 ceph-mon[68309]: pgmap v439: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s wr, 102 op/s 2024-03-20T08:59:39.080 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T08:59:39.112 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:39.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:38 smithi115 ceph-mon[72476]: pgmap v439: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s wr, 102 op/s 2024-03-20T08:59:39.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:38 smithi122 ceph-mon[73495]: pgmap v439: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s wr, 102 op/s 2024-03-20T08:59:39.440 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 75 16 108304 108288 90.2269 0 - 0.00913997 2024-03-20T08:59:39.470 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T08:59:39.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:39.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T08:59:39.573 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T08:59:39.599 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:39.618 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T08:59:39.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:39.621 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T08:59:39.722 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T08:59:39.751 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:39.953 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T08:59:39.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:39.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T08:59:39.958 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:59:40.059 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T08:59:40.093 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:40.440 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 76 16 108304 108288 89.0398 0 - 0.00913997 2024-03-20T08:59:40.860 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T08:59:40.883 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T08:59:40.908 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T08:59:40.936 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T08:59:40.945 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T08:59:40.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:40.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T08:59:40.958 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T08:59:40.982 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T08:59:41.007 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T08:59:41.029 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T08:59:41.039 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:40 smithi115 ceph-mon[72476]: pgmap v440: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 15 active+clean; 6.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s wr, 60 op/s 2024-03-20T08:59:41.049 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T08:59:41.050 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T08:59:41.072 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T08:59:41.080 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:40 smithi026 ceph-mon[68309]: pgmap v440: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 15 active+clean; 6.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s wr, 60 op/s 2024-03-20T08:59:41.084 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:41.096 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T08:59:41.119 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T08:59:41.120 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T08:59:41.120 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T08:59:41.120 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T08:59:41.120 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T08:59:41.120 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T08:59:41.120 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T08:59:41.120 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T08:59:41.120 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T08:59:41.121 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T08:59:41.121 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T08:59:41.121 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T08:59:41.121 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T08:59:41.125 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T08:59:41.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:41.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T08:59:41.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:40 smithi122 ceph-mon[73495]: pgmap v440: 17 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 15 active+clean; 6.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s wr, 60 op/s 2024-03-20T08:59:41.230 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T08:59:41.265 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:41.440 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 77 16 108541 108525 88.0759 1.05804 0.00383624 0.011347 2024-03-20T08:59:41.467 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T08:59:41.623 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T08:59:41.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:41.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T08:59:41.727 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T08:59:41.758 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:41.781 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T08:59:41.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:41.786 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T08:59:41.792 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:41 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2414347499' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T08:59:41.792 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:41 smithi026 ceph-mon[68309]: 2.4 scrub starts 2024-03-20T08:59:41.886 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T08:59:41.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:41 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2414347499' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T08:59:41.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:41 smithi115 ceph-mon[72476]: 2.4 scrub starts 2024-03-20T08:59:41.918 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:42.170 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T08:59:42.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:42.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T08:59:42.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:41 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2414347499' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T08:59:42.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:41 smithi122 ceph-mon[73495]: 2.4 scrub starts 2024-03-20T08:59:42.275 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T08:59:42.313 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:42.337 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T08:59:42.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:42.340 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T08:59:42.440 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 78 16 109410 109394 87.643 54.3125 0.00567796 0.0114037 2024-03-20T08:59:42.441 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T08:59:42.480 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:42.508 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T08:59:42.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:42.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T08:59:42.611 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T08:59:42.641 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:42.671 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T08:59:42.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:42.678 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T08:59:42.779 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T08:59:42.808 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T08:59:42.808 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T08:59:42.813 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:42.837 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T08:59:42.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:42.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T08:59:42.888 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:59:42.943 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T08:59:42.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:42 smithi122 ceph-mon[73495]: pgmap v441: 17 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 15 active+clean; 6.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s wr, 50 op/s 2024-03-20T08:59:42.972 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:42.976 INFO:teuthology.orchestra.run.smithi026.stderr:no valid command found; 10 closest matches: 2024-03-20T08:59:42.976 INFO:teuthology.orchestra.run.smithi026.stderr:0 2024-03-20T08:59:42.976 INFO:teuthology.orchestra.run.smithi026.stderr:1 2024-03-20T08:59:42.976 INFO:teuthology.orchestra.run.smithi026.stderr:2 2024-03-20T08:59:42.976 INFO:teuthology.orchestra.run.smithi026.stderr:abort 2024-03-20T08:59:42.976 INFO:teuthology.orchestra.run.smithi026.stderr:assert 2024-03-20T08:59:42.976 INFO:teuthology.orchestra.run.smithi026.stderr:bench [] [] [] [] 2024-03-20T08:59:42.976 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs debug_inject_read_zeros 2024-03-20T08:59:42.977 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs files list 2024-03-20T08:59:42.977 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs stats 2024-03-20T08:59:42.977 INFO:teuthology.orchestra.run.smithi026.stderr:bluestore allocator dump block 2024-03-20T08:59:42.977 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: invalid command 2024-03-20T08:59:43.075 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T08:59:43.077 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:42 smithi115 ceph-mon[72476]: pgmap v441: 17 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 15 active+clean; 6.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s wr, 50 op/s 2024-03-20T08:59:43.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:43.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T08:59:43.180 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T08:59:43.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:42 smithi026 ceph-mon[68309]: pgmap v441: 17 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 15 active+clean; 6.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s wr, 50 op/s 2024-03-20T08:59:43.253 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:43.294 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T08:59:43.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:43.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T08:59:43.402 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T08:59:43.433 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:43.440 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 79 16 110259 110243 87.2052 53.0625 0.0320992 0.0114622 2024-03-20T08:59:43.559 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T08:59:43.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:43.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T08:59:43.591 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-20T08:59:43.668 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T08:59:43.727 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:43.864 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T08:59:43.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:43.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T08:59:43.910 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T08:59:43.911 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-20T08:59:43.911 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.6 'trim stale osdmaps' 2024-03-20T08:59:43.939 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:59:43.982 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T08:59:44.014 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:43 smithi026 ceph-mon[68309]: 2.6 scrub starts 2024-03-20T08:59:44.015 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:44.045 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T08:59:44.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:44.053 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T08:59:44.154 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T08:59:44.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:43 smithi115 ceph-mon[72476]: 2.6 scrub starts 2024-03-20T08:59:44.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:43 smithi122 ceph-mon[73495]: 2.6 scrub starts 2024-03-20T08:59:44.186 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:44.211 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T08:59:44.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:44.219 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T08:59:44.320 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T08:59:44.354 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:44.377 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T08:59:44.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:44.383 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T08:59:44.440 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:2024-03-20T08:59:44.440959+0000 min lat: 0.00103856 max lat: 15.0259 avg lat: 0.0115181 2024-03-20T08:59:44.440 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T08:59:44.440 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 80 16 111044 111028 86.7284 49.0625 0.00396331 0.0115181 2024-03-20T08:59:44.443 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T08:59:44.484 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T08:59:44.512 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:44.534 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T08:59:44.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:44.540 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T08:59:44.642 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T08:59:44.761 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:44.828 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:44 smithi026 ceph-mon[68309]: pgmap v442: 17 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 15 active+clean; 6.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s wr, 41 op/s 2024-03-20T08:59:44.830 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:44 smithi122 ceph-mon[73495]: pgmap v442: 17 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 15 active+clean; 6.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s wr, 41 op/s 2024-03-20T08:59:44.998 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:59:45.032 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T08:59:45.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:45.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T08:59:45.139 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T08:59:45.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:44 smithi115 ceph-mon[72476]: pgmap v442: 17 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 15 active+clean; 6.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s wr, 41 op/s 2024-03-20T08:59:45.174 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:45.328 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T08:59:45.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:45.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T08:59:45.432 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T08:59:45.440 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 81 16 111778 111762 86.224 45.875 0.00811616 0.0115914 2024-03-20T08:59:45.460 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:45.481 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T08:59:45.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:45.486 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T08:59:45.587 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T08:59:45.659 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:45.894 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T08:59:45.894 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T08:59:45.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:45 smithi115 ceph-mon[72476]: 2.6 scrub ok 2024-03-20T08:59:45.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:45 smithi026 ceph-mon[68309]: 2.6 scrub ok 2024-03-20T08:59:46.134 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:59:46.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:45 smithi122 ceph-mon[73495]: 2.6 scrub ok 2024-03-20T08:59:46.440 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 82 16 112587 112571 85.7891 50.5625 0.00445974 0.0116489 2024-03-20T08:59:46.713 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-20T08:59:46.857 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T08:59:46.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:46.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T08:59:46.963 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T08:59:46.966 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T08:59:47.018 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:59:47.021 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:47.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:46 smithi115 ceph-mon[72476]: pgmap v443: 17 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 14 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s wr, 128 op/s 2024-03-20T08:59:47.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:46 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/610353236' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T08:59:47.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:46 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T08:59:47.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:46 smithi122 ceph-mon[73495]: pgmap v443: 17 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 14 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s wr, 128 op/s 2024-03-20T08:59:47.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:46 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/610353236' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T08:59:47.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:46 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T08:59:47.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:46 smithi026 ceph-mon[68309]: pgmap v443: 17 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 14 active+clean; 6.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s wr, 128 op/s 2024-03-20T08:59:47.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:46 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/610353236' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T08:59:47.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:46 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T08:59:47.298 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T08:59:47.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:47.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T08:59:47.403 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T08:59:47.440 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 83 16 113396 113380 85.3646 50.5625 0.0062378 0.0117054 2024-03-20T08:59:47.449 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:47.682 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T08:59:47.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:47.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T08:59:47.788 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T08:59:47.860 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:47.976 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:47 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:59:47.976 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:47 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:59:47.976 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:47 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T08:59:47.976 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:47 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T08:59:47.976 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:47 smithi026 ceph-mon[68309]: osdmap e149: 12 total, 12 up, 12 in 2024-03-20T08:59:47.976 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:47 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:59:47.976 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:47 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:59:47.976 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:47 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:59:47.976 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:47 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:59:47.976 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:47 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:59:47.977 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:47 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T08:59:47.977 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:47 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T08:59:48.158 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:47 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:59:48.158 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:47 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:59:48.158 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:47 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T08:59:48.158 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:47 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T08:59:48.158 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:47 smithi115 ceph-mon[72476]: osdmap e149: 12 total, 12 up, 12 in 2024-03-20T08:59:48.158 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:47 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:59:48.158 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:47 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:59:48.158 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:47 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:59:48.158 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:47 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:59:48.158 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:47 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:59:48.158 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:47 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T08:59:48.159 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:47 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T08:59:48.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:47 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:59:48.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:47 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:59:48.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:47 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T08:59:48.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:47 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T08:59:48.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:47 smithi122 ceph-mon[73495]: osdmap e149: 12 total, 12 up, 12 in 2024-03-20T08:59:48.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:47 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:59:48.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:47 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T08:59:48.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:47 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T08:59:48.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:47 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T08:59:48.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:47 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T08:59:48.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:47 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T08:59:48.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:47 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T08:59:48.244 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T08:59:48.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:48.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T08:59:48.356 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T08:59:48.390 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:48.411 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T08:59:48.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:48.414 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T08:59:48.440 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 84 16 114182 114166 84.9332 49.125 0.0152084 0.0117665 2024-03-20T08:59:48.515 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T08:59:48.551 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:48.572 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T08:59:48.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:48.575 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T08:59:48.676 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T08:59:48.735 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:48.915 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T08:59:48.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:48.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T08:59:49.028 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T08:59:49.058 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:49.066 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T08:59:49.066 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T08:59:49.086 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T08:59:49.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:49.091 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T08:59:49.192 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T08:59:49.226 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:49.325 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T08:59:49.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:49.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T08:59:49.374 INFO:teuthology.orchestra.run.smithi026.stderr:no valid command found; 10 closest matches: 2024-03-20T08:59:49.375 INFO:teuthology.orchestra.run.smithi026.stderr:0 2024-03-20T08:59:49.375 INFO:teuthology.orchestra.run.smithi026.stderr:1 2024-03-20T08:59:49.375 INFO:teuthology.orchestra.run.smithi026.stderr:2 2024-03-20T08:59:49.375 INFO:teuthology.orchestra.run.smithi026.stderr:abort 2024-03-20T08:59:49.375 INFO:teuthology.orchestra.run.smithi026.stderr:assert 2024-03-20T08:59:49.375 INFO:teuthology.orchestra.run.smithi026.stderr:bench [] [] [] [] 2024-03-20T08:59:49.375 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs debug_inject_read_zeros 2024-03-20T08:59:49.375 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs files list 2024-03-20T08:59:49.375 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs stats 2024-03-20T08:59:49.375 INFO:teuthology.orchestra.run.smithi026.stderr:bluestore allocator dump block 2024-03-20T08:59:49.375 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: invalid command 2024-03-20T08:59:49.429 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T08:59:49.441 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 85 16 114982 114966 84.5222 50 0.0225432 0.0118265 2024-03-20T08:59:49.461 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:49.517 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T08:59:49.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:49.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T08:59:49.624 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T08:59:49.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:49 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:59:49.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:49 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:59:49.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:49 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T08:59:49.651 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:49.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:49 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:59:49.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:49 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:59:49.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:49 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T08:59:49.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:49 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:59:49.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:49 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:59:49.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:49 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T08:59:49.846 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-20T08:59:49.848 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T08:59:49.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:49.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T08:59:49.924 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T08:59:49.952 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T08:59:49.973 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T08:59:50.007 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:50.035 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T08:59:50.090 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T08:59:50.124 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T08:59:50.161 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:59:50.173 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T08:59:50.222 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T08:59:50.245 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T08:59:50.252 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T08:59:50.290 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T08:59:50.331 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T08:59:50.385 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T08:59:50.392 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T08:59:50.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:50.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T08:59:50.406 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T08:59:50.407 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-20T08:59:50.407 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.7 'trim stale osdmaps' 2024-03-20T08:59:50.440 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 86 16 115786 115770 84.1236 50.25 0.0050993 0.0118766 2024-03-20T08:59:50.452 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T08:59:50.452 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T08:59:50.452 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T08:59:50.452 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T08:59:50.452 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T08:59:50.452 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T08:59:50.453 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T08:59:50.453 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T08:59:50.453 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T08:59:50.454 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T08:59:50.454 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T08:59:50.454 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T08:59:50.454 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T08:59:50.497 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T08:59:50.588 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:50.588 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:50 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:59:50.588 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:50 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:59:50.588 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:50 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:59:50.588 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:50 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T08:59:50.588 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:50 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/723091862' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T08:59:50.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:50 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:59:50.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:50 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:59:50.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:50 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:59:50.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:50 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T08:59:50.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:50 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/723091862' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T08:59:50.731 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T08:59:50.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:50.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T08:59:50.746 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:50 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T08:59:50.746 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:50 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T08:59:50.746 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:50 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T08:59:50.746 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:50 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T08:59:50.746 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:50 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/723091862' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T08:59:50.837 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T08:59:50.909 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:51.171 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T08:59:51.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:51.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T08:59:51.251 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:59:51.275 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T08:59:51.300 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:59:51.354 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:51.422 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:51 smithi026 ceph-mon[68309]: pgmap v446: 17 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 14 active+clean; 6.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s wr, 451 op/s 2024-03-20T08:59:51.441 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 87 16 116342 116326 83.5561 34.75 0.00651994 0.0119585 2024-03-20T08:59:51.571 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T08:59:51.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:51.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T08:59:51.577 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:51 smithi115 ceph-mon[72476]: pgmap v446: 17 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 14 active+clean; 6.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s wr, 451 op/s 2024-03-20T08:59:51.675 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T08:59:51.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:51 smithi122 ceph-mon[73495]: pgmap v446: 17 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 14 active+clean; 6.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s wr, 451 op/s 2024-03-20T08:59:51.707 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:51.738 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T08:59:51.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:51.741 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T08:59:51.842 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T08:59:51.882 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:51.906 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T08:59:51.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:51.909 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T08:59:52.010 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T08:59:52.042 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:52.224 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T08:59:52.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:52.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T08:59:52.274 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T08:59:52.274 INFO:teuthology.orchestra.run.smithi115.stdout: "size": 20, 2024-03-20T08:59:52.274 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 600, 2024-03-20T08:59:52.274 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [ 2024-03-20T08:59:52.274 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.274 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:41692 2.7 2:e8fd97ed:::benchmark_data_smithi122_107654_object41691:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-20T08:59:52.274 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:58:39.733162+0000", 2024-03-20T08:59:52.275 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 72.534265130999998, 2024-03-20T08:59:52.275 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.060791761999999999, 2024-03-20T08:59:52.275 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:52.275 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:52.275 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T08:59:52.275 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T08:59:52.275 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:59:52.275 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 41692 2024-03-20T08:59:52.275 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.275 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:52.275 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.275 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:52.276 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.733162+0000", 2024-03-20T08:59:52.276 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.276 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.276 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.276 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:52.276 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.733162+0000", 2024-03-20T08:59:52.276 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.276 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.276 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.276 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:52.276 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.733163+0000", 2024-03-20T08:59:52.276 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.435e-06 2024-03-20T08:59:52.276 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.277 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.277 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:52.277 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.733217+0000", 2024-03-20T08:59:52.277 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3427999999999997e-05 2024-03-20T08:59:52.277 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.277 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.277 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:52.277 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.733231+0000", 2024-03-20T08:59:52.277 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.45e-05 2024-03-20T08:59:52.277 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.277 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.277 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:52.277 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.733240+0000", 2024-03-20T08:59:52.277 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5129999999999996e-06 2024-03-20T08:59:52.278 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.278 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.278 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:52.278 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.733302+0000", 2024-03-20T08:59:52.278 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2216000000000002e-05 2024-03-20T08:59:52.278 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.278 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.278 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:52.278 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.733421+0000", 2024-03-20T08:59:52.279 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000119079 2024-03-20T08:59:52.279 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.279 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.279 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T08:59:52.279 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.733562+0000", 2024-03-20T08:59:52.279 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000141147 2024-03-20T08:59:52.279 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.279 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.279 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T08:59:52.279 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.734946+0000", 2024-03-20T08:59:52.280 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001384019 2024-03-20T08:59:52.280 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.280 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.280 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.280 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.735396+0000", 2024-03-20T08:59:52.280 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00044978999999999999 2024-03-20T08:59:52.280 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.280 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.280 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.280 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.793856+0000", 2024-03-20T08:59:52.280 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.058459839 2024-03-20T08:59:52.280 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.280 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.281 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:52.281 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.793902+0000", 2024-03-20T08:59:52.281 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6652e-05 2024-03-20T08:59:52.281 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.281 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.281 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:52.281 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.793953+0000", 2024-03-20T08:59:52.281 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1143999999999997e-05 2024-03-20T08:59:52.281 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.281 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:52.281 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.281 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.281 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.281 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:41702 2.7 2:e9cc37ef:::benchmark_data_smithi122_107654_object41701:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-20T08:59:52.282 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:58:39.736630+0000", 2024-03-20T08:59:52.282 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 72.530796910999996, 2024-03-20T08:59:52.282 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.058660687000000003, 2024-03-20T08:59:52.282 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:52.282 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:52.282 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T08:59:52.282 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T08:59:52.282 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:59:52.282 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 41702 2024-03-20T08:59:52.282 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.282 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:52.282 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.282 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:52.283 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.736630+0000", 2024-03-20T08:59:52.283 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.283 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.283 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.283 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:52.283 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.736630+0000", 2024-03-20T08:59:52.283 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.283 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.283 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.283 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:52.283 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.736631+0000", 2024-03-20T08:59:52.283 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.251e-06 2024-03-20T08:59:52.283 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.283 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.284 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:52.284 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.736694+0000", 2024-03-20T08:59:52.284 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.3272e-05 2024-03-20T08:59:52.284 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.284 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.284 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:52.284 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.736705+0000", 2024-03-20T08:59:52.284 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0159999999999999e-05 2024-03-20T08:59:52.284 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.284 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.285 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:52.285 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.736710+0000", 2024-03-20T08:59:52.285 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7509999999999997e-06 2024-03-20T08:59:52.285 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.285 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.285 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:52.285 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.736758+0000", 2024-03-20T08:59:52.285 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.7788999999999999e-05 2024-03-20T08:59:52.285 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.285 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.285 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:52.285 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.736848+0000", 2024-03-20T08:59:52.285 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.9874e-05 2024-03-20T08:59:52.285 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.285 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.286 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T08:59:52.286 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.736948+0000", 2024-03-20T08:59:52.286 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000100492 2024-03-20T08:59:52.286 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.286 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.286 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T08:59:52.286 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.738024+0000", 2024-03-20T08:59:52.286 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00107554 2024-03-20T08:59:52.286 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.286 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.286 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.286 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.738459+0000", 2024-03-20T08:59:52.286 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00043516799999999999 2024-03-20T08:59:52.287 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.287 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.287 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.287 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.795216+0000", 2024-03-20T08:59:52.287 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.056756661999999999 2024-03-20T08:59:52.287 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.287 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.287 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:52.287 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.795250+0000", 2024-03-20T08:59:52.287 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.4208999999999997e-05 2024-03-20T08:59:52.287 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.287 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.287 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:52.287 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.795291+0000", 2024-03-20T08:59:52.288 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.0519000000000003e-05 2024-03-20T08:59:52.288 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.288 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:52.288 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.288 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.288 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.288 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:53325 2.7 2:e50bf810:::benchmark_data_smithi122_107654_object53324:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-20T08:59:52.288 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:58:45.740926+0000", 2024-03-20T08:59:52.288 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 66.526500748999993, 2024-03-20T08:59:52.288 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.20845012600000001, 2024-03-20T08:59:52.288 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:52.288 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:52.289 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T08:59:52.289 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T08:59:52.289 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:59:52.289 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 53325 2024-03-20T08:59:52.289 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.289 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:52.289 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.289 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:52.289 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.740926+0000", 2024-03-20T08:59:52.290 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.290 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.290 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.290 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:52.290 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.740926+0000", 2024-03-20T08:59:52.290 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.290 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.291 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.291 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:52.291 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.740928+0000", 2024-03-20T08:59:52.291 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5519999999999999e-06 2024-03-20T08:59:52.291 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.291 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.291 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:52.291 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.740984+0000", 2024-03-20T08:59:52.291 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5915000000000003e-05 2024-03-20T08:59:52.291 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.291 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.291 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:52.291 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.740998+0000", 2024-03-20T08:59:52.292 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4769e-05 2024-03-20T08:59:52.292 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.292 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.292 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:52.292 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.741007+0000", 2024-03-20T08:59:52.292 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.6149999999999997e-06 2024-03-20T08:59:52.292 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.292 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.292 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:52.292 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.741078+0000", 2024-03-20T08:59:52.292 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.1155999999999994e-05 2024-03-20T08:59:52.292 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.292 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.293 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:52.293 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.741201+0000", 2024-03-20T08:59:52.293 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012243700000000001 2024-03-20T08:59:52.293 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.293 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.293 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T08:59:52.293 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.741344+0000", 2024-03-20T08:59:52.293 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00014315000000000001 2024-03-20T08:59:52.293 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.293 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.293 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T08:59:52.293 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.742780+0000", 2024-03-20T08:59:52.293 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0014363760000000001 2024-03-20T08:59:52.293 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.294 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.294 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.294 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.743656+0000", 2024-03-20T08:59:52.294 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000876307 2024-03-20T08:59:52.294 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.294 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.294 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.294 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.949309+0000", 2024-03-20T08:59:52.294 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.205652374 2024-03-20T08:59:52.294 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.294 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.294 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:52.294 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.949347+0000", 2024-03-20T08:59:52.295 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7993e-05 2024-03-20T08:59:52.295 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.295 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.295 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:52.295 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.949376+0000", 2024-03-20T08:59:52.295 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9482e-05 2024-03-20T08:59:52.296 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.296 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:52.296 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.296 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.296 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.296 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:53361 2.7 2:e45d35a1:::benchmark_data_smithi122_107654_object53360:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-20T08:59:52.296 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:58:45.759219+0000", 2024-03-20T08:59:52.296 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 66.508207862000006, 2024-03-20T08:59:52.297 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.19152682400000001, 2024-03-20T08:59:52.297 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:52.297 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:52.297 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T08:59:52.297 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T08:59:52.297 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:59:52.297 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 53361 2024-03-20T08:59:52.297 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.297 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:52.297 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.297 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:52.297 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.759219+0000", 2024-03-20T08:59:52.298 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.298 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.298 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.298 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:52.298 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.759219+0000", 2024-03-20T08:59:52.298 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.298 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.299 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.299 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:52.299 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.759221+0000", 2024-03-20T08:59:52.299 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7880000000000001e-06 2024-03-20T08:59:52.299 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.299 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.299 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:52.299 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.759272+0000", 2024-03-20T08:59:52.299 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0980000000000003e-05 2024-03-20T08:59:52.300 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.300 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.300 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:52.300 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.759288+0000", 2024-03-20T08:59:52.300 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5879000000000002e-05 2024-03-20T08:59:52.300 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.300 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.300 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:52.300 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.759295+0000", 2024-03-20T08:59:52.300 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.0589999999999997e-06 2024-03-20T08:59:52.301 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.301 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.301 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:52.301 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.759345+0000", 2024-03-20T08:59:52.301 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0821e-05 2024-03-20T08:59:52.301 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.301 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.301 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:52.301 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.759444+0000", 2024-03-20T08:59:52.302 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.8855000000000001e-05 2024-03-20T08:59:52.302 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.302 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.302 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T08:59:52.302 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.759536+0000", 2024-03-20T08:59:52.302 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.1990000000000005e-05 2024-03-20T08:59:52.302 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.303 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.303 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T08:59:52.303 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.761886+0000", 2024-03-20T08:59:52.303 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0023491950000000001 2024-03-20T08:59:52.303 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.303 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.303 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.303 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.765278+0000", 2024-03-20T08:59:52.304 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0033926640000000001 2024-03-20T08:59:52.304 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.304 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.304 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.304 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.950640+0000", 2024-03-20T08:59:52.304 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.18536202800000001 2024-03-20T08:59:52.304 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.304 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.304 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:52.304 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.950698+0000", 2024-03-20T08:59:52.304 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7547000000000003e-05 2024-03-20T08:59:52.304 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.305 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.305 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:52.305 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.950746+0000", 2024-03-20T08:59:52.305 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8018e-05 2024-03-20T08:59:52.305 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.305 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:52.305 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.305 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.305 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.305 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:53363 2.7 2:e993eef9:::benchmark_data_smithi122_107654_object53362:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-20T08:59:52.305 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:58:45.760351+0000", 2024-03-20T08:59:52.306 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 66.507075345999993, 2024-03-20T08:59:52.306 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.190487987, 2024-03-20T08:59:52.306 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:52.306 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:52.306 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T08:59:52.306 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T08:59:52.306 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:59:52.306 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 53363 2024-03-20T08:59:52.306 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.306 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:52.306 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.306 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:52.307 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.760351+0000", 2024-03-20T08:59:52.307 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.307 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.307 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.307 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:52.307 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.760351+0000", 2024-03-20T08:59:52.307 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.307 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.307 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.307 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:52.307 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.760352+0000", 2024-03-20T08:59:52.307 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.1299999999999998e-07 2024-03-20T08:59:52.307 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.307 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.308 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:52.308 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.760428+0000", 2024-03-20T08:59:52.308 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.5286999999999999e-05 2024-03-20T08:59:52.308 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.308 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.308 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:52.308 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.760441+0000", 2024-03-20T08:59:52.308 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2860000000000001e-05 2024-03-20T08:59:52.308 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.308 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.308 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:52.308 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.760448+0000", 2024-03-20T08:59:52.308 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.8189999999999992e-06 2024-03-20T08:59:52.308 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.309 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.309 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:52.309 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.760781+0000", 2024-03-20T08:59:52.309 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00033301199999999999 2024-03-20T08:59:52.309 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.309 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.309 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:52.309 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.760901+0000", 2024-03-20T08:59:52.309 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000119924 2024-03-20T08:59:52.309 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.309 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.309 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T08:59:52.309 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.761081+0000", 2024-03-20T08:59:52.309 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00018010500000000001 2024-03-20T08:59:52.309 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.310 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.310 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T08:59:52.310 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.761987+0000", 2024-03-20T08:59:52.310 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00090566900000000001 2024-03-20T08:59:52.310 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.310 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.310 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.310 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.765325+0000", 2024-03-20T08:59:52.310 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0033378000000000001 2024-03-20T08:59:52.310 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.310 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.310 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.311 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.950778+0000", 2024-03-20T08:59:52.311 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.18545304200000001 2024-03-20T08:59:52.311 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.311 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.311 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:52.311 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.950813+0000", 2024-03-20T08:59:52.311 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.472e-05 2024-03-20T08:59:52.311 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.311 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.311 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:52.311 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.950839+0000", 2024-03-20T08:59:52.311 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6835999999999999e-05 2024-03-20T08:59:52.311 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.311 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:52.311 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.312 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.312 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.312 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:53419 2.7 2:e710df74:::benchmark_data_smithi122_107654_object53418:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-20T08:59:52.312 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:58:45.791591+0000", 2024-03-20T08:59:52.312 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 66.475835936999999, 2024-03-20T08:59:52.312 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.15939167900000001, 2024-03-20T08:59:52.312 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:52.312 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:52.312 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T08:59:52.312 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T08:59:52.312 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:59:52.312 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 53419 2024-03-20T08:59:52.312 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.312 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:52.313 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.313 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:52.313 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.791591+0000", 2024-03-20T08:59:52.313 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.313 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.313 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.313 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:52.313 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.791591+0000", 2024-03-20T08:59:52.313 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.313 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.313 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.313 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:52.313 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.791592+0000", 2024-03-20T08:59:52.313 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2389999999999999e-06 2024-03-20T08:59:52.313 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.314 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.314 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:52.314 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.791637+0000", 2024-03-20T08:59:52.314 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4650000000000001e-05 2024-03-20T08:59:52.314 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.314 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.314 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:52.314 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.791651+0000", 2024-03-20T08:59:52.314 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4423999999999999e-05 2024-03-20T08:59:52.314 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.314 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.315 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:52.315 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.791660+0000", 2024-03-20T08:59:52.315 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.8259999999999999e-06 2024-03-20T08:59:52.315 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.315 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.315 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:52.315 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.791718+0000", 2024-03-20T08:59:52.315 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8168999999999998e-05 2024-03-20T08:59:52.315 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.315 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.315 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:52.315 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.791856+0000", 2024-03-20T08:59:52.315 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013817199999999999 2024-03-20T08:59:52.315 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.316 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.316 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T08:59:52.316 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.792032+0000", 2024-03-20T08:59:52.316 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00017524000000000001 2024-03-20T08:59:52.316 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.316 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.316 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T08:59:52.316 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.793458+0000", 2024-03-20T08:59:52.316 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0014260010000000001 2024-03-20T08:59:52.316 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.317 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.317 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.317 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.795766+0000", 2024-03-20T08:59:52.317 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0023082340000000002 2024-03-20T08:59:52.317 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.317 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.318 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.318 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.950879+0000", 2024-03-20T08:59:52.318 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.15511298200000001 2024-03-20T08:59:52.318 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.319 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.319 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:52.319 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.950933+0000", 2024-03-20T08:59:52.319 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4591e-05 2024-03-20T08:59:52.319 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.319 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.319 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:52.320 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.950983+0000", 2024-03-20T08:59:52.320 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9150999999999997e-05 2024-03-20T08:59:52.320 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.320 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:52.320 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.320 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.320 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.320 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:53451 2.7 2:ecf06e77:::benchmark_data_smithi122_107654_object53450:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-20T08:59:52.320 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:58:45.807989+0000", 2024-03-20T08:59:52.320 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 66.459437703999996, 2024-03-20T08:59:52.320 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.14417972800000001, 2024-03-20T08:59:52.320 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:52.320 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:52.320 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T08:59:52.321 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T08:59:52.321 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:59:52.321 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 53451 2024-03-20T08:59:52.321 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.321 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:52.321 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.321 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:52.321 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.807989+0000", 2024-03-20T08:59:52.321 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.321 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.321 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.321 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:52.321 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.807989+0000", 2024-03-20T08:59:52.321 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.322 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.322 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.322 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:52.322 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.807991+0000", 2024-03-20T08:59:52.322 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8369999999999999e-06 2024-03-20T08:59:52.322 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.322 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.322 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:52.322 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.808031+0000", 2024-03-20T08:59:52.322 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.9857999999999997e-05 2024-03-20T08:59:52.323 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.323 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.323 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:52.323 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.808050+0000", 2024-03-20T08:59:52.323 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9086e-05 2024-03-20T08:59:52.323 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.323 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.323 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:52.323 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.808060+0000", 2024-03-20T08:59:52.324 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.9399999999999997e-06 2024-03-20T08:59:52.324 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.324 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.324 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:52.324 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.808126+0000", 2024-03-20T08:59:52.324 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.5685999999999994e-05 2024-03-20T08:59:52.324 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.324 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.325 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:52.325 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.808275+0000", 2024-03-20T08:59:52.325 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00014987 2024-03-20T08:59:52.325 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.325 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.325 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T08:59:52.325 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.808415+0000", 2024-03-20T08:59:52.325 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013999699999999999 2024-03-20T08:59:52.326 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.326 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.326 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T08:59:52.326 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.810284+0000", 2024-03-20T08:59:52.326 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001868672 2024-03-20T08:59:52.326 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.326 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.326 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.326 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.811950+0000", 2024-03-20T08:59:52.327 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0016661930000000001 2024-03-20T08:59:52.327 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.327 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.327 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.327 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.952104+0000", 2024-03-20T08:59:52.327 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.140153905 2024-03-20T08:59:52.327 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.327 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.328 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:52.328 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.952143+0000", 2024-03-20T08:59:52.328 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.8816000000000002e-05 2024-03-20T08:59:52.328 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.328 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.328 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:52.328 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.952169+0000", 2024-03-20T08:59:52.328 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5868e-05 2024-03-20T08:59:52.328 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.329 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:52.329 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.329 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.329 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.329 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:53480 2.7 2:e99370e5:::benchmark_data_smithi122_107654_object53479:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-20T08:59:52.329 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:58:45.825869+0000", 2024-03-20T08:59:52.329 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 66.441557566, 2024-03-20T08:59:52.329 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.126442162, 2024-03-20T08:59:52.329 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:52.330 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:52.330 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T08:59:52.330 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T08:59:52.330 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:59:52.330 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 53480 2024-03-20T08:59:52.330 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.330 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:52.330 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.331 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:52.331 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.825869+0000", 2024-03-20T08:59:52.331 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.331 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.331 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.331 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:52.331 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.825869+0000", 2024-03-20T08:59:52.331 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.332 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.332 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.332 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:52.332 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.825872+0000", 2024-03-20T08:59:52.332 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3369999999999998e-06 2024-03-20T08:59:52.332 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.332 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.332 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:52.332 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.825916+0000", 2024-03-20T08:59:52.333 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4693000000000001e-05 2024-03-20T08:59:52.333 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.333 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.333 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:52.333 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.825938+0000", 2024-03-20T08:59:52.333 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1393999999999999e-05 2024-03-20T08:59:52.333 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.333 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.333 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:52.334 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.825948+0000", 2024-03-20T08:59:52.334 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0539e-05 2024-03-20T08:59:52.334 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.334 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.334 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:52.334 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.826015+0000", 2024-03-20T08:59:52.334 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.6328999999999995e-05 2024-03-20T08:59:52.334 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.335 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.335 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:52.335 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.826123+0000", 2024-03-20T08:59:52.335 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000108646 2024-03-20T08:59:52.335 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.335 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.335 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T08:59:52.335 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.826240+0000", 2024-03-20T08:59:52.335 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000116766 2024-03-20T08:59:52.336 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.336 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.336 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.336 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.831367+0000", 2024-03-20T08:59:52.336 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0051273869999999997 2024-03-20T08:59:52.336 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.336 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.336 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T08:59:52.336 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.832945+0000", 2024-03-20T08:59:52.337 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00157763 2024-03-20T08:59:52.337 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.337 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.337 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.337 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.952214+0000", 2024-03-20T08:59:52.337 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.11926896100000001 2024-03-20T08:59:52.337 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.337 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.337 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:52.338 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.952270+0000", 2024-03-20T08:59:52.338 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5899000000000001e-05 2024-03-20T08:59:52.338 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.338 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.338 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:52.338 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.952311+0000", 2024-03-20T08:59:52.338 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.1581000000000001e-05 2024-03-20T08:59:52.338 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.339 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:52.339 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.339 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.339 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.339 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:53481 2.7 2:eab72ea7:::benchmark_data_smithi122_107654_object53480:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-20T08:59:52.339 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:58:45.826029+0000", 2024-03-20T08:59:52.339 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 66.441397988999995, 2024-03-20T08:59:52.339 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.12836041500000001, 2024-03-20T08:59:52.339 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:52.339 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:52.339 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T08:59:52.339 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T08:59:52.339 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:59:52.339 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 53481 2024-03-20T08:59:52.340 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.340 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:52.340 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.340 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:52.340 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.826029+0000", 2024-03-20T08:59:52.340 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.340 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.340 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.340 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:52.340 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.826029+0000", 2024-03-20T08:59:52.340 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.340 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.340 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.340 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:52.340 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.826030+0000", 2024-03-20T08:59:52.341 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.0500000000000002e-07 2024-03-20T08:59:52.341 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.341 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.341 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:52.341 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.826102+0000", 2024-03-20T08:59:52.341 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.2596999999999996e-05 2024-03-20T08:59:52.341 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.341 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.341 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:52.341 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.826112+0000", 2024-03-20T08:59:52.341 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.9769999999999995e-06 2024-03-20T08:59:52.341 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.341 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.341 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:52.341 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.826120+0000", 2024-03-20T08:59:52.342 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3749999999999997e-06 2024-03-20T08:59:52.342 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.342 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.342 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:52.342 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.827847+0000", 2024-03-20T08:59:52.342 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001727606 2024-03-20T08:59:52.342 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.342 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.342 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:52.343 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.827972+0000", 2024-03-20T08:59:52.343 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012518699999999999 2024-03-20T08:59:52.343 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.343 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.343 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T08:59:52.343 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.828059+0000", 2024-03-20T08:59:52.343 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.6706999999999995e-05 2024-03-20T08:59:52.343 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.344 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.344 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.344 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.831411+0000", 2024-03-20T08:59:52.344 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0033521100000000002 2024-03-20T08:59:52.344 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.344 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.345 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T08:59:52.345 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.832951+0000", 2024-03-20T08:59:52.345 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001539522 2024-03-20T08:59:52.345 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.345 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.345 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.345 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.954317+0000", 2024-03-20T08:59:52.346 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.121365893 2024-03-20T08:59:52.346 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.346 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.346 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:52.346 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.954359+0000", 2024-03-20T08:59:52.346 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.2688999999999997e-05 2024-03-20T08:59:52.346 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.347 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.347 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:52.347 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.954389+0000", 2024-03-20T08:59:52.347 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9847000000000001e-05 2024-03-20T08:59:52.347 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.347 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:52.347 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.348 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.348 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.348 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:53483 2.7 2:e28a92ca:::benchmark_data_smithi122_107654_object53482:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-20T08:59:52.348 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:58:45.826502+0000", 2024-03-20T08:59:52.348 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 66.440925140999994, 2024-03-20T08:59:52.348 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.12796293, 2024-03-20T08:59:52.348 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:52.349 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:52.349 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T08:59:52.349 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T08:59:52.349 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:59:52.349 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 53483 2024-03-20T08:59:52.349 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.349 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:52.350 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.350 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:52.350 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.826502+0000", 2024-03-20T08:59:52.350 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.350 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.350 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.350 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:52.351 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.826502+0000", 2024-03-20T08:59:52.351 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.351 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.351 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.351 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:52.351 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.826502+0000", 2024-03-20T08:59:52.352 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.7899999999999998e-07 2024-03-20T08:59:52.352 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.352 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.352 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:52.352 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.826682+0000", 2024-03-20T08:59:52.352 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00017939599999999999 2024-03-20T08:59:52.352 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.352 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.352 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:52.352 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.826693+0000", 2024-03-20T08:59:52.352 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1296e-05 2024-03-20T08:59:52.352 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.352 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.353 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:52.353 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.826701+0000", 2024-03-20T08:59:52.353 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.8739999999999995e-06 2024-03-20T08:59:52.353 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.353 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.353 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:52.353 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.829703+0000", 2024-03-20T08:59:52.353 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.003002191 2024-03-20T08:59:52.353 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.353 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.353 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:52.353 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.829813+0000", 2024-03-20T08:59:52.353 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000109737 2024-03-20T08:59:52.353 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.354 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.354 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T08:59:52.354 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.829935+0000", 2024-03-20T08:59:52.354 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000121883 2024-03-20T08:59:52.354 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.354 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.354 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.354 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.831458+0000", 2024-03-20T08:59:52.354 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0015230280000000001 2024-03-20T08:59:52.354 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.354 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.354 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T08:59:52.354 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.832956+0000", 2024-03-20T08:59:52.355 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0014977530000000001 2024-03-20T08:59:52.355 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.355 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.355 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.355 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.954414+0000", 2024-03-20T08:59:52.355 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.121458809 2024-03-20T08:59:52.355 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.355 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.355 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:52.355 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.954443+0000", 2024-03-20T08:59:52.355 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9142e-05 2024-03-20T08:59:52.355 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.355 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.355 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:52.356 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.954465+0000", 2024-03-20T08:59:52.356 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1141999999999998e-05 2024-03-20T08:59:52.356 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.356 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:52.356 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.356 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.356 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.356 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:53490 2.7 2:e474dc8c:::benchmark_data_smithi122_107654_object53489:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-20T08:59:52.356 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:58:45.829205+0000", 2024-03-20T08:59:52.356 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 66.438222080000003, 2024-03-20T08:59:52.356 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.13008857400000001, 2024-03-20T08:59:52.356 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:52.357 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:52.357 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T08:59:52.357 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T08:59:52.357 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:59:52.357 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 53490 2024-03-20T08:59:52.357 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.357 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:52.357 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.357 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:52.357 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.829205+0000", 2024-03-20T08:59:52.357 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.357 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.358 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.358 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:52.358 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.829205+0000", 2024-03-20T08:59:52.358 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.358 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.358 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.358 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:52.358 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.829206+0000", 2024-03-20T08:59:52.358 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2780000000000001e-06 2024-03-20T08:59:52.358 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.358 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.358 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:52.358 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.829234+0000", 2024-03-20T08:59:52.358 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.7988999999999999e-05 2024-03-20T08:59:52.359 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.359 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.359 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:52.359 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.829250+0000", 2024-03-20T08:59:52.359 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6337999999999998e-05 2024-03-20T08:59:52.359 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.359 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.359 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:52.359 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.829258+0000", 2024-03-20T08:59:52.359 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7829999999999999e-06 2024-03-20T08:59:52.359 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.359 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.359 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:52.360 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.831499+0000", 2024-03-20T08:59:52.360 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0022408480000000001 2024-03-20T08:59:52.360 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.360 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.360 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:52.360 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.831635+0000", 2024-03-20T08:59:52.360 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013605 2024-03-20T08:59:52.360 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.360 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.360 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T08:59:52.360 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.831788+0000", 2024-03-20T08:59:52.360 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00015283500000000001 2024-03-20T08:59:52.360 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.360 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.360 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T08:59:52.361 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.833143+0000", 2024-03-20T08:59:52.361 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001355081 2024-03-20T08:59:52.361 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.361 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.361 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.361 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.834962+0000", 2024-03-20T08:59:52.361 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0018187329999999999 2024-03-20T08:59:52.361 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.361 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.361 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.361 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.959164+0000", 2024-03-20T08:59:52.361 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.124202327 2024-03-20T08:59:52.361 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.362 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.362 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:52.362 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.959225+0000", 2024-03-20T08:59:52.362 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1253999999999993e-05 2024-03-20T08:59:52.362 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.362 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.362 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:52.362 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.959293+0000", 2024-03-20T08:59:52.362 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.8058000000000005e-05 2024-03-20T08:59:52.363 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.363 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:52.363 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.363 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.363 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.363 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:53497 2.7 2:e0729c41:::benchmark_data_smithi122_107654_object53496:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-20T08:59:52.363 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:58:45.833199+0000", 2024-03-20T08:59:52.363 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 66.434227325999998, 2024-03-20T08:59:52.363 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.1262344, 2024-03-20T08:59:52.363 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:52.363 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:52.363 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T08:59:52.363 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T08:59:52.363 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:59:52.364 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 53497 2024-03-20T08:59:52.364 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.364 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:52.364 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.364 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:52.364 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.833199+0000", 2024-03-20T08:59:52.364 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.364 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.364 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.364 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:52.364 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.833199+0000", 2024-03-20T08:59:52.364 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.364 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.365 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.365 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:52.365 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.833201+0000", 2024-03-20T08:59:52.365 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.029e-06 2024-03-20T08:59:52.365 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.365 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.365 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:52.365 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.833277+0000", 2024-03-20T08:59:52.365 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.6118000000000005e-05 2024-03-20T08:59:52.365 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.365 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.365 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:52.365 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.833287+0000", 2024-03-20T08:59:52.365 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0753e-05 2024-03-20T08:59:52.365 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.366 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.366 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:52.366 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.833293+0000", 2024-03-20T08:59:52.366 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0299999999999999e-06 2024-03-20T08:59:52.366 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.366 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.366 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:52.366 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.833339+0000", 2024-03-20T08:59:52.366 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5467999999999999e-05 2024-03-20T08:59:52.366 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.366 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.366 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:52.367 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.833460+0000", 2024-03-20T08:59:52.367 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000121144 2024-03-20T08:59:52.367 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.367 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.367 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T08:59:52.367 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.833610+0000", 2024-03-20T08:59:52.367 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000150078 2024-03-20T08:59:52.367 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.367 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.368 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T08:59:52.368 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.835042+0000", 2024-03-20T08:59:52.368 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0014316140000000001 2024-03-20T08:59:52.368 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.368 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.368 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.368 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.835258+0000", 2024-03-20T08:59:52.368 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00021614099999999999 2024-03-20T08:59:52.368 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.368 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.368 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.368 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.959339+0000", 2024-03-20T08:59:52.368 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.12408108 2024-03-20T08:59:52.368 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.368 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.369 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:52.369 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.959394+0000", 2024-03-20T08:59:52.369 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5077999999999997e-05 2024-03-20T08:59:52.369 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.369 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.369 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:52.369 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.959434+0000", 2024-03-20T08:59:52.369 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.9867000000000003e-05 2024-03-20T08:59:52.369 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.369 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:52.369 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.369 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.369 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.369 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:53515 2.7 2:eb083dc9:::benchmark_data_smithi122_107654_object53514:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-20T08:59:52.370 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:58:45.846705+0000", 2024-03-20T08:59:52.370 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 66.420721942, 2024-03-20T08:59:52.370 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.112886466, 2024-03-20T08:59:52.370 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:52.370 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:52.370 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T08:59:52.370 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T08:59:52.370 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:59:52.370 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 53515 2024-03-20T08:59:52.370 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.370 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:52.370 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.370 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:52.371 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.846705+0000", 2024-03-20T08:59:52.371 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.371 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.371 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.371 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:52.371 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.846705+0000", 2024-03-20T08:59:52.371 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.371 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.371 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.371 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:52.371 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.846706+0000", 2024-03-20T08:59:52.371 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5510000000000001e-06 2024-03-20T08:59:52.371 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.371 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.372 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:52.372 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.846792+0000", 2024-03-20T08:59:52.372 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5709000000000002e-05 2024-03-20T08:59:52.372 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.372 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.372 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:52.372 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.846808+0000", 2024-03-20T08:59:52.372 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5855999999999999e-05 2024-03-20T08:59:52.372 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.372 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.373 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:52.373 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.846816+0000", 2024-03-20T08:59:52.373 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.593e-06 2024-03-20T08:59:52.373 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.373 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.373 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:52.373 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.846879+0000", 2024-03-20T08:59:52.373 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2949999999999999e-05 2024-03-20T08:59:52.373 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.373 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.373 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:52.373 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.847019+0000", 2024-03-20T08:59:52.373 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013999499999999999 2024-03-20T08:59:52.374 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.374 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.374 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T08:59:52.374 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.847170+0000", 2024-03-20T08:59:52.374 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00015141699999999999 2024-03-20T08:59:52.374 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.374 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.374 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T08:59:52.374 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.849020+0000", 2024-03-20T08:59:52.374 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001850078 2024-03-20T08:59:52.374 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.374 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.374 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.375 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.849103+0000", 2024-03-20T08:59:52.375 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.2665999999999998e-05 2024-03-20T08:59:52.375 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.375 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.375 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.375 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.959485+0000", 2024-03-20T08:59:52.375 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.110381998 2024-03-20T08:59:52.375 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.375 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.375 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:52.375 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.959544+0000", 2024-03-20T08:59:52.375 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9360000000000001e-05 2024-03-20T08:59:52.375 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.375 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.376 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:52.376 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.959591+0000", 2024-03-20T08:59:52.376 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.7293000000000003e-05 2024-03-20T08:59:52.376 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.376 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:52.376 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.376 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.376 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.376 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:53531 2.7 2:e510c7ab:::benchmark_data_smithi122_107654_object53530:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-20T08:59:52.376 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:58:45.856021+0000", 2024-03-20T08:59:52.376 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 66.411406192000001, 2024-03-20T08:59:52.376 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.103719694, 2024-03-20T08:59:52.376 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:52.376 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:52.377 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T08:59:52.377 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T08:59:52.377 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:59:52.377 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 53531 2024-03-20T08:59:52.377 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.377 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:52.377 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.377 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:52.377 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.856021+0000", 2024-03-20T08:59:52.377 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.377 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.377 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.377 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:52.377 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.856021+0000", 2024-03-20T08:59:52.377 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.378 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.378 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.378 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:52.378 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.856022+0000", 2024-03-20T08:59:52.378 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.139e-06 2024-03-20T08:59:52.378 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.378 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.379 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:52.379 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.856107+0000", 2024-03-20T08:59:52.379 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5086999999999994e-05 2024-03-20T08:59:52.379 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.379 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.379 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:52.379 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.856120+0000", 2024-03-20T08:59:52.380 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2857000000000001e-05 2024-03-20T08:59:52.380 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.380 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.380 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:52.380 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.856126+0000", 2024-03-20T08:59:52.380 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1330000000000004e-06 2024-03-20T08:59:52.380 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.380 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.380 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:52.380 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.856186+0000", 2024-03-20T08:59:52.380 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0346000000000001e-05 2024-03-20T08:59:52.380 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.380 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.381 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:52.381 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.856326+0000", 2024-03-20T08:59:52.381 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013999100000000001 2024-03-20T08:59:52.381 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.381 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.381 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T08:59:52.381 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.856432+0000", 2024-03-20T08:59:52.381 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000105714 2024-03-20T08:59:52.381 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.381 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.381 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T08:59:52.381 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.857775+0000", 2024-03-20T08:59:52.381 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0013426810000000001 2024-03-20T08:59:52.381 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.381 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.382 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.382 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.858605+0000", 2024-03-20T08:59:52.382 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00083076600000000003 2024-03-20T08:59:52.382 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.382 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.382 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.382 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.959657+0000", 2024-03-20T08:59:52.382 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.10105141099999999 2024-03-20T08:59:52.382 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.382 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.382 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:52.382 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.959705+0000", 2024-03-20T08:59:52.383 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8313999999999999e-05 2024-03-20T08:59:52.383 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.383 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.383 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:52.383 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.959740+0000", 2024-03-20T08:59:52.383 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.5255e-05 2024-03-20T08:59:52.383 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.383 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:52.383 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.383 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.383 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.383 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:53560 2.7 2:e40219b5:::benchmark_data_smithi122_107654_object53559:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-20T08:59:52.384 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:58:45.882597+0000", 2024-03-20T08:59:52.384 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 66.384830171000004, 2024-03-20T08:59:52.384 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.077277745999999994, 2024-03-20T08:59:52.384 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:52.384 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:52.384 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T08:59:52.384 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T08:59:52.384 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:59:52.384 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 53560 2024-03-20T08:59:52.384 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.384 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:52.384 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.384 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:52.385 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.882597+0000", 2024-03-20T08:59:52.385 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.385 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.385 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.385 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:52.385 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.882597+0000", 2024-03-20T08:59:52.385 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.385 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.385 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.385 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:52.385 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.882598+0000", 2024-03-20T08:59:52.385 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3239999999999999e-06 2024-03-20T08:59:52.385 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.385 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.386 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:52.386 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.882635+0000", 2024-03-20T08:59:52.386 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7320000000000002e-05 2024-03-20T08:59:52.386 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.386 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.386 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:52.386 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.882649+0000", 2024-03-20T08:59:52.386 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3332999999999999e-05 2024-03-20T08:59:52.386 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.386 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.386 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:52.386 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.882670+0000", 2024-03-20T08:59:52.386 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1049000000000001e-05 2024-03-20T08:59:52.387 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.387 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.387 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:52.387 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.882734+0000", 2024-03-20T08:59:52.387 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4554000000000006e-05 2024-03-20T08:59:52.387 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.387 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.387 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:52.387 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.882872+0000", 2024-03-20T08:59:52.387 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013735599999999999 2024-03-20T08:59:52.387 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.387 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.387 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T08:59:52.387 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.883030+0000", 2024-03-20T08:59:52.388 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000158286 2024-03-20T08:59:52.388 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.388 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.388 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T08:59:52.388 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.884714+0000", 2024-03-20T08:59:52.388 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0016846299999999999 2024-03-20T08:59:52.388 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.388 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.388 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.388 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.885218+0000", 2024-03-20T08:59:52.388 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00050378300000000005 2024-03-20T08:59:52.388 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.388 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.388 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.389 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.959786+0000", 2024-03-20T08:59:52.389 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.074567533000000005 2024-03-20T08:59:52.389 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.389 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.389 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:52.389 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.959838+0000", 2024-03-20T08:59:52.389 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2287000000000003e-05 2024-03-20T08:59:52.389 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.389 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.389 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:52.389 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.959874+0000", 2024-03-20T08:59:52.389 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6291000000000002e-05 2024-03-20T08:59:52.389 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.390 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:52.390 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.390 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.390 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.390 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:53568 2.7 2:efebfd9b:::benchmark_data_smithi122_107654_object53567:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-20T08:59:52.390 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:58:45.888935+0000", 2024-03-20T08:59:52.390 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 66.378491706000005, 2024-03-20T08:59:52.390 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.071078239000000001, 2024-03-20T08:59:52.390 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:52.390 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:52.390 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T08:59:52.391 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T08:59:52.391 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:59:52.391 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 53568 2024-03-20T08:59:52.391 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.391 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:52.391 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.391 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:52.391 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.888935+0000", 2024-03-20T08:59:52.391 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.391 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.391 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.391 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:52.391 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.888935+0000", 2024-03-20T08:59:52.392 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.392 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.392 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.392 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:52.392 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.888936+0000", 2024-03-20T08:59:52.392 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.372e-06 2024-03-20T08:59:52.392 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.392 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.392 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:52.392 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.888985+0000", 2024-03-20T08:59:52.392 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8231e-05 2024-03-20T08:59:52.392 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.392 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.392 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:52.393 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.888997+0000", 2024-03-20T08:59:52.393 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2293000000000001e-05 2024-03-20T08:59:52.393 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.393 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.393 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:52.393 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.889004+0000", 2024-03-20T08:59:52.393 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.7710000000000001e-06 2024-03-20T08:59:52.393 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.393 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.393 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:52.393 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.889057+0000", 2024-03-20T08:59:52.393 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3489000000000002e-05 2024-03-20T08:59:52.393 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.393 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.394 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:52.394 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.889195+0000", 2024-03-20T08:59:52.394 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013759699999999999 2024-03-20T08:59:52.394 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.394 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.394 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T08:59:52.394 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.889350+0000", 2024-03-20T08:59:52.394 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00015498899999999999 2024-03-20T08:59:52.394 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.394 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.395 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T08:59:52.395 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.890715+0000", 2024-03-20T08:59:52.395 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0013651570000000001 2024-03-20T08:59:52.395 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.395 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.395 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.395 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.891085+0000", 2024-03-20T08:59:52.395 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000370465 2024-03-20T08:59:52.395 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.395 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.395 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.396 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.959923+0000", 2024-03-20T08:59:52.396 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.068837630999999996 2024-03-20T08:59:52.396 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.396 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.396 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:52.396 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.959976+0000", 2024-03-20T08:59:52.396 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2496e-05 2024-03-20T08:59:52.396 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.396 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.396 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:52.396 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.960013+0000", 2024-03-20T08:59:52.396 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7747999999999998e-05 2024-03-20T08:59:52.396 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.396 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:52.397 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.397 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.397 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.397 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:53574 2.7 2:e09cc1d6:::benchmark_data_smithi122_107654_object53573:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-20T08:59:52.397 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:58:45.895630+0000", 2024-03-20T08:59:52.397 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 66.371796809000003, 2024-03-20T08:59:52.397 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.064527031999999998, 2024-03-20T08:59:52.397 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:52.397 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:52.397 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T08:59:52.397 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T08:59:52.397 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:59:52.397 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 53574 2024-03-20T08:59:52.397 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.397 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:52.398 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.398 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:52.398 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.895630+0000", 2024-03-20T08:59:52.398 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.398 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.398 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.399 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:52.399 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.895630+0000", 2024-03-20T08:59:52.399 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.399 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.399 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.399 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:52.400 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.895631+0000", 2024-03-20T08:59:52.400 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.009e-06 2024-03-20T08:59:52.400 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.400 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.400 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:52.400 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.895660+0000", 2024-03-20T08:59:52.400 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9028999999999999e-05 2024-03-20T08:59:52.401 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.401 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.401 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:52.401 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.895671+0000", 2024-03-20T08:59:52.401 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0919e-05 2024-03-20T08:59:52.401 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.402 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.402 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:52.402 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.895677+0000", 2024-03-20T08:59:52.402 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7250000000000002e-06 2024-03-20T08:59:52.402 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.402 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.403 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:52.403 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.895722+0000", 2024-03-20T08:59:52.403 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4997999999999998e-05 2024-03-20T08:59:52.403 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.403 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.403 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:52.404 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.895805+0000", 2024-03-20T08:59:52.404 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.3079999999999997e-05 2024-03-20T08:59:52.404 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.404 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.404 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T08:59:52.404 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.895900+0000", 2024-03-20T08:59:52.405 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.5571999999999999e-05 2024-03-20T08:59:52.405 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.405 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.405 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T08:59:52.405 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.897051+0000", 2024-03-20T08:59:52.405 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0011508060000000001 2024-03-20T08:59:52.406 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.406 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.406 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.406 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.897607+0000", 2024-03-20T08:59:52.406 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00055633400000000002 2024-03-20T08:59:52.406 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.407 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.407 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.407 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.960062+0000", 2024-03-20T08:59:52.407 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.062454161000000001 2024-03-20T08:59:52.407 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.407 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.408 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:52.408 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.960116+0000", 2024-03-20T08:59:52.408 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4750000000000003e-05 2024-03-20T08:59:52.408 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.408 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.409 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:52.409 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.960157+0000", 2024-03-20T08:59:52.409 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.0649000000000003e-05 2024-03-20T08:59:52.409 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.409 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:52.409 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.409 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.410 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.410 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:53581 2.7 2:e940c4ba:::benchmark_data_smithi122_107654_object53580:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-20T08:59:52.410 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:58:45.912437+0000", 2024-03-20T08:59:52.410 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 66.354990061999999, 2024-03-20T08:59:52.410 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.04784646, 2024-03-20T08:59:52.410 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:52.411 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:52.411 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T08:59:52.411 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T08:59:52.411 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:59:52.411 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 53581 2024-03-20T08:59:52.411 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.412 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:52.412 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.412 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:52.412 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.912437+0000", 2024-03-20T08:59:52.412 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.412 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.412 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.413 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:52.413 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.912437+0000", 2024-03-20T08:59:52.413 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.413 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.413 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.413 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:52.413 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.912439+0000", 2024-03-20T08:59:52.413 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3630000000000001e-06 2024-03-20T08:59:52.414 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.414 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.414 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:52.414 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.912493+0000", 2024-03-20T08:59:52.414 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4351999999999997e-05 2024-03-20T08:59:52.414 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.414 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.414 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:52.415 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T08:59:52.417 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.912522+0000", 2024-03-20T08:59:52.417 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8517000000000002e-05 2024-03-20T08:59:52.417 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.417 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.417 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:52.417 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.912549+0000", 2024-03-20T08:59:52.417 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6863999999999999e-05 2024-03-20T08:59:52.417 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.417 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.417 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:52.417 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.912618+0000", 2024-03-20T08:59:52.418 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.9330000000000002e-05 2024-03-20T08:59:52.418 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.418 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.418 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:52.418 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.912758+0000", 2024-03-20T08:59:52.418 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000140252 2024-03-20T08:59:52.418 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.418 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.418 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T08:59:52.418 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.912921+0000", 2024-03-20T08:59:52.418 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000162516 2024-03-20T08:59:52.418 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.419 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.419 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T08:59:52.419 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.914214+0000", 2024-03-20T08:59:52.419 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0012933280000000001 2024-03-20T08:59:52.419 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.419 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.419 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.419 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.915283+0000", 2024-03-20T08:59:52.419 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00106832 2024-03-20T08:59:52.419 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.419 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.419 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.419 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.960199+0000", 2024-03-20T08:59:52.420 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.044916606999999997 2024-03-20T08:59:52.420 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.420 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.420 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:52.420 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.960251+0000", 2024-03-20T08:59:52.420 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1734000000000003e-05 2024-03-20T08:59:52.420 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.420 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.420 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:52.420 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.960283+0000", 2024-03-20T08:59:52.420 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.2277000000000003e-05 2024-03-20T08:59:52.420 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.421 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:52.421 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.421 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.421 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.421 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:85603 2.7 2:e992b70d:::benchmark_data_smithi122_107654_object85602:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-03-20T08:59:52.421 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:59:07.717759+0000", 2024-03-20T08:59:52.421 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 44.549667911, 2024-03-20T08:59:52.421 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.040923001000000001, 2024-03-20T08:59:52.422 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:52.422 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:52.422 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T08:59:52.422 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T08:59:52.422 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:59:52.422 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 85603 2024-03-20T08:59:52.422 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.422 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:52.422 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.422 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:52.422 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:07.717759+0000", 2024-03-20T08:59:52.423 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.423 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.423 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.423 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:52.423 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:07.717759+0000", 2024-03-20T08:59:52.423 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.423 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.423 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.424 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:52.424 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:07.717761+0000", 2024-03-20T08:59:52.424 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6479999999999999e-06 2024-03-20T08:59:52.424 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.424 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.424 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:52.424 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:07.717815+0000", 2024-03-20T08:59:52.425 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4688999999999998e-05 2024-03-20T08:59:52.425 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.425 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.425 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:52.425 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:07.717829+0000", 2024-03-20T08:59:52.425 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.417e-05 2024-03-20T08:59:52.426 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.426 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.426 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:52.426 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:07.717839+0000", 2024-03-20T08:59:52.426 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.2140000000000002e-06 2024-03-20T08:59:52.426 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.426 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.426 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:52.426 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:07.717891+0000", 2024-03-20T08:59:52.426 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2085e-05 2024-03-20T08:59:52.426 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.427 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.427 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:52.427 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:07.717991+0000", 2024-03-20T08:59:52.427 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000100734 2024-03-20T08:59:52.427 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.427 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.427 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T08:59:52.427 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:07.718087+0000", 2024-03-20T08:59:52.427 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.5148000000000004e-05 2024-03-20T08:59:52.427 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.427 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.427 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T08:59:52.427 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:07.720066+0000", 2024-03-20T08:59:52.427 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0019794769999999999 2024-03-20T08:59:52.428 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.428 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.428 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.428 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:07.721149+0000", 2024-03-20T08:59:52.428 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0010825330000000001 2024-03-20T08:59:52.428 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.428 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.428 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.428 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:07.758573+0000", 2024-03-20T08:59:52.428 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.037423954000000002 2024-03-20T08:59:52.428 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.429 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.429 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:52.429 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:07.758638+0000", 2024-03-20T08:59:52.429 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.5519999999999996e-05 2024-03-20T08:59:52.429 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.429 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.429 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:52.429 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:07.758682+0000", 2024-03-20T08:59:52.429 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.3829000000000003e-05 2024-03-20T08:59:52.429 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.429 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:52.429 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.429 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.429 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.430 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:85609 2.7 2:ee84ae54:::benchmark_data_smithi122_107654_object85608:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-03-20T08:59:52.430 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:59:07.723529+0000", 2024-03-20T08:59:52.430 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 44.543898116000001, 2024-03-20T08:59:52.430 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.039704731, 2024-03-20T08:59:52.430 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T08:59:52.430 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T08:59:52.430 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T08:59:52.430 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T08:59:52.430 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T08:59:52.430 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 85609 2024-03-20T08:59:52.430 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.430 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T08:59:52.430 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.431 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T08:59:52.431 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:07.723529+0000", 2024-03-20T08:59:52.431 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.431 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.431 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.431 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T08:59:52.431 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:07.723529+0000", 2024-03-20T08:59:52.431 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T08:59:52.431 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.431 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.431 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T08:59:52.432 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:07.723531+0000", 2024-03-20T08:59:52.432 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5299999999999999e-06 2024-03-20T08:59:52.432 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.432 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.432 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T08:59:52.432 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:07.723599+0000", 2024-03-20T08:59:52.432 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.7402000000000003e-05 2024-03-20T08:59:52.432 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.432 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.432 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T08:59:52.432 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:07.723636+0000", 2024-03-20T08:59:52.432 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7392999999999999e-05 2024-03-20T08:59:52.432 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.432 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.433 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T08:59:52.433 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:07.723659+0000", 2024-03-20T08:59:52.433 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2872999999999999e-05 2024-03-20T08:59:52.433 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.433 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.433 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T08:59:52.433 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:07.723833+0000", 2024-03-20T08:59:52.433 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00017405 2024-03-20T08:59:52.433 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.433 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.433 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T08:59:52.433 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:07.723978+0000", 2024-03-20T08:59:52.433 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00014465300000000001 2024-03-20T08:59:52.434 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.434 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.434 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T08:59:52.434 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:07.724110+0000", 2024-03-20T08:59:52.434 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013277100000000001 2024-03-20T08:59:52.434 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.434 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.434 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T08:59:52.434 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:07.726586+0000", 2024-03-20T08:59:52.434 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0024756029999999998 2024-03-20T08:59:52.434 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.435 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.435 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.435 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:07.728036+0000", 2024-03-20T08:59:52.435 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0014495689999999999 2024-03-20T08:59:52.435 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.435 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.435 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T08:59:52.435 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:07.763165+0000", 2024-03-20T08:59:52.435 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.035129604000000002 2024-03-20T08:59:52.435 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.435 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.435 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T08:59:52.435 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:07.763208+0000", 2024-03-20T08:59:52.436 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.2484999999999997e-05 2024-03-20T08:59:52.436 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T08:59:52.436 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T08:59:52.436 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T08:59:52.436 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:59:07.763233+0000", 2024-03-20T08:59:52.436 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5797999999999999e-05 2024-03-20T08:59:52.436 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.436 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:52.436 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.436 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T08:59:52.436 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T08:59:52.437 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T08:59:52.444 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 88 16 117181 117165 83.2009 52.4375 0.00479595 0.0120131 2024-03-20T08:59:52.464 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:52.488 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T08:59:52.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:52.492 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T08:59:52.593 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T08:59:52.624 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:52.974 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T08:59:52.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:52.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T08:59:53.003 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-20T08:59:53.078 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T08:59:53.125 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:53.149 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T08:59:53.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:53.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T08:59:53.201 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T08:59:53.252 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T08:59:53.282 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:53.304 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T08:59:53.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:53.308 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T08:59:53.361 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:59:53.409 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T08:59:53.442 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 89 16 118020 118004 82.8552 52.4375 0.0131776 0.0120635 2024-03-20T08:59:53.475 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:53.508 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:53 smithi026 ceph-mon[68309]: pgmap v447: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 597 op/s 2024-03-20T08:59:53.508 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:53 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:59:53.508 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:53 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:59:53.508 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:53 smithi026 ceph-mon[68309]: osdmap e150: 12 total, 12 up, 12 in 2024-03-20T08:59:53.699 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:53 smithi122 ceph-mon[73495]: pgmap v447: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 597 op/s 2024-03-20T08:59:53.699 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:53 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:59:53.699 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:53 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:59:53.699 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:53 smithi122 ceph-mon[73495]: osdmap e150: 12 total, 12 up, 12 in 2024-03-20T08:59:53.699 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:53 smithi115 ceph-mon[72476]: pgmap v447: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 597 op/s 2024-03-20T08:59:53.700 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:53 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:59:53.700 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:53 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:59:53.700 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:53 smithi115 ceph-mon[72476]: osdmap e150: 12 total, 12 up, 12 in 2024-03-20T08:59:53.863 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T08:59:53.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:53.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T08:59:53.971 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T08:59:54.008 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:54.259 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T08:59:54.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:54.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T08:59:54.364 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T08:59:54.426 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:54.443 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 90 12 118782 118770 82.4665 47.875 0.0181858 0.0121167 2024-03-20T08:59:54.460 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T08:59:54.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:54.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T08:59:54.565 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T08:59:54.590 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:54 smithi115 ceph-mon[72476]: pgmap v449: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 788 op/s 2024-03-20T08:59:54.592 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:54.665 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:59:54.690 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T08:59:54.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:54.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T08:59:54.716 INFO:teuthology.orchestra.run.smithi026.stderr:no valid command found; 10 closest matches: 2024-03-20T08:59:54.716 INFO:teuthology.orchestra.run.smithi026.stderr:0 2024-03-20T08:59:54.717 INFO:teuthology.orchestra.run.smithi026.stderr:1 2024-03-20T08:59:54.717 INFO:teuthology.orchestra.run.smithi026.stderr:2 2024-03-20T08:59:54.717 INFO:teuthology.orchestra.run.smithi026.stderr:abort 2024-03-20T08:59:54.717 INFO:teuthology.orchestra.run.smithi026.stderr:assert 2024-03-20T08:59:54.717 INFO:teuthology.orchestra.run.smithi026.stderr:bench [] [] [] [] 2024-03-20T08:59:54.717 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs debug_inject_read_zeros 2024-03-20T08:59:54.717 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs files list 2024-03-20T08:59:54.717 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs stats 2024-03-20T08:59:54.717 INFO:teuthology.orchestra.run.smithi026.stderr:bluestore allocator dump block 2024-03-20T08:59:54.717 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: invalid command 2024-03-20T08:59:54.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:54 smithi026 ceph-mon[68309]: pgmap v449: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 788 op/s 2024-03-20T08:59:54.793 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T08:59:54.827 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:54.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:54 smithi122 ceph-mon[73495]: pgmap v449: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 788 op/s 2024-03-20T08:59:55.028 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T08:59:55.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:55.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T08:59:55.132 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T08:59:55.157 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:55.176 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T08:59:55.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:55.183 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T08:59:55.251 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T08:59:55.251 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T08:59:55.251 INFO:teuthology.orchestra.run.smithi115.stdout: "num_ops": 0 2024-03-20T08:59:55.252 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T08:59:55.283 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T08:59:55.312 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:55.443 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Total time run: 90.0665 2024-03-20T08:59:55.443 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Total writes made: 118782 2024-03-20T08:59:55.443 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Write size: 65536 2024-03-20T08:59:55.443 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Object size: 65536 2024-03-20T08:59:55.443 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Bandwidth (MB/sec): 82.4266 2024-03-20T08:59:55.444 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Stddev Bandwidth: 55.1352 2024-03-20T08:59:55.444 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Max bandwidth (MB/sec): 234.438 2024-03-20T08:59:55.444 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Min bandwidth (MB/sec): 0 2024-03-20T08:59:55.444 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Average IOPS: 1318 2024-03-20T08:59:55.444 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Stddev IOPS: 882.178 2024-03-20T08:59:55.444 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Max IOPS: 3751 2024-03-20T08:59:55.444 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Min IOPS: 0 2024-03-20T08:59:55.444 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Average Latency(s): 0.0121252 2024-03-20T08:59:55.444 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Stddev Latency(s): 0.173681 2024-03-20T08:59:55.444 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Max latency(s): 15.0259 2024-03-20T08:59:55.444 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Min latency(s): 0.00103856 2024-03-20T08:59:55.451 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Cleaning up (deleting benchmark objects) 2024-03-20T08:59:55.487 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T08:59:55.487 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-20T08:59:55.487 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.8 'trim stale osdmaps' 2024-03-20T08:59:55.533 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T08:59:55.596 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:55 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/425870463' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T08:59:55.597 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:55 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T08:59:55.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:55 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/425870463' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T08:59:55.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:55 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T08:59:55.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:55 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/425870463' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T08:59:55.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:55 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T08:59:55.960 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T08:59:55.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:55.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T08:59:55.981 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-20T08:59:56.068 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T08:59:56.104 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:56.126 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T08:59:56.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:56.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T08:59:56.232 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T08:59:56.276 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:56.303 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T08:59:56.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:56.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T08:59:56.389 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:59:56.408 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T08:59:56.440 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:56.468 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T08:59:56.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:56.475 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T08:59:56.576 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T08:59:56.610 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:56.647 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T08:59:56.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:56.654 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T08:59:56.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:56 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T08:59:56.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:56 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T08:59:56.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:56 smithi115 ceph-mon[72476]: pgmap v450: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 752 op/s 2024-03-20T08:59:56.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:56 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T08:59:56.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:56 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T08:59:56.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:56 smithi115 ceph-mon[72476]: osdmap e151: 12 total, 12 up, 12 in 2024-03-20T08:59:56.683 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:56 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T08:59:56.683 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:56 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T08:59:56.683 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:56 smithi026 ceph-mon[68309]: pgmap v450: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 752 op/s 2024-03-20T08:59:56.683 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:56 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T08:59:56.683 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:56 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T08:59:56.683 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:56 smithi026 ceph-mon[68309]: osdmap e151: 12 total, 12 up, 12 in 2024-03-20T08:59:56.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:56 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T08:59:56.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:56 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T08:59:56.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:56 smithi122 ceph-mon[73495]: pgmap v450: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 752 op/s 2024-03-20T08:59:56.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:56 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T08:59:56.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:56 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T08:59:56.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:56 smithi122 ceph-mon[73495]: osdmap e151: 12 total, 12 up, 12 in 2024-03-20T08:59:56.754 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T08:59:56.806 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:56.830 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T08:59:56.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:56.834 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T08:59:56.938 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T08:59:57.000 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:57.142 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T08:59:57.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:57.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T08:59:57.260 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T08:59:57.323 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:57.411 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T08:59:57.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:57.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T08:59:57.519 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T08:59:57.562 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:57.591 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T08:59:57.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:57.594 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T08:59:57.695 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T08:59:57.724 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:57.736 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T08:59:57.745 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T08:59:57.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:57.750 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T08:59:57.825 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T08:59:57.850 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T08:59:57.876 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T08:59:57.898 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:57.913 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T08:59:57.966 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T08:59:58.000 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T08:59:58.037 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T08:59:58.082 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T08:59:58.119 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T08:59:58.156 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T08:59:58.186 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T08:59:58.223 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T08:59:58.271 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T08:59:58.285 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T08:59:58.285 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T08:59:58.285 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T08:59:58.285 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T08:59:58.285 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T08:59:58.285 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T08:59:58.286 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T08:59:58.286 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T08:59:58.286 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T08:59:58.286 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T08:59:58.286 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T08:59:58.286 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T08:59:58.440 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T08:59:58.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:58.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T08:59:58.465 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T08:59:58.465 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T08:59:58.551 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T08:59:58.613 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:58.748 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T08:59:58.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:58.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T08:59:58.851 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T08:59:58.911 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:59.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:58 smithi115 ceph-mon[72476]: pgmap v452: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 777 op/s 2024-03-20T08:59:59.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 08:59:58 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/859140126' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T08:59:59.192 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T08:59:59.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:58 smithi122 ceph-mon[73495]: pgmap v452: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 777 op/s 2024-03-20T08:59:59.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 08:59:58 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/859140126' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T08:59:59.222 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-20T08:59:59.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:58 smithi026 ceph-mon[68309]: pgmap v452: 17 pgs: 1 active+clean+scrubbing+deep, 16 active+clean; 7.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 777 op/s 2024-03-20T08:59:59.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 08:59:58 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/859140126' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T08:59:59.558 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T08:59:59.819 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T08:59:59.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:59.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T08:59:59.927 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T08:59:59.957 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T08:59:59.980 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T08:59:59.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T08:59:59.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:00:00.084 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:00:00.119 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:00.148 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:00:00.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:00.153 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:00:00.172 INFO:teuthology.orchestra.run.smithi026.stderr:no valid command found; 10 closest matches: 2024-03-20T09:00:00.172 INFO:teuthology.orchestra.run.smithi026.stderr:0 2024-03-20T09:00:00.172 INFO:teuthology.orchestra.run.smithi026.stderr:1 2024-03-20T09:00:00.173 INFO:teuthology.orchestra.run.smithi026.stderr:2 2024-03-20T09:00:00.173 INFO:teuthology.orchestra.run.smithi026.stderr:abort 2024-03-20T09:00:00.173 INFO:teuthology.orchestra.run.smithi026.stderr:assert 2024-03-20T09:00:00.173 INFO:teuthology.orchestra.run.smithi026.stderr:bench [] [] [] [] 2024-03-20T09:00:00.173 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs debug_inject_read_zeros 2024-03-20T09:00:00.173 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs files list 2024-03-20T09:00:00.173 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs stats 2024-03-20T09:00:00.173 INFO:teuthology.orchestra.run.smithi026.stderr:bluestore allocator dump block 2024-03-20T09:00:00.174 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: invalid command 2024-03-20T09:00:00.254 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:00:00.334 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:00.482 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:00:00.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:00.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:00:00.596 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:00:00.633 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:00.754 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:00:00.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:00 smithi115 ceph-mon[72476]: pgmap v453: 17 pgs: 17 active+clean; 7.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 35 MiB/s wr, 747 op/s 2024-03-20T09:00:00.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:00 smithi115 ceph-mon[72476]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-03-20T09:00:00.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:00 smithi115 ceph-mon[72476]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-20T09:00:00.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:00 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:00:00.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:00 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:00:01.085 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:00:01.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:01.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:00:01.174 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:00 smithi026 ceph-mon[68309]: pgmap v453: 17 pgs: 17 active+clean; 7.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 35 MiB/s wr, 747 op/s 2024-03-20T09:00:01.175 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:00 smithi026 ceph-mon[68309]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-03-20T09:00:01.175 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:00 smithi026 ceph-mon[68309]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-20T09:00:01.175 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:00 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:00:01.175 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:00 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:00:01.190 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:00:01.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:00 smithi122 ceph-mon[73495]: pgmap v453: 17 pgs: 17 active+clean; 7.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 35 MiB/s wr, 747 op/s 2024-03-20T09:00:01.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:00 smithi122 ceph-mon[73495]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-03-20T09:00:01.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:00 smithi122 ceph-mon[73495]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-20T09:00:01.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:00 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:00:01.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:00 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:00:01.220 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T09:00:01.220 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-20T09:00:01.221 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.9 'trim stale osdmaps' 2024-03-20T09:00:01.224 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:01.328 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:00:01.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:01.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:00:01.433 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:00:01.530 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:01.567 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:00:01.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:01.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:00:01.678 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:00:01.678 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:00:01.680 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:00:01.717 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:01.845 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:01 smithi026 ceph-mon[68309]: osdmap e152: 12 total, 12 up, 12 in 2024-03-20T09:00:01.975 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:00:01.979 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:01 smithi115 ceph-mon[72476]: osdmap e152: 12 total, 12 up, 12 in 2024-03-20T09:00:01.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:01.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:00:02.082 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:00:02.156 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:02.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:01 smithi122 ceph-mon[73495]: osdmap e152: 12 total, 12 up, 12 in 2024-03-20T09:00:02.207 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:00:02.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:02.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:00:02.236 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:00:02.313 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:00:02.365 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:02.395 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:00:02.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:02.399 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:00:02.500 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:00:02.513 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_ops_in_flight 2024-03-20T09:00:02.520 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:00:02.562 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:02.870 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:02 smithi026 ceph-mon[68309]: pgmap v455: 17 pgs: 17 active+clean; 7.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 21 MiB/s wr, 741 op/s 2024-03-20T09:00:02.876 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:00:03.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:02 smithi115 ceph-mon[72476]: pgmap v455: 17 pgs: 17 active+clean; 7.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 21 MiB/s wr, 741 op/s 2024-03-20T09:00:03.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:02 smithi122 ceph-mon[73495]: pgmap v455: 17 pgs: 17 active+clean; 7.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 21 MiB/s wr, 741 op/s 2024-03-20T09:00:03.933 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:00:03.961 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:00:03.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:03.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:00:04.069 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:00:04.161 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:04.884 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:00:05.138 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:00:05.142 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:04 smithi026 ceph-mon[68309]: pgmap v456: 17 pgs: 17 active+clean; 6.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 9.8 MiB/s wr, 902 op/s 2024-03-20T09:00:05.142 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:04 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1385503746' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:00:05.142 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:04 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:00:05.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:05.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:00:05.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:04 smithi115 ceph-mon[72476]: pgmap v456: 17 pgs: 17 active+clean; 6.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 9.8 MiB/s wr, 902 op/s 2024-03-20T09:00:05.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:04 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1385503746' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:00:05.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:04 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:00:05.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:04 smithi122 ceph-mon[73495]: pgmap v456: 17 pgs: 17 active+clean; 6.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 9.8 MiB/s wr, 902 op/s 2024-03-20T09:00:05.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:04 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1385503746' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:00:05.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:04 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:00:05.248 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:00:05.290 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:05.554 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:00:05.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:05.560 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:00:05.661 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:00:05.718 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:06.123 INFO:teuthology.orchestra.run.smithi026.stderr:no valid command found; 10 closest matches: 2024-03-20T09:00:06.124 INFO:teuthology.orchestra.run.smithi026.stderr:0 2024-03-20T09:00:06.124 INFO:teuthology.orchestra.run.smithi026.stderr:1 2024-03-20T09:00:06.124 INFO:teuthology.orchestra.run.smithi026.stderr:2 2024-03-20T09:00:06.124 INFO:teuthology.orchestra.run.smithi026.stderr:abort 2024-03-20T09:00:06.124 INFO:teuthology.orchestra.run.smithi026.stderr:assert 2024-03-20T09:00:06.124 INFO:teuthology.orchestra.run.smithi026.stderr:bench [] [] [] [] 2024-03-20T09:00:06.124 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs debug_inject_read_zeros 2024-03-20T09:00:06.124 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs files list 2024-03-20T09:00:06.125 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs stats 2024-03-20T09:00:06.125 INFO:teuthology.orchestra.run.smithi026.stderr:bluestore allocator dump block 2024-03-20T09:00:06.125 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: invalid command 2024-03-20T09:00:06.139 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:05 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:00:06.139 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:05 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:00:06.140 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:05 smithi026 ceph-mon[68309]: osdmap e153: 12 total, 12 up, 12 in 2024-03-20T09:00:06.159 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:05 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:00:06.159 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:05 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:00:06.160 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:05 smithi115 ceph-mon[72476]: osdmap e153: 12 total, 12 up, 12 in 2024-03-20T09:00:06.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:05 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:00:06.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:05 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:00:06.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:05 smithi122 ceph-mon[73495]: osdmap e153: 12 total, 12 up, 12 in 2024-03-20T09:00:07.051 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:06 smithi026 ceph-mon[68309]: pgmap v458: 17 pgs: 17 active+clean; 6.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 1.7 MiB/s wr, 963 op/s 2024-03-20T09:00:07.051 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:06 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:00:07.051 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:06 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:00:07.089 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:00:07.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:07.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:00:07.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:06 smithi115 ceph-mon[72476]: pgmap v458: 17 pgs: 17 active+clean; 6.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 1.7 MiB/s wr, 963 op/s 2024-03-20T09:00:07.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:06 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:00:07.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:06 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:00:07.197 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:00:07.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:06 smithi122 ceph-mon[73495]: pgmap v458: 17 pgs: 17 active+clean; 6.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 1.7 MiB/s wr, 963 op/s 2024-03-20T09:00:07.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:06 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:00:07.199 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:06 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:00:07.229 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:07.251 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:00:07.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:07.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:00:07.356 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:00:07.393 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:07.414 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:00:07.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:07.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:00:07.518 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:00:07.547 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:07.559 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T09:00:07.560 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-20T09:00:07.560 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.10 'trim stale osdmaps' 2024-03-20T09:00:07.567 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:00:07.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:07.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:00:07.622 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:00:07.622 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:00:07.673 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:00:07.734 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:07.912 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:07 smithi026 ceph-mon[68309]: osdmap e154: 12 total, 12 up, 12 in 2024-03-20T09:00:08.011 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:00:08.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:08.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:00:08.117 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:00:08.186 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:08.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:07 smithi122 ceph-mon[73495]: osdmap e154: 12 total, 12 up, 12 in 2024-03-20T09:00:08.254 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:00:08.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:08.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:00:08.362 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:00:08.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:07 smithi115 ceph-mon[72476]: osdmap e154: 12 total, 12 up, 12 in 2024-03-20T09:00:08.414 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:08.493 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:00:08.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:08.498 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:00:08.529 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_blocked_ops 2024-03-20T09:00:08.599 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:00:08.629 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:08.633 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:00:08.654 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:00:08.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:08.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:00:08.724 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:00:08.755 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:00:08.757 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:00:08.784 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:08.802 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:00:08.806 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:00:08.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:08.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:00:08.859 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:00:08.900 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:00:08.911 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:00:08.938 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:00:08.967 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:00:08.982 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:00:08.990 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:09.021 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:00:09.022 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:00:09.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:09.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:00:09.065 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:00:09.115 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:00:09.127 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:00:09.153 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:00:09.168 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:08 smithi122 ceph-mon[73495]: 2.c scrub starts 2024-03-20T09:00:09.168 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:08 smithi122 ceph-mon[73495]: 1.0 scrub starts 2024-03-20T09:00:09.168 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:08 smithi122 ceph-mon[73495]: 1.0 scrub ok 2024-03-20T09:00:09.168 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:08 smithi122 ceph-mon[73495]: pgmap v460: 17 pgs: 17 active+clean; 6.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s 2024-03-20T09:00:09.168 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:08 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3678111796' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:00:09.173 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:09.188 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:00:09.197 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:00:09.197 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:00:09.197 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:00:09.197 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:00:09.197 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:00:09.197 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:00:09.197 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:00:09.197 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:00:09.197 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:00:09.198 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:00:09.198 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:00:09.198 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:00:09.211 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:00:09.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:09.215 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:00:09.215 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:08 smithi026 ceph-mon[68309]: 2.c scrub starts 2024-03-20T09:00:09.215 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:08 smithi026 ceph-mon[68309]: 1.0 scrub starts 2024-03-20T09:00:09.215 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:08 smithi026 ceph-mon[68309]: 1.0 scrub ok 2024-03-20T09:00:09.216 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:08 smithi026 ceph-mon[68309]: pgmap v460: 17 pgs: 17 active+clean; 6.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s 2024-03-20T09:00:09.216 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:08 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3678111796' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:00:09.315 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:00:09.376 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:09.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:08 smithi115 ceph-mon[72476]: 2.c scrub starts 2024-03-20T09:00:09.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:08 smithi115 ceph-mon[72476]: 1.0 scrub starts 2024-03-20T09:00:09.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:08 smithi115 ceph-mon[72476]: 1.0 scrub ok 2024-03-20T09:00:09.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:08 smithi115 ceph-mon[72476]: pgmap v460: 17 pgs: 17 active+clean; 6.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.14k op/s 2024-03-20T09:00:09.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:08 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3678111796' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:00:09.491 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:00:09.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:09.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:00:09.600 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:00:09.630 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:09.651 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:00:09.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:09.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:00:09.757 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:00:09.808 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:09.873 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:00:09.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:09.877 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:00:09.977 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:00:10.035 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:00:10.036 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:10.147 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:00:10.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:10.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:00:10.254 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:00:10.344 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:10.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:10 smithi026 ceph-mon[68309]: 2.c scrub ok 2024-03-20T09:00:10.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:10 smithi026 ceph-mon[68309]: pgmap v461: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s 2024-03-20T09:00:11.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:10 smithi122 ceph-mon[73495]: 2.c scrub ok 2024-03-20T09:00:11.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:10 smithi122 ceph-mon[73495]: pgmap v461: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s 2024-03-20T09:00:11.242 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:00:11.242 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:00:11.242 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-03-20T09:00:11.242 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-03-20T09:00:11.242 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:00:11.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:10 smithi115 ceph-mon[72476]: 2.c scrub ok 2024-03-20T09:00:11.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:10 smithi115 ceph-mon[72476]: pgmap v461: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.06k op/s 2024-03-20T09:00:12.200 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_historic_ops 2024-03-20T09:00:12.207 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:00:12.595 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:00:13.162 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:12 smithi122 ceph-mon[73495]: pgmap v462: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 948 op/s 2024-03-20T09:00:13.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:12 smithi026 ceph-mon[68309]: pgmap v462: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 948 op/s 2024-03-20T09:00:13.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:12 smithi115 ceph-mon[72476]: pgmap v462: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 948 op/s 2024-03-20T09:00:13.878 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:00:13.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:13.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:00:13.985 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:00:14.047 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:00:14.097 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:14.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:13 smithi026 ceph-mon[68309]: 2.4 scrub starts 2024-03-20T09:00:14.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:13 smithi115 ceph-mon[72476]: 2.4 scrub starts 2024-03-20T09:00:14.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:13 smithi122 ceph-mon[73495]: 2.4 scrub starts 2024-03-20T09:00:14.853 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:00:14.853 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-03-20T09:00:14.854 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-03-20T09:00:14.854 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-03-20T09:00:14.854 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.854 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:30172 2.f 2:f74da544:::benchmark_data_smithi122_107654_object30171:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-20T09:00:14.854 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:58:34.522939+0000", 2024-03-20T09:00:14.854 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 100.327818882, 2024-03-20T09:00:14.854 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20426802299999999, 2024-03-20T09:00:14.854 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:14.854 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:14.854 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:14.854 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:14.854 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:14.854 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 30172 2024-03-20T09:00:14.855 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.855 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:14.855 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.855 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:14.855 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.522939+0000", 2024-03-20T09:00:14.855 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.855 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.855 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.855 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:14.855 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.522939+0000", 2024-03-20T09:00:14.855 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.855 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.855 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.856 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:14.856 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.522939+0000", 2024-03-20T09:00:14.856 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.51e-07 2024-03-20T09:00:14.856 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.856 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.856 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:14.856 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.522962+0000", 2024-03-20T09:00:14.856 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2342000000000001e-05 2024-03-20T09:00:14.856 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.856 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.856 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:14.856 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.522970+0000", 2024-03-20T09:00:14.856 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6270000000000006e-06 2024-03-20T09:00:14.857 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.857 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.857 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:14.857 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.522977+0000", 2024-03-20T09:00:14.857 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9199999999999998e-06 2024-03-20T09:00:14.857 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.857 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.857 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:14.857 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.523080+0000", 2024-03-20T09:00:14.857 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000102877 2024-03-20T09:00:14.857 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.857 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.859 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:14.859 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.523228+0000", 2024-03-20T09:00:14.859 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014827199999999999 2024-03-20T09:00:14.859 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.859 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.859 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 4,9", 2024-03-20T09:00:14.859 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.523384+0000", 2024-03-20T09:00:14.859 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015502199999999999 2024-03-20T09:00:14.859 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.859 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.860 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.860 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.524537+0000", 2024-03-20T09:00:14.860 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001153196 2024-03-20T09:00:14.860 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.860 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.860 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:14.860 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.524942+0000", 2024-03-20T09:00:14.860 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00040574399999999998 2024-03-20T09:00:14.860 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.860 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.860 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.860 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.727133+0000", 2024-03-20T09:00:14.860 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20219045499999999 2024-03-20T09:00:14.860 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.861 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.861 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:14.861 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.727175+0000", 2024-03-20T09:00:14.861 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1795999999999998e-05 2024-03-20T09:00:14.861 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.861 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.861 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:14.861 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.727207+0000", 2024-03-20T09:00:14.861 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2221000000000002e-05 2024-03-20T09:00:14.861 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.861 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:14.861 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.861 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.861 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.862 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:30200 2.f 2:f0fc9aeb:::benchmark_data_smithi122_107654_object30199:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-20T09:00:14.862 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:58:34.537363+0000", 2024-03-20T09:00:14.862 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 100.31339435300001, 2024-03-20T09:00:14.862 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19084448800000001, 2024-03-20T09:00:14.862 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:14.862 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:14.862 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:14.862 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:14.862 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:14.862 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 30200 2024-03-20T09:00:14.862 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.862 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:14.862 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.862 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:14.863 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.537363+0000", 2024-03-20T09:00:14.863 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.863 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.863 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.863 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:14.863 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.537363+0000", 2024-03-20T09:00:14.863 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.863 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.863 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.863 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:14.863 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.537365+0000", 2024-03-20T09:00:14.863 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3599999999999999e-06 2024-03-20T09:00:14.863 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.864 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.864 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:14.864 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.537388+0000", 2024-03-20T09:00:14.864 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3697999999999999e-05 2024-03-20T09:00:14.864 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.864 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.864 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:14.864 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.537403+0000", 2024-03-20T09:00:14.864 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4127999999999999e-05 2024-03-20T09:00:14.864 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.864 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.864 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:14.864 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.537411+0000", 2024-03-20T09:00:14.864 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9130000000000003e-06 2024-03-20T09:00:14.865 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.865 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.865 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:14.865 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.537452+0000", 2024-03-20T09:00:14.865 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1961e-05 2024-03-20T09:00:14.865 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.865 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.865 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:14.865 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.537575+0000", 2024-03-20T09:00:14.865 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000122994 2024-03-20T09:00:14.865 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.865 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.865 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 4,9", 2024-03-20T09:00:14.866 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.537678+0000", 2024-03-20T09:00:14.866 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000102867 2024-03-20T09:00:14.866 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.866 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.866 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.866 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.539934+0000", 2024-03-20T09:00:14.866 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0022559329999999999 2024-03-20T09:00:14.866 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.866 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.866 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:14.866 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.540101+0000", 2024-03-20T09:00:14.866 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016721100000000001 2024-03-20T09:00:14.867 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.867 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.867 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.867 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.728114+0000", 2024-03-20T09:00:14.867 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.188012553 2024-03-20T09:00:14.867 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.867 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.867 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:14.867 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.728168+0000", 2024-03-20T09:00:14.868 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3684000000000003e-05 2024-03-20T09:00:14.868 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.868 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.868 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:14.868 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.728208+0000", 2024-03-20T09:00:14.868 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0185999999999998e-05 2024-03-20T09:00:14.868 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.868 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:14.868 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.868 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.868 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.868 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:30202 2.f 2:f47c1051:::benchmark_data_smithi122_107654_object30201:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-20T09:00:14.868 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:58:34.537517+0000", 2024-03-20T09:00:14.869 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 100.313240734, 2024-03-20T09:00:14.869 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19085913400000001, 2024-03-20T09:00:14.869 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:14.869 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:14.869 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:14.869 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:14.869 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:14.869 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 30202 2024-03-20T09:00:14.869 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.869 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:14.869 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.869 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:14.869 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.537517+0000", 2024-03-20T09:00:14.870 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.870 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.870 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.870 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:14.870 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.537517+0000", 2024-03-20T09:00:14.870 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.870 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.870 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.870 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:14.870 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.537518+0000", 2024-03-20T09:00:14.870 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4300000000000002e-07 2024-03-20T09:00:14.870 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.871 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.871 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:14.871 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.537539+0000", 2024-03-20T09:00:14.871 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.145e-05 2024-03-20T09:00:14.871 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.871 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.871 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:14.871 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.537547+0000", 2024-03-20T09:00:14.871 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1850000000000004e-06 2024-03-20T09:00:14.871 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.871 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.871 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:14.871 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.537554+0000", 2024-03-20T09:00:14.871 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7710000000000001e-06 2024-03-20T09:00:14.872 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.872 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.872 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:14.872 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.538652+0000", 2024-03-20T09:00:14.872 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010981910000000001 2024-03-20T09:00:14.872 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.872 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.872 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:14.872 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.538769+0000", 2024-03-20T09:00:14.872 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000116128 2024-03-20T09:00:14.872 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.872 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.872 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 4,9", 2024-03-20T09:00:14.872 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.538893+0000", 2024-03-20T09:00:14.873 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012465100000000001 2024-03-20T09:00:14.873 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.873 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.873 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.873 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.540043+0000", 2024-03-20T09:00:14.873 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0011499979999999999 2024-03-20T09:00:14.873 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.873 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.873 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:14.873 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.540107+0000", 2024-03-20T09:00:14.873 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3874000000000005e-05 2024-03-20T09:00:14.873 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.873 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.873 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.873 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.728258+0000", 2024-03-20T09:00:14.874 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18815111700000001 2024-03-20T09:00:14.874 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.874 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.874 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:14.874 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.728331+0000", 2024-03-20T09:00:14.874 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2816000000000002e-05 2024-03-20T09:00:14.874 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.874 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.874 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:14.874 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.728376+0000", 2024-03-20T09:00:14.874 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5210000000000003e-05 2024-03-20T09:00:14.874 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.874 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:14.874 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.875 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.875 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.875 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:30239 2.f 2:fb5d4aea:::benchmark_data_smithi122_107654_object30238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-20T09:00:14.875 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:58:34.553378+0000", 2024-03-20T09:00:14.875 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 100.297379452, 2024-03-20T09:00:14.875 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.17587739099999999, 2024-03-20T09:00:14.875 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:14.875 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:14.875 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:14.875 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:14.875 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:14.875 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 30239 2024-03-20T09:00:14.875 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.875 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:14.876 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.876 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:14.876 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.553378+0000", 2024-03-20T09:00:14.876 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.876 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.876 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.876 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:14.876 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.553378+0000", 2024-03-20T09:00:14.876 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.876 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.876 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.876 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:14.876 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.553380+0000", 2024-03-20T09:00:14.876 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.17e-06 2024-03-20T09:00:14.877 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.877 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.877 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:14.877 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.553399+0000", 2024-03-20T09:00:14.877 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9119000000000001e-05 2024-03-20T09:00:14.877 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.877 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.877 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:14.877 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.553409+0000", 2024-03-20T09:00:14.877 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0267e-05 2024-03-20T09:00:14.877 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.877 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.877 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:14.878 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.553415+0000", 2024-03-20T09:00:14.878 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2650000000000002e-06 2024-03-20T09:00:14.878 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.878 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.878 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:14.878 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.553459+0000", 2024-03-20T09:00:14.878 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4267000000000002e-05 2024-03-20T09:00:14.878 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.878 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.878 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:14.878 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.553566+0000", 2024-03-20T09:00:14.878 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000106897 2024-03-20T09:00:14.878 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.878 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.879 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 4,9", 2024-03-20T09:00:14.879 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.553714+0000", 2024-03-20T09:00:14.879 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014721600000000001 2024-03-20T09:00:14.879 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.879 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.879 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.879 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.556067+0000", 2024-03-20T09:00:14.879 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002353099 2024-03-20T09:00:14.879 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.879 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.879 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:14.879 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.557007+0000", 2024-03-20T09:00:14.879 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00094056299999999995 2024-03-20T09:00:14.879 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.879 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.880 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.880 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.729165+0000", 2024-03-20T09:00:14.880 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.17215812599999999 2024-03-20T09:00:14.880 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.880 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.880 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:14.880 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.729215+0000", 2024-03-20T09:00:14.880 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9298000000000001e-05 2024-03-20T09:00:14.880 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.880 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.880 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:14.880 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.729256+0000", 2024-03-20T09:00:14.880 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1103999999999998e-05 2024-03-20T09:00:14.880 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.880 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:14.881 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.881 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.881 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.881 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:30280 2.f 2:f92af2c9:::benchmark_data_smithi122_107654_object30279:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-20T09:00:14.881 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:58:34.572657+0000", 2024-03-20T09:00:14.881 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 100.27810116000001, 2024-03-20T09:00:14.881 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.15675993999999999, 2024-03-20T09:00:14.881 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:14.881 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:14.881 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:14.881 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:14.881 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:14.881 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 30280 2024-03-20T09:00:14.881 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.882 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:14.882 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.882 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:14.882 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.572657+0000", 2024-03-20T09:00:14.882 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.882 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.882 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.882 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:14.882 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.572657+0000", 2024-03-20T09:00:14.882 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.882 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.882 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.882 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:14.882 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.572657+0000", 2024-03-20T09:00:14.883 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.44e-07 2024-03-20T09:00:14.883 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.883 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.883 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:14.883 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.572678+0000", 2024-03-20T09:00:14.883 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0826999999999999e-05 2024-03-20T09:00:14.883 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.883 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.883 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:14.883 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.572688+0000", 2024-03-20T09:00:14.883 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0134e-05 2024-03-20T09:00:14.883 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.883 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.883 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:14.884 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.572695+0000", 2024-03-20T09:00:14.884 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7299999999999999e-06 2024-03-20T09:00:14.884 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.884 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.884 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:14.884 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.572750+0000", 2024-03-20T09:00:14.884 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5563999999999999e-05 2024-03-20T09:00:14.884 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.884 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.884 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:14.884 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.572891+0000", 2024-03-20T09:00:14.884 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000140767 2024-03-20T09:00:14.884 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.884 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.884 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 4,9", 2024-03-20T09:00:14.885 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.573034+0000", 2024-03-20T09:00:14.885 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014309699999999999 2024-03-20T09:00:14.885 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.885 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.885 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:14.885 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.574106+0000", 2024-03-20T09:00:14.885 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010715880000000001 2024-03-20T09:00:14.885 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.885 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.885 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.885 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.574513+0000", 2024-03-20T09:00:14.885 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00040760999999999998 2024-03-20T09:00:14.885 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.885 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.886 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.886 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.729314+0000", 2024-03-20T09:00:14.886 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.15480029200000001 2024-03-20T09:00:14.886 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.886 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.886 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:14.886 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.729373+0000", 2024-03-20T09:00:14.886 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9474e-05 2024-03-20T09:00:14.886 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.886 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.886 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:14.886 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.729417+0000", 2024-03-20T09:00:14.886 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3312999999999997e-05 2024-03-20T09:00:14.887 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.887 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:14.887 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.887 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.887 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.887 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:45936 2.b 2:d4186d16:::benchmark_data_smithi122_107654_object45935:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-20T09:00:14.887 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:58:41.726346+0000", 2024-03-20T09:00:14.887 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 93.124411975000001, 2024-03-20T09:00:14.887 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20713243200000001, 2024-03-20T09:00:14.887 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:14.887 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:14.888 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:14.888 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:14.888 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:14.888 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 45936 2024-03-20T09:00:14.888 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.888 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:14.888 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.888 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:14.888 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.726346+0000", 2024-03-20T09:00:14.888 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.888 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.888 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.888 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:14.888 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.726346+0000", 2024-03-20T09:00:14.888 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.889 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.889 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.889 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:14.889 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.726347+0000", 2024-03-20T09:00:14.889 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.122e-06 2024-03-20T09:00:14.889 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.889 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.889 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:14.889 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.726369+0000", 2024-03-20T09:00:14.889 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1987000000000001e-05 2024-03-20T09:00:14.889 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.889 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.889 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:14.889 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.726384+0000", 2024-03-20T09:00:14.890 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5325e-05 2024-03-20T09:00:14.890 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.890 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.890 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:14.890 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.726393+0000", 2024-03-20T09:00:14.890 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2719999999999997e-06 2024-03-20T09:00:14.890 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.890 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.890 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:14.890 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.727102+0000", 2024-03-20T09:00:14.890 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000709214 2024-03-20T09:00:14.890 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.890 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.890 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:14.891 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.727209+0000", 2024-03-20T09:00:14.891 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010774 2024-03-20T09:00:14.891 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.891 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.891 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,11", 2024-03-20T09:00:14.891 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.727320+0000", 2024-03-20T09:00:14.891 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000110469 2024-03-20T09:00:14.891 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.891 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.891 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:14.891 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.728754+0000", 2024-03-20T09:00:14.891 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014344469999999999 2024-03-20T09:00:14.891 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.891 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.892 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.892 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.730238+0000", 2024-03-20T09:00:14.892 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001483832 2024-03-20T09:00:14.892 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.892 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.892 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.892 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.933412+0000", 2024-03-20T09:00:14.892 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20317375600000001 2024-03-20T09:00:14.892 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.892 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.892 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:14.892 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.933447+0000", 2024-03-20T09:00:14.892 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4999999999999997e-05 2024-03-20T09:00:14.892 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.892 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.893 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:14.893 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.933478+0000", 2024-03-20T09:00:14.893 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1267999999999999e-05 2024-03-20T09:00:14.893 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.893 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:14.893 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.893 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.893 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.893 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:45963 2.b 2:d2808d5e:::benchmark_data_smithi122_107654_object45962:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-20T09:00:14.893 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:58:41.736859+0000", 2024-03-20T09:00:14.893 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 93.113898692000006, 2024-03-20T09:00:14.893 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19803585600000001, 2024-03-20T09:00:14.893 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:14.893 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:14.894 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:14.894 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:14.894 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:14.894 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 45963 2024-03-20T09:00:14.894 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.894 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:14.894 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.894 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:14.894 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.736859+0000", 2024-03-20T09:00:14.894 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.894 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.894 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.894 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:14.894 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.736859+0000", 2024-03-20T09:00:14.895 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.895 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.895 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.895 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:14.895 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.736860+0000", 2024-03-20T09:00:14.895 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5099999999999999e-07 2024-03-20T09:00:14.895 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.895 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.895 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:14.895 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.736878+0000", 2024-03-20T09:00:14.895 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8158000000000001e-05 2024-03-20T09:00:14.895 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.895 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.895 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:14.896 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.736888+0000", 2024-03-20T09:00:14.896 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0297e-05 2024-03-20T09:00:14.896 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.896 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.896 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:14.896 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.736896+0000", 2024-03-20T09:00:14.896 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3329999999999999e-06 2024-03-20T09:00:14.896 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.896 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.896 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:14.896 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.736989+0000", 2024-03-20T09:00:14.896 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3826999999999995e-05 2024-03-20T09:00:14.896 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.896 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.897 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:14.897 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.737115+0000", 2024-03-20T09:00:14.897 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012570100000000001 2024-03-20T09:00:14.897 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.897 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.897 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,11", 2024-03-20T09:00:14.897 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.737265+0000", 2024-03-20T09:00:14.897 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000150216 2024-03-20T09:00:14.897 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.897 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.897 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:14.897 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.739254+0000", 2024-03-20T09:00:14.897 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001988989 2024-03-20T09:00:14.897 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.898 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.898 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.898 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.740505+0000", 2024-03-20T09:00:14.898 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001250464 2024-03-20T09:00:14.898 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.898 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.898 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.898 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.934821+0000", 2024-03-20T09:00:14.898 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19431633200000001 2024-03-20T09:00:14.898 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.898 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.898 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:14.898 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.934857+0000", 2024-03-20T09:00:14.898 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5648000000000002e-05 2024-03-20T09:00:14.899 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.899 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.899 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:14.899 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.934895+0000", 2024-03-20T09:00:14.899 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.824e-05 2024-03-20T09:00:14.899 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.899 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:14.899 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.899 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.899 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.899 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:45969 2.b 2:d875b49b:::benchmark_data_smithi122_107654_object45968:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-20T09:00:14.899 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:58:41.740680+0000", 2024-03-20T09:00:14.899 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 93.110077619999998, 2024-03-20T09:00:14.900 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.194373935, 2024-03-20T09:00:14.900 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:14.900 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:14.900 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:14.900 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:14.900 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:14.900 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 45969 2024-03-20T09:00:14.900 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.900 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:14.900 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.900 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:14.900 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.740680+0000", 2024-03-20T09:00:14.900 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.900 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.901 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.901 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:14.901 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.740680+0000", 2024-03-20T09:00:14.901 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.901 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.901 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.901 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:14.901 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.740681+0000", 2024-03-20T09:00:14.901 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0999999999999998e-07 2024-03-20T09:00:14.901 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.901 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.901 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:14.901 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.740703+0000", 2024-03-20T09:00:14.901 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2073999999999999e-05 2024-03-20T09:00:14.901 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.902 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.902 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:14.902 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.740713+0000", 2024-03-20T09:00:14.902 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5829999999999996e-06 2024-03-20T09:00:14.902 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.902 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.902 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:14.902 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.740718+0000", 2024-03-20T09:00:14.902 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4219999999999999e-06 2024-03-20T09:00:14.902 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.902 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.902 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:14.902 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.740751+0000", 2024-03-20T09:00:14.903 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3002e-05 2024-03-20T09:00:14.903 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.903 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.903 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:14.903 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.740835+0000", 2024-03-20T09:00:14.903 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3548999999999996e-05 2024-03-20T09:00:14.903 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.903 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.903 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,11", 2024-03-20T09:00:14.903 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.740952+0000", 2024-03-20T09:00:14.903 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000117625 2024-03-20T09:00:14.903 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.903 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.903 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:14.903 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.743264+0000", 2024-03-20T09:00:14.904 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0023117189999999998 2024-03-20T09:00:14.904 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.904 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.904 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.904 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.745345+0000", 2024-03-20T09:00:14.904 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002081645 2024-03-20T09:00:14.904 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.904 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.904 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.904 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.934950+0000", 2024-03-20T09:00:14.904 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.189604936 2024-03-20T09:00:14.904 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.904 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.904 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:14.905 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.935005+0000", 2024-03-20T09:00:14.905 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4440000000000001e-05 2024-03-20T09:00:14.905 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.905 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.905 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:14.905 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.935054+0000", 2024-03-20T09:00:14.905 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9230000000000001e-05 2024-03-20T09:00:14.905 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.905 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:14.905 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.905 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.905 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.905 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:45972 2.b 2:d8091d0c:::benchmark_data_smithi122_107654_object45971:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-20T09:00:14.905 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:58:41.741861+0000", 2024-03-20T09:00:14.905 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 93.108897158000005, 2024-03-20T09:00:14.906 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.193411216, 2024-03-20T09:00:14.906 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:14.906 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:14.906 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:14.906 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:14.906 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:14.906 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 45972 2024-03-20T09:00:14.906 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.906 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:14.906 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.906 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:14.906 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.741861+0000", 2024-03-20T09:00:14.906 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.907 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.907 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.907 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:14.907 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.741861+0000", 2024-03-20T09:00:14.907 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.907 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.907 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.907 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:14.907 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.741861+0000", 2024-03-20T09:00:14.907 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4300000000000002e-07 2024-03-20T09:00:14.907 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.907 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.907 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:14.908 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.741880+0000", 2024-03-20T09:00:14.908 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8272000000000001e-05 2024-03-20T09:00:14.908 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.908 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.908 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:14.908 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.741890+0000", 2024-03-20T09:00:14.908 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0681e-05 2024-03-20T09:00:14.908 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.908 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.908 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:14.908 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.741898+0000", 2024-03-20T09:00:14.908 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6939999999999995e-06 2024-03-20T09:00:14.908 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.908 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.909 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:14.909 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.742119+0000", 2024-03-20T09:00:14.909 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000220934 2024-03-20T09:00:14.909 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.909 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.909 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:14.909 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.742229+0000", 2024-03-20T09:00:14.909 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011039699999999999 2024-03-20T09:00:14.909 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.909 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.909 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,11", 2024-03-20T09:00:14.909 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.742353+0000", 2024-03-20T09:00:14.909 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000123919 2024-03-20T09:00:14.909 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.909 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.910 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:14.910 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.745098+0000", 2024-03-20T09:00:14.910 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002744972 2024-03-20T09:00:14.910 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.910 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.910 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.910 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.745383+0000", 2024-03-20T09:00:14.910 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00028482300000000002 2024-03-20T09:00:14.910 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.910 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.910 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.910 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.935199+0000", 2024-03-20T09:00:14.910 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18981584100000001 2024-03-20T09:00:14.911 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.911 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.911 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:14.911 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.935242+0000", 2024-03-20T09:00:14.911 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2778999999999998e-05 2024-03-20T09:00:14.911 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.911 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.911 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:14.911 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.935272+0000", 2024-03-20T09:00:14.911 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0060999999999999e-05 2024-03-20T09:00:14.911 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.911 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:14.911 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.911 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.912 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.912 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:46011 2.b 2:d6099855:::benchmark_data_smithi122_107654_object46010:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-20T09:00:14.912 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:58:41.760400+0000", 2024-03-20T09:00:14.912 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 93.090357851999997, 2024-03-20T09:00:14.912 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.17582661799999999, 2024-03-20T09:00:14.912 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:14.912 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:14.912 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:14.912 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:14.912 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:14.912 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 46011 2024-03-20T09:00:14.912 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.912 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:14.912 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.912 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:14.913 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.760400+0000", 2024-03-20T09:00:14.913 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.913 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.913 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.913 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:14.913 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.760400+0000", 2024-03-20T09:00:14.913 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.913 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.913 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.913 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:14.913 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.760401+0000", 2024-03-20T09:00:14.913 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.373e-06 2024-03-20T09:00:14.913 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.913 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.914 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:14.914 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.760429+0000", 2024-03-20T09:00:14.914 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7317e-05 2024-03-20T09:00:14.914 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.914 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.914 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:14.914 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.760442+0000", 2024-03-20T09:00:14.914 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2901999999999999e-05 2024-03-20T09:00:14.914 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.914 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.914 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:14.914 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.760448+0000", 2024-03-20T09:00:14.914 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.613e-06 2024-03-20T09:00:14.914 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.914 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.915 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:14.915 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.760499+0000", 2024-03-20T09:00:14.915 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1325000000000001e-05 2024-03-20T09:00:14.915 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.915 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.915 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:14.915 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.760593+0000", 2024-03-20T09:00:14.915 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3119000000000001e-05 2024-03-20T09:00:14.915 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.915 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.915 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,11", 2024-03-20T09:00:14.915 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.760678+0000", 2024-03-20T09:00:14.915 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5385999999999999e-05 2024-03-20T09:00:14.915 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.916 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.916 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:14.916 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.762321+0000", 2024-03-20T09:00:14.916 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0016431219999999999 2024-03-20T09:00:14.916 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.916 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.916 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.916 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.762779+0000", 2024-03-20T09:00:14.916 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00045834199999999998 2024-03-20T09:00:14.916 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.916 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.916 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.916 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.936149+0000", 2024-03-20T09:00:14.916 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.17336975299999999 2024-03-20T09:00:14.916 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.917 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.917 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:14.917 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.936184+0000", 2024-03-20T09:00:14.917 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5027000000000002e-05 2024-03-20T09:00:14.917 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.917 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.917 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:14.917 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.936227+0000", 2024-03-20T09:00:14.917 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2339000000000002e-05 2024-03-20T09:00:14.917 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.917 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:14.917 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.917 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.917 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.918 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:46031 2.b 2:dc03c257:::benchmark_data_smithi122_107654_object46030:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-20T09:00:14.918 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:58:41.771436+0000", 2024-03-20T09:00:14.918 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 93.079322160000004, 2024-03-20T09:00:14.918 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.16497686, 2024-03-20T09:00:14.918 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:14.918 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:14.918 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:14.918 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:14.918 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:14.918 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 46031 2024-03-20T09:00:14.918 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.918 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:14.918 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.918 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:14.919 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.771436+0000", 2024-03-20T09:00:14.919 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.919 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.919 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.919 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:14.919 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.771436+0000", 2024-03-20T09:00:14.919 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.919 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.919 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.919 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:14.919 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.771437+0000", 2024-03-20T09:00:14.919 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1289999999999999e-06 2024-03-20T09:00:14.919 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.919 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.919 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:14.920 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.771461+0000", 2024-03-20T09:00:14.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4201000000000001e-05 2024-03-20T09:00:14.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.920 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.920 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:14.920 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.771474+0000", 2024-03-20T09:00:14.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2737e-05 2024-03-20T09:00:14.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.920 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.920 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:14.920 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.771480+0000", 2024-03-20T09:00:14.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5119999999999997e-06 2024-03-20T09:00:14.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.920 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.921 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:14.921 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.771535+0000", 2024-03-20T09:00:14.921 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4950000000000001e-05 2024-03-20T09:00:14.921 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.921 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.921 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:14.921 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.771684+0000", 2024-03-20T09:00:14.921 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014921699999999999 2024-03-20T09:00:14.921 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.921 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.921 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,11", 2024-03-20T09:00:14.921 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.771832+0000", 2024-03-20T09:00:14.921 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000147229 2024-03-20T09:00:14.921 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.922 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.922 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:14.922 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.775077+0000", 2024-03-20T09:00:14.922 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003245066 2024-03-20T09:00:14.922 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.922 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.922 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.922 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.775112+0000", 2024-03-20T09:00:14.922 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4975000000000003e-05 2024-03-20T09:00:14.922 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.922 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.922 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.922 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.936277+0000", 2024-03-20T09:00:14.922 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.16116520300000001 2024-03-20T09:00:14.923 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.923 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.923 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:14.923 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.936358+0000", 2024-03-20T09:00:14.923 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1163999999999997e-05 2024-03-20T09:00:14.923 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.923 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.923 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:14.923 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.936412+0000", 2024-03-20T09:00:14.923 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4477e-05 2024-03-20T09:00:14.923 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.923 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:14.923 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.923 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.924 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.924 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:46035 2.b 2:dfb4166b:::benchmark_data_smithi122_107654_object46034:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-20T09:00:14.924 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:58:41.773372+0000", 2024-03-20T09:00:14.924 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 93.077385754999995, 2024-03-20T09:00:14.924 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.16478121000000001, 2024-03-20T09:00:14.924 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:14.924 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:14.924 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:14.924 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:14.924 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:14.924 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 46035 2024-03-20T09:00:14.924 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.924 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:14.924 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.924 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:14.925 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.773372+0000", 2024-03-20T09:00:14.925 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.925 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.925 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.925 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:14.925 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.773372+0000", 2024-03-20T09:00:14.925 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.925 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.925 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.925 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:14.925 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.773373+0000", 2024-03-20T09:00:14.925 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.201e-06 2024-03-20T09:00:14.925 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.925 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:14.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.773397+0000", 2024-03-20T09:00:14.926 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3516e-05 2024-03-20T09:00:14.926 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.926 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:14.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.773411+0000", 2024-03-20T09:00:14.926 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4713e-05 2024-03-20T09:00:14.926 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.926 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:14.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.773420+0000", 2024-03-20T09:00:14.926 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9339999999999995e-06 2024-03-20T09:00:14.926 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.927 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.927 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:14.927 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.773472+0000", 2024-03-20T09:00:14.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1749000000000002e-05 2024-03-20T09:00:14.927 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.927 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.927 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:14.927 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.773600+0000", 2024-03-20T09:00:14.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012749199999999999 2024-03-20T09:00:14.927 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.927 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.927 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,11", 2024-03-20T09:00:14.928 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.773735+0000", 2024-03-20T09:00:14.928 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000135115 2024-03-20T09:00:14.928 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.928 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.928 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.928 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.775150+0000", 2024-03-20T09:00:14.928 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014154580000000001 2024-03-20T09:00:14.928 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.928 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.928 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:14.928 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.775338+0000", 2024-03-20T09:00:14.928 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00018755900000000001 2024-03-20T09:00:14.928 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.929 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.929 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.929 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.938067+0000", 2024-03-20T09:00:14.929 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.16272925499999999 2024-03-20T09:00:14.929 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.929 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.929 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:14.929 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.938117+0000", 2024-03-20T09:00:14.929 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0297000000000003e-05 2024-03-20T09:00:14.929 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.929 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.929 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:14.930 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.938153+0000", 2024-03-20T09:00:14.930 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5920999999999997e-05 2024-03-20T09:00:14.930 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.930 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:14.930 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.930 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.930 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.930 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:46046 2.b 2:d4b31499:::benchmark_data_smithi122_107654_object46045:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-20T09:00:14.930 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:58:41.779337+0000", 2024-03-20T09:00:14.930 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 93.071420364000005, 2024-03-20T09:00:14.930 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.15895119199999999, 2024-03-20T09:00:14.930 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:14.930 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:14.930 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:14.931 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:14.931 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:14.931 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 46046 2024-03-20T09:00:14.931 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.931 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:14.931 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.931 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:14.931 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.779337+0000", 2024-03-20T09:00:14.931 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.931 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.931 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.931 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:14.931 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.779337+0000", 2024-03-20T09:00:14.931 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.932 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.932 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.932 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:14.932 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.779339+0000", 2024-03-20T09:00:14.932 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3459999999999999e-06 2024-03-20T09:00:14.932 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.932 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.932 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:14.932 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.779360+0000", 2024-03-20T09:00:14.932 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1491000000000001e-05 2024-03-20T09:00:14.932 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.932 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.932 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:14.932 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.779373+0000", 2024-03-20T09:00:14.932 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2607e-05 2024-03-20T09:00:14.933 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.933 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:14.933 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.779380+0000", 2024-03-20T09:00:14.933 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3719999999999999e-06 2024-03-20T09:00:14.933 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.933 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:14.933 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.779430+0000", 2024-03-20T09:00:14.933 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9302000000000003e-05 2024-03-20T09:00:14.933 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.933 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:14.933 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.779536+0000", 2024-03-20T09:00:14.933 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000106007 2024-03-20T09:00:14.934 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.934 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.934 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,11", 2024-03-20T09:00:14.934 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.779618+0000", 2024-03-20T09:00:14.934 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2193000000000004e-05 2024-03-20T09:00:14.934 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.934 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.934 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:14.934 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.781794+0000", 2024-03-20T09:00:14.934 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0021761570000000002 2024-03-20T09:00:14.934 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.934 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.934 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.934 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.782212+0000", 2024-03-20T09:00:14.935 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000418525 2024-03-20T09:00:14.935 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.935 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.935 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.935 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.938201+0000", 2024-03-20T09:00:14.935 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.15598822500000001 2024-03-20T09:00:14.935 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.935 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.935 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:14.935 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.938249+0000", 2024-03-20T09:00:14.935 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8671000000000002e-05 2024-03-20T09:00:14.935 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.935 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.935 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:14.935 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.938289+0000", 2024-03-20T09:00:14.936 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9295999999999998e-05 2024-03-20T09:00:14.936 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.936 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:14.936 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.936 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.936 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.936 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:46049 2.b 2:dd5832d0:::benchmark_data_smithi122_107654_object46048:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-20T09:00:14.936 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:58:41.780028+0000", 2024-03-20T09:00:14.936 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 93.070729771000003, 2024-03-20T09:00:14.936 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.15985512700000001, 2024-03-20T09:00:14.936 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:14.936 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:14.936 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:14.936 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:14.937 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:14.937 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 46049 2024-03-20T09:00:14.937 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.937 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:14.937 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.937 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:14.937 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.780028+0000", 2024-03-20T09:00:14.937 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.937 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.937 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.937 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:14.937 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.780028+0000", 2024-03-20T09:00:14.937 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.937 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.938 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.938 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:14.938 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.780029+0000", 2024-03-20T09:00:14.938 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3e-07 2024-03-20T09:00:14.938 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.938 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.938 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:14.938 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.780048+0000", 2024-03-20T09:00:14.938 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9086999999999999e-05 2024-03-20T09:00:14.938 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.938 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.938 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:14.938 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.780059+0000", 2024-03-20T09:00:14.939 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1092e-05 2024-03-20T09:00:14.939 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.939 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.939 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:14.939 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.780066+0000", 2024-03-20T09:00:14.939 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9160000000000004e-06 2024-03-20T09:00:14.939 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.939 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.939 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:14.939 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.780853+0000", 2024-03-20T09:00:14.939 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00078682700000000003 2024-03-20T09:00:14.939 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.939 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.939 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:14.939 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.780922+0000", 2024-03-20T09:00:14.940 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9356000000000005e-05 2024-03-20T09:00:14.940 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.940 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.940 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,11", 2024-03-20T09:00:14.940 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.781012+0000", 2024-03-20T09:00:14.940 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9628000000000003e-05 2024-03-20T09:00:14.940 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.940 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.940 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:14.940 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.781939+0000", 2024-03-20T09:00:14.940 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00092758000000000001 2024-03-20T09:00:14.940 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.940 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.940 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.941 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.782394+0000", 2024-03-20T09:00:14.941 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00045468100000000002 2024-03-20T09:00:14.941 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.941 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.941 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.941 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.939785+0000", 2024-03-20T09:00:14.941 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.15739067300000001 2024-03-20T09:00:14.941 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.941 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.941 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:14.941 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.939839+0000", 2024-03-20T09:00:14.941 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4554e-05 2024-03-20T09:00:14.941 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.941 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.941 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:14.942 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.939883+0000", 2024-03-20T09:00:14.942 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4002999999999999e-05 2024-03-20T09:00:14.942 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.942 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:14.942 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.942 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.942 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.942 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:46060 2.b 2:dcad56ac:::benchmark_data_smithi122_107654_object46059:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-20T09:00:14.942 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:58:41.785384+0000", 2024-03-20T09:00:14.942 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 93.065374083999998, 2024-03-20T09:00:14.942 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.155417148, 2024-03-20T09:00:14.942 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:14.942 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:14.942 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:14.943 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:14.943 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:14.943 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 46060 2024-03-20T09:00:14.943 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.943 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:14.943 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.943 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:14.943 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.785384+0000", 2024-03-20T09:00:14.943 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.943 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.943 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.943 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:14.943 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.785384+0000", 2024-03-20T09:00:14.943 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.944 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.944 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.944 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:14.944 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.785384+0000", 2024-03-20T09:00:14.944 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4399999999999999e-07 2024-03-20T09:00:14.944 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.944 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.944 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:14.944 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.785405+0000", 2024-03-20T09:00:14.944 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1036e-05 2024-03-20T09:00:14.944 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.944 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.944 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:14.944 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.785412+0000", 2024-03-20T09:00:14.944 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6869999999999997e-06 2024-03-20T09:00:14.945 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.945 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.945 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:14.945 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.785417+0000", 2024-03-20T09:00:14.945 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4730000000000002e-06 2024-03-20T09:00:14.945 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.945 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.945 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:14.945 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.786616+0000", 2024-03-20T09:00:14.945 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001199061 2024-03-20T09:00:14.945 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.945 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.945 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:14.945 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.786735+0000", 2024-03-20T09:00:14.946 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000119101 2024-03-20T09:00:14.946 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.946 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.946 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,11", 2024-03-20T09:00:14.946 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.786842+0000", 2024-03-20T09:00:14.946 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000107414 2024-03-20T09:00:14.946 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.946 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.946 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:14.946 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.788237+0000", 2024-03-20T09:00:14.946 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0013949240000000001 2024-03-20T09:00:14.946 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.946 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.946 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.947 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.788525+0000", 2024-03-20T09:00:14.947 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000287662 2024-03-20T09:00:14.947 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.947 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.947 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.947 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.940725+0000", 2024-03-20T09:00:14.947 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.152200116 2024-03-20T09:00:14.947 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.947 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.947 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:14.947 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.940763+0000", 2024-03-20T09:00:14.947 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.782e-05 2024-03-20T09:00:14.947 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.947 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.947 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:14.948 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.940801+0000", 2024-03-20T09:00:14.948 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8109999999999999e-05 2024-03-20T09:00:14.948 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.948 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:14.948 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.948 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.948 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.948 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:116068 2.b 2:d4c3e9d8:::benchmark_data_smithi122_107654_object116067:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-03-20T09:00:14.948 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:59:50.846349+0000", 2024-03-20T09:00:14.948 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 24.004409119000002, 2024-03-20T09:00:14.948 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20256579399999999, 2024-03-20T09:00:14.948 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:14.948 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:14.948 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:14.948 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:14.949 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:14.949 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 116068 2024-03-20T09:00:14.949 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.949 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:14.949 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.949 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:14.949 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.846349+0000", 2024-03-20T09:00:14.949 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.949 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.949 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.949 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:14.949 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.846349+0000", 2024-03-20T09:00:14.949 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.949 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.950 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.950 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:14.950 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.846349+0000", 2024-03-20T09:00:14.950 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2800000000000002e-07 2024-03-20T09:00:14.950 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.950 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.950 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:14.950 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.846369+0000", 2024-03-20T09:00:14.950 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0156000000000001e-05 2024-03-20T09:00:14.950 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.950 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.950 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:14.950 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.846375+0000", 2024-03-20T09:00:14.951 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2430000000000002e-06 2024-03-20T09:00:14.951 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.951 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.951 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:14.951 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.846381+0000", 2024-03-20T09:00:14.951 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1900000000000003e-06 2024-03-20T09:00:14.951 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.951 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.951 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:14.951 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.846436+0000", 2024-03-20T09:00:14.951 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5296e-05 2024-03-20T09:00:14.951 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.951 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.951 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:14.951 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.846559+0000", 2024-03-20T09:00:14.952 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012265299999999999 2024-03-20T09:00:14.952 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.952 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.952 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,11", 2024-03-20T09:00:14.952 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.846698+0000", 2024-03-20T09:00:14.952 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013973600000000001 2024-03-20T09:00:14.952 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.952 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.952 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:14.952 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.852495+0000", 2024-03-20T09:00:14.952 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0057968539999999997 2024-03-20T09:00:14.952 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.952 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.952 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.953 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.853877+0000", 2024-03-20T09:00:14.953 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0013814599999999999 2024-03-20T09:00:14.953 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.953 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.953 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.953 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.048826+0000", 2024-03-20T09:00:14.953 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19494949 2024-03-20T09:00:14.953 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.953 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.953 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:14.953 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.048867+0000", 2024-03-20T09:00:14.953 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.091e-05 2024-03-20T09:00:14.953 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.954 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.954 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:14.954 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.048914+0000", 2024-03-20T09:00:14.954 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7377999999999999e-05 2024-03-20T09:00:14.954 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.954 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:14.954 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.954 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.954 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.954 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:116083 2.b 2:dcf6575c:::benchmark_data_smithi122_107654_object116082:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-03-20T09:00:14.954 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:59:50.860693+0000", 2024-03-20T09:00:14.954 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 23.990065165000001, 2024-03-20T09:00:14.954 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19728431599999999, 2024-03-20T09:00:14.954 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:14.955 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:14.955 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:14.955 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:14.955 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:14.955 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 116083 2024-03-20T09:00:14.955 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.955 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:14.955 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.955 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:14.955 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.860693+0000", 2024-03-20T09:00:14.955 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.955 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.955 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.955 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:14.956 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.860693+0000", 2024-03-20T09:00:14.956 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.956 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.956 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.956 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:14.956 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.860693+0000", 2024-03-20T09:00:14.956 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4199999999999995e-07 2024-03-20T09:00:14.956 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.956 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.956 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:14.956 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.860711+0000", 2024-03-20T09:00:14.956 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7963999999999999e-05 2024-03-20T09:00:14.956 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.956 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.956 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:14.957 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.860721+0000", 2024-03-20T09:00:14.957 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4350000000000003e-06 2024-03-20T09:00:14.957 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.957 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.957 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:14.957 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.860727+0000", 2024-03-20T09:00:14.957 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7780000000000004e-06 2024-03-20T09:00:14.957 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.957 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.957 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:14.957 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.860771+0000", 2024-03-20T09:00:14.957 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4039000000000003e-05 2024-03-20T09:00:14.957 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.957 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.958 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:14.958 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.860851+0000", 2024-03-20T09:00:14.958 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0817999999999998e-05 2024-03-20T09:00:14.958 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.958 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.958 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,11", 2024-03-20T09:00:14.958 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.860977+0000", 2024-03-20T09:00:14.958 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012594700000000001 2024-03-20T09:00:14.958 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.958 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.958 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:14.958 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.863719+0000", 2024-03-20T09:00:14.958 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027417460000000002 2024-03-20T09:00:14.958 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.959 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.959 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.959 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.866474+0000", 2024-03-20T09:00:14.959 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027546929999999999 2024-03-20T09:00:14.959 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.959 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.959 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.959 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.057869+0000", 2024-03-20T09:00:14.959 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19139492699999999 2024-03-20T09:00:14.959 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.959 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.959 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:14.959 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.057931+0000", 2024-03-20T09:00:14.959 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2445000000000003e-05 2024-03-20T09:00:14.960 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.960 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.960 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:14.960 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.057977+0000", 2024-03-20T09:00:14.960 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5782000000000003e-05 2024-03-20T09:00:14.960 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.960 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:14.960 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.960 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.960 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.960 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:116090 2.b 2:d8f8696b:::benchmark_data_smithi122_107654_object116089:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-03-20T09:00:14.960 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:59:50.868553+0000", 2024-03-20T09:00:14.960 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 23.982205275999998, 2024-03-20T09:00:14.960 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19503764200000001, 2024-03-20T09:00:14.960 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:14.961 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:14.961 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:14.961 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:14.961 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:14.961 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 116090 2024-03-20T09:00:14.961 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.961 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:14.961 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.961 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:14.961 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.868553+0000", 2024-03-20T09:00:14.961 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.961 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.961 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.961 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:14.962 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.868553+0000", 2024-03-20T09:00:14.962 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.962 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.962 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.962 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:14.962 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.868554+0000", 2024-03-20T09:00:14.962 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.517e-06 2024-03-20T09:00:14.962 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.962 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.962 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:14.962 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.868573+0000", 2024-03-20T09:00:14.962 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9397000000000001e-05 2024-03-20T09:00:14.962 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.962 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.962 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:14.963 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.868585+0000", 2024-03-20T09:00:14.963 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2016e-05 2024-03-20T09:00:14.963 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.963 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.963 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:14.963 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.868593+0000", 2024-03-20T09:00:14.963 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7340000000000008e-06 2024-03-20T09:00:14.963 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.963 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.963 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:14.963 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.868642+0000", 2024-03-20T09:00:14.963 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9298999999999997e-05 2024-03-20T09:00:14.963 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.964 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.964 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:14.964 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.868748+0000", 2024-03-20T09:00:14.964 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105164 2024-03-20T09:00:14.964 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.964 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.964 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,11", 2024-03-20T09:00:14.964 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.868886+0000", 2024-03-20T09:00:14.964 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000138696 2024-03-20T09:00:14.964 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.964 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.964 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:14.964 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.871701+0000", 2024-03-20T09:00:14.964 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0028147290000000002 2024-03-20T09:00:14.964 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.965 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.965 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.965 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.871796+0000", 2024-03-20T09:00:14.965 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4596e-05 2024-03-20T09:00:14.965 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.965 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.965 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.965 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.063503+0000", 2024-03-20T09:00:14.965 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19170742599999999 2024-03-20T09:00:14.965 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.965 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.965 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:14.965 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.063549+0000", 2024-03-20T09:00:14.965 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6158000000000001e-05 2024-03-20T09:00:14.966 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.966 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.966 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:14.966 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.063590+0000", 2024-03-20T09:00:14.966 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.091e-05 2024-03-20T09:00:14.966 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.966 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:14.966 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.966 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.966 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.966 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:116097 2.b 2:dcc82db7:::benchmark_data_smithi122_107654_object116096:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-03-20T09:00:14.966 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:59:50.881259+0000", 2024-03-20T09:00:14.967 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 23.969499207999998, 2024-03-20T09:00:14.967 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.182490387, 2024-03-20T09:00:14.967 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:14.967 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:14.967 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:14.967 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:14.967 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:14.967 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 116097 2024-03-20T09:00:14.967 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.968 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:14.968 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.968 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:14.968 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.881259+0000", 2024-03-20T09:00:14.968 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.968 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.968 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.968 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:14.968 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.881259+0000", 2024-03-20T09:00:14.968 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.968 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.968 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.968 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:14.968 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.881260+0000", 2024-03-20T09:00:14.969 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4249999999999999e-06 2024-03-20T09:00:14.969 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.969 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.969 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:14.969 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.881291+0000", 2024-03-20T09:00:14.969 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1489999999999998e-05 2024-03-20T09:00:14.969 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.969 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.969 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:14.969 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.881306+0000", 2024-03-20T09:00:14.969 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4092e-05 2024-03-20T09:00:14.969 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.969 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.969 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:14.969 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.881315+0000", 2024-03-20T09:00:14.970 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5999999999999996e-06 2024-03-20T09:00:14.970 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.970 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.970 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:14.970 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.881379+0000", 2024-03-20T09:00:14.970 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3346e-05 2024-03-20T09:00:14.970 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.970 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.970 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:14.970 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.881524+0000", 2024-03-20T09:00:14.970 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000145759 2024-03-20T09:00:14.970 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.970 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.970 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,11", 2024-03-20T09:00:14.971 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.881682+0000", 2024-03-20T09:00:14.971 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000158126 2024-03-20T09:00:14.971 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.971 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.971 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:14.971 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.884410+0000", 2024-03-20T09:00:14.971 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027277870000000002 2024-03-20T09:00:14.971 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.971 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.971 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.971 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.885722+0000", 2024-03-20T09:00:14.972 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0013118400000000001 2024-03-20T09:00:14.972 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.972 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.972 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.972 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.063641+0000", 2024-03-20T09:00:14.972 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.17791922499999999 2024-03-20T09:00:14.972 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.972 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.972 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:14.972 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.063704+0000", 2024-03-20T09:00:14.972 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3015000000000006e-05 2024-03-20T09:00:14.972 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.972 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.972 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:14.972 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.063749+0000", 2024-03-20T09:00:14.973 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4681999999999997e-05 2024-03-20T09:00:14.973 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.973 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:14.973 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.973 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.973 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.973 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:116107 2.b 2:de6b9bcd:::benchmark_data_smithi122_107654_object116106:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-03-20T09:00:14.973 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:59:50.888820+0000", 2024-03-20T09:00:14.973 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 23.961937487, 2024-03-20T09:00:14.973 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.175065884, 2024-03-20T09:00:14.973 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:14.973 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:14.973 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:14.973 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:14.974 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:14.974 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 116107 2024-03-20T09:00:14.974 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.974 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:14.974 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.974 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:14.974 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.888820+0000", 2024-03-20T09:00:14.974 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.974 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.974 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.974 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:14.974 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.888820+0000", 2024-03-20T09:00:14.974 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:14.974 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.975 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.975 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:14.975 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.888822+0000", 2024-03-20T09:00:14.975 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3790000000000001e-06 2024-03-20T09:00:14.975 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.975 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.975 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:14.975 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.888844+0000", 2024-03-20T09:00:14.975 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2010000000000001e-05 2024-03-20T09:00:14.975 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.975 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.975 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:14.975 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.888857+0000", 2024-03-20T09:00:14.975 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3675e-05 2024-03-20T09:00:14.975 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.976 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.976 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:14.976 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.888865+0000", 2024-03-20T09:00:14.976 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7090000000000002e-06 2024-03-20T09:00:14.976 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.976 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.976 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:14.976 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.888916+0000", 2024-03-20T09:00:14.976 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0494000000000001e-05 2024-03-20T09:00:14.976 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.976 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.976 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:14.976 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.889041+0000", 2024-03-20T09:00:14.976 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000125838 2024-03-20T09:00:14.977 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.977 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.977 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,11", 2024-03-20T09:00:14.977 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.889196+0000", 2024-03-20T09:00:14.977 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015424699999999999 2024-03-20T09:00:14.977 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.977 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.977 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:14.977 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.891889+0000", 2024-03-20T09:00:14.977 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0026937060000000001 2024-03-20T09:00:14.977 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.977 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.977 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.977 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.894398+0000", 2024-03-20T09:00:14.977 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0025083200000000001 2024-03-20T09:00:14.978 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.978 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.978 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:14.978 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.063797+0000", 2024-03-20T09:00:14.978 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.16939885700000001 2024-03-20T09:00:14.978 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.978 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.978 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:14.978 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.063860+0000", 2024-03-20T09:00:14.978 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3211000000000002e-05 2024-03-20T09:00:14.978 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:14.978 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:14.978 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:14.978 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.063886+0000", 2024-03-20T09:00:14.978 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6438e-05 2024-03-20T09:00:14.979 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.979 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:14.979 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.979 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:14.979 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:14.979 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:00:14.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:14.980 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:00:14.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:00:15.081 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:00:15.122 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:15.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:14 smithi122 ceph-mon[73495]: pgmap v463: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 947 op/s 2024-03-20T09:00:15.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:14 smithi026 ceph-mon[68309]: pgmap v463: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 947 op/s 2024-03-20T09:00:15.285 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:00:15.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:15.296 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:00:15.397 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:00:15.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:14 smithi115 ceph-mon[72476]: pgmap v463: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 947 op/s 2024-03-20T09:00:15.427 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:15.448 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:00:15.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:15.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:00:15.551 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:00:15.581 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:15.617 INFO:teuthology.orchestra.run.smithi026.stderr:no valid command found; 10 closest matches: 2024-03-20T09:00:15.617 INFO:teuthology.orchestra.run.smithi026.stderr:0 2024-03-20T09:00:15.617 INFO:teuthology.orchestra.run.smithi026.stderr:1 2024-03-20T09:00:15.617 INFO:teuthology.orchestra.run.smithi026.stderr:2 2024-03-20T09:00:15.618 INFO:teuthology.orchestra.run.smithi026.stderr:abort 2024-03-20T09:00:15.618 INFO:teuthology.orchestra.run.smithi026.stderr:assert 2024-03-20T09:00:15.618 INFO:teuthology.orchestra.run.smithi026.stderr:bench [] [] [] [] 2024-03-20T09:00:15.618 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs debug_inject_read_zeros 2024-03-20T09:00:15.618 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs files list 2024-03-20T09:00:15.618 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs stats 2024-03-20T09:00:15.618 INFO:teuthology.orchestra.run.smithi026.stderr:bluestore allocator dump block 2024-03-20T09:00:15.618 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: invalid command 2024-03-20T09:00:15.783 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:00:15.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:15.787 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:00:15.788 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_ops_in_flight 2024-03-20T09:00:15.888 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:00:15.945 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:16.036 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:00:16.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:16.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:00:16.145 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:00:16.174 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:16.191 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:00:16.197 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:00:16.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:16.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:00:16.301 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:00:16.370 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:16.583 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:00:17.059 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:00:17.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:17.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:00:17.083 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T09:00:17.084 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-20T09:00:17.084 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.11 'trim stale osdmaps' 2024-03-20T09:00:17.165 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:00:17.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:16 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:00:17.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:16 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:00:17.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:16 smithi122 ceph-mon[73495]: pgmap v464: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s 2024-03-20T09:00:17.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:16 smithi122 ceph-mon[73495]: 2.4 scrub ok 2024-03-20T09:00:17.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:16 smithi122 ceph-mon[73495]: osdmap e155: 12 total, 12 up, 12 in 2024-03-20T09:00:17.256 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:16 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:00:17.256 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:16 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:00:17.256 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:16 smithi026 ceph-mon[68309]: pgmap v464: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s 2024-03-20T09:00:17.256 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:16 smithi026 ceph-mon[68309]: 2.4 scrub ok 2024-03-20T09:00:17.257 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:16 smithi026 ceph-mon[68309]: osdmap e155: 12 total, 12 up, 12 in 2024-03-20T09:00:17.259 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:17.299 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:00:17.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:17.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:00:17.401 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:00:17.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:16 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:00:17.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:16 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:00:17.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:16 smithi115 ceph-mon[72476]: pgmap v464: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.01k op/s 2024-03-20T09:00:17.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:16 smithi115 ceph-mon[72476]: 2.4 scrub ok 2024-03-20T09:00:17.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:16 smithi115 ceph-mon[72476]: osdmap e155: 12 total, 12 up, 12 in 2024-03-20T09:00:17.440 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:17.797 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:00:17.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:17.801 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:00:17.902 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:00:17.949 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:17.974 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:00:18.025 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:00:18.075 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:00:18.110 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:00:18.115 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:00:18.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:18.118 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:00:18.155 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:00:18.183 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:00:18.219 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:00:18.224 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:00:18.278 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:00:18.305 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:18 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/142482995' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:00:18.317 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:00:18.320 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:18.370 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:00:18.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:18 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/142482995' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:00:18.407 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:00:18.442 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:00:18.442 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:00:18.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:18 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/142482995' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:00:18.457 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:00:18.466 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:00:18.466 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:00:18.466 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:00:18.466 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:00:18.467 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:00:18.467 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:00:18.467 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:00:18.467 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:00:18.467 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:00:18.467 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:00:18.467 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:00:18.467 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:00:19.177 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:19 smithi122 ceph-mon[73495]: pgmap v466: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 991 op/s 2024-03-20T09:00:19.208 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:00:19.214 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_blocked_ops 2024-03-20T09:00:19.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:19.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:00:19.330 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:00:19.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:19 smithi115 ceph-mon[72476]: pgmap v466: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 991 op/s 2024-03-20T09:00:19.406 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:19 smithi026 ceph-mon[68309]: pgmap v466: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 6.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 991 op/s 2024-03-20T09:00:19.411 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:19.586 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:00:19.604 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:00:19.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:19.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:00:19.711 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:00:19.748 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:19.795 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:00:19.928 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:00:19.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:19.936 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:00:20.037 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:00:20.046 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:00:20.096 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:20.166 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:00:20.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:20.172 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:00:20.273 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:00:20.347 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:20 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2305047918' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:00:20.347 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:20 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:00:20.350 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:20.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:20 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2305047918' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:00:20.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:20 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:00:20.435 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:00:20.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:20.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:00:20.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:20 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2305047918' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:00:20.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:20 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:00:20.539 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:00:20.566 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:20.584 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:00:20.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:20.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:00:20.689 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:00:20.722 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:20.744 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:00:20.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:20.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:00:20.849 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:00:20.886 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:21.094 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:00:21.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:21.098 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:00:21.105 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:21 smithi026 ceph-mon[68309]: pgmap v467: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 986 op/s 2024-03-20T09:00:21.105 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:21 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:00:21.105 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:21 smithi026 ceph-mon[68309]: Cluster is now healthy 2024-03-20T09:00:21.105 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:21 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:00:21.106 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:21 smithi026 ceph-mon[68309]: osdmap e156: 12 total, 12 up, 12 in 2024-03-20T09:00:21.199 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:00:21.229 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:21.251 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:00:21.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:21.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:00:21.356 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:00:21.386 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:21 smithi115 ceph-mon[72476]: pgmap v467: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 986 op/s 2024-03-20T09:00:21.386 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:21 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:00:21.386 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:21 smithi115 ceph-mon[72476]: Cluster is now healthy 2024-03-20T09:00:21.386 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:21 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:00:21.386 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:21 smithi115 ceph-mon[72476]: osdmap e156: 12 total, 12 up, 12 in 2024-03-20T09:00:21.387 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:21.413 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:00:21.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:21.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:00:21.424 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:21 smithi122 ceph-mon[73495]: pgmap v467: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 5.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 986 op/s 2024-03-20T09:00:21.424 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:21 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:00:21.424 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:21 smithi122 ceph-mon[73495]: Cluster is now healthy 2024-03-20T09:00:21.424 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:21 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:00:21.424 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:21 smithi122 ceph-mon[73495]: osdmap e156: 12 total, 12 up, 12 in 2024-03-20T09:00:21.517 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:00:21.562 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:21.644 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:00:21.666 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:00:21.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:21.671 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:00:21.772 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:00:21.829 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:00:21.829 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:00:21.892 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:22.061 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:00:22.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:22.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:00:22.171 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:00:22.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:22 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:00:22.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:22 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:00:22.205 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:22.232 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:00:22.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:22.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:00:22.337 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:00:22.378 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:22.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:22 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:00:22.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:22 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:00:22.459 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:00:22.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:22 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:00:22.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:22 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:00:22.541 INFO:teuthology.orchestra.run.smithi026.stderr:no valid command found; 10 closest matches: 2024-03-20T09:00:22.541 INFO:teuthology.orchestra.run.smithi026.stderr:0 2024-03-20T09:00:22.541 INFO:teuthology.orchestra.run.smithi026.stderr:1 2024-03-20T09:00:22.541 INFO:teuthology.orchestra.run.smithi026.stderr:2 2024-03-20T09:00:22.541 INFO:teuthology.orchestra.run.smithi026.stderr:abort 2024-03-20T09:00:22.541 INFO:teuthology.orchestra.run.smithi026.stderr:assert 2024-03-20T09:00:22.541 INFO:teuthology.orchestra.run.smithi026.stderr:bench [] [] [] [] 2024-03-20T09:00:22.541 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs debug_inject_read_zeros 2024-03-20T09:00:22.541 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs files list 2024-03-20T09:00:22.541 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs stats 2024-03-20T09:00:22.541 INFO:teuthology.orchestra.run.smithi026.stderr:bluestore allocator dump block 2024-03-20T09:00:22.541 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: invalid command 2024-03-20T09:00:22.740 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:00:22.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:22.745 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:00:22.746 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_historic_ops 2024-03-20T09:00:22.846 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:00:22.874 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:22.896 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:00:22.898 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:22.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:00:23.000 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:00:23.037 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:23.077 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:00:23.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:23.083 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:00:23.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:23 smithi115 ceph-mon[72476]: pgmap v469: 17 pgs: 17 active+clean; 5.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 916 op/s 2024-03-20T09:00:23.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:23 smithi115 ceph-mon[72476]: osdmap e157: 12 total, 12 up, 12 in 2024-03-20T09:00:23.159 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:00:23.184 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:00:23.220 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:23.242 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:00:23.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:23.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:00:23.350 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:00:23.381 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:23.404 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:00:23.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:23.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:00:23.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:23 smithi122 ceph-mon[73495]: pgmap v469: 17 pgs: 17 active+clean; 5.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 916 op/s 2024-03-20T09:00:23.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:23 smithi122 ceph-mon[73495]: osdmap e157: 12 total, 12 up, 12 in 2024-03-20T09:00:23.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:23 smithi026 ceph-mon[68309]: pgmap v469: 17 pgs: 17 active+clean; 5.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 916 op/s 2024-03-20T09:00:23.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:23 smithi026 ceph-mon[68309]: osdmap e157: 12 total, 12 up, 12 in 2024-03-20T09:00:23.509 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:00:23.538 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:23.560 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:00:23.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:23.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:00:23.671 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:00:23.710 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:23.736 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:00:24.166 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:00:24.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:24.171 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:00:24.272 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:00:24.307 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:24.410 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T09:00:24.411 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-20T09:00:24.772 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:00:24.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:24.778 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:00:24.880 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:00:24.957 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:25.008 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:00:25.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:25.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:00:25.081 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:00:25.119 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:00:25.149 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:00:25.158 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:25.202 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:00:25.238 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:00:25.289 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:00:25.330 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:00:25.372 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:00:25.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:25 smithi115 ceph-mon[72476]: pgmap v471: 17 pgs: 17 active+clean; 5.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 949 op/s 2024-03-20T09:00:25.416 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:00:25.427 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:25 smithi026 ceph-mon[68309]: pgmap v471: 17 pgs: 17 active+clean; 5.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 949 op/s 2024-03-20T09:00:25.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:25 smithi122 ceph-mon[73495]: pgmap v471: 17 pgs: 17 active+clean; 5.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 949 op/s 2024-03-20T09:00:25.456 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:00:25.496 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:00:25.514 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:00:25.514 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:00:25.520 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:00:25.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:25.528 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:00:25.529 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:00:25.580 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:00:25.629 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:00:25.630 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:00:25.631 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:00:25.632 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:00:25.632 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:00:25.632 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:00:25.632 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:00:25.632 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:00:25.632 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:00:25.632 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:00:25.632 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:00:25.632 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:00:25.632 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:00:25.632 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:00:25.670 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:25.803 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:00:25.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:25.809 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:00:25.910 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:00:25.943 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:25.968 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:00:25.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:25.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:00:26.073 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:00:26.095 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:26 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2256060821' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:00:26.095 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:26 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2256060821' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:00:26.110 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:26.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:26 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2256060821' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:00:26.650 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_ops_in_flight 2024-03-20T09:00:26.670 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:00:26.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:26.673 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:00:26.774 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:00:26.808 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:26.837 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:00:26.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:26.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:00:26.942 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:00:26.973 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:26.993 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:00:26.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:26.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:00:27.065 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:00:27.096 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:00:27.147 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:27.196 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:00:27.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:27.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:00:27.308 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:00:27.352 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:27.378 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:00:27.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:27.385 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:00:27.385 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:27 smithi122 ceph-mon[73495]: pgmap v472: 17 pgs: 17 active+clean; 5.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 926 op/s 2024-03-20T09:00:27.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:27 smithi115 ceph-mon[72476]: pgmap v472: 17 pgs: 17 active+clean; 5.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 926 op/s 2024-03-20T09:00:27.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:27 smithi026 ceph-mon[68309]: pgmap v472: 17 pgs: 17 active+clean; 5.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 926 op/s 2024-03-20T09:00:27.486 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:00:27.523 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:27.611 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:00:27.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:27.616 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:00:27.717 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:00:27.807 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:27.846 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:00:27.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:27.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:00:27.949 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:00:27.976 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:27.997 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:00:28.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:28.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:00:28.107 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:00:28.132 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:28 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:00:28.132 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:28 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:00:28.132 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:28 smithi026 ceph-mon[68309]: osdmap e158: 12 total, 12 up, 12 in 2024-03-20T09:00:28.142 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:28.165 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:00:28.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:28.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:00:28.272 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:00:28.321 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:28.365 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:00:28.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:28.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:00:28.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:28 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:00:28.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:28 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:00:28.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:28 smithi115 ceph-mon[72476]: osdmap e158: 12 total, 12 up, 12 in 2024-03-20T09:00:28.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:28 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:00:28.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:28 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:00:28.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:28 smithi122 ceph-mon[73495]: osdmap e158: 12 total, 12 up, 12 in 2024-03-20T09:00:28.472 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:00:28.504 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:28.533 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:00:28.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:28.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:00:28.637 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:00:28.690 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:28.727 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:00:28.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:28.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:00:28.838 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:00:28.890 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:28.933 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:00:28.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:28.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:00:29.039 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:00:29.087 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:29.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:29.153 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:00:29.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:00:29.187 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:00:29.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:29 smithi026 ceph-mon[68309]: pgmap v474: 17 pgs: 17 active+clean; 5.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 900 op/s 2024-03-20T09:00:29.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:29 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2732984925' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:00:29.224 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:29 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:00:29.228 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:00:29.254 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:00:29.286 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:29.312 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:00:29.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:29.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:00:29.324 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:00:29.324 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:00:29.324 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-03-20T09:00:29.324 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:00:29.350 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:29 smithi122 ceph-mon[73495]: pgmap v474: 17 pgs: 17 active+clean; 5.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 900 op/s 2024-03-20T09:00:29.350 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:29 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2732984925' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:00:29.350 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:29 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:00:29.411 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-03-20T09:00:29.412 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:00:29.412 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-20T09:00:29.412 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-20T09:00:29.416 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:00:29.464 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:29.614 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:00:29.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:29.616 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:00:29.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:29 smithi115 ceph-mon[72476]: pgmap v474: 17 pgs: 17 active+clean; 5.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 900 op/s 2024-03-20T09:00:29.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:29 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2732984925' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:00:29.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:29 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:00:29.717 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:00:29.746 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:29.771 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:00:29.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:29.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:00:29.877 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:00:29.908 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:29.929 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:00:29.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:29.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:00:30.017 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:00:30.032 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:00:30.061 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:00:30.107 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:30.295 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_blocked_ops 2024-03-20T09:00:30.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:30 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:00:30.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:30 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:00:30.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:30 smithi122 ceph-mon[73495]: osdmap e159: 12 total, 12 up, 12 in 2024-03-20T09:00:30.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:30 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:00:30.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:30 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:00:30.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:30 smithi026 ceph-mon[68309]: osdmap e159: 12 total, 12 up, 12 in 2024-03-20T09:00:30.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:30 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:00:30.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:30 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:00:30.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:30 smithi115 ceph-mon[72476]: osdmap e159: 12 total, 12 up, 12 in 2024-03-20T09:00:30.668 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:00:30.908 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:00:30.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:30.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:00:31.015 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:00:31.047 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:31.068 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:00:31.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:31.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:00:31.172 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:00:31.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:31 smithi122 ceph-mon[73495]: pgmap v476: 17 pgs: 17 active+clean; 5.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 908 op/s 2024-03-20T09:00:31.265 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:31.301 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:00:31.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:31.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:00:31.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:31 smithi115 ceph-mon[72476]: pgmap v476: 17 pgs: 17 active+clean; 5.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 908 op/s 2024-03-20T09:00:31.415 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:00:31.470 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:31.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:31 smithi026 ceph-mon[68309]: pgmap v476: 17 pgs: 17 active+clean; 5.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 908 op/s 2024-03-20T09:00:31.552 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:00:31.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:31.559 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:00:31.660 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:00:31.691 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:31.953 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:00:31.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:31.963 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:00:31.980 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:00:32.064 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:00:32.093 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:33.061 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:32 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:00:33.061 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:32 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:00:33.061 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:32 smithi026 ceph-mon[68309]: pgmap v477: 17 pgs: 17 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 879 op/s 2024-03-20T09:00:33.061 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:32 smithi026 ceph-mon[68309]: osdmap e160: 12 total, 12 up, 12 in 2024-03-20T09:00:33.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:32 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:00:33.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:32 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:00:33.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:32 smithi115 ceph-mon[72476]: pgmap v477: 17 pgs: 17 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 879 op/s 2024-03-20T09:00:33.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:32 smithi115 ceph-mon[72476]: osdmap e160: 12 total, 12 up, 12 in 2024-03-20T09:00:33.204 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:32 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:00:33.204 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:32 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:00:33.204 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:32 smithi122 ceph-mon[73495]: pgmap v477: 17 pgs: 17 active+clean; 5.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 879 op/s 2024-03-20T09:00:33.204 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:32 smithi122 ceph-mon[73495]: osdmap e160: 12 total, 12 up, 12 in 2024-03-20T09:00:33.459 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:00:33.466 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:00:33.523 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:00:33.558 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:00:33.579 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:00:33.632 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:00:33.676 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:00:33.729 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:00:33.767 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:00:33.794 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:00:33.795 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:00:33.795 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 161, 2024-03-20T09:00:33.795 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "852fbb68-e696-11ee-95c9-87774f69a715", 2024-03-20T09:00:33.795 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-03-20T08:48:35.281088+0000", 2024-03-20T09:00:33.795 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-03-20T09:00:33.236014+0000", 2024-03-20T09:00:33.795 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2024-03-20T08:56:17.873738+0000", 2024-03-20T09:00:33.795 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2024-03-20T08:55:59.486577+0000", 2024-03-20T09:00:33.795 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-20T09:00:33.795 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5801984, 2024-03-20T09:00:33.795 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2024-03-20T09:00:33.796 INFO:teuthology.orchestra.run.smithi026.stdout: "noscrub", 2024-03-20T09:00:33.796 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2024-03-20T09:00:33.796 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2024-03-20T09:00:33.796 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2024-03-20T09:00:33.796 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2024-03-20T09:00:33.796 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:00:33.796 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 26, 2024-03-20T09:00:33.796 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2024-03-20T09:00:33.796 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-20T09:00:33.796 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-20T09:00:33.796 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2024-03-20T09:00:33.796 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 2, 2024-03-20T09:00:33.796 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 12, 2024-03-20T09:00:33.796 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "luminous", 2024-03-20T09:00:33.797 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "jewel", 2024-03-20T09:00:33.797 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "squid", 2024-03-20T09:00:33.797 INFO:teuthology.orchestra.run.smithi026.stdout: "allow_crimson": false, 2024-03-20T09:00:33.797 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2024-03-20T09:00:33.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.797 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2024-03-20T09:00:33.797 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": ".mgr", 2024-03-20T09:00:33.797 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-03-20T08:52:17.442324+0000", 2024-03-20T09:00:33.797 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-03-20T09:00:33.797 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-03-20T09:00:33.797 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-03-20T09:00:33.797 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 3, 2024-03-20T09:00:33.797 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 2, 2024-03-20T09:00:33.797 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-03-20T09:00:33.798 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-03-20T09:00:33.798 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-03-20T09:00:33.798 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-03-20T09:00:33.798 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-20T09:00:33.798 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-03-20T09:00:33.798 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-03-20T09:00:33.798 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 1, 2024-03-20T09:00:33.798 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 1, 2024-03-20T09:00:33.798 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 1, 2024-03-20T09:00:33.798 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 1, 2024-03-20T09:00:33.799 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 1, 2024-03-20T09:00:33.799 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-03-20T09:00:33.799 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-03-20T09:00:33.799 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-03-20T09:00:33.799 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-03-20T09:00:33.799 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-03-20T09:00:33.799 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-03-20T09:00:33.799 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-03-20T09:00:33.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.799 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "131", 2024-03-20T09:00:33.799 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-03-20T09:00:33.800 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-20T09:00:33.800 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2024-03-20T09:00:33.800 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-03-20T09:00:33.800 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-03-20T09:00:33.800 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-03-20T09:00:33.800 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-03-20T09:00:33.800 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-03-20T09:00:33.800 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-03-20T09:00:33.800 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-03-20T09:00:33.800 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-03-20T09:00:33.800 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-03-20T09:00:33.800 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-03-20T09:00:33.800 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-03-20T09:00:33.800 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-03-20T09:00:33.801 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-03-20T09:00:33.801 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-03-20T09:00:33.801 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-03-20T09:00:33.801 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-20T09:00:33.801 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-20T09:00:33.801 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-20T09:00:33.801 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-03-20T09:00:33.801 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-03-20T09:00:33.801 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-03-20T09:00:33.801 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-03-20T09:00:33.801 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-03-20T09:00:33.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.801 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-03-20T09:00:33.801 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-03-20T09:00:33.802 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-03-20T09:00:33.802 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-03-20T09:00:33.802 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-03-20T09:00:33.802 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-03-20T09:00:33.802 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-03-20T09:00:33.802 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-03-20T09:00:33.802 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-03-20T09:00:33.802 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-03-20T09:00:33.802 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-03-20T09:00:33.802 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2024-03-20T09:00:33.802 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_max": 32, 2024-03-20T09:00:33.802 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2024-03-20T09:00:33.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.803 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-03-20T09:00:33.803 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": {} 2024-03-20T09:00:33.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.803 INFO:teuthology.orchestra.run.smithi026.stdout: "read_balance": { 2024-03-20T09:00:33.803 INFO:teuthology.orchestra.run.smithi026.stdout: "score_type": "Fair distribution", 2024-03-20T09:00:33.803 INFO:teuthology.orchestra.run.smithi026.stdout: "score_acting": 12, 2024-03-20T09:00:33.803 INFO:teuthology.orchestra.run.smithi026.stdout: "score_stable": 12, 2024-03-20T09:00:33.803 INFO:teuthology.orchestra.run.smithi026.stdout: "optimal_score": 0.25, 2024-03-20T09:00:33.803 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_acting": 3, 2024-03-20T09:00:33.803 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_stable": 3, 2024-03-20T09:00:33.803 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-03-20T09:00:33.803 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity": 1, 2024-03-20T09:00:33.803 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity_weighted": 1 2024-03-20T09:00:33.803 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.804 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 2, 2024-03-20T09:00:33.804 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "unique_pool_0", 2024-03-20T09:00:33.804 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-03-20T08:58:08.054071+0000", 2024-03-20T09:00:33.804 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-03-20T09:00:33.804 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-03-20T09:00:33.804 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-03-20T09:00:33.804 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 3, 2024-03-20T09:00:33.804 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 2, 2024-03-20T09:00:33.804 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-03-20T09:00:33.804 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-03-20T09:00:33.804 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-03-20T09:00:33.804 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-03-20T09:00:33.805 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-20T09:00:33.805 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-03-20T09:00:33.805 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-03-20T09:00:33.805 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 16, 2024-03-20T09:00:33.805 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 16, 2024-03-20T09:00:33.805 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 16, 2024-03-20T09:00:33.805 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 16, 2024-03-20T09:00:33.805 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 16, 2024-03-20T09:00:33.805 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-03-20T09:00:33.805 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-03-20T09:00:33.805 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-03-20T09:00:33.805 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-03-20T09:00:33.805 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-03-20T09:00:33.806 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-03-20T09:00:33.806 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-03-20T09:00:33.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.806 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "125", 2024-03-20T09:00:33.806 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-03-20T09:00:33.806 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-20T09:00:33.806 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2024-03-20T09:00:33.807 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-03-20T09:00:33.807 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-03-20T09:00:33.807 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-03-20T09:00:33.808 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-03-20T09:00:33.808 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-03-20T09:00:33.808 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-03-20T09:00:33.808 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-03-20T09:00:33.808 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-03-20T09:00:33.808 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-03-20T09:00:33.808 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-03-20T09:00:33.808 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-03-20T09:00:33.808 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-03-20T09:00:33.808 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-03-20T09:00:33.808 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-03-20T09:00:33.808 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-03-20T09:00:33.809 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-20T09:00:33.809 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-20T09:00:33.809 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-20T09:00:33.809 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-03-20T09:00:33.809 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-03-20T09:00:33.809 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-03-20T09:00:33.809 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-03-20T09:00:33.809 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-03-20T09:00:33.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.809 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-03-20T09:00:33.809 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-03-20T09:00:33.809 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-03-20T09:00:33.810 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-03-20T09:00:33.810 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-03-20T09:00:33.810 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-03-20T09:00:33.810 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-03-20T09:00:33.810 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-03-20T09:00:33.810 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-03-20T09:00:33.810 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-03-20T09:00:33.810 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-03-20T09:00:33.810 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2024-03-20T09:00:33.810 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-03-20T09:00:33.810 INFO:teuthology.orchestra.run.smithi026.stdout: "rados": {} 2024-03-20T09:00:33.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.810 INFO:teuthology.orchestra.run.smithi026.stdout: "read_balance": { 2024-03-20T09:00:33.810 INFO:teuthology.orchestra.run.smithi026.stdout: "score_type": "Fair distribution", 2024-03-20T09:00:33.811 INFO:teuthology.orchestra.run.smithi026.stdout: "score_acting": 2.25, 2024-03-20T09:00:33.811 INFO:teuthology.orchestra.run.smithi026.stdout: "score_stable": 2.25, 2024-03-20T09:00:33.811 INFO:teuthology.orchestra.run.smithi026.stdout: "optimal_score": 1, 2024-03-20T09:00:33.811 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_acting": 2.25, 2024-03-20T09:00:33.811 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_stable": 2.25, 2024-03-20T09:00:33.811 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-03-20T09:00:33.811 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity": 1, 2024-03-20T09:00:33.811 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity_weighted": 1 2024-03-20T09:00:33.811 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.811 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.811 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:00:33.811 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:00:33.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.811 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-03-20T09:00:33.812 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "ad3fd9c0-316f-4fe5-a63b-d480087c1729", 2024-03-20T09:00:33.812 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:00:33.812 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:00:33.812 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:00:33.812 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:00:33.812 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:00:33.812 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:00:33.812 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 8, 2024-03-20T09:00:33.812 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 62, 2024-03-20T09:00:33.812 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:00:33.812 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:00:33.812 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:00:33.812 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.813 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.813 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2024-03-20T09:00:33.813 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:00:33.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.813 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.813 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2024-03-20T09:00:33.813 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:00:33.814 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.814 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.815 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:00:33.815 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.815 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.815 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2024-03-20T09:00:33.815 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:00:33.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.815 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.815 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2024-03-20T09:00:33.816 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:00:33.816 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.816 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.816 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:00:33.816 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.816 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.816 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2024-03-20T09:00:33.816 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:00:33.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.816 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.817 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2024-03-20T09:00:33.817 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:00:33.817 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.817 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.817 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:00:33.817 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.817 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.817 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2024-03-20T09:00:33.817 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:00:33.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.818 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.818 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2024-03-20T09:00:33.818 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:00:33.818 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.818 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.818 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/2139805293", 2024-03-20T09:00:33.818 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/2139805293", 2024-03-20T09:00:33.818 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/2139805293", 2024-03-20T09:00:33.818 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/2139805293", 2024-03-20T09:00:33.818 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:00:33.818 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:00:33.818 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:00:33.819 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.819 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-03-20T09:00:33.819 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e85b94f5-2954-47a1-9c20-096f2606eb04", 2024-03-20T09:00:33.819 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:00:33.819 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:00:33.819 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:00:33.819 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:00:33.819 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:00:33.819 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:00:33.819 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2024-03-20T09:00:33.819 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 121, 2024-03-20T09:00:33.820 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:00:33.820 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:00:33.820 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:00:33.820 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.820 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.820 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2024-03-20T09:00:33.820 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:00:33.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.820 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.820 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2024-03-20T09:00:33.820 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:00:33.821 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.821 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.821 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:00:33.821 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.821 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.821 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2024-03-20T09:00:33.821 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:00:33.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.821 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.821 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2024-03-20T09:00:33.822 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:00:33.822 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.822 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.822 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:00:33.822 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.822 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.822 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2024-03-20T09:00:33.822 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:00:33.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.822 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.822 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2024-03-20T09:00:33.823 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:00:33.823 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.823 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.823 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:00:33.823 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.823 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.823 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2024-03-20T09:00:33.823 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:00:33.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.823 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.824 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2024-03-20T09:00:33.824 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:00:33.824 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.824 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.824 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/3853355134", 2024-03-20T09:00:33.824 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/3853355134", 2024-03-20T09:00:33.824 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/3853355134", 2024-03-20T09:00:33.824 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/3853355134", 2024-03-20T09:00:33.824 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:00:33.824 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:00:33.824 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:00:33.824 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.825 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-03-20T09:00:33.825 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "dd2b10f0-741c-47a4-b224-ac6bfe2c14ef", 2024-03-20T09:00:33.825 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:00:33.825 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:00:33.825 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:00:33.826 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:00:33.826 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:00:33.826 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:00:33.826 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 18, 2024-03-20T09:00:33.826 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 121, 2024-03-20T09:00:33.826 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:00:33.826 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:00:33.826 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:00:33.826 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.826 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.826 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2024-03-20T09:00:33.827 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:00:33.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.827 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.827 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2024-03-20T09:00:33.827 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:00:33.827 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.827 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.827 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:00:33.827 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.828 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.828 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2024-03-20T09:00:33.828 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:00:33.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.828 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.828 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2024-03-20T09:00:33.828 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:00:33.828 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.829 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.829 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:00:33.829 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.829 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.829 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2024-03-20T09:00:33.829 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:00:33.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.829 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.829 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2024-03-20T09:00:33.829 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:00:33.830 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.830 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.830 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:00:33.830 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.830 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.830 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2024-03-20T09:00:33.830 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:00:33.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.831 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.831 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2024-03-20T09:00:33.831 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:00:33.831 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.831 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.831 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/3993357967", 2024-03-20T09:00:33.831 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/3993357967", 2024-03-20T09:00:33.831 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/3993357967", 2024-03-20T09:00:33.831 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/3993357967", 2024-03-20T09:00:33.831 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:00:33.832 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:00:33.832 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:00:33.832 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.832 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-03-20T09:00:33.832 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1", 2024-03-20T09:00:33.832 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:00:33.832 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:00:33.832 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:00:33.832 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:00:33.832 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:00:33.832 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:00:33.832 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 26, 2024-03-20T09:00:33.833 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 0, 2024-03-20T09:00:33.833 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:00:33.833 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:00:33.833 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:00:33.833 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.833 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.833 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2024-03-20T09:00:33.833 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:00:33.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.833 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.833 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2024-03-20T09:00:33.834 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:00:33.834 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.834 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.834 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:00:33.834 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.834 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.834 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2024-03-20T09:00:33.834 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:00:33.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.834 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.834 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2024-03-20T09:00:33.835 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:00:33.835 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.835 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.835 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:00:33.835 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.835 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.836 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2024-03-20T09:00:33.836 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:00:33.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.836 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.836 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2024-03-20T09:00:33.836 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:00:33.836 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.836 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.836 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:00:33.836 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.837 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.837 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2024-03-20T09:00:33.837 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:00:33.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.837 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.837 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2024-03-20T09:00:33.837 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:00:33.837 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.837 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.837 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/4110225509", 2024-03-20T09:00:33.838 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/4110225509", 2024-03-20T09:00:33.838 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/4110225509", 2024-03-20T09:00:33.838 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/4110225509", 2024-03-20T09:00:33.838 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:00:33.838 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:00:33.838 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:00:33.838 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.838 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-03-20T09:00:33.838 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "f14e34d5-9829-42a3-9b25-31a0976c5ef7", 2024-03-20T09:00:33.839 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:00:33.839 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:00:33.839 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:00:33.839 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:00:33.839 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:00:33.839 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:00:33.839 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 32, 2024-03-20T09:00:33.839 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 0, 2024-03-20T09:00:33.839 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:00:33.839 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:00:33.839 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:00:33.839 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.840 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.840 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6800", 2024-03-20T09:00:33.840 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:00:33.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.840 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.840 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6801", 2024-03-20T09:00:33.840 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:00:33.840 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.840 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.840 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:00:33.840 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.841 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.841 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6802", 2024-03-20T09:00:33.841 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:00:33.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.841 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.841 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6803", 2024-03-20T09:00:33.841 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:00:33.841 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.842 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.842 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:00:33.842 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.842 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.842 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6806", 2024-03-20T09:00:33.842 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:00:33.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.842 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.842 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6807", 2024-03-20T09:00:33.842 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:00:33.843 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.843 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.843 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:00:33.843 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.843 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.843 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6804", 2024-03-20T09:00:33.843 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:00:33.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.843 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.843 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6805", 2024-03-20T09:00:33.844 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:00:33.844 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.844 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.844 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6801/477923058", 2024-03-20T09:00:33.844 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6803/477923058", 2024-03-20T09:00:33.844 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6807/477923058", 2024-03-20T09:00:33.844 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6805/477923058", 2024-03-20T09:00:33.844 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:00:33.844 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:00:33.844 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:00:33.844 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.845 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-03-20T09:00:33.845 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "cb431b6a-540d-4318-ae35-d379e1df0185", 2024-03-20T09:00:33.845 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:00:33.845 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:00:33.845 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:00:33.845 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:00:33.845 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:00:33.845 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:00:33.845 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 41, 2024-03-20T09:00:33.845 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 121, 2024-03-20T09:00:33.846 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:00:33.846 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:00:33.846 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:00:33.846 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.846 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.846 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6808", 2024-03-20T09:00:33.846 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:00:33.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.846 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.847 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6809", 2024-03-20T09:00:33.847 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:00:33.847 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.847 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.847 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:00:33.847 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.847 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.847 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6810", 2024-03-20T09:00:33.847 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:00:33.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.847 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.848 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6811", 2024-03-20T09:00:33.848 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:00:33.848 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.848 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.848 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:00:33.848 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.848 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.848 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6814", 2024-03-20T09:00:33.848 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:00:33.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.849 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.849 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6815", 2024-03-20T09:00:33.849 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:00:33.849 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.849 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.849 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:00:33.849 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.849 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.849 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6812", 2024-03-20T09:00:33.849 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:00:33.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.850 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.850 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6813", 2024-03-20T09:00:33.850 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:00:33.850 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.850 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.850 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6809/1339268003", 2024-03-20T09:00:33.850 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6811/1339268003", 2024-03-20T09:00:33.850 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6815/1339268003", 2024-03-20T09:00:33.850 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6813/1339268003", 2024-03-20T09:00:33.850 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:00:33.850 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:00:33.850 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:00:33.851 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.851 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2024-03-20T09:00:33.851 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "753ed830-e764-4ff3-9d9b-87e7fa748b81", 2024-03-20T09:00:33.851 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:00:33.851 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:00:33.851 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:00:33.851 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:00:33.851 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:00:33.851 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:00:33.851 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 51, 2024-03-20T09:00:33.851 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 121, 2024-03-20T09:00:33.851 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:00:33.852 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:00:33.852 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:00:33.852 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.852 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.852 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6816", 2024-03-20T09:00:33.852 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:00:33.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.852 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.852 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6817", 2024-03-20T09:00:33.852 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:00:33.852 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.852 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.853 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:00:33.853 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.853 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.853 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6818", 2024-03-20T09:00:33.853 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:00:33.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.853 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.853 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6819", 2024-03-20T09:00:33.853 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:00:33.853 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.853 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.854 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:00:33.854 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.854 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.854 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6822", 2024-03-20T09:00:33.854 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:00:33.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.854 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.854 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6823", 2024-03-20T09:00:33.854 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:00:33.854 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.854 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.855 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:00:33.855 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.855 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.855 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6820", 2024-03-20T09:00:33.855 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:00:33.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.855 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.855 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6821", 2024-03-20T09:00:33.855 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:00:33.855 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.855 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.856 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6817/3278301843", 2024-03-20T09:00:33.856 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6819/3278301843", 2024-03-20T09:00:33.856 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6823/3278301843", 2024-03-20T09:00:33.856 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6821/3278301843", 2024-03-20T09:00:33.856 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:00:33.856 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:00:33.856 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:00:33.857 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.857 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2024-03-20T09:00:33.857 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "55bcbd37-7a6f-4ab1-b5a9-91a951444d86", 2024-03-20T09:00:33.857 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:00:33.857 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:00:33.857 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:00:33.857 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:00:33.857 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:00:33.857 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:00:33.857 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 60, 2024-03-20T09:00:33.857 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 121, 2024-03-20T09:00:33.857 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:00:33.858 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:00:33.858 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:00:33.858 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.858 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.858 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6824", 2024-03-20T09:00:33.858 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:00:33.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.858 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.858 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6825", 2024-03-20T09:00:33.858 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:00:33.858 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.858 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.859 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:00:33.859 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.859 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.859 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6826", 2024-03-20T09:00:33.859 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:00:33.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.859 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.859 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6827", 2024-03-20T09:00:33.859 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:00:33.859 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.859 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.860 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:00:33.860 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.860 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.860 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6830", 2024-03-20T09:00:33.860 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:00:33.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.860 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.860 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6831", 2024-03-20T09:00:33.860 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:00:33.860 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.861 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.861 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:00:33.861 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.861 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.861 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6828", 2024-03-20T09:00:33.861 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:00:33.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.861 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.861 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6829", 2024-03-20T09:00:33.861 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:00:33.862 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.862 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.862 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6825/3152424001", 2024-03-20T09:00:33.862 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6827/3152424001", 2024-03-20T09:00:33.862 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6831/3152424001", 2024-03-20T09:00:33.862 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6829/3152424001", 2024-03-20T09:00:33.862 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:00:33.862 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:00:33.862 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:00:33.862 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.863 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 8, 2024-03-20T09:00:33.863 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "bc86248e-6bd0-4369-b8a2-e593bc2a610d", 2024-03-20T09:00:33.863 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:00:33.863 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:00:33.863 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:00:33.863 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:00:33.863 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:00:33.863 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:00:33.863 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 70, 2024-03-20T09:00:33.863 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 121, 2024-03-20T09:00:33.863 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:00:33.863 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:00:33.863 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:00:33.863 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.864 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.864 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6800", 2024-03-20T09:00:33.864 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:00:33.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.864 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.864 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6801", 2024-03-20T09:00:33.864 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:00:33.864 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.864 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.864 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:00:33.865 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.865 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.865 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6802", 2024-03-20T09:00:33.865 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:00:33.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.865 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.865 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6803", 2024-03-20T09:00:33.865 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:00:33.865 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.865 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.866 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:00:33.866 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.866 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.866 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6806", 2024-03-20T09:00:33.866 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:00:33.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.867 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.867 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6807", 2024-03-20T09:00:33.867 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:00:33.867 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.867 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.867 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:00:33.867 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.867 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.867 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6804", 2024-03-20T09:00:33.868 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:00:33.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.868 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.868 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6805", 2024-03-20T09:00:33.868 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:00:33.868 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.868 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.868 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6801/1309426740", 2024-03-20T09:00:33.868 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6803/1309426740", 2024-03-20T09:00:33.868 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6807/1309426740", 2024-03-20T09:00:33.868 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6805/1309426740", 2024-03-20T09:00:33.869 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:00:33.869 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:00:33.869 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:00:33.869 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.869 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 9, 2024-03-20T09:00:33.869 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b20a4aa-032a-4c09-ac75-153c554424e7", 2024-03-20T09:00:33.869 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:00:33.869 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:00:33.869 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:00:33.869 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:00:33.869 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:00:33.869 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:00:33.870 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 80, 2024-03-20T09:00:33.870 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 121, 2024-03-20T09:00:33.870 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:00:33.870 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:00:33.870 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:00:33.870 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.870 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.870 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6808", 2024-03-20T09:00:33.870 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:00:33.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.870 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.870 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6809", 2024-03-20T09:00:33.871 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:00:33.871 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.871 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.871 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:00:33.871 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.871 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.871 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6810", 2024-03-20T09:00:33.871 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:00:33.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.871 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.872 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6811", 2024-03-20T09:00:33.872 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:00:33.872 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.872 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.872 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:00:33.872 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.872 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.872 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6814", 2024-03-20T09:00:33.872 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:00:33.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.872 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.872 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6815", 2024-03-20T09:00:33.873 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:00:33.873 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.873 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.873 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:00:33.873 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.873 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.873 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6812", 2024-03-20T09:00:33.873 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:00:33.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.873 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.873 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6813", 2024-03-20T09:00:33.874 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:00:33.874 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.874 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.874 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6809/2953210076", 2024-03-20T09:00:33.874 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6811/2953210076", 2024-03-20T09:00:33.874 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6815/2953210076", 2024-03-20T09:00:33.874 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6813/2953210076", 2024-03-20T09:00:33.874 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:00:33.874 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:00:33.874 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:00:33.875 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.875 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 10, 2024-03-20T09:00:33.875 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e62c1fb3-5dd1-4d0d-997c-d20633d19361", 2024-03-20T09:00:33.875 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:00:33.875 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:00:33.875 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:00:33.875 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:00:33.875 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:00:33.875 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:00:33.875 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 89, 2024-03-20T09:00:33.875 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 0, 2024-03-20T09:00:33.875 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:00:33.876 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:00:33.876 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:00:33.876 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.876 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.876 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6816", 2024-03-20T09:00:33.876 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:00:33.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.876 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.876 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6817", 2024-03-20T09:00:33.876 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:00:33.876 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.876 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.877 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:00:33.877 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.877 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.877 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6818", 2024-03-20T09:00:33.877 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:00:33.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.877 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.877 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6819", 2024-03-20T09:00:33.877 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:00:33.877 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.877 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.878 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:00:33.878 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.878 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.878 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6822", 2024-03-20T09:00:33.878 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:00:33.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.878 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.878 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6823", 2024-03-20T09:00:33.878 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:00:33.878 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.878 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.879 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:00:33.879 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.879 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.879 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6820", 2024-03-20T09:00:33.879 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:00:33.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.879 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.879 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6821", 2024-03-20T09:00:33.879 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:00:33.879 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.879 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.880 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6817/409394340", 2024-03-20T09:00:33.880 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6819/409394340", 2024-03-20T09:00:33.880 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6823/409394340", 2024-03-20T09:00:33.880 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6821/409394340", 2024-03-20T09:00:33.880 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:00:33.880 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:00:33.880 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:00:33.880 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.880 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 11, 2024-03-20T09:00:33.880 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b83994d5-e5ef-48b0-9758-c26843cec3b6", 2024-03-20T09:00:33.880 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:00:33.880 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:00:33.881 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:00:33.881 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:00:33.881 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:00:33.881 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:00:33.881 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 96, 2024-03-20T09:00:33.881 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 121, 2024-03-20T09:00:33.881 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:00:33.881 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:00:33.881 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:00:33.881 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.881 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.881 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6824", 2024-03-20T09:00:33.881 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:00:33.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.882 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.882 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6825", 2024-03-20T09:00:33.882 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:00:33.882 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.882 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.882 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:00:33.882 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.882 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.882 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6826", 2024-03-20T09:00:33.882 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:00:33.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.883 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.883 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6827", 2024-03-20T09:00:33.883 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:00:33.883 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.883 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.883 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:00:33.883 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.883 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.883 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6830", 2024-03-20T09:00:33.883 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:00:33.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.884 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.884 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6831", 2024-03-20T09:00:33.884 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:00:33.884 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.884 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.884 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:00:33.884 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:33.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.884 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:33.884 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6828", 2024-03-20T09:00:33.884 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:00:33.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.885 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:33.885 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6829", 2024-03-20T09:00:33.885 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:00:33.885 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.885 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.885 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6825/2704069354", 2024-03-20T09:00:33.885 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6827/2704069354", 2024-03-20T09:00:33.885 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6831/2704069354", 2024-03-20T09:00:33.885 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6829/2704069354", 2024-03-20T09:00:33.885 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:00:33.885 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:00:33.885 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:00:33.885 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:33.886 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.886 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:00:33.886 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2024-03-20T09:00:33.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.886 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-03-20T09:00:33.886 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:00:33.886 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:00:33.886 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:00:33.886 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:00:33.886 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:00:33.886 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:00:15.458602+0000", 2024-03-20T09:00:33.886 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:00:33.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.887 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-03-20T09:00:33.887 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:00:33.887 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:00:33.887 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:00:33.887 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:00:33.887 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:00:33.887 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T08:59:07.091248+0000", 2024-03-20T09:00:33.887 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:00:33.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.887 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-03-20T09:00:33.887 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:00:33.887 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:00:33.887 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:00:33.887 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:00:33.888 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:00:33.888 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:00:06.154878+0000", 2024-03-20T09:00:33.888 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:00:33.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.888 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-03-20T09:00:33.888 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:00:33.888 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:00:33.888 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:00:33.888 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:00:33.888 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:00:33.888 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:00:31.384708+0000", 2024-03-20T09:00:33.888 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:00:33.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.889 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-03-20T09:00:33.889 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:00:33.889 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:00:33.889 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:00:33.889 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:00:33.889 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:00:33.889 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:00:21.542194+0000", 2024-03-20T09:00:33.889 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:00:33.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.889 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-03-20T09:00:33.889 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:00:33.889 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:00:33.890 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:00:33.890 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:00:33.890 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:00:33.890 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:00:00.111136+0000", 2024-03-20T09:00:33.890 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:00:33.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.890 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2024-03-20T09:00:33.890 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:00:33.890 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:00:33.890 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:00:33.890 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:00:33.890 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:00:33.890 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:00:32.621480+0000", 2024-03-20T09:00:33.891 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:00:33.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.891 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2024-03-20T09:00:33.891 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:00:33.891 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:00:33.891 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:00:33.891 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:00:33.891 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:00:33.891 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T08:59:18.725286+0000", 2024-03-20T09:00:33.891 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:00:33.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.891 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 8, 2024-03-20T09:00:33.891 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:00:33.892 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:00:33.892 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:00:33.892 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:00:33.892 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:00:33.892 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T08:59:52.093175+0000", 2024-03-20T09:00:33.892 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:00:33.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.892 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 9, 2024-03-20T09:00:33.892 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:00:33.892 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:00:33.892 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:00:33.892 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:00:33.892 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:00:33.893 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:00:26.483886+0000", 2024-03-20T09:00:33.893 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:00:33.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.893 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 10, 2024-03-20T09:00:33.893 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:00:33.893 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:00:33.893 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:00:33.893 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:00:33.893 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:00:33.893 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T08:59:55.463512+0000", 2024-03-20T09:00:33.893 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:00:33.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:33.894 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 11, 2024-03-20T09:00:33.894 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:00:33.894 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:00:33.894 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:00:33.894 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:00:33.894 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:00:33.894 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T08:59:46.097314+0000", 2024-03-20T09:00:33.894 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:00:33.894 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.894 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:00:33.894 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [], 2024-03-20T09:00:33.894 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2024-03-20T09:00:33.894 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_primaries": [], 2024-03-20T09:00:33.894 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2024-03-20T09:00:33.895 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2024-03-20T09:00:33.895 INFO:teuthology.orchestra.run.smithi026.stdout: "blocklist": { 2024-03-20T09:00:33.895 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/4291404191": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:00:33.895 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/642012566": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:00:33.895 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6801/1181459999": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:00:33.895 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/3984513952": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:00:33.895 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6800/1654945498": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:00:33.895 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6800/1181459999": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:00:33.895 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/137770981": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:00:33.895 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/1451237762": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:00:33.895 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/3938386631": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:00:33.895 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6801/1654945498": "2024-03-21T08:49:04.596754+0000" 2024-03-20T09:00:33.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.895 INFO:teuthology.orchestra.run.smithi026.stdout: "range_blocklist": {}, 2024-03-20T09:00:33.896 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2024-03-20T09:00:33.896 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2024-03-20T09:00:33.896 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2024-03-20T09:00:33.896 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2024-03-20T09:00:33.896 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2024-03-20T09:00:33.896 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2024-03-20T09:00:33.896 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2024-03-20T09:00:33.896 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:33.896 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2024-03-20T09:00:33.896 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2024-03-20T09:00:33.896 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2024-03-20T09:00:33.896 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2024-03-20T09:00:33.896 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {}, 2024-03-20T09:00:33.896 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": { 2024-03-20T09:00:33.897 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_enabled": false, 2024-03-20T09:00:33.897 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_bucket_count": 0, 2024-03-20T09:00:33.897 INFO:teuthology.orchestra.run.smithi026.stdout: "degraded_stretch_mode": 0, 2024-03-20T09:00:33.897 INFO:teuthology.orchestra.run.smithi026.stdout: "recovering_stretch_mode": 0, 2024-03-20T09:00:33.897 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_bucket": 0 2024-03-20T09:00:33.897 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:33.897 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:00:33.898 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:00:33.898 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:00:33.905 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:00:33.957 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:00:33.974 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:00:33.974 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:00:33.974 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:00:33.974 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:00:33.974 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:00:33.975 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:00:33.975 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:00:33.975 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:00:33.975 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:00:33.975 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:00:33.975 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:00:33.975 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:00:34.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:33 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:00:34.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:33 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:00:34.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:33 smithi115 ceph-mon[72476]: osdmap e161: 12 total, 12 up, 12 in 2024-03-20T09:00:34.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:33 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1473321531' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:00:34.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:33 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3522256091' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:00:34.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:33 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:00:34.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:33 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:00:34.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:33 smithi122 ceph-mon[73495]: osdmap e161: 12 total, 12 up, 12 in 2024-03-20T09:00:34.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:33 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1473321531' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:00:34.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:33 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3522256091' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:00:34.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:33 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:00:34.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:33 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:00:34.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:33 smithi026 ceph-mon[68309]: osdmap e161: 12 total, 12 up, 12 in 2024-03-20T09:00:34.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:33 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1473321531' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:00:34.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:33 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3522256091' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:00:35.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:35 smithi026 ceph-mon[68309]: pgmap v480: 17 pgs: 17 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 923 op/s 2024-03-20T09:00:35.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:35 smithi115 ceph-mon[72476]: pgmap v480: 17 pgs: 17 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 923 op/s 2024-03-20T09:00:35.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:35 smithi122 ceph-mon[73495]: pgmap v480: 17 pgs: 17 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 923 op/s 2024-03-20T09:00:36.716 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:00:36.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:36.721 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:00:36.822 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:00:36.858 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:36.865 INFO:tasks.thrashosds.thrasher:j is {'epoch': 161, 'fsid': '852fbb68-e696-11ee-95c9-87774f69a715', 'created': '2024-03-20T08:48:35.281088+0000', 'modified': '2024-03-20T09:00:33.236014+0000', 'last_up_change': '2024-03-20T08:56:17.873738+0000', 'last_in_change': '2024-03-20T08:55:59.486577+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-20T08:52:17.442324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '131', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'unique_pool_0', 'create_time': '2024-03-20T08:58:08.054071+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '125', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'ad3fd9c0-316f-4fe5-a63b-d480087c1729', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 62, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6802', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6803', 'nonce': 2139805293}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6804', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6805', 'nonce': 2139805293}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6808', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6809', 'nonce': 2139805293}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6806', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6807', 'nonce': 2139805293}]}, 'public_addr': '172.21.15.26:6803/2139805293', 'cluster_addr': '172.21.15.26:6805/2139805293', 'heartbeat_back_addr': '172.21.15.26:6809/2139805293', 'heartbeat_front_addr': '172.21.15.26:6807/2139805293', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'e85b94f5-2954-47a1-9c20-096f2606eb04', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 121, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6810', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6811', 'nonce': 3853355134}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6812', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6813', 'nonce': 3853355134}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6816', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6817', 'nonce': 3853355134}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6814', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6815', 'nonce': 3853355134}]}, 'public_addr': '172.21.15.26:6811/3853355134', 'cluster_addr': '172.21.15.26:6813/3853355134', 'heartbeat_back_addr': '172.21.15.26:6817/3853355134', 'heartbeat_front_addr': '172.21.15.26:6815/3853355134', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'dd2b10f0-741c-47a4-b224-ac6bfe2c14ef', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 121, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6818', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6819', 'nonce': 3993357967}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6820', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6821', 'nonce': 3993357967}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6824', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6825', 'nonce': 3993357967}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6822', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6823', 'nonce': 3993357967}]}, 'public_addr': '172.21.15.26:6819/3993357967', 'cluster_addr': '172.21.15.26:6821/3993357967', 'heartbeat_back_addr': '172.21.15.26:6825/3993357967', 'heartbeat_front_addr': '172.21.15.26:6823/3993357967', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6826', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6827', 'nonce': 4110225509}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6828', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6829', 'nonce': 4110225509}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6832', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6833', 'nonce': 4110225509}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6830', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6831', 'nonce': 4110225509}]}, 'public_addr': '172.21.15.26:6827/4110225509', 'cluster_addr': '172.21.15.26:6829/4110225509', 'heartbeat_back_addr': '172.21.15.26:6833/4110225509', 'heartbeat_front_addr': '172.21.15.26:6831/4110225509', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'f14e34d5-9829-42a3-9b25-31a0976c5ef7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6800', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6801', 'nonce': 477923058}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6802', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6803', 'nonce': 477923058}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6806', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6807', 'nonce': 477923058}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6804', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6805', 'nonce': 477923058}]}, 'public_addr': '172.21.15.115:6801/477923058', 'cluster_addr': '172.21.15.115:6803/477923058', 'heartbeat_back_addr': '172.21.15.115:6807/477923058', 'heartbeat_front_addr': '172.21.15.115:6805/477923058', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'cb431b6a-540d-4318-ae35-d379e1df0185', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 121, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6808', 'nonce': 1339268003}, {'type': 'v1', 'addr': '172.21.15.115:6809', 'nonce': 1339268003}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6810', 'nonce': 1339268003}, {'type': 'v1', 'addr': '172.21.15.115:6811', 'nonce': 1339268003}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6814', 'nonce': 1339268003}, {'type': 'v1', 'addr': '172.21.15.115:6815', 'nonce': 1339268003}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6812', 'nonce': 1339268003}, {'type': 'v1', 'addr': '172.21.15.115:6813', 'nonce': 1339268003}]}, 'public_addr': '172.21.15.115:6809/1339268003', 'cluster_addr': '172.21.15.115:6811/1339268003', 'heartbeat_back_addr': '172.21.15.115:6815/1339268003', 'heartbeat_front_addr': '172.21.15.115:6813/1339268003', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '753ed830-e764-4ff3-9d9b-87e7fa748b81', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 121, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6816', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6817', 'nonce': 3278301843}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6818', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6819', 'nonce': 3278301843}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6822', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6823', 'nonce': 3278301843}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6820', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6821', 'nonce': 3278301843}]}, 'public_addr': '172.21.15.115:6817/3278301843', 'cluster_addr': '172.21.15.115:6819/3278301843', 'heartbeat_back_addr': '172.21.15.115:6823/3278301843', 'heartbeat_front_addr': '172.21.15.115:6821/3278301843', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '55bcbd37-7a6f-4ab1-b5a9-91a951444d86', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 121, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6824', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6825', 'nonce': 3152424001}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6826', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6827', 'nonce': 3152424001}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6830', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6831', 'nonce': 3152424001}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6828', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6829', 'nonce': 3152424001}]}, 'public_addr': '172.21.15.115:6825/3152424001', 'cluster_addr': '172.21.15.115:6827/3152424001', 'heartbeat_back_addr': '172.21.15.115:6831/3152424001', 'heartbeat_front_addr': '172.21.15.115:6829/3152424001', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'bc86248e-6bd0-4369-b8a2-e593bc2a610d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 70, 'up_thru': 121, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6800', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6801', 'nonce': 1309426740}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6802', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6803', 'nonce': 1309426740}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6806', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6807', 'nonce': 1309426740}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6804', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6805', 'nonce': 1309426740}]}, 'public_addr': '172.21.15.122:6801/1309426740', 'cluster_addr': '172.21.15.122:6803/1309426740', 'heartbeat_back_addr': '172.21.15.122:6807/1309426740', 'heartbeat_front_addr': '172.21.15.122:6805/1309426740', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '9b20a4aa-032a-4c09-ac75-153c554424e7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 121, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6808', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6809', 'nonce': 2953210076}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6810', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6811', 'nonce': 2953210076}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6814', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6815', 'nonce': 2953210076}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6812', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6813', 'nonce': 2953210076}]}, 'public_addr': '172.21.15.122:6809/2953210076', 'cluster_addr': '172.21.15.122:6811/2953210076', 'heartbeat_back_addr': '172.21.15.122:6815/2953210076', 'heartbeat_front_addr': '172.21.15.122:6813/2953210076', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'e62c1fb3-5dd1-4d0d-997c-d20633d19361', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 89, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6816', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6817', 'nonce': 409394340}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6818', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6819', 'nonce': 409394340}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6822', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6823', 'nonce': 409394340}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6820', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6821', 'nonce': 409394340}]}, 'public_addr': '172.21.15.122:6817/409394340', 'cluster_addr': '172.21.15.122:6819/409394340', 'heartbeat_back_addr': '172.21.15.122:6823/409394340', 'heartbeat_front_addr': '172.21.15.122:6821/409394340', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'b83994d5-e5ef-48b0-9758-c26843cec3b6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 96, 'up_thru': 121, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6824', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6825', 'nonce': 2704069354}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6826', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6827', 'nonce': 2704069354}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6830', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6831', 'nonce': 2704069354}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6828', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6829', 'nonce': 2704069354}]}, 'public_addr': '172.21.15.122:6825/2704069354', 'cluster_addr': '172.21.15.122:6827/2704069354', 'heartbeat_back_addr': '172.21.15.122:6831/2704069354', 'heartbeat_front_addr': '172.21.15.122:6829/2704069354', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:00:15.458602+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T08:59:07.091248+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:00:06.154878+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:00:31.384708+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:00:21.542194+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:00:00.111136+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:00:32.621480+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T08:59:18.725286+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T08:59:52.093175+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:00:26.483886+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T08:59:55.463512+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T08:59:46.097314+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.26:0/4291404191': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/642012566': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:6801/1181459999': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/3984513952': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6800/1654945498': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6800/1181459999': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/137770981': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:0/1451237762': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/3938386631': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6801/1654945498': '2024-03-21T08:49:04.596754+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-20T09:00:36.866 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-03-20T09:00:37.245 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:00:37.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:37 smithi026 ceph-mon[68309]: pgmap v481: 17 pgs: 17 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 914 op/s 2024-03-20T09:00:37.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:37 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:00:37.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:37 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:00:37.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:37 smithi026 ceph-mon[68309]: osdmap e162: 12 total, 12 up, 12 in 2024-03-20T09:00:37.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:37 smithi115 ceph-mon[72476]: pgmap v481: 17 pgs: 17 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 914 op/s 2024-03-20T09:00:37.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:37 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:00:37.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:37 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:00:37.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:37 smithi115 ceph-mon[72476]: osdmap e162: 12 total, 12 up, 12 in 2024-03-20T09:00:37.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:37 smithi122 ceph-mon[73495]: pgmap v481: 17 pgs: 17 active+clean; 5.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 914 op/s 2024-03-20T09:00:37.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:37 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:00:37.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:37 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:00:37.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:37 smithi122 ceph-mon[73495]: osdmap e162: 12 total, 12 up, 12 in 2024-03-20T09:00:37.835 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:00:38.322 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:00:38.443 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:38 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1758868648' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:00:38.443 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:38 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:00:38.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:38 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1758868648' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:00:38.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:38 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:00:38.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:38 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1758868648' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:00:38.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:38 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:00:38.915 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:00:38.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:38.919 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:00:39.020 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:00:39.035 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:00:39.036 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:00:39.047 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:39.066 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:00:39.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:39.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:00:39.170 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:00:39.206 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:39.283 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:00:39.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:39.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:00:39.391 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:00:39.421 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:39.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:39 smithi026 ceph-mon[68309]: pgmap v483: 17 pgs: 17 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 999 op/s 2024-03-20T09:00:39.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:39 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:00:39.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:39 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:00:39.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:39 smithi026 ceph-mon[68309]: osdmap e163: 12 total, 12 up, 12 in 2024-03-20T09:00:39.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:39 smithi115 ceph-mon[72476]: pgmap v483: 17 pgs: 17 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 999 op/s 2024-03-20T09:00:39.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:39 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:00:39.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:39 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:00:39.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:39 smithi115 ceph-mon[72476]: osdmap e163: 12 total, 12 up, 12 in 2024-03-20T09:00:39.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:39 smithi122 ceph-mon[73495]: pgmap v483: 17 pgs: 17 active+clean; 4.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 999 op/s 2024-03-20T09:00:39.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:39 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:00:39.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:39 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:00:39.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:39 smithi122 ceph-mon[73495]: osdmap e163: 12 total, 12 up, 12 in 2024-03-20T09:00:39.922 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_historic_ops 2024-03-20T09:00:39.970 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:00:39.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:39.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:00:40.076 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:00:40.096 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:00:40.102 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:40.116 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:00:40.122 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:00:40.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:40.125 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:00:40.138 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:00:40.154 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:00:40.174 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:00:40.187 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:00:40.195 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:00:40.215 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:00:40.225 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:00:40.235 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:00:40.256 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:40.257 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:00:40.275 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:00:40.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:40.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:00:40.279 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:00:40.300 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:00:40.320 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:00:40.321 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:00:40.321 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:00:40.321 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:00:40.321 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:00:40.321 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:00:40.321 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:00:40.321 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:00:40.322 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:00:40.322 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:00:40.322 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:00:40.322 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:00:40.322 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:00:40.380 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:00:40.408 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:40.505 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:00:40.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:40 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3810587566' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:00:40.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:40.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:00:40.608 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:00:40.636 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:40.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:40 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3810587566' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:00:40.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:40 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3810587566' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:00:40.886 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:00:40.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:40.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:00:40.989 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:00:41.017 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:41.036 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:00:41.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:41.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:00:41.140 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:00:41.167 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:41.204 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:00:41.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:41.215 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:00:41.315 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:00:41.343 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:41.363 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:00:41.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:41.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:00:41.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:41 smithi122 ceph-mon[73495]: pgmap v485: 17 pgs: 17 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 891 op/s 2024-03-20T09:00:41.466 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:00:41.498 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:41.519 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:00:41.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:41.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:00:41.622 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:00:41.650 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:41 smithi026 ceph-mon[68309]: pgmap v485: 17 pgs: 17 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 891 op/s 2024-03-20T09:00:41.650 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:00:41.652 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:41.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:41 smithi115 ceph-mon[72476]: pgmap v485: 17 pgs: 17 active+clean; 4.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 891 op/s 2024-03-20T09:00:41.674 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:00:41.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:41.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:00:41.778 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:00:41.806 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:41.867 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-03-20T09:00:41.867 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:00:41.867 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-20T09:00:41.867 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-20T09:00:41.933 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:00:42.003 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:00:42.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:42.006 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:00:42.107 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:00:42.139 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:42.159 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:00:42.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:42.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:00:42.263 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:00:42.292 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:42.319 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:00:42.319 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-03-20T09:00:42.319 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-03-20T09:00:42.319 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-03-20T09:00:42.319 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.319 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.24550.0:1360 2.0 e125/121 2:06f96393:::benchmark_data_smithi122_107654_object1359:head v 125'77, mlcod=125'75)", 2024-03-20T09:00:42.320 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:58:24.752475+0000", 2024-03-20T09:00:42.320 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 137.53091983600001, 2024-03-20T09:00:42.320 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.030003946, 2024-03-20T09:00:42.320 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:42.320 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:42.320 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:42.320 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.320 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:42.320 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.752475+0000", 2024-03-20T09:00:42.320 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.320 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.320 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.320 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:42.321 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.752475+0000", 2024-03-20T09:00:42.321 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.321 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.321 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.321 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:42.321 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.752475+0000", 2024-03-20T09:00:42.321 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7999999999999995e-07 2024-03-20T09:00:42.321 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.321 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.321 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:42.321 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.752548+0000", 2024-03-20T09:00:42.321 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2780000000000005e-05 2024-03-20T09:00:42.321 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.321 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.322 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:42.322 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.752561+0000", 2024-03-20T09:00:42.322 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2313000000000001e-05 2024-03-20T09:00:42.322 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.322 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.322 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:42.322 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.752567+0000", 2024-03-20T09:00:42.322 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8199999999999999e-06 2024-03-20T09:00:42.322 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.322 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.322 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:42.322 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.781575+0000", 2024-03-20T09:00:42.323 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.029007380999999999 2024-03-20T09:00:42.323 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.323 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.323 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:42.323 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.781597+0000", 2024-03-20T09:00:42.323 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2206999999999999e-05 2024-03-20T09:00:42.323 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.323 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.323 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:42.323 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.782436+0000", 2024-03-20T09:00:42.323 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00083926599999999997 2024-03-20T09:00:42.323 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.323 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.324 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:42.324 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.782479+0000", 2024-03-20T09:00:42.324 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2599000000000003e-05 2024-03-20T09:00:42.324 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.324 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:42.324 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.324 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.324 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.324 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.24550.0:1361 2.7 e125/121 2:e2795900:::benchmark_data_smithi122_107654_object1360:head v 125'87, mlcod=125'84)", 2024-03-20T09:00:42.324 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:58:24.752693+0000", 2024-03-20T09:00:42.324 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 137.53070220000001, 2024-03-20T09:00:42.324 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.02981021, 2024-03-20T09:00:42.324 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:42.324 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:42.325 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:42.325 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.325 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:42.325 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.752693+0000", 2024-03-20T09:00:42.325 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.325 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.325 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.325 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:42.325 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.752693+0000", 2024-03-20T09:00:42.325 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.325 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.325 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.325 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:42.325 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.752693+0000", 2024-03-20T09:00:42.326 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3900000000000004e-07 2024-03-20T09:00:42.326 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.326 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.326 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:42.326 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.752789+0000", 2024-03-20T09:00:42.326 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5447999999999998e-05 2024-03-20T09:00:42.326 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.326 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.326 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:42.326 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.781602+0000", 2024-03-20T09:00:42.326 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.028812963 2024-03-20T09:00:42.326 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.326 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.326 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:42.327 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.781613+0000", 2024-03-20T09:00:42.327 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1681e-05 2024-03-20T09:00:42.327 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.327 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.327 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:42.327 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.781663+0000", 2024-03-20T09:00:42.327 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0136000000000002e-05 2024-03-20T09:00:42.327 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.327 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.327 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:42.327 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.781684+0000", 2024-03-20T09:00:42.327 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0860999999999999e-05 2024-03-20T09:00:42.328 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.328 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.328 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:42.328 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.782468+0000", 2024-03-20T09:00:42.328 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00078375700000000005 2024-03-20T09:00:42.328 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.328 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.328 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:42.328 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.782503+0000", 2024-03-20T09:00:42.328 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4724999999999997e-05 2024-03-20T09:00:42.328 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.328 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:42.328 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.328 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.329 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.329 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.24550.0:1364 2.0 e125/121 2:04548208:::benchmark_data_smithi122_107654_object1363:head v 125'78, mlcod=125'75)", 2024-03-20T09:00:42.329 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:58:24.753476+0000", 2024-03-20T09:00:42.329 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 137.529919001, 2024-03-20T09:00:42.329 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.029272663000000001, 2024-03-20T09:00:42.329 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:42.329 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:42.329 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:42.329 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.329 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:42.329 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.753476+0000", 2024-03-20T09:00:42.329 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.329 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.329 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.330 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:42.330 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.753476+0000", 2024-03-20T09:00:42.330 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.330 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.330 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.330 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:42.330 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.753476+0000", 2024-03-20T09:00:42.330 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8299999999999996e-07 2024-03-20T09:00:42.330 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.330 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.330 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:42.330 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.753504+0000", 2024-03-20T09:00:42.330 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7324000000000001e-05 2024-03-20T09:00:42.330 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.331 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.331 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:42.331 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.781569+0000", 2024-03-20T09:00:42.331 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.028065203 2024-03-20T09:00:42.331 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.331 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.331 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:42.331 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.781582+0000", 2024-03-20T09:00:42.331 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3142999999999999e-05 2024-03-20T09:00:42.331 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.331 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.331 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:42.331 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.781814+0000", 2024-03-20T09:00:42.331 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00023215000000000001 2024-03-20T09:00:42.331 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.332 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.332 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:42.332 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.781827+0000", 2024-03-20T09:00:42.332 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2493999999999999e-05 2024-03-20T09:00:42.332 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.332 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.332 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:42.332 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.782714+0000", 2024-03-20T09:00:42.332 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00088687 2024-03-20T09:00:42.333 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.333 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.333 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:42.333 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.782748+0000", 2024-03-20T09:00:42.333 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4696000000000001e-05 2024-03-20T09:00:42.333 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.333 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:42.333 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.333 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.333 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.333 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.24550.0:134686 2.0 e154/121 2:06c44837:::benchmark_data_smithi122_107654_object15901:head v 154'8325, mlcod=154'8323)", 2024-03-20T09:00:42.334 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:00:10.874658+0000", 2024-03-20T09:00:42.334 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 31.408736288, 2024-03-20T09:00:42.334 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.024269978000000001, 2024-03-20T09:00:42.334 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:42.334 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:42.334 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:42.334 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.334 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:42.334 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:10.874658+0000", 2024-03-20T09:00:42.334 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.334 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.334 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.335 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:42.335 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:10.874658+0000", 2024-03-20T09:00:42.335 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.335 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.335 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.335 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:42.335 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:10.874659+0000", 2024-03-20T09:00:42.335 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5000000000000001e-07 2024-03-20T09:00:42.335 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.335 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.335 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:42.335 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:10.874667+0000", 2024-03-20T09:00:42.335 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9300000000000003e-06 2024-03-20T09:00:42.335 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.336 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.336 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:42.336 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:10.874684+0000", 2024-03-20T09:00:42.336 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6708999999999999e-05 2024-03-20T09:00:42.336 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.336 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.336 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:42.336 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:10.874692+0000", 2024-03-20T09:00:42.336 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9780000000000005e-06 2024-03-20T09:00:42.336 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.336 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.336 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:42.336 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:10.874795+0000", 2024-03-20T09:00:42.336 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000102786 2024-03-20T09:00:42.337 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.337 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.337 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:42.337 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:10.874816+0000", 2024-03-20T09:00:42.337 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1060000000000002e-05 2024-03-20T09:00:42.337 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.337 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.337 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:42.337 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:10.898885+0000", 2024-03-20T09:00:42.337 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.024069426000000001 2024-03-20T09:00:42.337 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.337 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.337 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:42.338 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:10.898928+0000", 2024-03-20T09:00:42.338 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3238999999999997e-05 2024-03-20T09:00:42.338 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.338 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:42.338 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.338 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.338 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.338 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.24550.0:135138 2.0 e154/121 2:02320770:::benchmark_data_smithi122_107654_object16353:head v 154'8344, mlcod=154'8342)", 2024-03-20T09:00:42.338 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:00:11.287923+0000", 2024-03-20T09:00:42.338 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 30.995471405, 2024-03-20T09:00:42.338 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019597415, 2024-03-20T09:00:42.338 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:42.338 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:42.339 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:42.339 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.339 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:42.339 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.287923+0000", 2024-03-20T09:00:42.339 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.339 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.339 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.339 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:42.339 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.287923+0000", 2024-03-20T09:00:42.339 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.339 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.339 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.339 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:42.339 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.287924+0000", 2024-03-20T09:00:42.340 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.032e-06 2024-03-20T09:00:42.340 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.340 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.340 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:42.340 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.287933+0000", 2024-03-20T09:00:42.340 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2050000000000002e-06 2024-03-20T09:00:42.340 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.340 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.340 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:42.340 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.287949+0000", 2024-03-20T09:00:42.340 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6526000000000001e-05 2024-03-20T09:00:42.340 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.340 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.340 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:42.341 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.287959+0000", 2024-03-20T09:00:42.341 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6260000000000007e-06 2024-03-20T09:00:42.341 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.341 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.341 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:42.341 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.288046+0000", 2024-03-20T09:00:42.341 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7361000000000006e-05 2024-03-20T09:00:42.341 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.341 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.341 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:42.341 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.288070+0000", 2024-03-20T09:00:42.341 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4031000000000001e-05 2024-03-20T09:00:42.341 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.341 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.342 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:42.342 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.307479+0000", 2024-03-20T09:00:42.342 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019409342 2024-03-20T09:00:42.342 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.342 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.342 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:42.342 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.307521+0000", 2024-03-20T09:00:42.342 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1291999999999997e-05 2024-03-20T09:00:42.342 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.342 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:42.342 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.342 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.342 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.343 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.24550.0:135139 2.0 e154/121 2:0b0a0cda:::benchmark_data_smithi122_107654_object16354:head v 154'8345, mlcod=154'8342)", 2024-03-20T09:00:42.343 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:00:11.288850+0000", 2024-03-20T09:00:42.343 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 30.994544866999998, 2024-03-20T09:00:42.343 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.034181531000000001, 2024-03-20T09:00:42.343 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:42.343 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:42.343 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:42.343 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.343 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:42.343 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.288850+0000", 2024-03-20T09:00:42.343 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.343 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.343 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.343 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:42.344 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.288850+0000", 2024-03-20T09:00:42.344 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.344 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.344 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.344 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:42.344 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.288851+0000", 2024-03-20T09:00:42.344 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1900000000000002e-07 2024-03-20T09:00:42.344 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.344 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.344 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:42.344 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.288857+0000", 2024-03-20T09:00:42.344 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1430000000000003e-06 2024-03-20T09:00:42.344 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.344 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.344 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:42.345 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.288870+0000", 2024-03-20T09:00:42.345 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3397000000000001e-05 2024-03-20T09:00:42.345 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.345 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.345 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:42.345 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.288877+0000", 2024-03-20T09:00:42.345 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4690000000000002e-06 2024-03-20T09:00:42.345 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.345 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.345 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:42.345 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.300389+0000", 2024-03-20T09:00:42.345 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011512302 2024-03-20T09:00:42.345 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.345 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.346 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:42.346 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.300413+0000", 2024-03-20T09:00:42.346 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4578000000000001e-05 2024-03-20T09:00:42.346 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.346 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.346 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:42.346 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.322981+0000", 2024-03-20T09:00:42.346 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.022567907000000002 2024-03-20T09:00:42.346 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.346 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.346 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:42.346 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.323031+0000", 2024-03-20T09:00:42.346 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0015999999999997e-05 2024-03-20T09:00:42.346 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.347 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:42.347 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.347 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.347 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.347 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.24550.0:135150 2.0 e154/121 2:025d5434:::benchmark_data_smithi122_107654_object16365:head v 154'8346, mlcod=154'8342)", 2024-03-20T09:00:42.347 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:00:11.300075+0000", 2024-03-20T09:00:42.347 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 30.983319810000001, 2024-03-20T09:00:42.347 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.022990370999999999, 2024-03-20T09:00:42.347 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:42.347 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:42.347 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:42.347 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.347 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:42.347 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.300075+0000", 2024-03-20T09:00:42.348 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.348 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.348 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.348 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:42.348 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.300075+0000", 2024-03-20T09:00:42.348 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.348 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.348 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.348 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:42.348 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.300076+0000", 2024-03-20T09:00:42.348 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.161e-06 2024-03-20T09:00:42.348 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.348 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.349 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:42.349 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.300086+0000", 2024-03-20T09:00:42.349 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0018e-05 2024-03-20T09:00:42.349 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.349 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.349 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:42.349 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.300102+0000", 2024-03-20T09:00:42.349 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5970000000000001e-05 2024-03-20T09:00:42.349 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.349 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.349 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:42.349 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.300112+0000", 2024-03-20T09:00:42.349 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.842e-06 2024-03-20T09:00:42.349 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.350 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.350 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:42.350 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.303825+0000", 2024-03-20T09:00:42.350 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003712777 2024-03-20T09:00:42.350 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.350 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.350 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:42.350 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.303846+0000", 2024-03-20T09:00:42.350 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1060000000000002e-05 2024-03-20T09:00:42.350 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.350 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.350 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:42.350 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.323034+0000", 2024-03-20T09:00:42.350 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019188377999999999 2024-03-20T09:00:42.351 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.351 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.351 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:42.351 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.323065+0000", 2024-03-20T09:00:42.351 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1164999999999997e-05 2024-03-20T09:00:42.351 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.351 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:42.351 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.351 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.351 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.351 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.24550.0:135153 2.0 e154/121 2:0663c568:::benchmark_data_smithi122_107654_object16368:head v 154'8347, mlcod=154'8342)", 2024-03-20T09:00:42.351 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:00:11.300867+0000", 2024-03-20T09:00:42.351 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 30.982528187, 2024-03-20T09:00:42.351 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.022227958999999999, 2024-03-20T09:00:42.352 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:42.352 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:42.352 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:42.352 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.352 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:42.352 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.300867+0000", 2024-03-20T09:00:42.352 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.352 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.352 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.352 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:42.352 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.300867+0000", 2024-03-20T09:00:42.352 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.352 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.352 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.353 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:42.353 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.300867+0000", 2024-03-20T09:00:42.353 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4099999999999998e-07 2024-03-20T09:00:42.353 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.353 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.353 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:42.353 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.300873+0000", 2024-03-20T09:00:42.353 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7710000000000003e-06 2024-03-20T09:00:42.353 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.353 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.353 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:42.353 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.300883+0000", 2024-03-20T09:00:42.353 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7159999999999998e-06 2024-03-20T09:00:42.353 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.354 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.354 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:42.354 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.300889+0000", 2024-03-20T09:00:42.354 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7150000000000001e-06 2024-03-20T09:00:42.354 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.354 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.354 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:42.354 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.307548+0000", 2024-03-20T09:00:42.354 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0066589149999999996 2024-03-20T09:00:42.354 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.354 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.354 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:42.354 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.307565+0000", 2024-03-20T09:00:42.354 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6728e-05 2024-03-20T09:00:42.355 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.355 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.355 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:42.355 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.323067+0000", 2024-03-20T09:00:42.355 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015502203000000001 2024-03-20T09:00:42.355 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.355 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.355 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:42.355 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.323094+0000", 2024-03-20T09:00:42.355 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7169999999999999e-05 2024-03-20T09:00:42.355 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.355 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:42.355 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.355 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.356 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.356 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.24550.0:135154 2.0 e154/121 2:002815b7:::benchmark_data_smithi122_107654_object16369:head v 154'8348, mlcod=154'8342)", 2024-03-20T09:00:42.356 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:00:11.305922+0000", 2024-03-20T09:00:42.356 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 30.977472247000001, 2024-03-20T09:00:42.356 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017218555999999999, 2024-03-20T09:00:42.356 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:42.356 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:42.356 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:42.356 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.356 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:42.356 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.305922+0000", 2024-03-20T09:00:42.356 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.356 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.356 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.357 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:42.357 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.305922+0000", 2024-03-20T09:00:42.357 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.357 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.357 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.357 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:42.357 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.305924+0000", 2024-03-20T09:00:42.357 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.164e-06 2024-03-20T09:00:42.357 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.357 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.357 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:42.357 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.305932+0000", 2024-03-20T09:00:42.357 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4999999999999999e-06 2024-03-20T09:00:42.358 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.358 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.358 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:42.358 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.305946+0000", 2024-03-20T09:00:42.358 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4215e-05 2024-03-20T09:00:42.358 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.358 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.358 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:42.358 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.305955+0000", 2024-03-20T09:00:42.358 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2870000000000004e-06 2024-03-20T09:00:42.358 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.358 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.358 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:42.358 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.310669+0000", 2024-03-20T09:00:42.359 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0047146820000000004 2024-03-20T09:00:42.359 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.359 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.359 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:42.359 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.310682+0000", 2024-03-20T09:00:42.359 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3038e-05 2024-03-20T09:00:42.359 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.359 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.359 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:42.359 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.323099+0000", 2024-03-20T09:00:42.359 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012416536000000001 2024-03-20T09:00:42.359 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.359 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.359 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:42.359 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.323141+0000", 2024-03-20T09:00:42.360 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2134e-05 2024-03-20T09:00:42.360 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.360 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:42.360 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.360 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.360 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.360 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.24550.0:135156 2.0 e154/121 2:0df3b349:::benchmark_data_smithi122_107654_object16371:head v 154'8349, mlcod=154'8342)", 2024-03-20T09:00:42.360 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:00:11.309037+0000", 2024-03-20T09:00:42.360 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 30.974357648000002, 2024-03-20T09:00:42.360 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017428322999999999, 2024-03-20T09:00:42.360 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:42.361 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:42.361 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:42.361 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.361 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:42.361 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.309037+0000", 2024-03-20T09:00:42.361 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.361 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.361 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.361 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:42.361 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.309037+0000", 2024-03-20T09:00:42.361 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.361 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.361 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.361 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:42.362 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.309038+0000", 2024-03-20T09:00:42.362 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8400000000000003e-07 2024-03-20T09:00:42.362 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.362 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.362 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:42.362 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.309045+0000", 2024-03-20T09:00:42.362 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4630000000000004e-06 2024-03-20T09:00:42.362 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.362 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.362 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:42.362 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.309055+0000", 2024-03-20T09:00:42.362 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3959999999999995e-06 2024-03-20T09:00:42.362 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.363 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.363 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:42.363 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.309062+0000", 2024-03-20T09:00:42.363 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8539999999999999e-06 2024-03-20T09:00:42.363 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.363 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.363 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:42.363 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.313597+0000", 2024-03-20T09:00:42.363 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0045356939999999998 2024-03-20T09:00:42.363 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.363 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.363 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:42.363 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.313611+0000", 2024-03-20T09:00:42.363 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3549e-05 2024-03-20T09:00:42.364 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.364 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.364 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:42.364 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.326420+0000", 2024-03-20T09:00:42.364 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012808686999999999 2024-03-20T09:00:42.364 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.364 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.364 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:42.364 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.326465+0000", 2024-03-20T09:00:42.364 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5796e-05 2024-03-20T09:00:42.364 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.364 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:42.364 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.365 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.365 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.365 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.24550.0:135157 2.0 e154/121 2:0d47b275:::benchmark_data_smithi122_107654_object16372:head v 154'8350, mlcod=154'8342)", 2024-03-20T09:00:42.365 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:00:11.309075+0000", 2024-03-20T09:00:42.365 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 30.974319502, 2024-03-20T09:00:42.365 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017410459, 2024-03-20T09:00:42.365 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:42.365 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:42.365 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:42.365 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.365 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:42.365 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.309075+0000", 2024-03-20T09:00:42.365 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.365 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.366 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.366 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:42.366 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.309075+0000", 2024-03-20T09:00:42.366 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.366 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.366 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.366 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:42.366 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.309075+0000", 2024-03-20T09:00:42.366 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5899999999999998e-07 2024-03-20T09:00:42.366 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.366 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.366 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:42.366 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.309079+0000", 2024-03-20T09:00:42.366 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.76e-06 2024-03-20T09:00:42.367 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.367 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.367 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:42.367 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.309084+0000", 2024-03-20T09:00:42.367 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1270000000000002e-06 2024-03-20T09:00:42.367 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.367 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.367 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:42.367 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.309088+0000", 2024-03-20T09:00:42.367 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0119999999999999e-06 2024-03-20T09:00:42.367 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.367 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.367 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:42.368 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.319390+0000", 2024-03-20T09:00:42.368 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010301319 2024-03-20T09:00:42.368 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.368 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.368 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:42.368 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.319412+0000", 2024-03-20T09:00:42.368 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2215999999999999e-05 2024-03-20T09:00:42.368 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.368 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.368 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:42.368 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.326469+0000", 2024-03-20T09:00:42.368 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0070573040000000004 2024-03-20T09:00:42.368 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.368 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.369 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:42.369 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.326486+0000", 2024-03-20T09:00:42.369 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6461999999999999e-05 2024-03-20T09:00:42.369 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.369 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:42.369 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.369 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.369 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.369 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.24550.0:139556 2.0 e154/121 2:01ea06d7:::benchmark_data_smithi122_107654_object20771:head v 154'8615, mlcod=154'8613)", 2024-03-20T09:00:42.369 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:00:15.777342+0000", 2024-03-20T09:00:42.369 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 26.506052689000001, 2024-03-20T09:00:42.369 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.022714399, 2024-03-20T09:00:42.369 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:42.369 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:42.370 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:42.370 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.370 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:42.370 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.777342+0000", 2024-03-20T09:00:42.370 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.370 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.370 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.370 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:42.370 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.777342+0000", 2024-03-20T09:00:42.370 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.370 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.370 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.370 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:42.370 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.777343+0000", 2024-03-20T09:00:42.371 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0589999999999999e-06 2024-03-20T09:00:42.371 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.371 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.371 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:42.371 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.777350+0000", 2024-03-20T09:00:42.371 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9120000000000001e-06 2024-03-20T09:00:42.371 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.371 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.371 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:42.371 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.777362+0000", 2024-03-20T09:00:42.371 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1661999999999999e-05 2024-03-20T09:00:42.371 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.371 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.371 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:42.372 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.777368+0000", 2024-03-20T09:00:42.372 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6239999999999996e-06 2024-03-20T09:00:42.372 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.372 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.372 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:42.372 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.777423+0000", 2024-03-20T09:00:42.372 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4243999999999998e-05 2024-03-20T09:00:42.372 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.372 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.372 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:42.372 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.777446+0000", 2024-03-20T09:00:42.372 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3319000000000002e-05 2024-03-20T09:00:42.372 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.372 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.373 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:42.373 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.800022+0000", 2024-03-20T09:00:42.373 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.022576269 2024-03-20T09:00:42.373 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.373 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.373 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:42.373 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.800056+0000", 2024-03-20T09:00:42.373 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4310000000000002e-05 2024-03-20T09:00:42.373 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.373 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:42.373 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.373 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.373 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.373 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.24550.0:139560 2.0 e154/121 2:05b961a6:::benchmark_data_smithi122_107654_object20775:head v 154'8616, mlcod=154'8613)", 2024-03-20T09:00:42.374 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:00:15.777389+0000", 2024-03-20T09:00:42.374 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 26.506006068000001, 2024-03-20T09:00:42.374 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.022693640000000001, 2024-03-20T09:00:42.374 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:42.374 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:42.374 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:42.374 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.374 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:42.374 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.777389+0000", 2024-03-20T09:00:42.374 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.374 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.374 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.374 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:42.374 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.777389+0000", 2024-03-20T09:00:42.375 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.375 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.375 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.375 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:42.375 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.777389+0000", 2024-03-20T09:00:42.375 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2599999999999998e-07 2024-03-20T09:00:42.375 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.375 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.375 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:42.375 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.777394+0000", 2024-03-20T09:00:42.375 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6399999999999996e-06 2024-03-20T09:00:42.375 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.375 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.375 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:42.376 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.777399+0000", 2024-03-20T09:00:42.376 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5350000000000004e-06 2024-03-20T09:00:42.376 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.376 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.376 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:42.376 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.777404+0000", 2024-03-20T09:00:42.376 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8489999999999997e-06 2024-03-20T09:00:42.376 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.376 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.376 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:42.376 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.780533+0000", 2024-03-20T09:00:42.376 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0031285190000000002 2024-03-20T09:00:42.376 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.376 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.377 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:42.377 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.780553+0000", 2024-03-20T09:00:42.377 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0752000000000001e-05 2024-03-20T09:00:42.377 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.377 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.377 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:42.377 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.800059+0000", 2024-03-20T09:00:42.377 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019505663999999999 2024-03-20T09:00:42.377 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.377 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.377 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:42.377 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.800082+0000", 2024-03-20T09:00:42.377 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3354999999999999e-05 2024-03-20T09:00:42.377 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.378 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:42.378 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.378 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.378 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.378 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.24550.0:139562 2.0 e154/121 2:0310d4c1:::benchmark_data_smithi122_107654_object20777:head v 154'8617, mlcod=154'8613)", 2024-03-20T09:00:42.378 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:00:15.779830+0000", 2024-03-20T09:00:42.378 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 26.503564957000002, 2024-03-20T09:00:42.378 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.020278715999999999, 2024-03-20T09:00:42.378 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:42.378 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:42.378 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:42.378 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.378 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:42.378 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.779830+0000", 2024-03-20T09:00:42.379 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.379 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.379 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.379 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:42.379 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.779830+0000", 2024-03-20T09:00:42.379 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.379 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.379 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.379 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:42.379 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.779831+0000", 2024-03-20T09:00:42.379 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3399999999999997e-07 2024-03-20T09:00:42.379 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.379 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.379 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:42.380 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.779838+0000", 2024-03-20T09:00:42.380 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7079999999999997e-06 2024-03-20T09:00:42.380 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.380 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.380 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:42.380 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.779859+0000", 2024-03-20T09:00:42.380 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0547000000000002e-05 2024-03-20T09:00:42.380 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.380 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.380 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:42.380 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.779868+0000", 2024-03-20T09:00:42.380 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5839999999999995e-06 2024-03-20T09:00:42.380 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.381 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.381 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:42.381 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.784435+0000", 2024-03-20T09:00:42.381 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0045674979999999997 2024-03-20T09:00:42.381 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.381 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.381 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:42.381 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.784459+0000", 2024-03-20T09:00:42.381 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3634000000000001e-05 2024-03-20T09:00:42.381 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.381 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.381 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:42.381 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.800084+0000", 2024-03-20T09:00:42.381 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015625239999999999 2024-03-20T09:00:42.381 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.382 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.382 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:42.382 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.800108+0000", 2024-03-20T09:00:42.382 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4570999999999999e-05 2024-03-20T09:00:42.382 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.382 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:42.382 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.382 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.382 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.382 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.24550.0:139573 2.0 e154/121 2:02269d87:::benchmark_data_smithi122_107654_object20788:head v 154'8621, mlcod=154'8613)", 2024-03-20T09:00:42.382 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:00:15.789501+0000", 2024-03-20T09:00:42.382 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 26.493893599, 2024-03-20T09:00:42.382 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017877295000000001, 2024-03-20T09:00:42.383 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:42.383 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:42.383 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:42.383 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.383 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:42.383 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.789501+0000", 2024-03-20T09:00:42.383 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.383 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.383 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.383 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:42.383 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.789501+0000", 2024-03-20T09:00:42.383 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.383 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.383 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.384 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:42.384 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.789502+0000", 2024-03-20T09:00:42.384 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8800000000000001e-07 2024-03-20T09:00:42.384 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.384 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.384 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:42.384 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.789510+0000", 2024-03-20T09:00:42.384 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4980000000000006e-06 2024-03-20T09:00:42.384 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.384 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.384 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:42.384 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.789527+0000", 2024-03-20T09:00:42.384 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6059e-05 2024-03-20T09:00:42.384 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.384 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.385 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:42.385 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.789535+0000", 2024-03-20T09:00:42.385 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5869999999999993e-06 2024-03-20T09:00:42.385 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.385 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.385 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:42.385 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.800549+0000", 2024-03-20T09:00:42.385 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011013762999999999 2024-03-20T09:00:42.385 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.385 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.385 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:42.385 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.800569+0000", 2024-03-20T09:00:42.385 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0479000000000002e-05 2024-03-20T09:00:42.385 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.386 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.386 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:42.386 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.807315+0000", 2024-03-20T09:00:42.386 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0067459870000000002 2024-03-20T09:00:42.386 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.386 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.386 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:42.386 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:15.807378+0000", 2024-03-20T09:00:42.386 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3034000000000007e-05 2024-03-20T09:00:42.386 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.386 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:42.386 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.386 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.386 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.387 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.24550.0:149859 2.4 e157/121 2:26ae561b:::benchmark_data_smithi122_107654_object31074:head v 157'9382, mlcod=157'9380)", 2024-03-20T09:00:42.387 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:00:26.625477+0000", 2024-03-20T09:00:42.387 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 15.65791739, 2024-03-20T09:00:42.387 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017211601999999999, 2024-03-20T09:00:42.387 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:42.387 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:42.387 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:42.387 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.387 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:42.387 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:26.625477+0000", 2024-03-20T09:00:42.387 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.387 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.387 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.387 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:42.388 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:26.625477+0000", 2024-03-20T09:00:42.390 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.390 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.390 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.390 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:42.390 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:26.625479+0000", 2024-03-20T09:00:42.390 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.412e-06 2024-03-20T09:00:42.390 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.390 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.390 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:42.390 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:26.625492+0000", 2024-03-20T09:00:42.390 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.306e-05 2024-03-20T09:00:42.390 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.390 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.391 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:42.391 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:26.625511+0000", 2024-03-20T09:00:42.391 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9210000000000001e-05 2024-03-20T09:00:42.391 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.391 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.391 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:42.391 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:26.625522+0000", 2024-03-20T09:00:42.391 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1018e-05 2024-03-20T09:00:42.391 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.391 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.391 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:42.391 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:26.625800+0000", 2024-03-20T09:00:42.391 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00027755500000000002 2024-03-20T09:00:42.391 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.392 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.392 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:42.392 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:26.625825+0000", 2024-03-20T09:00:42.392 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5896000000000001e-05 2024-03-20T09:00:42.392 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.392 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.392 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:42.392 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:26.642641+0000", 2024-03-20T09:00:42.392 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016815457999999998 2024-03-20T09:00:42.392 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.392 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.392 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:42.392 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:26.642689+0000", 2024-03-20T09:00:42.392 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7992999999999999e-05 2024-03-20T09:00:42.393 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.393 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:42.393 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.393 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.393 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.393 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.24550.0:150183 2.7 e157/121 2:ec12418d:::benchmark_data_smithi122_107654_object31398:head v 157'9393, mlcod=157'9391)", 2024-03-20T09:00:42.393 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:00:26.972940+0000", 2024-03-20T09:00:42.393 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 15.310454579, 2024-03-20T09:00:42.393 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018073583000000001, 2024-03-20T09:00:42.393 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:42.393 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:42.393 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:42.393 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.393 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:42.394 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:26.972940+0000", 2024-03-20T09:00:42.394 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.394 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.394 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.394 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:42.394 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:26.972940+0000", 2024-03-20T09:00:42.394 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.394 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.394 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.394 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:42.394 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:26.972941+0000", 2024-03-20T09:00:42.394 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.105e-06 2024-03-20T09:00:42.394 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.395 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.395 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:42.395 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:26.972952+0000", 2024-03-20T09:00:42.395 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0550000000000001e-05 2024-03-20T09:00:42.395 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.395 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.395 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:42.395 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:26.972968+0000", 2024-03-20T09:00:42.395 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5724000000000001e-05 2024-03-20T09:00:42.395 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.395 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.395 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:42.395 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:26.972977+0000", 2024-03-20T09:00:42.395 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0219999999999993e-06 2024-03-20T09:00:42.396 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.396 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.396 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:42.396 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:26.973084+0000", 2024-03-20T09:00:42.396 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000107697 2024-03-20T09:00:42.396 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.396 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.396 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:42.396 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:26.973103+0000", 2024-03-20T09:00:42.396 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8947999999999999e-05 2024-03-20T09:00:42.396 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.396 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.396 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:42.396 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:26.990966+0000", 2024-03-20T09:00:42.396 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017863179 2024-03-20T09:00:42.397 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.397 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.397 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:42.397 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:26.991014+0000", 2024-03-20T09:00:42.397 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7358000000000003e-05 2024-03-20T09:00:42.397 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.397 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:42.397 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.397 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.397 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.397 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.24550.0:152497 2.4 e159/121 2:2c50e55f:::benchmark_data_smithi122_107654_object33712:head v 159'9581, mlcod=159'9578)", 2024-03-20T09:00:42.397 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:00:29.426270+0000", 2024-03-20T09:00:42.397 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 12.857125098999999, 2024-03-20T09:00:42.397 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019963825000000001, 2024-03-20T09:00:42.398 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:42.398 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:42.398 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:42.398 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.398 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:42.398 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:29.426270+0000", 2024-03-20T09:00:42.398 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.398 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.398 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.398 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:42.398 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:29.426270+0000", 2024-03-20T09:00:42.398 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.398 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.399 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.399 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:42.399 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:29.426271+0000", 2024-03-20T09:00:42.399 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.198e-06 2024-03-20T09:00:42.399 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.399 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.399 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:42.399 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:29.426279+0000", 2024-03-20T09:00:42.399 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8420000000000005e-06 2024-03-20T09:00:42.399 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.399 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.399 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:42.399 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:29.426302+0000", 2024-03-20T09:00:42.399 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3339000000000001e-05 2024-03-20T09:00:42.400 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.400 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.400 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:42.400 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:29.426310+0000", 2024-03-20T09:00:42.400 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.96e-06 2024-03-20T09:00:42.400 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.400 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.400 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:42.400 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:29.426404+0000", 2024-03-20T09:00:42.400 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3966000000000001e-05 2024-03-20T09:00:42.400 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.400 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.400 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:42.400 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:29.426425+0000", 2024-03-20T09:00:42.401 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0888000000000001e-05 2024-03-20T09:00:42.401 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.401 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.401 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:42.401 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:29.446172+0000", 2024-03-20T09:00:42.401 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019746889 2024-03-20T09:00:42.401 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.401 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.401 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:42.401 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:29.446233+0000", 2024-03-20T09:00:42.401 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1742999999999995e-05 2024-03-20T09:00:42.401 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.401 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:42.401 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.402 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.402 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.402 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.24550.0:153547 2.4 e159/121 2:20c29b93:::benchmark_data_smithi122_107654_object34762:head v 159'9644, mlcod=159'9641)", 2024-03-20T09:00:42.402 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:00:30.607448+0000", 2024-03-20T09:00:42.402 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 11.675946551999999, 2024-03-20T09:00:42.402 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019246914, 2024-03-20T09:00:42.402 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:42.402 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:42.402 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:42.402 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.402 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:42.402 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:30.607448+0000", 2024-03-20T09:00:42.402 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.402 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.403 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.403 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:42.403 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:30.607448+0000", 2024-03-20T09:00:42.403 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.403 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.403 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.403 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:42.403 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:30.607449+0000", 2024-03-20T09:00:42.403 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2600000000000001e-07 2024-03-20T09:00:42.403 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.403 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.403 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:42.403 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:30.607454+0000", 2024-03-20T09:00:42.403 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4369999999999998e-06 2024-03-20T09:00:42.403 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.404 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.404 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:42.404 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:30.607463+0000", 2024-03-20T09:00:42.404 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6640000000000004e-06 2024-03-20T09:00:42.404 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.404 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.404 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:42.404 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:30.607469+0000", 2024-03-20T09:00:42.404 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4970000000000001e-06 2024-03-20T09:00:42.404 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.404 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.404 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:42.404 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:30.612373+0000", 2024-03-20T09:00:42.404 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0049042290000000004 2024-03-20T09:00:42.405 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.405 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.405 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:42.405 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:30.612393+0000", 2024-03-20T09:00:42.405 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0443000000000001e-05 2024-03-20T09:00:42.405 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.405 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.405 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:42.405 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:30.626646+0000", 2024-03-20T09:00:42.405 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.01425271 2024-03-20T09:00:42.405 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.405 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.405 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:42.405 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:30.626695+0000", 2024-03-20T09:00:42.405 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9107999999999998e-05 2024-03-20T09:00:42.406 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.406 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:42.406 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.406 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.406 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.406 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.24550.0:153639 2.0 e159/121 2:088ac428:::benchmark_data_smithi122_107654_object34854:head v 159'9446, mlcod=159'9443)", 2024-03-20T09:00:42.406 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:00:30.707775+0000", 2024-03-20T09:00:42.406 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 11.575619939999999, 2024-03-20T09:00:42.406 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.021158989999999999, 2024-03-20T09:00:42.406 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:42.406 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:42.406 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:42.406 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.407 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:42.407 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:30.707775+0000", 2024-03-20T09:00:42.407 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.407 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.407 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.407 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:42.407 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:30.707775+0000", 2024-03-20T09:00:42.407 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:42.407 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.407 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.407 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:42.407 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:30.707776+0000", 2024-03-20T09:00:42.407 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1829999999999999e-06 2024-03-20T09:00:42.407 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.408 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.408 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:42.408 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:30.707786+0000", 2024-03-20T09:00:42.408 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0132999999999999e-05 2024-03-20T09:00:42.408 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.408 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.408 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:42.408 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:30.707806+0000", 2024-03-20T09:00:42.408 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9656e-05 2024-03-20T09:00:42.408 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.408 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.408 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:42.408 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:30.707815+0000", 2024-03-20T09:00:42.408 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2930000000000006e-06 2024-03-20T09:00:42.408 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.409 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.409 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:42.409 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:30.707873+0000", 2024-03-20T09:00:42.409 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7966e-05 2024-03-20T09:00:42.409 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.409 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.409 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:42.409 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:30.707894+0000", 2024-03-20T09:00:42.409 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0726000000000001e-05 2024-03-20T09:00:42.409 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.409 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.409 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:42.409 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:30.728903+0000", 2024-03-20T09:00:42.409 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.021009773999999998 2024-03-20T09:00:42.410 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:42.410 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:42.410 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:42.410 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:30.728934+0000", 2024-03-20T09:00:42.410 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0258999999999999e-05 2024-03-20T09:00:42.410 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.410 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:42.410 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.410 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:42.410 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:42.410 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:00:42.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:42.412 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:00:42.412 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:00:42.513 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:00:42.543 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:42.559 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:00:42.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:42.562 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:00:42.663 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:00:42.693 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:42.699 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:00:42.714 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:00:42.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:42.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:00:42.820 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:00:42.848 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:42.908 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:00:42.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:42 smithi026 ceph-mon[68309]: pgmap v486: 17 pgs: 17 active+clean; 4.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 764 op/s 2024-03-20T09:00:42.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {auth=28,config=2,health=12,kv=53,logm=5} crc {auth=3842626008,config=3599132250,health=179246142,kv=3157025254,logm=1916008925}) 2024-03-20T09:00:42.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2614714409}) 2024-03-20T09:00:42.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3957732173}) 2024-03-20T09:00:42.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=949486325}) 2024-03-20T09:00:42.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2421990428}) 2024-03-20T09:00:42.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1589229951}) 2024-03-20T09:00:42.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1243207356}) 2024-03-20T09:00:42.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4152758543}) 2024-03-20T09:00:42.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2920391713}) 2024-03-20T09:00:42.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2045570403}) 2024-03-20T09:00:42.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2500037183}) 2024-03-20T09:00:42.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=465494191}) 2024-03-20T09:00:42.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2916002799}) 2024-03-20T09:00:42.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2396952289}) 2024-03-20T09:00:42.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=708052665}) 2024-03-20T09:00:42.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1156589624}) 2024-03-20T09:00:42.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4132078778}) 2024-03-20T09:00:42.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=873307856}) 2024-03-20T09:00:42.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=877198285}) 2024-03-20T09:00:42.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4122260855}) 2024-03-20T09:00:42.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3919049215}) 2024-03-20T09:00:42.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1193946668}) 2024-03-20T09:00:42.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1807344774}) 2024-03-20T09:00:42.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=811814791}) 2024-03-20T09:00:42.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=20,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=46} crc {logm=332830007,mdsmap=3984425268,mgr=427508465,mgr_command_descs=3831600018,mgr_metadata=2702834764,mgrstat=2516955680,mon_config_key=4257406585}) 2024-03-20T09:00:42.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=421847133}) 2024-03-20T09:00:42.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=679321205}) 2024-03-20T09:00:42.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=32,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=50} crc {mon_config_key=1750444256,monmap=3481182261,osd_metadata=813835912,osd_pg_creating=309268299,osdmap=1581046758}) 2024-03-20T09:00:42.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=63} crc {osdmap=2588021307}) 2024-03-20T09:00:43.085 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:42 smithi122 ceph-mon[73495]: pgmap v486: 17 pgs: 17 active+clean; 4.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 764 op/s 2024-03-20T09:00:43.085 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {auth=28,config=2,health=12,kv=53,logm=5} crc {auth=3842626008,config=3599132250,health=179246142,kv=3157025254,logm=1916008925}) 2024-03-20T09:00:43.085 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2614714409}) 2024-03-20T09:00:43.085 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3957732173}) 2024-03-20T09:00:43.086 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=949486325}) 2024-03-20T09:00:43.086 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2421990428}) 2024-03-20T09:00:43.086 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1589229951}) 2024-03-20T09:00:43.086 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1243207356}) 2024-03-20T09:00:43.086 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4152758543}) 2024-03-20T09:00:43.086 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2920391713}) 2024-03-20T09:00:43.086 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2045570403}) 2024-03-20T09:00:43.086 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2500037183}) 2024-03-20T09:00:43.086 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=465494191}) 2024-03-20T09:00:43.086 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2916002799}) 2024-03-20T09:00:43.086 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2396952289}) 2024-03-20T09:00:43.086 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=708052665}) 2024-03-20T09:00:43.086 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1156589624}) 2024-03-20T09:00:43.086 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4132078778}) 2024-03-20T09:00:43.087 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=873307856}) 2024-03-20T09:00:43.087 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=877198285}) 2024-03-20T09:00:43.087 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4122260855}) 2024-03-20T09:00:43.087 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3919049215}) 2024-03-20T09:00:43.087 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1193946668}) 2024-03-20T09:00:43.087 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1807344774}) 2024-03-20T09:00:43.087 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=811814791}) 2024-03-20T09:00:43.087 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=20,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=46} crc {logm=332830007,mdsmap=3984425268,mgr=427508465,mgr_command_descs=3831600018,mgr_metadata=2702834764,mgrstat=2516955680,mon_config_key=4257406585}) 2024-03-20T09:00:43.087 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=421847133}) 2024-03-20T09:00:43.087 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=679321205}) 2024-03-20T09:00:43.087 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=32,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=50} crc {mon_config_key=1750444256,monmap=3481182261,osd_metadata=813835912,osd_pg_creating=309268299,osdmap=1581046758}) 2024-03-20T09:00:43.087 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=63} crc {osdmap=2588021307}) 2024-03-20T09:00:43.096 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:00:43.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:43.099 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:00:43.111 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_ops_in_flight 2024-03-20T09:00:43.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:42 smithi115 ceph-mon[72476]: pgmap v486: 17 pgs: 17 active+clean; 4.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 764 op/s 2024-03-20T09:00:43.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {auth=28,config=2,health=12,kv=53,logm=5} crc {auth=3842626008,config=3599132250,health=179246142,kv=3157025254,logm=1916008925}) 2024-03-20T09:00:43.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2614714409}) 2024-03-20T09:00:43.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3957732173}) 2024-03-20T09:00:43.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=949486325}) 2024-03-20T09:00:43.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2421990428}) 2024-03-20T09:00:43.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1589229951}) 2024-03-20T09:00:43.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1243207356}) 2024-03-20T09:00:43.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4152758543}) 2024-03-20T09:00:43.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2920391713}) 2024-03-20T09:00:43.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2045570403}) 2024-03-20T09:00:43.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2500037183}) 2024-03-20T09:00:43.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=465494191}) 2024-03-20T09:00:43.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2916002799}) 2024-03-20T09:00:43.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2396952289}) 2024-03-20T09:00:43.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=708052665}) 2024-03-20T09:00:43.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1156589624}) 2024-03-20T09:00:43.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4132078778}) 2024-03-20T09:00:43.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=873307856}) 2024-03-20T09:00:43.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=877198285}) 2024-03-20T09:00:43.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4122260855}) 2024-03-20T09:00:43.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3919049215}) 2024-03-20T09:00:43.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1193946668}) 2024-03-20T09:00:43.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1807344774}) 2024-03-20T09:00:43.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=811814791}) 2024-03-20T09:00:43.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=20,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=46} crc {logm=332830007,mdsmap=3984425268,mgr=427508465,mgr_command_descs=3831600018,mgr_metadata=2702834764,mgrstat=2516955680,mon_config_key=4257406585}) 2024-03-20T09:00:43.158 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=421847133}) 2024-03-20T09:00:43.158 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=679321205}) 2024-03-20T09:00:43.158 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=32,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=50} crc {mon_config_key=1750444256,monmap=3481182261,osd_metadata=813835912,osd_pg_creating=309268299,osdmap=1581046758}) 2024-03-20T09:00:43.158 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=63} crc {osdmap=2588021307}) 2024-03-20T09:00:43.200 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:00:43.232 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:43.252 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:00:43.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:43.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:00:43.357 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:00:43.383 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:43.411 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:00:43.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:43.414 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:00:43.418 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:00:43.514 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:00:43.543 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:43.564 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:00:43.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:43.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:00:43.668 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:00:43.696 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:43.718 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:00:43.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:43.721 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:00:43.822 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:00:43.856 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:43.876 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:00:43.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:43.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:00:43.980 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:00:44.013 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:44.033 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:00:44.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:44.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:00:44.138 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:00:44.166 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:44.185 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:00:44.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:44.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:00:44.289 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:00:44.316 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:44.808 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:00:44.828 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:44 smithi026 ceph-mon[68309]: pgmap v487: 17 pgs: 17 active+clean; 4.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 621 op/s 2024-03-20T09:00:44.828 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:44 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2997016181' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:00:44.828 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:44 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:00:45.065 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:00:45.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:45.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:00:45.117 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:44 smithi122 ceph-mon[73495]: pgmap v487: 17 pgs: 17 active+clean; 4.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 621 op/s 2024-03-20T09:00:45.117 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:44 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2997016181' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:00:45.117 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:44 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:00:45.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:44 smithi115 ceph-mon[72476]: pgmap v487: 17 pgs: 17 active+clean; 4.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 621 op/s 2024-03-20T09:00:45.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:44 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2997016181' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:00:45.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:44 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:00:45.168 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:00:45.197 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:45.400 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:00:45.402 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:00:45.402 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:00:45.402 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-03-20T09:00:45.402 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:00:45.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:45.404 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:00:45.505 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:00:45.509 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:00:45.509 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:00:45.509 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 164, 2024-03-20T09:00:45.509 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "852fbb68-e696-11ee-95c9-87774f69a715", 2024-03-20T09:00:45.509 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-03-20T08:48:35.281088+0000", 2024-03-20T09:00:45.509 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-03-20T09:00:44.794357+0000", 2024-03-20T09:00:45.509 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2024-03-20T08:56:17.873738+0000", 2024-03-20T09:00:45.509 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2024-03-20T08:55:59.486577+0000", 2024-03-20T09:00:45.509 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-20T09:00:45.510 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5804032, 2024-03-20T09:00:45.510 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2024-03-20T09:00:45.510 INFO:teuthology.orchestra.run.smithi026.stdout: "nodeep-scrub", 2024-03-20T09:00:45.510 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2024-03-20T09:00:45.510 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2024-03-20T09:00:45.510 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2024-03-20T09:00:45.510 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2024-03-20T09:00:45.510 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:00:45.510 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 26, 2024-03-20T09:00:45.510 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2024-03-20T09:00:45.510 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-20T09:00:45.510 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-20T09:00:45.510 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2024-03-20T09:00:45.510 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 2, 2024-03-20T09:00:45.511 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 12, 2024-03-20T09:00:45.511 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "luminous", 2024-03-20T09:00:45.511 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "jewel", 2024-03-20T09:00:45.511 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "squid", 2024-03-20T09:00:45.511 INFO:teuthology.orchestra.run.smithi026.stdout: "allow_crimson": false, 2024-03-20T09:00:45.511 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2024-03-20T09:00:45.511 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.511 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2024-03-20T09:00:45.511 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": ".mgr", 2024-03-20T09:00:45.511 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-03-20T08:52:17.442324+0000", 2024-03-20T09:00:45.511 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-03-20T09:00:45.511 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-03-20T09:00:45.511 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-03-20T09:00:45.511 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 3, 2024-03-20T09:00:45.512 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 2, 2024-03-20T09:00:45.512 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-03-20T09:00:45.512 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-03-20T09:00:45.512 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-03-20T09:00:45.512 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-03-20T09:00:45.512 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-20T09:00:45.512 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-03-20T09:00:45.512 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-03-20T09:00:45.512 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 1, 2024-03-20T09:00:45.512 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 1, 2024-03-20T09:00:45.512 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 1, 2024-03-20T09:00:45.513 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 1, 2024-03-20T09:00:45.513 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 1, 2024-03-20T09:00:45.513 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-03-20T09:00:45.513 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-03-20T09:00:45.513 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-03-20T09:00:45.513 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-03-20T09:00:45.513 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-03-20T09:00:45.513 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-03-20T09:00:45.513 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-03-20T09:00:45.513 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.514 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "131", 2024-03-20T09:00:45.514 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-03-20T09:00:45.514 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-20T09:00:45.514 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2024-03-20T09:00:45.514 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-03-20T09:00:45.514 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-03-20T09:00:45.514 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-03-20T09:00:45.514 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-03-20T09:00:45.514 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-03-20T09:00:45.514 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-03-20T09:00:45.515 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-03-20T09:00:45.515 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-03-20T09:00:45.515 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-03-20T09:00:45.515 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-03-20T09:00:45.515 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-03-20T09:00:45.515 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-03-20T09:00:45.515 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-03-20T09:00:45.515 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-03-20T09:00:45.515 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-03-20T09:00:45.515 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-20T09:00:45.515 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-20T09:00:45.515 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-20T09:00:45.515 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-03-20T09:00:45.516 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-03-20T09:00:45.516 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-03-20T09:00:45.516 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-03-20T09:00:45.516 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-03-20T09:00:45.516 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.516 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-03-20T09:00:45.516 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-03-20T09:00:45.516 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-03-20T09:00:45.516 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-03-20T09:00:45.516 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-03-20T09:00:45.516 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-03-20T09:00:45.516 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-03-20T09:00:45.516 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-03-20T09:00:45.517 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-03-20T09:00:45.517 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-03-20T09:00:45.517 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-03-20T09:00:45.517 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2024-03-20T09:00:45.517 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_max": 32, 2024-03-20T09:00:45.517 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2024-03-20T09:00:45.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.517 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-03-20T09:00:45.517 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": {} 2024-03-20T09:00:45.517 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.517 INFO:teuthology.orchestra.run.smithi026.stdout: "read_balance": { 2024-03-20T09:00:45.517 INFO:teuthology.orchestra.run.smithi026.stdout: "score_type": "Fair distribution", 2024-03-20T09:00:45.518 INFO:teuthology.orchestra.run.smithi026.stdout: "score_acting": 12, 2024-03-20T09:00:45.518 INFO:teuthology.orchestra.run.smithi026.stdout: "score_stable": 12, 2024-03-20T09:00:45.518 INFO:teuthology.orchestra.run.smithi026.stdout: "optimal_score": 0.25, 2024-03-20T09:00:45.518 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_acting": 3, 2024-03-20T09:00:45.518 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_stable": 3, 2024-03-20T09:00:45.519 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-03-20T09:00:45.519 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity": 1, 2024-03-20T09:00:45.519 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity_weighted": 1 2024-03-20T09:00:45.519 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.519 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.519 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.519 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 2, 2024-03-20T09:00:45.519 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "unique_pool_0", 2024-03-20T09:00:45.520 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-03-20T08:58:08.054071+0000", 2024-03-20T09:00:45.520 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-03-20T09:00:45.520 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-03-20T09:00:45.520 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-03-20T09:00:45.520 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 3, 2024-03-20T09:00:45.520 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 2, 2024-03-20T09:00:45.520 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-03-20T09:00:45.520 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-03-20T09:00:45.520 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-03-20T09:00:45.520 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-03-20T09:00:45.520 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-20T09:00:45.520 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-03-20T09:00:45.520 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-03-20T09:00:45.520 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 16, 2024-03-20T09:00:45.521 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 16, 2024-03-20T09:00:45.521 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 16, 2024-03-20T09:00:45.521 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 16, 2024-03-20T09:00:45.521 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 16, 2024-03-20T09:00:45.521 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-03-20T09:00:45.521 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-03-20T09:00:45.521 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-03-20T09:00:45.521 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-03-20T09:00:45.521 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-03-20T09:00:45.521 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-03-20T09:00:45.521 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-03-20T09:00:45.521 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.521 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "125", 2024-03-20T09:00:45.521 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-03-20T09:00:45.522 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-20T09:00:45.522 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2024-03-20T09:00:45.522 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-03-20T09:00:45.522 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-03-20T09:00:45.522 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-03-20T09:00:45.522 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-03-20T09:00:45.522 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-03-20T09:00:45.522 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-03-20T09:00:45.522 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-03-20T09:00:45.522 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-03-20T09:00:45.522 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-03-20T09:00:45.522 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-03-20T09:00:45.522 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-03-20T09:00:45.522 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-03-20T09:00:45.522 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-03-20T09:00:45.523 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-03-20T09:00:45.523 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-03-20T09:00:45.523 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-20T09:00:45.523 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-20T09:00:45.523 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-20T09:00:45.523 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-03-20T09:00:45.523 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-03-20T09:00:45.523 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-03-20T09:00:45.523 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-03-20T09:00:45.523 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-03-20T09:00:45.523 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.523 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-03-20T09:00:45.523 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-03-20T09:00:45.523 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-03-20T09:00:45.524 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-03-20T09:00:45.524 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-03-20T09:00:45.524 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-03-20T09:00:45.524 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-03-20T09:00:45.524 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-03-20T09:00:45.524 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-03-20T09:00:45.524 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-03-20T09:00:45.524 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-03-20T09:00:45.524 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2024-03-20T09:00:45.525 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-03-20T09:00:45.525 INFO:teuthology.orchestra.run.smithi026.stdout: "rados": {} 2024-03-20T09:00:45.525 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.525 INFO:teuthology.orchestra.run.smithi026.stdout: "read_balance": { 2024-03-20T09:00:45.526 INFO:teuthology.orchestra.run.smithi026.stdout: "score_type": "Fair distribution", 2024-03-20T09:00:45.526 INFO:teuthology.orchestra.run.smithi026.stdout: "score_acting": 2.25, 2024-03-20T09:00:45.526 INFO:teuthology.orchestra.run.smithi026.stdout: "score_stable": 2.25, 2024-03-20T09:00:45.526 INFO:teuthology.orchestra.run.smithi026.stdout: "optimal_score": 1, 2024-03-20T09:00:45.526 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_acting": 2.25, 2024-03-20T09:00:45.526 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_stable": 2.25, 2024-03-20T09:00:45.526 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-03-20T09:00:45.526 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity": 1, 2024-03-20T09:00:45.526 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity_weighted": 1 2024-03-20T09:00:45.527 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.527 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.527 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:00:45.527 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:00:45.527 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.527 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-03-20T09:00:45.527 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "ad3fd9c0-316f-4fe5-a63b-d480087c1729", 2024-03-20T09:00:45.527 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:00:45.527 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:00:45.527 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:00:45.528 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:00:45.528 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:00:45.528 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:00:45.528 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 8, 2024-03-20T09:00:45.528 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 62, 2024-03-20T09:00:45.528 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:00:45.528 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:00:45.528 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:00:45.528 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.528 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.528 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.528 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2024-03-20T09:00:45.528 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:00:45.529 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.529 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.530 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.530 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2024-03-20T09:00:45.530 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:00:45.530 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.530 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.530 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.530 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:00:45.530 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.530 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.530 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.530 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2024-03-20T09:00:45.531 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:00:45.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.531 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.531 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2024-03-20T09:00:45.531 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:00:45.531 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.531 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.531 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.531 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:00:45.531 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.531 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.531 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.532 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2024-03-20T09:00:45.532 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:00:45.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.532 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.532 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2024-03-20T09:00:45.532 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:00:45.532 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.532 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.532 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.532 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:00:45.532 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.532 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.533 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.533 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2024-03-20T09:00:45.533 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:00:45.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.533 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.533 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.533 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2024-03-20T09:00:45.533 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:00:45.533 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.533 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.533 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.533 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/2139805293", 2024-03-20T09:00:45.534 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/2139805293", 2024-03-20T09:00:45.534 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/2139805293", 2024-03-20T09:00:45.534 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/2139805293", 2024-03-20T09:00:45.534 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:00:45.534 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:00:45.534 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:00:45.534 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.534 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.534 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.534 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-03-20T09:00:45.534 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e85b94f5-2954-47a1-9c20-096f2606eb04", 2024-03-20T09:00:45.534 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:00:45.534 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:00:45.535 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:00:45.535 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:00:45.536 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:00:45.536 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:00:45.536 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2024-03-20T09:00:45.537 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 121, 2024-03-20T09:00:45.537 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:00:45.537 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:00:45.537 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:00:45.537 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.537 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.538 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.538 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2024-03-20T09:00:45.538 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:00:45.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.538 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.538 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.538 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2024-03-20T09:00:45.538 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:00:45.538 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.538 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.538 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.539 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:00:45.539 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.539 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.539 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2024-03-20T09:00:45.539 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:00:45.539 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.539 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.539 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.539 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2024-03-20T09:00:45.539 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:00:45.539 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.540 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.540 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.540 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:00:45.540 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.540 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.540 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.540 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2024-03-20T09:00:45.541 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:00:45.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.541 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.541 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2024-03-20T09:00:45.541 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:00:45.541 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.541 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.541 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.541 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:00:45.541 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.541 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.541 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.542 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2024-03-20T09:00:45.542 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:00:45.542 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.542 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.542 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.542 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2024-03-20T09:00:45.543 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:00:45.543 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.543 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.543 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.543 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/3853355134", 2024-03-20T09:00:45.543 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/3853355134", 2024-03-20T09:00:45.543 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/3853355134", 2024-03-20T09:00:45.543 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/3853355134", 2024-03-20T09:00:45.543 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:00:45.544 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:00:45.544 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:00:45.544 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.544 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.544 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.545 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-03-20T09:00:45.545 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "dd2b10f0-741c-47a4-b224-ac6bfe2c14ef", 2024-03-20T09:00:45.545 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:00:45.545 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:00:45.545 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:00:45.545 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:00:45.545 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:00:45.545 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:00:45.545 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 18, 2024-03-20T09:00:45.545 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 121, 2024-03-20T09:00:45.545 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:00:45.545 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:00:45.545 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:00:45.546 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.546 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.546 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2024-03-20T09:00:45.546 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:00:45.546 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.546 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.546 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.546 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2024-03-20T09:00:45.546 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:00:45.546 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.547 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.547 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:00:45.547 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.547 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.547 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2024-03-20T09:00:45.547 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:00:45.547 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.547 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.547 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.547 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2024-03-20T09:00:45.547 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:00:45.548 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.548 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.548 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:00:45.548 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.548 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.548 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2024-03-20T09:00:45.548 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:00:45.548 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.548 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.549 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.549 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2024-03-20T09:00:45.549 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:00:45.549 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.549 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.549 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.549 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:00:45.549 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.549 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.549 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.549 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2024-03-20T09:00:45.549 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:00:45.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.550 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.550 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.550 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2024-03-20T09:00:45.550 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:00:45.550 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.550 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.550 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.550 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/3993357967", 2024-03-20T09:00:45.550 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/3993357967", 2024-03-20T09:00:45.550 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/3993357967", 2024-03-20T09:00:45.551 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/3993357967", 2024-03-20T09:00:45.551 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:00:45.551 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:00:45.551 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:00:45.551 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.551 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.551 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.551 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-03-20T09:00:45.551 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1", 2024-03-20T09:00:45.551 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:00:45.551 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:00:45.551 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:00:45.552 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:00:45.552 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:00:45.552 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:00:45.552 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 26, 2024-03-20T09:00:45.552 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 0, 2024-03-20T09:00:45.552 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:00:45.552 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:00:45.552 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:00:45.552 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.553 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.553 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2024-03-20T09:00:45.553 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:00:45.553 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.553 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.553 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.554 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2024-03-20T09:00:45.554 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:00:45.554 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.554 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.554 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.554 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:00:45.555 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.555 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.555 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2024-03-20T09:00:45.555 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:00:45.555 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.555 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.556 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.556 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2024-03-20T09:00:45.556 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:00:45.556 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.556 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.556 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.556 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:00:45.556 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.556 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.557 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.557 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2024-03-20T09:00:45.557 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:00:45.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.557 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.557 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.557 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2024-03-20T09:00:45.557 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:00:45.557 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.557 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.557 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.557 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:00:45.558 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.558 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.558 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2024-03-20T09:00:45.558 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:00:45.558 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.558 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.559 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.559 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2024-03-20T09:00:45.559 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:00:45.559 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.559 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.559 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.559 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/4110225509", 2024-03-20T09:00:45.559 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/4110225509", 2024-03-20T09:00:45.559 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/4110225509", 2024-03-20T09:00:45.559 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/4110225509", 2024-03-20T09:00:45.559 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:00:45.560 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:00:45.560 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:00:45.560 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.560 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-03-20T09:00:45.560 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "f14e34d5-9829-42a3-9b25-31a0976c5ef7", 2024-03-20T09:00:45.560 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:00:45.560 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:00:45.560 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:00:45.560 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:00:45.560 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:00:45.561 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:00:45.561 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 32, 2024-03-20T09:00:45.561 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 0, 2024-03-20T09:00:45.561 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:00:45.561 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:00:45.561 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:00:45.561 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.561 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.561 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6800", 2024-03-20T09:00:45.561 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:00:45.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.562 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.562 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6801", 2024-03-20T09:00:45.562 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:00:45.562 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.562 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.562 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:00:45.562 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.562 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.562 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6802", 2024-03-20T09:00:45.563 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:00:45.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.564 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.564 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6803", 2024-03-20T09:00:45.564 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:00:45.564 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.564 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.564 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:00:45.564 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.564 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.564 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6806", 2024-03-20T09:00:45.565 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:00:45.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.565 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.565 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6807", 2024-03-20T09:00:45.566 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:00:45.566 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.566 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.566 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:00:45.566 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.567 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.567 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6804", 2024-03-20T09:00:45.567 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:00:45.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.567 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.567 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6805", 2024-03-20T09:00:45.567 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:00:45.567 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.567 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.567 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6801/477923058", 2024-03-20T09:00:45.568 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6803/477923058", 2024-03-20T09:00:45.568 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6807/477923058", 2024-03-20T09:00:45.568 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6805/477923058", 2024-03-20T09:00:45.568 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:00:45.568 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:00:45.568 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:00:45.568 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.568 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-03-20T09:00:45.568 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "cb431b6a-540d-4318-ae35-d379e1df0185", 2024-03-20T09:00:45.568 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:00:45.569 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:00:45.569 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:00:45.569 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:00:45.569 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:00:45.569 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:00:45.569 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 41, 2024-03-20T09:00:45.569 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 121, 2024-03-20T09:00:45.569 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:00:45.569 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:00:45.569 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:00:45.569 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.570 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.570 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6808", 2024-03-20T09:00:45.570 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:00:45.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.570 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.570 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6809", 2024-03-20T09:00:45.570 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:00:45.570 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.570 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.571 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:00:45.571 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.571 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.571 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6810", 2024-03-20T09:00:45.571 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:00:45.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.571 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.571 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6811", 2024-03-20T09:00:45.571 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:00:45.571 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.572 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.572 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:00:45.572 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.572 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.572 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6814", 2024-03-20T09:00:45.572 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:00:45.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.572 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.572 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6815", 2024-03-20T09:00:45.573 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:00:45.573 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.573 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.573 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:00:45.573 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.573 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.573 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6812", 2024-03-20T09:00:45.573 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:00:45.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.574 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.574 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6813", 2024-03-20T09:00:45.574 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:00:45.574 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.574 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.574 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6809/1339268003", 2024-03-20T09:00:45.574 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6811/1339268003", 2024-03-20T09:00:45.575 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6815/1339268003", 2024-03-20T09:00:45.575 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6813/1339268003", 2024-03-20T09:00:45.575 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:00:45.575 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:00:45.575 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:00:45.575 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.576 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2024-03-20T09:00:45.576 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "753ed830-e764-4ff3-9d9b-87e7fa748b81", 2024-03-20T09:00:45.576 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:00:45.576 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:00:45.576 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:00:45.576 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:00:45.576 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:00:45.576 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:00:45.576 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 51, 2024-03-20T09:00:45.576 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 121, 2024-03-20T09:00:45.576 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:00:45.577 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:00:45.577 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:00:45.577 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.577 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.577 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6816", 2024-03-20T09:00:45.577 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:00:45.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.577 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.577 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6817", 2024-03-20T09:00:45.577 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:00:45.578 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.578 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.578 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:00:45.578 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.578 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.578 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6818", 2024-03-20T09:00:45.578 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:00:45.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.579 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.579 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6819", 2024-03-20T09:00:45.579 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:00:45.579 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.579 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.579 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:00:45.579 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.579 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.579 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6822", 2024-03-20T09:00:45.579 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:00:45.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.580 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.580 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6823", 2024-03-20T09:00:45.580 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:00:45.580 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.580 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.580 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:00:45.580 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.580 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.580 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6820", 2024-03-20T09:00:45.581 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:00:45.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.581 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.581 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6821", 2024-03-20T09:00:45.581 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:00:45.581 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.581 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.581 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6817/3278301843", 2024-03-20T09:00:45.581 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6819/3278301843", 2024-03-20T09:00:45.581 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6823/3278301843", 2024-03-20T09:00:45.582 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6821/3278301843", 2024-03-20T09:00:45.582 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:00:45.582 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:00:45.582 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:00:45.582 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.582 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2024-03-20T09:00:45.582 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "55bcbd37-7a6f-4ab1-b5a9-91a951444d86", 2024-03-20T09:00:45.582 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:00:45.582 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:00:45.582 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:00:45.583 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:00:45.583 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:00:45.583 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:00:45.583 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 60, 2024-03-20T09:00:45.583 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 121, 2024-03-20T09:00:45.583 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:00:45.583 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:00:45.583 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:00:45.583 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.583 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.583 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6824", 2024-03-20T09:00:45.584 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:00:45.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.584 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.584 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6825", 2024-03-20T09:00:45.584 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:00:45.585 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.585 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.585 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:00:45.585 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.585 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.585 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6826", 2024-03-20T09:00:45.586 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:00:45.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.586 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.586 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6827", 2024-03-20T09:00:45.586 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:00:45.586 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.587 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.587 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:00:45.587 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.587 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.587 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6830", 2024-03-20T09:00:45.587 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:00:45.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.587 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.587 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6831", 2024-03-20T09:00:45.588 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:00:45.588 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.588 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.588 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:00:45.588 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.588 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.588 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6828", 2024-03-20T09:00:45.588 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:00:45.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.589 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.589 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6829", 2024-03-20T09:00:45.589 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:00:45.589 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.589 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.589 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6825/3152424001", 2024-03-20T09:00:45.589 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6827/3152424001", 2024-03-20T09:00:45.589 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6831/3152424001", 2024-03-20T09:00:45.589 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6829/3152424001", 2024-03-20T09:00:45.589 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:00:45.589 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:00:45.589 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:00:45.590 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.590 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 8, 2024-03-20T09:00:45.590 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "bc86248e-6bd0-4369-b8a2-e593bc2a610d", 2024-03-20T09:00:45.590 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:00:45.590 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:00:45.590 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:00:45.590 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:00:45.590 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:00:45.590 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:00:45.590 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 70, 2024-03-20T09:00:45.591 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 121, 2024-03-20T09:00:45.591 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:00:45.591 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:00:45.591 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:00:45.591 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.591 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.591 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6800", 2024-03-20T09:00:45.591 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:00:45.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.591 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.592 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6801", 2024-03-20T09:00:45.592 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:00:45.592 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.592 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.592 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:00:45.592 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.592 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.593 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6802", 2024-03-20T09:00:45.593 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:00:45.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.593 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.593 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6803", 2024-03-20T09:00:45.593 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:00:45.593 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.593 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.593 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:00:45.593 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.594 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.594 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6806", 2024-03-20T09:00:45.594 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:00:45.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.594 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.595 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6807", 2024-03-20T09:00:45.595 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:00:45.595 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.595 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.595 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:00:45.595 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.595 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.595 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6804", 2024-03-20T09:00:45.595 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:00:45.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.596 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.596 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6805", 2024-03-20T09:00:45.596 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:00:45.596 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.596 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.597 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6801/1309426740", 2024-03-20T09:00:45.597 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6803/1309426740", 2024-03-20T09:00:45.597 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6807/1309426740", 2024-03-20T09:00:45.597 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6805/1309426740", 2024-03-20T09:00:45.597 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:00:45.597 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:00:45.598 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:00:45.598 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.598 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 9, 2024-03-20T09:00:45.598 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b20a4aa-032a-4c09-ac75-153c554424e7", 2024-03-20T09:00:45.598 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:00:45.598 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:00:45.598 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:00:45.598 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:00:45.599 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:00:45.599 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:00:45.599 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 80, 2024-03-20T09:00:45.599 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 121, 2024-03-20T09:00:45.599 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:00:45.599 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:00:45.599 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:00:45.599 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.599 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.599 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6808", 2024-03-20T09:00:45.600 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:00:45.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.600 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.600 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6809", 2024-03-20T09:00:45.600 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:00:45.600 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.600 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.600 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:00:45.600 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.600 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.601 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6810", 2024-03-20T09:00:45.601 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:00:45.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.601 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.601 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6811", 2024-03-20T09:00:45.601 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:00:45.601 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.601 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.602 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:00:45.602 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.602 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.602 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6814", 2024-03-20T09:00:45.602 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:00:45.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.602 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.602 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6815", 2024-03-20T09:00:45.602 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:00:45.603 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.603 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.603 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:00:45.603 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.603 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.603 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6812", 2024-03-20T09:00:45.603 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:00:45.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.604 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.604 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6813", 2024-03-20T09:00:45.604 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:00:45.604 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.604 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.604 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6809/2953210076", 2024-03-20T09:00:45.604 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6811/2953210076", 2024-03-20T09:00:45.604 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6815/2953210076", 2024-03-20T09:00:45.605 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6813/2953210076", 2024-03-20T09:00:45.605 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:00:45.605 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:00:45.605 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:00:45.605 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.605 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 10, 2024-03-20T09:00:45.605 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e62c1fb3-5dd1-4d0d-997c-d20633d19361", 2024-03-20T09:00:45.606 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:00:45.606 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:00:45.606 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:00:45.606 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:00:45.606 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:00:45.606 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:00:45.606 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 89, 2024-03-20T09:00:45.606 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 0, 2024-03-20T09:00:45.606 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:00:45.606 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:00:45.606 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:00:45.606 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.607 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.608 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6816", 2024-03-20T09:00:45.608 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:00:45.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.608 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.608 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6817", 2024-03-20T09:00:45.608 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:00:45.608 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.608 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.608 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:00:45.608 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.609 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.609 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6818", 2024-03-20T09:00:45.609 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:00:45.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.609 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.609 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6819", 2024-03-20T09:00:45.609 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:00:45.609 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.609 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.612 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:00:45.612 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.612 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.613 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6822", 2024-03-20T09:00:45.613 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:00:45.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.613 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.613 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6823", 2024-03-20T09:00:45.613 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:00:45.613 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.613 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.613 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:00:45.614 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.614 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.614 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6820", 2024-03-20T09:00:45.614 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:00:45.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.614 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.614 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6821", 2024-03-20T09:00:45.614 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:00:45.615 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.615 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.615 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6817/409394340", 2024-03-20T09:00:45.615 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6819/409394340", 2024-03-20T09:00:45.615 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6823/409394340", 2024-03-20T09:00:45.615 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6821/409394340", 2024-03-20T09:00:45.615 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:00:45.615 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:00:45.615 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:00:45.615 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.615 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 11, 2024-03-20T09:00:45.616 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b83994d5-e5ef-48b0-9758-c26843cec3b6", 2024-03-20T09:00:45.616 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:00:45.616 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:00:45.616 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:00:45.616 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:00:45.616 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:00:45.616 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:00:45.616 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 96, 2024-03-20T09:00:45.616 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 121, 2024-03-20T09:00:45.616 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:00:45.616 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:00:45.616 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:00:45.616 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.617 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.617 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6824", 2024-03-20T09:00:45.617 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:00:45.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.617 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.617 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6825", 2024-03-20T09:00:45.617 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:00:45.617 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.617 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.617 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:00:45.617 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.618 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.618 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6826", 2024-03-20T09:00:45.618 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:00:45.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.618 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.618 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6827", 2024-03-20T09:00:45.618 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:00:45.618 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.618 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.618 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:00:45.618 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.619 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.619 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6830", 2024-03-20T09:00:45.619 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:00:45.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.619 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.619 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6831", 2024-03-20T09:00:45.619 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:00:45.619 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.619 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.619 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:00:45.619 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:00:45.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.619 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:00:45.620 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6828", 2024-03-20T09:00:45.620 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:00:45.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.620 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:00:45.620 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6829", 2024-03-20T09:00:45.620 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:00:45.620 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.620 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.620 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6825/2704069354", 2024-03-20T09:00:45.620 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6827/2704069354", 2024-03-20T09:00:45.620 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6831/2704069354", 2024-03-20T09:00:45.620 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6829/2704069354", 2024-03-20T09:00:45.621 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:00:45.621 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:00:45.621 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:00:45.621 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:00:45.621 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.621 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:00:45.621 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2024-03-20T09:00:45.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.621 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-03-20T09:00:45.621 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:00:45.621 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:00:45.621 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:00:45.621 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:00:45.621 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:00:45.622 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:00:15.458602+0000", 2024-03-20T09:00:45.622 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:00:45.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.622 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-03-20T09:00:45.622 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:00:45.622 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:00:45.622 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:00:45.622 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:00:45.622 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:00:45.622 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:00:36.085528+0000", 2024-03-20T09:00:45.622 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:00:45.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.623 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-03-20T09:00:45.623 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:00:45.623 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:00:45.623 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:00:45.623 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:00:45.623 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:00:45.623 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:00:06.154878+0000", 2024-03-20T09:00:45.623 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:00:45.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.623 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-03-20T09:00:45.623 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:00:45.623 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:00:45.623 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:00:45.624 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:00:45.624 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:00:45.624 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:00:31.384708+0000", 2024-03-20T09:00:45.624 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:00:45.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.624 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-03-20T09:00:45.624 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:00:45.624 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:00:45.624 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:00:45.624 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:00:45.624 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:00:45.624 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:00:21.542194+0000", 2024-03-20T09:00:45.624 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:00:45.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.625 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-03-20T09:00:45.625 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:00:45.625 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:00:45.625 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:00:45.625 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:00:45.625 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:00:45.625 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:00:00.111136+0000", 2024-03-20T09:00:45.625 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:00:45.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.625 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2024-03-20T09:00:45.625 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:00:45.625 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:00:45.625 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:00:45.626 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:00:45.626 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:00:45.626 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:00:32.621480+0000", 2024-03-20T09:00:45.626 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:00:45.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.626 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2024-03-20T09:00:45.626 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:00:45.626 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:00:45.626 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:00:45.626 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:00:45.626 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:00:45.626 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T08:59:18.725286+0000", 2024-03-20T09:00:45.627 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:00:45.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.627 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 8, 2024-03-20T09:00:45.627 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:00:45.627 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:00:45.627 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:00:45.627 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:00:45.627 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:00:45.627 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T08:59:52.093175+0000", 2024-03-20T09:00:45.627 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:00:45.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.627 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 9, 2024-03-20T09:00:45.628 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:00:45.628 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:00:45.628 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:00:45.628 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:00:45.628 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:00:45.628 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:00:26.483886+0000", 2024-03-20T09:00:45.628 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:00:45.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.628 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 10, 2024-03-20T09:00:45.628 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:00:45.628 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:00:45.628 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:00:45.628 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:00:45.629 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:00:45.629 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T08:59:55.463512+0000", 2024-03-20T09:00:45.629 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:00:45.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:00:45.629 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 11, 2024-03-20T09:00:45.629 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:00:45.629 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:00:45.629 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:00:45.629 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:00:45.629 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:00:45.629 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T08:59:46.097314+0000", 2024-03-20T09:00:45.629 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:00:45.630 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.630 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:00:45.630 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [], 2024-03-20T09:00:45.630 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2024-03-20T09:00:45.630 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_primaries": [], 2024-03-20T09:00:45.630 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2024-03-20T09:00:45.630 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2024-03-20T09:00:45.630 INFO:teuthology.orchestra.run.smithi026.stdout: "blocklist": { 2024-03-20T09:00:45.630 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/4291404191": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:00:45.630 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/642012566": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:00:45.630 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6801/1181459999": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:00:45.630 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/3984513952": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:00:45.630 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6800/1654945498": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:00:45.630 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6800/1181459999": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:00:45.631 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/137770981": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:00:45.631 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/1451237762": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:00:45.631 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/3938386631": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:00:45.631 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6801/1654945498": "2024-03-21T08:49:04.596754+0000" 2024-03-20T09:00:45.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.631 INFO:teuthology.orchestra.run.smithi026.stdout: "range_blocklist": {}, 2024-03-20T09:00:45.631 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2024-03-20T09:00:45.631 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2024-03-20T09:00:45.631 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2024-03-20T09:00:45.631 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2024-03-20T09:00:45.631 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2024-03-20T09:00:45.631 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2024-03-20T09:00:45.631 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2024-03-20T09:00:45.631 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:00:45.632 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2024-03-20T09:00:45.632 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2024-03-20T09:00:45.632 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2024-03-20T09:00:45.632 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2024-03-20T09:00:45.632 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {}, 2024-03-20T09:00:45.632 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": { 2024-03-20T09:00:45.632 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_enabled": false, 2024-03-20T09:00:45.632 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_bucket_count": 0, 2024-03-20T09:00:45.632 INFO:teuthology.orchestra.run.smithi026.stdout: "degraded_stretch_mode": 0, 2024-03-20T09:00:45.632 INFO:teuthology.orchestra.run.smithi026.stdout: "recovering_stretch_mode": 0, 2024-03-20T09:00:45.632 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_bucket": 0 2024-03-20T09:00:45.632 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:00:45.632 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:00:45.633 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:45.682 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:00:45.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:45.685 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:00:45.785 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:00:45.818 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:45.839 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:00:45.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:45.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:00:45.943 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:00:45.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:45 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:00:45.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:45 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:00:45.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:45 smithi122 ceph-mon[73495]: osdmap e164: 12 total, 12 up, 12 in 2024-03-20T09:00:45.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:45 smithi122 ceph-mon[73495]: 2.5 scrub starts 2024-03-20T09:00:45.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:45 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1557515237' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:00:45.974 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:46.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:45 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:00:46.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:45 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:00:46.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:45 smithi115 ceph-mon[72476]: osdmap e164: 12 total, 12 up, 12 in 2024-03-20T09:00:46.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:45 smithi115 ceph-mon[72476]: 2.5 scrub starts 2024-03-20T09:00:46.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:45 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1557515237' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:00:46.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:45 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:00:46.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:45 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:00:46.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:45 smithi026 ceph-mon[68309]: osdmap e164: 12 total, 12 up, 12 in 2024-03-20T09:00:46.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:45 smithi026 ceph-mon[68309]: 2.5 scrub starts 2024-03-20T09:00:46.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:45 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1557515237' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:00:46.252 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_blocked_ops 2024-03-20T09:00:46.509 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:00:46.777 INFO:tasks.thrashosds.thrasher:j is {'epoch': 164, 'fsid': '852fbb68-e696-11ee-95c9-87774f69a715', 'created': '2024-03-20T08:48:35.281088+0000', 'modified': '2024-03-20T09:00:44.794357+0000', 'last_up_change': '2024-03-20T08:56:17.873738+0000', 'last_in_change': '2024-03-20T08:55:59.486577+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-20T08:52:17.442324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '131', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'unique_pool_0', 'create_time': '2024-03-20T08:58:08.054071+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '125', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'ad3fd9c0-316f-4fe5-a63b-d480087c1729', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 62, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6802', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6803', 'nonce': 2139805293}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6804', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6805', 'nonce': 2139805293}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6808', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6809', 'nonce': 2139805293}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6806', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6807', 'nonce': 2139805293}]}, 'public_addr': '172.21.15.26:6803/2139805293', 'cluster_addr': '172.21.15.26:6805/2139805293', 'heartbeat_back_addr': '172.21.15.26:6809/2139805293', 'heartbeat_front_addr': '172.21.15.26:6807/2139805293', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'e85b94f5-2954-47a1-9c20-096f2606eb04', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 121, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6810', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6811', 'nonce': 3853355134}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6812', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6813', 'nonce': 3853355134}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6816', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6817', 'nonce': 3853355134}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6814', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6815', 'nonce': 3853355134}]}, 'public_addr': '172.21.15.26:6811/3853355134', 'cluster_addr': '172.21.15.26:6813/3853355134', 'heartbeat_back_addr': '172.21.15.26:6817/3853355134', 'heartbeat_front_addr': '172.21.15.26:6815/3853355134', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'dd2b10f0-741c-47a4-b224-ac6bfe2c14ef', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 121, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6818', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6819', 'nonce': 3993357967}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6820', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6821', 'nonce': 3993357967}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6824', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6825', 'nonce': 3993357967}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6822', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6823', 'nonce': 3993357967}]}, 'public_addr': '172.21.15.26:6819/3993357967', 'cluster_addr': '172.21.15.26:6821/3993357967', 'heartbeat_back_addr': '172.21.15.26:6825/3993357967', 'heartbeat_front_addr': '172.21.15.26:6823/3993357967', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6826', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6827', 'nonce': 4110225509}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6828', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6829', 'nonce': 4110225509}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6832', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6833', 'nonce': 4110225509}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6830', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6831', 'nonce': 4110225509}]}, 'public_addr': '172.21.15.26:6827/4110225509', 'cluster_addr': '172.21.15.26:6829/4110225509', 'heartbeat_back_addr': '172.21.15.26:6833/4110225509', 'heartbeat_front_addr': '172.21.15.26:6831/4110225509', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'f14e34d5-9829-42a3-9b25-31a0976c5ef7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6800', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6801', 'nonce': 477923058}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6802', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6803', 'nonce': 477923058}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6806', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6807', 'nonce': 477923058}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6804', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6805', 'nonce': 477923058}]}, 'public_addr': '172.21.15.115:6801/477923058', 'cluster_addr': '172.21.15.115:6803/477923058', 'heartbeat_back_addr': '172.21.15.115:6807/477923058', 'heartbeat_front_addr': '172.21.15.115:6805/477923058', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'cb431b6a-540d-4318-ae35-d379e1df0185', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 121, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6808', 'nonce': 1339268003}, {'type': 'v1', 'addr': '172.21.15.115:6809', 'nonce': 1339268003}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6810', 'nonce': 1339268003}, {'type': 'v1', 'addr': '172.21.15.115:6811', 'nonce': 1339268003}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6814', 'nonce': 1339268003}, {'type': 'v1', 'addr': '172.21.15.115:6815', 'nonce': 1339268003}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6812', 'nonce': 1339268003}, {'type': 'v1', 'addr': '172.21.15.115:6813', 'nonce': 1339268003}]}, 'public_addr': '172.21.15.115:6809/1339268003', 'cluster_addr': '172.21.15.115:6811/1339268003', 'heartbeat_back_addr': '172.21.15.115:6815/1339268003', 'heartbeat_front_addr': '172.21.15.115:6813/1339268003', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '753ed830-e764-4ff3-9d9b-87e7fa748b81', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 121, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6816', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6817', 'nonce': 3278301843}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6818', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6819', 'nonce': 3278301843}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6822', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6823', 'nonce': 3278301843}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6820', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6821', 'nonce': 3278301843}]}, 'public_addr': '172.21.15.115:6817/3278301843', 'cluster_addr': '172.21.15.115:6819/3278301843', 'heartbeat_back_addr': '172.21.15.115:6823/3278301843', 'heartbeat_front_addr': '172.21.15.115:6821/3278301843', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '55bcbd37-7a6f-4ab1-b5a9-91a951444d86', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 121, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6824', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6825', 'nonce': 3152424001}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6826', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6827', 'nonce': 3152424001}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6830', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6831', 'nonce': 3152424001}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6828', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6829', 'nonce': 3152424001}]}, 'public_addr': '172.21.15.115:6825/3152424001', 'cluster_addr': '172.21.15.115:6827/3152424001', 'heartbeat_back_addr': '172.21.15.115:6831/3152424001', 'heartbeat_front_addr': '172.21.15.115:6829/3152424001', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'bc86248e-6bd0-4369-b8a2-e593bc2a610d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 70, 'up_thru': 121, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6800', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6801', 'nonce': 1309426740}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6802', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6803', 'nonce': 1309426740}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6806', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6807', 'nonce': 1309426740}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6804', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6805', 'nonce': 1309426740}]}, 'public_addr': '172.21.15.122:6801/1309426740', 'cluster_addr': '172.21.15.122:6803/1309426740', 'heartbeat_back_addr': '172.21.15.122:6807/1309426740', 'heartbeat_front_addr': '172.21.15.122:6805/1309426740', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '9b20a4aa-032a-4c09-ac75-153c554424e7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 121, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6808', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6809', 'nonce': 2953210076}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6810', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6811', 'nonce': 2953210076}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6814', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6815', 'nonce': 2953210076}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6812', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6813', 'nonce': 2953210076}]}, 'public_addr': '172.21.15.122:6809/2953210076', 'cluster_addr': '172.21.15.122:6811/2953210076', 'heartbeat_back_addr': '172.21.15.122:6815/2953210076', 'heartbeat_front_addr': '172.21.15.122:6813/2953210076', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'e62c1fb3-5dd1-4d0d-997c-d20633d19361', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 89, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6816', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6817', 'nonce': 409394340}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6818', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6819', 'nonce': 409394340}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6822', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6823', 'nonce': 409394340}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6820', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6821', 'nonce': 409394340}]}, 'public_addr': '172.21.15.122:6817/409394340', 'cluster_addr': '172.21.15.122:6819/409394340', 'heartbeat_back_addr': '172.21.15.122:6823/409394340', 'heartbeat_front_addr': '172.21.15.122:6821/409394340', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'b83994d5-e5ef-48b0-9758-c26843cec3b6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 96, 'up_thru': 121, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6824', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6825', 'nonce': 2704069354}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6826', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6827', 'nonce': 2704069354}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6830', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6831', 'nonce': 2704069354}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6828', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6829', 'nonce': 2704069354}]}, 'public_addr': '172.21.15.122:6825/2704069354', 'cluster_addr': '172.21.15.122:6827/2704069354', 'heartbeat_back_addr': '172.21.15.122:6831/2704069354', 'heartbeat_front_addr': '172.21.15.122:6829/2704069354', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:00:15.458602+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:00:36.085528+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:00:06.154878+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:00:31.384708+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:00:21.542194+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:00:00.111136+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:00:32.621480+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T08:59:18.725286+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T08:59:52.093175+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:00:26.483886+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T08:59:55.463512+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T08:59:46.097314+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.26:0/4291404191': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/642012566': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:6801/1181459999': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/3984513952': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6800/1654945498': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6800/1181459999': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/137770981': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:0/1451237762': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/3938386631': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6801/1654945498': '2024-03-21T08:49:04.596754+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-20T09:00:46.777 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:00:47.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:46 smithi115 ceph-mon[72476]: pgmap v489: 17 pgs: 17 active+clean; 4.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 372 op/s 2024-03-20T09:00:47.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:46 smithi115 ceph-mon[72476]: 2.5 scrub ok 2024-03-20T09:00:47.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:46 smithi122 ceph-mon[73495]: pgmap v489: 17 pgs: 17 active+clean; 4.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 372 op/s 2024-03-20T09:00:47.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:46 smithi122 ceph-mon[73495]: 2.5 scrub ok 2024-03-20T09:00:47.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:46 smithi026 ceph-mon[68309]: pgmap v489: 17 pgs: 17 active+clean; 4.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 372 op/s 2024-03-20T09:00:47.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:46 smithi026 ceph-mon[68309]: 2.5 scrub ok 2024-03-20T09:00:47.894 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:00:47.924 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:00:47.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:47.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:00:48.031 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:00:48.059 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:48.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:47 smithi115 ceph-mon[72476]: 2.1 scrub starts 2024-03-20T09:00:48.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:47 smithi122 ceph-mon[73495]: 2.1 scrub starts 2024-03-20T09:00:48.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:47 smithi026 ceph-mon[68309]: 2.1 scrub starts 2024-03-20T09:00:48.543 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:00:48.543 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:00:48.543 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-03-20T09:00:48.543 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-03-20T09:00:48.544 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:00:48.945 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:48 smithi122 ceph-mon[73495]: 2.1 scrub ok 2024-03-20T09:00:48.945 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:48 smithi122 ceph-mon[73495]: pgmap v490: 17 pgs: 17 active+clean; 4.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 234 op/s 2024-03-20T09:00:48.945 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:48 smithi122 ceph-mon[73495]: 2.f scrub starts 2024-03-20T09:00:48.945 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:48 smithi122 ceph-mon[73495]: 2.9 scrub starts 2024-03-20T09:00:49.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:48 smithi115 ceph-mon[72476]: 2.1 scrub ok 2024-03-20T09:00:49.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:48 smithi115 ceph-mon[72476]: pgmap v490: 17 pgs: 17 active+clean; 4.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 234 op/s 2024-03-20T09:00:49.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:48 smithi115 ceph-mon[72476]: 2.f scrub starts 2024-03-20T09:00:49.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:48 smithi115 ceph-mon[72476]: 2.9 scrub starts 2024-03-20T09:00:49.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:48 smithi026 ceph-mon[68309]: 2.1 scrub ok 2024-03-20T09:00:49.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:48 smithi026 ceph-mon[68309]: pgmap v490: 17 pgs: 17 active+clean; 4.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 234 op/s 2024-03-20T09:00:49.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:48 smithi026 ceph-mon[68309]: 2.f scrub starts 2024-03-20T09:00:49.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:48 smithi026 ceph-mon[68309]: 2.9 scrub starts 2024-03-20T09:00:49.454 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_historic_ops 2024-03-20T09:00:49.826 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:00:49.917 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:00:49.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:49.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:00:49.931 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:00:50.030 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:00:50.061 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:50.095 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:00:50.098 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:49 smithi115 ceph-mon[72476]: 2.0 scrub starts 2024-03-20T09:00:50.098 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:49 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:00:50.098 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:49 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:00:50.098 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:49 smithi115 ceph-mon[72476]: 2.f scrub ok 2024-03-20T09:00:50.099 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:49 smithi115 ceph-mon[72476]: 2.9 scrub ok 2024-03-20T09:00:50.099 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:49 smithi115 ceph-mon[72476]: osdmap e165: 12 total, 12 up, 12 in 2024-03-20T09:00:50.099 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:49 smithi115 ceph-mon[72476]: 2.3 scrub starts 2024-03-20T09:00:50.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:50.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:00:50.154 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:49 smithi122 ceph-mon[73495]: 2.0 scrub starts 2024-03-20T09:00:50.154 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:49 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:00:50.154 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:49 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:00:50.154 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:49 smithi122 ceph-mon[73495]: 2.f scrub ok 2024-03-20T09:00:50.154 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:49 smithi122 ceph-mon[73495]: 2.9 scrub ok 2024-03-20T09:00:50.154 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:49 smithi122 ceph-mon[73495]: osdmap e165: 12 total, 12 up, 12 in 2024-03-20T09:00:50.154 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:49 smithi122 ceph-mon[73495]: 2.3 scrub starts 2024-03-20T09:00:50.201 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:00:50.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:49 smithi026 ceph-mon[68309]: 2.0 scrub starts 2024-03-20T09:00:50.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:49 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:00:50.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:49 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:00:50.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:49 smithi026 ceph-mon[68309]: 2.f scrub ok 2024-03-20T09:00:50.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:49 smithi026 ceph-mon[68309]: 2.9 scrub ok 2024-03-20T09:00:50.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:49 smithi026 ceph-mon[68309]: osdmap e165: 12 total, 12 up, 12 in 2024-03-20T09:00:50.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:49 smithi026 ceph-mon[68309]: 2.3 scrub starts 2024-03-20T09:00:50.289 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:50.543 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:00:50.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:50.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:00:50.646 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:00:50.744 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:50.845 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:50 smithi026 ceph-mon[68309]: 2.0 scrub ok 2024-03-20T09:00:50.845 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:50 smithi026 ceph-mon[68309]: pgmap v492: 17 pgs: 17 active+clean; 4.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 76 op/s 2024-03-20T09:00:50.846 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:50 smithi026 ceph-mon[68309]: 2.3 scrub ok 2024-03-20T09:00:50.846 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:50 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:00:50.846 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:50 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:00:50.846 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:50 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:00:50.846 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:50 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:00:50.846 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:50 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:00:51.025 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:00:51.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:51.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:00:51.041 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:00:51.128 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:00:51.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:50 smithi115 ceph-mon[72476]: 2.0 scrub ok 2024-03-20T09:00:51.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:50 smithi115 ceph-mon[72476]: pgmap v492: 17 pgs: 17 active+clean; 4.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 76 op/s 2024-03-20T09:00:51.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:50 smithi115 ceph-mon[72476]: 2.3 scrub ok 2024-03-20T09:00:51.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:50 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:00:51.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:50 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:00:51.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:50 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:00:51.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:50 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:00:51.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:50 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:00:51.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:50 smithi122 ceph-mon[73495]: 2.0 scrub ok 2024-03-20T09:00:51.199 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:50 smithi122 ceph-mon[73495]: pgmap v492: 17 pgs: 17 active+clean; 4.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 76 op/s 2024-03-20T09:00:51.199 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:50 smithi122 ceph-mon[73495]: 2.3 scrub ok 2024-03-20T09:00:51.199 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:50 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:00:51.199 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:50 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:00:51.199 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:50 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:00:51.199 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:50 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:00:51.199 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:50 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:00:51.200 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:51.323 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:00:51.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:51.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:00:51.428 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:00:51.477 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:51.978 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:00:52.057 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:00:52.097 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:00:52.150 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:00:52.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:51 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:00:52.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:51 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:00:52.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:51 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:00:52.192 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:00:52.201 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:51 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:00:52.201 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:51 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:00:52.201 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:51 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:00:52.214 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:00:52.224 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:00:52.224 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-03-20T09:00:52.224 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-03-20T09:00:52.224 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-03-20T09:00:52.224 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.224 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:115843 2.c 2:3f68602e:::benchmark_data_smithi122_107654_object115842:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-03-20T09:00:52.225 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:59:50.515708+0000", 2024-03-20T09:00:52.225 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 61.704616565999999, 2024-03-20T09:00:52.225 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.13167431900000001, 2024-03-20T09:00:52.225 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:52.225 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:52.225 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:52.225 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:52.225 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:52.225 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 115843 2024-03-20T09:00:52.225 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.225 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:52.225 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.225 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:52.226 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.515708+0000", 2024-03-20T09:00:52.226 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.226 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.226 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.226 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:52.226 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.515708+0000", 2024-03-20T09:00:52.226 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.226 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.226 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.226 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:52.226 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.515710+0000", 2024-03-20T09:00:52.226 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4959999999999999e-06 2024-03-20T09:00:52.226 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.227 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.227 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:52.227 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.515731+0000", 2024-03-20T09:00:52.227 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1185000000000001e-05 2024-03-20T09:00:52.227 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.227 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.227 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:52.227 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.515744+0000", 2024-03-20T09:00:52.227 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.311e-05 2024-03-20T09:00:52.227 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.227 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.227 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:52.227 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.515753+0000", 2024-03-20T09:00:52.228 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3850000000000002e-06 2024-03-20T09:00:52.228 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.228 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.228 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:52.228 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.560294+0000", 2024-03-20T09:00:52.228 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.044541973999999998 2024-03-20T09:00:52.228 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.228 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.228 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:52.228 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.560410+0000", 2024-03-20T09:00:52.228 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000115351 2024-03-20T09:00:52.228 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.228 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.228 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,1", 2024-03-20T09:00:52.229 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.560567+0000", 2024-03-20T09:00:52.229 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015703700000000001 2024-03-20T09:00:52.229 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.229 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.229 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:52.229 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.567381+0000", 2024-03-20T09:00:52.229 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.006813759 2024-03-20T09:00:52.229 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.229 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.229 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.229 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.631954+0000", 2024-03-20T09:00:52.229 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.064573832999999997 2024-03-20T09:00:52.229 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.230 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.230 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.230 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.647294+0000", 2024-03-20T09:00:52.230 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015339159 2024-03-20T09:00:52.230 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.230 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.230 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:52.230 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.647340+0000", 2024-03-20T09:00:52.230 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6183000000000002e-05 2024-03-20T09:00:52.230 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.230 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.230 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:52.230 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.647383+0000", 2024-03-20T09:00:52.230 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2846999999999998e-05 2024-03-20T09:00:52.231 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.231 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:52.231 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.231 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.231 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.231 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:115947 2.c 2:346cf7fc:::benchmark_data_smithi122_107654_object115946:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-03-20T09:00:52.231 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:59:50.671148+0000", 2024-03-20T09:00:52.231 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 61.549176834999997, 2024-03-20T09:00:52.231 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.128749166, 2024-03-20T09:00:52.231 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:52.231 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:52.231 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:52.231 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:52.231 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:52.232 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 115947 2024-03-20T09:00:52.232 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.232 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:52.232 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.232 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:52.232 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.671148+0000", 2024-03-20T09:00:52.232 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.232 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.232 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.232 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:52.232 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.671148+0000", 2024-03-20T09:00:52.232 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.232 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.232 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.233 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:52.233 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.671150+0000", 2024-03-20T09:00:52.233 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.703e-06 2024-03-20T09:00:52.233 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.233 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.233 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:52.233 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.671172+0000", 2024-03-20T09:00:52.233 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2717999999999998e-05 2024-03-20T09:00:52.233 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.233 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.233 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:52.233 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.671187+0000", 2024-03-20T09:00:52.233 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4749999999999999e-05 2024-03-20T09:00:52.233 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.234 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.234 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:52.234 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.671198+0000", 2024-03-20T09:00:52.234 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0550000000000001e-05 2024-03-20T09:00:52.234 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.234 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.234 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:52.234 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.738775+0000", 2024-03-20T09:00:52.234 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.067577172000000005 2024-03-20T09:00:52.234 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.234 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.234 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:52.234 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.738869+0000", 2024-03-20T09:00:52.235 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3741999999999998e-05 2024-03-20T09:00:52.235 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.235 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.235 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,1", 2024-03-20T09:00:52.235 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.738982+0000", 2024-03-20T09:00:52.235 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000113268 2024-03-20T09:00:52.235 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.235 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.235 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:52.235 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.750866+0000", 2024-03-20T09:00:52.235 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011883569 2024-03-20T09:00:52.235 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.235 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.235 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.235 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.787430+0000", 2024-03-20T09:00:52.236 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.036564450999999998 2024-03-20T09:00:52.236 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.236 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.236 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.236 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.799814+0000", 2024-03-20T09:00:52.236 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012383674000000001 2024-03-20T09:00:52.236 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.236 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.236 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:52.236 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.799855+0000", 2024-03-20T09:00:52.236 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1522e-05 2024-03-20T09:00:52.236 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.236 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.236 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:52.237 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.799897+0000", 2024-03-20T09:00:52.237 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2046999999999999e-05 2024-03-20T09:00:52.237 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.237 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:52.237 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.237 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.237 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.237 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:115952 2.c 2:39c2ddfa:::benchmark_data_smithi122_107654_object115951:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-03-20T09:00:52.237 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:59:50.677374+0000", 2024-03-20T09:00:52.237 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 61.542950941999997, 2024-03-20T09:00:52.237 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.12579528600000001, 2024-03-20T09:00:52.237 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:52.237 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:52.237 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:52.238 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:52.238 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:52.238 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 115952 2024-03-20T09:00:52.238 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.238 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:52.238 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.238 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:52.238 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.677374+0000", 2024-03-20T09:00:52.238 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.238 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.238 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.238 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:52.238 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.677374+0000", 2024-03-20T09:00:52.239 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.239 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.239 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.239 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:52.239 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.677375+0000", 2024-03-20T09:00:52.239 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.511e-06 2024-03-20T09:00:52.239 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.239 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.239 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:52.239 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.677398+0000", 2024-03-20T09:00:52.239 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.249e-05 2024-03-20T09:00:52.239 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.239 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.239 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:52.240 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.677412+0000", 2024-03-20T09:00:52.240 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3614e-05 2024-03-20T09:00:52.240 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.240 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.240 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:52.240 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.677420+0000", 2024-03-20T09:00:52.240 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2859999999999999e-06 2024-03-20T09:00:52.240 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.240 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.240 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:52.240 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.742479+0000", 2024-03-20T09:00:52.240 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.065059078000000006 2024-03-20T09:00:52.240 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.240 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.241 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:52.241 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.742612+0000", 2024-03-20T09:00:52.241 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013335300000000001 2024-03-20T09:00:52.241 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.241 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.241 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,1", 2024-03-20T09:00:52.241 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.742765+0000", 2024-03-20T09:00:52.241 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015314899999999999 2024-03-20T09:00:52.241 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.241 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.241 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:52.241 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.758153+0000", 2024-03-20T09:00:52.241 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015387875000000001 2024-03-20T09:00:52.241 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.242 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.242 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.242 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.796958+0000", 2024-03-20T09:00:52.242 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.038805124000000003 2024-03-20T09:00:52.242 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.242 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.242 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.242 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.803075+0000", 2024-03-20T09:00:52.242 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0061162380000000004 2024-03-20T09:00:52.242 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.242 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.242 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:52.242 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.803110+0000", 2024-03-20T09:00:52.242 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5670000000000002e-05 2024-03-20T09:00:52.243 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.243 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.243 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:52.243 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.803169+0000", 2024-03-20T09:00:52.243 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8897999999999998e-05 2024-03-20T09:00:52.243 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.243 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:52.243 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.243 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.243 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.243 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:116213 2.c 2:3f17461b:::benchmark_data_smithi122_107654_object116212:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-03-20T09:00:52.243 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:59:51.205319+0000", 2024-03-20T09:00:52.243 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 61.015005485000003, 2024-03-20T09:00:52.243 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19140834600000001, 2024-03-20T09:00:52.244 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:52.244 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:52.244 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:52.244 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:52.244 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:52.244 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 116213 2024-03-20T09:00:52.244 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.244 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:52.244 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.244 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:52.244 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.205319+0000", 2024-03-20T09:00:52.244 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.244 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.244 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.245 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:52.245 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.205319+0000", 2024-03-20T09:00:52.245 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.245 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.245 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.245 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:52.245 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.205321+0000", 2024-03-20T09:00:52.245 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8390000000000001e-06 2024-03-20T09:00:52.245 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.245 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.245 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:52.245 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.205342+0000", 2024-03-20T09:00:52.245 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0903e-05 2024-03-20T09:00:52.245 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.246 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.246 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:52.246 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.205356+0000", 2024-03-20T09:00:52.246 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3617e-05 2024-03-20T09:00:52.246 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.246 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.246 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:52.246 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.205364+0000", 2024-03-20T09:00:52.246 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1610000000000003e-06 2024-03-20T09:00:52.246 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.246 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.246 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:52.246 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.230109+0000", 2024-03-20T09:00:52.247 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.024744629000000001 2024-03-20T09:00:52.247 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.247 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.247 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:52.247 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.230204+0000", 2024-03-20T09:00:52.247 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5529e-05 2024-03-20T09:00:52.247 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.247 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.247 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,1", 2024-03-20T09:00:52.247 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.230319+0000", 2024-03-20T09:00:52.247 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000114744 2024-03-20T09:00:52.247 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.247 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.247 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:52.247 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.232694+0000", 2024-03-20T09:00:52.248 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0023751190000000002 2024-03-20T09:00:52.248 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.248 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.248 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.248 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.288571+0000", 2024-03-20T09:00:52.248 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.055876985999999997 2024-03-20T09:00:52.248 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.248 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.248 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.248 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.396625+0000", 2024-03-20T09:00:52.248 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.108054493 2024-03-20T09:00:52.248 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.248 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.248 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:52.249 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.396669+0000", 2024-03-20T09:00:52.249 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3891999999999999e-05 2024-03-20T09:00:52.249 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.249 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.249 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:52.249 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.396728+0000", 2024-03-20T09:00:52.249 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8434000000000003e-05 2024-03-20T09:00:52.249 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.249 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:52.249 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.249 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.249 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.249 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:116273 2.c 2:3ab71007:::benchmark_data_smithi122_107654_object116272:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-03-20T09:00:52.249 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:59:51.338183+0000", 2024-03-20T09:00:52.250 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 60.882141754000003, 2024-03-20T09:00:52.250 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.12596252299999999, 2024-03-20T09:00:52.250 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:52.250 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:52.250 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:52.250 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:52.250 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:52.250 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 116273 2024-03-20T09:00:52.250 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.250 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:52.250 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.250 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:52.250 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.338183+0000", 2024-03-20T09:00:52.250 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.251 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.251 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.251 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:52.251 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.338183+0000", 2024-03-20T09:00:52.251 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.251 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.251 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.251 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:52.251 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.338184+0000", 2024-03-20T09:00:52.251 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1699999999999997e-07 2024-03-20T09:00:52.251 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.251 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.251 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:52.252 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.338204+0000", 2024-03-20T09:00:52.252 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.967e-05 2024-03-20T09:00:52.252 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.252 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.252 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:52.252 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.338215+0000", 2024-03-20T09:00:52.252 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0848e-05 2024-03-20T09:00:52.252 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.252 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.252 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:52.252 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.338220+0000", 2024-03-20T09:00:52.252 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7790000000000001e-06 2024-03-20T09:00:52.252 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.253 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.253 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:52.253 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.397514+0000", 2024-03-20T09:00:52.253 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.059293534000000002 2024-03-20T09:00:52.253 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.253 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.253 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:52.253 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.397636+0000", 2024-03-20T09:00:52.253 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012247000000000001 2024-03-20T09:00:52.253 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.253 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.253 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,1", 2024-03-20T09:00:52.253 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.397806+0000", 2024-03-20T09:00:52.253 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016917700000000001 2024-03-20T09:00:52.254 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.254 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.254 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:52.254 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.408840+0000", 2024-03-20T09:00:52.254 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011034197000000001 2024-03-20T09:00:52.254 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.254 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.254 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.254 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.453995+0000", 2024-03-20T09:00:52.254 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.045155379000000002 2024-03-20T09:00:52.254 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.254 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.254 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.254 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.464024+0000", 2024-03-20T09:00:52.255 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010028944999999999 2024-03-20T09:00:52.255 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.255 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.255 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:52.255 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.464086+0000", 2024-03-20T09:00:52.255 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1505000000000002e-05 2024-03-20T09:00:52.255 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.255 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.255 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:52.255 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.464146+0000", 2024-03-20T09:00:52.255 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0102000000000002e-05 2024-03-20T09:00:52.255 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.255 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:52.255 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.256 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.256 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.256 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:116409 2.c 2:3ea2ea5a:::benchmark_data_smithi122_107654_object116408:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-03-20T09:00:52.256 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:59:51.516307+0000", 2024-03-20T09:00:52.256 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 60.704017761999999, 2024-03-20T09:00:52.256 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.134582808, 2024-03-20T09:00:52.256 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:52.256 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:52.256 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:52.256 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:52.256 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:52.256 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 116409 2024-03-20T09:00:52.256 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.256 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:52.257 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.257 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:52.257 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.516307+0000", 2024-03-20T09:00:52.257 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.257 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.257 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.257 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:52.257 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.516307+0000", 2024-03-20T09:00:52.257 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.257 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.257 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.257 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:52.257 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.516308+0000", 2024-03-20T09:00:52.257 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1620000000000001e-06 2024-03-20T09:00:52.258 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.258 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.258 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:52.258 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.516333+0000", 2024-03-20T09:00:52.258 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4780999999999999e-05 2024-03-20T09:00:52.258 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.258 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.258 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:52.258 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.516345+0000", 2024-03-20T09:00:52.258 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1600000000000001e-05 2024-03-20T09:00:52.258 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.258 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.258 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:52.259 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.516351+0000", 2024-03-20T09:00:52.259 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9769999999999997e-06 2024-03-20T09:00:52.259 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.259 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.259 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:52.259 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.575834+0000", 2024-03-20T09:00:52.259 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.059483108999999999 2024-03-20T09:00:52.259 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.259 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.259 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:52.259 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.575924+0000", 2024-03-20T09:00:52.259 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0581000000000006e-05 2024-03-20T09:00:52.260 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.260 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.260 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,1", 2024-03-20T09:00:52.260 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.576017+0000", 2024-03-20T09:00:52.260 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2802999999999999e-05 2024-03-20T09:00:52.260 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.260 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.260 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:52.260 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.587383+0000", 2024-03-20T09:00:52.260 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011366134999999999 2024-03-20T09:00:52.260 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.260 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.261 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.261 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.618598+0000", 2024-03-20T09:00:52.261 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.031215204 2024-03-20T09:00:52.261 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.261 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.261 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.261 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.650800+0000", 2024-03-20T09:00:52.261 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.032201322999999997 2024-03-20T09:00:52.261 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.261 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.261 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:52.261 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.650840+0000", 2024-03-20T09:00:52.261 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0602999999999997e-05 2024-03-20T09:00:52.261 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.262 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.262 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:52.262 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.650890+0000", 2024-03-20T09:00:52.262 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9530000000000002e-05 2024-03-20T09:00:52.262 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.262 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:52.262 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.262 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.262 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.262 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:116410 2.c 2:3d71f892:::benchmark_data_smithi122_107654_object116409:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-03-20T09:00:52.262 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:59:51.518113+0000", 2024-03-20T09:00:52.262 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 60.702212302, 2024-03-20T09:00:52.262 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.144706217, 2024-03-20T09:00:52.262 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:52.263 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:52.263 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:52.263 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:52.263 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:52.263 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 116410 2024-03-20T09:00:52.263 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.263 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:52.263 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.263 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:52.263 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.518113+0000", 2024-03-20T09:00:52.263 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.263 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.263 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.263 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:52.264 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.518113+0000", 2024-03-20T09:00:52.264 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.264 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.264 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.264 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:52.264 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.518113+0000", 2024-03-20T09:00:52.264 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6899999999999996e-07 2024-03-20T09:00:52.264 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.264 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.264 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:52.264 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.518135+0000", 2024-03-20T09:00:52.264 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1141999999999998e-05 2024-03-20T09:00:52.264 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.264 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.264 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:52.265 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.518146+0000", 2024-03-20T09:00:52.265 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1093e-05 2024-03-20T09:00:52.265 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.265 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.265 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:52.265 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.518153+0000", 2024-03-20T09:00:52.265 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0659999999999998e-06 2024-03-20T09:00:52.265 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.265 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.265 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:52.265 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.578928+0000", 2024-03-20T09:00:52.265 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.060775820000000001 2024-03-20T09:00:52.265 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.265 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.266 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:52.266 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.579024+0000", 2024-03-20T09:00:52.266 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5587999999999994e-05 2024-03-20T09:00:52.266 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.266 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.266 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,1", 2024-03-20T09:00:52.266 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.579129+0000", 2024-03-20T09:00:52.266 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010485299999999999 2024-03-20T09:00:52.266 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.266 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.266 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:52.266 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.587470+0000", 2024-03-20T09:00:52.266 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0083414030000000007 2024-03-20T09:00:52.266 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.267 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.267 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.267 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.618656+0000", 2024-03-20T09:00:52.267 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.031186077999999999 2024-03-20T09:00:52.267 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.267 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.267 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.267 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.662726+0000", 2024-03-20T09:00:52.267 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.044069097000000002 2024-03-20T09:00:52.267 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.267 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.267 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:52.267 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.662762+0000", 2024-03-20T09:00:52.268 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6257999999999997e-05 2024-03-20T09:00:52.268 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.268 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.268 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:52.268 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.662819+0000", 2024-03-20T09:00:52.268 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7049999999999998e-05 2024-03-20T09:00:52.268 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.268 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:52.268 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.268 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.268 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.268 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:116531 2.c 2:354436dd:::benchmark_data_smithi122_107654_object116530:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-03-20T09:00:52.268 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:59:51.677494+0000", 2024-03-20T09:00:52.268 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 60.542831026999998, 2024-03-20T09:00:52.269 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.12885465700000001, 2024-03-20T09:00:52.269 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:52.269 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:52.269 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:52.269 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:52.269 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:52.269 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 116531 2024-03-20T09:00:52.269 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.269 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:52.269 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.269 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:52.269 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.677494+0000", 2024-03-20T09:00:52.269 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.270 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.270 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.270 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:52.270 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.677494+0000", 2024-03-20T09:00:52.270 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.270 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.270 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.270 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:52.270 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.677494+0000", 2024-03-20T09:00:52.270 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6699999999999999e-07 2024-03-20T09:00:52.270 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.270 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.270 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:52.271 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.677522+0000", 2024-03-20T09:00:52.271 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.779e-05 2024-03-20T09:00:52.271 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.271 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.271 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:52.271 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.677531+0000", 2024-03-20T09:00:52.271 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2620000000000004e-06 2024-03-20T09:00:52.271 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.271 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.271 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:52.271 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.677538+0000", 2024-03-20T09:00:52.271 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.737e-06 2024-03-20T09:00:52.271 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.271 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.271 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:52.272 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.758795+0000", 2024-03-20T09:00:52.272 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.081256825000000005 2024-03-20T09:00:52.272 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.272 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.272 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:52.272 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.758900+0000", 2024-03-20T09:00:52.272 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105486 2024-03-20T09:00:52.272 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.272 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.272 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,1", 2024-03-20T09:00:52.272 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.759017+0000", 2024-03-20T09:00:52.272 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000116604 2024-03-20T09:00:52.272 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.273 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.273 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:52.273 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.769878+0000", 2024-03-20T09:00:52.273 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010861050000000001 2024-03-20T09:00:52.273 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.273 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.273 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.273 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.790374+0000", 2024-03-20T09:00:52.273 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.020495607999999998 2024-03-20T09:00:52.273 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.273 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.273 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.273 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.806224+0000", 2024-03-20T09:00:52.273 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015850181000000001 2024-03-20T09:00:52.274 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.274 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.274 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:52.274 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.806273+0000", 2024-03-20T09:00:52.274 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9041e-05 2024-03-20T09:00:52.274 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.274 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.274 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:52.274 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.806349+0000", 2024-03-20T09:00:52.274 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5605999999999994e-05 2024-03-20T09:00:52.274 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.274 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:52.274 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.275 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.275 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.275 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:116542 2.c 2:3edd835c:::benchmark_data_smithi122_107654_object116541:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-03-20T09:00:52.275 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:59:51.682426+0000", 2024-03-20T09:00:52.275 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 60.537898859000002, 2024-03-20T09:00:52.275 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.12410114999999999, 2024-03-20T09:00:52.275 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:52.275 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:52.275 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:52.275 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:52.275 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:52.275 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 116542 2024-03-20T09:00:52.275 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.276 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:52.276 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.276 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:52.276 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.682426+0000", 2024-03-20T09:00:52.276 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.276 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.276 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.276 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:52.276 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.682426+0000", 2024-03-20T09:00:52.276 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.276 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.276 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.276 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:52.276 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.682428+0000", 2024-03-20T09:00:52.277 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.815e-06 2024-03-20T09:00:52.277 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.277 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.277 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:52.277 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.682449+0000", 2024-03-20T09:00:52.277 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1039999999999998e-05 2024-03-20T09:00:52.277 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.277 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.277 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:52.277 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.682462+0000", 2024-03-20T09:00:52.277 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3186e-05 2024-03-20T09:00:52.277 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.277 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.277 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:52.278 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.682469+0000", 2024-03-20T09:00:52.278 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.306e-06 2024-03-20T09:00:52.278 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.278 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.278 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:52.278 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.762194+0000", 2024-03-20T09:00:52.278 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.079725070999999995 2024-03-20T09:00:52.278 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.278 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.278 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:52.278 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.762308+0000", 2024-03-20T09:00:52.278 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000113618 2024-03-20T09:00:52.278 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.278 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.279 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,1", 2024-03-20T09:00:52.279 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.762436+0000", 2024-03-20T09:00:52.279 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012765300000000001 2024-03-20T09:00:52.279 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.279 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.279 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:52.279 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.769885+0000", 2024-03-20T09:00:52.279 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0074489580000000003 2024-03-20T09:00:52.279 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.279 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.279 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.279 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.790416+0000", 2024-03-20T09:00:52.279 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.020531514000000001 2024-03-20T09:00:52.279 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.280 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.280 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.280 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.806405+0000", 2024-03-20T09:00:52.280 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015988498 2024-03-20T09:00:52.280 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.280 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.280 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:52.280 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.806469+0000", 2024-03-20T09:00:52.280 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4306000000000004e-05 2024-03-20T09:00:52.280 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.280 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.280 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:52.280 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.806527+0000", 2024-03-20T09:00:52.280 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8184999999999999e-05 2024-03-20T09:00:52.281 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.281 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:52.281 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.281 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.281 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.281 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:116677 2.c 2:394ac6ab:::benchmark_data_smithi122_107654_object116676:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-03-20T09:00:52.281 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:59:51.856481+0000", 2024-03-20T09:00:52.281 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 60.363843709999998, 2024-03-20T09:00:52.281 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.131526797, 2024-03-20T09:00:52.281 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:52.281 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:52.281 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:52.281 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:52.282 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:52.282 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 116677 2024-03-20T09:00:52.282 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.282 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:52.282 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.282 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:52.282 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.856481+0000", 2024-03-20T09:00:52.282 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.282 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.282 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.282 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:52.282 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.856481+0000", 2024-03-20T09:00:52.282 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.282 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.283 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.283 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:52.283 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.856482+0000", 2024-03-20T09:00:52.283 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2300000000000001e-06 2024-03-20T09:00:52.283 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.283 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.283 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:52.283 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.856507+0000", 2024-03-20T09:00:52.283 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4219999999999999e-05 2024-03-20T09:00:52.283 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.283 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.283 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:52.283 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.856525+0000", 2024-03-20T09:00:52.283 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8280000000000001e-05 2024-03-20T09:00:52.284 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.284 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.284 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:52.284 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.856535+0000", 2024-03-20T09:00:52.284 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6670000000000008e-06 2024-03-20T09:00:52.284 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.284 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.284 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:52.284 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.935763+0000", 2024-03-20T09:00:52.284 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.079228648999999998 2024-03-20T09:00:52.284 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.284 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.284 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:52.284 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.935885+0000", 2024-03-20T09:00:52.285 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000121458 2024-03-20T09:00:52.285 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.285 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.285 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,1", 2024-03-20T09:00:52.285 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.936006+0000", 2024-03-20T09:00:52.285 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012094100000000001 2024-03-20T09:00:52.285 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.285 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.285 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:52.285 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.939351+0000", 2024-03-20T09:00:52.285 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0033454299999999999 2024-03-20T09:00:52.285 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.285 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.285 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.286 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.970538+0000", 2024-03-20T09:00:52.286 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.031187109000000001 2024-03-20T09:00:52.286 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.286 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.286 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.286 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.987928+0000", 2024-03-20T09:00:52.286 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017390137 2024-03-20T09:00:52.286 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.286 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.286 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:52.286 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.987970+0000", 2024-03-20T09:00:52.286 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1918000000000001e-05 2024-03-20T09:00:52.286 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.286 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.287 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:52.287 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.988008+0000", 2024-03-20T09:00:52.287 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7758e-05 2024-03-20T09:00:52.287 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.287 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:52.287 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.287 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.287 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.287 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:117546 2.c 2:3bbf0116:::benchmark_data_smithi122_107654_object117545:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-20T09:00:52.287 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:59:52.871253+0000", 2024-03-20T09:00:52.287 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 59.349072354, 2024-03-20T09:00:52.287 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.13200684300000001, 2024-03-20T09:00:52.287 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:52.287 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:52.288 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:52.288 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:52.288 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:52.288 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 117546 2024-03-20T09:00:52.288 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.288 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:52.288 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.288 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:52.288 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:52.871253+0000", 2024-03-20T09:00:52.288 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.288 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.288 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.288 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:52.288 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:52.871253+0000", 2024-03-20T09:00:52.288 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.289 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.289 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.289 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:52.289 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:52.871254+0000", 2024-03-20T09:00:52.289 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.22e-06 2024-03-20T09:00:52.289 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.289 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.289 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:52.289 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:52.871273+0000", 2024-03-20T09:00:52.289 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9443999999999999e-05 2024-03-20T09:00:52.289 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.289 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.289 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:52.289 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:52.871291+0000", 2024-03-20T09:00:52.290 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7385e-05 2024-03-20T09:00:52.290 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.290 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.290 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:52.290 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:52.871299+0000", 2024-03-20T09:00:52.290 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8869999999999998e-06 2024-03-20T09:00:52.290 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.290 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.290 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:52.290 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:52.941859+0000", 2024-03-20T09:00:52.290 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.070559223000000004 2024-03-20T09:00:52.290 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.290 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.290 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:52.291 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:52.941968+0000", 2024-03-20T09:00:52.291 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010958 2024-03-20T09:00:52.291 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.291 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.291 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,1", 2024-03-20T09:00:52.291 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:52.942079+0000", 2024-03-20T09:00:52.291 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011029199999999999 2024-03-20T09:00:52.291 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.291 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.291 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:52.291 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:52.945779+0000", 2024-03-20T09:00:52.291 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0036999620000000002 2024-03-20T09:00:52.291 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.292 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.292 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.292 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:52.973775+0000", 2024-03-20T09:00:52.292 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.027996061999999999 2024-03-20T09:00:52.292 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.292 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.292 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.292 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:53.003165+0000", 2024-03-20T09:00:52.292 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.029390039999999999 2024-03-20T09:00:52.292 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.292 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.292 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:52.292 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:53.003203+0000", 2024-03-20T09:00:52.292 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8470000000000003e-05 2024-03-20T09:00:52.292 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.293 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.293 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:52.293 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:53.003259+0000", 2024-03-20T09:00:52.293 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6277999999999999e-05 2024-03-20T09:00:52.293 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.293 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:52.293 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.293 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.293 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.293 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:117971 2.c 2:3693bb2f:::benchmark_data_smithi122_107654_object117970:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-20T09:00:52.293 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:59:53.375705+0000", 2024-03-20T09:00:52.293 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 58.844620183000004, 2024-03-20T09:00:52.293 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.13271301299999999, 2024-03-20T09:00:52.293 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:52.294 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:52.294 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:52.294 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:52.294 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:52.294 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 117971 2024-03-20T09:00:52.294 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.294 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:52.294 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.294 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:52.294 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:53.375705+0000", 2024-03-20T09:00:52.294 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.294 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.294 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.294 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:52.295 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:53.375705+0000", 2024-03-20T09:00:52.295 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.295 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.295 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.295 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:52.295 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:53.375706+0000", 2024-03-20T09:00:52.295 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.311e-06 2024-03-20T09:00:52.295 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.295 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.295 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:52.295 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:53.375727+0000", 2024-03-20T09:00:52.295 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0781e-05 2024-03-20T09:00:52.295 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.295 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.296 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:52.296 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:53.375740+0000", 2024-03-20T09:00:52.296 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2867999999999999e-05 2024-03-20T09:00:52.296 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.296 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.296 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:52.296 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:53.375748+0000", 2024-03-20T09:00:52.296 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0339999999999997e-06 2024-03-20T09:00:52.296 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.296 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.296 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:52.296 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:53.434566+0000", 2024-03-20T09:00:52.296 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.058817849999999998 2024-03-20T09:00:52.296 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.296 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.297 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:52.297 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:53.434705+0000", 2024-03-20T09:00:52.297 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013945 2024-03-20T09:00:52.297 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.297 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.297 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,1", 2024-03-20T09:00:52.297 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:53.434888+0000", 2024-03-20T09:00:52.297 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00018330199999999999 2024-03-20T09:00:52.297 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.297 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.297 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:52.297 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:53.438487+0000", 2024-03-20T09:00:52.297 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0035990800000000002 2024-03-20T09:00:52.297 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.298 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.298 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.298 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:53.497234+0000", 2024-03-20T09:00:52.298 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.058746782999999997 2024-03-20T09:00:52.298 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.298 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.298 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.298 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:53.508349+0000", 2024-03-20T09:00:52.298 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011114565999999999 2024-03-20T09:00:52.298 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.298 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.298 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:52.298 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:53.508386+0000", 2024-03-20T09:00:52.298 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7353e-05 2024-03-20T09:00:52.299 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.299 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.299 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:52.299 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:53.508418+0000", 2024-03-20T09:00:52.299 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1634999999999998e-05 2024-03-20T09:00:52.299 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.299 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:52.299 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.299 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.299 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.299 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:117976 2.c 2:3d201fb1:::benchmark_data_smithi122_107654_object117975:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-20T09:00:52.299 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:59:53.384872+0000", 2024-03-20T09:00:52.299 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 58.835452930000002, 2024-03-20T09:00:52.300 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.123695034, 2024-03-20T09:00:52.300 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:52.300 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:52.300 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:52.300 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:52.300 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:52.300 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 117976 2024-03-20T09:00:52.300 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.300 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:52.300 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.300 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:52.300 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:53.384872+0000", 2024-03-20T09:00:52.300 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.300 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.301 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.301 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:52.301 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:53.384872+0000", 2024-03-20T09:00:52.301 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.301 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.301 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.301 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:52.301 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:53.384873+0000", 2024-03-20T09:00:52.301 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.223e-06 2024-03-20T09:00:52.301 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.301 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.301 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:52.301 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:53.384891+0000", 2024-03-20T09:00:52.301 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8043999999999999e-05 2024-03-20T09:00:52.302 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.302 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.302 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:52.302 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:53.384902+0000", 2024-03-20T09:00:52.302 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0325e-05 2024-03-20T09:00:52.302 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.302 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.302 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:52.302 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:53.384909+0000", 2024-03-20T09:00:52.302 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4440000000000002e-06 2024-03-20T09:00:52.302 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.302 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.302 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:52.303 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:53.439380+0000", 2024-03-20T09:00:52.303 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.054471118999999998 2024-03-20T09:00:52.303 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.303 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.303 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:52.303 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:53.439533+0000", 2024-03-20T09:00:52.303 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015242399999999999 2024-03-20T09:00:52.303 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.303 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.303 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,1", 2024-03-20T09:00:52.303 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:53.439700+0000", 2024-03-20T09:00:52.303 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000167646 2024-03-20T09:00:52.303 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.303 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.304 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:52.304 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:53.449669+0000", 2024-03-20T09:00:52.304 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0099692889999999992 2024-03-20T09:00:52.304 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.304 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.304 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.304 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:53.497269+0000", 2024-03-20T09:00:52.304 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.047599771999999999 2024-03-20T09:00:52.304 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.304 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.304 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.305 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:53.508462+0000", 2024-03-20T09:00:52.305 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011192739 2024-03-20T09:00:52.305 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.305 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.305 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:52.305 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:53.508518+0000", 2024-03-20T09:00:52.305 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5705999999999998e-05 2024-03-20T09:00:52.305 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.305 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.305 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:52.305 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:53.508567+0000", 2024-03-20T09:00:52.305 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9302999999999999e-05 2024-03-20T09:00:52.306 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.306 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:52.306 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.306 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.306 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.306 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:118463 2.c 2:34ab857c:::benchmark_data_smithi122_107654_object118462:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-20T09:00:52.306 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:59:54.004321+0000", 2024-03-20T09:00:52.306 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 58.216003532000002, 2024-03-20T09:00:52.306 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.13158202499999999, 2024-03-20T09:00:52.306 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:52.306 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:52.306 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:52.306 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:52.307 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:52.307 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 118463 2024-03-20T09:00:52.307 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.307 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:52.307 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.307 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:52.307 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.004321+0000", 2024-03-20T09:00:52.307 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.307 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.307 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.307 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:52.307 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.004321+0000", 2024-03-20T09:00:52.307 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.308 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.308 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.308 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:52.308 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.004323+0000", 2024-03-20T09:00:52.308 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2190000000000001e-06 2024-03-20T09:00:52.308 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.308 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.308 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:52.308 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.004346+0000", 2024-03-20T09:00:52.308 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3237000000000001e-05 2024-03-20T09:00:52.308 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.308 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.308 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:52.308 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.004360+0000", 2024-03-20T09:00:52.309 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4406999999999999e-05 2024-03-20T09:00:52.309 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.309 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.309 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:52.309 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.004369+0000", 2024-03-20T09:00:52.309 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0240000000000003e-06 2024-03-20T09:00:52.309 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.309 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.309 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:52.309 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.081208+0000", 2024-03-20T09:00:52.309 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.076839114 2024-03-20T09:00:52.309 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.309 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.310 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:52.310 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.081323+0000", 2024-03-20T09:00:52.310 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011497600000000001 2024-03-20T09:00:52.310 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.310 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.310 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,1", 2024-03-20T09:00:52.310 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.081438+0000", 2024-03-20T09:00:52.310 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000114838 2024-03-20T09:00:52.310 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.310 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.310 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:52.310 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.084536+0000", 2024-03-20T09:00:52.310 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0030978669999999998 2024-03-20T09:00:52.311 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.311 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.311 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.311 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.127195+0000", 2024-03-20T09:00:52.311 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.042658579000000002 2024-03-20T09:00:52.311 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.311 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.311 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.311 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.135819+0000", 2024-03-20T09:00:52.311 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0086247960000000005 2024-03-20T09:00:52.311 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.311 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.311 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:52.311 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.135858+0000", 2024-03-20T09:00:52.312 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8810999999999998e-05 2024-03-20T09:00:52.312 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.312 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.312 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:52.312 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.135903+0000", 2024-03-20T09:00:52.312 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5157000000000002e-05 2024-03-20T09:00:52.312 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.312 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:52.312 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.312 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.312 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.312 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:118730 2.c 2:3d905c19:::benchmark_data_smithi122_107654_object118729:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-20T09:00:52.312 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:59:54.346667+0000", 2024-03-20T09:00:52.313 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 57.873658122999998, 2024-03-20T09:00:52.313 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.134685058, 2024-03-20T09:00:52.313 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:52.313 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:52.313 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:52.313 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:52.313 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:52.313 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 118730 2024-03-20T09:00:52.313 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.313 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:52.313 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.313 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:52.313 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.346667+0000", 2024-03-20T09:00:52.313 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.313 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.314 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.314 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:52.314 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.346667+0000", 2024-03-20T09:00:52.314 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.314 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.314 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.314 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:52.314 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.346668+0000", 2024-03-20T09:00:52.314 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6330000000000001e-06 2024-03-20T09:00:52.314 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.314 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.314 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:52.314 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.346691+0000", 2024-03-20T09:00:52.315 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2135000000000001e-05 2024-03-20T09:00:52.315 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.315 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.315 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:52.315 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.346703+0000", 2024-03-20T09:00:52.315 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2296999999999999e-05 2024-03-20T09:00:52.315 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.315 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.315 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:52.315 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.346712+0000", 2024-03-20T09:00:52.315 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7520000000000002e-06 2024-03-20T09:00:52.315 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.315 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.315 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:52.316 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.402153+0000", 2024-03-20T09:00:52.316 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.055441035 2024-03-20T09:00:52.316 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.316 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.316 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:52.316 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.402306+0000", 2024-03-20T09:00:52.316 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015310599999999999 2024-03-20T09:00:52.316 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.316 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.316 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,1", 2024-03-20T09:00:52.316 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.402469+0000", 2024-03-20T09:00:52.316 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016302800000000001 2024-03-20T09:00:52.316 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.316 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.316 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:52.317 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.420185+0000", 2024-03-20T09:00:52.317 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017716671999999999 2024-03-20T09:00:52.317 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.317 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.317 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.317 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.477037+0000", 2024-03-20T09:00:52.317 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.056851930000000002 2024-03-20T09:00:52.317 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.317 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.317 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.317 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.481213+0000", 2024-03-20T09:00:52.317 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0041753149999999998 2024-03-20T09:00:52.317 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.317 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.318 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:52.318 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.481274+0000", 2024-03-20T09:00:52.318 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1255999999999998e-05 2024-03-20T09:00:52.318 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.318 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.318 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:52.318 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.481352+0000", 2024-03-20T09:00:52.318 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7898999999999994e-05 2024-03-20T09:00:52.318 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.318 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:52.318 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.318 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.318 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.318 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:118735 2.c 2:31f42c05:::benchmark_data_smithi122_107654_object118734:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-20T09:00:52.319 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:59:54.352319+0000", 2024-03-20T09:00:52.319 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 57.868005783000001, 2024-03-20T09:00:52.319 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.129169543, 2024-03-20T09:00:52.319 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:52.319 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:52.319 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:52.319 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:52.319 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:52.319 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 118735 2024-03-20T09:00:52.319 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.319 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:52.319 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.319 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:52.319 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.352319+0000", 2024-03-20T09:00:52.319 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.320 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.320 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.320 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:52.320 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.352319+0000", 2024-03-20T09:00:52.320 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.320 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.320 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.320 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:52.320 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.352320+0000", 2024-03-20T09:00:52.320 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.288e-06 2024-03-20T09:00:52.320 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.320 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.320 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:52.321 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.352336+0000", 2024-03-20T09:00:52.321 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5990000000000001e-05 2024-03-20T09:00:52.321 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.321 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.321 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:52.321 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.352349+0000", 2024-03-20T09:00:52.321 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2268e-05 2024-03-20T09:00:52.321 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.321 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.321 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:52.321 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.352356+0000", 2024-03-20T09:00:52.321 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2420000000000003e-06 2024-03-20T09:00:52.321 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.321 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.322 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:52.322 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.406002+0000", 2024-03-20T09:00:52.322 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.053646218000000002 2024-03-20T09:00:52.322 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.322 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.322 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:52.322 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.406139+0000", 2024-03-20T09:00:52.322 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013686699999999999 2024-03-20T09:00:52.322 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.322 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.322 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,1", 2024-03-20T09:00:52.322 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.406326+0000", 2024-03-20T09:00:52.322 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00018697900000000001 2024-03-20T09:00:52.322 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.323 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.323 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:52.323 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.420190+0000", 2024-03-20T09:00:52.323 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013863723 2024-03-20T09:00:52.323 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.323 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.323 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.323 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.477097+0000", 2024-03-20T09:00:52.323 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.056906887000000003 2024-03-20T09:00:52.323 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.323 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.323 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.323 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.481398+0000", 2024-03-20T09:00:52.323 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.004301661 2024-03-20T09:00:52.324 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.324 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.324 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:52.324 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.481434+0000", 2024-03-20T09:00:52.324 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5731000000000001e-05 2024-03-20T09:00:52.324 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.324 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.324 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:52.324 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.481489+0000", 2024-03-20T09:00:52.324 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4688999999999998e-05 2024-03-20T09:00:52.324 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.324 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:52.324 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.324 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.325 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.325 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:118745 2.c 2:34a6a08c:::benchmark_data_smithi122_107654_object118744:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-20T09:00:52.325 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:59:54.355991+0000", 2024-03-20T09:00:52.325 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 57.864333567000003, 2024-03-20T09:00:52.325 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.12565564900000001, 2024-03-20T09:00:52.325 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:52.325 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:52.325 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:52.325 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:52.325 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:52.325 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 118745 2024-03-20T09:00:52.325 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.325 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:52.325 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.325 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:52.326 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.355991+0000", 2024-03-20T09:00:52.326 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.326 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.326 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.326 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:52.326 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.355991+0000", 2024-03-20T09:00:52.326 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.326 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.326 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.326 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:52.326 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.355993+0000", 2024-03-20T09:00:52.326 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5209999999999999e-06 2024-03-20T09:00:52.326 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.326 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.327 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:52.327 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.356016+0000", 2024-03-20T09:00:52.327 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2789000000000002e-05 2024-03-20T09:00:52.327 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.327 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.327 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:52.327 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.356036+0000", 2024-03-20T09:00:52.327 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9888999999999999e-05 2024-03-20T09:00:52.327 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.327 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.327 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:52.327 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.356045+0000", 2024-03-20T09:00:52.327 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0669999999999998e-06 2024-03-20T09:00:52.327 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.328 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.328 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:52.328 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.409370+0000", 2024-03-20T09:00:52.328 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.053325526999999998 2024-03-20T09:00:52.328 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.328 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.328 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:52.328 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.409465+0000", 2024-03-20T09:00:52.328 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5038999999999996e-05 2024-03-20T09:00:52.328 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.328 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.328 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,1", 2024-03-20T09:00:52.328 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.409586+0000", 2024-03-20T09:00:52.328 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012050800000000001 2024-03-20T09:00:52.329 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.329 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.329 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:52.329 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.425452+0000", 2024-03-20T09:00:52.329 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015865840999999999 2024-03-20T09:00:52.329 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.329 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.329 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.329 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.477131+0000", 2024-03-20T09:00:52.329 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.051679010999999997 2024-03-20T09:00:52.329 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.329 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.329 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.330 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.481538+0000", 2024-03-20T09:00:52.330 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0044075140000000004 2024-03-20T09:00:52.330 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.330 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.330 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:52.330 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.481595+0000", 2024-03-20T09:00:52.330 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7188000000000002e-05 2024-03-20T09:00:52.330 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.330 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.330 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:52.330 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.481647+0000", 2024-03-20T09:00:52.330 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1755000000000002e-05 2024-03-20T09:00:52.330 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.330 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:52.331 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.331 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.331 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.331 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:118747 2.c 2:319652a6:::benchmark_data_smithi122_107654_object118746:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-20T09:00:52.331 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:59:54.357450+0000", 2024-03-20T09:00:52.331 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 57.862874499999997, 2024-03-20T09:00:52.331 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.13811911099999999, 2024-03-20T09:00:52.331 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:52.331 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:52.331 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:52.331 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:52.331 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:52.331 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 118747 2024-03-20T09:00:52.331 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.332 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:52.332 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.332 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:52.332 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.357450+0000", 2024-03-20T09:00:52.332 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.332 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.332 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.332 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:52.332 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.357450+0000", 2024-03-20T09:00:52.332 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.332 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.332 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.332 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:52.332 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.357451+0000", 2024-03-20T09:00:52.333 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0720000000000001e-06 2024-03-20T09:00:52.333 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.333 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.333 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:52.333 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.357476+0000", 2024-03-20T09:00:52.333 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4400999999999999e-05 2024-03-20T09:00:52.333 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.333 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.333 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:52.333 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.357490+0000", 2024-03-20T09:00:52.333 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3735e-05 2024-03-20T09:00:52.333 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.333 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.333 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:52.334 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.357498+0000", 2024-03-20T09:00:52.334 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9379999999999992e-06 2024-03-20T09:00:52.334 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.334 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.334 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:52.334 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.412214+0000", 2024-03-20T09:00:52.334 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.054716527000000001 2024-03-20T09:00:52.334 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.334 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.334 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:52.334 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.412352+0000", 2024-03-20T09:00:52.334 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000138236 2024-03-20T09:00:52.335 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.335 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.335 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,1", 2024-03-20T09:00:52.335 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.412531+0000", 2024-03-20T09:00:52.335 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00017862600000000001 2024-03-20T09:00:52.335 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.335 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.335 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:52.335 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.425458+0000", 2024-03-20T09:00:52.335 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012926582000000001 2024-03-20T09:00:52.335 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.335 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.335 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.335 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.481684+0000", 2024-03-20T09:00:52.336 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.056226037999999999 2024-03-20T09:00:52.336 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.336 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.336 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.336 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.495463+0000", 2024-03-20T09:00:52.336 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013779777999999999 2024-03-20T09:00:52.336 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.336 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.336 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:52.336 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.495504+0000", 2024-03-20T09:00:52.336 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1060999999999999e-05 2024-03-20T09:00:52.336 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.337 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.337 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:52.337 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.495570+0000", 2024-03-20T09:00:52.337 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5116999999999993e-05 2024-03-20T09:00:52.337 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.337 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:52.337 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.337 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.337 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.337 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:118825 2.c 2:3b28bb18:::benchmark_data_smithi122_107654_object40:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-20T09:00:52.337 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:59:55.490736+0000", 2024-03-20T09:00:52.337 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 56.729589216000001, 2024-03-20T09:00:52.337 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.14062445800000001, 2024-03-20T09:00:52.337 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:52.338 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:52.338 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:52.338 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:52.338 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:52.338 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 118825 2024-03-20T09:00:52.338 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.338 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:52.338 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.338 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:52.338 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:55.490736+0000", 2024-03-20T09:00:52.338 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.338 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.338 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.339 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:52.339 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:55.490736+0000", 2024-03-20T09:00:52.339 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.339 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.339 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.339 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:52.339 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:55.490737+0000", 2024-03-20T09:00:52.339 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3650000000000001e-06 2024-03-20T09:00:52.339 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.339 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.339 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:52.339 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:55.490742+0000", 2024-03-20T09:00:52.339 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4020000000000001e-06 2024-03-20T09:00:52.340 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.340 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.340 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:52.340 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:55.490756+0000", 2024-03-20T09:00:52.340 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3805000000000001e-05 2024-03-20T09:00:52.340 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.340 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.340 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:52.340 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:55.490765+0000", 2024-03-20T09:00:52.340 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7309999999999999e-06 2024-03-20T09:00:52.340 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.340 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.340 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:52.340 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:55.565645+0000", 2024-03-20T09:00:52.341 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.074879596000000007 2024-03-20T09:00:52.341 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.341 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.341 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:52.341 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:55.565770+0000", 2024-03-20T09:00:52.341 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012557399999999999 2024-03-20T09:00:52.341 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.341 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.341 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,1", 2024-03-20T09:00:52.341 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:55.565897+0000", 2024-03-20T09:00:52.341 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012646599999999999 2024-03-20T09:00:52.341 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.341 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.341 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:52.342 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:55.575641+0000", 2024-03-20T09:00:52.342 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0097443489999999994 2024-03-20T09:00:52.342 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.342 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.342 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.342 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:55.626130+0000", 2024-03-20T09:00:52.342 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.050489499 2024-03-20T09:00:52.342 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.342 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.342 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.342 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:55.631262+0000", 2024-03-20T09:00:52.342 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0051313230000000001 2024-03-20T09:00:52.342 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.343 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.343 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:52.343 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:55.631306+0000", 2024-03-20T09:00:52.343 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4311000000000003e-05 2024-03-20T09:00:52.343 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.343 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.343 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:52.343 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:55.631360+0000", 2024-03-20T09:00:52.343 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4036999999999998e-05 2024-03-20T09:00:52.343 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.343 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:52.343 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.343 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.343 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.344 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:118853 2.c 2:35aa9c37:::benchmark_data_smithi122_107654_object68:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-03-20T09:00:52.344 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:59:55.518772+0000", 2024-03-20T09:00:52.344 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 56.701552759000002, 2024-03-20T09:00:52.344 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.124689865, 2024-03-20T09:00:52.344 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:00:52.344 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:00:52.344 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:00:52.344 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:00:52.344 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:00:52.344 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 118853 2024-03-20T09:00:52.344 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.344 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:00:52.344 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.345 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:00:52.345 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:55.518772+0000", 2024-03-20T09:00:52.345 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.345 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.345 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.345 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:00:52.345 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:55.518772+0000", 2024-03-20T09:00:52.345 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:00:52.345 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.345 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.345 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:00:52.345 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:55.518773+0000", 2024-03-20T09:00:52.345 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.297e-06 2024-03-20T09:00:52.345 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.346 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.346 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:00:52.346 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:55.518788+0000", 2024-03-20T09:00:52.346 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4518999999999999e-05 2024-03-20T09:00:52.346 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.346 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.346 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:00:52.346 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:55.518800+0000", 2024-03-20T09:00:52.346 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1627999999999999e-05 2024-03-20T09:00:52.346 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.346 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.346 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:00:52.346 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:55.518807+0000", 2024-03-20T09:00:52.346 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2389999999999996e-06 2024-03-20T09:00:52.347 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.347 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.347 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:00:52.347 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:55.569126+0000", 2024-03-20T09:00:52.347 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.050319286999999997 2024-03-20T09:00:52.347 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.347 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.347 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:00:52.347 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:55.569257+0000", 2024-03-20T09:00:52.347 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013093499999999999 2024-03-20T09:00:52.347 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.347 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.347 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,1", 2024-03-20T09:00:52.347 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:55.569404+0000", 2024-03-20T09:00:52.348 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014715900000000001 2024-03-20T09:00:52.348 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.348 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.348 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:00:52.348 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:55.579746+0000", 2024-03-20T09:00:52.348 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010341515000000001 2024-03-20T09:00:52.348 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.348 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.348 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.348 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:55.626183+0000", 2024-03-20T09:00:52.348 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.046437025999999999 2024-03-20T09:00:52.348 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.348 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.348 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:00:52.348 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:55.643393+0000", 2024-03-20T09:00:52.349 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017210744 2024-03-20T09:00:52.349 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.349 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.349 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:00:52.349 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:55.643431+0000", 2024-03-20T09:00:52.349 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7481000000000002e-05 2024-03-20T09:00:52.349 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:00:52.349 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:00:52.349 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:00:52.349 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:55.643462+0000", 2024-03-20T09:00:52.349 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1034999999999997e-05 2024-03-20T09:00:52.349 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.349 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:52.349 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.350 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:00:52.350 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:00:52.350 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:00:52.350 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:51 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:00:52.351 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:51 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:00:52.351 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:51 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:00:52.351 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:00:52.351 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:00:52.351 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:00:52.351 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:00:52.369 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:00:52.409 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:00:52.449 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:00:52.454 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:00:52.454 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:00:52.455 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:00:52.455 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:00:52.455 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:00:52.455 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:00:52.455 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:00:52.455 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:00:52.455 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:00:52.455 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:00:52.455 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:00:52.455 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:00:52.819 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:00:52.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:52.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:00:52.926 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:00:52.943 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:52 smithi122 ceph-mon[73495]: 2.a scrub starts 2024-03-20T09:00:52.944 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:52 smithi122 ceph-mon[73495]: pgmap v493: 17 pgs: 17 active+clean; 4.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 39 op/s 2024-03-20T09:00:52.944 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:52 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/94445238' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:00:52.954 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:52 smithi026 ceph-mon[68309]: 2.a scrub starts 2024-03-20T09:00:52.954 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:52 smithi026 ceph-mon[68309]: pgmap v493: 17 pgs: 17 active+clean; 4.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 39 op/s 2024-03-20T09:00:52.955 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:52 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/94445238' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:00:52.977 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:52.989 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-20T09:00:53.006 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:00:53.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:53.010 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:00:53.111 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:00:53.141 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:52 smithi115 ceph-mon[72476]: 2.a scrub starts 2024-03-20T09:00:53.141 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:52 smithi115 ceph-mon[72476]: pgmap v493: 17 pgs: 17 active+clean; 4.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 39 op/s 2024-03-20T09:00:53.141 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:52 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/94445238' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:00:53.144 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:53.168 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:00:53.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:53.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:00:53.275 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:00:53.303 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:53.324 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:00:53.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:53.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:00:53.428 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:00:53.478 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:53.507 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:00:53.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:53.514 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:00:53.614 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:00:53.644 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:53.667 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:00:53.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:53.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:00:53.771 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:00:53.802 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:53.826 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:00:53.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:53.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:00:53.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:53 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:00:53.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:53 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:00:53.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:53 smithi115 ceph-mon[72476]: 2.a scrub ok 2024-03-20T09:00:53.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:53 smithi115 ceph-mon[72476]: 2.8 scrub starts 2024-03-20T09:00:53.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:53 smithi115 ceph-mon[72476]: 2.e scrub starts 2024-03-20T09:00:53.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:53 smithi115 ceph-mon[72476]: osdmap e166: 12 total, 12 up, 12 in 2024-03-20T09:00:53.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:53 smithi115 ceph-mon[72476]: 2.b scrub starts 2024-03-20T09:00:53.931 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:00:53.976 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:54.010 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:00:54.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:54.014 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:00:54.115 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:00:54.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:53 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:00:54.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:53 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:00:54.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:53 smithi122 ceph-mon[73495]: 2.a scrub ok 2024-03-20T09:00:54.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:53 smithi122 ceph-mon[73495]: 2.8 scrub starts 2024-03-20T09:00:54.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:53 smithi122 ceph-mon[73495]: 2.e scrub starts 2024-03-20T09:00:54.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:53 smithi122 ceph-mon[73495]: osdmap e166: 12 total, 12 up, 12 in 2024-03-20T09:00:54.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:53 smithi122 ceph-mon[73495]: 2.b scrub starts 2024-03-20T09:00:54.202 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:53 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:00:54.202 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:53 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:00:54.202 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:53 smithi026 ceph-mon[68309]: 2.a scrub ok 2024-03-20T09:00:54.202 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:53 smithi026 ceph-mon[68309]: 2.8 scrub starts 2024-03-20T09:00:54.202 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:53 smithi026 ceph-mon[68309]: 2.e scrub starts 2024-03-20T09:00:54.202 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:53 smithi026 ceph-mon[68309]: osdmap e166: 12 total, 12 up, 12 in 2024-03-20T09:00:54.202 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:53 smithi026 ceph-mon[68309]: 2.b scrub starts 2024-03-20T09:00:54.216 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:54.448 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:00:54.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:54.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:00:54.557 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:00:54.618 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:54.651 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:00:54.652 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:00:54.654 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:00:54.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:54.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:00:54.761 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:00:54.795 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:54.826 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:00:54.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:54.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:00:54.935 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:00:54.982 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:55.008 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:00:55.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:55.012 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:00:55.113 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:00:55.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:54 smithi115 ceph-mon[72476]: pgmap v495: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 193 op/s 2024-03-20T09:00:55.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:54 smithi115 ceph-mon[72476]: 2.8 scrub ok 2024-03-20T09:00:55.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:54 smithi115 ceph-mon[72476]: 2.d scrub starts 2024-03-20T09:00:55.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:54 smithi115 ceph-mon[72476]: 2.b scrub ok 2024-03-20T09:00:55.158 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:54 smithi026 ceph-mon[68309]: pgmap v495: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 193 op/s 2024-03-20T09:00:55.158 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:54 smithi026 ceph-mon[68309]: 2.8 scrub ok 2024-03-20T09:00:55.158 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:54 smithi026 ceph-mon[68309]: 2.d scrub starts 2024-03-20T09:00:55.158 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:54 smithi026 ceph-mon[68309]: 2.b scrub ok 2024-03-20T09:00:55.164 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:55.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:54 smithi122 ceph-mon[73495]: pgmap v495: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 193 op/s 2024-03-20T09:00:55.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:54 smithi122 ceph-mon[73495]: 2.8 scrub ok 2024-03-20T09:00:55.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:54 smithi122 ceph-mon[73495]: 2.d scrub starts 2024-03-20T09:00:55.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:54 smithi122 ceph-mon[73495]: 2.b scrub ok 2024-03-20T09:00:55.283 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:00:55.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:55.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:00:55.387 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:00:55.417 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:55.731 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:00:55.733 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":495,"stamp":"2024-03-20T09:00:53.507607+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4939973176,"num_objects":75372,"num_object_clones":0,"num_object_copies":226116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75372,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":162377,"num_write_kb":7604987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160706,"ondisk_log_size":160706,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":15348284,"kb_used_data":14536200,"kb_used_omap":18,"kb_used_meta":811309,"kb_avail":1109593540,"statfs":{"total":1151940427776,"available":1136223784960,"internally_reserved":0,"allocated":14885068800,"data_stored":14881802266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":830780731},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,5,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-101515264,"num_objects":-1549,"num_object_clones":0,"num_object_copies":-4647,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1549,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003295"},"pg_stats":[{"pgid":"2.f","version":"165'10007","reported_seq":10234,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-03-20T09:00:51.509340+0000","last_change":"2024-03-20T09:00:48.749086+0000","last_active":"2024-03-20T09:00:51.509340+0000","last_peered":"2024-03-20T09:00:51.509340+0000","last_clean":"2024-03-20T09:00:51.509340+0000","last_became_active":"2024-03-20T08:58:10.027677+0000","last_became_peered":"2024-03-20T08:58:10.027677+0000","last_unstale":"2024-03-20T09:00:51.509340+0000","last_undegraded":"2024-03-20T09:00:51.509340+0000","last_fullsized":"2024-03-20T09:00:51.509340+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'9900","last_scrub_stamp":"2024-03-20T09:00:48.748963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:00:48.748963+0000","objects_scrubbed":4676,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:12.710297+0000","scrub_duration":532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299433984,"num_objects":4569,"num_object_clones":0,"num_object_copies":13707,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10007,"num_write_kb":466432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"165'10115","reported_seq":10324,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-03-20T09:00:49.647499+0000","last_change":"2024-03-20T09:00:15.743746+0000","last_active":"2024-03-20T09:00:49.647499+0000","last_peered":"2024-03-20T09:00:49.647499+0000","last_clean":"2024-03-20T09:00:49.647499+0000","last_became_active":"2024-03-20T08:58:10.015946+0000","last_became_peered":"2024-03-20T08:58:10.015946+0000","last_unstale":"2024-03-20T09:00:49.647499+0000","last_undegraded":"2024-03-20T09:00:49.647499+0000","last_fullsized":"2024-03-20T09:00:49.647499+0000","mapping_epoch":121,"log_start":"125'100","ondisk_log_start":"125'100","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'8757","last_scrub_stamp":"2024-03-20T09:00:15.743623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:00:15.743623+0000","objects_scrubbed":6227,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:01:36.603416+0000","scrub_duration":2344,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314507264,"num_objects":4799,"num_object_clones":0,"num_object_copies":14397,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10115,"num_write_kb":477248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"165'10127","reported_seq":10301,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-03-20T09:00:52.062826+0000","last_change":"2024-03-20T08:59:32.348108+0000","last_active":"2024-03-20T09:00:52.062826+0000","last_peered":"2024-03-20T09:00:52.062826+0000","last_clean":"2024-03-20T09:00:52.062826+0000","last_became_active":"2024-03-20T08:58:09.933090+0000","last_became_peered":"2024-03-20T08:58:09.933090+0000","last_unstale":"2024-03-20T09:00:52.062826+0000","last_undegraded":"2024-03-20T09:00:52.062826+0000","last_fullsized":"2024-03-20T09:00:52.062826+0000","mapping_epoch":121,"log_start":"125'100","ondisk_log_start":"125'100","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'6787","last_scrub_stamp":"2024-03-20T08:59:32.347948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T08:59:32.347948+0000","objects_scrubbed":6787,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:00:59.629856+0000","scrub_duration":635,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306774016,"num_objects":4681,"num_object_clones":0,"num_object_copies":14043,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10127,"num_write_kb":473856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"166'10084","reported_seq":10313,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-03-20T09:00:53.229988+0000","last_change":"2024-03-20T09:00:47.460883+0000","last_active":"2024-03-20T09:00:53.229988+0000","last_peered":"2024-03-20T09:00:53.229988+0000","last_clean":"2024-03-20T09:00:53.229988+0000","last_became_active":"2024-03-20T08:58:10.027595+0000","last_became_peered":"2024-03-20T08:58:10.027595+0000","last_unstale":"2024-03-20T09:00:53.229988+0000","last_undegraded":"2024-03-20T09:00:53.229988+0000","last_fullsized":"2024-03-20T09:00:53.229988+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'9875","last_scrub_stamp":"2024-03-20T09:00:47.460723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:00:47.460723+0000","objects_scrubbed":4831,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:01:54.542606+0000","scrub_duration":528,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302907392,"num_objects":4622,"num_object_clones":0,"num_object_copies":13866,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10084,"num_write_kb":470592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"165'10460","reported_seq":10691,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-03-20T09:00:52.079039+0000","last_change":"2024-03-20T09:00:49.806152+0000","last_active":"2024-03-20T09:00:52.079039+0000","last_peered":"2024-03-20T09:00:52.079039+0000","last_clean":"2024-03-20T09:00:52.079039+0000","last_became_active":"2024-03-20T08:58:09.929029+0000","last_became_peered":"2024-03-20T08:58:09.929029+0000","last_unstale":"2024-03-20T09:00:52.079039+0000","last_undegraded":"2024-03-20T09:00:52.079039+0000","last_fullsized":"2024-03-20T09:00:52.079039+0000","mapping_epoch":121,"log_start":"125'400","ondisk_log_start":"125'400","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'10360","last_scrub_stamp":"2024-03-20T09:00:49.805840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:00:49.805840+0000","objects_scrubbed":4895,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:14.860484+0000","scrub_duration":877,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312606720,"num_objects":4770,"num_object_clones":0,"num_object_copies":14310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10460,"num_write_kb":487360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":494,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-03-20T09:00:48.815807+0000","last_change":"2024-03-20T09:00:07.376260+0000","last_active":"2024-03-20T09:00:48.815807+0000","last_peered":"2024-03-20T09:00:48.815807+0000","last_clean":"2024-03-20T09:00:48.815807+0000","last_became_active":"2024-03-20T08:56:21.910528+0000","last_became_peered":"2024-03-20T08:56:21.910528+0000","last_unstale":"2024-03-20T09:00:48.815807+0000","last_undegraded":"2024-03-20T09:00:48.815807+0000","last_fullsized":"2024-03-20T09:00:48.815807+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:00:07.376166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:00:07.376166+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:01:31.145342+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"165'9950","reported_seq":10179,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-03-20T09:00:50.929523+0000","last_change":"2024-03-20T09:00:49.146954+0000","last_active":"2024-03-20T09:00:50.929523+0000","last_peered":"2024-03-20T09:00:50.929523+0000","last_clean":"2024-03-20T09:00:50.929523+0000","last_became_active":"2024-03-20T08:58:09.933985+0000","last_became_peered":"2024-03-20T08:58:09.933985+0000","last_unstale":"2024-03-20T09:00:50.929523+0000","last_undegraded":"2024-03-20T09:00:50.929523+0000","last_fullsized":"2024-03-20T09:00:50.929523+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'9863","last_scrub_stamp":"2024-03-20T09:00:49.146848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:00:49.146848+0000","objects_scrubbed":4833,"log_size":9950,"log_dups_size":0,"ondisk_log_size":9950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:01:49.401312+0000","scrub_duration":527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311033856,"num_objects":4746,"num_object_clones":0,"num_object_copies":14238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9950,"num_write_kb":470272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"165'10153","reported_seq":10383,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-03-20T09:00:51.443484+0000","last_change":"2024-03-20T09:00:45.728660+0000","last_active":"2024-03-20T09:00:51.443484+0000","last_peered":"2024-03-20T09:00:51.443484+0000","last_clean":"2024-03-20T09:00:51.443484+0000","last_became_active":"2024-03-20T08:58:10.027686+0000","last_became_peered":"2024-03-20T08:58:10.027686+0000","last_unstale":"2024-03-20T09:00:51.443484+0000","last_undegraded":"2024-03-20T09:00:51.443484+0000","last_fullsized":"2024-03-20T09:00:51.443484+0000","mapping_epoch":121,"log_start":"125'100","ondisk_log_start":"125'100","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'10042","last_scrub_stamp":"2024-03-20T09:00:45.728563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:00:45.728563+0000","objects_scrubbed":4802,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:01:51.770911+0000","scrub_duration":527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307429376,"num_objects":4691,"num_object_clones":0,"num_object_copies":14073,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10153,"num_write_kb":475008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"165'10152","reported_seq":10327,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-03-20T09:00:49.690620+0000","last_change":"2024-03-20T08:59:45.000055+0000","last_active":"2024-03-20T09:00:49.690620+0000","last_peered":"2024-03-20T09:00:49.690620+0000","last_clean":"2024-03-20T09:00:49.690620+0000","last_became_active":"2024-03-20T08:58:10.027648+0000","last_became_peered":"2024-03-20T08:58:10.027648+0000","last_unstale":"2024-03-20T09:00:49.690620+0000","last_undegraded":"2024-03-20T09:00:49.690620+0000","last_fullsized":"2024-03-20T09:00:49.690620+0000","mapping_epoch":121,"log_start":"125'100","ondisk_log_start":"125'100","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"148'7022","last_scrub_stamp":"2024-03-20T08:59:44.999792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T08:59:44.999792+0000","objects_scrubbed":6977,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:00:56.056731+0000","scrub_duration":1861,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313524224,"num_objects":4784,"num_object_clones":0,"num_object_copies":14352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10152,"num_write_kb":477952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"165'10106","reported_seq":10280,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-03-20T09:00:49.801689+0000","last_change":"2024-03-20T08:59:34.434294+0000","last_active":"2024-03-20T09:00:49.801689+0000","last_peered":"2024-03-20T09:00:49.801689+0000","last_clean":"2024-03-20T09:00:49.801689+0000","last_became_active":"2024-03-20T08:58:09.931796+0000","last_became_peered":"2024-03-20T08:58:09.931796+0000","last_unstale":"2024-03-20T09:00:49.801689+0000","last_undegraded":"2024-03-20T09:00:49.801689+0000","last_fullsized":"2024-03-20T09:00:49.801689+0000","mapping_epoch":121,"log_start":"125'100","ondisk_log_start":"125'100","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'6788","last_scrub_stamp":"2024-03-20T08:59:34.434106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T08:59:34.434106+0000","objects_scrubbed":6788,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:00:54.013460+0000","scrub_duration":729,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313655296,"num_objects":4786,"num_object_clones":0,"num_object_copies":14358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10106,"num_write_kb":476544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"165'10055","reported_seq":10228,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-03-20T09:00:50.916857+0000","last_change":"2024-03-20T08:59:28.967499+0000","last_active":"2024-03-20T09:00:50.916857+0000","last_peered":"2024-03-20T09:00:50.916857+0000","last_clean":"2024-03-20T09:00:50.916857+0000","last_became_active":"2024-03-20T08:58:10.016062+0000","last_became_peered":"2024-03-20T08:58:10.016062+0000","last_unstale":"2024-03-20T09:00:50.916857+0000","last_undegraded":"2024-03-20T09:00:50.916857+0000","last_fullsized":"2024-03-20T09:00:50.916857+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'6713","last_scrub_stamp":"2024-03-20T08:59:28.967391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T08:59:28.967391+0000","objects_scrubbed":6713,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":687,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304676864,"num_objects":4649,"num_object_clones":0,"num_object_copies":13947,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10055,"num_write_kb":470528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"165'10333","reported_seq":10565,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-03-20T09:00:49.723221+0000","last_change":"2024-03-20T09:00:48.795741+0000","last_active":"2024-03-20T09:00:49.723221+0000","last_peered":"2024-03-20T09:00:49.723221+0000","last_clean":"2024-03-20T09:00:49.723221+0000","last_became_active":"2024-03-20T08:58:10.027614+0000","last_became_peered":"2024-03-20T08:58:10.027614+0000","last_unstale":"2024-03-20T09:00:49.723221+0000","last_undegraded":"2024-03-20T09:00:49.723221+0000","last_fullsized":"2024-03-20T09:00:49.723221+0000","mapping_epoch":121,"log_start":"125'300","ondisk_log_start":"125'300","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'10297","last_scrub_stamp":"2024-03-20T09:00:48.795610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:00:48.795610+0000","objects_scrubbed":4877,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:10.924576+0000","scrub_duration":504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":317259776,"num_objects":4841,"num_object_clones":0,"num_object_copies":14523,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10333,"num_write_kb":485568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"163'10136","reported_seq":10292,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-03-20T09:00:45.422374+0000","last_change":"2024-03-20T08:59:30.565395+0000","last_active":"2024-03-20T09:00:45.422374+0000","last_peered":"2024-03-20T09:00:45.422374+0000","last_clean":"2024-03-20T09:00:45.422374+0000","last_became_active":"2024-03-20T08:58:10.027712+0000","last_became_peered":"2024-03-20T08:58:10.027712+0000","last_unstale":"2024-03-20T09:00:45.422374+0000","last_undegraded":"2024-03-20T09:00:45.422374+0000","last_fullsized":"2024-03-20T09:00:45.422374+0000","mapping_epoch":121,"log_start":"125'100","ondisk_log_start":"125'100","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'6780","last_scrub_stamp":"2024-03-20T08:59:30.565263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T08:59:30.565263+0000","objects_scrubbed":6780,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:00:49.334226+0000","scrub_duration":687,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314834944,"num_objects":4804,"num_object_clones":0,"num_object_copies":14412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10136,"num_write_kb":478080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"165'10141","reported_seq":10315,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-03-20T09:00:51.533443+0000","last_change":"2024-03-20T08:59:27.948008+0000","last_active":"2024-03-20T09:00:51.533443+0000","last_peered":"2024-03-20T09:00:51.533443+0000","last_clean":"2024-03-20T09:00:51.533443+0000","last_became_active":"2024-03-20T08:58:09.935063+0000","last_became_peered":"2024-03-20T08:58:09.935063+0000","last_unstale":"2024-03-20T09:00:51.533443+0000","last_undegraded":"2024-03-20T09:00:51.533443+0000","last_fullsized":"2024-03-20T09:00:51.533443+0000","mapping_epoch":121,"log_start":"125'100","ondisk_log_start":"125'100","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'6763","last_scrub_stamp":"2024-03-20T08:59:27.947886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T08:59:27.947886+0000","objects_scrubbed":6763,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":722,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307036160,"num_objects":4685,"num_object_clones":0,"num_object_copies":14055,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10141,"num_write_kb":474432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"163'13808","reported_seq":14176,"reported_epoch":164,"state":"active+clean","last_fresh":"2024-03-20T09:00:45.422441+0000","last_change":"2024-03-20T09:00:09.317786+0000","last_active":"2024-03-20T09:00:45.422441+0000","last_peered":"2024-03-20T09:00:45.422441+0000","last_clean":"2024-03-20T09:00:45.422441+0000","last_became_active":"2024-03-20T08:58:10.016313+0000","last_became_peered":"2024-03-20T08:58:10.016313+0000","last_unstale":"2024-03-20T09:00:45.422441+0000","last_undegraded":"2024-03-20T09:00:45.422441+0000","last_fullsized":"2024-03-20T09:00:45.422441+0000","mapping_epoch":121,"log_start":"133'3800","ondisk_log_start":"133'3800","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'12090","last_scrub_stamp":"2024-03-20T09:00:09.317614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:00:09.317614+0000","objects_scrubbed":6610,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:01:20.580827+0000","scrub_duration":2021,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316735488,"num_objects":4833,"num_object_clones":0,"num_object_copies":14499,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9935,"num_write_kb":472576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"165'10174","reported_seq":10348,"reported_epoch":165,"state":"active+clean","last_fresh":"2024-03-20T09:00:52.819973+0000","last_change":"2024-03-20T08:59:28.482394+0000","last_active":"2024-03-20T09:00:52.819973+0000","last_peered":"2024-03-20T09:00:52.819973+0000","last_clean":"2024-03-20T09:00:52.819973+0000","last_became_active":"2024-03-20T08:58:10.027826+0000","last_became_peered":"2024-03-20T08:58:10.027826+0000","last_unstale":"2024-03-20T09:00:52.819973+0000","last_undegraded":"2024-03-20T09:00:52.819973+0000","last_fullsized":"2024-03-20T09:00:52.819973+0000","mapping_epoch":121,"log_start":"125'100","ondisk_log_start":"125'100","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'6682","last_scrub_stamp":"2024-03-20T08:59:28.482236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T08:59:28.482236+0000","objects_scrubbed":6682,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":650,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294453272,"num_objects":4494,"num_object_clones":0,"num_object_copies":13482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4494,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10174,"num_write_kb":469313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"166'10268","reported_seq":10459,"reported_epoch":166,"state":"active+clean+scrubbing","last_fresh":"2024-03-20T09:00:53.259985+0000","last_change":"2024-03-20T09:00:52.866812+0000","last_active":"2024-03-20T09:00:53.259985+0000","last_peered":"2024-03-20T09:00:53.259985+0000","last_clean":"2024-03-20T09:00:53.259985+0000","last_became_active":"2024-03-20T08:58:10.027586+0000","last_became_peered":"2024-03-20T08:58:10.027586+0000","last_unstale":"2024-03-20T09:00:53.259985+0000","last_undegraded":"2024-03-20T09:00:53.259985+0000","last_fullsized":"2024-03-20T09:00:53.259985+0000","mapping_epoch":121,"log_start":"125'200","ondisk_log_start":"125'200","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'6609","last_scrub_stamp":"2024-03-20T08:59:23.634655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T08:59:23.634655+0000","objects_scrubbed":1199,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"scrubbing for 0s","scrub_duration":2034,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302514176,"num_objects":4616,"num_object_clones":0,"num_object_copies":13848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10268,"num_write_kb":476288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":4939382808,"num_objects":75370,"num_object_clones":0,"num_object_copies":226110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75370,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162196,"num_write_kb":7602049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14807543808,"data_stored":14807531592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160569,"ondisk_log_size":160569,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860474,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":989180,"kb_used_data":926064,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92755972,"statfs":{"total":95995035648,"available":94982115328,"internally_reserved":0,"allocated":948289536,"data_stored":948019831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":64616900},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":89,"seq":382252089407,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":955456,"kb_used_data":910388,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":92789696,"statfs":{"total":95995035648,"available":95016648704,"internally_reserved":0,"allocated":932237312,"data_stored":931966613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":46135744},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":80,"seq":343597383751,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1870208,"kb_used_data":1776248,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":91874944,"statfs":{"total":95995035648,"available":94079942656,"internally_reserved":0,"allocated":1818877952,"data_stored":1818603181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":95877568},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":70,"seq":300647710795,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1266364,"kb_used_data":1199028,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92478788,"statfs":{"total":95995035648,"available":94698278912,"internally_reserved":0,"allocated":1227804672,"data_stored":1227533993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":60,"seq":257698037840,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1592196,"kb_used_data":1524732,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92152956,"statfs":{"total":95995035648,"available":94364626944,"internally_reserved":0,"allocated":1561325568,"data_stored":1561051337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332181,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":660616,"kb_used_data":615548,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93084536,"statfs":{"total":95995035648,"available":95318564864,"internally_reserved":0,"allocated":630321152,"data_stored":630046921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574959,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1634944,"kb_used_data":1545460,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":92110208,"statfs":{"total":95995035648,"available":94320852992,"internally_reserved":0,"allocated":1582551040,"data_stored":1582280361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91617734},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738486,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1591036,"kb_used_data":1497332,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":92154116,"statfs":{"total":95995035648,"available":94365814784,"internally_reserved":0,"allocated":1533267968,"data_stored":1532997289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411433,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1260860,"kb_used_data":1193588,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92484292,"statfs":{"total":95995035648,"available":94703915008,"internally_reserved":0,"allocated":1222234112,"data_stored":1221963433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149796,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1921728,"kb_used_data":1836408,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":91823424,"statfs":{"total":95995035648,"available":94027186176,"internally_reserved":0,"allocated":1880481792,"data_stored":1880207041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":87357895},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953568,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":953668,"kb_used_data":904440,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92791484,"statfs":{"total":95995035648,"available":95018479616,"internally_reserved":0,"allocated":926146560,"data_stored":925871809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659226,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":652028,"kb_used_data":606964,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93093124,"statfs":{"total":95995035648,"available":95327358976,"internally_reserved":0,"allocated":621531136,"data_stored":621260457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1527316480,"data_stored":1527316480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1574895616,"data_stored":1574895616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1216217088,"data_stored":1216217088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1871777792,"data_stored":1871773720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":918884352,"data_stored":918880280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":619315200,"data_stored":619315200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":628162560,"data_stored":628162560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1553399808,"data_stored":1553399808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1222180864,"data_stored":1222180864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1797918720,"data_stored":1797914648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":930414592,"data_stored":930414592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":947060736,"data_stored":947060736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:00:55.735 INFO:tasks.thrashosds.thrasher:Setting 2.0 to [2, 1, 0, 7, 4, 5] 2024-03-20T09:00:55.735 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.0', '2', '1', '0', '7', '4', '5'] 2024-03-20T09:00:55.735 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pg-upmap-items 2.0 2 1 0 7 4 5 2024-03-20T09:00:55.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:55 smithi026 ceph-mon[68309]: 2.e scrub ok 2024-03-20T09:00:55.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:55 smithi026 ceph-mon[68309]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:00:55.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:55 smithi026 ceph-mon[68309]: 2.7 scrub starts 2024-03-20T09:00:55.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:55 smithi026 ceph-mon[68309]: 2.d scrub ok 2024-03-20T09:00:56.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:55 smithi115 ceph-mon[72476]: 2.e scrub ok 2024-03-20T09:00:56.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:55 smithi115 ceph-mon[72476]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:00:56.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:55 smithi115 ceph-mon[72476]: 2.7 scrub starts 2024-03-20T09:00:56.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:55 smithi115 ceph-mon[72476]: 2.d scrub ok 2024-03-20T09:00:56.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:55 smithi122 ceph-mon[73495]: 2.e scrub ok 2024-03-20T09:00:56.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:55 smithi122 ceph-mon[73495]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:00:56.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:55 smithi122 ceph-mon[73495]: 2.7 scrub starts 2024-03-20T09:00:56.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:55 smithi122 ceph-mon[73495]: 2.d scrub ok 2024-03-20T09:00:56.267 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:00:56.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:56.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:00:56.371 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:00:56.406 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:56.424 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:00:56.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:56.427 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:00:56.455 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:00:56.528 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:00:56.556 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:56.633 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:00:56.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:56.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:00:56.739 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:00:56.768 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:56.881 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:00:56.949 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:00:56.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:56.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:00:57.055 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:00:57.088 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:57.109 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:00:57.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:57.112 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:00:57.134 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:00:57.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:56 smithi115 ceph-mon[72476]: 2.7 scrub ok 2024-03-20T09:00:57.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:56 smithi115 ceph-mon[72476]: pgmap v496: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 383 op/s 2024-03-20T09:00:57.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:56 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:00:57.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:56 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:00:57.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:56 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/225984823' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:00:57.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:56 smithi122 ceph-mon[73495]: 2.7 scrub ok 2024-03-20T09:00:57.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:56 smithi122 ceph-mon[73495]: pgmap v496: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 383 op/s 2024-03-20T09:00:57.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:56 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:00:57.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:56 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:00:57.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:56 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/225984823' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:00:57.212 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:00:57.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:56 smithi026 ceph-mon[68309]: 2.7 scrub ok 2024-03-20T09:00:57.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:56 smithi026 ceph-mon[68309]: pgmap v496: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 383 op/s 2024-03-20T09:00:57.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:56 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:00:57.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:56 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:00:57.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:56 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/225984823' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:00:57.241 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:57.261 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:00:57.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:57.264 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:00:57.365 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:00:57.400 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:57.422 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:00:57.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:57.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:00:57.526 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:00:57.556 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:57.575 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:00:57.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:57.578 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:00:57.679 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:00:57.711 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:57.732 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:00:57.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:57.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:00:57.836 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:00:57.866 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:57.990 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:00:57.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:57.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:00:58.094 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:00:58.124 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:57 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:00:58.124 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:57 smithi122 ceph-mon[73495]: Cluster is now healthy 2024-03-20T09:00:58.124 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:57 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/225984823' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:00:58.124 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:57 smithi122 ceph-mon[73495]: osdmap e167: 12 total, 12 up, 12 in 2024-03-20T09:00:58.126 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:58.147 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:00:58.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:58.150 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:00:58.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:57 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:00:58.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:57 smithi115 ceph-mon[72476]: Cluster is now healthy 2024-03-20T09:00:58.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:57 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/225984823' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:00:58.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:57 smithi115 ceph-mon[72476]: osdmap e167: 12 total, 12 up, 12 in 2024-03-20T09:00:58.203 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:57 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:00:58.204 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:57 smithi026 ceph-mon[68309]: Cluster is now healthy 2024-03-20T09:00:58.204 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:57 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/225984823' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:00:58.204 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:57 smithi026 ceph-mon[68309]: osdmap e167: 12 total, 12 up, 12 in 2024-03-20T09:00:58.251 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:00:58.283 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:58.305 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:00:58.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:58.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:00:58.409 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:00:58.438 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:58.458 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:00:58.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:58.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:00:58.562 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:00:58.595 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:58.615 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:00:58.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:58.618 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:00:58.719 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:00:58.749 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:59.019 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:00:58 smithi026 ceph-mon[68309]: pgmap v498: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 547 op/s 2024-03-20T09:00:59.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:00:58 smithi115 ceph-mon[72476]: pgmap v498: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 547 op/s 2024-03-20T09:00:59.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:00:58 smithi122 ceph-mon[73495]: pgmap v498: 17 pgs: 2 active+clean+scrubbing, 15 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 547 op/s 2024-03-20T09:00:59.298 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:00:59.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:59.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:00:59.403 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:00:59.420 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:00:59.420 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:00:59.435 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:59.456 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:00:59.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:59.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:00:59.543 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:00:59.560 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:00:59.564 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:00:59.592 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:59.614 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:00:59.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:59.617 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:00:59.718 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:00:59.746 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:00:59.946 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-20T09:00:59.946 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:00:59.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:00:59.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:01:00.049 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:01:00.081 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:00.102 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:01:00.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:00.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:01:00.206 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:01:00.237 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:00.257 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:01:00.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:00.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:01:00.361 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:01:00.390 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:00.443 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:01:00.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:00.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:01:00.547 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:01:00.579 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:00.600 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:01:00.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:00.603 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:01:00.704 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:01:00.735 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:00.984 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:01:00.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:00.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:01:01.074 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:01:01.088 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:01:01.107 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:01:01.119 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:01.140 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:01:01.142 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:00 smithi122 ceph-mon[73495]: pgmap v499: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 566 op/s 2024-03-20T09:01:01.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:01.143 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:01:01.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:00 smithi115 ceph-mon[72476]: pgmap v499: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 566 op/s 2024-03-20T09:01:01.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:00 smithi026 ceph-mon[68309]: pgmap v499: 17 pgs: 1 active+clean+scrubbing, 16 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 566 op/s 2024-03-20T09:01:01.245 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:01:01.274 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:01.294 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:01:01.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:01.297 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:01:01.398 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:01:01.425 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:01.444 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:01:01.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:01.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:01:01.547 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:01:01.574 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:01.592 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:01:01.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:01.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:01:01.696 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:01:01.728 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:01.751 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:01:01.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:01.754 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:01:01.855 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:01:01.884 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:01 smithi026 ceph-mon[68309]: 2.2 scrub starts 2024-03-20T09:01:01.884 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:01 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3972766195' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.0", "id": [2, 1, 0, 7, 4, 5]}]: dispatch 2024-03-20T09:01:01.884 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:01 smithi026 ceph-mon[68309]: 2.2 scrub ok 2024-03-20T09:01:01.890 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:01.891 INFO:teuthology.orchestra.run.smithi026.stderr:set 2.0 pg_upmap_items mapping to [2->1,0->7,4->5] 2024-03-20T09:01:01.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:01 smithi122 ceph-mon[73495]: 2.2 scrub starts 2024-03-20T09:01:01.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:01 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3972766195' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.0", "id": [2, 1, 0, 7, 4, 5]}]: dispatch 2024-03-20T09:01:01.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:01 smithi122 ceph-mon[73495]: 2.2 scrub ok 2024-03-20T09:01:02.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:01 smithi115 ceph-mon[72476]: 2.2 scrub starts 2024-03-20T09:01:02.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:01 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3972766195' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.0", "id": [2, 1, 0, 7, 4, 5]}]: dispatch 2024-03-20T09:01:02.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:01 smithi115 ceph-mon[72476]: 2.2 scrub ok 2024-03-20T09:01:03.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:02 smithi115 ceph-mon[72476]: pgmap v500: 17 pgs: 17 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 455 op/s 2024-03-20T09:01:03.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:02 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3972766195' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.0", "id": [2, 1, 0, 7, 4, 5]}]': finished 2024-03-20T09:01:03.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:02 smithi115 ceph-mon[72476]: osdmap e168: 12 total, 12 up, 12 in 2024-03-20T09:01:03.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:02 smithi122 ceph-mon[73495]: pgmap v500: 17 pgs: 17 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 455 op/s 2024-03-20T09:01:03.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:02 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3972766195' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.0", "id": [2, 1, 0, 7, 4, 5]}]': finished 2024-03-20T09:01:03.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:02 smithi122 ceph-mon[73495]: osdmap e168: 12 total, 12 up, 12 in 2024-03-20T09:01:03.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:02 smithi026 ceph-mon[68309]: pgmap v500: 17 pgs: 17 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 455 op/s 2024-03-20T09:01:03.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:02 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3972766195' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.0", "id": [2, 1, 0, 7, 4, 5]}]': finished 2024-03-20T09:01:03.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:02 smithi026 ceph-mon[68309]: osdmap e168: 12 total, 12 up, 12 in 2024-03-20T09:01:04.245 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:01:04.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:04.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:01:04.272 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:01:04.292 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:01:04.311 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:01:04.331 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:01:04.350 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:01:04.351 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:01:04.372 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:01:04.381 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:04.396 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:01:04.417 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:01:04.437 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:01:04.459 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:01:04.480 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:01:04.481 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:01:04.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:04.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:01:04.503 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:01:04.506 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:01:04.506 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:01:04.506 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:01:04.506 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:01:04.506 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:01:04.506 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:01:04.506 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:01:04.506 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:01:04.506 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:01:04.506 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:01:04.506 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:01:04.585 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:01:04.617 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:04.638 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:01:04.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:04.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:01:04.742 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:01:04.774 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:05.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:04 smithi115 ceph-mon[72476]: pgmap v502: 17 pgs: 17 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 236 op/s 2024-03-20T09:01:05.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:04 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4102793074' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:01:05.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:04 smithi122 ceph-mon[73495]: pgmap v502: 17 pgs: 17 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 236 op/s 2024-03-20T09:01:05.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:04 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4102793074' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:01:05.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:04 smithi026 ceph-mon[68309]: pgmap v502: 17 pgs: 17 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 236 op/s 2024-03-20T09:01:05.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:04 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4102793074' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:01:05.624 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:01:05.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:05.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:01:05.729 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:01:05.759 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:05.900 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:01:06.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:05 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4025594891' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:01:06.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:05 smithi115 ceph-mon[72476]: 2.6 scrub starts 2024-03-20T09:01:06.189 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:01:06.191 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:05 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4025594891' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:01:06.191 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:05 smithi026 ceph-mon[68309]: 2.6 scrub starts 2024-03-20T09:01:06.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:06.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:01:06.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:05 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4025594891' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:01:06.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:05 smithi122 ceph-mon[73495]: 2.6 scrub starts 2024-03-20T09:01:06.293 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:01:06.325 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:06.346 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:01:06.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:06.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:01:06.451 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:01:06.481 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:06.501 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:01:06.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:06.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:01:06.605 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:01:06.638 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:06.659 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:01:06.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:06.662 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:01:06.763 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:01:06.794 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:06.814 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:01:06.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:06.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:01:06.918 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:01:06.949 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:06.994 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:01:06.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:06.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:01:07.027 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:01:07.027 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:01:07.027 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-03-20T09:01:07.027 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-03-20T09:01:07.027 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:01:07.098 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:01:07.127 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:06 smithi115 ceph-mon[72476]: pgmap v503: 17 pgs: 17 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 102 op/s 2024-03-20T09:01:07.127 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:06 smithi115 ceph-mon[72476]: 2.6 scrub ok 2024-03-20T09:01:07.127 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:06 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:01:07.127 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:06 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4025594891' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:01:07.128 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:06 smithi115 ceph-mon[72476]: osdmap e169: 12 total, 12 up, 12 in 2024-03-20T09:01:07.129 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:07.149 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:01:07.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:07.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:01:07.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:06 smithi122 ceph-mon[73495]: pgmap v503: 17 pgs: 17 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 102 op/s 2024-03-20T09:01:07.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:06 smithi122 ceph-mon[73495]: 2.6 scrub ok 2024-03-20T09:01:07.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:06 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:01:07.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:06 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4025594891' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:01:07.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:06 smithi122 ceph-mon[73495]: osdmap e169: 12 total, 12 up, 12 in 2024-03-20T09:01:07.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:06 smithi026 ceph-mon[68309]: pgmap v503: 17 pgs: 17 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 102 op/s 2024-03-20T09:01:07.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:06 smithi026 ceph-mon[68309]: 2.6 scrub ok 2024-03-20T09:01:07.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:06 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:01:07.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:06 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4025594891' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:01:07.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:06 smithi026 ceph-mon[68309]: osdmap e169: 12 total, 12 up, 12 in 2024-03-20T09:01:07.253 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:01:07.283 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:07.303 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:01:07.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:07.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:01:07.407 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:01:07.440 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:07.461 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:01:07.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:07.464 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:01:07.565 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:01:07.596 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:07.616 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:01:07.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:07.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:01:07.640 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:01:07.694 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-20T09:01:07.719 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:01:07.751 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:07.773 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:01:07.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:07.776 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:01:07.877 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:01:07.907 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:07.927 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:01:07.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:07.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:01:07.932 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:01:08.032 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:01:08.062 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:08.082 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:01:08.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:08.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:01:08.157 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:01:08.186 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:01:08.218 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:08.240 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:01:08.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:08.243 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:01:08.344 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:01:08.372 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:08.393 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:01:08.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:08.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:01:08.497 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:01:08.527 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:08.604 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:01:08.876 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:01:08.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:08.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:01:08.980 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:01:09.013 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:09.034 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:01:09.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:09.037 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:01:09.138 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:01:09.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:08 smithi115 ceph-mon[72476]: pgmap v505: 17 pgs: 17 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 14 op/s 2024-03-20T09:01:09.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:08 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:01:09.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:08 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:01:09.166 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:08 smithi122 ceph-mon[73495]: pgmap v505: 17 pgs: 17 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 14 op/s 2024-03-20T09:01:09.166 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:08 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:01:09.166 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:08 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:01:09.168 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:09.188 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:01:09.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:09.191 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:01:09.193 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:08 smithi026 ceph-mon[68309]: pgmap v505: 17 pgs: 17 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 14 op/s 2024-03-20T09:01:09.193 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:08 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:01:09.193 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:08 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:01:09.292 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:01:09.320 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:09.340 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:01:09.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:09.343 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:01:09.444 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:01:09.475 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:09.497 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:01:09.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:09.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:01:09.600 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:01:09.629 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:09.649 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:01:09.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:09.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:01:09.753 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:01:09.785 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:09.807 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:01:09.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:09.810 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:01:09.911 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:01:09.940 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:09.959 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:01:09.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:09.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:01:10.063 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:01:10.096 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:09 smithi122 ceph-mon[73495]: osdmap e170: 12 total, 12 up, 12 in 2024-03-20T09:01:10.096 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:09 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1445420551' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:01:10.098 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:10.132 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:01:10.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:10.139 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:01:10.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:09 smithi115 ceph-mon[72476]: osdmap e170: 12 total, 12 up, 12 in 2024-03-20T09:01:10.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:09 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1445420551' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:01:10.225 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:09 smithi026 ceph-mon[68309]: osdmap e170: 12 total, 12 up, 12 in 2024-03-20T09:01:10.226 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:09 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1445420551' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:01:10.240 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:01:10.275 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:10.301 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:01:10.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:10.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:01:10.408 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:01:10.496 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:10.497 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:01:10.497 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:01:10.497 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:01:10.497 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:01:10.498 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:01:10.498 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:01:10.498 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:01:10.498 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:01:10.498 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:01:10.498 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:01:10.498 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:01:10.498 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:01:10.498 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:01:10.498 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:01:10.498 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:01:10.498 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:01:10.499 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:01:10.499 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:01:10.499 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:01:10.499 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:01:10.499 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:01:10.499 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:01:10.499 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:01:10.499 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:01:10.720 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:01:10.720 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:01:10.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:10.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:01:10.823 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:01:10.854 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:10.889 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:01:10.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:10.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:01:10.995 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:01:11.019 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:10 smithi026 ceph-mon[68309]: pgmap v507: 17 pgs: 17 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:01:11.021 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-03-20T09:01:11.022 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:01:11.022 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:01:11.023 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:11.043 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:01:11.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:11.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:01:11.152 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:01:11.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:10 smithi115 ceph-mon[72476]: pgmap v507: 17 pgs: 17 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:01:11.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:10 smithi122 ceph-mon[73495]: pgmap v507: 17 pgs: 17 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:01:11.189 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:11.211 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:01:11.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:11.214 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:01:11.315 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:01:11.361 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:11.385 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:01:11.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:11.388 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:01:11.489 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:01:11.546 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:11.636 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:01:11.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:11.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:01:11.744 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:01:11.783 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:11.840 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:01:11.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:11.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:01:11.945 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:01:11.972 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:11.994 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:01:11.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:11.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:01:12.099 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:01:12.148 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:12.163 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:01:12.163 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:01:12.188 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:01:12.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:12.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:01:12.306 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:01:12.370 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:12.397 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:01:12.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:12.401 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:01:12.502 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:01:12.561 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:12.632 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:01:12.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:12.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:01:12.747 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:01:12.788 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:12.814 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:01:12.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:12.820 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:01:12.922 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:01:12.957 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:12.983 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:01:12.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:12.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:01:13.087 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:01:13.148 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:13.186 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:01:13.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:13.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:01:13.189 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:01:13.192 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:12 smithi026 ceph-mon[68309]: pgmap v508: 17 pgs: 17 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 12 op/s 2024-03-20T09:01:13.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:12 smithi122 ceph-mon[73495]: pgmap v508: 17 pgs: 17 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 12 op/s 2024-03-20T09:01:13.291 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:01:13.317 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:12 smithi115 ceph-mon[72476]: pgmap v508: 17 pgs: 17 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 12 op/s 2024-03-20T09:01:13.319 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:13.340 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:01:13.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:13.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:01:13.429 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:01:13.446 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:01:13.548 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:13.711 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:01:13.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:13.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:01:13.817 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:01:13.850 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:13.884 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:01:13.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:13.895 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:01:13.996 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:01:14.061 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:14.154 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:01:14.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:14.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:01:14.205 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-20T09:01:14.260 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:01:14.294 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:14.322 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:01:14.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:14.328 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:01:14.429 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:01:14.455 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:14.473 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:01:14.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:14.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:01:14.577 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:01:14.617 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:14.646 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:01:14.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:14.651 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:01:14.752 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:01:14.780 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:14.806 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:01:14.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:14.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:01:14.915 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:01:14.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:14 smithi026 ceph-mon[68309]: pgmap v509: 17 pgs: 17 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 125 op/s 2024-03-20T09:01:14.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:14 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1135002007' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:01:14.979 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:01:14.989 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:15.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:14 smithi115 ceph-mon[72476]: pgmap v509: 17 pgs: 17 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 125 op/s 2024-03-20T09:01:15.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:14 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1135002007' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:01:15.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:14 smithi122 ceph-mon[73495]: pgmap v509: 17 pgs: 17 active+clean; 4.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 125 op/s 2024-03-20T09:01:15.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:14 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1135002007' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:01:15.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:15 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:01:15.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:15 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1135002007' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:01:15.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:15 smithi026 ceph-mon[68309]: osdmap e171: 12 total, 12 up, 12 in 2024-03-20T09:01:16.072 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:01:16.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:15 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:01:16.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:15 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1135002007' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:01:16.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:15 smithi115 ceph-mon[72476]: osdmap e171: 12 total, 12 up, 12 in 2024-03-20T09:01:16.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:15 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:01:16.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:15 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1135002007' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:01:16.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:15 smithi122 ceph-mon[73495]: osdmap e171: 12 total, 12 up, 12 in 2024-03-20T09:01:16.796 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:01:16.804 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:01:16.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:16.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:01:16.908 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:01:16.974 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:17.120 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:01:17.123 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:01:17.224 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:16 smithi026 ceph-mon[68309]: pgmap v511: 17 pgs: 17 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 306 op/s 2024-03-20T09:01:17.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:16 smithi115 ceph-mon[72476]: pgmap v511: 17 pgs: 17 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 306 op/s 2024-03-20T09:01:17.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:16 smithi122 ceph-mon[73495]: pgmap v511: 17 pgs: 17 active+clean; 4.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 306 op/s 2024-03-20T09:01:18.105 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:01:18.185 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":511,"stamp":"2024-03-20T09:01:15.519318+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4555211320,"num_objects":69501,"num_object_clones":0,"num_object_copies":208503,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69501,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":168248,"num_write_kb":7604987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160877,"ondisk_log_size":160877,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":14292248,"kb_used_data":13392836,"kb_used_omap":18,"kb_used_meta":898733,"kb_avail":1110649576,"statfs":{"total":1151940427776,"available":1137305165824,"internally_reserved":0,"allocated":13714264064,"data_stored":13711308548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":920302907},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-160628736,"num_objects":-2451,"num_object_clones":0,"num_object_copies":-7353,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2451,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.008940"},"pg_stats":[{"pgid":"2.f","version":"170'10258","reported_seq":10494,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-20T09:01:11.552385+0000","last_change":"2024-03-20T09:00:48.749086+0000","last_active":"2024-03-20T09:01:11.552385+0000","last_peered":"2024-03-20T09:01:11.552385+0000","last_clean":"2024-03-20T09:01:11.552385+0000","last_became_active":"2024-03-20T08:58:10.027677+0000","last_became_peered":"2024-03-20T08:58:10.027677+0000","last_unstale":"2024-03-20T09:01:11.552385+0000","last_undegraded":"2024-03-20T09:01:11.552385+0000","last_fullsized":"2024-03-20T09:01:11.552385+0000","mapping_epoch":121,"log_start":"125'200","ondisk_log_start":"125'200","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'9900","last_scrub_stamp":"2024-03-20T09:00:48.748963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:00:48.748963+0000","objects_scrubbed":4676,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:12.710297+0000","scrub_duration":532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282984448,"num_objects":4318,"num_object_clones":0,"num_object_copies":12954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10258,"num_write_kb":466432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"170'10647","reported_seq":10866,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-20T09:01:14.709371+0000","last_change":"2024-03-20T09:00:15.743746+0000","last_active":"2024-03-20T09:01:14.709371+0000","last_peered":"2024-03-20T09:01:14.709371+0000","last_clean":"2024-03-20T09:01:14.709371+0000","last_became_active":"2024-03-20T08:58:10.015946+0000","last_became_peered":"2024-03-20T08:58:10.015946+0000","last_unstale":"2024-03-20T09:01:14.709371+0000","last_undegraded":"2024-03-20T09:01:14.709371+0000","last_fullsized":"2024-03-20T09:01:14.709371+0000","mapping_epoch":121,"log_start":"125'600","ondisk_log_start":"125'600","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'8757","last_scrub_stamp":"2024-03-20T09:00:15.743623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:00:15.743623+0000","objects_scrubbed":6227,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:01:36.603416+0000","scrub_duration":2344,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279642112,"num_objects":4267,"num_object_clones":0,"num_object_copies":12801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10647,"num_write_kb":477248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"170'10399","reported_seq":10636,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-20T09:01:12.054366+0000","last_change":"2024-03-20T09:01:01.312938+0000","last_active":"2024-03-20T09:01:12.054366+0000","last_peered":"2024-03-20T09:01:12.054366+0000","last_clean":"2024-03-20T09:01:12.054366+0000","last_became_active":"2024-03-20T08:58:09.933090+0000","last_became_peered":"2024-03-20T08:58:09.933090+0000","last_unstale":"2024-03-20T09:01:12.054366+0000","last_undegraded":"2024-03-20T09:01:12.054366+0000","last_fullsized":"2024-03-20T09:01:12.054366+0000","mapping_epoch":121,"log_start":"125'300","ondisk_log_start":"125'300","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'10289","last_scrub_stamp":"2024-03-20T09:01:01.312817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:01:01.312817+0000","objects_scrubbed":4519,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:08.660376+0000","scrub_duration":435,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288948224,"num_objects":4409,"num_object_clones":0,"num_object_copies":13227,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10399,"num_write_kb":473856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"170'10374","reported_seq":10611,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-20T09:01:13.244522+0000","last_change":"2024-03-20T09:00:47.460883+0000","last_active":"2024-03-20T09:01:13.244522+0000","last_peered":"2024-03-20T09:01:13.244522+0000","last_clean":"2024-03-20T09:01:13.244522+0000","last_became_active":"2024-03-20T08:58:10.027595+0000","last_became_peered":"2024-03-20T08:58:10.027595+0000","last_unstale":"2024-03-20T09:01:13.244522+0000","last_undegraded":"2024-03-20T09:01:13.244522+0000","last_fullsized":"2024-03-20T09:01:13.244522+0000","mapping_epoch":121,"log_start":"125'300","ondisk_log_start":"125'300","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'9875","last_scrub_stamp":"2024-03-20T09:00:47.460723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:00:47.460723+0000","objects_scrubbed":4831,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:01:54.542606+0000","scrub_duration":528,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283901952,"num_objects":4332,"num_object_clones":0,"num_object_copies":12996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10374,"num_write_kb":470592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"170'10720","reported_seq":10961,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-20T09:01:12.042883+0000","last_change":"2024-03-20T09:00:49.806152+0000","last_active":"2024-03-20T09:01:12.042883+0000","last_peered":"2024-03-20T09:01:12.042883+0000","last_clean":"2024-03-20T09:01:12.042883+0000","last_became_active":"2024-03-20T08:58:09.929029+0000","last_became_peered":"2024-03-20T08:58:09.929029+0000","last_unstale":"2024-03-20T09:01:12.042883+0000","last_undegraded":"2024-03-20T09:01:12.042883+0000","last_fullsized":"2024-03-20T09:01:12.042883+0000","mapping_epoch":121,"log_start":"125'700","ondisk_log_start":"125'700","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'10360","last_scrub_stamp":"2024-03-20T09:00:49.805840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:00:49.805840+0000","objects_scrubbed":4895,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:14.860484+0000","scrub_duration":877,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295567360,"num_objects":4510,"num_object_clones":0,"num_object_copies":13530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10720,"num_write_kb":487360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":504,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-20T09:01:09.523100+0000","last_change":"2024-03-20T09:00:07.376260+0000","last_active":"2024-03-20T09:01:09.523100+0000","last_peered":"2024-03-20T09:01:09.523100+0000","last_clean":"2024-03-20T09:01:09.523100+0000","last_became_active":"2024-03-20T08:56:21.910528+0000","last_became_peered":"2024-03-20T08:56:21.910528+0000","last_unstale":"2024-03-20T09:01:09.523100+0000","last_undegraded":"2024-03-20T09:01:09.523100+0000","last_fullsized":"2024-03-20T09:01:09.523100+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:00:07.376166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:00:07.376166+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:01:31.145342+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"170'10225","reported_seq":10464,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-20T09:01:10.888234+0000","last_change":"2024-03-20T09:00:49.146954+0000","last_active":"2024-03-20T09:01:10.888234+0000","last_peered":"2024-03-20T09:01:10.888234+0000","last_clean":"2024-03-20T09:01:10.888234+0000","last_became_active":"2024-03-20T08:58:09.933985+0000","last_became_peered":"2024-03-20T08:58:09.933985+0000","last_unstale":"2024-03-20T09:01:10.888234+0000","last_undegraded":"2024-03-20T09:01:10.888234+0000","last_fullsized":"2024-03-20T09:01:10.888234+0000","mapping_epoch":121,"log_start":"125'200","ondisk_log_start":"125'200","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'9863","last_scrub_stamp":"2024-03-20T09:00:49.146848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:00:49.146848+0000","objects_scrubbed":4833,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:01:49.401312+0000","scrub_duration":527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293011456,"num_objects":4471,"num_object_clones":0,"num_object_copies":13413,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10225,"num_write_kb":470272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"170'10413","reported_seq":10653,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-20T09:01:11.520865+0000","last_change":"2024-03-20T09:00:45.728660+0000","last_active":"2024-03-20T09:01:11.520865+0000","last_peered":"2024-03-20T09:01:11.520865+0000","last_clean":"2024-03-20T09:01:11.520865+0000","last_became_active":"2024-03-20T08:58:10.027686+0000","last_became_peered":"2024-03-20T08:58:10.027686+0000","last_unstale":"2024-03-20T09:01:11.520865+0000","last_undegraded":"2024-03-20T09:01:11.520865+0000","last_fullsized":"2024-03-20T09:01:11.520865+0000","mapping_epoch":121,"log_start":"125'400","ondisk_log_start":"125'400","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'10042","last_scrub_stamp":"2024-03-20T09:00:45.728563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:00:45.728563+0000","objects_scrubbed":4802,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:01:51.770911+0000","scrub_duration":527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290390016,"num_objects":4431,"num_object_clones":0,"num_object_copies":13293,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10413,"num_write_kb":475008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"170'10685","reported_seq":10923,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-20T09:01:14.616140+0000","last_change":"2024-03-20T09:01:05.787896+0000","last_active":"2024-03-20T09:01:14.616140+0000","last_peered":"2024-03-20T09:01:14.616140+0000","last_clean":"2024-03-20T09:01:14.616140+0000","last_became_active":"2024-03-20T08:58:10.027648+0000","last_became_peered":"2024-03-20T08:58:10.027648+0000","last_unstale":"2024-03-20T09:01:14.616140+0000","last_undegraded":"2024-03-20T09:01:14.616140+0000","last_fullsized":"2024-03-20T09:01:14.616140+0000","mapping_epoch":121,"log_start":"125'600","ondisk_log_start":"125'600","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'10448","last_scrub_stamp":"2024-03-20T09:01:05.787737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:01:05.787737+0000","objects_scrubbed":4488,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:08.404662+0000","scrub_duration":434,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278593536,"num_objects":4251,"num_object_clones":0,"num_object_copies":12753,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10685,"num_write_kb":477952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"170'10634","reported_seq":10873,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-20T09:01:14.852097+0000","last_change":"2024-03-20T09:00:55.481213+0000","last_active":"2024-03-20T09:01:14.852097+0000","last_peered":"2024-03-20T09:01:14.852097+0000","last_clean":"2024-03-20T09:01:14.852097+0000","last_became_active":"2024-03-20T08:58:09.931796+0000","last_became_peered":"2024-03-20T08:58:09.931796+0000","last_unstale":"2024-03-20T09:01:14.852097+0000","last_undegraded":"2024-03-20T09:01:14.852097+0000","last_fullsized":"2024-03-20T09:01:14.852097+0000","mapping_epoch":121,"log_start":"125'600","ondisk_log_start":"125'600","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'10375","last_scrub_stamp":"2024-03-20T09:00:55.481104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:00:55.481104+0000","objects_scrubbed":4526,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:21.563910+0000","scrub_duration":696,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279052288,"num_objects":4258,"num_object_clones":0,"num_object_copies":12774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10634,"num_write_kb":476544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"170'10317","reported_seq":10554,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-20T09:01:10.917075+0000","last_change":"2024-03-20T09:00:53.604045+0000","last_active":"2024-03-20T09:01:10.917075+0000","last_peered":"2024-03-20T09:01:10.917075+0000","last_clean":"2024-03-20T09:01:10.917075+0000","last_became_active":"2024-03-20T08:58:10.016062+0000","last_became_peered":"2024-03-20T08:58:10.016062+0000","last_unstale":"2024-03-20T09:01:10.917075+0000","last_undegraded":"2024-03-20T09:01:10.917075+0000","last_fullsized":"2024-03-20T09:01:10.917075+0000","mapping_epoch":121,"log_start":"125'300","ondisk_log_start":"125'300","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'10185","last_scrub_stamp":"2024-03-20T09:00:53.603924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:00:53.603924+0000","objects_scrubbed":4540,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:18.362421+0000","scrub_duration":932,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287506432,"num_objects":4387,"num_object_clones":0,"num_object_copies":13161,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10317,"num_write_kb":470528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"170'10878","reported_seq":11120,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-20T09:01:14.724086+0000","last_change":"2024-03-20T09:00:48.795741+0000","last_active":"2024-03-20T09:01:14.724086+0000","last_peered":"2024-03-20T09:01:14.724086+0000","last_clean":"2024-03-20T09:01:14.724086+0000","last_became_active":"2024-03-20T08:58:10.027614+0000","last_became_peered":"2024-03-20T08:58:10.027614+0000","last_unstale":"2024-03-20T09:01:14.724086+0000","last_undegraded":"2024-03-20T09:01:14.724086+0000","last_fullsized":"2024-03-20T09:01:14.724086+0000","mapping_epoch":121,"log_start":"125'800","ondisk_log_start":"125'800","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'10297","last_scrub_stamp":"2024-03-20T09:00:48.795610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:00:48.795610+0000","objects_scrubbed":4877,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:10.924576+0000","scrub_duration":504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281542656,"num_objects":4296,"num_object_clones":0,"num_object_copies":12888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10878,"num_write_kb":485568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"170'10664","reported_seq":10903,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-20T09:01:14.099091+0000","last_change":"2024-03-20T09:00:52.401762+0000","last_active":"2024-03-20T09:01:14.099091+0000","last_peered":"2024-03-20T09:01:14.099091+0000","last_clean":"2024-03-20T09:01:14.099091+0000","last_became_active":"2024-03-20T08:58:10.027712+0000","last_became_peered":"2024-03-20T08:58:10.027712+0000","last_unstale":"2024-03-20T09:01:14.099091+0000","last_undegraded":"2024-03-20T09:01:14.099091+0000","last_fullsized":"2024-03-20T09:01:14.099091+0000","mapping_epoch":121,"log_start":"125'600","ondisk_log_start":"125'600","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'10310","last_scrub_stamp":"2024-03-20T09:00:52.401561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:00:52.401561+0000","objects_scrubbed":4659,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:00.047259+0000","scrub_duration":1181,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280231936,"num_objects":4276,"num_object_clones":0,"num_object_copies":12828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10664,"num_write_kb":478080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"170'10423","reported_seq":10661,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-20T09:01:11.497921+0000","last_change":"2024-03-20T09:00:54.016339+0000","last_active":"2024-03-20T09:01:11.497921+0000","last_peered":"2024-03-20T09:01:11.497921+0000","last_clean":"2024-03-20T09:01:11.497921+0000","last_became_active":"2024-03-20T08:58:09.935063+0000","last_became_peered":"2024-03-20T08:58:09.935063+0000","last_unstale":"2024-03-20T09:01:11.497921+0000","last_undegraded":"2024-03-20T09:01:11.497921+0000","last_fullsized":"2024-03-20T09:01:11.497921+0000","mapping_epoch":121,"log_start":"125'400","ondisk_log_start":"125'400","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'10275","last_scrub_stamp":"2024-03-20T09:00:54.016170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:00:54.016170+0000","objects_scrubbed":4576,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:20.494969+0000","scrub_duration":903,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288555008,"num_objects":4403,"num_object_clones":0,"num_object_copies":13209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10423,"num_write_kb":474432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"170'14348","reported_seq":14728,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-20T09:01:14.134931+0000","last_change":"2024-03-20T09:00:09.317786+0000","last_active":"2024-03-20T09:01:14.134931+0000","last_peered":"2024-03-20T09:01:14.134931+0000","last_clean":"2024-03-20T09:01:14.134931+0000","last_became_active":"2024-03-20T08:58:10.016313+0000","last_became_peered":"2024-03-20T08:58:10.016313+0000","last_unstale":"2024-03-20T09:01:14.134931+0000","last_undegraded":"2024-03-20T09:01:14.134931+0000","last_fullsized":"2024-03-20T09:01:14.134931+0000","mapping_epoch":121,"log_start":"134'4300","ondisk_log_start":"134'4300","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'12090","last_scrub_stamp":"2024-03-20T09:00:09.317614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:00:09.317614+0000","objects_scrubbed":6610,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:01:20.580827+0000","scrub_duration":2021,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281346048,"num_objects":4293,"num_object_clones":0,"num_object_copies":12879,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10475,"num_write_kb":472576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"170'10436","reported_seq":10672,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-20T09:01:12.847455+0000","last_change":"2024-03-20T09:00:54.964123+0000","last_active":"2024-03-20T09:01:12.847455+0000","last_peered":"2024-03-20T09:01:12.847455+0000","last_clean":"2024-03-20T09:01:12.847455+0000","last_became_active":"2024-03-20T08:58:10.027826+0000","last_became_peered":"2024-03-20T08:58:10.027826+0000","last_unstale":"2024-03-20T09:01:12.847455+0000","last_undegraded":"2024-03-20T09:01:12.847455+0000","last_fullsized":"2024-03-20T09:01:12.847455+0000","mapping_epoch":121,"log_start":"125'400","ondisk_log_start":"125'400","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'10263","last_scrub_stamp":"2024-03-20T09:00:54.963885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:00:54.963885+0000","objects_scrubbed":4429,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:17.874888+0000","scrub_duration":1047,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":277282840,"num_objects":4232,"num_object_clones":0,"num_object_copies":12696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4232,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10436,"num_write_kb":469313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"170'10519","reported_seq":10758,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-20T09:01:13.156481+0000","last_change":"2024-03-20T09:00:54.407001+0000","last_active":"2024-03-20T09:01:13.156481+0000","last_peered":"2024-03-20T09:01:13.156481+0000","last_clean":"2024-03-20T09:01:13.156481+0000","last_became_active":"2024-03-20T08:58:10.027586+0000","last_became_peered":"2024-03-20T08:58:10.027586+0000","last_unstale":"2024-03-20T09:01:13.156481+0000","last_undegraded":"2024-03-20T09:01:13.156481+0000","last_fullsized":"2024-03-20T09:01:13.156481+0000","mapping_epoch":121,"log_start":"125'500","ondisk_log_start":"125'500","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'10327","last_scrub_stamp":"2024-03-20T09:00:54.406683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:00:54.406683+0000","objects_scrubbed":4596,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:05.211717+0000","scrub_duration":1534,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":286064640,"num_objects":4365,"num_object_clones":0,"num_object_copies":13095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10519,"num_write_kb":476288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":4554620952,"num_objects":69499,"num_object_clones":0,"num_object_copies":208497,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69499,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168067,"num_write_kb":7602049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13635956736,"data_stored":13635944520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160740,"ondisk_log_size":160740,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860479,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":903176,"kb_used_data":831744,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92841976,"statfs":{"total":95995035648,"available":95070183424,"internally_reserved":0,"allocated":851705856,"data_stored":851457902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":73136580},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":89,"seq":382252089411,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":917180,"kb_used_data":867832,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92827972,"statfs":{"total":95995035648,"available":95055843328,"internally_reserved":0,"allocated":888659968,"data_stored":888415566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":80,"seq":343597383755,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1756484,"kb_used_data":1657276,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":91988668,"statfs":{"total":95995035648,"available":94196396032,"internally_reserved":0,"allocated":1697050624,"data_stored":1696802150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":101579200},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":70,"seq":300647710799,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1203324,"kb_used_data":1131768,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92541828,"statfs":{"total":95995035648,"available":94762831872,"internally_reserved":0,"allocated":1158930432,"data_stored":1158686050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":60,"seq":257698037844,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1505732,"kb_used_data":1434176,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92239420,"statfs":{"total":95995035648,"available":94453166080,"internally_reserved":0,"allocated":1468596224,"data_stored":1468348290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332186,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":596104,"kb_used_data":546880,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93149048,"statfs":{"total":95995035648,"available":95384625152,"internally_reserved":0,"allocated":560005120,"data_stored":559757186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574964,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1469056,"kb_used_data":1371128,"kb_used_omap":1,"kb_used_meta":97790,"kb_avail":92276096,"statfs":{"total":95995035648,"available":94490722304,"internally_reserved":0,"allocated":1404035072,"data_stored":1403790690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":100137414},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738490,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1495424,"kb_used_data":1414200,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":92249728,"statfs":{"total":95995035648,"available":94463721472,"internally_reserved":0,"allocated":1448140800,"data_stored":1447896418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411437,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1196736,"kb_used_data":1125304,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92548416,"statfs":{"total":95995035648,"available":94769577984,"internally_reserved":0,"allocated":1152311296,"data_stored":1152066914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149801,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1747140,"kb_used_data":1635452,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":91998012,"statfs":{"total":95995035648,"available":94205964288,"internally_reserved":0,"allocated":1674702848,"data_stored":1674454394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":114358727},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953573,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":880772,"kb_used_data":805180,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92864380,"statfs":{"total":95995035648,"available":95093125120,"internally_reserved":0,"allocated":824504320,"data_stored":824255866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659230,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":621120,"kb_used_data":571896,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93124032,"statfs":{"total":95995035648,"available":95359008768,"internally_reserved":0,"allocated":585621504,"data_stored":585377122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1439105024,"data_stored":1439105024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1398079488,"data_stored":1398079488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1145896960,"data_stored":1145896960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1664290816,"data_stored":1664286744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":818417664,"data_stored":818413592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":584515584,"data_stored":584515584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":557907968,"data_stored":557907968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1464008704,"data_stored":1464008704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1150943232,"data_stored":1150943232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1694961664,"data_stored":1694957592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":877658112,"data_stored":877658112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":840171520,"data_stored":840171520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:01:18.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:18 smithi115 ceph-mon[72476]: from='client.24833 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:01:18.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:18 smithi122 ceph-mon[73495]: from='client.24833 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:01:18.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:18 smithi026 ceph-mon[68309]: from='client.24833 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:01:19.367 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:19 smithi026 ceph-mon[68309]: pgmap v512: 17 pgs: 17 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 505 op/s 2024-03-20T09:01:19.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:19 smithi115 ceph-mon[72476]: pgmap v512: 17 pgs: 17 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 505 op/s 2024-03-20T09:01:19.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:19 smithi122 ceph-mon[73495]: pgmap v512: 17 pgs: 17 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 505 op/s 2024-03-20T09:01:19.716 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:01:19.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:19.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:01:19.821 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:01:19.879 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:19.905 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:01:19.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:19.913 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:01:20.014 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:01:20.046 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:20.079 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:01:20.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:20.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:01:20.186 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:01:20.220 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:20.243 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:01:20.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:20.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:01:20.351 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:01:20.438 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:20.473 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:01:20.473 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:01:20.912 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:01:20.912 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:01:20.934 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:01:20.983 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:01:21.024 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:01:21.077 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:01:21.122 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:01:21.166 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:01:21.211 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:01:21.255 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:01:21.292 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:01:21.327 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:01:21.336 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:01:21.336 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:01:21.336 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:01:21.336 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:01:21.336 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:01:21.336 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:01:21.337 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:01:21.337 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:01:21.337 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:01:21.337 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:01:21.337 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:01:21.337 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:01:21.373 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:21 smithi026 ceph-mon[68309]: pgmap v513: 17 pgs: 17 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 572 op/s 2024-03-20T09:01:21.373 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:21 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/194299660' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:01:21.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:21 smithi115 ceph-mon[72476]: pgmap v513: 17 pgs: 17 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 572 op/s 2024-03-20T09:01:21.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:21 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/194299660' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:01:21.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:21 smithi122 ceph-mon[73495]: pgmap v513: 17 pgs: 17 active+clean; 4.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 572 op/s 2024-03-20T09:01:21.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:21 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/194299660' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:01:21.547 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:01:21.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:21.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:01:21.586 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-20T09:01:21.660 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:01:21.691 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:21.710 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:01:21.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:21.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:01:21.817 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:01:21.905 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:23.187 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-03-20T09:01:23.187 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:01:23.187 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-20T09:01:23.187 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-20T09:01:23.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:23 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:01:23.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:23 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:01:23.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:23 smithi115 ceph-mon[72476]: pgmap v514: 17 pgs: 17 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 731 op/s 2024-03-20T09:01:23.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:23 smithi115 ceph-mon[72476]: osdmap e172: 12 total, 12 up, 12 in 2024-03-20T09:01:23.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:23 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:01:23.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:23 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:01:23.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:23 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:01:23.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:23 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:01:23.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:23 smithi122 ceph-mon[73495]: pgmap v514: 17 pgs: 17 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 731 op/s 2024-03-20T09:01:23.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:23 smithi122 ceph-mon[73495]: osdmap e172: 12 total, 12 up, 12 in 2024-03-20T09:01:23.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:23 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:01:23.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:23 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:01:23.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:23 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:01:23.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:23 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:01:23.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:23 smithi026 ceph-mon[68309]: pgmap v514: 17 pgs: 17 active+clean; 3.9 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 731 op/s 2024-03-20T09:01:23.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:23 smithi026 ceph-mon[68309]: osdmap e172: 12 total, 12 up, 12 in 2024-03-20T09:01:23.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:23 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:01:23.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:23 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:01:24.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:24 smithi115 ceph-mon[72476]: osdmap e173: 12 total, 12 up, 12 in 2024-03-20T09:01:24.452 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:24 smithi122 ceph-mon[73495]: osdmap e173: 12 total, 12 up, 12 in 2024-03-20T09:01:24.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:24 smithi026 ceph-mon[68309]: osdmap e173: 12 total, 12 up, 12 in 2024-03-20T09:01:24.998 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:01:25.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:25.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:01:25.104 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:01:25.131 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:01:25.190 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:25.195 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:01:25.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:25 smithi115 ceph-mon[72476]: pgmap v517: 17 pgs: 17 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 917 op/s 2024-03-20T09:01:25.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:25 smithi122 ceph-mon[73495]: pgmap v517: 17 pgs: 17 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 917 op/s 2024-03-20T09:01:25.447 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:25 smithi026 ceph-mon[68309]: pgmap v517: 17 pgs: 17 active+clean; 3.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 917 op/s 2024-03-20T09:01:25.563 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:01:25.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:25.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:01:25.671 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:01:25.773 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:26.050 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:01:26.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:26.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:01:26.159 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:01:26.192 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:26.215 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:01:26.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:26.218 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:01:26.319 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:01:26.361 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:26.393 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:01:26.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:26.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:01:26.500 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:01:26.575 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:27.025 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:01:27.052 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:01:27.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:27.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:01:27.166 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:01:27.197 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:27.231 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:01:27.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:27.235 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:01:27.336 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:01:27.364 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:27.386 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:01:27.388 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:27 smithi115 ceph-mon[72476]: pgmap v518: 17 pgs: 17 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 889 op/s 2024-03-20T09:01:27.388 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:27 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:01:27.388 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:27 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:01:27.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:27.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:01:27.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:27 smithi122 ceph-mon[73495]: pgmap v518: 17 pgs: 17 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 889 op/s 2024-03-20T09:01:27.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:27 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:01:27.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:27 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:01:27.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:27 smithi026 ceph-mon[68309]: pgmap v518: 17 pgs: 17 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 889 op/s 2024-03-20T09:01:27.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:27 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:01:27.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:27 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:01:27.490 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:01:27.498 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:01:27.519 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:27.538 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:01:27.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:27.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:01:27.642 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:01:27.676 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:27.702 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:01:27.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:27.706 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:01:27.807 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:01:27.886 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:28.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:28 smithi115 ceph-mon[72476]: osdmap e174: 12 total, 12 up, 12 in 2024-03-20T09:01:28.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:28 smithi122 ceph-mon[73495]: osdmap e174: 12 total, 12 up, 12 in 2024-03-20T09:01:28.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:28 smithi026 ceph-mon[68309]: osdmap e174: 12 total, 12 up, 12 in 2024-03-20T09:01:28.500 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:01:28.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:28.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:01:28.609 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:01:28.642 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:28.669 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:01:28.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:28.672 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:01:28.773 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:01:28.775 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:01:28.776 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:01:28.776 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 174, 2024-03-20T09:01:28.776 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "852fbb68-e696-11ee-95c9-87774f69a715", 2024-03-20T09:01:28.776 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-03-20T08:48:35.281088+0000", 2024-03-20T09:01:28.776 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-03-20T09:01:27.109801+0000", 2024-03-20T09:01:28.776 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2024-03-20T08:56:17.873738+0000", 2024-03-20T09:01:28.776 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2024-03-20T08:55:59.486577+0000", 2024-03-20T09:01:28.776 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-20T09:01:28.776 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5806080, 2024-03-20T09:01:28.776 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2024-03-20T09:01:28.776 INFO:teuthology.orchestra.run.smithi026.stdout: "nodeep-scrub", 2024-03-20T09:01:28.776 INFO:teuthology.orchestra.run.smithi026.stdout: "noscrub", 2024-03-20T09:01:28.777 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2024-03-20T09:01:28.777 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2024-03-20T09:01:28.777 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2024-03-20T09:01:28.777 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2024-03-20T09:01:28.777 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:01:28.777 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 26, 2024-03-20T09:01:28.777 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2024-03-20T09:01:28.777 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-20T09:01:28.777 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-20T09:01:28.777 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2024-03-20T09:01:28.777 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 2, 2024-03-20T09:01:28.777 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 12, 2024-03-20T09:01:28.777 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "luminous", 2024-03-20T09:01:28.777 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "jewel", 2024-03-20T09:01:28.778 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "squid", 2024-03-20T09:01:28.778 INFO:teuthology.orchestra.run.smithi026.stdout: "allow_crimson": false, 2024-03-20T09:01:28.778 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2024-03-20T09:01:28.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.778 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2024-03-20T09:01:28.778 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": ".mgr", 2024-03-20T09:01:28.778 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-03-20T08:52:17.442324+0000", 2024-03-20T09:01:28.778 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-03-20T09:01:28.778 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-03-20T09:01:28.778 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-03-20T09:01:28.778 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 3, 2024-03-20T09:01:28.778 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 2, 2024-03-20T09:01:28.778 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-03-20T09:01:28.779 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-03-20T09:01:28.779 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-03-20T09:01:28.779 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-03-20T09:01:28.779 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-20T09:01:28.779 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-03-20T09:01:28.779 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-03-20T09:01:28.779 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 1, 2024-03-20T09:01:28.779 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 1, 2024-03-20T09:01:28.779 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 1, 2024-03-20T09:01:28.779 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 1, 2024-03-20T09:01:28.779 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 1, 2024-03-20T09:01:28.779 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-03-20T09:01:28.780 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-03-20T09:01:28.780 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-03-20T09:01:28.780 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-03-20T09:01:28.780 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-03-20T09:01:28.780 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-03-20T09:01:28.780 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-03-20T09:01:28.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.780 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "131", 2024-03-20T09:01:28.780 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-03-20T09:01:28.780 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-20T09:01:28.780 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2024-03-20T09:01:28.780 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-03-20T09:01:28.780 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-03-20T09:01:28.780 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-03-20T09:01:28.781 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-03-20T09:01:28.781 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-03-20T09:01:28.781 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-03-20T09:01:28.781 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-03-20T09:01:28.781 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-03-20T09:01:28.781 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-03-20T09:01:28.781 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-03-20T09:01:28.781 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-03-20T09:01:28.781 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-03-20T09:01:28.781 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-03-20T09:01:28.781 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-03-20T09:01:28.781 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-03-20T09:01:28.781 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-20T09:01:28.781 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-20T09:01:28.782 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-20T09:01:28.782 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-03-20T09:01:28.782 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-03-20T09:01:28.782 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-03-20T09:01:28.782 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-03-20T09:01:28.782 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-03-20T09:01:28.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.782 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-03-20T09:01:28.782 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-03-20T09:01:28.782 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-03-20T09:01:28.782 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-03-20T09:01:28.782 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-03-20T09:01:28.782 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-03-20T09:01:28.783 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-03-20T09:01:28.783 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-03-20T09:01:28.783 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-03-20T09:01:28.783 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-03-20T09:01:28.783 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-03-20T09:01:28.783 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2024-03-20T09:01:28.783 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_max": 32, 2024-03-20T09:01:28.783 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2024-03-20T09:01:28.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.784 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-03-20T09:01:28.784 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": {} 2024-03-20T09:01:28.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.784 INFO:teuthology.orchestra.run.smithi026.stdout: "read_balance": { 2024-03-20T09:01:28.784 INFO:teuthology.orchestra.run.smithi026.stdout: "score_type": "Fair distribution", 2024-03-20T09:01:28.784 INFO:teuthology.orchestra.run.smithi026.stdout: "score_acting": 12, 2024-03-20T09:01:28.784 INFO:teuthology.orchestra.run.smithi026.stdout: "score_stable": 12, 2024-03-20T09:01:28.784 INFO:teuthology.orchestra.run.smithi026.stdout: "optimal_score": 0.25, 2024-03-20T09:01:28.784 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_acting": 3, 2024-03-20T09:01:28.784 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_stable": 3, 2024-03-20T09:01:28.784 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-03-20T09:01:28.784 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity": 1, 2024-03-20T09:01:28.784 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity_weighted": 1 2024-03-20T09:01:28.785 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.785 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 2, 2024-03-20T09:01:28.785 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "unique_pool_0", 2024-03-20T09:01:28.785 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-03-20T08:58:08.054071+0000", 2024-03-20T09:01:28.785 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-03-20T09:01:28.785 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-03-20T09:01:28.785 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-03-20T09:01:28.785 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 3, 2024-03-20T09:01:28.785 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 2, 2024-03-20T09:01:28.785 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-03-20T09:01:28.785 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-03-20T09:01:28.786 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-03-20T09:01:28.786 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-03-20T09:01:28.786 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-20T09:01:28.786 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-03-20T09:01:28.786 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-03-20T09:01:28.786 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 16, 2024-03-20T09:01:28.786 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 16, 2024-03-20T09:01:28.786 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 16, 2024-03-20T09:01:28.786 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 16, 2024-03-20T09:01:28.786 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 16, 2024-03-20T09:01:28.786 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-03-20T09:01:28.786 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-03-20T09:01:28.787 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-03-20T09:01:28.787 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-03-20T09:01:28.787 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-03-20T09:01:28.787 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-03-20T09:01:28.787 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-03-20T09:01:28.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.787 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "125", 2024-03-20T09:01:28.787 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-03-20T09:01:28.787 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-20T09:01:28.787 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2024-03-20T09:01:28.787 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-03-20T09:01:28.787 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-03-20T09:01:28.787 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-03-20T09:01:28.787 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-03-20T09:01:28.787 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-03-20T09:01:28.788 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-03-20T09:01:28.788 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-03-20T09:01:28.788 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-03-20T09:01:28.788 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-03-20T09:01:28.788 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-03-20T09:01:28.788 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-03-20T09:01:28.788 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-03-20T09:01:28.788 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-03-20T09:01:28.788 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-03-20T09:01:28.788 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-03-20T09:01:28.788 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-20T09:01:28.788 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-20T09:01:28.788 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-20T09:01:28.788 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-03-20T09:01:28.789 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-03-20T09:01:28.789 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-03-20T09:01:28.789 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-03-20T09:01:28.789 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-03-20T09:01:28.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.789 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-03-20T09:01:28.789 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-03-20T09:01:28.789 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-03-20T09:01:28.789 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-03-20T09:01:28.789 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-03-20T09:01:28.789 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-03-20T09:01:28.789 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-03-20T09:01:28.789 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-03-20T09:01:28.789 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-03-20T09:01:28.790 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-03-20T09:01:28.790 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-03-20T09:01:28.790 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2024-03-20T09:01:28.790 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-03-20T09:01:28.790 INFO:teuthology.orchestra.run.smithi026.stdout: "rados": {} 2024-03-20T09:01:28.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.790 INFO:teuthology.orchestra.run.smithi026.stdout: "read_balance": { 2024-03-20T09:01:28.790 INFO:teuthology.orchestra.run.smithi026.stdout: "score_type": "Fair distribution", 2024-03-20T09:01:28.790 INFO:teuthology.orchestra.run.smithi026.stdout: "score_acting": 2.25, 2024-03-20T09:01:28.791 INFO:teuthology.orchestra.run.smithi026.stdout: "score_stable": 2.25, 2024-03-20T09:01:28.791 INFO:teuthology.orchestra.run.smithi026.stdout: "optimal_score": 1, 2024-03-20T09:01:28.791 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_acting": 2.25, 2024-03-20T09:01:28.791 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_stable": 2.25, 2024-03-20T09:01:28.791 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-03-20T09:01:28.792 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity": 1, 2024-03-20T09:01:28.792 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity_weighted": 1 2024-03-20T09:01:28.792 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.792 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.792 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:01:28.792 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:01:28.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.792 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-03-20T09:01:28.792 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "ad3fd9c0-316f-4fe5-a63b-d480087c1729", 2024-03-20T09:01:28.792 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:01:28.792 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:01:28.793 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:01:28.793 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:01:28.793 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:01:28.793 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:01:28.793 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 8, 2024-03-20T09:01:28.793 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 62, 2024-03-20T09:01:28.793 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:01:28.793 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:01:28.793 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:01:28.793 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.793 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.794 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2024-03-20T09:01:28.794 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:01:28.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.794 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.794 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2024-03-20T09:01:28.794 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:01:28.794 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.794 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.795 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:01:28.795 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.795 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.795 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2024-03-20T09:01:28.795 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:01:28.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.795 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.795 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2024-03-20T09:01:28.795 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:01:28.795 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.795 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.796 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:01:28.796 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.796 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.796 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2024-03-20T09:01:28.796 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:01:28.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.796 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.796 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2024-03-20T09:01:28.796 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:01:28.796 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.796 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.797 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:01:28.797 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.797 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.797 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2024-03-20T09:01:28.797 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:01:28.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.797 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.797 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2024-03-20T09:01:28.797 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:01:28.797 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.797 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.798 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/2139805293", 2024-03-20T09:01:28.798 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/2139805293", 2024-03-20T09:01:28.798 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/2139805293", 2024-03-20T09:01:28.798 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/2139805293", 2024-03-20T09:01:28.798 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:01:28.798 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:01:28.798 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:01:28.798 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.798 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-03-20T09:01:28.798 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e85b94f5-2954-47a1-9c20-096f2606eb04", 2024-03-20T09:01:28.798 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:01:28.799 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:01:28.799 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:01:28.799 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:01:28.799 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:01:28.799 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:01:28.799 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2024-03-20T09:01:28.799 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 121, 2024-03-20T09:01:28.799 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:01:28.799 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:01:28.799 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:01:28.799 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.800 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.800 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2024-03-20T09:01:28.800 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:01:28.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.800 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.800 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2024-03-20T09:01:28.800 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:01:28.800 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.800 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.800 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:01:28.800 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.801 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.801 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2024-03-20T09:01:28.801 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:01:28.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.801 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.801 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2024-03-20T09:01:28.801 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:01:28.801 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.801 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.801 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:01:28.801 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.802 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.802 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2024-03-20T09:01:28.802 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:01:28.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.802 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.802 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2024-03-20T09:01:28.802 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:01:28.803 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.803 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.803 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:01:28.803 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.804 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.804 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2024-03-20T09:01:28.804 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:01:28.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.804 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.805 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2024-03-20T09:01:28.805 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:01:28.805 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.805 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.805 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/3853355134", 2024-03-20T09:01:28.806 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/3853355134", 2024-03-20T09:01:28.806 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/3853355134", 2024-03-20T09:01:28.806 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/3853355134", 2024-03-20T09:01:28.806 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:01:28.806 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:01:28.806 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:01:28.807 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.807 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-03-20T09:01:28.807 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "dd2b10f0-741c-47a4-b224-ac6bfe2c14ef", 2024-03-20T09:01:28.807 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:01:28.808 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:01:28.808 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:01:28.808 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:01:28.808 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:01:28.808 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:01:28.809 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 18, 2024-03-20T09:01:28.809 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 121, 2024-03-20T09:01:28.809 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:01:28.809 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:01:28.809 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:01:28.809 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.810 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.810 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2024-03-20T09:01:28.810 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:01:28.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.811 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.811 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2024-03-20T09:01:28.811 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:01:28.811 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.811 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.812 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:01:28.812 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.812 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.812 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2024-03-20T09:01:28.812 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:01:28.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.813 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.813 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2024-03-20T09:01:28.813 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:01:28.813 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.813 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.813 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:01:28.814 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.814 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.814 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2024-03-20T09:01:28.814 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:01:28.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.815 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.815 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2024-03-20T09:01:28.815 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:01:28.815 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.815 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.816 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:01:28.816 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.816 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.816 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2024-03-20T09:01:28.816 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:01:28.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.817 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.817 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2024-03-20T09:01:28.817 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:01:28.817 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.817 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.817 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/3993357967", 2024-03-20T09:01:28.818 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/3993357967", 2024-03-20T09:01:28.818 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/3993357967", 2024-03-20T09:01:28.818 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/3993357967", 2024-03-20T09:01:28.818 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:01:28.818 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:01:28.818 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:01:28.818 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.819 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-03-20T09:01:28.819 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1", 2024-03-20T09:01:28.819 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:01:28.819 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:01:28.819 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:01:28.819 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:01:28.820 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:01:28.820 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:01:28.820 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 26, 2024-03-20T09:01:28.820 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 0, 2024-03-20T09:01:28.820 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:01:28.820 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:01:28.820 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:01:28.821 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.821 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.821 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2024-03-20T09:01:28.821 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:01:28.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.822 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.822 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2024-03-20T09:01:28.822 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:01:28.822 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.822 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.823 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:01:28.823 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.823 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.823 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2024-03-20T09:01:28.824 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:01:28.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.824 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.824 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2024-03-20T09:01:28.824 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:01:28.824 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.824 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.825 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:01:28.825 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.825 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.825 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2024-03-20T09:01:28.825 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:01:28.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.825 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.826 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2024-03-20T09:01:28.826 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:01:28.826 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.826 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.826 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:01:28.826 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.827 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.827 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2024-03-20T09:01:28.827 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:01:28.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.827 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.827 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2024-03-20T09:01:28.828 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:01:28.828 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.828 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.828 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/4110225509", 2024-03-20T09:01:28.828 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/4110225509", 2024-03-20T09:01:28.828 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/4110225509", 2024-03-20T09:01:28.829 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/4110225509", 2024-03-20T09:01:28.829 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:01:28.829 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:01:28.829 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:01:28.829 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.830 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-03-20T09:01:28.830 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "f14e34d5-9829-42a3-9b25-31a0976c5ef7", 2024-03-20T09:01:28.830 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:01:28.830 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:01:28.830 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:01:28.831 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:01:28.831 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:01:28.831 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:01:28.831 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 32, 2024-03-20T09:01:28.831 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 0, 2024-03-20T09:01:28.831 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:01:28.831 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:01:28.831 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:01:28.831 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.831 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.832 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6800", 2024-03-20T09:01:28.832 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:01:28.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.832 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.832 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6801", 2024-03-20T09:01:28.832 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:01:28.832 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.832 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.832 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:01:28.833 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.833 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.833 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6802", 2024-03-20T09:01:28.833 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:01:28.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.833 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.834 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6803", 2024-03-20T09:01:28.834 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:01:28.834 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.834 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.834 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:01:28.834 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.834 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.834 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6806", 2024-03-20T09:01:28.834 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:01:28.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.835 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.835 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6807", 2024-03-20T09:01:28.835 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:01:28.835 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.835 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.835 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:01:28.835 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.835 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.836 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6804", 2024-03-20T09:01:28.836 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:01:28.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.836 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.836 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6805", 2024-03-20T09:01:28.836 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:01:28.836 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.836 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.836 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6801/477923058", 2024-03-20T09:01:28.837 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6803/477923058", 2024-03-20T09:01:28.837 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6807/477923058", 2024-03-20T09:01:28.837 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6805/477923058", 2024-03-20T09:01:28.837 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:01:28.837 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:01:28.837 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:01:28.837 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.837 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-03-20T09:01:28.837 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "cb431b6a-540d-4318-ae35-d379e1df0185", 2024-03-20T09:01:28.837 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:01:28.837 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:01:28.837 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:01:28.837 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:01:28.838 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:01:28.838 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:01:28.838 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 41, 2024-03-20T09:01:28.838 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 121, 2024-03-20T09:01:28.838 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:01:28.838 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:01:28.838 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:01:28.838 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.838 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.838 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6808", 2024-03-20T09:01:28.838 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:01:28.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.839 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.839 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6809", 2024-03-20T09:01:28.839 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:01:28.839 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.839 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.839 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:01:28.839 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.839 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.840 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6810", 2024-03-20T09:01:28.840 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:01:28.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.840 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.840 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6811", 2024-03-20T09:01:28.840 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:01:28.840 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.840 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.840 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:01:28.840 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.840 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.841 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6814", 2024-03-20T09:01:28.841 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:01:28.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.841 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.841 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6815", 2024-03-20T09:01:28.841 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:01:28.842 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.842 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.842 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:01:28.842 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.842 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.842 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6812", 2024-03-20T09:01:28.842 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:01:28.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.843 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.843 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6813", 2024-03-20T09:01:28.843 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:01:28.843 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.843 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.843 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6809/1339268003", 2024-03-20T09:01:28.843 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6811/1339268003", 2024-03-20T09:01:28.843 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6815/1339268003", 2024-03-20T09:01:28.843 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6813/1339268003", 2024-03-20T09:01:28.843 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:01:28.843 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:01:28.843 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:01:28.843 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.844 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2024-03-20T09:01:28.844 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "753ed830-e764-4ff3-9d9b-87e7fa748b81", 2024-03-20T09:01:28.844 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:01:28.844 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:01:28.844 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:01:28.844 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:01:28.844 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:01:28.844 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:01:28.844 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 51, 2024-03-20T09:01:28.844 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 121, 2024-03-20T09:01:28.844 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:01:28.844 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:01:28.844 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:01:28.845 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.845 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.845 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6816", 2024-03-20T09:01:28.845 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:01:28.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.845 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.845 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6817", 2024-03-20T09:01:28.846 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:01:28.846 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.846 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.846 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:01:28.846 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.847 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.847 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6818", 2024-03-20T09:01:28.847 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:01:28.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.847 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.847 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6819", 2024-03-20T09:01:28.847 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:01:28.848 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.848 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.848 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:01:28.848 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.848 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.848 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6822", 2024-03-20T09:01:28.848 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:01:28.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.849 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.849 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6823", 2024-03-20T09:01:28.849 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:01:28.849 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.849 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.849 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:01:28.850 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.850 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.850 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6820", 2024-03-20T09:01:28.850 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:01:28.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.850 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.850 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6821", 2024-03-20T09:01:28.853 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:01:28.853 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.853 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.854 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6817/3278301843", 2024-03-20T09:01:28.854 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6819/3278301843", 2024-03-20T09:01:28.854 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6823/3278301843", 2024-03-20T09:01:28.854 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6821/3278301843", 2024-03-20T09:01:28.854 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:01:28.855 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:01:28.855 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:01:28.855 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.855 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2024-03-20T09:01:28.856 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "55bcbd37-7a6f-4ab1-b5a9-91a951444d86", 2024-03-20T09:01:28.856 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:01:28.856 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:01:28.856 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:01:28.856 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:01:28.856 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:01:28.857 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:01:28.857 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 60, 2024-03-20T09:01:28.857 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 121, 2024-03-20T09:01:28.857 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:01:28.857 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:01:28.857 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:01:28.858 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.858 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.858 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6824", 2024-03-20T09:01:28.858 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:01:28.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.859 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.859 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6825", 2024-03-20T09:01:28.859 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:01:28.859 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.860 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.860 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:01:28.860 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.860 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.861 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6826", 2024-03-20T09:01:28.861 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:01:28.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.861 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.861 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6827", 2024-03-20T09:01:28.862 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:01:28.862 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.862 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.862 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:01:28.862 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.863 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.863 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6830", 2024-03-20T09:01:28.863 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:01:28.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.863 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.863 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6831", 2024-03-20T09:01:28.863 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:01:28.863 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.864 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.864 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:01:28.864 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.864 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.864 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6828", 2024-03-20T09:01:28.864 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:01:28.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.864 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.864 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6829", 2024-03-20T09:01:28.864 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:01:28.864 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.865 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.865 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6825/3152424001", 2024-03-20T09:01:28.865 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6827/3152424001", 2024-03-20T09:01:28.865 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6831/3152424001", 2024-03-20T09:01:28.865 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6829/3152424001", 2024-03-20T09:01:28.865 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:01:28.865 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:01:28.865 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:01:28.865 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.865 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 8, 2024-03-20T09:01:28.865 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "bc86248e-6bd0-4369-b8a2-e593bc2a610d", 2024-03-20T09:01:28.866 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:01:28.866 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:01:28.866 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:01:28.866 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:01:28.866 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:01:28.866 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:01:28.866 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 70, 2024-03-20T09:01:28.866 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 121, 2024-03-20T09:01:28.866 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:01:28.866 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:01:28.866 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:01:28.866 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.866 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.867 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6800", 2024-03-20T09:01:28.867 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:01:28.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.867 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.867 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6801", 2024-03-20T09:01:28.867 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:01:28.867 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.867 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.867 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:01:28.867 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.867 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.868 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6802", 2024-03-20T09:01:28.868 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:01:28.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.868 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.868 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6803", 2024-03-20T09:01:28.868 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:01:28.868 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.868 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.868 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:01:28.868 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.868 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.869 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6806", 2024-03-20T09:01:28.869 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:01:28.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.869 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.869 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6807", 2024-03-20T09:01:28.869 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:01:28.869 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.869 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.869 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:01:28.869 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.869 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.870 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6804", 2024-03-20T09:01:28.870 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:01:28.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.870 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.870 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6805", 2024-03-20T09:01:28.870 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:01:28.870 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.870 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.870 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6801/1309426740", 2024-03-20T09:01:28.870 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6803/1309426740", 2024-03-20T09:01:28.870 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6807/1309426740", 2024-03-20T09:01:28.870 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6805/1309426740", 2024-03-20T09:01:28.871 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:01:28.871 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:01:28.871 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:01:28.871 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.871 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 9, 2024-03-20T09:01:28.871 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b20a4aa-032a-4c09-ac75-153c554424e7", 2024-03-20T09:01:28.871 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:01:28.871 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:01:28.871 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:01:28.871 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:01:28.871 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:01:28.871 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:01:28.872 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 80, 2024-03-20T09:01:28.872 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 121, 2024-03-20T09:01:28.872 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:01:28.872 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:01:28.872 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:01:28.872 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.872 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.872 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6808", 2024-03-20T09:01:28.872 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:01:28.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.872 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.872 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6809", 2024-03-20T09:01:28.872 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:01:28.873 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.873 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.873 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:01:28.873 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.873 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.873 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6810", 2024-03-20T09:01:28.873 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:01:28.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.873 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.873 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6811", 2024-03-20T09:01:28.873 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:01:28.874 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.874 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.874 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:01:28.874 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.874 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.874 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6814", 2024-03-20T09:01:28.874 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:01:28.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.874 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.874 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6815", 2024-03-20T09:01:28.874 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:01:28.875 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.875 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.875 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:01:28.875 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.875 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.875 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6812", 2024-03-20T09:01:28.875 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:01:28.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.875 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.875 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6813", 2024-03-20T09:01:28.876 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:01:28.876 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.876 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.876 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6809/2953210076", 2024-03-20T09:01:28.876 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6811/2953210076", 2024-03-20T09:01:28.876 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6815/2953210076", 2024-03-20T09:01:28.876 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6813/2953210076", 2024-03-20T09:01:28.876 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:01:28.876 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:01:28.876 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:01:28.876 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.877 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 10, 2024-03-20T09:01:28.877 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e62c1fb3-5dd1-4d0d-997c-d20633d19361", 2024-03-20T09:01:28.877 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:01:28.877 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:01:28.877 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:01:28.877 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:01:28.877 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:01:28.877 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:01:28.877 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 89, 2024-03-20T09:01:28.877 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 0, 2024-03-20T09:01:28.877 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:01:28.877 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:01:28.877 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:01:28.877 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.878 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.878 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6816", 2024-03-20T09:01:28.878 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:01:28.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.878 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.878 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6817", 2024-03-20T09:01:28.878 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:01:28.878 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.878 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.878 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:01:28.878 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.879 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.879 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6818", 2024-03-20T09:01:28.879 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:01:28.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.879 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.879 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6819", 2024-03-20T09:01:28.879 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:01:28.879 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.879 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.879 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:01:28.879 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.880 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.880 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6822", 2024-03-20T09:01:28.880 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:01:28.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.880 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.880 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6823", 2024-03-20T09:01:28.880 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:01:28.880 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.880 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.880 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:01:28.880 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.881 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.881 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6820", 2024-03-20T09:01:28.881 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:01:28.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.881 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.881 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6821", 2024-03-20T09:01:28.881 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:01:28.881 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.881 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.881 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6817/409394340", 2024-03-20T09:01:28.881 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6819/409394340", 2024-03-20T09:01:28.881 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6823/409394340", 2024-03-20T09:01:28.882 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6821/409394340", 2024-03-20T09:01:28.882 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:01:28.882 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:01:28.882 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:01:28.882 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.882 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 11, 2024-03-20T09:01:28.882 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b83994d5-e5ef-48b0-9758-c26843cec3b6", 2024-03-20T09:01:28.882 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:01:28.882 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:01:28.882 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:01:28.882 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:01:28.882 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:01:28.882 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:01:28.883 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 96, 2024-03-20T09:01:28.883 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 121, 2024-03-20T09:01:28.883 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:01:28.883 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:01:28.883 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:01:28.883 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.883 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.883 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6824", 2024-03-20T09:01:28.883 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:01:28.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.883 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.883 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6825", 2024-03-20T09:01:28.884 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:01:28.884 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.884 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.884 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:01:28.884 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.884 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.884 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6826", 2024-03-20T09:01:28.884 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:01:28.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.884 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.884 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6827", 2024-03-20T09:01:28.885 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:01:28.885 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.885 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.885 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:01:28.885 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.885 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.885 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6830", 2024-03-20T09:01:28.885 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:01:28.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.885 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.885 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6831", 2024-03-20T09:01:28.886 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:01:28.886 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.886 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.886 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:01:28.886 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:01:28.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.886 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:01:28.886 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6828", 2024-03-20T09:01:28.886 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:01:28.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.886 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:01:28.886 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6829", 2024-03-20T09:01:28.887 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:01:28.887 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.887 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.887 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6825/2704069354", 2024-03-20T09:01:28.887 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6827/2704069354", 2024-03-20T09:01:28.887 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6831/2704069354", 2024-03-20T09:01:28.887 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6829/2704069354", 2024-03-20T09:01:28.887 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:01:28.887 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:01:28.887 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:01:28.887 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:01:28.887 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.887 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:01:28.888 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2024-03-20T09:01:28.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.888 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-03-20T09:01:28.888 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:01:28.888 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:01:28.888 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:01:28.888 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:01:28.888 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:01:28.888 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:00:15.458602+0000", 2024-03-20T09:01:28.888 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:01:28.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.888 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-03-20T09:01:28.888 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:01:28.888 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:01:28.889 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:01:28.889 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:01:28.889 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:01:28.889 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:00:36.085528+0000", 2024-03-20T09:01:28.889 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:01:28.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.889 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-03-20T09:01:28.889 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:01:28.889 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:01:28.889 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:01:28.889 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:01:28.889 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:01:28.889 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:01:08.169482+0000", 2024-03-20T09:01:28.890 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:01:28.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.890 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-03-20T09:01:28.890 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:01:28.890 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:01:28.890 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:01:28.890 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:01:28.890 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:01:28.890 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:00:31.384708+0000", 2024-03-20T09:01:28.890 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:01:28.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.890 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-03-20T09:01:28.891 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:01:28.891 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:01:28.891 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:01:28.891 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:01:28.891 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:01:28.891 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:01:22.483283+0000", 2024-03-20T09:01:28.891 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:01:28.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.891 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-03-20T09:01:28.891 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:01:28.891 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:01:28.891 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:01:28.891 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:01:28.891 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:01:28.892 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:01:26.278480+0000", 2024-03-20T09:01:28.892 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:01:28.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.892 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2024-03-20T09:01:28.892 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:01:28.892 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:01:28.892 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:01:28.892 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:01:28.892 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:01:28.892 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:00:32.621480+0000", 2024-03-20T09:01:28.892 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:01:28.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.893 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2024-03-20T09:01:28.893 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:01:28.893 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:01:28.893 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:01:28.893 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:01:28.893 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:01:28.893 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:00:48.693641+0000", 2024-03-20T09:01:28.893 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:01:28.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.893 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 8, 2024-03-20T09:01:28.893 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:01:28.893 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:01:28.893 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:01:28.894 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:01:28.894 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:01:28.894 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:00:56.583909+0000", 2024-03-20T09:01:28.894 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:01:28.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.894 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 9, 2024-03-20T09:01:28.894 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:01:28.894 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:01:28.894 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:01:28.894 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:01:28.894 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:01:28.894 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:00:26.483886+0000", 2024-03-20T09:01:28.894 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:01:28.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.895 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 10, 2024-03-20T09:01:28.895 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:01:28.895 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:01:28.895 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:01:28.895 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:01:28.895 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:01:28.895 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:01:21.494541+0000", 2024-03-20T09:01:28.895 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:01:28.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:01:28.895 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 11, 2024-03-20T09:01:28.896 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:01:28.896 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:01:28.896 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:01:28.896 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:01:28.896 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:01:28.896 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:00:52.392962+0000", 2024-03-20T09:01:28.896 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:01:28.896 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.896 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:01:28.896 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [], 2024-03-20T09:01:28.896 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2024-03-20T09:01:28.896 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_primaries": [], 2024-03-20T09:01:28.896 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2024-03-20T09:01:28.896 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2024-03-20T09:01:28.896 INFO:teuthology.orchestra.run.smithi026.stdout: "blocklist": { 2024-03-20T09:01:28.897 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/4291404191": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:01:28.897 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/642012566": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:01:28.897 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6801/1181459999": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:01:28.897 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/3984513952": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:01:28.897 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6800/1654945498": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:01:28.897 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6800/1181459999": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:01:28.897 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/137770981": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:01:28.897 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/1451237762": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:01:28.897 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/3938386631": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:01:28.897 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6801/1654945498": "2024-03-21T08:49:04.596754+0000" 2024-03-20T09:01:28.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.897 INFO:teuthology.orchestra.run.smithi026.stdout: "range_blocklist": {}, 2024-03-20T09:01:28.897 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2024-03-20T09:01:28.898 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2024-03-20T09:01:28.898 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2024-03-20T09:01:28.898 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2024-03-20T09:01:28.898 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2024-03-20T09:01:28.898 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2024-03-20T09:01:28.898 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2024-03-20T09:01:28.898 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:01:28.898 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2024-03-20T09:01:28.898 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2024-03-20T09:01:28.898 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2024-03-20T09:01:28.898 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2024-03-20T09:01:28.898 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {}, 2024-03-20T09:01:28.898 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": { 2024-03-20T09:01:28.899 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_enabled": false, 2024-03-20T09:01:28.899 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_bucket_count": 0, 2024-03-20T09:01:28.899 INFO:teuthology.orchestra.run.smithi026.stdout: "degraded_stretch_mode": 0, 2024-03-20T09:01:28.899 INFO:teuthology.orchestra.run.smithi026.stdout: "recovering_stretch_mode": 0, 2024-03-20T09:01:28.899 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_bucket": 0 2024-03-20T09:01:28.899 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:01:28.899 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:01:28.900 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:29.159 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:29 smithi026 ceph-mon[68309]: pgmap v520: 17 pgs: 17 active+clean; 3.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.00k op/s 2024-03-20T09:01:29.159 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:29 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3956533218' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:01:29.223 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:01:29.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:29.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:01:29.341 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:01:29.373 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:29.395 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:01:29.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:29.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:01:29.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:29 smithi122 ceph-mon[73495]: pgmap v520: 17 pgs: 17 active+clean; 3.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.00k op/s 2024-03-20T09:01:29.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:29 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3956533218' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:01:29.486 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:01:29.502 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:01:29.536 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:29.562 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:01:29.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:29.569 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:01:29.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:29 smithi115 ceph-mon[72476]: pgmap v520: 17 pgs: 17 active+clean; 3.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.00k op/s 2024-03-20T09:01:29.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:29 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3956533218' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:01:29.670 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:01:29.701 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:29.730 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:01:29.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:29.737 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:01:29.838 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:01:29.867 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:29.893 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:01:29.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:29.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:01:29.914 INFO:tasks.thrashosds.thrasher:j is {'epoch': 174, 'fsid': '852fbb68-e696-11ee-95c9-87774f69a715', 'created': '2024-03-20T08:48:35.281088+0000', 'modified': '2024-03-20T09:01:27.109801+0000', 'last_up_change': '2024-03-20T08:56:17.873738+0000', 'last_in_change': '2024-03-20T08:55:59.486577+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-20T08:52:17.442324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '131', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'unique_pool_0', 'create_time': '2024-03-20T08:58:08.054071+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '125', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'ad3fd9c0-316f-4fe5-a63b-d480087c1729', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 62, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6802', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6803', 'nonce': 2139805293}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6804', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6805', 'nonce': 2139805293}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6808', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6809', 'nonce': 2139805293}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6806', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6807', 'nonce': 2139805293}]}, 'public_addr': '172.21.15.26:6803/2139805293', 'cluster_addr': '172.21.15.26:6805/2139805293', 'heartbeat_back_addr': '172.21.15.26:6809/2139805293', 'heartbeat_front_addr': '172.21.15.26:6807/2139805293', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'e85b94f5-2954-47a1-9c20-096f2606eb04', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 121, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6810', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6811', 'nonce': 3853355134}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6812', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6813', 'nonce': 3853355134}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6816', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6817', 'nonce': 3853355134}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6814', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6815', 'nonce': 3853355134}]}, 'public_addr': '172.21.15.26:6811/3853355134', 'cluster_addr': '172.21.15.26:6813/3853355134', 'heartbeat_back_addr': '172.21.15.26:6817/3853355134', 'heartbeat_front_addr': '172.21.15.26:6815/3853355134', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'dd2b10f0-741c-47a4-b224-ac6bfe2c14ef', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 121, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6818', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6819', 'nonce': 3993357967}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6820', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6821', 'nonce': 3993357967}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6824', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6825', 'nonce': 3993357967}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6822', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6823', 'nonce': 3993357967}]}, 'public_addr': '172.21.15.26:6819/3993357967', 'cluster_addr': '172.21.15.26:6821/3993357967', 'heartbeat_back_addr': '172.21.15.26:6825/3993357967', 'heartbeat_front_addr': '172.21.15.26:6823/3993357967', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6826', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6827', 'nonce': 4110225509}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6828', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6829', 'nonce': 4110225509}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6832', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6833', 'nonce': 4110225509}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6830', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6831', 'nonce': 4110225509}]}, 'public_addr': '172.21.15.26:6827/4110225509', 'cluster_addr': '172.21.15.26:6829/4110225509', 'heartbeat_back_addr': '172.21.15.26:6833/4110225509', 'heartbeat_front_addr': '172.21.15.26:6831/4110225509', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'f14e34d5-9829-42a3-9b25-31a0976c5ef7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6800', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6801', 'nonce': 477923058}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6802', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6803', 'nonce': 477923058}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6806', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6807', 'nonce': 477923058}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6804', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6805', 'nonce': 477923058}]}, 'public_addr': '172.21.15.115:6801/477923058', 'cluster_addr': '172.21.15.115:6803/477923058', 'heartbeat_back_addr': '172.21.15.115:6807/477923058', 'heartbeat_front_addr': '172.21.15.115:6805/477923058', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'cb431b6a-540d-4318-ae35-d379e1df0185', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 121, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6808', 'nonce': 1339268003}, {'type': 'v1', 'addr': '172.21.15.115:6809', 'nonce': 1339268003}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6810', 'nonce': 1339268003}, {'type': 'v1', 'addr': '172.21.15.115:6811', 'nonce': 1339268003}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6814', 'nonce': 1339268003}, {'type': 'v1', 'addr': '172.21.15.115:6815', 'nonce': 1339268003}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6812', 'nonce': 1339268003}, {'type': 'v1', 'addr': '172.21.15.115:6813', 'nonce': 1339268003}]}, 'public_addr': '172.21.15.115:6809/1339268003', 'cluster_addr': '172.21.15.115:6811/1339268003', 'heartbeat_back_addr': '172.21.15.115:6815/1339268003', 'heartbeat_front_addr': '172.21.15.115:6813/1339268003', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '753ed830-e764-4ff3-9d9b-87e7fa748b81', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 121, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6816', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6817', 'nonce': 3278301843}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6818', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6819', 'nonce': 3278301843}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6822', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6823', 'nonce': 3278301843}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6820', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6821', 'nonce': 3278301843}]}, 'public_addr': '172.21.15.115:6817/3278301843', 'cluster_addr': '172.21.15.115:6819/3278301843', 'heartbeat_back_addr': '172.21.15.115:6823/3278301843', 'heartbeat_front_addr': '172.21.15.115:6821/3278301843', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '55bcbd37-7a6f-4ab1-b5a9-91a951444d86', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 121, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6824', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6825', 'nonce': 3152424001}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6826', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6827', 'nonce': 3152424001}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6830', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6831', 'nonce': 3152424001}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6828', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6829', 'nonce': 3152424001}]}, 'public_addr': '172.21.15.115:6825/3152424001', 'cluster_addr': '172.21.15.115:6827/3152424001', 'heartbeat_back_addr': '172.21.15.115:6831/3152424001', 'heartbeat_front_addr': '172.21.15.115:6829/3152424001', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'bc86248e-6bd0-4369-b8a2-e593bc2a610d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 70, 'up_thru': 121, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6800', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6801', 'nonce': 1309426740}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6802', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6803', 'nonce': 1309426740}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6806', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6807', 'nonce': 1309426740}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6804', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6805', 'nonce': 1309426740}]}, 'public_addr': '172.21.15.122:6801/1309426740', 'cluster_addr': '172.21.15.122:6803/1309426740', 'heartbeat_back_addr': '172.21.15.122:6807/1309426740', 'heartbeat_front_addr': '172.21.15.122:6805/1309426740', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '9b20a4aa-032a-4c09-ac75-153c554424e7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 121, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6808', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6809', 'nonce': 2953210076}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6810', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6811', 'nonce': 2953210076}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6814', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6815', 'nonce': 2953210076}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6812', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6813', 'nonce': 2953210076}]}, 'public_addr': '172.21.15.122:6809/2953210076', 'cluster_addr': '172.21.15.122:6811/2953210076', 'heartbeat_back_addr': '172.21.15.122:6815/2953210076', 'heartbeat_front_addr': '172.21.15.122:6813/2953210076', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'e62c1fb3-5dd1-4d0d-997c-d20633d19361', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 89, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6816', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6817', 'nonce': 409394340}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6818', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6819', 'nonce': 409394340}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6822', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6823', 'nonce': 409394340}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6820', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6821', 'nonce': 409394340}]}, 'public_addr': '172.21.15.122:6817/409394340', 'cluster_addr': '172.21.15.122:6819/409394340', 'heartbeat_back_addr': '172.21.15.122:6823/409394340', 'heartbeat_front_addr': '172.21.15.122:6821/409394340', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'b83994d5-e5ef-48b0-9758-c26843cec3b6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 96, 'up_thru': 121, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6824', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6825', 'nonce': 2704069354}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6826', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6827', 'nonce': 2704069354}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6830', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6831', 'nonce': 2704069354}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6828', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6829', 'nonce': 2704069354}]}, 'public_addr': '172.21.15.122:6825/2704069354', 'cluster_addr': '172.21.15.122:6827/2704069354', 'heartbeat_back_addr': '172.21.15.122:6831/2704069354', 'heartbeat_front_addr': '172.21.15.122:6829/2704069354', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:00:15.458602+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:00:36.085528+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:01:08.169482+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:00:31.384708+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:01:22.483283+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:01:26.278480+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:00:32.621480+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:00:48.693641+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:00:56.583909+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:00:26.483886+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:01:21.494541+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:00:52.392962+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.26:0/4291404191': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/642012566': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:6801/1181459999': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/3984513952': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6800/1654945498': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6800/1181459999': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/137770981': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:0/1451237762': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/3938386631': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6801/1654945498': '2024-03-21T08:49:04.596754+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-20T09:01:29.914 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-03-20T09:01:29.996 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:01:30.024 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:30.043 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:01:30.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:30.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:01:30.147 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:01:30.164 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:30 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/460360862' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:01:30.164 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:30 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:01:30.182 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:30.201 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:01:30.209 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:01:30.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:30.216 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:01:30.317 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:01:30.348 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:30.393 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:01:30.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:30.403 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:01:30.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:30 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/460360862' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:01:30.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:30 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:01:30.503 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:01:30.570 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:30.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:30 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/460360862' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:01:30.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:30 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:01:30.697 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:01:30.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:30.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:01:30.805 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:01:30.835 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:30.870 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:01:30.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:30.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:01:30.977 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:01:31.007 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:31.030 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:01:31.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:31.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:01:31.135 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:01:31.169 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:31.199 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:01:31.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:31.202 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:01:31.303 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:01:31.338 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:31.370 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:01:31.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:31.380 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:01:31.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:31 smithi122 ceph-mon[73495]: pgmap v521: 17 pgs: 17 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 876 op/s 2024-03-20T09:01:31.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:31 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:01:31.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:31 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:01:31.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:31 smithi122 ceph-mon[73495]: osdmap e175: 12 total, 12 up, 12 in 2024-03-20T09:01:31.479 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:31 smithi026 ceph-mon[68309]: pgmap v521: 17 pgs: 17 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 876 op/s 2024-03-20T09:01:31.479 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:31 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:01:31.479 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:31 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:01:31.479 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:31 smithi026 ceph-mon[68309]: osdmap e175: 12 total, 12 up, 12 in 2024-03-20T09:01:31.481 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:01:31.514 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:31.541 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:01:31.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:31.550 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:01:31.651 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:01:31.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:31 smithi115 ceph-mon[72476]: pgmap v521: 17 pgs: 17 active+clean; 3.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 876 op/s 2024-03-20T09:01:31.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:31 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:01:31.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:31 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:01:31.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:31 smithi115 ceph-mon[72476]: osdmap e175: 12 total, 12 up, 12 in 2024-03-20T09:01:31.685 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:31.718 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:01:31.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:31.723 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:01:31.824 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:01:31.854 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:31.880 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:01:31.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:31.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:01:31.986 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:01:32.018 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:32.041 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:01:32.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:32.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:01:32.145 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:01:32.180 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:32.202 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:01:32.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:32.210 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:01:32.259 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:01:32.260 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:01:32.260 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-03-20T09:01:32.260 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-03-20T09:01:32.260 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:01:32.310 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:01:32.342 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:32.369 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:01:32.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:32.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:01:32.473 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:01:32.557 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:32.936 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-20T09:01:33.133 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:01:33.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:33.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:01:33.239 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:01:33.270 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:33.299 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:01:33.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:33.303 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:01:33.404 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:01:33.442 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:33.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:33 smithi026 ceph-mon[68309]: pgmap v523: 17 pgs: 17 active+clean; 3.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 834 op/s 2024-03-20T09:01:33.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:33 smithi026 ceph-mon[68309]: 2.c scrub starts 2024-03-20T09:01:33.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:33 smithi026 ceph-mon[68309]: 2.c scrub ok 2024-03-20T09:01:33.473 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:01:33.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:33.482 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:01:33.486 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:33 smithi122 ceph-mon[73495]: pgmap v523: 17 pgs: 17 active+clean; 3.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 834 op/s 2024-03-20T09:01:33.486 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:33 smithi122 ceph-mon[73495]: 2.c scrub starts 2024-03-20T09:01:33.487 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:33 smithi122 ceph-mon[73495]: 2.c scrub ok 2024-03-20T09:01:33.584 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:01:33.620 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:33.651 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:01:33.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:33 smithi115 ceph-mon[72476]: pgmap v523: 17 pgs: 17 active+clean; 3.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 834 op/s 2024-03-20T09:01:33.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:33 smithi115 ceph-mon[72476]: 2.c scrub starts 2024-03-20T09:01:33.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:33 smithi115 ceph-mon[72476]: 2.c scrub ok 2024-03-20T09:01:33.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:33.658 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:01:33.759 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:01:33.848 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:33.887 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:01:33.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:33.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:01:33.936 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:01:33.986 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:01:34.000 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:01:34.016 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:01:34.052 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:01:34.076 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:34.123 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:01:34.176 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:01:34.216 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:01:34.217 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:34 smithi026 ceph-mon[68309]: 1.0 scrub starts 2024-03-20T09:01:34.217 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:34 smithi026 ceph-mon[68309]: 1.0 scrub ok 2024-03-20T09:01:34.217 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:34 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2787243860' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:01:34.263 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:01:34.300 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:01:34.339 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:01:34.339 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:01:34.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:34.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:01:34.390 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:01:34.443 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:01:34.444 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:01:34.473 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:34 smithi122 ceph-mon[73495]: 1.0 scrub starts 2024-03-20T09:01:34.473 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:34 smithi122 ceph-mon[73495]: 1.0 scrub ok 2024-03-20T09:01:34.473 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:34 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2787243860' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:01:34.473 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:34.497 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:01:34.498 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:01:34.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:34.504 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:01:34.514 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:01:34.514 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:01:34.514 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:01:34.515 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:01:34.515 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:01:34.515 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:01:34.515 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:01:34.515 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:01:34.515 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:01:34.515 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:01:34.515 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:01:34.515 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:01:34.605 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:01:34.640 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:34.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:34 smithi115 ceph-mon[72476]: 1.0 scrub starts 2024-03-20T09:01:34.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:34 smithi115 ceph-mon[72476]: 1.0 scrub ok 2024-03-20T09:01:34.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:34 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2787243860' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:01:34.660 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:01:34.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:34.666 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:01:34.767 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:01:34.812 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:34.841 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:01:34.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:34.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:01:34.915 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-03-20T09:01:34.915 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:01:34.916 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-03-20T09:01:34.916 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:01:34.946 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:01:34.994 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:35.022 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:01:35.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:35.027 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:01:35.128 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:01:35.159 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:35.199 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:01:35.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:35.205 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:01:35.306 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:01:35.371 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:35 smithi026 ceph-mon[68309]: pgmap v524: 17 pgs: 17 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 891 op/s 2024-03-20T09:01:35.374 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:35.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:35 smithi122 ceph-mon[73495]: pgmap v524: 17 pgs: 17 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 891 op/s 2024-03-20T09:01:35.472 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:01:35.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:35.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:01:35.579 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:01:35.617 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:35.631 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:01:35.647 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:01:35.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:35.650 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:01:35.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:35 smithi115 ceph-mon[72476]: pgmap v524: 17 pgs: 17 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 891 op/s 2024-03-20T09:01:35.751 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:01:35.825 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:36.073 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:01:36.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:36.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:01:36.185 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:01:36.237 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:36.248 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:01:36.355 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:01:36.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:36.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:01:36.460 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:01:36.540 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:36.592 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:01:36.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:36.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:01:36.707 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:01:36.738 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:36.763 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:01:36.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:36.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:01:36.869 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:01:36.898 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:36.920 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:01:36.925 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:01:36.925 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:01:36.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:36.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:01:37.027 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:01:37.099 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:37.240 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:01:37.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:37.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:01:37.350 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:01:37.397 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:37.425 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:01:37.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:37.428 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:01:37.529 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:01:37.589 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:37 smithi026 ceph-mon[68309]: pgmap v525: 17 pgs: 17 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 801 op/s 2024-03-20T09:01:37.590 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:37.640 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:01:37.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:37.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:01:37.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:37 smithi115 ceph-mon[72476]: pgmap v525: 17 pgs: 17 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 801 op/s 2024-03-20T09:01:37.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:37 smithi122 ceph-mon[73495]: pgmap v525: 17 pgs: 17 active+clean; 3.2 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 801 op/s 2024-03-20T09:01:37.744 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:01:37.806 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:38.129 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:01:38.256 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-20T09:01:38.583 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:01:38.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:38.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:01:38.596 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:01:38.690 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:01:38.721 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:38.743 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:01:38.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:38.747 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:01:38.848 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:01:38.898 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:38.940 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:01:38.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:38.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:01:39.050 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:01:39.082 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:39.129 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:01:39.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:39.134 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:01:39.235 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:01:39.278 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:39.306 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:01:39.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:39.311 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:01:39.412 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:01:39.442 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:39.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:39 smithi122 ceph-mon[73495]: pgmap v526: 17 pgs: 17 active+clean; 3.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 857 op/s 2024-03-20T09:01:39.468 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:01:39.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:39.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:01:39.538 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:01:39.544 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:01:39.575 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:01:39.599 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:39 smithi115 ceph-mon[72476]: pgmap v526: 17 pgs: 17 active+clean; 3.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 857 op/s 2024-03-20T09:01:39.601 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:39.626 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:01:39.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:39.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:01:39.703 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:39 smithi026 ceph-mon[68309]: pgmap v526: 17 pgs: 17 active+clean; 3.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 857 op/s 2024-03-20T09:01:39.729 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:01:39.803 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:40.177 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:01:40.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:40.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:01:40.282 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:01:40.351 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:40.703 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:01:40.722 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:01:41.426 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:01:41.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:41.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:01:41.535 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:01:41.572 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:41 smithi115 ceph-mon[72476]: pgmap v527: 17 pgs: 17 active+clean; 3.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 846 op/s 2024-03-20T09:01:41.572 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:41 smithi115 ceph-mon[72476]: 2.4 scrub starts 2024-03-20T09:01:41.574 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:41.611 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:01:41.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:41.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:01:41.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:41 smithi122 ceph-mon[73495]: pgmap v527: 17 pgs: 17 active+clean; 3.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 846 op/s 2024-03-20T09:01:41.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:41 smithi122 ceph-mon[73495]: 2.4 scrub starts 2024-03-20T09:01:41.716 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:01:41.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:41 smithi026 ceph-mon[68309]: pgmap v527: 17 pgs: 17 active+clean; 3.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 846 op/s 2024-03-20T09:01:41.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:41 smithi026 ceph-mon[68309]: 2.4 scrub starts 2024-03-20T09:01:41.748 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:41.771 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:01:41.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:41.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:01:41.798 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":527,"stamp":"2024-03-20T09:01:39.545870+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3187016248,"num_objects":48624,"num_object_clones":0,"num_object_copies":145872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48624,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":189125,"num_write_kb":7604987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160854,"ondisk_log_size":160854,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":10441012,"kb_used_data":9461396,"kb_used_omap":18,"kb_used_meta":979245,"kb_avail":1114500812,"statfs":{"total":1151940427776,"available":1141248831488,"internally_reserved":0,"allocated":9688469504,"data_stored":9685240028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":1002747195},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,5,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-555352064,"num_objects":-8474,"num_object_clones":0,"num_object_copies":-25422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8474,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.013299"},"pg_stats":[{"pgid":"2.f","version":"175'11552","reported_seq":11799,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-20T09:01:36.529831+0000","last_change":"2024-03-20T09:00:48.749086+0000","last_active":"2024-03-20T09:01:36.529831+0000","last_peered":"2024-03-20T09:01:36.529831+0000","last_clean":"2024-03-20T09:01:36.529831+0000","last_became_active":"2024-03-20T08:58:10.027677+0000","last_became_peered":"2024-03-20T08:58:10.027677+0000","last_unstale":"2024-03-20T09:01:36.529831+0000","last_undegraded":"2024-03-20T09:01:36.529831+0000","last_fullsized":"2024-03-20T09:01:36.529831+0000","mapping_epoch":121,"log_start":"126'1500","ondisk_log_start":"126'1500","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'9900","last_scrub_stamp":"2024-03-20T09:00:48.748963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:00:48.748963+0000","objects_scrubbed":4676,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:12.710297+0000","scrub_duration":532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198180864,"num_objects":3024,"num_object_clones":0,"num_object_copies":9072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11552,"num_write_kb":466432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"175'11793","reported_seq":12021,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-20T09:01:34.778048+0000","last_change":"2024-03-20T09:00:15.743746+0000","last_active":"2024-03-20T09:01:34.778048+0000","last_peered":"2024-03-20T09:01:34.778048+0000","last_clean":"2024-03-20T09:01:34.778048+0000","last_became_active":"2024-03-20T08:58:10.015946+0000","last_became_peered":"2024-03-20T08:58:10.015946+0000","last_unstale":"2024-03-20T09:01:34.778048+0000","last_undegraded":"2024-03-20T09:01:34.778048+0000","last_fullsized":"2024-03-20T09:01:34.778048+0000","mapping_epoch":121,"log_start":"126'1700","ondisk_log_start":"126'1700","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'8757","last_scrub_stamp":"2024-03-20T09:00:15.743623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:00:15.743623+0000","objects_scrubbed":6227,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:01:36.603416+0000","scrub_duration":2344,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204537856,"num_objects":3121,"num_object_clones":0,"num_object_copies":9363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11793,"num_write_kb":477248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"175'11820","reported_seq":12068,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-20T09:01:37.067836+0000","last_change":"2024-03-20T09:01:01.312938+0000","last_active":"2024-03-20T09:01:37.067836+0000","last_peered":"2024-03-20T09:01:37.067836+0000","last_clean":"2024-03-20T09:01:37.067836+0000","last_became_active":"2024-03-20T08:58:09.933090+0000","last_became_peered":"2024-03-20T08:58:09.933090+0000","last_unstale":"2024-03-20T09:01:37.067836+0000","last_undegraded":"2024-03-20T09:01:37.067836+0000","last_fullsized":"2024-03-20T09:01:37.067836+0000","mapping_epoch":121,"log_start":"126'1800","ondisk_log_start":"126'1800","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'10289","last_scrub_stamp":"2024-03-20T09:01:01.312817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:01:01.312817+0000","objects_scrubbed":4519,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:08.660376+0000","scrub_duration":435,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195821568,"num_objects":2988,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11820,"num_write_kb":473856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"175'11764","reported_seq":12010,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-20T09:01:38.281375+0000","last_change":"2024-03-20T09:00:47.460883+0000","last_active":"2024-03-20T09:01:38.281375+0000","last_peered":"2024-03-20T09:01:38.281375+0000","last_clean":"2024-03-20T09:01:38.281375+0000","last_became_active":"2024-03-20T08:58:10.027595+0000","last_became_peered":"2024-03-20T08:58:10.027595+0000","last_unstale":"2024-03-20T09:01:38.281375+0000","last_undegraded":"2024-03-20T09:01:38.281375+0000","last_fullsized":"2024-03-20T09:01:38.281375+0000","mapping_epoch":121,"log_start":"126'1700","ondisk_log_start":"126'1700","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'9875","last_scrub_stamp":"2024-03-20T09:00:47.460723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:00:47.460723+0000","objects_scrubbed":4831,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:01:54.542606+0000","scrub_duration":528,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192806912,"num_objects":2942,"num_object_clones":0,"num_object_copies":8826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11764,"num_write_kb":470592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"175'12157","reported_seq":12408,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-20T09:01:37.067411+0000","last_change":"2024-03-20T09:00:49.806152+0000","last_active":"2024-03-20T09:01:37.067411+0000","last_peered":"2024-03-20T09:01:37.067411+0000","last_clean":"2024-03-20T09:01:37.067411+0000","last_became_active":"2024-03-20T08:58:09.929029+0000","last_became_peered":"2024-03-20T08:58:09.929029+0000","last_unstale":"2024-03-20T09:01:37.067411+0000","last_undegraded":"2024-03-20T09:01:37.067411+0000","last_fullsized":"2024-03-20T09:01:37.067411+0000","mapping_epoch":121,"log_start":"127'2100","ondisk_log_start":"127'2100","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'10360","last_scrub_stamp":"2024-03-20T09:00:49.805840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:00:49.805840+0000","objects_scrubbed":4895,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:14.860484+0000","scrub_duration":877,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201392128,"num_objects":3073,"num_object_clones":0,"num_object_copies":9219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12157,"num_write_kb":487360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":522,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-20T09:01:33.809420+0000","last_change":"2024-03-20T09:01:33.809459+0000","last_active":"2024-03-20T09:01:33.809420+0000","last_peered":"2024-03-20T09:01:33.809420+0000","last_clean":"2024-03-20T09:01:33.809420+0000","last_became_active":"2024-03-20T08:56:21.910528+0000","last_became_peered":"2024-03-20T08:56:21.910528+0000","last_unstale":"2024-03-20T09:01:33.809420+0000","last_undegraded":"2024-03-20T09:01:33.809420+0000","last_fullsized":"2024-03-20T09:01:33.809420+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:01:33.809365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:01:33.809365+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:44.946329+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"175'11581","reported_seq":11830,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-20T09:01:35.852217+0000","last_change":"2024-03-20T09:00:49.146954+0000","last_active":"2024-03-20T09:01:35.852217+0000","last_peered":"2024-03-20T09:01:35.852217+0000","last_clean":"2024-03-20T09:01:35.852217+0000","last_became_active":"2024-03-20T08:58:09.933985+0000","last_became_peered":"2024-03-20T08:58:09.933985+0000","last_unstale":"2024-03-20T09:01:35.852217+0000","last_undegraded":"2024-03-20T09:01:35.852217+0000","last_fullsized":"2024-03-20T09:01:35.852217+0000","mapping_epoch":121,"log_start":"126'1500","ondisk_log_start":"126'1500","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'9863","last_scrub_stamp":"2024-03-20T09:00:49.146848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:00:49.146848+0000","objects_scrubbed":4833,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:01:49.401312+0000","scrub_duration":527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204144640,"num_objects":3115,"num_object_clones":0,"num_object_copies":9345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11581,"num_write_kb":470272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"175'11779","reported_seq":12030,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-20T09:01:36.569893+0000","last_change":"2024-03-20T09:00:45.728660+0000","last_active":"2024-03-20T09:01:36.569893+0000","last_peered":"2024-03-20T09:01:36.569893+0000","last_clean":"2024-03-20T09:01:36.569893+0000","last_became_active":"2024-03-20T08:58:10.027686+0000","last_became_peered":"2024-03-20T08:58:10.027686+0000","last_unstale":"2024-03-20T09:01:36.569893+0000","last_undegraded":"2024-03-20T09:01:36.569893+0000","last_fullsized":"2024-03-20T09:01:36.569893+0000","mapping_epoch":121,"log_start":"126'1700","ondisk_log_start":"126'1700","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'10042","last_scrub_stamp":"2024-03-20T09:00:45.728563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:00:45.728563+0000","objects_scrubbed":4802,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:01:51.770911+0000","scrub_duration":527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200867840,"num_objects":3065,"num_object_clones":0,"num_object_copies":9195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11779,"num_write_kb":475008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"175'11767","reported_seq":12015,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-20T09:01:34.668973+0000","last_change":"2024-03-20T09:01:05.787896+0000","last_active":"2024-03-20T09:01:34.668973+0000","last_peered":"2024-03-20T09:01:34.668973+0000","last_clean":"2024-03-20T09:01:34.668973+0000","last_became_active":"2024-03-20T08:58:10.027648+0000","last_became_peered":"2024-03-20T08:58:10.027648+0000","last_unstale":"2024-03-20T09:01:34.668973+0000","last_undegraded":"2024-03-20T09:01:34.668973+0000","last_fullsized":"2024-03-20T09:01:34.668973+0000","mapping_epoch":121,"log_start":"126'1700","ondisk_log_start":"126'1700","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'10448","last_scrub_stamp":"2024-03-20T09:01:05.787737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:01:05.787737+0000","objects_scrubbed":4488,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:08.404662+0000","scrub_duration":434,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207683584,"num_objects":3169,"num_object_clones":0,"num_object_copies":9507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11767,"num_write_kb":477952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"175'11750","reported_seq":11999,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-20T09:01:34.793567+0000","last_change":"2024-03-20T09:00:55.481213+0000","last_active":"2024-03-20T09:01:34.793567+0000","last_peered":"2024-03-20T09:01:34.793567+0000","last_clean":"2024-03-20T09:01:34.793567+0000","last_became_active":"2024-03-20T08:58:09.931796+0000","last_became_peered":"2024-03-20T08:58:09.931796+0000","last_unstale":"2024-03-20T09:01:34.793567+0000","last_undegraded":"2024-03-20T09:01:34.793567+0000","last_fullsized":"2024-03-20T09:01:34.793567+0000","mapping_epoch":121,"log_start":"126'1700","ondisk_log_start":"126'1700","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'10375","last_scrub_stamp":"2024-03-20T09:00:55.481104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:00:55.481104+0000","objects_scrubbed":4526,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:21.563910+0000","scrub_duration":696,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205914112,"num_objects":3142,"num_object_clones":0,"num_object_copies":9426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11750,"num_write_kb":476544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"175'11658","reported_seq":11905,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-20T09:01:35.914477+0000","last_change":"2024-03-20T09:00:53.604045+0000","last_active":"2024-03-20T09:01:35.914477+0000","last_peered":"2024-03-20T09:01:35.914477+0000","last_clean":"2024-03-20T09:01:35.914477+0000","last_became_active":"2024-03-20T08:58:10.016062+0000","last_became_peered":"2024-03-20T08:58:10.016062+0000","last_unstale":"2024-03-20T09:01:35.914477+0000","last_undegraded":"2024-03-20T09:01:35.914477+0000","last_fullsized":"2024-03-20T09:01:35.914477+0000","mapping_epoch":121,"log_start":"126'1600","ondisk_log_start":"126'1600","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'10185","last_scrub_stamp":"2024-03-20T09:00:53.603924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:00:53.603924+0000","objects_scrubbed":4540,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:18.362421+0000","scrub_duration":932,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199622656,"num_objects":3046,"num_object_clones":0,"num_object_copies":9138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11658,"num_write_kb":470528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"175'12007","reported_seq":12257,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-20T09:01:34.751308+0000","last_change":"2024-03-20T09:00:48.795741+0000","last_active":"2024-03-20T09:01:34.751308+0000","last_peered":"2024-03-20T09:01:34.751308+0000","last_clean":"2024-03-20T09:01:34.751308+0000","last_became_active":"2024-03-20T08:58:10.027614+0000","last_became_peered":"2024-03-20T08:58:10.027614+0000","last_unstale":"2024-03-20T09:01:34.751308+0000","last_undegraded":"2024-03-20T09:01:34.751308+0000","last_fullsized":"2024-03-20T09:01:34.751308+0000","mapping_epoch":121,"log_start":"127'2000","ondisk_log_start":"127'2000","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'10297","last_scrub_stamp":"2024-03-20T09:00:48.795610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:00:48.795610+0000","objects_scrubbed":4877,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:10.924576+0000","scrub_duration":504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207552512,"num_objects":3167,"num_object_clones":0,"num_object_copies":9501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12007,"num_write_kb":485568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"175'12015","reported_seq":12264,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-20T09:01:39.122551+0000","last_change":"2024-03-20T09:00:52.401762+0000","last_active":"2024-03-20T09:01:39.122551+0000","last_peered":"2024-03-20T09:01:39.122551+0000","last_clean":"2024-03-20T09:01:39.122551+0000","last_became_active":"2024-03-20T08:58:10.027712+0000","last_became_peered":"2024-03-20T08:58:10.027712+0000","last_unstale":"2024-03-20T09:01:39.122551+0000","last_undegraded":"2024-03-20T09:01:39.122551+0000","last_fullsized":"2024-03-20T09:01:39.122551+0000","mapping_epoch":121,"log_start":"127'2000","ondisk_log_start":"127'2000","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'10310","last_scrub_stamp":"2024-03-20T09:00:52.401561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:00:52.401561+0000","objects_scrubbed":4659,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:00.047259+0000","scrub_duration":1181,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191692800,"num_objects":2925,"num_object_clones":0,"num_object_copies":8775,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12015,"num_write_kb":478080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"175'11814","reported_seq":12062,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-20T09:01:36.517578+0000","last_change":"2024-03-20T09:00:54.016339+0000","last_active":"2024-03-20T09:01:36.517578+0000","last_peered":"2024-03-20T09:01:36.517578+0000","last_clean":"2024-03-20T09:01:36.517578+0000","last_became_active":"2024-03-20T08:58:09.935063+0000","last_became_peered":"2024-03-20T08:58:09.935063+0000","last_unstale":"2024-03-20T09:01:36.517578+0000","last_undegraded":"2024-03-20T09:01:36.517578+0000","last_fullsized":"2024-03-20T09:01:36.517578+0000","mapping_epoch":121,"log_start":"126'1800","ondisk_log_start":"126'1800","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'10275","last_scrub_stamp":"2024-03-20T09:00:54.016170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:00:54.016170+0000","objects_scrubbed":4576,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:20.494969+0000","scrub_duration":903,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197394432,"num_objects":3012,"num_object_clones":0,"num_object_copies":9036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11814,"num_write_kb":474432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.c","version":"175'15716","reported_seq":16146,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-20T09:01:39.120176+0000","last_change":"2024-03-20T09:01:33.035234+0000","last_active":"2024-03-20T09:01:39.120176+0000","last_peered":"2024-03-20T09:01:39.120176+0000","last_clean":"2024-03-20T09:01:39.120176+0000","last_became_active":"2024-03-20T08:58:10.016313+0000","last_became_peered":"2024-03-20T08:58:10.016313+0000","last_unstale":"2024-03-20T09:01:39.120176+0000","last_undegraded":"2024-03-20T09:01:39.120176+0000","last_fullsized":"2024-03-20T09:01:39.120176+0000","mapping_epoch":121,"log_start":"140'5700","ondisk_log_start":"140'5700","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'15408","last_scrub_stamp":"2024-03-20T09:01:33.035015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:01:33.035015+0000","objects_scrubbed":3249,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:43.570157+0000","scrub_duration":804,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191692800,"num_objects":2925,"num_object_clones":0,"num_object_copies":8775,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11843,"num_write_kb":472576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.d","version":"175'11735","reported_seq":11981,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-20T09:01:37.868829+0000","last_change":"2024-03-20T09:00:54.964123+0000","last_active":"2024-03-20T09:01:37.868829+0000","last_peered":"2024-03-20T09:01:37.868829+0000","last_clean":"2024-03-20T09:01:37.868829+0000","last_became_active":"2024-03-20T08:58:10.027826+0000","last_became_peered":"2024-03-20T08:58:10.027826+0000","last_unstale":"2024-03-20T09:01:37.868829+0000","last_undegraded":"2024-03-20T09:01:37.868829+0000","last_fullsized":"2024-03-20T09:01:37.868829+0000","mapping_epoch":121,"log_start":"126'1700","ondisk_log_start":"126'1700","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'10263","last_scrub_stamp":"2024-03-20T09:00:54.963885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:00:54.963885+0000","objects_scrubbed":4429,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:17.874888+0000","scrub_duration":1047,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192151576,"num_objects":2933,"num_object_clones":0,"num_object_copies":8799,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2933,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11735,"num_write_kb":469313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.e","version":"175'11909","reported_seq":12156,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-20T09:01:38.285600+0000","last_change":"2024-03-20T09:00:54.407001+0000","last_active":"2024-03-20T09:01:38.285600+0000","last_peered":"2024-03-20T09:01:38.285600+0000","last_clean":"2024-03-20T09:01:38.285600+0000","last_became_active":"2024-03-20T08:58:10.027586+0000","last_became_peered":"2024-03-20T08:58:10.027586+0000","last_unstale":"2024-03-20T09:01:38.285600+0000","last_undegraded":"2024-03-20T09:01:38.285600+0000","last_fullsized":"2024-03-20T09:01:38.285600+0000","mapping_epoch":121,"log_start":"126'1900","ondisk_log_start":"126'1900","created":121,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'10327","last_scrub_stamp":"2024-03-20T09:00:54.406683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:58:08.911564+0000","last_clean_scrub_stamp":"2024-03-20T09:00:54.406683+0000","objects_scrubbed":4596,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:02:05.211717+0000","scrub_duration":1534,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194969600,"num_objects":2975,"num_object_clones":0,"num_object_copies":8925,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11909,"num_write_kb":476288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":3186425880,"num_objects":48622,"num_object_clones":0,"num_object_copies":145866,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48622,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":188944,"num_write_kb":7602049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9590812672,"data_stored":9590800456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160717,"ondisk_log_size":160717,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2644640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4792320,"data_stored":4760352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860484,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":639172,"kb_used_data":567740,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93105980,"statfs":{"total":95995035648,"available":95340523520,"internally_reserved":0,"allocated":581365760,"data_stored":581094976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":73136580},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":89,"seq":382252089416,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":660028,"kb_used_data":592756,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93085124,"statfs":{"total":95995035648,"available":95319166976,"internally_reserved":0,"allocated":606982144,"data_stored":606714912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":68876736},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":80,"seq":343597383760,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1277504,"kb_used_data":1161656,"kb_used_omap":1,"kb_used_meta":115838,"kb_avail":92467648,"statfs":{"total":95995035648,"available":94686871552,"internally_reserved":0,"allocated":1189535744,"data_stored":1189264440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":118618560},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":70,"seq":300647710804,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":864124,"kb_used_data":788468,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92881028,"statfs":{"total":95995035648,"available":95110172672,"internally_reserved":0,"allocated":807391232,"data_stored":807124020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":60,"seq":257698037849,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1075780,"kb_used_data":990460,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":92669372,"statfs":{"total":95995035648,"available":94893436928,"internally_reserved":0,"allocated":1014231040,"data_stored":1013960276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":87357898},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332190,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":460548,"kb_used_data":403004,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93284604,"statfs":{"total":95995035648,"available":95523434496,"internally_reserved":0,"allocated":412676096,"data_stored":412405332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574968,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1115580,"kb_used_data":1016180,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92629572,"statfs":{"total":95995035648,"available":94852681728,"internally_reserved":0,"allocated":1040568320,"data_stored":1040301108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":101579206},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738495,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1075324,"kb_used_data":967796,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92669828,"statfs":{"total":95995035648,"available":94893903872,"internally_reserved":0,"allocated":991023104,"data_stored":990755892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411442,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":838076,"kb_used_data":766644,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92907076,"statfs":{"total":95995035648,"available":95136845824,"internally_reserved":0,"allocated":785043456,"data_stored":784776244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149805,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1309696,"kb_used_data":1210488,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92435456,"statfs":{"total":95995035648,"available":94653906944,"internally_reserved":0,"allocated":1239539712,"data_stored":1239268428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":101579207},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953577,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":672128,"kb_used_data":600696,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93073024,"statfs":{"total":95995035648,"available":95306776576,"internally_reserved":0,"allocated":615112704,"data_stored":614841420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659235,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":453052,"kb_used_data":395508,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93292100,"statfs":{"total":95995035648,"available":95531110400,"internally_reserved":0,"allocated":405000192,"data_stored":404732980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":985137152,"data_stored":985137152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1030619136,"data_stored":1030619136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":777322496,"data_stored":777322496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1232736256,"data_stored":1232732184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":607916032,"data_stored":607911960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":397213696,"data_stored":397213696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":409403392,"data_stored":409403392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1011810304,"data_stored":1011810304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":797048832,"data_stored":797048832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1166479360,"data_stored":1166475288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":604504064,"data_stored":604504064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":570621952,"data_stored":570621952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:01:41.799 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-20T09:01:41.800 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pool set .mgr pg_num 11 2024-03-20T09:01:41.875 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:01:41.903 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:41.925 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:01:41.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:41.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:01:42.030 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:01:42.098 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:42.130 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:01:42.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:42.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:01:42.234 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:01:42.302 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:42.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:42 smithi115 ceph-mon[72476]: from='client.24781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:01:42.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:42 smithi115 ceph-mon[72476]: 2.4 scrub ok 2024-03-20T09:01:42.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:42 smithi122 ceph-mon[73495]: from='client.24781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:01:42.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:42 smithi122 ceph-mon[73495]: 2.4 scrub ok 2024-03-20T09:01:42.724 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:42 smithi026 ceph-mon[68309]: from='client.24781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:01:42.724 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:42 smithi026 ceph-mon[68309]: 2.4 scrub ok 2024-03-20T09:01:43.252 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:01:43.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:43.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:01:43.327 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:01:43.357 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:01:43.442 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:43.477 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:43 smithi026 ceph-mon[68309]: pgmap v528: 17 pgs: 17 active+clean; 2.9 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 794 op/s 2024-03-20T09:01:43.478 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:43 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/24189360' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:01:43.478 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:43 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:01:43.604 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:01:43.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:43.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:01:43.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:43 smithi115 ceph-mon[72476]: pgmap v528: 17 pgs: 17 active+clean; 2.9 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 794 op/s 2024-03-20T09:01:43.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:43 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/24189360' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:01:43.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:43 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:01:43.699 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:43 smithi122 ceph-mon[73495]: pgmap v528: 17 pgs: 17 active+clean; 2.9 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 794 op/s 2024-03-20T09:01:43.699 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:43 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/24189360' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:01:43.699 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:43 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:01:43.718 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:01:43.762 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:43.795 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:01:43.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:43.806 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:01:43.907 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:01:43.940 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:43.968 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:01:43.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:43.974 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:01:44.075 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:01:44.109 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:44.133 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:01:44.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:44.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:01:44.239 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:01:44.341 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:44.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:44 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:01:44.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:44 smithi115 ceph-mon[72476]: Cluster is now healthy 2024-03-20T09:01:44.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:44 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:01:44.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:44 smithi115 ceph-mon[72476]: osdmap e176: 12 total, 12 up, 12 in 2024-03-20T09:01:44.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:44 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:01:44.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:44 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:01:44.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:44 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:01:44.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:44 smithi122 ceph-mon[73495]: Cluster is now healthy 2024-03-20T09:01:44.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:44 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:01:44.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:44 smithi122 ceph-mon[73495]: osdmap e176: 12 total, 12 up, 12 in 2024-03-20T09:01:44.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:44 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:01:44.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:44 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:01:44.724 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:44 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:01:44.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:44 smithi026 ceph-mon[68309]: Cluster is now healthy 2024-03-20T09:01:44.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:44 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:01:44.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:44 smithi026 ceph-mon[68309]: osdmap e176: 12 total, 12 up, 12 in 2024-03-20T09:01:44.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:44 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:01:44.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:44 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:01:44.926 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:01:44.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:44.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:01:45.039 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:01:45.095 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:45.128 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:01:45.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:45.131 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:01:45.234 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:01:45.267 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:45.290 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:01:45.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:45.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:01:45.393 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:01:45.477 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:45.576 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:01:45.576 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:01:45.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:45 smithi115 ceph-mon[72476]: pgmap v530: 17 pgs: 17 active+clean; 2.7 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 852 op/s 2024-03-20T09:01:45.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:45 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:01:45.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:45 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:01:45.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:45 smithi115 ceph-mon[72476]: osdmap e177: 12 total, 12 up, 12 in 2024-03-20T09:01:45.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:45 smithi122 ceph-mon[73495]: pgmap v530: 17 pgs: 17 active+clean; 2.7 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 852 op/s 2024-03-20T09:01:45.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:45 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:01:45.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:45 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:01:45.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:45 smithi122 ceph-mon[73495]: osdmap e177: 12 total, 12 up, 12 in 2024-03-20T09:01:45.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:45 smithi026 ceph-mon[68309]: pgmap v530: 17 pgs: 17 active+clean; 2.7 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 852 op/s 2024-03-20T09:01:45.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:45 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:01:45.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:45 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:01:45.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:45 smithi026 ceph-mon[68309]: osdmap e177: 12 total, 12 up, 12 in 2024-03-20T09:01:45.860 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:01:45.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:45.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:01:45.967 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:01:45.995 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:46.022 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:01:46.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:46.031 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:01:46.132 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:01:46.198 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:46.389 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:01:46.436 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-20T09:01:46.462 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:46 smithi026 ceph-mon[68309]: osdmap e178: 12 total, 12 up, 12 in 2024-03-20T09:01:46.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:46 smithi115 ceph-mon[72476]: osdmap e178: 12 total, 12 up, 12 in 2024-03-20T09:01:46.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:46 smithi122 ceph-mon[73495]: osdmap e178: 12 total, 12 up, 12 in 2024-03-20T09:01:46.892 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:01:46.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:46.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:01:46.999 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:01:47.031 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:47.056 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:01:47.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:47.062 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:01:47.162 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:01:47.193 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:47.214 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:01:47.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:47.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:01:47.318 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:01:47.346 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:01:47.346 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:01:47.346 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:01:47.346 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:01:47.347 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:01:47.366 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:01:47.377 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:47.408 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:01:47.448 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:01:47.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:47 smithi026 ceph-mon[68309]: pgmap v533: 17 pgs: 17 active+clean; 2.6 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 911 op/s 2024-03-20T09:01:47.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:47 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3481590229' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:01:47.501 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:01:47.544 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:01:47.576 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:01:47.618 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:01:47.618 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:01:47.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:47.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:01:47.642 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:01:47.643 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:01:47.643 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:01:47.643 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:01:47.643 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:01:47.643 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:01:47.643 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:01:47.643 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:01:47.643 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:01:47.643 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:01:47.643 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:01:47.643 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:01:47.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:47 smithi115 ceph-mon[72476]: pgmap v533: 17 pgs: 17 active+clean; 2.6 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 911 op/s 2024-03-20T09:01:47.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:47 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3481590229' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:01:47.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:47 smithi122 ceph-mon[73495]: pgmap v533: 17 pgs: 17 active+clean; 2.6 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 911 op/s 2024-03-20T09:01:47.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:47 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3481590229' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:01:47.724 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:01:47.753 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:47.776 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:01:47.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:47.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:01:47.880 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:01:47.955 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:48.005 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:01:48.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:48.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:01:48.111 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:01:48.145 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:48.172 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:01:48.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:48.184 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:01:48.285 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:01:48.330 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:01:48.355 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:48.906 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:01:48.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:48.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:01:49.025 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:01:49.059 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:49.084 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:01:49.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:49.094 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:01:49.195 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:01:49.230 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:49.255 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:01:49.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:49.259 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:01:49.267 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:01:49.360 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:01:49.387 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:49.406 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:01:49.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:49.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:01:49.510 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:01:49.607 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:49 smithi026 ceph-mon[68309]: pgmap v534: 17 pgs: 17 active+clean; 2.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 957 op/s 2024-03-20T09:01:49.622 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:49.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:49 smithi115 ceph-mon[72476]: pgmap v534: 17 pgs: 17 active+clean; 2.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 957 op/s 2024-03-20T09:01:49.680 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:01:49.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:49.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:01:49.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:49 smithi122 ceph-mon[73495]: pgmap v534: 17 pgs: 17 active+clean; 2.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 957 op/s 2024-03-20T09:01:49.786 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:01:49.838 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:49.875 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:01:49.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:49.881 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:01:49.982 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:01:50.068 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:50.549 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:01:50.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:50.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:01:50.663 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:01:50.695 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:50.721 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:01:50.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:50.727 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:01:50.828 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:01:50.858 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:50.886 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:01:50.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:50.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:01:50.991 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:01:51.024 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:51.050 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:01:51.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:51.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:01:51.154 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:01:51.223 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:51.272 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:01:51.291 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:01:51.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:51.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:01:51.397 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:01:51.420 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:01:51.426 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:51.447 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:01:51.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:51.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:01:51.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:51 smithi026 ceph-mon[68309]: pgmap v535: 17 pgs: 17 active+clean; 2.4 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 799 op/s 2024-03-20T09:01:51.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:51 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:01:51.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:51 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:01:51.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:51 smithi026 ceph-mon[68309]: osdmap e179: 12 total, 12 up, 12 in 2024-03-20T09:01:51.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:51 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:01:51.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:51 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:01:51.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:51 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2930944964' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-20T09:01:51.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:51 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-20T09:01:51.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:51 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:01:51.511 INFO:teuthology.orchestra.run.smithi026.stderr:set pool 1 pg_num to 11 2024-03-20T09:01:51.551 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:01:51.579 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:51.605 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:01:51.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:51.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:01:51.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:51 smithi122 ceph-mon[73495]: pgmap v535: 17 pgs: 17 active+clean; 2.4 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 799 op/s 2024-03-20T09:01:51.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:51 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:01:51.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:51 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:01:51.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:51 smithi122 ceph-mon[73495]: osdmap e179: 12 total, 12 up, 12 in 2024-03-20T09:01:51.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:51 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:01:51.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:51 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:01:51.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:51 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2930944964' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-20T09:01:51.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:51 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-20T09:01:51.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:51 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:01:51.712 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:01:51.741 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:51 smithi115 ceph-mon[72476]: pgmap v535: 17 pgs: 17 active+clean; 2.4 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 799 op/s 2024-03-20T09:01:51.741 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:51 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:01:51.741 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:51 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:01:51.741 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:51 smithi115 ceph-mon[72476]: osdmap e179: 12 total, 12 up, 12 in 2024-03-20T09:01:51.741 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:51 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:01:51.741 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:51 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:01:51.741 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:51 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2930944964' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-20T09:01:51.742 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:51 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-20T09:01:51.742 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:51 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:01:51.743 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:51.765 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:01:51.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:51.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:01:51.870 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:01:51.941 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:51.986 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:01:51.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:51.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:01:52.094 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:01:52.220 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:52.267 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:01:52.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:52.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:01:52.372 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:01:52.403 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:52.426 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:01:52.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:52.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:01:52.533 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:01:52.617 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:52.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:52 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:01:52.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:52 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:01:52.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:52 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-20T09:01:52.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:52 smithi115 ceph-mon[72476]: osdmap e180: 12 total, 12 up, 12 in 2024-03-20T09:01:52.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:52 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-20T09:01:52.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:52 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-20T09:01:52.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:52 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:01:52.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:52 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:01:52.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:52 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:01:52.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:52 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:01:52.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:52 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:01:52.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:52 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-20T09:01:52.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:52 smithi026 ceph-mon[68309]: osdmap e180: 12 total, 12 up, 12 in 2024-03-20T09:01:52.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:52 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-20T09:01:52.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:52 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-20T09:01:52.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:52 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:01:52.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:52 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:01:52.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:52 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:01:52.730 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:01:52.730 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:01:52.731 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-03-20T09:01:52.731 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-03-20T09:01:52.731 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:01:52.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:52 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:01:52.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:52 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:01:52.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:52 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-20T09:01:52.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:52 smithi122 ceph-mon[73495]: osdmap e180: 12 total, 12 up, 12 in 2024-03-20T09:01:52.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:52 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-20T09:01:52.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:52 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-20T09:01:52.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:52 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:01:52.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:52 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:01:52.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:52 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:01:53.732 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:53 smithi026 ceph-mon[68309]: pgmap v538: 17 pgs: 17 active+clean; 2.3 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 809 op/s 2024-03-20T09:01:53.732 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:53 smithi026 ceph-mon[68309]: 2.5 scrub starts 2024-03-20T09:01:53.732 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:53 smithi026 ceph-mon[68309]: 2.5 scrub ok 2024-03-20T09:01:53.732 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:53 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-20T09:01:53.732 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:53 smithi026 ceph-mon[68309]: osdmap e181: 12 total, 12 up, 12 in 2024-03-20T09:01:53.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:53 smithi115 ceph-mon[72476]: pgmap v538: 17 pgs: 17 active+clean; 2.3 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 809 op/s 2024-03-20T09:01:53.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:53 smithi115 ceph-mon[72476]: 2.5 scrub starts 2024-03-20T09:01:53.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:53 smithi115 ceph-mon[72476]: 2.5 scrub ok 2024-03-20T09:01:53.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:53 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-20T09:01:53.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:53 smithi115 ceph-mon[72476]: osdmap e181: 12 total, 12 up, 12 in 2024-03-20T09:01:53.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:53 smithi122 ceph-mon[73495]: pgmap v538: 17 pgs: 17 active+clean; 2.3 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 809 op/s 2024-03-20T09:01:53.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:53 smithi122 ceph-mon[73495]: 2.5 scrub starts 2024-03-20T09:01:53.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:53 smithi122 ceph-mon[73495]: 2.5 scrub ok 2024-03-20T09:01:53.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:53 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-20T09:01:53.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:53 smithi122 ceph-mon[73495]: osdmap e181: 12 total, 12 up, 12 in 2024-03-20T09:01:54.026 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:01:54.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:54.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:01:54.138 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:01:54.178 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:54.202 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:01:54.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:54.208 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:01:54.309 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:01:54.346 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:54.379 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:01:54.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:54.386 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:01:54.481 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:01:54.487 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:01:54.520 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:54.530 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-20T09:01:54.540 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:01:54.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:54.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:01:54.648 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:01:54.679 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:54.715 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:01:54.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:54.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:01:54.790 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:54 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:01:54.791 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:54 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:01:54.791 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:54 smithi026 ceph-mon[68309]: osdmap e182: 12 total, 12 up, 12 in 2024-03-20T09:01:54.791 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:54 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-20T09:01:54.791 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:54 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-20T09:01:54.791 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:54 smithi026 ceph-mon[68309]: 1.0 scrub starts 2024-03-20T09:01:54.791 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:54 smithi026 ceph-mon[68309]: 1.0 scrub ok 2024-03-20T09:01:54.822 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:01:54.855 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:54 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:01:54.855 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:54 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:01:54.855 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:54 smithi115 ceph-mon[72476]: osdmap e182: 12 total, 12 up, 12 in 2024-03-20T09:01:54.855 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:54 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-20T09:01:54.855 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:54 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-20T09:01:54.855 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:54 smithi115 ceph-mon[72476]: 1.0 scrub starts 2024-03-20T09:01:54.855 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:54 smithi115 ceph-mon[72476]: 1.0 scrub ok 2024-03-20T09:01:54.856 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:54.876 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:01:54.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:54.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:01:54.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:54 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:01:54.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:54 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:01:54.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:54 smithi122 ceph-mon[73495]: osdmap e182: 12 total, 12 up, 12 in 2024-03-20T09:01:54.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:54 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-20T09:01:54.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:54 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-20T09:01:54.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:54 smithi122 ceph-mon[73495]: 1.0 scrub starts 2024-03-20T09:01:54.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:54 smithi122 ceph-mon[73495]: 1.0 scrub ok 2024-03-20T09:01:54.982 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:01:55.066 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:55.122 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:01:55.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:55.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:01:55.226 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:01:55.299 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:55.681 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:01:55.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:55.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:01:55.795 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:01:55.821 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:55 smithi115 ceph-mon[72476]: pgmap v541: 27 pgs: 10 unknown, 17 active+clean; 2.2 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 952 op/s 2024-03-20T09:01:55.821 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:55 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-20T09:01:55.822 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:55 smithi115 ceph-mon[72476]: osdmap e183: 12 total, 12 up, 12 in 2024-03-20T09:01:55.822 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:55 smithi115 ceph-mon[72476]: osdmap e184: 12 total, 12 up, 12 in 2024-03-20T09:01:55.823 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:55.851 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:01:55.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:55.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:01:55.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:55 smithi122 ceph-mon[73495]: pgmap v541: 27 pgs: 10 unknown, 17 active+clean; 2.2 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 952 op/s 2024-03-20T09:01:55.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:55 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-20T09:01:55.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:55 smithi122 ceph-mon[73495]: osdmap e183: 12 total, 12 up, 12 in 2024-03-20T09:01:55.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:55 smithi122 ceph-mon[73495]: osdmap e184: 12 total, 12 up, 12 in 2024-03-20T09:01:55.955 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:01:55.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:55 smithi026 ceph-mon[68309]: pgmap v541: 27 pgs: 10 unknown, 17 active+clean; 2.2 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 952 op/s 2024-03-20T09:01:55.975 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:55 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-20T09:01:55.975 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:55 smithi026 ceph-mon[68309]: osdmap e183: 12 total, 12 up, 12 in 2024-03-20T09:01:55.975 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:55 smithi026 ceph-mon[68309]: osdmap e184: 12 total, 12 up, 12 in 2024-03-20T09:01:55.992 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:56.025 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:01:56.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:56.031 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:01:56.132 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:01:56.163 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:56.186 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:01:56.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:01:56.194 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:01:56.295 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:01:56.355 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:01:56.602 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:56 smithi026 ceph-mon[68309]: 2.1 scrub starts 2024-03-20T09:01:56.603 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:56 smithi026 ceph-mon[68309]: 2.0 scrub starts 2024-03-20T09:01:56.603 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:56 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3113430928' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:01:56.603 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:56 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:01:56.603 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:56 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:01:56.603 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:56 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:01:56.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:56 smithi115 ceph-mon[72476]: 2.1 scrub starts 2024-03-20T09:01:56.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:56 smithi115 ceph-mon[72476]: 2.0 scrub starts 2024-03-20T09:01:56.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:56 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3113430928' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:01:56.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:56 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:01:56.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:56 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:01:56.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:56 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:01:56.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:56 smithi122 ceph-mon[73495]: 2.1 scrub starts 2024-03-20T09:01:56.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:56 smithi122 ceph-mon[73495]: 2.0 scrub starts 2024-03-20T09:01:56.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:56 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3113430928' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:01:56.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:56 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:01:56.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:56 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:01:56.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:56 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:01:57.058 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:01:57.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:57 smithi115 ceph-mon[72476]: pgmap v544: 27 pgs: 10 unknown, 17 active+clean; 2.1 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 67 KiB/s wr, 953 op/s; 23 KiB/s, 0 objects/s recovering 2024-03-20T09:01:57.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:57 smithi115 ceph-mon[72476]: 2.0 scrub ok 2024-03-20T09:01:57.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:57 smithi115 ceph-mon[72476]: osdmap e185: 12 total, 12 up, 12 in 2024-03-20T09:01:57.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:57 smithi115 ceph-mon[72476]: osdmap e186: 12 total, 12 up, 12 in 2024-03-20T09:01:57.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:57 smithi122 ceph-mon[73495]: pgmap v544: 27 pgs: 10 unknown, 17 active+clean; 2.1 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 67 KiB/s wr, 953 op/s; 23 KiB/s, 0 objects/s recovering 2024-03-20T09:01:57.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:57 smithi122 ceph-mon[73495]: 2.0 scrub ok 2024-03-20T09:01:57.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:57 smithi122 ceph-mon[73495]: osdmap e185: 12 total, 12 up, 12 in 2024-03-20T09:01:57.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:57 smithi122 ceph-mon[73495]: osdmap e186: 12 total, 12 up, 12 in 2024-03-20T09:01:57.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:57 smithi026 ceph-mon[68309]: pgmap v544: 27 pgs: 10 unknown, 17 active+clean; 2.1 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 67 KiB/s wr, 953 op/s; 23 KiB/s, 0 objects/s recovering 2024-03-20T09:01:57.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:57 smithi026 ceph-mon[68309]: 2.0 scrub ok 2024-03-20T09:01:57.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:57 smithi026 ceph-mon[68309]: osdmap e185: 12 total, 12 up, 12 in 2024-03-20T09:01:57.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:57 smithi026 ceph-mon[68309]: osdmap e186: 12 total, 12 up, 12 in 2024-03-20T09:01:58.623 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-03-20T09:01:58.623 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:01:58.623 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-20T09:01:58.623 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 0.058084 2024-03-20T09:01:58.624 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd primary-affinity 7 0.058084476150756625 2024-03-20T09:01:58.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:58 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:01:58.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:58 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:01:58.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:58 smithi026 ceph-mon[68309]: pgmap v547: 27 pgs: 1 peering, 1 active+clean+scrubbing, 2 remapped+peering, 7 unknown, 16 active+clean; 2.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 857 op/s 2024-03-20T09:01:58.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:58 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:01:58.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:58 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:01:58.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:58 smithi026 ceph-mon[68309]: osdmap e187: 12 total, 12 up, 12 in 2024-03-20T09:01:58.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:01:58 smithi026 ceph-mon[68309]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-20T09:01:59.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:58 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:01:59.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:58 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:01:59.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:58 smithi115 ceph-mon[72476]: pgmap v547: 27 pgs: 1 peering, 1 active+clean+scrubbing, 2 remapped+peering, 7 unknown, 16 active+clean; 2.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 857 op/s 2024-03-20T09:01:59.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:58 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:01:59.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:58 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:01:59.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:58 smithi115 ceph-mon[72476]: osdmap e187: 12 total, 12 up, 12 in 2024-03-20T09:01:59.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:01:58 smithi115 ceph-mon[72476]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-20T09:01:59.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:58 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:01:59.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:58 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:01:59.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:58 smithi122 ceph-mon[73495]: pgmap v547: 27 pgs: 1 peering, 1 active+clean+scrubbing, 2 remapped+peering, 7 unknown, 16 active+clean; 2.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 857 op/s 2024-03-20T09:01:59.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:58 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:01:59.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:58 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:01:59.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:58 smithi122 ceph-mon[73495]: osdmap e187: 12 total, 12 up, 12 in 2024-03-20T09:01:59.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:01:58 smithi122 ceph-mon[73495]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-20T09:02:00.656 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:02:00.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:00.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:02:00.679 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:02:00.764 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:02:00.879 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:01.129 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:00 smithi026 ceph-mon[68309]: pgmap v549: 27 pgs: 1 peering, 1 active+clean+scrubbing, 2 remapped+peering, 5 unknown, 18 active+clean; 1.9 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 781 op/s 2024-03-20T09:02:01.148 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:02:01.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:01.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:02:01.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:00 smithi115 ceph-mon[72476]: pgmap v549: 27 pgs: 1 peering, 1 active+clean+scrubbing, 2 remapped+peering, 5 unknown, 18 active+clean; 1.9 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 781 op/s 2024-03-20T09:02:01.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:00 smithi122 ceph-mon[73495]: pgmap v549: 27 pgs: 1 peering, 1 active+clean+scrubbing, 2 remapped+peering, 5 unknown, 18 active+clean; 1.9 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 781 op/s 2024-03-20T09:02:01.255 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:02:01.271 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:02:01.334 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:01.401 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:02:01.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:01.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:02:01.510 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:02:01.559 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:01.589 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:02:01.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:01.593 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:02:01.694 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:02:01.805 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:01.881 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:02:01.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:01.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:02:01.909 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:02:01.957 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:02:01.991 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:02:02.002 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:02:02.053 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:01 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1607301412' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:02:02.057 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:02.071 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:02:02.119 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:02:02.120 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:02:02.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:02.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:02:02.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:01 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1607301412' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:02:02.164 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:02:02.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:01 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1607301412' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:02:02.211 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:02:02.225 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:02:02.260 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:02.268 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:02:02.288 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:02:02.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:02.293 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:02:02.339 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:02:02.378 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:02:02.394 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:02:02.411 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:02:02.423 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:02.445 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:02:02.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:02.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:02:02.450 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:02:02.452 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:02:02.452 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:02:02.452 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:02:02.452 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:02:02.452 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:02:02.452 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:02:02.452 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:02:02.452 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:02:02.452 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:02:02.452 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:02:02.453 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:02:02.453 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:02:02.549 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:02:02.580 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:02.608 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:02:02.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:02.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:02:02.714 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:02:02.782 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:02.822 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:02:02.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:02.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:02:02.934 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:02:02.966 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:02.991 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:02:02.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:02.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:02:03.100 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:02:03.141 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:02 smithi122 ceph-mon[73495]: pgmap v550: 27 pgs: 1 peering, 26 active+clean; 1.8 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 827 op/s 2024-03-20T09:02:03.145 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:03.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:02 smithi115 ceph-mon[72476]: pgmap v550: 27 pgs: 1 peering, 26 active+clean; 1.8 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 827 op/s 2024-03-20T09:02:03.164 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:02:03.169 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:02:03.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:03.174 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:02:03.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:02 smithi026 ceph-mon[68309]: pgmap v550: 27 pgs: 1 peering, 26 active+clean; 1.8 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 827 op/s 2024-03-20T09:02:03.275 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:02:03.307 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:03.327 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:02:03.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:03.332 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:02:03.433 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:02:03.504 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:03.901 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:02:03.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:03.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:02:04.007 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:02:04.035 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:04.065 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:02:04.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:04.071 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:02:04.172 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:02:04.199 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:03 smithi122 ceph-mon[73495]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-20T09:02:04.201 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:04.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:03 smithi026 ceph-mon[68309]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-20T09:02:04.235 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:02:04.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:04.242 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:02:04.343 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:02:04.387 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:04.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:03 smithi115 ceph-mon[72476]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-20T09:02:04.414 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:02:04.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:04.422 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:02:04.523 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:02:04.589 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:04.766 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:02:04.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:04.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:02:04.877 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:02:04.908 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:04.932 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:02:04.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:04.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:02:05.036 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:02:05.061 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:04 smithi026 ceph-mon[68309]: pgmap v551: 27 pgs: 27 active+clean; 1.7 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 266 KiB/s wr, 991 op/s; 152 KiB/s, 0 objects/s recovering 2024-03-20T09:02:05.084 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:05.114 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:02:05.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:05.119 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:02:05.158 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:04 smithi115 ceph-mon[72476]: pgmap v551: 27 pgs: 27 active+clean; 1.7 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 266 KiB/s wr, 991 op/s; 152 KiB/s, 0 objects/s recovering 2024-03-20T09:02:05.219 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:02:05.259 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:04 smithi122 ceph-mon[73495]: pgmap v551: 27 pgs: 27 active+clean; 1.7 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 266 KiB/s wr, 991 op/s; 152 KiB/s, 0 objects/s recovering 2024-03-20T09:02:05.262 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:05.293 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:02:05.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:05.301 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:02:05.302 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:02:05.402 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:02:05.462 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:02:05.462 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:05.778 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:02:05.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:05.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:02:05.884 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:02:05.919 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:05.942 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:02:05.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:05.944 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:02:05.977 INFO:teuthology.orchestra.run.smithi026.stderr:set osd.7 primary-affinity to 0.0580845 (838062) 2024-03-20T09:02:06.045 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:02:06.072 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:06.083 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:05 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2344331652' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.058084476150756625}]: dispatch 2024-03-20T09:02:06.083 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:05 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.058084476150756625}]: dispatch 2024-03-20T09:02:06.091 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:02:06.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:06.094 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:02:06.195 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:02:06.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:05 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2344331652' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.058084476150756625}]: dispatch 2024-03-20T09:02:06.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:05 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.058084476150756625}]: dispatch 2024-03-20T09:02:06.220 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:06.239 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:02:06.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:06.241 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:02:06.343 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:02:06.368 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:06.386 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:02:06.431 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:05 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2344331652' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.058084476150756625}]: dispatch 2024-03-20T09:02:06.431 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:05 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.058084476150756625}]: dispatch 2024-03-20T09:02:06.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:06.433 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:02:06.534 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:02:06.561 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:06.579 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:02:06.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:06.582 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:02:06.683 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:02:06.714 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:06.734 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:02:06.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:06.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:02:06.769 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:02:06.776 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:02:06.776 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:02:06.838 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:02:06.871 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:06.892 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:02:06.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:06.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:02:06.997 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:02:07.028 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:07.051 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:02:07.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:07.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:02:07.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:06 smithi115 ceph-mon[72476]: pgmap v552: 27 pgs: 27 active+clean; 1.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 214 KiB/s wr, 889 op/s; 123 KiB/s, 0 objects/s recovering 2024-03-20T09:02:07.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:06 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.058084476150756625}]': finished 2024-03-20T09:02:07.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:06 smithi115 ceph-mon[72476]: osdmap e188: 12 total, 12 up, 12 in 2024-03-20T09:02:07.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:06 smithi115 ceph-mon[72476]: osdmap e189: 12 total, 12 up, 12 in 2024-03-20T09:02:07.159 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:02:07.188 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:07.210 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:02:07.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:07.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:02:07.317 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:02:07.346 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:06 smithi122 ceph-mon[73495]: pgmap v552: 27 pgs: 27 active+clean; 1.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 214 KiB/s wr, 889 op/s; 123 KiB/s, 0 objects/s recovering 2024-03-20T09:02:07.346 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:06 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.058084476150756625}]': finished 2024-03-20T09:02:07.346 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:06 smithi122 ceph-mon[73495]: osdmap e188: 12 total, 12 up, 12 in 2024-03-20T09:02:07.346 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:06 smithi122 ceph-mon[73495]: osdmap e189: 12 total, 12 up, 12 in 2024-03-20T09:02:07.347 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:07.378 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:02:07.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:07.382 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:02:07.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:06 smithi026 ceph-mon[68309]: pgmap v552: 27 pgs: 27 active+clean; 1.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 214 KiB/s wr, 889 op/s; 123 KiB/s, 0 objects/s recovering 2024-03-20T09:02:07.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:06 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.058084476150756625}]': finished 2024-03-20T09:02:07.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:06 smithi026 ceph-mon[68309]: osdmap e188: 12 total, 12 up, 12 in 2024-03-20T09:02:07.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:06 smithi026 ceph-mon[68309]: osdmap e189: 12 total, 12 up, 12 in 2024-03-20T09:02:07.483 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:02:07.558 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:07.775 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:02:07.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:07.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:02:07.885 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:02:07.940 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:07.969 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:02:07.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:07.974 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:02:08.076 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:02:08.108 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:08.131 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:02:08.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:08.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:02:08.235 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:02:08.265 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:08.284 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:02:08.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:08.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:02:08.388 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:02:08.418 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:08.448 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:02:08.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:08.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:02:08.552 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:02:08.555 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-20T09:02:08.582 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:08.607 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:02:08.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:08.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:02:08.713 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:02:08.803 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:08.842 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:02:08.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:08.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:02:08.950 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:02:08.987 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:09.011 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:02:09.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:09.014 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:02:09.044 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:08 smithi026 ceph-mon[68309]: pgmap v555: 27 pgs: 1 peering, 26 active+clean; 1.5 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 134 KiB/s wr, 892 op/s; 47 KiB/s, 0 objects/s recovering 2024-03-20T09:02:09.115 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:02:09.142 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:09.164 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:02:09.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:09.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:02:09.267 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:02:09.294 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:08 smithi122 ceph-mon[73495]: pgmap v555: 27 pgs: 1 peering, 26 active+clean; 1.5 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 134 KiB/s wr, 892 op/s; 47 KiB/s, 0 objects/s recovering 2024-03-20T09:02:09.297 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:09.323 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:02:09.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:09.329 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:02:09.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:08 smithi115 ceph-mon[72476]: pgmap v555: 27 pgs: 1 peering, 26 active+clean; 1.5 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 134 KiB/s wr, 892 op/s; 47 KiB/s, 0 objects/s recovering 2024-03-20T09:02:09.429 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:02:09.468 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:09.493 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:02:09.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:09.496 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:02:09.597 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:02:09.640 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:09.670 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:02:09.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:09.675 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:02:09.776 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:02:09.844 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:10.028 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:02:10.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:09 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3180633555' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:02:10.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:09 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3180633555' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:02:10.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:09 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3180633555' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:02:11.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:10 smithi026 ceph-mon[68309]: pgmap v556: 27 pgs: 1 peering, 26 active+clean; 1.4 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 845 op/s; 23 KiB/s, 0 objects/s recovering 2024-03-20T09:02:11.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:10 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:02:11.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:10 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3180633555' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:02:11.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:10 smithi026 ceph-mon[68309]: osdmap e190: 12 total, 12 up, 12 in 2024-03-20T09:02:11.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:10 smithi115 ceph-mon[72476]: pgmap v556: 27 pgs: 1 peering, 26 active+clean; 1.4 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 845 op/s; 23 KiB/s, 0 objects/s recovering 2024-03-20T09:02:11.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:10 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:02:11.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:10 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3180633555' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:02:11.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:10 smithi115 ceph-mon[72476]: osdmap e190: 12 total, 12 up, 12 in 2024-03-20T09:02:11.426 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:02:11.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:11.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:02:11.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:10 smithi122 ceph-mon[73495]: pgmap v556: 27 pgs: 1 peering, 26 active+clean; 1.4 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 845 op/s; 23 KiB/s, 0 objects/s recovering 2024-03-20T09:02:11.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:10 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:02:11.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:10 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3180633555' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:02:11.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:10 smithi122 ceph-mon[73495]: osdmap e190: 12 total, 12 up, 12 in 2024-03-20T09:02:11.529 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:02:11.620 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:11.658 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:02:11.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:11.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:02:11.768 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:02:11.770 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:02:11.789 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:02:11.835 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:11.839 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:02:11.895 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:02:11.951 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:02:12.005 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:02:12.044 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:02:12.075 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:02:12.117 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:02:12.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:12.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:02:12.133 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:02:12.163 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:02:12.199 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:02:12.222 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:02:12.247 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:02:12.266 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:12 smithi122 ceph-mon[73495]: pgmap v558: 27 pgs: 27 active+clean; 1.3 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 667 op/s; 31 KiB/s, 0 objects/s recovering 2024-03-20T09:02:12.266 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:12 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2549318995' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:02:12.271 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:12.277 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:02:12.287 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:02:12.287 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:02:12.287 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:02:12.287 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:02:12.287 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:02:12.287 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:02:12.288 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:02:12.288 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:02:12.288 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:02:12.288 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:02:12.288 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:02:12.288 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:02:12.296 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:02:12.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:12.306 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:02:12.311 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:12 smithi026 ceph-mon[68309]: pgmap v558: 27 pgs: 27 active+clean; 1.3 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 667 op/s; 31 KiB/s, 0 objects/s recovering 2024-03-20T09:02:12.311 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:12 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2549318995' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:02:12.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:12 smithi115 ceph-mon[72476]: pgmap v558: 27 pgs: 27 active+clean; 1.3 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 667 op/s; 31 KiB/s, 0 objects/s recovering 2024-03-20T09:02:12.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:12 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2549318995' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:02:12.407 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:02:12.434 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:12.456 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:02:12.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:12.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:02:12.561 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:02:12.633 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:12.910 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:02:12.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:12.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:02:13.021 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:02:13.071 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:13.113 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:02:13.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:13.116 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:02:13.217 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:02:13.299 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:13.417 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:02:13.472 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:02:13.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:13.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:02:13.589 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:02:13.637 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:13.901 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-03-20T09:02:13.902 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:02:13.902 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-03-20T09:02:13.902 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-20T09:02:13.902 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-20T09:02:13.903 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-03-20T09:02:13.929 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:13 smithi115 ceph-mon[72476]: pgmap v559: 27 pgs: 27 active+clean; 1.2 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 35 KiB/s wr, 842 op/s; 25 KiB/s, 0 objects/s recovering 2024-03-20T09:02:13.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:13 smithi122 ceph-mon[73495]: pgmap v559: 27 pgs: 27 active+clean; 1.2 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 35 KiB/s wr, 842 op/s; 25 KiB/s, 0 objects/s recovering 2024-03-20T09:02:13.976 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:13 smithi026 ceph-mon[68309]: pgmap v559: 27 pgs: 27 active+clean; 1.2 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 35 KiB/s wr, 842 op/s; 25 KiB/s, 0 objects/s recovering 2024-03-20T09:02:14.028 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:02:14.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:14.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:02:14.129 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:02:14.137 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:02:14.181 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:14.215 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:02:14.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:14.218 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:02:14.241 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:02:14.319 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:02:14.351 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:14.554 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:02:14.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:14.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:02:14.659 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:02:14.737 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:14.898 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:02:14.898 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:02:14.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:02:14.898 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.24550.0:221630 2.6 e190/121 2:665735b2:::benchmark_data_smithi122_107654_object102845:head v 190'13941, mlcod=190'13939)", 2024-03-20T09:02:14.898 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:02:14.873731+0000", 2024-03-20T09:02:14.898 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.014905211, 2024-03-20T09:02:14.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014936245000000001, 2024-03-20T09:02:14.898 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:02:14.899 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:02:14.899 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:02:14.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:02:14.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:02:14.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:02:14.873731+0000", 2024-03-20T09:02:14.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:02:14.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:02:14.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:02:14.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:02:14.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:02:14.873731+0000", 2024-03-20T09:02:14.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:02:14.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:02:14.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:02:14.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:02:14.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:02:14.873732+0000", 2024-03-20T09:02:14.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.6200000000000006e-07 2024-03-20T09:02:14.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:02:14.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:02:14.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:02:14.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:02:14.873740+0000", 2024-03-20T09:02:14.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9009999999999994e-06 2024-03-20T09:02:14.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:02:14.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:02:14.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:02:14.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:02:14.873752+0000", 2024-03-20T09:02:14.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2459e-05 2024-03-20T09:02:14.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:02:14.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:02:14.901 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:02:14.901 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:02:14.873760+0000", 2024-03-20T09:02:14.901 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1920000000000005e-06 2024-03-20T09:02:14.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:02:14.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:02:14.901 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:02:14.901 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:02:14.878866+0000", 2024-03-20T09:02:14.901 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0051055440000000001 2024-03-20T09:02:14.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:02:14.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:02:14.901 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:02:14.901 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:02:14.878891+0000", 2024-03-20T09:02:14.901 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4983000000000001e-05 2024-03-20T09:02:14.902 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:02:14.902 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:02:14.902 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:02:14.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:02:14.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:02:14.902 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.24550.0:221631 2.e e190/121 2:7743f7ff:::benchmark_data_smithi122_107654_object102846:head v 190'13905, mlcod=190'13902)", 2024-03-20T09:02:14.902 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:02:14.874822+0000", 2024-03-20T09:02:14.902 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.013813949000000001, 2024-03-20T09:02:14.902 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013918246, 2024-03-20T09:02:14.902 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:02:14.902 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:02:14.902 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:02:14.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:02:14.902 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:02:14.903 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:02:14.874822+0000", 2024-03-20T09:02:14.903 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:02:14.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:02:14.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:02:14.903 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:02:14.903 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:02:14.874822+0000", 2024-03-20T09:02:14.903 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:02:14.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:02:14.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:02:14.903 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:02:14.903 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:02:14.874823+0000", 2024-03-20T09:02:14.903 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2600000000000002e-07 2024-03-20T09:02:14.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:02:14.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:02:14.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:02:14.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:02:14.874831+0000", 2024-03-20T09:02:14.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6399999999999997e-06 2024-03-20T09:02:14.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:02:14.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:02:14.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:02:14.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:02:14.874845+0000", 2024-03-20T09:02:14.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3978000000000001e-05 2024-03-20T09:02:14.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:02:14.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:02:14.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:02:14.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:02:14.874852+0000", 2024-03-20T09:02:14.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7959999999999996e-06 2024-03-20T09:02:14.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:02:14.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:02:14.905 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:02:14.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:02:14.883727+0000", 2024-03-20T09:02:14.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0088744649999999998 2024-03-20T09:02:14.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:02:14.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:02:14.905 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:02:14.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:02:14.883753+0000", 2024-03-20T09:02:14.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6474000000000001e-05 2024-03-20T09:02:14.905 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:02:14.905 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:02:14.905 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:02:14.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:02:14.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:02:14.906 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.24550.0:221637 2.e e190/121)", 2024-03-20T09:02:14.906 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:02:14.879609+0000", 2024-03-20T09:02:14.906 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0090275149999999998, 2024-03-20T09:02:14.906 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092003280000000007, 2024-03-20T09:02:14.906 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:02:14.906 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "queued for pg", 2024-03-20T09:02:14.906 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:02:14.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:02:14.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:02:14.906 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:02:14.879609+0000", 2024-03-20T09:02:14.907 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:02:14.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:02:14.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:02:14.907 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:02:14.907 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:02:14.879609+0000", 2024-03-20T09:02:14.907 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:02:14.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:02:14.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:02:14.907 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:02:14.907 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:02:14.879609+0000", 2024-03-20T09:02:14.907 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7800000000000001e-07 2024-03-20T09:02:14.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:02:14.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:02:14.907 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:02:14.907 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:02:14.879617+0000", 2024-03-20T09:02:14.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3440000000000003e-06 2024-03-20T09:02:14.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:02:14.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:02:14.908 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:02:14.908 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:02:14.879631+0000", 2024-03-20T09:02:14.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4319e-05 2024-03-20T09:02:14.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:02:14.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:02:14.908 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:02:14.908 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:02:14.879639+0000", 2024-03-20T09:02:14.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9969999999999998e-06 2024-03-20T09:02:14.908 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:02:14.908 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:02:14.908 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:02:14.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:02:14.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:02:14.909 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.24550.0:221647 2.a e190/121 2:5f6f5773:::benchmark_data_smithi122_107654_object102862:head v 190'13926, mlcod=190'13924)", 2024-03-20T09:02:14.909 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:02:14.887427+0000", 2024-03-20T09:02:14.909 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0012092159999999999, 2024-03-20T09:02:14.909 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001433398, 2024-03-20T09:02:14.909 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:02:14.909 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:02:14.909 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:02:14.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:02:14.909 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:02:14.909 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:02:14.887427+0000", 2024-03-20T09:02:14.909 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:02:14.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:02:14.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:02:14.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:02:14.910 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:02:14.887427+0000", 2024-03-20T09:02:14.910 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:02:14.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:02:14.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:02:14.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:02:14.910 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:02:14.887428+0000", 2024-03-20T09:02:14.910 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.052e-06 2024-03-20T09:02:14.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:02:14.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:02:14.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:02:14.910 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:02:14.887436+0000", 2024-03-20T09:02:14.910 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3259999999999996e-06 2024-03-20T09:02:14.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:02:14.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:02:14.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:02:14.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:02:14.887450+0000", 2024-03-20T09:02:14.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3176000000000001e-05 2024-03-20T09:02:14.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:02:14.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:02:14.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:02:14.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:02:14.887458+0000", 2024-03-20T09:02:14.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6659999999999997e-06 2024-03-20T09:02:14.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:02:14.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:02:14.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:02:14.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:02:14.887534+0000", 2024-03-20T09:02:14.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6310000000000006e-05 2024-03-20T09:02:14.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:02:14.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:02:14.912 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:02:14.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:02:14.887556+0000", 2024-03-20T09:02:14.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1401000000000001e-05 2024-03-20T09:02:14.912 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:02:14.912 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:02:14.912 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:02:14.912 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:02:14.912 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:02:14.912 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 4 2024-03-20T09:02:14.912 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:02:15.509 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:02:15.666 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:02:15.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:15.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:02:15.784 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:02:15.822 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:15.861 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:02:15.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:15.868 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:02:15.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:15 smithi122 ceph-mon[73495]: pgmap v560: 27 pgs: 27 active+clean; 1.1 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 31 KiB/s wr, 828 op/s; 22 KiB/s, 0 objects/s recovering 2024-03-20T09:02:15.969 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:02:15.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:15 smithi026 ceph-mon[68309]: pgmap v560: 27 pgs: 27 active+clean; 1.1 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 31 KiB/s wr, 828 op/s; 22 KiB/s, 0 objects/s recovering 2024-03-20T09:02:16.007 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:16.045 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:02:16.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:16.048 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:02:16.149 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:02:16.152 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:15 smithi115 ceph-mon[72476]: pgmap v560: 27 pgs: 27 active+clean; 1.1 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 31 KiB/s wr, 828 op/s; 22 KiB/s, 0 objects/s recovering 2024-03-20T09:02:16.179 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:16.216 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:02:16.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:16.224 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:02:16.324 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:02:16.411 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:16.454 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:02:16.454 INFO:teuthology.orchestra.run.smithi115.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-20T09:02:16.454 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:02:16.778 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:02:16.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:16.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:02:16.805 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-20T09:02:16.890 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:02:16.922 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:17.250 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:02:17.277 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:02:17.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:17.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:02:17.382 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:02:17.422 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:17.461 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:02:17.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:17.466 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:02:17.567 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:02:17.634 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:17.707 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:02:17.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:17.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:02:17.812 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:02:17.840 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:17.867 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:02:17.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:17.878 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:02:17.979 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:02:18.019 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:18.046 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:02:18.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:18.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:02:18.077 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:02:18.150 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:02:18.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:17 smithi115 ceph-mon[72476]: pgmap v561: 27 pgs: 27 active+clean; 987 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 921 B/s rd, 27 KiB/s wr, 828 op/s; 19 KiB/s, 0 objects/s recovering 2024-03-20T09:02:18.191 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:18.193 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:17 smithi026 ceph-mon[68309]: pgmap v561: 27 pgs: 27 active+clean; 987 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 921 B/s rd, 27 KiB/s wr, 828 op/s; 19 KiB/s, 0 objects/s recovering 2024-03-20T09:02:18.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:17 smithi122 ceph-mon[73495]: pgmap v561: 27 pgs: 27 active+clean; 987 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 921 B/s rd, 27 KiB/s wr, 828 op/s; 19 KiB/s, 0 objects/s recovering 2024-03-20T09:02:18.215 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:02:18.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:18.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:02:18.320 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:02:18.381 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:18.601 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:02:18.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:18.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:02:18.705 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:02:18.734 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:18.753 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:02:18.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:18.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:02:18.859 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:02:18.889 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:18.910 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:02:18.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:18.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:02:19.018 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:02:19.046 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:19.068 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:02:19.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:19.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:02:19.176 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:02:19.207 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:19.227 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:02:19.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:19.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:02:19.331 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:02:19.377 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:19.399 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:02:19.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:19.402 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:02:19.503 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:02:19.538 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:19.557 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:02:19.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:19.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:02:19.662 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:02:19.674 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:19 smithi122 ceph-mon[73495]: pgmap v562: 27 pgs: 27 active+clean; 886 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 843 op/s 2024-03-20T09:02:19.691 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:19.715 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:02:19.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:19 smithi026 ceph-mon[68309]: pgmap v562: 27 pgs: 27 active+clean; 886 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 843 op/s 2024-03-20T09:02:19.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:19.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:02:19.823 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:02:19.869 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:19.904 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:02:19.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:19 smithi115 ceph-mon[72476]: pgmap v562: 27 pgs: 27 active+clean; 886 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 843 op/s 2024-03-20T09:02:19.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:19.910 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:02:20.011 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:02:20.054 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:20.082 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:02:20.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:20.087 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:02:20.188 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:02:20.228 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:20.262 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:02:20.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:20.267 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:02:20.368 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:02:20.402 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:20.425 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:02:20.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:20.432 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:02:20.533 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:02:20.572 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:20.608 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:02:20.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:20.611 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:02:20.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:20 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/596337242' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:02:20.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:20 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:02:20.702 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:02:20.712 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:02:20.794 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:20 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/596337242' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:02:20.794 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:20 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:02:20.800 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:20.950 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:02:20.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:20.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:02:21.057 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:02:21.067 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:02:21.097 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:02:21.152 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:02:21.152 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:02:21.152 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-03-20T09:02:21.152 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-03-20T09:02:21.152 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:02:21.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:20 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/596337242' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:02:21.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:20 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:02:21.164 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:21.172 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:02:21.233 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:02:21.268 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:02:21.317 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:02:21.360 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:02:21.395 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:02:21.412 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:02:21.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:21.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:02:21.442 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:02:21.494 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:02:21.525 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:02:21.535 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:02:21.552 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:21.568 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:02:21.574 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:02:21.574 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:02:21.574 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:02:21.574 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:02:21.575 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:02:21.575 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:02:21.575 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:02:21.575 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:02:21.575 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:02:21.575 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:02:21.575 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:02:21.575 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:02:21.575 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:02:21.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:21.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:02:21.681 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:02:21.702 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:21 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:02:21.702 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:21 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:02:21.702 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:21 smithi026 ceph-mon[68309]: osdmap e191: 12 total, 12 up, 12 in 2024-03-20T09:02:21.702 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:21 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3589612756' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:02:21.702 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:21 smithi026 ceph-mon[68309]: from='client.24881 172.21.15.26:0/3589612756' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:02:21.702 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:21 smithi026 ceph-mon[68309]: 1.8 scrub starts 2024-03-20T09:02:21.702 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:21 smithi026 ceph-mon[68309]: 1.8 scrub ok 2024-03-20T09:02:21.703 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:21 smithi026 ceph-mon[68309]: pgmap v564: 27 pgs: 27 active+clean; 803 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 863 op/s 2024-03-20T09:02:21.728 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:21.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:21 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:02:21.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:21 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:02:21.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:21 smithi115 ceph-mon[72476]: osdmap e191: 12 total, 12 up, 12 in 2024-03-20T09:02:21.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:21 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3589612756' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:02:21.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:21 smithi115 ceph-mon[72476]: from='client.24881 172.21.15.26:0/3589612756' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:02:21.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:21 smithi115 ceph-mon[72476]: 1.8 scrub starts 2024-03-20T09:02:21.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:21 smithi115 ceph-mon[72476]: 1.8 scrub ok 2024-03-20T09:02:21.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:21 smithi115 ceph-mon[72476]: pgmap v564: 27 pgs: 27 active+clean; 803 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 863 op/s 2024-03-20T09:02:21.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:21 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:02:21.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:21 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:02:21.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:21 smithi122 ceph-mon[73495]: osdmap e191: 12 total, 12 up, 12 in 2024-03-20T09:02:21.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:21 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3589612756' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:02:21.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:21 smithi122 ceph-mon[73495]: from='client.24881 172.21.15.26:0/3589612756' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:02:21.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:21 smithi122 ceph-mon[73495]: 1.8 scrub starts 2024-03-20T09:02:21.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:21 smithi122 ceph-mon[73495]: 1.8 scrub ok 2024-03-20T09:02:21.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:21 smithi122 ceph-mon[73495]: pgmap v564: 27 pgs: 27 active+clean; 803 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 863 op/s 2024-03-20T09:02:22.235 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-03-20T09:02:22.235 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:02:22.235 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-20T09:02:22.235 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 1.000000 2024-03-20T09:02:22.235 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd primary-affinity 4 1 2024-03-20T09:02:22.503 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:02:22.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:22.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:02:22.611 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:02:22.661 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:22.776 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:02:22.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:22.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:02:22.884 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:02:22.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:22 smithi026 ceph-mon[68309]: 1.a scrub starts 2024-03-20T09:02:22.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:22 smithi026 ceph-mon[68309]: 1.a scrub ok 2024-03-20T09:02:22.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:22 smithi026 ceph-mon[68309]: 1.7 scrub starts 2024-03-20T09:02:22.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:22 smithi026 ceph-mon[68309]: 1.7 scrub ok 2024-03-20T09:02:22.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:22 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:02:22.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:22 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:02:22.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:22 smithi026 ceph-mon[68309]: 2.2 scrub starts 2024-03-20T09:02:22.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:22 smithi026 ceph-mon[68309]: 2.2 scrub ok 2024-03-20T09:02:22.971 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:23.098 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:02:23.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:23.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:02:23.140 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-20T09:02:23.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:22 smithi115 ceph-mon[72476]: 1.a scrub starts 2024-03-20T09:02:23.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:22 smithi115 ceph-mon[72476]: 1.a scrub ok 2024-03-20T09:02:23.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:22 smithi115 ceph-mon[72476]: 1.7 scrub starts 2024-03-20T09:02:23.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:22 smithi115 ceph-mon[72476]: 1.7 scrub ok 2024-03-20T09:02:23.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:22 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:02:23.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:22 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:02:23.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:22 smithi115 ceph-mon[72476]: 2.2 scrub starts 2024-03-20T09:02:23.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:22 smithi115 ceph-mon[72476]: 2.2 scrub ok 2024-03-20T09:02:23.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:22 smithi122 ceph-mon[73495]: 1.a scrub starts 2024-03-20T09:02:23.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:22 smithi122 ceph-mon[73495]: 1.a scrub ok 2024-03-20T09:02:23.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:22 smithi122 ceph-mon[73495]: 1.7 scrub starts 2024-03-20T09:02:23.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:22 smithi122 ceph-mon[73495]: 1.7 scrub ok 2024-03-20T09:02:23.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:22 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:02:23.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:22 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:02:23.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:22 smithi122 ceph-mon[73495]: 2.2 scrub starts 2024-03-20T09:02:23.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:22 smithi122 ceph-mon[73495]: 2.2 scrub ok 2024-03-20T09:02:23.207 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:02:23.255 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:23.281 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:02:23.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:23.284 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:02:23.385 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:02:23.454 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:23.531 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:02:23.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:23.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:02:23.636 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:02:23.639 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:02:23.672 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:23.675 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:02:23.694 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:02:23.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:23.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:02:23.799 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:02:23.847 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:23.878 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:02:23.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:23.885 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:02:23.986 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:02:24.038 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:23 smithi026 ceph-mon[68309]: 2.8 scrub starts 2024-03-20T09:02:24.039 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:23 smithi026 ceph-mon[68309]: 2.8 scrub ok 2024-03-20T09:02:24.039 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:23 smithi026 ceph-mon[68309]: osdmap e192: 12 total, 12 up, 12 in 2024-03-20T09:02:24.039 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:23 smithi026 ceph-mon[68309]: 2.6 scrub starts 2024-03-20T09:02:24.039 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:23 smithi026 ceph-mon[68309]: pgmap v566: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 647 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 875 op/s 2024-03-20T09:02:24.046 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:24.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:23 smithi115 ceph-mon[72476]: 2.8 scrub starts 2024-03-20T09:02:24.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:23 smithi115 ceph-mon[72476]: 2.8 scrub ok 2024-03-20T09:02:24.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:23 smithi115 ceph-mon[72476]: osdmap e192: 12 total, 12 up, 12 in 2024-03-20T09:02:24.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:23 smithi115 ceph-mon[72476]: 2.6 scrub starts 2024-03-20T09:02:24.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:23 smithi115 ceph-mon[72476]: pgmap v566: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 647 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 875 op/s 2024-03-20T09:02:24.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:23 smithi122 ceph-mon[73495]: 2.8 scrub starts 2024-03-20T09:02:24.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:23 smithi122 ceph-mon[73495]: 2.8 scrub ok 2024-03-20T09:02:24.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:23 smithi122 ceph-mon[73495]: osdmap e192: 12 total, 12 up, 12 in 2024-03-20T09:02:24.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:23 smithi122 ceph-mon[73495]: 2.6 scrub starts 2024-03-20T09:02:24.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:23 smithi122 ceph-mon[73495]: pgmap v566: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 647 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 875 op/s 2024-03-20T09:02:24.451 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:02:24.777 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:24 smithi026 ceph-mon[68309]: 2.6 scrub ok 2024-03-20T09:02:24.777 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:24 smithi026 ceph-mon[68309]: 1.0 scrub starts 2024-03-20T09:02:24.777 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:24 smithi026 ceph-mon[68309]: 1.0 scrub ok 2024-03-20T09:02:24.778 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:24 smithi026 ceph-mon[68309]: 1.2 scrub starts 2024-03-20T09:02:24.778 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:24 smithi026 ceph-mon[68309]: 1.2 scrub ok 2024-03-20T09:02:24.778 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:24 smithi026 ceph-mon[68309]: 2.1 scrub starts 2024-03-20T09:02:25.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:24 smithi115 ceph-mon[72476]: 2.6 scrub ok 2024-03-20T09:02:25.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:24 smithi115 ceph-mon[72476]: 1.0 scrub starts 2024-03-20T09:02:25.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:24 smithi115 ceph-mon[72476]: 1.0 scrub ok 2024-03-20T09:02:25.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:24 smithi115 ceph-mon[72476]: 1.2 scrub starts 2024-03-20T09:02:25.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:24 smithi115 ceph-mon[72476]: 1.2 scrub ok 2024-03-20T09:02:25.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:24 smithi115 ceph-mon[72476]: 2.1 scrub starts 2024-03-20T09:02:25.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:24 smithi122 ceph-mon[73495]: 2.6 scrub ok 2024-03-20T09:02:25.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:24 smithi122 ceph-mon[73495]: 1.0 scrub starts 2024-03-20T09:02:25.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:24 smithi122 ceph-mon[73495]: 1.0 scrub ok 2024-03-20T09:02:25.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:24 smithi122 ceph-mon[73495]: 1.2 scrub starts 2024-03-20T09:02:25.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:24 smithi122 ceph-mon[73495]: 1.2 scrub ok 2024-03-20T09:02:25.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:24 smithi122 ceph-mon[73495]: 2.1 scrub starts 2024-03-20T09:02:25.591 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:02:25.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:25 smithi026 ceph-mon[68309]: 1.4 scrub starts 2024-03-20T09:02:25.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:25 smithi026 ceph-mon[68309]: 1.4 scrub ok 2024-03-20T09:02:25.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:25 smithi026 ceph-mon[68309]: 2.1 scrub ok 2024-03-20T09:02:25.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:25 smithi026 ceph-mon[68309]: 2.d scrub starts 2024-03-20T09:02:25.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:25 smithi026 ceph-mon[68309]: 2.d scrub ok 2024-03-20T09:02:25.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:25 smithi026 ceph-mon[68309]: 2.e scrub starts 2024-03-20T09:02:25.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:25 smithi026 ceph-mon[68309]: pgmap v567: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 547 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 870 op/s 2024-03-20T09:02:26.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:25 smithi115 ceph-mon[72476]: 1.4 scrub starts 2024-03-20T09:02:26.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:25 smithi115 ceph-mon[72476]: 1.4 scrub ok 2024-03-20T09:02:26.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:25 smithi115 ceph-mon[72476]: 2.1 scrub ok 2024-03-20T09:02:26.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:25 smithi115 ceph-mon[72476]: 2.d scrub starts 2024-03-20T09:02:26.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:25 smithi115 ceph-mon[72476]: 2.d scrub ok 2024-03-20T09:02:26.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:25 smithi115 ceph-mon[72476]: 2.e scrub starts 2024-03-20T09:02:26.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:25 smithi115 ceph-mon[72476]: pgmap v567: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 547 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 870 op/s 2024-03-20T09:02:26.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:25 smithi122 ceph-mon[73495]: 1.4 scrub starts 2024-03-20T09:02:26.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:25 smithi122 ceph-mon[73495]: 1.4 scrub ok 2024-03-20T09:02:26.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:25 smithi122 ceph-mon[73495]: 2.1 scrub ok 2024-03-20T09:02:26.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:25 smithi122 ceph-mon[73495]: 2.d scrub starts 2024-03-20T09:02:26.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:25 smithi122 ceph-mon[73495]: 2.d scrub ok 2024-03-20T09:02:26.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:25 smithi122 ceph-mon[73495]: 2.e scrub starts 2024-03-20T09:02:26.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:25 smithi122 ceph-mon[73495]: pgmap v567: 27 pgs: 1 active+clean+scrubbing, 26 active+clean; 547 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 870 op/s 2024-03-20T09:02:26.258 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:02:26.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:26.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:02:26.361 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:02:26.440 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:26.508 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:02:26.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:26.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:02:26.613 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:02:26.645 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:26.666 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:02:26.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:26.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:02:26.770 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:02:26.799 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:26 smithi026 ceph-mon[68309]: 1.3 scrub starts 2024-03-20T09:02:26.799 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:26 smithi026 ceph-mon[68309]: 1.3 scrub ok 2024-03-20T09:02:26.799 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:26 smithi026 ceph-mon[68309]: 2.e scrub ok 2024-03-20T09:02:26.800 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:26 smithi026 ceph-mon[68309]: 1.9 scrub starts 2024-03-20T09:02:26.800 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:26 smithi026 ceph-mon[68309]: 1.9 scrub ok 2024-03-20T09:02:26.800 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:26 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:02:26.800 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:26 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:02:26.800 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:26 smithi026 ceph-mon[68309]: 2.7 scrub starts 2024-03-20T09:02:26.800 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:26 smithi026 ceph-mon[68309]: osdmap e193: 12 total, 12 up, 12 in 2024-03-20T09:02:26.800 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:26 smithi026 ceph-mon[68309]: 2.7 scrub ok 2024-03-20T09:02:26.800 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:26 smithi026 ceph-mon[68309]: 1.1 scrub starts 2024-03-20T09:02:26.800 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:26 smithi026 ceph-mon[68309]: 1.1 scrub ok 2024-03-20T09:02:26.804 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:26.834 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:02:26.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:26.837 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:02:26.855 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:02:26.855 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:02:26.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:26 smithi115 ceph-mon[72476]: 1.3 scrub starts 2024-03-20T09:02:26.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:26 smithi115 ceph-mon[72476]: 1.3 scrub ok 2024-03-20T09:02:26.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:26 smithi115 ceph-mon[72476]: 2.e scrub ok 2024-03-20T09:02:26.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:26 smithi115 ceph-mon[72476]: 1.9 scrub starts 2024-03-20T09:02:26.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:26 smithi115 ceph-mon[72476]: 1.9 scrub ok 2024-03-20T09:02:26.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:26 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:02:26.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:26 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:02:26.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:26 smithi115 ceph-mon[72476]: 2.7 scrub starts 2024-03-20T09:02:26.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:26 smithi115 ceph-mon[72476]: osdmap e193: 12 total, 12 up, 12 in 2024-03-20T09:02:26.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:26 smithi115 ceph-mon[72476]: 2.7 scrub ok 2024-03-20T09:02:26.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:26 smithi115 ceph-mon[72476]: 1.1 scrub starts 2024-03-20T09:02:26.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:26 smithi115 ceph-mon[72476]: 1.1 scrub ok 2024-03-20T09:02:26.937 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:02:26.997 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:27.020 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:02:27.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:27.027 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:02:27.128 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:02:27.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:26 smithi122 ceph-mon[73495]: 1.3 scrub starts 2024-03-20T09:02:27.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:26 smithi122 ceph-mon[73495]: 1.3 scrub ok 2024-03-20T09:02:27.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:26 smithi122 ceph-mon[73495]: 2.e scrub ok 2024-03-20T09:02:27.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:26 smithi122 ceph-mon[73495]: 1.9 scrub starts 2024-03-20T09:02:27.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:26 smithi122 ceph-mon[73495]: 1.9 scrub ok 2024-03-20T09:02:27.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:26 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:02:27.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:26 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:02:27.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:26 smithi122 ceph-mon[73495]: 2.7 scrub starts 2024-03-20T09:02:27.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:26 smithi122 ceph-mon[73495]: osdmap e193: 12 total, 12 up, 12 in 2024-03-20T09:02:27.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:26 smithi122 ceph-mon[73495]: 2.7 scrub ok 2024-03-20T09:02:27.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:26 smithi122 ceph-mon[73495]: 1.1 scrub starts 2024-03-20T09:02:27.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:26 smithi122 ceph-mon[73495]: 1.1 scrub ok 2024-03-20T09:02:27.225 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:27.333 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:02:27.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:27.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:02:27.456 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:02:27.490 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:27.528 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:02:27.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:27.542 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:02:27.643 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:02:27.691 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:27.737 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:02:27.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:27.745 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:02:27.815 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-20T09:02:27.816 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:02:27.846 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:02:27.873 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:02:27.905 INFO:teuthology.orchestra.run.smithi026.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-20T09:02:27.921 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:28.127 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:27 smithi026 ceph-mon[68309]: 2.9 scrub starts 2024-03-20T09:02:28.127 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:27 smithi026 ceph-mon[68309]: 2.9 scrub ok 2024-03-20T09:02:28.128 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:27 smithi026 ceph-mon[68309]: 1.6 scrub starts 2024-03-20T09:02:28.128 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:27 smithi026 ceph-mon[68309]: 1.6 scrub ok 2024-03-20T09:02:28.128 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:27 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2876357730' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-20T09:02:28.128 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:27 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-20T09:02:28.128 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:27 smithi026 ceph-mon[68309]: pgmap v569: 27 pgs: 27 active+clean; 449 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 839 op/s 2024-03-20T09:02:28.136 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:02:28.149 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:02:28.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:28.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:02:28.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:27 smithi115 ceph-mon[72476]: 2.9 scrub starts 2024-03-20T09:02:28.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:27 smithi115 ceph-mon[72476]: 2.9 scrub ok 2024-03-20T09:02:28.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:27 smithi115 ceph-mon[72476]: 1.6 scrub starts 2024-03-20T09:02:28.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:27 smithi115 ceph-mon[72476]: 1.6 scrub ok 2024-03-20T09:02:28.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:27 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2876357730' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-20T09:02:28.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:27 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-20T09:02:28.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:27 smithi115 ceph-mon[72476]: pgmap v569: 27 pgs: 27 active+clean; 449 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 839 op/s 2024-03-20T09:02:28.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:27 smithi122 ceph-mon[73495]: 2.9 scrub starts 2024-03-20T09:02:28.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:27 smithi122 ceph-mon[73495]: 2.9 scrub ok 2024-03-20T09:02:28.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:27 smithi122 ceph-mon[73495]: 1.6 scrub starts 2024-03-20T09:02:28.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:27 smithi122 ceph-mon[73495]: 1.6 scrub ok 2024-03-20T09:02:28.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:27 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2876357730' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-20T09:02:28.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:27 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-20T09:02:28.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:27 smithi122 ceph-mon[73495]: pgmap v569: 27 pgs: 27 active+clean; 449 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 839 op/s 2024-03-20T09:02:28.257 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:02:28.286 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:28.315 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:02:28.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:28.321 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:02:28.423 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:02:28.465 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:28.521 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:02:28.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:28.527 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:02:28.628 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:02:28.664 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:28.893 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:28 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-20T09:02:28.894 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:28 smithi026 ceph-mon[68309]: osdmap e194: 12 total, 12 up, 12 in 2024-03-20T09:02:28.894 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:28 smithi026 ceph-mon[68309]: 1.5 scrub starts 2024-03-20T09:02:28.894 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:28 smithi026 ceph-mon[68309]: 1.5 scrub ok 2024-03-20T09:02:28.894 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:28 smithi026 ceph-mon[68309]: 2.3 scrub starts 2024-03-20T09:02:28.894 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:28 smithi026 ceph-mon[68309]: 2.3 scrub ok 2024-03-20T09:02:29.119 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:02:29.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:29.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:02:29.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:28 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-20T09:02:29.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:28 smithi115 ceph-mon[72476]: osdmap e194: 12 total, 12 up, 12 in 2024-03-20T09:02:29.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:28 smithi115 ceph-mon[72476]: 1.5 scrub starts 2024-03-20T09:02:29.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:28 smithi115 ceph-mon[72476]: 1.5 scrub ok 2024-03-20T09:02:29.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:28 smithi115 ceph-mon[72476]: 2.3 scrub starts 2024-03-20T09:02:29.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:28 smithi115 ceph-mon[72476]: 2.3 scrub ok 2024-03-20T09:02:29.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:28 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-20T09:02:29.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:28 smithi122 ceph-mon[73495]: osdmap e194: 12 total, 12 up, 12 in 2024-03-20T09:02:29.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:28 smithi122 ceph-mon[73495]: 1.5 scrub starts 2024-03-20T09:02:29.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:28 smithi122 ceph-mon[73495]: 1.5 scrub ok 2024-03-20T09:02:29.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:28 smithi122 ceph-mon[73495]: 2.3 scrub starts 2024-03-20T09:02:29.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:28 smithi122 ceph-mon[73495]: 2.3 scrub ok 2024-03-20T09:02:29.225 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:02:29.278 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:29.321 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:02:29.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:29.329 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:02:29.430 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:02:29.473 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:29.525 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:02:29.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:29.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:02:29.632 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:02:29.666 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:29.688 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:02:29.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:29.692 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:02:29.793 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:02:29.834 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:29.872 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:02:29.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:29.875 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:02:29.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:29 smithi122 ceph-mon[73495]: pgmap v571: 27 pgs: 27 active+clean; 348 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 838 op/s 2024-03-20T09:02:29.977 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:02:30.025 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:29 smithi026 ceph-mon[68309]: pgmap v571: 27 pgs: 27 active+clean; 348 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 838 op/s 2024-03-20T09:02:30.035 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:30.253 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:29 smithi115 ceph-mon[72476]: pgmap v571: 27 pgs: 27 active+clean; 348 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 838 op/s 2024-03-20T09:02:30.910 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:02:30.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:30.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:02:31.015 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:02:31.093 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:31.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:30 smithi122 ceph-mon[73495]: 2.f scrub starts 2024-03-20T09:02:31.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:30 smithi122 ceph-mon[73495]: 2.f scrub ok 2024-03-20T09:02:31.207 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:02:31.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:31.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:30 smithi026 ceph-mon[68309]: 2.f scrub starts 2024-03-20T09:02:31.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:30 smithi026 ceph-mon[68309]: 2.f scrub ok 2024-03-20T09:02:31.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:02:31.325 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:02:31.353 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:30 smithi115 ceph-mon[72476]: 2.f scrub starts 2024-03-20T09:02:31.353 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:30 smithi115 ceph-mon[72476]: 2.f scrub ok 2024-03-20T09:02:31.355 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:31.691 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:02:31.743 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:02:31.781 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:02:31.829 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:02:31.863 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:02:31.895 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:02:31.991 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:02:32.026 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:02:32.027 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:02:32.050 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:02:32.120 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:02:32.171 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:02:32.186 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:31 smithi115 ceph-mon[72476]: 2.a scrub starts 2024-03-20T09:02:32.186 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:31 smithi115 ceph-mon[72476]: 2.a scrub ok 2024-03-20T09:02:32.186 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:31 smithi115 ceph-mon[72476]: 2.b scrub starts 2024-03-20T09:02:32.186 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:31 smithi115 ceph-mon[72476]: 2.b scrub ok 2024-03-20T09:02:32.186 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:31 smithi115 ceph-mon[72476]: pgmap v572: 27 pgs: 27 active+clean; 264 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 746 op/s 2024-03-20T09:02:32.187 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:31 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/656170327' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:02:32.191 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:02:32.191 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:02:32.191 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:02:32.191 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:02:32.191 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:02:32.191 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:02:32.191 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:02:32.192 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:02:32.192 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:02:32.192 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:02:32.192 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:02:32.192 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:02:32.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:31 smithi122 ceph-mon[73495]: 2.a scrub starts 2024-03-20T09:02:32.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:31 smithi122 ceph-mon[73495]: 2.a scrub ok 2024-03-20T09:02:32.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:31 smithi122 ceph-mon[73495]: 2.b scrub starts 2024-03-20T09:02:32.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:31 smithi122 ceph-mon[73495]: 2.b scrub ok 2024-03-20T09:02:32.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:31 smithi122 ceph-mon[73495]: pgmap v572: 27 pgs: 27 active+clean; 264 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 746 op/s 2024-03-20T09:02:32.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:31 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/656170327' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:02:32.212 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:02:32.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:32.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:02:32.217 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:31 smithi026 ceph-mon[68309]: 2.a scrub starts 2024-03-20T09:02:32.217 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:31 smithi026 ceph-mon[68309]: 2.a scrub ok 2024-03-20T09:02:32.217 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:31 smithi026 ceph-mon[68309]: 2.b scrub starts 2024-03-20T09:02:32.217 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:31 smithi026 ceph-mon[68309]: 2.b scrub ok 2024-03-20T09:02:32.217 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:31 smithi026 ceph-mon[68309]: pgmap v572: 27 pgs: 27 active+clean; 264 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 746 op/s 2024-03-20T09:02:32.217 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:31 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/656170327' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:02:32.317 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:02:32.347 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:32.548 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:02:32.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:32.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:02:32.655 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:02:32.692 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:32.733 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:02:32.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:32.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:02:32.839 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:02:32.885 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:32.903 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:02:32.903 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [ 2024-03-20T09:02:32.903 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:32.903 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:237221 2.f e194/121 2:fee35c71:::benchmark_data_smithi122_107654_object118436:head v 194'14560, mlcod=194'14558)", 2024-03-20T09:02:32.903 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:02:32.898857+0000", 2024-03-20T09:02:32.903 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.0042596259999999999, 2024-03-20T09:02:32.903 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.004287852, 2024-03-20T09:02:32.904 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:32.904 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:02:32.904 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:32.904 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:32.904 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:32.904 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:32.898857+0000", 2024-03-20T09:02:32.904 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:32.904 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:32.904 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:32.904 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:32.904 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:32.898857+0000", 2024-03-20T09:02:32.904 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:32.904 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:32.904 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:32.905 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:32.905 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:32.898858+0000", 2024-03-20T09:02:32.905 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0759999999999999e-06 2024-03-20T09:02:32.905 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:32.905 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:32.905 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:32.905 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:32.898866+0000", 2024-03-20T09:02:32.905 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.2879999999999993e-06 2024-03-20T09:02:32.905 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:32.905 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:32.905 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:32.905 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:32.898876+0000", 2024-03-20T09:02:32.905 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.9040000000000004e-06 2024-03-20T09:02:32.905 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:32.906 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:32.906 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:32.906 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:32.898886+0000", 2024-03-20T09:02:32.906 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0015e-05 2024-03-20T09:02:32.906 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:32.906 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:32.906 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:32.906 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:32.898952+0000", 2024-03-20T09:02:32.906 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.5803999999999995e-05 2024-03-20T09:02:32.906 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:32.906 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:32.906 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:32.907 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:32.898980+0000", 2024-03-20T09:02:32.907 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8104999999999999e-05 2024-03-20T09:02:32.907 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:32.907 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:32.907 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:32.907 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:32.907 INFO:teuthology.orchestra.run.smithi115.stdout: ], 2024-03-20T09:02:32.907 INFO:teuthology.orchestra.run.smithi115.stdout: "num_ops": 1 2024-03-20T09:02:32.907 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:02:32.913 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:02:32.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:32.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:02:33.019 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:02:33.063 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:33.103 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:02:33.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:33.108 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:02:33.209 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:02:33.245 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:33.269 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Removed 118782 objects 2024-03-20T09:02:33.270 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Clean up completed and total clean up time :157.819 2024-03-20T09:02:33.320 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:02:33.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:33.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:02:33.428 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:02:33.455 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:33.474 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:02:33.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:33.476 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:02:33.577 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:02:33.604 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:33.622 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:02:33.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:33.625 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:02:33.701 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-20T09:02:33.726 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:02:33.759 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:33.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:33 smithi122 ceph-mon[73495]: pgmap v573: 27 pgs: 27 active+clean; 113 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 852 op/s 2024-03-20T09:02:33.957 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:02:33.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:33 smithi026 ceph-mon[68309]: pgmap v573: 27 pgs: 27 active+clean; 113 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 852 op/s 2024-03-20T09:02:33.961 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:02:33.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:33.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:02:34.066 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:02:34.091 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:33 smithi115 ceph-mon[72476]: pgmap v573: 27 pgs: 27 active+clean; 113 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 852 op/s 2024-03-20T09:02:34.094 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:34.191 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:02:34.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:34.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:02:34.295 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:02:34.324 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:34.343 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:02:34.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:34.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:02:34.447 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:02:34.475 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:34.495 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:02:34.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:34.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:02:34.520 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-03-20T09:02:34.521 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-03-20T09:02:34.572 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-03-20T09:02:34.573 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:02:34.573 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-03-20T09:02:34.573 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight-by-utilization 110 3.0 2024-03-20T09:02:34.598 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:02:34.653 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:34.678 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:02:34.679 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:02:34.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:34.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:02:34.782 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:02:34.803 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:34 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1833415325' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:02:34.812 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:34.910 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:02:34.912 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:34 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1833415325' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:02:34.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:34.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:02:34.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:34 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1833415325' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:02:35.014 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:02:35.047 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:35.068 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:02:35.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:35.072 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:02:35.173 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:02:35.204 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:35.369 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:02:35.374 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:02:35.453 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:02:35.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:35.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:02:35.557 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:02:35.585 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:35.623 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:02:35.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:35.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:02:35.721 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:02:35.721 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:02:35.721 INFO:teuthology.orchestra.run.smithi115.stdout: "complaint_time": 30, 2024-03-20T09:02:35.721 INFO:teuthology.orchestra.run.smithi115.stdout: "num_blocked_ops": 0 2024-03-20T09:02:35.721 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:02:35.727 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:02:35.742 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:35 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:02:35.742 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:35 smithi115 ceph-mon[72476]: Cluster is now healthy 2024-03-20T09:02:35.742 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:35 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1833415325' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:02:35.742 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:35 smithi115 ceph-mon[72476]: osdmap e195: 12 total, 12 up, 12 in 2024-03-20T09:02:35.742 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:35 smithi115 ceph-mon[72476]: pgmap v575: 27 pgs: 27 active+clean; 39 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 810 op/s 2024-03-20T09:02:35.760 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:35.782 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:02:35.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:35.785 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:02:35.853 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:35 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:02:35.853 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:35 smithi026 ceph-mon[68309]: Cluster is now healthy 2024-03-20T09:02:35.853 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:35 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1833415325' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:02:35.853 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:35 smithi026 ceph-mon[68309]: osdmap e195: 12 total, 12 up, 12 in 2024-03-20T09:02:35.853 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:35 smithi026 ceph-mon[68309]: pgmap v575: 27 pgs: 27 active+clean; 39 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 810 op/s 2024-03-20T09:02:35.886 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:02:35.915 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:35.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:35 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:02:35.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:35 smithi122 ceph-mon[73495]: Cluster is now healthy 2024-03-20T09:02:35.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:35 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1833415325' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:02:35.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:35 smithi122 ceph-mon[73495]: osdmap e195: 12 total, 12 up, 12 in 2024-03-20T09:02:35.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:35 smithi122 ceph-mon[73495]: pgmap v575: 27 pgs: 27 active+clean; 39 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 810 op/s 2024-03-20T09:02:35.986 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:02:36.012 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:02:36.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:36.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:02:36.118 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:02:36.146 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:36.168 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:02:36.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:36.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:02:36.272 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:02:36.304 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:36.326 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:02:36.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:36.329 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:02:36.430 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:02:36.458 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:36.491 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-20T09:02:36.556 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:02:36.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:36.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:02:36.660 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:02:36.692 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:36.713 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:02:36.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:36.716 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:02:36.746 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:02:36.816 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:02:36.846 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:36.868 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:02:36.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:36.871 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:02:36.972 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:02:37.001 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:37.021 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:02:37.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:37.023 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:02:37.124 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:02:37.153 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:37.377 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:02:37.455 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:02:37.680 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:02:37.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:37.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:02:37.785 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:02:37.817 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:37.838 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:02:37.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:37.842 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:02:37.943 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:02:37.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:37 smithi026 ceph-mon[68309]: osdmap e196: 12 total, 12 up, 12 in 2024-03-20T09:02:37.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:37 smithi026 ceph-mon[68309]: pgmap v577: 27 pgs: 27 active+clean; 3.7 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 681 op/s 2024-03-20T09:02:37.972 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:37.993 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:02:37.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:37.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:02:38.024 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:37 smithi115 ceph-mon[72476]: osdmap e196: 12 total, 12 up, 12 in 2024-03-20T09:02:38.024 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:37 smithi115 ceph-mon[72476]: pgmap v577: 27 pgs: 27 active+clean; 3.7 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 681 op/s 2024-03-20T09:02:38.097 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:02:38.126 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:38.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:37 smithi122 ceph-mon[73495]: osdmap e196: 12 total, 12 up, 12 in 2024-03-20T09:02:38.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:37 smithi122 ceph-mon[73495]: pgmap v577: 27 pgs: 27 active+clean; 3.7 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 681 op/s 2024-03-20T09:02:38.674 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:02:38.674 INFO:teuthology.orchestra.run.smithi115.stdout: "size": 20, 2024-03-20T09:02:38.674 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 600, 2024-03-20T09:02:38.674 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [ 2024-03-20T09:02:38.674 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.675 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:152768 2.d e159/121 2:b73f0106:::benchmark_data_smithi122_107654_object33983:head v 159'9528, mlcod=159'9525)", 2024-03-20T09:02:38.675 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:00:29.749867+0000", 2024-03-20T09:02:38.675 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 128.922149305, 2024-03-20T09:02:38.675 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.017064526, 2024-03-20T09:02:38.675 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:38.675 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:38.675 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:38.675 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.675 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:38.675 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:00:29.749867+0000", 2024-03-20T09:02:38.675 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.675 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.675 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.675 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:38.676 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:00:29.749867+0000", 2024-03-20T09:02:38.676 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.676 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.676 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.676 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:38.676 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:00:29.749867+0000", 2024-03-20T09:02:38.676 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.41e-07 2024-03-20T09:02:38.676 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.676 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.676 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:38.676 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:00:29.749871+0000", 2024-03-20T09:02:38.676 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.631e-06 2024-03-20T09:02:38.676 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.676 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.677 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:38.677 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:00:29.749876+0000", 2024-03-20T09:02:38.677 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1769999999999997e-06 2024-03-20T09:02:38.677 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.677 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.677 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:38.677 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:00:29.749880+0000", 2024-03-20T09:02:38.677 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4349999999999999e-06 2024-03-20T09:02:38.677 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.677 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.677 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:38.677 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:00:29.753500+0000", 2024-03-20T09:02:38.677 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0036193699999999998 2024-03-20T09:02:38.678 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.678 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.678 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:38.678 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:00:29.753522+0000", 2024-03-20T09:02:38.678 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2560000000000001e-05 2024-03-20T09:02:38.678 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.678 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.678 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:38.678 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:00:29.766896+0000", 2024-03-20T09:02:38.678 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.013373599 2024-03-20T09:02:38.678 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.678 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.678 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:38.679 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:00:29.766931+0000", 2024-03-20T09:02:38.679 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.5512999999999997e-05 2024-03-20T09:02:38.679 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.679 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:38.679 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.679 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.679 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.679 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:175615 2.d e171/121 2:b8c0e4d3:::benchmark_data_smithi122_107654_object56830:head v 171'10882, mlcod=171'10879)", 2024-03-20T09:02:38.679 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:01:21.380359+0000", 2024-03-20T09:02:38.679 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 77.291657125, 2024-03-20T09:02:38.679 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.019220312999999999, 2024-03-20T09:02:38.679 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:38.679 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:38.680 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:38.680 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.680 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:38.680 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:21.380359+0000", 2024-03-20T09:02:38.680 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.680 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.680 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.680 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:38.680 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:21.380359+0000", 2024-03-20T09:02:38.680 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.680 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.680 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.680 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:38.680 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:21.380360+0000", 2024-03-20T09:02:38.681 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.35e-06 2024-03-20T09:02:38.681 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.681 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.681 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:38.681 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:21.380367+0000", 2024-03-20T09:02:38.681 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.8510000000000001e-06 2024-03-20T09:02:38.681 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.681 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.681 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:38.681 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:21.380380+0000", 2024-03-20T09:02:38.681 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2927e-05 2024-03-20T09:02:38.681 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.681 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.681 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:38.682 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:21.380387+0000", 2024-03-20T09:02:38.682 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.4510000000000003e-06 2024-03-20T09:02:38.682 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.682 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.682 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:38.682 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:21.380441+0000", 2024-03-20T09:02:38.682 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3452999999999998e-05 2024-03-20T09:02:38.682 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.682 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.682 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:38.682 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:21.380464+0000", 2024-03-20T09:02:38.682 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3405999999999999e-05 2024-03-20T09:02:38.682 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.683 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.683 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:38.683 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:21.399532+0000", 2024-03-20T09:02:38.683 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.019067229000000002 2024-03-20T09:02:38.683 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.683 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.683 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:38.683 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:21.399579+0000", 2024-03-20T09:02:38.683 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.7645999999999997e-05 2024-03-20T09:02:38.683 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.683 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:38.683 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.683 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.683 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.684 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:175617 2.d e171/121 2:bdf2ee32:::benchmark_data_smithi122_107654_object56832:head v 171'10883, mlcod=171'10879)", 2024-03-20T09:02:38.684 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:01:21.382335+0000", 2024-03-20T09:02:38.684 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 77.289680677999996, 2024-03-20T09:02:38.684 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.017278963000000001, 2024-03-20T09:02:38.684 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:38.684 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:38.684 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:38.684 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.684 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:38.684 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:21.382335+0000", 2024-03-20T09:02:38.684 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.684 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.684 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.684 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:38.685 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:21.382335+0000", 2024-03-20T09:02:38.685 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.685 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.685 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.685 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:38.685 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:21.382336+0000", 2024-03-20T09:02:38.685 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.5099999999999999e-07 2024-03-20T09:02:38.685 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.685 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.685 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:38.685 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:21.382342+0000", 2024-03-20T09:02:38.685 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7089999999999999e-06 2024-03-20T09:02:38.685 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.685 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.685 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:38.686 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:21.382351+0000", 2024-03-20T09:02:38.686 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.3850000000000008e-06 2024-03-20T09:02:38.686 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.686 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.686 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:38.686 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:21.382356+0000", 2024-03-20T09:02:38.686 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1660000000000002e-06 2024-03-20T09:02:38.686 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.686 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.686 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:38.686 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:21.384254+0000", 2024-03-20T09:02:38.686 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0018981549999999999 2024-03-20T09:02:38.686 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.686 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.687 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:38.687 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:21.384270+0000", 2024-03-20T09:02:38.687 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5187e-05 2024-03-20T09:02:38.687 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.687 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.687 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:38.687 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:21.399585+0000", 2024-03-20T09:02:38.687 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.01531545 2024-03-20T09:02:38.687 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.687 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.687 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:38.687 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:21.399614+0000", 2024-03-20T09:02:38.687 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9260000000000001e-05 2024-03-20T09:02:38.687 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.687 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:38.688 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.688 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.688 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.688 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:175623 2.d e171/121 2:b1dd92b1:::benchmark_data_smithi122_107654_object56838:head v 171'10884, mlcod=171'10880)", 2024-03-20T09:02:38.688 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:01:21.386378+0000", 2024-03-20T09:02:38.688 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 77.285637503000004, 2024-03-20T09:02:38.688 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.019248683999999999, 2024-03-20T09:02:38.688 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:38.688 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:38.688 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:38.688 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.688 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:38.688 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:21.386378+0000", 2024-03-20T09:02:38.689 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.689 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.689 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.689 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:38.689 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:21.386378+0000", 2024-03-20T09:02:38.689 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.689 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.689 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.689 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:38.689 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:21.386380+0000", 2024-03-20T09:02:38.689 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0189999999999999e-06 2024-03-20T09:02:38.689 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.689 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.689 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:38.690 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:21.386389+0000", 2024-03-20T09:02:38.690 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.7089999999999997e-06 2024-03-20T09:02:38.690 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.690 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.690 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:38.690 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:21.386404+0000", 2024-03-20T09:02:38.690 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5114e-05 2024-03-20T09:02:38.690 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.690 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.690 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:38.690 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:21.386413+0000", 2024-03-20T09:02:38.690 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.6489999999999997e-06 2024-03-20T09:02:38.690 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.690 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.691 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:38.691 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:21.392359+0000", 2024-03-20T09:02:38.691 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0059456020000000004 2024-03-20T09:02:38.691 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.691 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.691 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:38.691 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:21.392382+0000", 2024-03-20T09:02:38.691 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3462999999999999e-05 2024-03-20T09:02:38.691 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.691 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.691 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:38.691 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:21.405591+0000", 2024-03-20T09:02:38.691 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.013208981 2024-03-20T09:02:38.691 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.692 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.692 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:38.692 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:21.405627+0000", 2024-03-20T09:02:38.692 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6146999999999998e-05 2024-03-20T09:02:38.692 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.692 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:38.692 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.692 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.692 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.692 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:185588 2.5 e175/121 2:a570ae3d:::benchmark_data_smithi122_107654_object66803:head v 175'11606, mlcod=175'11602)", 2024-03-20T09:02:38.692 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:01:32.832255+0000", 2024-03-20T09:02:38.692 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 65.839761179999996, 2024-03-20T09:02:38.692 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.020438211000000001, 2024-03-20T09:02:38.692 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:38.693 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:38.693 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:38.693 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.693 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:38.693 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832255+0000", 2024-03-20T09:02:38.693 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.693 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.693 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.693 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:38.693 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832255+0000", 2024-03-20T09:02:38.693 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.693 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.693 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.693 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:38.694 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832255+0000", 2024-03-20T09:02:38.694 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7100000000000001e-07 2024-03-20T09:02:38.694 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.694 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.694 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:38.694 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832257+0000", 2024-03-20T09:02:38.694 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2249999999999999e-06 2024-03-20T09:02:38.694 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.694 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.694 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:38.694 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832260+0000", 2024-03-20T09:02:38.694 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.2100000000000002e-06 2024-03-20T09:02:38.694 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.694 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.694 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:38.695 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832263+0000", 2024-03-20T09:02:38.695 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6199999999999999e-06 2024-03-20T09:02:38.695 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.695 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.695 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:38.695 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.835395+0000", 2024-03-20T09:02:38.695 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0031315959999999999 2024-03-20T09:02:38.695 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.695 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.695 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:38.695 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.835408+0000", 2024-03-20T09:02:38.695 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.328e-05 2024-03-20T09:02:38.695 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.695 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.696 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:38.696 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.852650+0000", 2024-03-20T09:02:38.696 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.017242499000000001 2024-03-20T09:02:38.696 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.696 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.696 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:38.696 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.852693+0000", 2024-03-20T09:02:38.696 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.261e-05 2024-03-20T09:02:38.696 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.696 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:38.696 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.696 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.696 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.696 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:185591 2.5 e175/121 2:aefd965d:::benchmark_data_smithi122_107654_object66806:head v 175'11607, mlcod=175'11602)", 2024-03-20T09:02:38.697 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:01:32.832267+0000", 2024-03-20T09:02:38.697 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 65.839748599999993, 2024-03-20T09:02:38.697 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.02045632, 2024-03-20T09:02:38.697 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:38.697 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:38.697 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:38.697 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.697 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:38.697 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832267+0000", 2024-03-20T09:02:38.697 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.697 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.697 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.697 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:38.697 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832267+0000", 2024-03-20T09:02:38.698 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.698 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.698 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.698 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:38.698 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832268+0000", 2024-03-20T09:02:38.698 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.14e-07 2024-03-20T09:02:38.698 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.698 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.698 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:38.698 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832274+0000", 2024-03-20T09:02:38.698 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4320000000000004e-06 2024-03-20T09:02:38.698 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.698 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.698 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:38.699 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832277+0000", 2024-03-20T09:02:38.699 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.3799999999999998e-06 2024-03-20T09:02:38.699 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.699 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.699 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:38.699 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832280+0000", 2024-03-20T09:02:38.699 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.1080000000000001e-06 2024-03-20T09:02:38.699 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.699 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.699 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:38.699 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.838235+0000", 2024-03-20T09:02:38.699 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0059548409999999998 2024-03-20T09:02:38.699 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.699 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.700 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:38.700 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.838247+0000", 2024-03-20T09:02:38.700 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2051e-05 2024-03-20T09:02:38.700 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.700 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.700 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:38.700 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.852697+0000", 2024-03-20T09:02:38.700 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.014449267 2024-03-20T09:02:38.700 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.700 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.700 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:38.700 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.852724+0000", 2024-03-20T09:02:38.700 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.7127e-05 2024-03-20T09:02:38.700 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.700 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:38.701 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.701 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.701 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.701 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:185615 2.5 e175/121 2:a2e93340:::benchmark_data_smithi122_107654_object66830:head v 175'11608, mlcod=175'11602)", 2024-03-20T09:02:38.701 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:01:32.832297+0000", 2024-03-20T09:02:38.701 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 65.839719407000004, 2024-03-20T09:02:38.701 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.030412412999999999, 2024-03-20T09:02:38.701 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:38.701 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:38.701 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:38.701 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.701 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:38.701 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832297+0000", 2024-03-20T09:02:38.701 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.702 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.702 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.702 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:38.702 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832297+0000", 2024-03-20T09:02:38.702 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.702 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.702 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.702 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:38.702 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832297+0000", 2024-03-20T09:02:38.702 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.18e-07 2024-03-20T09:02:38.702 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.702 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.702 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:38.703 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832299+0000", 2024-03-20T09:02:38.703 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9099999999999999e-06 2024-03-20T09:02:38.703 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.703 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.703 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:38.703 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832301+0000", 2024-03-20T09:02:38.703 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8250000000000001e-06 2024-03-20T09:02:38.703 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.703 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.703 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:38.703 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832304+0000", 2024-03-20T09:02:38.703 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9040000000000001e-06 2024-03-20T09:02:38.703 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.703 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.703 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:38.704 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.841247+0000", 2024-03-20T09:02:38.704 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0089422189999999995 2024-03-20T09:02:38.704 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.704 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.704 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:38.704 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.841268+0000", 2024-03-20T09:02:38.704 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1427e-05 2024-03-20T09:02:38.704 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.704 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.704 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:38.704 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.862669+0000", 2024-03-20T09:02:38.704 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.021401153999999999 2024-03-20T09:02:38.704 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.704 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.705 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:38.705 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.862709+0000", 2024-03-20T09:02:38.705 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.9855999999999999e-05 2024-03-20T09:02:38.705 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.705 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:38.705 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.705 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.705 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.705 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:185628 2.5 e175/121 2:a6144487:::benchmark_data_smithi122_107654_object66843:head v 175'11609, mlcod=175'11602)", 2024-03-20T09:02:38.705 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:01:32.832309+0000", 2024-03-20T09:02:38.705 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 65.839706645999996, 2024-03-20T09:02:38.705 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.030430852000000001, 2024-03-20T09:02:38.705 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:38.705 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:38.705 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:38.706 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.706 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:38.706 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832309+0000", 2024-03-20T09:02:38.706 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.706 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.706 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.706 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:38.706 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832309+0000", 2024-03-20T09:02:38.706 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.706 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.706 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.706 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:38.706 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832309+0000", 2024-03-20T09:02:38.706 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4600000000000001e-07 2024-03-20T09:02:38.707 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.707 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.707 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:38.707 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832312+0000", 2024-03-20T09:02:38.707 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.362e-06 2024-03-20T09:02:38.707 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.707 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.707 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:38.707 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832315+0000", 2024-03-20T09:02:38.707 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9699999999999999e-06 2024-03-20T09:02:38.707 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.707 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.707 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:38.707 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832317+0000", 2024-03-20T09:02:38.708 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.441e-06 2024-03-20T09:02:38.708 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.708 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.708 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:38.708 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.844932+0000", 2024-03-20T09:02:38.708 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.012614428 2024-03-20T09:02:38.708 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.708 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.708 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:38.708 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.844946+0000", 2024-03-20T09:02:38.708 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4243000000000001e-05 2024-03-20T09:02:38.708 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.708 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.708 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:38.709 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.862713+0000", 2024-03-20T09:02:38.709 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.017766889000000001 2024-03-20T09:02:38.709 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.709 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.709 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:38.709 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.862740+0000", 2024-03-20T09:02:38.709 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.7373e-05 2024-03-20T09:02:38.709 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.709 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:38.709 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.709 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.709 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.709 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:185645 2.5 e175/121 2:a013e27a:::benchmark_data_smithi122_107654_object66860:head v 175'11610, mlcod=175'11602)", 2024-03-20T09:02:38.709 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:01:32.832331+0000", 2024-03-20T09:02:38.710 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 65.839684500999994, 2024-03-20T09:02:38.710 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.034058443000000001, 2024-03-20T09:02:38.710 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:38.710 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:38.710 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:38.710 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.710 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:38.710 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832331+0000", 2024-03-20T09:02:38.710 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.710 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.710 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.710 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:38.710 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832331+0000", 2024-03-20T09:02:38.711 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.711 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.711 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.711 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:38.711 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832332+0000", 2024-03-20T09:02:38.711 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3300000000000001e-07 2024-03-20T09:02:38.711 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.711 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.711 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:38.711 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832334+0000", 2024-03-20T09:02:38.711 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0760000000000001e-06 2024-03-20T09:02:38.711 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.711 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.711 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:38.712 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832337+0000", 2024-03-20T09:02:38.712 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8320000000000002e-06 2024-03-20T09:02:38.712 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.712 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.712 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:38.712 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832339+0000", 2024-03-20T09:02:38.712 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.244e-06 2024-03-20T09:02:38.712 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.712 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.712 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:38.712 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.848364+0000", 2024-03-20T09:02:38.712 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.016025542 2024-03-20T09:02:38.712 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.712 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.712 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:38.713 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.848386+0000", 2024-03-20T09:02:38.713 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1205000000000001e-05 2024-03-20T09:02:38.713 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.713 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.713 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:38.713 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.866350+0000", 2024-03-20T09:02:38.713 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.017964237000000001 2024-03-20T09:02:38.713 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.713 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.713 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:38.713 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.866390+0000", 2024-03-20T09:02:38.713 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.0173999999999999e-05 2024-03-20T09:02:38.713 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.713 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:38.714 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.714 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.714 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.714 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:185672 2.5 e175/121 2:a65db905:::benchmark_data_smithi122_107654_object66887:head v 175'11611, mlcod=175'11602)", 2024-03-20T09:02:38.714 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:01:32.832344+0000", 2024-03-20T09:02:38.714 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 65.839671977999998, 2024-03-20T09:02:38.714 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.034075934000000002, 2024-03-20T09:02:38.714 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:38.714 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:38.714 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:38.714 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.714 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:38.714 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832344+0000", 2024-03-20T09:02:38.714 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.715 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.715 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.715 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:38.715 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832344+0000", 2024-03-20T09:02:38.715 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.715 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.715 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.715 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:38.715 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832344+0000", 2024-03-20T09:02:38.715 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2700000000000001e-07 2024-03-20T09:02:38.715 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.715 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.715 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:38.715 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832346+0000", 2024-03-20T09:02:38.715 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1940000000000001e-06 2024-03-20T09:02:38.715 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.716 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.716 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:38.716 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832350+0000", 2024-03-20T09:02:38.716 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.179e-06 2024-03-20T09:02:38.716 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.716 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.716 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:38.716 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832352+0000", 2024-03-20T09:02:38.716 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3319999999999999e-06 2024-03-20T09:02:38.716 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.716 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.716 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:38.716 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.852766+0000", 2024-03-20T09:02:38.717 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.020414252000000001 2024-03-20T09:02:38.717 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.717 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.717 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:38.717 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.852782+0000", 2024-03-20T09:02:38.717 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5588000000000001e-05 2024-03-20T09:02:38.717 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.717 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.717 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:38.717 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.866392+0000", 2024-03-20T09:02:38.717 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.013610726 2024-03-20T09:02:38.717 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.717 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.717 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:38.718 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.866420+0000", 2024-03-20T09:02:38.718 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.7535999999999999e-05 2024-03-20T09:02:38.718 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.718 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:38.718 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.718 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.718 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.718 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:185682 2.5 e175/121 2:a5587409:::benchmark_data_smithi122_107654_object66897:head v 175'11612, mlcod=175'11602)", 2024-03-20T09:02:38.718 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:01:32.832392+0000", 2024-03-20T09:02:38.718 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 65.839623988, 2024-03-20T09:02:38.718 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.034054375999999997, 2024-03-20T09:02:38.718 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:38.718 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:38.719 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:38.719 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.719 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:38.719 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832392+0000", 2024-03-20T09:02:38.719 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.719 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.719 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.719 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:38.719 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832392+0000", 2024-03-20T09:02:38.719 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.719 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.719 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.719 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:38.719 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832392+0000", 2024-03-20T09:02:38.720 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1999999999999999e-07 2024-03-20T09:02:38.720 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.720 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.720 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:38.720 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832411+0000", 2024-03-20T09:02:38.720 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9235000000000001e-05 2024-03-20T09:02:38.720 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.720 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.720 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:38.720 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832416+0000", 2024-03-20T09:02:38.720 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5669999999999996e-06 2024-03-20T09:02:38.720 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.720 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.720 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:38.721 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832419+0000", 2024-03-20T09:02:38.721 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.4999999999999999e-06 2024-03-20T09:02:38.721 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.721 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.721 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:38.721 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.855706+0000", 2024-03-20T09:02:38.721 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.023286937000000001 2024-03-20T09:02:38.721 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.721 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.721 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:38.721 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.855718+0000", 2024-03-20T09:02:38.721 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1222e-05 2024-03-20T09:02:38.721 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.721 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.721 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:38.722 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.866422+0000", 2024-03-20T09:02:38.722 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.010703957 2024-03-20T09:02:38.722 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.722 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.722 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:38.722 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.866446+0000", 2024-03-20T09:02:38.722 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.4837999999999998e-05 2024-03-20T09:02:38.722 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.722 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:38.722 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.722 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.722 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.722 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:185683 2.5 e175/121 2:a7b575af:::benchmark_data_smithi122_107654_object66898:head v 175'11613, mlcod=175'11602)", 2024-03-20T09:02:38.723 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:01:32.832424+0000", 2024-03-20T09:02:38.723 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 65.839591650000003, 2024-03-20T09:02:38.723 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.038061003000000003, 2024-03-20T09:02:38.723 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:38.723 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:38.723 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:38.723 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.723 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:38.723 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832424+0000", 2024-03-20T09:02:38.723 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.723 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.723 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.723 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:38.723 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832424+0000", 2024-03-20T09:02:38.724 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.724 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.724 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.724 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:38.724 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832424+0000", 2024-03-20T09:02:38.724 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1999999999999999e-07 2024-03-20T09:02:38.724 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.724 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.724 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:38.724 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832426+0000", 2024-03-20T09:02:38.724 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9269999999999999e-06 2024-03-20T09:02:38.724 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.724 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.724 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:38.725 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832430+0000", 2024-03-20T09:02:38.725 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.4939999999999999e-06 2024-03-20T09:02:38.725 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.725 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.725 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:38.725 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832433+0000", 2024-03-20T09:02:38.725 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6709999999999999e-06 2024-03-20T09:02:38.725 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.725 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.725 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:38.725 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.858583+0000", 2024-03-20T09:02:38.725 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.026150679 2024-03-20T09:02:38.725 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.725 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.725 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:38.726 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.858607+0000", 2024-03-20T09:02:38.726 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3536999999999998e-05 2024-03-20T09:02:38.726 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.726 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.726 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:38.726 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.870454+0000", 2024-03-20T09:02:38.726 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.011847206000000001 2024-03-20T09:02:38.726 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.726 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.726 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:38.726 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.870485+0000", 2024-03-20T09:02:38.726 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.1368999999999997e-05 2024-03-20T09:02:38.726 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.726 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:38.727 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.727 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.727 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.727 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:185688 2.5 e175/121 2:af98be93:::benchmark_data_smithi122_107654_object66903:head v 175'11614, mlcod=175'11602)", 2024-03-20T09:02:38.727 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:01:32.832437+0000", 2024-03-20T09:02:38.727 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 65.839578743999994, 2024-03-20T09:02:38.727 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.038076793999999997, 2024-03-20T09:02:38.727 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:38.727 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:38.727 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:38.727 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.727 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:38.727 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832437+0000", 2024-03-20T09:02:38.727 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.728 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.728 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.728 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:38.728 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832437+0000", 2024-03-20T09:02:38.728 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.728 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.728 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.728 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:38.728 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832437+0000", 2024-03-20T09:02:38.728 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.23e-07 2024-03-20T09:02:38.728 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.728 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.728 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:38.728 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832440+0000", 2024-03-20T09:02:38.729 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2419999999999999e-06 2024-03-20T09:02:38.729 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.729 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.729 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:38.729 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832443+0000", 2024-03-20T09:02:38.729 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6090000000000001e-06 2024-03-20T09:02:38.729 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.729 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.729 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:38.729 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.832446+0000", 2024-03-20T09:02:38.729 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5959999999999998e-06 2024-03-20T09:02:38.729 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.729 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.729 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:38.729 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.862781+0000", 2024-03-20T09:02:38.730 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.030335556 2024-03-20T09:02:38.730 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.730 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.730 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:38.730 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.862803+0000", 2024-03-20T09:02:38.730 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1393999999999999e-05 2024-03-20T09:02:38.730 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.730 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.730 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:38.730 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.870488+0000", 2024-03-20T09:02:38.730 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.007684906 2024-03-20T09:02:38.730 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.730 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.731 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:38.731 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.870514+0000", 2024-03-20T09:02:38.731 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6367999999999999e-05 2024-03-20T09:02:38.731 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.731 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:38.731 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.731 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.731 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.731 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:185704 2.d e175/121 2:b8bd90f8:::benchmark_data_smithi122_107654_object66919:head v 175'11485, mlcod=175'11483)", 2024-03-20T09:02:38.731 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:01:32.845737+0000", 2024-03-20T09:02:38.731 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 65.826279143999997, 2024-03-20T09:02:38.731 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.024963553999999999, 2024-03-20T09:02:38.731 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:38.731 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:38.731 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:38.732 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.732 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:38.732 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.845737+0000", 2024-03-20T09:02:38.732 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.732 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.732 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.732 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:38.732 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.845737+0000", 2024-03-20T09:02:38.732 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.732 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.732 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.732 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:38.732 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.845738+0000", 2024-03-20T09:02:38.732 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.4799999999999997e-07 2024-03-20T09:02:38.732 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.733 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.733 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:38.733 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.845746+0000", 2024-03-20T09:02:38.733 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.7949999999999996e-06 2024-03-20T09:02:38.733 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.733 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.733 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:38.733 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.845758+0000", 2024-03-20T09:02:38.733 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1036000000000001e-05 2024-03-20T09:02:38.733 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.733 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.733 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:38.733 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.845765+0000", 2024-03-20T09:02:38.733 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.0129999999999996e-06 2024-03-20T09:02:38.734 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.734 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.734 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:38.734 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.862772+0000", 2024-03-20T09:02:38.734 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.017007186000000001 2024-03-20T09:02:38.734 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.734 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.734 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:38.734 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.862790+0000", 2024-03-20T09:02:38.734 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8692999999999999e-05 2024-03-20T09:02:38.734 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.734 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.734 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:38.734 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.870670+0000", 2024-03-20T09:02:38.735 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0078799920000000006 2024-03-20T09:02:38.735 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.735 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.735 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:38.735 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.870700+0000", 2024-03-20T09:02:38.735 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9991000000000001e-05 2024-03-20T09:02:38.735 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.735 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:38.735 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.735 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.735 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.735 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:185707 2.d e175/121 2:b9799cab:::benchmark_data_smithi122_107654_object66922:head v 175'11486, mlcod=175'11483)", 2024-03-20T09:02:38.735 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:01:32.849624+0000", 2024-03-20T09:02:38.735 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 65.822392355000005, 2024-03-20T09:02:38.735 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.021185671999999999, 2024-03-20T09:02:38.736 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:38.736 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:38.736 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:38.736 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.736 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:38.736 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.849624+0000", 2024-03-20T09:02:38.736 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.736 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.736 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.736 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:38.736 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.849624+0000", 2024-03-20T09:02:38.736 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.736 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.736 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.736 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:38.737 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.849625+0000", 2024-03-20T09:02:38.737 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1009999999999999e-06 2024-03-20T09:02:38.737 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.737 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.737 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:38.737 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.849637+0000", 2024-03-20T09:02:38.737 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1939e-05 2024-03-20T09:02:38.737 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.737 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.737 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:38.737 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.849650+0000", 2024-03-20T09:02:38.737 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2999999999999999e-05 2024-03-20T09:02:38.737 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.737 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.738 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:38.738 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.849658+0000", 2024-03-20T09:02:38.738 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.9999999999999996e-06 2024-03-20T09:02:38.738 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.738 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.738 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:38.738 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.866372+0000", 2024-03-20T09:02:38.738 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.016714545000000001 2024-03-20T09:02:38.738 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.738 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.738 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:38.738 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.866392+0000", 2024-03-20T09:02:38.738 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0111000000000001e-05 2024-03-20T09:02:38.738 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.739 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.739 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:38.739 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.870789+0000", 2024-03-20T09:02:38.739 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0043963659999999996 2024-03-20T09:02:38.739 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.739 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.739 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:38.739 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:32.870809+0000", 2024-03-20T09:02:38.739 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0610000000000001e-05 2024-03-20T09:02:38.739 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.739 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:38.739 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.739 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.739 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.739 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:204675 2.5 e183/121 2:a4081342:::benchmark_data_smithi122_107654_object85890:head v 183'12763, mlcod=183'12761)", 2024-03-20T09:02:38.740 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:01:54.682828+0000", 2024-03-20T09:02:38.740 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 43.989188265999999, 2024-03-20T09:02:38.740 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.016981319000000002, 2024-03-20T09:02:38.740 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:38.740 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:38.740 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:38.740 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.740 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:38.740 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:54.682828+0000", 2024-03-20T09:02:38.740 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.740 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.740 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.740 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:38.740 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:54.682828+0000", 2024-03-20T09:02:38.740 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.741 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.741 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.741 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:38.741 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:54.682828+0000", 2024-03-20T09:02:38.741 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.0900000000000001e-07 2024-03-20T09:02:38.741 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.741 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.741 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:38.741 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:54.682835+0000", 2024-03-20T09:02:38.741 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.6200000000000001e-06 2024-03-20T09:02:38.741 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.741 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.741 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:38.741 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:54.682845+0000", 2024-03-20T09:02:38.741 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.4890000000000001e-06 2024-03-20T09:02:38.742 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.742 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.742 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:38.742 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:54.682852+0000", 2024-03-20T09:02:38.742 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.2230000000000001e-06 2024-03-20T09:02:38.742 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.742 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.742 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:38.742 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:54.686398+0000", 2024-03-20T09:02:38.742 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0035459150000000002 2024-03-20T09:02:38.742 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.742 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.742 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:38.742 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:54.686421+0000", 2024-03-20T09:02:38.743 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3646999999999999e-05 2024-03-20T09:02:38.743 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.743 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.743 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:38.743 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:54.699759+0000", 2024-03-20T09:02:38.743 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.013337587999999999 2024-03-20T09:02:38.743 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.743 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.743 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:38.743 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:54.699809+0000", 2024-03-20T09:02:38.743 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0127999999999998e-05 2024-03-20T09:02:38.743 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.743 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:38.743 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.744 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.744 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.744 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:219186 2.d e190/121 2:bb919463:::benchmark_data_smithi122_107654_object100401:head v 190'13535, mlcod=190'13530)", 2024-03-20T09:02:38.744 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:02:12.033479+0000", 2024-03-20T09:02:38.744 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 26.638537191000001, 2024-03-20T09:02:38.744 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.017258596000000001, 2024-03-20T09:02:38.744 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:38.744 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:38.744 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:38.744 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.744 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:38.744 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:12.033479+0000", 2024-03-20T09:02:38.744 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.744 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.745 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.745 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:38.745 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:12.033479+0000", 2024-03-20T09:02:38.745 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.745 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.745 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.745 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:38.745 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:12.033479+0000", 2024-03-20T09:02:38.745 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9200000000000001e-07 2024-03-20T09:02:38.745 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.745 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.745 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:38.745 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:12.033487+0000", 2024-03-20T09:02:38.745 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.8769999999999993e-06 2024-03-20T09:02:38.746 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.746 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.746 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:38.746 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:12.033499+0000", 2024-03-20T09:02:38.746 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2014999999999999e-05 2024-03-20T09:02:38.746 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.746 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.746 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:38.746 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:12.033506+0000", 2024-03-20T09:02:38.746 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.9569999999999996e-06 2024-03-20T09:02:38.746 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.746 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.746 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:38.746 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:12.036974+0000", 2024-03-20T09:02:38.747 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0034676360000000001 2024-03-20T09:02:38.747 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.747 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.747 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:38.747 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:12.036992+0000", 2024-03-20T09:02:38.747 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8261e-05 2024-03-20T09:02:38.747 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.747 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.747 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:38.747 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:12.050685+0000", 2024-03-20T09:02:38.747 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.013692734999999999 2024-03-20T09:02:38.747 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.747 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.747 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:38.747 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:12.050737+0000", 2024-03-20T09:02:38.748 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2522999999999998e-05 2024-03-20T09:02:38.748 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.748 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:38.748 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.748 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.748 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.748 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:224316 2.d e190/121 2:b381e73c:::benchmark_data_smithi122_107654_object105531:head v 190'13860, mlcod=190'13858)", 2024-03-20T09:02:38.748 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:02:18.004140+0000", 2024-03-20T09:02:38.748 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 20.667876196000002, 2024-03-20T09:02:38.748 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.022379915, 2024-03-20T09:02:38.748 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:38.748 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:38.748 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:38.748 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.749 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:38.749 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:18.004140+0000", 2024-03-20T09:02:38.749 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.749 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.749 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.749 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:38.749 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:18.004140+0000", 2024-03-20T09:02:38.749 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.749 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.749 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.749 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:38.749 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:18.004141+0000", 2024-03-20T09:02:38.749 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4479999999999999e-06 2024-03-20T09:02:38.749 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.750 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.750 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:38.750 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:18.004153+0000", 2024-03-20T09:02:38.750 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1668000000000001e-05 2024-03-20T09:02:38.750 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.750 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.750 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:38.750 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:18.004174+0000", 2024-03-20T09:02:38.750 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1369000000000002e-05 2024-03-20T09:02:38.750 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.750 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.750 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:38.750 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:18.004190+0000", 2024-03-20T09:02:38.750 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5841e-05 2024-03-20T09:02:38.751 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.751 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.751 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:38.751 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:18.004262+0000", 2024-03-20T09:02:38.751 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.2352000000000001e-05 2024-03-20T09:02:38.751 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.751 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.751 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:38.751 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:18.004292+0000", 2024-03-20T09:02:38.751 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9352e-05 2024-03-20T09:02:38.751 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.751 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.751 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:38.751 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:18.026484+0000", 2024-03-20T09:02:38.751 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.022191684999999999 2024-03-20T09:02:38.752 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.752 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.752 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:38.752 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:18.026520+0000", 2024-03-20T09:02:38.752 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6199999999999999e-05 2024-03-20T09:02:38.752 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.752 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:38.752 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.752 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.752 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.752 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:224328 2.5 e190/121 2:a329e22e:::benchmark_data_smithi122_107654_object105543:head v 190'13979, mlcod=190'13975)", 2024-03-20T09:02:38.752 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:02:18.017222+0000", 2024-03-20T09:02:38.752 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 20.654793926, 2024-03-20T09:02:38.752 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.017322126, 2024-03-20T09:02:38.752 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:38.753 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:38.753 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:38.753 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.753 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:38.753 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:18.017222+0000", 2024-03-20T09:02:38.753 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.753 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.753 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.753 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:38.753 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:18.017222+0000", 2024-03-20T09:02:38.753 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.753 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.753 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.753 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:38.754 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:18.017223+0000", 2024-03-20T09:02:38.754 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3099999999999997e-07 2024-03-20T09:02:38.754 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.754 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.754 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:38.754 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:18.017228+0000", 2024-03-20T09:02:38.754 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9529999999999998e-06 2024-03-20T09:02:38.754 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.754 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.754 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:38.754 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:18.017235+0000", 2024-03-20T09:02:38.754 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.5109999999999997e-06 2024-03-20T09:02:38.754 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.755 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.755 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:38.755 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:18.017240+0000", 2024-03-20T09:02:38.755 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2190000000000003e-06 2024-03-20T09:02:38.755 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.755 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.755 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:38.755 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:18.022254+0000", 2024-03-20T09:02:38.755 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0050139479999999998 2024-03-20T09:02:38.755 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.755 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.755 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:38.755 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:18.022281+0000", 2024-03-20T09:02:38.755 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6995000000000002e-05 2024-03-20T09:02:38.756 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.756 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.756 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:38.756 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:18.034500+0000", 2024-03-20T09:02:38.756 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.01221881 2024-03-20T09:02:38.756 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.756 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.756 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:38.756 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:18.034544+0000", 2024-03-20T09:02:38.756 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.3958999999999997e-05 2024-03-20T09:02:38.756 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.756 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:38.756 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.756 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.757 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.757 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24550.0:225810 2.f e190/121 2:fe88b30f:::benchmark_data_smithi122_107654_object107025:head v 190'13877, mlcod=190'13874)", 2024-03-20T09:02:38.757 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:02:19.704523+0000", 2024-03-20T09:02:38.757 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 18.967493113, 2024-03-20T09:02:38.757 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.021305477999999999, 2024-03-20T09:02:38.757 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:38.757 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:38.757 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:38.757 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.757 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:38.757 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:19.704523+0000", 2024-03-20T09:02:38.757 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.757 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.757 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.758 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:38.758 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:19.704523+0000", 2024-03-20T09:02:38.758 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:38.758 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.758 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.758 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:38.758 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:19.704523+0000", 2024-03-20T09:02:38.758 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7000000000000005e-07 2024-03-20T09:02:38.758 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.758 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.758 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:38.758 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:19.704528+0000", 2024-03-20T09:02:38.758 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.7369999999999997e-06 2024-03-20T09:02:38.758 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.759 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.759 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:38.759 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:19.704536+0000", 2024-03-20T09:02:38.759 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.9510000000000006e-06 2024-03-20T09:02:38.759 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.759 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.759 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:38.759 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:19.704542+0000", 2024-03-20T09:02:38.759 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5369999999999997e-06 2024-03-20T09:02:38.759 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.759 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.759 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:38.759 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:19.708470+0000", 2024-03-20T09:02:38.759 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.003928384 2024-03-20T09:02:38.759 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.760 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.760 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:38.760 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:19.708493+0000", 2024-03-20T09:02:38.760 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2756e-05 2024-03-20T09:02:38.760 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.760 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.760 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:38.760 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:19.725789+0000", 2024-03-20T09:02:38.760 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.017296087000000002 2024-03-20T09:02:38.760 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:38.760 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:38.760 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:38.760 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:02:19.725828+0000", 2024-03-20T09:02:38.760 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.9456000000000003e-05 2024-03-20T09:02:38.760 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.761 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:38.761 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.761 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:38.761 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:38.761 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:02:38.984 INFO:teuthology.orchestra.run.smithi026.stdout:moved 4 / 81 (4.93827%) 2024-03-20T09:02:38.985 INFO:teuthology.orchestra.run.smithi026.stdout:avg 6.75 2024-03-20T09:02:38.985 INFO:teuthology.orchestra.run.smithi026.stdout:stddev 1.73805 -> 1.53433 (expected baseline 2.48747) 2024-03-20T09:02:38.985 INFO:teuthology.orchestra.run.smithi026.stdout:min osd.5 with 3 -> 4 pgs (0.444444 -> 0.592593 * mean) 2024-03-20T09:02:38.985 INFO:teuthology.orchestra.run.smithi026.stdout:max osd.9 with 10 -> 8 pgs (1.48148 -> 1.18519 * mean) 2024-03-20T09:02:38.985 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:02:38.985 INFO:teuthology.orchestra.run.smithi026.stdout:oload 110 2024-03-20T09:02:38.985 INFO:teuthology.orchestra.run.smithi026.stdout:max_change 3 2024-03-20T09:02:38.985 INFO:teuthology.orchestra.run.smithi026.stdout:max_change_osds 4 2024-03-20T09:02:38.985 INFO:teuthology.orchestra.run.smithi026.stdout:average_utilization 0.0010 2024-03-20T09:02:38.985 INFO:teuthology.orchestra.run.smithi026.stdout:overload_utilization 0.0010 2024-03-20T09:02:38.985 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3 weight 1.0000 -> 0.6832 2024-03-20T09:02:38.985 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9 weight 1.0000 -> 0.7781 2024-03-20T09:02:38.986 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0 weight 1.0000 -> 0.8244 2024-03-20T09:02:38.986 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1 weight 1.0000 -> 0.8575 2024-03-20T09:02:38.991 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:38 smithi115 ceph-mon[72476]: from='client.15009 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:02:38.991 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:38 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'54030','1':'56196','3':'44775','9':'50993'}"}]: dispatch 2024-03-20T09:02:38.991 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:38 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'54030','1':'56196','3':'44775','9':'50993'}"}]: dispatch 2024-03-20T09:02:39.176 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:02:39.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:39.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:02:39.179 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:38 smithi026 ceph-mon[68309]: from='client.15009 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:02:39.179 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:38 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'54030','1':'56196','3':'44775','9':'50993'}"}]: dispatch 2024-03-20T09:02:39.179 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:38 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'54030','1':'56196','3':'44775','9':'50993'}"}]: dispatch 2024-03-20T09:02:39.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:38 smithi122 ceph-mon[73495]: from='client.15009 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:02:39.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:38 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'54030','1':'56196','3':'44775','9':'50993'}"}]: dispatch 2024-03-20T09:02:39.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:38 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'54030','1':'56196','3':'44775','9':'50993'}"}]: dispatch 2024-03-20T09:02:39.280 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:02:39.309 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:39.339 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-20T09:02:39.381 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:02:39.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:39.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:02:39.486 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:02:39.516 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:39.599 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:02:39.613 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:02:39.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:39.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:02:39.717 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:02:39.748 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:39.769 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:02:39.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:39.772 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:02:39.873 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:02:39.902 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:39.921 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:02:39.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:39.924 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:02:40.025 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:02:40.053 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:40.080 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:02:40.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:40.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:02:40.115 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:39 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'54030','1':'56196','3':'44775','9':'50993'}"}]': finished 2024-03-20T09:02:40.115 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:39 smithi115 ceph-mon[72476]: osdmap e197: 12 total, 12 up, 12 in 2024-03-20T09:02:40.115 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:39 smithi115 ceph-mon[72476]: pgmap v579: 27 pgs: 27 active+clean; 577 KiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 299 op/s 2024-03-20T09:02:40.116 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:39 smithi115 ceph-mon[72476]: osdmap e198: 12 total, 12 up, 12 in 2024-03-20T09:02:40.184 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:02:40.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:39 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'54030','1':'56196','3':'44775','9':'50993'}"}]': finished 2024-03-20T09:02:40.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:39 smithi122 ceph-mon[73495]: osdmap e197: 12 total, 12 up, 12 in 2024-03-20T09:02:40.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:39 smithi122 ceph-mon[73495]: pgmap v579: 27 pgs: 27 active+clean; 577 KiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 299 op/s 2024-03-20T09:02:40.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:39 smithi122 ceph-mon[73495]: osdmap e198: 12 total, 12 up, 12 in 2024-03-20T09:02:40.211 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:40.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:39 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'54030','1':'56196','3':'44775','9':'50993'}"}]': finished 2024-03-20T09:02:40.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:39 smithi026 ceph-mon[68309]: osdmap e197: 12 total, 12 up, 12 in 2024-03-20T09:02:40.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:39 smithi026 ceph-mon[68309]: pgmap v579: 27 pgs: 27 active+clean; 577 KiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 299 op/s 2024-03-20T09:02:40.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:39 smithi026 ceph-mon[68309]: osdmap e198: 12 total, 12 up, 12 in 2024-03-20T09:02:40.335 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:02:40.366 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-03-20T09:02:40.367 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight-by-utilization 1000 3.0 2024-03-20T09:02:40.895 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'unique_pool_0' removed 2024-03-20T09:02:40.902 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:40 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3022127197' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-20T09:02:40.902 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:40 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4048274004' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:02:40.902 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:40 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3022127197' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-20T09:02:40.902 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:40 smithi026 ceph-mon[68309]: osdmap e199: 12 total, 12 up, 12 in 2024-03-20T09:02:40.940 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:02:40.941 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:02:40.941 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:02:40.941 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:02:40.954 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:02:40.959 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:02:40.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:40.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:02:40.977 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:02:40.997 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:02:41.018 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:02:41.039 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:02:41.060 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:02:41.064 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:02:41.080 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:02:41.094 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:41.102 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:02:41.103 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:02:41.103 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:02:41.103 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:02:41.103 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:02:41.103 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:02:41.103 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:02:41.104 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:02:41.104 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:02:41.104 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:02:41.104 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:02:41.104 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:02:41.104 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:02:41.147 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:02:41.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:41.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:02:41.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:40 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3022127197' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-20T09:02:41.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:40 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4048274004' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:02:41.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:40 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3022127197' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-20T09:02:41.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:40 smithi115 ceph-mon[72476]: osdmap e199: 12 total, 12 up, 12 in 2024-03-20T09:02:41.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:40 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3022127197' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-20T09:02:41.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:40 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4048274004' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:02:41.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:40 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3022127197' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-20T09:02:41.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:40 smithi122 ceph-mon[73495]: osdmap e199: 12 total, 12 up, 12 in 2024-03-20T09:02:41.250 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:02:41.278 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:41.507 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:02:41.507 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:02:41.527 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:02:41.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:41.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:02:41.631 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:02:41.657 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:41.675 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:02:41.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:41.678 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:02:41.779 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:02:41.808 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:41.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:41 smithi122 ceph-mon[73495]: pgmap v582: 11 pgs: 1 active+undersized+remapped, 1 remapped+peering, 9 active+clean; 577 KiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:02:41.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:41 smithi122 ceph-mon[73495]: osdmap e200: 12 total, 12 up, 12 in 2024-03-20T09:02:41.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:41 smithi026 ceph-mon[68309]: pgmap v582: 11 pgs: 1 active+undersized+remapped, 1 remapped+peering, 9 active+clean; 577 KiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:02:41.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:41 smithi026 ceph-mon[68309]: osdmap e200: 12 total, 12 up, 12 in 2024-03-20T09:02:42.056 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:02:42.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:42.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:02:42.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:41 smithi115 ceph-mon[72476]: pgmap v582: 11 pgs: 1 active+undersized+remapped, 1 remapped+peering, 9 active+clean; 577 KiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:02:42.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:41 smithi115 ceph-mon[72476]: osdmap e200: 12 total, 12 up, 12 in 2024-03-20T09:02:42.160 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:02:42.188 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:42.241 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:02:42.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:42.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:02:42.253 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-20T09:02:42.349 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:02:42.377 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:42.395 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:02:42.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:42.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:02:42.499 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:02:42.519 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:02:42.527 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:42.545 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:02:42.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:42.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:02:42.649 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:02:42.655 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-20T09:02:42.656 INFO:tasks.radosbench:Beginning radosbench... 2024-03-20T09:02:42.656 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-03-20T09:02:42.656 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pool create unique_pool_1 16 2024-03-20T09:02:42.681 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:42.702 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:02:42.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:42.706 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:02:42.807 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:02:42.835 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:42.854 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:02:42.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:42.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:02:42.958 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:02:42.990 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:43.011 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:02:43.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:43.014 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:02:43.089 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:02:43.116 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:02:43.142 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:43.266 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:02:43.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:43.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:02:43.370 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:02:43.403 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:43.423 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:02:43.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:43.427 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:02:43.528 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:02:43.558 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:43.578 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:02:43.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:43.581 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:02:43.682 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:02:43.712 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:43.786 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:02:43.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:43.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:02:43.890 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:02:43.918 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:43.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:43 smithi026 ceph-mon[68309]: osdmap e201: 12 total, 12 up, 12 in 2024-03-20T09:02:43.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:43 smithi026 ceph-mon[68309]: pgmap v585: 11 pgs: 1 active+undersized+remapped, 1 remapped+peering, 9 active+clean; 577 KiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:02:44.084 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:43 smithi115 ceph-mon[72476]: osdmap e201: 12 total, 12 up, 12 in 2024-03-20T09:02:44.084 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:43 smithi115 ceph-mon[72476]: pgmap v585: 11 pgs: 1 active+undersized+remapped, 1 remapped+peering, 9 active+clean; 577 KiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:02:44.095 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:02:44.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:44.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:02:44.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:43 smithi122 ceph-mon[73495]: osdmap e201: 12 total, 12 up, 12 in 2024-03-20T09:02:44.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:43 smithi122 ceph-mon[73495]: pgmap v585: 11 pgs: 1 active+undersized+remapped, 1 remapped+peering, 9 active+clean; 577 KiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:02:44.200 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:02:44.232 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:44.254 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:02:44.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:44.257 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:02:44.345 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:02:44.345 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:02:44.358 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:02:44.388 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:44.415 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:02:44.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:44.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:02:44.524 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:02:44.554 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:44.626 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:02:44.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:44.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:02:44.730 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:02:44.759 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:44.882 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:02:44.888 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:02:44.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:44 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1097357173' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:02:44.906 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:02:44.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:44.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:02:45.022 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:02:45.049 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:45.084 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:02:45.102 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:02:45.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:45.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:02:45.110 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:44 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1097357173' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:02:45.126 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-20T09:02:45.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:44 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1097357173' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:02:45.206 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:02:45.237 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:45.260 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:02:45.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:45.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:02:45.364 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:02:45.392 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:45.437 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:02:45.516 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:02:45.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:45.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:02:45.556 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:02:45.622 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:02:45.652 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:45.798 INFO:teuthology.orchestra.run.smithi026.stdout:moved 0 / 33 (0%) 2024-03-20T09:02:45.798 INFO:teuthology.orchestra.run.smithi026.stdout:avg 2.75 2024-03-20T09:02:45.798 INFO:teuthology.orchestra.run.smithi026.stdout:stddev 1.23322 -> 1.23322 (expected baseline 1.58771) 2024-03-20T09:02:45.798 INFO:teuthology.orchestra.run.smithi026.stdout:min osd.1 with 1 -> 1 pgs (0.363636 -> 0.363636 * mean) 2024-03-20T09:02:45.798 INFO:teuthology.orchestra.run.smithi026.stdout:max osd.6 with 5 -> 5 pgs (1.81818 -> 1.81818 * mean) 2024-03-20T09:02:45.799 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:02:45.799 INFO:teuthology.orchestra.run.smithi026.stdout:oload 1000 2024-03-20T09:02:45.799 INFO:teuthology.orchestra.run.smithi026.stdout:max_change 3 2024-03-20T09:02:45.799 INFO:teuthology.orchestra.run.smithi026.stdout:max_change_osds 4 2024-03-20T09:02:45.799 INFO:teuthology.orchestra.run.smithi026.stdout:average_utilization 0.0010 2024-03-20T09:02:45.799 INFO:teuthology.orchestra.run.smithi026.stdout:overload_utilization 0.0096 2024-03-20T09:02:45.800 INFO:teuthology.orchestra.run.smithi026.stderr:no change 2024-03-20T09:02:45.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:45 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:02:45.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:45 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1097357173' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:02:45.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:45 smithi026 ceph-mon[68309]: osdmap e202: 12 total, 12 up, 12 in 2024-03-20T09:02:45.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:45 smithi026 ceph-mon[68309]: pgmap v587: 11 pgs: 1 active+undersized+remapped, 10 active+clean; 577 KiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:02:45.982 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:45 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:02:45.983 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:45 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1097357173' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:02:45.983 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:45 smithi115 ceph-mon[72476]: osdmap e202: 12 total, 12 up, 12 in 2024-03-20T09:02:45.983 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:45 smithi115 ceph-mon[72476]: pgmap v587: 11 pgs: 1 active+undersized+remapped, 10 active+clean; 577 KiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:02:46.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:45 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:02:46.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:45 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1097357173' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:02:46.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:45 smithi122 ceph-mon[73495]: osdmap e202: 12 total, 12 up, 12 in 2024-03-20T09:02:46.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:45 smithi122 ceph-mon[73495]: pgmap v587: 11 pgs: 1 active+undersized+remapped, 10 active+clean; 577 KiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:02:46.900 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:02:46.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:46.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:02:46.989 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:46 smithi115 ceph-mon[72476]: from='client.24926 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:02:46.989 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:46 smithi115 ceph-mon[72476]: osdmap e203: 12 total, 12 up, 12 in 2024-03-20T09:02:47.005 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:02:47.037 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:47.058 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:02:47.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:47.061 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:02:47.162 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:02:47.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:46 smithi122 ceph-mon[73495]: from='client.24926 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:02:47.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:46 smithi122 ceph-mon[73495]: osdmap e203: 12 total, 12 up, 12 in 2024-03-20T09:02:47.192 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:47.212 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:02:47.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:47.215 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:02:47.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:46 smithi026 ceph-mon[68309]: from='client.24926 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:02:47.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:46 smithi026 ceph-mon[68309]: osdmap e203: 12 total, 12 up, 12 in 2024-03-20T09:02:47.283 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:02:47.284 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:02:47.316 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:02:47.348 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:47.380 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-03-20T09:02:47.380 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight-by-utilization 1000 0.05 2024-03-20T09:02:47.546 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:02:47.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:47.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:02:47.650 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:02:47.680 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:47.932 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:02:47.935 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:47 smithi026 ceph-mon[68309]: pgmap v589: 11 pgs: 11 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:02:47.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:47.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:02:47.974 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-20T09:02:48.037 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:02:48.063 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:48.082 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:02:48.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:48.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:02:48.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:47 smithi115 ceph-mon[72476]: pgmap v589: 11 pgs: 11 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:02:48.186 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:02:48.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:47 smithi122 ceph-mon[73495]: pgmap v589: 11 pgs: 11 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:02:48.216 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:48.239 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:02:48.633 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:02:48.665 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:02:48.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:48.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:02:48.769 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:02:48.798 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:48.873 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:02:48.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:48.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:02:48.887 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'unique_pool_1' created 2024-03-20T09:02:48.981 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:02:49.008 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:48 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2236783772' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-20T09:02:49.010 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:49.029 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:02:49.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:49.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:02:49.134 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:02:49.161 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:49.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:48 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2236783772' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-20T09:02:49.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:48 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2236783772' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-20T09:02:49.509 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:02:49.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:49.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:02:49.614 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:02:49.646 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:49.667 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:02:49.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:49.670 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:02:49.771 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:02:49.809 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:49.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:49 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:02:49.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:49 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:02:49.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:49 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2236783772' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-20T09:02:49.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:49 smithi122 ceph-mon[73495]: osdmap e204: 12 total, 12 up, 12 in 2024-03-20T09:02:49.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:49 smithi122 ceph-mon[73495]: pgmap v591: 27 pgs: 4 creating+peering, 12 unknown, 11 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:02:50.016 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:02:50.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:50.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:02:50.047 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:49 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:02:50.047 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:49 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:02:50.048 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:49 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2236783772' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-20T09:02:50.048 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:49 smithi026 ceph-mon[68309]: osdmap e204: 12 total, 12 up, 12 in 2024-03-20T09:02:50.048 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:49 smithi026 ceph-mon[68309]: pgmap v591: 27 pgs: 4 creating+peering, 12 unknown, 11 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:02:50.049 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:02:50.049 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:02:50.072 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:49 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:02:50.072 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:49 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:02:50.072 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:49 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2236783772' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-20T09:02:50.072 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:49 smithi115 ceph-mon[72476]: osdmap e204: 12 total, 12 up, 12 in 2024-03-20T09:02:50.072 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:49 smithi115 ceph-mon[72476]: pgmap v591: 27 pgs: 4 creating+peering, 12 unknown, 11 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:02:50.120 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:02:50.150 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:50.603 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:02:50.787 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-03-20T09:02:50.791 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-20T09:02:51.058 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:02:51.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:50 smithi115 ceph-mon[72476]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-20T09:02:51.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:50 smithi115 ceph-mon[72476]: osdmap e205: 12 total, 12 up, 12 in 2024-03-20T09:02:51.190 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:50 smithi026 ceph-mon[68309]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-20T09:02:51.190 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:50 smithi026 ceph-mon[68309]: osdmap e205: 12 total, 12 up, 12 in 2024-03-20T09:02:51.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:50 smithi122 ceph-mon[73495]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-20T09:02:51.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:50 smithi122 ceph-mon[73495]: osdmap e205: 12 total, 12 up, 12 in 2024-03-20T09:02:51.201 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:02:51.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:51.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:02:51.306 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:02:51.334 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:51.433 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:02:51.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:51.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:02:51.537 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:02:51.565 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:51.949 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:02:51.949 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:02:51.950 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:02:51.970 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:02:51.994 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:02:52.014 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:02:52.034 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:02:52.054 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:02:52.075 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:02:52.096 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:02:52.103 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:02:52.105 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:02:52.138 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:02:52.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:51 smithi115 ceph-mon[72476]: osdmap e206: 12 total, 12 up, 12 in 2024-03-20T09:02:52.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:51 smithi115 ceph-mon[72476]: pgmap v594: 27 pgs: 6 creating+peering, 6 unknown, 15 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:02:52.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:51 smithi115 ceph-mon[72476]: osdmap e207: 12 total, 12 up, 12 in 2024-03-20T09:02:52.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:51 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1118216736' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:02:52.177 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:02:52.178 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:02:52.178 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:02:52.178 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:02:52.178 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:02:52.178 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:02:52.178 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:02:52.178 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:02:52.179 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:02:52.179 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:02:52.179 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:02:52.179 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:02:52.179 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:02:52.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:51 smithi122 ceph-mon[73495]: osdmap e206: 12 total, 12 up, 12 in 2024-03-20T09:02:52.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:51 smithi122 ceph-mon[73495]: pgmap v594: 27 pgs: 6 creating+peering, 6 unknown, 15 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:02:52.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:51 smithi122 ceph-mon[73495]: osdmap e207: 12 total, 12 up, 12 in 2024-03-20T09:02:52.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:51 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1118216736' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:02:52.201 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:51 smithi026 ceph-mon[68309]: osdmap e206: 12 total, 12 up, 12 in 2024-03-20T09:02:52.201 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:51 smithi026 ceph-mon[68309]: pgmap v594: 27 pgs: 6 creating+peering, 6 unknown, 15 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:02:52.201 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:51 smithi026 ceph-mon[68309]: osdmap e207: 12 total, 12 up, 12 in 2024-03-20T09:02:52.201 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:51 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1118216736' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:02:52.243 INFO:teuthology.orchestra.run.smithi026.stdout:moved 0 / 81 (0%) 2024-03-20T09:02:52.243 INFO:teuthology.orchestra.run.smithi026.stdout:avg 6.75 2024-03-20T09:02:52.243 INFO:teuthology.orchestra.run.smithi026.stdout:stddev 2.16506 -> 2.16506 (expected baseline 2.48747) 2024-03-20T09:02:52.243 INFO:teuthology.orchestra.run.smithi026.stdout:min osd.1 with 4 -> 4 pgs (0.592593 -> 0.592593 * mean) 2024-03-20T09:02:52.243 INFO:teuthology.orchestra.run.smithi026.stdout:max osd.4 with 11 -> 11 pgs (1.62963 -> 1.62963 * mean) 2024-03-20T09:02:52.243 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:02:52.244 INFO:teuthology.orchestra.run.smithi026.stdout:oload 1000 2024-03-20T09:02:52.244 INFO:teuthology.orchestra.run.smithi026.stdout:max_change 0.05 2024-03-20T09:02:52.244 INFO:teuthology.orchestra.run.smithi026.stdout:max_change_osds 4 2024-03-20T09:02:52.244 INFO:teuthology.orchestra.run.smithi026.stdout:average_utilization 0.0010 2024-03-20T09:02:52.244 INFO:teuthology.orchestra.run.smithi026.stdout:overload_utilization 0.0099 2024-03-20T09:02:52.246 INFO:teuthology.orchestra.run.smithi026.stderr:no change 2024-03-20T09:02:52.414 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:02:52.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:52.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:02:52.519 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:02:52.551 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:52.572 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:02:52.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:52.576 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:02:52.677 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:02:52.707 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:52.960 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:02:52.960 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:02:52.960 INFO:teuthology.orchestra.run.smithi115.stdout: "complaint_time": 30, 2024-03-20T09:02:52.961 INFO:teuthology.orchestra.run.smithi115.stdout: "num_blocked_ops": 0 2024-03-20T09:02:52.961 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:02:52.987 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:52 smithi115 ceph-mon[72476]: from='client.24944 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:02:52.987 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:52 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:02:53.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:52 smithi122 ceph-mon[73495]: from='client.24944 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:02:53.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:52 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:02:53.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:52 smithi026 ceph-mon[68309]: from='client.24944 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:02:53.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:52 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:02:53.459 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:02:53.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:53.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:02:53.563 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:02:53.592 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:53.610 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:02:53.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:53.613 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:02:53.686 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-20T09:02:53.714 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:02:53.757 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:53.777 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:02:53.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:53.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:02:53.881 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:02:53.911 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:53.931 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:02:53.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:53.934 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:02:53.956 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:02:54.035 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:02:54.064 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:54.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:53 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:02:54.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:53 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:02:54.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:53 smithi122 ceph-mon[73495]: osdmap e208: 12 total, 12 up, 12 in 2024-03-20T09:02:54.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:53 smithi122 ceph-mon[73495]: pgmap v597: 27 pgs: 6 creating+peering, 21 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:02:54.214 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:53 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:02:54.214 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:53 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:02:54.214 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:53 smithi115 ceph-mon[72476]: osdmap e208: 12 total, 12 up, 12 in 2024-03-20T09:02:54.214 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:53 smithi115 ceph-mon[72476]: pgmap v597: 27 pgs: 6 creating+peering, 21 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:02:54.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:53 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:02:54.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:53 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:02:54.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:53 smithi026 ceph-mon[68309]: osdmap e208: 12 total, 12 up, 12 in 2024-03-20T09:02:54.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:53 smithi026 ceph-mon[68309]: pgmap v597: 27 pgs: 6 creating+peering, 21 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:02:54.262 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:02:54.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:54.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:02:54.288 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-03-20T09:02:54.288 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd test-reweight-by-utilization 1000 3.0 2024-03-20T09:02:54.366 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:02:54.399 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:54.420 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:02:54.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:54.423 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:02:54.524 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:02:54.554 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:54.575 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:02:54.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:54.577 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:02:54.677 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:02:54.706 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:54.725 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:02:54.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:54.728 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:02:54.829 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:02:54.859 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:55.209 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:02:55.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:55.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:02:55.313 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:02:55.345 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:55.368 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:02:55.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:55.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:02:55.475 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:02:55.502 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:55.574 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:02:55.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:55.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:02:55.679 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:02:55.707 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:55.830 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:02:55.832 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:55 smithi115 ceph-mon[72476]: pgmap v598: 27 pgs: 27 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:02:55.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:55.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:02:55.846 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:02:55.846 INFO:teuthology.orchestra.run.smithi115.stdout: "size": 20, 2024-03-20T09:02:55.846 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 600, 2024-03-20T09:02:55.846 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [ 2024-03-20T09:02:55.846 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.846 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:41692 2.7 2:e8fd97ed:::benchmark_data_smithi122_107654_object41691:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-20T09:02:55.846 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:58:39.733162+0000", 2024-03-20T09:02:55.847 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 256.11017763699999, 2024-03-20T09:02:55.847 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.060791761999999999, 2024-03-20T09:02:55.847 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:55.847 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:55.847 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:02:55.847 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T09:02:55.847 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:02:55.847 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 41692 2024-03-20T09:02:55.847 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.847 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:55.847 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.847 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:55.847 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.733162+0000", 2024-03-20T09:02:55.847 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.848 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.848 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.848 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:55.848 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.733162+0000", 2024-03-20T09:02:55.848 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.848 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.848 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.848 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:55.848 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.733163+0000", 2024-03-20T09:02:55.848 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.435e-06 2024-03-20T09:02:55.848 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.848 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.848 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:55.848 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.733217+0000", 2024-03-20T09:02:55.849 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3427999999999997e-05 2024-03-20T09:02:55.849 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.849 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.849 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:55.849 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.733231+0000", 2024-03-20T09:02:55.849 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.45e-05 2024-03-20T09:02:55.849 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.849 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.849 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:55.849 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.733240+0000", 2024-03-20T09:02:55.849 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5129999999999996e-06 2024-03-20T09:02:55.849 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.849 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.849 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:55.850 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.733302+0000", 2024-03-20T09:02:55.850 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2216000000000002e-05 2024-03-20T09:02:55.850 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.850 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.850 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:55.850 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.733421+0000", 2024-03-20T09:02:55.850 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000119079 2024-03-20T09:02:55.850 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.850 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.850 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T09:02:55.850 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.733562+0000", 2024-03-20T09:02:55.850 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000141147 2024-03-20T09:02:55.850 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.850 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.851 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:02:55.851 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.734946+0000", 2024-03-20T09:02:55.851 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001384019 2024-03-20T09:02:55.851 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.851 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.851 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.851 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.735396+0000", 2024-03-20T09:02:55.851 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00044978999999999999 2024-03-20T09:02:55.851 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.851 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.851 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.851 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.793856+0000", 2024-03-20T09:02:55.851 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.058459839 2024-03-20T09:02:55.851 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.852 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.852 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:55.852 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.793902+0000", 2024-03-20T09:02:55.852 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6652e-05 2024-03-20T09:02:55.852 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.852 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.852 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:55.852 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.793953+0000", 2024-03-20T09:02:55.852 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1143999999999997e-05 2024-03-20T09:02:55.852 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.852 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:55.852 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.852 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.852 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.853 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:41702 2.7 2:e9cc37ef:::benchmark_data_smithi122_107654_object41701:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-20T09:02:55.853 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:58:39.736630+0000", 2024-03-20T09:02:55.853 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 256.10670941699999, 2024-03-20T09:02:55.853 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.058660687000000003, 2024-03-20T09:02:55.853 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:55.853 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:55.853 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:02:55.853 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T09:02:55.853 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:02:55.853 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 41702 2024-03-20T09:02:55.853 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.853 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:55.853 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.854 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:55.854 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.736630+0000", 2024-03-20T09:02:55.854 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.854 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.854 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.854 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:55.854 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.736630+0000", 2024-03-20T09:02:55.854 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.854 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.854 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.854 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:55.854 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.736631+0000", 2024-03-20T09:02:55.854 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.251e-06 2024-03-20T09:02:55.855 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.855 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.855 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:55.855 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.736694+0000", 2024-03-20T09:02:55.855 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.3272e-05 2024-03-20T09:02:55.855 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.855 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.855 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:55.855 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.736705+0000", 2024-03-20T09:02:55.855 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0159999999999999e-05 2024-03-20T09:02:55.855 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.855 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.855 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:55.855 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.736710+0000", 2024-03-20T09:02:55.855 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7509999999999997e-06 2024-03-20T09:02:55.856 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.856 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.856 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:55.856 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.736758+0000", 2024-03-20T09:02:55.856 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.7788999999999999e-05 2024-03-20T09:02:55.856 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.856 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.856 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:55.856 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.736848+0000", 2024-03-20T09:02:55.856 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.9874e-05 2024-03-20T09:02:55.856 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.856 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.856 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T09:02:55.856 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.736948+0000", 2024-03-20T09:02:55.857 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000100492 2024-03-20T09:02:55.857 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.857 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.857 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:02:55.857 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.738024+0000", 2024-03-20T09:02:55.857 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00107554 2024-03-20T09:02:55.857 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.857 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.857 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.857 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.738459+0000", 2024-03-20T09:02:55.857 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00043516799999999999 2024-03-20T09:02:55.857 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.857 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.857 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.857 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.795216+0000", 2024-03-20T09:02:55.858 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.056756661999999999 2024-03-20T09:02:55.858 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.858 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.858 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:55.858 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.795250+0000", 2024-03-20T09:02:55.858 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.4208999999999997e-05 2024-03-20T09:02:55.858 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.858 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.858 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:55.858 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:39.795291+0000", 2024-03-20T09:02:55.858 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.0519000000000003e-05 2024-03-20T09:02:55.858 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.858 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:55.858 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.859 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.859 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.859 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:53325 2.7 2:e50bf810:::benchmark_data_smithi122_107654_object53324:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-20T09:02:55.859 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:58:45.740926+0000", 2024-03-20T09:02:55.859 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 250.10241325499999, 2024-03-20T09:02:55.859 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.20845012600000001, 2024-03-20T09:02:55.859 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:55.859 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:55.859 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:02:55.859 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T09:02:55.859 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:02:55.859 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 53325 2024-03-20T09:02:55.859 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.859 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:55.860 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.860 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:55.860 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.740926+0000", 2024-03-20T09:02:55.860 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.860 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.860 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.860 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:55.860 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.740926+0000", 2024-03-20T09:02:55.860 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.860 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.860 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.860 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:55.860 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.740928+0000", 2024-03-20T09:02:55.860 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5519999999999999e-06 2024-03-20T09:02:55.860 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.861 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.861 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:55.861 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.740984+0000", 2024-03-20T09:02:55.861 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5915000000000003e-05 2024-03-20T09:02:55.861 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.861 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.861 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:55.861 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.740998+0000", 2024-03-20T09:02:55.861 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4769e-05 2024-03-20T09:02:55.861 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.861 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.861 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:55.861 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.741007+0000", 2024-03-20T09:02:55.861 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.6149999999999997e-06 2024-03-20T09:02:55.862 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.862 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.862 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:55.862 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.741078+0000", 2024-03-20T09:02:55.862 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.1155999999999994e-05 2024-03-20T09:02:55.862 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.862 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.862 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:55.862 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.741201+0000", 2024-03-20T09:02:55.862 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012243700000000001 2024-03-20T09:02:55.862 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.862 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.862 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T09:02:55.862 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.741344+0000", 2024-03-20T09:02:55.862 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00014315000000000001 2024-03-20T09:02:55.863 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.863 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.863 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:02:55.863 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.742780+0000", 2024-03-20T09:02:55.863 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0014363760000000001 2024-03-20T09:02:55.863 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.863 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.863 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.863 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.743656+0000", 2024-03-20T09:02:55.863 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000876307 2024-03-20T09:02:55.863 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.863 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.863 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.863 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.949309+0000", 2024-03-20T09:02:55.863 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.205652374 2024-03-20T09:02:55.864 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.864 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.864 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:55.864 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.949347+0000", 2024-03-20T09:02:55.864 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7993e-05 2024-03-20T09:02:55.864 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.864 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.864 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:55.864 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.949376+0000", 2024-03-20T09:02:55.864 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9482e-05 2024-03-20T09:02:55.864 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.864 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:55.864 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.864 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.865 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.865 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:53361 2.7 2:e45d35a1:::benchmark_data_smithi122_107654_object53360:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-20T09:02:55.865 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:58:45.759219+0000", 2024-03-20T09:02:55.865 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 250.08412036799999, 2024-03-20T09:02:55.865 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.19152682400000001, 2024-03-20T09:02:55.865 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:55.865 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:55.865 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:02:55.865 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T09:02:55.865 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:02:55.865 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 53361 2024-03-20T09:02:55.865 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.865 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:55.865 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.866 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:55.866 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.759219+0000", 2024-03-20T09:02:55.866 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.866 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.866 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.866 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:55.866 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.759219+0000", 2024-03-20T09:02:55.866 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.866 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.866 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.866 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:55.866 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.759221+0000", 2024-03-20T09:02:55.866 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7880000000000001e-06 2024-03-20T09:02:55.866 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.866 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.867 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:55.867 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.759272+0000", 2024-03-20T09:02:55.867 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0980000000000003e-05 2024-03-20T09:02:55.867 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.867 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.867 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:55.867 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.759288+0000", 2024-03-20T09:02:55.867 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5879000000000002e-05 2024-03-20T09:02:55.867 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.867 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.867 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:55.867 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.759295+0000", 2024-03-20T09:02:55.867 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.0589999999999997e-06 2024-03-20T09:02:55.868 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.868 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.868 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:55.868 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.759345+0000", 2024-03-20T09:02:55.868 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0821e-05 2024-03-20T09:02:55.868 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.868 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.868 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:55.868 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.759444+0000", 2024-03-20T09:02:55.868 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.8855000000000001e-05 2024-03-20T09:02:55.868 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.868 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.868 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T09:02:55.868 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.759536+0000", 2024-03-20T09:02:55.868 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.1990000000000005e-05 2024-03-20T09:02:55.869 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.869 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.869 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:02:55.869 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.761886+0000", 2024-03-20T09:02:55.869 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0023491950000000001 2024-03-20T09:02:55.869 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.869 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.869 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.869 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.765278+0000", 2024-03-20T09:02:55.869 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0033926640000000001 2024-03-20T09:02:55.869 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.869 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.869 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.869 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.950640+0000", 2024-03-20T09:02:55.869 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.18536202800000001 2024-03-20T09:02:55.870 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.870 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.870 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:55.870 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.950698+0000", 2024-03-20T09:02:55.870 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7547000000000003e-05 2024-03-20T09:02:55.870 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.870 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.870 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:55.870 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.950746+0000", 2024-03-20T09:02:55.870 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8018e-05 2024-03-20T09:02:55.870 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.870 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:55.870 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.871 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.871 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.871 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:53363 2.7 2:e993eef9:::benchmark_data_smithi122_107654_object53362:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-20T09:02:55.871 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:58:45.760351+0000", 2024-03-20T09:02:55.871 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 250.082987852, 2024-03-20T09:02:55.871 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.190487987, 2024-03-20T09:02:55.871 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:55.871 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:55.871 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:02:55.871 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T09:02:55.871 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:02:55.871 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 53363 2024-03-20T09:02:55.871 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.871 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:55.871 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.872 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:55.872 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.760351+0000", 2024-03-20T09:02:55.872 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.872 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.872 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.872 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:55.872 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.760351+0000", 2024-03-20T09:02:55.872 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.872 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.872 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.872 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:55.872 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.760352+0000", 2024-03-20T09:02:55.872 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.1299999999999998e-07 2024-03-20T09:02:55.872 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.873 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.873 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:55.873 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.760428+0000", 2024-03-20T09:02:55.873 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.5286999999999999e-05 2024-03-20T09:02:55.873 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.873 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.873 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:55.873 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.760441+0000", 2024-03-20T09:02:55.873 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2860000000000001e-05 2024-03-20T09:02:55.873 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.873 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.873 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:55.873 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.760448+0000", 2024-03-20T09:02:55.873 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.8189999999999992e-06 2024-03-20T09:02:55.874 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.874 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.874 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:55.874 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.760781+0000", 2024-03-20T09:02:55.874 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00033301199999999999 2024-03-20T09:02:55.874 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.874 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.874 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:55.874 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.760901+0000", 2024-03-20T09:02:55.874 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000119924 2024-03-20T09:02:55.874 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.874 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.874 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T09:02:55.874 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.761081+0000", 2024-03-20T09:02:55.875 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00018010500000000001 2024-03-20T09:02:55.875 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.875 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.875 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:02:55.875 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.761987+0000", 2024-03-20T09:02:55.875 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00090566900000000001 2024-03-20T09:02:55.875 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.875 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.875 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.875 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.765325+0000", 2024-03-20T09:02:55.875 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0033378000000000001 2024-03-20T09:02:55.875 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.875 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.875 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.876 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.950778+0000", 2024-03-20T09:02:55.876 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.18545304200000001 2024-03-20T09:02:55.876 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.876 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.876 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:55.876 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.950813+0000", 2024-03-20T09:02:55.876 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.472e-05 2024-03-20T09:02:55.876 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.876 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.876 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:55.876 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.950839+0000", 2024-03-20T09:02:55.876 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6835999999999999e-05 2024-03-20T09:02:55.876 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.876 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:55.877 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.877 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.877 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.877 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:53419 2.7 2:e710df74:::benchmark_data_smithi122_107654_object53418:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-20T09:02:55.877 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:58:45.791591+0000", 2024-03-20T09:02:55.877 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 250.05174844300001, 2024-03-20T09:02:55.877 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.15939167900000001, 2024-03-20T09:02:55.877 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:55.877 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:55.877 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:02:55.877 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T09:02:55.877 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:02:55.877 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 53419 2024-03-20T09:02:55.878 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.878 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:55.878 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.878 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:55.878 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.791591+0000", 2024-03-20T09:02:55.878 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.878 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.878 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.878 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:55.878 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.791591+0000", 2024-03-20T09:02:55.878 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.878 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.878 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.878 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:55.879 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.791592+0000", 2024-03-20T09:02:55.879 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2389999999999999e-06 2024-03-20T09:02:55.879 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.879 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.879 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:55.879 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.791637+0000", 2024-03-20T09:02:55.879 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4650000000000001e-05 2024-03-20T09:02:55.879 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.879 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.879 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:55.879 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.791651+0000", 2024-03-20T09:02:55.879 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4423999999999999e-05 2024-03-20T09:02:55.879 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.879 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.880 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:55.880 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.791660+0000", 2024-03-20T09:02:55.880 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.8259999999999999e-06 2024-03-20T09:02:55.880 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.880 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.880 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:55.880 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.791718+0000", 2024-03-20T09:02:55.880 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8168999999999998e-05 2024-03-20T09:02:55.880 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.880 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.880 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:55.880 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.791856+0000", 2024-03-20T09:02:55.880 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013817199999999999 2024-03-20T09:02:55.880 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.880 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.881 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T09:02:55.881 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.792032+0000", 2024-03-20T09:02:55.881 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00017524000000000001 2024-03-20T09:02:55.881 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.881 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.881 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:02:55.881 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.793458+0000", 2024-03-20T09:02:55.881 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0014260010000000001 2024-03-20T09:02:55.881 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.881 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.882 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.882 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.795766+0000", 2024-03-20T09:02:55.882 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0023082340000000002 2024-03-20T09:02:55.882 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.882 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.882 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.882 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.950879+0000", 2024-03-20T09:02:55.882 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.15511298200000001 2024-03-20T09:02:55.882 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.882 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.882 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:55.882 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.950933+0000", 2024-03-20T09:02:55.882 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4591e-05 2024-03-20T09:02:55.882 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.883 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.883 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:55.883 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.950983+0000", 2024-03-20T09:02:55.883 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9150999999999997e-05 2024-03-20T09:02:55.883 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.883 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:55.883 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.883 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.883 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.883 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:53451 2.7 2:ecf06e77:::benchmark_data_smithi122_107654_object53450:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-20T09:02:55.883 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:58:45.807989+0000", 2024-03-20T09:02:55.883 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 250.03535020999999, 2024-03-20T09:02:55.883 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.14417972800000001, 2024-03-20T09:02:55.883 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:55.883 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:55.884 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:02:55.884 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T09:02:55.884 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:02:55.884 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 53451 2024-03-20T09:02:55.884 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.884 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:55.884 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.884 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:55.884 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.807989+0000", 2024-03-20T09:02:55.884 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.884 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.884 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.884 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:55.884 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.807989+0000", 2024-03-20T09:02:55.885 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.885 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.885 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.885 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:55.885 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.807991+0000", 2024-03-20T09:02:55.885 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8369999999999999e-06 2024-03-20T09:02:55.885 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.885 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.885 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:55.885 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.808031+0000", 2024-03-20T09:02:55.885 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.9857999999999997e-05 2024-03-20T09:02:55.885 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.885 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.885 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:55.885 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.808050+0000", 2024-03-20T09:02:55.886 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9086e-05 2024-03-20T09:02:55.886 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.886 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.886 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:55.886 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.808060+0000", 2024-03-20T09:02:55.886 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.9399999999999997e-06 2024-03-20T09:02:55.886 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.886 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.886 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:55.886 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.808126+0000", 2024-03-20T09:02:55.886 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.5685999999999994e-05 2024-03-20T09:02:55.886 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.886 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.886 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:55.887 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.808275+0000", 2024-03-20T09:02:55.887 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00014987 2024-03-20T09:02:55.887 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.887 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.887 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T09:02:55.887 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.808415+0000", 2024-03-20T09:02:55.887 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013999699999999999 2024-03-20T09:02:55.887 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.887 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.887 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:02:55.887 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.810284+0000", 2024-03-20T09:02:55.887 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001868672 2024-03-20T09:02:55.887 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.887 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.888 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.888 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.811950+0000", 2024-03-20T09:02:55.888 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0016661930000000001 2024-03-20T09:02:55.888 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.888 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.888 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.888 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.952104+0000", 2024-03-20T09:02:55.888 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.140153905 2024-03-20T09:02:55.888 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.888 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.888 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:55.888 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.952143+0000", 2024-03-20T09:02:55.888 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.8816000000000002e-05 2024-03-20T09:02:55.888 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.888 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.889 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:55.889 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.952169+0000", 2024-03-20T09:02:55.889 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5868e-05 2024-03-20T09:02:55.889 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.889 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:55.889 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.889 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.889 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.889 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:53480 2.7 2:e99370e5:::benchmark_data_smithi122_107654_object53479:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-20T09:02:55.889 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:58:45.825869+0000", 2024-03-20T09:02:55.889 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 250.01747007200001, 2024-03-20T09:02:55.889 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.126442162, 2024-03-20T09:02:55.889 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:55.889 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:55.889 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:02:55.890 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T09:02:55.890 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:02:55.890 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 53480 2024-03-20T09:02:55.890 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.890 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:55.890 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.890 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:55.890 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.825869+0000", 2024-03-20T09:02:55.890 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.890 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.890 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.890 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:55.890 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.825869+0000", 2024-03-20T09:02:55.891 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.891 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.891 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.891 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:55.891 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.825872+0000", 2024-03-20T09:02:55.891 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3369999999999998e-06 2024-03-20T09:02:55.891 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.891 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.891 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:55.891 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.825916+0000", 2024-03-20T09:02:55.891 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4693000000000001e-05 2024-03-20T09:02:55.891 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.891 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.891 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:55.891 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.825938+0000", 2024-03-20T09:02:55.892 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1393999999999999e-05 2024-03-20T09:02:55.892 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.892 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.892 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:55.892 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.825948+0000", 2024-03-20T09:02:55.892 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0539e-05 2024-03-20T09:02:55.892 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.892 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.892 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:55.892 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.826015+0000", 2024-03-20T09:02:55.892 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.6328999999999995e-05 2024-03-20T09:02:55.892 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.892 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.892 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:55.893 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.826123+0000", 2024-03-20T09:02:55.893 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000108646 2024-03-20T09:02:55.893 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.893 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.893 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T09:02:55.893 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.826240+0000", 2024-03-20T09:02:55.893 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000116766 2024-03-20T09:02:55.893 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.893 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.893 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.893 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.831367+0000", 2024-03-20T09:02:55.893 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0051273869999999997 2024-03-20T09:02:55.893 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.893 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.894 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:02:55.894 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.832945+0000", 2024-03-20T09:02:55.894 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00157763 2024-03-20T09:02:55.894 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.894 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.894 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.894 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.952214+0000", 2024-03-20T09:02:55.894 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.11926896100000001 2024-03-20T09:02:55.894 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.894 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.894 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:55.894 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.952270+0000", 2024-03-20T09:02:55.894 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5899000000000001e-05 2024-03-20T09:02:55.894 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.894 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.895 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:55.895 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.952311+0000", 2024-03-20T09:02:55.895 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.1581000000000001e-05 2024-03-20T09:02:55.895 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.895 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:55.895 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.895 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.895 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.895 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:53481 2.7 2:eab72ea7:::benchmark_data_smithi122_107654_object53480:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-20T09:02:55.895 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:58:45.826029+0000", 2024-03-20T09:02:55.895 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 250.017310495, 2024-03-20T09:02:55.895 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.12836041500000001, 2024-03-20T09:02:55.895 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:55.896 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:55.896 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:02:55.896 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T09:02:55.896 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:02:55.896 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 53481 2024-03-20T09:02:55.896 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.896 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:55.896 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.896 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:55.896 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.826029+0000", 2024-03-20T09:02:55.896 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.896 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.896 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.896 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:55.896 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.826029+0000", 2024-03-20T09:02:55.897 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.897 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.897 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.897 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:55.897 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.826030+0000", 2024-03-20T09:02:55.897 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.0500000000000002e-07 2024-03-20T09:02:55.897 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.897 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.897 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:55.897 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.826102+0000", 2024-03-20T09:02:55.897 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.2596999999999996e-05 2024-03-20T09:02:55.897 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.897 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.897 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:55.898 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.826112+0000", 2024-03-20T09:02:55.898 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.9769999999999995e-06 2024-03-20T09:02:55.898 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.898 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.898 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:55.898 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.826120+0000", 2024-03-20T09:02:55.898 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3749999999999997e-06 2024-03-20T09:02:55.898 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.898 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.898 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:55.898 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.827847+0000", 2024-03-20T09:02:55.898 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001727606 2024-03-20T09:02:55.898 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.898 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.899 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:55.899 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.827972+0000", 2024-03-20T09:02:55.899 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012518699999999999 2024-03-20T09:02:55.899 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.899 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.899 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T09:02:55.899 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.828059+0000", 2024-03-20T09:02:55.899 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.6706999999999995e-05 2024-03-20T09:02:55.899 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.899 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.899 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.899 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.831411+0000", 2024-03-20T09:02:55.899 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0033521100000000002 2024-03-20T09:02:55.899 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.899 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.900 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:02:55.900 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.832951+0000", 2024-03-20T09:02:55.900 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001539522 2024-03-20T09:02:55.900 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.900 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.900 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.900 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.954317+0000", 2024-03-20T09:02:55.900 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.121365893 2024-03-20T09:02:55.900 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.900 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.900 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:55.900 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.954359+0000", 2024-03-20T09:02:55.900 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.2688999999999997e-05 2024-03-20T09:02:55.900 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.901 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.901 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:55.901 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.954389+0000", 2024-03-20T09:02:55.901 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9847000000000001e-05 2024-03-20T09:02:55.901 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.901 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:55.901 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.901 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.901 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.901 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:53483 2.7 2:e28a92ca:::benchmark_data_smithi122_107654_object53482:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-20T09:02:55.901 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:58:45.826502+0000", 2024-03-20T09:02:55.901 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 250.01683764699999, 2024-03-20T09:02:55.901 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.12796293, 2024-03-20T09:02:55.901 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:55.902 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:55.902 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:02:55.902 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T09:02:55.902 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:02:55.902 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 53483 2024-03-20T09:02:55.902 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.902 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:55.902 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.902 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:55.902 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.826502+0000", 2024-03-20T09:02:55.902 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.902 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.902 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.902 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:55.903 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.826502+0000", 2024-03-20T09:02:55.903 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.903 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.903 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.903 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:55.903 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.826502+0000", 2024-03-20T09:02:55.903 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.7899999999999998e-07 2024-03-20T09:02:55.903 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.903 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.903 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:55.903 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.826682+0000", 2024-03-20T09:02:55.903 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00017939599999999999 2024-03-20T09:02:55.903 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.903 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.903 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:55.904 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.826693+0000", 2024-03-20T09:02:55.904 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1296e-05 2024-03-20T09:02:55.904 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.904 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.904 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:55.904 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.826701+0000", 2024-03-20T09:02:55.904 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.8739999999999995e-06 2024-03-20T09:02:55.904 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.904 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.904 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:55.904 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.829703+0000", 2024-03-20T09:02:55.904 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.003002191 2024-03-20T09:02:55.904 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.904 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.904 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:55.904 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.829813+0000", 2024-03-20T09:02:55.905 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000109737 2024-03-20T09:02:55.905 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.905 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.905 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T09:02:55.905 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.829935+0000", 2024-03-20T09:02:55.905 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000121883 2024-03-20T09:02:55.905 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.905 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.905 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.905 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.831458+0000", 2024-03-20T09:02:55.905 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0015230280000000001 2024-03-20T09:02:55.905 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.905 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.905 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:02:55.905 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.832956+0000", 2024-03-20T09:02:55.906 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0014977530000000001 2024-03-20T09:02:55.906 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.906 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.906 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.906 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.954414+0000", 2024-03-20T09:02:55.906 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.121458809 2024-03-20T09:02:55.906 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.906 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.906 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:55.906 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.954443+0000", 2024-03-20T09:02:55.906 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9142e-05 2024-03-20T09:02:55.906 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.906 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.907 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:55.907 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.954465+0000", 2024-03-20T09:02:55.907 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1141999999999998e-05 2024-03-20T09:02:55.907 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.907 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:55.907 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.907 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.907 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.907 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:53490 2.7 2:e474dc8c:::benchmark_data_smithi122_107654_object53489:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-20T09:02:55.907 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:58:45.829205+0000", 2024-03-20T09:02:55.907 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 250.01413458600001, 2024-03-20T09:02:55.907 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.13008857400000001, 2024-03-20T09:02:55.907 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:55.907 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:55.907 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:02:55.908 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T09:02:55.908 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:02:55.908 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 53490 2024-03-20T09:02:55.908 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.908 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:55.908 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.908 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:55.908 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.829205+0000", 2024-03-20T09:02:55.908 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.908 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.908 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.908 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:55.908 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.829205+0000", 2024-03-20T09:02:55.908 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.909 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.909 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.909 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:55.909 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.829206+0000", 2024-03-20T09:02:55.909 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2780000000000001e-06 2024-03-20T09:02:55.909 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.909 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.909 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:55.909 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.829234+0000", 2024-03-20T09:02:55.909 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.7988999999999999e-05 2024-03-20T09:02:55.909 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.909 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.909 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:55.909 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.829250+0000", 2024-03-20T09:02:55.910 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6337999999999998e-05 2024-03-20T09:02:55.910 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.910 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.910 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:55.910 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.829258+0000", 2024-03-20T09:02:55.910 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7829999999999999e-06 2024-03-20T09:02:55.910 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.910 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.910 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:55.910 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.831499+0000", 2024-03-20T09:02:55.910 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0022408480000000001 2024-03-20T09:02:55.910 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.910 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.910 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:55.910 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.831635+0000", 2024-03-20T09:02:55.911 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013605 2024-03-20T09:02:55.911 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.911 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.911 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T09:02:55.911 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.831788+0000", 2024-03-20T09:02:55.911 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00015283500000000001 2024-03-20T09:02:55.911 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.911 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.911 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:02:55.911 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.833143+0000", 2024-03-20T09:02:55.911 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001355081 2024-03-20T09:02:55.911 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.911 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.911 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.911 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.834962+0000", 2024-03-20T09:02:55.912 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0018187329999999999 2024-03-20T09:02:55.912 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.912 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.912 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.912 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.959164+0000", 2024-03-20T09:02:55.912 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.124202327 2024-03-20T09:02:55.912 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.912 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.912 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:55.912 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.959225+0000", 2024-03-20T09:02:55.912 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1253999999999993e-05 2024-03-20T09:02:55.912 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.912 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.912 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:55.913 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.959293+0000", 2024-03-20T09:02:55.913 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.8058000000000005e-05 2024-03-20T09:02:55.913 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.913 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:55.913 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.913 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.913 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.913 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:53497 2.7 2:e0729c41:::benchmark_data_smithi122_107654_object53496:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-20T09:02:55.913 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:58:45.833199+0000", 2024-03-20T09:02:55.913 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 250.01013983199999, 2024-03-20T09:02:55.913 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.1262344, 2024-03-20T09:02:55.913 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:55.913 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:55.913 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:02:55.914 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T09:02:55.914 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:02:55.914 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 53497 2024-03-20T09:02:55.914 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.914 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:55.914 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.914 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:55.914 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.833199+0000", 2024-03-20T09:02:55.914 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.914 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.914 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.914 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:55.914 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.833199+0000", 2024-03-20T09:02:55.915 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.915 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.915 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.915 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:55.915 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.833201+0000", 2024-03-20T09:02:55.915 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.029e-06 2024-03-20T09:02:55.915 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.915 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.915 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:55.915 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.833277+0000", 2024-03-20T09:02:55.915 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.6118000000000005e-05 2024-03-20T09:02:55.915 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.915 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.915 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:55.915 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.833287+0000", 2024-03-20T09:02:55.916 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0753e-05 2024-03-20T09:02:55.916 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.916 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.916 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:55.916 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.833293+0000", 2024-03-20T09:02:55.916 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0299999999999999e-06 2024-03-20T09:02:55.916 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.916 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.916 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:55.916 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.833339+0000", 2024-03-20T09:02:55.916 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5467999999999999e-05 2024-03-20T09:02:55.916 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.916 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.916 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:55.917 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.833460+0000", 2024-03-20T09:02:55.917 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000121144 2024-03-20T09:02:55.917 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.917 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.917 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T09:02:55.917 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.833610+0000", 2024-03-20T09:02:55.917 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000150078 2024-03-20T09:02:55.917 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.917 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.917 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:02:55.917 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.835042+0000", 2024-03-20T09:02:55.917 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0014316140000000001 2024-03-20T09:02:55.917 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.917 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.918 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.918 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.835258+0000", 2024-03-20T09:02:55.918 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00021614099999999999 2024-03-20T09:02:55.918 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.918 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.918 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.918 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.959339+0000", 2024-03-20T09:02:55.918 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.12408108 2024-03-20T09:02:55.918 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.918 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.918 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:55.918 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.959394+0000", 2024-03-20T09:02:55.918 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5077999999999997e-05 2024-03-20T09:02:55.919 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.919 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.919 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:55.919 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.959434+0000", 2024-03-20T09:02:55.919 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.9867000000000003e-05 2024-03-20T09:02:55.919 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.919 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:55.919 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.919 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.919 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.919 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:53515 2.7 2:eb083dc9:::benchmark_data_smithi122_107654_object53514:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-20T09:02:55.919 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:58:45.846705+0000", 2024-03-20T09:02:55.919 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 249.99663444800001, 2024-03-20T09:02:55.919 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.112886466, 2024-03-20T09:02:55.919 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:55.920 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:55.920 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:02:55.920 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T09:02:55.920 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:02:55.920 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 53515 2024-03-20T09:02:55.920 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.920 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:55.920 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.920 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:55.920 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.846705+0000", 2024-03-20T09:02:55.920 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.920 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.920 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.920 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:55.921 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.846705+0000", 2024-03-20T09:02:55.921 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.921 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.921 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.921 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:55.921 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.846706+0000", 2024-03-20T09:02:55.921 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5510000000000001e-06 2024-03-20T09:02:55.921 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.921 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.921 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:55.921 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.846792+0000", 2024-03-20T09:02:55.921 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5709000000000002e-05 2024-03-20T09:02:55.921 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.921 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.921 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:55.922 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.846808+0000", 2024-03-20T09:02:55.922 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5855999999999999e-05 2024-03-20T09:02:55.922 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.922 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.922 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:55.922 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.846816+0000", 2024-03-20T09:02:55.922 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.593e-06 2024-03-20T09:02:55.922 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.922 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.922 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:55.922 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.846879+0000", 2024-03-20T09:02:55.922 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2949999999999999e-05 2024-03-20T09:02:55.922 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.923 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.923 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:55.923 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.847019+0000", 2024-03-20T09:02:55.923 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013999499999999999 2024-03-20T09:02:55.923 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.923 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.923 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T09:02:55.923 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.847170+0000", 2024-03-20T09:02:55.923 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00015141699999999999 2024-03-20T09:02:55.923 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.923 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.923 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:02:55.923 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.849020+0000", 2024-03-20T09:02:55.923 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001850078 2024-03-20T09:02:55.923 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.924 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.924 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.924 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.849103+0000", 2024-03-20T09:02:55.924 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.2665999999999998e-05 2024-03-20T09:02:55.924 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.924 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.924 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.924 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.959485+0000", 2024-03-20T09:02:55.924 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.110381998 2024-03-20T09:02:55.924 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.924 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.924 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:55.924 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.959544+0000", 2024-03-20T09:02:55.924 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9360000000000001e-05 2024-03-20T09:02:55.925 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.925 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.925 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:55.925 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.959591+0000", 2024-03-20T09:02:55.925 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.7293000000000003e-05 2024-03-20T09:02:55.925 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.925 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:55.925 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.925 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.925 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.925 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:53531 2.7 2:e510c7ab:::benchmark_data_smithi122_107654_object53530:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-20T09:02:55.925 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:58:45.856021+0000", 2024-03-20T09:02:55.925 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 249.987318698, 2024-03-20T09:02:55.925 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.103719694, 2024-03-20T09:02:55.926 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:55.926 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:55.926 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:02:55.926 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T09:02:55.926 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:02:55.926 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 53531 2024-03-20T09:02:55.926 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.926 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:55.926 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.926 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:55.926 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.856021+0000", 2024-03-20T09:02:55.926 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.926 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.926 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.927 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:55.927 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.856021+0000", 2024-03-20T09:02:55.927 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.927 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.927 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.927 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:55.927 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.856022+0000", 2024-03-20T09:02:55.927 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.139e-06 2024-03-20T09:02:55.927 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.927 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.927 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:55.927 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.856107+0000", 2024-03-20T09:02:55.927 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5086999999999994e-05 2024-03-20T09:02:55.927 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.927 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.928 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:55.928 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.856120+0000", 2024-03-20T09:02:55.928 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2857000000000001e-05 2024-03-20T09:02:55.928 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.928 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.928 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:55.928 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.856126+0000", 2024-03-20T09:02:55.928 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1330000000000004e-06 2024-03-20T09:02:55.928 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.928 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.928 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:55.928 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.856186+0000", 2024-03-20T09:02:55.928 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0346000000000001e-05 2024-03-20T09:02:55.928 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.929 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.929 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:55.929 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.856326+0000", 2024-03-20T09:02:55.929 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013999100000000001 2024-03-20T09:02:55.929 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.929 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.929 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T09:02:55.929 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.856432+0000", 2024-03-20T09:02:55.929 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000105714 2024-03-20T09:02:55.929 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.929 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.929 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:02:55.929 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.857775+0000", 2024-03-20T09:02:55.929 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0013426810000000001 2024-03-20T09:02:55.930 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.930 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.930 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.930 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.858605+0000", 2024-03-20T09:02:55.930 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00083076600000000003 2024-03-20T09:02:55.930 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.930 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.930 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.930 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.959657+0000", 2024-03-20T09:02:55.930 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.10105141099999999 2024-03-20T09:02:55.930 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.930 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.930 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:55.930 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.959705+0000", 2024-03-20T09:02:55.931 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8313999999999999e-05 2024-03-20T09:02:55.931 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.931 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.931 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:55.931 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.959740+0000", 2024-03-20T09:02:55.931 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.5255e-05 2024-03-20T09:02:55.931 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.931 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:55.931 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.931 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.931 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.931 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:53560 2.7 2:e40219b5:::benchmark_data_smithi122_107654_object53559:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-20T09:02:55.931 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:58:45.882597+0000", 2024-03-20T09:02:55.931 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 249.96074267700001, 2024-03-20T09:02:55.931 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.077277745999999994, 2024-03-20T09:02:55.932 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:55.932 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:55.932 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:02:55.932 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T09:02:55.932 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:02:55.932 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 53560 2024-03-20T09:02:55.932 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.932 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:55.932 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.932 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:55.932 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.882597+0000", 2024-03-20T09:02:55.932 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.932 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.932 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.933 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:55.933 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.882597+0000", 2024-03-20T09:02:55.933 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.933 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.933 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.933 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:55.933 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.882598+0000", 2024-03-20T09:02:55.933 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3239999999999999e-06 2024-03-20T09:02:55.933 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.933 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.933 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:55.933 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.882635+0000", 2024-03-20T09:02:55.933 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7320000000000002e-05 2024-03-20T09:02:55.933 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.933 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.934 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:55.934 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.882649+0000", 2024-03-20T09:02:55.934 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3332999999999999e-05 2024-03-20T09:02:55.934 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.934 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.934 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:55.934 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.882670+0000", 2024-03-20T09:02:55.934 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1049000000000001e-05 2024-03-20T09:02:55.934 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.934 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.934 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:55.934 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.882734+0000", 2024-03-20T09:02:55.934 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4554000000000006e-05 2024-03-20T09:02:55.934 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.935 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.935 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:55.935 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.882872+0000", 2024-03-20T09:02:55.935 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013735599999999999 2024-03-20T09:02:55.935 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.935 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.935 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T09:02:55.935 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.883030+0000", 2024-03-20T09:02:55.935 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000158286 2024-03-20T09:02:55.935 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.935 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.935 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:02:55.935 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.884714+0000", 2024-03-20T09:02:55.935 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0016846299999999999 2024-03-20T09:02:55.936 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.936 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.936 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.936 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.885218+0000", 2024-03-20T09:02:55.936 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00050378300000000005 2024-03-20T09:02:55.936 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.936 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.936 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.936 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.959786+0000", 2024-03-20T09:02:55.936 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.074567533000000005 2024-03-20T09:02:55.936 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.936 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.936 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:55.936 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.959838+0000", 2024-03-20T09:02:55.937 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2287000000000003e-05 2024-03-20T09:02:55.937 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.937 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.937 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:55.937 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.959874+0000", 2024-03-20T09:02:55.937 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6291000000000002e-05 2024-03-20T09:02:55.937 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.937 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:55.937 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.937 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.937 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.937 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:53568 2.7 2:efebfd9b:::benchmark_data_smithi122_107654_object53567:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-20T09:02:55.937 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:58:45.888935+0000", 2024-03-20T09:02:55.937 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 249.95440421199999, 2024-03-20T09:02:55.937 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.071078239000000001, 2024-03-20T09:02:55.938 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:55.938 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:55.938 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:02:55.938 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T09:02:55.938 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:02:55.938 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 53568 2024-03-20T09:02:55.938 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.938 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:55.938 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.938 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:55.938 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.888935+0000", 2024-03-20T09:02:55.938 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.938 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.939 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.939 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:55.939 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.888935+0000", 2024-03-20T09:02:55.939 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.939 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.939 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.939 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:55.939 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.888936+0000", 2024-03-20T09:02:55.939 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.372e-06 2024-03-20T09:02:55.939 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.939 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.939 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:55.939 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.888985+0000", 2024-03-20T09:02:55.939 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8231e-05 2024-03-20T09:02:55.940 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.940 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.940 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:55.940 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.888997+0000", 2024-03-20T09:02:55.940 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2293000000000001e-05 2024-03-20T09:02:55.940 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.940 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.940 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:55.940 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.889004+0000", 2024-03-20T09:02:55.940 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.7710000000000001e-06 2024-03-20T09:02:55.940 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.940 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.940 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:55.940 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.889057+0000", 2024-03-20T09:02:55.941 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3489000000000002e-05 2024-03-20T09:02:55.941 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.941 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.941 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:55.941 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.889195+0000", 2024-03-20T09:02:55.941 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013759699999999999 2024-03-20T09:02:55.941 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.941 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.941 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T09:02:55.941 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.889350+0000", 2024-03-20T09:02:55.941 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00015498899999999999 2024-03-20T09:02:55.941 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.941 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.941 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:02:55.942 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.890715+0000", 2024-03-20T09:02:55.942 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0013651570000000001 2024-03-20T09:02:55.942 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.942 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.942 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.942 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.891085+0000", 2024-03-20T09:02:55.942 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000370465 2024-03-20T09:02:55.942 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.942 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.942 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.942 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.959923+0000", 2024-03-20T09:02:55.942 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.068837630999999996 2024-03-20T09:02:55.942 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.942 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.943 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:55.943 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.959976+0000", 2024-03-20T09:02:55.943 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2496e-05 2024-03-20T09:02:55.943 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.943 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.943 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:55.943 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.960013+0000", 2024-03-20T09:02:55.943 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7747999999999998e-05 2024-03-20T09:02:55.943 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.943 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:55.943 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.943 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.943 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.943 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:53574 2.7 2:e09cc1d6:::benchmark_data_smithi122_107654_object53573:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-20T09:02:55.943 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:58:45.895630+0000", 2024-03-20T09:02:55.944 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 249.947709315, 2024-03-20T09:02:55.944 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.064527031999999998, 2024-03-20T09:02:55.944 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:55.944 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:55.944 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:02:55.944 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T09:02:55.944 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:02:55.944 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 53574 2024-03-20T09:02:55.944 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.944 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:55.944 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.944 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:55.944 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.895630+0000", 2024-03-20T09:02:55.944 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.944 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.945 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.945 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:55.945 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.895630+0000", 2024-03-20T09:02:55.945 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.945 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.945 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.945 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:55.945 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.895631+0000", 2024-03-20T09:02:55.945 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.009e-06 2024-03-20T09:02:55.945 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.945 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.945 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:55.945 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.895660+0000", 2024-03-20T09:02:55.945 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9028999999999999e-05 2024-03-20T09:02:55.946 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.946 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.946 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:55.946 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.895671+0000", 2024-03-20T09:02:55.946 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0919e-05 2024-03-20T09:02:55.946 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.946 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.946 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:55.946 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.895677+0000", 2024-03-20T09:02:55.946 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7250000000000002e-06 2024-03-20T09:02:55.946 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.946 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.946 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:55.947 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.895722+0000", 2024-03-20T09:02:55.947 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4997999999999998e-05 2024-03-20T09:02:55.947 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.947 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.947 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:55.947 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.895805+0000", 2024-03-20T09:02:55.947 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.3079999999999997e-05 2024-03-20T09:02:55.947 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.947 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.947 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T09:02:55.947 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.895900+0000", 2024-03-20T09:02:55.947 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.5571999999999999e-05 2024-03-20T09:02:55.947 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.947 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.948 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:02:55.948 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.897051+0000", 2024-03-20T09:02:55.948 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0011508060000000001 2024-03-20T09:02:55.948 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.948 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.948 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.948 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.897607+0000", 2024-03-20T09:02:55.948 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00055633400000000002 2024-03-20T09:02:55.948 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.948 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.948 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.948 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.960062+0000", 2024-03-20T09:02:55.948 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.062454161000000001 2024-03-20T09:02:55.948 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.949 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.949 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:55.949 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.960116+0000", 2024-03-20T09:02:55.949 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4750000000000003e-05 2024-03-20T09:02:55.949 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.949 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.949 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:55.949 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.960157+0000", 2024-03-20T09:02:55.949 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.0649000000000003e-05 2024-03-20T09:02:55.949 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.949 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:55.949 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.949 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.949 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.949 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:53581 2.7 2:e940c4ba:::benchmark_data_smithi122_107654_object53580:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-20T09:02:55.949 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T08:58:45.912437+0000", 2024-03-20T09:02:55.950 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 249.93090256799999, 2024-03-20T09:02:55.950 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.04784646, 2024-03-20T09:02:55.950 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:55.950 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:55.950 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:02:55.950 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T09:02:55.950 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:02:55.950 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 53581 2024-03-20T09:02:55.950 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.950 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:55.950 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.950 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:55.950 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.912437+0000", 2024-03-20T09:02:55.950 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.950 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.951 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.951 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:55.951 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.912437+0000", 2024-03-20T09:02:55.951 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.951 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.951 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.951 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:55.951 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.912439+0000", 2024-03-20T09:02:55.951 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3630000000000001e-06 2024-03-20T09:02:55.951 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.951 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.951 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:55.951 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.912493+0000", 2024-03-20T09:02:55.951 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4351999999999997e-05 2024-03-20T09:02:55.951 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.952 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.952 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:55.952 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.912522+0000", 2024-03-20T09:02:55.952 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8517000000000002e-05 2024-03-20T09:02:55.952 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.952 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.952 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:55.952 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.912549+0000", 2024-03-20T09:02:55.952 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6863999999999999e-05 2024-03-20T09:02:55.952 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.952 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.952 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:55.952 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.912618+0000", 2024-03-20T09:02:55.952 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.9330000000000002e-05 2024-03-20T09:02:55.953 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.953 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.953 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:55.953 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.912758+0000", 2024-03-20T09:02:55.953 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000140252 2024-03-20T09:02:55.953 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.953 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.953 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T09:02:55.953 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.912921+0000", 2024-03-20T09:02:55.953 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000162516 2024-03-20T09:02:55.953 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.953 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.953 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:02:55.953 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.914214+0000", 2024-03-20T09:02:55.953 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0012933280000000001 2024-03-20T09:02:55.954 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.954 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.954 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.954 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.915283+0000", 2024-03-20T09:02:55.954 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00106832 2024-03-20T09:02:55.954 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.954 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.954 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.954 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.960199+0000", 2024-03-20T09:02:55.954 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.044916606999999997 2024-03-20T09:02:55.954 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.954 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.954 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:55.954 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.960251+0000", 2024-03-20T09:02:55.955 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1734000000000003e-05 2024-03-20T09:02:55.955 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.955 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.955 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:55.955 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T08:58:45.960283+0000", 2024-03-20T09:02:55.955 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.2277000000000003e-05 2024-03-20T09:02:55.955 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.955 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:55.955 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.955 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.955 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.955 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:160999 2.7 2:e164057e:::benchmark_data_smithi122_107654_object42214:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-03-20T09:02:55.955 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:00:49.583634+0000", 2024-03-20T09:02:55.955 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 126.259705599, 2024-03-20T09:02:55.956 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.048285641999999997, 2024-03-20T09:02:55.956 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:55.956 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:55.956 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:02:55.956 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T09:02:55.956 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:02:55.956 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 160999 2024-03-20T09:02:55.956 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.956 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:55.956 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.956 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:55.956 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:00:49.583634+0000", 2024-03-20T09:02:55.956 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.956 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.957 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.957 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:55.957 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:00:49.583634+0000", 2024-03-20T09:02:55.957 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.957 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.957 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.957 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:55.957 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:00:49.583635+0000", 2024-03-20T09:02:55.957 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4139999999999999e-06 2024-03-20T09:02:55.957 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.957 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.957 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:55.957 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:00:49.583641+0000", 2024-03-20T09:02:55.957 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1809999999999998e-06 2024-03-20T09:02:55.957 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.958 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.958 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:55.958 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:00:49.583654+0000", 2024-03-20T09:02:55.958 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2653000000000001e-05 2024-03-20T09:02:55.958 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.958 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.958 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:55.958 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:00:49.583662+0000", 2024-03-20T09:02:55.958 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.3720000000000005e-06 2024-03-20T09:02:55.958 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.958 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.958 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:55.958 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:00:49.583805+0000", 2024-03-20T09:02:55.958 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00014282699999999999 2024-03-20T09:02:55.959 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.959 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.959 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:55.959 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:00:49.583902+0000", 2024-03-20T09:02:55.959 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.7003000000000006e-05 2024-03-20T09:02:55.959 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.959 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.959 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T09:02:55.959 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:00:49.583968+0000", 2024-03-20T09:02:55.959 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.5393000000000002e-05 2024-03-20T09:02:55.959 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.959 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.959 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:02:55.959 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:00:49.587735+0000", 2024-03-20T09:02:55.959 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0037677309999999999 2024-03-20T09:02:55.960 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.960 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.960 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.960 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:00:49.596503+0000", 2024-03-20T09:02:55.960 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.008768049 2024-03-20T09:02:55.960 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.960 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.960 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.960 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:00:49.631815+0000", 2024-03-20T09:02:55.960 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.035311411000000001 2024-03-20T09:02:55.960 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.960 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.960 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:55.960 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:00:49.631873+0000", 2024-03-20T09:02:55.961 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8567999999999999e-05 2024-03-20T09:02:55.961 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.961 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.961 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:55.961 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:00:49.631919+0000", 2024-03-20T09:02:55.961 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.604e-05 2024-03-20T09:02:55.961 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.961 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:55.961 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.961 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.961 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.961 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24550.0:177880 2.7 2:e2d17c61:::benchmark_data_smithi122_107654_object59095:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-03-20T09:02:55.961 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:01:23.854338+0000", 2024-03-20T09:02:55.961 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 91.989000951999998, 2024-03-20T09:02:55.962 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.048086619999999997, 2024-03-20T09:02:55.962 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:02:55.962 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:02:55.962 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:02:55.962 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24550", 2024-03-20T09:02:55.962 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:02:55.962 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 177880 2024-03-20T09:02:55.962 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.962 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:02:55.962 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.962 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:02:55.962 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:23.854338+0000", 2024-03-20T09:02:55.962 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.962 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.963 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.963 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:02:55.963 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:23.854338+0000", 2024-03-20T09:02:55.963 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:02:55.963 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.963 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.963 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:02:55.963 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:23.854340+0000", 2024-03-20T09:02:55.963 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5120000000000001e-06 2024-03-20T09:02:55.963 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.963 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.963 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:02:55.963 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:23.854347+0000", 2024-03-20T09:02:55.963 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.8190000000000003e-06 2024-03-20T09:02:55.964 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.964 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.964 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:02:55.964 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:23.854371+0000", 2024-03-20T09:02:55.964 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.4355999999999999e-05 2024-03-20T09:02:55.964 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.964 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.964 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:02:55.964 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:23.854392+0000", 2024-03-20T09:02:55.964 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1075000000000001e-05 2024-03-20T09:02:55.964 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.964 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.964 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:02:55.964 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:23.854466+0000", 2024-03-20T09:02:55.964 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3715999999999997e-05 2024-03-20T09:02:55.965 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.965 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.965 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:02:55.965 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:23.854620+0000", 2024-03-20T09:02:55.965 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000154568 2024-03-20T09:02:55.965 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.965 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.965 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,10", 2024-03-20T09:02:55.965 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:23.854703+0000", 2024-03-20T09:02:55.965 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.3070000000000003e-05 2024-03-20T09:02:55.965 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.965 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.965 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.965 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:23.859636+0000", 2024-03-20T09:02:55.966 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0049327559999999999 2024-03-20T09:02:55.966 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.966 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.966 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:02:55.966 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:23.859658+0000", 2024-03-20T09:02:55.966 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1729000000000002e-05 2024-03-20T09:02:55.966 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.966 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.966 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:02:55.966 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:23.902306+0000", 2024-03-20T09:02:55.966 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.042648299000000001 2024-03-20T09:02:55.966 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.966 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.966 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:02:55.967 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:23.902370+0000", 2024-03-20T09:02:55.967 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4109999999999994e-05 2024-03-20T09:02:55.967 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:02:55.967 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:02:55.967 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:02:55.967 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:01:23.902425+0000", 2024-03-20T09:02:55.967 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4610000000000001e-05 2024-03-20T09:02:55.967 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.967 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:55.967 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.967 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:02:55.967 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:02:55.967 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:02:55.968 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:02:55.969 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:55 smithi122 ceph-mon[73495]: pgmap v598: 27 pgs: 27 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:02:55.969 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:02:55.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:55 smithi026 ceph-mon[68309]: pgmap v598: 27 pgs: 27 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:02:55.995 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:56.067 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:02:56.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:56.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:02:56.172 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:02:56.201 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:56.525 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-20T09:02:56.550 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:02:56.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:56.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:02:56.654 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:02:56.681 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:56.681 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:02:56.700 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:02:56.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:56.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:02:56.804 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:02:56.816 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:02:56.837 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:56.858 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:02:56.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:56.861 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:02:56.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:56 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3392607289' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:02:56.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:56 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3392607289' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:02:56.962 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:02:56.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:56 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3392607289' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:02:56.990 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:57.088 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:02:57.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:57.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:02:57.192 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:02:57.219 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:57.396 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:02:57.682 INFO:teuthology.orchestra.run.smithi026.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-03-20T09:02:57.947 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:57 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:02:57.947 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:57 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3392607289' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:02:57.947 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:57 smithi026 ceph-mon[68309]: osdmap e209: 12 total, 12 up, 12 in 2024-03-20T09:02:57.947 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:57 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2789616126' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-20T09:02:57.947 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:57 smithi026 ceph-mon[68309]: pgmap v600: 27 pgs: 27 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:02:57.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:57 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:02:57.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:57 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3392607289' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:02:57.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:57 smithi122 ceph-mon[73495]: osdmap e209: 12 total, 12 up, 12 in 2024-03-20T09:02:57.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:57 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2789616126' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-20T09:02:57.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:57 smithi122 ceph-mon[73495]: pgmap v600: 27 pgs: 27 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:02:58.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:57 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:02:58.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:57 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3392607289' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:02:58.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:57 smithi115 ceph-mon[72476]: osdmap e209: 12 total, 12 up, 12 in 2024-03-20T09:02:58.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:57 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2789616126' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-20T09:02:58.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:57 smithi115 ceph-mon[72476]: pgmap v600: 27 pgs: 27 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:02:58.169 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:02:58.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:58.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:02:58.273 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:02:58.300 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:58.448 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:02:58.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:58.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:02:58.552 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:02:58.585 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:58.607 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:02:58.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:58.610 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:02:58.711 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:02:58.721 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:02:58.721 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:02:58.721 INFO:teuthology.orchestra.run.smithi115.stdout: "num_ops": 0 2024-03-20T09:02:58.721 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:02:58.735 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:58 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2789616126' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-20T09:02:58.735 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:58 smithi115 ceph-mon[72476]: osdmap e210: 12 total, 12 up, 12 in 2024-03-20T09:02:58.740 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:58.759 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:02:58.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:58.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:02:58.863 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:02:58.891 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:58 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2789616126' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-20T09:02:58.892 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:58 smithi122 ceph-mon[73495]: osdmap e210: 12 total, 12 up, 12 in 2024-03-20T09:02:58.894 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:58.914 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:02:58.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:58.918 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:02:58.922 INFO:teuthology.orchestra.run.smithi026.stdout:moved 0 / 81 (0%) 2024-03-20T09:02:58.922 INFO:teuthology.orchestra.run.smithi026.stdout:avg 6.75 2024-03-20T09:02:58.923 INFO:teuthology.orchestra.run.smithi026.stdout:stddev 2.16506 -> 2.16506 (expected baseline 2.48747) 2024-03-20T09:02:58.923 INFO:teuthology.orchestra.run.smithi026.stdout:min osd.1 with 4 -> 4 pgs (0.592593 -> 0.592593 * mean) 2024-03-20T09:02:58.923 INFO:teuthology.orchestra.run.smithi026.stdout:max osd.4 with 11 -> 11 pgs (1.62963 -> 1.62963 * mean) 2024-03-20T09:02:58.923 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:02:58.923 INFO:teuthology.orchestra.run.smithi026.stdout:oload 1000 2024-03-20T09:02:58.923 INFO:teuthology.orchestra.run.smithi026.stdout:max_change 3 2024-03-20T09:02:58.923 INFO:teuthology.orchestra.run.smithi026.stdout:max_change_osds 4 2024-03-20T09:02:58.923 INFO:teuthology.orchestra.run.smithi026.stdout:average_utilization 0.0010 2024-03-20T09:02:58.923 INFO:teuthology.orchestra.run.smithi026.stdout:overload_utilization 0.0100 2024-03-20T09:02:58.923 INFO:teuthology.orchestra.run.smithi026.stderr:no change 2024-03-20T09:02:58.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:58 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2789616126' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-20T09:02:58.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:58 smithi026 ceph-mon[68309]: osdmap e210: 12 total, 12 up, 12 in 2024-03-20T09:02:59.019 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:02:59.047 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:59.066 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:02:59.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:59.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:02:59.177 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:02:59.210 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:59.231 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:02:59.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:59.235 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:02:59.336 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:02:59.366 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:59.474 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-20T09:02:59.523 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:02:59.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:02:59.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:02:59.530 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-03-20T09:02:59.530 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight-by-utilization 110 3.0 2024-03-20T09:02:59.628 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:02:59.655 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:02:59.737 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:02:59.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:59 smithi115 ceph-mon[72476]: from='client.15066 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:02:59.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:02:59 smithi115 ceph-mon[72476]: pgmap v602: 27 pgs: 27 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:02:59.909 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:59 smithi026 ceph-mon[68309]: from='client.15066 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:02:59.909 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:02:59 smithi026 ceph-mon[68309]: pgmap v602: 27 pgs: 27 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:02:59.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:59 smithi122 ceph-mon[73495]: from='client.15066 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:02:59.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:02:59 smithi122 ceph-mon[73495]: pgmap v602: 27 pgs: 27 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:03:00.004 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:03:00.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:00.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:03:00.107 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:03:00.135 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:00.501 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:03:00.546 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:03:00.768 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:00 smithi026 ceph-mon[68309]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-20T09:03:00.768 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:00 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:03:00.768 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:00 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:03:00.768 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:00 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:03:00.768 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:00 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:03:00.768 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:00 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:03:00.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:00 smithi122 ceph-mon[73495]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-20T09:03:00.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:00 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:03:00.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:00 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:03:00.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:00 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:03:00.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:00 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:03:00.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:00 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:03:00.970 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:00 smithi115 ceph-mon[72476]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-20T09:03:00.970 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:00 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:03:00.970 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:00 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:03:00.970 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:00 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:03:00.970 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:00 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:03:00.970 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:00 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:03:00.983 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:03:00.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:00.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:03:01.088 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:03:01.121 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:01.142 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:03:01.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:01.145 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:03:01.246 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:03:01.276 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:01.295 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:03:01.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:01.299 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:03:01.400 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:03:01.432 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:01.547 DEBUG:teuthology.orchestra.run.smithi122:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-03-20T09:03:01.549 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-20T09:03:01.550 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-20T09:03:01.601 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:hints = 1 2024-03-20T09:03:01.601 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-20T09:03:01.601 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Object prefix: benchmark_data_smithi122_119671 2024-03-20T09:03:01.601 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T09:03:01.601 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 0 0 0 0 0 0 - 0 2024-03-20T09:03:01.610 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:03:01.610 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:03:01.610 INFO:teuthology.orchestra.run.smithi115.stdout: "complaint_time": 30, 2024-03-20T09:03:01.610 INFO:teuthology.orchestra.run.smithi115.stdout: "num_blocked_ops": 0 2024-03-20T09:03:01.610 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:03:01.784 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:03:01.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:01.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:03:01.890 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:03:01.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:01 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:03:01.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:01 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:03:01.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:01 smithi115 ceph-mon[72476]: osdmap e211: 12 total, 12 up, 12 in 2024-03-20T09:03:01.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:01 smithi115 ceph-mon[72476]: pgmap v604: 27 pgs: 27 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:03:01.935 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:01.944 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:03:01.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:01 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:03:01.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:01 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:03:01.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:01 smithi122 ceph-mon[73495]: osdmap e211: 12 total, 12 up, 12 in 2024-03-20T09:03:01.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:01 smithi122 ceph-mon[73495]: pgmap v604: 27 pgs: 27 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:03:01.950 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:01 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:03:01.950 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:01 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:03:01.951 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:01 smithi026 ceph-mon[68309]: osdmap e211: 12 total, 12 up, 12 in 2024-03-20T09:03:01.951 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:01 smithi026 ceph-mon[68309]: pgmap v604: 27 pgs: 27 active+clean; 577 KiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:03:02.344 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-20T09:03:02.400 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:03:02.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:02.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:03:02.509 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:03:02.543 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:02.566 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:03:02.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:02.570 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:03:02.601 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 1 16 3596 3580 223.722 223.75 0.00128144 0.00412678 2024-03-20T09:03:02.671 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:03:02.692 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:03:02.708 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:02.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:02.787 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:03:02.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:03:02.888 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:03:02.930 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:02.960 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:03:02.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:02.964 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:03:03.065 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:03:03.108 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:03.540 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:03:03.577 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:03:03.601 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 2 16 7263 7247 226.444 229.188 0.00277187 0.00440644 2024-03-20T09:03:03.621 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:03:03.660 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:03:03.683 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:03:03.698 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:03:03.737 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:03:03.771 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:03:03.792 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:03:03.813 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:03:03.836 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:03:03.858 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:03:03.882 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:03:03.883 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:03:03.883 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:03:03.883 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:03:03.884 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:03:03.884 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:03:03.884 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:03:03.884 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:03:03.884 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:03:03.884 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:03:03.884 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:03:03.884 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:03:03.884 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:03:03.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:03 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1878282946' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:03:03.907 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:03 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1878282946' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:03:03.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:03 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1878282946' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:03:04.571 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:03:04.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:04.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:03:04.581 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:04 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:03:04.581 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:04 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:03:04.581 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:04 smithi026 ceph-mon[68309]: osdmap e212: 12 total, 12 up, 12 in 2024-03-20T09:03:04.581 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:04 smithi026 ceph-mon[68309]: pgmap v606: 27 pgs: 27 active+clean; 51 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s wr, 135 op/s 2024-03-20T09:03:04.581 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:04 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'32653','3':'35063','9':'34736'}"}]: dispatch 2024-03-20T09:03:04.581 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:04 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'32653','3':'35063','9':'34736'}"}]: dispatch 2024-03-20T09:03:04.597 INFO:teuthology.orchestra.run.smithi026.stdout:moved 9 / 81 (11.1111%) 2024-03-20T09:03:04.597 INFO:teuthology.orchestra.run.smithi026.stdout:avg 6.75 2024-03-20T09:03:04.598 INFO:teuthology.orchestra.run.smithi026.stdout:stddev 2.16506 -> 3.03109 (expected baseline 2.48747) 2024-03-20T09:03:04.598 INFO:teuthology.orchestra.run.smithi026.stdout:min osd.1 with 4 -> 5 pgs (0.592593 -> 0.740741 * mean) 2024-03-20T09:03:04.598 INFO:teuthology.orchestra.run.smithi026.stdout:max osd.4 with 11 -> 12 pgs (1.62963 -> 1.77778 * mean) 2024-03-20T09:03:04.598 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:03:04.598 INFO:teuthology.orchestra.run.smithi026.stdout:oload 110 2024-03-20T09:03:04.598 INFO:teuthology.orchestra.run.smithi026.stdout:max_change 3 2024-03-20T09:03:04.598 INFO:teuthology.orchestra.run.smithi026.stdout:max_change_osds 4 2024-03-20T09:03:04.598 INFO:teuthology.orchestra.run.smithi026.stdout:average_utilization 0.0011 2024-03-20T09:03:04.598 INFO:teuthology.orchestra.run.smithi026.stdout:overload_utilization 0.0012 2024-03-20T09:03:04.598 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0 weight 0.8244 -> 0.4982 2024-03-20T09:03:04.598 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9 weight 0.7781 -> 0.5300 2024-03-20T09:03:04.599 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3 weight 0.6832 -> 0.5350 2024-03-20T09:03:04.601 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 3 16 10565 10549 219.749 206.375 0.0094802 0.00453383 2024-03-20T09:03:04.676 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:03:04.708 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:04.831 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:04 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:03:04.831 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:04 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:03:04.831 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:04 smithi115 ceph-mon[72476]: osdmap e212: 12 total, 12 up, 12 in 2024-03-20T09:03:04.831 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:04 smithi115 ceph-mon[72476]: pgmap v606: 27 pgs: 27 active+clean; 51 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s wr, 135 op/s 2024-03-20T09:03:04.831 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:04 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'32653','3':'35063','9':'34736'}"}]: dispatch 2024-03-20T09:03:04.831 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:04 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'32653','3':'35063','9':'34736'}"}]: dispatch 2024-03-20T09:03:04.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:04 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:03:04.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:04 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:03:04.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:04 smithi122 ceph-mon[73495]: osdmap e212: 12 total, 12 up, 12 in 2024-03-20T09:03:04.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:04 smithi122 ceph-mon[73495]: pgmap v606: 27 pgs: 27 active+clean; 51 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s wr, 135 op/s 2024-03-20T09:03:04.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:04 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'32653','3':'35063','9':'34736'}"}]: dispatch 2024-03-20T09:03:04.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:04 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'32653','3':'35063','9':'34736'}"}]: dispatch 2024-03-20T09:03:05.602 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 4 16 10587 10571 165.156 1.375 0.00187923 0.0045335 2024-03-20T09:03:05.825 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:05 smithi115 ceph-mon[72476]: from='client.24919 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:03:05.826 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:05 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'32653','3':'35063','9':'34736'}"}]': finished 2024-03-20T09:03:05.826 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:05 smithi115 ceph-mon[72476]: osdmap e213: 12 total, 12 up, 12 in 2024-03-20T09:03:05.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:05 smithi122 ceph-mon[73495]: from='client.24919 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:03:05.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:05 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'32653','3':'35063','9':'34736'}"}]': finished 2024-03-20T09:03:05.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:05 smithi122 ceph-mon[73495]: osdmap e213: 12 total, 12 up, 12 in 2024-03-20T09:03:05.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:05 smithi026 ceph-mon[68309]: from='client.24919 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:03:05.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:05 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'32653','3':'35063','9':'34736'}"}]': finished 2024-03-20T09:03:05.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:05 smithi026 ceph-mon[68309]: osdmap e213: 12 total, 12 up, 12 in 2024-03-20T09:03:06.142 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:03:06.142 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:03:06.603 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 5 16 10587 10571 132.099 0 - 0.0045335 2024-03-20T09:03:06.604 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:06 smithi026 ceph-mon[68309]: osdmap e214: 12 total, 12 up, 12 in 2024-03-20T09:03:06.604 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:06 smithi026 ceph-mon[68309]: pgmap v609: 27 pgs: 1 peering, 3 unknown, 23 active+clean; 471 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 117 MiB/s wr, 1.87k op/s; 23 KiB/s, 0 objects/s recovering 2024-03-20T09:03:06.604 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:06 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:03:06.604 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:06 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:03:06.804 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:06 smithi115 ceph-mon[72476]: osdmap e214: 12 total, 12 up, 12 in 2024-03-20T09:03:06.804 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:06 smithi115 ceph-mon[72476]: pgmap v609: 27 pgs: 1 peering, 3 unknown, 23 active+clean; 471 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 117 MiB/s wr, 1.87k op/s; 23 KiB/s, 0 objects/s recovering 2024-03-20T09:03:06.804 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:06 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:03:06.804 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:06 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:03:06.858 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_ops_in_flight 2024-03-20T09:03:06.898 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:06 smithi122 ceph-mon[73495]: osdmap e214: 12 total, 12 up, 12 in 2024-03-20T09:03:06.898 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:06 smithi122 ceph-mon[73495]: pgmap v609: 27 pgs: 1 peering, 3 unknown, 23 active+clean; 471 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 117 MiB/s wr, 1.87k op/s; 23 KiB/s, 0 objects/s recovering 2024-03-20T09:03:06.898 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:06 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:03:06.898 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:06 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:03:07.237 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:03:07.603 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 6 15 12877 12862 133.933 71.5938 0.00543854 0.00745746 2024-03-20T09:03:07.814 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:07 smithi122 ceph-mon[73495]: osdmap e215: 12 total, 12 up, 12 in 2024-03-20T09:03:07.814 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:07 smithi122 ceph-mon[73495]: osdmap e216: 12 total, 12 up, 12 in 2024-03-20T09:03:07.814 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:07 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:03:07.814 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:07 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:03:07.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:07 smithi115 ceph-mon[72476]: osdmap e215: 12 total, 12 up, 12 in 2024-03-20T09:03:07.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:07 smithi115 ceph-mon[72476]: osdmap e216: 12 total, 12 up, 12 in 2024-03-20T09:03:07.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:07 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:03:07.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:07 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:03:07.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:07 smithi026 ceph-mon[68309]: osdmap e215: 12 total, 12 up, 12 in 2024-03-20T09:03:07.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:07 smithi026 ceph-mon[68309]: osdmap e216: 12 total, 12 up, 12 in 2024-03-20T09:03:07.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:07 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:03:07.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:07 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:03:08.604 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 7 16 15467 15451 137.9 161.812 0.00983594 0.0072455 2024-03-20T09:03:08.908 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:08 smithi115 ceph-mon[72476]: pgmap v612: 27 pgs: 1 active+recovering+undersized+remapped, 2 remapped+peering, 1 peering, 3 unknown, 20 active+clean; 659 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 152 MiB/s wr, 2.44k op/s; 559/31617 objects misplaced (1.768%); 1.2 MiB/s, 19 objects/s recovering 2024-03-20T09:03:08.908 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:08 smithi115 ceph-mon[72476]: osdmap e217: 12 total, 12 up, 12 in 2024-03-20T09:03:08.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:08 smithi122 ceph-mon[73495]: pgmap v612: 27 pgs: 1 active+recovering+undersized+remapped, 2 remapped+peering, 1 peering, 3 unknown, 20 active+clean; 659 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 152 MiB/s wr, 2.44k op/s; 559/31617 objects misplaced (1.768%); 1.2 MiB/s, 19 objects/s recovering 2024-03-20T09:03:08.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:08 smithi122 ceph-mon[73495]: osdmap e217: 12 total, 12 up, 12 in 2024-03-20T09:03:08.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:08 smithi026 ceph-mon[68309]: pgmap v612: 27 pgs: 1 active+recovering+undersized+remapped, 2 remapped+peering, 1 peering, 3 unknown, 20 active+clean; 659 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 152 MiB/s wr, 2.44k op/s; 559/31617 objects misplaced (1.768%); 1.2 MiB/s, 19 objects/s recovering 2024-03-20T09:03:08.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:08 smithi026 ceph-mon[68309]: osdmap e217: 12 total, 12 up, 12 in 2024-03-20T09:03:09.487 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:03:09.488 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:03:09.604 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 8 16 18062 18046 140.934 162.188 0.00339278 0.00709007 2024-03-20T09:03:10.218 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_blocked_ops 2024-03-20T09:03:10.362 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:03:10.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:10.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:03:10.466 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:03:10.496 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:10.514 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:03:10.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:10.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:03:10.518 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:03:10.604 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 9 16 18168 18152 126.013 6.625 0.00240065 0.00708042 2024-03-20T09:03:10.618 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:03:10.652 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:10.674 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:03:10.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:10.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:03:10.778 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:03:10.807 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:10.826 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:03:10.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:10.829 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:03:10.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:10 smithi115 ceph-mon[72476]: pgmap v614: 27 pgs: 3 active+recovering+undersized+remapped, 2 remapped+peering, 1 peering, 3 unknown, 18 active+clean; 781 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.24k op/s; 715/37482 objects misplaced (1.908%); 20 MiB/s, 319 objects/s recovering 2024-03-20T09:03:10.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:10 smithi115 ceph-mon[72476]: osdmap e218: 12 total, 12 up, 12 in 2024-03-20T09:03:10.930 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:03:10.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:10 smithi122 ceph-mon[73495]: pgmap v614: 27 pgs: 3 active+recovering+undersized+remapped, 2 remapped+peering, 1 peering, 3 unknown, 18 active+clean; 781 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.24k op/s; 715/37482 objects misplaced (1.908%); 20 MiB/s, 319 objects/s recovering 2024-03-20T09:03:10.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:10 smithi122 ceph-mon[73495]: osdmap e218: 12 total, 12 up, 12 in 2024-03-20T09:03:10.957 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:10.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:10 smithi026 ceph-mon[68309]: pgmap v614: 27 pgs: 3 active+recovering+undersized+remapped, 2 remapped+peering, 1 peering, 3 unknown, 18 active+clean; 781 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.24k op/s; 715/37482 objects misplaced (1.908%); 20 MiB/s, 319 objects/s recovering 2024-03-20T09:03:10.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:10 smithi026 ceph-mon[68309]: osdmap e218: 12 total, 12 up, 12 in 2024-03-20T09:03:11.029 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:03:11.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:11.032 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:03:11.133 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:03:11.161 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:11.181 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:03:11.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:11.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:03:11.285 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:03:11.311 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:11.330 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:03:11.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:11.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:03:11.434 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:03:11.461 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:11.604 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 10 16 18247 18231 113.908 4.9375 0.00180025 0.00794058 2024-03-20T09:03:11.636 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:03:11.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:11 smithi122 ceph-mon[73495]: osdmap e219: 12 total, 12 up, 12 in 2024-03-20T09:03:11.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:11 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/274416010' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:03:11.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:11 smithi122 ceph-mon[73495]: pgmap v617: 27 pgs: 1 peering, 4 active+recovering+undersized+remapped, 1 active+undersized+remapped, 21 active+clean; 985 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s wr, 1.59k op/s; 1049/47268 objects misplaced (2.219%); 39 MiB/s, 616 objects/s recovering 2024-03-20T09:03:11.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:11 smithi115 ceph-mon[72476]: osdmap e219: 12 total, 12 up, 12 in 2024-03-20T09:03:11.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:11 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/274416010' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:03:11.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:11 smithi115 ceph-mon[72476]: pgmap v617: 27 pgs: 1 peering, 4 active+recovering+undersized+remapped, 1 active+undersized+remapped, 21 active+clean; 985 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s wr, 1.59k op/s; 1049/47268 objects misplaced (2.219%); 39 MiB/s, 616 objects/s recovering 2024-03-20T09:03:11.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:11 smithi026 ceph-mon[68309]: osdmap e219: 12 total, 12 up, 12 in 2024-03-20T09:03:11.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:11 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/274416010' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:03:11.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:11 smithi026 ceph-mon[68309]: pgmap v617: 27 pgs: 1 peering, 4 active+recovering+undersized+remapped, 1 active+undersized+remapped, 21 active+clean; 985 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 100 MiB/s wr, 1.59k op/s; 1049/47268 objects misplaced (2.219%); 39 MiB/s, 616 objects/s recovering 2024-03-20T09:03:12.114 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:03:12.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:12.125 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:03:12.226 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:03:12.271 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:12.381 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:03:12.527 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:03:12.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:12.532 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:03:12.604 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 11 16 21253 21237 120.629 187.875 0.00373691 0.00828351 2024-03-20T09:03:12.632 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:03:12.673 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:12.692 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:03:12.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:12.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:03:12.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:12 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:03:12.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:12 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/274416010' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:03:12.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:12 smithi026 ceph-mon[68309]: osdmap e220: 12 total, 12 up, 12 in 2024-03-20T09:03:12.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:12 smithi026 ceph-mon[68309]: osdmap e221: 12 total, 12 up, 12 in 2024-03-20T09:03:12.746 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:03:12.796 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:03:12.805 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:12 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:03:12.805 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:12 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/274416010' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:03:12.806 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:12 smithi122 ceph-mon[73495]: osdmap e220: 12 total, 12 up, 12 in 2024-03-20T09:03:12.806 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:12 smithi122 ceph-mon[73495]: osdmap e221: 12 total, 12 up, 12 in 2024-03-20T09:03:12.822 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:03:12.822 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:03:12.824 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:12.842 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:03:12.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:12.845 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:03:12.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:12 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:03:12.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:12 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/274416010' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:03:12.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:12 smithi115 ceph-mon[72476]: osdmap e220: 12 total, 12 up, 12 in 2024-03-20T09:03:12.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:12 smithi115 ceph-mon[72476]: osdmap e221: 12 total, 12 up, 12 in 2024-03-20T09:03:12.946 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:03:13.001 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:13.031 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:03:13.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:13.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:03:13.136 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:03:13.178 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:13.545 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_historic_ops 2024-03-20T09:03:13.605 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 12 15 23688 23673 123.263 152.25 0.00324187 0.00810737 2024-03-20T09:03:13.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:13 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:03:13.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:13 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:03:13.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:13 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:03:13.911 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:03:13.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:13 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:03:13.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:13 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:03:13.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:13 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:03:13.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:13 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:03:13.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:13 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:03:13.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:13 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:03:14.037 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:03:14.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:14.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:03:14.141 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:03:14.180 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:14.290 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:03:14.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:14.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:03:14.395 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:03:14.435 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:14.465 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:03:14.557 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-03-20T09:03:14.557 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:03:14.557 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_1 2024-03-20T09:03:14.557 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:03:14.605 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 13 16 25959 25943 124.693 141.875 0.00237097 0.00801511 2024-03-20T09:03:14.660 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:03:14.689 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:03:14.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:14.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:03:14.700 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:03:14.729 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:03:14.769 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:03:14.799 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:03:14.800 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:03:14.821 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:03:14.824 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:14.843 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:03:14.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:14.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:03:14.847 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:03:14.866 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:03:14.869 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:03:14.893 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:03:14.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:14 smithi115 ceph-mon[72476]: 1.a scrub starts 2024-03-20T09:03:14.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:14 smithi115 ceph-mon[72476]: 1.a scrub ok 2024-03-20T09:03:14.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:14 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:03:14.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:14 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:03:14.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:14 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:03:14.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:14 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:03:14.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:14 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/254634416' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:03:14.922 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:03:14.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:14 smithi122 ceph-mon[73495]: 1.a scrub starts 2024-03-20T09:03:14.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:14 smithi122 ceph-mon[73495]: 1.a scrub ok 2024-03-20T09:03:14.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:14 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:03:14.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:14 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:03:14.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:14 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:03:14.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:14 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:03:14.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:14 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/254634416' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:03:14.946 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:03:14.951 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:03:14.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:14 smithi026 ceph-mon[68309]: 1.a scrub starts 2024-03-20T09:03:14.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:14 smithi026 ceph-mon[68309]: 1.a scrub ok 2024-03-20T09:03:14.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:14 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:03:14.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:14 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:03:14.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:14 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:03:14.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:14 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:03:14.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:14 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/254634416' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:03:14.978 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:03:14.981 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:03:14.981 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:03:14.982 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:03:14.982 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:03:14.982 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:03:14.982 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:03:14.982 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:03:14.982 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:03:14.982 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:03:14.982 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:03:14.982 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:03:14.982 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:03:14.998 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:15.030 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:03:15.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:15.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:03:15.136 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:03:15.178 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:15.205 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:03:15.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:15.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:03:15.311 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:03:15.352 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:15.605 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 14 15 28142 28127 125.535 136.5 0.00501836 0.00796035 2024-03-20T09:03:15.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:15 smithi115 ceph-mon[72476]: pgmap v621: 27 pgs: 27 active+clean; 1.5 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 120 MiB/s wr, 1.91k op/s; 28 MiB/s, 449 objects/s recovering 2024-03-20T09:03:15.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:15 smithi122 ceph-mon[73495]: pgmap v621: 27 pgs: 27 active+clean; 1.5 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 120 MiB/s wr, 1.91k op/s; 28 MiB/s, 449 objects/s recovering 2024-03-20T09:03:15.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:15 smithi026 ceph-mon[68309]: pgmap v621: 27 pgs: 27 active+clean; 1.5 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 120 MiB/s wr, 1.91k op/s; 28 MiB/s, 449 objects/s recovering 2024-03-20T09:03:16.095 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:03:16.095 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-03-20T09:03:16.095 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-03-20T09:03:16.095 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-03-20T09:03:16.095 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.095 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:30172 2.f 2:f74da544:::benchmark_data_smithi122_107654_object30171:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-20T09:03:16.095 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:58:34.522939+0000", 2024-03-20T09:03:16.095 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 281.56803751500001, 2024-03-20T09:03:16.095 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20426802299999999, 2024-03-20T09:03:16.096 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:16.096 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:16.096 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:16.096 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:03:16.096 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:03:16.096 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 30172 2024-03-20T09:03:16.096 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.096 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:16.096 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.096 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:16.096 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.522939+0000", 2024-03-20T09:03:16.096 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.097 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.097 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.097 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:16.097 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.522939+0000", 2024-03-20T09:03:16.097 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.097 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.097 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.097 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:16.097 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.522939+0000", 2024-03-20T09:03:16.097 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.51e-07 2024-03-20T09:03:16.097 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.097 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.097 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:16.097 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.522962+0000", 2024-03-20T09:03:16.097 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2342000000000001e-05 2024-03-20T09:03:16.098 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.098 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.098 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:16.098 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.522970+0000", 2024-03-20T09:03:16.098 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6270000000000006e-06 2024-03-20T09:03:16.098 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.098 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.098 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:16.098 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.522977+0000", 2024-03-20T09:03:16.098 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9199999999999998e-06 2024-03-20T09:03:16.098 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.098 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.098 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:16.098 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.523080+0000", 2024-03-20T09:03:16.099 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000102877 2024-03-20T09:03:16.099 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.099 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.099 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:16.099 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.523228+0000", 2024-03-20T09:03:16.099 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014827199999999999 2024-03-20T09:03:16.099 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.099 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.099 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 4,9", 2024-03-20T09:03:16.099 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.523384+0000", 2024-03-20T09:03:16.099 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015502199999999999 2024-03-20T09:03:16.099 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.099 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.099 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.099 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.524537+0000", 2024-03-20T09:03:16.100 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001153196 2024-03-20T09:03:16.100 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.100 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.100 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:16.100 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.524942+0000", 2024-03-20T09:03:16.100 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00040574399999999998 2024-03-20T09:03:16.100 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.100 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.100 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.100 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.727133+0000", 2024-03-20T09:03:16.100 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20219045499999999 2024-03-20T09:03:16.100 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.100 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.100 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:16.101 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.727175+0000", 2024-03-20T09:03:16.101 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1795999999999998e-05 2024-03-20T09:03:16.101 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.101 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.101 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:16.101 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.727207+0000", 2024-03-20T09:03:16.101 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2221000000000002e-05 2024-03-20T09:03:16.101 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.101 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:16.101 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.101 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.101 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.101 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:30200 2.f 2:f0fc9aeb:::benchmark_data_smithi122_107654_object30199:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-20T09:03:16.101 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:58:34.537363+0000", 2024-03-20T09:03:16.102 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 281.55361298600002, 2024-03-20T09:03:16.102 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19084448800000001, 2024-03-20T09:03:16.102 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:16.102 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:16.102 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:16.102 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:03:16.102 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:03:16.102 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 30200 2024-03-20T09:03:16.102 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.102 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:16.102 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.102 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:16.102 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.537363+0000", 2024-03-20T09:03:16.103 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.103 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.103 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.103 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:16.103 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.537363+0000", 2024-03-20T09:03:16.103 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.103 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.103 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.103 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:16.103 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.537365+0000", 2024-03-20T09:03:16.103 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3599999999999999e-06 2024-03-20T09:03:16.103 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.103 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.103 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:16.104 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.537388+0000", 2024-03-20T09:03:16.104 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3697999999999999e-05 2024-03-20T09:03:16.104 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.104 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.104 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:16.104 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.537403+0000", 2024-03-20T09:03:16.104 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4127999999999999e-05 2024-03-20T09:03:16.104 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.104 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.104 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:16.105 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.537411+0000", 2024-03-20T09:03:16.105 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9130000000000003e-06 2024-03-20T09:03:16.105 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.105 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.105 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:16.105 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.537452+0000", 2024-03-20T09:03:16.105 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1961e-05 2024-03-20T09:03:16.105 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.105 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.105 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:16.105 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.537575+0000", 2024-03-20T09:03:16.105 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000122994 2024-03-20T09:03:16.105 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.106 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.106 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 4,9", 2024-03-20T09:03:16.106 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.537678+0000", 2024-03-20T09:03:16.106 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000102867 2024-03-20T09:03:16.106 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.106 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.106 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.106 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.539934+0000", 2024-03-20T09:03:16.106 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0022559329999999999 2024-03-20T09:03:16.106 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.106 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.106 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:16.106 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.540101+0000", 2024-03-20T09:03:16.107 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016721100000000001 2024-03-20T09:03:16.107 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.107 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.107 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.107 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.728114+0000", 2024-03-20T09:03:16.107 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.188012553 2024-03-20T09:03:16.107 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.107 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.107 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:16.107 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.728168+0000", 2024-03-20T09:03:16.107 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3684000000000003e-05 2024-03-20T09:03:16.107 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.107 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.108 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:16.108 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.728208+0000", 2024-03-20T09:03:16.108 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0185999999999998e-05 2024-03-20T09:03:16.108 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.108 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:16.108 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.108 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.108 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.108 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:30202 2.f 2:f47c1051:::benchmark_data_smithi122_107654_object30201:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-20T09:03:16.108 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:58:34.537517+0000", 2024-03-20T09:03:16.108 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 281.55345936700002, 2024-03-20T09:03:16.108 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19085913400000001, 2024-03-20T09:03:16.108 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:16.108 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:16.108 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:16.109 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:03:16.109 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:03:16.109 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 30202 2024-03-20T09:03:16.109 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.109 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:16.109 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.109 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:16.109 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.537517+0000", 2024-03-20T09:03:16.109 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.109 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.109 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.109 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:16.109 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.537517+0000", 2024-03-20T09:03:16.110 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.110 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.110 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.110 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:16.110 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.537518+0000", 2024-03-20T09:03:16.110 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4300000000000002e-07 2024-03-20T09:03:16.110 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.110 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.110 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:16.110 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.537539+0000", 2024-03-20T09:03:16.110 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.145e-05 2024-03-20T09:03:16.110 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.110 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.111 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:16.111 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.537547+0000", 2024-03-20T09:03:16.111 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1850000000000004e-06 2024-03-20T09:03:16.111 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.111 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.111 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:16.111 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.537554+0000", 2024-03-20T09:03:16.111 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7710000000000001e-06 2024-03-20T09:03:16.111 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.111 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.111 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:16.111 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.538652+0000", 2024-03-20T09:03:16.111 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010981910000000001 2024-03-20T09:03:16.111 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.112 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.112 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:16.112 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.538769+0000", 2024-03-20T09:03:16.112 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000116128 2024-03-20T09:03:16.112 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.112 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.112 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 4,9", 2024-03-20T09:03:16.112 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.538893+0000", 2024-03-20T09:03:16.112 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012465100000000001 2024-03-20T09:03:16.112 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.112 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.112 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.112 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.540043+0000", 2024-03-20T09:03:16.112 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0011499979999999999 2024-03-20T09:03:16.112 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.113 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.113 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:16.113 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.540107+0000", 2024-03-20T09:03:16.113 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3874000000000005e-05 2024-03-20T09:03:16.113 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.113 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.113 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.113 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.728258+0000", 2024-03-20T09:03:16.113 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18815111700000001 2024-03-20T09:03:16.113 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.113 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.113 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:16.113 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.728331+0000", 2024-03-20T09:03:16.114 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2816000000000002e-05 2024-03-20T09:03:16.114 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.114 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.114 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:16.114 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.728376+0000", 2024-03-20T09:03:16.114 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5210000000000003e-05 2024-03-20T09:03:16.114 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.114 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:16.114 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.114 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.114 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.114 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:30239 2.f 2:fb5d4aea:::benchmark_data_smithi122_107654_object30238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-20T09:03:16.115 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:58:34.553378+0000", 2024-03-20T09:03:16.115 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 281.53759808500001, 2024-03-20T09:03:16.115 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.17587739099999999, 2024-03-20T09:03:16.115 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:16.115 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:16.115 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:16.115 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:03:16.115 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:03:16.115 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 30239 2024-03-20T09:03:16.115 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.115 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:16.115 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.115 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:16.115 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.553378+0000", 2024-03-20T09:03:16.116 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.116 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.116 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.116 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:16.116 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.553378+0000", 2024-03-20T09:03:16.116 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.116 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.116 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.116 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:16.116 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.553380+0000", 2024-03-20T09:03:16.116 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.17e-06 2024-03-20T09:03:16.116 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.116 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.117 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:16.117 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.553399+0000", 2024-03-20T09:03:16.117 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9119000000000001e-05 2024-03-20T09:03:16.117 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.117 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.117 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:16.117 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.553409+0000", 2024-03-20T09:03:16.117 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0267e-05 2024-03-20T09:03:16.117 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.118 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.118 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:16.118 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.553415+0000", 2024-03-20T09:03:16.118 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2650000000000002e-06 2024-03-20T09:03:16.118 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.118 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.118 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:16.118 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.553459+0000", 2024-03-20T09:03:16.118 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4267000000000002e-05 2024-03-20T09:03:16.118 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.118 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.118 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:16.118 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.553566+0000", 2024-03-20T09:03:16.118 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000106897 2024-03-20T09:03:16.119 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.119 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.119 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 4,9", 2024-03-20T09:03:16.119 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.553714+0000", 2024-03-20T09:03:16.119 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014721600000000001 2024-03-20T09:03:16.119 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.119 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.119 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.119 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.556067+0000", 2024-03-20T09:03:16.119 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002353099 2024-03-20T09:03:16.119 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.119 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.119 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:16.119 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.557007+0000", 2024-03-20T09:03:16.120 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00094056299999999995 2024-03-20T09:03:16.120 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.120 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.120 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.120 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.729165+0000", 2024-03-20T09:03:16.120 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.17215812599999999 2024-03-20T09:03:16.120 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.120 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.120 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:16.120 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.729215+0000", 2024-03-20T09:03:16.120 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9298000000000001e-05 2024-03-20T09:03:16.120 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.120 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.120 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:16.121 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:34.729256+0000", 2024-03-20T09:03:16.121 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1103999999999998e-05 2024-03-20T09:03:16.121 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.121 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:16.121 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.121 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.121 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.121 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:45936 2.b 2:d4186d16:::benchmark_data_smithi122_107654_object45935:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-20T09:03:16.121 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:58:41.726346+0000", 2024-03-20T09:03:16.121 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 274.36463060800003, 2024-03-20T09:03:16.121 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20713243200000001, 2024-03-20T09:03:16.121 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:16.121 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:16.121 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:16.121 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:03:16.122 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:03:16.122 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 45936 2024-03-20T09:03:16.122 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.122 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:16.122 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.122 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:16.122 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.726346+0000", 2024-03-20T09:03:16.122 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.122 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.122 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.122 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:16.122 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.726346+0000", 2024-03-20T09:03:16.122 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.122 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.123 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.123 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:16.123 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.726347+0000", 2024-03-20T09:03:16.123 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.122e-06 2024-03-20T09:03:16.123 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.123 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.123 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:16.123 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.726369+0000", 2024-03-20T09:03:16.123 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1987000000000001e-05 2024-03-20T09:03:16.123 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.123 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.123 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:16.123 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.726384+0000", 2024-03-20T09:03:16.123 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5325e-05 2024-03-20T09:03:16.124 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.124 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.124 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:16.124 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.726393+0000", 2024-03-20T09:03:16.124 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2719999999999997e-06 2024-03-20T09:03:16.124 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.124 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.124 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:16.124 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.727102+0000", 2024-03-20T09:03:16.124 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000709214 2024-03-20T09:03:16.124 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.124 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.124 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:16.125 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.727209+0000", 2024-03-20T09:03:16.125 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010774 2024-03-20T09:03:16.125 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.125 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.125 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,11", 2024-03-20T09:03:16.125 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.727320+0000", 2024-03-20T09:03:16.125 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000110469 2024-03-20T09:03:16.125 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.125 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.125 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:16.125 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.728754+0000", 2024-03-20T09:03:16.125 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014344469999999999 2024-03-20T09:03:16.125 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.125 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.126 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.126 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.730238+0000", 2024-03-20T09:03:16.126 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001483832 2024-03-20T09:03:16.126 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.126 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.126 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.126 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.933412+0000", 2024-03-20T09:03:16.126 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20317375600000001 2024-03-20T09:03:16.126 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.126 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.126 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:16.126 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.933447+0000", 2024-03-20T09:03:16.126 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4999999999999997e-05 2024-03-20T09:03:16.126 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.127 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.127 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:16.127 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.933478+0000", 2024-03-20T09:03:16.127 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1267999999999999e-05 2024-03-20T09:03:16.127 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.127 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:16.127 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.127 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.127 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.127 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:45963 2.b 2:d2808d5e:::benchmark_data_smithi122_107654_object45962:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-20T09:03:16.127 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:58:41.736859+0000", 2024-03-20T09:03:16.127 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 274.354117325, 2024-03-20T09:03:16.127 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19803585600000001, 2024-03-20T09:03:16.128 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:16.128 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:16.128 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:16.128 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:03:16.128 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:03:16.128 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 45963 2024-03-20T09:03:16.128 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.128 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:16.128 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.128 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:16.128 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.736859+0000", 2024-03-20T09:03:16.128 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.128 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.128 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.128 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:16.129 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.736859+0000", 2024-03-20T09:03:16.129 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.129 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.129 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.129 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:16.129 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.736860+0000", 2024-03-20T09:03:16.129 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5099999999999999e-07 2024-03-20T09:03:16.129 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.129 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.129 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:16.129 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.736878+0000", 2024-03-20T09:03:16.129 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8158000000000001e-05 2024-03-20T09:03:16.129 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.129 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.130 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:16.130 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.736888+0000", 2024-03-20T09:03:16.130 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0297e-05 2024-03-20T09:03:16.130 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.130 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.130 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:16.130 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.736896+0000", 2024-03-20T09:03:16.130 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3329999999999999e-06 2024-03-20T09:03:16.130 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.130 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.130 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:16.130 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.736989+0000", 2024-03-20T09:03:16.130 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3826999999999995e-05 2024-03-20T09:03:16.131 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.131 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.131 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:16.131 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.737115+0000", 2024-03-20T09:03:16.131 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012570100000000001 2024-03-20T09:03:16.131 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.131 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.131 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,11", 2024-03-20T09:03:16.131 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.737265+0000", 2024-03-20T09:03:16.131 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000150216 2024-03-20T09:03:16.131 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.131 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.131 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:16.131 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.739254+0000", 2024-03-20T09:03:16.132 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001988989 2024-03-20T09:03:16.132 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.132 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.132 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.132 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.740505+0000", 2024-03-20T09:03:16.132 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001250464 2024-03-20T09:03:16.132 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.132 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.132 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.132 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.934821+0000", 2024-03-20T09:03:16.132 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19431633200000001 2024-03-20T09:03:16.132 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.132 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.132 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:16.132 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.934857+0000", 2024-03-20T09:03:16.133 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5648000000000002e-05 2024-03-20T09:03:16.133 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.133 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.133 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:16.133 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.934895+0000", 2024-03-20T09:03:16.133 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.824e-05 2024-03-20T09:03:16.133 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.133 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:16.133 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.133 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.133 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.133 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:45969 2.b 2:d875b49b:::benchmark_data_smithi122_107654_object45968:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-20T09:03:16.133 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:58:41.740680+0000", 2024-03-20T09:03:16.134 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 274.35029625300001, 2024-03-20T09:03:16.134 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.194373935, 2024-03-20T09:03:16.134 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:16.134 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:16.134 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:16.134 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:03:16.134 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:03:16.134 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 45969 2024-03-20T09:03:16.134 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.134 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:16.134 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.134 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:16.134 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.740680+0000", 2024-03-20T09:03:16.134 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.135 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.135 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.135 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:16.135 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.740680+0000", 2024-03-20T09:03:16.135 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.135 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.135 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.135 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:16.135 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.740681+0000", 2024-03-20T09:03:16.135 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0999999999999998e-07 2024-03-20T09:03:16.136 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.136 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.136 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:16.136 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.740703+0000", 2024-03-20T09:03:16.136 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2073999999999999e-05 2024-03-20T09:03:16.136 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.136 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.136 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:16.136 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.740713+0000", 2024-03-20T09:03:16.136 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5829999999999996e-06 2024-03-20T09:03:16.136 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.136 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.136 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:16.137 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.740718+0000", 2024-03-20T09:03:16.137 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4219999999999999e-06 2024-03-20T09:03:16.137 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.137 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.137 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:16.137 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.740751+0000", 2024-03-20T09:03:16.137 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3002e-05 2024-03-20T09:03:16.137 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.137 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.137 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:16.137 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.740835+0000", 2024-03-20T09:03:16.137 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3548999999999996e-05 2024-03-20T09:03:16.137 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.137 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.138 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,11", 2024-03-20T09:03:16.138 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.740952+0000", 2024-03-20T09:03:16.138 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000117625 2024-03-20T09:03:16.138 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.138 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.138 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:16.138 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.743264+0000", 2024-03-20T09:03:16.138 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0023117189999999998 2024-03-20T09:03:16.138 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.138 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.138 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.138 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.745345+0000", 2024-03-20T09:03:16.138 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002081645 2024-03-20T09:03:16.139 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.139 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.139 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.139 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.934950+0000", 2024-03-20T09:03:16.139 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.189604936 2024-03-20T09:03:16.139 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.139 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.139 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:16.139 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.935005+0000", 2024-03-20T09:03:16.139 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4440000000000001e-05 2024-03-20T09:03:16.139 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.139 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.139 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:16.139 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.935054+0000", 2024-03-20T09:03:16.140 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9230000000000001e-05 2024-03-20T09:03:16.140 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.140 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:16.140 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.140 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.140 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.140 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:45972 2.b 2:d8091d0c:::benchmark_data_smithi122_107654_object45971:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-20T09:03:16.140 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:58:41.741861+0000", 2024-03-20T09:03:16.140 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 274.34911579099997, 2024-03-20T09:03:16.140 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.193411216, 2024-03-20T09:03:16.140 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:16.140 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:16.140 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:16.140 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:03:16.141 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:03:16.141 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 45972 2024-03-20T09:03:16.141 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.141 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:16.141 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.141 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:16.141 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.741861+0000", 2024-03-20T09:03:16.141 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.141 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.141 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.141 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:16.141 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.741861+0000", 2024-03-20T09:03:16.141 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.141 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.142 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.142 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:16.142 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.741861+0000", 2024-03-20T09:03:16.142 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4300000000000002e-07 2024-03-20T09:03:16.142 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.142 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.142 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:16.142 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.741880+0000", 2024-03-20T09:03:16.142 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8272000000000001e-05 2024-03-20T09:03:16.142 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.142 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.142 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:16.142 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.741890+0000", 2024-03-20T09:03:16.142 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0681e-05 2024-03-20T09:03:16.143 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.143 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.143 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:16.143 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.741898+0000", 2024-03-20T09:03:16.143 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6939999999999995e-06 2024-03-20T09:03:16.143 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.143 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.143 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:16.143 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.742119+0000", 2024-03-20T09:03:16.143 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000220934 2024-03-20T09:03:16.143 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.143 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.143 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:16.143 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.742229+0000", 2024-03-20T09:03:16.144 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011039699999999999 2024-03-20T09:03:16.144 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.144 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.144 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,11", 2024-03-20T09:03:16.144 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.742353+0000", 2024-03-20T09:03:16.144 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000123919 2024-03-20T09:03:16.144 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.144 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.144 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:16.144 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.745098+0000", 2024-03-20T09:03:16.144 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002744972 2024-03-20T09:03:16.144 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.144 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.144 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.145 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.745383+0000", 2024-03-20T09:03:16.145 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00028482300000000002 2024-03-20T09:03:16.145 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.145 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.145 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.145 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.935199+0000", 2024-03-20T09:03:16.145 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18981584100000001 2024-03-20T09:03:16.145 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.145 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.145 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:16.145 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.935242+0000", 2024-03-20T09:03:16.145 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2778999999999998e-05 2024-03-20T09:03:16.145 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.145 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.146 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:16.146 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.935272+0000", 2024-03-20T09:03:16.146 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0060999999999999e-05 2024-03-20T09:03:16.146 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.146 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:16.146 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.146 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.146 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.146 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:46011 2.b 2:d6099855:::benchmark_data_smithi122_107654_object46010:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-20T09:03:16.146 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:58:41.760400+0000", 2024-03-20T09:03:16.146 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 274.33057648499999, 2024-03-20T09:03:16.146 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.17582661799999999, 2024-03-20T09:03:16.146 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:16.147 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:16.147 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:16.147 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:03:16.147 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:03:16.147 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 46011 2024-03-20T09:03:16.147 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.147 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:16.147 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.147 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:16.147 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.760400+0000", 2024-03-20T09:03:16.147 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.147 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.147 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.147 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:16.147 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.760400+0000", 2024-03-20T09:03:16.148 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.148 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.148 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.148 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:16.148 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.760401+0000", 2024-03-20T09:03:16.148 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.373e-06 2024-03-20T09:03:16.148 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.148 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.148 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:16.148 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.760429+0000", 2024-03-20T09:03:16.148 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7317e-05 2024-03-20T09:03:16.148 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.148 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.148 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:16.149 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.760442+0000", 2024-03-20T09:03:16.149 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2901999999999999e-05 2024-03-20T09:03:16.149 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.149 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.149 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:16.149 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.760448+0000", 2024-03-20T09:03:16.149 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.613e-06 2024-03-20T09:03:16.149 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.149 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.149 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:16.149 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.760499+0000", 2024-03-20T09:03:16.149 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1325000000000001e-05 2024-03-20T09:03:16.149 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.149 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.150 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:16.150 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.760593+0000", 2024-03-20T09:03:16.150 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3119000000000001e-05 2024-03-20T09:03:16.150 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.150 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.150 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,11", 2024-03-20T09:03:16.150 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.760678+0000", 2024-03-20T09:03:16.150 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5385999999999999e-05 2024-03-20T09:03:16.150 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.150 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.150 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:16.150 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.762321+0000", 2024-03-20T09:03:16.150 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0016431219999999999 2024-03-20T09:03:16.151 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.151 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.151 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.151 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.762779+0000", 2024-03-20T09:03:16.151 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00045834199999999998 2024-03-20T09:03:16.151 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.151 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.151 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.151 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.936149+0000", 2024-03-20T09:03:16.151 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.17336975299999999 2024-03-20T09:03:16.151 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.151 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.151 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:16.151 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.936184+0000", 2024-03-20T09:03:16.152 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5027000000000002e-05 2024-03-20T09:03:16.152 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.152 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.152 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:16.152 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:41.936227+0000", 2024-03-20T09:03:16.152 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2339000000000002e-05 2024-03-20T09:03:16.152 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.152 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:16.152 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.152 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.152 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.152 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:116068 2.b 2:d4c3e9d8:::benchmark_data_smithi122_107654_object116067:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-03-20T09:03:16.152 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:59:50.846349+0000", 2024-03-20T09:03:16.152 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 205.24462775200001, 2024-03-20T09:03:16.153 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20256579399999999, 2024-03-20T09:03:16.153 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:16.153 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:16.153 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:16.153 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:03:16.153 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:03:16.153 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 116068 2024-03-20T09:03:16.153 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.153 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:16.153 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.153 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:16.153 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.846349+0000", 2024-03-20T09:03:16.153 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.153 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.154 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.154 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:16.154 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.846349+0000", 2024-03-20T09:03:16.154 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.154 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.154 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.154 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:16.154 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.846349+0000", 2024-03-20T09:03:16.154 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2800000000000002e-07 2024-03-20T09:03:16.154 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.154 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.154 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:16.154 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.846369+0000", 2024-03-20T09:03:16.154 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0156000000000001e-05 2024-03-20T09:03:16.155 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.155 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.155 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:16.155 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.846375+0000", 2024-03-20T09:03:16.155 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2430000000000002e-06 2024-03-20T09:03:16.155 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.155 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.155 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:16.155 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.846381+0000", 2024-03-20T09:03:16.155 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1900000000000003e-06 2024-03-20T09:03:16.155 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.155 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.155 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:16.155 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.846436+0000", 2024-03-20T09:03:16.155 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5296e-05 2024-03-20T09:03:16.156 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.156 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.156 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:16.156 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.846559+0000", 2024-03-20T09:03:16.156 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012265299999999999 2024-03-20T09:03:16.156 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.156 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.156 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,11", 2024-03-20T09:03:16.156 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.846698+0000", 2024-03-20T09:03:16.156 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013973600000000001 2024-03-20T09:03:16.156 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.156 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.156 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:16.157 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.852495+0000", 2024-03-20T09:03:16.157 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0057968539999999997 2024-03-20T09:03:16.157 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.157 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.157 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.157 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.853877+0000", 2024-03-20T09:03:16.157 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0013814599999999999 2024-03-20T09:03:16.157 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.157 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.157 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.157 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.048826+0000", 2024-03-20T09:03:16.157 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19494949 2024-03-20T09:03:16.157 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.157 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.158 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:16.158 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.048867+0000", 2024-03-20T09:03:16.158 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.091e-05 2024-03-20T09:03:16.158 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.158 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.158 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:16.158 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.048914+0000", 2024-03-20T09:03:16.158 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7377999999999999e-05 2024-03-20T09:03:16.158 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.158 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:16.158 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.158 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.158 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.158 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:116083 2.b 2:dcf6575c:::benchmark_data_smithi122_107654_object116082:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-03-20T09:03:16.159 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:59:50.860693+0000", 2024-03-20T09:03:16.159 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 205.23028379799999, 2024-03-20T09:03:16.159 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19728431599999999, 2024-03-20T09:03:16.159 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:16.159 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:16.159 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:16.159 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:03:16.159 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:03:16.159 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 116083 2024-03-20T09:03:16.159 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.159 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:16.159 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.159 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:16.159 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.860693+0000", 2024-03-20T09:03:16.160 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.160 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.160 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.160 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:16.160 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.860693+0000", 2024-03-20T09:03:16.160 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.160 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.160 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.160 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:16.160 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.860693+0000", 2024-03-20T09:03:16.160 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4199999999999995e-07 2024-03-20T09:03:16.160 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.160 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.160 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:16.161 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.860711+0000", 2024-03-20T09:03:16.161 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7963999999999999e-05 2024-03-20T09:03:16.161 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.161 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.161 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:16.161 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.860721+0000", 2024-03-20T09:03:16.161 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4350000000000003e-06 2024-03-20T09:03:16.161 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.161 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.161 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:16.161 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.860727+0000", 2024-03-20T09:03:16.161 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7780000000000004e-06 2024-03-20T09:03:16.161 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.161 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.161 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:16.162 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.860771+0000", 2024-03-20T09:03:16.162 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4039000000000003e-05 2024-03-20T09:03:16.162 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.162 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.162 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:16.162 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.860851+0000", 2024-03-20T09:03:16.162 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0817999999999998e-05 2024-03-20T09:03:16.162 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.162 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.162 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,11", 2024-03-20T09:03:16.162 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.860977+0000", 2024-03-20T09:03:16.162 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012594700000000001 2024-03-20T09:03:16.162 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.162 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.163 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:16.163 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.863719+0000", 2024-03-20T09:03:16.163 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027417460000000002 2024-03-20T09:03:16.163 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.163 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.163 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.163 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.866474+0000", 2024-03-20T09:03:16.163 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027546929999999999 2024-03-20T09:03:16.163 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.163 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.163 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.163 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.057869+0000", 2024-03-20T09:03:16.163 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19139492699999999 2024-03-20T09:03:16.163 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.164 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.164 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:16.164 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.057931+0000", 2024-03-20T09:03:16.164 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2445000000000003e-05 2024-03-20T09:03:16.164 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.164 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.164 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:16.164 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.057977+0000", 2024-03-20T09:03:16.164 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5782000000000003e-05 2024-03-20T09:03:16.164 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.164 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:16.164 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.164 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.165 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.165 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:116090 2.b 2:d8f8696b:::benchmark_data_smithi122_107654_object116089:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-03-20T09:03:16.165 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:59:50.868553+0000", 2024-03-20T09:03:16.165 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 205.22242390900001, 2024-03-20T09:03:16.165 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19503764200000001, 2024-03-20T09:03:16.165 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:16.165 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:16.165 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:16.165 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:03:16.165 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:03:16.165 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 116090 2024-03-20T09:03:16.165 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.165 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:16.165 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.166 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:16.166 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.868553+0000", 2024-03-20T09:03:16.166 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.166 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.166 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.166 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:16.166 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.868553+0000", 2024-03-20T09:03:16.166 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.166 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.166 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.166 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:16.166 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.868554+0000", 2024-03-20T09:03:16.166 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.517e-06 2024-03-20T09:03:16.167 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.167 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.167 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:16.167 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.868573+0000", 2024-03-20T09:03:16.167 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9397000000000001e-05 2024-03-20T09:03:16.167 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.167 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.167 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:16.167 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.868585+0000", 2024-03-20T09:03:16.167 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2016e-05 2024-03-20T09:03:16.167 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.167 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.167 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:16.167 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.868593+0000", 2024-03-20T09:03:16.168 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7340000000000008e-06 2024-03-20T09:03:16.168 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.168 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.168 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:16.168 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.868642+0000", 2024-03-20T09:03:16.168 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9298999999999997e-05 2024-03-20T09:03:16.168 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.168 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.168 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:16.168 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.868748+0000", 2024-03-20T09:03:16.168 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105164 2024-03-20T09:03:16.168 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.168 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.168 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,11", 2024-03-20T09:03:16.169 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.868886+0000", 2024-03-20T09:03:16.169 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000138696 2024-03-20T09:03:16.169 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.169 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.169 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:16.169 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.871701+0000", 2024-03-20T09:03:16.169 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0028147290000000002 2024-03-20T09:03:16.169 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.169 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.169 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.169 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.871796+0000", 2024-03-20T09:03:16.169 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4596e-05 2024-03-20T09:03:16.169 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.169 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.170 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.170 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.063503+0000", 2024-03-20T09:03:16.170 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19170742599999999 2024-03-20T09:03:16.170 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.170 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.170 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:16.170 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.063549+0000", 2024-03-20T09:03:16.170 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6158000000000001e-05 2024-03-20T09:03:16.170 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.170 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.170 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:16.170 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.063590+0000", 2024-03-20T09:03:16.170 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.091e-05 2024-03-20T09:03:16.171 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.171 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:16.171 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.171 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.171 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.171 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:116097 2.b 2:dcc82db7:::benchmark_data_smithi122_107654_object116096:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-03-20T09:03:16.171 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:59:50.881259+0000", 2024-03-20T09:03:16.171 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 205.20971784100001, 2024-03-20T09:03:16.171 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.182490387, 2024-03-20T09:03:16.171 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:16.171 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:16.171 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:16.171 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:03:16.171 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:03:16.171 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 116097 2024-03-20T09:03:16.172 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.172 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:16.172 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.172 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:16.172 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.881259+0000", 2024-03-20T09:03:16.172 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.172 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.172 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.172 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:16.172 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.881259+0000", 2024-03-20T09:03:16.172 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.172 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.172 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.172 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:16.173 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.881260+0000", 2024-03-20T09:03:16.173 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4249999999999999e-06 2024-03-20T09:03:16.173 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.173 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.173 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:16.173 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.881291+0000", 2024-03-20T09:03:16.173 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1489999999999998e-05 2024-03-20T09:03:16.173 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.173 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.173 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:16.173 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.881306+0000", 2024-03-20T09:03:16.173 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4092e-05 2024-03-20T09:03:16.173 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.173 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.174 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:16.174 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.881315+0000", 2024-03-20T09:03:16.174 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5999999999999996e-06 2024-03-20T09:03:16.174 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.174 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.174 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:16.174 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.881379+0000", 2024-03-20T09:03:16.174 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3346e-05 2024-03-20T09:03:16.174 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.174 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.174 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:16.174 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.881524+0000", 2024-03-20T09:03:16.174 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000145759 2024-03-20T09:03:16.175 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.175 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.175 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,11", 2024-03-20T09:03:16.175 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.881682+0000", 2024-03-20T09:03:16.175 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000158126 2024-03-20T09:03:16.175 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.175 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.175 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:16.175 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.884410+0000", 2024-03-20T09:03:16.175 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027277870000000002 2024-03-20T09:03:16.175 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.175 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.175 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.175 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:50.885722+0000", 2024-03-20T09:03:16.176 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0013118400000000001 2024-03-20T09:03:16.176 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.176 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.176 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.176 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.063641+0000", 2024-03-20T09:03:16.176 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.17791922499999999 2024-03-20T09:03:16.176 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.176 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.176 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:16.176 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.063704+0000", 2024-03-20T09:03:16.176 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3015000000000006e-05 2024-03-20T09:03:16.176 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.176 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.176 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:16.177 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.063749+0000", 2024-03-20T09:03:16.177 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4681999999999997e-05 2024-03-20T09:03:16.177 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.177 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:16.177 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.177 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.177 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.177 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:185562 2.5 2:a002230b:::benchmark_data_smithi122_107654_object66777:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-20T09:03:16.177 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:01:32.624249+0000", 2024-03-20T09:03:16.177 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 103.46672727799999, 2024-03-20T09:03:16.177 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20806407800000001, 2024-03-20T09:03:16.177 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:16.177 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:16.177 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:16.178 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:03:16.178 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:03:16.178 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 185562 2024-03-20T09:03:16.178 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.178 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:16.178 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.178 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:16.178 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.624249+0000", 2024-03-20T09:03:16.178 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.178 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.178 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.178 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:16.178 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.624249+0000", 2024-03-20T09:03:16.179 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.179 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.179 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.179 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:16.179 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.624250+0000", 2024-03-20T09:03:16.179 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1930000000000001e-06 2024-03-20T09:03:16.179 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.179 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.179 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:16.179 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.624256+0000", 2024-03-20T09:03:16.179 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0929999999999999e-06 2024-03-20T09:03:16.179 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.179 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.179 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:16.180 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.624269+0000", 2024-03-20T09:03:16.180 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.26e-05 2024-03-20T09:03:16.180 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.180 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.180 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:16.180 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.624279+0000", 2024-03-20T09:03:16.180 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0249e-05 2024-03-20T09:03:16.180 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.180 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.180 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:16.180 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.624328+0000", 2024-03-20T09:03:16.180 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8386000000000001e-05 2024-03-20T09:03:16.180 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.180 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.181 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:16.181 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.624430+0000", 2024-03-20T09:03:16.181 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010203499999999999 2024-03-20T09:03:16.181 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.181 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.181 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,4", 2024-03-20T09:03:16.181 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.624528+0000", 2024-03-20T09:03:16.181 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.8733999999999994e-05 2024-03-20T09:03:16.181 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.181 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.181 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:16.181 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.634445+0000", 2024-03-20T09:03:16.181 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0099167649999999993 2024-03-20T09:03:16.181 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.182 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.182 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.182 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.639754+0000", 2024-03-20T09:03:16.182 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0053083039999999998 2024-03-20T09:03:16.182 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.182 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.182 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.182 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.832194+0000", 2024-03-20T09:03:16.182 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19244091899999999 2024-03-20T09:03:16.182 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.182 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.182 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:16.182 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.832255+0000", 2024-03-20T09:03:16.182 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1036000000000003e-05 2024-03-20T09:03:16.183 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.183 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.183 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:16.183 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.832313+0000", 2024-03-20T09:03:16.183 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7763999999999998e-05 2024-03-20T09:03:16.183 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.183 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:16.183 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.183 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.183 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.183 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:185568 2.5 2:a62ec9a1:::benchmark_data_smithi122_107654_object66783:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-20T09:03:16.183 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:01:32.624392+0000", 2024-03-20T09:03:16.183 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 103.466584497, 2024-03-20T09:03:16.183 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.22075340199999999, 2024-03-20T09:03:16.184 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:16.184 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:16.184 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:16.184 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:03:16.184 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:03:16.184 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 185568 2024-03-20T09:03:16.184 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.184 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:16.184 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.184 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:16.184 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.624392+0000", 2024-03-20T09:03:16.184 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.184 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.184 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.184 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:16.185 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.624392+0000", 2024-03-20T09:03:16.185 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.185 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.185 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.185 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:16.185 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.624393+0000", 2024-03-20T09:03:16.185 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3099999999999996e-07 2024-03-20T09:03:16.185 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.185 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.185 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:16.185 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.624398+0000", 2024-03-20T09:03:16.185 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.938e-06 2024-03-20T09:03:16.185 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.185 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.186 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:16.186 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.624407+0000", 2024-03-20T09:03:16.186 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5899999999999997e-06 2024-03-20T09:03:16.186 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.186 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.186 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:16.186 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.624414+0000", 2024-03-20T09:03:16.186 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0820000000000001e-06 2024-03-20T09:03:16.186 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.186 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.186 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:16.186 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.629815+0000", 2024-03-20T09:03:16.186 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0054001700000000001 2024-03-20T09:03:16.187 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.187 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.187 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:16.187 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.629938+0000", 2024-03-20T09:03:16.187 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012370299999999999 2024-03-20T09:03:16.187 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.187 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.187 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,4", 2024-03-20T09:03:16.187 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.630056+0000", 2024-03-20T09:03:16.187 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000117316 2024-03-20T09:03:16.187 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.187 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.187 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:16.188 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.634535+0000", 2024-03-20T09:03:16.188 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0044798099999999999 2024-03-20T09:03:16.188 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.188 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.188 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.188 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.639894+0000", 2024-03-20T09:03:16.188 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0053589329999999998 2024-03-20T09:03:16.188 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.188 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.188 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.188 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.845027+0000", 2024-03-20T09:03:16.188 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.205132488 2024-03-20T09:03:16.188 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.188 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.188 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:16.189 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.845100+0000", 2024-03-20T09:03:16.189 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2849000000000007e-05 2024-03-20T09:03:16.189 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.189 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.189 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:16.189 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.845145+0000", 2024-03-20T09:03:16.189 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5692000000000002e-05 2024-03-20T09:03:16.189 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.189 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:16.189 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.189 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.189 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.189 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:185581 2.f 2:f574bb59:::benchmark_data_smithi122_107654_object66796:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-20T09:03:16.190 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:01:32.639940+0000", 2024-03-20T09:03:16.190 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 103.45103689299999, 2024-03-20T09:03:16.190 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.201438917, 2024-03-20T09:03:16.190 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:16.190 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:16.190 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:16.190 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:03:16.190 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:03:16.190 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 185581 2024-03-20T09:03:16.190 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.190 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:16.190 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.190 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:16.190 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.639940+0000", 2024-03-20T09:03:16.191 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.191 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.191 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.191 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:16.191 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.639940+0000", 2024-03-20T09:03:16.191 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.191 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.191 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.191 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:16.191 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.639941+0000", 2024-03-20T09:03:16.191 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.372e-06 2024-03-20T09:03:16.191 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.191 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.191 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:16.192 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.639947+0000", 2024-03-20T09:03:16.192 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9780000000000002e-06 2024-03-20T09:03:16.192 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.192 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.192 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:16.192 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.639959+0000", 2024-03-20T09:03:16.192 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2362e-05 2024-03-20T09:03:16.192 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.192 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.192 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:16.192 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.639966+0000", 2024-03-20T09:03:16.192 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2049999999999996e-06 2024-03-20T09:03:16.192 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.192 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.193 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:16.193 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.640019+0000", 2024-03-20T09:03:16.193 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2485999999999999e-05 2024-03-20T09:03:16.193 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.193 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.193 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:16.193 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.640150+0000", 2024-03-20T09:03:16.193 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013122000000000001 2024-03-20T09:03:16.193 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.193 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.193 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 4,9", 2024-03-20T09:03:16.193 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.640267+0000", 2024-03-20T09:03:16.193 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000116663 2024-03-20T09:03:16.193 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.194 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.194 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:16.194 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.644329+0000", 2024-03-20T09:03:16.194 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040625749999999997 2024-03-20T09:03:16.194 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.194 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.194 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.194 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.647507+0000", 2024-03-20T09:03:16.194 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003177994 2024-03-20T09:03:16.194 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.194 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.194 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.194 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.841311+0000", 2024-03-20T09:03:16.194 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19380404900000001 2024-03-20T09:03:16.195 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.195 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.195 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:16.195 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.841351+0000", 2024-03-20T09:03:16.195 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9962999999999997e-05 2024-03-20T09:03:16.195 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.195 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.195 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:16.195 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.841378+0000", 2024-03-20T09:03:16.195 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.705e-05 2024-03-20T09:03:16.195 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.195 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:16.195 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.195 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.196 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.196 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:185588 2.5 2:a570ae3d:::benchmark_data_smithi122_107654_object66803:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-20T09:03:16.196 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:01:32.649315+0000", 2024-03-20T09:03:16.196 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 103.441661326, 2024-03-20T09:03:16.196 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.203669501, 2024-03-20T09:03:16.196 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:16.196 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:16.196 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:16.196 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:03:16.196 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:03:16.196 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 185588 2024-03-20T09:03:16.196 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.196 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:16.197 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.197 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:16.197 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.649315+0000", 2024-03-20T09:03:16.197 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.197 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.197 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.197 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:16.197 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.649315+0000", 2024-03-20T09:03:16.197 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.197 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.197 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.197 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:16.197 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.649317+0000", 2024-03-20T09:03:16.197 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.446e-06 2024-03-20T09:03:16.198 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.198 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.198 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:16.198 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.649324+0000", 2024-03-20T09:03:16.198 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1130000000000003e-06 2024-03-20T09:03:16.198 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.198 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.198 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:16.198 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.649339+0000", 2024-03-20T09:03:16.198 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5633000000000001e-05 2024-03-20T09:03:16.198 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.198 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.198 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:16.198 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.649348+0000", 2024-03-20T09:03:16.199 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9169999999999995e-06 2024-03-20T09:03:16.199 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.199 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.199 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:16.199 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.649405+0000", 2024-03-20T09:03:16.199 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.698e-05 2024-03-20T09:03:16.199 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.199 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.199 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:16.199 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.649542+0000", 2024-03-20T09:03:16.199 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000136935 2024-03-20T09:03:16.199 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.199 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.199 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,4", 2024-03-20T09:03:16.200 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.649672+0000", 2024-03-20T09:03:16.200 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013025600000000001 2024-03-20T09:03:16.200 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.200 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.200 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.200 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.657929+0000", 2024-03-20T09:03:16.200 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0082568629999999997 2024-03-20T09:03:16.200 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.200 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.200 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:16.200 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.657945+0000", 2024-03-20T09:03:16.200 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5486999999999999e-05 2024-03-20T09:03:16.200 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.200 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.201 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.201 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.852883+0000", 2024-03-20T09:03:16.201 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19493843199999999 2024-03-20T09:03:16.201 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.201 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.201 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:16.201 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.852936+0000", 2024-03-20T09:03:16.201 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3270999999999999e-05 2024-03-20T09:03:16.201 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.201 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.201 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:16.201 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.852985+0000", 2024-03-20T09:03:16.201 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8167999999999997e-05 2024-03-20T09:03:16.201 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.201 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:16.202 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.202 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.202 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.202 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:185591 2.5 2:aefd965d:::benchmark_data_smithi122_107654_object66806:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-20T09:03:16.202 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:01:32.658407+0000", 2024-03-20T09:03:16.202 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 103.432569633, 2024-03-20T09:03:16.202 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.194743105, 2024-03-20T09:03:16.202 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:16.202 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:16.202 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:16.202 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:03:16.202 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:03:16.202 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 185591 2024-03-20T09:03:16.203 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.203 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:16.203 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.203 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:16.203 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.658407+0000", 2024-03-20T09:03:16.203 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.203 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.203 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.203 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:16.203 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.658407+0000", 2024-03-20T09:03:16.203 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.203 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.203 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.203 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:16.204 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.658408+0000", 2024-03-20T09:03:16.204 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5e-06 2024-03-20T09:03:16.204 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.204 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.204 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:16.204 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.658415+0000", 2024-03-20T09:03:16.204 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0099999999999998e-06 2024-03-20T09:03:16.204 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.204 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.204 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:16.204 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.658431+0000", 2024-03-20T09:03:16.204 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5358000000000001e-05 2024-03-20T09:03:16.204 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.204 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.205 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:16.205 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.658439+0000", 2024-03-20T09:03:16.205 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.721e-06 2024-03-20T09:03:16.205 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.205 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.205 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:16.205 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.658488+0000", 2024-03-20T09:03:16.205 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8650999999999999e-05 2024-03-20T09:03:16.205 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.205 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.205 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:16.205 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.658592+0000", 2024-03-20T09:03:16.205 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104246 2024-03-20T09:03:16.205 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.205 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.206 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,4", 2024-03-20T09:03:16.206 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.658672+0000", 2024-03-20T09:03:16.206 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9739000000000003e-05 2024-03-20T09:03:16.206 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.206 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.206 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:16.206 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.663815+0000", 2024-03-20T09:03:16.206 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0051429090000000002 2024-03-20T09:03:16.206 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.206 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.206 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.206 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.667108+0000", 2024-03-20T09:03:16.206 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0032932349999999998 2024-03-20T09:03:16.206 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.207 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.207 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.207 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.853053+0000", 2024-03-20T09:03:16.207 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18594439600000001 2024-03-20T09:03:16.207 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.207 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.207 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:16.207 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.853108+0000", 2024-03-20T09:03:16.207 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5331999999999998e-05 2024-03-20T09:03:16.207 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.207 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.207 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:16.207 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.853150+0000", 2024-03-20T09:03:16.208 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2008000000000002e-05 2024-03-20T09:03:16.208 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.208 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:16.208 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.208 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.208 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.208 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:185600 2.f 2:f995c3b9:::benchmark_data_smithi122_107654_object66815:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-20T09:03:16.208 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:01:32.658682+0000", 2024-03-20T09:03:16.208 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 103.432294024, 2024-03-20T09:03:16.208 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18279983399999999, 2024-03-20T09:03:16.208 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:16.208 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:16.208 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:16.208 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:03:16.209 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:03:16.209 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 185600 2024-03-20T09:03:16.209 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.209 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:16.209 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.209 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:16.209 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.658682+0000", 2024-03-20T09:03:16.209 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.209 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.209 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.209 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:16.209 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.658682+0000", 2024-03-20T09:03:16.209 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.209 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.210 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.210 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:16.210 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.658683+0000", 2024-03-20T09:03:16.210 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9500000000000003e-07 2024-03-20T09:03:16.210 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.210 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.210 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:16.210 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.658686+0000", 2024-03-20T09:03:16.210 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7470000000000001e-06 2024-03-20T09:03:16.210 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.210 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.210 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:16.210 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.658692+0000", 2024-03-20T09:03:16.211 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5010000000000001e-06 2024-03-20T09:03:16.211 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.211 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.211 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:16.211 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.658698+0000", 2024-03-20T09:03:16.211 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6099999999999997e-06 2024-03-20T09:03:16.211 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.211 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.211 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:16.211 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.658760+0000", 2024-03-20T09:03:16.211 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.224e-05 2024-03-20T09:03:16.211 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.211 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.211 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:16.212 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.658882+0000", 2024-03-20T09:03:16.212 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000121863 2024-03-20T09:03:16.212 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.212 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.212 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 4,9", 2024-03-20T09:03:16.212 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.659002+0000", 2024-03-20T09:03:16.212 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011997699999999999 2024-03-20T09:03:16.212 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.212 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.212 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:16.212 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.667427+0000", 2024-03-20T09:03:16.212 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0084247769999999996 2024-03-20T09:03:16.212 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.212 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.213 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.213 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.671513+0000", 2024-03-20T09:03:16.213 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040864990000000004 2024-03-20T09:03:16.213 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.213 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.213 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.213 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.841409+0000", 2024-03-20T09:03:16.213 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.169896253 2024-03-20T09:03:16.213 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.213 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.213 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:16.213 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.841442+0000", 2024-03-20T09:03:16.213 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2387e-05 2024-03-20T09:03:16.213 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.214 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.214 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:16.214 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.841482+0000", 2024-03-20T09:03:16.214 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0485000000000003e-05 2024-03-20T09:03:16.214 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.214 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:16.214 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.214 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.214 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.214 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:185615 2.5 2:a2e93340:::benchmark_data_smithi122_107654_object66830:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-20T09:03:16.214 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:01:32.681959+0000", 2024-03-20T09:03:16.214 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 103.409017818, 2024-03-20T09:03:16.214 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18416463, 2024-03-20T09:03:16.214 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:16.215 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:16.215 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:16.215 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:03:16.215 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:03:16.215 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 185615 2024-03-20T09:03:16.215 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.215 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:16.215 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.215 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:16.215 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.681959+0000", 2024-03-20T09:03:16.215 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.215 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.215 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.215 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:16.216 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.681959+0000", 2024-03-20T09:03:16.216 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:16.216 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.216 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.216 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:16.216 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.681960+0000", 2024-03-20T09:03:16.216 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6819999999999999e-06 2024-03-20T09:03:16.216 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.216 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.216 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:16.216 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.681968+0000", 2024-03-20T09:03:16.216 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.729e-06 2024-03-20T09:03:16.216 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.216 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.217 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:16.217 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.681986+0000", 2024-03-20T09:03:16.217 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7961999999999998e-05 2024-03-20T09:03:16.217 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.217 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.217 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:16.217 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.681996+0000", 2024-03-20T09:03:16.217 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0285999999999999e-05 2024-03-20T09:03:16.217 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.217 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.217 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:16.217 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.682063+0000", 2024-03-20T09:03:16.217 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6970000000000004e-05 2024-03-20T09:03:16.217 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.217 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.218 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:16.218 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.682184+0000", 2024-03-20T09:03:16.218 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000120957 2024-03-20T09:03:16.218 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.218 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.218 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,4", 2024-03-20T09:03:16.218 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.682305+0000", 2024-03-20T09:03:16.218 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000120597 2024-03-20T09:03:16.218 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.218 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.218 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:16.218 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.687029+0000", 2024-03-20T09:03:16.218 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0047239170000000002 2024-03-20T09:03:16.219 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.219 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.219 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.219 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.687609+0000", 2024-03-20T09:03:16.219 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00058059300000000002 2024-03-20T09:03:16.219 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.219 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.219 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:16.219 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.866021+0000", 2024-03-20T09:03:16.219 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.17841179800000001 2024-03-20T09:03:16.219 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.219 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.219 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:16.219 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.866081+0000", 2024-03-20T09:03:16.219 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9664000000000003e-05 2024-03-20T09:03:16.220 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:16.220 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:16.220 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:16.220 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.866123+0000", 2024-03-20T09:03:16.220 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2475000000000002e-05 2024-03-20T09:03:16.220 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.220 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:16.220 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.220 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:16.220 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:16.220 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:03:16.313 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:03:16.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:16.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:03:16.331 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:03:16.419 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:03:16.469 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:16.504 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:03:16.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:16.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:03:16.605 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 15 16 30695 30679 127.799 159.5 0.00292808 0.00779952 2024-03-20T09:03:16.609 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:03:16.642 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:16.852 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:16 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:03:16.852 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:16 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:03:16.894 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:03:16.896 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_ops_in_flight 2024-03-20T09:03:16.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:16.898 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:03:16.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:16 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:03:16.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:16 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:03:16.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:16 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:03:16.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:16 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:03:16.999 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:03:17.050 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:17.080 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:03:17.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:17.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:03:17.191 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:03:17.239 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:17.269 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:03:17.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:17.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:03:17.309 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:03:17.373 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:03:17.399 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:17.605 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 16 15 32513 32498 126.916 113.688 0.00616071 0.00787433 2024-03-20T09:03:17.886 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:03:17.917 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:17 smithi026 ceph-mon[68309]: osdmap e222: 12 total, 12 up, 12 in 2024-03-20T09:03:17.917 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:17 smithi026 ceph-mon[68309]: osdmap e223: 12 total, 12 up, 12 in 2024-03-20T09:03:17.917 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:17 smithi026 ceph-mon[68309]: pgmap v624: 27 pgs: 27 active+clean; 1.7 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 125 MiB/s wr, 2.00k op/s; 11 MiB/s, 175 objects/s recovering 2024-03-20T09:03:17.917 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:17 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/757502587' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:03:17.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:17 smithi122 ceph-mon[73495]: osdmap e222: 12 total, 12 up, 12 in 2024-03-20T09:03:17.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:17 smithi122 ceph-mon[73495]: osdmap e223: 12 total, 12 up, 12 in 2024-03-20T09:03:17.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:17 smithi122 ceph-mon[73495]: pgmap v624: 27 pgs: 27 active+clean; 1.7 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 125 MiB/s wr, 2.00k op/s; 11 MiB/s, 175 objects/s recovering 2024-03-20T09:03:17.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:17 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/757502587' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:03:18.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:17 smithi115 ceph-mon[72476]: osdmap e222: 12 total, 12 up, 12 in 2024-03-20T09:03:18.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:17 smithi115 ceph-mon[72476]: osdmap e223: 12 total, 12 up, 12 in 2024-03-20T09:03:18.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:17 smithi115 ceph-mon[72476]: pgmap v624: 27 pgs: 27 active+clean; 1.7 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 125 MiB/s wr, 2.00k op/s; 11 MiB/s, 175 objects/s recovering 2024-03-20T09:03:18.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:17 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/757502587' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:03:18.606 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 17 16 35002 34986 128.592 155.5 0.00515543 0.00777213 2024-03-20T09:03:18.849 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:03:18.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:18 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:03:18.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:18 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:03:18.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:18 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:03:18.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:18 smithi122 ceph-mon[73495]: Cluster is now healthy 2024-03-20T09:03:18.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:18 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/757502587' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:03:18.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:18 smithi122 ceph-mon[73495]: osdmap e224: 12 total, 12 up, 12 in 2024-03-20T09:03:18.947 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:18 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:03:18.947 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:18 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:03:18.947 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:18 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:03:18.947 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:18 smithi026 ceph-mon[68309]: Cluster is now healthy 2024-03-20T09:03:18.947 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:18 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/757502587' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:03:18.948 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:18 smithi026 ceph-mon[68309]: osdmap e224: 12 total, 12 up, 12 in 2024-03-20T09:03:19.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:18 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:03:19.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:18 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:03:19.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:18 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:03:19.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:18 smithi115 ceph-mon[72476]: Cluster is now healthy 2024-03-20T09:03:19.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:18 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/757502587' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:03:19.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:18 smithi115 ceph-mon[72476]: osdmap e224: 12 total, 12 up, 12 in 2024-03-20T09:03:19.400 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:03:19.605 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 18 16 37078 37062 128.656 129.75 0.0036398 0.00776816 2024-03-20T09:03:19.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:19 smithi122 ceph-mon[73495]: pgmap v626: 27 pgs: 27 active+clean; 1.9 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 128 MiB/s wr, 2.04k op/s; 5.1 MiB/s, 81 objects/s recovering 2024-03-20T09:03:19.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:19 smithi026 ceph-mon[68309]: pgmap v626: 27 pgs: 27 active+clean; 1.9 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 128 MiB/s wr, 2.04k op/s; 5.1 MiB/s, 81 objects/s recovering 2024-03-20T09:03:20.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:19 smithi115 ceph-mon[72476]: pgmap v626: 27 pgs: 27 active+clean; 1.9 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 128 MiB/s wr, 2.04k op/s; 5.1 MiB/s, 81 objects/s recovering 2024-03-20T09:03:20.252 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:03:20.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:20.258 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:03:20.358 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:03:20.399 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:20.556 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:03:20.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:20.561 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:03:20.606 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 19 16 39372 39356 129.43 143.375 0.00256811 0.00772141 2024-03-20T09:03:20.662 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:03:20.716 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:20.752 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:03:20.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:20.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:03:20.806 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:03:20.859 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:03:20.905 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:20.930 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:03:20.931 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:20 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:03:20.932 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:20 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:03:20.932 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:20 smithi115 ceph-mon[72476]: osdmap e225: 12 total, 12 up, 12 in 2024-03-20T09:03:20.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:20.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:03:20.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:20 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:03:20.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:20 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:03:20.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:20 smithi122 ceph-mon[73495]: osdmap e225: 12 total, 12 up, 12 in 2024-03-20T09:03:20.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:20 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:03:20.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:20 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:03:20.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:20 smithi026 ceph-mon[68309]: osdmap e225: 12 total, 12 up, 12 in 2024-03-20T09:03:21.033 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:03:21.079 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:21.116 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:03:21.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:21.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:03:21.224 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:03:21.257 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:21.463 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:03:21.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:21.467 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:03:21.568 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:03:21.606 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:2024-03-20T09:03:21.606451+0000 min lat: 0.001051 max lat: 2.05653 avg lat: 0.00771322 2024-03-20T09:03:21.606 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T09:03:21.606 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 20 16 41487 41471 129.567 132.188 0.0100812 0.00771322 2024-03-20T09:03:21.606 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:21.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:21 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:03:21.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:21 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:03:21.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:21 smithi122 ceph-mon[73495]: from='client.24943 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:03:21.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:21 smithi122 ceph-mon[73495]: pgmap v628: 27 pgs: 27 active+clean; 2.2 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 138 MiB/s wr, 2.21k op/s 2024-03-20T09:03:21.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:21 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:03:21.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:21 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:03:21.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:21 smithi026 ceph-mon[68309]: from='client.24943 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:03:21.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:21 smithi026 ceph-mon[68309]: pgmap v628: 27 pgs: 27 active+clean; 2.2 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 138 MiB/s wr, 2.21k op/s 2024-03-20T09:03:21.891 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:03:21.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:21.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:03:21.997 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:03:22.032 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:21 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:03:22.032 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:21 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:03:22.032 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:21 smithi115 ceph-mon[72476]: from='client.24943 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:03:22.032 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:21 smithi115 ceph-mon[72476]: pgmap v628: 27 pgs: 27 active+clean; 2.2 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 138 MiB/s wr, 2.21k op/s 2024-03-20T09:03:22.041 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:22.069 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:03:22.069 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-20T09:03:21.282+0000 7f1349a3a700 0 --2- 172.21.15.26:0/1428854240 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1344101a30 0x7f134419ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-03-20T09:03:22.070 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:03:22.074 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:03:22.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:22.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:03:22.127 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:03:22.128 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:03:22.128 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-03-20T09:03:22.128 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:03:22.180 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:03:22.225 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:22.259 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:03:22.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:22.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:03:22.363 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:03:22.398 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:22.606 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 21 16 43403 43387 129.099 119.75 0.00368787 0.00774187 2024-03-20T09:03:22.756 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:03:22.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:22.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:03:22.778 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":627,"stamp":"2024-03-20T09:03:19.674469+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2003829280,"num_objects":30569,"num_object_clones":0,"num_object_copies":91707,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30567,"num_write_kb":1956288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3971,"num_bytes_recovered":260243456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32074,"ondisk_log_size":32074,"up":81,"acting":81,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":86,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":6971676,"kb_used_data":5724284,"kb_used_omap":18,"kb_used_meta":1247213,"kb_avail":1117970148,"statfs":{"total":1151940427776,"available":1144801431552,"internally_reserved":0,"allocated":5861666816,"data_stored":5856512572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":1277146427},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":433782784,"num_objects":6619,"num_object_clones":0,"num_object_copies":19857,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6619,"num_write_kb":423616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.070273"},"pg_stats":[{"pgid":"3.f","version":"224'2224","reported_seq":2274,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-03-20T09:03:19.204729+0000","last_change":"2024-03-20T09:02:49.921660+0000","last_active":"2024-03-20T09:03:19.204729+0000","last_peered":"2024-03-20T09:03:19.204729+0000","last_clean":"2024-03-20T09:03:19.204729+0000","last_became_active":"2024-03-20T09:02:49.921153+0000","last_became_peered":"2024-03-20T09:02:49.921153+0000","last_unstale":"2024-03-20T09:03:19.204729+0000","last_undegraded":"2024-03-20T09:03:19.204729+0000","last_fullsized":"2024-03-20T09:03:19.204729+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":2224,"log_dups_size":0,"ondisk_log_size":2224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:04:06.579353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145752064,"num_objects":2224,"num_object_clones":0,"num_object_copies":6672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2224,"num_write_kb":142336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,8],"acting":[11,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"224'2323","reported_seq":3728,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-03-20T09:03:19.204980+0000","last_change":"2024-03-20T09:03:10.635160+0000","last_active":"2024-03-20T09:03:19.204980+0000","last_peered":"2024-03-20T09:03:19.204980+0000","last_clean":"2024-03-20T09:03:19.204980+0000","last_became_active":"2024-03-20T09:03:10.633041+0000","last_became_peered":"2024-03-20T09:03:10.633041+0000","last_unstale":"2024-03-20T09:03:19.204980+0000","last_undegraded":"2024-03-20T09:03:19.204980+0000","last_fullsized":"2024-03-20T09:03:19.204980+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":2323,"log_dups_size":0,"ondisk_log_size":2323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:04:08.491161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152240128,"num_objects":2323,"num_object_clones":0,"num_object_copies":6969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2323,"num_write_kb":148672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":672,"num_bytes_recovered":44040192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9],"acting":[11,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.d","version":"223'1953","reported_seq":2000,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-03-20T09:03:17.120344+0000","last_change":"2024-03-20T09:02:49.921481+0000","last_active":"2024-03-20T09:03:17.120344+0000","last_peered":"2024-03-20T09:03:17.120344+0000","last_clean":"2024-03-20T09:03:17.120344+0000","last_became_active":"2024-03-20T09:02:49.921160+0000","last_became_peered":"2024-03-20T09:02:49.921160+0000","last_unstale":"2024-03-20T09:03:17.120344+0000","last_undegraded":"2024-03-20T09:03:17.120344+0000","last_fullsized":"2024-03-20T09:03:17.120344+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":1953,"log_dups_size":0,"ondisk_log_size":1953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:04:07.713258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127991808,"num_objects":1953,"num_object_clones":0,"num_object_copies":5859,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1953,"num_write_kb":124992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,9],"acting":[5,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"223'1939","reported_seq":1987,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-03-20T09:03:17.100602+0000","last_change":"2024-03-20T09:02:49.893721+0000","last_active":"2024-03-20T09:03:17.100602+0000","last_peered":"2024-03-20T09:03:17.100602+0000","last_clean":"2024-03-20T09:03:17.100602+0000","last_became_active":"2024-03-20T09:02:49.893394+0000","last_became_peered":"2024-03-20T09:02:49.893394+0000","last_unstale":"2024-03-20T09:03:17.100602+0000","last_undegraded":"2024-03-20T09:03:17.100602+0000","last_fullsized":"2024-03-20T09:03:17.100602+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":1939,"log_dups_size":0,"ondisk_log_size":1939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:04:03.912815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127074304,"num_objects":1939,"num_object_clones":0,"num_object_copies":5817,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1939,"num_write_kb":124096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"221'1953","reported_seq":1996,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-20T09:03:16.616278+0000","last_change":"2024-03-20T09:02:49.921268+0000","last_active":"2024-03-20T09:03:16.616278+0000","last_peered":"2024-03-20T09:03:16.616278+0000","last_clean":"2024-03-20T09:03:16.616278+0000","last_became_active":"2024-03-20T09:02:49.920857+0000","last_became_peered":"2024-03-20T09:02:49.920857+0000","last_unstale":"2024-03-20T09:03:16.616278+0000","last_undegraded":"2024-03-20T09:03:16.616278+0000","last_fullsized":"2024-03-20T09:03:16.616278+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":1953,"log_dups_size":0,"ondisk_log_size":1953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:03:58.497193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127991808,"num_objects":1953,"num_object_clones":0,"num_object_copies":5859,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1953,"num_write_kb":124992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.5","version":"223'2030","reported_seq":3362,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-03-20T09:03:17.128799+0000","last_change":"2024-03-20T09:03:11.638864+0000","last_active":"2024-03-20T09:03:17.128799+0000","last_peered":"2024-03-20T09:03:17.128799+0000","last_clean":"2024-03-20T09:03:17.128799+0000","last_became_active":"2024-03-20T09:03:11.636554+0000","last_became_peered":"2024-03-20T09:03:11.636554+0000","last_unstale":"2024-03-20T09:03:17.128799+0000","last_undegraded":"2024-03-20T09:03:17.128799+0000","last_fullsized":"2024-03-20T09:03:17.128799+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":2030,"log_dups_size":0,"ondisk_log_size":2030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:03:54.071840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133038080,"num_objects":2030,"num_object_clones":0,"num_object_copies":6090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2030,"num_write_kb":129920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":637,"num_bytes_recovered":41746432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"20'137","reported_seq":333,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-20T09:03:11.870133+0000","last_change":"2024-03-20T09:02:42.863702+0000","last_active":"2024-03-20T09:03:11.870133+0000","last_peered":"2024-03-20T09:03:11.870133+0000","last_clean":"2024-03-20T09:03:11.870133+0000","last_became_active":"2024-03-20T09:02:42.863240+0000","last_became_peered":"2024-03-20T09:02:42.863240+0000","last_unstale":"2024-03-20T09:03:11.870133+0000","last_undegraded":"2024-03-20T09:03:11.870133+0000","last_fullsized":"2024-03-20T09:03:11.870133+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:21.734957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:21.734957+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:03:28.236144+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"221'1717","reported_seq":1760,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-20T09:03:15.228661+0000","last_change":"2024-03-20T09:02:49.895676+0000","last_active":"2024-03-20T09:03:15.228661+0000","last_peered":"2024-03-20T09:03:15.228661+0000","last_clean":"2024-03-20T09:03:15.228661+0000","last_became_active":"2024-03-20T09:02:49.895272+0000","last_became_peered":"2024-03-20T09:02:49.895272+0000","last_unstale":"2024-03-20T09:03:15.228661+0000","last_undegraded":"2024-03-20T09:03:15.228661+0000","last_fullsized":"2024-03-20T09:03:15.228661+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":1717,"log_dups_size":0,"ondisk_log_size":1717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:04:12.084447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112525312,"num_objects":1717,"num_object_clones":0,"num_object_copies":5151,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1717,"num_write_kb":109888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"20'137","reported_seq":336,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-20T09:03:11.866130+0000","last_change":"2024-03-20T09:03:07.873347+0000","last_active":"2024-03-20T09:03:11.866130+0000","last_peered":"2024-03-20T09:03:11.866130+0000","last_clean":"2024-03-20T09:03:11.866130+0000","last_became_active":"2024-03-20T09:03:07.872941+0000","last_became_peered":"2024-03-20T09:03:07.872941+0000","last_unstale":"2024-03-20T09:03:11.866130+0000","last_undegraded":"2024-03-20T09:03:11.866130+0000","last_fullsized":"2024-03-20T09:03:11.866130+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:25.927693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:25.927693+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:03:34.222725+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,11],"acting":[10,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"224'2268","reported_seq":3702,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-03-20T09:03:19.200488+0000","last_change":"2024-03-20T09:03:10.633077+0000","last_active":"2024-03-20T09:03:19.200488+0000","last_peered":"2024-03-20T09:03:19.200488+0000","last_clean":"2024-03-20T09:03:19.200488+0000","last_became_active":"2024-03-20T09:03:10.631273+0000","last_became_peered":"2024-03-20T09:03:10.631273+0000","last_unstale":"2024-03-20T09:03:19.200488+0000","last_undegraded":"2024-03-20T09:03:19.200488+0000","last_fullsized":"2024-03-20T09:03:19.200488+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":2268,"log_dups_size":0,"ondisk_log_size":2268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:03:59.872028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148635648,"num_objects":2268,"num_object_clones":0,"num_object_copies":6804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2268,"num_write_kb":145152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":687,"num_bytes_recovered":45023232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,1],"acting":[11,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.2","version":"20'137","reported_seq":634,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-03-20T09:03:16.879631+0000","last_change":"2024-03-20T09:02:24.000401+0000","last_active":"2024-03-20T09:03:16.879631+0000","last_peered":"2024-03-20T09:03:16.879631+0000","last_clean":"2024-03-20T09:03:16.879631+0000","last_became_active":"2024-03-20T09:02:06.963614+0000","last_became_peered":"2024-03-20T09:02:06.963614+0000","last_unstale":"2024-03-20T09:03:16.879631+0000","last_undegraded":"2024-03-20T09:03:16.879631+0000","last_fullsized":"2024-03-20T09:03:16.879631+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:24.000204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:24.000204+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:03:51.807913+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"221'1660","reported_seq":3030,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-20T09:03:14.713726+0000","last_change":"2024-03-20T09:03:11.640237+0000","last_active":"2024-03-20T09:03:14.713726+0000","last_peered":"2024-03-20T09:03:14.713726+0000","last_clean":"2024-03-20T09:03:14.713726+0000","last_became_active":"2024-03-20T09:03:11.637935+0000","last_became_peered":"2024-03-20T09:03:11.637935+0000","last_unstale":"2024-03-20T09:03:14.713726+0000","last_undegraded":"2024-03-20T09:03:14.713726+0000","last_fullsized":"2024-03-20T09:03:14.713726+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":1660,"log_dups_size":0,"ondisk_log_size":1660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:03:58.425347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108789760,"num_objects":1660,"num_object_clones":0,"num_object_copies":4980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1660,"num_write_kb":106240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":658,"num_bytes_recovered":43122688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":403,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-03-20T09:03:17.889352+0000","last_change":"2024-03-20T09:02:23.383277+0000","last_active":"2024-03-20T09:03:17.889352+0000","last_peered":"2024-03-20T09:03:17.889352+0000","last_clean":"2024-03-20T09:03:17.889352+0000","last_became_active":"2024-03-20T09:02:06.964051+0000","last_became_peered":"2024-03-20T09:02:06.964051+0000","last_unstale":"2024-03-20T09:03:17.889352+0000","last_undegraded":"2024-03-20T09:03:17.889352+0000","last_fullsized":"2024-03-20T09:03:17.889352+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:23.383145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:23.383145+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:03:38.935982+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1","version":"223'2046","reported_seq":2094,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-03-20T09:03:17.202004+0000","last_change":"2024-03-20T09:02:49.893607+0000","last_active":"2024-03-20T09:03:17.202004+0000","last_peered":"2024-03-20T09:03:17.202004+0000","last_clean":"2024-03-20T09:03:17.202004+0000","last_became_active":"2024-03-20T09:02:49.893244+0000","last_became_peered":"2024-03-20T09:02:49.893244+0000","last_unstale":"2024-03-20T09:03:17.202004+0000","last_undegraded":"2024-03-20T09:03:17.202004+0000","last_fullsized":"2024-03-20T09:03:17.202004+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":2046,"log_dups_size":0,"ondisk_log_size":2046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:04:05.613961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134086656,"num_objects":2046,"num_object_clones":0,"num_object_copies":6138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2046,"num_write_kb":130944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,10],"acting":[0,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"20'137","reported_seq":314,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-20T09:03:11.872429+0000","last_change":"2024-03-20T09:02:24.936689+0000","last_active":"2024-03-20T09:03:11.872429+0000","last_peered":"2024-03-20T09:03:11.872429+0000","last_clean":"2024-03-20T09:03:11.872429+0000","last_became_active":"2024-03-20T09:01:57.862486+0000","last_became_peered":"2024-03-20T09:01:57.862486+0000","last_unstale":"2024-03-20T09:03:11.872429+0000","last_undegraded":"2024-03-20T09:03:11.872429+0000","last_fullsized":"2024-03-20T09:03:11.872429+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:24.936556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:24.936556+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:03:48.646357+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"221'1596","reported_seq":1639,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-20T09:03:14.708806+0000","last_change":"2024-03-20T09:02:49.897561+0000","last_active":"2024-03-20T09:03:14.708806+0000","last_peered":"2024-03-20T09:03:14.708806+0000","last_clean":"2024-03-20T09:03:14.708806+0000","last_became_active":"2024-03-20T09:02:49.897287+0000","last_became_peered":"2024-03-20T09:02:49.897287+0000","last_unstale":"2024-03-20T09:03:14.708806+0000","last_undegraded":"2024-03-20T09:03:14.708806+0000","last_fullsized":"2024-03-20T09:03:14.708806+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":1596,"log_dups_size":0,"ondisk_log_size":1596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:03:52.376273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104595456,"num_objects":1596,"num_object_clones":0,"num_object_copies":4788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1596,"num_write_kb":102144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"20'137","reported_seq":327,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-20T09:03:11.866185+0000","last_change":"2024-03-20T09:03:07.876977+0000","last_active":"2024-03-20T09:03:11.866185+0000","last_peered":"2024-03-20T09:03:11.866185+0000","last_clean":"2024-03-20T09:03:11.866185+0000","last_became_active":"2024-03-20T09:03:07.876617+0000","last_became_peered":"2024-03-20T09:03:07.876617+0000","last_unstale":"2024-03-20T09:03:11.866185+0000","last_undegraded":"2024-03-20T09:03:11.866185+0000","last_fullsized":"2024-03-20T09:03:11.866185+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:26.880624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:26.880624+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:03:55.611860+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,4],"acting":[10,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.7","version":"221'1611","reported_seq":1654,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-20T09:03:15.095014+0000","last_change":"2024-03-20T09:02:49.921094+0000","last_active":"2024-03-20T09:03:15.095014+0000","last_peered":"2024-03-20T09:03:15.095014+0000","last_clean":"2024-03-20T09:03:15.095014+0000","last_became_active":"2024-03-20T09:02:49.920789+0000","last_became_peered":"2024-03-20T09:02:49.920789+0000","last_unstale":"2024-03-20T09:03:15.095014+0000","last_undegraded":"2024-03-20T09:03:15.095014+0000","last_fullsized":"2024-03-20T09:03:15.095014+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":1611,"log_dups_size":0,"ondisk_log_size":1611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:03:58.884923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105578496,"num_objects":1611,"num_object_clones":0,"num_object_copies":4833,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1611,"num_write_kb":103104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"20'137","reported_seq":318,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-20T09:03:11.866391+0000","last_change":"2024-03-20T09:02:27.912312+0000","last_active":"2024-03-20T09:03:11.866391+0000","last_peered":"2024-03-20T09:03:11.866391+0000","last_clean":"2024-03-20T09:03:11.866391+0000","last_became_active":"2024-03-20T09:01:57.869661+0000","last_became_peered":"2024-03-20T09:01:57.869661+0000","last_unstale":"2024-03-20T09:03:11.866391+0000","last_undegraded":"2024-03-20T09:03:11.866391+0000","last_fullsized":"2024-03-20T09:03:11.866391+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:27.912168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:27.912168+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:03:33.629842+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.6","version":"221'1698","reported_seq":3103,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-20T09:03:14.726973+0000","last_change":"2024-03-20T09:03:11.641901+0000","last_active":"2024-03-20T09:03:14.726973+0000","last_peered":"2024-03-20T09:03:14.726973+0000","last_clean":"2024-03-20T09:03:14.726973+0000","last_became_active":"2024-03-20T09:03:11.638887+0000","last_became_peered":"2024-03-20T09:03:11.638887+0000","last_unstale":"2024-03-20T09:03:14.726973+0000","last_undegraded":"2024-03-20T09:03:14.726973+0000","last_fullsized":"2024-03-20T09:03:14.726973+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":1698,"log_dups_size":0,"ondisk_log_size":1698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:04:14.325147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111280128,"num_objects":1698,"num_object_clones":0,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1698,"num_write_kb":108672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":675,"num_bytes_recovered":44236800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"20'137","reported_seq":420,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-03-20T09:03:17.889463+0000","last_change":"2024-03-20T09:02:24.397397+0000","last_active":"2024-03-20T09:03:17.889463+0000","last_peered":"2024-03-20T09:03:17.889463+0000","last_clean":"2024-03-20T09:03:17.889463+0000","last_became_active":"2024-03-20T09:01:57.861749+0000","last_became_peered":"2024-03-20T09:01:57.861749+0000","last_unstale":"2024-03-20T09:03:17.889463+0000","last_undegraded":"2024-03-20T09:03:17.889463+0000","last_fullsized":"2024-03-20T09:03:17.889463+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:24.397259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:24.397259+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:03:34.066614+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.8","version":"224'2255","reported_seq":2305,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-03-20T09:03:19.204794+0000","last_change":"2024-03-20T09:02:49.896667+0000","last_active":"2024-03-20T09:03:19.204794+0000","last_peered":"2024-03-20T09:03:19.204794+0000","last_clean":"2024-03-20T09:03:19.204794+0000","last_became_active":"2024-03-20T09:02:49.896349+0000","last_became_peered":"2024-03-20T09:02:49.896349+0000","last_unstale":"2024-03-20T09:03:19.204794+0000","last_undegraded":"2024-03-20T09:03:19.204794+0000","last_fullsized":"2024-03-20T09:03:19.204794+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":2255,"log_dups_size":0,"ondisk_log_size":2255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:03:50.271772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147783680,"num_objects":2255,"num_object_clones":0,"num_object_copies":6765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2255,"num_write_kb":144320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"20'137","reported_seq":272,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-20T09:03:13.525382+0000","last_change":"2024-03-20T09:03:13.525409+0000","last_active":"2024-03-20T09:03:13.525382+0000","last_peered":"2024-03-20T09:03:13.525382+0000","last_clean":"2024-03-20T09:03:13.525382+0000","last_became_active":"2024-03-20T09:03:07.875459+0000","last_became_peered":"2024-03-20T09:03:07.875459+0000","last_unstale":"2024-03-20T09:03:13.525382+0000","last_undegraded":"2024-03-20T09:03:13.525382+0000","last_fullsized":"2024-03-20T09:03:13.525382+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:03:13.525325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:03:13.525325+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:04:22.073540+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"221'1658","reported_seq":2996,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-20T09:03:15.090274+0000","last_change":"2024-03-20T09:03:10.635413+0000","last_active":"2024-03-20T09:03:15.090274+0000","last_peered":"2024-03-20T09:03:15.090274+0000","last_clean":"2024-03-20T09:03:15.090274+0000","last_became_active":"2024-03-20T09:03:10.633998+0000","last_became_peered":"2024-03-20T09:03:10.633998+0000","last_unstale":"2024-03-20T09:03:15.090274+0000","last_undegraded":"2024-03-20T09:03:15.090274+0000","last_fullsized":"2024-03-20T09:03:15.090274+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":1658,"log_dups_size":0,"ondisk_log_size":1658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:03:52.516145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108658688,"num_objects":1658,"num_object_clones":0,"num_object_copies":4974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1658,"num_write_kb":106112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":642,"num_bytes_recovered":42074112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,11],"acting":[3,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"20'137","reported_seq":636,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-03-20T09:03:17.891423+0000","last_change":"2024-03-20T09:02:25.540219+0000","last_active":"2024-03-20T09:03:17.891423+0000","last_peered":"2024-03-20T09:03:17.891423+0000","last_clean":"2024-03-20T09:03:17.891423+0000","last_became_active":"2024-03-20T09:02:06.962814+0000","last_became_peered":"2024-03-20T09:02:06.962814+0000","last_unstale":"2024-03-20T09:03:17.891423+0000","last_undegraded":"2024-03-20T09:03:17.891423+0000","last_fullsized":"2024-03-20T09:03:17.891423+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:25.540102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:25.540102+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:03:46.963634+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,2],"acting":[9,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.a","version":"221'1636","reported_seq":1680,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-20T09:03:14.723238+0000","last_change":"2024-03-20T09:02:49.896615+0000","last_active":"2024-03-20T09:03:14.723238+0000","last_peered":"2024-03-20T09:03:14.723238+0000","last_clean":"2024-03-20T09:03:14.723238+0000","last_became_active":"2024-03-20T09:02:49.896263+0000","last_became_peered":"2024-03-20T09:02:49.896263+0000","last_unstale":"2024-03-20T09:03:14.723238+0000","last_undegraded":"2024-03-20T09:03:14.723238+0000","last_fullsized":"2024-03-20T09:03:14.723238+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":1636,"log_dups_size":0,"ondisk_log_size":1636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:04:11.864532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107216896,"num_objects":1636,"num_object_clones":0,"num_object_copies":4908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1636,"num_write_kb":104704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,8],"acting":[6,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"20'137","reported_seq":322,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-03-20T09:03:16.871213+0000","last_change":"2024-03-20T09:02:21.099690+0000","last_active":"2024-03-20T09:03:16.871213+0000","last_peered":"2024-03-20T09:03:16.871213+0000","last_clean":"2024-03-20T09:03:16.871213+0000","last_became_active":"2024-03-20T09:01:57.871509+0000","last_became_peered":"2024-03-20T09:01:57.871509+0000","last_unstale":"2024-03-20T09:03:16.871213+0000","last_undegraded":"2024-03-20T09:03:16.871213+0000","last_fullsized":"2024-03-20T09:03:16.871213+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:21.099608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:21.099608+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:03:39.779070+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":2003238912,"num_objects":30567,"num_object_clones":0,"num_object_copies":91701,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30567,"num_write_kb":1956288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3971,"num_bytes_recovered":260243456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6071255040,"data_stored":6071255040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30567,"ondisk_log_size":30567,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1507,"ondisk_log_size":1507,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860504,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1218248,"kb_used_data":1114880,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92526904,"statfs":{"total":95995035648,"available":94747549696,"internally_reserved":0,"allocated":1141637120,"data_stored":1141197574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":105839044},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":89,"seq":382252089436,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":183268,"kb_used_data":120156,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93561884,"statfs":{"total":95995035648,"available":95807369216,"internally_reserved":0,"allocated":123039744,"data_stored":122613742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":64616896},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":80,"seq":343597383780,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":387452,"kb_used_data":266036,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93357700,"statfs":{"total":95995035648,"available":95598284800,"internally_reserved":0,"allocated":272420864,"data_stored":271985816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":124320192},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":70,"seq":300647710824,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":458340,"kb_used_data":346652,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93286812,"statfs":{"total":95995035648,"available":95525695488,"internally_reserved":0,"allocated":354971648,"data_stored":354545654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114358726},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":60,"seq":257698037869,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":720996,"kb_used_data":573212,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93024156,"statfs":{"total":95995035648,"available":95256735744,"internally_reserved":0,"allocated":586969088,"data_stored":586543094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":151321034},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332210,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":684132,"kb_used_data":616860,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93061020,"statfs":{"total":95995035648,"available":95294484480,"internally_reserved":0,"allocated":631664640,"data_stored":631238646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574988,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":518820,"kb_used_data":393244,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93226332,"statfs":{"total":95995035648,"available":95463763968,"internally_reserved":0,"allocated":402681856,"data_stored":402255862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":128580038},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738515,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":469284,"kb_used_data":339548,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93275868,"statfs":{"total":95995035648,"available":95514488832,"internally_reserved":0,"allocated":347697152,"data_stored":347271158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411462,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":225092,"kb_used_data":135548,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93520060,"statfs":{"total":95995035648,"available":95764541440,"internally_reserved":0,"allocated":138801152,"data_stored":138362560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149825,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":483948,"kb_used_data":366692,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":93261204,"statfs":{"total":95995035648,"available":95499472896,"internally_reserved":0,"allocated":375492608,"data_stored":375063062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":120060359},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953597,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":956132,"kb_used_data":852764,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92789020,"statfs":{"total":95995035648,"available":95015956480,"internally_reserved":0,"allocated":873230336,"data_stored":872804342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659256,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":665964,"kb_used_data":598692,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93079188,"statfs":{"total":95995035648,"available":95313088512,"internally_reserved":0,"allocated":613060608,"data_stored":612631062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":359268352,"data_stored":359268352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":431489024,"data_stored":431489024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":141754368,"data_stored":141754368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":391839744,"data_stored":391839744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":900268032,"data_stored":900268032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":646774784,"data_stored":646774784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":645857280,"data_stored":645857280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593362944,"data_stored":593362944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":376569856,"data_stored":376569856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":273874944,"data_stored":273874944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":124190720,"data_stored":124190720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1186004992,"data_stored":1186004992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:03:22.780 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-20T09:03:22.780 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pool set unique_pool_1 pg_num 26 2024-03-20T09:03:22.847 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:22 smithi122 ceph-mon[73495]: osdmap e226: 12 total, 12 up, 12 in 2024-03-20T09:03:22.847 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:22 smithi122 ceph-mon[73495]: osdmap e227: 12 total, 12 up, 12 in 2024-03-20T09:03:22.860 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:03:22.888 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_blocked_ops 2024-03-20T09:03:22.915 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:22.946 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:03:22.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:22.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:03:22.954 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:22 smithi115 ceph-mon[72476]: osdmap e226: 12 total, 12 up, 12 in 2024-03-20T09:03:22.954 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:22 smithi115 ceph-mon[72476]: osdmap e227: 12 total, 12 up, 12 in 2024-03-20T09:03:22.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:22 smithi026 ceph-mon[68309]: osdmap e226: 12 total, 12 up, 12 in 2024-03-20T09:03:22.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:22 smithi026 ceph-mon[68309]: osdmap e227: 12 total, 12 up, 12 in 2024-03-20T09:03:23.051 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:03:23.097 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:23.127 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:03:23.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:23.133 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:03:23.235 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:03:23.270 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:03:23.281 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:23.316 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:03:23.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:23.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:03:23.339 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:03:23.404 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:03:23.420 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:03:23.421 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:03:23.457 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:03:23.462 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:23.497 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:03:23.530 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:03:23.564 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:03:23.598 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:03:23.599 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:03:23.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:23.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:03:23.606 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 22 16 45050 45034 127.91 102.938 0.00945294 0.00781333 2024-03-20T09:03:23.636 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:03:23.666 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:03:23.694 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:03:23.702 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:03:23.729 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:03:23.747 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:23.758 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:03:23.794 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:03:23.795 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:03:23.795 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:03:23.795 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:03:23.795 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:03:23.795 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:03:23.795 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:03:23.795 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:03:23.795 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:03:23.795 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:03:23.796 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:03:23.796 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:03:23.796 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:03:23.830 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:23 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:03:23.830 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:23 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:03:23.830 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:23 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1313466485' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:03:23.830 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:23 smithi122 ceph-mon[73495]: pgmap v631: 27 pgs: 27 active+clean; 2.4 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 123 MiB/s wr, 1.96k op/s 2024-03-20T09:03:23.900 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:03:23.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:23.905 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:23 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:03:23.905 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:23 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:03:23.905 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:23 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1313466485' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:03:23.905 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:23 smithi026 ceph-mon[68309]: pgmap v631: 27 pgs: 27 active+clean; 2.4 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 123 MiB/s wr, 1.96k op/s 2024-03-20T09:03:23.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:03:24.007 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:03:24.047 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:24.107 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:03:24.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:24.110 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:03:24.158 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:23 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:03:24.159 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:23 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:03:24.159 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:23 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1313466485' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:03:24.159 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:23 smithi115 ceph-mon[72476]: pgmap v631: 27 pgs: 27 active+clean; 2.4 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 123 MiB/s wr, 1.96k op/s 2024-03-20T09:03:24.211 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:03:24.253 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:24.277 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:03:24.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:24.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:03:24.384 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:03:24.420 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:24.607 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 23 16 47050 47034 127.778 125 0.00434817 0.00782118 2024-03-20T09:03:24.628 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:03:24.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:24.633 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:03:24.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:24 smithi026 ceph-mon[68309]: osdmap e228: 12 total, 12 up, 12 in 2024-03-20T09:03:24.734 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:03:24.772 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:24.973 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:24 smithi122 ceph-mon[73495]: osdmap e228: 12 total, 12 up, 12 in 2024-03-20T09:03:25.124 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:03:25.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:25.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:03:25.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:24 smithi115 ceph-mon[72476]: osdmap e228: 12 total, 12 up, 12 in 2024-03-20T09:03:25.231 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:03:25.266 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:25.300 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:03:25.300 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:03:25.420 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:03:25.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:25.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:03:25.524 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:03:25.566 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:25.594 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:03:25.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:25.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:03:25.607 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 24 16 49030 49014 127.61 123.75 0.00295436 0.0078319 2024-03-20T09:03:25.697 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:03:25.736 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:25.908 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:25 smithi115 ceph-mon[72476]: pgmap v633: 27 pgs: 27 active+clean; 2.8 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 160 MiB/s wr, 2.56k op/s 2024-03-20T09:03:25.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:25 smithi122 ceph-mon[73495]: pgmap v633: 27 pgs: 27 active+clean; 2.8 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 160 MiB/s wr, 2.56k op/s 2024-03-20T09:03:26.052 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_historic_ops 2024-03-20T09:03:26.089 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:03:26.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:26.094 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:03:26.097 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:25 smithi026 ceph-mon[68309]: pgmap v633: 27 pgs: 27 active+clean; 2.8 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 160 MiB/s wr, 2.56k op/s 2024-03-20T09:03:26.194 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:03:26.233 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:26.450 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:03:26.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:26.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:03:26.467 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:03:26.563 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:03:26.594 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:26.607 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 25 16 50939 50923 127.277 119.312 0.00301829 0.00784483 2024-03-20T09:03:27.444 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:03:27.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:27.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:03:27.550 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:03:27.598 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:27.607 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 26 16 52532 52516 126.211 99.5625 0.00463807 0.00791899 2024-03-20T09:03:27.641 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:03:27.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:27.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:03:27.749 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:03:27.790 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:27.815 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:03:27.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:27.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:03:27.881 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:27 smithi115 ceph-mon[72476]: osdmap e229: 12 total, 12 up, 12 in 2024-03-20T09:03:27.881 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:27 smithi115 ceph-mon[72476]: from='client.15108 172.21.15.26:0/244410876' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]: dispatch 2024-03-20T09:03:27.882 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:27 smithi115 ceph-mon[72476]: pgmap v635: 27 pgs: 27 active+clean; 3.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 129 MiB/s wr, 2.07k op/s 2024-03-20T09:03:27.882 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:27 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3590479153' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:03:27.882 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:27 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:03:27.883 INFO:teuthology.orchestra.run.smithi026.stderr:set pool 3 pg_num to 26 2024-03-20T09:03:27.891 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:03:27.919 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:03:27.960 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:28.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:27 smithi122 ceph-mon[73495]: osdmap e229: 12 total, 12 up, 12 in 2024-03-20T09:03:28.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:27 smithi122 ceph-mon[73495]: from='client.15108 172.21.15.26:0/244410876' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]: dispatch 2024-03-20T09:03:28.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:27 smithi122 ceph-mon[73495]: pgmap v635: 27 pgs: 27 active+clean; 3.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 129 MiB/s wr, 2.07k op/s 2024-03-20T09:03:28.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:27 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3590479153' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:03:28.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:27 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:03:28.212 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:03:28.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:28.217 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:03:28.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:27 smithi026 ceph-mon[68309]: osdmap e229: 12 total, 12 up, 12 in 2024-03-20T09:03:28.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:27 smithi026 ceph-mon[68309]: from='client.15108 172.21.15.26:0/244410876' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]: dispatch 2024-03-20T09:03:28.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:27 smithi026 ceph-mon[68309]: pgmap v635: 27 pgs: 27 active+clean; 3.0 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 129 MiB/s wr, 2.07k op/s 2024-03-20T09:03:28.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:27 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3590479153' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:03:28.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:27 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:03:28.318 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:03:28.354 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:28.432 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:03:28.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:28.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:03:28.546 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:03:28.556 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:03:28.556 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:03:28.589 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:28.608 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 27 16 54271 54255 125.562 108.688 0.00827144 0.0079597 2024-03-20T09:03:28.619 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:03:28.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:28.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:03:28.726 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:03:28.790 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:28.823 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:03:28.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:28.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:03:28.881 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:28 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:03:28.882 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:28 smithi115 ceph-mon[72476]: from='client.15108 172.21.15.26:0/244410876' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]': finished 2024-03-20T09:03:28.882 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:28 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:03:28.882 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:28 smithi115 ceph-mon[72476]: osdmap e230: 12 total, 12 up, 12 in 2024-03-20T09:03:28.882 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:28 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:03:28.882 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:28 smithi122 ceph-mon[73495]: from='client.15108 172.21.15.26:0/244410876' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]': finished 2024-03-20T09:03:28.882 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:28 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:03:28.882 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:28 smithi122 ceph-mon[73495]: osdmap e230: 12 total, 12 up, 12 in 2024-03-20T09:03:28.927 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:03:28.963 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:29.045 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:03:29.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:29.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:03:29.151 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:03:29.152 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:03:29.183 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:28 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:03:29.183 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:28 smithi026 ceph-mon[68309]: from='client.15108 172.21.15.26:0/244410876' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]': finished 2024-03-20T09:03:29.183 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:28 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:03:29.183 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:28 smithi026 ceph-mon[68309]: osdmap e230: 12 total, 12 up, 12 in 2024-03-20T09:03:29.200 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:29.236 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:03:29.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:29.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:03:29.340 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:03:29.387 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:29.398 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_ops_in_flight 2024-03-20T09:03:29.420 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:03:29.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:29.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:03:29.510 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:03:29.526 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:03:29.576 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:29.604 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:03:29.608 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 28 16 55932 55916 124.785 103.812 0.00871095 0.00800948 2024-03-20T09:03:29.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:29.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:03:29.710 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:03:29.756 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:29.774 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:03:29.791 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:03:29.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:29.795 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:03:29.888 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:29 smithi115 ceph-mon[72476]: pgmap v637: 27 pgs: 27 active+clean; 3.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 124 MiB/s wr, 1.98k op/s 2024-03-20T09:03:29.888 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:29 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-20T09:03:29.895 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:03:30.010 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:29 smithi122 ceph-mon[73495]: pgmap v637: 27 pgs: 27 active+clean; 3.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 124 MiB/s wr, 1.98k op/s 2024-03-20T09:03:30.010 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:29 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-20T09:03:30.016 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:30.021 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:29 smithi026 ceph-mon[68309]: pgmap v637: 27 pgs: 27 active+clean; 3.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 124 MiB/s wr, 1.98k op/s 2024-03-20T09:03:30.021 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:29 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-20T09:03:30.053 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:03:30.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:30.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:03:30.158 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:03:30.188 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:30.286 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:03:30.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:30.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:03:30.390 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:03:30.418 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:30.610 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 29 16 56436 56420 121.569 31.5 0.052204 0.0080286 2024-03-20T09:03:30.868 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:03:30.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:30.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:03:30.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:30 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-20T09:03:30.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:30 smithi122 ceph-mon[73495]: osdmap e231: 12 total, 12 up, 12 in 2024-03-20T09:03:30.972 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:03:31.024 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:31.059 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:03:31.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:31.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:03:31.121 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:30 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-20T09:03:31.122 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:30 smithi026 ceph-mon[68309]: osdmap e231: 12 total, 12 up, 12 in 2024-03-20T09:03:31.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:30 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-20T09:03:31.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:30 smithi115 ceph-mon[72476]: osdmap e231: 12 total, 12 up, 12 in 2024-03-20T09:03:31.165 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:03:31.209 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:31.235 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:03:31.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:31.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:03:31.340 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:03:31.367 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:03:31.391 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:31.424 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:03:31.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:31.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:03:31.475 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:03:31.518 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:03:31.531 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:03:31.557 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:03:31.572 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:31.593 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:03:31.601 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:03:31.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:31.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:03:31.607 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 30 16 57899 57883 120.564 91.4375 0.00555595 0.00828993 2024-03-20T09:03:31.626 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:03:31.660 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:03:31.693 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:03:31.703 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:03:31.704 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:03:31.706 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:03:31.719 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:03:31.759 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:03:31.764 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:31.770 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:03:31.793 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:03:31.795 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:03:31.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:31.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:03:31.823 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:03:31.853 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:03:31.854 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:03:31.854 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:03:31.854 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:03:31.855 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:03:31.855 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:03:31.855 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:03:31.855 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:03:31.855 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:03:31.855 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:03:31.855 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:03:31.855 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:03:31.855 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:03:31.898 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:03:31.926 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:31.963 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:03:31.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:31.966 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:03:32.067 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:03:32.097 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:31 smithi122 ceph-mon[73495]: osdmap e232: 12 total, 12 up, 12 in 2024-03-20T09:03:32.097 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:31 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3572592551' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:03:32.097 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:31 smithi122 ceph-mon[73495]: pgmap v640: 37 pgs: 2 peering, 10 unknown, 25 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 110 MiB/s wr, 1.76k op/s 2024-03-20T09:03:32.097 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:31 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-20T09:03:32.097 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:31 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-20T09:03:32.097 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:31 smithi122 ceph-mon[73495]: osdmap e233: 12 total, 12 up, 12 in 2024-03-20T09:03:32.098 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:32.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:31 smithi115 ceph-mon[72476]: osdmap e232: 12 total, 12 up, 12 in 2024-03-20T09:03:32.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:31 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3572592551' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:03:32.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:31 smithi115 ceph-mon[72476]: pgmap v640: 37 pgs: 2 peering, 10 unknown, 25 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 110 MiB/s wr, 1.76k op/s 2024-03-20T09:03:32.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:31 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-20T09:03:32.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:31 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-20T09:03:32.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:31 smithi115 ceph-mon[72476]: osdmap e233: 12 total, 12 up, 12 in 2024-03-20T09:03:32.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:31 smithi026 ceph-mon[68309]: osdmap e232: 12 total, 12 up, 12 in 2024-03-20T09:03:32.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:31 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3572592551' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:03:32.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:31 smithi026 ceph-mon[68309]: pgmap v640: 37 pgs: 2 peering, 10 unknown, 25 active+clean; 3.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 110 MiB/s wr, 1.76k op/s 2024-03-20T09:03:32.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:31 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-20T09:03:32.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:31 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-20T09:03:32.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:31 smithi026 ceph-mon[68309]: osdmap e233: 12 total, 12 up, 12 in 2024-03-20T09:03:32.462 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_blocked_ops 2024-03-20T09:03:32.608 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 31 16 58515 58499 117.917 38.5 0.00334185 0.00828074 2024-03-20T09:03:32.718 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:03:33.447 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:03:33.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:33.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:03:33.552 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:03:33.582 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:33.608 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 32 16 58515 58499 114.232 0 - 0.00828074 2024-03-20T09:03:33.860 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-03-20T09:03:33.860 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:03:33.861 INFO:tasks.thrashosds.thrasher:Reweighting osd 8 to 0.6007040763735481 2024-03-20T09:03:33.861 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 8 0.6007040763735481 2024-03-20T09:03:33.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:33 smithi026 ceph-mon[68309]: osdmap e234: 12 total, 12 up, 12 in 2024-03-20T09:03:33.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:33 smithi026 ceph-mon[68309]: pgmap v643: 37 pgs: 2 peering, 9 unknown, 26 active+clean; 3.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s wr, 112 op/s 2024-03-20T09:03:34.058 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:03:34.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:34.063 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:03:34.158 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:33 smithi115 ceph-mon[72476]: osdmap e234: 12 total, 12 up, 12 in 2024-03-20T09:03:34.158 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:33 smithi115 ceph-mon[72476]: pgmap v643: 37 pgs: 2 peering, 9 unknown, 26 active+clean; 3.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s wr, 112 op/s 2024-03-20T09:03:34.163 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:03:34.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:33 smithi122 ceph-mon[73495]: osdmap e234: 12 total, 12 up, 12 in 2024-03-20T09:03:34.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:33 smithi122 ceph-mon[73495]: pgmap v643: 37 pgs: 2 peering, 9 unknown, 26 active+clean; 3.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s wr, 112 op/s 2024-03-20T09:03:34.231 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:34.262 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:03:34.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:34.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:03:34.338 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:03:34.372 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:03:34.439 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:34.480 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:03:34.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:34.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:03:34.586 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:03:34.608 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 33 16 59005 58989 111.699 15.3125 0.0109237 0.00894768 2024-03-20T09:03:34.634 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:34.711 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:03:34.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:34.714 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:03:34.742 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:03:34.742 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:03:34.742 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-03-20T09:03:34.743 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-03-20T09:03:34.743 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:03:34.815 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:03:34.865 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:34.916 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:03:35.064 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:03:35.111 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:34 smithi026 ceph-mon[68309]: osdmap e235: 12 total, 12 up, 12 in 2024-03-20T09:03:35.111 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:34 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1268868476' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:03:35.111 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:34 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:03:35.137 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:34 smithi122 ceph-mon[73495]: osdmap e235: 12 total, 12 up, 12 in 2024-03-20T09:03:35.137 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:34 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1268868476' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:03:35.137 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:34 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:03:35.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:34 smithi115 ceph-mon[72476]: osdmap e235: 12 total, 12 up, 12 in 2024-03-20T09:03:35.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:34 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1268868476' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:03:35.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:34 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:03:35.520 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_historic_ops 2024-03-20T09:03:35.608 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 34 16 60062 60046 110.357 66.0625 0.0330329 0.0090562 2024-03-20T09:03:35.911 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:03:35.965 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:03:35.969 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:35 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:03:35.969 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:35 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:03:35.969 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:35 smithi026 ceph-mon[68309]: osdmap e236: 12 total, 12 up, 12 in 2024-03-20T09:03:35.969 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:35 smithi026 ceph-mon[68309]: pgmap v646: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 2 unknown, 28 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s wr, 152 op/s; 7304/165663 objects degraded (4.409%); 21212/165663 objects misplaced (12.804%) 2024-03-20T09:03:35.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:35.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:03:36.071 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:03:36.105 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:36.158 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:35 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:03:36.158 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:35 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:03:36.158 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:35 smithi115 ceph-mon[72476]: osdmap e236: 12 total, 12 up, 12 in 2024-03-20T09:03:36.158 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:35 smithi115 ceph-mon[72476]: pgmap v646: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 2 unknown, 28 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s wr, 152 op/s; 7304/165663 objects degraded (4.409%); 21212/165663 objects misplaced (12.804%) 2024-03-20T09:03:36.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:35 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:03:36.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:35 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:03:36.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:35 smithi122 ceph-mon[73495]: osdmap e236: 12 total, 12 up, 12 in 2024-03-20T09:03:36.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:35 smithi122 ceph-mon[73495]: pgmap v646: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 2 unknown, 28 active+clean; 3.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s wr, 152 op/s; 7304/165663 objects degraded (4.409%); 21212/165663 objects misplaced (12.804%) 2024-03-20T09:03:36.208 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:03:36.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:36.211 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:03:36.312 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:03:36.360 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:36.402 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:03:36.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:36.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:03:36.508 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:03:36.549 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:36.608 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 35 16 60941 60925 108.773 54.9375 0.0150871 0.0091883 2024-03-20T09:03:36.634 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:03:36.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:36.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:03:36.749 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:03:36.788 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:36.913 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:03:36.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:36.917 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:03:36.919 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:36 smithi026 ceph-mon[68309]: Health check failed: Degraded data redundancy: 7304/165663 objects degraded (4.409%), 2 pgs degraded (PG_DEGRADED) 2024-03-20T09:03:36.919 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:36 smithi026 ceph-mon[68309]: osdmap e237: 12 total, 12 up, 12 in 2024-03-20T09:03:37.018 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:03:37.047 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:36 smithi122 ceph-mon[73495]: Health check failed: Degraded data redundancy: 7304/165663 objects degraded (4.409%), 2 pgs degraded (PG_DEGRADED) 2024-03-20T09:03:37.048 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:36 smithi122 ceph-mon[73495]: osdmap e237: 12 total, 12 up, 12 in 2024-03-20T09:03:37.049 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:37.190 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:03:37.202 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:03:37.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:37.206 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:03:37.307 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:03:37.338 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:37.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:36 smithi115 ceph-mon[72476]: Health check failed: Degraded data redundancy: 7304/165663 objects degraded (4.409%), 2 pgs degraded (PG_DEGRADED) 2024-03-20T09:03:37.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:36 smithi115 ceph-mon[72476]: osdmap e237: 12 total, 12 up, 12 in 2024-03-20T09:03:37.544 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:03:37.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:37.549 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:03:37.608 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 36 16 61953 61937 107.509 63.25 0.0137979 0.00929657 2024-03-20T09:03:37.650 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:03:37.685 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:37.717 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:03:37.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:37.719 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:03:37.820 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:03:37.869 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:37.885 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:03:37.885 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-03-20T09:03:37.885 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-03-20T09:03:37.885 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-03-20T09:03:37.885 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.885 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.24550.0:1360 2.0 e125/121 2:06f96393:::benchmark_data_smithi122_107654_object1359:head v 125'77, mlcod=125'75)", 2024-03-20T09:03:37.885 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:58:24.752475+0000", 2024-03-20T09:03:37.885 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 313.12922410700003, 2024-03-20T09:03:37.885 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.030003946, 2024-03-20T09:03:37.885 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:37.886 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:37.886 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:37.886 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.886 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:37.886 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.752475+0000", 2024-03-20T09:03:37.886 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:37.886 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.886 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.886 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:37.886 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.752475+0000", 2024-03-20T09:03:37.886 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:37.886 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.886 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.887 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:37.887 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.752475+0000", 2024-03-20T09:03:37.887 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7999999999999995e-07 2024-03-20T09:03:37.887 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.887 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.887 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:37.887 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.752548+0000", 2024-03-20T09:03:37.887 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2780000000000005e-05 2024-03-20T09:03:37.887 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.887 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.887 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:37.887 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.752561+0000", 2024-03-20T09:03:37.887 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2313000000000001e-05 2024-03-20T09:03:37.888 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.888 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.888 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:37.888 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.752567+0000", 2024-03-20T09:03:37.888 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8199999999999999e-06 2024-03-20T09:03:37.888 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.888 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.888 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:37.888 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.781575+0000", 2024-03-20T09:03:37.888 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.029007380999999999 2024-03-20T09:03:37.888 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.888 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.888 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:37.889 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.781597+0000", 2024-03-20T09:03:37.889 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2206999999999999e-05 2024-03-20T09:03:37.889 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.889 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.889 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:37.889 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.782436+0000", 2024-03-20T09:03:37.889 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00083926599999999997 2024-03-20T09:03:37.889 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.889 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.889 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:37.889 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.782479+0000", 2024-03-20T09:03:37.889 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2599000000000003e-05 2024-03-20T09:03:37.889 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:37.889 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:37.890 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:37.890 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.890 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.890 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.24550.0:1361 2.7 e125/121 2:e2795900:::benchmark_data_smithi122_107654_object1360:head v 125'87, mlcod=125'84)", 2024-03-20T09:03:37.890 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:58:24.752693+0000", 2024-03-20T09:03:37.890 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 313.12900647100003, 2024-03-20T09:03:37.890 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.02981021, 2024-03-20T09:03:37.890 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:37.890 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:37.890 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:37.890 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.890 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:37.891 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.752693+0000", 2024-03-20T09:03:37.891 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:37.891 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.891 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.891 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:37.891 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.752693+0000", 2024-03-20T09:03:37.891 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:37.891 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.891 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.891 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:37.891 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.752693+0000", 2024-03-20T09:03:37.891 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3900000000000004e-07 2024-03-20T09:03:37.891 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.891 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.892 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:37.892 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.752789+0000", 2024-03-20T09:03:37.892 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5447999999999998e-05 2024-03-20T09:03:37.892 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.892 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.892 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:37.892 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.781602+0000", 2024-03-20T09:03:37.892 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.028812963 2024-03-20T09:03:37.892 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.892 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.892 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:37.892 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.781613+0000", 2024-03-20T09:03:37.892 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1681e-05 2024-03-20T09:03:37.892 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.893 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.893 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:37.893 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.781663+0000", 2024-03-20T09:03:37.893 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0136000000000002e-05 2024-03-20T09:03:37.893 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.893 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.893 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:37.893 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.781684+0000", 2024-03-20T09:03:37.893 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0860999999999999e-05 2024-03-20T09:03:37.893 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.893 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.893 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:37.894 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.782468+0000", 2024-03-20T09:03:37.894 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00078375700000000005 2024-03-20T09:03:37.894 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.894 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.894 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:37.894 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.782503+0000", 2024-03-20T09:03:37.894 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4724999999999997e-05 2024-03-20T09:03:37.894 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:37.894 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:37.894 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:37.894 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.894 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.894 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.24550.0:1364 2.0 e125/121 2:04548208:::benchmark_data_smithi122_107654_object1363:head v 125'78, mlcod=125'75)", 2024-03-20T09:03:37.895 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:58:24.753476+0000", 2024-03-20T09:03:37.895 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 313.12822327200001, 2024-03-20T09:03:37.895 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.029272663000000001, 2024-03-20T09:03:37.895 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:37.895 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:37.895 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:37.895 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.895 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:37.895 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.753476+0000", 2024-03-20T09:03:37.895 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:37.895 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.895 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.895 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:37.896 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.753476+0000", 2024-03-20T09:03:37.896 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:37.896 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.896 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.896 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:37.896 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.753476+0000", 2024-03-20T09:03:37.896 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8299999999999996e-07 2024-03-20T09:03:37.896 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.896 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.896 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:37.896 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.753504+0000", 2024-03-20T09:03:37.896 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7324000000000001e-05 2024-03-20T09:03:37.896 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.896 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.897 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:37.897 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.781569+0000", 2024-03-20T09:03:37.897 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.028065203 2024-03-20T09:03:37.897 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.897 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.897 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:37.897 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.781582+0000", 2024-03-20T09:03:37.897 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3142999999999999e-05 2024-03-20T09:03:37.897 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.897 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.897 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:37.897 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.781814+0000", 2024-03-20T09:03:37.897 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00023215000000000001 2024-03-20T09:03:37.897 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.897 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.898 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:37.898 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.781827+0000", 2024-03-20T09:03:37.898 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2493999999999999e-05 2024-03-20T09:03:37.898 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.898 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.898 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:37.898 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.782714+0000", 2024-03-20T09:03:37.898 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00088687 2024-03-20T09:03:37.898 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.898 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.898 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:37.898 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:58:24.782748+0000", 2024-03-20T09:03:37.898 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4696000000000001e-05 2024-03-20T09:03:37.899 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:37.899 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:37.899 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:37.899 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.899 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.899 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.24550.0:135139 2.0 e154/121 2:0b0a0cda:::benchmark_data_smithi122_107654_object16354:head v 154'8345, mlcod=154'8342)", 2024-03-20T09:03:37.899 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:00:11.288850+0000", 2024-03-20T09:03:37.899 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 206.59284913799999, 2024-03-20T09:03:37.899 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.034181531000000001, 2024-03-20T09:03:37.899 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:37.899 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:37.900 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:37.900 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.900 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:37.900 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.288850+0000", 2024-03-20T09:03:37.900 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:37.900 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.900 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.900 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:37.900 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.288850+0000", 2024-03-20T09:03:37.900 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:37.900 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.900 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.901 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:37.901 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.288851+0000", 2024-03-20T09:03:37.901 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1900000000000002e-07 2024-03-20T09:03:37.901 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.901 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.901 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:37.901 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.288857+0000", 2024-03-20T09:03:37.901 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1430000000000003e-06 2024-03-20T09:03:37.901 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.901 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.901 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:37.901 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.288870+0000", 2024-03-20T09:03:37.902 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3397000000000001e-05 2024-03-20T09:03:37.902 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.902 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.902 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:37.902 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.288877+0000", 2024-03-20T09:03:37.902 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4690000000000002e-06 2024-03-20T09:03:37.902 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.902 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.903 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:37.903 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.300389+0000", 2024-03-20T09:03:37.903 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011512302 2024-03-20T09:03:37.903 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.903 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.903 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:37.903 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.300413+0000", 2024-03-20T09:03:37.903 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4578000000000001e-05 2024-03-20T09:03:37.903 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.903 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.903 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:37.903 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.322981+0000", 2024-03-20T09:03:37.903 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.022567907000000002 2024-03-20T09:03:37.904 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.904 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.904 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:37.904 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:11.323031+0000", 2024-03-20T09:03:37.904 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0015999999999997e-05 2024-03-20T09:03:37.904 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:37.904 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:37.904 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:37.904 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.904 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.904 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.24550.0:206170 2.0 e184/121 2:023ee901:::benchmark_data_smithi122_107654_object87385:head v 184'12729, mlcod=184'12727)", 2024-03-20T09:03:37.904 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:01:56.432630+0000", 2024-03-20T09:03:37.905 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 101.449069157, 2024-03-20T09:03:37.905 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.029929188999999998, 2024-03-20T09:03:37.905 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:37.905 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:37.905 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:37.905 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.905 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:37.905 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:56.432630+0000", 2024-03-20T09:03:37.906 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:37.906 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.906 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.906 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:37.906 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:56.432630+0000", 2024-03-20T09:03:37.906 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:37.906 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.906 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.906 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:37.906 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:56.432631+0000", 2024-03-20T09:03:37.906 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.8400000000000002e-07 2024-03-20T09:03:37.906 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.906 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.907 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:37.907 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:56.432637+0000", 2024-03-20T09:03:37.907 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5699999999999998e-06 2024-03-20T09:03:37.907 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.907 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.907 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:37.907 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:56.432650+0000", 2024-03-20T09:03:37.907 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.279e-05 2024-03-20T09:03:37.907 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.907 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.907 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:37.907 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:56.432657+0000", 2024-03-20T09:03:37.907 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3339999999999996e-06 2024-03-20T09:03:37.907 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.908 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.908 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:37.908 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:56.435335+0000", 2024-03-20T09:03:37.908 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0026776019999999998 2024-03-20T09:03:37.908 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.908 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.908 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:37.908 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:56.435356+0000", 2024-03-20T09:03:37.908 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1180999999999999e-05 2024-03-20T09:03:37.908 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.908 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.908 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:37.909 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:56.462515+0000", 2024-03-20T09:03:37.909 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.027158808999999999 2024-03-20T09:03:37.909 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.909 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.909 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:37.909 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:56.462559+0000", 2024-03-20T09:03:37.909 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3918999999999997e-05 2024-03-20T09:03:37.909 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:37.909 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:37.909 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:37.909 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.909 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.909 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:222462 2.0 2:02798eea:::benchmark_data_smithi122_107654_object103677:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-03-20T09:03:37.909 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:02:15.796002+0000", 2024-03-20T09:03:37.910 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 82.085696475000006, 2024-03-20T09:03:37.910 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035486458999999998, 2024-03-20T09:03:37.910 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:37.910 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:37.910 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:37.910 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:03:37.910 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:03:37.910 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 222462 2024-03-20T09:03:37.910 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.910 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:37.910 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.910 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:37.911 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.796002+0000", 2024-03-20T09:03:37.911 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:37.911 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.911 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.911 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:37.911 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.796002+0000", 2024-03-20T09:03:37.911 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:37.911 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.911 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.911 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:37.911 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.796003+0000", 2024-03-20T09:03:37.911 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5499999999999998e-07 2024-03-20T09:03:37.911 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.911 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.912 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:37.912 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.796006+0000", 2024-03-20T09:03:37.912 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.016e-06 2024-03-20T09:03:37.912 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.912 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.912 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:37.912 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.796013+0000", 2024-03-20T09:03:37.912 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4059999999999999e-06 2024-03-20T09:03:37.912 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.912 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.912 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:37.912 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.796019+0000", 2024-03-20T09:03:37.912 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7610000000000004e-06 2024-03-20T09:03:37.912 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.912 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.912 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:37.913 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.799853+0000", 2024-03-20T09:03:37.913 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0038338930000000001 2024-03-20T09:03:37.913 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.913 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.913 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:37.913 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.799982+0000", 2024-03-20T09:03:37.913 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012874699999999999 2024-03-20T09:03:37.913 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.913 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.913 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:03:37.913 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.800112+0000", 2024-03-20T09:03:37.913 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000130099 2024-03-20T09:03:37.913 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.913 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.913 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:37.913 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.808146+0000", 2024-03-20T09:03:37.914 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0080337759999999994 2024-03-20T09:03:37.914 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.914 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.914 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:37.914 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.813019+0000", 2024-03-20T09:03:37.914 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0048727900000000001 2024-03-20T09:03:37.914 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.914 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.914 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:37.914 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.831382+0000", 2024-03-20T09:03:37.914 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018363121 2024-03-20T09:03:37.914 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.915 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.915 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:37.915 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.831442+0000", 2024-03-20T09:03:37.915 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0315000000000001e-05 2024-03-20T09:03:37.915 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.915 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.915 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:37.915 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.831489+0000", 2024-03-20T09:03:37.915 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6980000000000001e-05 2024-03-20T09:03:37.915 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:37.915 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:37.915 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:37.916 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.916 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.916 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:222466 2.0 2:0549dde6:::benchmark_data_smithi122_107654_object103681:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-03-20T09:03:37.916 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:02:15.804025+0000", 2024-03-20T09:03:37.916 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 82.077674189000007, 2024-03-20T09:03:37.916 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.027556961000000001, 2024-03-20T09:03:37.916 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:37.916 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:37.916 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:37.916 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:03:37.916 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:03:37.916 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 222466 2024-03-20T09:03:37.916 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.916 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:37.917 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.917 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:37.917 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.804025+0000", 2024-03-20T09:03:37.917 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:37.917 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.917 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.917 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:37.917 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.804025+0000", 2024-03-20T09:03:37.917 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:37.917 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.917 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.917 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:37.917 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.804026+0000", 2024-03-20T09:03:37.918 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.384e-06 2024-03-20T09:03:37.918 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.918 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.918 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:37.918 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.804032+0000", 2024-03-20T09:03:37.918 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7579999999999998e-06 2024-03-20T09:03:37.918 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.918 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.918 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:37.918 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.804047+0000", 2024-03-20T09:03:37.918 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5081e-05 2024-03-20T09:03:37.918 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.918 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.918 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:37.919 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.804056+0000", 2024-03-20T09:03:37.919 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2480000000000002e-06 2024-03-20T09:03:37.919 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.919 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.919 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:37.919 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.804261+0000", 2024-03-20T09:03:37.919 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00020518399999999999 2024-03-20T09:03:37.919 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.919 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.919 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:37.919 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.804380+0000", 2024-03-20T09:03:37.919 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000118595 2024-03-20T09:03:37.919 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.919 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.919 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:03:37.919 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.804505+0000", 2024-03-20T09:03:37.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000124578 2024-03-20T09:03:37.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.920 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.920 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:37.920 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.812987+0000", 2024-03-20T09:03:37.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0084828109999999998 2024-03-20T09:03:37.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.920 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.920 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:37.920 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.813053+0000", 2024-03-20T09:03:37.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5596e-05 2024-03-20T09:03:37.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.920 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.920 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:37.921 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.831528+0000", 2024-03-20T09:03:37.921 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018475017999999999 2024-03-20T09:03:37.921 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.921 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.921 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:37.921 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.831561+0000", 2024-03-20T09:03:37.921 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3194000000000001e-05 2024-03-20T09:03:37.921 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.921 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.921 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:37.921 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.831582+0000", 2024-03-20T09:03:37.921 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0514e-05 2024-03-20T09:03:37.921 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:37.921 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:37.922 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:37.922 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.922 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.922 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:222469 2.0 2:097c581c:::benchmark_data_smithi122_107654_object103684:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-03-20T09:03:37.922 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:02:15.804081+0000", 2024-03-20T09:03:37.922 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 82.077617798999995, 2024-03-20T09:03:37.922 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.032435215000000003, 2024-03-20T09:03:37.922 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:37.922 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:37.922 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:37.922 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:03:37.922 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:03:37.922 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 222469 2024-03-20T09:03:37.923 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.923 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:37.923 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.923 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:37.923 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.804081+0000", 2024-03-20T09:03:37.923 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:37.923 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.923 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.923 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:37.923 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.804081+0000", 2024-03-20T09:03:37.923 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:37.923 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.923 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.923 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:37.923 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.804082+0000", 2024-03-20T09:03:37.924 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0900000000000002e-07 2024-03-20T09:03:37.924 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.924 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.924 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:37.924 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.804085+0000", 2024-03-20T09:03:37.924 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2179999999999999e-06 2024-03-20T09:03:37.924 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.924 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.924 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:37.924 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.804092+0000", 2024-03-20T09:03:37.924 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9689999999999997e-06 2024-03-20T09:03:37.924 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.924 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.924 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:37.924 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.804097+0000", 2024-03-20T09:03:37.924 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1309999999999997e-06 2024-03-20T09:03:37.925 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.925 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.925 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:37.925 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.808182+0000", 2024-03-20T09:03:37.925 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040851009999999998 2024-03-20T09:03:37.925 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.925 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.925 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:37.925 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.808316+0000", 2024-03-20T09:03:37.925 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013360499999999999 2024-03-20T09:03:37.925 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.925 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:03:37.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.808452+0000", 2024-03-20T09:03:37.926 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013671500000000001 2024-03-20T09:03:37.926 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.926 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:37.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.813093+0000", 2024-03-20T09:03:37.926 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0046406700000000004 2024-03-20T09:03:37.926 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.926 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:37.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.814755+0000", 2024-03-20T09:03:37.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0016617229999999999 2024-03-20T09:03:37.927 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.927 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.927 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:37.927 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.836407+0000", 2024-03-20T09:03:37.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.021651987000000001 2024-03-20T09:03:37.927 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.927 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.927 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:37.927 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.836468+0000", 2024-03-20T09:03:37.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0982e-05 2024-03-20T09:03:37.927 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.927 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.927 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:37.927 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:15.836516+0000", 2024-03-20T09:03:37.928 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8605e-05 2024-03-20T09:03:37.928 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:37.928 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:37.928 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:37.928 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.928 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.928 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:222979 2.0 2:0ef9c1ca:::benchmark_data_smithi122_107654_object104194:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-03-20T09:03:37.928 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:02:16.406601+0000", 2024-03-20T09:03:37.928 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 81.475097794999996, 2024-03-20T09:03:37.928 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.030039033999999999, 2024-03-20T09:03:37.928 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:37.928 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:37.928 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:37.929 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:03:37.929 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:03:37.929 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 222979 2024-03-20T09:03:37.929 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.929 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:37.929 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.929 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:37.929 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:16.406601+0000", 2024-03-20T09:03:37.929 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:37.929 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.930 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.930 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:37.930 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:16.406601+0000", 2024-03-20T09:03:37.930 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:37.930 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.930 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.930 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:37.930 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:16.406602+0000", 2024-03-20T09:03:37.930 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5400000000000001e-07 2024-03-20T09:03:37.930 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.930 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.930 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:37.930 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:16.406605+0000", 2024-03-20T09:03:37.931 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2509999999999999e-06 2024-03-20T09:03:37.931 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.931 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.931 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:37.931 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:16.406614+0000", 2024-03-20T09:03:37.931 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4859999999999997e-06 2024-03-20T09:03:37.931 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.931 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.931 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:37.931 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:16.406619+0000", 2024-03-20T09:03:37.931 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8130000000000001e-06 2024-03-20T09:03:37.931 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.931 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.931 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:37.931 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:16.411090+0000", 2024-03-20T09:03:37.932 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0044703659999999999 2024-03-20T09:03:37.932 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.932 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.932 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:37.932 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:16.411221+0000", 2024-03-20T09:03:37.932 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013126899999999999 2024-03-20T09:03:37.932 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.932 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.932 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:03:37.932 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:16.411332+0000", 2024-03-20T09:03:37.932 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000111227 2024-03-20T09:03:37.932 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.932 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:37.933 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:16.415560+0000", 2024-03-20T09:03:37.933 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0042274829999999998 2024-03-20T09:03:37.933 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.933 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:37.933 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:16.415870+0000", 2024-03-20T09:03:37.933 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00031056299999999997 2024-03-20T09:03:37.933 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.933 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:37.933 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:16.436524+0000", 2024-03-20T09:03:37.933 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.020654045999999999 2024-03-20T09:03:37.933 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.934 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.934 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:37.934 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:16.436586+0000", 2024-03-20T09:03:37.934 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1216000000000005e-05 2024-03-20T09:03:37.934 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.934 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.934 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:37.934 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:16.436640+0000", 2024-03-20T09:03:37.934 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4660000000000002e-05 2024-03-20T09:03:37.934 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:37.934 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:37.934 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:37.934 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.935 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.935 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:224489 2.0 2:00ab959f:::benchmark_data_smithi122_107654_object105704:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-03-20T09:03:37.935 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:02:18.191514+0000", 2024-03-20T09:03:37.935 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 79.690184821000003, 2024-03-20T09:03:37.935 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.028073587000000001, 2024-03-20T09:03:37.935 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:37.935 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:37.935 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:37.935 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:03:37.935 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:03:37.935 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 224489 2024-03-20T09:03:37.936 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.936 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:37.936 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.936 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:37.936 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.191514+0000", 2024-03-20T09:03:37.936 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:37.936 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.936 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.936 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:37.936 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.191514+0000", 2024-03-20T09:03:37.936 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:37.936 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.936 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.937 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:37.937 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.191515+0000", 2024-03-20T09:03:37.937 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.33e-06 2024-03-20T09:03:37.937 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.937 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.937 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:37.937 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.191519+0000", 2024-03-20T09:03:37.937 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7060000000000002e-06 2024-03-20T09:03:37.937 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.937 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.937 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:37.937 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.191529+0000", 2024-03-20T09:03:37.937 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.8980000000000008e-06 2024-03-20T09:03:37.937 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.938 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.938 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:37.938 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.191535+0000", 2024-03-20T09:03:37.938 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8950000000000003e-06 2024-03-20T09:03:37.938 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.938 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.938 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:37.938 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.191637+0000", 2024-03-20T09:03:37.938 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000101587 2024-03-20T09:03:37.938 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.938 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.938 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:37.938 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.191769+0000", 2024-03-20T09:03:37.939 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000132461 2024-03-20T09:03:37.939 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.939 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.939 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:03:37.939 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.191890+0000", 2024-03-20T09:03:37.939 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000120759 2024-03-20T09:03:37.939 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.939 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.939 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:37.939 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.202453+0000", 2024-03-20T09:03:37.939 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010563359 2024-03-20T09:03:37.939 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.939 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.939 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:37.940 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.202468+0000", 2024-03-20T09:03:37.940 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5322e-05 2024-03-20T09:03:37.940 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.940 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.940 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:37.940 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.219521+0000", 2024-03-20T09:03:37.940 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017052706000000001 2024-03-20T09:03:37.940 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.940 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.940 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:37.940 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.219558+0000", 2024-03-20T09:03:37.940 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7224000000000001e-05 2024-03-20T09:03:37.940 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.940 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.940 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:37.941 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.219588+0000", 2024-03-20T09:03:37.941 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.934e-05 2024-03-20T09:03:37.941 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:37.941 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:37.941 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:37.941 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.941 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.941 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:224773 2.0 2:0687dd70:::benchmark_data_smithi122_107654_object105988:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-03-20T09:03:37.941 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:02:18.512719+0000", 2024-03-20T09:03:37.941 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 79.368979671000005, 2024-03-20T09:03:37.941 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.028305433000000001, 2024-03-20T09:03:37.941 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:37.941 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:37.941 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:37.942 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:03:37.942 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:03:37.942 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 224773 2024-03-20T09:03:37.942 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.942 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:37.942 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.942 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:37.942 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.512719+0000", 2024-03-20T09:03:37.942 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:37.942 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.942 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.942 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:37.942 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.512719+0000", 2024-03-20T09:03:37.943 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:37.943 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.943 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.943 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:37.943 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.512721+0000", 2024-03-20T09:03:37.943 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8819999999999999e-06 2024-03-20T09:03:37.943 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.943 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.943 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:37.943 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.512729+0000", 2024-03-20T09:03:37.943 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9200000000000004e-06 2024-03-20T09:03:37.943 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.943 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.943 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:37.944 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.512749+0000", 2024-03-20T09:03:37.944 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9894999999999998e-05 2024-03-20T09:03:37.944 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.944 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.944 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:37.944 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.512761+0000", 2024-03-20T09:03:37.944 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1592e-05 2024-03-20T09:03:37.944 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.944 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.944 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:37.944 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.512826+0000", 2024-03-20T09:03:37.944 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5757000000000001e-05 2024-03-20T09:03:37.944 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.944 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.944 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:37.944 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.512953+0000", 2024-03-20T09:03:37.945 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000126742 2024-03-20T09:03:37.945 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.945 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.945 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:03:37.945 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.513048+0000", 2024-03-20T09:03:37.945 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4906000000000002e-05 2024-03-20T09:03:37.945 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.945 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.945 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:37.945 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.516529+0000", 2024-03-20T09:03:37.945 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0034813909999999999 2024-03-20T09:03:37.945 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.945 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.945 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:37.945 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.521180+0000", 2024-03-20T09:03:37.946 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0046503339999999999 2024-03-20T09:03:37.946 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.946 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.946 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:37.946 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.540922+0000", 2024-03-20T09:03:37.946 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.019741986999999999 2024-03-20T09:03:37.946 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.946 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.946 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:37.946 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.540982+0000", 2024-03-20T09:03:37.946 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9880000000000003e-05 2024-03-20T09:03:37.946 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.946 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.947 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:37.947 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.541025+0000", 2024-03-20T09:03:37.947 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3146999999999999e-05 2024-03-20T09:03:37.947 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:37.947 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:37.947 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:37.947 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.947 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.947 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:225052 2.0 2:0a2c87e2:::benchmark_data_smithi122_107654_object106267:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-03-20T09:03:37.947 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:02:18.838769+0000", 2024-03-20T09:03:37.947 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 79.042930365999993, 2024-03-20T09:03:37.947 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.028886377000000001, 2024-03-20T09:03:37.947 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:37.947 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:37.947 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:37.948 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:03:37.948 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:03:37.948 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 225052 2024-03-20T09:03:37.948 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.948 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:37.948 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.948 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:37.948 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.838769+0000", 2024-03-20T09:03:37.948 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:37.948 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.948 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.948 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:37.948 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.838769+0000", 2024-03-20T09:03:37.948 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:37.949 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.949 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.949 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:37.949 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.838769+0000", 2024-03-20T09:03:37.949 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0299999999999998e-07 2024-03-20T09:03:37.949 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.949 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.949 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:37.949 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.838776+0000", 2024-03-20T09:03:37.949 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8070000000000002e-06 2024-03-20T09:03:37.949 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.949 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.949 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:37.949 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.838784+0000", 2024-03-20T09:03:37.950 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8040000000000002e-06 2024-03-20T09:03:37.950 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.950 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.950 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:37.950 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.838789+0000", 2024-03-20T09:03:37.950 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2850000000000002e-06 2024-03-20T09:03:37.950 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.950 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.950 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:37.950 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.845766+0000", 2024-03-20T09:03:37.950 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0069770459999999998 2024-03-20T09:03:37.950 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.950 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.951 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:37.951 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.845913+0000", 2024-03-20T09:03:37.951 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014707000000000001 2024-03-20T09:03:37.951 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.951 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.951 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:03:37.951 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.846041+0000", 2024-03-20T09:03:37.951 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000127105 2024-03-20T09:03:37.951 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.951 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.951 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:37.951 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.858246+0000", 2024-03-20T09:03:37.952 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.01220544 2024-03-20T09:03:37.952 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.952 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.952 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:37.952 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.858309+0000", 2024-03-20T09:03:37.952 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3215999999999999e-05 2024-03-20T09:03:37.952 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.952 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.952 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:37.952 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.867550+0000", 2024-03-20T09:03:37.952 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0092404919999999995 2024-03-20T09:03:37.952 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.952 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.953 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:37.953 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.867606+0000", 2024-03-20T09:03:37.953 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6817000000000002e-05 2024-03-20T09:03:37.953 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.953 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.953 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:37.953 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:18.867655+0000", 2024-03-20T09:03:37.953 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8492000000000003e-05 2024-03-20T09:03:37.953 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:37.953 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:37.953 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:37.953 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.953 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.953 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:236966 2.0 2:0f597f71:::benchmark_data_smithi122_107654_object118181:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-03-20T09:03:37.954 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:02:32.619454+0000", 2024-03-20T09:03:37.954 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 65.262244956999993, 2024-03-20T09:03:37.954 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.033299764000000003, 2024-03-20T09:03:37.954 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:37.954 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:37.954 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:37.954 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:03:37.954 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:03:37.954 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 236966 2024-03-20T09:03:37.954 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.954 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:37.954 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.955 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:37.955 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.619454+0000", 2024-03-20T09:03:37.955 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:37.955 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.955 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.955 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:37.955 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.619454+0000", 2024-03-20T09:03:37.955 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:37.955 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.955 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.955 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:37.955 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.619455+0000", 2024-03-20T09:03:37.955 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3069999999999999e-06 2024-03-20T09:03:37.955 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.955 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.956 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:37.956 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.619461+0000", 2024-03-20T09:03:37.956 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5230000000000003e-06 2024-03-20T09:03:37.956 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.956 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.956 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:37.956 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.619474+0000", 2024-03-20T09:03:37.956 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2852e-05 2024-03-20T09:03:37.956 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.956 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.956 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:37.956 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.619482+0000", 2024-03-20T09:03:37.956 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4509999999999992e-06 2024-03-20T09:03:37.956 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.957 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.957 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:37.957 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.621795+0000", 2024-03-20T09:03:37.957 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0023133350000000001 2024-03-20T09:03:37.957 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.957 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.957 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:37.957 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.621873+0000", 2024-03-20T09:03:37.957 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7892000000000006e-05 2024-03-20T09:03:37.957 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.957 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.957 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:03:37.957 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.621991+0000", 2024-03-20T09:03:37.957 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000117868 2024-03-20T09:03:37.958 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.958 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.958 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:37.958 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.626393+0000", 2024-03-20T09:03:37.958 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0044020500000000002 2024-03-20T09:03:37.958 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.958 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.958 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:37.958 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.628455+0000", 2024-03-20T09:03:37.958 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0020614309999999999 2024-03-20T09:03:37.958 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.958 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.959 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:37.959 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.652650+0000", 2024-03-20T09:03:37.959 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.024195357000000001 2024-03-20T09:03:37.959 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.959 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.959 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:37.959 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.652708+0000", 2024-03-20T09:03:37.959 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7797999999999998e-05 2024-03-20T09:03:37.959 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.959 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.959 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:37.959 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.652754+0000", 2024-03-20T09:03:37.959 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5899999999999998e-05 2024-03-20T09:03:37.959 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:37.960 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:37.960 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:37.960 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.960 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.960 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:237019 2.0 2:051a62d4:::benchmark_data_smithi122_107654_object118234:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-03-20T09:03:37.960 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:02:32.675552+0000", 2024-03-20T09:03:37.960 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 65.206146955999998, 2024-03-20T09:03:37.960 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.027085027000000001, 2024-03-20T09:03:37.960 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:37.960 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:37.960 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:37.960 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:03:37.960 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:03:37.960 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 237019 2024-03-20T09:03:37.960 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.961 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:37.961 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.961 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:37.961 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.675552+0000", 2024-03-20T09:03:37.961 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:37.961 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.961 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.961 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:37.961 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.675552+0000", 2024-03-20T09:03:37.961 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:37.961 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.961 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.961 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:37.961 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.675554+0000", 2024-03-20T09:03:37.961 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.601e-06 2024-03-20T09:03:37.962 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.962 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.962 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:37.962 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.675561+0000", 2024-03-20T09:03:37.962 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3939999999999998e-06 2024-03-20T09:03:37.962 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.962 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.962 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:37.962 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.675578+0000", 2024-03-20T09:03:37.962 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6824999999999999e-05 2024-03-20T09:03:37.962 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.962 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.962 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:37.962 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.675587+0000", 2024-03-20T09:03:37.963 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3889999999999994e-06 2024-03-20T09:03:37.963 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.963 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.963 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:37.963 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.675647+0000", 2024-03-20T09:03:37.963 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9988000000000002e-05 2024-03-20T09:03:37.963 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.963 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.963 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:37.963 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.675781+0000", 2024-03-20T09:03:37.963 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013360499999999999 2024-03-20T09:03:37.963 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.963 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.963 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:03:37.964 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.675913+0000", 2024-03-20T09:03:37.964 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000131848 2024-03-20T09:03:37.964 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.964 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.964 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:37.964 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.688640+0000", 2024-03-20T09:03:37.964 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012727581 2024-03-20T09:03:37.964 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.964 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.964 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:37.964 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.688669+0000", 2024-03-20T09:03:37.964 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8753000000000001e-05 2024-03-20T09:03:37.964 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.964 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.965 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:37.965 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.702532+0000", 2024-03-20T09:03:37.965 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013862535 2024-03-20T09:03:37.965 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.965 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.965 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:37.965 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.702590+0000", 2024-03-20T09:03:37.965 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8396000000000001e-05 2024-03-20T09:03:37.965 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.965 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.965 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:37.965 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.702637+0000", 2024-03-20T09:03:37.965 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7111999999999999e-05 2024-03-20T09:03:37.965 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:37.966 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:37.966 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:37.966 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.966 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.966 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:237021 2.0 2:0dda45d3:::benchmark_data_smithi122_107654_object118236:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-03-20T09:03:37.966 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:02:32.675690+0000", 2024-03-20T09:03:37.966 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 65.206009242999997, 2024-03-20T09:03:37.966 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.027186926, 2024-03-20T09:03:37.966 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:37.966 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:37.966 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:37.966 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:03:37.966 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:03:37.967 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 237021 2024-03-20T09:03:37.967 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.967 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:37.967 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.967 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:37.967 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.675690+0000", 2024-03-20T09:03:37.967 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:37.967 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.967 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.967 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:37.967 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.675690+0000", 2024-03-20T09:03:37.967 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:37.967 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.967 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.968 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:37.968 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.675691+0000", 2024-03-20T09:03:37.968 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7100000000000001e-07 2024-03-20T09:03:37.968 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.968 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.968 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:37.968 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.675694+0000", 2024-03-20T09:03:37.968 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6789999999999998e-06 2024-03-20T09:03:37.968 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.968 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.969 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:37.969 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.675703+0000", 2024-03-20T09:03:37.969 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.935e-06 2024-03-20T09:03:37.969 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.969 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.969 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:37.969 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.675709+0000", 2024-03-20T09:03:37.969 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0950000000000001e-06 2024-03-20T09:03:37.969 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.969 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.969 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:37.969 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.679605+0000", 2024-03-20T09:03:37.969 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003895426 2024-03-20T09:03:37.969 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.970 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.970 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:37.970 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.679684+0000", 2024-03-20T09:03:37.970 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9072999999999993e-05 2024-03-20T09:03:37.970 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.970 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.970 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:03:37.970 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.679759+0000", 2024-03-20T09:03:37.970 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4813000000000004e-05 2024-03-20T09:03:37.970 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.974 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.974 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:37.974 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.688677+0000", 2024-03-20T09:03:37.974 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0089188380000000001 2024-03-20T09:03:37.974 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.974 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.974 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:37.974 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.688705+0000", 2024-03-20T09:03:37.974 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7826000000000001e-05 2024-03-20T09:03:37.974 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.974 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.974 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:37.975 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.702783+0000", 2024-03-20T09:03:37.975 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014077532 2024-03-20T09:03:37.975 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.975 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.975 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:37.975 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.702838+0000", 2024-03-20T09:03:37.975 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5608e-05 2024-03-20T09:03:37.975 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.975 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.975 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:37.975 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:02:32.702877+0000", 2024-03-20T09:03:37.975 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8330000000000001e-05 2024-03-20T09:03:37.975 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:37.975 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:37.976 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:37.976 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.976 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.976 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:58468 3.11 3:8a579397:::benchmark_data_smithi122_119671_object58467:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e233)", 2024-03-20T09:03:37.976 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:03:31.877406+0000", 2024-03-20T09:03:37.976 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 6.0042930439999997, 2024-03-20T09:03:37.976 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1325015989999998, 2024-03-20T09:03:37.976 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:37.976 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:03:37.976 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:37.976 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:03:37.976 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:03:37.976 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 58468 2024-03-20T09:03:37.976 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.977 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:37.977 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.977 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:37.977 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:31.877406+0000", 2024-03-20T09:03:37.977 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:37.977 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.977 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.977 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:37.977 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:31.877406+0000", 2024-03-20T09:03:37.977 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:37.977 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.977 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.977 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:37.978 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:31.877407+0000", 2024-03-20T09:03:37.978 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2410000000000001e-06 2024-03-20T09:03:37.978 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.978 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.978 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:37.978 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:31.877450+0000", 2024-03-20T09:03:37.978 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2409e-05 2024-03-20T09:03:37.978 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.978 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.978 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:37.978 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:31.877464+0000", 2024-03-20T09:03:37.978 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4269e-05 2024-03-20T09:03:37.979 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.979 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.979 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:37.979 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:31.877480+0000", 2024-03-20T09:03:37.979 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6087999999999999e-05 2024-03-20T09:03:37.979 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.979 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.979 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:37.979 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:31.877537+0000", 2024-03-20T09:03:37.979 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7481000000000001e-05 2024-03-20T09:03:37.979 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.980 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.980 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-03-20T09:03:37.980 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:31.877545+0000", 2024-03-20T09:03:37.980 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1400000000000002e-06 2024-03-20T09:03:37.980 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.980 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.980 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:37.980 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:31.879569+0000", 2024-03-20T09:03:37.980 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0020248639999999999 2024-03-20T09:03:37.980 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.980 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.980 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:03:37.980 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:31.879580+0000", 2024-03-20T09:03:37.981 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.049e-05 2024-03-20T09:03:37.981 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.981 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.981 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:37.981 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:32.879523+0000", 2024-03-20T09:03:37.981 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.99994274699999997 2024-03-20T09:03:37.981 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.981 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.981 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:03:37.981 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:32.879531+0000", 2024-03-20T09:03:37.981 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8210000000000008e-06 2024-03-20T09:03:37.981 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.982 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.982 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:37.982 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:34.009826+0000", 2024-03-20T09:03:37.982 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1302948269999999 2024-03-20T09:03:37.982 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.982 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.982 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:37.982 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:34.009908+0000", 2024-03-20T09:03:37.982 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1222000000000003e-05 2024-03-20T09:03:37.982 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:37.982 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:37.982 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:37.982 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.983 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.983 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:58498 3.11 3:88396c3e:::benchmark_data_smithi122_119671_object58497:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-20T09:03:37.983 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:03:31.892645+0000", 2024-03-20T09:03:37.983 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.9890543430000003, 2024-03-20T09:03:37.983 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1635402199999998, 2024-03-20T09:03:37.983 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:37.983 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:37.983 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:37.983 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:03:37.983 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:03:37.983 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 58498 2024-03-20T09:03:37.983 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.984 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:37.984 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.984 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:37.984 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:31.892645+0000", 2024-03-20T09:03:37.984 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:37.984 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.984 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.984 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:37.984 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:31.892645+0000", 2024-03-20T09:03:37.984 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:37.984 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.984 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.984 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:37.985 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:31.892646+0000", 2024-03-20T09:03:37.985 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5939999999999999e-06 2024-03-20T09:03:37.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.985 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.985 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:37.985 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:31.892668+0000", 2024-03-20T09:03:37.985 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1296000000000001e-05 2024-03-20T09:03:37.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.985 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.985 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:37.985 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:31.892680+0000", 2024-03-20T09:03:37.985 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2108e-05 2024-03-20T09:03:37.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.986 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.986 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:37.986 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:31.892686+0000", 2024-03-20T09:03:37.986 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.691e-06 2024-03-20T09:03:37.986 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.986 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.986 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:37.986 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:31.892735+0000", 2024-03-20T09:03:37.986 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8303999999999997e-05 2024-03-20T09:03:37.986 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.987 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.987 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:03:37.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:31.892741+0000", 2024-03-20T09:03:37.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7909999999999999e-06 2024-03-20T09:03:37.987 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.987 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.987 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:37.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:32.879561+0000", 2024-03-20T09:03:37.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.98681976199999999 2024-03-20T09:03:37.987 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.987 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.987 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:03:37.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:32.879571+0000", 2024-03-20T09:03:37.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.8030000000000008e-06 2024-03-20T09:03:37.988 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.988 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:37.988 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:34.010640+0000", 2024-03-20T09:03:37.988 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1310692099999999 2024-03-20T09:03:37.988 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.988 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:37.988 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:34.010819+0000", 2024-03-20T09:03:37.988 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000178631 2024-03-20T09:03:37.988 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.988 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 4,6", 2024-03-20T09:03:37.988 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:34.011023+0000", 2024-03-20T09:03:37.989 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000204098 2024-03-20T09:03:37.989 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.989 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.989 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:37.989 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:34.012845+0000", 2024-03-20T09:03:37.989 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001821865 2024-03-20T09:03:37.989 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.989 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.989 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:37.989 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:34.054904+0000", 2024-03-20T09:03:37.989 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.042058933 2024-03-20T09:03:37.990 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.990 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:37.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:34.055525+0000", 2024-03-20T09:03:37.990 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00062162299999999997 2024-03-20T09:03:37.990 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.990 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:37.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:34.055593+0000", 2024-03-20T09:03:37.990 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7848999999999994e-05 2024-03-20T09:03:37.990 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.990 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:37.991 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:34.056185+0000", 2024-03-20T09:03:37.991 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00059166199999999996 2024-03-20T09:03:37.991 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:37.991 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:37.991 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:37.991 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.991 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.991 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:58507 3.11 3:8ad9a5b3:::benchmark_data_smithi122_119671_object58506:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-20T09:03:37.991 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:03:31.898563+0000", 2024-03-20T09:03:37.991 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.9831363619999998, 2024-03-20T09:03:37.991 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1581842039999999, 2024-03-20T09:03:37.991 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:37.991 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:37.991 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:37.992 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:03:37.992 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:03:37.992 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 58507 2024-03-20T09:03:37.992 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.992 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:37.992 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.992 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:37.992 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:31.898563+0000", 2024-03-20T09:03:37.992 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:37.992 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.992 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.992 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:37.992 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:31.898563+0000", 2024-03-20T09:03:37.992 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:37.993 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.993 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.993 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:37.993 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:31.898565+0000", 2024-03-20T09:03:37.993 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0310000000000001e-06 2024-03-20T09:03:37.993 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.993 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.993 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:37.993 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:31.898590+0000", 2024-03-20T09:03:37.993 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5658999999999999e-05 2024-03-20T09:03:37.993 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.993 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.993 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:37.994 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:31.898604+0000", 2024-03-20T09:03:37.994 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4012999999999999e-05 2024-03-20T09:03:37.994 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.994 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.994 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:37.994 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:31.898611+0000", 2024-03-20T09:03:37.994 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3779999999999998e-06 2024-03-20T09:03:37.994 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.994 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.994 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:37.995 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:31.898676+0000", 2024-03-20T09:03:37.995 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5368000000000002e-05 2024-03-20T09:03:37.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.995 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.995 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:03:37.995 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:31.898688+0000", 2024-03-20T09:03:37.995 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1457999999999999e-05 2024-03-20T09:03:37.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.995 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.995 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:37.995 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:32.879610+0000", 2024-03-20T09:03:37.995 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.98092234700000003 2024-03-20T09:03:37.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.996 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:03:37.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:32.879620+0000", 2024-03-20T09:03:37.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0368e-05 2024-03-20T09:03:37.996 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.996 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:37.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:34.012152+0000", 2024-03-20T09:03:37.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1325312830000001 2024-03-20T09:03:37.996 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.996 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:37.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:34.012236+0000", 2024-03-20T09:03:37.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4252000000000006e-05 2024-03-20T09:03:37.997 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.997 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.997 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 4,6", 2024-03-20T09:03:37.997 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:34.012347+0000", 2024-03-20T09:03:37.997 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000111611 2024-03-20T09:03:37.997 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.997 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.997 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:37.997 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:34.012955+0000", 2024-03-20T09:03:37.997 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00060725899999999997 2024-03-20T09:03:37.997 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.997 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.997 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:37.997 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:34.054963+0000", 2024-03-20T09:03:37.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.042008669999999998 2024-03-20T09:03:37.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.998 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.998 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:37.998 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:34.056223+0000", 2024-03-20T09:03:37.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001259556 2024-03-20T09:03:37.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.998 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.998 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:37.998 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:34.056258+0000", 2024-03-20T09:03:37.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5178000000000001e-05 2024-03-20T09:03:37.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.998 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.998 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:37.999 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:34.056747+0000", 2024-03-20T09:03:37.999 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00048877299999999996 2024-03-20T09:03:37.999 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:37.999 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:37.999 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:37.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:37.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:37.999 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:58786 3.11 3:8f6b111f:::benchmark_data_smithi122_119671_object58785:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e235)", 2024-03-20T09:03:37.999 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:03:34.387432+0000", 2024-03-20T09:03:37.999 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 3.4942666, 2024-03-20T09:03:37.999 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.027833529999999999, 2024-03-20T09:03:37.999 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:37.999 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:38.000 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:38.000 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:03:38.000 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:03:38.000 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 58786 2024-03-20T09:03:38.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:38.000 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:38.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:38.000 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:38.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:34.387432+0000", 2024-03-20T09:03:38.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:38.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:38.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:38.000 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:38.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:34.387432+0000", 2024-03-20T09:03:38.001 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:38.001 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:38.001 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:38.001 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:38.001 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:34.387434+0000", 2024-03-20T09:03:38.001 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.494e-06 2024-03-20T09:03:38.001 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:38.001 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:38.001 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:38.001 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:34.387476+0000", 2024-03-20T09:03:38.001 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2512000000000002e-05 2024-03-20T09:03:38.001 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:38.001 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:38.001 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:38.002 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:34.387490+0000", 2024-03-20T09:03:38.002 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3937000000000001e-05 2024-03-20T09:03:38.002 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:38.002 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:38.002 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:38.002 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:34.387498+0000", 2024-03-20T09:03:38.002 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8790000000000003e-06 2024-03-20T09:03:38.002 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:38.002 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:38.002 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:38.002 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:34.388077+0000", 2024-03-20T09:03:38.002 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00057889900000000002 2024-03-20T09:03:38.002 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:38.002 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:38.003 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:38.003 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:34.388220+0000", 2024-03-20T09:03:38.003 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014286299999999999 2024-03-20T09:03:38.003 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:38.003 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:38.003 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 4,6", 2024-03-20T09:03:38.003 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:34.388395+0000", 2024-03-20T09:03:38.003 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000174771 2024-03-20T09:03:38.003 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:38.003 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:38.003 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:38.003 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:34.389496+0000", 2024-03-20T09:03:38.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0011014009999999999 2024-03-20T09:03:38.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:38.004 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:38.004 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:38.004 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:34.391533+0000", 2024-03-20T09:03:38.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0020371629999999998 2024-03-20T09:03:38.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:38.004 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:38.004 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:38.004 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:34.414373+0000", 2024-03-20T09:03:38.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.022839272000000001 2024-03-20T09:03:38.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:38.004 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:38.004 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:38.005 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:34.414423+0000", 2024-03-20T09:03:38.005 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0336000000000001e-05 2024-03-20T09:03:38.005 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:38.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:38.005 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:38.005 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:34.415266+0000", 2024-03-20T09:03:38.005 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00084300300000000005 2024-03-20T09:03:38.005 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:38.005 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:38.005 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:38.005 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:38.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:38.006 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:62162 3.11 3:8cb2b033:::benchmark_data_smithi122_119671_object62161:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-20T09:03:38.006 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:03:37.807042+0000", 2024-03-20T09:03:38.006 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.074656697999999994, 2024-03-20T09:03:38.006 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.026835424, 2024-03-20T09:03:38.006 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:38.006 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:38.006 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:38.006 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:03:38.006 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:03:38.006 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 62162 2024-03-20T09:03:38.006 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:38.006 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:38.006 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:38.007 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:38.007 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:37.807042+0000", 2024-03-20T09:03:38.007 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:38.007 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:38.007 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:38.007 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:38.007 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:37.807042+0000", 2024-03-20T09:03:38.007 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:38.007 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:38.007 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:38.007 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:38.007 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:37.807044+0000", 2024-03-20T09:03:38.007 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.322e-06 2024-03-20T09:03:38.008 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:38.008 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:38.008 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:38.008 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:37.807070+0000", 2024-03-20T09:03:38.008 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6733e-05 2024-03-20T09:03:38.008 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:38.008 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:38.008 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:38.008 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:37.807085+0000", 2024-03-20T09:03:38.008 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4939999999999999e-05 2024-03-20T09:03:38.008 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:38.008 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:38.008 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:38.008 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:37.807094+0000", 2024-03-20T09:03:38.009 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8950000000000004e-06 2024-03-20T09:03:38.009 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:38.009 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:38.009 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:38.009 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:37.810302+0000", 2024-03-20T09:03:38.009 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003207414 2024-03-20T09:03:38.009 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:38.009 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:38.009 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:38.009 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:37.810489+0000", 2024-03-20T09:03:38.009 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00018771900000000001 2024-03-20T09:03:38.009 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:38.009 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:38.009 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 4,6", 2024-03-20T09:03:38.009 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:37.810695+0000", 2024-03-20T09:03:38.010 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00020523500000000001 2024-03-20T09:03:38.010 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:38.010 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:38.010 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:38.010 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:37.811957+0000", 2024-03-20T09:03:38.010 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0012628769999999999 2024-03-20T09:03:38.010 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:38.010 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:38.010 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:38.010 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:37.821090+0000", 2024-03-20T09:03:38.010 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0091323410000000004 2024-03-20T09:03:38.010 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:38.010 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:38.010 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:38.011 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:37.832892+0000", 2024-03-20T09:03:38.011 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011802142999999999 2024-03-20T09:03:38.011 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:38.011 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:38.011 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:38.011 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:37.832972+0000", 2024-03-20T09:03:38.011 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0556999999999995e-05 2024-03-20T09:03:38.011 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:38.011 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:38.011 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:38.011 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:37.833878+0000", 2024-03-20T09:03:38.011 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00090524799999999999 2024-03-20T09:03:38.011 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:38.011 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:38.011 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:38.012 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:38.012 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:38.012 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:03:38.012 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.8 to 0.600704 (99c7) 2024-03-20T09:03:38.013 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:37 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1264612687' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 0.6007040763735481}]: dispatch 2024-03-20T09:03:38.013 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:37 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 0.6007040763735481}]: dispatch 2024-03-20T09:03:38.013 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:37 smithi122 ceph-mon[73495]: pgmap v648: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 113 MiB/s wr, 1.80k op/s; 7304/179508 objects degraded (4.069%); 33330/179508 objects misplaced (18.567%); 12 MiB/s, 197 objects/s recovering 2024-03-20T09:03:38.071 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:03:38.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:38.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:03:38.176 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:03:38.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:37 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1264612687' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 0.6007040763735481}]: dispatch 2024-03-20T09:03:38.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:37 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 0.6007040763735481}]: dispatch 2024-03-20T09:03:38.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:37 smithi026 ceph-mon[68309]: pgmap v648: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 113 MiB/s wr, 1.80k op/s; 7304/179508 objects degraded (4.069%); 33330/179508 objects misplaced (18.567%); 12 MiB/s, 197 objects/s recovering 2024-03-20T09:03:38.213 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:38.273 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:03:38.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:38.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:03:38.377 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:03:38.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:37 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1264612687' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 0.6007040763735481}]: dispatch 2024-03-20T09:03:38.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:37 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 0.6007040763735481}]: dispatch 2024-03-20T09:03:38.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:37 smithi115 ceph-mon[72476]: pgmap v648: 37 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 27 active+clean; 3.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 113 MiB/s wr, 1.80k op/s; 7304/179508 objects degraded (4.069%); 33330/179508 objects misplaced (18.567%); 12 MiB/s, 197 objects/s recovering 2024-03-20T09:03:38.416 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:38.445 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:03:38.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:38.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:03:38.549 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:03:38.599 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:38.608 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 37 16 62403 62387 105.363 28.125 0.00493183 0.00933098 2024-03-20T09:03:38.626 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:03:38.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:38.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:03:38.701 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_ops_in_flight 2024-03-20T09:03:38.731 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:03:38.773 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:38.799 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:03:38.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:38.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:03:38.904 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:03:38.949 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:38.976 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:03:38.978 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:03:38.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:38.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:03:38.994 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:03:39.081 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:03:39.117 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:38 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 0.6007040763735481}]': finished 2024-03-20T09:03:39.117 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:38 smithi026 ceph-mon[68309]: osdmap e238: 12 total, 12 up, 12 in 2024-03-20T09:03:39.121 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:39.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:38 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 0.6007040763735481}]': finished 2024-03-20T09:03:39.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:38 smithi115 ceph-mon[72476]: osdmap e238: 12 total, 12 up, 12 in 2024-03-20T09:03:39.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:38 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 0.6007040763735481}]': finished 2024-03-20T09:03:39.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:38 smithi122 ceph-mon[73495]: osdmap e238: 12 total, 12 up, 12 in 2024-03-20T09:03:39.230 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:03:39.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:39.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:03:39.337 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:03:39.366 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:39.386 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:03:39.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:39.389 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:03:39.490 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:03:39.518 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:39.608 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 38 16 62403 62387 102.591 0 - 0.00933098 2024-03-20T09:03:39.858 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:03:39.867 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:03:39.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:39.870 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:03:39.920 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:03:39.958 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:03:39.971 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:03:39.993 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:03:40.009 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:40.160 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:03:40.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:40.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:03:40.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:39 smithi122 ceph-mon[73495]: osdmap e239: 12 total, 12 up, 12 in 2024-03-20T09:03:40.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:39 smithi122 ceph-mon[73495]: pgmap v651: 37 pgs: 2 remapped+peering, 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 26 active+clean; 3.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 127 MiB/s wr, 2.03k op/s; 7304/181674 objects degraded (4.020%); 28211/181674 objects misplaced (15.528%); 15 MiB/s, 245 objects/s recovering 2024-03-20T09:03:40.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:39 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3862477307' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:03:40.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:39 smithi026 ceph-mon[68309]: osdmap e239: 12 total, 12 up, 12 in 2024-03-20T09:03:40.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:39 smithi026 ceph-mon[68309]: pgmap v651: 37 pgs: 2 remapped+peering, 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 26 active+clean; 3.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 127 MiB/s wr, 2.03k op/s; 7304/181674 objects degraded (4.020%); 28211/181674 objects misplaced (15.528%); 15 MiB/s, 245 objects/s recovering 2024-03-20T09:03:40.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:39 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3862477307' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:03:40.229 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:03:40.265 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:03:40.273 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:03:40.311 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:03:40.312 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:40.349 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:03:40.371 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:03:40.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:40.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:03:40.387 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:03:40.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:39 smithi115 ceph-mon[72476]: osdmap e239: 12 total, 12 up, 12 in 2024-03-20T09:03:40.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:39 smithi115 ceph-mon[72476]: pgmap v651: 37 pgs: 2 remapped+peering, 2 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 26 active+clean; 3.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 127 MiB/s wr, 2.03k op/s; 7304/181674 objects degraded (4.020%); 28211/181674 objects misplaced (15.528%); 15 MiB/s, 245 objects/s recovering 2024-03-20T09:03:40.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:39 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3862477307' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:03:40.420 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:03:40.450 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:03:40.482 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:03:40.486 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:03:40.518 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:03:40.523 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:03:40.523 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:03:40.523 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:03:40.523 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:03:40.523 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:03:40.523 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:03:40.523 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:03:40.523 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:03:40.523 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:03:40.524 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:03:40.524 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:03:40.527 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:40.608 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 39 16 62625 62609 100.316 6.9375 0.00551071 0.00970755 2024-03-20T09:03:40.888 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:03:40.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:40.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:03:40.997 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:03:41.029 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:41.154 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:03:41.156 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:40 smithi122 ceph-mon[73495]: osdmap e240: 12 total, 12 up, 12 in 2024-03-20T09:03:41.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:41.158 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:03:41.198 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:03:41.199 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:03:41.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:40 smithi026 ceph-mon[68309]: osdmap e240: 12 total, 12 up, 12 in 2024-03-20T09:03:41.259 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:03:41.288 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:41.308 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:03:41.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:41.311 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:03:41.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:40 smithi115 ceph-mon[72476]: osdmap e240: 12 total, 12 up, 12 in 2024-03-20T09:03:41.412 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:03:41.447 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:41.491 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:03:41.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:41.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:03:41.596 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:03:41.608 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:2024-03-20T09:03:41.609159+0000 min lat: 0.001051 max lat: 2.26105 avg lat: 0.00970755 2024-03-20T09:03:41.608 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T09:03:41.609 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 40 16 62625 62609 97.8087 0 - 0.00970755 2024-03-20T09:03:41.634 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:41.661 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:03:41.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:41.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:03:41.766 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:03:41.802 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:42.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:41 smithi122 ceph-mon[73495]: pgmap v653: 37 pgs: 3 remapped+peering, 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 25 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s; 3620/186582 objects degraded (1.940%); 25859/186582 objects misplaced (13.859%); 49 MiB/s, 790 objects/s recovering 2024-03-20T09:03:42.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:41 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 7304/181674 objects degraded (4.020%), 2 pgs degraded (PG_DEGRADED) 2024-03-20T09:03:42.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:41 smithi122 ceph-mon[73495]: osdmap e241: 12 total, 12 up, 12 in 2024-03-20T09:03:42.215 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_blocked_ops 2024-03-20T09:03:42.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:41 smithi026 ceph-mon[68309]: pgmap v653: 37 pgs: 3 remapped+peering, 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 25 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s; 3620/186582 objects degraded (1.940%); 25859/186582 objects misplaced (13.859%); 49 MiB/s, 790 objects/s recovering 2024-03-20T09:03:42.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:41 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 7304/181674 objects degraded (4.020%), 2 pgs degraded (PG_DEGRADED) 2024-03-20T09:03:42.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:41 smithi026 ceph-mon[68309]: osdmap e241: 12 total, 12 up, 12 in 2024-03-20T09:03:42.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:41 smithi115 ceph-mon[72476]: pgmap v653: 37 pgs: 3 remapped+peering, 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 25 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s; 3620/186582 objects degraded (1.940%); 25859/186582 objects misplaced (13.859%); 49 MiB/s, 790 objects/s recovering 2024-03-20T09:03:42.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:41 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 7304/181674 objects degraded (4.020%), 2 pgs degraded (PG_DEGRADED) 2024-03-20T09:03:42.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:41 smithi115 ceph-mon[72476]: osdmap e241: 12 total, 12 up, 12 in 2024-03-20T09:03:42.459 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:03:42.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:42.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:03:42.512 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:03:42.564 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:03:42.604 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:42.608 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 41 16 62625 62609 95.4233 0 - 0.00970755 2024-03-20T09:03:42.633 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:03:42.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:42.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:03:42.736 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:03:42.772 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:43.532 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:03:43.609 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 42 16 62770 62754 93.3674 3.02083 0.00461954 0.010684 2024-03-20T09:03:43.884 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:43 smithi122 ceph-mon[73495]: pgmap v655: 37 pgs: 3 remapped+peering, 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 25 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s wr, 461 op/s; 3620/187389 objects degraded (1.932%); 25859/187389 objects misplaced (13.800%); 36 MiB/s, 577 objects/s recovering 2024-03-20T09:03:43.891 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-03-20T09:03:43.891 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:03:43.892 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-03-20T09:03:43.892 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:03:43.930 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:43 smithi026 ceph-mon[68309]: pgmap v655: 37 pgs: 3 remapped+peering, 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 25 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s wr, 461 op/s; 3620/187389 objects degraded (1.932%); 25859/187389 objects misplaced (13.800%); 36 MiB/s, 577 objects/s recovering 2024-03-20T09:03:44.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:43 smithi115 ceph-mon[72476]: pgmap v655: 37 pgs: 3 remapped+peering, 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 25 active+clean; 3.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s wr, 461 op/s; 3620/187389 objects degraded (1.932%); 25859/187389 objects misplaced (13.800%); 36 MiB/s, 577 objects/s recovering 2024-03-20T09:03:44.565 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:03:44.565 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:03:44.609 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 43 16 63695 63679 92.5405 57.8125 0.00872423 0.0108011 2024-03-20T09:03:45.028 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:03:45.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:45.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:03:45.133 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:03:45.175 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:45.315 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:03:45.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:45.322 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:03:45.352 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_historic_ops 2024-03-20T09:03:45.423 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:03:45.461 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:03:45.465 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:03:45.482 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:45.527 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:03:45.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:45.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:03:45.609 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 44 16 64679 64663 91.8349 61.5 0.00475266 0.0108835 2024-03-20T09:03:45.634 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:03:45.682 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:45.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:45 smithi122 ceph-mon[73495]: pgmap v656: 37 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 25 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s wr, 547 op/s; 12806/190464 objects degraded (6.724%); 19549/190464 objects misplaced (10.264%); 61 MiB/s, 969 objects/s recovering 2024-03-20T09:03:45.724 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:03:45.747 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:03:45.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:45.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:03:45.852 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:03:45.909 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:45 smithi115 ceph-mon[72476]: pgmap v656: 37 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 25 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s wr, 547 op/s; 12806/190464 objects degraded (6.724%); 19549/190464 objects misplaced (10.264%); 61 MiB/s, 969 objects/s recovering 2024-03-20T09:03:45.922 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:45.941 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:45 smithi026 ceph-mon[68309]: pgmap v656: 37 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 25 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 34 MiB/s wr, 547 op/s; 12806/190464 objects degraded (6.724%); 19549/190464 objects misplaced (10.264%); 61 MiB/s, 969 objects/s recovering 2024-03-20T09:03:45.963 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:03:45.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:45.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:03:46.071 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:03:46.106 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:46.132 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:03:46.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:46.135 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:03:46.236 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:03:46.286 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:46.609 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 45 15 65664 65649 91.1636 61.625 0.0165964 0.0109642 2024-03-20T09:03:46.738 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:03:46.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:46 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4166941755' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:03:46.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:46 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4166941755' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:03:47.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:46 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4166941755' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:03:47.609 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 46 16 66614 66598 90.4711 59.3125 0.0104718 0.0110481 2024-03-20T09:03:47.699 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:47 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:03:47.699 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:47 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4166941755' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:03:47.700 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:47 smithi026 ceph-mon[68309]: osdmap e242: 12 total, 12 up, 12 in 2024-03-20T09:03:47.700 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:47 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 12806/190464 objects degraded (6.724%), 4 pgs degraded (PG_DEGRADED) 2024-03-20T09:03:47.700 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:47 smithi026 ceph-mon[68309]: osdmap e243: 12 total, 12 up, 12 in 2024-03-20T09:03:47.700 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:47 smithi026 ceph-mon[68309]: 3.11 scrub starts 2024-03-20T09:03:47.700 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:47 smithi026 ceph-mon[68309]: pgmap v659: 37 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 25 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s wr, 368 op/s; 12806/193215 objects degraded (6.628%); 17667/193215 objects misplaced (9.144%); 46 MiB/s, 739 objects/s recovering 2024-03-20T09:03:48.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:47 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:03:48.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:47 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4166941755' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:03:48.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:47 smithi115 ceph-mon[72476]: osdmap e242: 12 total, 12 up, 12 in 2024-03-20T09:03:48.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:47 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 12806/190464 objects degraded (6.724%), 4 pgs degraded (PG_DEGRADED) 2024-03-20T09:03:48.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:47 smithi115 ceph-mon[72476]: osdmap e243: 12 total, 12 up, 12 in 2024-03-20T09:03:48.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:47 smithi115 ceph-mon[72476]: 3.11 scrub starts 2024-03-20T09:03:48.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:47 smithi115 ceph-mon[72476]: pgmap v659: 37 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 25 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s wr, 368 op/s; 12806/193215 objects degraded (6.628%); 17667/193215 objects misplaced (9.144%); 46 MiB/s, 739 objects/s recovering 2024-03-20T09:03:48.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:47 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:03:48.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:47 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4166941755' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:03:48.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:47 smithi122 ceph-mon[73495]: osdmap e242: 12 total, 12 up, 12 in 2024-03-20T09:03:48.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:47 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 12806/190464 objects degraded (6.724%), 4 pgs degraded (PG_DEGRADED) 2024-03-20T09:03:48.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:47 smithi122 ceph-mon[73495]: osdmap e243: 12 total, 12 up, 12 in 2024-03-20T09:03:48.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:47 smithi122 ceph-mon[73495]: 3.11 scrub starts 2024-03-20T09:03:48.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:47 smithi122 ceph-mon[73495]: pgmap v659: 37 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 25 active+clean; 3.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s wr, 368 op/s; 12806/193215 objects degraded (6.628%); 17667/193215 objects misplaced (9.144%); 46 MiB/s, 739 objects/s recovering 2024-03-20T09:03:48.609 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 47 16 67660 67644 89.9371 65.375 0.0435799 0.0111142 2024-03-20T09:03:49.035 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:48 smithi026 ceph-mon[68309]: 3.11 scrub ok 2024-03-20T09:03:49.045 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:03:49.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:49.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:03:49.152 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:03:49.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:48 smithi122 ceph-mon[73495]: 3.11 scrub ok 2024-03-20T09:03:49.198 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:49.305 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:03:49.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:49.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:03:49.375 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:03:49.407 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:03:49.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:48 smithi115 ceph-mon[72476]: 3.11 scrub ok 2024-03-20T09:03:49.412 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:03:49.440 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:03:49.474 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:49.477 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:03:49.514 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:03:49.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:49.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:03:49.520 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:03:49.559 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:03:49.599 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:03:49.609 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 48 16 68339 68323 88.9474 42.4375 0.00643936 0.0111427 2024-03-20T09:03:49.620 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:03:49.637 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:03:49.662 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:49.667 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:03:49.683 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:03:49.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:49.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:03:49.703 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:03:49.741 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:03:49.775 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:03:49.785 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:03:49.785 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:03:49.785 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:03:49.786 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:03:49.786 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:03:49.786 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:03:49.786 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:03:49.786 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:03:49.786 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:03:49.786 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:03:49.786 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:03:49.786 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:03:49.786 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:03:49.826 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:49.853 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:03:49.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:49.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:03:49.941 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:03:49.952 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:03:49.958 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:03:49.998 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:50.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:49 smithi115 ceph-mon[72476]: osdmap e244: 12 total, 12 up, 12 in 2024-03-20T09:03:50.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:49 smithi115 ceph-mon[72476]: 1.5 scrub starts 2024-03-20T09:03:50.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:49 smithi115 ceph-mon[72476]: 1.5 scrub ok 2024-03-20T09:03:50.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:49 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1059666979' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:03:50.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:49 smithi115 ceph-mon[72476]: 3.1 scrub starts 2024-03-20T09:03:50.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:49 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:03:50.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:49 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:03:50.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:49 smithi115 ceph-mon[72476]: pgmap v661: 37 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 25 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s wr, 491 op/s; 12806/196236 objects degraded (6.526%); 17667/196236 objects misplaced (9.003%); 46 MiB/s, 739 objects/s recovering 2024-03-20T09:03:50.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:49 smithi122 ceph-mon[73495]: osdmap e244: 12 total, 12 up, 12 in 2024-03-20T09:03:50.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:49 smithi122 ceph-mon[73495]: 1.5 scrub starts 2024-03-20T09:03:50.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:49 smithi122 ceph-mon[73495]: 1.5 scrub ok 2024-03-20T09:03:50.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:49 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1059666979' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:03:50.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:49 smithi122 ceph-mon[73495]: 3.1 scrub starts 2024-03-20T09:03:50.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:49 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:03:50.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:49 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:03:50.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:49 smithi122 ceph-mon[73495]: pgmap v661: 37 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 25 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s wr, 491 op/s; 12806/196236 objects degraded (6.526%); 17667/196236 objects misplaced (9.003%); 46 MiB/s, 739 objects/s recovering 2024-03-20T09:03:50.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:49 smithi026 ceph-mon[68309]: osdmap e244: 12 total, 12 up, 12 in 2024-03-20T09:03:50.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:49 smithi026 ceph-mon[68309]: 1.5 scrub starts 2024-03-20T09:03:50.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:49 smithi026 ceph-mon[68309]: 1.5 scrub ok 2024-03-20T09:03:50.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:49 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1059666979' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:03:50.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:49 smithi026 ceph-mon[68309]: 3.1 scrub starts 2024-03-20T09:03:50.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:49 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:03:50.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:49 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:03:50.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:49 smithi026 ceph-mon[68309]: pgmap v661: 37 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 25 active+clean; 4.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s wr, 491 op/s; 12806/196236 objects degraded (6.526%); 17667/196236 objects misplaced (9.003%); 46 MiB/s, 739 objects/s recovering 2024-03-20T09:03:50.361 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:03:50.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:50.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:03:50.469 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:03:50.504 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:50.609 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 49 15 69013 68998 87.9931 42.1875 0.0168635 0.0113597 2024-03-20T09:03:50.811 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:03:50.858 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":662,"stamp":"2024-03-20T09:03:49.910667+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4361028128,"num_objects":66537,"num_object_clones":0,"num_object_copies":199611,"num_objects_missing_on_primary":1675,"num_objects_missing":1675,"num_objects_degraded":9186,"num_objects_misplaced":19130,"num_objects_unfound":0,"num_objects_dirty":66537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66535,"num_write_kb":4258240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13354,"num_bytes_recovered":875167744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103467,"ondisk_log_size":103467,"up":111,"acting":99,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":140,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":15428284,"kb_used_data":13869956,"kb_used_omap":18,"kb_used_meta":1558125,"kb_avail":1109513540,"statfs":{"total":1151940427776,"available":1136141864960,"internally_reserved":0,"allocated":14202834944,"data_stored":14196619004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":1595520315},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":199819264,"num_objects":3049,"num_object_clones":0,"num_object_copies":9147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3620,"num_objects_misplaced":-419,"num_objects_unfound":0,"num_objects_dirty":3049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3049,"num_write_kb":195136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2293,"num_bytes_recovered":150274048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.298526"},"pg_stats":[{"pgid":"3.19","version":"241'3798","reported_seq":6746,"reported_epoch":241,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-20T09:03:46.001239+0000","last_change":"2024-03-20T09:03:34.153923+0000","last_active":"2024-03-20T09:03:46.001239+0000","last_peered":"2024-03-20T09:03:46.001239+0000","last_clean":"2024-03-20T09:03:31.864218+0000","last_became_active":"2024-03-20T09:03:34.071349+0000","last_became_peered":"2024-03-20T09:03:34.071349+0000","last_unstale":"2024-03-20T09:03:46.001239+0000","last_undegraded":"2024-03-20T09:03:46.001239+0000","last_fullsized":"2024-03-20T09:03:33.881302+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":3798,"log_dups_size":0,"ondisk_log_size":3798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133693440,"num_objects":2040,"num_object_clones":0,"num_object_copies":6120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3380,"num_objects_unfound":0,"num_objects_dirty":2040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2040,"num_write_kb":130560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":713,"num_bytes_recovered":46727168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[7,1],"avail_no_missing":["7","8","11"],"object_location_counts":[{"shards":"1,7","objects":913},{"shards":"1,7,8,11","objects":1},{"shards":"7,8,11","objects":1126}],"blocked_by":[],"up_primary":1,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"20'137","reported_seq":362,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-20T09:03:46.886326+0000","last_change":"2024-03-20T09:02:21.099690+0000","last_active":"2024-03-20T09:03:46.886326+0000","last_peered":"2024-03-20T09:03:46.886326+0000","last_clean":"2024-03-20T09:03:46.886326+0000","last_became_active":"2024-03-20T09:01:57.871509+0000","last_became_peered":"2024-03-20T09:01:57.871509+0000","last_unstale":"2024-03-20T09:03:46.886326+0000","last_undegraded":"2024-03-20T09:03:46.886326+0000","last_fullsized":"2024-03-20T09:03:46.886326+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:21.099608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:21.099608+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"244'4276","reported_seq":4369,"reported_epoch":244,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-20T09:03:49.443918+0000","last_change":"2024-03-20T09:03:40.094172+0000","last_active":"2024-03-20T09:03:49.443918+0000","last_peered":"2024-03-20T09:03:49.443918+0000","last_clean":"2024-03-20T09:03:37.897876+0000","last_became_active":"2024-03-20T09:03:40.086905+0000","last_became_peered":"2024-03-20T09:03:40.086905+0000","last_unstale":"2024-03-20T09:03:49.443918+0000","last_undegraded":"2024-03-20T09:03:39.972959+0000","last_fullsized":"2024-03-20T09:03:39.947357+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":4276,"log_dups_size":0,"ondisk_log_size":4276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280231936,"num_objects":4276,"num_object_clones":0,"num_object_copies":12828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3877,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4276,"num_write_kb":273664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4],"avail_no_missing":["6","4","8"],"object_location_counts":[{"shards":"4,6","objects":399},{"shards":"4,6,8","objects":3876}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"20'137","reported_seq":674,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-20T09:03:46.894932+0000","last_change":"2024-03-20T09:02:25.540219+0000","last_active":"2024-03-20T09:03:46.894932+0000","last_peered":"2024-03-20T09:03:46.894932+0000","last_clean":"2024-03-20T09:03:46.894932+0000","last_became_active":"2024-03-20T09:02:06.962814+0000","last_became_peered":"2024-03-20T09:02:06.962814+0000","last_unstale":"2024-03-20T09:03:46.894932+0000","last_undegraded":"2024-03-20T09:03:46.894932+0000","last_fullsized":"2024-03-20T09:03:46.894932+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:25.540102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:25.540102+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:03:46.963634+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,2],"acting":[9,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.b","version":"241'4032","reported_seq":5413,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-20T09:03:45.096549+0000","last_change":"2024-03-20T09:03:10.635413+0000","last_active":"2024-03-20T09:03:45.096549+0000","last_peered":"2024-03-20T09:03:45.096549+0000","last_clean":"2024-03-20T09:03:45.096549+0000","last_became_active":"2024-03-20T09:03:10.633998+0000","last_became_peered":"2024-03-20T09:03:10.633998+0000","last_unstale":"2024-03-20T09:03:45.096549+0000","last_undegraded":"2024-03-20T09:03:45.096549+0000","last_fullsized":"2024-03-20T09:03:45.096549+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":4032,"log_dups_size":0,"ondisk_log_size":4032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:04:06.442895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264241152,"num_objects":4032,"num_object_clones":0,"num_object_copies":12096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4032,"num_write_kb":258048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":642,"num_bytes_recovered":42074112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,11],"acting":[3,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"20'137","reported_seq":318,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-03-20T09:03:48.901668+0000","last_change":"2024-03-20T09:03:13.525409+0000","last_active":"2024-03-20T09:03:48.901668+0000","last_peered":"2024-03-20T09:03:48.901668+0000","last_clean":"2024-03-20T09:03:48.901668+0000","last_became_active":"2024-03-20T09:03:07.875459+0000","last_became_peered":"2024-03-20T09:03:07.875459+0000","last_unstale":"2024-03-20T09:03:48.901668+0000","last_undegraded":"2024-03-20T09:03:48.901668+0000","last_fullsized":"2024-03-20T09:03:48.901668+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:03:13.525325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:03:13.525325+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:04:22.073540+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"244'3907","reported_seq":4006,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-03-20T09:03:49.055646+0000","last_change":"2024-03-20T09:03:30.919404+0000","last_active":"2024-03-20T09:03:49.055646+0000","last_peered":"2024-03-20T09:03:49.055646+0000","last_clean":"2024-03-20T09:03:49.055646+0000","last_became_active":"2024-03-20T09:03:30.916325+0000","last_became_peered":"2024-03-20T09:03:30.916325+0000","last_unstale":"2024-03-20T09:03:49.055646+0000","last_undegraded":"2024-03-20T09:03:49.055646+0000","last_fullsized":"2024-03-20T09:03:49.055646+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":3907,"log_dups_size":0,"ondisk_log_size":3907,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:03:49.236207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140410880,"num_objects":2142,"num_object_clones":0,"num_object_copies":6426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2142,"num_write_kb":137120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"20'137","reported_seq":463,"reported_epoch":244,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-20T09:03:48.895452+0000","last_change":"2024-03-20T09:03:39.957765+0000","last_active":"2024-03-20T09:03:48.895452+0000","last_peered":"2024-03-20T09:03:48.895452+0000","last_clean":"2024-03-20T09:03:36.885236+0000","last_became_active":"2024-03-20T09:03:39.957540+0000","last_became_peered":"2024-03-20T09:03:39.957540+0000","last_unstale":"2024-03-20T09:03:48.895452+0000","last_undegraded":"2024-03-20T09:03:39.946354+0000","last_fullsized":"2024-03-20T09:03:39.945788+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:24.397259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:24.397259+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7],"avail_no_missing":["11","7","8"],"object_location_counts":[{"shards":"7,8,11","objects":1}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.6","version":"244'3963","reported_seq":5423,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-03-20T09:03:49.388719+0000","last_change":"2024-03-20T09:03:30.918879+0000","last_active":"2024-03-20T09:03:49.388719+0000","last_peered":"2024-03-20T09:03:49.388719+0000","last_clean":"2024-03-20T09:03:49.388719+0000","last_became_active":"2024-03-20T09:03:30.916582+0000","last_became_peered":"2024-03-20T09:03:30.916582+0000","last_unstale":"2024-03-20T09:03:49.388719+0000","last_undegraded":"2024-03-20T09:03:49.388719+0000","last_fullsized":"2024-03-20T09:03:49.388719+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":3963,"log_dups_size":0,"ondisk_log_size":3963,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:03:53.988076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140967936,"num_objects":2151,"num_object_clones":0,"num_object_copies":6453,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2151,"num_write_kb":137664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":337,"num_bytes_recovered":22118400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"20'137","reported_seq":358,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-20T09:03:41.877631+0000","last_change":"2024-03-20T09:02:27.912312+0000","last_active":"2024-03-20T09:03:41.877631+0000","last_peered":"2024-03-20T09:03:41.877631+0000","last_clean":"2024-03-20T09:03:41.877631+0000","last_became_active":"2024-03-20T09:01:57.869661+0000","last_became_peered":"2024-03-20T09:01:57.869661+0000","last_unstale":"2024-03-20T09:03:41.877631+0000","last_undegraded":"2024-03-20T09:03:41.877631+0000","last_fullsized":"2024-03-20T09:03:41.877631+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:27.912168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:27.912168+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.7","version":"241'3751","reported_seq":3844,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-20T09:03:45.094526+0000","last_change":"2024-03-20T09:03:30.911757+0000","last_active":"2024-03-20T09:03:45.094526+0000","last_peered":"2024-03-20T09:03:45.094526+0000","last_clean":"2024-03-20T09:03:45.094526+0000","last_became_active":"2024-03-20T09:03:30.910426+0000","last_became_peered":"2024-03-20T09:03:30.910426+0000","last_unstale":"2024-03-20T09:03:45.094526+0000","last_undegraded":"2024-03-20T09:03:45.094526+0000","last_fullsized":"2024-03-20T09:03:45.094526+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":3751,"log_dups_size":0,"ondisk_log_size":3751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:03:48.354958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130056192,"num_objects":1984,"num_object_clones":0,"num_object_copies":5952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1984,"num_write_kb":127008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"20'137","reported_seq":367,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-20T09:03:41.877658+0000","last_change":"2024-03-20T09:03:07.876977+0000","last_active":"2024-03-20T09:03:41.877658+0000","last_peered":"2024-03-20T09:03:41.877658+0000","last_clean":"2024-03-20T09:03:41.877658+0000","last_became_active":"2024-03-20T09:03:07.876617+0000","last_became_peered":"2024-03-20T09:03:07.876617+0000","last_unstale":"2024-03-20T09:03:41.877658+0000","last_undegraded":"2024-03-20T09:03:41.877658+0000","last_fullsized":"2024-03-20T09:03:41.877658+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:26.880624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:26.880624+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:03:55.611860+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,4],"acting":[10,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.4","version":"244'3863","reported_seq":3961,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-03-20T09:03:49.363115+0000","last_change":"2024-03-20T09:03:30.922330+0000","last_active":"2024-03-20T09:03:49.363115+0000","last_peered":"2024-03-20T09:03:49.363115+0000","last_clean":"2024-03-20T09:03:49.363115+0000","last_became_active":"2024-03-20T09:03:30.919651+0000","last_became_peered":"2024-03-20T09:03:30.919651+0000","last_unstale":"2024-03-20T09:03:49.363115+0000","last_undegraded":"2024-03-20T09:03:49.363115+0000","last_fullsized":"2024-03-20T09:03:49.363115+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":3863,"log_dups_size":0,"ondisk_log_size":3863,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:03:53.988047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139296768,"num_objects":2125,"num_object_clones":0,"num_object_copies":6375,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2125,"num_write_kb":136032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"20'137","reported_seq":360,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-03-20T09:03:48.891418+0000","last_change":"2024-03-20T09:02:24.936689+0000","last_active":"2024-03-20T09:03:48.891418+0000","last_peered":"2024-03-20T09:03:48.891418+0000","last_clean":"2024-03-20T09:03:48.891418+0000","last_became_active":"2024-03-20T09:01:57.862486+0000","last_became_peered":"2024-03-20T09:01:57.862486+0000","last_unstale":"2024-03-20T09:03:48.891418+0000","last_undegraded":"2024-03-20T09:03:48.891418+0000","last_fullsized":"2024-03-20T09:03:48.891418+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:24.936556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:24.936556+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"243'3873","reported_seq":3970,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-20T09:03:47.208882+0000","last_change":"2024-03-20T09:03:30.923705+0000","last_active":"2024-03-20T09:03:47.208882+0000","last_peered":"2024-03-20T09:03:47.208882+0000","last_clean":"2024-03-20T09:03:47.208882+0000","last_became_active":"2024-03-20T09:03:30.921890+0000","last_became_peered":"2024-03-20T09:03:30.921890+0000","last_unstale":"2024-03-20T09:03:47.208882+0000","last_undegraded":"2024-03-20T09:03:47.208882+0000","last_fullsized":"2024-03-20T09:03:47.208882+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":3873,"log_dups_size":0,"ondisk_log_size":3873,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:03:48.373300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137428992,"num_objects":2097,"num_object_clones":0,"num_object_copies":6291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2097,"num_write_kb":134208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,10],"acting":[0,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":443,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-03-20T09:03:48.895095+0000","last_change":"2024-03-20T09:02:23.383277+0000","last_active":"2024-03-20T09:03:48.895095+0000","last_peered":"2024-03-20T09:03:48.895095+0000","last_clean":"2024-03-20T09:03:48.895095+0000","last_became_active":"2024-03-20T09:02:06.964051+0000","last_became_peered":"2024-03-20T09:02:06.964051+0000","last_unstale":"2024-03-20T09:03:48.895095+0000","last_undegraded":"2024-03-20T09:03:48.895095+0000","last_fullsized":"2024-03-20T09:03:48.895095+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:23.383145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:23.383145+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"244'3919","reported_seq":5344,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-03-20T09:03:49.190895+0000","last_change":"2024-03-20T09:03:30.926740+0000","last_active":"2024-03-20T09:03:49.190895+0000","last_peered":"2024-03-20T09:03:49.190895+0000","last_clean":"2024-03-20T09:03:49.190895+0000","last_became_active":"2024-03-20T09:03:30.924712+0000","last_became_peered":"2024-03-20T09:03:30.924712+0000","last_unstale":"2024-03-20T09:03:49.190895+0000","last_undegraded":"2024-03-20T09:03:49.190895+0000","last_fullsized":"2024-03-20T09:03:49.190895+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":3919,"log_dups_size":0,"ondisk_log_size":3919,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:03:53.573554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140509184,"num_objects":2144,"num_object_clones":0,"num_object_copies":6432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2144,"num_write_kb":137216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":329,"num_bytes_recovered":21561344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"20'137","reported_seq":674,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-20T09:03:46.887517+0000","last_change":"2024-03-20T09:02:24.000401+0000","last_active":"2024-03-20T09:03:46.887517+0000","last_peered":"2024-03-20T09:03:46.887517+0000","last_clean":"2024-03-20T09:03:46.887517+0000","last_became_active":"2024-03-20T09:02:06.963614+0000","last_became_peered":"2024-03-20T09:02:06.963614+0000","last_unstale":"2024-03-20T09:03:46.887517+0000","last_undegraded":"2024-03-20T09:03:46.887517+0000","last_fullsized":"2024-03-20T09:03:46.887517+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:24.000204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:24.000204+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:03:51.807913+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"244'3840","reported_seq":5323,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-03-20T09:03:49.175328+0000","last_change":"2024-03-20T09:03:30.928051+0000","last_active":"2024-03-20T09:03:49.175328+0000","last_peered":"2024-03-20T09:03:49.175328+0000","last_clean":"2024-03-20T09:03:49.175328+0000","last_became_active":"2024-03-20T09:03:30.926683+0000","last_became_peered":"2024-03-20T09:03:30.926683+0000","last_unstale":"2024-03-20T09:03:49.175328+0000","last_undegraded":"2024-03-20T09:03:49.175328+0000","last_fullsized":"2024-03-20T09:03:49.175328+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":3840,"log_dups_size":0,"ondisk_log_size":3840,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:03:49.236252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137363456,"num_objects":2096,"num_object_clones":0,"num_object_copies":6288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2096,"num_write_kb":134144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":343,"num_bytes_recovered":22511616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,1],"acting":[11,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1","version":"20'137","reported_seq":376,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-20T09:03:41.877931+0000","last_change":"2024-03-20T09:03:07.873347+0000","last_active":"2024-03-20T09:03:41.877931+0000","last_peered":"2024-03-20T09:03:41.877931+0000","last_clean":"2024-03-20T09:03:41.877931+0000","last_became_active":"2024-03-20T09:03:07.872941+0000","last_became_peered":"2024-03-20T09:03:07.872941+0000","last_unstale":"2024-03-20T09:03:41.877931+0000","last_undegraded":"2024-03-20T09:03:41.877931+0000","last_fullsized":"2024-03-20T09:03:41.877931+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:25.927693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:25.927693+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,11],"acting":[10,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3","version":"241'3815","reported_seq":3907,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-20T09:03:45.190379+0000","last_change":"2024-03-20T09:03:30.939580+0000","last_active":"2024-03-20T09:03:45.190379+0000","last_peered":"2024-03-20T09:03:45.190379+0000","last_clean":"2024-03-20T09:03:45.190379+0000","last_became_active":"2024-03-20T09:03:30.937809+0000","last_became_peered":"2024-03-20T09:03:30.937809+0000","last_unstale":"2024-03-20T09:03:45.190379+0000","last_undegraded":"2024-03-20T09:03:45.190379+0000","last_fullsized":"2024-03-20T09:03:45.190379+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":3815,"log_dups_size":0,"ondisk_log_size":3815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:03:48.597808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132481024,"num_objects":2021,"num_object_clones":0,"num_object_copies":6063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2021,"num_write_kb":129376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"20'137","reported_seq":379,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-03-20T09:03:48.901451+0000","last_change":"2024-03-20T09:02:42.863702+0000","last_active":"2024-03-20T09:03:48.901451+0000","last_peered":"2024-03-20T09:03:48.901451+0000","last_clean":"2024-03-20T09:03:48.901451+0000","last_became_active":"2024-03-20T09:02:42.863240+0000","last_became_peered":"2024-03-20T09:02:42.863240+0000","last_unstale":"2024-03-20T09:03:48.901451+0000","last_undegraded":"2024-03-20T09:03:48.901451+0000","last_fullsized":"2024-03-20T09:03:48.901451+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:21.734957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:21.734957+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"243'3864","reported_seq":5245,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-20T09:03:47.121581+0000","last_change":"2024-03-20T09:03:30.933949+0000","last_active":"2024-03-20T09:03:47.121581+0000","last_peered":"2024-03-20T09:03:47.121581+0000","last_clean":"2024-03-20T09:03:47.121581+0000","last_became_active":"2024-03-20T09:03:30.931920+0000","last_became_peered":"2024-03-20T09:03:30.931920+0000","last_unstale":"2024-03-20T09:03:47.121581+0000","last_undegraded":"2024-03-20T09:03:47.121581+0000","last_fullsized":"2024-03-20T09:03:47.121581+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":3864,"log_dups_size":0,"ondisk_log_size":3864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:03:49.215198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136216576,"num_objects":2078,"num_object_clones":0,"num_object_copies":6234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2078,"num_write_kb":133024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":318,"num_bytes_recovered":20873216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"241'3776","reported_seq":3867,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-20T09:03:46.558009+0000","last_change":"2024-03-20T09:03:30.912463+0000","last_active":"2024-03-20T09:03:46.558009+0000","last_peered":"2024-03-20T09:03:46.558009+0000","last_clean":"2024-03-20T09:03:46.558009+0000","last_became_active":"2024-03-20T09:03:30.910402+0000","last_became_peered":"2024-03-20T09:03:30.910402+0000","last_unstale":"2024-03-20T09:03:46.558009+0000","last_undegraded":"2024-03-20T09:03:46.558009+0000","last_fullsized":"2024-03-20T09:03:46.558009+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":3776,"log_dups_size":0,"ondisk_log_size":3776,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:03:47.985104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132251648,"num_objects":2018,"num_object_clones":0,"num_object_copies":6054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2018,"num_write_kb":129152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.c","version":"243'4102","reported_seq":4193,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-20T09:03:47.117241+0000","last_change":"2024-03-20T09:02:49.893721+0000","last_active":"2024-03-20T09:03:47.117241+0000","last_peered":"2024-03-20T09:03:47.117241+0000","last_clean":"2024-03-20T09:03:47.117241+0000","last_became_active":"2024-03-20T09:02:49.893394+0000","last_became_peered":"2024-03-20T09:02:49.893394+0000","last_unstale":"2024-03-20T09:03:47.117241+0000","last_undegraded":"2024-03-20T09:03:47.117241+0000","last_fullsized":"2024-03-20T09:03:47.117241+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":4102,"log_dups_size":0,"ondisk_log_size":4102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:03:55.593370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268828672,"num_objects":4102,"num_object_clones":0,"num_object_copies":12306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4102,"num_write_kb":262528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"243'4062","reported_seq":4153,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-20T09:03:47.090645+0000","last_change":"2024-03-20T09:02:49.921481+0000","last_active":"2024-03-20T09:03:47.090645+0000","last_peered":"2024-03-20T09:03:47.090645+0000","last_clean":"2024-03-20T09:03:47.090645+0000","last_became_active":"2024-03-20T09:02:49.921160+0000","last_became_peered":"2024-03-20T09:02:49.921160+0000","last_unstale":"2024-03-20T09:03:47.090645+0000","last_undegraded":"2024-03-20T09:03:47.090645+0000","last_fullsized":"2024-03-20T09:03:47.090645+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":4062,"log_dups_size":0,"ondisk_log_size":4062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:04:07.144343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266207232,"num_objects":4062,"num_object_clones":0,"num_object_copies":12186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4062,"num_write_kb":259968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,9],"acting":[5,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"244'4259","reported_seq":5707,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-03-20T09:03:49.151242+0000","last_change":"2024-03-20T09:03:10.635160+0000","last_active":"2024-03-20T09:03:49.151242+0000","last_peered":"2024-03-20T09:03:49.151242+0000","last_clean":"2024-03-20T09:03:49.151242+0000","last_became_active":"2024-03-20T09:03:10.633041+0000","last_became_peered":"2024-03-20T09:03:10.633041+0000","last_unstale":"2024-03-20T09:03:49.151242+0000","last_undegraded":"2024-03-20T09:03:49.151242+0000","last_fullsized":"2024-03-20T09:03:49.151242+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":4259,"log_dups_size":0,"ondisk_log_size":4259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:04:08.374454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279117824,"num_objects":4259,"num_object_clones":0,"num_object_copies":12777,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4259,"num_write_kb":272576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":672,"num_bytes_recovered":44040192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9],"acting":[11,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.f","version":"244'4155","reported_seq":4248,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-03-20T09:03:49.199488+0000","last_change":"2024-03-20T09:02:49.921660+0000","last_active":"2024-03-20T09:03:49.199488+0000","last_peered":"2024-03-20T09:03:49.199488+0000","last_clean":"2024-03-20T09:03:49.199488+0000","last_became_active":"2024-03-20T09:02:49.921153+0000","last_became_peered":"2024-03-20T09:02:49.921153+0000","last_unstale":"2024-03-20T09:03:49.199488+0000","last_undegraded":"2024-03-20T09:03:49.199488+0000","last_fullsized":"2024-03-20T09:03:49.199488+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":4155,"log_dups_size":0,"ondisk_log_size":4155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:04:10.211495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272302080,"num_objects":4155,"num_object_clones":0,"num_object_copies":12465,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4155,"num_write_kb":265920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,8],"acting":[11,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.10","version":"244'3886","reported_seq":9538,"reported_epoch":244,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-20T09:03:49.895580+0000","last_change":"2024-03-20T09:03:34.215955+0000","last_active":"2024-03-20T09:03:49.895580+0000","last_peered":"2024-03-20T09:03:49.895580+0000","last_clean":"2024-03-20T09:03:31.829008+0000","last_became_active":"2024-03-20T09:03:34.185671+0000","last_became_peered":"2024-03-20T09:03:34.185671+0000","last_unstale":"2024-03-20T09:03:49.895580+0000","last_undegraded":"2024-03-20T09:03:49.895580+0000","last_fullsized":"2024-03-20T09:03:33.902042+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":3886,"log_dups_size":0,"ondisk_log_size":3886,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140378112,"num_objects":2142,"num_object_clones":0,"num_object_copies":6426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":780,"num_objects_unfound":0,"num_objects_dirty":2142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2142,"num_write_kb":137088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1733,"num_bytes_recovered":113541120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,5],"acting":[6,5],"avail_no_missing":["6","1","11"],"object_location_counts":[{"shards":"1,6,11","objects":390},{"shards":"5,6","objects":1752}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"241'3838","reported_seq":7671,"reported_epoch":241,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-20T09:03:46.127646+0000","last_change":"2024-03-20T09:03:34.055595+0000","last_active":"2024-03-20T09:03:46.127646+0000","last_peered":"2024-03-20T09:03:46.127646+0000","last_clean":"2024-03-20T09:03:31.872349+0000","last_became_active":"2024-03-20T09:03:34.008061+0000","last_became_peered":"2024-03-20T09:03:34.008061+0000","last_unstale":"2024-03-20T09:03:46.127646+0000","last_undegraded":"2024-03-20T09:03:46.127646+0000","last_fullsized":"2024-03-20T09:03:33.881293+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":3838,"log_dups_size":0,"ondisk_log_size":3838,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135135232,"num_objects":2062,"num_object_clones":0,"num_object_copies":6186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2062,"num_write_kb":131968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1869,"num_bytes_recovered":122486784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,6],"acting":[10,4],"avail_no_missing":["10","0","4"],"object_location_counts":[{"shards":"0,4,10","objects":1},{"shards":"4,10","objects":2061}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.12","version":"241'3780","reported_seq":7174,"reported_epoch":241,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-20T09:03:45.251589+0000","last_change":"2024-03-20T09:03:37.954525+0000","last_active":"2024-03-20T09:03:45.251589+0000","last_peered":"2024-03-20T09:03:45.251589+0000","last_clean":"2024-03-20T09:03:31.842124+0000","last_became_active":"2024-03-20T09:03:34.061531+0000","last_became_peered":"2024-03-20T09:03:34.061531+0000","last_unstale":"2024-03-20T09:03:45.251589+0000","last_undegraded":"2024-03-20T09:03:45.251589+0000","last_fullsized":"2024-03-20T09:03:33.883191+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":3780,"log_dups_size":0,"ondisk_log_size":3780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131399680,"num_objects":2005,"num_object_clones":0,"num_object_copies":6015,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4049,"num_objects_unfound":0,"num_objects_dirty":2005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2005,"num_write_kb":128320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":822,"num_bytes_recovered":53870592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[4,1],"avail_no_missing":["4","3","5","6"],"object_location_counts":[{"shards":"1,4","objects":655},{"shards":"1,4,5,6","objects":1},{"shards":"4,5,6","objects":1349}],"blocked_by":[],"up_primary":1,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"241'3824","reported_seq":6637,"reported_epoch":241,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-20T09:03:45.254223+0000","last_change":"2024-03-20T09:03:34.092565+0000","last_active":"2024-03-20T09:03:45.254223+0000","last_peered":"2024-03-20T09:03:45.254223+0000","last_clean":"2024-03-20T09:03:31.875152+0000","last_became_active":"2024-03-20T09:03:34.070822+0000","last_became_peered":"2024-03-20T09:03:34.070822+0000","last_unstale":"2024-03-20T09:03:45.254223+0000","last_undegraded":"2024-03-20T09:03:45.254223+0000","last_fullsized":"2024-03-20T09:03:33.882704+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":3824,"log_dups_size":0,"ondisk_log_size":3824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133070848,"num_objects":2031,"num_object_clones":0,"num_object_copies":6093,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1838,"num_objects_unfound":0,"num_objects_dirty":2031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2031,"num_write_kb":129952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":906,"num_bytes_recovered":59375616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,2],"avail_no_missing":["4","0","6"],"object_location_counts":[{"shards":"0,4,6","objects":919},{"shards":"2,4","objects":1112}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"244'3863","reported_seq":8689,"reported_epoch":244,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-20T09:03:49.890080+0000","last_change":"2024-03-20T09:03:34.102790+0000","last_active":"2024-03-20T09:03:49.890080+0000","last_peered":"2024-03-20T09:03:49.890080+0000","last_clean":"2024-03-20T09:03:31.874481+0000","last_became_active":"2024-03-20T09:03:34.060209+0000","last_became_peered":"2024-03-20T09:03:34.060209+0000","last_unstale":"2024-03-20T09:03:49.890080+0000","last_undegraded":"2024-03-20T09:03:49.890080+0000","last_fullsized":"2024-03-20T09:03:33.882045+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":3863,"log_dups_size":0,"ondisk_log_size":3863,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139296768,"num_objects":2126,"num_object_clones":0,"num_object_copies":6378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":434,"num_objects_unfound":0,"num_objects_dirty":2126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2126,"num_write_kb":136032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1575,"num_bytes_recovered":103219200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,11],"acting":[1,11],"avail_no_missing":["1","5","11"],"object_location_counts":[{"shards":"1,5,11","objects":217},{"shards":"1,11","objects":1909}],"blocked_by":[],"up_primary":4,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"244'3963","reported_seq":5992,"reported_epoch":244,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-20T09:03:49.143211+0000","last_change":"2024-03-20T09:03:43.309938+0000","last_active":"2024-03-20T09:03:49.143211+0000","last_peered":"2024-03-20T09:03:49.143211+0000","last_clean":"2024-03-20T09:03:31.866916+0000","last_became_active":"2024-03-20T09:03:43.290722+0000","last_became_peered":"2024-03-20T09:03:43.290722+0000","last_unstale":"2024-03-20T09:03:49.143211+0000","last_undegraded":"2024-03-20T09:03:41.005184+0000","last_fullsized":"2024-03-20T09:03:41.002007+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":3963,"log_dups_size":0,"ondisk_log_size":3963,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142704640,"num_objects":2178,"num_object_clones":0,"num_object_copies":6534,"num_objects_missing_on_primary":1675,"num_objects_missing":1675,"num_objects_degraded":5308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2178,"num_write_kb":139360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":479,"num_bytes_recovered":31358976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,4],"acting":[11,7],"avail_no_missing":["2","3","5","6"],"object_location_counts":[{"shards":"2,5,6","objects":1675},{"shards":"2,5,6,7,8,11","objects":159},{"shards":"5,6,7,8,11","objects":1},{"shards":"5,7,8,11","objects":123},{"shards":"7,11","objects":220}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.16","version":"241'3850","reported_seq":8115,"reported_epoch":241,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-20T09:03:44.881898+0000","last_change":"2024-03-20T09:03:34.062138+0000","last_active":"2024-03-20T09:03:44.881898+0000","last_peered":"2024-03-20T09:03:44.881898+0000","last_clean":"2024-03-20T09:03:31.872748+0000","last_became_active":"2024-03-20T09:03:34.040331+0000","last_became_peered":"2024-03-20T09:03:34.040331+0000","last_unstale":"2024-03-20T09:03:44.881898+0000","last_undegraded":"2024-03-20T09:03:44.881898+0000","last_fullsized":"2024-03-20T09:03:33.881412+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":3850,"log_dups_size":0,"ondisk_log_size":3850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133562368,"num_objects":2038,"num_object_clones":0,"num_object_copies":6114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":458,"num_objects_unfound":0,"num_objects_dirty":2038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2038,"num_write_kb":130432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1743,"num_bytes_recovered":114196480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[1,7],"avail_no_missing":["1","0","4","7"],"object_location_counts":[{"shards":"1,4,7","objects":458},{"shards":"1,7","objects":1580}],"blocked_by":[],"up_primary":5,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"241'3785","reported_seq":7066,"reported_epoch":241,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-20T09:03:45.110302+0000","last_change":"2024-03-20T09:03:34.098721+0000","last_active":"2024-03-20T09:03:45.110302+0000","last_peered":"2024-03-20T09:03:45.110302+0000","last_clean":"2024-03-20T09:03:31.879207+0000","last_became_active":"2024-03-20T09:03:34.062582+0000","last_became_peered":"2024-03-20T09:03:34.062582+0000","last_unstale":"2024-03-20T09:03:45.110302+0000","last_undegraded":"2024-03-20T09:03:45.110302+0000","last_fullsized":"2024-03-20T09:03:33.887282+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":3785,"log_dups_size":0,"ondisk_log_size":3785,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132284416,"num_objects":2019,"num_object_clones":0,"num_object_copies":6057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2991,"num_objects_unfound":0,"num_objects_dirty":2019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2019,"num_write_kb":129184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":52166656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[3,1],"avail_no_missing":["3","7","11"],"object_location_counts":[{"shards":"1,3","objects":1022},{"shards":"3,7,11","objects":997}],"blocked_by":[],"up_primary":5,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"244'3916","reported_seq":4325,"reported_epoch":244,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-20T09:03:49.882577+0000","last_change":"2024-03-20T09:03:48.621553+0000","last_active":"2024-03-20T09:03:49.882577+0000","last_peered":"2024-03-20T09:03:49.882577+0000","last_clean":"2024-03-20T09:03:31.840153+0000","last_became_active":"2024-03-20T09:03:34.084203+0000","last_became_peered":"2024-03-20T09:03:34.084203+0000","last_unstale":"2024-03-20T09:03:49.882577+0000","last_undegraded":"2024-03-20T09:03:49.882577+0000","last_fullsized":"2024-03-20T09:03:33.881383+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":3916,"log_dups_size":0,"ondisk_log_size":3916,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141000704,"num_objects":2152,"num_object_clones":0,"num_object_copies":6456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5199,"num_objects_unfound":0,"num_objects_dirty":2152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2152,"num_write_kb":137696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":5046272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,7],"acting":[1,2],"avail_no_missing":["1","4","11"],"object_location_counts":[{"shards":"1,2","objects":419},{"shards":"1,4,11","objects":1733}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":4360437760,"num_objects":66535,"num_object_clones":0,"num_object_copies":199605,"num_objects_missing_on_primary":1675,"num_objects_missing":1675,"num_objects_degraded":9185,"num_objects_misplaced":19130,"num_objects_unfound":0,"num_objects_dirty":66535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66535,"num_write_kb":4258240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13354,"num_bytes_recovered":875167744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14317060096,"data_stored":14317060096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101960,"ondisk_log_size":101960,"up":78,"acting":67,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1507,"ondisk_log_size":1507,"up":33,"acting":32,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860510,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2271996,"kb_used_data":2124208,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":91473156,"statfs":{"total":95995035648,"available":93668511744,"internally_reserved":0,"allocated":2175188992,"data_stored":2174663795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":151321028},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":89,"seq":382252089442,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":336532,"kb_used_data":265100,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93408620,"statfs":{"total":95995035648,"available":95650426880,"internally_reserved":0,"allocated":271462400,"data_stored":270949853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":73136576},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":80,"seq":343597383786,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":750764,"kb_used_data":621028,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":92994388,"statfs":{"total":95995035648,"available":95226253312,"internally_reserved":0,"allocated":635932672,"data_stored":635411704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":132839872},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":70,"seq":300647710830,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":998484,"kb_used_data":868748,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":92746668,"statfs":{"total":95995035648,"available":94972588032,"internally_reserved":0,"allocated":889597952,"data_stored":889085413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132839878},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":60,"seq":257698037875,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1618772,"kb_used_data":1438988,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":92126380,"statfs":{"total":95995035648,"available":94337413120,"internally_reserved":0,"allocated":1473523712,"data_stored":1473011173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":184023498},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332217,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1929472,"kb_used_data":1817776,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":91815680,"statfs":{"total":95995035648,"available":94019256320,"internally_reserved":0,"allocated":1861402624,"data_stored":1860877435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114358726},"hb_peers":[0,1,2,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574995,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1462980,"kb_used_data":1297136,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":92282172,"statfs":{"total":95995035648,"available":94496944128,"internally_reserved":0,"allocated":1328267264,"data_stored":1327742075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":169802182},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738521,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":637588,"kb_used_data":507852,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93107564,"statfs":{"total":95995035648,"available":95342145536,"internally_reserved":0,"allocated":520040448,"data_stored":519527909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411468,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":504244,"kb_used_data":410604,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93240908,"statfs":{"total":95995035648,"available":95478689792,"internally_reserved":0,"allocated":420458496,"data_stored":419933984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149831,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":956188,"kb_used_data":820884,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":92788964,"statfs":{"total":95995035648,"available":95015899136,"internally_reserved":0,"allocated":840585216,"data_stored":840069125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":138541511},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953603,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2313812,"kb_used_data":2166028,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":91431340,"statfs":{"total":95995035648,"available":93625692160,"internally_reserved":0,"allocated":2218012672,"data_stored":2217500133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659262,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1647452,"kb_used_data":1531604,"kb_used_omap":1,"kb_used_meta":115838,"kb_avail":92097700,"statfs":{"total":95995035648,"available":94308044800,"internally_reserved":0,"allocated":1568362496,"data_stored":1567846405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":118618565},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":520552448,"data_stored":520552448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1346502656,"data_stored":1346502656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":427950080,"data_stored":427950080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":850198528,"data_stored":850198528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2246901760,"data_stored":2246901760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1599602688,"data_stored":1599602688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1861943296,"data_stored":1861943296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1482424320,"data_stored":1482424320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":898695168,"data_stored":898695168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":638189568,"data_stored":638189568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":270532608,"data_stored":270532608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2173566976,"data_stored":2173566976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:03:50.861 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-20T09:03:50.861 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pool set .mgr pg_num 21 2024-03-20T09:03:51.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:50 smithi122 ceph-mon[73495]: 3.1 scrub ok 2024-03-20T09:03:51.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:50 smithi122 ceph-mon[73495]: osdmap e245: 12 total, 12 up, 12 in 2024-03-20T09:03:51.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:50 smithi122 ceph-mon[73495]: from='client.25003 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:03:51.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:50 smithi026 ceph-mon[68309]: 3.1 scrub ok 2024-03-20T09:03:51.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:50 smithi026 ceph-mon[68309]: osdmap e245: 12 total, 12 up, 12 in 2024-03-20T09:03:51.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:50 smithi026 ceph-mon[68309]: from='client.25003 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:03:51.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:50 smithi115 ceph-mon[72476]: 3.1 scrub ok 2024-03-20T09:03:51.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:50 smithi115 ceph-mon[72476]: osdmap e245: 12 total, 12 up, 12 in 2024-03-20T09:03:51.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:50 smithi115 ceph-mon[72476]: from='client.25003 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:03:51.532 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:03:51.550 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:03:51.609 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 50 16 69948 69932 87.4006 58.375 0.0112541 0.0114342 2024-03-20T09:03:52.133 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:51 smithi026 ceph-mon[68309]: pgmap v663: 37 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 26 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s wr, 403 op/s; 9186/203427 objects degraded (4.516%); 15750/203427 objects misplaced (7.742%); 5.2 MiB/s, 83 objects/s recovering 2024-03-20T09:03:52.133 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:51 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 12806/196236 objects degraded (6.526%), 4 pgs degraded (PG_DEGRADED) 2024-03-20T09:03:52.133 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:51 smithi026 ceph-mon[68309]: osdmap e246: 12 total, 12 up, 12 in 2024-03-20T09:03:52.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:51 smithi122 ceph-mon[73495]: pgmap v663: 37 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 26 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s wr, 403 op/s; 9186/203427 objects degraded (4.516%); 15750/203427 objects misplaced (7.742%); 5.2 MiB/s, 83 objects/s recovering 2024-03-20T09:03:52.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:51 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 12806/196236 objects degraded (6.526%), 4 pgs degraded (PG_DEGRADED) 2024-03-20T09:03:52.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:51 smithi122 ceph-mon[73495]: osdmap e246: 12 total, 12 up, 12 in 2024-03-20T09:03:52.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:51 smithi115 ceph-mon[72476]: pgmap v663: 37 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 26 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s wr, 403 op/s; 9186/203427 objects degraded (4.516%); 15750/203427 objects misplaced (7.742%); 5.2 MiB/s, 83 objects/s recovering 2024-03-20T09:03:52.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:51 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 12806/196236 objects degraded (6.526%), 4 pgs degraded (PG_DEGRADED) 2024-03-20T09:03:52.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:51 smithi115 ceph-mon[72476]: osdmap e246: 12 total, 12 up, 12 in 2024-03-20T09:03:52.609 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 51 16 70783 70767 86.7102 52.1875 0.00592328 0.0114644 2024-03-20T09:03:53.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:52 smithi122 ceph-mon[73495]: 1.1 scrub starts 2024-03-20T09:03:53.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:52 smithi122 ceph-mon[73495]: 1.1 scrub ok 2024-03-20T09:03:53.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:52 smithi122 ceph-mon[73495]: osdmap e247: 12 total, 12 up, 12 in 2024-03-20T09:03:53.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:52 smithi026 ceph-mon[68309]: 1.1 scrub starts 2024-03-20T09:03:53.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:52 smithi026 ceph-mon[68309]: 1.1 scrub ok 2024-03-20T09:03:53.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:52 smithi026 ceph-mon[68309]: osdmap e247: 12 total, 12 up, 12 in 2024-03-20T09:03:53.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:52 smithi115 ceph-mon[72476]: 1.1 scrub starts 2024-03-20T09:03:53.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:52 smithi115 ceph-mon[72476]: 1.1 scrub ok 2024-03-20T09:03:53.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:52 smithi115 ceph-mon[72476]: osdmap e247: 12 total, 12 up, 12 in 2024-03-20T09:03:53.446 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:03:53.609 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 52 16 71678 71662 86.1183 55.9375 0.0162778 0.0116075 2024-03-20T09:03:54.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:53 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:03:54.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:53 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:03:54.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:53 smithi026 ceph-mon[68309]: pgmap v666: 37 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 27 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s wr, 54 op/s; 9186/206763 objects degraded (4.443%); 15292/206763 objects misplaced (7.396%); 11 MiB/s, 183 objects/s recovering 2024-03-20T09:03:54.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:53 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:03:54.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:53 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:03:54.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:53 smithi115 ceph-mon[72476]: pgmap v666: 37 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 27 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s wr, 54 op/s; 9186/206763 objects degraded (4.443%); 15292/206763 objects misplaced (7.396%); 11 MiB/s, 183 objects/s recovering 2024-03-20T09:03:54.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:53 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:03:54.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:53 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:03:54.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:53 smithi122 ceph-mon[73495]: pgmap v666: 37 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 27 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s wr, 54 op/s; 9186/206763 objects degraded (4.443%); 15292/206763 objects misplaced (7.396%); 11 MiB/s, 183 objects/s recovering 2024-03-20T09:03:54.610 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 53 16 72530 72514 85.4982 53.25 0.00350223 0.0116223 2024-03-20T09:03:54.956 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:54 smithi026 ceph-mon[68309]: osdmap e248: 12 total, 12 up, 12 in 2024-03-20T09:03:55.025 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:03:55.194 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:03:55.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:55.200 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:03:55.301 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:03:55.342 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:55.370 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:03:55.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:55.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:03:55.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:54 smithi115 ceph-mon[72476]: osdmap e248: 12 total, 12 up, 12 in 2024-03-20T09:03:55.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:54 smithi122 ceph-mon[73495]: osdmap e248: 12 total, 12 up, 12 in 2024-03-20T09:03:55.474 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:03:55.510 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:55.610 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 54 16 73280 73264 84.7829 46.875 0.0104495 0.0117904 2024-03-20T09:03:55.983 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:03:56.025 INFO:teuthology.orchestra.run.smithi026.stderr:set pool 1 pg_num to 21 2024-03-20T09:03:56.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:55 smithi026 ceph-mon[68309]: osdmap e249: 12 total, 12 up, 12 in 2024-03-20T09:03:56.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:55 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1910101533' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:03:56.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:55 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:03:56.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:55 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1245823093' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-20T09:03:56.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:55 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-20T09:03:56.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:55 smithi026 ceph-mon[68309]: pgmap v669: 37 pgs: 1 peering, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 28 active+clean; 4.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s wr, 455 op/s; 9186/215346 objects degraded (4.266%); 9345/215346 objects misplaced (4.340%); 44 MiB/s, 703 objects/s recovering 2024-03-20T09:03:56.332 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:55 smithi122 ceph-mon[73495]: osdmap e249: 12 total, 12 up, 12 in 2024-03-20T09:03:56.332 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:56 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1910101533' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:03:56.332 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:56 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:03:56.332 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:56 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1245823093' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-20T09:03:56.332 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:56 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-20T09:03:56.332 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:56 smithi122 ceph-mon[73495]: pgmap v669: 37 pgs: 1 peering, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 28 active+clean; 4.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s wr, 455 op/s; 9186/215346 objects degraded (4.266%); 9345/215346 objects misplaced (4.340%); 44 MiB/s, 703 objects/s recovering 2024-03-20T09:03:56.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:55 smithi115 ceph-mon[72476]: osdmap e249: 12 total, 12 up, 12 in 2024-03-20T09:03:56.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:55 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1910101533' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:03:56.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:55 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:03:56.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:55 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1245823093' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-20T09:03:56.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:55 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-03-20T09:03:56.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:55 smithi115 ceph-mon[72476]: pgmap v669: 37 pgs: 1 peering, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 28 active+clean; 4.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s wr, 455 op/s; 9186/215346 objects degraded (4.266%); 9345/215346 objects misplaced (4.340%); 44 MiB/s, 703 objects/s recovering 2024-03-20T09:03:56.610 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 55 16 73802 73786 83.8346 32.625 0.00654133 0.011812 2024-03-20T09:03:56.627 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:03:56.627 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-03-20T09:03:56.627 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-03-20T09:03:56.627 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-03-20T09:03:56.627 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.628 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:116213 2.c 2:3f17461b:::benchmark_data_smithi122_107654_object116212:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-03-20T09:03:56.628 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:59:51.205319+0000", 2024-03-20T09:03:56.628 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 245.41622082000001, 2024-03-20T09:03:56.628 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19140834600000001, 2024-03-20T09:03:56.628 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:56.628 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:56.628 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:56.628 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:03:56.628 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:03:56.628 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 116213 2024-03-20T09:03:56.628 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.628 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:56.628 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.629 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:56.629 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.205319+0000", 2024-03-20T09:03:56.629 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.629 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.629 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.629 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:56.629 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.205319+0000", 2024-03-20T09:03:56.629 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.629 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.629 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.629 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:56.629 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.205321+0000", 2024-03-20T09:03:56.629 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8390000000000001e-06 2024-03-20T09:03:56.629 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.630 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.630 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:56.630 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.205342+0000", 2024-03-20T09:03:56.630 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0903e-05 2024-03-20T09:03:56.630 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.630 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.630 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:56.630 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.205356+0000", 2024-03-20T09:03:56.630 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3617e-05 2024-03-20T09:03:56.630 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.630 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.630 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:56.630 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.205364+0000", 2024-03-20T09:03:56.630 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1610000000000003e-06 2024-03-20T09:03:56.631 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.631 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.631 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:56.631 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.230109+0000", 2024-03-20T09:03:56.631 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.024744629000000001 2024-03-20T09:03:56.631 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.631 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.631 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:56.631 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.230204+0000", 2024-03-20T09:03:56.631 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5529e-05 2024-03-20T09:03:56.631 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.631 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.631 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,1", 2024-03-20T09:03:56.632 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.230319+0000", 2024-03-20T09:03:56.632 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000114744 2024-03-20T09:03:56.632 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.632 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.632 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:56.632 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.232694+0000", 2024-03-20T09:03:56.632 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0023751190000000002 2024-03-20T09:03:56.632 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.632 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.632 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.632 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.288571+0000", 2024-03-20T09:03:56.632 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.055876985999999997 2024-03-20T09:03:56.632 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.633 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.633 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.633 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.396625+0000", 2024-03-20T09:03:56.633 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.108054493 2024-03-20T09:03:56.633 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.633 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.633 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:56.633 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.396669+0000", 2024-03-20T09:03:56.633 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3891999999999999e-05 2024-03-20T09:03:56.633 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.633 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.633 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:56.633 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.396728+0000", 2024-03-20T09:03:56.633 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8434000000000003e-05 2024-03-20T09:03:56.634 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.634 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:56.634 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.634 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.634 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.634 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:116410 2.c 2:3d71f892:::benchmark_data_smithi122_107654_object116409:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-03-20T09:03:56.634 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:59:51.518113+0000", 2024-03-20T09:03:56.634 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 245.10342763700001, 2024-03-20T09:03:56.634 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.144706217, 2024-03-20T09:03:56.634 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:56.634 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:56.634 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:56.634 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:03:56.635 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:03:56.635 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 116410 2024-03-20T09:03:56.635 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.635 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:56.635 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.635 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:56.635 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.518113+0000", 2024-03-20T09:03:56.635 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.635 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.635 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.635 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:56.635 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.518113+0000", 2024-03-20T09:03:56.635 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.635 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.636 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.636 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:56.636 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.518113+0000", 2024-03-20T09:03:56.636 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6899999999999996e-07 2024-03-20T09:03:56.636 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.636 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.636 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:56.636 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.518135+0000", 2024-03-20T09:03:56.636 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1141999999999998e-05 2024-03-20T09:03:56.636 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.636 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.636 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:56.636 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.518146+0000", 2024-03-20T09:03:56.636 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1093e-05 2024-03-20T09:03:56.636 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.637 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.637 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:56.637 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.518153+0000", 2024-03-20T09:03:56.637 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0659999999999998e-06 2024-03-20T09:03:56.637 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.637 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.637 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:56.637 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.578928+0000", 2024-03-20T09:03:56.637 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.060775820000000001 2024-03-20T09:03:56.637 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.637 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.637 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:56.637 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.579024+0000", 2024-03-20T09:03:56.637 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5587999999999994e-05 2024-03-20T09:03:56.637 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.638 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.638 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,1", 2024-03-20T09:03:56.638 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.579129+0000", 2024-03-20T09:03:56.638 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010485299999999999 2024-03-20T09:03:56.638 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.638 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.638 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:56.638 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.587470+0000", 2024-03-20T09:03:56.638 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0083414030000000007 2024-03-20T09:03:56.638 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.638 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.638 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.638 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.618656+0000", 2024-03-20T09:03:56.639 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.031186077999999999 2024-03-20T09:03:56.639 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.639 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.639 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.639 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.662726+0000", 2024-03-20T09:03:56.639 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.044069097000000002 2024-03-20T09:03:56.639 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.639 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.639 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:56.639 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.662762+0000", 2024-03-20T09:03:56.639 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6257999999999997e-05 2024-03-20T09:03:56.639 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.639 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.639 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:56.640 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:51.662819+0000", 2024-03-20T09:03:56.640 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7049999999999998e-05 2024-03-20T09:03:56.640 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.640 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:56.640 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.640 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.640 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.640 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:2722 3.e 3:73746d84:::benchmark_data_smithi122_119671_object2721:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-03-20T09:03:56.640 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:03:02.169404+0000", 2024-03-20T09:03:56.640 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 54.452135875000003, 2024-03-20T09:03:56.640 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.204325325, 2024-03-20T09:03:56.640 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:56.640 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:56.640 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:56.640 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:03:56.641 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:03:56.641 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2722 2024-03-20T09:03:56.641 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.641 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:56.641 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.641 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:56.641 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.169404+0000", 2024-03-20T09:03:56.641 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.641 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.641 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.641 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:56.641 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.169404+0000", 2024-03-20T09:03:56.641 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.641 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.642 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.642 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:56.642 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.169405+0000", 2024-03-20T09:03:56.642 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.23e-07 2024-03-20T09:03:56.642 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.642 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.642 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:56.642 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.169424+0000", 2024-03-20T09:03:56.642 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8627e-05 2024-03-20T09:03:56.642 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.642 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.642 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:56.643 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.169435+0000", 2024-03-20T09:03:56.643 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1072e-05 2024-03-20T09:03:56.643 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.643 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.643 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:56.643 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.169441+0000", 2024-03-20T09:03:56.643 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5320000000000003e-06 2024-03-20T09:03:56.643 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.643 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.643 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:56.643 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.169494+0000", 2024-03-20T09:03:56.643 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2723999999999999e-05 2024-03-20T09:03:56.643 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.643 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.644 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:56.644 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.169594+0000", 2024-03-20T09:03:56.644 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9522e-05 2024-03-20T09:03:56.644 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.644 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.644 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,4", 2024-03-20T09:03:56.644 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.169701+0000", 2024-03-20T09:03:56.644 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000107909 2024-03-20T09:03:56.644 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.644 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.644 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:56.644 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.170382+0000", 2024-03-20T09:03:56.644 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00068083200000000003 2024-03-20T09:03:56.644 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.645 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.645 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.645 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.170465+0000", 2024-03-20T09:03:56.645 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2216e-05 2024-03-20T09:03:56.645 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.645 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.645 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.645 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.373556+0000", 2024-03-20T09:03:56.645 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20309166100000001 2024-03-20T09:03:56.645 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.645 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.645 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:56.646 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.373691+0000", 2024-03-20T09:03:56.646 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013524300000000001 2024-03-20T09:03:56.646 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.646 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.646 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:56.646 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.373730+0000", 2024-03-20T09:03:56.646 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8263999999999998e-05 2024-03-20T09:03:56.646 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.646 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:56.646 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.646 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.646 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.646 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:2729 3.e 3:791f8992:::benchmark_data_smithi122_119671_object2728:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-03-20T09:03:56.647 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:03:02.171204+0000", 2024-03-20T09:03:56.647 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 54.450336153000002, 2024-03-20T09:03:56.647 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20275626899999999, 2024-03-20T09:03:56.647 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:56.647 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:56.647 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:56.647 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:03:56.647 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:03:56.647 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2729 2024-03-20T09:03:56.647 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.647 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:56.647 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.647 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:56.647 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.171204+0000", 2024-03-20T09:03:56.648 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.648 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.648 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.648 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:56.648 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.171204+0000", 2024-03-20T09:03:56.648 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.648 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.648 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.648 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:56.648 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.171205+0000", 2024-03-20T09:03:56.648 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6100000000000001e-07 2024-03-20T09:03:56.648 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.648 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.648 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:56.649 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.171221+0000", 2024-03-20T09:03:56.649 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6745999999999999e-05 2024-03-20T09:03:56.649 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.649 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.649 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:56.649 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.171227+0000", 2024-03-20T09:03:56.649 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2499999999999997e-06 2024-03-20T09:03:56.649 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.649 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.649 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:56.649 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.171230+0000", 2024-03-20T09:03:56.649 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.782e-06 2024-03-20T09:03:56.649 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.649 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.650 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:56.650 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.171277+0000", 2024-03-20T09:03:56.650 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6643e-05 2024-03-20T09:03:56.650 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.650 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.650 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:56.650 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.171407+0000", 2024-03-20T09:03:56.650 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013011699999999999 2024-03-20T09:03:56.650 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.650 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.650 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,4", 2024-03-20T09:03:56.650 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.171513+0000", 2024-03-20T09:03:56.650 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105942 2024-03-20T09:03:56.650 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.651 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.651 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:56.651 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.172517+0000", 2024-03-20T09:03:56.651 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010036299999999999 2024-03-20T09:03:56.651 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.651 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.651 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.651 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.173417+0000", 2024-03-20T09:03:56.651 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00090022300000000002 2024-03-20T09:03:56.651 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.651 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.651 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.651 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.373859+0000", 2024-03-20T09:03:56.652 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.200442447 2024-03-20T09:03:56.652 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.652 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.652 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:56.652 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.373914+0000", 2024-03-20T09:03:56.652 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4314000000000002e-05 2024-03-20T09:03:56.652 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.652 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.652 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:56.652 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.373960+0000", 2024-03-20T09:03:56.652 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6613999999999998e-05 2024-03-20T09:03:56.652 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.652 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:56.652 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.653 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.653 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.653 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:2734 3.e 3:7535f8d9:::benchmark_data_smithi122_119671_object2733:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-03-20T09:03:56.653 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:03:02.171311+0000", 2024-03-20T09:03:56.653 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 54.450229032000003, 2024-03-20T09:03:56.653 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20282460299999999, 2024-03-20T09:03:56.653 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:56.653 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:56.653 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:56.653 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:03:56.653 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:03:56.653 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2734 2024-03-20T09:03:56.653 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.653 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:56.654 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.654 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:56.654 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.171311+0000", 2024-03-20T09:03:56.654 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.654 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.654 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.654 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:56.654 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.171311+0000", 2024-03-20T09:03:56.654 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.654 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.654 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.654 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:56.654 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.171312+0000", 2024-03-20T09:03:56.655 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3000000000000001e-07 2024-03-20T09:03:56.655 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.655 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.655 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:56.655 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.171331+0000", 2024-03-20T09:03:56.655 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.88e-05 2024-03-20T09:03:56.655 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.655 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.655 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:56.655 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.171337+0000", 2024-03-20T09:03:56.655 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6839999999999999e-06 2024-03-20T09:03:56.656 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.656 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.656 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:56.656 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.171343+0000", 2024-03-20T09:03:56.656 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3349999999999997e-06 2024-03-20T09:03:56.656 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.656 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.656 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:56.656 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.171800+0000", 2024-03-20T09:03:56.656 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00045792899999999999 2024-03-20T09:03:56.656 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.656 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.656 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:56.657 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.171931+0000", 2024-03-20T09:03:56.657 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013026300000000001 2024-03-20T09:03:56.657 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.657 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.657 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,4", 2024-03-20T09:03:56.657 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.172068+0000", 2024-03-20T09:03:56.657 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000137677 2024-03-20T09:03:56.657 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.657 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.657 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:56.657 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.172600+0000", 2024-03-20T09:03:56.657 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00053186800000000001 2024-03-20T09:03:56.657 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.658 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.658 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.658 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.173472+0000", 2024-03-20T09:03:56.658 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00087161200000000002 2024-03-20T09:03:56.658 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.658 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.658 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.658 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.374026+0000", 2024-03-20T09:03:56.658 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20055383399999999 2024-03-20T09:03:56.658 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.658 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.658 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:56.658 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.374091+0000", 2024-03-20T09:03:56.659 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5424000000000002e-05 2024-03-20T09:03:56.659 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.659 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.659 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:56.659 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.374136+0000", 2024-03-20T09:03:56.659 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4647000000000001e-05 2024-03-20T09:03:56.659 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.659 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:56.659 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.659 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.659 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.659 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:2747 3.8 3:135d5110:::benchmark_data_smithi122_119671_object2746:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-03-20T09:03:56.659 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:03:02.174976+0000", 2024-03-20T09:03:56.659 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 54.446564441, 2024-03-20T09:03:56.659 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19865374499999999, 2024-03-20T09:03:56.659 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:56.660 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:56.660 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:56.660 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:03:56.660 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:03:56.660 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2747 2024-03-20T09:03:56.660 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.660 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:56.660 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.660 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:56.660 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.174976+0000", 2024-03-20T09:03:56.660 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.660 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.660 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.660 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:56.660 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.174976+0000", 2024-03-20T09:03:56.661 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.661 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.661 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.661 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:56.661 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.174977+0000", 2024-03-20T09:03:56.661 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5600000000000004e-07 2024-03-20T09:03:56.661 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.661 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.661 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:56.661 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.174995+0000", 2024-03-20T09:03:56.661 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8185999999999998e-05 2024-03-20T09:03:56.661 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.661 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.662 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:56.662 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.175006+0000", 2024-03-20T09:03:56.662 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1389000000000001e-05 2024-03-20T09:03:56.662 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.662 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.662 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:56.662 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.175014+0000", 2024-03-20T09:03:56.662 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1079999999999993e-06 2024-03-20T09:03:56.662 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.662 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.662 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:56.662 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.175063+0000", 2024-03-20T09:03:56.662 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8665000000000002e-05 2024-03-20T09:03:56.663 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.663 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.663 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:56.663 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.175189+0000", 2024-03-20T09:03:56.663 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000125415 2024-03-20T09:03:56.663 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.663 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.663 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,4", 2024-03-20T09:03:56.663 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.175341+0000", 2024-03-20T09:03:56.663 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015202400000000001 2024-03-20T09:03:56.663 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.663 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.663 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:56.664 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.175703+0000", 2024-03-20T09:03:56.664 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00036208600000000002 2024-03-20T09:03:56.664 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.665 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.665 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.665 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.176625+0000", 2024-03-20T09:03:56.665 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00092236900000000003 2024-03-20T09:03:56.665 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.665 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.665 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.665 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.373439+0000", 2024-03-20T09:03:56.665 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.196813722 2024-03-20T09:03:56.665 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.665 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.665 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:56.665 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.373552+0000", 2024-03-20T09:03:56.666 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000113251 2024-03-20T09:03:56.666 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.666 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.666 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:56.666 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.373630+0000", 2024-03-20T09:03:56.666 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7574e-05 2024-03-20T09:03:56.666 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.666 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:56.666 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.666 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.666 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.666 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:2760 3.e 3:7d05dc79:::benchmark_data_smithi122_119671_object2759:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-03-20T09:03:56.666 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:03:02.178397+0000", 2024-03-20T09:03:56.667 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 54.443143663000001, 2024-03-20T09:03:56.667 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19658330099999999, 2024-03-20T09:03:56.667 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:56.667 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:56.667 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:56.667 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:03:56.667 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:03:56.667 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2760 2024-03-20T09:03:56.667 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.667 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:56.667 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.667 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:56.667 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.178397+0000", 2024-03-20T09:03:56.667 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.668 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.668 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.668 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:56.668 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.178397+0000", 2024-03-20T09:03:56.668 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.668 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.668 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.668 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:56.668 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.178398+0000", 2024-03-20T09:03:56.668 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2699999999999998e-07 2024-03-20T09:03:56.668 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.668 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.668 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:56.669 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.178417+0000", 2024-03-20T09:03:56.669 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.978e-05 2024-03-20T09:03:56.669 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.669 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.669 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:56.669 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.178430+0000", 2024-03-20T09:03:56.669 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2988e-05 2024-03-20T09:03:56.669 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.669 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.669 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:56.669 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.178438+0000", 2024-03-20T09:03:56.669 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3819999999999998e-06 2024-03-20T09:03:56.669 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.669 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.669 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:56.670 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.178502+0000", 2024-03-20T09:03:56.670 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4113000000000001e-05 2024-03-20T09:03:56.670 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.670 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.670 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:56.670 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.178589+0000", 2024-03-20T09:03:56.670 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7066999999999998e-05 2024-03-20T09:03:56.670 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.670 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.670 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,4", 2024-03-20T09:03:56.670 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.178702+0000", 2024-03-20T09:03:56.670 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000113142 2024-03-20T09:03:56.670 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.671 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.671 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:56.671 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.179118+0000", 2024-03-20T09:03:56.671 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00041638999999999999 2024-03-20T09:03:56.671 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.671 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.671 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.671 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.180589+0000", 2024-03-20T09:03:56.671 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014707489999999999 2024-03-20T09:03:56.671 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.671 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.671 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.671 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.374879+0000", 2024-03-20T09:03:56.671 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19428980600000001 2024-03-20T09:03:56.672 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.672 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.672 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:56.672 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.374939+0000", 2024-03-20T09:03:56.672 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9762999999999997e-05 2024-03-20T09:03:56.672 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.672 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.672 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:56.672 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.374980+0000", 2024-03-20T09:03:56.673 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1193999999999999e-05 2024-03-20T09:03:56.673 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.673 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:56.673 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.673 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.673 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.673 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:2762 3.8 3:1a72c311:::benchmark_data_smithi122_119671_object2761:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-03-20T09:03:56.673 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:03:02.178729+0000", 2024-03-20T09:03:56.673 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 54.442810958999999, 2024-03-20T09:03:56.673 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19541729099999999, 2024-03-20T09:03:56.673 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:56.673 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:56.673 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:56.674 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:03:56.674 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:03:56.674 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2762 2024-03-20T09:03:56.674 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.674 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:56.674 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.674 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:56.674 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.178729+0000", 2024-03-20T09:03:56.674 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.674 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.674 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.674 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:56.674 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.178729+0000", 2024-03-20T09:03:56.674 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.675 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.675 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.675 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:56.675 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.178730+0000", 2024-03-20T09:03:56.675 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7599999999999996e-07 2024-03-20T09:03:56.675 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.675 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.675 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:56.675 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.178751+0000", 2024-03-20T09:03:56.675 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0828000000000001e-05 2024-03-20T09:03:56.675 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.675 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.675 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:56.675 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.178761+0000", 2024-03-20T09:03:56.676 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6239999999999997e-06 2024-03-20T09:03:56.676 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.676 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.676 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:56.676 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.178768+0000", 2024-03-20T09:03:56.676 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1080000000000004e-06 2024-03-20T09:03:56.676 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.676 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.676 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:56.677 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.178809+0000", 2024-03-20T09:03:56.677 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1133000000000001e-05 2024-03-20T09:03:56.677 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.677 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.677 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:56.677 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.178896+0000", 2024-03-20T09:03:56.677 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7102999999999996e-05 2024-03-20T09:03:56.677 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.677 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.677 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,4", 2024-03-20T09:03:56.677 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.178981+0000", 2024-03-20T09:03:56.677 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5328999999999996e-05 2024-03-20T09:03:56.677 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.678 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.678 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:56.678 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.179386+0000", 2024-03-20T09:03:56.678 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00040516899999999998 2024-03-20T09:03:56.678 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.678 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.678 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.678 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.180852+0000", 2024-03-20T09:03:56.678 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001465963 2024-03-20T09:03:56.678 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.678 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.678 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.678 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.374032+0000", 2024-03-20T09:03:56.678 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19317979699999999 2024-03-20T09:03:56.679 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.679 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.679 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:56.679 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.374097+0000", 2024-03-20T09:03:56.679 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5159000000000004e-05 2024-03-20T09:03:56.679 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.679 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.679 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:56.679 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.374147+0000", 2024-03-20T09:03:56.679 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9202000000000001e-05 2024-03-20T09:03:56.679 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.679 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:56.679 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.680 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.680 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.680 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:2776 3.e 3:7bb8a6f8:::benchmark_data_smithi122_119671_object2775:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-03-20T09:03:56.680 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:03:02.181315+0000", 2024-03-20T09:03:56.680 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 54.440225292000001, 2024-03-20T09:03:56.680 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.193822832, 2024-03-20T09:03:56.680 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:56.680 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:56.680 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:56.680 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:03:56.680 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:03:56.680 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2776 2024-03-20T09:03:56.680 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.681 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:56.681 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.681 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:56.681 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.181315+0000", 2024-03-20T09:03:56.681 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.681 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.681 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.681 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:56.681 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.181315+0000", 2024-03-20T09:03:56.681 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.681 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.681 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.681 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:56.681 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.181316+0000", 2024-03-20T09:03:56.682 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7599999999999996e-07 2024-03-20T09:03:56.682 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.682 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.682 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:56.682 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.181334+0000", 2024-03-20T09:03:56.682 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7713000000000001e-05 2024-03-20T09:03:56.682 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.682 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.682 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:56.682 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.181345+0000", 2024-03-20T09:03:56.682 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1291000000000001e-05 2024-03-20T09:03:56.682 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.682 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.682 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:56.683 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.181350+0000", 2024-03-20T09:03:56.683 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8500000000000002e-06 2024-03-20T09:03:56.683 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.683 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.683 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:56.683 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.181399+0000", 2024-03-20T09:03:56.683 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9061999999999999e-05 2024-03-20T09:03:56.683 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.683 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.683 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:56.683 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.181490+0000", 2024-03-20T09:03:56.683 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1607999999999994e-05 2024-03-20T09:03:56.683 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.683 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.683 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,4", 2024-03-20T09:03:56.684 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.181600+0000", 2024-03-20T09:03:56.684 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000110048 2024-03-20T09:03:56.684 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.684 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.684 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:56.684 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.182251+0000", 2024-03-20T09:03:56.684 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00065016599999999998 2024-03-20T09:03:56.684 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.684 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.684 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.684 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.183057+0000", 2024-03-20T09:03:56.684 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00080644699999999996 2024-03-20T09:03:56.684 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.684 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.685 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.685 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.375040+0000", 2024-03-20T09:03:56.685 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19198335999999999 2024-03-20T09:03:56.685 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.685 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.685 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:56.685 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.375097+0000", 2024-03-20T09:03:56.685 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6925999999999997e-05 2024-03-20T09:03:56.685 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.685 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.685 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:56.685 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.375138+0000", 2024-03-20T09:03:56.686 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0485000000000003e-05 2024-03-20T09:03:56.686 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.686 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:56.686 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.686 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.686 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.686 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:2784 3.8 3:1c8a57b1:::benchmark_data_smithi122_119671_object2783:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-03-20T09:03:56.686 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:03:02.183233+0000", 2024-03-20T09:03:56.686 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 54.438307719999997, 2024-03-20T09:03:56.686 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.191073138, 2024-03-20T09:03:56.686 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:56.686 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:56.687 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:56.687 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:03:56.687 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:03:56.687 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2784 2024-03-20T09:03:56.687 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.687 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:56.687 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.687 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:56.687 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.183233+0000", 2024-03-20T09:03:56.687 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.687 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.687 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.687 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:56.687 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.183233+0000", 2024-03-20T09:03:56.688 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.688 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.688 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.688 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:56.688 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.183233+0000", 2024-03-20T09:03:56.688 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1500000000000004e-07 2024-03-20T09:03:56.688 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.688 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.688 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:56.688 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.183251+0000", 2024-03-20T09:03:56.688 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8241e-05 2024-03-20T09:03:56.688 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.689 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.689 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:56.689 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.183259+0000", 2024-03-20T09:03:56.689 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5800000000000003e-06 2024-03-20T09:03:56.689 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.689 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.689 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:56.689 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.183275+0000", 2024-03-20T09:03:56.689 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6419e-05 2024-03-20T09:03:56.689 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.689 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.690 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:56.690 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.183329+0000", 2024-03-20T09:03:56.690 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3400999999999999e-05 2024-03-20T09:03:56.690 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.690 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.690 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:56.690 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.183452+0000", 2024-03-20T09:03:56.690 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012280799999999999 2024-03-20T09:03:56.690 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.690 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.690 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,4", 2024-03-20T09:03:56.690 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.183597+0000", 2024-03-20T09:03:56.690 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014497699999999999 2024-03-20T09:03:56.690 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.691 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.691 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:56.691 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.184179+0000", 2024-03-20T09:03:56.691 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00058238400000000003 2024-03-20T09:03:56.691 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.691 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.691 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.691 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.184637+0000", 2024-03-20T09:03:56.691 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00045812700000000001 2024-03-20T09:03:56.691 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.691 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.691 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.692 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.374204+0000", 2024-03-20T09:03:56.692 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.189567289 2024-03-20T09:03:56.692 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.692 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.692 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:56.692 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.374255+0000", 2024-03-20T09:03:56.692 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0213000000000002e-05 2024-03-20T09:03:56.692 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.692 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.692 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:56.693 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.374306+0000", 2024-03-20T09:03:56.693 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1084000000000001e-05 2024-03-20T09:03:56.693 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.693 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:56.693 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.693 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.693 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.693 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:2791 3.e 3:79e056e0:::benchmark_data_smithi122_119671_object2790:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-03-20T09:03:56.693 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:03:02.186501+0000", 2024-03-20T09:03:56.693 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 54.435039553999999, 2024-03-20T09:03:56.693 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18877754799999999, 2024-03-20T09:03:56.693 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:56.694 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:56.694 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:56.694 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:03:56.694 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:03:56.694 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2791 2024-03-20T09:03:56.694 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.694 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:56.694 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.694 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:56.694 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.186501+0000", 2024-03-20T09:03:56.694 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.694 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.694 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.694 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:56.695 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.186501+0000", 2024-03-20T09:03:56.695 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.695 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.695 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.695 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:56.695 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.186502+0000", 2024-03-20T09:03:56.695 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1800000000000004e-07 2024-03-20T09:03:56.695 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.695 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.695 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:56.695 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.186518+0000", 2024-03-20T09:03:56.695 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6551000000000001e-05 2024-03-20T09:03:56.695 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.695 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.696 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:56.696 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.186528+0000", 2024-03-20T09:03:56.696 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0311e-05 2024-03-20T09:03:56.696 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.696 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.696 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:56.696 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.186534+0000", 2024-03-20T09:03:56.696 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3539999999999999e-06 2024-03-20T09:03:56.696 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.696 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.696 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:56.696 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.186622+0000", 2024-03-20T09:03:56.696 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8361000000000003e-05 2024-03-20T09:03:56.697 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.697 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.697 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:56.697 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.186774+0000", 2024-03-20T09:03:56.697 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000152244 2024-03-20T09:03:56.697 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.697 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.697 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,4", 2024-03-20T09:03:56.697 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.186947+0000", 2024-03-20T09:03:56.697 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00017305000000000001 2024-03-20T09:03:56.697 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.697 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.697 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:56.697 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.187796+0000", 2024-03-20T09:03:56.698 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00084884900000000004 2024-03-20T09:03:56.698 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.698 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.698 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.698 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.188442+0000", 2024-03-20T09:03:56.698 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00064533899999999998 2024-03-20T09:03:56.698 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.698 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.698 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.698 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.375195+0000", 2024-03-20T09:03:56.698 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.186752894 2024-03-20T09:03:56.698 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.698 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.699 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:56.699 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.375244+0000", 2024-03-20T09:03:56.699 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9357000000000002e-05 2024-03-20T09:03:56.699 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.699 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.699 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:56.699 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.375278+0000", 2024-03-20T09:03:56.699 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4319999999999997e-05 2024-03-20T09:03:56.699 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.699 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:56.699 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.699 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.699 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.700 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:2797 3.e 3:7fb7a927:::benchmark_data_smithi122_119671_object2796:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-03-20T09:03:56.700 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:03:02.187238+0000", 2024-03-20T09:03:56.700 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 54.434301892000001, 2024-03-20T09:03:56.700 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.188219831, 2024-03-20T09:03:56.700 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:56.700 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:56.700 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:56.700 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:03:56.700 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:03:56.700 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2797 2024-03-20T09:03:56.700 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.700 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:56.700 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.700 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:56.701 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.187238+0000", 2024-03-20T09:03:56.701 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.701 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.701 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.701 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:56.701 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.187238+0000", 2024-03-20T09:03:56.701 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.701 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.701 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.702 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:56.702 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.187239+0000", 2024-03-20T09:03:56.702 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0699999999999996e-07 2024-03-20T09:03:56.702 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.702 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.702 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:56.702 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.187256+0000", 2024-03-20T09:03:56.702 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6803000000000002e-05 2024-03-20T09:03:56.702 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.702 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.702 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:56.702 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.187269+0000", 2024-03-20T09:03:56.703 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2877999999999999e-05 2024-03-20T09:03:56.703 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.703 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.703 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:56.703 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.187278+0000", 2024-03-20T09:03:56.703 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3589999999999997e-06 2024-03-20T09:03:56.703 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.703 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.704 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:56.704 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.187332+0000", 2024-03-20T09:03:56.705 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4011000000000002e-05 2024-03-20T09:03:56.705 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.705 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.705 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:56.706 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.187422+0000", 2024-03-20T09:03:56.706 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.975e-05 2024-03-20T09:03:56.706 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.706 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.706 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,4", 2024-03-20T09:03:56.706 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.187516+0000", 2024-03-20T09:03:56.706 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4011999999999994e-05 2024-03-20T09:03:56.706 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.706 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.706 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:56.706 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.188391+0000", 2024-03-20T09:03:56.706 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00087528699999999998 2024-03-20T09:03:56.706 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.707 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.707 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.707 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.188907+0000", 2024-03-20T09:03:56.707 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00051633699999999996 2024-03-20T09:03:56.707 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.707 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.707 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.707 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.375339+0000", 2024-03-20T09:03:56.707 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18643184099999999 2024-03-20T09:03:56.707 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.707 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.707 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:56.707 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.375403+0000", 2024-03-20T09:03:56.707 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3361999999999994e-05 2024-03-20T09:03:56.707 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.708 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.708 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:56.708 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.375458+0000", 2024-03-20T09:03:56.708 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5483999999999999e-05 2024-03-20T09:03:56.708 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.708 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:56.708 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.708 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.708 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.708 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:2798 3.e 3:7ea16f80:::benchmark_data_smithi122_119671_object2797:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-03-20T09:03:56.708 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:03:02.187834+0000", 2024-03-20T09:03:56.708 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 54.433706342999997, 2024-03-20T09:03:56.708 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18778505100000001, 2024-03-20T09:03:56.708 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:56.709 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:56.709 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:56.709 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:03:56.709 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:03:56.709 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2798 2024-03-20T09:03:56.709 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.709 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:56.709 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.709 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:56.709 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.187834+0000", 2024-03-20T09:03:56.709 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.709 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.709 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.709 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:56.709 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.187834+0000", 2024-03-20T09:03:56.710 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.710 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.710 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.710 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:56.710 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.187835+0000", 2024-03-20T09:03:56.710 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3399999999999998e-07 2024-03-20T09:03:56.710 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.710 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.710 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:56.710 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.187852+0000", 2024-03-20T09:03:56.710 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7234000000000001e-05 2024-03-20T09:03:56.710 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.710 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.710 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:56.711 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.187859+0000", 2024-03-20T09:03:56.711 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5000000000000002e-06 2024-03-20T09:03:56.711 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.711 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.711 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:56.711 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.187872+0000", 2024-03-20T09:03:56.711 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3016999999999999e-05 2024-03-20T09:03:56.711 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.711 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.711 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:56.711 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.187919+0000", 2024-03-20T09:03:56.711 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6326999999999999e-05 2024-03-20T09:03:56.711 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.711 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.711 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:56.712 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.188023+0000", 2024-03-20T09:03:56.712 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010461199999999999 2024-03-20T09:03:56.712 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.712 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.712 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,4", 2024-03-20T09:03:56.712 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.188142+0000", 2024-03-20T09:03:56.712 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000118309 2024-03-20T09:03:56.712 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.712 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.712 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:56.712 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.188564+0000", 2024-03-20T09:03:56.712 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00042198799999999999 2024-03-20T09:03:56.712 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.712 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.713 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.713 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.189112+0000", 2024-03-20T09:03:56.713 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00054863500000000001 2024-03-20T09:03:56.713 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.713 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.713 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.713 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.375527+0000", 2024-03-20T09:03:56.713 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.186414408 2024-03-20T09:03:56.713 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.713 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.713 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:56.713 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.375575+0000", 2024-03-20T09:03:56.713 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8613999999999999e-05 2024-03-20T09:03:56.713 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.713 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.714 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:56.714 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.375619+0000", 2024-03-20T09:03:56.714 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3673e-05 2024-03-20T09:03:56.714 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.714 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:56.714 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.714 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.714 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.714 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:2803 3.8 3:1fc8c2c7:::benchmark_data_smithi122_119671_object2802:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-03-20T09:03:56.714 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:03:02.189426+0000", 2024-03-20T09:03:56.714 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 54.432114134999999, 2024-03-20T09:03:56.714 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18500308700000001, 2024-03-20T09:03:56.714 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:56.715 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:56.715 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:56.715 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:03:56.715 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:03:56.715 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2803 2024-03-20T09:03:56.715 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.715 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:56.715 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.716 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:56.716 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.189426+0000", 2024-03-20T09:03:56.716 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.716 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.716 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.716 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:56.716 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.189426+0000", 2024-03-20T09:03:56.716 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.716 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.716 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.716 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:56.716 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.189427+0000", 2024-03-20T09:03:56.716 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7100000000000002e-07 2024-03-20T09:03:56.716 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.717 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.717 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:56.717 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.189471+0000", 2024-03-20T09:03:56.717 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4688000000000003e-05 2024-03-20T09:03:56.717 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.717 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.717 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:56.717 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.189480+0000", 2024-03-20T09:03:56.717 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2789999999999998e-06 2024-03-20T09:03:56.717 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.717 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.717 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:56.717 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.189486+0000", 2024-03-20T09:03:56.717 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9900000000000002e-06 2024-03-20T09:03:56.718 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.718 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.718 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:56.718 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.189527+0000", 2024-03-20T09:03:56.718 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0975e-05 2024-03-20T09:03:56.718 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.718 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.718 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:56.718 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.189702+0000", 2024-03-20T09:03:56.718 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00017593599999999999 2024-03-20T09:03:56.718 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.718 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.718 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,4", 2024-03-20T09:03:56.718 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.189837+0000", 2024-03-20T09:03:56.719 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000134511 2024-03-20T09:03:56.719 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.719 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.719 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:56.719 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.190299+0000", 2024-03-20T09:03:56.719 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00046235999999999998 2024-03-20T09:03:56.719 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.719 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.719 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.719 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.190659+0000", 2024-03-20T09:03:56.719 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00035968699999999997 2024-03-20T09:03:56.719 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.719 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.719 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.719 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.374368+0000", 2024-03-20T09:03:56.720 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18370908499999999 2024-03-20T09:03:56.720 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.720 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.720 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:56.720 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.374405+0000", 2024-03-20T09:03:56.720 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.667e-05 2024-03-20T09:03:56.720 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.720 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.720 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:56.720 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.374429+0000", 2024-03-20T09:03:56.720 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4434999999999999e-05 2024-03-20T09:03:56.720 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.720 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:56.720 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.721 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.721 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.721 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:2806 3.8 3:17720c29:::benchmark_data_smithi122_119671_object2805:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-03-20T09:03:56.721 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:03:02.190720+0000", 2024-03-20T09:03:56.721 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 54.430819984000003, 2024-03-20T09:03:56.721 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18384916400000001, 2024-03-20T09:03:56.721 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:56.721 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:56.721 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:56.721 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:03:56.721 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:03:56.721 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2806 2024-03-20T09:03:56.721 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.721 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:56.721 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.722 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:56.722 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.190720+0000", 2024-03-20T09:03:56.722 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.722 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.722 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.722 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:56.722 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.190720+0000", 2024-03-20T09:03:56.722 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.722 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.722 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.722 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:56.722 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.190721+0000", 2024-03-20T09:03:56.722 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6400000000000001e-07 2024-03-20T09:03:56.723 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.723 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.723 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:56.723 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.190743+0000", 2024-03-20T09:03:56.723 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1619000000000001e-05 2024-03-20T09:03:56.723 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.723 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.723 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:56.723 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.190753+0000", 2024-03-20T09:03:56.723 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0399e-05 2024-03-20T09:03:56.723 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.723 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.723 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:56.723 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.190759+0000", 2024-03-20T09:03:56.723 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9610000000000002e-06 2024-03-20T09:03:56.724 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.724 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.724 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:56.724 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.190799+0000", 2024-03-20T09:03:56.724 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9786999999999997e-05 2024-03-20T09:03:56.724 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.724 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.724 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:56.724 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.190888+0000", 2024-03-20T09:03:56.724 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9103000000000004e-05 2024-03-20T09:03:56.724 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.724 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.724 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,4", 2024-03-20T09:03:56.724 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.190973+0000", 2024-03-20T09:03:56.724 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5117999999999994e-05 2024-03-20T09:03:56.725 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.725 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.725 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:56.725 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.191330+0000", 2024-03-20T09:03:56.725 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000356652 2024-03-20T09:03:56.725 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.725 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.725 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.725 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.191726+0000", 2024-03-20T09:03:56.725 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00039610600000000002 2024-03-20T09:03:56.725 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.725 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.725 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.726 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.374481+0000", 2024-03-20T09:03:56.726 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18275464 2024-03-20T09:03:56.726 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.726 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.726 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:56.726 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.374531+0000", 2024-03-20T09:03:56.726 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0367000000000001e-05 2024-03-20T09:03:56.726 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.726 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.727 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:56.727 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.374569+0000", 2024-03-20T09:03:56.727 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8547999999999998e-05 2024-03-20T09:03:56.727 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.727 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:56.727 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.727 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.727 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.727 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:2813 3.8 3:1b56fc87:::benchmark_data_smithi122_119671_object2812:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-03-20T09:03:56.727 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:03:02.193415+0000", 2024-03-20T09:03:56.727 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 54.428125321000003, 2024-03-20T09:03:56.727 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.181444946, 2024-03-20T09:03:56.727 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:56.727 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:56.728 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:56.728 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:03:56.728 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:03:56.728 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2813 2024-03-20T09:03:56.728 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.728 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:56.728 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.728 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:56.728 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.193415+0000", 2024-03-20T09:03:56.728 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.728 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.728 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.728 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:56.728 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.193415+0000", 2024-03-20T09:03:56.729 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.729 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.729 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.729 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:56.729 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.193416+0000", 2024-03-20T09:03:56.729 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.47e-07 2024-03-20T09:03:56.729 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.729 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.729 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:56.729 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.193440+0000", 2024-03-20T09:03:56.729 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4142e-05 2024-03-20T09:03:56.729 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.729 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.729 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:56.730 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.193453+0000", 2024-03-20T09:03:56.730 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2719e-05 2024-03-20T09:03:56.730 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.730 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.730 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:56.730 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.193473+0000", 2024-03-20T09:03:56.730 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0352000000000001e-05 2024-03-20T09:03:56.730 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.730 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.730 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:56.730 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.193517+0000", 2024-03-20T09:03:56.730 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3807999999999998e-05 2024-03-20T09:03:56.730 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.730 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.731 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:56.731 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.193590+0000", 2024-03-20T09:03:56.731 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3415000000000001e-05 2024-03-20T09:03:56.731 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.731 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.731 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,4", 2024-03-20T09:03:56.731 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.193675+0000", 2024-03-20T09:03:56.731 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4783999999999993e-05 2024-03-20T09:03:56.731 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.731 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.731 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:56.731 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.194096+0000", 2024-03-20T09:03:56.731 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00042082000000000003 2024-03-20T09:03:56.731 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.732 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.732 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.732 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.194536+0000", 2024-03-20T09:03:56.732 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000440216 2024-03-20T09:03:56.732 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.732 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.732 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.732 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.374768+0000", 2024-03-20T09:03:56.732 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.180231481 2024-03-20T09:03:56.732 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.732 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.732 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:56.732 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.374820+0000", 2024-03-20T09:03:56.732 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2092999999999997e-05 2024-03-20T09:03:56.732 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.733 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.733 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:56.733 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.374860+0000", 2024-03-20T09:03:56.733 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0169000000000001e-05 2024-03-20T09:03:56.733 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.733 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:56.733 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.733 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.733 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.733 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:2821 3.e 3:78bd3bb9:::benchmark_data_smithi122_119671_object2820:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-03-20T09:03:56.733 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:03:02.198401+0000", 2024-03-20T09:03:56.733 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 54.423139712999998, 2024-03-20T09:03:56.733 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.17733315899999999, 2024-03-20T09:03:56.733 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:56.734 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:56.734 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:56.734 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:03:56.734 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:03:56.734 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2821 2024-03-20T09:03:56.734 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.734 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:56.734 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.734 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:56.734 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.198401+0000", 2024-03-20T09:03:56.734 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.734 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.734 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.735 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:56.735 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.198401+0000", 2024-03-20T09:03:56.735 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.735 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.735 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.735 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:56.735 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.198401+0000", 2024-03-20T09:03:56.735 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8800000000000002e-07 2024-03-20T09:03:56.735 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.735 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.735 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:56.735 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.198418+0000", 2024-03-20T09:03:56.735 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7108e-05 2024-03-20T09:03:56.735 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.735 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.736 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:56.736 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.198429+0000", 2024-03-20T09:03:56.736 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0316000000000001e-05 2024-03-20T09:03:56.736 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.736 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.736 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:56.736 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.198437+0000", 2024-03-20T09:03:56.736 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7859999999999997e-06 2024-03-20T09:03:56.736 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.736 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.736 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:56.736 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.198488+0000", 2024-03-20T09:03:56.736 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1078000000000001e-05 2024-03-20T09:03:56.737 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.737 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.737 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:56.737 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.198571+0000", 2024-03-20T09:03:56.737 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3300999999999994e-05 2024-03-20T09:03:56.737 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.737 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.737 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,4", 2024-03-20T09:03:56.737 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.198659+0000", 2024-03-20T09:03:56.737 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8394999999999996e-05 2024-03-20T09:03:56.737 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.737 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.737 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:56.737 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.198996+0000", 2024-03-20T09:03:56.737 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00033649299999999999 2024-03-20T09:03:56.738 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.738 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.738 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.738 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.199346+0000", 2024-03-20T09:03:56.738 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00035049000000000001 2024-03-20T09:03:56.738 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.738 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.738 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.738 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.375672+0000", 2024-03-20T09:03:56.738 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.176326013 2024-03-20T09:03:56.738 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.738 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.738 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:56.738 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.375712+0000", 2024-03-20T09:03:56.739 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9576999999999997e-05 2024-03-20T09:03:56.739 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.739 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.739 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:56.739 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.375734+0000", 2024-03-20T09:03:56.739 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1814000000000002e-05 2024-03-20T09:03:56.739 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.739 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:56.739 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.739 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.739 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.739 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:2824 3.e 3:7c5ee3b3:::benchmark_data_smithi122_119671_object2823:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-03-20T09:03:56.739 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:03:02.201753+0000", 2024-03-20T09:03:56.739 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 54.41978701, 2024-03-20T09:03:56.740 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.17413394400000001, 2024-03-20T09:03:56.740 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:56.740 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:56.740 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:56.740 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:03:56.740 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:03:56.740 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 2824 2024-03-20T09:03:56.740 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.740 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:56.740 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.740 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:56.740 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.201753+0000", 2024-03-20T09:03:56.740 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.740 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.740 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.741 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:56.741 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.201753+0000", 2024-03-20T09:03:56.741 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.741 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.741 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.741 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:56.741 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.201754+0000", 2024-03-20T09:03:56.741 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4099999999999998e-07 2024-03-20T09:03:56.741 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.741 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.741 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:56.741 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.201770+0000", 2024-03-20T09:03:56.741 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6291999999999999e-05 2024-03-20T09:03:56.741 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.741 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.742 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:56.742 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.201779+0000", 2024-03-20T09:03:56.742 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0710000000000001e-06 2024-03-20T09:03:56.742 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.742 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.742 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:56.742 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.201785+0000", 2024-03-20T09:03:56.742 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3170000000000001e-06 2024-03-20T09:03:56.742 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.742 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.742 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:56.742 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.201830+0000", 2024-03-20T09:03:56.742 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5015000000000002e-05 2024-03-20T09:03:56.743 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.743 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.743 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:56.743 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.201909+0000", 2024-03-20T09:03:56.743 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9519999999999998e-05 2024-03-20T09:03:56.743 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.743 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.743 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,4", 2024-03-20T09:03:56.743 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.201996+0000", 2024-03-20T09:03:56.743 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6380999999999998e-05 2024-03-20T09:03:56.743 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.743 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.743 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:56.743 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.202307+0000", 2024-03-20T09:03:56.743 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000311708 2024-03-20T09:03:56.744 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.744 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.744 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.744 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.202592+0000", 2024-03-20T09:03:56.744 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00028430000000000003 2024-03-20T09:03:56.744 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.744 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.744 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.744 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.375792+0000", 2024-03-20T09:03:56.744 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.17320011099999999 2024-03-20T09:03:56.744 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.744 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.744 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:56.744 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.375846+0000", 2024-03-20T09:03:56.745 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3949000000000001e-05 2024-03-20T09:03:56.745 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.745 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.745 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:56.745 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:02.375887+0000", 2024-03-20T09:03:56.745 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1539000000000003e-05 2024-03-20T09:03:56.745 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.745 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:56.745 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.745 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.745 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.745 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:69772 3.15 3:a9b85009:::benchmark_data_smithi122_119671_object69771:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-03-20T09:03:56.745 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:03:51.409951+0000", 2024-03-20T09:03:56.745 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.2115887819999998, 2024-03-20T09:03:56.745 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.152158235, 2024-03-20T09:03:56.746 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:56.746 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:56.746 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:56.746 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:03:56.746 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:03:56.746 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 69772 2024-03-20T09:03:56.746 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.746 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:56.746 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.746 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:56.746 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:51.409951+0000", 2024-03-20T09:03:56.746 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.746 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.746 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.747 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:56.747 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:51.409951+0000", 2024-03-20T09:03:56.747 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.747 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.747 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.747 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:56.747 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:51.409953+0000", 2024-03-20T09:03:56.747 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.522e-06 2024-03-20T09:03:56.747 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.747 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.747 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:56.747 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:51.409976+0000", 2024-03-20T09:03:56.747 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3016000000000001e-05 2024-03-20T09:03:56.747 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.748 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.748 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:56.748 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:51.409989+0000", 2024-03-20T09:03:56.748 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3005e-05 2024-03-20T09:03:56.748 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.748 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.748 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:56.748 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:51.409998+0000", 2024-03-20T09:03:56.748 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.704e-06 2024-03-20T09:03:56.748 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.748 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.748 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:56.748 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:51.461052+0000", 2024-03-20T09:03:56.748 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.051054233999999997 2024-03-20T09:03:56.749 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.749 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.749 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:56.749 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:51.464010+0000", 2024-03-20T09:03:56.749 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0029581329999999999 2024-03-20T09:03:56.749 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.749 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.749 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 4,7", 2024-03-20T09:03:56.749 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:51.468991+0000", 2024-03-20T09:03:56.749 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0049811159999999998 2024-03-20T09:03:56.749 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.749 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.749 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.749 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:51.522181+0000", 2024-03-20T09:03:56.750 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.053190119000000001 2024-03-20T09:03:56.750 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.750 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.750 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:56.750 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:51.522656+0000", 2024-03-20T09:03:56.750 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00047508899999999999 2024-03-20T09:03:56.750 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.750 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.750 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.750 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:51.525116+0000", 2024-03-20T09:03:56.750 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0024601010000000001 2024-03-20T09:03:56.750 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.750 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.750 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:56.751 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:51.526822+0000", 2024-03-20T09:03:56.751 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0017052059999999999 2024-03-20T09:03:56.751 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.751 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.751 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:56.751 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:51.562110+0000", 2024-03-20T09:03:56.751 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.035287989999999998 2024-03-20T09:03:56.751 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.751 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:56.751 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.751 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.751 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.751 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:69777 3.15 3:a83e9080:::benchmark_data_smithi122_119671_object69776:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-03-20T09:03:56.751 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:03:51.414194+0000", 2024-03-20T09:03:56.752 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.2073459260000003, 2024-03-20T09:03:56.752 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.156580622, 2024-03-20T09:03:56.752 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:03:56.752 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:03:56.752 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:03:56.752 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:03:56.752 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:03:56.752 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 69777 2024-03-20T09:03:56.752 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.752 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:03:56.752 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.752 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:03:56.752 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:51.414194+0000", 2024-03-20T09:03:56.752 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.752 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.753 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.753 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:03:56.753 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:51.414194+0000", 2024-03-20T09:03:56.753 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:03:56.753 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.753 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.753 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:03:56.753 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:51.414195+0000", 2024-03-20T09:03:56.753 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.048e-06 2024-03-20T09:03:56.753 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.753 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.753 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:03:56.753 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:51.414214+0000", 2024-03-20T09:03:56.753 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8827000000000001e-05 2024-03-20T09:03:56.753 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.754 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.754 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:03:56.754 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:51.414226+0000", 2024-03-20T09:03:56.754 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1814000000000001e-05 2024-03-20T09:03:56.754 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.754 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.754 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:03:56.754 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:51.414234+0000", 2024-03-20T09:03:56.754 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4190000000000002e-06 2024-03-20T09:03:56.754 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.754 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.754 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:03:56.754 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:51.471744+0000", 2024-03-20T09:03:56.754 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.057509823000000002 2024-03-20T09:03:56.755 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.755 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.755 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:03:56.755 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:51.475187+0000", 2024-03-20T09:03:56.755 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0034430939999999998 2024-03-20T09:03:56.755 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.755 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.755 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 4,7", 2024-03-20T09:03:56.755 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:51.479710+0000", 2024-03-20T09:03:56.755 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0045228059999999999 2024-03-20T09:03:56.755 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.755 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.755 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:03:56.755 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:51.522933+0000", 2024-03-20T09:03:56.756 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.043222536999999998 2024-03-20T09:03:56.756 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.756 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.756 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.756 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:51.523825+0000", 2024-03-20T09:03:56.756 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00089238000000000002 2024-03-20T09:03:56.756 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.756 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.756 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:03:56.756 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:51.562902+0000", 2024-03-20T09:03:56.756 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.039077293999999999 2024-03-20T09:03:56.756 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.756 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.756 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:03:56.756 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:51.564576+0000", 2024-03-20T09:03:56.757 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0016731630000000001 2024-03-20T09:03:56.757 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:03:56.757 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:03:56.757 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:03:56.757 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:51.570775+0000", 2024-03-20T09:03:56.757 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0061994169999999996 2024-03-20T09:03:56.757 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.757 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:56.757 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.757 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:03:56.757 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:03:56.757 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:03:57.358 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:56 smithi122 ceph-mon[73495]: 1.6 scrub starts 2024-03-20T09:03:57.359 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:56 smithi122 ceph-mon[73495]: 1.6 scrub ok 2024-03-20T09:03:57.359 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:56 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:03:57.359 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:56 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:03:57.359 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:56 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-03-20T09:03:57.359 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:56 smithi122 ceph-mon[73495]: osdmap e250: 12 total, 12 up, 12 in 2024-03-20T09:03:57.359 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:56 smithi122 ceph-mon[73495]: 1.8 scrub starts 2024-03-20T09:03:57.359 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:56 smithi122 ceph-mon[73495]: 1.8 scrub ok 2024-03-20T09:03:57.359 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:56 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 9186/215346 objects degraded (4.266%), 3 pgs degraded (PG_DEGRADED) 2024-03-20T09:03:57.359 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:56 smithi122 ceph-mon[73495]: osdmap e251: 12 total, 12 up, 12 in 2024-03-20T09:03:57.400 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-20T09:03:57.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:56 smithi115 ceph-mon[72476]: 1.6 scrub starts 2024-03-20T09:03:57.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:56 smithi115 ceph-mon[72476]: 1.6 scrub ok 2024-03-20T09:03:57.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:56 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:03:57.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:56 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:03:57.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:56 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-03-20T09:03:57.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:56 smithi115 ceph-mon[72476]: osdmap e250: 12 total, 12 up, 12 in 2024-03-20T09:03:57.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:56 smithi115 ceph-mon[72476]: 1.8 scrub starts 2024-03-20T09:03:57.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:56 smithi115 ceph-mon[72476]: 1.8 scrub ok 2024-03-20T09:03:57.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:56 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 9186/215346 objects degraded (4.266%), 3 pgs degraded (PG_DEGRADED) 2024-03-20T09:03:57.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:56 smithi115 ceph-mon[72476]: osdmap e251: 12 total, 12 up, 12 in 2024-03-20T09:03:57.435 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:56 smithi026 ceph-mon[68309]: 1.6 scrub starts 2024-03-20T09:03:57.436 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:56 smithi026 ceph-mon[68309]: 1.6 scrub ok 2024-03-20T09:03:57.436 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:56 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:03:57.436 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:56 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:03:57.436 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:56 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-03-20T09:03:57.436 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:56 smithi026 ceph-mon[68309]: osdmap e250: 12 total, 12 up, 12 in 2024-03-20T09:03:57.436 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:56 smithi026 ceph-mon[68309]: 1.8 scrub starts 2024-03-20T09:03:57.436 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:56 smithi026 ceph-mon[68309]: 1.8 scrub ok 2024-03-20T09:03:57.436 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:56 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 9186/215346 objects degraded (4.266%), 3 pgs degraded (PG_DEGRADED) 2024-03-20T09:03:57.436 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:56 smithi026 ceph-mon[68309]: osdmap e251: 12 total, 12 up, 12 in 2024-03-20T09:03:57.610 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 56 16 74571 74555 83.1958 48.0625 0.0142591 0.0120142 2024-03-20T09:03:58.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:57 smithi026 ceph-mon[68309]: 1.9 scrub starts 2024-03-20T09:03:58.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:57 smithi026 ceph-mon[68309]: 1.9 scrub ok 2024-03-20T09:03:58.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:57 smithi026 ceph-mon[68309]: 3.17 scrub starts 2024-03-20T09:03:58.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:57 smithi026 ceph-mon[68309]: pgmap v672: 37 pgs: 1 peering, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 29 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s; 9186/219129 objects degraded (4.192%); 7836/219129 objects misplaced (3.576%); 45 MiB/s, 715 objects/s recovering 2024-03-20T09:03:58.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:57 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-20T09:03:58.370 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:03:58.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:58.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:03:58.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:57 smithi115 ceph-mon[72476]: 1.9 scrub starts 2024-03-20T09:03:58.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:57 smithi115 ceph-mon[72476]: 1.9 scrub ok 2024-03-20T09:03:58.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:57 smithi115 ceph-mon[72476]: 3.17 scrub starts 2024-03-20T09:03:58.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:57 smithi115 ceph-mon[72476]: pgmap v672: 37 pgs: 1 peering, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 29 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s; 9186/219129 objects degraded (4.192%); 7836/219129 objects misplaced (3.576%); 45 MiB/s, 715 objects/s recovering 2024-03-20T09:03:58.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:57 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-20T09:03:58.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:57 smithi122 ceph-mon[73495]: 1.9 scrub starts 2024-03-20T09:03:58.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:57 smithi122 ceph-mon[73495]: 1.9 scrub ok 2024-03-20T09:03:58.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:57 smithi122 ceph-mon[73495]: 3.17 scrub starts 2024-03-20T09:03:58.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:57 smithi122 ceph-mon[73495]: pgmap v672: 37 pgs: 1 peering, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 29 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s; 9186/219129 objects degraded (4.192%); 7836/219129 objects misplaced (3.576%); 45 MiB/s, 715 objects/s recovering 2024-03-20T09:03:58.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:57 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-03-20T09:03:58.474 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:03:58.524 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:58.563 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:03:58.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:58.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:03:58.610 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 57 16 75659 75643 82.9291 68 0.00644655 0.0120532 2024-03-20T09:03:58.671 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:03:58.707 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:58.838 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:03:58.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:58.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:03:58.944 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:03:58.980 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:59.008 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:03:59.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:59.011 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:03:59.112 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:03:59.177 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:59.249 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:58 smithi026 ceph-mon[68309]: 3.17 scrub ok 2024-03-20T09:03:59.249 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:58 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-03-20T09:03:59.250 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:58 smithi026 ceph-mon[68309]: osdmap e252: 12 total, 12 up, 12 in 2024-03-20T09:03:59.250 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:58 smithi026 ceph-mon[68309]: 3.c deep-scrub starts 2024-03-20T09:03:59.250 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:58 smithi026 ceph-mon[68309]: osdmap e253: 12 total, 12 up, 12 in 2024-03-20T09:03:59.256 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:03:59.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:59.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:03:59.367 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:03:59.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:58 smithi115 ceph-mon[72476]: 3.17 scrub ok 2024-03-20T09:03:59.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:58 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-03-20T09:03:59.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:58 smithi115 ceph-mon[72476]: osdmap e252: 12 total, 12 up, 12 in 2024-03-20T09:03:59.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:58 smithi115 ceph-mon[72476]: 3.c deep-scrub starts 2024-03-20T09:03:59.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:58 smithi115 ceph-mon[72476]: osdmap e253: 12 total, 12 up, 12 in 2024-03-20T09:03:59.443 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:59.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:58 smithi122 ceph-mon[73495]: 3.17 scrub ok 2024-03-20T09:03:59.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:58 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-03-20T09:03:59.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:58 smithi122 ceph-mon[73495]: osdmap e252: 12 total, 12 up, 12 in 2024-03-20T09:03:59.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:58 smithi122 ceph-mon[73495]: 3.c deep-scrub starts 2024-03-20T09:03:59.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:58 smithi122 ceph-mon[73495]: osdmap e253: 12 total, 12 up, 12 in 2024-03-20T09:03:59.477 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:03:59.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:59.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:03:59.588 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:03:59.610 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 58 16 76519 76503 82.426 53.75 0.0251491 0.0121245 2024-03-20T09:03:59.626 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:59.837 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:03:59.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:03:59.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:03:59.869 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:03:59.895 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:03:59.938 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:03:59.943 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:03:59.965 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:03:59.982 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:03:59.995 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:04:00.039 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:04:00.076 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:04:00.114 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:04:00.138 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:04:00.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:00.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:04:00.146 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:04:00.175 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:04:00.205 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:04:00.233 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:04:00.243 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:04:00.256 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:04:00.259 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:04:00.260 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:04:00.260 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:04:00.260 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:04:00.260 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:04:00.260 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:04:00.260 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:04:00.260 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:04:00.260 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:04:00.260 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:04:00.260 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:04:00.260 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:04:00.273 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:59 smithi122 ceph-mon[73495]: 3.5 scrub starts 2024-03-20T09:04:00.273 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:59 smithi122 ceph-mon[73495]: pgmap v675: 47 pgs: 7 unknown, 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 33 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 515 op/s; 9185/221538 objects degraded (4.146%); 7836/221538 objects misplaced (3.537%); 7.9 MiB/s, 125 objects/s recovering 2024-03-20T09:04:00.273 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:59 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-20T09:04:00.273 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:03:59 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3986748850' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:04:00.274 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:00.282 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:59 smithi026 ceph-mon[68309]: 3.5 scrub starts 2024-03-20T09:04:00.282 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:59 smithi026 ceph-mon[68309]: pgmap v675: 47 pgs: 7 unknown, 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 33 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 515 op/s; 9185/221538 objects degraded (4.146%); 7836/221538 objects misplaced (3.537%); 7.9 MiB/s, 125 objects/s recovering 2024-03-20T09:04:00.282 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:59 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-20T09:04:00.282 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:03:59 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3986748850' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:04:00.296 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:04:00.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:00.299 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:04:00.400 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:04:00.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:59 smithi115 ceph-mon[72476]: 3.5 scrub starts 2024-03-20T09:04:00.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:59 smithi115 ceph-mon[72476]: pgmap v675: 47 pgs: 7 unknown, 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 33 active+clean; 4.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 515 op/s; 9185/221538 objects degraded (4.146%); 7836/221538 objects misplaced (3.537%); 7.9 MiB/s, 125 objects/s recovering 2024-03-20T09:04:00.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:59 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-03-20T09:04:00.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:03:59 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3986748850' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:04:00.448 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:00.475 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:04:00.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:00.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:04:00.579 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:04:00.610 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 59 16 77236 77220 81.7885 44.8125 0.00204062 0.0121527 2024-03-20T09:04:00.617 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:00.685 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:04:00.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:00.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:04:00.789 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:04:00.816 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:00.834 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:04:00.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:00.837 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:04:00.873 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:04:00.938 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:04:00.965 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:00.983 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:04:00.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:00.986 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:04:01.087 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:04:01.115 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:01 smithi122 ceph-mon[73495]: 1.2 deep-scrub starts 2024-03-20T09:04:01.116 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:01 smithi122 ceph-mon[73495]: 1.2 deep-scrub ok 2024-03-20T09:04:01.116 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:01 smithi122 ceph-mon[73495]: 1.5 scrub starts 2024-03-20T09:04:01.116 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:01 smithi122 ceph-mon[73495]: 1.5 scrub ok 2024-03-20T09:04:01.116 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:01 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-03-20T09:04:01.116 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:01 smithi122 ceph-mon[73495]: osdmap e254: 12 total, 12 up, 12 in 2024-03-20T09:04:01.117 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:01 smithi122 ceph-mon[73495]: 3.5 scrub ok 2024-03-20T09:04:01.117 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:01.142 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:04:01.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:01.146 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:04:01.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:01 smithi026 ceph-mon[68309]: 1.2 deep-scrub starts 2024-03-20T09:04:01.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:01 smithi026 ceph-mon[68309]: 1.2 deep-scrub ok 2024-03-20T09:04:01.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:01 smithi026 ceph-mon[68309]: 1.5 scrub starts 2024-03-20T09:04:01.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:01 smithi026 ceph-mon[68309]: 1.5 scrub ok 2024-03-20T09:04:01.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:01 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-03-20T09:04:01.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:01 smithi026 ceph-mon[68309]: osdmap e254: 12 total, 12 up, 12 in 2024-03-20T09:04:01.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:01 smithi026 ceph-mon[68309]: 3.5 scrub ok 2024-03-20T09:04:01.247 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:04:01.280 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:01.303 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:04:01.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:01.308 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:04:01.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:01 smithi115 ceph-mon[72476]: 1.2 deep-scrub starts 2024-03-20T09:04:01.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:01 smithi115 ceph-mon[72476]: 1.2 deep-scrub ok 2024-03-20T09:04:01.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:01 smithi115 ceph-mon[72476]: 1.5 scrub starts 2024-03-20T09:04:01.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:01 smithi115 ceph-mon[72476]: 1.5 scrub ok 2024-03-20T09:04:01.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:01 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-03-20T09:04:01.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:01 smithi115 ceph-mon[72476]: osdmap e254: 12 total, 12 up, 12 in 2024-03-20T09:04:01.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:01 smithi115 ceph-mon[72476]: 3.5 scrub ok 2024-03-20T09:04:01.408 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:04:01.441 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:01.610 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:2024-03-20T09:04:01.610855+0000 min lat: 0.001051 max lat: 5.41234 avg lat: 0.0123421 2024-03-20T09:04:01.610 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T09:04:01.610 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 60 16 77763 77747 80.9743 32.9375 0.0191196 0.0123421 2024-03-20T09:04:01.666 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:04:01.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:01.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:04:01.774 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:04:01.809 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:02.031 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:04:02.072 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:04:02.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:02.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:04:02.106 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:04:02.106 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:04:02.177 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:04:02.221 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:02.269 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:04:02.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:02.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:04:02.274 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:02 smithi115 ceph-mon[72476]: 3.7 scrub starts 2024-03-20T09:04:02.274 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:02 smithi115 ceph-mon[72476]: 3.7 scrub ok 2024-03-20T09:04:02.274 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:02 smithi115 ceph-mon[72476]: 1.1 scrub starts 2024-03-20T09:04:02.274 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:02 smithi115 ceph-mon[72476]: 1.1 scrub ok 2024-03-20T09:04:02.275 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:02 smithi115 ceph-mon[72476]: osdmap e255: 12 total, 12 up, 12 in 2024-03-20T09:04:02.275 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:02 smithi115 ceph-mon[72476]: 3.16 scrub starts 2024-03-20T09:04:02.275 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:02 smithi115 ceph-mon[72476]: pgmap v678: 47 pgs: 4 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 705 op/s; 5309/227592 objects degraded (2.333%); 9111/227592 objects misplaced (4.003%); 16 MiB/s, 251 objects/s recovering 2024-03-20T09:04:02.275 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:02 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 9185/221538 objects degraded (4.146%), 2 pgs degraded (PG_DEGRADED) 2024-03-20T09:04:02.375 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:04:02.408 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:02 smithi026 ceph-mon[68309]: 3.7 scrub starts 2024-03-20T09:04:02.408 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:02 smithi026 ceph-mon[68309]: 3.7 scrub ok 2024-03-20T09:04:02.408 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:02 smithi026 ceph-mon[68309]: 1.1 scrub starts 2024-03-20T09:04:02.408 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:02 smithi026 ceph-mon[68309]: 1.1 scrub ok 2024-03-20T09:04:02.408 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:02 smithi026 ceph-mon[68309]: osdmap e255: 12 total, 12 up, 12 in 2024-03-20T09:04:02.408 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:02 smithi026 ceph-mon[68309]: 3.16 scrub starts 2024-03-20T09:04:02.408 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:02 smithi026 ceph-mon[68309]: pgmap v678: 47 pgs: 4 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 705 op/s; 5309/227592 objects degraded (2.333%); 9111/227592 objects misplaced (4.003%); 16 MiB/s, 251 objects/s recovering 2024-03-20T09:04:02.409 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:02 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 9185/221538 objects degraded (4.146%), 2 pgs degraded (PG_DEGRADED) 2024-03-20T09:04:02.411 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:02.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:02 smithi122 ceph-mon[73495]: 3.7 scrub starts 2024-03-20T09:04:02.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:02 smithi122 ceph-mon[73495]: 3.7 scrub ok 2024-03-20T09:04:02.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:02 smithi122 ceph-mon[73495]: 1.1 scrub starts 2024-03-20T09:04:02.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:02 smithi122 ceph-mon[73495]: 1.1 scrub ok 2024-03-20T09:04:02.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:02 smithi122 ceph-mon[73495]: osdmap e255: 12 total, 12 up, 12 in 2024-03-20T09:04:02.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:02 smithi122 ceph-mon[73495]: 3.16 scrub starts 2024-03-20T09:04:02.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:02 smithi122 ceph-mon[73495]: pgmap v678: 47 pgs: 4 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 38 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 705 op/s; 5309/227592 objects degraded (2.333%); 9111/227592 objects misplaced (4.003%); 16 MiB/s, 251 objects/s recovering 2024-03-20T09:04:02.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:02 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 9185/221538 objects degraded (4.146%), 2 pgs degraded (PG_DEGRADED) 2024-03-20T09:04:02.610 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 61 16 78699 78683 80.6058 58.5 0.0241672 0.012399 2024-03-20T09:04:03.357 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:03 smithi026 ceph-mon[68309]: 3.16 scrub ok 2024-03-20T09:04:03.357 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:03 smithi026 ceph-mon[68309]: 1.6 scrub starts 2024-03-20T09:04:03.357 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:03 smithi026 ceph-mon[68309]: 1.6 scrub ok 2024-03-20T09:04:03.357 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:03 smithi026 ceph-mon[68309]: osdmap e256: 12 total, 12 up, 12 in 2024-03-20T09:04:03.363 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:04:03.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:03.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:04:03.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:03 smithi115 ceph-mon[72476]: 3.16 scrub ok 2024-03-20T09:04:03.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:03 smithi115 ceph-mon[72476]: 1.6 scrub starts 2024-03-20T09:04:03.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:03 smithi115 ceph-mon[72476]: 1.6 scrub ok 2024-03-20T09:04:03.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:03 smithi115 ceph-mon[72476]: osdmap e256: 12 total, 12 up, 12 in 2024-03-20T09:04:03.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:03 smithi122 ceph-mon[73495]: 3.16 scrub ok 2024-03-20T09:04:03.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:03 smithi122 ceph-mon[73495]: 1.6 scrub starts 2024-03-20T09:04:03.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:03 smithi122 ceph-mon[73495]: 1.6 scrub ok 2024-03-20T09:04:03.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:03 smithi122 ceph-mon[73495]: osdmap e256: 12 total, 12 up, 12 in 2024-03-20T09:04:03.468 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:04:03.505 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:03.610 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 62 16 79643 79627 80.2573 59 0.0140692 0.0124528 2024-03-20T09:04:04.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:04 smithi115 ceph-mon[72476]: osdmap e257: 12 total, 12 up, 12 in 2024-03-20T09:04:04.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:04 smithi115 ceph-mon[72476]: pgmap v681: 47 pgs: 1 peering, 36 active+clean, 3 active+recovering+undersized+remapped, 3 unknown, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped; 4.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5309/231072 objects degraded (2.298%); 9111/231072 objects misplaced (3.943%) 2024-03-20T09:04:04.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:04 smithi122 ceph-mon[73495]: osdmap e257: 12 total, 12 up, 12 in 2024-03-20T09:04:04.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:04 smithi122 ceph-mon[73495]: pgmap v681: 47 pgs: 1 peering, 36 active+clean, 3 active+recovering+undersized+remapped, 3 unknown, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped; 4.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5309/231072 objects degraded (2.298%); 9111/231072 objects misplaced (3.943%) 2024-03-20T09:04:04.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:04 smithi026 ceph-mon[68309]: osdmap e257: 12 total, 12 up, 12 in 2024-03-20T09:04:04.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:04 smithi026 ceph-mon[68309]: pgmap v681: 47 pgs: 1 peering, 36 active+clean, 3 active+recovering+undersized+remapped, 3 unknown, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped; 4.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5309/231072 objects degraded (2.298%); 9111/231072 objects misplaced (3.943%) 2024-03-20T09:04:04.610 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 63 16 80570 80554 79.9029 57.9375 0.00490934 0.0125082 2024-03-20T09:04:04.672 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-03-20T09:04:04.672 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:04:04.672 INFO:tasks.thrashosds.thrasher:Removing osd 9, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-03-20T09:04:04.672 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd out 9 2024-03-20T09:04:05.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:05 smithi115 ceph-mon[72476]: osdmap e258: 12 total, 12 up, 12 in 2024-03-20T09:04:05.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:05 smithi115 ceph-mon[72476]: 1.10 scrub starts 2024-03-20T09:04:05.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:05 smithi115 ceph-mon[72476]: 1.10 scrub ok 2024-03-20T09:04:05.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:05 smithi122 ceph-mon[73495]: osdmap e258: 12 total, 12 up, 12 in 2024-03-20T09:04:05.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:05 smithi122 ceph-mon[73495]: 1.10 scrub starts 2024-03-20T09:04:05.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:05 smithi122 ceph-mon[73495]: 1.10 scrub ok 2024-03-20T09:04:05.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:05 smithi026 ceph-mon[68309]: osdmap e258: 12 total, 12 up, 12 in 2024-03-20T09:04:05.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:05 smithi026 ceph-mon[68309]: 1.10 scrub starts 2024-03-20T09:04:05.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:05 smithi026 ceph-mon[68309]: 1.10 scrub ok 2024-03-20T09:04:05.610 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 64 16 81400 81384 79.465 51.875 0.00383391 0.0125434 2024-03-20T09:04:05.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:05.663 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:04:05.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:04:05.764 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:04:05.791 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:05.813 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:04:05.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:05.820 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:04:05.921 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:04:05.948 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:05.971 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:04:05.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:05.975 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:04:06.076 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:04:06.108 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:06.136 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:04:06.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:06.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:04:06.251 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:04:06.315 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:06.348 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:04:06.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:06.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:04:06.388 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:06 smithi026 ceph-mon[68309]: 1.c scrub starts 2024-03-20T09:04:06.388 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:06 smithi026 ceph-mon[68309]: 1.c scrub ok 2024-03-20T09:04:06.389 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:06 smithi026 ceph-mon[68309]: 1.11 scrub starts 2024-03-20T09:04:06.389 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:06 smithi026 ceph-mon[68309]: 1.11 scrub ok 2024-03-20T09:04:06.389 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:06 smithi026 ceph-mon[68309]: osdmap e259: 12 total, 12 up, 12 in 2024-03-20T09:04:06.389 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:06 smithi026 ceph-mon[68309]: 1.12 scrub starts 2024-03-20T09:04:06.389 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:06 smithi026 ceph-mon[68309]: 1.12 scrub ok 2024-03-20T09:04:06.389 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:06 smithi026 ceph-mon[68309]: pgmap v684: 47 pgs: 1 peering, 39 active+clean, 2 active+recovering+undersized+remapped, 1 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 1 active+undersized+remapped; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1 op/s; 5309/239355 objects degraded (2.218%); 6190/239355 objects misplaced (2.586%) 2024-03-20T09:04:06.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:06 smithi115 ceph-mon[72476]: 1.c scrub starts 2024-03-20T09:04:06.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:06 smithi115 ceph-mon[72476]: 1.c scrub ok 2024-03-20T09:04:06.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:06 smithi115 ceph-mon[72476]: 1.11 scrub starts 2024-03-20T09:04:06.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:06 smithi115 ceph-mon[72476]: 1.11 scrub ok 2024-03-20T09:04:06.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:06 smithi115 ceph-mon[72476]: osdmap e259: 12 total, 12 up, 12 in 2024-03-20T09:04:06.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:06 smithi115 ceph-mon[72476]: 1.12 scrub starts 2024-03-20T09:04:06.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:06 smithi115 ceph-mon[72476]: 1.12 scrub ok 2024-03-20T09:04:06.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:06 smithi115 ceph-mon[72476]: pgmap v684: 47 pgs: 1 peering, 39 active+clean, 2 active+recovering+undersized+remapped, 1 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 1 active+undersized+remapped; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1 op/s; 5309/239355 objects degraded (2.218%); 6190/239355 objects misplaced (2.586%) 2024-03-20T09:04:06.426 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-20T09:04:06.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:06 smithi122 ceph-mon[73495]: 1.c scrub starts 2024-03-20T09:04:06.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:06 smithi122 ceph-mon[73495]: 1.c scrub ok 2024-03-20T09:04:06.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:06 smithi122 ceph-mon[73495]: 1.11 scrub starts 2024-03-20T09:04:06.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:06 smithi122 ceph-mon[73495]: 1.11 scrub ok 2024-03-20T09:04:06.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:06 smithi122 ceph-mon[73495]: osdmap e259: 12 total, 12 up, 12 in 2024-03-20T09:04:06.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:06 smithi122 ceph-mon[73495]: 1.12 scrub starts 2024-03-20T09:04:06.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:06 smithi122 ceph-mon[73495]: 1.12 scrub ok 2024-03-20T09:04:06.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:06 smithi122 ceph-mon[73495]: pgmap v684: 47 pgs: 1 peering, 39 active+clean, 2 active+recovering+undersized+remapped, 1 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 1 active+undersized+remapped; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1 op/s; 5309/239355 objects degraded (2.218%); 6190/239355 objects misplaced (2.586%) 2024-03-20T09:04:06.450 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:04:06.507 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:06.543 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:04:06.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:06.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:04:06.611 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 65 16 81986 81970 78.8059 36.625 0.00443168 0.0126811 2024-03-20T09:04:06.648 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:04:06.677 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:06.701 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:04:06.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:06.704 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:04:06.805 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:04:06.875 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:06.876 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:04:06.918 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:04:06.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:06.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:04:06.928 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:04:07.028 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:04:07.062 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:07.100 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:04:07.127 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:07 smithi115 ceph-mon[72476]: osdmap e260: 12 total, 12 up, 12 in 2024-03-20T09:04:07.127 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:07 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 5309/239355 objects degraded (2.218%), 2 pgs degraded (PG_DEGRADED) 2024-03-20T09:04:07.127 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:07 smithi115 ceph-mon[72476]: osdmap e261: 12 total, 12 up, 12 in 2024-03-20T09:04:07.127 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:07 smithi026 ceph-mon[68309]: osdmap e260: 12 total, 12 up, 12 in 2024-03-20T09:04:07.127 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:07 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 5309/239355 objects degraded (2.218%), 2 pgs degraded (PG_DEGRADED) 2024-03-20T09:04:07.127 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:07 smithi026 ceph-mon[68309]: osdmap e261: 12 total, 12 up, 12 in 2024-03-20T09:04:07.142 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:04:07.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:07.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:04:07.246 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:04:07.289 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:07.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:07 smithi122 ceph-mon[73495]: osdmap e260: 12 total, 12 up, 12 in 2024-03-20T09:04:07.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:07 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 5309/239355 objects degraded (2.218%), 2 pgs degraded (PG_DEGRADED) 2024-03-20T09:04:07.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:07 smithi122 ceph-mon[73495]: osdmap e261: 12 total, 12 up, 12 in 2024-03-20T09:04:07.611 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 66 16 82988 82972 78.5607 62.625 0.0322576 0.0127235 2024-03-20T09:04:08.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:08 smithi115 ceph-mon[72476]: pgmap v687: 47 pgs: 1 peering, 40 active+clean, 2 active+recovering+undersized+remapped, 1 unknown, 1 active+clean+scrubbing+deep, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped; 5.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 27 MiB/s wr, 430 op/s; 5309/243522 objects degraded (2.180%); 6190/243522 objects misplaced (2.542%); 18 MiB/s, 280 objects/s recovering 2024-03-20T09:04:08.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:08 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:04:08.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:08 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:04:08.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:08 smithi122 ceph-mon[73495]: pgmap v687: 47 pgs: 1 peering, 40 active+clean, 2 active+recovering+undersized+remapped, 1 unknown, 1 active+clean+scrubbing+deep, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped; 5.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 27 MiB/s wr, 430 op/s; 5309/243522 objects degraded (2.180%); 6190/243522 objects misplaced (2.542%); 18 MiB/s, 280 objects/s recovering 2024-03-20T09:04:08.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:08 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:04:08.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:08 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:04:08.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:08 smithi026 ceph-mon[68309]: pgmap v687: 47 pgs: 1 peering, 40 active+clean, 2 active+recovering+undersized+remapped, 1 unknown, 1 active+clean+scrubbing+deep, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped; 5.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 27 MiB/s wr, 430 op/s; 5309/243522 objects degraded (2.180%); 6190/243522 objects misplaced (2.542%); 18 MiB/s, 280 objects/s recovering 2024-03-20T09:04:08.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:08 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:04:08.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:08 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:04:08.611 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 67 16 83934 83918 78.2706 59.125 0.00388104 0.0127707 2024-03-20T09:04:08.747 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:04:08.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:08.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:04:08.852 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:04:08.891 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:08.950 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:04:08.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:08.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:04:09.052 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:04:09.108 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:09.142 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:09 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:04:09.142 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:09 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:04:09.143 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:09 smithi026 ceph-mon[68309]: osdmap e262: 12 total, 12 up, 12 in 2024-03-20T09:04:09.143 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:09 smithi026 ceph-mon[68309]: 3.d scrub starts 2024-03-20T09:04:09.143 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:04:09.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:09.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:04:09.259 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:04:09.291 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:09.313 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:04:09.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:09.316 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:04:09.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:09 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:04:09.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:09 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:04:09.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:09 smithi115 ceph-mon[72476]: osdmap e262: 12 total, 12 up, 12 in 2024-03-20T09:04:09.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:09 smithi115 ceph-mon[72476]: 3.d scrub starts 2024-03-20T09:04:09.417 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:04:09.445 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:09 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:04:09.445 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:09 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:04:09.445 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:09 smithi122 ceph-mon[73495]: osdmap e262: 12 total, 12 up, 12 in 2024-03-20T09:04:09.445 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:09 smithi122 ceph-mon[73495]: 3.d scrub starts 2024-03-20T09:04:09.447 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:09.468 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:04:09.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:09 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mon-a[68286]: 2024-03-20T09:04:09.259+0000 7f683364b700 -1 mon.a@0(leader).osd e263 definitely_dead 0 2024-03-20T09:04:09.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:09.474 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:04:09.575 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:04:09.603 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:04:09.610 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:09.611 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 68 16 84693 84677 77.8171 47.4375 0.00244295 0.0128159 2024-03-20T09:04:09.634 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:04:09.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:09.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:04:09.739 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:04:09.766 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:09.788 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:04:09.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:09.792 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:04:09.893 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:04:09.925 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:09.945 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:04:09.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:09.948 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:04:10.049 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:04:10.077 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:10.097 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:04:10.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:10.100 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:04:10.161 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:04:10.162 INFO:teuthology.orchestra.run.smithi026.stderr:marked out osd.9. 2024-03-20T09:04:10.201 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:04:10.234 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:10.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:10 smithi122 ceph-mon[73495]: osdmap e263: 12 total, 12 up, 12 in 2024-03-20T09:04:10.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:10 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2991126427' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["9"]}]: dispatch 2024-03-20T09:04:10.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:10 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["9"]}]: dispatch 2024-03-20T09:04:10.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:10 smithi122 ceph-mon[73495]: Client client.admin marked osd.9 out, while it was still marked up 2024-03-20T09:04:10.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:10 smithi122 ceph-mon[73495]: 3.d scrub ok 2024-03-20T09:04:10.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:10 smithi122 ceph-mon[73495]: pgmap v690: 47 pgs: 42 active+clean, 3 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped; 5.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 581 op/s; 1/246333 objects degraded (0.000%); 11352/246333 objects misplaced (4.608%); 741 KiB/s, 11 objects/s recovering 2024-03-20T09:04:10.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:10 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/626353650' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:04:10.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:10 smithi026 ceph-mon[68309]: osdmap e263: 12 total, 12 up, 12 in 2024-03-20T09:04:10.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:10 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2991126427' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["9"]}]: dispatch 2024-03-20T09:04:10.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:10 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["9"]}]: dispatch 2024-03-20T09:04:10.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:10 smithi026 ceph-mon[68309]: Client client.admin marked osd.9 out, while it was still marked up 2024-03-20T09:04:10.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:10 smithi026 ceph-mon[68309]: 3.d scrub ok 2024-03-20T09:04:10.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:10 smithi026 ceph-mon[68309]: pgmap v690: 47 pgs: 42 active+clean, 3 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped; 5.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 581 op/s; 1/246333 objects degraded (0.000%); 11352/246333 objects misplaced (4.608%); 741 KiB/s, 11 objects/s recovering 2024-03-20T09:04:10.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:10 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/626353650' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:04:10.611 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 69 16 84946 84930 76.9185 15.8125 0.00264388 0.0129282 2024-03-20T09:04:10.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:10 smithi115 ceph-mon[72476]: osdmap e263: 12 total, 12 up, 12 in 2024-03-20T09:04:10.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:10 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2991126427' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["9"]}]: dispatch 2024-03-20T09:04:10.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:10 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["9"]}]: dispatch 2024-03-20T09:04:10.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:10 smithi115 ceph-mon[72476]: Client client.admin marked osd.9 out, while it was still marked up 2024-03-20T09:04:10.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:10 smithi115 ceph-mon[72476]: 3.d scrub ok 2024-03-20T09:04:10.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:10 smithi115 ceph-mon[72476]: pgmap v690: 47 pgs: 42 active+clean, 3 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped; 5.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 581 op/s; 1/246333 objects degraded (0.000%); 11352/246333 objects misplaced (4.608%); 741 KiB/s, 11 objects/s recovering 2024-03-20T09:04:10.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:10 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/626353650' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:04:11.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:11 smithi115 ceph-mon[72476]: 3.4 scrub starts 2024-03-20T09:04:11.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:11 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:04:11.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:11 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["9"]}]': finished 2024-03-20T09:04:11.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:11 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/626353650' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:04:11.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:11 smithi115 ceph-mon[72476]: osdmap e264: 12 total, 12 up, 11 in 2024-03-20T09:04:11.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:11 smithi122 ceph-mon[73495]: 3.4 scrub starts 2024-03-20T09:04:11.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:11 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:04:11.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:11 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["9"]}]': finished 2024-03-20T09:04:11.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:11 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/626353650' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:04:11.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:11 smithi122 ceph-mon[73495]: osdmap e264: 12 total, 12 up, 11 in 2024-03-20T09:04:11.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:11 smithi026 ceph-mon[68309]: 3.4 scrub starts 2024-03-20T09:04:11.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:11 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:04:11.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:11 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["9"]}]': finished 2024-03-20T09:04:11.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:11 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/626353650' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:04:11.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:11 smithi026 ceph-mon[68309]: osdmap e264: 12 total, 12 up, 11 in 2024-03-20T09:04:11.611 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 70 16 84946 84930 75.8198 0 - 0.0129282 2024-03-20T09:04:11.685 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:04:11.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:11.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:04:11.790 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:04:11.816 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:11.834 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:04:11.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:11.839 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:04:11.939 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:04:11.969 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:12.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:12 smithi026 ceph-mon[68309]: osdmap e265: 12 total, 12 up, 11 in 2024-03-20T09:04:12.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:12 smithi026 ceph-mon[68309]: pgmap v693: 47 pgs: 43 active+clean, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep; 5.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 44 MiB/s wr, 700 op/s; 1/251934 objects degraded (0.000%); 9876/251934 objects misplaced (3.920%); 11 MiB/s, 179 objects/s recovering 2024-03-20T09:04:12.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:12 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 1/246333 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-03-20T09:04:12.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:12 smithi122 ceph-mon[73495]: osdmap e265: 12 total, 12 up, 11 in 2024-03-20T09:04:12.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:12 smithi122 ceph-mon[73495]: pgmap v693: 47 pgs: 43 active+clean, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep; 5.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 44 MiB/s wr, 700 op/s; 1/251934 objects degraded (0.000%); 9876/251934 objects misplaced (3.920%); 11 MiB/s, 179 objects/s recovering 2024-03-20T09:04:12.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:12 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 1/246333 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-03-20T09:04:12.611 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 71 16 85124 85108 74.9086 5.5625 0.00602329 0.0133414 2024-03-20T09:04:12.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:12 smithi115 ceph-mon[72476]: osdmap e265: 12 total, 12 up, 11 in 2024-03-20T09:04:12.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:12 smithi115 ceph-mon[72476]: pgmap v693: 47 pgs: 43 active+clean, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep; 5.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 44 MiB/s wr, 700 op/s; 1/251934 objects degraded (0.000%); 9876/251934 objects misplaced (3.920%); 11 MiB/s, 179 objects/s recovering 2024-03-20T09:04:12.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:12 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 1/246333 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-03-20T09:04:13.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:13 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:04:13.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:13 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:04:13.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:13 smithi122 ceph-mon[73495]: osdmap e266: 12 total, 12 up, 11 in 2024-03-20T09:04:13.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:13 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:04:13.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:13 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:04:13.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:13 smithi026 ceph-mon[68309]: osdmap e266: 12 total, 12 up, 11 in 2024-03-20T09:04:13.611 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 72 16 85817 85801 74.4698 43.3125 0.0246533 0.0134219 2024-03-20T09:04:13.649 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:04:13.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:13.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:04:13.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:13 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:04:13.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:13 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:04:13.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:13 smithi115 ceph-mon[72476]: osdmap e266: 12 total, 12 up, 11 in 2024-03-20T09:04:13.756 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:04:13.790 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:14.200 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:04:14.200 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:04:14.244 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:04:14.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:14.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:04:14.350 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:04:14.399 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:14.433 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:04:14.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:14.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:04:14.462 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:14 smithi026 ceph-mon[68309]: osdmap e267: 12 total, 12 up, 11 in 2024-03-20T09:04:14.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:14 smithi026 ceph-mon[68309]: pgmap v696: 47 pgs: 2 peering, 1 activating+undersized+degraded+remapped, 41 active+clean, 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped; 5.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 2.5 KiB/s rd, 42 MiB/s wr, 677 op/s; 5245/254382 objects degraded (2.062%); 8373/254382 objects misplaced (3.292%); 18 MiB/s, 292 objects/s recovering 2024-03-20T09:04:14.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:14 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:04:14.549 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:04:14.611 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 73 16 86441 86425 73.9839 39 0.0266936 0.0135105 2024-03-20T09:04:14.633 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:14 smithi115 ceph-mon[72476]: osdmap e267: 12 total, 12 up, 11 in 2024-03-20T09:04:14.633 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:14 smithi115 ceph-mon[72476]: pgmap v696: 47 pgs: 2 peering, 1 activating+undersized+degraded+remapped, 41 active+clean, 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped; 5.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 2.5 KiB/s rd, 42 MiB/s wr, 677 op/s; 5245/254382 objects degraded (2.062%); 8373/254382 objects misplaced (3.292%); 18 MiB/s, 292 objects/s recovering 2024-03-20T09:04:14.633 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:14 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:04:14.636 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:14.672 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:04:14.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:14.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:04:14.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:14 smithi122 ceph-mon[73495]: osdmap e267: 12 total, 12 up, 11 in 2024-03-20T09:04:14.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:14 smithi122 ceph-mon[73495]: pgmap v696: 47 pgs: 2 peering, 1 activating+undersized+degraded+remapped, 41 active+clean, 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped; 5.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 2.5 KiB/s rd, 42 MiB/s wr, 677 op/s; 5245/254382 objects degraded (2.062%); 8373/254382 objects misplaced (3.292%); 18 MiB/s, 292 objects/s recovering 2024-03-20T09:04:14.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:14 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:04:14.776 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:04:14.810 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:15.161 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-20T09:04:15.327 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:04:15.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:15.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:04:15.403 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:04:15.431 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:04:15.432 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:04:15.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:15 smithi026 ceph-mon[68309]: osdmap e268: 12 total, 12 up, 11 in 2024-03-20T09:04:15.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:15 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:04:15.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:15 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:04:15.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:15 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:04:15.462 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:04:15.469 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:15.505 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:04:15.531 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:04:15.562 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:04:15.594 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:04:15.611 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 74 16 87128 87112 73.5643 42.9375 0.00428868 0.0135868 2024-03-20T09:04:15.623 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:04:15.649 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:04:15.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:15 smithi115 ceph-mon[72476]: osdmap e268: 12 total, 12 up, 11 in 2024-03-20T09:04:15.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:15 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:04:15.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:15 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:04:15.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:15 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:04:15.673 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:04:15.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:15 smithi122 ceph-mon[73495]: osdmap e268: 12 total, 12 up, 11 in 2024-03-20T09:04:15.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:15 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:04:15.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:15 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:04:15.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:15 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:04:15.698 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:04:15.726 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:04:15.726 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:04:15.726 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:04:15.726 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:04:15.726 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:04:15.727 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:04:15.727 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:04:15.727 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:04:15.727 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:04:15.727 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:04:15.727 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:04:15.727 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:04:15.727 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:04:16.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:16 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1024652108' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:04:16.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:16 smithi026 ceph-mon[68309]: pgmap v698: 47 pgs: 2 peering, 1 activating+undersized+degraded+remapped, 39 active+clean, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep; 5.3 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 2.2 KiB/s rd, 29 MiB/s wr, 470 op/s; 5245/258153 objects degraded (2.032%); 15277/258153 objects misplaced (5.918%); 18 MiB/s, 284 objects/s recovering 2024-03-20T09:04:16.533 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:04:16.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:16.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:04:16.596 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:04:16.611 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 75 16 87770 87754 73.1184 40.125 0.021536 0.0136693 2024-03-20T09:04:16.637 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:04:16.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:16 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1024652108' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:04:16.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:16 smithi115 ceph-mon[72476]: pgmap v698: 47 pgs: 2 peering, 1 activating+undersized+degraded+remapped, 39 active+clean, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep; 5.3 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 2.2 KiB/s rd, 29 MiB/s wr, 470 op/s; 5245/258153 objects degraded (2.032%); 15277/258153 objects misplaced (5.918%); 18 MiB/s, 284 objects/s recovering 2024-03-20T09:04:16.667 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:16 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1024652108' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:04:16.667 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:16 smithi122 ceph-mon[73495]: pgmap v698: 47 pgs: 2 peering, 1 activating+undersized+degraded+remapped, 39 active+clean, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep; 5.3 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 2.2 KiB/s rd, 29 MiB/s wr, 470 op/s; 5245/258153 objects degraded (2.032%); 15277/258153 objects misplaced (5.918%); 18 MiB/s, 284 objects/s recovering 2024-03-20T09:04:16.670 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:16.693 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:04:16.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:16.697 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:04:16.797 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:04:16.833 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:16.855 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:04:16.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:16.858 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:04:16.961 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:04:16.995 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:17.016 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:04:17.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:17.019 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:04:17.120 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:04:17.183 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:17.212 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:04:17.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:17.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:04:17.315 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:04:17.341 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:17 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 5245/258153 objects degraded (2.032%), 2 pgs degraded (PG_DEGRADED) 2024-03-20T09:04:17.341 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:17 smithi122 ceph-mon[73495]: osdmap e269: 12 total, 12 up, 11 in 2024-03-20T09:04:17.343 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:17.368 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:04:17.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:17.374 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:04:17.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:17 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 5245/258153 objects degraded (2.032%), 2 pgs degraded (PG_DEGRADED) 2024-03-20T09:04:17.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:17 smithi026 ceph-mon[68309]: osdmap e269: 12 total, 12 up, 11 in 2024-03-20T09:04:17.475 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:04:17.514 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:17 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 5245/258153 objects degraded (2.032%), 2 pgs degraded (PG_DEGRADED) 2024-03-20T09:04:17.514 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:17 smithi115 ceph-mon[72476]: osdmap e269: 12 total, 12 up, 11 in 2024-03-20T09:04:17.524 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:17.564 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:04:17.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:17.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:04:17.611 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 76 16 88472 88456 72.7336 43.875 0.00457139 0.0137438 2024-03-20T09:04:17.675 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:04:17.708 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:17.832 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:04:18.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:18 smithi026 ceph-mon[68309]: pgmap v700: 47 pgs: 2 peering, 39 active+clean, 4 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped; 5.3 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 1.8 KiB/s rd, 34 MiB/s wr, 553 op/s; 1/261177 objects degraded (0.000%); 20390/261177 objects misplaced (7.807%); 16 MiB/s, 256 objects/s recovering 2024-03-20T09:04:18.533 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:04:18.612 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 77 16 89174 89158 72.3588 43.875 0.00509411 0.0138152 2024-03-20T09:04:18.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:18 smithi115 ceph-mon[72476]: pgmap v700: 47 pgs: 2 peering, 39 active+clean, 4 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped; 5.3 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 1.8 KiB/s rd, 34 MiB/s wr, 553 op/s; 1/261177 objects degraded (0.000%); 20390/261177 objects misplaced (7.807%); 16 MiB/s, 256 objects/s recovering 2024-03-20T09:04:18.684 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11] out_osds: [9] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-03-20T09:04:18.684 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:04:18.684 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-03-20T09:04:18.684 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-20T09:04:18.685 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-20T09:04:18.685 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-03-20T09:04:18.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:18 smithi122 ceph-mon[73495]: pgmap v700: 47 pgs: 2 peering, 39 active+clean, 4 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped; 5.3 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 1.8 KiB/s rd, 34 MiB/s wr, 553 op/s; 1/261177 objects degraded (0.000%); 20390/261177 objects misplaced (7.807%); 16 MiB/s, 256 objects/s recovering 2024-03-20T09:04:19.274 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:04:19.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:19.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:04:19.381 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:04:19.418 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:19.438 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:04:19.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:19.441 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:04:19.542 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:04:19.600 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:19.612 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 78 16 89877 89861 71.9944 43.9375 0.0388148 0.0138815 2024-03-20T09:04:19.635 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:04:19.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:19.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:04:19.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:19 smithi026 ceph-mon[68309]: pgmap v701: 47 pgs: 41 active+clean, 4 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped; 5.4 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 1.5 KiB/s rd, 31 MiB/s wr, 504 op/s; 1/263676 objects degraded (0.000%); 19906/263676 objects misplaced (7.549%); 17 MiB/s, 272 objects/s recovering 2024-03-20T09:04:19.743 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:04:19.773 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:19.795 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:04:19.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:19.798 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:04:19.899 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:04:19.912 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:19 smithi115 ceph-mon[72476]: pgmap v701: 47 pgs: 41 active+clean, 4 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped; 5.4 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 1.5 KiB/s rd, 31 MiB/s wr, 504 op/s; 1/263676 objects degraded (0.000%); 19906/263676 objects misplaced (7.549%); 17 MiB/s, 272 objects/s recovering 2024-03-20T09:04:19.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:19 smithi122 ceph-mon[73495]: pgmap v701: 47 pgs: 41 active+clean, 4 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped; 5.4 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 1.5 KiB/s rd, 31 MiB/s wr, 504 op/s; 1/263676 objects degraded (0.000%); 19906/263676 objects misplaced (7.549%); 17 MiB/s, 272 objects/s recovering 2024-03-20T09:04:19.992 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:20.037 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:04:20.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:20.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:04:20.144 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:04:20.208 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:20.251 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:04:20.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:20.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:04:20.335 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:04:20.336 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:04:20.340 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:04:20.356 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:04:20.413 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:20.612 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 79 16 90559 90543 71.6226 42.625 0.0121265 0.0139569 2024-03-20T09:04:20.619 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:04:20.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:20.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:04:20.724 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:04:20.755 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:20.777 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:04:20.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:20.780 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:04:20.881 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:04:20.953 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:20.991 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:04:20.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:20.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:04:21.099 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:04:21.190 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:21.247 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:04:21.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:21.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:04:21.255 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:21 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:04:21.255 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:21 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:04:21.255 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:21 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:04:21.255 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:21 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:04:21.255 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:21 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:04:21.255 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:21 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:04:21.255 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:21 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:04:21.355 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:04:21.427 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:21.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:21 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:04:21.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:21 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:04:21.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:21 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:04:21.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:21 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:04:21.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:21 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:04:21.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:21 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:04:21.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:21 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:04:21.463 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:04:21.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:21 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:04:21.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:21 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:04:21.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:21 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:04:21.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:21 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:04:21.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:21 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:04:21.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:21 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:04:21.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:21 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:04:21.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:21.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:04:21.576 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:04:21.612 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:2024-03-20T09:04:21.612441+0000 min lat: 0.001051 max lat: 5.41234 avg lat: 0.0140326 2024-03-20T09:04:21.612 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T09:04:21.612 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 80 16 91188 91172 71.2187 39.3125 0.0124399 0.0140326 2024-03-20T09:04:21.631 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:21.678 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:04:21.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:21.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:04:21.786 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:04:21.815 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:21.835 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:04:21.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:21.839 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:04:21.940 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:04:21.971 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:21.993 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:04:21.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:21.998 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:04:22.098 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:04:22.126 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:22.150 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:04:22.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:22.154 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:04:22.255 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:04:22.291 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:22.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:22 smithi115 ceph-mon[72476]: pgmap v702: 47 pgs: 41 active+clean, 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep; 5.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 37 MiB/s wr, 587 op/s; 1/268482 objects degraded (0.000%); 19282/268482 objects misplaced (7.182%); 15 MiB/s, 234 objects/s recovering 2024-03-20T09:04:22.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:22 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 1/263676 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-03-20T09:04:22.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:22 smithi122 ceph-mon[73495]: pgmap v702: 47 pgs: 41 active+clean, 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep; 5.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 37 MiB/s wr, 587 op/s; 1/268482 objects degraded (0.000%); 19282/268482 objects misplaced (7.182%); 15 MiB/s, 234 objects/s recovering 2024-03-20T09:04:22.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:22 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 1/263676 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-03-20T09:04:22.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:22 smithi026 ceph-mon[68309]: pgmap v702: 47 pgs: 41 active+clean, 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep; 5.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 37 MiB/s wr, 587 op/s; 1/268482 objects degraded (0.000%); 19282/268482 objects misplaced (7.182%); 15 MiB/s, 234 objects/s recovering 2024-03-20T09:04:22.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:22 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 1/263676 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-03-20T09:04:22.612 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 81 16 91853 91837 70.8525 41.5625 0.00810637 0.0141085 2024-03-20T09:04:23.009 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:04:23.010 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-03-20T09:04:23.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-03-20T09:04:23.010 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:04:23.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.010 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:17428 2.c e149/121 2:38a16a5f:::benchmark_data_smithi122_107654_object14237:head v 149'10155, mlcod=149'10147)", 2024-03-20T09:04:23.010 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:59:49.925151+0000", 2024-03-20T09:04:23.010 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 272.907128253, 2024-03-20T09:04:23.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.098968790000000001, 2024-03-20T09:04:23.010 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:04:23.010 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:04:23.010 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:04:23.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:04:23.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:49.925151+0000", 2024-03-20T09:04:23.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:04:23.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:49.925151+0000", 2024-03-20T09:04:23.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:04:23.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:49.925152+0000", 2024-03-20T09:04:23.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8800000000000002e-07 2024-03-20T09:04:23.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:04:23.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:49.925158+0000", 2024-03-20T09:04:23.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5819999999999998e-06 2024-03-20T09:04:23.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:04:23.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:49.925171+0000", 2024-03-20T09:04:23.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2597e-05 2024-03-20T09:04:23.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:04:23.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:49.925178+0000", 2024-03-20T09:04:23.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6739999999999997e-06 2024-03-20T09:04:23.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:04:23.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:49.935649+0000", 2024-03-20T09:04:23.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010470577999999999 2024-03-20T09:04:23.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:04:23.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:49.935674+0000", 2024-03-20T09:04:23.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4850000000000001e-05 2024-03-20T09:04:23.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:04:23.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.024099+0000", 2024-03-20T09:04:23.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.088425064999999997 2024-03-20T09:04:23.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:04:23.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.024120+0000", 2024-03-20T09:04:23.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0755999999999999e-05 2024-03-20T09:04:23.014 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.014 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:04:23.014 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.015 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:17429 2.c e149/121 2:38a1796b:::benchmark_data_smithi122_107654_object86769:head v 149'10156, mlcod=149'10147)", 2024-03-20T09:04:23.015 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:59:49.928040+0000", 2024-03-20T09:04:23.015 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 272.90423888499998, 2024-03-20T09:04:23.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.096102691000000004, 2024-03-20T09:04:23.015 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:04:23.015 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:04:23.015 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:04:23.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:04:23.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:49.928040+0000", 2024-03-20T09:04:23.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:04:23.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:49.928040+0000", 2024-03-20T09:04:23.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:04:23.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:49.928041+0000", 2024-03-20T09:04:23.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3900000000000004e-07 2024-03-20T09:04:23.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:04:23.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:49.928050+0000", 2024-03-20T09:04:23.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0860000000000008e-06 2024-03-20T09:04:23.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:04:23.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:49.928064+0000", 2024-03-20T09:04:23.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.346e-05 2024-03-20T09:04:23.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:04:23.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:49.928071+0000", 2024-03-20T09:04:23.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.311e-06 2024-03-20T09:04:23.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:04:23.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:49.939744+0000", 2024-03-20T09:04:23.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01167341 2024-03-20T09:04:23.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:04:23.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:49.939764+0000", 2024-03-20T09:04:23.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9936e-05 2024-03-20T09:04:23.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:04:23.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.024122+0000", 2024-03-20T09:04:23.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.084357565999999995 2024-03-20T09:04:23.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:04:23.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.024143+0000", 2024-03-20T09:04:23.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1183e-05 2024-03-20T09:04:23.019 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.019 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:04:23.019 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.019 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:17431 2.c e149/121 2:38a1d19b:::benchmark_data_smithi122_107654_object58048:head v 149'10157, mlcod=149'10147)", 2024-03-20T09:04:23.019 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:59:49.931983+0000", 2024-03-20T09:04:23.020 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 272.90029610099998, 2024-03-20T09:04:23.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.092183867000000003, 2024-03-20T09:04:23.020 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:04:23.020 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:04:23.020 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:04:23.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:04:23.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:49.931983+0000", 2024-03-20T09:04:23.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:04:23.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:49.931983+0000", 2024-03-20T09:04:23.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:04:23.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:49.931984+0000", 2024-03-20T09:04:23.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2900000000000002e-07 2024-03-20T09:04:23.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:04:23.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:49.931995+0000", 2024-03-20T09:04:23.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1038e-05 2024-03-20T09:04:23.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:04:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:49.932004+0000", 2024-03-20T09:04:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3149999999999998e-06 2024-03-20T09:04:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:04:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:49.932012+0000", 2024-03-20T09:04:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6199999999999999e-06 2024-03-20T09:04:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:04:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:49.943554+0000", 2024-03-20T09:04:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011541813999999999 2024-03-20T09:04:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:04:23.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:49.943570+0000", 2024-03-20T09:04:23.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6215e-05 2024-03-20T09:04:23.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:04:23.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.024146+0000", 2024-03-20T09:04:23.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.080575989000000001 2024-03-20T09:04:23.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:04:23.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.024167+0000", 2024-03-20T09:04:23.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1047e-05 2024-03-20T09:04:23.024 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.024 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:04:23.024 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.024 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:17470 2.c e149/121 2:38acf834:::benchmark_data_smithi122_107654_object87711:head v 149'10189, mlcod=149'10179)", 2024-03-20T09:04:23.024 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:59:50.146536+0000", 2024-03-20T09:04:23.024 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 272.68574291900001, 2024-03-20T09:04:23.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.091690715000000006, 2024-03-20T09:04:23.024 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:04:23.024 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:04:23.024 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:04:23.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:04:23.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.146536+0000", 2024-03-20T09:04:23.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:04:23.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.146536+0000", 2024-03-20T09:04:23.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:04:23.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.146538+0000", 2024-03-20T09:04:23.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.339e-06 2024-03-20T09:04:23.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:04:23.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.146549+0000", 2024-03-20T09:04:23.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1252e-05 2024-03-20T09:04:23.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:04:23.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.146559+0000", 2024-03-20T09:04:23.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9389999999999992e-06 2024-03-20T09:04:23.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:04:23.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.146566+0000", 2024-03-20T09:04:23.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0079999999999996e-06 2024-03-20T09:04:23.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:04:23.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.152760+0000", 2024-03-20T09:04:23.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0061945510000000004 2024-03-20T09:04:23.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:04:23.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.152778+0000", 2024-03-20T09:04:23.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8022000000000001e-05 2024-03-20T09:04:23.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:04:23.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.238185+0000", 2024-03-20T09:04:23.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.085406968 2024-03-20T09:04:23.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:04:23.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.238227+0000", 2024-03-20T09:04:23.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1635999999999999e-05 2024-03-20T09:04:23.028 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.028 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:04:23.028 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.028 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:17508 2.c e149/121 2:38b477e0:::benchmark_data_smithi122_107654_object37264:head v 149'10218, mlcod=149'10213)", 2024-03-20T09:04:23.028 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:59:50.330155+0000", 2024-03-20T09:04:23.029 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 272.50212394099998, 2024-03-20T09:04:23.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.103256659, 2024-03-20T09:04:23.029 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:04:23.029 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:04:23.029 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:04:23.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:04:23.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.330155+0000", 2024-03-20T09:04:23.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:04:23.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.330155+0000", 2024-03-20T09:04:23.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:04:23.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.330156+0000", 2024-03-20T09:04:23.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0400000000000005e-07 2024-03-20T09:04:23.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:04:23.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.330163+0000", 2024-03-20T09:04:23.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2370000000000003e-06 2024-03-20T09:04:23.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:04:23.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.330176+0000", 2024-03-20T09:04:23.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2621e-05 2024-03-20T09:04:23.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:04:23.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.330183+0000", 2024-03-20T09:04:23.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5680000000000002e-06 2024-03-20T09:04:23.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:04:23.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.335130+0000", 2024-03-20T09:04:23.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004946893 2024-03-20T09:04:23.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:04:23.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.335143+0000", 2024-03-20T09:04:23.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2879e-05 2024-03-20T09:04:23.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:04:23.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.433381+0000", 2024-03-20T09:04:23.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.098237611000000002 2024-03-20T09:04:23.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:04:23.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.433412+0000", 2024-03-20T09:04:23.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1046000000000001e-05 2024-03-20T09:04:23.032 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.033 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:04:23.033 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.033 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.24550.0:115709 2.4 e149/121 2:24991c2b:::benchmark_data_smithi122_107654_object115708:head v 149'7261, mlcod=149'7257)", 2024-03-20T09:04:23.033 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:59:50.330304+0000", 2024-03-20T09:04:23.033 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 272.50197546800001, 2024-03-20T09:04:23.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.103074317, 2024-03-20T09:04:23.033 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:04:23.033 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:04:23.033 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:04:23.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:04:23.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.330304+0000", 2024-03-20T09:04:23.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:04:23.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.330304+0000", 2024-03-20T09:04:23.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:04:23.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.330305+0000", 2024-03-20T09:04:23.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1699999999999997e-07 2024-03-20T09:04:23.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:04:23.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.330328+0000", 2024-03-20T09:04:23.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3584e-05 2024-03-20T09:04:23.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:04:23.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.330340+0000", 2024-03-20T09:04:23.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1975e-05 2024-03-20T09:04:23.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:04:23.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.330347+0000", 2024-03-20T09:04:23.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9120000000000001e-06 2024-03-20T09:04:23.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:04:23.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.332019+0000", 2024-03-20T09:04:23.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001671886 2024-03-20T09:04:23.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:04:23.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.332036+0000", 2024-03-20T09:04:23.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7093000000000001e-05 2024-03-20T09:04:23.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:04:23.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.433323+0000", 2024-03-20T09:04:23.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.101287346 2024-03-20T09:04:23.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:04:23.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.433378+0000", 2024-03-20T09:04:23.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4703999999999997e-05 2024-03-20T09:04:23.037 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.037 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:04:23.037 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.038 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:17509 2.c e149/121 2:38b4cf0d:::benchmark_data_smithi122_107654_object95569:head v 149'10219, mlcod=149'10213)", 2024-03-20T09:04:23.038 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:59:50.333005+0000", 2024-03-20T09:04:23.038 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 272.49927407000001, 2024-03-20T09:04:23.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.10043097199999999, 2024-03-20T09:04:23.038 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:04:23.038 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:04:23.038 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:04:23.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:04:23.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.333005+0000", 2024-03-20T09:04:23.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:04:23.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.333005+0000", 2024-03-20T09:04:23.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:04:23.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.333006+0000", 2024-03-20T09:04:23.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1900000000000001e-07 2024-03-20T09:04:23.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:04:23.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.333015+0000", 2024-03-20T09:04:23.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5089999999999993e-06 2024-03-20T09:04:23.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:04:23.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.333027+0000", 2024-03-20T09:04:23.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2459999999999999e-05 2024-03-20T09:04:23.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:04:23.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.333035+0000", 2024-03-20T09:04:23.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7340000000000008e-06 2024-03-20T09:04:23.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:04:23.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.340219+0000", 2024-03-20T09:04:23.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0071845889999999999 2024-03-20T09:04:23.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:04:23.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.340241+0000", 2024-03-20T09:04:23.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1974e-05 2024-03-20T09:04:23.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:04:23.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.433414+0000", 2024-03-20T09:04:23.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.093172431999999999 2024-03-20T09:04:23.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:04:23.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.433436+0000", 2024-03-20T09:04:23.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2354999999999999e-05 2024-03-20T09:04:23.042 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.042 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:04:23.042 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.042 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:17510 2.c e149/121 2:38b4f64a:::benchmark_data_smithi122_107654_object10487:head v 149'10220, mlcod=149'10213)", 2024-03-20T09:04:23.042 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:59:50.338994+0000", 2024-03-20T09:04:23.042 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 272.493285394, 2024-03-20T09:04:23.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.094464838999999995, 2024-03-20T09:04:23.043 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:04:23.043 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:04:23.043 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:04:23.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:04:23.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.338994+0000", 2024-03-20T09:04:23.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:04:23.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.338994+0000", 2024-03-20T09:04:23.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:04:23.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.338995+0000", 2024-03-20T09:04:23.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.099e-06 2024-03-20T09:04:23.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:04:23.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.339007+0000", 2024-03-20T09:04:23.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1938e-05 2024-03-20T09:04:23.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:04:23.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.339021+0000", 2024-03-20T09:04:23.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4619e-05 2024-03-20T09:04:23.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:04:23.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.339031+0000", 2024-03-20T09:04:23.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3670000000000003e-06 2024-03-20T09:04:23.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:04:23.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.344638+0000", 2024-03-20T09:04:23.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056069279999999997 2024-03-20T09:04:23.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:04:23.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.344656+0000", 2024-03-20T09:04:23.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8479999999999999e-05 2024-03-20T09:04:23.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:04:23.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.433438+0000", 2024-03-20T09:04:23.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.088781475999999998 2024-03-20T09:04:23.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:04:23.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.433459+0000", 2024-03-20T09:04:23.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0931999999999999e-05 2024-03-20T09:04:23.047 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.047 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:04:23.047 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.047 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:17511 2.c e149/121 2:38b623b2:::benchmark_data_smithi122_107654_object1337:head v 149'10221, mlcod=149'10213)", 2024-03-20T09:04:23.047 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:59:50.340205+0000", 2024-03-20T09:04:23.047 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 272.49207460100001, 2024-03-20T09:04:23.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.093276827000000007, 2024-03-20T09:04:23.047 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:04:23.047 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:04:23.048 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:04:23.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:04:23.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.340205+0000", 2024-03-20T09:04:23.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:04:23.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.340205+0000", 2024-03-20T09:04:23.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:04:23.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.340205+0000", 2024-03-20T09:04:23.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5700000000000002e-07 2024-03-20T09:04:23.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:04:23.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.340215+0000", 2024-03-20T09:04:23.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4749999999999999e-06 2024-03-20T09:04:23.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:04:23.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.340224+0000", 2024-03-20T09:04:23.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1190000000000003e-06 2024-03-20T09:04:23.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:04:23.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.340230+0000", 2024-03-20T09:04:23.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5100000000000004e-06 2024-03-20T09:04:23.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:04:23.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.349292+0000", 2024-03-20T09:04:23.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0090621369999999996 2024-03-20T09:04:23.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:04:23.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.349314+0000", 2024-03-20T09:04:23.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1554999999999999e-05 2024-03-20T09:04:23.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:04:23.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.433460+0000", 2024-03-20T09:04:23.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.084146361000000003 2024-03-20T09:04:23.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:04:23.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:50.433481+0000", 2024-03-20T09:04:23.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1112999999999999e-05 2024-03-20T09:04:23.051 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.051 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:04:23.051 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.052 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.24550.0:116213 2.c e149/121 2:3f17461b:::benchmark_data_smithi122_107654_object116212:head v 149'10411, mlcod=149'10395)", 2024-03-20T09:04:23.052 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:59:51.230510+0000", 2024-03-20T09:04:23.052 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 271.60176898100002, 2024-03-20T09:04:23.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.093438833999999998, 2024-03-20T09:04:23.052 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:04:23.052 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:04:23.052 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:04:23.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:04:23.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.230510+0000", 2024-03-20T09:04:23.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:04:23.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.230510+0000", 2024-03-20T09:04:23.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:04:23.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.230511+0000", 2024-03-20T09:04:23.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.164e-06 2024-03-20T09:04:23.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:04:23.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.230541+0000", 2024-03-20T09:04:23.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9907e-05 2024-03-20T09:04:23.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:04:23.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.230556+0000", 2024-03-20T09:04:23.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4969e-05 2024-03-20T09:04:23.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:04:23.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.230565+0000", 2024-03-20T09:04:23.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6000000000000007e-06 2024-03-20T09:04:23.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:04:23.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.284839+0000", 2024-03-20T09:04:23.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.054274373000000001 2024-03-20T09:04:23.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:04:23.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.284862+0000", 2024-03-20T09:04:23.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3025999999999999e-05 2024-03-20T09:04:23.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:04:23.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.323889+0000", 2024-03-20T09:04:23.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039026480000000002 2024-03-20T09:04:23.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:04:23.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.323949+0000", 2024-03-20T09:04:23.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0315000000000001e-05 2024-03-20T09:04:23.056 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.056 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:04:23.056 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.056 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.24550.0:116196 2.4 e149/121 2:2cf10b31:::benchmark_data_smithi122_107654_object116195:head v 149'7293, mlcod=149'7288)", 2024-03-20T09:04:23.057 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:59:51.297724+0000", 2024-03-20T09:04:23.057 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 271.53455518099997, 2024-03-20T09:04:23.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.097007000999999995, 2024-03-20T09:04:23.057 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:04:23.057 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:04:23.057 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:04:23.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:04:23.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.297724+0000", 2024-03-20T09:04:23.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.057 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:04:23.057 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.297724+0000", 2024-03-20T09:04:23.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:04:23.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.297725+0000", 2024-03-20T09:04:23.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3600000000000003e-07 2024-03-20T09:04:23.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:04:23.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.297746+0000", 2024-03-20T09:04:23.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0783000000000001e-05 2024-03-20T09:04:23.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:04:23.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.297760+0000", 2024-03-20T09:04:23.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4630000000000001e-05 2024-03-20T09:04:23.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:04:23.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.297767+0000", 2024-03-20T09:04:23.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7309999999999996e-06 2024-03-20T09:04:23.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:04:23.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.322559+0000", 2024-03-20T09:04:23.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.024792076 2024-03-20T09:04:23.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:04:23.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.322579+0000", 2024-03-20T09:04:23.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9660000000000002e-05 2024-03-20T09:04:23.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:04:23.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.394690+0000", 2024-03-20T09:04:23.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.072111335999999998 2024-03-20T09:04:23.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:04:23.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.394731+0000", 2024-03-20T09:04:23.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1048999999999999e-05 2024-03-20T09:04:23.061 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.061 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:04:23.061 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.061 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.24550.0:116218 2.4 e149/121 2:26370d48:::benchmark_data_smithi122_107654_object116217:head v 149'7296, mlcod=149'7288)", 2024-03-20T09:04:23.061 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:59:51.313700+0000", 2024-03-20T09:04:23.061 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 271.51857916799997, 2024-03-20T09:04:23.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.092583980999999996, 2024-03-20T09:04:23.061 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:04:23.061 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:04:23.061 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:04:23.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:04:23.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.313700+0000", 2024-03-20T09:04:23.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:04:23.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.313700+0000", 2024-03-20T09:04:23.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:04:23.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.313700+0000", 2024-03-20T09:04:23.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6600000000000002e-07 2024-03-20T09:04:23.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:04:23.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.313725+0000", 2024-03-20T09:04:23.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4202999999999998e-05 2024-03-20T09:04:23.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:04:23.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.313733+0000", 2024-03-20T09:04:23.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.507e-06 2024-03-20T09:04:23.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:04:23.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.313739+0000", 2024-03-20T09:04:23.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2380000000000002e-06 2024-03-20T09:04:23.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:04:23.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.332315+0000", 2024-03-20T09:04:23.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018575873 2024-03-20T09:04:23.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:04:23.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.332331+0000", 2024-03-20T09:04:23.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5417000000000002e-05 2024-03-20T09:04:23.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:04:23.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.406237+0000", 2024-03-20T09:04:23.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.073906779000000006 2024-03-20T09:04:23.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:04:23.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.406284+0000", 2024-03-20T09:04:23.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6597999999999997e-05 2024-03-20T09:04:23.065 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.065 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:04:23.065 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.066 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.24550.0:116236 2.4 e149/121 2:274ff9a3:::benchmark_data_smithi122_107654_object116235:head v 149'7297, mlcod=149'7288)", 2024-03-20T09:04:23.066 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:59:51.314458+0000", 2024-03-20T09:04:23.066 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 271.51782123700002, 2024-03-20T09:04:23.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.091855622999999997, 2024-03-20T09:04:23.066 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:04:23.066 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:04:23.066 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:04:23.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:04:23.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.314458+0000", 2024-03-20T09:04:23.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:04:23.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.314458+0000", 2024-03-20T09:04:23.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:04:23.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.314458+0000", 2024-03-20T09:04:23.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5499999999999998e-07 2024-03-20T09:04:23.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:04:23.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.314476+0000", 2024-03-20T09:04:23.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7467999999999999e-05 2024-03-20T09:04:23.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:04:23.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.314487+0000", 2024-03-20T09:04:23.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0730000000000001e-05 2024-03-20T09:04:23.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:04:23.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.314495+0000", 2024-03-20T09:04:23.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6989999999999992e-06 2024-03-20T09:04:23.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:04:23.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.335646+0000", 2024-03-20T09:04:23.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021150380999999999 2024-03-20T09:04:23.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:04:23.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.335666+0000", 2024-03-20T09:04:23.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9853000000000001e-05 2024-03-20T09:04:23.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:04:23.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.406287+0000", 2024-03-20T09:04:23.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.070621499000000004 2024-03-20T09:04:23.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:04:23.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:51.406314+0000", 2024-03-20T09:04:23.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6537999999999999e-05 2024-03-20T09:04:23.070 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.070 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:04:23.070 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.070 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:18194 2.c e150/121 2:39c891b1:::benchmark_data_smithi122_107654_object35198:head v 150'10772, mlcod=150'10766)", 2024-03-20T09:04:23.070 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:59:53.391352+0000", 2024-03-20T09:04:23.070 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 269.44092687599999, 2024-03-20T09:04:23.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.096555207000000004, 2024-03-20T09:04:23.070 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:04:23.070 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:04:23.071 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:04:23.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:04:23.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.391352+0000", 2024-03-20T09:04:23.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:04:23.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.391352+0000", 2024-03-20T09:04:23.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:04:23.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.391353+0000", 2024-03-20T09:04:23.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1200000000000001e-07 2024-03-20T09:04:23.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:04:23.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.391358+0000", 2024-03-20T09:04:23.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1699999999999996e-06 2024-03-20T09:04:23.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:04:23.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.391368+0000", 2024-03-20T09:04:23.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4579999999999999e-06 2024-03-20T09:04:23.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:04:23.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.391375+0000", 2024-03-20T09:04:23.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.153e-06 2024-03-20T09:04:23.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:04:23.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.395956+0000", 2024-03-20T09:04:23.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0045807130000000001 2024-03-20T09:04:23.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:04:23.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.395977+0000", 2024-03-20T09:04:23.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0811000000000001e-05 2024-03-20T09:04:23.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:04:23.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.487849+0000", 2024-03-20T09:04:23.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.091872316999999995 2024-03-20T09:04:23.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:04:23.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.487908+0000", 2024-03-20T09:04:23.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8672999999999999e-05 2024-03-20T09:04:23.074 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.074 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:04:23.074 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.075 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:18195 2.c e150/121 2:39c9b1a4:::benchmark_data_smithi122_107654_object89166:head v 150'10773, mlcod=150'10766)", 2024-03-20T09:04:23.075 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:59:53.391392+0000", 2024-03-20T09:04:23.075 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 269.44088745599998, 2024-03-20T09:04:23.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.115279932, 2024-03-20T09:04:23.075 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:04:23.075 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:04:23.075 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:04:23.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:04:23.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.391392+0000", 2024-03-20T09:04:23.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:04:23.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.391392+0000", 2024-03-20T09:04:23.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:04:23.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.391392+0000", 2024-03-20T09:04:23.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5899999999999998e-07 2024-03-20T09:04:23.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:04:23.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.391396+0000", 2024-03-20T09:04:23.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9700000000000001e-06 2024-03-20T09:04:23.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:04:23.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.391402+0000", 2024-03-20T09:04:23.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6250000000000004e-06 2024-03-20T09:04:23.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:04:23.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.391406+0000", 2024-03-20T09:04:23.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4290000000000003e-06 2024-03-20T09:04:23.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:04:23.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.400151+0000", 2024-03-20T09:04:23.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0087445639999999998 2024-03-20T09:04:23.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:04:23.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.400169+0000", 2024-03-20T09:04:23.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8698999999999998e-05 2024-03-20T09:04:23.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:04:23.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.506620+0000", 2024-03-20T09:04:23.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.106450249 2024-03-20T09:04:23.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:04:23.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.506672+0000", 2024-03-20T09:04:23.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2136999999999999e-05 2024-03-20T09:04:23.079 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.079 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:04:23.079 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.079 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:18196 2.c e150/121 2:39c9c790:::benchmark_data_smithi122_107654_object54584:head v 150'10774, mlcod=150'10766)", 2024-03-20T09:04:23.079 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:59:53.393573+0000", 2024-03-20T09:04:23.079 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 269.43870574300001, 2024-03-20T09:04:23.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.11313285200000001, 2024-03-20T09:04:23.079 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:04:23.079 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:04:23.079 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:04:23.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:04:23.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.393573+0000", 2024-03-20T09:04:23.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:04:23.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.393573+0000", 2024-03-20T09:04:23.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:04:23.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.393574+0000", 2024-03-20T09:04:23.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1500000000000004e-07 2024-03-20T09:04:23.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:04:23.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.393580+0000", 2024-03-20T09:04:23.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4990000000000002e-06 2024-03-20T09:04:23.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:04:23.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.393589+0000", 2024-03-20T09:04:23.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5859999999999994e-06 2024-03-20T09:04:23.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:04:23.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.393595+0000", 2024-03-20T09:04:23.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8000000000000004e-06 2024-03-20T09:04:23.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:04:23.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.407016+0000", 2024-03-20T09:04:23.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013421367 2024-03-20T09:04:23.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:04:23.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.407037+0000", 2024-03-20T09:04:23.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0727999999999999e-05 2024-03-20T09:04:23.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:04:23.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.506675+0000", 2024-03-20T09:04:23.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.099638172999999997 2024-03-20T09:04:23.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:04:23.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.506706+0000", 2024-03-20T09:04:23.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0984e-05 2024-03-20T09:04:23.083 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.083 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:04:23.083 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.084 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:18197 2.c e150/121 2:39cab7e3:::benchmark_data_smithi122_107654_object23118:head v 150'10775, mlcod=150'10766)", 2024-03-20T09:04:23.084 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:59:53.397373+0000", 2024-03-20T09:04:23.084 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 269.43490630299999, 2024-03-20T09:04:23.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.10935983000000001, 2024-03-20T09:04:23.084 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:04:23.084 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:04:23.085 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:04:23.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:04:23.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.397373+0000", 2024-03-20T09:04:23.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:04:23.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.397373+0000", 2024-03-20T09:04:23.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:04:23.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.397374+0000", 2024-03-20T09:04:23.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0599999999999999e-07 2024-03-20T09:04:23.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:04:23.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.397382+0000", 2024-03-20T09:04:23.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9659999999999996e-06 2024-03-20T09:04:23.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:04:23.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.397400+0000", 2024-03-20T09:04:23.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7878999999999999e-05 2024-03-20T09:04:23.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.087 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:04:23.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.397408+0000", 2024-03-20T09:04:23.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7909999999999993e-06 2024-03-20T09:04:23.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.087 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:04:23.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.411757+0000", 2024-03-20T09:04:23.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014348334000000001 2024-03-20T09:04:23.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:04:23.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.411776+0000", 2024-03-20T09:04:23.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9641e-05 2024-03-20T09:04:23.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:04:23.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.506709+0000", 2024-03-20T09:04:23.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.094932384999999994 2024-03-20T09:04:23.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:04:23.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.506733+0000", 2024-03-20T09:04:23.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3927999999999999e-05 2024-03-20T09:04:23.089 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.089 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:04:23.089 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.089 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.24550.0:117977 2.4 e150/121 2:23cb42c7:::benchmark_data_smithi122_107654_object117976:head v 150'7411, mlcod=150'7409)", 2024-03-20T09:04:23.089 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:59:53.397433+0000", 2024-03-20T09:04:23.089 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 269.43484642200002, 2024-03-20T09:04:23.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.109332975, 2024-03-20T09:04:23.089 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:04:23.089 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:04:23.089 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:04:23.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:04:23.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.397433+0000", 2024-03-20T09:04:23.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:04:23.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.397433+0000", 2024-03-20T09:04:23.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:04:23.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.397433+0000", 2024-03-20T09:04:23.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0299999999999999e-07 2024-03-20T09:04:23.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:04:23.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.397459+0000", 2024-03-20T09:04:23.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5449999999999999e-05 2024-03-20T09:04:23.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:04:23.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.397467+0000", 2024-03-20T09:04:23.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5599999999999994e-06 2024-03-20T09:04:23.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:04:23.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.397473+0000", 2024-03-20T09:04:23.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0270000000000001e-06 2024-03-20T09:04:23.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:04:23.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.404773+0000", 2024-03-20T09:04:23.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0072997000000000001 2024-03-20T09:04:23.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:04:23.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.404791+0000", 2024-03-20T09:04:23.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7946000000000001e-05 2024-03-20T09:04:23.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:04:23.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.506735+0000", 2024-03-20T09:04:23.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.101943993 2024-03-20T09:04:23.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:04:23.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.506766+0000", 2024-03-20T09:04:23.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0796000000000001e-05 2024-03-20T09:04:23.093 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.093 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:04:23.093 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.093 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:18198 2.c e150/121 2:39cbcc8e:::benchmark_data_smithi122_107654_object77773:head v 150'10776, mlcod=150'10766)", 2024-03-20T09:04:23.093 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:59:53.398613+0000", 2024-03-20T09:04:23.093 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 269.433666211, 2024-03-20T09:04:23.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.108170949, 2024-03-20T09:04:23.094 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:04:23.094 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:04:23.094 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:04:23.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:04:23.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.398613+0000", 2024-03-20T09:04:23.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:04:23.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.398613+0000", 2024-03-20T09:04:23.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:04:23.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.398614+0000", 2024-03-20T09:04:23.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0399999999999995e-07 2024-03-20T09:04:23.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:04:23.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.398620+0000", 2024-03-20T09:04:23.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.263e-06 2024-03-20T09:04:23.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:04:23.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.398631+0000", 2024-03-20T09:04:23.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1205e-05 2024-03-20T09:04:23.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:04:23.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.398638+0000", 2024-03-20T09:04:23.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1010000000000003e-06 2024-03-20T09:04:23.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:04:23.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.416585+0000", 2024-03-20T09:04:23.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017946444999999998 2024-03-20T09:04:23.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:04:23.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.416604+0000", 2024-03-20T09:04:23.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9701e-05 2024-03-20T09:04:23.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:04:23.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.506769+0000", 2024-03-20T09:04:23.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.090164266000000007 2024-03-20T09:04:23.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:04:23.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:53.506784+0000", 2024-03-20T09:04:23.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5264000000000002e-05 2024-03-20T09:04:23.098 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.098 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:04:23.098 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.099 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:18421 2.c e150/121 2:3a295916:::benchmark_data_smithi122_107654_object98583:head v 150'10954, mlcod=150'10937)", 2024-03-20T09:04:23.099 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:59:54.378986+0000", 2024-03-20T09:04:23.099 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 268.45329286200001, 2024-03-20T09:04:23.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.094296002000000004, 2024-03-20T09:04:23.099 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:04:23.099 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:04:23.099 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:04:23.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:04:23.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:54.378986+0000", 2024-03-20T09:04:23.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:04:23.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:54.378986+0000", 2024-03-20T09:04:23.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:04:23.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:04:23.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:54.378987+0000", 2024-03-20T09:04:23.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.79e-07 2024-03-20T09:04:23.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:04:23.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:54.378990+0000", 2024-03-20T09:04:23.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7900000000000001e-06 2024-03-20T09:04:23.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:04:23.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:54.378996+0000", 2024-03-20T09:04:23.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8100000000000003e-06 2024-03-20T09:04:23.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:04:23.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:54.379001+0000", 2024-03-20T09:04:23.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.729e-06 2024-03-20T09:04:23.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:04:23.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:54.388449+0000", 2024-03-20T09:04:23.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094477569999999993 2024-03-20T09:04:23.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:04:23.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:54.388480+0000", 2024-03-20T09:04:23.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0846999999999998e-05 2024-03-20T09:04:23.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:04:23.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:54.473225+0000", 2024-03-20T09:04:23.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.084745037999999995 2024-03-20T09:04:23.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:04:23.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:04:23.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:04:23.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:54.473282+0000", 2024-03-20T09:04:23.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7751999999999999e-05 2024-03-20T09:04:23.103 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.103 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:04:23.103 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.103 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:04:23.103 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:04:23.103 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:04:23.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:23 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:04:23.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:23 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:04:23.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:23 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:04:23.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:23 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:04:23.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:23 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:04:23.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:23 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:04:23.612 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 82 16 92519 92503 70.4961 41.625 0.0137814 0.014173 2024-03-20T09:04:24.355 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:04:24.377 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:04:24.404 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:04:24.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:24 smithi115 ceph-mon[72476]: osdmap e270: 12 total, 12 up, 11 in 2024-03-20T09:04:24.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:24 smithi115 ceph-mon[72476]: pgmap v704: 47 pgs: 41 active+clean, 4 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped; 5.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 35 MiB/s wr, 563 op/s; 1/271677 objects degraded (0.000%); 19146/271677 objects misplaced (7.047%); 9.7 MiB/s, 155 objects/s recovering 2024-03-20T09:04:24.431 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:04:24.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:24 smithi122 ceph-mon[73495]: osdmap e270: 12 total, 12 up, 11 in 2024-03-20T09:04:24.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:24 smithi122 ceph-mon[73495]: pgmap v704: 47 pgs: 41 active+clean, 4 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped; 5.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 35 MiB/s wr, 563 op/s; 1/271677 objects degraded (0.000%); 19146/271677 objects misplaced (7.047%); 9.7 MiB/s, 155 objects/s recovering 2024-03-20T09:04:24.461 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:04:24.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:24 smithi026 ceph-mon[68309]: osdmap e270: 12 total, 12 up, 11 in 2024-03-20T09:04:24.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:24 smithi026 ceph-mon[68309]: pgmap v704: 47 pgs: 41 active+clean, 4 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped; 5.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 35 MiB/s wr, 563 op/s; 1/271677 objects degraded (0.000%); 19146/271677 objects misplaced (7.047%); 9.7 MiB/s, 155 objects/s recovering 2024-03-20T09:04:24.498 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:04:24.524 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:04:24.546 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:04:24.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:24.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:04:24.558 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:04:24.583 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:04:24.609 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:04:24.612 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 83 16 93314 93298 70.2453 49.6875 0.00939363 0.0142307 2024-03-20T09:04:24.633 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:04:24.657 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:04:24.658 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:04:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:04:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:04:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:04:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:04:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:04:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:04:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:04:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:04:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:04:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:04:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:04:24.669 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:04:24.699 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:25.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:25 smithi115 ceph-mon[72476]: 3.c deep-scrub ok 2024-03-20T09:04:25.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:25 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2807429038' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:04:25.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:25 smithi122 ceph-mon[73495]: 3.c deep-scrub ok 2024-03-20T09:04:25.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:25 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2807429038' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:04:25.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:25 smithi026 ceph-mon[68309]: 3.c deep-scrub ok 2024-03-20T09:04:25.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:25 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2807429038' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:04:25.612 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 84 15 94220 94205 70.0839 56.6875 0.0123805 0.0142632 2024-03-20T09:04:25.762 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:04:25.762 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-20T09:04:25.762 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:04:26.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:26 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:04:26.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:26 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:04:26.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:26 smithi115 ceph-mon[72476]: osdmap e271: 12 total, 12 up, 11 in 2024-03-20T09:04:26.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:26 smithi115 ceph-mon[72476]: pgmap v706: 47 pgs: 41 active+clean, 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep; 5.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 45 MiB/s wr, 722 op/s; 1/278526 objects degraded (0.000%); 17972/278526 objects misplaced (6.453%); 17 MiB/s, 270 objects/s recovering 2024-03-20T09:04:26.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:26 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:04:26.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:26 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:04:26.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:26 smithi122 ceph-mon[73495]: osdmap e271: 12 total, 12 up, 11 in 2024-03-20T09:04:26.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:26 smithi122 ceph-mon[73495]: pgmap v706: 47 pgs: 41 active+clean, 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep; 5.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 45 MiB/s wr, 722 op/s; 1/278526 objects degraded (0.000%); 17972/278526 objects misplaced (6.453%); 17 MiB/s, 270 objects/s recovering 2024-03-20T09:04:26.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:26 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:04:26.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:26 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:04:26.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:26 smithi026 ceph-mon[68309]: osdmap e271: 12 total, 12 up, 11 in 2024-03-20T09:04:26.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:26 smithi026 ceph-mon[68309]: pgmap v706: 47 pgs: 41 active+clean, 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep; 5.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 45 MiB/s wr, 722 op/s; 1/278526 objects degraded (0.000%); 17972/278526 objects misplaced (6.453%); 17 MiB/s, 270 objects/s recovering 2024-03-20T09:04:26.527 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-20T09:04:26.580 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:04:26.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:26.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:04:26.612 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 85 16 95112 95096 69.9145 55.6875 0.0266353 0.0142984 2024-03-20T09:04:26.684 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:04:26.724 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:26.804 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:04:26.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:26.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:04:26.908 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:04:26.941 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:27.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:27 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 1/278526 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-03-20T09:04:27.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:27 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 1/278526 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-03-20T09:04:27.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:27 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 1/278526 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-03-20T09:04:27.614 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 86 16 96054 96038 69.7861 58.875 0.0185212 0.0143245 2024-03-20T09:04:27.692 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:04:27.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:27.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:04:27.796 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:04:27.826 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:27.852 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:04:27.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:27.855 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:04:27.957 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:04:27.987 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:28.008 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:04:28.015 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:04:28.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:28.019 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:04:28.121 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:04:28.135 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:28 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:04:28.135 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:28 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:04:28.135 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:28 smithi122 ceph-mon[73495]: pgmap v707: 47 pgs: 42 active+clean, 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped; 5.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 49 MiB/s wr, 780 op/s; 1/282417 objects degraded (0.000%); 17844/282417 objects misplaced (6.318%); 15 MiB/s, 242 objects/s recovering 2024-03-20T09:04:28.188 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:28.225 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:04:28.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:28.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:04:28.335 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:04:28.373 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:28.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:28 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:04:28.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:28 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:04:28.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:28 smithi115 ceph-mon[72476]: pgmap v707: 47 pgs: 42 active+clean, 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped; 5.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 49 MiB/s wr, 780 op/s; 1/282417 objects degraded (0.000%); 17844/282417 objects misplaced (6.318%); 15 MiB/s, 242 objects/s recovering 2024-03-20T09:04:28.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:28 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:04:28.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:28 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:04:28.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:28 smithi026 ceph-mon[68309]: pgmap v707: 47 pgs: 42 active+clean, 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped; 5.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 49 MiB/s wr, 780 op/s; 1/282417 objects degraded (0.000%); 17844/282417 objects misplaced (6.318%); 15 MiB/s, 242 objects/s recovering 2024-03-20T09:04:28.612 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 87 16 96971 96955 69.6426 57.3125 0.0152062 0.014353 2024-03-20T09:04:29.186 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:04:29.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:29 smithi026 ceph-mon[68309]: osdmap e272: 12 total, 12 up, 11 in 2024-03-20T09:04:29.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:29 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/167074379' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:04:29.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:29 smithi115 ceph-mon[72476]: osdmap e272: 12 total, 12 up, 11 in 2024-03-20T09:04:29.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:29 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/167074379' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:04:29.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:29 smithi122 ceph-mon[73495]: osdmap e272: 12 total, 12 up, 11 in 2024-03-20T09:04:29.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:29 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/167074379' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:04:29.612 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 88 16 97922 97906 69.5266 59.4375 0.003956 0.014377 2024-03-20T09:04:29.726 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:04:29.934 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:04:29.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:29.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:04:30.039 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:04:30.069 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:30.092 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:04:30.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:30.095 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:04:30.178 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:30 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:04:30.179 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:30 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/167074379' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:04:30.179 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:30 smithi026 ceph-mon[68309]: osdmap e273: 12 total, 12 up, 11 in 2024-03-20T09:04:30.179 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:30 smithi026 ceph-mon[68309]: pgmap v710: 47 pgs: 42 active+clean, 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped; 5.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 49 MiB/s wr, 787 op/s; 1/285849 objects degraded (0.000%); 17395/285849 objects misplaced (6.085%); 16 MiB/s, 252 objects/s recovering 2024-03-20T09:04:30.195 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:04:30.238 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:30.260 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:04:30.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:30.270 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:04:30.371 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:04:30.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:30 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:04:30.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:30 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/167074379' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:04:30.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:30 smithi122 ceph-mon[73495]: osdmap e273: 12 total, 12 up, 11 in 2024-03-20T09:04:30.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:30 smithi122 ceph-mon[73495]: pgmap v710: 47 pgs: 42 active+clean, 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped; 5.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 49 MiB/s wr, 787 op/s; 1/285849 objects degraded (0.000%); 17395/285849 objects misplaced (6.085%); 16 MiB/s, 252 objects/s recovering 2024-03-20T09:04:30.467 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:30 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:04:30.468 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:30 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/167074379' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:04:30.468 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:30 smithi115 ceph-mon[72476]: osdmap e273: 12 total, 12 up, 11 in 2024-03-20T09:04:30.468 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:30 smithi115 ceph-mon[72476]: pgmap v710: 47 pgs: 42 active+clean, 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped; 5.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 49 MiB/s wr, 787 op/s; 1/285849 objects degraded (0.000%); 17395/285849 objects misplaced (6.085%); 16 MiB/s, 252 objects/s recovering 2024-03-20T09:04:30.468 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:30.510 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:04:30.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:30.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:04:30.612 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 89 16 98856 98840 69.4013 58.375 0.00988426 0.0144041 2024-03-20T09:04:30.614 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:04:30.660 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:30.688 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:04:30.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:30.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:04:30.792 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:04:30.837 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:30.906 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:04:30.907 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:04:31.613 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 90 4 99779 99775 69.2794 58.4375 0.0117124 0.0144297 2024-03-20T09:04:31.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:31 smithi122 ceph-mon[73495]: pgmap v711: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 42 active+clean, 4 active+recovering+undersized+remapped; 5.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 51 MiB/s wr, 814 op/s; 1/292398 objects degraded (0.000%); 16682/292398 objects misplaced (5.705%); 11 MiB/s, 180 objects/s recovering 2024-03-20T09:04:31.956 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:04:31.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:31 smithi026 ceph-mon[68309]: pgmap v711: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 42 active+clean, 4 active+recovering+undersized+remapped; 5.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 51 MiB/s wr, 814 op/s; 1/292398 objects degraded (0.000%); 16682/292398 objects misplaced (5.705%); 11 MiB/s, 180 objects/s recovering 2024-03-20T09:04:32.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:31 smithi115 ceph-mon[72476]: pgmap v711: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 42 active+clean, 4 active+recovering+undersized+remapped; 5.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 51 MiB/s wr, 814 op/s; 1/292398 objects degraded (0.000%); 16682/292398 objects misplaced (5.705%); 11 MiB/s, 180 objects/s recovering 2024-03-20T09:04:32.259 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11] out_osds: [9] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-03-20T09:04:32.259 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:04:32.259 INFO:tasks.thrashosds.thrasher:Killing osd 5, live_osds are [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-03-20T09:04:32.260 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-20T09:04:32.260 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.5 2024-03-20T09:04:32.613 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Total time run: 90.0165 2024-03-20T09:04:32.613 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Total writes made: 99779 2024-03-20T09:04:32.613 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Write size: 65536 2024-03-20T09:04:32.613 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Object size: 65536 2024-03-20T09:04:32.613 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Bandwidth (MB/sec): 69.2783 2024-03-20T09:04:32.613 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Stddev Bandwidth: 52.5735 2024-03-20T09:04:32.613 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Max bandwidth (MB/sec): 229.188 2024-03-20T09:04:32.613 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Min bandwidth (MB/sec): 0 2024-03-20T09:04:32.613 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Average IOPS: 1108 2024-03-20T09:04:32.614 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Stddev IOPS: 841.18 2024-03-20T09:04:32.614 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Max IOPS: 3667 2024-03-20T09:04:32.614 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Min IOPS: 0 2024-03-20T09:04:32.614 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Average Latency(s): 0.0144302 2024-03-20T09:04:32.614 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Stddev Latency(s): 0.0803357 2024-03-20T09:04:32.614 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Max latency(s): 5.41234 2024-03-20T09:04:32.614 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Min latency(s): 0.001051 2024-03-20T09:04:32.629 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Cleaning up (deleting benchmark objects) 2024-03-20T09:04:32.656 INFO:journalctl@ceph.osd.5.smithi115.stdout:Mar 20 09:04:32 smithi115 systemd[1]: Stopping Ceph osd.5 for 852fbb68-e696-11ee-95c9-87774f69a715... 2024-03-20T09:04:32.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:32 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 1/292398 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-03-20T09:04:32.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:32 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 1/292398 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-03-20T09:04:33.158 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:32 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 1/292398 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 2024-03-20T09:04:33.491 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:04:33.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:33.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:04:33.511 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:04:33.595 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:04:33.631 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:33.655 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:04:33.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:33.661 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:04:33.762 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:04:33.794 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:33.813 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:04:33.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:33.816 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:04:33.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:33 smithi115 ceph-mon[72476]: osd.5 marked itself down and dead 2024-03-20T09:04:33.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:33 smithi115 ceph-mon[72476]: pgmap v712: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 42 active+clean, 4 active+recovering+undersized+remapped; 6.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 46 MiB/s wr, 734 op/s; 1/296016 objects degraded (0.000%); 16522/296016 objects misplaced (5.581%); 10 MiB/s, 167 objects/s recovering 2024-03-20T09:04:33.906 INFO:journalctl@ceph.osd.5.smithi115.stdout:Mar 20 09:04:33 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-5[85004]: 2024-03-20T09:04:33.586+0000 7fa7546ed700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-20T09:04:33.906 INFO:journalctl@ceph.osd.5.smithi115.stdout:Mar 20 09:04:33 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-5[85004]: 2024-03-20T09:04:33.586+0000 7fa7546ed700 -1 osd.5 273 *** Got signal Terminated *** 2024-03-20T09:04:33.907 INFO:journalctl@ceph.osd.5.smithi115.stdout:Mar 20 09:04:33 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-5[85004]: 2024-03-20T09:04:33.586+0000 7fa7546ed700 -1 osd.5 273 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-20T09:04:33.917 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:04:33.942 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:33 smithi122 ceph-mon[73495]: osd.5 marked itself down and dead 2024-03-20T09:04:33.942 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:33 smithi122 ceph-mon[73495]: pgmap v712: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 42 active+clean, 4 active+recovering+undersized+remapped; 6.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 46 MiB/s wr, 734 op/s; 1/296016 objects degraded (0.000%); 16522/296016 objects misplaced (5.581%); 10 MiB/s, 167 objects/s recovering 2024-03-20T09:04:33.944 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:33.965 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:04:33.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:33.967 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:04:33.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:33 smithi026 ceph-mon[68309]: osd.5 marked itself down and dead 2024-03-20T09:04:33.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:33 smithi026 ceph-mon[68309]: pgmap v712: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 42 active+clean, 4 active+recovering+undersized+remapped; 6.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 46 MiB/s wr, 734 op/s; 1/296016 objects degraded (0.000%); 16522/296016 objects misplaced (5.581%); 10 MiB/s, 167 objects/s recovering 2024-03-20T09:04:34.068 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:04:34.101 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:34.123 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:04:34.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:34.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:04:34.228 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:04:34.256 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:34.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:34 smithi026 ceph-mon[68309]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-20T09:04:34.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:34 smithi026 ceph-mon[68309]: osdmap e274: 12 total, 11 up, 11 in 2024-03-20T09:04:35.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:34 smithi115 ceph-mon[72476]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-20T09:04:35.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:34 smithi115 ceph-mon[72476]: osdmap e274: 12 total, 11 up, 11 in 2024-03-20T09:04:35.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:34 smithi122 ceph-mon[73495]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-20T09:04:35.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:34 smithi122 ceph-mon[73495]: osdmap e274: 12 total, 11 up, 11 in 2024-03-20T09:04:35.906 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:04:35.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:35.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:04:36.011 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:04:36.012 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:35 smithi026 ceph-mon[68309]: osdmap e275: 12 total, 11 up, 11 in 2024-03-20T09:04:36.012 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:35 smithi026 ceph-mon[68309]: pgmap v715: 47 pgs: 2 remapped+peering, 1 peering, 3 stale+active+clean, 5 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 30 active+clean, 3 active+undersized, 1 active+recovering+undersized+remapped; 6.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 158 B/s rd, 53 MiB/s wr, 1.01k op/s; 19908/296331 objects degraded (6.718%); 8381/296331 objects misplaced (2.828%); 27 MiB/s, 430 objects/s recovering 2024-03-20T09:04:36.043 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:35 smithi115 ceph-mon[72476]: osdmap e275: 12 total, 11 up, 11 in 2024-03-20T09:04:36.043 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:35 smithi115 ceph-mon[72476]: pgmap v715: 47 pgs: 2 remapped+peering, 1 peering, 3 stale+active+clean, 5 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 30 active+clean, 3 active+undersized, 1 active+recovering+undersized+remapped; 6.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 158 B/s rd, 53 MiB/s wr, 1.01k op/s; 19908/296331 objects degraded (6.718%); 8381/296331 objects misplaced (2.828%); 27 MiB/s, 430 objects/s recovering 2024-03-20T09:04:36.046 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:36.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:35 smithi122 ceph-mon[73495]: osdmap e275: 12 total, 11 up, 11 in 2024-03-20T09:04:36.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:35 smithi122 ceph-mon[73495]: pgmap v715: 47 pgs: 2 remapped+peering, 1 peering, 3 stale+active+clean, 5 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 30 active+clean, 3 active+undersized, 1 active+recovering+undersized+remapped; 6.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 158 B/s rd, 53 MiB/s wr, 1.01k op/s; 19908/296331 objects degraded (6.718%); 8381/296331 objects misplaced (2.828%); 27 MiB/s, 430 objects/s recovering 2024-03-20T09:04:36.419 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-20T09:04:36.854 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:04:37.084 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:36 smithi026 ceph-mon[68309]: osdmap e276: 12 total, 11 up, 11 in 2024-03-20T09:04:37.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:36 smithi115 ceph-mon[72476]: osdmap e276: 12 total, 11 up, 11 in 2024-03-20T09:04:37.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:36 smithi122 ceph-mon[73495]: osdmap e276: 12 total, 11 up, 11 in 2024-03-20T09:04:37.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:37 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 19908/296331 objects degraded (6.718%), 7 pgs degraded (PG_DEGRADED) 2024-03-20T09:04:37.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:37 smithi026 ceph-mon[68309]: osdmap e277: 12 total, 11 up, 11 in 2024-03-20T09:04:37.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:37 smithi026 ceph-mon[68309]: pgmap v718: 47 pgs: 2 remapped+peering, 1 peering, 2 stale+active+clean, 6 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 29 active+clean, 4 active+undersized, 1 active+recovering+undersized+remapped; 6.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 1.5 KiB/s rd, 17 MiB/s wr, 558 op/s; 19909/295989 objects degraded (6.726%); 8381/295989 objects misplaced (2.832%); 24 MiB/s, 390 objects/s recovering 2024-03-20T09:04:38.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:37 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 19908/296331 objects degraded (6.718%), 7 pgs degraded (PG_DEGRADED) 2024-03-20T09:04:38.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:37 smithi115 ceph-mon[72476]: osdmap e277: 12 total, 11 up, 11 in 2024-03-20T09:04:38.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:37 smithi115 ceph-mon[72476]: pgmap v718: 47 pgs: 2 remapped+peering, 1 peering, 2 stale+active+clean, 6 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 29 active+clean, 4 active+undersized, 1 active+recovering+undersized+remapped; 6.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 1.5 KiB/s rd, 17 MiB/s wr, 558 op/s; 19909/295989 objects degraded (6.726%); 8381/295989 objects misplaced (2.832%); 24 MiB/s, 390 objects/s recovering 2024-03-20T09:04:38.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:37 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 19908/296331 objects degraded (6.718%), 7 pgs degraded (PG_DEGRADED) 2024-03-20T09:04:38.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:37 smithi122 ceph-mon[73495]: osdmap e277: 12 total, 11 up, 11 in 2024-03-20T09:04:38.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:37 smithi122 ceph-mon[73495]: pgmap v718: 47 pgs: 2 remapped+peering, 1 peering, 2 stale+active+clean, 6 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 29 active+clean, 4 active+undersized, 1 active+recovering+undersized+remapped; 6.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 1.5 KiB/s rd, 17 MiB/s wr, 558 op/s; 19909/295989 objects degraded (6.726%); 8381/295989 objects misplaced (2.832%); 24 MiB/s, 390 objects/s recovering 2024-03-20T09:04:38.501 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:04:38.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:38.507 INFO:journalctl@ceph.osd.5.smithi115.stdout:Mar 20 09:04:38 smithi115 bash[125125]: ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-5 2024-03-20T09:04:38.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:04:38.542 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:04:38.608 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:04:38.638 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:39.187 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:04:39.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:39.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:04:39.236 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:04:39.256 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:04:39.275 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:04:39.292 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:04:39.297 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:04:39.318 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:04:39.324 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:39.340 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:04:39.352 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:04:39.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:39.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:04:39.361 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:04:39.383 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:04:39.404 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:04:39.426 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:04:39.447 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:04:39.448 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:04:39.448 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:04:39.448 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:04:39.448 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:04:39.448 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:04:39.448 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-20T09:04:39.448 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:04:39.448 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:04:39.448 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:04:39.448 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:04:39.448 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:04:39.449 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:04:39.458 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:04:39.489 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:39.588 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:04:39.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:39.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:04:39.691 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:04:39.726 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:39.827 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:04:39.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:39.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:04:39.932 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:04:39.958 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:39.977 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:04:39.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:39.980 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:04:40.081 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:04:40.109 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:40.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:39 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:04:40.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:39 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:04:40.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:39 smithi122 ceph-mon[73495]: osdmap e278: 12 total, 11 up, 11 in 2024-03-20T09:04:40.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:39 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/658522049' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:04:40.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:39 smithi122 ceph-mon[73495]: pgmap v720: 47 pgs: 2 remapped+peering, 7 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 28 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 6.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 3.3 KiB/s rd, 8.5 MiB/s wr, 341 op/s; 22973/295704 objects degraded (7.769%); 7724/295704 objects misplaced (2.612%); 22 MiB/s, 350 objects/s recovering 2024-03-20T09:04:40.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:39 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:04:40.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:39 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:04:40.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:39 smithi026 ceph-mon[68309]: osdmap e278: 12 total, 11 up, 11 in 2024-03-20T09:04:40.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:39 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/658522049' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:04:40.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:39 smithi026 ceph-mon[68309]: pgmap v720: 47 pgs: 2 remapped+peering, 7 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 28 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 6.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 3.3 KiB/s rd, 8.5 MiB/s wr, 341 op/s; 22973/295704 objects degraded (7.769%); 7724/295704 objects misplaced (2.612%); 22 MiB/s, 350 objects/s recovering 2024-03-20T09:04:40.273 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:39 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:04:40.273 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:39 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:04:40.273 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:39 smithi115 ceph-mon[72476]: osdmap e278: 12 total, 11 up, 11 in 2024-03-20T09:04:40.273 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:39 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/658522049' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:04:40.273 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:39 smithi115 ceph-mon[72476]: pgmap v720: 47 pgs: 2 remapped+peering, 7 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 28 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 6.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 3.3 KiB/s rd, 8.5 MiB/s wr, 341 op/s; 22973/295704 objects degraded (7.769%); 7724/295704 objects misplaced (2.612%); 22 MiB/s, 350 objects/s recovering 2024-03-20T09:04:40.280 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.5.service' 2024-03-20T09:04:40.326 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-20T09:04:40.326 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-20T09:04:40.326 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.5 2024-03-20T09:04:40.326 INFO:tasks.thrashosds.thrasher:remote for osd 5 is ubuntu@smithi115.front.sepia.ceph.com 2024-03-20T09:04:40.326 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -n osd.5 --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op list-pgs 2024-03-20T09:04:40.360 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:04:40.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:40.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:04:40.371 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:04:40.372 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:04:40.464 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:04:40.491 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:40.510 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:04:40.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:40.513 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:04:40.569 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/osd.5/config 2024-03-20T09:04:40.613 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:04:40.642 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:40.841 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:04:40.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:40.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:04:40.946 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:04:40.973 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:40.992 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:04:40.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:40.994 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:04:41.095 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:04:41.128 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:41.185 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:40 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:04:41.185 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:40 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:04:41.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:40 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:04:41.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:40 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:04:41.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:40 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:04:41.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:40 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:04:41.479 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:04:41.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:41.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:04:41.583 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:04:41.609 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:41.630 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:04:41.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:41.633 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:04:41.736 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:04:41.766 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:41.824 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:04:41.824 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:04:41.824 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-03-20T09:04:41.824 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-03-20T09:04:41.824 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:04:41.921 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:04:42.069 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:41 smithi115 ceph-mon[72476]: osdmap e279: 12 total, 11 up, 11 in 2024-03-20T09:04:42.069 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:41 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1647954475' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:04:42.069 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:41 smithi115 ceph-mon[72476]: pgmap v722: 47 pgs: 2 remapped+peering, 7 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 28 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 6.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 2.5 KiB/s rd, 242 KiB/s wr, 54 op/s; 22962/295590 objects degraded (7.768%); 7724/295590 objects misplaced (2.613%); 6.4 MiB/s, 102 objects/s recovering 2024-03-20T09:04:42.069 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:41 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 22973/295704 objects degraded (7.769%), 9 pgs degraded (PG_DEGRADED) 2024-03-20T09:04:42.115 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:04:42.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:42.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:04:42.125 INFO:teuthology.orchestra.run.smithi115.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-20T09:04:42.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:41 smithi122 ceph-mon[73495]: osdmap e279: 12 total, 11 up, 11 in 2024-03-20T09:04:42.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:41 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1647954475' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:04:42.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:41 smithi122 ceph-mon[73495]: pgmap v722: 47 pgs: 2 remapped+peering, 7 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 28 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 6.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 2.5 KiB/s rd, 242 KiB/s wr, 54 op/s; 22962/295590 objects degraded (7.768%); 7724/295590 objects misplaced (2.613%); 6.4 MiB/s, 102 objects/s recovering 2024-03-20T09:04:42.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:41 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 22973/295704 objects degraded (7.769%), 9 pgs degraded (PG_DEGRADED) 2024-03-20T09:04:42.219 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:04:42.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:41 smithi026 ceph-mon[68309]: osdmap e279: 12 total, 11 up, 11 in 2024-03-20T09:04:42.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:41 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1647954475' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:04:42.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:41 smithi026 ceph-mon[68309]: pgmap v722: 47 pgs: 2 remapped+peering, 7 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 28 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 6.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 2.5 KiB/s rd, 242 KiB/s wr, 54 op/s; 22962/295590 objects degraded (7.768%); 7724/295590 objects misplaced (2.613%); 6.4 MiB/s, 102 objects/s recovering 2024-03-20T09:04:42.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:41 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 22973/295704 objects degraded (7.769%), 9 pgs degraded (PG_DEGRADED) 2024-03-20T09:04:42.250 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:42.373 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:04:42.454 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:04:42.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:42.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:04:42.558 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:04:42.587 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:42.788 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:04:42.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:42.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:04:42.893 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:04:42.921 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:42.941 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:04:42.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:42.944 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:04:43.045 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:04:43.072 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:43.094 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:04:43.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:43.097 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:04:43.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:42 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:04:43.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:42 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1647954475' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:04:43.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:42 smithi122 ceph-mon[73495]: osdmap e280: 12 total, 11 up, 11 in 2024-03-20T09:04:43.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:42 smithi122 ceph-mon[73495]: 1.11 scrub starts 2024-03-20T09:04:43.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:42 smithi122 ceph-mon[73495]: 1.11 scrub ok 2024-03-20T09:04:43.198 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:04:43.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:42 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:04:43.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:42 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1647954475' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:04:43.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:42 smithi026 ceph-mon[68309]: osdmap e280: 12 total, 11 up, 11 in 2024-03-20T09:04:43.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:42 smithi026 ceph-mon[68309]: 1.11 scrub starts 2024-03-20T09:04:43.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:42 smithi026 ceph-mon[68309]: 1.11 scrub ok 2024-03-20T09:04:43.228 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:43.309 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:04:43.327 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:04:43.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:43.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:04:43.343 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-20T09:04:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:42 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:04:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:42 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1647954475' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:04:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:42 smithi115 ceph-mon[72476]: osdmap e280: 12 total, 11 up, 11 in 2024-03-20T09:04:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:42 smithi115 ceph-mon[72476]: 1.11 scrub starts 2024-03-20T09:04:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:42 smithi115 ceph-mon[72476]: 1.11 scrub ok 2024-03-20T09:04:43.431 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:04:43.458 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:43.586 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:04:43.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:43.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:04:43.692 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:04:43.725 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:43.819 INFO:teuthology.orchestra.run.smithi115.stdout:3.d 2024-03-20T09:04:43.819 INFO:teuthology.orchestra.run.smithi115.stdout:1.8 2024-03-20T09:04:43.819 INFO:teuthology.orchestra.run.smithi115.stdout:3.16 2024-03-20T09:04:43.819 INFO:teuthology.orchestra.run.smithi115.stdout:1.7 2024-03-20T09:04:43.819 INFO:teuthology.orchestra.run.smithi115.stdout:1.10 2024-03-20T09:04:43.819 INFO:teuthology.orchestra.run.smithi115.stdout:3.10 2024-03-20T09:04:43.819 INFO:teuthology.orchestra.run.smithi115.stdout:3.15 2024-03-20T09:04:43.819 INFO:teuthology.orchestra.run.smithi115.stdout:3.5 2024-03-20T09:04:43.819 INFO:teuthology.orchestra.run.smithi115.stdout:1.b 2024-03-20T09:04:43.819 INFO:teuthology.orchestra.run.smithi115.stdout:1.12 2024-03-20T09:04:43.820 INFO:teuthology.orchestra.run.smithi115.stdout:3.17 2024-03-20T09:04:43.820 INFO:teuthology.orchestra.run.smithi115.stdout:1.9 2024-03-20T09:04:43.820 INFO:teuthology.orchestra.run.smithi115.stdout:1.f 2024-03-20T09:04:43.820 INFO:teuthology.orchestra.run.smithi115.stdout:3.2 2024-03-20T09:04:43.820 INFO:teuthology.orchestra.run.smithi115.stdout:1.c 2024-03-20T09:04:43.820 INFO:teuthology.orchestra.run.smithi115.stdout:3.c 2024-03-20T09:04:43.820 INFO:teuthology.orchestra.run.smithi115.stdout:3.4 2024-03-20T09:04:43.923 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:43 smithi026 ceph-mon[68309]: 1.14 scrub starts 2024-03-20T09:04:43.923 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:43 smithi026 ceph-mon[68309]: 1.14 scrub ok 2024-03-20T09:04:43.923 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:43 smithi026 ceph-mon[68309]: 3.9 scrub starts 2024-03-20T09:04:43.923 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:43 smithi026 ceph-mon[68309]: 3.9 scrub ok 2024-03-20T09:04:43.923 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:43 smithi026 ceph-mon[68309]: pgmap v724: 47 pgs: 2 remapped+peering, 7 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 28 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 6.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 26 op/s; 22962/295590 objects degraded (7.768%); 7724/295590 objects misplaced (2.613%); 6.3 MiB/s, 100 objects/s recovering 2024-03-20T09:04:44.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:43 smithi122 ceph-mon[73495]: 1.14 scrub starts 2024-03-20T09:04:44.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:43 smithi122 ceph-mon[73495]: 1.14 scrub ok 2024-03-20T09:04:44.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:43 smithi122 ceph-mon[73495]: 3.9 scrub starts 2024-03-20T09:04:44.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:43 smithi122 ceph-mon[73495]: 3.9 scrub ok 2024-03-20T09:04:44.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:43 smithi122 ceph-mon[73495]: pgmap v724: 47 pgs: 2 remapped+peering, 7 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 28 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 6.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 26 op/s; 22962/295590 objects degraded (7.768%); 7724/295590 objects misplaced (2.613%); 6.3 MiB/s, 100 objects/s recovering 2024-03-20T09:04:44.354 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:43 smithi115 ceph-mon[72476]: 1.14 scrub starts 2024-03-20T09:04:44.354 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:43 smithi115 ceph-mon[72476]: 1.14 scrub ok 2024-03-20T09:04:44.354 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:43 smithi115 ceph-mon[72476]: 3.9 scrub starts 2024-03-20T09:04:44.354 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:43 smithi115 ceph-mon[72476]: 3.9 scrub ok 2024-03-20T09:04:44.354 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:43 smithi115 ceph-mon[72476]: pgmap v724: 47 pgs: 2 remapped+peering, 7 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 28 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 6.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 26 op/s; 22962/295590 objects degraded (7.768%); 7724/295590 objects misplaced (2.613%); 6.3 MiB/s, 100 objects/s recovering 2024-03-20T09:04:44.600 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:04:44.814 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:04:45.122 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:44 smithi026 ceph-mon[68309]: 3.18 scrub starts 2024-03-20T09:04:45.122 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:44 smithi026 ceph-mon[68309]: osdmap e281: 12 total, 11 up, 11 in 2024-03-20T09:04:45.122 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:44 smithi026 ceph-mon[68309]: 3.18 scrub ok 2024-03-20T09:04:45.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:44 smithi115 ceph-mon[72476]: 3.18 scrub starts 2024-03-20T09:04:45.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:44 smithi115 ceph-mon[72476]: osdmap e281: 12 total, 11 up, 11 in 2024-03-20T09:04:45.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:44 smithi115 ceph-mon[72476]: 3.18 scrub ok 2024-03-20T09:04:45.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:44 smithi122 ceph-mon[73495]: 3.18 scrub starts 2024-03-20T09:04:45.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:44 smithi122 ceph-mon[73495]: osdmap e281: 12 total, 11 up, 11 in 2024-03-20T09:04:45.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:44 smithi122 ceph-mon[73495]: 3.18 scrub ok 2024-03-20T09:04:45.289 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -n osd.5 --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op export --pgid 3.d --file /var/log/ceph/exp.3.d.5 2024-03-20T09:04:45.511 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/osd.5/config 2024-03-20T09:04:45.668 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:04:45.690 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:04:45.710 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:04:45.730 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:04:45.752 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:04:46.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:45 smithi115 ceph-mon[72476]: osdmap e282: 12 total, 11 up, 11 in 2024-03-20T09:04:46.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:45 smithi115 ceph-mon[72476]: pgmap v727: 47 pgs: 2 remapped+peering, 7 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 28 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 6.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 22955/295590 objects degraded (7.766%); 7014/295590 objects misplaced (2.373%); 9.3 MiB/s, 148 objects/s recovering 2024-03-20T09:04:46.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:45 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1027026699' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:04:46.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:45 smithi115 ceph-mon[72476]: 3.6 scrub starts 2024-03-20T09:04:46.187 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:45 smithi026 ceph-mon[68309]: osdmap e282: 12 total, 11 up, 11 in 2024-03-20T09:04:46.187 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:45 smithi026 ceph-mon[68309]: pgmap v727: 47 pgs: 2 remapped+peering, 7 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 28 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 6.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 22955/295590 objects degraded (7.766%); 7014/295590 objects misplaced (2.373%); 9.3 MiB/s, 148 objects/s recovering 2024-03-20T09:04:46.187 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:45 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1027026699' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:04:46.187 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:45 smithi026 ceph-mon[68309]: 3.6 scrub starts 2024-03-20T09:04:46.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:45 smithi122 ceph-mon[73495]: osdmap e282: 12 total, 11 up, 11 in 2024-03-20T09:04:46.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:45 smithi122 ceph-mon[73495]: pgmap v727: 47 pgs: 2 remapped+peering, 7 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 28 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 6.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 22955/295590 objects degraded (7.766%); 7014/295590 objects misplaced (2.373%); 9.3 MiB/s, 148 objects/s recovering 2024-03-20T09:04:46.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:45 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1027026699' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:04:46.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:45 smithi122 ceph-mon[73495]: 3.6 scrub starts 2024-03-20T09:04:46.205 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:04:46.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:46.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:04:46.257 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:04:46.310 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:04:46.337 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:46.356 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:04:46.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:46.359 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:04:46.460 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:04:46.492 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:46.514 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:04:46.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:46.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:04:46.593 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:04:46.615 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:04:46.618 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:04:46.640 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:04:46.649 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:46.662 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:04:46.683 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:04:46.707 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:04:46.707 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:04:46.707 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:04:46.707 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:04:46.707 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:04:46.708 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:04:46.708 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-20T09:04:46.708 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:04:46.708 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:04:46.708 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:04:46.708 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:04:46.708 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:04:46.708 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:04:46.970 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:04:46.971 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:04:47.060 INFO:teuthology.orchestra.run.smithi115.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-20T09:04:47.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:46 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:04:47.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:46 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:04:47.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:46 smithi122 ceph-mon[73495]: osdmap e283: 12 total, 11 up, 11 in 2024-03-20T09:04:47.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:46 smithi122 ceph-mon[73495]: 3.6 scrub ok 2024-03-20T09:04:47.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:46 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 22955/295590 objects degraded (7.766%), 9 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:04:47.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:46 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:04:47.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:46 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:04:47.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:46 smithi026 ceph-mon[68309]: osdmap e283: 12 total, 11 up, 11 in 2024-03-20T09:04:47.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:46 smithi026 ceph-mon[68309]: 3.6 scrub ok 2024-03-20T09:04:47.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:46 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 22955/295590 objects degraded (7.766%), 9 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:04:47.319 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:46 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:04:47.319 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:46 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:04:47.319 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:46 smithi115 ceph-mon[72476]: osdmap e283: 12 total, 11 up, 11 in 2024-03-20T09:04:47.319 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:46 smithi115 ceph-mon[72476]: 3.6 scrub ok 2024-03-20T09:04:47.319 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:46 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 22955/295590 objects degraded (7.766%), 9 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:04:47.776 INFO:teuthology.orchestra.run.smithi115.stderr:Exporting 3.d info 3.d( v 273'6244 (0'0,273'6244] local-lis/les=265/266 n=6132 ec=204/204 lis/c=265/204 les/c/f=266/205/0 sis=265) 2024-03-20T09:04:47.814 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0031471:::benchmark_data_smithi122_119671_object58867:head# 2024-03-20T09:04:47.814 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0033314:::benchmark_data_smithi122_119671_object89042:head# 2024-03-20T09:04:47.815 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b00350bb:::benchmark_data_smithi122_119671_object99725:head# 2024-03-20T09:04:47.815 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b00424b4:::benchmark_data_smithi122_119671_object49302:head# 2024-03-20T09:04:47.816 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0051513:::benchmark_data_smithi122_119671_object51935:head# 2024-03-20T09:04:47.816 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0058005:::benchmark_data_smithi122_119671_object21971:head# 2024-03-20T09:04:47.816 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0062a02:::benchmark_data_smithi122_119671_object55216:head# 2024-03-20T09:04:47.817 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b00670ba:::benchmark_data_smithi122_119671_object23821:head# 2024-03-20T09:04:47.817 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b006bb2c:::benchmark_data_smithi122_119671_object4970:head# 2024-03-20T09:04:47.820 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b006df72:::benchmark_data_smithi122_119671_object38541:head# 2024-03-20T09:04:47.821 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0075821:::benchmark_data_smithi122_119671_object9798:head# 2024-03-20T09:04:47.821 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0087e88:::benchmark_data_smithi122_119671_object35896:head# 2024-03-20T09:04:47.822 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0097cb5:::benchmark_data_smithi122_119671_object28209:head# 2024-03-20T09:04:47.822 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b00a4363:::benchmark_data_smithi122_119671_object14608:head# 2024-03-20T09:04:47.822 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b00a91e6:::benchmark_data_smithi122_119671_object85862:head# 2024-03-20T09:04:47.823 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b00b2e23:::benchmark_data_smithi122_119671_object30901:head# 2024-03-20T09:04:47.823 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b00b4fb7:::benchmark_data_smithi122_119671_object84815:head# 2024-03-20T09:04:47.823 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b00c2228:::benchmark_data_smithi122_119671_object59276:head# 2024-03-20T09:04:47.824 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b00c44b2:::benchmark_data_smithi122_119671_object2674:head# 2024-03-20T09:04:47.824 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b00c90c6:::benchmark_data_smithi122_119671_object54403:head# 2024-03-20T09:04:47.824 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b00d5f32:::benchmark_data_smithi122_119671_object77195:head# 2024-03-20T09:04:47.827 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b00d74a6:::benchmark_data_smithi122_119671_object85488:head# 2024-03-20T09:04:47.827 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b00e59db:::benchmark_data_smithi122_119671_object35545:head# 2024-03-20T09:04:47.828 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b00ef923:::benchmark_data_smithi122_119671_object28263:head# 2024-03-20T09:04:47.831 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b00f7737:::benchmark_data_smithi122_119671_object10228:head# 2024-03-20T09:04:47.831 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b00f8dfc:::benchmark_data_smithi122_119671_object84523:head# 2024-03-20T09:04:47.831 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b00fc565:::benchmark_data_smithi122_119671_object70527:head# 2024-03-20T09:04:47.832 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b01056b3:::benchmark_data_smithi122_119671_object49802:head# 2024-03-20T09:04:47.832 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0113397:::benchmark_data_smithi122_119671_object30866:head# 2024-03-20T09:04:47.833 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b011a669:::benchmark_data_smithi122_119671_object69855:head# 2024-03-20T09:04:47.833 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b012015f:::benchmark_data_smithi122_119671_object68636:head# 2024-03-20T09:04:47.833 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0128f1c:::benchmark_data_smithi122_119671_object86793:head# 2024-03-20T09:04:47.834 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b012bc68:::benchmark_data_smithi122_119671_object50838:head# 2024-03-20T09:04:47.835 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b012dcee:::benchmark_data_smithi122_119671_object64248:head# 2024-03-20T09:04:47.836 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b01489b5:::benchmark_data_smithi122_119671_object37405:head# 2024-03-20T09:04:47.837 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0156536:::benchmark_data_smithi122_119671_object31524:head# 2024-03-20T09:04:47.837 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0159ba6:::benchmark_data_smithi122_119671_object39452:head# 2024-03-20T09:04:47.837 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0174e02:::benchmark_data_smithi122_119671_object3715:head# 2024-03-20T09:04:47.837 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b01839c1:::benchmark_data_smithi122_119671_object9885:head# 2024-03-20T09:04:47.837 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b018cf96:::benchmark_data_smithi122_119671_object68414:head# 2024-03-20T09:04:47.838 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b019b426:::benchmark_data_smithi122_119671_object57153:head# 2024-03-20T09:04:47.838 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b01a9b4d:::benchmark_data_smithi122_119671_object11135:head# 2024-03-20T09:04:47.838 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b01af513:::benchmark_data_smithi122_119671_object83415:head# 2024-03-20T09:04:47.839 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b01bbc79:::benchmark_data_smithi122_119671_object14714:head# 2024-03-20T09:04:47.839 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b01c6931:::benchmark_data_smithi122_119671_object53383:head# 2024-03-20T09:04:47.839 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b01d4e2b:::benchmark_data_smithi122_119671_object76655:head# 2024-03-20T09:04:47.840 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b01df5c7:::benchmark_data_smithi122_119671_object5659:head# 2024-03-20T09:04:47.840 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b01e4e2c:::benchmark_data_smithi122_119671_object49358:head# 2024-03-20T09:04:47.841 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b01f0010:::benchmark_data_smithi122_119671_object51691:head# 2024-03-20T09:04:47.841 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b01f9a18:::benchmark_data_smithi122_119671_object62016:head# 2024-03-20T09:04:47.841 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0202d9c:::benchmark_data_smithi122_119671_object13902:head# 2024-03-20T09:04:47.842 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b021359b:::benchmark_data_smithi122_119671_object55417:head# 2024-03-20T09:04:47.842 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b021afa6:::benchmark_data_smithi122_119671_object94327:head# 2024-03-20T09:04:47.843 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b021fbea:::benchmark_data_smithi122_119671_object66761:head# 2024-03-20T09:04:47.843 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0226064:::benchmark_data_smithi122_119671_object36050:head# 2024-03-20T09:04:47.843 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b022f606:::benchmark_data_smithi122_119671_object88507:head# 2024-03-20T09:04:47.844 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b025599e:::benchmark_data_smithi122_119671_object9541:head# 2024-03-20T09:04:47.844 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b025fe5b:::benchmark_data_smithi122_119671_object83150:head# 2024-03-20T09:04:47.844 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0274cf6:::benchmark_data_smithi122_119671_object48421:head# 2024-03-20T09:04:47.845 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b02844cb:::benchmark_data_smithi122_119671_object41052:head# 2024-03-20T09:04:47.845 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b02a6a47:::benchmark_data_smithi122_119671_object69995:head# 2024-03-20T09:04:47.846 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b02a8951:::benchmark_data_smithi122_119671_object79865:head# 2024-03-20T09:04:47.846 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b02a9c70:::benchmark_data_smithi122_119671_object10523:head# 2024-03-20T09:04:47.847 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b02af41e:::benchmark_data_smithi122_119671_object83315:head# 2024-03-20T09:04:47.847 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b02c8af0:::benchmark_data_smithi122_119671_object69622:head# 2024-03-20T09:04:47.848 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b02ca104:::benchmark_data_smithi122_119671_object40282:head# 2024-03-20T09:04:47.848 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b02cf590:::benchmark_data_smithi122_119671_object87299:head# 2024-03-20T09:04:47.848 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b02d476c:::benchmark_data_smithi122_119671_object38004:head# 2024-03-20T09:04:47.849 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b02dc697:::benchmark_data_smithi122_119671_object62495:head# 2024-03-20T09:04:47.849 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b02e2362:::benchmark_data_smithi122_119671_object86290:head# 2024-03-20T09:04:47.850 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b02f4561:::benchmark_data_smithi122_119671_object82223:head# 2024-03-20T09:04:47.850 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b02ffc1a:::benchmark_data_smithi122_119671_object98092:head# 2024-03-20T09:04:47.850 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b03037a7:::benchmark_data_smithi122_119671_object44634:head# 2024-03-20T09:04:47.851 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0313eb8:::benchmark_data_smithi122_119671_object40708:head# 2024-03-20T09:04:47.851 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b031c65f:::benchmark_data_smithi122_119671_object36907:head# 2024-03-20T09:04:47.852 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0325d9c:::benchmark_data_smithi122_119671_object9777:head# 2024-03-20T09:04:47.852 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b032f23c:::benchmark_data_smithi122_119671_object84716:head# 2024-03-20T09:04:47.853 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b032fc07:::benchmark_data_smithi122_119671_object58159:head# 2024-03-20T09:04:47.853 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0333845:::benchmark_data_smithi122_119671_object49335:head# 2024-03-20T09:04:47.854 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b033455b:::benchmark_data_smithi122_119671_object84945:head# 2024-03-20T09:04:47.854 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b033f5ff:::benchmark_data_smithi122_119671_object29977:head# 2024-03-20T09:04:47.855 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b034bec2:::benchmark_data_smithi122_119671_object26621:head# 2024-03-20T09:04:47.855 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b035486f:::benchmark_data_smithi122_119671_object48612:head# 2024-03-20T09:04:47.856 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0360a7d:::benchmark_data_smithi122_119671_object70438:head# 2024-03-20T09:04:47.856 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b03620eb:::benchmark_data_smithi122_119671_object24903:head# 2024-03-20T09:04:47.857 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0363a3e:::benchmark_data_smithi122_119671_object3849:head# 2024-03-20T09:04:47.857 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b036e4fa:::benchmark_data_smithi122_119671_object42649:head# 2024-03-20T09:04:47.857 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0371f13:::benchmark_data_smithi122_119671_object54710:head# 2024-03-20T09:04:47.858 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0385716:::benchmark_data_smithi122_119671_object87948:head# 2024-03-20T09:04:47.858 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0389f52:::benchmark_data_smithi122_119671_object90186:head# 2024-03-20T09:04:47.859 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b038baa5:::benchmark_data_smithi122_119671_object74580:head# 2024-03-20T09:04:47.859 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b039a049:::benchmark_data_smithi122_119671_object25546:head# 2024-03-20T09:04:47.859 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b039aece:::benchmark_data_smithi122_119671_object46005:head# 2024-03-20T09:04:47.860 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b03a31dd:::benchmark_data_smithi122_119671_object66685:head# 2024-03-20T09:04:47.860 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b03b19d9:::benchmark_data_smithi122_119671_object47270:head# 2024-03-20T09:04:47.861 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b03b563d:::benchmark_data_smithi122_119671_object7362:head# 2024-03-20T09:04:47.861 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b03b97aa:::benchmark_data_smithi122_119671_object15624:head# 2024-03-20T09:04:47.862 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b03c9b8b:::benchmark_data_smithi122_119671_object90627:head# 2024-03-20T09:04:47.862 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b03d9be6:::benchmark_data_smithi122_119671_object96333:head# 2024-03-20T09:04:47.862 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b03e14b0:::benchmark_data_smithi122_119671_object11427:head# 2024-03-20T09:04:47.863 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0405bc0:::benchmark_data_smithi122_119671_object10688:head# 2024-03-20T09:04:47.863 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b040fa09:::benchmark_data_smithi122_119671_object36069:head# 2024-03-20T09:04:47.863 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0422a09:::benchmark_data_smithi122_119671_object71802:head# 2024-03-20T09:04:47.864 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b043ab7a:::benchmark_data_smithi122_119671_object9079:head# 2024-03-20T09:04:47.864 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b043ca99:::benchmark_data_smithi122_119671_object65584:head# 2024-03-20T09:04:47.865 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0451505:::benchmark_data_smithi122_119671_object87515:head# 2024-03-20T09:04:47.865 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0451cb5:::benchmark_data_smithi122_119671_object84713:head# 2024-03-20T09:04:47.865 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0452724:::benchmark_data_smithi122_119671_object34581:head# 2024-03-20T09:04:47.866 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b04636e6:::benchmark_data_smithi122_119671_object92591:head# 2024-03-20T09:04:47.866 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b046ae84:::benchmark_data_smithi122_119671_object5081:head# 2024-03-20T09:04:47.867 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b047147a:::benchmark_data_smithi122_119671_object24333:head# 2024-03-20T09:04:47.867 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b047dd04:::benchmark_data_smithi122_119671_object51653:head# 2024-03-20T09:04:47.868 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0487075:::benchmark_data_smithi122_119671_object75736:head# 2024-03-20T09:04:47.868 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0489aaa:::benchmark_data_smithi122_119671_object72095:head# 2024-03-20T09:04:47.869 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b048fec6:::benchmark_data_smithi122_119671_object16721:head# 2024-03-20T09:04:47.869 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0496317:::benchmark_data_smithi122_119671_object27701:head# 2024-03-20T09:04:47.869 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0497297:::benchmark_data_smithi122_119671_object90113:head# 2024-03-20T09:04:47.870 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0499edc:::benchmark_data_smithi122_119671_object58885:head# 2024-03-20T09:04:47.870 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b04bee2e:::benchmark_data_smithi122_119671_object86376:head# 2024-03-20T09:04:47.870 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b04cf6a3:::benchmark_data_smithi122_119671_object81168:head# 2024-03-20T09:04:47.871 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b04dd9be:::benchmark_data_smithi122_119671_object30980:head# 2024-03-20T09:04:47.871 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b04e364c:::benchmark_data_smithi122_119671_object70388:head# 2024-03-20T09:04:47.871 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b04e929c:::benchmark_data_smithi122_119671_object78730:head# 2024-03-20T09:04:47.872 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b04f7078:::benchmark_data_smithi122_119671_object54988:head# 2024-03-20T09:04:47.872 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b05006e4:::benchmark_data_smithi122_119671_object34791:head# 2024-03-20T09:04:47.873 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b050fff1:::benchmark_data_smithi122_119671_object78826:head# 2024-03-20T09:04:47.873 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0510552:::benchmark_data_smithi122_119671_object78776:head# 2024-03-20T09:04:47.873 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b052278f:::benchmark_data_smithi122_119671_object76364:head# 2024-03-20T09:04:47.874 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0522d63:::benchmark_data_smithi122_119671_object96352:head# 2024-03-20T09:04:47.874 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b052943c:::benchmark_data_smithi122_119671_object20315:head# 2024-03-20T09:04:47.874 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0537ed9:::benchmark_data_smithi122_119671_object91918:head# 2024-03-20T09:04:47.875 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0542746:::benchmark_data_smithi122_119671_object80657:head# 2024-03-20T09:04:47.875 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b055046b:::benchmark_data_smithi122_119671_object84043:head# 2024-03-20T09:04:47.876 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b05510c5:::benchmark_data_smithi122_119671_object27982:head# 2024-03-20T09:04:47.876 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0554bfb:::benchmark_data_smithi122_119671_object30501:head# 2024-03-20T09:04:47.877 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b056295b:::benchmark_data_smithi122_119671_object5881:head# 2024-03-20T09:04:47.877 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0576026:::benchmark_data_smithi122_119671_object57052:head# 2024-03-20T09:04:47.877 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0584395:::benchmark_data_smithi122_119671_object87836:head# 2024-03-20T09:04:47.878 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b05a7e36:::benchmark_data_smithi122_119671_object63637:head# 2024-03-20T09:04:47.878 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b05aacf2:::benchmark_data_smithi122_119671_object7859:head# 2024-03-20T09:04:47.879 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b05b4219:::benchmark_data_smithi122_119671_object7902:head# 2024-03-20T09:04:47.879 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b05b6897:::benchmark_data_smithi122_119671_object66539:head# 2024-03-20T09:04:47.879 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b05cd655:::benchmark_data_smithi122_119671_object96258:head# 2024-03-20T09:04:47.880 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b05d2b1c:::benchmark_data_smithi122_119671_object69714:head# 2024-03-20T09:04:47.880 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b05dfa17:::benchmark_data_smithi122_119671_object6391:head# 2024-03-20T09:04:47.881 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b05eafcb:::benchmark_data_smithi122_119671_object85560:head# 2024-03-20T09:04:47.881 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b05fa81f:::benchmark_data_smithi122_119671_object55984:head# 2024-03-20T09:04:47.882 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b060edcf:::benchmark_data_smithi122_119671_object38452:head# 2024-03-20T09:04:47.882 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0621538:::benchmark_data_smithi122_119671_object1968:head# 2024-03-20T09:04:47.882 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b06586dd:::benchmark_data_smithi122_119671_object30565:head# 2024-03-20T09:04:47.883 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b065fc9c:::benchmark_data_smithi122_119671_object29185:head# 2024-03-20T09:04:47.883 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0681ad1:::benchmark_data_smithi122_119671_object75132:head# 2024-03-20T09:04:47.883 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0687022:::benchmark_data_smithi122_119671_object49731:head# 2024-03-20T09:04:47.884 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b068e20d:::benchmark_data_smithi122_119671_object30444:head# 2024-03-20T09:04:47.884 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0691003:::benchmark_data_smithi122_119671_object4744:head# 2024-03-20T09:04:47.885 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b06a54e5:::benchmark_data_smithi122_119671_object82334:head# 2024-03-20T09:04:47.885 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b06aeb5a:::benchmark_data_smithi122_119671_object87650:head# 2024-03-20T09:04:47.886 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b06bdb38:::benchmark_data_smithi122_119671_object64347:head# 2024-03-20T09:04:47.886 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b06c5af6:::benchmark_data_smithi122_119671_object10664:head# 2024-03-20T09:04:47.887 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b06d6033:::benchmark_data_smithi122_119671_object12750:head# 2024-03-20T09:04:47.887 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b06e2d2e:::benchmark_data_smithi122_119671_object64587:head# 2024-03-20T09:04:47.888 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b06ed7f1:::benchmark_data_smithi122_119671_object78845:head# 2024-03-20T09:04:47.888 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b06ffe42:::benchmark_data_smithi122_119671_object45956:head# 2024-03-20T09:04:47.888 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b070296b:::benchmark_data_smithi122_119671_object57282:head# 2024-03-20T09:04:47.889 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b070608f:::benchmark_data_smithi122_119671_object20268:head# 2024-03-20T09:04:47.889 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b070ce9f:::benchmark_data_smithi122_119671_object43292:head# 2024-03-20T09:04:47.889 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0717535:::benchmark_data_smithi122_119671_object51066:head# 2024-03-20T09:04:47.890 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0724009:::benchmark_data_smithi122_119671_object34637:head# 2024-03-20T09:04:47.890 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b07326cf:::benchmark_data_smithi122_119671_object5768:head# 2024-03-20T09:04:47.891 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0740939:::benchmark_data_smithi122_119671_object43997:head# 2024-03-20T09:04:47.891 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b074f43f:::benchmark_data_smithi122_119671_object92165:head# 2024-03-20T09:04:47.891 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b07597e1:::benchmark_data_smithi122_119671_object12920:head# 2024-03-20T09:04:47.892 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b075acbc:::benchmark_data_smithi122_119671_object73890:head# 2024-03-20T09:04:47.892 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0774a20:::benchmark_data_smithi122_119671_object51787:head# 2024-03-20T09:04:47.893 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0780112:::benchmark_data_smithi122_119671_object7868:head# 2024-03-20T09:04:47.893 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b07831d1:::benchmark_data_smithi122_119671_object22394:head# 2024-03-20T09:04:47.894 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b07955fd:::benchmark_data_smithi122_119671_object2819:head# 2024-03-20T09:04:47.897 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b07a13b1:::benchmark_data_smithi122_119671_object25568:head# 2024-03-20T09:04:47.898 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b07a87b2:::benchmark_data_smithi122_119671_object56725:head# 2024-03-20T09:04:47.898 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b07ab961:::benchmark_data_smithi122_119671_object22298:head# 2024-03-20T09:04:47.898 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b07c8987:::benchmark_data_smithi122_119671_object78868:head# 2024-03-20T09:04:47.899 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b07cfb6d:::benchmark_data_smithi122_119671_object29179:head# 2024-03-20T09:04:47.900 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b07e98fa:::benchmark_data_smithi122_119671_object22289:head# 2024-03-20T09:04:47.900 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b07f3298:::benchmark_data_smithi122_119671_object58680:head# 2024-03-20T09:04:47.900 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b07fa2f3:::benchmark_data_smithi122_119671_object92125:head# 2024-03-20T09:04:47.901 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b07fd3a8:::benchmark_data_smithi122_119671_object25248:head# 2024-03-20T09:04:47.902 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0807059:::benchmark_data_smithi122_119671_object66325:head# 2024-03-20T09:04:47.902 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b081bf04:::benchmark_data_smithi122_119671_object24877:head# 2024-03-20T09:04:47.902 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b082d912:::benchmark_data_smithi122_119671_object49496:head# 2024-03-20T09:04:47.903 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0836c13:::benchmark_data_smithi122_119671_object76194:head# 2024-03-20T09:04:47.903 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b083766b:::benchmark_data_smithi122_119671_object52386:head# 2024-03-20T09:04:47.903 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b083af58:::benchmark_data_smithi122_119671_object51057:head# 2024-03-20T09:04:47.904 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b083e2ae:::benchmark_data_smithi122_119671_object47903:head# 2024-03-20T09:04:47.904 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0845671:::benchmark_data_smithi122_119671_object19363:head# 2024-03-20T09:04:47.905 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b084b221:::benchmark_data_smithi122_119671_object68735:head# 2024-03-20T09:04:47.905 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b08574f3:::benchmark_data_smithi122_119671_object74086:head# 2024-03-20T09:04:47.906 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b085b033:::benchmark_data_smithi122_119671_object78546:head# 2024-03-20T09:04:47.910 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0866967:::benchmark_data_smithi122_119671_object24521:head# 2024-03-20T09:04:47.910 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b08814e0:::benchmark_data_smithi122_119671_object73084:head# 2024-03-20T09:04:47.910 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0885e58:::benchmark_data_smithi122_119671_object47522:head# 2024-03-20T09:04:47.910 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b088718a:::benchmark_data_smithi122_119671_object32164:head# 2024-03-20T09:04:47.910 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0888500:::benchmark_data_smithi122_119671_object12329:head# 2024-03-20T09:04:47.910 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0888a20:::benchmark_data_smithi122_119671_object65600:head# 2024-03-20T09:04:47.910 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b08973c5:::benchmark_data_smithi122_119671_object49911:head# 2024-03-20T09:04:47.911 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b089e7a5:::benchmark_data_smithi122_119671_object11189:head# 2024-03-20T09:04:47.911 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b08a6077:::benchmark_data_smithi122_119671_object85292:head# 2024-03-20T09:04:47.911 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b08a6e52:::benchmark_data_smithi122_119671_object99350:head# 2024-03-20T09:04:47.911 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b08aa0e9:::benchmark_data_smithi122_119671_object43343:head# 2024-03-20T09:04:47.912 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b08adff7:::benchmark_data_smithi122_119671_object39141:head# 2024-03-20T09:04:47.913 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b08aedb4:::benchmark_data_smithi122_119671_object38236:head# 2024-03-20T09:04:47.913 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b08b18d2:::benchmark_data_smithi122_119671_object23034:head# 2024-03-20T09:04:47.913 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b08b5e8d:::benchmark_data_smithi122_119671_object52138:head# 2024-03-20T09:04:47.914 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b08c35cb:::benchmark_data_smithi122_119671_object70468:head# 2024-03-20T09:04:47.914 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b08c7b97:::benchmark_data_smithi122_119671_object88535:head# 2024-03-20T09:04:47.915 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b08e9aac:::benchmark_data_smithi122_119671_object44620:head# 2024-03-20T09:04:47.915 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b08ea8e9:::benchmark_data_smithi122_119671_object89499:head# 2024-03-20T09:04:47.915 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b08f80a3:::benchmark_data_smithi122_119671_object38416:head# 2024-03-20T09:04:47.916 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b08f93c0:::benchmark_data_smithi122_119671_object53738:head# 2024-03-20T09:04:47.916 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b09043dd:::benchmark_data_smithi122_119671_object69649:head# 2024-03-20T09:04:47.917 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b09057ab:::benchmark_data_smithi122_119671_object22895:head# 2024-03-20T09:04:47.917 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0914683:::benchmark_data_smithi122_119671_object4627:head# 2024-03-20T09:04:47.917 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0918973:::benchmark_data_smithi122_119671_object46641:head# 2024-03-20T09:04:47.918 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b09219ae:::benchmark_data_smithi122_119671_object16852:head# 2024-03-20T09:04:47.918 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0926805:::benchmark_data_smithi122_119671_object20227:head# 2024-03-20T09:04:47.919 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b092e190:::benchmark_data_smithi122_119671_object27866:head# 2024-03-20T09:04:47.919 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b093466f:::benchmark_data_smithi122_119671_object65155:head# 2024-03-20T09:04:47.920 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0942f0b:::benchmark_data_smithi122_119671_object32943:head# 2024-03-20T09:04:47.920 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b09451b5:::benchmark_data_smithi122_119671_object41406:head# 2024-03-20T09:04:47.920 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b095d025:::benchmark_data_smithi122_119671_object44500:head# 2024-03-20T09:04:47.921 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0966aa8:::benchmark_data_smithi122_119671_object85132:head# 2024-03-20T09:04:47.921 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b09766d6:::benchmark_data_smithi122_119671_object80445:head# 2024-03-20T09:04:47.921 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b098cd44:::benchmark_data_smithi122_119671_object47761:head# 2024-03-20T09:04:47.922 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b098e8db:::benchmark_data_smithi122_119671_object12024:head# 2024-03-20T09:04:47.922 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b09a4781:::benchmark_data_smithi122_119671_object77463:head# 2024-03-20T09:04:47.923 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b09b8cec:::benchmark_data_smithi122_119671_object47863:head# 2024-03-20T09:04:47.923 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b09bb3a0:::benchmark_data_smithi122_119671_object91218:head# 2024-03-20T09:04:47.925 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b09cfc28:::benchmark_data_smithi122_119671_object13685:head# 2024-03-20T09:04:47.925 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b09e5990:::benchmark_data_smithi122_119671_object26333:head# 2024-03-20T09:04:47.926 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b09e91f0:::benchmark_data_smithi122_119671_object45590:head# 2024-03-20T09:04:47.926 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0a092c5:::benchmark_data_smithi122_119671_object43186:head# 2024-03-20T09:04:47.927 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0a103c2:::benchmark_data_smithi122_119671_object75321:head# 2024-03-20T09:04:47.928 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0a10e61:::benchmark_data_smithi122_119671_object97876:head# 2024-03-20T09:04:47.928 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0a149f4:::benchmark_data_smithi122_119671_object71474:head# 2024-03-20T09:04:47.929 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0a309bf:::benchmark_data_smithi122_119671_object80688:head# 2024-03-20T09:04:47.929 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0a32701:::benchmark_data_smithi122_119671_object73521:head# 2024-03-20T09:04:47.929 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0a33c03:::benchmark_data_smithi122_119671_object22311:head# 2024-03-20T09:04:47.930 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0a3eb81:::benchmark_data_smithi122_119671_object84084:head# 2024-03-20T09:04:47.930 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0a45e2e:::benchmark_data_smithi122_119671_object23112:head# 2024-03-20T09:04:47.931 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0a4c7d7:::benchmark_data_smithi122_119671_object83561:head# 2024-03-20T09:04:47.931 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0a5596c:::benchmark_data_smithi122_119671_object46042:head# 2024-03-20T09:04:47.932 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0a6037d:::benchmark_data_smithi122_119671_object11073:head# 2024-03-20T09:04:47.935 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0a60a8d:::benchmark_data_smithi122_119671_object56289:head# 2024-03-20T09:04:47.935 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0a67daa:::benchmark_data_smithi122_119671_object34371:head# 2024-03-20T09:04:47.936 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0a74e71:::benchmark_data_smithi122_119671_object96110:head# 2024-03-20T09:04:47.936 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0a881e0:::benchmark_data_smithi122_119671_object6008:head# 2024-03-20T09:04:47.937 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0a8d531:::benchmark_data_smithi122_119671_object88304:head# 2024-03-20T09:04:47.937 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0a97630:::benchmark_data_smithi122_119671_object52585:head# 2024-03-20T09:04:47.939 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0aaafef:::benchmark_data_smithi122_119671_object87310:head# 2024-03-20T09:04:47.939 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0ab8452:::benchmark_data_smithi122_119671_object73993:head# 2024-03-20T09:04:47.940 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0acd803:::benchmark_data_smithi122_119671_object73139:head# 2024-03-20T09:04:47.940 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0ad3537:::benchmark_data_smithi122_119671_object98712:head# 2024-03-20T09:04:47.941 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0ad524f:::benchmark_data_smithi122_119671_object58379:head# 2024-03-20T09:04:47.941 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0ae5622:::benchmark_data_smithi122_119671_object71829:head# 2024-03-20T09:04:47.942 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0ae5b08:::benchmark_data_smithi122_119671_object54832:head# 2024-03-20T09:04:47.942 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0aec6fa:::benchmark_data_smithi122_119671_object66541:head# 2024-03-20T09:04:47.943 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0b1e965:::benchmark_data_smithi122_119671_object83991:head# 2024-03-20T09:04:47.943 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0b27289:::benchmark_data_smithi122_119671_object85951:head# 2024-03-20T09:04:47.943 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0b27ed3:::benchmark_data_smithi122_119671_object57212:head# 2024-03-20T09:04:47.944 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0b29f3d:::benchmark_data_smithi122_119671_object73011:head# 2024-03-20T09:04:47.944 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0b36165:::benchmark_data_smithi122_119671_object83383:head# 2024-03-20T09:04:47.945 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0b38d02:::benchmark_data_smithi122_119671_object92751:head# 2024-03-20T09:04:47.945 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0b4d589:::benchmark_data_smithi122_119671_object87502:head# 2024-03-20T09:04:47.945 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0b63f50:::benchmark_data_smithi122_119671_object27297:head# 2024-03-20T09:04:47.946 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0b69cb7:::benchmark_data_smithi122_119671_object64466:head# 2024-03-20T09:04:47.946 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0b6f2f7:::benchmark_data_smithi122_119671_object34037:head# 2024-03-20T09:04:47.947 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0b73b40:::benchmark_data_smithi122_119671_object51573:head# 2024-03-20T09:04:47.947 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0b7d50d:::benchmark_data_smithi122_119671_object16518:head# 2024-03-20T09:04:47.948 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0b80f81:::benchmark_data_smithi122_119671_object71768:head# 2024-03-20T09:04:47.948 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0b83705:::benchmark_data_smithi122_119671_object53901:head# 2024-03-20T09:04:47.948 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0b8c27d:::benchmark_data_smithi122_119671_object46493:head# 2024-03-20T09:04:47.949 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0b8c5f6:::benchmark_data_smithi122_119671_object89995:head# 2024-03-20T09:04:47.949 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0bac264:::benchmark_data_smithi122_119671_object14421:head# 2024-03-20T09:04:47.950 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0bc2a2b:::benchmark_data_smithi122_119671_object55880:head# 2024-03-20T09:04:47.950 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0bc5e72:::benchmark_data_smithi122_119671_object35449:head# 2024-03-20T09:04:47.950 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0bc8bb1:::benchmark_data_smithi122_119671_object54861:head# 2024-03-20T09:04:47.951 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0bcb7cf:::benchmark_data_smithi122_119671_object10542:head# 2024-03-20T09:04:47.951 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0bd2cb2:::benchmark_data_smithi122_119671_object80663:head# 2024-03-20T09:04:47.952 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0bd651b:::benchmark_data_smithi122_119671_object27977:head# 2024-03-20T09:04:47.952 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0bd7c20:::benchmark_data_smithi122_119671_object56868:head# 2024-03-20T09:04:47.953 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0bd910a:::benchmark_data_smithi122_119671_object90735:head# 2024-03-20T09:04:47.953 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0beb7b7:::benchmark_data_smithi122_119671_object20805:head# 2024-03-20T09:04:47.954 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0bec12e:::benchmark_data_smithi122_119671_object59863:head# 2024-03-20T09:04:47.954 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0beefd3:::benchmark_data_smithi122_119671_object9054:head# 2024-03-20T09:04:47.954 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0bf072b:::benchmark_data_smithi122_119671_object62313:head# 2024-03-20T09:04:47.955 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0bf3e42:::benchmark_data_smithi122_119671_object83171:head# 2024-03-20T09:04:47.955 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0bf45e3:::benchmark_data_smithi122_119671_object97414:head# 2024-03-20T09:04:47.956 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0c040d2:::benchmark_data_smithi122_119671_object41746:head# 2024-03-20T09:04:47.956 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0c1c996:::benchmark_data_smithi122_119671_object66971:head# 2024-03-20T09:04:47.957 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0c207bb:::benchmark_data_smithi122_119671_object65552:head# 2024-03-20T09:04:47.958 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0c4b6f3:::benchmark_data_smithi122_119671_object27232:head# 2024-03-20T09:04:47.958 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0c517f4:::benchmark_data_smithi122_119671_object83910:head# 2024-03-20T09:04:47.959 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0c678ec:::benchmark_data_smithi122_119671_object29728:head# 2024-03-20T09:04:47.959 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0c73bbb:::benchmark_data_smithi122_119671_object41704:head# 2024-03-20T09:04:47.960 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0c78f36:::benchmark_data_smithi122_119671_object5920:head# 2024-03-20T09:04:47.960 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0c9b74a:::benchmark_data_smithi122_119671_object31542:head# 2024-03-20T09:04:47.961 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0ca3ca8:::benchmark_data_smithi122_119671_object27073:head# 2024-03-20T09:04:47.961 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0cb2a12:::benchmark_data_smithi122_119671_object31440:head# 2024-03-20T09:04:47.962 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0cb5147:::benchmark_data_smithi122_119671_object10325:head# 2024-03-20T09:04:47.962 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0cc4be5:::benchmark_data_smithi122_119671_object35758:head# 2024-03-20T09:04:47.962 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0cc4e50:::benchmark_data_smithi122_119671_object1895:head# 2024-03-20T09:04:47.963 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0ccc620:::benchmark_data_smithi122_119671_object16950:head# 2024-03-20T09:04:47.963 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0cd103e:::benchmark_data_smithi122_119671_object95557:head# 2024-03-20T09:04:47.964 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0ce3538:::benchmark_data_smithi122_119671_object14552:head# 2024-03-20T09:04:47.964 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0ce3755:::benchmark_data_smithi122_119671_object17936:head# 2024-03-20T09:04:47.965 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0cef7e0:::benchmark_data_smithi122_119671_object76417:head# 2024-03-20T09:04:47.965 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0cefca3:::benchmark_data_smithi122_119671_object55143:head# 2024-03-20T09:04:47.966 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0cf6f4f:::benchmark_data_smithi122_119671_object4474:head# 2024-03-20T09:04:47.966 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0cfdbfa:::benchmark_data_smithi122_119671_object98749:head# 2024-03-20T09:04:47.966 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0d10d70:::benchmark_data_smithi122_119671_object72485:head# 2024-03-20T09:04:47.967 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0d1ad2d:::benchmark_data_smithi122_119671_object55352:head# 2024-03-20T09:04:47.968 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0d1e5b0:::benchmark_data_smithi122_119671_object12355:head# 2024-03-20T09:04:47.968 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0d310c4:::benchmark_data_smithi122_119671_object7225:head# 2024-03-20T09:04:47.968 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0d3c4d1:::benchmark_data_smithi122_119671_object28577:head# 2024-03-20T09:04:47.969 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0d4670c:::benchmark_data_smithi122_119671_object33710:head# 2024-03-20T09:04:47.969 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0d48cd2:::benchmark_data_smithi122_119671_object46950:head# 2024-03-20T09:04:47.970 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0d659a8:::benchmark_data_smithi122_119671_object89269:head# 2024-03-20T09:04:47.970 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0d758fc:::benchmark_data_smithi122_119671_object72701:head# 2024-03-20T09:04:47.970 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0d879f1:::benchmark_data_smithi122_119671_object85853:head# 2024-03-20T09:04:47.971 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0d98009:::benchmark_data_smithi122_119671_object46216:head# 2024-03-20T09:04:47.971 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0da7947:::benchmark_data_smithi122_119671_object58613:head# 2024-03-20T09:04:47.972 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0daad32:::benchmark_data_smithi122_119671_object63455:head# 2024-03-20T09:04:47.972 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0dae60f:::benchmark_data_smithi122_119671_object37898:head# 2024-03-20T09:04:47.973 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0db3512:::benchmark_data_smithi122_119671_object12853:head# 2024-03-20T09:04:47.973 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0dc8398:::benchmark_data_smithi122_119671_object64119:head# 2024-03-20T09:04:47.973 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0dcce8e:::benchmark_data_smithi122_119671_object16593:head# 2024-03-20T09:04:47.974 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0de3c71:::benchmark_data_smithi122_119671_object79982:head# 2024-03-20T09:04:47.974 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0de8980:::benchmark_data_smithi122_119671_object83311:head# 2024-03-20T09:04:47.975 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0debc98:::benchmark_data_smithi122_119671_object30802:head# 2024-03-20T09:04:47.976 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0dee826:::benchmark_data_smithi122_119671_object52295:head# 2024-03-20T09:04:47.976 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0defe8f:::benchmark_data_smithi122_119671_object83152:head# 2024-03-20T09:04:47.976 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0df3053:::benchmark_data_smithi122_119671_object31948:head# 2024-03-20T09:04:47.977 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0df8cc7:::benchmark_data_smithi122_119671_object98340:head# 2024-03-20T09:04:47.977 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0dfc4db:::benchmark_data_smithi122_119671_object97251:head# 2024-03-20T09:04:47.978 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0e04298:::benchmark_data_smithi122_119671_object30902:head# 2024-03-20T09:04:47.978 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0e14699:::benchmark_data_smithi122_119671_object39195:head# 2024-03-20T09:04:47.979 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0e1b50b:::benchmark_data_smithi122_119671_object64221:head# 2024-03-20T09:04:47.981 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0e224b1:::benchmark_data_smithi122_119671_object22957:head# 2024-03-20T09:04:47.981 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0e32ce2:::benchmark_data_smithi122_119671_object22052:head# 2024-03-20T09:04:47.982 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0e5a551:::benchmark_data_smithi122_119671_object98498:head# 2024-03-20T09:04:47.982 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0e5b140:::benchmark_data_smithi122_119671_object86054:head# 2024-03-20T09:04:47.983 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0e6daf3:::benchmark_data_smithi122_119671_object20042:head# 2024-03-20T09:04:47.983 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0e6f333:::benchmark_data_smithi122_119671_object39255:head# 2024-03-20T09:04:47.983 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0e7740b:::benchmark_data_smithi122_119671_object93155:head# 2024-03-20T09:04:47.984 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0e79bf3:::benchmark_data_smithi122_119671_object97237:head# 2024-03-20T09:04:47.984 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0e7e245:::benchmark_data_smithi122_119671_object21917:head# 2024-03-20T09:04:47.985 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0e88d72:::benchmark_data_smithi122_119671_object49945:head# 2024-03-20T09:04:47.985 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0e8b039:::benchmark_data_smithi122_119671_object54366:head# 2024-03-20T09:04:47.986 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0e8db0b:::benchmark_data_smithi122_119671_object3356:head# 2024-03-20T09:04:47.986 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0ea4301:::benchmark_data_smithi122_119671_object51008:head# 2024-03-20T09:04:47.986 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0ea9a5f:::benchmark_data_smithi122_119671_object75090:head# 2024-03-20T09:04:47.987 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0eb1df1:::benchmark_data_smithi122_119671_object87207:head# 2024-03-20T09:04:47.987 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0eb2d7e:::benchmark_data_smithi122_119671_object85550:head# 2024-03-20T09:04:47.988 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0eb4cd5:::benchmark_data_smithi122_119671_object85182:head# 2024-03-20T09:04:47.988 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0eb6b77:::benchmark_data_smithi122_119671_object54957:head# 2024-03-20T09:04:47.988 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0eb85fa:::benchmark_data_smithi122_119671_object22306:head# 2024-03-20T09:04:47.989 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0ebe43c:::benchmark_data_smithi122_119671_object40521:head# 2024-03-20T09:04:47.989 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0ebea0d:::benchmark_data_smithi122_119671_object67828:head# 2024-03-20T09:04:47.990 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0ed457a:::benchmark_data_smithi122_119671_object66240:head# 2024-03-20T09:04:47.990 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0edbcb7:::benchmark_data_smithi122_119671_object1057:head# 2024-03-20T09:04:47.991 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0ef9419:::benchmark_data_smithi122_119671_object75982:head# 2024-03-20T09:04:47.991 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0f04452:::benchmark_data_smithi122_119671_object20461:head# 2024-03-20T09:04:47.992 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0f17efb:::benchmark_data_smithi122_119671_object52479:head# 2024-03-20T09:04:47.992 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0f25a15:::benchmark_data_smithi122_119671_object46520:head# 2024-03-20T09:04:47.992 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0f2bae1:::benchmark_data_smithi122_119671_object11750:head# 2024-03-20T09:04:47.993 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0f3c86b:::benchmark_data_smithi122_119671_object63456:head# 2024-03-20T09:04:47.993 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0f438f5:::benchmark_data_smithi122_119671_object57655:head# 2024-03-20T09:04:47.994 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0f4c8ee:::benchmark_data_smithi122_119671_object64098:head# 2024-03-20T09:04:47.994 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0f656c5:::benchmark_data_smithi122_119671_object78835:head# 2024-03-20T09:04:47.995 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0f6ba7c:::benchmark_data_smithi122_119671_object27631:head# 2024-03-20T09:04:47.995 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0f72b15:::benchmark_data_smithi122_119671_object40332:head# 2024-03-20T09:04:47.995 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0f7dc29:::benchmark_data_smithi122_119671_object58392:head# 2024-03-20T09:04:47.996 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0f801e4:::benchmark_data_smithi122_119671_object74442:head# 2024-03-20T09:04:47.996 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0fa04b4:::benchmark_data_smithi122_119671_object44380:head# 2024-03-20T09:04:47.997 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0fa549e:::benchmark_data_smithi122_119671_object5017:head# 2024-03-20T09:04:47.997 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0fb8a21:::benchmark_data_smithi122_119671_object78194:head# 2024-03-20T09:04:47.998 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0fbb76d:::benchmark_data_smithi122_119671_object84513:head# 2024-03-20T09:04:47.998 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0fc5108:::benchmark_data_smithi122_119671_object3968:head# 2024-03-20T09:04:47.998 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0fe6f18:::benchmark_data_smithi122_119671_object61033:head# 2024-03-20T09:04:47.999 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0fed03c:::benchmark_data_smithi122_119671_object99413:head# 2024-03-20T09:04:47.999 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0fed1f0:::benchmark_data_smithi122_119671_object18752:head# 2024-03-20T09:04:48.000 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b0ff5c67:::benchmark_data_smithi122_119671_object59220:head# 2024-03-20T09:04:48.000 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1006b7e:::benchmark_data_smithi122_119671_object15358:head# 2024-03-20T09:04:48.000 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b10164a7:::benchmark_data_smithi122_119671_object20654:head# 2024-03-20T09:04:48.001 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1019a38:::benchmark_data_smithi122_119671_object26078:head# 2024-03-20T09:04:48.001 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1031e17:::benchmark_data_smithi122_119671_object74674:head# 2024-03-20T09:04:48.002 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1039c0e:::benchmark_data_smithi122_119671_object36872:head# 2024-03-20T09:04:48.002 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b10486ae:::benchmark_data_smithi122_119671_object47803:head# 2024-03-20T09:04:48.002 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1054417:::benchmark_data_smithi122_119671_object54064:head# 2024-03-20T09:04:48.003 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b107b2bb:::benchmark_data_smithi122_119671_object9518:head# 2024-03-20T09:04:48.003 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b108c726:::benchmark_data_smithi122_119671_object82251:head# 2024-03-20T09:04:48.004 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b10aaae9:::benchmark_data_smithi122_119671_object4649:head# 2024-03-20T09:04:48.004 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b10b3dd3:::benchmark_data_smithi122_119671_object76363:head# 2024-03-20T09:04:48.004 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b10b6d31:::benchmark_data_smithi122_119671_object21104:head# 2024-03-20T09:04:48.005 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b10c700d:::benchmark_data_smithi122_119671_object1985:head# 2024-03-20T09:04:48.005 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b10d924b:::benchmark_data_smithi122_119671_object22362:head# 2024-03-20T09:04:48.006 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b10de806:::benchmark_data_smithi122_119671_object77173:head# 2024-03-20T09:04:48.006 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b10e5b96:::benchmark_data_smithi122_119671_object4829:head# 2024-03-20T09:04:48.007 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b10f021c:::benchmark_data_smithi122_119671_object27719:head# 2024-03-20T09:04:48.007 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b10fcbe3:::benchmark_data_smithi122_119671_object39273:head# 2024-03-20T09:04:48.008 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b10fd1a8:::benchmark_data_smithi122_119671_object8384:head# 2024-03-20T09:04:48.008 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1113bf4:::benchmark_data_smithi122_119671_object22446:head# 2024-03-20T09:04:48.009 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1129c52:::benchmark_data_smithi122_119671_object91664:head# 2024-03-20T09:04:48.009 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b112f41d:::benchmark_data_smithi122_119671_object97739:head# 2024-03-20T09:04:48.009 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b11358a6:::benchmark_data_smithi122_119671_object81154:head# 2024-03-20T09:04:48.010 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b11428e0:::benchmark_data_smithi122_119671_object81199:head# 2024-03-20T09:04:48.010 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1155de4:::benchmark_data_smithi122_119671_object91814:head# 2024-03-20T09:04:48.011 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1159e09:::benchmark_data_smithi122_119671_object79643:head# 2024-03-20T09:04:48.011 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b116cb88:::benchmark_data_smithi122_119671_object87817:head# 2024-03-20T09:04:48.012 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b117a0ef:::benchmark_data_smithi122_119671_object10540:head# 2024-03-20T09:04:48.012 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b119521e:::benchmark_data_smithi122_119671_object15992:head# 2024-03-20T09:04:48.012 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b11964bb:::benchmark_data_smithi122_119671_object8296:head# 2024-03-20T09:04:48.013 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b119a299:::benchmark_data_smithi122_119671_object50200:head# 2024-03-20T09:04:48.013 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b11a09f3:::benchmark_data_smithi122_119671_object43188:head# 2024-03-20T09:04:48.014 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b11a3545:::benchmark_data_smithi122_119671_object99768:head# 2024-03-20T09:04:48.014 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b11a4e63:::benchmark_data_smithi122_119671_object3454:head# 2024-03-20T09:04:48.015 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b11ab501:::benchmark_data_smithi122_119671_object44226:head# 2024-03-20T09:04:48.015 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b11adacb:::benchmark_data_smithi122_119671_object29775:head# 2024-03-20T09:04:48.015 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b11b8093:::benchmark_data_smithi122_119671_object63182:head# 2024-03-20T09:04:48.016 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b11c0c57:::benchmark_data_smithi122_119671_object8818:head# 2024-03-20T09:04:48.016 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b11cdc1f:::benchmark_data_smithi122_119671_object29214:head# 2024-03-20T09:04:48.017 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b11d0baf:::benchmark_data_smithi122_119671_object3269:head# 2024-03-20T09:04:48.017 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b11dbf0c:::benchmark_data_smithi122_119671_object5103:head# 2024-03-20T09:04:48.017 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b11dd973:::benchmark_data_smithi122_119671_object82917:head# 2024-03-20T09:04:48.018 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b11eb72d:::benchmark_data_smithi122_119671_object42308:head# 2024-03-20T09:04:48.018 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b11ed802:::benchmark_data_smithi122_119671_object36549:head# 2024-03-20T09:04:48.019 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b120391b:::benchmark_data_smithi122_119671_object17689:head# 2024-03-20T09:04:48.019 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b120578c:::benchmark_data_smithi122_119671_object83357:head# 2024-03-20T09:04:48.019 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b120ba69:::benchmark_data_smithi122_119671_object51813:head# 2024-03-20T09:04:48.020 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b120d9ea:::benchmark_data_smithi122_119671_object84659:head# 2024-03-20T09:04:48.020 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b12304d1:::benchmark_data_smithi122_119671_object47639:head# 2024-03-20T09:04:48.021 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b12316c5:::benchmark_data_smithi122_119671_object99116:head# 2024-03-20T09:04:48.021 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1241a5c:::benchmark_data_smithi122_119671_object29199:head# 2024-03-20T09:04:48.021 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b124d7c7:::benchmark_data_smithi122_119671_object40694:head# 2024-03-20T09:04:48.022 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b124f657:::benchmark_data_smithi122_119671_object56442:head# 2024-03-20T09:04:48.022 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b12752f4:::benchmark_data_smithi122_119671_object47344:head# 2024-03-20T09:04:48.023 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1278594:::benchmark_data_smithi122_119671_object17569:head# 2024-03-20T09:04:48.023 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b12839fc:::benchmark_data_smithi122_119671_object18084:head# 2024-03-20T09:04:48.024 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1296782:::benchmark_data_smithi122_119671_object5714:head# 2024-03-20T09:04:48.024 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b12aafbe:::benchmark_data_smithi122_119671_object21998:head# 2024-03-20T09:04:48.024 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b12c679f:::benchmark_data_smithi122_119671_object97548:head# 2024-03-20T09:04:48.025 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b12cd820:::benchmark_data_smithi122_119671_object68447:head# 2024-03-20T09:04:48.025 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b12dc91f:::benchmark_data_smithi122_119671_object90129:head# 2024-03-20T09:04:48.026 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b12e61d2:::benchmark_data_smithi122_119671_object34726:head# 2024-03-20T09:04:48.026 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b12f6026:::benchmark_data_smithi122_119671_object34187:head# 2024-03-20T09:04:48.026 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b12f6ca7:::benchmark_data_smithi122_119671_object21522:head# 2024-03-20T09:04:48.027 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b12f78d7:::benchmark_data_smithi122_119671_object89847:head# 2024-03-20T09:04:48.027 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b12f8e76:::benchmark_data_smithi122_119671_object14007:head# 2024-03-20T09:04:48.028 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b12ff9a0:::benchmark_data_smithi122_119671_object60390:head# 2024-03-20T09:04:48.030 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b130147f:::benchmark_data_smithi122_119671_object22146:head# 2024-03-20T09:04:48.030 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1307515:::benchmark_data_smithi122_119671_object76172:head# 2024-03-20T09:04:48.031 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b130912d:::benchmark_data_smithi122_119671_object97853:head# 2024-03-20T09:04:48.031 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b130e5e2:::benchmark_data_smithi122_119671_object63395:head# 2024-03-20T09:04:48.032 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1310f92:::benchmark_data_smithi122_119671_object11997:head# 2024-03-20T09:04:48.032 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b132e36f:::benchmark_data_smithi122_119671_object89892:head# 2024-03-20T09:04:48.033 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b133ae86:::benchmark_data_smithi122_119671_object28764:head# 2024-03-20T09:04:48.033 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b133f5e1:::benchmark_data_smithi122_119671_object92200:head# 2024-03-20T09:04:48.034 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1346a78:::benchmark_data_smithi122_119671_object74779:head# 2024-03-20T09:04:48.034 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b134b090:::benchmark_data_smithi122_119671_object53375:head# 2024-03-20T09:04:48.035 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b137f03d:::benchmark_data_smithi122_119671_object99525:head# 2024-03-20T09:04:48.036 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1382c22:::benchmark_data_smithi122_119671_object7973:head# 2024-03-20T09:04:48.037 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1386124:::benchmark_data_smithi122_119671_object78347:head# 2024-03-20T09:04:48.037 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1399f42:::benchmark_data_smithi122_119671_object29480:head# 2024-03-20T09:04:48.037 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b13bf863:::benchmark_data_smithi122_119671_object58940:head# 2024-03-20T09:04:48.038 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b13dd1ba:::benchmark_data_smithi122_119671_object30342:head# 2024-03-20T09:04:48.038 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b13e37ed:::benchmark_data_smithi122_119671_object63606:head# 2024-03-20T09:04:48.039 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b13edcb6:::benchmark_data_smithi122_119671_object75509:head# 2024-03-20T09:04:48.039 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1412885:::benchmark_data_smithi122_119671_object48325:head# 2024-03-20T09:04:48.040 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b14204d9:::benchmark_data_smithi122_119671_object53207:head# 2024-03-20T09:04:48.040 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b14247d0:::benchmark_data_smithi122_119671_object2365:head# 2024-03-20T09:04:48.040 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1426146:::benchmark_data_smithi122_119671_object99237:head# 2024-03-20T09:04:48.041 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1447b22:::benchmark_data_smithi122_119671_object62547:head# 2024-03-20T09:04:48.041 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1448d65:::benchmark_data_smithi122_119671_object67039:head# 2024-03-20T09:04:48.042 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b146735a:::benchmark_data_smithi122_119671_object9203:head# 2024-03-20T09:04:48.042 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b146a779:::benchmark_data_smithi122_119671_object20344:head# 2024-03-20T09:04:48.043 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1470836:::benchmark_data_smithi122_119671_object92288:head# 2024-03-20T09:04:48.043 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1479190:::benchmark_data_smithi122_119671_object96215:head# 2024-03-20T09:04:48.043 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b147acf0:::benchmark_data_smithi122_119671_object43711:head# 2024-03-20T09:04:48.044 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b147e316:::benchmark_data_smithi122_119671_object56196:head# 2024-03-20T09:04:48.044 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1486951:::benchmark_data_smithi122_119671_object9118:head# 2024-03-20T09:04:48.045 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b14a2e16:::benchmark_data_smithi122_119671_object73834:head# 2024-03-20T09:04:48.045 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b14a343f:::benchmark_data_smithi122_119671_object90099:head# 2024-03-20T09:04:48.046 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b14b7e77:::benchmark_data_smithi122_119671_object36591:head# 2024-03-20T09:04:48.046 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b14ba3d6:::benchmark_data_smithi122_119671_object56104:head# 2024-03-20T09:04:48.046 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b14bd22e:::benchmark_data_smithi122_119671_object66793:head# 2024-03-20T09:04:48.047 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b14ca1b5:::benchmark_data_smithi122_119671_object9583:head# 2024-03-20T09:04:48.047 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b14cb3e5:::benchmark_data_smithi122_119671_object68675:head# 2024-03-20T09:04:48.048 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b14e585a:::benchmark_data_smithi122_119671_object34588:head# 2024-03-20T09:04:48.049 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b14eab8c:::benchmark_data_smithi122_119671_object91083:head# 2024-03-20T09:04:48.049 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b14ef4c4:::benchmark_data_smithi122_119671_object25000:head# 2024-03-20T09:04:48.049 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b14f1f32:::benchmark_data_smithi122_119671_object8354:head# 2024-03-20T09:04:48.050 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b14f3828:::benchmark_data_smithi122_119671_object95623:head# 2024-03-20T09:04:48.050 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b150c503:::benchmark_data_smithi122_119671_object97678:head# 2024-03-20T09:04:48.051 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b150f8b6:::benchmark_data_smithi122_119671_object43193:head# 2024-03-20T09:04:48.051 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1513461:::benchmark_data_smithi122_119671_object64981:head# 2024-03-20T09:04:48.051 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1515c96:::benchmark_data_smithi122_119671_object32062:head# 2024-03-20T09:04:48.052 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b15168e6:::benchmark_data_smithi122_119671_object90279:head# 2024-03-20T09:04:48.055 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1519266:::benchmark_data_smithi122_119671_object41688:head# 2024-03-20T09:04:48.055 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b151a863:::benchmark_data_smithi122_119671_object65370:head# 2024-03-20T09:04:48.055 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b15250b4:::benchmark_data_smithi122_119671_object7806:head# 2024-03-20T09:04:48.055 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b15319c6:::benchmark_data_smithi122_119671_object60314:head# 2024-03-20T09:04:48.055 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1536592:::benchmark_data_smithi122_119671_object72223:head# 2024-03-20T09:04:48.055 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b154c552:::benchmark_data_smithi122_119671_object85558:head# 2024-03-20T09:04:48.056 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b155ea13:::benchmark_data_smithi122_119671_object16083:head# 2024-03-20T09:04:48.056 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1561b0f:::benchmark_data_smithi122_119671_object67401:head# 2024-03-20T09:04:48.056 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b157658f:::benchmark_data_smithi122_119671_object44102:head# 2024-03-20T09:04:48.056 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1580712:::benchmark_data_smithi122_119671_object30871:head# 2024-03-20T09:04:48.057 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1580f02:::benchmark_data_smithi122_119671_object63064:head# 2024-03-20T09:04:48.057 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b158136c:::benchmark_data_smithi122_119671_object87359:head# 2024-03-20T09:04:48.058 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b159a887:::benchmark_data_smithi122_119671_object45172:head# 2024-03-20T09:04:48.058 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b159ec81:::benchmark_data_smithi122_119671_object54425:head# 2024-03-20T09:04:48.058 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b15a8f8d:::benchmark_data_smithi122_119671_object21731:head# 2024-03-20T09:04:48.059 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b15b17b0:::benchmark_data_smithi122_119671_object66780:head# 2024-03-20T09:04:48.060 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b15b9c8e:::benchmark_data_smithi122_119671_object20464:head# 2024-03-20T09:04:48.060 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b15be42a:::benchmark_data_smithi122_119671_object58230:head# 2024-03-20T09:04:48.060 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b15c6408:::benchmark_data_smithi122_119671_object58690:head# 2024-03-20T09:04:48.061 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b15d641a:::benchmark_data_smithi122_119671_object46587:head# 2024-03-20T09:04:48.062 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b15e2e5c:::benchmark_data_smithi122_119671_object40586:head# 2024-03-20T09:04:48.062 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b15ebd78:::benchmark_data_smithi122_119671_object2749:head# 2024-03-20T09:04:48.063 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b15fefc6:::benchmark_data_smithi122_119671_object31417:head# 2024-03-20T09:04:48.064 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1608934:::benchmark_data_smithi122_119671_object98046:head# 2024-03-20T09:04:48.064 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b160cb14:::benchmark_data_smithi122_119671_object42965:head# 2024-03-20T09:04:48.065 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1613a07:::benchmark_data_smithi122_119671_object3118:head# 2024-03-20T09:04:48.065 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b16159a0:::benchmark_data_smithi122_119671_object19652:head# 2024-03-20T09:04:48.066 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1621597:::benchmark_data_smithi122_119671_object78702:head# 2024-03-20T09:04:48.066 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1622276:::benchmark_data_smithi122_119671_object98385:head# 2024-03-20T09:04:48.067 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b163cd4e:::benchmark_data_smithi122_119671_object31843:head# 2024-03-20T09:04:48.068 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b16495f2:::benchmark_data_smithi122_119671_object17195:head# 2024-03-20T09:04:48.068 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1649915:::benchmark_data_smithi122_119671_object17150:head# 2024-03-20T09:04:48.068 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b164a34e:::benchmark_data_smithi122_119671_object76028:head# 2024-03-20T09:04:48.068 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b164d87e:::benchmark_data_smithi122_119671_object1955:head# 2024-03-20T09:04:48.069 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b165154b:::benchmark_data_smithi122_119671_object60968:head# 2024-03-20T09:04:48.069 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b165684b:::benchmark_data_smithi122_119671_object95328:head# 2024-03-20T09:04:48.069 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1659e43:::benchmark_data_smithi122_119671_object46961:head# 2024-03-20T09:04:48.070 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b16607f5:::benchmark_data_smithi122_119671_object25424:head# 2024-03-20T09:04:48.071 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b16607fd:::benchmark_data_smithi122_119671_object67181:head# 2024-03-20T09:04:48.071 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1694688:::benchmark_data_smithi122_119671_object84594:head# 2024-03-20T09:04:48.071 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b169671b:::benchmark_data_smithi122_119671_object25647:head# 2024-03-20T09:04:48.072 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b16b1eaa:::benchmark_data_smithi122_119671_object32341:head# 2024-03-20T09:04:48.073 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b16becaf:::benchmark_data_smithi122_119671_object55687:head# 2024-03-20T09:04:48.073 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b16d2c7d:::benchmark_data_smithi122_119671_object47508:head# 2024-03-20T09:04:48.075 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b16e0b71:::benchmark_data_smithi122_119671_object77507:head# 2024-03-20T09:04:48.075 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b16e2772:::benchmark_data_smithi122_119671_object64562:head# 2024-03-20T09:04:48.076 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b16e6745:::benchmark_data_smithi122_119671_object95605:head# 2024-03-20T09:04:48.076 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b16f70d2:::benchmark_data_smithi122_119671_object87321:head# 2024-03-20T09:04:48.077 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b16fab07:::benchmark_data_smithi122_119671_object30031:head# 2024-03-20T09:04:48.077 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b16ff496:::benchmark_data_smithi122_119671_object86482:head# 2024-03-20T09:04:48.077 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b171ac5d:::benchmark_data_smithi122_119671_object65442:head# 2024-03-20T09:04:48.078 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1723f64:::benchmark_data_smithi122_119671_object27434:head# 2024-03-20T09:04:48.078 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1748b2e:::benchmark_data_smithi122_119671_object4925:head# 2024-03-20T09:04:48.079 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1751e29:::benchmark_data_smithi122_119671_object62859:head# 2024-03-20T09:04:48.079 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1753575:::benchmark_data_smithi122_119671_object30472:head# 2024-03-20T09:04:48.080 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1756077:::benchmark_data_smithi122_119671_object7621:head# 2024-03-20T09:04:48.080 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b175673e:::benchmark_data_smithi122_119671_object65461:head# 2024-03-20T09:04:48.080 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1759f64:::benchmark_data_smithi122_119671_object62802:head# 2024-03-20T09:04:48.081 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b175b521:::benchmark_data_smithi122_119671_object89020:head# 2024-03-20T09:04:48.081 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b175c240:::benchmark_data_smithi122_119671_object15139:head# 2024-03-20T09:04:48.082 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b176fdee:::benchmark_data_smithi122_119671_object77158:head# 2024-03-20T09:04:48.082 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1789fe9:::benchmark_data_smithi122_119671_object51510:head# 2024-03-20T09:04:48.082 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b17b61a1:::benchmark_data_smithi122_119671_object17921:head# 2024-03-20T09:04:48.083 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b17bcd9a:::benchmark_data_smithi122_119671_object33573:head# 2024-03-20T09:04:48.083 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b17c0991:::benchmark_data_smithi122_119671_object85079:head# 2024-03-20T09:04:48.084 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b17c567a:::benchmark_data_smithi122_119671_object33370:head# 2024-03-20T09:04:48.084 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b17cedb0:::benchmark_data_smithi122_119671_object91075:head# 2024-03-20T09:04:48.084 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b17d6c82:::benchmark_data_smithi122_119671_object38866:head# 2024-03-20T09:04:48.085 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b17e7ce5:::benchmark_data_smithi122_119671_object83761:head# 2024-03-20T09:04:48.085 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b17e81bf:::benchmark_data_smithi122_119671_object90235:head# 2024-03-20T09:04:48.086 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b17ee6c4:::benchmark_data_smithi122_119671_object95346:head# 2024-03-20T09:04:48.086 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b180c38a:::benchmark_data_smithi122_119671_object31054:head# 2024-03-20T09:04:48.086 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1810533:::benchmark_data_smithi122_119671_object61511:head# 2024-03-20T09:04:48.087 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b18276a0:::benchmark_data_smithi122_119671_object59136:head# 2024-03-20T09:04:48.087 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b182c2d8:::benchmark_data_smithi122_119671_object7561:head# 2024-03-20T09:04:48.087 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b18346a4:::benchmark_data_smithi122_119671_object24836:head# 2024-03-20T09:04:48.088 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b183ef05:::benchmark_data_smithi122_119671_object45482:head# 2024-03-20T09:04:48.088 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1849232:::benchmark_data_smithi122_119671_object1144:head# 2024-03-20T09:04:48.089 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b185c8b8:::benchmark_data_smithi122_119671_object77834:head# 2024-03-20T09:04:48.089 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b186875d:::benchmark_data_smithi122_119671_object49704:head# 2024-03-20T09:04:48.090 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b186a8c6:::benchmark_data_smithi122_119671_object85994:head# 2024-03-20T09:04:48.090 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b18838b1:::benchmark_data_smithi122_119671_object53111:head# 2024-03-20T09:04:48.090 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b189765d:::benchmark_data_smithi122_119671_object6741:head# 2024-03-20T09:04:48.091 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1898a88:::benchmark_data_smithi122_119671_object86453:head# 2024-03-20T09:04:48.091 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b189d92e:::benchmark_data_smithi122_119671_object92426:head# 2024-03-20T09:04:48.092 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b189f68e:::benchmark_data_smithi122_119671_object24872:head# 2024-03-20T09:04:48.092 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b18a6b44:::benchmark_data_smithi122_119671_object95719:head# 2024-03-20T09:04:48.092 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b18a9d8a:::benchmark_data_smithi122_119671_object84529:head# 2024-03-20T09:04:48.093 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b18b368c:::benchmark_data_smithi122_119671_object28425:head# 2024-03-20T09:04:48.093 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b18bf211:::benchmark_data_smithi122_119671_object5612:head# 2024-03-20T09:04:48.094 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b18c2547:::benchmark_data_smithi122_119671_object16263:head# 2024-03-20T09:04:48.094 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b18c2fbb:::benchmark_data_smithi122_119671_object61053:head# 2024-03-20T09:04:48.094 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b18e5684:::benchmark_data_smithi122_119671_object76604:head# 2024-03-20T09:04:48.096 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b18fd62c:::benchmark_data_smithi122_119671_object78245:head# 2024-03-20T09:04:48.096 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b18fdf40:::benchmark_data_smithi122_119671_object75850:head# 2024-03-20T09:04:48.097 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1913ba8:::benchmark_data_smithi122_119671_object51012:head# 2024-03-20T09:04:48.097 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b19162fd:::benchmark_data_smithi122_119671_object53033:head# 2024-03-20T09:04:48.098 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b192830d:::benchmark_data_smithi122_119671_object22445:head# 2024-03-20T09:04:48.098 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b19320a2:::benchmark_data_smithi122_119671_object62775:head# 2024-03-20T09:04:48.099 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b193a8de:::benchmark_data_smithi122_119671_object73526:head# 2024-03-20T09:04:48.099 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b193ea4a:::benchmark_data_smithi122_119671_object89865:head# 2024-03-20T09:04:48.099 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b194bad5:::benchmark_data_smithi122_119671_object67515:head# 2024-03-20T09:04:48.100 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b196180a:::benchmark_data_smithi122_119671_object42551:head# 2024-03-20T09:04:48.100 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1967e13:::benchmark_data_smithi122_119671_object34527:head# 2024-03-20T09:04:48.101 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1968497:::benchmark_data_smithi122_119671_object95902:head# 2024-03-20T09:04:48.101 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b196b3cf:::benchmark_data_smithi122_119671_object84501:head# 2024-03-20T09:04:48.101 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b196deba:::benchmark_data_smithi122_119671_object47336:head# 2024-03-20T09:04:48.102 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b197b042:::benchmark_data_smithi122_119671_object82707:head# 2024-03-20T09:04:48.102 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b197caff:::benchmark_data_smithi122_119671_object4571:head# 2024-03-20T09:04:48.103 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b19819fc:::benchmark_data_smithi122_119671_object44904:head# 2024-03-20T09:04:48.103 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b198c036:::benchmark_data_smithi122_119671_object4376:head# 2024-03-20T09:04:48.104 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b198e623:::benchmark_data_smithi122_119671_object35712:head# 2024-03-20T09:04:48.104 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b198fbeb:::benchmark_data_smithi122_119671_object18903:head# 2024-03-20T09:04:48.104 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b199430a:::benchmark_data_smithi122_119671_object24684:head# 2024-03-20T09:04:48.105 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1997288:::benchmark_data_smithi122_119671_object15458:head# 2024-03-20T09:04:48.105 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b19a8e86:::benchmark_data_smithi122_119671_object91909:head# 2024-03-20T09:04:48.106 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b19ad2d7:::benchmark_data_smithi122_119671_object98638:head# 2024-03-20T09:04:48.106 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b19b1f6c:::benchmark_data_smithi122_119671_object42828:head# 2024-03-20T09:04:48.106 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b19c8091:::benchmark_data_smithi122_119671_object73468:head# 2024-03-20T09:04:48.107 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b19cd33c:::benchmark_data_smithi122_119671_object8996:head# 2024-03-20T09:04:48.107 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b19cd572:::benchmark_data_smithi122_119671_object41181:head# 2024-03-20T09:04:48.108 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b19de0bf:::benchmark_data_smithi122_119671_object30188:head# 2024-03-20T09:04:48.108 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b19fce86:::benchmark_data_smithi122_119671_object74671:head# 2024-03-20T09:04:48.108 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1a00a97:::benchmark_data_smithi122_119671_object92345:head# 2024-03-20T09:04:48.109 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1a08c55:::benchmark_data_smithi122_119671_object42332:head# 2024-03-20T09:04:48.109 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1a14127:::benchmark_data_smithi122_119671_object37286:head# 2024-03-20T09:04:48.109 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1a1d9e4:::benchmark_data_smithi122_119671_object51360:head# 2024-03-20T09:04:48.110 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1a219a2:::benchmark_data_smithi122_119671_object93597:head# 2024-03-20T09:04:48.110 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1a27d98:::benchmark_data_smithi122_119671_object94193:head# 2024-03-20T09:04:48.111 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1a46eb4:::benchmark_data_smithi122_119671_object14634:head# 2024-03-20T09:04:48.111 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1a4be3b:::benchmark_data_smithi122_119671_object73459:head# 2024-03-20T09:04:48.111 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1a4c77d:::benchmark_data_smithi122_119671_object48844:head# 2024-03-20T09:04:48.112 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1a4ec29:::benchmark_data_smithi122_119671_object78330:head# 2024-03-20T09:04:48.112 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1a5b395:::benchmark_data_smithi122_119671_object81012:head# 2024-03-20T09:04:48.113 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1a6d7ba:::benchmark_data_smithi122_119671_object36560:head# 2024-03-20T09:04:48.113 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1a7ae69:::benchmark_data_smithi122_119671_object77369:head# 2024-03-20T09:04:48.113 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1a7fd0d:::benchmark_data_smithi122_119671_object17623:head# 2024-03-20T09:04:48.114 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1a82001:::benchmark_data_smithi122_119671_object74833:head# 2024-03-20T09:04:48.114 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1a99919:::benchmark_data_smithi122_119671_object21798:head# 2024-03-20T09:04:48.115 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1aac9cc:::benchmark_data_smithi122_119671_object61041:head# 2024-03-20T09:04:48.119 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1abc240:::benchmark_data_smithi122_119671_object74391:head# 2024-03-20T09:04:48.119 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1ac88f4:::benchmark_data_smithi122_119671_object99602:head# 2024-03-20T09:04:48.119 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1acc9b9:::benchmark_data_smithi122_119671_object53364:head# 2024-03-20T09:04:48.119 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1accf29:::benchmark_data_smithi122_119671_object65023:head# 2024-03-20T09:04:48.119 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1ad4c8e:::benchmark_data_smithi122_119671_object99420:head# 2024-03-20T09:04:48.119 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1ad7ca4:::benchmark_data_smithi122_119671_object21282:head# 2024-03-20T09:04:48.119 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1ad8ded:::benchmark_data_smithi122_119671_object70883:head# 2024-03-20T09:04:48.119 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1ae2f99:::benchmark_data_smithi122_119671_object37013:head# 2024-03-20T09:04:48.119 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1af0cd7:::benchmark_data_smithi122_119671_object30461:head# 2024-03-20T09:04:48.120 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1af12b0:::benchmark_data_smithi122_119671_object29690:head# 2024-03-20T09:04:48.120 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1af2a30:::benchmark_data_smithi122_119671_object45146:head# 2024-03-20T09:04:48.120 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1b023e9:::benchmark_data_smithi122_119671_object62793:head# 2024-03-20T09:04:48.120 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1b0b9f7:::benchmark_data_smithi122_119671_object56003:head# 2024-03-20T09:04:48.121 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1b1014c:::benchmark_data_smithi122_119671_object21929:head# 2024-03-20T09:04:48.121 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1b223f2:::benchmark_data_smithi122_119671_object47106:head# 2024-03-20T09:04:48.121 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1b29c82:::benchmark_data_smithi122_119671_object73942:head# 2024-03-20T09:04:48.121 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1b2bf73:::benchmark_data_smithi122_119671_object90248:head# 2024-03-20T09:04:48.122 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1b3b5d0:::benchmark_data_smithi122_119671_object13370:head# 2024-03-20T09:04:48.122 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1b3c80c:::benchmark_data_smithi122_119671_object44616:head# 2024-03-20T09:04:48.123 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1b41376:::benchmark_data_smithi122_119671_object58236:head# 2024-03-20T09:04:48.123 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1b43578:::benchmark_data_smithi122_119671_object16108:head# 2024-03-20T09:04:48.123 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1b507a2:::benchmark_data_smithi122_119671_object73480:head# 2024-03-20T09:04:48.124 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1b5a684:::benchmark_data_smithi122_119671_object30210:head# 2024-03-20T09:04:48.124 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1b5abda:::benchmark_data_smithi122_119671_object62390:head# 2024-03-20T09:04:48.125 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1b61265:::benchmark_data_smithi122_119671_object70257:head# 2024-03-20T09:04:48.125 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1b6840f:::benchmark_data_smithi122_119671_object39089:head# 2024-03-20T09:04:48.126 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1b6a032:::benchmark_data_smithi122_119671_object9954:head# 2024-03-20T09:04:48.126 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1b6b058:::benchmark_data_smithi122_119671_object39268:head# 2024-03-20T09:04:48.127 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1b6cde3:::benchmark_data_smithi122_119671_object48439:head# 2024-03-20T09:04:48.127 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1bbe552:::benchmark_data_smithi122_119671_object98610:head# 2024-03-20T09:04:48.128 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1bdca38:::benchmark_data_smithi122_119671_object33556:head# 2024-03-20T09:04:48.128 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1bdf85e:::benchmark_data_smithi122_119671_object6659:head# 2024-03-20T09:04:48.128 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1be28ad:::benchmark_data_smithi122_119671_object36018:head# 2024-03-20T09:04:48.129 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1beaf27:::benchmark_data_smithi122_119671_object10485:head# 2024-03-20T09:04:48.129 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1bece77:::benchmark_data_smithi122_119671_object76384:head# 2024-03-20T09:04:48.129 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1bf45cd:::benchmark_data_smithi122_119671_object86156:head# 2024-03-20T09:04:48.130 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1c038fc:::benchmark_data_smithi122_119671_object82959:head# 2024-03-20T09:04:48.130 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1c05554:::benchmark_data_smithi122_119671_object93471:head# 2024-03-20T09:04:48.131 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1c129ee:::benchmark_data_smithi122_119671_object75857:head# 2024-03-20T09:04:48.131 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1c13d56:::benchmark_data_smithi122_119671_object83523:head# 2024-03-20T09:04:48.132 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1c2e6ba:::benchmark_data_smithi122_119671_object3161:head# 2024-03-20T09:04:48.133 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1c3ba7a:::benchmark_data_smithi122_119671_object22475:head# 2024-03-20T09:04:48.133 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1c43195:::benchmark_data_smithi122_119671_object60786:head# 2024-03-20T09:04:48.134 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1c4d3bd:::benchmark_data_smithi122_119671_object47554:head# 2024-03-20T09:04:48.134 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1c5f2c9:::benchmark_data_smithi122_119671_object67581:head# 2024-03-20T09:04:48.134 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1c6d9c8:::benchmark_data_smithi122_119671_object68283:head# 2024-03-20T09:04:48.135 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1c7bc00:::benchmark_data_smithi122_119671_object23975:head# 2024-03-20T09:04:48.135 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1c7c3a4:::benchmark_data_smithi122_119671_object56836:head# 2024-03-20T09:04:48.136 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1c7e700:::benchmark_data_smithi122_119671_object98240:head# 2024-03-20T09:04:48.137 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1c90eaf:::benchmark_data_smithi122_119671_object91076:head# 2024-03-20T09:04:48.137 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1c951be:::benchmark_data_smithi122_119671_object55380:head# 2024-03-20T09:04:48.137 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1c95b42:::benchmark_data_smithi122_119671_object73694:head# 2024-03-20T09:04:48.138 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1c9e653:::benchmark_data_smithi122_119671_object78065:head# 2024-03-20T09:04:48.138 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1ca0cff:::benchmark_data_smithi122_119671_object11488:head# 2024-03-20T09:04:48.139 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1ca13e0:::benchmark_data_smithi122_119671_object68113:head# 2024-03-20T09:04:48.139 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1ca3dcd:::benchmark_data_smithi122_119671_object48043:head# 2024-03-20T09:04:48.140 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1ca664a:::benchmark_data_smithi122_119671_object6190:head# 2024-03-20T09:04:48.140 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1ca9b8e:::benchmark_data_smithi122_119671_object40889:head# 2024-03-20T09:04:48.141 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1caa934:::benchmark_data_smithi122_119671_object79485:head# 2024-03-20T09:04:48.141 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1caece3:::benchmark_data_smithi122_119671_object3831:head# 2024-03-20T09:04:48.142 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1cbfa33:::benchmark_data_smithi122_119671_object87875:head# 2024-03-20T09:04:48.142 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1cc582e:::benchmark_data_smithi122_119671_object56379:head# 2024-03-20T09:04:48.143 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1cc590a:::benchmark_data_smithi122_119671_object86606:head# 2024-03-20T09:04:48.143 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1ccb9c5:::benchmark_data_smithi122_119671_object35249:head# 2024-03-20T09:04:48.144 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1cce97a:::benchmark_data_smithi122_119671_object37707:head# 2024-03-20T09:04:48.144 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1ce1b70:::benchmark_data_smithi122_119671_object97905:head# 2024-03-20T09:04:48.145 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1cec431:::benchmark_data_smithi122_119671_object5854:head# 2024-03-20T09:04:48.145 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1cf45b2:::benchmark_data_smithi122_119671_object92088:head# 2024-03-20T09:04:48.146 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1cf4843:::benchmark_data_smithi122_119671_object87956:head# 2024-03-20T09:04:48.146 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1cfcba0:::benchmark_data_smithi122_119671_object7131:head# 2024-03-20T09:04:48.147 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1d0a67b:::benchmark_data_smithi122_119671_object90251:head# 2024-03-20T09:04:48.147 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1d0fdbb:::benchmark_data_smithi122_119671_object39244:head# 2024-03-20T09:04:48.148 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1d4a6c6:::benchmark_data_smithi122_119671_object36977:head# 2024-03-20T09:04:48.148 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1d4e62d:::benchmark_data_smithi122_119671_object21442:head# 2024-03-20T09:04:48.149 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1d5975d:::benchmark_data_smithi122_119671_object34361:head# 2024-03-20T09:04:48.149 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1d72e2b:::benchmark_data_smithi122_119671_object8715:head# 2024-03-20T09:04:48.149 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1d8a6d6:::benchmark_data_smithi122_119671_object5701:head# 2024-03-20T09:04:48.150 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1da2ab7:::benchmark_data_smithi122_119671_object81339:head# 2024-03-20T09:04:48.150 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1da3286:::benchmark_data_smithi122_119671_object5432:head# 2024-03-20T09:04:48.151 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1da48ee:::benchmark_data_smithi122_119671_object59763:head# 2024-03-20T09:04:48.151 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1da7a02:::benchmark_data_smithi122_119671_object95587:head# 2024-03-20T09:04:48.152 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1dac1ab:::benchmark_data_smithi122_119671_object15005:head# 2024-03-20T09:04:48.152 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1db244d:::benchmark_data_smithi122_119671_object10835:head# 2024-03-20T09:04:48.153 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1dc29aa:::benchmark_data_smithi122_119671_object28601:head# 2024-03-20T09:04:48.153 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1ddaf1c:::benchmark_data_smithi122_119671_object49246:head# 2024-03-20T09:04:48.153 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1ddcab3:::benchmark_data_smithi122_119671_object25983:head# 2024-03-20T09:04:48.154 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1df656a:::benchmark_data_smithi122_119671_object20338:head# 2024-03-20T09:04:48.155 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1dff63b:::benchmark_data_smithi122_119671_object14147:head# 2024-03-20T09:04:48.155 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1e02660:::benchmark_data_smithi122_119671_object14841:head# 2024-03-20T09:04:48.156 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1e05312:::benchmark_data_smithi122_119671_object1108:head# 2024-03-20T09:04:48.156 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1e0e7fc:::benchmark_data_smithi122_119671_object1374:head# 2024-03-20T09:04:48.156 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1e1a999:::benchmark_data_smithi122_119671_object15209:head# 2024-03-20T09:04:48.157 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1e1c882:::benchmark_data_smithi122_119671_object74464:head# 2024-03-20T09:04:48.157 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1e21f15:::benchmark_data_smithi122_119671_object49289:head# 2024-03-20T09:04:48.158 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1e26233:::benchmark_data_smithi122_119671_object39435:head# 2024-03-20T09:04:48.158 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1e2cab1:::benchmark_data_smithi122_119671_object72528:head# 2024-03-20T09:04:48.159 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1e2f83e:::benchmark_data_smithi122_119671_object70300:head# 2024-03-20T09:04:48.159 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1e3133e:::benchmark_data_smithi122_119671_object77979:head# 2024-03-20T09:04:48.160 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1e4e251:::benchmark_data_smithi122_119671_object92753:head# 2024-03-20T09:04:48.160 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1e50767:::benchmark_data_smithi122_119671_object30762:head# 2024-03-20T09:04:48.161 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1e5b193:::benchmark_data_smithi122_119671_object31829:head# 2024-03-20T09:04:48.161 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1e5fdf3:::benchmark_data_smithi122_119671_object85379:head# 2024-03-20T09:04:48.161 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1e608ab:::benchmark_data_smithi122_119671_object26335:head# 2024-03-20T09:04:48.162 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1e6117d:::benchmark_data_smithi122_119671_object85465:head# 2024-03-20T09:04:48.162 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1e6f46b:::benchmark_data_smithi122_119671_object62009:head# 2024-03-20T09:04:48.163 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1e77844:::benchmark_data_smithi122_119671_object15496:head# 2024-03-20T09:04:48.163 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1e7e8cc:::benchmark_data_smithi122_119671_object37167:head# 2024-03-20T09:04:48.163 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1e8999a:::benchmark_data_smithi122_119671_object37428:head# 2024-03-20T09:04:48.164 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1e8de57:::benchmark_data_smithi122_119671_object29970:head# 2024-03-20T09:04:48.164 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1e91e52:::benchmark_data_smithi122_119671_object41315:head# 2024-03-20T09:04:48.165 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1e9a751:::benchmark_data_smithi122_119671_object16759:head# 2024-03-20T09:04:48.165 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1ea9fbe:::benchmark_data_smithi122_119671_object19241:head# 2024-03-20T09:04:48.166 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1eaaf14:::benchmark_data_smithi122_119671_object4050:head# 2024-03-20T09:04:48.166 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1eb6377:::benchmark_data_smithi122_119671_object24579:head# 2024-03-20T09:04:48.167 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1ebb391:::benchmark_data_smithi122_119671_object68022:head# 2024-03-20T09:04:48.167 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1ebbaca:::benchmark_data_smithi122_119671_object3751:head# 2024-03-20T09:04:48.167 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1ec1a61:::benchmark_data_smithi122_119671_object89550:head# 2024-03-20T09:04:48.168 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1ed911e:::benchmark_data_smithi122_119671_object62987:head# 2024-03-20T09:04:48.168 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1ee76a1:::benchmark_data_smithi122_119671_object38151:head# 2024-03-20T09:04:48.169 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1f0b18d:::benchmark_data_smithi122_119671_object86229:head# 2024-03-20T09:04:48.169 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1f0b7a7:::benchmark_data_smithi122_119671_object17072:head# 2024-03-20T09:04:48.170 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1f17084:::benchmark_data_smithi122_119671_object64014:head# 2024-03-20T09:04:48.170 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1f2aaa3:::benchmark_data_smithi122_119671_object69355:head# 2024-03-20T09:04:48.171 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1f2cbca:::benchmark_data_smithi122_119671_object6844:head# 2024-03-20T09:04:48.171 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1f5986a:::benchmark_data_smithi122_119671_object85122:head# 2024-03-20T09:04:48.171 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1f65b4d:::benchmark_data_smithi122_119671_object9876:head# 2024-03-20T09:04:48.172 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1f692b5:::benchmark_data_smithi122_119671_object14574:head# 2024-03-20T09:04:48.172 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1f79e0e:::benchmark_data_smithi122_119671_object28624:head# 2024-03-20T09:04:48.173 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1f84efa:::benchmark_data_smithi122_119671_object72039:head# 2024-03-20T09:04:48.173 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1f8e414:::benchmark_data_smithi122_119671_object58808:head# 2024-03-20T09:04:48.173 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1f8e87e:::benchmark_data_smithi122_119671_object86730:head# 2024-03-20T09:04:48.174 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1f8e92e:::benchmark_data_smithi122_119671_object52140:head# 2024-03-20T09:04:48.174 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1f9bfb6:::benchmark_data_smithi122_119671_object76382:head# 2024-03-20T09:04:48.175 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1f9f95d:::benchmark_data_smithi122_119671_object36213:head# 2024-03-20T09:04:48.175 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1fab800:::benchmark_data_smithi122_119671_object50486:head# 2024-03-20T09:04:48.175 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1fb8a9f:::benchmark_data_smithi122_119671_object10937:head# 2024-03-20T09:04:48.176 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1fc8947:::benchmark_data_smithi122_119671_object64712:head# 2024-03-20T09:04:48.176 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1fcb5c6:::benchmark_data_smithi122_119671_object1451:head# 2024-03-20T09:04:48.176 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1fd2d51:::benchmark_data_smithi122_119671_object85026:head# 2024-03-20T09:04:48.177 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1ff61e0:::benchmark_data_smithi122_119671_object84547:head# 2024-03-20T09:04:48.177 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b1fff8f9:::benchmark_data_smithi122_119671_object41162:head# 2024-03-20T09:04:48.177 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2001cf2:::benchmark_data_smithi122_119671_object82022:head# 2024-03-20T09:04:48.178 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2007b09:::benchmark_data_smithi122_119671_object9922:head# 2024-03-20T09:04:48.178 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b20182e8:::benchmark_data_smithi122_119671_object24155:head# 2024-03-20T09:04:48.179 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b201cbf3:::benchmark_data_smithi122_119671_object70496:head# 2024-03-20T09:04:48.179 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b201d39c:::benchmark_data_smithi122_119671_object42916:head# 2024-03-20T09:04:48.180 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b202ea2a:::benchmark_data_smithi122_119671_object58614:head# 2024-03-20T09:04:48.180 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b202f9b7:::benchmark_data_smithi122_119671_object81159:head# 2024-03-20T09:04:48.181 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b20331b0:::benchmark_data_smithi122_119671_object56914:head# 2024-03-20T09:04:48.181 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2042e4f:::benchmark_data_smithi122_119671_object10593:head# 2024-03-20T09:04:48.181 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2050499:::benchmark_data_smithi122_119671_object10177:head# 2024-03-20T09:04:48.182 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b206719e:::benchmark_data_smithi122_119671_object14592:head# 2024-03-20T09:04:48.182 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b207d6ab:::benchmark_data_smithi122_119671_object45632:head# 2024-03-20T09:04:48.183 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b207f543:::benchmark_data_smithi122_119671_object58260:head# 2024-03-20T09:04:48.183 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2082d51:::benchmark_data_smithi122_119671_object63032:head# 2024-03-20T09:04:48.183 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b209cf13:::benchmark_data_smithi122_119671_object5774:head# 2024-03-20T09:04:48.184 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b20b71d6:::benchmark_data_smithi122_119671_object75378:head# 2024-03-20T09:04:48.184 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b20bff4c:::benchmark_data_smithi122_119671_object58319:head# 2024-03-20T09:04:48.185 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b20c6daa:::benchmark_data_smithi122_119671_object35445:head# 2024-03-20T09:04:48.185 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b20ce561:::benchmark_data_smithi122_119671_object85681:head# 2024-03-20T09:04:48.185 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b20d3f73:::benchmark_data_smithi122_119671_object65902:head# 2024-03-20T09:04:48.186 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b20d7df5:::benchmark_data_smithi122_119671_object49612:head# 2024-03-20T09:04:48.186 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b20e925a:::benchmark_data_smithi122_119671_object10529:head# 2024-03-20T09:04:48.187 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b20eebd8:::benchmark_data_smithi122_119671_object76975:head# 2024-03-20T09:04:48.187 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b20f46aa:::benchmark_data_smithi122_119671_object67641:head# 2024-03-20T09:04:48.188 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b20f61e7:::benchmark_data_smithi122_119671_object14460:head# 2024-03-20T09:04:48.188 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b20fa327:::benchmark_data_smithi122_119671_object20382:head# 2024-03-20T09:04:48.189 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2106157:::benchmark_data_smithi122_119671_object84192:head# 2024-03-20T09:04:48.189 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b210cf05:::benchmark_data_smithi122_119671_object22729:head# 2024-03-20T09:04:48.189 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b211cb93:::benchmark_data_smithi122_119671_object7822:head# 2024-03-20T09:04:48.190 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b212e205:::benchmark_data_smithi122_119671_object50779:head# 2024-03-20T09:04:48.191 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b21372be:::benchmark_data_smithi122_119671_object92462:head# 2024-03-20T09:04:48.191 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2151ed2:::benchmark_data_smithi122_119671_object69068:head# 2024-03-20T09:04:48.191 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b215e6e1:::benchmark_data_smithi122_119671_object74864:head# 2024-03-20T09:04:48.194 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b21618cd:::benchmark_data_smithi122_119671_object9475:head# 2024-03-20T09:04:48.195 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b21636bb:::benchmark_data_smithi122_119671_object4714:head# 2024-03-20T09:04:48.196 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b217b35f:::benchmark_data_smithi122_119671_object12776:head# 2024-03-20T09:04:48.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:47 smithi122 ceph-mon[73495]: 1.3 scrub starts 2024-03-20T09:04:48.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:47 smithi122 ceph-mon[73495]: 1.3 scrub ok 2024-03-20T09:04:48.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:47 smithi122 ceph-mon[73495]: pgmap v729: 47 pgs: 1 active+clean+remapped, 1 activating+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+undersized+degraded+remapped, 29 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 6.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 28599/295467 objects degraded (9.679%); 13490/295467 objects misplaced (4.566%); 36 MiB/s, 590 objects/s recovering 2024-03-20T09:04:48.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:47 smithi122 ceph-mon[73495]: 3.19 scrub starts 2024-03-20T09:04:48.197 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2186c95:::benchmark_data_smithi122_119671_object4292:head# 2024-03-20T09:04:48.198 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2186e7e:::benchmark_data_smithi122_119671_object49135:head# 2024-03-20T09:04:48.198 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b218cec3:::benchmark_data_smithi122_119671_object38963:head# 2024-03-20T09:04:48.198 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b21ae879:::benchmark_data_smithi122_119671_object61884:head# 2024-03-20T09:04:48.198 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b21b66c1:::benchmark_data_smithi122_119671_object15157:head# 2024-03-20T09:04:48.198 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b21bc29c:::benchmark_data_smithi122_119671_object11826:head# 2024-03-20T09:04:48.199 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b21f675a:::benchmark_data_smithi122_119671_object77607:head# 2024-03-20T09:04:48.199 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2215bff:::benchmark_data_smithi122_119671_object25092:head# 2024-03-20T09:04:48.200 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b221f5ed:::benchmark_data_smithi122_119671_object75388:head# 2024-03-20T09:04:48.200 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2223e90:::benchmark_data_smithi122_119671_object19055:head# 2024-03-20T09:04:48.201 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b223008f:::benchmark_data_smithi122_119671_object45275:head# 2024-03-20T09:04:48.201 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2231020:::benchmark_data_smithi122_119671_object97622:head# 2024-03-20T09:04:48.202 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b223713c:::benchmark_data_smithi122_119671_object65983:head# 2024-03-20T09:04:48.202 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b223d141:::benchmark_data_smithi122_119671_object90467:head# 2024-03-20T09:04:48.202 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b225edf4:::benchmark_data_smithi122_119671_object80785:head# 2024-03-20T09:04:48.203 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b225f556:::benchmark_data_smithi122_119671_object45077:head# 2024-03-20T09:04:48.206 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b22657e4:::benchmark_data_smithi122_119671_object3722:head# 2024-03-20T09:04:48.206 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b226b841:::benchmark_data_smithi122_119671_object88430:head# 2024-03-20T09:04:48.207 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b226dfbc:::benchmark_data_smithi122_119671_object59982:head# 2024-03-20T09:04:48.207 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2279c65:::benchmark_data_smithi122_119671_object44987:head# 2024-03-20T09:04:48.207 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b22a23a2:::benchmark_data_smithi122_119671_object55100:head# 2024-03-20T09:04:48.208 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b22af9bc:::benchmark_data_smithi122_119671_object22317:head# 2024-03-20T09:04:48.208 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b22b1d68:::benchmark_data_smithi122_119671_object37606:head# 2024-03-20T09:04:48.209 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b22b36c5:::benchmark_data_smithi122_119671_object22792:head# 2024-03-20T09:04:48.209 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b22b416d:::benchmark_data_smithi122_119671_object45144:head# 2024-03-20T09:04:48.209 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b22c94eb:::benchmark_data_smithi122_119671_object9830:head# 2024-03-20T09:04:48.210 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b22d5832:::benchmark_data_smithi122_119671_object27033:head# 2024-03-20T09:04:48.210 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b22dac6b:::benchmark_data_smithi122_119671_object44918:head# 2024-03-20T09:04:48.211 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b22e541d:::benchmark_data_smithi122_119671_object40541:head# 2024-03-20T09:04:48.211 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b22f821f:::benchmark_data_smithi122_119671_object6254:head# 2024-03-20T09:04:48.212 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b22f9800:::benchmark_data_smithi122_119671_object72937:head# 2024-03-20T09:04:48.212 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2307ba7:::benchmark_data_smithi122_119671_object41501:head# 2024-03-20T09:04:48.213 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b23192f8:::benchmark_data_smithi122_119671_object48018:head# 2024-03-20T09:04:48.213 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b231d354:::benchmark_data_smithi122_119671_object74453:head# 2024-03-20T09:04:48.213 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2326bbf:::benchmark_data_smithi122_119671_object92717:head# 2024-03-20T09:04:48.214 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b232a7e9:::benchmark_data_smithi122_119671_object9780:head# 2024-03-20T09:04:48.214 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b232e4fe:::benchmark_data_smithi122_119671_object8520:head# 2024-03-20T09:04:48.215 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2337aa1:::benchmark_data_smithi122_119671_object28185:head# 2024-03-20T09:04:48.215 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b233bb2d:::benchmark_data_smithi122_119671_object16628:head# 2024-03-20T09:04:48.216 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b233c4b9:::benchmark_data_smithi122_119671_object48663:head# 2024-03-20T09:04:48.216 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2343827:::benchmark_data_smithi122_119671_object18882:head# 2024-03-20T09:04:48.216 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b23457a7:::benchmark_data_smithi122_119671_object33256:head# 2024-03-20T09:04:48.217 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b23597bd:::benchmark_data_smithi122_119671_object4553:head# 2024-03-20T09:04:48.217 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b235e6ee:::benchmark_data_smithi122_119671_object18969:head# 2024-03-20T09:04:48.218 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b23721a0:::benchmark_data_smithi122_119671_object33946:head# 2024-03-20T09:04:48.218 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b237695a:::benchmark_data_smithi122_119671_object89323:head# 2024-03-20T09:04:48.219 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b237a956:::benchmark_data_smithi122_119671_object60759:head# 2024-03-20T09:04:48.219 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b238e1a4:::benchmark_data_smithi122_119671_object39289:head# 2024-03-20T09:04:48.220 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b23a00f1:::benchmark_data_smithi122_119671_object25748:head# 2024-03-20T09:04:48.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:47 smithi026 ceph-mon[68309]: 1.3 scrub starts 2024-03-20T09:04:48.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:47 smithi026 ceph-mon[68309]: 1.3 scrub ok 2024-03-20T09:04:48.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:47 smithi026 ceph-mon[68309]: pgmap v729: 47 pgs: 1 active+clean+remapped, 1 activating+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+undersized+degraded+remapped, 29 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 6.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 28599/295467 objects degraded (9.679%); 13490/295467 objects misplaced (4.566%); 36 MiB/s, 590 objects/s recovering 2024-03-20T09:04:48.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:47 smithi026 ceph-mon[68309]: 3.19 scrub starts 2024-03-20T09:04:48.221 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b23aa6e7:::benchmark_data_smithi122_119671_object95138:head# 2024-03-20T09:04:48.221 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b23af437:::benchmark_data_smithi122_119671_object40875:head# 2024-03-20T09:04:48.221 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b23c6153:::benchmark_data_smithi122_119671_object76447:head# 2024-03-20T09:04:48.222 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b23fc64f:::benchmark_data_smithi122_119671_object76394:head# 2024-03-20T09:04:48.222 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b242d0d7:::benchmark_data_smithi122_119671_object30326:head# 2024-03-20T09:04:48.223 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b242f6b4:::benchmark_data_smithi122_119671_object97087:head# 2024-03-20T09:04:48.223 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b242fb39:::benchmark_data_smithi122_119671_object65509:head# 2024-03-20T09:04:48.224 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b243fb88:::benchmark_data_smithi122_119671_object20132:head# 2024-03-20T09:04:48.224 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2445319:::benchmark_data_smithi122_119671_object66225:head# 2024-03-20T09:04:48.224 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2447118:::benchmark_data_smithi122_119671_object70865:head# 2024-03-20T09:04:48.225 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b245a384:::benchmark_data_smithi122_119671_object9658:head# 2024-03-20T09:04:48.227 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b245d360:::benchmark_data_smithi122_119671_object95161:head# 2024-03-20T09:04:48.228 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b246dd7f:::benchmark_data_smithi122_119671_object2466:head# 2024-03-20T09:04:48.230 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b247a281:::benchmark_data_smithi122_119671_object13469:head# 2024-03-20T09:04:48.230 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b247e872:::benchmark_data_smithi122_119671_object41935:head# 2024-03-20T09:04:48.231 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b24951ed:::benchmark_data_smithi122_119671_object47017:head# 2024-03-20T09:04:48.231 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b249a5d8:::benchmark_data_smithi122_119671_object72256:head# 2024-03-20T09:04:48.232 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b249dd2b:::benchmark_data_smithi122_119671_object41957:head# 2024-03-20T09:04:48.232 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b24abda6:::benchmark_data_smithi122_119671_object6120:head# 2024-03-20T09:04:48.233 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b24bada4:::benchmark_data_smithi122_119671_object8717:head# 2024-03-20T09:04:48.233 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b24bd5c8:::benchmark_data_smithi122_119671_object93142:head# 2024-03-20T09:04:48.233 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b24cf720:::benchmark_data_smithi122_119671_object61223:head# 2024-03-20T09:04:48.234 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b24d8e1c:::benchmark_data_smithi122_119671_object28283:head# 2024-03-20T09:04:48.234 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b24dc05d:::benchmark_data_smithi122_119671_object80398:head# 2024-03-20T09:04:48.235 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b24e117a:::benchmark_data_smithi122_119671_object59302:head# 2024-03-20T09:04:48.235 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b24ea8ad:::benchmark_data_smithi122_119671_object39333:head# 2024-03-20T09:04:48.235 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b24ec9a4:::benchmark_data_smithi122_119671_object71349:head# 2024-03-20T09:04:48.237 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2503cf5:::benchmark_data_smithi122_119671_object33482:head# 2024-03-20T09:04:48.237 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2523cf9:::benchmark_data_smithi122_119671_object96001:head# 2024-03-20T09:04:48.238 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b252a024:::benchmark_data_smithi122_119671_object8125:head# 2024-03-20T09:04:48.238 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b253822f:::benchmark_data_smithi122_119671_object76495:head# 2024-03-20T09:04:48.239 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2544794:::benchmark_data_smithi122_119671_object84860:head# 2024-03-20T09:04:48.239 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b25498ed:::benchmark_data_smithi122_119671_object47700:head# 2024-03-20T09:04:48.240 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b255622d:::benchmark_data_smithi122_119671_object47671:head# 2024-03-20T09:04:48.240 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b255acc5:::benchmark_data_smithi122_119671_object66085:head# 2024-03-20T09:04:48.242 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b255e358:::benchmark_data_smithi122_119671_object52161:head# 2024-03-20T09:04:48.242 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b255e66c:::benchmark_data_smithi122_119671_object24235:head# 2024-03-20T09:04:48.242 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b25607d3:::benchmark_data_smithi122_119671_object33328:head# 2024-03-20T09:04:48.243 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2567b1f:::benchmark_data_smithi122_119671_object80731:head# 2024-03-20T09:04:48.243 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b259653c:::benchmark_data_smithi122_119671_object72192:head# 2024-03-20T09:04:48.244 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b25a01e9:::benchmark_data_smithi122_119671_object58963:head# 2024-03-20T09:04:48.244 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b25a8c24:::benchmark_data_smithi122_119671_object68389:head# 2024-03-20T09:04:48.247 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b25b3662:::benchmark_data_smithi122_119671_object75057:head# 2024-03-20T09:04:48.247 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b25b796f:::benchmark_data_smithi122_119671_object45694:head# 2024-03-20T09:04:48.248 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b25bb26a:::benchmark_data_smithi122_119671_object90721:head# 2024-03-20T09:04:48.248 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b25c833a:::benchmark_data_smithi122_119671_object17622:head# 2024-03-20T09:04:48.249 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b25c9866:::benchmark_data_smithi122_119671_object13436:head# 2024-03-20T09:04:48.249 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b25cccc7:::benchmark_data_smithi122_119671_object67945:head# 2024-03-20T09:04:48.249 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b25d732c:::benchmark_data_smithi122_119671_object30441:head# 2024-03-20T09:04:48.250 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b25e0966:::benchmark_data_smithi122_119671_object31236:head# 2024-03-20T09:04:48.250 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b25ed2ca:::benchmark_data_smithi122_119671_object87001:head# 2024-03-20T09:04:48.251 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b25f0a55:::benchmark_data_smithi122_119671_object43621:head# 2024-03-20T09:04:48.251 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b25f4ec2:::benchmark_data_smithi122_119671_object70986:head# 2024-03-20T09:04:48.252 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b26215f3:::benchmark_data_smithi122_119671_object31182:head# 2024-03-20T09:04:48.252 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b263510e:::benchmark_data_smithi122_119671_object84823:head# 2024-03-20T09:04:48.252 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2635542:::benchmark_data_smithi122_119671_object73909:head# 2024-03-20T09:04:48.253 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b263f158:::benchmark_data_smithi122_119671_object54908:head# 2024-03-20T09:04:48.253 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2641e79:::benchmark_data_smithi122_119671_object42222:head# 2024-03-20T09:04:48.254 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b266e8f4:::benchmark_data_smithi122_119671_object62372:head# 2024-03-20T09:04:48.254 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b267a1b0:::benchmark_data_smithi122_119671_object91949:head# 2024-03-20T09:04:48.255 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b26816ed:::benchmark_data_smithi122_119671_object13340:head# 2024-03-20T09:04:48.255 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2690fba:::benchmark_data_smithi122_119671_object97449:head# 2024-03-20T09:04:48.255 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2695429:::benchmark_data_smithi122_119671_object79319:head# 2024-03-20T09:04:48.256 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b269a062:::benchmark_data_smithi122_119671_object35244:head# 2024-03-20T09:04:48.256 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b26a3665:::benchmark_data_smithi122_119671_object38696:head# 2024-03-20T09:04:48.257 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b26a8379:::benchmark_data_smithi122_119671_object73770:head# 2024-03-20T09:04:48.257 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b26adbfa:::benchmark_data_smithi122_119671_object89734:head# 2024-03-20T09:04:48.257 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b26b02a8:::benchmark_data_smithi122_119671_object23926:head# 2024-03-20T09:04:48.258 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b26ba7e2:::benchmark_data_smithi122_119671_object21518:head# 2024-03-20T09:04:48.258 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b26bab88:::benchmark_data_smithi122_119671_object9004:head# 2024-03-20T09:04:48.259 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b26d0e45:::benchmark_data_smithi122_119671_object86893:head# 2024-03-20T09:04:48.259 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b26d775c:::benchmark_data_smithi122_119671_object23327:head# 2024-03-20T09:04:48.259 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b26f3e36:::benchmark_data_smithi122_119671_object2179:head# 2024-03-20T09:04:48.260 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b26fa22a:::benchmark_data_smithi122_119671_object61677:head# 2024-03-20T09:04:48.260 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b271aaaf:::benchmark_data_smithi122_119671_object51655:head# 2024-03-20T09:04:48.261 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2733ab9:::benchmark_data_smithi122_119671_object30102:head# 2024-03-20T09:04:48.261 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2756ba7:::benchmark_data_smithi122_119671_object87701:head# 2024-03-20T09:04:48.261 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b276045f:::benchmark_data_smithi122_119671_object36962:head# 2024-03-20T09:04:48.262 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b276b9a9:::benchmark_data_smithi122_119671_object50154:head# 2024-03-20T09:04:48.262 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b27798be:::benchmark_data_smithi122_119671_object87744:head# 2024-03-20T09:04:48.263 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b277ec5d:::benchmark_data_smithi122_119671_object34128:head# 2024-03-20T09:04:48.263 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b278888c:::benchmark_data_smithi122_119671_object62179:head# 2024-03-20T09:04:48.263 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2796db3:::benchmark_data_smithi122_119671_object94873:head# 2024-03-20T09:04:48.264 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b27ab3f6:::benchmark_data_smithi122_119671_object95447:head# 2024-03-20T09:04:48.264 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b27aca48:::benchmark_data_smithi122_119671_object48507:head# 2024-03-20T09:04:48.265 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b27b6f22:::benchmark_data_smithi122_119671_object88777:head# 2024-03-20T09:04:48.265 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b27bc9c4:::benchmark_data_smithi122_119671_object18731:head# 2024-03-20T09:04:48.266 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b27c2a98:::benchmark_data_smithi122_119671_object53824:head# 2024-03-20T09:04:48.266 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b27c80c7:::benchmark_data_smithi122_119671_object7600:head# 2024-03-20T09:04:48.266 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b27c9d8a:::benchmark_data_smithi122_119671_object4025:head# 2024-03-20T09:04:48.267 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b27e2804:::benchmark_data_smithi122_119671_object9155:head# 2024-03-20T09:04:48.267 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b27e49ee:::benchmark_data_smithi122_119671_object20809:head# 2024-03-20T09:04:48.268 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b28126e5:::benchmark_data_smithi122_119671_object63258:head# 2024-03-20T09:04:48.268 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b281487b:::benchmark_data_smithi122_119671_object51476:head# 2024-03-20T09:04:48.269 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b281671f:::benchmark_data_smithi122_119671_object65528:head# 2024-03-20T09:04:48.269 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2822738:::benchmark_data_smithi122_119671_object81033:head# 2024-03-20T09:04:48.269 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2830752:::benchmark_data_smithi122_119671_object23598:head# 2024-03-20T09:04:48.270 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2832306:::benchmark_data_smithi122_119671_object68511:head# 2024-03-20T09:04:48.271 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2839a8b:::benchmark_data_smithi122_119671_object5206:head# 2024-03-20T09:04:48.271 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b283b3a0:::benchmark_data_smithi122_119671_object50464:head# 2024-03-20T09:04:48.272 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b283ee3c:::benchmark_data_smithi122_119671_object52371:head# 2024-03-20T09:04:48.272 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2844600:::benchmark_data_smithi122_119671_object42088:head# 2024-03-20T09:04:48.273 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2848ff0:::benchmark_data_smithi122_119671_object97978:head# 2024-03-20T09:04:48.273 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b28535b1:::benchmark_data_smithi122_119671_object32930:head# 2024-03-20T09:04:48.273 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b28564e5:::benchmark_data_smithi122_119671_object40028:head# 2024-03-20T09:04:48.273 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b285cf0d:::benchmark_data_smithi122_119671_object36407:head# 2024-03-20T09:04:48.274 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b28728dc:::benchmark_data_smithi122_119671_object65239:head# 2024-03-20T09:04:48.274 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b287592c:::benchmark_data_smithi122_119671_object76304:head# 2024-03-20T09:04:48.275 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2883378:::benchmark_data_smithi122_119671_object60853:head# 2024-03-20T09:04:48.275 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2887627:::benchmark_data_smithi122_119671_object89462:head# 2024-03-20T09:04:48.275 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b28914cf:::benchmark_data_smithi122_119671_object61285:head# 2024-03-20T09:04:48.276 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b28b05fb:::benchmark_data_smithi122_119671_object97026:head# 2024-03-20T09:04:48.276 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b28b452c:::benchmark_data_smithi122_119671_object42125:head# 2024-03-20T09:04:48.277 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b28b82a7:::benchmark_data_smithi122_119671_object19226:head# 2024-03-20T09:04:48.277 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b28c9456:::benchmark_data_smithi122_119671_object63560:head# 2024-03-20T09:04:48.278 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b28eed67:::benchmark_data_smithi122_119671_object58034:head# 2024-03-20T09:04:48.278 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b28f1a77:::benchmark_data_smithi122_119671_object66409:head# 2024-03-20T09:04:48.278 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b28f71f1:::benchmark_data_smithi122_119671_object3874:head# 2024-03-20T09:04:48.279 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b28ffbe9:::benchmark_data_smithi122_119671_object12749:head# 2024-03-20T09:04:48.280 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2907b7a:::benchmark_data_smithi122_119671_object78778:head# 2024-03-20T09:04:48.280 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2907e58:::benchmark_data_smithi122_119671_object67494:head# 2024-03-20T09:04:48.280 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b292967d:::benchmark_data_smithi122_119671_object89845:head# 2024-03-20T09:04:48.281 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2967960:::benchmark_data_smithi122_119671_object37674:head# 2024-03-20T09:04:48.281 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b296a1af:::benchmark_data_smithi122_119671_object34654:head# 2024-03-20T09:04:48.282 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2971b62:::benchmark_data_smithi122_119671_object47316:head# 2024-03-20T09:04:48.282 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b297a4ba:::benchmark_data_smithi122_119671_object22591:head# 2024-03-20T09:04:48.282 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b297b4be:::benchmark_data_smithi122_119671_object28067:head# 2024-03-20T09:04:48.283 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b297bee4:::benchmark_data_smithi122_119671_object59664:head# 2024-03-20T09:04:48.283 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2984913:::benchmark_data_smithi122_119671_object84576:head# 2024-03-20T09:04:48.284 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b298f043:::benchmark_data_smithi122_119671_object86640:head# 2024-03-20T09:04:48.284 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b298f896:::benchmark_data_smithi122_119671_object8111:head# 2024-03-20T09:04:48.285 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b29909c9:::benchmark_data_smithi122_119671_object17816:head# 2024-03-20T09:04:48.285 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b299acc8:::benchmark_data_smithi122_119671_object8933:head# 2024-03-20T09:04:48.285 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b299af40:::benchmark_data_smithi122_119671_object10364:head# 2024-03-20T09:04:48.286 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b29a8590:::benchmark_data_smithi122_119671_object36037:head# 2024-03-20T09:04:48.286 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b29b5742:::benchmark_data_smithi122_119671_object55628:head# 2024-03-20T09:04:48.287 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b29c9435:::benchmark_data_smithi122_119671_object79945:head# 2024-03-20T09:04:48.287 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b29cac20:::benchmark_data_smithi122_119671_object92843:head# 2024-03-20T09:04:48.288 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b29cd263:::benchmark_data_smithi122_119671_object82011:head# 2024-03-20T09:04:48.290 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b29e78c4:::benchmark_data_smithi122_119671_object12241:head# 2024-03-20T09:04:48.291 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b29f39e1:::benchmark_data_smithi122_119671_object54968:head# 2024-03-20T09:04:48.291 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b29f98e0:::benchmark_data_smithi122_119671_object56759:head# 2024-03-20T09:04:48.292 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2a051cb:::benchmark_data_smithi122_119671_object10167:head# 2024-03-20T09:04:48.292 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2a092d0:::benchmark_data_smithi122_119671_object6633:head# 2024-03-20T09:04:48.292 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2a3bb7d:::benchmark_data_smithi122_119671_object55057:head# 2024-03-20T09:04:48.293 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2a45119:::benchmark_data_smithi122_119671_object14434:head# 2024-03-20T09:04:48.293 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2a4a1e6:::benchmark_data_smithi122_119671_object7585:head# 2024-03-20T09:04:48.294 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2a4eb3e:::benchmark_data_smithi122_119671_object56233:head# 2024-03-20T09:04:48.294 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2a514cf:::benchmark_data_smithi122_119671_object90768:head# 2024-03-20T09:04:48.295 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2a5b9b5:::benchmark_data_smithi122_119671_object17213:head# 2024-03-20T09:04:48.295 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2a6ca94:::benchmark_data_smithi122_119671_object49212:head# 2024-03-20T09:04:48.296 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2a7a888:::benchmark_data_smithi122_119671_object97887:head# 2024-03-20T09:04:48.296 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2abadd9:::benchmark_data_smithi122_119671_object48231:head# 2024-03-20T09:04:48.297 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2abfb6f:::benchmark_data_smithi122_119671_object94536:head# 2024-03-20T09:04:48.297 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2ad52da:::benchmark_data_smithi122_119671_object67213:head# 2024-03-20T09:04:48.297 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2ad935a:::benchmark_data_smithi122_119671_object88246:head# 2024-03-20T09:04:48.298 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2ae0d34:::benchmark_data_smithi122_119671_object18568:head# 2024-03-20T09:04:48.298 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2ae38f5:::benchmark_data_smithi122_119671_object10693:head# 2024-03-20T09:04:48.299 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2ae3baf:::benchmark_data_smithi122_119671_object24827:head# 2024-03-20T09:04:48.299 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2ae543c:::benchmark_data_smithi122_119671_object81583:head# 2024-03-20T09:04:48.299 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2aea3ec:::benchmark_data_smithi122_119671_object51255:head# 2024-03-20T09:04:48.300 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2af2068:::benchmark_data_smithi122_119671_object84189:head# 2024-03-20T09:04:48.300 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:04:48.301 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2af8242:::benchmark_data_smithi122_119671_object48929:head# 2024-03-20T09:04:48.301 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2b02297:::benchmark_data_smithi122_119671_object34472:head# 2024-03-20T09:04:48.301 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2b02d9a:::benchmark_data_smithi122_119671_object88501:head# 2024-03-20T09:04:48.301 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2b0f1e8:::benchmark_data_smithi122_119671_object17534:head# 2024-03-20T09:04:48.302 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2b147de:::benchmark_data_smithi122_119671_object44727:head# 2024-03-20T09:04:48.302 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2b1fe15:::benchmark_data_smithi122_119671_object45111:head# 2024-03-20T09:04:48.303 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2b3a5fa:::benchmark_data_smithi122_119671_object56901:head# 2024-03-20T09:04:48.303 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2b55eb4:::benchmark_data_smithi122_119671_object29826:head# 2024-03-20T09:04:48.304 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2b75f78:::benchmark_data_smithi122_119671_object28183:head# 2024-03-20T09:04:48.304 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2b7ae98:::benchmark_data_smithi122_119671_object35468:head# 2024-03-20T09:04:48.305 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2b7ddb5:::benchmark_data_smithi122_119671_object82418:head# 2024-03-20T09:04:48.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:48.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:04:48.307 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2b88957:::benchmark_data_smithi122_119671_object88950:head# 2024-03-20T09:04:48.307 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2bae7b7:::benchmark_data_smithi122_119671_object39787:head# 2024-03-20T09:04:48.308 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2baed2e:::benchmark_data_smithi122_119671_object73561:head# 2024-03-20T09:04:48.308 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2bbcaf4:::benchmark_data_smithi122_119671_object99758:head# 2024-03-20T09:04:48.309 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2bd11a8:::benchmark_data_smithi122_119671_object29829:head# 2024-03-20T09:04:48.309 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2bd1a45:::benchmark_data_smithi122_119671_object96197:head# 2024-03-20T09:04:48.309 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2bd2841:::benchmark_data_smithi122_119671_object76954:head# 2024-03-20T09:04:48.310 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2bd2bfd:::benchmark_data_smithi122_119671_object41215:head# 2024-03-20T09:04:48.310 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2bd4d1d:::benchmark_data_smithi122_119671_object88662:head# 2024-03-20T09:04:48.311 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2be30d1:::benchmark_data_smithi122_119671_object87652:head# 2024-03-20T09:04:48.311 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2beccf2:::benchmark_data_smithi122_119671_object58254:head# 2024-03-20T09:04:48.312 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2c04319:::benchmark_data_smithi122_119671_object9419:head# 2024-03-20T09:04:48.312 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2c09f01:::benchmark_data_smithi122_119671_object66337:head# 2024-03-20T09:04:48.313 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2c245bd:::benchmark_data_smithi122_119671_object41819:head# 2024-03-20T09:04:48.313 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2c25d52:::benchmark_data_smithi122_119671_object4918:head# 2024-03-20T09:04:48.313 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2c41ba3:::benchmark_data_smithi122_119671_object63298:head# 2024-03-20T09:04:48.314 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2c458ff:::benchmark_data_smithi122_119671_object4690:head# 2024-03-20T09:04:48.314 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2c4a852:::benchmark_data_smithi122_119671_object78248:head# 2024-03-20T09:04:48.315 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2c56c90:::benchmark_data_smithi122_119671_object62273:head# 2024-03-20T09:04:48.315 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2c65716:::benchmark_data_smithi122_119671_object68174:head# 2024-03-20T09:04:48.316 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2c681cb:::benchmark_data_smithi122_119671_object50316:head# 2024-03-20T09:04:48.316 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2c79261:::benchmark_data_smithi122_119671_object99576:head# 2024-03-20T09:04:48.316 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2c7d9fc:::benchmark_data_smithi122_119671_object99318:head# 2024-03-20T09:04:48.317 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2c8486e:::benchmark_data_smithi122_119671_object11463:head# 2024-03-20T09:04:48.317 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2c92f94:::benchmark_data_smithi122_119671_object33561:head# 2024-03-20T09:04:48.318 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2ca6fd8:::benchmark_data_smithi122_119671_object25268:head# 2024-03-20T09:04:48.318 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2cbaeaa:::benchmark_data_smithi122_119671_object58902:head# 2024-03-20T09:04:48.318 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2cd55c9:::benchmark_data_smithi122_119671_object53550:head# 2024-03-20T09:04:48.319 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2ce8e9e:::benchmark_data_smithi122_119671_object98693:head# 2024-03-20T09:04:48.319 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2cebe52:::benchmark_data_smithi122_119671_object63146:head# 2024-03-20T09:04:48.321 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2d03082:::benchmark_data_smithi122_119671_object88733:head# 2024-03-20T09:04:48.322 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2d15cc8:::benchmark_data_smithi122_119671_object57726:head# 2024-03-20T09:04:48.322 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2d1cee5:::benchmark_data_smithi122_119671_object48404:head# 2024-03-20T09:04:48.322 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2d301de:::benchmark_data_smithi122_119671_object57276:head# 2024-03-20T09:04:48.322 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2d33c30:::benchmark_data_smithi122_119671_object86879:head# 2024-03-20T09:04:48.322 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2d35c8f:::benchmark_data_smithi122_119671_object30101:head# 2024-03-20T09:04:48.322 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2d3c745:::benchmark_data_smithi122_119671_object77184:head# 2024-03-20T09:04:48.323 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2d5fd9d:::benchmark_data_smithi122_119671_object6237:head# 2024-03-20T09:04:48.325 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2d6e133:::benchmark_data_smithi122_119671_object72548:head# 2024-03-20T09:04:48.326 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2d8b121:::benchmark_data_smithi122_119671_object71324:head# 2024-03-20T09:04:48.326 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2d97113:::benchmark_data_smithi122_119671_object9288:head# 2024-03-20T09:04:48.326 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2dadd73:::benchmark_data_smithi122_119671_object49359:head# 2024-03-20T09:04:48.326 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2db58ba:::benchmark_data_smithi122_119671_object82285:head# 2024-03-20T09:04:48.326 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2db6632:::benchmark_data_smithi122_119671_object17100:head# 2024-03-20T09:04:48.326 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2dcba1f:::benchmark_data_smithi122_119671_object79162:head# 2024-03-20T09:04:48.327 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2dd641c:::benchmark_data_smithi122_119671_object92042:head# 2024-03-20T09:04:48.327 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2ddde11:::benchmark_data_smithi122_119671_object58030:head# 2024-03-20T09:04:48.328 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2de22a5:::benchmark_data_smithi122_119671_object12495:head# 2024-03-20T09:04:48.328 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2de4c04:::benchmark_data_smithi122_119671_object81134:head# 2024-03-20T09:04:48.328 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2def9f9:::benchmark_data_smithi122_119671_object99398:head# 2024-03-20T09:04:48.328 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2dfa72f:::benchmark_data_smithi122_119671_object65832:head# 2024-03-20T09:04:48.330 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2dfb7d1:::benchmark_data_smithi122_119671_object41899:head# 2024-03-20T09:04:48.330 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2e00a6a:::benchmark_data_smithi122_119671_object27179:head# 2024-03-20T09:04:48.330 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2e03105:::benchmark_data_smithi122_119671_object21912:head# 2024-03-20T09:04:48.330 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2e09fe5:::benchmark_data_smithi122_119671_object76208:head# 2024-03-20T09:04:48.330 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2e0a52f:::benchmark_data_smithi122_119671_object18696:head# 2024-03-20T09:04:48.331 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2e1e854:::benchmark_data_smithi122_119671_object90686:head# 2024-03-20T09:04:48.331 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2e25ea8:::benchmark_data_smithi122_119671_object56351:head# 2024-03-20T09:04:48.331 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2e3cdd6:::benchmark_data_smithi122_119671_object4339:head# 2024-03-20T09:04:48.331 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2e4206c:::benchmark_data_smithi122_119671_object10646:head# 2024-03-20T09:04:48.332 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2e46778:::benchmark_data_smithi122_119671_object65360:head# 2024-03-20T09:04:48.332 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2e4bb4f:::benchmark_data_smithi122_119671_object99649:head# 2024-03-20T09:04:48.332 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2e5b6a3:::benchmark_data_smithi122_119671_object5339:head# 2024-03-20T09:04:48.333 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2e5e4b9:::benchmark_data_smithi122_119671_object40723:head# 2024-03-20T09:04:48.333 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2e6421c:::benchmark_data_smithi122_119671_object83575:head# 2024-03-20T09:04:48.334 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2e89cf7:::benchmark_data_smithi122_119671_object80862:head# 2024-03-20T09:04:48.334 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2e8e64a:::benchmark_data_smithi122_119671_object86630:head# 2024-03-20T09:04:48.334 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2e90497:::benchmark_data_smithi122_119671_object23758:head# 2024-03-20T09:04:48.335 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2e932a1:::benchmark_data_smithi122_119671_object91801:head# 2024-03-20T09:04:48.335 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2ea5663:::benchmark_data_smithi122_119671_object55530:head# 2024-03-20T09:04:48.336 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2eb2163:::benchmark_data_smithi122_119671_object89553:head# 2024-03-20T09:04:48.336 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2ebadf0:::benchmark_data_smithi122_119671_object24110:head# 2024-03-20T09:04:48.337 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2ebb3b5:::benchmark_data_smithi122_119671_object2705:head# 2024-03-20T09:04:48.337 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2ec5fc6:::benchmark_data_smithi122_119671_object85721:head# 2024-03-20T09:04:48.337 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2ec9ed0:::benchmark_data_smithi122_119671_object88032:head# 2024-03-20T09:04:48.338 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2ecab53:::benchmark_data_smithi122_119671_object15097:head# 2024-03-20T09:04:48.338 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2ed6ea1:::benchmark_data_smithi122_119671_object40474:head# 2024-03-20T09:04:48.339 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2ed728a:::benchmark_data_smithi122_119671_object95613:head# 2024-03-20T09:04:48.339 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2ef14a5:::benchmark_data_smithi122_119671_object60731:head# 2024-03-20T09:04:48.339 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2effe13:::benchmark_data_smithi122_119671_object66153:head# 2024-03-20T09:04:48.340 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2f11d36:::benchmark_data_smithi122_119671_object28831:head# 2024-03-20T09:04:48.340 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2f1e4d2:::benchmark_data_smithi122_119671_object52263:head# 2024-03-20T09:04:48.341 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2f253ab:::benchmark_data_smithi122_119671_object28148:head# 2024-03-20T09:04:48.341 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2f270da:::benchmark_data_smithi122_119671_object82643:head# 2024-03-20T09:04:48.342 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2f36192:::benchmark_data_smithi122_119671_object87521:head# 2024-03-20T09:04:48.342 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2f3dcbf:::benchmark_data_smithi122_119671_object49607:head# 2024-03-20T09:04:48.342 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2f578e3:::benchmark_data_smithi122_119671_object90375:head# 2024-03-20T09:04:48.343 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2f5fb16:::benchmark_data_smithi122_119671_object17185:head# 2024-03-20T09:04:48.343 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2f6631c:::benchmark_data_smithi122_119671_object73527:head# 2024-03-20T09:04:48.344 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2f69cd9:::benchmark_data_smithi122_119671_object87963:head# 2024-03-20T09:04:48.344 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2f7f63c:::benchmark_data_smithi122_119671_object89694:head# 2024-03-20T09:04:48.344 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2f823b2:::benchmark_data_smithi122_119671_object12349:head# 2024-03-20T09:04:48.345 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2f934d5:::benchmark_data_smithi122_119671_object26777:head# 2024-03-20T09:04:48.345 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2f9619b:::benchmark_data_smithi122_119671_object16775:head# 2024-03-20T09:04:48.346 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2fa72b9:::benchmark_data_smithi122_119671_object43918:head# 2024-03-20T09:04:48.346 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2faba73:::benchmark_data_smithi122_119671_object65920:head# 2024-03-20T09:04:48.347 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2fb5c48:::benchmark_data_smithi122_119671_object20898:head# 2024-03-20T09:04:48.347 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2fc7113:::benchmark_data_smithi122_119671_object8235:head# 2024-03-20T09:04:48.347 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2fd23b1:::benchmark_data_smithi122_119671_object12509:head# 2024-03-20T09:04:48.348 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2fd9620:::benchmark_data_smithi122_119671_object65048:head# 2024-03-20T09:04:48.348 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b2fef3d0:::benchmark_data_smithi122_119671_object82274:head# 2024-03-20T09:04:48.349 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b301b66c:::benchmark_data_smithi122_119671_object3261:head# 2024-03-20T09:04:48.349 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b301c795:::benchmark_data_smithi122_119671_object35805:head# 2024-03-20T09:04:48.350 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3028623:::benchmark_data_smithi122_119671_object37361:head# 2024-03-20T09:04:48.350 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3036cb7:::benchmark_data_smithi122_119671_object33184:head# 2024-03-20T09:04:48.350 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b303933f:::benchmark_data_smithi122_119671_object61398:head# 2024-03-20T09:04:48.351 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b30453d3:::benchmark_data_smithi122_119671_object79779:head# 2024-03-20T09:04:48.351 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3061895:::benchmark_data_smithi122_119671_object38015:head# 2024-03-20T09:04:48.352 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b306ba93:::benchmark_data_smithi122_119671_object6530:head# 2024-03-20T09:04:48.352 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b307b3d5:::benchmark_data_smithi122_119671_object57479:head# 2024-03-20T09:04:48.353 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3081207:::benchmark_data_smithi122_119671_object69500:head# 2024-03-20T09:04:48.353 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b309e32a:::benchmark_data_smithi122_119671_object26258:head# 2024-03-20T09:04:48.354 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b30a67dd:::benchmark_data_smithi122_119671_object99449:head# 2024-03-20T09:04:48.354 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b30ba6f1:::benchmark_data_smithi122_119671_object60608:head# 2024-03-20T09:04:48.355 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b30e0839:::benchmark_data_smithi122_119671_object17552:head# 2024-03-20T09:04:48.355 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b30e4394:::benchmark_data_smithi122_119671_object65658:head# 2024-03-20T09:04:48.355 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b30e6311:::benchmark_data_smithi122_119671_object61492:head# 2024-03-20T09:04:48.356 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b30ec037:::benchmark_data_smithi122_119671_object35824:head# 2024-03-20T09:04:48.356 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b30ec409:::benchmark_data_smithi122_119671_object94359:head# 2024-03-20T09:04:48.357 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b30fbf36:::benchmark_data_smithi122_119671_object93060:head# 2024-03-20T09:04:48.357 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3117842:::benchmark_data_smithi122_119671_object40503:head# 2024-03-20T09:04:48.359 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3128d67:::benchmark_data_smithi122_119671_object46164:head# 2024-03-20T09:04:48.359 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b312f968:::benchmark_data_smithi122_119671_object53362:head# 2024-03-20T09:04:48.360 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3136911:::benchmark_data_smithi122_119671_object37371:head# 2024-03-20T09:04:48.360 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3138ed1:::benchmark_data_smithi122_119671_object47258:head# 2024-03-20T09:04:48.360 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b316a53e:::benchmark_data_smithi122_119671_object57950:head# 2024-03-20T09:04:48.361 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b316fbcd:::benchmark_data_smithi122_119671_object1903:head# 2024-03-20T09:04:48.361 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b317831f:::benchmark_data_smithi122_119671_object52532:head# 2024-03-20T09:04:48.362 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b317cb7e:::benchmark_data_smithi122_119671_object12917:head# 2024-03-20T09:04:48.362 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3184647:::benchmark_data_smithi122_119671_object68686:head# 2024-03-20T09:04:48.363 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3185dfd:::benchmark_data_smithi122_119671_object91019:head# 2024-03-20T09:04:48.363 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3191027:::benchmark_data_smithi122_119671_object78529:head# 2024-03-20T09:04:48.364 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b31978f2:::benchmark_data_smithi122_119671_object11388:head# 2024-03-20T09:04:48.365 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b319a93c:::benchmark_data_smithi122_119671_object19655:head# 2024-03-20T09:04:48.365 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b319caa3:::benchmark_data_smithi122_119671_object53991:head# 2024-03-20T09:04:48.366 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b31abf16:::benchmark_data_smithi122_119671_object36468:head# 2024-03-20T09:04:48.366 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b31b0d8d:::benchmark_data_smithi122_119671_object72879:head# 2024-03-20T09:04:48.367 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b31b460b:::benchmark_data_smithi122_119671_object54303:head# 2024-03-20T09:04:48.367 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b31c1210:::benchmark_data_smithi122_119671_object24079:head# 2024-03-20T09:04:48.367 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b31c4692:::benchmark_data_smithi122_119671_object67631:head# 2024-03-20T09:04:48.368 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b31ca9dc:::benchmark_data_smithi122_119671_object27857:head# 2024-03-20T09:04:48.369 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b31d71a0:::benchmark_data_smithi122_119671_object36946:head# 2024-03-20T09:04:48.369 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b31defa6:::benchmark_data_smithi122_119671_object15391:head# 2024-03-20T09:04:48.369 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b31ecd99:::benchmark_data_smithi122_119671_object22650:head# 2024-03-20T09:04:48.370 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b31fa148:::benchmark_data_smithi122_119671_object25840:head# 2024-03-20T09:04:48.370 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b31faaa3:::benchmark_data_smithi122_119671_object86438:head# 2024-03-20T09:04:48.371 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3212905:::benchmark_data_smithi122_119671_object27580:head# 2024-03-20T09:04:48.371 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3217fd7:::benchmark_data_smithi122_119671_object56538:head# 2024-03-20T09:04:48.372 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b321f413:::benchmark_data_smithi122_119671_object41670:head# 2024-03-20T09:04:48.372 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3226a3e:::benchmark_data_smithi122_119671_object25620:head# 2024-03-20T09:04:48.373 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b323aa07:::benchmark_data_smithi122_119671_object51238:head# 2024-03-20T09:04:48.373 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b323d5b7:::benchmark_data_smithi122_119671_object60981:head# 2024-03-20T09:04:48.374 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b323ded7:::benchmark_data_smithi122_119671_object75229:head# 2024-03-20T09:04:48.374 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3247ea9:::benchmark_data_smithi122_119671_object46120:head# 2024-03-20T09:04:48.375 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b324da61:::benchmark_data_smithi122_119671_object84466:head# 2024-03-20T09:04:48.375 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3279d58:::benchmark_data_smithi122_119671_object33667:head# 2024-03-20T09:04:48.376 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3279fcc:::benchmark_data_smithi122_119671_object18203:head# 2024-03-20T09:04:48.376 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3283b77:::benchmark_data_smithi122_119671_object98235:head# 2024-03-20T09:04:48.376 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b32a2783:::benchmark_data_smithi122_119671_object48083:head# 2024-03-20T09:04:48.377 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b32a3cf3:::benchmark_data_smithi122_119671_object18604:head# 2024-03-20T09:04:48.378 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b32b40b2:::benchmark_data_smithi122_119671_object7497:head# 2024-03-20T09:04:48.378 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b32c2b7b:::benchmark_data_smithi122_119671_object51843:head# 2024-03-20T09:04:48.378 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b32c3611:::benchmark_data_smithi122_119671_object73212:head# 2024-03-20T09:04:48.379 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b32eeb45:::benchmark_data_smithi122_119671_object32324:head# 2024-03-20T09:04:48.379 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b330ff24:::benchmark_data_smithi122_119671_object58813:head# 2024-03-20T09:04:48.379 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3315bce:::benchmark_data_smithi122_119671_object56943:head# 2024-03-20T09:04:48.380 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3318d33:::benchmark_data_smithi122_119671_object86813:head# 2024-03-20T09:04:48.380 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b331b559:::benchmark_data_smithi122_119671_object34809:head# 2024-03-20T09:04:48.381 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b332a91c:::benchmark_data_smithi122_119671_object69054:head# 2024-03-20T09:04:48.381 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b332aa38:::benchmark_data_smithi122_119671_object60477:head# 2024-03-20T09:04:48.382 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b332ca9d:::benchmark_data_smithi122_119671_object30743:head# 2024-03-20T09:04:48.382 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3336077:::benchmark_data_smithi122_119671_object64546:head# 2024-03-20T09:04:48.382 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b333c4fb:::benchmark_data_smithi122_119671_object97811:head# 2024-03-20T09:04:48.383 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3340c2e:::benchmark_data_smithi122_119671_object91371:head# 2024-03-20T09:04:48.383 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b334aa7b:::benchmark_data_smithi122_119671_object76991:head# 2024-03-20T09:04:48.383 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b334eb35:::benchmark_data_smithi122_119671_object71193:head# 2024-03-20T09:04:48.384 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3352209:::benchmark_data_smithi122_119671_object14678:head# 2024-03-20T09:04:48.384 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3354d60:::benchmark_data_smithi122_119671_object9557:head# 2024-03-20T09:04:48.385 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3356a5a:::benchmark_data_smithi122_119671_object99059:head# 2024-03-20T09:04:48.385 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3367f0d:::benchmark_data_smithi122_119671_object97824:head# 2024-03-20T09:04:48.386 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b336eae0:::benchmark_data_smithi122_119671_object8472:head# 2024-03-20T09:04:48.386 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3375741:::benchmark_data_smithi122_119671_object48687:head# 2024-03-20T09:04:48.387 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b337cf95:::benchmark_data_smithi122_119671_object52556:head# 2024-03-20T09:04:48.387 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b338308b:::benchmark_data_smithi122_119671_object83263:head# 2024-03-20T09:04:48.388 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b33864a2:::benchmark_data_smithi122_119671_object98026:head# 2024-03-20T09:04:48.388 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3397acd:::benchmark_data_smithi122_119671_object68043:head# 2024-03-20T09:04:48.389 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b33ab69f:::benchmark_data_smithi122_119671_object66486:head# 2024-03-20T09:04:48.389 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b33b8895:::benchmark_data_smithi122_119671_object13177:head# 2024-03-20T09:04:48.390 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b33b8903:::benchmark_data_smithi122_119671_object53398:head# 2024-03-20T09:04:48.390 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b33b95d6:::benchmark_data_smithi122_119671_object71554:head# 2024-03-20T09:04:48.391 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b33c0298:::benchmark_data_smithi122_119671_object18598:head# 2024-03-20T09:04:48.391 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b33c3aa7:::benchmark_data_smithi122_119671_object16944:head# 2024-03-20T09:04:48.391 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b33cd44f:::benchmark_data_smithi122_119671_object52320:head# 2024-03-20T09:04:48.392 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b33df655:::benchmark_data_smithi122_119671_object39159:head# 2024-03-20T09:04:48.394 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b33e7fd9:::benchmark_data_smithi122_119671_object82483:head# 2024-03-20T09:04:48.395 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b33f1a2b:::benchmark_data_smithi122_119671_object94714:head# 2024-03-20T09:04:48.395 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3403174:::benchmark_data_smithi122_119671_object4629:head# 2024-03-20T09:04:48.396 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b340fe06:::benchmark_data_smithi122_119671_object84557:head# 2024-03-20T09:04:48.396 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3411ac7:::benchmark_data_smithi122_119671_object84952:head# 2024-03-20T09:04:48.397 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3422b57:::benchmark_data_smithi122_119671_object66994:head# 2024-03-20T09:04:48.397 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3434ab4:::benchmark_data_smithi122_119671_object89493:head# 2024-03-20T09:04:48.397 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b343d424:::benchmark_data_smithi122_119671_object75963:head# 2024-03-20T09:04:48.398 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3444aaa:::benchmark_data_smithi122_119671_object40403:head# 2024-03-20T09:04:48.398 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3446d07:::benchmark_data_smithi122_119671_object73644:head# 2024-03-20T09:04:48.399 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b346f3d0:::benchmark_data_smithi122_119671_object30008:head# 2024-03-20T09:04:48.399 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b347f87c:::benchmark_data_smithi122_119671_object95335:head# 2024-03-20T09:04:48.399 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b349a6ee:::benchmark_data_smithi122_119671_object87664:head# 2024-03-20T09:04:48.399 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b349dd29:::benchmark_data_smithi122_119671_object95000:head# 2024-03-20T09:04:48.403 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b349e322:::benchmark_data_smithi122_119671_object70493:head# 2024-03-20T09:04:48.403 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b349f399:::benchmark_data_smithi122_119671_object68270:head# 2024-03-20T09:04:48.404 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b34a2ec0:::benchmark_data_smithi122_119671_object51111:head# 2024-03-20T09:04:48.405 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b34a3cfb:::benchmark_data_smithi122_119671_object35626:head# 2024-03-20T09:04:48.405 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b34a5456:::benchmark_data_smithi122_119671_object2111:head# 2024-03-20T09:04:48.405 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b34ab85e:::benchmark_data_smithi122_119671_object51926:head# 2024-03-20T09:04:48.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:47 smithi115 ceph-mon[72476]: 1.3 scrub starts 2024-03-20T09:04:48.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:47 smithi115 ceph-mon[72476]: 1.3 scrub ok 2024-03-20T09:04:48.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:47 smithi115 ceph-mon[72476]: pgmap v729: 47 pgs: 1 active+clean+remapped, 1 activating+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+undersized+degraded+remapped, 29 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 6.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 28599/295467 objects degraded (9.679%); 13490/295467 objects misplaced (4.566%); 36 MiB/s, 590 objects/s recovering 2024-03-20T09:04:48.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:47 smithi115 ceph-mon[72476]: 3.19 scrub starts 2024-03-20T09:04:48.406 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b34b2813:::benchmark_data_smithi122_119671_object3611:head# 2024-03-20T09:04:48.406 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b34b8aa1:::benchmark_data_smithi122_119671_object5613:head# 2024-03-20T09:04:48.406 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b34c23bc:::benchmark_data_smithi122_119671_object88142:head# 2024-03-20T09:04:48.407 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:04:48.408 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b34ce07d:::benchmark_data_smithi122_119671_object44373:head# 2024-03-20T09:04:48.408 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b34cf86a:::benchmark_data_smithi122_119671_object93523:head# 2024-03-20T09:04:48.408 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b34d3ef4:::benchmark_data_smithi122_119671_object11560:head# 2024-03-20T09:04:48.409 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b34e3a82:::benchmark_data_smithi122_119671_object26482:head# 2024-03-20T09:04:48.409 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b34f0f83:::benchmark_data_smithi122_119671_object93282:head# 2024-03-20T09:04:48.409 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b34fa239:::benchmark_data_smithi122_119671_object59251:head# 2024-03-20T09:04:48.409 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b350d8df:::benchmark_data_smithi122_119671_object99444:head# 2024-03-20T09:04:48.411 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3512db9:::benchmark_data_smithi122_119671_object82782:head# 2024-03-20T09:04:48.411 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3512fcc:::benchmark_data_smithi122_119671_object77550:head# 2024-03-20T09:04:48.411 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3518f52:::benchmark_data_smithi122_119671_object62557:head# 2024-03-20T09:04:48.411 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3524a86:::benchmark_data_smithi122_119671_object17527:head# 2024-03-20T09:04:48.412 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b35268c0:::benchmark_data_smithi122_119671_object19867:head# 2024-03-20T09:04:48.412 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3531ad1:::benchmark_data_smithi122_119671_object89637:head# 2024-03-20T09:04:48.412 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3531b5d:::benchmark_data_smithi122_119671_object46263:head# 2024-03-20T09:04:48.413 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3540b17:::benchmark_data_smithi122_119671_object88819:head# 2024-03-20T09:04:48.413 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b355349b:::benchmark_data_smithi122_119671_object68450:head# 2024-03-20T09:04:48.414 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b355623e:::benchmark_data_smithi122_119671_object71676:head# 2024-03-20T09:04:48.414 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3563035:::benchmark_data_smithi122_119671_object89198:head# 2024-03-20T09:04:48.415 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b35634bd:::benchmark_data_smithi122_119671_object87520:head# 2024-03-20T09:04:48.415 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3567935:::benchmark_data_smithi122_119671_object29548:head# 2024-03-20T09:04:48.416 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3571fc7:::benchmark_data_smithi122_119671_object74222:head# 2024-03-20T09:04:48.418 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3579d09:::benchmark_data_smithi122_119671_object64095:head# 2024-03-20T09:04:48.419 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b35867da:::benchmark_data_smithi122_119671_object7948:head# 2024-03-20T09:04:48.419 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3587684:::benchmark_data_smithi122_119671_object15978:head# 2024-03-20T09:04:48.420 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b35bf10c:::benchmark_data_smithi122_119671_object51867:head# 2024-03-20T09:04:48.420 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b35bf8cd:::benchmark_data_smithi122_119671_object24892:head# 2024-03-20T09:04:48.420 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b35c4b30:::benchmark_data_smithi122_119671_object1192:head# 2024-03-20T09:04:48.421 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b35c5551:::benchmark_data_smithi122_119671_object45656:head# 2024-03-20T09:04:48.422 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b35d918d:::benchmark_data_smithi122_119671_object48174:head# 2024-03-20T09:04:48.422 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b35e166a:::benchmark_data_smithi122_119671_object85712:head# 2024-03-20T09:04:48.423 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b35fd4fe:::benchmark_data_smithi122_119671_object47605:head# 2024-03-20T09:04:48.423 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b361d576:::benchmark_data_smithi122_119671_object81575:head# 2024-03-20T09:04:48.423 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3635267:::benchmark_data_smithi122_119671_object3188:head# 2024-03-20T09:04:48.424 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3637f23:::benchmark_data_smithi122_119671_object64493:head# 2024-03-20T09:04:48.424 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3641b9c:::benchmark_data_smithi122_119671_object40221:head# 2024-03-20T09:04:48.424 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b365c09d:::benchmark_data_smithi122_119671_object22634:head# 2024-03-20T09:04:48.427 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3669efb:::benchmark_data_smithi122_119671_object86017:head# 2024-03-20T09:04:48.428 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b366a264:::benchmark_data_smithi122_119671_object53764:head# 2024-03-20T09:04:48.428 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b366bd14:::benchmark_data_smithi122_119671_object12606:head# 2024-03-20T09:04:48.430 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b36807f3:::benchmark_data_smithi122_119671_object81879:head# 2024-03-20T09:04:48.430 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b368cb15:::benchmark_data_smithi122_119671_object62311:head# 2024-03-20T09:04:48.433 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3691ed1:::benchmark_data_smithi122_119671_object16019:head# 2024-03-20T09:04:48.433 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3695d48:::benchmark_data_smithi122_119671_object62698:head# 2024-03-20T09:04:48.433 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b369921b:::benchmark_data_smithi122_119671_object28747:head# 2024-03-20T09:04:48.433 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3699b1e:::benchmark_data_smithi122_119671_object51543:head# 2024-03-20T09:04:48.433 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3699e7a:::benchmark_data_smithi122_119671_object13576:head# 2024-03-20T09:04:48.434 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b369dc98:::benchmark_data_smithi122_119671_object84759:head# 2024-03-20T09:04:48.434 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b36a4562:::benchmark_data_smithi122_119671_object63095:head# 2024-03-20T09:04:48.434 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b36ac56c:::benchmark_data_smithi122_119671_object43933:head# 2024-03-20T09:04:48.434 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b36b694e:::benchmark_data_smithi122_119671_object79345:head# 2024-03-20T09:04:48.434 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b36c71bf:::benchmark_data_smithi122_119671_object19095:head# 2024-03-20T09:04:48.435 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:48.436 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b36d7678:::benchmark_data_smithi122_119671_object19950:head# 2024-03-20T09:04:48.436 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b36de627:::benchmark_data_smithi122_119671_object51231:head# 2024-03-20T09:04:48.436 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b36e1230:::benchmark_data_smithi122_119671_object45935:head# 2024-03-20T09:04:48.437 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b37019d4:::benchmark_data_smithi122_119671_object68046:head# 2024-03-20T09:04:48.437 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3707b5d:::benchmark_data_smithi122_119671_object81971:head# 2024-03-20T09:04:48.437 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b371a541:::benchmark_data_smithi122_119671_object24468:head# 2024-03-20T09:04:48.438 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3723285:::benchmark_data_smithi122_119671_object21989:head# 2024-03-20T09:04:48.438 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b372d2b3:::benchmark_data_smithi122_119671_object23021:head# 2024-03-20T09:04:48.438 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3743cc3:::benchmark_data_smithi122_119671_object54739:head# 2024-03-20T09:04:48.439 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3760f37:::benchmark_data_smithi122_119671_object43019:head# 2024-03-20T09:04:48.439 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3763fc6:::benchmark_data_smithi122_119671_object48320:head# 2024-03-20T09:04:48.440 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b376b862:::benchmark_data_smithi122_119671_object7002:head# 2024-03-20T09:04:48.441 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b376ed3d:::benchmark_data_smithi122_119671_object29958:head# 2024-03-20T09:04:48.441 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b377a913:::benchmark_data_smithi122_119671_object10608:head# 2024-03-20T09:04:48.441 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3797262:::benchmark_data_smithi122_119671_object80286:head# 2024-03-20T09:04:48.442 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b379fa93:::benchmark_data_smithi122_119671_object43520:head# 2024-03-20T09:04:48.442 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b37a0ce0:::benchmark_data_smithi122_119671_object9227:head# 2024-03-20T09:04:48.442 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b37bae1f:::benchmark_data_smithi122_119671_object53541:head# 2024-03-20T09:04:48.443 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b37c1b32:::benchmark_data_smithi122_119671_object15185:head# 2024-03-20T09:04:48.444 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b37d0841:::benchmark_data_smithi122_119671_object13843:head# 2024-03-20T09:04:48.445 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b37e5110:::benchmark_data_smithi122_119671_object52043:head# 2024-03-20T09:04:48.446 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b37f6ef2:::benchmark_data_smithi122_119671_object2567:head# 2024-03-20T09:04:48.446 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b37f88ed:::benchmark_data_smithi122_119671_object23899:head# 2024-03-20T09:04:48.446 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b380b81a:::benchmark_data_smithi122_119671_object89815:head# 2024-03-20T09:04:48.446 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b38300da:::benchmark_data_smithi122_119671_object45982:head# 2024-03-20T09:04:48.447 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b383205c:::benchmark_data_smithi122_119671_object21098:head# 2024-03-20T09:04:48.448 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3836132:::benchmark_data_smithi122_119671_object96937:head# 2024-03-20T09:04:48.448 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b38386f1:::benchmark_data_smithi122_119671_object15388:head# 2024-03-20T09:04:48.448 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b383bd10:::benchmark_data_smithi122_119671_object83306:head# 2024-03-20T09:04:48.448 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3846db9:::benchmark_data_smithi122_119671_object44225:head# 2024-03-20T09:04:48.448 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3848ebe:::benchmark_data_smithi122_119671_object97581:head# 2024-03-20T09:04:48.448 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b384b18e:::benchmark_data_smithi122_119671_object3566:head# 2024-03-20T09:04:48.449 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b38519e4:::benchmark_data_smithi122_119671_object97078:head# 2024-03-20T09:04:48.449 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b385ee66:::benchmark_data_smithi122_119671_object61914:head# 2024-03-20T09:04:48.450 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b387911d:::benchmark_data_smithi122_119671_object69886:head# 2024-03-20T09:04:48.450 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b388a2eb:::benchmark_data_smithi122_119671_object58741:head# 2024-03-20T09:04:48.451 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b38a1720:::benchmark_data_smithi122_119671_object1761:head# 2024-03-20T09:04:48.452 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b38a70e2:::benchmark_data_smithi122_119671_object48987:head# 2024-03-20T09:04:48.452 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b38aa894:::benchmark_data_smithi122_119671_object38900:head# 2024-03-20T09:04:48.452 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b38aa910:::benchmark_data_smithi122_119671_object62909:head# 2024-03-20T09:04:48.453 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b38ab00a:::benchmark_data_smithi122_119671_object26416:head# 2024-03-20T09:04:48.453 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b38be173:::benchmark_data_smithi122_119671_object94649:head# 2024-03-20T09:04:48.453 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b38fe79e:::benchmark_data_smithi122_119671_object81230:head# 2024-03-20T09:04:48.453 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3906b0f:::benchmark_data_smithi122_119671_object67818:head# 2024-03-20T09:04:48.454 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b391879e:::benchmark_data_smithi122_119671_object49014:head# 2024-03-20T09:04:48.454 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b392946e:::benchmark_data_smithi122_119671_object31055:head# 2024-03-20T09:04:48.455 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b39332c2:::benchmark_data_smithi122_119671_object16357:head# 2024-03-20T09:04:48.455 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b39343a7:::benchmark_data_smithi122_119671_object32986:head# 2024-03-20T09:04:48.456 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3939911:::benchmark_data_smithi122_119671_object92622:head# 2024-03-20T09:04:48.456 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:04:48.458 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3944600:::benchmark_data_smithi122_119671_object46402:head# 2024-03-20T09:04:48.458 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b39467a3:::benchmark_data_smithi122_119671_object34563:head# 2024-03-20T09:04:48.459 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3951a17:::benchmark_data_smithi122_119671_object67913:head# 2024-03-20T09:04:48.459 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b39591d1:::benchmark_data_smithi122_119671_object94459:head# 2024-03-20T09:04:48.460 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3961feb:::benchmark_data_smithi122_119671_object8794:head# 2024-03-20T09:04:48.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:48.461 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:04:48.463 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3968641:::benchmark_data_smithi122_119671_object23680:head# 2024-03-20T09:04:48.463 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b396f49f:::benchmark_data_smithi122_119671_object48600:head# 2024-03-20T09:04:48.463 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b397a6f5:::benchmark_data_smithi122_119671_object88752:head# 2024-03-20T09:04:48.463 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3983ec6:::benchmark_data_smithi122_119671_object27786:head# 2024-03-20T09:04:48.463 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b398ae9b:::benchmark_data_smithi122_119671_object89507:head# 2024-03-20T09:04:48.463 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b398f707:::benchmark_data_smithi122_119671_object39450:head# 2024-03-20T09:04:48.464 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b399026b:::benchmark_data_smithi122_119671_object48616:head# 2024-03-20T09:04:48.467 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3994c17:::benchmark_data_smithi122_119671_object62837:head# 2024-03-20T09:04:48.467 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b399c976:::benchmark_data_smithi122_119671_object54118:head# 2024-03-20T09:04:48.467 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b39af048:::benchmark_data_smithi122_119671_object60545:head# 2024-03-20T09:04:48.467 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b39bb471:::benchmark_data_smithi122_119671_object90632:head# 2024-03-20T09:04:48.467 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b39c2861:::benchmark_data_smithi122_119671_object55306:head# 2024-03-20T09:04:48.468 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b39c62d5:::benchmark_data_smithi122_119671_object36792:head# 2024-03-20T09:04:48.468 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b39cd99a:::benchmark_data_smithi122_119671_object85816:head# 2024-03-20T09:04:48.468 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b39da333:::benchmark_data_smithi122_119671_object20607:head# 2024-03-20T09:04:48.469 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b39e835e:::benchmark_data_smithi122_119671_object14153:head# 2024-03-20T09:04:48.471 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b39fa414:::benchmark_data_smithi122_119671_object25519:head# 2024-03-20T09:04:48.472 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3a0968b:::benchmark_data_smithi122_119671_object80027:head# 2024-03-20T09:04:48.472 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3a0d66c:::benchmark_data_smithi122_119671_object83270:head# 2024-03-20T09:04:48.473 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3a23e79:::benchmark_data_smithi122_119671_object30937:head# 2024-03-20T09:04:48.473 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3a24c26:::benchmark_data_smithi122_119671_object58549:head# 2024-03-20T09:04:48.474 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3a2c8e1:::benchmark_data_smithi122_119671_object65259:head# 2024-03-20T09:04:48.474 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3a4c009:::benchmark_data_smithi122_119671_object85246:head# 2024-03-20T09:04:48.475 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3a4da28:::benchmark_data_smithi122_119671_object85288:head# 2024-03-20T09:04:48.475 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3a602c7:::benchmark_data_smithi122_119671_object41168:head# 2024-03-20T09:04:48.476 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3a7021a:::benchmark_data_smithi122_119671_object28547:head# 2024-03-20T09:04:48.476 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3a7707e:::benchmark_data_smithi122_119671_object83271:head# 2024-03-20T09:04:48.477 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3a89b80:::benchmark_data_smithi122_119671_object85108:head# 2024-03-20T09:04:48.477 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3a9f84a:::benchmark_data_smithi122_119671_object47989:head# 2024-03-20T09:04:48.477 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3aa9fb0:::benchmark_data_smithi122_119671_object53628:head# 2024-03-20T09:04:48.478 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3ab7d18:::benchmark_data_smithi122_119671_object48342:head# 2024-03-20T09:04:48.479 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3ac4bb3:::benchmark_data_smithi122_119671_object26312:head# 2024-03-20T09:04:48.482 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3ad683e:::benchmark_data_smithi122_119671_object15213:head# 2024-03-20T09:04:48.482 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3ada778:::benchmark_data_smithi122_119671_object88226:head# 2024-03-20T09:04:48.483 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3adbdb7:::benchmark_data_smithi122_119671_object19912:head# 2024-03-20T09:04:48.483 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3ae2586:::benchmark_data_smithi122_119671_object96688:head# 2024-03-20T09:04:48.484 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3aea23d:::benchmark_data_smithi122_119671_object63131:head# 2024-03-20T09:04:48.484 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3aea75e:::benchmark_data_smithi122_119671_object38449:head# 2024-03-20T09:04:48.485 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3afb2f2:::benchmark_data_smithi122_119671_object75063:head# 2024-03-20T09:04:48.485 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3afc7e9:::benchmark_data_smithi122_119671_object50009:head# 2024-03-20T09:04:48.486 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3b1b46a:::benchmark_data_smithi122_119671_object58373:head# 2024-03-20T09:04:48.486 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3b247ae:::benchmark_data_smithi122_119671_object3871:head# 2024-03-20T09:04:48.487 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3b447f3:::benchmark_data_smithi122_119671_object78128:head# 2024-03-20T09:04:48.487 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3b52213:::benchmark_data_smithi122_119671_object73829:head# 2024-03-20T09:04:48.487 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3b524c2:::benchmark_data_smithi122_119671_object20871:head# 2024-03-20T09:04:48.488 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3b698c4:::benchmark_data_smithi122_119671_object75325:head# 2024-03-20T09:04:48.488 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3b706f4:::benchmark_data_smithi122_119671_object54080:head# 2024-03-20T09:04:48.489 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3b7465f:::benchmark_data_smithi122_119671_object6035:head# 2024-03-20T09:04:48.489 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3b7500c:::benchmark_data_smithi122_119671_object39744:head# 2024-03-20T09:04:48.489 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3b88660:::benchmark_data_smithi122_119671_object2132:head# 2024-03-20T09:04:48.490 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3ba5609:::benchmark_data_smithi122_119671_object88404:head# 2024-03-20T09:04:48.491 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3bacfc5:::benchmark_data_smithi122_119671_object62419:head# 2024-03-20T09:04:48.491 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3bb845b:::benchmark_data_smithi122_119671_object91096:head# 2024-03-20T09:04:48.491 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3bbb9d8:::benchmark_data_smithi122_119671_object48564:head# 2024-03-20T09:04:48.491 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3bc0e6a:::benchmark_data_smithi122_119671_object49034:head# 2024-03-20T09:04:48.492 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3bc13af:::benchmark_data_smithi122_119671_object91850:head# 2024-03-20T09:04:48.492 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3bef708:::benchmark_data_smithi122_119671_object57310:head# 2024-03-20T09:04:48.492 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3bef84a:::benchmark_data_smithi122_119671_object37086:head# 2024-03-20T09:04:48.493 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3bff6f0:::benchmark_data_smithi122_119671_object62682:head# 2024-03-20T09:04:48.493 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3bffbd8:::benchmark_data_smithi122_119671_object73348:head# 2024-03-20T09:04:48.494 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3c0940a:::benchmark_data_smithi122_119671_object44715:head# 2024-03-20T09:04:48.494 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3c0e936:::benchmark_data_smithi122_119671_object2753:head# 2024-03-20T09:04:48.495 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3c25ca3:::benchmark_data_smithi122_119671_object61863:head# 2024-03-20T09:04:48.495 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3c31559:::benchmark_data_smithi122_119671_object28606:head# 2024-03-20T09:04:48.496 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3c34c3a:::benchmark_data_smithi122_119671_object42972:head# 2024-03-20T09:04:48.496 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3c385f5:::benchmark_data_smithi122_119671_object63661:head# 2024-03-20T09:04:48.497 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3c3e28d:::benchmark_data_smithi122_119671_object20780:head# 2024-03-20T09:04:48.497 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3c47ffe:::benchmark_data_smithi122_119671_object9008:head# 2024-03-20T09:04:48.498 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3c593c8:::benchmark_data_smithi122_119671_object47520:head# 2024-03-20T09:04:48.498 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3c7e2b4:::benchmark_data_smithi122_119671_object11120:head# 2024-03-20T09:04:48.499 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3c95104:::benchmark_data_smithi122_119671_object64230:head# 2024-03-20T09:04:48.499 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3c98c09:::benchmark_data_smithi122_119671_object47230:head# 2024-03-20T09:04:48.499 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3c9ea73:::benchmark_data_smithi122_119671_object33613:head# 2024-03-20T09:04:48.500 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3ca3ca6:::benchmark_data_smithi122_119671_object68122:head# 2024-03-20T09:04:48.500 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3caa26f:::benchmark_data_smithi122_119671_object22973:head# 2024-03-20T09:04:48.501 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3cc0362:::benchmark_data_smithi122_119671_object6337:head# 2024-03-20T09:04:48.501 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3cc80f8:::benchmark_data_smithi122_119671_object92780:head# 2024-03-20T09:04:48.502 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3cd3cf6:::benchmark_data_smithi122_119671_object95842:head# 2024-03-20T09:04:48.502 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3cd4bc4:::benchmark_data_smithi122_119671_object64976:head# 2024-03-20T09:04:48.502 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3cd7a31:::benchmark_data_smithi122_119671_object62623:head# 2024-03-20T09:04:48.503 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3cdecb4:::benchmark_data_smithi122_119671_object26001:head# 2024-03-20T09:04:48.503 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3cf029d:::benchmark_data_smithi122_119671_object89288:head# 2024-03-20T09:04:48.504 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3cf4b68:::benchmark_data_smithi122_119671_object52550:head# 2024-03-20T09:04:48.504 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3cf4ff1:::benchmark_data_smithi122_119671_object11151:head# 2024-03-20T09:04:48.504 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3cf7fa6:::benchmark_data_smithi122_119671_object14800:head# 2024-03-20T09:04:48.505 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3cfb777:::benchmark_data_smithi122_119671_object16668:head# 2024-03-20T09:04:48.505 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3d07ed4:::benchmark_data_smithi122_119671_object46155:head# 2024-03-20T09:04:48.506 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3d28091:::benchmark_data_smithi122_119671_object80140:head# 2024-03-20T09:04:48.506 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3d2cec1:::benchmark_data_smithi122_119671_object92001:head# 2024-03-20T09:04:48.507 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3d2f16e:::benchmark_data_smithi122_119671_object45617:head# 2024-03-20T09:04:48.507 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3d336e3:::benchmark_data_smithi122_119671_object44397:head# 2024-03-20T09:04:48.507 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3d49508:::benchmark_data_smithi122_119671_object52103:head# 2024-03-20T09:04:48.508 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3d55be2:::benchmark_data_smithi122_119671_object5994:head# 2024-03-20T09:04:48.508 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3d6006d:::benchmark_data_smithi122_119671_object28108:head# 2024-03-20T09:04:48.509 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3d71de4:::benchmark_data_smithi122_119671_object2542:head# 2024-03-20T09:04:48.509 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3d79450:::benchmark_data_smithi122_119671_object26350:head# 2024-03-20T09:04:48.509 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3d7e9b7:::benchmark_data_smithi122_119671_object6942:head# 2024-03-20T09:04:48.510 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3d85384:::benchmark_data_smithi122_119671_object22270:head# 2024-03-20T09:04:48.510 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3d85554:::benchmark_data_smithi122_119671_object53864:head# 2024-03-20T09:04:48.511 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3d8f57a:::benchmark_data_smithi122_119671_object95913:head# 2024-03-20T09:04:48.511 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3d980b9:::benchmark_data_smithi122_119671_object9678:head# 2024-03-20T09:04:48.512 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3d986f3:::benchmark_data_smithi122_119671_object45927:head# 2024-03-20T09:04:48.512 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3da0cd8:::benchmark_data_smithi122_119671_object89656:head# 2024-03-20T09:04:48.512 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3da91a0:::benchmark_data_smithi122_119671_object71895:head# 2024-03-20T09:04:48.513 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3da9a8e:::benchmark_data_smithi122_119671_object94928:head# 2024-03-20T09:04:48.513 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3dad57c:::benchmark_data_smithi122_119671_object27035:head# 2024-03-20T09:04:48.514 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3dc2195:::benchmark_data_smithi122_119671_object28891:head# 2024-03-20T09:04:48.514 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3dd3780:::benchmark_data_smithi122_119671_object58714:head# 2024-03-20T09:04:48.515 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3dd7077:::benchmark_data_smithi122_119671_object29253:head# 2024-03-20T09:04:48.515 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3dd7cc1:::benchmark_data_smithi122_119671_object81226:head# 2024-03-20T09:04:48.516 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3e082df:::benchmark_data_smithi122_119671_object81138:head# 2024-03-20T09:04:48.516 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3e173f7:::benchmark_data_smithi122_119671_object72222:head# 2024-03-20T09:04:48.517 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3e20a2c:::benchmark_data_smithi122_119671_object87676:head# 2024-03-20T09:04:48.517 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3e26746:::benchmark_data_smithi122_119671_object78924:head# 2024-03-20T09:04:48.518 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3e26b00:::benchmark_data_smithi122_119671_object93398:head# 2024-03-20T09:04:48.518 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3e2fe9a:::benchmark_data_smithi122_119671_object3773:head# 2024-03-20T09:04:48.518 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3e4939e:::benchmark_data_smithi122_119671_object3062:head# 2024-03-20T09:04:48.519 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3e5275a:::benchmark_data_smithi122_119671_object29359:head# 2024-03-20T09:04:48.519 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3e5fadc:::benchmark_data_smithi122_119671_object65203:head# 2024-03-20T09:04:48.520 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3e622ee:::benchmark_data_smithi122_119671_object54028:head# 2024-03-20T09:04:48.520 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3e7b9cd:::benchmark_data_smithi122_119671_object71467:head# 2024-03-20T09:04:48.520 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3e7cb28:::benchmark_data_smithi122_119671_object99532:head# 2024-03-20T09:04:48.521 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3e81e5d:::benchmark_data_smithi122_119671_object76207:head# 2024-03-20T09:04:48.521 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3e8277f:::benchmark_data_smithi122_119671_object36106:head# 2024-03-20T09:04:48.522 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3ebb87c:::benchmark_data_smithi122_119671_object72412:head# 2024-03-20T09:04:48.522 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3ebea25:::benchmark_data_smithi122_119671_object80742:head# 2024-03-20T09:04:48.523 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3ec1e85:::benchmark_data_smithi122_119671_object96548:head# 2024-03-20T09:04:48.523 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3ec897a:::benchmark_data_smithi122_119671_object98258:head# 2024-03-20T09:04:48.524 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3ec9a95:::benchmark_data_smithi122_119671_object49515:head# 2024-03-20T09:04:48.526 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3ed289e:::benchmark_data_smithi122_119671_object17670:head# 2024-03-20T09:04:48.526 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3ed61fe:::benchmark_data_smithi122_119671_object37825:head# 2024-03-20T09:04:48.526 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3ed6eb0:::benchmark_data_smithi122_119671_object28076:head# 2024-03-20T09:04:48.526 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3edac4c:::benchmark_data_smithi122_119671_object99697:head# 2024-03-20T09:04:48.527 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3ef2adc:::benchmark_data_smithi122_119671_object65303:head# 2024-03-20T09:04:48.527 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3efc376:::benchmark_data_smithi122_119671_object63248:head# 2024-03-20T09:04:48.527 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3f02d1f:::benchmark_data_smithi122_119671_object69003:head# 2024-03-20T09:04:48.528 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3f12a1e:::benchmark_data_smithi122_119671_object64918:head# 2024-03-20T09:04:48.528 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3f15a48:::benchmark_data_smithi122_119671_object37588:head# 2024-03-20T09:04:48.529 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3f1da78:::benchmark_data_smithi122_119671_object20542:head# 2024-03-20T09:04:48.529 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3f23797:::benchmark_data_smithi122_119671_object96541:head# 2024-03-20T09:04:48.529 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3f2d2df:::benchmark_data_smithi122_119671_object16355:head# 2024-03-20T09:04:48.530 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3f3081e:::benchmark_data_smithi122_119671_object28042:head# 2024-03-20T09:04:48.530 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3f47994:::benchmark_data_smithi122_119671_object68947:head# 2024-03-20T09:04:48.530 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3f55c4f:::benchmark_data_smithi122_119671_object49729:head# 2024-03-20T09:04:48.531 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3f5e670:::benchmark_data_smithi122_119671_object15146:head# 2024-03-20T09:04:48.531 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3f79625:::benchmark_data_smithi122_119671_object47472:head# 2024-03-20T09:04:48.532 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3f79de4:::benchmark_data_smithi122_119671_object21407:head# 2024-03-20T09:04:48.532 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3f81201:::benchmark_data_smithi122_119671_object65547:head# 2024-03-20T09:04:48.533 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3f89ef2:::benchmark_data_smithi122_119671_object25037:head# 2024-03-20T09:04:48.533 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3f8da35:::benchmark_data_smithi122_119671_object41889:head# 2024-03-20T09:04:48.534 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3f8db82:::benchmark_data_smithi122_119671_object69518:head# 2024-03-20T09:04:48.534 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3fac0db:::benchmark_data_smithi122_119671_object96885:head# 2024-03-20T09:04:48.535 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3fb3df7:::benchmark_data_smithi122_119671_object77505:head# 2024-03-20T09:04:48.535 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3fb6e4a:::benchmark_data_smithi122_119671_object59729:head# 2024-03-20T09:04:48.535 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b3ffb5cd:::benchmark_data_smithi122_119671_object99465:head# 2024-03-20T09:04:48.536 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b401d109:::benchmark_data_smithi122_119671_object26507:head# 2024-03-20T09:04:48.536 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b401fce0:::benchmark_data_smithi122_119671_object73184:head# 2024-03-20T09:04:48.537 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b40201c1:::benchmark_data_smithi122_119671_object79218:head# 2024-03-20T09:04:48.537 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b402d068:::benchmark_data_smithi122_119671_object95865:head# 2024-03-20T09:04:48.538 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4031faf:::benchmark_data_smithi122_119671_object90494:head# 2024-03-20T09:04:48.539 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4036c65:::benchmark_data_smithi122_119671_object66777:head# 2024-03-20T09:04:48.539 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b403dbb7:::benchmark_data_smithi122_119671_object34094:head# 2024-03-20T09:04:48.539 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4049906:::benchmark_data_smithi122_119671_object80467:head# 2024-03-20T09:04:48.540 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4051be6:::benchmark_data_smithi122_119671_object53587:head# 2024-03-20T09:04:48.540 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b405d78b:::benchmark_data_smithi122_119671_object21849:head# 2024-03-20T09:04:48.541 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4070bc3:::benchmark_data_smithi122_119671_object16992:head# 2024-03-20T09:04:48.541 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b407697f:::benchmark_data_smithi122_119671_object62677:head# 2024-03-20T09:04:48.541 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b407c156:::benchmark_data_smithi122_119671_object67372:head# 2024-03-20T09:04:48.542 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4082939:::benchmark_data_smithi122_119671_object88649:head# 2024-03-20T09:04:48.542 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b40837ae:::benchmark_data_smithi122_119671_object91137:head# 2024-03-20T09:04:48.543 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b40a9a95:::benchmark_data_smithi122_119671_object16497:head# 2024-03-20T09:04:48.543 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b40ac00d:::benchmark_data_smithi122_119671_object43632:head# 2024-03-20T09:04:48.544 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b40b8eeb:::benchmark_data_smithi122_119671_object50396:head# 2024-03-20T09:04:48.544 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b40ce6fc:::benchmark_data_smithi122_119671_object88371:head# 2024-03-20T09:04:48.544 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b40d5a4e:::benchmark_data_smithi122_119671_object24220:head# 2024-03-20T09:04:48.545 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b40d6de3:::benchmark_data_smithi122_119671_object95040:head# 2024-03-20T09:04:48.545 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b40d862a:::benchmark_data_smithi122_119671_object29264:head# 2024-03-20T09:04:48.546 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b40d95f8:::benchmark_data_smithi122_119671_object82174:head# 2024-03-20T09:04:48.546 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b40e103b:::benchmark_data_smithi122_119671_object13693:head# 2024-03-20T09:04:48.547 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b40f5fbb:::benchmark_data_smithi122_119671_object65256:head# 2024-03-20T09:04:48.547 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b40f9981:::benchmark_data_smithi122_119671_object83718:head# 2024-03-20T09:04:48.548 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b412e3c8:::benchmark_data_smithi122_119671_object44320:head# 2024-03-20T09:04:48.548 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b41343ed:::benchmark_data_smithi122_119671_object30939:head# 2024-03-20T09:04:48.548 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b414562d:::benchmark_data_smithi122_119671_object71343:head# 2024-03-20T09:04:48.549 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b41468f2:::benchmark_data_smithi122_119671_object45096:head# 2024-03-20T09:04:48.549 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4149fe2:::benchmark_data_smithi122_119671_object44329:head# 2024-03-20T09:04:48.550 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4152b25:::benchmark_data_smithi122_119671_object3664:head# 2024-03-20T09:04:48.550 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4154c0b:::benchmark_data_smithi122_119671_object96629:head# 2024-03-20T09:04:48.550 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4156a4a:::benchmark_data_smithi122_119671_object25288:head# 2024-03-20T09:04:48.551 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b41656eb:::benchmark_data_smithi122_119671_object50296:head# 2024-03-20T09:04:48.551 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4167e8f:::benchmark_data_smithi122_119671_object37600:head# 2024-03-20T09:04:48.552 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b416cf9e:::benchmark_data_smithi122_119671_object59856:head# 2024-03-20T09:04:48.552 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4173e9c:::benchmark_data_smithi122_119671_object44052:head# 2024-03-20T09:04:48.552 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4190e9a:::benchmark_data_smithi122_119671_object3975:head# 2024-03-20T09:04:48.553 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4193b29:::benchmark_data_smithi122_119671_object12377:head# 2024-03-20T09:04:48.553 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b419da45:::benchmark_data_smithi122_119671_object75348:head# 2024-03-20T09:04:48.554 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b41ab3e7:::benchmark_data_smithi122_119671_object37501:head# 2024-03-20T09:04:48.554 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b41ae040:::benchmark_data_smithi122_119671_object8223:head# 2024-03-20T09:04:48.555 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b41b24e9:::benchmark_data_smithi122_119671_object7453:head# 2024-03-20T09:04:48.555 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b41bdd14:::benchmark_data_smithi122_119671_object23037:head# 2024-03-20T09:04:48.555 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b41c7602:::benchmark_data_smithi122_119671_object51406:head# 2024-03-20T09:04:48.556 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b41c8024:::benchmark_data_smithi122_119671_object78489:head# 2024-03-20T09:04:48.556 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b41c8e79:::benchmark_data_smithi122_119671_object2482:head# 2024-03-20T09:04:48.556 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b41d14f0:::benchmark_data_smithi122_119671_object90580:head# 2024-03-20T09:04:48.557 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b41d9ccf:::benchmark_data_smithi122_119671_object87651:head# 2024-03-20T09:04:48.557 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b41dec81:::benchmark_data_smithi122_119671_object30897:head# 2024-03-20T09:04:48.558 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b41e5224:::benchmark_data_smithi122_119671_object57584:head# 2024-03-20T09:04:48.558 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b42027fe:::benchmark_data_smithi122_119671_object91023:head# 2024-03-20T09:04:48.558 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4207554:::benchmark_data_smithi122_119671_object47330:head# 2024-03-20T09:04:48.559 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4207b7a:::benchmark_data_smithi122_119671_object53673:head# 2024-03-20T09:04:48.559 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b420acea:::benchmark_data_smithi122_119671_object13538:head# 2024-03-20T09:04:48.560 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b420de57:::benchmark_data_smithi122_119671_object37316:head# 2024-03-20T09:04:48.560 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b420ff2e:::benchmark_data_smithi122_119671_object21343:head# 2024-03-20T09:04:48.561 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b421fa7e:::benchmark_data_smithi122_119671_object84796:head# 2024-03-20T09:04:48.561 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4220d5b:::benchmark_data_smithi122_119671_object44389:head# 2024-03-20T09:04:48.561 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:04:48.563 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4239fb8:::benchmark_data_smithi122_119671_object5161:head# 2024-03-20T09:04:48.563 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4243277:::benchmark_data_smithi122_119671_object59530:head# 2024-03-20T09:04:48.563 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4245c93:::benchmark_data_smithi122_119671_object50601:head# 2024-03-20T09:04:48.563 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b424c90f:::benchmark_data_smithi122_119671_object3020:head# 2024-03-20T09:04:48.565 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b424f215:::benchmark_data_smithi122_119671_object39511:head# 2024-03-20T09:04:48.565 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4262e50:::benchmark_data_smithi122_119671_object14513:head# 2024-03-20T09:04:48.565 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4276239:::benchmark_data_smithi122_119671_object17416:head# 2024-03-20T09:04:48.565 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4277874:::benchmark_data_smithi122_119671_object89419:head# 2024-03-20T09:04:48.565 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4278fbd:::benchmark_data_smithi122_119671_object65523:head# 2024-03-20T09:04:48.566 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4285629:::benchmark_data_smithi122_119671_object54898:head# 2024-03-20T09:04:48.566 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b428590e:::benchmark_data_smithi122_119671_object51589:head# 2024-03-20T09:04:48.566 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b42884fa:::benchmark_data_smithi122_119671_object7023:head# 2024-03-20T09:04:48.567 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4288b53:::benchmark_data_smithi122_119671_object5628:head# 2024-03-20T09:04:48.567 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b42991a5:::benchmark_data_smithi122_119671_object38935:head# 2024-03-20T09:04:48.568 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b42ad6a5:::benchmark_data_smithi122_119671_object62684:head# 2024-03-20T09:04:48.570 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b42c3a37:::benchmark_data_smithi122_119671_object19944:head# 2024-03-20T09:04:48.570 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b42c934a:::benchmark_data_smithi122_119671_object36175:head# 2024-03-20T09:04:48.570 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b42d4a8e:::benchmark_data_smithi122_119671_object67334:head# 2024-03-20T09:04:48.570 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b42d549b:::benchmark_data_smithi122_119671_object44224:head# 2024-03-20T09:04:48.572 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b42dc4a8:::benchmark_data_smithi122_119671_object93608:head# 2024-03-20T09:04:48.572 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b42dec4f:::benchmark_data_smithi122_119671_object54768:head# 2024-03-20T09:04:48.572 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b42e9089:::benchmark_data_smithi122_119671_object94249:head# 2024-03-20T09:04:48.572 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b42ef5e9:::benchmark_data_smithi122_119671_object38739:head# 2024-03-20T09:04:48.574 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b42f7761:::benchmark_data_smithi122_119671_object50492:head# 2024-03-20T09:04:48.574 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b431552d:::benchmark_data_smithi122_119671_object70844:head# 2024-03-20T09:04:48.574 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4332ad4:::benchmark_data_smithi122_119671_object84719:head# 2024-03-20T09:04:48.575 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b433ab74:::benchmark_data_smithi122_119671_object22356:head# 2024-03-20T09:04:48.575 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b43527b5:::benchmark_data_smithi122_119671_object87269:head# 2024-03-20T09:04:48.576 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b435994a:::benchmark_data_smithi122_119671_object1826:head# 2024-03-20T09:04:48.576 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b435b41a:::benchmark_data_smithi122_119671_object15511:head# 2024-03-20T09:04:48.576 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b435f9d5:::benchmark_data_smithi122_119671_object47327:head# 2024-03-20T09:04:48.577 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4365716:::benchmark_data_smithi122_119671_object61176:head# 2024-03-20T09:04:48.580 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4378b67:::benchmark_data_smithi122_119671_object40376:head# 2024-03-20T09:04:48.580 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4379b90:::benchmark_data_smithi122_119671_object43508:head# 2024-03-20T09:04:48.581 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b43819ae:::benchmark_data_smithi122_119671_object16308:head# 2024-03-20T09:04:48.581 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b43a6f3c:::benchmark_data_smithi122_119671_object41036:head# 2024-03-20T09:04:48.581 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b43b0348:::benchmark_data_smithi122_119671_object40977:head# 2024-03-20T09:04:48.581 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b43c1d25:::benchmark_data_smithi122_119671_object80300:head# 2024-03-20T09:04:48.581 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b43c6a11:::benchmark_data_smithi122_119671_object49331:head# 2024-03-20T09:04:48.581 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b43d3555:::benchmark_data_smithi122_119671_object71930:head# 2024-03-20T09:04:48.582 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b43d60bc:::benchmark_data_smithi122_119671_object95467:head# 2024-03-20T09:04:48.584 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b43d9b2a:::benchmark_data_smithi122_119671_object80724:head# 2024-03-20T09:04:48.584 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b43e73f0:::benchmark_data_smithi122_119671_object17553:head# 2024-03-20T09:04:48.584 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b43ff5b7:::benchmark_data_smithi122_119671_object61887:head# 2024-03-20T09:04:48.584 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4406f06:::benchmark_data_smithi122_119671_object16533:head# 2024-03-20T09:04:48.584 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b440b893:::benchmark_data_smithi122_119671_object51703:head# 2024-03-20T09:04:48.585 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b441538f:::benchmark_data_smithi122_119671_object89812:head# 2024-03-20T09:04:48.585 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b44175a0:::benchmark_data_smithi122_119671_object42772:head# 2024-03-20T09:04:48.585 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b441ee3c:::benchmark_data_smithi122_119671_object34635:head# 2024-03-20T09:04:48.585 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4420aeb:::benchmark_data_smithi122_119671_object61415:head# 2024-03-20T09:04:48.588 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b44238b2:::benchmark_data_smithi122_119671_object61061:head# 2024-03-20T09:04:48.589 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b442522c:::benchmark_data_smithi122_119671_object18785:head# 2024-03-20T09:04:48.589 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b442b09f:::benchmark_data_smithi122_119671_object27300:head# 2024-03-20T09:04:48.589 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b442fc19:::benchmark_data_smithi122_119671_object41018:head# 2024-03-20T09:04:48.591 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4431fb8:::benchmark_data_smithi122_119671_object19547:head# 2024-03-20T09:04:48.591 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b443a5b2:::benchmark_data_smithi122_119671_object99544:head# 2024-03-20T09:04:48.591 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b443cd88:::benchmark_data_smithi122_119671_object72500:head# 2024-03-20T09:04:48.591 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b444b8e1:::benchmark_data_smithi122_119671_object52691:head# 2024-03-20T09:04:48.592 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b446dc72:::benchmark_data_smithi122_119671_object37308:head# 2024-03-20T09:04:48.592 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4470ba9:::benchmark_data_smithi122_119671_object6667:head# 2024-03-20T09:04:48.592 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4477259:::benchmark_data_smithi122_119671_object66985:head# 2024-03-20T09:04:48.593 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b447b44b:::benchmark_data_smithi122_119671_object14865:head# 2024-03-20T09:04:48.593 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b44878a0:::benchmark_data_smithi122_119671_object69505:head# 2024-03-20T09:04:48.593 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b448df36:::benchmark_data_smithi122_119671_object65607:head# 2024-03-20T09:04:48.594 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b449feb5:::benchmark_data_smithi122_119671_object72439:head# 2024-03-20T09:04:48.596 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b44ad1de:::benchmark_data_smithi122_119671_object33992:head# 2024-03-20T09:04:48.596 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b44af0f0:::benchmark_data_smithi122_119671_object18018:head# 2024-03-20T09:04:48.597 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b44ca358:::benchmark_data_smithi122_119671_object30670:head# 2024-03-20T09:04:48.597 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b44cd3a6:::benchmark_data_smithi122_119671_object67876:head# 2024-03-20T09:04:48.597 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b44d276d:::benchmark_data_smithi122_119671_object5075:head# 2024-03-20T09:04:48.597 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b44e5b18:::benchmark_data_smithi122_119671_object66748:head# 2024-03-20T09:04:48.597 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b44ef330:::benchmark_data_smithi122_119671_object5991:head# 2024-03-20T09:04:48.600 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b44f01a5:::benchmark_data_smithi122_119671_object13052:head# 2024-03-20T09:04:48.600 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4501bd6:::benchmark_data_smithi122_119671_object34902:head# 2024-03-20T09:04:48.600 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4516b0f:::benchmark_data_smithi122_119671_object49561:head# 2024-03-20T09:04:48.600 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4521092:::benchmark_data_smithi122_119671_object11474:head# 2024-03-20T09:04:48.601 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b45391dd:::benchmark_data_smithi122_119671_object30269:head# 2024-03-20T09:04:48.601 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b45398c2:::benchmark_data_smithi122_119671_object89282:head# 2024-03-20T09:04:48.601 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4548bbc:::benchmark_data_smithi122_119671_object95460:head# 2024-03-20T09:04:48.601 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b455dcff:::benchmark_data_smithi122_119671_object63211:head# 2024-03-20T09:04:48.602 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b455ddab:::benchmark_data_smithi122_119671_object87011:head# 2024-03-20T09:04:48.602 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b455de31:::benchmark_data_smithi122_119671_object90560:head# 2024-03-20T09:04:48.602 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:48.602 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b456679a:::benchmark_data_smithi122_119671_object72169:head# 2024-03-20T09:04:48.602 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b456ebed:::benchmark_data_smithi122_119671_object74857:head# 2024-03-20T09:04:48.603 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b45a4253:::benchmark_data_smithi122_119671_object3060:head# 2024-03-20T09:04:48.603 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b45ab0e1:::benchmark_data_smithi122_119671_object73476:head# 2024-03-20T09:04:48.604 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b45ae483:::benchmark_data_smithi122_119671_object49348:head# 2024-03-20T09:04:48.604 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b45b3685:::benchmark_data_smithi122_119671_object10176:head# 2024-03-20T09:04:48.604 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b45ba32f:::benchmark_data_smithi122_119671_object47582:head# 2024-03-20T09:04:48.605 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b45ca44c:::benchmark_data_smithi122_119671_object46067:head# 2024-03-20T09:04:48.605 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b45cbd12:::benchmark_data_smithi122_119671_object58328:head# 2024-03-20T09:04:48.605 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b45ea323:::benchmark_data_smithi122_119671_object96448:head# 2024-03-20T09:04:48.606 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b46088af:::benchmark_data_smithi122_119671_object64494:head# 2024-03-20T09:04:48.606 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4609dfe:::benchmark_data_smithi122_119671_object49647:head# 2024-03-20T09:04:48.607 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b460df17:::benchmark_data_smithi122_119671_object49920:head# 2024-03-20T09:04:48.607 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b46156bb:::benchmark_data_smithi122_119671_object9789:head# 2024-03-20T09:04:48.608 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b46221c4:::benchmark_data_smithi122_119671_object11831:head# 2024-03-20T09:04:48.608 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4623519:::benchmark_data_smithi122_119671_object76258:head# 2024-03-20T09:04:48.612 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4635663:::benchmark_data_smithi122_119671_object97305:head# 2024-03-20T09:04:48.612 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b463db4e:::benchmark_data_smithi122_119671_object15505:head# 2024-03-20T09:04:48.612 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b464bb9f:::benchmark_data_smithi122_119671_object87138:head# 2024-03-20T09:04:48.612 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4672d0d:::benchmark_data_smithi122_119671_object39728:head# 2024-03-20T09:04:48.612 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b467b6e5:::benchmark_data_smithi122_119671_object93187:head# 2024-03-20T09:04:48.612 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b46848e3:::benchmark_data_smithi122_119671_object50732:head# 2024-03-20T09:04:48.612 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b46864bf:::benchmark_data_smithi122_119671_object33820:head# 2024-03-20T09:04:48.612 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b46872f6:::benchmark_data_smithi122_119671_object28868:head# 2024-03-20T09:04:48.613 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b469641e:::benchmark_data_smithi122_119671_object68863:head# 2024-03-20T09:04:48.613 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b46a773e:::benchmark_data_smithi122_119671_object83656:head# 2024-03-20T09:04:48.613 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b46aafb4:::benchmark_data_smithi122_119671_object99165:head# 2024-03-20T09:04:48.614 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b46ac119:::benchmark_data_smithi122_119671_object33356:head# 2024-03-20T09:04:48.614 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b46bdc9b:::benchmark_data_smithi122_119671_object11845:head# 2024-03-20T09:04:48.614 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b46c868b:::benchmark_data_smithi122_119671_object30687:head# 2024-03-20T09:04:48.615 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b46cc16e:::benchmark_data_smithi122_119671_object70861:head# 2024-03-20T09:04:48.615 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b46e0ac6:::benchmark_data_smithi122_119671_object36730:head# 2024-03-20T09:04:48.616 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b46e94ab:::benchmark_data_smithi122_119671_object69567:head# 2024-03-20T09:04:48.616 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b46eba48:::benchmark_data_smithi122_119671_object17980:head# 2024-03-20T09:04:48.616 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b46f7377:::benchmark_data_smithi122_119671_object64330:head# 2024-03-20T09:04:48.617 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4707ede:::benchmark_data_smithi122_119671_object22927:head# 2024-03-20T09:04:48.617 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b471d044:::benchmark_data_smithi122_119671_object12193:head# 2024-03-20T09:04:48.618 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b472b73a:::benchmark_data_smithi122_119671_object57404:head# 2024-03-20T09:04:48.619 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b472e9e7:::benchmark_data_smithi122_119671_object73657:head# 2024-03-20T09:04:48.619 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4743755:::benchmark_data_smithi122_119671_object97729:head# 2024-03-20T09:04:48.619 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4749192:::benchmark_data_smithi122_119671_object39949:head# 2024-03-20T09:04:48.619 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b474e9e9:::benchmark_data_smithi122_119671_object50806:head# 2024-03-20T09:04:48.620 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4760a53:::benchmark_data_smithi122_119671_object41046:head# 2024-03-20T09:04:48.620 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4794423:::benchmark_data_smithi122_119671_object89207:head# 2024-03-20T09:04:48.621 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b479d4a6:::benchmark_data_smithi122_119671_object98294:head# 2024-03-20T09:04:48.622 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b47a1576:::benchmark_data_smithi122_119671_object36931:head# 2024-03-20T09:04:48.622 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b47a9641:::benchmark_data_smithi122_119671_object32630:head# 2024-03-20T09:04:48.622 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b47b5756:::benchmark_data_smithi122_119671_object76486:head# 2024-03-20T09:04:48.623 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b47b5ecc:::benchmark_data_smithi122_119671_object82333:head# 2024-03-20T09:04:48.623 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b47c92c7:::benchmark_data_smithi122_119671_object70912:head# 2024-03-20T09:04:48.624 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b47cfe69:::benchmark_data_smithi122_119671_object43364:head# 2024-03-20T09:04:48.624 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b47d6cc1:::benchmark_data_smithi122_119671_object87462:head# 2024-03-20T09:04:48.624 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b47eb736:::benchmark_data_smithi122_119671_object39782:head# 2024-03-20T09:04:48.625 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:04:48.628 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b47ef6de:::benchmark_data_smithi122_119671_object96119:head# 2024-03-20T09:04:48.628 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b480880b:::benchmark_data_smithi122_119671_object96345:head# 2024-03-20T09:04:48.628 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b480b322:::benchmark_data_smithi122_119671_object94230:head# 2024-03-20T09:04:48.628 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4829e1f:::benchmark_data_smithi122_119671_object20204:head# 2024-03-20T09:04:48.628 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b482b6d3:::benchmark_data_smithi122_119671_object92979:head# 2024-03-20T09:04:48.628 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b482bf7b:::benchmark_data_smithi122_119671_object13885:head# 2024-03-20T09:04:48.628 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b48351be:::benchmark_data_smithi122_119671_object79281:head# 2024-03-20T09:04:48.629 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4845220:::benchmark_data_smithi122_119671_object54321:head# 2024-03-20T09:04:48.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:48.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:04:48.630 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4849443:::benchmark_data_smithi122_119671_object93188:head# 2024-03-20T09:04:48.631 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b485347a:::benchmark_data_smithi122_119671_object4608:head# 2024-03-20T09:04:48.631 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b485e760:::benchmark_data_smithi122_119671_object87199:head# 2024-03-20T09:04:48.631 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4861d3b:::benchmark_data_smithi122_119671_object33414:head# 2024-03-20T09:04:48.631 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4864346:::benchmark_data_smithi122_119671_object93911:head# 2024-03-20T09:04:48.631 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4866e7c:::benchmark_data_smithi122_119671_object74873:head# 2024-03-20T09:04:48.631 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4867df6:::benchmark_data_smithi122_119671_object89895:head# 2024-03-20T09:04:48.632 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4877aef:::benchmark_data_smithi122_119671_object60057:head# 2024-03-20T09:04:48.632 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b487d2ec:::benchmark_data_smithi122_119671_object22726:head# 2024-03-20T09:04:48.632 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b487e2b0:::benchmark_data_smithi122_119671_object51649:head# 2024-03-20T09:04:48.632 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b48814d8:::benchmark_data_smithi122_119671_object55188:head# 2024-03-20T09:04:48.633 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b489d413:::benchmark_data_smithi122_119671_object60330:head# 2024-03-20T09:04:48.633 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b48a3c59:::benchmark_data_smithi122_119671_object71880:head# 2024-03-20T09:04:48.634 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b48aa1df:::benchmark_data_smithi122_119671_object25466:head# 2024-03-20T09:04:48.634 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b48b43e8:::benchmark_data_smithi122_119671_object64215:head# 2024-03-20T09:04:48.635 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b48be5c9:::benchmark_data_smithi122_119671_object78982:head# 2024-03-20T09:04:48.635 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b48bfe29:::benchmark_data_smithi122_119671_object74254:head# 2024-03-20T09:04:48.635 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b48c2b1e:::benchmark_data_smithi122_119671_object8568:head# 2024-03-20T09:04:48.636 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b48c72b3:::benchmark_data_smithi122_119671_object43600:head# 2024-03-20T09:04:48.636 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b48d44de:::benchmark_data_smithi122_119671_object16385:head# 2024-03-20T09:04:48.637 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b48d4889:::benchmark_data_smithi122_119671_object66420:head# 2024-03-20T09:04:48.637 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b48dcb7a:::benchmark_data_smithi122_119671_object7033:head# 2024-03-20T09:04:48.638 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b48e35e0:::benchmark_data_smithi122_119671_object66020:head# 2024-03-20T09:04:48.638 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4900f06:::benchmark_data_smithi122_119671_object66880:head# 2024-03-20T09:04:48.639 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4908bee:::benchmark_data_smithi122_119671_object77654:head# 2024-03-20T09:04:48.639 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4908d89:::benchmark_data_smithi122_119671_object33297:head# 2024-03-20T09:04:48.640 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4910710:::benchmark_data_smithi122_119671_object26674:head# 2024-03-20T09:04:48.640 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b49164e9:::benchmark_data_smithi122_119671_object78331:head# 2024-03-20T09:04:48.640 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b491e82c:::benchmark_data_smithi122_119671_object19543:head# 2024-03-20T09:04:48.641 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b492c26b:::benchmark_data_smithi122_119671_object45997:head# 2024-03-20T09:04:48.641 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4936947:::benchmark_data_smithi122_119671_object80038:head# 2024-03-20T09:04:48.642 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4945776:::benchmark_data_smithi122_119671_object10155:head# 2024-03-20T09:04:48.643 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4949468:::benchmark_data_smithi122_119671_object53087:head# 2024-03-20T09:04:48.643 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b494b111:::benchmark_data_smithi122_119671_object23799:head# 2024-03-20T09:04:48.643 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4951b2b:::benchmark_data_smithi122_119671_object64834:head# 2024-03-20T09:04:48.644 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4956618:::benchmark_data_smithi122_119671_object39031:head# 2024-03-20T09:04:48.644 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b495a3e1:::benchmark_data_smithi122_119671_object76075:head# 2024-03-20T09:04:48.644 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4970871:::benchmark_data_smithi122_119671_object96281:head# 2024-03-20T09:04:48.645 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b49794e6:::benchmark_data_smithi122_119671_object58710:head# 2024-03-20T09:04:48.645 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4981d3c:::benchmark_data_smithi122_119671_object64135:head# 2024-03-20T09:04:48.646 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b49915c9:::benchmark_data_smithi122_119671_object56806:head# 2024-03-20T09:04:48.647 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b499ab97:::benchmark_data_smithi122_119671_object99702:head# 2024-03-20T09:04:48.648 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b49b21dc:::benchmark_data_smithi122_119671_object37120:head# 2024-03-20T09:04:48.648 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b49c2466:::benchmark_data_smithi122_119671_object27310:head# 2024-03-20T09:04:48.649 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b49dfb01:::benchmark_data_smithi122_119671_object5447:head# 2024-03-20T09:04:48.649 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b49ebfc6:::benchmark_data_smithi122_119671_object61045:head# 2024-03-20T09:04:48.650 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4a021c6:::benchmark_data_smithi122_119671_object59301:head# 2024-03-20T09:04:48.650 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4a123a8:::benchmark_data_smithi122_119671_object5642:head# 2024-03-20T09:04:48.650 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4a1e2c9:::benchmark_data_smithi122_119671_object53697:head# 2024-03-20T09:04:48.651 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4a2add4:::benchmark_data_smithi122_119671_object67539:head# 2024-03-20T09:04:48.651 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4a393c4:::benchmark_data_smithi122_119671_object53852:head# 2024-03-20T09:04:48.652 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4a4dab1:::benchmark_data_smithi122_119671_object61808:head# 2024-03-20T09:04:48.652 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4a4e570:::benchmark_data_smithi122_119671_object70184:head# 2024-03-20T09:04:48.653 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4a53592:::benchmark_data_smithi122_119671_object61100:head# 2024-03-20T09:04:48.654 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4a60305:::benchmark_data_smithi122_119671_object1979:head# 2024-03-20T09:04:48.654 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4a67db7:::benchmark_data_smithi122_119671_object55337:head# 2024-03-20T09:04:48.655 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4a706d0:::benchmark_data_smithi122_119671_object62904:head# 2024-03-20T09:04:48.658 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4a70f25:::benchmark_data_smithi122_119671_object1986:head# 2024-03-20T09:04:48.658 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4a8cf6f:::benchmark_data_smithi122_119671_object86580:head# 2024-03-20T09:04:48.659 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4a8ee56:::benchmark_data_smithi122_119671_object10610:head# 2024-03-20T09:04:48.659 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4a916ad:::benchmark_data_smithi122_119671_object54060:head# 2024-03-20T09:04:48.659 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4a9f9ee:::benchmark_data_smithi122_119671_object16726:head# 2024-03-20T09:04:48.660 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4ab4863:::benchmark_data_smithi122_119671_object80226:head# 2024-03-20T09:04:48.660 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4ab88eb:::benchmark_data_smithi122_119671_object82273:head# 2024-03-20T09:04:48.662 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4abb361:::benchmark_data_smithi122_119671_object39702:head# 2024-03-20T09:04:48.663 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4adc617:::benchmark_data_smithi122_119671_object92449:head# 2024-03-20T09:04:48.663 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4ae07b6:::benchmark_data_smithi122_119671_object90923:head# 2024-03-20T09:04:48.664 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4af8b3a:::benchmark_data_smithi122_119671_object27603:head# 2024-03-20T09:04:48.664 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4b022f5:::benchmark_data_smithi122_119671_object74690:head# 2024-03-20T09:04:48.665 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4b0b593:::benchmark_data_smithi122_119671_object10770:head# 2024-03-20T09:04:48.665 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4b101bf:::benchmark_data_smithi122_119671_object22305:head# 2024-03-20T09:04:48.665 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4b1053e:::benchmark_data_smithi122_119671_object98559:head# 2024-03-20T09:04:48.666 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4b2ca89:::benchmark_data_smithi122_119671_object66352:head# 2024-03-20T09:04:48.667 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4b45376:::benchmark_data_smithi122_119671_object22839:head# 2024-03-20T09:04:48.668 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4b503d5:::benchmark_data_smithi122_119671_object66176:head# 2024-03-20T09:04:48.671 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4b54d6e:::benchmark_data_smithi122_119671_object43731:head# 2024-03-20T09:04:48.671 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4b579b6:::benchmark_data_smithi122_119671_object37210:head# 2024-03-20T09:04:48.672 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4b58bfd:::benchmark_data_smithi122_119671_object35433:head# 2024-03-20T09:04:48.672 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4b59d9d:::benchmark_data_smithi122_119671_object76577:head# 2024-03-20T09:04:48.673 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4b6d36c:::benchmark_data_smithi122_119671_object24245:head# 2024-03-20T09:04:48.673 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4b6d741:::benchmark_data_smithi122_119671_object84762:head# 2024-03-20T09:04:48.674 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4b6dca0:::benchmark_data_smithi122_119671_object95085:head# 2024-03-20T09:04:48.674 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4b6eedf:::benchmark_data_smithi122_119671_object36751:head# 2024-03-20T09:04:48.675 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4b6f58a:::benchmark_data_smithi122_119671_object94597:head# 2024-03-20T09:04:48.675 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4b71954:::benchmark_data_smithi122_119671_object29097:head# 2024-03-20T09:04:48.675 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4b72574:::benchmark_data_smithi122_119671_object49343:head# 2024-03-20T09:04:48.676 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4b8217c:::benchmark_data_smithi122_119671_object62683:head# 2024-03-20T09:04:48.676 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4b85bd9:::benchmark_data_smithi122_119671_object14762:head# 2024-03-20T09:04:48.676 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4b905b8:::benchmark_data_smithi122_119671_object41797:head# 2024-03-20T09:04:48.677 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4b96bd6:::benchmark_data_smithi122_119671_object5599:head# 2024-03-20T09:04:48.677 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4b9a682:::benchmark_data_smithi122_119671_object63083:head# 2024-03-20T09:04:48.678 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4ba92ef:::benchmark_data_smithi122_119671_object17378:head# 2024-03-20T09:04:48.680 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4baf830:::benchmark_data_smithi122_119671_object75690:head# 2024-03-20T09:04:48.680 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4bb39f7:::benchmark_data_smithi122_119671_object66731:head# 2024-03-20T09:04:48.680 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4bc4926:::benchmark_data_smithi122_119671_object51526:head# 2024-03-20T09:04:48.681 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4bc8be8:::benchmark_data_smithi122_119671_object8637:head# 2024-03-20T09:04:48.681 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4bce129:::benchmark_data_smithi122_119671_object62638:head# 2024-03-20T09:04:48.682 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4bd4e5c:::benchmark_data_smithi122_119671_object58015:head# 2024-03-20T09:04:48.682 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4bd7f5c:::benchmark_data_smithi122_119671_object55362:head# 2024-03-20T09:04:48.683 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4beb429:::benchmark_data_smithi122_119671_object9824:head# 2024-03-20T09:04:48.683 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4bf57dc:::benchmark_data_smithi122_119671_object25398:head# 2024-03-20T09:04:48.684 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4bfbf1b:::benchmark_data_smithi122_119671_object47692:head# 2024-03-20T09:04:48.684 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4c0cc23:::benchmark_data_smithi122_119671_object78169:head# 2024-03-20T09:04:48.684 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4c175ae:::benchmark_data_smithi122_119671_object14013:head# 2024-03-20T09:04:48.685 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4c23dd3:::benchmark_data_smithi122_119671_object60105:head# 2024-03-20T09:04:48.685 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4c4d7d5:::benchmark_data_smithi122_119671_object98574:head# 2024-03-20T09:04:48.685 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4c5c771:::benchmark_data_smithi122_119671_object97079:head# 2024-03-20T09:04:48.686 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4c5e2d0:::benchmark_data_smithi122_119671_object36605:head# 2024-03-20T09:04:48.686 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4c6ab10:::benchmark_data_smithi122_119671_object95625:head# 2024-03-20T09:04:48.687 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4c6cb10:::benchmark_data_smithi122_119671_object49751:head# 2024-03-20T09:04:48.687 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4c74e31:::benchmark_data_smithi122_119671_object34729:head# 2024-03-20T09:04:48.687 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4c7f15e:::benchmark_data_smithi122_119671_object85783:head# 2024-03-20T09:04:48.688 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4c87758:::benchmark_data_smithi122_119671_object37229:head# 2024-03-20T09:04:48.688 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4c9cb54:::benchmark_data_smithi122_119671_object51640:head# 2024-03-20T09:04:48.689 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4c9cd48:::benchmark_data_smithi122_119671_object77402:head# 2024-03-20T09:04:48.689 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4ca1f7c:::benchmark_data_smithi122_119671_object43002:head# 2024-03-20T09:04:48.690 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4ca82af:::benchmark_data_smithi122_119671_object83452:head# 2024-03-20T09:04:48.690 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4cad607:::benchmark_data_smithi122_119671_object47754:head# 2024-03-20T09:04:48.690 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4cb313a:::benchmark_data_smithi122_119671_object96438:head# 2024-03-20T09:04:48.691 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4cb65e2:::benchmark_data_smithi122_119671_object30307:head# 2024-03-20T09:04:48.691 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4cbd045:::benchmark_data_smithi122_119671_object41624:head# 2024-03-20T09:04:48.691 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4cbe55c:::benchmark_data_smithi122_119671_object51700:head# 2024-03-20T09:04:48.692 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4cbf418:::benchmark_data_smithi122_119671_object55955:head# 2024-03-20T09:04:48.692 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4cd07b7:::benchmark_data_smithi122_119671_object91351:head# 2024-03-20T09:04:48.692 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4cd3cee:::benchmark_data_smithi122_119671_object7986:head# 2024-03-20T09:04:48.693 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4cd6f19:::benchmark_data_smithi122_119671_object74443:head# 2024-03-20T09:04:48.693 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4cdb020:::benchmark_data_smithi122_119671_object89548:head# 2024-03-20T09:04:48.694 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4ce6c44:::benchmark_data_smithi122_119671_object95216:head# 2024-03-20T09:04:48.694 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4cef7bd:::benchmark_data_smithi122_119671_object59390:head# 2024-03-20T09:04:48.695 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4cefe6b:::benchmark_data_smithi122_119671_object80050:head# 2024-03-20T09:04:48.695 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4cf275a:::benchmark_data_smithi122_119671_object59915:head# 2024-03-20T09:04:48.695 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4cff128:::benchmark_data_smithi122_119671_object6761:head# 2024-03-20T09:04:48.696 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4d0bdc7:::benchmark_data_smithi122_119671_object95903:head# 2024-03-20T09:04:48.696 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4d14e45:::benchmark_data_smithi122_119671_object30327:head# 2024-03-20T09:04:48.697 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4d39ab4:::benchmark_data_smithi122_119671_object29376:head# 2024-03-20T09:04:48.697 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4d43719:::benchmark_data_smithi122_119671_object88602:head# 2024-03-20T09:04:48.698 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4d4c57d:::benchmark_data_smithi122_119671_object76418:head# 2024-03-20T09:04:48.698 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4d5aa97:::benchmark_data_smithi122_119671_object53460:head# 2024-03-20T09:04:48.698 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4d7c38a:::benchmark_data_smithi122_119671_object75067:head# 2024-03-20T09:04:48.699 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4d7ea46:::benchmark_data_smithi122_119671_object75578:head# 2024-03-20T09:04:48.699 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4d83b76:::benchmark_data_smithi122_119671_object50370:head# 2024-03-20T09:04:48.700 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4d9d1c4:::benchmark_data_smithi122_119671_object63936:head# 2024-03-20T09:04:48.700 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4da4772:::benchmark_data_smithi122_119671_object33984:head# 2024-03-20T09:04:48.701 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4db06c7:::benchmark_data_smithi122_119671_object28271:head# 2024-03-20T09:04:48.701 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4dc1e59:::benchmark_data_smithi122_119671_object45029:head# 2024-03-20T09:04:48.702 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4dccc3c:::benchmark_data_smithi122_119671_object28846:head# 2024-03-20T09:04:48.702 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4dd6f49:::benchmark_data_smithi122_119671_object3958:head# 2024-03-20T09:04:48.703 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4de38d7:::benchmark_data_smithi122_119671_object73646:head# 2024-03-20T09:04:48.703 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4de3ee5:::benchmark_data_smithi122_119671_object45870:head# 2024-03-20T09:04:48.704 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4dfe456:::benchmark_data_smithi122_119671_object15246:head# 2024-03-20T09:04:48.704 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4e0f8f6:::benchmark_data_smithi122_119671_object24465:head# 2024-03-20T09:04:48.704 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4e153d1:::benchmark_data_smithi122_119671_object20784:head# 2024-03-20T09:04:48.704 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4e2adaa:::benchmark_data_smithi122_119671_object47855:head# 2024-03-20T09:04:48.705 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4e2c87d:::benchmark_data_smithi122_119671_object10794:head# 2024-03-20T09:04:48.706 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4e5292c:::benchmark_data_smithi122_119671_object9157:head# 2024-03-20T09:04:48.706 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4e56904:::benchmark_data_smithi122_119671_object95809:head# 2024-03-20T09:04:48.707 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4e5c34c:::benchmark_data_smithi122_119671_object65287:head# 2024-03-20T09:04:48.707 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4e91def:::benchmark_data_smithi122_119671_object54733:head# 2024-03-20T09:04:48.707 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4ea49bf:::benchmark_data_smithi122_119671_object49242:head# 2024-03-20T09:04:48.708 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4ebca98:::benchmark_data_smithi122_119671_object3393:head# 2024-03-20T09:04:48.708 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4ec3559:::benchmark_data_smithi122_119671_object56823:head# 2024-03-20T09:04:48.708 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4ec64f6:::benchmark_data_smithi122_119671_object83094:head# 2024-03-20T09:04:48.709 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4ed1073:::benchmark_data_smithi122_119671_object1960:head# 2024-03-20T09:04:48.709 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4eda4c4:::benchmark_data_smithi122_119671_object55334:head# 2024-03-20T09:04:48.710 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4eff95f:::benchmark_data_smithi122_119671_object77132:head# 2024-03-20T09:04:48.710 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4f134fd:::benchmark_data_smithi122_119671_object52280:head# 2024-03-20T09:04:48.711 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4f29ac0:::benchmark_data_smithi122_119671_object32694:head# 2024-03-20T09:04:48.711 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4f31134:::benchmark_data_smithi122_119671_object92829:head# 2024-03-20T09:04:48.711 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4f45799:::benchmark_data_smithi122_119671_object26381:head# 2024-03-20T09:04:48.712 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4f46617:::benchmark_data_smithi122_119671_object80263:head# 2024-03-20T09:04:48.712 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4f5c17e:::benchmark_data_smithi122_119671_object6878:head# 2024-03-20T09:04:48.712 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4f655b7:::benchmark_data_smithi122_119671_object78151:head# 2024-03-20T09:04:48.713 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4f6d21a:::benchmark_data_smithi122_119671_object19696:head# 2024-03-20T09:04:48.713 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4f6daf3:::benchmark_data_smithi122_119671_object20093:head# 2024-03-20T09:04:48.713 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4f73307:::benchmark_data_smithi122_119671_object13908:head# 2024-03-20T09:04:48.714 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4f80d61:::benchmark_data_smithi122_119671_object62707:head# 2024-03-20T09:04:48.714 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4f8bb65:::benchmark_data_smithi122_119671_object68424:head# 2024-03-20T09:04:48.715 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4f90a58:::benchmark_data_smithi122_119671_object23739:head# 2024-03-20T09:04:48.715 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4f9cc29:::benchmark_data_smithi122_119671_object17170:head# 2024-03-20T09:04:48.715 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4fba6b1:::benchmark_data_smithi122_119671_object77526:head# 2024-03-20T09:04:48.716 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4fbb6f8:::benchmark_data_smithi122_119671_object64624:head# 2024-03-20T09:04:48.716 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4fbdfe8:::benchmark_data_smithi122_119671_object64422:head# 2024-03-20T09:04:48.717 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4fc1f31:::benchmark_data_smithi122_119671_object82149:head# 2024-03-20T09:04:48.718 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4fc7610:::benchmark_data_smithi122_119671_object4403:head# 2024-03-20T09:04:48.718 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4fe3d39:::benchmark_data_smithi122_119671_object94913:head# 2024-03-20T09:04:48.718 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4ff445c:::benchmark_data_smithi122_119671_object49960:head# 2024-03-20T09:04:48.718 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b4ffb9a8:::benchmark_data_smithi122_119671_object76465:head# 2024-03-20T09:04:48.719 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b50010f3:::benchmark_data_smithi122_119671_object13453:head# 2024-03-20T09:04:48.719 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b500ebe4:::benchmark_data_smithi122_119671_object77506:head# 2024-03-20T09:04:48.719 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b501744e:::benchmark_data_smithi122_119671_object3260:head# 2024-03-20T09:04:48.720 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b50200dc:::benchmark_data_smithi122_119671_object32759:head# 2024-03-20T09:04:48.720 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5021e22:::benchmark_data_smithi122_119671_object47047:head# 2024-03-20T09:04:48.721 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b502b180:::benchmark_data_smithi122_119671_object16209:head# 2024-03-20T09:04:48.721 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b504070c:::benchmark_data_smithi122_119671_object78345:head# 2024-03-20T09:04:48.721 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b505560d:::benchmark_data_smithi122_119671_object28231:head# 2024-03-20T09:04:48.722 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b506b9be:::benchmark_data_smithi122_119671_object72335:head# 2024-03-20T09:04:48.723 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b506c5da:::benchmark_data_smithi122_119671_object20225:head# 2024-03-20T09:04:48.723 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5071572:::benchmark_data_smithi122_119671_object40749:head# 2024-03-20T09:04:48.723 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b507b44e:::benchmark_data_smithi122_119671_object57779:head# 2024-03-20T09:04:48.724 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-20T09:04:48.725 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b508662d:::benchmark_data_smithi122_119671_object73683:head# 2024-03-20T09:04:48.725 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5091751:::benchmark_data_smithi122_119671_object1311:head# 2024-03-20T09:04:48.725 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b50c0536:::benchmark_data_smithi122_119671_object63034:head# 2024-03-20T09:04:48.725 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b50ca0f2:::benchmark_data_smithi122_119671_object9045:head# 2024-03-20T09:04:48.726 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b50cbdad:::benchmark_data_smithi122_119671_object68437:head# 2024-03-20T09:04:48.726 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b50ccb62:::benchmark_data_smithi122_119671_object62965:head# 2024-03-20T09:04:48.726 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b50ce447:::benchmark_data_smithi122_119671_object70960:head# 2024-03-20T09:04:48.726 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b50d9114:::benchmark_data_smithi122_119671_object11537:head# 2024-03-20T09:04:48.726 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b50dac26:::benchmark_data_smithi122_119671_object47831:head# 2024-03-20T09:04:48.731 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b50e21e3:::benchmark_data_smithi122_119671_object21133:head# 2024-03-20T09:04:48.731 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b50e395b:::benchmark_data_smithi122_119671_object48910:head# 2024-03-20T09:04:48.731 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b50f327c:::benchmark_data_smithi122_119671_object43327:head# 2024-03-20T09:04:48.731 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b50f55ad:::benchmark_data_smithi122_119671_object8714:head# 2024-03-20T09:04:48.731 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b51086dd:::benchmark_data_smithi122_119671_object75937:head# 2024-03-20T09:04:48.731 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5108b67:::benchmark_data_smithi122_119671_object30528:head# 2024-03-20T09:04:48.732 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b511cb78:::benchmark_data_smithi122_119671_object77447:head# 2024-03-20T09:04:48.732 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b511d033:::benchmark_data_smithi122_119671_object82806:head# 2024-03-20T09:04:48.732 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b512aebb:::benchmark_data_smithi122_119671_object5975:head# 2024-03-20T09:04:48.732 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b513360b:::benchmark_data_smithi122_119671_object11740:head# 2024-03-20T09:04:48.732 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5134172:::benchmark_data_smithi122_119671_object46691:head# 2024-03-20T09:04:48.732 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b51367d9:::benchmark_data_smithi122_119671_object95499:head# 2024-03-20T09:04:48.732 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:04:48.733 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5158f57:::benchmark_data_smithi122_119671_object13814:head# 2024-03-20T09:04:48.733 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5167312:::benchmark_data_smithi122_119671_object9258:head# 2024-03-20T09:04:48.734 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b516ca8e:::benchmark_data_smithi122_119671_object16018:head# 2024-03-20T09:04:48.734 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b516e8a8:::benchmark_data_smithi122_119671_object36402:head# 2024-03-20T09:04:48.734 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5176266:::benchmark_data_smithi122_119671_object29463:head# 2024-03-20T09:04:48.735 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b519cfcf:::benchmark_data_smithi122_119671_object14239:head# 2024-03-20T09:04:48.735 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b51a0eca:::benchmark_data_smithi122_119671_object81440:head# 2024-03-20T09:04:48.735 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b51a6a67:::benchmark_data_smithi122_119671_object77954:head# 2024-03-20T09:04:48.735 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b51ab924:::benchmark_data_smithi122_119671_object6775:head# 2024-03-20T09:04:48.736 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b51b0889:::benchmark_data_smithi122_119671_object1534:head# 2024-03-20T09:04:48.736 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b51b7c83:::benchmark_data_smithi122_119671_object77947:head# 2024-03-20T09:04:48.736 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b51c2b90:::benchmark_data_smithi122_119671_object11265:head# 2024-03-20T09:04:48.737 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b51d3f09:::benchmark_data_smithi122_119671_object26157:head# 2024-03-20T09:04:48.737 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b51d409f:::benchmark_data_smithi122_119671_object72740:head# 2024-03-20T09:04:48.738 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b51d7746:::benchmark_data_smithi122_119671_object82051:head# 2024-03-20T09:04:48.738 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b51f1757:::benchmark_data_smithi122_119671_object30705:head# 2024-03-20T09:04:48.739 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b51f5e24:::benchmark_data_smithi122_119671_object93765:head# 2024-03-20T09:04:48.739 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b51f9995:::benchmark_data_smithi122_119671_object71757:head# 2024-03-20T09:04:48.742 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b51ff1c0:::benchmark_data_smithi122_119671_object13429:head# 2024-03-20T09:04:48.742 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b520a0a8:::benchmark_data_smithi122_119671_object8148:head# 2024-03-20T09:04:48.743 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b520b4e8:::benchmark_data_smithi122_119671_object19824:head# 2024-03-20T09:04:48.743 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b521a0dd:::benchmark_data_smithi122_119671_object79880:head# 2024-03-20T09:04:48.744 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b521e6f2:::benchmark_data_smithi122_119671_object97306:head# 2024-03-20T09:04:48.744 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5221605:::benchmark_data_smithi122_119671_object34165:head# 2024-03-20T09:04:48.745 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5225003:::benchmark_data_smithi122_119671_object83444:head# 2024-03-20T09:04:48.745 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5233660:::benchmark_data_smithi122_119671_object97621:head# 2024-03-20T09:04:48.745 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5235a2b:::benchmark_data_smithi122_119671_object55980:head# 2024-03-20T09:04:48.746 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b523c8c5:::benchmark_data_smithi122_119671_object13133:head# 2024-03-20T09:04:48.747 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b523d164:::benchmark_data_smithi122_119671_object80707:head# 2024-03-20T09:04:48.747 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b523eeb7:::benchmark_data_smithi122_119671_object83204:head# 2024-03-20T09:04:48.748 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b523f5c9:::benchmark_data_smithi122_119671_object27301:head# 2024-03-20T09:04:48.748 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b524202c:::benchmark_data_smithi122_119671_object9508:head# 2024-03-20T09:04:48.750 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5247ba2:::benchmark_data_smithi122_119671_object88327:head# 2024-03-20T09:04:48.750 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b52517e5:::benchmark_data_smithi122_119671_object87390:head# 2024-03-20T09:04:48.752 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5267331:::benchmark_data_smithi122_119671_object7935:head# 2024-03-20T09:04:48.753 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5267f1e:::benchmark_data_smithi122_119671_object3310:head# 2024-03-20T09:04:48.753 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5283f15:::benchmark_data_smithi122_119671_object42876:head# 2024-03-20T09:04:48.753 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b52a4a59:::benchmark_data_smithi122_119671_object84585:head# 2024-03-20T09:04:48.754 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b52b259a:::benchmark_data_smithi122_119671_object36909:head# 2024-03-20T09:04:48.754 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b52dbc35:::benchmark_data_smithi122_119671_object45285:head# 2024-03-20T09:04:48.754 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b52f27c9:::benchmark_data_smithi122_119671_object62689:head# 2024-03-20T09:04:48.755 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b52fe60b:::benchmark_data_smithi122_119671_object93840:head# 2024-03-20T09:04:48.755 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5323f17:::benchmark_data_smithi122_119671_object57164:head# 2024-03-20T09:04:48.756 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b532a197:::benchmark_data_smithi122_119671_object84135:head# 2024-03-20T09:04:48.757 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b533de8a:::benchmark_data_smithi122_119671_object7347:head# 2024-03-20T09:04:48.757 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b534d1d1:::benchmark_data_smithi122_119671_object82514:head# 2024-03-20T09:04:48.757 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b534f22c:::benchmark_data_smithi122_119671_object69267:head# 2024-03-20T09:04:48.758 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5359370:::benchmark_data_smithi122_119671_object88489:head# 2024-03-20T09:04:48.758 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b536b057:::benchmark_data_smithi122_119671_object10833:head# 2024-03-20T09:04:48.758 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5372e63:::benchmark_data_smithi122_119671_object13168:head# 2024-03-20T09:04:48.759 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b537f180:::benchmark_data_smithi122_119671_object61362:head# 2024-03-20T09:04:48.759 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5384c90:::benchmark_data_smithi122_119671_object34028:head# 2024-03-20T09:04:48.760 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b539275d:::benchmark_data_smithi122_119671_object37089:head# 2024-03-20T09:04:48.760 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5394a94:::benchmark_data_smithi122_119671_object3400:head# 2024-03-20T09:04:48.760 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b53a0be4:::benchmark_data_smithi122_119671_object72401:head# 2024-03-20T09:04:48.761 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b53a5ddf:::benchmark_data_smithi122_119671_object88933:head# 2024-03-20T09:04:48.761 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:48.762 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b53ab526:::benchmark_data_smithi122_119671_object12264:head# 2024-03-20T09:04:48.762 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b53b34cd:::benchmark_data_smithi122_119671_object51775:head# 2024-03-20T09:04:48.762 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b53beddb:::benchmark_data_smithi122_119671_object67424:head# 2024-03-20T09:04:48.763 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b53c437b:::benchmark_data_smithi122_119671_object78349:head# 2024-03-20T09:04:48.763 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b53c6504:::benchmark_data_smithi122_119671_object79340:head# 2024-03-20T09:04:48.764 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b53ec34c:::benchmark_data_smithi122_119671_object84376:head# 2024-03-20T09:04:48.764 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b53ed7b4:::benchmark_data_smithi122_119671_object21030:head# 2024-03-20T09:04:48.764 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b540332f:::benchmark_data_smithi122_119671_object47721:head# 2024-03-20T09:04:48.765 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5412d43:::benchmark_data_smithi122_119671_object30161:head# 2024-03-20T09:04:48.766 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5419f63:::benchmark_data_smithi122_119671_object53046:head# 2024-03-20T09:04:48.766 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b541cb45:::benchmark_data_smithi122_119671_object57628:head# 2024-03-20T09:04:48.766 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b54363af:::benchmark_data_smithi122_119671_object36543:head# 2024-03-20T09:04:48.767 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b543b845:::benchmark_data_smithi122_119671_object7691:head# 2024-03-20T09:04:48.767 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b54476ba:::benchmark_data_smithi122_119671_object43448:head# 2024-03-20T09:04:48.767 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5451fa3:::benchmark_data_smithi122_119671_object44926:head# 2024-03-20T09:04:48.768 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5458b0b:::benchmark_data_smithi122_119671_object12859:head# 2024-03-20T09:04:48.768 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b545ad99:::benchmark_data_smithi122_119671_object24473:head# 2024-03-20T09:04:48.769 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b54675bc:::benchmark_data_smithi122_119671_object3780:head# 2024-03-20T09:04:48.769 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5471327:::benchmark_data_smithi122_119671_object92516:head# 2024-03-20T09:04:48.770 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b547391a:::benchmark_data_smithi122_119671_object90293:head# 2024-03-20T09:04:48.770 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b547a82e:::benchmark_data_smithi122_119671_object72309:head# 2024-03-20T09:04:48.770 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5481d0e:::benchmark_data_smithi122_119671_object48133:head# 2024-03-20T09:04:48.771 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b548a459:::benchmark_data_smithi122_119671_object97974:head# 2024-03-20T09:04:48.771 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b54928ea:::benchmark_data_smithi122_119671_object80031:head# 2024-03-20T09:04:48.772 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5499ddb:::benchmark_data_smithi122_119671_object84067:head# 2024-03-20T09:04:48.772 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b54a506e:::benchmark_data_smithi122_119671_object29168:head# 2024-03-20T09:04:48.773 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b54ac415:::benchmark_data_smithi122_119671_object24715:head# 2024-03-20T09:04:48.773 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b54b272a:::benchmark_data_smithi122_119671_object44851:head# 2024-03-20T09:04:48.774 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b54b3409:::benchmark_data_smithi122_119671_object53523:head# 2024-03-20T09:04:48.774 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b54c083a:::benchmark_data_smithi122_119671_object43793:head# 2024-03-20T09:04:48.774 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b54c80d0:::benchmark_data_smithi122_119671_object30674:head# 2024-03-20T09:04:48.775 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b54c8eef:::benchmark_data_smithi122_119671_object28934:head# 2024-03-20T09:04:48.775 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b54c9b2e:::benchmark_data_smithi122_119671_object81459:head# 2024-03-20T09:04:48.776 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b54cce4e:::benchmark_data_smithi122_119671_object16539:head# 2024-03-20T09:04:48.777 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b54e06a9:::benchmark_data_smithi122_119671_object80101:head# 2024-03-20T09:04:48.777 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b54e3fe9:::benchmark_data_smithi122_119671_object4541:head# 2024-03-20T09:04:48.777 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b54f2375:::benchmark_data_smithi122_119671_object65691:head# 2024-03-20T09:04:48.778 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b550130f:::benchmark_data_smithi122_119671_object82244:head# 2024-03-20T09:04:48.778 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5506eb3:::benchmark_data_smithi122_119671_object87885:head# 2024-03-20T09:04:48.779 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5508ace:::benchmark_data_smithi122_119671_object72089:head# 2024-03-20T09:04:48.779 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b550d2c5:::benchmark_data_smithi122_119671_object63703:head# 2024-03-20T09:04:48.780 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5513328:::benchmark_data_smithi122_119671_object37249:head# 2024-03-20T09:04:48.780 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5514e5d:::benchmark_data_smithi122_119671_object96147:head# 2024-03-20T09:04:48.780 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:04:48.781 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b551bb2b:::benchmark_data_smithi122_119671_object66990:head# 2024-03-20T09:04:48.781 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b553702d:::benchmark_data_smithi122_119671_object68868:head# 2024-03-20T09:04:48.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:48.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:04:48.784 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b55399f2:::benchmark_data_smithi122_119671_object67164:head# 2024-03-20T09:04:48.784 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5540ccf:::benchmark_data_smithi122_119671_object98788:head# 2024-03-20T09:04:48.784 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b554b46d:::benchmark_data_smithi122_119671_object1467:head# 2024-03-20T09:04:48.784 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b555580c:::benchmark_data_smithi122_119671_object43367:head# 2024-03-20T09:04:48.785 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b555668c:::benchmark_data_smithi122_119671_object27101:head# 2024-03-20T09:04:48.785 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b555e56d:::benchmark_data_smithi122_119671_object27702:head# 2024-03-20T09:04:48.785 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5564f21:::benchmark_data_smithi122_119671_object5333:head# 2024-03-20T09:04:48.786 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5567450:::benchmark_data_smithi122_119671_object92256:head# 2024-03-20T09:04:48.786 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5568e42:::benchmark_data_smithi122_119671_object36850:head# 2024-03-20T09:04:48.787 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b556b744:::benchmark_data_smithi122_119671_object57967:head# 2024-03-20T09:04:48.787 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b556c0aa:::benchmark_data_smithi122_119671_object81144:head# 2024-03-20T09:04:48.787 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b557c206:::benchmark_data_smithi122_119671_object77881:head# 2024-03-20T09:04:48.788 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5583444:::benchmark_data_smithi122_119671_object64305:head# 2024-03-20T09:04:48.788 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b559ce48:::benchmark_data_smithi122_119671_object54206:head# 2024-03-20T09:04:48.789 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b55a5444:::benchmark_data_smithi122_119671_object11348:head# 2024-03-20T09:04:48.789 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b55a7dba:::benchmark_data_smithi122_119671_object15991:head# 2024-03-20T09:04:48.790 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b55b9c1d:::benchmark_data_smithi122_119671_object56512:head# 2024-03-20T09:04:48.790 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b55cc7c1:::benchmark_data_smithi122_119671_object8975:head# 2024-03-20T09:04:48.791 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b55cd232:::benchmark_data_smithi122_119671_object43244:head# 2024-03-20T09:04:48.791 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b55d1876:::benchmark_data_smithi122_119671_object53919:head# 2024-03-20T09:04:48.792 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b55d40d8:::benchmark_data_smithi122_119671_object77790:head# 2024-03-20T09:04:48.792 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b55d7f21:::benchmark_data_smithi122_119671_object71126:head# 2024-03-20T09:04:48.792 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b55dfce4:::benchmark_data_smithi122_119671_object18168:head# 2024-03-20T09:04:48.793 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b55e569d:::benchmark_data_smithi122_119671_object75003:head# 2024-03-20T09:04:48.793 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b55f077b:::benchmark_data_smithi122_119671_object52638:head# 2024-03-20T09:04:48.793 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b55f7075:::benchmark_data_smithi122_119671_object34306:head# 2024-03-20T09:04:48.794 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b55f7c5c:::benchmark_data_smithi122_119671_object85621:head# 2024-03-20T09:04:48.794 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b55f86f3:::benchmark_data_smithi122_119671_object94758:head# 2024-03-20T09:04:48.795 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b55fbf5b:::benchmark_data_smithi122_119671_object44934:head# 2024-03-20T09:04:48.795 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b55fdb19:::benchmark_data_smithi122_119671_object64804:head# 2024-03-20T09:04:48.795 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5615a7f:::benchmark_data_smithi122_119671_object73142:head# 2024-03-20T09:04:48.796 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b56207b0:::benchmark_data_smithi122_119671_object55627:head# 2024-03-20T09:04:48.796 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5625664:::benchmark_data_smithi122_119671_object60468:head# 2024-03-20T09:04:48.797 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b562842e:::benchmark_data_smithi122_119671_object89178:head# 2024-03-20T09:04:48.797 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5645813:::benchmark_data_smithi122_119671_object9321:head# 2024-03-20T09:04:48.798 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5686717:::benchmark_data_smithi122_119671_object69607:head# 2024-03-20T09:04:48.798 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5689d84:::benchmark_data_smithi122_119671_object37931:head# 2024-03-20T09:04:48.799 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b568d2fc:::benchmark_data_smithi122_119671_object12459:head# 2024-03-20T09:04:48.800 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b56933cf:::benchmark_data_smithi122_119671_object64774:head# 2024-03-20T09:04:48.800 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b56a0052:::benchmark_data_smithi122_119671_object65088:head# 2024-03-20T09:04:48.801 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b56b36f9:::benchmark_data_smithi122_119671_object78283:head# 2024-03-20T09:04:48.801 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b56be2da:::benchmark_data_smithi122_119671_object30306:head# 2024-03-20T09:04:48.802 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b56ca53b:::benchmark_data_smithi122_119671_object47109:head# 2024-03-20T09:04:48.802 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b56cc376:::benchmark_data_smithi122_119671_object48776:head# 2024-03-20T09:04:48.802 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b56cf064:::benchmark_data_smithi122_119671_object10504:head# 2024-03-20T09:04:48.803 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b56e7b22:::benchmark_data_smithi122_119671_object56174:head# 2024-03-20T09:04:48.803 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b56f407d:::benchmark_data_smithi122_119671_object51069:head# 2024-03-20T09:04:48.804 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b56fd7c1:::benchmark_data_smithi122_119671_object27860:head# 2024-03-20T09:04:48.804 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b571673d:::benchmark_data_smithi122_119671_object64829:head# 2024-03-20T09:04:48.804 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b571a65e:::benchmark_data_smithi122_119671_object76867:head# 2024-03-20T09:04:48.805 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b571b08b:::benchmark_data_smithi122_119671_object36162:head# 2024-03-20T09:04:48.805 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b571f948:::benchmark_data_smithi122_119671_object74239:head# 2024-03-20T09:04:48.806 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b572318d:::benchmark_data_smithi122_119671_object36227:head# 2024-03-20T09:04:48.807 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5731d2e:::benchmark_data_smithi122_119671_object5687:head# 2024-03-20T09:04:48.808 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5751b50:::benchmark_data_smithi122_119671_object65399:head# 2024-03-20T09:04:48.808 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5757fea:::benchmark_data_smithi122_119671_object75770:head# 2024-03-20T09:04:48.808 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5759a5a:::benchmark_data_smithi122_119671_object44920:head# 2024-03-20T09:04:48.808 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5765564:::benchmark_data_smithi122_119671_object52017:head# 2024-03-20T09:04:48.809 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5773aeb:::benchmark_data_smithi122_119671_object83272:head# 2024-03-20T09:04:48.809 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b57875b0:::benchmark_data_smithi122_119671_object18553:head# 2024-03-20T09:04:48.809 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b579a71e:::benchmark_data_smithi122_119671_object74237:head# 2024-03-20T09:04:48.810 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b57b7f2a:::benchmark_data_smithi122_119671_object99222:head# 2024-03-20T09:04:48.810 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b57bddfc:::benchmark_data_smithi122_119671_object15281:head# 2024-03-20T09:04:48.811 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b57caf40:::benchmark_data_smithi122_119671_object91971:head# 2024-03-20T09:04:48.811 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b57e264c:::benchmark_data_smithi122_119671_object18749:head# 2024-03-20T09:04:48.811 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b57e7174:::benchmark_data_smithi122_119671_object93406:head# 2024-03-20T09:04:48.812 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b57eee26:::benchmark_data_smithi122_119671_object56439:head# 2024-03-20T09:04:48.812 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b57f1b68:::benchmark_data_smithi122_119671_object33242:head# 2024-03-20T09:04:48.813 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b57f417f:::benchmark_data_smithi122_119671_object24101:head# 2024-03-20T09:04:48.813 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5804184:::benchmark_data_smithi122_119671_object35938:head# 2024-03-20T09:04:48.814 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b580a075:::benchmark_data_smithi122_119671_object85471:head# 2024-03-20T09:04:48.814 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b58167fe:::benchmark_data_smithi122_119671_object97616:head# 2024-03-20T09:04:48.814 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b581b187:::benchmark_data_smithi122_119671_object99326:head# 2024-03-20T09:04:48.815 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b581f9e7:::benchmark_data_smithi122_119671_object52499:head# 2024-03-20T09:04:48.815 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5824f27:::benchmark_data_smithi122_119671_object17046:head# 2024-03-20T09:04:48.816 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5826053:::benchmark_data_smithi122_119671_object91140:head# 2024-03-20T09:04:48.816 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5829b74:::benchmark_data_smithi122_119671_object19780:head# 2024-03-20T09:04:48.817 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b582d336:::benchmark_data_smithi122_119671_object68805:head# 2024-03-20T09:04:48.817 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b583cc5a:::benchmark_data_smithi122_119671_object61983:head# 2024-03-20T09:04:48.817 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b583f9fa:::benchmark_data_smithi122_119671_object74171:head# 2024-03-20T09:04:48.818 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5847f74:::benchmark_data_smithi122_119671_object81808:head# 2024-03-20T09:04:48.818 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5848ebf:::benchmark_data_smithi122_119671_object97480:head# 2024-03-20T09:04:48.819 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b586a3bd:::benchmark_data_smithi122_119671_object41905:head# 2024-03-20T09:04:48.819 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b58812ee:::benchmark_data_smithi122_119671_object53416:head# 2024-03-20T09:04:48.819 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5886304:::benchmark_data_smithi122_119671_object67400:head# 2024-03-20T09:04:48.820 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5894d83:::benchmark_data_smithi122_119671_object81756:head# 2024-03-20T09:04:48.820 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5896369:::benchmark_data_smithi122_119671_object49210:head# 2024-03-20T09:04:48.820 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b589b7dd:::benchmark_data_smithi122_119671_object99744:head# 2024-03-20T09:04:48.821 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b589c534:::benchmark_data_smithi122_119671_object84266:head# 2024-03-20T09:04:48.821 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b58c0362:::benchmark_data_smithi122_119671_object9059:head# 2024-03-20T09:04:48.822 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b58fbdae:::benchmark_data_smithi122_119671_object91698:head# 2024-03-20T09:04:48.822 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5901cb8:::benchmark_data_smithi122_119671_object13553:head# 2024-03-20T09:04:48.823 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5906662:::benchmark_data_smithi122_119671_object6274:head# 2024-03-20T09:04:48.823 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b590bd1a:::benchmark_data_smithi122_119671_object11347:head# 2024-03-20T09:04:48.824 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b590dd48:::benchmark_data_smithi122_119671_object51068:head# 2024-03-20T09:04:48.824 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b591f103:::benchmark_data_smithi122_119671_object84653:head# 2024-03-20T09:04:48.826 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b592a88e:::benchmark_data_smithi122_119671_object12169:head# 2024-03-20T09:04:48.827 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b59330f5:::benchmark_data_smithi122_119671_object64642:head# 2024-03-20T09:04:48.827 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5950c58:::benchmark_data_smithi122_119671_object83548:head# 2024-03-20T09:04:48.828 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b59542f9:::benchmark_data_smithi122_119671_object22629:head# 2024-03-20T09:04:48.828 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5960d1a:::benchmark_data_smithi122_119671_object87044:head# 2024-03-20T09:04:48.829 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b596f078:::benchmark_data_smithi122_119671_object30968:head# 2024-03-20T09:04:48.829 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5970b4f:::benchmark_data_smithi122_119671_object56516:head# 2024-03-20T09:04:48.829 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5983928:::benchmark_data_smithi122_119671_object16288:head# 2024-03-20T09:04:48.830 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5989baf:::benchmark_data_smithi122_119671_object26226:head# 2024-03-20T09:04:48.830 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b598ddaa:::benchmark_data_smithi122_119671_object64062:head# 2024-03-20T09:04:48.830 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b59903e8:::benchmark_data_smithi122_119671_object88707:head# 2024-03-20T09:04:48.831 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b599290a:::benchmark_data_smithi122_119671_object91264:head# 2024-03-20T09:04:48.831 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b59a693f:::benchmark_data_smithi122_119671_object84338:head# 2024-03-20T09:04:48.832 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b59b1b29:::benchmark_data_smithi122_119671_object30436:head# 2024-03-20T09:04:48.832 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b59b32e5:::benchmark_data_smithi122_119671_object49777:head# 2024-03-20T09:04:48.833 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b59b5f57:::benchmark_data_smithi122_119671_object76977:head# 2024-03-20T09:04:48.833 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b59c89e4:::benchmark_data_smithi122_119671_object42411:head# 2024-03-20T09:04:48.833 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b59d2478:::benchmark_data_smithi122_119671_object9820:head# 2024-03-20T09:04:48.834 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5a1b26f:::benchmark_data_smithi122_119671_object43547:head# 2024-03-20T09:04:48.834 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5a27932:::benchmark_data_smithi122_119671_object59646:head# 2024-03-20T09:04:48.835 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5a39766:::benchmark_data_smithi122_119671_object3803:head# 2024-03-20T09:04:48.835 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5a42b20:::benchmark_data_smithi122_119671_object13117:head# 2024-03-20T09:04:48.835 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5a461eb:::benchmark_data_smithi122_119671_object9093:head# 2024-03-20T09:04:48.836 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5a46863:::benchmark_data_smithi122_119671_object6191:head# 2024-03-20T09:04:48.836 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5a499dc:::benchmark_data_smithi122_119671_object85343:head# 2024-03-20T09:04:48.837 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5a516c8:::benchmark_data_smithi122_119671_object44178:head# 2024-03-20T09:04:48.837 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5a64639:::benchmark_data_smithi122_119671_object68070:head# 2024-03-20T09:04:48.838 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5a6e76c:::benchmark_data_smithi122_119671_object88652:head# 2024-03-20T09:04:48.838 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5a72657:::benchmark_data_smithi122_119671_object37273:head# 2024-03-20T09:04:48.838 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5a73e61:::benchmark_data_smithi122_119671_object60648:head# 2024-03-20T09:04:48.839 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5a85d59:::benchmark_data_smithi122_119671_object89601:head# 2024-03-20T09:04:48.839 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5a88d77:::benchmark_data_smithi122_119671_object57733:head# 2024-03-20T09:04:48.840 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5a97a0f:::benchmark_data_smithi122_119671_object63063:head# 2024-03-20T09:04:48.840 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5aac35c:::benchmark_data_smithi122_119671_object86921:head# 2024-03-20T09:04:48.840 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5aae135:::benchmark_data_smithi122_119671_object73859:head# 2024-03-20T09:04:48.841 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5ab458a:::benchmark_data_smithi122_119671_object22566:head# 2024-03-20T09:04:48.841 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5abb906:::benchmark_data_smithi122_119671_object38854:head# 2024-03-20T09:04:48.842 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5ac462f:::benchmark_data_smithi122_119671_object21497:head# 2024-03-20T09:04:48.842 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5ac5345:::benchmark_data_smithi122_119671_object90809:head# 2024-03-20T09:04:48.843 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5ac9dea:::benchmark_data_smithi122_119671_object49100:head# 2024-03-20T09:04:48.843 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5ad8550:::benchmark_data_smithi122_119671_object41481:head# 2024-03-20T09:04:48.844 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5ad932f:::benchmark_data_smithi122_119671_object29921:head# 2024-03-20T09:04:48.844 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5ad9793:::benchmark_data_smithi122_119671_object66577:head# 2024-03-20T09:04:48.844 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5ada1cf:::benchmark_data_smithi122_119671_object85374:head# 2024-03-20T09:04:48.845 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5ae311b:::benchmark_data_smithi122_119671_object96413:head# 2024-03-20T09:04:48.845 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5b060f5:::benchmark_data_smithi122_119671_object19264:head# 2024-03-20T09:04:48.846 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5b0afd1:::benchmark_data_smithi122_119671_object6654:head# 2024-03-20T09:04:48.846 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5b0cfdb:::benchmark_data_smithi122_119671_object11908:head# 2024-03-20T09:04:48.846 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5b0d6f4:::benchmark_data_smithi122_119671_object75208:head# 2024-03-20T09:04:48.847 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5b1f71c:::benchmark_data_smithi122_119671_object57424:head# 2024-03-20T09:04:48.847 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5b21802:::benchmark_data_smithi122_119671_object90539:head# 2024-03-20T09:04:48.848 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5b21ddb:::benchmark_data_smithi122_119671_object49967:head# 2024-03-20T09:04:48.850 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5b32afc:::benchmark_data_smithi122_119671_object24764:head# 2024-03-20T09:04:48.850 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5b32bb1:::benchmark_data_smithi122_119671_object6850:head# 2024-03-20T09:04:48.850 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5b36d0c:::benchmark_data_smithi122_119671_object86351:head# 2024-03-20T09:04:48.850 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5b36e75:::benchmark_data_smithi122_119671_object83119:head# 2024-03-20T09:04:48.850 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5b3acb4:::benchmark_data_smithi122_119671_object30318:head# 2024-03-20T09:04:48.851 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5b40376:::benchmark_data_smithi122_119671_object81590:head# 2024-03-20T09:04:48.851 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5b607f7:::benchmark_data_smithi122_119671_object17808:head# 2024-03-20T09:04:48.851 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5b73e3d:::benchmark_data_smithi122_119671_object64312:head# 2024-03-20T09:04:48.852 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5b75eb0:::benchmark_data_smithi122_119671_object11843:head# 2024-03-20T09:04:48.852 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5b828ec:::benchmark_data_smithi122_119671_object32337:head# 2024-03-20T09:04:48.853 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5b91eee:::benchmark_data_smithi122_119671_object48764:head# 2024-03-20T09:04:48.853 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5ba1f0c:::benchmark_data_smithi122_119671_object57686:head# 2024-03-20T09:04:48.853 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5bac81d:::benchmark_data_smithi122_119671_object22808:head# 2024-03-20T09:04:48.854 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5bacc1b:::benchmark_data_smithi122_119671_object48344:head# 2024-03-20T09:04:48.854 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5bb0bb0:::benchmark_data_smithi122_119671_object11076:head# 2024-03-20T09:04:48.854 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5bb7a27:::benchmark_data_smithi122_119671_object60266:head# 2024-03-20T09:04:48.855 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5bc5753:::benchmark_data_smithi122_119671_object20800:head# 2024-03-20T09:04:48.855 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5bc8262:::benchmark_data_smithi122_119671_object26822:head# 2024-03-20T09:04:48.856 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5bd4238:::benchmark_data_smithi122_119671_object63871:head# 2024-03-20T09:04:48.856 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5bdc41f:::benchmark_data_smithi122_119671_object34853:head# 2024-03-20T09:04:48.857 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5becf51:::benchmark_data_smithi122_119671_object98496:head# 2024-03-20T09:04:48.857 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5bf2392:::benchmark_data_smithi122_119671_object61000:head# 2024-03-20T09:04:48.858 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5c0c3fb:::benchmark_data_smithi122_119671_object13289:head# 2024-03-20T09:04:48.861 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5c137dc:::benchmark_data_smithi122_119671_object1771:head# 2024-03-20T09:04:48.861 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5c21e7b:::benchmark_data_smithi122_119671_object90351:head# 2024-03-20T09:04:48.861 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5c2475a:::benchmark_data_smithi122_119671_object33565:head# 2024-03-20T09:04:48.861 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5c2ad17:::benchmark_data_smithi122_119671_object82924:head# 2024-03-20T09:04:48.862 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5c51456:::benchmark_data_smithi122_119671_object80812:head# 2024-03-20T09:04:48.862 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5c52c78:::benchmark_data_smithi122_119671_object53736:head# 2024-03-20T09:04:48.863 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5c52e04:::benchmark_data_smithi122_119671_object58957:head# 2024-03-20T09:04:48.863 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5c60f75:::benchmark_data_smithi122_119671_object41306:head# 2024-03-20T09:04:48.863 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5c61fa1:::benchmark_data_smithi122_119671_object2700:head# 2024-03-20T09:04:48.863 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5c70f1a:::benchmark_data_smithi122_119671_object47087:head# 2024-03-20T09:04:48.864 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5c81492:::benchmark_data_smithi122_119671_object9382:head# 2024-03-20T09:04:48.864 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5c846ba:::benchmark_data_smithi122_119671_object46212:head# 2024-03-20T09:04:48.864 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5c86702:::benchmark_data_smithi122_119671_object60093:head# 2024-03-20T09:04:48.865 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5c88b59:::benchmark_data_smithi122_119671_object45830:head# 2024-03-20T09:04:48.865 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5c8bf45:::benchmark_data_smithi122_119671_object83359:head# 2024-03-20T09:04:48.865 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5c91992:::benchmark_data_smithi122_119671_object24265:head# 2024-03-20T09:04:48.866 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5ca416b:::benchmark_data_smithi122_119671_object90987:head# 2024-03-20T09:04:48.866 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5caafa3:::benchmark_data_smithi122_119671_object91199:head# 2024-03-20T09:04:48.867 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5cd5769:::benchmark_data_smithi122_119671_object53138:head# 2024-03-20T09:04:48.867 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5cdb171:::benchmark_data_smithi122_119671_object55950:head# 2024-03-20T09:04:48.868 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5ce3f54:::benchmark_data_smithi122_119671_object54338:head# 2024-03-20T09:04:48.868 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5ce9bea:::benchmark_data_smithi122_119671_object55669:head# 2024-03-20T09:04:48.868 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5cf544d:::benchmark_data_smithi122_119671_object31447:head# 2024-03-20T09:04:48.869 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5cfa94e:::benchmark_data_smithi122_119671_object35955:head# 2024-03-20T09:04:48.869 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5cfc6c6:::benchmark_data_smithi122_119671_object17948:head# 2024-03-20T09:04:48.870 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5cffc77:::benchmark_data_smithi122_119671_object43763:head# 2024-03-20T09:04:48.871 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5d1541d:::benchmark_data_smithi122_119671_object97256:head# 2024-03-20T09:04:48.871 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5d1f7bc:::benchmark_data_smithi122_119671_object10263:head# 2024-03-20T09:04:48.871 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5d26c11:::benchmark_data_smithi122_119671_object55588:head# 2024-03-20T09:04:48.872 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5d2e9e7:::benchmark_data_smithi122_119671_object72878:head# 2024-03-20T09:04:48.872 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5d34eff:::benchmark_data_smithi122_119671_object66103:head# 2024-03-20T09:04:48.872 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5d3d8ec:::benchmark_data_smithi122_119671_object31530:head# 2024-03-20T09:04:48.873 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5d4b538:::benchmark_data_smithi122_119671_object57171:head# 2024-03-20T09:04:48.873 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5d4d8b6:::benchmark_data_smithi122_119671_object4218:head# 2024-03-20T09:04:48.874 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5d4e8b4:::benchmark_data_smithi122_119671_object16368:head# 2024-03-20T09:04:48.874 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5d56baf:::benchmark_data_smithi122_119671_object65643:head# 2024-03-20T09:04:48.875 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5d63feb:::benchmark_data_smithi122_119671_object45413:head# 2024-03-20T09:04:48.876 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5d67535:::benchmark_data_smithi122_119671_object86126:head# 2024-03-20T09:04:48.877 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5d68643:::benchmark_data_smithi122_119671_object22919:head# 2024-03-20T09:04:48.877 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5d70c2e:::benchmark_data_smithi122_119671_object66386:head# 2024-03-20T09:04:48.877 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5d7d271:::benchmark_data_smithi122_119671_object18020:head# 2024-03-20T09:04:48.877 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5d808e3:::benchmark_data_smithi122_119671_object66593:head# 2024-03-20T09:04:48.878 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5d88e55:::benchmark_data_smithi122_119671_object13932:head# 2024-03-20T09:04:48.878 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5d8efb7:::benchmark_data_smithi122_119671_object26995:head# 2024-03-20T09:04:48.878 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5d921a3:::benchmark_data_smithi122_119671_object87574:head# 2024-03-20T09:04:48.879 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5d9ea76:::benchmark_data_smithi122_119671_object5086:head# 2024-03-20T09:04:48.879 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5dd9be6:::benchmark_data_smithi122_119671_object29086:head# 2024-03-20T09:04:48.880 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5ddfba2:::benchmark_data_smithi122_119671_object71869:head# 2024-03-20T09:04:48.880 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5df3334:::benchmark_data_smithi122_119671_object17838:head# 2024-03-20T09:04:48.881 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5df3b4d:::benchmark_data_smithi122_119671_object92684:head# 2024-03-20T09:04:48.881 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5df4f28:::benchmark_data_smithi122_119671_object81653:head# 2024-03-20T09:04:48.882 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5e01d99:::benchmark_data_smithi122_119671_object12484:head# 2024-03-20T09:04:48.882 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5e0f1d5:::benchmark_data_smithi122_119671_object82660:head# 2024-03-20T09:04:48.883 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5e1bf40:::benchmark_data_smithi122_119671_object88277:head# 2024-03-20T09:04:48.883 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5e2043a:::benchmark_data_smithi122_119671_object27782:head# 2024-03-20T09:04:48.883 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5e22a4b:::benchmark_data_smithi122_119671_object6427:head# 2024-03-20T09:04:48.884 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:04:48.884 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5e2be8f:::benchmark_data_smithi122_119671_object25151:head# 2024-03-20T09:04:48.885 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5e359af:::benchmark_data_smithi122_119671_object65785:head# 2024-03-20T09:04:48.886 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5e360d1:::benchmark_data_smithi122_119671_object76924:head# 2024-03-20T09:04:48.887 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5e49f31:::benchmark_data_smithi122_119671_object10250:head# 2024-03-20T09:04:48.887 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5e4e0bc:::benchmark_data_smithi122_119671_object1119:head# 2024-03-20T09:04:48.887 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5e53f4f:::benchmark_data_smithi122_119671_object77089:head# 2024-03-20T09:04:48.888 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5e5a78e:::benchmark_data_smithi122_119671_object59573:head# 2024-03-20T09:04:48.888 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5e64715:::benchmark_data_smithi122_119671_object55445:head# 2024-03-20T09:04:48.888 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5e66c96:::benchmark_data_smithi122_119671_object86070:head# 2024-03-20T09:04:48.889 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5e69272:::benchmark_data_smithi122_119671_object4279:head# 2024-03-20T09:04:48.889 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5e72e4f:::benchmark_data_smithi122_119671_object63573:head# 2024-03-20T09:04:48.890 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5e73cb9:::benchmark_data_smithi122_119671_object59466:head# 2024-03-20T09:04:48.890 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5e82d0f:::benchmark_data_smithi122_119671_object43661:head# 2024-03-20T09:04:48.890 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5e87d59:::benchmark_data_smithi122_119671_object40184:head# 2024-03-20T09:04:48.891 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5e9031a:::benchmark_data_smithi122_119671_object76860:head# 2024-03-20T09:04:48.892 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5e9285e:::benchmark_data_smithi122_119671_object46857:head# 2024-03-20T09:04:48.892 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5e98e4f:::benchmark_data_smithi122_119671_object63473:head# 2024-03-20T09:04:48.892 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5e9a310:::benchmark_data_smithi122_119671_object39390:head# 2024-03-20T09:04:48.893 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5e9aa81:::benchmark_data_smithi122_119671_object69447:head# 2024-03-20T09:04:48.893 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5e9f261:::benchmark_data_smithi122_119671_object53826:head# 2024-03-20T09:04:48.893 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5e9fa6f:::benchmark_data_smithi122_119671_object84816:head# 2024-03-20T09:04:48.894 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5eacd91:::benchmark_data_smithi122_119671_object8380:head# 2024-03-20T09:04:48.894 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5ebc4c9:::benchmark_data_smithi122_119671_object28414:head# 2024-03-20T09:04:48.895 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5ebcdd2:::benchmark_data_smithi122_119671_object45257:head# 2024-03-20T09:04:48.895 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5ebf3c0:::benchmark_data_smithi122_119671_object8004:head# 2024-03-20T09:04:48.895 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5ec4fd9:::benchmark_data_smithi122_119671_object7278:head# 2024-03-20T09:04:48.896 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5ed547a:::benchmark_data_smithi122_119671_object71598:head# 2024-03-20T09:04:48.896 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5ed8806:::benchmark_data_smithi122_119671_object91241:head# 2024-03-20T09:04:48.897 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5edbd01:::benchmark_data_smithi122_119671_object80180:head# 2024-03-20T09:04:48.897 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5edef39:::benchmark_data_smithi122_119671_object66396:head# 2024-03-20T09:04:48.897 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5ee16dc:::benchmark_data_smithi122_119671_object57116:head# 2024-03-20T09:04:48.898 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5ee4be0:::benchmark_data_smithi122_119671_object43475:head# 2024-03-20T09:04:48.898 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5ef2f18:::benchmark_data_smithi122_119671_object85814:head# 2024-03-20T09:04:48.899 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5f01905:::benchmark_data_smithi122_119671_object77093:head# 2024-03-20T09:04:48.899 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5f04d9d:::benchmark_data_smithi122_119671_object81673:head# 2024-03-20T09:04:48.900 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5f14f39:::benchmark_data_smithi122_119671_object42022:head# 2024-03-20T09:04:48.900 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5f27b33:::benchmark_data_smithi122_119671_object88162:head# 2024-03-20T09:04:48.901 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5f3822c:::benchmark_data_smithi122_119671_object35167:head# 2024-03-20T09:04:48.901 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5f643fd:::benchmark_data_smithi122_119671_object23756:head# 2024-03-20T09:04:48.902 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5f7c58e:::benchmark_data_smithi122_119671_object92975:head# 2024-03-20T09:04:48.902 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5f7ca81:::benchmark_data_smithi122_119671_object16870:head# 2024-03-20T09:04:48.902 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5f8f94c:::benchmark_data_smithi122_119671_object39292:head# 2024-03-20T09:04:48.903 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5f97787:::benchmark_data_smithi122_119671_object7560:head# 2024-03-20T09:04:48.903 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5f9b957:::benchmark_data_smithi122_119671_object81941:head# 2024-03-20T09:04:48.904 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5fa786a:::benchmark_data_smithi122_119671_object50442:head# 2024-03-20T09:04:48.904 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5fbd960:::benchmark_data_smithi122_119671_object64113:head# 2024-03-20T09:04:48.904 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5fd9798:::benchmark_data_smithi122_119671_object75249:head# 2024-03-20T09:04:48.905 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5fed19a:::benchmark_data_smithi122_119671_object77836:head# 2024-03-20T09:04:48.905 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5ff3eed:::benchmark_data_smithi122_119671_object54353:head# 2024-03-20T09:04:48.906 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5ff5ac7:::benchmark_data_smithi122_119671_object15681:head# 2024-03-20T09:04:48.906 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b5ffeae3:::benchmark_data_smithi122_119671_object82873:head# 2024-03-20T09:04:48.907 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6011054:::benchmark_data_smithi122_119671_object52055:head# 2024-03-20T09:04:48.907 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6011c08:::benchmark_data_smithi122_119671_object16566:head# 2024-03-20T09:04:48.908 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b603129f:::benchmark_data_smithi122_119671_object75785:head# 2024-03-20T09:04:48.908 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6039258:::benchmark_data_smithi122_119671_object47006:head# 2024-03-20T09:04:48.909 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b60588c7:::benchmark_data_smithi122_119671_object21455:head# 2024-03-20T09:04:48.909 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b605b223:::benchmark_data_smithi122_119671_object62891:head# 2024-03-20T09:04:48.909 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b605d428:::benchmark_data_smithi122_119671_object31562:head# 2024-03-20T09:04:48.910 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6066da2:::benchmark_data_smithi122_119671_object65606:head# 2024-03-20T09:04:48.910 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6068276:::benchmark_data_smithi122_119671_object26605:head# 2024-03-20T09:04:48.911 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b606bc02:::benchmark_data_smithi122_119671_object21396:head# 2024-03-20T09:04:48.911 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b607f63d:::benchmark_data_smithi122_119671_object95098:head# 2024-03-20T09:04:48.912 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b608b61c:::benchmark_data_smithi122_119671_object32352:head# 2024-03-20T09:04:48.913 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b609c036:::benchmark_data_smithi122_119671_object83220:head# 2024-03-20T09:04:48.913 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:48.913 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b60ccc37:::benchmark_data_smithi122_119671_object95071:head# 2024-03-20T09:04:48.913 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b60d0189:::benchmark_data_smithi122_119671_object26977:head# 2024-03-20T09:04:48.914 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b60d0566:::benchmark_data_smithi122_119671_object73081:head# 2024-03-20T09:04:48.914 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b60d244c:::benchmark_data_smithi122_119671_object28858:head# 2024-03-20T09:04:48.915 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b60f951b:::benchmark_data_smithi122_119671_object57170:head# 2024-03-20T09:04:48.915 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b60fbbce:::benchmark_data_smithi122_119671_object35905:head# 2024-03-20T09:04:48.916 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6101524:::benchmark_data_smithi122_119671_object79617:head# 2024-03-20T09:04:48.916 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b611af16:::benchmark_data_smithi122_119671_object52843:head# 2024-03-20T09:04:48.917 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b612aecd:::benchmark_data_smithi122_119671_object17995:head# 2024-03-20T09:04:48.917 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b613f003:::benchmark_data_smithi122_119671_object79060:head# 2024-03-20T09:04:48.917 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b61404b7:::benchmark_data_smithi122_119671_object65965:head# 2024-03-20T09:04:48.918 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6146ba7:::benchmark_data_smithi122_119671_object63770:head# 2024-03-20T09:04:48.918 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6152e76:::benchmark_data_smithi122_119671_object60440:head# 2024-03-20T09:04:48.919 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b615a272:::benchmark_data_smithi122_119671_object66250:head# 2024-03-20T09:04:48.919 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b61727b0:::benchmark_data_smithi122_119671_object13700:head# 2024-03-20T09:04:48.920 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6182bc0:::benchmark_data_smithi122_119671_object88520:head# 2024-03-20T09:04:48.920 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b618c8ef:::benchmark_data_smithi122_119671_object49862:head# 2024-03-20T09:04:48.921 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b618e237:::benchmark_data_smithi122_119671_object83673:head# 2024-03-20T09:04:48.921 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b618e987:::benchmark_data_smithi122_119671_object90383:head# 2024-03-20T09:04:48.922 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b619908c:::benchmark_data_smithi122_119671_object38913:head# 2024-03-20T09:04:48.923 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b619b06d:::benchmark_data_smithi122_119671_object66972:head# 2024-03-20T09:04:48.923 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b619cdd0:::benchmark_data_smithi122_119671_object82480:head# 2024-03-20T09:04:48.923 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b61a15e9:::benchmark_data_smithi122_119671_object7551:head# 2024-03-20T09:04:48.923 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b61a21fc:::benchmark_data_smithi122_119671_object47499:head# 2024-03-20T09:04:48.924 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b61b8f83:::benchmark_data_smithi122_119671_object19470:head# 2024-03-20T09:04:48.924 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b61c7aea:::benchmark_data_smithi122_119671_object10563:head# 2024-03-20T09:04:48.925 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b61d2bb1:::benchmark_data_smithi122_119671_object65404:head# 2024-03-20T09:04:48.927 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b61e8c3b:::benchmark_data_smithi122_119671_object82789:head# 2024-03-20T09:04:48.927 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b61f2e23:::benchmark_data_smithi122_119671_object29808:head# 2024-03-20T09:04:48.928 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b61f42b8:::benchmark_data_smithi122_119671_object26959:head# 2024-03-20T09:04:48.928 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b61f6b2c:::benchmark_data_smithi122_119671_object34234:head# 2024-03-20T09:04:48.928 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b61f7c5c:::benchmark_data_smithi122_119671_object48775:head# 2024-03-20T09:04:48.928 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b61ff6d4:::benchmark_data_smithi122_119671_object67647:head# 2024-03-20T09:04:48.929 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6206bb5:::benchmark_data_smithi122_119671_object50750:head# 2024-03-20T09:04:48.929 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b621ca5d:::benchmark_data_smithi122_119671_object56795:head# 2024-03-20T09:04:48.929 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b621f406:::benchmark_data_smithi122_119671_object28931:head# 2024-03-20T09:04:48.929 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b62272e1:::benchmark_data_smithi122_119671_object40229:head# 2024-03-20T09:04:48.930 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6245765:::benchmark_data_smithi122_119671_object67892:head# 2024-03-20T09:04:48.934 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6250943:::benchmark_data_smithi122_119671_object67307:head# 2024-03-20T09:04:48.936 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b625fe73:::benchmark_data_smithi122_119671_object88076:head# 2024-03-20T09:04:48.936 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6282d8e:::benchmark_data_smithi122_119671_object53308:head# 2024-03-20T09:04:48.936 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b628495b:::benchmark_data_smithi122_119671_object95402:head# 2024-03-20T09:04:48.936 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b62864aa:::benchmark_data_smithi122_119671_object60574:head# 2024-03-20T09:04:48.936 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b629ddd9:::benchmark_data_smithi122_119671_object18645:head# 2024-03-20T09:04:48.936 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b62b18db:::benchmark_data_smithi122_119671_object96985:head# 2024-03-20T09:04:48.936 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b62bd262:::benchmark_data_smithi122_119671_object79714:head# 2024-03-20T09:04:48.937 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b62c9131:::benchmark_data_smithi122_119671_object52688:head# 2024-03-20T09:04:48.937 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b62ca02e:::benchmark_data_smithi122_119671_object59167:head# 2024-03-20T09:04:48.937 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b62d36c8:::benchmark_data_smithi122_119671_object8428:head# 2024-03-20T09:04:48.937 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b62e819b:::benchmark_data_smithi122_119671_object51863:head# 2024-03-20T09:04:48.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:48.940 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:04:48.940 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:04:48.942 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b62eecdf:::benchmark_data_smithi122_119671_object95219:head# 2024-03-20T09:04:48.942 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b62fc818:::benchmark_data_smithi122_119671_object62439:head# 2024-03-20T09:04:48.942 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6308e53:::benchmark_data_smithi122_119671_object29076:head# 2024-03-20T09:04:48.942 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b630abc8:::benchmark_data_smithi122_119671_object92731:head# 2024-03-20T09:04:48.942 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b630ec04:::benchmark_data_smithi122_119671_object60356:head# 2024-03-20T09:04:48.942 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b632ca04:::benchmark_data_smithi122_119671_object91937:head# 2024-03-20T09:04:48.942 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6343677:::benchmark_data_smithi122_119671_object83100:head# 2024-03-20T09:04:48.942 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b634473f:::benchmark_data_smithi122_119671_object78612:head# 2024-03-20T09:04:48.942 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6346a75:::benchmark_data_smithi122_119671_object69892:head# 2024-03-20T09:04:48.942 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6348385:::benchmark_data_smithi122_119671_object72359:head# 2024-03-20T09:04:48.942 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6350464:::benchmark_data_smithi122_119671_object27504:head# 2024-03-20T09:04:48.942 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6355441:::benchmark_data_smithi122_119671_object56633:head# 2024-03-20T09:04:48.943 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6362f40:::benchmark_data_smithi122_119671_object55436:head# 2024-03-20T09:04:48.943 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b63867a2:::benchmark_data_smithi122_119671_object68494:head# 2024-03-20T09:04:48.944 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b639f9de:::benchmark_data_smithi122_119671_object35018:head# 2024-03-20T09:04:48.944 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b63a1439:::benchmark_data_smithi122_119671_object78956:head# 2024-03-20T09:04:48.945 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b63a8a0b:::benchmark_data_smithi122_119671_object11869:head# 2024-03-20T09:04:48.945 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b63b0d6d:::benchmark_data_smithi122_119671_object80174:head# 2024-03-20T09:04:48.946 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b63b2b6c:::benchmark_data_smithi122_119671_object70074:head# 2024-03-20T09:04:48.946 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b63bec39:::benchmark_data_smithi122_119671_object7014:head# 2024-03-20T09:04:48.946 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b63d0b89:::benchmark_data_smithi122_119671_object98448:head# 2024-03-20T09:04:48.947 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b63d657b:::benchmark_data_smithi122_119671_object53406:head# 2024-03-20T09:04:48.947 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b63fc100:::benchmark_data_smithi122_119671_object14437:head# 2024-03-20T09:04:48.948 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6406233:::benchmark_data_smithi122_119671_object79403:head# 2024-03-20T09:04:48.948 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6412aea:::benchmark_data_smithi122_119671_object58959:head# 2024-03-20T09:04:48.949 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:48 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:04:48.949 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:48 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:04:48.949 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:48 smithi026 ceph-mon[68309]: osdmap e284: 12 total, 11 up, 11 in 2024-03-20T09:04:48.949 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:48 smithi026 ceph-mon[68309]: 3.19 scrub ok 2024-03-20T09:04:48.949 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:48 smithi026 ceph-mon[68309]: 1.13 scrub starts 2024-03-20T09:04:48.949 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:48 smithi026 ceph-mon[68309]: 1.13 scrub ok 2024-03-20T09:04:48.950 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b641813b:::benchmark_data_smithi122_119671_object96891:head# 2024-03-20T09:04:48.950 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b641a352:::benchmark_data_smithi122_119671_object69706:head# 2024-03-20T09:04:48.950 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6429026:::benchmark_data_smithi122_119671_object57152:head# 2024-03-20T09:04:48.950 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b642cac2:::benchmark_data_smithi122_119671_object59644:head# 2024-03-20T09:04:48.951 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b643199e:::benchmark_data_smithi122_119671_object3111:head# 2024-03-20T09:04:48.951 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6439e7f:::benchmark_data_smithi122_119671_object19716:head# 2024-03-20T09:04:48.952 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b643bfee:::benchmark_data_smithi122_119671_object33949:head# 2024-03-20T09:04:48.952 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b645fdd0:::benchmark_data_smithi122_119671_object17194:head# 2024-03-20T09:04:48.953 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6473168:::benchmark_data_smithi122_119671_object31623:head# 2024-03-20T09:04:48.953 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b64775b7:::benchmark_data_smithi122_119671_object7725:head# 2024-03-20T09:04:48.954 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b647f1d7:::benchmark_data_smithi122_119671_object96231:head# 2024-03-20T09:04:48.954 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6487364:::benchmark_data_smithi122_119671_object46692:head# 2024-03-20T09:04:48.954 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b648d189:::benchmark_data_smithi122_119671_object80572:head# 2024-03-20T09:04:48.955 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b64966e5:::benchmark_data_smithi122_119671_object93087:head# 2024-03-20T09:04:48.955 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b64a4f88:::benchmark_data_smithi122_119671_object94833:head# 2024-03-20T09:04:48.956 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b64ac886:::benchmark_data_smithi122_119671_object83395:head# 2024-03-20T09:04:48.956 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b64b1dfd:::benchmark_data_smithi122_119671_object9564:head# 2024-03-20T09:04:48.957 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b64b5e3a:::benchmark_data_smithi122_119671_object86738:head# 2024-03-20T09:04:48.957 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b64c57fa:::benchmark_data_smithi122_119671_object85754:head# 2024-03-20T09:04:48.957 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b64de769:::benchmark_data_smithi122_119671_object91397:head# 2024-03-20T09:04:48.958 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b64e3edb:::benchmark_data_smithi122_119671_object41424:head# 2024-03-20T09:04:48.958 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b64fed01:::benchmark_data_smithi122_119671_object65456:head# 2024-03-20T09:04:48.959 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b650336b:::benchmark_data_smithi122_119671_object18755:head# 2024-03-20T09:04:48.959 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b65060fb:::benchmark_data_smithi122_119671_object34479:head# 2024-03-20T09:04:48.959 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6512ef1:::benchmark_data_smithi122_119671_object62779:head# 2024-03-20T09:04:48.960 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b65161d9:::benchmark_data_smithi122_119671_object64403:head# 2024-03-20T09:04:48.960 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b652589e:::benchmark_data_smithi122_119671_object36506:head# 2024-03-20T09:04:48.961 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6526170:::benchmark_data_smithi122_119671_object71322:head# 2024-03-20T09:04:48.961 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b652a48f:::benchmark_data_smithi122_119671_object70819:head# 2024-03-20T09:04:48.962 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6540eed:::benchmark_data_smithi122_119671_object61529:head# 2024-03-20T09:04:48.963 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6542375:::benchmark_data_smithi122_119671_object39393:head# 2024-03-20T09:04:48.963 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6544cd7:::benchmark_data_smithi122_119671_object24061:head# 2024-03-20T09:04:48.963 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b65456ae:::benchmark_data_smithi122_119671_object7261:head# 2024-03-20T09:04:48.963 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6556b9d:::benchmark_data_smithi122_119671_object58120:head# 2024-03-20T09:04:48.963 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6581863:::benchmark_data_smithi122_119671_object81577:head# 2024-03-20T09:04:48.964 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b658ed87:::benchmark_data_smithi122_119671_object93236:head# 2024-03-20T09:04:48.964 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b659b2af:::benchmark_data_smithi122_119671_object16333:head# 2024-03-20T09:04:48.965 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b659da76:::benchmark_data_smithi122_119671_object95383:head# 2024-03-20T09:04:48.966 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b65af8d1:::benchmark_data_smithi122_119671_object47798:head# 2024-03-20T09:04:48.966 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b65bf206:::benchmark_data_smithi122_119671_object58390:head# 2024-03-20T09:04:48.967 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b65c0dfd:::benchmark_data_smithi122_119671_object86211:head# 2024-03-20T09:04:48.967 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b65c3e48:::benchmark_data_smithi122_119671_object90247:head# 2024-03-20T09:04:48.967 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b65c8180:::benchmark_data_smithi122_119671_object12101:head# 2024-03-20T09:04:48.967 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b65d595b:::benchmark_data_smithi122_119671_object65030:head# 2024-03-20T09:04:48.968 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b65d5ea3:::benchmark_data_smithi122_119671_object72199:head# 2024-03-20T09:04:48.968 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b65d66be:::benchmark_data_smithi122_119671_object24490:head# 2024-03-20T09:04:48.968 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b65db49f:::benchmark_data_smithi122_119671_object24549:head# 2024-03-20T09:04:48.969 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b65ea3bb:::benchmark_data_smithi122_119671_object17007:head# 2024-03-20T09:04:48.969 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b65ec451:::benchmark_data_smithi122_119671_object66491:head# 2024-03-20T09:04:48.970 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b65edd75:::benchmark_data_smithi122_119671_object94776:head# 2024-03-20T09:04:48.970 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b65f843b:::benchmark_data_smithi122_119671_object62185:head# 2024-03-20T09:04:48.970 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6604524:::benchmark_data_smithi122_119671_object34868:head# 2024-03-20T09:04:48.971 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b660fbc6:::benchmark_data_smithi122_119671_object96307:head# 2024-03-20T09:04:48.971 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6621860:::benchmark_data_smithi122_119671_object2726:head# 2024-03-20T09:04:48.972 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b66220b0:::benchmark_data_smithi122_119671_object39768:head# 2024-03-20T09:04:48.972 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6629e67:::benchmark_data_smithi122_119671_object65757:head# 2024-03-20T09:04:48.972 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b663473f:::benchmark_data_smithi122_119671_object75434:head# 2024-03-20T09:04:48.973 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b66347d2:::benchmark_data_smithi122_119671_object96037:head# 2024-03-20T09:04:48.973 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b663fe06:::benchmark_data_smithi122_119671_object93881:head# 2024-03-20T09:04:48.974 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6649020:::benchmark_data_smithi122_119671_object87326:head# 2024-03-20T09:04:48.974 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b664b66b:::benchmark_data_smithi122_119671_object83902:head# 2024-03-20T09:04:48.974 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6651862:::benchmark_data_smithi122_119671_object38583:head# 2024-03-20T09:04:48.975 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b66551c9:::benchmark_data_smithi122_119671_object4990:head# 2024-03-20T09:04:48.975 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6658e3f:::benchmark_data_smithi122_119671_object27071:head# 2024-03-20T09:04:48.976 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b665c38a:::benchmark_data_smithi122_119671_object56737:head# 2024-03-20T09:04:48.976 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6662437:::benchmark_data_smithi122_119671_object32826:head# 2024-03-20T09:04:48.976 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b666db4d:::benchmark_data_smithi122_119671_object93163:head# 2024-03-20T09:04:48.977 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6677217:::benchmark_data_smithi122_119671_object37487:head# 2024-03-20T09:04:48.977 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b667e78a:::benchmark_data_smithi122_119671_object68529:head# 2024-03-20T09:04:48.978 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b667f43d:::benchmark_data_smithi122_119671_object36829:head# 2024-03-20T09:04:48.978 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b667f89b:::benchmark_data_smithi122_119671_object79733:head# 2024-03-20T09:04:48.978 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6688184:::benchmark_data_smithi122_119671_object74203:head# 2024-03-20T09:04:48.979 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6696332:::benchmark_data_smithi122_119671_object56683:head# 2024-03-20T09:04:48.979 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b66a898a:::benchmark_data_smithi122_119671_object13783:head# 2024-03-20T09:04:48.980 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b66c788b:::benchmark_data_smithi122_119671_object48478:head# 2024-03-20T09:04:48.980 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b66d65e5:::benchmark_data_smithi122_119671_object31428:head# 2024-03-20T09:04:48.980 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b66e02b1:::benchmark_data_smithi122_119671_object18461:head# 2024-03-20T09:04:48.981 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b66e9f68:::benchmark_data_smithi122_119671_object1743:head# 2024-03-20T09:04:48.981 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b66f1d50:::benchmark_data_smithi122_119671_object10218:head# 2024-03-20T09:04:48.982 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b66f2f90:::benchmark_data_smithi122_119671_object16982:head# 2024-03-20T09:04:48.982 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b66f3793:::benchmark_data_smithi122_119671_object30244:head# 2024-03-20T09:04:48.983 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b66f4f45:::benchmark_data_smithi122_119671_object32729:head# 2024-03-20T09:04:48.983 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b66f93aa:::benchmark_data_smithi122_119671_object83147:head# 2024-03-20T09:04:48.983 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6711729:::benchmark_data_smithi122_119671_object44675:head# 2024-03-20T09:04:48.985 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b67178c6:::benchmark_data_smithi122_119671_object61610:head# 2024-03-20T09:04:48.985 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6748830:::benchmark_data_smithi122_119671_object95855:head# 2024-03-20T09:04:48.985 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b67521da:::benchmark_data_smithi122_119671_object16342:head# 2024-03-20T09:04:48.986 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b675948a:::benchmark_data_smithi122_119671_object67704:head# 2024-03-20T09:04:48.986 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b675d941:::benchmark_data_smithi122_119671_object26159:head# 2024-03-20T09:04:48.986 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6790c76:::benchmark_data_smithi122_119671_object23206:head# 2024-03-20T09:04:48.988 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b67b687c:::benchmark_data_smithi122_119671_object31727:head# 2024-03-20T09:04:48.988 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b67bde42:::benchmark_data_smithi122_119671_object42488:head# 2024-03-20T09:04:48.988 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b67ccbae:::benchmark_data_smithi122_119671_object68135:head# 2024-03-20T09:04:48.988 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b67f886d:::benchmark_data_smithi122_119671_object92084:head# 2024-03-20T09:04:48.988 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6803bbc:::benchmark_data_smithi122_119671_object62574:head# 2024-03-20T09:04:48.989 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6807780:::benchmark_data_smithi122_119671_object94707:head# 2024-03-20T09:04:48.989 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6818d66:::benchmark_data_smithi122_119671_object5772:head# 2024-03-20T09:04:48.990 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b681edcc:::benchmark_data_smithi122_119671_object11742:head# 2024-03-20T09:04:48.990 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6822dd1:::benchmark_data_smithi122_119671_object26295:head# 2024-03-20T09:04:48.990 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6827131:::benchmark_data_smithi122_119671_object98492:head# 2024-03-20T09:04:48.991 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6831441:::benchmark_data_smithi122_119671_object26037:head# 2024-03-20T09:04:48.991 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b68379ee:::benchmark_data_smithi122_119671_object55230:head# 2024-03-20T09:04:48.992 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6838d97:::benchmark_data_smithi122_119671_object31900:head# 2024-03-20T09:04:48.992 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6839736:::benchmark_data_smithi122_119671_object5076:head# 2024-03-20T09:04:48.992 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b683d59d:::benchmark_data_smithi122_119671_object53337:head# 2024-03-20T09:04:48.993 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6841ffe:::benchmark_data_smithi122_119671_object12557:head# 2024-03-20T09:04:48.993 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b684a295:::benchmark_data_smithi122_119671_object53871:head# 2024-03-20T09:04:48.993 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b685d6b1:::benchmark_data_smithi122_119671_object64395:head# 2024-03-20T09:04:48.994 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6862f28:::benchmark_data_smithi122_119671_object89864:head# 2024-03-20T09:04:48.994 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b68797e9:::benchmark_data_smithi122_119671_object87929:head# 2024-03-20T09:04:48.995 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b687a779:::benchmark_data_smithi122_119671_object68796:head# 2024-03-20T09:04:48.996 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b687bbc2:::benchmark_data_smithi122_119671_object40383:head# 2024-03-20T09:04:48.996 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b687eded:::benchmark_data_smithi122_119671_object70983:head# 2024-03-20T09:04:48.996 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b68920c7:::benchmark_data_smithi122_119671_object31274:head# 2024-03-20T09:04:48.997 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b689757a:::benchmark_data_smithi122_119671_object3719:head# 2024-03-20T09:04:48.997 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b68a505d:::benchmark_data_smithi122_119671_object87690:head# 2024-03-20T09:04:48.998 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b68a6534:::benchmark_data_smithi122_119671_object19322:head# 2024-03-20T09:04:48.998 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b68adf22:::benchmark_data_smithi122_119671_object73949:head# 2024-03-20T09:04:48.998 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b68bac2b:::benchmark_data_smithi122_119671_object58780:head# 2024-03-20T09:04:48.999 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b68bfa7a:::benchmark_data_smithi122_119671_object97241:head# 2024-03-20T09:04:48.999 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b68ca38b:::benchmark_data_smithi122_119671_object4460:head# 2024-03-20T09:04:49.000 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b68e1ca3:::benchmark_data_smithi122_119671_object45229:head# 2024-03-20T09:04:49.000 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b68e9682:::benchmark_data_smithi122_119671_object96190:head# 2024-03-20T09:04:49.001 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b68ee67d:::benchmark_data_smithi122_119671_object96377:head# 2024-03-20T09:04:49.001 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b68f5d2d:::benchmark_data_smithi122_119671_object28227:head# 2024-03-20T09:04:49.001 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6904fc1:::benchmark_data_smithi122_119671_object78475:head# 2024-03-20T09:04:49.002 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b690d77a:::benchmark_data_smithi122_119671_object48918:head# 2024-03-20T09:04:49.004 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b691190b:::benchmark_data_smithi122_119671_object95732:head# 2024-03-20T09:04:49.004 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b69128e4:::benchmark_data_smithi122_119671_object51529:head# 2024-03-20T09:04:49.004 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6937ecb:::benchmark_data_smithi122_119671_object52742:head# 2024-03-20T09:04:49.004 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b69453db:::benchmark_data_smithi122_119671_object94769:head# 2024-03-20T09:04:49.004 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6945750:::benchmark_data_smithi122_119671_object57260:head# 2024-03-20T09:04:49.005 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b695836c:::benchmark_data_smithi122_119671_object95433:head# 2024-03-20T09:04:49.005 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6967475:::benchmark_data_smithi122_119671_object78401:head# 2024-03-20T09:04:49.005 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6970255:::benchmark_data_smithi122_119671_object94040:head# 2024-03-20T09:04:49.006 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6972935:::benchmark_data_smithi122_119671_object81794:head# 2024-03-20T09:04:49.006 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b69803be:::benchmark_data_smithi122_119671_object64860:head# 2024-03-20T09:04:49.006 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b69845ef:::benchmark_data_smithi122_119671_object98069:head# 2024-03-20T09:04:49.007 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b698ead3:::benchmark_data_smithi122_119671_object10984:head# 2024-03-20T09:04:49.007 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6990b66:::benchmark_data_smithi122_119671_object68955:head# 2024-03-20T09:04:49.007 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6991b20:::benchmark_data_smithi122_119671_object73671:head# 2024-03-20T09:04:49.008 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6994a58:::benchmark_data_smithi122_119671_object80014:head# 2024-03-20T09:04:49.008 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6996f48:::benchmark_data_smithi122_119671_object5489:head# 2024-03-20T09:04:49.010 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b69a6d61:::benchmark_data_smithi122_119671_object25745:head# 2024-03-20T09:04:49.010 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b69a77c6:::benchmark_data_smithi122_119671_object48625:head# 2024-03-20T09:04:49.010 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b69aec82:::benchmark_data_smithi122_119671_object51347:head# 2024-03-20T09:04:49.010 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b69af119:::benchmark_data_smithi122_119671_object67757:head# 2024-03-20T09:04:49.011 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b69b347c:::benchmark_data_smithi122_119671_object35061:head# 2024-03-20T09:04:49.011 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b69b9f47:::benchmark_data_smithi122_119671_object60571:head# 2024-03-20T09:04:49.011 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b69bdcf6:::benchmark_data_smithi122_119671_object57439:head# 2024-03-20T09:04:49.012 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b69c1f31:::benchmark_data_smithi122_119671_object29007:head# 2024-03-20T09:04:49.012 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b69c3015:::benchmark_data_smithi122_119671_object32073:head# 2024-03-20T09:04:49.012 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b69cc08b:::benchmark_data_smithi122_119671_object84586:head# 2024-03-20T09:04:49.013 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b69cc39c:::benchmark_data_smithi122_119671_object7260:head# 2024-03-20T09:04:49.013 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b69de826:::benchmark_data_smithi122_119671_object97096:head# 2024-03-20T09:04:49.013 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b69e53e1:::benchmark_data_smithi122_119671_object2815:head# 2024-03-20T09:04:49.014 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b69f0204:::benchmark_data_smithi122_119671_object72179:head# 2024-03-20T09:04:49.014 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b69f2a26:::benchmark_data_smithi122_119671_object38157:head# 2024-03-20T09:04:49.015 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b69ffac9:::benchmark_data_smithi122_119671_object32053:head# 2024-03-20T09:04:49.015 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6a06066:::benchmark_data_smithi122_119671_object32262:head# 2024-03-20T09:04:49.015 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6a06798:::benchmark_data_smithi122_119671_object32412:head# 2024-03-20T09:04:49.016 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6a10b73:::benchmark_data_smithi122_119671_object88485:head# 2024-03-20T09:04:49.016 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6a12f3b:::benchmark_data_smithi122_119671_object76149:head# 2024-03-20T09:04:49.017 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6a4201a:::benchmark_data_smithi122_119671_object40497:head# 2024-03-20T09:04:49.017 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6a486f6:::benchmark_data_smithi122_119671_object31575:head# 2024-03-20T09:04:49.017 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6a4873e:::benchmark_data_smithi122_119671_object62221:head# 2024-03-20T09:04:49.018 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6a4e865:::benchmark_data_smithi122_119671_object76934:head# 2024-03-20T09:04:49.018 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6a55ea6:::benchmark_data_smithi122_119671_object49770:head# 2024-03-20T09:04:49.019 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6a5630f:::benchmark_data_smithi122_119671_object82684:head# 2024-03-20T09:04:49.019 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6a62d4f:::benchmark_data_smithi122_119671_object83738:head# 2024-03-20T09:04:49.021 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6a64d93:::benchmark_data_smithi122_119671_object31701:head# 2024-03-20T09:04:49.021 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6a72cda:::benchmark_data_smithi122_119671_object56413:head# 2024-03-20T09:04:49.021 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6a7e67a:::benchmark_data_smithi122_119671_object53203:head# 2024-03-20T09:04:49.021 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6a98cb5:::benchmark_data_smithi122_119671_object19179:head# 2024-03-20T09:04:49.022 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6a9c522:::benchmark_data_smithi122_119671_object4151:head# 2024-03-20T09:04:49.022 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6aab175:::benchmark_data_smithi122_119671_object89144:head# 2024-03-20T09:04:49.022 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6ad8447:::benchmark_data_smithi122_119671_object15218:head# 2024-03-20T09:04:49.022 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6adbe03:::benchmark_data_smithi122_119671_object73352:head# 2024-03-20T09:04:49.023 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6af12ae:::benchmark_data_smithi122_119671_object45139:head# 2024-03-20T09:04:49.023 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6af62e2:::benchmark_data_smithi122_119671_object55779:head# 2024-03-20T09:04:49.026 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6b0295c:::benchmark_data_smithi122_119671_object75272:head# 2024-03-20T09:04:49.026 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6b038fd:::benchmark_data_smithi122_119671_object66727:head# 2024-03-20T09:04:49.026 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6b1842d:::benchmark_data_smithi122_119671_object6584:head# 2024-03-20T09:04:49.027 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6b1ba6f:::benchmark_data_smithi122_119671_object17286:head# 2024-03-20T09:04:49.027 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6b2d164:::benchmark_data_smithi122_119671_object2617:head# 2024-03-20T09:04:49.028 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6b531e0:::benchmark_data_smithi122_119671_object91376:head# 2024-03-20T09:04:49.028 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6b69cf4:::benchmark_data_smithi122_119671_object28221:head# 2024-03-20T09:04:49.029 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6b72bd1:::benchmark_data_smithi122_119671_object28358:head# 2024-03-20T09:04:49.029 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6b7a8e1:::benchmark_data_smithi122_119671_object91027:head# 2024-03-20T09:04:49.030 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6b86467:::benchmark_data_smithi122_119671_object51767:head# 2024-03-20T09:04:49.030 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6b894cf:::benchmark_data_smithi122_119671_object49300:head# 2024-03-20T09:04:49.031 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6b8c83d:::benchmark_data_smithi122_119671_object3799:head# 2024-03-20T09:04:49.031 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6b9755f:::benchmark_data_smithi122_119671_object11852:head# 2024-03-20T09:04:49.032 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6bb4444:::benchmark_data_smithi122_119671_object77959:head# 2024-03-20T09:04:49.032 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6bbe393:::benchmark_data_smithi122_119671_object52254:head# 2024-03-20T09:04:49.033 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6bd2f43:::benchmark_data_smithi122_119671_object77487:head# 2024-03-20T09:04:49.033 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6bdd2b5:::benchmark_data_smithi122_119671_object3257:head# 2024-03-20T09:04:49.033 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6beb3a8:::benchmark_data_smithi122_119671_object14210:head# 2024-03-20T09:04:49.034 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6c02e0d:::benchmark_data_smithi122_119671_object15939:head# 2024-03-20T09:04:49.035 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6c0700f:::benchmark_data_smithi122_119671_object71547:head# 2024-03-20T09:04:49.035 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6c0a8db:::benchmark_data_smithi122_119671_object45043:head# 2024-03-20T09:04:49.035 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6c18ed2:::benchmark_data_smithi122_119671_object88955:head# 2024-03-20T09:04:49.036 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6c1e2fc:::benchmark_data_smithi122_119671_object8856:head# 2024-03-20T09:04:49.036 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6c24fbc:::benchmark_data_smithi122_119671_object50722:head# 2024-03-20T09:04:49.037 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6c2a9c7:::benchmark_data_smithi122_119671_object62937:head# 2024-03-20T09:04:49.037 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6c44794:::benchmark_data_smithi122_119671_object35711:head# 2024-03-20T09:04:49.038 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6c46534:::benchmark_data_smithi122_119671_object68033:head# 2024-03-20T09:04:49.038 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6c4e447:::benchmark_data_smithi122_119671_object90927:head# 2024-03-20T09:04:49.039 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6c582db:::benchmark_data_smithi122_119671_object7242:head# 2024-03-20T09:04:49.039 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6c72516:::benchmark_data_smithi122_119671_object99243:head# 2024-03-20T09:04:49.040 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6c88c0f:::benchmark_data_smithi122_119671_object23944:head# 2024-03-20T09:04:49.040 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6c8af66:::benchmark_data_smithi122_119671_object16691:head# 2024-03-20T09:04:49.041 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6c9490f:::benchmark_data_smithi122_119671_object4303:head# 2024-03-20T09:04:49.041 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:04:49.042 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6c9ef92:::benchmark_data_smithi122_119671_object27914:head# 2024-03-20T09:04:49.042 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6ca15f7:::benchmark_data_smithi122_119671_object93536:head# 2024-03-20T09:04:49.043 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6ca9212:::benchmark_data_smithi122_119671_object87488:head# 2024-03-20T09:04:49.043 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6caaf6c:::benchmark_data_smithi122_119671_object45280:head# 2024-03-20T09:04:49.043 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6caf9d6:::benchmark_data_smithi122_119671_object34108:head# 2024-03-20T09:04:49.043 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6cb44e0:::benchmark_data_smithi122_119671_object95202:head# 2024-03-20T09:04:49.045 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6cb73bb:::benchmark_data_smithi122_119671_object17660:head# 2024-03-20T09:04:49.045 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6cc330e:::benchmark_data_smithi122_119671_object12864:head# 2024-03-20T09:04:49.045 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6cc3f52:::benchmark_data_smithi122_119671_object52363:head# 2024-03-20T09:04:49.045 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6cf1736:::benchmark_data_smithi122_119671_object19207:head# 2024-03-20T09:04:49.046 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6cf4a04:::benchmark_data_smithi122_119671_object25740:head# 2024-03-20T09:04:49.046 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6d06756:::benchmark_data_smithi122_119671_object4556:head# 2024-03-20T09:04:49.047 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6d08541:::benchmark_data_smithi122_119671_object43094:head# 2024-03-20T09:04:49.047 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6d33290:::benchmark_data_smithi122_119671_object19865:head# 2024-03-20T09:04:49.048 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6d4fbd2:::benchmark_data_smithi122_119671_object82881:head# 2024-03-20T09:04:49.048 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6d621d4:::benchmark_data_smithi122_119671_object6146:head# 2024-03-20T09:04:49.049 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6d79e73:::benchmark_data_smithi122_119671_object60073:head# 2024-03-20T09:04:49.049 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6d849bc:::benchmark_data_smithi122_119671_object11020:head# 2024-03-20T09:04:49.050 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6d9b3f2:::benchmark_data_smithi122_119671_object1254:head# 2024-03-20T09:04:49.050 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6da06ef:::benchmark_data_smithi122_119671_object14017:head# 2024-03-20T09:04:49.050 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6da2e8e:::benchmark_data_smithi122_119671_object47944:head# 2024-03-20T09:04:49.051 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6db16ea:::benchmark_data_smithi122_119671_object90198:head# 2024-03-20T09:04:49.051 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6db265e:::benchmark_data_smithi122_119671_object77054:head# 2024-03-20T09:04:49.052 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6dbe151:::benchmark_data_smithi122_119671_object37519:head# 2024-03-20T09:04:49.052 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6de2612:::benchmark_data_smithi122_119671_object13883:head# 2024-03-20T09:04:49.053 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6df3c0b:::benchmark_data_smithi122_119671_object22602:head# 2024-03-20T09:04:49.053 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6df6a08:::benchmark_data_smithi122_119671_object54797:head# 2024-03-20T09:04:49.054 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6e06fbb:::benchmark_data_smithi122_119671_object20933:head# 2024-03-20T09:04:49.054 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6e0a823:::benchmark_data_smithi122_119671_object5744:head# 2024-03-20T09:04:49.054 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6e0ec9c:::benchmark_data_smithi122_119671_object48926:head# 2024-03-20T09:04:49.055 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6e110d5:::benchmark_data_smithi122_119671_object85453:head# 2024-03-20T09:04:49.056 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6e161c8:::benchmark_data_smithi122_119671_object96529:head# 2024-03-20T09:04:49.056 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6e1cf4a:::benchmark_data_smithi122_119671_object55059:head# 2024-03-20T09:04:49.056 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6e2ed66:::benchmark_data_smithi122_119671_object43513:head# 2024-03-20T09:04:49.057 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6e3b3b3:::benchmark_data_smithi122_119671_object51679:head# 2024-03-20T09:04:49.057 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6e3b897:::benchmark_data_smithi122_119671_object17043:head# 2024-03-20T09:04:49.058 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6e3d493:::benchmark_data_smithi122_119671_object48214:head# 2024-03-20T09:04:49.058 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6e69c6a:::benchmark_data_smithi122_119671_object24660:head# 2024-03-20T09:04:49.059 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6e7456c:::benchmark_data_smithi122_119671_object75652:head# 2024-03-20T09:04:49.059 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6e7bcde:::benchmark_data_smithi122_119671_object26808:head# 2024-03-20T09:04:49.060 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6e7cacc:::benchmark_data_smithi122_119671_object62443:head# 2024-03-20T09:04:49.060 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6e92074:::benchmark_data_smithi122_119671_object94655:head# 2024-03-20T09:04:49.061 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6e994ee:::benchmark_data_smithi122_119671_object50602:head# 2024-03-20T09:04:49.061 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6eaabd5:::benchmark_data_smithi122_119671_object78793:head# 2024-03-20T09:04:49.061 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6eac363:::benchmark_data_smithi122_119671_object80126:head# 2024-03-20T09:04:49.062 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6eb30db:::benchmark_data_smithi122_119671_object12686:head# 2024-03-20T09:04:49.063 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6ec167f:::benchmark_data_smithi122_119671_object77172:head# 2024-03-20T09:04:49.063 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6ecd543:::benchmark_data_smithi122_119671_object87727:head# 2024-03-20T09:04:49.063 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6ecdaa6:::benchmark_data_smithi122_119671_object80831:head# 2024-03-20T09:04:49.064 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6f0f843:::benchmark_data_smithi122_119671_object63236:head# 2024-03-20T09:04:49.064 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6f15f55:::benchmark_data_smithi122_119671_object92559:head# 2024-03-20T09:04:49.065 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6f19b5a:::benchmark_data_smithi122_119671_object31577:head# 2024-03-20T09:04:49.065 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6f1ffb6:::benchmark_data_smithi122_119671_object93139:head# 2024-03-20T09:04:49.066 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6f20000:::benchmark_data_smithi122_119671_object2486:head# 2024-03-20T09:04:49.066 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6f22c53:::benchmark_data_smithi122_119671_object43316:head# 2024-03-20T09:04:49.067 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6f2bf60:::benchmark_data_smithi122_119671_object24363:head# 2024-03-20T09:04:49.067 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6f2c052:::benchmark_data_smithi122_119671_object96366:head# 2024-03-20T09:04:49.069 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6f2f41b:::benchmark_data_smithi122_119671_object96512:head# 2024-03-20T09:04:49.069 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6f39f98:::benchmark_data_smithi122_119671_object45835:head# 2024-03-20T09:04:49.069 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6f42458:::benchmark_data_smithi122_119671_object18955:head# 2024-03-20T09:04:49.069 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6f434c4:::benchmark_data_smithi122_119671_object40083:head# 2024-03-20T09:04:49.069 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6f6426c:::benchmark_data_smithi122_119671_object98451:head# 2024-03-20T09:04:49.070 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6f7eb24:::benchmark_data_smithi122_119671_object86894:head# 2024-03-20T09:04:49.070 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6f86ed5:::benchmark_data_smithi122_119671_object4528:head# 2024-03-20T09:04:49.071 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6f8b636:::benchmark_data_smithi122_119671_object69836:head# 2024-03-20T09:04:49.071 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6f97790:::benchmark_data_smithi122_119671_object81699:head# 2024-03-20T09:04:49.072 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6f9c3aa:::benchmark_data_smithi122_119671_object97427:head# 2024-03-20T09:04:49.072 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:49.073 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6f9d137:::benchmark_data_smithi122_119671_object16470:head# 2024-03-20T09:04:49.073 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6fae7cb:::benchmark_data_smithi122_119671_object78392:head# 2024-03-20T09:04:49.073 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6fbcb4a:::benchmark_data_smithi122_119671_object73918:head# 2024-03-20T09:04:49.074 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6fd69ef:::benchmark_data_smithi122_119671_object55735:head# 2024-03-20T09:04:49.074 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6fdd09b:::benchmark_data_smithi122_119671_object39424:head# 2024-03-20T09:04:49.075 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6fe4ebc:::benchmark_data_smithi122_119671_object11735:head# 2024-03-20T09:04:49.075 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6fe9b20:::benchmark_data_smithi122_119671_object82280:head# 2024-03-20T09:04:49.076 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6ff2d75:::benchmark_data_smithi122_119671_object11638:head# 2024-03-20T09:04:49.077 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6ff6697:::benchmark_data_smithi122_119671_object39523:head# 2024-03-20T09:04:49.077 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6ffa448:::benchmark_data_smithi122_119671_object14124:head# 2024-03-20T09:04:49.077 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6ffc092:::benchmark_data_smithi122_119671_object38187:head# 2024-03-20T09:04:49.078 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b6ffe28e:::benchmark_data_smithi122_119671_object79014:head# 2024-03-20T09:04:49.078 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7001c4d:::benchmark_data_smithi122_119671_object18312:head# 2024-03-20T09:04:49.079 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b700d579:::benchmark_data_smithi122_119671_object94074:head# 2024-03-20T09:04:49.079 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b700eeff:::benchmark_data_smithi122_119671_object72836:head# 2024-03-20T09:04:49.079 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b70115f7:::benchmark_data_smithi122_119671_object60080:head# 2024-03-20T09:04:49.080 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7011c9d:::benchmark_data_smithi122_119671_object91243:head# 2024-03-20T09:04:49.080 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7016235:::benchmark_data_smithi122_119671_object22671:head# 2024-03-20T09:04:49.081 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7016e87:::benchmark_data_smithi122_119671_object68318:head# 2024-03-20T09:04:49.081 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b701abc7:::benchmark_data_smithi122_119671_object1739:head# 2024-03-20T09:04:49.082 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b70234bd:::benchmark_data_smithi122_119671_object51845:head# 2024-03-20T09:04:49.082 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b702903b:::benchmark_data_smithi122_119671_object77625:head# 2024-03-20T09:04:49.083 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7037569:::benchmark_data_smithi122_119671_object40632:head# 2024-03-20T09:04:49.084 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b705a797:::benchmark_data_smithi122_119671_object27837:head# 2024-03-20T09:04:49.084 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7076ac6:::benchmark_data_smithi122_119671_object59907:head# 2024-03-20T09:04:49.084 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7078fc8:::benchmark_data_smithi122_119671_object82317:head# 2024-03-20T09:04:49.085 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b707ae39:::benchmark_data_smithi122_119671_object66302:head# 2024-03-20T09:04:49.085 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7097fb6:::benchmark_data_smithi122_119671_object84026:head# 2024-03-20T09:04:49.086 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b709fd8d:::benchmark_data_smithi122_119671_object78294:head# 2024-03-20T09:04:49.086 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b70ae732:::benchmark_data_smithi122_119671_object2927:head# 2024-03-20T09:04:49.087 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b70bb914:::benchmark_data_smithi122_119671_object38276:head# 2024-03-20T09:04:49.087 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b70c863f:::benchmark_data_smithi122_119671_object62782:head# 2024-03-20T09:04:49.087 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b70c9fa8:::benchmark_data_smithi122_119671_object7989:head# 2024-03-20T09:04:49.088 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b70ca16c:::benchmark_data_smithi122_119671_object22006:head# 2024-03-20T09:04:49.089 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b70e4768:::benchmark_data_smithi122_119671_object67565:head# 2024-03-20T09:04:49.089 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b70e7502:::benchmark_data_smithi122_119671_object95620:head# 2024-03-20T09:04:49.089 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b70ea86b:::benchmark_data_smithi122_119671_object88802:head# 2024-03-20T09:04:49.090 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b70eb862:::benchmark_data_smithi122_119671_object98352:head# 2024-03-20T09:04:49.090 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7102519:::benchmark_data_smithi122_119671_object8869:head# 2024-03-20T09:04:49.091 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b71092d1:::benchmark_data_smithi122_119671_object56173:head# 2024-03-20T09:04:49.091 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b71102c9:::benchmark_data_smithi122_119671_object85259:head# 2024-03-20T09:04:49.092 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b71226ae:::benchmark_data_smithi122_119671_object22090:head# 2024-03-20T09:04:49.092 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7131c87:::benchmark_data_smithi122_119671_object44752:head# 2024-03-20T09:04:49.092 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b71321f1:::benchmark_data_smithi122_119671_object47020:head# 2024-03-20T09:04:49.093 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7136eea:::benchmark_data_smithi122_119671_object36683:head# 2024-03-20T09:04:49.093 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b713d48d:::benchmark_data_smithi122_119671_object63206:head# 2024-03-20T09:04:49.093 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7145f9a:::benchmark_data_smithi122_119671_object80538:head# 2024-03-20T09:04:49.094 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7158488:::benchmark_data_smithi122_119671_object19607:head# 2024-03-20T09:04:49.094 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7158850:::benchmark_data_smithi122_119671_object85335:head# 2024-03-20T09:04:49.095 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b715d021:::benchmark_data_smithi122_119671_object93658:head# 2024-03-20T09:04:49.095 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b71624d8:::benchmark_data_smithi122_119671_object49501:head# 2024-03-20T09:04:49.095 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7167bb1:::benchmark_data_smithi122_119671_object64644:head# 2024-03-20T09:04:49.096 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7177ce9:::benchmark_data_smithi122_119671_object32251:head# 2024-03-20T09:04:49.096 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b717d13d:::benchmark_data_smithi122_119671_object71673:head# 2024-03-20T09:04:49.097 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7188605:::benchmark_data_smithi122_119671_object95278:head# 2024-03-20T09:04:49.097 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b71a65db:::benchmark_data_smithi122_119671_object97906:head# 2024-03-20T09:04:49.097 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b71cb74f:::benchmark_data_smithi122_119671_object18468:head# 2024-03-20T09:04:49.098 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b71f2a11:::benchmark_data_smithi122_119671_object52038:head# 2024-03-20T09:04:49.098 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7207870:::benchmark_data_smithi122_119671_object53336:head# 2024-03-20T09:04:49.099 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7210887:::benchmark_data_smithi122_119671_object53552:head# 2024-03-20T09:04:49.099 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7210e24:::benchmark_data_smithi122_119671_object52414:head# 2024-03-20T09:04:49.099 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7211e73:::benchmark_data_smithi122_119671_object90674:head# 2024-03-20T09:04:49.100 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b721e988:::benchmark_data_smithi122_119671_object75879:head# 2024-03-20T09:04:49.100 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b724be15:::benchmark_data_smithi122_119671_object20177:head# 2024-03-20T09:04:49.101 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b724ef0f:::benchmark_data_smithi122_119671_object42259:head# 2024-03-20T09:04:49.101 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b725f5ed:::benchmark_data_smithi122_119671_object9364:head# 2024-03-20T09:04:49.101 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b726bbf9:::benchmark_data_smithi122_119671_object90588:head# 2024-03-20T09:04:49.102 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b726f1d4:::benchmark_data_smithi122_119671_object16238:head# 2024-03-20T09:04:49.102 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7276e6d:::benchmark_data_smithi122_119671_object69186:head# 2024-03-20T09:04:49.103 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7277722:::benchmark_data_smithi122_119671_object10759:head# 2024-03-20T09:04:49.103 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b727e477:::benchmark_data_smithi122_119671_object11419:head# 2024-03-20T09:04:49.103 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b728004b:::benchmark_data_smithi122_119671_object35401:head# 2024-03-20T09:04:49.104 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7281805:::benchmark_data_smithi122_119671_object69196:head# 2024-03-20T09:04:49.104 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7289657:::benchmark_data_smithi122_119671_object12324:head# 2024-03-20T09:04:49.105 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b728d4ed:::benchmark_data_smithi122_119671_object62021:head# 2024-03-20T09:04:49.105 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b729c17a:::benchmark_data_smithi122_119671_object30594:head# 2024-03-20T09:04:49.105 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b729e344:::benchmark_data_smithi122_119671_object71809:head# 2024-03-20T09:04:49.106 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b72a1498:::benchmark_data_smithi122_119671_object10463:head# 2024-03-20T09:04:49.106 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b72a64e6:::benchmark_data_smithi122_119671_object64274:head# 2024-03-20T09:04:49.106 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b72b1366:::benchmark_data_smithi122_119671_object21189:head# 2024-03-20T09:04:49.107 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b72b3a6c:::benchmark_data_smithi122_119671_object33945:head# 2024-03-20T09:04:49.107 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b72bcc54:::benchmark_data_smithi122_119671_object6726:head# 2024-03-20T09:04:49.108 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b72cd70d:::benchmark_data_smithi122_119671_object60706:head# 2024-03-20T09:04:49.108 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b72e9636:::benchmark_data_smithi122_119671_object19615:head# 2024-03-20T09:04:49.109 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b72fbe2d:::benchmark_data_smithi122_119671_object95057:head# 2024-03-20T09:04:49.109 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b72fd2d2:::benchmark_data_smithi122_119671_object84411:head# 2024-03-20T09:04:49.109 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b72fdc64:::benchmark_data_smithi122_119671_object36150:head# 2024-03-20T09:04:49.110 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7300698:::benchmark_data_smithi122_119671_object4268:head# 2024-03-20T09:04:49.110 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b730d8ef:::benchmark_data_smithi122_119671_object63188:head# 2024-03-20T09:04:49.111 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7319b25:::benchmark_data_smithi122_119671_object88064:head# 2024-03-20T09:04:49.111 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b731da59:::benchmark_data_smithi122_119671_object54466:head# 2024-03-20T09:04:49.112 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7337fb7:::benchmark_data_smithi122_119671_object30978:head# 2024-03-20T09:04:49.112 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b733bdb0:::benchmark_data_smithi122_119671_object38575:head# 2024-03-20T09:04:49.112 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7347388:::benchmark_data_smithi122_119671_object80673:head# 2024-03-20T09:04:49.113 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b735567a:::benchmark_data_smithi122_119671_object15430:head# 2024-03-20T09:04:49.113 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7364b95:::benchmark_data_smithi122_119671_object32328:head# 2024-03-20T09:04:49.113 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b736b039:::benchmark_data_smithi122_119671_object65587:head# 2024-03-20T09:04:49.114 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b73781a9:::benchmark_data_smithi122_119671_object80233:head# 2024-03-20T09:04:49.114 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7387f1b:::benchmark_data_smithi122_119671_object27867:head# 2024-03-20T09:04:49.115 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b738a55f:::benchmark_data_smithi122_119671_object34347:head# 2024-03-20T09:04:49.115 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b738fc20:::benchmark_data_smithi122_119671_object9821:head# 2024-03-20T09:04:49.115 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b73909e1:::benchmark_data_smithi122_119671_object31799:head# 2024-03-20T09:04:49.116 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7392eec:::benchmark_data_smithi122_119671_object67289:head# 2024-03-20T09:04:49.116 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b73a257f:::benchmark_data_smithi122_119671_object39887:head# 2024-03-20T09:04:49.117 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b73ce401:::benchmark_data_smithi122_119671_object86318:head# 2024-03-20T09:04:49.117 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b73cff94:::benchmark_data_smithi122_119671_object30671:head# 2024-03-20T09:04:49.117 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b73dda0c:::benchmark_data_smithi122_119671_object32295:head# 2024-03-20T09:04:49.118 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b73f78c0:::benchmark_data_smithi122_119671_object78859:head# 2024-03-20T09:04:49.118 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b740670e:::benchmark_data_smithi122_119671_object96777:head# 2024-03-20T09:04:49.119 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b740825b:::benchmark_data_smithi122_119671_object15940:head# 2024-03-20T09:04:49.119 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b740b304:::benchmark_data_smithi122_119671_object8761:head# 2024-03-20T09:04:49.119 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b740df10:::benchmark_data_smithi122_119671_object4272:head# 2024-03-20T09:04:49.120 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b740f507:::benchmark_data_smithi122_119671_object39210:head# 2024-03-20T09:04:49.120 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7420ba3:::benchmark_data_smithi122_119671_object93014:head# 2024-03-20T09:04:49.121 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7427ccd:::benchmark_data_smithi122_119671_object20835:head# 2024-03-20T09:04:49.121 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7432684:::benchmark_data_smithi122_119671_object11963:head# 2024-03-20T09:04:49.121 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b743a03e:::benchmark_data_smithi122_119671_object90553:head# 2024-03-20T09:04:49.122 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7447676:::benchmark_data_smithi122_119671_object71415:head# 2024-03-20T09:04:49.122 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b744df15:::benchmark_data_smithi122_119671_object65724:head# 2024-03-20T09:04:49.123 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b745117c:::benchmark_data_smithi122_119671_object57840:head# 2024-03-20T09:04:49.123 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b745c1ae:::benchmark_data_smithi122_119671_object18086:head# 2024-03-20T09:04:49.124 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b74603cd:::benchmark_data_smithi122_119671_object98425:head# 2024-03-20T09:04:49.124 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b74745d4:::benchmark_data_smithi122_119671_object57777:head# 2024-03-20T09:04:49.124 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7476996:::benchmark_data_smithi122_119671_object89376:head# 2024-03-20T09:04:49.125 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b748a66e:::benchmark_data_smithi122_119671_object42205:head# 2024-03-20T09:04:49.125 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b748dbde:::benchmark_data_smithi122_119671_object37122:head# 2024-03-20T09:04:49.125 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b749802d:::benchmark_data_smithi122_119671_object74099:head# 2024-03-20T09:04:49.126 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b74ab604:::benchmark_data_smithi122_119671_object83731:head# 2024-03-20T09:04:49.127 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b74c3797:::benchmark_data_smithi122_119671_object41824:head# 2024-03-20T09:04:49.127 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b74caca9:::benchmark_data_smithi122_119671_object46084:head# 2024-03-20T09:04:49.127 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b74d4dda:::benchmark_data_smithi122_119671_object52714:head# 2024-03-20T09:04:49.128 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b74e1905:::benchmark_data_smithi122_119671_object42309:head# 2024-03-20T09:04:49.128 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b74e47a6:::benchmark_data_smithi122_119671_object70003:head# 2024-03-20T09:04:49.129 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b74e5710:::benchmark_data_smithi122_119671_object94919:head# 2024-03-20T09:04:49.129 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b74ec752:::benchmark_data_smithi122_119671_object5513:head# 2024-03-20T09:04:49.130 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b74ed708:::benchmark_data_smithi122_119671_object87280:head# 2024-03-20T09:04:49.130 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b74fb7b4:::benchmark_data_smithi122_119671_object54251:head# 2024-03-20T09:04:49.131 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b74fddd5:::benchmark_data_smithi122_119671_object47456:head# 2024-03-20T09:04:49.131 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b75085fa:::benchmark_data_smithi122_119671_object19428:head# 2024-03-20T09:04:49.131 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7511368:::benchmark_data_smithi122_119671_object62228:head# 2024-03-20T09:04:49.132 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7513490:::benchmark_data_smithi122_119671_object25781:head# 2024-03-20T09:04:49.132 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7513792:::benchmark_data_smithi122_119671_object93120:head# 2024-03-20T09:04:49.133 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7513d6e:::benchmark_data_smithi122_119671_object33434:head# 2024-03-20T09:04:49.133 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b754961f:::benchmark_data_smithi122_119671_object58814:head# 2024-03-20T09:04:49.134 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7556d4a:::benchmark_data_smithi122_119671_object84262:head# 2024-03-20T09:04:49.134 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7562a7e:::benchmark_data_smithi122_119671_object58123:head# 2024-03-20T09:04:49.135 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7570058:::benchmark_data_smithi122_119671_object24508:head# 2024-03-20T09:04:49.135 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b75725b6:::benchmark_data_smithi122_119671_object9989:head# 2024-03-20T09:04:49.135 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7586b48:::benchmark_data_smithi122_119671_object22718:head# 2024-03-20T09:04:49.138 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b758b757:::benchmark_data_smithi122_119671_object92675:head# 2024-03-20T09:04:49.138 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7597cdd:::benchmark_data_smithi122_119671_object26249:head# 2024-03-20T09:04:49.138 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b759dd05:::benchmark_data_smithi122_119671_object2806:head# 2024-03-20T09:04:49.138 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b75a6647:::benchmark_data_smithi122_119671_object9813:head# 2024-03-20T09:04:49.138 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b75b8196:::benchmark_data_smithi122_119671_object42287:head# 2024-03-20T09:04:49.139 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b75b971c:::benchmark_data_smithi122_119671_object73066:head# 2024-03-20T09:04:49.139 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b75bd073:::benchmark_data_smithi122_119671_object85594:head# 2024-03-20T09:04:49.139 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b75d1c78:::benchmark_data_smithi122_119671_object40006:head# 2024-03-20T09:04:49.140 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b75d683c:::benchmark_data_smithi122_119671_object19859:head# 2024-03-20T09:04:49.140 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b75ef690:::benchmark_data_smithi122_119671_object3086:head# 2024-03-20T09:04:49.141 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b75ffc70:::benchmark_data_smithi122_119671_object71186:head# 2024-03-20T09:04:49.141 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b76025d4:::benchmark_data_smithi122_119671_object50865:head# 2024-03-20T09:04:49.142 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7602fa2:::benchmark_data_smithi122_119671_object38732:head# 2024-03-20T09:04:49.142 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7604669:::benchmark_data_smithi122_119671_object85701:head# 2024-03-20T09:04:49.142 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b76069bd:::benchmark_data_smithi122_119671_object78070:head# 2024-03-20T09:04:49.143 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b760b412:::benchmark_data_smithi122_119671_object39833:head# 2024-03-20T09:04:49.143 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b761272b:::benchmark_data_smithi122_119671_object32285:head# 2024-03-20T09:04:49.144 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b76129f5:::benchmark_data_smithi122_119671_object97943:head# 2024-03-20T09:04:49.144 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b76172ec:::benchmark_data_smithi122_119671_object84454:head# 2024-03-20T09:04:49.145 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b76181a4:::benchmark_data_smithi122_119671_object14747:head# 2024-03-20T09:04:49.145 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b761b27f:::benchmark_data_smithi122_119671_object44903:head# 2024-03-20T09:04:49.145 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7621088:::benchmark_data_smithi122_119671_object97987:head# 2024-03-20T09:04:49.146 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b762ffcf:::benchmark_data_smithi122_119671_object59613:head# 2024-03-20T09:04:49.146 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b76433f0:::benchmark_data_smithi122_119671_object26047:head# 2024-03-20T09:04:49.147 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b76570c1:::benchmark_data_smithi122_119671_object50479:head# 2024-03-20T09:04:49.147 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b766f1ad:::benchmark_data_smithi122_119671_object3991:head# 2024-03-20T09:04:49.148 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b76801ef:::benchmark_data_smithi122_119671_object85976:head# 2024-03-20T09:04:49.148 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b769d6ed:::benchmark_data_smithi122_119671_object77349:head# 2024-03-20T09:04:49.149 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b76bb989:::benchmark_data_smithi122_119671_object95373:head# 2024-03-20T09:04:49.149 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b76bd1d0:::benchmark_data_smithi122_119671_object70528:head# 2024-03-20T09:04:49.150 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b76c7c13:::benchmark_data_smithi122_119671_object84936:head# 2024-03-20T09:04:49.150 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b76c8e32:::benchmark_data_smithi122_119671_object69039:head# 2024-03-20T09:04:49.150 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b76d139b:::benchmark_data_smithi122_119671_object7514:head# 2024-03-20T09:04:49.151 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b76fa1ab:::benchmark_data_smithi122_119671_object45702:head# 2024-03-20T09:04:49.152 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b76fff39:::benchmark_data_smithi122_119671_object27500:head# 2024-03-20T09:04:49.152 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7706124:::benchmark_data_smithi122_119671_object48924:head# 2024-03-20T09:04:49.153 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7714984:::benchmark_data_smithi122_119671_object31226:head# 2024-03-20T09:04:49.153 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7718e90:::benchmark_data_smithi122_119671_object24115:head# 2024-03-20T09:04:49.154 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b771b03c:::benchmark_data_smithi122_119671_object35195:head# 2024-03-20T09:04:49.154 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b771ce36:::benchmark_data_smithi122_119671_object24747:head# 2024-03-20T09:04:49.155 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b771e887:::benchmark_data_smithi122_119671_object79603:head# 2024-03-20T09:04:49.155 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7720bed:::benchmark_data_smithi122_119671_object9366:head# 2024-03-20T09:04:49.156 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7727993:::benchmark_data_smithi122_119671_object36234:head# 2024-03-20T09:04:49.156 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b772af94:::benchmark_data_smithi122_119671_object18044:head# 2024-03-20T09:04:49.157 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b772ffa8:::benchmark_data_smithi122_119671_object90482:head# 2024-03-20T09:04:49.157 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b774bbce:::benchmark_data_smithi122_119671_object38780:head# 2024-03-20T09:04:49.158 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7750090:::benchmark_data_smithi122_119671_object12164:head# 2024-03-20T09:04:49.158 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b777c54b:::benchmark_data_smithi122_119671_object57322:head# 2024-03-20T09:04:49.159 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b77849fc:::benchmark_data_smithi122_119671_object3169:head# 2024-03-20T09:04:49.159 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7786505:::benchmark_data_smithi122_119671_object75476:head# 2024-03-20T09:04:49.160 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7789927:::benchmark_data_smithi122_119671_object97973:head# 2024-03-20T09:04:49.160 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b778af9f:::benchmark_data_smithi122_119671_object24491:head# 2024-03-20T09:04:49.160 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b778ed92:::benchmark_data_smithi122_119671_object78096:head# 2024-03-20T09:04:49.161 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7792ca0:::benchmark_data_smithi122_119671_object2925:head# 2024-03-20T09:04:49.161 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b779c89c:::benchmark_data_smithi122_119671_object32886:head# 2024-03-20T09:04:49.162 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b779f260:::benchmark_data_smithi122_119671_object10389:head# 2024-03-20T09:04:49.162 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b77bbb35:::benchmark_data_smithi122_119671_object87237:head# 2024-03-20T09:04:49.162 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b77bd48c:::benchmark_data_smithi122_119671_object34972:head# 2024-03-20T09:04:49.163 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b77d8f4a:::benchmark_data_smithi122_119671_object7272:head# 2024-03-20T09:04:49.163 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b77e9436:::benchmark_data_smithi122_119671_object28056:head# 2024-03-20T09:04:49.164 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b78031a1:::benchmark_data_smithi122_119671_object99188:head# 2024-03-20T09:04:49.164 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7806356:::benchmark_data_smithi122_119671_object44530:head# 2024-03-20T09:04:49.165 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b78091d2:::benchmark_data_smithi122_119671_object92817:head# 2024-03-20T09:04:49.165 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b780e66a:::benchmark_data_smithi122_119671_object12969:head# 2024-03-20T09:04:49.166 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7812fbb:::benchmark_data_smithi122_119671_object90963:head# 2024-03-20T09:04:49.166 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b781314b:::benchmark_data_smithi122_119671_object36904:head# 2024-03-20T09:04:49.166 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b78149d9:::benchmark_data_smithi122_119671_object81717:head# 2024-03-20T09:04:49.167 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7819fed:::benchmark_data_smithi122_119671_object16731:head# 2024-03-20T09:04:49.167 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b782b710:::benchmark_data_smithi122_119671_object25435:head# 2024-03-20T09:04:49.168 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b782c966:::benchmark_data_smithi122_119671_object28342:head# 2024-03-20T09:04:49.168 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b782df3c:::benchmark_data_smithi122_119671_object48121:head# 2024-03-20T09:04:49.168 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7839eb5:::benchmark_data_smithi122_119671_object76732:head# 2024-03-20T09:04:49.169 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b784ddf8:::benchmark_data_smithi122_119671_object19856:head# 2024-03-20T09:04:49.169 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b784ffb0:::benchmark_data_smithi122_119671_object53135:head# 2024-03-20T09:04:49.170 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b78653f5:::benchmark_data_smithi122_119671_object31123:head# 2024-03-20T09:04:49.170 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b78a68bd:::benchmark_data_smithi122_119671_object17727:head# 2024-03-20T09:04:49.171 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b78b1bd2:::benchmark_data_smithi122_119671_object10499:head# 2024-03-20T09:04:49.171 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b78b7ce1:::benchmark_data_smithi122_119671_object90825:head# 2024-03-20T09:04:49.171 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b78c6358:::benchmark_data_smithi122_119671_object3119:head# 2024-03-20T09:04:49.172 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b78c8ee5:::benchmark_data_smithi122_119671_object56581:head# 2024-03-20T09:04:49.175 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b78cf910:::benchmark_data_smithi122_119671_object88343:head# 2024-03-20T09:04:49.176 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b78e7532:::benchmark_data_smithi122_119671_object80939:head# 2024-03-20T09:04:49.176 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b78f4620:::benchmark_data_smithi122_119671_object21504:head# 2024-03-20T09:04:49.176 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7904eb8:::benchmark_data_smithi122_119671_object88555:head# 2024-03-20T09:04:49.177 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7908d5c:::benchmark_data_smithi122_119671_object29847:head# 2024-03-20T09:04:49.177 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b791f48f:::benchmark_data_smithi122_119671_object59154:head# 2024-03-20T09:04:49.177 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7920019:::benchmark_data_smithi122_119671_object54933:head# 2024-03-20T09:04:49.178 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b79210dd:::benchmark_data_smithi122_119671_object33842:head# 2024-03-20T09:04:49.178 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b793740d:::benchmark_data_smithi122_119671_object27463:head# 2024-03-20T09:04:49.179 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b793ae8a:::benchmark_data_smithi122_119671_object47665:head# 2024-03-20T09:04:49.179 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b793eb3e:::benchmark_data_smithi122_119671_object86965:head# 2024-03-20T09:04:49.179 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7956cff:::benchmark_data_smithi122_119671_object85987:head# 2024-03-20T09:04:49.180 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7977b9c:::benchmark_data_smithi122_119671_object73242:head# 2024-03-20T09:04:49.180 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b79b320e:::benchmark_data_smithi122_119671_object71576:head# 2024-03-20T09:04:49.181 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b79be3b4:::benchmark_data_smithi122_119671_object7731:head# 2024-03-20T09:04:49.181 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b79bfdf6:::benchmark_data_smithi122_119671_object65838:head# 2024-03-20T09:04:49.183 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b79c83fa:::benchmark_data_smithi122_119671_object54229:head# 2024-03-20T09:04:49.183 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b79d0112:::benchmark_data_smithi122_119671_object33742:head# 2024-03-20T09:04:49.183 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b79eb156:::benchmark_data_smithi122_119671_object1150:head# 2024-03-20T09:04:49.183 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b79f1082:::benchmark_data_smithi122_119671_object96351:head# 2024-03-20T09:04:49.184 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b79fa2ff:::benchmark_data_smithi122_119671_object34004:head# 2024-03-20T09:04:49.184 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7a03da8:::benchmark_data_smithi122_119671_object4107:head# 2024-03-20T09:04:49.184 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7a1bc94:::benchmark_data_smithi122_119671_object8300:head# 2024-03-20T09:04:49.184 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7a20b60:::benchmark_data_smithi122_119671_object27699:head# 2024-03-20T09:04:49.185 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7a2447d:::benchmark_data_smithi122_119671_object73152:head# 2024-03-20T09:04:49.185 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7a33733:::benchmark_data_smithi122_119671_object46248:head# 2024-03-20T09:04:49.186 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7a4315a:::benchmark_data_smithi122_119671_object92581:head# 2024-03-20T09:04:49.187 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7a44835:::benchmark_data_smithi122_119671_object15835:head# 2024-03-20T09:04:49.187 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7a59a06:::benchmark_data_smithi122_119671_object57794:head# 2024-03-20T09:04:49.187 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7a5cb18:::benchmark_data_smithi122_119671_object11017:head# 2024-03-20T09:04:49.188 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7a637c7:::benchmark_data_smithi122_119671_object5969:head# 2024-03-20T09:04:49.188 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7a66831:::benchmark_data_smithi122_119671_object67911:head# 2024-03-20T09:04:49.188 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7a67bb3:::benchmark_data_smithi122_119671_object78525:head# 2024-03-20T09:04:49.189 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7a75830:::benchmark_data_smithi122_119671_object94282:head# 2024-03-20T09:04:49.189 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7a7d9f9:::benchmark_data_smithi122_119671_object47913:head# 2024-03-20T09:04:49.190 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7ab658d:::benchmark_data_smithi122_119671_object84109:head# 2024-03-20T09:04:49.190 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7ac440d:::benchmark_data_smithi122_119671_object19883:head# 2024-03-20T09:04:49.191 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7ac6e99:::benchmark_data_smithi122_119671_object84666:head# 2024-03-20T09:04:49.191 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7ac7a1b:::benchmark_data_smithi122_119671_object79194:head# 2024-03-20T09:04:49.191 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7ac827a:::benchmark_data_smithi122_119671_object27607:head# 2024-03-20T09:04:49.192 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7acb358:::benchmark_data_smithi122_119671_object58167:head# 2024-03-20T09:04:49.192 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7acb5e1:::benchmark_data_smithi122_119671_object78038:head# 2024-03-20T09:04:49.193 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7ad5e2f:::benchmark_data_smithi122_119671_object46912:head# 2024-03-20T09:04:49.193 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7ad8f60:::benchmark_data_smithi122_119671_object84332:head# 2024-03-20T09:04:49.194 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7ae7beb:::benchmark_data_smithi122_119671_object38012:head# 2024-03-20T09:04:49.194 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7af7ce1:::benchmark_data_smithi122_119671_object6506:head# 2024-03-20T09:04:49.194 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7afc777:::benchmark_data_smithi122_119671_object25221:head# 2024-03-20T09:04:49.195 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7b035df:::benchmark_data_smithi122_119671_object11273:head# 2024-03-20T09:04:49.195 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7b037ed:::benchmark_data_smithi122_119671_object54360:head# 2024-03-20T09:04:49.196 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7b164d8:::benchmark_data_smithi122_119671_object80067:head# 2024-03-20T09:04:49.196 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7b2f23f:::benchmark_data_smithi122_119671_object98133:head# 2024-03-20T09:04:49.196 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7b4934b:::benchmark_data_smithi122_119671_object49088:head# 2024-03-20T09:04:49.197 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7b4bafe:::benchmark_data_smithi122_119671_object30040:head# 2024-03-20T09:04:49.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:48 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:04:49.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:48 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:04:49.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:48 smithi122 ceph-mon[73495]: osdmap e284: 12 total, 11 up, 11 in 2024-03-20T09:04:49.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:48 smithi122 ceph-mon[73495]: 3.19 scrub ok 2024-03-20T09:04:49.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:48 smithi122 ceph-mon[73495]: 1.13 scrub starts 2024-03-20T09:04:49.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:48 smithi122 ceph-mon[73495]: 1.13 scrub ok 2024-03-20T09:04:49.199 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7b516de:::benchmark_data_smithi122_119671_object14712:head# 2024-03-20T09:04:49.199 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7b5790a:::benchmark_data_smithi122_119671_object19852:head# 2024-03-20T09:04:49.200 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7b5cb85:::benchmark_data_smithi122_119671_object78171:head# 2024-03-20T09:04:49.200 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7b5e5bc:::benchmark_data_smithi122_119671_object4620:head# 2024-03-20T09:04:49.200 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7b72cd2:::benchmark_data_smithi122_119671_object41006:head# 2024-03-20T09:04:49.200 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7b72cdc:::benchmark_data_smithi122_119671_object36090:head# 2024-03-20T09:04:49.201 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7b72e92:::benchmark_data_smithi122_119671_object33671:head# 2024-03-20T09:04:49.201 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7b76fad:::benchmark_data_smithi122_119671_object5766:head# 2024-03-20T09:04:49.201 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7b94fd9:::benchmark_data_smithi122_119671_object68524:head# 2024-03-20T09:04:49.201 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7badd49:::benchmark_data_smithi122_119671_object96753:head# 2024-03-20T09:04:49.203 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7bb6fe6:::benchmark_data_smithi122_119671_object17900:head# 2024-03-20T09:04:49.203 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7bbe7ec:::benchmark_data_smithi122_119671_object38355:head# 2024-03-20T09:04:49.203 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7bc3bbc:::benchmark_data_smithi122_119671_object42316:head# 2024-03-20T09:04:49.203 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7be6dda:::benchmark_data_smithi122_119671_object37778:head# 2024-03-20T09:04:49.204 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7bf62f5:::benchmark_data_smithi122_119671_object24013:head# 2024-03-20T09:04:49.204 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7bfee56:::benchmark_data_smithi122_119671_object74638:head# 2024-03-20T09:04:49.204 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7c1d39d:::benchmark_data_smithi122_119671_object30681:head# 2024-03-20T09:04:49.205 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7c43e86:::benchmark_data_smithi122_119671_object96007:head# 2024-03-20T09:04:49.205 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7c54a85:::benchmark_data_smithi122_119671_object90576:head# 2024-03-20T09:04:49.205 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7c59723:::benchmark_data_smithi122_119671_object47597:head# 2024-03-20T09:04:49.206 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7c688b1:::benchmark_data_smithi122_119671_object62953:head# 2024-03-20T09:04:49.207 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7c84acb:::benchmark_data_smithi122_119671_object84891:head# 2024-03-20T09:04:49.207 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7c8cd0b:::benchmark_data_smithi122_119671_object63187:head# 2024-03-20T09:04:49.208 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7c96e2d:::benchmark_data_smithi122_119671_object66446:head# 2024-03-20T09:04:49.208 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7ca5c8e:::benchmark_data_smithi122_119671_object76806:head# 2024-03-20T09:04:49.209 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7cab9fb:::benchmark_data_smithi122_119671_object89935:head# 2024-03-20T09:04:49.210 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7cbcbeb:::benchmark_data_smithi122_119671_object69943:head# 2024-03-20T09:04:49.210 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7cd560b:::benchmark_data_smithi122_119671_object65194:head# 2024-03-20T09:04:49.210 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7cddb9d:::benchmark_data_smithi122_119671_object35368:head# 2024-03-20T09:04:49.211 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7cf04ed:::benchmark_data_smithi122_119671_object18770:head# 2024-03-20T09:04:49.217 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7cf5d70:::benchmark_data_smithi122_119671_object52409:head# 2024-03-20T09:04:49.218 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7cf9fdf:::benchmark_data_smithi122_119671_object36411:head# 2024-03-20T09:04:49.218 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7cfa66d:::benchmark_data_smithi122_119671_object25584:head# 2024-03-20T09:04:49.218 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7cfcd99:::benchmark_data_smithi122_119671_object7732:head# 2024-03-20T09:04:49.219 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7d0bb42:::benchmark_data_smithi122_119671_object32065:head# 2024-03-20T09:04:49.219 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7d299ce:::benchmark_data_smithi122_119671_object58491:head# 2024-03-20T09:04:49.219 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7d383cd:::benchmark_data_smithi122_119671_object22651:head# 2024-03-20T09:04:49.219 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7d3d599:::benchmark_data_smithi122_119671_object23310:head# 2024-03-20T09:04:49.219 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7d3eff2:::benchmark_data_smithi122_119671_object42697:head# 2024-03-20T09:04:49.219 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7d4d855:::benchmark_data_smithi122_119671_object5676:head# 2024-03-20T09:04:49.219 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7d53a7d:::benchmark_data_smithi122_119671_object6141:head# 2024-03-20T09:04:49.219 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7d569ba:::benchmark_data_smithi122_119671_object43862:head# 2024-03-20T09:04:49.219 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7d7a4a5:::benchmark_data_smithi122_119671_object15747:head# 2024-03-20T09:04:49.219 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7d7f8de:::benchmark_data_smithi122_119671_object8295:head# 2024-03-20T09:04:49.219 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7d99157:::benchmark_data_smithi122_119671_object45151:head# 2024-03-20T09:04:49.219 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7daa3f0:::benchmark_data_smithi122_119671_object38533:head# 2024-03-20T09:04:49.219 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7db5e53:::benchmark_data_smithi122_119671_object78165:head# 2024-03-20T09:04:49.220 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7dc7759:::benchmark_data_smithi122_119671_object71736:head# 2024-03-20T09:04:49.220 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7de2b19:::benchmark_data_smithi122_119671_object29384:head# 2024-03-20T09:04:49.221 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7dffc1e:::benchmark_data_smithi122_119671_object63532:head# 2024-03-20T09:04:49.221 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7e0da74:::benchmark_data_smithi122_119671_object6988:head# 2024-03-20T09:04:49.221 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7e186c3:::benchmark_data_smithi122_119671_object14964:head# 2024-03-20T09:04:49.221 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7e2430d:::benchmark_data_smithi122_119671_object59678:head# 2024-03-20T09:04:49.222 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7e40e4f:::benchmark_data_smithi122_119671_object35360:head# 2024-03-20T09:04:49.222 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7e4b3c5:::benchmark_data_smithi122_119671_object66838:head# 2024-03-20T09:04:49.222 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7e4cc01:::benchmark_data_smithi122_119671_object50789:head# 2024-03-20T09:04:49.223 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7e4d4bc:::benchmark_data_smithi122_119671_object29291:head# 2024-03-20T09:04:49.223 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7e4e985:::benchmark_data_smithi122_119671_object34662:head# 2024-03-20T09:04:49.223 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7e50f5e:::benchmark_data_smithi122_119671_object3597:head# 2024-03-20T09:04:49.223 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7e579c8:::benchmark_data_smithi122_119671_object4081:head# 2024-03-20T09:04:49.224 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7e63e94:::benchmark_data_smithi122_119671_object57989:head# 2024-03-20T09:04:49.224 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7e6f6e2:::benchmark_data_smithi122_119671_object14023:head# 2024-03-20T09:04:49.225 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7e6fe27:::benchmark_data_smithi122_119671_object34322:head# 2024-03-20T09:04:49.225 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7e86796:::benchmark_data_smithi122_119671_object13958:head# 2024-03-20T09:04:49.225 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7ea578c:::benchmark_data_smithi122_119671_object96042:head# 2024-03-20T09:04:49.227 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7ea7b79:::benchmark_data_smithi122_119671_object10432:head# 2024-03-20T09:04:49.227 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7ec881b:::benchmark_data_smithi122_119671_object85395:head# 2024-03-20T09:04:49.227 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7ecd36e:::benchmark_data_smithi122_119671_object61778:head# 2024-03-20T09:04:49.228 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7ed0d21:::benchmark_data_smithi122_119671_object87835:head# 2024-03-20T09:04:49.228 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7ed2031:::benchmark_data_smithi122_119671_object43248:head# 2024-03-20T09:04:49.228 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7ed7436:::benchmark_data_smithi122_119671_object7715:head# 2024-03-20T09:04:49.229 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7ef5de4:::benchmark_data_smithi122_119671_object97783:head# 2024-03-20T09:04:49.229 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7ef9a74:::benchmark_data_smithi122_119671_object7610:head# 2024-03-20T09:04:49.230 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7f10722:::benchmark_data_smithi122_119671_object95476:head# 2024-03-20T09:04:49.230 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7f14cf7:::benchmark_data_smithi122_119671_object72091:head# 2024-03-20T09:04:49.231 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7f1ab0d:::benchmark_data_smithi122_119671_object94637:head# 2024-03-20T09:04:49.231 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7f2be3f:::benchmark_data_smithi122_119671_object25948:head# 2024-03-20T09:04:49.232 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7f44fc3:::benchmark_data_smithi122_119671_object64845:head# 2024-03-20T09:04:49.232 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7f450c3:::benchmark_data_smithi122_119671_object28563:head# 2024-03-20T09:04:49.233 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7f46517:::benchmark_data_smithi122_119671_object62667:head# 2024-03-20T09:04:49.233 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7f550e2:::benchmark_data_smithi122_119671_object37477:head# 2024-03-20T09:04:49.233 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7f8b335:::benchmark_data_smithi122_119671_object42223:head# 2024-03-20T09:04:49.234 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7f94a74:::benchmark_data_smithi122_119671_object40426:head# 2024-03-20T09:04:49.234 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7fa065d:::benchmark_data_smithi122_119671_object74938:head# 2024-03-20T09:04:49.234 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7fa80fc:::benchmark_data_smithi122_119671_object38864:head# 2024-03-20T09:04:49.235 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7fad547:::benchmark_data_smithi122_119671_object35139:head# 2024-03-20T09:04:49.235 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7fb043a:::benchmark_data_smithi122_119671_object44740:head# 2024-03-20T09:04:49.236 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7fb448e:::benchmark_data_smithi122_119671_object76906:head# 2024-03-20T09:04:49.237 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7fb6d57:::benchmark_data_smithi122_119671_object56850:head# 2024-03-20T09:04:49.237 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7fbba0c:::benchmark_data_smithi122_119671_object19928:head# 2024-03-20T09:04:49.237 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7fbddb4:::benchmark_data_smithi122_119671_object44209:head# 2024-03-20T09:04:49.238 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7fd7a52:::benchmark_data_smithi122_119671_object83601:head# 2024-03-20T09:04:49.238 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7fd88d8:::benchmark_data_smithi122_119671_object87272:head# 2024-03-20T09:04:49.239 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7fdb1dc:::benchmark_data_smithi122_119671_object28352:head# 2024-03-20T09:04:49.239 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7fdcbc8:::benchmark_data_smithi122_119671_object68383:head# 2024-03-20T09:04:49.240 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7fe0556:::benchmark_data_smithi122_119671_object71275:head# 2024-03-20T09:04:49.240 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7fe28a6:::benchmark_data_smithi122_119671_object11375:head# 2024-03-20T09:04:49.241 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7ff05f5:::benchmark_data_smithi122_119671_object16767:head# 2024-03-20T09:04:49.241 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b7ff2d72:::benchmark_data_smithi122_119671_object88271:head# 2024-03-20T09:04:49.242 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b800202b:::benchmark_data_smithi122_119671_object19881:head# 2024-03-20T09:04:49.242 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b80036f3:::benchmark_data_smithi122_119671_object55752:head# 2024-03-20T09:04:49.242 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8008e71:::benchmark_data_smithi122_119671_object70820:head# 2024-03-20T09:04:49.243 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b801a664:::benchmark_data_smithi122_119671_object65520:head# 2024-03-20T09:04:49.243 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b802ada9:::benchmark_data_smithi122_119671_object73030:head# 2024-03-20T09:04:49.244 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b802bd0e:::benchmark_data_smithi122_119671_object20391:head# 2024-03-20T09:04:49.244 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8032b0b:::benchmark_data_smithi122_119671_object91305:head# 2024-03-20T09:04:49.245 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b803967d:::benchmark_data_smithi122_119671_object52425:head# 2024-03-20T09:04:49.245 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b803aa79:::benchmark_data_smithi122_119671_object35762:head# 2024-03-20T09:04:49.245 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b803d11c:::benchmark_data_smithi122_119671_object55385:head# 2024-03-20T09:04:49.246 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b803f500:::benchmark_data_smithi122_119671_object84556:head# 2024-03-20T09:04:49.246 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8055109:::benchmark_data_smithi122_119671_object38302:head# 2024-03-20T09:04:49.246 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8057a52:::benchmark_data_smithi122_119671_object4994:head# 2024-03-20T09:04:49.247 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b80597b5:::benchmark_data_smithi122_119671_object15314:head# 2024-03-20T09:04:49.247 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b80727c5:::benchmark_data_smithi122_119671_object27129:head# 2024-03-20T09:04:49.248 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b80755c2:::benchmark_data_smithi122_119671_object64041:head# 2024-03-20T09:04:49.248 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8075c7f:::benchmark_data_smithi122_119671_object47146:head# 2024-03-20T09:04:49.248 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b807684e:::benchmark_data_smithi122_119671_object57881:head# 2024-03-20T09:04:49.249 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8089bf2:::benchmark_data_smithi122_119671_object4491:head# 2024-03-20T09:04:49.249 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8096e5b:::benchmark_data_smithi122_119671_object66453:head# 2024-03-20T09:04:49.250 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b80b08e5:::benchmark_data_smithi122_119671_object25090:head# 2024-03-20T09:04:49.250 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b80b59bf:::benchmark_data_smithi122_119671_object33132:head# 2024-03-20T09:04:49.250 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b80b8a68:::benchmark_data_smithi122_119671_object24412:head# 2024-03-20T09:04:49.251 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b80b96ef:::benchmark_data_smithi122_119671_object93241:head# 2024-03-20T09:04:49.251 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b80c0db9:::benchmark_data_smithi122_119671_object91088:head# 2024-03-20T09:04:49.251 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b80c817f:::benchmark_data_smithi122_119671_object44914:head# 2024-03-20T09:04:49.252 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b80cd1ac:::benchmark_data_smithi122_119671_object12900:head# 2024-03-20T09:04:49.252 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b80d21d9:::benchmark_data_smithi122_119671_object50161:head# 2024-03-20T09:04:49.253 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b80d71e4:::benchmark_data_smithi122_119671_object72501:head# 2024-03-20T09:04:49.253 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b80e0c0e:::benchmark_data_smithi122_119671_object56301:head# 2024-03-20T09:04:49.253 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b80ef054:::benchmark_data_smithi122_119671_object51244:head# 2024-03-20T09:04:49.254 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b80ffa3e:::benchmark_data_smithi122_119671_object57575:head# 2024-03-20T09:04:49.254 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b81009fb:::benchmark_data_smithi122_119671_object20049:head# 2024-03-20T09:04:49.255 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b810714b:::benchmark_data_smithi122_119671_object36804:head# 2024-03-20T09:04:49.255 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8107cc2:::benchmark_data_smithi122_119671_object82987:head# 2024-03-20T09:04:49.256 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8117d41:::benchmark_data_smithi122_119671_object69773:head# 2024-03-20T09:04:49.256 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8128b1f:::benchmark_data_smithi122_119671_object95893:head# 2024-03-20T09:04:49.256 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b81299f3:::benchmark_data_smithi122_119671_object37178:head# 2024-03-20T09:04:49.257 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8133784:::benchmark_data_smithi122_119671_object63647:head# 2024-03-20T09:04:49.257 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8136bb3:::benchmark_data_smithi122_119671_object88472:head# 2024-03-20T09:04:49.258 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b813ea46:::benchmark_data_smithi122_119671_object80804:head# 2024-03-20T09:04:49.258 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8147ca8:::benchmark_data_smithi122_119671_object36478:head# 2024-03-20T09:04:49.259 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b814fcd2:::benchmark_data_smithi122_119671_object28802:head# 2024-03-20T09:04:49.259 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8151f24:::benchmark_data_smithi122_119671_object18548:head# 2024-03-20T09:04:49.259 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8154d00:::benchmark_data_smithi122_119671_object15750:head# 2024-03-20T09:04:49.259 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8168b6b:::benchmark_data_smithi122_119671_object74433:head# 2024-03-20T09:04:49.260 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8169c09:::benchmark_data_smithi122_119671_object88014:head# 2024-03-20T09:04:49.260 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8170530:::benchmark_data_smithi122_119671_object82332:head# 2024-03-20T09:04:49.261 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8179c4c:::benchmark_data_smithi122_119671_object67903:head# 2024-03-20T09:04:49.261 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b817cf91:::benchmark_data_smithi122_119671_object96836:head# 2024-03-20T09:04:49.261 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b81822ab:::benchmark_data_smithi122_119671_object97128:head# 2024-03-20T09:04:49.262 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b81860be:::benchmark_data_smithi122_119671_object26728:head# 2024-03-20T09:04:49.262 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8188de5:::benchmark_data_smithi122_119671_object39634:head# 2024-03-20T09:04:49.263 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b818f3c0:::benchmark_data_smithi122_119671_object72695:head# 2024-03-20T09:04:49.263 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b81a2176:::benchmark_data_smithi122_119671_object68372:head# 2024-03-20T09:04:49.264 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b81a28d6:::benchmark_data_smithi122_119671_object43363:head# 2024-03-20T09:04:49.265 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b81acc4e:::benchmark_data_smithi122_119671_object15872:head# 2024-03-20T09:04:49.265 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b81b0201:::benchmark_data_smithi122_119671_object11053:head# 2024-03-20T09:04:49.265 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b81b124c:::benchmark_data_smithi122_119671_object79010:head# 2024-03-20T09:04:49.265 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b81b480f:::benchmark_data_smithi122_119671_object9680:head# 2024-03-20T09:04:49.265 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b81b6299:::benchmark_data_smithi122_119671_object15800:head# 2024-03-20T09:04:49.266 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b81b6c38:::benchmark_data_smithi122_119671_object76367:head# 2024-03-20T09:04:49.266 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b81d950e:::benchmark_data_smithi122_119671_object20291:head# 2024-03-20T09:04:49.266 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b81e839b:::benchmark_data_smithi122_119671_object2742:head# 2024-03-20T09:04:49.267 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8202855:::benchmark_data_smithi122_119671_object88471:head# 2024-03-20T09:04:49.267 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8204f8d:::benchmark_data_smithi122_119671_object97482:head# 2024-03-20T09:04:49.267 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b820b03a:::benchmark_data_smithi122_119671_object54490:head# 2024-03-20T09:04:49.268 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8212aed:::benchmark_data_smithi122_119671_object81501:head# 2024-03-20T09:04:49.268 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b821709b:::benchmark_data_smithi122_119671_object74732:head# 2024-03-20T09:04:49.269 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b821bd6e:::benchmark_data_smithi122_119671_object44976:head# 2024-03-20T09:04:49.269 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b82214ca:::benchmark_data_smithi122_119671_object92305:head# 2024-03-20T09:04:49.269 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8229cc9:::benchmark_data_smithi122_119671_object84921:head# 2024-03-20T09:04:49.270 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b823ae7f:::benchmark_data_smithi122_119671_object66023:head# 2024-03-20T09:04:49.270 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b825aeb6:::benchmark_data_smithi122_119671_object81073:head# 2024-03-20T09:04:49.271 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8267c81:::benchmark_data_smithi122_119671_object72063:head# 2024-03-20T09:04:49.271 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8268a4d:::benchmark_data_smithi122_119671_object44309:head# 2024-03-20T09:04:49.272 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b826b983:::benchmark_data_smithi122_119671_object1518:head# 2024-03-20T09:04:49.272 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8273313:::benchmark_data_smithi122_119671_object97936:head# 2024-03-20T09:04:49.272 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8296675:::benchmark_data_smithi122_119671_object24428:head# 2024-03-20T09:04:49.273 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8296db0:::benchmark_data_smithi122_119671_object71970:head# 2024-03-20T09:04:49.273 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8299ca2:::benchmark_data_smithi122_119671_object75094:head# 2024-03-20T09:04:49.274 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b82a3695:::benchmark_data_smithi122_119671_object43007:head# 2024-03-20T09:04:49.277 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b82a75de:::benchmark_data_smithi122_119671_object53994:head# 2024-03-20T09:04:49.277 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b82abf93:::benchmark_data_smithi122_119671_object68251:head# 2024-03-20T09:04:49.278 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b82af4a8:::benchmark_data_smithi122_119671_object54237:head# 2024-03-20T09:04:49.278 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b82b6d19:::benchmark_data_smithi122_119671_object6840:head# 2024-03-20T09:04:49.278 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b82b9821:::benchmark_data_smithi122_119671_object18342:head# 2024-03-20T09:04:49.279 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b82cd1d5:::benchmark_data_smithi122_119671_object19827:head# 2024-03-20T09:04:49.279 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b82dc905:::benchmark_data_smithi122_119671_object72997:head# 2024-03-20T09:04:49.280 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b82e3f5d:::benchmark_data_smithi122_119671_object26428:head# 2024-03-20T09:04:49.280 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b82ea7b3:::benchmark_data_smithi122_119671_object84007:head# 2024-03-20T09:04:49.280 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b830a67f:::benchmark_data_smithi122_119671_object30849:head# 2024-03-20T09:04:49.281 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b830ae7a:::benchmark_data_smithi122_119671_object93516:head# 2024-03-20T09:04:49.281 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8311b32:::benchmark_data_smithi122_119671_object17918:head# 2024-03-20T09:04:49.282 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b831dfa1:::benchmark_data_smithi122_119671_object50449:head# 2024-03-20T09:04:49.282 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b834826f:::benchmark_data_smithi122_119671_object58434:head# 2024-03-20T09:04:49.283 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8359478:::benchmark_data_smithi122_119671_object82960:head# 2024-03-20T09:04:49.283 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b835fd7a:::benchmark_data_smithi122_119671_object1839:head# 2024-03-20T09:04:49.284 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b836265b:::benchmark_data_smithi122_119671_object89854:head# 2024-03-20T09:04:49.284 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b83660d3:::benchmark_data_smithi122_119671_object92385:head# 2024-03-20T09:04:49.285 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b83688d7:::benchmark_data_smithi122_119671_object41792:head# 2024-03-20T09:04:49.285 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b837abd6:::benchmark_data_smithi122_119671_object14795:head# 2024-03-20T09:04:49.286 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8396819:::benchmark_data_smithi122_119671_object23565:head# 2024-03-20T09:04:49.286 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b83aa4f2:::benchmark_data_smithi122_119671_object68133:head# 2024-03-20T09:04:49.286 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b83ab218:::benchmark_data_smithi122_119671_object76003:head# 2024-03-20T09:04:49.287 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b83d564c:::benchmark_data_smithi122_119671_object62156:head# 2024-03-20T09:04:49.287 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b83df189:::benchmark_data_smithi122_119671_object45339:head# 2024-03-20T09:04:49.288 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b83e3062:::benchmark_data_smithi122_119671_object2747:head# 2024-03-20T09:04:49.288 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b83e3ce9:::benchmark_data_smithi122_119671_object74482:head# 2024-03-20T09:04:49.289 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b83ef985:::benchmark_data_smithi122_119671_object17998:head# 2024-03-20T09:04:49.289 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b83fd5e2:::benchmark_data_smithi122_119671_object78595:head# 2024-03-20T09:04:49.289 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8410f5e:::benchmark_data_smithi122_119671_object88715:head# 2024-03-20T09:04:49.290 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8417cb9:::benchmark_data_smithi122_119671_object55182:head# 2024-03-20T09:04:49.290 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8427d05:::benchmark_data_smithi122_119671_object90275:head# 2024-03-20T09:04:49.291 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b842e76f:::benchmark_data_smithi122_119671_object86557:head# 2024-03-20T09:04:49.291 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b84371b7:::benchmark_data_smithi122_119671_object75535:head# 2024-03-20T09:04:49.292 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b843daf8:::benchmark_data_smithi122_119671_object49103:head# 2024-03-20T09:04:49.292 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8441013:::benchmark_data_smithi122_119671_object73899:head# 2024-03-20T09:04:49.293 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b844af22:::benchmark_data_smithi122_119671_object38752:head# 2024-03-20T09:04:49.293 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b844be7b:::benchmark_data_smithi122_119671_object29238:head# 2024-03-20T09:04:49.293 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b845a16f:::benchmark_data_smithi122_119671_object47310:head# 2024-03-20T09:04:49.294 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b846e617:::benchmark_data_smithi122_119671_object58875:head# 2024-03-20T09:04:49.294 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8470302:::benchmark_data_smithi122_119671_object64541:head# 2024-03-20T09:04:49.295 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8473f26:::benchmark_data_smithi122_119671_object62027:head# 2024-03-20T09:04:49.295 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b847b42a:::benchmark_data_smithi122_119671_object42732:head# 2024-03-20T09:04:49.295 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8483761:::benchmark_data_smithi122_119671_object47377:head# 2024-03-20T09:04:49.296 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8484633:::benchmark_data_smithi122_119671_object65070:head# 2024-03-20T09:04:49.297 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8486c19:::benchmark_data_smithi122_119671_object92246:head# 2024-03-20T09:04:49.297 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b848c968:::benchmark_data_smithi122_119671_object41104:head# 2024-03-20T09:04:49.298 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b849b576:::benchmark_data_smithi122_119671_object1221:head# 2024-03-20T09:04:49.298 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b84a1600:::benchmark_data_smithi122_119671_object85328:head# 2024-03-20T09:04:49.298 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b84abced:::benchmark_data_smithi122_119671_object93549:head# 2024-03-20T09:04:49.299 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b84be974:::benchmark_data_smithi122_119671_object92245:head# 2024-03-20T09:04:49.299 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b84c9740:::benchmark_data_smithi122_119671_object63739:head# 2024-03-20T09:04:49.300 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b84da0ae:::benchmark_data_smithi122_119671_object9857:head# 2024-03-20T09:04:49.301 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b84dc184:::benchmark_data_smithi122_119671_object3006:head# 2024-03-20T09:04:49.301 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b850e72a:::benchmark_data_smithi122_119671_object22917:head# 2024-03-20T09:04:49.301 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b851681b:::benchmark_data_smithi122_119671_object53315:head# 2024-03-20T09:04:49.303 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8519a37:::benchmark_data_smithi122_119671_object40166:head# 2024-03-20T09:04:49.303 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b85581be:::benchmark_data_smithi122_119671_object97481:head# 2024-03-20T09:04:49.303 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8587608:::benchmark_data_smithi122_119671_object99769:head# 2024-03-20T09:04:49.304 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b858fa84:::benchmark_data_smithi122_119671_object41578:head# 2024-03-20T09:04:49.304 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b85935f3:::benchmark_data_smithi122_119671_object31082:head# 2024-03-20T09:04:49.304 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b859439f:::benchmark_data_smithi122_119671_object92641:head# 2024-03-20T09:04:49.304 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b85a113d:::benchmark_data_smithi122_119671_object44557:head# 2024-03-20T09:04:49.305 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b85ae805:::benchmark_data_smithi122_119671_object66188:head# 2024-03-20T09:04:49.305 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b85b0dd5:::benchmark_data_smithi122_119671_object42443:head# 2024-03-20T09:04:49.305 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b85caaf1:::benchmark_data_smithi122_119671_object59164:head# 2024-03-20T09:04:49.305 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b85cd9bb:::benchmark_data_smithi122_119671_object67050:head# 2024-03-20T09:04:49.306 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b85d064a:::benchmark_data_smithi122_119671_object36961:head# 2024-03-20T09:04:49.306 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b85d983b:::benchmark_data_smithi122_119671_object75774:head# 2024-03-20T09:04:49.307 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b85ed74b:::benchmark_data_smithi122_119671_object60079:head# 2024-03-20T09:04:49.307 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b85f98c8:::benchmark_data_smithi122_119671_object13180:head# 2024-03-20T09:04:49.308 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b85fa685:::benchmark_data_smithi122_119671_object28895:head# 2024-03-20T09:04:49.308 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b85fee2e:::benchmark_data_smithi122_119671_object73540:head# 2024-03-20T09:04:49.309 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8605cd7:::benchmark_data_smithi122_119671_object89503:head# 2024-03-20T09:04:49.309 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8615e71:::benchmark_last_metadata:head# 2024-03-20T09:04:49.310 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b861f63c:::benchmark_data_smithi122_119671_object86594:head# 2024-03-20T09:04:49.310 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b86209c1:::benchmark_data_smithi122_119671_object98303:head# 2024-03-20T09:04:49.310 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8620fd9:::benchmark_data_smithi122_119671_object64445:head# 2024-03-20T09:04:49.311 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b862746a:::benchmark_data_smithi122_119671_object66158:head# 2024-03-20T09:04:49.311 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b862e2e2:::benchmark_data_smithi122_119671_object67845:head# 2024-03-20T09:04:49.312 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8635a4a:::benchmark_data_smithi122_119671_object7857:head# 2024-03-20T09:04:49.312 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8636b44:::benchmark_data_smithi122_119671_object1613:head# 2024-03-20T09:04:49.313 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b863cc08:::benchmark_data_smithi122_119671_object42679:head# 2024-03-20T09:04:49.314 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b864b115:::benchmark_data_smithi122_119671_object59637:head# 2024-03-20T09:04:49.314 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b864f509:::benchmark_data_smithi122_119671_object62251:head# 2024-03-20T09:04:49.314 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8650a8c:::benchmark_data_smithi122_119671_object65833:head# 2024-03-20T09:04:49.315 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8652145:::benchmark_data_smithi122_119671_object12379:head# 2024-03-20T09:04:49.315 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b86573aa:::benchmark_data_smithi122_119671_object4056:head# 2024-03-20T09:04:49.315 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b866b87f:::benchmark_data_smithi122_119671_object12930:head# 2024-03-20T09:04:49.316 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b867a0ed:::benchmark_data_smithi122_119671_object16485:head# 2024-03-20T09:04:49.316 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8692db8:::benchmark_data_smithi122_119671_object10527:head# 2024-03-20T09:04:49.317 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b869634d:::benchmark_data_smithi122_119671_object40215:head# 2024-03-20T09:04:49.318 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8697444:::benchmark_data_smithi122_119671_object34965:head# 2024-03-20T09:04:49.318 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b86befdc:::benchmark_data_smithi122_119671_object69454:head# 2024-03-20T09:04:49.318 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b86cedb6:::benchmark_data_smithi122_119671_object82532:head# 2024-03-20T09:04:49.319 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b86cf43f:::benchmark_data_smithi122_119671_object45753:head# 2024-03-20T09:04:49.319 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b86cf902:::benchmark_data_smithi122_119671_object89219:head# 2024-03-20T09:04:49.320 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b86e58b3:::benchmark_data_smithi122_119671_object44453:head# 2024-03-20T09:04:49.320 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b86f8343:::benchmark_data_smithi122_119671_object27581:head# 2024-03-20T09:04:49.321 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8700b85:::benchmark_data_smithi122_119671_object54329:head# 2024-03-20T09:04:49.321 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b87143e0:::benchmark_data_smithi122_119671_object53522:head# 2024-03-20T09:04:49.322 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8733634:::benchmark_data_smithi122_119671_object24618:head# 2024-03-20T09:04:49.322 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b873d2e1:::benchmark_data_smithi122_119671_object7447:head# 2024-03-20T09:04:49.322 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b873fc0d:::benchmark_data_smithi122_119671_object43279:head# 2024-03-20T09:04:49.323 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8743862:::benchmark_data_smithi122_119671_object38483:head# 2024-03-20T09:04:49.323 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8746227:::benchmark_data_smithi122_119671_object80223:head# 2024-03-20T09:04:49.324 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8748eb8:::benchmark_data_smithi122_119671_object39784:head# 2024-03-20T09:04:49.324 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8755d86:::benchmark_data_smithi122_119671_object67794:head# 2024-03-20T09:04:49.325 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8763470:::benchmark_data_smithi122_119671_object25763:head# 2024-03-20T09:04:49.325 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8766c6a:::benchmark_data_smithi122_119671_object11327:head# 2024-03-20T09:04:49.326 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8771e09:::benchmark_data_smithi122_119671_object51078:head# 2024-03-20T09:04:49.326 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b877dd03:::benchmark_data_smithi122_119671_object17375:head# 2024-03-20T09:04:49.327 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b878104b:::benchmark_data_smithi122_119671_object32706:head# 2024-03-20T09:04:49.327 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8788aea:::benchmark_data_smithi122_119671_object5651:head# 2024-03-20T09:04:49.328 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b878e11b:::benchmark_data_smithi122_119671_object93526:head# 2024-03-20T09:04:49.328 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b87a2d08:::benchmark_data_smithi122_119671_object35383:head# 2024-03-20T09:04:49.329 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b87b71a8:::benchmark_data_smithi122_119671_object44866:head# 2024-03-20T09:04:49.329 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b87c8d7e:::benchmark_data_smithi122_119671_object62022:head# 2024-03-20T09:04:49.329 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b87d1b73:::benchmark_data_smithi122_119671_object8712:head# 2024-03-20T09:04:49.330 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b87d27b1:::benchmark_data_smithi122_119671_object88730:head# 2024-03-20T09:04:49.331 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b87e6e1e:::benchmark_data_smithi122_119671_object99191:head# 2024-03-20T09:04:49.331 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b87e8af1:::benchmark_data_smithi122_119671_object59064:head# 2024-03-20T09:04:49.332 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b87ea27f:::benchmark_data_smithi122_119671_object96479:head# 2024-03-20T09:04:49.332 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b87ea693:::benchmark_data_smithi122_119671_object64768:head# 2024-03-20T09:04:49.332 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b87f1890:::benchmark_data_smithi122_119671_object49989:head# 2024-03-20T09:04:49.333 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b88096b4:::benchmark_data_smithi122_119671_object49976:head# 2024-03-20T09:04:49.334 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8819e19:::benchmark_data_smithi122_119671_object80903:head# 2024-03-20T09:04:49.334 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b882f26e:::benchmark_data_smithi122_119671_object4814:head# 2024-03-20T09:04:49.334 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8849713:::benchmark_data_smithi122_119671_object20688:head# 2024-03-20T09:04:49.335 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b885e32d:::benchmark_data_smithi122_119671_object76055:head# 2024-03-20T09:04:49.335 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b886e33d:::benchmark_data_smithi122_119671_object24532:head# 2024-03-20T09:04:49.336 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8882a99:::benchmark_data_smithi122_119671_object65484:head# 2024-03-20T09:04:49.336 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8886c68:::benchmark_data_smithi122_119671_object58750:head# 2024-03-20T09:04:49.337 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b889d088:::benchmark_data_smithi122_119671_object72788:head# 2024-03-20T09:04:49.337 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b88a3411:::benchmark_data_smithi122_119671_object26339:head# 2024-03-20T09:04:49.337 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b88aee5b:::benchmark_data_smithi122_119671_object46409:head# 2024-03-20T09:04:49.338 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b88ba9af:::benchmark_data_smithi122_119671_object70515:head# 2024-03-20T09:04:49.338 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b88c8480:::benchmark_data_smithi122_119671_object60336:head# 2024-03-20T09:04:49.343 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b88d9e7a:::benchmark_data_smithi122_119671_object28091:head# 2024-03-20T09:04:49.344 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b88daa04:::benchmark_data_smithi122_119671_object81061:head# 2024-03-20T09:04:49.344 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b88e09d9:::benchmark_data_smithi122_119671_object35504:head# 2024-03-20T09:04:49.344 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b88fc2f6:::benchmark_data_smithi122_119671_object5597:head# 2024-03-20T09:04:49.344 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b891100c:::benchmark_data_smithi122_119671_object83940:head# 2024-03-20T09:04:49.344 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b89131c0:::benchmark_data_smithi122_119671_object67662:head# 2024-03-20T09:04:49.344 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b891b3ec:::benchmark_data_smithi122_119671_object94979:head# 2024-03-20T09:04:49.344 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8927c01:::benchmark_data_smithi122_119671_object13955:head# 2024-03-20T09:04:49.344 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8929f3a:::benchmark_data_smithi122_119671_object16661:head# 2024-03-20T09:04:49.344 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8938aa6:::benchmark_data_smithi122_119671_object21312:head# 2024-03-20T09:04:49.344 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b893a1ed:::benchmark_data_smithi122_119671_object59576:head# 2024-03-20T09:04:49.344 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8940d4d:::benchmark_data_smithi122_119671_object90270:head# 2024-03-20T09:04:49.345 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b894f64f:::benchmark_data_smithi122_119671_object47617:head# 2024-03-20T09:04:49.345 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8954849:::benchmark_data_smithi122_119671_object37336:head# 2024-03-20T09:04:49.345 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b896a67d:::benchmark_data_smithi122_119671_object55050:head# 2024-03-20T09:04:49.346 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b89945c6:::benchmark_data_smithi122_119671_object43812:head# 2024-03-20T09:04:49.346 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8996442:::benchmark_data_smithi122_119671_object28125:head# 2024-03-20T09:04:49.346 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8998378:::benchmark_data_smithi122_119671_object63888:head# 2024-03-20T09:04:49.346 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b899cc7f:::benchmark_data_smithi122_119671_object27653:head# 2024-03-20T09:04:49.347 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b89af1ff:::benchmark_data_smithi122_119671_object10767:head# 2024-03-20T09:04:49.347 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b89b0b55:::benchmark_data_smithi122_119671_object12893:head# 2024-03-20T09:04:49.347 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b89e5f63:::benchmark_data_smithi122_119671_object36363:head# 2024-03-20T09:04:49.348 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b89f594d:::benchmark_data_smithi122_119671_object20416:head# 2024-03-20T09:04:49.348 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8a0c9bd:::benchmark_data_smithi122_119671_object78613:head# 2024-03-20T09:04:49.349 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8a127b7:::benchmark_data_smithi122_119671_object75997:head# 2024-03-20T09:04:49.349 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8a18938:::benchmark_data_smithi122_119671_object64731:head# 2024-03-20T09:04:49.349 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8a2309e:::benchmark_data_smithi122_119671_object45130:head# 2024-03-20T09:04:49.350 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8a2c671:::benchmark_data_smithi122_119671_object69681:head# 2024-03-20T09:04:49.350 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8a3642c:::benchmark_data_smithi122_119671_object29187:head# 2024-03-20T09:04:49.350 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8a3b1f9:::benchmark_data_smithi122_119671_object5828:head# 2024-03-20T09:04:49.351 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8a40cb8:::benchmark_data_smithi122_119671_object69917:head# 2024-03-20T09:04:49.351 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8a46f3b:::benchmark_data_smithi122_119671_object36295:head# 2024-03-20T09:04:49.352 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8a50781:::benchmark_data_smithi122_119671_object14724:head# 2024-03-20T09:04:49.352 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8a685c1:::benchmark_data_smithi122_119671_object38686:head# 2024-03-20T09:04:49.353 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8a7a014:::benchmark_data_smithi122_119671_object12476:head# 2024-03-20T09:04:49.353 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8a8cc71:::benchmark_data_smithi122_119671_object83424:head# 2024-03-20T09:04:49.353 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8a8dd47:::benchmark_data_smithi122_119671_object83102:head# 2024-03-20T09:04:49.354 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8aa7734:::benchmark_data_smithi122_119671_object36202:head# 2024-03-20T09:04:49.354 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8ab657a:::benchmark_data_smithi122_119671_object73231:head# 2024-03-20T09:04:49.355 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8abe03d:::benchmark_data_smithi122_119671_object48514:head# 2024-03-20T09:04:49.355 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8ae8427:::benchmark_data_smithi122_119671_object58749:head# 2024-03-20T09:04:49.355 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8aecbc3:::benchmark_data_smithi122_119671_object3797:head# 2024-03-20T09:04:49.356 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8b0199c:::benchmark_data_smithi122_119671_object73548:head# 2024-03-20T09:04:49.356 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8b0e73e:::benchmark_data_smithi122_119671_object88630:head# 2024-03-20T09:04:49.357 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8b10637:::benchmark_data_smithi122_119671_object43953:head# 2024-03-20T09:04:49.357 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8b10f2b:::benchmark_data_smithi122_119671_object12032:head# 2024-03-20T09:04:49.357 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8b1df2a:::benchmark_data_smithi122_119671_object97161:head# 2024-03-20T09:04:49.358 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8b22e0b:::benchmark_data_smithi122_119671_object60226:head# 2024-03-20T09:04:49.358 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8b267e5:::benchmark_data_smithi122_119671_object55108:head# 2024-03-20T09:04:49.359 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8b27705:::benchmark_data_smithi122_119671_object68231:head# 2024-03-20T09:04:49.359 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8b324a0:::benchmark_data_smithi122_119671_object21969:head# 2024-03-20T09:04:49.359 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8b33d1e:::benchmark_data_smithi122_119671_object98531:head# 2024-03-20T09:04:49.360 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8b4030f:::benchmark_data_smithi122_119671_object2534:head# 2024-03-20T09:04:49.360 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8b42edd:::benchmark_data_smithi122_119671_object79798:head# 2024-03-20T09:04:49.360 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8b4cdbb:::benchmark_data_smithi122_119671_object42418:head# 2024-03-20T09:04:49.361 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8b5150f:::benchmark_data_smithi122_119671_object70956:head# 2024-03-20T09:04:49.361 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8b601db:::benchmark_data_smithi122_119671_object68404:head# 2024-03-20T09:04:49.362 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8b700ab:::benchmark_data_smithi122_119671_object52400:head# 2024-03-20T09:04:49.362 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8b70191:::benchmark_data_smithi122_119671_object87162:head# 2024-03-20T09:04:49.362 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8b8056c:::benchmark_data_smithi122_119671_object43130:head# 2024-03-20T09:04:49.363 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8b87560:::benchmark_data_smithi122_119671_object68852:head# 2024-03-20T09:04:49.363 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8b8e965:::benchmark_data_smithi122_119671_object83891:head# 2024-03-20T09:04:49.364 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8b97574:::benchmark_data_smithi122_119671_object88976:head# 2024-03-20T09:04:49.364 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8bac7be:::benchmark_data_smithi122_119671_object90076:head# 2024-03-20T09:04:49.364 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8bae45c:::benchmark_data_smithi122_119671_object71577:head# 2024-03-20T09:04:49.365 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8baf0f3:::benchmark_data_smithi122_119671_object29960:head# 2024-03-20T09:04:49.365 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8bb2aa5:::benchmark_data_smithi122_119671_object84206:head# 2024-03-20T09:04:49.365 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8bb9e7a:::benchmark_data_smithi122_119671_object62550:head# 2024-03-20T09:04:49.366 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8bbd115:::benchmark_data_smithi122_119671_object72218:head# 2024-03-20T09:04:49.366 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8bbfc5f:::benchmark_data_smithi122_119671_object92895:head# 2024-03-20T09:04:49.367 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8bd4acc:::benchmark_data_smithi122_119671_object86327:head# 2024-03-20T09:04:49.367 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8bd6582:::benchmark_data_smithi122_119671_object10196:head# 2024-03-20T09:04:49.368 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8be6b90:::benchmark_data_smithi122_119671_object10230:head# 2024-03-20T09:04:49.368 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8be8f89:::benchmark_data_smithi122_119671_object16890:head# 2024-03-20T09:04:49.368 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8bf6fd1:::benchmark_data_smithi122_119671_object46326:head# 2024-03-20T09:04:49.369 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8bfd623:::benchmark_data_smithi122_119671_object88318:head# 2024-03-20T09:04:49.369 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8c07091:::benchmark_data_smithi122_119671_object13623:head# 2024-03-20T09:04:49.370 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8c17004:::benchmark_data_smithi122_119671_object12413:head# 2024-03-20T09:04:49.370 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8c1763b:::benchmark_data_smithi122_119671_object89584:head# 2024-03-20T09:04:49.370 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8c200bc:::benchmark_data_smithi122_119671_object34988:head# 2024-03-20T09:04:49.371 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8c284ee:::benchmark_data_smithi122_119671_object35937:head# 2024-03-20T09:04:49.371 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8c2d3a4:::benchmark_data_smithi122_119671_object93965:head# 2024-03-20T09:04:49.372 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8c3ffa1:::benchmark_data_smithi122_119671_object61432:head# 2024-03-20T09:04:49.372 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8c5b828:::benchmark_data_smithi122_119671_object24765:head# 2024-03-20T09:04:49.372 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8c5e562:::benchmark_data_smithi122_119671_object61081:head# 2024-03-20T09:04:49.373 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8c6a095:::benchmark_data_smithi122_119671_object46135:head# 2024-03-20T09:04:49.373 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8c747f2:::benchmark_data_smithi122_119671_object78931:head# 2024-03-20T09:04:49.373 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8c7c592:::benchmark_data_smithi122_119671_object15864:head# 2024-03-20T09:04:49.374 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8c7d806:::benchmark_data_smithi122_119671_object37742:head# 2024-03-20T09:04:49.374 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8c8a149:::benchmark_data_smithi122_119671_object80787:head# 2024-03-20T09:04:49.375 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8cb6a28:::benchmark_data_smithi122_119671_object29517:head# 2024-03-20T09:04:49.375 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8cbfbf1:::benchmark_data_smithi122_119671_object32231:head# 2024-03-20T09:04:49.375 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8ce4304:::benchmark_data_smithi122_119671_object64678:head# 2024-03-20T09:04:49.376 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8ce5bb5:::benchmark_data_smithi122_119671_object9303:head# 2024-03-20T09:04:49.376 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8ced76c:::benchmark_data_smithi122_119671_object98007:head# 2024-03-20T09:04:49.377 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8cf76df:::benchmark_data_smithi122_119671_object94762:head# 2024-03-20T09:04:49.377 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8d00031:::benchmark_data_smithi122_119671_object44998:head# 2024-03-20T09:04:49.377 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8d18f6e:::benchmark_data_smithi122_119671_object48791:head# 2024-03-20T09:04:49.377 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8d21a6b:::benchmark_data_smithi122_119671_object79566:head# 2024-03-20T09:04:49.378 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8d2a0b8:::benchmark_data_smithi122_119671_object36713:head# 2024-03-20T09:04:49.378 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8d31a04:::benchmark_data_smithi122_119671_object28715:head# 2024-03-20T09:04:49.379 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8d3edf6:::benchmark_data_smithi122_119671_object26863:head# 2024-03-20T09:04:49.379 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8d45102:::benchmark_data_smithi122_119671_object24402:head# 2024-03-20T09:04:49.379 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8d51923:::benchmark_data_smithi122_119671_object36634:head# 2024-03-20T09:04:49.380 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8d54d85:::benchmark_data_smithi122_119671_object23025:head# 2024-03-20T09:04:49.380 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8d6416d:::benchmark_data_smithi122_119671_object29574:head# 2024-03-20T09:04:49.380 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8d65239:::benchmark_data_smithi122_119671_object6911:head# 2024-03-20T09:04:49.381 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8d740f0:::benchmark_data_smithi122_119671_object78560:head# 2024-03-20T09:04:49.381 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8d9153a:::benchmark_data_smithi122_119671_object11155:head# 2024-03-20T09:04:49.382 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8d97ae6:::benchmark_data_smithi122_119671_object23035:head# 2024-03-20T09:04:49.382 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8d9f08b:::benchmark_data_smithi122_119671_object16987:head# 2024-03-20T09:04:49.382 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8da387c:::benchmark_data_smithi122_119671_object78847:head# 2024-03-20T09:04:49.383 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8da67a4:::benchmark_data_smithi122_119671_object34991:head# 2024-03-20T09:04:49.383 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8dc150b:::benchmark_data_smithi122_119671_object63087:head# 2024-03-20T09:04:49.384 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8dca9f5:::benchmark_data_smithi122_119671_object78646:head# 2024-03-20T09:04:49.384 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8dd16e3:::benchmark_data_smithi122_119671_object86404:head# 2024-03-20T09:04:49.385 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8dd491b:::benchmark_data_smithi122_119671_object36842:head# 2024-03-20T09:04:49.385 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8ddbaa3:::benchmark_data_smithi122_119671_object34432:head# 2024-03-20T09:04:49.386 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8de255b:::benchmark_data_smithi122_119671_object5583:head# 2024-03-20T09:04:49.386 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8de2621:::benchmark_data_smithi122_119671_object54238:head# 2024-03-20T09:04:49.386 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8decd5d:::benchmark_data_smithi122_119671_object93377:head# 2024-03-20T09:04:49.387 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8deefaf:::benchmark_data_smithi122_119671_object20465:head# 2024-03-20T09:04:49.387 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8df4194:::benchmark_data_smithi122_119671_object76485:head# 2024-03-20T09:04:49.388 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8e0085c:::benchmark_data_smithi122_119671_object31272:head# 2024-03-20T09:04:49.388 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8e0ab25:::benchmark_data_smithi122_119671_object47572:head# 2024-03-20T09:04:49.388 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8e216ea:::benchmark_data_smithi122_119671_object55934:head# 2024-03-20T09:04:49.389 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8e2922e:::benchmark_data_smithi122_119671_object30961:head# 2024-03-20T09:04:49.389 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8e2ad4f:::benchmark_data_smithi122_119671_object36924:head# 2024-03-20T09:04:49.391 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8e380a9:::benchmark_data_smithi122_119671_object97262:head# 2024-03-20T09:04:49.392 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8e3f74f:::benchmark_data_smithi122_119671_object44956:head# 2024-03-20T09:04:49.392 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8e55598:::benchmark_data_smithi122_119671_object64515:head# 2024-03-20T09:04:49.392 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8e7422a:::benchmark_data_smithi122_119671_object97619:head# 2024-03-20T09:04:49.392 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8e89798:::benchmark_data_smithi122_119671_object73894:head# 2024-03-20T09:04:49.393 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8eb2918:::benchmark_data_smithi122_119671_object71652:head# 2024-03-20T09:04:49.393 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8ebbf78:::benchmark_data_smithi122_119671_object32942:head# 2024-03-20T09:04:49.394 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8ec4946:::benchmark_data_smithi122_119671_object79676:head# 2024-03-20T09:04:49.394 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8ece423:::benchmark_data_smithi122_119671_object1065:head# 2024-03-20T09:04:49.394 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8ed88c7:::benchmark_data_smithi122_119671_object39997:head# 2024-03-20T09:04:49.395 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8ef3d30:::benchmark_data_smithi122_119671_object35995:head# 2024-03-20T09:04:49.395 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8ef4609:::benchmark_data_smithi122_119671_object23871:head# 2024-03-20T09:04:49.396 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8f05312:::benchmark_data_smithi122_119671_object34149:head# 2024-03-20T09:04:49.396 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8f05a0f:::benchmark_data_smithi122_119671_object82719:head# 2024-03-20T09:04:49.396 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8f14219:::benchmark_data_smithi122_119671_object49410:head# 2024-03-20T09:04:49.397 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8f23c75:::benchmark_data_smithi122_119671_object99381:head# 2024-03-20T09:04:49.397 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8f2cc87:::benchmark_data_smithi122_119671_object34464:head# 2024-03-20T09:04:49.397 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8f2fabf:::benchmark_data_smithi122_119671_object3369:head# 2024-03-20T09:04:49.398 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8f41033:::benchmark_data_smithi122_119671_object63458:head# 2024-03-20T09:04:49.398 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8f428df:::benchmark_data_smithi122_119671_object81213:head# 2024-03-20T09:04:49.398 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8f44bad:::benchmark_data_smithi122_119671_object58567:head# 2024-03-20T09:04:49.399 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8f5234a:::benchmark_data_smithi122_119671_object8867:head# 2024-03-20T09:04:49.399 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8f69853:::benchmark_data_smithi122_119671_object28064:head# 2024-03-20T09:04:49.400 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8f736a5:::benchmark_data_smithi122_119671_object10021:head# 2024-03-20T09:04:49.400 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8f7a2ee:::benchmark_data_smithi122_119671_object25768:head# 2024-03-20T09:04:49.401 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8f7ce74:::benchmark_data_smithi122_119671_object18370:head# 2024-03-20T09:04:49.401 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8f82fab:::benchmark_data_smithi122_119671_object55374:head# 2024-03-20T09:04:49.401 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8f9102b:::benchmark_data_smithi122_119671_object77709:head# 2024-03-20T09:04:49.401 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8f9cab8:::benchmark_data_smithi122_119671_object61826:head# 2024-03-20T09:04:49.402 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8f9f4e0:::benchmark_data_smithi122_119671_object71523:head# 2024-03-20T09:04:49.402 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8faa30a:::benchmark_data_smithi122_119671_object1300:head# 2024-03-20T09:04:49.402 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8facbf8:::benchmark_data_smithi122_119671_object86230:head# 2024-03-20T09:04:49.403 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8fae347:::benchmark_data_smithi122_119671_object16452:head# 2024-03-20T09:04:49.403 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8fb902e:::benchmark_data_smithi122_119671_object45083:head# 2024-03-20T09:04:49.403 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8fc4c79:::benchmark_data_smithi122_119671_object77732:head# 2024-03-20T09:04:49.403 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8fcb669:::benchmark_data_smithi122_119671_object35609:head# 2024-03-20T09:04:49.404 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8fd0810:::benchmark_data_smithi122_119671_object64189:head# 2024-03-20T09:04:49.404 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8fd7e92:::benchmark_data_smithi122_119671_object86303:head# 2024-03-20T09:04:49.405 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8fe7588:::benchmark_data_smithi122_119671_object44491:head# 2024-03-20T09:04:49.405 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8fee053:::benchmark_data_smithi122_119671_object85541:head# 2024-03-20T09:04:49.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:48 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:04:49.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:48 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:04:49.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:48 smithi115 ceph-mon[72476]: osdmap e284: 12 total, 11 up, 11 in 2024-03-20T09:04:49.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:48 smithi115 ceph-mon[72476]: 3.19 scrub ok 2024-03-20T09:04:49.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:48 smithi115 ceph-mon[72476]: 1.13 scrub starts 2024-03-20T09:04:49.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:48 smithi115 ceph-mon[72476]: 1.13 scrub ok 2024-03-20T09:04:49.407 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b8ff7ab1:::benchmark_data_smithi122_119671_object64068:head# 2024-03-20T09:04:49.407 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b900722a:::benchmark_data_smithi122_119671_object48408:head# 2024-03-20T09:04:49.407 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b900db6c:::benchmark_data_smithi122_119671_object11840:head# 2024-03-20T09:04:49.407 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9019f56:::benchmark_data_smithi122_119671_object7578:head# 2024-03-20T09:04:49.407 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9043c7d:::benchmark_data_smithi122_119671_object52338:head# 2024-03-20T09:04:49.408 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b904f8b8:::benchmark_data_smithi122_119671_object78537:head# 2024-03-20T09:04:49.408 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b90537c1:::benchmark_data_smithi122_119671_object91972:head# 2024-03-20T09:04:49.408 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b905429b:::benchmark_data_smithi122_119671_object78224:head# 2024-03-20T09:04:49.409 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b905d63b:::benchmark_data_smithi122_119671_object56952:head# 2024-03-20T09:04:49.409 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9063833:::benchmark_data_smithi122_119671_object9495:head# 2024-03-20T09:04:49.410 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b907dae6:::benchmark_data_smithi122_119671_object15594:head# 2024-03-20T09:04:49.410 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b908409e:::benchmark_data_smithi122_119671_object59541:head# 2024-03-20T09:04:49.411 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9087d89:::benchmark_data_smithi122_119671_object88270:head# 2024-03-20T09:04:49.411 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b908d36e:::benchmark_data_smithi122_119671_object8961:head# 2024-03-20T09:04:49.411 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b909330e:::benchmark_data_smithi122_119671_object40141:head# 2024-03-20T09:04:49.412 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b909b155:::benchmark_data_smithi122_119671_object66697:head# 2024-03-20T09:04:49.412 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b90a53fc:::benchmark_data_smithi122_119671_object37256:head# 2024-03-20T09:04:49.412 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b90acc01:::benchmark_data_smithi122_119671_object30867:head# 2024-03-20T09:04:49.413 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b90acfac:::benchmark_data_smithi122_119671_object15028:head# 2024-03-20T09:04:49.413 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b90b75c6:::benchmark_data_smithi122_119671_object53480:head# 2024-03-20T09:04:49.413 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b90b8999:::benchmark_data_smithi122_119671_object70877:head# 2024-03-20T09:04:49.414 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b90bc89c:::benchmark_data_smithi122_119671_object48200:head# 2024-03-20T09:04:49.414 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b90c58c3:::benchmark_data_smithi122_119671_object81286:head# 2024-03-20T09:04:49.415 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b90ca221:::benchmark_data_smithi122_119671_object27128:head# 2024-03-20T09:04:49.415 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b90cb372:::benchmark_data_smithi122_119671_object98075:head# 2024-03-20T09:04:49.416 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b90d3ffb:::benchmark_data_smithi122_119671_object46258:head# 2024-03-20T09:04:49.416 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b90d924d:::benchmark_data_smithi122_119671_object72610:head# 2024-03-20T09:04:49.416 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b90ec698:::benchmark_data_smithi122_119671_object54664:head# 2024-03-20T09:04:49.417 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b90f8a34:::benchmark_data_smithi122_119671_object64648:head# 2024-03-20T09:04:49.417 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b90f9e9b:::benchmark_data_smithi122_119671_object11002:head# 2024-03-20T09:04:49.419 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9118a5a:::benchmark_data_smithi122_119671_object2568:head# 2024-03-20T09:04:49.419 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9131ec0:::benchmark_data_smithi122_119671_object86386:head# 2024-03-20T09:04:49.420 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9158141:::benchmark_data_smithi122_119671_object38674:head# 2024-03-20T09:04:49.420 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b915ab48:::benchmark_data_smithi122_119671_object79509:head# 2024-03-20T09:04:49.420 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b915df4f:::benchmark_data_smithi122_119671_object87068:head# 2024-03-20T09:04:49.421 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b915e834:::benchmark_data_smithi122_119671_object94794:head# 2024-03-20T09:04:49.421 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9160479:::benchmark_data_smithi122_119671_object37192:head# 2024-03-20T09:04:49.421 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9160e98:::benchmark_data_smithi122_119671_object49154:head# 2024-03-20T09:04:49.422 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9174135:::benchmark_data_smithi122_119671_object21735:head# 2024-03-20T09:04:49.422 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9174ef9:::benchmark_data_smithi122_119671_object86677:head# 2024-03-20T09:04:49.423 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9176c96:::benchmark_data_smithi122_119671_object11738:head# 2024-03-20T09:04:49.423 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9177e75:::benchmark_data_smithi122_119671_object19925:head# 2024-03-20T09:04:49.424 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b91855c9:::benchmark_data_smithi122_119671_object18137:head# 2024-03-20T09:04:49.424 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9189a5a:::benchmark_data_smithi122_119671_object13838:head# 2024-03-20T09:04:49.424 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9197ff9:::benchmark_data_smithi122_119671_object25265:head# 2024-03-20T09:04:49.425 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9199107:::benchmark_data_smithi122_119671_object73641:head# 2024-03-20T09:04:49.425 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b91a7e1d:::benchmark_data_smithi122_119671_object31352:head# 2024-03-20T09:04:49.426 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b91baa5e:::benchmark_data_smithi122_119671_object66975:head# 2024-03-20T09:04:49.426 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b91cf95d:::benchmark_data_smithi122_119671_object10573:head# 2024-03-20T09:04:49.427 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b91d07bb:::benchmark_data_smithi122_119671_object42529:head# 2024-03-20T09:04:49.427 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b91d38c7:::benchmark_data_smithi122_119671_object92814:head# 2024-03-20T09:04:49.427 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b91e6a75:::benchmark_data_smithi122_119671_object66686:head# 2024-03-20T09:04:49.428 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b91f2091:::benchmark_data_smithi122_119671_object2184:head# 2024-03-20T09:04:49.428 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b91f4598:::benchmark_data_smithi122_119671_object22343:head# 2024-03-20T09:04:49.428 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9200b46:::benchmark_data_smithi122_119671_object74133:head# 2024-03-20T09:04:49.429 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b920402f:::benchmark_data_smithi122_119671_object59176:head# 2024-03-20T09:04:49.429 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b920fddc:::benchmark_data_smithi122_119671_object93782:head# 2024-03-20T09:04:49.430 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b92140ce:::benchmark_data_smithi122_119671_object11226:head# 2024-03-20T09:04:49.430 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b922615c:::benchmark_data_smithi122_119671_object79505:head# 2024-03-20T09:04:49.430 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9233030:::benchmark_data_smithi122_119671_object36251:head# 2024-03-20T09:04:49.432 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b923e7f2:::benchmark_data_smithi122_119671_object33709:head# 2024-03-20T09:04:49.432 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b92437d9:::benchmark_data_smithi122_119671_object25391:head# 2024-03-20T09:04:49.433 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9248625:::benchmark_data_smithi122_119671_object57894:head# 2024-03-20T09:04:49.433 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9256901:::benchmark_data_smithi122_119671_object54057:head# 2024-03-20T09:04:49.434 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b927748b:::benchmark_data_smithi122_119671_object18895:head# 2024-03-20T09:04:49.434 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9278251:::benchmark_data_smithi122_119671_object30760:head# 2024-03-20T09:04:49.435 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b928133e:::benchmark_data_smithi122_119671_object84785:head# 2024-03-20T09:04:49.435 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9288f57:::benchmark_data_smithi122_119671_object47946:head# 2024-03-20T09:04:49.435 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b928ac8d:::benchmark_data_smithi122_119671_object47876:head# 2024-03-20T09:04:49.436 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b928c174:::benchmark_data_smithi122_119671_object21395:head# 2024-03-20T09:04:49.436 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9293eb1:::benchmark_data_smithi122_119671_object17432:head# 2024-03-20T09:04:49.436 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b929da67:::benchmark_data_smithi122_119671_object32965:head# 2024-03-20T09:04:49.437 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b92a7c47:::benchmark_data_smithi122_119671_object30782:head# 2024-03-20T09:04:49.437 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b92d1194:::benchmark_data_smithi122_119671_object53741:head# 2024-03-20T09:04:49.438 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b92d3f2d:::benchmark_data_smithi122_119671_object62000:head# 2024-03-20T09:04:49.438 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b92df8bb:::benchmark_data_smithi122_119671_object9745:head# 2024-03-20T09:04:49.441 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b92e61bd:::benchmark_data_smithi122_119671_object3032:head# 2024-03-20T09:04:49.442 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b92eaf42:::benchmark_data_smithi122_119671_object29580:head# 2024-03-20T09:04:49.442 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b92f2ad2:::benchmark_data_smithi122_119671_object60367:head# 2024-03-20T09:04:49.442 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b930409f:::benchmark_data_smithi122_119671_object12942:head# 2024-03-20T09:04:49.443 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b931e70d:::benchmark_data_smithi122_119671_object24916:head# 2024-03-20T09:04:49.445 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b932d345:::benchmark_data_smithi122_119671_object40701:head# 2024-03-20T09:04:49.446 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9355175:::benchmark_data_smithi122_119671_object91862:head# 2024-03-20T09:04:49.446 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9357e85:::benchmark_data_smithi122_119671_object27226:head# 2024-03-20T09:04:49.447 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9361b94:::benchmark_data_smithi122_119671_object99153:head# 2024-03-20T09:04:49.448 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b93776be:::benchmark_data_smithi122_119671_object27390:head# 2024-03-20T09:04:49.449 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b937b627:::benchmark_data_smithi122_119671_object17129:head# 2024-03-20T09:04:49.449 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b937bc76:::benchmark_data_smithi122_119671_object62651:head# 2024-03-20T09:04:49.450 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9389186:::benchmark_data_smithi122_119671_object53020:head# 2024-03-20T09:04:49.450 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b938b7f2:::benchmark_data_smithi122_119671_object96325:head# 2024-03-20T09:04:49.450 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b93925aa:::benchmark_data_smithi122_119671_object1935:head# 2024-03-20T09:04:49.451 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b939becd:::benchmark_data_smithi122_119671_object2067:head# 2024-03-20T09:04:49.451 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b93b04b5:::benchmark_data_smithi122_119671_object72497:head# 2024-03-20T09:04:49.453 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b93b1843:::benchmark_data_smithi122_119671_object83040:head# 2024-03-20T09:04:49.454 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b93b756f:::benchmark_data_smithi122_119671_object56002:head# 2024-03-20T09:04:49.454 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b93b88b5:::benchmark_data_smithi122_119671_object59078:head# 2024-03-20T09:04:49.455 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b93c2489:::benchmark_data_smithi122_119671_object5620:head# 2024-03-20T09:04:49.455 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b93c4647:::benchmark_data_smithi122_119671_object43940:head# 2024-03-20T09:04:49.455 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b93d63b8:::benchmark_data_smithi122_119671_object66828:head# 2024-03-20T09:04:49.455 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b93db787:::benchmark_data_smithi122_119671_object31992:head# 2024-03-20T09:04:49.456 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b93e0d5e:::benchmark_data_smithi122_119671_object28738:head# 2024-03-20T09:04:49.456 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b93efd92:::benchmark_data_smithi122_119671_object14418:head# 2024-03-20T09:04:49.457 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b93f3da3:::benchmark_data_smithi122_119671_object95038:head# 2024-03-20T09:04:49.458 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b93f74b5:::benchmark_data_smithi122_119671_object28941:head# 2024-03-20T09:04:49.458 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9402414:::benchmark_data_smithi122_119671_object64682:head# 2024-03-20T09:04:49.458 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b94074ba:::benchmark_data_smithi122_119671_object67960:head# 2024-03-20T09:04:49.459 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b940dea0:::benchmark_data_smithi122_119671_object74923:head# 2024-03-20T09:04:49.459 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9414979:::benchmark_data_smithi122_119671_object30957:head# 2024-03-20T09:04:49.464 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9435cd2:::benchmark_data_smithi122_119671_object84714:head# 2024-03-20T09:04:49.464 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b94381c1:::benchmark_data_smithi122_119671_object20890:head# 2024-03-20T09:04:49.464 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9446cca:::benchmark_data_smithi122_119671_object97261:head# 2024-03-20T09:04:49.465 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9446e62:::benchmark_data_smithi122_119671_object47843:head# 2024-03-20T09:04:49.465 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b94491f7:::benchmark_data_smithi122_119671_object86470:head# 2024-03-20T09:04:49.465 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b94559a6:::benchmark_data_smithi122_119671_object50588:head# 2024-03-20T09:04:49.465 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9458212:::benchmark_data_smithi122_119671_object16986:head# 2024-03-20T09:04:49.465 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b945c112:::benchmark_data_smithi122_119671_object47314:head# 2024-03-20T09:04:49.466 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9461258:::benchmark_data_smithi122_119671_object62310:head# 2024-03-20T09:04:49.467 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b947016f:::benchmark_data_smithi122_119671_object72459:head# 2024-03-20T09:04:49.474 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b94774d5:::benchmark_data_smithi122_119671_object77610:head# 2024-03-20T09:04:49.474 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b947ac8d:::benchmark_data_smithi122_119671_object6848:head# 2024-03-20T09:04:49.475 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b947f9f1:::benchmark_data_smithi122_119671_object2619:head# 2024-03-20T09:04:49.476 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9481e76:::benchmark_data_smithi122_119671_object98578:head# 2024-03-20T09:04:49.476 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9483ba2:::benchmark_data_smithi122_119671_object36449:head# 2024-03-20T09:04:49.476 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b948c72a:::benchmark_data_smithi122_119671_object95633:head# 2024-03-20T09:04:49.476 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b94a7244:::benchmark_data_smithi122_119671_object16390:head# 2024-03-20T09:04:49.476 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b94a9f18:::benchmark_data_smithi122_119671_object63319:head# 2024-03-20T09:04:49.476 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b94b5a8b:::benchmark_data_smithi122_119671_object99092:head# 2024-03-20T09:04:49.476 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b94b7e10:::benchmark_data_smithi122_119671_object79172:head# 2024-03-20T09:04:49.476 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b94cca34:::benchmark_data_smithi122_119671_object84795:head# 2024-03-20T09:04:49.476 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b94d4ed6:::benchmark_data_smithi122_119671_object86656:head# 2024-03-20T09:04:49.476 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b94db033:::benchmark_data_smithi122_119671_object42208:head# 2024-03-20T09:04:49.477 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b94e6699:::benchmark_data_smithi122_119671_object78878:head# 2024-03-20T09:04:49.477 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b94eaac4:::benchmark_data_smithi122_119671_object57482:head# 2024-03-20T09:04:49.478 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b94ec385:::benchmark_data_smithi122_119671_object99739:head# 2024-03-20T09:04:49.479 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b94fa278:::benchmark_data_smithi122_119671_object91759:head# 2024-03-20T09:04:49.479 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9505ed4:::benchmark_data_smithi122_119671_object77766:head# 2024-03-20T09:04:49.480 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9509231:::benchmark_data_smithi122_119671_object51606:head# 2024-03-20T09:04:49.482 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9521450:::benchmark_data_smithi122_119671_object23130:head# 2024-03-20T09:04:49.486 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9534c64:::benchmark_data_smithi122_119671_object58722:head# 2024-03-20T09:04:49.487 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9539eae:::benchmark_data_smithi122_119671_object4126:head# 2024-03-20T09:04:49.487 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b953c4ff:::benchmark_data_smithi122_119671_object82291:head# 2024-03-20T09:04:49.487 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9549cde:::benchmark_data_smithi122_119671_object18655:head# 2024-03-20T09:04:49.488 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b954c1ae:::benchmark_data_smithi122_119671_object71853:head# 2024-03-20T09:04:49.488 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b954f05b:::benchmark_data_smithi122_119671_object8728:head# 2024-03-20T09:04:49.488 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b955bbec:::benchmark_data_smithi122_119671_object2898:head# 2024-03-20T09:04:49.491 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9563bc2:::benchmark_data_smithi122_119671_object61277:head# 2024-03-20T09:04:49.491 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b956858c:::benchmark_data_smithi122_119671_object54545:head# 2024-03-20T09:04:49.495 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b956b3f8:::benchmark_data_smithi122_119671_object20613:head# 2024-03-20T09:04:49.495 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9572b44:::benchmark_data_smithi122_119671_object66151:head# 2024-03-20T09:04:49.495 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b958936a:::benchmark_data_smithi122_119671_object34291:head# 2024-03-20T09:04:49.495 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9589a9e:::benchmark_data_smithi122_119671_object98916:head# 2024-03-20T09:04:49.497 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b958d381:::benchmark_data_smithi122_119671_object75429:head# 2024-03-20T09:04:49.497 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9591ec5:::benchmark_data_smithi122_119671_object92865:head# 2024-03-20T09:04:49.497 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9594b15:::benchmark_data_smithi122_119671_object42796:head# 2024-03-20T09:04:49.497 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b95967fa:::benchmark_data_smithi122_119671_object8756:head# 2024-03-20T09:04:49.497 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b959942c:::benchmark_data_smithi122_119671_object2909:head# 2024-03-20T09:04:49.497 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b95a60c8:::benchmark_data_smithi122_119671_object95745:head# 2024-03-20T09:04:49.497 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b95b137a:::benchmark_data_smithi122_119671_object91048:head# 2024-03-20T09:04:49.497 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b95bfbfe:::benchmark_data_smithi122_119671_object51398:head# 2024-03-20T09:04:49.497 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b95c6009:::benchmark_data_smithi122_119671_object65128:head# 2024-03-20T09:04:49.498 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b95f5697:::benchmark_data_smithi122_119671_object73848:head# 2024-03-20T09:04:49.498 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b95febc4:::benchmark_data_smithi122_119671_object95843:head# 2024-03-20T09:04:49.499 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b961b33e:::benchmark_data_smithi122_119671_object93447:head# 2024-03-20T09:04:49.499 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9620096:::benchmark_data_smithi122_119671_object34064:head# 2024-03-20T09:04:49.500 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b96225bc:::benchmark_data_smithi122_119671_object62452:head# 2024-03-20T09:04:49.500 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b962320d:::benchmark_data_smithi122_119671_object58838:head# 2024-03-20T09:04:49.502 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9626240:::benchmark_data_smithi122_119671_object83211:head# 2024-03-20T09:04:49.503 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b962af8f:::benchmark_data_smithi122_119671_object13230:head# 2024-03-20T09:04:49.503 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b962b557:::benchmark_data_smithi122_119671_object18735:head# 2024-03-20T09:04:49.503 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b962fb5f:::benchmark_data_smithi122_119671_object91725:head# 2024-03-20T09:04:49.503 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9631498:::benchmark_data_smithi122_119671_object69764:head# 2024-03-20T09:04:49.503 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b96343ff:::benchmark_data_smithi122_119671_object99274:head# 2024-03-20T09:04:49.504 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9634acb:::benchmark_data_smithi122_119671_object10797:head# 2024-03-20T09:04:49.504 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9639325:::benchmark_data_smithi122_119671_object77348:head# 2024-03-20T09:04:49.505 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9639cb5:::benchmark_data_smithi122_119671_object17154:head# 2024-03-20T09:04:49.505 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9644b24:::benchmark_data_smithi122_119671_object55747:head# 2024-03-20T09:04:49.505 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b965f38c:::benchmark_data_smithi122_119671_object53892:head# 2024-03-20T09:04:49.506 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b966ff50:::benchmark_data_smithi122_119671_object85091:head# 2024-03-20T09:04:49.507 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b96703b8:::benchmark_data_smithi122_119671_object42246:head# 2024-03-20T09:04:49.507 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b967624c:::benchmark_data_smithi122_119671_object97369:head# 2024-03-20T09:04:49.507 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b96766f0:::benchmark_data_smithi122_119671_object41636:head# 2024-03-20T09:04:49.508 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9695070:::benchmark_data_smithi122_119671_object37577:head# 2024-03-20T09:04:49.508 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b96a9ffe:::benchmark_data_smithi122_119671_object5452:head# 2024-03-20T09:04:49.509 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b96b5258:::benchmark_data_smithi122_119671_object43978:head# 2024-03-20T09:04:49.509 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b96bbd5c:::benchmark_data_smithi122_119671_object7374:head# 2024-03-20T09:04:49.510 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b96cff93:::benchmark_data_smithi122_119671_object34609:head# 2024-03-20T09:04:49.511 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b96ddae3:::benchmark_data_smithi122_119671_object97627:head# 2024-03-20T09:04:49.512 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b96e22f9:::benchmark_data_smithi122_119671_object46283:head# 2024-03-20T09:04:49.512 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b96e6d44:::benchmark_data_smithi122_119671_object23711:head# 2024-03-20T09:04:49.512 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b96e7fcf:::benchmark_data_smithi122_119671_object62398:head# 2024-03-20T09:04:49.513 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b96f73ef:::benchmark_data_smithi122_119671_object87672:head# 2024-03-20T09:04:49.513 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b97071bf:::benchmark_data_smithi122_119671_object3112:head# 2024-03-20T09:04:49.513 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b97100c3:::benchmark_data_smithi122_119671_object98487:head# 2024-03-20T09:04:49.514 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b971917c:::benchmark_data_smithi122_119671_object73519:head# 2024-03-20T09:04:49.514 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b97280ef:::benchmark_data_smithi122_119671_object36451:head# 2024-03-20T09:04:49.515 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9728bd5:::benchmark_data_smithi122_119671_object97086:head# 2024-03-20T09:04:49.517 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b97438ea:::benchmark_data_smithi122_119671_object49953:head# 2024-03-20T09:04:49.517 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b974a3b5:::benchmark_data_smithi122_119671_object31070:head# 2024-03-20T09:04:49.517 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b975443d:::benchmark_data_smithi122_119671_object31307:head# 2024-03-20T09:04:49.519 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b975eb36:::benchmark_data_smithi122_119671_object79479:head# 2024-03-20T09:04:49.519 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b97639c7:::benchmark_data_smithi122_119671_object19744:head# 2024-03-20T09:04:49.522 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b976a180:::benchmark_data_smithi122_119671_object74112:head# 2024-03-20T09:04:49.524 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b976fc70:::benchmark_data_smithi122_119671_object20284:head# 2024-03-20T09:04:49.524 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9775e72:::benchmark_data_smithi122_119671_object85572:head# 2024-03-20T09:04:49.525 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b979105e:::benchmark_data_smithi122_119671_object41188:head# 2024-03-20T09:04:49.525 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b97b7215:::benchmark_data_smithi122_119671_object45295:head# 2024-03-20T09:04:49.526 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b97b8ef1:::benchmark_data_smithi122_119671_object1352:head# 2024-03-20T09:04:49.526 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b97c90e0:::benchmark_data_smithi122_119671_object8225:head# 2024-03-20T09:04:49.526 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b97dba9e:::benchmark_data_smithi122_119671_object7906:head# 2024-03-20T09:04:49.526 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b97f1a1e:::benchmark_data_smithi122_119671_object6765:head# 2024-03-20T09:04:49.528 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b97fd53d:::benchmark_data_smithi122_119671_object89204:head# 2024-03-20T09:04:49.528 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b98028f0:::benchmark_data_smithi122_119671_object14142:head# 2024-03-20T09:04:49.528 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b980ce57:::benchmark_data_smithi122_119671_object83613:head# 2024-03-20T09:04:49.528 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b980f749:::benchmark_data_smithi122_119671_object12714:head# 2024-03-20T09:04:49.529 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b98237a8:::benchmark_data_smithi122_119671_object13414:head# 2024-03-20T09:04:49.529 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b983cc1c:::benchmark_data_smithi122_119671_object52344:head# 2024-03-20T09:04:49.529 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b983d186:::benchmark_data_smithi122_119671_object99600:head# 2024-03-20T09:04:49.530 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b98404c2:::benchmark_data_smithi122_119671_object84415:head# 2024-03-20T09:04:49.530 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9845df1:::benchmark_data_smithi122_119671_object94103:head# 2024-03-20T09:04:49.531 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b984ccdb:::benchmark_data_smithi122_119671_object10609:head# 2024-03-20T09:04:49.532 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b985df26:::benchmark_data_smithi122_119671_object18112:head# 2024-03-20T09:04:49.532 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b98818db:::benchmark_data_smithi122_119671_object52517:head# 2024-03-20T09:04:49.534 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9885714:::benchmark_data_smithi122_119671_object23646:head# 2024-03-20T09:04:49.534 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b988da1b:::benchmark_data_smithi122_119671_object54424:head# 2024-03-20T09:04:49.534 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b989debb:::benchmark_data_smithi122_119671_object57757:head# 2024-03-20T09:04:49.534 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b98a0a4d:::benchmark_data_smithi122_119671_object55510:head# 2024-03-20T09:04:49.534 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b98a0c10:::benchmark_data_smithi122_119671_object82293:head# 2024-03-20T09:04:49.535 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b98a0f13:::benchmark_data_smithi122_119671_object9287:head# 2024-03-20T09:04:49.535 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b98a4fb9:::benchmark_data_smithi122_119671_object27707:head# 2024-03-20T09:04:49.535 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b98ae761:::benchmark_data_smithi122_119671_object73728:head# 2024-03-20T09:04:49.535 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b98b0492:::benchmark_data_smithi122_119671_object80278:head# 2024-03-20T09:04:49.536 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b98ba5d0:::benchmark_data_smithi122_119671_object93417:head# 2024-03-20T09:04:49.537 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b98bfb00:::benchmark_data_smithi122_119671_object22432:head# 2024-03-20T09:04:49.537 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b98c07b9:::benchmark_data_smithi122_119671_object45653:head# 2024-03-20T09:04:49.537 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b98c1567:::benchmark_data_smithi122_119671_object15328:head# 2024-03-20T09:04:49.537 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b98c5f50:::benchmark_data_smithi122_119671_object69333:head# 2024-03-20T09:04:49.538 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b98c8aad:::benchmark_data_smithi122_119671_object64180:head# 2024-03-20T09:04:49.538 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b98cbc25:::benchmark_data_smithi122_119671_object4242:head# 2024-03-20T09:04:49.540 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b98f15c6:::benchmark_data_smithi122_119671_object4046:head# 2024-03-20T09:04:49.540 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b98f3b0d:::benchmark_data_smithi122_119671_object22377:head# 2024-03-20T09:04:49.540 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b98f606a:::benchmark_data_smithi122_119671_object12592:head# 2024-03-20T09:04:49.540 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b98f8afe:::benchmark_data_smithi122_119671_object81314:head# 2024-03-20T09:04:49.540 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9906bb6:::benchmark_data_smithi122_119671_object31425:head# 2024-03-20T09:04:49.541 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b99091a2:::benchmark_data_smithi122_119671_object45742:head# 2024-03-20T09:04:49.541 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9913257:::benchmark_data_smithi122_119671_object24074:head# 2024-03-20T09:04:49.542 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9924a02:::benchmark_data_smithi122_119671_object35960:head# 2024-03-20T09:04:49.542 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b992fb8b:::benchmark_data_smithi122_119671_object76547:head# 2024-03-20T09:04:49.545 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9937851:::benchmark_data_smithi122_119671_object47420:head# 2024-03-20T09:04:49.545 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b993cfd7:::benchmark_data_smithi122_119671_object26844:head# 2024-03-20T09:04:49.545 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9942518:::benchmark_data_smithi122_119671_object3004:head# 2024-03-20T09:04:49.545 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b99496fc:::benchmark_data_smithi122_119671_object13042:head# 2024-03-20T09:04:49.546 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b994d04e:::benchmark_data_smithi122_119671_object57981:head# 2024-03-20T09:04:49.546 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b994d545:::benchmark_data_smithi122_119671_object39357:head# 2024-03-20T09:04:49.546 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b994f44c:::benchmark_data_smithi122_119671_object32717:head# 2024-03-20T09:04:49.547 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9973030:::benchmark_data_smithi122_119671_object91835:head# 2024-03-20T09:04:49.547 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b997767c:::benchmark_data_smithi122_119671_object22216:head# 2024-03-20T09:04:49.548 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9985f99:::benchmark_data_smithi122_119671_object1284:head# 2024-03-20T09:04:49.548 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b998a9b0:::benchmark_data_smithi122_119671_object39600:head# 2024-03-20T09:04:49.549 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b998eed8:::benchmark_data_smithi122_119671_object53809:head# 2024-03-20T09:04:49.549 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b999dc13:::benchmark_data_smithi122_119671_object73979:head# 2024-03-20T09:04:49.549 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b99a1b4e:::benchmark_data_smithi122_119671_object58301:head# 2024-03-20T09:04:49.549 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b99a7edb:::benchmark_data_smithi122_119671_object31826:head# 2024-03-20T09:04:49.550 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b99a9d85:::benchmark_data_smithi122_119671_object65309:head# 2024-03-20T09:04:49.550 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b99b4a88:::benchmark_data_smithi122_119671_object57638:head# 2024-03-20T09:04:49.551 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b99b8d84:::benchmark_data_smithi122_119671_object6572:head# 2024-03-20T09:04:49.551 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b99bbd8c:::benchmark_data_smithi122_119671_object39095:head# 2024-03-20T09:04:49.552 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b99bc167:::benchmark_data_smithi122_119671_object85595:head# 2024-03-20T09:04:49.552 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b99cadd6:::benchmark_data_smithi122_119671_object20256:head# 2024-03-20T09:04:49.552 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b99dd343:::benchmark_data_smithi122_119671_object41923:head# 2024-03-20T09:04:49.554 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b99ea8cc:::benchmark_data_smithi122_119671_object6377:head# 2024-03-20T09:04:49.554 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b99ede2c:::benchmark_data_smithi122_119671_object19419:head# 2024-03-20T09:04:49.554 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b99fa2d1:::benchmark_data_smithi122_119671_object65984:head# 2024-03-20T09:04:49.554 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9a0a05e:::benchmark_data_smithi122_119671_object7778:head# 2024-03-20T09:04:49.555 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9a136e5:::benchmark_data_smithi122_119671_object46515:head# 2024-03-20T09:04:49.555 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9a15cdf:::benchmark_data_smithi122_119671_object58424:head# 2024-03-20T09:04:49.555 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9a288ac:::benchmark_data_smithi122_119671_object56376:head# 2024-03-20T09:04:49.556 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9a2d9f8:::benchmark_data_smithi122_119671_object71611:head# 2024-03-20T09:04:49.556 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9a37213:::benchmark_data_smithi122_119671_object64440:head# 2024-03-20T09:04:49.557 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9a4c393:::benchmark_data_smithi122_119671_object45608:head# 2024-03-20T09:04:49.557 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9a4cdf3:::benchmark_data_smithi122_119671_object21807:head# 2024-03-20T09:04:49.558 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9a60f50:::benchmark_data_smithi122_119671_object51664:head# 2024-03-20T09:04:49.558 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9a698f2:::benchmark_data_smithi122_119671_object72653:head# 2024-03-20T09:04:49.559 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9a6b89b:::benchmark_data_smithi122_119671_object19479:head# 2024-03-20T09:04:49.559 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9a6c304:::benchmark_data_smithi122_119671_object92917:head# 2024-03-20T09:04:49.560 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9a6c5f5:::benchmark_data_smithi122_119671_object51271:head# 2024-03-20T09:04:49.560 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9a797ff:::benchmark_data_smithi122_119671_object35951:head# 2024-03-20T09:04:49.560 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9a7dda0:::benchmark_data_smithi122_119671_object55144:head# 2024-03-20T09:04:49.561 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9aa4275:::benchmark_data_smithi122_119671_object46827:head# 2024-03-20T09:04:49.561 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9aa5320:::benchmark_data_smithi122_119671_object77228:head# 2024-03-20T09:04:49.562 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9ab1f85:::benchmark_data_smithi122_119671_object81726:head# 2024-03-20T09:04:49.562 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9ac9af0:::benchmark_data_smithi122_119671_object12173:head# 2024-03-20T09:04:49.562 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9ada4f6:::benchmark_data_smithi122_119671_object14104:head# 2024-03-20T09:04:49.568 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9ae0524:::benchmark_data_smithi122_119671_object19563:head# 2024-03-20T09:04:49.568 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9aeb6fd:::benchmark_data_smithi122_119671_object95598:head# 2024-03-20T09:04:49.568 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9aec28f:::benchmark_data_smithi122_119671_object29362:head# 2024-03-20T09:04:49.568 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9aecea9:::benchmark_data_smithi122_119671_object22042:head# 2024-03-20T09:04:49.568 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9afa6a2:::benchmark_data_smithi122_119671_object79029:head# 2024-03-20T09:04:49.568 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9afb612:::benchmark_data_smithi122_119671_object31762:head# 2024-03-20T09:04:49.568 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9afbd37:::benchmark_data_smithi122_119671_object24883:head# 2024-03-20T09:04:49.568 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9afc52a:::benchmark_data_smithi122_119671_object16716:head# 2024-03-20T09:04:49.568 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9b03612:::benchmark_data_smithi122_119671_object6651:head# 2024-03-20T09:04:49.568 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9b1608f:::benchmark_data_smithi122_119671_object18738:head# 2024-03-20T09:04:49.568 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9b1b761:::benchmark_data_smithi122_119671_object71955:head# 2024-03-20T09:04:49.569 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9b2ff13:::benchmark_data_smithi122_119671_object20834:head# 2024-03-20T09:04:49.569 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9b37523:::benchmark_data_smithi122_119671_object19599:head# 2024-03-20T09:04:49.569 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9b39895:::benchmark_data_smithi122_119671_object51140:head# 2024-03-20T09:04:49.569 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9b4297f:::benchmark_data_smithi122_119671_object80793:head# 2024-03-20T09:04:49.569 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9b4c750:::benchmark_data_smithi122_119671_object93218:head# 2024-03-20T09:04:49.570 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9b4e08a:::benchmark_data_smithi122_119671_object23231:head# 2024-03-20T09:04:49.570 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9b5300e:::benchmark_data_smithi122_119671_object16265:head# 2024-03-20T09:04:49.570 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9b558b1:::benchmark_data_smithi122_119671_object90626:head# 2024-03-20T09:04:49.571 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9b5cc53:::benchmark_data_smithi122_119671_object6451:head# 2024-03-20T09:04:49.571 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9b78c4e:::benchmark_data_smithi122_119671_object90193:head# 2024-03-20T09:04:49.571 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9b85dde:::benchmark_data_smithi122_119671_object82600:head# 2024-03-20T09:04:49.572 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9b8b00b:::benchmark_data_smithi122_119671_object46372:head# 2024-03-20T09:04:49.572 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9b9e1af:::benchmark_data_smithi122_119671_object34566:head# 2024-03-20T09:04:49.572 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9ba33d8:::benchmark_data_smithi122_119671_object7547:head# 2024-03-20T09:04:49.573 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9bb24b7:::benchmark_data_smithi122_119671_object40490:head# 2024-03-20T09:04:49.573 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9bb350b:::benchmark_data_smithi122_119671_object68057:head# 2024-03-20T09:04:49.574 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9bb65d1:::benchmark_data_smithi122_119671_object20368:head# 2024-03-20T09:04:49.574 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9bb7b12:::benchmark_data_smithi122_119671_object42436:head# 2024-03-20T09:04:49.575 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9bcd50f:::benchmark_data_smithi122_119671_object8040:head# 2024-03-20T09:04:49.575 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9bd934d:::benchmark_data_smithi122_119671_object88936:head# 2024-03-20T09:04:49.575 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9bdb907:::benchmark_data_smithi122_119671_object40815:head# 2024-03-20T09:04:49.576 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9be559e:::benchmark_data_smithi122_119671_object45660:head# 2024-03-20T09:04:49.576 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9be8d34:::benchmark_data_smithi122_119671_object84366:head# 2024-03-20T09:04:49.577 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9bf3464:::benchmark_data_smithi122_119671_object80045:head# 2024-03-20T09:04:49.577 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9bfcf37:::benchmark_data_smithi122_119671_object47186:head# 2024-03-20T09:04:49.578 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9c02a6a:::benchmark_data_smithi122_119671_object86896:head# 2024-03-20T09:04:49.578 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9c0ac2c:::benchmark_data_smithi122_119671_object79727:head# 2024-03-20T09:04:49.578 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9c105de:::benchmark_data_smithi122_119671_object19361:head# 2024-03-20T09:04:49.579 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9c1f438:::benchmark_data_smithi122_119671_object7566:head# 2024-03-20T09:04:49.579 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9c30386:::benchmark_data_smithi122_119671_object28372:head# 2024-03-20T09:04:49.580 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9c377cc:::benchmark_data_smithi122_119671_object81489:head# 2024-03-20T09:04:49.580 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9c39c1a:::benchmark_data_smithi122_119671_object8266:head# 2024-03-20T09:04:49.580 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9c3dca8:::benchmark_data_smithi122_119671_object41948:head# 2024-03-20T09:04:49.581 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9c6940b:::benchmark_data_smithi122_119671_object94116:head# 2024-03-20T09:04:49.581 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9c69ecd:::benchmark_data_smithi122_119671_object69280:head# 2024-03-20T09:04:49.582 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9c9046a:::benchmark_data_smithi122_119671_object70131:head# 2024-03-20T09:04:49.582 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9cb66cd:::benchmark_data_smithi122_119671_object15160:head# 2024-03-20T09:04:49.582 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9cb6923:::benchmark_data_smithi122_119671_object2480:head# 2024-03-20T09:04:49.583 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9ccd2ec:::benchmark_data_smithi122_119671_object31036:head# 2024-03-20T09:04:49.583 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9cd4e3a:::benchmark_data_smithi122_119671_object60448:head# 2024-03-20T09:04:49.584 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9cdab2d:::benchmark_data_smithi122_119671_object88295:head# 2024-03-20T09:04:49.584 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9cde496:::benchmark_data_smithi122_119671_object16223:head# 2024-03-20T09:04:49.585 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9d14139:::benchmark_data_smithi122_119671_object59524:head# 2024-03-20T09:04:49.585 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9d2ae0c:::benchmark_data_smithi122_119671_object76091:head# 2024-03-20T09:04:49.586 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9d3a052:::benchmark_data_smithi122_119671_object96266:head# 2024-03-20T09:04:49.587 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9d3c9cd:::benchmark_data_smithi122_119671_object9458:head# 2024-03-20T09:04:49.587 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9d3fe28:::benchmark_data_smithi122_119671_object70803:head# 2024-03-20T09:04:49.587 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9d40f1f:::benchmark_data_smithi122_119671_object10530:head# 2024-03-20T09:04:49.587 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9d4a8a6:::benchmark_data_smithi122_119671_object35413:head# 2024-03-20T09:04:49.588 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9d52130:::benchmark_data_smithi122_119671_object42115:head# 2024-03-20T09:04:49.588 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9d61366:::benchmark_data_smithi122_119671_object69900:head# 2024-03-20T09:04:49.589 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9d65931:::benchmark_data_smithi122_119671_object1133:head# 2024-03-20T09:04:49.589 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9d7eca4:::benchmark_data_smithi122_119671_object27779:head# 2024-03-20T09:04:49.590 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9d8f484:::benchmark_data_smithi122_119671_object32955:head# 2024-03-20T09:04:49.590 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9d9565e:::benchmark_data_smithi122_119671_object12071:head# 2024-03-20T09:04:49.590 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9da7912:::benchmark_data_smithi122_119671_object15752:head# 2024-03-20T09:04:49.590 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9da987b:::benchmark_data_smithi122_119671_object56417:head# 2024-03-20T09:04:49.591 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9dabe92:::benchmark_data_smithi122_119671_object16401:head# 2024-03-20T09:04:49.591 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9db27a6:::benchmark_data_smithi122_119671_object80771:head# 2024-03-20T09:04:49.591 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9db9400:::benchmark_data_smithi122_119671_object50586:head# 2024-03-20T09:04:49.592 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9dbc43e:::benchmark_data_smithi122_119671_object70200:head# 2024-03-20T09:04:49.592 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9dce5c9:::benchmark_data_smithi122_119671_object89445:head# 2024-03-20T09:04:49.593 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9de7c6b:::benchmark_data_smithi122_119671_object80884:head# 2024-03-20T09:04:49.593 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9def8d5:::benchmark_data_smithi122_119671_object95019:head# 2024-03-20T09:04:49.593 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9df687f:::benchmark_data_smithi122_119671_object81904:head# 2024-03-20T09:04:49.594 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9e1a707:::benchmark_data_smithi122_119671_object15451:head# 2024-03-20T09:04:49.594 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9e2d8ad:::benchmark_data_smithi122_119671_object32689:head# 2024-03-20T09:04:49.595 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9e44a2d:::benchmark_data_smithi122_119671_object30709:head# 2024-03-20T09:04:49.596 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9e4e41c:::benchmark_data_smithi122_119671_object84577:head# 2024-03-20T09:04:49.596 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9e52ab7:::benchmark_data_smithi122_119671_object99743:head# 2024-03-20T09:04:49.597 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9e53c9a:::benchmark_data_smithi122_119671_object8807:head# 2024-03-20T09:04:49.597 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9e55a69:::benchmark_data_smithi122_119671_object85568:head# 2024-03-20T09:04:49.597 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9e70e8e:::benchmark_data_smithi122_119671_object21099:head# 2024-03-20T09:04:49.598 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9e78199:::benchmark_data_smithi122_119671_object56126:head# 2024-03-20T09:04:49.598 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9e7f2b0:::benchmark_data_smithi122_119671_object77883:head# 2024-03-20T09:04:49.599 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9e89fbd:::benchmark_data_smithi122_119671_object96884:head# 2024-03-20T09:04:49.599 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9e8f5fb:::benchmark_data_smithi122_119671_object85676:head# 2024-03-20T09:04:49.600 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9e93694:::benchmark_data_smithi122_119671_object53907:head# 2024-03-20T09:04:49.600 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9eb766a:::benchmark_data_smithi122_119671_object53756:head# 2024-03-20T09:04:49.600 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9ec164b:::benchmark_data_smithi122_119671_object20034:head# 2024-03-20T09:04:49.601 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9ecdc34:::benchmark_data_smithi122_119671_object64491:head# 2024-03-20T09:04:49.601 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:04:49.602 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:04:49.602 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9ed20eb:::benchmark_data_smithi122_119671_object79456:head# 2024-03-20T09:04:49.603 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9eeeff1:::benchmark_data_smithi122_119671_object15562:head# 2024-03-20T09:04:49.603 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9ef12d2:::benchmark_data_smithi122_119671_object14524:head# 2024-03-20T09:04:49.603 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9ef698f:::benchmark_data_smithi122_119671_object32250:head# 2024-03-20T09:04:49.603 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9ef7f2b:::benchmark_data_smithi122_119671_object34723:head# 2024-03-20T09:04:49.603 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9ef928d:::benchmark_data_smithi122_119671_object1054:head# 2024-03-20T09:04:49.604 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9f1509e:::benchmark_data_smithi122_119671_object60927:head# 2024-03-20T09:04:49.604 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9f1b96b:::benchmark_data_smithi122_119671_object58553:head# 2024-03-20T09:04:49.605 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9f1e4b0:::benchmark_data_smithi122_119671_object24839:head# 2024-03-20T09:04:49.605 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9f232fc:::benchmark_data_smithi122_119671_object82859:head# 2024-03-20T09:04:49.605 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9f2ed3f:::benchmark_data_smithi122_119671_object35113:head# 2024-03-20T09:04:49.606 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9f3189c:::benchmark_data_smithi122_119671_object35853:head# 2024-03-20T09:04:49.606 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9f3e263:::benchmark_data_smithi122_119671_object12653:head# 2024-03-20T09:04:49.606 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9f4237a:::benchmark_data_smithi122_119671_object2531:head# 2024-03-20T09:04:49.607 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9f44b45:::benchmark_data_smithi122_119671_object28225:head# 2024-03-20T09:04:49.607 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9f63f02:::benchmark_data_smithi122_119671_object10165:head# 2024-03-20T09:04:49.608 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9f778ad:::benchmark_data_smithi122_119671_object4604:head# 2024-03-20T09:04:49.608 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9f7faa4:::benchmark_data_smithi122_119671_object25063:head# 2024-03-20T09:04:49.609 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9f905a7:::benchmark_data_smithi122_119671_object52723:head# 2024-03-20T09:04:49.609 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9f9460c:::benchmark_data_smithi122_119671_object91050:head# 2024-03-20T09:04:49.610 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9f9b366:::benchmark_data_smithi122_119671_object53348:head# 2024-03-20T09:04:49.610 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9f9d8c4:::benchmark_data_smithi122_119671_object66597:head# 2024-03-20T09:04:49.610 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9f9f2c5:::benchmark_data_smithi122_119671_object85850:head# 2024-03-20T09:04:49.611 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9fa566e:::benchmark_data_smithi122_119671_object87582:head# 2024-03-20T09:04:49.611 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9fa8329:::benchmark_data_smithi122_119671_object4258:head# 2024-03-20T09:04:49.611 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9fae878:::benchmark_data_smithi122_119671_object42516:head# 2024-03-20T09:04:49.612 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9fbef25:::benchmark_data_smithi122_119671_object16425:head# 2024-03-20T09:04:49.612 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9fd46f9:::benchmark_data_smithi122_119671_object88324:head# 2024-03-20T09:04:49.613 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9fd9067:::benchmark_data_smithi122_119671_object94931:head# 2024-03-20T09:04:49.613 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9fddc35:::benchmark_data_smithi122_119671_object23636:head# 2024-03-20T09:04:49.613 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9ff680a:::benchmark_data_smithi122_119671_object34193:head# 2024-03-20T09:04:49.614 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:b9ff7dca:::benchmark_data_smithi122_119671_object10016:head# 2024-03-20T09:04:49.614 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba018677:::benchmark_data_smithi122_119671_object27484:head# 2024-03-20T09:04:49.614 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba0205a0:::benchmark_data_smithi122_119671_object54452:head# 2024-03-20T09:04:49.615 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba020dc4:::benchmark_data_smithi122_119671_object9945:head# 2024-03-20T09:04:49.615 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba046e98:::benchmark_data_smithi122_119671_object54433:head# 2024-03-20T09:04:49.615 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba050028:::benchmark_data_smithi122_119671_object51849:head# 2024-03-20T09:04:49.616 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba0550b8:::benchmark_data_smithi122_119671_object56736:head# 2024-03-20T09:04:49.616 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba086918:::benchmark_data_smithi122_119671_object75858:head# 2024-03-20T09:04:49.617 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba08a11c:::benchmark_data_smithi122_119671_object55285:head# 2024-03-20T09:04:49.617 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba09194b:::benchmark_data_smithi122_119671_object38755:head# 2024-03-20T09:04:49.618 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba092fff:::benchmark_data_smithi122_119671_object62373:head# 2024-03-20T09:04:49.618 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba09d9bf:::benchmark_data_smithi122_119671_object35495:head# 2024-03-20T09:04:49.618 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba09f961:::benchmark_data_smithi122_119671_object91933:head# 2024-03-20T09:04:49.619 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba09f9bf:::benchmark_data_smithi122_119671_object32339:head# 2024-03-20T09:04:49.619 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba0a08be:::benchmark_data_smithi122_119671_object13708:head# 2024-03-20T09:04:49.620 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba0a0f5f:::benchmark_data_smithi122_119671_object92369:head# 2024-03-20T09:04:49.620 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba0ad2ce:::benchmark_data_smithi122_119671_object18872:head# 2024-03-20T09:04:49.621 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba0c7cbd:::benchmark_data_smithi122_119671_object1476:head# 2024-03-20T09:04:49.621 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba0d9cdb:::benchmark_data_smithi122_119671_object46057:head# 2024-03-20T09:04:49.621 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba0df7be:::benchmark_data_smithi122_119671_object89057:head# 2024-03-20T09:04:49.622 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba0e2b05:::benchmark_data_smithi122_119671_object39124:head# 2024-03-20T09:04:49.622 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba0e8c44:::benchmark_data_smithi122_119671_object39016:head# 2024-03-20T09:04:49.623 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba0ec75e:::benchmark_data_smithi122_119671_object14775:head# 2024-03-20T09:04:49.624 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba0ecf87:::benchmark_data_smithi122_119671_object73652:head# 2024-03-20T09:04:49.624 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba102e77:::benchmark_data_smithi122_119671_object23518:head# 2024-03-20T09:04:49.625 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba11dd01:::benchmark_data_smithi122_119671_object33389:head# 2024-03-20T09:04:49.625 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba126cd0:::benchmark_data_smithi122_119671_object12621:head# 2024-03-20T09:04:49.625 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba147044:::benchmark_data_smithi122_119671_object70404:head# 2024-03-20T09:04:49.626 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba14a968:::benchmark_data_smithi122_119671_object89768:head# 2024-03-20T09:04:49.626 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba167357:::benchmark_data_smithi122_119671_object47821:head# 2024-03-20T09:04:49.626 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba16ce89:::benchmark_data_smithi122_119671_object63779:head# 2024-03-20T09:04:49.627 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba16de77:::benchmark_data_smithi122_119671_object82191:head# 2024-03-20T09:04:49.627 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba1932e8:::benchmark_data_smithi122_119671_object70114:head# 2024-03-20T09:04:49.627 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba197b3e:::benchmark_data_smithi122_119671_object56038:head# 2024-03-20T09:04:49.628 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba1b43b4:::benchmark_data_smithi122_119671_object41774:head# 2024-03-20T09:04:49.628 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba1b8c46:::benchmark_data_smithi122_119671_object49476:head# 2024-03-20T09:04:49.629 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba1bc0c7:::benchmark_data_smithi122_119671_object67640:head# 2024-03-20T09:04:49.629 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba1c811e:::benchmark_data_smithi122_119671_object41902:head# 2024-03-20T09:04:49.629 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba1c85d9:::benchmark_data_smithi122_119671_object49147:head# 2024-03-20T09:04:49.630 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba1cb781:::benchmark_data_smithi122_119671_object1966:head# 2024-03-20T09:04:49.630 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba1cf693:::benchmark_data_smithi122_119671_object68416:head# 2024-03-20T09:04:49.630 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba1e1a34:::benchmark_data_smithi122_119671_object93352:head# 2024-03-20T09:04:49.631 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba1e3ed7:::benchmark_data_smithi122_119671_object97152:head# 2024-03-20T09:04:49.631 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba1f0126:::benchmark_data_smithi122_119671_object67258:head# 2024-03-20T09:04:49.632 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba1fc2c1:::benchmark_data_smithi122_119671_object51610:head# 2024-03-20T09:04:49.632 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba200d92:::benchmark_data_smithi122_119671_object96074:head# 2024-03-20T09:04:49.633 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba219d2a:::benchmark_data_smithi122_119671_object54231:head# 2024-03-20T09:04:49.633 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba21a1e1:::benchmark_data_smithi122_119671_object25127:head# 2024-03-20T09:04:49.633 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba21d200:::benchmark_data_smithi122_119671_object52436:head# 2024-03-20T09:04:49.634 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba22fc67:::benchmark_data_smithi122_119671_object39005:head# 2024-03-20T09:04:49.634 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba233054:::benchmark_data_smithi122_119671_object14004:head# 2024-03-20T09:04:49.635 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba241078:::benchmark_data_smithi122_119671_object95712:head# 2024-03-20T09:04:49.635 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba245921:::benchmark_data_smithi122_119671_object19754:head# 2024-03-20T09:04:49.635 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba24ca86:::benchmark_data_smithi122_119671_object28797:head# 2024-03-20T09:04:49.635 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba2565dc:::benchmark_data_smithi122_119671_object91563:head# 2024-03-20T09:04:49.636 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba2698b2:::benchmark_data_smithi122_119671_object40753:head# 2024-03-20T09:04:49.636 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba26d9d9:::benchmark_data_smithi122_119671_object14420:head# 2024-03-20T09:04:49.637 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba27a9c3:::benchmark_data_smithi122_119671_object21797:head# 2024-03-20T09:04:49.637 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba27c738:::benchmark_data_smithi122_119671_object59173:head# 2024-03-20T09:04:49.637 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba28ade0:::benchmark_data_smithi122_119671_object41111:head# 2024-03-20T09:04:49.638 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba298870:::benchmark_data_smithi122_119671_object16276:head# 2024-03-20T09:04:49.638 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba2b7be9:::benchmark_data_smithi122_119671_object40076:head# 2024-03-20T09:04:49.639 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba2c8585:::benchmark_data_smithi122_119671_object25755:head# 2024-03-20T09:04:49.639 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba2caf12:::benchmark_data_smithi122_119671_object10187:head# 2024-03-20T09:04:49.639 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba2d25b9:::benchmark_data_smithi122_119671_object7790:head# 2024-03-20T09:04:49.640 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba2d8227:::benchmark_data_smithi122_119671_object52662:head# 2024-03-20T09:04:49.640 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba2dbd22:::benchmark_data_smithi122_119671_object13212:head# 2024-03-20T09:04:49.641 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba2f5f28:::benchmark_data_smithi122_119671_object95315:head# 2024-03-20T09:04:49.641 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba2fb430:::benchmark_data_smithi122_119671_object40993:head# 2024-03-20T09:04:49.641 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba30cb67:::benchmark_data_smithi122_119671_object27184:head# 2024-03-20T09:04:49.642 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba30da18:::benchmark_data_smithi122_119671_object3114:head# 2024-03-20T09:04:49.642 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba30f695:::benchmark_data_smithi122_119671_object23389:head# 2024-03-20T09:04:49.642 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba314340:::benchmark_data_smithi122_119671_object83660:head# 2024-03-20T09:04:49.643 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba316459:::benchmark_data_smithi122_119671_object57812:head# 2024-03-20T09:04:49.643 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba337055:::benchmark_data_smithi122_119671_object21552:head# 2024-03-20T09:04:49.644 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba33da45:::benchmark_data_smithi122_119671_object16675:head# 2024-03-20T09:04:49.644 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba3499e8:::benchmark_data_smithi122_119671_object15265:head# 2024-03-20T09:04:49.644 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba34faf2:::benchmark_data_smithi122_119671_object99279:head# 2024-03-20T09:04:49.645 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba350514:::benchmark_data_smithi122_119671_object57737:head# 2024-03-20T09:04:49.645 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba35129d:::benchmark_data_smithi122_119671_object57354:head# 2024-03-20T09:04:49.646 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba355c89:::benchmark_data_smithi122_119671_object38119:head# 2024-03-20T09:04:49.646 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba361184:::benchmark_data_smithi122_119671_object58468:head# 2024-03-20T09:04:49.646 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba365b7c:::benchmark_data_smithi122_119671_object17060:head# 2024-03-20T09:04:49.647 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba365d70:::benchmark_data_smithi122_119671_object15846:head# 2024-03-20T09:04:49.647 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba389094:::benchmark_data_smithi122_119671_object63567:head# 2024-03-20T09:04:49.648 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba38939d:::benchmark_data_smithi122_119671_object42416:head# 2024-03-20T09:04:49.648 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba3901fd:::benchmark_data_smithi122_119671_object41789:head# 2024-03-20T09:04:49.649 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba3929c2:::benchmark_data_smithi122_119671_object12498:head# 2024-03-20T09:04:49.649 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba397022:::benchmark_data_smithi122_119671_object44438:head# 2024-03-20T09:04:49.649 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba39de65:::benchmark_data_smithi122_119671_object60597:head# 2024-03-20T09:04:49.650 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba3b0b12:::benchmark_data_smithi122_119671_object38618:head# 2024-03-20T09:04:49.650 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba3b3970:::benchmark_data_smithi122_119671_object40940:head# 2024-03-20T09:04:49.650 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba3df839:::benchmark_data_smithi122_119671_object70642:head# 2024-03-20T09:04:49.651 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba3ea4b2:::benchmark_data_smithi122_119671_object91254:head# 2024-03-20T09:04:49.651 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba3f6767:::benchmark_data_smithi122_119671_object62470:head# 2024-03-20T09:04:49.651 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba3f7008:::benchmark_data_smithi122_119671_object74349:head# 2024-03-20T09:04:49.652 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba3faaf5:::benchmark_data_smithi122_119671_object21527:head# 2024-03-20T09:04:49.652 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba40ac83:::benchmark_data_smithi122_119671_object21451:head# 2024-03-20T09:04:49.652 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba40f0b9:::benchmark_data_smithi122_119671_object26341:head# 2024-03-20T09:04:49.653 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba41131f:::benchmark_data_smithi122_119671_object82878:head# 2024-03-20T09:04:49.653 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba434ca6:::benchmark_data_smithi122_119671_object75496:head# 2024-03-20T09:04:49.654 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba4516bd:::benchmark_data_smithi122_119671_object68186:head# 2024-03-20T09:04:49.654 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba4568c2:::benchmark_data_smithi122_119671_object88146:head# 2024-03-20T09:04:49.655 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba456ca6:::benchmark_data_smithi122_119671_object11303:head# 2024-03-20T09:04:49.655 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba460d9e:::benchmark_data_smithi122_119671_object56135:head# 2024-03-20T09:04:49.656 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba46f83f:::benchmark_data_smithi122_119671_object44730:head# 2024-03-20T09:04:49.656 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba4915cd:::benchmark_data_smithi122_119671_object29014:head# 2024-03-20T09:04:49.656 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba49499f:::benchmark_data_smithi122_119671_object29845:head# 2024-03-20T09:04:49.657 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba497077:::benchmark_data_smithi122_119671_object21208:head# 2024-03-20T09:04:49.657 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba49a6e0:::benchmark_data_smithi122_119671_object44201:head# 2024-03-20T09:04:49.658 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba4c875b:::benchmark_data_smithi122_119671_object47332:head# 2024-03-20T09:04:49.660 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba4cb67d:::benchmark_data_smithi122_119671_object43300:head# 2024-03-20T09:04:49.661 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba4cfa14:::benchmark_data_smithi122_119671_object23118:head# 2024-03-20T09:04:49.661 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba4d1877:::benchmark_data_smithi122_119671_object86922:head# 2024-03-20T09:04:49.662 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba4d50e8:::benchmark_data_smithi122_119671_object25076:head# 2024-03-20T09:04:49.662 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba4de147:::benchmark_data_smithi122_119671_object92178:head# 2024-03-20T09:04:49.666 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba4e2fc0:::benchmark_data_smithi122_119671_object69960:head# 2024-03-20T09:04:49.666 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba4e371e:::benchmark_data_smithi122_119671_object39852:head# 2024-03-20T09:04:49.666 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba4e410e:::benchmark_data_smithi122_119671_object7016:head# 2024-03-20T09:04:49.667 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba4ee07d:::benchmark_data_smithi122_119671_object23214:head# 2024-03-20T09:04:49.667 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba4ef9f0:::benchmark_data_smithi122_119671_object62840:head# 2024-03-20T09:04:49.668 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba5012c6:::benchmark_data_smithi122_119671_object46410:head# 2024-03-20T09:04:49.668 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba513cb1:::benchmark_data_smithi122_119671_object18561:head# 2024-03-20T09:04:49.669 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba51c43d:::benchmark_data_smithi122_119671_object43417:head# 2024-03-20T09:04:49.669 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba51fde2:::benchmark_data_smithi122_119671_object32827:head# 2024-03-20T09:04:49.669 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba521103:::benchmark_data_smithi122_119671_object87118:head# 2024-03-20T09:04:49.670 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba52974e:::benchmark_data_smithi122_119671_object27250:head# 2024-03-20T09:04:49.670 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba52be56:::benchmark_data_smithi122_119671_object3616:head# 2024-03-20T09:04:49.671 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba53079e:::benchmark_data_smithi122_119671_object9903:head# 2024-03-20T09:04:49.671 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba540bfd:::benchmark_data_smithi122_119671_object59240:head# 2024-03-20T09:04:49.671 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba543392:::benchmark_data_smithi122_119671_object82834:head# 2024-03-20T09:04:49.672 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba548f4e:::benchmark_data_smithi122_119671_object19217:head# 2024-03-20T09:04:49.673 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba5582d2:::benchmark_data_smithi122_119671_object19142:head# 2024-03-20T09:04:49.674 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba58c58c:::benchmark_data_smithi122_119671_object78751:head# 2024-03-20T09:04:49.674 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba58cfd6:::benchmark_data_smithi122_119671_object34342:head# 2024-03-20T09:04:49.675 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba5918d0:::benchmark_data_smithi122_119671_object85242:head# 2024-03-20T09:04:49.675 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba59f1c3:::benchmark_data_smithi122_119671_object82085:head# 2024-03-20T09:04:49.676 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba5a75d2:::benchmark_data_smithi122_119671_object2189:head# 2024-03-20T09:04:49.676 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba5b8b10:::benchmark_data_smithi122_119671_object73786:head# 2024-03-20T09:04:49.677 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba5bf534:::benchmark_data_smithi122_119671_object58396:head# 2024-03-20T09:04:49.677 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba5c84ec:::benchmark_data_smithi122_119671_object93306:head# 2024-03-20T09:04:49.678 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba5d1359:::benchmark_data_smithi122_119671_object78479:head# 2024-03-20T09:04:49.678 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba5dabb4:::benchmark_data_smithi122_119671_object27450:head# 2024-03-20T09:04:49.679 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba5dc533:::benchmark_data_smithi122_119671_object76039:head# 2024-03-20T09:04:49.679 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba5dd25e:::benchmark_data_smithi122_119671_object82175:head# 2024-03-20T09:04:49.679 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba5e56c2:::benchmark_data_smithi122_119671_object25503:head# 2024-03-20T09:04:49.680 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba5f4971:::benchmark_data_smithi122_119671_object28806:head# 2024-03-20T09:04:49.680 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba5f50c2:::benchmark_data_smithi122_119671_object1448:head# 2024-03-20T09:04:49.683 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba5fea89:::benchmark_data_smithi122_119671_object76965:head# 2024-03-20T09:04:49.683 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba608a8e:::benchmark_data_smithi122_119671_object55516:head# 2024-03-20T09:04:49.684 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba609695:::benchmark_data_smithi122_119671_object32417:head# 2024-03-20T09:04:49.684 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba612258:::benchmark_data_smithi122_119671_object62959:head# 2024-03-20T09:04:49.684 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba6227c5:::benchmark_data_smithi122_119671_object51669:head# 2024-03-20T09:04:49.685 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba625fc8:::benchmark_data_smithi122_119671_object68204:head# 2024-03-20T09:04:49.685 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba6298e9:::benchmark_data_smithi122_119671_object38230:head# 2024-03-20T09:04:49.686 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba63ca6a:::benchmark_data_smithi122_119671_object36056:head# 2024-03-20T09:04:49.688 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba64154d:::benchmark_data_smithi122_119671_object23885:head# 2024-03-20T09:04:49.688 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba642f91:::benchmark_data_smithi122_119671_object95033:head# 2024-03-20T09:04:49.691 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba65b8b1:::benchmark_data_smithi122_119671_object56563:head# 2024-03-20T09:04:49.694 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba67579f:::benchmark_data_smithi122_119671_object52332:head# 2024-03-20T09:04:49.694 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba67d1de:::benchmark_data_smithi122_119671_object81324:head# 2024-03-20T09:04:49.695 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba699673:::benchmark_data_smithi122_119671_object74286:head# 2024-03-20T09:04:49.695 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba6aed3b:::benchmark_data_smithi122_119671_object53571:head# 2024-03-20T09:04:49.695 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba6b3715:::benchmark_data_smithi122_119671_object70779:head# 2024-03-20T09:04:49.696 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba6c7d4b:::benchmark_data_smithi122_119671_object90369:head# 2024-03-20T09:04:49.698 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba6ccca1:::benchmark_data_smithi122_119671_object19735:head# 2024-03-20T09:04:49.700 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba6d1494:::benchmark_data_smithi122_119671_object13325:head# 2024-03-20T09:04:49.701 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba6de6e7:::benchmark_data_smithi122_119671_object45534:head# 2024-03-20T09:04:49.701 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba716418:::benchmark_data_smithi122_119671_object72557:head# 2024-03-20T09:04:49.701 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba722304:::benchmark_data_smithi122_119671_object89612:head# 2024-03-20T09:04:49.702 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba72481a:::benchmark_data_smithi122_119671_object69160:head# 2024-03-20T09:04:49.702 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba73784b:::benchmark_data_smithi122_119671_object9131:head# 2024-03-20T09:04:49.706 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba7436e1:::benchmark_data_smithi122_119671_object95479:head# 2024-03-20T09:04:49.706 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba7557bf:::benchmark_data_smithi122_119671_object80137:head# 2024-03-20T09:04:49.706 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba75ea28:::benchmark_data_smithi122_119671_object40531:head# 2024-03-20T09:04:49.706 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba76a074:::benchmark_data_smithi122_119671_object25896:head# 2024-03-20T09:04:49.706 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba7c8512:::benchmark_data_smithi122_119671_object71751:head# 2024-03-20T09:04:49.706 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba7d3efb:::benchmark_data_smithi122_119671_object1560:head# 2024-03-20T09:04:49.706 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba7de744:::benchmark_data_smithi122_119671_object33413:head# 2024-03-20T09:04:49.707 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba7e0801:::benchmark_data_smithi122_119671_object82300:head# 2024-03-20T09:04:49.707 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba7e2e6e:::benchmark_data_smithi122_119671_object15278:head# 2024-03-20T09:04:49.707 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba7e4cda:::benchmark_data_smithi122_119671_object26054:head# 2024-03-20T09:04:49.708 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba7ecfbc:::benchmark_data_smithi122_119671_object74200:head# 2024-03-20T09:04:49.708 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba7f4751:::benchmark_data_smithi122_119671_object5430:head# 2024-03-20T09:04:49.708 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba824838:::benchmark_data_smithi122_119671_object80811:head# 2024-03-20T09:04:49.709 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba82bdbf:::benchmark_data_smithi122_119671_object46823:head# 2024-03-20T09:04:49.711 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba8605c2:::benchmark_data_smithi122_119671_object62825:head# 2024-03-20T09:04:49.711 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba87a4a0:::benchmark_data_smithi122_119671_object80813:head# 2024-03-20T09:04:49.711 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba87ec6f:::benchmark_data_smithi122_119671_object31600:head# 2024-03-20T09:04:49.711 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba880ae8:::benchmark_data_smithi122_119671_object31107:head# 2024-03-20T09:04:49.712 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba880b8f:::benchmark_data_smithi122_119671_object30545:head# 2024-03-20T09:04:49.712 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba88c222:::benchmark_data_smithi122_119671_object80313:head# 2024-03-20T09:04:49.712 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba8a2853:::benchmark_data_smithi122_119671_object6239:head# 2024-03-20T09:04:49.713 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba8addfa:::benchmark_data_smithi122_119671_object97530:head# 2024-03-20T09:04:49.713 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba8bbe78:::benchmark_data_smithi122_119671_object91617:head# 2024-03-20T09:04:49.714 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba8be9ac:::benchmark_data_smithi122_119671_object41664:head# 2024-03-20T09:04:49.714 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba8c87b3:::benchmark_data_smithi122_119671_object84425:head# 2024-03-20T09:04:49.715 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba8c9900:::benchmark_data_smithi122_119671_object7450:head# 2024-03-20T09:04:49.715 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba8d766f:::benchmark_data_smithi122_119671_object69285:head# 2024-03-20T09:04:49.716 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba8ea004:::benchmark_data_smithi122_119671_object61643:head# 2024-03-20T09:04:49.716 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba8eb75d:::benchmark_data_smithi122_119671_object81764:head# 2024-03-20T09:04:49.716 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba8f2c76:::benchmark_data_smithi122_119671_object25338:head# 2024-03-20T09:04:49.717 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba8fa5b8:::benchmark_data_smithi122_119671_object11442:head# 2024-03-20T09:04:49.717 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba94b513:::benchmark_data_smithi122_119671_object77926:head# 2024-03-20T09:04:49.718 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba94d05d:::benchmark_data_smithi122_119671_object38771:head# 2024-03-20T09:04:49.721 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba94da50:::benchmark_data_smithi122_119671_object20782:head# 2024-03-20T09:04:49.721 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba94eb85:::benchmark_data_smithi122_119671_object58374:head# 2024-03-20T09:04:49.721 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba968efd:::benchmark_data_smithi122_119671_object73007:head# 2024-03-20T09:04:49.721 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba96a90b:::benchmark_data_smithi122_119671_object17387:head# 2024-03-20T09:04:49.721 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba96db7d:::benchmark_data_smithi122_119671_object73516:head# 2024-03-20T09:04:49.721 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba971839:::benchmark_data_smithi122_119671_object56982:head# 2024-03-20T09:04:49.722 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba976ee5:::benchmark_data_smithi122_119671_object52759:head# 2024-03-20T09:04:49.722 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba97e222:::benchmark_data_smithi122_119671_object91002:head# 2024-03-20T09:04:49.722 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba9867ae:::benchmark_data_smithi122_119671_object26677:head# 2024-03-20T09:04:49.722 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba992878:::benchmark_data_smithi122_119671_object18962:head# 2024-03-20T09:04:49.723 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba99e067:::benchmark_data_smithi122_119671_object79483:head# 2024-03-20T09:04:49.723 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba9a06d7:::benchmark_data_smithi122_119671_object86722:head# 2024-03-20T09:04:49.727 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba9a74c3:::benchmark_data_smithi122_119671_object23278:head# 2024-03-20T09:04:49.727 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba9aaf83:::benchmark_data_smithi122_119671_object70728:head# 2024-03-20T09:04:49.727 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba9baa2e:::benchmark_data_smithi122_119671_object60582:head# 2024-03-20T09:04:49.728 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba9be5a7:::benchmark_data_smithi122_119671_object55001:head# 2024-03-20T09:04:49.728 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba9c43da:::benchmark_data_smithi122_119671_object11643:head# 2024-03-20T09:04:49.729 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba9c5ef2:::benchmark_data_smithi122_119671_object69697:head# 2024-03-20T09:04:49.729 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba9dd4c8:::benchmark_data_smithi122_119671_object75717:head# 2024-03-20T09:04:49.729 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba9e52af:::benchmark_data_smithi122_119671_object95639:head# 2024-03-20T09:04:49.730 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba9eb24b:::benchmark_data_smithi122_119671_object69672:head# 2024-03-20T09:04:49.730 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:ba9ff202:::benchmark_data_smithi122_119671_object98503:head# 2024-03-20T09:04:49.732 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baa01b3e:::benchmark_data_smithi122_119671_object50894:head# 2024-03-20T09:04:49.732 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baa091d6:::benchmark_data_smithi122_119671_object31983:head# 2024-03-20T09:04:49.732 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baa1218c:::benchmark_data_smithi122_119671_object50583:head# 2024-03-20T09:04:49.733 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baa137e9:::benchmark_data_smithi122_119671_object57851:head# 2024-03-20T09:04:49.733 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baa1e6a9:::benchmark_data_smithi122_119671_object3816:head# 2024-03-20T09:04:49.733 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baa2c4ce:::benchmark_data_smithi122_119671_object34695:head# 2024-03-20T09:04:49.733 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baa3a88f:::benchmark_data_smithi122_119671_object6764:head# 2024-03-20T09:04:49.733 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baa4053e:::benchmark_data_smithi122_119671_object16846:head# 2024-03-20T09:04:49.734 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baa42b27:::benchmark_data_smithi122_119671_object8214:head# 2024-03-20T09:04:49.734 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baa4915a:::benchmark_data_smithi122_119671_object10817:head# 2024-03-20T09:04:49.734 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baa57c95:::benchmark_data_smithi122_119671_object20290:head# 2024-03-20T09:04:49.735 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baa5cf84:::benchmark_data_smithi122_119671_object62414:head# 2024-03-20T09:04:49.736 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baa5dec9:::benchmark_data_smithi122_119671_object31320:head# 2024-03-20T09:04:49.736 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baa65cac:::benchmark_data_smithi122_119671_object21458:head# 2024-03-20T09:04:49.737 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baa670a2:::benchmark_data_smithi122_119671_object86220:head# 2024-03-20T09:04:49.738 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baa6d94b:::benchmark_data_smithi122_119671_object59353:head# 2024-03-20T09:04:49.738 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baa821d7:::benchmark_data_smithi122_119671_object48822:head# 2024-03-20T09:04:49.738 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baa8a604:::benchmark_data_smithi122_119671_object38036:head# 2024-03-20T09:04:49.738 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baa98fb3:::benchmark_data_smithi122_119671_object9490:head# 2024-03-20T09:04:49.739 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baa99e0d:::benchmark_data_smithi122_119671_object65977:head# 2024-03-20T09:04:49.739 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baabc7e0:::benchmark_data_smithi122_119671_object36800:head# 2024-03-20T09:04:49.740 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baabf4ce:::benchmark_data_smithi122_119671_object19922:head# 2024-03-20T09:04:49.740 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baacd221:::benchmark_data_smithi122_119671_object23399:head# 2024-03-20T09:04:49.740 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baacefc3:::benchmark_data_smithi122_119671_object35899:head# 2024-03-20T09:04:49.741 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baae05d5:::benchmark_data_smithi122_119671_object62246:head# 2024-03-20T09:04:49.741 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baae3d92:::benchmark_data_smithi122_119671_object4395:head# 2024-03-20T09:04:49.742 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baaf3089:::benchmark_data_smithi122_119671_object97665:head# 2024-03-20T09:04:49.742 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baaf3e4b:::benchmark_data_smithi122_119671_object52895:head# 2024-03-20T09:04:49.742 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baaf7277:::benchmark_data_smithi122_119671_object1433:head# 2024-03-20T09:04:49.743 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bab06ea6:::benchmark_data_smithi122_119671_object12195:head# 2024-03-20T09:04:49.743 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bab0b7c0:::benchmark_data_smithi122_119671_object4359:head# 2024-03-20T09:04:49.743 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bab17554:::benchmark_data_smithi122_119671_object89977:head# 2024-03-20T09:04:49.744 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bab27437:::benchmark_data_smithi122_119671_object38668:head# 2024-03-20T09:04:49.744 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bab3b3ee:::benchmark_data_smithi122_119671_object42599:head# 2024-03-20T09:04:49.745 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bab42f6a:::benchmark_data_smithi122_119671_object2917:head# 2024-03-20T09:04:49.745 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bab4f98c:::benchmark_data_smithi122_119671_object98288:head# 2024-03-20T09:04:49.746 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bab5ae1a:::benchmark_data_smithi122_119671_object37953:head# 2024-03-20T09:04:49.746 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bab6ab5e:::benchmark_data_smithi122_119671_object97174:head# 2024-03-20T09:04:49.752 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bab7f7f5:::benchmark_data_smithi122_119671_object64844:head# 2024-03-20T09:04:49.752 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bab84668:::benchmark_data_smithi122_119671_object21677:head# 2024-03-20T09:04:49.752 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bab9b168:::benchmark_data_smithi122_119671_object25379:head# 2024-03-20T09:04:49.752 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baba536b:::benchmark_data_smithi122_119671_object17017:head# 2024-03-20T09:04:49.752 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baba71be:::benchmark_data_smithi122_119671_object40107:head# 2024-03-20T09:04:49.752 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baba9bba:::benchmark_data_smithi122_119671_object25056:head# 2024-03-20T09:04:49.752 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:babaf4ee:::benchmark_data_smithi122_119671_object21610:head# 2024-03-20T09:04:49.752 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:babb57d7:::benchmark_data_smithi122_119671_object34144:head# 2024-03-20T09:04:49.753 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:babb7666:::benchmark_data_smithi122_119671_object42852:head# 2024-03-20T09:04:49.753 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:babc1af4:::benchmark_data_smithi122_119671_object62364:head# 2024-03-20T09:04:49.753 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:babc51c7:::benchmark_data_smithi122_119671_object33816:head# 2024-03-20T09:04:49.753 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:babc6ffe:::benchmark_data_smithi122_119671_object96486:head# 2024-03-20T09:04:49.753 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:babd449b:::benchmark_data_smithi122_119671_object9668:head# 2024-03-20T09:04:49.753 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:babd9fc3:::benchmark_data_smithi122_119671_object57627:head# 2024-03-20T09:04:49.753 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:babe7350:::benchmark_data_smithi122_119671_object41581:head# 2024-03-20T09:04:49.754 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:babfbf48:::benchmark_data_smithi122_119671_object59785:head# 2024-03-20T09:04:49.754 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bac01339:::benchmark_data_smithi122_119671_object91590:head# 2024-03-20T09:04:49.754 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bac020d3:::benchmark_data_smithi122_119671_object15836:head# 2024-03-20T09:04:49.754 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bac03571:::benchmark_data_smithi122_119671_object44749:head# 2024-03-20T09:04:49.754 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bac04ca3:::benchmark_data_smithi122_119671_object19337:head# 2024-03-20T09:04:49.755 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bac05012:::benchmark_data_smithi122_119671_object52734:head# 2024-03-20T09:04:49.755 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bac06ac7:::benchmark_data_smithi122_119671_object91601:head# 2024-03-20T09:04:49.755 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bac1e0cf:::benchmark_data_smithi122_119671_object53768:head# 2024-03-20T09:04:49.756 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bac20591:::benchmark_data_smithi122_119671_object75567:head# 2024-03-20T09:04:49.756 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bac20a7b:::benchmark_data_smithi122_119671_object36997:head# 2024-03-20T09:04:49.756 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bac2ae68:::benchmark_data_smithi122_119671_object70990:head# 2024-03-20T09:04:49.757 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bac40f39:::benchmark_data_smithi122_119671_object1671:head# 2024-03-20T09:04:49.757 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bac50b37:::benchmark_data_smithi122_119671_object4856:head# 2024-03-20T09:04:49.758 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bac563a6:::benchmark_data_smithi122_119671_object25588:head# 2024-03-20T09:04:49.758 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bac56526:::benchmark_data_smithi122_119671_object27834:head# 2024-03-20T09:04:49.759 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bac5a80d:::benchmark_data_smithi122_119671_object33096:head# 2024-03-20T09:04:49.759 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bac6fb38:::benchmark_data_smithi122_119671_object37484:head# 2024-03-20T09:04:49.759 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bac76649:::benchmark_data_smithi122_119671_object22779:head# 2024-03-20T09:04:49.760 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bac85f2d:::benchmark_data_smithi122_119671_object95395:head# 2024-03-20T09:04:49.760 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bac9ccd2:::benchmark_data_smithi122_119671_object24940:head# 2024-03-20T09:04:49.761 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baca15ad:::benchmark_data_smithi122_119671_object37870:head# 2024-03-20T09:04:49.761 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baccb4af:::benchmark_data_smithi122_119671_object97559:head# 2024-03-20T09:04:49.761 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bacd0709:::benchmark_data_smithi122_119671_object15708:head# 2024-03-20T09:04:49.762 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bace42aa:::benchmark_data_smithi122_119671_object10467:head# 2024-03-20T09:04:49.762 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bace5f9e:::benchmark_data_smithi122_119671_object77594:head# 2024-03-20T09:04:49.762 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baceda40:::benchmark_data_smithi122_119671_object28989:head# 2024-03-20T09:04:49.763 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bacf71d8:::benchmark_data_smithi122_119671_object19430:head# 2024-03-20T09:04:49.763 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bacf9762:::benchmark_data_smithi122_119671_object79575:head# 2024-03-20T09:04:49.764 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bacfceae:::benchmark_data_smithi122_119671_object46188:head# 2024-03-20T09:04:49.764 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bad067ea:::benchmark_data_smithi122_119671_object17890:head# 2024-03-20T09:04:49.764 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bad0ff23:::benchmark_data_smithi122_119671_object86440:head# 2024-03-20T09:04:49.765 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bad399ff:::benchmark_data_smithi122_119671_object15205:head# 2024-03-20T09:04:49.765 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bad3b076:::benchmark_data_smithi122_119671_object65573:head# 2024-03-20T09:04:49.766 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bad437f7:::benchmark_data_smithi122_119671_object58493:head# 2024-03-20T09:04:49.766 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bad54f90:::benchmark_data_smithi122_119671_object4184:head# 2024-03-20T09:04:49.766 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bad5b5ad:::benchmark_data_smithi122_119671_object55709:head# 2024-03-20T09:04:49.767 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bad5f198:::benchmark_data_smithi122_119671_object8592:head# 2024-03-20T09:04:49.767 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bad7b57d:::benchmark_data_smithi122_119671_object49158:head# 2024-03-20T09:04:49.768 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bad7de60:::benchmark_data_smithi122_119671_object2317:head# 2024-03-20T09:04:49.768 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bad8bbeb:::benchmark_data_smithi122_119671_object22601:head# 2024-03-20T09:04:49.768 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bad8f966:::benchmark_data_smithi122_119671_object60136:head# 2024-03-20T09:04:49.769 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bada0cf9:::benchmark_data_smithi122_119671_object91207:head# 2024-03-20T09:04:49.769 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bada8ca6:::benchmark_data_smithi122_119671_object38207:head# 2024-03-20T09:04:49.770 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:badab945:::benchmark_data_smithi122_119671_object58125:head# 2024-03-20T09:04:49.771 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:badb1f91:::benchmark_data_smithi122_119671_object77677:head# 2024-03-20T09:04:49.771 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:badd5728:::benchmark_data_smithi122_119671_object60411:head# 2024-03-20T09:04:49.772 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bade4ec3:::benchmark_data_smithi122_119671_object18176:head# 2024-03-20T09:04:49.772 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:badf0281:::benchmark_data_smithi122_119671_object63018:head# 2024-03-20T09:04:49.772 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:badfdc3a:::benchmark_data_smithi122_119671_object27763:head# 2024-03-20T09:04:49.773 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bae20793:::benchmark_data_smithi122_119671_object19330:head# 2024-03-20T09:04:49.773 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bae3309a:::benchmark_data_smithi122_119671_object13014:head# 2024-03-20T09:04:49.774 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bae3826b:::benchmark_data_smithi122_119671_object27379:head# 2024-03-20T09:04:49.774 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bae61227:::benchmark_data_smithi122_119671_object88684:head# 2024-03-20T09:04:49.774 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bae63f8c:::benchmark_data_smithi122_119671_object7774:head# 2024-03-20T09:04:49.775 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bae77786:::benchmark_data_smithi122_119671_object45236:head# 2024-03-20T09:04:49.775 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bae81113:::benchmark_data_smithi122_119671_object62797:head# 2024-03-20T09:04:49.776 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bae8bb71:::benchmark_data_smithi122_119671_object29174:head# 2024-03-20T09:04:49.776 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bae9435c:::benchmark_data_smithi122_119671_object25356:head# 2024-03-20T09:04:49.777 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baebade3:::benchmark_data_smithi122_119671_object64434:head# 2024-03-20T09:04:49.777 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baebbba3:::benchmark_data_smithi122_119671_object29523:head# 2024-03-20T09:04:49.778 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baec299b:::benchmark_data_smithi122_119671_object93242:head# 2024-03-20T09:04:49.778 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baed7c2b:::benchmark_data_smithi122_119671_object55072:head# 2024-03-20T09:04:49.779 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baedb226:::benchmark_data_smithi122_119671_object58228:head# 2024-03-20T09:04:49.779 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baedb438:::benchmark_data_smithi122_119671_object15292:head# 2024-03-20T09:04:49.779 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baede2c5:::benchmark_data_smithi122_119671_object44989:head# 2024-03-20T09:04:49.780 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baede6d8:::benchmark_data_smithi122_119671_object57236:head# 2024-03-20T09:04:49.780 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baef2e92:::benchmark_data_smithi122_119671_object46199:head# 2024-03-20T09:04:49.781 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baef94d0:::benchmark_data_smithi122_119671_object87350:head# 2024-03-20T09:04:49.781 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baefcfff:::benchmark_data_smithi122_119671_object57224:head# 2024-03-20T09:04:49.781 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baf018e0:::benchmark_data_smithi122_119671_object61997:head# 2024-03-20T09:04:49.782 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baf09bd1:::benchmark_data_smithi122_119671_object28046:head# 2024-03-20T09:04:49.782 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baf1100f:::benchmark_data_smithi122_119671_object68329:head# 2024-03-20T09:04:49.783 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baf124aa:::benchmark_data_smithi122_119671_object41882:head# 2024-03-20T09:04:49.784 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baf1ff53:::benchmark_data_smithi122_119671_object40353:head# 2024-03-20T09:04:49.784 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baf29e77:::benchmark_data_smithi122_119671_object79265:head# 2024-03-20T09:04:49.784 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baf2aa73:::benchmark_data_smithi122_119671_object62656:head# 2024-03-20T09:04:49.785 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baf3d633:::benchmark_data_smithi122_119671_object35647:head# 2024-03-20T09:04:49.785 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baf4146a:::benchmark_data_smithi122_119671_object35731:head# 2024-03-20T09:04:49.786 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baf43e86:::benchmark_data_smithi122_119671_object87667:head# 2024-03-20T09:04:49.786 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baf480e3:::benchmark_data_smithi122_119671_object68483:head# 2024-03-20T09:04:49.787 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baf4c773:::benchmark_data_smithi122_119671_object32586:head# 2024-03-20T09:04:49.787 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baf54b13:::benchmark_data_smithi122_119671_object24644:head# 2024-03-20T09:04:49.788 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baf75d4b:::benchmark_data_smithi122_119671_object45435:head# 2024-03-20T09:04:49.788 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baf8432f:::benchmark_data_smithi122_119671_object24993:head# 2024-03-20T09:04:49.789 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baf870ff:::benchmark_data_smithi122_119671_object22279:head# 2024-03-20T09:04:49.789 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baf89457:::benchmark_data_smithi122_119671_object66851:head# 2024-03-20T09:04:49.789 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:baf9d592:::benchmark_data_smithi122_119671_object48939:head# 2024-03-20T09:04:49.790 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bafab91a:::benchmark_data_smithi122_119671_object93281:head# 2024-03-20T09:04:49.790 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bafaeaa6:::benchmark_data_smithi122_119671_object22483:head# 2024-03-20T09:04:49.791 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bafb6838:::benchmark_data_smithi122_119671_object60564:head# 2024-03-20T09:04:49.791 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bafb9aff:::benchmark_data_smithi122_119671_object82033:head# 2024-03-20T09:04:49.792 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bafcb123:::benchmark_data_smithi122_119671_object1916:head# 2024-03-20T09:04:49.792 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bafe4192:::benchmark_data_smithi122_119671_object78683:head# 2024-03-20T09:04:49.793 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb0229d6:::benchmark_data_smithi122_119671_object24701:head# 2024-03-20T09:04:49.793 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb02c3ff:::benchmark_data_smithi122_119671_object35280:head# 2024-03-20T09:04:49.793 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb02d9d9:::benchmark_data_smithi122_119671_object19634:head# 2024-03-20T09:04:49.794 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb058d12:::benchmark_data_smithi122_119671_object94761:head# 2024-03-20T09:04:49.794 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb05d0ae:::benchmark_data_smithi122_119671_object24368:head# 2024-03-20T09:04:49.795 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb06116e:::benchmark_data_smithi122_119671_object37632:head# 2024-03-20T09:04:49.795 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb0645d1:::benchmark_data_smithi122_119671_object44623:head# 2024-03-20T09:04:49.795 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb0669a7:::benchmark_data_smithi122_119671_object3328:head# 2024-03-20T09:04:49.795 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb0741b3:::benchmark_data_smithi122_119671_object14696:head# 2024-03-20T09:04:49.796 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb078d5a:::benchmark_data_smithi122_119671_object95062:head# 2024-03-20T09:04:49.796 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb07998d:::benchmark_data_smithi122_119671_object84042:head# 2024-03-20T09:04:49.797 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb08c630:::benchmark_data_smithi122_119671_object61411:head# 2024-03-20T09:04:49.797 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb09d227:::benchmark_data_smithi122_119671_object73020:head# 2024-03-20T09:04:49.797 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb0b219c:::benchmark_data_smithi122_119671_object93907:head# 2024-03-20T09:04:49.798 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb0bb3ab:::benchmark_data_smithi122_119671_object56230:head# 2024-03-20T09:04:49.798 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb0bbf68:::benchmark_data_smithi122_119671_object97397:head# 2024-03-20T09:04:49.798 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb0c3e0c:::benchmark_data_smithi122_119671_object10225:head# 2024-03-20T09:04:49.799 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb0c71fc:::benchmark_data_smithi122_119671_object38384:head# 2024-03-20T09:04:49.799 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb0c9ff9:::benchmark_data_smithi122_119671_object64833:head# 2024-03-20T09:04:49.799 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb0d1470:::benchmark_data_smithi122_119671_object17714:head# 2024-03-20T09:04:49.800 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb0d68bb:::benchmark_data_smithi122_119671_object33420:head# 2024-03-20T09:04:49.800 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb101a30:::benchmark_data_smithi122_119671_object35454:head# 2024-03-20T09:04:49.801 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb102b2f:::benchmark_data_smithi122_119671_object61641:head# 2024-03-20T09:04:49.801 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb10dfe7:::benchmark_data_smithi122_119671_object79163:head# 2024-03-20T09:04:49.802 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb112f88:::benchmark_data_smithi122_119671_object11607:head# 2024-03-20T09:04:49.802 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb11f9a6:::benchmark_data_smithi122_119671_object34734:head# 2024-03-20T09:04:49.802 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb12ce4e:::benchmark_data_smithi122_119671_object7020:head# 2024-03-20T09:04:49.803 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb14252c:::benchmark_data_smithi122_119671_object42118:head# 2024-03-20T09:04:49.803 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb14c208:::benchmark_data_smithi122_119671_object51714:head# 2024-03-20T09:04:49.804 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb150d21:::benchmark_data_smithi122_119671_object96462:head# 2024-03-20T09:04:49.804 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb151743:::benchmark_data_smithi122_119671_object61602:head# 2024-03-20T09:04:49.805 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb156b54:::benchmark_data_smithi122_119671_object61004:head# 2024-03-20T09:04:49.805 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb163f6d:::benchmark_data_smithi122_119671_object24974:head# 2024-03-20T09:04:49.805 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb17cad0:::benchmark_data_smithi122_119671_object57570:head# 2024-03-20T09:04:49.806 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb183d02:::benchmark_data_smithi122_119671_object61569:head# 2024-03-20T09:04:49.806 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb18e300:::benchmark_data_smithi122_119671_object73592:head# 2024-03-20T09:04:49.807 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb18eb04:::benchmark_data_smithi122_119671_object78531:head# 2024-03-20T09:04:49.807 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb1995fe:::benchmark_data_smithi122_119671_object50415:head# 2024-03-20T09:04:49.807 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb199888:::benchmark_data_smithi122_119671_object17726:head# 2024-03-20T09:04:49.808 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb1a1cf6:::benchmark_data_smithi122_119671_object84679:head# 2024-03-20T09:04:49.808 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb1a2219:::benchmark_data_smithi122_119671_object36843:head# 2024-03-20T09:04:49.809 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb1a8637:::benchmark_data_smithi122_119671_object12331:head# 2024-03-20T09:04:49.809 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb1bb1f4:::benchmark_data_smithi122_119671_object98196:head# 2024-03-20T09:04:49.810 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb1e1c84:::benchmark_data_smithi122_119671_object96793:head# 2024-03-20T09:04:49.810 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb1ef7ed:::benchmark_data_smithi122_119671_object84652:head# 2024-03-20T09:04:49.811 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb1f5ac3:::benchmark_data_smithi122_119671_object98319:head# 2024-03-20T09:04:49.811 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb21644d:::benchmark_data_smithi122_119671_object51778:head# 2024-03-20T09:04:49.811 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb217190:::benchmark_data_smithi122_119671_object95618:head# 2024-03-20T09:04:49.812 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb218ee9:::benchmark_data_smithi122_119671_object25297:head# 2024-03-20T09:04:49.812 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb21949b:::benchmark_data_smithi122_119671_object92241:head# 2024-03-20T09:04:49.812 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb21c771:::benchmark_data_smithi122_119671_object42383:head# 2024-03-20T09:04:49.813 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb21dcf7:::benchmark_data_smithi122_119671_object98865:head# 2024-03-20T09:04:49.813 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb22d906:::benchmark_data_smithi122_119671_object13855:head# 2024-03-20T09:04:49.814 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb22f378:::benchmark_data_smithi122_119671_object62928:head# 2024-03-20T09:04:49.814 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb2341d0:::benchmark_data_smithi122_119671_object63081:head# 2024-03-20T09:04:49.815 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb23682b:::benchmark_data_smithi122_119671_object51518:head# 2024-03-20T09:04:49.815 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb239bd4:::benchmark_data_smithi122_119671_object77224:head# 2024-03-20T09:04:49.816 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb264bca:::benchmark_data_smithi122_119671_object8502:head# 2024-03-20T09:04:49.817 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb282f7a:::benchmark_data_smithi122_119671_object50937:head# 2024-03-20T09:04:49.817 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb28acfc:::benchmark_data_smithi122_119671_object53440:head# 2024-03-20T09:04:49.818 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb28c810:::benchmark_data_smithi122_119671_object21380:head# 2024-03-20T09:04:49.818 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb29b98c:::benchmark_data_smithi122_119671_object66787:head# 2024-03-20T09:04:49.819 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb2a4bba:::benchmark_data_smithi122_119671_object26860:head# 2024-03-20T09:04:49.820 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb2a5009:::benchmark_data_smithi122_119671_object56535:head# 2024-03-20T09:04:49.820 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb2a9a92:::benchmark_data_smithi122_119671_object67528:head# 2024-03-20T09:04:49.821 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb2b96ab:::benchmark_data_smithi122_119671_object85460:head# 2024-03-20T09:04:49.821 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb2c7f61:::benchmark_data_smithi122_119671_object42228:head# 2024-03-20T09:04:49.821 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb2e0f33:::benchmark_data_smithi122_119671_object30395:head# 2024-03-20T09:04:49.822 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb2e7b92:::benchmark_data_smithi122_119671_object31068:head# 2024-03-20T09:04:49.822 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb2ea86b:::benchmark_data_smithi122_119671_object26746:head# 2024-03-20T09:04:49.823 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb2fc436:::benchmark_data_smithi122_119671_object84628:head# 2024-03-20T09:04:49.823 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb31c073:::benchmark_data_smithi122_119671_object1874:head# 2024-03-20T09:04:49.824 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:04:49.824 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb330f36:::benchmark_data_smithi122_119671_object46679:head# 2024-03-20T09:04:49.824 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb3357a1:::benchmark_data_smithi122_119671_object55482:head# 2024-03-20T09:04:49.825 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb33c5f3:::benchmark_data_smithi122_119671_object17794:head# 2024-03-20T09:04:49.825 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb33f22f:::benchmark_data_smithi122_119671_object51466:head# 2024-03-20T09:04:49.825 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb350a24:::benchmark_data_smithi122_119671_object39987:head# 2024-03-20T09:04:49.826 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb3541fc:::benchmark_data_smithi122_119671_object12639:head# 2024-03-20T09:04:49.826 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb360b9d:::benchmark_data_smithi122_119671_object16517:head# 2024-03-20T09:04:49.826 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb387b7d:::benchmark_data_smithi122_119671_object97686:head# 2024-03-20T09:04:49.827 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb395ec5:::benchmark_data_smithi122_119671_object23453:head# 2024-03-20T09:04:49.827 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb3e50f6:::benchmark_data_smithi122_119671_object50916:head# 2024-03-20T09:04:49.828 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb3ecee4:::benchmark_data_smithi122_119671_object57634:head# 2024-03-20T09:04:49.828 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb3f7aab:::benchmark_data_smithi122_119671_object54023:head# 2024-03-20T09:04:49.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:49.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:04:49.830 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb40486d:::benchmark_data_smithi122_119671_object93593:head# 2024-03-20T09:04:49.830 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb406470:::benchmark_data_smithi122_119671_object44842:head# 2024-03-20T09:04:49.830 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb40d18a:::benchmark_data_smithi122_119671_object91794:head# 2024-03-20T09:04:49.830 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb40eab9:::benchmark_data_smithi122_119671_object44288:head# 2024-03-20T09:04:49.831 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb419f40:::benchmark_data_smithi122_119671_object91871:head# 2024-03-20T09:04:49.831 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb41e56d:::benchmark_data_smithi122_119671_object15276:head# 2024-03-20T09:04:49.831 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb426bc7:::benchmark_data_smithi122_119671_object91782:head# 2024-03-20T09:04:49.831 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb42abd7:::benchmark_data_smithi122_119671_object84239:head# 2024-03-20T09:04:49.832 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb42c1c4:::benchmark_data_smithi122_119671_object27403:head# 2024-03-20T09:04:49.832 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb431813:::benchmark_data_smithi122_119671_object8894:head# 2024-03-20T09:04:49.833 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb440c58:::benchmark_data_smithi122_119671_object39682:head# 2024-03-20T09:04:49.833 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb4470a0:::benchmark_data_smithi122_119671_object27682:head# 2024-03-20T09:04:49.833 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb4478a2:::benchmark_data_smithi122_119671_object7953:head# 2024-03-20T09:04:49.834 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb4535b4:::benchmark_data_smithi122_119671_object65641:head# 2024-03-20T09:04:49.834 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb456d07:::benchmark_data_smithi122_119671_object38422:head# 2024-03-20T09:04:49.835 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb46bb3c:::benchmark_data_smithi122_119671_object79392:head# 2024-03-20T09:04:49.835 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb470ba7:::benchmark_data_smithi122_119671_object79085:head# 2024-03-20T09:04:49.836 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb472a82:::benchmark_data_smithi122_119671_object69522:head# 2024-03-20T09:04:49.836 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb4764b5:::benchmark_data_smithi122_119671_object67361:head# 2024-03-20T09:04:49.836 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb47c572:::benchmark_data_smithi122_119671_object17709:head# 2024-03-20T09:04:49.837 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb480c08:::benchmark_data_smithi122_119671_object46504:head# 2024-03-20T09:04:49.838 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb489db0:::benchmark_data_smithi122_119671_object25715:head# 2024-03-20T09:04:49.838 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb4ae3d2:::benchmark_data_smithi122_119671_object40131:head# 2024-03-20T09:04:49.839 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb4c3b15:::benchmark_data_smithi122_119671_object66152:head# 2024-03-20T09:04:49.839 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb4c6675:::benchmark_data_smithi122_119671_object89759:head# 2024-03-20T09:04:49.839 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb4d5232:::benchmark_data_smithi122_119671_object65184:head# 2024-03-20T09:04:49.839 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb4dd579:::benchmark_data_smithi122_119671_object76999:head# 2024-03-20T09:04:49.840 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb4e7810:::benchmark_data_smithi122_119671_object27209:head# 2024-03-20T09:04:49.840 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb4f7860:::benchmark_data_smithi122_119671_object83346:head# 2024-03-20T09:04:49.841 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb4ff0df:::benchmark_data_smithi122_119671_object18496:head# 2024-03-20T09:04:49.841 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb5102f3:::benchmark_data_smithi122_119671_object25719:head# 2024-03-20T09:04:49.841 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb510390:::benchmark_data_smithi122_119671_object94568:head# 2024-03-20T09:04:49.842 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb51b71f:::benchmark_data_smithi122_119671_object71851:head# 2024-03-20T09:04:49.842 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb52124f:::benchmark_data_smithi122_119671_object81451:head# 2024-03-20T09:04:49.843 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb527791:::benchmark_data_smithi122_119671_object57814:head# 2024-03-20T09:04:49.843 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb5475a0:::benchmark_data_smithi122_119671_object6153:head# 2024-03-20T09:04:49.843 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb54e1ce:::benchmark_data_smithi122_119671_object40071:head# 2024-03-20T09:04:49.844 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb54f40f:::benchmark_data_smithi122_119671_object12923:head# 2024-03-20T09:04:49.844 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb550e0b:::benchmark_data_smithi122_119671_object71619:head# 2024-03-20T09:04:49.845 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb55604e:::benchmark_data_smithi122_119671_object90429:head# 2024-03-20T09:04:49.845 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb5671c3:::benchmark_data_smithi122_119671_object59141:head# 2024-03-20T09:04:49.846 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb56a80f:::benchmark_data_smithi122_119671_object74429:head# 2024-03-20T09:04:49.846 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb56ba38:::benchmark_data_smithi122_119671_object19127:head# 2024-03-20T09:04:49.846 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb5792b2:::benchmark_data_smithi122_119671_object96513:head# 2024-03-20T09:04:49.847 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb596261:::benchmark_data_smithi122_119671_object52676:head# 2024-03-20T09:04:49.847 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb5a3620:::benchmark_data_smithi122_119671_object19675:head# 2024-03-20T09:04:49.848 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb5a7d96:::benchmark_data_smithi122_119671_object37626:head# 2024-03-20T09:04:49.848 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb5c0c79:::benchmark_data_smithi122_119671_object94088:head# 2024-03-20T09:04:49.848 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb5c8338:::benchmark_data_smithi122_119671_object19272:head# 2024-03-20T09:04:49.849 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb5c9eea:::benchmark_data_smithi122_119671_object15744:head# 2024-03-20T09:04:49.849 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb5d1270:::benchmark_data_smithi122_119671_object41696:head# 2024-03-20T09:04:49.850 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb5ea2f6:::benchmark_data_smithi122_119671_object97191:head# 2024-03-20T09:04:49.850 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb5f7166:::benchmark_data_smithi122_119671_object89630:head# 2024-03-20T09:04:49.851 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb5fc51d:::benchmark_data_smithi122_119671_object51394:head# 2024-03-20T09:04:49.851 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb61e9e3:::benchmark_data_smithi122_119671_object72801:head# 2024-03-20T09:04:49.851 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb623da2:::benchmark_data_smithi122_119671_object69625:head# 2024-03-20T09:04:49.852 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb62fec0:::benchmark_data_smithi122_119671_object3976:head# 2024-03-20T09:04:49.852 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb648ddd:::benchmark_data_smithi122_119671_object7501:head# 2024-03-20T09:04:49.852 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb649d02:::benchmark_data_smithi122_119671_object24399:head# 2024-03-20T09:04:49.853 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb649ffb:::benchmark_data_smithi122_119671_object61300:head# 2024-03-20T09:04:49.853 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb64db07:::benchmark_data_smithi122_119671_object65775:head# 2024-03-20T09:04:49.854 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb678889:::benchmark_data_smithi122_119671_object65152:head# 2024-03-20T09:04:49.854 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb6bef05:::benchmark_data_smithi122_119671_object73122:head# 2024-03-20T09:04:49.854 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb6c50a0:::benchmark_data_smithi122_119671_object5290:head# 2024-03-20T09:04:49.855 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb6c71d1:::benchmark_data_smithi122_119671_object21834:head# 2024-03-20T09:04:49.855 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb6ca104:::benchmark_data_smithi122_119671_object82143:head# 2024-03-20T09:04:49.856 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb6e6071:::benchmark_data_smithi122_119671_object13435:head# 2024-03-20T09:04:49.856 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb6ecfe3:::benchmark_data_smithi122_119671_object32259:head# 2024-03-20T09:04:49.857 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb6f2754:::benchmark_data_smithi122_119671_object68884:head# 2024-03-20T09:04:49.857 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb6ff720:::benchmark_data_smithi122_119671_object72677:head# 2024-03-20T09:04:49.857 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb7063f3:::benchmark_data_smithi122_119671_object64281:head# 2024-03-20T09:04:49.858 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb708f00:::benchmark_data_smithi122_119671_object86355:head# 2024-03-20T09:04:49.858 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb711236:::benchmark_data_smithi122_119671_object65557:head# 2024-03-20T09:04:49.859 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb71acdc:::benchmark_data_smithi122_119671_object71278:head# 2024-03-20T09:04:49.859 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb722566:::benchmark_data_smithi122_119671_object48649:head# 2024-03-20T09:04:49.859 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb722aab:::benchmark_data_smithi122_119671_object58846:head# 2024-03-20T09:04:49.859 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb72da50:::benchmark_data_smithi122_119671_object11031:head# 2024-03-20T09:04:49.860 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb73412f:::benchmark_data_smithi122_119671_object23190:head# 2024-03-20T09:04:49.860 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb7410bb:::benchmark_data_smithi122_119671_object96017:head# 2024-03-20T09:04:49.861 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb744e1e:::benchmark_data_smithi122_119671_object47691:head# 2024-03-20T09:04:49.861 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb74ba47:::benchmark_data_smithi122_119671_object25116:head# 2024-03-20T09:04:49.862 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb75533f:::benchmark_data_smithi122_119671_object29559:head# 2024-03-20T09:04:49.862 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb7621aa:::benchmark_data_smithi122_119671_object54574:head# 2024-03-20T09:04:49.862 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb7652f2:::benchmark_data_smithi122_119671_object29361:head# 2024-03-20T09:04:49.863 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb784b6c:::benchmark_data_smithi122_119671_object38104:head# 2024-03-20T09:04:49.863 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb7a93b9:::benchmark_data_smithi122_119671_object88913:head# 2024-03-20T09:04:49.863 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb7ad152:::benchmark_data_smithi122_119671_object34070:head# 2024-03-20T09:04:49.864 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb7d7908:::benchmark_data_smithi122_119671_object26337:head# 2024-03-20T09:04:49.864 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb7ed8de:::benchmark_data_smithi122_119671_object30015:head# 2024-03-20T09:04:49.865 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb7eec4c:::benchmark_data_smithi122_119671_object70033:head# 2024-03-20T09:04:49.865 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb7f3d50:::benchmark_data_smithi122_119671_object29759:head# 2024-03-20T09:04:49.865 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb82102a:::benchmark_data_smithi122_119671_object1278:head# 2024-03-20T09:04:49.866 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb8492ed:::benchmark_data_smithi122_119671_object18710:head# 2024-03-20T09:04:49.866 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb84c761:::benchmark_data_smithi122_119671_object12047:head# 2024-03-20T09:04:49.867 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb863661:::benchmark_data_smithi122_119671_object43664:head# 2024-03-20T09:04:49.867 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb86a35d:::benchmark_data_smithi122_119671_object60529:head# 2024-03-20T09:04:49.867 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb86d46f:::benchmark_data_smithi122_119671_object97139:head# 2024-03-20T09:04:49.868 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb88070c:::benchmark_data_smithi122_119671_object45595:head# 2024-03-20T09:04:49.868 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb881e56:::benchmark_data_smithi122_119671_object79080:head# 2024-03-20T09:04:49.869 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb88b541:::benchmark_data_smithi122_119671_object42178:head# 2024-03-20T09:04:49.869 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb88cba7:::benchmark_data_smithi122_119671_object17221:head# 2024-03-20T09:04:49.869 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb8aa04b:::benchmark_data_smithi122_119671_object86165:head# 2024-03-20T09:04:49.870 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb8b1db2:::benchmark_data_smithi122_119671_object57745:head# 2024-03-20T09:04:49.870 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb8ba9ce:::benchmark_data_smithi122_119671_object12846:head# 2024-03-20T09:04:49.870 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb8bbfd5:::benchmark_data_smithi122_119671_object77221:head# 2024-03-20T09:04:49.871 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb8beedd:::benchmark_data_smithi122_119671_object25564:head# 2024-03-20T09:04:49.871 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb8d69ec:::benchmark_data_smithi122_119671_object78229:head# 2024-03-20T09:04:49.871 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb8f6296:::benchmark_data_smithi122_119671_object48130:head# 2024-03-20T09:04:49.872 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb90624a:::benchmark_data_smithi122_119671_object62912:head# 2024-03-20T09:04:49.872 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb907b77:::benchmark_data_smithi122_119671_object83983:head# 2024-03-20T09:04:49.873 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb90ea38:::benchmark_data_smithi122_119671_object16243:head# 2024-03-20T09:04:49.873 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb912115:::benchmark_data_smithi122_119671_object88374:head# 2024-03-20T09:04:49.873 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb916492:::benchmark_data_smithi122_119671_object18679:head# 2024-03-20T09:04:49.874 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb924312:::benchmark_data_smithi122_119671_object49857:head# 2024-03-20T09:04:49.874 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb92cb41:::benchmark_data_smithi122_119671_object45019:head# 2024-03-20T09:04:49.874 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb9300a5:::benchmark_data_smithi122_119671_object47308:head# 2024-03-20T09:04:49.875 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb94194c:::benchmark_data_smithi122_119671_object41477:head# 2024-03-20T09:04:49.875 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb948575:::benchmark_data_smithi122_119671_object51315:head# 2024-03-20T09:04:49.876 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb95a0be:::benchmark_data_smithi122_119671_object86335:head# 2024-03-20T09:04:49.876 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb95a8c0:::benchmark_data_smithi122_119671_object38623:head# 2024-03-20T09:04:49.877 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb95c76f:::benchmark_data_smithi122_119671_object47467:head# 2024-03-20T09:04:49.877 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb963745:::benchmark_data_smithi122_119671_object63861:head# 2024-03-20T09:04:49.877 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb96e0e6:::benchmark_data_smithi122_119671_object83795:head# 2024-03-20T09:04:49.878 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb971461:::benchmark_data_smithi122_119671_object81704:head# 2024-03-20T09:04:49.878 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb97d2bf:::benchmark_data_smithi122_119671_object51780:head# 2024-03-20T09:04:49.878 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb97e43d:::benchmark_data_smithi122_119671_object38654:head# 2024-03-20T09:04:49.879 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb98a5c8:::benchmark_data_smithi122_119671_object89820:head# 2024-03-20T09:04:49.879 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb99a472:::benchmark_data_smithi122_119671_object12707:head# 2024-03-20T09:04:49.880 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb9a4862:::benchmark_data_smithi122_119671_object21448:head# 2024-03-20T09:04:49.880 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb9a79ca:::benchmark_data_smithi122_119671_object52150:head# 2024-03-20T09:04:49.881 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb9ae458:::benchmark_data_smithi122_119671_object32735:head# 2024-03-20T09:04:49.881 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb9b688f:::benchmark_data_smithi122_119671_object26784:head# 2024-03-20T09:04:49.881 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb9b6c0f:::benchmark_data_smithi122_119671_object89704:head# 2024-03-20T09:04:49.882 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb9c42f7:::benchmark_data_smithi122_119671_object91615:head# 2024-03-20T09:04:49.882 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb9c6044:::benchmark_data_smithi122_119671_object34812:head# 2024-03-20T09:04:49.883 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb9cc740:::benchmark_data_smithi122_119671_object11015:head# 2024-03-20T09:04:49.883 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb9cf26c:::benchmark_data_smithi122_119671_object74694:head# 2024-03-20T09:04:49.883 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb9d73ad:::benchmark_data_smithi122_119671_object64809:head# 2024-03-20T09:04:49.884 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb9db120:::benchmark_data_smithi122_119671_object85663:head# 2024-03-20T09:04:49.884 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb9e0398:::benchmark_data_smithi122_119671_object23105:head# 2024-03-20T09:04:49.885 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb9e1538:::benchmark_data_smithi122_119671_object44957:head# 2024-03-20T09:04:49.885 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb9f28c5:::benchmark_data_smithi122_119671_object45921:head# 2024-03-20T09:04:49.886 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb9f956b:::benchmark_data_smithi122_119671_object81116:head# 2024-03-20T09:04:49.886 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bb9f9f7b:::benchmark_data_smithi122_119671_object72522:head# 2024-03-20T09:04:49.887 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bba09424:::benchmark_data_smithi122_119671_object46355:head# 2024-03-20T09:04:49.887 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bba1c657:::benchmark_data_smithi122_119671_object88850:head# 2024-03-20T09:04:49.887 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bba23bae:::benchmark_data_smithi122_119671_object78291:head# 2024-03-20T09:04:49.888 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bba3848b:::benchmark_data_smithi122_119671_object46930:head# 2024-03-20T09:04:49.888 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bba40123:::benchmark_data_smithi122_119671_object43054:head# 2024-03-20T09:04:49.889 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bba6ae3c:::benchmark_data_smithi122_119671_object92258:head# 2024-03-20T09:04:49.889 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bba6af21:::benchmark_data_smithi122_119671_object2543:head# 2024-03-20T09:04:49.889 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bba6b7c0:::benchmark_data_smithi122_119671_object1586:head# 2024-03-20T09:04:49.890 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bba76e7d:::benchmark_data_smithi122_119671_object50630:head# 2024-03-20T09:04:49.890 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bba81831:::benchmark_data_smithi122_119671_object13189:head# 2024-03-20T09:04:49.891 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bba82f91:::benchmark_data_smithi122_119671_object16473:head# 2024-03-20T09:04:49.891 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bba87980:::benchmark_data_smithi122_119671_object32275:head# 2024-03-20T09:04:49.891 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bba8af9f:::benchmark_data_smithi122_119671_object8995:head# 2024-03-20T09:04:49.892 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bba9767e:::benchmark_data_smithi122_119671_object15267:head# 2024-03-20T09:04:49.892 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbaa016f:::benchmark_data_smithi122_119671_object64991:head# 2024-03-20T09:04:49.893 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbaaac80:::benchmark_data_smithi122_119671_object47458:head# 2024-03-20T09:04:49.893 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbab2739:::benchmark_data_smithi122_119671_object96942:head# 2024-03-20T09:04:49.894 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:04:49.894 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbab2812:::benchmark_data_smithi122_119671_object19869:head# 2024-03-20T09:04:49.894 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbab8b29:::benchmark_data_smithi122_119671_object7345:head# 2024-03-20T09:04:49.894 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbabec99:::benchmark_data_smithi122_119671_object46718:head# 2024-03-20T09:04:49.895 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbaced88:::benchmark_data_smithi122_119671_object92937:head# 2024-03-20T09:04:49.895 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbacff06:::benchmark_data_smithi122_119671_object47530:head# 2024-03-20T09:04:49.896 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbad1f1b:::benchmark_data_smithi122_119671_object4583:head# 2024-03-20T09:04:49.896 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbaefdfa:::benchmark_data_smithi122_119671_object75093:head# 2024-03-20T09:04:49.897 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbb197df:::benchmark_data_smithi122_119671_object68303:head# 2024-03-20T09:04:49.897 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbb1ace3:::benchmark_data_smithi122_119671_object72440:head# 2024-03-20T09:04:49.898 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbb1f5ec:::benchmark_data_smithi122_119671_object61707:head# 2024-03-20T09:04:49.898 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbb220c2:::benchmark_data_smithi122_119671_object59191:head# 2024-03-20T09:04:49.898 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbb26c0e:::benchmark_data_smithi122_119671_object36972:head# 2024-03-20T09:04:49.899 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbb281db:::benchmark_data_smithi122_119671_object35342:head# 2024-03-20T09:04:49.899 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbb30dd1:::benchmark_data_smithi122_119671_object59845:head# 2024-03-20T09:04:49.900 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbb3c83d:::benchmark_data_smithi122_119671_object48340:head# 2024-03-20T09:04:49.900 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbb3ff89:::benchmark_data_smithi122_119671_object81263:head# 2024-03-20T09:04:49.900 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbb4d977:::benchmark_data_smithi122_119671_object95254:head# 2024-03-20T09:04:49.901 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbb4f6ea:::benchmark_data_smithi122_119671_object63388:head# 2024-03-20T09:04:49.901 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbb50648:::benchmark_data_smithi122_119671_object95765:head# 2024-03-20T09:04:49.902 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbb531a2:::benchmark_data_smithi122_119671_object99605:head# 2024-03-20T09:04:49.902 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbb54cfb:::benchmark_data_smithi122_119671_object98819:head# 2024-03-20T09:04:49.903 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbb56e14:::benchmark_data_smithi122_119671_object8660:head# 2024-03-20T09:04:49.903 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbb58fc4:::benchmark_data_smithi122_119671_object40508:head# 2024-03-20T09:04:49.903 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbb724ec:::benchmark_data_smithi122_119671_object67740:head# 2024-03-20T09:04:49.903 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbb7df5e:::benchmark_data_smithi122_119671_object8087:head# 2024-03-20T09:04:49.904 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbb9ca3d:::benchmark_data_smithi122_119671_object67864:head# 2024-03-20T09:04:49.904 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbbbcf69:::benchmark_data_smithi122_119671_object34228:head# 2024-03-20T09:04:49.905 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbbd1413:::benchmark_data_smithi122_119671_object28071:head# 2024-03-20T09:04:49.905 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbbd6d47:::benchmark_data_smithi122_119671_object56490:head# 2024-03-20T09:04:49.905 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbbdc794:::benchmark_data_smithi122_119671_object4743:head# 2024-03-20T09:04:49.906 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbbdff6f:::benchmark_data_smithi122_119671_object20201:head# 2024-03-20T09:04:49.906 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbc0296e:::benchmark_data_smithi122_119671_object14364:head# 2024-03-20T09:04:49.907 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbc02a03:::benchmark_data_smithi122_119671_object38133:head# 2024-03-20T09:04:49.907 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbc17b28:::benchmark_data_smithi122_119671_object56151:head# 2024-03-20T09:04:49.907 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbc17fa4:::benchmark_data_smithi122_119671_object54547:head# 2024-03-20T09:04:49.908 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbc1ea6e:::benchmark_data_smithi122_119671_object59477:head# 2024-03-20T09:04:49.908 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbc497f4:::benchmark_data_smithi122_119671_object69513:head# 2024-03-20T09:04:49.909 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbc4e890:::benchmark_data_smithi122_119671_object37236:head# 2024-03-20T09:04:49.909 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbc4f54c:::benchmark_data_smithi122_119671_object12410:head# 2024-03-20T09:04:49.910 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbc64927:::benchmark_data_smithi122_119671_object83344:head# 2024-03-20T09:04:49.910 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbc9621f:::benchmark_data_smithi122_119671_object93445:head# 2024-03-20T09:04:49.911 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbca0a47:::benchmark_data_smithi122_119671_object61875:head# 2024-03-20T09:04:49.911 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbca715c:::benchmark_data_smithi122_119671_object42785:head# 2024-03-20T09:04:49.911 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbcaf927:::benchmark_data_smithi122_119671_object29140:head# 2024-03-20T09:04:49.911 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbcb1d97:::benchmark_data_smithi122_119671_object5509:head# 2024-03-20T09:04:49.912 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbcec9bd:::benchmark_data_smithi122_119671_object47596:head# 2024-03-20T09:04:49.912 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbcecdc1:::benchmark_data_smithi122_119671_object70695:head# 2024-03-20T09:04:49.913 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbced0f9:::benchmark_data_smithi122_119671_object22658:head# 2024-03-20T09:04:49.913 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbd05727:::benchmark_data_smithi122_119671_object92907:head# 2024-03-20T09:04:49.913 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbd11f75:::benchmark_data_smithi122_119671_object77793:head# 2024-03-20T09:04:49.914 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbd1dd99:::benchmark_data_smithi122_119671_object74615:head# 2024-03-20T09:04:49.914 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbd1f9d4:::benchmark_data_smithi122_119671_object7952:head# 2024-03-20T09:04:49.914 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbd1fac1:::benchmark_data_smithi122_119671_object70056:head# 2024-03-20T09:04:49.915 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbd2fa92:::benchmark_data_smithi122_119671_object74672:head# 2024-03-20T09:04:49.915 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbd3f961:::benchmark_data_smithi122_119671_object99621:head# 2024-03-20T09:04:49.916 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbd4f539:::benchmark_data_smithi122_119671_object78368:head# 2024-03-20T09:04:49.916 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbd50f5b:::benchmark_data_smithi122_119671_object60781:head# 2024-03-20T09:04:49.917 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbd53db8:::benchmark_data_smithi122_119671_object89039:head# 2024-03-20T09:04:49.917 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbd557eb:::benchmark_data_smithi122_119671_object84939:head# 2024-03-20T09:04:49.917 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbd7afb6:::benchmark_data_smithi122_119671_object23687:head# 2024-03-20T09:04:49.918 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbd82060:::benchmark_data_smithi122_119671_object97488:head# 2024-03-20T09:04:49.918 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbd883ef:::benchmark_data_smithi122_119671_object59407:head# 2024-03-20T09:04:49.919 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbd8ba89:::benchmark_data_smithi122_119671_object84058:head# 2024-03-20T09:04:49.919 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbda9da3:::benchmark_data_smithi122_119671_object53250:head# 2024-03-20T09:04:49.919 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbdaf813:::benchmark_data_smithi122_119671_object50542:head# 2024-03-20T09:04:49.920 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbdba078:::benchmark_data_smithi122_119671_object70722:head# 2024-03-20T09:04:49.920 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbdbcc23:::benchmark_data_smithi122_119671_object3900:head# 2024-03-20T09:04:49.920 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbdc05fa:::benchmark_data_smithi122_119671_object67377:head# 2024-03-20T09:04:49.921 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbdc0bc5:::benchmark_data_smithi122_119671_object5855:head# 2024-03-20T09:04:49.921 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbdce493:::benchmark_data_smithi122_119671_object90935:head# 2024-03-20T09:04:49.921 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbdd02d4:::benchmark_data_smithi122_119671_object33190:head# 2024-03-20T09:04:49.922 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbde1596:::benchmark_data_smithi122_119671_object74375:head# 2024-03-20T09:04:49.922 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbde23b7:::benchmark_data_smithi122_119671_object24047:head# 2024-03-20T09:04:49.923 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbdedf04:::benchmark_data_smithi122_119671_object29142:head# 2024-03-20T09:04:49.923 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbdf5db2:::benchmark_data_smithi122_119671_object98096:head# 2024-03-20T09:04:49.924 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbe00014:::benchmark_data_smithi122_119671_object91459:head# 2024-03-20T09:04:49.924 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbe07025:::benchmark_data_smithi122_119671_object54627:head# 2024-03-20T09:04:49.924 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbe16fb0:::benchmark_data_smithi122_119671_object32817:head# 2024-03-20T09:04:49.925 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbe18dc1:::benchmark_data_smithi122_119671_object55393:head# 2024-03-20T09:04:49.925 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbe48346:::benchmark_data_smithi122_119671_object53575:head# 2024-03-20T09:04:49.926 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbe48757:::benchmark_data_smithi122_119671_object51505:head# 2024-03-20T09:04:49.926 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbe54605:::benchmark_data_smithi122_119671_object29278:head# 2024-03-20T09:04:49.927 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbe5e018:::benchmark_data_smithi122_119671_object34784:head# 2024-03-20T09:04:49.927 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbe6420b:::benchmark_data_smithi122_119671_object39848:head# 2024-03-20T09:04:49.927 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbe6ea66:::benchmark_data_smithi122_119671_object35221:head# 2024-03-20T09:04:49.928 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbe7a60e:::benchmark_data_smithi122_119671_object1531:head# 2024-03-20T09:04:49.928 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbe84ce1:::benchmark_data_smithi122_119671_object45930:head# 2024-03-20T09:04:49.929 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbe856e5:::benchmark_data_smithi122_119671_object19089:head# 2024-03-20T09:04:49.929 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbe997d1:::benchmark_data_smithi122_119671_object74534:head# 2024-03-20T09:04:49.929 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:04:49.930 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbea3bd0:::benchmark_data_smithi122_119671_object44074:head# 2024-03-20T09:04:49.930 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbeb06b3:::benchmark_data_smithi122_119671_object22184:head# 2024-03-20T09:04:49.930 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbeb7db0:::benchmark_data_smithi122_119671_object44764:head# 2024-03-20T09:04:49.931 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbeb9d6c:::benchmark_data_smithi122_119671_object5788:head# 2024-03-20T09:04:49.931 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbec561e:::benchmark_data_smithi122_119671_object83728:head# 2024-03-20T09:04:49.932 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbeca5a8:::benchmark_data_smithi122_119671_object68147:head# 2024-03-20T09:04:49.932 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbecf716:::benchmark_data_smithi122_119671_object81805:head# 2024-03-20T09:04:49.933 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbee2f0b:::benchmark_data_smithi122_119671_object47081:head# 2024-03-20T09:04:49.933 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbee4f87:::benchmark_data_smithi122_119671_object42017:head# 2024-03-20T09:04:49.933 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbeeecef:::benchmark_data_smithi122_119671_object82540:head# 2024-03-20T09:04:49.934 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbef0c4f:::benchmark_data_smithi122_119671_object15777:head# 2024-03-20T09:04:49.934 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbef2633:::benchmark_data_smithi122_119671_object38172:head# 2024-03-20T09:04:49.935 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbef7ff5:::benchmark_data_smithi122_119671_object83134:head# 2024-03-20T09:04:49.935 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbf0ace8:::benchmark_data_smithi122_119671_object16617:head# 2024-03-20T09:04:49.936 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbf0b42d:::benchmark_data_smithi122_119671_object33869:head# 2024-03-20T09:04:49.936 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbf1d4bf:::benchmark_data_smithi122_119671_object28976:head# 2024-03-20T09:04:49.936 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbf244de:::benchmark_data_smithi122_119671_object30170:head# 2024-03-20T09:04:49.937 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbf25014:::benchmark_data_smithi122_119671_object86634:head# 2024-03-20T09:04:49.937 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbf2826d:::benchmark_data_smithi122_119671_object36917:head# 2024-03-20T09:04:49.938 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbf29edb:::benchmark_data_smithi122_119671_object97439:head# 2024-03-20T09:04:49.938 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbf2a7d5:::benchmark_data_smithi122_119671_object23746:head# 2024-03-20T09:04:49.939 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbf2c746:::benchmark_data_smithi122_119671_object53658:head# 2024-03-20T09:04:49.939 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbf2cab3:::benchmark_data_smithi122_119671_object77143:head# 2024-03-20T09:04:49.939 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbf33b7a:::benchmark_data_smithi122_119671_object59717:head# 2024-03-20T09:04:49.942 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbf33ff7:::benchmark_data_smithi122_119671_object91012:head# 2024-03-20T09:04:49.942 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbf439d1:::benchmark_data_smithi122_119671_object61938:head# 2024-03-20T09:04:49.943 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbf472aa:::benchmark_data_smithi122_119671_object47766:head# 2024-03-20T09:04:49.943 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbf58b9f:::benchmark_data_smithi122_119671_object80328:head# 2024-03-20T09:04:49.944 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbf63419:::benchmark_data_smithi122_119671_object96820:head# 2024-03-20T09:04:49.944 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbf6b7f0:::benchmark_data_smithi122_119671_object2369:head# 2024-03-20T09:04:49.945 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbf6f0c4:::benchmark_data_smithi122_119671_object83626:head# 2024-03-20T09:04:49.945 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbf7ad23:::benchmark_data_smithi122_119671_object6281:head# 2024-03-20T09:04:49.946 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbf7dd2c:::benchmark_data_smithi122_119671_object50374:head# 2024-03-20T09:04:49.946 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbf7fc96:::benchmark_data_smithi122_119671_object23227:head# 2024-03-20T09:04:49.946 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbf8251d:::benchmark_data_smithi122_119671_object2660:head# 2024-03-20T09:04:49.947 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbf9101b:::benchmark_data_smithi122_119671_object70014:head# 2024-03-20T09:04:49.947 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbfb9acf:::benchmark_data_smithi122_119671_object47485:head# 2024-03-20T09:04:49.948 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbfba9ad:::benchmark_data_smithi122_119671_object98359:head# 2024-03-20T09:04:49.948 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbfd62ee:::benchmark_data_smithi122_119671_object73041:head# 2024-03-20T09:04:49.949 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbfdb270:::benchmark_data_smithi122_119671_object85642:head# 2024-03-20T09:04:49.949 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbfe2c57:::benchmark_data_smithi122_119671_object40696:head# 2024-03-20T09:04:49.949 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbfe74eb:::benchmark_data_smithi122_119671_object55154:head# 2024-03-20T09:04:49.950 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbff68c4:::benchmark_data_smithi122_119671_object4247:head# 2024-03-20T09:04:49.950 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:04:49.951 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bbffcd01:::benchmark_data_smithi122_119671_object81817:head# 2024-03-20T09:04:49.951 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc000983:::benchmark_data_smithi122_119671_object78254:head# 2024-03-20T09:04:49.951 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc0026ba:::benchmark_data_smithi122_119671_object60930:head# 2024-03-20T09:04:49.952 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc004012:::benchmark_data_smithi122_119671_object20466:head# 2024-03-20T09:04:49.952 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc007a2d:::benchmark_data_smithi122_119671_object89716:head# 2024-03-20T09:04:49.953 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc01d290:::benchmark_data_smithi122_119671_object27926:head# 2024-03-20T09:04:49.953 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc01e906:::benchmark_data_smithi122_119671_object54780:head# 2024-03-20T09:04:49.954 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc020ceb:::benchmark_data_smithi122_119671_object90919:head# 2024-03-20T09:04:49.954 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc0243cb:::benchmark_data_smithi122_119671_object76739:head# 2024-03-20T09:04:49.954 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc046001:::benchmark_data_smithi122_119671_object75134:head# 2024-03-20T09:04:49.955 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc048331:::benchmark_data_smithi122_119671_object86935:head# 2024-03-20T09:04:49.957 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc04addd:::benchmark_data_smithi122_119671_object25978:head# 2024-03-20T09:04:49.959 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc0618a0:::benchmark_data_smithi122_119671_object82134:head# 2024-03-20T09:04:49.959 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc064f83:::benchmark_data_smithi122_119671_object18289:head# 2024-03-20T09:04:49.959 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc0692f4:::benchmark_data_smithi122_119671_object2410:head# 2024-03-20T09:04:49.959 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc0698fa:::benchmark_data_smithi122_119671_object83058:head# 2024-03-20T09:04:49.959 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc06a2a8:::benchmark_data_smithi122_119671_object5124:head# 2024-03-20T09:04:49.959 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc0707cc:::benchmark_data_smithi122_119671_object39488:head# 2024-03-20T09:04:49.959 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc086f54:::benchmark_data_smithi122_119671_object41014:head# 2024-03-20T09:04:49.960 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc09e473:::benchmark_data_smithi122_119671_object12531:head# 2024-03-20T09:04:49.960 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc0a9fa5:::benchmark_data_smithi122_119671_object18215:head# 2024-03-20T09:04:49.960 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc0afcca:::benchmark_data_smithi122_119671_object19114:head# 2024-03-20T09:04:49.961 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc0c4839:::benchmark_data_smithi122_119671_object82225:head# 2024-03-20T09:04:49.961 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc0c74a3:::benchmark_data_smithi122_119671_object54126:head# 2024-03-20T09:04:49.961 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc0dfe69:::benchmark_data_smithi122_119671_object60718:head# 2024-03-20T09:04:49.961 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc0e0595:::benchmark_data_smithi122_119671_object27862:head# 2024-03-20T09:04:49.961 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc0e998a:::benchmark_data_smithi122_119671_object35444:head# 2024-03-20T09:04:49.962 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc0ee886:::benchmark_data_smithi122_119671_object27277:head# 2024-03-20T09:04:49.962 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc0f1db8:::benchmark_data_smithi122_119671_object52327:head# 2024-03-20T09:04:49.962 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc0f2147:::benchmark_data_smithi122_119671_object44135:head# 2024-03-20T09:04:49.966 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc10692f:::benchmark_data_smithi122_119671_object64108:head# 2024-03-20T09:04:49.966 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc10eeb2:::benchmark_data_smithi122_119671_object27546:head# 2024-03-20T09:04:49.966 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc1136e4:::benchmark_data_smithi122_119671_object13402:head# 2024-03-20T09:04:49.967 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc115849:::benchmark_data_smithi122_119671_object92395:head# 2024-03-20T09:04:49.968 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc12c424:::benchmark_data_smithi122_119671_object84928:head# 2024-03-20T09:04:49.969 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc1363a1:::benchmark_data_smithi122_119671_object84566:head# 2024-03-20T09:04:49.969 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc1381cf:::benchmark_data_smithi122_119671_object90763:head# 2024-03-20T09:04:49.969 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc14a0ad:::benchmark_data_smithi122_119671_object62100:head# 2024-03-20T09:04:49.969 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:49.970 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc15778b:::benchmark_data_smithi122_119671_object49971:head# 2024-03-20T09:04:49.970 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc178aad:::benchmark_data_smithi122_119671_object74041:head# 2024-03-20T09:04:49.970 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc17dff1:::benchmark_data_smithi122_119671_object99427:head# 2024-03-20T09:04:49.970 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc1846cf:::benchmark_data_smithi122_119671_object40681:head# 2024-03-20T09:04:49.971 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc184df7:::benchmark_data_smithi122_119671_object90023:head# 2024-03-20T09:04:49.971 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc191807:::benchmark_data_smithi122_119671_object83823:head# 2024-03-20T09:04:49.971 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc19466a:::benchmark_data_smithi122_119671_object62092:head# 2024-03-20T09:04:49.972 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc19dc2b:::benchmark_data_smithi122_119671_object76700:head# 2024-03-20T09:04:49.972 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc1ada60:::benchmark_data_smithi122_119671_object16985:head# 2024-03-20T09:04:49.972 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc1c5535:::benchmark_data_smithi122_119671_object44609:head# 2024-03-20T09:04:49.973 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc1c6ef2:::benchmark_data_smithi122_119671_object47943:head# 2024-03-20T09:04:49.973 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc1c9f8e:::benchmark_data_smithi122_119671_object19140:head# 2024-03-20T09:04:49.974 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc1d10a1:::benchmark_data_smithi122_119671_object49518:head# 2024-03-20T09:04:49.974 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc1d4b97:::benchmark_data_smithi122_119671_object85749:head# 2024-03-20T09:04:49.975 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc1e0d78:::benchmark_data_smithi122_119671_object49373:head# 2024-03-20T09:04:49.975 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc1e4609:::benchmark_data_smithi122_119671_object52604:head# 2024-03-20T09:04:49.975 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc1f2758:::benchmark_data_smithi122_119671_object97497:head# 2024-03-20T09:04:49.976 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc21231e:::benchmark_data_smithi122_119671_object86975:head# 2024-03-20T09:04:49.976 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc22b0a4:::benchmark_data_smithi122_119671_object83799:head# 2024-03-20T09:04:49.977 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc22dc98:::benchmark_data_smithi122_119671_object74107:head# 2024-03-20T09:04:49.979 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc24b940:::benchmark_data_smithi122_119671_object61490:head# 2024-03-20T09:04:49.980 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc259e66:::benchmark_data_smithi122_119671_object90298:head# 2024-03-20T09:04:49.980 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc25eba0:::benchmark_data_smithi122_119671_object79892:head# 2024-03-20T09:04:49.980 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc26106e:::benchmark_data_smithi122_119671_object80284:head# 2024-03-20T09:04:49.980 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc26645c:::benchmark_data_smithi122_119671_object3671:head# 2024-03-20T09:04:49.980 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc27c863:::benchmark_data_smithi122_119671_object69991:head# 2024-03-20T09:04:49.981 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc27cb18:::benchmark_data_smithi122_119671_object77762:head# 2024-03-20T09:04:49.981 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc28c877:::benchmark_data_smithi122_119671_object2766:head# 2024-03-20T09:04:49.981 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc2dd1b6:::benchmark_data_smithi122_119671_object75814:head# 2024-03-20T09:04:49.981 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc2ebbbe:::benchmark_data_smithi122_119671_object23953:head# 2024-03-20T09:04:49.982 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc2ec479:::benchmark_data_smithi122_119671_object46022:head# 2024-03-20T09:04:49.982 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc2f378f:::benchmark_data_smithi122_119671_object62258:head# 2024-03-20T09:04:49.982 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc302f45:::benchmark_data_smithi122_119671_object13624:head# 2024-03-20T09:04:49.983 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc30cefb:::benchmark_data_smithi122_119671_object22722:head# 2024-03-20T09:04:49.983 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc31d3a7:::benchmark_data_smithi122_119671_object46837:head# 2024-03-20T09:04:49.983 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc321cdd:::benchmark_data_smithi122_119671_object14965:head# 2024-03-20T09:04:49.984 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc323401:::benchmark_data_smithi122_119671_object52656:head# 2024-03-20T09:04:49.984 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc323922:::benchmark_data_smithi122_119671_object60143:head# 2024-03-20T09:04:49.985 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc329e5e:::benchmark_data_smithi122_119671_object4375:head# 2024-03-20T09:04:49.985 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc32e9c8:::benchmark_data_smithi122_119671_object40443:head# 2024-03-20T09:04:49.985 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc3308ff:::benchmark_data_smithi122_119671_object67716:head# 2024-03-20T09:04:49.986 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc33643f:::benchmark_data_smithi122_119671_object16150:head# 2024-03-20T09:04:49.986 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc339308:::benchmark_data_smithi122_119671_object23487:head# 2024-03-20T09:04:49.986 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc34bb1f:::benchmark_data_smithi122_119671_object28316:head# 2024-03-20T09:04:49.987 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc36c27c:::benchmark_data_smithi122_119671_object69932:head# 2024-03-20T09:04:49.987 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc379935:::benchmark_data_smithi122_119671_object14929:head# 2024-03-20T09:04:49.988 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc37f178:::benchmark_data_smithi122_119671_object85113:head# 2024-03-20T09:04:49.988 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc380113:::benchmark_data_smithi122_119671_object13858:head# 2024-03-20T09:04:49.988 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc3812d1:::benchmark_data_smithi122_119671_object40013:head# 2024-03-20T09:04:49.989 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc38855e:::benchmark_data_smithi122_119671_object58980:head# 2024-03-20T09:04:49.989 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc38b093:::benchmark_data_smithi122_119671_object18087:head# 2024-03-20T09:04:49.990 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc393680:::benchmark_data_smithi122_119671_object38634:head# 2024-03-20T09:04:49.990 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc3a6e23:::benchmark_data_smithi122_119671_object28648:head# 2024-03-20T09:04:49.991 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc3a9b8e:::benchmark_data_smithi122_119671_object11404:head# 2024-03-20T09:04:49.991 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc3ab47b:::benchmark_data_smithi122_119671_object93851:head# 2024-03-20T09:04:49.991 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc3ab611:::benchmark_data_smithi122_119671_object72419:head# 2024-03-20T09:04:49.992 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc3ae039:::benchmark_data_smithi122_119671_object66202:head# 2024-03-20T09:04:49.992 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc3bc6e8:::benchmark_data_smithi122_119671_object90659:head# 2024-03-20T09:04:49.993 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc3bcb48:::benchmark_data_smithi122_119671_object42209:head# 2024-03-20T09:04:49.993 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc3be9da:::benchmark_data_smithi122_119671_object89933:head# 2024-03-20T09:04:49.993 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc3dba61:::benchmark_data_smithi122_119671_object23577:head# 2024-03-20T09:04:49.994 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc3dd318:::benchmark_data_smithi122_119671_object73357:head# 2024-03-20T09:04:49.994 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc3e1f66:::benchmark_data_smithi122_119671_object49836:head# 2024-03-20T09:04:49.994 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc3f7239:::benchmark_data_smithi122_119671_object64726:head# 2024-03-20T09:04:49.995 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc403131:::benchmark_data_smithi122_119671_object45104:head# 2024-03-20T09:04:49.995 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc4092c5:::benchmark_data_smithi122_119671_object18658:head# 2024-03-20T09:04:49.996 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc40df23:::benchmark_data_smithi122_119671_object41620:head# 2024-03-20T09:04:49.996 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc414500:::benchmark_data_smithi122_119671_object19012:head# 2024-03-20T09:04:49.997 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc428a2d:::benchmark_data_smithi122_119671_object86399:head# 2024-03-20T09:04:49.997 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc43077e:::benchmark_data_smithi122_119671_object30455:head# 2024-03-20T09:04:49.998 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc433f18:::benchmark_data_smithi122_119671_object99205:head# 2024-03-20T09:04:49.998 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc44fb02:::benchmark_data_smithi122_119671_object90679:head# 2024-03-20T09:04:49.999 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc451ea0:::benchmark_data_smithi122_119671_object51124:head# 2024-03-20T09:04:49.999 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc452f6a:::benchmark_data_smithi122_119671_object86196:head# 2024-03-20T09:04:49.999 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc45ecf9:::benchmark_data_smithi122_119671_object88826:head# 2024-03-20T09:04:50.002 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc460102:::benchmark_data_smithi122_119671_object28805:head# 2024-03-20T09:04:50.002 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc461174:::benchmark_data_smithi122_119671_object76634:head# 2024-03-20T09:04:50.002 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc47be4d:::benchmark_data_smithi122_119671_object7530:head# 2024-03-20T09:04:50.002 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc4981ba:::benchmark_data_smithi122_119671_object21842:head# 2024-03-20T09:04:50.002 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc49ebb8:::benchmark_data_smithi122_119671_object64486:head# 2024-03-20T09:04:50.002 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc4a6262:::benchmark_data_smithi122_119671_object92343:head# 2024-03-20T09:04:50.004 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc4ad7ca:::benchmark_data_smithi122_119671_object35268:head# 2024-03-20T09:04:50.004 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc4b3981:::benchmark_data_smithi122_119671_object78794:head# 2024-03-20T09:04:50.004 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc4ca755:::benchmark_data_smithi122_119671_object27664:head# 2024-03-20T09:04:50.007 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc4df9b6:::benchmark_data_smithi122_119671_object40912:head# 2024-03-20T09:04:50.007 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc4eae99:::benchmark_data_smithi122_119671_object67480:head# 2024-03-20T09:04:50.007 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc4f1347:::benchmark_data_smithi122_119671_object42361:head# 2024-03-20T09:04:50.008 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc4fd2f0:::benchmark_data_smithi122_119671_object39718:head# 2024-03-20T09:04:50.009 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc5281bc:::benchmark_data_smithi122_119671_object95382:head# 2024-03-20T09:04:50.009 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc52c164:::benchmark_data_smithi122_119671_object53604:head# 2024-03-20T09:04:50.009 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc55d723:::benchmark_data_smithi122_119671_object9762:head# 2024-03-20T09:04:50.009 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc560e4e:::benchmark_data_smithi122_119671_object67948:head# 2024-03-20T09:04:50.010 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc561be9:::benchmark_data_smithi122_119671_object92705:head# 2024-03-20T09:04:50.010 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc57b2d4:::benchmark_data_smithi122_119671_object30330:head# 2024-03-20T09:04:50.010 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc57d3d8:::benchmark_data_smithi122_119671_object54419:head# 2024-03-20T09:04:50.011 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc598a06:::benchmark_data_smithi122_119671_object15912:head# 2024-03-20T09:04:50.011 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc59deaa:::benchmark_data_smithi122_119671_object7036:head# 2024-03-20T09:04:50.012 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc5a1434:::benchmark_data_smithi122_119671_object73939:head# 2024-03-20T09:04:50.012 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc5a8842:::benchmark_data_smithi122_119671_object58625:head# 2024-03-20T09:04:50.012 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc5ca43b:::benchmark_data_smithi122_119671_object33341:head# 2024-03-20T09:04:50.013 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc5cb58b:::benchmark_data_smithi122_119671_object52675:head# 2024-03-20T09:04:50.013 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc5cc822:::benchmark_data_smithi122_119671_object69502:head# 2024-03-20T09:04:50.014 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc5d37ff:::benchmark_data_smithi122_119671_object75899:head# 2024-03-20T09:04:50.014 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc5d744e:::benchmark_data_smithi122_119671_object57551:head# 2024-03-20T09:04:50.015 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc5d919c:::benchmark_data_smithi122_119671_object27135:head# 2024-03-20T09:04:50.015 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc5da734:::benchmark_data_smithi122_119671_object55485:head# 2024-03-20T09:04:50.015 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc5dca20:::benchmark_data_smithi122_119671_object29471:head# 2024-03-20T09:04:50.015 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc5e6efc:::benchmark_data_smithi122_119671_object83922:head# 2024-03-20T09:04:50.016 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc5eca10:::benchmark_data_smithi122_119671_object76874:head# 2024-03-20T09:04:50.016 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc61d2cc:::benchmark_data_smithi122_119671_object31846:head# 2024-03-20T09:04:50.017 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc63df75:::benchmark_data_smithi122_119671_object33353:head# 2024-03-20T09:04:50.017 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc65aad7:::benchmark_data_smithi122_119671_object64053:head# 2024-03-20T09:04:50.017 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc65b185:::benchmark_data_smithi122_119671_object97193:head# 2024-03-20T09:04:50.018 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc65b428:::benchmark_data_smithi122_119671_object40486:head# 2024-03-20T09:04:50.018 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc65d158:::benchmark_data_smithi122_119671_object36273:head# 2024-03-20T09:04:50.019 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc663715:::benchmark_data_smithi122_119671_object23743:head# 2024-03-20T09:04:50.019 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc688e77:::benchmark_data_smithi122_119671_object44488:head# 2024-03-20T09:04:50.020 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc68d32e:::benchmark_data_smithi122_119671_object20066:head# 2024-03-20T09:04:50.020 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc694a73:::benchmark_data_smithi122_119671_object9351:head# 2024-03-20T09:04:50.020 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc696e81:::benchmark_data_smithi122_119671_object36186:head# 2024-03-20T09:04:50.021 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc699428:::benchmark_data_smithi122_119671_object85298:head# 2024-03-20T09:04:50.021 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc6a015a:::benchmark_data_smithi122_119671_object51792:head# 2024-03-20T09:04:50.022 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc6bda50:::benchmark_data_smithi122_119671_object99007:head# 2024-03-20T09:04:50.022 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc6c3b67:::benchmark_data_smithi122_119671_object37401:head# 2024-03-20T09:04:50.022 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc6cfa5d:::benchmark_data_smithi122_119671_object29583:head# 2024-03-20T09:04:50.023 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc6d8e47:::benchmark_data_smithi122_119671_object65770:head# 2024-03-20T09:04:50.023 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc6db57b:::benchmark_data_smithi122_119671_object97124:head# 2024-03-20T09:04:50.024 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc6e5a03:::benchmark_data_smithi122_119671_object29825:head# 2024-03-20T09:04:50.024 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc6fd9af:::benchmark_data_smithi122_119671_object77389:head# 2024-03-20T09:04:50.024 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc6ffe66:::benchmark_data_smithi122_119671_object23032:head# 2024-03-20T09:04:50.025 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc7019f1:::benchmark_data_smithi122_119671_object32507:head# 2024-03-20T09:04:50.025 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc709a20:::benchmark_data_smithi122_119671_object29717:head# 2024-03-20T09:04:50.025 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc74e155:::benchmark_data_smithi122_119671_object86108:head# 2024-03-20T09:04:50.026 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc760a8a:::benchmark_data_smithi122_119671_object71683:head# 2024-03-20T09:04:50.026 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc76cb94:::benchmark_data_smithi122_119671_object8845:head# 2024-03-20T09:04:50.027 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc7822f3:::benchmark_data_smithi122_119671_object53302:head# 2024-03-20T09:04:50.028 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc78b4a4:::benchmark_data_smithi122_119671_object32376:head# 2024-03-20T09:04:50.029 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc798812:::benchmark_data_smithi122_119671_object82363:head# 2024-03-20T09:04:50.029 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc79c4cf:::benchmark_data_smithi122_119671_object74612:head# 2024-03-20T09:04:50.029 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc79e003:::benchmark_data_smithi122_119671_object26752:head# 2024-03-20T09:04:50.030 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc7dc511:::benchmark_data_smithi122_119671_object30863:head# 2024-03-20T09:04:50.030 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc7e1632:::benchmark_data_smithi122_119671_object64841:head# 2024-03-20T09:04:50.030 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc7e8e5a:::benchmark_data_smithi122_119671_object13074:head# 2024-03-20T09:04:50.032 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc7eb84d:::benchmark_data_smithi122_119671_object8663:head# 2024-03-20T09:04:50.033 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc7f05e6:::benchmark_data_smithi122_119671_object55177:head# 2024-03-20T09:04:50.033 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc7f2235:::benchmark_data_smithi122_119671_object68352:head# 2024-03-20T09:04:50.034 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc8160bf:::benchmark_data_smithi122_119671_object22796:head# 2024-03-20T09:04:50.035 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc81f87f:::benchmark_data_smithi122_119671_object12446:head# 2024-03-20T09:04:50.035 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc823894:::benchmark_data_smithi122_119671_object28536:head# 2024-03-20T09:04:50.038 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc82b08c:::benchmark_data_smithi122_119671_object55755:head# 2024-03-20T09:04:50.038 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc82fa66:::benchmark_data_smithi122_119671_object97761:head# 2024-03-20T09:04:50.039 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc835247:::benchmark_data_smithi122_119671_object17058:head# 2024-03-20T09:04:50.039 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc83568e:::benchmark_data_smithi122_119671_object89772:head# 2024-03-20T09:04:50.039 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc84bf38:::benchmark_data_smithi122_119671_object3942:head# 2024-03-20T09:04:50.040 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc872459:::benchmark_data_smithi122_119671_object51139:head# 2024-03-20T09:04:50.040 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc87dd55:::benchmark_data_smithi122_119671_object68218:head# 2024-03-20T09:04:50.042 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc87dff5:::benchmark_data_smithi122_119671_object72282:head# 2024-03-20T09:04:50.042 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc881225:::benchmark_data_smithi122_119671_object13904:head# 2024-03-20T09:04:50.043 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc892e5c:::benchmark_data_smithi122_119671_object95078:head# 2024-03-20T09:04:50.043 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc89e3e5:::benchmark_data_smithi122_119671_object43583:head# 2024-03-20T09:04:50.044 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc8a64bd:::benchmark_data_smithi122_119671_object64724:head# 2024-03-20T09:04:50.044 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc8b8f9a:::benchmark_data_smithi122_119671_object49933:head# 2024-03-20T09:04:50.045 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc8bf321:::benchmark_data_smithi122_119671_object11942:head# 2024-03-20T09:04:50.045 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc8c63b8:::benchmark_data_smithi122_119671_object37886:head# 2024-03-20T09:04:50.045 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc8c9c44:::benchmark_data_smithi122_119671_object18181:head# 2024-03-20T09:04:50.046 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc8d1d02:::benchmark_data_smithi122_119671_object83831:head# 2024-03-20T09:04:50.046 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc8dc6dd:::benchmark_data_smithi122_119671_object48985:head# 2024-03-20T09:04:50.047 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc8df157:::benchmark_data_smithi122_119671_object84002:head# 2024-03-20T09:04:50.047 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc8eafd1:::benchmark_data_smithi122_119671_object88565:head# 2024-03-20T09:04:50.047 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc8f6bca:::benchmark_data_smithi122_119671_object69348:head# 2024-03-20T09:04:50.048 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc90890e:::benchmark_data_smithi122_119671_object42330:head# 2024-03-20T09:04:50.048 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc909d7a:::benchmark_data_smithi122_119671_object40492:head# 2024-03-20T09:04:50.049 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc90d132:::benchmark_data_smithi122_119671_object4205:head# 2024-03-20T09:04:50.049 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc912332:::benchmark_data_smithi122_119671_object17587:head# 2024-03-20T09:04:50.049 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc913575:::benchmark_data_smithi122_119671_object52356:head# 2024-03-20T09:04:50.050 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc915809:::benchmark_data_smithi122_119671_object82953:head# 2024-03-20T09:04:50.050 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc9174d6:::benchmark_data_smithi122_119671_object79760:head# 2024-03-20T09:04:50.051 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc9188d2:::benchmark_data_smithi122_119671_object66669:head# 2024-03-20T09:04:50.051 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc91fb79:::benchmark_data_smithi122_119671_object61622:head# 2024-03-20T09:04:50.051 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc92319b:::benchmark_data_smithi122_119671_object10301:head# 2024-03-20T09:04:50.052 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc931e3e:::benchmark_data_smithi122_119671_object9977:head# 2024-03-20T09:04:50.052 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc93667d:::benchmark_data_smithi122_119671_object30694:head# 2024-03-20T09:04:50.052 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc936873:::benchmark_data_smithi122_119671_object41637:head# 2024-03-20T09:04:50.053 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc937c56:::benchmark_data_smithi122_119671_object53731:head# 2024-03-20T09:04:50.053 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc93b306:::benchmark_data_smithi122_119671_object68462:head# 2024-03-20T09:04:50.054 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc940976:::benchmark_data_smithi122_119671_object28723:head# 2024-03-20T09:04:50.054 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc9481c3:::benchmark_data_smithi122_119671_object65136:head# 2024-03-20T09:04:50.055 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc95de1f:::benchmark_data_smithi122_119671_object50065:head# 2024-03-20T09:04:50.055 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc96cecb:::benchmark_data_smithi122_119671_object81103:head# 2024-03-20T09:04:50.056 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc9996a7:::benchmark_data_smithi122_119671_object74511:head# 2024-03-20T09:04:50.056 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc99b5e4:::benchmark_data_smithi122_119671_object80712:head# 2024-03-20T09:04:50.056 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc99d07c:::benchmark_data_smithi122_119671_object7903:head# 2024-03-20T09:04:50.057 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc9a01f1:::benchmark_data_smithi122_119671_object90989:head# 2024-03-20T09:04:50.057 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc9b0295:::benchmark_data_smithi122_119671_object23912:head# 2024-03-20T09:04:50.058 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc9b68e2:::benchmark_data_smithi122_119671_object57392:head# 2024-03-20T09:04:50.058 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc9c4608:::benchmark_data_smithi122_119671_object30217:head# 2024-03-20T09:04:50.059 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc9e1933:::benchmark_data_smithi122_119671_object40915:head# 2024-03-20T09:04:50.059 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc9ee484:::benchmark_data_smithi122_119671_object60264:head# 2024-03-20T09:04:50.060 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bc9f2029:::benchmark_data_smithi122_119671_object72135:head# 2024-03-20T09:04:50.060 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bca0266e:::benchmark_data_smithi122_119671_object13129:head# 2024-03-20T09:04:50.060 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bca07155:::benchmark_data_smithi122_119671_object47040:head# 2024-03-20T09:04:50.061 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bca14ab0:::benchmark_data_smithi122_119671_object27877:head# 2024-03-20T09:04:50.062 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bca1b015:::benchmark_data_smithi122_119671_object43204:head# 2024-03-20T09:04:50.062 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bca3537f:::benchmark_data_smithi122_119671_object75436:head# 2024-03-20T09:04:50.063 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bca4a292:::benchmark_data_smithi122_119671_object69181:head# 2024-03-20T09:04:50.063 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bca4a78c:::benchmark_data_smithi122_119671_object42240:head# 2024-03-20T09:04:50.063 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bca55329:::benchmark_data_smithi122_119671_object13439:head# 2024-03-20T09:04:50.063 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bca55e75:::benchmark_data_smithi122_119671_object54046:head# 2024-03-20T09:04:50.064 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bca61629:::benchmark_data_smithi122_119671_object29344:head# 2024-03-20T09:04:50.065 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bca61a97:::benchmark_data_smithi122_119671_object72592:head# 2024-03-20T09:04:50.065 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bca62b66:::benchmark_data_smithi122_119671_object66878:head# 2024-03-20T09:04:50.066 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bca65e81:::benchmark_data_smithi122_119671_object73624:head# 2024-03-20T09:04:50.066 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bca7b70f:::benchmark_data_smithi122_119671_object63245:head# 2024-03-20T09:04:50.067 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bca8fe7c:::benchmark_data_smithi122_119671_object26760:head# 2024-03-20T09:04:50.067 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bca9cd4f:::benchmark_data_smithi122_119671_object49068:head# 2024-03-20T09:04:50.068 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcaadeab:::benchmark_data_smithi122_119671_object78674:head# 2024-03-20T09:04:50.068 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcab5d2a:::benchmark_data_smithi122_119671_object6481:head# 2024-03-20T09:04:50.068 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcad409e:::benchmark_data_smithi122_119671_object32814:head# 2024-03-20T09:04:50.069 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcaf93d4:::benchmark_data_smithi122_119671_object11320:head# 2024-03-20T09:04:50.069 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcaf9b3f:::benchmark_data_smithi122_119671_object41517:head# 2024-03-20T09:04:50.070 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcb00b48:::benchmark_data_smithi122_119671_object74307:head# 2024-03-20T09:04:50.070 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcb1f794:::benchmark_data_smithi122_119671_object75487:head# 2024-03-20T09:04:50.071 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcb30cb9:::benchmark_data_smithi122_119671_object59088:head# 2024-03-20T09:04:50.071 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcb5b610:::benchmark_data_smithi122_119671_object87699:head# 2024-03-20T09:04:50.072 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcb5bd62:::benchmark_data_smithi122_119671_object95230:head# 2024-03-20T09:04:50.072 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcb607d6:::benchmark_data_smithi122_119671_object9191:head# 2024-03-20T09:04:50.073 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcb65d9f:::benchmark_data_smithi122_119671_object48450:head# 2024-03-20T09:04:50.073 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcb6eeef:::benchmark_data_smithi122_119671_object36183:head# 2024-03-20T09:04:50.073 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcb77b7c:::benchmark_data_smithi122_119671_object1686:head# 2024-03-20T09:04:50.074 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcb8095f:::benchmark_data_smithi122_119671_object54718:head# 2024-03-20T09:04:50.074 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcb851dd:::benchmark_data_smithi122_119671_object19988:head# 2024-03-20T09:04:50.075 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcb8b4d2:::benchmark_data_smithi122_119671_object1551:head# 2024-03-20T09:04:50.078 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcb8faf8:::benchmark_data_smithi122_119671_object78582:head# 2024-03-20T09:04:50.078 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcb962fe:::benchmark_data_smithi122_119671_object79203:head# 2024-03-20T09:04:50.078 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcb9d418:::benchmark_data_smithi122_119671_object27236:head# 2024-03-20T09:04:50.078 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcba2355:::benchmark_data_smithi122_119671_object70144:head# 2024-03-20T09:04:50.078 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcbab49d:::benchmark_data_smithi122_119671_object55587:head# 2024-03-20T09:04:50.078 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcbb197d:::benchmark_data_smithi122_119671_object95239:head# 2024-03-20T09:04:50.078 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcbbf3b3:::benchmark_data_smithi122_119671_object97915:head# 2024-03-20T09:04:50.081 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcbe192f:::benchmark_data_smithi122_119671_object82368:head# 2024-03-20T09:04:50.081 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcbf11f2:::benchmark_data_smithi122_119671_object62609:head# 2024-03-20T09:04:50.082 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcbf58aa:::benchmark_data_smithi122_119671_object74064:head# 2024-03-20T09:04:50.082 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcbf9cd8:::benchmark_data_smithi122_119671_object56259:head# 2024-03-20T09:04:50.082 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcc0deec:::benchmark_data_smithi122_119671_object38348:head# 2024-03-20T09:04:50.082 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcc274b8:::benchmark_data_smithi122_119671_object12854:head# 2024-03-20T09:04:50.083 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcc3a4b5:::benchmark_data_smithi122_119671_object44218:head# 2024-03-20T09:04:50.084 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcc3ca2e:::benchmark_data_smithi122_119671_object17559:head# 2024-03-20T09:04:50.084 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcc3d49d:::benchmark_data_smithi122_119671_object49261:head# 2024-03-20T09:04:50.084 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcc3ded1:::benchmark_data_smithi122_119671_object99167:head# 2024-03-20T09:04:50.085 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcc4d284:::benchmark_data_smithi122_119671_object81566:head# 2024-03-20T09:04:50.085 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcc64c25:::benchmark_data_smithi122_119671_object45622:head# 2024-03-20T09:04:50.086 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcc99a85:::benchmark_data_smithi122_119671_object75066:head# 2024-03-20T09:04:50.086 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcca4899:::benchmark_data_smithi122_119671_object81079:head# 2024-03-20T09:04:50.087 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcca9741:::benchmark_data_smithi122_119671_object69129:head# 2024-03-20T09:04:50.087 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bccb3fb2:::benchmark_data_smithi122_119671_object26187:head# 2024-03-20T09:04:50.087 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bccd0064:::benchmark_data_smithi122_119671_object64582:head# 2024-03-20T09:04:50.088 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bccd0b79:::benchmark_data_smithi122_119671_object94070:head# 2024-03-20T09:04:50.088 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bccdbe57:::benchmark_data_smithi122_119671_object5746:head# 2024-03-20T09:04:50.089 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bccdde08:::benchmark_data_smithi122_119671_object29866:head# 2024-03-20T09:04:50.089 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcce62db:::benchmark_data_smithi122_119671_object5983:head# 2024-03-20T09:04:50.089 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcd0184b:::benchmark_data_smithi122_119671_object51032:head# 2024-03-20T09:04:50.090 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcd04981:::benchmark_data_smithi122_119671_object73784:head# 2024-03-20T09:04:50.090 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcd0be0d:::benchmark_data_smithi122_119671_object1946:head# 2024-03-20T09:04:50.091 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcd1f18d:::benchmark_data_smithi122_119671_object47900:head# 2024-03-20T09:04:50.091 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcd391b2:::benchmark_data_smithi122_119671_object6145:head# 2024-03-20T09:04:50.092 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcd3f8c0:::benchmark_data_smithi122_119671_object83704:head# 2024-03-20T09:04:50.092 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcd488cf:::benchmark_data_smithi122_119671_object93409:head# 2024-03-20T09:04:50.093 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcd527d9:::benchmark_data_smithi122_119671_object88400:head# 2024-03-20T09:04:50.093 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcd746c9:::benchmark_data_smithi122_119671_object77821:head# 2024-03-20T09:04:50.093 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcd7fbc1:::benchmark_data_smithi122_119671_object92101:head# 2024-03-20T09:04:50.094 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcd9f19a:::benchmark_data_smithi122_119671_object25689:head# 2024-03-20T09:04:50.094 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcda56d6:::benchmark_data_smithi122_119671_object39143:head# 2024-03-20T09:04:50.095 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcdae7ed:::benchmark_data_smithi122_119671_object38106:head# 2024-03-20T09:04:50.095 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcdaeee7:::benchmark_data_smithi122_119671_object57291:head# 2024-03-20T09:04:50.095 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcdbe3fb:::benchmark_data_smithi122_119671_object54905:head# 2024-03-20T09:04:50.096 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcdc5aff:::benchmark_data_smithi122_119671_object61422:head# 2024-03-20T09:04:50.097 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcdd0718:::benchmark_data_smithi122_119671_object22174:head# 2024-03-20T09:04:50.097 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcdd1d7c:::benchmark_data_smithi122_119671_object69449:head# 2024-03-20T09:04:50.097 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bce009a1:::benchmark_data_smithi122_119671_object58105:head# 2024-03-20T09:04:50.098 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bce0e9a2:::benchmark_data_smithi122_119671_object50723:head# 2024-03-20T09:04:50.098 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bce27829:::benchmark_data_smithi122_119671_object39796:head# 2024-03-20T09:04:50.099 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bce30fec:::benchmark_data_smithi122_119671_object28517:head# 2024-03-20T09:04:50.099 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bce4277c:::benchmark_data_smithi122_119671_object67217:head# 2024-03-20T09:04:50.100 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bce49c1c:::benchmark_data_smithi122_119671_object45854:head# 2024-03-20T09:04:50.100 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bce4e76f:::benchmark_data_smithi122_119671_object11418:head# 2024-03-20T09:04:50.100 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bce531ea:::benchmark_data_smithi122_119671_object44807:head# 2024-03-20T09:04:50.101 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bce58b87:::benchmark_data_smithi122_119671_object24888:head# 2024-03-20T09:04:50.101 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bce5b795:::benchmark_data_smithi122_119671_object73828:head# 2024-03-20T09:04:50.103 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bce64fe5:::benchmark_data_smithi122_119671_object15549:head# 2024-03-20T09:04:50.104 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bce684f3:::benchmark_data_smithi122_119671_object80552:head# 2024-03-20T09:04:50.104 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bce7a426:::benchmark_data_smithi122_119671_object67786:head# 2024-03-20T09:04:50.105 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bce7cbc9:::benchmark_data_smithi122_119671_object92223:head# 2024-03-20T09:04:50.105 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bce87abd:::benchmark_data_smithi122_119671_object9884:head# 2024-03-20T09:04:50.106 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bce87d60:::benchmark_data_smithi122_119671_object89349:head# 2024-03-20T09:04:50.106 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bce8f4c1:::benchmark_data_smithi122_119671_object76275:head# 2024-03-20T09:04:50.106 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bce9103d:::benchmark_data_smithi122_119671_object14622:head# 2024-03-20T09:04:50.107 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcea1f89:::benchmark_data_smithi122_119671_object21590:head# 2024-03-20T09:04:50.108 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcecb46e:::benchmark_data_smithi122_119671_object25532:head# 2024-03-20T09:04:50.108 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bced6726:::benchmark_data_smithi122_119671_object71967:head# 2024-03-20T09:04:50.108 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bceefc2e:::benchmark_data_smithi122_119671_object88320:head# 2024-03-20T09:04:50.109 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcef6c07:::benchmark_data_smithi122_119671_object63708:head# 2024-03-20T09:04:50.109 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcef74ce:::benchmark_data_smithi122_119671_object76467:head# 2024-03-20T09:04:50.110 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcf19f94:::benchmark_data_smithi122_119671_object75127:head# 2024-03-20T09:04:50.110 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcf1d9cb:::benchmark_data_smithi122_119671_object95511:head# 2024-03-20T09:04:50.110 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcf219fc:::benchmark_data_smithi122_119671_object43352:head# 2024-03-20T09:04:50.111 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcf2373b:::benchmark_data_smithi122_119671_object9391:head# 2024-03-20T09:04:50.111 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcf360bb:::benchmark_data_smithi122_119671_object98540:head# 2024-03-20T09:04:50.112 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcf44b8b:::benchmark_data_smithi122_119671_object99001:head# 2024-03-20T09:04:50.112 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcf45e7f:::benchmark_data_smithi122_119671_object63490:head# 2024-03-20T09:04:50.112 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcf50e97:::benchmark_data_smithi122_119671_object43690:head# 2024-03-20T09:04:50.114 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcf54eff:::benchmark_data_smithi122_119671_object28250:head# 2024-03-20T09:04:50.115 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcf59252:::benchmark_data_smithi122_119671_object23677:head# 2024-03-20T09:04:50.115 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcf5c077:::benchmark_data_smithi122_119671_object32128:head# 2024-03-20T09:04:50.116 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcf5d7ff:::benchmark_data_smithi122_119671_object58331:head# 2024-03-20T09:04:50.116 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcf70525:::benchmark_data_smithi122_119671_object36313:head# 2024-03-20T09:04:50.117 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcf7142a:::benchmark_data_smithi122_119671_object23673:head# 2024-03-20T09:04:50.117 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcf7d3be:::benchmark_data_smithi122_119671_object74867:head# 2024-03-20T09:04:50.118 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcf7df71:::benchmark_data_smithi122_119671_object73162:head# 2024-03-20T09:04:50.118 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcf833ef:::benchmark_data_smithi122_119671_object59698:head# 2024-03-20T09:04:50.118 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcf83da4:::benchmark_data_smithi122_119671_object5878:head# 2024-03-20T09:04:50.119 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcf9238f:::benchmark_data_smithi122_119671_object14939:head# 2024-03-20T09:04:50.119 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcf94fac:::benchmark_data_smithi122_119671_object53820:head# 2024-03-20T09:04:50.120 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcf9d9fe:::benchmark_data_smithi122_119671_object95679:head# 2024-03-20T09:04:50.120 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcf9e106:::benchmark_data_smithi122_119671_object52508:head# 2024-03-20T09:04:50.122 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcfa0a38:::benchmark_data_smithi122_119671_object60577:head# 2024-03-20T09:04:50.123 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcfe49fa:::benchmark_data_smithi122_119671_object45549:head# 2024-03-20T09:04:50.123 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bcff34e6:::benchmark_data_smithi122_119671_object93125:head# 2024-03-20T09:04:50.124 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd000f91:::benchmark_data_smithi122_119671_object30578:head# 2024-03-20T09:04:50.124 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd02ef06:::benchmark_data_smithi122_119671_object7351:head# 2024-03-20T09:04:50.125 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd03610e:::benchmark_data_smithi122_119671_object11733:head# 2024-03-20T09:04:50.127 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd040e66:::benchmark_data_smithi122_119671_object75084:head# 2024-03-20T09:04:50.127 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd06d9ab:::benchmark_data_smithi122_119671_object73950:head# 2024-03-20T09:04:50.128 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd071eba:::benchmark_data_smithi122_119671_object57381:head# 2024-03-20T09:04:50.128 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd07882c:::benchmark_data_smithi122_119671_object62643:head# 2024-03-20T09:04:50.129 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd07b187:::benchmark_data_smithi122_119671_object34500:head# 2024-03-20T09:04:50.129 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd07f3d5:::benchmark_data_smithi122_119671_object25917:head# 2024-03-20T09:04:50.129 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd081180:::benchmark_data_smithi122_119671_object1645:head# 2024-03-20T09:04:50.130 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd09652b:::benchmark_data_smithi122_119671_object97849:head# 2024-03-20T09:04:50.130 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd099007:::benchmark_data_smithi122_119671_object41524:head# 2024-03-20T09:04:50.131 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd0b4087:::benchmark_data_smithi122_119671_object82294:head# 2024-03-20T09:04:50.131 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd0be456:::benchmark_data_smithi122_119671_object45535:head# 2024-03-20T09:04:50.131 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd0cef23:::benchmark_data_smithi122_119671_object95963:head# 2024-03-20T09:04:50.132 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd0d6429:::benchmark_data_smithi122_119671_object94373:head# 2024-03-20T09:04:50.132 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd0e30af:::benchmark_data_smithi122_119671_object43489:head# 2024-03-20T09:04:50.133 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd0ee820:::benchmark_data_smithi122_119671_object57208:head# 2024-03-20T09:04:50.133 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd0f5d3b:::benchmark_data_smithi122_119671_object60417:head# 2024-03-20T09:04:50.133 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd0f91f1:::benchmark_data_smithi122_119671_object10425:head# 2024-03-20T09:04:50.134 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd103b85:::benchmark_data_smithi122_119671_object79862:head# 2024-03-20T09:04:50.134 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd1090aa:::benchmark_data_smithi122_119671_object65150:head# 2024-03-20T09:04:50.135 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd10c91f:::benchmark_data_smithi122_119671_object80561:head# 2024-03-20T09:04:50.135 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd117b57:::benchmark_data_smithi122_119671_object83261:head# 2024-03-20T09:04:50.136 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd119a5c:::benchmark_data_smithi122_119671_object38344:head# 2024-03-20T09:04:50.136 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd13c2b6:::benchmark_data_smithi122_119671_object79987:head# 2024-03-20T09:04:50.137 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd13f2e1:::benchmark_data_smithi122_119671_object62503:head# 2024-03-20T09:04:50.137 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd1431e7:::benchmark_data_smithi122_119671_object51405:head# 2024-03-20T09:04:50.137 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd143238:::benchmark_data_smithi122_119671_object37670:head# 2024-03-20T09:04:50.138 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd14691a:::benchmark_data_smithi122_119671_object61433:head# 2024-03-20T09:04:50.138 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd14b614:::benchmark_data_smithi122_119671_object47951:head# 2024-03-20T09:04:50.141 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd150d4c:::benchmark_data_smithi122_119671_object95177:head# 2024-03-20T09:04:50.142 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd163750:::benchmark_data_smithi122_119671_object85622:head# 2024-03-20T09:04:50.142 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd1753d3:::benchmark_data_smithi122_119671_object59215:head# 2024-03-20T09:04:50.142 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd1770e2:::benchmark_data_smithi122_119671_object31058:head# 2024-03-20T09:04:50.143 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd17916b:::benchmark_data_smithi122_119671_object26049:head# 2024-03-20T09:04:50.143 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd17a85e:::benchmark_data_smithi122_119671_object21172:head# 2024-03-20T09:04:50.143 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:49 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/19587992' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:04:50.144 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:49 smithi026 ceph-mon[68309]: pgmap v731: 47 pgs: 1 active+clean+remapped, 1 activating+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 29 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 6.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 28606/294816 objects degraded (9.703%); 12685/294816 objects misplaced (4.303%); 25 MiB/s, 410 objects/s recovering 2024-03-20T09:04:50.144 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:49 smithi026 ceph-mon[68309]: 3.12 scrub starts 2024-03-20T09:04:50.144 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd18961d:::benchmark_data_smithi122_119671_object4796:head# 2024-03-20T09:04:50.144 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd190652:::benchmark_data_smithi122_119671_object99343:head# 2024-03-20T09:04:50.144 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd1913e7:::benchmark_data_smithi122_119671_object27405:head# 2024-03-20T09:04:50.145 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd19b601:::benchmark_data_smithi122_119671_object50677:head# 2024-03-20T09:04:50.145 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd19d0e8:::benchmark_data_smithi122_119671_object84946:head# 2024-03-20T09:04:50.145 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd1a1b5f:::benchmark_data_smithi122_119671_object84497:head# 2024-03-20T09:04:50.146 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd1ad707:::benchmark_data_smithi122_119671_object50319:head# 2024-03-20T09:04:50.146 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd1b2e2e:::benchmark_data_smithi122_119671_object73602:head# 2024-03-20T09:04:50.146 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd1c0db0:::benchmark_data_smithi122_119671_object50323:head# 2024-03-20T09:04:50.147 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd1d19ae:::benchmark_data_smithi122_119671_object11229:head# 2024-03-20T09:04:50.147 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd1de2b1:::benchmark_data_smithi122_119671_object51419:head# 2024-03-20T09:04:50.148 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd1e304e:::benchmark_data_smithi122_119671_object96406:head# 2024-03-20T09:04:50.148 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd1f3aef:::benchmark_data_smithi122_119671_object73042:head# 2024-03-20T09:04:50.149 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd1f4efc:::benchmark_data_smithi122_119671_object78146:head# 2024-03-20T09:04:50.149 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd2052a8:::benchmark_data_smithi122_119671_object14206:head# 2024-03-20T09:04:50.149 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd20dd73:::benchmark_data_smithi122_119671_object76583:head# 2024-03-20T09:04:50.149 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd20f2a4:::benchmark_data_smithi122_119671_object23560:head# 2024-03-20T09:04:50.150 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd20f315:::benchmark_data_smithi122_119671_object83629:head# 2024-03-20T09:04:50.150 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd2113b0:::benchmark_data_smithi122_119671_object17506:head# 2024-03-20T09:04:50.151 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd22bc48:::benchmark_data_smithi122_119671_object74047:head# 2024-03-20T09:04:50.151 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd25a771:::benchmark_data_smithi122_119671_object9673:head# 2024-03-20T09:04:50.152 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd2670f9:::benchmark_data_smithi122_119671_object68531:head# 2024-03-20T09:04:50.152 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd26e127:::benchmark_data_smithi122_119671_object4421:head# 2024-03-20T09:04:50.153 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd26f86d:::benchmark_data_smithi122_119671_object19527:head# 2024-03-20T09:04:50.153 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd27a723:::benchmark_data_smithi122_119671_object49494:head# 2024-03-20T09:04:50.154 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd282d09:::benchmark_data_smithi122_119671_object91392:head# 2024-03-20T09:04:50.154 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd289a33:::benchmark_data_smithi122_119671_object35144:head# 2024-03-20T09:04:50.155 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd28bc41:::benchmark_data_smithi122_119671_object45629:head# 2024-03-20T09:04:50.155 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd28ecd1:::benchmark_data_smithi122_119671_object91484:head# 2024-03-20T09:04:50.155 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd29a776:::benchmark_data_smithi122_119671_object56559:head# 2024-03-20T09:04:50.156 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd2a6a20:::benchmark_data_smithi122_119671_object53381:head# 2024-03-20T09:04:50.156 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd2a9d17:::benchmark_data_smithi122_119671_object26873:head# 2024-03-20T09:04:50.156 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd2ab425:::benchmark_data_smithi122_119671_object71513:head# 2024-03-20T09:04:50.157 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd2b42dc:::benchmark_data_smithi122_119671_object46323:head# 2024-03-20T09:04:50.157 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd2cfc63:::benchmark_data_smithi122_119671_object12575:head# 2024-03-20T09:04:50.158 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd2d0772:::benchmark_data_smithi122_119671_object85294:head# 2024-03-20T09:04:50.158 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd2d4786:::benchmark_data_smithi122_119671_object22103:head# 2024-03-20T09:04:50.162 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd2f29b0:::benchmark_data_smithi122_119671_object94999:head# 2024-03-20T09:04:50.162 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd2f44c5:::benchmark_data_smithi122_119671_object40982:head# 2024-03-20T09:04:50.162 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd2f970e:::benchmark_data_smithi122_119671_object39154:head# 2024-03-20T09:04:50.162 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd30c66c:::benchmark_data_smithi122_119671_object92932:head# 2024-03-20T09:04:50.163 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd3100de:::benchmark_data_smithi122_119671_object14005:head# 2024-03-20T09:04:50.163 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd312f97:::benchmark_data_smithi122_119671_object85476:head# 2024-03-20T09:04:50.163 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd314917:::benchmark_data_smithi122_119671_object68180:head# 2024-03-20T09:04:50.163 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd31c5ad:::benchmark_data_smithi122_119671_object16798:head# 2024-03-20T09:04:50.163 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd321f12:::benchmark_data_smithi122_119671_object78074:head# 2024-03-20T09:04:50.163 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd326ab3:::benchmark_data_smithi122_119671_object66251:head# 2024-03-20T09:04:50.164 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd32a828:::benchmark_data_smithi122_119671_object63075:head# 2024-03-20T09:04:50.164 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd32c192:::benchmark_data_smithi122_119671_object90450:head# 2024-03-20T09:04:50.164 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd338739:::benchmark_data_smithi122_119671_object78083:head# 2024-03-20T09:04:50.164 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd33c081:::benchmark_data_smithi122_119671_object77421:head# 2024-03-20T09:04:50.164 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd353517:::benchmark_data_smithi122_119671_object27843:head# 2024-03-20T09:04:50.165 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd35845c:::benchmark_data_smithi122_119671_object76842:head# 2024-03-20T09:04:50.166 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd3586df:::benchmark_data_smithi122_119671_object43899:head# 2024-03-20T09:04:50.166 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd36e017:::benchmark_data_smithi122_119671_object71298:head# 2024-03-20T09:04:50.167 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd379a5a:::benchmark_data_smithi122_119671_object18984:head# 2024-03-20T09:04:50.167 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd38308e:::benchmark_data_smithi122_119671_object60926:head# 2024-03-20T09:04:50.167 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd386940:::benchmark_data_smithi122_119671_object70472:head# 2024-03-20T09:04:50.168 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd3aa8ce:::benchmark_data_smithi122_119671_object76211:head# 2024-03-20T09:04:50.168 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd3afd00:::benchmark_data_smithi122_119671_object43058:head# 2024-03-20T09:04:50.169 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd3b44cf:::benchmark_data_smithi122_119671_object12265:head# 2024-03-20T09:04:50.169 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd3dcdb6:::benchmark_data_smithi122_119671_object95769:head# 2024-03-20T09:04:50.170 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd3eefc2:::benchmark_data_smithi122_119671_object42499:head# 2024-03-20T09:04:50.170 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:04:50.170 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd3efd65:::benchmark_data_smithi122_119671_object86152:head# 2024-03-20T09:04:50.171 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd3f0395:::benchmark_data_smithi122_119671_object38498:head# 2024-03-20T09:04:50.171 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd3fbf56:::benchmark_data_smithi122_119671_object91403:head# 2024-03-20T09:04:50.171 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd400a9e:::benchmark_data_smithi122_119671_object11507:head# 2024-03-20T09:04:50.172 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd403b54:::benchmark_data_smithi122_119671_object68984:head# 2024-03-20T09:04:50.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:50.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:04:50.173 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd40474a:::benchmark_data_smithi122_119671_object77772:head# 2024-03-20T09:04:50.173 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd4055b0:::benchmark_data_smithi122_119671_object73849:head# 2024-03-20T09:04:50.174 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd4199c7:::benchmark_data_smithi122_119671_object9915:head# 2024-03-20T09:04:50.175 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd434343:::benchmark_data_smithi122_119671_object71357:head# 2024-03-20T09:04:50.175 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd439729:::benchmark_data_smithi122_119671_object38723:head# 2024-03-20T09:04:50.176 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd44b992:::benchmark_data_smithi122_119671_object78274:head# 2024-03-20T09:04:50.176 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd4591ab:::benchmark_data_smithi122_119671_object60858:head# 2024-03-20T09:04:50.177 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd469213:::benchmark_data_smithi122_119671_object23133:head# 2024-03-20T09:04:50.177 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd49482d:::benchmark_data_smithi122_119671_object76968:head# 2024-03-20T09:04:50.177 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd49bacd:::benchmark_data_smithi122_119671_object39630:head# 2024-03-20T09:04:50.181 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd49c358:::benchmark_data_smithi122_119671_object94937:head# 2024-03-20T09:04:50.181 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd4a9d21:::benchmark_data_smithi122_119671_object17398:head# 2024-03-20T09:04:50.181 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd4b8c44:::benchmark_data_smithi122_119671_object64205:head# 2024-03-20T09:04:50.181 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd4b8f5b:::benchmark_data_smithi122_119671_object94162:head# 2024-03-20T09:04:50.181 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd4bb488:::benchmark_data_smithi122_119671_object30462:head# 2024-03-20T09:04:50.181 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd4c1953:::benchmark_data_smithi122_119671_object95970:head# 2024-03-20T09:04:50.181 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd4c4b53:::benchmark_data_smithi122_119671_object52154:head# 2024-03-20T09:04:50.182 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd4dd027:::benchmark_data_smithi122_119671_object92759:head# 2024-03-20T09:04:50.182 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd4e648d:::benchmark_data_smithi122_119671_object48094:head# 2024-03-20T09:04:50.182 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd4f6849:::benchmark_data_smithi122_119671_object75771:head# 2024-03-20T09:04:50.184 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd4f9ef3:::benchmark_data_smithi122_119671_object65626:head# 2024-03-20T09:04:50.184 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd511878:::benchmark_data_smithi122_119671_object55575:head# 2024-03-20T09:04:50.185 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd51384b:::benchmark_data_smithi122_119671_object56227:head# 2024-03-20T09:04:50.185 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd516235:::benchmark_data_smithi122_119671_object92813:head# 2024-03-20T09:04:50.185 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd51c98a:::benchmark_data_smithi122_119671_object75019:head# 2024-03-20T09:04:50.188 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd52df2b:::benchmark_data_smithi122_119671_object18885:head# 2024-03-20T09:04:50.189 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd53d48a:::benchmark_data_smithi122_119671_object6643:head# 2024-03-20T09:04:50.189 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd53f69e:::benchmark_data_smithi122_119671_object76718:head# 2024-03-20T09:04:50.189 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd540af5:::benchmark_data_smithi122_119671_object53122:head# 2024-03-20T09:04:50.189 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd541fdf:::benchmark_data_smithi122_119671_object80997:head# 2024-03-20T09:04:50.190 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd548739:::benchmark_data_smithi122_119671_object73178:head# 2024-03-20T09:04:50.190 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd54b0a7:::benchmark_data_smithi122_119671_object84295:head# 2024-03-20T09:04:50.190 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd54c655:::benchmark_data_smithi122_119671_object59305:head# 2024-03-20T09:04:50.191 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd550590:::benchmark_data_smithi122_119671_object34010:head# 2024-03-20T09:04:50.191 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd559043:::benchmark_data_smithi122_119671_object81762:head# 2024-03-20T09:04:50.191 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd5616e9:::benchmark_data_smithi122_119671_object71134:head# 2024-03-20T09:04:50.192 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd569ed4:::benchmark_data_smithi122_119671_object52745:head# 2024-03-20T09:04:50.192 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd56a63d:::benchmark_data_smithi122_119671_object65095:head# 2024-03-20T09:04:50.193 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd56feac:::benchmark_data_smithi122_119671_object74102:head# 2024-03-20T09:04:50.193 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd5831ba:::benchmark_data_smithi122_119671_object20214:head# 2024-03-20T09:04:50.193 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd58d960:::benchmark_data_smithi122_119671_object79897:head# 2024-03-20T09:04:50.194 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd59b7a3:::benchmark_data_smithi122_119671_object38740:head# 2024-03-20T09:04:50.194 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd59c7ca:::benchmark_data_smithi122_119671_object80606:head# 2024-03-20T09:04:50.195 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd5a30f0:::benchmark_data_smithi122_119671_object18253:head# 2024-03-20T09:04:50.195 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd5aa794:::benchmark_data_smithi122_119671_object36181:head# 2024-03-20T09:04:50.195 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd5cad3a:::benchmark_data_smithi122_119671_object78207:head# 2024-03-20T09:04:50.196 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd5d39e1:::benchmark_data_smithi122_119671_object92537:head# 2024-03-20T09:04:50.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:49 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/19587992' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:04:50.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:49 smithi122 ceph-mon[73495]: pgmap v731: 47 pgs: 1 active+clean+remapped, 1 activating+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 29 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 6.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 28606/294816 objects degraded (9.703%); 12685/294816 objects misplaced (4.303%); 25 MiB/s, 410 objects/s recovering 2024-03-20T09:04:50.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:49 smithi122 ceph-mon[73495]: 3.12 scrub starts 2024-03-20T09:04:50.197 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd5d82dc:::benchmark_data_smithi122_119671_object98334:head# 2024-03-20T09:04:50.197 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd5dade4:::benchmark_data_smithi122_119671_object60097:head# 2024-03-20T09:04:50.197 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd5e9150:::benchmark_data_smithi122_119671_object48905:head# 2024-03-20T09:04:50.198 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd5f7c01:::benchmark_data_smithi122_119671_object56869:head# 2024-03-20T09:04:50.198 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd6273f3:::benchmark_data_smithi122_119671_object7982:head# 2024-03-20T09:04:50.198 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd62ceac:::benchmark_data_smithi122_119671_object81372:head# 2024-03-20T09:04:50.199 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd644841:::benchmark_data_smithi122_119671_object62181:head# 2024-03-20T09:04:50.199 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd646181:::benchmark_data_smithi122_119671_object33783:head# 2024-03-20T09:04:50.200 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd64f595:::benchmark_data_smithi122_119671_object50713:head# 2024-03-20T09:04:50.200 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd652b67:::benchmark_data_smithi122_119671_object88938:head# 2024-03-20T09:04:50.201 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd6598ba:::benchmark_data_smithi122_119671_object43400:head# 2024-03-20T09:04:50.201 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd667c76:::benchmark_data_smithi122_119671_object20599:head# 2024-03-20T09:04:50.201 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd67b8bb:::benchmark_data_smithi122_119671_object7526:head# 2024-03-20T09:04:50.202 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd69a5e1:::benchmark_data_smithi122_119671_object23659:head# 2024-03-20T09:04:50.202 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd6a6a72:::benchmark_data_smithi122_119671_object4585:head# 2024-03-20T09:04:50.202 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd6a8f86:::benchmark_data_smithi122_119671_object27130:head# 2024-03-20T09:04:50.203 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd6b94cd:::benchmark_data_smithi122_119671_object93344:head# 2024-03-20T09:04:50.203 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd6bb468:::benchmark_data_smithi122_119671_object8283:head# 2024-03-20T09:04:50.204 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd6c19d0:::benchmark_data_smithi122_119671_object64735:head# 2024-03-20T09:04:50.204 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd6c1ce3:::benchmark_data_smithi122_119671_object71256:head# 2024-03-20T09:04:50.205 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd6c9acc:::benchmark_data_smithi122_119671_object35485:head# 2024-03-20T09:04:50.205 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd6cc160:::benchmark_data_smithi122_119671_object20819:head# 2024-03-20T09:04:50.205 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd6d6b2d:::benchmark_data_smithi122_119671_object24238:head# 2024-03-20T09:04:50.206 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd6d91d3:::benchmark_data_smithi122_119671_object87668:head# 2024-03-20T09:04:50.206 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd6df201:::benchmark_data_smithi122_119671_object66567:head# 2024-03-20T09:04:50.206 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd6e9a6e:::benchmark_data_smithi122_119671_object10318:head# 2024-03-20T09:04:50.207 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd6f3e15:::benchmark_data_smithi122_119671_object40232:head# 2024-03-20T09:04:50.207 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd6f810b:::benchmark_data_smithi122_119671_object78232:head# 2024-03-20T09:04:50.208 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd700ce2:::benchmark_data_smithi122_119671_object31794:head# 2024-03-20T09:04:50.208 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd7019bb:::benchmark_data_smithi122_119671_object95297:head# 2024-03-20T09:04:50.293 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:04:50.322 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd709e20:::benchmark_data_smithi122_119671_object22996:head# 2024-03-20T09:04:50.322 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd70a008:::benchmark_data_smithi122_119671_object86280:head# 2024-03-20T09:04:50.322 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd70f320:::benchmark_data_smithi122_119671_object78577:head# 2024-03-20T09:04:50.322 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd711001:::benchmark_data_smithi122_119671_object56336:head# 2024-03-20T09:04:50.322 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd7286e7:::benchmark_data_smithi122_119671_object80331:head# 2024-03-20T09:04:50.323 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd72da8f:::benchmark_data_smithi122_119671_object35326:head# 2024-03-20T09:04:50.323 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd74c281:::benchmark_data_smithi122_119671_object60518:head# 2024-03-20T09:04:50.323 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd74e666:::benchmark_data_smithi122_119671_object32765:head# 2024-03-20T09:04:50.323 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd755700:::benchmark_data_smithi122_119671_object53127:head# 2024-03-20T09:04:50.323 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd75c8a6:::benchmark_data_smithi122_119671_object36984:head# 2024-03-20T09:04:50.323 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd7685ea:::benchmark_data_smithi122_119671_object17243:head# 2024-03-20T09:04:50.323 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd76cff2:::benchmark_data_smithi122_119671_object98198:head# 2024-03-20T09:04:50.323 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd777cb4:::benchmark_data_smithi122_119671_object94050:head# 2024-03-20T09:04:50.323 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd7b3e03:::benchmark_data_smithi122_119671_object61231:head# 2024-03-20T09:04:50.323 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd7c8086:::benchmark_data_smithi122_119671_object6693:head# 2024-03-20T09:04:50.323 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd7ce254:::benchmark_data_smithi122_119671_object42750:head# 2024-03-20T09:04:50.323 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd7d8a01:::benchmark_data_smithi122_119671_object37645:head# 2024-03-20T09:04:50.323 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd7dee82:::benchmark_data_smithi122_119671_object53854:head# 2024-03-20T09:04:50.324 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd7df91f:::benchmark_data_smithi122_119671_object75152:head# 2024-03-20T09:04:50.324 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd7e7824:::benchmark_data_smithi122_119671_object41116:head# 2024-03-20T09:04:50.324 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd7f1611:::benchmark_data_smithi122_119671_object92237:head# 2024-03-20T09:04:50.324 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd7f81c0:::benchmark_data_smithi122_119671_object67491:head# 2024-03-20T09:04:50.324 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd801101:::benchmark_data_smithi122_119671_object74399:head# 2024-03-20T09:04:50.324 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd805f22:::benchmark_data_smithi122_119671_object21759:head# 2024-03-20T09:04:50.324 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd80ef14:::benchmark_data_smithi122_119671_object71941:head# 2024-03-20T09:04:50.324 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd817b11:::benchmark_data_smithi122_119671_object62421:head# 2024-03-20T09:04:50.324 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd8210ed:::benchmark_data_smithi122_119671_object80119:head# 2024-03-20T09:04:50.324 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd8288be:::benchmark_data_smithi122_119671_object82173:head# 2024-03-20T09:04:50.324 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd846a89:::benchmark_data_smithi122_119671_object78736:head# 2024-03-20T09:04:50.325 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd84d53d:::benchmark_data_smithi122_119671_object25470:head# 2024-03-20T09:04:50.325 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd84f5b1:::benchmark_data_smithi122_119671_object58720:head# 2024-03-20T09:04:50.325 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd85463e:::benchmark_data_smithi122_119671_object7820:head# 2024-03-20T09:04:50.325 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd857fc8:::benchmark_data_smithi122_119671_object26690:head# 2024-03-20T09:04:50.325 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd866398:::benchmark_data_smithi122_119671_object33480:head# 2024-03-20T09:04:50.325 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd86bd38:::benchmark_data_smithi122_119671_object22137:head# 2024-03-20T09:04:50.325 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd870f7e:::benchmark_data_smithi122_119671_object48865:head# 2024-03-20T09:04:50.325 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd871336:::benchmark_data_smithi122_119671_object59133:head# 2024-03-20T09:04:50.325 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd8760a4:::benchmark_data_smithi122_119671_object6521:head# 2024-03-20T09:04:50.325 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd88ad20:::benchmark_data_smithi122_119671_object5799:head# 2024-03-20T09:04:50.325 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd89c056:::benchmark_data_smithi122_119671_object73279:head# 2024-03-20T09:04:50.325 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd8a91da:::benchmark_data_smithi122_119671_object92692:head# 2024-03-20T09:04:50.325 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd8b4c67:::benchmark_data_smithi122_119671_object25139:head# 2024-03-20T09:04:50.326 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd8ca23d:::benchmark_data_smithi122_119671_object27775:head# 2024-03-20T09:04:50.326 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd8ce1ec:::benchmark_data_smithi122_119671_object76766:head# 2024-03-20T09:04:50.326 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd8dbe5b:::benchmark_data_smithi122_119671_object36613:head# 2024-03-20T09:04:50.326 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd8dded4:::benchmark_data_smithi122_119671_object49779:head# 2024-03-20T09:04:50.326 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd90bb06:::benchmark_data_smithi122_119671_object57742:head# 2024-03-20T09:04:50.326 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd92c3d6:::benchmark_data_smithi122_119671_object6356:head# 2024-03-20T09:04:50.326 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd931ddf:::benchmark_data_smithi122_119671_object70415:head# 2024-03-20T09:04:50.326 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd93a8c7:::benchmark_data_smithi122_119671_object60822:head# 2024-03-20T09:04:50.326 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd960050:::benchmark_data_smithi122_119671_object99120:head# 2024-03-20T09:04:50.326 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd965dc1:::benchmark_data_smithi122_119671_object78336:head# 2024-03-20T09:04:50.326 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd96a2df:::benchmark_data_smithi122_119671_object90317:head# 2024-03-20T09:04:50.326 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd973224:::benchmark_data_smithi122_119671_object99199:head# 2024-03-20T09:04:50.327 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd975139:::benchmark_data_smithi122_119671_object14479:head# 2024-03-20T09:04:50.327 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd97f774:::benchmark_data_smithi122_119671_object60845:head# 2024-03-20T09:04:50.327 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd980810:::benchmark_data_smithi122_119671_object82001:head# 2024-03-20T09:04:50.327 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd98a188:::benchmark_data_smithi122_119671_object58182:head# 2024-03-20T09:04:50.327 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd99902a:::benchmark_data_smithi122_119671_object33765:head# 2024-03-20T09:04:50.327 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd9b39c9:::benchmark_data_smithi122_119671_object41854:head# 2024-03-20T09:04:50.327 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd9cb0b9:::benchmark_data_smithi122_119671_object34708:head# 2024-03-20T09:04:50.327 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd9d9fe3:::benchmark_data_smithi122_119671_object98043:head# 2024-03-20T09:04:50.327 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bd9f2e86:::benchmark_data_smithi122_119671_object93868:head# 2024-03-20T09:04:50.327 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bda01553:::benchmark_data_smithi122_119671_object20529:head# 2024-03-20T09:04:50.327 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bda03a3e:::benchmark_data_smithi122_119671_object47814:head# 2024-03-20T09:04:50.327 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bda0f464:::benchmark_data_smithi122_119671_object23410:head# 2024-03-20T09:04:50.328 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bda1aad3:::benchmark_data_smithi122_119671_object25406:head# 2024-03-20T09:04:50.328 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bda1d6f9:::benchmark_data_smithi122_119671_object86127:head# 2024-03-20T09:04:50.328 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bda26902:::benchmark_data_smithi122_119671_object58561:head# 2024-03-20T09:04:50.328 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bda2ed78:::benchmark_data_smithi122_119671_object22301:head# 2024-03-20T09:04:50.328 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bda34f14:::benchmark_data_smithi122_119671_object17499:head# 2024-03-20T09:04:50.328 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bda3609c:::benchmark_data_smithi122_119671_object26071:head# 2024-03-20T09:04:50.328 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bda4f8a8:::benchmark_data_smithi122_119671_object85152:head# 2024-03-20T09:04:50.328 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bda56562:::benchmark_data_smithi122_119671_object30439:head# 2024-03-20T09:04:50.328 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bda5772f:::benchmark_data_smithi122_119671_object52100:head# 2024-03-20T09:04:50.328 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bda7253e:::benchmark_data_smithi122_119671_object33409:head# 2024-03-20T09:04:50.328 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bda7e18b:::benchmark_data_smithi122_119671_object66754:head# 2024-03-20T09:04:50.328 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bda9ab3e:::benchmark_data_smithi122_119671_object98720:head# 2024-03-20T09:04:50.329 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bda9beb2:::benchmark_data_smithi122_119671_object21474:head# 2024-03-20T09:04:50.329 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdaa2228:::benchmark_data_smithi122_119671_object26068:head# 2024-03-20T09:04:50.329 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdaa9dfb:::benchmark_data_smithi122_119671_object88516:head# 2024-03-20T09:04:50.329 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdaab9da:::benchmark_data_smithi122_119671_object57877:head# 2024-03-20T09:04:50.329 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdaac687:::benchmark_data_smithi122_119671_object80692:head# 2024-03-20T09:04:50.329 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdab7e10:::benchmark_data_smithi122_119671_object40814:head# 2024-03-20T09:04:50.329 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdacbac9:::benchmark_data_smithi122_119671_object33424:head# 2024-03-20T09:04:50.329 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdad723f:::benchmark_data_smithi122_119671_object45022:head# 2024-03-20T09:04:50.329 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdae3c1a:::benchmark_data_smithi122_119671_object29402:head# 2024-03-20T09:04:50.329 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdafb051:::benchmark_data_smithi122_119671_object91785:head# 2024-03-20T09:04:50.329 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdb042cb:::benchmark_data_smithi122_119671_object17483:head# 2024-03-20T09:04:50.329 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdb0ca86:::benchmark_data_smithi122_119671_object74220:head# 2024-03-20T09:04:50.329 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdb11e1a:::benchmark_data_smithi122_119671_object6773:head# 2024-03-20T09:04:50.329 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdb12c5a:::benchmark_data_smithi122_119671_object61883:head# 2024-03-20T09:04:50.330 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdb15fad:::benchmark_data_smithi122_119671_object44072:head# 2024-03-20T09:04:50.330 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdb1e5ec:::benchmark_data_smithi122_119671_object26781:head# 2024-03-20T09:04:50.330 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdb2a609:::benchmark_data_smithi122_119671_object87852:head# 2024-03-20T09:04:50.330 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdb2f48a:::benchmark_data_smithi122_119671_object40384:head# 2024-03-20T09:04:50.330 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdb475e3:::benchmark_data_smithi122_119671_object83684:head# 2024-03-20T09:04:50.330 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdb4903f:::benchmark_data_smithi122_119671_object34649:head# 2024-03-20T09:04:50.330 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdb4bf32:::benchmark_data_smithi122_119671_object93096:head# 2024-03-20T09:04:50.330 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdb532b0:::benchmark_data_smithi122_119671_object73534:head# 2024-03-20T09:04:50.330 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdb57e99:::benchmark_data_smithi122_119671_object50901:head# 2024-03-20T09:04:50.330 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdb63a9c:::benchmark_data_smithi122_119671_object1870:head# 2024-03-20T09:04:50.330 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdb66bce:::benchmark_data_smithi122_119671_object97546:head# 2024-03-20T09:04:50.330 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdb71c3a:::benchmark_data_smithi122_119671_object9456:head# 2024-03-20T09:04:50.330 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdb74eee:::benchmark_data_smithi122_119671_object47742:head# 2024-03-20T09:04:50.331 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdb949bd:::benchmark_data_smithi122_119671_object16259:head# 2024-03-20T09:04:50.331 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdb9d4c4:::benchmark_data_smithi122_119671_object59094:head# 2024-03-20T09:04:50.331 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdba1836:::benchmark_data_smithi122_119671_object28862:head# 2024-03-20T09:04:50.331 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdba4815:::benchmark_data_smithi122_119671_object52877:head# 2024-03-20T09:04:50.331 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdbbb384:::benchmark_data_smithi122_119671_object43162:head# 2024-03-20T09:04:50.331 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdbbfd36:::benchmark_data_smithi122_119671_object96481:head# 2024-03-20T09:04:50.331 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdbc9130:::benchmark_data_smithi122_119671_object19566:head# 2024-03-20T09:04:50.331 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdbd547c:::benchmark_data_smithi122_119671_object1170:head# 2024-03-20T09:04:50.331 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdbe42b0:::benchmark_data_smithi122_119671_object62647:head# 2024-03-20T09:04:50.331 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdbeb57c:::benchmark_data_smithi122_119671_object84668:head# 2024-03-20T09:04:50.331 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdbf0b86:::benchmark_data_smithi122_119671_object81505:head# 2024-03-20T09:04:50.331 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdbfd87e:::benchmark_data_smithi122_119671_object65415:head# 2024-03-20T09:04:50.331 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdc07ffe:::benchmark_data_smithi122_119671_object37144:head# 2024-03-20T09:04:50.332 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdc0bbb2:::benchmark_data_smithi122_119671_object77974:head# 2024-03-20T09:04:50.332 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdc15793:::benchmark_data_smithi122_119671_object7173:head# 2024-03-20T09:04:50.332 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdc1b35a:::benchmark_data_smithi122_119671_object92281:head# 2024-03-20T09:04:50.332 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdc1c071:::benchmark_data_smithi122_119671_object15247:head# 2024-03-20T09:04:50.332 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdc25a56:::benchmark_data_smithi122_119671_object25729:head# 2024-03-20T09:04:50.332 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdc44391:::benchmark_data_smithi122_119671_object63594:head# 2024-03-20T09:04:50.332 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdc456fd:::benchmark_data_smithi122_119671_object62397:head# 2024-03-20T09:04:50.332 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdc5a978:::benchmark_data_smithi122_119671_object44448:head# 2024-03-20T09:04:50.332 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdc5cc60:::benchmark_data_smithi122_119671_object42878:head# 2024-03-20T09:04:50.332 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdc69e24:::benchmark_data_smithi122_119671_object6171:head# 2024-03-20T09:04:50.332 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdc6c003:::benchmark_data_smithi122_119671_object76571:head# 2024-03-20T09:04:50.332 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdc6f35f:::benchmark_data_smithi122_119671_object83760:head# 2024-03-20T09:04:50.333 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdc6ff24:::benchmark_data_smithi122_119671_object18350:head# 2024-03-20T09:04:50.333 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdc715a0:::benchmark_data_smithi122_119671_object91938:head# 2024-03-20T09:04:50.333 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdc763c5:::benchmark_data_smithi122_119671_object8604:head# 2024-03-20T09:04:50.333 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdc790d4:::benchmark_data_smithi122_119671_object15238:head# 2024-03-20T09:04:50.333 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdc7c60e:::benchmark_data_smithi122_119671_object74659:head# 2024-03-20T09:04:50.333 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdc81350:::benchmark_data_smithi122_119671_object64387:head# 2024-03-20T09:04:50.333 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdc84620:::benchmark_data_smithi122_119671_object85253:head# 2024-03-20T09:04:50.333 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdc9ac2d:::benchmark_data_smithi122_119671_object89390:head# 2024-03-20T09:04:50.333 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdc9f088:::benchmark_data_smithi122_119671_object96728:head# 2024-03-20T09:04:50.333 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdcaf3b3:::benchmark_data_smithi122_119671_object26197:head# 2024-03-20T09:04:50.333 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdcb762e:::benchmark_data_smithi122_119671_object93224:head# 2024-03-20T09:04:50.333 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdcbbcf6:::benchmark_data_smithi122_119671_object8608:head# 2024-03-20T09:04:50.333 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdcc6175:::benchmark_data_smithi122_119671_object99058:head# 2024-03-20T09:04:50.333 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdcc6417:::benchmark_data_smithi122_119671_object66443:head# 2024-03-20T09:04:50.334 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdcd0fcc:::benchmark_data_smithi122_119671_object36244:head# 2024-03-20T09:04:50.334 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdcdf243:::benchmark_data_smithi122_119671_object44199:head# 2024-03-20T09:04:50.334 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdce8e64:::benchmark_data_smithi122_119671_object69044:head# 2024-03-20T09:04:50.334 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdcf5e68:::benchmark_data_smithi122_119671_object99272:head# 2024-03-20T09:04:50.334 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdd090d9:::benchmark_data_smithi122_119671_object21636:head# 2024-03-20T09:04:50.334 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdd23b0b:::benchmark_data_smithi122_119671_object29130:head# 2024-03-20T09:04:50.334 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdd24da3:::benchmark_data_smithi122_119671_object57664:head# 2024-03-20T09:04:50.334 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdd393c7:::benchmark_data_smithi122_119671_object32083:head# 2024-03-20T09:04:50.334 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdd3c564:::benchmark_data_smithi122_119671_object62532:head# 2024-03-20T09:04:50.334 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdd3e37c:::benchmark_data_smithi122_119671_object91755:head# 2024-03-20T09:04:50.334 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdd4e98b:::benchmark_data_smithi122_119671_object75750:head# 2024-03-20T09:04:50.334 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdd598a6:::benchmark_data_smithi122_119671_object63068:head# 2024-03-20T09:04:50.334 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdd64ffb:::benchmark_data_smithi122_119671_object17262:head# 2024-03-20T09:04:50.335 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdd66933:::benchmark_data_smithi122_119671_object36092:head# 2024-03-20T09:04:50.335 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdd6a5a6:::benchmark_data_smithi122_119671_object91285:head# 2024-03-20T09:04:50.335 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdd8b543:::benchmark_data_smithi122_119671_object16702:head# 2024-03-20T09:04:50.335 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bddb4abf:::benchmark_data_smithi122_119671_object67837:head# 2024-03-20T09:04:50.335 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bddc1944:::benchmark_data_smithi122_119671_object43482:head# 2024-03-20T09:04:50.335 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bddc98bb:::benchmark_data_smithi122_119671_object53812:head# 2024-03-20T09:04:50.335 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bddd675d:::benchmark_data_smithi122_119671_object18963:head# 2024-03-20T09:04:50.335 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdde5fe6:::benchmark_data_smithi122_119671_object8647:head# 2024-03-20T09:04:50.335 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bddee6ff:::benchmark_data_smithi122_119671_object52086:head# 2024-03-20T09:04:50.335 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bddf79cb:::benchmark_data_smithi122_119671_object4212:head# 2024-03-20T09:04:50.335 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bde0d7e8:::benchmark_data_smithi122_119671_object31157:head# 2024-03-20T09:04:50.335 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bde1bc17:::benchmark_data_smithi122_119671_object61205:head# 2024-03-20T09:04:50.335 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bde274b4:::benchmark_data_smithi122_119671_object4129:head# 2024-03-20T09:04:50.335 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bde28b44:::benchmark_data_smithi122_119671_object86207:head# 2024-03-20T09:04:50.336 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bde2b885:::benchmark_data_smithi122_119671_object86854:head# 2024-03-20T09:04:50.336 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bde2b917:::benchmark_data_smithi122_119671_object47486:head# 2024-03-20T09:04:50.336 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bde4c5f5:::benchmark_data_smithi122_119671_object77223:head# 2024-03-20T09:04:50.336 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bde519cf:::benchmark_data_smithi122_119671_object65342:head# 2024-03-20T09:04:50.336 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bde67528:::benchmark_data_smithi122_119671_object28824:head# 2024-03-20T09:04:50.336 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bde6ee9d:::benchmark_data_smithi122_119671_object48245:head# 2024-03-20T09:04:50.336 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bde753b6:::benchmark_data_smithi122_119671_object91970:head# 2024-03-20T09:04:50.336 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bde798cc:::benchmark_data_smithi122_119671_object17446:head# 2024-03-20T09:04:50.336 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bde8d5e0:::benchmark_data_smithi122_119671_object86346:head# 2024-03-20T09:04:50.336 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bde8f6e3:::benchmark_data_smithi122_119671_object31202:head# 2024-03-20T09:04:50.336 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bde9227a:::benchmark_data_smithi122_119671_object12569:head# 2024-03-20T09:04:50.336 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bde95cd5:::benchmark_data_smithi122_119671_object7279:head# 2024-03-20T09:04:50.336 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bde9d4ec:::benchmark_data_smithi122_119671_object37320:head# 2024-03-20T09:04:50.337 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdea4db6:::benchmark_data_smithi122_119671_object83130:head# 2024-03-20T09:04:50.337 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdea8b5e:::benchmark_data_smithi122_119671_object80088:head# 2024-03-20T09:04:50.337 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdeb470e:::benchmark_data_smithi122_119671_object70564:head# 2024-03-20T09:04:50.337 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdec77ff:::benchmark_data_smithi122_119671_object91753:head# 2024-03-20T09:04:50.337 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdecfa36:::benchmark_data_smithi122_119671_object25373:head# 2024-03-20T09:04:50.337 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bded3936:::benchmark_data_smithi122_119671_object55479:head# 2024-03-20T09:04:50.337 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdf07ac6:::benchmark_data_smithi122_119671_object79514:head# 2024-03-20T09:04:50.337 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdf10ad4:::benchmark_data_smithi122_119671_object99362:head# 2024-03-20T09:04:50.337 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdf1f216:::benchmark_data_smithi122_119671_object22652:head# 2024-03-20T09:04:50.337 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdf60762:::benchmark_data_smithi122_119671_object94600:head# 2024-03-20T09:04:50.337 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdf7a00d:::benchmark_data_smithi122_119671_object82136:head# 2024-03-20T09:04:50.337 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdf86af4:::benchmark_data_smithi122_119671_object63022:head# 2024-03-20T09:04:50.337 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdf908e3:::benchmark_data_smithi122_119671_object66493:head# 2024-03-20T09:04:50.338 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdf94adc:::benchmark_data_smithi122_119671_object63551:head# 2024-03-20T09:04:50.338 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdf9ceca:::benchmark_data_smithi122_119671_object68943:head# 2024-03-20T09:04:50.338 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdf9f010:::benchmark_data_smithi122_119671_object92765:head# 2024-03-20T09:04:50.338 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdfcdf0f:::benchmark_data_smithi122_119671_object55869:head# 2024-03-20T09:04:50.338 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdfead9f:::benchmark_data_smithi122_119671_object29573:head# 2024-03-20T09:04:50.338 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdfecbcf:::benchmark_data_smithi122_119671_object72246:head# 2024-03-20T09:04:50.338 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdfed21e:::benchmark_data_smithi122_119671_object30222:head# 2024-03-20T09:04:50.338 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdff069c:::benchmark_data_smithi122_119671_object69040:head# 2024-03-20T09:04:50.338 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdff08cd:::benchmark_data_smithi122_119671_object99299:head# 2024-03-20T09:04:50.338 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdff8744:::benchmark_data_smithi122_119671_object33286:head# 2024-03-20T09:04:50.339 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bdffb1f6:::benchmark_data_smithi122_119671_object58627:head# 2024-03-20T09:04:50.339 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be001b84:::benchmark_data_smithi122_119671_object12341:head# 2024-03-20T09:04:50.339 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be013106:::benchmark_data_smithi122_119671_object15341:head# 2024-03-20T09:04:50.339 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be013c8d:::benchmark_data_smithi122_119671_object8090:head# 2024-03-20T09:04:50.339 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be018b62:::benchmark_data_smithi122_119671_object31418:head# 2024-03-20T09:04:50.339 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be0350cd:::benchmark_data_smithi122_119671_object74559:head# 2024-03-20T09:04:50.339 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be035272:::benchmark_data_smithi122_119671_object11414:head# 2024-03-20T09:04:50.339 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be03e0b4:::benchmark_data_smithi122_119671_object48614:head# 2024-03-20T09:04:50.339 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be04fa16:::benchmark_data_smithi122_119671_object70303:head# 2024-03-20T09:04:50.339 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be0517e0:::benchmark_data_smithi122_119671_object56147:head# 2024-03-20T09:04:50.339 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be055e04:::benchmark_data_smithi122_119671_object27118:head# 2024-03-20T09:04:50.339 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be0582a0:::benchmark_data_smithi122_119671_object79000:head# 2024-03-20T09:04:50.340 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be063a86:::benchmark_data_smithi122_119671_object87772:head# 2024-03-20T09:04:50.340 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be0681c8:::benchmark_data_smithi122_119671_object78417:head# 2024-03-20T09:04:50.340 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be068464:::benchmark_data_smithi122_119671_object7864:head# 2024-03-20T09:04:50.340 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be069960:::benchmark_data_smithi122_119671_object20926:head# 2024-03-20T09:04:50.340 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be0bff7c:::benchmark_data_smithi122_119671_object52397:head# 2024-03-20T09:04:50.340 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be0dbbd5:::benchmark_data_smithi122_119671_object67121:head# 2024-03-20T09:04:50.340 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be0ebc9f:::benchmark_data_smithi122_119671_object68828:head# 2024-03-20T09:04:50.340 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be0fc01b:::benchmark_data_smithi122_119671_object42844:head# 2024-03-20T09:04:50.340 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be0ffc57:::benchmark_data_smithi122_119671_object3219:head# 2024-03-20T09:04:50.340 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be112ddd:::benchmark_data_smithi122_119671_object6379:head# 2024-03-20T09:04:50.340 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be11d8b1:::benchmark_data_smithi122_119671_object65976:head# 2024-03-20T09:04:50.340 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be129286:::benchmark_data_smithi122_119671_object52483:head# 2024-03-20T09:04:50.340 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be135856:::benchmark_data_smithi122_119671_object31283:head# 2024-03-20T09:04:50.340 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be144b0a:::benchmark_data_smithi122_119671_object77181:head# 2024-03-20T09:04:50.341 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be14a0fc:::benchmark_data_smithi122_119671_object98622:head# 2024-03-20T09:04:50.341 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be14d450:::benchmark_data_smithi122_119671_object48113:head# 2024-03-20T09:04:50.341 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be15a54b:::benchmark_data_smithi122_119671_object48129:head# 2024-03-20T09:04:50.341 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be16aa7c:::benchmark_data_smithi122_119671_object23840:head# 2024-03-20T09:04:50.341 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be18955f:::benchmark_data_smithi122_119671_object46214:head# 2024-03-20T09:04:50.341 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be19656d:::benchmark_data_smithi122_119671_object2832:head# 2024-03-20T09:04:50.341 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be1983e9:::benchmark_data_smithi122_119671_object32522:head# 2024-03-20T09:04:50.341 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be19ebf2:::benchmark_data_smithi122_119671_object98681:head# 2024-03-20T09:04:50.341 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be1a58c1:::benchmark_data_smithi122_119671_object84054:head# 2024-03-20T09:04:50.341 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be1abc2f:::benchmark_data_smithi122_119671_object71355:head# 2024-03-20T09:04:50.341 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be1b76e3:::benchmark_data_smithi122_119671_object11377:head# 2024-03-20T09:04:50.341 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be1cccd0:::benchmark_data_smithi122_119671_object74287:head# 2024-03-20T09:04:50.341 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be1d16f0:::benchmark_data_smithi122_119671_object31819:head# 2024-03-20T09:04:50.341 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be1d44fd:::benchmark_data_smithi122_119671_object12214:head# 2024-03-20T09:04:50.342 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be1d8d5c:::benchmark_data_smithi122_119671_object66822:head# 2024-03-20T09:04:50.342 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be1db8e2:::benchmark_data_smithi122_119671_object76436:head# 2024-03-20T09:04:50.342 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be1e46f3:::benchmark_data_smithi122_119671_object62461:head# 2024-03-20T09:04:50.343 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:50.346 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be1e6e2b:::benchmark_data_smithi122_119671_object83352:head# 2024-03-20T09:04:50.346 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be1f8d74:::benchmark_data_smithi122_119671_object10746:head# 2024-03-20T09:04:50.346 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be1fb54c:::benchmark_data_smithi122_119671_object79195:head# 2024-03-20T09:04:50.346 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be2162ad:::benchmark_data_smithi122_119671_object53742:head# 2024-03-20T09:04:50.346 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be225bbd:::benchmark_data_smithi122_119671_object71618:head# 2024-03-20T09:04:50.346 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be2314bf:::benchmark_data_smithi122_119671_object77355:head# 2024-03-20T09:04:50.346 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be236354:::benchmark_data_smithi122_119671_object65021:head# 2024-03-20T09:04:50.346 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be254684:::benchmark_data_smithi122_119671_object17680:head# 2024-03-20T09:04:50.346 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be260554:::benchmark_data_smithi122_119671_object70639:head# 2024-03-20T09:04:50.346 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be26248c:::benchmark_data_smithi122_119671_object39706:head# 2024-03-20T09:04:50.347 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be264c02:::benchmark_data_smithi122_119671_object70741:head# 2024-03-20T09:04:50.347 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be264dc6:::benchmark_data_smithi122_119671_object48437:head# 2024-03-20T09:04:50.347 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be277b1e:::benchmark_data_smithi122_119671_object76668:head# 2024-03-20T09:04:50.347 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be289406:::benchmark_data_smithi122_119671_object91721:head# 2024-03-20T09:04:50.347 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be289af0:::benchmark_data_smithi122_119671_object81646:head# 2024-03-20T09:04:50.347 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be28a06a:::benchmark_data_smithi122_119671_object90757:head# 2024-03-20T09:04:50.347 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be292bf5:::benchmark_data_smithi122_119671_object49991:head# 2024-03-20T09:04:50.347 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be2aad4f:::benchmark_data_smithi122_119671_object14289:head# 2024-03-20T09:04:50.347 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be2c06de:::benchmark_data_smithi122_119671_object7209:head# 2024-03-20T09:04:50.347 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be2c2d58:::benchmark_data_smithi122_119671_object92893:head# 2024-03-20T09:04:50.347 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be2dc89c:::benchmark_data_smithi122_119671_object48828:head# 2024-03-20T09:04:50.347 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be2de023:::benchmark_data_smithi122_119671_object63608:head# 2024-03-20T09:04:50.347 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be2e009b:::benchmark_data_smithi122_119671_object31220:head# 2024-03-20T09:04:50.348 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be2ea2ac:::benchmark_data_smithi122_119671_object7865:head# 2024-03-20T09:04:50.348 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be2f41e9:::benchmark_data_smithi122_119671_object97092:head# 2024-03-20T09:04:50.348 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be30d369:::benchmark_data_smithi122_119671_object43243:head# 2024-03-20T09:04:50.348 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be321431:::benchmark_data_smithi122_119671_object81595:head# 2024-03-20T09:04:50.348 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be32a3fe:::benchmark_data_smithi122_119671_object41546:head# 2024-03-20T09:04:50.348 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be333b59:::benchmark_data_smithi122_119671_object84533:head# 2024-03-20T09:04:50.348 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be33d61d:::benchmark_data_smithi122_119671_object35279:head# 2024-03-20T09:04:50.348 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be37b333:::benchmark_data_smithi122_119671_object57624:head# 2024-03-20T09:04:50.348 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be3804f5:::benchmark_data_smithi122_119671_object47909:head# 2024-03-20T09:04:50.348 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be3930c1:::benchmark_data_smithi122_119671_object16810:head# 2024-03-20T09:04:50.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:50.351 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:04:50.351 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:04:50.352 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be3a04c2:::benchmark_data_smithi122_119671_object13826:head# 2024-03-20T09:04:50.352 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be3a5c58:::benchmark_data_smithi122_119671_object32174:head# 2024-03-20T09:04:50.352 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be3c8a1b:::benchmark_data_smithi122_119671_object79856:head# 2024-03-20T09:04:50.352 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be3c9a80:::benchmark_data_smithi122_119671_object28687:head# 2024-03-20T09:04:50.352 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be3ddbef:::benchmark_data_smithi122_119671_object76154:head# 2024-03-20T09:04:50.352 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be3e269f:::benchmark_data_smithi122_119671_object31549:head# 2024-03-20T09:04:50.353 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be3f1677:::benchmark_data_smithi122_119671_object97776:head# 2024-03-20T09:04:50.353 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be3f83d0:::benchmark_data_smithi122_119671_object61215:head# 2024-03-20T09:04:50.353 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be405317:::benchmark_data_smithi122_119671_object57778:head# 2024-03-20T09:04:50.353 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be412ba6:::benchmark_data_smithi122_119671_object91720:head# 2024-03-20T09:04:50.353 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be41cb94:::benchmark_data_smithi122_119671_object41179:head# 2024-03-20T09:04:50.353 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be4216da:::benchmark_data_smithi122_119671_object26589:head# 2024-03-20T09:04:50.353 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be42a559:::benchmark_data_smithi122_119671_object13541:head# 2024-03-20T09:04:50.354 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be42f0a4:::benchmark_data_smithi122_119671_object62808:head# 2024-03-20T09:04:50.354 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be442bc6:::benchmark_data_smithi122_119671_object94898:head# 2024-03-20T09:04:50.354 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be4473a8:::benchmark_data_smithi122_119671_object73295:head# 2024-03-20T09:04:50.354 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be44d8db:::benchmark_data_smithi122_119671_object85926:head# 2024-03-20T09:04:50.354 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be45d436:::benchmark_data_smithi122_119671_object91692:head# 2024-03-20T09:04:50.355 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be4626d5:::benchmark_data_smithi122_119671_object98811:head# 2024-03-20T09:04:50.355 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be467cec:::benchmark_data_smithi122_119671_object7981:head# 2024-03-20T09:04:50.355 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be469ab1:::benchmark_data_smithi122_119671_object3216:head# 2024-03-20T09:04:50.356 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be47e083:::benchmark_data_smithi122_119671_object21162:head# 2024-03-20T09:04:50.356 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be47f0a8:::benchmark_data_smithi122_119671_object9731:head# 2024-03-20T09:04:50.356 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be483333:::benchmark_data_smithi122_119671_object33730:head# 2024-03-20T09:04:50.357 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be484b6c:::benchmark_data_smithi122_119671_object48671:head# 2024-03-20T09:04:50.357 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be485775:::benchmark_data_smithi122_119671_object29307:head# 2024-03-20T09:04:50.357 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be489887:::benchmark_data_smithi122_119671_object45072:head# 2024-03-20T09:04:50.358 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be48b879:::benchmark_data_smithi122_119671_object87315:head# 2024-03-20T09:04:50.359 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be48c126:::benchmark_data_smithi122_119671_object80675:head# 2024-03-20T09:04:50.359 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be490d6e:::benchmark_data_smithi122_119671_object33331:head# 2024-03-20T09:04:50.359 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be4d1d30:::benchmark_data_smithi122_119671_object88960:head# 2024-03-20T09:04:50.360 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be4e5b82:::benchmark_data_smithi122_119671_object80319:head# 2024-03-20T09:04:50.360 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be4e92ad:::benchmark_data_smithi122_119671_object56994:head# 2024-03-20T09:04:50.360 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be4fd133:::benchmark_data_smithi122_119671_object66177:head# 2024-03-20T09:04:50.361 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be50c70a:::benchmark_data_smithi122_119671_object74508:head# 2024-03-20T09:04:50.361 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be51288d:::benchmark_data_smithi122_119671_object93410:head# 2024-03-20T09:04:50.361 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be51f8fd:::benchmark_data_smithi122_119671_object73252:head# 2024-03-20T09:04:50.362 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be52378a:::benchmark_data_smithi122_119671_object4265:head# 2024-03-20T09:04:50.362 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be52621f:::benchmark_data_smithi122_119671_object6868:head# 2024-03-20T09:04:50.363 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be52ce09:::benchmark_data_smithi122_119671_object71902:head# 2024-03-20T09:04:50.366 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be52fa9b:::benchmark_data_smithi122_119671_object56798:head# 2024-03-20T09:04:50.366 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be538f90:::benchmark_data_smithi122_119671_object26533:head# 2024-03-20T09:04:50.366 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be53aa52:::benchmark_data_smithi122_119671_object33799:head# 2024-03-20T09:04:50.366 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be54bc67:::benchmark_data_smithi122_119671_object41641:head# 2024-03-20T09:04:50.367 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be54d2f6:::benchmark_data_smithi122_119671_object62125:head# 2024-03-20T09:04:50.367 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be57211b:::benchmark_data_smithi122_119671_object60892:head# 2024-03-20T09:04:50.369 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be57a6b4:::benchmark_data_smithi122_119671_object78631:head# 2024-03-20T09:04:50.371 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be586ebd:::benchmark_data_smithi122_119671_object19457:head# 2024-03-20T09:04:50.371 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be58fa3b:::benchmark_data_smithi122_119671_object14710:head# 2024-03-20T09:04:50.373 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be59aa7f:::benchmark_data_smithi122_119671_object47498:head# 2024-03-20T09:04:50.374 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be5b3f24:::benchmark_data_smithi122_119671_object23347:head# 2024-03-20T09:04:50.376 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be5bda0c:::benchmark_data_smithi122_119671_object89096:head# 2024-03-20T09:04:50.377 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be5bf7aa:::benchmark_data_smithi122_119671_object32464:head# 2024-03-20T09:04:50.377 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be5c043f:::benchmark_data_smithi122_119671_object60383:head# 2024-03-20T09:04:50.378 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be5c22ef:::benchmark_data_smithi122_119671_object18085:head# 2024-03-20T09:04:50.378 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be5cc4af:::benchmark_data_smithi122_119671_object73658:head# 2024-03-20T09:04:50.379 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be5f2589:::benchmark_data_smithi122_119671_object28963:head# 2024-03-20T09:04:50.379 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be5febc4:::benchmark_data_smithi122_119671_object39442:head# 2024-03-20T09:04:50.379 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be5fff33:::benchmark_data_smithi122_119671_object69225:head# 2024-03-20T09:04:50.380 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be607714:::benchmark_data_smithi122_119671_object34030:head# 2024-03-20T09:04:50.380 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be60dacb:::benchmark_data_smithi122_119671_object54811:head# 2024-03-20T09:04:50.381 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be60f0d0:::benchmark_data_smithi122_119671_object86740:head# 2024-03-20T09:04:50.381 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be615398:::benchmark_data_smithi122_119671_object30774:head# 2024-03-20T09:04:50.381 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be61fcaf:::benchmark_data_smithi122_119671_object12404:head# 2024-03-20T09:04:50.382 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be621810:::benchmark_data_smithi122_119671_object79431:head# 2024-03-20T09:04:50.382 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be624c53:::benchmark_data_smithi122_119671_object70676:head# 2024-03-20T09:04:50.383 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be633ef5:::benchmark_data_smithi122_119671_object75397:head# 2024-03-20T09:04:50.383 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be639423:::benchmark_data_smithi122_119671_object14571:head# 2024-03-20T09:04:50.383 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be639f18:::benchmark_data_smithi122_119671_object17434:head# 2024-03-20T09:04:50.384 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be63f244:::benchmark_data_smithi122_119671_object53221:head# 2024-03-20T09:04:50.384 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be64eda6:::benchmark_data_smithi122_119671_object48858:head# 2024-03-20T09:04:50.385 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be6600f8:::benchmark_data_smithi122_119671_object38818:head# 2024-03-20T09:04:50.385 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be660a33:::benchmark_data_smithi122_119671_object1428:head# 2024-03-20T09:04:50.385 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be66b12e:::benchmark_data_smithi122_119671_object78080:head# 2024-03-20T09:04:50.386 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be673701:::benchmark_data_smithi122_119671_object64583:head# 2024-03-20T09:04:50.386 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be689dcc:::benchmark_data_smithi122_119671_object65486:head# 2024-03-20T09:04:50.388 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be68a8fa:::benchmark_data_smithi122_119671_object8584:head# 2024-03-20T09:04:50.388 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be68f402:::benchmark_data_smithi122_119671_object70052:head# 2024-03-20T09:04:50.388 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be6921a9:::benchmark_data_smithi122_119671_object81289:head# 2024-03-20T09:04:50.388 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be6a34a0:::benchmark_data_smithi122_119671_object56777:head# 2024-03-20T09:04:50.389 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be6a3e20:::benchmark_data_smithi122_119671_object29324:head# 2024-03-20T09:04:50.389 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be6aedaa:::benchmark_data_smithi122_119671_object21209:head# 2024-03-20T09:04:50.389 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be6b2c1b:::benchmark_data_smithi122_119671_object76671:head# 2024-03-20T09:04:50.390 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be6bcf55:::benchmark_data_smithi122_119671_object62732:head# 2024-03-20T09:04:50.390 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be6d64e4:::benchmark_data_smithi122_119671_object88780:head# 2024-03-20T09:04:50.391 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be6d6838:::benchmark_data_smithi122_119671_object75416:head# 2024-03-20T09:04:50.391 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be6da178:::benchmark_data_smithi122_119671_object21088:head# 2024-03-20T09:04:50.392 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be6df2b5:::benchmark_data_smithi122_119671_object6652:head# 2024-03-20T09:04:50.394 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be6e5546:::benchmark_data_smithi122_119671_object97705:head# 2024-03-20T09:04:50.395 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be6e6e9e:::benchmark_data_smithi122_119671_object2879:head# 2024-03-20T09:04:50.395 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be6f57ad:::benchmark_data_smithi122_119671_object62872:head# 2024-03-20T09:04:50.396 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be6f8ce1:::benchmark_data_smithi122_119671_object84780:head# 2024-03-20T09:04:50.396 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be6fb5ac:::benchmark_data_smithi122_119671_object69185:head# 2024-03-20T09:04:50.396 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be70123e:::benchmark_data_smithi122_119671_object41228:head# 2024-03-20T09:04:50.397 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be70f62d:::benchmark_data_smithi122_119671_object32877:head# 2024-03-20T09:04:50.397 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be714846:::benchmark_data_smithi122_119671_object55496:head# 2024-03-20T09:04:50.398 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be7154c8:::benchmark_data_smithi122_119671_object90532:head# 2024-03-20T09:04:50.398 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be7177c0:::benchmark_data_smithi122_119671_object83843:head# 2024-03-20T09:04:50.398 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be71f07b:::benchmark_data_smithi122_119671_object52185:head# 2024-03-20T09:04:50.399 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be71fdfc:::benchmark_data_smithi122_119671_object87013:head# 2024-03-20T09:04:50.399 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be72cbe3:::benchmark_data_smithi122_119671_object16447:head# 2024-03-20T09:04:50.400 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be73f453:::benchmark_data_smithi122_119671_object89351:head# 2024-03-20T09:04:50.400 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be74634c:::benchmark_data_smithi122_119671_object53803:head# 2024-03-20T09:04:50.401 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be785b8f:::benchmark_data_smithi122_119671_object45086:head# 2024-03-20T09:04:50.401 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be78ad3e:::benchmark_data_smithi122_119671_object92087:head# 2024-03-20T09:04:50.402 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be78c785:::benchmark_data_smithi122_119671_object78068:head# 2024-03-20T09:04:50.402 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be799ef5:::benchmark_data_smithi122_119671_object35719:head# 2024-03-20T09:04:50.403 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be79b8a5:::benchmark_data_smithi122_119671_object87522:head# 2024-03-20T09:04:50.404 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be79cf38:::benchmark_data_smithi122_119671_object3714:head# 2024-03-20T09:04:50.404 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be79ddd6:::benchmark_data_smithi122_119671_object75895:head# 2024-03-20T09:04:50.405 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be7abae1:::benchmark_data_smithi122_119671_object74715:head# 2024-03-20T09:04:50.405 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be7c27a6:::benchmark_data_smithi122_119671_object94662:head# 2024-03-20T09:04:50.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:49 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/19587992' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:04:50.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:49 smithi115 ceph-mon[72476]: pgmap v731: 47 pgs: 1 active+clean+remapped, 1 activating+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 29 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 6.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 28606/294816 objects degraded (9.703%); 12685/294816 objects misplaced (4.303%); 25 MiB/s, 410 objects/s recovering 2024-03-20T09:04:50.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:49 smithi115 ceph-mon[72476]: 3.12 scrub starts 2024-03-20T09:04:50.407 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be7da385:::benchmark_data_smithi122_119671_object76557:head# 2024-03-20T09:04:50.407 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be7dc485:::benchmark_data_smithi122_119671_object30120:head# 2024-03-20T09:04:50.407 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be7f6ca0:::benchmark_data_smithi122_119671_object12785:head# 2024-03-20T09:04:50.407 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be7f991c:::benchmark_data_smithi122_119671_object12658:head# 2024-03-20T09:04:50.407 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be804bbe:::benchmark_data_smithi122_119671_object84767:head# 2024-03-20T09:04:50.408 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be807993:::benchmark_data_smithi122_119671_object6328:head# 2024-03-20T09:04:50.408 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be825287:::benchmark_data_smithi122_119671_object20193:head# 2024-03-20T09:04:50.408 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be827e9c:::benchmark_data_smithi122_119671_object93807:head# 2024-03-20T09:04:50.408 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be83eac3:::benchmark_data_smithi122_119671_object15349:head# 2024-03-20T09:04:50.409 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be844c9e:::benchmark_data_smithi122_119671_object33027:head# 2024-03-20T09:04:50.409 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be8480f3:::benchmark_data_smithi122_119671_object21308:head# 2024-03-20T09:04:50.409 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be84acc1:::benchmark_data_smithi122_119671_object86331:head# 2024-03-20T09:04:50.410 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be84b0be:::benchmark_data_smithi122_119671_object35673:head# 2024-03-20T09:04:50.410 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be850f90:::benchmark_data_smithi122_119671_object77988:head# 2024-03-20T09:04:50.411 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be8521d1:::benchmark_data_smithi122_119671_object56097:head# 2024-03-20T09:04:50.411 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be85694d:::benchmark_data_smithi122_119671_object92794:head# 2024-03-20T09:04:50.411 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be858790:::benchmark_data_smithi122_119671_object94015:head# 2024-03-20T09:04:50.412 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be8601b3:::benchmark_data_smithi122_119671_object39128:head# 2024-03-20T09:04:50.412 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be860524:::benchmark_data_smithi122_119671_object37124:head# 2024-03-20T09:04:50.413 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be868b3e:::benchmark_data_smithi122_119671_object16672:head# 2024-03-20T09:04:50.413 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be875c30:::benchmark_data_smithi122_119671_object19723:head# 2024-03-20T09:04:50.414 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be878d84:::benchmark_data_smithi122_119671_object37350:head# 2024-03-20T09:04:50.414 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be87b34e:::benchmark_data_smithi122_119671_object15973:head# 2024-03-20T09:04:50.414 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be882b1f:::benchmark_data_smithi122_119671_object91434:head# 2024-03-20T09:04:50.414 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be8856d1:::benchmark_data_smithi122_119671_object87608:head# 2024-03-20T09:04:50.415 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be89fcf3:::benchmark_data_smithi122_119671_object98302:head# 2024-03-20T09:04:50.415 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be8a50fa:::benchmark_data_smithi122_119671_object42945:head# 2024-03-20T09:04:50.416 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be8a8b21:::benchmark_data_smithi122_119671_object45028:head# 2024-03-20T09:04:50.416 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be8c066e:::benchmark_data_smithi122_119671_object15698:head# 2024-03-20T09:04:50.416 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be8cc2a3:::benchmark_data_smithi122_119671_object68519:head# 2024-03-20T09:04:50.417 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be8dec1d:::benchmark_data_smithi122_119671_object16845:head# 2024-03-20T09:04:50.417 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be8e86ab:::benchmark_data_smithi122_119671_object84964:head# 2024-03-20T09:04:50.418 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be8ef1db:::benchmark_data_smithi122_119671_object26366:head# 2024-03-20T09:04:50.418 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be8f0744:::benchmark_data_smithi122_119671_object41871:head# 2024-03-20T09:04:50.418 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be8f1ac9:::benchmark_data_smithi122_119671_object3296:head# 2024-03-20T09:04:50.419 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be9076d1:::benchmark_data_smithi122_119671_object26293:head# 2024-03-20T09:04:50.419 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be90775a:::benchmark_data_smithi122_119671_object41630:head# 2024-03-20T09:04:50.419 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be9155b6:::benchmark_data_smithi122_119671_object65331:head# 2024-03-20T09:04:50.420 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be9209a5:::benchmark_data_smithi122_119671_object24743:head# 2024-03-20T09:04:50.420 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be934c58:::benchmark_data_smithi122_119671_object11878:head# 2024-03-20T09:04:50.421 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be93e312:::benchmark_data_smithi122_119671_object21301:head# 2024-03-20T09:04:50.421 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be944019:::benchmark_data_smithi122_119671_object81597:head# 2024-03-20T09:04:50.421 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be94c698:::benchmark_data_smithi122_119671_object21889:head# 2024-03-20T09:04:50.422 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be94d538:::benchmark_data_smithi122_119671_object10372:head# 2024-03-20T09:04:50.422 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be94eb45:::benchmark_data_smithi122_119671_object18024:head# 2024-03-20T09:04:50.422 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be94ee7f:::benchmark_data_smithi122_119671_object14509:head# 2024-03-20T09:04:50.423 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be951e60:::benchmark_data_smithi122_119671_object79790:head# 2024-03-20T09:04:50.423 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be952a26:::benchmark_data_smithi122_119671_object85661:head# 2024-03-20T09:04:50.424 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be980620:::benchmark_data_smithi122_119671_object42781:head# 2024-03-20T09:04:50.424 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be981512:::benchmark_data_smithi122_119671_object93959:head# 2024-03-20T09:04:50.425 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be9ac3bf:::benchmark_data_smithi122_119671_object37482:head# 2024-03-20T09:04:50.425 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be9c734e:::benchmark_data_smithi122_119671_object49150:head# 2024-03-20T09:04:50.426 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be9da27e:::benchmark_data_smithi122_119671_object73607:head# 2024-03-20T09:04:50.426 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be9e2e0b:::benchmark_data_smithi122_119671_object46712:head# 2024-03-20T09:04:50.426 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be9eff78:::benchmark_data_smithi122_119671_object17783:head# 2024-03-20T09:04:50.427 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:be9f5588:::benchmark_data_smithi122_119671_object12610:head# 2024-03-20T09:04:50.427 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bea0327f:::benchmark_data_smithi122_119671_object76487:head# 2024-03-20T09:04:50.428 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bea0337a:::benchmark_data_smithi122_119671_object85287:head# 2024-03-20T09:04:50.428 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bea03779:::benchmark_data_smithi122_119671_object93388:head# 2024-03-20T09:04:50.428 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bea102b7:::benchmark_data_smithi122_119671_object39811:head# 2024-03-20T09:04:50.429 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bea11d86:::benchmark_data_smithi122_119671_object19897:head# 2024-03-20T09:04:50.429 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bea22050:::benchmark_data_smithi122_119671_object83490:head# 2024-03-20T09:04:50.429 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bea2d1c4:::benchmark_data_smithi122_119671_object33785:head# 2024-03-20T09:04:50.430 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bea3333a:::benchmark_data_smithi122_119671_object63435:head# 2024-03-20T09:04:50.430 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bea33b66:::benchmark_data_smithi122_119671_object38503:head# 2024-03-20T09:04:50.431 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bea4160b:::benchmark_data_smithi122_119671_object32823:head# 2024-03-20T09:04:50.431 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bea49e7c:::benchmark_data_smithi122_119671_object38908:head# 2024-03-20T09:04:50.431 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bea4a193:::benchmark_data_smithi122_119671_object6556:head# 2024-03-20T09:04:50.432 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bea61917:::benchmark_data_smithi122_119671_object8858:head# 2024-03-20T09:04:50.432 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bea64b2b:::benchmark_data_smithi122_119671_object90404:head# 2024-03-20T09:04:50.432 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bea65069:::benchmark_data_smithi122_119671_object47239:head# 2024-03-20T09:04:50.433 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bea6c166:::benchmark_data_smithi122_119671_object46698:head# 2024-03-20T09:04:50.434 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bea6dc03:::benchmark_data_smithi122_119671_object98858:head# 2024-03-20T09:04:50.434 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bea80076:::benchmark_data_smithi122_119671_object15475:head# 2024-03-20T09:04:50.434 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bea8ec1e:::benchmark_data_smithi122_119671_object73454:head# 2024-03-20T09:04:50.435 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:beaa36da:::benchmark_data_smithi122_119671_object27940:head# 2024-03-20T09:04:50.435 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:beaa6033:::benchmark_data_smithi122_119671_object2292:head# 2024-03-20T09:04:50.435 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:beacb9b1:::benchmark_data_smithi122_119671_object94664:head# 2024-03-20T09:04:50.436 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bead82e5:::benchmark_data_smithi122_119671_object5946:head# 2024-03-20T09:04:50.437 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:beb06093:::benchmark_data_smithi122_119671_object2822:head# 2024-03-20T09:04:50.437 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:beb0827a:::benchmark_data_smithi122_119671_object70874:head# 2024-03-20T09:04:50.437 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:beb0b7a6:::benchmark_data_smithi122_119671_object5809:head# 2024-03-20T09:04:50.437 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:beb0d3b0:::benchmark_data_smithi122_119671_object9457:head# 2024-03-20T09:04:50.438 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:beb0f204:::benchmark_data_smithi122_119671_object37661:head# 2024-03-20T09:04:50.438 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:beb2331c:::benchmark_data_smithi122_119671_object76310:head# 2024-03-20T09:04:50.438 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:beb2c953:::benchmark_data_smithi122_119671_object57405:head# 2024-03-20T09:04:50.439 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:beb2e08e:::benchmark_data_smithi122_119671_object9216:head# 2024-03-20T09:04:50.439 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:beb338de:::benchmark_data_smithi122_119671_object6425:head# 2024-03-20T09:04:50.440 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:beb3e54f:::benchmark_data_smithi122_119671_object18011:head# 2024-03-20T09:04:50.441 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:beb48688:::benchmark_data_smithi122_119671_object23058:head# 2024-03-20T09:04:50.442 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:beb4c6c5:::benchmark_data_smithi122_119671_object19623:head# 2024-03-20T09:04:50.443 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:beb58d58:::benchmark_data_smithi122_119671_object94190:head# 2024-03-20T09:04:50.443 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:beb5a9a4:::benchmark_data_smithi122_119671_object16959:head# 2024-03-20T09:04:50.444 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:beb7693f:::benchmark_data_smithi122_119671_object82261:head# 2024-03-20T09:04:50.444 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:beb801a4:::benchmark_data_smithi122_119671_object50487:head# 2024-03-20T09:04:50.445 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:beb869ba:::benchmark_data_smithi122_119671_object71989:head# 2024-03-20T09:04:50.445 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:beb8972e:::benchmark_data_smithi122_119671_object30607:head# 2024-03-20T09:04:50.446 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:beb8fc9d:::benchmark_data_smithi122_119671_object89388:head# 2024-03-20T09:04:50.446 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bebba2a3:::benchmark_data_smithi122_119671_object76657:head# 2024-03-20T09:04:50.447 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bebd4fbf:::benchmark_data_smithi122_119671_object47028:head# 2024-03-20T09:04:50.448 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bebd7ee6:::benchmark_data_smithi122_119671_object6678:head# 2024-03-20T09:04:50.449 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bebd889f:::benchmark_data_smithi122_119671_object30297:head# 2024-03-20T09:04:50.449 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bebf686d:::benchmark_data_smithi122_119671_object79964:head# 2024-03-20T09:04:50.450 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bebff038:::benchmark_data_smithi122_119671_object39747:head# 2024-03-20T09:04:50.450 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bec0a682:::benchmark_data_smithi122_119671_object46782:head# 2024-03-20T09:04:50.450 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bec1843a:::benchmark_data_smithi122_119671_object66805:head# 2024-03-20T09:04:50.451 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bec19845:::benchmark_data_smithi122_119671_object78789:head# 2024-03-20T09:04:50.451 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bec1d981:::benchmark_data_smithi122_119671_object73552:head# 2024-03-20T09:04:50.451 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bec1fe46:::benchmark_data_smithi122_119671_object67776:head# 2024-03-20T09:04:50.452 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:04:50.453 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bec24bac:::benchmark_data_smithi122_119671_object77623:head# 2024-03-20T09:04:50.453 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bec25437:::benchmark_data_smithi122_119671_object27732:head# 2024-03-20T09:04:50.453 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bec27e55:::benchmark_data_smithi122_119671_object1356:head# 2024-03-20T09:04:50.453 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bec2fedd:::benchmark_data_smithi122_119671_object33055:head# 2024-03-20T09:04:50.454 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bec392e7:::benchmark_data_smithi122_119671_object63369:head# 2024-03-20T09:04:50.457 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bec4a8cb:::benchmark_data_smithi122_119671_object45432:head# 2024-03-20T09:04:50.457 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bec5b116:::benchmark_data_smithi122_119671_object5062:head# 2024-03-20T09:04:50.457 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bec68172:::benchmark_data_smithi122_119671_object35049:head# 2024-03-20T09:04:50.457 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bec74f4c:::benchmark_data_smithi122_119671_object26537:head# 2024-03-20T09:04:50.457 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bec7613c:::benchmark_data_smithi122_119671_object51206:head# 2024-03-20T09:04:50.457 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bec7a9f7:::benchmark_data_smithi122_119671_object46740:head# 2024-03-20T09:04:50.457 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bec87fc2:::benchmark_data_smithi122_119671_object71263:head# 2024-03-20T09:04:50.457 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bec9a559:::benchmark_data_smithi122_119671_object64737:head# 2024-03-20T09:04:50.458 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:beca273b:::benchmark_data_smithi122_119671_object26942:head# 2024-03-20T09:04:50.459 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:becad4e4:::benchmark_data_smithi122_119671_object83918:head# 2024-03-20T09:04:50.459 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:becb0ffb:::benchmark_data_smithi122_119671_object40204:head# 2024-03-20T09:04:50.459 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:becb7473:::benchmark_data_smithi122_119671_object59371:head# 2024-03-20T09:04:50.460 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:becb9cc9:::benchmark_data_smithi122_119671_object27952:head# 2024-03-20T09:04:50.460 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:becca1bb:::benchmark_data_smithi122_119671_object83958:head# 2024-03-20T09:04:50.460 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:beccb32e:::benchmark_data_smithi122_119671_object39613:head# 2024-03-20T09:04:50.461 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:beccd9e3:::benchmark_data_smithi122_119671_object78532:head# 2024-03-20T09:04:50.461 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:becd3cfb:::benchmark_data_smithi122_119671_object83166:head# 2024-03-20T09:04:50.462 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bece5c2c:::benchmark_data_smithi122_119671_object43233:head# 2024-03-20T09:04:50.463 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bed08d6c:::benchmark_data_smithi122_119671_object64129:head# 2024-03-20T09:04:50.463 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bed0a553:::benchmark_data_smithi122_119671_object20686:head# 2024-03-20T09:04:50.463 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bed3978d:::benchmark_data_smithi122_119671_object63181:head# 2024-03-20T09:04:50.464 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bed42393:::benchmark_data_smithi122_119671_object4039:head# 2024-03-20T09:04:50.464 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bed45071:::benchmark_data_smithi122_119671_object80723:head# 2024-03-20T09:04:50.465 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bed55ef3:::benchmark_data_smithi122_119671_object67839:head# 2024-03-20T09:04:50.465 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bed69030:::benchmark_data_smithi122_119671_object74370:head# 2024-03-20T09:04:50.466 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bed8a13e:::benchmark_data_smithi122_119671_object61434:head# 2024-03-20T09:04:50.466 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bed8e87e:::benchmark_data_smithi122_119671_object51337:head# 2024-03-20T09:04:50.467 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:beda8477:::benchmark_data_smithi122_119671_object87842:head# 2024-03-20T09:04:50.467 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:beda8fea:::benchmark_data_smithi122_119671_object78049:head# 2024-03-20T09:04:50.468 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:beda981a:::benchmark_data_smithi122_119671_object72572:head# 2024-03-20T09:04:50.468 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bedb2e86:::benchmark_data_smithi122_119671_object14111:head# 2024-03-20T09:04:50.468 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bedb6bbe:::benchmark_data_smithi122_119671_object15565:head# 2024-03-20T09:04:50.469 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bedb7594:::benchmark_data_smithi122_119671_object96684:head# 2024-03-20T09:04:50.469 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bedc40ba:::benchmark_data_smithi122_119671_object72889:head# 2024-03-20T09:04:50.472 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bedc6836:::benchmark_data_smithi122_119671_object57156:head# 2024-03-20T09:04:50.473 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bedc7af3:::benchmark_data_smithi122_119671_object44284:head# 2024-03-20T09:04:50.473 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bedcd284:::benchmark_data_smithi122_119671_object64267:head# 2024-03-20T09:04:50.473 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bedcf23f:::benchmark_data_smithi122_119671_object16546:head# 2024-03-20T09:04:50.474 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bede5907:::benchmark_data_smithi122_119671_object11315:head# 2024-03-20T09:04:50.474 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bede647f:::benchmark_data_smithi122_119671_object85984:head# 2024-03-20T09:04:50.474 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bede9891:::benchmark_data_smithi122_119671_object99705:head# 2024-03-20T09:04:50.474 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bedec74f:::benchmark_data_smithi122_119671_object80817:head# 2024-03-20T09:04:50.475 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bee00d2c:::benchmark_data_smithi122_119671_object34705:head# 2024-03-20T09:04:50.475 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bee05e28:::benchmark_data_smithi122_119671_object70701:head# 2024-03-20T09:04:50.475 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bee060ad:::benchmark_data_smithi122_119671_object9783:head# 2024-03-20T09:04:50.476 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bee1cc06:::benchmark_data_smithi122_119671_object65398:head# 2024-03-20T09:04:50.476 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bee33e86:::benchmark_data_smithi122_119671_object26133:head# 2024-03-20T09:04:50.477 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bee44a0e:::benchmark_data_smithi122_119671_object37363:head# 2024-03-20T09:04:50.477 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bee46110:::benchmark_data_smithi122_119671_object74587:head# 2024-03-20T09:04:50.477 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bee49a07:::benchmark_data_smithi122_119671_object58667:head# 2024-03-20T09:04:50.480 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bee522a5:::benchmark_data_smithi122_119671_object89739:head# 2024-03-20T09:04:50.480 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bee53285:::benchmark_data_smithi122_119671_object30982:head# 2024-03-20T09:04:50.480 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bee59f34:::benchmark_data_smithi122_119671_object62801:head# 2024-03-20T09:04:50.480 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bee5af18:::benchmark_data_smithi122_119671_object6778:head# 2024-03-20T09:04:50.481 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bee63d10:::benchmark_data_smithi122_119671_object95748:head# 2024-03-20T09:04:50.481 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bee6b9d7:::benchmark_data_smithi122_119671_object60480:head# 2024-03-20T09:04:50.481 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bee73543:::benchmark_data_smithi122_119671_object19497:head# 2024-03-20T09:04:50.481 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bee8a42b:::benchmark_data_smithi122_119671_object83568:head# 2024-03-20T09:04:50.482 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:50.483 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bee9a632:::benchmark_data_smithi122_119671_object41591:head# 2024-03-20T09:04:50.483 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:beea8392:::benchmark_data_smithi122_119671_object76336:head# 2024-03-20T09:04:50.484 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:beecdf84:::benchmark_data_smithi122_119671_object7830:head# 2024-03-20T09:04:50.484 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bef05e9a:::benchmark_data_smithi122_119671_object62308:head# 2024-03-20T09:04:50.485 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bef138eb:::benchmark_data_smithi122_119671_object25047:head# 2024-03-20T09:04:50.486 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bef15b1f:::benchmark_data_smithi122_119671_object25445:head# 2024-03-20T09:04:50.486 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bef170fd:::benchmark_data_smithi122_119671_object19670:head# 2024-03-20T09:04:50.487 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bef1a16c:::benchmark_data_smithi122_119671_object79543:head# 2024-03-20T09:04:50.487 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bef1e28c:::benchmark_data_smithi122_119671_object77985:head# 2024-03-20T09:04:50.488 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bef1e508:::benchmark_data_smithi122_119671_object16163:head# 2024-03-20T09:04:50.488 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bef1fd58:::benchmark_data_smithi122_119671_object29461:head# 2024-03-20T09:04:50.489 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bef28402:::benchmark_data_smithi122_119671_object81627:head# 2024-03-20T09:04:50.489 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bef3a9e4:::benchmark_data_smithi122_119671_object71774:head# 2024-03-20T09:04:50.490 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bef3c795:::benchmark_data_smithi122_119671_object92179:head# 2024-03-20T09:04:50.490 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bef7b1f1:::benchmark_data_smithi122_119671_object85953:head# 2024-03-20T09:04:50.490 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bef81064:::benchmark_data_smithi122_119671_object79352:head# 2024-03-20T09:04:50.491 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bef835c1:::benchmark_data_smithi122_119671_object35886:head# 2024-03-20T09:04:50.492 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bef863c6:::benchmark_data_smithi122_119671_object11891:head# 2024-03-20T09:04:50.493 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bef8e81e:::benchmark_data_smithi122_119671_object53557:head# 2024-03-20T09:04:50.494 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bef8fee6:::benchmark_data_smithi122_119671_object98135:head# 2024-03-20T09:04:50.494 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bef917f4:::benchmark_data_smithi122_119671_object78946:head# 2024-03-20T09:04:50.496 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bef926ca:::benchmark_data_smithi122_119671_object16313:head# 2024-03-20T09:04:50.496 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bef94e83:::benchmark_data_smithi122_119671_object10139:head# 2024-03-20T09:04:50.497 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bef98d2b:::benchmark_data_smithi122_119671_object31384:head# 2024-03-20T09:04:50.497 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bef9d3bf:::benchmark_data_smithi122_119671_object34308:head# 2024-03-20T09:04:50.497 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:befa96e4:::benchmark_data_smithi122_119671_object6919:head# 2024-03-20T09:04:50.497 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:befaa0da:::benchmark_data_smithi122_119671_object17983:head# 2024-03-20T09:04:50.497 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:befac815:::benchmark_data_smithi122_119671_object38456:head# 2024-03-20T09:04:50.498 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:befad0f6:::benchmark_data_smithi122_119671_object6580:head# 2024-03-20T09:04:50.498 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:befad3a2:::benchmark_data_smithi122_119671_object10424:head# 2024-03-20T09:04:50.499 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:befaf371:::benchmark_data_smithi122_119671_object81251:head# 2024-03-20T09:04:50.499 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:befb9213:::benchmark_data_smithi122_119671_object47451:head# 2024-03-20T09:04:50.499 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:befcdfe5:::benchmark_data_smithi122_119671_object69879:head# 2024-03-20T09:04:50.500 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:befd4c90:::benchmark_data_smithi122_119671_object31609:head# 2024-03-20T09:04:50.500 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:befe0d1f:::benchmark_data_smithi122_119671_object39395:head# 2024-03-20T09:04:50.502 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:04:50.502 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:befe4987:::benchmark_data_smithi122_119671_object59837:head# 2024-03-20T09:04:50.502 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:befe95a7:::benchmark_data_smithi122_119671_object11350:head# 2024-03-20T09:04:50.503 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:beff7d1f:::benchmark_data_smithi122_119671_object25673:head# 2024-03-20T09:04:50.503 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf009008:::benchmark_data_smithi122_119671_object54006:head# 2024-03-20T09:04:50.503 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf01bac1:::benchmark_data_smithi122_119671_object97046:head# 2024-03-20T09:04:50.503 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf02ff0b:::benchmark_data_smithi122_119671_object83669:head# 2024-03-20T09:04:50.503 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf049a8e:::benchmark_data_smithi122_119671_object77577:head# 2024-03-20T09:04:50.504 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf04c195:::benchmark_data_smithi122_119671_object35164:head# 2024-03-20T09:04:50.504 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf06545f:::benchmark_data_smithi122_119671_object75846:head# 2024-03-20T09:04:50.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:50.505 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:04:50.506 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf06fbff:::benchmark_data_smithi122_119671_object39232:head# 2024-03-20T09:04:50.506 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf072c8e:::benchmark_data_smithi122_119671_object87601:head# 2024-03-20T09:04:50.506 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf075462:::benchmark_data_smithi122_119671_object37950:head# 2024-03-20T09:04:50.506 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf076964:::benchmark_data_smithi122_119671_object39203:head# 2024-03-20T09:04:50.506 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf07a62b:::benchmark_data_smithi122_119671_object84974:head# 2024-03-20T09:04:50.506 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf09fc63:::benchmark_data_smithi122_119671_object60686:head# 2024-03-20T09:04:50.507 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf0a05e6:::benchmark_data_smithi122_119671_object14776:head# 2024-03-20T09:04:50.507 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf0a78fd:::benchmark_data_smithi122_119671_object84726:head# 2024-03-20T09:04:50.508 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf0b6a24:::benchmark_data_smithi122_119671_object92533:head# 2024-03-20T09:04:50.509 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf0ba3b5:::benchmark_data_smithi122_119671_object28886:head# 2024-03-20T09:04:50.509 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf0d9b5c:::benchmark_data_smithi122_119671_object76988:head# 2024-03-20T09:04:50.510 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf0ee001:::benchmark_data_smithi122_119671_object29933:head# 2024-03-20T09:04:50.510 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf0eebd2:::benchmark_data_smithi122_119671_object25611:head# 2024-03-20T09:04:50.510 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf0f6f47:::benchmark_data_smithi122_119671_object69975:head# 2024-03-20T09:04:50.511 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf0fee57:::benchmark_data_smithi122_119671_object82448:head# 2024-03-20T09:04:50.511 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf114505:::benchmark_data_smithi122_119671_object57673:head# 2024-03-20T09:04:50.512 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf11544a:::benchmark_data_smithi122_119671_object99522:head# 2024-03-20T09:04:50.512 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf11f5db:::benchmark_data_smithi122_119671_object13355:head# 2024-03-20T09:04:50.512 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf1205d9:::benchmark_data_smithi122_119671_object99030:head# 2024-03-20T09:04:50.513 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf12700e:::benchmark_data_smithi122_119671_object90586:head# 2024-03-20T09:04:50.513 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf12b382:::benchmark_data_smithi122_119671_object28888:head# 2024-03-20T09:04:50.513 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf13135b:::benchmark_data_smithi122_119671_object37199:head# 2024-03-20T09:04:50.514 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf1314c6:::benchmark_data_smithi122_119671_object90100:head# 2024-03-20T09:04:50.514 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf1472f6:::benchmark_data_smithi122_119671_object89906:head# 2024-03-20T09:04:50.515 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf150d3d:::benchmark_data_smithi122_119671_object86915:head# 2024-03-20T09:04:50.515 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf157643:::benchmark_data_smithi122_119671_object8914:head# 2024-03-20T09:04:50.515 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf1579ba:::benchmark_data_smithi122_119671_object9988:head# 2024-03-20T09:04:50.516 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf159320:::benchmark_data_smithi122_119671_object71585:head# 2024-03-20T09:04:50.516 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf1632a5:::benchmark_data_smithi122_119671_object30394:head# 2024-03-20T09:04:50.517 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf1649fd:::benchmark_data_smithi122_119671_object15880:head# 2024-03-20T09:04:50.517 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf17966e:::benchmark_data_smithi122_119671_object44505:head# 2024-03-20T09:04:50.518 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf17ed6e:::benchmark_data_smithi122_119671_object83144:head# 2024-03-20T09:04:50.518 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf18a5a0:::benchmark_data_smithi122_119671_object24649:head# 2024-03-20T09:04:50.518 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf18cd3e:::benchmark_data_smithi122_119671_object52584:head# 2024-03-20T09:04:50.519 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf1ace22:::benchmark_data_smithi122_119671_object56237:head# 2024-03-20T09:04:50.519 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf1c0524:::benchmark_data_smithi122_119671_object3702:head# 2024-03-20T09:04:50.520 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf1c2f4e:::benchmark_data_smithi122_119671_object97748:head# 2024-03-20T09:04:50.520 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf200d34:::benchmark_data_smithi122_119671_object40213:head# 2024-03-20T09:04:50.520 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf208a40:::benchmark_data_smithi122_119671_object18036:head# 2024-03-20T09:04:50.521 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf2103cd:::benchmark_data_smithi122_119671_object16064:head# 2024-03-20T09:04:50.521 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf211088:::benchmark_data_smithi122_119671_object60321:head# 2024-03-20T09:04:50.522 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf21f281:::benchmark_data_smithi122_119671_object98741:head# 2024-03-20T09:04:50.522 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf257800:::benchmark_data_smithi122_119671_object54698:head# 2024-03-20T09:04:50.523 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf2578b6:::benchmark_data_smithi122_119671_object68062:head# 2024-03-20T09:04:50.523 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf2753fa:::benchmark_data_smithi122_119671_object6812:head# 2024-03-20T09:04:50.523 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf283c74:::benchmark_data_smithi122_119671_object44300:head# 2024-03-20T09:04:50.524 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf285d80:::benchmark_data_smithi122_119671_object41307:head# 2024-03-20T09:04:50.524 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf28c256:::benchmark_data_smithi122_119671_object73115:head# 2024-03-20T09:04:50.525 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf297e27:::benchmark_data_smithi122_119671_object67353:head# 2024-03-20T09:04:50.525 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf2bdcc2:::benchmark_data_smithi122_119671_object19315:head# 2024-03-20T09:04:50.525 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf2bfa73:::benchmark_data_smithi122_119671_object58584:head# 2024-03-20T09:04:50.526 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf2cc131:::benchmark_data_smithi122_119671_object74355:head# 2024-03-20T09:04:50.526 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf2e182d:::benchmark_data_smithi122_119671_object26994:head# 2024-03-20T09:04:50.527 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf2f395a:::benchmark_data_smithi122_119671_object47339:head# 2024-03-20T09:04:50.527 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf311e71:::benchmark_data_smithi122_119671_object42476:head# 2024-03-20T09:04:50.528 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf314f3a:::benchmark_data_smithi122_119671_object84991:head# 2024-03-20T09:04:50.528 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf320351:::benchmark_data_smithi122_119671_object71030:head# 2024-03-20T09:04:50.528 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf32afc0:::benchmark_data_smithi122_119671_object47885:head# 2024-03-20T09:04:50.529 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf343551:::benchmark_data_smithi122_119671_object79810:head# 2024-03-20T09:04:50.529 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf3444a4:::benchmark_data_smithi122_119671_object98418:head# 2024-03-20T09:04:50.529 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf34e7c1:::benchmark_data_smithi122_119671_object49038:head# 2024-03-20T09:04:50.530 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf35a852:::benchmark_data_smithi122_119671_object51893:head# 2024-03-20T09:04:50.530 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf368c16:::benchmark_data_smithi122_119671_object91190:head# 2024-03-20T09:04:50.531 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf383b2b:::benchmark_data_smithi122_119671_object23781:head# 2024-03-20T09:04:50.531 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf38a887:::benchmark_data_smithi122_119671_object41294:head# 2024-03-20T09:04:50.532 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf38eebb:::benchmark_data_smithi122_119671_object53872:head# 2024-03-20T09:04:50.532 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf3a5434:::benchmark_data_smithi122_119671_object43568:head# 2024-03-20T09:04:50.533 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf3c3690:::benchmark_data_smithi122_119671_object75939:head# 2024-03-20T09:04:50.533 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf3c4336:::benchmark_data_smithi122_119671_object77701:head# 2024-03-20T09:04:50.533 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf3c5431:::benchmark_data_smithi122_119671_object59381:head# 2024-03-20T09:04:50.534 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf3c7dfe:::benchmark_data_smithi122_119671_object77426:head# 2024-03-20T09:04:50.534 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf3e1294:::benchmark_data_smithi122_119671_object51170:head# 2024-03-20T09:04:50.535 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf3e2f8d:::benchmark_data_smithi122_119671_object59377:head# 2024-03-20T09:04:50.535 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf3f8582:::benchmark_data_smithi122_119671_object18948:head# 2024-03-20T09:04:50.536 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf3fdd9f:::benchmark_data_smithi122_119671_object87796:head# 2024-03-20T09:04:50.536 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf404997:::benchmark_data_smithi122_119671_object23343:head# 2024-03-20T09:04:50.537 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf40cdbf:::benchmark_data_smithi122_119671_object22632:head# 2024-03-20T09:04:50.537 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf40dcb3:::benchmark_data_smithi122_119671_object92364:head# 2024-03-20T09:04:50.537 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf41b14a:::benchmark_data_smithi122_119671_object44121:head# 2024-03-20T09:04:50.538 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf41d6f8:::benchmark_data_smithi122_119671_object57576:head# 2024-03-20T09:04:50.538 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf424a8c:::benchmark_data_smithi122_119671_object55964:head# 2024-03-20T09:04:50.539 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf42e205:::benchmark_data_smithi122_119671_object11975:head# 2024-03-20T09:04:50.539 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf43cfd1:::benchmark_data_smithi122_119671_object15883:head# 2024-03-20T09:04:50.540 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf43e1bf:::benchmark_data_smithi122_119671_object94552:head# 2024-03-20T09:04:50.540 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf43ec53:::benchmark_data_smithi122_119671_object50325:head# 2024-03-20T09:04:50.541 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf4414f5:::benchmark_data_smithi122_119671_object75297:head# 2024-03-20T09:04:50.541 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf44437c:::benchmark_data_smithi122_119671_object43119:head# 2024-03-20T09:04:50.541 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf4479b2:::benchmark_data_smithi122_119671_object14894:head# 2024-03-20T09:04:50.542 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf4558e3:::benchmark_data_smithi122_119671_object16111:head# 2024-03-20T09:04:50.542 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf4596c9:::benchmark_data_smithi122_119671_object76026:head# 2024-03-20T09:04:50.543 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf464659:::benchmark_data_smithi122_119671_object87873:head# 2024-03-20T09:04:50.543 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf465605:::benchmark_data_smithi122_119671_object44649:head# 2024-03-20T09:04:50.543 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf47f23f:::benchmark_data_smithi122_119671_object39026:head# 2024-03-20T09:04:50.544 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf486739:::benchmark_data_smithi122_119671_object10309:head# 2024-03-20T09:04:50.544 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf487a5b:::benchmark_data_smithi122_119671_object10838:head# 2024-03-20T09:04:50.545 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf48817f:::benchmark_data_smithi122_119671_object36832:head# 2024-03-20T09:04:50.545 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf498351:::benchmark_data_smithi122_119671_object27137:head# 2024-03-20T09:04:50.545 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf49884d:::benchmark_data_smithi122_119671_object31234:head# 2024-03-20T09:04:50.546 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf4a0e61:::benchmark_data_smithi122_119671_object77992:head# 2024-03-20T09:04:50.546 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf4ab5a2:::benchmark_data_smithi122_119671_object6617:head# 2024-03-20T09:04:50.546 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf4b1458:::benchmark_data_smithi122_119671_object38231:head# 2024-03-20T09:04:50.547 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf4bb4fe:::benchmark_data_smithi122_119671_object45479:head# 2024-03-20T09:04:50.547 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf4c33f7:::benchmark_data_smithi122_119671_object72628:head# 2024-03-20T09:04:50.547 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf4ccf40:::benchmark_data_smithi122_119671_object92231:head# 2024-03-20T09:04:50.548 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf4d1c18:::benchmark_data_smithi122_119671_object65018:head# 2024-03-20T09:04:50.548 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf4dce04:::benchmark_data_smithi122_119671_object64404:head# 2024-03-20T09:04:50.549 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf4df44e:::benchmark_data_smithi122_119671_object61035:head# 2024-03-20T09:04:50.549 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf4ef6f8:::benchmark_data_smithi122_119671_object40945:head# 2024-03-20T09:04:50.549 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf4f7be6:::benchmark_data_smithi122_119671_object76814:head# 2024-03-20T09:04:50.550 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf5006bb:::benchmark_data_smithi122_119671_object32803:head# 2024-03-20T09:04:50.550 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf50d36d:::benchmark_data_smithi122_119671_object88095:head# 2024-03-20T09:04:50.551 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf50ead6:::benchmark_data_smithi122_119671_object79328:head# 2024-03-20T09:04:50.551 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf543b6a:::benchmark_data_smithi122_119671_object97845:head# 2024-03-20T09:04:50.553 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf560436:::benchmark_data_smithi122_119671_object93825:head# 2024-03-20T09:04:50.553 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf566150:::benchmark_data_smithi122_119671_object32859:head# 2024-03-20T09:04:50.553 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf571cae:::benchmark_data_smithi122_119671_object36642:head# 2024-03-20T09:04:50.553 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf5865db:::benchmark_data_smithi122_119671_object63547:head# 2024-03-20T09:04:50.554 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf58ce1d:::benchmark_data_smithi122_119671_object52082:head# 2024-03-20T09:04:50.555 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf58d360:::benchmark_data_smithi122_119671_object47687:head# 2024-03-20T09:04:50.555 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf594681:::benchmark_data_smithi122_119671_object17671:head# 2024-03-20T09:04:50.555 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf599dcb:::benchmark_data_smithi122_119671_object48644:head# 2024-03-20T09:04:50.556 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf59e657:::benchmark_data_smithi122_119671_object72832:head# 2024-03-20T09:04:50.556 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf5a94ce:::benchmark_data_smithi122_119671_object32982:head# 2024-03-20T09:04:50.557 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf5bf30b:::benchmark_data_smithi122_119671_object50297:head# 2024-03-20T09:04:50.557 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf5cb558:::benchmark_data_smithi122_119671_object24565:head# 2024-03-20T09:04:50.558 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf5e83a8:::benchmark_data_smithi122_119671_object11887:head# 2024-03-20T09:04:50.558 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf5eaa97:::benchmark_data_smithi122_119671_object92737:head# 2024-03-20T09:04:50.558 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf5f5e54:::benchmark_data_smithi122_119671_object71632:head# 2024-03-20T09:04:50.559 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf5fd96a:::benchmark_data_smithi122_119671_object2520:head# 2024-03-20T09:04:50.560 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf6177c7:::benchmark_data_smithi122_119671_object59382:head# 2024-03-20T09:04:50.560 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf618700:::benchmark_data_smithi122_119671_object43975:head# 2024-03-20T09:04:50.560 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf630644:::benchmark_data_smithi122_119671_object47076:head# 2024-03-20T09:04:50.560 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf630679:::benchmark_data_smithi122_119671_object51092:head# 2024-03-20T09:04:50.561 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf63673b:::benchmark_data_smithi122_119671_object55914:head# 2024-03-20T09:04:50.561 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf639ed7:::benchmark_data_smithi122_119671_object55514:head# 2024-03-20T09:04:50.562 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf64d246:::benchmark_data_smithi122_119671_object57838:head# 2024-03-20T09:04:50.562 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf659e1f:::benchmark_data_smithi122_119671_object27704:head# 2024-03-20T09:04:50.563 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf6703ab:::benchmark_data_smithi122_119671_object63224:head# 2024-03-20T09:04:50.563 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf6a8ade:::benchmark_data_smithi122_119671_object2163:head# 2024-03-20T09:04:50.563 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf6b72a3:::benchmark_data_smithi122_119671_object64017:head# 2024-03-20T09:04:50.564 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf6bb9cd:::benchmark_data_smithi122_119671_object66449:head# 2024-03-20T09:04:50.564 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf6bdca8:::benchmark_data_smithi122_119671_object92552:head# 2024-03-20T09:04:50.565 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf6dfdc8:::benchmark_data_smithi122_119671_object54654:head# 2024-03-20T09:04:50.566 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf6e0607:::benchmark_data_smithi122_119671_object96669:head# 2024-03-20T09:04:50.566 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf6ec1c9:::benchmark_data_smithi122_119671_object58115:head# 2024-03-20T09:04:50.567 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf700429:::benchmark_data_smithi122_119671_object42654:head# 2024-03-20T09:04:50.567 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf7012cb:::benchmark_data_smithi122_119671_object96951:head# 2024-03-20T09:04:50.568 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf703815:::benchmark_data_smithi122_119671_object23806:head# 2024-03-20T09:04:50.568 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf705e11:::benchmark_data_smithi122_119671_object60268:head# 2024-03-20T09:04:50.568 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf706e07:::benchmark_data_smithi122_119671_object51911:head# 2024-03-20T09:04:50.569 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf70ad5a:::benchmark_data_smithi122_119671_object94779:head# 2024-03-20T09:04:50.569 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf70b716:::benchmark_data_smithi122_119671_object14232:head# 2024-03-20T09:04:50.570 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf714fdc:::benchmark_data_smithi122_119671_object60591:head# 2024-03-20T09:04:50.570 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf71c900:::benchmark_data_smithi122_119671_object49810:head# 2024-03-20T09:04:50.575 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf721296:::benchmark_data_smithi122_119671_object41729:head# 2024-03-20T09:04:50.575 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf7222d4:::benchmark_data_smithi122_119671_object52674:head# 2024-03-20T09:04:50.575 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf728a15:::benchmark_data_smithi122_119671_object45395:head# 2024-03-20T09:04:50.575 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf74447c:::benchmark_data_smithi122_119671_object54310:head# 2024-03-20T09:04:50.575 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf74944a:::benchmark_data_smithi122_119671_object10781:head# 2024-03-20T09:04:50.575 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf74ceb5:::benchmark_data_smithi122_119671_object51452:head# 2024-03-20T09:04:50.576 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf765b99:::benchmark_data_smithi122_119671_object37729:head# 2024-03-20T09:04:50.576 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf76619b:::benchmark_data_smithi122_119671_object67165:head# 2024-03-20T09:04:50.576 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf76e5ab:::benchmark_data_smithi122_119671_object62120:head# 2024-03-20T09:04:50.576 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf78c036:::benchmark_data_smithi122_119671_object90881:head# 2024-03-20T09:04:50.576 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf79c34e:::benchmark_data_smithi122_119671_object88020:head# 2024-03-20T09:04:50.576 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf7a6e26:::benchmark_data_smithi122_119671_object27912:head# 2024-03-20T09:04:50.577 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf7ac28b:::benchmark_data_smithi122_119671_object39719:head# 2024-03-20T09:04:50.577 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf7ad551:::benchmark_data_smithi122_119671_object11672:head# 2024-03-20T09:04:50.577 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf7c34a4:::benchmark_data_smithi122_119671_object62783:head# 2024-03-20T09:04:50.577 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf7c6b21:::benchmark_data_smithi122_119671_object42588:head# 2024-03-20T09:04:50.577 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf7d54d2:::benchmark_data_smithi122_119671_object14729:head# 2024-03-20T09:04:50.577 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf7e8ce8:::benchmark_data_smithi122_119671_object72581:head# 2024-03-20T09:04:50.579 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf7fd982:::benchmark_data_smithi122_119671_object27145:head# 2024-03-20T09:04:50.580 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf7fe68a:::benchmark_data_smithi122_119671_object15891:head# 2024-03-20T09:04:50.583 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf819d27:::benchmark_data_smithi122_119671_object56972:head# 2024-03-20T09:04:50.583 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf85a3ee:::benchmark_data_smithi122_119671_object6660:head# 2024-03-20T09:04:50.585 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf891032:::benchmark_data_smithi122_119671_object32357:head# 2024-03-20T09:04:50.586 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf896d1b:::benchmark_data_smithi122_119671_object9378:head# 2024-03-20T09:04:50.586 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf89ebd5:::benchmark_data_smithi122_119671_object69935:head# 2024-03-20T09:04:50.586 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf8a8304:::benchmark_data_smithi122_119671_object52751:head# 2024-03-20T09:04:50.587 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf8a8a8c:::benchmark_data_smithi122_119671_object97628:head# 2024-03-20T09:04:50.587 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf8b750c:::benchmark_data_smithi122_119671_object54145:head# 2024-03-20T09:04:50.588 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf8c1be9:::benchmark_data_smithi122_119671_object91965:head# 2024-03-20T09:04:50.589 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf8cdd0e:::benchmark_data_smithi122_119671_object26127:head# 2024-03-20T09:04:50.589 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf8e6d76:::benchmark_data_smithi122_119671_object74248:head# 2024-03-20T09:04:50.590 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf8f3962:::benchmark_data_smithi122_119671_object10827:head# 2024-03-20T09:04:50.590 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf91b32b:::benchmark_data_smithi122_119671_object4446:head# 2024-03-20T09:04:50.591 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf92f0ea:::benchmark_data_smithi122_119671_object42265:head# 2024-03-20T09:04:50.591 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf93a0d7:::benchmark_data_smithi122_119671_object80485:head# 2024-03-20T09:04:50.591 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf93c3cf:::benchmark_data_smithi122_119671_object12546:head# 2024-03-20T09:04:50.592 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf942496:::benchmark_data_smithi122_119671_object86170:head# 2024-03-20T09:04:50.592 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf9470e1:::benchmark_data_smithi122_119671_object62743:head# 2024-03-20T09:04:50.592 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf94a6a4:::benchmark_data_smithi122_119671_object50080:head# 2024-03-20T09:04:50.593 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf94ace7:::benchmark_data_smithi122_119671_object42920:head# 2024-03-20T09:04:50.593 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf94f251:::benchmark_data_smithi122_119671_object30495:head# 2024-03-20T09:04:50.594 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf9609b0:::benchmark_data_smithi122_119671_object94097:head# 2024-03-20T09:04:50.594 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf988bef:::benchmark_data_smithi122_119671_object46119:head# 2024-03-20T09:04:50.594 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf99acf9:::benchmark_data_smithi122_119671_object92387:head# 2024-03-20T09:04:50.595 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf9a3733:::benchmark_data_smithi122_119671_object9162:head# 2024-03-20T09:04:50.595 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf9a5a41:::benchmark_data_smithi122_119671_object83962:head# 2024-03-20T09:04:50.597 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf9d12e4:::benchmark_data_smithi122_119671_object22080:head# 2024-03-20T09:04:50.597 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf9d5272:::benchmark_data_smithi122_119671_object29647:head# 2024-03-20T09:04:50.598 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bf9e363e:::benchmark_data_smithi122_119671_object81524:head# 2024-03-20T09:04:50.598 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfa06aba:::benchmark_data_smithi122_119671_object63903:head# 2024-03-20T09:04:50.599 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfa10891:::benchmark_data_smithi122_119671_object46680:head# 2024-03-20T09:04:50.599 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfa11979:::benchmark_data_smithi122_119671_object93684:head# 2024-03-20T09:04:50.599 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfa1ac8c:::benchmark_data_smithi122_119671_object7513:head# 2024-03-20T09:04:50.600 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfa1c5d1:::benchmark_data_smithi122_119671_object26393:head# 2024-03-20T09:04:50.603 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfa26516:::benchmark_data_smithi122_119671_object66721:head# 2024-03-20T09:04:50.603 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfa2cf3a:::benchmark_data_smithi122_119671_object82702:head# 2024-03-20T09:04:50.604 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfa36df0:::benchmark_data_smithi122_119671_object41003:head# 2024-03-20T09:04:50.604 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfa375e5:::benchmark_data_smithi122_119671_object50429:head# 2024-03-20T09:04:50.605 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfa559a5:::benchmark_data_smithi122_119671_object41356:head# 2024-03-20T09:04:50.605 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfa6c7c1:::benchmark_data_smithi122_119671_object66189:head# 2024-03-20T09:04:50.605 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:04:50.606 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfa86139:::benchmark_data_smithi122_119671_object76687:head# 2024-03-20T09:04:50.607 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfaa1b35:::benchmark_data_smithi122_119671_object14105:head# 2024-03-20T09:04:50.609 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfaa3826:::benchmark_data_smithi122_119671_object78836:head# 2024-03-20T09:04:50.609 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfaa645f:::benchmark_data_smithi122_119671_object94205:head# 2024-03-20T09:04:50.609 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfaa66af:::benchmark_data_smithi122_119671_object15833:head# 2024-03-20T09:04:50.609 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfaa7c60:::benchmark_data_smithi122_119671_object21510:head# 2024-03-20T09:04:50.609 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfacc072:::benchmark_data_smithi122_119671_object79766:head# 2024-03-20T09:04:50.609 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfacccc7:::benchmark_data_smithi122_119671_object88191:head# 2024-03-20T09:04:50.609 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bface4fb:::benchmark_data_smithi122_119671_object15484:head# 2024-03-20T09:04:50.610 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfad1e28:::benchmark_data_smithi122_119671_object2194:head# 2024-03-20T09:04:50.610 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfad6a43:::benchmark_data_smithi122_119671_object73433:head# 2024-03-20T09:04:50.610 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfad9e91:::benchmark_data_smithi122_119671_object70519:head# 2024-03-20T09:04:50.611 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfae54ce:::benchmark_data_smithi122_119671_object72705:head# 2024-03-20T09:04:50.611 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfafbb88:::benchmark_data_smithi122_119671_object9976:head# 2024-03-20T09:04:50.612 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfafdb35:::benchmark_data_smithi122_119671_object33033:head# 2024-03-20T09:04:50.612 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfaff92c:::benchmark_data_smithi122_119671_object20404:head# 2024-03-20T09:04:50.613 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfb06ac1:::benchmark_data_smithi122_119671_object42086:head# 2024-03-20T09:04:50.613 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfb0cadb:::benchmark_data_smithi122_119671_object39586:head# 2024-03-20T09:04:50.613 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfb18863:::benchmark_data_smithi122_119671_object80475:head# 2024-03-20T09:04:50.616 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfb1eb42:::benchmark_data_smithi122_119671_object2515:head# 2024-03-20T09:04:50.617 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfb247be:::benchmark_data_smithi122_119671_object8990:head# 2024-03-20T09:04:50.618 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfb32b5c:::benchmark_data_smithi122_119671_object6402:head# 2024-03-20T09:04:50.619 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfb5176c:::benchmark_data_smithi122_119671_object4266:head# 2024-03-20T09:04:50.619 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfb5743c:::benchmark_data_smithi122_119671_object84917:head# 2024-03-20T09:04:50.619 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfb5d1eb:::benchmark_data_smithi122_119671_object7007:head# 2024-03-20T09:04:50.619 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfb64736:::benchmark_data_smithi122_119671_object39297:head# 2024-03-20T09:04:50.621 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfb64b71:::benchmark_data_smithi122_119671_object61409:head# 2024-03-20T09:04:50.622 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfb77197:::benchmark_data_smithi122_119671_object27729:head# 2024-03-20T09:04:50.622 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfb84e73:::benchmark_data_smithi122_119671_object17233:head# 2024-03-20T09:04:50.623 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfb883ce:::benchmark_data_smithi122_119671_object12162:head# 2024-03-20T09:04:50.623 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfb8904d:::benchmark_data_smithi122_119671_object39374:head# 2024-03-20T09:04:50.624 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfb8c01a:::benchmark_data_smithi122_119671_object87823:head# 2024-03-20T09:04:50.624 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfb8db47:::benchmark_data_smithi122_119671_object50743:head# 2024-03-20T09:04:50.625 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfb8f5fc:::benchmark_data_smithi122_119671_object92792:head# 2024-03-20T09:04:50.626 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfbae6e8:::benchmark_data_smithi122_119671_object27316:head# 2024-03-20T09:04:50.626 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfbb2932:::benchmark_data_smithi122_119671_object62192:head# 2024-03-20T09:04:50.627 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfbbd2b2:::benchmark_data_smithi122_119671_object36985:head# 2024-03-20T09:04:50.627 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfbbe7c9:::benchmark_data_smithi122_119671_object9684:head# 2024-03-20T09:04:50.627 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfbd5c52:::benchmark_data_smithi122_119671_object85612:head# 2024-03-20T09:04:50.628 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfbda608:::benchmark_data_smithi122_119671_object68913:head# 2024-03-20T09:04:50.628 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfbe5601:::benchmark_data_smithi122_119671_object78502:head# 2024-03-20T09:04:50.628 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfbeeabf:::benchmark_data_smithi122_119671_object69893:head# 2024-03-20T09:04:50.629 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfc02894:::benchmark_data_smithi122_119671_object74153:head# 2024-03-20T09:04:50.629 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfc091ac:::benchmark_data_smithi122_119671_object49237:head# 2024-03-20T09:04:50.630 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfc13454:::benchmark_data_smithi122_119671_object97083:head# 2024-03-20T09:04:50.630 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfc1c368:::benchmark_data_smithi122_119671_object94558:head# 2024-03-20T09:04:50.631 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfc2c174:::benchmark_data_smithi122_119671_object36633:head# 2024-03-20T09:04:50.631 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfc3d06f:::benchmark_data_smithi122_119671_object34929:head# 2024-03-20T09:04:50.631 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfc49334:::benchmark_data_smithi122_119671_object61591:head# 2024-03-20T09:04:50.632 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfc4cdd3:::benchmark_data_smithi122_119671_object89891:head# 2024-03-20T09:04:50.632 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfc5a3b1:::benchmark_data_smithi122_119671_object53223:head# 2024-03-20T09:04:50.633 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfc5a6ba:::benchmark_data_smithi122_119671_object55383:head# 2024-03-20T09:04:50.633 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfc7bcb9:::benchmark_data_smithi122_119671_object13169:head# 2024-03-20T09:04:50.634 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfc84cd3:::benchmark_data_smithi122_119671_object9140:head# 2024-03-20T09:04:50.634 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfc9bbf6:::benchmark_data_smithi122_119671_object94560:head# 2024-03-20T09:04:50.634 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfc9dfe4:::benchmark_data_smithi122_119671_object76315:head# 2024-03-20T09:04:50.635 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfcaa5d1:::benchmark_data_smithi122_119671_object99067:head# 2024-03-20T09:04:50.635 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:50.636 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfcabb75:::benchmark_data_smithi122_119671_object23209:head# 2024-03-20T09:04:50.636 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfcaeb33:::benchmark_data_smithi122_119671_object76681:head# 2024-03-20T09:04:50.636 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfcc5b3f:::benchmark_data_smithi122_119671_object38059:head# 2024-03-20T09:04:50.636 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfccde9b:::benchmark_data_smithi122_119671_object52761:head# 2024-03-20T09:04:50.637 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfcddcea:::benchmark_data_smithi122_119671_object26385:head# 2024-03-20T09:04:50.637 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfcec624:::benchmark_data_smithi122_119671_object90745:head# 2024-03-20T09:04:50.638 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfcf0e53:::benchmark_data_smithi122_119671_object32406:head# 2024-03-20T09:04:50.638 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfd04519:::benchmark_data_smithi122_119671_object8177:head# 2024-03-20T09:04:50.638 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfd11f96:::benchmark_data_smithi122_119671_object53126:head# 2024-03-20T09:04:50.639 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfd34431:::benchmark_data_smithi122_119671_object13433:head# 2024-03-20T09:04:50.639 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfd49be6:::benchmark_data_smithi122_119671_object46334:head# 2024-03-20T09:04:50.640 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfd4b838:::benchmark_data_smithi122_119671_object95678:head# 2024-03-20T09:04:50.642 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfd50c6a:::benchmark_data_smithi122_119671_object91878:head# 2024-03-20T09:04:50.642 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfd56ce4:::benchmark_data_smithi122_119671_object79035:head# 2024-03-20T09:04:50.642 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfd5819f:::benchmark_data_smithi122_119671_object44444:head# 2024-03-20T09:04:50.643 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfd745fa:::benchmark_data_smithi122_119671_object72236:head# 2024-03-20T09:04:50.647 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfd88c25:::benchmark_data_smithi122_119671_object85607:head# 2024-03-20T09:04:50.648 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfdaa325:::benchmark_data_smithi122_119671_object65925:head# 2024-03-20T09:04:50.648 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfdbabdb:::benchmark_data_smithi122_119671_object95738:head# 2024-03-20T09:04:50.648 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfdd0c89:::benchmark_data_smithi122_119671_object86521:head# 2024-03-20T09:04:50.648 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfdd1173:::benchmark_data_smithi122_119671_object3812:head# 2024-03-20T09:04:50.648 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfdd9217:::benchmark_data_smithi122_119671_object68600:head# 2024-03-20T09:04:50.648 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfde2554:::benchmark_data_smithi122_119671_object75145:head# 2024-03-20T09:04:50.648 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfdeb00e:::benchmark_data_smithi122_119671_object76820:head# 2024-03-20T09:04:50.648 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfdfdde4:::benchmark_data_smithi122_119671_object77289:head# 2024-03-20T09:04:50.648 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfe00360:::benchmark_data_smithi122_119671_object29420:head# 2024-03-20T09:04:50.648 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfe103dd:::benchmark_data_smithi122_119671_object2076:head# 2024-03-20T09:04:50.648 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfe17689:::benchmark_data_smithi122_119671_object7384:head# 2024-03-20T09:04:50.648 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfe1e239:::benchmark_data_smithi122_119671_object85874:head# 2024-03-20T09:04:50.649 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfe239a0:::benchmark_data_smithi122_119671_object3316:head# 2024-03-20T09:04:50.649 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfe272e0:::benchmark_data_smithi122_119671_object91247:head# 2024-03-20T09:04:50.649 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfe29792:::benchmark_data_smithi122_119671_object5990:head# 2024-03-20T09:04:50.649 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfe2a18e:::benchmark_data_smithi122_119671_object63975:head# 2024-03-20T09:04:50.650 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfe3035f:::benchmark_data_smithi122_119671_object44819:head# 2024-03-20T09:04:50.650 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfe314c0:::benchmark_data_smithi122_119671_object91742:head# 2024-03-20T09:04:50.650 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfe44865:::benchmark_data_smithi122_119671_object29441:head# 2024-03-20T09:04:50.651 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfe47bde:::benchmark_data_smithi122_119671_object3915:head# 2024-03-20T09:04:50.651 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfe48fcb:::benchmark_data_smithi122_119671_object25023:head# 2024-03-20T09:04:50.652 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfe4fa6f:::benchmark_data_smithi122_119671_object63699:head# 2024-03-20T09:04:50.652 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfe7112f:::benchmark_data_smithi122_119671_object40591:head# 2024-03-20T09:04:50.654 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfe86df6:::benchmark_data_smithi122_119671_object90985:head# 2024-03-20T09:04:50.654 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfe9540c:::benchmark_data_smithi122_119671_object15335:head# 2024-03-20T09:04:50.654 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfe973fd:::benchmark_data_smithi122_119671_object46408:head# 2024-03-20T09:04:50.655 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfe9b754:::benchmark_data_smithi122_119671_object71015:head# 2024-03-20T09:04:50.655 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfea2c0d:::benchmark_data_smithi122_119671_object22757:head# 2024-03-20T09:04:50.655 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfea7df2:::benchmark_data_smithi122_119671_object60995:head# 2024-03-20T09:04:50.656 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfea8fef:::benchmark_data_smithi122_119671_object94401:head# 2024-03-20T09:04:50.656 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfeb83e2:::benchmark_data_smithi122_119671_object86506:head# 2024-03-20T09:04:50.656 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:04:50.657 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfebbcb7:::benchmark_data_smithi122_119671_object26828:head# 2024-03-20T09:04:50.657 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfec307e:::benchmark_data_smithi122_119671_object18304:head# 2024-03-20T09:04:50.657 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfec4999:::benchmark_data_smithi122_119671_object3645:head# 2024-03-20T09:04:50.657 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfeda674:::benchmark_data_smithi122_119671_object9548:head# 2024-03-20T09:04:50.658 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfee81cc:::benchmark_data_smithi122_119671_object47067:head# 2024-03-20T09:04:50.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:50.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:04:50.661 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfeef177:::benchmark_data_smithi122_119671_object20889:head# 2024-03-20T09:04:50.661 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfeef60c:::benchmark_data_smithi122_119671_object91331:head# 2024-03-20T09:04:50.661 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfef26c4:::benchmark_data_smithi122_119671_object65709:head# 2024-03-20T09:04:50.661 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfef7ead:::benchmark_data_smithi122_119671_object17773:head# 2024-03-20T09:04:50.661 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bff0436f:::benchmark_data_smithi122_119671_object78318:head# 2024-03-20T09:04:50.661 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bff2ad38:::benchmark_data_smithi122_119671_object27757:head# 2024-03-20T09:04:50.661 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bff2c8ad:::benchmark_data_smithi122_119671_object11333:head# 2024-03-20T09:04:50.661 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bff30e7d:::benchmark_data_smithi122_119671_object6154:head# 2024-03-20T09:04:50.662 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bff38646:::benchmark_data_smithi122_119671_object82446:head# 2024-03-20T09:04:50.662 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bff3d50c:::benchmark_data_smithi122_119671_object36006:head# 2024-03-20T09:04:50.662 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bff4bcc5:::benchmark_data_smithi122_119671_object40002:head# 2024-03-20T09:04:50.663 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bff68072:::benchmark_data_smithi122_119671_object36396:head# 2024-03-20T09:04:50.663 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bff6efce:::benchmark_data_smithi122_119671_object82416:head# 2024-03-20T09:04:50.663 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bff7326d:::benchmark_data_smithi122_119671_object67460:head# 2024-03-20T09:04:50.664 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bff74b95:::benchmark_data_smithi122_119671_object41766:head# 2024-03-20T09:04:50.664 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bff79aaf:::benchmark_data_smithi122_119671_object93656:head# 2024-03-20T09:04:50.665 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bff7c364:::benchmark_data_smithi122_119671_object87564:head# 2024-03-20T09:04:50.665 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bff8cf1f:::benchmark_data_smithi122_119671_object18766:head# 2024-03-20T09:04:50.665 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bff939ca:::benchmark_data_smithi122_119671_object11279:head# 2024-03-20T09:04:50.666 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bffc4ad8:::benchmark_data_smithi122_119671_object73274:head# 2024-03-20T09:04:50.666 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bffcef6b:::benchmark_data_smithi122_119671_object90569:head# 2024-03-20T09:04:50.667 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfff01d5:::benchmark_data_smithi122_119671_object74042:head# 2024-03-20T09:04:50.667 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfff9394:::benchmark_data_smithi122_119671_object37883:head# 2024-03-20T09:04:50.667 INFO:teuthology.orchestra.run.smithi115.stderr:Read #3:bfff9fcd:::benchmark_data_smithi122_119671_object55710:head# 2024-03-20T09:04:50.670 INFO:teuthology.orchestra.run.smithi115.stderr:Export successful 2024-03-20T09:04:50.760 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:04:50.788 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:50.807 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:04:50.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:50.810 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:04:50.911 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:04:50.941 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:51.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:50 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:04:51.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:50 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/19587992' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:04:51.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:50 smithi122 ceph-mon[73495]: osdmap e285: 12 total, 11 up, 11 in 2024-03-20T09:04:51.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:50 smithi122 ceph-mon[73495]: 3.12 scrub ok 2024-03-20T09:04:51.214 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:50 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:04:51.214 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:50 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/19587992' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:04:51.214 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:50 smithi115 ceph-mon[72476]: osdmap e285: 12 total, 11 up, 11 in 2024-03-20T09:04:51.214 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:50 smithi115 ceph-mon[72476]: 3.12 scrub ok 2024-03-20T09:04:51.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:50 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:04:51.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:50 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/19587992' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:04:51.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:50 smithi026 ceph-mon[68309]: osdmap e285: 12 total, 11 up, 11 in 2024-03-20T09:04:51.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:50 smithi026 ceph-mon[68309]: 3.12 scrub ok 2024-03-20T09:04:51.290 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:04:51.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:51.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:04:51.394 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:04:51.423 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:51.603 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:04:52.075 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -n osd.5 --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --force --op remove --pgid 3.d 2024-03-20T09:04:52.076 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:04:52.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:52.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:04:52.179 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:04:52.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:51 smithi122 ceph-mon[73495]: pgmap v733: 47 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped, 1 activating+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 28 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 6.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 28470/293523 objects degraded (9.699%); 12685/293523 objects misplaced (4.322%) 2024-03-20T09:04:52.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:51 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 28606/294816 objects degraded (9.703%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:04:52.205 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:52.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:51 smithi026 ceph-mon[68309]: pgmap v733: 47 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped, 1 activating+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 28 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 6.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 28470/293523 objects degraded (9.699%); 12685/293523 objects misplaced (4.322%) 2024-03-20T09:04:52.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:51 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 28606/294816 objects degraded (9.703%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:04:52.224 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:04:52.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:52.226 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:04:52.327 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:04:52.353 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:52.374 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/osd.5/config 2024-03-20T09:04:52.374 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:04:52.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:52.377 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:04:52.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:51 smithi115 ceph-mon[72476]: pgmap v733: 47 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped, 1 activating+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 28 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 6.0 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 28470/293523 objects degraded (9.699%); 12685/293523 objects misplaced (4.322%) 2024-03-20T09:04:52.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:51 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 28606/294816 objects degraded (9.703%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:04:52.478 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:04:52.505 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:52.528 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:04:52.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:52.530 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:04:52.631 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:04:52.665 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:52.701 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:04:52.701 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:04:52.704 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:04:52.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:52.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:04:52.713 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:04:52.809 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:04:52.838 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:52.860 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:04:52.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:52.863 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:04:52.964 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:04:52.994 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:53.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:52 smithi122 ceph-mon[73495]: 3.1 deep-scrub starts 2024-03-20T09:04:53.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:52 smithi026 ceph-mon[68309]: 3.1 deep-scrub starts 2024-03-20T09:04:53.250 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:04:53.252 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:52 smithi115 ceph-mon[72476]: 3.1 deep-scrub starts 2024-03-20T09:04:53.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:53.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:04:53.280 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:04:53.354 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:04:53.384 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:53.411 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:04:53.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:53.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:04:53.520 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:04:53.553 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:53.628 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:04:53.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:53.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:04:53.733 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:04:53.764 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:53.865 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:04:53.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:53.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:04:53.904 INFO:teuthology.orchestra.run.smithi115.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-20T09:04:53.972 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:04:54.001 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:54.023 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:04:54.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:54.026 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:04:54.127 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:04:54.153 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:53 smithi115 ceph-mon[72476]: pgmap v734: 47 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 27 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 28792/292206 objects degraded (9.853%); 12151/292206 objects misplaced (4.158%) 2024-03-20T09:04:54.155 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:54.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:53 smithi026 ceph-mon[68309]: pgmap v734: 47 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 27 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 28792/292206 objects degraded (9.853%); 12151/292206 objects misplaced (4.158%) 2024-03-20T09:04:54.357 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:04:54.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:54.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:04:54.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:53 smithi122 ceph-mon[73495]: pgmap v734: 47 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 27 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 28792/292206 objects degraded (9.853%); 12151/292206 objects misplaced (4.158%) 2024-03-20T09:04:54.463 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:04:54.489 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:54.512 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:04:54.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:54.515 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:04:54.616 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:04:54.646 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:54.668 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:04:54.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:54.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:04:54.711 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-20T09:04:54.773 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:04:54.812 INFO:teuthology.orchestra.run.smithi115.stdout: marking collection for removal 2024-03-20T09:04:54.812 INFO:teuthology.orchestra.run.smithi115.stdout:setting '_remove' omap key 2024-03-20T09:04:54.812 INFO:teuthology.orchestra.run.smithi115.stdout:finish_remove_pgs 3.d_head removing 3.d 2024-03-20T09:04:54.834 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:54.913 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:04:54.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:54.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:04:54.938 INFO:teuthology.orchestra.run.smithi115.stdout:Remove successful 2024-03-20T09:04:55.017 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:04:55.050 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:55.177 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:04:55.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:55.183 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:04:55.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:04:55.223 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:04:55.258 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:04:55.284 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:04:55.285 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:04:55.309 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:04:55.318 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:55.332 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:04:55.347 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:04:55.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:55.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:04:55.356 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:04:55.385 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:04:55.407 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:04:55.433 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:04:55.454 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:04:55.457 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:04:55.466 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:55 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3580245852' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:04:55.482 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:04:55.482 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:04:55.482 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:04:55.482 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:04:55.483 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:04:55.483 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:04:55.483 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-20T09:04:55.483 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:04:55.483 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:04:55.483 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:04:55.483 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:04:55.483 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:04:55.483 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:04:55.492 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:55.514 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:04:55.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:55.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:04:55.620 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:04:55.651 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:55.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:55 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3580245852' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:04:55.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:55 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3580245852' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:04:55.727 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:04:55.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:55.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:04:55.834 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:04:55.863 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:56.222 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -n osd.5 --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.3.d.5 2024-03-20T09:04:56.305 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:04:56.305 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:04:56.312 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:04:56.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:56.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:04:56.415 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:04:56.440 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:56 smithi115 ceph-mon[72476]: pgmap v735: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 28618/289704 objects degraded (9.878%); 12023/289704 objects misplaced (4.150%) 2024-03-20T09:04:56.443 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:56.468 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:04:56.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:56.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:04:56.478 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/osd.5/config 2024-03-20T09:04:56.575 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:04:56.604 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:56.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:56 smithi122 ceph-mon[73495]: pgmap v735: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 28618/289704 objects degraded (9.878%); 12023/289704 objects misplaced (4.150%) 2024-03-20T09:04:56.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:56 smithi026 ceph-mon[68309]: pgmap v735: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 28618/289704 objects degraded (9.878%); 12023/289704 objects misplaced (4.150%) 2024-03-20T09:04:56.726 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:04:56.804 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:04:56.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:56.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:04:56.917 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:04:56.944 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:56.963 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:04:56.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:56.966 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:04:57.067 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:04:57.096 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:57.346 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:04:57.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:57.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:04:57.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:57 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 28618/289704 objects degraded (9.878%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:04:57.451 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:04:57.481 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:57.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:57 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 28618/289704 objects degraded (9.878%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:04:57.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:57 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 28618/289704 objects degraded (9.878%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:04:58.033 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:04:58.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:58.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:04:58.060 INFO:teuthology.orchestra.run.smithi115.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-20T09:04:58.138 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:04:58.164 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:58.182 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:04:58.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:58.184 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:04:58.286 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:04:58.306 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:04:58.312 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:58.330 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:04:58.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:58.333 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:04:58.336 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:58 smithi115 ceph-mon[72476]: pgmap v736: 47 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 28514/288747 objects degraded (9.875%); 11994/288747 objects misplaced (4.154%) 2024-03-20T09:04:58.434 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:04:58.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:58 smithi122 ceph-mon[73495]: pgmap v736: 47 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 28514/288747 objects degraded (9.875%); 11994/288747 objects misplaced (4.154%) 2024-03-20T09:04:58.461 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:58.605 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:04:58.606 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:58 smithi026 ceph-mon[68309]: pgmap v736: 47 pgs: 1 active+clean+remapped, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 28514/288747 objects degraded (9.875%); 11994/288747 objects misplaced (4.154%) 2024-03-20T09:04:58.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:58.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:04:58.709 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:04:58.739 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:58.816 INFO:teuthology.orchestra.run.smithi115.stdout:get_pg_num_history pg_num_history pg_num_history(e273 pg_nums {1={19=1,181=11,252=21},2={121=16},3={204=16,231=26}} deleted_pools 199,2) 2024-03-20T09:04:58.816 INFO:teuthology.orchestra.run.smithi115.stdout:Importing pgid 3.d 2024-03-20T09:04:58.822 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0031471:::benchmark_data_smithi122_119671_object58867:head# 2024-03-20T09:04:58.822 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.822 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0033314:::benchmark_data_smithi122_119671_object89042:head# 2024-03-20T09:04:58.822 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.823 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b00350bb:::benchmark_data_smithi122_119671_object99725:head# 2024-03-20T09:04:58.823 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.825 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b00424b4:::benchmark_data_smithi122_119671_object49302:head# 2024-03-20T09:04:58.825 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.825 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0051513:::benchmark_data_smithi122_119671_object51935:head# 2024-03-20T09:04:58.825 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.825 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0058005:::benchmark_data_smithi122_119671_object21971:head# 2024-03-20T09:04:58.825 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.825 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0062a02:::benchmark_data_smithi122_119671_object55216:head# 2024-03-20T09:04:58.825 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.825 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b00670ba:::benchmark_data_smithi122_119671_object23821:head# 2024-03-20T09:04:58.825 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.826 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b006bb2c:::benchmark_data_smithi122_119671_object4970:head# 2024-03-20T09:04:58.826 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.826 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b006df72:::benchmark_data_smithi122_119671_object38541:head# 2024-03-20T09:04:58.826 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.826 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0075821:::benchmark_data_smithi122_119671_object9798:head# 2024-03-20T09:04:58.826 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.826 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0087e88:::benchmark_data_smithi122_119671_object35896:head# 2024-03-20T09:04:58.826 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.826 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0097cb5:::benchmark_data_smithi122_119671_object28209:head# 2024-03-20T09:04:58.826 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.839 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b00a4363:::benchmark_data_smithi122_119671_object14608:head# 2024-03-20T09:04:58.840 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.840 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b00a91e6:::benchmark_data_smithi122_119671_object85862:head# 2024-03-20T09:04:58.840 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.840 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b00b2e23:::benchmark_data_smithi122_119671_object30901:head# 2024-03-20T09:04:58.840 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.840 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b00b4fb7:::benchmark_data_smithi122_119671_object84815:head# 2024-03-20T09:04:58.840 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.841 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b00c2228:::benchmark_data_smithi122_119671_object59276:head# 2024-03-20T09:04:58.841 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.841 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b00c44b2:::benchmark_data_smithi122_119671_object2674:head# 2024-03-20T09:04:58.841 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.841 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b00c90c6:::benchmark_data_smithi122_119671_object54403:head# 2024-03-20T09:04:58.841 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.841 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b00d5f32:::benchmark_data_smithi122_119671_object77195:head# 2024-03-20T09:04:58.841 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.841 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b00d74a6:::benchmark_data_smithi122_119671_object85488:head# 2024-03-20T09:04:58.841 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.841 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b00e59db:::benchmark_data_smithi122_119671_object35545:head# 2024-03-20T09:04:58.841 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.842 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b00ef923:::benchmark_data_smithi122_119671_object28263:head# 2024-03-20T09:04:58.842 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.842 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b00f7737:::benchmark_data_smithi122_119671_object10228:head# 2024-03-20T09:04:58.842 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.842 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b00f8dfc:::benchmark_data_smithi122_119671_object84523:head# 2024-03-20T09:04:58.842 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.842 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b00fc565:::benchmark_data_smithi122_119671_object70527:head# 2024-03-20T09:04:58.842 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.842 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b01056b3:::benchmark_data_smithi122_119671_object49802:head# 2024-03-20T09:04:58.842 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.842 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0113397:::benchmark_data_smithi122_119671_object30866:head# 2024-03-20T09:04:58.842 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.842 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b011a669:::benchmark_data_smithi122_119671_object69855:head# 2024-03-20T09:04:58.843 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.843 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b012015f:::benchmark_data_smithi122_119671_object68636:head# 2024-03-20T09:04:58.843 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.843 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0128f1c:::benchmark_data_smithi122_119671_object86793:head# 2024-03-20T09:04:58.843 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.843 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b012bc68:::benchmark_data_smithi122_119671_object50838:head# 2024-03-20T09:04:58.843 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.843 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b012dcee:::benchmark_data_smithi122_119671_object64248:head# 2024-03-20T09:04:58.843 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.843 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b01489b5:::benchmark_data_smithi122_119671_object37405:head# 2024-03-20T09:04:58.843 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.843 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0156536:::benchmark_data_smithi122_119671_object31524:head# 2024-03-20T09:04:58.843 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.844 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0159ba6:::benchmark_data_smithi122_119671_object39452:head# 2024-03-20T09:04:58.844 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.844 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0174e02:::benchmark_data_smithi122_119671_object3715:head# 2024-03-20T09:04:58.844 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.844 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b01839c1:::benchmark_data_smithi122_119671_object9885:head# 2024-03-20T09:04:58.844 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.844 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b018cf96:::benchmark_data_smithi122_119671_object68414:head# 2024-03-20T09:04:58.844 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.844 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b019b426:::benchmark_data_smithi122_119671_object57153:head# 2024-03-20T09:04:58.844 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.845 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b01a9b4d:::benchmark_data_smithi122_119671_object11135:head# 2024-03-20T09:04:58.845 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.845 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b01af513:::benchmark_data_smithi122_119671_object83415:head# 2024-03-20T09:04:58.845 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.845 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b01bbc79:::benchmark_data_smithi122_119671_object14714:head# 2024-03-20T09:04:58.845 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.845 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b01c6931:::benchmark_data_smithi122_119671_object53383:head# 2024-03-20T09:04:58.845 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.846 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b01d4e2b:::benchmark_data_smithi122_119671_object76655:head# 2024-03-20T09:04:58.846 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.846 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b01df5c7:::benchmark_data_smithi122_119671_object5659:head# 2024-03-20T09:04:58.846 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.846 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b01e4e2c:::benchmark_data_smithi122_119671_object49358:head# 2024-03-20T09:04:58.846 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.846 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b01f0010:::benchmark_data_smithi122_119671_object51691:head# 2024-03-20T09:04:58.846 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.846 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b01f9a18:::benchmark_data_smithi122_119671_object62016:head# 2024-03-20T09:04:58.846 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.846 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0202d9c:::benchmark_data_smithi122_119671_object13902:head# 2024-03-20T09:04:58.846 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.847 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b021359b:::benchmark_data_smithi122_119671_object55417:head# 2024-03-20T09:04:58.847 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.847 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b021afa6:::benchmark_data_smithi122_119671_object94327:head# 2024-03-20T09:04:58.847 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.847 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b021fbea:::benchmark_data_smithi122_119671_object66761:head# 2024-03-20T09:04:58.847 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.847 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0226064:::benchmark_data_smithi122_119671_object36050:head# 2024-03-20T09:04:58.847 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.847 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b022f606:::benchmark_data_smithi122_119671_object88507:head# 2024-03-20T09:04:58.847 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.847 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b025599e:::benchmark_data_smithi122_119671_object9541:head# 2024-03-20T09:04:58.847 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.847 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b025fe5b:::benchmark_data_smithi122_119671_object83150:head# 2024-03-20T09:04:58.848 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.848 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0274cf6:::benchmark_data_smithi122_119671_object48421:head# 2024-03-20T09:04:58.848 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.848 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b02844cb:::benchmark_data_smithi122_119671_object41052:head# 2024-03-20T09:04:58.848 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.848 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b02a6a47:::benchmark_data_smithi122_119671_object69995:head# 2024-03-20T09:04:58.848 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.848 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b02a8951:::benchmark_data_smithi122_119671_object79865:head# 2024-03-20T09:04:58.848 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.848 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b02a9c70:::benchmark_data_smithi122_119671_object10523:head# 2024-03-20T09:04:58.848 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.848 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b02af41e:::benchmark_data_smithi122_119671_object83315:head# 2024-03-20T09:04:58.848 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.849 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b02c8af0:::benchmark_data_smithi122_119671_object69622:head# 2024-03-20T09:04:58.849 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.849 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b02ca104:::benchmark_data_smithi122_119671_object40282:head# 2024-03-20T09:04:58.849 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.849 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b02cf590:::benchmark_data_smithi122_119671_object87299:head# 2024-03-20T09:04:58.849 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.849 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b02d476c:::benchmark_data_smithi122_119671_object38004:head# 2024-03-20T09:04:58.849 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.849 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b02dc697:::benchmark_data_smithi122_119671_object62495:head# 2024-03-20T09:04:58.849 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.849 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b02e2362:::benchmark_data_smithi122_119671_object86290:head# 2024-03-20T09:04:58.849 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.849 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b02f4561:::benchmark_data_smithi122_119671_object82223:head# 2024-03-20T09:04:58.849 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.870 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b02ffc1a:::benchmark_data_smithi122_119671_object98092:head# 2024-03-20T09:04:58.870 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.870 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b03037a7:::benchmark_data_smithi122_119671_object44634:head# 2024-03-20T09:04:58.870 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.870 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0313eb8:::benchmark_data_smithi122_119671_object40708:head# 2024-03-20T09:04:58.870 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.871 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b031c65f:::benchmark_data_smithi122_119671_object36907:head# 2024-03-20T09:04:58.871 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.871 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0325d9c:::benchmark_data_smithi122_119671_object9777:head# 2024-03-20T09:04:58.871 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.871 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b032f23c:::benchmark_data_smithi122_119671_object84716:head# 2024-03-20T09:04:58.871 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.871 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b032fc07:::benchmark_data_smithi122_119671_object58159:head# 2024-03-20T09:04:58.871 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.871 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0333845:::benchmark_data_smithi122_119671_object49335:head# 2024-03-20T09:04:58.871 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.871 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b033455b:::benchmark_data_smithi122_119671_object84945:head# 2024-03-20T09:04:58.871 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.871 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b033f5ff:::benchmark_data_smithi122_119671_object29977:head# 2024-03-20T09:04:58.872 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.872 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b034bec2:::benchmark_data_smithi122_119671_object26621:head# 2024-03-20T09:04:58.872 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.872 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b035486f:::benchmark_data_smithi122_119671_object48612:head# 2024-03-20T09:04:58.872 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.872 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0360a7d:::benchmark_data_smithi122_119671_object70438:head# 2024-03-20T09:04:58.872 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.872 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b03620eb:::benchmark_data_smithi122_119671_object24903:head# 2024-03-20T09:04:58.872 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.872 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0363a3e:::benchmark_data_smithi122_119671_object3849:head# 2024-03-20T09:04:58.872 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.872 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b036e4fa:::benchmark_data_smithi122_119671_object42649:head# 2024-03-20T09:04:58.872 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.873 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0371f13:::benchmark_data_smithi122_119671_object54710:head# 2024-03-20T09:04:58.873 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.873 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0385716:::benchmark_data_smithi122_119671_object87948:head# 2024-03-20T09:04:58.873 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.873 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0389f52:::benchmark_data_smithi122_119671_object90186:head# 2024-03-20T09:04:58.873 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.873 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b038baa5:::benchmark_data_smithi122_119671_object74580:head# 2024-03-20T09:04:58.873 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.873 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b039a049:::benchmark_data_smithi122_119671_object25546:head# 2024-03-20T09:04:58.873 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.873 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b039aece:::benchmark_data_smithi122_119671_object46005:head# 2024-03-20T09:04:58.873 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.873 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b03a31dd:::benchmark_data_smithi122_119671_object66685:head# 2024-03-20T09:04:58.873 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.874 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b03b19d9:::benchmark_data_smithi122_119671_object47270:head# 2024-03-20T09:04:58.874 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.874 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b03b563d:::benchmark_data_smithi122_119671_object7362:head# 2024-03-20T09:04:58.874 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.874 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b03b97aa:::benchmark_data_smithi122_119671_object15624:head# 2024-03-20T09:04:58.874 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.874 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b03c9b8b:::benchmark_data_smithi122_119671_object90627:head# 2024-03-20T09:04:58.874 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.874 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b03d9be6:::benchmark_data_smithi122_119671_object96333:head# 2024-03-20T09:04:58.874 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.874 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b03e14b0:::benchmark_data_smithi122_119671_object11427:head# 2024-03-20T09:04:58.874 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.874 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0405bc0:::benchmark_data_smithi122_119671_object10688:head# 2024-03-20T09:04:58.875 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.875 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b040fa09:::benchmark_data_smithi122_119671_object36069:head# 2024-03-20T09:04:58.875 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.875 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0422a09:::benchmark_data_smithi122_119671_object71802:head# 2024-03-20T09:04:58.875 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.875 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b043ab7a:::benchmark_data_smithi122_119671_object9079:head# 2024-03-20T09:04:58.875 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.875 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b043ca99:::benchmark_data_smithi122_119671_object65584:head# 2024-03-20T09:04:58.875 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.875 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0451505:::benchmark_data_smithi122_119671_object87515:head# 2024-03-20T09:04:58.875 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.875 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0451cb5:::benchmark_data_smithi122_119671_object84713:head# 2024-03-20T09:04:58.875 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.875 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0452724:::benchmark_data_smithi122_119671_object34581:head# 2024-03-20T09:04:58.876 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.876 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b04636e6:::benchmark_data_smithi122_119671_object92591:head# 2024-03-20T09:04:58.876 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.876 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b046ae84:::benchmark_data_smithi122_119671_object5081:head# 2024-03-20T09:04:58.876 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.876 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b047147a:::benchmark_data_smithi122_119671_object24333:head# 2024-03-20T09:04:58.876 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.876 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b047dd04:::benchmark_data_smithi122_119671_object51653:head# 2024-03-20T09:04:58.876 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.876 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0487075:::benchmark_data_smithi122_119671_object75736:head# 2024-03-20T09:04:58.876 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.876 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0489aaa:::benchmark_data_smithi122_119671_object72095:head# 2024-03-20T09:04:58.876 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.876 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b048fec6:::benchmark_data_smithi122_119671_object16721:head# 2024-03-20T09:04:58.877 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.877 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0496317:::benchmark_data_smithi122_119671_object27701:head# 2024-03-20T09:04:58.877 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.877 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0497297:::benchmark_data_smithi122_119671_object90113:head# 2024-03-20T09:04:58.877 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.877 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0499edc:::benchmark_data_smithi122_119671_object58885:head# 2024-03-20T09:04:58.877 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.877 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b04bee2e:::benchmark_data_smithi122_119671_object86376:head# 2024-03-20T09:04:58.877 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.877 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b04cf6a3:::benchmark_data_smithi122_119671_object81168:head# 2024-03-20T09:04:58.877 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.877 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b04dd9be:::benchmark_data_smithi122_119671_object30980:head# 2024-03-20T09:04:58.877 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.877 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b04e364c:::benchmark_data_smithi122_119671_object70388:head# 2024-03-20T09:04:58.878 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.878 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b04e929c:::benchmark_data_smithi122_119671_object78730:head# 2024-03-20T09:04:58.878 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.878 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b04f7078:::benchmark_data_smithi122_119671_object54988:head# 2024-03-20T09:04:58.878 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.878 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b05006e4:::benchmark_data_smithi122_119671_object34791:head# 2024-03-20T09:04:58.878 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.878 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b050fff1:::benchmark_data_smithi122_119671_object78826:head# 2024-03-20T09:04:58.878 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.878 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0510552:::benchmark_data_smithi122_119671_object78776:head# 2024-03-20T09:04:58.878 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.878 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b052278f:::benchmark_data_smithi122_119671_object76364:head# 2024-03-20T09:04:58.878 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.879 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0522d63:::benchmark_data_smithi122_119671_object96352:head# 2024-03-20T09:04:58.879 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.879 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b052943c:::benchmark_data_smithi122_119671_object20315:head# 2024-03-20T09:04:58.879 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.879 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0537ed9:::benchmark_data_smithi122_119671_object91918:head# 2024-03-20T09:04:58.879 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.879 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0542746:::benchmark_data_smithi122_119671_object80657:head# 2024-03-20T09:04:58.879 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.879 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b055046b:::benchmark_data_smithi122_119671_object84043:head# 2024-03-20T09:04:58.879 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.880 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b05510c5:::benchmark_data_smithi122_119671_object27982:head# 2024-03-20T09:04:58.880 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.880 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0554bfb:::benchmark_data_smithi122_119671_object30501:head# 2024-03-20T09:04:58.880 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.880 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b056295b:::benchmark_data_smithi122_119671_object5881:head# 2024-03-20T09:04:58.880 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.880 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0576026:::benchmark_data_smithi122_119671_object57052:head# 2024-03-20T09:04:58.880 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.880 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0584395:::benchmark_data_smithi122_119671_object87836:head# 2024-03-20T09:04:58.880 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.880 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b05a7e36:::benchmark_data_smithi122_119671_object63637:head# 2024-03-20T09:04:58.880 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.880 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b05aacf2:::benchmark_data_smithi122_119671_object7859:head# 2024-03-20T09:04:58.880 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.881 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b05b4219:::benchmark_data_smithi122_119671_object7902:head# 2024-03-20T09:04:58.881 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.881 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b05b6897:::benchmark_data_smithi122_119671_object66539:head# 2024-03-20T09:04:58.881 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.881 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b05cd655:::benchmark_data_smithi122_119671_object96258:head# 2024-03-20T09:04:58.881 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.881 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b05d2b1c:::benchmark_data_smithi122_119671_object69714:head# 2024-03-20T09:04:58.881 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.881 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b05dfa17:::benchmark_data_smithi122_119671_object6391:head# 2024-03-20T09:04:58.881 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.881 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b05eafcb:::benchmark_data_smithi122_119671_object85560:head# 2024-03-20T09:04:58.881 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.881 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b05fa81f:::benchmark_data_smithi122_119671_object55984:head# 2024-03-20T09:04:58.881 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.882 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b060edcf:::benchmark_data_smithi122_119671_object38452:head# 2024-03-20T09:04:58.882 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.882 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0621538:::benchmark_data_smithi122_119671_object1968:head# 2024-03-20T09:04:58.882 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.882 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b06586dd:::benchmark_data_smithi122_119671_object30565:head# 2024-03-20T09:04:58.882 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.882 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b065fc9c:::benchmark_data_smithi122_119671_object29185:head# 2024-03-20T09:04:58.882 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.882 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0681ad1:::benchmark_data_smithi122_119671_object75132:head# 2024-03-20T09:04:58.882 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.882 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0687022:::benchmark_data_smithi122_119671_object49731:head# 2024-03-20T09:04:58.882 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.882 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b068e20d:::benchmark_data_smithi122_119671_object30444:head# 2024-03-20T09:04:58.882 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.883 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0691003:::benchmark_data_smithi122_119671_object4744:head# 2024-03-20T09:04:58.883 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.883 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b06a54e5:::benchmark_data_smithi122_119671_object82334:head# 2024-03-20T09:04:58.883 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.883 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b06aeb5a:::benchmark_data_smithi122_119671_object87650:head# 2024-03-20T09:04:58.883 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.883 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b06bdb38:::benchmark_data_smithi122_119671_object64347:head# 2024-03-20T09:04:58.883 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.883 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b06c5af6:::benchmark_data_smithi122_119671_object10664:head# 2024-03-20T09:04:58.883 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.883 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b06d6033:::benchmark_data_smithi122_119671_object12750:head# 2024-03-20T09:04:58.883 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.883 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b06e2d2e:::benchmark_data_smithi122_119671_object64587:head# 2024-03-20T09:04:58.883 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.883 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b06ed7f1:::benchmark_data_smithi122_119671_object78845:head# 2024-03-20T09:04:58.884 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.884 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b06ffe42:::benchmark_data_smithi122_119671_object45956:head# 2024-03-20T09:04:58.884 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.884 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b070296b:::benchmark_data_smithi122_119671_object57282:head# 2024-03-20T09:04:58.884 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.884 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b070608f:::benchmark_data_smithi122_119671_object20268:head# 2024-03-20T09:04:58.884 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.884 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b070ce9f:::benchmark_data_smithi122_119671_object43292:head# 2024-03-20T09:04:58.884 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.884 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0717535:::benchmark_data_smithi122_119671_object51066:head# 2024-03-20T09:04:58.884 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.884 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0724009:::benchmark_data_smithi122_119671_object34637:head# 2024-03-20T09:04:58.884 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.885 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b07326cf:::benchmark_data_smithi122_119671_object5768:head# 2024-03-20T09:04:58.885 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.885 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0740939:::benchmark_data_smithi122_119671_object43997:head# 2024-03-20T09:04:58.885 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.885 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b074f43f:::benchmark_data_smithi122_119671_object92165:head# 2024-03-20T09:04:58.885 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.885 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b07597e1:::benchmark_data_smithi122_119671_object12920:head# 2024-03-20T09:04:58.885 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.885 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b075acbc:::benchmark_data_smithi122_119671_object73890:head# 2024-03-20T09:04:58.885 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.885 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0774a20:::benchmark_data_smithi122_119671_object51787:head# 2024-03-20T09:04:58.885 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.885 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0780112:::benchmark_data_smithi122_119671_object7868:head# 2024-03-20T09:04:58.885 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.886 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b07831d1:::benchmark_data_smithi122_119671_object22394:head# 2024-03-20T09:04:58.886 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.886 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b07955fd:::benchmark_data_smithi122_119671_object2819:head# 2024-03-20T09:04:58.886 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.886 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b07a13b1:::benchmark_data_smithi122_119671_object25568:head# 2024-03-20T09:04:58.886 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.886 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b07a87b2:::benchmark_data_smithi122_119671_object56725:head# 2024-03-20T09:04:58.886 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.886 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b07ab961:::benchmark_data_smithi122_119671_object22298:head# 2024-03-20T09:04:58.886 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.886 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b07c8987:::benchmark_data_smithi122_119671_object78868:head# 2024-03-20T09:04:58.886 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.886 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b07cfb6d:::benchmark_data_smithi122_119671_object29179:head# 2024-03-20T09:04:58.886 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.887 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b07e98fa:::benchmark_data_smithi122_119671_object22289:head# 2024-03-20T09:04:58.887 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.887 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b07f3298:::benchmark_data_smithi122_119671_object58680:head# 2024-03-20T09:04:58.887 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.887 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b07fa2f3:::benchmark_data_smithi122_119671_object92125:head# 2024-03-20T09:04:58.887 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.887 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b07fd3a8:::benchmark_data_smithi122_119671_object25248:head# 2024-03-20T09:04:58.887 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.887 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0807059:::benchmark_data_smithi122_119671_object66325:head# 2024-03-20T09:04:58.887 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.887 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b081bf04:::benchmark_data_smithi122_119671_object24877:head# 2024-03-20T09:04:58.887 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.887 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b082d912:::benchmark_data_smithi122_119671_object49496:head# 2024-03-20T09:04:58.887 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.888 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0836c13:::benchmark_data_smithi122_119671_object76194:head# 2024-03-20T09:04:58.888 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.888 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b083766b:::benchmark_data_smithi122_119671_object52386:head# 2024-03-20T09:04:58.888 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.888 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b083af58:::benchmark_data_smithi122_119671_object51057:head# 2024-03-20T09:04:58.888 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.888 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b083e2ae:::benchmark_data_smithi122_119671_object47903:head# 2024-03-20T09:04:58.888 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.888 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0845671:::benchmark_data_smithi122_119671_object19363:head# 2024-03-20T09:04:58.888 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.888 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b084b221:::benchmark_data_smithi122_119671_object68735:head# 2024-03-20T09:04:58.888 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.888 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b08574f3:::benchmark_data_smithi122_119671_object74086:head# 2024-03-20T09:04:58.888 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.889 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b085b033:::benchmark_data_smithi122_119671_object78546:head# 2024-03-20T09:04:58.889 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.889 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:04:58.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:58.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:04:58.940 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0866967:::benchmark_data_smithi122_119671_object24521:head# 2024-03-20T09:04:58.940 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.940 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b08814e0:::benchmark_data_smithi122_119671_object73084:head# 2024-03-20T09:04:58.941 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.941 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0885e58:::benchmark_data_smithi122_119671_object47522:head# 2024-03-20T09:04:58.941 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.941 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b088718a:::benchmark_data_smithi122_119671_object32164:head# 2024-03-20T09:04:58.941 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.941 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0888500:::benchmark_data_smithi122_119671_object12329:head# 2024-03-20T09:04:58.941 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.941 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0888a20:::benchmark_data_smithi122_119671_object65600:head# 2024-03-20T09:04:58.941 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.941 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b08973c5:::benchmark_data_smithi122_119671_object49911:head# 2024-03-20T09:04:58.941 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.941 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b089e7a5:::benchmark_data_smithi122_119671_object11189:head# 2024-03-20T09:04:58.941 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.941 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b08a6077:::benchmark_data_smithi122_119671_object85292:head# 2024-03-20T09:04:58.942 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.942 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b08a6e52:::benchmark_data_smithi122_119671_object99350:head# 2024-03-20T09:04:58.942 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.942 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b08aa0e9:::benchmark_data_smithi122_119671_object43343:head# 2024-03-20T09:04:58.942 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.942 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b08adff7:::benchmark_data_smithi122_119671_object39141:head# 2024-03-20T09:04:58.942 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.942 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b08aedb4:::benchmark_data_smithi122_119671_object38236:head# 2024-03-20T09:04:58.942 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.942 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b08b18d2:::benchmark_data_smithi122_119671_object23034:head# 2024-03-20T09:04:58.942 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.942 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b08b5e8d:::benchmark_data_smithi122_119671_object52138:head# 2024-03-20T09:04:58.942 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.942 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b08c35cb:::benchmark_data_smithi122_119671_object70468:head# 2024-03-20T09:04:58.943 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.943 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b08c7b97:::benchmark_data_smithi122_119671_object88535:head# 2024-03-20T09:04:58.943 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.943 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b08e9aac:::benchmark_data_smithi122_119671_object44620:head# 2024-03-20T09:04:58.943 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.943 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b08ea8e9:::benchmark_data_smithi122_119671_object89499:head# 2024-03-20T09:04:58.943 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.943 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b08f80a3:::benchmark_data_smithi122_119671_object38416:head# 2024-03-20T09:04:58.943 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.943 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b08f93c0:::benchmark_data_smithi122_119671_object53738:head# 2024-03-20T09:04:58.943 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.943 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b09043dd:::benchmark_data_smithi122_119671_object69649:head# 2024-03-20T09:04:58.943 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.943 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b09057ab:::benchmark_data_smithi122_119671_object22895:head# 2024-03-20T09:04:58.944 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.944 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0914683:::benchmark_data_smithi122_119671_object4627:head# 2024-03-20T09:04:58.944 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.944 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0918973:::benchmark_data_smithi122_119671_object46641:head# 2024-03-20T09:04:58.944 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.944 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b09219ae:::benchmark_data_smithi122_119671_object16852:head# 2024-03-20T09:04:58.944 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.944 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0926805:::benchmark_data_smithi122_119671_object20227:head# 2024-03-20T09:04:58.944 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.944 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b092e190:::benchmark_data_smithi122_119671_object27866:head# 2024-03-20T09:04:58.944 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.944 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b093466f:::benchmark_data_smithi122_119671_object65155:head# 2024-03-20T09:04:58.944 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.944 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0942f0b:::benchmark_data_smithi122_119671_object32943:head# 2024-03-20T09:04:58.945 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.945 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b09451b5:::benchmark_data_smithi122_119671_object41406:head# 2024-03-20T09:04:58.945 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.945 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b095d025:::benchmark_data_smithi122_119671_object44500:head# 2024-03-20T09:04:58.945 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.945 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0966aa8:::benchmark_data_smithi122_119671_object85132:head# 2024-03-20T09:04:58.945 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.945 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b09766d6:::benchmark_data_smithi122_119671_object80445:head# 2024-03-20T09:04:58.945 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.945 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b098cd44:::benchmark_data_smithi122_119671_object47761:head# 2024-03-20T09:04:58.945 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.945 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b098e8db:::benchmark_data_smithi122_119671_object12024:head# 2024-03-20T09:04:58.945 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.945 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b09a4781:::benchmark_data_smithi122_119671_object77463:head# 2024-03-20T09:04:58.946 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.946 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b09b8cec:::benchmark_data_smithi122_119671_object47863:head# 2024-03-20T09:04:58.946 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.946 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b09bb3a0:::benchmark_data_smithi122_119671_object91218:head# 2024-03-20T09:04:58.946 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.946 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b09cfc28:::benchmark_data_smithi122_119671_object13685:head# 2024-03-20T09:04:58.946 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.946 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b09e5990:::benchmark_data_smithi122_119671_object26333:head# 2024-03-20T09:04:58.946 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.946 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b09e91f0:::benchmark_data_smithi122_119671_object45590:head# 2024-03-20T09:04:58.946 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.946 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0a092c5:::benchmark_data_smithi122_119671_object43186:head# 2024-03-20T09:04:58.946 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.947 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0a103c2:::benchmark_data_smithi122_119671_object75321:head# 2024-03-20T09:04:58.947 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.947 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0a10e61:::benchmark_data_smithi122_119671_object97876:head# 2024-03-20T09:04:58.947 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.947 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0a149f4:::benchmark_data_smithi122_119671_object71474:head# 2024-03-20T09:04:58.947 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.947 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0a309bf:::benchmark_data_smithi122_119671_object80688:head# 2024-03-20T09:04:58.947 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.947 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0a32701:::benchmark_data_smithi122_119671_object73521:head# 2024-03-20T09:04:58.947 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.947 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0a33c03:::benchmark_data_smithi122_119671_object22311:head# 2024-03-20T09:04:58.947 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.947 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0a3eb81:::benchmark_data_smithi122_119671_object84084:head# 2024-03-20T09:04:58.948 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.948 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0a45e2e:::benchmark_data_smithi122_119671_object23112:head# 2024-03-20T09:04:58.948 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.948 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0a4c7d7:::benchmark_data_smithi122_119671_object83561:head# 2024-03-20T09:04:58.948 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.948 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0a5596c:::benchmark_data_smithi122_119671_object46042:head# 2024-03-20T09:04:58.948 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.948 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0a6037d:::benchmark_data_smithi122_119671_object11073:head# 2024-03-20T09:04:58.948 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.948 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0a60a8d:::benchmark_data_smithi122_119671_object56289:head# 2024-03-20T09:04:58.948 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.948 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0a67daa:::benchmark_data_smithi122_119671_object34371:head# 2024-03-20T09:04:58.948 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.948 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0a74e71:::benchmark_data_smithi122_119671_object96110:head# 2024-03-20T09:04:58.949 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.949 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0a881e0:::benchmark_data_smithi122_119671_object6008:head# 2024-03-20T09:04:58.949 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.949 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0a8d531:::benchmark_data_smithi122_119671_object88304:head# 2024-03-20T09:04:58.949 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.949 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0a97630:::benchmark_data_smithi122_119671_object52585:head# 2024-03-20T09:04:58.949 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.949 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0aaafef:::benchmark_data_smithi122_119671_object87310:head# 2024-03-20T09:04:58.949 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.949 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0ab8452:::benchmark_data_smithi122_119671_object73993:head# 2024-03-20T09:04:58.949 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.949 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0acd803:::benchmark_data_smithi122_119671_object73139:head# 2024-03-20T09:04:58.949 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.950 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0ad3537:::benchmark_data_smithi122_119671_object98712:head# 2024-03-20T09:04:58.950 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.950 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0ad524f:::benchmark_data_smithi122_119671_object58379:head# 2024-03-20T09:04:58.950 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.950 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0ae5622:::benchmark_data_smithi122_119671_object71829:head# 2024-03-20T09:04:58.950 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.950 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0ae5b08:::benchmark_data_smithi122_119671_object54832:head# 2024-03-20T09:04:58.950 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.950 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0aec6fa:::benchmark_data_smithi122_119671_object66541:head# 2024-03-20T09:04:58.950 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.950 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0b1e965:::benchmark_data_smithi122_119671_object83991:head# 2024-03-20T09:04:58.950 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.950 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0b27289:::benchmark_data_smithi122_119671_object85951:head# 2024-03-20T09:04:58.950 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.951 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0b27ed3:::benchmark_data_smithi122_119671_object57212:head# 2024-03-20T09:04:58.951 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.951 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0b29f3d:::benchmark_data_smithi122_119671_object73011:head# 2024-03-20T09:04:58.951 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.951 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0b36165:::benchmark_data_smithi122_119671_object83383:head# 2024-03-20T09:04:58.951 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.951 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0b38d02:::benchmark_data_smithi122_119671_object92751:head# 2024-03-20T09:04:58.951 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.951 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0b4d589:::benchmark_data_smithi122_119671_object87502:head# 2024-03-20T09:04:58.951 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.951 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0b63f50:::benchmark_data_smithi122_119671_object27297:head# 2024-03-20T09:04:58.951 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.951 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0b69cb7:::benchmark_data_smithi122_119671_object64466:head# 2024-03-20T09:04:58.952 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.952 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0b6f2f7:::benchmark_data_smithi122_119671_object34037:head# 2024-03-20T09:04:58.952 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.952 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0b73b40:::benchmark_data_smithi122_119671_object51573:head# 2024-03-20T09:04:58.952 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.952 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0b7d50d:::benchmark_data_smithi122_119671_object16518:head# 2024-03-20T09:04:58.952 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.952 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0b80f81:::benchmark_data_smithi122_119671_object71768:head# 2024-03-20T09:04:58.952 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.952 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0b83705:::benchmark_data_smithi122_119671_object53901:head# 2024-03-20T09:04:58.952 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.952 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0b8c27d:::benchmark_data_smithi122_119671_object46493:head# 2024-03-20T09:04:58.952 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.952 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0b8c5f6:::benchmark_data_smithi122_119671_object89995:head# 2024-03-20T09:04:58.953 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.953 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0bac264:::benchmark_data_smithi122_119671_object14421:head# 2024-03-20T09:04:58.953 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.953 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0bc2a2b:::benchmark_data_smithi122_119671_object55880:head# 2024-03-20T09:04:58.953 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.953 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0bc5e72:::benchmark_data_smithi122_119671_object35449:head# 2024-03-20T09:04:58.953 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.953 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0bc8bb1:::benchmark_data_smithi122_119671_object54861:head# 2024-03-20T09:04:58.953 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.953 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0bcb7cf:::benchmark_data_smithi122_119671_object10542:head# 2024-03-20T09:04:58.953 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.953 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0bd2cb2:::benchmark_data_smithi122_119671_object80663:head# 2024-03-20T09:04:58.953 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.954 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0bd651b:::benchmark_data_smithi122_119671_object27977:head# 2024-03-20T09:04:58.954 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.954 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0bd7c20:::benchmark_data_smithi122_119671_object56868:head# 2024-03-20T09:04:58.954 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.954 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0bd910a:::benchmark_data_smithi122_119671_object90735:head# 2024-03-20T09:04:58.954 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.954 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0beb7b7:::benchmark_data_smithi122_119671_object20805:head# 2024-03-20T09:04:58.954 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.954 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0bec12e:::benchmark_data_smithi122_119671_object59863:head# 2024-03-20T09:04:58.954 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.954 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0beefd3:::benchmark_data_smithi122_119671_object9054:head# 2024-03-20T09:04:58.954 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.955 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0bf072b:::benchmark_data_smithi122_119671_object62313:head# 2024-03-20T09:04:58.955 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.955 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0bf3e42:::benchmark_data_smithi122_119671_object83171:head# 2024-03-20T09:04:58.955 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.955 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0bf45e3:::benchmark_data_smithi122_119671_object97414:head# 2024-03-20T09:04:58.955 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.955 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0c040d2:::benchmark_data_smithi122_119671_object41746:head# 2024-03-20T09:04:58.955 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.955 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0c1c996:::benchmark_data_smithi122_119671_object66971:head# 2024-03-20T09:04:58.955 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.955 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0c207bb:::benchmark_data_smithi122_119671_object65552:head# 2024-03-20T09:04:58.955 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.955 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0c4b6f3:::benchmark_data_smithi122_119671_object27232:head# 2024-03-20T09:04:58.955 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.956 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0c517f4:::benchmark_data_smithi122_119671_object83910:head# 2024-03-20T09:04:58.956 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.956 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0c678ec:::benchmark_data_smithi122_119671_object29728:head# 2024-03-20T09:04:58.956 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.956 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0c73bbb:::benchmark_data_smithi122_119671_object41704:head# 2024-03-20T09:04:58.956 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.956 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0c78f36:::benchmark_data_smithi122_119671_object5920:head# 2024-03-20T09:04:58.956 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.956 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0c9b74a:::benchmark_data_smithi122_119671_object31542:head# 2024-03-20T09:04:58.956 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.956 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0ca3ca8:::benchmark_data_smithi122_119671_object27073:head# 2024-03-20T09:04:58.956 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.957 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0cb2a12:::benchmark_data_smithi122_119671_object31440:head# 2024-03-20T09:04:58.957 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.957 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0cb5147:::benchmark_data_smithi122_119671_object10325:head# 2024-03-20T09:04:58.957 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.957 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0cc4be5:::benchmark_data_smithi122_119671_object35758:head# 2024-03-20T09:04:58.957 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.957 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0cc4e50:::benchmark_data_smithi122_119671_object1895:head# 2024-03-20T09:04:58.957 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.957 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0ccc620:::benchmark_data_smithi122_119671_object16950:head# 2024-03-20T09:04:58.957 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.957 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0cd103e:::benchmark_data_smithi122_119671_object95557:head# 2024-03-20T09:04:58.957 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.957 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0ce3538:::benchmark_data_smithi122_119671_object14552:head# 2024-03-20T09:04:58.957 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.957 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0ce3755:::benchmark_data_smithi122_119671_object17936:head# 2024-03-20T09:04:58.957 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.958 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0cef7e0:::benchmark_data_smithi122_119671_object76417:head# 2024-03-20T09:04:58.958 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.958 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0cefca3:::benchmark_data_smithi122_119671_object55143:head# 2024-03-20T09:04:58.958 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.958 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0cf6f4f:::benchmark_data_smithi122_119671_object4474:head# 2024-03-20T09:04:58.958 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.958 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0cfdbfa:::benchmark_data_smithi122_119671_object98749:head# 2024-03-20T09:04:58.958 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.958 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0d10d70:::benchmark_data_smithi122_119671_object72485:head# 2024-03-20T09:04:58.958 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.958 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0d1ad2d:::benchmark_data_smithi122_119671_object55352:head# 2024-03-20T09:04:58.958 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.958 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0d1e5b0:::benchmark_data_smithi122_119671_object12355:head# 2024-03-20T09:04:58.959 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.959 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0d310c4:::benchmark_data_smithi122_119671_object7225:head# 2024-03-20T09:04:58.959 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.959 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0d3c4d1:::benchmark_data_smithi122_119671_object28577:head# 2024-03-20T09:04:58.959 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.959 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0d4670c:::benchmark_data_smithi122_119671_object33710:head# 2024-03-20T09:04:58.959 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.959 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0d48cd2:::benchmark_data_smithi122_119671_object46950:head# 2024-03-20T09:04:58.959 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.959 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0d659a8:::benchmark_data_smithi122_119671_object89269:head# 2024-03-20T09:04:58.959 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.959 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0d758fc:::benchmark_data_smithi122_119671_object72701:head# 2024-03-20T09:04:58.959 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.959 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0d879f1:::benchmark_data_smithi122_119671_object85853:head# 2024-03-20T09:04:58.959 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.960 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0d98009:::benchmark_data_smithi122_119671_object46216:head# 2024-03-20T09:04:58.960 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.961 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0da7947:::benchmark_data_smithi122_119671_object58613:head# 2024-03-20T09:04:58.961 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.962 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0daad32:::benchmark_data_smithi122_119671_object63455:head# 2024-03-20T09:04:58.962 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.962 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0dae60f:::benchmark_data_smithi122_119671_object37898:head# 2024-03-20T09:04:58.962 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.962 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0db3512:::benchmark_data_smithi122_119671_object12853:head# 2024-03-20T09:04:58.962 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.962 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0dc8398:::benchmark_data_smithi122_119671_object64119:head# 2024-03-20T09:04:58.962 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.962 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0dcce8e:::benchmark_data_smithi122_119671_object16593:head# 2024-03-20T09:04:58.962 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.962 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0de3c71:::benchmark_data_smithi122_119671_object79982:head# 2024-03-20T09:04:58.962 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.962 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0de8980:::benchmark_data_smithi122_119671_object83311:head# 2024-03-20T09:04:58.963 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.963 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0debc98:::benchmark_data_smithi122_119671_object30802:head# 2024-03-20T09:04:58.963 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.963 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0dee826:::benchmark_data_smithi122_119671_object52295:head# 2024-03-20T09:04:58.963 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.963 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0defe8f:::benchmark_data_smithi122_119671_object83152:head# 2024-03-20T09:04:58.963 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.963 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0df3053:::benchmark_data_smithi122_119671_object31948:head# 2024-03-20T09:04:58.963 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.963 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0df8cc7:::benchmark_data_smithi122_119671_object98340:head# 2024-03-20T09:04:58.963 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.963 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0dfc4db:::benchmark_data_smithi122_119671_object97251:head# 2024-03-20T09:04:58.963 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.963 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0e04298:::benchmark_data_smithi122_119671_object30902:head# 2024-03-20T09:04:58.964 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.964 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0e14699:::benchmark_data_smithi122_119671_object39195:head# 2024-03-20T09:04:58.964 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.964 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0e1b50b:::benchmark_data_smithi122_119671_object64221:head# 2024-03-20T09:04:58.964 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.964 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0e224b1:::benchmark_data_smithi122_119671_object22957:head# 2024-03-20T09:04:58.964 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.964 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0e32ce2:::benchmark_data_smithi122_119671_object22052:head# 2024-03-20T09:04:58.964 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.964 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0e5a551:::benchmark_data_smithi122_119671_object98498:head# 2024-03-20T09:04:58.964 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.964 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0e5b140:::benchmark_data_smithi122_119671_object86054:head# 2024-03-20T09:04:58.964 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.965 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0e6daf3:::benchmark_data_smithi122_119671_object20042:head# 2024-03-20T09:04:58.965 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.965 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0e6f333:::benchmark_data_smithi122_119671_object39255:head# 2024-03-20T09:04:58.965 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.965 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0e7740b:::benchmark_data_smithi122_119671_object93155:head# 2024-03-20T09:04:58.965 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.965 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0e79bf3:::benchmark_data_smithi122_119671_object97237:head# 2024-03-20T09:04:58.965 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.965 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0e7e245:::benchmark_data_smithi122_119671_object21917:head# 2024-03-20T09:04:58.965 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.965 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0e88d72:::benchmark_data_smithi122_119671_object49945:head# 2024-03-20T09:04:58.965 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.965 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0e8b039:::benchmark_data_smithi122_119671_object54366:head# 2024-03-20T09:04:58.965 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.966 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0e8db0b:::benchmark_data_smithi122_119671_object3356:head# 2024-03-20T09:04:58.966 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.966 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0ea4301:::benchmark_data_smithi122_119671_object51008:head# 2024-03-20T09:04:58.966 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.966 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0ea9a5f:::benchmark_data_smithi122_119671_object75090:head# 2024-03-20T09:04:58.966 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.966 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0eb1df1:::benchmark_data_smithi122_119671_object87207:head# 2024-03-20T09:04:58.966 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.966 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0eb2d7e:::benchmark_data_smithi122_119671_object85550:head# 2024-03-20T09:04:58.966 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.966 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0eb4cd5:::benchmark_data_smithi122_119671_object85182:head# 2024-03-20T09:04:58.966 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.966 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0eb6b77:::benchmark_data_smithi122_119671_object54957:head# 2024-03-20T09:04:58.967 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.967 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0eb85fa:::benchmark_data_smithi122_119671_object22306:head# 2024-03-20T09:04:58.967 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.967 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0ebe43c:::benchmark_data_smithi122_119671_object40521:head# 2024-03-20T09:04:58.967 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.967 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0ebea0d:::benchmark_data_smithi122_119671_object67828:head# 2024-03-20T09:04:58.967 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.967 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0ed457a:::benchmark_data_smithi122_119671_object66240:head# 2024-03-20T09:04:58.967 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.967 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0edbcb7:::benchmark_data_smithi122_119671_object1057:head# 2024-03-20T09:04:58.967 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.967 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0ef9419:::benchmark_data_smithi122_119671_object75982:head# 2024-03-20T09:04:58.967 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.967 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0f04452:::benchmark_data_smithi122_119671_object20461:head# 2024-03-20T09:04:58.967 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.968 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0f17efb:::benchmark_data_smithi122_119671_object52479:head# 2024-03-20T09:04:58.968 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.968 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0f25a15:::benchmark_data_smithi122_119671_object46520:head# 2024-03-20T09:04:58.968 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.968 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0f2bae1:::benchmark_data_smithi122_119671_object11750:head# 2024-03-20T09:04:58.968 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.968 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0f3c86b:::benchmark_data_smithi122_119671_object63456:head# 2024-03-20T09:04:58.968 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.968 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0f438f5:::benchmark_data_smithi122_119671_object57655:head# 2024-03-20T09:04:58.968 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.968 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0f4c8ee:::benchmark_data_smithi122_119671_object64098:head# 2024-03-20T09:04:58.968 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.968 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0f656c5:::benchmark_data_smithi122_119671_object78835:head# 2024-03-20T09:04:58.968 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.968 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0f6ba7c:::benchmark_data_smithi122_119671_object27631:head# 2024-03-20T09:04:58.969 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.969 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0f72b15:::benchmark_data_smithi122_119671_object40332:head# 2024-03-20T09:04:58.969 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.969 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0f7dc29:::benchmark_data_smithi122_119671_object58392:head# 2024-03-20T09:04:58.969 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.969 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0f801e4:::benchmark_data_smithi122_119671_object74442:head# 2024-03-20T09:04:58.969 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.969 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0fa04b4:::benchmark_data_smithi122_119671_object44380:head# 2024-03-20T09:04:58.969 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.969 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0fa549e:::benchmark_data_smithi122_119671_object5017:head# 2024-03-20T09:04:58.969 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.969 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0fb8a21:::benchmark_data_smithi122_119671_object78194:head# 2024-03-20T09:04:58.969 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.969 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0fbb76d:::benchmark_data_smithi122_119671_object84513:head# 2024-03-20T09:04:58.970 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.970 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0fc5108:::benchmark_data_smithi122_119671_object3968:head# 2024-03-20T09:04:58.970 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.970 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0fe6f18:::benchmark_data_smithi122_119671_object61033:head# 2024-03-20T09:04:58.970 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.970 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0fed03c:::benchmark_data_smithi122_119671_object99413:head# 2024-03-20T09:04:58.970 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.970 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0fed1f0:::benchmark_data_smithi122_119671_object18752:head# 2024-03-20T09:04:58.970 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.970 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b0ff5c67:::benchmark_data_smithi122_119671_object59220:head# 2024-03-20T09:04:58.970 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.970 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1006b7e:::benchmark_data_smithi122_119671_object15358:head# 2024-03-20T09:04:58.970 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.971 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b10164a7:::benchmark_data_smithi122_119671_object20654:head# 2024-03-20T09:04:58.971 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.971 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1019a38:::benchmark_data_smithi122_119671_object26078:head# 2024-03-20T09:04:58.971 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.971 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1031e17:::benchmark_data_smithi122_119671_object74674:head# 2024-03-20T09:04:58.971 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.971 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1039c0e:::benchmark_data_smithi122_119671_object36872:head# 2024-03-20T09:04:58.971 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.971 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b10486ae:::benchmark_data_smithi122_119671_object47803:head# 2024-03-20T09:04:58.971 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.971 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1054417:::benchmark_data_smithi122_119671_object54064:head# 2024-03-20T09:04:58.971 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.971 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b107b2bb:::benchmark_data_smithi122_119671_object9518:head# 2024-03-20T09:04:58.971 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.972 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b108c726:::benchmark_data_smithi122_119671_object82251:head# 2024-03-20T09:04:58.972 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.972 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b10aaae9:::benchmark_data_smithi122_119671_object4649:head# 2024-03-20T09:04:58.972 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.972 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b10b3dd3:::benchmark_data_smithi122_119671_object76363:head# 2024-03-20T09:04:58.972 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.972 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b10b6d31:::benchmark_data_smithi122_119671_object21104:head# 2024-03-20T09:04:58.972 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.972 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b10c700d:::benchmark_data_smithi122_119671_object1985:head# 2024-03-20T09:04:58.972 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.972 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b10d924b:::benchmark_data_smithi122_119671_object22362:head# 2024-03-20T09:04:58.972 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.972 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b10de806:::benchmark_data_smithi122_119671_object77173:head# 2024-03-20T09:04:58.972 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.972 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b10e5b96:::benchmark_data_smithi122_119671_object4829:head# 2024-03-20T09:04:58.973 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.973 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b10f021c:::benchmark_data_smithi122_119671_object27719:head# 2024-03-20T09:04:58.973 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.973 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b10fcbe3:::benchmark_data_smithi122_119671_object39273:head# 2024-03-20T09:04:58.973 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.973 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b10fd1a8:::benchmark_data_smithi122_119671_object8384:head# 2024-03-20T09:04:58.973 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.973 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1113bf4:::benchmark_data_smithi122_119671_object22446:head# 2024-03-20T09:04:58.973 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.973 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1129c52:::benchmark_data_smithi122_119671_object91664:head# 2024-03-20T09:04:58.973 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.973 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b112f41d:::benchmark_data_smithi122_119671_object97739:head# 2024-03-20T09:04:58.973 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.973 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b11358a6:::benchmark_data_smithi122_119671_object81154:head# 2024-03-20T09:04:58.974 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.974 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b11428e0:::benchmark_data_smithi122_119671_object81199:head# 2024-03-20T09:04:58.974 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.974 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1155de4:::benchmark_data_smithi122_119671_object91814:head# 2024-03-20T09:04:58.974 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.974 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1159e09:::benchmark_data_smithi122_119671_object79643:head# 2024-03-20T09:04:58.974 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.974 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b116cb88:::benchmark_data_smithi122_119671_object87817:head# 2024-03-20T09:04:58.974 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.974 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b117a0ef:::benchmark_data_smithi122_119671_object10540:head# 2024-03-20T09:04:58.974 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.974 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b119521e:::benchmark_data_smithi122_119671_object15992:head# 2024-03-20T09:04:58.974 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.974 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b11964bb:::benchmark_data_smithi122_119671_object8296:head# 2024-03-20T09:04:58.975 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.975 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b119a299:::benchmark_data_smithi122_119671_object50200:head# 2024-03-20T09:04:58.975 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.975 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b11a09f3:::benchmark_data_smithi122_119671_object43188:head# 2024-03-20T09:04:58.975 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.975 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b11a3545:::benchmark_data_smithi122_119671_object99768:head# 2024-03-20T09:04:58.975 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.975 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b11a4e63:::benchmark_data_smithi122_119671_object3454:head# 2024-03-20T09:04:58.975 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.975 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b11ab501:::benchmark_data_smithi122_119671_object44226:head# 2024-03-20T09:04:58.975 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.975 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b11adacb:::benchmark_data_smithi122_119671_object29775:head# 2024-03-20T09:04:58.975 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.975 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b11b8093:::benchmark_data_smithi122_119671_object63182:head# 2024-03-20T09:04:58.975 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.976 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b11c0c57:::benchmark_data_smithi122_119671_object8818:head# 2024-03-20T09:04:58.976 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.976 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b11cdc1f:::benchmark_data_smithi122_119671_object29214:head# 2024-03-20T09:04:58.976 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.976 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b11d0baf:::benchmark_data_smithi122_119671_object3269:head# 2024-03-20T09:04:58.976 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.976 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b11dbf0c:::benchmark_data_smithi122_119671_object5103:head# 2024-03-20T09:04:58.976 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.976 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b11dd973:::benchmark_data_smithi122_119671_object82917:head# 2024-03-20T09:04:58.976 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.976 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b11eb72d:::benchmark_data_smithi122_119671_object42308:head# 2024-03-20T09:04:58.976 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.976 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b11ed802:::benchmark_data_smithi122_119671_object36549:head# 2024-03-20T09:04:58.976 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.977 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b120391b:::benchmark_data_smithi122_119671_object17689:head# 2024-03-20T09:04:58.977 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.977 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b120578c:::benchmark_data_smithi122_119671_object83357:head# 2024-03-20T09:04:58.977 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.977 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b120ba69:::benchmark_data_smithi122_119671_object51813:head# 2024-03-20T09:04:58.977 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.977 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b120d9ea:::benchmark_data_smithi122_119671_object84659:head# 2024-03-20T09:04:58.977 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.977 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b12304d1:::benchmark_data_smithi122_119671_object47639:head# 2024-03-20T09:04:58.977 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.977 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b12316c5:::benchmark_data_smithi122_119671_object99116:head# 2024-03-20T09:04:58.977 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.977 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1241a5c:::benchmark_data_smithi122_119671_object29199:head# 2024-03-20T09:04:58.977 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.978 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b124d7c7:::benchmark_data_smithi122_119671_object40694:head# 2024-03-20T09:04:58.978 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.978 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b124f657:::benchmark_data_smithi122_119671_object56442:head# 2024-03-20T09:04:58.978 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.978 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b12752f4:::benchmark_data_smithi122_119671_object47344:head# 2024-03-20T09:04:58.978 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.978 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1278594:::benchmark_data_smithi122_119671_object17569:head# 2024-03-20T09:04:58.978 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.978 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b12839fc:::benchmark_data_smithi122_119671_object18084:head# 2024-03-20T09:04:58.978 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.978 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1296782:::benchmark_data_smithi122_119671_object5714:head# 2024-03-20T09:04:58.978 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.978 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b12aafbe:::benchmark_data_smithi122_119671_object21998:head# 2024-03-20T09:04:58.979 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.979 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b12c679f:::benchmark_data_smithi122_119671_object97548:head# 2024-03-20T09:04:58.979 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.979 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b12cd820:::benchmark_data_smithi122_119671_object68447:head# 2024-03-20T09:04:58.979 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.979 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b12dc91f:::benchmark_data_smithi122_119671_object90129:head# 2024-03-20T09:04:58.979 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.979 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b12e61d2:::benchmark_data_smithi122_119671_object34726:head# 2024-03-20T09:04:58.979 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.979 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b12f6026:::benchmark_data_smithi122_119671_object34187:head# 2024-03-20T09:04:58.979 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.979 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b12f6ca7:::benchmark_data_smithi122_119671_object21522:head# 2024-03-20T09:04:58.979 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.979 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b12f78d7:::benchmark_data_smithi122_119671_object89847:head# 2024-03-20T09:04:58.980 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.980 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b12f8e76:::benchmark_data_smithi122_119671_object14007:head# 2024-03-20T09:04:58.980 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.980 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b12ff9a0:::benchmark_data_smithi122_119671_object60390:head# 2024-03-20T09:04:58.980 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.980 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b130147f:::benchmark_data_smithi122_119671_object22146:head# 2024-03-20T09:04:58.980 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.980 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1307515:::benchmark_data_smithi122_119671_object76172:head# 2024-03-20T09:04:58.980 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.980 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b130912d:::benchmark_data_smithi122_119671_object97853:head# 2024-03-20T09:04:58.980 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.980 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b130e5e2:::benchmark_data_smithi122_119671_object63395:head# 2024-03-20T09:04:58.980 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.980 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1310f92:::benchmark_data_smithi122_119671_object11997:head# 2024-03-20T09:04:58.980 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.981 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b132e36f:::benchmark_data_smithi122_119671_object89892:head# 2024-03-20T09:04:58.981 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.981 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b133ae86:::benchmark_data_smithi122_119671_object28764:head# 2024-03-20T09:04:58.981 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.981 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b133f5e1:::benchmark_data_smithi122_119671_object92200:head# 2024-03-20T09:04:58.981 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.981 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1346a78:::benchmark_data_smithi122_119671_object74779:head# 2024-03-20T09:04:58.981 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.981 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b134b090:::benchmark_data_smithi122_119671_object53375:head# 2024-03-20T09:04:58.981 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.981 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b137f03d:::benchmark_data_smithi122_119671_object99525:head# 2024-03-20T09:04:58.981 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.981 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1382c22:::benchmark_data_smithi122_119671_object7973:head# 2024-03-20T09:04:58.981 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.982 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1386124:::benchmark_data_smithi122_119671_object78347:head# 2024-03-20T09:04:58.982 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.982 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1399f42:::benchmark_data_smithi122_119671_object29480:head# 2024-03-20T09:04:58.982 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.982 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b13bf863:::benchmark_data_smithi122_119671_object58940:head# 2024-03-20T09:04:58.982 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.982 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b13dd1ba:::benchmark_data_smithi122_119671_object30342:head# 2024-03-20T09:04:58.982 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.982 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b13e37ed:::benchmark_data_smithi122_119671_object63606:head# 2024-03-20T09:04:58.982 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.982 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b13edcb6:::benchmark_data_smithi122_119671_object75509:head# 2024-03-20T09:04:58.982 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.982 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1412885:::benchmark_data_smithi122_119671_object48325:head# 2024-03-20T09:04:58.982 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.983 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b14204d9:::benchmark_data_smithi122_119671_object53207:head# 2024-03-20T09:04:58.983 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.983 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b14247d0:::benchmark_data_smithi122_119671_object2365:head# 2024-03-20T09:04:58.983 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.983 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1426146:::benchmark_data_smithi122_119671_object99237:head# 2024-03-20T09:04:58.983 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.983 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1447b22:::benchmark_data_smithi122_119671_object62547:head# 2024-03-20T09:04:58.983 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.983 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1448d65:::benchmark_data_smithi122_119671_object67039:head# 2024-03-20T09:04:58.983 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.983 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b146735a:::benchmark_data_smithi122_119671_object9203:head# 2024-03-20T09:04:58.983 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.983 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b146a779:::benchmark_data_smithi122_119671_object20344:head# 2024-03-20T09:04:58.983 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.984 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1470836:::benchmark_data_smithi122_119671_object92288:head# 2024-03-20T09:04:58.984 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.984 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1479190:::benchmark_data_smithi122_119671_object96215:head# 2024-03-20T09:04:58.984 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.984 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b147acf0:::benchmark_data_smithi122_119671_object43711:head# 2024-03-20T09:04:58.984 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.984 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b147e316:::benchmark_data_smithi122_119671_object56196:head# 2024-03-20T09:04:58.984 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.984 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1486951:::benchmark_data_smithi122_119671_object9118:head# 2024-03-20T09:04:58.984 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.984 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b14a2e16:::benchmark_data_smithi122_119671_object73834:head# 2024-03-20T09:04:58.984 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.984 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b14a343f:::benchmark_data_smithi122_119671_object90099:head# 2024-03-20T09:04:58.984 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.985 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b14b7e77:::benchmark_data_smithi122_119671_object36591:head# 2024-03-20T09:04:58.985 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.985 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b14ba3d6:::benchmark_data_smithi122_119671_object56104:head# 2024-03-20T09:04:58.985 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:58.985 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b14bd22e:::benchmark_data_smithi122_119671_object66793:head# 2024-03-20T09:04:58.985 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.000 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:04:59.043 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:59.065 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b14ca1b5:::benchmark_data_smithi122_119671_object9583:head# 2024-03-20T09:04:59.065 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.065 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b14cb3e5:::benchmark_data_smithi122_119671_object68675:head# 2024-03-20T09:04:59.065 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.065 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b14e585a:::benchmark_data_smithi122_119671_object34588:head# 2024-03-20T09:04:59.065 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.065 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b14eab8c:::benchmark_data_smithi122_119671_object91083:head# 2024-03-20T09:04:59.065 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.065 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b14ef4c4:::benchmark_data_smithi122_119671_object25000:head# 2024-03-20T09:04:59.065 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.065 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b14f1f32:::benchmark_data_smithi122_119671_object8354:head# 2024-03-20T09:04:59.065 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.066 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b14f3828:::benchmark_data_smithi122_119671_object95623:head# 2024-03-20T09:04:59.066 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.066 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b150c503:::benchmark_data_smithi122_119671_object97678:head# 2024-03-20T09:04:59.066 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.066 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b150f8b6:::benchmark_data_smithi122_119671_object43193:head# 2024-03-20T09:04:59.066 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.066 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1513461:::benchmark_data_smithi122_119671_object64981:head# 2024-03-20T09:04:59.066 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.066 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1515c96:::benchmark_data_smithi122_119671_object32062:head# 2024-03-20T09:04:59.066 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.066 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b15168e6:::benchmark_data_smithi122_119671_object90279:head# 2024-03-20T09:04:59.066 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.067 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1519266:::benchmark_data_smithi122_119671_object41688:head# 2024-03-20T09:04:59.067 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.067 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b151a863:::benchmark_data_smithi122_119671_object65370:head# 2024-03-20T09:04:59.067 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.067 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b15250b4:::benchmark_data_smithi122_119671_object7806:head# 2024-03-20T09:04:59.067 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.067 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b15319c6:::benchmark_data_smithi122_119671_object60314:head# 2024-03-20T09:04:59.067 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.067 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1536592:::benchmark_data_smithi122_119671_object72223:head# 2024-03-20T09:04:59.067 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.067 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b154c552:::benchmark_data_smithi122_119671_object85558:head# 2024-03-20T09:04:59.067 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.067 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b155ea13:::benchmark_data_smithi122_119671_object16083:head# 2024-03-20T09:04:59.067 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.068 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1561b0f:::benchmark_data_smithi122_119671_object67401:head# 2024-03-20T09:04:59.068 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.068 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b157658f:::benchmark_data_smithi122_119671_object44102:head# 2024-03-20T09:04:59.068 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.068 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1580712:::benchmark_data_smithi122_119671_object30871:head# 2024-03-20T09:04:59.068 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.068 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1580f02:::benchmark_data_smithi122_119671_object63064:head# 2024-03-20T09:04:59.068 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.068 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b158136c:::benchmark_data_smithi122_119671_object87359:head# 2024-03-20T09:04:59.068 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.068 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b159a887:::benchmark_data_smithi122_119671_object45172:head# 2024-03-20T09:04:59.068 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.068 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b159ec81:::benchmark_data_smithi122_119671_object54425:head# 2024-03-20T09:04:59.069 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.069 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b15a8f8d:::benchmark_data_smithi122_119671_object21731:head# 2024-03-20T09:04:59.069 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.069 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b15b17b0:::benchmark_data_smithi122_119671_object66780:head# 2024-03-20T09:04:59.069 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.069 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b15b9c8e:::benchmark_data_smithi122_119671_object20464:head# 2024-03-20T09:04:59.069 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.069 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b15be42a:::benchmark_data_smithi122_119671_object58230:head# 2024-03-20T09:04:59.069 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.069 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b15c6408:::benchmark_data_smithi122_119671_object58690:head# 2024-03-20T09:04:59.069 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.069 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b15d641a:::benchmark_data_smithi122_119671_object46587:head# 2024-03-20T09:04:59.069 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.070 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b15e2e5c:::benchmark_data_smithi122_119671_object40586:head# 2024-03-20T09:04:59.070 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.070 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b15ebd78:::benchmark_data_smithi122_119671_object2749:head# 2024-03-20T09:04:59.070 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.070 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b15fefc6:::benchmark_data_smithi122_119671_object31417:head# 2024-03-20T09:04:59.070 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.070 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1608934:::benchmark_data_smithi122_119671_object98046:head# 2024-03-20T09:04:59.070 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.070 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b160cb14:::benchmark_data_smithi122_119671_object42965:head# 2024-03-20T09:04:59.070 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.070 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1613a07:::benchmark_data_smithi122_119671_object3118:head# 2024-03-20T09:04:59.070 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.070 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b16159a0:::benchmark_data_smithi122_119671_object19652:head# 2024-03-20T09:04:59.071 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.071 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1621597:::benchmark_data_smithi122_119671_object78702:head# 2024-03-20T09:04:59.071 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.071 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1622276:::benchmark_data_smithi122_119671_object98385:head# 2024-03-20T09:04:59.071 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.071 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b163cd4e:::benchmark_data_smithi122_119671_object31843:head# 2024-03-20T09:04:59.071 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.071 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b16495f2:::benchmark_data_smithi122_119671_object17195:head# 2024-03-20T09:04:59.071 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.071 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1649915:::benchmark_data_smithi122_119671_object17150:head# 2024-03-20T09:04:59.071 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.071 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b164a34e:::benchmark_data_smithi122_119671_object76028:head# 2024-03-20T09:04:59.071 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.072 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b164d87e:::benchmark_data_smithi122_119671_object1955:head# 2024-03-20T09:04:59.072 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.072 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b165154b:::benchmark_data_smithi122_119671_object60968:head# 2024-03-20T09:04:59.072 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.072 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b165684b:::benchmark_data_smithi122_119671_object95328:head# 2024-03-20T09:04:59.072 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.072 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1659e43:::benchmark_data_smithi122_119671_object46961:head# 2024-03-20T09:04:59.072 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.072 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b16607f5:::benchmark_data_smithi122_119671_object25424:head# 2024-03-20T09:04:59.072 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.072 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b16607fd:::benchmark_data_smithi122_119671_object67181:head# 2024-03-20T09:04:59.072 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.072 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1694688:::benchmark_data_smithi122_119671_object84594:head# 2024-03-20T09:04:59.072 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.073 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b169671b:::benchmark_data_smithi122_119671_object25647:head# 2024-03-20T09:04:59.073 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.073 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b16b1eaa:::benchmark_data_smithi122_119671_object32341:head# 2024-03-20T09:04:59.073 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.073 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b16becaf:::benchmark_data_smithi122_119671_object55687:head# 2024-03-20T09:04:59.073 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.073 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b16d2c7d:::benchmark_data_smithi122_119671_object47508:head# 2024-03-20T09:04:59.073 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.073 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b16e0b71:::benchmark_data_smithi122_119671_object77507:head# 2024-03-20T09:04:59.073 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.073 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b16e2772:::benchmark_data_smithi122_119671_object64562:head# 2024-03-20T09:04:59.073 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.073 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b16e6745:::benchmark_data_smithi122_119671_object95605:head# 2024-03-20T09:04:59.074 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.074 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b16f70d2:::benchmark_data_smithi122_119671_object87321:head# 2024-03-20T09:04:59.074 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.074 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b16fab07:::benchmark_data_smithi122_119671_object30031:head# 2024-03-20T09:04:59.074 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.074 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b16ff496:::benchmark_data_smithi122_119671_object86482:head# 2024-03-20T09:04:59.074 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.074 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b171ac5d:::benchmark_data_smithi122_119671_object65442:head# 2024-03-20T09:04:59.074 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.074 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1723f64:::benchmark_data_smithi122_119671_object27434:head# 2024-03-20T09:04:59.074 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.074 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1748b2e:::benchmark_data_smithi122_119671_object4925:head# 2024-03-20T09:04:59.074 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.074 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1751e29:::benchmark_data_smithi122_119671_object62859:head# 2024-03-20T09:04:59.075 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.075 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1753575:::benchmark_data_smithi122_119671_object30472:head# 2024-03-20T09:04:59.075 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.075 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1756077:::benchmark_data_smithi122_119671_object7621:head# 2024-03-20T09:04:59.075 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.075 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b175673e:::benchmark_data_smithi122_119671_object65461:head# 2024-03-20T09:04:59.075 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.075 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1759f64:::benchmark_data_smithi122_119671_object62802:head# 2024-03-20T09:04:59.075 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.075 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b175b521:::benchmark_data_smithi122_119671_object89020:head# 2024-03-20T09:04:59.075 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.075 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b175c240:::benchmark_data_smithi122_119671_object15139:head# 2024-03-20T09:04:59.075 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.075 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b176fdee:::benchmark_data_smithi122_119671_object77158:head# 2024-03-20T09:04:59.075 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.076 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1789fe9:::benchmark_data_smithi122_119671_object51510:head# 2024-03-20T09:04:59.076 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.076 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b17b61a1:::benchmark_data_smithi122_119671_object17921:head# 2024-03-20T09:04:59.076 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.076 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b17bcd9a:::benchmark_data_smithi122_119671_object33573:head# 2024-03-20T09:04:59.076 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.076 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b17c0991:::benchmark_data_smithi122_119671_object85079:head# 2024-03-20T09:04:59.076 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.076 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b17c567a:::benchmark_data_smithi122_119671_object33370:head# 2024-03-20T09:04:59.076 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.076 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b17cedb0:::benchmark_data_smithi122_119671_object91075:head# 2024-03-20T09:04:59.076 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.076 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b17d6c82:::benchmark_data_smithi122_119671_object38866:head# 2024-03-20T09:04:59.076 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.077 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b17e7ce5:::benchmark_data_smithi122_119671_object83761:head# 2024-03-20T09:04:59.077 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.077 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b17e81bf:::benchmark_data_smithi122_119671_object90235:head# 2024-03-20T09:04:59.077 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.077 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b17ee6c4:::benchmark_data_smithi122_119671_object95346:head# 2024-03-20T09:04:59.077 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.077 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b180c38a:::benchmark_data_smithi122_119671_object31054:head# 2024-03-20T09:04:59.077 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.077 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1810533:::benchmark_data_smithi122_119671_object61511:head# 2024-03-20T09:04:59.077 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.077 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b18276a0:::benchmark_data_smithi122_119671_object59136:head# 2024-03-20T09:04:59.077 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.077 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b182c2d8:::benchmark_data_smithi122_119671_object7561:head# 2024-03-20T09:04:59.077 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.077 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b18346a4:::benchmark_data_smithi122_119671_object24836:head# 2024-03-20T09:04:59.078 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.078 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b183ef05:::benchmark_data_smithi122_119671_object45482:head# 2024-03-20T09:04:59.078 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.078 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1849232:::benchmark_data_smithi122_119671_object1144:head# 2024-03-20T09:04:59.078 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.078 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b185c8b8:::benchmark_data_smithi122_119671_object77834:head# 2024-03-20T09:04:59.078 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.078 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b186875d:::benchmark_data_smithi122_119671_object49704:head# 2024-03-20T09:04:59.078 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.078 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b186a8c6:::benchmark_data_smithi122_119671_object85994:head# 2024-03-20T09:04:59.078 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.078 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b18838b1:::benchmark_data_smithi122_119671_object53111:head# 2024-03-20T09:04:59.078 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.078 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b189765d:::benchmark_data_smithi122_119671_object6741:head# 2024-03-20T09:04:59.079 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.079 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1898a88:::benchmark_data_smithi122_119671_object86453:head# 2024-03-20T09:04:59.079 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.079 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b189d92e:::benchmark_data_smithi122_119671_object92426:head# 2024-03-20T09:04:59.079 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.079 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b189f68e:::benchmark_data_smithi122_119671_object24872:head# 2024-03-20T09:04:59.079 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.079 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b18a6b44:::benchmark_data_smithi122_119671_object95719:head# 2024-03-20T09:04:59.079 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.079 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b18a9d8a:::benchmark_data_smithi122_119671_object84529:head# 2024-03-20T09:04:59.079 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.079 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b18b368c:::benchmark_data_smithi122_119671_object28425:head# 2024-03-20T09:04:59.079 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.079 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b18bf211:::benchmark_data_smithi122_119671_object5612:head# 2024-03-20T09:04:59.080 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.080 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b18c2547:::benchmark_data_smithi122_119671_object16263:head# 2024-03-20T09:04:59.080 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.080 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b18c2fbb:::benchmark_data_smithi122_119671_object61053:head# 2024-03-20T09:04:59.080 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.080 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b18e5684:::benchmark_data_smithi122_119671_object76604:head# 2024-03-20T09:04:59.080 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.080 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b18fd62c:::benchmark_data_smithi122_119671_object78245:head# 2024-03-20T09:04:59.080 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.080 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b18fdf40:::benchmark_data_smithi122_119671_object75850:head# 2024-03-20T09:04:59.080 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.080 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1913ba8:::benchmark_data_smithi122_119671_object51012:head# 2024-03-20T09:04:59.080 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.080 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b19162fd:::benchmark_data_smithi122_119671_object53033:head# 2024-03-20T09:04:59.081 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.081 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b192830d:::benchmark_data_smithi122_119671_object22445:head# 2024-03-20T09:04:59.081 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.081 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b19320a2:::benchmark_data_smithi122_119671_object62775:head# 2024-03-20T09:04:59.081 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.081 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b193a8de:::benchmark_data_smithi122_119671_object73526:head# 2024-03-20T09:04:59.081 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.081 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b193ea4a:::benchmark_data_smithi122_119671_object89865:head# 2024-03-20T09:04:59.081 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.081 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b194bad5:::benchmark_data_smithi122_119671_object67515:head# 2024-03-20T09:04:59.081 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.081 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b196180a:::benchmark_data_smithi122_119671_object42551:head# 2024-03-20T09:04:59.081 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.081 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1967e13:::benchmark_data_smithi122_119671_object34527:head# 2024-03-20T09:04:59.081 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.082 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1968497:::benchmark_data_smithi122_119671_object95902:head# 2024-03-20T09:04:59.082 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.082 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b196b3cf:::benchmark_data_smithi122_119671_object84501:head# 2024-03-20T09:04:59.082 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.082 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b196deba:::benchmark_data_smithi122_119671_object47336:head# 2024-03-20T09:04:59.082 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.082 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b197b042:::benchmark_data_smithi122_119671_object82707:head# 2024-03-20T09:04:59.082 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.082 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b197caff:::benchmark_data_smithi122_119671_object4571:head# 2024-03-20T09:04:59.082 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.082 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b19819fc:::benchmark_data_smithi122_119671_object44904:head# 2024-03-20T09:04:59.082 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.082 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b198c036:::benchmark_data_smithi122_119671_object4376:head# 2024-03-20T09:04:59.082 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.083 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b198e623:::benchmark_data_smithi122_119671_object35712:head# 2024-03-20T09:04:59.083 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.083 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b198fbeb:::benchmark_data_smithi122_119671_object18903:head# 2024-03-20T09:04:59.083 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.083 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b199430a:::benchmark_data_smithi122_119671_object24684:head# 2024-03-20T09:04:59.083 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.083 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1997288:::benchmark_data_smithi122_119671_object15458:head# 2024-03-20T09:04:59.083 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.083 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b19a8e86:::benchmark_data_smithi122_119671_object91909:head# 2024-03-20T09:04:59.083 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.083 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b19ad2d7:::benchmark_data_smithi122_119671_object98638:head# 2024-03-20T09:04:59.083 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.083 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b19b1f6c:::benchmark_data_smithi122_119671_object42828:head# 2024-03-20T09:04:59.083 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.084 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b19c8091:::benchmark_data_smithi122_119671_object73468:head# 2024-03-20T09:04:59.084 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.084 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b19cd33c:::benchmark_data_smithi122_119671_object8996:head# 2024-03-20T09:04:59.084 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.084 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b19cd572:::benchmark_data_smithi122_119671_object41181:head# 2024-03-20T09:04:59.084 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.084 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b19de0bf:::benchmark_data_smithi122_119671_object30188:head# 2024-03-20T09:04:59.084 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.084 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b19fce86:::benchmark_data_smithi122_119671_object74671:head# 2024-03-20T09:04:59.084 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.084 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1a00a97:::benchmark_data_smithi122_119671_object92345:head# 2024-03-20T09:04:59.084 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.084 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1a08c55:::benchmark_data_smithi122_119671_object42332:head# 2024-03-20T09:04:59.084 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.085 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1a14127:::benchmark_data_smithi122_119671_object37286:head# 2024-03-20T09:04:59.085 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.085 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1a1d9e4:::benchmark_data_smithi122_119671_object51360:head# 2024-03-20T09:04:59.085 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.085 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1a219a2:::benchmark_data_smithi122_119671_object93597:head# 2024-03-20T09:04:59.085 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.085 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1a27d98:::benchmark_data_smithi122_119671_object94193:head# 2024-03-20T09:04:59.085 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.085 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1a46eb4:::benchmark_data_smithi122_119671_object14634:head# 2024-03-20T09:04:59.085 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.085 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1a4be3b:::benchmark_data_smithi122_119671_object73459:head# 2024-03-20T09:04:59.085 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.085 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1a4c77d:::benchmark_data_smithi122_119671_object48844:head# 2024-03-20T09:04:59.085 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.086 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1a4ec29:::benchmark_data_smithi122_119671_object78330:head# 2024-03-20T09:04:59.086 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.086 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1a5b395:::benchmark_data_smithi122_119671_object81012:head# 2024-03-20T09:04:59.086 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.086 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1a6d7ba:::benchmark_data_smithi122_119671_object36560:head# 2024-03-20T09:04:59.086 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.086 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1a7ae69:::benchmark_data_smithi122_119671_object77369:head# 2024-03-20T09:04:59.086 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.086 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1a7fd0d:::benchmark_data_smithi122_119671_object17623:head# 2024-03-20T09:04:59.086 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.086 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1a82001:::benchmark_data_smithi122_119671_object74833:head# 2024-03-20T09:04:59.086 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.086 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1a99919:::benchmark_data_smithi122_119671_object21798:head# 2024-03-20T09:04:59.086 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.087 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1aac9cc:::benchmark_data_smithi122_119671_object61041:head# 2024-03-20T09:04:59.087 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.087 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1abc240:::benchmark_data_smithi122_119671_object74391:head# 2024-03-20T09:04:59.087 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.087 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1ac88f4:::benchmark_data_smithi122_119671_object99602:head# 2024-03-20T09:04:59.087 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.087 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1acc9b9:::benchmark_data_smithi122_119671_object53364:head# 2024-03-20T09:04:59.087 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.087 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1accf29:::benchmark_data_smithi122_119671_object65023:head# 2024-03-20T09:04:59.087 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.087 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1ad4c8e:::benchmark_data_smithi122_119671_object99420:head# 2024-03-20T09:04:59.087 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.087 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1ad7ca4:::benchmark_data_smithi122_119671_object21282:head# 2024-03-20T09:04:59.087 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.087 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1ad8ded:::benchmark_data_smithi122_119671_object70883:head# 2024-03-20T09:04:59.088 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.088 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1ae2f99:::benchmark_data_smithi122_119671_object37013:head# 2024-03-20T09:04:59.088 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.088 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1af0cd7:::benchmark_data_smithi122_119671_object30461:head# 2024-03-20T09:04:59.088 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.088 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1af12b0:::benchmark_data_smithi122_119671_object29690:head# 2024-03-20T09:04:59.088 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.088 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1af2a30:::benchmark_data_smithi122_119671_object45146:head# 2024-03-20T09:04:59.088 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.088 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1b023e9:::benchmark_data_smithi122_119671_object62793:head# 2024-03-20T09:04:59.088 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.088 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1b0b9f7:::benchmark_data_smithi122_119671_object56003:head# 2024-03-20T09:04:59.088 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.088 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1b1014c:::benchmark_data_smithi122_119671_object21929:head# 2024-03-20T09:04:59.089 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.089 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1b223f2:::benchmark_data_smithi122_119671_object47106:head# 2024-03-20T09:04:59.089 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.089 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1b29c82:::benchmark_data_smithi122_119671_object73942:head# 2024-03-20T09:04:59.089 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.089 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1b2bf73:::benchmark_data_smithi122_119671_object90248:head# 2024-03-20T09:04:59.089 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.089 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1b3b5d0:::benchmark_data_smithi122_119671_object13370:head# 2024-03-20T09:04:59.089 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.089 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1b3c80c:::benchmark_data_smithi122_119671_object44616:head# 2024-03-20T09:04:59.089 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.089 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1b41376:::benchmark_data_smithi122_119671_object58236:head# 2024-03-20T09:04:59.089 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.089 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1b43578:::benchmark_data_smithi122_119671_object16108:head# 2024-03-20T09:04:59.089 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.090 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1b507a2:::benchmark_data_smithi122_119671_object73480:head# 2024-03-20T09:04:59.090 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.090 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1b5a684:::benchmark_data_smithi122_119671_object30210:head# 2024-03-20T09:04:59.090 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.090 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1b5abda:::benchmark_data_smithi122_119671_object62390:head# 2024-03-20T09:04:59.090 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.090 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1b61265:::benchmark_data_smithi122_119671_object70257:head# 2024-03-20T09:04:59.090 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.090 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1b6840f:::benchmark_data_smithi122_119671_object39089:head# 2024-03-20T09:04:59.090 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.090 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1b6a032:::benchmark_data_smithi122_119671_object9954:head# 2024-03-20T09:04:59.090 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.090 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1b6b058:::benchmark_data_smithi122_119671_object39268:head# 2024-03-20T09:04:59.090 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.091 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1b6cde3:::benchmark_data_smithi122_119671_object48439:head# 2024-03-20T09:04:59.091 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.091 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1bbe552:::benchmark_data_smithi122_119671_object98610:head# 2024-03-20T09:04:59.091 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.091 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1bdca38:::benchmark_data_smithi122_119671_object33556:head# 2024-03-20T09:04:59.091 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.091 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1bdf85e:::benchmark_data_smithi122_119671_object6659:head# 2024-03-20T09:04:59.091 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.091 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1be28ad:::benchmark_data_smithi122_119671_object36018:head# 2024-03-20T09:04:59.091 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.091 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1beaf27:::benchmark_data_smithi122_119671_object10485:head# 2024-03-20T09:04:59.091 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.091 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1bece77:::benchmark_data_smithi122_119671_object76384:head# 2024-03-20T09:04:59.091 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.091 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1bf45cd:::benchmark_data_smithi122_119671_object86156:head# 2024-03-20T09:04:59.092 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.092 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1c038fc:::benchmark_data_smithi122_119671_object82959:head# 2024-03-20T09:04:59.092 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.092 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1c05554:::benchmark_data_smithi122_119671_object93471:head# 2024-03-20T09:04:59.092 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.092 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1c129ee:::benchmark_data_smithi122_119671_object75857:head# 2024-03-20T09:04:59.092 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.092 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1c13d56:::benchmark_data_smithi122_119671_object83523:head# 2024-03-20T09:04:59.092 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.092 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1c2e6ba:::benchmark_data_smithi122_119671_object3161:head# 2024-03-20T09:04:59.092 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.092 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1c3ba7a:::benchmark_data_smithi122_119671_object22475:head# 2024-03-20T09:04:59.092 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.092 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1c43195:::benchmark_data_smithi122_119671_object60786:head# 2024-03-20T09:04:59.093 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.093 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1c4d3bd:::benchmark_data_smithi122_119671_object47554:head# 2024-03-20T09:04:59.093 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.093 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1c5f2c9:::benchmark_data_smithi122_119671_object67581:head# 2024-03-20T09:04:59.093 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.093 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1c6d9c8:::benchmark_data_smithi122_119671_object68283:head# 2024-03-20T09:04:59.093 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.093 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1c7bc00:::benchmark_data_smithi122_119671_object23975:head# 2024-03-20T09:04:59.093 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.093 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1c7c3a4:::benchmark_data_smithi122_119671_object56836:head# 2024-03-20T09:04:59.093 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.093 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1c7e700:::benchmark_data_smithi122_119671_object98240:head# 2024-03-20T09:04:59.093 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.093 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1c90eaf:::benchmark_data_smithi122_119671_object91076:head# 2024-03-20T09:04:59.094 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.094 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1c951be:::benchmark_data_smithi122_119671_object55380:head# 2024-03-20T09:04:59.094 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.094 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1c95b42:::benchmark_data_smithi122_119671_object73694:head# 2024-03-20T09:04:59.094 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.094 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1c9e653:::benchmark_data_smithi122_119671_object78065:head# 2024-03-20T09:04:59.094 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.094 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1ca0cff:::benchmark_data_smithi122_119671_object11488:head# 2024-03-20T09:04:59.094 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.094 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1ca13e0:::benchmark_data_smithi122_119671_object68113:head# 2024-03-20T09:04:59.094 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.094 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1ca3dcd:::benchmark_data_smithi122_119671_object48043:head# 2024-03-20T09:04:59.094 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.094 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1ca664a:::benchmark_data_smithi122_119671_object6190:head# 2024-03-20T09:04:59.095 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.095 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1ca9b8e:::benchmark_data_smithi122_119671_object40889:head# 2024-03-20T09:04:59.095 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.095 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1caa934:::benchmark_data_smithi122_119671_object79485:head# 2024-03-20T09:04:59.095 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.095 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1caece3:::benchmark_data_smithi122_119671_object3831:head# 2024-03-20T09:04:59.095 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.095 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1cbfa33:::benchmark_data_smithi122_119671_object87875:head# 2024-03-20T09:04:59.095 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.095 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1cc582e:::benchmark_data_smithi122_119671_object56379:head# 2024-03-20T09:04:59.095 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.095 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1cc590a:::benchmark_data_smithi122_119671_object86606:head# 2024-03-20T09:04:59.095 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.095 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1ccb9c5:::benchmark_data_smithi122_119671_object35249:head# 2024-03-20T09:04:59.095 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.096 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1cce97a:::benchmark_data_smithi122_119671_object37707:head# 2024-03-20T09:04:59.096 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.096 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1ce1b70:::benchmark_data_smithi122_119671_object97905:head# 2024-03-20T09:04:59.096 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.096 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1cec431:::benchmark_data_smithi122_119671_object5854:head# 2024-03-20T09:04:59.096 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.096 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1cf45b2:::benchmark_data_smithi122_119671_object92088:head# 2024-03-20T09:04:59.096 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.096 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1cf4843:::benchmark_data_smithi122_119671_object87956:head# 2024-03-20T09:04:59.097 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.097 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1cfcba0:::benchmark_data_smithi122_119671_object7131:head# 2024-03-20T09:04:59.097 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.097 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1d0a67b:::benchmark_data_smithi122_119671_object90251:head# 2024-03-20T09:04:59.097 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.097 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1d0fdbb:::benchmark_data_smithi122_119671_object39244:head# 2024-03-20T09:04:59.097 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.097 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1d4a6c6:::benchmark_data_smithi122_119671_object36977:head# 2024-03-20T09:04:59.097 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.097 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1d4e62d:::benchmark_data_smithi122_119671_object21442:head# 2024-03-20T09:04:59.097 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.097 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1d5975d:::benchmark_data_smithi122_119671_object34361:head# 2024-03-20T09:04:59.097 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.097 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1d72e2b:::benchmark_data_smithi122_119671_object8715:head# 2024-03-20T09:04:59.098 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.098 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1d8a6d6:::benchmark_data_smithi122_119671_object5701:head# 2024-03-20T09:04:59.098 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.098 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1da2ab7:::benchmark_data_smithi122_119671_object81339:head# 2024-03-20T09:04:59.098 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.098 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1da3286:::benchmark_data_smithi122_119671_object5432:head# 2024-03-20T09:04:59.098 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.098 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1da48ee:::benchmark_data_smithi122_119671_object59763:head# 2024-03-20T09:04:59.098 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.098 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1da7a02:::benchmark_data_smithi122_119671_object95587:head# 2024-03-20T09:04:59.098 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.098 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1dac1ab:::benchmark_data_smithi122_119671_object15005:head# 2024-03-20T09:04:59.099 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.099 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1db244d:::benchmark_data_smithi122_119671_object10835:head# 2024-03-20T09:04:59.099 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.099 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1dc29aa:::benchmark_data_smithi122_119671_object28601:head# 2024-03-20T09:04:59.099 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.099 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1ddaf1c:::benchmark_data_smithi122_119671_object49246:head# 2024-03-20T09:04:59.099 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.099 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1ddcab3:::benchmark_data_smithi122_119671_object25983:head# 2024-03-20T09:04:59.099 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.099 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1df656a:::benchmark_data_smithi122_119671_object20338:head# 2024-03-20T09:04:59.099 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.099 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1dff63b:::benchmark_data_smithi122_119671_object14147:head# 2024-03-20T09:04:59.099 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.099 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1e02660:::benchmark_data_smithi122_119671_object14841:head# 2024-03-20T09:04:59.100 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.100 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1e05312:::benchmark_data_smithi122_119671_object1108:head# 2024-03-20T09:04:59.100 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.100 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1e0e7fc:::benchmark_data_smithi122_119671_object1374:head# 2024-03-20T09:04:59.100 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.100 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1e1a999:::benchmark_data_smithi122_119671_object15209:head# 2024-03-20T09:04:59.100 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.100 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1e1c882:::benchmark_data_smithi122_119671_object74464:head# 2024-03-20T09:04:59.100 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.100 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1e21f15:::benchmark_data_smithi122_119671_object49289:head# 2024-03-20T09:04:59.100 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.100 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1e26233:::benchmark_data_smithi122_119671_object39435:head# 2024-03-20T09:04:59.100 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.100 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1e2cab1:::benchmark_data_smithi122_119671_object72528:head# 2024-03-20T09:04:59.101 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.101 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1e2f83e:::benchmark_data_smithi122_119671_object70300:head# 2024-03-20T09:04:59.101 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.101 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1e3133e:::benchmark_data_smithi122_119671_object77979:head# 2024-03-20T09:04:59.101 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.101 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1e4e251:::benchmark_data_smithi122_119671_object92753:head# 2024-03-20T09:04:59.101 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.101 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1e50767:::benchmark_data_smithi122_119671_object30762:head# 2024-03-20T09:04:59.101 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.101 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1e5b193:::benchmark_data_smithi122_119671_object31829:head# 2024-03-20T09:04:59.101 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.101 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1e5fdf3:::benchmark_data_smithi122_119671_object85379:head# 2024-03-20T09:04:59.101 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.101 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1e608ab:::benchmark_data_smithi122_119671_object26335:head# 2024-03-20T09:04:59.102 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.102 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1e6117d:::benchmark_data_smithi122_119671_object85465:head# 2024-03-20T09:04:59.102 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.102 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1e6f46b:::benchmark_data_smithi122_119671_object62009:head# 2024-03-20T09:04:59.102 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.102 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1e77844:::benchmark_data_smithi122_119671_object15496:head# 2024-03-20T09:04:59.102 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.102 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1e7e8cc:::benchmark_data_smithi122_119671_object37167:head# 2024-03-20T09:04:59.102 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.102 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1e8999a:::benchmark_data_smithi122_119671_object37428:head# 2024-03-20T09:04:59.102 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.102 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1e8de57:::benchmark_data_smithi122_119671_object29970:head# 2024-03-20T09:04:59.102 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.102 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1e91e52:::benchmark_data_smithi122_119671_object41315:head# 2024-03-20T09:04:59.103 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.103 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1e9a751:::benchmark_data_smithi122_119671_object16759:head# 2024-03-20T09:04:59.103 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.103 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1ea9fbe:::benchmark_data_smithi122_119671_object19241:head# 2024-03-20T09:04:59.103 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.103 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1eaaf14:::benchmark_data_smithi122_119671_object4050:head# 2024-03-20T09:04:59.103 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.103 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1eb6377:::benchmark_data_smithi122_119671_object24579:head# 2024-03-20T09:04:59.103 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.103 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1ebb391:::benchmark_data_smithi122_119671_object68022:head# 2024-03-20T09:04:59.103 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.103 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1ebbaca:::benchmark_data_smithi122_119671_object3751:head# 2024-03-20T09:04:59.103 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.103 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1ec1a61:::benchmark_data_smithi122_119671_object89550:head# 2024-03-20T09:04:59.104 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.104 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1ed911e:::benchmark_data_smithi122_119671_object62987:head# 2024-03-20T09:04:59.104 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.104 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1ee76a1:::benchmark_data_smithi122_119671_object38151:head# 2024-03-20T09:04:59.104 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.104 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1f0b18d:::benchmark_data_smithi122_119671_object86229:head# 2024-03-20T09:04:59.104 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.104 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1f0b7a7:::benchmark_data_smithi122_119671_object17072:head# 2024-03-20T09:04:59.104 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.104 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1f17084:::benchmark_data_smithi122_119671_object64014:head# 2024-03-20T09:04:59.104 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.104 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1f2aaa3:::benchmark_data_smithi122_119671_object69355:head# 2024-03-20T09:04:59.104 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.104 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1f2cbca:::benchmark_data_smithi122_119671_object6844:head# 2024-03-20T09:04:59.105 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.105 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1f5986a:::benchmark_data_smithi122_119671_object85122:head# 2024-03-20T09:04:59.105 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.105 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1f65b4d:::benchmark_data_smithi122_119671_object9876:head# 2024-03-20T09:04:59.105 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.105 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1f692b5:::benchmark_data_smithi122_119671_object14574:head# 2024-03-20T09:04:59.105 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.105 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1f79e0e:::benchmark_data_smithi122_119671_object28624:head# 2024-03-20T09:04:59.105 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.105 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1f84efa:::benchmark_data_smithi122_119671_object72039:head# 2024-03-20T09:04:59.105 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.105 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1f8e414:::benchmark_data_smithi122_119671_object58808:head# 2024-03-20T09:04:59.105 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.106 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1f8e87e:::benchmark_data_smithi122_119671_object86730:head# 2024-03-20T09:04:59.106 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.106 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1f8e92e:::benchmark_data_smithi122_119671_object52140:head# 2024-03-20T09:04:59.106 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.106 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1f9bfb6:::benchmark_data_smithi122_119671_object76382:head# 2024-03-20T09:04:59.106 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.106 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1f9f95d:::benchmark_data_smithi122_119671_object36213:head# 2024-03-20T09:04:59.106 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.106 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1fab800:::benchmark_data_smithi122_119671_object50486:head# 2024-03-20T09:04:59.106 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.106 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1fb8a9f:::benchmark_data_smithi122_119671_object10937:head# 2024-03-20T09:04:59.106 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.106 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1fc8947:::benchmark_data_smithi122_119671_object64712:head# 2024-03-20T09:04:59.107 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.107 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1fcb5c6:::benchmark_data_smithi122_119671_object1451:head# 2024-03-20T09:04:59.107 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.107 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1fd2d51:::benchmark_data_smithi122_119671_object85026:head# 2024-03-20T09:04:59.107 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.107 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1ff61e0:::benchmark_data_smithi122_119671_object84547:head# 2024-03-20T09:04:59.107 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.107 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b1fff8f9:::benchmark_data_smithi122_119671_object41162:head# 2024-03-20T09:04:59.107 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.107 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2001cf2:::benchmark_data_smithi122_119671_object82022:head# 2024-03-20T09:04:59.107 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.107 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2007b09:::benchmark_data_smithi122_119671_object9922:head# 2024-03-20T09:04:59.107 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.108 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b20182e8:::benchmark_data_smithi122_119671_object24155:head# 2024-03-20T09:04:59.108 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.108 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b201cbf3:::benchmark_data_smithi122_119671_object70496:head# 2024-03-20T09:04:59.108 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.108 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b201d39c:::benchmark_data_smithi122_119671_object42916:head# 2024-03-20T09:04:59.108 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.108 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b202ea2a:::benchmark_data_smithi122_119671_object58614:head# 2024-03-20T09:04:59.108 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.108 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b202f9b7:::benchmark_data_smithi122_119671_object81159:head# 2024-03-20T09:04:59.108 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.108 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b20331b0:::benchmark_data_smithi122_119671_object56914:head# 2024-03-20T09:04:59.108 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.108 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2042e4f:::benchmark_data_smithi122_119671_object10593:head# 2024-03-20T09:04:59.108 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.109 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2050499:::benchmark_data_smithi122_119671_object10177:head# 2024-03-20T09:04:59.109 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.109 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b206719e:::benchmark_data_smithi122_119671_object14592:head# 2024-03-20T09:04:59.109 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.109 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b207d6ab:::benchmark_data_smithi122_119671_object45632:head# 2024-03-20T09:04:59.109 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.109 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b207f543:::benchmark_data_smithi122_119671_object58260:head# 2024-03-20T09:04:59.109 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.109 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2082d51:::benchmark_data_smithi122_119671_object63032:head# 2024-03-20T09:04:59.109 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.109 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b209cf13:::benchmark_data_smithi122_119671_object5774:head# 2024-03-20T09:04:59.109 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.109 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b20b71d6:::benchmark_data_smithi122_119671_object75378:head# 2024-03-20T09:04:59.109 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.110 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b20bff4c:::benchmark_data_smithi122_119671_object58319:head# 2024-03-20T09:04:59.110 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.110 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b20c6daa:::benchmark_data_smithi122_119671_object35445:head# 2024-03-20T09:04:59.110 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.110 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b20ce561:::benchmark_data_smithi122_119671_object85681:head# 2024-03-20T09:04:59.110 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.110 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b20d3f73:::benchmark_data_smithi122_119671_object65902:head# 2024-03-20T09:04:59.110 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.110 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b20d7df5:::benchmark_data_smithi122_119671_object49612:head# 2024-03-20T09:04:59.110 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.110 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b20e925a:::benchmark_data_smithi122_119671_object10529:head# 2024-03-20T09:04:59.110 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.110 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b20eebd8:::benchmark_data_smithi122_119671_object76975:head# 2024-03-20T09:04:59.110 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.111 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b20f46aa:::benchmark_data_smithi122_119671_object67641:head# 2024-03-20T09:04:59.111 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.111 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b20f61e7:::benchmark_data_smithi122_119671_object14460:head# 2024-03-20T09:04:59.111 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.111 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b20fa327:::benchmark_data_smithi122_119671_object20382:head# 2024-03-20T09:04:59.111 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.111 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2106157:::benchmark_data_smithi122_119671_object84192:head# 2024-03-20T09:04:59.111 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.111 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b210cf05:::benchmark_data_smithi122_119671_object22729:head# 2024-03-20T09:04:59.111 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.111 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b211cb93:::benchmark_data_smithi122_119671_object7822:head# 2024-03-20T09:04:59.111 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.111 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b212e205:::benchmark_data_smithi122_119671_object50779:head# 2024-03-20T09:04:59.111 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.112 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b21372be:::benchmark_data_smithi122_119671_object92462:head# 2024-03-20T09:04:59.112 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.112 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2151ed2:::benchmark_data_smithi122_119671_object69068:head# 2024-03-20T09:04:59.112 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.112 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b215e6e1:::benchmark_data_smithi122_119671_object74864:head# 2024-03-20T09:04:59.112 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.112 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b21618cd:::benchmark_data_smithi122_119671_object9475:head# 2024-03-20T09:04:59.112 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.112 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b21636bb:::benchmark_data_smithi122_119671_object4714:head# 2024-03-20T09:04:59.112 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.112 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b217b35f:::benchmark_data_smithi122_119671_object12776:head# 2024-03-20T09:04:59.112 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.112 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2186c95:::benchmark_data_smithi122_119671_object4292:head# 2024-03-20T09:04:59.112 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.112 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2186e7e:::benchmark_data_smithi122_119671_object49135:head# 2024-03-20T09:04:59.113 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.113 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b218cec3:::benchmark_data_smithi122_119671_object38963:head# 2024-03-20T09:04:59.113 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.113 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b21ae879:::benchmark_data_smithi122_119671_object61884:head# 2024-03-20T09:04:59.113 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.113 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b21b66c1:::benchmark_data_smithi122_119671_object15157:head# 2024-03-20T09:04:59.113 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.113 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b21bc29c:::benchmark_data_smithi122_119671_object11826:head# 2024-03-20T09:04:59.113 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.113 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b21f675a:::benchmark_data_smithi122_119671_object77607:head# 2024-03-20T09:04:59.113 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.113 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2215bff:::benchmark_data_smithi122_119671_object25092:head# 2024-03-20T09:04:59.113 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.113 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b221f5ed:::benchmark_data_smithi122_119671_object75388:head# 2024-03-20T09:04:59.114 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.114 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2223e90:::benchmark_data_smithi122_119671_object19055:head# 2024-03-20T09:04:59.114 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.114 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b223008f:::benchmark_data_smithi122_119671_object45275:head# 2024-03-20T09:04:59.114 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.114 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2231020:::benchmark_data_smithi122_119671_object97622:head# 2024-03-20T09:04:59.114 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.114 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b223713c:::benchmark_data_smithi122_119671_object65983:head# 2024-03-20T09:04:59.114 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.114 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b223d141:::benchmark_data_smithi122_119671_object90467:head# 2024-03-20T09:04:59.114 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.114 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b225edf4:::benchmark_data_smithi122_119671_object80785:head# 2024-03-20T09:04:59.114 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.114 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b225f556:::benchmark_data_smithi122_119671_object45077:head# 2024-03-20T09:04:59.115 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.115 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b22657e4:::benchmark_data_smithi122_119671_object3722:head# 2024-03-20T09:04:59.115 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.115 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b226b841:::benchmark_data_smithi122_119671_object88430:head# 2024-03-20T09:04:59.115 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.115 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b226dfbc:::benchmark_data_smithi122_119671_object59982:head# 2024-03-20T09:04:59.115 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.115 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2279c65:::benchmark_data_smithi122_119671_object44987:head# 2024-03-20T09:04:59.115 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.115 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b22a23a2:::benchmark_data_smithi122_119671_object55100:head# 2024-03-20T09:04:59.115 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.115 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b22af9bc:::benchmark_data_smithi122_119671_object22317:head# 2024-03-20T09:04:59.115 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.115 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b22b1d68:::benchmark_data_smithi122_119671_object37606:head# 2024-03-20T09:04:59.115 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.116 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b22b36c5:::benchmark_data_smithi122_119671_object22792:head# 2024-03-20T09:04:59.116 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.116 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b22b416d:::benchmark_data_smithi122_119671_object45144:head# 2024-03-20T09:04:59.116 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.116 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b22c94eb:::benchmark_data_smithi122_119671_object9830:head# 2024-03-20T09:04:59.116 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.116 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b22d5832:::benchmark_data_smithi122_119671_object27033:head# 2024-03-20T09:04:59.116 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.116 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b22dac6b:::benchmark_data_smithi122_119671_object44918:head# 2024-03-20T09:04:59.116 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.116 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b22e541d:::benchmark_data_smithi122_119671_object40541:head# 2024-03-20T09:04:59.116 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.116 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b22f821f:::benchmark_data_smithi122_119671_object6254:head# 2024-03-20T09:04:59.116 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.117 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b22f9800:::benchmark_data_smithi122_119671_object72937:head# 2024-03-20T09:04:59.117 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.117 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2307ba7:::benchmark_data_smithi122_119671_object41501:head# 2024-03-20T09:04:59.117 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.117 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b23192f8:::benchmark_data_smithi122_119671_object48018:head# 2024-03-20T09:04:59.117 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.117 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b231d354:::benchmark_data_smithi122_119671_object74453:head# 2024-03-20T09:04:59.117 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.117 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2326bbf:::benchmark_data_smithi122_119671_object92717:head# 2024-03-20T09:04:59.117 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.117 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b232a7e9:::benchmark_data_smithi122_119671_object9780:head# 2024-03-20T09:04:59.117 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.117 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b232e4fe:::benchmark_data_smithi122_119671_object8520:head# 2024-03-20T09:04:59.117 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.117 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2337aa1:::benchmark_data_smithi122_119671_object28185:head# 2024-03-20T09:04:59.118 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.118 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b233bb2d:::benchmark_data_smithi122_119671_object16628:head# 2024-03-20T09:04:59.118 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.118 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b233c4b9:::benchmark_data_smithi122_119671_object48663:head# 2024-03-20T09:04:59.118 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.118 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2343827:::benchmark_data_smithi122_119671_object18882:head# 2024-03-20T09:04:59.118 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.118 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b23457a7:::benchmark_data_smithi122_119671_object33256:head# 2024-03-20T09:04:59.118 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.118 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b23597bd:::benchmark_data_smithi122_119671_object4553:head# 2024-03-20T09:04:59.118 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.118 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b235e6ee:::benchmark_data_smithi122_119671_object18969:head# 2024-03-20T09:04:59.118 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.118 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b23721a0:::benchmark_data_smithi122_119671_object33946:head# 2024-03-20T09:04:59.119 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.119 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b237695a:::benchmark_data_smithi122_119671_object89323:head# 2024-03-20T09:04:59.119 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.119 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b237a956:::benchmark_data_smithi122_119671_object60759:head# 2024-03-20T09:04:59.119 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.119 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b238e1a4:::benchmark_data_smithi122_119671_object39289:head# 2024-03-20T09:04:59.119 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.119 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b23a00f1:::benchmark_data_smithi122_119671_object25748:head# 2024-03-20T09:04:59.119 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.119 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b23aa6e7:::benchmark_data_smithi122_119671_object95138:head# 2024-03-20T09:04:59.119 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.119 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b23af437:::benchmark_data_smithi122_119671_object40875:head# 2024-03-20T09:04:59.119 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.119 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b23c6153:::benchmark_data_smithi122_119671_object76447:head# 2024-03-20T09:04:59.119 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.120 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b23fc64f:::benchmark_data_smithi122_119671_object76394:head# 2024-03-20T09:04:59.120 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.120 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b242d0d7:::benchmark_data_smithi122_119671_object30326:head# 2024-03-20T09:04:59.120 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.120 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b242f6b4:::benchmark_data_smithi122_119671_object97087:head# 2024-03-20T09:04:59.120 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.120 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b242fb39:::benchmark_data_smithi122_119671_object65509:head# 2024-03-20T09:04:59.120 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.120 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b243fb88:::benchmark_data_smithi122_119671_object20132:head# 2024-03-20T09:04:59.120 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.120 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2445319:::benchmark_data_smithi122_119671_object66225:head# 2024-03-20T09:04:59.120 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.120 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2447118:::benchmark_data_smithi122_119671_object70865:head# 2024-03-20T09:04:59.121 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.121 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b245a384:::benchmark_data_smithi122_119671_object9658:head# 2024-03-20T09:04:59.121 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.121 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b245d360:::benchmark_data_smithi122_119671_object95161:head# 2024-03-20T09:04:59.121 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.121 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b246dd7f:::benchmark_data_smithi122_119671_object2466:head# 2024-03-20T09:04:59.121 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.121 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b247a281:::benchmark_data_smithi122_119671_object13469:head# 2024-03-20T09:04:59.121 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.121 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b247e872:::benchmark_data_smithi122_119671_object41935:head# 2024-03-20T09:04:59.121 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.121 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b24951ed:::benchmark_data_smithi122_119671_object47017:head# 2024-03-20T09:04:59.121 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.121 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b249a5d8:::benchmark_data_smithi122_119671_object72256:head# 2024-03-20T09:04:59.122 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.122 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b249dd2b:::benchmark_data_smithi122_119671_object41957:head# 2024-03-20T09:04:59.122 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.122 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b24abda6:::benchmark_data_smithi122_119671_object6120:head# 2024-03-20T09:04:59.122 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.122 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b24bada4:::benchmark_data_smithi122_119671_object8717:head# 2024-03-20T09:04:59.122 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.122 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b24bd5c8:::benchmark_data_smithi122_119671_object93142:head# 2024-03-20T09:04:59.122 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.122 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b24cf720:::benchmark_data_smithi122_119671_object61223:head# 2024-03-20T09:04:59.122 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.122 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b24d8e1c:::benchmark_data_smithi122_119671_object28283:head# 2024-03-20T09:04:59.122 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.122 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b24dc05d:::benchmark_data_smithi122_119671_object80398:head# 2024-03-20T09:04:59.123 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.123 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b24e117a:::benchmark_data_smithi122_119671_object59302:head# 2024-03-20T09:04:59.123 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.123 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b24ea8ad:::benchmark_data_smithi122_119671_object39333:head# 2024-03-20T09:04:59.123 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.123 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b24ec9a4:::benchmark_data_smithi122_119671_object71349:head# 2024-03-20T09:04:59.123 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.123 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2503cf5:::benchmark_data_smithi122_119671_object33482:head# 2024-03-20T09:04:59.123 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.123 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2523cf9:::benchmark_data_smithi122_119671_object96001:head# 2024-03-20T09:04:59.123 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.123 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b252a024:::benchmark_data_smithi122_119671_object8125:head# 2024-03-20T09:04:59.123 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.123 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b253822f:::benchmark_data_smithi122_119671_object76495:head# 2024-03-20T09:04:59.124 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.124 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2544794:::benchmark_data_smithi122_119671_object84860:head# 2024-03-20T09:04:59.124 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.124 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b25498ed:::benchmark_data_smithi122_119671_object47700:head# 2024-03-20T09:04:59.124 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.124 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b255622d:::benchmark_data_smithi122_119671_object47671:head# 2024-03-20T09:04:59.124 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.124 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b255acc5:::benchmark_data_smithi122_119671_object66085:head# 2024-03-20T09:04:59.124 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.124 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b255e358:::benchmark_data_smithi122_119671_object52161:head# 2024-03-20T09:04:59.124 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.124 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b255e66c:::benchmark_data_smithi122_119671_object24235:head# 2024-03-20T09:04:59.124 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.124 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b25607d3:::benchmark_data_smithi122_119671_object33328:head# 2024-03-20T09:04:59.125 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.125 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2567b1f:::benchmark_data_smithi122_119671_object80731:head# 2024-03-20T09:04:59.125 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.125 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b259653c:::benchmark_data_smithi122_119671_object72192:head# 2024-03-20T09:04:59.125 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.125 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b25a01e9:::benchmark_data_smithi122_119671_object58963:head# 2024-03-20T09:04:59.125 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.125 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b25a8c24:::benchmark_data_smithi122_119671_object68389:head# 2024-03-20T09:04:59.125 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.125 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b25b3662:::benchmark_data_smithi122_119671_object75057:head# 2024-03-20T09:04:59.125 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.125 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b25b796f:::benchmark_data_smithi122_119671_object45694:head# 2024-03-20T09:04:59.125 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.125 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b25bb26a:::benchmark_data_smithi122_119671_object90721:head# 2024-03-20T09:04:59.126 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.126 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b25c833a:::benchmark_data_smithi122_119671_object17622:head# 2024-03-20T09:04:59.126 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.126 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b25c9866:::benchmark_data_smithi122_119671_object13436:head# 2024-03-20T09:04:59.126 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.126 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b25cccc7:::benchmark_data_smithi122_119671_object67945:head# 2024-03-20T09:04:59.126 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.126 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b25d732c:::benchmark_data_smithi122_119671_object30441:head# 2024-03-20T09:04:59.126 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.126 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b25e0966:::benchmark_data_smithi122_119671_object31236:head# 2024-03-20T09:04:59.126 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.126 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b25ed2ca:::benchmark_data_smithi122_119671_object87001:head# 2024-03-20T09:04:59.126 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.126 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b25f0a55:::benchmark_data_smithi122_119671_object43621:head# 2024-03-20T09:04:59.127 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.127 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b25f4ec2:::benchmark_data_smithi122_119671_object70986:head# 2024-03-20T09:04:59.127 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.127 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b26215f3:::benchmark_data_smithi122_119671_object31182:head# 2024-03-20T09:04:59.127 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.127 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b263510e:::benchmark_data_smithi122_119671_object84823:head# 2024-03-20T09:04:59.127 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.127 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2635542:::benchmark_data_smithi122_119671_object73909:head# 2024-03-20T09:04:59.127 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.127 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b263f158:::benchmark_data_smithi122_119671_object54908:head# 2024-03-20T09:04:59.127 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.127 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2641e79:::benchmark_data_smithi122_119671_object42222:head# 2024-03-20T09:04:59.127 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.127 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b266e8f4:::benchmark_data_smithi122_119671_object62372:head# 2024-03-20T09:04:59.128 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.128 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b267a1b0:::benchmark_data_smithi122_119671_object91949:head# 2024-03-20T09:04:59.128 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.128 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b26816ed:::benchmark_data_smithi122_119671_object13340:head# 2024-03-20T09:04:59.128 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.128 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2690fba:::benchmark_data_smithi122_119671_object97449:head# 2024-03-20T09:04:59.128 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.128 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2695429:::benchmark_data_smithi122_119671_object79319:head# 2024-03-20T09:04:59.128 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.128 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b269a062:::benchmark_data_smithi122_119671_object35244:head# 2024-03-20T09:04:59.128 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.128 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b26a3665:::benchmark_data_smithi122_119671_object38696:head# 2024-03-20T09:04:59.128 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.128 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b26a8379:::benchmark_data_smithi122_119671_object73770:head# 2024-03-20T09:04:59.128 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.129 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b26adbfa:::benchmark_data_smithi122_119671_object89734:head# 2024-03-20T09:04:59.129 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.129 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b26b02a8:::benchmark_data_smithi122_119671_object23926:head# 2024-03-20T09:04:59.129 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.129 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b26ba7e2:::benchmark_data_smithi122_119671_object21518:head# 2024-03-20T09:04:59.129 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.129 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b26bab88:::benchmark_data_smithi122_119671_object9004:head# 2024-03-20T09:04:59.129 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.129 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b26d0e45:::benchmark_data_smithi122_119671_object86893:head# 2024-03-20T09:04:59.129 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.129 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b26d775c:::benchmark_data_smithi122_119671_object23327:head# 2024-03-20T09:04:59.129 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.129 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b26f3e36:::benchmark_data_smithi122_119671_object2179:head# 2024-03-20T09:04:59.129 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.130 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b26fa22a:::benchmark_data_smithi122_119671_object61677:head# 2024-03-20T09:04:59.130 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.130 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b271aaaf:::benchmark_data_smithi122_119671_object51655:head# 2024-03-20T09:04:59.130 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.130 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2733ab9:::benchmark_data_smithi122_119671_object30102:head# 2024-03-20T09:04:59.130 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.130 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2756ba7:::benchmark_data_smithi122_119671_object87701:head# 2024-03-20T09:04:59.130 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.130 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b276045f:::benchmark_data_smithi122_119671_object36962:head# 2024-03-20T09:04:59.130 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.130 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b276b9a9:::benchmark_data_smithi122_119671_object50154:head# 2024-03-20T09:04:59.130 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.130 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b27798be:::benchmark_data_smithi122_119671_object87744:head# 2024-03-20T09:04:59.131 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.131 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b277ec5d:::benchmark_data_smithi122_119671_object34128:head# 2024-03-20T09:04:59.131 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.131 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b278888c:::benchmark_data_smithi122_119671_object62179:head# 2024-03-20T09:04:59.131 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.131 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2796db3:::benchmark_data_smithi122_119671_object94873:head# 2024-03-20T09:04:59.131 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.131 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b27ab3f6:::benchmark_data_smithi122_119671_object95447:head# 2024-03-20T09:04:59.131 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.131 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b27aca48:::benchmark_data_smithi122_119671_object48507:head# 2024-03-20T09:04:59.131 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.131 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b27b6f22:::benchmark_data_smithi122_119671_object88777:head# 2024-03-20T09:04:59.131 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.131 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b27bc9c4:::benchmark_data_smithi122_119671_object18731:head# 2024-03-20T09:04:59.131 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.132 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b27c2a98:::benchmark_data_smithi122_119671_object53824:head# 2024-03-20T09:04:59.132 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.132 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b27c80c7:::benchmark_data_smithi122_119671_object7600:head# 2024-03-20T09:04:59.132 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.132 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b27c9d8a:::benchmark_data_smithi122_119671_object4025:head# 2024-03-20T09:04:59.132 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.132 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b27e2804:::benchmark_data_smithi122_119671_object9155:head# 2024-03-20T09:04:59.132 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.132 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b27e49ee:::benchmark_data_smithi122_119671_object20809:head# 2024-03-20T09:04:59.132 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.132 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b28126e5:::benchmark_data_smithi122_119671_object63258:head# 2024-03-20T09:04:59.132 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.132 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b281487b:::benchmark_data_smithi122_119671_object51476:head# 2024-03-20T09:04:59.132 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.133 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b281671f:::benchmark_data_smithi122_119671_object65528:head# 2024-03-20T09:04:59.133 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.133 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2822738:::benchmark_data_smithi122_119671_object81033:head# 2024-03-20T09:04:59.133 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.133 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2830752:::benchmark_data_smithi122_119671_object23598:head# 2024-03-20T09:04:59.133 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.133 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2832306:::benchmark_data_smithi122_119671_object68511:head# 2024-03-20T09:04:59.133 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.133 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2839a8b:::benchmark_data_smithi122_119671_object5206:head# 2024-03-20T09:04:59.133 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.133 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b283b3a0:::benchmark_data_smithi122_119671_object50464:head# 2024-03-20T09:04:59.133 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.133 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b283ee3c:::benchmark_data_smithi122_119671_object52371:head# 2024-03-20T09:04:59.134 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.134 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2844600:::benchmark_data_smithi122_119671_object42088:head# 2024-03-20T09:04:59.134 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.134 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2848ff0:::benchmark_data_smithi122_119671_object97978:head# 2024-03-20T09:04:59.134 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.134 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b28535b1:::benchmark_data_smithi122_119671_object32930:head# 2024-03-20T09:04:59.134 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.134 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b28564e5:::benchmark_data_smithi122_119671_object40028:head# 2024-03-20T09:04:59.134 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.134 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b285cf0d:::benchmark_data_smithi122_119671_object36407:head# 2024-03-20T09:04:59.134 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.134 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b28728dc:::benchmark_data_smithi122_119671_object65239:head# 2024-03-20T09:04:59.134 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.134 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b287592c:::benchmark_data_smithi122_119671_object76304:head# 2024-03-20T09:04:59.135 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.135 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2883378:::benchmark_data_smithi122_119671_object60853:head# 2024-03-20T09:04:59.135 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.135 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2887627:::benchmark_data_smithi122_119671_object89462:head# 2024-03-20T09:04:59.135 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.135 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b28914cf:::benchmark_data_smithi122_119671_object61285:head# 2024-03-20T09:04:59.135 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.135 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b28b05fb:::benchmark_data_smithi122_119671_object97026:head# 2024-03-20T09:04:59.135 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.135 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b28b452c:::benchmark_data_smithi122_119671_object42125:head# 2024-03-20T09:04:59.135 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.135 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b28b82a7:::benchmark_data_smithi122_119671_object19226:head# 2024-03-20T09:04:59.135 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.135 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b28c9456:::benchmark_data_smithi122_119671_object63560:head# 2024-03-20T09:04:59.136 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.136 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b28eed67:::benchmark_data_smithi122_119671_object58034:head# 2024-03-20T09:04:59.136 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.136 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b28f1a77:::benchmark_data_smithi122_119671_object66409:head# 2024-03-20T09:04:59.136 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.136 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b28f71f1:::benchmark_data_smithi122_119671_object3874:head# 2024-03-20T09:04:59.136 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.136 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b28ffbe9:::benchmark_data_smithi122_119671_object12749:head# 2024-03-20T09:04:59.136 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.136 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2907b7a:::benchmark_data_smithi122_119671_object78778:head# 2024-03-20T09:04:59.136 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.136 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2907e58:::benchmark_data_smithi122_119671_object67494:head# 2024-03-20T09:04:59.136 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.136 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b292967d:::benchmark_data_smithi122_119671_object89845:head# 2024-03-20T09:04:59.136 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.137 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2967960:::benchmark_data_smithi122_119671_object37674:head# 2024-03-20T09:04:59.137 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.137 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b296a1af:::benchmark_data_smithi122_119671_object34654:head# 2024-03-20T09:04:59.137 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.137 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2971b62:::benchmark_data_smithi122_119671_object47316:head# 2024-03-20T09:04:59.137 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.137 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b297a4ba:::benchmark_data_smithi122_119671_object22591:head# 2024-03-20T09:04:59.137 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.137 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b297b4be:::benchmark_data_smithi122_119671_object28067:head# 2024-03-20T09:04:59.137 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.137 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b297bee4:::benchmark_data_smithi122_119671_object59664:head# 2024-03-20T09:04:59.137 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.137 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2984913:::benchmark_data_smithi122_119671_object84576:head# 2024-03-20T09:04:59.137 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.138 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b298f043:::benchmark_data_smithi122_119671_object86640:head# 2024-03-20T09:04:59.138 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.138 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b298f896:::benchmark_data_smithi122_119671_object8111:head# 2024-03-20T09:04:59.138 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.138 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b29909c9:::benchmark_data_smithi122_119671_object17816:head# 2024-03-20T09:04:59.138 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.138 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b299acc8:::benchmark_data_smithi122_119671_object8933:head# 2024-03-20T09:04:59.138 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.138 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b299af40:::benchmark_data_smithi122_119671_object10364:head# 2024-03-20T09:04:59.138 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.138 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b29a8590:::benchmark_data_smithi122_119671_object36037:head# 2024-03-20T09:04:59.138 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.138 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b29b5742:::benchmark_data_smithi122_119671_object55628:head# 2024-03-20T09:04:59.138 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.139 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b29c9435:::benchmark_data_smithi122_119671_object79945:head# 2024-03-20T09:04:59.139 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.139 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b29cac20:::benchmark_data_smithi122_119671_object92843:head# 2024-03-20T09:04:59.139 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.139 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b29cd263:::benchmark_data_smithi122_119671_object82011:head# 2024-03-20T09:04:59.139 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.139 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b29e78c4:::benchmark_data_smithi122_119671_object12241:head# 2024-03-20T09:04:59.139 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.139 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b29f39e1:::benchmark_data_smithi122_119671_object54968:head# 2024-03-20T09:04:59.139 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.139 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b29f98e0:::benchmark_data_smithi122_119671_object56759:head# 2024-03-20T09:04:59.139 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.139 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2a051cb:::benchmark_data_smithi122_119671_object10167:head# 2024-03-20T09:04:59.139 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.139 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2a092d0:::benchmark_data_smithi122_119671_object6633:head# 2024-03-20T09:04:59.140 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.140 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2a3bb7d:::benchmark_data_smithi122_119671_object55057:head# 2024-03-20T09:04:59.140 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.155 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:04:59.162 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:04:59.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:59.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:04:59.271 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2a45119:::benchmark_data_smithi122_119671_object14434:head# 2024-03-20T09:04:59.271 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.271 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2a4a1e6:::benchmark_data_smithi122_119671_object7585:head# 2024-03-20T09:04:59.271 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.271 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2a4eb3e:::benchmark_data_smithi122_119671_object56233:head# 2024-03-20T09:04:59.271 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.271 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2a514cf:::benchmark_data_smithi122_119671_object90768:head# 2024-03-20T09:04:59.271 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.272 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2a5b9b5:::benchmark_data_smithi122_119671_object17213:head# 2024-03-20T09:04:59.272 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.272 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2a6ca94:::benchmark_data_smithi122_119671_object49212:head# 2024-03-20T09:04:59.272 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.272 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2a7a888:::benchmark_data_smithi122_119671_object97887:head# 2024-03-20T09:04:59.272 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.272 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2abadd9:::benchmark_data_smithi122_119671_object48231:head# 2024-03-20T09:04:59.272 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.272 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2abfb6f:::benchmark_data_smithi122_119671_object94536:head# 2024-03-20T09:04:59.272 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.272 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2ad52da:::benchmark_data_smithi122_119671_object67213:head# 2024-03-20T09:04:59.272 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.272 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2ad935a:::benchmark_data_smithi122_119671_object88246:head# 2024-03-20T09:04:59.272 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.273 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2ae0d34:::benchmark_data_smithi122_119671_object18568:head# 2024-03-20T09:04:59.273 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.273 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2ae38f5:::benchmark_data_smithi122_119671_object10693:head# 2024-03-20T09:04:59.273 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.273 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2ae3baf:::benchmark_data_smithi122_119671_object24827:head# 2024-03-20T09:04:59.273 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.273 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2ae543c:::benchmark_data_smithi122_119671_object81583:head# 2024-03-20T09:04:59.273 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.273 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2aea3ec:::benchmark_data_smithi122_119671_object51255:head# 2024-03-20T09:04:59.273 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.273 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2af2068:::benchmark_data_smithi122_119671_object84189:head# 2024-03-20T09:04:59.273 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.273 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2af8242:::benchmark_data_smithi122_119671_object48929:head# 2024-03-20T09:04:59.273 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.274 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2b02297:::benchmark_data_smithi122_119671_object34472:head# 2024-03-20T09:04:59.274 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.274 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2b02d9a:::benchmark_data_smithi122_119671_object88501:head# 2024-03-20T09:04:59.274 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.274 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2b0f1e8:::benchmark_data_smithi122_119671_object17534:head# 2024-03-20T09:04:59.274 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.274 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2b147de:::benchmark_data_smithi122_119671_object44727:head# 2024-03-20T09:04:59.274 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.274 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2b1fe15:::benchmark_data_smithi122_119671_object45111:head# 2024-03-20T09:04:59.274 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.274 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2b3a5fa:::benchmark_data_smithi122_119671_object56901:head# 2024-03-20T09:04:59.274 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.274 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2b55eb4:::benchmark_data_smithi122_119671_object29826:head# 2024-03-20T09:04:59.274 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.275 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2b75f78:::benchmark_data_smithi122_119671_object28183:head# 2024-03-20T09:04:59.275 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.275 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2b7ae98:::benchmark_data_smithi122_119671_object35468:head# 2024-03-20T09:04:59.275 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.275 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2b7ddb5:::benchmark_data_smithi122_119671_object82418:head# 2024-03-20T09:04:59.275 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.275 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2b88957:::benchmark_data_smithi122_119671_object88950:head# 2024-03-20T09:04:59.275 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.275 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2bae7b7:::benchmark_data_smithi122_119671_object39787:head# 2024-03-20T09:04:59.275 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.275 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2baed2e:::benchmark_data_smithi122_119671_object73561:head# 2024-03-20T09:04:59.275 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.275 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2bbcaf4:::benchmark_data_smithi122_119671_object99758:head# 2024-03-20T09:04:59.275 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.276 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2bd11a8:::benchmark_data_smithi122_119671_object29829:head# 2024-03-20T09:04:59.276 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.276 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2bd1a45:::benchmark_data_smithi122_119671_object96197:head# 2024-03-20T09:04:59.276 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.276 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2bd2841:::benchmark_data_smithi122_119671_object76954:head# 2024-03-20T09:04:59.276 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.276 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2bd2bfd:::benchmark_data_smithi122_119671_object41215:head# 2024-03-20T09:04:59.276 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.276 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2bd4d1d:::benchmark_data_smithi122_119671_object88662:head# 2024-03-20T09:04:59.276 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.276 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2be30d1:::benchmark_data_smithi122_119671_object87652:head# 2024-03-20T09:04:59.276 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.276 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2beccf2:::benchmark_data_smithi122_119671_object58254:head# 2024-03-20T09:04:59.276 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.277 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2c04319:::benchmark_data_smithi122_119671_object9419:head# 2024-03-20T09:04:59.277 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.277 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2c09f01:::benchmark_data_smithi122_119671_object66337:head# 2024-03-20T09:04:59.277 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.277 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2c245bd:::benchmark_data_smithi122_119671_object41819:head# 2024-03-20T09:04:59.277 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.277 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2c25d52:::benchmark_data_smithi122_119671_object4918:head# 2024-03-20T09:04:59.277 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.277 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2c41ba3:::benchmark_data_smithi122_119671_object63298:head# 2024-03-20T09:04:59.277 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.277 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2c458ff:::benchmark_data_smithi122_119671_object4690:head# 2024-03-20T09:04:59.277 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.277 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2c4a852:::benchmark_data_smithi122_119671_object78248:head# 2024-03-20T09:04:59.277 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.278 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2c56c90:::benchmark_data_smithi122_119671_object62273:head# 2024-03-20T09:04:59.278 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.278 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2c65716:::benchmark_data_smithi122_119671_object68174:head# 2024-03-20T09:04:59.278 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.278 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2c681cb:::benchmark_data_smithi122_119671_object50316:head# 2024-03-20T09:04:59.278 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.278 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2c79261:::benchmark_data_smithi122_119671_object99576:head# 2024-03-20T09:04:59.278 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.278 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2c7d9fc:::benchmark_data_smithi122_119671_object99318:head# 2024-03-20T09:04:59.278 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.278 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2c8486e:::benchmark_data_smithi122_119671_object11463:head# 2024-03-20T09:04:59.278 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.278 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2c92f94:::benchmark_data_smithi122_119671_object33561:head# 2024-03-20T09:04:59.279 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.279 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2ca6fd8:::benchmark_data_smithi122_119671_object25268:head# 2024-03-20T09:04:59.279 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.279 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2cbaeaa:::benchmark_data_smithi122_119671_object58902:head# 2024-03-20T09:04:59.279 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.279 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2cd55c9:::benchmark_data_smithi122_119671_object53550:head# 2024-03-20T09:04:59.279 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.279 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2ce8e9e:::benchmark_data_smithi122_119671_object98693:head# 2024-03-20T09:04:59.279 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.279 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2cebe52:::benchmark_data_smithi122_119671_object63146:head# 2024-03-20T09:04:59.279 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.279 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2d03082:::benchmark_data_smithi122_119671_object88733:head# 2024-03-20T09:04:59.279 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.279 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2d15cc8:::benchmark_data_smithi122_119671_object57726:head# 2024-03-20T09:04:59.280 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.280 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2d1cee5:::benchmark_data_smithi122_119671_object48404:head# 2024-03-20T09:04:59.280 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.280 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2d301de:::benchmark_data_smithi122_119671_object57276:head# 2024-03-20T09:04:59.280 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.280 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2d33c30:::benchmark_data_smithi122_119671_object86879:head# 2024-03-20T09:04:59.280 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.280 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2d35c8f:::benchmark_data_smithi122_119671_object30101:head# 2024-03-20T09:04:59.280 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.280 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2d3c745:::benchmark_data_smithi122_119671_object77184:head# 2024-03-20T09:04:59.280 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.280 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2d5fd9d:::benchmark_data_smithi122_119671_object6237:head# 2024-03-20T09:04:59.280 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.281 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2d6e133:::benchmark_data_smithi122_119671_object72548:head# 2024-03-20T09:04:59.281 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.281 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2d8b121:::benchmark_data_smithi122_119671_object71324:head# 2024-03-20T09:04:59.281 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.281 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2d97113:::benchmark_data_smithi122_119671_object9288:head# 2024-03-20T09:04:59.281 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.281 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2dadd73:::benchmark_data_smithi122_119671_object49359:head# 2024-03-20T09:04:59.281 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.281 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2db58ba:::benchmark_data_smithi122_119671_object82285:head# 2024-03-20T09:04:59.281 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.281 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2db6632:::benchmark_data_smithi122_119671_object17100:head# 2024-03-20T09:04:59.281 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.281 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2dcba1f:::benchmark_data_smithi122_119671_object79162:head# 2024-03-20T09:04:59.281 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.282 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2dd641c:::benchmark_data_smithi122_119671_object92042:head# 2024-03-20T09:04:59.282 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.282 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2ddde11:::benchmark_data_smithi122_119671_object58030:head# 2024-03-20T09:04:59.282 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.282 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2de22a5:::benchmark_data_smithi122_119671_object12495:head# 2024-03-20T09:04:59.282 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.282 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2de4c04:::benchmark_data_smithi122_119671_object81134:head# 2024-03-20T09:04:59.282 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.282 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2def9f9:::benchmark_data_smithi122_119671_object99398:head# 2024-03-20T09:04:59.282 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.282 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2dfa72f:::benchmark_data_smithi122_119671_object65832:head# 2024-03-20T09:04:59.282 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.282 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2dfb7d1:::benchmark_data_smithi122_119671_object41899:head# 2024-03-20T09:04:59.282 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.283 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2e00a6a:::benchmark_data_smithi122_119671_object27179:head# 2024-03-20T09:04:59.283 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.283 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2e03105:::benchmark_data_smithi122_119671_object21912:head# 2024-03-20T09:04:59.283 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.283 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2e09fe5:::benchmark_data_smithi122_119671_object76208:head# 2024-03-20T09:04:59.283 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.283 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2e0a52f:::benchmark_data_smithi122_119671_object18696:head# 2024-03-20T09:04:59.283 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.283 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2e1e854:::benchmark_data_smithi122_119671_object90686:head# 2024-03-20T09:04:59.283 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.284 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2e25ea8:::benchmark_data_smithi122_119671_object56351:head# 2024-03-20T09:04:59.284 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.284 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2e3cdd6:::benchmark_data_smithi122_119671_object4339:head# 2024-03-20T09:04:59.284 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.284 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2e4206c:::benchmark_data_smithi122_119671_object10646:head# 2024-03-20T09:04:59.284 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.284 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2e46778:::benchmark_data_smithi122_119671_object65360:head# 2024-03-20T09:04:59.284 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.284 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2e4bb4f:::benchmark_data_smithi122_119671_object99649:head# 2024-03-20T09:04:59.284 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.284 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2e5b6a3:::benchmark_data_smithi122_119671_object5339:head# 2024-03-20T09:04:59.284 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.285 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2e5e4b9:::benchmark_data_smithi122_119671_object40723:head# 2024-03-20T09:04:59.285 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.285 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2e6421c:::benchmark_data_smithi122_119671_object83575:head# 2024-03-20T09:04:59.285 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.285 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2e89cf7:::benchmark_data_smithi122_119671_object80862:head# 2024-03-20T09:04:59.285 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.285 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2e8e64a:::benchmark_data_smithi122_119671_object86630:head# 2024-03-20T09:04:59.285 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.285 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2e90497:::benchmark_data_smithi122_119671_object23758:head# 2024-03-20T09:04:59.285 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.286 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2e932a1:::benchmark_data_smithi122_119671_object91801:head# 2024-03-20T09:04:59.286 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.286 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2ea5663:::benchmark_data_smithi122_119671_object55530:head# 2024-03-20T09:04:59.286 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.286 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2eb2163:::benchmark_data_smithi122_119671_object89553:head# 2024-03-20T09:04:59.286 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.286 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2ebadf0:::benchmark_data_smithi122_119671_object24110:head# 2024-03-20T09:04:59.286 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.286 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2ebb3b5:::benchmark_data_smithi122_119671_object2705:head# 2024-03-20T09:04:59.286 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.286 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2ec5fc6:::benchmark_data_smithi122_119671_object85721:head# 2024-03-20T09:04:59.286 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.287 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2ec9ed0:::benchmark_data_smithi122_119671_object88032:head# 2024-03-20T09:04:59.287 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.287 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2ecab53:::benchmark_data_smithi122_119671_object15097:head# 2024-03-20T09:04:59.287 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.287 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2ed6ea1:::benchmark_data_smithi122_119671_object40474:head# 2024-03-20T09:04:59.287 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.287 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2ed728a:::benchmark_data_smithi122_119671_object95613:head# 2024-03-20T09:04:59.287 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.287 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2ef14a5:::benchmark_data_smithi122_119671_object60731:head# 2024-03-20T09:04:59.287 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.287 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2effe13:::benchmark_data_smithi122_119671_object66153:head# 2024-03-20T09:04:59.288 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.288 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2f11d36:::benchmark_data_smithi122_119671_object28831:head# 2024-03-20T09:04:59.288 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.288 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2f1e4d2:::benchmark_data_smithi122_119671_object52263:head# 2024-03-20T09:04:59.288 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.288 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2f253ab:::benchmark_data_smithi122_119671_object28148:head# 2024-03-20T09:04:59.288 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.288 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2f270da:::benchmark_data_smithi122_119671_object82643:head# 2024-03-20T09:04:59.288 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.288 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2f36192:::benchmark_data_smithi122_119671_object87521:head# 2024-03-20T09:04:59.288 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.289 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2f3dcbf:::benchmark_data_smithi122_119671_object49607:head# 2024-03-20T09:04:59.289 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.289 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2f578e3:::benchmark_data_smithi122_119671_object90375:head# 2024-03-20T09:04:59.289 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.289 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2f5fb16:::benchmark_data_smithi122_119671_object17185:head# 2024-03-20T09:04:59.289 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.289 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2f6631c:::benchmark_data_smithi122_119671_object73527:head# 2024-03-20T09:04:59.289 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.289 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2f69cd9:::benchmark_data_smithi122_119671_object87963:head# 2024-03-20T09:04:59.289 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.289 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2f7f63c:::benchmark_data_smithi122_119671_object89694:head# 2024-03-20T09:04:59.289 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.290 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2f823b2:::benchmark_data_smithi122_119671_object12349:head# 2024-03-20T09:04:59.290 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.290 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2f934d5:::benchmark_data_smithi122_119671_object26777:head# 2024-03-20T09:04:59.290 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.290 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2f9619b:::benchmark_data_smithi122_119671_object16775:head# 2024-03-20T09:04:59.290 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.290 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2fa72b9:::benchmark_data_smithi122_119671_object43918:head# 2024-03-20T09:04:59.290 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.290 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2faba73:::benchmark_data_smithi122_119671_object65920:head# 2024-03-20T09:04:59.290 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.290 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2fb5c48:::benchmark_data_smithi122_119671_object20898:head# 2024-03-20T09:04:59.290 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.291 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2fc7113:::benchmark_data_smithi122_119671_object8235:head# 2024-03-20T09:04:59.291 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.291 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2fd23b1:::benchmark_data_smithi122_119671_object12509:head# 2024-03-20T09:04:59.291 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.291 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2fd9620:::benchmark_data_smithi122_119671_object65048:head# 2024-03-20T09:04:59.291 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.291 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b2fef3d0:::benchmark_data_smithi122_119671_object82274:head# 2024-03-20T09:04:59.291 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.291 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b301b66c:::benchmark_data_smithi122_119671_object3261:head# 2024-03-20T09:04:59.291 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.291 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b301c795:::benchmark_data_smithi122_119671_object35805:head# 2024-03-20T09:04:59.291 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.292 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3028623:::benchmark_data_smithi122_119671_object37361:head# 2024-03-20T09:04:59.292 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.292 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3036cb7:::benchmark_data_smithi122_119671_object33184:head# 2024-03-20T09:04:59.292 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.292 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b303933f:::benchmark_data_smithi122_119671_object61398:head# 2024-03-20T09:04:59.292 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.292 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b30453d3:::benchmark_data_smithi122_119671_object79779:head# 2024-03-20T09:04:59.292 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.292 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3061895:::benchmark_data_smithi122_119671_object38015:head# 2024-03-20T09:04:59.292 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.292 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b306ba93:::benchmark_data_smithi122_119671_object6530:head# 2024-03-20T09:04:59.292 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.293 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b307b3d5:::benchmark_data_smithi122_119671_object57479:head# 2024-03-20T09:04:59.293 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.293 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3081207:::benchmark_data_smithi122_119671_object69500:head# 2024-03-20T09:04:59.293 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.293 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b309e32a:::benchmark_data_smithi122_119671_object26258:head# 2024-03-20T09:04:59.293 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.293 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b30a67dd:::benchmark_data_smithi122_119671_object99449:head# 2024-03-20T09:04:59.293 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.293 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b30ba6f1:::benchmark_data_smithi122_119671_object60608:head# 2024-03-20T09:04:59.293 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.293 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b30e0839:::benchmark_data_smithi122_119671_object17552:head# 2024-03-20T09:04:59.293 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.294 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b30e4394:::benchmark_data_smithi122_119671_object65658:head# 2024-03-20T09:04:59.294 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.294 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b30e6311:::benchmark_data_smithi122_119671_object61492:head# 2024-03-20T09:04:59.294 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.294 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b30ec037:::benchmark_data_smithi122_119671_object35824:head# 2024-03-20T09:04:59.294 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.294 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b30ec409:::benchmark_data_smithi122_119671_object94359:head# 2024-03-20T09:04:59.294 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.294 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b30fbf36:::benchmark_data_smithi122_119671_object93060:head# 2024-03-20T09:04:59.295 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.295 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3117842:::benchmark_data_smithi122_119671_object40503:head# 2024-03-20T09:04:59.295 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.295 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3128d67:::benchmark_data_smithi122_119671_object46164:head# 2024-03-20T09:04:59.295 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.295 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b312f968:::benchmark_data_smithi122_119671_object53362:head# 2024-03-20T09:04:59.295 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.295 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3136911:::benchmark_data_smithi122_119671_object37371:head# 2024-03-20T09:04:59.295 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.295 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3138ed1:::benchmark_data_smithi122_119671_object47258:head# 2024-03-20T09:04:59.295 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.296 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b316a53e:::benchmark_data_smithi122_119671_object57950:head# 2024-03-20T09:04:59.296 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.296 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b316fbcd:::benchmark_data_smithi122_119671_object1903:head# 2024-03-20T09:04:59.296 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.296 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b317831f:::benchmark_data_smithi122_119671_object52532:head# 2024-03-20T09:04:59.296 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.296 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b317cb7e:::benchmark_data_smithi122_119671_object12917:head# 2024-03-20T09:04:59.296 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.296 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3184647:::benchmark_data_smithi122_119671_object68686:head# 2024-03-20T09:04:59.296 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.296 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3185dfd:::benchmark_data_smithi122_119671_object91019:head# 2024-03-20T09:04:59.297 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.297 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3191027:::benchmark_data_smithi122_119671_object78529:head# 2024-03-20T09:04:59.297 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.297 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b31978f2:::benchmark_data_smithi122_119671_object11388:head# 2024-03-20T09:04:59.297 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.297 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b319a93c:::benchmark_data_smithi122_119671_object19655:head# 2024-03-20T09:04:59.297 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.297 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b319caa3:::benchmark_data_smithi122_119671_object53991:head# 2024-03-20T09:04:59.297 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.297 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b31abf16:::benchmark_data_smithi122_119671_object36468:head# 2024-03-20T09:04:59.297 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.297 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b31b0d8d:::benchmark_data_smithi122_119671_object72879:head# 2024-03-20T09:04:59.298 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.298 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b31b460b:::benchmark_data_smithi122_119671_object54303:head# 2024-03-20T09:04:59.298 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.298 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b31c1210:::benchmark_data_smithi122_119671_object24079:head# 2024-03-20T09:04:59.298 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.298 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b31c4692:::benchmark_data_smithi122_119671_object67631:head# 2024-03-20T09:04:59.298 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.298 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b31ca9dc:::benchmark_data_smithi122_119671_object27857:head# 2024-03-20T09:04:59.298 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.298 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b31d71a0:::benchmark_data_smithi122_119671_object36946:head# 2024-03-20T09:04:59.298 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.298 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b31defa6:::benchmark_data_smithi122_119671_object15391:head# 2024-03-20T09:04:59.299 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.299 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b31ecd99:::benchmark_data_smithi122_119671_object22650:head# 2024-03-20T09:04:59.299 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.299 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b31fa148:::benchmark_data_smithi122_119671_object25840:head# 2024-03-20T09:04:59.299 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.299 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b31faaa3:::benchmark_data_smithi122_119671_object86438:head# 2024-03-20T09:04:59.299 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.299 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3212905:::benchmark_data_smithi122_119671_object27580:head# 2024-03-20T09:04:59.299 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.299 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3217fd7:::benchmark_data_smithi122_119671_object56538:head# 2024-03-20T09:04:59.299 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.299 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b321f413:::benchmark_data_smithi122_119671_object41670:head# 2024-03-20T09:04:59.299 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.300 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3226a3e:::benchmark_data_smithi122_119671_object25620:head# 2024-03-20T09:04:59.300 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.300 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b323aa07:::benchmark_data_smithi122_119671_object51238:head# 2024-03-20T09:04:59.300 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.300 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b323d5b7:::benchmark_data_smithi122_119671_object60981:head# 2024-03-20T09:04:59.300 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.300 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b323ded7:::benchmark_data_smithi122_119671_object75229:head# 2024-03-20T09:04:59.300 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.300 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3247ea9:::benchmark_data_smithi122_119671_object46120:head# 2024-03-20T09:04:59.300 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.300 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b324da61:::benchmark_data_smithi122_119671_object84466:head# 2024-03-20T09:04:59.300 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.301 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3279d58:::benchmark_data_smithi122_119671_object33667:head# 2024-03-20T09:04:59.301 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.301 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3279fcc:::benchmark_data_smithi122_119671_object18203:head# 2024-03-20T09:04:59.301 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.301 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3283b77:::benchmark_data_smithi122_119671_object98235:head# 2024-03-20T09:04:59.301 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.301 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b32a2783:::benchmark_data_smithi122_119671_object48083:head# 2024-03-20T09:04:59.301 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.301 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b32a3cf3:::benchmark_data_smithi122_119671_object18604:head# 2024-03-20T09:04:59.301 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.301 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b32b40b2:::benchmark_data_smithi122_119671_object7497:head# 2024-03-20T09:04:59.301 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.301 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b32c2b7b:::benchmark_data_smithi122_119671_object51843:head# 2024-03-20T09:04:59.302 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.302 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b32c3611:::benchmark_data_smithi122_119671_object73212:head# 2024-03-20T09:04:59.302 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.302 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b32eeb45:::benchmark_data_smithi122_119671_object32324:head# 2024-03-20T09:04:59.302 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.302 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b330ff24:::benchmark_data_smithi122_119671_object58813:head# 2024-03-20T09:04:59.302 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.302 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3315bce:::benchmark_data_smithi122_119671_object56943:head# 2024-03-20T09:04:59.302 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.302 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3318d33:::benchmark_data_smithi122_119671_object86813:head# 2024-03-20T09:04:59.302 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.303 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b331b559:::benchmark_data_smithi122_119671_object34809:head# 2024-03-20T09:04:59.303 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.303 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b332a91c:::benchmark_data_smithi122_119671_object69054:head# 2024-03-20T09:04:59.303 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.303 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b332aa38:::benchmark_data_smithi122_119671_object60477:head# 2024-03-20T09:04:59.303 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.303 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b332ca9d:::benchmark_data_smithi122_119671_object30743:head# 2024-03-20T09:04:59.303 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.303 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3336077:::benchmark_data_smithi122_119671_object64546:head# 2024-03-20T09:04:59.303 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.303 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b333c4fb:::benchmark_data_smithi122_119671_object97811:head# 2024-03-20T09:04:59.303 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.304 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3340c2e:::benchmark_data_smithi122_119671_object91371:head# 2024-03-20T09:04:59.304 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.304 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b334aa7b:::benchmark_data_smithi122_119671_object76991:head# 2024-03-20T09:04:59.304 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.304 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b334eb35:::benchmark_data_smithi122_119671_object71193:head# 2024-03-20T09:04:59.304 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.304 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3352209:::benchmark_data_smithi122_119671_object14678:head# 2024-03-20T09:04:59.304 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.304 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3354d60:::benchmark_data_smithi122_119671_object9557:head# 2024-03-20T09:04:59.304 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.304 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3356a5a:::benchmark_data_smithi122_119671_object99059:head# 2024-03-20T09:04:59.304 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.305 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3367f0d:::benchmark_data_smithi122_119671_object97824:head# 2024-03-20T09:04:59.305 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.305 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b336eae0:::benchmark_data_smithi122_119671_object8472:head# 2024-03-20T09:04:59.305 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.305 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3375741:::benchmark_data_smithi122_119671_object48687:head# 2024-03-20T09:04:59.305 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.305 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b337cf95:::benchmark_data_smithi122_119671_object52556:head# 2024-03-20T09:04:59.305 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.305 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b338308b:::benchmark_data_smithi122_119671_object83263:head# 2024-03-20T09:04:59.305 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.305 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b33864a2:::benchmark_data_smithi122_119671_object98026:head# 2024-03-20T09:04:59.305 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.306 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3397acd:::benchmark_data_smithi122_119671_object68043:head# 2024-03-20T09:04:59.306 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.306 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b33ab69f:::benchmark_data_smithi122_119671_object66486:head# 2024-03-20T09:04:59.306 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.306 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b33b8895:::benchmark_data_smithi122_119671_object13177:head# 2024-03-20T09:04:59.306 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.306 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b33b8903:::benchmark_data_smithi122_119671_object53398:head# 2024-03-20T09:04:59.306 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.306 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b33b95d6:::benchmark_data_smithi122_119671_object71554:head# 2024-03-20T09:04:59.306 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.306 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b33c0298:::benchmark_data_smithi122_119671_object18598:head# 2024-03-20T09:04:59.306 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.307 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b33c3aa7:::benchmark_data_smithi122_119671_object16944:head# 2024-03-20T09:04:59.307 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.307 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b33cd44f:::benchmark_data_smithi122_119671_object52320:head# 2024-03-20T09:04:59.307 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.307 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b33df655:::benchmark_data_smithi122_119671_object39159:head# 2024-03-20T09:04:59.307 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.307 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b33e7fd9:::benchmark_data_smithi122_119671_object82483:head# 2024-03-20T09:04:59.307 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.307 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b33f1a2b:::benchmark_data_smithi122_119671_object94714:head# 2024-03-20T09:04:59.307 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.307 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3403174:::benchmark_data_smithi122_119671_object4629:head# 2024-03-20T09:04:59.307 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.308 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b340fe06:::benchmark_data_smithi122_119671_object84557:head# 2024-03-20T09:04:59.308 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.308 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3411ac7:::benchmark_data_smithi122_119671_object84952:head# 2024-03-20T09:04:59.308 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.308 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3422b57:::benchmark_data_smithi122_119671_object66994:head# 2024-03-20T09:04:59.308 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.308 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3434ab4:::benchmark_data_smithi122_119671_object89493:head# 2024-03-20T09:04:59.308 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.308 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b343d424:::benchmark_data_smithi122_119671_object75963:head# 2024-03-20T09:04:59.308 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.308 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3444aaa:::benchmark_data_smithi122_119671_object40403:head# 2024-03-20T09:04:59.308 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.309 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3446d07:::benchmark_data_smithi122_119671_object73644:head# 2024-03-20T09:04:59.309 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.309 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b346f3d0:::benchmark_data_smithi122_119671_object30008:head# 2024-03-20T09:04:59.309 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.309 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b347f87c:::benchmark_data_smithi122_119671_object95335:head# 2024-03-20T09:04:59.309 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.309 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b349a6ee:::benchmark_data_smithi122_119671_object87664:head# 2024-03-20T09:04:59.309 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.309 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b349dd29:::benchmark_data_smithi122_119671_object95000:head# 2024-03-20T09:04:59.309 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.309 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b349e322:::benchmark_data_smithi122_119671_object70493:head# 2024-03-20T09:04:59.310 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.310 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b349f399:::benchmark_data_smithi122_119671_object68270:head# 2024-03-20T09:04:59.310 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.310 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b34a2ec0:::benchmark_data_smithi122_119671_object51111:head# 2024-03-20T09:04:59.310 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.310 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b34a3cfb:::benchmark_data_smithi122_119671_object35626:head# 2024-03-20T09:04:59.310 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.310 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b34a5456:::benchmark_data_smithi122_119671_object2111:head# 2024-03-20T09:04:59.310 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.310 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b34ab85e:::benchmark_data_smithi122_119671_object51926:head# 2024-03-20T09:04:59.310 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.311 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b34b2813:::benchmark_data_smithi122_119671_object3611:head# 2024-03-20T09:04:59.311 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.311 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b34b8aa1:::benchmark_data_smithi122_119671_object5613:head# 2024-03-20T09:04:59.311 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.311 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b34c23bc:::benchmark_data_smithi122_119671_object88142:head# 2024-03-20T09:04:59.311 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.311 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b34ce07d:::benchmark_data_smithi122_119671_object44373:head# 2024-03-20T09:04:59.311 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.311 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b34cf86a:::benchmark_data_smithi122_119671_object93523:head# 2024-03-20T09:04:59.311 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.311 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b34d3ef4:::benchmark_data_smithi122_119671_object11560:head# 2024-03-20T09:04:59.312 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.312 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b34e3a82:::benchmark_data_smithi122_119671_object26482:head# 2024-03-20T09:04:59.312 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.312 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b34f0f83:::benchmark_data_smithi122_119671_object93282:head# 2024-03-20T09:04:59.312 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.312 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b34fa239:::benchmark_data_smithi122_119671_object59251:head# 2024-03-20T09:04:59.312 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.312 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b350d8df:::benchmark_data_smithi122_119671_object99444:head# 2024-03-20T09:04:59.312 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.312 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3512db9:::benchmark_data_smithi122_119671_object82782:head# 2024-03-20T09:04:59.312 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.312 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3512fcc:::benchmark_data_smithi122_119671_object77550:head# 2024-03-20T09:04:59.312 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.313 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3518f52:::benchmark_data_smithi122_119671_object62557:head# 2024-03-20T09:04:59.313 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.313 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3524a86:::benchmark_data_smithi122_119671_object17527:head# 2024-03-20T09:04:59.313 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.313 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b35268c0:::benchmark_data_smithi122_119671_object19867:head# 2024-03-20T09:04:59.313 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.313 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3531ad1:::benchmark_data_smithi122_119671_object89637:head# 2024-03-20T09:04:59.313 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.313 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3531b5d:::benchmark_data_smithi122_119671_object46263:head# 2024-03-20T09:04:59.313 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.313 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3540b17:::benchmark_data_smithi122_119671_object88819:head# 2024-03-20T09:04:59.313 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.314 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b355349b:::benchmark_data_smithi122_119671_object68450:head# 2024-03-20T09:04:59.314 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.314 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b355623e:::benchmark_data_smithi122_119671_object71676:head# 2024-03-20T09:04:59.314 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.314 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3563035:::benchmark_data_smithi122_119671_object89198:head# 2024-03-20T09:04:59.314 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.314 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b35634bd:::benchmark_data_smithi122_119671_object87520:head# 2024-03-20T09:04:59.314 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.314 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3567935:::benchmark_data_smithi122_119671_object29548:head# 2024-03-20T09:04:59.314 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.314 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3571fc7:::benchmark_data_smithi122_119671_object74222:head# 2024-03-20T09:04:59.315 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.315 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3579d09:::benchmark_data_smithi122_119671_object64095:head# 2024-03-20T09:04:59.315 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.315 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b35867da:::benchmark_data_smithi122_119671_object7948:head# 2024-03-20T09:04:59.315 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.315 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3587684:::benchmark_data_smithi122_119671_object15978:head# 2024-03-20T09:04:59.315 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.315 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b35bf10c:::benchmark_data_smithi122_119671_object51867:head# 2024-03-20T09:04:59.315 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.315 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b35bf8cd:::benchmark_data_smithi122_119671_object24892:head# 2024-03-20T09:04:59.315 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.315 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b35c4b30:::benchmark_data_smithi122_119671_object1192:head# 2024-03-20T09:04:59.316 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.316 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b35c5551:::benchmark_data_smithi122_119671_object45656:head# 2024-03-20T09:04:59.316 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.316 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b35d918d:::benchmark_data_smithi122_119671_object48174:head# 2024-03-20T09:04:59.316 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.316 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b35e166a:::benchmark_data_smithi122_119671_object85712:head# 2024-03-20T09:04:59.316 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.316 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b35fd4fe:::benchmark_data_smithi122_119671_object47605:head# 2024-03-20T09:04:59.316 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.316 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b361d576:::benchmark_data_smithi122_119671_object81575:head# 2024-03-20T09:04:59.316 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.316 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3635267:::benchmark_data_smithi122_119671_object3188:head# 2024-03-20T09:04:59.316 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.317 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3637f23:::benchmark_data_smithi122_119671_object64493:head# 2024-03-20T09:04:59.317 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.317 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3641b9c:::benchmark_data_smithi122_119671_object40221:head# 2024-03-20T09:04:59.317 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.317 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b365c09d:::benchmark_data_smithi122_119671_object22634:head# 2024-03-20T09:04:59.317 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.317 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3669efb:::benchmark_data_smithi122_119671_object86017:head# 2024-03-20T09:04:59.317 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.317 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b366a264:::benchmark_data_smithi122_119671_object53764:head# 2024-03-20T09:04:59.317 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.317 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b366bd14:::benchmark_data_smithi122_119671_object12606:head# 2024-03-20T09:04:59.317 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.318 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b36807f3:::benchmark_data_smithi122_119671_object81879:head# 2024-03-20T09:04:59.318 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.318 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b368cb15:::benchmark_data_smithi122_119671_object62311:head# 2024-03-20T09:04:59.318 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.318 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3691ed1:::benchmark_data_smithi122_119671_object16019:head# 2024-03-20T09:04:59.318 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.318 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3695d48:::benchmark_data_smithi122_119671_object62698:head# 2024-03-20T09:04:59.318 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.318 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b369921b:::benchmark_data_smithi122_119671_object28747:head# 2024-03-20T09:04:59.318 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.318 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3699b1e:::benchmark_data_smithi122_119671_object51543:head# 2024-03-20T09:04:59.319 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.319 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3699e7a:::benchmark_data_smithi122_119671_object13576:head# 2024-03-20T09:04:59.319 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.319 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b369dc98:::benchmark_data_smithi122_119671_object84759:head# 2024-03-20T09:04:59.319 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.319 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b36a4562:::benchmark_data_smithi122_119671_object63095:head# 2024-03-20T09:04:59.319 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.319 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b36ac56c:::benchmark_data_smithi122_119671_object43933:head# 2024-03-20T09:04:59.319 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.319 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b36b694e:::benchmark_data_smithi122_119671_object79345:head# 2024-03-20T09:04:59.319 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.319 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b36c71bf:::benchmark_data_smithi122_119671_object19095:head# 2024-03-20T09:04:59.320 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.320 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b36d7678:::benchmark_data_smithi122_119671_object19950:head# 2024-03-20T09:04:59.320 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.320 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b36de627:::benchmark_data_smithi122_119671_object51231:head# 2024-03-20T09:04:59.320 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.320 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b36e1230:::benchmark_data_smithi122_119671_object45935:head# 2024-03-20T09:04:59.320 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.320 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b37019d4:::benchmark_data_smithi122_119671_object68046:head# 2024-03-20T09:04:59.320 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.320 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3707b5d:::benchmark_data_smithi122_119671_object81971:head# 2024-03-20T09:04:59.320 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.320 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b371a541:::benchmark_data_smithi122_119671_object24468:head# 2024-03-20T09:04:59.321 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.321 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3723285:::benchmark_data_smithi122_119671_object21989:head# 2024-03-20T09:04:59.321 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.321 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b372d2b3:::benchmark_data_smithi122_119671_object23021:head# 2024-03-20T09:04:59.321 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.321 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3743cc3:::benchmark_data_smithi122_119671_object54739:head# 2024-03-20T09:04:59.321 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.321 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3760f37:::benchmark_data_smithi122_119671_object43019:head# 2024-03-20T09:04:59.321 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.321 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3763fc6:::benchmark_data_smithi122_119671_object48320:head# 2024-03-20T09:04:59.321 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.322 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b376b862:::benchmark_data_smithi122_119671_object7002:head# 2024-03-20T09:04:59.322 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.322 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b376ed3d:::benchmark_data_smithi122_119671_object29958:head# 2024-03-20T09:04:59.322 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.322 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b377a913:::benchmark_data_smithi122_119671_object10608:head# 2024-03-20T09:04:59.322 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.322 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3797262:::benchmark_data_smithi122_119671_object80286:head# 2024-03-20T09:04:59.322 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.322 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b379fa93:::benchmark_data_smithi122_119671_object43520:head# 2024-03-20T09:04:59.322 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.322 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b37a0ce0:::benchmark_data_smithi122_119671_object9227:head# 2024-03-20T09:04:59.322 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.323 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b37bae1f:::benchmark_data_smithi122_119671_object53541:head# 2024-03-20T09:04:59.323 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.323 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b37c1b32:::benchmark_data_smithi122_119671_object15185:head# 2024-03-20T09:04:59.323 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.323 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b37d0841:::benchmark_data_smithi122_119671_object13843:head# 2024-03-20T09:04:59.323 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.323 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b37e5110:::benchmark_data_smithi122_119671_object52043:head# 2024-03-20T09:04:59.323 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.323 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b37f6ef2:::benchmark_data_smithi122_119671_object2567:head# 2024-03-20T09:04:59.323 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.323 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b37f88ed:::benchmark_data_smithi122_119671_object23899:head# 2024-03-20T09:04:59.324 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.324 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b380b81a:::benchmark_data_smithi122_119671_object89815:head# 2024-03-20T09:04:59.324 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.324 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b38300da:::benchmark_data_smithi122_119671_object45982:head# 2024-03-20T09:04:59.324 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.324 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b383205c:::benchmark_data_smithi122_119671_object21098:head# 2024-03-20T09:04:59.324 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.324 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3836132:::benchmark_data_smithi122_119671_object96937:head# 2024-03-20T09:04:59.324 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.324 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b38386f1:::benchmark_data_smithi122_119671_object15388:head# 2024-03-20T09:04:59.324 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.324 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b383bd10:::benchmark_data_smithi122_119671_object83306:head# 2024-03-20T09:04:59.325 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.325 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3846db9:::benchmark_data_smithi122_119671_object44225:head# 2024-03-20T09:04:59.325 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.325 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3848ebe:::benchmark_data_smithi122_119671_object97581:head# 2024-03-20T09:04:59.325 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.325 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b384b18e:::benchmark_data_smithi122_119671_object3566:head# 2024-03-20T09:04:59.325 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.325 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b38519e4:::benchmark_data_smithi122_119671_object97078:head# 2024-03-20T09:04:59.325 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.325 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b385ee66:::benchmark_data_smithi122_119671_object61914:head# 2024-03-20T09:04:59.325 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.326 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b387911d:::benchmark_data_smithi122_119671_object69886:head# 2024-03-20T09:04:59.326 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.326 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b388a2eb:::benchmark_data_smithi122_119671_object58741:head# 2024-03-20T09:04:59.326 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.326 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b38a1720:::benchmark_data_smithi122_119671_object1761:head# 2024-03-20T09:04:59.326 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.326 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b38a70e2:::benchmark_data_smithi122_119671_object48987:head# 2024-03-20T09:04:59.326 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.326 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b38aa894:::benchmark_data_smithi122_119671_object38900:head# 2024-03-20T09:04:59.326 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.326 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b38aa910:::benchmark_data_smithi122_119671_object62909:head# 2024-03-20T09:04:59.326 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.327 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b38ab00a:::benchmark_data_smithi122_119671_object26416:head# 2024-03-20T09:04:59.327 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.327 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b38be173:::benchmark_data_smithi122_119671_object94649:head# 2024-03-20T09:04:59.327 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.327 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b38fe79e:::benchmark_data_smithi122_119671_object81230:head# 2024-03-20T09:04:59.327 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.327 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3906b0f:::benchmark_data_smithi122_119671_object67818:head# 2024-03-20T09:04:59.327 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.327 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b391879e:::benchmark_data_smithi122_119671_object49014:head# 2024-03-20T09:04:59.327 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.327 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b392946e:::benchmark_data_smithi122_119671_object31055:head# 2024-03-20T09:04:59.327 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.328 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b39332c2:::benchmark_data_smithi122_119671_object16357:head# 2024-03-20T09:04:59.328 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.328 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b39343a7:::benchmark_data_smithi122_119671_object32986:head# 2024-03-20T09:04:59.328 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.328 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3939911:::benchmark_data_smithi122_119671_object92622:head# 2024-03-20T09:04:59.328 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.328 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3944600:::benchmark_data_smithi122_119671_object46402:head# 2024-03-20T09:04:59.328 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.328 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b39467a3:::benchmark_data_smithi122_119671_object34563:head# 2024-03-20T09:04:59.328 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.328 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3951a17:::benchmark_data_smithi122_119671_object67913:head# 2024-03-20T09:04:59.329 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.329 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b39591d1:::benchmark_data_smithi122_119671_object94459:head# 2024-03-20T09:04:59.329 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.329 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3961feb:::benchmark_data_smithi122_119671_object8794:head# 2024-03-20T09:04:59.329 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.329 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3968641:::benchmark_data_smithi122_119671_object23680:head# 2024-03-20T09:04:59.329 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.329 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b396f49f:::benchmark_data_smithi122_119671_object48600:head# 2024-03-20T09:04:59.329 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.329 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b397a6f5:::benchmark_data_smithi122_119671_object88752:head# 2024-03-20T09:04:59.329 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.329 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3983ec6:::benchmark_data_smithi122_119671_object27786:head# 2024-03-20T09:04:59.329 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.330 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b398ae9b:::benchmark_data_smithi122_119671_object89507:head# 2024-03-20T09:04:59.330 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.330 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b398f707:::benchmark_data_smithi122_119671_object39450:head# 2024-03-20T09:04:59.330 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.330 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b399026b:::benchmark_data_smithi122_119671_object48616:head# 2024-03-20T09:04:59.330 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.330 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3994c17:::benchmark_data_smithi122_119671_object62837:head# 2024-03-20T09:04:59.330 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.330 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b399c976:::benchmark_data_smithi122_119671_object54118:head# 2024-03-20T09:04:59.330 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.330 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b39af048:::benchmark_data_smithi122_119671_object60545:head# 2024-03-20T09:04:59.330 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.331 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b39bb471:::benchmark_data_smithi122_119671_object90632:head# 2024-03-20T09:04:59.331 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.331 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b39c2861:::benchmark_data_smithi122_119671_object55306:head# 2024-03-20T09:04:59.331 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.331 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b39c62d5:::benchmark_data_smithi122_119671_object36792:head# 2024-03-20T09:04:59.331 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.331 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b39cd99a:::benchmark_data_smithi122_119671_object85816:head# 2024-03-20T09:04:59.331 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.331 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b39da333:::benchmark_data_smithi122_119671_object20607:head# 2024-03-20T09:04:59.331 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.331 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b39e835e:::benchmark_data_smithi122_119671_object14153:head# 2024-03-20T09:04:59.331 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.332 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b39fa414:::benchmark_data_smithi122_119671_object25519:head# 2024-03-20T09:04:59.332 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.332 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3a0968b:::benchmark_data_smithi122_119671_object80027:head# 2024-03-20T09:04:59.332 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.332 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3a0d66c:::benchmark_data_smithi122_119671_object83270:head# 2024-03-20T09:04:59.332 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.332 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3a23e79:::benchmark_data_smithi122_119671_object30937:head# 2024-03-20T09:04:59.332 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.332 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3a24c26:::benchmark_data_smithi122_119671_object58549:head# 2024-03-20T09:04:59.332 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.332 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3a2c8e1:::benchmark_data_smithi122_119671_object65259:head# 2024-03-20T09:04:59.332 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.332 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3a4c009:::benchmark_data_smithi122_119671_object85246:head# 2024-03-20T09:04:59.333 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.333 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3a4da28:::benchmark_data_smithi122_119671_object85288:head# 2024-03-20T09:04:59.333 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.333 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3a602c7:::benchmark_data_smithi122_119671_object41168:head# 2024-03-20T09:04:59.333 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.333 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3a7021a:::benchmark_data_smithi122_119671_object28547:head# 2024-03-20T09:04:59.333 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.333 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3a7707e:::benchmark_data_smithi122_119671_object83271:head# 2024-03-20T09:04:59.333 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.333 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3a89b80:::benchmark_data_smithi122_119671_object85108:head# 2024-03-20T09:04:59.333 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.333 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3a9f84a:::benchmark_data_smithi122_119671_object47989:head# 2024-03-20T09:04:59.334 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.334 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3aa9fb0:::benchmark_data_smithi122_119671_object53628:head# 2024-03-20T09:04:59.334 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.334 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3ab7d18:::benchmark_data_smithi122_119671_object48342:head# 2024-03-20T09:04:59.334 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.334 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3ac4bb3:::benchmark_data_smithi122_119671_object26312:head# 2024-03-20T09:04:59.334 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.334 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3ad683e:::benchmark_data_smithi122_119671_object15213:head# 2024-03-20T09:04:59.334 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.334 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3ada778:::benchmark_data_smithi122_119671_object88226:head# 2024-03-20T09:04:59.334 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.334 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3adbdb7:::benchmark_data_smithi122_119671_object19912:head# 2024-03-20T09:04:59.335 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.335 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3ae2586:::benchmark_data_smithi122_119671_object96688:head# 2024-03-20T09:04:59.335 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.335 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3aea23d:::benchmark_data_smithi122_119671_object63131:head# 2024-03-20T09:04:59.335 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.335 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3aea75e:::benchmark_data_smithi122_119671_object38449:head# 2024-03-20T09:04:59.335 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.335 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3afb2f2:::benchmark_data_smithi122_119671_object75063:head# 2024-03-20T09:04:59.335 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.335 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3afc7e9:::benchmark_data_smithi122_119671_object50009:head# 2024-03-20T09:04:59.335 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.336 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3b1b46a:::benchmark_data_smithi122_119671_object58373:head# 2024-03-20T09:04:59.336 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.336 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3b247ae:::benchmark_data_smithi122_119671_object3871:head# 2024-03-20T09:04:59.336 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.336 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3b447f3:::benchmark_data_smithi122_119671_object78128:head# 2024-03-20T09:04:59.336 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.336 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3b52213:::benchmark_data_smithi122_119671_object73829:head# 2024-03-20T09:04:59.336 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.336 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3b524c2:::benchmark_data_smithi122_119671_object20871:head# 2024-03-20T09:04:59.336 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.336 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3b698c4:::benchmark_data_smithi122_119671_object75325:head# 2024-03-20T09:04:59.336 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.336 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3b706f4:::benchmark_data_smithi122_119671_object54080:head# 2024-03-20T09:04:59.337 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.337 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3b7465f:::benchmark_data_smithi122_119671_object6035:head# 2024-03-20T09:04:59.337 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.337 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3b7500c:::benchmark_data_smithi122_119671_object39744:head# 2024-03-20T09:04:59.337 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.337 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3b88660:::benchmark_data_smithi122_119671_object2132:head# 2024-03-20T09:04:59.337 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.337 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3ba5609:::benchmark_data_smithi122_119671_object88404:head# 2024-03-20T09:04:59.337 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.337 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3bacfc5:::benchmark_data_smithi122_119671_object62419:head# 2024-03-20T09:04:59.337 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.337 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3bb845b:::benchmark_data_smithi122_119671_object91096:head# 2024-03-20T09:04:59.338 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.338 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3bbb9d8:::benchmark_data_smithi122_119671_object48564:head# 2024-03-20T09:04:59.338 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.338 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3bc0e6a:::benchmark_data_smithi122_119671_object49034:head# 2024-03-20T09:04:59.338 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.338 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3bc13af:::benchmark_data_smithi122_119671_object91850:head# 2024-03-20T09:04:59.338 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.338 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3bef708:::benchmark_data_smithi122_119671_object57310:head# 2024-03-20T09:04:59.338 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.338 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3bef84a:::benchmark_data_smithi122_119671_object37086:head# 2024-03-20T09:04:59.338 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.339 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3bff6f0:::benchmark_data_smithi122_119671_object62682:head# 2024-03-20T09:04:59.339 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.339 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3bffbd8:::benchmark_data_smithi122_119671_object73348:head# 2024-03-20T09:04:59.339 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.339 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3c0940a:::benchmark_data_smithi122_119671_object44715:head# 2024-03-20T09:04:59.339 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.339 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3c0e936:::benchmark_data_smithi122_119671_object2753:head# 2024-03-20T09:04:59.339 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.339 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3c25ca3:::benchmark_data_smithi122_119671_object61863:head# 2024-03-20T09:04:59.339 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.339 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3c31559:::benchmark_data_smithi122_119671_object28606:head# 2024-03-20T09:04:59.340 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.340 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3c34c3a:::benchmark_data_smithi122_119671_object42972:head# 2024-03-20T09:04:59.340 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.340 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3c385f5:::benchmark_data_smithi122_119671_object63661:head# 2024-03-20T09:04:59.340 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.340 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3c3e28d:::benchmark_data_smithi122_119671_object20780:head# 2024-03-20T09:04:59.340 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.340 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3c47ffe:::benchmark_data_smithi122_119671_object9008:head# 2024-03-20T09:04:59.340 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.340 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3c593c8:::benchmark_data_smithi122_119671_object47520:head# 2024-03-20T09:04:59.340 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.340 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3c7e2b4:::benchmark_data_smithi122_119671_object11120:head# 2024-03-20T09:04:59.341 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.341 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3c95104:::benchmark_data_smithi122_119671_object64230:head# 2024-03-20T09:04:59.341 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.341 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3c98c09:::benchmark_data_smithi122_119671_object47230:head# 2024-03-20T09:04:59.341 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.341 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3c9ea73:::benchmark_data_smithi122_119671_object33613:head# 2024-03-20T09:04:59.341 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.341 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3ca3ca6:::benchmark_data_smithi122_119671_object68122:head# 2024-03-20T09:04:59.341 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.341 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3caa26f:::benchmark_data_smithi122_119671_object22973:head# 2024-03-20T09:04:59.341 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.341 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3cc0362:::benchmark_data_smithi122_119671_object6337:head# 2024-03-20T09:04:59.342 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.342 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3cc80f8:::benchmark_data_smithi122_119671_object92780:head# 2024-03-20T09:04:59.342 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.342 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3cd3cf6:::benchmark_data_smithi122_119671_object95842:head# 2024-03-20T09:04:59.342 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.342 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3cd4bc4:::benchmark_data_smithi122_119671_object64976:head# 2024-03-20T09:04:59.342 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.342 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3cd7a31:::benchmark_data_smithi122_119671_object62623:head# 2024-03-20T09:04:59.342 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.342 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3cdecb4:::benchmark_data_smithi122_119671_object26001:head# 2024-03-20T09:04:59.342 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.342 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3cf029d:::benchmark_data_smithi122_119671_object89288:head# 2024-03-20T09:04:59.343 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.343 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3cf4b68:::benchmark_data_smithi122_119671_object52550:head# 2024-03-20T09:04:59.343 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.343 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3cf4ff1:::benchmark_data_smithi122_119671_object11151:head# 2024-03-20T09:04:59.343 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.343 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3cf7fa6:::benchmark_data_smithi122_119671_object14800:head# 2024-03-20T09:04:59.343 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.343 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3cfb777:::benchmark_data_smithi122_119671_object16668:head# 2024-03-20T09:04:59.343 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.343 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3d07ed4:::benchmark_data_smithi122_119671_object46155:head# 2024-03-20T09:04:59.343 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.343 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3d28091:::benchmark_data_smithi122_119671_object80140:head# 2024-03-20T09:04:59.344 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.344 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3d2cec1:::benchmark_data_smithi122_119671_object92001:head# 2024-03-20T09:04:59.344 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.344 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3d2f16e:::benchmark_data_smithi122_119671_object45617:head# 2024-03-20T09:04:59.344 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.344 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3d336e3:::benchmark_data_smithi122_119671_object44397:head# 2024-03-20T09:04:59.344 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.344 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3d49508:::benchmark_data_smithi122_119671_object52103:head# 2024-03-20T09:04:59.344 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.344 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3d55be2:::benchmark_data_smithi122_119671_object5994:head# 2024-03-20T09:04:59.344 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.344 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3d6006d:::benchmark_data_smithi122_119671_object28108:head# 2024-03-20T09:04:59.345 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.345 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3d71de4:::benchmark_data_smithi122_119671_object2542:head# 2024-03-20T09:04:59.345 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.345 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3d79450:::benchmark_data_smithi122_119671_object26350:head# 2024-03-20T09:04:59.345 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.345 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3d7e9b7:::benchmark_data_smithi122_119671_object6942:head# 2024-03-20T09:04:59.345 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.345 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3d85384:::benchmark_data_smithi122_119671_object22270:head# 2024-03-20T09:04:59.345 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.345 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3d85554:::benchmark_data_smithi122_119671_object53864:head# 2024-03-20T09:04:59.345 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.346 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3d8f57a:::benchmark_data_smithi122_119671_object95913:head# 2024-03-20T09:04:59.346 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.346 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3d980b9:::benchmark_data_smithi122_119671_object9678:head# 2024-03-20T09:04:59.346 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.346 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3d986f3:::benchmark_data_smithi122_119671_object45927:head# 2024-03-20T09:04:59.346 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.346 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3da0cd8:::benchmark_data_smithi122_119671_object89656:head# 2024-03-20T09:04:59.346 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.346 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3da91a0:::benchmark_data_smithi122_119671_object71895:head# 2024-03-20T09:04:59.346 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.346 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3da9a8e:::benchmark_data_smithi122_119671_object94928:head# 2024-03-20T09:04:59.347 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.347 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3dad57c:::benchmark_data_smithi122_119671_object27035:head# 2024-03-20T09:04:59.347 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.347 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3dc2195:::benchmark_data_smithi122_119671_object28891:head# 2024-03-20T09:04:59.347 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.347 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3dd3780:::benchmark_data_smithi122_119671_object58714:head# 2024-03-20T09:04:59.347 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.347 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3dd7077:::benchmark_data_smithi122_119671_object29253:head# 2024-03-20T09:04:59.347 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.347 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3dd7cc1:::benchmark_data_smithi122_119671_object81226:head# 2024-03-20T09:04:59.347 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.347 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3e082df:::benchmark_data_smithi122_119671_object81138:head# 2024-03-20T09:04:59.348 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.348 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3e173f7:::benchmark_data_smithi122_119671_object72222:head# 2024-03-20T09:04:59.348 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.348 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3e20a2c:::benchmark_data_smithi122_119671_object87676:head# 2024-03-20T09:04:59.348 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.348 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3e26746:::benchmark_data_smithi122_119671_object78924:head# 2024-03-20T09:04:59.348 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.348 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3e26b00:::benchmark_data_smithi122_119671_object93398:head# 2024-03-20T09:04:59.348 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.348 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3e2fe9a:::benchmark_data_smithi122_119671_object3773:head# 2024-03-20T09:04:59.348 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.348 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3e4939e:::benchmark_data_smithi122_119671_object3062:head# 2024-03-20T09:04:59.348 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.349 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3e5275a:::benchmark_data_smithi122_119671_object29359:head# 2024-03-20T09:04:59.349 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.349 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3e5fadc:::benchmark_data_smithi122_119671_object65203:head# 2024-03-20T09:04:59.349 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.349 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3e622ee:::benchmark_data_smithi122_119671_object54028:head# 2024-03-20T09:04:59.349 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.349 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3e7b9cd:::benchmark_data_smithi122_119671_object71467:head# 2024-03-20T09:04:59.349 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.349 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3e7cb28:::benchmark_data_smithi122_119671_object99532:head# 2024-03-20T09:04:59.349 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.349 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3e81e5d:::benchmark_data_smithi122_119671_object76207:head# 2024-03-20T09:04:59.350 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.350 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3e8277f:::benchmark_data_smithi122_119671_object36106:head# 2024-03-20T09:04:59.350 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.350 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3ebb87c:::benchmark_data_smithi122_119671_object72412:head# 2024-03-20T09:04:59.350 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.350 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3ebea25:::benchmark_data_smithi122_119671_object80742:head# 2024-03-20T09:04:59.350 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.350 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3ec1e85:::benchmark_data_smithi122_119671_object96548:head# 2024-03-20T09:04:59.350 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.350 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3ec897a:::benchmark_data_smithi122_119671_object98258:head# 2024-03-20T09:04:59.350 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.350 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3ec9a95:::benchmark_data_smithi122_119671_object49515:head# 2024-03-20T09:04:59.351 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.351 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3ed289e:::benchmark_data_smithi122_119671_object17670:head# 2024-03-20T09:04:59.351 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.351 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3ed61fe:::benchmark_data_smithi122_119671_object37825:head# 2024-03-20T09:04:59.351 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.351 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3ed6eb0:::benchmark_data_smithi122_119671_object28076:head# 2024-03-20T09:04:59.351 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.351 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3edac4c:::benchmark_data_smithi122_119671_object99697:head# 2024-03-20T09:04:59.351 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.351 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3ef2adc:::benchmark_data_smithi122_119671_object65303:head# 2024-03-20T09:04:59.352 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.352 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3efc376:::benchmark_data_smithi122_119671_object63248:head# 2024-03-20T09:04:59.352 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.352 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3f02d1f:::benchmark_data_smithi122_119671_object69003:head# 2024-03-20T09:04:59.352 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.352 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3f12a1e:::benchmark_data_smithi122_119671_object64918:head# 2024-03-20T09:04:59.352 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.352 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3f15a48:::benchmark_data_smithi122_119671_object37588:head# 2024-03-20T09:04:59.352 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.352 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3f1da78:::benchmark_data_smithi122_119671_object20542:head# 2024-03-20T09:04:59.352 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.352 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3f23797:::benchmark_data_smithi122_119671_object96541:head# 2024-03-20T09:04:59.353 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.353 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3f2d2df:::benchmark_data_smithi122_119671_object16355:head# 2024-03-20T09:04:59.353 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.353 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3f3081e:::benchmark_data_smithi122_119671_object28042:head# 2024-03-20T09:04:59.353 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.353 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3f47994:::benchmark_data_smithi122_119671_object68947:head# 2024-03-20T09:04:59.353 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.353 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3f55c4f:::benchmark_data_smithi122_119671_object49729:head# 2024-03-20T09:04:59.353 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.353 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3f5e670:::benchmark_data_smithi122_119671_object15146:head# 2024-03-20T09:04:59.353 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.353 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3f79625:::benchmark_data_smithi122_119671_object47472:head# 2024-03-20T09:04:59.354 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.354 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3f79de4:::benchmark_data_smithi122_119671_object21407:head# 2024-03-20T09:04:59.354 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.354 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3f81201:::benchmark_data_smithi122_119671_object65547:head# 2024-03-20T09:04:59.354 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.354 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3f89ef2:::benchmark_data_smithi122_119671_object25037:head# 2024-03-20T09:04:59.354 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.354 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3f8da35:::benchmark_data_smithi122_119671_object41889:head# 2024-03-20T09:04:59.354 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.354 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3f8db82:::benchmark_data_smithi122_119671_object69518:head# 2024-03-20T09:04:59.354 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.355 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3fac0db:::benchmark_data_smithi122_119671_object96885:head# 2024-03-20T09:04:59.355 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.355 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3fb3df7:::benchmark_data_smithi122_119671_object77505:head# 2024-03-20T09:04:59.355 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.355 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3fb6e4a:::benchmark_data_smithi122_119671_object59729:head# 2024-03-20T09:04:59.355 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.355 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b3ffb5cd:::benchmark_data_smithi122_119671_object99465:head# 2024-03-20T09:04:59.355 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.355 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b401d109:::benchmark_data_smithi122_119671_object26507:head# 2024-03-20T09:04:59.355 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.355 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b401fce0:::benchmark_data_smithi122_119671_object73184:head# 2024-03-20T09:04:59.355 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.356 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b40201c1:::benchmark_data_smithi122_119671_object79218:head# 2024-03-20T09:04:59.356 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.356 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b402d068:::benchmark_data_smithi122_119671_object95865:head# 2024-03-20T09:04:59.356 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.356 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4031faf:::benchmark_data_smithi122_119671_object90494:head# 2024-03-20T09:04:59.356 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.356 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4036c65:::benchmark_data_smithi122_119671_object66777:head# 2024-03-20T09:04:59.356 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.356 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b403dbb7:::benchmark_data_smithi122_119671_object34094:head# 2024-03-20T09:04:59.356 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.356 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4049906:::benchmark_data_smithi122_119671_object80467:head# 2024-03-20T09:04:59.357 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.357 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4051be6:::benchmark_data_smithi122_119671_object53587:head# 2024-03-20T09:04:59.357 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.357 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b405d78b:::benchmark_data_smithi122_119671_object21849:head# 2024-03-20T09:04:59.357 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.357 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4070bc3:::benchmark_data_smithi122_119671_object16992:head# 2024-03-20T09:04:59.357 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.357 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b407697f:::benchmark_data_smithi122_119671_object62677:head# 2024-03-20T09:04:59.357 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.357 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b407c156:::benchmark_data_smithi122_119671_object67372:head# 2024-03-20T09:04:59.357 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.357 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4082939:::benchmark_data_smithi122_119671_object88649:head# 2024-03-20T09:04:59.358 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.358 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b40837ae:::benchmark_data_smithi122_119671_object91137:head# 2024-03-20T09:04:59.358 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.358 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b40a9a95:::benchmark_data_smithi122_119671_object16497:head# 2024-03-20T09:04:59.358 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.358 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b40ac00d:::benchmark_data_smithi122_119671_object43632:head# 2024-03-20T09:04:59.358 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.358 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b40b8eeb:::benchmark_data_smithi122_119671_object50396:head# 2024-03-20T09:04:59.358 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.358 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b40ce6fc:::benchmark_data_smithi122_119671_object88371:head# 2024-03-20T09:04:59.359 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.359 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b40d5a4e:::benchmark_data_smithi122_119671_object24220:head# 2024-03-20T09:04:59.359 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.359 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b40d6de3:::benchmark_data_smithi122_119671_object95040:head# 2024-03-20T09:04:59.359 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.359 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b40d862a:::benchmark_data_smithi122_119671_object29264:head# 2024-03-20T09:04:59.359 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.359 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b40d95f8:::benchmark_data_smithi122_119671_object82174:head# 2024-03-20T09:04:59.359 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.359 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b40e103b:::benchmark_data_smithi122_119671_object13693:head# 2024-03-20T09:04:59.359 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.359 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b40f5fbb:::benchmark_data_smithi122_119671_object65256:head# 2024-03-20T09:04:59.360 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.360 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b40f9981:::benchmark_data_smithi122_119671_object83718:head# 2024-03-20T09:04:59.360 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.360 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b412e3c8:::benchmark_data_smithi122_119671_object44320:head# 2024-03-20T09:04:59.360 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.360 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b41343ed:::benchmark_data_smithi122_119671_object30939:head# 2024-03-20T09:04:59.360 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.360 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b414562d:::benchmark_data_smithi122_119671_object71343:head# 2024-03-20T09:04:59.360 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.360 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b41468f2:::benchmark_data_smithi122_119671_object45096:head# 2024-03-20T09:04:59.360 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.361 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4149fe2:::benchmark_data_smithi122_119671_object44329:head# 2024-03-20T09:04:59.361 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.361 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4152b25:::benchmark_data_smithi122_119671_object3664:head# 2024-03-20T09:04:59.361 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.361 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4154c0b:::benchmark_data_smithi122_119671_object96629:head# 2024-03-20T09:04:59.361 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.361 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4156a4a:::benchmark_data_smithi122_119671_object25288:head# 2024-03-20T09:04:59.361 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.361 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b41656eb:::benchmark_data_smithi122_119671_object50296:head# 2024-03-20T09:04:59.361 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.361 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4167e8f:::benchmark_data_smithi122_119671_object37600:head# 2024-03-20T09:04:59.361 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.362 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b416cf9e:::benchmark_data_smithi122_119671_object59856:head# 2024-03-20T09:04:59.362 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.362 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4173e9c:::benchmark_data_smithi122_119671_object44052:head# 2024-03-20T09:04:59.362 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.362 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4190e9a:::benchmark_data_smithi122_119671_object3975:head# 2024-03-20T09:04:59.362 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.362 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4193b29:::benchmark_data_smithi122_119671_object12377:head# 2024-03-20T09:04:59.362 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.362 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b419da45:::benchmark_data_smithi122_119671_object75348:head# 2024-03-20T09:04:59.362 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.362 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b41ab3e7:::benchmark_data_smithi122_119671_object37501:head# 2024-03-20T09:04:59.363 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.363 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b41ae040:::benchmark_data_smithi122_119671_object8223:head# 2024-03-20T09:04:59.363 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.363 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b41b24e9:::benchmark_data_smithi122_119671_object7453:head# 2024-03-20T09:04:59.363 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.363 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b41bdd14:::benchmark_data_smithi122_119671_object23037:head# 2024-03-20T09:04:59.363 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.363 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b41c7602:::benchmark_data_smithi122_119671_object51406:head# 2024-03-20T09:04:59.363 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.363 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b41c8024:::benchmark_data_smithi122_119671_object78489:head# 2024-03-20T09:04:59.364 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.364 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b41c8e79:::benchmark_data_smithi122_119671_object2482:head# 2024-03-20T09:04:59.364 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.364 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b41d14f0:::benchmark_data_smithi122_119671_object90580:head# 2024-03-20T09:04:59.364 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.364 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b41d9ccf:::benchmark_data_smithi122_119671_object87651:head# 2024-03-20T09:04:59.364 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.364 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b41dec81:::benchmark_data_smithi122_119671_object30897:head# 2024-03-20T09:04:59.364 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.364 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b41e5224:::benchmark_data_smithi122_119671_object57584:head# 2024-03-20T09:04:59.364 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.364 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b42027fe:::benchmark_data_smithi122_119671_object91023:head# 2024-03-20T09:04:59.365 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.365 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4207554:::benchmark_data_smithi122_119671_object47330:head# 2024-03-20T09:04:59.365 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.365 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4207b7a:::benchmark_data_smithi122_119671_object53673:head# 2024-03-20T09:04:59.365 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.365 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b420acea:::benchmark_data_smithi122_119671_object13538:head# 2024-03-20T09:04:59.365 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.365 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b420de57:::benchmark_data_smithi122_119671_object37316:head# 2024-03-20T09:04:59.365 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.365 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b420ff2e:::benchmark_data_smithi122_119671_object21343:head# 2024-03-20T09:04:59.365 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.365 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b421fa7e:::benchmark_data_smithi122_119671_object84796:head# 2024-03-20T09:04:59.366 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.366 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4220d5b:::benchmark_data_smithi122_119671_object44389:head# 2024-03-20T09:04:59.366 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.366 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4239fb8:::benchmark_data_smithi122_119671_object5161:head# 2024-03-20T09:04:59.366 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.366 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4243277:::benchmark_data_smithi122_119671_object59530:head# 2024-03-20T09:04:59.366 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.366 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4245c93:::benchmark_data_smithi122_119671_object50601:head# 2024-03-20T09:04:59.366 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.366 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b424c90f:::benchmark_data_smithi122_119671_object3020:head# 2024-03-20T09:04:59.366 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.367 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b424f215:::benchmark_data_smithi122_119671_object39511:head# 2024-03-20T09:04:59.367 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.367 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4262e50:::benchmark_data_smithi122_119671_object14513:head# 2024-03-20T09:04:59.367 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.367 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4276239:::benchmark_data_smithi122_119671_object17416:head# 2024-03-20T09:04:59.367 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.367 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4277874:::benchmark_data_smithi122_119671_object89419:head# 2024-03-20T09:04:59.367 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.367 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4278fbd:::benchmark_data_smithi122_119671_object65523:head# 2024-03-20T09:04:59.368 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.368 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4285629:::benchmark_data_smithi122_119671_object54898:head# 2024-03-20T09:04:59.368 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.368 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b428590e:::benchmark_data_smithi122_119671_object51589:head# 2024-03-20T09:04:59.368 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.368 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b42884fa:::benchmark_data_smithi122_119671_object7023:head# 2024-03-20T09:04:59.368 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.369 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4288b53:::benchmark_data_smithi122_119671_object5628:head# 2024-03-20T09:04:59.369 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.369 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b42991a5:::benchmark_data_smithi122_119671_object38935:head# 2024-03-20T09:04:59.369 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.369 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b42ad6a5:::benchmark_data_smithi122_119671_object62684:head# 2024-03-20T09:04:59.369 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.369 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b42c3a37:::benchmark_data_smithi122_119671_object19944:head# 2024-03-20T09:04:59.369 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.369 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b42c934a:::benchmark_data_smithi122_119671_object36175:head# 2024-03-20T09:04:59.369 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.370 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b42d4a8e:::benchmark_data_smithi122_119671_object67334:head# 2024-03-20T09:04:59.370 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.370 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b42d549b:::benchmark_data_smithi122_119671_object44224:head# 2024-03-20T09:04:59.370 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.370 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b42dc4a8:::benchmark_data_smithi122_119671_object93608:head# 2024-03-20T09:04:59.370 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.370 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b42dec4f:::benchmark_data_smithi122_119671_object54768:head# 2024-03-20T09:04:59.370 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.370 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b42e9089:::benchmark_data_smithi122_119671_object94249:head# 2024-03-20T09:04:59.370 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.370 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b42ef5e9:::benchmark_data_smithi122_119671_object38739:head# 2024-03-20T09:04:59.371 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.371 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b42f7761:::benchmark_data_smithi122_119671_object50492:head# 2024-03-20T09:04:59.371 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.371 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b431552d:::benchmark_data_smithi122_119671_object70844:head# 2024-03-20T09:04:59.371 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.371 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4332ad4:::benchmark_data_smithi122_119671_object84719:head# 2024-03-20T09:04:59.371 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.371 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b433ab74:::benchmark_data_smithi122_119671_object22356:head# 2024-03-20T09:04:59.371 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.371 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b43527b5:::benchmark_data_smithi122_119671_object87269:head# 2024-03-20T09:04:59.371 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.373 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b435994a:::benchmark_data_smithi122_119671_object1826:head# 2024-03-20T09:04:59.373 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.374 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b435b41a:::benchmark_data_smithi122_119671_object15511:head# 2024-03-20T09:04:59.374 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.374 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b435f9d5:::benchmark_data_smithi122_119671_object47327:head# 2024-03-20T09:04:59.374 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.374 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4365716:::benchmark_data_smithi122_119671_object61176:head# 2024-03-20T09:04:59.374 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.374 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4378b67:::benchmark_data_smithi122_119671_object40376:head# 2024-03-20T09:04:59.374 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.374 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4379b90:::benchmark_data_smithi122_119671_object43508:head# 2024-03-20T09:04:59.374 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.374 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b43819ae:::benchmark_data_smithi122_119671_object16308:head# 2024-03-20T09:04:59.375 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.375 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b43a6f3c:::benchmark_data_smithi122_119671_object41036:head# 2024-03-20T09:04:59.375 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.375 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b43b0348:::benchmark_data_smithi122_119671_object40977:head# 2024-03-20T09:04:59.375 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.375 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b43c1d25:::benchmark_data_smithi122_119671_object80300:head# 2024-03-20T09:04:59.375 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.375 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b43c6a11:::benchmark_data_smithi122_119671_object49331:head# 2024-03-20T09:04:59.375 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.375 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b43d3555:::benchmark_data_smithi122_119671_object71930:head# 2024-03-20T09:04:59.375 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.375 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b43d60bc:::benchmark_data_smithi122_119671_object95467:head# 2024-03-20T09:04:59.376 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.376 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b43d9b2a:::benchmark_data_smithi122_119671_object80724:head# 2024-03-20T09:04:59.376 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.376 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b43e73f0:::benchmark_data_smithi122_119671_object17553:head# 2024-03-20T09:04:59.376 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.376 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b43ff5b7:::benchmark_data_smithi122_119671_object61887:head# 2024-03-20T09:04:59.376 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.376 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4406f06:::benchmark_data_smithi122_119671_object16533:head# 2024-03-20T09:04:59.376 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.376 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b440b893:::benchmark_data_smithi122_119671_object51703:head# 2024-03-20T09:04:59.376 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.376 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b441538f:::benchmark_data_smithi122_119671_object89812:head# 2024-03-20T09:04:59.377 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.377 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b44175a0:::benchmark_data_smithi122_119671_object42772:head# 2024-03-20T09:04:59.377 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.377 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b441ee3c:::benchmark_data_smithi122_119671_object34635:head# 2024-03-20T09:04:59.377 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.377 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4420aeb:::benchmark_data_smithi122_119671_object61415:head# 2024-03-20T09:04:59.377 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.377 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b44238b2:::benchmark_data_smithi122_119671_object61061:head# 2024-03-20T09:04:59.377 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.377 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b442522c:::benchmark_data_smithi122_119671_object18785:head# 2024-03-20T09:04:59.378 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.378 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b442b09f:::benchmark_data_smithi122_119671_object27300:head# 2024-03-20T09:04:59.378 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.378 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b442fc19:::benchmark_data_smithi122_119671_object41018:head# 2024-03-20T09:04:59.378 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.378 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4431fb8:::benchmark_data_smithi122_119671_object19547:head# 2024-03-20T09:04:59.378 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.378 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b443a5b2:::benchmark_data_smithi122_119671_object99544:head# 2024-03-20T09:04:59.378 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.378 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b443cd88:::benchmark_data_smithi122_119671_object72500:head# 2024-03-20T09:04:59.378 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.378 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b444b8e1:::benchmark_data_smithi122_119671_object52691:head# 2024-03-20T09:04:59.379 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.379 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b446dc72:::benchmark_data_smithi122_119671_object37308:head# 2024-03-20T09:04:59.379 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.379 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4470ba9:::benchmark_data_smithi122_119671_object6667:head# 2024-03-20T09:04:59.379 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.379 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4477259:::benchmark_data_smithi122_119671_object66985:head# 2024-03-20T09:04:59.379 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.379 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b447b44b:::benchmark_data_smithi122_119671_object14865:head# 2024-03-20T09:04:59.379 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.379 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b44878a0:::benchmark_data_smithi122_119671_object69505:head# 2024-03-20T09:04:59.379 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.380 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b448df36:::benchmark_data_smithi122_119671_object65607:head# 2024-03-20T09:04:59.380 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.380 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b449feb5:::benchmark_data_smithi122_119671_object72439:head# 2024-03-20T09:04:59.380 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.380 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b44ad1de:::benchmark_data_smithi122_119671_object33992:head# 2024-03-20T09:04:59.380 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.380 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b44af0f0:::benchmark_data_smithi122_119671_object18018:head# 2024-03-20T09:04:59.380 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.380 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b44ca358:::benchmark_data_smithi122_119671_object30670:head# 2024-03-20T09:04:59.380 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.380 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b44cd3a6:::benchmark_data_smithi122_119671_object67876:head# 2024-03-20T09:04:59.380 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.381 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b44d276d:::benchmark_data_smithi122_119671_object5075:head# 2024-03-20T09:04:59.381 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.381 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b44e5b18:::benchmark_data_smithi122_119671_object66748:head# 2024-03-20T09:04:59.381 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.381 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b44ef330:::benchmark_data_smithi122_119671_object5991:head# 2024-03-20T09:04:59.381 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.381 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b44f01a5:::benchmark_data_smithi122_119671_object13052:head# 2024-03-20T09:04:59.381 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.381 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4501bd6:::benchmark_data_smithi122_119671_object34902:head# 2024-03-20T09:04:59.381 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.381 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4516b0f:::benchmark_data_smithi122_119671_object49561:head# 2024-03-20T09:04:59.382 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.382 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4521092:::benchmark_data_smithi122_119671_object11474:head# 2024-03-20T09:04:59.382 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.382 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b45391dd:::benchmark_data_smithi122_119671_object30269:head# 2024-03-20T09:04:59.382 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.382 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b45398c2:::benchmark_data_smithi122_119671_object89282:head# 2024-03-20T09:04:59.382 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.382 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4548bbc:::benchmark_data_smithi122_119671_object95460:head# 2024-03-20T09:04:59.382 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.382 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b455dcff:::benchmark_data_smithi122_119671_object63211:head# 2024-03-20T09:04:59.382 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.383 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b455ddab:::benchmark_data_smithi122_119671_object87011:head# 2024-03-20T09:04:59.383 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.383 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b455de31:::benchmark_data_smithi122_119671_object90560:head# 2024-03-20T09:04:59.383 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.383 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b456679a:::benchmark_data_smithi122_119671_object72169:head# 2024-03-20T09:04:59.383 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.383 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b456ebed:::benchmark_data_smithi122_119671_object74857:head# 2024-03-20T09:04:59.383 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.383 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b45a4253:::benchmark_data_smithi122_119671_object3060:head# 2024-03-20T09:04:59.383 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.383 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b45ab0e1:::benchmark_data_smithi122_119671_object73476:head# 2024-03-20T09:04:59.384 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.384 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b45ae483:::benchmark_data_smithi122_119671_object49348:head# 2024-03-20T09:04:59.384 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.384 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b45b3685:::benchmark_data_smithi122_119671_object10176:head# 2024-03-20T09:04:59.384 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.384 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b45ba32f:::benchmark_data_smithi122_119671_object47582:head# 2024-03-20T09:04:59.384 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.384 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b45ca44c:::benchmark_data_smithi122_119671_object46067:head# 2024-03-20T09:04:59.384 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.384 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b45cbd12:::benchmark_data_smithi122_119671_object58328:head# 2024-03-20T09:04:59.384 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.385 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b45ea323:::benchmark_data_smithi122_119671_object96448:head# 2024-03-20T09:04:59.385 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.385 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b46088af:::benchmark_data_smithi122_119671_object64494:head# 2024-03-20T09:04:59.385 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.385 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4609dfe:::benchmark_data_smithi122_119671_object49647:head# 2024-03-20T09:04:59.385 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.385 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b460df17:::benchmark_data_smithi122_119671_object49920:head# 2024-03-20T09:04:59.385 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.385 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b46156bb:::benchmark_data_smithi122_119671_object9789:head# 2024-03-20T09:04:59.385 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.385 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b46221c4:::benchmark_data_smithi122_119671_object11831:head# 2024-03-20T09:04:59.385 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.386 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4623519:::benchmark_data_smithi122_119671_object76258:head# 2024-03-20T09:04:59.386 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.386 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4635663:::benchmark_data_smithi122_119671_object97305:head# 2024-03-20T09:04:59.386 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.386 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b463db4e:::benchmark_data_smithi122_119671_object15505:head# 2024-03-20T09:04:59.386 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.386 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b464bb9f:::benchmark_data_smithi122_119671_object87138:head# 2024-03-20T09:04:59.386 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.386 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4672d0d:::benchmark_data_smithi122_119671_object39728:head# 2024-03-20T09:04:59.386 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.386 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b467b6e5:::benchmark_data_smithi122_119671_object93187:head# 2024-03-20T09:04:59.387 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.387 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b46848e3:::benchmark_data_smithi122_119671_object50732:head# 2024-03-20T09:04:59.387 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.387 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b46864bf:::benchmark_data_smithi122_119671_object33820:head# 2024-03-20T09:04:59.387 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.387 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b46872f6:::benchmark_data_smithi122_119671_object28868:head# 2024-03-20T09:04:59.387 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.387 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b469641e:::benchmark_data_smithi122_119671_object68863:head# 2024-03-20T09:04:59.387 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.387 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b46a773e:::benchmark_data_smithi122_119671_object83656:head# 2024-03-20T09:04:59.388 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.388 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b46aafb4:::benchmark_data_smithi122_119671_object99165:head# 2024-03-20T09:04:59.388 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.388 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b46ac119:::benchmark_data_smithi122_119671_object33356:head# 2024-03-20T09:04:59.388 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.388 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b46bdc9b:::benchmark_data_smithi122_119671_object11845:head# 2024-03-20T09:04:59.388 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.388 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b46c868b:::benchmark_data_smithi122_119671_object30687:head# 2024-03-20T09:04:59.388 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.388 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b46cc16e:::benchmark_data_smithi122_119671_object70861:head# 2024-03-20T09:04:59.388 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.389 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b46e0ac6:::benchmark_data_smithi122_119671_object36730:head# 2024-03-20T09:04:59.389 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.389 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b46e94ab:::benchmark_data_smithi122_119671_object69567:head# 2024-03-20T09:04:59.389 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.389 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b46eba48:::benchmark_data_smithi122_119671_object17980:head# 2024-03-20T09:04:59.389 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.389 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b46f7377:::benchmark_data_smithi122_119671_object64330:head# 2024-03-20T09:04:59.389 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.389 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4707ede:::benchmark_data_smithi122_119671_object22927:head# 2024-03-20T09:04:59.389 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.389 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b471d044:::benchmark_data_smithi122_119671_object12193:head# 2024-03-20T09:04:59.389 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.390 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b472b73a:::benchmark_data_smithi122_119671_object57404:head# 2024-03-20T09:04:59.390 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.390 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b472e9e7:::benchmark_data_smithi122_119671_object73657:head# 2024-03-20T09:04:59.390 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.390 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4743755:::benchmark_data_smithi122_119671_object97729:head# 2024-03-20T09:04:59.390 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.390 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4749192:::benchmark_data_smithi122_119671_object39949:head# 2024-03-20T09:04:59.390 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.390 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b474e9e9:::benchmark_data_smithi122_119671_object50806:head# 2024-03-20T09:04:59.390 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.390 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4760a53:::benchmark_data_smithi122_119671_object41046:head# 2024-03-20T09:04:59.391 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.391 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4794423:::benchmark_data_smithi122_119671_object89207:head# 2024-03-20T09:04:59.391 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.391 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b479d4a6:::benchmark_data_smithi122_119671_object98294:head# 2024-03-20T09:04:59.391 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.391 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b47a1576:::benchmark_data_smithi122_119671_object36931:head# 2024-03-20T09:04:59.391 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.391 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b47a9641:::benchmark_data_smithi122_119671_object32630:head# 2024-03-20T09:04:59.391 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.391 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b47b5756:::benchmark_data_smithi122_119671_object76486:head# 2024-03-20T09:04:59.391 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.392 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b47b5ecc:::benchmark_data_smithi122_119671_object82333:head# 2024-03-20T09:04:59.392 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.392 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b47c92c7:::benchmark_data_smithi122_119671_object70912:head# 2024-03-20T09:04:59.392 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.392 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b47cfe69:::benchmark_data_smithi122_119671_object43364:head# 2024-03-20T09:04:59.392 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.392 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b47d6cc1:::benchmark_data_smithi122_119671_object87462:head# 2024-03-20T09:04:59.392 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.392 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b47eb736:::benchmark_data_smithi122_119671_object39782:head# 2024-03-20T09:04:59.392 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.392 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b47ef6de:::benchmark_data_smithi122_119671_object96119:head# 2024-03-20T09:04:59.392 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.393 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b480880b:::benchmark_data_smithi122_119671_object96345:head# 2024-03-20T09:04:59.393 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.393 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b480b322:::benchmark_data_smithi122_119671_object94230:head# 2024-03-20T09:04:59.393 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.393 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4829e1f:::benchmark_data_smithi122_119671_object20204:head# 2024-03-20T09:04:59.393 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.393 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b482b6d3:::benchmark_data_smithi122_119671_object92979:head# 2024-03-20T09:04:59.393 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.393 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b482bf7b:::benchmark_data_smithi122_119671_object13885:head# 2024-03-20T09:04:59.393 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.393 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b48351be:::benchmark_data_smithi122_119671_object79281:head# 2024-03-20T09:04:59.393 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.394 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4845220:::benchmark_data_smithi122_119671_object54321:head# 2024-03-20T09:04:59.394 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.394 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4849443:::benchmark_data_smithi122_119671_object93188:head# 2024-03-20T09:04:59.394 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.394 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b485347a:::benchmark_data_smithi122_119671_object4608:head# 2024-03-20T09:04:59.394 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.394 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b485e760:::benchmark_data_smithi122_119671_object87199:head# 2024-03-20T09:04:59.394 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.394 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4861d3b:::benchmark_data_smithi122_119671_object33414:head# 2024-03-20T09:04:59.394 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.395 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4864346:::benchmark_data_smithi122_119671_object93911:head# 2024-03-20T09:04:59.395 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.395 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4866e7c:::benchmark_data_smithi122_119671_object74873:head# 2024-03-20T09:04:59.395 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.395 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4867df6:::benchmark_data_smithi122_119671_object89895:head# 2024-03-20T09:04:59.395 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.395 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4877aef:::benchmark_data_smithi122_119671_object60057:head# 2024-03-20T09:04:59.395 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.395 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b487d2ec:::benchmark_data_smithi122_119671_object22726:head# 2024-03-20T09:04:59.395 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.395 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b487e2b0:::benchmark_data_smithi122_119671_object51649:head# 2024-03-20T09:04:59.395 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.396 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b48814d8:::benchmark_data_smithi122_119671_object55188:head# 2024-03-20T09:04:59.396 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.396 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b489d413:::benchmark_data_smithi122_119671_object60330:head# 2024-03-20T09:04:59.396 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.396 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b48a3c59:::benchmark_data_smithi122_119671_object71880:head# 2024-03-20T09:04:59.396 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.396 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b48aa1df:::benchmark_data_smithi122_119671_object25466:head# 2024-03-20T09:04:59.396 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.396 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b48b43e8:::benchmark_data_smithi122_119671_object64215:head# 2024-03-20T09:04:59.396 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.397 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b48be5c9:::benchmark_data_smithi122_119671_object78982:head# 2024-03-20T09:04:59.397 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.397 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b48bfe29:::benchmark_data_smithi122_119671_object74254:head# 2024-03-20T09:04:59.397 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.397 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b48c2b1e:::benchmark_data_smithi122_119671_object8568:head# 2024-03-20T09:04:59.397 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.397 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b48c72b3:::benchmark_data_smithi122_119671_object43600:head# 2024-03-20T09:04:59.397 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.397 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b48d44de:::benchmark_data_smithi122_119671_object16385:head# 2024-03-20T09:04:59.397 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.398 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b48d4889:::benchmark_data_smithi122_119671_object66420:head# 2024-03-20T09:04:59.398 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.398 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b48dcb7a:::benchmark_data_smithi122_119671_object7033:head# 2024-03-20T09:04:59.398 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.398 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b48e35e0:::benchmark_data_smithi122_119671_object66020:head# 2024-03-20T09:04:59.398 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.398 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4900f06:::benchmark_data_smithi122_119671_object66880:head# 2024-03-20T09:04:59.398 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.398 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4908bee:::benchmark_data_smithi122_119671_object77654:head# 2024-03-20T09:04:59.399 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.399 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4908d89:::benchmark_data_smithi122_119671_object33297:head# 2024-03-20T09:04:59.399 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.399 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4910710:::benchmark_data_smithi122_119671_object26674:head# 2024-03-20T09:04:59.399 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.399 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b49164e9:::benchmark_data_smithi122_119671_object78331:head# 2024-03-20T09:04:59.399 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.399 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b491e82c:::benchmark_data_smithi122_119671_object19543:head# 2024-03-20T09:04:59.399 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.399 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b492c26b:::benchmark_data_smithi122_119671_object45997:head# 2024-03-20T09:04:59.399 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.400 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4936947:::benchmark_data_smithi122_119671_object80038:head# 2024-03-20T09:04:59.400 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.400 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4945776:::benchmark_data_smithi122_119671_object10155:head# 2024-03-20T09:04:59.400 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.400 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4949468:::benchmark_data_smithi122_119671_object53087:head# 2024-03-20T09:04:59.400 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.400 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b494b111:::benchmark_data_smithi122_119671_object23799:head# 2024-03-20T09:04:59.400 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.400 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4951b2b:::benchmark_data_smithi122_119671_object64834:head# 2024-03-20T09:04:59.400 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.400 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4956618:::benchmark_data_smithi122_119671_object39031:head# 2024-03-20T09:04:59.400 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.401 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b495a3e1:::benchmark_data_smithi122_119671_object76075:head# 2024-03-20T09:04:59.401 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.401 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4970871:::benchmark_data_smithi122_119671_object96281:head# 2024-03-20T09:04:59.401 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.401 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b49794e6:::benchmark_data_smithi122_119671_object58710:head# 2024-03-20T09:04:59.401 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.401 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4981d3c:::benchmark_data_smithi122_119671_object64135:head# 2024-03-20T09:04:59.401 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.401 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b49915c9:::benchmark_data_smithi122_119671_object56806:head# 2024-03-20T09:04:59.401 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.401 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b499ab97:::benchmark_data_smithi122_119671_object99702:head# 2024-03-20T09:04:59.401 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.402 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b49b21dc:::benchmark_data_smithi122_119671_object37120:head# 2024-03-20T09:04:59.402 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.402 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b49c2466:::benchmark_data_smithi122_119671_object27310:head# 2024-03-20T09:04:59.402 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.402 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b49dfb01:::benchmark_data_smithi122_119671_object5447:head# 2024-03-20T09:04:59.402 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.402 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b49ebfc6:::benchmark_data_smithi122_119671_object61045:head# 2024-03-20T09:04:59.402 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.402 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4a021c6:::benchmark_data_smithi122_119671_object59301:head# 2024-03-20T09:04:59.403 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.403 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4a123a8:::benchmark_data_smithi122_119671_object5642:head# 2024-03-20T09:04:59.403 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.403 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4a1e2c9:::benchmark_data_smithi122_119671_object53697:head# 2024-03-20T09:04:59.403 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.403 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4a2add4:::benchmark_data_smithi122_119671_object67539:head# 2024-03-20T09:04:59.403 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.403 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4a393c4:::benchmark_data_smithi122_119671_object53852:head# 2024-03-20T09:04:59.403 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.403 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4a4dab1:::benchmark_data_smithi122_119671_object61808:head# 2024-03-20T09:04:59.403 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.403 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4a4e570:::benchmark_data_smithi122_119671_object70184:head# 2024-03-20T09:04:59.404 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.404 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4a53592:::benchmark_data_smithi122_119671_object61100:head# 2024-03-20T09:04:59.404 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.404 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4a60305:::benchmark_data_smithi122_119671_object1979:head# 2024-03-20T09:04:59.404 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.404 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4a67db7:::benchmark_data_smithi122_119671_object55337:head# 2024-03-20T09:04:59.404 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.404 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4a706d0:::benchmark_data_smithi122_119671_object62904:head# 2024-03-20T09:04:59.404 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.404 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4a70f25:::benchmark_data_smithi122_119671_object1986:head# 2024-03-20T09:04:59.405 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.405 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4a8cf6f:::benchmark_data_smithi122_119671_object86580:head# 2024-03-20T09:04:59.405 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.405 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4a8ee56:::benchmark_data_smithi122_119671_object10610:head# 2024-03-20T09:04:59.405 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.405 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4a916ad:::benchmark_data_smithi122_119671_object54060:head# 2024-03-20T09:04:59.405 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.405 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4a9f9ee:::benchmark_data_smithi122_119671_object16726:head# 2024-03-20T09:04:59.405 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.405 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4ab4863:::benchmark_data_smithi122_119671_object80226:head# 2024-03-20T09:04:59.405 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.406 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4ab88eb:::benchmark_data_smithi122_119671_object82273:head# 2024-03-20T09:04:59.406 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.406 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4abb361:::benchmark_data_smithi122_119671_object39702:head# 2024-03-20T09:04:59.406 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.406 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4adc617:::benchmark_data_smithi122_119671_object92449:head# 2024-03-20T09:04:59.406 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.406 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4ae07b6:::benchmark_data_smithi122_119671_object90923:head# 2024-03-20T09:04:59.406 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.406 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4af8b3a:::benchmark_data_smithi122_119671_object27603:head# 2024-03-20T09:04:59.406 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.407 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4b022f5:::benchmark_data_smithi122_119671_object74690:head# 2024-03-20T09:04:59.407 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.407 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4b0b593:::benchmark_data_smithi122_119671_object10770:head# 2024-03-20T09:04:59.407 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.407 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4b101bf:::benchmark_data_smithi122_119671_object22305:head# 2024-03-20T09:04:59.407 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.407 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4b1053e:::benchmark_data_smithi122_119671_object98559:head# 2024-03-20T09:04:59.407 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.407 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4b2ca89:::benchmark_data_smithi122_119671_object66352:head# 2024-03-20T09:04:59.407 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.407 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4b45376:::benchmark_data_smithi122_119671_object22839:head# 2024-03-20T09:04:59.407 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.408 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4b503d5:::benchmark_data_smithi122_119671_object66176:head# 2024-03-20T09:04:59.408 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.408 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4b54d6e:::benchmark_data_smithi122_119671_object43731:head# 2024-03-20T09:04:59.408 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.408 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4b579b6:::benchmark_data_smithi122_119671_object37210:head# 2024-03-20T09:04:59.408 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.408 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4b58bfd:::benchmark_data_smithi122_119671_object35433:head# 2024-03-20T09:04:59.408 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.408 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4b59d9d:::benchmark_data_smithi122_119671_object76577:head# 2024-03-20T09:04:59.408 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.408 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4b6d36c:::benchmark_data_smithi122_119671_object24245:head# 2024-03-20T09:04:59.408 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.409 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4b6d741:::benchmark_data_smithi122_119671_object84762:head# 2024-03-20T09:04:59.409 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.409 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4b6dca0:::benchmark_data_smithi122_119671_object95085:head# 2024-03-20T09:04:59.409 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.409 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4b6eedf:::benchmark_data_smithi122_119671_object36751:head# 2024-03-20T09:04:59.409 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.409 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4b6f58a:::benchmark_data_smithi122_119671_object94597:head# 2024-03-20T09:04:59.409 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.409 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4b71954:::benchmark_data_smithi122_119671_object29097:head# 2024-03-20T09:04:59.409 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.409 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4b72574:::benchmark_data_smithi122_119671_object49343:head# 2024-03-20T09:04:59.410 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.410 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4b8217c:::benchmark_data_smithi122_119671_object62683:head# 2024-03-20T09:04:59.410 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.410 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4b85bd9:::benchmark_data_smithi122_119671_object14762:head# 2024-03-20T09:04:59.410 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.410 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4b905b8:::benchmark_data_smithi122_119671_object41797:head# 2024-03-20T09:04:59.410 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.410 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4b96bd6:::benchmark_data_smithi122_119671_object5599:head# 2024-03-20T09:04:59.410 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.410 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4b9a682:::benchmark_data_smithi122_119671_object63083:head# 2024-03-20T09:04:59.410 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.410 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4ba92ef:::benchmark_data_smithi122_119671_object17378:head# 2024-03-20T09:04:59.411 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.411 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4baf830:::benchmark_data_smithi122_119671_object75690:head# 2024-03-20T09:04:59.411 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.411 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4bb39f7:::benchmark_data_smithi122_119671_object66731:head# 2024-03-20T09:04:59.411 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.411 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4bc4926:::benchmark_data_smithi122_119671_object51526:head# 2024-03-20T09:04:59.411 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.411 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4bc8be8:::benchmark_data_smithi122_119671_object8637:head# 2024-03-20T09:04:59.411 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.411 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4bce129:::benchmark_data_smithi122_119671_object62638:head# 2024-03-20T09:04:59.411 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.411 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4bd4e5c:::benchmark_data_smithi122_119671_object58015:head# 2024-03-20T09:04:59.412 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.412 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4bd7f5c:::benchmark_data_smithi122_119671_object55362:head# 2024-03-20T09:04:59.412 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.412 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4beb429:::benchmark_data_smithi122_119671_object9824:head# 2024-03-20T09:04:59.412 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.412 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4bf57dc:::benchmark_data_smithi122_119671_object25398:head# 2024-03-20T09:04:59.412 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.412 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4bfbf1b:::benchmark_data_smithi122_119671_object47692:head# 2024-03-20T09:04:59.412 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.412 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4c0cc23:::benchmark_data_smithi122_119671_object78169:head# 2024-03-20T09:04:59.412 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.412 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4c175ae:::benchmark_data_smithi122_119671_object14013:head# 2024-03-20T09:04:59.413 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.413 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4c23dd3:::benchmark_data_smithi122_119671_object60105:head# 2024-03-20T09:04:59.413 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.413 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4c4d7d5:::benchmark_data_smithi122_119671_object98574:head# 2024-03-20T09:04:59.413 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.413 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4c5c771:::benchmark_data_smithi122_119671_object97079:head# 2024-03-20T09:04:59.413 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.413 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4c5e2d0:::benchmark_data_smithi122_119671_object36605:head# 2024-03-20T09:04:59.413 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.413 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4c6ab10:::benchmark_data_smithi122_119671_object95625:head# 2024-03-20T09:04:59.413 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.413 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4c6cb10:::benchmark_data_smithi122_119671_object49751:head# 2024-03-20T09:04:59.413 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.414 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4c74e31:::benchmark_data_smithi122_119671_object34729:head# 2024-03-20T09:04:59.414 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.414 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4c7f15e:::benchmark_data_smithi122_119671_object85783:head# 2024-03-20T09:04:59.414 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.414 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4c87758:::benchmark_data_smithi122_119671_object37229:head# 2024-03-20T09:04:59.414 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.414 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4c9cb54:::benchmark_data_smithi122_119671_object51640:head# 2024-03-20T09:04:59.414 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.414 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4c9cd48:::benchmark_data_smithi122_119671_object77402:head# 2024-03-20T09:04:59.414 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.414 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4ca1f7c:::benchmark_data_smithi122_119671_object43002:head# 2024-03-20T09:04:59.415 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.415 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4ca82af:::benchmark_data_smithi122_119671_object83452:head# 2024-03-20T09:04:59.415 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.415 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4cad607:::benchmark_data_smithi122_119671_object47754:head# 2024-03-20T09:04:59.415 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.415 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4cb313a:::benchmark_data_smithi122_119671_object96438:head# 2024-03-20T09:04:59.415 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.415 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4cb65e2:::benchmark_data_smithi122_119671_object30307:head# 2024-03-20T09:04:59.415 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.415 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4cbd045:::benchmark_data_smithi122_119671_object41624:head# 2024-03-20T09:04:59.415 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.416 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4cbe55c:::benchmark_data_smithi122_119671_object51700:head# 2024-03-20T09:04:59.416 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.416 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4cbf418:::benchmark_data_smithi122_119671_object55955:head# 2024-03-20T09:04:59.416 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.416 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4cd07b7:::benchmark_data_smithi122_119671_object91351:head# 2024-03-20T09:04:59.416 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.416 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4cd3cee:::benchmark_data_smithi122_119671_object7986:head# 2024-03-20T09:04:59.416 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.416 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4cd6f19:::benchmark_data_smithi122_119671_object74443:head# 2024-03-20T09:04:59.416 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.416 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4cdb020:::benchmark_data_smithi122_119671_object89548:head# 2024-03-20T09:04:59.417 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.417 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4ce6c44:::benchmark_data_smithi122_119671_object95216:head# 2024-03-20T09:04:59.417 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.417 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4cef7bd:::benchmark_data_smithi122_119671_object59390:head# 2024-03-20T09:04:59.417 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.417 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4cefe6b:::benchmark_data_smithi122_119671_object80050:head# 2024-03-20T09:04:59.417 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.417 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4cf275a:::benchmark_data_smithi122_119671_object59915:head# 2024-03-20T09:04:59.417 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.417 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4cff128:::benchmark_data_smithi122_119671_object6761:head# 2024-03-20T09:04:59.417 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.417 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4d0bdc7:::benchmark_data_smithi122_119671_object95903:head# 2024-03-20T09:04:59.417 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.418 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4d14e45:::benchmark_data_smithi122_119671_object30327:head# 2024-03-20T09:04:59.418 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.418 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4d39ab4:::benchmark_data_smithi122_119671_object29376:head# 2024-03-20T09:04:59.418 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.418 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4d43719:::benchmark_data_smithi122_119671_object88602:head# 2024-03-20T09:04:59.418 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.418 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4d4c57d:::benchmark_data_smithi122_119671_object76418:head# 2024-03-20T09:04:59.418 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.418 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4d5aa97:::benchmark_data_smithi122_119671_object53460:head# 2024-03-20T09:04:59.418 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.418 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4d7c38a:::benchmark_data_smithi122_119671_object75067:head# 2024-03-20T09:04:59.419 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.419 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:04:59.447 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:59.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:59.469 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:04:59.470 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:04:59.475 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:04:59 smithi026 ceph-mon[68309]: 3.1 deep-scrub ok 2024-03-20T09:04:59.572 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:04:59.611 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:04:59.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:04:59.636 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:04:59.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:04:59.641 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:04:59.641 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:04:59.641 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-03-20T09:04:59.641 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-03-20T09:04:59.641 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:04:59.688 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4d7ea46:::benchmark_data_smithi122_119671_object75578:head# 2024-03-20T09:04:59.688 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.688 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4d83b76:::benchmark_data_smithi122_119671_object50370:head# 2024-03-20T09:04:59.688 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.688 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4d9d1c4:::benchmark_data_smithi122_119671_object63936:head# 2024-03-20T09:04:59.688 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.688 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4da4772:::benchmark_data_smithi122_119671_object33984:head# 2024-03-20T09:04:59.688 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.688 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4db06c7:::benchmark_data_smithi122_119671_object28271:head# 2024-03-20T09:04:59.688 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.689 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4dc1e59:::benchmark_data_smithi122_119671_object45029:head# 2024-03-20T09:04:59.689 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.689 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4dccc3c:::benchmark_data_smithi122_119671_object28846:head# 2024-03-20T09:04:59.689 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.689 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4dd6f49:::benchmark_data_smithi122_119671_object3958:head# 2024-03-20T09:04:59.689 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.689 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4de38d7:::benchmark_data_smithi122_119671_object73646:head# 2024-03-20T09:04:59.689 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.689 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4de3ee5:::benchmark_data_smithi122_119671_object45870:head# 2024-03-20T09:04:59.689 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.689 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4dfe456:::benchmark_data_smithi122_119671_object15246:head# 2024-03-20T09:04:59.689 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.690 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4e0f8f6:::benchmark_data_smithi122_119671_object24465:head# 2024-03-20T09:04:59.690 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.690 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4e153d1:::benchmark_data_smithi122_119671_object20784:head# 2024-03-20T09:04:59.690 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.690 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4e2adaa:::benchmark_data_smithi122_119671_object47855:head# 2024-03-20T09:04:59.690 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.690 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4e2c87d:::benchmark_data_smithi122_119671_object10794:head# 2024-03-20T09:04:59.690 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.690 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4e5292c:::benchmark_data_smithi122_119671_object9157:head# 2024-03-20T09:04:59.690 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.690 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4e56904:::benchmark_data_smithi122_119671_object95809:head# 2024-03-20T09:04:59.690 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.691 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4e5c34c:::benchmark_data_smithi122_119671_object65287:head# 2024-03-20T09:04:59.691 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.691 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4e91def:::benchmark_data_smithi122_119671_object54733:head# 2024-03-20T09:04:59.691 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.691 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4ea49bf:::benchmark_data_smithi122_119671_object49242:head# 2024-03-20T09:04:59.691 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.691 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4ebca98:::benchmark_data_smithi122_119671_object3393:head# 2024-03-20T09:04:59.691 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.691 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4ec3559:::benchmark_data_smithi122_119671_object56823:head# 2024-03-20T09:04:59.691 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.691 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4ec64f6:::benchmark_data_smithi122_119671_object83094:head# 2024-03-20T09:04:59.691 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.692 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4ed1073:::benchmark_data_smithi122_119671_object1960:head# 2024-03-20T09:04:59.692 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.692 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4eda4c4:::benchmark_data_smithi122_119671_object55334:head# 2024-03-20T09:04:59.692 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.692 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4eff95f:::benchmark_data_smithi122_119671_object77132:head# 2024-03-20T09:04:59.692 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.692 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4f134fd:::benchmark_data_smithi122_119671_object52280:head# 2024-03-20T09:04:59.692 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.692 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4f29ac0:::benchmark_data_smithi122_119671_object32694:head# 2024-03-20T09:04:59.692 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.692 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4f31134:::benchmark_data_smithi122_119671_object92829:head# 2024-03-20T09:04:59.692 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.693 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4f45799:::benchmark_data_smithi122_119671_object26381:head# 2024-03-20T09:04:59.693 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.693 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4f46617:::benchmark_data_smithi122_119671_object80263:head# 2024-03-20T09:04:59.693 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.693 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4f5c17e:::benchmark_data_smithi122_119671_object6878:head# 2024-03-20T09:04:59.693 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.693 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4f655b7:::benchmark_data_smithi122_119671_object78151:head# 2024-03-20T09:04:59.693 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.693 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4f6d21a:::benchmark_data_smithi122_119671_object19696:head# 2024-03-20T09:04:59.693 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.693 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4f6daf3:::benchmark_data_smithi122_119671_object20093:head# 2024-03-20T09:04:59.694 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.694 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4f73307:::benchmark_data_smithi122_119671_object13908:head# 2024-03-20T09:04:59.694 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.694 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4f80d61:::benchmark_data_smithi122_119671_object62707:head# 2024-03-20T09:04:59.694 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.694 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4f8bb65:::benchmark_data_smithi122_119671_object68424:head# 2024-03-20T09:04:59.694 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.694 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4f90a58:::benchmark_data_smithi122_119671_object23739:head# 2024-03-20T09:04:59.694 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.694 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4f9cc29:::benchmark_data_smithi122_119671_object17170:head# 2024-03-20T09:04:59.694 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.695 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4fba6b1:::benchmark_data_smithi122_119671_object77526:head# 2024-03-20T09:04:59.695 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.695 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4fbb6f8:::benchmark_data_smithi122_119671_object64624:head# 2024-03-20T09:04:59.695 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.695 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4fbdfe8:::benchmark_data_smithi122_119671_object64422:head# 2024-03-20T09:04:59.695 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.695 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4fc1f31:::benchmark_data_smithi122_119671_object82149:head# 2024-03-20T09:04:59.695 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.695 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4fc7610:::benchmark_data_smithi122_119671_object4403:head# 2024-03-20T09:04:59.695 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.695 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4fe3d39:::benchmark_data_smithi122_119671_object94913:head# 2024-03-20T09:04:59.696 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.696 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4ff445c:::benchmark_data_smithi122_119671_object49960:head# 2024-03-20T09:04:59.696 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.696 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b4ffb9a8:::benchmark_data_smithi122_119671_object76465:head# 2024-03-20T09:04:59.696 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.696 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b50010f3:::benchmark_data_smithi122_119671_object13453:head# 2024-03-20T09:04:59.696 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.696 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b500ebe4:::benchmark_data_smithi122_119671_object77506:head# 2024-03-20T09:04:59.696 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.696 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b501744e:::benchmark_data_smithi122_119671_object3260:head# 2024-03-20T09:04:59.696 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.697 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b50200dc:::benchmark_data_smithi122_119671_object32759:head# 2024-03-20T09:04:59.697 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.697 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5021e22:::benchmark_data_smithi122_119671_object47047:head# 2024-03-20T09:04:59.697 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.697 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b502b180:::benchmark_data_smithi122_119671_object16209:head# 2024-03-20T09:04:59.697 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.697 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b504070c:::benchmark_data_smithi122_119671_object78345:head# 2024-03-20T09:04:59.697 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.697 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b505560d:::benchmark_data_smithi122_119671_object28231:head# 2024-03-20T09:04:59.697 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.697 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b506b9be:::benchmark_data_smithi122_119671_object72335:head# 2024-03-20T09:04:59.697 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.698 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b506c5da:::benchmark_data_smithi122_119671_object20225:head# 2024-03-20T09:04:59.698 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.698 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5071572:::benchmark_data_smithi122_119671_object40749:head# 2024-03-20T09:04:59.698 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.698 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b507b44e:::benchmark_data_smithi122_119671_object57779:head# 2024-03-20T09:04:59.698 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.698 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b508662d:::benchmark_data_smithi122_119671_object73683:head# 2024-03-20T09:04:59.698 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.698 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5091751:::benchmark_data_smithi122_119671_object1311:head# 2024-03-20T09:04:59.698 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.698 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b50c0536:::benchmark_data_smithi122_119671_object63034:head# 2024-03-20T09:04:59.698 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.699 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b50ca0f2:::benchmark_data_smithi122_119671_object9045:head# 2024-03-20T09:04:59.699 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.699 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b50cbdad:::benchmark_data_smithi122_119671_object68437:head# 2024-03-20T09:04:59.699 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.699 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b50ccb62:::benchmark_data_smithi122_119671_object62965:head# 2024-03-20T09:04:59.699 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.699 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b50ce447:::benchmark_data_smithi122_119671_object70960:head# 2024-03-20T09:04:59.699 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.699 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b50d9114:::benchmark_data_smithi122_119671_object11537:head# 2024-03-20T09:04:59.699 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.699 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b50dac26:::benchmark_data_smithi122_119671_object47831:head# 2024-03-20T09:04:59.699 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.700 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b50e21e3:::benchmark_data_smithi122_119671_object21133:head# 2024-03-20T09:04:59.700 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.700 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b50e395b:::benchmark_data_smithi122_119671_object48910:head# 2024-03-20T09:04:59.700 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.700 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b50f327c:::benchmark_data_smithi122_119671_object43327:head# 2024-03-20T09:04:59.700 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.700 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b50f55ad:::benchmark_data_smithi122_119671_object8714:head# 2024-03-20T09:04:59.700 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.700 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b51086dd:::benchmark_data_smithi122_119671_object75937:head# 2024-03-20T09:04:59.700 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.700 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5108b67:::benchmark_data_smithi122_119671_object30528:head# 2024-03-20T09:04:59.700 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.701 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b511cb78:::benchmark_data_smithi122_119671_object77447:head# 2024-03-20T09:04:59.701 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.701 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b511d033:::benchmark_data_smithi122_119671_object82806:head# 2024-03-20T09:04:59.701 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.701 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b512aebb:::benchmark_data_smithi122_119671_object5975:head# 2024-03-20T09:04:59.701 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.701 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b513360b:::benchmark_data_smithi122_119671_object11740:head# 2024-03-20T09:04:59.701 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.701 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5134172:::benchmark_data_smithi122_119671_object46691:head# 2024-03-20T09:04:59.701 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.701 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b51367d9:::benchmark_data_smithi122_119671_object95499:head# 2024-03-20T09:04:59.701 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.701 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5158f57:::benchmark_data_smithi122_119671_object13814:head# 2024-03-20T09:04:59.702 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.702 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5167312:::benchmark_data_smithi122_119671_object9258:head# 2024-03-20T09:04:59.702 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.702 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b516ca8e:::benchmark_data_smithi122_119671_object16018:head# 2024-03-20T09:04:59.702 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.702 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b516e8a8:::benchmark_data_smithi122_119671_object36402:head# 2024-03-20T09:04:59.702 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.702 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5176266:::benchmark_data_smithi122_119671_object29463:head# 2024-03-20T09:04:59.702 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.702 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b519cfcf:::benchmark_data_smithi122_119671_object14239:head# 2024-03-20T09:04:59.702 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.702 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b51a0eca:::benchmark_data_smithi122_119671_object81440:head# 2024-03-20T09:04:59.703 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.703 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b51a6a67:::benchmark_data_smithi122_119671_object77954:head# 2024-03-20T09:04:59.703 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.703 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b51ab924:::benchmark_data_smithi122_119671_object6775:head# 2024-03-20T09:04:59.703 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.703 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b51b0889:::benchmark_data_smithi122_119671_object1534:head# 2024-03-20T09:04:59.703 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.703 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b51b7c83:::benchmark_data_smithi122_119671_object77947:head# 2024-03-20T09:04:59.703 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.703 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b51c2b90:::benchmark_data_smithi122_119671_object11265:head# 2024-03-20T09:04:59.703 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.703 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b51d3f09:::benchmark_data_smithi122_119671_object26157:head# 2024-03-20T09:04:59.703 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.704 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b51d409f:::benchmark_data_smithi122_119671_object72740:head# 2024-03-20T09:04:59.704 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.704 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b51d7746:::benchmark_data_smithi122_119671_object82051:head# 2024-03-20T09:04:59.704 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.704 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b51f1757:::benchmark_data_smithi122_119671_object30705:head# 2024-03-20T09:04:59.704 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.704 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b51f5e24:::benchmark_data_smithi122_119671_object93765:head# 2024-03-20T09:04:59.704 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.704 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b51f9995:::benchmark_data_smithi122_119671_object71757:head# 2024-03-20T09:04:59.704 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.704 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b51ff1c0:::benchmark_data_smithi122_119671_object13429:head# 2024-03-20T09:04:59.705 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.705 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b520a0a8:::benchmark_data_smithi122_119671_object8148:head# 2024-03-20T09:04:59.705 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.705 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b520b4e8:::benchmark_data_smithi122_119671_object19824:head# 2024-03-20T09:04:59.705 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.705 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b521a0dd:::benchmark_data_smithi122_119671_object79880:head# 2024-03-20T09:04:59.705 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.705 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b521e6f2:::benchmark_data_smithi122_119671_object97306:head# 2024-03-20T09:04:59.705 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.705 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5221605:::benchmark_data_smithi122_119671_object34165:head# 2024-03-20T09:04:59.705 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.705 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5225003:::benchmark_data_smithi122_119671_object83444:head# 2024-03-20T09:04:59.706 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.706 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5233660:::benchmark_data_smithi122_119671_object97621:head# 2024-03-20T09:04:59.706 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.706 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5235a2b:::benchmark_data_smithi122_119671_object55980:head# 2024-03-20T09:04:59.706 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.706 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b523c8c5:::benchmark_data_smithi122_119671_object13133:head# 2024-03-20T09:04:59.706 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.706 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b523d164:::benchmark_data_smithi122_119671_object80707:head# 2024-03-20T09:04:59.706 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.706 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b523eeb7:::benchmark_data_smithi122_119671_object83204:head# 2024-03-20T09:04:59.706 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.706 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b523f5c9:::benchmark_data_smithi122_119671_object27301:head# 2024-03-20T09:04:59.707 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.707 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b524202c:::benchmark_data_smithi122_119671_object9508:head# 2024-03-20T09:04:59.707 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.707 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5247ba2:::benchmark_data_smithi122_119671_object88327:head# 2024-03-20T09:04:59.707 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.707 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b52517e5:::benchmark_data_smithi122_119671_object87390:head# 2024-03-20T09:04:59.707 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.707 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5267331:::benchmark_data_smithi122_119671_object7935:head# 2024-03-20T09:04:59.707 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.707 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5267f1e:::benchmark_data_smithi122_119671_object3310:head# 2024-03-20T09:04:59.707 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.707 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5283f15:::benchmark_data_smithi122_119671_object42876:head# 2024-03-20T09:04:59.708 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.708 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b52a4a59:::benchmark_data_smithi122_119671_object84585:head# 2024-03-20T09:04:59.708 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.708 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b52b259a:::benchmark_data_smithi122_119671_object36909:head# 2024-03-20T09:04:59.708 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.708 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b52dbc35:::benchmark_data_smithi122_119671_object45285:head# 2024-03-20T09:04:59.708 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.708 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b52f27c9:::benchmark_data_smithi122_119671_object62689:head# 2024-03-20T09:04:59.708 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.708 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b52fe60b:::benchmark_data_smithi122_119671_object93840:head# 2024-03-20T09:04:59.708 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.708 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5323f17:::benchmark_data_smithi122_119671_object57164:head# 2024-03-20T09:04:59.709 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.709 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b532a197:::benchmark_data_smithi122_119671_object84135:head# 2024-03-20T09:04:59.709 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.709 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b533de8a:::benchmark_data_smithi122_119671_object7347:head# 2024-03-20T09:04:59.709 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.709 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b534d1d1:::benchmark_data_smithi122_119671_object82514:head# 2024-03-20T09:04:59.709 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.709 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b534f22c:::benchmark_data_smithi122_119671_object69267:head# 2024-03-20T09:04:59.709 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.709 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5359370:::benchmark_data_smithi122_119671_object88489:head# 2024-03-20T09:04:59.709 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.709 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b536b057:::benchmark_data_smithi122_119671_object10833:head# 2024-03-20T09:04:59.709 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.710 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5372e63:::benchmark_data_smithi122_119671_object13168:head# 2024-03-20T09:04:59.710 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.710 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b537f180:::benchmark_data_smithi122_119671_object61362:head# 2024-03-20T09:04:59.710 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.710 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5384c90:::benchmark_data_smithi122_119671_object34028:head# 2024-03-20T09:04:59.710 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.710 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b539275d:::benchmark_data_smithi122_119671_object37089:head# 2024-03-20T09:04:59.710 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.710 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5394a94:::benchmark_data_smithi122_119671_object3400:head# 2024-03-20T09:04:59.710 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.710 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b53a0be4:::benchmark_data_smithi122_119671_object72401:head# 2024-03-20T09:04:59.710 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.711 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b53a5ddf:::benchmark_data_smithi122_119671_object88933:head# 2024-03-20T09:04:59.711 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.711 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b53ab526:::benchmark_data_smithi122_119671_object12264:head# 2024-03-20T09:04:59.711 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.711 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b53b34cd:::benchmark_data_smithi122_119671_object51775:head# 2024-03-20T09:04:59.711 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.711 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b53beddb:::benchmark_data_smithi122_119671_object67424:head# 2024-03-20T09:04:59.711 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.711 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b53c437b:::benchmark_data_smithi122_119671_object78349:head# 2024-03-20T09:04:59.711 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.711 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b53c6504:::benchmark_data_smithi122_119671_object79340:head# 2024-03-20T09:04:59.712 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.712 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b53ec34c:::benchmark_data_smithi122_119671_object84376:head# 2024-03-20T09:04:59.712 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.712 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b53ed7b4:::benchmark_data_smithi122_119671_object21030:head# 2024-03-20T09:04:59.712 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.712 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b540332f:::benchmark_data_smithi122_119671_object47721:head# 2024-03-20T09:04:59.712 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.712 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5412d43:::benchmark_data_smithi122_119671_object30161:head# 2024-03-20T09:04:59.712 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.712 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5419f63:::benchmark_data_smithi122_119671_object53046:head# 2024-03-20T09:04:59.712 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.712 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b541cb45:::benchmark_data_smithi122_119671_object57628:head# 2024-03-20T09:04:59.713 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.713 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b54363af:::benchmark_data_smithi122_119671_object36543:head# 2024-03-20T09:04:59.713 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.713 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b543b845:::benchmark_data_smithi122_119671_object7691:head# 2024-03-20T09:04:59.713 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.713 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b54476ba:::benchmark_data_smithi122_119671_object43448:head# 2024-03-20T09:04:59.713 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.713 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5451fa3:::benchmark_data_smithi122_119671_object44926:head# 2024-03-20T09:04:59.713 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.713 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5458b0b:::benchmark_data_smithi122_119671_object12859:head# 2024-03-20T09:04:59.713 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.713 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b545ad99:::benchmark_data_smithi122_119671_object24473:head# 2024-03-20T09:04:59.714 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.714 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b54675bc:::benchmark_data_smithi122_119671_object3780:head# 2024-03-20T09:04:59.714 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.714 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5471327:::benchmark_data_smithi122_119671_object92516:head# 2024-03-20T09:04:59.714 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.714 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b547391a:::benchmark_data_smithi122_119671_object90293:head# 2024-03-20T09:04:59.714 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.714 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b547a82e:::benchmark_data_smithi122_119671_object72309:head# 2024-03-20T09:04:59.714 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.714 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5481d0e:::benchmark_data_smithi122_119671_object48133:head# 2024-03-20T09:04:59.714 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.715 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b548a459:::benchmark_data_smithi122_119671_object97974:head# 2024-03-20T09:04:59.715 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.715 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b54928ea:::benchmark_data_smithi122_119671_object80031:head# 2024-03-20T09:04:59.715 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.715 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5499ddb:::benchmark_data_smithi122_119671_object84067:head# 2024-03-20T09:04:59.715 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.715 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b54a506e:::benchmark_data_smithi122_119671_object29168:head# 2024-03-20T09:04:59.715 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.715 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b54ac415:::benchmark_data_smithi122_119671_object24715:head# 2024-03-20T09:04:59.715 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.715 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b54b272a:::benchmark_data_smithi122_119671_object44851:head# 2024-03-20T09:04:59.715 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.716 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b54b3409:::benchmark_data_smithi122_119671_object53523:head# 2024-03-20T09:04:59.716 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.716 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b54c083a:::benchmark_data_smithi122_119671_object43793:head# 2024-03-20T09:04:59.716 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.716 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b54c80d0:::benchmark_data_smithi122_119671_object30674:head# 2024-03-20T09:04:59.716 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.716 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b54c8eef:::benchmark_data_smithi122_119671_object28934:head# 2024-03-20T09:04:59.716 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.716 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b54c9b2e:::benchmark_data_smithi122_119671_object81459:head# 2024-03-20T09:04:59.716 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.716 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b54cce4e:::benchmark_data_smithi122_119671_object16539:head# 2024-03-20T09:04:59.716 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.717 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b54e06a9:::benchmark_data_smithi122_119671_object80101:head# 2024-03-20T09:04:59.717 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.717 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b54e3fe9:::benchmark_data_smithi122_119671_object4541:head# 2024-03-20T09:04:59.717 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.717 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b54f2375:::benchmark_data_smithi122_119671_object65691:head# 2024-03-20T09:04:59.717 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.717 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b550130f:::benchmark_data_smithi122_119671_object82244:head# 2024-03-20T09:04:59.717 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.717 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5506eb3:::benchmark_data_smithi122_119671_object87885:head# 2024-03-20T09:04:59.717 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.717 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5508ace:::benchmark_data_smithi122_119671_object72089:head# 2024-03-20T09:04:59.717 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.718 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b550d2c5:::benchmark_data_smithi122_119671_object63703:head# 2024-03-20T09:04:59.718 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.718 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5513328:::benchmark_data_smithi122_119671_object37249:head# 2024-03-20T09:04:59.718 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.718 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5514e5d:::benchmark_data_smithi122_119671_object96147:head# 2024-03-20T09:04:59.718 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.718 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b551bb2b:::benchmark_data_smithi122_119671_object66990:head# 2024-03-20T09:04:59.718 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.718 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b553702d:::benchmark_data_smithi122_119671_object68868:head# 2024-03-20T09:04:59.718 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.718 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b55399f2:::benchmark_data_smithi122_119671_object67164:head# 2024-03-20T09:04:59.718 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.719 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5540ccf:::benchmark_data_smithi122_119671_object98788:head# 2024-03-20T09:04:59.719 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.719 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b554b46d:::benchmark_data_smithi122_119671_object1467:head# 2024-03-20T09:04:59.719 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.719 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b555580c:::benchmark_data_smithi122_119671_object43367:head# 2024-03-20T09:04:59.719 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.719 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b555668c:::benchmark_data_smithi122_119671_object27101:head# 2024-03-20T09:04:59.719 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.719 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b555e56d:::benchmark_data_smithi122_119671_object27702:head# 2024-03-20T09:04:59.719 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.719 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5564f21:::benchmark_data_smithi122_119671_object5333:head# 2024-03-20T09:04:59.720 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.720 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5567450:::benchmark_data_smithi122_119671_object92256:head# 2024-03-20T09:04:59.720 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.720 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5568e42:::benchmark_data_smithi122_119671_object36850:head# 2024-03-20T09:04:59.720 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.720 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b556b744:::benchmark_data_smithi122_119671_object57967:head# 2024-03-20T09:04:59.720 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.720 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b556c0aa:::benchmark_data_smithi122_119671_object81144:head# 2024-03-20T09:04:59.720 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.720 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b557c206:::benchmark_data_smithi122_119671_object77881:head# 2024-03-20T09:04:59.720 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.720 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5583444:::benchmark_data_smithi122_119671_object64305:head# 2024-03-20T09:04:59.721 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.721 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b559ce48:::benchmark_data_smithi122_119671_object54206:head# 2024-03-20T09:04:59.721 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.721 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b55a5444:::benchmark_data_smithi122_119671_object11348:head# 2024-03-20T09:04:59.721 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.721 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b55a7dba:::benchmark_data_smithi122_119671_object15991:head# 2024-03-20T09:04:59.721 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.721 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b55b9c1d:::benchmark_data_smithi122_119671_object56512:head# 2024-03-20T09:04:59.721 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.721 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b55cc7c1:::benchmark_data_smithi122_119671_object8975:head# 2024-03-20T09:04:59.721 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.722 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b55cd232:::benchmark_data_smithi122_119671_object43244:head# 2024-03-20T09:04:59.722 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.722 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b55d1876:::benchmark_data_smithi122_119671_object53919:head# 2024-03-20T09:04:59.722 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.722 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b55d40d8:::benchmark_data_smithi122_119671_object77790:head# 2024-03-20T09:04:59.722 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.722 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b55d7f21:::benchmark_data_smithi122_119671_object71126:head# 2024-03-20T09:04:59.722 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.722 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b55dfce4:::benchmark_data_smithi122_119671_object18168:head# 2024-03-20T09:04:59.722 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.722 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b55e569d:::benchmark_data_smithi122_119671_object75003:head# 2024-03-20T09:04:59.722 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.723 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b55f077b:::benchmark_data_smithi122_119671_object52638:head# 2024-03-20T09:04:59.723 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.723 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b55f7075:::benchmark_data_smithi122_119671_object34306:head# 2024-03-20T09:04:59.723 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.723 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b55f7c5c:::benchmark_data_smithi122_119671_object85621:head# 2024-03-20T09:04:59.723 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.723 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b55f86f3:::benchmark_data_smithi122_119671_object94758:head# 2024-03-20T09:04:59.723 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.723 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b55fbf5b:::benchmark_data_smithi122_119671_object44934:head# 2024-03-20T09:04:59.723 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.723 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b55fdb19:::benchmark_data_smithi122_119671_object64804:head# 2024-03-20T09:04:59.723 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.724 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5615a7f:::benchmark_data_smithi122_119671_object73142:head# 2024-03-20T09:04:59.724 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.724 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b56207b0:::benchmark_data_smithi122_119671_object55627:head# 2024-03-20T09:04:59.724 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.724 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5625664:::benchmark_data_smithi122_119671_object60468:head# 2024-03-20T09:04:59.724 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.724 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b562842e:::benchmark_data_smithi122_119671_object89178:head# 2024-03-20T09:04:59.724 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.724 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5645813:::benchmark_data_smithi122_119671_object9321:head# 2024-03-20T09:04:59.724 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.724 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5686717:::benchmark_data_smithi122_119671_object69607:head# 2024-03-20T09:04:59.725 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.725 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5689d84:::benchmark_data_smithi122_119671_object37931:head# 2024-03-20T09:04:59.725 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.725 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b568d2fc:::benchmark_data_smithi122_119671_object12459:head# 2024-03-20T09:04:59.725 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.725 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b56933cf:::benchmark_data_smithi122_119671_object64774:head# 2024-03-20T09:04:59.725 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.725 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b56a0052:::benchmark_data_smithi122_119671_object65088:head# 2024-03-20T09:04:59.725 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.725 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b56b36f9:::benchmark_data_smithi122_119671_object78283:head# 2024-03-20T09:04:59.725 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.725 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b56be2da:::benchmark_data_smithi122_119671_object30306:head# 2024-03-20T09:04:59.726 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.726 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b56ca53b:::benchmark_data_smithi122_119671_object47109:head# 2024-03-20T09:04:59.726 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.726 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b56cc376:::benchmark_data_smithi122_119671_object48776:head# 2024-03-20T09:04:59.726 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.726 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b56cf064:::benchmark_data_smithi122_119671_object10504:head# 2024-03-20T09:04:59.726 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.726 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b56e7b22:::benchmark_data_smithi122_119671_object56174:head# 2024-03-20T09:04:59.726 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.726 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b56f407d:::benchmark_data_smithi122_119671_object51069:head# 2024-03-20T09:04:59.726 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.727 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b56fd7c1:::benchmark_data_smithi122_119671_object27860:head# 2024-03-20T09:04:59.727 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.727 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b571673d:::benchmark_data_smithi122_119671_object64829:head# 2024-03-20T09:04:59.727 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.727 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b571a65e:::benchmark_data_smithi122_119671_object76867:head# 2024-03-20T09:04:59.727 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.727 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b571b08b:::benchmark_data_smithi122_119671_object36162:head# 2024-03-20T09:04:59.727 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.727 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b571f948:::benchmark_data_smithi122_119671_object74239:head# 2024-03-20T09:04:59.727 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.727 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b572318d:::benchmark_data_smithi122_119671_object36227:head# 2024-03-20T09:04:59.727 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.728 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5731d2e:::benchmark_data_smithi122_119671_object5687:head# 2024-03-20T09:04:59.728 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.728 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5751b50:::benchmark_data_smithi122_119671_object65399:head# 2024-03-20T09:04:59.728 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.728 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5757fea:::benchmark_data_smithi122_119671_object75770:head# 2024-03-20T09:04:59.728 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.728 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5759a5a:::benchmark_data_smithi122_119671_object44920:head# 2024-03-20T09:04:59.728 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.728 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5765564:::benchmark_data_smithi122_119671_object52017:head# 2024-03-20T09:04:59.728 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.728 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5773aeb:::benchmark_data_smithi122_119671_object83272:head# 2024-03-20T09:04:59.729 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.729 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b57875b0:::benchmark_data_smithi122_119671_object18553:head# 2024-03-20T09:04:59.729 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.729 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b579a71e:::benchmark_data_smithi122_119671_object74237:head# 2024-03-20T09:04:59.729 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.729 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b57b7f2a:::benchmark_data_smithi122_119671_object99222:head# 2024-03-20T09:04:59.729 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.729 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b57bddfc:::benchmark_data_smithi122_119671_object15281:head# 2024-03-20T09:04:59.729 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.729 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b57caf40:::benchmark_data_smithi122_119671_object91971:head# 2024-03-20T09:04:59.729 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.729 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b57e264c:::benchmark_data_smithi122_119671_object18749:head# 2024-03-20T09:04:59.730 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.730 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b57e7174:::benchmark_data_smithi122_119671_object93406:head# 2024-03-20T09:04:59.730 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.730 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b57eee26:::benchmark_data_smithi122_119671_object56439:head# 2024-03-20T09:04:59.730 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.730 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b57f1b68:::benchmark_data_smithi122_119671_object33242:head# 2024-03-20T09:04:59.730 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.730 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b57f417f:::benchmark_data_smithi122_119671_object24101:head# 2024-03-20T09:04:59.730 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.730 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5804184:::benchmark_data_smithi122_119671_object35938:head# 2024-03-20T09:04:59.730 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.731 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b580a075:::benchmark_data_smithi122_119671_object85471:head# 2024-03-20T09:04:59.731 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.731 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b58167fe:::benchmark_data_smithi122_119671_object97616:head# 2024-03-20T09:04:59.731 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.731 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b581b187:::benchmark_data_smithi122_119671_object99326:head# 2024-03-20T09:04:59.731 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.731 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b581f9e7:::benchmark_data_smithi122_119671_object52499:head# 2024-03-20T09:04:59.731 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.731 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5824f27:::benchmark_data_smithi122_119671_object17046:head# 2024-03-20T09:04:59.731 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.731 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5826053:::benchmark_data_smithi122_119671_object91140:head# 2024-03-20T09:04:59.732 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.732 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5829b74:::benchmark_data_smithi122_119671_object19780:head# 2024-03-20T09:04:59.732 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.732 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b582d336:::benchmark_data_smithi122_119671_object68805:head# 2024-03-20T09:04:59.732 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.732 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b583cc5a:::benchmark_data_smithi122_119671_object61983:head# 2024-03-20T09:04:59.732 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.732 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b583f9fa:::benchmark_data_smithi122_119671_object74171:head# 2024-03-20T09:04:59.732 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.732 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5847f74:::benchmark_data_smithi122_119671_object81808:head# 2024-03-20T09:04:59.732 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.732 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5848ebf:::benchmark_data_smithi122_119671_object97480:head# 2024-03-20T09:04:59.733 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.733 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b586a3bd:::benchmark_data_smithi122_119671_object41905:head# 2024-03-20T09:04:59.733 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.733 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b58812ee:::benchmark_data_smithi122_119671_object53416:head# 2024-03-20T09:04:59.733 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.733 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5886304:::benchmark_data_smithi122_119671_object67400:head# 2024-03-20T09:04:59.733 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.733 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5894d83:::benchmark_data_smithi122_119671_object81756:head# 2024-03-20T09:04:59.733 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.733 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5896369:::benchmark_data_smithi122_119671_object49210:head# 2024-03-20T09:04:59.734 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.734 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b589b7dd:::benchmark_data_smithi122_119671_object99744:head# 2024-03-20T09:04:59.734 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.734 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b589c534:::benchmark_data_smithi122_119671_object84266:head# 2024-03-20T09:04:59.734 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.734 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b58c0362:::benchmark_data_smithi122_119671_object9059:head# 2024-03-20T09:04:59.734 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.734 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b58fbdae:::benchmark_data_smithi122_119671_object91698:head# 2024-03-20T09:04:59.734 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.734 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5901cb8:::benchmark_data_smithi122_119671_object13553:head# 2024-03-20T09:04:59.734 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.734 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5906662:::benchmark_data_smithi122_119671_object6274:head# 2024-03-20T09:04:59.734 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.735 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b590bd1a:::benchmark_data_smithi122_119671_object11347:head# 2024-03-20T09:04:59.735 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.735 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b590dd48:::benchmark_data_smithi122_119671_object51068:head# 2024-03-20T09:04:59.735 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.735 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b591f103:::benchmark_data_smithi122_119671_object84653:head# 2024-03-20T09:04:59.735 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.735 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b592a88e:::benchmark_data_smithi122_119671_object12169:head# 2024-03-20T09:04:59.735 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.735 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b59330f5:::benchmark_data_smithi122_119671_object64642:head# 2024-03-20T09:04:59.735 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.735 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5950c58:::benchmark_data_smithi122_119671_object83548:head# 2024-03-20T09:04:59.735 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.736 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b59542f9:::benchmark_data_smithi122_119671_object22629:head# 2024-03-20T09:04:59.736 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.736 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5960d1a:::benchmark_data_smithi122_119671_object87044:head# 2024-03-20T09:04:59.736 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.736 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b596f078:::benchmark_data_smithi122_119671_object30968:head# 2024-03-20T09:04:59.736 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.736 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5970b4f:::benchmark_data_smithi122_119671_object56516:head# 2024-03-20T09:04:59.736 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.736 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5983928:::benchmark_data_smithi122_119671_object16288:head# 2024-03-20T09:04:59.736 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.736 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5989baf:::benchmark_data_smithi122_119671_object26226:head# 2024-03-20T09:04:59.737 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.737 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b598ddaa:::benchmark_data_smithi122_119671_object64062:head# 2024-03-20T09:04:59.737 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.737 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b59903e8:::benchmark_data_smithi122_119671_object88707:head# 2024-03-20T09:04:59.737 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.737 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b599290a:::benchmark_data_smithi122_119671_object91264:head# 2024-03-20T09:04:59.737 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.737 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b59a693f:::benchmark_data_smithi122_119671_object84338:head# 2024-03-20T09:04:59.737 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.737 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b59b1b29:::benchmark_data_smithi122_119671_object30436:head# 2024-03-20T09:04:59.737 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.738 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b59b32e5:::benchmark_data_smithi122_119671_object49777:head# 2024-03-20T09:04:59.738 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.738 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b59b5f57:::benchmark_data_smithi122_119671_object76977:head# 2024-03-20T09:04:59.738 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.738 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b59c89e4:::benchmark_data_smithi122_119671_object42411:head# 2024-03-20T09:04:59.738 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.738 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b59d2478:::benchmark_data_smithi122_119671_object9820:head# 2024-03-20T09:04:59.738 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.738 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5a1b26f:::benchmark_data_smithi122_119671_object43547:head# 2024-03-20T09:04:59.738 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.738 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5a27932:::benchmark_data_smithi122_119671_object59646:head# 2024-03-20T09:04:59.738 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.739 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5a39766:::benchmark_data_smithi122_119671_object3803:head# 2024-03-20T09:04:59.739 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.739 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5a42b20:::benchmark_data_smithi122_119671_object13117:head# 2024-03-20T09:04:59.739 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.739 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5a461eb:::benchmark_data_smithi122_119671_object9093:head# 2024-03-20T09:04:59.739 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.739 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5a46863:::benchmark_data_smithi122_119671_object6191:head# 2024-03-20T09:04:59.739 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.739 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5a499dc:::benchmark_data_smithi122_119671_object85343:head# 2024-03-20T09:04:59.739 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.739 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5a516c8:::benchmark_data_smithi122_119671_object44178:head# 2024-03-20T09:04:59.739 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.740 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5a64639:::benchmark_data_smithi122_119671_object68070:head# 2024-03-20T09:04:59.740 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.740 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5a6e76c:::benchmark_data_smithi122_119671_object88652:head# 2024-03-20T09:04:59.740 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.740 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5a72657:::benchmark_data_smithi122_119671_object37273:head# 2024-03-20T09:04:59.740 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.740 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5a73e61:::benchmark_data_smithi122_119671_object60648:head# 2024-03-20T09:04:59.740 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.740 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5a85d59:::benchmark_data_smithi122_119671_object89601:head# 2024-03-20T09:04:59.740 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.740 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5a88d77:::benchmark_data_smithi122_119671_object57733:head# 2024-03-20T09:04:59.740 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.741 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5a97a0f:::benchmark_data_smithi122_119671_object63063:head# 2024-03-20T09:04:59.741 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.741 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5aac35c:::benchmark_data_smithi122_119671_object86921:head# 2024-03-20T09:04:59.741 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.741 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5aae135:::benchmark_data_smithi122_119671_object73859:head# 2024-03-20T09:04:59.741 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.741 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5ab458a:::benchmark_data_smithi122_119671_object22566:head# 2024-03-20T09:04:59.741 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.741 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5abb906:::benchmark_data_smithi122_119671_object38854:head# 2024-03-20T09:04:59.741 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.741 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5ac462f:::benchmark_data_smithi122_119671_object21497:head# 2024-03-20T09:04:59.742 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.742 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5ac5345:::benchmark_data_smithi122_119671_object90809:head# 2024-03-20T09:04:59.742 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.742 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5ac9dea:::benchmark_data_smithi122_119671_object49100:head# 2024-03-20T09:04:59.742 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.742 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5ad8550:::benchmark_data_smithi122_119671_object41481:head# 2024-03-20T09:04:59.742 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.742 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5ad932f:::benchmark_data_smithi122_119671_object29921:head# 2024-03-20T09:04:59.742 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.742 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5ad9793:::benchmark_data_smithi122_119671_object66577:head# 2024-03-20T09:04:59.742 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.742 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5ada1cf:::benchmark_data_smithi122_119671_object85374:head# 2024-03-20T09:04:59.743 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.743 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5ae311b:::benchmark_data_smithi122_119671_object96413:head# 2024-03-20T09:04:59.743 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.743 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5b060f5:::benchmark_data_smithi122_119671_object19264:head# 2024-03-20T09:04:59.743 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.743 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5b0afd1:::benchmark_data_smithi122_119671_object6654:head# 2024-03-20T09:04:59.743 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.743 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5b0cfdb:::benchmark_data_smithi122_119671_object11908:head# 2024-03-20T09:04:59.743 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.743 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5b0d6f4:::benchmark_data_smithi122_119671_object75208:head# 2024-03-20T09:04:59.743 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.744 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5b1f71c:::benchmark_data_smithi122_119671_object57424:head# 2024-03-20T09:04:59.744 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.744 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5b21802:::benchmark_data_smithi122_119671_object90539:head# 2024-03-20T09:04:59.744 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.744 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5b21ddb:::benchmark_data_smithi122_119671_object49967:head# 2024-03-20T09:04:59.744 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.744 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5b32afc:::benchmark_data_smithi122_119671_object24764:head# 2024-03-20T09:04:59.744 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.744 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5b32bb1:::benchmark_data_smithi122_119671_object6850:head# 2024-03-20T09:04:59.744 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.744 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5b36d0c:::benchmark_data_smithi122_119671_object86351:head# 2024-03-20T09:04:59.744 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.744 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5b36e75:::benchmark_data_smithi122_119671_object83119:head# 2024-03-20T09:04:59.745 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.745 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5b3acb4:::benchmark_data_smithi122_119671_object30318:head# 2024-03-20T09:04:59.745 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.745 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5b40376:::benchmark_data_smithi122_119671_object81590:head# 2024-03-20T09:04:59.745 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.745 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5b607f7:::benchmark_data_smithi122_119671_object17808:head# 2024-03-20T09:04:59.745 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.745 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5b73e3d:::benchmark_data_smithi122_119671_object64312:head# 2024-03-20T09:04:59.745 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.745 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5b75eb0:::benchmark_data_smithi122_119671_object11843:head# 2024-03-20T09:04:59.745 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.745 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5b828ec:::benchmark_data_smithi122_119671_object32337:head# 2024-03-20T09:04:59.746 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.746 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5b91eee:::benchmark_data_smithi122_119671_object48764:head# 2024-03-20T09:04:59.746 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.746 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5ba1f0c:::benchmark_data_smithi122_119671_object57686:head# 2024-03-20T09:04:59.746 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.746 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5bac81d:::benchmark_data_smithi122_119671_object22808:head# 2024-03-20T09:04:59.746 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.746 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5bacc1b:::benchmark_data_smithi122_119671_object48344:head# 2024-03-20T09:04:59.746 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.746 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5bb0bb0:::benchmark_data_smithi122_119671_object11076:head# 2024-03-20T09:04:59.746 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.747 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5bb7a27:::benchmark_data_smithi122_119671_object60266:head# 2024-03-20T09:04:59.747 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.747 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5bc5753:::benchmark_data_smithi122_119671_object20800:head# 2024-03-20T09:04:59.747 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.747 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5bc8262:::benchmark_data_smithi122_119671_object26822:head# 2024-03-20T09:04:59.747 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.747 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5bd4238:::benchmark_data_smithi122_119671_object63871:head# 2024-03-20T09:04:59.747 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.747 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5bdc41f:::benchmark_data_smithi122_119671_object34853:head# 2024-03-20T09:04:59.747 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.747 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5becf51:::benchmark_data_smithi122_119671_object98496:head# 2024-03-20T09:04:59.747 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.748 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5bf2392:::benchmark_data_smithi122_119671_object61000:head# 2024-03-20T09:04:59.748 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.748 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5c0c3fb:::benchmark_data_smithi122_119671_object13289:head# 2024-03-20T09:04:59.748 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.748 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5c137dc:::benchmark_data_smithi122_119671_object1771:head# 2024-03-20T09:04:59.748 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.748 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5c21e7b:::benchmark_data_smithi122_119671_object90351:head# 2024-03-20T09:04:59.748 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.748 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5c2475a:::benchmark_data_smithi122_119671_object33565:head# 2024-03-20T09:04:59.748 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.748 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5c2ad17:::benchmark_data_smithi122_119671_object82924:head# 2024-03-20T09:04:59.748 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.749 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5c51456:::benchmark_data_smithi122_119671_object80812:head# 2024-03-20T09:04:59.749 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.749 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5c52c78:::benchmark_data_smithi122_119671_object53736:head# 2024-03-20T09:04:59.749 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.749 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5c52e04:::benchmark_data_smithi122_119671_object58957:head# 2024-03-20T09:04:59.749 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.749 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5c60f75:::benchmark_data_smithi122_119671_object41306:head# 2024-03-20T09:04:59.749 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.749 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5c61fa1:::benchmark_data_smithi122_119671_object2700:head# 2024-03-20T09:04:59.749 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.749 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5c70f1a:::benchmark_data_smithi122_119671_object47087:head# 2024-03-20T09:04:59.749 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.750 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5c81492:::benchmark_data_smithi122_119671_object9382:head# 2024-03-20T09:04:59.750 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.750 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5c846ba:::benchmark_data_smithi122_119671_object46212:head# 2024-03-20T09:04:59.750 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.750 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5c86702:::benchmark_data_smithi122_119671_object60093:head# 2024-03-20T09:04:59.750 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.750 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5c88b59:::benchmark_data_smithi122_119671_object45830:head# 2024-03-20T09:04:59.750 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.750 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5c8bf45:::benchmark_data_smithi122_119671_object83359:head# 2024-03-20T09:04:59.750 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.750 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5c91992:::benchmark_data_smithi122_119671_object24265:head# 2024-03-20T09:04:59.750 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.751 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5ca416b:::benchmark_data_smithi122_119671_object90987:head# 2024-03-20T09:04:59.751 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.751 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5caafa3:::benchmark_data_smithi122_119671_object91199:head# 2024-03-20T09:04:59.751 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.751 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5cd5769:::benchmark_data_smithi122_119671_object53138:head# 2024-03-20T09:04:59.751 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.751 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5cdb171:::benchmark_data_smithi122_119671_object55950:head# 2024-03-20T09:04:59.751 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.751 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5ce3f54:::benchmark_data_smithi122_119671_object54338:head# 2024-03-20T09:04:59.751 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.751 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5ce9bea:::benchmark_data_smithi122_119671_object55669:head# 2024-03-20T09:04:59.751 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.751 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5cf544d:::benchmark_data_smithi122_119671_object31447:head# 2024-03-20T09:04:59.752 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.752 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5cfa94e:::benchmark_data_smithi122_119671_object35955:head# 2024-03-20T09:04:59.752 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.752 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5cfc6c6:::benchmark_data_smithi122_119671_object17948:head# 2024-03-20T09:04:59.752 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.752 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5cffc77:::benchmark_data_smithi122_119671_object43763:head# 2024-03-20T09:04:59.752 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.752 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5d1541d:::benchmark_data_smithi122_119671_object97256:head# 2024-03-20T09:04:59.752 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.752 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5d1f7bc:::benchmark_data_smithi122_119671_object10263:head# 2024-03-20T09:04:59.752 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.752 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5d26c11:::benchmark_data_smithi122_119671_object55588:head# 2024-03-20T09:04:59.753 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.753 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5d2e9e7:::benchmark_data_smithi122_119671_object72878:head# 2024-03-20T09:04:59.753 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.753 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5d34eff:::benchmark_data_smithi122_119671_object66103:head# 2024-03-20T09:04:59.753 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.753 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5d3d8ec:::benchmark_data_smithi122_119671_object31530:head# 2024-03-20T09:04:59.753 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.753 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5d4b538:::benchmark_data_smithi122_119671_object57171:head# 2024-03-20T09:04:59.753 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.753 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5d4d8b6:::benchmark_data_smithi122_119671_object4218:head# 2024-03-20T09:04:59.753 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.753 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5d4e8b4:::benchmark_data_smithi122_119671_object16368:head# 2024-03-20T09:04:59.754 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.754 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5d56baf:::benchmark_data_smithi122_119671_object65643:head# 2024-03-20T09:04:59.754 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.754 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5d63feb:::benchmark_data_smithi122_119671_object45413:head# 2024-03-20T09:04:59.754 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.754 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5d67535:::benchmark_data_smithi122_119671_object86126:head# 2024-03-20T09:04:59.754 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.754 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5d68643:::benchmark_data_smithi122_119671_object22919:head# 2024-03-20T09:04:59.755 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.755 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5d70c2e:::benchmark_data_smithi122_119671_object66386:head# 2024-03-20T09:04:59.755 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.755 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5d7d271:::benchmark_data_smithi122_119671_object18020:head# 2024-03-20T09:04:59.755 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.755 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5d808e3:::benchmark_data_smithi122_119671_object66593:head# 2024-03-20T09:04:59.755 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.755 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5d88e55:::benchmark_data_smithi122_119671_object13932:head# 2024-03-20T09:04:59.755 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.755 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5d8efb7:::benchmark_data_smithi122_119671_object26995:head# 2024-03-20T09:04:59.755 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.755 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5d921a3:::benchmark_data_smithi122_119671_object87574:head# 2024-03-20T09:04:59.756 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.756 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5d9ea76:::benchmark_data_smithi122_119671_object5086:head# 2024-03-20T09:04:59.756 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.756 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5dd9be6:::benchmark_data_smithi122_119671_object29086:head# 2024-03-20T09:04:59.756 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.756 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5ddfba2:::benchmark_data_smithi122_119671_object71869:head# 2024-03-20T09:04:59.756 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.756 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5df3334:::benchmark_data_smithi122_119671_object17838:head# 2024-03-20T09:04:59.756 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.756 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5df3b4d:::benchmark_data_smithi122_119671_object92684:head# 2024-03-20T09:04:59.756 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.756 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5df4f28:::benchmark_data_smithi122_119671_object81653:head# 2024-03-20T09:04:59.757 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.757 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5e01d99:::benchmark_data_smithi122_119671_object12484:head# 2024-03-20T09:04:59.757 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.757 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5e0f1d5:::benchmark_data_smithi122_119671_object82660:head# 2024-03-20T09:04:59.757 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.757 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5e1bf40:::benchmark_data_smithi122_119671_object88277:head# 2024-03-20T09:04:59.757 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.757 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5e2043a:::benchmark_data_smithi122_119671_object27782:head# 2024-03-20T09:04:59.757 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.757 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5e22a4b:::benchmark_data_smithi122_119671_object6427:head# 2024-03-20T09:04:59.757 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.757 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5e2be8f:::benchmark_data_smithi122_119671_object25151:head# 2024-03-20T09:04:59.758 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.758 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5e359af:::benchmark_data_smithi122_119671_object65785:head# 2024-03-20T09:04:59.758 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.758 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5e360d1:::benchmark_data_smithi122_119671_object76924:head# 2024-03-20T09:04:59.758 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.758 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5e49f31:::benchmark_data_smithi122_119671_object10250:head# 2024-03-20T09:04:59.758 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.758 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5e4e0bc:::benchmark_data_smithi122_119671_object1119:head# 2024-03-20T09:04:59.758 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.758 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5e53f4f:::benchmark_data_smithi122_119671_object77089:head# 2024-03-20T09:04:59.758 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.759 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5e5a78e:::benchmark_data_smithi122_119671_object59573:head# 2024-03-20T09:04:59.759 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.759 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5e64715:::benchmark_data_smithi122_119671_object55445:head# 2024-03-20T09:04:59.759 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.759 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5e66c96:::benchmark_data_smithi122_119671_object86070:head# 2024-03-20T09:04:59.759 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.759 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5e69272:::benchmark_data_smithi122_119671_object4279:head# 2024-03-20T09:04:59.759 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.759 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5e72e4f:::benchmark_data_smithi122_119671_object63573:head# 2024-03-20T09:04:59.759 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.759 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5e73cb9:::benchmark_data_smithi122_119671_object59466:head# 2024-03-20T09:04:59.759 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.760 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5e82d0f:::benchmark_data_smithi122_119671_object43661:head# 2024-03-20T09:04:59.760 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.760 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5e87d59:::benchmark_data_smithi122_119671_object40184:head# 2024-03-20T09:04:59.760 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.760 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5e9031a:::benchmark_data_smithi122_119671_object76860:head# 2024-03-20T09:04:59.760 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.760 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5e9285e:::benchmark_data_smithi122_119671_object46857:head# 2024-03-20T09:04:59.760 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.760 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5e98e4f:::benchmark_data_smithi122_119671_object63473:head# 2024-03-20T09:04:59.760 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.760 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5e9a310:::benchmark_data_smithi122_119671_object39390:head# 2024-03-20T09:04:59.760 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.761 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5e9aa81:::benchmark_data_smithi122_119671_object69447:head# 2024-03-20T09:04:59.761 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.761 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5e9f261:::benchmark_data_smithi122_119671_object53826:head# 2024-03-20T09:04:59.761 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.761 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5e9fa6f:::benchmark_data_smithi122_119671_object84816:head# 2024-03-20T09:04:59.761 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.761 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5eacd91:::benchmark_data_smithi122_119671_object8380:head# 2024-03-20T09:04:59.761 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.761 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5ebc4c9:::benchmark_data_smithi122_119671_object28414:head# 2024-03-20T09:04:59.761 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.761 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5ebcdd2:::benchmark_data_smithi122_119671_object45257:head# 2024-03-20T09:04:59.761 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.762 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5ebf3c0:::benchmark_data_smithi122_119671_object8004:head# 2024-03-20T09:04:59.762 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.762 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5ec4fd9:::benchmark_data_smithi122_119671_object7278:head# 2024-03-20T09:04:59.762 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.762 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5ed547a:::benchmark_data_smithi122_119671_object71598:head# 2024-03-20T09:04:59.762 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.762 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5ed8806:::benchmark_data_smithi122_119671_object91241:head# 2024-03-20T09:04:59.762 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.762 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5edbd01:::benchmark_data_smithi122_119671_object80180:head# 2024-03-20T09:04:59.762 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.762 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5edef39:::benchmark_data_smithi122_119671_object66396:head# 2024-03-20T09:04:59.762 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.763 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5ee16dc:::benchmark_data_smithi122_119671_object57116:head# 2024-03-20T09:04:59.763 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.763 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5ee4be0:::benchmark_data_smithi122_119671_object43475:head# 2024-03-20T09:04:59.763 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.763 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5ef2f18:::benchmark_data_smithi122_119671_object85814:head# 2024-03-20T09:04:59.763 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.763 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5f01905:::benchmark_data_smithi122_119671_object77093:head# 2024-03-20T09:04:59.763 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.763 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5f04d9d:::benchmark_data_smithi122_119671_object81673:head# 2024-03-20T09:04:59.763 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.763 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5f14f39:::benchmark_data_smithi122_119671_object42022:head# 2024-03-20T09:04:59.763 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.763 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5f27b33:::benchmark_data_smithi122_119671_object88162:head# 2024-03-20T09:04:59.764 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.764 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5f3822c:::benchmark_data_smithi122_119671_object35167:head# 2024-03-20T09:04:59.764 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.764 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5f643fd:::benchmark_data_smithi122_119671_object23756:head# 2024-03-20T09:04:59.764 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.764 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5f7c58e:::benchmark_data_smithi122_119671_object92975:head# 2024-03-20T09:04:59.764 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.764 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5f7ca81:::benchmark_data_smithi122_119671_object16870:head# 2024-03-20T09:04:59.764 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.764 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5f8f94c:::benchmark_data_smithi122_119671_object39292:head# 2024-03-20T09:04:59.764 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.764 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5f97787:::benchmark_data_smithi122_119671_object7560:head# 2024-03-20T09:04:59.765 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.765 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5f9b957:::benchmark_data_smithi122_119671_object81941:head# 2024-03-20T09:04:59.765 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.765 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5fa786a:::benchmark_data_smithi122_119671_object50442:head# 2024-03-20T09:04:59.765 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.765 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5fbd960:::benchmark_data_smithi122_119671_object64113:head# 2024-03-20T09:04:59.765 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.765 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5fd9798:::benchmark_data_smithi122_119671_object75249:head# 2024-03-20T09:04:59.765 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.765 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5fed19a:::benchmark_data_smithi122_119671_object77836:head# 2024-03-20T09:04:59.765 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.765 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5ff3eed:::benchmark_data_smithi122_119671_object54353:head# 2024-03-20T09:04:59.765 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.766 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5ff5ac7:::benchmark_data_smithi122_119671_object15681:head# 2024-03-20T09:04:59.766 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.766 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b5ffeae3:::benchmark_data_smithi122_119671_object82873:head# 2024-03-20T09:04:59.766 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.766 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6011054:::benchmark_data_smithi122_119671_object52055:head# 2024-03-20T09:04:59.766 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.766 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6011c08:::benchmark_data_smithi122_119671_object16566:head# 2024-03-20T09:04:59.766 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.766 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b603129f:::benchmark_data_smithi122_119671_object75785:head# 2024-03-20T09:04:59.766 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.766 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6039258:::benchmark_data_smithi122_119671_object47006:head# 2024-03-20T09:04:59.766 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.767 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b60588c7:::benchmark_data_smithi122_119671_object21455:head# 2024-03-20T09:04:59.767 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.767 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b605b223:::benchmark_data_smithi122_119671_object62891:head# 2024-03-20T09:04:59.767 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.767 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b605d428:::benchmark_data_smithi122_119671_object31562:head# 2024-03-20T09:04:59.767 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.767 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6066da2:::benchmark_data_smithi122_119671_object65606:head# 2024-03-20T09:04:59.767 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.767 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6068276:::benchmark_data_smithi122_119671_object26605:head# 2024-03-20T09:04:59.767 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.767 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b606bc02:::benchmark_data_smithi122_119671_object21396:head# 2024-03-20T09:04:59.767 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.768 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b607f63d:::benchmark_data_smithi122_119671_object95098:head# 2024-03-20T09:04:59.768 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.768 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b608b61c:::benchmark_data_smithi122_119671_object32352:head# 2024-03-20T09:04:59.768 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.768 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b609c036:::benchmark_data_smithi122_119671_object83220:head# 2024-03-20T09:04:59.768 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.768 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b60ccc37:::benchmark_data_smithi122_119671_object95071:head# 2024-03-20T09:04:59.768 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.768 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b60d0189:::benchmark_data_smithi122_119671_object26977:head# 2024-03-20T09:04:59.768 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.768 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b60d0566:::benchmark_data_smithi122_119671_object73081:head# 2024-03-20T09:04:59.768 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.768 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b60d244c:::benchmark_data_smithi122_119671_object28858:head# 2024-03-20T09:04:59.769 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.769 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b60f951b:::benchmark_data_smithi122_119671_object57170:head# 2024-03-20T09:04:59.769 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.769 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b60fbbce:::benchmark_data_smithi122_119671_object35905:head# 2024-03-20T09:04:59.769 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.769 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6101524:::benchmark_data_smithi122_119671_object79617:head# 2024-03-20T09:04:59.769 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.769 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b611af16:::benchmark_data_smithi122_119671_object52843:head# 2024-03-20T09:04:59.769 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.769 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b612aecd:::benchmark_data_smithi122_119671_object17995:head# 2024-03-20T09:04:59.769 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.769 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b613f003:::benchmark_data_smithi122_119671_object79060:head# 2024-03-20T09:04:59.770 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.770 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b61404b7:::benchmark_data_smithi122_119671_object65965:head# 2024-03-20T09:04:59.770 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.770 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6146ba7:::benchmark_data_smithi122_119671_object63770:head# 2024-03-20T09:04:59.770 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.770 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6152e76:::benchmark_data_smithi122_119671_object60440:head# 2024-03-20T09:04:59.770 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.770 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b615a272:::benchmark_data_smithi122_119671_object66250:head# 2024-03-20T09:04:59.770 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.770 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b61727b0:::benchmark_data_smithi122_119671_object13700:head# 2024-03-20T09:04:59.770 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.771 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6182bc0:::benchmark_data_smithi122_119671_object88520:head# 2024-03-20T09:04:59.771 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.771 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b618c8ef:::benchmark_data_smithi122_119671_object49862:head# 2024-03-20T09:04:59.771 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.771 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b618e237:::benchmark_data_smithi122_119671_object83673:head# 2024-03-20T09:04:59.771 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.771 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b618e987:::benchmark_data_smithi122_119671_object90383:head# 2024-03-20T09:04:59.771 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.771 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b619908c:::benchmark_data_smithi122_119671_object38913:head# 2024-03-20T09:04:59.771 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.771 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b619b06d:::benchmark_data_smithi122_119671_object66972:head# 2024-03-20T09:04:59.772 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.772 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b619cdd0:::benchmark_data_smithi122_119671_object82480:head# 2024-03-20T09:04:59.772 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.772 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b61a15e9:::benchmark_data_smithi122_119671_object7551:head# 2024-03-20T09:04:59.772 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.772 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b61a21fc:::benchmark_data_smithi122_119671_object47499:head# 2024-03-20T09:04:59.772 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.772 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b61b8f83:::benchmark_data_smithi122_119671_object19470:head# 2024-03-20T09:04:59.772 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.772 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b61c7aea:::benchmark_data_smithi122_119671_object10563:head# 2024-03-20T09:04:59.772 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.772 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b61d2bb1:::benchmark_data_smithi122_119671_object65404:head# 2024-03-20T09:04:59.773 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.773 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b61e8c3b:::benchmark_data_smithi122_119671_object82789:head# 2024-03-20T09:04:59.773 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.773 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b61f2e23:::benchmark_data_smithi122_119671_object29808:head# 2024-03-20T09:04:59.773 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.773 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b61f42b8:::benchmark_data_smithi122_119671_object26959:head# 2024-03-20T09:04:59.773 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.773 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b61f6b2c:::benchmark_data_smithi122_119671_object34234:head# 2024-03-20T09:04:59.773 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.773 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b61f7c5c:::benchmark_data_smithi122_119671_object48775:head# 2024-03-20T09:04:59.773 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.774 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b61ff6d4:::benchmark_data_smithi122_119671_object67647:head# 2024-03-20T09:04:59.774 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.774 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6206bb5:::benchmark_data_smithi122_119671_object50750:head# 2024-03-20T09:04:59.774 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.774 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b621ca5d:::benchmark_data_smithi122_119671_object56795:head# 2024-03-20T09:04:59.774 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.774 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b621f406:::benchmark_data_smithi122_119671_object28931:head# 2024-03-20T09:04:59.774 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.774 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b62272e1:::benchmark_data_smithi122_119671_object40229:head# 2024-03-20T09:04:59.774 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.774 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6245765:::benchmark_data_smithi122_119671_object67892:head# 2024-03-20T09:04:59.774 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.775 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6250943:::benchmark_data_smithi122_119671_object67307:head# 2024-03-20T09:04:59.775 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.775 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b625fe73:::benchmark_data_smithi122_119671_object88076:head# 2024-03-20T09:04:59.775 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.775 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6282d8e:::benchmark_data_smithi122_119671_object53308:head# 2024-03-20T09:04:59.775 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.775 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b628495b:::benchmark_data_smithi122_119671_object95402:head# 2024-03-20T09:04:59.775 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.775 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b62864aa:::benchmark_data_smithi122_119671_object60574:head# 2024-03-20T09:04:59.775 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.775 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b629ddd9:::benchmark_data_smithi122_119671_object18645:head# 2024-03-20T09:04:59.776 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.776 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b62b18db:::benchmark_data_smithi122_119671_object96985:head# 2024-03-20T09:04:59.776 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.776 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b62bd262:::benchmark_data_smithi122_119671_object79714:head# 2024-03-20T09:04:59.776 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.776 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b62c9131:::benchmark_data_smithi122_119671_object52688:head# 2024-03-20T09:04:59.776 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.776 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b62ca02e:::benchmark_data_smithi122_119671_object59167:head# 2024-03-20T09:04:59.776 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.776 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b62d36c8:::benchmark_data_smithi122_119671_object8428:head# 2024-03-20T09:04:59.776 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.776 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b62e819b:::benchmark_data_smithi122_119671_object51863:head# 2024-03-20T09:04:59.777 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.777 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b62eecdf:::benchmark_data_smithi122_119671_object95219:head# 2024-03-20T09:04:59.777 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.777 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b62fc818:::benchmark_data_smithi122_119671_object62439:head# 2024-03-20T09:04:59.777 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.777 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6308e53:::benchmark_data_smithi122_119671_object29076:head# 2024-03-20T09:04:59.777 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.777 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b630abc8:::benchmark_data_smithi122_119671_object92731:head# 2024-03-20T09:04:59.777 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.777 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b630ec04:::benchmark_data_smithi122_119671_object60356:head# 2024-03-20T09:04:59.777 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.777 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b632ca04:::benchmark_data_smithi122_119671_object91937:head# 2024-03-20T09:04:59.778 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.778 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6343677:::benchmark_data_smithi122_119671_object83100:head# 2024-03-20T09:04:59.778 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.778 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b634473f:::benchmark_data_smithi122_119671_object78612:head# 2024-03-20T09:04:59.778 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.778 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6346a75:::benchmark_data_smithi122_119671_object69892:head# 2024-03-20T09:04:59.778 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.778 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6348385:::benchmark_data_smithi122_119671_object72359:head# 2024-03-20T09:04:59.778 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.778 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6350464:::benchmark_data_smithi122_119671_object27504:head# 2024-03-20T09:04:59.778 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.778 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6355441:::benchmark_data_smithi122_119671_object56633:head# 2024-03-20T09:04:59.779 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.779 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6362f40:::benchmark_data_smithi122_119671_object55436:head# 2024-03-20T09:04:59.779 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.779 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b63867a2:::benchmark_data_smithi122_119671_object68494:head# 2024-03-20T09:04:59.779 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.779 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b639f9de:::benchmark_data_smithi122_119671_object35018:head# 2024-03-20T09:04:59.779 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.779 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b63a1439:::benchmark_data_smithi122_119671_object78956:head# 2024-03-20T09:04:59.779 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.779 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b63a8a0b:::benchmark_data_smithi122_119671_object11869:head# 2024-03-20T09:04:59.779 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.779 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b63b0d6d:::benchmark_data_smithi122_119671_object80174:head# 2024-03-20T09:04:59.780 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.780 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b63b2b6c:::benchmark_data_smithi122_119671_object70074:head# 2024-03-20T09:04:59.780 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.780 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b63bec39:::benchmark_data_smithi122_119671_object7014:head# 2024-03-20T09:04:59.780 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.780 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b63d0b89:::benchmark_data_smithi122_119671_object98448:head# 2024-03-20T09:04:59.780 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.780 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b63d657b:::benchmark_data_smithi122_119671_object53406:head# 2024-03-20T09:04:59.780 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.780 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b63fc100:::benchmark_data_smithi122_119671_object14437:head# 2024-03-20T09:04:59.780 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.780 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6406233:::benchmark_data_smithi122_119671_object79403:head# 2024-03-20T09:04:59.781 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.781 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6412aea:::benchmark_data_smithi122_119671_object58959:head# 2024-03-20T09:04:59.781 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.781 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b641813b:::benchmark_data_smithi122_119671_object96891:head# 2024-03-20T09:04:59.781 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.781 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b641a352:::benchmark_data_smithi122_119671_object69706:head# 2024-03-20T09:04:59.781 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.781 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6429026:::benchmark_data_smithi122_119671_object57152:head# 2024-03-20T09:04:59.781 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.781 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b642cac2:::benchmark_data_smithi122_119671_object59644:head# 2024-03-20T09:04:59.781 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.781 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b643199e:::benchmark_data_smithi122_119671_object3111:head# 2024-03-20T09:04:59.782 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.782 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6439e7f:::benchmark_data_smithi122_119671_object19716:head# 2024-03-20T09:04:59.782 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.782 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b643bfee:::benchmark_data_smithi122_119671_object33949:head# 2024-03-20T09:04:59.782 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.782 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b645fdd0:::benchmark_data_smithi122_119671_object17194:head# 2024-03-20T09:04:59.782 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.782 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6473168:::benchmark_data_smithi122_119671_object31623:head# 2024-03-20T09:04:59.782 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.782 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b64775b7:::benchmark_data_smithi122_119671_object7725:head# 2024-03-20T09:04:59.782 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.782 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b647f1d7:::benchmark_data_smithi122_119671_object96231:head# 2024-03-20T09:04:59.783 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.783 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6487364:::benchmark_data_smithi122_119671_object46692:head# 2024-03-20T09:04:59.783 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.783 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b648d189:::benchmark_data_smithi122_119671_object80572:head# 2024-03-20T09:04:59.783 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.783 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b64966e5:::benchmark_data_smithi122_119671_object93087:head# 2024-03-20T09:04:59.783 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.783 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b64a4f88:::benchmark_data_smithi122_119671_object94833:head# 2024-03-20T09:04:59.783 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.783 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b64ac886:::benchmark_data_smithi122_119671_object83395:head# 2024-03-20T09:04:59.785 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.785 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b64b1dfd:::benchmark_data_smithi122_119671_object9564:head# 2024-03-20T09:04:59.785 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.786 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b64b5e3a:::benchmark_data_smithi122_119671_object86738:head# 2024-03-20T09:04:59.786 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.786 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b64c57fa:::benchmark_data_smithi122_119671_object85754:head# 2024-03-20T09:04:59.786 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.786 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b64de769:::benchmark_data_smithi122_119671_object91397:head# 2024-03-20T09:04:59.786 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.786 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b64e3edb:::benchmark_data_smithi122_119671_object41424:head# 2024-03-20T09:04:59.786 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.786 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b64fed01:::benchmark_data_smithi122_119671_object65456:head# 2024-03-20T09:04:59.786 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.786 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b650336b:::benchmark_data_smithi122_119671_object18755:head# 2024-03-20T09:04:59.786 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.787 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b65060fb:::benchmark_data_smithi122_119671_object34479:head# 2024-03-20T09:04:59.787 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.787 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6512ef1:::benchmark_data_smithi122_119671_object62779:head# 2024-03-20T09:04:59.787 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.787 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b65161d9:::benchmark_data_smithi122_119671_object64403:head# 2024-03-20T09:04:59.787 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.787 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b652589e:::benchmark_data_smithi122_119671_object36506:head# 2024-03-20T09:04:59.787 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.788 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6526170:::benchmark_data_smithi122_119671_object71322:head# 2024-03-20T09:04:59.788 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.788 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b652a48f:::benchmark_data_smithi122_119671_object70819:head# 2024-03-20T09:04:59.788 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.788 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6540eed:::benchmark_data_smithi122_119671_object61529:head# 2024-03-20T09:04:59.788 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.788 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6542375:::benchmark_data_smithi122_119671_object39393:head# 2024-03-20T09:04:59.788 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.788 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6544cd7:::benchmark_data_smithi122_119671_object24061:head# 2024-03-20T09:04:59.788 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.788 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b65456ae:::benchmark_data_smithi122_119671_object7261:head# 2024-03-20T09:04:59.788 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.789 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6556b9d:::benchmark_data_smithi122_119671_object58120:head# 2024-03-20T09:04:59.789 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.789 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6581863:::benchmark_data_smithi122_119671_object81577:head# 2024-03-20T09:04:59.789 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.789 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b658ed87:::benchmark_data_smithi122_119671_object93236:head# 2024-03-20T09:04:59.789 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.789 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b659b2af:::benchmark_data_smithi122_119671_object16333:head# 2024-03-20T09:04:59.789 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.789 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b659da76:::benchmark_data_smithi122_119671_object95383:head# 2024-03-20T09:04:59.789 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.789 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b65af8d1:::benchmark_data_smithi122_119671_object47798:head# 2024-03-20T09:04:59.789 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.790 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b65bf206:::benchmark_data_smithi122_119671_object58390:head# 2024-03-20T09:04:59.790 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.790 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b65c0dfd:::benchmark_data_smithi122_119671_object86211:head# 2024-03-20T09:04:59.790 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.790 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b65c3e48:::benchmark_data_smithi122_119671_object90247:head# 2024-03-20T09:04:59.790 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.790 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b65c8180:::benchmark_data_smithi122_119671_object12101:head# 2024-03-20T09:04:59.790 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.790 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b65d595b:::benchmark_data_smithi122_119671_object65030:head# 2024-03-20T09:04:59.790 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.790 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b65d5ea3:::benchmark_data_smithi122_119671_object72199:head# 2024-03-20T09:04:59.790 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.791 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b65d66be:::benchmark_data_smithi122_119671_object24490:head# 2024-03-20T09:04:59.791 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.791 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b65db49f:::benchmark_data_smithi122_119671_object24549:head# 2024-03-20T09:04:59.791 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.791 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b65ea3bb:::benchmark_data_smithi122_119671_object17007:head# 2024-03-20T09:04:59.791 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.791 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b65ec451:::benchmark_data_smithi122_119671_object66491:head# 2024-03-20T09:04:59.791 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.791 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b65edd75:::benchmark_data_smithi122_119671_object94776:head# 2024-03-20T09:04:59.791 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.791 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b65f843b:::benchmark_data_smithi122_119671_object62185:head# 2024-03-20T09:04:59.791 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.791 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6604524:::benchmark_data_smithi122_119671_object34868:head# 2024-03-20T09:04:59.792 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.792 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b660fbc6:::benchmark_data_smithi122_119671_object96307:head# 2024-03-20T09:04:59.792 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.792 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6621860:::benchmark_data_smithi122_119671_object2726:head# 2024-03-20T09:04:59.792 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.792 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b66220b0:::benchmark_data_smithi122_119671_object39768:head# 2024-03-20T09:04:59.792 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.792 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6629e67:::benchmark_data_smithi122_119671_object65757:head# 2024-03-20T09:04:59.792 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.792 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b663473f:::benchmark_data_smithi122_119671_object75434:head# 2024-03-20T09:04:59.792 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.793 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b66347d2:::benchmark_data_smithi122_119671_object96037:head# 2024-03-20T09:04:59.793 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.793 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b663fe06:::benchmark_data_smithi122_119671_object93881:head# 2024-03-20T09:04:59.793 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.793 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6649020:::benchmark_data_smithi122_119671_object87326:head# 2024-03-20T09:04:59.793 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.793 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b664b66b:::benchmark_data_smithi122_119671_object83902:head# 2024-03-20T09:04:59.793 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.793 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6651862:::benchmark_data_smithi122_119671_object38583:head# 2024-03-20T09:04:59.793 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.793 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b66551c9:::benchmark_data_smithi122_119671_object4990:head# 2024-03-20T09:04:59.793 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.794 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6658e3f:::benchmark_data_smithi122_119671_object27071:head# 2024-03-20T09:04:59.794 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.794 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b665c38a:::benchmark_data_smithi122_119671_object56737:head# 2024-03-20T09:04:59.794 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.794 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6662437:::benchmark_data_smithi122_119671_object32826:head# 2024-03-20T09:04:59.794 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.794 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b666db4d:::benchmark_data_smithi122_119671_object93163:head# 2024-03-20T09:04:59.794 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.794 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6677217:::benchmark_data_smithi122_119671_object37487:head# 2024-03-20T09:04:59.794 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.794 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b667e78a:::benchmark_data_smithi122_119671_object68529:head# 2024-03-20T09:04:59.795 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.795 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b667f43d:::benchmark_data_smithi122_119671_object36829:head# 2024-03-20T09:04:59.795 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.795 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b667f89b:::benchmark_data_smithi122_119671_object79733:head# 2024-03-20T09:04:59.795 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.795 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6688184:::benchmark_data_smithi122_119671_object74203:head# 2024-03-20T09:04:59.795 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.795 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6696332:::benchmark_data_smithi122_119671_object56683:head# 2024-03-20T09:04:59.795 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.795 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b66a898a:::benchmark_data_smithi122_119671_object13783:head# 2024-03-20T09:04:59.795 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.795 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b66c788b:::benchmark_data_smithi122_119671_object48478:head# 2024-03-20T09:04:59.796 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.796 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b66d65e5:::benchmark_data_smithi122_119671_object31428:head# 2024-03-20T09:04:59.796 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.796 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b66e02b1:::benchmark_data_smithi122_119671_object18461:head# 2024-03-20T09:04:59.796 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.796 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b66e9f68:::benchmark_data_smithi122_119671_object1743:head# 2024-03-20T09:04:59.796 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.796 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b66f1d50:::benchmark_data_smithi122_119671_object10218:head# 2024-03-20T09:04:59.796 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.796 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b66f2f90:::benchmark_data_smithi122_119671_object16982:head# 2024-03-20T09:04:59.796 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.796 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b66f3793:::benchmark_data_smithi122_119671_object30244:head# 2024-03-20T09:04:59.797 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.797 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b66f4f45:::benchmark_data_smithi122_119671_object32729:head# 2024-03-20T09:04:59.797 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.797 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b66f93aa:::benchmark_data_smithi122_119671_object83147:head# 2024-03-20T09:04:59.797 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.797 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6711729:::benchmark_data_smithi122_119671_object44675:head# 2024-03-20T09:04:59.797 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.797 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b67178c6:::benchmark_data_smithi122_119671_object61610:head# 2024-03-20T09:04:59.797 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.797 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6748830:::benchmark_data_smithi122_119671_object95855:head# 2024-03-20T09:04:59.797 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.798 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b67521da:::benchmark_data_smithi122_119671_object16342:head# 2024-03-20T09:04:59.798 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.798 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b675948a:::benchmark_data_smithi122_119671_object67704:head# 2024-03-20T09:04:59.798 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.798 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b675d941:::benchmark_data_smithi122_119671_object26159:head# 2024-03-20T09:04:59.798 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.798 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6790c76:::benchmark_data_smithi122_119671_object23206:head# 2024-03-20T09:04:59.798 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.798 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b67b687c:::benchmark_data_smithi122_119671_object31727:head# 2024-03-20T09:04:59.798 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.798 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b67bde42:::benchmark_data_smithi122_119671_object42488:head# 2024-03-20T09:04:59.799 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.799 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b67ccbae:::benchmark_data_smithi122_119671_object68135:head# 2024-03-20T09:04:59.799 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.799 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b67f886d:::benchmark_data_smithi122_119671_object92084:head# 2024-03-20T09:04:59.799 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.799 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6803bbc:::benchmark_data_smithi122_119671_object62574:head# 2024-03-20T09:04:59.799 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.799 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6807780:::benchmark_data_smithi122_119671_object94707:head# 2024-03-20T09:04:59.799 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.799 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6818d66:::benchmark_data_smithi122_119671_object5772:head# 2024-03-20T09:04:59.799 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.799 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b681edcc:::benchmark_data_smithi122_119671_object11742:head# 2024-03-20T09:04:59.799 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.800 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6822dd1:::benchmark_data_smithi122_119671_object26295:head# 2024-03-20T09:04:59.800 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.800 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6827131:::benchmark_data_smithi122_119671_object98492:head# 2024-03-20T09:04:59.800 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.800 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6831441:::benchmark_data_smithi122_119671_object26037:head# 2024-03-20T09:04:59.800 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.800 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b68379ee:::benchmark_data_smithi122_119671_object55230:head# 2024-03-20T09:04:59.800 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.800 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6838d97:::benchmark_data_smithi122_119671_object31900:head# 2024-03-20T09:04:59.800 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.800 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6839736:::benchmark_data_smithi122_119671_object5076:head# 2024-03-20T09:04:59.800 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.801 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b683d59d:::benchmark_data_smithi122_119671_object53337:head# 2024-03-20T09:04:59.801 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.801 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6841ffe:::benchmark_data_smithi122_119671_object12557:head# 2024-03-20T09:04:59.801 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.801 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b684a295:::benchmark_data_smithi122_119671_object53871:head# 2024-03-20T09:04:59.801 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.801 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b685d6b1:::benchmark_data_smithi122_119671_object64395:head# 2024-03-20T09:04:59.801 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.801 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6862f28:::benchmark_data_smithi122_119671_object89864:head# 2024-03-20T09:04:59.801 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.801 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b68797e9:::benchmark_data_smithi122_119671_object87929:head# 2024-03-20T09:04:59.802 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.802 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b687a779:::benchmark_data_smithi122_119671_object68796:head# 2024-03-20T09:04:59.802 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.802 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b687bbc2:::benchmark_data_smithi122_119671_object40383:head# 2024-03-20T09:04:59.802 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.802 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b687eded:::benchmark_data_smithi122_119671_object70983:head# 2024-03-20T09:04:59.802 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.802 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b68920c7:::benchmark_data_smithi122_119671_object31274:head# 2024-03-20T09:04:59.802 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.802 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b689757a:::benchmark_data_smithi122_119671_object3719:head# 2024-03-20T09:04:59.802 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.802 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b68a505d:::benchmark_data_smithi122_119671_object87690:head# 2024-03-20T09:04:59.803 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.803 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b68a6534:::benchmark_data_smithi122_119671_object19322:head# 2024-03-20T09:04:59.803 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.803 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b68adf22:::benchmark_data_smithi122_119671_object73949:head# 2024-03-20T09:04:59.803 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.803 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b68bac2b:::benchmark_data_smithi122_119671_object58780:head# 2024-03-20T09:04:59.803 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.803 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b68bfa7a:::benchmark_data_smithi122_119671_object97241:head# 2024-03-20T09:04:59.803 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.803 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b68ca38b:::benchmark_data_smithi122_119671_object4460:head# 2024-03-20T09:04:59.803 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.804 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b68e1ca3:::benchmark_data_smithi122_119671_object45229:head# 2024-03-20T09:04:59.804 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.804 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b68e9682:::benchmark_data_smithi122_119671_object96190:head# 2024-03-20T09:04:59.804 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.804 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b68ee67d:::benchmark_data_smithi122_119671_object96377:head# 2024-03-20T09:04:59.804 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.804 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b68f5d2d:::benchmark_data_smithi122_119671_object28227:head# 2024-03-20T09:04:59.804 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.804 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6904fc1:::benchmark_data_smithi122_119671_object78475:head# 2024-03-20T09:04:59.804 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.804 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b690d77a:::benchmark_data_smithi122_119671_object48918:head# 2024-03-20T09:04:59.804 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.805 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b691190b:::benchmark_data_smithi122_119671_object95732:head# 2024-03-20T09:04:59.805 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.805 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b69128e4:::benchmark_data_smithi122_119671_object51529:head# 2024-03-20T09:04:59.805 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.805 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6937ecb:::benchmark_data_smithi122_119671_object52742:head# 2024-03-20T09:04:59.805 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.805 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b69453db:::benchmark_data_smithi122_119671_object94769:head# 2024-03-20T09:04:59.805 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.805 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6945750:::benchmark_data_smithi122_119671_object57260:head# 2024-03-20T09:04:59.805 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.805 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b695836c:::benchmark_data_smithi122_119671_object95433:head# 2024-03-20T09:04:59.806 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.806 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6967475:::benchmark_data_smithi122_119671_object78401:head# 2024-03-20T09:04:59.806 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.806 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6970255:::benchmark_data_smithi122_119671_object94040:head# 2024-03-20T09:04:59.806 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.806 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6972935:::benchmark_data_smithi122_119671_object81794:head# 2024-03-20T09:04:59.806 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.806 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b69803be:::benchmark_data_smithi122_119671_object64860:head# 2024-03-20T09:04:59.806 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.806 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b69845ef:::benchmark_data_smithi122_119671_object98069:head# 2024-03-20T09:04:59.806 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.806 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b698ead3:::benchmark_data_smithi122_119671_object10984:head# 2024-03-20T09:04:59.807 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.807 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6990b66:::benchmark_data_smithi122_119671_object68955:head# 2024-03-20T09:04:59.807 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.807 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6991b20:::benchmark_data_smithi122_119671_object73671:head# 2024-03-20T09:04:59.807 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.807 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6994a58:::benchmark_data_smithi122_119671_object80014:head# 2024-03-20T09:04:59.807 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.807 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6996f48:::benchmark_data_smithi122_119671_object5489:head# 2024-03-20T09:04:59.807 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.807 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b69a6d61:::benchmark_data_smithi122_119671_object25745:head# 2024-03-20T09:04:59.807 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.807 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b69a77c6:::benchmark_data_smithi122_119671_object48625:head# 2024-03-20T09:04:59.808 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.808 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b69aec82:::benchmark_data_smithi122_119671_object51347:head# 2024-03-20T09:04:59.808 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.808 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b69af119:::benchmark_data_smithi122_119671_object67757:head# 2024-03-20T09:04:59.808 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.808 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b69b347c:::benchmark_data_smithi122_119671_object35061:head# 2024-03-20T09:04:59.808 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.808 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b69b9f47:::benchmark_data_smithi122_119671_object60571:head# 2024-03-20T09:04:59.808 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.808 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b69bdcf6:::benchmark_data_smithi122_119671_object57439:head# 2024-03-20T09:04:59.808 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.808 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b69c1f31:::benchmark_data_smithi122_119671_object29007:head# 2024-03-20T09:04:59.809 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.809 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b69c3015:::benchmark_data_smithi122_119671_object32073:head# 2024-03-20T09:04:59.809 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.809 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b69cc08b:::benchmark_data_smithi122_119671_object84586:head# 2024-03-20T09:04:59.809 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.809 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b69cc39c:::benchmark_data_smithi122_119671_object7260:head# 2024-03-20T09:04:59.809 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.809 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b69de826:::benchmark_data_smithi122_119671_object97096:head# 2024-03-20T09:04:59.809 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.809 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b69e53e1:::benchmark_data_smithi122_119671_object2815:head# 2024-03-20T09:04:59.809 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.809 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b69f0204:::benchmark_data_smithi122_119671_object72179:head# 2024-03-20T09:04:59.809 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.810 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b69f2a26:::benchmark_data_smithi122_119671_object38157:head# 2024-03-20T09:04:59.810 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.810 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b69ffac9:::benchmark_data_smithi122_119671_object32053:head# 2024-03-20T09:04:59.810 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.810 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6a06066:::benchmark_data_smithi122_119671_object32262:head# 2024-03-20T09:04:59.810 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.810 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6a06798:::benchmark_data_smithi122_119671_object32412:head# 2024-03-20T09:04:59.810 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.810 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6a10b73:::benchmark_data_smithi122_119671_object88485:head# 2024-03-20T09:04:59.810 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.810 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6a12f3b:::benchmark_data_smithi122_119671_object76149:head# 2024-03-20T09:04:59.811 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.811 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6a4201a:::benchmark_data_smithi122_119671_object40497:head# 2024-03-20T09:04:59.811 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.811 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6a486f6:::benchmark_data_smithi122_119671_object31575:head# 2024-03-20T09:04:59.811 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.811 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6a4873e:::benchmark_data_smithi122_119671_object62221:head# 2024-03-20T09:04:59.811 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.811 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6a4e865:::benchmark_data_smithi122_119671_object76934:head# 2024-03-20T09:04:59.811 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.811 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6a55ea6:::benchmark_data_smithi122_119671_object49770:head# 2024-03-20T09:04:59.812 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.812 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6a5630f:::benchmark_data_smithi122_119671_object82684:head# 2024-03-20T09:04:59.812 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.812 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6a62d4f:::benchmark_data_smithi122_119671_object83738:head# 2024-03-20T09:04:59.812 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.813 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6a64d93:::benchmark_data_smithi122_119671_object31701:head# 2024-03-20T09:04:59.813 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.813 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6a72cda:::benchmark_data_smithi122_119671_object56413:head# 2024-03-20T09:04:59.813 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.813 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6a7e67a:::benchmark_data_smithi122_119671_object53203:head# 2024-03-20T09:04:59.813 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.813 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6a98cb5:::benchmark_data_smithi122_119671_object19179:head# 2024-03-20T09:04:59.813 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.813 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6a9c522:::benchmark_data_smithi122_119671_object4151:head# 2024-03-20T09:04:59.814 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.814 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6aab175:::benchmark_data_smithi122_119671_object89144:head# 2024-03-20T09:04:59.814 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.814 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6ad8447:::benchmark_data_smithi122_119671_object15218:head# 2024-03-20T09:04:59.814 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.814 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6adbe03:::benchmark_data_smithi122_119671_object73352:head# 2024-03-20T09:04:59.814 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.814 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6af12ae:::benchmark_data_smithi122_119671_object45139:head# 2024-03-20T09:04:59.814 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.815 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6af62e2:::benchmark_data_smithi122_119671_object55779:head# 2024-03-20T09:04:59.815 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.815 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6b0295c:::benchmark_data_smithi122_119671_object75272:head# 2024-03-20T09:04:59.815 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.815 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6b038fd:::benchmark_data_smithi122_119671_object66727:head# 2024-03-20T09:04:59.815 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.815 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6b1842d:::benchmark_data_smithi122_119671_object6584:head# 2024-03-20T09:04:59.815 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.815 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6b1ba6f:::benchmark_data_smithi122_119671_object17286:head# 2024-03-20T09:04:59.815 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.815 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6b2d164:::benchmark_data_smithi122_119671_object2617:head# 2024-03-20T09:04:59.815 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.815 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6b531e0:::benchmark_data_smithi122_119671_object91376:head# 2024-03-20T09:04:59.815 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.816 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6b69cf4:::benchmark_data_smithi122_119671_object28221:head# 2024-03-20T09:04:59.816 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.816 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6b72bd1:::benchmark_data_smithi122_119671_object28358:head# 2024-03-20T09:04:59.816 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.816 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6b7a8e1:::benchmark_data_smithi122_119671_object91027:head# 2024-03-20T09:04:59.816 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.816 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6b86467:::benchmark_data_smithi122_119671_object51767:head# 2024-03-20T09:04:59.816 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.816 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6b894cf:::benchmark_data_smithi122_119671_object49300:head# 2024-03-20T09:04:59.816 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.816 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6b8c83d:::benchmark_data_smithi122_119671_object3799:head# 2024-03-20T09:04:59.816 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.816 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6b9755f:::benchmark_data_smithi122_119671_object11852:head# 2024-03-20T09:04:59.817 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.817 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6bb4444:::benchmark_data_smithi122_119671_object77959:head# 2024-03-20T09:04:59.817 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.817 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6bbe393:::benchmark_data_smithi122_119671_object52254:head# 2024-03-20T09:04:59.817 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.817 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6bd2f43:::benchmark_data_smithi122_119671_object77487:head# 2024-03-20T09:04:59.817 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.818 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6bdd2b5:::benchmark_data_smithi122_119671_object3257:head# 2024-03-20T09:04:59.818 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.818 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6beb3a8:::benchmark_data_smithi122_119671_object14210:head# 2024-03-20T09:04:59.818 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.818 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6c02e0d:::benchmark_data_smithi122_119671_object15939:head# 2024-03-20T09:04:59.818 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.818 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6c0700f:::benchmark_data_smithi122_119671_object71547:head# 2024-03-20T09:04:59.818 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.818 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6c0a8db:::benchmark_data_smithi122_119671_object45043:head# 2024-03-20T09:04:59.818 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.818 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6c18ed2:::benchmark_data_smithi122_119671_object88955:head# 2024-03-20T09:04:59.819 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.819 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6c1e2fc:::benchmark_data_smithi122_119671_object8856:head# 2024-03-20T09:04:59.819 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.819 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6c24fbc:::benchmark_data_smithi122_119671_object50722:head# 2024-03-20T09:04:59.819 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.819 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6c2a9c7:::benchmark_data_smithi122_119671_object62937:head# 2024-03-20T09:04:59.819 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.819 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6c44794:::benchmark_data_smithi122_119671_object35711:head# 2024-03-20T09:04:59.819 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.819 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6c46534:::benchmark_data_smithi122_119671_object68033:head# 2024-03-20T09:04:59.820 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.820 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6c4e447:::benchmark_data_smithi122_119671_object90927:head# 2024-03-20T09:04:59.820 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.820 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6c582db:::benchmark_data_smithi122_119671_object7242:head# 2024-03-20T09:04:59.820 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.820 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6c72516:::benchmark_data_smithi122_119671_object99243:head# 2024-03-20T09:04:59.820 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.820 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6c88c0f:::benchmark_data_smithi122_119671_object23944:head# 2024-03-20T09:04:59.820 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.820 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6c8af66:::benchmark_data_smithi122_119671_object16691:head# 2024-03-20T09:04:59.820 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.820 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6c9490f:::benchmark_data_smithi122_119671_object4303:head# 2024-03-20T09:04:59.820 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.820 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6c9ef92:::benchmark_data_smithi122_119671_object27914:head# 2024-03-20T09:04:59.821 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.821 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6ca15f7:::benchmark_data_smithi122_119671_object93536:head# 2024-03-20T09:04:59.821 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.821 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6ca9212:::benchmark_data_smithi122_119671_object87488:head# 2024-03-20T09:04:59.821 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.821 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6caaf6c:::benchmark_data_smithi122_119671_object45280:head# 2024-03-20T09:04:59.821 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.821 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6caf9d6:::benchmark_data_smithi122_119671_object34108:head# 2024-03-20T09:04:59.821 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.821 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6cb44e0:::benchmark_data_smithi122_119671_object95202:head# 2024-03-20T09:04:59.821 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.821 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6cb73bb:::benchmark_data_smithi122_119671_object17660:head# 2024-03-20T09:04:59.821 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.822 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6cc330e:::benchmark_data_smithi122_119671_object12864:head# 2024-03-20T09:04:59.822 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.822 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6cc3f52:::benchmark_data_smithi122_119671_object52363:head# 2024-03-20T09:04:59.822 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.822 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6cf1736:::benchmark_data_smithi122_119671_object19207:head# 2024-03-20T09:04:59.822 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.822 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6cf4a04:::benchmark_data_smithi122_119671_object25740:head# 2024-03-20T09:04:59.822 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.822 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6d06756:::benchmark_data_smithi122_119671_object4556:head# 2024-03-20T09:04:59.822 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.822 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6d08541:::benchmark_data_smithi122_119671_object43094:head# 2024-03-20T09:04:59.822 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.822 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6d33290:::benchmark_data_smithi122_119671_object19865:head# 2024-03-20T09:04:59.823 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.823 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6d4fbd2:::benchmark_data_smithi122_119671_object82881:head# 2024-03-20T09:04:59.823 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.823 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6d621d4:::benchmark_data_smithi122_119671_object6146:head# 2024-03-20T09:04:59.823 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.823 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6d79e73:::benchmark_data_smithi122_119671_object60073:head# 2024-03-20T09:04:59.823 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.823 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6d849bc:::benchmark_data_smithi122_119671_object11020:head# 2024-03-20T09:04:59.823 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.823 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6d9b3f2:::benchmark_data_smithi122_119671_object1254:head# 2024-03-20T09:04:59.823 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.823 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6da06ef:::benchmark_data_smithi122_119671_object14017:head# 2024-03-20T09:04:59.823 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.823 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6da2e8e:::benchmark_data_smithi122_119671_object47944:head# 2024-03-20T09:04:59.824 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.824 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6db16ea:::benchmark_data_smithi122_119671_object90198:head# 2024-03-20T09:04:59.824 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.824 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6db265e:::benchmark_data_smithi122_119671_object77054:head# 2024-03-20T09:04:59.824 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.824 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6dbe151:::benchmark_data_smithi122_119671_object37519:head# 2024-03-20T09:04:59.824 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.824 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6de2612:::benchmark_data_smithi122_119671_object13883:head# 2024-03-20T09:04:59.824 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.824 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6df3c0b:::benchmark_data_smithi122_119671_object22602:head# 2024-03-20T09:04:59.824 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.824 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6df6a08:::benchmark_data_smithi122_119671_object54797:head# 2024-03-20T09:04:59.824 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.824 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6e06fbb:::benchmark_data_smithi122_119671_object20933:head# 2024-03-20T09:04:59.825 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.825 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6e0a823:::benchmark_data_smithi122_119671_object5744:head# 2024-03-20T09:04:59.825 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.825 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6e0ec9c:::benchmark_data_smithi122_119671_object48926:head# 2024-03-20T09:04:59.825 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.825 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6e110d5:::benchmark_data_smithi122_119671_object85453:head# 2024-03-20T09:04:59.825 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.825 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6e161c8:::benchmark_data_smithi122_119671_object96529:head# 2024-03-20T09:04:59.825 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.825 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6e1cf4a:::benchmark_data_smithi122_119671_object55059:head# 2024-03-20T09:04:59.825 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.825 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6e2ed66:::benchmark_data_smithi122_119671_object43513:head# 2024-03-20T09:04:59.825 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.826 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6e3b3b3:::benchmark_data_smithi122_119671_object51679:head# 2024-03-20T09:04:59.826 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.826 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6e3b897:::benchmark_data_smithi122_119671_object17043:head# 2024-03-20T09:04:59.826 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.826 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6e3d493:::benchmark_data_smithi122_119671_object48214:head# 2024-03-20T09:04:59.826 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.826 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6e69c6a:::benchmark_data_smithi122_119671_object24660:head# 2024-03-20T09:04:59.826 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.826 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6e7456c:::benchmark_data_smithi122_119671_object75652:head# 2024-03-20T09:04:59.826 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.826 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6e7bcde:::benchmark_data_smithi122_119671_object26808:head# 2024-03-20T09:04:59.826 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.826 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6e7cacc:::benchmark_data_smithi122_119671_object62443:head# 2024-03-20T09:04:59.827 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.827 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6e92074:::benchmark_data_smithi122_119671_object94655:head# 2024-03-20T09:04:59.827 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.827 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6e994ee:::benchmark_data_smithi122_119671_object50602:head# 2024-03-20T09:04:59.827 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.827 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6eaabd5:::benchmark_data_smithi122_119671_object78793:head# 2024-03-20T09:04:59.827 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.827 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6eac363:::benchmark_data_smithi122_119671_object80126:head# 2024-03-20T09:04:59.827 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.827 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6eb30db:::benchmark_data_smithi122_119671_object12686:head# 2024-03-20T09:04:59.827 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.827 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6ec167f:::benchmark_data_smithi122_119671_object77172:head# 2024-03-20T09:04:59.827 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.828 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6ecd543:::benchmark_data_smithi122_119671_object87727:head# 2024-03-20T09:04:59.828 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.828 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6ecdaa6:::benchmark_data_smithi122_119671_object80831:head# 2024-03-20T09:04:59.828 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.828 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6f0f843:::benchmark_data_smithi122_119671_object63236:head# 2024-03-20T09:04:59.828 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.828 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6f15f55:::benchmark_data_smithi122_119671_object92559:head# 2024-03-20T09:04:59.828 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.829 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6f19b5a:::benchmark_data_smithi122_119671_object31577:head# 2024-03-20T09:04:59.829 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.829 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6f1ffb6:::benchmark_data_smithi122_119671_object93139:head# 2024-03-20T09:04:59.829 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.829 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6f20000:::benchmark_data_smithi122_119671_object2486:head# 2024-03-20T09:04:59.829 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.829 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6f22c53:::benchmark_data_smithi122_119671_object43316:head# 2024-03-20T09:04:59.829 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.829 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6f2bf60:::benchmark_data_smithi122_119671_object24363:head# 2024-03-20T09:04:59.830 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.830 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6f2c052:::benchmark_data_smithi122_119671_object96366:head# 2024-03-20T09:04:59.830 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.830 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6f2f41b:::benchmark_data_smithi122_119671_object96512:head# 2024-03-20T09:04:59.830 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.830 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6f39f98:::benchmark_data_smithi122_119671_object45835:head# 2024-03-20T09:04:59.830 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.830 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6f42458:::benchmark_data_smithi122_119671_object18955:head# 2024-03-20T09:04:59.830 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.830 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6f434c4:::benchmark_data_smithi122_119671_object40083:head# 2024-03-20T09:04:59.830 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.831 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6f6426c:::benchmark_data_smithi122_119671_object98451:head# 2024-03-20T09:04:59.831 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.831 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6f7eb24:::benchmark_data_smithi122_119671_object86894:head# 2024-03-20T09:04:59.831 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.831 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6f86ed5:::benchmark_data_smithi122_119671_object4528:head# 2024-03-20T09:04:59.832 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.832 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6f8b636:::benchmark_data_smithi122_119671_object69836:head# 2024-03-20T09:04:59.832 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.832 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6f97790:::benchmark_data_smithi122_119671_object81699:head# 2024-03-20T09:04:59.832 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.832 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6f9c3aa:::benchmark_data_smithi122_119671_object97427:head# 2024-03-20T09:04:59.832 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.832 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6f9d137:::benchmark_data_smithi122_119671_object16470:head# 2024-03-20T09:04:59.832 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.832 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6fae7cb:::benchmark_data_smithi122_119671_object78392:head# 2024-03-20T09:04:59.832 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.832 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6fbcb4a:::benchmark_data_smithi122_119671_object73918:head# 2024-03-20T09:04:59.833 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.833 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6fd69ef:::benchmark_data_smithi122_119671_object55735:head# 2024-03-20T09:04:59.833 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.833 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6fdd09b:::benchmark_data_smithi122_119671_object39424:head# 2024-03-20T09:04:59.833 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.833 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6fe4ebc:::benchmark_data_smithi122_119671_object11735:head# 2024-03-20T09:04:59.833 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.833 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6fe9b20:::benchmark_data_smithi122_119671_object82280:head# 2024-03-20T09:04:59.833 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.833 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6ff2d75:::benchmark_data_smithi122_119671_object11638:head# 2024-03-20T09:04:59.833 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.833 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6ff6697:::benchmark_data_smithi122_119671_object39523:head# 2024-03-20T09:04:59.833 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.833 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6ffa448:::benchmark_data_smithi122_119671_object14124:head# 2024-03-20T09:04:59.834 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.834 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6ffc092:::benchmark_data_smithi122_119671_object38187:head# 2024-03-20T09:04:59.834 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.834 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b6ffe28e:::benchmark_data_smithi122_119671_object79014:head# 2024-03-20T09:04:59.834 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.834 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7001c4d:::benchmark_data_smithi122_119671_object18312:head# 2024-03-20T09:04:59.834 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.834 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b700d579:::benchmark_data_smithi122_119671_object94074:head# 2024-03-20T09:04:59.834 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.834 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b700eeff:::benchmark_data_smithi122_119671_object72836:head# 2024-03-20T09:04:59.834 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.834 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b70115f7:::benchmark_data_smithi122_119671_object60080:head# 2024-03-20T09:04:59.834 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.834 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7011c9d:::benchmark_data_smithi122_119671_object91243:head# 2024-03-20T09:04:59.835 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.835 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7016235:::benchmark_data_smithi122_119671_object22671:head# 2024-03-20T09:04:59.835 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.835 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7016e87:::benchmark_data_smithi122_119671_object68318:head# 2024-03-20T09:04:59.835 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.835 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b701abc7:::benchmark_data_smithi122_119671_object1739:head# 2024-03-20T09:04:59.835 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.835 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b70234bd:::benchmark_data_smithi122_119671_object51845:head# 2024-03-20T09:04:59.835 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.835 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b702903b:::benchmark_data_smithi122_119671_object77625:head# 2024-03-20T09:04:59.835 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.835 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7037569:::benchmark_data_smithi122_119671_object40632:head# 2024-03-20T09:04:59.835 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.835 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b705a797:::benchmark_data_smithi122_119671_object27837:head# 2024-03-20T09:04:59.836 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.836 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7076ac6:::benchmark_data_smithi122_119671_object59907:head# 2024-03-20T09:04:59.836 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.836 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7078fc8:::benchmark_data_smithi122_119671_object82317:head# 2024-03-20T09:04:59.836 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.836 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b707ae39:::benchmark_data_smithi122_119671_object66302:head# 2024-03-20T09:04:59.836 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.836 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7097fb6:::benchmark_data_smithi122_119671_object84026:head# 2024-03-20T09:04:59.836 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.836 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b709fd8d:::benchmark_data_smithi122_119671_object78294:head# 2024-03-20T09:04:59.836 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.836 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b70ae732:::benchmark_data_smithi122_119671_object2927:head# 2024-03-20T09:04:59.836 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.836 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b70bb914:::benchmark_data_smithi122_119671_object38276:head# 2024-03-20T09:04:59.837 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.837 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b70c863f:::benchmark_data_smithi122_119671_object62782:head# 2024-03-20T09:04:59.837 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.837 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b70c9fa8:::benchmark_data_smithi122_119671_object7989:head# 2024-03-20T09:04:59.837 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.837 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b70ca16c:::benchmark_data_smithi122_119671_object22006:head# 2024-03-20T09:04:59.837 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.837 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b70e4768:::benchmark_data_smithi122_119671_object67565:head# 2024-03-20T09:04:59.837 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.837 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b70e7502:::benchmark_data_smithi122_119671_object95620:head# 2024-03-20T09:04:59.837 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.837 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b70ea86b:::benchmark_data_smithi122_119671_object88802:head# 2024-03-20T09:04:59.837 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.838 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b70eb862:::benchmark_data_smithi122_119671_object98352:head# 2024-03-20T09:04:59.838 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.838 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7102519:::benchmark_data_smithi122_119671_object8869:head# 2024-03-20T09:04:59.838 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.838 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b71092d1:::benchmark_data_smithi122_119671_object56173:head# 2024-03-20T09:04:59.838 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.838 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b71102c9:::benchmark_data_smithi122_119671_object85259:head# 2024-03-20T09:04:59.838 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.838 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b71226ae:::benchmark_data_smithi122_119671_object22090:head# 2024-03-20T09:04:59.839 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.839 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7131c87:::benchmark_data_smithi122_119671_object44752:head# 2024-03-20T09:04:59.839 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.839 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b71321f1:::benchmark_data_smithi122_119671_object47020:head# 2024-03-20T09:04:59.839 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.839 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7136eea:::benchmark_data_smithi122_119671_object36683:head# 2024-03-20T09:04:59.839 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.839 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b713d48d:::benchmark_data_smithi122_119671_object63206:head# 2024-03-20T09:04:59.839 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.839 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7145f9a:::benchmark_data_smithi122_119671_object80538:head# 2024-03-20T09:04:59.839 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.839 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7158488:::benchmark_data_smithi122_119671_object19607:head# 2024-03-20T09:04:59.839 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.839 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7158850:::benchmark_data_smithi122_119671_object85335:head# 2024-03-20T09:04:59.840 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.840 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b715d021:::benchmark_data_smithi122_119671_object93658:head# 2024-03-20T09:04:59.840 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.840 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b71624d8:::benchmark_data_smithi122_119671_object49501:head# 2024-03-20T09:04:59.840 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.840 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7167bb1:::benchmark_data_smithi122_119671_object64644:head# 2024-03-20T09:04:59.840 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.840 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7177ce9:::benchmark_data_smithi122_119671_object32251:head# 2024-03-20T09:04:59.840 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.840 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b717d13d:::benchmark_data_smithi122_119671_object71673:head# 2024-03-20T09:04:59.840 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.840 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7188605:::benchmark_data_smithi122_119671_object95278:head# 2024-03-20T09:04:59.840 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.840 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b71a65db:::benchmark_data_smithi122_119671_object97906:head# 2024-03-20T09:04:59.841 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.841 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b71cb74f:::benchmark_data_smithi122_119671_object18468:head# 2024-03-20T09:04:59.841 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.841 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b71f2a11:::benchmark_data_smithi122_119671_object52038:head# 2024-03-20T09:04:59.841 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.841 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7207870:::benchmark_data_smithi122_119671_object53336:head# 2024-03-20T09:04:59.841 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.841 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7210887:::benchmark_data_smithi122_119671_object53552:head# 2024-03-20T09:04:59.841 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.841 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7210e24:::benchmark_data_smithi122_119671_object52414:head# 2024-03-20T09:04:59.841 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.841 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7211e73:::benchmark_data_smithi122_119671_object90674:head# 2024-03-20T09:04:59.841 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.842 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b721e988:::benchmark_data_smithi122_119671_object75879:head# 2024-03-20T09:04:59.842 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.842 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b724be15:::benchmark_data_smithi122_119671_object20177:head# 2024-03-20T09:04:59.842 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.842 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b724ef0f:::benchmark_data_smithi122_119671_object42259:head# 2024-03-20T09:04:59.842 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.842 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b725f5ed:::benchmark_data_smithi122_119671_object9364:head# 2024-03-20T09:04:59.842 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.842 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b726bbf9:::benchmark_data_smithi122_119671_object90588:head# 2024-03-20T09:04:59.842 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.842 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b726f1d4:::benchmark_data_smithi122_119671_object16238:head# 2024-03-20T09:04:59.842 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.842 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7276e6d:::benchmark_data_smithi122_119671_object69186:head# 2024-03-20T09:04:59.842 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.843 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7277722:::benchmark_data_smithi122_119671_object10759:head# 2024-03-20T09:04:59.843 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.843 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b727e477:::benchmark_data_smithi122_119671_object11419:head# 2024-03-20T09:04:59.843 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.843 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b728004b:::benchmark_data_smithi122_119671_object35401:head# 2024-03-20T09:04:59.843 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.843 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7281805:::benchmark_data_smithi122_119671_object69196:head# 2024-03-20T09:04:59.843 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.843 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7289657:::benchmark_data_smithi122_119671_object12324:head# 2024-03-20T09:04:59.843 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.843 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b728d4ed:::benchmark_data_smithi122_119671_object62021:head# 2024-03-20T09:04:59.843 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.843 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b729c17a:::benchmark_data_smithi122_119671_object30594:head# 2024-03-20T09:04:59.844 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.844 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b729e344:::benchmark_data_smithi122_119671_object71809:head# 2024-03-20T09:04:59.844 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.844 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b72a1498:::benchmark_data_smithi122_119671_object10463:head# 2024-03-20T09:04:59.844 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.844 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b72a64e6:::benchmark_data_smithi122_119671_object64274:head# 2024-03-20T09:04:59.844 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.844 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b72b1366:::benchmark_data_smithi122_119671_object21189:head# 2024-03-20T09:04:59.844 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.844 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b72b3a6c:::benchmark_data_smithi122_119671_object33945:head# 2024-03-20T09:04:59.844 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.844 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b72bcc54:::benchmark_data_smithi122_119671_object6726:head# 2024-03-20T09:04:59.844 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.844 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b72cd70d:::benchmark_data_smithi122_119671_object60706:head# 2024-03-20T09:04:59.845 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.845 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b72e9636:::benchmark_data_smithi122_119671_object19615:head# 2024-03-20T09:04:59.845 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.845 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b72fbe2d:::benchmark_data_smithi122_119671_object95057:head# 2024-03-20T09:04:59.845 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.845 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b72fd2d2:::benchmark_data_smithi122_119671_object84411:head# 2024-03-20T09:04:59.845 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.845 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b72fdc64:::benchmark_data_smithi122_119671_object36150:head# 2024-03-20T09:04:59.845 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.845 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7300698:::benchmark_data_smithi122_119671_object4268:head# 2024-03-20T09:04:59.845 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.845 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b730d8ef:::benchmark_data_smithi122_119671_object63188:head# 2024-03-20T09:04:59.845 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.845 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7319b25:::benchmark_data_smithi122_119671_object88064:head# 2024-03-20T09:04:59.846 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.846 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b731da59:::benchmark_data_smithi122_119671_object54466:head# 2024-03-20T09:04:59.846 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.846 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7337fb7:::benchmark_data_smithi122_119671_object30978:head# 2024-03-20T09:04:59.846 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.846 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b733bdb0:::benchmark_data_smithi122_119671_object38575:head# 2024-03-20T09:04:59.846 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.846 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7347388:::benchmark_data_smithi122_119671_object80673:head# 2024-03-20T09:04:59.846 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.846 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b735567a:::benchmark_data_smithi122_119671_object15430:head# 2024-03-20T09:04:59.846 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.846 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7364b95:::benchmark_data_smithi122_119671_object32328:head# 2024-03-20T09:04:59.846 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.847 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b736b039:::benchmark_data_smithi122_119671_object65587:head# 2024-03-20T09:04:59.847 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.847 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b73781a9:::benchmark_data_smithi122_119671_object80233:head# 2024-03-20T09:04:59.847 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.847 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7387f1b:::benchmark_data_smithi122_119671_object27867:head# 2024-03-20T09:04:59.847 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.847 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b738a55f:::benchmark_data_smithi122_119671_object34347:head# 2024-03-20T09:04:59.847 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.847 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b738fc20:::benchmark_data_smithi122_119671_object9821:head# 2024-03-20T09:04:59.847 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.847 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b73909e1:::benchmark_data_smithi122_119671_object31799:head# 2024-03-20T09:04:59.847 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.847 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7392eec:::benchmark_data_smithi122_119671_object67289:head# 2024-03-20T09:04:59.847 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.847 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b73a257f:::benchmark_data_smithi122_119671_object39887:head# 2024-03-20T09:04:59.848 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.848 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b73ce401:::benchmark_data_smithi122_119671_object86318:head# 2024-03-20T09:04:59.848 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.848 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b73cff94:::benchmark_data_smithi122_119671_object30671:head# 2024-03-20T09:04:59.848 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.848 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b73dda0c:::benchmark_data_smithi122_119671_object32295:head# 2024-03-20T09:04:59.848 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.848 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b73f78c0:::benchmark_data_smithi122_119671_object78859:head# 2024-03-20T09:04:59.848 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.848 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b740670e:::benchmark_data_smithi122_119671_object96777:head# 2024-03-20T09:04:59.848 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.849 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b740825b:::benchmark_data_smithi122_119671_object15940:head# 2024-03-20T09:04:59.849 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.849 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b740b304:::benchmark_data_smithi122_119671_object8761:head# 2024-03-20T09:04:59.849 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.849 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b740df10:::benchmark_data_smithi122_119671_object4272:head# 2024-03-20T09:04:59.849 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.849 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b740f507:::benchmark_data_smithi122_119671_object39210:head# 2024-03-20T09:04:59.849 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.849 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7420ba3:::benchmark_data_smithi122_119671_object93014:head# 2024-03-20T09:04:59.849 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.849 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7427ccd:::benchmark_data_smithi122_119671_object20835:head# 2024-03-20T09:04:59.849 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.850 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7432684:::benchmark_data_smithi122_119671_object11963:head# 2024-03-20T09:04:59.850 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.850 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b743a03e:::benchmark_data_smithi122_119671_object90553:head# 2024-03-20T09:04:59.850 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.850 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7447676:::benchmark_data_smithi122_119671_object71415:head# 2024-03-20T09:04:59.850 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.850 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b744df15:::benchmark_data_smithi122_119671_object65724:head# 2024-03-20T09:04:59.850 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.850 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b745117c:::benchmark_data_smithi122_119671_object57840:head# 2024-03-20T09:04:59.850 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.850 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b745c1ae:::benchmark_data_smithi122_119671_object18086:head# 2024-03-20T09:04:59.850 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.850 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b74603cd:::benchmark_data_smithi122_119671_object98425:head# 2024-03-20T09:04:59.850 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.851 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b74745d4:::benchmark_data_smithi122_119671_object57777:head# 2024-03-20T09:04:59.851 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.851 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7476996:::benchmark_data_smithi122_119671_object89376:head# 2024-03-20T09:04:59.851 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.851 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b748a66e:::benchmark_data_smithi122_119671_object42205:head# 2024-03-20T09:04:59.851 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.851 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b748dbde:::benchmark_data_smithi122_119671_object37122:head# 2024-03-20T09:04:59.851 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.851 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b749802d:::benchmark_data_smithi122_119671_object74099:head# 2024-03-20T09:04:59.851 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.851 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b74ab604:::benchmark_data_smithi122_119671_object83731:head# 2024-03-20T09:04:59.851 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.851 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b74c3797:::benchmark_data_smithi122_119671_object41824:head# 2024-03-20T09:04:59.852 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.852 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b74caca9:::benchmark_data_smithi122_119671_object46084:head# 2024-03-20T09:04:59.852 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.852 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b74d4dda:::benchmark_data_smithi122_119671_object52714:head# 2024-03-20T09:04:59.852 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.852 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b74e1905:::benchmark_data_smithi122_119671_object42309:head# 2024-03-20T09:04:59.852 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.852 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b74e47a6:::benchmark_data_smithi122_119671_object70003:head# 2024-03-20T09:04:59.852 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.852 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b74e5710:::benchmark_data_smithi122_119671_object94919:head# 2024-03-20T09:04:59.852 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.852 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b74ec752:::benchmark_data_smithi122_119671_object5513:head# 2024-03-20T09:04:59.852 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.852 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b74ed708:::benchmark_data_smithi122_119671_object87280:head# 2024-03-20T09:04:59.853 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.853 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b74fb7b4:::benchmark_data_smithi122_119671_object54251:head# 2024-03-20T09:04:59.853 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.853 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b74fddd5:::benchmark_data_smithi122_119671_object47456:head# 2024-03-20T09:04:59.853 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.853 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b75085fa:::benchmark_data_smithi122_119671_object19428:head# 2024-03-20T09:04:59.853 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.853 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7511368:::benchmark_data_smithi122_119671_object62228:head# 2024-03-20T09:04:59.853 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.853 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7513490:::benchmark_data_smithi122_119671_object25781:head# 2024-03-20T09:04:59.853 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.853 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7513792:::benchmark_data_smithi122_119671_object93120:head# 2024-03-20T09:04:59.853 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.853 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7513d6e:::benchmark_data_smithi122_119671_object33434:head# 2024-03-20T09:04:59.854 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.854 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b754961f:::benchmark_data_smithi122_119671_object58814:head# 2024-03-20T09:04:59.854 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.854 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7556d4a:::benchmark_data_smithi122_119671_object84262:head# 2024-03-20T09:04:59.854 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.854 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7562a7e:::benchmark_data_smithi122_119671_object58123:head# 2024-03-20T09:04:59.854 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.854 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7570058:::benchmark_data_smithi122_119671_object24508:head# 2024-03-20T09:04:59.854 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.854 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b75725b6:::benchmark_data_smithi122_119671_object9989:head# 2024-03-20T09:04:59.854 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.854 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7586b48:::benchmark_data_smithi122_119671_object22718:head# 2024-03-20T09:04:59.854 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.855 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b758b757:::benchmark_data_smithi122_119671_object92675:head# 2024-03-20T09:04:59.855 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.855 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7597cdd:::benchmark_data_smithi122_119671_object26249:head# 2024-03-20T09:04:59.855 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.855 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b759dd05:::benchmark_data_smithi122_119671_object2806:head# 2024-03-20T09:04:59.855 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.855 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b75a6647:::benchmark_data_smithi122_119671_object9813:head# 2024-03-20T09:04:59.855 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.855 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b75b8196:::benchmark_data_smithi122_119671_object42287:head# 2024-03-20T09:04:59.855 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.855 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b75b971c:::benchmark_data_smithi122_119671_object73066:head# 2024-03-20T09:04:59.855 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.855 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b75bd073:::benchmark_data_smithi122_119671_object85594:head# 2024-03-20T09:04:59.855 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.856 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b75d1c78:::benchmark_data_smithi122_119671_object40006:head# 2024-03-20T09:04:59.856 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.856 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b75d683c:::benchmark_data_smithi122_119671_object19859:head# 2024-03-20T09:04:59.856 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.856 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b75ef690:::benchmark_data_smithi122_119671_object3086:head# 2024-03-20T09:04:59.856 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.856 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b75ffc70:::benchmark_data_smithi122_119671_object71186:head# 2024-03-20T09:04:59.856 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.856 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b76025d4:::benchmark_data_smithi122_119671_object50865:head# 2024-03-20T09:04:59.856 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.856 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7602fa2:::benchmark_data_smithi122_119671_object38732:head# 2024-03-20T09:04:59.856 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.856 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7604669:::benchmark_data_smithi122_119671_object85701:head# 2024-03-20T09:04:59.856 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.857 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b76069bd:::benchmark_data_smithi122_119671_object78070:head# 2024-03-20T09:04:59.857 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.857 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b760b412:::benchmark_data_smithi122_119671_object39833:head# 2024-03-20T09:04:59.857 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.857 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b761272b:::benchmark_data_smithi122_119671_object32285:head# 2024-03-20T09:04:59.857 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.857 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b76129f5:::benchmark_data_smithi122_119671_object97943:head# 2024-03-20T09:04:59.857 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.857 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b76172ec:::benchmark_data_smithi122_119671_object84454:head# 2024-03-20T09:04:59.857 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.857 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b76181a4:::benchmark_data_smithi122_119671_object14747:head# 2024-03-20T09:04:59.857 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.857 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b761b27f:::benchmark_data_smithi122_119671_object44903:head# 2024-03-20T09:04:59.857 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.858 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7621088:::benchmark_data_smithi122_119671_object97987:head# 2024-03-20T09:04:59.858 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.858 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b762ffcf:::benchmark_data_smithi122_119671_object59613:head# 2024-03-20T09:04:59.858 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.858 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b76433f0:::benchmark_data_smithi122_119671_object26047:head# 2024-03-20T09:04:59.858 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.858 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b76570c1:::benchmark_data_smithi122_119671_object50479:head# 2024-03-20T09:04:59.858 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.858 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b766f1ad:::benchmark_data_smithi122_119671_object3991:head# 2024-03-20T09:04:59.858 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.858 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b76801ef:::benchmark_data_smithi122_119671_object85976:head# 2024-03-20T09:04:59.858 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.858 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b769d6ed:::benchmark_data_smithi122_119671_object77349:head# 2024-03-20T09:04:59.858 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.859 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b76bb989:::benchmark_data_smithi122_119671_object95373:head# 2024-03-20T09:04:59.859 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.859 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b76bd1d0:::benchmark_data_smithi122_119671_object70528:head# 2024-03-20T09:04:59.859 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.859 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b76c7c13:::benchmark_data_smithi122_119671_object84936:head# 2024-03-20T09:04:59.859 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.859 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b76c8e32:::benchmark_data_smithi122_119671_object69039:head# 2024-03-20T09:04:59.859 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.860 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b76d139b:::benchmark_data_smithi122_119671_object7514:head# 2024-03-20T09:04:59.860 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.860 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b76fa1ab:::benchmark_data_smithi122_119671_object45702:head# 2024-03-20T09:04:59.860 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.860 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b76fff39:::benchmark_data_smithi122_119671_object27500:head# 2024-03-20T09:04:59.860 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.860 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7706124:::benchmark_data_smithi122_119671_object48924:head# 2024-03-20T09:04:59.860 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.860 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7714984:::benchmark_data_smithi122_119671_object31226:head# 2024-03-20T09:04:59.860 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.860 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7718e90:::benchmark_data_smithi122_119671_object24115:head# 2024-03-20T09:04:59.860 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.860 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b771b03c:::benchmark_data_smithi122_119671_object35195:head# 2024-03-20T09:04:59.860 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.861 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b771ce36:::benchmark_data_smithi122_119671_object24747:head# 2024-03-20T09:04:59.861 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.861 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b771e887:::benchmark_data_smithi122_119671_object79603:head# 2024-03-20T09:04:59.861 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.861 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7720bed:::benchmark_data_smithi122_119671_object9366:head# 2024-03-20T09:04:59.861 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.861 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7727993:::benchmark_data_smithi122_119671_object36234:head# 2024-03-20T09:04:59.861 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.861 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b772af94:::benchmark_data_smithi122_119671_object18044:head# 2024-03-20T09:04:59.861 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.861 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b772ffa8:::benchmark_data_smithi122_119671_object90482:head# 2024-03-20T09:04:59.861 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.861 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b774bbce:::benchmark_data_smithi122_119671_object38780:head# 2024-03-20T09:04:59.861 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.862 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7750090:::benchmark_data_smithi122_119671_object12164:head# 2024-03-20T09:04:59.862 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.862 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b777c54b:::benchmark_data_smithi122_119671_object57322:head# 2024-03-20T09:04:59.862 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.862 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b77849fc:::benchmark_data_smithi122_119671_object3169:head# 2024-03-20T09:04:59.862 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.862 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7786505:::benchmark_data_smithi122_119671_object75476:head# 2024-03-20T09:04:59.862 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.862 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7789927:::benchmark_data_smithi122_119671_object97973:head# 2024-03-20T09:04:59.862 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.862 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b778af9f:::benchmark_data_smithi122_119671_object24491:head# 2024-03-20T09:04:59.862 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.862 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b778ed92:::benchmark_data_smithi122_119671_object78096:head# 2024-03-20T09:04:59.862 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.863 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7792ca0:::benchmark_data_smithi122_119671_object2925:head# 2024-03-20T09:04:59.863 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.863 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b779c89c:::benchmark_data_smithi122_119671_object32886:head# 2024-03-20T09:04:59.863 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.863 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b779f260:::benchmark_data_smithi122_119671_object10389:head# 2024-03-20T09:04:59.863 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.863 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b77bbb35:::benchmark_data_smithi122_119671_object87237:head# 2024-03-20T09:04:59.863 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.863 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b77bd48c:::benchmark_data_smithi122_119671_object34972:head# 2024-03-20T09:04:59.863 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.863 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b77d8f4a:::benchmark_data_smithi122_119671_object7272:head# 2024-03-20T09:04:59.863 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.863 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b77e9436:::benchmark_data_smithi122_119671_object28056:head# 2024-03-20T09:04:59.863 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.864 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b78031a1:::benchmark_data_smithi122_119671_object99188:head# 2024-03-20T09:04:59.864 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.864 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7806356:::benchmark_data_smithi122_119671_object44530:head# 2024-03-20T09:04:59.864 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.864 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b78091d2:::benchmark_data_smithi122_119671_object92817:head# 2024-03-20T09:04:59.864 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.864 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b780e66a:::benchmark_data_smithi122_119671_object12969:head# 2024-03-20T09:04:59.864 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.864 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7812fbb:::benchmark_data_smithi122_119671_object90963:head# 2024-03-20T09:04:59.864 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.864 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b781314b:::benchmark_data_smithi122_119671_object36904:head# 2024-03-20T09:04:59.864 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.864 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b78149d9:::benchmark_data_smithi122_119671_object81717:head# 2024-03-20T09:04:59.864 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.865 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7819fed:::benchmark_data_smithi122_119671_object16731:head# 2024-03-20T09:04:59.865 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.865 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b782b710:::benchmark_data_smithi122_119671_object25435:head# 2024-03-20T09:04:59.865 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.865 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b782c966:::benchmark_data_smithi122_119671_object28342:head# 2024-03-20T09:04:59.865 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.865 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b782df3c:::benchmark_data_smithi122_119671_object48121:head# 2024-03-20T09:04:59.865 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.865 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7839eb5:::benchmark_data_smithi122_119671_object76732:head# 2024-03-20T09:04:59.865 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.865 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b784ddf8:::benchmark_data_smithi122_119671_object19856:head# 2024-03-20T09:04:59.865 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.865 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b784ffb0:::benchmark_data_smithi122_119671_object53135:head# 2024-03-20T09:04:59.865 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.866 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b78653f5:::benchmark_data_smithi122_119671_object31123:head# 2024-03-20T09:04:59.866 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.866 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b78a68bd:::benchmark_data_smithi122_119671_object17727:head# 2024-03-20T09:04:59.866 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.866 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b78b1bd2:::benchmark_data_smithi122_119671_object10499:head# 2024-03-20T09:04:59.866 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.866 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b78b7ce1:::benchmark_data_smithi122_119671_object90825:head# 2024-03-20T09:04:59.866 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.866 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b78c6358:::benchmark_data_smithi122_119671_object3119:head# 2024-03-20T09:04:59.866 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.866 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b78c8ee5:::benchmark_data_smithi122_119671_object56581:head# 2024-03-20T09:04:59.866 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.866 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b78cf910:::benchmark_data_smithi122_119671_object88343:head# 2024-03-20T09:04:59.867 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.867 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b78e7532:::benchmark_data_smithi122_119671_object80939:head# 2024-03-20T09:04:59.867 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.867 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b78f4620:::benchmark_data_smithi122_119671_object21504:head# 2024-03-20T09:04:59.867 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.867 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7904eb8:::benchmark_data_smithi122_119671_object88555:head# 2024-03-20T09:04:59.867 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.867 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7908d5c:::benchmark_data_smithi122_119671_object29847:head# 2024-03-20T09:04:59.867 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.867 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b791f48f:::benchmark_data_smithi122_119671_object59154:head# 2024-03-20T09:04:59.867 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.867 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7920019:::benchmark_data_smithi122_119671_object54933:head# 2024-03-20T09:04:59.867 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.868 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b79210dd:::benchmark_data_smithi122_119671_object33842:head# 2024-03-20T09:04:59.868 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.868 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b793740d:::benchmark_data_smithi122_119671_object27463:head# 2024-03-20T09:04:59.868 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.868 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b793ae8a:::benchmark_data_smithi122_119671_object47665:head# 2024-03-20T09:04:59.868 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.868 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b793eb3e:::benchmark_data_smithi122_119671_object86965:head# 2024-03-20T09:04:59.869 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.869 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7956cff:::benchmark_data_smithi122_119671_object85987:head# 2024-03-20T09:04:59.869 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.869 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7977b9c:::benchmark_data_smithi122_119671_object73242:head# 2024-03-20T09:04:59.869 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.869 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b79b320e:::benchmark_data_smithi122_119671_object71576:head# 2024-03-20T09:04:59.869 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.869 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b79be3b4:::benchmark_data_smithi122_119671_object7731:head# 2024-03-20T09:04:59.870 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.870 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b79bfdf6:::benchmark_data_smithi122_119671_object65838:head# 2024-03-20T09:04:59.870 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.870 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b79c83fa:::benchmark_data_smithi122_119671_object54229:head# 2024-03-20T09:04:59.870 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.870 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b79d0112:::benchmark_data_smithi122_119671_object33742:head# 2024-03-20T09:04:59.870 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.870 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b79eb156:::benchmark_data_smithi122_119671_object1150:head# 2024-03-20T09:04:59.870 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.870 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b79f1082:::benchmark_data_smithi122_119671_object96351:head# 2024-03-20T09:04:59.870 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.870 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b79fa2ff:::benchmark_data_smithi122_119671_object34004:head# 2024-03-20T09:04:59.870 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.871 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7a03da8:::benchmark_data_smithi122_119671_object4107:head# 2024-03-20T09:04:59.871 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.871 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7a1bc94:::benchmark_data_smithi122_119671_object8300:head# 2024-03-20T09:04:59.871 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.871 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7a20b60:::benchmark_data_smithi122_119671_object27699:head# 2024-03-20T09:04:59.871 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.871 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7a2447d:::benchmark_data_smithi122_119671_object73152:head# 2024-03-20T09:04:59.871 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.871 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7a33733:::benchmark_data_smithi122_119671_object46248:head# 2024-03-20T09:04:59.871 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.871 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7a4315a:::benchmark_data_smithi122_119671_object92581:head# 2024-03-20T09:04:59.871 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.871 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7a44835:::benchmark_data_smithi122_119671_object15835:head# 2024-03-20T09:04:59.871 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.872 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7a59a06:::benchmark_data_smithi122_119671_object57794:head# 2024-03-20T09:04:59.872 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.872 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7a5cb18:::benchmark_data_smithi122_119671_object11017:head# 2024-03-20T09:04:59.872 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.872 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7a637c7:::benchmark_data_smithi122_119671_object5969:head# 2024-03-20T09:04:59.872 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.872 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7a66831:::benchmark_data_smithi122_119671_object67911:head# 2024-03-20T09:04:59.872 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.872 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7a67bb3:::benchmark_data_smithi122_119671_object78525:head# 2024-03-20T09:04:59.872 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.872 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7a75830:::benchmark_data_smithi122_119671_object94282:head# 2024-03-20T09:04:59.872 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.872 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7a7d9f9:::benchmark_data_smithi122_119671_object47913:head# 2024-03-20T09:04:59.872 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.873 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7ab658d:::benchmark_data_smithi122_119671_object84109:head# 2024-03-20T09:04:59.873 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.873 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7ac440d:::benchmark_data_smithi122_119671_object19883:head# 2024-03-20T09:04:59.873 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.873 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7ac6e99:::benchmark_data_smithi122_119671_object84666:head# 2024-03-20T09:04:59.873 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.873 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7ac7a1b:::benchmark_data_smithi122_119671_object79194:head# 2024-03-20T09:04:59.873 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.873 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7ac827a:::benchmark_data_smithi122_119671_object27607:head# 2024-03-20T09:04:59.873 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.873 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7acb358:::benchmark_data_smithi122_119671_object58167:head# 2024-03-20T09:04:59.873 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.873 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7acb5e1:::benchmark_data_smithi122_119671_object78038:head# 2024-03-20T09:04:59.874 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.874 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7ad5e2f:::benchmark_data_smithi122_119671_object46912:head# 2024-03-20T09:04:59.874 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.874 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7ad8f60:::benchmark_data_smithi122_119671_object84332:head# 2024-03-20T09:04:59.874 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.874 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7ae7beb:::benchmark_data_smithi122_119671_object38012:head# 2024-03-20T09:04:59.874 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.874 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7af7ce1:::benchmark_data_smithi122_119671_object6506:head# 2024-03-20T09:04:59.874 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.874 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7afc777:::benchmark_data_smithi122_119671_object25221:head# 2024-03-20T09:04:59.874 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.874 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7b035df:::benchmark_data_smithi122_119671_object11273:head# 2024-03-20T09:04:59.874 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.874 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7b037ed:::benchmark_data_smithi122_119671_object54360:head# 2024-03-20T09:04:59.875 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.875 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7b164d8:::benchmark_data_smithi122_119671_object80067:head# 2024-03-20T09:04:59.875 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.875 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7b2f23f:::benchmark_data_smithi122_119671_object98133:head# 2024-03-20T09:04:59.875 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.875 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7b4934b:::benchmark_data_smithi122_119671_object49088:head# 2024-03-20T09:04:59.875 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.875 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7b4bafe:::benchmark_data_smithi122_119671_object30040:head# 2024-03-20T09:04:59.875 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.875 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7b516de:::benchmark_data_smithi122_119671_object14712:head# 2024-03-20T09:04:59.875 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.875 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7b5790a:::benchmark_data_smithi122_119671_object19852:head# 2024-03-20T09:04:59.875 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.875 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7b5cb85:::benchmark_data_smithi122_119671_object78171:head# 2024-03-20T09:04:59.876 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.876 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7b5e5bc:::benchmark_data_smithi122_119671_object4620:head# 2024-03-20T09:04:59.876 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.876 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7b72cd2:::benchmark_data_smithi122_119671_object41006:head# 2024-03-20T09:04:59.876 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.876 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7b72cdc:::benchmark_data_smithi122_119671_object36090:head# 2024-03-20T09:04:59.876 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.876 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7b72e92:::benchmark_data_smithi122_119671_object33671:head# 2024-03-20T09:04:59.876 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.876 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7b76fad:::benchmark_data_smithi122_119671_object5766:head# 2024-03-20T09:04:59.876 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.876 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7b94fd9:::benchmark_data_smithi122_119671_object68524:head# 2024-03-20T09:04:59.876 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.876 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7badd49:::benchmark_data_smithi122_119671_object96753:head# 2024-03-20T09:04:59.877 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.877 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7bb6fe6:::benchmark_data_smithi122_119671_object17900:head# 2024-03-20T09:04:59.877 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.877 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7bbe7ec:::benchmark_data_smithi122_119671_object38355:head# 2024-03-20T09:04:59.877 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.877 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7bc3bbc:::benchmark_data_smithi122_119671_object42316:head# 2024-03-20T09:04:59.877 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.877 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7be6dda:::benchmark_data_smithi122_119671_object37778:head# 2024-03-20T09:04:59.877 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.877 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7bf62f5:::benchmark_data_smithi122_119671_object24013:head# 2024-03-20T09:04:59.877 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.877 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7bfee56:::benchmark_data_smithi122_119671_object74638:head# 2024-03-20T09:04:59.877 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.877 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7c1d39d:::benchmark_data_smithi122_119671_object30681:head# 2024-03-20T09:04:59.878 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.878 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7c43e86:::benchmark_data_smithi122_119671_object96007:head# 2024-03-20T09:04:59.878 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.878 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7c54a85:::benchmark_data_smithi122_119671_object90576:head# 2024-03-20T09:04:59.878 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.878 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7c59723:::benchmark_data_smithi122_119671_object47597:head# 2024-03-20T09:04:59.878 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.878 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7c688b1:::benchmark_data_smithi122_119671_object62953:head# 2024-03-20T09:04:59.878 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.878 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7c84acb:::benchmark_data_smithi122_119671_object84891:head# 2024-03-20T09:04:59.878 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.878 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7c8cd0b:::benchmark_data_smithi122_119671_object63187:head# 2024-03-20T09:04:59.878 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.879 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7c96e2d:::benchmark_data_smithi122_119671_object66446:head# 2024-03-20T09:04:59.879 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.879 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7ca5c8e:::benchmark_data_smithi122_119671_object76806:head# 2024-03-20T09:04:59.879 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.879 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7cab9fb:::benchmark_data_smithi122_119671_object89935:head# 2024-03-20T09:04:59.879 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.879 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7cbcbeb:::benchmark_data_smithi122_119671_object69943:head# 2024-03-20T09:04:59.879 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.879 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7cd560b:::benchmark_data_smithi122_119671_object65194:head# 2024-03-20T09:04:59.879 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.879 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7cddb9d:::benchmark_data_smithi122_119671_object35368:head# 2024-03-20T09:04:59.879 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.879 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7cf04ed:::benchmark_data_smithi122_119671_object18770:head# 2024-03-20T09:04:59.880 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.880 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7cf5d70:::benchmark_data_smithi122_119671_object52409:head# 2024-03-20T09:04:59.880 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.880 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7cf9fdf:::benchmark_data_smithi122_119671_object36411:head# 2024-03-20T09:04:59.880 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.880 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7cfa66d:::benchmark_data_smithi122_119671_object25584:head# 2024-03-20T09:04:59.880 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.880 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7cfcd99:::benchmark_data_smithi122_119671_object7732:head# 2024-03-20T09:04:59.880 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.880 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7d0bb42:::benchmark_data_smithi122_119671_object32065:head# 2024-03-20T09:04:59.881 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.881 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7d299ce:::benchmark_data_smithi122_119671_object58491:head# 2024-03-20T09:04:59.881 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.881 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7d383cd:::benchmark_data_smithi122_119671_object22651:head# 2024-03-20T09:04:59.881 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.881 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7d3d599:::benchmark_data_smithi122_119671_object23310:head# 2024-03-20T09:04:59.881 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.881 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7d3eff2:::benchmark_data_smithi122_119671_object42697:head# 2024-03-20T09:04:59.881 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.881 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7d4d855:::benchmark_data_smithi122_119671_object5676:head# 2024-03-20T09:04:59.881 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.881 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7d53a7d:::benchmark_data_smithi122_119671_object6141:head# 2024-03-20T09:04:59.881 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.882 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7d569ba:::benchmark_data_smithi122_119671_object43862:head# 2024-03-20T09:04:59.882 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.882 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7d7a4a5:::benchmark_data_smithi122_119671_object15747:head# 2024-03-20T09:04:59.882 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.882 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7d7f8de:::benchmark_data_smithi122_119671_object8295:head# 2024-03-20T09:04:59.882 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.882 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7d99157:::benchmark_data_smithi122_119671_object45151:head# 2024-03-20T09:04:59.882 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.882 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7daa3f0:::benchmark_data_smithi122_119671_object38533:head# 2024-03-20T09:04:59.882 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.882 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7db5e53:::benchmark_data_smithi122_119671_object78165:head# 2024-03-20T09:04:59.882 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.882 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7dc7759:::benchmark_data_smithi122_119671_object71736:head# 2024-03-20T09:04:59.883 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.883 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7de2b19:::benchmark_data_smithi122_119671_object29384:head# 2024-03-20T09:04:59.883 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.883 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7dffc1e:::benchmark_data_smithi122_119671_object63532:head# 2024-03-20T09:04:59.883 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.883 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7e0da74:::benchmark_data_smithi122_119671_object6988:head# 2024-03-20T09:04:59.883 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.883 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7e186c3:::benchmark_data_smithi122_119671_object14964:head# 2024-03-20T09:04:59.883 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.883 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7e2430d:::benchmark_data_smithi122_119671_object59678:head# 2024-03-20T09:04:59.883 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.883 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7e40e4f:::benchmark_data_smithi122_119671_object35360:head# 2024-03-20T09:04:59.883 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.883 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7e4b3c5:::benchmark_data_smithi122_119671_object66838:head# 2024-03-20T09:04:59.884 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.884 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7e4cc01:::benchmark_data_smithi122_119671_object50789:head# 2024-03-20T09:04:59.884 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.884 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7e4d4bc:::benchmark_data_smithi122_119671_object29291:head# 2024-03-20T09:04:59.884 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.884 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7e4e985:::benchmark_data_smithi122_119671_object34662:head# 2024-03-20T09:04:59.884 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.884 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7e50f5e:::benchmark_data_smithi122_119671_object3597:head# 2024-03-20T09:04:59.884 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.884 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7e579c8:::benchmark_data_smithi122_119671_object4081:head# 2024-03-20T09:04:59.884 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.884 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7e63e94:::benchmark_data_smithi122_119671_object57989:head# 2024-03-20T09:04:59.884 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.884 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7e6f6e2:::benchmark_data_smithi122_119671_object14023:head# 2024-03-20T09:04:59.885 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.885 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7e6fe27:::benchmark_data_smithi122_119671_object34322:head# 2024-03-20T09:04:59.885 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.885 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7e86796:::benchmark_data_smithi122_119671_object13958:head# 2024-03-20T09:04:59.885 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.885 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7ea578c:::benchmark_data_smithi122_119671_object96042:head# 2024-03-20T09:04:59.885 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.885 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7ea7b79:::benchmark_data_smithi122_119671_object10432:head# 2024-03-20T09:04:59.885 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.885 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7ec881b:::benchmark_data_smithi122_119671_object85395:head# 2024-03-20T09:04:59.885 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.885 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7ecd36e:::benchmark_data_smithi122_119671_object61778:head# 2024-03-20T09:04:59.885 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.885 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7ed0d21:::benchmark_data_smithi122_119671_object87835:head# 2024-03-20T09:04:59.886 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.886 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7ed2031:::benchmark_data_smithi122_119671_object43248:head# 2024-03-20T09:04:59.886 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.886 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7ed7436:::benchmark_data_smithi122_119671_object7715:head# 2024-03-20T09:04:59.886 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.886 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7ef5de4:::benchmark_data_smithi122_119671_object97783:head# 2024-03-20T09:04:59.886 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.886 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7ef9a74:::benchmark_data_smithi122_119671_object7610:head# 2024-03-20T09:04:59.886 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.886 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7f10722:::benchmark_data_smithi122_119671_object95476:head# 2024-03-20T09:04:59.886 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.886 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7f14cf7:::benchmark_data_smithi122_119671_object72091:head# 2024-03-20T09:04:59.886 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.887 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7f1ab0d:::benchmark_data_smithi122_119671_object94637:head# 2024-03-20T09:04:59.887 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.887 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7f2be3f:::benchmark_data_smithi122_119671_object25948:head# 2024-03-20T09:04:59.887 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.887 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7f44fc3:::benchmark_data_smithi122_119671_object64845:head# 2024-03-20T09:04:59.887 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.887 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7f450c3:::benchmark_data_smithi122_119671_object28563:head# 2024-03-20T09:04:59.887 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.887 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7f46517:::benchmark_data_smithi122_119671_object62667:head# 2024-03-20T09:04:59.887 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.887 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7f550e2:::benchmark_data_smithi122_119671_object37477:head# 2024-03-20T09:04:59.887 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.887 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7f8b335:::benchmark_data_smithi122_119671_object42223:head# 2024-03-20T09:04:59.887 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.888 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7f94a74:::benchmark_data_smithi122_119671_object40426:head# 2024-03-20T09:04:59.888 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.888 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7fa065d:::benchmark_data_smithi122_119671_object74938:head# 2024-03-20T09:04:59.888 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.888 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7fa80fc:::benchmark_data_smithi122_119671_object38864:head# 2024-03-20T09:04:59.888 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.888 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7fad547:::benchmark_data_smithi122_119671_object35139:head# 2024-03-20T09:04:59.888 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.888 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7fb043a:::benchmark_data_smithi122_119671_object44740:head# 2024-03-20T09:04:59.888 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.888 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7fb448e:::benchmark_data_smithi122_119671_object76906:head# 2024-03-20T09:04:59.888 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.888 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7fb6d57:::benchmark_data_smithi122_119671_object56850:head# 2024-03-20T09:04:59.888 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.889 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7fbba0c:::benchmark_data_smithi122_119671_object19928:head# 2024-03-20T09:04:59.889 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.889 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7fbddb4:::benchmark_data_smithi122_119671_object44209:head# 2024-03-20T09:04:59.889 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.889 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7fd7a52:::benchmark_data_smithi122_119671_object83601:head# 2024-03-20T09:04:59.889 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.889 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7fd88d8:::benchmark_data_smithi122_119671_object87272:head# 2024-03-20T09:04:59.889 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.889 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7fdb1dc:::benchmark_data_smithi122_119671_object28352:head# 2024-03-20T09:04:59.889 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.889 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7fdcbc8:::benchmark_data_smithi122_119671_object68383:head# 2024-03-20T09:04:59.889 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.889 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7fe0556:::benchmark_data_smithi122_119671_object71275:head# 2024-03-20T09:04:59.889 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.890 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7fe28a6:::benchmark_data_smithi122_119671_object11375:head# 2024-03-20T09:04:59.890 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.890 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7ff05f5:::benchmark_data_smithi122_119671_object16767:head# 2024-03-20T09:04:59.890 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.890 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b7ff2d72:::benchmark_data_smithi122_119671_object88271:head# 2024-03-20T09:04:59.890 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.890 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b800202b:::benchmark_data_smithi122_119671_object19881:head# 2024-03-20T09:04:59.890 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.891 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b80036f3:::benchmark_data_smithi122_119671_object55752:head# 2024-03-20T09:04:59.891 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.891 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8008e71:::benchmark_data_smithi122_119671_object70820:head# 2024-03-20T09:04:59.891 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.891 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b801a664:::benchmark_data_smithi122_119671_object65520:head# 2024-03-20T09:04:59.891 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.891 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b802ada9:::benchmark_data_smithi122_119671_object73030:head# 2024-03-20T09:04:59.891 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.891 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b802bd0e:::benchmark_data_smithi122_119671_object20391:head# 2024-03-20T09:04:59.891 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.891 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8032b0b:::benchmark_data_smithi122_119671_object91305:head# 2024-03-20T09:04:59.891 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.891 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b803967d:::benchmark_data_smithi122_119671_object52425:head# 2024-03-20T09:04:59.892 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.892 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b803aa79:::benchmark_data_smithi122_119671_object35762:head# 2024-03-20T09:04:59.892 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.892 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b803d11c:::benchmark_data_smithi122_119671_object55385:head# 2024-03-20T09:04:59.892 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.892 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b803f500:::benchmark_data_smithi122_119671_object84556:head# 2024-03-20T09:04:59.892 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.893 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8055109:::benchmark_data_smithi122_119671_object38302:head# 2024-03-20T09:04:59.893 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.893 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8057a52:::benchmark_data_smithi122_119671_object4994:head# 2024-03-20T09:04:59.893 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.893 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b80597b5:::benchmark_data_smithi122_119671_object15314:head# 2024-03-20T09:04:59.893 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.893 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b80727c5:::benchmark_data_smithi122_119671_object27129:head# 2024-03-20T09:04:59.893 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.893 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b80755c2:::benchmark_data_smithi122_119671_object64041:head# 2024-03-20T09:04:59.893 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.893 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8075c7f:::benchmark_data_smithi122_119671_object47146:head# 2024-03-20T09:04:59.894 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.894 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b807684e:::benchmark_data_smithi122_119671_object57881:head# 2024-03-20T09:04:59.894 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.894 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8089bf2:::benchmark_data_smithi122_119671_object4491:head# 2024-03-20T09:04:59.894 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.894 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8096e5b:::benchmark_data_smithi122_119671_object66453:head# 2024-03-20T09:04:59.894 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.894 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b80b08e5:::benchmark_data_smithi122_119671_object25090:head# 2024-03-20T09:04:59.894 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.894 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b80b59bf:::benchmark_data_smithi122_119671_object33132:head# 2024-03-20T09:04:59.895 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.895 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b80b8a68:::benchmark_data_smithi122_119671_object24412:head# 2024-03-20T09:04:59.895 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.895 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b80b96ef:::benchmark_data_smithi122_119671_object93241:head# 2024-03-20T09:04:59.895 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.895 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b80c0db9:::benchmark_data_smithi122_119671_object91088:head# 2024-03-20T09:04:59.895 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.895 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b80c817f:::benchmark_data_smithi122_119671_object44914:head# 2024-03-20T09:04:59.895 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.895 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b80cd1ac:::benchmark_data_smithi122_119671_object12900:head# 2024-03-20T09:04:59.895 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.895 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b80d21d9:::benchmark_data_smithi122_119671_object50161:head# 2024-03-20T09:04:59.895 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.895 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b80d71e4:::benchmark_data_smithi122_119671_object72501:head# 2024-03-20T09:04:59.896 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.896 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b80e0c0e:::benchmark_data_smithi122_119671_object56301:head# 2024-03-20T09:04:59.896 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.896 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b80ef054:::benchmark_data_smithi122_119671_object51244:head# 2024-03-20T09:04:59.896 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.896 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b80ffa3e:::benchmark_data_smithi122_119671_object57575:head# 2024-03-20T09:04:59.896 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.896 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b81009fb:::benchmark_data_smithi122_119671_object20049:head# 2024-03-20T09:04:59.896 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.896 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b810714b:::benchmark_data_smithi122_119671_object36804:head# 2024-03-20T09:04:59.896 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.896 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8107cc2:::benchmark_data_smithi122_119671_object82987:head# 2024-03-20T09:04:59.896 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.897 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8117d41:::benchmark_data_smithi122_119671_object69773:head# 2024-03-20T09:04:59.897 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.897 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8128b1f:::benchmark_data_smithi122_119671_object95893:head# 2024-03-20T09:04:59.897 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.897 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b81299f3:::benchmark_data_smithi122_119671_object37178:head# 2024-03-20T09:04:59.897 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.897 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8133784:::benchmark_data_smithi122_119671_object63647:head# 2024-03-20T09:04:59.897 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.897 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8136bb3:::benchmark_data_smithi122_119671_object88472:head# 2024-03-20T09:04:59.897 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.897 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b813ea46:::benchmark_data_smithi122_119671_object80804:head# 2024-03-20T09:04:59.897 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.897 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8147ca8:::benchmark_data_smithi122_119671_object36478:head# 2024-03-20T09:04:59.898 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.898 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b814fcd2:::benchmark_data_smithi122_119671_object28802:head# 2024-03-20T09:04:59.898 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.898 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8151f24:::benchmark_data_smithi122_119671_object18548:head# 2024-03-20T09:04:59.898 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.898 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8154d00:::benchmark_data_smithi122_119671_object15750:head# 2024-03-20T09:04:59.898 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.898 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8168b6b:::benchmark_data_smithi122_119671_object74433:head# 2024-03-20T09:04:59.898 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.898 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8169c09:::benchmark_data_smithi122_119671_object88014:head# 2024-03-20T09:04:59.898 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.898 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8170530:::benchmark_data_smithi122_119671_object82332:head# 2024-03-20T09:04:59.898 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.898 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8179c4c:::benchmark_data_smithi122_119671_object67903:head# 2024-03-20T09:04:59.899 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.899 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b817cf91:::benchmark_data_smithi122_119671_object96836:head# 2024-03-20T09:04:59.899 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.899 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b81822ab:::benchmark_data_smithi122_119671_object97128:head# 2024-03-20T09:04:59.899 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.899 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b81860be:::benchmark_data_smithi122_119671_object26728:head# 2024-03-20T09:04:59.899 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.899 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8188de5:::benchmark_data_smithi122_119671_object39634:head# 2024-03-20T09:04:59.899 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.899 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b818f3c0:::benchmark_data_smithi122_119671_object72695:head# 2024-03-20T09:04:59.899 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.899 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b81a2176:::benchmark_data_smithi122_119671_object68372:head# 2024-03-20T09:04:59.899 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.900 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b81a28d6:::benchmark_data_smithi122_119671_object43363:head# 2024-03-20T09:04:59.900 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.900 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b81acc4e:::benchmark_data_smithi122_119671_object15872:head# 2024-03-20T09:04:59.900 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.900 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b81b0201:::benchmark_data_smithi122_119671_object11053:head# 2024-03-20T09:04:59.900 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.900 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b81b124c:::benchmark_data_smithi122_119671_object79010:head# 2024-03-20T09:04:59.900 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.900 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b81b480f:::benchmark_data_smithi122_119671_object9680:head# 2024-03-20T09:04:59.901 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.901 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b81b6299:::benchmark_data_smithi122_119671_object15800:head# 2024-03-20T09:04:59.901 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.901 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b81b6c38:::benchmark_data_smithi122_119671_object76367:head# 2024-03-20T09:04:59.901 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.901 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b81d950e:::benchmark_data_smithi122_119671_object20291:head# 2024-03-20T09:04:59.901 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.901 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b81e839b:::benchmark_data_smithi122_119671_object2742:head# 2024-03-20T09:04:59.901 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.901 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8202855:::benchmark_data_smithi122_119671_object88471:head# 2024-03-20T09:04:59.901 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.901 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8204f8d:::benchmark_data_smithi122_119671_object97482:head# 2024-03-20T09:04:59.902 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.902 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b820b03a:::benchmark_data_smithi122_119671_object54490:head# 2024-03-20T09:04:59.902 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.902 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8212aed:::benchmark_data_smithi122_119671_object81501:head# 2024-03-20T09:04:59.902 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.902 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b821709b:::benchmark_data_smithi122_119671_object74732:head# 2024-03-20T09:04:59.902 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.902 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b821bd6e:::benchmark_data_smithi122_119671_object44976:head# 2024-03-20T09:04:59.902 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.902 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b82214ca:::benchmark_data_smithi122_119671_object92305:head# 2024-03-20T09:04:59.902 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.902 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8229cc9:::benchmark_data_smithi122_119671_object84921:head# 2024-03-20T09:04:59.902 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.903 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b823ae7f:::benchmark_data_smithi122_119671_object66023:head# 2024-03-20T09:04:59.903 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.903 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b825aeb6:::benchmark_data_smithi122_119671_object81073:head# 2024-03-20T09:04:59.903 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.903 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8267c81:::benchmark_data_smithi122_119671_object72063:head# 2024-03-20T09:04:59.903 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.903 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8268a4d:::benchmark_data_smithi122_119671_object44309:head# 2024-03-20T09:04:59.903 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.903 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b826b983:::benchmark_data_smithi122_119671_object1518:head# 2024-03-20T09:04:59.903 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.903 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8273313:::benchmark_data_smithi122_119671_object97936:head# 2024-03-20T09:04:59.903 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.903 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8296675:::benchmark_data_smithi122_119671_object24428:head# 2024-03-20T09:04:59.903 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.904 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8296db0:::benchmark_data_smithi122_119671_object71970:head# 2024-03-20T09:04:59.904 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.904 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8299ca2:::benchmark_data_smithi122_119671_object75094:head# 2024-03-20T09:04:59.904 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.904 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b82a3695:::benchmark_data_smithi122_119671_object43007:head# 2024-03-20T09:04:59.904 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.904 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b82a75de:::benchmark_data_smithi122_119671_object53994:head# 2024-03-20T09:04:59.904 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.904 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b82abf93:::benchmark_data_smithi122_119671_object68251:head# 2024-03-20T09:04:59.904 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.904 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b82af4a8:::benchmark_data_smithi122_119671_object54237:head# 2024-03-20T09:04:59.904 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.904 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b82b6d19:::benchmark_data_smithi122_119671_object6840:head# 2024-03-20T09:04:59.904 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.905 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b82b9821:::benchmark_data_smithi122_119671_object18342:head# 2024-03-20T09:04:59.905 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.905 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b82cd1d5:::benchmark_data_smithi122_119671_object19827:head# 2024-03-20T09:04:59.905 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.905 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b82dc905:::benchmark_data_smithi122_119671_object72997:head# 2024-03-20T09:04:59.905 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.905 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b82e3f5d:::benchmark_data_smithi122_119671_object26428:head# 2024-03-20T09:04:59.905 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.905 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b82ea7b3:::benchmark_data_smithi122_119671_object84007:head# 2024-03-20T09:04:59.905 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.905 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b830a67f:::benchmark_data_smithi122_119671_object30849:head# 2024-03-20T09:04:59.905 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.905 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b830ae7a:::benchmark_data_smithi122_119671_object93516:head# 2024-03-20T09:04:59.906 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.906 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8311b32:::benchmark_data_smithi122_119671_object17918:head# 2024-03-20T09:04:59.906 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.906 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b831dfa1:::benchmark_data_smithi122_119671_object50449:head# 2024-03-20T09:04:59.906 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.906 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b834826f:::benchmark_data_smithi122_119671_object58434:head# 2024-03-20T09:04:59.906 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.906 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8359478:::benchmark_data_smithi122_119671_object82960:head# 2024-03-20T09:04:59.906 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.906 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b835fd7a:::benchmark_data_smithi122_119671_object1839:head# 2024-03-20T09:04:59.906 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.906 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b836265b:::benchmark_data_smithi122_119671_object89854:head# 2024-03-20T09:04:59.907 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.907 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b83660d3:::benchmark_data_smithi122_119671_object92385:head# 2024-03-20T09:04:59.907 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.907 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b83688d7:::benchmark_data_smithi122_119671_object41792:head# 2024-03-20T09:04:59.907 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.907 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b837abd6:::benchmark_data_smithi122_119671_object14795:head# 2024-03-20T09:04:59.907 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.907 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8396819:::benchmark_data_smithi122_119671_object23565:head# 2024-03-20T09:04:59.907 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.907 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b83aa4f2:::benchmark_data_smithi122_119671_object68133:head# 2024-03-20T09:04:59.907 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.907 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b83ab218:::benchmark_data_smithi122_119671_object76003:head# 2024-03-20T09:04:59.907 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.907 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b83d564c:::benchmark_data_smithi122_119671_object62156:head# 2024-03-20T09:04:59.908 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.908 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b83df189:::benchmark_data_smithi122_119671_object45339:head# 2024-03-20T09:04:59.908 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.908 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b83e3062:::benchmark_data_smithi122_119671_object2747:head# 2024-03-20T09:04:59.908 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.908 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b83e3ce9:::benchmark_data_smithi122_119671_object74482:head# 2024-03-20T09:04:59.908 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.908 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b83ef985:::benchmark_data_smithi122_119671_object17998:head# 2024-03-20T09:04:59.908 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.908 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b83fd5e2:::benchmark_data_smithi122_119671_object78595:head# 2024-03-20T09:04:59.908 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.908 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8410f5e:::benchmark_data_smithi122_119671_object88715:head# 2024-03-20T09:04:59.908 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.908 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8417cb9:::benchmark_data_smithi122_119671_object55182:head# 2024-03-20T09:04:59.909 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.909 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8427d05:::benchmark_data_smithi122_119671_object90275:head# 2024-03-20T09:04:59.909 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.909 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b842e76f:::benchmark_data_smithi122_119671_object86557:head# 2024-03-20T09:04:59.909 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.909 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b84371b7:::benchmark_data_smithi122_119671_object75535:head# 2024-03-20T09:04:59.909 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.909 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b843daf8:::benchmark_data_smithi122_119671_object49103:head# 2024-03-20T09:04:59.909 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.909 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8441013:::benchmark_data_smithi122_119671_object73899:head# 2024-03-20T09:04:59.909 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.909 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b844af22:::benchmark_data_smithi122_119671_object38752:head# 2024-03-20T09:04:59.909 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.909 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b844be7b:::benchmark_data_smithi122_119671_object29238:head# 2024-03-20T09:04:59.910 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.910 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b845a16f:::benchmark_data_smithi122_119671_object47310:head# 2024-03-20T09:04:59.910 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.910 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b846e617:::benchmark_data_smithi122_119671_object58875:head# 2024-03-20T09:04:59.910 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.910 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8470302:::benchmark_data_smithi122_119671_object64541:head# 2024-03-20T09:04:59.910 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.910 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8473f26:::benchmark_data_smithi122_119671_object62027:head# 2024-03-20T09:04:59.910 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.910 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b847b42a:::benchmark_data_smithi122_119671_object42732:head# 2024-03-20T09:04:59.910 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.910 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8483761:::benchmark_data_smithi122_119671_object47377:head# 2024-03-20T09:04:59.910 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.911 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8484633:::benchmark_data_smithi122_119671_object65070:head# 2024-03-20T09:04:59.911 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.911 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8486c19:::benchmark_data_smithi122_119671_object92246:head# 2024-03-20T09:04:59.911 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.911 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b848c968:::benchmark_data_smithi122_119671_object41104:head# 2024-03-20T09:04:59.911 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.911 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b849b576:::benchmark_data_smithi122_119671_object1221:head# 2024-03-20T09:04:59.911 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.911 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b84a1600:::benchmark_data_smithi122_119671_object85328:head# 2024-03-20T09:04:59.912 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.912 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b84abced:::benchmark_data_smithi122_119671_object93549:head# 2024-03-20T09:04:59.912 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.912 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b84be974:::benchmark_data_smithi122_119671_object92245:head# 2024-03-20T09:04:59.912 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.912 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b84c9740:::benchmark_data_smithi122_119671_object63739:head# 2024-03-20T09:04:59.912 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.912 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b84da0ae:::benchmark_data_smithi122_119671_object9857:head# 2024-03-20T09:04:59.912 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.912 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b84dc184:::benchmark_data_smithi122_119671_object3006:head# 2024-03-20T09:04:59.912 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.912 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b850e72a:::benchmark_data_smithi122_119671_object22917:head# 2024-03-20T09:04:59.912 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.912 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b851681b:::benchmark_data_smithi122_119671_object53315:head# 2024-03-20T09:04:59.913 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.913 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8519a37:::benchmark_data_smithi122_119671_object40166:head# 2024-03-20T09:04:59.913 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.913 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b85581be:::benchmark_data_smithi122_119671_object97481:head# 2024-03-20T09:04:59.913 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.913 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8587608:::benchmark_data_smithi122_119671_object99769:head# 2024-03-20T09:04:59.913 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.913 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b858fa84:::benchmark_data_smithi122_119671_object41578:head# 2024-03-20T09:04:59.913 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.913 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b85935f3:::benchmark_data_smithi122_119671_object31082:head# 2024-03-20T09:04:59.914 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.914 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b859439f:::benchmark_data_smithi122_119671_object92641:head# 2024-03-20T09:04:59.914 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.914 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b85a113d:::benchmark_data_smithi122_119671_object44557:head# 2024-03-20T09:04:59.914 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.914 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b85ae805:::benchmark_data_smithi122_119671_object66188:head# 2024-03-20T09:04:59.914 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.914 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b85b0dd5:::benchmark_data_smithi122_119671_object42443:head# 2024-03-20T09:04:59.914 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.914 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b85caaf1:::benchmark_data_smithi122_119671_object59164:head# 2024-03-20T09:04:59.915 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.915 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b85cd9bb:::benchmark_data_smithi122_119671_object67050:head# 2024-03-20T09:04:59.915 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.915 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b85d064a:::benchmark_data_smithi122_119671_object36961:head# 2024-03-20T09:04:59.915 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.915 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b85d983b:::benchmark_data_smithi122_119671_object75774:head# 2024-03-20T09:04:59.915 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.915 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b85ed74b:::benchmark_data_smithi122_119671_object60079:head# 2024-03-20T09:04:59.915 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.915 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b85f98c8:::benchmark_data_smithi122_119671_object13180:head# 2024-03-20T09:04:59.915 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.915 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b85fa685:::benchmark_data_smithi122_119671_object28895:head# 2024-03-20T09:04:59.915 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.916 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b85fee2e:::benchmark_data_smithi122_119671_object73540:head# 2024-03-20T09:04:59.916 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.916 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8605cd7:::benchmark_data_smithi122_119671_object89503:head# 2024-03-20T09:04:59.916 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.916 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8615e71:::benchmark_last_metadata:head# 2024-03-20T09:04:59.916 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.916 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b861f63c:::benchmark_data_smithi122_119671_object86594:head# 2024-03-20T09:04:59.916 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.916 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b86209c1:::benchmark_data_smithi122_119671_object98303:head# 2024-03-20T09:04:59.916 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.916 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8620fd9:::benchmark_data_smithi122_119671_object64445:head# 2024-03-20T09:04:59.916 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.916 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b862746a:::benchmark_data_smithi122_119671_object66158:head# 2024-03-20T09:04:59.916 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.917 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b862e2e2:::benchmark_data_smithi122_119671_object67845:head# 2024-03-20T09:04:59.917 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.917 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8635a4a:::benchmark_data_smithi122_119671_object7857:head# 2024-03-20T09:04:59.917 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.917 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8636b44:::benchmark_data_smithi122_119671_object1613:head# 2024-03-20T09:04:59.917 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.917 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b863cc08:::benchmark_data_smithi122_119671_object42679:head# 2024-03-20T09:04:59.917 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.917 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b864b115:::benchmark_data_smithi122_119671_object59637:head# 2024-03-20T09:04:59.917 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.917 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b864f509:::benchmark_data_smithi122_119671_object62251:head# 2024-03-20T09:04:59.917 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.918 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8650a8c:::benchmark_data_smithi122_119671_object65833:head# 2024-03-20T09:04:59.918 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.918 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8652145:::benchmark_data_smithi122_119671_object12379:head# 2024-03-20T09:04:59.918 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.918 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b86573aa:::benchmark_data_smithi122_119671_object4056:head# 2024-03-20T09:04:59.918 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.918 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b866b87f:::benchmark_data_smithi122_119671_object12930:head# 2024-03-20T09:04:59.918 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.918 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b867a0ed:::benchmark_data_smithi122_119671_object16485:head# 2024-03-20T09:04:59.918 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.919 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8692db8:::benchmark_data_smithi122_119671_object10527:head# 2024-03-20T09:04:59.919 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.919 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b869634d:::benchmark_data_smithi122_119671_object40215:head# 2024-03-20T09:04:59.919 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.919 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8697444:::benchmark_data_smithi122_119671_object34965:head# 2024-03-20T09:04:59.919 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.919 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b86befdc:::benchmark_data_smithi122_119671_object69454:head# 2024-03-20T09:04:59.919 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.919 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b86cedb6:::benchmark_data_smithi122_119671_object82532:head# 2024-03-20T09:04:59.919 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.919 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b86cf43f:::benchmark_data_smithi122_119671_object45753:head# 2024-03-20T09:04:59.919 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.919 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b86cf902:::benchmark_data_smithi122_119671_object89219:head# 2024-03-20T09:04:59.920 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.920 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b86e58b3:::benchmark_data_smithi122_119671_object44453:head# 2024-03-20T09:04:59.920 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.920 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b86f8343:::benchmark_data_smithi122_119671_object27581:head# 2024-03-20T09:04:59.920 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.920 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8700b85:::benchmark_data_smithi122_119671_object54329:head# 2024-03-20T09:04:59.920 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.920 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b87143e0:::benchmark_data_smithi122_119671_object53522:head# 2024-03-20T09:04:59.920 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.921 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8733634:::benchmark_data_smithi122_119671_object24618:head# 2024-03-20T09:04:59.921 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.921 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b873d2e1:::benchmark_data_smithi122_119671_object7447:head# 2024-03-20T09:04:59.921 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.921 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b873fc0d:::benchmark_data_smithi122_119671_object43279:head# 2024-03-20T09:04:59.921 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.921 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8743862:::benchmark_data_smithi122_119671_object38483:head# 2024-03-20T09:04:59.921 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.921 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8746227:::benchmark_data_smithi122_119671_object80223:head# 2024-03-20T09:04:59.921 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.922 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8748eb8:::benchmark_data_smithi122_119671_object39784:head# 2024-03-20T09:04:59.922 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.922 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8755d86:::benchmark_data_smithi122_119671_object67794:head# 2024-03-20T09:04:59.922 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.922 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8763470:::benchmark_data_smithi122_119671_object25763:head# 2024-03-20T09:04:59.922 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.922 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8766c6a:::benchmark_data_smithi122_119671_object11327:head# 2024-03-20T09:04:59.922 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.922 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8771e09:::benchmark_data_smithi122_119671_object51078:head# 2024-03-20T09:04:59.922 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.922 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b877dd03:::benchmark_data_smithi122_119671_object17375:head# 2024-03-20T09:04:59.922 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.922 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b878104b:::benchmark_data_smithi122_119671_object32706:head# 2024-03-20T09:04:59.922 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.923 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8788aea:::benchmark_data_smithi122_119671_object5651:head# 2024-03-20T09:04:59.923 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.923 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b878e11b:::benchmark_data_smithi122_119671_object93526:head# 2024-03-20T09:04:59.923 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.923 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b87a2d08:::benchmark_data_smithi122_119671_object35383:head# 2024-03-20T09:04:59.923 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.923 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b87b71a8:::benchmark_data_smithi122_119671_object44866:head# 2024-03-20T09:04:59.923 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.923 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b87c8d7e:::benchmark_data_smithi122_119671_object62022:head# 2024-03-20T09:04:59.923 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.923 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b87d1b73:::benchmark_data_smithi122_119671_object8712:head# 2024-03-20T09:04:59.923 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.923 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b87d27b1:::benchmark_data_smithi122_119671_object88730:head# 2024-03-20T09:04:59.924 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.924 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b87e6e1e:::benchmark_data_smithi122_119671_object99191:head# 2024-03-20T09:04:59.924 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.924 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b87e8af1:::benchmark_data_smithi122_119671_object59064:head# 2024-03-20T09:04:59.924 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.924 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b87ea27f:::benchmark_data_smithi122_119671_object96479:head# 2024-03-20T09:04:59.924 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.924 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b87ea693:::benchmark_data_smithi122_119671_object64768:head# 2024-03-20T09:04:59.924 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.924 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b87f1890:::benchmark_data_smithi122_119671_object49989:head# 2024-03-20T09:04:59.924 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.924 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b88096b4:::benchmark_data_smithi122_119671_object49976:head# 2024-03-20T09:04:59.924 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.925 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8819e19:::benchmark_data_smithi122_119671_object80903:head# 2024-03-20T09:04:59.925 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.925 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b882f26e:::benchmark_data_smithi122_119671_object4814:head# 2024-03-20T09:04:59.925 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.925 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8849713:::benchmark_data_smithi122_119671_object20688:head# 2024-03-20T09:04:59.925 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.925 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b885e32d:::benchmark_data_smithi122_119671_object76055:head# 2024-03-20T09:04:59.925 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.925 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b886e33d:::benchmark_data_smithi122_119671_object24532:head# 2024-03-20T09:04:59.925 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.925 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8882a99:::benchmark_data_smithi122_119671_object65484:head# 2024-03-20T09:04:59.925 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.925 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8886c68:::benchmark_data_smithi122_119671_object58750:head# 2024-03-20T09:04:59.925 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.926 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b889d088:::benchmark_data_smithi122_119671_object72788:head# 2024-03-20T09:04:59.926 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.926 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b88a3411:::benchmark_data_smithi122_119671_object26339:head# 2024-03-20T09:04:59.926 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.926 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b88aee5b:::benchmark_data_smithi122_119671_object46409:head# 2024-03-20T09:04:59.926 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.926 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b88ba9af:::benchmark_data_smithi122_119671_object70515:head# 2024-03-20T09:04:59.926 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.926 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b88c8480:::benchmark_data_smithi122_119671_object60336:head# 2024-03-20T09:04:59.926 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.926 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b88d9e7a:::benchmark_data_smithi122_119671_object28091:head# 2024-03-20T09:04:59.926 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.927 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b88daa04:::benchmark_data_smithi122_119671_object81061:head# 2024-03-20T09:04:59.927 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.927 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b88e09d9:::benchmark_data_smithi122_119671_object35504:head# 2024-03-20T09:04:59.927 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.927 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b88fc2f6:::benchmark_data_smithi122_119671_object5597:head# 2024-03-20T09:04:59.927 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.928 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b891100c:::benchmark_data_smithi122_119671_object83940:head# 2024-03-20T09:04:59.928 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.928 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b89131c0:::benchmark_data_smithi122_119671_object67662:head# 2024-03-20T09:04:59.928 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.928 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b891b3ec:::benchmark_data_smithi122_119671_object94979:head# 2024-03-20T09:04:59.928 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.928 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8927c01:::benchmark_data_smithi122_119671_object13955:head# 2024-03-20T09:04:59.928 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.928 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8929f3a:::benchmark_data_smithi122_119671_object16661:head# 2024-03-20T09:04:59.929 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.929 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8938aa6:::benchmark_data_smithi122_119671_object21312:head# 2024-03-20T09:04:59.929 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.929 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b893a1ed:::benchmark_data_smithi122_119671_object59576:head# 2024-03-20T09:04:59.929 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.929 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8940d4d:::benchmark_data_smithi122_119671_object90270:head# 2024-03-20T09:04:59.929 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.929 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b894f64f:::benchmark_data_smithi122_119671_object47617:head# 2024-03-20T09:04:59.929 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.929 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8954849:::benchmark_data_smithi122_119671_object37336:head# 2024-03-20T09:04:59.930 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.930 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b896a67d:::benchmark_data_smithi122_119671_object55050:head# 2024-03-20T09:04:59.930 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.930 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b89945c6:::benchmark_data_smithi122_119671_object43812:head# 2024-03-20T09:04:59.930 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.930 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8996442:::benchmark_data_smithi122_119671_object28125:head# 2024-03-20T09:04:59.930 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.930 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8998378:::benchmark_data_smithi122_119671_object63888:head# 2024-03-20T09:04:59.930 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.930 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b899cc7f:::benchmark_data_smithi122_119671_object27653:head# 2024-03-20T09:04:59.930 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.930 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b89af1ff:::benchmark_data_smithi122_119671_object10767:head# 2024-03-20T09:04:59.931 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.931 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b89b0b55:::benchmark_data_smithi122_119671_object12893:head# 2024-03-20T09:04:59.931 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.931 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b89e5f63:::benchmark_data_smithi122_119671_object36363:head# 2024-03-20T09:04:59.931 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.931 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b89f594d:::benchmark_data_smithi122_119671_object20416:head# 2024-03-20T09:04:59.931 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.931 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8a0c9bd:::benchmark_data_smithi122_119671_object78613:head# 2024-03-20T09:04:59.931 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.931 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8a127b7:::benchmark_data_smithi122_119671_object75997:head# 2024-03-20T09:04:59.931 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.931 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8a18938:::benchmark_data_smithi122_119671_object64731:head# 2024-03-20T09:04:59.931 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.931 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8a2309e:::benchmark_data_smithi122_119671_object45130:head# 2024-03-20T09:04:59.932 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.932 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8a2c671:::benchmark_data_smithi122_119671_object69681:head# 2024-03-20T09:04:59.932 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.932 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8a3642c:::benchmark_data_smithi122_119671_object29187:head# 2024-03-20T09:04:59.932 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.932 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8a3b1f9:::benchmark_data_smithi122_119671_object5828:head# 2024-03-20T09:04:59.932 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.932 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8a40cb8:::benchmark_data_smithi122_119671_object69917:head# 2024-03-20T09:04:59.932 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.932 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8a46f3b:::benchmark_data_smithi122_119671_object36295:head# 2024-03-20T09:04:59.932 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.932 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8a50781:::benchmark_data_smithi122_119671_object14724:head# 2024-03-20T09:04:59.932 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.932 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8a685c1:::benchmark_data_smithi122_119671_object38686:head# 2024-03-20T09:04:59.933 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.933 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8a7a014:::benchmark_data_smithi122_119671_object12476:head# 2024-03-20T09:04:59.933 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.933 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8a8cc71:::benchmark_data_smithi122_119671_object83424:head# 2024-03-20T09:04:59.933 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.933 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8a8dd47:::benchmark_data_smithi122_119671_object83102:head# 2024-03-20T09:04:59.933 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.933 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8aa7734:::benchmark_data_smithi122_119671_object36202:head# 2024-03-20T09:04:59.933 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.933 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8ab657a:::benchmark_data_smithi122_119671_object73231:head# 2024-03-20T09:04:59.933 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.933 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8abe03d:::benchmark_data_smithi122_119671_object48514:head# 2024-03-20T09:04:59.934 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.934 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8ae8427:::benchmark_data_smithi122_119671_object58749:head# 2024-03-20T09:04:59.934 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.934 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8aecbc3:::benchmark_data_smithi122_119671_object3797:head# 2024-03-20T09:04:59.934 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.934 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8b0199c:::benchmark_data_smithi122_119671_object73548:head# 2024-03-20T09:04:59.934 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.934 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8b0e73e:::benchmark_data_smithi122_119671_object88630:head# 2024-03-20T09:04:59.934 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.934 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8b10637:::benchmark_data_smithi122_119671_object43953:head# 2024-03-20T09:04:59.934 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.935 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8b10f2b:::benchmark_data_smithi122_119671_object12032:head# 2024-03-20T09:04:59.935 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.935 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8b1df2a:::benchmark_data_smithi122_119671_object97161:head# 2024-03-20T09:04:59.935 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.935 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8b22e0b:::benchmark_data_smithi122_119671_object60226:head# 2024-03-20T09:04:59.935 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.935 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8b267e5:::benchmark_data_smithi122_119671_object55108:head# 2024-03-20T09:04:59.935 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.935 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8b27705:::benchmark_data_smithi122_119671_object68231:head# 2024-03-20T09:04:59.935 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.935 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8b324a0:::benchmark_data_smithi122_119671_object21969:head# 2024-03-20T09:04:59.935 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.935 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8b33d1e:::benchmark_data_smithi122_119671_object98531:head# 2024-03-20T09:04:59.936 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.936 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8b4030f:::benchmark_data_smithi122_119671_object2534:head# 2024-03-20T09:04:59.936 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.936 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8b42edd:::benchmark_data_smithi122_119671_object79798:head# 2024-03-20T09:04:59.936 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.936 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8b4cdbb:::benchmark_data_smithi122_119671_object42418:head# 2024-03-20T09:04:59.936 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.936 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8b5150f:::benchmark_data_smithi122_119671_object70956:head# 2024-03-20T09:04:59.936 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.936 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8b601db:::benchmark_data_smithi122_119671_object68404:head# 2024-03-20T09:04:59.936 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.936 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8b700ab:::benchmark_data_smithi122_119671_object52400:head# 2024-03-20T09:04:59.936 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.936 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8b70191:::benchmark_data_smithi122_119671_object87162:head# 2024-03-20T09:04:59.936 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.937 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8b8056c:::benchmark_data_smithi122_119671_object43130:head# 2024-03-20T09:04:59.937 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.937 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8b87560:::benchmark_data_smithi122_119671_object68852:head# 2024-03-20T09:04:59.937 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.937 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8b8e965:::benchmark_data_smithi122_119671_object83891:head# 2024-03-20T09:04:59.937 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.937 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8b97574:::benchmark_data_smithi122_119671_object88976:head# 2024-03-20T09:04:59.937 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.937 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8bac7be:::benchmark_data_smithi122_119671_object90076:head# 2024-03-20T09:04:59.937 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.937 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8bae45c:::benchmark_data_smithi122_119671_object71577:head# 2024-03-20T09:04:59.937 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.937 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8baf0f3:::benchmark_data_smithi122_119671_object29960:head# 2024-03-20T09:04:59.937 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.938 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8bb2aa5:::benchmark_data_smithi122_119671_object84206:head# 2024-03-20T09:04:59.938 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.938 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8bb9e7a:::benchmark_data_smithi122_119671_object62550:head# 2024-03-20T09:04:59.938 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.938 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8bbd115:::benchmark_data_smithi122_119671_object72218:head# 2024-03-20T09:04:59.938 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.938 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8bbfc5f:::benchmark_data_smithi122_119671_object92895:head# 2024-03-20T09:04:59.938 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.938 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8bd4acc:::benchmark_data_smithi122_119671_object86327:head# 2024-03-20T09:04:59.938 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.938 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8bd6582:::benchmark_data_smithi122_119671_object10196:head# 2024-03-20T09:04:59.938 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.938 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8be6b90:::benchmark_data_smithi122_119671_object10230:head# 2024-03-20T09:04:59.939 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.939 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8be8f89:::benchmark_data_smithi122_119671_object16890:head# 2024-03-20T09:04:59.939 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.939 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8bf6fd1:::benchmark_data_smithi122_119671_object46326:head# 2024-03-20T09:04:59.939 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.939 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8bfd623:::benchmark_data_smithi122_119671_object88318:head# 2024-03-20T09:04:59.939 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.939 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8c07091:::benchmark_data_smithi122_119671_object13623:head# 2024-03-20T09:04:59.939 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.939 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8c17004:::benchmark_data_smithi122_119671_object12413:head# 2024-03-20T09:04:59.940 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.940 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8c1763b:::benchmark_data_smithi122_119671_object89584:head# 2024-03-20T09:04:59.940 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.940 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8c200bc:::benchmark_data_smithi122_119671_object34988:head# 2024-03-20T09:04:59.940 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.940 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8c284ee:::benchmark_data_smithi122_119671_object35937:head# 2024-03-20T09:04:59.940 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.940 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8c2d3a4:::benchmark_data_smithi122_119671_object93965:head# 2024-03-20T09:04:59.940 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.940 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8c3ffa1:::benchmark_data_smithi122_119671_object61432:head# 2024-03-20T09:04:59.940 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.940 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8c5b828:::benchmark_data_smithi122_119671_object24765:head# 2024-03-20T09:04:59.940 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.940 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8c5e562:::benchmark_data_smithi122_119671_object61081:head# 2024-03-20T09:04:59.941 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.941 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8c6a095:::benchmark_data_smithi122_119671_object46135:head# 2024-03-20T09:04:59.941 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.941 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8c747f2:::benchmark_data_smithi122_119671_object78931:head# 2024-03-20T09:04:59.941 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.941 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8c7c592:::benchmark_data_smithi122_119671_object15864:head# 2024-03-20T09:04:59.941 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.941 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8c7d806:::benchmark_data_smithi122_119671_object37742:head# 2024-03-20T09:04:59.941 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.941 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8c8a149:::benchmark_data_smithi122_119671_object80787:head# 2024-03-20T09:04:59.941 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.941 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8cb6a28:::benchmark_data_smithi122_119671_object29517:head# 2024-03-20T09:04:59.941 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.941 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8cbfbf1:::benchmark_data_smithi122_119671_object32231:head# 2024-03-20T09:04:59.941 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.942 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8ce4304:::benchmark_data_smithi122_119671_object64678:head# 2024-03-20T09:04:59.942 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.942 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8ce5bb5:::benchmark_data_smithi122_119671_object9303:head# 2024-03-20T09:04:59.942 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.942 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8ced76c:::benchmark_data_smithi122_119671_object98007:head# 2024-03-20T09:04:59.942 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.942 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8cf76df:::benchmark_data_smithi122_119671_object94762:head# 2024-03-20T09:04:59.942 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.942 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8d00031:::benchmark_data_smithi122_119671_object44998:head# 2024-03-20T09:04:59.942 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.942 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8d18f6e:::benchmark_data_smithi122_119671_object48791:head# 2024-03-20T09:04:59.942 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.942 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8d21a6b:::benchmark_data_smithi122_119671_object79566:head# 2024-03-20T09:04:59.943 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.943 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8d2a0b8:::benchmark_data_smithi122_119671_object36713:head# 2024-03-20T09:04:59.943 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.943 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8d31a04:::benchmark_data_smithi122_119671_object28715:head# 2024-03-20T09:04:59.943 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.943 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8d3edf6:::benchmark_data_smithi122_119671_object26863:head# 2024-03-20T09:04:59.943 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.943 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8d45102:::benchmark_data_smithi122_119671_object24402:head# 2024-03-20T09:04:59.943 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.943 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8d51923:::benchmark_data_smithi122_119671_object36634:head# 2024-03-20T09:04:59.943 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.943 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8d54d85:::benchmark_data_smithi122_119671_object23025:head# 2024-03-20T09:04:59.943 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.943 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8d6416d:::benchmark_data_smithi122_119671_object29574:head# 2024-03-20T09:04:59.944 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.944 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8d65239:::benchmark_data_smithi122_119671_object6911:head# 2024-03-20T09:04:59.944 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.944 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8d740f0:::benchmark_data_smithi122_119671_object78560:head# 2024-03-20T09:04:59.944 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.944 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8d9153a:::benchmark_data_smithi122_119671_object11155:head# 2024-03-20T09:04:59.944 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.944 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8d97ae6:::benchmark_data_smithi122_119671_object23035:head# 2024-03-20T09:04:59.944 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.944 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8d9f08b:::benchmark_data_smithi122_119671_object16987:head# 2024-03-20T09:04:59.944 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.944 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8da387c:::benchmark_data_smithi122_119671_object78847:head# 2024-03-20T09:04:59.944 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.944 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8da67a4:::benchmark_data_smithi122_119671_object34991:head# 2024-03-20T09:04:59.945 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.945 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8dc150b:::benchmark_data_smithi122_119671_object63087:head# 2024-03-20T09:04:59.945 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.945 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8dca9f5:::benchmark_data_smithi122_119671_object78646:head# 2024-03-20T09:04:59.945 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.945 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8dd16e3:::benchmark_data_smithi122_119671_object86404:head# 2024-03-20T09:04:59.945 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.945 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8dd491b:::benchmark_data_smithi122_119671_object36842:head# 2024-03-20T09:04:59.945 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.945 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8ddbaa3:::benchmark_data_smithi122_119671_object34432:head# 2024-03-20T09:04:59.945 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.945 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8de255b:::benchmark_data_smithi122_119671_object5583:head# 2024-03-20T09:04:59.945 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.945 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8de2621:::benchmark_data_smithi122_119671_object54238:head# 2024-03-20T09:04:59.945 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.946 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8decd5d:::benchmark_data_smithi122_119671_object93377:head# 2024-03-20T09:04:59.946 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.946 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8deefaf:::benchmark_data_smithi122_119671_object20465:head# 2024-03-20T09:04:59.946 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.946 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8df4194:::benchmark_data_smithi122_119671_object76485:head# 2024-03-20T09:04:59.946 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.946 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8e0085c:::benchmark_data_smithi122_119671_object31272:head# 2024-03-20T09:04:59.946 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.946 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8e0ab25:::benchmark_data_smithi122_119671_object47572:head# 2024-03-20T09:04:59.946 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.946 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8e216ea:::benchmark_data_smithi122_119671_object55934:head# 2024-03-20T09:04:59.946 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.946 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8e2922e:::benchmark_data_smithi122_119671_object30961:head# 2024-03-20T09:04:59.946 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.947 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8e2ad4f:::benchmark_data_smithi122_119671_object36924:head# 2024-03-20T09:04:59.947 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.947 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8e380a9:::benchmark_data_smithi122_119671_object97262:head# 2024-03-20T09:04:59.947 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.947 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8e3f74f:::benchmark_data_smithi122_119671_object44956:head# 2024-03-20T09:04:59.947 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.947 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8e55598:::benchmark_data_smithi122_119671_object64515:head# 2024-03-20T09:04:59.947 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.947 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8e7422a:::benchmark_data_smithi122_119671_object97619:head# 2024-03-20T09:04:59.947 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.947 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8e89798:::benchmark_data_smithi122_119671_object73894:head# 2024-03-20T09:04:59.947 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.947 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8eb2918:::benchmark_data_smithi122_119671_object71652:head# 2024-03-20T09:04:59.947 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.947 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8ebbf78:::benchmark_data_smithi122_119671_object32942:head# 2024-03-20T09:04:59.948 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.948 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8ec4946:::benchmark_data_smithi122_119671_object79676:head# 2024-03-20T09:04:59.948 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:04:59.948 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:04:59 smithi115 ceph-mon[72476]: 3.1 deep-scrub ok 2024-03-20T09:04:59.950 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:04:59.952 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:04:59 smithi122 ceph-mon[73495]: 3.1 deep-scrub ok 2024-03-20T09:04:59.986 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:00.082 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:05:00.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:00.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:05:00.194 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:05:00.264 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:05:00.373 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8ece423:::benchmark_data_smithi122_119671_object1065:head# 2024-03-20T09:05:00.373 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.373 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8ed88c7:::benchmark_data_smithi122_119671_object39997:head# 2024-03-20T09:05:00.373 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.373 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8ef3d30:::benchmark_data_smithi122_119671_object35995:head# 2024-03-20T09:05:00.374 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.374 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8ef4609:::benchmark_data_smithi122_119671_object23871:head# 2024-03-20T09:05:00.374 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.374 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8f05312:::benchmark_data_smithi122_119671_object34149:head# 2024-03-20T09:05:00.374 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.374 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8f05a0f:::benchmark_data_smithi122_119671_object82719:head# 2024-03-20T09:05:00.374 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.374 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8f14219:::benchmark_data_smithi122_119671_object49410:head# 2024-03-20T09:05:00.374 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.374 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8f23c75:::benchmark_data_smithi122_119671_object99381:head# 2024-03-20T09:05:00.374 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.374 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8f2cc87:::benchmark_data_smithi122_119671_object34464:head# 2024-03-20T09:05:00.374 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.375 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8f2fabf:::benchmark_data_smithi122_119671_object3369:head# 2024-03-20T09:05:00.375 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.375 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8f41033:::benchmark_data_smithi122_119671_object63458:head# 2024-03-20T09:05:00.375 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.375 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8f428df:::benchmark_data_smithi122_119671_object81213:head# 2024-03-20T09:05:00.375 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.375 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8f44bad:::benchmark_data_smithi122_119671_object58567:head# 2024-03-20T09:05:00.375 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.375 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8f5234a:::benchmark_data_smithi122_119671_object8867:head# 2024-03-20T09:05:00.375 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.375 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8f69853:::benchmark_data_smithi122_119671_object28064:head# 2024-03-20T09:05:00.375 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.375 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8f736a5:::benchmark_data_smithi122_119671_object10021:head# 2024-03-20T09:05:00.375 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.375 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8f7a2ee:::benchmark_data_smithi122_119671_object25768:head# 2024-03-20T09:05:00.375 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.376 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8f7ce74:::benchmark_data_smithi122_119671_object18370:head# 2024-03-20T09:05:00.376 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.376 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8f82fab:::benchmark_data_smithi122_119671_object55374:head# 2024-03-20T09:05:00.376 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.376 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8f9102b:::benchmark_data_smithi122_119671_object77709:head# 2024-03-20T09:05:00.376 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.376 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8f9cab8:::benchmark_data_smithi122_119671_object61826:head# 2024-03-20T09:05:00.376 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.376 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8f9f4e0:::benchmark_data_smithi122_119671_object71523:head# 2024-03-20T09:05:00.376 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.376 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8faa30a:::benchmark_data_smithi122_119671_object1300:head# 2024-03-20T09:05:00.376 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.376 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8facbf8:::benchmark_data_smithi122_119671_object86230:head# 2024-03-20T09:05:00.376 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.377 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8fae347:::benchmark_data_smithi122_119671_object16452:head# 2024-03-20T09:05:00.377 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.377 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8fb902e:::benchmark_data_smithi122_119671_object45083:head# 2024-03-20T09:05:00.377 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.377 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8fc4c79:::benchmark_data_smithi122_119671_object77732:head# 2024-03-20T09:05:00.377 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.377 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8fcb669:::benchmark_data_smithi122_119671_object35609:head# 2024-03-20T09:05:00.377 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.377 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8fd0810:::benchmark_data_smithi122_119671_object64189:head# 2024-03-20T09:05:00.377 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.377 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8fd7e92:::benchmark_data_smithi122_119671_object86303:head# 2024-03-20T09:05:00.377 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.377 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8fe7588:::benchmark_data_smithi122_119671_object44491:head# 2024-03-20T09:05:00.378 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.378 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8fee053:::benchmark_data_smithi122_119671_object85541:head# 2024-03-20T09:05:00.378 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.378 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b8ff7ab1:::benchmark_data_smithi122_119671_object64068:head# 2024-03-20T09:05:00.378 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.378 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b900722a:::benchmark_data_smithi122_119671_object48408:head# 2024-03-20T09:05:00.378 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.378 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b900db6c:::benchmark_data_smithi122_119671_object11840:head# 2024-03-20T09:05:00.378 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.378 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9019f56:::benchmark_data_smithi122_119671_object7578:head# 2024-03-20T09:05:00.378 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.378 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9043c7d:::benchmark_data_smithi122_119671_object52338:head# 2024-03-20T09:05:00.378 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.378 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b904f8b8:::benchmark_data_smithi122_119671_object78537:head# 2024-03-20T09:05:00.379 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.379 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b90537c1:::benchmark_data_smithi122_119671_object91972:head# 2024-03-20T09:05:00.379 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.379 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b905429b:::benchmark_data_smithi122_119671_object78224:head# 2024-03-20T09:05:00.379 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.379 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b905d63b:::benchmark_data_smithi122_119671_object56952:head# 2024-03-20T09:05:00.379 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.379 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9063833:::benchmark_data_smithi122_119671_object9495:head# 2024-03-20T09:05:00.379 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.379 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b907dae6:::benchmark_data_smithi122_119671_object15594:head# 2024-03-20T09:05:00.379 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.379 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b908409e:::benchmark_data_smithi122_119671_object59541:head# 2024-03-20T09:05:00.379 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.379 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9087d89:::benchmark_data_smithi122_119671_object88270:head# 2024-03-20T09:05:00.379 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.379 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b908d36e:::benchmark_data_smithi122_119671_object8961:head# 2024-03-20T09:05:00.380 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.380 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b909330e:::benchmark_data_smithi122_119671_object40141:head# 2024-03-20T09:05:00.380 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.380 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b909b155:::benchmark_data_smithi122_119671_object66697:head# 2024-03-20T09:05:00.380 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.380 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b90a53fc:::benchmark_data_smithi122_119671_object37256:head# 2024-03-20T09:05:00.380 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.380 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b90acc01:::benchmark_data_smithi122_119671_object30867:head# 2024-03-20T09:05:00.380 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.380 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b90acfac:::benchmark_data_smithi122_119671_object15028:head# 2024-03-20T09:05:00.380 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.380 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b90b75c6:::benchmark_data_smithi122_119671_object53480:head# 2024-03-20T09:05:00.380 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.380 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b90b8999:::benchmark_data_smithi122_119671_object70877:head# 2024-03-20T09:05:00.381 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.381 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b90bc89c:::benchmark_data_smithi122_119671_object48200:head# 2024-03-20T09:05:00.381 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.381 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b90c58c3:::benchmark_data_smithi122_119671_object81286:head# 2024-03-20T09:05:00.381 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.381 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b90ca221:::benchmark_data_smithi122_119671_object27128:head# 2024-03-20T09:05:00.381 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.381 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b90cb372:::benchmark_data_smithi122_119671_object98075:head# 2024-03-20T09:05:00.381 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.381 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b90d3ffb:::benchmark_data_smithi122_119671_object46258:head# 2024-03-20T09:05:00.381 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.381 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b90d924d:::benchmark_data_smithi122_119671_object72610:head# 2024-03-20T09:05:00.381 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.381 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b90ec698:::benchmark_data_smithi122_119671_object54664:head# 2024-03-20T09:05:00.381 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.382 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b90f8a34:::benchmark_data_smithi122_119671_object64648:head# 2024-03-20T09:05:00.382 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.382 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b90f9e9b:::benchmark_data_smithi122_119671_object11002:head# 2024-03-20T09:05:00.382 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.382 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9118a5a:::benchmark_data_smithi122_119671_object2568:head# 2024-03-20T09:05:00.382 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.382 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9131ec0:::benchmark_data_smithi122_119671_object86386:head# 2024-03-20T09:05:00.382 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.382 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9158141:::benchmark_data_smithi122_119671_object38674:head# 2024-03-20T09:05:00.382 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.382 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b915ab48:::benchmark_data_smithi122_119671_object79509:head# 2024-03-20T09:05:00.382 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.382 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b915df4f:::benchmark_data_smithi122_119671_object87068:head# 2024-03-20T09:05:00.382 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.383 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b915e834:::benchmark_data_smithi122_119671_object94794:head# 2024-03-20T09:05:00.383 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.383 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9160479:::benchmark_data_smithi122_119671_object37192:head# 2024-03-20T09:05:00.383 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.383 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9160e98:::benchmark_data_smithi122_119671_object49154:head# 2024-03-20T09:05:00.383 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.383 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9174135:::benchmark_data_smithi122_119671_object21735:head# 2024-03-20T09:05:00.383 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.383 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9174ef9:::benchmark_data_smithi122_119671_object86677:head# 2024-03-20T09:05:00.383 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.383 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9176c96:::benchmark_data_smithi122_119671_object11738:head# 2024-03-20T09:05:00.383 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.383 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9177e75:::benchmark_data_smithi122_119671_object19925:head# 2024-03-20T09:05:00.383 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.383 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b91855c9:::benchmark_data_smithi122_119671_object18137:head# 2024-03-20T09:05:00.384 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.384 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9189a5a:::benchmark_data_smithi122_119671_object13838:head# 2024-03-20T09:05:00.384 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.384 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9197ff9:::benchmark_data_smithi122_119671_object25265:head# 2024-03-20T09:05:00.384 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.384 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9199107:::benchmark_data_smithi122_119671_object73641:head# 2024-03-20T09:05:00.384 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.384 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b91a7e1d:::benchmark_data_smithi122_119671_object31352:head# 2024-03-20T09:05:00.384 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.384 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b91baa5e:::benchmark_data_smithi122_119671_object66975:head# 2024-03-20T09:05:00.384 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.384 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b91cf95d:::benchmark_data_smithi122_119671_object10573:head# 2024-03-20T09:05:00.384 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.384 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b91d07bb:::benchmark_data_smithi122_119671_object42529:head# 2024-03-20T09:05:00.384 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.384 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b91d38c7:::benchmark_data_smithi122_119671_object92814:head# 2024-03-20T09:05:00.385 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.385 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b91e6a75:::benchmark_data_smithi122_119671_object66686:head# 2024-03-20T09:05:00.385 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.385 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b91f2091:::benchmark_data_smithi122_119671_object2184:head# 2024-03-20T09:05:00.385 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.385 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b91f4598:::benchmark_data_smithi122_119671_object22343:head# 2024-03-20T09:05:00.385 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.385 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9200b46:::benchmark_data_smithi122_119671_object74133:head# 2024-03-20T09:05:00.385 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.385 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b920402f:::benchmark_data_smithi122_119671_object59176:head# 2024-03-20T09:05:00.385 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.385 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b920fddc:::benchmark_data_smithi122_119671_object93782:head# 2024-03-20T09:05:00.386 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.386 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b92140ce:::benchmark_data_smithi122_119671_object11226:head# 2024-03-20T09:05:00.386 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.386 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b922615c:::benchmark_data_smithi122_119671_object79505:head# 2024-03-20T09:05:00.386 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.386 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9233030:::benchmark_data_smithi122_119671_object36251:head# 2024-03-20T09:05:00.386 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.386 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b923e7f2:::benchmark_data_smithi122_119671_object33709:head# 2024-03-20T09:05:00.386 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.386 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b92437d9:::benchmark_data_smithi122_119671_object25391:head# 2024-03-20T09:05:00.386 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.386 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9248625:::benchmark_data_smithi122_119671_object57894:head# 2024-03-20T09:05:00.386 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.387 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9256901:::benchmark_data_smithi122_119671_object54057:head# 2024-03-20T09:05:00.387 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.387 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b927748b:::benchmark_data_smithi122_119671_object18895:head# 2024-03-20T09:05:00.387 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.387 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9278251:::benchmark_data_smithi122_119671_object30760:head# 2024-03-20T09:05:00.387 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.387 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b928133e:::benchmark_data_smithi122_119671_object84785:head# 2024-03-20T09:05:00.387 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.387 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9288f57:::benchmark_data_smithi122_119671_object47946:head# 2024-03-20T09:05:00.387 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.387 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b928ac8d:::benchmark_data_smithi122_119671_object47876:head# 2024-03-20T09:05:00.387 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.387 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b928c174:::benchmark_data_smithi122_119671_object21395:head# 2024-03-20T09:05:00.387 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.388 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9293eb1:::benchmark_data_smithi122_119671_object17432:head# 2024-03-20T09:05:00.388 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.388 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b929da67:::benchmark_data_smithi122_119671_object32965:head# 2024-03-20T09:05:00.388 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.388 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b92a7c47:::benchmark_data_smithi122_119671_object30782:head# 2024-03-20T09:05:00.388 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.388 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b92d1194:::benchmark_data_smithi122_119671_object53741:head# 2024-03-20T09:05:00.388 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.388 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b92d3f2d:::benchmark_data_smithi122_119671_object62000:head# 2024-03-20T09:05:00.388 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.388 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b92df8bb:::benchmark_data_smithi122_119671_object9745:head# 2024-03-20T09:05:00.388 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.388 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b92e61bd:::benchmark_data_smithi122_119671_object3032:head# 2024-03-20T09:05:00.388 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.389 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b92eaf42:::benchmark_data_smithi122_119671_object29580:head# 2024-03-20T09:05:00.389 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.389 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b92f2ad2:::benchmark_data_smithi122_119671_object60367:head# 2024-03-20T09:05:00.389 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.389 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b930409f:::benchmark_data_smithi122_119671_object12942:head# 2024-03-20T09:05:00.389 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.389 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b931e70d:::benchmark_data_smithi122_119671_object24916:head# 2024-03-20T09:05:00.389 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.389 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b932d345:::benchmark_data_smithi122_119671_object40701:head# 2024-03-20T09:05:00.389 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.389 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9355175:::benchmark_data_smithi122_119671_object91862:head# 2024-03-20T09:05:00.389 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.389 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9357e85:::benchmark_data_smithi122_119671_object27226:head# 2024-03-20T09:05:00.389 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.390 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9361b94:::benchmark_data_smithi122_119671_object99153:head# 2024-03-20T09:05:00.390 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.390 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b93776be:::benchmark_data_smithi122_119671_object27390:head# 2024-03-20T09:05:00.390 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.390 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b937b627:::benchmark_data_smithi122_119671_object17129:head# 2024-03-20T09:05:00.390 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.390 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b937bc76:::benchmark_data_smithi122_119671_object62651:head# 2024-03-20T09:05:00.390 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.390 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9389186:::benchmark_data_smithi122_119671_object53020:head# 2024-03-20T09:05:00.390 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.390 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b938b7f2:::benchmark_data_smithi122_119671_object96325:head# 2024-03-20T09:05:00.390 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.390 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b93925aa:::benchmark_data_smithi122_119671_object1935:head# 2024-03-20T09:05:00.391 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.391 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b939becd:::benchmark_data_smithi122_119671_object2067:head# 2024-03-20T09:05:00.391 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.391 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b93b04b5:::benchmark_data_smithi122_119671_object72497:head# 2024-03-20T09:05:00.391 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.391 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b93b1843:::benchmark_data_smithi122_119671_object83040:head# 2024-03-20T09:05:00.391 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.391 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b93b756f:::benchmark_data_smithi122_119671_object56002:head# 2024-03-20T09:05:00.391 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.391 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b93b88b5:::benchmark_data_smithi122_119671_object59078:head# 2024-03-20T09:05:00.391 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.391 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b93c2489:::benchmark_data_smithi122_119671_object5620:head# 2024-03-20T09:05:00.391 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.392 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b93c4647:::benchmark_data_smithi122_119671_object43940:head# 2024-03-20T09:05:00.392 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.392 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b93d63b8:::benchmark_data_smithi122_119671_object66828:head# 2024-03-20T09:05:00.392 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.392 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b93db787:::benchmark_data_smithi122_119671_object31992:head# 2024-03-20T09:05:00.392 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.392 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b93e0d5e:::benchmark_data_smithi122_119671_object28738:head# 2024-03-20T09:05:00.392 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.392 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b93efd92:::benchmark_data_smithi122_119671_object14418:head# 2024-03-20T09:05:00.392 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.392 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b93f3da3:::benchmark_data_smithi122_119671_object95038:head# 2024-03-20T09:05:00.392 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.392 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b93f74b5:::benchmark_data_smithi122_119671_object28941:head# 2024-03-20T09:05:00.392 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.393 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9402414:::benchmark_data_smithi122_119671_object64682:head# 2024-03-20T09:05:00.393 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.393 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b94074ba:::benchmark_data_smithi122_119671_object67960:head# 2024-03-20T09:05:00.393 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.393 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b940dea0:::benchmark_data_smithi122_119671_object74923:head# 2024-03-20T09:05:00.393 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.393 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9414979:::benchmark_data_smithi122_119671_object30957:head# 2024-03-20T09:05:00.393 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.393 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9435cd2:::benchmark_data_smithi122_119671_object84714:head# 2024-03-20T09:05:00.393 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.393 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b94381c1:::benchmark_data_smithi122_119671_object20890:head# 2024-03-20T09:05:00.393 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.393 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9446cca:::benchmark_data_smithi122_119671_object97261:head# 2024-03-20T09:05:00.394 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.394 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9446e62:::benchmark_data_smithi122_119671_object47843:head# 2024-03-20T09:05:00.394 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.394 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b94491f7:::benchmark_data_smithi122_119671_object86470:head# 2024-03-20T09:05:00.394 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.394 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b94559a6:::benchmark_data_smithi122_119671_object50588:head# 2024-03-20T09:05:00.394 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.394 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9458212:::benchmark_data_smithi122_119671_object16986:head# 2024-03-20T09:05:00.394 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.394 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b945c112:::benchmark_data_smithi122_119671_object47314:head# 2024-03-20T09:05:00.394 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.394 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9461258:::benchmark_data_smithi122_119671_object62310:head# 2024-03-20T09:05:00.394 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.395 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b947016f:::benchmark_data_smithi122_119671_object72459:head# 2024-03-20T09:05:00.395 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.395 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b94774d5:::benchmark_data_smithi122_119671_object77610:head# 2024-03-20T09:05:00.395 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.395 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b947ac8d:::benchmark_data_smithi122_119671_object6848:head# 2024-03-20T09:05:00.395 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.395 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b947f9f1:::benchmark_data_smithi122_119671_object2619:head# 2024-03-20T09:05:00.395 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.395 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9481e76:::benchmark_data_smithi122_119671_object98578:head# 2024-03-20T09:05:00.395 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.395 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9483ba2:::benchmark_data_smithi122_119671_object36449:head# 2024-03-20T09:05:00.395 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.395 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b948c72a:::benchmark_data_smithi122_119671_object95633:head# 2024-03-20T09:05:00.395 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.396 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b94a7244:::benchmark_data_smithi122_119671_object16390:head# 2024-03-20T09:05:00.396 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.396 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b94a9f18:::benchmark_data_smithi122_119671_object63319:head# 2024-03-20T09:05:00.396 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.396 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b94b5a8b:::benchmark_data_smithi122_119671_object99092:head# 2024-03-20T09:05:00.396 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.396 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b94b7e10:::benchmark_data_smithi122_119671_object79172:head# 2024-03-20T09:05:00.396 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.396 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b94cca34:::benchmark_data_smithi122_119671_object84795:head# 2024-03-20T09:05:00.396 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.396 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b94d4ed6:::benchmark_data_smithi122_119671_object86656:head# 2024-03-20T09:05:00.396 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.396 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b94db033:::benchmark_data_smithi122_119671_object42208:head# 2024-03-20T09:05:00.396 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.397 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b94e6699:::benchmark_data_smithi122_119671_object78878:head# 2024-03-20T09:05:00.397 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.397 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b94eaac4:::benchmark_data_smithi122_119671_object57482:head# 2024-03-20T09:05:00.397 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.397 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b94ec385:::benchmark_data_smithi122_119671_object99739:head# 2024-03-20T09:05:00.397 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.397 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b94fa278:::benchmark_data_smithi122_119671_object91759:head# 2024-03-20T09:05:00.397 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.397 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9505ed4:::benchmark_data_smithi122_119671_object77766:head# 2024-03-20T09:05:00.397 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.397 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9509231:::benchmark_data_smithi122_119671_object51606:head# 2024-03-20T09:05:00.397 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.397 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9521450:::benchmark_data_smithi122_119671_object23130:head# 2024-03-20T09:05:00.397 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.398 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9534c64:::benchmark_data_smithi122_119671_object58722:head# 2024-03-20T09:05:00.398 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.398 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9539eae:::benchmark_data_smithi122_119671_object4126:head# 2024-03-20T09:05:00.398 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.398 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b953c4ff:::benchmark_data_smithi122_119671_object82291:head# 2024-03-20T09:05:00.398 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.398 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9549cde:::benchmark_data_smithi122_119671_object18655:head# 2024-03-20T09:05:00.398 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.398 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b954c1ae:::benchmark_data_smithi122_119671_object71853:head# 2024-03-20T09:05:00.398 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.398 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b954f05b:::benchmark_data_smithi122_119671_object8728:head# 2024-03-20T09:05:00.398 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.399 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b955bbec:::benchmark_data_smithi122_119671_object2898:head# 2024-03-20T09:05:00.399 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.399 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9563bc2:::benchmark_data_smithi122_119671_object61277:head# 2024-03-20T09:05:00.399 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.399 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b956858c:::benchmark_data_smithi122_119671_object54545:head# 2024-03-20T09:05:00.399 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.399 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b956b3f8:::benchmark_data_smithi122_119671_object20613:head# 2024-03-20T09:05:00.399 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.399 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9572b44:::benchmark_data_smithi122_119671_object66151:head# 2024-03-20T09:05:00.399 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.399 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b958936a:::benchmark_data_smithi122_119671_object34291:head# 2024-03-20T09:05:00.399 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.399 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9589a9e:::benchmark_data_smithi122_119671_object98916:head# 2024-03-20T09:05:00.399 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.400 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b958d381:::benchmark_data_smithi122_119671_object75429:head# 2024-03-20T09:05:00.400 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.400 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9591ec5:::benchmark_data_smithi122_119671_object92865:head# 2024-03-20T09:05:00.400 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.400 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9594b15:::benchmark_data_smithi122_119671_object42796:head# 2024-03-20T09:05:00.400 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.400 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b95967fa:::benchmark_data_smithi122_119671_object8756:head# 2024-03-20T09:05:00.400 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.400 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b959942c:::benchmark_data_smithi122_119671_object2909:head# 2024-03-20T09:05:00.400 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.400 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b95a60c8:::benchmark_data_smithi122_119671_object95745:head# 2024-03-20T09:05:00.400 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.400 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b95b137a:::benchmark_data_smithi122_119671_object91048:head# 2024-03-20T09:05:00.400 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.401 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b95bfbfe:::benchmark_data_smithi122_119671_object51398:head# 2024-03-20T09:05:00.401 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.401 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b95c6009:::benchmark_data_smithi122_119671_object65128:head# 2024-03-20T09:05:00.401 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.401 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b95f5697:::benchmark_data_smithi122_119671_object73848:head# 2024-03-20T09:05:00.401 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.401 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b95febc4:::benchmark_data_smithi122_119671_object95843:head# 2024-03-20T09:05:00.401 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.401 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b961b33e:::benchmark_data_smithi122_119671_object93447:head# 2024-03-20T09:05:00.401 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.401 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9620096:::benchmark_data_smithi122_119671_object34064:head# 2024-03-20T09:05:00.401 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.401 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b96225bc:::benchmark_data_smithi122_119671_object62452:head# 2024-03-20T09:05:00.402 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.402 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b962320d:::benchmark_data_smithi122_119671_object58838:head# 2024-03-20T09:05:00.402 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.402 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9626240:::benchmark_data_smithi122_119671_object83211:head# 2024-03-20T09:05:00.402 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.402 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b962af8f:::benchmark_data_smithi122_119671_object13230:head# 2024-03-20T09:05:00.402 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.402 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b962b557:::benchmark_data_smithi122_119671_object18735:head# 2024-03-20T09:05:00.402 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.402 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b962fb5f:::benchmark_data_smithi122_119671_object91725:head# 2024-03-20T09:05:00.402 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.402 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9631498:::benchmark_data_smithi122_119671_object69764:head# 2024-03-20T09:05:00.402 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.403 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b96343ff:::benchmark_data_smithi122_119671_object99274:head# 2024-03-20T09:05:00.403 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.403 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9634acb:::benchmark_data_smithi122_119671_object10797:head# 2024-03-20T09:05:00.403 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.403 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9639325:::benchmark_data_smithi122_119671_object77348:head# 2024-03-20T09:05:00.403 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.403 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9639cb5:::benchmark_data_smithi122_119671_object17154:head# 2024-03-20T09:05:00.403 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.403 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9644b24:::benchmark_data_smithi122_119671_object55747:head# 2024-03-20T09:05:00.403 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.403 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b965f38c:::benchmark_data_smithi122_119671_object53892:head# 2024-03-20T09:05:00.403 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.403 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b966ff50:::benchmark_data_smithi122_119671_object85091:head# 2024-03-20T09:05:00.403 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.404 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b96703b8:::benchmark_data_smithi122_119671_object42246:head# 2024-03-20T09:05:00.404 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.404 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b967624c:::benchmark_data_smithi122_119671_object97369:head# 2024-03-20T09:05:00.404 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.404 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b96766f0:::benchmark_data_smithi122_119671_object41636:head# 2024-03-20T09:05:00.404 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.404 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9695070:::benchmark_data_smithi122_119671_object37577:head# 2024-03-20T09:05:00.404 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.404 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b96a9ffe:::benchmark_data_smithi122_119671_object5452:head# 2024-03-20T09:05:00.404 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.404 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b96b5258:::benchmark_data_smithi122_119671_object43978:head# 2024-03-20T09:05:00.404 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.404 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b96bbd5c:::benchmark_data_smithi122_119671_object7374:head# 2024-03-20T09:05:00.405 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.405 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b96cff93:::benchmark_data_smithi122_119671_object34609:head# 2024-03-20T09:05:00.405 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.405 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b96ddae3:::benchmark_data_smithi122_119671_object97627:head# 2024-03-20T09:05:00.405 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.405 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b96e22f9:::benchmark_data_smithi122_119671_object46283:head# 2024-03-20T09:05:00.405 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.405 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b96e6d44:::benchmark_data_smithi122_119671_object23711:head# 2024-03-20T09:05:00.405 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.405 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b96e7fcf:::benchmark_data_smithi122_119671_object62398:head# 2024-03-20T09:05:00.405 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.405 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b96f73ef:::benchmark_data_smithi122_119671_object87672:head# 2024-03-20T09:05:00.405 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.405 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b97071bf:::benchmark_data_smithi122_119671_object3112:head# 2024-03-20T09:05:00.406 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.406 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b97100c3:::benchmark_data_smithi122_119671_object98487:head# 2024-03-20T09:05:00.406 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.406 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b971917c:::benchmark_data_smithi122_119671_object73519:head# 2024-03-20T09:05:00.406 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.406 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b97280ef:::benchmark_data_smithi122_119671_object36451:head# 2024-03-20T09:05:00.406 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.406 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9728bd5:::benchmark_data_smithi122_119671_object97086:head# 2024-03-20T09:05:00.406 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.406 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b97438ea:::benchmark_data_smithi122_119671_object49953:head# 2024-03-20T09:05:00.406 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.406 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b974a3b5:::benchmark_data_smithi122_119671_object31070:head# 2024-03-20T09:05:00.406 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.407 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b975443d:::benchmark_data_smithi122_119671_object31307:head# 2024-03-20T09:05:00.407 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.407 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b975eb36:::benchmark_data_smithi122_119671_object79479:head# 2024-03-20T09:05:00.407 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.407 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b97639c7:::benchmark_data_smithi122_119671_object19744:head# 2024-03-20T09:05:00.407 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.407 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b976a180:::benchmark_data_smithi122_119671_object74112:head# 2024-03-20T09:05:00.407 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.407 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b976fc70:::benchmark_data_smithi122_119671_object20284:head# 2024-03-20T09:05:00.407 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.407 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9775e72:::benchmark_data_smithi122_119671_object85572:head# 2024-03-20T09:05:00.407 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.407 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b979105e:::benchmark_data_smithi122_119671_object41188:head# 2024-03-20T09:05:00.408 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.408 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b97b7215:::benchmark_data_smithi122_119671_object45295:head# 2024-03-20T09:05:00.408 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.408 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b97b8ef1:::benchmark_data_smithi122_119671_object1352:head# 2024-03-20T09:05:00.408 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.408 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b97c90e0:::benchmark_data_smithi122_119671_object8225:head# 2024-03-20T09:05:00.408 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.408 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b97dba9e:::benchmark_data_smithi122_119671_object7906:head# 2024-03-20T09:05:00.408 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.408 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b97f1a1e:::benchmark_data_smithi122_119671_object6765:head# 2024-03-20T09:05:00.408 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.408 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b97fd53d:::benchmark_data_smithi122_119671_object89204:head# 2024-03-20T09:05:00.408 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.409 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b98028f0:::benchmark_data_smithi122_119671_object14142:head# 2024-03-20T09:05:00.409 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.409 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b980ce57:::benchmark_data_smithi122_119671_object83613:head# 2024-03-20T09:05:00.409 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.409 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b980f749:::benchmark_data_smithi122_119671_object12714:head# 2024-03-20T09:05:00.409 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.409 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b98237a8:::benchmark_data_smithi122_119671_object13414:head# 2024-03-20T09:05:00.409 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.409 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b983cc1c:::benchmark_data_smithi122_119671_object52344:head# 2024-03-20T09:05:00.409 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.409 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b983d186:::benchmark_data_smithi122_119671_object99600:head# 2024-03-20T09:05:00.409 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.409 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b98404c2:::benchmark_data_smithi122_119671_object84415:head# 2024-03-20T09:05:00.409 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.410 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9845df1:::benchmark_data_smithi122_119671_object94103:head# 2024-03-20T09:05:00.410 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.410 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b984ccdb:::benchmark_data_smithi122_119671_object10609:head# 2024-03-20T09:05:00.410 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.410 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b985df26:::benchmark_data_smithi122_119671_object18112:head# 2024-03-20T09:05:00.410 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.410 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b98818db:::benchmark_data_smithi122_119671_object52517:head# 2024-03-20T09:05:00.410 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.410 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9885714:::benchmark_data_smithi122_119671_object23646:head# 2024-03-20T09:05:00.410 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.410 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b988da1b:::benchmark_data_smithi122_119671_object54424:head# 2024-03-20T09:05:00.410 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.410 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b989debb:::benchmark_data_smithi122_119671_object57757:head# 2024-03-20T09:05:00.411 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.411 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b98a0a4d:::benchmark_data_smithi122_119671_object55510:head# 2024-03-20T09:05:00.411 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.411 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b98a0c10:::benchmark_data_smithi122_119671_object82293:head# 2024-03-20T09:05:00.411 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.411 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b98a0f13:::benchmark_data_smithi122_119671_object9287:head# 2024-03-20T09:05:00.411 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.411 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b98a4fb9:::benchmark_data_smithi122_119671_object27707:head# 2024-03-20T09:05:00.411 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.411 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b98ae761:::benchmark_data_smithi122_119671_object73728:head# 2024-03-20T09:05:00.411 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.411 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b98b0492:::benchmark_data_smithi122_119671_object80278:head# 2024-03-20T09:05:00.411 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.411 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b98ba5d0:::benchmark_data_smithi122_119671_object93417:head# 2024-03-20T09:05:00.411 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.412 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b98bfb00:::benchmark_data_smithi122_119671_object22432:head# 2024-03-20T09:05:00.412 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.412 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b98c07b9:::benchmark_data_smithi122_119671_object45653:head# 2024-03-20T09:05:00.412 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.412 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b98c1567:::benchmark_data_smithi122_119671_object15328:head# 2024-03-20T09:05:00.412 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.412 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b98c5f50:::benchmark_data_smithi122_119671_object69333:head# 2024-03-20T09:05:00.412 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.412 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b98c8aad:::benchmark_data_smithi122_119671_object64180:head# 2024-03-20T09:05:00.412 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.412 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b98cbc25:::benchmark_data_smithi122_119671_object4242:head# 2024-03-20T09:05:00.412 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.412 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b98f15c6:::benchmark_data_smithi122_119671_object4046:head# 2024-03-20T09:05:00.412 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.413 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b98f3b0d:::benchmark_data_smithi122_119671_object22377:head# 2024-03-20T09:05:00.413 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.413 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b98f606a:::benchmark_data_smithi122_119671_object12592:head# 2024-03-20T09:05:00.413 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.413 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b98f8afe:::benchmark_data_smithi122_119671_object81314:head# 2024-03-20T09:05:00.413 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.413 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9906bb6:::benchmark_data_smithi122_119671_object31425:head# 2024-03-20T09:05:00.413 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.413 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b99091a2:::benchmark_data_smithi122_119671_object45742:head# 2024-03-20T09:05:00.413 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.413 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9913257:::benchmark_data_smithi122_119671_object24074:head# 2024-03-20T09:05:00.413 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.413 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9924a02:::benchmark_data_smithi122_119671_object35960:head# 2024-03-20T09:05:00.413 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.414 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b992fb8b:::benchmark_data_smithi122_119671_object76547:head# 2024-03-20T09:05:00.414 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.414 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9937851:::benchmark_data_smithi122_119671_object47420:head# 2024-03-20T09:05:00.414 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.414 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b993cfd7:::benchmark_data_smithi122_119671_object26844:head# 2024-03-20T09:05:00.414 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.414 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9942518:::benchmark_data_smithi122_119671_object3004:head# 2024-03-20T09:05:00.414 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.414 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b99496fc:::benchmark_data_smithi122_119671_object13042:head# 2024-03-20T09:05:00.414 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.414 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b994d04e:::benchmark_data_smithi122_119671_object57981:head# 2024-03-20T09:05:00.414 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.414 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b994d545:::benchmark_data_smithi122_119671_object39357:head# 2024-03-20T09:05:00.415 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.415 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b994f44c:::benchmark_data_smithi122_119671_object32717:head# 2024-03-20T09:05:00.415 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.415 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9973030:::benchmark_data_smithi122_119671_object91835:head# 2024-03-20T09:05:00.415 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.415 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b997767c:::benchmark_data_smithi122_119671_object22216:head# 2024-03-20T09:05:00.415 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.415 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9985f99:::benchmark_data_smithi122_119671_object1284:head# 2024-03-20T09:05:00.415 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.415 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b998a9b0:::benchmark_data_smithi122_119671_object39600:head# 2024-03-20T09:05:00.415 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.415 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b998eed8:::benchmark_data_smithi122_119671_object53809:head# 2024-03-20T09:05:00.415 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.416 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b999dc13:::benchmark_data_smithi122_119671_object73979:head# 2024-03-20T09:05:00.416 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.416 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b99a1b4e:::benchmark_data_smithi122_119671_object58301:head# 2024-03-20T09:05:00.416 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.416 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b99a7edb:::benchmark_data_smithi122_119671_object31826:head# 2024-03-20T09:05:00.416 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.416 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b99a9d85:::benchmark_data_smithi122_119671_object65309:head# 2024-03-20T09:05:00.416 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.416 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b99b4a88:::benchmark_data_smithi122_119671_object57638:head# 2024-03-20T09:05:00.416 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.416 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b99b8d84:::benchmark_data_smithi122_119671_object6572:head# 2024-03-20T09:05:00.416 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.416 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b99bbd8c:::benchmark_data_smithi122_119671_object39095:head# 2024-03-20T09:05:00.416 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.417 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b99bc167:::benchmark_data_smithi122_119671_object85595:head# 2024-03-20T09:05:00.417 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.417 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b99cadd6:::benchmark_data_smithi122_119671_object20256:head# 2024-03-20T09:05:00.417 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.417 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b99dd343:::benchmark_data_smithi122_119671_object41923:head# 2024-03-20T09:05:00.417 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.417 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b99ea8cc:::benchmark_data_smithi122_119671_object6377:head# 2024-03-20T09:05:00.417 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.417 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b99ede2c:::benchmark_data_smithi122_119671_object19419:head# 2024-03-20T09:05:00.417 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.417 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b99fa2d1:::benchmark_data_smithi122_119671_object65984:head# 2024-03-20T09:05:00.417 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.417 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9a0a05e:::benchmark_data_smithi122_119671_object7778:head# 2024-03-20T09:05:00.417 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.418 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9a136e5:::benchmark_data_smithi122_119671_object46515:head# 2024-03-20T09:05:00.418 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.418 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9a15cdf:::benchmark_data_smithi122_119671_object58424:head# 2024-03-20T09:05:00.418 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.418 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9a288ac:::benchmark_data_smithi122_119671_object56376:head# 2024-03-20T09:05:00.418 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.418 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9a2d9f8:::benchmark_data_smithi122_119671_object71611:head# 2024-03-20T09:05:00.418 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.418 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9a37213:::benchmark_data_smithi122_119671_object64440:head# 2024-03-20T09:05:00.418 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.418 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9a4c393:::benchmark_data_smithi122_119671_object45608:head# 2024-03-20T09:05:00.418 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.418 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9a4cdf3:::benchmark_data_smithi122_119671_object21807:head# 2024-03-20T09:05:00.418 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.419 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9a60f50:::benchmark_data_smithi122_119671_object51664:head# 2024-03-20T09:05:00.419 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.419 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9a698f2:::benchmark_data_smithi122_119671_object72653:head# 2024-03-20T09:05:00.419 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.419 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9a6b89b:::benchmark_data_smithi122_119671_object19479:head# 2024-03-20T09:05:00.419 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.419 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9a6c304:::benchmark_data_smithi122_119671_object92917:head# 2024-03-20T09:05:00.419 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.419 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9a6c5f5:::benchmark_data_smithi122_119671_object51271:head# 2024-03-20T09:05:00.419 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.419 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9a797ff:::benchmark_data_smithi122_119671_object35951:head# 2024-03-20T09:05:00.419 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.419 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9a7dda0:::benchmark_data_smithi122_119671_object55144:head# 2024-03-20T09:05:00.419 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.419 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9aa4275:::benchmark_data_smithi122_119671_object46827:head# 2024-03-20T09:05:00.420 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.420 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9aa5320:::benchmark_data_smithi122_119671_object77228:head# 2024-03-20T09:05:00.420 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.420 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9ab1f85:::benchmark_data_smithi122_119671_object81726:head# 2024-03-20T09:05:00.420 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.420 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9ac9af0:::benchmark_data_smithi122_119671_object12173:head# 2024-03-20T09:05:00.420 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.420 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9ada4f6:::benchmark_data_smithi122_119671_object14104:head# 2024-03-20T09:05:00.420 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.420 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9ae0524:::benchmark_data_smithi122_119671_object19563:head# 2024-03-20T09:05:00.420 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.420 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9aeb6fd:::benchmark_data_smithi122_119671_object95598:head# 2024-03-20T09:05:00.420 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.420 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9aec28f:::benchmark_data_smithi122_119671_object29362:head# 2024-03-20T09:05:00.421 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.421 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9aecea9:::benchmark_data_smithi122_119671_object22042:head# 2024-03-20T09:05:00.421 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.421 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9afa6a2:::benchmark_data_smithi122_119671_object79029:head# 2024-03-20T09:05:00.421 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.421 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9afb612:::benchmark_data_smithi122_119671_object31762:head# 2024-03-20T09:05:00.421 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.421 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9afbd37:::benchmark_data_smithi122_119671_object24883:head# 2024-03-20T09:05:00.421 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.421 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9afc52a:::benchmark_data_smithi122_119671_object16716:head# 2024-03-20T09:05:00.421 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.421 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9b03612:::benchmark_data_smithi122_119671_object6651:head# 2024-03-20T09:05:00.421 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.421 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9b1608f:::benchmark_data_smithi122_119671_object18738:head# 2024-03-20T09:05:00.422 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.422 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9b1b761:::benchmark_data_smithi122_119671_object71955:head# 2024-03-20T09:05:00.422 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.422 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9b2ff13:::benchmark_data_smithi122_119671_object20834:head# 2024-03-20T09:05:00.422 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.422 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9b37523:::benchmark_data_smithi122_119671_object19599:head# 2024-03-20T09:05:00.422 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.422 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9b39895:::benchmark_data_smithi122_119671_object51140:head# 2024-03-20T09:05:00.422 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.422 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9b4297f:::benchmark_data_smithi122_119671_object80793:head# 2024-03-20T09:05:00.422 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.422 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9b4c750:::benchmark_data_smithi122_119671_object93218:head# 2024-03-20T09:05:00.422 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.422 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9b4e08a:::benchmark_data_smithi122_119671_object23231:head# 2024-03-20T09:05:00.423 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.423 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9b5300e:::benchmark_data_smithi122_119671_object16265:head# 2024-03-20T09:05:00.423 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.423 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9b558b1:::benchmark_data_smithi122_119671_object90626:head# 2024-03-20T09:05:00.423 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.423 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9b5cc53:::benchmark_data_smithi122_119671_object6451:head# 2024-03-20T09:05:00.423 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.423 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9b78c4e:::benchmark_data_smithi122_119671_object90193:head# 2024-03-20T09:05:00.423 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.423 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9b85dde:::benchmark_data_smithi122_119671_object82600:head# 2024-03-20T09:05:00.423 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.423 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9b8b00b:::benchmark_data_smithi122_119671_object46372:head# 2024-03-20T09:05:00.423 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.423 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9b9e1af:::benchmark_data_smithi122_119671_object34566:head# 2024-03-20T09:05:00.424 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.424 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9ba33d8:::benchmark_data_smithi122_119671_object7547:head# 2024-03-20T09:05:00.424 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.424 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9bb24b7:::benchmark_data_smithi122_119671_object40490:head# 2024-03-20T09:05:00.424 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.424 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9bb350b:::benchmark_data_smithi122_119671_object68057:head# 2024-03-20T09:05:00.424 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.424 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9bb65d1:::benchmark_data_smithi122_119671_object20368:head# 2024-03-20T09:05:00.424 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.424 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9bb7b12:::benchmark_data_smithi122_119671_object42436:head# 2024-03-20T09:05:00.424 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.424 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9bcd50f:::benchmark_data_smithi122_119671_object8040:head# 2024-03-20T09:05:00.424 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.424 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9bd934d:::benchmark_data_smithi122_119671_object88936:head# 2024-03-20T09:05:00.425 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.425 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9bdb907:::benchmark_data_smithi122_119671_object40815:head# 2024-03-20T09:05:00.425 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.425 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9be559e:::benchmark_data_smithi122_119671_object45660:head# 2024-03-20T09:05:00.425 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.425 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9be8d34:::benchmark_data_smithi122_119671_object84366:head# 2024-03-20T09:05:00.425 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.425 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9bf3464:::benchmark_data_smithi122_119671_object80045:head# 2024-03-20T09:05:00.425 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.425 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9bfcf37:::benchmark_data_smithi122_119671_object47186:head# 2024-03-20T09:05:00.425 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.425 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9c02a6a:::benchmark_data_smithi122_119671_object86896:head# 2024-03-20T09:05:00.425 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.425 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9c0ac2c:::benchmark_data_smithi122_119671_object79727:head# 2024-03-20T09:05:00.426 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.426 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9c105de:::benchmark_data_smithi122_119671_object19361:head# 2024-03-20T09:05:00.426 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.426 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9c1f438:::benchmark_data_smithi122_119671_object7566:head# 2024-03-20T09:05:00.426 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.426 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9c30386:::benchmark_data_smithi122_119671_object28372:head# 2024-03-20T09:05:00.426 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.426 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9c377cc:::benchmark_data_smithi122_119671_object81489:head# 2024-03-20T09:05:00.426 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.426 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9c39c1a:::benchmark_data_smithi122_119671_object8266:head# 2024-03-20T09:05:00.426 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.426 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9c3dca8:::benchmark_data_smithi122_119671_object41948:head# 2024-03-20T09:05:00.426 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.427 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9c6940b:::benchmark_data_smithi122_119671_object94116:head# 2024-03-20T09:05:00.427 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.427 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9c69ecd:::benchmark_data_smithi122_119671_object69280:head# 2024-03-20T09:05:00.427 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.427 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9c9046a:::benchmark_data_smithi122_119671_object70131:head# 2024-03-20T09:05:00.427 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.427 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9cb66cd:::benchmark_data_smithi122_119671_object15160:head# 2024-03-20T09:05:00.427 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.427 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9cb6923:::benchmark_data_smithi122_119671_object2480:head# 2024-03-20T09:05:00.427 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.427 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9ccd2ec:::benchmark_data_smithi122_119671_object31036:head# 2024-03-20T09:05:00.427 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.427 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9cd4e3a:::benchmark_data_smithi122_119671_object60448:head# 2024-03-20T09:05:00.427 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.428 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9cdab2d:::benchmark_data_smithi122_119671_object88295:head# 2024-03-20T09:05:00.428 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.428 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9cde496:::benchmark_data_smithi122_119671_object16223:head# 2024-03-20T09:05:00.428 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.428 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9d14139:::benchmark_data_smithi122_119671_object59524:head# 2024-03-20T09:05:00.428 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.428 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9d2ae0c:::benchmark_data_smithi122_119671_object76091:head# 2024-03-20T09:05:00.428 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.428 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9d3a052:::benchmark_data_smithi122_119671_object96266:head# 2024-03-20T09:05:00.428 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.428 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9d3c9cd:::benchmark_data_smithi122_119671_object9458:head# 2024-03-20T09:05:00.428 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.428 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9d3fe28:::benchmark_data_smithi122_119671_object70803:head# 2024-03-20T09:05:00.428 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.429 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9d40f1f:::benchmark_data_smithi122_119671_object10530:head# 2024-03-20T09:05:00.429 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.429 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9d4a8a6:::benchmark_data_smithi122_119671_object35413:head# 2024-03-20T09:05:00.429 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.429 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9d52130:::benchmark_data_smithi122_119671_object42115:head# 2024-03-20T09:05:00.429 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.429 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9d61366:::benchmark_data_smithi122_119671_object69900:head# 2024-03-20T09:05:00.429 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.429 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9d65931:::benchmark_data_smithi122_119671_object1133:head# 2024-03-20T09:05:00.429 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.429 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9d7eca4:::benchmark_data_smithi122_119671_object27779:head# 2024-03-20T09:05:00.429 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.429 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9d8f484:::benchmark_data_smithi122_119671_object32955:head# 2024-03-20T09:05:00.429 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.429 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9d9565e:::benchmark_data_smithi122_119671_object12071:head# 2024-03-20T09:05:00.430 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.430 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9da7912:::benchmark_data_smithi122_119671_object15752:head# 2024-03-20T09:05:00.430 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.430 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9da987b:::benchmark_data_smithi122_119671_object56417:head# 2024-03-20T09:05:00.430 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.430 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9dabe92:::benchmark_data_smithi122_119671_object16401:head# 2024-03-20T09:05:00.430 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.430 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9db27a6:::benchmark_data_smithi122_119671_object80771:head# 2024-03-20T09:05:00.430 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.430 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9db9400:::benchmark_data_smithi122_119671_object50586:head# 2024-03-20T09:05:00.430 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.430 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9dbc43e:::benchmark_data_smithi122_119671_object70200:head# 2024-03-20T09:05:00.430 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.430 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9dce5c9:::benchmark_data_smithi122_119671_object89445:head# 2024-03-20T09:05:00.431 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.431 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9de7c6b:::benchmark_data_smithi122_119671_object80884:head# 2024-03-20T09:05:00.431 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.431 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9def8d5:::benchmark_data_smithi122_119671_object95019:head# 2024-03-20T09:05:00.431 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.431 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9df687f:::benchmark_data_smithi122_119671_object81904:head# 2024-03-20T09:05:00.431 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.431 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9e1a707:::benchmark_data_smithi122_119671_object15451:head# 2024-03-20T09:05:00.431 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.431 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9e2d8ad:::benchmark_data_smithi122_119671_object32689:head# 2024-03-20T09:05:00.431 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.431 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9e44a2d:::benchmark_data_smithi122_119671_object30709:head# 2024-03-20T09:05:00.431 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.432 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9e4e41c:::benchmark_data_smithi122_119671_object84577:head# 2024-03-20T09:05:00.432 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.432 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9e52ab7:::benchmark_data_smithi122_119671_object99743:head# 2024-03-20T09:05:00.432 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.432 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9e53c9a:::benchmark_data_smithi122_119671_object8807:head# 2024-03-20T09:05:00.432 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.432 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9e55a69:::benchmark_data_smithi122_119671_object85568:head# 2024-03-20T09:05:00.432 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.432 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9e70e8e:::benchmark_data_smithi122_119671_object21099:head# 2024-03-20T09:05:00.432 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.432 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9e78199:::benchmark_data_smithi122_119671_object56126:head# 2024-03-20T09:05:00.432 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.432 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9e7f2b0:::benchmark_data_smithi122_119671_object77883:head# 2024-03-20T09:05:00.433 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.433 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9e89fbd:::benchmark_data_smithi122_119671_object96884:head# 2024-03-20T09:05:00.433 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.433 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9e8f5fb:::benchmark_data_smithi122_119671_object85676:head# 2024-03-20T09:05:00.433 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.433 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9e93694:::benchmark_data_smithi122_119671_object53907:head# 2024-03-20T09:05:00.433 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.433 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9eb766a:::benchmark_data_smithi122_119671_object53756:head# 2024-03-20T09:05:00.433 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.433 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9ec164b:::benchmark_data_smithi122_119671_object20034:head# 2024-03-20T09:05:00.433 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.433 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9ecdc34:::benchmark_data_smithi122_119671_object64491:head# 2024-03-20T09:05:00.433 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.433 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9ed20eb:::benchmark_data_smithi122_119671_object79456:head# 2024-03-20T09:05:00.433 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.434 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9eeeff1:::benchmark_data_smithi122_119671_object15562:head# 2024-03-20T09:05:00.434 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.434 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9ef12d2:::benchmark_data_smithi122_119671_object14524:head# 2024-03-20T09:05:00.434 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.434 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9ef698f:::benchmark_data_smithi122_119671_object32250:head# 2024-03-20T09:05:00.434 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.434 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9ef7f2b:::benchmark_data_smithi122_119671_object34723:head# 2024-03-20T09:05:00.434 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.434 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9ef928d:::benchmark_data_smithi122_119671_object1054:head# 2024-03-20T09:05:00.434 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.434 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9f1509e:::benchmark_data_smithi122_119671_object60927:head# 2024-03-20T09:05:00.434 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.434 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9f1b96b:::benchmark_data_smithi122_119671_object58553:head# 2024-03-20T09:05:00.435 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.435 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9f1e4b0:::benchmark_data_smithi122_119671_object24839:head# 2024-03-20T09:05:00.435 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.435 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9f232fc:::benchmark_data_smithi122_119671_object82859:head# 2024-03-20T09:05:00.435 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.435 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9f2ed3f:::benchmark_data_smithi122_119671_object35113:head# 2024-03-20T09:05:00.435 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.435 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9f3189c:::benchmark_data_smithi122_119671_object35853:head# 2024-03-20T09:05:00.435 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.435 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9f3e263:::benchmark_data_smithi122_119671_object12653:head# 2024-03-20T09:05:00.435 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.435 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9f4237a:::benchmark_data_smithi122_119671_object2531:head# 2024-03-20T09:05:00.435 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.436 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9f44b45:::benchmark_data_smithi122_119671_object28225:head# 2024-03-20T09:05:00.436 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.436 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9f63f02:::benchmark_data_smithi122_119671_object10165:head# 2024-03-20T09:05:00.436 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.436 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9f778ad:::benchmark_data_smithi122_119671_object4604:head# 2024-03-20T09:05:00.436 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.436 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9f7faa4:::benchmark_data_smithi122_119671_object25063:head# 2024-03-20T09:05:00.436 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.436 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9f905a7:::benchmark_data_smithi122_119671_object52723:head# 2024-03-20T09:05:00.436 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.436 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9f9460c:::benchmark_data_smithi122_119671_object91050:head# 2024-03-20T09:05:00.436 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.436 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9f9b366:::benchmark_data_smithi122_119671_object53348:head# 2024-03-20T09:05:00.436 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.437 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9f9d8c4:::benchmark_data_smithi122_119671_object66597:head# 2024-03-20T09:05:00.437 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.437 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9f9f2c5:::benchmark_data_smithi122_119671_object85850:head# 2024-03-20T09:05:00.437 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.437 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9fa566e:::benchmark_data_smithi122_119671_object87582:head# 2024-03-20T09:05:00.437 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.437 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9fa8329:::benchmark_data_smithi122_119671_object4258:head# 2024-03-20T09:05:00.437 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.437 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9fae878:::benchmark_data_smithi122_119671_object42516:head# 2024-03-20T09:05:00.437 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.437 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9fbef25:::benchmark_data_smithi122_119671_object16425:head# 2024-03-20T09:05:00.437 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.437 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9fd46f9:::benchmark_data_smithi122_119671_object88324:head# 2024-03-20T09:05:00.438 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.438 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9fd9067:::benchmark_data_smithi122_119671_object94931:head# 2024-03-20T09:05:00.438 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.438 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9fddc35:::benchmark_data_smithi122_119671_object23636:head# 2024-03-20T09:05:00.438 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.438 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9ff680a:::benchmark_data_smithi122_119671_object34193:head# 2024-03-20T09:05:00.438 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.438 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:b9ff7dca:::benchmark_data_smithi122_119671_object10016:head# 2024-03-20T09:05:00.438 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.438 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba018677:::benchmark_data_smithi122_119671_object27484:head# 2024-03-20T09:05:00.438 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.438 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba0205a0:::benchmark_data_smithi122_119671_object54452:head# 2024-03-20T09:05:00.438 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.438 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba020dc4:::benchmark_data_smithi122_119671_object9945:head# 2024-03-20T09:05:00.439 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.439 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba046e98:::benchmark_data_smithi122_119671_object54433:head# 2024-03-20T09:05:00.439 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.439 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba050028:::benchmark_data_smithi122_119671_object51849:head# 2024-03-20T09:05:00.439 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.439 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba0550b8:::benchmark_data_smithi122_119671_object56736:head# 2024-03-20T09:05:00.439 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.439 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba086918:::benchmark_data_smithi122_119671_object75858:head# 2024-03-20T09:05:00.439 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.439 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba08a11c:::benchmark_data_smithi122_119671_object55285:head# 2024-03-20T09:05:00.439 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.439 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba09194b:::benchmark_data_smithi122_119671_object38755:head# 2024-03-20T09:05:00.439 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.440 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba092fff:::benchmark_data_smithi122_119671_object62373:head# 2024-03-20T09:05:00.440 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.440 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba09d9bf:::benchmark_data_smithi122_119671_object35495:head# 2024-03-20T09:05:00.440 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.440 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba09f961:::benchmark_data_smithi122_119671_object91933:head# 2024-03-20T09:05:00.440 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.440 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba09f9bf:::benchmark_data_smithi122_119671_object32339:head# 2024-03-20T09:05:00.440 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.440 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba0a08be:::benchmark_data_smithi122_119671_object13708:head# 2024-03-20T09:05:00.440 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.440 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba0a0f5f:::benchmark_data_smithi122_119671_object92369:head# 2024-03-20T09:05:00.440 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.440 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba0ad2ce:::benchmark_data_smithi122_119671_object18872:head# 2024-03-20T09:05:00.440 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.441 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba0c7cbd:::benchmark_data_smithi122_119671_object1476:head# 2024-03-20T09:05:00.441 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.441 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba0d9cdb:::benchmark_data_smithi122_119671_object46057:head# 2024-03-20T09:05:00.441 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.441 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba0df7be:::benchmark_data_smithi122_119671_object89057:head# 2024-03-20T09:05:00.441 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.441 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba0e2b05:::benchmark_data_smithi122_119671_object39124:head# 2024-03-20T09:05:00.441 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.441 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba0e8c44:::benchmark_data_smithi122_119671_object39016:head# 2024-03-20T09:05:00.441 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.441 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba0ec75e:::benchmark_data_smithi122_119671_object14775:head# 2024-03-20T09:05:00.441 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.442 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba0ecf87:::benchmark_data_smithi122_119671_object73652:head# 2024-03-20T09:05:00.442 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.442 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba102e77:::benchmark_data_smithi122_119671_object23518:head# 2024-03-20T09:05:00.442 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.442 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba11dd01:::benchmark_data_smithi122_119671_object33389:head# 2024-03-20T09:05:00.442 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.442 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba126cd0:::benchmark_data_smithi122_119671_object12621:head# 2024-03-20T09:05:00.442 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.442 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba147044:::benchmark_data_smithi122_119671_object70404:head# 2024-03-20T09:05:00.442 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.442 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba14a968:::benchmark_data_smithi122_119671_object89768:head# 2024-03-20T09:05:00.442 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.442 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba167357:::benchmark_data_smithi122_119671_object47821:head# 2024-03-20T09:05:00.443 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.443 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba16ce89:::benchmark_data_smithi122_119671_object63779:head# 2024-03-20T09:05:00.443 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.443 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba16de77:::benchmark_data_smithi122_119671_object82191:head# 2024-03-20T09:05:00.443 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.443 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba1932e8:::benchmark_data_smithi122_119671_object70114:head# 2024-03-20T09:05:00.443 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.443 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba197b3e:::benchmark_data_smithi122_119671_object56038:head# 2024-03-20T09:05:00.443 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.443 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba1b43b4:::benchmark_data_smithi122_119671_object41774:head# 2024-03-20T09:05:00.443 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.443 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba1b8c46:::benchmark_data_smithi122_119671_object49476:head# 2024-03-20T09:05:00.444 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.444 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba1bc0c7:::benchmark_data_smithi122_119671_object67640:head# 2024-03-20T09:05:00.444 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.444 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba1c811e:::benchmark_data_smithi122_119671_object41902:head# 2024-03-20T09:05:00.444 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.444 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba1c85d9:::benchmark_data_smithi122_119671_object49147:head# 2024-03-20T09:05:00.444 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.444 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba1cb781:::benchmark_data_smithi122_119671_object1966:head# 2024-03-20T09:05:00.444 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.444 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba1cf693:::benchmark_data_smithi122_119671_object68416:head# 2024-03-20T09:05:00.444 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.444 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba1e1a34:::benchmark_data_smithi122_119671_object93352:head# 2024-03-20T09:05:00.444 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.444 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba1e3ed7:::benchmark_data_smithi122_119671_object97152:head# 2024-03-20T09:05:00.445 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.445 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba1f0126:::benchmark_data_smithi122_119671_object67258:head# 2024-03-20T09:05:00.445 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.445 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba1fc2c1:::benchmark_data_smithi122_119671_object51610:head# 2024-03-20T09:05:00.445 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.445 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba200d92:::benchmark_data_smithi122_119671_object96074:head# 2024-03-20T09:05:00.445 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.445 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba219d2a:::benchmark_data_smithi122_119671_object54231:head# 2024-03-20T09:05:00.445 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.445 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba21a1e1:::benchmark_data_smithi122_119671_object25127:head# 2024-03-20T09:05:00.445 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.445 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba21d200:::benchmark_data_smithi122_119671_object52436:head# 2024-03-20T09:05:00.445 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.446 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba22fc67:::benchmark_data_smithi122_119671_object39005:head# 2024-03-20T09:05:00.446 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.446 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba233054:::benchmark_data_smithi122_119671_object14004:head# 2024-03-20T09:05:00.446 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.446 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba241078:::benchmark_data_smithi122_119671_object95712:head# 2024-03-20T09:05:00.446 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.446 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba245921:::benchmark_data_smithi122_119671_object19754:head# 2024-03-20T09:05:00.446 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.446 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba24ca86:::benchmark_data_smithi122_119671_object28797:head# 2024-03-20T09:05:00.446 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.446 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba2565dc:::benchmark_data_smithi122_119671_object91563:head# 2024-03-20T09:05:00.446 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.446 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba2698b2:::benchmark_data_smithi122_119671_object40753:head# 2024-03-20T09:05:00.446 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.447 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba26d9d9:::benchmark_data_smithi122_119671_object14420:head# 2024-03-20T09:05:00.447 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.447 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba27a9c3:::benchmark_data_smithi122_119671_object21797:head# 2024-03-20T09:05:00.447 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.447 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba27c738:::benchmark_data_smithi122_119671_object59173:head# 2024-03-20T09:05:00.447 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.447 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba28ade0:::benchmark_data_smithi122_119671_object41111:head# 2024-03-20T09:05:00.447 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.447 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba298870:::benchmark_data_smithi122_119671_object16276:head# 2024-03-20T09:05:00.447 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.447 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba2b7be9:::benchmark_data_smithi122_119671_object40076:head# 2024-03-20T09:05:00.448 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.448 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba2c8585:::benchmark_data_smithi122_119671_object25755:head# 2024-03-20T09:05:00.448 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.448 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba2caf12:::benchmark_data_smithi122_119671_object10187:head# 2024-03-20T09:05:00.448 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.448 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba2d25b9:::benchmark_data_smithi122_119671_object7790:head# 2024-03-20T09:05:00.448 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.448 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba2d8227:::benchmark_data_smithi122_119671_object52662:head# 2024-03-20T09:05:00.448 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.448 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba2dbd22:::benchmark_data_smithi122_119671_object13212:head# 2024-03-20T09:05:00.448 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.448 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba2f5f28:::benchmark_data_smithi122_119671_object95315:head# 2024-03-20T09:05:00.448 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.449 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba2fb430:::benchmark_data_smithi122_119671_object40993:head# 2024-03-20T09:05:00.449 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.449 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba30cb67:::benchmark_data_smithi122_119671_object27184:head# 2024-03-20T09:05:00.449 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.449 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba30da18:::benchmark_data_smithi122_119671_object3114:head# 2024-03-20T09:05:00.449 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.449 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba30f695:::benchmark_data_smithi122_119671_object23389:head# 2024-03-20T09:05:00.449 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.449 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba314340:::benchmark_data_smithi122_119671_object83660:head# 2024-03-20T09:05:00.449 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.449 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba316459:::benchmark_data_smithi122_119671_object57812:head# 2024-03-20T09:05:00.449 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.450 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba337055:::benchmark_data_smithi122_119671_object21552:head# 2024-03-20T09:05:00.450 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.450 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba33da45:::benchmark_data_smithi122_119671_object16675:head# 2024-03-20T09:05:00.450 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.450 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba3499e8:::benchmark_data_smithi122_119671_object15265:head# 2024-03-20T09:05:00.450 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.450 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba34faf2:::benchmark_data_smithi122_119671_object99279:head# 2024-03-20T09:05:00.450 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.450 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba350514:::benchmark_data_smithi122_119671_object57737:head# 2024-03-20T09:05:00.450 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.450 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba35129d:::benchmark_data_smithi122_119671_object57354:head# 2024-03-20T09:05:00.450 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.450 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba355c89:::benchmark_data_smithi122_119671_object38119:head# 2024-03-20T09:05:00.451 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.451 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba361184:::benchmark_data_smithi122_119671_object58468:head# 2024-03-20T09:05:00.451 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.451 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba365b7c:::benchmark_data_smithi122_119671_object17060:head# 2024-03-20T09:05:00.451 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.451 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba365d70:::benchmark_data_smithi122_119671_object15846:head# 2024-03-20T09:05:00.451 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.451 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba389094:::benchmark_data_smithi122_119671_object63567:head# 2024-03-20T09:05:00.451 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.451 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba38939d:::benchmark_data_smithi122_119671_object42416:head# 2024-03-20T09:05:00.451 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.451 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba3901fd:::benchmark_data_smithi122_119671_object41789:head# 2024-03-20T09:05:00.451 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.451 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba3929c2:::benchmark_data_smithi122_119671_object12498:head# 2024-03-20T09:05:00.452 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.452 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba397022:::benchmark_data_smithi122_119671_object44438:head# 2024-03-20T09:05:00.452 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.452 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba39de65:::benchmark_data_smithi122_119671_object60597:head# 2024-03-20T09:05:00.452 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.452 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba3b0b12:::benchmark_data_smithi122_119671_object38618:head# 2024-03-20T09:05:00.452 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.452 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba3b3970:::benchmark_data_smithi122_119671_object40940:head# 2024-03-20T09:05:00.452 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.452 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba3df839:::benchmark_data_smithi122_119671_object70642:head# 2024-03-20T09:05:00.452 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.452 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba3ea4b2:::benchmark_data_smithi122_119671_object91254:head# 2024-03-20T09:05:00.452 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.452 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba3f6767:::benchmark_data_smithi122_119671_object62470:head# 2024-03-20T09:05:00.453 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.453 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba3f7008:::benchmark_data_smithi122_119671_object74349:head# 2024-03-20T09:05:00.453 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.453 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba3faaf5:::benchmark_data_smithi122_119671_object21527:head# 2024-03-20T09:05:00.453 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.453 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba40ac83:::benchmark_data_smithi122_119671_object21451:head# 2024-03-20T09:05:00.453 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.453 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba40f0b9:::benchmark_data_smithi122_119671_object26341:head# 2024-03-20T09:05:00.453 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.453 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba41131f:::benchmark_data_smithi122_119671_object82878:head# 2024-03-20T09:05:00.453 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.453 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba434ca6:::benchmark_data_smithi122_119671_object75496:head# 2024-03-20T09:05:00.453 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.453 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba4516bd:::benchmark_data_smithi122_119671_object68186:head# 2024-03-20T09:05:00.453 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.454 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba4568c2:::benchmark_data_smithi122_119671_object88146:head# 2024-03-20T09:05:00.454 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.454 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba456ca6:::benchmark_data_smithi122_119671_object11303:head# 2024-03-20T09:05:00.454 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.454 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba460d9e:::benchmark_data_smithi122_119671_object56135:head# 2024-03-20T09:05:00.454 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.454 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba46f83f:::benchmark_data_smithi122_119671_object44730:head# 2024-03-20T09:05:00.454 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.454 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba4915cd:::benchmark_data_smithi122_119671_object29014:head# 2024-03-20T09:05:00.454 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.454 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba49499f:::benchmark_data_smithi122_119671_object29845:head# 2024-03-20T09:05:00.454 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.454 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba497077:::benchmark_data_smithi122_119671_object21208:head# 2024-03-20T09:05:00.455 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.455 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba49a6e0:::benchmark_data_smithi122_119671_object44201:head# 2024-03-20T09:05:00.455 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.455 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba4c875b:::benchmark_data_smithi122_119671_object47332:head# 2024-03-20T09:05:00.455 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.455 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba4cb67d:::benchmark_data_smithi122_119671_object43300:head# 2024-03-20T09:05:00.455 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.455 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba4cfa14:::benchmark_data_smithi122_119671_object23118:head# 2024-03-20T09:05:00.455 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.455 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba4d1877:::benchmark_data_smithi122_119671_object86922:head# 2024-03-20T09:05:00.455 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.455 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba4d50e8:::benchmark_data_smithi122_119671_object25076:head# 2024-03-20T09:05:00.455 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.455 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba4de147:::benchmark_data_smithi122_119671_object92178:head# 2024-03-20T09:05:00.456 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.456 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba4e2fc0:::benchmark_data_smithi122_119671_object69960:head# 2024-03-20T09:05:00.456 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.456 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba4e371e:::benchmark_data_smithi122_119671_object39852:head# 2024-03-20T09:05:00.456 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.456 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba4e410e:::benchmark_data_smithi122_119671_object7016:head# 2024-03-20T09:05:00.456 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.456 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba4ee07d:::benchmark_data_smithi122_119671_object23214:head# 2024-03-20T09:05:00.456 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.456 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba4ef9f0:::benchmark_data_smithi122_119671_object62840:head# 2024-03-20T09:05:00.456 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.456 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba5012c6:::benchmark_data_smithi122_119671_object46410:head# 2024-03-20T09:05:00.456 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.456 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba513cb1:::benchmark_data_smithi122_119671_object18561:head# 2024-03-20T09:05:00.457 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.457 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba51c43d:::benchmark_data_smithi122_119671_object43417:head# 2024-03-20T09:05:00.457 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.457 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba51fde2:::benchmark_data_smithi122_119671_object32827:head# 2024-03-20T09:05:00.457 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.457 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba521103:::benchmark_data_smithi122_119671_object87118:head# 2024-03-20T09:05:00.457 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.457 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba52974e:::benchmark_data_smithi122_119671_object27250:head# 2024-03-20T09:05:00.457 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.457 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba52be56:::benchmark_data_smithi122_119671_object3616:head# 2024-03-20T09:05:00.457 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.457 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba53079e:::benchmark_data_smithi122_119671_object9903:head# 2024-03-20T09:05:00.457 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.458 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba540bfd:::benchmark_data_smithi122_119671_object59240:head# 2024-03-20T09:05:00.458 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.458 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba543392:::benchmark_data_smithi122_119671_object82834:head# 2024-03-20T09:05:00.458 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.458 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba548f4e:::benchmark_data_smithi122_119671_object19217:head# 2024-03-20T09:05:00.458 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.458 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba5582d2:::benchmark_data_smithi122_119671_object19142:head# 2024-03-20T09:05:00.458 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.458 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba58c58c:::benchmark_data_smithi122_119671_object78751:head# 2024-03-20T09:05:00.458 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.458 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba58cfd6:::benchmark_data_smithi122_119671_object34342:head# 2024-03-20T09:05:00.458 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.458 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba5918d0:::benchmark_data_smithi122_119671_object85242:head# 2024-03-20T09:05:00.458 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.459 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba59f1c3:::benchmark_data_smithi122_119671_object82085:head# 2024-03-20T09:05:00.459 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.459 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba5a75d2:::benchmark_data_smithi122_119671_object2189:head# 2024-03-20T09:05:00.459 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.459 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba5b8b10:::benchmark_data_smithi122_119671_object73786:head# 2024-03-20T09:05:00.459 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.459 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba5bf534:::benchmark_data_smithi122_119671_object58396:head# 2024-03-20T09:05:00.459 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.459 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba5c84ec:::benchmark_data_smithi122_119671_object93306:head# 2024-03-20T09:05:00.459 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.459 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba5d1359:::benchmark_data_smithi122_119671_object78479:head# 2024-03-20T09:05:00.459 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.460 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba5dabb4:::benchmark_data_smithi122_119671_object27450:head# 2024-03-20T09:05:00.460 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.460 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba5dc533:::benchmark_data_smithi122_119671_object76039:head# 2024-03-20T09:05:00.460 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.460 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba5dd25e:::benchmark_data_smithi122_119671_object82175:head# 2024-03-20T09:05:00.460 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.460 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba5e56c2:::benchmark_data_smithi122_119671_object25503:head# 2024-03-20T09:05:00.460 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.460 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba5f4971:::benchmark_data_smithi122_119671_object28806:head# 2024-03-20T09:05:00.460 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.460 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba5f50c2:::benchmark_data_smithi122_119671_object1448:head# 2024-03-20T09:05:00.460 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.460 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba5fea89:::benchmark_data_smithi122_119671_object76965:head# 2024-03-20T09:05:00.461 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.461 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba608a8e:::benchmark_data_smithi122_119671_object55516:head# 2024-03-20T09:05:00.461 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.461 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba609695:::benchmark_data_smithi122_119671_object32417:head# 2024-03-20T09:05:00.461 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.461 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba612258:::benchmark_data_smithi122_119671_object62959:head# 2024-03-20T09:05:00.461 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.461 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba6227c5:::benchmark_data_smithi122_119671_object51669:head# 2024-03-20T09:05:00.461 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.461 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba625fc8:::benchmark_data_smithi122_119671_object68204:head# 2024-03-20T09:05:00.461 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.461 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba6298e9:::benchmark_data_smithi122_119671_object38230:head# 2024-03-20T09:05:00.461 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.462 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba63ca6a:::benchmark_data_smithi122_119671_object36056:head# 2024-03-20T09:05:00.462 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.462 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba64154d:::benchmark_data_smithi122_119671_object23885:head# 2024-03-20T09:05:00.462 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.462 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba642f91:::benchmark_data_smithi122_119671_object95033:head# 2024-03-20T09:05:00.462 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.462 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba65b8b1:::benchmark_data_smithi122_119671_object56563:head# 2024-03-20T09:05:00.462 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.462 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba67579f:::benchmark_data_smithi122_119671_object52332:head# 2024-03-20T09:05:00.462 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.462 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba67d1de:::benchmark_data_smithi122_119671_object81324:head# 2024-03-20T09:05:00.462 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.462 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba699673:::benchmark_data_smithi122_119671_object74286:head# 2024-03-20T09:05:00.462 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.463 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba6aed3b:::benchmark_data_smithi122_119671_object53571:head# 2024-03-20T09:05:00.463 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.463 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba6b3715:::benchmark_data_smithi122_119671_object70779:head# 2024-03-20T09:05:00.463 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.463 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba6c7d4b:::benchmark_data_smithi122_119671_object90369:head# 2024-03-20T09:05:00.463 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.463 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba6ccca1:::benchmark_data_smithi122_119671_object19735:head# 2024-03-20T09:05:00.463 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.463 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba6d1494:::benchmark_data_smithi122_119671_object13325:head# 2024-03-20T09:05:00.463 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.463 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba6de6e7:::benchmark_data_smithi122_119671_object45534:head# 2024-03-20T09:05:00.463 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.463 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba716418:::benchmark_data_smithi122_119671_object72557:head# 2024-03-20T09:05:00.463 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.464 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba722304:::benchmark_data_smithi122_119671_object89612:head# 2024-03-20T09:05:00.464 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.464 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba72481a:::benchmark_data_smithi122_119671_object69160:head# 2024-03-20T09:05:00.464 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.464 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba73784b:::benchmark_data_smithi122_119671_object9131:head# 2024-03-20T09:05:00.464 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.464 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba7436e1:::benchmark_data_smithi122_119671_object95479:head# 2024-03-20T09:05:00.464 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.464 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba7557bf:::benchmark_data_smithi122_119671_object80137:head# 2024-03-20T09:05:00.464 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.464 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba75ea28:::benchmark_data_smithi122_119671_object40531:head# 2024-03-20T09:05:00.464 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.464 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba76a074:::benchmark_data_smithi122_119671_object25896:head# 2024-03-20T09:05:00.464 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.465 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba7c8512:::benchmark_data_smithi122_119671_object71751:head# 2024-03-20T09:05:00.465 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.465 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba7d3efb:::benchmark_data_smithi122_119671_object1560:head# 2024-03-20T09:05:00.465 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.465 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba7de744:::benchmark_data_smithi122_119671_object33413:head# 2024-03-20T09:05:00.465 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.465 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba7e0801:::benchmark_data_smithi122_119671_object82300:head# 2024-03-20T09:05:00.465 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.465 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba7e2e6e:::benchmark_data_smithi122_119671_object15278:head# 2024-03-20T09:05:00.465 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.465 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba7e4cda:::benchmark_data_smithi122_119671_object26054:head# 2024-03-20T09:05:00.465 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.465 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba7ecfbc:::benchmark_data_smithi122_119671_object74200:head# 2024-03-20T09:05:00.465 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.466 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba7f4751:::benchmark_data_smithi122_119671_object5430:head# 2024-03-20T09:05:00.466 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.466 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba824838:::benchmark_data_smithi122_119671_object80811:head# 2024-03-20T09:05:00.466 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.466 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba82bdbf:::benchmark_data_smithi122_119671_object46823:head# 2024-03-20T09:05:00.466 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.466 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba8605c2:::benchmark_data_smithi122_119671_object62825:head# 2024-03-20T09:05:00.466 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.466 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba87a4a0:::benchmark_data_smithi122_119671_object80813:head# 2024-03-20T09:05:00.466 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.466 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba87ec6f:::benchmark_data_smithi122_119671_object31600:head# 2024-03-20T09:05:00.466 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.466 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba880ae8:::benchmark_data_smithi122_119671_object31107:head# 2024-03-20T09:05:00.467 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.467 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba880b8f:::benchmark_data_smithi122_119671_object30545:head# 2024-03-20T09:05:00.467 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.467 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba88c222:::benchmark_data_smithi122_119671_object80313:head# 2024-03-20T09:05:00.467 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.467 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba8a2853:::benchmark_data_smithi122_119671_object6239:head# 2024-03-20T09:05:00.467 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.467 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba8addfa:::benchmark_data_smithi122_119671_object97530:head# 2024-03-20T09:05:00.467 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.467 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba8bbe78:::benchmark_data_smithi122_119671_object91617:head# 2024-03-20T09:05:00.467 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.467 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba8be9ac:::benchmark_data_smithi122_119671_object41664:head# 2024-03-20T09:05:00.467 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.467 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba8c87b3:::benchmark_data_smithi122_119671_object84425:head# 2024-03-20T09:05:00.467 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.468 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba8c9900:::benchmark_data_smithi122_119671_object7450:head# 2024-03-20T09:05:00.468 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.468 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba8d766f:::benchmark_data_smithi122_119671_object69285:head# 2024-03-20T09:05:00.468 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.468 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba8ea004:::benchmark_data_smithi122_119671_object61643:head# 2024-03-20T09:05:00.468 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.468 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba8eb75d:::benchmark_data_smithi122_119671_object81764:head# 2024-03-20T09:05:00.468 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.468 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba8f2c76:::benchmark_data_smithi122_119671_object25338:head# 2024-03-20T09:05:00.468 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.468 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba8fa5b8:::benchmark_data_smithi122_119671_object11442:head# 2024-03-20T09:05:00.468 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.468 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba94b513:::benchmark_data_smithi122_119671_object77926:head# 2024-03-20T09:05:00.468 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.469 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba94d05d:::benchmark_data_smithi122_119671_object38771:head# 2024-03-20T09:05:00.469 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.469 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba94da50:::benchmark_data_smithi122_119671_object20782:head# 2024-03-20T09:05:00.469 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.469 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba94eb85:::benchmark_data_smithi122_119671_object58374:head# 2024-03-20T09:05:00.469 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.469 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba968efd:::benchmark_data_smithi122_119671_object73007:head# 2024-03-20T09:05:00.469 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.469 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba96a90b:::benchmark_data_smithi122_119671_object17387:head# 2024-03-20T09:05:00.469 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.469 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba96db7d:::benchmark_data_smithi122_119671_object73516:head# 2024-03-20T09:05:00.469 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.469 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba971839:::benchmark_data_smithi122_119671_object56982:head# 2024-03-20T09:05:00.469 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.470 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba976ee5:::benchmark_data_smithi122_119671_object52759:head# 2024-03-20T09:05:00.470 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.470 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba97e222:::benchmark_data_smithi122_119671_object91002:head# 2024-03-20T09:05:00.470 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.470 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba9867ae:::benchmark_data_smithi122_119671_object26677:head# 2024-03-20T09:05:00.470 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.470 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba992878:::benchmark_data_smithi122_119671_object18962:head# 2024-03-20T09:05:00.470 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.470 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba99e067:::benchmark_data_smithi122_119671_object79483:head# 2024-03-20T09:05:00.470 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.470 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba9a06d7:::benchmark_data_smithi122_119671_object86722:head# 2024-03-20T09:05:00.470 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.470 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba9a74c3:::benchmark_data_smithi122_119671_object23278:head# 2024-03-20T09:05:00.471 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.471 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba9aaf83:::benchmark_data_smithi122_119671_object70728:head# 2024-03-20T09:05:00.471 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.471 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba9baa2e:::benchmark_data_smithi122_119671_object60582:head# 2024-03-20T09:05:00.471 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.471 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba9be5a7:::benchmark_data_smithi122_119671_object55001:head# 2024-03-20T09:05:00.471 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.471 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba9c43da:::benchmark_data_smithi122_119671_object11643:head# 2024-03-20T09:05:00.471 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.471 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba9c5ef2:::benchmark_data_smithi122_119671_object69697:head# 2024-03-20T09:05:00.471 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.471 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba9dd4c8:::benchmark_data_smithi122_119671_object75717:head# 2024-03-20T09:05:00.471 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.472 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba9e52af:::benchmark_data_smithi122_119671_object95639:head# 2024-03-20T09:05:00.472 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.472 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba9eb24b:::benchmark_data_smithi122_119671_object69672:head# 2024-03-20T09:05:00.472 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.472 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:ba9ff202:::benchmark_data_smithi122_119671_object98503:head# 2024-03-20T09:05:00.472 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.472 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baa01b3e:::benchmark_data_smithi122_119671_object50894:head# 2024-03-20T09:05:00.472 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.472 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baa091d6:::benchmark_data_smithi122_119671_object31983:head# 2024-03-20T09:05:00.472 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.472 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baa1218c:::benchmark_data_smithi122_119671_object50583:head# 2024-03-20T09:05:00.472 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.472 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baa137e9:::benchmark_data_smithi122_119671_object57851:head# 2024-03-20T09:05:00.473 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.473 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baa1e6a9:::benchmark_data_smithi122_119671_object3816:head# 2024-03-20T09:05:00.473 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.473 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baa2c4ce:::benchmark_data_smithi122_119671_object34695:head# 2024-03-20T09:05:00.473 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.473 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baa3a88f:::benchmark_data_smithi122_119671_object6764:head# 2024-03-20T09:05:00.473 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.473 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baa4053e:::benchmark_data_smithi122_119671_object16846:head# 2024-03-20T09:05:00.473 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.473 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baa42b27:::benchmark_data_smithi122_119671_object8214:head# 2024-03-20T09:05:00.473 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.473 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baa4915a:::benchmark_data_smithi122_119671_object10817:head# 2024-03-20T09:05:00.474 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.474 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baa57c95:::benchmark_data_smithi122_119671_object20290:head# 2024-03-20T09:05:00.474 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.474 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baa5cf84:::benchmark_data_smithi122_119671_object62414:head# 2024-03-20T09:05:00.474 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.474 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baa5dec9:::benchmark_data_smithi122_119671_object31320:head# 2024-03-20T09:05:00.474 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.474 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baa65cac:::benchmark_data_smithi122_119671_object21458:head# 2024-03-20T09:05:00.474 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.474 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baa670a2:::benchmark_data_smithi122_119671_object86220:head# 2024-03-20T09:05:00.474 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.474 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baa6d94b:::benchmark_data_smithi122_119671_object59353:head# 2024-03-20T09:05:00.474 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.474 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baa821d7:::benchmark_data_smithi122_119671_object48822:head# 2024-03-20T09:05:00.475 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.475 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baa8a604:::benchmark_data_smithi122_119671_object38036:head# 2024-03-20T09:05:00.475 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.475 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baa98fb3:::benchmark_data_smithi122_119671_object9490:head# 2024-03-20T09:05:00.475 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.475 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baa99e0d:::benchmark_data_smithi122_119671_object65977:head# 2024-03-20T09:05:00.475 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.475 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baabc7e0:::benchmark_data_smithi122_119671_object36800:head# 2024-03-20T09:05:00.475 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.475 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baabf4ce:::benchmark_data_smithi122_119671_object19922:head# 2024-03-20T09:05:00.475 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.475 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baacd221:::benchmark_data_smithi122_119671_object23399:head# 2024-03-20T09:05:00.475 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.476 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baacefc3:::benchmark_data_smithi122_119671_object35899:head# 2024-03-20T09:05:00.476 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.476 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baae05d5:::benchmark_data_smithi122_119671_object62246:head# 2024-03-20T09:05:00.476 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.476 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baae3d92:::benchmark_data_smithi122_119671_object4395:head# 2024-03-20T09:05:00.476 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.476 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baaf3089:::benchmark_data_smithi122_119671_object97665:head# 2024-03-20T09:05:00.476 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.476 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baaf3e4b:::benchmark_data_smithi122_119671_object52895:head# 2024-03-20T09:05:00.476 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.477 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baaf7277:::benchmark_data_smithi122_119671_object1433:head# 2024-03-20T09:05:00.477 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.477 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bab06ea6:::benchmark_data_smithi122_119671_object12195:head# 2024-03-20T09:05:00.477 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.477 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bab0b7c0:::benchmark_data_smithi122_119671_object4359:head# 2024-03-20T09:05:00.477 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.477 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bab17554:::benchmark_data_smithi122_119671_object89977:head# 2024-03-20T09:05:00.477 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.477 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bab27437:::benchmark_data_smithi122_119671_object38668:head# 2024-03-20T09:05:00.477 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.477 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bab3b3ee:::benchmark_data_smithi122_119671_object42599:head# 2024-03-20T09:05:00.477 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.478 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bab42f6a:::benchmark_data_smithi122_119671_object2917:head# 2024-03-20T09:05:00.478 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.478 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bab4f98c:::benchmark_data_smithi122_119671_object98288:head# 2024-03-20T09:05:00.478 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.478 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bab5ae1a:::benchmark_data_smithi122_119671_object37953:head# 2024-03-20T09:05:00.478 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.478 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bab6ab5e:::benchmark_data_smithi122_119671_object97174:head# 2024-03-20T09:05:00.478 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.478 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bab7f7f5:::benchmark_data_smithi122_119671_object64844:head# 2024-03-20T09:05:00.478 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.478 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bab84668:::benchmark_data_smithi122_119671_object21677:head# 2024-03-20T09:05:00.478 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.478 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bab9b168:::benchmark_data_smithi122_119671_object25379:head# 2024-03-20T09:05:00.479 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.479 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baba536b:::benchmark_data_smithi122_119671_object17017:head# 2024-03-20T09:05:00.479 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.479 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baba71be:::benchmark_data_smithi122_119671_object40107:head# 2024-03-20T09:05:00.479 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.479 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baba9bba:::benchmark_data_smithi122_119671_object25056:head# 2024-03-20T09:05:00.479 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.479 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:babaf4ee:::benchmark_data_smithi122_119671_object21610:head# 2024-03-20T09:05:00.479 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.479 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:babb57d7:::benchmark_data_smithi122_119671_object34144:head# 2024-03-20T09:05:00.479 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.479 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:babb7666:::benchmark_data_smithi122_119671_object42852:head# 2024-03-20T09:05:00.480 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.480 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:babc1af4:::benchmark_data_smithi122_119671_object62364:head# 2024-03-20T09:05:00.480 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.480 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:babc51c7:::benchmark_data_smithi122_119671_object33816:head# 2024-03-20T09:05:00.480 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.480 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:babc6ffe:::benchmark_data_smithi122_119671_object96486:head# 2024-03-20T09:05:00.480 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.480 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:babd449b:::benchmark_data_smithi122_119671_object9668:head# 2024-03-20T09:05:00.480 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.480 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:babd9fc3:::benchmark_data_smithi122_119671_object57627:head# 2024-03-20T09:05:00.480 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.480 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:babe7350:::benchmark_data_smithi122_119671_object41581:head# 2024-03-20T09:05:00.480 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.480 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:babfbf48:::benchmark_data_smithi122_119671_object59785:head# 2024-03-20T09:05:00.480 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.481 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bac01339:::benchmark_data_smithi122_119671_object91590:head# 2024-03-20T09:05:00.481 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.481 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bac020d3:::benchmark_data_smithi122_119671_object15836:head# 2024-03-20T09:05:00.481 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.481 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bac03571:::benchmark_data_smithi122_119671_object44749:head# 2024-03-20T09:05:00.481 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.481 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bac04ca3:::benchmark_data_smithi122_119671_object19337:head# 2024-03-20T09:05:00.481 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.481 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bac05012:::benchmark_data_smithi122_119671_object52734:head# 2024-03-20T09:05:00.481 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.481 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bac06ac7:::benchmark_data_smithi122_119671_object91601:head# 2024-03-20T09:05:00.481 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.481 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bac1e0cf:::benchmark_data_smithi122_119671_object53768:head# 2024-03-20T09:05:00.482 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.482 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bac20591:::benchmark_data_smithi122_119671_object75567:head# 2024-03-20T09:05:00.482 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.482 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bac20a7b:::benchmark_data_smithi122_119671_object36997:head# 2024-03-20T09:05:00.482 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.482 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bac2ae68:::benchmark_data_smithi122_119671_object70990:head# 2024-03-20T09:05:00.482 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.482 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bac40f39:::benchmark_data_smithi122_119671_object1671:head# 2024-03-20T09:05:00.482 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.482 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bac50b37:::benchmark_data_smithi122_119671_object4856:head# 2024-03-20T09:05:00.482 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.482 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bac563a6:::benchmark_data_smithi122_119671_object25588:head# 2024-03-20T09:05:00.482 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.483 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bac56526:::benchmark_data_smithi122_119671_object27834:head# 2024-03-20T09:05:00.483 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.483 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bac5a80d:::benchmark_data_smithi122_119671_object33096:head# 2024-03-20T09:05:00.483 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.483 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bac6fb38:::benchmark_data_smithi122_119671_object37484:head# 2024-03-20T09:05:00.483 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.483 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bac76649:::benchmark_data_smithi122_119671_object22779:head# 2024-03-20T09:05:00.483 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.483 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bac85f2d:::benchmark_data_smithi122_119671_object95395:head# 2024-03-20T09:05:00.483 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.483 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bac9ccd2:::benchmark_data_smithi122_119671_object24940:head# 2024-03-20T09:05:00.483 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.483 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baca15ad:::benchmark_data_smithi122_119671_object37870:head# 2024-03-20T09:05:00.484 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.484 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baccb4af:::benchmark_data_smithi122_119671_object97559:head# 2024-03-20T09:05:00.484 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.484 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bacd0709:::benchmark_data_smithi122_119671_object15708:head# 2024-03-20T09:05:00.484 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.484 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bace42aa:::benchmark_data_smithi122_119671_object10467:head# 2024-03-20T09:05:00.484 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.484 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bace5f9e:::benchmark_data_smithi122_119671_object77594:head# 2024-03-20T09:05:00.484 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.484 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baceda40:::benchmark_data_smithi122_119671_object28989:head# 2024-03-20T09:05:00.484 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.484 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bacf71d8:::benchmark_data_smithi122_119671_object19430:head# 2024-03-20T09:05:00.485 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.485 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bacf9762:::benchmark_data_smithi122_119671_object79575:head# 2024-03-20T09:05:00.485 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.485 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bacfceae:::benchmark_data_smithi122_119671_object46188:head# 2024-03-20T09:05:00.485 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.485 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bad067ea:::benchmark_data_smithi122_119671_object17890:head# 2024-03-20T09:05:00.485 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.485 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bad0ff23:::benchmark_data_smithi122_119671_object86440:head# 2024-03-20T09:05:00.485 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.485 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bad399ff:::benchmark_data_smithi122_119671_object15205:head# 2024-03-20T09:05:00.485 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.485 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bad3b076:::benchmark_data_smithi122_119671_object65573:head# 2024-03-20T09:05:00.485 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.486 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bad437f7:::benchmark_data_smithi122_119671_object58493:head# 2024-03-20T09:05:00.486 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.486 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bad54f90:::benchmark_data_smithi122_119671_object4184:head# 2024-03-20T09:05:00.486 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.486 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bad5b5ad:::benchmark_data_smithi122_119671_object55709:head# 2024-03-20T09:05:00.486 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.486 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bad5f198:::benchmark_data_smithi122_119671_object8592:head# 2024-03-20T09:05:00.486 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.486 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bad7b57d:::benchmark_data_smithi122_119671_object49158:head# 2024-03-20T09:05:00.486 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.486 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bad7de60:::benchmark_data_smithi122_119671_object2317:head# 2024-03-20T09:05:00.486 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.486 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bad8bbeb:::benchmark_data_smithi122_119671_object22601:head# 2024-03-20T09:05:00.487 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.487 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bad8f966:::benchmark_data_smithi122_119671_object60136:head# 2024-03-20T09:05:00.487 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.487 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bada0cf9:::benchmark_data_smithi122_119671_object91207:head# 2024-03-20T09:05:00.487 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.487 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bada8ca6:::benchmark_data_smithi122_119671_object38207:head# 2024-03-20T09:05:00.487 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.487 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:badab945:::benchmark_data_smithi122_119671_object58125:head# 2024-03-20T09:05:00.487 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.487 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:badb1f91:::benchmark_data_smithi122_119671_object77677:head# 2024-03-20T09:05:00.487 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.487 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:badd5728:::benchmark_data_smithi122_119671_object60411:head# 2024-03-20T09:05:00.487 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.487 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bade4ec3:::benchmark_data_smithi122_119671_object18176:head# 2024-03-20T09:05:00.488 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.488 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:badf0281:::benchmark_data_smithi122_119671_object63018:head# 2024-03-20T09:05:00.488 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.488 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:badfdc3a:::benchmark_data_smithi122_119671_object27763:head# 2024-03-20T09:05:00.488 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.488 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bae20793:::benchmark_data_smithi122_119671_object19330:head# 2024-03-20T09:05:00.488 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.488 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bae3309a:::benchmark_data_smithi122_119671_object13014:head# 2024-03-20T09:05:00.488 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.488 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bae3826b:::benchmark_data_smithi122_119671_object27379:head# 2024-03-20T09:05:00.488 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.488 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bae61227:::benchmark_data_smithi122_119671_object88684:head# 2024-03-20T09:05:00.488 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.488 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bae63f8c:::benchmark_data_smithi122_119671_object7774:head# 2024-03-20T09:05:00.489 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.489 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bae77786:::benchmark_data_smithi122_119671_object45236:head# 2024-03-20T09:05:00.489 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.489 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bae81113:::benchmark_data_smithi122_119671_object62797:head# 2024-03-20T09:05:00.489 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.489 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bae8bb71:::benchmark_data_smithi122_119671_object29174:head# 2024-03-20T09:05:00.489 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.489 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bae9435c:::benchmark_data_smithi122_119671_object25356:head# 2024-03-20T09:05:00.489 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.489 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baebade3:::benchmark_data_smithi122_119671_object64434:head# 2024-03-20T09:05:00.489 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.489 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baebbba3:::benchmark_data_smithi122_119671_object29523:head# 2024-03-20T09:05:00.489 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.489 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baec299b:::benchmark_data_smithi122_119671_object93242:head# 2024-03-20T09:05:00.490 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.490 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baed7c2b:::benchmark_data_smithi122_119671_object55072:head# 2024-03-20T09:05:00.490 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.490 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baedb226:::benchmark_data_smithi122_119671_object58228:head# 2024-03-20T09:05:00.490 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.490 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baedb438:::benchmark_data_smithi122_119671_object15292:head# 2024-03-20T09:05:00.490 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.490 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baede2c5:::benchmark_data_smithi122_119671_object44989:head# 2024-03-20T09:05:00.490 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.490 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baede6d8:::benchmark_data_smithi122_119671_object57236:head# 2024-03-20T09:05:00.490 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.490 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baef2e92:::benchmark_data_smithi122_119671_object46199:head# 2024-03-20T09:05:00.490 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.491 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baef94d0:::benchmark_data_smithi122_119671_object87350:head# 2024-03-20T09:05:00.491 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.491 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baefcfff:::benchmark_data_smithi122_119671_object57224:head# 2024-03-20T09:05:00.491 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.491 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baf018e0:::benchmark_data_smithi122_119671_object61997:head# 2024-03-20T09:05:00.491 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.491 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baf09bd1:::benchmark_data_smithi122_119671_object28046:head# 2024-03-20T09:05:00.491 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.491 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baf1100f:::benchmark_data_smithi122_119671_object68329:head# 2024-03-20T09:05:00.491 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.491 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baf124aa:::benchmark_data_smithi122_119671_object41882:head# 2024-03-20T09:05:00.491 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.491 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baf1ff53:::benchmark_data_smithi122_119671_object40353:head# 2024-03-20T09:05:00.492 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.492 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baf29e77:::benchmark_data_smithi122_119671_object79265:head# 2024-03-20T09:05:00.492 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.492 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baf2aa73:::benchmark_data_smithi122_119671_object62656:head# 2024-03-20T09:05:00.492 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.492 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baf3d633:::benchmark_data_smithi122_119671_object35647:head# 2024-03-20T09:05:00.492 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.492 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baf4146a:::benchmark_data_smithi122_119671_object35731:head# 2024-03-20T09:05:00.492 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.492 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baf43e86:::benchmark_data_smithi122_119671_object87667:head# 2024-03-20T09:05:00.492 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.492 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baf480e3:::benchmark_data_smithi122_119671_object68483:head# 2024-03-20T09:05:00.492 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.492 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baf4c773:::benchmark_data_smithi122_119671_object32586:head# 2024-03-20T09:05:00.493 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.493 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baf54b13:::benchmark_data_smithi122_119671_object24644:head# 2024-03-20T09:05:00.493 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.493 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baf75d4b:::benchmark_data_smithi122_119671_object45435:head# 2024-03-20T09:05:00.493 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.493 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baf8432f:::benchmark_data_smithi122_119671_object24993:head# 2024-03-20T09:05:00.493 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.493 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baf870ff:::benchmark_data_smithi122_119671_object22279:head# 2024-03-20T09:05:00.493 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.493 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baf89457:::benchmark_data_smithi122_119671_object66851:head# 2024-03-20T09:05:00.493 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.493 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:baf9d592:::benchmark_data_smithi122_119671_object48939:head# 2024-03-20T09:05:00.493 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.493 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bafab91a:::benchmark_data_smithi122_119671_object93281:head# 2024-03-20T09:05:00.494 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.494 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bafaeaa6:::benchmark_data_smithi122_119671_object22483:head# 2024-03-20T09:05:00.494 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.494 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bafb6838:::benchmark_data_smithi122_119671_object60564:head# 2024-03-20T09:05:00.494 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.494 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bafb9aff:::benchmark_data_smithi122_119671_object82033:head# 2024-03-20T09:05:00.494 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.494 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bafcb123:::benchmark_data_smithi122_119671_object1916:head# 2024-03-20T09:05:00.494 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.494 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bafe4192:::benchmark_data_smithi122_119671_object78683:head# 2024-03-20T09:05:00.494 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.494 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb0229d6:::benchmark_data_smithi122_119671_object24701:head# 2024-03-20T09:05:00.494 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.494 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb02c3ff:::benchmark_data_smithi122_119671_object35280:head# 2024-03-20T09:05:00.495 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.495 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb02d9d9:::benchmark_data_smithi122_119671_object19634:head# 2024-03-20T09:05:00.495 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.495 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb058d12:::benchmark_data_smithi122_119671_object94761:head# 2024-03-20T09:05:00.495 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.495 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb05d0ae:::benchmark_data_smithi122_119671_object24368:head# 2024-03-20T09:05:00.495 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.495 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb06116e:::benchmark_data_smithi122_119671_object37632:head# 2024-03-20T09:05:00.495 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.495 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb0645d1:::benchmark_data_smithi122_119671_object44623:head# 2024-03-20T09:05:00.495 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.495 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb0669a7:::benchmark_data_smithi122_119671_object3328:head# 2024-03-20T09:05:00.495 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.495 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb0741b3:::benchmark_data_smithi122_119671_object14696:head# 2024-03-20T09:05:00.495 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.496 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb078d5a:::benchmark_data_smithi122_119671_object95062:head# 2024-03-20T09:05:00.496 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.496 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb07998d:::benchmark_data_smithi122_119671_object84042:head# 2024-03-20T09:05:00.496 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.496 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb08c630:::benchmark_data_smithi122_119671_object61411:head# 2024-03-20T09:05:00.496 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.496 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb09d227:::benchmark_data_smithi122_119671_object73020:head# 2024-03-20T09:05:00.496 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.496 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb0b219c:::benchmark_data_smithi122_119671_object93907:head# 2024-03-20T09:05:00.496 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.496 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb0bb3ab:::benchmark_data_smithi122_119671_object56230:head# 2024-03-20T09:05:00.496 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.496 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb0bbf68:::benchmark_data_smithi122_119671_object97397:head# 2024-03-20T09:05:00.496 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.497 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb0c3e0c:::benchmark_data_smithi122_119671_object10225:head# 2024-03-20T09:05:00.497 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.497 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb0c71fc:::benchmark_data_smithi122_119671_object38384:head# 2024-03-20T09:05:00.497 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.497 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb0c9ff9:::benchmark_data_smithi122_119671_object64833:head# 2024-03-20T09:05:00.497 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.497 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb0d1470:::benchmark_data_smithi122_119671_object17714:head# 2024-03-20T09:05:00.497 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.497 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb0d68bb:::benchmark_data_smithi122_119671_object33420:head# 2024-03-20T09:05:00.497 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.497 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb101a30:::benchmark_data_smithi122_119671_object35454:head# 2024-03-20T09:05:00.497 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.497 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb102b2f:::benchmark_data_smithi122_119671_object61641:head# 2024-03-20T09:05:00.497 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.498 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb10dfe7:::benchmark_data_smithi122_119671_object79163:head# 2024-03-20T09:05:00.498 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.498 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb112f88:::benchmark_data_smithi122_119671_object11607:head# 2024-03-20T09:05:00.498 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.498 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb11f9a6:::benchmark_data_smithi122_119671_object34734:head# 2024-03-20T09:05:00.498 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.498 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb12ce4e:::benchmark_data_smithi122_119671_object7020:head# 2024-03-20T09:05:00.498 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.498 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb14252c:::benchmark_data_smithi122_119671_object42118:head# 2024-03-20T09:05:00.498 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.498 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb14c208:::benchmark_data_smithi122_119671_object51714:head# 2024-03-20T09:05:00.498 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.498 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb150d21:::benchmark_data_smithi122_119671_object96462:head# 2024-03-20T09:05:00.499 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.499 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb151743:::benchmark_data_smithi122_119671_object61602:head# 2024-03-20T09:05:00.499 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.499 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb156b54:::benchmark_data_smithi122_119671_object61004:head# 2024-03-20T09:05:00.499 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.499 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb163f6d:::benchmark_data_smithi122_119671_object24974:head# 2024-03-20T09:05:00.499 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.499 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb17cad0:::benchmark_data_smithi122_119671_object57570:head# 2024-03-20T09:05:00.499 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.499 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb183d02:::benchmark_data_smithi122_119671_object61569:head# 2024-03-20T09:05:00.499 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.499 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb18e300:::benchmark_data_smithi122_119671_object73592:head# 2024-03-20T09:05:00.499 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.499 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb18eb04:::benchmark_data_smithi122_119671_object78531:head# 2024-03-20T09:05:00.500 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.500 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb1995fe:::benchmark_data_smithi122_119671_object50415:head# 2024-03-20T09:05:00.500 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.500 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb199888:::benchmark_data_smithi122_119671_object17726:head# 2024-03-20T09:05:00.500 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.500 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb1a1cf6:::benchmark_data_smithi122_119671_object84679:head# 2024-03-20T09:05:00.500 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.500 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb1a2219:::benchmark_data_smithi122_119671_object36843:head# 2024-03-20T09:05:00.500 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.500 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb1a8637:::benchmark_data_smithi122_119671_object12331:head# 2024-03-20T09:05:00.500 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.500 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb1bb1f4:::benchmark_data_smithi122_119671_object98196:head# 2024-03-20T09:05:00.500 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.501 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb1e1c84:::benchmark_data_smithi122_119671_object96793:head# 2024-03-20T09:05:00.501 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.501 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb1ef7ed:::benchmark_data_smithi122_119671_object84652:head# 2024-03-20T09:05:00.501 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.501 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb1f5ac3:::benchmark_data_smithi122_119671_object98319:head# 2024-03-20T09:05:00.501 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.501 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb21644d:::benchmark_data_smithi122_119671_object51778:head# 2024-03-20T09:05:00.501 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.501 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb217190:::benchmark_data_smithi122_119671_object95618:head# 2024-03-20T09:05:00.501 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.501 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb218ee9:::benchmark_data_smithi122_119671_object25297:head# 2024-03-20T09:05:00.501 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.501 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb21949b:::benchmark_data_smithi122_119671_object92241:head# 2024-03-20T09:05:00.501 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.502 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb21c771:::benchmark_data_smithi122_119671_object42383:head# 2024-03-20T09:05:00.502 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.502 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb21dcf7:::benchmark_data_smithi122_119671_object98865:head# 2024-03-20T09:05:00.502 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.502 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb22d906:::benchmark_data_smithi122_119671_object13855:head# 2024-03-20T09:05:00.502 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.502 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb22f378:::benchmark_data_smithi122_119671_object62928:head# 2024-03-20T09:05:00.502 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.502 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb2341d0:::benchmark_data_smithi122_119671_object63081:head# 2024-03-20T09:05:00.502 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.502 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb23682b:::benchmark_data_smithi122_119671_object51518:head# 2024-03-20T09:05:00.502 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.502 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb239bd4:::benchmark_data_smithi122_119671_object77224:head# 2024-03-20T09:05:00.502 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.503 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb264bca:::benchmark_data_smithi122_119671_object8502:head# 2024-03-20T09:05:00.503 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.503 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb282f7a:::benchmark_data_smithi122_119671_object50937:head# 2024-03-20T09:05:00.503 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.503 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb28acfc:::benchmark_data_smithi122_119671_object53440:head# 2024-03-20T09:05:00.503 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.503 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb28c810:::benchmark_data_smithi122_119671_object21380:head# 2024-03-20T09:05:00.503 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.503 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb29b98c:::benchmark_data_smithi122_119671_object66787:head# 2024-03-20T09:05:00.503 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.503 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb2a4bba:::benchmark_data_smithi122_119671_object26860:head# 2024-03-20T09:05:00.503 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.503 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb2a5009:::benchmark_data_smithi122_119671_object56535:head# 2024-03-20T09:05:00.503 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.503 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb2a9a92:::benchmark_data_smithi122_119671_object67528:head# 2024-03-20T09:05:00.504 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.504 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb2b96ab:::benchmark_data_smithi122_119671_object85460:head# 2024-03-20T09:05:00.504 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.504 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb2c7f61:::benchmark_data_smithi122_119671_object42228:head# 2024-03-20T09:05:00.504 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.504 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb2e0f33:::benchmark_data_smithi122_119671_object30395:head# 2024-03-20T09:05:00.504 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.504 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb2e7b92:::benchmark_data_smithi122_119671_object31068:head# 2024-03-20T09:05:00.504 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.504 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb2ea86b:::benchmark_data_smithi122_119671_object26746:head# 2024-03-20T09:05:00.504 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.504 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb2fc436:::benchmark_data_smithi122_119671_object84628:head# 2024-03-20T09:05:00.504 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.505 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb31c073:::benchmark_data_smithi122_119671_object1874:head# 2024-03-20T09:05:00.505 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.505 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb330f36:::benchmark_data_smithi122_119671_object46679:head# 2024-03-20T09:05:00.505 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.505 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb3357a1:::benchmark_data_smithi122_119671_object55482:head# 2024-03-20T09:05:00.505 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.505 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb33c5f3:::benchmark_data_smithi122_119671_object17794:head# 2024-03-20T09:05:00.505 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.505 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb33f22f:::benchmark_data_smithi122_119671_object51466:head# 2024-03-20T09:05:00.505 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.505 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb350a24:::benchmark_data_smithi122_119671_object39987:head# 2024-03-20T09:05:00.505 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.505 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb3541fc:::benchmark_data_smithi122_119671_object12639:head# 2024-03-20T09:05:00.506 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.506 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb360b9d:::benchmark_data_smithi122_119671_object16517:head# 2024-03-20T09:05:00.506 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.506 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb387b7d:::benchmark_data_smithi122_119671_object97686:head# 2024-03-20T09:05:00.506 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.506 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb395ec5:::benchmark_data_smithi122_119671_object23453:head# 2024-03-20T09:05:00.506 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.506 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb3e50f6:::benchmark_data_smithi122_119671_object50916:head# 2024-03-20T09:05:00.506 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.506 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb3ecee4:::benchmark_data_smithi122_119671_object57634:head# 2024-03-20T09:05:00.506 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.506 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb3f7aab:::benchmark_data_smithi122_119671_object54023:head# 2024-03-20T09:05:00.506 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.507 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb40486d:::benchmark_data_smithi122_119671_object93593:head# 2024-03-20T09:05:00.507 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.507 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb406470:::benchmark_data_smithi122_119671_object44842:head# 2024-03-20T09:05:00.507 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.507 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb40d18a:::benchmark_data_smithi122_119671_object91794:head# 2024-03-20T09:05:00.507 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.507 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb40eab9:::benchmark_data_smithi122_119671_object44288:head# 2024-03-20T09:05:00.507 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.507 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb419f40:::benchmark_data_smithi122_119671_object91871:head# 2024-03-20T09:05:00.507 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.507 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb41e56d:::benchmark_data_smithi122_119671_object15276:head# 2024-03-20T09:05:00.507 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.507 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb426bc7:::benchmark_data_smithi122_119671_object91782:head# 2024-03-20T09:05:00.507 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.508 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb42abd7:::benchmark_data_smithi122_119671_object84239:head# 2024-03-20T09:05:00.508 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.508 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb42c1c4:::benchmark_data_smithi122_119671_object27403:head# 2024-03-20T09:05:00.508 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.508 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb431813:::benchmark_data_smithi122_119671_object8894:head# 2024-03-20T09:05:00.508 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.508 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb440c58:::benchmark_data_smithi122_119671_object39682:head# 2024-03-20T09:05:00.508 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.508 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb4470a0:::benchmark_data_smithi122_119671_object27682:head# 2024-03-20T09:05:00.508 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.508 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb4478a2:::benchmark_data_smithi122_119671_object7953:head# 2024-03-20T09:05:00.508 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.508 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb4535b4:::benchmark_data_smithi122_119671_object65641:head# 2024-03-20T09:05:00.509 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.509 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb456d07:::benchmark_data_smithi122_119671_object38422:head# 2024-03-20T09:05:00.509 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.509 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb46bb3c:::benchmark_data_smithi122_119671_object79392:head# 2024-03-20T09:05:00.509 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.509 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb470ba7:::benchmark_data_smithi122_119671_object79085:head# 2024-03-20T09:05:00.509 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.509 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb472a82:::benchmark_data_smithi122_119671_object69522:head# 2024-03-20T09:05:00.509 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.509 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb4764b5:::benchmark_data_smithi122_119671_object67361:head# 2024-03-20T09:05:00.509 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.509 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb47c572:::benchmark_data_smithi122_119671_object17709:head# 2024-03-20T09:05:00.509 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.509 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb480c08:::benchmark_data_smithi122_119671_object46504:head# 2024-03-20T09:05:00.510 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.510 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb489db0:::benchmark_data_smithi122_119671_object25715:head# 2024-03-20T09:05:00.510 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.510 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb4ae3d2:::benchmark_data_smithi122_119671_object40131:head# 2024-03-20T09:05:00.510 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.510 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb4c3b15:::benchmark_data_smithi122_119671_object66152:head# 2024-03-20T09:05:00.510 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.510 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb4c6675:::benchmark_data_smithi122_119671_object89759:head# 2024-03-20T09:05:00.510 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.510 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb4d5232:::benchmark_data_smithi122_119671_object65184:head# 2024-03-20T09:05:00.510 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.510 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb4dd579:::benchmark_data_smithi122_119671_object76999:head# 2024-03-20T09:05:00.510 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.510 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb4e7810:::benchmark_data_smithi122_119671_object27209:head# 2024-03-20T09:05:00.511 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.511 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb4f7860:::benchmark_data_smithi122_119671_object83346:head# 2024-03-20T09:05:00.511 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.511 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb4ff0df:::benchmark_data_smithi122_119671_object18496:head# 2024-03-20T09:05:00.511 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.511 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb5102f3:::benchmark_data_smithi122_119671_object25719:head# 2024-03-20T09:05:00.511 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.511 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb510390:::benchmark_data_smithi122_119671_object94568:head# 2024-03-20T09:05:00.511 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.511 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb51b71f:::benchmark_data_smithi122_119671_object71851:head# 2024-03-20T09:05:00.511 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.511 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb52124f:::benchmark_data_smithi122_119671_object81451:head# 2024-03-20T09:05:00.511 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.511 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb527791:::benchmark_data_smithi122_119671_object57814:head# 2024-03-20T09:05:00.512 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.512 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb5475a0:::benchmark_data_smithi122_119671_object6153:head# 2024-03-20T09:05:00.512 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.512 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb54e1ce:::benchmark_data_smithi122_119671_object40071:head# 2024-03-20T09:05:00.512 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.512 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb54f40f:::benchmark_data_smithi122_119671_object12923:head# 2024-03-20T09:05:00.512 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.512 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb550e0b:::benchmark_data_smithi122_119671_object71619:head# 2024-03-20T09:05:00.512 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.512 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb55604e:::benchmark_data_smithi122_119671_object90429:head# 2024-03-20T09:05:00.512 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.512 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb5671c3:::benchmark_data_smithi122_119671_object59141:head# 2024-03-20T09:05:00.512 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.512 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb56a80f:::benchmark_data_smithi122_119671_object74429:head# 2024-03-20T09:05:00.513 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.513 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb56ba38:::benchmark_data_smithi122_119671_object19127:head# 2024-03-20T09:05:00.513 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.513 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb5792b2:::benchmark_data_smithi122_119671_object96513:head# 2024-03-20T09:05:00.513 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.513 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb596261:::benchmark_data_smithi122_119671_object52676:head# 2024-03-20T09:05:00.513 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.513 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb5a3620:::benchmark_data_smithi122_119671_object19675:head# 2024-03-20T09:05:00.513 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.513 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb5a7d96:::benchmark_data_smithi122_119671_object37626:head# 2024-03-20T09:05:00.513 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.513 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb5c0c79:::benchmark_data_smithi122_119671_object94088:head# 2024-03-20T09:05:00.513 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.514 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb5c8338:::benchmark_data_smithi122_119671_object19272:head# 2024-03-20T09:05:00.514 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.514 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb5c9eea:::benchmark_data_smithi122_119671_object15744:head# 2024-03-20T09:05:00.514 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.514 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb5d1270:::benchmark_data_smithi122_119671_object41696:head# 2024-03-20T09:05:00.514 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.514 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb5ea2f6:::benchmark_data_smithi122_119671_object97191:head# 2024-03-20T09:05:00.514 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.514 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb5f7166:::benchmark_data_smithi122_119671_object89630:head# 2024-03-20T09:05:00.514 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.514 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb5fc51d:::benchmark_data_smithi122_119671_object51394:head# 2024-03-20T09:05:00.514 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.514 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb61e9e3:::benchmark_data_smithi122_119671_object72801:head# 2024-03-20T09:05:00.515 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.515 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb623da2:::benchmark_data_smithi122_119671_object69625:head# 2024-03-20T09:05:00.515 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.515 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb62fec0:::benchmark_data_smithi122_119671_object3976:head# 2024-03-20T09:05:00.515 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.515 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb648ddd:::benchmark_data_smithi122_119671_object7501:head# 2024-03-20T09:05:00.515 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.515 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb649d02:::benchmark_data_smithi122_119671_object24399:head# 2024-03-20T09:05:00.515 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.515 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb649ffb:::benchmark_data_smithi122_119671_object61300:head# 2024-03-20T09:05:00.515 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.515 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb64db07:::benchmark_data_smithi122_119671_object65775:head# 2024-03-20T09:05:00.515 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.516 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb678889:::benchmark_data_smithi122_119671_object65152:head# 2024-03-20T09:05:00.516 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.516 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb6bef05:::benchmark_data_smithi122_119671_object73122:head# 2024-03-20T09:05:00.516 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.516 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb6c50a0:::benchmark_data_smithi122_119671_object5290:head# 2024-03-20T09:05:00.516 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.516 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb6c71d1:::benchmark_data_smithi122_119671_object21834:head# 2024-03-20T09:05:00.516 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.516 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb6ca104:::benchmark_data_smithi122_119671_object82143:head# 2024-03-20T09:05:00.516 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.516 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb6e6071:::benchmark_data_smithi122_119671_object13435:head# 2024-03-20T09:05:00.516 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.516 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb6ecfe3:::benchmark_data_smithi122_119671_object32259:head# 2024-03-20T09:05:00.516 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.517 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb6f2754:::benchmark_data_smithi122_119671_object68884:head# 2024-03-20T09:05:00.517 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.517 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb6ff720:::benchmark_data_smithi122_119671_object72677:head# 2024-03-20T09:05:00.517 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.517 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb7063f3:::benchmark_data_smithi122_119671_object64281:head# 2024-03-20T09:05:00.517 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.517 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb708f00:::benchmark_data_smithi122_119671_object86355:head# 2024-03-20T09:05:00.517 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.517 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb711236:::benchmark_data_smithi122_119671_object65557:head# 2024-03-20T09:05:00.517 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.517 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb71acdc:::benchmark_data_smithi122_119671_object71278:head# 2024-03-20T09:05:00.517 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.517 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb722566:::benchmark_data_smithi122_119671_object48649:head# 2024-03-20T09:05:00.518 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.518 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb722aab:::benchmark_data_smithi122_119671_object58846:head# 2024-03-20T09:05:00.518 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.518 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb72da50:::benchmark_data_smithi122_119671_object11031:head# 2024-03-20T09:05:00.518 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.518 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb73412f:::benchmark_data_smithi122_119671_object23190:head# 2024-03-20T09:05:00.518 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.518 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb7410bb:::benchmark_data_smithi122_119671_object96017:head# 2024-03-20T09:05:00.518 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.518 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb744e1e:::benchmark_data_smithi122_119671_object47691:head# 2024-03-20T09:05:00.518 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.518 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb74ba47:::benchmark_data_smithi122_119671_object25116:head# 2024-03-20T09:05:00.518 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.519 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb75533f:::benchmark_data_smithi122_119671_object29559:head# 2024-03-20T09:05:00.519 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.519 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb7621aa:::benchmark_data_smithi122_119671_object54574:head# 2024-03-20T09:05:00.519 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.519 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb7652f2:::benchmark_data_smithi122_119671_object29361:head# 2024-03-20T09:05:00.519 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.519 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb784b6c:::benchmark_data_smithi122_119671_object38104:head# 2024-03-20T09:05:00.519 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.519 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb7a93b9:::benchmark_data_smithi122_119671_object88913:head# 2024-03-20T09:05:00.519 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.519 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb7ad152:::benchmark_data_smithi122_119671_object34070:head# 2024-03-20T09:05:00.519 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.520 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb7d7908:::benchmark_data_smithi122_119671_object26337:head# 2024-03-20T09:05:00.520 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.520 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb7ed8de:::benchmark_data_smithi122_119671_object30015:head# 2024-03-20T09:05:00.520 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.520 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb7eec4c:::benchmark_data_smithi122_119671_object70033:head# 2024-03-20T09:05:00.520 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.520 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb7f3d50:::benchmark_data_smithi122_119671_object29759:head# 2024-03-20T09:05:00.520 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.520 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb82102a:::benchmark_data_smithi122_119671_object1278:head# 2024-03-20T09:05:00.520 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.520 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb8492ed:::benchmark_data_smithi122_119671_object18710:head# 2024-03-20T09:05:00.520 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.520 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb84c761:::benchmark_data_smithi122_119671_object12047:head# 2024-03-20T09:05:00.521 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.521 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb863661:::benchmark_data_smithi122_119671_object43664:head# 2024-03-20T09:05:00.521 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.521 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb86a35d:::benchmark_data_smithi122_119671_object60529:head# 2024-03-20T09:05:00.521 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.521 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb86d46f:::benchmark_data_smithi122_119671_object97139:head# 2024-03-20T09:05:00.521 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.521 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb88070c:::benchmark_data_smithi122_119671_object45595:head# 2024-03-20T09:05:00.521 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.521 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb881e56:::benchmark_data_smithi122_119671_object79080:head# 2024-03-20T09:05:00.521 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.521 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb88b541:::benchmark_data_smithi122_119671_object42178:head# 2024-03-20T09:05:00.521 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.522 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb88cba7:::benchmark_data_smithi122_119671_object17221:head# 2024-03-20T09:05:00.522 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.522 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb8aa04b:::benchmark_data_smithi122_119671_object86165:head# 2024-03-20T09:05:00.522 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.522 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb8b1db2:::benchmark_data_smithi122_119671_object57745:head# 2024-03-20T09:05:00.522 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.522 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb8ba9ce:::benchmark_data_smithi122_119671_object12846:head# 2024-03-20T09:05:00.522 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.522 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb8bbfd5:::benchmark_data_smithi122_119671_object77221:head# 2024-03-20T09:05:00.522 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.522 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb8beedd:::benchmark_data_smithi122_119671_object25564:head# 2024-03-20T09:05:00.522 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.523 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb8d69ec:::benchmark_data_smithi122_119671_object78229:head# 2024-03-20T09:05:00.523 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.523 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb8f6296:::benchmark_data_smithi122_119671_object48130:head# 2024-03-20T09:05:00.523 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.523 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb90624a:::benchmark_data_smithi122_119671_object62912:head# 2024-03-20T09:05:00.523 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.523 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb907b77:::benchmark_data_smithi122_119671_object83983:head# 2024-03-20T09:05:00.523 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.523 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb90ea38:::benchmark_data_smithi122_119671_object16243:head# 2024-03-20T09:05:00.523 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.523 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb912115:::benchmark_data_smithi122_119671_object88374:head# 2024-03-20T09:05:00.523 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.523 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb916492:::benchmark_data_smithi122_119671_object18679:head# 2024-03-20T09:05:00.524 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.524 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb924312:::benchmark_data_smithi122_119671_object49857:head# 2024-03-20T09:05:00.524 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.524 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb92cb41:::benchmark_data_smithi122_119671_object45019:head# 2024-03-20T09:05:00.524 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.524 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb9300a5:::benchmark_data_smithi122_119671_object47308:head# 2024-03-20T09:05:00.524 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.524 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb94194c:::benchmark_data_smithi122_119671_object41477:head# 2024-03-20T09:05:00.524 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.524 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb948575:::benchmark_data_smithi122_119671_object51315:head# 2024-03-20T09:05:00.524 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.524 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb95a0be:::benchmark_data_smithi122_119671_object86335:head# 2024-03-20T09:05:00.525 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.525 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb95a8c0:::benchmark_data_smithi122_119671_object38623:head# 2024-03-20T09:05:00.525 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.525 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb95c76f:::benchmark_data_smithi122_119671_object47467:head# 2024-03-20T09:05:00.525 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.525 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb963745:::benchmark_data_smithi122_119671_object63861:head# 2024-03-20T09:05:00.525 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.525 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb96e0e6:::benchmark_data_smithi122_119671_object83795:head# 2024-03-20T09:05:00.525 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.525 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb971461:::benchmark_data_smithi122_119671_object81704:head# 2024-03-20T09:05:00.525 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.525 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb97d2bf:::benchmark_data_smithi122_119671_object51780:head# 2024-03-20T09:05:00.525 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.525 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb97e43d:::benchmark_data_smithi122_119671_object38654:head# 2024-03-20T09:05:00.526 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.526 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb98a5c8:::benchmark_data_smithi122_119671_object89820:head# 2024-03-20T09:05:00.526 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.526 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb99a472:::benchmark_data_smithi122_119671_object12707:head# 2024-03-20T09:05:00.526 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.526 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb9a4862:::benchmark_data_smithi122_119671_object21448:head# 2024-03-20T09:05:00.526 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.526 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb9a79ca:::benchmark_data_smithi122_119671_object52150:head# 2024-03-20T09:05:00.526 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.526 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb9ae458:::benchmark_data_smithi122_119671_object32735:head# 2024-03-20T09:05:00.526 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.526 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb9b688f:::benchmark_data_smithi122_119671_object26784:head# 2024-03-20T09:05:00.526 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.527 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb9b6c0f:::benchmark_data_smithi122_119671_object89704:head# 2024-03-20T09:05:00.527 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.527 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb9c42f7:::benchmark_data_smithi122_119671_object91615:head# 2024-03-20T09:05:00.527 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.527 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb9c6044:::benchmark_data_smithi122_119671_object34812:head# 2024-03-20T09:05:00.527 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.527 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb9cc740:::benchmark_data_smithi122_119671_object11015:head# 2024-03-20T09:05:00.527 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.527 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb9cf26c:::benchmark_data_smithi122_119671_object74694:head# 2024-03-20T09:05:00.527 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.527 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb9d73ad:::benchmark_data_smithi122_119671_object64809:head# 2024-03-20T09:05:00.527 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.527 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb9db120:::benchmark_data_smithi122_119671_object85663:head# 2024-03-20T09:05:00.527 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.528 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb9e0398:::benchmark_data_smithi122_119671_object23105:head# 2024-03-20T09:05:00.528 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.528 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb9e1538:::benchmark_data_smithi122_119671_object44957:head# 2024-03-20T09:05:00.528 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.528 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb9f28c5:::benchmark_data_smithi122_119671_object45921:head# 2024-03-20T09:05:00.528 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.528 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb9f956b:::benchmark_data_smithi122_119671_object81116:head# 2024-03-20T09:05:00.528 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.528 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bb9f9f7b:::benchmark_data_smithi122_119671_object72522:head# 2024-03-20T09:05:00.528 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.528 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bba09424:::benchmark_data_smithi122_119671_object46355:head# 2024-03-20T09:05:00.528 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.528 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bba1c657:::benchmark_data_smithi122_119671_object88850:head# 2024-03-20T09:05:00.529 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.529 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bba23bae:::benchmark_data_smithi122_119671_object78291:head# 2024-03-20T09:05:00.529 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.529 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bba3848b:::benchmark_data_smithi122_119671_object46930:head# 2024-03-20T09:05:00.529 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.529 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bba40123:::benchmark_data_smithi122_119671_object43054:head# 2024-03-20T09:05:00.529 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.529 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bba6ae3c:::benchmark_data_smithi122_119671_object92258:head# 2024-03-20T09:05:00.529 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.529 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bba6af21:::benchmark_data_smithi122_119671_object2543:head# 2024-03-20T09:05:00.529 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.529 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bba6b7c0:::benchmark_data_smithi122_119671_object1586:head# 2024-03-20T09:05:00.529 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.529 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bba76e7d:::benchmark_data_smithi122_119671_object50630:head# 2024-03-20T09:05:00.529 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.530 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bba81831:::benchmark_data_smithi122_119671_object13189:head# 2024-03-20T09:05:00.530 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.530 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bba82f91:::benchmark_data_smithi122_119671_object16473:head# 2024-03-20T09:05:00.530 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.530 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bba87980:::benchmark_data_smithi122_119671_object32275:head# 2024-03-20T09:05:00.530 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.530 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bba8af9f:::benchmark_data_smithi122_119671_object8995:head# 2024-03-20T09:05:00.530 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.530 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bba9767e:::benchmark_data_smithi122_119671_object15267:head# 2024-03-20T09:05:00.530 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.530 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbaa016f:::benchmark_data_smithi122_119671_object64991:head# 2024-03-20T09:05:00.530 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.530 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbaaac80:::benchmark_data_smithi122_119671_object47458:head# 2024-03-20T09:05:00.531 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.531 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbab2739:::benchmark_data_smithi122_119671_object96942:head# 2024-03-20T09:05:00.531 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.531 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbab2812:::benchmark_data_smithi122_119671_object19869:head# 2024-03-20T09:05:00.531 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.531 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbab8b29:::benchmark_data_smithi122_119671_object7345:head# 2024-03-20T09:05:00.531 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.531 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbabec99:::benchmark_data_smithi122_119671_object46718:head# 2024-03-20T09:05:00.531 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.531 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbaced88:::benchmark_data_smithi122_119671_object92937:head# 2024-03-20T09:05:00.531 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.531 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbacff06:::benchmark_data_smithi122_119671_object47530:head# 2024-03-20T09:05:00.531 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.531 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbad1f1b:::benchmark_data_smithi122_119671_object4583:head# 2024-03-20T09:05:00.532 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.532 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbaefdfa:::benchmark_data_smithi122_119671_object75093:head# 2024-03-20T09:05:00.532 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.532 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbb197df:::benchmark_data_smithi122_119671_object68303:head# 2024-03-20T09:05:00.532 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.532 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbb1ace3:::benchmark_data_smithi122_119671_object72440:head# 2024-03-20T09:05:00.532 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.532 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbb1f5ec:::benchmark_data_smithi122_119671_object61707:head# 2024-03-20T09:05:00.532 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.532 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbb220c2:::benchmark_data_smithi122_119671_object59191:head# 2024-03-20T09:05:00.532 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.532 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbb26c0e:::benchmark_data_smithi122_119671_object36972:head# 2024-03-20T09:05:00.532 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.532 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbb281db:::benchmark_data_smithi122_119671_object35342:head# 2024-03-20T09:05:00.533 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.533 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbb30dd1:::benchmark_data_smithi122_119671_object59845:head# 2024-03-20T09:05:00.533 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.533 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbb3c83d:::benchmark_data_smithi122_119671_object48340:head# 2024-03-20T09:05:00.533 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.533 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbb3ff89:::benchmark_data_smithi122_119671_object81263:head# 2024-03-20T09:05:00.533 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.533 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbb4d977:::benchmark_data_smithi122_119671_object95254:head# 2024-03-20T09:05:00.533 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.533 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbb4f6ea:::benchmark_data_smithi122_119671_object63388:head# 2024-03-20T09:05:00.533 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.533 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbb50648:::benchmark_data_smithi122_119671_object95765:head# 2024-03-20T09:05:00.533 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.533 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbb531a2:::benchmark_data_smithi122_119671_object99605:head# 2024-03-20T09:05:00.533 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.534 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbb54cfb:::benchmark_data_smithi122_119671_object98819:head# 2024-03-20T09:05:00.534 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.534 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbb56e14:::benchmark_data_smithi122_119671_object8660:head# 2024-03-20T09:05:00.534 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.534 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbb58fc4:::benchmark_data_smithi122_119671_object40508:head# 2024-03-20T09:05:00.534 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.534 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbb724ec:::benchmark_data_smithi122_119671_object67740:head# 2024-03-20T09:05:00.534 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.534 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbb7df5e:::benchmark_data_smithi122_119671_object8087:head# 2024-03-20T09:05:00.534 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.534 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbb9ca3d:::benchmark_data_smithi122_119671_object67864:head# 2024-03-20T09:05:00.534 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.534 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbbbcf69:::benchmark_data_smithi122_119671_object34228:head# 2024-03-20T09:05:00.534 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.535 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbbd1413:::benchmark_data_smithi122_119671_object28071:head# 2024-03-20T09:05:00.535 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.535 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbbd6d47:::benchmark_data_smithi122_119671_object56490:head# 2024-03-20T09:05:00.535 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.535 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbbdc794:::benchmark_data_smithi122_119671_object4743:head# 2024-03-20T09:05:00.535 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.535 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbbdff6f:::benchmark_data_smithi122_119671_object20201:head# 2024-03-20T09:05:00.535 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.535 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbc0296e:::benchmark_data_smithi122_119671_object14364:head# 2024-03-20T09:05:00.535 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.535 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbc02a03:::benchmark_data_smithi122_119671_object38133:head# 2024-03-20T09:05:00.535 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.535 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbc17b28:::benchmark_data_smithi122_119671_object56151:head# 2024-03-20T09:05:00.535 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.536 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbc17fa4:::benchmark_data_smithi122_119671_object54547:head# 2024-03-20T09:05:00.536 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.536 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbc1ea6e:::benchmark_data_smithi122_119671_object59477:head# 2024-03-20T09:05:00.536 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.536 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbc497f4:::benchmark_data_smithi122_119671_object69513:head# 2024-03-20T09:05:00.536 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.536 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbc4e890:::benchmark_data_smithi122_119671_object37236:head# 2024-03-20T09:05:00.536 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.536 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbc4f54c:::benchmark_data_smithi122_119671_object12410:head# 2024-03-20T09:05:00.536 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.536 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbc64927:::benchmark_data_smithi122_119671_object83344:head# 2024-03-20T09:05:00.536 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.536 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbc9621f:::benchmark_data_smithi122_119671_object93445:head# 2024-03-20T09:05:00.536 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.537 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbca0a47:::benchmark_data_smithi122_119671_object61875:head# 2024-03-20T09:05:00.537 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.537 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbca715c:::benchmark_data_smithi122_119671_object42785:head# 2024-03-20T09:05:00.537 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.537 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbcaf927:::benchmark_data_smithi122_119671_object29140:head# 2024-03-20T09:05:00.537 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.537 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbcb1d97:::benchmark_data_smithi122_119671_object5509:head# 2024-03-20T09:05:00.537 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.537 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbcec9bd:::benchmark_data_smithi122_119671_object47596:head# 2024-03-20T09:05:00.537 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.537 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbcecdc1:::benchmark_data_smithi122_119671_object70695:head# 2024-03-20T09:05:00.537 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.537 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbced0f9:::benchmark_data_smithi122_119671_object22658:head# 2024-03-20T09:05:00.537 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.537 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbd05727:::benchmark_data_smithi122_119671_object92907:head# 2024-03-20T09:05:00.538 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.538 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbd11f75:::benchmark_data_smithi122_119671_object77793:head# 2024-03-20T09:05:00.538 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.538 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbd1dd99:::benchmark_data_smithi122_119671_object74615:head# 2024-03-20T09:05:00.538 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.538 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbd1f9d4:::benchmark_data_smithi122_119671_object7952:head# 2024-03-20T09:05:00.538 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.538 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbd1fac1:::benchmark_data_smithi122_119671_object70056:head# 2024-03-20T09:05:00.538 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.538 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbd2fa92:::benchmark_data_smithi122_119671_object74672:head# 2024-03-20T09:05:00.538 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.538 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbd3f961:::benchmark_data_smithi122_119671_object99621:head# 2024-03-20T09:05:00.538 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.538 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbd4f539:::benchmark_data_smithi122_119671_object78368:head# 2024-03-20T09:05:00.539 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.539 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbd50f5b:::benchmark_data_smithi122_119671_object60781:head# 2024-03-20T09:05:00.539 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.539 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbd53db8:::benchmark_data_smithi122_119671_object89039:head# 2024-03-20T09:05:00.539 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.539 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbd557eb:::benchmark_data_smithi122_119671_object84939:head# 2024-03-20T09:05:00.539 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.539 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbd7afb6:::benchmark_data_smithi122_119671_object23687:head# 2024-03-20T09:05:00.539 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.539 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbd82060:::benchmark_data_smithi122_119671_object97488:head# 2024-03-20T09:05:00.539 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.539 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbd883ef:::benchmark_data_smithi122_119671_object59407:head# 2024-03-20T09:05:00.539 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.539 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbd8ba89:::benchmark_data_smithi122_119671_object84058:head# 2024-03-20T09:05:00.540 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.540 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbda9da3:::benchmark_data_smithi122_119671_object53250:head# 2024-03-20T09:05:00.540 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.540 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbdaf813:::benchmark_data_smithi122_119671_object50542:head# 2024-03-20T09:05:00.540 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.540 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbdba078:::benchmark_data_smithi122_119671_object70722:head# 2024-03-20T09:05:00.540 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.540 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbdbcc23:::benchmark_data_smithi122_119671_object3900:head# 2024-03-20T09:05:00.540 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.540 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbdc05fa:::benchmark_data_smithi122_119671_object67377:head# 2024-03-20T09:05:00.540 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.540 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbdc0bc5:::benchmark_data_smithi122_119671_object5855:head# 2024-03-20T09:05:00.540 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.540 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbdce493:::benchmark_data_smithi122_119671_object90935:head# 2024-03-20T09:05:00.541 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.541 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbdd02d4:::benchmark_data_smithi122_119671_object33190:head# 2024-03-20T09:05:00.541 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.541 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbde1596:::benchmark_data_smithi122_119671_object74375:head# 2024-03-20T09:05:00.541 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.541 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbde23b7:::benchmark_data_smithi122_119671_object24047:head# 2024-03-20T09:05:00.541 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.541 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbdedf04:::benchmark_data_smithi122_119671_object29142:head# 2024-03-20T09:05:00.541 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.541 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbdf5db2:::benchmark_data_smithi122_119671_object98096:head# 2024-03-20T09:05:00.541 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.541 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbe00014:::benchmark_data_smithi122_119671_object91459:head# 2024-03-20T09:05:00.541 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.541 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbe07025:::benchmark_data_smithi122_119671_object54627:head# 2024-03-20T09:05:00.542 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.542 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbe16fb0:::benchmark_data_smithi122_119671_object32817:head# 2024-03-20T09:05:00.542 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.542 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbe18dc1:::benchmark_data_smithi122_119671_object55393:head# 2024-03-20T09:05:00.542 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.542 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbe48346:::benchmark_data_smithi122_119671_object53575:head# 2024-03-20T09:05:00.542 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.542 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbe48757:::benchmark_data_smithi122_119671_object51505:head# 2024-03-20T09:05:00.542 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.542 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbe54605:::benchmark_data_smithi122_119671_object29278:head# 2024-03-20T09:05:00.542 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.542 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbe5e018:::benchmark_data_smithi122_119671_object34784:head# 2024-03-20T09:05:00.542 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.543 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbe6420b:::benchmark_data_smithi122_119671_object39848:head# 2024-03-20T09:05:00.543 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.543 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbe6ea66:::benchmark_data_smithi122_119671_object35221:head# 2024-03-20T09:05:00.543 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.543 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbe7a60e:::benchmark_data_smithi122_119671_object1531:head# 2024-03-20T09:05:00.543 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.543 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbe84ce1:::benchmark_data_smithi122_119671_object45930:head# 2024-03-20T09:05:00.543 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.543 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbe856e5:::benchmark_data_smithi122_119671_object19089:head# 2024-03-20T09:05:00.543 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.543 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbe997d1:::benchmark_data_smithi122_119671_object74534:head# 2024-03-20T09:05:00.543 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.543 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbea3bd0:::benchmark_data_smithi122_119671_object44074:head# 2024-03-20T09:05:00.543 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.544 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbeb06b3:::benchmark_data_smithi122_119671_object22184:head# 2024-03-20T09:05:00.544 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.544 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbeb7db0:::benchmark_data_smithi122_119671_object44764:head# 2024-03-20T09:05:00.544 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.544 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbeb9d6c:::benchmark_data_smithi122_119671_object5788:head# 2024-03-20T09:05:00.544 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.544 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbec561e:::benchmark_data_smithi122_119671_object83728:head# 2024-03-20T09:05:00.544 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.544 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbeca5a8:::benchmark_data_smithi122_119671_object68147:head# 2024-03-20T09:05:00.544 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.544 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbecf716:::benchmark_data_smithi122_119671_object81805:head# 2024-03-20T09:05:00.544 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.544 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbee2f0b:::benchmark_data_smithi122_119671_object47081:head# 2024-03-20T09:05:00.544 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.545 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbee4f87:::benchmark_data_smithi122_119671_object42017:head# 2024-03-20T09:05:00.545 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.545 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbeeecef:::benchmark_data_smithi122_119671_object82540:head# 2024-03-20T09:05:00.545 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.545 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbef0c4f:::benchmark_data_smithi122_119671_object15777:head# 2024-03-20T09:05:00.545 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.545 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbef2633:::benchmark_data_smithi122_119671_object38172:head# 2024-03-20T09:05:00.545 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.545 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbef7ff5:::benchmark_data_smithi122_119671_object83134:head# 2024-03-20T09:05:00.545 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.545 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbf0ace8:::benchmark_data_smithi122_119671_object16617:head# 2024-03-20T09:05:00.545 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.545 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbf0b42d:::benchmark_data_smithi122_119671_object33869:head# 2024-03-20T09:05:00.545 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.546 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbf1d4bf:::benchmark_data_smithi122_119671_object28976:head# 2024-03-20T09:05:00.546 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.546 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbf244de:::benchmark_data_smithi122_119671_object30170:head# 2024-03-20T09:05:00.546 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.546 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbf25014:::benchmark_data_smithi122_119671_object86634:head# 2024-03-20T09:05:00.546 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.546 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbf2826d:::benchmark_data_smithi122_119671_object36917:head# 2024-03-20T09:05:00.546 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.546 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbf29edb:::benchmark_data_smithi122_119671_object97439:head# 2024-03-20T09:05:00.546 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.546 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbf2a7d5:::benchmark_data_smithi122_119671_object23746:head# 2024-03-20T09:05:00.546 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.546 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbf2c746:::benchmark_data_smithi122_119671_object53658:head# 2024-03-20T09:05:00.546 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.547 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbf2cab3:::benchmark_data_smithi122_119671_object77143:head# 2024-03-20T09:05:00.547 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.547 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbf33b7a:::benchmark_data_smithi122_119671_object59717:head# 2024-03-20T09:05:00.547 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.547 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbf33ff7:::benchmark_data_smithi122_119671_object91012:head# 2024-03-20T09:05:00.547 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.547 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbf439d1:::benchmark_data_smithi122_119671_object61938:head# 2024-03-20T09:05:00.547 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.547 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbf472aa:::benchmark_data_smithi122_119671_object47766:head# 2024-03-20T09:05:00.547 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.547 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbf58b9f:::benchmark_data_smithi122_119671_object80328:head# 2024-03-20T09:05:00.547 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.547 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbf63419:::benchmark_data_smithi122_119671_object96820:head# 2024-03-20T09:05:00.547 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.548 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbf6b7f0:::benchmark_data_smithi122_119671_object2369:head# 2024-03-20T09:05:00.548 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.548 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbf6f0c4:::benchmark_data_smithi122_119671_object83626:head# 2024-03-20T09:05:00.548 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.548 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbf7ad23:::benchmark_data_smithi122_119671_object6281:head# 2024-03-20T09:05:00.548 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.548 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbf7dd2c:::benchmark_data_smithi122_119671_object50374:head# 2024-03-20T09:05:00.548 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.548 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbf7fc96:::benchmark_data_smithi122_119671_object23227:head# 2024-03-20T09:05:00.548 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.548 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbf8251d:::benchmark_data_smithi122_119671_object2660:head# 2024-03-20T09:05:00.548 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.548 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbf9101b:::benchmark_data_smithi122_119671_object70014:head# 2024-03-20T09:05:00.549 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.549 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbfb9acf:::benchmark_data_smithi122_119671_object47485:head# 2024-03-20T09:05:00.549 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.549 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbfba9ad:::benchmark_data_smithi122_119671_object98359:head# 2024-03-20T09:05:00.549 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.549 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbfd62ee:::benchmark_data_smithi122_119671_object73041:head# 2024-03-20T09:05:00.549 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.549 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbfdb270:::benchmark_data_smithi122_119671_object85642:head# 2024-03-20T09:05:00.549 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.549 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbfe2c57:::benchmark_data_smithi122_119671_object40696:head# 2024-03-20T09:05:00.549 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.549 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbfe74eb:::benchmark_data_smithi122_119671_object55154:head# 2024-03-20T09:05:00.549 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.549 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbff68c4:::benchmark_data_smithi122_119671_object4247:head# 2024-03-20T09:05:00.550 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.550 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bbffcd01:::benchmark_data_smithi122_119671_object81817:head# 2024-03-20T09:05:00.550 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.550 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc000983:::benchmark_data_smithi122_119671_object78254:head# 2024-03-20T09:05:00.550 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.550 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc0026ba:::benchmark_data_smithi122_119671_object60930:head# 2024-03-20T09:05:00.550 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.550 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc004012:::benchmark_data_smithi122_119671_object20466:head# 2024-03-20T09:05:00.550 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.550 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc007a2d:::benchmark_data_smithi122_119671_object89716:head# 2024-03-20T09:05:00.550 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.550 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc01d290:::benchmark_data_smithi122_119671_object27926:head# 2024-03-20T09:05:00.550 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.551 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc01e906:::benchmark_data_smithi122_119671_object54780:head# 2024-03-20T09:05:00.551 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.551 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc020ceb:::benchmark_data_smithi122_119671_object90919:head# 2024-03-20T09:05:00.551 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.551 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc0243cb:::benchmark_data_smithi122_119671_object76739:head# 2024-03-20T09:05:00.551 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.551 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc046001:::benchmark_data_smithi122_119671_object75134:head# 2024-03-20T09:05:00.551 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.551 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc048331:::benchmark_data_smithi122_119671_object86935:head# 2024-03-20T09:05:00.551 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.551 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc04addd:::benchmark_data_smithi122_119671_object25978:head# 2024-03-20T09:05:00.551 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.551 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc0618a0:::benchmark_data_smithi122_119671_object82134:head# 2024-03-20T09:05:00.552 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.552 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc064f83:::benchmark_data_smithi122_119671_object18289:head# 2024-03-20T09:05:00.552 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.552 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc0692f4:::benchmark_data_smithi122_119671_object2410:head# 2024-03-20T09:05:00.552 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.552 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc0698fa:::benchmark_data_smithi122_119671_object83058:head# 2024-03-20T09:05:00.552 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.552 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc06a2a8:::benchmark_data_smithi122_119671_object5124:head# 2024-03-20T09:05:00.552 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.552 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc0707cc:::benchmark_data_smithi122_119671_object39488:head# 2024-03-20T09:05:00.552 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.552 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc086f54:::benchmark_data_smithi122_119671_object41014:head# 2024-03-20T09:05:00.552 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.552 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc09e473:::benchmark_data_smithi122_119671_object12531:head# 2024-03-20T09:05:00.553 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.553 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc0a9fa5:::benchmark_data_smithi122_119671_object18215:head# 2024-03-20T09:05:00.553 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.553 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc0afcca:::benchmark_data_smithi122_119671_object19114:head# 2024-03-20T09:05:00.553 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.553 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc0c4839:::benchmark_data_smithi122_119671_object82225:head# 2024-03-20T09:05:00.553 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.553 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc0c74a3:::benchmark_data_smithi122_119671_object54126:head# 2024-03-20T09:05:00.553 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.553 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc0dfe69:::benchmark_data_smithi122_119671_object60718:head# 2024-03-20T09:05:00.553 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.553 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc0e0595:::benchmark_data_smithi122_119671_object27862:head# 2024-03-20T09:05:00.553 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.553 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc0e998a:::benchmark_data_smithi122_119671_object35444:head# 2024-03-20T09:05:00.554 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.554 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc0ee886:::benchmark_data_smithi122_119671_object27277:head# 2024-03-20T09:05:00.554 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.554 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc0f1db8:::benchmark_data_smithi122_119671_object52327:head# 2024-03-20T09:05:00.554 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.554 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc0f2147:::benchmark_data_smithi122_119671_object44135:head# 2024-03-20T09:05:00.554 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.554 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc10692f:::benchmark_data_smithi122_119671_object64108:head# 2024-03-20T09:05:00.554 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.554 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc10eeb2:::benchmark_data_smithi122_119671_object27546:head# 2024-03-20T09:05:00.554 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.554 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc1136e4:::benchmark_data_smithi122_119671_object13402:head# 2024-03-20T09:05:00.554 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.555 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc115849:::benchmark_data_smithi122_119671_object92395:head# 2024-03-20T09:05:00.555 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.555 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc12c424:::benchmark_data_smithi122_119671_object84928:head# 2024-03-20T09:05:00.555 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.555 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc1363a1:::benchmark_data_smithi122_119671_object84566:head# 2024-03-20T09:05:00.555 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.555 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc1381cf:::benchmark_data_smithi122_119671_object90763:head# 2024-03-20T09:05:00.555 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.555 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc14a0ad:::benchmark_data_smithi122_119671_object62100:head# 2024-03-20T09:05:00.555 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.555 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc15778b:::benchmark_data_smithi122_119671_object49971:head# 2024-03-20T09:05:00.555 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.555 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc178aad:::benchmark_data_smithi122_119671_object74041:head# 2024-03-20T09:05:00.556 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.556 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc17dff1:::benchmark_data_smithi122_119671_object99427:head# 2024-03-20T09:05:00.556 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.556 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc1846cf:::benchmark_data_smithi122_119671_object40681:head# 2024-03-20T09:05:00.556 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.556 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc184df7:::benchmark_data_smithi122_119671_object90023:head# 2024-03-20T09:05:00.556 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.556 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc191807:::benchmark_data_smithi122_119671_object83823:head# 2024-03-20T09:05:00.556 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.556 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc19466a:::benchmark_data_smithi122_119671_object62092:head# 2024-03-20T09:05:00.556 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.556 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc19dc2b:::benchmark_data_smithi122_119671_object76700:head# 2024-03-20T09:05:00.557 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.557 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc1ada60:::benchmark_data_smithi122_119671_object16985:head# 2024-03-20T09:05:00.557 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.557 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc1c5535:::benchmark_data_smithi122_119671_object44609:head# 2024-03-20T09:05:00.557 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.557 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc1c6ef2:::benchmark_data_smithi122_119671_object47943:head# 2024-03-20T09:05:00.557 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.557 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc1c9f8e:::benchmark_data_smithi122_119671_object19140:head# 2024-03-20T09:05:00.557 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.557 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc1d10a1:::benchmark_data_smithi122_119671_object49518:head# 2024-03-20T09:05:00.557 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.557 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc1d4b97:::benchmark_data_smithi122_119671_object85749:head# 2024-03-20T09:05:00.557 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.558 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc1e0d78:::benchmark_data_smithi122_119671_object49373:head# 2024-03-20T09:05:00.558 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.558 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc1e4609:::benchmark_data_smithi122_119671_object52604:head# 2024-03-20T09:05:00.558 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.558 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc1f2758:::benchmark_data_smithi122_119671_object97497:head# 2024-03-20T09:05:00.558 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.558 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc21231e:::benchmark_data_smithi122_119671_object86975:head# 2024-03-20T09:05:00.558 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.558 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc22b0a4:::benchmark_data_smithi122_119671_object83799:head# 2024-03-20T09:05:00.558 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.558 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc22dc98:::benchmark_data_smithi122_119671_object74107:head# 2024-03-20T09:05:00.558 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.558 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc24b940:::benchmark_data_smithi122_119671_object61490:head# 2024-03-20T09:05:00.559 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.559 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc259e66:::benchmark_data_smithi122_119671_object90298:head# 2024-03-20T09:05:00.559 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.559 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc25eba0:::benchmark_data_smithi122_119671_object79892:head# 2024-03-20T09:05:00.559 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.559 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc26106e:::benchmark_data_smithi122_119671_object80284:head# 2024-03-20T09:05:00.559 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.559 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc26645c:::benchmark_data_smithi122_119671_object3671:head# 2024-03-20T09:05:00.559 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.559 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc27c863:::benchmark_data_smithi122_119671_object69991:head# 2024-03-20T09:05:00.559 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.559 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc27cb18:::benchmark_data_smithi122_119671_object77762:head# 2024-03-20T09:05:00.559 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.560 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc28c877:::benchmark_data_smithi122_119671_object2766:head# 2024-03-20T09:05:00.560 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.560 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc2dd1b6:::benchmark_data_smithi122_119671_object75814:head# 2024-03-20T09:05:00.560 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.560 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc2ebbbe:::benchmark_data_smithi122_119671_object23953:head# 2024-03-20T09:05:00.560 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.560 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc2ec479:::benchmark_data_smithi122_119671_object46022:head# 2024-03-20T09:05:00.560 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.560 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc2f378f:::benchmark_data_smithi122_119671_object62258:head# 2024-03-20T09:05:00.560 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.560 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc302f45:::benchmark_data_smithi122_119671_object13624:head# 2024-03-20T09:05:00.560 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.560 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc30cefb:::benchmark_data_smithi122_119671_object22722:head# 2024-03-20T09:05:00.561 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.561 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc31d3a7:::benchmark_data_smithi122_119671_object46837:head# 2024-03-20T09:05:00.561 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.561 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc321cdd:::benchmark_data_smithi122_119671_object14965:head# 2024-03-20T09:05:00.561 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.561 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc323401:::benchmark_data_smithi122_119671_object52656:head# 2024-03-20T09:05:00.561 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.561 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc323922:::benchmark_data_smithi122_119671_object60143:head# 2024-03-20T09:05:00.561 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.561 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc329e5e:::benchmark_data_smithi122_119671_object4375:head# 2024-03-20T09:05:00.561 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.561 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc32e9c8:::benchmark_data_smithi122_119671_object40443:head# 2024-03-20T09:05:00.561 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.561 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc3308ff:::benchmark_data_smithi122_119671_object67716:head# 2024-03-20T09:05:00.562 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.562 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc33643f:::benchmark_data_smithi122_119671_object16150:head# 2024-03-20T09:05:00.562 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.562 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc339308:::benchmark_data_smithi122_119671_object23487:head# 2024-03-20T09:05:00.562 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.562 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc34bb1f:::benchmark_data_smithi122_119671_object28316:head# 2024-03-20T09:05:00.562 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.562 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc36c27c:::benchmark_data_smithi122_119671_object69932:head# 2024-03-20T09:05:00.562 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.562 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc379935:::benchmark_data_smithi122_119671_object14929:head# 2024-03-20T09:05:00.562 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.562 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc37f178:::benchmark_data_smithi122_119671_object85113:head# 2024-03-20T09:05:00.562 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.562 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc380113:::benchmark_data_smithi122_119671_object13858:head# 2024-03-20T09:05:00.563 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.563 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc3812d1:::benchmark_data_smithi122_119671_object40013:head# 2024-03-20T09:05:00.563 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.563 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc38855e:::benchmark_data_smithi122_119671_object58980:head# 2024-03-20T09:05:00.563 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.563 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc38b093:::benchmark_data_smithi122_119671_object18087:head# 2024-03-20T09:05:00.563 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.563 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc393680:::benchmark_data_smithi122_119671_object38634:head# 2024-03-20T09:05:00.563 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.563 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc3a6e23:::benchmark_data_smithi122_119671_object28648:head# 2024-03-20T09:05:00.563 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.563 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc3a9b8e:::benchmark_data_smithi122_119671_object11404:head# 2024-03-20T09:05:00.563 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.563 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc3ab47b:::benchmark_data_smithi122_119671_object93851:head# 2024-03-20T09:05:00.564 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.564 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc3ab611:::benchmark_data_smithi122_119671_object72419:head# 2024-03-20T09:05:00.564 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.564 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc3ae039:::benchmark_data_smithi122_119671_object66202:head# 2024-03-20T09:05:00.564 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.564 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc3bc6e8:::benchmark_data_smithi122_119671_object90659:head# 2024-03-20T09:05:00.564 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.564 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc3bcb48:::benchmark_data_smithi122_119671_object42209:head# 2024-03-20T09:05:00.564 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.564 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc3be9da:::benchmark_data_smithi122_119671_object89933:head# 2024-03-20T09:05:00.564 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.564 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc3dba61:::benchmark_data_smithi122_119671_object23577:head# 2024-03-20T09:05:00.564 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.564 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc3dd318:::benchmark_data_smithi122_119671_object73357:head# 2024-03-20T09:05:00.565 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.565 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc3e1f66:::benchmark_data_smithi122_119671_object49836:head# 2024-03-20T09:05:00.565 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.565 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc3f7239:::benchmark_data_smithi122_119671_object64726:head# 2024-03-20T09:05:00.565 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.565 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc403131:::benchmark_data_smithi122_119671_object45104:head# 2024-03-20T09:05:00.565 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.565 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc4092c5:::benchmark_data_smithi122_119671_object18658:head# 2024-03-20T09:05:00.565 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.565 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc40df23:::benchmark_data_smithi122_119671_object41620:head# 2024-03-20T09:05:00.565 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.565 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc414500:::benchmark_data_smithi122_119671_object19012:head# 2024-03-20T09:05:00.565 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.565 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc428a2d:::benchmark_data_smithi122_119671_object86399:head# 2024-03-20T09:05:00.566 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.566 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc43077e:::benchmark_data_smithi122_119671_object30455:head# 2024-03-20T09:05:00.566 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.566 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc433f18:::benchmark_data_smithi122_119671_object99205:head# 2024-03-20T09:05:00.566 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.566 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc44fb02:::benchmark_data_smithi122_119671_object90679:head# 2024-03-20T09:05:00.566 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.566 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc451ea0:::benchmark_data_smithi122_119671_object51124:head# 2024-03-20T09:05:00.566 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.566 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc452f6a:::benchmark_data_smithi122_119671_object86196:head# 2024-03-20T09:05:00.566 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.566 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc45ecf9:::benchmark_data_smithi122_119671_object88826:head# 2024-03-20T09:05:00.566 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.566 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc460102:::benchmark_data_smithi122_119671_object28805:head# 2024-03-20T09:05:00.567 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.567 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc461174:::benchmark_data_smithi122_119671_object76634:head# 2024-03-20T09:05:00.567 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.567 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc47be4d:::benchmark_data_smithi122_119671_object7530:head# 2024-03-20T09:05:00.567 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.567 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc4981ba:::benchmark_data_smithi122_119671_object21842:head# 2024-03-20T09:05:00.567 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.567 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc49ebb8:::benchmark_data_smithi122_119671_object64486:head# 2024-03-20T09:05:00.567 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.567 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc4a6262:::benchmark_data_smithi122_119671_object92343:head# 2024-03-20T09:05:00.567 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.567 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc4ad7ca:::benchmark_data_smithi122_119671_object35268:head# 2024-03-20T09:05:00.567 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.567 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc4b3981:::benchmark_data_smithi122_119671_object78794:head# 2024-03-20T09:05:00.568 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.568 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc4ca755:::benchmark_data_smithi122_119671_object27664:head# 2024-03-20T09:05:00.568 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.568 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc4df9b6:::benchmark_data_smithi122_119671_object40912:head# 2024-03-20T09:05:00.568 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.568 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc4eae99:::benchmark_data_smithi122_119671_object67480:head# 2024-03-20T09:05:00.568 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.568 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc4f1347:::benchmark_data_smithi122_119671_object42361:head# 2024-03-20T09:05:00.568 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.568 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc4fd2f0:::benchmark_data_smithi122_119671_object39718:head# 2024-03-20T09:05:00.568 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.568 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc5281bc:::benchmark_data_smithi122_119671_object95382:head# 2024-03-20T09:05:00.568 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.568 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc52c164:::benchmark_data_smithi122_119671_object53604:head# 2024-03-20T09:05:00.569 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.569 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc55d723:::benchmark_data_smithi122_119671_object9762:head# 2024-03-20T09:05:00.569 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.569 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc560e4e:::benchmark_data_smithi122_119671_object67948:head# 2024-03-20T09:05:00.569 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.569 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc561be9:::benchmark_data_smithi122_119671_object92705:head# 2024-03-20T09:05:00.569 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.569 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc57b2d4:::benchmark_data_smithi122_119671_object30330:head# 2024-03-20T09:05:00.569 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.569 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc57d3d8:::benchmark_data_smithi122_119671_object54419:head# 2024-03-20T09:05:00.569 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.569 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc598a06:::benchmark_data_smithi122_119671_object15912:head# 2024-03-20T09:05:00.569 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.570 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc59deaa:::benchmark_data_smithi122_119671_object7036:head# 2024-03-20T09:05:00.570 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.570 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc5a1434:::benchmark_data_smithi122_119671_object73939:head# 2024-03-20T09:05:00.570 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.570 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc5a8842:::benchmark_data_smithi122_119671_object58625:head# 2024-03-20T09:05:00.570 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.570 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc5ca43b:::benchmark_data_smithi122_119671_object33341:head# 2024-03-20T09:05:00.570 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.570 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc5cb58b:::benchmark_data_smithi122_119671_object52675:head# 2024-03-20T09:05:00.570 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.570 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc5cc822:::benchmark_data_smithi122_119671_object69502:head# 2024-03-20T09:05:00.570 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.570 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc5d37ff:::benchmark_data_smithi122_119671_object75899:head# 2024-03-20T09:05:00.570 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.571 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc5d744e:::benchmark_data_smithi122_119671_object57551:head# 2024-03-20T09:05:00.571 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.571 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc5d919c:::benchmark_data_smithi122_119671_object27135:head# 2024-03-20T09:05:00.571 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.571 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc5da734:::benchmark_data_smithi122_119671_object55485:head# 2024-03-20T09:05:00.571 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.571 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc5dca20:::benchmark_data_smithi122_119671_object29471:head# 2024-03-20T09:05:00.571 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.571 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc5e6efc:::benchmark_data_smithi122_119671_object83922:head# 2024-03-20T09:05:00.571 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.571 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc5eca10:::benchmark_data_smithi122_119671_object76874:head# 2024-03-20T09:05:00.571 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.571 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc61d2cc:::benchmark_data_smithi122_119671_object31846:head# 2024-03-20T09:05:00.571 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.571 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc63df75:::benchmark_data_smithi122_119671_object33353:head# 2024-03-20T09:05:00.572 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.572 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc65aad7:::benchmark_data_smithi122_119671_object64053:head# 2024-03-20T09:05:00.572 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.572 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc65b185:::benchmark_data_smithi122_119671_object97193:head# 2024-03-20T09:05:00.572 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.572 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc65b428:::benchmark_data_smithi122_119671_object40486:head# 2024-03-20T09:05:00.572 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.572 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc65d158:::benchmark_data_smithi122_119671_object36273:head# 2024-03-20T09:05:00.572 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.572 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc663715:::benchmark_data_smithi122_119671_object23743:head# 2024-03-20T09:05:00.572 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.572 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc688e77:::benchmark_data_smithi122_119671_object44488:head# 2024-03-20T09:05:00.572 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.572 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc68d32e:::benchmark_data_smithi122_119671_object20066:head# 2024-03-20T09:05:00.573 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.573 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc694a73:::benchmark_data_smithi122_119671_object9351:head# 2024-03-20T09:05:00.573 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.573 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc696e81:::benchmark_data_smithi122_119671_object36186:head# 2024-03-20T09:05:00.573 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.573 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc699428:::benchmark_data_smithi122_119671_object85298:head# 2024-03-20T09:05:00.573 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.573 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc6a015a:::benchmark_data_smithi122_119671_object51792:head# 2024-03-20T09:05:00.573 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.573 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc6bda50:::benchmark_data_smithi122_119671_object99007:head# 2024-03-20T09:05:00.573 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.573 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc6c3b67:::benchmark_data_smithi122_119671_object37401:head# 2024-03-20T09:05:00.573 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.573 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc6cfa5d:::benchmark_data_smithi122_119671_object29583:head# 2024-03-20T09:05:00.574 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.574 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc6d8e47:::benchmark_data_smithi122_119671_object65770:head# 2024-03-20T09:05:00.574 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.574 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc6db57b:::benchmark_data_smithi122_119671_object97124:head# 2024-03-20T09:05:00.574 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.574 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc6e5a03:::benchmark_data_smithi122_119671_object29825:head# 2024-03-20T09:05:00.574 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.574 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc6fd9af:::benchmark_data_smithi122_119671_object77389:head# 2024-03-20T09:05:00.574 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.574 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc6ffe66:::benchmark_data_smithi122_119671_object23032:head# 2024-03-20T09:05:00.574 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.574 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc7019f1:::benchmark_data_smithi122_119671_object32507:head# 2024-03-20T09:05:00.574 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.575 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc709a20:::benchmark_data_smithi122_119671_object29717:head# 2024-03-20T09:05:00.575 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.575 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc74e155:::benchmark_data_smithi122_119671_object86108:head# 2024-03-20T09:05:00.575 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.575 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc760a8a:::benchmark_data_smithi122_119671_object71683:head# 2024-03-20T09:05:00.575 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.575 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc76cb94:::benchmark_data_smithi122_119671_object8845:head# 2024-03-20T09:05:00.575 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.575 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc7822f3:::benchmark_data_smithi122_119671_object53302:head# 2024-03-20T09:05:00.575 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.575 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc78b4a4:::benchmark_data_smithi122_119671_object32376:head# 2024-03-20T09:05:00.575 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.575 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc798812:::benchmark_data_smithi122_119671_object82363:head# 2024-03-20T09:05:00.575 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.576 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc79c4cf:::benchmark_data_smithi122_119671_object74612:head# 2024-03-20T09:05:00.576 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.576 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc79e003:::benchmark_data_smithi122_119671_object26752:head# 2024-03-20T09:05:00.576 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.576 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc7dc511:::benchmark_data_smithi122_119671_object30863:head# 2024-03-20T09:05:00.576 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.576 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc7e1632:::benchmark_data_smithi122_119671_object64841:head# 2024-03-20T09:05:00.576 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.576 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc7e8e5a:::benchmark_data_smithi122_119671_object13074:head# 2024-03-20T09:05:00.576 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.576 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc7eb84d:::benchmark_data_smithi122_119671_object8663:head# 2024-03-20T09:05:00.576 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.576 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc7f05e6:::benchmark_data_smithi122_119671_object55177:head# 2024-03-20T09:05:00.576 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.577 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc7f2235:::benchmark_data_smithi122_119671_object68352:head# 2024-03-20T09:05:00.577 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.577 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc8160bf:::benchmark_data_smithi122_119671_object22796:head# 2024-03-20T09:05:00.577 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.577 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc81f87f:::benchmark_data_smithi122_119671_object12446:head# 2024-03-20T09:05:00.577 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.577 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc823894:::benchmark_data_smithi122_119671_object28536:head# 2024-03-20T09:05:00.577 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.577 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc82b08c:::benchmark_data_smithi122_119671_object55755:head# 2024-03-20T09:05:00.577 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.577 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc82fa66:::benchmark_data_smithi122_119671_object97761:head# 2024-03-20T09:05:00.577 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.577 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc835247:::benchmark_data_smithi122_119671_object17058:head# 2024-03-20T09:05:00.577 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.578 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc83568e:::benchmark_data_smithi122_119671_object89772:head# 2024-03-20T09:05:00.578 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.578 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc84bf38:::benchmark_data_smithi122_119671_object3942:head# 2024-03-20T09:05:00.578 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.578 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc872459:::benchmark_data_smithi122_119671_object51139:head# 2024-03-20T09:05:00.578 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.578 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc87dd55:::benchmark_data_smithi122_119671_object68218:head# 2024-03-20T09:05:00.578 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.578 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc87dff5:::benchmark_data_smithi122_119671_object72282:head# 2024-03-20T09:05:00.578 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.578 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc881225:::benchmark_data_smithi122_119671_object13904:head# 2024-03-20T09:05:00.578 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.578 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc892e5c:::benchmark_data_smithi122_119671_object95078:head# 2024-03-20T09:05:00.578 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.579 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc89e3e5:::benchmark_data_smithi122_119671_object43583:head# 2024-03-20T09:05:00.579 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.579 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc8a64bd:::benchmark_data_smithi122_119671_object64724:head# 2024-03-20T09:05:00.579 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.579 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc8b8f9a:::benchmark_data_smithi122_119671_object49933:head# 2024-03-20T09:05:00.579 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.579 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc8bf321:::benchmark_data_smithi122_119671_object11942:head# 2024-03-20T09:05:00.579 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.579 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc8c63b8:::benchmark_data_smithi122_119671_object37886:head# 2024-03-20T09:05:00.579 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.579 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc8c9c44:::benchmark_data_smithi122_119671_object18181:head# 2024-03-20T09:05:00.579 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.579 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc8d1d02:::benchmark_data_smithi122_119671_object83831:head# 2024-03-20T09:05:00.579 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.580 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc8dc6dd:::benchmark_data_smithi122_119671_object48985:head# 2024-03-20T09:05:00.580 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.580 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc8df157:::benchmark_data_smithi122_119671_object84002:head# 2024-03-20T09:05:00.580 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.580 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc8eafd1:::benchmark_data_smithi122_119671_object88565:head# 2024-03-20T09:05:00.580 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.580 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc8f6bca:::benchmark_data_smithi122_119671_object69348:head# 2024-03-20T09:05:00.580 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.580 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc90890e:::benchmark_data_smithi122_119671_object42330:head# 2024-03-20T09:05:00.580 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.580 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc909d7a:::benchmark_data_smithi122_119671_object40492:head# 2024-03-20T09:05:00.580 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.580 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc90d132:::benchmark_data_smithi122_119671_object4205:head# 2024-03-20T09:05:00.580 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.581 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc912332:::benchmark_data_smithi122_119671_object17587:head# 2024-03-20T09:05:00.581 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.581 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc913575:::benchmark_data_smithi122_119671_object52356:head# 2024-03-20T09:05:00.581 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.581 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc915809:::benchmark_data_smithi122_119671_object82953:head# 2024-03-20T09:05:00.581 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.581 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc9174d6:::benchmark_data_smithi122_119671_object79760:head# 2024-03-20T09:05:00.581 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.581 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc9188d2:::benchmark_data_smithi122_119671_object66669:head# 2024-03-20T09:05:00.581 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.581 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc91fb79:::benchmark_data_smithi122_119671_object61622:head# 2024-03-20T09:05:00.581 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.581 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc92319b:::benchmark_data_smithi122_119671_object10301:head# 2024-03-20T09:05:00.581 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.582 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc931e3e:::benchmark_data_smithi122_119671_object9977:head# 2024-03-20T09:05:00.582 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.582 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc93667d:::benchmark_data_smithi122_119671_object30694:head# 2024-03-20T09:05:00.582 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.582 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc936873:::benchmark_data_smithi122_119671_object41637:head# 2024-03-20T09:05:00.582 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.582 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc937c56:::benchmark_data_smithi122_119671_object53731:head# 2024-03-20T09:05:00.582 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.582 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc93b306:::benchmark_data_smithi122_119671_object68462:head# 2024-03-20T09:05:00.582 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.582 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc940976:::benchmark_data_smithi122_119671_object28723:head# 2024-03-20T09:05:00.583 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.583 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc9481c3:::benchmark_data_smithi122_119671_object65136:head# 2024-03-20T09:05:00.583 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.583 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc95de1f:::benchmark_data_smithi122_119671_object50065:head# 2024-03-20T09:05:00.583 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.583 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc96cecb:::benchmark_data_smithi122_119671_object81103:head# 2024-03-20T09:05:00.583 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.583 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc9996a7:::benchmark_data_smithi122_119671_object74511:head# 2024-03-20T09:05:00.583 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.583 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc99b5e4:::benchmark_data_smithi122_119671_object80712:head# 2024-03-20T09:05:00.583 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.583 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc99d07c:::benchmark_data_smithi122_119671_object7903:head# 2024-03-20T09:05:00.583 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.583 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc9a01f1:::benchmark_data_smithi122_119671_object90989:head# 2024-03-20T09:05:00.583 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.584 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc9b0295:::benchmark_data_smithi122_119671_object23912:head# 2024-03-20T09:05:00.584 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.584 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc9b68e2:::benchmark_data_smithi122_119671_object57392:head# 2024-03-20T09:05:00.584 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.584 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc9c4608:::benchmark_data_smithi122_119671_object30217:head# 2024-03-20T09:05:00.584 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.584 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc9e1933:::benchmark_data_smithi122_119671_object40915:head# 2024-03-20T09:05:00.584 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.584 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc9ee484:::benchmark_data_smithi122_119671_object60264:head# 2024-03-20T09:05:00.584 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.584 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bc9f2029:::benchmark_data_smithi122_119671_object72135:head# 2024-03-20T09:05:00.584 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.584 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bca0266e:::benchmark_data_smithi122_119671_object13129:head# 2024-03-20T09:05:00.584 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.585 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bca07155:::benchmark_data_smithi122_119671_object47040:head# 2024-03-20T09:05:00.585 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.585 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bca14ab0:::benchmark_data_smithi122_119671_object27877:head# 2024-03-20T09:05:00.585 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.585 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bca1b015:::benchmark_data_smithi122_119671_object43204:head# 2024-03-20T09:05:00.585 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.585 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bca3537f:::benchmark_data_smithi122_119671_object75436:head# 2024-03-20T09:05:00.585 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.585 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bca4a292:::benchmark_data_smithi122_119671_object69181:head# 2024-03-20T09:05:00.585 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.585 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bca4a78c:::benchmark_data_smithi122_119671_object42240:head# 2024-03-20T09:05:00.585 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.585 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bca55329:::benchmark_data_smithi122_119671_object13439:head# 2024-03-20T09:05:00.586 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.586 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bca55e75:::benchmark_data_smithi122_119671_object54046:head# 2024-03-20T09:05:00.586 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.586 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bca61629:::benchmark_data_smithi122_119671_object29344:head# 2024-03-20T09:05:00.586 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.586 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bca61a97:::benchmark_data_smithi122_119671_object72592:head# 2024-03-20T09:05:00.586 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.586 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bca62b66:::benchmark_data_smithi122_119671_object66878:head# 2024-03-20T09:05:00.586 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.586 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bca65e81:::benchmark_data_smithi122_119671_object73624:head# 2024-03-20T09:05:00.586 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.586 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bca7b70f:::benchmark_data_smithi122_119671_object63245:head# 2024-03-20T09:05:00.586 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.586 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bca8fe7c:::benchmark_data_smithi122_119671_object26760:head# 2024-03-20T09:05:00.587 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.587 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bca9cd4f:::benchmark_data_smithi122_119671_object49068:head# 2024-03-20T09:05:00.587 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.587 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcaadeab:::benchmark_data_smithi122_119671_object78674:head# 2024-03-20T09:05:00.587 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.587 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcab5d2a:::benchmark_data_smithi122_119671_object6481:head# 2024-03-20T09:05:00.587 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.587 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcad409e:::benchmark_data_smithi122_119671_object32814:head# 2024-03-20T09:05:00.587 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.587 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcaf93d4:::benchmark_data_smithi122_119671_object11320:head# 2024-03-20T09:05:00.587 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.587 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcaf9b3f:::benchmark_data_smithi122_119671_object41517:head# 2024-03-20T09:05:00.587 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.587 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcb00b48:::benchmark_data_smithi122_119671_object74307:head# 2024-03-20T09:05:00.588 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.588 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcb1f794:::benchmark_data_smithi122_119671_object75487:head# 2024-03-20T09:05:00.588 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.588 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcb30cb9:::benchmark_data_smithi122_119671_object59088:head# 2024-03-20T09:05:00.588 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.588 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcb5b610:::benchmark_data_smithi122_119671_object87699:head# 2024-03-20T09:05:00.589 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.589 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcb5bd62:::benchmark_data_smithi122_119671_object95230:head# 2024-03-20T09:05:00.589 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.589 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcb607d6:::benchmark_data_smithi122_119671_object9191:head# 2024-03-20T09:05:00.589 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.589 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcb65d9f:::benchmark_data_smithi122_119671_object48450:head# 2024-03-20T09:05:00.589 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.589 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcb6eeef:::benchmark_data_smithi122_119671_object36183:head# 2024-03-20T09:05:00.589 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.589 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcb77b7c:::benchmark_data_smithi122_119671_object1686:head# 2024-03-20T09:05:00.589 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.589 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcb8095f:::benchmark_data_smithi122_119671_object54718:head# 2024-03-20T09:05:00.589 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.589 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcb851dd:::benchmark_data_smithi122_119671_object19988:head# 2024-03-20T09:05:00.590 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.590 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcb8b4d2:::benchmark_data_smithi122_119671_object1551:head# 2024-03-20T09:05:00.590 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.590 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcb8faf8:::benchmark_data_smithi122_119671_object78582:head# 2024-03-20T09:05:00.590 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.590 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcb962fe:::benchmark_data_smithi122_119671_object79203:head# 2024-03-20T09:05:00.590 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.590 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcb9d418:::benchmark_data_smithi122_119671_object27236:head# 2024-03-20T09:05:00.590 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.590 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcba2355:::benchmark_data_smithi122_119671_object70144:head# 2024-03-20T09:05:00.590 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.590 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcbab49d:::benchmark_data_smithi122_119671_object55587:head# 2024-03-20T09:05:00.590 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.591 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcbb197d:::benchmark_data_smithi122_119671_object95239:head# 2024-03-20T09:05:00.591 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.591 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcbbf3b3:::benchmark_data_smithi122_119671_object97915:head# 2024-03-20T09:05:00.591 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.591 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcbe192f:::benchmark_data_smithi122_119671_object82368:head# 2024-03-20T09:05:00.591 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.591 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcbf11f2:::benchmark_data_smithi122_119671_object62609:head# 2024-03-20T09:05:00.591 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.591 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcbf58aa:::benchmark_data_smithi122_119671_object74064:head# 2024-03-20T09:05:00.591 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.591 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcbf9cd8:::benchmark_data_smithi122_119671_object56259:head# 2024-03-20T09:05:00.591 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.591 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcc0deec:::benchmark_data_smithi122_119671_object38348:head# 2024-03-20T09:05:00.591 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.592 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcc274b8:::benchmark_data_smithi122_119671_object12854:head# 2024-03-20T09:05:00.592 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.592 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcc3a4b5:::benchmark_data_smithi122_119671_object44218:head# 2024-03-20T09:05:00.592 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.592 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcc3ca2e:::benchmark_data_smithi122_119671_object17559:head# 2024-03-20T09:05:00.592 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.592 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcc3d49d:::benchmark_data_smithi122_119671_object49261:head# 2024-03-20T09:05:00.592 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.592 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcc3ded1:::benchmark_data_smithi122_119671_object99167:head# 2024-03-20T09:05:00.592 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.592 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcc4d284:::benchmark_data_smithi122_119671_object81566:head# 2024-03-20T09:05:00.592 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.592 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcc64c25:::benchmark_data_smithi122_119671_object45622:head# 2024-03-20T09:05:00.592 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.593 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcc99a85:::benchmark_data_smithi122_119671_object75066:head# 2024-03-20T09:05:00.593 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.593 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcca4899:::benchmark_data_smithi122_119671_object81079:head# 2024-03-20T09:05:00.593 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.593 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcca9741:::benchmark_data_smithi122_119671_object69129:head# 2024-03-20T09:05:00.593 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.593 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bccb3fb2:::benchmark_data_smithi122_119671_object26187:head# 2024-03-20T09:05:00.593 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.593 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bccd0064:::benchmark_data_smithi122_119671_object64582:head# 2024-03-20T09:05:00.593 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.593 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bccd0b79:::benchmark_data_smithi122_119671_object94070:head# 2024-03-20T09:05:00.593 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.593 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bccdbe57:::benchmark_data_smithi122_119671_object5746:head# 2024-03-20T09:05:00.593 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.594 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bccdde08:::benchmark_data_smithi122_119671_object29866:head# 2024-03-20T09:05:00.594 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.594 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcce62db:::benchmark_data_smithi122_119671_object5983:head# 2024-03-20T09:05:00.594 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.594 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcd0184b:::benchmark_data_smithi122_119671_object51032:head# 2024-03-20T09:05:00.594 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.594 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcd04981:::benchmark_data_smithi122_119671_object73784:head# 2024-03-20T09:05:00.594 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.594 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcd0be0d:::benchmark_data_smithi122_119671_object1946:head# 2024-03-20T09:05:00.594 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.594 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcd1f18d:::benchmark_data_smithi122_119671_object47900:head# 2024-03-20T09:05:00.594 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.594 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcd391b2:::benchmark_data_smithi122_119671_object6145:head# 2024-03-20T09:05:00.594 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.595 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcd3f8c0:::benchmark_data_smithi122_119671_object83704:head# 2024-03-20T09:05:00.595 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.595 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcd488cf:::benchmark_data_smithi122_119671_object93409:head# 2024-03-20T09:05:00.595 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.595 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcd527d9:::benchmark_data_smithi122_119671_object88400:head# 2024-03-20T09:05:00.595 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.595 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcd746c9:::benchmark_data_smithi122_119671_object77821:head# 2024-03-20T09:05:00.595 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.595 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcd7fbc1:::benchmark_data_smithi122_119671_object92101:head# 2024-03-20T09:05:00.595 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.595 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcd9f19a:::benchmark_data_smithi122_119671_object25689:head# 2024-03-20T09:05:00.595 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.595 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcda56d6:::benchmark_data_smithi122_119671_object39143:head# 2024-03-20T09:05:00.595 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.596 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcdae7ed:::benchmark_data_smithi122_119671_object38106:head# 2024-03-20T09:05:00.596 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.596 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcdaeee7:::benchmark_data_smithi122_119671_object57291:head# 2024-03-20T09:05:00.596 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.596 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcdbe3fb:::benchmark_data_smithi122_119671_object54905:head# 2024-03-20T09:05:00.596 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.596 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcdc5aff:::benchmark_data_smithi122_119671_object61422:head# 2024-03-20T09:05:00.596 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.596 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcdd0718:::benchmark_data_smithi122_119671_object22174:head# 2024-03-20T09:05:00.596 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.596 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcdd1d7c:::benchmark_data_smithi122_119671_object69449:head# 2024-03-20T09:05:00.596 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.596 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bce009a1:::benchmark_data_smithi122_119671_object58105:head# 2024-03-20T09:05:00.596 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.597 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bce0e9a2:::benchmark_data_smithi122_119671_object50723:head# 2024-03-20T09:05:00.597 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.597 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bce27829:::benchmark_data_smithi122_119671_object39796:head# 2024-03-20T09:05:00.597 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.597 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bce30fec:::benchmark_data_smithi122_119671_object28517:head# 2024-03-20T09:05:00.597 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.597 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bce4277c:::benchmark_data_smithi122_119671_object67217:head# 2024-03-20T09:05:00.597 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.597 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bce49c1c:::benchmark_data_smithi122_119671_object45854:head# 2024-03-20T09:05:00.597 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.597 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bce4e76f:::benchmark_data_smithi122_119671_object11418:head# 2024-03-20T09:05:00.597 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.597 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bce531ea:::benchmark_data_smithi122_119671_object44807:head# 2024-03-20T09:05:00.598 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.598 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bce58b87:::benchmark_data_smithi122_119671_object24888:head# 2024-03-20T09:05:00.598 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.598 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bce5b795:::benchmark_data_smithi122_119671_object73828:head# 2024-03-20T09:05:00.598 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.598 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bce64fe5:::benchmark_data_smithi122_119671_object15549:head# 2024-03-20T09:05:00.598 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.598 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bce684f3:::benchmark_data_smithi122_119671_object80552:head# 2024-03-20T09:05:00.598 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.602 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bce7a426:::benchmark_data_smithi122_119671_object67786:head# 2024-03-20T09:05:00.602 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.602 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bce7cbc9:::benchmark_data_smithi122_119671_object92223:head# 2024-03-20T09:05:00.602 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.602 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bce87abd:::benchmark_data_smithi122_119671_object9884:head# 2024-03-20T09:05:00.602 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.602 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bce87d60:::benchmark_data_smithi122_119671_object89349:head# 2024-03-20T09:05:00.602 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.602 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bce8f4c1:::benchmark_data_smithi122_119671_object76275:head# 2024-03-20T09:05:00.603 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.603 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bce9103d:::benchmark_data_smithi122_119671_object14622:head# 2024-03-20T09:05:00.603 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.603 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcea1f89:::benchmark_data_smithi122_119671_object21590:head# 2024-03-20T09:05:00.603 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.603 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcecb46e:::benchmark_data_smithi122_119671_object25532:head# 2024-03-20T09:05:00.603 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.603 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bced6726:::benchmark_data_smithi122_119671_object71967:head# 2024-03-20T09:05:00.603 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.603 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bceefc2e:::benchmark_data_smithi122_119671_object88320:head# 2024-03-20T09:05:00.603 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.603 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcef6c07:::benchmark_data_smithi122_119671_object63708:head# 2024-03-20T09:05:00.603 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.603 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcef74ce:::benchmark_data_smithi122_119671_object76467:head# 2024-03-20T09:05:00.604 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.604 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcf19f94:::benchmark_data_smithi122_119671_object75127:head# 2024-03-20T09:05:00.604 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.604 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcf1d9cb:::benchmark_data_smithi122_119671_object95511:head# 2024-03-20T09:05:00.604 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.604 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcf219fc:::benchmark_data_smithi122_119671_object43352:head# 2024-03-20T09:05:00.604 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.604 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcf2373b:::benchmark_data_smithi122_119671_object9391:head# 2024-03-20T09:05:00.604 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.604 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcf360bb:::benchmark_data_smithi122_119671_object98540:head# 2024-03-20T09:05:00.604 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.604 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcf44b8b:::benchmark_data_smithi122_119671_object99001:head# 2024-03-20T09:05:00.604 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.604 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcf45e7f:::benchmark_data_smithi122_119671_object63490:head# 2024-03-20T09:05:00.605 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.605 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcf50e97:::benchmark_data_smithi122_119671_object43690:head# 2024-03-20T09:05:00.605 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.605 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcf54eff:::benchmark_data_smithi122_119671_object28250:head# 2024-03-20T09:05:00.605 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.605 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcf59252:::benchmark_data_smithi122_119671_object23677:head# 2024-03-20T09:05:00.605 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.605 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcf5c077:::benchmark_data_smithi122_119671_object32128:head# 2024-03-20T09:05:00.605 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.605 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcf5d7ff:::benchmark_data_smithi122_119671_object58331:head# 2024-03-20T09:05:00.605 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.605 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcf70525:::benchmark_data_smithi122_119671_object36313:head# 2024-03-20T09:05:00.605 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.605 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcf7142a:::benchmark_data_smithi122_119671_object23673:head# 2024-03-20T09:05:00.606 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.606 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcf7d3be:::benchmark_data_smithi122_119671_object74867:head# 2024-03-20T09:05:00.606 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.606 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcf7df71:::benchmark_data_smithi122_119671_object73162:head# 2024-03-20T09:05:00.606 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.606 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcf833ef:::benchmark_data_smithi122_119671_object59698:head# 2024-03-20T09:05:00.606 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.606 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcf83da4:::benchmark_data_smithi122_119671_object5878:head# 2024-03-20T09:05:00.606 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.606 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcf9238f:::benchmark_data_smithi122_119671_object14939:head# 2024-03-20T09:05:00.606 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.606 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcf94fac:::benchmark_data_smithi122_119671_object53820:head# 2024-03-20T09:05:00.606 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.607 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcf9d9fe:::benchmark_data_smithi122_119671_object95679:head# 2024-03-20T09:05:00.607 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.607 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcf9e106:::benchmark_data_smithi122_119671_object52508:head# 2024-03-20T09:05:00.607 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.607 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcfa0a38:::benchmark_data_smithi122_119671_object60577:head# 2024-03-20T09:05:00.607 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.607 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcfe49fa:::benchmark_data_smithi122_119671_object45549:head# 2024-03-20T09:05:00.607 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.607 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bcff34e6:::benchmark_data_smithi122_119671_object93125:head# 2024-03-20T09:05:00.607 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.607 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd000f91:::benchmark_data_smithi122_119671_object30578:head# 2024-03-20T09:05:00.607 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.607 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd02ef06:::benchmark_data_smithi122_119671_object7351:head# 2024-03-20T09:05:00.607 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.608 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd03610e:::benchmark_data_smithi122_119671_object11733:head# 2024-03-20T09:05:00.608 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.608 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd040e66:::benchmark_data_smithi122_119671_object75084:head# 2024-03-20T09:05:00.608 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.608 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd06d9ab:::benchmark_data_smithi122_119671_object73950:head# 2024-03-20T09:05:00.608 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.608 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd071eba:::benchmark_data_smithi122_119671_object57381:head# 2024-03-20T09:05:00.608 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.608 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd07882c:::benchmark_data_smithi122_119671_object62643:head# 2024-03-20T09:05:00.608 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.608 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd07b187:::benchmark_data_smithi122_119671_object34500:head# 2024-03-20T09:05:00.608 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.608 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd07f3d5:::benchmark_data_smithi122_119671_object25917:head# 2024-03-20T09:05:00.609 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.609 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd081180:::benchmark_data_smithi122_119671_object1645:head# 2024-03-20T09:05:00.609 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.609 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd09652b:::benchmark_data_smithi122_119671_object97849:head# 2024-03-20T09:05:00.609 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.609 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd099007:::benchmark_data_smithi122_119671_object41524:head# 2024-03-20T09:05:00.609 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.609 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd0b4087:::benchmark_data_smithi122_119671_object82294:head# 2024-03-20T09:05:00.609 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.609 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd0be456:::benchmark_data_smithi122_119671_object45535:head# 2024-03-20T09:05:00.609 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.609 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd0cef23:::benchmark_data_smithi122_119671_object95963:head# 2024-03-20T09:05:00.609 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.609 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd0d6429:::benchmark_data_smithi122_119671_object94373:head# 2024-03-20T09:05:00.610 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.610 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd0e30af:::benchmark_data_smithi122_119671_object43489:head# 2024-03-20T09:05:00.610 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.610 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd0ee820:::benchmark_data_smithi122_119671_object57208:head# 2024-03-20T09:05:00.610 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.610 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd0f5d3b:::benchmark_data_smithi122_119671_object60417:head# 2024-03-20T09:05:00.610 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.610 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd0f91f1:::benchmark_data_smithi122_119671_object10425:head# 2024-03-20T09:05:00.610 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.610 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd103b85:::benchmark_data_smithi122_119671_object79862:head# 2024-03-20T09:05:00.610 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.610 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd1090aa:::benchmark_data_smithi122_119671_object65150:head# 2024-03-20T09:05:00.610 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.611 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd10c91f:::benchmark_data_smithi122_119671_object80561:head# 2024-03-20T09:05:00.611 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.611 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd117b57:::benchmark_data_smithi122_119671_object83261:head# 2024-03-20T09:05:00.611 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.611 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd119a5c:::benchmark_data_smithi122_119671_object38344:head# 2024-03-20T09:05:00.611 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.611 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd13c2b6:::benchmark_data_smithi122_119671_object79987:head# 2024-03-20T09:05:00.611 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.611 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd13f2e1:::benchmark_data_smithi122_119671_object62503:head# 2024-03-20T09:05:00.611 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.611 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd1431e7:::benchmark_data_smithi122_119671_object51405:head# 2024-03-20T09:05:00.611 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.611 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd143238:::benchmark_data_smithi122_119671_object37670:head# 2024-03-20T09:05:00.611 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.611 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd14691a:::benchmark_data_smithi122_119671_object61433:head# 2024-03-20T09:05:00.612 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.612 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd14b614:::benchmark_data_smithi122_119671_object47951:head# 2024-03-20T09:05:00.612 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.612 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd150d4c:::benchmark_data_smithi122_119671_object95177:head# 2024-03-20T09:05:00.612 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.612 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd163750:::benchmark_data_smithi122_119671_object85622:head# 2024-03-20T09:05:00.612 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.612 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd1753d3:::benchmark_data_smithi122_119671_object59215:head# 2024-03-20T09:05:00.612 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.612 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd1770e2:::benchmark_data_smithi122_119671_object31058:head# 2024-03-20T09:05:00.612 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.612 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd17916b:::benchmark_data_smithi122_119671_object26049:head# 2024-03-20T09:05:00.612 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.612 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd17a85e:::benchmark_data_smithi122_119671_object21172:head# 2024-03-20T09:05:00.612 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.613 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd18961d:::benchmark_data_smithi122_119671_object4796:head# 2024-03-20T09:05:00.613 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.613 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd190652:::benchmark_data_smithi122_119671_object99343:head# 2024-03-20T09:05:00.613 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.613 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd1913e7:::benchmark_data_smithi122_119671_object27405:head# 2024-03-20T09:05:00.613 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.613 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd19b601:::benchmark_data_smithi122_119671_object50677:head# 2024-03-20T09:05:00.613 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.613 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd19d0e8:::benchmark_data_smithi122_119671_object84946:head# 2024-03-20T09:05:00.613 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.613 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd1a1b5f:::benchmark_data_smithi122_119671_object84497:head# 2024-03-20T09:05:00.613 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.613 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd1ad707:::benchmark_data_smithi122_119671_object50319:head# 2024-03-20T09:05:00.614 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.614 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd1b2e2e:::benchmark_data_smithi122_119671_object73602:head# 2024-03-20T09:05:00.614 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.614 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd1c0db0:::benchmark_data_smithi122_119671_object50323:head# 2024-03-20T09:05:00.614 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.614 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd1d19ae:::benchmark_data_smithi122_119671_object11229:head# 2024-03-20T09:05:00.614 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.614 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd1de2b1:::benchmark_data_smithi122_119671_object51419:head# 2024-03-20T09:05:00.614 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.614 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd1e304e:::benchmark_data_smithi122_119671_object96406:head# 2024-03-20T09:05:00.614 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.614 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd1f3aef:::benchmark_data_smithi122_119671_object73042:head# 2024-03-20T09:05:00.614 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.614 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd1f4efc:::benchmark_data_smithi122_119671_object78146:head# 2024-03-20T09:05:00.615 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.615 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd2052a8:::benchmark_data_smithi122_119671_object14206:head# 2024-03-20T09:05:00.615 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.615 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd20dd73:::benchmark_data_smithi122_119671_object76583:head# 2024-03-20T09:05:00.615 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.615 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd20f2a4:::benchmark_data_smithi122_119671_object23560:head# 2024-03-20T09:05:00.615 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.615 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd20f315:::benchmark_data_smithi122_119671_object83629:head# 2024-03-20T09:05:00.615 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.615 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd2113b0:::benchmark_data_smithi122_119671_object17506:head# 2024-03-20T09:05:00.615 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.615 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd22bc48:::benchmark_data_smithi122_119671_object74047:head# 2024-03-20T09:05:00.615 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.615 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd25a771:::benchmark_data_smithi122_119671_object9673:head# 2024-03-20T09:05:00.615 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.616 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd2670f9:::benchmark_data_smithi122_119671_object68531:head# 2024-03-20T09:05:00.616 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.616 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd26e127:::benchmark_data_smithi122_119671_object4421:head# 2024-03-20T09:05:00.616 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.616 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd26f86d:::benchmark_data_smithi122_119671_object19527:head# 2024-03-20T09:05:00.616 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.616 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd27a723:::benchmark_data_smithi122_119671_object49494:head# 2024-03-20T09:05:00.616 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.616 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd282d09:::benchmark_data_smithi122_119671_object91392:head# 2024-03-20T09:05:00.616 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.616 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd289a33:::benchmark_data_smithi122_119671_object35144:head# 2024-03-20T09:05:00.616 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.616 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd28bc41:::benchmark_data_smithi122_119671_object45629:head# 2024-03-20T09:05:00.616 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.617 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd28ecd1:::benchmark_data_smithi122_119671_object91484:head# 2024-03-20T09:05:00.617 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.617 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd29a776:::benchmark_data_smithi122_119671_object56559:head# 2024-03-20T09:05:00.617 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.617 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd2a6a20:::benchmark_data_smithi122_119671_object53381:head# 2024-03-20T09:05:00.617 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.617 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd2a9d17:::benchmark_data_smithi122_119671_object26873:head# 2024-03-20T09:05:00.617 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.617 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd2ab425:::benchmark_data_smithi122_119671_object71513:head# 2024-03-20T09:05:00.617 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.617 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd2b42dc:::benchmark_data_smithi122_119671_object46323:head# 2024-03-20T09:05:00.617 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.617 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd2cfc63:::benchmark_data_smithi122_119671_object12575:head# 2024-03-20T09:05:00.617 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.618 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd2d0772:::benchmark_data_smithi122_119671_object85294:head# 2024-03-20T09:05:00.618 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.618 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd2d4786:::benchmark_data_smithi122_119671_object22103:head# 2024-03-20T09:05:00.618 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.618 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd2f29b0:::benchmark_data_smithi122_119671_object94999:head# 2024-03-20T09:05:00.618 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.618 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd2f44c5:::benchmark_data_smithi122_119671_object40982:head# 2024-03-20T09:05:00.618 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.618 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd2f970e:::benchmark_data_smithi122_119671_object39154:head# 2024-03-20T09:05:00.618 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.618 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd30c66c:::benchmark_data_smithi122_119671_object92932:head# 2024-03-20T09:05:00.618 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.618 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd3100de:::benchmark_data_smithi122_119671_object14005:head# 2024-03-20T09:05:00.618 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.619 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd312f97:::benchmark_data_smithi122_119671_object85476:head# 2024-03-20T09:05:00.619 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.619 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd314917:::benchmark_data_smithi122_119671_object68180:head# 2024-03-20T09:05:00.619 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.619 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd31c5ad:::benchmark_data_smithi122_119671_object16798:head# 2024-03-20T09:05:00.619 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.619 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd321f12:::benchmark_data_smithi122_119671_object78074:head# 2024-03-20T09:05:00.619 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.619 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd326ab3:::benchmark_data_smithi122_119671_object66251:head# 2024-03-20T09:05:00.619 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.619 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd32a828:::benchmark_data_smithi122_119671_object63075:head# 2024-03-20T09:05:00.619 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.619 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd32c192:::benchmark_data_smithi122_119671_object90450:head# 2024-03-20T09:05:00.619 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.620 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd338739:::benchmark_data_smithi122_119671_object78083:head# 2024-03-20T09:05:00.620 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.620 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd33c081:::benchmark_data_smithi122_119671_object77421:head# 2024-03-20T09:05:00.620 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.620 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd353517:::benchmark_data_smithi122_119671_object27843:head# 2024-03-20T09:05:00.620 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.620 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd35845c:::benchmark_data_smithi122_119671_object76842:head# 2024-03-20T09:05:00.620 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.620 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd3586df:::benchmark_data_smithi122_119671_object43899:head# 2024-03-20T09:05:00.620 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.620 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd36e017:::benchmark_data_smithi122_119671_object71298:head# 2024-03-20T09:05:00.620 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.620 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd379a5a:::benchmark_data_smithi122_119671_object18984:head# 2024-03-20T09:05:00.620 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.621 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd38308e:::benchmark_data_smithi122_119671_object60926:head# 2024-03-20T09:05:00.621 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.621 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd386940:::benchmark_data_smithi122_119671_object70472:head# 2024-03-20T09:05:00.621 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.621 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd3aa8ce:::benchmark_data_smithi122_119671_object76211:head# 2024-03-20T09:05:00.621 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.621 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd3afd00:::benchmark_data_smithi122_119671_object43058:head# 2024-03-20T09:05:00.621 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.621 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd3b44cf:::benchmark_data_smithi122_119671_object12265:head# 2024-03-20T09:05:00.621 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.621 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd3dcdb6:::benchmark_data_smithi122_119671_object95769:head# 2024-03-20T09:05:00.621 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.621 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd3eefc2:::benchmark_data_smithi122_119671_object42499:head# 2024-03-20T09:05:00.621 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.621 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd3efd65:::benchmark_data_smithi122_119671_object86152:head# 2024-03-20T09:05:00.622 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.622 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd3f0395:::benchmark_data_smithi122_119671_object38498:head# 2024-03-20T09:05:00.622 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.622 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd3fbf56:::benchmark_data_smithi122_119671_object91403:head# 2024-03-20T09:05:00.622 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.622 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd400a9e:::benchmark_data_smithi122_119671_object11507:head# 2024-03-20T09:05:00.622 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.622 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd403b54:::benchmark_data_smithi122_119671_object68984:head# 2024-03-20T09:05:00.622 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.622 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd40474a:::benchmark_data_smithi122_119671_object77772:head# 2024-03-20T09:05:00.622 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.622 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd4055b0:::benchmark_data_smithi122_119671_object73849:head# 2024-03-20T09:05:00.622 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.623 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd4199c7:::benchmark_data_smithi122_119671_object9915:head# 2024-03-20T09:05:00.623 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.623 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd434343:::benchmark_data_smithi122_119671_object71357:head# 2024-03-20T09:05:00.623 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.623 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd439729:::benchmark_data_smithi122_119671_object38723:head# 2024-03-20T09:05:00.623 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.623 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd44b992:::benchmark_data_smithi122_119671_object78274:head# 2024-03-20T09:05:00.623 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.623 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd4591ab:::benchmark_data_smithi122_119671_object60858:head# 2024-03-20T09:05:00.623 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.623 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd469213:::benchmark_data_smithi122_119671_object23133:head# 2024-03-20T09:05:00.623 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.623 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd49482d:::benchmark_data_smithi122_119671_object76968:head# 2024-03-20T09:05:00.623 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.624 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd49bacd:::benchmark_data_smithi122_119671_object39630:head# 2024-03-20T09:05:00.624 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.624 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd49c358:::benchmark_data_smithi122_119671_object94937:head# 2024-03-20T09:05:00.624 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.624 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd4a9d21:::benchmark_data_smithi122_119671_object17398:head# 2024-03-20T09:05:00.624 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.624 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd4b8c44:::benchmark_data_smithi122_119671_object64205:head# 2024-03-20T09:05:00.624 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.624 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd4b8f5b:::benchmark_data_smithi122_119671_object94162:head# 2024-03-20T09:05:00.624 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.624 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd4bb488:::benchmark_data_smithi122_119671_object30462:head# 2024-03-20T09:05:00.624 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.624 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd4c1953:::benchmark_data_smithi122_119671_object95970:head# 2024-03-20T09:05:00.624 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.624 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd4c4b53:::benchmark_data_smithi122_119671_object52154:head# 2024-03-20T09:05:00.625 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.625 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd4dd027:::benchmark_data_smithi122_119671_object92759:head# 2024-03-20T09:05:00.625 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.625 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd4e648d:::benchmark_data_smithi122_119671_object48094:head# 2024-03-20T09:05:00.625 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.625 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd4f6849:::benchmark_data_smithi122_119671_object75771:head# 2024-03-20T09:05:00.625 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.625 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd4f9ef3:::benchmark_data_smithi122_119671_object65626:head# 2024-03-20T09:05:00.625 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.625 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd511878:::benchmark_data_smithi122_119671_object55575:head# 2024-03-20T09:05:00.625 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.625 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd51384b:::benchmark_data_smithi122_119671_object56227:head# 2024-03-20T09:05:00.625 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.625 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd516235:::benchmark_data_smithi122_119671_object92813:head# 2024-03-20T09:05:00.626 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.626 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd51c98a:::benchmark_data_smithi122_119671_object75019:head# 2024-03-20T09:05:00.626 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.626 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd52df2b:::benchmark_data_smithi122_119671_object18885:head# 2024-03-20T09:05:00.626 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.626 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd53d48a:::benchmark_data_smithi122_119671_object6643:head# 2024-03-20T09:05:00.626 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.626 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd53f69e:::benchmark_data_smithi122_119671_object76718:head# 2024-03-20T09:05:00.626 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.626 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd540af5:::benchmark_data_smithi122_119671_object53122:head# 2024-03-20T09:05:00.626 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.626 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd541fdf:::benchmark_data_smithi122_119671_object80997:head# 2024-03-20T09:05:00.626 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.627 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd548739:::benchmark_data_smithi122_119671_object73178:head# 2024-03-20T09:05:00.627 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.627 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd54b0a7:::benchmark_data_smithi122_119671_object84295:head# 2024-03-20T09:05:00.627 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.627 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd54c655:::benchmark_data_smithi122_119671_object59305:head# 2024-03-20T09:05:00.627 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.627 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd550590:::benchmark_data_smithi122_119671_object34010:head# 2024-03-20T09:05:00.627 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.627 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd559043:::benchmark_data_smithi122_119671_object81762:head# 2024-03-20T09:05:00.627 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.627 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd5616e9:::benchmark_data_smithi122_119671_object71134:head# 2024-03-20T09:05:00.627 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.627 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd569ed4:::benchmark_data_smithi122_119671_object52745:head# 2024-03-20T09:05:00.627 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.628 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd56a63d:::benchmark_data_smithi122_119671_object65095:head# 2024-03-20T09:05:00.628 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.628 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd56feac:::benchmark_data_smithi122_119671_object74102:head# 2024-03-20T09:05:00.628 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.628 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd5831ba:::benchmark_data_smithi122_119671_object20214:head# 2024-03-20T09:05:00.628 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.628 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd58d960:::benchmark_data_smithi122_119671_object79897:head# 2024-03-20T09:05:00.628 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.628 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd59b7a3:::benchmark_data_smithi122_119671_object38740:head# 2024-03-20T09:05:00.628 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.628 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd59c7ca:::benchmark_data_smithi122_119671_object80606:head# 2024-03-20T09:05:00.628 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.628 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd5a30f0:::benchmark_data_smithi122_119671_object18253:head# 2024-03-20T09:05:00.628 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.628 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd5aa794:::benchmark_data_smithi122_119671_object36181:head# 2024-03-20T09:05:00.629 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.629 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd5cad3a:::benchmark_data_smithi122_119671_object78207:head# 2024-03-20T09:05:00.629 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.629 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd5d39e1:::benchmark_data_smithi122_119671_object92537:head# 2024-03-20T09:05:00.629 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.629 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd5d82dc:::benchmark_data_smithi122_119671_object98334:head# 2024-03-20T09:05:00.629 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.629 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd5dade4:::benchmark_data_smithi122_119671_object60097:head# 2024-03-20T09:05:00.629 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.629 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd5e9150:::benchmark_data_smithi122_119671_object48905:head# 2024-03-20T09:05:00.629 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.629 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd5f7c01:::benchmark_data_smithi122_119671_object56869:head# 2024-03-20T09:05:00.629 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.629 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd6273f3:::benchmark_data_smithi122_119671_object7982:head# 2024-03-20T09:05:00.630 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.630 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd62ceac:::benchmark_data_smithi122_119671_object81372:head# 2024-03-20T09:05:00.630 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.630 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd644841:::benchmark_data_smithi122_119671_object62181:head# 2024-03-20T09:05:00.630 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.630 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd646181:::benchmark_data_smithi122_119671_object33783:head# 2024-03-20T09:05:00.630 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.630 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd64f595:::benchmark_data_smithi122_119671_object50713:head# 2024-03-20T09:05:00.630 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.630 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd652b67:::benchmark_data_smithi122_119671_object88938:head# 2024-03-20T09:05:00.630 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.630 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd6598ba:::benchmark_data_smithi122_119671_object43400:head# 2024-03-20T09:05:00.630 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.630 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd667c76:::benchmark_data_smithi122_119671_object20599:head# 2024-03-20T09:05:00.631 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.631 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd67b8bb:::benchmark_data_smithi122_119671_object7526:head# 2024-03-20T09:05:00.631 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.631 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd69a5e1:::benchmark_data_smithi122_119671_object23659:head# 2024-03-20T09:05:00.631 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.631 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd6a6a72:::benchmark_data_smithi122_119671_object4585:head# 2024-03-20T09:05:00.631 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.631 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd6a8f86:::benchmark_data_smithi122_119671_object27130:head# 2024-03-20T09:05:00.631 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.631 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd6b94cd:::benchmark_data_smithi122_119671_object93344:head# 2024-03-20T09:05:00.631 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.631 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd6bb468:::benchmark_data_smithi122_119671_object8283:head# 2024-03-20T09:05:00.631 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.631 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd6c19d0:::benchmark_data_smithi122_119671_object64735:head# 2024-03-20T09:05:00.632 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.632 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd6c1ce3:::benchmark_data_smithi122_119671_object71256:head# 2024-03-20T09:05:00.632 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.632 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd6c9acc:::benchmark_data_smithi122_119671_object35485:head# 2024-03-20T09:05:00.632 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.632 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd6cc160:::benchmark_data_smithi122_119671_object20819:head# 2024-03-20T09:05:00.632 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.632 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd6d6b2d:::benchmark_data_smithi122_119671_object24238:head# 2024-03-20T09:05:00.632 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.632 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd6d91d3:::benchmark_data_smithi122_119671_object87668:head# 2024-03-20T09:05:00.632 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.632 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd6df201:::benchmark_data_smithi122_119671_object66567:head# 2024-03-20T09:05:00.632 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.633 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd6e9a6e:::benchmark_data_smithi122_119671_object10318:head# 2024-03-20T09:05:00.633 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.633 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd6f3e15:::benchmark_data_smithi122_119671_object40232:head# 2024-03-20T09:05:00.633 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.633 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd6f810b:::benchmark_data_smithi122_119671_object78232:head# 2024-03-20T09:05:00.633 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.633 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd700ce2:::benchmark_data_smithi122_119671_object31794:head# 2024-03-20T09:05:00.633 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.633 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd7019bb:::benchmark_data_smithi122_119671_object95297:head# 2024-03-20T09:05:00.633 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.633 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd709e20:::benchmark_data_smithi122_119671_object22996:head# 2024-03-20T09:05:00.633 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.633 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd70a008:::benchmark_data_smithi122_119671_object86280:head# 2024-03-20T09:05:00.633 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.634 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd70f320:::benchmark_data_smithi122_119671_object78577:head# 2024-03-20T09:05:00.634 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.634 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd711001:::benchmark_data_smithi122_119671_object56336:head# 2024-03-20T09:05:00.634 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.634 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd7286e7:::benchmark_data_smithi122_119671_object80331:head# 2024-03-20T09:05:00.634 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.634 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd72da8f:::benchmark_data_smithi122_119671_object35326:head# 2024-03-20T09:05:00.634 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.634 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd74c281:::benchmark_data_smithi122_119671_object60518:head# 2024-03-20T09:05:00.634 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.634 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd74e666:::benchmark_data_smithi122_119671_object32765:head# 2024-03-20T09:05:00.634 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.634 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd755700:::benchmark_data_smithi122_119671_object53127:head# 2024-03-20T09:05:00.634 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.635 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd75c8a6:::benchmark_data_smithi122_119671_object36984:head# 2024-03-20T09:05:00.635 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.635 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd7685ea:::benchmark_data_smithi122_119671_object17243:head# 2024-03-20T09:05:00.635 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.635 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd76cff2:::benchmark_data_smithi122_119671_object98198:head# 2024-03-20T09:05:00.635 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.635 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd777cb4:::benchmark_data_smithi122_119671_object94050:head# 2024-03-20T09:05:00.635 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.635 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd7b3e03:::benchmark_data_smithi122_119671_object61231:head# 2024-03-20T09:05:00.635 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.635 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd7c8086:::benchmark_data_smithi122_119671_object6693:head# 2024-03-20T09:05:00.635 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.635 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd7ce254:::benchmark_data_smithi122_119671_object42750:head# 2024-03-20T09:05:00.635 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.636 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd7d8a01:::benchmark_data_smithi122_119671_object37645:head# 2024-03-20T09:05:00.636 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.636 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd7dee82:::benchmark_data_smithi122_119671_object53854:head# 2024-03-20T09:05:00.636 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.636 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd7df91f:::benchmark_data_smithi122_119671_object75152:head# 2024-03-20T09:05:00.636 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.636 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd7e7824:::benchmark_data_smithi122_119671_object41116:head# 2024-03-20T09:05:00.636 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.636 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd7f1611:::benchmark_data_smithi122_119671_object92237:head# 2024-03-20T09:05:00.636 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.636 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd7f81c0:::benchmark_data_smithi122_119671_object67491:head# 2024-03-20T09:05:00.636 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.636 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd801101:::benchmark_data_smithi122_119671_object74399:head# 2024-03-20T09:05:00.636 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.637 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd805f22:::benchmark_data_smithi122_119671_object21759:head# 2024-03-20T09:05:00.637 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.637 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd80ef14:::benchmark_data_smithi122_119671_object71941:head# 2024-03-20T09:05:00.637 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.637 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd817b11:::benchmark_data_smithi122_119671_object62421:head# 2024-03-20T09:05:00.637 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.637 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd8210ed:::benchmark_data_smithi122_119671_object80119:head# 2024-03-20T09:05:00.637 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.637 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd8288be:::benchmark_data_smithi122_119671_object82173:head# 2024-03-20T09:05:00.637 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.637 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd846a89:::benchmark_data_smithi122_119671_object78736:head# 2024-03-20T09:05:00.637 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.637 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd84d53d:::benchmark_data_smithi122_119671_object25470:head# 2024-03-20T09:05:00.638 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.638 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd84f5b1:::benchmark_data_smithi122_119671_object58720:head# 2024-03-20T09:05:00.638 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.638 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd85463e:::benchmark_data_smithi122_119671_object7820:head# 2024-03-20T09:05:00.638 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.638 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd857fc8:::benchmark_data_smithi122_119671_object26690:head# 2024-03-20T09:05:00.638 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.638 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd866398:::benchmark_data_smithi122_119671_object33480:head# 2024-03-20T09:05:00.638 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.638 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd86bd38:::benchmark_data_smithi122_119671_object22137:head# 2024-03-20T09:05:00.638 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.638 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd870f7e:::benchmark_data_smithi122_119671_object48865:head# 2024-03-20T09:05:00.638 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.639 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd871336:::benchmark_data_smithi122_119671_object59133:head# 2024-03-20T09:05:00.639 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.639 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd8760a4:::benchmark_data_smithi122_119671_object6521:head# 2024-03-20T09:05:00.639 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.639 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd88ad20:::benchmark_data_smithi122_119671_object5799:head# 2024-03-20T09:05:00.639 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.639 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd89c056:::benchmark_data_smithi122_119671_object73279:head# 2024-03-20T09:05:00.639 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.639 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd8a91da:::benchmark_data_smithi122_119671_object92692:head# 2024-03-20T09:05:00.639 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.639 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd8b4c67:::benchmark_data_smithi122_119671_object25139:head# 2024-03-20T09:05:00.639 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.639 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd8ca23d:::benchmark_data_smithi122_119671_object27775:head# 2024-03-20T09:05:00.640 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.640 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd8ce1ec:::benchmark_data_smithi122_119671_object76766:head# 2024-03-20T09:05:00.640 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.640 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd8dbe5b:::benchmark_data_smithi122_119671_object36613:head# 2024-03-20T09:05:00.640 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.640 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd8dded4:::benchmark_data_smithi122_119671_object49779:head# 2024-03-20T09:05:00.640 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.640 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd90bb06:::benchmark_data_smithi122_119671_object57742:head# 2024-03-20T09:05:00.640 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.640 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd92c3d6:::benchmark_data_smithi122_119671_object6356:head# 2024-03-20T09:05:00.640 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.640 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd931ddf:::benchmark_data_smithi122_119671_object70415:head# 2024-03-20T09:05:00.640 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.640 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd93a8c7:::benchmark_data_smithi122_119671_object60822:head# 2024-03-20T09:05:00.641 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.641 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd960050:::benchmark_data_smithi122_119671_object99120:head# 2024-03-20T09:05:00.641 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.641 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd965dc1:::benchmark_data_smithi122_119671_object78336:head# 2024-03-20T09:05:00.641 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.641 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd96a2df:::benchmark_data_smithi122_119671_object90317:head# 2024-03-20T09:05:00.641 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.641 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd973224:::benchmark_data_smithi122_119671_object99199:head# 2024-03-20T09:05:00.641 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.641 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd975139:::benchmark_data_smithi122_119671_object14479:head# 2024-03-20T09:05:00.641 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.641 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd97f774:::benchmark_data_smithi122_119671_object60845:head# 2024-03-20T09:05:00.641 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.642 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd980810:::benchmark_data_smithi122_119671_object82001:head# 2024-03-20T09:05:00.642 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.642 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd98a188:::benchmark_data_smithi122_119671_object58182:head# 2024-03-20T09:05:00.642 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.642 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd99902a:::benchmark_data_smithi122_119671_object33765:head# 2024-03-20T09:05:00.642 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.642 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd9b39c9:::benchmark_data_smithi122_119671_object41854:head# 2024-03-20T09:05:00.642 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.642 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd9cb0b9:::benchmark_data_smithi122_119671_object34708:head# 2024-03-20T09:05:00.642 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.642 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd9d9fe3:::benchmark_data_smithi122_119671_object98043:head# 2024-03-20T09:05:00.642 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.642 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bd9f2e86:::benchmark_data_smithi122_119671_object93868:head# 2024-03-20T09:05:00.643 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.643 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bda01553:::benchmark_data_smithi122_119671_object20529:head# 2024-03-20T09:05:00.643 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.643 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bda03a3e:::benchmark_data_smithi122_119671_object47814:head# 2024-03-20T09:05:00.643 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.643 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bda0f464:::benchmark_data_smithi122_119671_object23410:head# 2024-03-20T09:05:00.643 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.643 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bda1aad3:::benchmark_data_smithi122_119671_object25406:head# 2024-03-20T09:05:00.643 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.643 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bda1d6f9:::benchmark_data_smithi122_119671_object86127:head# 2024-03-20T09:05:00.643 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.643 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bda26902:::benchmark_data_smithi122_119671_object58561:head# 2024-03-20T09:05:00.643 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.643 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bda2ed78:::benchmark_data_smithi122_119671_object22301:head# 2024-03-20T09:05:00.644 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.644 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bda34f14:::benchmark_data_smithi122_119671_object17499:head# 2024-03-20T09:05:00.644 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.644 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bda3609c:::benchmark_data_smithi122_119671_object26071:head# 2024-03-20T09:05:00.644 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.644 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bda4f8a8:::benchmark_data_smithi122_119671_object85152:head# 2024-03-20T09:05:00.644 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.644 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bda56562:::benchmark_data_smithi122_119671_object30439:head# 2024-03-20T09:05:00.644 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.644 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bda5772f:::benchmark_data_smithi122_119671_object52100:head# 2024-03-20T09:05:00.644 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.644 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bda7253e:::benchmark_data_smithi122_119671_object33409:head# 2024-03-20T09:05:00.644 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.644 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bda7e18b:::benchmark_data_smithi122_119671_object66754:head# 2024-03-20T09:05:00.644 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.645 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bda9ab3e:::benchmark_data_smithi122_119671_object98720:head# 2024-03-20T09:05:00.645 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.645 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bda9beb2:::benchmark_data_smithi122_119671_object21474:head# 2024-03-20T09:05:00.645 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.645 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdaa2228:::benchmark_data_smithi122_119671_object26068:head# 2024-03-20T09:05:00.645 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.645 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdaa9dfb:::benchmark_data_smithi122_119671_object88516:head# 2024-03-20T09:05:00.645 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.645 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdaab9da:::benchmark_data_smithi122_119671_object57877:head# 2024-03-20T09:05:00.645 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.645 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdaac687:::benchmark_data_smithi122_119671_object80692:head# 2024-03-20T09:05:00.645 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.645 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdab7e10:::benchmark_data_smithi122_119671_object40814:head# 2024-03-20T09:05:00.646 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.646 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdacbac9:::benchmark_data_smithi122_119671_object33424:head# 2024-03-20T09:05:00.646 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.646 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdad723f:::benchmark_data_smithi122_119671_object45022:head# 2024-03-20T09:05:00.646 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.646 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdae3c1a:::benchmark_data_smithi122_119671_object29402:head# 2024-03-20T09:05:00.646 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.646 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdafb051:::benchmark_data_smithi122_119671_object91785:head# 2024-03-20T09:05:00.646 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.646 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdb042cb:::benchmark_data_smithi122_119671_object17483:head# 2024-03-20T09:05:00.646 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.646 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdb0ca86:::benchmark_data_smithi122_119671_object74220:head# 2024-03-20T09:05:00.646 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.646 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdb11e1a:::benchmark_data_smithi122_119671_object6773:head# 2024-03-20T09:05:00.647 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.647 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdb12c5a:::benchmark_data_smithi122_119671_object61883:head# 2024-03-20T09:05:00.647 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.647 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdb15fad:::benchmark_data_smithi122_119671_object44072:head# 2024-03-20T09:05:00.647 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.647 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdb1e5ec:::benchmark_data_smithi122_119671_object26781:head# 2024-03-20T09:05:00.647 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.647 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdb2a609:::benchmark_data_smithi122_119671_object87852:head# 2024-03-20T09:05:00.647 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.647 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdb2f48a:::benchmark_data_smithi122_119671_object40384:head# 2024-03-20T09:05:00.647 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.647 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdb475e3:::benchmark_data_smithi122_119671_object83684:head# 2024-03-20T09:05:00.647 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.648 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdb4903f:::benchmark_data_smithi122_119671_object34649:head# 2024-03-20T09:05:00.648 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.648 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdb4bf32:::benchmark_data_smithi122_119671_object93096:head# 2024-03-20T09:05:00.648 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.648 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdb532b0:::benchmark_data_smithi122_119671_object73534:head# 2024-03-20T09:05:00.648 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.648 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdb57e99:::benchmark_data_smithi122_119671_object50901:head# 2024-03-20T09:05:00.648 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.648 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdb63a9c:::benchmark_data_smithi122_119671_object1870:head# 2024-03-20T09:05:00.648 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.648 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdb66bce:::benchmark_data_smithi122_119671_object97546:head# 2024-03-20T09:05:00.648 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.648 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdb71c3a:::benchmark_data_smithi122_119671_object9456:head# 2024-03-20T09:05:00.648 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.649 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdb74eee:::benchmark_data_smithi122_119671_object47742:head# 2024-03-20T09:05:00.649 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.649 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdb949bd:::benchmark_data_smithi122_119671_object16259:head# 2024-03-20T09:05:00.649 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.649 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdb9d4c4:::benchmark_data_smithi122_119671_object59094:head# 2024-03-20T09:05:00.649 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.649 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdba1836:::benchmark_data_smithi122_119671_object28862:head# 2024-03-20T09:05:00.649 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.649 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdba4815:::benchmark_data_smithi122_119671_object52877:head# 2024-03-20T09:05:00.649 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.649 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdbbb384:::benchmark_data_smithi122_119671_object43162:head# 2024-03-20T09:05:00.649 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.649 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdbbfd36:::benchmark_data_smithi122_119671_object96481:head# 2024-03-20T09:05:00.649 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.650 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdbc9130:::benchmark_data_smithi122_119671_object19566:head# 2024-03-20T09:05:00.650 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.650 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdbd547c:::benchmark_data_smithi122_119671_object1170:head# 2024-03-20T09:05:00.650 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.650 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdbe42b0:::benchmark_data_smithi122_119671_object62647:head# 2024-03-20T09:05:00.650 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.650 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdbeb57c:::benchmark_data_smithi122_119671_object84668:head# 2024-03-20T09:05:00.650 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.650 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdbf0b86:::benchmark_data_smithi122_119671_object81505:head# 2024-03-20T09:05:00.650 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.650 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdbfd87e:::benchmark_data_smithi122_119671_object65415:head# 2024-03-20T09:05:00.651 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.651 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdc07ffe:::benchmark_data_smithi122_119671_object37144:head# 2024-03-20T09:05:00.651 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.651 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdc0bbb2:::benchmark_data_smithi122_119671_object77974:head# 2024-03-20T09:05:00.651 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.651 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdc15793:::benchmark_data_smithi122_119671_object7173:head# 2024-03-20T09:05:00.651 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.651 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdc1b35a:::benchmark_data_smithi122_119671_object92281:head# 2024-03-20T09:05:00.651 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.651 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdc1c071:::benchmark_data_smithi122_119671_object15247:head# 2024-03-20T09:05:00.651 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.651 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdc25a56:::benchmark_data_smithi122_119671_object25729:head# 2024-03-20T09:05:00.651 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.651 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdc44391:::benchmark_data_smithi122_119671_object63594:head# 2024-03-20T09:05:00.652 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.652 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdc456fd:::benchmark_data_smithi122_119671_object62397:head# 2024-03-20T09:05:00.652 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.652 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdc5a978:::benchmark_data_smithi122_119671_object44448:head# 2024-03-20T09:05:00.652 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.652 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdc5cc60:::benchmark_data_smithi122_119671_object42878:head# 2024-03-20T09:05:00.652 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.652 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdc69e24:::benchmark_data_smithi122_119671_object6171:head# 2024-03-20T09:05:00.652 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.652 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdc6c003:::benchmark_data_smithi122_119671_object76571:head# 2024-03-20T09:05:00.652 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.652 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdc6f35f:::benchmark_data_smithi122_119671_object83760:head# 2024-03-20T09:05:00.652 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.652 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdc6ff24:::benchmark_data_smithi122_119671_object18350:head# 2024-03-20T09:05:00.653 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.653 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdc715a0:::benchmark_data_smithi122_119671_object91938:head# 2024-03-20T09:05:00.653 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.653 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdc763c5:::benchmark_data_smithi122_119671_object8604:head# 2024-03-20T09:05:00.653 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.653 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdc790d4:::benchmark_data_smithi122_119671_object15238:head# 2024-03-20T09:05:00.653 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.653 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdc7c60e:::benchmark_data_smithi122_119671_object74659:head# 2024-03-20T09:05:00.653 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.653 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdc81350:::benchmark_data_smithi122_119671_object64387:head# 2024-03-20T09:05:00.653 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.653 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdc84620:::benchmark_data_smithi122_119671_object85253:head# 2024-03-20T09:05:00.653 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.653 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdc9ac2d:::benchmark_data_smithi122_119671_object89390:head# 2024-03-20T09:05:00.653 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.654 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdc9f088:::benchmark_data_smithi122_119671_object96728:head# 2024-03-20T09:05:00.654 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.654 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdcaf3b3:::benchmark_data_smithi122_119671_object26197:head# 2024-03-20T09:05:00.654 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.654 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdcb762e:::benchmark_data_smithi122_119671_object93224:head# 2024-03-20T09:05:00.654 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.654 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdcbbcf6:::benchmark_data_smithi122_119671_object8608:head# 2024-03-20T09:05:00.654 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.654 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdcc6175:::benchmark_data_smithi122_119671_object99058:head# 2024-03-20T09:05:00.654 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.654 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdcc6417:::benchmark_data_smithi122_119671_object66443:head# 2024-03-20T09:05:00.654 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.654 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdcd0fcc:::benchmark_data_smithi122_119671_object36244:head# 2024-03-20T09:05:00.654 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.655 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdcdf243:::benchmark_data_smithi122_119671_object44199:head# 2024-03-20T09:05:00.655 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.655 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdce8e64:::benchmark_data_smithi122_119671_object69044:head# 2024-03-20T09:05:00.655 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.655 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdcf5e68:::benchmark_data_smithi122_119671_object99272:head# 2024-03-20T09:05:00.655 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.655 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdd090d9:::benchmark_data_smithi122_119671_object21636:head# 2024-03-20T09:05:00.655 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.655 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdd23b0b:::benchmark_data_smithi122_119671_object29130:head# 2024-03-20T09:05:00.655 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.655 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdd24da3:::benchmark_data_smithi122_119671_object57664:head# 2024-03-20T09:05:00.655 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.655 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdd393c7:::benchmark_data_smithi122_119671_object32083:head# 2024-03-20T09:05:00.656 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.656 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdd3c564:::benchmark_data_smithi122_119671_object62532:head# 2024-03-20T09:05:00.656 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.656 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdd3e37c:::benchmark_data_smithi122_119671_object91755:head# 2024-03-20T09:05:00.656 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.656 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdd4e98b:::benchmark_data_smithi122_119671_object75750:head# 2024-03-20T09:05:00.656 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.656 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdd598a6:::benchmark_data_smithi122_119671_object63068:head# 2024-03-20T09:05:00.656 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.656 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdd64ffb:::benchmark_data_smithi122_119671_object17262:head# 2024-03-20T09:05:00.656 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.656 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdd66933:::benchmark_data_smithi122_119671_object36092:head# 2024-03-20T09:05:00.656 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.656 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdd6a5a6:::benchmark_data_smithi122_119671_object91285:head# 2024-03-20T09:05:00.657 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.657 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdd8b543:::benchmark_data_smithi122_119671_object16702:head# 2024-03-20T09:05:00.657 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.657 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bddb4abf:::benchmark_data_smithi122_119671_object67837:head# 2024-03-20T09:05:00.657 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.657 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bddc1944:::benchmark_data_smithi122_119671_object43482:head# 2024-03-20T09:05:00.657 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.657 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bddc98bb:::benchmark_data_smithi122_119671_object53812:head# 2024-03-20T09:05:00.657 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.657 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bddd675d:::benchmark_data_smithi122_119671_object18963:head# 2024-03-20T09:05:00.657 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.657 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdde5fe6:::benchmark_data_smithi122_119671_object8647:head# 2024-03-20T09:05:00.657 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.657 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bddee6ff:::benchmark_data_smithi122_119671_object52086:head# 2024-03-20T09:05:00.658 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.658 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bddf79cb:::benchmark_data_smithi122_119671_object4212:head# 2024-03-20T09:05:00.658 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.658 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bde0d7e8:::benchmark_data_smithi122_119671_object31157:head# 2024-03-20T09:05:00.658 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.658 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bde1bc17:::benchmark_data_smithi122_119671_object61205:head# 2024-03-20T09:05:00.658 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.658 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bde274b4:::benchmark_data_smithi122_119671_object4129:head# 2024-03-20T09:05:00.658 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.658 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bde28b44:::benchmark_data_smithi122_119671_object86207:head# 2024-03-20T09:05:00.658 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.658 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bde2b885:::benchmark_data_smithi122_119671_object86854:head# 2024-03-20T09:05:00.658 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.658 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bde2b917:::benchmark_data_smithi122_119671_object47486:head# 2024-03-20T09:05:00.659 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.659 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bde4c5f5:::benchmark_data_smithi122_119671_object77223:head# 2024-03-20T09:05:00.659 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.659 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bde519cf:::benchmark_data_smithi122_119671_object65342:head# 2024-03-20T09:05:00.659 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.659 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bde67528:::benchmark_data_smithi122_119671_object28824:head# 2024-03-20T09:05:00.659 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.659 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bde6ee9d:::benchmark_data_smithi122_119671_object48245:head# 2024-03-20T09:05:00.659 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.659 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bde753b6:::benchmark_data_smithi122_119671_object91970:head# 2024-03-20T09:05:00.660 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.660 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bde798cc:::benchmark_data_smithi122_119671_object17446:head# 2024-03-20T09:05:00.660 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.660 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bde8d5e0:::benchmark_data_smithi122_119671_object86346:head# 2024-03-20T09:05:00.660 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.660 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bde8f6e3:::benchmark_data_smithi122_119671_object31202:head# 2024-03-20T09:05:00.660 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.660 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bde9227a:::benchmark_data_smithi122_119671_object12569:head# 2024-03-20T09:05:00.660 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.660 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bde95cd5:::benchmark_data_smithi122_119671_object7279:head# 2024-03-20T09:05:00.660 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.660 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bde9d4ec:::benchmark_data_smithi122_119671_object37320:head# 2024-03-20T09:05:00.660 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.660 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdea4db6:::benchmark_data_smithi122_119671_object83130:head# 2024-03-20T09:05:00.661 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.661 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdea8b5e:::benchmark_data_smithi122_119671_object80088:head# 2024-03-20T09:05:00.661 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.661 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdeb470e:::benchmark_data_smithi122_119671_object70564:head# 2024-03-20T09:05:00.661 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.661 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdec77ff:::benchmark_data_smithi122_119671_object91753:head# 2024-03-20T09:05:00.661 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.661 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdecfa36:::benchmark_data_smithi122_119671_object25373:head# 2024-03-20T09:05:00.661 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.661 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bded3936:::benchmark_data_smithi122_119671_object55479:head# 2024-03-20T09:05:00.661 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.661 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdf07ac6:::benchmark_data_smithi122_119671_object79514:head# 2024-03-20T09:05:00.661 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.661 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdf10ad4:::benchmark_data_smithi122_119671_object99362:head# 2024-03-20T09:05:00.661 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.662 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdf1f216:::benchmark_data_smithi122_119671_object22652:head# 2024-03-20T09:05:00.662 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.662 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdf60762:::benchmark_data_smithi122_119671_object94600:head# 2024-03-20T09:05:00.662 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.662 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdf7a00d:::benchmark_data_smithi122_119671_object82136:head# 2024-03-20T09:05:00.662 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.662 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdf86af4:::benchmark_data_smithi122_119671_object63022:head# 2024-03-20T09:05:00.662 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.662 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdf908e3:::benchmark_data_smithi122_119671_object66493:head# 2024-03-20T09:05:00.662 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.662 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdf94adc:::benchmark_data_smithi122_119671_object63551:head# 2024-03-20T09:05:00.662 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.662 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdf9ceca:::benchmark_data_smithi122_119671_object68943:head# 2024-03-20T09:05:00.662 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.663 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdf9f010:::benchmark_data_smithi122_119671_object92765:head# 2024-03-20T09:05:00.663 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.663 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdfcdf0f:::benchmark_data_smithi122_119671_object55869:head# 2024-03-20T09:05:00.663 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.663 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdfead9f:::benchmark_data_smithi122_119671_object29573:head# 2024-03-20T09:05:00.663 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.663 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdfecbcf:::benchmark_data_smithi122_119671_object72246:head# 2024-03-20T09:05:00.663 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.663 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdfed21e:::benchmark_data_smithi122_119671_object30222:head# 2024-03-20T09:05:00.663 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.663 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdff069c:::benchmark_data_smithi122_119671_object69040:head# 2024-03-20T09:05:00.663 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.663 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdff08cd:::benchmark_data_smithi122_119671_object99299:head# 2024-03-20T09:05:00.664 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.664 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdff8744:::benchmark_data_smithi122_119671_object33286:head# 2024-03-20T09:05:00.664 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.664 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bdffb1f6:::benchmark_data_smithi122_119671_object58627:head# 2024-03-20T09:05:00.664 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.664 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be001b84:::benchmark_data_smithi122_119671_object12341:head# 2024-03-20T09:05:00.664 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.664 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be013106:::benchmark_data_smithi122_119671_object15341:head# 2024-03-20T09:05:00.664 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.664 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be013c8d:::benchmark_data_smithi122_119671_object8090:head# 2024-03-20T09:05:00.664 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.664 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be018b62:::benchmark_data_smithi122_119671_object31418:head# 2024-03-20T09:05:00.665 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.665 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be0350cd:::benchmark_data_smithi122_119671_object74559:head# 2024-03-20T09:05:00.665 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.665 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be035272:::benchmark_data_smithi122_119671_object11414:head# 2024-03-20T09:05:00.665 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.665 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be03e0b4:::benchmark_data_smithi122_119671_object48614:head# 2024-03-20T09:05:00.665 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.665 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be04fa16:::benchmark_data_smithi122_119671_object70303:head# 2024-03-20T09:05:00.665 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.665 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be0517e0:::benchmark_data_smithi122_119671_object56147:head# 2024-03-20T09:05:00.665 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.665 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be055e04:::benchmark_data_smithi122_119671_object27118:head# 2024-03-20T09:05:00.665 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.665 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be0582a0:::benchmark_data_smithi122_119671_object79000:head# 2024-03-20T09:05:00.666 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.666 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be063a86:::benchmark_data_smithi122_119671_object87772:head# 2024-03-20T09:05:00.666 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.666 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be0681c8:::benchmark_data_smithi122_119671_object78417:head# 2024-03-20T09:05:00.666 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.666 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be068464:::benchmark_data_smithi122_119671_object7864:head# 2024-03-20T09:05:00.666 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.666 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be069960:::benchmark_data_smithi122_119671_object20926:head# 2024-03-20T09:05:00.666 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.666 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be0bff7c:::benchmark_data_smithi122_119671_object52397:head# 2024-03-20T09:05:00.666 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.666 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be0dbbd5:::benchmark_data_smithi122_119671_object67121:head# 2024-03-20T09:05:00.666 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.666 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be0ebc9f:::benchmark_data_smithi122_119671_object68828:head# 2024-03-20T09:05:00.667 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.667 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be0fc01b:::benchmark_data_smithi122_119671_object42844:head# 2024-03-20T09:05:00.667 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.667 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be0ffc57:::benchmark_data_smithi122_119671_object3219:head# 2024-03-20T09:05:00.667 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.667 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be112ddd:::benchmark_data_smithi122_119671_object6379:head# 2024-03-20T09:05:00.667 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.667 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be11d8b1:::benchmark_data_smithi122_119671_object65976:head# 2024-03-20T09:05:00.667 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.667 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be129286:::benchmark_data_smithi122_119671_object52483:head# 2024-03-20T09:05:00.667 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.667 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be135856:::benchmark_data_smithi122_119671_object31283:head# 2024-03-20T09:05:00.667 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.667 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be144b0a:::benchmark_data_smithi122_119671_object77181:head# 2024-03-20T09:05:00.667 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.668 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be14a0fc:::benchmark_data_smithi122_119671_object98622:head# 2024-03-20T09:05:00.668 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.668 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be14d450:::benchmark_data_smithi122_119671_object48113:head# 2024-03-20T09:05:00.668 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.668 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be15a54b:::benchmark_data_smithi122_119671_object48129:head# 2024-03-20T09:05:00.668 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.668 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be16aa7c:::benchmark_data_smithi122_119671_object23840:head# 2024-03-20T09:05:00.668 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.668 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be18955f:::benchmark_data_smithi122_119671_object46214:head# 2024-03-20T09:05:00.668 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.668 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be19656d:::benchmark_data_smithi122_119671_object2832:head# 2024-03-20T09:05:00.668 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.668 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be1983e9:::benchmark_data_smithi122_119671_object32522:head# 2024-03-20T09:05:00.668 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.669 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be19ebf2:::benchmark_data_smithi122_119671_object98681:head# 2024-03-20T09:05:00.669 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.669 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be1a58c1:::benchmark_data_smithi122_119671_object84054:head# 2024-03-20T09:05:00.669 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.669 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be1abc2f:::benchmark_data_smithi122_119671_object71355:head# 2024-03-20T09:05:00.669 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.669 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be1b76e3:::benchmark_data_smithi122_119671_object11377:head# 2024-03-20T09:05:00.669 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.669 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be1cccd0:::benchmark_data_smithi122_119671_object74287:head# 2024-03-20T09:05:00.669 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.669 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be1d16f0:::benchmark_data_smithi122_119671_object31819:head# 2024-03-20T09:05:00.669 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.669 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be1d44fd:::benchmark_data_smithi122_119671_object12214:head# 2024-03-20T09:05:00.669 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.669 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be1d8d5c:::benchmark_data_smithi122_119671_object66822:head# 2024-03-20T09:05:00.670 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.670 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be1db8e2:::benchmark_data_smithi122_119671_object76436:head# 2024-03-20T09:05:00.670 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.670 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be1e46f3:::benchmark_data_smithi122_119671_object62461:head# 2024-03-20T09:05:00.670 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.670 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be1e6e2b:::benchmark_data_smithi122_119671_object83352:head# 2024-03-20T09:05:00.670 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.670 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be1f8d74:::benchmark_data_smithi122_119671_object10746:head# 2024-03-20T09:05:00.670 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.670 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be1fb54c:::benchmark_data_smithi122_119671_object79195:head# 2024-03-20T09:05:00.670 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.670 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be2162ad:::benchmark_data_smithi122_119671_object53742:head# 2024-03-20T09:05:00.670 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.670 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be225bbd:::benchmark_data_smithi122_119671_object71618:head# 2024-03-20T09:05:00.671 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.671 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be2314bf:::benchmark_data_smithi122_119671_object77355:head# 2024-03-20T09:05:00.671 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.671 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be236354:::benchmark_data_smithi122_119671_object65021:head# 2024-03-20T09:05:00.671 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.671 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be254684:::benchmark_data_smithi122_119671_object17680:head# 2024-03-20T09:05:00.671 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.671 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be260554:::benchmark_data_smithi122_119671_object70639:head# 2024-03-20T09:05:00.671 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.671 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be26248c:::benchmark_data_smithi122_119671_object39706:head# 2024-03-20T09:05:00.671 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.671 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be264c02:::benchmark_data_smithi122_119671_object70741:head# 2024-03-20T09:05:00.671 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.671 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be264dc6:::benchmark_data_smithi122_119671_object48437:head# 2024-03-20T09:05:00.672 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.672 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be277b1e:::benchmark_data_smithi122_119671_object76668:head# 2024-03-20T09:05:00.672 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.672 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be289406:::benchmark_data_smithi122_119671_object91721:head# 2024-03-20T09:05:00.672 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.672 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be289af0:::benchmark_data_smithi122_119671_object81646:head# 2024-03-20T09:05:00.672 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.672 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be28a06a:::benchmark_data_smithi122_119671_object90757:head# 2024-03-20T09:05:00.672 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.672 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be292bf5:::benchmark_data_smithi122_119671_object49991:head# 2024-03-20T09:05:00.672 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.672 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be2aad4f:::benchmark_data_smithi122_119671_object14289:head# 2024-03-20T09:05:00.672 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.672 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be2c06de:::benchmark_data_smithi122_119671_object7209:head# 2024-03-20T09:05:00.673 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.673 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be2c2d58:::benchmark_data_smithi122_119671_object92893:head# 2024-03-20T09:05:00.673 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.673 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be2dc89c:::benchmark_data_smithi122_119671_object48828:head# 2024-03-20T09:05:00.673 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.673 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be2de023:::benchmark_data_smithi122_119671_object63608:head# 2024-03-20T09:05:00.673 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.673 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be2e009b:::benchmark_data_smithi122_119671_object31220:head# 2024-03-20T09:05:00.673 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.673 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be2ea2ac:::benchmark_data_smithi122_119671_object7865:head# 2024-03-20T09:05:00.673 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.673 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be2f41e9:::benchmark_data_smithi122_119671_object97092:head# 2024-03-20T09:05:00.673 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.673 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be30d369:::benchmark_data_smithi122_119671_object43243:head# 2024-03-20T09:05:00.674 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.674 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be321431:::benchmark_data_smithi122_119671_object81595:head# 2024-03-20T09:05:00.674 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.674 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be32a3fe:::benchmark_data_smithi122_119671_object41546:head# 2024-03-20T09:05:00.674 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.674 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be333b59:::benchmark_data_smithi122_119671_object84533:head# 2024-03-20T09:05:00.674 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.674 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be33d61d:::benchmark_data_smithi122_119671_object35279:head# 2024-03-20T09:05:00.674 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.674 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be37b333:::benchmark_data_smithi122_119671_object57624:head# 2024-03-20T09:05:00.674 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.674 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be3804f5:::benchmark_data_smithi122_119671_object47909:head# 2024-03-20T09:05:00.674 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.675 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be3930c1:::benchmark_data_smithi122_119671_object16810:head# 2024-03-20T09:05:00.675 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.675 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be3a04c2:::benchmark_data_smithi122_119671_object13826:head# 2024-03-20T09:05:00.675 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.675 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be3a5c58:::benchmark_data_smithi122_119671_object32174:head# 2024-03-20T09:05:00.675 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.675 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be3c8a1b:::benchmark_data_smithi122_119671_object79856:head# 2024-03-20T09:05:00.675 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.675 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be3c9a80:::benchmark_data_smithi122_119671_object28687:head# 2024-03-20T09:05:00.675 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.675 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be3ddbef:::benchmark_data_smithi122_119671_object76154:head# 2024-03-20T09:05:00.675 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.675 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be3e269f:::benchmark_data_smithi122_119671_object31549:head# 2024-03-20T09:05:00.676 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.676 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be3f1677:::benchmark_data_smithi122_119671_object97776:head# 2024-03-20T09:05:00.676 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.676 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be3f83d0:::benchmark_data_smithi122_119671_object61215:head# 2024-03-20T09:05:00.676 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.676 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be405317:::benchmark_data_smithi122_119671_object57778:head# 2024-03-20T09:05:00.676 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.676 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be412ba6:::benchmark_data_smithi122_119671_object91720:head# 2024-03-20T09:05:00.676 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.676 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be41cb94:::benchmark_data_smithi122_119671_object41179:head# 2024-03-20T09:05:00.676 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.676 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be4216da:::benchmark_data_smithi122_119671_object26589:head# 2024-03-20T09:05:00.676 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.676 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be42a559:::benchmark_data_smithi122_119671_object13541:head# 2024-03-20T09:05:00.677 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.677 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be42f0a4:::benchmark_data_smithi122_119671_object62808:head# 2024-03-20T09:05:00.677 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.677 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be442bc6:::benchmark_data_smithi122_119671_object94898:head# 2024-03-20T09:05:00.677 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.677 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be4473a8:::benchmark_data_smithi122_119671_object73295:head# 2024-03-20T09:05:00.677 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.677 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be44d8db:::benchmark_data_smithi122_119671_object85926:head# 2024-03-20T09:05:00.677 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.677 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be45d436:::benchmark_data_smithi122_119671_object91692:head# 2024-03-20T09:05:00.677 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.677 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be4626d5:::benchmark_data_smithi122_119671_object98811:head# 2024-03-20T09:05:00.678 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.678 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be467cec:::benchmark_data_smithi122_119671_object7981:head# 2024-03-20T09:05:00.678 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.678 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be469ab1:::benchmark_data_smithi122_119671_object3216:head# 2024-03-20T09:05:00.678 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.678 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be47e083:::benchmark_data_smithi122_119671_object21162:head# 2024-03-20T09:05:00.678 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.678 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be47f0a8:::benchmark_data_smithi122_119671_object9731:head# 2024-03-20T09:05:00.678 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.678 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be483333:::benchmark_data_smithi122_119671_object33730:head# 2024-03-20T09:05:00.678 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.678 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be484b6c:::benchmark_data_smithi122_119671_object48671:head# 2024-03-20T09:05:00.678 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.679 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be485775:::benchmark_data_smithi122_119671_object29307:head# 2024-03-20T09:05:00.679 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.679 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be489887:::benchmark_data_smithi122_119671_object45072:head# 2024-03-20T09:05:00.679 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.679 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be48b879:::benchmark_data_smithi122_119671_object87315:head# 2024-03-20T09:05:00.679 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.679 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be48c126:::benchmark_data_smithi122_119671_object80675:head# 2024-03-20T09:05:00.679 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.679 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be490d6e:::benchmark_data_smithi122_119671_object33331:head# 2024-03-20T09:05:00.679 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.679 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be4d1d30:::benchmark_data_smithi122_119671_object88960:head# 2024-03-20T09:05:00.679 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.679 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be4e5b82:::benchmark_data_smithi122_119671_object80319:head# 2024-03-20T09:05:00.679 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.680 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be4e92ad:::benchmark_data_smithi122_119671_object56994:head# 2024-03-20T09:05:00.680 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.680 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be4fd133:::benchmark_data_smithi122_119671_object66177:head# 2024-03-20T09:05:00.680 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.680 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be50c70a:::benchmark_data_smithi122_119671_object74508:head# 2024-03-20T09:05:00.680 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.680 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be51288d:::benchmark_data_smithi122_119671_object93410:head# 2024-03-20T09:05:00.680 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.680 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be51f8fd:::benchmark_data_smithi122_119671_object73252:head# 2024-03-20T09:05:00.680 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.680 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be52378a:::benchmark_data_smithi122_119671_object4265:head# 2024-03-20T09:05:00.680 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.680 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be52621f:::benchmark_data_smithi122_119671_object6868:head# 2024-03-20T09:05:00.680 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.681 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be52ce09:::benchmark_data_smithi122_119671_object71902:head# 2024-03-20T09:05:00.681 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.681 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be52fa9b:::benchmark_data_smithi122_119671_object56798:head# 2024-03-20T09:05:00.681 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.681 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be538f90:::benchmark_data_smithi122_119671_object26533:head# 2024-03-20T09:05:00.681 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.681 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be53aa52:::benchmark_data_smithi122_119671_object33799:head# 2024-03-20T09:05:00.681 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.681 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be54bc67:::benchmark_data_smithi122_119671_object41641:head# 2024-03-20T09:05:00.681 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.681 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be54d2f6:::benchmark_data_smithi122_119671_object62125:head# 2024-03-20T09:05:00.681 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.681 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be57211b:::benchmark_data_smithi122_119671_object60892:head# 2024-03-20T09:05:00.682 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.682 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be57a6b4:::benchmark_data_smithi122_119671_object78631:head# 2024-03-20T09:05:00.682 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.682 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be586ebd:::benchmark_data_smithi122_119671_object19457:head# 2024-03-20T09:05:00.682 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.682 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be58fa3b:::benchmark_data_smithi122_119671_object14710:head# 2024-03-20T09:05:00.682 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.682 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be59aa7f:::benchmark_data_smithi122_119671_object47498:head# 2024-03-20T09:05:00.682 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.682 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be5b3f24:::benchmark_data_smithi122_119671_object23347:head# 2024-03-20T09:05:00.682 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.682 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be5bda0c:::benchmark_data_smithi122_119671_object89096:head# 2024-03-20T09:05:00.682 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.682 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be5bf7aa:::benchmark_data_smithi122_119671_object32464:head# 2024-03-20T09:05:00.683 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.683 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be5c043f:::benchmark_data_smithi122_119671_object60383:head# 2024-03-20T09:05:00.683 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.683 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be5c22ef:::benchmark_data_smithi122_119671_object18085:head# 2024-03-20T09:05:00.683 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.683 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be5cc4af:::benchmark_data_smithi122_119671_object73658:head# 2024-03-20T09:05:00.683 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.683 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be5f2589:::benchmark_data_smithi122_119671_object28963:head# 2024-03-20T09:05:00.683 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.683 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be5febc4:::benchmark_data_smithi122_119671_object39442:head# 2024-03-20T09:05:00.683 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.683 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be5fff33:::benchmark_data_smithi122_119671_object69225:head# 2024-03-20T09:05:00.683 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.683 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be607714:::benchmark_data_smithi122_119671_object34030:head# 2024-03-20T09:05:00.684 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.684 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be60dacb:::benchmark_data_smithi122_119671_object54811:head# 2024-03-20T09:05:00.684 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.684 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be60f0d0:::benchmark_data_smithi122_119671_object86740:head# 2024-03-20T09:05:00.684 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.684 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be615398:::benchmark_data_smithi122_119671_object30774:head# 2024-03-20T09:05:00.684 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.684 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be61fcaf:::benchmark_data_smithi122_119671_object12404:head# 2024-03-20T09:05:00.684 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.684 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be621810:::benchmark_data_smithi122_119671_object79431:head# 2024-03-20T09:05:00.684 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.684 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be624c53:::benchmark_data_smithi122_119671_object70676:head# 2024-03-20T09:05:00.684 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.685 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be633ef5:::benchmark_data_smithi122_119671_object75397:head# 2024-03-20T09:05:00.685 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.685 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be639423:::benchmark_data_smithi122_119671_object14571:head# 2024-03-20T09:05:00.685 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.685 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be639f18:::benchmark_data_smithi122_119671_object17434:head# 2024-03-20T09:05:00.685 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.685 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be63f244:::benchmark_data_smithi122_119671_object53221:head# 2024-03-20T09:05:00.685 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.685 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be64eda6:::benchmark_data_smithi122_119671_object48858:head# 2024-03-20T09:05:00.685 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.685 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be6600f8:::benchmark_data_smithi122_119671_object38818:head# 2024-03-20T09:05:00.685 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.685 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be660a33:::benchmark_data_smithi122_119671_object1428:head# 2024-03-20T09:05:00.686 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.686 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be66b12e:::benchmark_data_smithi122_119671_object78080:head# 2024-03-20T09:05:00.686 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.686 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be673701:::benchmark_data_smithi122_119671_object64583:head# 2024-03-20T09:05:00.686 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.686 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be689dcc:::benchmark_data_smithi122_119671_object65486:head# 2024-03-20T09:05:00.686 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.686 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be68a8fa:::benchmark_data_smithi122_119671_object8584:head# 2024-03-20T09:05:00.686 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.686 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be68f402:::benchmark_data_smithi122_119671_object70052:head# 2024-03-20T09:05:00.686 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.686 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be6921a9:::benchmark_data_smithi122_119671_object81289:head# 2024-03-20T09:05:00.686 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.687 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be6a34a0:::benchmark_data_smithi122_119671_object56777:head# 2024-03-20T09:05:00.687 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.687 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be6a3e20:::benchmark_data_smithi122_119671_object29324:head# 2024-03-20T09:05:00.687 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.687 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be6aedaa:::benchmark_data_smithi122_119671_object21209:head# 2024-03-20T09:05:00.687 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.687 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be6b2c1b:::benchmark_data_smithi122_119671_object76671:head# 2024-03-20T09:05:00.687 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.687 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be6bcf55:::benchmark_data_smithi122_119671_object62732:head# 2024-03-20T09:05:00.687 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.687 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be6d64e4:::benchmark_data_smithi122_119671_object88780:head# 2024-03-20T09:05:00.687 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.687 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be6d6838:::benchmark_data_smithi122_119671_object75416:head# 2024-03-20T09:05:00.687 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.688 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be6da178:::benchmark_data_smithi122_119671_object21088:head# 2024-03-20T09:05:00.688 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.688 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be6df2b5:::benchmark_data_smithi122_119671_object6652:head# 2024-03-20T09:05:00.688 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.688 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be6e5546:::benchmark_data_smithi122_119671_object97705:head# 2024-03-20T09:05:00.688 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.688 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be6e6e9e:::benchmark_data_smithi122_119671_object2879:head# 2024-03-20T09:05:00.688 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.688 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be6f57ad:::benchmark_data_smithi122_119671_object62872:head# 2024-03-20T09:05:00.688 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.688 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be6f8ce1:::benchmark_data_smithi122_119671_object84780:head# 2024-03-20T09:05:00.688 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.689 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be6fb5ac:::benchmark_data_smithi122_119671_object69185:head# 2024-03-20T09:05:00.689 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.689 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be70123e:::benchmark_data_smithi122_119671_object41228:head# 2024-03-20T09:05:00.689 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.689 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be70f62d:::benchmark_data_smithi122_119671_object32877:head# 2024-03-20T09:05:00.689 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.689 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be714846:::benchmark_data_smithi122_119671_object55496:head# 2024-03-20T09:05:00.689 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.689 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be7154c8:::benchmark_data_smithi122_119671_object90532:head# 2024-03-20T09:05:00.689 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.690 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be7177c0:::benchmark_data_smithi122_119671_object83843:head# 2024-03-20T09:05:00.690 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.690 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be71f07b:::benchmark_data_smithi122_119671_object52185:head# 2024-03-20T09:05:00.690 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.690 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be71fdfc:::benchmark_data_smithi122_119671_object87013:head# 2024-03-20T09:05:00.690 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.690 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be72cbe3:::benchmark_data_smithi122_119671_object16447:head# 2024-03-20T09:05:00.690 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.690 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be73f453:::benchmark_data_smithi122_119671_object89351:head# 2024-03-20T09:05:00.690 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.690 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be74634c:::benchmark_data_smithi122_119671_object53803:head# 2024-03-20T09:05:00.690 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.690 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be785b8f:::benchmark_data_smithi122_119671_object45086:head# 2024-03-20T09:05:00.690 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.691 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be78ad3e:::benchmark_data_smithi122_119671_object92087:head# 2024-03-20T09:05:00.691 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.691 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be78c785:::benchmark_data_smithi122_119671_object78068:head# 2024-03-20T09:05:00.691 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.691 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be799ef5:::benchmark_data_smithi122_119671_object35719:head# 2024-03-20T09:05:00.691 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.691 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be79b8a5:::benchmark_data_smithi122_119671_object87522:head# 2024-03-20T09:05:00.691 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.691 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be79cf38:::benchmark_data_smithi122_119671_object3714:head# 2024-03-20T09:05:00.691 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.691 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be79ddd6:::benchmark_data_smithi122_119671_object75895:head# 2024-03-20T09:05:00.691 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.691 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be7abae1:::benchmark_data_smithi122_119671_object74715:head# 2024-03-20T09:05:00.691 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.692 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be7c27a6:::benchmark_data_smithi122_119671_object94662:head# 2024-03-20T09:05:00.692 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.692 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be7da385:::benchmark_data_smithi122_119671_object76557:head# 2024-03-20T09:05:00.692 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.692 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be7dc485:::benchmark_data_smithi122_119671_object30120:head# 2024-03-20T09:05:00.692 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.692 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be7f6ca0:::benchmark_data_smithi122_119671_object12785:head# 2024-03-20T09:05:00.692 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.692 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be7f991c:::benchmark_data_smithi122_119671_object12658:head# 2024-03-20T09:05:00.692 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.692 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be804bbe:::benchmark_data_smithi122_119671_object84767:head# 2024-03-20T09:05:00.692 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.692 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be807993:::benchmark_data_smithi122_119671_object6328:head# 2024-03-20T09:05:00.692 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.692 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be825287:::benchmark_data_smithi122_119671_object20193:head# 2024-03-20T09:05:00.693 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.693 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be827e9c:::benchmark_data_smithi122_119671_object93807:head# 2024-03-20T09:05:00.693 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.693 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be83eac3:::benchmark_data_smithi122_119671_object15349:head# 2024-03-20T09:05:00.693 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.693 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be844c9e:::benchmark_data_smithi122_119671_object33027:head# 2024-03-20T09:05:00.693 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.693 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be8480f3:::benchmark_data_smithi122_119671_object21308:head# 2024-03-20T09:05:00.693 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.693 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be84acc1:::benchmark_data_smithi122_119671_object86331:head# 2024-03-20T09:05:00.693 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.693 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be84b0be:::benchmark_data_smithi122_119671_object35673:head# 2024-03-20T09:05:00.693 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.693 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be850f90:::benchmark_data_smithi122_119671_object77988:head# 2024-03-20T09:05:00.694 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.694 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be8521d1:::benchmark_data_smithi122_119671_object56097:head# 2024-03-20T09:05:00.694 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.694 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be85694d:::benchmark_data_smithi122_119671_object92794:head# 2024-03-20T09:05:00.694 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.694 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be858790:::benchmark_data_smithi122_119671_object94015:head# 2024-03-20T09:05:00.694 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.694 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be8601b3:::benchmark_data_smithi122_119671_object39128:head# 2024-03-20T09:05:00.694 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.694 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be860524:::benchmark_data_smithi122_119671_object37124:head# 2024-03-20T09:05:00.694 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.695 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be868b3e:::benchmark_data_smithi122_119671_object16672:head# 2024-03-20T09:05:00.695 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.695 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be875c30:::benchmark_data_smithi122_119671_object19723:head# 2024-03-20T09:05:00.695 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.695 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be878d84:::benchmark_data_smithi122_119671_object37350:head# 2024-03-20T09:05:00.695 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.695 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be87b34e:::benchmark_data_smithi122_119671_object15973:head# 2024-03-20T09:05:00.695 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.695 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be882b1f:::benchmark_data_smithi122_119671_object91434:head# 2024-03-20T09:05:00.695 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.695 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be8856d1:::benchmark_data_smithi122_119671_object87608:head# 2024-03-20T09:05:00.695 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.695 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be89fcf3:::benchmark_data_smithi122_119671_object98302:head# 2024-03-20T09:05:00.695 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.696 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be8a50fa:::benchmark_data_smithi122_119671_object42945:head# 2024-03-20T09:05:00.696 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.696 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be8a8b21:::benchmark_data_smithi122_119671_object45028:head# 2024-03-20T09:05:00.696 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.696 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be8c066e:::benchmark_data_smithi122_119671_object15698:head# 2024-03-20T09:05:00.696 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.696 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be8cc2a3:::benchmark_data_smithi122_119671_object68519:head# 2024-03-20T09:05:00.696 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.696 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be8dec1d:::benchmark_data_smithi122_119671_object16845:head# 2024-03-20T09:05:00.696 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.696 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be8e86ab:::benchmark_data_smithi122_119671_object84964:head# 2024-03-20T09:05:00.696 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.696 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be8ef1db:::benchmark_data_smithi122_119671_object26366:head# 2024-03-20T09:05:00.696 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.697 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be8f0744:::benchmark_data_smithi122_119671_object41871:head# 2024-03-20T09:05:00.697 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.697 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be8f1ac9:::benchmark_data_smithi122_119671_object3296:head# 2024-03-20T09:05:00.697 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.697 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be9076d1:::benchmark_data_smithi122_119671_object26293:head# 2024-03-20T09:05:00.697 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.697 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be90775a:::benchmark_data_smithi122_119671_object41630:head# 2024-03-20T09:05:00.697 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.697 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be9155b6:::benchmark_data_smithi122_119671_object65331:head# 2024-03-20T09:05:00.697 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.697 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be9209a5:::benchmark_data_smithi122_119671_object24743:head# 2024-03-20T09:05:00.697 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.697 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be934c58:::benchmark_data_smithi122_119671_object11878:head# 2024-03-20T09:05:00.697 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.697 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be93e312:::benchmark_data_smithi122_119671_object21301:head# 2024-03-20T09:05:00.698 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.698 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be944019:::benchmark_data_smithi122_119671_object81597:head# 2024-03-20T09:05:00.698 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.698 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be94c698:::benchmark_data_smithi122_119671_object21889:head# 2024-03-20T09:05:00.698 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.698 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be94d538:::benchmark_data_smithi122_119671_object10372:head# 2024-03-20T09:05:00.698 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.698 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be94eb45:::benchmark_data_smithi122_119671_object18024:head# 2024-03-20T09:05:00.698 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.698 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be94ee7f:::benchmark_data_smithi122_119671_object14509:head# 2024-03-20T09:05:00.698 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.698 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be951e60:::benchmark_data_smithi122_119671_object79790:head# 2024-03-20T09:05:00.698 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.699 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be952a26:::benchmark_data_smithi122_119671_object85661:head# 2024-03-20T09:05:00.699 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.699 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be980620:::benchmark_data_smithi122_119671_object42781:head# 2024-03-20T09:05:00.699 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.699 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be981512:::benchmark_data_smithi122_119671_object93959:head# 2024-03-20T09:05:00.699 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.699 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be9ac3bf:::benchmark_data_smithi122_119671_object37482:head# 2024-03-20T09:05:00.699 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.699 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be9c734e:::benchmark_data_smithi122_119671_object49150:head# 2024-03-20T09:05:00.699 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.699 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be9da27e:::benchmark_data_smithi122_119671_object73607:head# 2024-03-20T09:05:00.699 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.699 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be9e2e0b:::benchmark_data_smithi122_119671_object46712:head# 2024-03-20T09:05:00.700 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.700 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be9eff78:::benchmark_data_smithi122_119671_object17783:head# 2024-03-20T09:05:00.700 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.700 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:be9f5588:::benchmark_data_smithi122_119671_object12610:head# 2024-03-20T09:05:00.700 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.700 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bea0327f:::benchmark_data_smithi122_119671_object76487:head# 2024-03-20T09:05:00.700 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.700 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bea0337a:::benchmark_data_smithi122_119671_object85287:head# 2024-03-20T09:05:00.700 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.700 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bea03779:::benchmark_data_smithi122_119671_object93388:head# 2024-03-20T09:05:00.700 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.700 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bea102b7:::benchmark_data_smithi122_119671_object39811:head# 2024-03-20T09:05:00.701 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.701 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bea11d86:::benchmark_data_smithi122_119671_object19897:head# 2024-03-20T09:05:00.701 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.701 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bea22050:::benchmark_data_smithi122_119671_object83490:head# 2024-03-20T09:05:00.701 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.701 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bea2d1c4:::benchmark_data_smithi122_119671_object33785:head# 2024-03-20T09:05:00.701 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.701 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bea3333a:::benchmark_data_smithi122_119671_object63435:head# 2024-03-20T09:05:00.701 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.701 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bea33b66:::benchmark_data_smithi122_119671_object38503:head# 2024-03-20T09:05:00.701 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.701 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bea4160b:::benchmark_data_smithi122_119671_object32823:head# 2024-03-20T09:05:00.702 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.702 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bea49e7c:::benchmark_data_smithi122_119671_object38908:head# 2024-03-20T09:05:00.702 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.702 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bea4a193:::benchmark_data_smithi122_119671_object6556:head# 2024-03-20T09:05:00.702 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.702 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bea61917:::benchmark_data_smithi122_119671_object8858:head# 2024-03-20T09:05:00.702 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.702 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bea64b2b:::benchmark_data_smithi122_119671_object90404:head# 2024-03-20T09:05:00.702 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.702 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bea65069:::benchmark_data_smithi122_119671_object47239:head# 2024-03-20T09:05:00.702 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.702 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bea6c166:::benchmark_data_smithi122_119671_object46698:head# 2024-03-20T09:05:00.702 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.703 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bea6dc03:::benchmark_data_smithi122_119671_object98858:head# 2024-03-20T09:05:00.703 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.703 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bea80076:::benchmark_data_smithi122_119671_object15475:head# 2024-03-20T09:05:00.703 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.703 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bea8ec1e:::benchmark_data_smithi122_119671_object73454:head# 2024-03-20T09:05:00.703 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.703 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:beaa36da:::benchmark_data_smithi122_119671_object27940:head# 2024-03-20T09:05:00.703 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.703 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:beaa6033:::benchmark_data_smithi122_119671_object2292:head# 2024-03-20T09:05:00.703 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.703 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:beacb9b1:::benchmark_data_smithi122_119671_object94664:head# 2024-03-20T09:05:00.703 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.703 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bead82e5:::benchmark_data_smithi122_119671_object5946:head# 2024-03-20T09:05:00.703 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.704 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:beb06093:::benchmark_data_smithi122_119671_object2822:head# 2024-03-20T09:05:00.704 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.704 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:beb0827a:::benchmark_data_smithi122_119671_object70874:head# 2024-03-20T09:05:00.704 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.704 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:beb0b7a6:::benchmark_data_smithi122_119671_object5809:head# 2024-03-20T09:05:00.704 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.704 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:beb0d3b0:::benchmark_data_smithi122_119671_object9457:head# 2024-03-20T09:05:00.704 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.704 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:beb0f204:::benchmark_data_smithi122_119671_object37661:head# 2024-03-20T09:05:00.704 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.704 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:beb2331c:::benchmark_data_smithi122_119671_object76310:head# 2024-03-20T09:05:00.704 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.704 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:beb2c953:::benchmark_data_smithi122_119671_object57405:head# 2024-03-20T09:05:00.704 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.704 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:beb2e08e:::benchmark_data_smithi122_119671_object9216:head# 2024-03-20T09:05:00.705 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.705 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:beb338de:::benchmark_data_smithi122_119671_object6425:head# 2024-03-20T09:05:00.705 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.705 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:beb3e54f:::benchmark_data_smithi122_119671_object18011:head# 2024-03-20T09:05:00.705 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.705 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:beb48688:::benchmark_data_smithi122_119671_object23058:head# 2024-03-20T09:05:00.705 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.705 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:beb4c6c5:::benchmark_data_smithi122_119671_object19623:head# 2024-03-20T09:05:00.705 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.705 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:beb58d58:::benchmark_data_smithi122_119671_object94190:head# 2024-03-20T09:05:00.705 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.705 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:beb5a9a4:::benchmark_data_smithi122_119671_object16959:head# 2024-03-20T09:05:00.705 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.705 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:beb7693f:::benchmark_data_smithi122_119671_object82261:head# 2024-03-20T09:05:00.706 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.706 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:beb801a4:::benchmark_data_smithi122_119671_object50487:head# 2024-03-20T09:05:00.706 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.706 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:beb869ba:::benchmark_data_smithi122_119671_object71989:head# 2024-03-20T09:05:00.706 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.706 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:beb8972e:::benchmark_data_smithi122_119671_object30607:head# 2024-03-20T09:05:00.706 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.706 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:beb8fc9d:::benchmark_data_smithi122_119671_object89388:head# 2024-03-20T09:05:00.706 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.706 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bebba2a3:::benchmark_data_smithi122_119671_object76657:head# 2024-03-20T09:05:00.706 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.706 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bebd4fbf:::benchmark_data_smithi122_119671_object47028:head# 2024-03-20T09:05:00.706 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.706 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bebd7ee6:::benchmark_data_smithi122_119671_object6678:head# 2024-03-20T09:05:00.707 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.707 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bebd889f:::benchmark_data_smithi122_119671_object30297:head# 2024-03-20T09:05:00.707 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.707 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bebf686d:::benchmark_data_smithi122_119671_object79964:head# 2024-03-20T09:05:00.707 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.707 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bebff038:::benchmark_data_smithi122_119671_object39747:head# 2024-03-20T09:05:00.707 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.707 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bec0a682:::benchmark_data_smithi122_119671_object46782:head# 2024-03-20T09:05:00.707 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.707 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bec1843a:::benchmark_data_smithi122_119671_object66805:head# 2024-03-20T09:05:00.707 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.707 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bec19845:::benchmark_data_smithi122_119671_object78789:head# 2024-03-20T09:05:00.707 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.707 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bec1d981:::benchmark_data_smithi122_119671_object73552:head# 2024-03-20T09:05:00.708 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.708 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bec1fe46:::benchmark_data_smithi122_119671_object67776:head# 2024-03-20T09:05:00.708 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.708 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bec24bac:::benchmark_data_smithi122_119671_object77623:head# 2024-03-20T09:05:00.708 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.708 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bec25437:::benchmark_data_smithi122_119671_object27732:head# 2024-03-20T09:05:00.708 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.708 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bec27e55:::benchmark_data_smithi122_119671_object1356:head# 2024-03-20T09:05:00.708 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.708 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bec2fedd:::benchmark_data_smithi122_119671_object33055:head# 2024-03-20T09:05:00.708 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.708 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bec392e7:::benchmark_data_smithi122_119671_object63369:head# 2024-03-20T09:05:00.708 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.708 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bec4a8cb:::benchmark_data_smithi122_119671_object45432:head# 2024-03-20T09:05:00.709 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.709 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bec5b116:::benchmark_data_smithi122_119671_object5062:head# 2024-03-20T09:05:00.709 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.709 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bec68172:::benchmark_data_smithi122_119671_object35049:head# 2024-03-20T09:05:00.709 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.709 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bec74f4c:::benchmark_data_smithi122_119671_object26537:head# 2024-03-20T09:05:00.709 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.709 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bec7613c:::benchmark_data_smithi122_119671_object51206:head# 2024-03-20T09:05:00.709 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.709 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bec7a9f7:::benchmark_data_smithi122_119671_object46740:head# 2024-03-20T09:05:00.709 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.709 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bec87fc2:::benchmark_data_smithi122_119671_object71263:head# 2024-03-20T09:05:00.709 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.709 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bec9a559:::benchmark_data_smithi122_119671_object64737:head# 2024-03-20T09:05:00.710 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.710 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:beca273b:::benchmark_data_smithi122_119671_object26942:head# 2024-03-20T09:05:00.710 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.710 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:becad4e4:::benchmark_data_smithi122_119671_object83918:head# 2024-03-20T09:05:00.710 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.710 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:becb0ffb:::benchmark_data_smithi122_119671_object40204:head# 2024-03-20T09:05:00.710 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.710 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:becb7473:::benchmark_data_smithi122_119671_object59371:head# 2024-03-20T09:05:00.710 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.710 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:becb9cc9:::benchmark_data_smithi122_119671_object27952:head# 2024-03-20T09:05:00.710 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.710 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:becca1bb:::benchmark_data_smithi122_119671_object83958:head# 2024-03-20T09:05:00.710 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.710 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:beccb32e:::benchmark_data_smithi122_119671_object39613:head# 2024-03-20T09:05:00.711 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.711 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:beccd9e3:::benchmark_data_smithi122_119671_object78532:head# 2024-03-20T09:05:00.711 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.711 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:becd3cfb:::benchmark_data_smithi122_119671_object83166:head# 2024-03-20T09:05:00.711 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.711 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bece5c2c:::benchmark_data_smithi122_119671_object43233:head# 2024-03-20T09:05:00.711 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.711 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bed08d6c:::benchmark_data_smithi122_119671_object64129:head# 2024-03-20T09:05:00.711 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.711 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bed0a553:::benchmark_data_smithi122_119671_object20686:head# 2024-03-20T09:05:00.711 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.711 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bed3978d:::benchmark_data_smithi122_119671_object63181:head# 2024-03-20T09:05:00.711 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.711 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bed42393:::benchmark_data_smithi122_119671_object4039:head# 2024-03-20T09:05:00.711 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.712 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bed45071:::benchmark_data_smithi122_119671_object80723:head# 2024-03-20T09:05:00.712 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.712 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bed55ef3:::benchmark_data_smithi122_119671_object67839:head# 2024-03-20T09:05:00.712 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.712 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bed69030:::benchmark_data_smithi122_119671_object74370:head# 2024-03-20T09:05:00.712 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.712 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bed8a13e:::benchmark_data_smithi122_119671_object61434:head# 2024-03-20T09:05:00.712 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.712 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bed8e87e:::benchmark_data_smithi122_119671_object51337:head# 2024-03-20T09:05:00.712 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.712 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:beda8477:::benchmark_data_smithi122_119671_object87842:head# 2024-03-20T09:05:00.712 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.712 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:beda8fea:::benchmark_data_smithi122_119671_object78049:head# 2024-03-20T09:05:00.712 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.713 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:beda981a:::benchmark_data_smithi122_119671_object72572:head# 2024-03-20T09:05:00.713 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.713 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bedb2e86:::benchmark_data_smithi122_119671_object14111:head# 2024-03-20T09:05:00.713 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.713 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bedb6bbe:::benchmark_data_smithi122_119671_object15565:head# 2024-03-20T09:05:00.713 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.713 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bedb7594:::benchmark_data_smithi122_119671_object96684:head# 2024-03-20T09:05:00.713 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.713 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bedc40ba:::benchmark_data_smithi122_119671_object72889:head# 2024-03-20T09:05:00.713 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.713 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bedc6836:::benchmark_data_smithi122_119671_object57156:head# 2024-03-20T09:05:00.713 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.713 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bedc7af3:::benchmark_data_smithi122_119671_object44284:head# 2024-03-20T09:05:00.714 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.714 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bedcd284:::benchmark_data_smithi122_119671_object64267:head# 2024-03-20T09:05:00.714 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.714 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bedcf23f:::benchmark_data_smithi122_119671_object16546:head# 2024-03-20T09:05:00.714 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.714 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bede5907:::benchmark_data_smithi122_119671_object11315:head# 2024-03-20T09:05:00.714 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.714 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bede647f:::benchmark_data_smithi122_119671_object85984:head# 2024-03-20T09:05:00.714 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.714 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bede9891:::benchmark_data_smithi122_119671_object99705:head# 2024-03-20T09:05:00.714 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.714 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bedec74f:::benchmark_data_smithi122_119671_object80817:head# 2024-03-20T09:05:00.714 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.715 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bee00d2c:::benchmark_data_smithi122_119671_object34705:head# 2024-03-20T09:05:00.715 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.715 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bee05e28:::benchmark_data_smithi122_119671_object70701:head# 2024-03-20T09:05:00.715 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.715 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bee060ad:::benchmark_data_smithi122_119671_object9783:head# 2024-03-20T09:05:00.715 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.715 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bee1cc06:::benchmark_data_smithi122_119671_object65398:head# 2024-03-20T09:05:00.715 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.715 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bee33e86:::benchmark_data_smithi122_119671_object26133:head# 2024-03-20T09:05:00.715 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.715 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bee44a0e:::benchmark_data_smithi122_119671_object37363:head# 2024-03-20T09:05:00.715 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.715 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bee46110:::benchmark_data_smithi122_119671_object74587:head# 2024-03-20T09:05:00.715 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.716 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bee49a07:::benchmark_data_smithi122_119671_object58667:head# 2024-03-20T09:05:00.716 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.716 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bee522a5:::benchmark_data_smithi122_119671_object89739:head# 2024-03-20T09:05:00.716 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.716 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bee53285:::benchmark_data_smithi122_119671_object30982:head# 2024-03-20T09:05:00.716 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.716 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bee59f34:::benchmark_data_smithi122_119671_object62801:head# 2024-03-20T09:05:00.716 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.716 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bee5af18:::benchmark_data_smithi122_119671_object6778:head# 2024-03-20T09:05:00.716 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.716 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bee63d10:::benchmark_data_smithi122_119671_object95748:head# 2024-03-20T09:05:00.716 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.717 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bee6b9d7:::benchmark_data_smithi122_119671_object60480:head# 2024-03-20T09:05:00.717 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.717 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bee73543:::benchmark_data_smithi122_119671_object19497:head# 2024-03-20T09:05:00.717 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.717 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bee8a42b:::benchmark_data_smithi122_119671_object83568:head# 2024-03-20T09:05:00.717 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.717 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bee9a632:::benchmark_data_smithi122_119671_object41591:head# 2024-03-20T09:05:00.717 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.717 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:beea8392:::benchmark_data_smithi122_119671_object76336:head# 2024-03-20T09:05:00.717 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.717 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:beecdf84:::benchmark_data_smithi122_119671_object7830:head# 2024-03-20T09:05:00.717 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.717 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bef05e9a:::benchmark_data_smithi122_119671_object62308:head# 2024-03-20T09:05:00.717 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.718 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bef138eb:::benchmark_data_smithi122_119671_object25047:head# 2024-03-20T09:05:00.718 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.718 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bef15b1f:::benchmark_data_smithi122_119671_object25445:head# 2024-03-20T09:05:00.718 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.718 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bef170fd:::benchmark_data_smithi122_119671_object19670:head# 2024-03-20T09:05:00.718 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.718 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bef1a16c:::benchmark_data_smithi122_119671_object79543:head# 2024-03-20T09:05:00.718 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.718 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bef1e28c:::benchmark_data_smithi122_119671_object77985:head# 2024-03-20T09:05:00.718 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.718 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bef1e508:::benchmark_data_smithi122_119671_object16163:head# 2024-03-20T09:05:00.718 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.718 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bef1fd58:::benchmark_data_smithi122_119671_object29461:head# 2024-03-20T09:05:00.718 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.719 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bef28402:::benchmark_data_smithi122_119671_object81627:head# 2024-03-20T09:05:00.719 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.719 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bef3a9e4:::benchmark_data_smithi122_119671_object71774:head# 2024-03-20T09:05:00.719 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.719 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bef3c795:::benchmark_data_smithi122_119671_object92179:head# 2024-03-20T09:05:00.719 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.719 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bef7b1f1:::benchmark_data_smithi122_119671_object85953:head# 2024-03-20T09:05:00.719 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.719 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bef81064:::benchmark_data_smithi122_119671_object79352:head# 2024-03-20T09:05:00.719 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.719 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bef835c1:::benchmark_data_smithi122_119671_object35886:head# 2024-03-20T09:05:00.719 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.719 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bef863c6:::benchmark_data_smithi122_119671_object11891:head# 2024-03-20T09:05:00.719 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.720 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bef8e81e:::benchmark_data_smithi122_119671_object53557:head# 2024-03-20T09:05:00.720 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.720 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bef8fee6:::benchmark_data_smithi122_119671_object98135:head# 2024-03-20T09:05:00.720 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.720 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bef917f4:::benchmark_data_smithi122_119671_object78946:head# 2024-03-20T09:05:00.720 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.720 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bef926ca:::benchmark_data_smithi122_119671_object16313:head# 2024-03-20T09:05:00.720 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.720 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bef94e83:::benchmark_data_smithi122_119671_object10139:head# 2024-03-20T09:05:00.720 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.720 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bef98d2b:::benchmark_data_smithi122_119671_object31384:head# 2024-03-20T09:05:00.720 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.720 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bef9d3bf:::benchmark_data_smithi122_119671_object34308:head# 2024-03-20T09:05:00.720 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.721 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:befa96e4:::benchmark_data_smithi122_119671_object6919:head# 2024-03-20T09:05:00.721 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.721 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:befaa0da:::benchmark_data_smithi122_119671_object17983:head# 2024-03-20T09:05:00.721 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.721 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:befac815:::benchmark_data_smithi122_119671_object38456:head# 2024-03-20T09:05:00.721 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.721 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:befad0f6:::benchmark_data_smithi122_119671_object6580:head# 2024-03-20T09:05:00.721 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.721 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:befad3a2:::benchmark_data_smithi122_119671_object10424:head# 2024-03-20T09:05:00.721 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.721 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:befaf371:::benchmark_data_smithi122_119671_object81251:head# 2024-03-20T09:05:00.721 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.721 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:befb9213:::benchmark_data_smithi122_119671_object47451:head# 2024-03-20T09:05:00.721 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.722 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:befcdfe5:::benchmark_data_smithi122_119671_object69879:head# 2024-03-20T09:05:00.722 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.722 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:befd4c90:::benchmark_data_smithi122_119671_object31609:head# 2024-03-20T09:05:00.722 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.722 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:befe0d1f:::benchmark_data_smithi122_119671_object39395:head# 2024-03-20T09:05:00.722 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.722 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:befe4987:::benchmark_data_smithi122_119671_object59837:head# 2024-03-20T09:05:00.722 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.722 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:befe95a7:::benchmark_data_smithi122_119671_object11350:head# 2024-03-20T09:05:00.722 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.722 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:beff7d1f:::benchmark_data_smithi122_119671_object25673:head# 2024-03-20T09:05:00.722 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.722 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf009008:::benchmark_data_smithi122_119671_object54006:head# 2024-03-20T09:05:00.722 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.723 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf01bac1:::benchmark_data_smithi122_119671_object97046:head# 2024-03-20T09:05:00.723 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.723 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf02ff0b:::benchmark_data_smithi122_119671_object83669:head# 2024-03-20T09:05:00.723 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.723 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf049a8e:::benchmark_data_smithi122_119671_object77577:head# 2024-03-20T09:05:00.723 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.723 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf04c195:::benchmark_data_smithi122_119671_object35164:head# 2024-03-20T09:05:00.723 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.723 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf06545f:::benchmark_data_smithi122_119671_object75846:head# 2024-03-20T09:05:00.723 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.723 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf06fbff:::benchmark_data_smithi122_119671_object39232:head# 2024-03-20T09:05:00.723 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.724 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf072c8e:::benchmark_data_smithi122_119671_object87601:head# 2024-03-20T09:05:00.724 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.724 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf075462:::benchmark_data_smithi122_119671_object37950:head# 2024-03-20T09:05:00.724 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.724 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf076964:::benchmark_data_smithi122_119671_object39203:head# 2024-03-20T09:05:00.724 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.724 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf07a62b:::benchmark_data_smithi122_119671_object84974:head# 2024-03-20T09:05:00.724 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.724 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf09fc63:::benchmark_data_smithi122_119671_object60686:head# 2024-03-20T09:05:00.724 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.724 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf0a05e6:::benchmark_data_smithi122_119671_object14776:head# 2024-03-20T09:05:00.724 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.724 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf0a78fd:::benchmark_data_smithi122_119671_object84726:head# 2024-03-20T09:05:00.724 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.724 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf0b6a24:::benchmark_data_smithi122_119671_object92533:head# 2024-03-20T09:05:00.725 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.725 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf0ba3b5:::benchmark_data_smithi122_119671_object28886:head# 2024-03-20T09:05:00.725 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.725 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf0d9b5c:::benchmark_data_smithi122_119671_object76988:head# 2024-03-20T09:05:00.725 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.725 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf0ee001:::benchmark_data_smithi122_119671_object29933:head# 2024-03-20T09:05:00.725 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.725 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf0eebd2:::benchmark_data_smithi122_119671_object25611:head# 2024-03-20T09:05:00.725 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.725 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf0f6f47:::benchmark_data_smithi122_119671_object69975:head# 2024-03-20T09:05:00.725 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.725 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf0fee57:::benchmark_data_smithi122_119671_object82448:head# 2024-03-20T09:05:00.725 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.725 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf114505:::benchmark_data_smithi122_119671_object57673:head# 2024-03-20T09:05:00.725 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.726 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf11544a:::benchmark_data_smithi122_119671_object99522:head# 2024-03-20T09:05:00.726 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.726 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf11f5db:::benchmark_data_smithi122_119671_object13355:head# 2024-03-20T09:05:00.726 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.726 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf1205d9:::benchmark_data_smithi122_119671_object99030:head# 2024-03-20T09:05:00.726 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.726 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf12700e:::benchmark_data_smithi122_119671_object90586:head# 2024-03-20T09:05:00.726 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.726 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf12b382:::benchmark_data_smithi122_119671_object28888:head# 2024-03-20T09:05:00.726 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.726 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf13135b:::benchmark_data_smithi122_119671_object37199:head# 2024-03-20T09:05:00.726 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.726 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf1314c6:::benchmark_data_smithi122_119671_object90100:head# 2024-03-20T09:05:00.726 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.727 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf1472f6:::benchmark_data_smithi122_119671_object89906:head# 2024-03-20T09:05:00.727 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.727 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf150d3d:::benchmark_data_smithi122_119671_object86915:head# 2024-03-20T09:05:00.727 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.727 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf157643:::benchmark_data_smithi122_119671_object8914:head# 2024-03-20T09:05:00.727 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.727 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf1579ba:::benchmark_data_smithi122_119671_object9988:head# 2024-03-20T09:05:00.727 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.727 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf159320:::benchmark_data_smithi122_119671_object71585:head# 2024-03-20T09:05:00.727 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.727 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf1632a5:::benchmark_data_smithi122_119671_object30394:head# 2024-03-20T09:05:00.727 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.727 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf1649fd:::benchmark_data_smithi122_119671_object15880:head# 2024-03-20T09:05:00.728 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.728 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf17966e:::benchmark_data_smithi122_119671_object44505:head# 2024-03-20T09:05:00.728 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.728 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf17ed6e:::benchmark_data_smithi122_119671_object83144:head# 2024-03-20T09:05:00.728 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.728 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf18a5a0:::benchmark_data_smithi122_119671_object24649:head# 2024-03-20T09:05:00.728 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.728 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf18cd3e:::benchmark_data_smithi122_119671_object52584:head# 2024-03-20T09:05:00.728 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.728 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf1ace22:::benchmark_data_smithi122_119671_object56237:head# 2024-03-20T09:05:00.728 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.728 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf1c0524:::benchmark_data_smithi122_119671_object3702:head# 2024-03-20T09:05:00.728 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.728 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf1c2f4e:::benchmark_data_smithi122_119671_object97748:head# 2024-03-20T09:05:00.728 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.729 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf200d34:::benchmark_data_smithi122_119671_object40213:head# 2024-03-20T09:05:00.729 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.729 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf208a40:::benchmark_data_smithi122_119671_object18036:head# 2024-03-20T09:05:00.729 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.729 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf2103cd:::benchmark_data_smithi122_119671_object16064:head# 2024-03-20T09:05:00.729 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.729 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf211088:::benchmark_data_smithi122_119671_object60321:head# 2024-03-20T09:05:00.729 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.729 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf21f281:::benchmark_data_smithi122_119671_object98741:head# 2024-03-20T09:05:00.729 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.729 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf257800:::benchmark_data_smithi122_119671_object54698:head# 2024-03-20T09:05:00.729 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.729 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf2578b6:::benchmark_data_smithi122_119671_object68062:head# 2024-03-20T09:05:00.729 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.730 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf2753fa:::benchmark_data_smithi122_119671_object6812:head# 2024-03-20T09:05:00.730 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.730 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf283c74:::benchmark_data_smithi122_119671_object44300:head# 2024-03-20T09:05:00.730 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.730 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf285d80:::benchmark_data_smithi122_119671_object41307:head# 2024-03-20T09:05:00.730 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.730 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf28c256:::benchmark_data_smithi122_119671_object73115:head# 2024-03-20T09:05:00.730 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.730 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf297e27:::benchmark_data_smithi122_119671_object67353:head# 2024-03-20T09:05:00.730 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.730 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf2bdcc2:::benchmark_data_smithi122_119671_object19315:head# 2024-03-20T09:05:00.730 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.730 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf2bfa73:::benchmark_data_smithi122_119671_object58584:head# 2024-03-20T09:05:00.730 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.731 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf2cc131:::benchmark_data_smithi122_119671_object74355:head# 2024-03-20T09:05:00.731 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.731 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf2e182d:::benchmark_data_smithi122_119671_object26994:head# 2024-03-20T09:05:00.731 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.731 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf2f395a:::benchmark_data_smithi122_119671_object47339:head# 2024-03-20T09:05:00.731 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.731 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf311e71:::benchmark_data_smithi122_119671_object42476:head# 2024-03-20T09:05:00.731 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.731 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf314f3a:::benchmark_data_smithi122_119671_object84991:head# 2024-03-20T09:05:00.731 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.731 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf320351:::benchmark_data_smithi122_119671_object71030:head# 2024-03-20T09:05:00.731 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.731 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf32afc0:::benchmark_data_smithi122_119671_object47885:head# 2024-03-20T09:05:00.731 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.732 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf343551:::benchmark_data_smithi122_119671_object79810:head# 2024-03-20T09:05:00.732 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.732 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf3444a4:::benchmark_data_smithi122_119671_object98418:head# 2024-03-20T09:05:00.732 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.732 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf34e7c1:::benchmark_data_smithi122_119671_object49038:head# 2024-03-20T09:05:00.732 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.732 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf35a852:::benchmark_data_smithi122_119671_object51893:head# 2024-03-20T09:05:00.732 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.732 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf368c16:::benchmark_data_smithi122_119671_object91190:head# 2024-03-20T09:05:00.732 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.733 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf383b2b:::benchmark_data_smithi122_119671_object23781:head# 2024-03-20T09:05:00.733 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.733 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf38a887:::benchmark_data_smithi122_119671_object41294:head# 2024-03-20T09:05:00.733 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.733 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf38eebb:::benchmark_data_smithi122_119671_object53872:head# 2024-03-20T09:05:00.733 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.733 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf3a5434:::benchmark_data_smithi122_119671_object43568:head# 2024-03-20T09:05:00.733 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.733 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf3c3690:::benchmark_data_smithi122_119671_object75939:head# 2024-03-20T09:05:00.733 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.733 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf3c4336:::benchmark_data_smithi122_119671_object77701:head# 2024-03-20T09:05:00.733 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.733 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf3c5431:::benchmark_data_smithi122_119671_object59381:head# 2024-03-20T09:05:00.734 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.734 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf3c7dfe:::benchmark_data_smithi122_119671_object77426:head# 2024-03-20T09:05:00.734 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.734 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf3e1294:::benchmark_data_smithi122_119671_object51170:head# 2024-03-20T09:05:00.734 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.734 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf3e2f8d:::benchmark_data_smithi122_119671_object59377:head# 2024-03-20T09:05:00.734 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.734 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf3f8582:::benchmark_data_smithi122_119671_object18948:head# 2024-03-20T09:05:00.734 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.734 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf3fdd9f:::benchmark_data_smithi122_119671_object87796:head# 2024-03-20T09:05:00.734 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.734 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf404997:::benchmark_data_smithi122_119671_object23343:head# 2024-03-20T09:05:00.734 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.735 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf40cdbf:::benchmark_data_smithi122_119671_object22632:head# 2024-03-20T09:05:00.735 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.735 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf40dcb3:::benchmark_data_smithi122_119671_object92364:head# 2024-03-20T09:05:00.735 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.735 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf41b14a:::benchmark_data_smithi122_119671_object44121:head# 2024-03-20T09:05:00.735 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.735 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf41d6f8:::benchmark_data_smithi122_119671_object57576:head# 2024-03-20T09:05:00.735 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.735 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf424a8c:::benchmark_data_smithi122_119671_object55964:head# 2024-03-20T09:05:00.735 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.735 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf42e205:::benchmark_data_smithi122_119671_object11975:head# 2024-03-20T09:05:00.735 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.735 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf43cfd1:::benchmark_data_smithi122_119671_object15883:head# 2024-03-20T09:05:00.735 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.736 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf43e1bf:::benchmark_data_smithi122_119671_object94552:head# 2024-03-20T09:05:00.736 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.736 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf43ec53:::benchmark_data_smithi122_119671_object50325:head# 2024-03-20T09:05:00.736 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.736 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf4414f5:::benchmark_data_smithi122_119671_object75297:head# 2024-03-20T09:05:00.736 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.736 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf44437c:::benchmark_data_smithi122_119671_object43119:head# 2024-03-20T09:05:00.736 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.736 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf4479b2:::benchmark_data_smithi122_119671_object14894:head# 2024-03-20T09:05:00.736 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.736 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf4558e3:::benchmark_data_smithi122_119671_object16111:head# 2024-03-20T09:05:00.736 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.736 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf4596c9:::benchmark_data_smithi122_119671_object76026:head# 2024-03-20T09:05:00.736 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.737 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf464659:::benchmark_data_smithi122_119671_object87873:head# 2024-03-20T09:05:00.737 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.737 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf465605:::benchmark_data_smithi122_119671_object44649:head# 2024-03-20T09:05:00.737 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.737 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf47f23f:::benchmark_data_smithi122_119671_object39026:head# 2024-03-20T09:05:00.737 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.737 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf486739:::benchmark_data_smithi122_119671_object10309:head# 2024-03-20T09:05:00.737 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.737 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf487a5b:::benchmark_data_smithi122_119671_object10838:head# 2024-03-20T09:05:00.737 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.737 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf48817f:::benchmark_data_smithi122_119671_object36832:head# 2024-03-20T09:05:00.737 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.737 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf498351:::benchmark_data_smithi122_119671_object27137:head# 2024-03-20T09:05:00.737 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.738 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf49884d:::benchmark_data_smithi122_119671_object31234:head# 2024-03-20T09:05:00.738 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.738 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf4a0e61:::benchmark_data_smithi122_119671_object77992:head# 2024-03-20T09:05:00.738 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.738 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf4ab5a2:::benchmark_data_smithi122_119671_object6617:head# 2024-03-20T09:05:00.738 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.738 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf4b1458:::benchmark_data_smithi122_119671_object38231:head# 2024-03-20T09:05:00.738 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.738 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf4bb4fe:::benchmark_data_smithi122_119671_object45479:head# 2024-03-20T09:05:00.738 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.738 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf4c33f7:::benchmark_data_smithi122_119671_object72628:head# 2024-03-20T09:05:00.738 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.739 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf4ccf40:::benchmark_data_smithi122_119671_object92231:head# 2024-03-20T09:05:00.739 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.739 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf4d1c18:::benchmark_data_smithi122_119671_object65018:head# 2024-03-20T09:05:00.739 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.739 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf4dce04:::benchmark_data_smithi122_119671_object64404:head# 2024-03-20T09:05:00.739 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.739 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf4df44e:::benchmark_data_smithi122_119671_object61035:head# 2024-03-20T09:05:00.739 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.739 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf4ef6f8:::benchmark_data_smithi122_119671_object40945:head# 2024-03-20T09:05:00.739 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.739 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf4f7be6:::benchmark_data_smithi122_119671_object76814:head# 2024-03-20T09:05:00.739 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.739 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf5006bb:::benchmark_data_smithi122_119671_object32803:head# 2024-03-20T09:05:00.739 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.739 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf50d36d:::benchmark_data_smithi122_119671_object88095:head# 2024-03-20T09:05:00.740 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.740 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf50ead6:::benchmark_data_smithi122_119671_object79328:head# 2024-03-20T09:05:00.740 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.740 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf543b6a:::benchmark_data_smithi122_119671_object97845:head# 2024-03-20T09:05:00.740 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.740 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf560436:::benchmark_data_smithi122_119671_object93825:head# 2024-03-20T09:05:00.740 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.740 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf566150:::benchmark_data_smithi122_119671_object32859:head# 2024-03-20T09:05:00.740 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.740 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf571cae:::benchmark_data_smithi122_119671_object36642:head# 2024-03-20T09:05:00.740 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.740 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf5865db:::benchmark_data_smithi122_119671_object63547:head# 2024-03-20T09:05:00.740 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.741 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf58ce1d:::benchmark_data_smithi122_119671_object52082:head# 2024-03-20T09:05:00.741 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.741 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf58d360:::benchmark_data_smithi122_119671_object47687:head# 2024-03-20T09:05:00.741 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.741 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf594681:::benchmark_data_smithi122_119671_object17671:head# 2024-03-20T09:05:00.741 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.741 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf599dcb:::benchmark_data_smithi122_119671_object48644:head# 2024-03-20T09:05:00.741 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.741 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf59e657:::benchmark_data_smithi122_119671_object72832:head# 2024-03-20T09:05:00.741 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.741 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf5a94ce:::benchmark_data_smithi122_119671_object32982:head# 2024-03-20T09:05:00.741 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.741 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf5bf30b:::benchmark_data_smithi122_119671_object50297:head# 2024-03-20T09:05:00.741 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.742 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf5cb558:::benchmark_data_smithi122_119671_object24565:head# 2024-03-20T09:05:00.742 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.742 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf5e83a8:::benchmark_data_smithi122_119671_object11887:head# 2024-03-20T09:05:00.742 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.742 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf5eaa97:::benchmark_data_smithi122_119671_object92737:head# 2024-03-20T09:05:00.742 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.742 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf5f5e54:::benchmark_data_smithi122_119671_object71632:head# 2024-03-20T09:05:00.742 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.742 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf5fd96a:::benchmark_data_smithi122_119671_object2520:head# 2024-03-20T09:05:00.742 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.742 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf6177c7:::benchmark_data_smithi122_119671_object59382:head# 2024-03-20T09:05:00.742 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.742 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf618700:::benchmark_data_smithi122_119671_object43975:head# 2024-03-20T09:05:00.742 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.743 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf630644:::benchmark_data_smithi122_119671_object47076:head# 2024-03-20T09:05:00.743 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.743 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf630679:::benchmark_data_smithi122_119671_object51092:head# 2024-03-20T09:05:00.743 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.743 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf63673b:::benchmark_data_smithi122_119671_object55914:head# 2024-03-20T09:05:00.743 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.743 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf639ed7:::benchmark_data_smithi122_119671_object55514:head# 2024-03-20T09:05:00.743 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.743 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf64d246:::benchmark_data_smithi122_119671_object57838:head# 2024-03-20T09:05:00.743 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.743 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf659e1f:::benchmark_data_smithi122_119671_object27704:head# 2024-03-20T09:05:00.743 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.743 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf6703ab:::benchmark_data_smithi122_119671_object63224:head# 2024-03-20T09:05:00.744 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.744 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf6a8ade:::benchmark_data_smithi122_119671_object2163:head# 2024-03-20T09:05:00.744 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.744 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf6b72a3:::benchmark_data_smithi122_119671_object64017:head# 2024-03-20T09:05:00.744 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.744 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf6bb9cd:::benchmark_data_smithi122_119671_object66449:head# 2024-03-20T09:05:00.744 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.744 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf6bdca8:::benchmark_data_smithi122_119671_object92552:head# 2024-03-20T09:05:00.744 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.744 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf6dfdc8:::benchmark_data_smithi122_119671_object54654:head# 2024-03-20T09:05:00.744 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.744 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf6e0607:::benchmark_data_smithi122_119671_object96669:head# 2024-03-20T09:05:00.744 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.745 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf6ec1c9:::benchmark_data_smithi122_119671_object58115:head# 2024-03-20T09:05:00.745 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.745 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf700429:::benchmark_data_smithi122_119671_object42654:head# 2024-03-20T09:05:00.745 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.745 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf7012cb:::benchmark_data_smithi122_119671_object96951:head# 2024-03-20T09:05:00.745 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.745 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf703815:::benchmark_data_smithi122_119671_object23806:head# 2024-03-20T09:05:00.745 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.745 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf705e11:::benchmark_data_smithi122_119671_object60268:head# 2024-03-20T09:05:00.745 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.745 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf706e07:::benchmark_data_smithi122_119671_object51911:head# 2024-03-20T09:05:00.746 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.746 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf70ad5a:::benchmark_data_smithi122_119671_object94779:head# 2024-03-20T09:05:00.746 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.746 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf70b716:::benchmark_data_smithi122_119671_object14232:head# 2024-03-20T09:05:00.746 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.746 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf714fdc:::benchmark_data_smithi122_119671_object60591:head# 2024-03-20T09:05:00.746 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.746 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf71c900:::benchmark_data_smithi122_119671_object49810:head# 2024-03-20T09:05:00.746 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.746 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf721296:::benchmark_data_smithi122_119671_object41729:head# 2024-03-20T09:05:00.746 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.746 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf7222d4:::benchmark_data_smithi122_119671_object52674:head# 2024-03-20T09:05:00.747 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.747 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf728a15:::benchmark_data_smithi122_119671_object45395:head# 2024-03-20T09:05:00.747 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.747 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf74447c:::benchmark_data_smithi122_119671_object54310:head# 2024-03-20T09:05:00.747 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.747 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf74944a:::benchmark_data_smithi122_119671_object10781:head# 2024-03-20T09:05:00.747 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.747 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf74ceb5:::benchmark_data_smithi122_119671_object51452:head# 2024-03-20T09:05:00.747 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.747 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf765b99:::benchmark_data_smithi122_119671_object37729:head# 2024-03-20T09:05:00.747 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.747 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf76619b:::benchmark_data_smithi122_119671_object67165:head# 2024-03-20T09:05:00.747 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.747 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf76e5ab:::benchmark_data_smithi122_119671_object62120:head# 2024-03-20T09:05:00.748 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.748 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf78c036:::benchmark_data_smithi122_119671_object90881:head# 2024-03-20T09:05:00.748 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.748 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf79c34e:::benchmark_data_smithi122_119671_object88020:head# 2024-03-20T09:05:00.748 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.748 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf7a6e26:::benchmark_data_smithi122_119671_object27912:head# 2024-03-20T09:05:00.748 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.748 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf7ac28b:::benchmark_data_smithi122_119671_object39719:head# 2024-03-20T09:05:00.748 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.748 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf7ad551:::benchmark_data_smithi122_119671_object11672:head# 2024-03-20T09:05:00.748 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.748 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf7c34a4:::benchmark_data_smithi122_119671_object62783:head# 2024-03-20T09:05:00.748 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.749 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf7c6b21:::benchmark_data_smithi122_119671_object42588:head# 2024-03-20T09:05:00.749 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.749 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf7d54d2:::benchmark_data_smithi122_119671_object14729:head# 2024-03-20T09:05:00.749 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.749 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf7e8ce8:::benchmark_data_smithi122_119671_object72581:head# 2024-03-20T09:05:00.749 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.749 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf7fd982:::benchmark_data_smithi122_119671_object27145:head# 2024-03-20T09:05:00.749 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.749 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf7fe68a:::benchmark_data_smithi122_119671_object15891:head# 2024-03-20T09:05:00.749 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.749 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf819d27:::benchmark_data_smithi122_119671_object56972:head# 2024-03-20T09:05:00.749 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.749 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf85a3ee:::benchmark_data_smithi122_119671_object6660:head# 2024-03-20T09:05:00.749 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.750 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf891032:::benchmark_data_smithi122_119671_object32357:head# 2024-03-20T09:05:00.750 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.750 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf896d1b:::benchmark_data_smithi122_119671_object9378:head# 2024-03-20T09:05:00.750 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.750 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf89ebd5:::benchmark_data_smithi122_119671_object69935:head# 2024-03-20T09:05:00.750 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.750 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf8a8304:::benchmark_data_smithi122_119671_object52751:head# 2024-03-20T09:05:00.750 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.750 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf8a8a8c:::benchmark_data_smithi122_119671_object97628:head# 2024-03-20T09:05:00.750 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.750 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf8b750c:::benchmark_data_smithi122_119671_object54145:head# 2024-03-20T09:05:00.750 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.750 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf8c1be9:::benchmark_data_smithi122_119671_object91965:head# 2024-03-20T09:05:00.751 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.751 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf8cdd0e:::benchmark_data_smithi122_119671_object26127:head# 2024-03-20T09:05:00.751 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.751 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf8e6d76:::benchmark_data_smithi122_119671_object74248:head# 2024-03-20T09:05:00.751 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.751 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf8f3962:::benchmark_data_smithi122_119671_object10827:head# 2024-03-20T09:05:00.751 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.751 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf91b32b:::benchmark_data_smithi122_119671_object4446:head# 2024-03-20T09:05:00.751 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.751 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf92f0ea:::benchmark_data_smithi122_119671_object42265:head# 2024-03-20T09:05:00.751 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.751 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf93a0d7:::benchmark_data_smithi122_119671_object80485:head# 2024-03-20T09:05:00.751 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.751 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf93c3cf:::benchmark_data_smithi122_119671_object12546:head# 2024-03-20T09:05:00.752 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.752 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf942496:::benchmark_data_smithi122_119671_object86170:head# 2024-03-20T09:05:00.752 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.752 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf9470e1:::benchmark_data_smithi122_119671_object62743:head# 2024-03-20T09:05:00.752 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.752 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf94a6a4:::benchmark_data_smithi122_119671_object50080:head# 2024-03-20T09:05:00.752 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.752 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf94ace7:::benchmark_data_smithi122_119671_object42920:head# 2024-03-20T09:05:00.752 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.752 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf94f251:::benchmark_data_smithi122_119671_object30495:head# 2024-03-20T09:05:00.752 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.752 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf9609b0:::benchmark_data_smithi122_119671_object94097:head# 2024-03-20T09:05:00.752 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.752 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf988bef:::benchmark_data_smithi122_119671_object46119:head# 2024-03-20T09:05:00.753 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.753 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf99acf9:::benchmark_data_smithi122_119671_object92387:head# 2024-03-20T09:05:00.753 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.753 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf9a3733:::benchmark_data_smithi122_119671_object9162:head# 2024-03-20T09:05:00.753 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.753 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf9a5a41:::benchmark_data_smithi122_119671_object83962:head# 2024-03-20T09:05:00.753 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.753 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf9d12e4:::benchmark_data_smithi122_119671_object22080:head# 2024-03-20T09:05:00.753 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.753 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf9d5272:::benchmark_data_smithi122_119671_object29647:head# 2024-03-20T09:05:00.753 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.753 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bf9e363e:::benchmark_data_smithi122_119671_object81524:head# 2024-03-20T09:05:00.753 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.753 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfa06aba:::benchmark_data_smithi122_119671_object63903:head# 2024-03-20T09:05:00.754 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.754 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfa10891:::benchmark_data_smithi122_119671_object46680:head# 2024-03-20T09:05:00.754 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.754 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfa11979:::benchmark_data_smithi122_119671_object93684:head# 2024-03-20T09:05:00.754 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.754 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfa1ac8c:::benchmark_data_smithi122_119671_object7513:head# 2024-03-20T09:05:00.754 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.754 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfa1c5d1:::benchmark_data_smithi122_119671_object26393:head# 2024-03-20T09:05:00.754 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.754 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfa26516:::benchmark_data_smithi122_119671_object66721:head# 2024-03-20T09:05:00.754 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.754 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfa2cf3a:::benchmark_data_smithi122_119671_object82702:head# 2024-03-20T09:05:00.754 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.755 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfa36df0:::benchmark_data_smithi122_119671_object41003:head# 2024-03-20T09:05:00.755 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.755 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfa375e5:::benchmark_data_smithi122_119671_object50429:head# 2024-03-20T09:05:00.755 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.755 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfa559a5:::benchmark_data_smithi122_119671_object41356:head# 2024-03-20T09:05:00.755 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.755 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfa6c7c1:::benchmark_data_smithi122_119671_object66189:head# 2024-03-20T09:05:00.755 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.755 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfa86139:::benchmark_data_smithi122_119671_object76687:head# 2024-03-20T09:05:00.755 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.755 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfaa1b35:::benchmark_data_smithi122_119671_object14105:head# 2024-03-20T09:05:00.755 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.755 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfaa3826:::benchmark_data_smithi122_119671_object78836:head# 2024-03-20T09:05:00.755 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.755 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfaa645f:::benchmark_data_smithi122_119671_object94205:head# 2024-03-20T09:05:00.756 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.756 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfaa66af:::benchmark_data_smithi122_119671_object15833:head# 2024-03-20T09:05:00.756 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.756 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfaa7c60:::benchmark_data_smithi122_119671_object21510:head# 2024-03-20T09:05:00.756 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.756 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfacc072:::benchmark_data_smithi122_119671_object79766:head# 2024-03-20T09:05:00.756 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.756 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfacccc7:::benchmark_data_smithi122_119671_object88191:head# 2024-03-20T09:05:00.756 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.756 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bface4fb:::benchmark_data_smithi122_119671_object15484:head# 2024-03-20T09:05:00.756 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.756 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfad1e28:::benchmark_data_smithi122_119671_object2194:head# 2024-03-20T09:05:00.756 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.756 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfad6a43:::benchmark_data_smithi122_119671_object73433:head# 2024-03-20T09:05:00.757 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.757 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfad9e91:::benchmark_data_smithi122_119671_object70519:head# 2024-03-20T09:05:00.757 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.757 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfae54ce:::benchmark_data_smithi122_119671_object72705:head# 2024-03-20T09:05:00.757 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.757 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfafbb88:::benchmark_data_smithi122_119671_object9976:head# 2024-03-20T09:05:00.757 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.757 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfafdb35:::benchmark_data_smithi122_119671_object33033:head# 2024-03-20T09:05:00.757 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.757 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfaff92c:::benchmark_data_smithi122_119671_object20404:head# 2024-03-20T09:05:00.757 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.757 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfb06ac1:::benchmark_data_smithi122_119671_object42086:head# 2024-03-20T09:05:00.757 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.757 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfb0cadb:::benchmark_data_smithi122_119671_object39586:head# 2024-03-20T09:05:00.758 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.758 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfb18863:::benchmark_data_smithi122_119671_object80475:head# 2024-03-20T09:05:00.758 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.758 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfb1eb42:::benchmark_data_smithi122_119671_object2515:head# 2024-03-20T09:05:00.758 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.758 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfb247be:::benchmark_data_smithi122_119671_object8990:head# 2024-03-20T09:05:00.758 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.758 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfb32b5c:::benchmark_data_smithi122_119671_object6402:head# 2024-03-20T09:05:00.758 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.758 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfb5176c:::benchmark_data_smithi122_119671_object4266:head# 2024-03-20T09:05:00.758 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.758 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfb5743c:::benchmark_data_smithi122_119671_object84917:head# 2024-03-20T09:05:00.758 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.758 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfb5d1eb:::benchmark_data_smithi122_119671_object7007:head# 2024-03-20T09:05:00.759 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.759 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfb64736:::benchmark_data_smithi122_119671_object39297:head# 2024-03-20T09:05:00.759 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.759 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfb64b71:::benchmark_data_smithi122_119671_object61409:head# 2024-03-20T09:05:00.759 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.759 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfb77197:::benchmark_data_smithi122_119671_object27729:head# 2024-03-20T09:05:00.759 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.759 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfb84e73:::benchmark_data_smithi122_119671_object17233:head# 2024-03-20T09:05:00.759 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.759 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfb883ce:::benchmark_data_smithi122_119671_object12162:head# 2024-03-20T09:05:00.759 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.759 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfb8904d:::benchmark_data_smithi122_119671_object39374:head# 2024-03-20T09:05:00.759 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.759 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfb8c01a:::benchmark_data_smithi122_119671_object87823:head# 2024-03-20T09:05:00.760 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.760 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfb8db47:::benchmark_data_smithi122_119671_object50743:head# 2024-03-20T09:05:00.760 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.760 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfb8f5fc:::benchmark_data_smithi122_119671_object92792:head# 2024-03-20T09:05:00.760 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.760 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfbae6e8:::benchmark_data_smithi122_119671_object27316:head# 2024-03-20T09:05:00.760 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.760 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfbb2932:::benchmark_data_smithi122_119671_object62192:head# 2024-03-20T09:05:00.760 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.760 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfbbd2b2:::benchmark_data_smithi122_119671_object36985:head# 2024-03-20T09:05:00.760 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.760 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfbbe7c9:::benchmark_data_smithi122_119671_object9684:head# 2024-03-20T09:05:00.760 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.760 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfbd5c52:::benchmark_data_smithi122_119671_object85612:head# 2024-03-20T09:05:00.761 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.761 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfbda608:::benchmark_data_smithi122_119671_object68913:head# 2024-03-20T09:05:00.761 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.761 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfbe5601:::benchmark_data_smithi122_119671_object78502:head# 2024-03-20T09:05:00.761 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.761 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfbeeabf:::benchmark_data_smithi122_119671_object69893:head# 2024-03-20T09:05:00.761 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.761 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfc02894:::benchmark_data_smithi122_119671_object74153:head# 2024-03-20T09:05:00.761 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.761 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfc091ac:::benchmark_data_smithi122_119671_object49237:head# 2024-03-20T09:05:00.761 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.761 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfc13454:::benchmark_data_smithi122_119671_object97083:head# 2024-03-20T09:05:00.761 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.761 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfc1c368:::benchmark_data_smithi122_119671_object94558:head# 2024-03-20T09:05:00.762 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.762 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfc2c174:::benchmark_data_smithi122_119671_object36633:head# 2024-03-20T09:05:00.762 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.762 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfc3d06f:::benchmark_data_smithi122_119671_object34929:head# 2024-03-20T09:05:00.762 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.762 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfc49334:::benchmark_data_smithi122_119671_object61591:head# 2024-03-20T09:05:00.762 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.762 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfc4cdd3:::benchmark_data_smithi122_119671_object89891:head# 2024-03-20T09:05:00.762 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.762 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfc5a3b1:::benchmark_data_smithi122_119671_object53223:head# 2024-03-20T09:05:00.762 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.762 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfc5a6ba:::benchmark_data_smithi122_119671_object55383:head# 2024-03-20T09:05:00.762 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.762 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfc7bcb9:::benchmark_data_smithi122_119671_object13169:head# 2024-03-20T09:05:00.763 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.763 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfc84cd3:::benchmark_data_smithi122_119671_object9140:head# 2024-03-20T09:05:00.763 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.763 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfc9bbf6:::benchmark_data_smithi122_119671_object94560:head# 2024-03-20T09:05:00.763 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.763 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfc9dfe4:::benchmark_data_smithi122_119671_object76315:head# 2024-03-20T09:05:00.763 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.763 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfcaa5d1:::benchmark_data_smithi122_119671_object99067:head# 2024-03-20T09:05:00.763 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.763 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfcabb75:::benchmark_data_smithi122_119671_object23209:head# 2024-03-20T09:05:00.763 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.763 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfcaeb33:::benchmark_data_smithi122_119671_object76681:head# 2024-03-20T09:05:00.764 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.764 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfcc5b3f:::benchmark_data_smithi122_119671_object38059:head# 2024-03-20T09:05:00.764 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.764 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfccde9b:::benchmark_data_smithi122_119671_object52761:head# 2024-03-20T09:05:00.764 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.764 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfcddcea:::benchmark_data_smithi122_119671_object26385:head# 2024-03-20T09:05:00.764 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.764 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfcec624:::benchmark_data_smithi122_119671_object90745:head# 2024-03-20T09:05:00.764 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.764 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfcf0e53:::benchmark_data_smithi122_119671_object32406:head# 2024-03-20T09:05:00.764 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.764 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfd04519:::benchmark_data_smithi122_119671_object8177:head# 2024-03-20T09:05:00.765 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.765 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfd11f96:::benchmark_data_smithi122_119671_object53126:head# 2024-03-20T09:05:00.765 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.765 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfd34431:::benchmark_data_smithi122_119671_object13433:head# 2024-03-20T09:05:00.765 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.765 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfd49be6:::benchmark_data_smithi122_119671_object46334:head# 2024-03-20T09:05:00.765 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.765 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfd4b838:::benchmark_data_smithi122_119671_object95678:head# 2024-03-20T09:05:00.765 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.765 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfd50c6a:::benchmark_data_smithi122_119671_object91878:head# 2024-03-20T09:05:00.765 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.765 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfd56ce4:::benchmark_data_smithi122_119671_object79035:head# 2024-03-20T09:05:00.765 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.765 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfd5819f:::benchmark_data_smithi122_119671_object44444:head# 2024-03-20T09:05:00.766 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.766 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfd745fa:::benchmark_data_smithi122_119671_object72236:head# 2024-03-20T09:05:00.766 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.766 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfd88c25:::benchmark_data_smithi122_119671_object85607:head# 2024-03-20T09:05:00.766 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.766 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfdaa325:::benchmark_data_smithi122_119671_object65925:head# 2024-03-20T09:05:00.766 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.766 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfdbabdb:::benchmark_data_smithi122_119671_object95738:head# 2024-03-20T09:05:00.766 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.766 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfdd0c89:::benchmark_data_smithi122_119671_object86521:head# 2024-03-20T09:05:00.766 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.766 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfdd1173:::benchmark_data_smithi122_119671_object3812:head# 2024-03-20T09:05:00.766 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.767 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfdd9217:::benchmark_data_smithi122_119671_object68600:head# 2024-03-20T09:05:00.767 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.767 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfde2554:::benchmark_data_smithi122_119671_object75145:head# 2024-03-20T09:05:00.767 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.767 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfdeb00e:::benchmark_data_smithi122_119671_object76820:head# 2024-03-20T09:05:00.767 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.767 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfdfdde4:::benchmark_data_smithi122_119671_object77289:head# 2024-03-20T09:05:00.767 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.767 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfe00360:::benchmark_data_smithi122_119671_object29420:head# 2024-03-20T09:05:00.767 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.767 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfe103dd:::benchmark_data_smithi122_119671_object2076:head# 2024-03-20T09:05:00.767 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.767 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfe17689:::benchmark_data_smithi122_119671_object7384:head# 2024-03-20T09:05:00.768 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.768 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfe1e239:::benchmark_data_smithi122_119671_object85874:head# 2024-03-20T09:05:00.768 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.768 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfe239a0:::benchmark_data_smithi122_119671_object3316:head# 2024-03-20T09:05:00.768 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.768 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfe272e0:::benchmark_data_smithi122_119671_object91247:head# 2024-03-20T09:05:00.768 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.768 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfe29792:::benchmark_data_smithi122_119671_object5990:head# 2024-03-20T09:05:00.768 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.768 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfe2a18e:::benchmark_data_smithi122_119671_object63975:head# 2024-03-20T09:05:00.768 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.768 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfe3035f:::benchmark_data_smithi122_119671_object44819:head# 2024-03-20T09:05:00.768 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.768 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfe314c0:::benchmark_data_smithi122_119671_object91742:head# 2024-03-20T09:05:00.769 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.769 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfe44865:::benchmark_data_smithi122_119671_object29441:head# 2024-03-20T09:05:00.769 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.769 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfe47bde:::benchmark_data_smithi122_119671_object3915:head# 2024-03-20T09:05:00.769 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.769 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfe48fcb:::benchmark_data_smithi122_119671_object25023:head# 2024-03-20T09:05:00.769 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.769 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfe4fa6f:::benchmark_data_smithi122_119671_object63699:head# 2024-03-20T09:05:00.769 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.769 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfe7112f:::benchmark_data_smithi122_119671_object40591:head# 2024-03-20T09:05:00.769 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.769 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfe86df6:::benchmark_data_smithi122_119671_object90985:head# 2024-03-20T09:05:00.769 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.770 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfe9540c:::benchmark_data_smithi122_119671_object15335:head# 2024-03-20T09:05:00.770 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.770 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfe973fd:::benchmark_data_smithi122_119671_object46408:head# 2024-03-20T09:05:00.770 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.770 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfe9b754:::benchmark_data_smithi122_119671_object71015:head# 2024-03-20T09:05:00.770 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.770 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfea2c0d:::benchmark_data_smithi122_119671_object22757:head# 2024-03-20T09:05:00.770 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.770 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfea7df2:::benchmark_data_smithi122_119671_object60995:head# 2024-03-20T09:05:00.770 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.770 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfea8fef:::benchmark_data_smithi122_119671_object94401:head# 2024-03-20T09:05:00.770 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.770 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfeb83e2:::benchmark_data_smithi122_119671_object86506:head# 2024-03-20T09:05:00.770 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.771 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfebbcb7:::benchmark_data_smithi122_119671_object26828:head# 2024-03-20T09:05:00.771 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.771 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfec307e:::benchmark_data_smithi122_119671_object18304:head# 2024-03-20T09:05:00.771 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.771 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfec4999:::benchmark_data_smithi122_119671_object3645:head# 2024-03-20T09:05:00.771 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.771 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfeda674:::benchmark_data_smithi122_119671_object9548:head# 2024-03-20T09:05:00.771 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.771 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfee81cc:::benchmark_data_smithi122_119671_object47067:head# 2024-03-20T09:05:00.771 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.771 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfeef177:::benchmark_data_smithi122_119671_object20889:head# 2024-03-20T09:05:00.771 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.771 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfeef60c:::benchmark_data_smithi122_119671_object91331:head# 2024-03-20T09:05:00.771 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.772 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfef26c4:::benchmark_data_smithi122_119671_object65709:head# 2024-03-20T09:05:00.772 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.772 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfef7ead:::benchmark_data_smithi122_119671_object17773:head# 2024-03-20T09:05:00.772 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.772 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bff0436f:::benchmark_data_smithi122_119671_object78318:head# 2024-03-20T09:05:00.772 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.772 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bff2ad38:::benchmark_data_smithi122_119671_object27757:head# 2024-03-20T09:05:00.772 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.772 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bff2c8ad:::benchmark_data_smithi122_119671_object11333:head# 2024-03-20T09:05:00.772 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.772 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bff30e7d:::benchmark_data_smithi122_119671_object6154:head# 2024-03-20T09:05:00.772 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.773 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bff38646:::benchmark_data_smithi122_119671_object82446:head# 2024-03-20T09:05:00.773 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.773 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bff3d50c:::benchmark_data_smithi122_119671_object36006:head# 2024-03-20T09:05:00.773 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.773 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bff4bcc5:::benchmark_data_smithi122_119671_object40002:head# 2024-03-20T09:05:00.773 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.773 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bff68072:::benchmark_data_smithi122_119671_object36396:head# 2024-03-20T09:05:00.773 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.773 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bff6efce:::benchmark_data_smithi122_119671_object82416:head# 2024-03-20T09:05:00.773 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.773 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bff7326d:::benchmark_data_smithi122_119671_object67460:head# 2024-03-20T09:05:00.773 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.773 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bff74b95:::benchmark_data_smithi122_119671_object41766:head# 2024-03-20T09:05:00.773 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.774 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bff79aaf:::benchmark_data_smithi122_119671_object93656:head# 2024-03-20T09:05:00.774 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.774 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bff7c364:::benchmark_data_smithi122_119671_object87564:head# 2024-03-20T09:05:00.774 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.774 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bff8cf1f:::benchmark_data_smithi122_119671_object18766:head# 2024-03-20T09:05:00.774 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.774 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bff939ca:::benchmark_data_smithi122_119671_object11279:head# 2024-03-20T09:05:00.774 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.774 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bffc4ad8:::benchmark_data_smithi122_119671_object73274:head# 2024-03-20T09:05:00.774 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.774 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bffcef6b:::benchmark_data_smithi122_119671_object90569:head# 2024-03-20T09:05:00.774 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.774 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfff01d5:::benchmark_data_smithi122_119671_object74042:head# 2024-03-20T09:05:00.775 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.775 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfff9394:::benchmark_data_smithi122_119671_object37883:head# 2024-03-20T09:05:00.775 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.775 INFO:teuthology.orchestra.run.smithi115.stdout:Write #3:bfff9fcd:::benchmark_data_smithi122_119671_object55710:head# 2024-03-20T09:05:00.775 INFO:teuthology.orchestra.run.smithi115.stdout:snapset 0=[]:{} 2024-03-20T09:05:00.775 INFO:teuthology.orchestra.run.smithi115.stdout:write_pg epoch 273 info 3.d( v 273'6244 (0'0,273'6244] local-lis/les=265/266 n=6132 ec=204/204 lis/c=265/204 les/c/f=266/205/0 sis=265) 2024-03-20T09:05:00.776 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:00.776 INFO:teuthology.orchestra.run.smithi115.stdout:Import successful 2024-03-20T09:05:00.777 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:00 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:05:00.777 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:00 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:05:00.777 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:00 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/743011813' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:05:00.777 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:00 smithi122 ceph-mon[73495]: pgmap v737: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 28498/287688 objects degraded (9.906%); 11965/287688 objects misplaced (4.159%) 2024-03-20T09:05:00.777 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:00 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:05:00.777 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:00 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:05:00.778 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:00 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/743011813' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:05:00.778 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:00 smithi026 ceph-mon[68309]: pgmap v737: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 28498/287688 objects degraded (9.906%); 11965/287688 objects misplaced (4.159%) 2024-03-20T09:05:00.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:00.778 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:05:00.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:05:00.779 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:00 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:05:00.779 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:00 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:05:00.779 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:00 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/743011813' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:05:00.779 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:00 smithi115 ceph-mon[72476]: pgmap v737: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 28498/287688 objects degraded (9.906%); 11965/287688 objects misplaced (4.159%) 2024-03-20T09:05:00.879 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:05:00.911 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:01.077 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-20T09:05:01.261 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:05:01.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:01.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:05:01.365 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:05:01.396 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:01.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:01 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:05:01.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:01 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/743011813' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:05:01.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:01 smithi115 ceph-mon[72476]: osdmap e286: 12 total, 11 up, 11 in 2024-03-20T09:05:01.696 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:01 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:05:01.697 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:01 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/743011813' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:05:01.697 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:01 smithi026 ceph-mon[68309]: osdmap e286: 12 total, 11 up, 11 in 2024-03-20T09:05:01.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:01 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:05:01.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:01 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/743011813' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:05:01.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:01 smithi122 ceph-mon[73495]: osdmap e286: 12 total, 11 up, 11 in 2024-03-20T09:05:02.219 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:05:02.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:02 smithi026 ceph-mon[68309]: pgmap v739: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 28335/286176 objects degraded (9.901%); 11965/286176 objects misplaced (4.181%) 2024-03-20T09:05:02.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:02 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 28498/287688 objects degraded (9.906%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:05:02.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:02 smithi026 ceph-mon[68309]: osdmap e287: 12 total, 11 up, 11 in 2024-03-20T09:05:02.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:02 smithi115 ceph-mon[72476]: pgmap v739: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 28335/286176 objects degraded (9.901%); 11965/286176 objects misplaced (4.181%) 2024-03-20T09:05:02.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:02 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 28498/287688 objects degraded (9.906%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:05:02.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:02 smithi115 ceph-mon[72476]: osdmap e287: 12 total, 11 up, 11 in 2024-03-20T09:05:02.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:02 smithi122 ceph-mon[73495]: pgmap v739: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 28335/286176 objects degraded (9.901%); 11965/286176 objects misplaced (4.181%) 2024-03-20T09:05:02.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:02 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 28498/287688 objects degraded (9.906%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:05:02.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:02 smithi122 ceph-mon[73495]: osdmap e287: 12 total, 11 up, 11 in 2024-03-20T09:05:03.348 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:05:03.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:03.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:05:03.452 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:05:03.483 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:03.725 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:05:03.753 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:03 smithi026 ceph-mon[68309]: pgmap v741: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 29 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 194 op/s; 28184/285030 objects degraded (9.888%); 11927/285030 objects misplaced (4.184%); 0 B/s, 0 objects/s recovering 2024-03-20T09:05:04.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:03 smithi115 ceph-mon[72476]: pgmap v741: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 29 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 194 op/s; 28184/285030 objects degraded (9.888%); 11927/285030 objects misplaced (4.184%); 0 B/s, 0 objects/s recovering 2024-03-20T09:05:04.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:03 smithi122 ceph-mon[73495]: pgmap v741: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 29 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 194 op/s; 28184/285030 objects degraded (9.888%); 11927/285030 objects misplaced (4.184%); 0 B/s, 0 objects/s recovering 2024-03-20T09:05:04.748 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:05:04.769 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:05:04.791 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:05:04.814 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:05:04.836 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:05:05.059 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:05:05.079 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:05:05.100 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:05:05.122 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:05:05.140 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:05:05.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:04 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3565919606' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:05:05.162 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:05:05.162 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:05:05.162 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:05:05.162 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:05:05.162 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:05:05.162 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:05:05.163 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-20T09:05:05.163 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:05:05.163 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:05:05.163 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:05:05.163 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:05:05.163 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:05:05.163 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:05:05.177 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:04 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3565919606' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:05:05.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:04 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3565919606' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:05:05.921 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:05:05.922 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:05:05.933 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:05:05.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:05.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:05:05.964 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:05:06.040 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:05:06.067 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:05 smithi026 ceph-mon[68309]: pgmap v742: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 29 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 287 op/s; 27988/281835 objects degraded (9.931%); 11884/281835 objects misplaced (4.217%); 0 B/s, 0 objects/s recovering 2024-03-20T09:05:06.070 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:06.129 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:05 smithi115 ceph-mon[72476]: pgmap v742: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 29 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 287 op/s; 27988/281835 objects degraded (9.931%); 11884/281835 objects misplaced (4.217%); 0 B/s, 0 objects/s recovering 2024-03-20T09:05:06.170 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/exp.3.d.5 2024-03-20T09:05:06.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:05 smithi122 ceph-mon[73495]: pgmap v742: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 29 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 287 op/s; 27988/281835 objects degraded (9.931%); 11884/281835 objects misplaced (4.217%); 0 B/s, 0 objects/s recovering 2024-03-20T09:05:06.219 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:05:06.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:06.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:05:06.323 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:05:06.350 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:06.368 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:05:06.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:06.371 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:05:06.472 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:05:06.498 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:06.516 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:05:06.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:06.519 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:05:06.620 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:05:06.648 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:06.674 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:05:06.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:06.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:05:06.688 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:05:06.688 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:05:06.777 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:05:06.804 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:06.824 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:05:06.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:06.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:05:06.928 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:05:06.954 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:06.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:06 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 27988/281835 objects degraded (9.931%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:05:06.973 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:05:06.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:06.976 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:05:07.077 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:05:07.104 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:07.123 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:05:07.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:07.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:05:07.227 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:05:07.251 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:06 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 27988/281835 objects degraded (9.931%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:05:07.254 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:07.275 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:05:07.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:07.278 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:05:07.379 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:05:07.405 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:07.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:06 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 27988/281835 objects degraded (9.931%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:05:07.423 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:05:07.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:07.426 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:05:07.528 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:05:07.554 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:07.572 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:05:07.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:07.574 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:05:07.675 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:05:07.706 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:07.745 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:05:07.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:07.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:05:07.851 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:05:07.877 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:07.896 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:05:07.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:07.898 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:05:07.923 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:05:07.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:07 smithi026 ceph-mon[68309]: pgmap v743: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 29 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 295 op/s; 27857/280599 objects degraded (9.928%); 11830/280599 objects misplaced (4.216%) 2024-03-20T09:05:07.998 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:05:08.024 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:08.043 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:05:08.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:08.046 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:05:08.147 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:05:08.171 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:07 smithi122 ceph-mon[73495]: pgmap v743: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 29 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 295 op/s; 27857/280599 objects degraded (9.928%); 11830/280599 objects misplaced (4.216%) 2024-03-20T09:05:08.173 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:08.191 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:05:08.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:08.194 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:05:08.295 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:05:08.323 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:08.345 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:05:08.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:08.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:05:08.367 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-20T09:05:08.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:07 smithi115 ceph-mon[72476]: pgmap v743: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 29 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 295 op/s; 27857/280599 objects degraded (9.928%); 11830/280599 objects misplaced (4.216%) 2024-03-20T09:05:08.449 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:05:08.475 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:08.495 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:05:08.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:08.497 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:05:08.598 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:05:08.627 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:08.646 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:05:08.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:08.648 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:05:08.749 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:05:08.777 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:08.795 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:05:08.800 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:05:08.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:08.804 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:05:08.905 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:05:08.934 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:08.956 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:05:08.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:08.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:05:09.061 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:05:09.088 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:09.103 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:05:09.110 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:05:09.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:09.114 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:05:09.218 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:05:09.244 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:09.263 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:05:09.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:09.265 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:05:09.366 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:05:09.393 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:09 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2674282083' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:05:09.394 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:09 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:05:09.397 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:09.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:09 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2674282083' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:05:09.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:09 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:05:09.417 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:05:09.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:09.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:05:09.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:09 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2674282083' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:05:09.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:09 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:05:09.521 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:05:09.549 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:09.704 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:05:10.198 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:05:10.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:10.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:05:10.303 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:05:10.330 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:10.349 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:05:10.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:10.352 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:05:10.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:10 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:05:10.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:10 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:05:10.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:10 smithi115 ceph-mon[72476]: osdmap e288: 12 total, 11 up, 11 in 2024-03-20T09:05:10.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:10 smithi115 ceph-mon[72476]: pgmap v745: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 29 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 278 op/s; 27824/279498 objects degraded (9.955%); 11830/279498 objects misplaced (4.233%) 2024-03-20T09:05:10.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:10 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:05:10.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:10 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:05:10.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:10 smithi122 ceph-mon[73495]: osdmap e288: 12 total, 11 up, 11 in 2024-03-20T09:05:10.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:10 smithi122 ceph-mon[73495]: pgmap v745: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 29 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 278 op/s; 27824/279498 objects degraded (9.955%); 11830/279498 objects misplaced (4.233%) 2024-03-20T09:05:10.452 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:05:10.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:10 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:05:10.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:10 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:05:10.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:10 smithi026 ceph-mon[68309]: osdmap e288: 12 total, 11 up, 11 in 2024-03-20T09:05:10.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:10 smithi026 ceph-mon[68309]: pgmap v745: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 29 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 278 op/s; 27824/279498 objects degraded (9.955%); 11830/279498 objects misplaced (4.233%) 2024-03-20T09:05:10.482 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:11.266 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11] out_osds: [9] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11] 2024-03-20T09:05:11.267 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:05:11.267 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11] 2024-03-20T09:05:11.268 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd out 3 2024-03-20T09:05:11.859 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:11 smithi026 ceph-mon[68309]: pgmap v746: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 29 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 261 op/s; 27646/277755 objects degraded (9.953%); 11830/277755 objects misplaced (4.259%) 2024-03-20T09:05:11.860 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:11 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:05:11.860 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:11 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:05:11.860 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:11 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:05:12.091 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:05:12.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:12.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:05:12.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:11 smithi115 ceph-mon[72476]: pgmap v746: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 29 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 261 op/s; 27646/277755 objects degraded (9.953%); 11830/277755 objects misplaced (4.259%) 2024-03-20T09:05:12.203 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:11 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:05:12.203 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:11 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:05:12.203 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:11 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:05:12.203 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:05:12.204 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:11 smithi122 ceph-mon[73495]: pgmap v746: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 29 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 261 op/s; 27646/277755 objects degraded (9.953%); 11830/277755 objects misplaced (4.259%) 2024-03-20T09:05:12.204 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:11 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:05:12.204 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:11 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:05:12.204 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:11 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:05:12.220 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:05:12.230 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:12.383 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:05:12.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:12.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:05:12.488 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:05:12.518 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:12.569 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:05:12.592 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:05:12.611 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:05:12.637 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:05:12.657 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:05:12.679 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:05:12.701 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:05:12.756 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:05:12.756 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:05:12.756 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-03-20T09:05:12.756 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:05:12.770 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:05:12.772 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:12 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:05:12.772 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:12 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 27646/277755 objects degraded (9.953%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:05:12.772 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:12 smithi026 ceph-mon[68309]: osdmap e289: 12 total, 11 up, 11 in 2024-03-20T09:05:12.772 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:12 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3455097948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:05:12.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:12.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:05:12.874 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:05:12.900 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:12.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:12 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:05:12.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:12 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 27646/277755 objects degraded (9.953%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:05:12.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:12 smithi115 ceph-mon[72476]: osdmap e289: 12 total, 11 up, 11 in 2024-03-20T09:05:12.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:12 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3455097948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:05:12.920 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:05:12.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:12.923 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:05:12.923 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:05:12.942 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:05:12.962 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:05:12.984 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:05:12.985 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:05:12.985 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:05:12.985 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:05:12.985 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:05:12.985 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:05:12.985 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-20T09:05:12.985 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:05:12.985 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:05:12.985 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:05:12.986 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:05:12.986 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:05:12.986 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:05:13.021 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:05:13.024 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:05:13.060 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:13.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:12 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:05:13.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:12 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 27646/277755 objects degraded (9.953%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:05:13.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:12 smithi122 ceph-mon[73495]: osdmap e289: 12 total, 11 up, 11 in 2024-03-20T09:05:13.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:12 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3455097948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:05:13.211 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:05:13.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:13.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:05:13.315 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:05:13.341 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:13.362 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:05:13.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:13.364 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:05:13.465 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:05:13.495 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:13.519 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:05:13.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:13.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:05:13.625 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:05:13.653 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:13.676 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:05:13.677 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:05:13.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:13.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:05:13.719 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-20T09:05:13.782 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:05:13.811 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:13.830 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:05:13.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:13.835 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:05:13.936 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:05:13.972 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:14.355 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:05:14.355 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:05:14.462 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:05:14.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:14.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:05:14.566 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:05:14.595 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:14.616 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:05:14.617 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:14 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:05:14.617 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:14 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:05:14.617 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:14 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:05:14.618 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:14 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:05:14.618 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:14 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:05:14.618 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:14 smithi115 ceph-mon[72476]: pgmap v748: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 29 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 218 op/s; 27488/276600 objects degraded (9.938%); 11789/276600 objects misplaced (4.262%) 2024-03-20T09:05:14.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:14.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:05:14.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:14 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:05:14.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:14 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:05:14.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:14 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:05:14.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:14 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:05:14.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:14 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:05:14.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:14 smithi122 ceph-mon[73495]: pgmap v748: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 29 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 218 op/s; 27488/276600 objects degraded (9.938%); 11789/276600 objects misplaced (4.262%) 2024-03-20T09:05:14.720 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:05:14.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:14 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:05:14.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:14 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:05:14.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:14 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:05:14.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:14 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:05:14.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:14 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:05:14.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:14 smithi026 ceph-mon[68309]: pgmap v748: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 29 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 218 op/s; 27488/276600 objects degraded (9.938%); 11789/276600 objects misplaced (4.262%) 2024-03-20T09:05:14.752 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:14.774 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:05:14.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:14.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:05:14.877 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:05:14.905 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:15.966 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:05:15.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:15 smithi026 ceph-mon[68309]: pgmap v749: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 29 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 276 op/s; 27274/273972 objects degraded (9.955%); 11789/273972 objects misplaced (4.303%) 2024-03-20T09:05:16.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:15 smithi115 ceph-mon[72476]: pgmap v749: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 29 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 276 op/s; 27274/273972 objects degraded (9.955%); 11789/273972 objects misplaced (4.303%) 2024-03-20T09:05:16.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:15 smithi122 ceph-mon[73495]: pgmap v749: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 29 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 276 op/s; 27274/273972 objects degraded (9.955%); 11789/273972 objects misplaced (4.303%) 2024-03-20T09:05:16.357 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:05:16.713 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:05:16.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:16.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:05:16.823 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:05:16.856 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:16.962 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:05:16.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:16.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:05:16.976 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:16 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 27274/273972 objects degraded (9.955%), 9 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:05:17.067 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:05:17.093 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:17.115 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:05:17.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:17.122 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:05:17.223 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:05:17.253 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:16 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 27274/273972 objects degraded (9.955%), 9 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:05:17.261 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:17.276 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:05:17.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:17.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:05:17.384 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:05:17.408 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:16 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 27274/273972 objects degraded (9.955%), 9 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:05:17.409 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:17.432 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:05:17.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:17.433 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:05:17.534 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:05:17.564 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:17.581 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:05:17.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:17.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:05:17.687 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:05:17.725 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:17.827 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:05:17.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:17.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:05:17.930 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:05:17.967 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:18.145 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:17 smithi026 ceph-mon[68309]: pgmap v750: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 29 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 294 op/s; 27146/272826 objects degraded (9.950%); 11746/272826 objects misplaced (4.305%) 2024-03-20T09:05:18.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:17 smithi115 ceph-mon[72476]: pgmap v750: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 29 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 294 op/s; 27146/272826 objects degraded (9.950%); 11746/272826 objects misplaced (4.305%) 2024-03-20T09:05:18.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:17 smithi122 ceph-mon[73495]: pgmap v750: 47 pgs: 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 29 active+clean, 1 active+recovering+undersized+remapped, 7 active+undersized; 5.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 294 op/s; 27146/272826 objects degraded (9.950%); 11746/272826 objects misplaced (4.305%) 2024-03-20T09:05:18.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:18 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mon-a[68286]: 2024-03-20T09:05:18.179+0000 7f683364b700 -1 mon.a@0(leader).osd e289 definitely_dead 0 2024-03-20T09:05:18.585 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:05:18.618 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:05:18.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:18.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:05:18.723 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:05:18.752 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:18.850 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:05:18.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:18.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:05:18.954 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:05:18.970 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:05:18.970 INFO:teuthology.orchestra.run.smithi026.stderr:marked out osd.3. 2024-03-20T09:05:18.990 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:19.091 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:05:19.094 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:18 smithi026 ceph-mon[68309]: Health check failed: 0 slow ops, oldest one blocked for 35 sec, osd.6 has slow ops (SLOW_OPS) 2024-03-20T09:05:19.094 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:18 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/557626637' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:05:19.094 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:18 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/450743347' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-20T09:05:19.094 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:18 smithi026 ceph-mon[68309]: Client client.admin marked osd.3 out, while it was still marked up 2024-03-20T09:05:19.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:19.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:05:19.196 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:05:19.228 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:19.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:18 smithi115 ceph-mon[72476]: Health check failed: 0 slow ops, oldest one blocked for 35 sec, osd.6 has slow ops (SLOW_OPS) 2024-03-20T09:05:19.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:18 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/557626637' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:05:19.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:18 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/450743347' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-20T09:05:19.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:18 smithi115 ceph-mon[72476]: Client client.admin marked osd.3 out, while it was still marked up 2024-03-20T09:05:19.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:18 smithi122 ceph-mon[73495]: Health check failed: 0 slow ops, oldest one blocked for 35 sec, osd.6 has slow ops (SLOW_OPS) 2024-03-20T09:05:19.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:18 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/557626637' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:05:19.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:18 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/450743347' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-20T09:05:19.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:18 smithi122 ceph-mon[73495]: Client client.admin marked osd.3 out, while it was still marked up 2024-03-20T09:05:19.877 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:05:19.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:19.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:05:19.980 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:05:20.007 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:20.026 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:05:20.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:20.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:05:20.130 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:05:20.158 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:20.186 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:05:20.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:20.189 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:05:20.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:19 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:05:20.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:19 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/557626637' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:05:20.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:19 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/450743347' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-03-20T09:05:20.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:19 smithi026 ceph-mon[68309]: osdmap e290: 12 total, 11 up, 10 in 2024-03-20T09:05:20.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:19 smithi026 ceph-mon[68309]: pgmap v752: 47 pgs: 1 remapped+peering, 1 unknown, 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 7 active+undersized; 5.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 246 op/s; 27090/271830 objects degraded (9.966%); 9949/271830 objects misplaced (3.660%); 80 KiB/s, 1 objects/s recovering 2024-03-20T09:05:20.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:19 smithi026 ceph-mon[68309]: osdmap e291: 12 total, 11 up, 10 in 2024-03-20T09:05:20.291 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:05:20.315 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:19 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:05:20.315 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:19 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/557626637' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:05:20.315 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:19 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/450743347' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-03-20T09:05:20.315 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:19 smithi122 ceph-mon[73495]: osdmap e290: 12 total, 11 up, 10 in 2024-03-20T09:05:20.315 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:19 smithi122 ceph-mon[73495]: pgmap v752: 47 pgs: 1 remapped+peering, 1 unknown, 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 7 active+undersized; 5.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 246 op/s; 27090/271830 objects degraded (9.966%); 9949/271830 objects misplaced (3.660%); 80 KiB/s, 1 objects/s recovering 2024-03-20T09:05:20.315 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:19 smithi122 ceph-mon[73495]: osdmap e291: 12 total, 11 up, 10 in 2024-03-20T09:05:20.317 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:20.336 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:05:20.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:20.340 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:05:20.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:19 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:05:20.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:19 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/557626637' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:05:20.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:19 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/450743347' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-03-20T09:05:20.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:19 smithi115 ceph-mon[72476]: osdmap e290: 12 total, 11 up, 10 in 2024-03-20T09:05:20.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:19 smithi115 ceph-mon[72476]: pgmap v752: 47 pgs: 1 remapped+peering, 1 unknown, 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 7 active+undersized; 5.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 246 op/s; 27090/271830 objects degraded (9.966%); 9949/271830 objects misplaced (3.660%); 80 KiB/s, 1 objects/s recovering 2024-03-20T09:05:20.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:19 smithi115 ceph-mon[72476]: osdmap e291: 12 total, 11 up, 10 in 2024-03-20T09:05:20.441 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:05:20.472 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:20.546 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:05:20.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:20.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:05:20.650 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:05:20.676 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:20.696 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:05:20.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:20.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:05:20.800 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:05:20.830 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:20.869 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:05:20.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:20.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:05:20.974 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:05:21.002 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:21.029 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:05:21.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:21.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:05:21.133 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:05:21.167 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:21.270 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:05:21.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:21.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:05:21.340 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:05:21.363 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:05:21.376 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:05:21.384 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:05:21.404 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:05:21.404 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:21.428 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:05:21.430 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:05:21.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:21.435 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:05:21.450 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:05:21.479 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:05:21.505 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:05:21.529 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:05:21.536 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:05:21.556 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:05:21.568 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:21.587 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:05:21.588 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:05:21.588 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:05:21.588 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:05:21.588 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:05:21.588 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:05:21.588 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-20T09:05:21.588 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:05:21.588 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:05:21.588 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:05:21.588 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:05:21.588 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:05:21.588 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:05:21.923 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:05:21.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:21.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:05:22.023 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:05:22.023 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:05:22.023 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-03-20T09:05:22.023 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-03-20T09:05:22.023 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:05:22.028 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:05:22.058 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:22.080 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:05:22.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:22.084 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:05:22.185 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:05:22.213 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:22.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:21 smithi026 ceph-mon[68309]: osdmap e292: 12 total, 11 up, 10 in 2024-03-20T09:05:22.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:21 smithi026 ceph-mon[68309]: 3.c starting backfill to osd.10 from (0'0,0'0] MAX to 289'10731 2024-03-20T09:05:22.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:21 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:05:22.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:21 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:05:22.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:21 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2515406447' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:05:22.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:21 smithi026 ceph-mon[68309]: pgmap v755: 47 pgs: 1 peering, 3 remapped+peering, 1 unknown, 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 25 active+clean, 7 active+undersized; 5.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 180 op/s; 26990/270723 objects degraded (9.970%); 9949/270723 objects misplaced (3.675%); 107 KiB/s, 1 objects/s recovering 2024-03-20T09:05:22.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:21 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:05:22.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:21 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:05:22.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:21 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:05:22.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:21 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:05:22.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:21 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 27090/271830 objects degraded (9.966%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:05:22.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:21 smithi026 ceph-mon[68309]: osdmap e293: 12 total, 11 up, 10 in 2024-03-20T09:05:22.235 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:05:22.237 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:21 smithi122 ceph-mon[73495]: osdmap e292: 12 total, 11 up, 10 in 2024-03-20T09:05:22.237 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:21 smithi122 ceph-mon[73495]: 3.c starting backfill to osd.10 from (0'0,0'0] MAX to 289'10731 2024-03-20T09:05:22.237 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:21 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:05:22.237 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:21 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:05:22.237 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:21 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2515406447' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:05:22.237 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:21 smithi122 ceph-mon[73495]: pgmap v755: 47 pgs: 1 peering, 3 remapped+peering, 1 unknown, 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 25 active+clean, 7 active+undersized; 5.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 180 op/s; 26990/270723 objects degraded (9.970%); 9949/270723 objects misplaced (3.675%); 107 KiB/s, 1 objects/s recovering 2024-03-20T09:05:22.237 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:21 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:05:22.237 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:21 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:05:22.238 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:21 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:05:22.238 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:21 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:05:22.238 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:21 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 27090/271830 objects degraded (9.966%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:05:22.238 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:21 smithi122 ceph-mon[73495]: osdmap e293: 12 total, 11 up, 10 in 2024-03-20T09:05:22.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:22.239 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:05:22.340 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:05:22.369 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:21 smithi115 ceph-mon[72476]: osdmap e292: 12 total, 11 up, 10 in 2024-03-20T09:05:22.369 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:21 smithi115 ceph-mon[72476]: 3.c starting backfill to osd.10 from (0'0,0'0] MAX to 289'10731 2024-03-20T09:05:22.369 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:21 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:05:22.369 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:21 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:05:22.369 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:21 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2515406447' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:05:22.369 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:21 smithi115 ceph-mon[72476]: pgmap v755: 47 pgs: 1 peering, 3 remapped+peering, 1 unknown, 1 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 7 active+undersized+degraded, 1 active+recovering+degraded+remapped, 25 active+clean, 7 active+undersized; 5.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 180 op/s; 26990/270723 objects degraded (9.970%); 9949/270723 objects misplaced (3.675%); 107 KiB/s, 1 objects/s recovering 2024-03-20T09:05:22.370 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:21 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:05:22.370 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:21 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:05:22.370 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:21 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:05:22.370 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:21 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:05:22.370 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:21 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 27090/271830 objects degraded (9.966%), 9 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:05:22.370 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:21 smithi115 ceph-mon[72476]: osdmap e293: 12 total, 11 up, 10 in 2024-03-20T09:05:22.372 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:22.392 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:05:22.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:22.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:05:22.496 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:05:22.527 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:22.766 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-20T09:05:22.778 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:05:22.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:22.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:05:22.882 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:05:22.916 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:22.941 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:05:22.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:22.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:05:23.044 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:05:23.074 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:23.492 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:05:23.493 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:05:23.622 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:05:23.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:23.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:05:23.644 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:05:23.726 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:05:23.757 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:23.781 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:05:23.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:23.787 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:05:23.888 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:05:23.917 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:23.939 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:05:23.941 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:23.942 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:05:23.942 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:05:24.042 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:05:24.073 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:23 smithi122 ceph-mon[73495]: osdmap e294: 12 total, 11 up, 10 in 2024-03-20T09:05:24.073 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:23 smithi122 ceph-mon[73495]: pgmap v758: 47 pgs: 2 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 peering, 3 remapped+peering, 1 unknown, 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 25 active+clean, 6 active+undersized; 5.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 163 op/s; 26325/269871 objects degraded (9.755%); 15603/269871 objects misplaced (5.782%); 8.4 MiB/s, 133 objects/s recovering 2024-03-20T09:05:24.073 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:23 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:05:24.073 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:23 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:05:24.074 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:23 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:05:24.074 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:24.092 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:05:24.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:24.097 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:05:24.198 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:05:24.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:23 smithi026 ceph-mon[68309]: osdmap e294: 12 total, 11 up, 10 in 2024-03-20T09:05:24.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:23 smithi026 ceph-mon[68309]: pgmap v758: 47 pgs: 2 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 peering, 3 remapped+peering, 1 unknown, 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 25 active+clean, 6 active+undersized; 5.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 163 op/s; 26325/269871 objects degraded (9.755%); 15603/269871 objects misplaced (5.782%); 8.4 MiB/s, 133 objects/s recovering 2024-03-20T09:05:24.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:23 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:05:24.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:23 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:05:24.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:23 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:05:24.226 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:24.255 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:05:24.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:24.259 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:05:24.360 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:05:24.390 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:24.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:23 smithi115 ceph-mon[72476]: osdmap e294: 12 total, 11 up, 10 in 2024-03-20T09:05:24.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:23 smithi115 ceph-mon[72476]: pgmap v758: 47 pgs: 2 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 peering, 3 remapped+peering, 1 unknown, 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 25 active+clean, 6 active+undersized; 5.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 163 op/s; 26325/269871 objects degraded (9.755%); 15603/269871 objects misplaced (5.782%); 8.4 MiB/s, 133 objects/s recovering 2024-03-20T09:05:24.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:23 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:05:24.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:23 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:05:24.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:23 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:05:24.409 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:05:24.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:24.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:05:24.513 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:05:24.545 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:24.567 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:05:24.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:24.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:05:24.671 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:05:24.705 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:24.730 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:05:24.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:24.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:05:24.834 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:05:24.868 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:24.893 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:05:24.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:24.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:05:24.899 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:05:24.997 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:05:25.027 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:25.047 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:05:25.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:25.050 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:05:25.145 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:24 smithi026 ceph-mon[68309]: 1.8 scrub starts 2024-03-20T09:05:25.146 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:24 smithi026 ceph-mon[68309]: 1.8 scrub ok 2024-03-20T09:05:25.146 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:24 smithi026 ceph-mon[68309]: Health check cleared: SLOW_OPS (was: 0 slow ops, oldest one blocked for 35 sec, osd.6 has slow ops) 2024-03-20T09:05:25.151 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:05:25.179 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:24 smithi115 ceph-mon[72476]: 1.8 scrub starts 2024-03-20T09:05:25.179 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:24 smithi115 ceph-mon[72476]: 1.8 scrub ok 2024-03-20T09:05:25.179 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:24 smithi115 ceph-mon[72476]: Health check cleared: SLOW_OPS (was: 0 slow ops, oldest one blocked for 35 sec, osd.6 has slow ops) 2024-03-20T09:05:25.181 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:25.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:24 smithi122 ceph-mon[73495]: 1.8 scrub starts 2024-03-20T09:05:25.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:24 smithi122 ceph-mon[73495]: 1.8 scrub ok 2024-03-20T09:05:25.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:24 smithi122 ceph-mon[73495]: Health check cleared: SLOW_OPS (was: 0 slow ops, oldest one blocked for 35 sec, osd.6 has slow ops) 2024-03-20T09:05:25.205 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:05:25.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:25.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:05:25.311 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:05:25.340 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:25.494 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:05:26.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:25 smithi122 ceph-mon[73495]: pgmap v759: 47 pgs: 2 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 6 active+undersized; 5.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 153 op/s; 29348/268584 objects degraded (10.927%); 28578/268584 objects misplaced (10.640%); 18 MiB/s, 280 objects/s recovering 2024-03-20T09:05:26.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:25 smithi026 ceph-mon[68309]: pgmap v759: 47 pgs: 2 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 6 active+undersized; 5.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 153 op/s; 29348/268584 objects degraded (10.927%); 28578/268584 objects misplaced (10.640%); 18 MiB/s, 280 objects/s recovering 2024-03-20T09:05:26.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:25 smithi115 ceph-mon[72476]: pgmap v759: 47 pgs: 2 active+clean+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 6 active+undersized; 5.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 153 op/s; 29348/268584 objects degraded (10.927%); 28578/268584 objects misplaced (10.640%); 18 MiB/s, 280 objects/s recovering 2024-03-20T09:05:26.490 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:05:26.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:26.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:05:26.595 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:05:26.622 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:26.643 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:05:26.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:26.648 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:05:26.742 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:05:26.748 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:05:26.753 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:05:26.753 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:05:26.781 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:26.983 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:05:26.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:26.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:05:27.087 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:05:27.115 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:27.136 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:05:27.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:27.139 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:05:27.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:26 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:05:27.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:26 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:05:27.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:26 smithi122 ceph-mon[73495]: osdmap e295: 12 total, 11 up, 10 in 2024-03-20T09:05:27.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:26 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 29348/268584 objects degraded (10.927%), 13 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:05:27.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:26 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:05:27.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:26 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:05:27.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:26 smithi026 ceph-mon[68309]: osdmap e295: 12 total, 11 up, 10 in 2024-03-20T09:05:27.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:26 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 29348/268584 objects degraded (10.927%), 13 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:05:27.240 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:05:27.272 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:27.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:26 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:05:27.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:26 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:05:27.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:26 smithi115 ceph-mon[72476]: osdmap e295: 12 total, 11 up, 10 in 2024-03-20T09:05:27.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:26 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 29348/268584 objects degraded (10.927%), 13 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:05:27.632 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-20T09:05:27.723 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:05:27.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:27.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:05:27.829 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:05:27.866 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:27.954 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:05:28.069 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:05:28.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:28.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:05:28.173 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:05:28.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:27 smithi122 ceph-mon[73495]: pgmap v761: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 6 active+undersized; 5.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 155 op/s; 27374/267924 objects degraded (10.217%); 28508/267924 objects misplaced (10.640%); 36 MiB/s, 579 objects/s recovering 2024-03-20T09:05:28.202 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:27 smithi026 ceph-mon[68309]: pgmap v761: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 6 active+undersized; 5.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 155 op/s; 27374/267924 objects degraded (10.217%); 28508/267924 objects misplaced (10.640%); 36 MiB/s, 579 objects/s recovering 2024-03-20T09:05:28.205 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:28.212 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:27 smithi115 ceph-mon[72476]: pgmap v761: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 6 active+undersized; 5.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 155 op/s; 27374/267924 objects degraded (10.217%); 28508/267924 objects misplaced (10.640%); 36 MiB/s, 579 objects/s recovering 2024-03-20T09:05:28.355 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:05:28.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:28.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:05:28.459 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:05:28.491 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:28.511 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:05:28.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:28.514 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:05:28.615 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:05:28.648 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:28.648 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7, 8, 10, 11] out_osds: [9, 3] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11] 2024-03-20T09:05:28.648 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:05:28.648 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-20T09:05:28.648 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-20T09:05:28.801 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:05:28.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:28.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:05:28.906 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:05:28.937 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:05:28.943 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:29.128 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:28 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/854643819' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:05:29.145 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:05:29.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:29.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:05:29.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:28 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/854643819' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:05:29.248 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:05:29.274 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:28 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/854643819' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:05:29.277 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:29.351 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:05:29.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:29.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:05:29.456 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:05:29.486 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:29.503 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:05:29.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:29.506 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:05:29.593 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:05:29.607 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:05:29.615 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:05:29.634 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:29.641 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:05:29.654 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:05:29.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:29.657 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:05:29.665 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:05:29.690 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:05:29.715 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:05:29.736 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:05:29.757 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:05:29.758 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:05:29.777 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:05:29.788 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:29.797 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:05:29.824 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:05:29.825 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:05:29.825 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:05:29.825 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:05:29.825 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:05:29.825 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:05:29.825 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-20T09:05:29.825 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:05:29.825 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:05:29.825 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:05:29.826 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:05:29.826 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:05:29.826 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:05:29.844 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:05:29.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:29.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:05:29.947 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:05:29.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:29 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:05:29.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:29 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:05:29.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:29 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:05:29.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:29 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/854643819' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:05:29.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:29 smithi026 ceph-mon[68309]: osdmap e296: 12 total, 11 up, 10 in 2024-03-20T09:05:29.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:29 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/773774494' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:05:29.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:29 smithi026 ceph-mon[68309]: pgmap v763: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 6 active+undersized; 5.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 119 op/s; 27342/267519 objects degraded (10.221%); 28157/267519 objects misplaced (10.525%); 30 MiB/s, 487 objects/s recovering 2024-03-20T09:05:29.979 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:29.999 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:05:29.999 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [ 2024-03-20T09:05:29.999 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:30.000 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "MOSDPGPushReply(3.b 296/291 [PushReplyOp(3:d659b937:::benchmark_data_smithi122_119671_object19671:head)])", 2024-03-20T09:05:30.000 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:29.996951+0000", 2024-03-20T09:05:30.000 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.0022337030000000001, 2024-03-20T09:05:30.000 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0022556210000000002, 2024-03-20T09:05:30.000 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:30.000 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:05:30.000 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:30.000 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:30.000 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:30.000 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:29.996951+0000", 2024-03-20T09:05:30.000 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:30.000 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:30.000 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:30.000 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:30.001 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:29.996951+0000", 2024-03-20T09:05:30.001 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:30.001 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:30.001 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:30.001 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:30.001 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:29.996952+0000", 2024-03-20T09:05:30.001 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.032e-06 2024-03-20T09:05:30.001 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:30.001 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:30.001 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:30.001 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:29.996958+0000", 2024-03-20T09:05:30.001 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.421e-06 2024-03-20T09:05:30.001 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:30.001 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:30.002 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:30.002 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:29.996972+0000", 2024-03-20T09:05:30.002 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3423e-05 2024-03-20T09:05:30.002 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:30.002 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:30.002 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:30.002 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:29.996981+0000", 2024-03-20T09:05:30.002 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.0110000000000006e-06 2024-03-20T09:05:30.002 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:30.002 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:30.002 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:30.002 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:29.997034+0000", 2024-03-20T09:05:30.002 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3650000000000003e-05 2024-03-20T09:05:30.002 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:30.003 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:30.003 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:30.003 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:30.003 INFO:teuthology.orchestra.run.smithi115.stdout: ], 2024-03-20T09:05:30.003 INFO:teuthology.orchestra.run.smithi115.stdout: "num_ops": 1 2024-03-20T09:05:30.003 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:05:30.026 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:29 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:05:30.026 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:29 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:05:30.026 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:29 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:05:30.026 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:29 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/854643819' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:05:30.026 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:29 smithi115 ceph-mon[72476]: osdmap e296: 12 total, 11 up, 10 in 2024-03-20T09:05:30.026 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:29 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/773774494' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:05:30.026 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:29 smithi115 ceph-mon[72476]: pgmap v763: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 6 active+undersized; 5.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 119 op/s; 27342/267519 objects degraded (10.221%); 28157/267519 objects misplaced (10.525%); 30 MiB/s, 487 objects/s recovering 2024-03-20T09:05:30.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:29 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:05:30.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:29 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:05:30.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:29 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:05:30.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:29 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/854643819' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:05:30.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:29 smithi122 ceph-mon[73495]: osdmap e296: 12 total, 11 up, 10 in 2024-03-20T09:05:30.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:29 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/773774494' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:05:30.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:29 smithi122 ceph-mon[73495]: pgmap v763: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 6 active+undersized; 5.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 119 op/s; 27342/267519 objects degraded (10.221%); 28157/267519 objects misplaced (10.525%); 30 MiB/s, 487 objects/s recovering 2024-03-20T09:05:30.840 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-20T09:05:31.174 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:05:31.488 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:05:31.489 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:05:31.530 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:05:31.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:31.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:05:31.563 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:05:31.634 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:05:31.663 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:31.684 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:05:31.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:31.687 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:05:31.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:31 smithi026 ceph-mon[68309]: pgmap v764: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 6 active+undersized; 5.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 110 op/s; 27312/267171 objects degraded (10.223%); 26932/267171 objects misplaced (10.080%); 35 MiB/s, 562 objects/s recovering 2024-03-20T09:05:31.787 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:05:31.820 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:31.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:31 smithi122 ceph-mon[73495]: pgmap v764: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 6 active+undersized; 5.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 110 op/s; 27312/267171 objects degraded (10.223%); 26932/267171 objects misplaced (10.080%); 35 MiB/s, 562 objects/s recovering 2024-03-20T09:05:32.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:31 smithi115 ceph-mon[72476]: pgmap v764: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 6 active+undersized; 5.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 110 op/s; 27312/267171 objects degraded (10.223%); 26932/267171 objects misplaced (10.080%); 35 MiB/s, 562 objects/s recovering 2024-03-20T09:05:32.169 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:05:32.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:32 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 27312/267171 objects degraded (10.223%), 13 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:05:32.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:32 smithi026 ceph-mon[68309]: osdmap e297: 12 total, 11 up, 10 in 2024-03-20T09:05:33.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:32 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 27312/267171 objects degraded (10.223%), 13 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:05:33.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:32 smithi115 ceph-mon[72476]: osdmap e297: 12 total, 11 up, 10 in 2024-03-20T09:05:33.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:32 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 27312/267171 objects degraded (10.223%), 13 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:05:33.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:32 smithi122 ceph-mon[73495]: osdmap e297: 12 total, 11 up, 10 in 2024-03-20T09:05:33.323 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:05:33.490 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:05:33.614 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:05:33.614 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:05:33.614 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 297, 2024-03-20T09:05:33.614 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "852fbb68-e696-11ee-95c9-87774f69a715", 2024-03-20T09:05:33.614 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-03-20T08:48:35.281088+0000", 2024-03-20T09:05:33.614 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-03-20T09:05:31.907888+0000", 2024-03-20T09:05:33.615 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2024-03-20T09:04:33.695633+0000", 2024-03-20T09:05:33.615 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2024-03-20T09:05:18.957778+0000", 2024-03-20T09:05:33.615 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-20T09:05:33.615 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5799936, 2024-03-20T09:05:33.615 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2024-03-20T09:05:33.615 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2024-03-20T09:05:33.615 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2024-03-20T09:05:33.615 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2024-03-20T09:05:33.615 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2024-03-20T09:05:33.615 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:05:33.615 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 26, 2024-03-20T09:05:33.615 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2024-03-20T09:05:33.615 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-20T09:05:33.615 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-20T09:05:33.615 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2024-03-20T09:05:33.616 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 3, 2024-03-20T09:05:33.616 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 12, 2024-03-20T09:05:33.616 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "luminous", 2024-03-20T09:05:33.616 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "jewel", 2024-03-20T09:05:33.616 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "squid", 2024-03-20T09:05:33.616 INFO:teuthology.orchestra.run.smithi026.stdout: "allow_crimson": false, 2024-03-20T09:05:33.616 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2024-03-20T09:05:33.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.616 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2024-03-20T09:05:33.616 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": ".mgr", 2024-03-20T09:05:33.616 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-03-20T08:52:17.442324+0000", 2024-03-20T09:05:33.616 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-03-20T09:05:33.616 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-03-20T09:05:33.617 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-03-20T09:05:33.617 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 3, 2024-03-20T09:05:33.617 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 2, 2024-03-20T09:05:33.617 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-03-20T09:05:33.617 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-03-20T09:05:33.617 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-03-20T09:05:33.617 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-03-20T09:05:33.617 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-20T09:05:33.617 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-03-20T09:05:33.617 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-03-20T09:05:33.617 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 21, 2024-03-20T09:05:33.617 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 21, 2024-03-20T09:05:33.618 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 21, 2024-03-20T09:05:33.618 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 21, 2024-03-20T09:05:33.618 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 21, 2024-03-20T09:05:33.618 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-03-20T09:05:33.618 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-03-20T09:05:33.618 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-03-20T09:05:33.618 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-03-20T09:05:33.618 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-03-20T09:05:33.618 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-03-20T09:05:33.618 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-03-20T09:05:33.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.618 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "254", 2024-03-20T09:05:33.618 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-03-20T09:05:33.619 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-20T09:05:33.619 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "252", 2024-03-20T09:05:33.619 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-03-20T09:05:33.619 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-03-20T09:05:33.619 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-03-20T09:05:33.619 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-03-20T09:05:33.619 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-03-20T09:05:33.619 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-03-20T09:05:33.619 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-03-20T09:05:33.619 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-03-20T09:05:33.619 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-03-20T09:05:33.619 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-03-20T09:05:33.619 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-03-20T09:05:33.619 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-03-20T09:05:33.620 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-03-20T09:05:33.620 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-03-20T09:05:33.620 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-03-20T09:05:33.620 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-20T09:05:33.620 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-20T09:05:33.620 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-20T09:05:33.620 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-03-20T09:05:33.620 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-03-20T09:05:33.620 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-03-20T09:05:33.620 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-03-20T09:05:33.620 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-03-20T09:05:33.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.621 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-03-20T09:05:33.621 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-03-20T09:05:33.621 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-03-20T09:05:33.621 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-03-20T09:05:33.621 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-03-20T09:05:33.621 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-03-20T09:05:33.621 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-03-20T09:05:33.621 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-03-20T09:05:33.621 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-03-20T09:05:33.621 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-03-20T09:05:33.621 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-03-20T09:05:33.621 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2024-03-20T09:05:33.621 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_max": 32, 2024-03-20T09:05:33.621 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2024-03-20T09:05:33.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.622 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-03-20T09:05:33.622 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": {} 2024-03-20T09:05:33.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.622 INFO:teuthology.orchestra.run.smithi026.stdout: "read_balance": { 2024-03-20T09:05:33.622 INFO:teuthology.orchestra.run.smithi026.stdout: "score_type": "Fair distribution", 2024-03-20T09:05:33.622 INFO:teuthology.orchestra.run.smithi026.stdout: "score_acting": 2.2799999713897705, 2024-03-20T09:05:33.622 INFO:teuthology.orchestra.run.smithi026.stdout: "score_stable": 2.2799999713897705, 2024-03-20T09:05:33.622 INFO:teuthology.orchestra.run.smithi026.stdout: "optimal_score": 0.75, 2024-03-20T09:05:33.622 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_acting": 1.7100000381469727, 2024-03-20T09:05:33.622 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_stable": 1.7100000381469727, 2024-03-20T09:05:33.622 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity_weighted": 0.92150616645812988, 2024-03-20T09:05:33.622 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity": 0.92000001668930054, 2024-03-20T09:05:33.623 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-03-20T09:05:33.623 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.623 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 3, 2024-03-20T09:05:33.623 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "unique_pool_1", 2024-03-20T09:05:33.623 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-03-20T09:02:48.375175+0000", 2024-03-20T09:05:33.623 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-03-20T09:05:33.623 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-03-20T09:05:33.623 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-03-20T09:05:33.623 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 3, 2024-03-20T09:05:33.623 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 2, 2024-03-20T09:05:33.623 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-03-20T09:05:33.623 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-03-20T09:05:33.624 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-03-20T09:05:33.624 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-03-20T09:05:33.624 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-20T09:05:33.624 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-03-20T09:05:33.624 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-03-20T09:05:33.624 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 26, 2024-03-20T09:05:33.624 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 26, 2024-03-20T09:05:33.624 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 26, 2024-03-20T09:05:33.624 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 26, 2024-03-20T09:05:33.624 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 26, 2024-03-20T09:05:33.624 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-03-20T09:05:33.624 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-03-20T09:05:33.624 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-03-20T09:05:33.624 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-03-20T09:05:33.625 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-03-20T09:05:33.625 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-03-20T09:05:33.625 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-03-20T09:05:33.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.625 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "233", 2024-03-20T09:05:33.625 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-03-20T09:05:33.625 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-20T09:05:33.625 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "231", 2024-03-20T09:05:33.625 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-03-20T09:05:33.625 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-03-20T09:05:33.625 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-03-20T09:05:33.625 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-03-20T09:05:33.626 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-03-20T09:05:33.626 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-03-20T09:05:33.626 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-03-20T09:05:33.626 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-03-20T09:05:33.626 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-03-20T09:05:33.626 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-03-20T09:05:33.626 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-03-20T09:05:33.626 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-03-20T09:05:33.626 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-03-20T09:05:33.626 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-03-20T09:05:33.626 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-03-20T09:05:33.626 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-20T09:05:33.626 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-20T09:05:33.627 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-20T09:05:33.627 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-03-20T09:05:33.627 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-03-20T09:05:33.627 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-03-20T09:05:33.627 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-03-20T09:05:33.627 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-03-20T09:05:33.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.627 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-03-20T09:05:33.627 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-03-20T09:05:33.627 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-03-20T09:05:33.627 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-03-20T09:05:33.627 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-03-20T09:05:33.627 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-03-20T09:05:33.627 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-03-20T09:05:33.628 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-03-20T09:05:33.628 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-03-20T09:05:33.628 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-03-20T09:05:33.628 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-03-20T09:05:33.628 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2024-03-20T09:05:33.628 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-03-20T09:05:33.628 INFO:teuthology.orchestra.run.smithi026.stdout: "rados": {} 2024-03-20T09:05:33.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.628 INFO:teuthology.orchestra.run.smithi026.stdout: "read_balance": { 2024-03-20T09:05:33.628 INFO:teuthology.orchestra.run.smithi026.stdout: "score_type": "Fair distribution", 2024-03-20T09:05:33.628 INFO:teuthology.orchestra.run.smithi026.stdout: "score_acting": 2.7699999809265137, 2024-03-20T09:05:33.628 INFO:teuthology.orchestra.run.smithi026.stdout: "score_stable": 2.7699999809265137, 2024-03-20T09:05:33.628 INFO:teuthology.orchestra.run.smithi026.stdout: "optimal_score": 0.75, 2024-03-20T09:05:33.628 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_acting": 2.0799999237060547, 2024-03-20T09:05:33.629 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_stable": 2.0799999237060547, 2024-03-20T09:05:33.629 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity_weighted": 0.92150616645812988, 2024-03-20T09:05:33.629 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity": 0.92000001668930054, 2024-03-20T09:05:33.629 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-03-20T09:05:33.629 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.629 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.629 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:05:33.629 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:05:33.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.629 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-03-20T09:05:33.629 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "ad3fd9c0-316f-4fe5-a63b-d480087c1729", 2024-03-20T09:05:33.629 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:05:33.629 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:05:33.629 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0.4982452392578125, 2024-03-20T09:05:33.630 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:05:33.630 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:05:33.630 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:05:33.630 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 8, 2024-03-20T09:05:33.630 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 274, 2024-03-20T09:05:33.630 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:05:33.630 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:05:33.630 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:05:33.630 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.630 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.630 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2024-03-20T09:05:33.630 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:05:33.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.631 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.631 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2024-03-20T09:05:33.631 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:05:33.631 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.631 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.631 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:05:33.631 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.631 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.631 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2024-03-20T09:05:33.631 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:05:33.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.632 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.632 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2024-03-20T09:05:33.632 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:05:33.632 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.632 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.632 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:05:33.632 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.632 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.632 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2024-03-20T09:05:33.632 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:05:33.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.633 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.633 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2024-03-20T09:05:33.633 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:05:33.633 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.633 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.633 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:05:33.633 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.633 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.633 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2024-03-20T09:05:33.633 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:05:33.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.634 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.634 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2024-03-20T09:05:33.634 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:05:33.634 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.634 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.634 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/2139805293", 2024-03-20T09:05:33.634 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/2139805293", 2024-03-20T09:05:33.634 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/2139805293", 2024-03-20T09:05:33.634 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/2139805293", 2024-03-20T09:05:33.634 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:05:33.634 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:05:33.635 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:05:33.635 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.635 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-03-20T09:05:33.635 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e85b94f5-2954-47a1-9c20-096f2606eb04", 2024-03-20T09:05:33.635 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:05:33.635 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:05:33.635 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0.85748291015625, 2024-03-20T09:05:33.635 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:05:33.635 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:05:33.635 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:05:33.635 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2024-03-20T09:05:33.635 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 291, 2024-03-20T09:05:33.636 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:05:33.636 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:05:33.636 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:05:33.636 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.636 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.636 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2024-03-20T09:05:33.636 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:05:33.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.636 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.636 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2024-03-20T09:05:33.636 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:05:33.636 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.637 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.637 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:05:33.637 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.637 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.637 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2024-03-20T09:05:33.637 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:05:33.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.637 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.637 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2024-03-20T09:05:33.637 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:05:33.637 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.637 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.638 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:05:33.638 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.638 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.638 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2024-03-20T09:05:33.638 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:05:33.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.638 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.638 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2024-03-20T09:05:33.638 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:05:33.638 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.639 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.639 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:05:33.639 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.639 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.639 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2024-03-20T09:05:33.639 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:05:33.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.639 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.639 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2024-03-20T09:05:33.639 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:05:33.639 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.639 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.640 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/3853355134", 2024-03-20T09:05:33.640 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/3853355134", 2024-03-20T09:05:33.640 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/3853355134", 2024-03-20T09:05:33.640 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/3853355134", 2024-03-20T09:05:33.640 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:05:33.640 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:05:33.640 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:05:33.640 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.640 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-03-20T09:05:33.640 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "dd2b10f0-741c-47a4-b224-ac6bfe2c14ef", 2024-03-20T09:05:33.641 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:05:33.641 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:05:33.641 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:05:33.641 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:05:33.641 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:05:33.641 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:05:33.641 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 18, 2024-03-20T09:05:33.641 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 293, 2024-03-20T09:05:33.641 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:05:33.641 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:05:33.641 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:05:33.641 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.641 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.642 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2024-03-20T09:05:33.642 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:05:33.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.642 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.642 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2024-03-20T09:05:33.642 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:05:33.642 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.642 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.642 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:05:33.642 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.642 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.643 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2024-03-20T09:05:33.643 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:05:33.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.643 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.643 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2024-03-20T09:05:33.643 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:05:33.643 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.643 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.643 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:05:33.643 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.644 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.644 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2024-03-20T09:05:33.644 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:05:33.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.644 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.644 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2024-03-20T09:05:33.644 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:05:33.644 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.644 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.644 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:05:33.644 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.645 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.645 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2024-03-20T09:05:33.645 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:05:33.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.645 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.645 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2024-03-20T09:05:33.645 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:05:33.645 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.645 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.645 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/3993357967", 2024-03-20T09:05:33.645 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/3993357967", 2024-03-20T09:05:33.645 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/3993357967", 2024-03-20T09:05:33.646 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/3993357967", 2024-03-20T09:05:33.646 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:05:33.646 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:05:33.646 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:05:33.646 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.646 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-03-20T09:05:33.646 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1", 2024-03-20T09:05:33.646 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:05:33.646 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 0, 2024-03-20T09:05:33.646 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-03-20T09:05:33.646 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:05:33.646 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:05:33.647 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:05:33.647 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 26, 2024-03-20T09:05:33.647 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 274, 2024-03-20T09:05:33.647 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:05:33.647 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:05:33.647 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:05:33.647 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.647 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.647 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2024-03-20T09:05:33.647 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:05:33.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.647 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.647 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2024-03-20T09:05:33.648 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:05:33.648 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.648 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.648 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:05:33.648 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.648 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.648 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2024-03-20T09:05:33.648 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:05:33.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.648 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.648 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2024-03-20T09:05:33.649 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:05:33.649 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.649 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.649 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:05:33.649 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.649 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.649 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2024-03-20T09:05:33.649 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:05:33.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.649 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.650 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2024-03-20T09:05:33.650 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:05:33.650 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.650 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.650 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:05:33.650 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.650 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.651 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2024-03-20T09:05:33.651 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:05:33.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.651 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.651 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2024-03-20T09:05:33.651 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:05:33.651 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.651 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.651 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/4110225509", 2024-03-20T09:05:33.651 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/4110225509", 2024-03-20T09:05:33.651 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/4110225509", 2024-03-20T09:05:33.651 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/4110225509", 2024-03-20T09:05:33.652 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:05:33.652 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:05:33.652 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:05:33.652 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.652 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-03-20T09:05:33.652 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "f14e34d5-9829-42a3-9b25-31a0976c5ef7", 2024-03-20T09:05:33.652 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:05:33.652 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:05:33.652 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:05:33.652 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:05:33.652 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:05:33.652 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:05:33.653 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 32, 2024-03-20T09:05:33.653 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 293, 2024-03-20T09:05:33.653 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:05:33.653 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:05:33.653 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:05:33.653 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.653 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.653 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6800", 2024-03-20T09:05:33.653 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:05:33.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.653 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.653 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6801", 2024-03-20T09:05:33.654 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:05:33.654 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.654 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.654 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:05:33.654 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.654 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.654 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6802", 2024-03-20T09:05:33.654 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:05:33.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.654 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.654 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6803", 2024-03-20T09:05:33.655 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:05:33.655 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.655 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.655 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:05:33.655 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.655 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.655 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6806", 2024-03-20T09:05:33.655 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:05:33.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.655 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.655 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6807", 2024-03-20T09:05:33.656 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:05:33.656 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.656 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.656 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:05:33.656 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.656 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.656 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6804", 2024-03-20T09:05:33.656 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:05:33.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.656 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.656 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6805", 2024-03-20T09:05:33.657 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:05:33.657 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.657 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.657 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6801/477923058", 2024-03-20T09:05:33.657 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6803/477923058", 2024-03-20T09:05:33.657 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6807/477923058", 2024-03-20T09:05:33.657 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6805/477923058", 2024-03-20T09:05:33.657 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:05:33.657 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:05:33.657 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:05:33.657 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.658 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-03-20T09:05:33.658 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "cb431b6a-540d-4318-ae35-d379e1df0185", 2024-03-20T09:05:33.658 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 0, 2024-03-20T09:05:33.658 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:05:33.658 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:05:33.658 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:05:33.658 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:05:33.658 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:05:33.658 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 41, 2024-03-20T09:05:33.658 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 265, 2024-03-20T09:05:33.658 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 274, 2024-03-20T09:05:33.658 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:05:33.659 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:05:33.659 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.659 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.659 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6808", 2024-03-20T09:05:33.659 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:05:33.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.659 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.659 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6809", 2024-03-20T09:05:33.659 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:05:33.659 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.659 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.660 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:05:33.660 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.660 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.660 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6810", 2024-03-20T09:05:33.660 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:05:33.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.660 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.660 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6811", 2024-03-20T09:05:33.660 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:05:33.660 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.660 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.661 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:05:33.661 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.661 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.661 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6814", 2024-03-20T09:05:33.661 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:05:33.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.661 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.661 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6815", 2024-03-20T09:05:33.661 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:05:33.661 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.661 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.662 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:05:33.662 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.662 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.662 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6812", 2024-03-20T09:05:33.662 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:05:33.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.662 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.662 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6813", 2024-03-20T09:05:33.662 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:05:33.662 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.662 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.663 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6809/1339268003", 2024-03-20T09:05:33.663 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6811/1339268003", 2024-03-20T09:05:33.663 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6815/1339268003", 2024-03-20T09:05:33.663 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6813/1339268003", 2024-03-20T09:05:33.663 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:05:33.663 INFO:teuthology.orchestra.run.smithi026.stdout: "exists" 2024-03-20T09:05:33.663 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.663 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2024-03-20T09:05:33.663 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "753ed830-e764-4ff3-9d9b-87e7fa748b81", 2024-03-20T09:05:33.664 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:05:33.664 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:05:33.664 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:05:33.664 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:05:33.664 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:05:33.664 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:05:33.664 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 51, 2024-03-20T09:05:33.664 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 291, 2024-03-20T09:05:33.664 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:05:33.664 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:05:33.664 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:05:33.664 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.664 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.665 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6816", 2024-03-20T09:05:33.665 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:05:33.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.665 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.665 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6817", 2024-03-20T09:05:33.665 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:05:33.665 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.665 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.665 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:05:33.665 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.665 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.666 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6818", 2024-03-20T09:05:33.666 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:05:33.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.666 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.666 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6819", 2024-03-20T09:05:33.666 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:05:33.666 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.666 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.666 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:05:33.666 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.666 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.667 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6822", 2024-03-20T09:05:33.667 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:05:33.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.667 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.667 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6823", 2024-03-20T09:05:33.667 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:05:33.667 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.667 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.667 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:05:33.667 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.667 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.668 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6820", 2024-03-20T09:05:33.668 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:05:33.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.668 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.668 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6821", 2024-03-20T09:05:33.668 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:05:33.668 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.668 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.668 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6817/3278301843", 2024-03-20T09:05:33.668 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6819/3278301843", 2024-03-20T09:05:33.668 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6823/3278301843", 2024-03-20T09:05:33.668 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6821/3278301843", 2024-03-20T09:05:33.669 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:05:33.669 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:05:33.669 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:05:33.669 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.669 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2024-03-20T09:05:33.669 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "55bcbd37-7a6f-4ab1-b5a9-91a951444d86", 2024-03-20T09:05:33.669 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:05:33.669 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:05:33.669 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:05:33.669 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 0.058074951171875, 2024-03-20T09:05:33.669 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:05:33.669 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:05:33.670 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 60, 2024-03-20T09:05:33.670 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 274, 2024-03-20T09:05:33.670 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:05:33.670 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:05:33.670 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:05:33.670 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.670 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.670 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6824", 2024-03-20T09:05:33.670 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:05:33.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.670 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.670 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6825", 2024-03-20T09:05:33.671 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:05:33.671 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.671 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.671 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:05:33.671 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.671 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.671 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6826", 2024-03-20T09:05:33.671 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:05:33.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.671 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.671 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6827", 2024-03-20T09:05:33.672 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:05:33.672 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.672 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.672 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:05:33.672 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.672 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.672 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6830", 2024-03-20T09:05:33.672 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:05:33.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.672 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.672 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6831", 2024-03-20T09:05:33.673 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:05:33.673 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.673 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.673 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:05:33.673 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.673 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.673 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6828", 2024-03-20T09:05:33.673 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:05:33.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.673 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.673 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6829", 2024-03-20T09:05:33.674 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:05:33.674 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.674 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.674 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6825/3152424001", 2024-03-20T09:05:33.674 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6827/3152424001", 2024-03-20T09:05:33.674 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6831/3152424001", 2024-03-20T09:05:33.674 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6829/3152424001", 2024-03-20T09:05:33.674 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:05:33.674 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:05:33.674 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:05:33.674 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.675 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 8, 2024-03-20T09:05:33.675 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "bc86248e-6bd0-4369-b8a2-e593bc2a610d", 2024-03-20T09:05:33.675 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:05:33.675 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:05:33.675 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0.6006927490234375, 2024-03-20T09:05:33.675 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:05:33.675 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:05:33.675 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:05:33.675 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 70, 2024-03-20T09:05:33.675 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 233, 2024-03-20T09:05:33.675 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:05:33.675 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:05:33.675 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:05:33.675 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.676 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.676 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6800", 2024-03-20T09:05:33.676 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:05:33.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.676 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.676 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6801", 2024-03-20T09:05:33.676 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:05:33.676 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.676 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.676 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:05:33.677 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.677 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.677 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6802", 2024-03-20T09:05:33.677 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:05:33.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.677 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.677 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6803", 2024-03-20T09:05:33.677 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:05:33.677 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.677 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.677 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:05:33.678 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.678 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.678 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6806", 2024-03-20T09:05:33.678 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:05:33.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.678 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.678 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6807", 2024-03-20T09:05:33.678 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:05:33.678 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.678 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.679 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:05:33.679 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.679 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.679 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6804", 2024-03-20T09:05:33.679 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:05:33.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.679 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.679 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6805", 2024-03-20T09:05:33.679 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:05:33.679 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.679 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.680 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6801/1309426740", 2024-03-20T09:05:33.680 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6803/1309426740", 2024-03-20T09:05:33.680 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6807/1309426740", 2024-03-20T09:05:33.680 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6805/1309426740", 2024-03-20T09:05:33.680 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:05:33.680 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:05:33.680 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:05:33.680 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.680 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 9, 2024-03-20T09:05:33.680 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b20a4aa-032a-4c09-ac75-153c554424e7", 2024-03-20T09:05:33.680 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:05:33.680 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 0, 2024-03-20T09:05:33.680 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-03-20T09:05:33.681 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:05:33.681 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:05:33.681 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:05:33.681 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 80, 2024-03-20T09:05:33.681 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 257, 2024-03-20T09:05:33.681 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:05:33.681 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:05:33.681 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:05:33.681 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.681 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.681 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6808", 2024-03-20T09:05:33.681 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:05:33.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.682 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.682 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6809", 2024-03-20T09:05:33.682 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:05:33.682 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.682 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.682 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:05:33.682 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.682 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.682 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6810", 2024-03-20T09:05:33.682 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:05:33.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.683 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.683 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6811", 2024-03-20T09:05:33.683 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:05:33.683 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.683 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.683 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:05:33.683 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.683 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.683 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6814", 2024-03-20T09:05:33.683 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:05:33.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.684 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.684 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6815", 2024-03-20T09:05:33.684 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:05:33.684 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.684 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.684 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:05:33.684 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.684 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.684 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6812", 2024-03-20T09:05:33.684 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:05:33.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.685 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.685 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6813", 2024-03-20T09:05:33.685 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:05:33.685 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.685 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.685 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6809/2953210076", 2024-03-20T09:05:33.685 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6811/2953210076", 2024-03-20T09:05:33.685 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6815/2953210076", 2024-03-20T09:05:33.685 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6813/2953210076", 2024-03-20T09:05:33.685 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:05:33.685 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:05:33.685 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:05:33.685 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.686 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 10, 2024-03-20T09:05:33.686 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e62c1fb3-5dd1-4d0d-997c-d20633d19361", 2024-03-20T09:05:33.686 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:05:33.686 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:05:33.686 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:05:33.686 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:05:33.686 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:05:33.686 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:05:33.686 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 89, 2024-03-20T09:05:33.686 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 274, 2024-03-20T09:05:33.686 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:05:33.687 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:05:33.687 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:05:33.687 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.687 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.687 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6816", 2024-03-20T09:05:33.687 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:05:33.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.687 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.687 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6817", 2024-03-20T09:05:33.687 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:05:33.687 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.687 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.688 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:05:33.688 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.688 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.688 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6818", 2024-03-20T09:05:33.688 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:05:33.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.688 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.688 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6819", 2024-03-20T09:05:33.688 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:05:33.688 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.688 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.689 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:05:33.689 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.689 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.689 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6822", 2024-03-20T09:05:33.689 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:05:33.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.689 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.689 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6823", 2024-03-20T09:05:33.689 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:05:33.689 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.689 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.690 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:05:33.690 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.690 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.690 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6820", 2024-03-20T09:05:33.690 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:05:33.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.690 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.690 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6821", 2024-03-20T09:05:33.690 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:05:33.690 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.690 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.691 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6817/409394340", 2024-03-20T09:05:33.691 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6819/409394340", 2024-03-20T09:05:33.691 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6823/409394340", 2024-03-20T09:05:33.691 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6821/409394340", 2024-03-20T09:05:33.691 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:05:33.691 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:05:33.691 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:05:33.691 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.691 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 11, 2024-03-20T09:05:33.691 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b83994d5-e5ef-48b0-9758-c26843cec3b6", 2024-03-20T09:05:33.691 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:05:33.692 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:05:33.692 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:05:33.692 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:05:33.692 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:05:33.692 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:05:33.692 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 96, 2024-03-20T09:05:33.692 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 291, 2024-03-20T09:05:33.692 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:05:33.692 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:05:33.692 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:05:33.692 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.692 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.692 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6824", 2024-03-20T09:05:33.692 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:05:33.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.693 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.693 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6825", 2024-03-20T09:05:33.693 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:05:33.693 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.693 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.693 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:05:33.693 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.693 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.693 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6826", 2024-03-20T09:05:33.693 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:05:33.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.694 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.694 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6827", 2024-03-20T09:05:33.694 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:05:33.694 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.694 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.694 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:05:33.694 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.694 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.694 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6830", 2024-03-20T09:05:33.694 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:05:33.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.695 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.695 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6831", 2024-03-20T09:05:33.695 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:05:33.695 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.695 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.695 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:05:33.695 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:05:33.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.695 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:05:33.695 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6828", 2024-03-20T09:05:33.695 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:05:33.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.696 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:05:33.696 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6829", 2024-03-20T09:05:33.696 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:05:33.696 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.696 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.696 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6825/2704069354", 2024-03-20T09:05:33.696 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6827/2704069354", 2024-03-20T09:05:33.696 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6831/2704069354", 2024-03-20T09:05:33.696 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6829/2704069354", 2024-03-20T09:05:33.696 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:05:33.697 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:05:33.697 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:05:33.697 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.697 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.697 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:05:33.697 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2024-03-20T09:05:33.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.697 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-03-20T09:05:33.697 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:05:33.697 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:05:33.697 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:05:33.697 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:05:33.697 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:05:33.697 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:05:28.667693+0000", 2024-03-20T09:05:33.697 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:05:33.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.698 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-03-20T09:05:33.698 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:05:33.698 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:05:33.698 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:05:33.698 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:05:33.698 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:05:33.698 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:05:25.355751+0000", 2024-03-20T09:05:33.698 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:05:33.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.698 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-03-20T09:05:33.698 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:05:33.699 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:05:33.699 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:05:33.699 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:05:33.699 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:05:33.699 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:04:47.452084+0000", 2024-03-20T09:05:33.699 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:05:33.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.699 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-03-20T09:05:33.699 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:05:33.699 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:05:33.699 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:05:33.699 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:05:33.699 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 35063, 2024-03-20T09:05:33.700 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:04:38.382339+0000", 2024-03-20T09:05:33.700 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:05:33.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.700 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-03-20T09:05:33.700 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:05:33.700 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:05:33.700 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:05:33.700 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:05:33.700 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:05:33.700 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:04:59.307481+0000", 2024-03-20T09:05:33.700 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:05:33.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.701 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-03-20T09:05:33.701 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:05:33.701 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:05:33.701 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:05:33.701 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:05:33.701 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:05:33.701 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:03:53.609882+0000", 2024-03-20T09:05:33.701 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 273 2024-03-20T09:05:33.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.701 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2024-03-20T09:05:33.701 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:05:33.701 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:05:33.701 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:05:33.702 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:05:33.702 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:05:33.702 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:04:27.557282+0000", 2024-03-20T09:05:33.702 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:05:33.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.702 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2024-03-20T09:05:33.702 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:05:33.702 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:05:33.702 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:05:33.702 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:05:33.702 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:05:33.702 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:04:22.949240+0000", 2024-03-20T09:05:33.702 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:05:33.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.703 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 8, 2024-03-20T09:05:33.703 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:05:33.703 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:05:33.703 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:05:33.703 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:05:33.703 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:05:33.703 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:04:40.313726+0000", 2024-03-20T09:05:33.703 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:05:33.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.703 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 9, 2024-03-20T09:05:33.703 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:05:33.704 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:05:33.704 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:05:33.704 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:05:33.704 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 34736, 2024-03-20T09:05:33.704 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:04:24.794574+0000", 2024-03-20T09:05:33.704 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:05:33.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.704 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 10, 2024-03-20T09:05:33.704 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:05:33.704 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:05:33.704 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:05:33.704 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:05:33.704 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:05:33.705 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:05:21.018930+0000", 2024-03-20T09:05:33.705 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:05:33.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.705 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 11, 2024-03-20T09:05:33.705 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:05:33.705 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:05:33.705 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:05:33.705 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:05:33.705 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:05:33.705 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:04:45.248536+0000", 2024-03-20T09:05:33.705 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:05:33.705 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.705 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:05:33.705 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [], 2024-03-20T09:05:33.706 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2024-03-20T09:05:33.706 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_primaries": [], 2024-03-20T09:05:33.706 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [ 2024-03-20T09:05:33.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.706 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "1.8", 2024-03-20T09:05:33.706 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:05:33.706 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-03-20T09:05:33.706 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-03-20T09:05:33.706 INFO:teuthology.orchestra.run.smithi026.stdout: 3 2024-03-20T09:05:33.706 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.706 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.7", 2024-03-20T09:05:33.706 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:05:33.707 INFO:teuthology.orchestra.run.smithi026.stdout: 11, 2024-03-20T09:05:33.707 INFO:teuthology.orchestra.run.smithi026.stdout: 7 2024-03-20T09:05:33.707 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.707 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.b", 2024-03-20T09:05:33.707 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:05:33.707 INFO:teuthology.orchestra.run.smithi026.stdout: 4, 2024-03-20T09:05:33.707 INFO:teuthology.orchestra.run.smithi026.stdout: 11 2024-03-20T09:05:33.707 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.707 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.c", 2024-03-20T09:05:33.707 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:05:33.708 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-03-20T09:05:33.708 INFO:teuthology.orchestra.run.smithi026.stdout: 3 2024-03-20T09:05:33.708 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.708 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.d", 2024-03-20T09:05:33.708 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:05:33.708 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-03-20T09:05:33.708 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-03-20T09:05:33.708 INFO:teuthology.orchestra.run.smithi026.stdout: 9 2024-03-20T09:05:33.708 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.708 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.e", 2024-03-20T09:05:33.708 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:05:33.709 INFO:teuthology.orchestra.run.smithi026.stdout: 11, 2024-03-20T09:05:33.709 INFO:teuthology.orchestra.run.smithi026.stdout: 4 2024-03-20T09:05:33.709 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.709 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.10", 2024-03-20T09:05:33.709 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:05:33.709 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-03-20T09:05:33.709 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-03-20T09:05:33.709 INFO:teuthology.orchestra.run.smithi026.stdout: 9 2024-03-20T09:05:33.709 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.709 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.15", 2024-03-20T09:05:33.710 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:05:33.710 INFO:teuthology.orchestra.run.smithi026.stdout: 11, 2024-03-20T09:05:33.710 INFO:teuthology.orchestra.run.smithi026.stdout: 7 2024-03-20T09:05:33.710 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:05:33.710 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.19", 2024-03-20T09:05:33.710 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:05:33.710 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-03-20T09:05:33.710 INFO:teuthology.orchestra.run.smithi026.stdout: 4 2024-03-20T09:05:33.710 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:05:33.710 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.710 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:05:33.710 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2024-03-20T09:05:33.711 INFO:teuthology.orchestra.run.smithi026.stdout: "blocklist": { 2024-03-20T09:05:33.711 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/4291404191": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:05:33.711 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/642012566": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:05:33.711 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6801/1181459999": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:05:33.711 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/3984513952": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:05:33.711 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6800/1654945498": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:05:33.711 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6800/1181459999": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:05:33.711 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/137770981": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:05:33.711 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/1451237762": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:05:33.711 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/3938386631": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:05:33.711 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6801/1654945498": "2024-03-21T08:49:04.596754+0000" 2024-03-20T09:05:33.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.711 INFO:teuthology.orchestra.run.smithi026.stdout: "range_blocklist": {}, 2024-03-20T09:05:33.711 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2024-03-20T09:05:33.711 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2024-03-20T09:05:33.712 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2024-03-20T09:05:33.712 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2024-03-20T09:05:33.712 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2024-03-20T09:05:33.712 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2024-03-20T09:05:33.712 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2024-03-20T09:05:33.712 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:05:33.712 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2024-03-20T09:05:33.712 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2024-03-20T09:05:33.712 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2024-03-20T09:05:33.712 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2024-03-20T09:05:33.712 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {}, 2024-03-20T09:05:33.712 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": { 2024-03-20T09:05:33.712 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_enabled": false, 2024-03-20T09:05:33.713 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_bucket_count": 0, 2024-03-20T09:05:33.713 INFO:teuthology.orchestra.run.smithi026.stdout: "degraded_stretch_mode": 0, 2024-03-20T09:05:33.713 INFO:teuthology.orchestra.run.smithi026.stdout: "recovering_stretch_mode": 0, 2024-03-20T09:05:33.713 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_bucket": 0 2024-03-20T09:05:33.713 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:05:33.713 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:05:33.773 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:05:33.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:33.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:05:33.880 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:05:33.908 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:33 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:05:33.909 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:33 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:05:33.909 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:33 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1396096290' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:05:33.909 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:33 smithi026 ceph-mon[68309]: pgmap v766: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 6 active+undersized; 5.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 86 op/s; 27228/266544 objects degraded (10.215%); 26853/266544 objects misplaced (10.075%); 28 MiB/s, 450 objects/s recovering 2024-03-20T09:05:33.911 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:34.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:33 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:05:34.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:33 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:05:34.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:33 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1396096290' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:05:34.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:33 smithi115 ceph-mon[72476]: pgmap v766: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 6 active+undersized; 5.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 86 op/s; 27228/266544 objects degraded (10.215%); 26853/266544 objects misplaced (10.075%); 28 MiB/s, 450 objects/s recovering 2024-03-20T09:05:34.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:33 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:05:34.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:33 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:05:34.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:33 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1396096290' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:05:34.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:33 smithi122 ceph-mon[73495]: pgmap v766: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 6 active+undersized; 5.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 86 op/s; 27228/266544 objects degraded (10.215%); 26853/266544 objects misplaced (10.075%); 28 MiB/s, 450 objects/s recovering 2024-03-20T09:05:34.356 INFO:tasks.thrashosds.thrasher:j is {'epoch': 297, 'fsid': '852fbb68-e696-11ee-95c9-87774f69a715', 'created': '2024-03-20T08:48:35.281088+0000', 'modified': '2024-03-20T09:05:31.907888+0000', 'last_up_change': '2024-03-20T09:04:33.695633+0000', 'last_in_change': '2024-03-20T09:05:18.957778+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-20T08:52:17.442324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '254', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '252', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.2799999713897705, 'score_stable': 2.2799999713897705, 'optimal_score': 0.75, 'raw_score_acting': 1.7100000381469727, 'raw_score_stable': 1.7100000381469727, 'primary_affinity_weighted': 0.9215061664581299, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 3, 'pool_name': 'unique_pool_1', 'create_time': '2024-03-20T09:02:48.375175+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '233', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '231', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.7699999809265137, 'score_stable': 2.7699999809265137, 'optimal_score': 0.75, 'raw_score_acting': 2.0799999237060547, 'raw_score_stable': 2.0799999237060547, 'primary_affinity_weighted': 0.9215061664581299, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}], 'osds': [{'osd': 0, 'uuid': 'ad3fd9c0-316f-4fe5-a63b-d480087c1729', 'up': 1, 'in': 1, 'weight': 0.4982452392578125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 274, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6802', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6803', 'nonce': 2139805293}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6804', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6805', 'nonce': 2139805293}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6808', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6809', 'nonce': 2139805293}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6806', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6807', 'nonce': 2139805293}]}, 'public_addr': '172.21.15.26:6803/2139805293', 'cluster_addr': '172.21.15.26:6805/2139805293', 'heartbeat_back_addr': '172.21.15.26:6809/2139805293', 'heartbeat_front_addr': '172.21.15.26:6807/2139805293', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'e85b94f5-2954-47a1-9c20-096f2606eb04', 'up': 1, 'in': 1, 'weight': 0.85748291015625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 291, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6810', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6811', 'nonce': 3853355134}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6812', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6813', 'nonce': 3853355134}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6816', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6817', 'nonce': 3853355134}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6814', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6815', 'nonce': 3853355134}]}, 'public_addr': '172.21.15.26:6811/3853355134', 'cluster_addr': '172.21.15.26:6813/3853355134', 'heartbeat_back_addr': '172.21.15.26:6817/3853355134', 'heartbeat_front_addr': '172.21.15.26:6815/3853355134', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'dd2b10f0-741c-47a4-b224-ac6bfe2c14ef', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 293, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6818', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6819', 'nonce': 3993357967}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6820', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6821', 'nonce': 3993357967}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6824', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6825', 'nonce': 3993357967}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6822', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6823', 'nonce': 3993357967}]}, 'public_addr': '172.21.15.26:6819/3993357967', 'cluster_addr': '172.21.15.26:6821/3993357967', 'heartbeat_back_addr': '172.21.15.26:6825/3993357967', 'heartbeat_front_addr': '172.21.15.26:6823/3993357967', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 274, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6826', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6827', 'nonce': 4110225509}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6828', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6829', 'nonce': 4110225509}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6832', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6833', 'nonce': 4110225509}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6830', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6831', 'nonce': 4110225509}]}, 'public_addr': '172.21.15.26:6827/4110225509', 'cluster_addr': '172.21.15.26:6829/4110225509', 'heartbeat_back_addr': '172.21.15.26:6833/4110225509', 'heartbeat_front_addr': '172.21.15.26:6831/4110225509', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'f14e34d5-9829-42a3-9b25-31a0976c5ef7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 293, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6800', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6801', 'nonce': 477923058}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6802', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6803', 'nonce': 477923058}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6806', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6807', 'nonce': 477923058}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6804', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6805', 'nonce': 477923058}]}, 'public_addr': '172.21.15.115:6801/477923058', 'cluster_addr': '172.21.15.115:6803/477923058', 'heartbeat_back_addr': '172.21.15.115:6807/477923058', 'heartbeat_front_addr': '172.21.15.115:6805/477923058', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'cb431b6a-540d-4318-ae35-d379e1df0185', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 265, 'down_at': 274, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6808', 'nonce': 1339268003}, {'type': 'v1', 'addr': '172.21.15.115:6809', 'nonce': 1339268003}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6810', 'nonce': 1339268003}, {'type': 'v1', 'addr': '172.21.15.115:6811', 'nonce': 1339268003}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6814', 'nonce': 1339268003}, {'type': 'v1', 'addr': '172.21.15.115:6815', 'nonce': 1339268003}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6812', 'nonce': 1339268003}, {'type': 'v1', 'addr': '172.21.15.115:6813', 'nonce': 1339268003}]}, 'public_addr': '172.21.15.115:6809/1339268003', 'cluster_addr': '172.21.15.115:6811/1339268003', 'heartbeat_back_addr': '172.21.15.115:6815/1339268003', 'heartbeat_front_addr': '172.21.15.115:6813/1339268003', 'state': ['exists']}, {'osd': 6, 'uuid': '753ed830-e764-4ff3-9d9b-87e7fa748b81', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 291, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6816', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6817', 'nonce': 3278301843}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6818', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6819', 'nonce': 3278301843}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6822', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6823', 'nonce': 3278301843}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6820', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6821', 'nonce': 3278301843}]}, 'public_addr': '172.21.15.115:6817/3278301843', 'cluster_addr': '172.21.15.115:6819/3278301843', 'heartbeat_back_addr': '172.21.15.115:6823/3278301843', 'heartbeat_front_addr': '172.21.15.115:6821/3278301843', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '55bcbd37-7a6f-4ab1-b5a9-91a951444d86', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.058074951171875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 274, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6824', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6825', 'nonce': 3152424001}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6826', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6827', 'nonce': 3152424001}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6830', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6831', 'nonce': 3152424001}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6828', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6829', 'nonce': 3152424001}]}, 'public_addr': '172.21.15.115:6825/3152424001', 'cluster_addr': '172.21.15.115:6827/3152424001', 'heartbeat_back_addr': '172.21.15.115:6831/3152424001', 'heartbeat_front_addr': '172.21.15.115:6829/3152424001', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'bc86248e-6bd0-4369-b8a2-e593bc2a610d', 'up': 1, 'in': 1, 'weight': 0.6006927490234375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 70, 'up_thru': 233, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6800', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6801', 'nonce': 1309426740}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6802', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6803', 'nonce': 1309426740}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6806', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6807', 'nonce': 1309426740}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6804', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6805', 'nonce': 1309426740}]}, 'public_addr': '172.21.15.122:6801/1309426740', 'cluster_addr': '172.21.15.122:6803/1309426740', 'heartbeat_back_addr': '172.21.15.122:6807/1309426740', 'heartbeat_front_addr': '172.21.15.122:6805/1309426740', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '9b20a4aa-032a-4c09-ac75-153c554424e7', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 257, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6808', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6809', 'nonce': 2953210076}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6810', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6811', 'nonce': 2953210076}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6814', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6815', 'nonce': 2953210076}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6812', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6813', 'nonce': 2953210076}]}, 'public_addr': '172.21.15.122:6809/2953210076', 'cluster_addr': '172.21.15.122:6811/2953210076', 'heartbeat_back_addr': '172.21.15.122:6815/2953210076', 'heartbeat_front_addr': '172.21.15.122:6813/2953210076', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'e62c1fb3-5dd1-4d0d-997c-d20633d19361', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 89, 'up_thru': 274, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6816', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6817', 'nonce': 409394340}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6818', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6819', 'nonce': 409394340}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6822', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6823', 'nonce': 409394340}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6820', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6821', 'nonce': 409394340}]}, 'public_addr': '172.21.15.122:6817/409394340', 'cluster_addr': '172.21.15.122:6819/409394340', 'heartbeat_back_addr': '172.21.15.122:6823/409394340', 'heartbeat_front_addr': '172.21.15.122:6821/409394340', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'b83994d5-e5ef-48b0-9758-c26843cec3b6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 96, 'up_thru': 291, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6824', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6825', 'nonce': 2704069354}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6826', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6827', 'nonce': 2704069354}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6830', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6831', 'nonce': 2704069354}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6828', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6829', 'nonce': 2704069354}]}, 'public_addr': '172.21.15.122:6825/2704069354', 'cluster_addr': '172.21.15.122:6827/2704069354', 'heartbeat_back_addr': '172.21.15.122:6831/2704069354', 'heartbeat_front_addr': '172.21.15.122:6829/2704069354', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:05:28.667693+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:05:25.355751+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:04:47.452084+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 35063, 'last_purged_snaps_scrub': '2024-03-20T09:04:38.382339+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:04:59.307481+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:03:53.609882+0000', 'dead_epoch': 273}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:04:27.557282+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:04:22.949240+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:04:40.313726+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 34736, 'last_purged_snaps_scrub': '2024-03-20T09:04:24.794574+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:05:21.018930+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:04:45.248536+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '1.8', 'osds': [2, 4, 3]}, {'pgid': '3.7', 'osds': [11, 7]}, {'pgid': '3.b', 'osds': [4, 11]}, {'pgid': '3.c', 'osds': [6, 3]}, {'pgid': '3.d', 'osds': [6, 7, 9]}, {'pgid': '3.e', 'osds': [11, 4]}, {'pgid': '3.10', 'osds': [6, 2, 9]}, {'pgid': '3.15', 'osds': [11, 7]}, {'pgid': '3.19', 'osds': [1, 4]}], 'primary_temp': [], 'blocklist': {'172.21.15.26:0/4291404191': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/642012566': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:6801/1181459999': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/3984513952': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6800/1654945498': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6800/1181459999': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/137770981': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:0/1451237762': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/3938386631': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6801/1654945498': '2024-03-21T08:49:04.596754+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-20T09:05:34.356 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-03-20T09:05:34.741 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:05:34.862 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:05:34.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:34.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:05:34.966 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:05:34.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:34 smithi026 ceph-mon[68309]: osdmap e298: 12 total, 11 up, 10 in 2024-03-20T09:05:34.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:34 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:05:34.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:34 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:05:34.995 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:35.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:34 smithi115 ceph-mon[72476]: osdmap e298: 12 total, 11 up, 10 in 2024-03-20T09:05:35.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:34 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:05:35.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:34 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:05:35.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:34 smithi122 ceph-mon[73495]: osdmap e298: 12 total, 11 up, 10 in 2024-03-20T09:05:35.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:34 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:05:35.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:34 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:05:35.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:35 smithi026 ceph-mon[68309]: osdmap e299: 12 total, 11 up, 10 in 2024-03-20T09:05:35.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:35 smithi026 ceph-mon[68309]: pgmap v769: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 6 active+undersized; 5.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 104 op/s; 27167/265632 objects degraded (10.227%); 24709/265632 objects misplaced (9.302%); 35 MiB/s, 577 objects/s recovering 2024-03-20T09:05:36.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:35 smithi115 ceph-mon[72476]: osdmap e299: 12 total, 11 up, 10 in 2024-03-20T09:05:36.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:35 smithi115 ceph-mon[72476]: pgmap v769: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 6 active+undersized; 5.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 104 op/s; 27167/265632 objects degraded (10.227%); 24709/265632 objects misplaced (9.302%); 35 MiB/s, 577 objects/s recovering 2024-03-20T09:05:36.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:35 smithi122 ceph-mon[73495]: osdmap e299: 12 total, 11 up, 10 in 2024-03-20T09:05:36.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:35 smithi122 ceph-mon[73495]: pgmap v769: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 6 active+undersized; 5.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 104 op/s; 27167/265632 objects degraded (10.227%); 24709/265632 objects misplaced (9.302%); 35 MiB/s, 577 objects/s recovering 2024-03-20T09:05:36.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:36 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:05:36.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:36 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:05:36.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:36 smithi026 ceph-mon[68309]: osdmap e300: 12 total, 11 up, 10 in 2024-03-20T09:05:36.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:36 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1909650729' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:05:37.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:36 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:05:37.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:36 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:05:37.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:36 smithi115 ceph-mon[72476]: osdmap e300: 12 total, 11 up, 10 in 2024-03-20T09:05:37.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:36 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1909650729' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:05:37.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:36 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:05:37.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:36 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:05:37.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:36 smithi122 ceph-mon[73495]: osdmap e300: 12 total, 11 up, 10 in 2024-03-20T09:05:37.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:36 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1909650729' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:05:37.594 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:05:37.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:37 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:05:37.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:37 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1909650729' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:05:37.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:37 smithi026 ceph-mon[68309]: osdmap e301: 12 total, 11 up, 10 in 2024-03-20T09:05:37.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:37 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 27167/265632 objects degraded (10.227%), 13 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-03-20T09:05:37.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:37 smithi026 ceph-mon[68309]: pgmap v772: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 6 active+undersized; 5.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 113 op/s; 27102/265185 objects degraded (10.220%); 24632/265185 objects misplaced (9.289%); 34 MiB/s, 559 objects/s recovering 2024-03-20T09:05:38.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:37 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:05:38.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:37 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1909650729' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:05:38.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:37 smithi115 ceph-mon[72476]: osdmap e301: 12 total, 11 up, 10 in 2024-03-20T09:05:38.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:37 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 27167/265632 objects degraded (10.227%), 13 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-03-20T09:05:38.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:37 smithi115 ceph-mon[72476]: pgmap v772: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 6 active+undersized; 5.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 113 op/s; 27102/265185 objects degraded (10.220%); 24632/265185 objects misplaced (9.289%); 34 MiB/s, 559 objects/s recovering 2024-03-20T09:05:38.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:37 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:05:38.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:37 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1909650729' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:05:38.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:37 smithi122 ceph-mon[73495]: osdmap e301: 12 total, 11 up, 10 in 2024-03-20T09:05:38.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:37 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 27167/265632 objects degraded (10.227%), 13 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-03-20T09:05:38.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:37 smithi122 ceph-mon[73495]: pgmap v772: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 6 active+undersized; 5.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 113 op/s; 27102/265185 objects degraded (10.220%); 24632/265185 objects misplaced (9.289%); 34 MiB/s, 559 objects/s recovering 2024-03-20T09:05:39.357 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7, 8, 10, 11] out_osds: [9, 3] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11] 2024-03-20T09:05:39.357 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:05:39.357 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-03-20T09:05:39.358 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight-by-utilization 110 3.0 2024-03-20T09:05:39.777 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:05:39.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:39.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:05:39.881 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:05:39.909 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:39.927 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:05:39.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:39.930 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:05:39.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:39 smithi026 ceph-mon[68309]: pgmap v773: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 6 active+undersized; 5.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 105 op/s; 27043/264663 objects degraded (10.218%); 24197/264663 objects misplaced (9.143%); 27 MiB/s, 444 objects/s recovering 2024-03-20T09:05:40.030 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:05:40.067 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:39 smithi122 ceph-mon[73495]: pgmap v773: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 6 active+undersized; 5.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 105 op/s; 27043/264663 objects degraded (10.218%); 24197/264663 objects misplaced (9.143%); 27 MiB/s, 444 objects/s recovering 2024-03-20T09:05:40.068 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:40.089 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:05:40.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:40.091 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:05:40.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:39 smithi115 ceph-mon[72476]: pgmap v773: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 6 active+undersized; 5.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 105 op/s; 27043/264663 objects degraded (10.218%); 24197/264663 objects misplaced (9.143%); 27 MiB/s, 444 objects/s recovering 2024-03-20T09:05:40.192 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:05:40.225 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:41.670 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:05:41.671 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:05:41.671 INFO:teuthology.orchestra.run.smithi115.stdout: "complaint_time": 30, 2024-03-20T09:05:41.671 INFO:teuthology.orchestra.run.smithi115.stdout: "num_blocked_ops": 0 2024-03-20T09:05:41.671 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:05:41.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:41 smithi026 ceph-mon[68309]: pgmap v774: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 27 active+clean, 6 active+undersized; 5.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 98 op/s; 26992/264078 objects degraded (10.221%); 23814/264078 objects misplaced (9.018%); 23 MiB/s, 423 objects/s recovering 2024-03-20T09:05:42.017 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:41 smithi115 ceph-mon[72476]: pgmap v774: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 27 active+clean, 6 active+undersized; 5.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 98 op/s; 26992/264078 objects degraded (10.221%); 23814/264078 objects misplaced (9.018%); 23 MiB/s, 423 objects/s recovering 2024-03-20T09:05:42.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:41 smithi122 ceph-mon[73495]: pgmap v774: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 27 active+clean, 6 active+undersized; 5.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 98 op/s; 26992/264078 objects degraded (10.221%); 23814/264078 objects misplaced (9.018%); 23 MiB/s, 423 objects/s recovering 2024-03-20T09:05:42.348 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-20T09:05:42.680 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:05:42.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:05:42.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:05:42.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 26992/264078 objects degraded (10.221%), 12 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-03-20T09:05:42.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=232672708,config=3599132250,health=947575964,kv=4220410864}) 2024-03-20T09:05:42.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=3325467583,logm=470351074}) 2024-03-20T09:05:42.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=471040182}) 2024-03-20T09:05:42.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=815604070}) 2024-03-20T09:05:42.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2007105632}) 2024-03-20T09:05:42.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3740096149}) 2024-03-20T09:05:42.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3795267886}) 2024-03-20T09:05:42.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2203058994}) 2024-03-20T09:05:42.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2193314884}) 2024-03-20T09:05:42.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=639917931}) 2024-03-20T09:05:42.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2351137051}) 2024-03-20T09:05:42.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066117093}) 2024-03-20T09:05:42.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4106568028}) 2024-03-20T09:05:42.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2684561070}) 2024-03-20T09:05:42.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=777158838}) 2024-03-20T09:05:42.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3424711402}) 2024-03-20T09:05:42.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=71875596}) 2024-03-20T09:05:42.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3564559035}) 2024-03-20T09:05:42.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1551809073}) 2024-03-20T09:05:42.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4237774710}) 2024-03-20T09:05:42.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4162281157}) 2024-03-20T09:05:42.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3283980421}) 2024-03-20T09:05:42.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1561583635}) 2024-03-20T09:05:42.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3407208921}) 2024-03-20T09:05:42.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3506827474}) 2024-03-20T09:05:42.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1635750056}) 2024-03-20T09:05:42.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2823011674}) 2024-03-20T09:05:42.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2746114490}) 2024-03-20T09:05:42.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=330603124}) 2024-03-20T09:05:42.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1885476539}) 2024-03-20T09:05:42.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1577647937}) 2024-03-20T09:05:42.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3186591992}) 2024-03-20T09:05:42.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=74,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=2} crc {logm=2847818656,mdsmap=3984425268,mgr=427508465,mgr_command_descs=3831600018,mgr_metadata=2702834764,mgrstat=2918875162}) 2024-03-20T09:05:42.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=14,mon_config_key=86} crc {mgrstat=991574895,mon_config_key=568942078}) 2024-03-20T09:05:42.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1350994881}) 2024-03-20T09:05:42.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=92,monmap=5,osd_metadata=3} crc {mon_config_key=360779685,monmap=3481182261,osd_metadata=1402321155}) 2024-03-20T09:05:42.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=9,osd_pg_creating=1,osdmap=90} crc {osd_metadata=2746910592,osd_pg_creating=2148541147,osdmap=3580702051}) 2024-03-20T09:05:42.975 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2303323582}) 2024-03-20T09:05:42.975 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=10} crc {osdmap=2726515318}) 2024-03-20T09:05:42.975 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:05:42.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:05:42.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 26992/264078 objects degraded (10.221%), 12 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-03-20T09:05:42.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=232672708,config=3599132250,health=947575964,kv=4220410864}) 2024-03-20T09:05:42.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=3325467583,logm=470351074}) 2024-03-20T09:05:42.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=471040182}) 2024-03-20T09:05:42.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=815604070}) 2024-03-20T09:05:42.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2007105632}) 2024-03-20T09:05:42.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3740096149}) 2024-03-20T09:05:42.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3795267886}) 2024-03-20T09:05:42.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2203058994}) 2024-03-20T09:05:42.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2193314884}) 2024-03-20T09:05:42.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=639917931}) 2024-03-20T09:05:42.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2351137051}) 2024-03-20T09:05:42.977 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066117093}) 2024-03-20T09:05:42.977 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4106568028}) 2024-03-20T09:05:42.977 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2684561070}) 2024-03-20T09:05:42.977 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=777158838}) 2024-03-20T09:05:42.977 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3424711402}) 2024-03-20T09:05:42.977 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=71875596}) 2024-03-20T09:05:42.977 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3564559035}) 2024-03-20T09:05:42.977 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1551809073}) 2024-03-20T09:05:42.977 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4237774710}) 2024-03-20T09:05:42.977 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4162281157}) 2024-03-20T09:05:42.977 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3283980421}) 2024-03-20T09:05:42.977 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1561583635}) 2024-03-20T09:05:42.977 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3407208921}) 2024-03-20T09:05:42.978 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3506827474}) 2024-03-20T09:05:42.978 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1635750056}) 2024-03-20T09:05:42.978 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2823011674}) 2024-03-20T09:05:42.978 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2746114490}) 2024-03-20T09:05:42.978 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=330603124}) 2024-03-20T09:05:42.978 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1885476539}) 2024-03-20T09:05:42.978 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1577647937}) 2024-03-20T09:05:42.978 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3186591992}) 2024-03-20T09:05:42.978 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=74,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=2} crc {logm=2847818656,mdsmap=3984425268,mgr=427508465,mgr_command_descs=3831600018,mgr_metadata=2702834764,mgrstat=2918875162}) 2024-03-20T09:05:42.978 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=14,mon_config_key=86} crc {mgrstat=991574895,mon_config_key=568942078}) 2024-03-20T09:05:42.978 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1350994881}) 2024-03-20T09:05:42.978 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=92,monmap=5,osd_metadata=3} crc {mon_config_key=360779685,monmap=3481182261,osd_metadata=1402321155}) 2024-03-20T09:05:42.979 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=9,osd_pg_creating=1,osdmap=90} crc {osd_metadata=2746910592,osd_pg_creating=2148541147,osdmap=3580702051}) 2024-03-20T09:05:42.979 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2303323582}) 2024-03-20T09:05:42.979 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=10} crc {osdmap=2726515318}) 2024-03-20T09:05:43.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:05:43.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:05:43.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 26992/264078 objects degraded (10.221%), 12 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-03-20T09:05:43.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=232672708,config=3599132250,health=947575964,kv=4220410864}) 2024-03-20T09:05:43.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=3325467583,logm=470351074}) 2024-03-20T09:05:43.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=471040182}) 2024-03-20T09:05:43.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=815604070}) 2024-03-20T09:05:43.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2007105632}) 2024-03-20T09:05:43.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3740096149}) 2024-03-20T09:05:43.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3795267886}) 2024-03-20T09:05:43.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2203058994}) 2024-03-20T09:05:43.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2193314884}) 2024-03-20T09:05:43.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=639917931}) 2024-03-20T09:05:43.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2351137051}) 2024-03-20T09:05:43.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1066117093}) 2024-03-20T09:05:43.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4106568028}) 2024-03-20T09:05:43.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2684561070}) 2024-03-20T09:05:43.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=777158838}) 2024-03-20T09:05:43.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3424711402}) 2024-03-20T09:05:43.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=71875596}) 2024-03-20T09:05:43.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3564559035}) 2024-03-20T09:05:43.199 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1551809073}) 2024-03-20T09:05:43.199 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4237774710}) 2024-03-20T09:05:43.199 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4162281157}) 2024-03-20T09:05:43.199 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3283980421}) 2024-03-20T09:05:43.199 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1561583635}) 2024-03-20T09:05:43.199 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3407208921}) 2024-03-20T09:05:43.199 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3506827474}) 2024-03-20T09:05:43.199 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1635750056}) 2024-03-20T09:05:43.199 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2823011674}) 2024-03-20T09:05:43.199 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2746114490}) 2024-03-20T09:05:43.199 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=330603124}) 2024-03-20T09:05:43.199 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1885476539}) 2024-03-20T09:05:43.199 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1577647937}) 2024-03-20T09:05:43.199 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3186591992}) 2024-03-20T09:05:43.200 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=74,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=2} crc {logm=2847818656,mdsmap=3984425268,mgr=427508465,mgr_command_descs=3831600018,mgr_metadata=2702834764,mgrstat=2918875162}) 2024-03-20T09:05:43.200 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=14,mon_config_key=86} crc {mgrstat=991574895,mon_config_key=568942078}) 2024-03-20T09:05:43.200 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1350994881}) 2024-03-20T09:05:43.200 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=92,monmap=5,osd_metadata=3} crc {mon_config_key=360779685,monmap=3481182261,osd_metadata=1402321155}) 2024-03-20T09:05:43.200 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=9,osd_pg_creating=1,osdmap=90} crc {osd_metadata=2746910592,osd_pg_creating=2148541147,osdmap=3580702051}) 2024-03-20T09:05:43.200 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2303323582}) 2024-03-20T09:05:43.200 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=10} crc {osdmap=2726515318}) 2024-03-20T09:05:43.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:43 smithi026 ceph-mon[68309]: osdmap e302: 12 total, 11 up, 10 in 2024-03-20T09:05:43.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:43 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:05:43.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:43 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:05:43.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:43 smithi026 ceph-mon[68309]: pgmap v776: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 27 active+clean, 6 active+undersized; 5.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 92 op/s; 26910/263430 objects degraded (10.215%); 23728/263430 objects misplaced (9.007%); 6.7 MiB/s, 139 objects/s recovering 2024-03-20T09:05:44.050 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:43 smithi115 ceph-mon[72476]: osdmap e302: 12 total, 11 up, 10 in 2024-03-20T09:05:44.050 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:43 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:05:44.050 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:43 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:05:44.050 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:43 smithi115 ceph-mon[72476]: pgmap v776: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 27 active+clean, 6 active+undersized; 5.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 92 op/s; 26910/263430 objects degraded (10.215%); 23728/263430 objects misplaced (9.007%); 6.7 MiB/s, 139 objects/s recovering 2024-03-20T09:05:44.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:43 smithi122 ceph-mon[73495]: osdmap e302: 12 total, 11 up, 10 in 2024-03-20T09:05:44.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:43 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:05:44.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:43 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:05:44.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:43 smithi122 ceph-mon[73495]: pgmap v776: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 27 active+clean, 6 active+undersized; 5.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 92 op/s; 26910/263430 objects degraded (10.215%); 23728/263430 objects misplaced (9.007%); 6.7 MiB/s, 139 objects/s recovering 2024-03-20T09:05:44.699 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:05:44.699 INFO:teuthology.orchestra.run.smithi115.stdout: "size": 20, 2024-03-20T09:05:44.699 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 600, 2024-03-20T09:05:44.699 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [ 2024-03-20T09:05:44.699 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.699 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:58488 3.13 3:cea12a8b:::benchmark_data_smithi122_119671_object58487:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-20T09:05:44.699 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:03:31.889749+0000", 2024-03-20T09:05:44.699 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 132.80561242499999, 2024-03-20T09:05:44.700 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2189091009999999, 2024-03-20T09:05:44.700 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:44.700 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:05:44.700 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:44.700 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:44.700 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:44.700 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 58488 2024-03-20T09:05:44.700 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.700 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:44.700 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.700 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:44.700 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:31.889749+0000", 2024-03-20T09:05:44.700 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.700 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.701 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.701 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:44.701 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:31.889749+0000", 2024-03-20T09:05:44.701 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.701 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.701 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.701 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:44.701 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:31.889750+0000", 2024-03-20T09:05:44.702 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2330000000000001e-06 2024-03-20T09:05:44.702 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.702 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.702 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:44.702 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:31.889782+0000", 2024-03-20T09:05:44.702 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.1603000000000002e-05 2024-03-20T09:05:44.702 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.702 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.702 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:44.702 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:31.889795+0000", 2024-03-20T09:05:44.702 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2795e-05 2024-03-20T09:05:44.702 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.703 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.703 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:44.703 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:31.889802+0000", 2024-03-20T09:05:44.703 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.968e-06 2024-03-20T09:05:44.703 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.703 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.703 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.703 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:31.889852+0000", 2024-03-20T09:05:44.703 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0238999999999998e-05 2024-03-20T09:05:44.703 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.703 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.703 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:05:44.704 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:31.889862+0000", 2024-03-20T09:05:44.704 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0242e-05 2024-03-20T09:05:44.704 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.704 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.704 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.704 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:32.881820+0000", 2024-03-20T09:05:44.704 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.99195747300000003 2024-03-20T09:05:44.704 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.704 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.704 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:05:44.704 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:32.881832+0000", 2024-03-20T09:05:44.704 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2510999999999999e-05 2024-03-20T09:05:44.704 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.704 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.705 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.705 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:34.078202+0000", 2024-03-20T09:05:44.705 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1963698899999999 2024-03-20T09:05:44.705 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.705 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.705 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:05:44.705 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:34.078357+0000", 2024-03-20T09:05:44.705 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00015463 2024-03-20T09:05:44.705 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.705 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.705 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,7", 2024-03-20T09:05:44.705 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:34.078550+0000", 2024-03-20T09:05:44.705 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00019360199999999999 2024-03-20T09:05:44.705 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.705 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.706 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:44.706 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:34.081024+0000", 2024-03-20T09:05:44.706 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0024738989999999999 2024-03-20T09:05:44.706 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.706 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.706 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:44.706 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:34.082689+0000", 2024-03-20T09:05:44.706 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0016651509999999999 2024-03-20T09:05:44.706 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.706 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.706 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:05:44.706 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:34.107846+0000", 2024-03-20T09:05:44.706 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.025156614000000001 2024-03-20T09:05:44.707 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.707 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.707 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:05:44.707 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:34.108008+0000", 2024-03-20T09:05:44.707 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000162118 2024-03-20T09:05:44.707 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.707 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.707 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:44.707 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:34.108658+0000", 2024-03-20T09:05:44.707 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000650133 2024-03-20T09:05:44.707 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.707 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:44.707 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.708 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.708 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.708 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:58502 3.14 3.37943b74 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-20T09:05:44.708 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:03:31.896215+0000", 2024-03-20T09:05:44.708 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 132.799146079, 2024-03-20T09:05:44.708 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.98446493599999996, 2024-03-20T09:05:44.708 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:44.708 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:05:44.708 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:44.708 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:44.708 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:44.708 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 58502 2024-03-20T09:05:44.708 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.708 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:44.709 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.709 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:44.709 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:31.896215+0000", 2024-03-20T09:05:44.709 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.709 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.709 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.709 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:44.709 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:31.896215+0000", 2024-03-20T09:05:44.709 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.709 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.709 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.709 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:44.709 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:31.896217+0000", 2024-03-20T09:05:44.709 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.071e-06 2024-03-20T09:05:44.710 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.710 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.710 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:44.710 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:31.896271+0000", 2024-03-20T09:05:44.710 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4758e-05 2024-03-20T09:05:44.710 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.710 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.710 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:44.710 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:31.896285+0000", 2024-03-20T09:05:44.710 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3304e-05 2024-03-20T09:05:44.710 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.710 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.710 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:44.711 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:31.896291+0000", 2024-03-20T09:05:44.711 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0290000000000002e-06 2024-03-20T09:05:44.711 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.711 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.711 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.711 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:31.896347+0000", 2024-03-20T09:05:44.711 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6406000000000002e-05 2024-03-20T09:05:44.711 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.711 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.711 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:05:44.711 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:31.896357+0000", 2024-03-20T09:05:44.711 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.7280000000000007e-06 2024-03-20T09:05:44.711 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.711 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.711 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.712 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:32.880662+0000", 2024-03-20T09:05:44.712 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.98430528299999998 2024-03-20T09:05:44.712 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.712 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.712 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:44.712 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:32.880680+0000", 2024-03-20T09:05:44.712 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8357000000000001e-05 2024-03-20T09:05:44.712 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.712 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:44.712 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.712 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.712 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.712 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:58511 3.14 3.c2de34 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-20T09:05:44.713 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:03:31.903113+0000", 2024-03-20T09:05:44.713 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 132.79224888499999, 2024-03-20T09:05:44.713 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.97828768399999999, 2024-03-20T09:05:44.713 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:44.713 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:05:44.713 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:44.713 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:44.713 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:44.713 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 58511 2024-03-20T09:05:44.713 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.713 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:44.713 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.713 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:44.713 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:31.903113+0000", 2024-03-20T09:05:44.713 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.714 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.714 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.714 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:44.714 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:31.903113+0000", 2024-03-20T09:05:44.714 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.714 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.714 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.714 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:44.714 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:31.903113+0000", 2024-03-20T09:05:44.714 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.2799999999999995e-07 2024-03-20T09:05:44.714 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.714 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.715 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:44.715 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:31.903168+0000", 2024-03-20T09:05:44.715 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4392999999999999e-05 2024-03-20T09:05:44.715 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.715 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.715 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:44.715 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:31.903185+0000", 2024-03-20T09:05:44.715 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7215999999999999e-05 2024-03-20T09:05:44.715 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.715 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.715 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:44.715 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:31.903191+0000", 2024-03-20T09:05:44.715 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9699999999999996e-06 2024-03-20T09:05:44.715 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.716 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.716 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.716 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:31.903235+0000", 2024-03-20T09:05:44.716 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4056e-05 2024-03-20T09:05:44.716 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.716 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.716 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:05:44.716 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:31.903241+0000", 2024-03-20T09:05:44.716 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5269999999999998e-06 2024-03-20T09:05:44.716 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.716 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.716 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.716 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:32.881381+0000", 2024-03-20T09:05:44.716 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.97814043799999995 2024-03-20T09:05:44.717 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.717 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.717 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:44.717 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:32.881400+0000", 2024-03-20T09:05:44.717 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9256e-05 2024-03-20T09:05:44.717 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.717 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:44.717 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.717 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.717 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.717 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:58467 3.12 3:4b74932c:::benchmark_data_smithi122_119671_object58466:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e234)", 2024-03-20T09:05:44.717 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:03:32.876666+0000", 2024-03-20T09:05:44.717 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 131.81869585999999, 2024-03-20T09:05:44.718 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.251213599, 2024-03-20T09:05:44.718 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:44.718 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:05:44.718 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:44.718 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:44.718 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:44.718 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 58467 2024-03-20T09:05:44.718 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.718 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:44.718 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.718 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:44.718 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:32.876666+0000", 2024-03-20T09:05:44.718 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.718 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.719 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.719 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:44.719 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:32.876666+0000", 2024-03-20T09:05:44.719 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.719 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.719 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.719 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:44.719 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:32.876668+0000", 2024-03-20T09:05:44.719 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.888e-06 2024-03-20T09:05:44.719 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.719 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.719 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:44.719 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:32.876853+0000", 2024-03-20T09:05:44.719 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000185651 2024-03-20T09:05:44.720 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.720 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.720 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:44.720 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:32.876893+0000", 2024-03-20T09:05:44.720 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.9365e-05 2024-03-20T09:05:44.720 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.720 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.720 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:44.720 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:32.876906+0000", 2024-03-20T09:05:44.720 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3266e-05 2024-03-20T09:05:44.720 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.720 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.720 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.720 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:32.877080+0000", 2024-03-20T09:05:44.721 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00017366699999999999 2024-03-20T09:05:44.721 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.721 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.721 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op must wait for map", 2024-03-20T09:05:44.721 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:32.877088+0000", 2024-03-20T09:05:44.721 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5159999999999994e-06 2024-03-20T09:05:44.721 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.721 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.721 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.721 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:32.881376+0000", 2024-03-20T09:05:44.721 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0042881610000000004 2024-03-20T09:05:44.721 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.721 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.721 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:05:44.722 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:32.881389+0000", 2024-03-20T09:05:44.722 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2911000000000001e-05 2024-03-20T09:05:44.722 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.722 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.722 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.722 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:34.064679+0000", 2024-03-20T09:05:44.722 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1832895809999999 2024-03-20T09:05:44.722 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.722 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.722 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for degraded object", 2024-03-20T09:05:44.722 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:34.066742+0000", 2024-03-20T09:05:44.722 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0020637189999999999 2024-03-20T09:05:44.722 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.723 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.723 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.723 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:34.127741+0000", 2024-03-20T09:05:44.723 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.060998122000000002 2024-03-20T09:05:44.723 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.723 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.723 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:44.723 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:34.127879+0000", 2024-03-20T09:05:44.723 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013875199999999999 2024-03-20T09:05:44.723 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.723 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:44.723 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.723 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.723 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.723 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:58497 3.12 3:49b9de71:::benchmark_data_smithi122_119671_object58496:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e234)", 2024-03-20T09:05:44.724 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:03:32.876947+0000", 2024-03-20T09:05:44.724 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 131.81841463000001, 2024-03-20T09:05:44.724 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.251715229, 2024-03-20T09:05:44.724 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:44.724 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:05:44.724 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:44.724 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:44.724 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:44.724 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 58497 2024-03-20T09:05:44.724 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.724 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:44.724 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.724 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:44.724 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:32.876947+0000", 2024-03-20T09:05:44.725 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.725 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.725 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.725 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:44.725 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:32.876947+0000", 2024-03-20T09:05:44.725 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.725 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.725 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.725 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:44.725 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:32.876948+0000", 2024-03-20T09:05:44.725 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3300000000000001e-07 2024-03-20T09:05:44.725 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.725 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.725 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:44.726 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:32.876991+0000", 2024-03-20T09:05:44.726 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.3720000000000002e-05 2024-03-20T09:05:44.726 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.726 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.726 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:44.726 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:32.877002+0000", 2024-03-20T09:05:44.726 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0509999999999999e-05 2024-03-20T09:05:44.726 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.726 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.726 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:44.726 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:32.877010+0000", 2024-03-20T09:05:44.726 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.0949999999999996e-06 2024-03-20T09:05:44.726 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.726 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.727 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.727 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:32.877154+0000", 2024-03-20T09:05:44.727 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00014383700000000001 2024-03-20T09:05:44.727 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.727 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.727 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting_for_map not empty", 2024-03-20T09:05:44.727 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:32.877159+0000", 2024-03-20T09:05:44.727 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0989999999999998e-06 2024-03-20T09:05:44.727 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.727 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.727 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.727 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:32.881780+0000", 2024-03-20T09:05:44.727 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0046207519999999997 2024-03-20T09:05:44.727 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.728 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.728 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:05:44.728 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:32.881789+0000", 2024-03-20T09:05:44.728 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.7399999999999999e-06 2024-03-20T09:05:44.728 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.728 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.728 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.728 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:34.084582+0000", 2024-03-20T09:05:44.728 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2027926769999999 2024-03-20T09:05:44.728 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.728 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.728 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:05:44.728 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:34.084736+0000", 2024-03-20T09:05:44.728 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00015432600000000001 2024-03-20T09:05:44.729 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.729 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.729 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,2,7", 2024-03-20T09:05:44.729 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:34.084954+0000", 2024-03-20T09:05:44.729 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00021793 2024-03-20T09:05:44.729 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.729 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.729 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:44.729 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:34.126086+0000", 2024-03-20T09:05:44.729 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.041131320999999998 2024-03-20T09:05:44.729 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.729 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.729 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:05:44.729 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:34.126103+0000", 2024-03-20T09:05:44.730 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6875e-05 2024-03-20T09:05:44.730 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.730 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.730 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:44.730 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:34.126150+0000", 2024-03-20T09:05:44.730 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.7725999999999997e-05 2024-03-20T09:05:44.730 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.730 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.730 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:44.730 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:34.127922+0000", 2024-03-20T09:05:44.730 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00177133 2024-03-20T09:05:44.730 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.730 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.730 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:05:44.731 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:34.127977+0000", 2024-03-20T09:05:44.731 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5046000000000001e-05 2024-03-20T09:05:44.731 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.731 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.731 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:44.731 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:34.128662+0000", 2024-03-20T09:05:44.731 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00068551199999999999 2024-03-20T09:05:44.731 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.731 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:44.731 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.731 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.731 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.731 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:70252 3.14 3:292c9f46:::benchmark_data_smithi122_119671_object70251:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-03-20T09:05:44.731 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:03:51.934006+0000", 2024-03-20T09:05:44.732 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 112.76135546, 2024-03-20T09:05:44.732 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.98078653000000005, 2024-03-20T09:05:44.732 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:44.732 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:05:44.732 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:44.732 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:44.732 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:44.732 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 70252 2024-03-20T09:05:44.732 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.732 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:44.732 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.732 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:44.732 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:51.934006+0000", 2024-03-20T09:05:44.732 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.733 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.733 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.733 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:44.733 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:51.934006+0000", 2024-03-20T09:05:44.733 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.733 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.733 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.733 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:44.733 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:51.934007+0000", 2024-03-20T09:05:44.733 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4219999999999999e-06 2024-03-20T09:05:44.733 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.733 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.733 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:44.733 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:51.934069+0000", 2024-03-20T09:05:44.734 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1357000000000002e-05 2024-03-20T09:05:44.734 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.734 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.734 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:44.734 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:51.934080+0000", 2024-03-20T09:05:44.734 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0993e-05 2024-03-20T09:05:44.734 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.734 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.734 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:44.734 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:51.934088+0000", 2024-03-20T09:05:44.734 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.259e-06 2024-03-20T09:05:44.734 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.734 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.734 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.735 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:51.934168+0000", 2024-03-20T09:05:44.735 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.9424000000000004e-05 2024-03-20T09:05:44.735 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.735 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.735 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:05:44.735 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:51.934195+0000", 2024-03-20T09:05:44.735 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.7881e-05 2024-03-20T09:05:44.735 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.735 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.735 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.735 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:52.900081+0000", 2024-03-20T09:05:44.735 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.96588523100000001 2024-03-20T09:05:44.735 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.735 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.736 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:05:44.736 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:52.900248+0000", 2024-03-20T09:05:44.736 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00016738500000000001 2024-03-20T09:05:44.736 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.736 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.736 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 8,11", 2024-03-20T09:05:44.736 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:52.900409+0000", 2024-03-20T09:05:44.736 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000161309 2024-03-20T09:05:44.736 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.736 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.736 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:44.736 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:52.907026+0000", 2024-03-20T09:05:44.736 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0066164400000000003 2024-03-20T09:05:44.736 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.737 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.737 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:44.737 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:52.907068+0000", 2024-03-20T09:05:44.737 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.2095000000000003e-05 2024-03-20T09:05:44.737 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.737 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.737 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:05:44.737 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:52.914723+0000", 2024-03-20T09:05:44.737 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0076546740000000002 2024-03-20T09:05:44.737 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.737 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.737 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:05:44.737 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:52.914755+0000", 2024-03-20T09:05:44.737 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.2199999999999997e-05 2024-03-20T09:05:44.738 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.738 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.738 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:44.738 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:52.914793+0000", 2024-03-20T09:05:44.738 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.786e-05 2024-03-20T09:05:44.738 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.738 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:44.738 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.738 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.738 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.738 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:70292 3.14 3:2b3e8093:::benchmark_data_smithi122_119671_object70291:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-03-20T09:05:44.738 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:03:51.966407+0000", 2024-03-20T09:05:44.738 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 112.728954835, 2024-03-20T09:05:44.738 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.94847097400000002, 2024-03-20T09:05:44.739 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:44.739 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:05:44.739 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:44.739 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:44.739 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:44.739 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 70292 2024-03-20T09:05:44.739 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.739 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:44.739 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.739 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:44.739 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:51.966407+0000", 2024-03-20T09:05:44.739 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.739 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.740 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.740 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:44.740 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:51.966407+0000", 2024-03-20T09:05:44.740 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.740 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.740 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.740 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:44.740 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:51.966408+0000", 2024-03-20T09:05:44.740 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.2399999999999996e-07 2024-03-20T09:05:44.740 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.741 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.741 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:44.741 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:51.966429+0000", 2024-03-20T09:05:44.741 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1729000000000002e-05 2024-03-20T09:05:44.741 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.741 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.741 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:44.741 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:51.966438+0000", 2024-03-20T09:05:44.741 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.2400000000000007e-06 2024-03-20T09:05:44.742 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.742 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.742 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:44.742 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:51.966445+0000", 2024-03-20T09:05:44.742 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.4279999999999999e-06 2024-03-20T09:05:44.742 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.743 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.743 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.743 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:51.966576+0000", 2024-03-20T09:05:44.743 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013136499999999999 2024-03-20T09:05:44.743 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.743 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.743 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:05:44.743 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:51.966588+0000", 2024-03-20T09:05:44.743 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1399e-05 2024-03-20T09:05:44.743 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.743 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.743 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.743 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:52.901610+0000", 2024-03-20T09:05:44.743 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.93502221699999999 2024-03-20T09:05:44.744 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.744 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.744 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:05:44.744 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:52.901722+0000", 2024-03-20T09:05:44.744 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000112388 2024-03-20T09:05:44.744 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.744 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.744 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 8,11", 2024-03-20T09:05:44.744 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:52.901857+0000", 2024-03-20T09:05:44.744 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000134148 2024-03-20T09:05:44.744 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.744 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.744 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:44.744 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:52.907107+0000", 2024-03-20T09:05:44.745 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0052504190000000001 2024-03-20T09:05:44.745 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.745 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.745 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:44.745 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:52.914613+0000", 2024-03-20T09:05:44.745 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0075062879999999998 2024-03-20T09:05:44.746 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.746 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.746 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:05:44.746 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:52.914803+0000", 2024-03-20T09:05:44.746 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00019023999999999999 2024-03-20T09:05:44.746 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.746 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.746 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:05:44.746 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:52.914843+0000", 2024-03-20T09:05:44.746 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.9888999999999997e-05 2024-03-20T09:05:44.746 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.746 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.746 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:44.747 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:52.914878+0000", 2024-03-20T09:05:44.747 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.43e-05 2024-03-20T09:05:44.747 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.747 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:44.747 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.747 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.747 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.747 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:70326 3.14 3:2be50bbb:::benchmark_data_smithi122_119671_object70325:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-03-20T09:05:44.747 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:03:52.005217+0000", 2024-03-20T09:05:44.747 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 112.690144776, 2024-03-20T09:05:44.747 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.90981233100000003, 2024-03-20T09:05:44.747 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:44.747 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:05:44.747 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:44.748 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:44.748 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:44.748 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 70326 2024-03-20T09:05:44.748 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.748 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:44.748 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.748 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:44.748 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:52.005217+0000", 2024-03-20T09:05:44.748 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.748 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.748 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.748 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:44.748 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:52.005217+0000", 2024-03-20T09:05:44.748 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.748 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.749 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.749 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:44.749 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:52.005219+0000", 2024-03-20T09:05:44.749 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.838e-06 2024-03-20T09:05:44.749 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.749 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.749 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:44.749 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:52.005254+0000", 2024-03-20T09:05:44.749 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.5722000000000001e-05 2024-03-20T09:05:44.749 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.749 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.749 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:44.749 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:52.005268+0000", 2024-03-20T09:05:44.749 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3678e-05 2024-03-20T09:05:44.750 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.750 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.750 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:44.750 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:52.005276+0000", 2024-03-20T09:05:44.750 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.8399999999999995e-06 2024-03-20T09:05:44.750 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.750 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.750 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.750 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:52.005331+0000", 2024-03-20T09:05:44.750 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5333e-05 2024-03-20T09:05:44.750 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.750 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.750 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:05:44.750 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:52.005340+0000", 2024-03-20T09:05:44.751 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.3799999999999994e-06 2024-03-20T09:05:44.751 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.751 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.751 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.751 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:52.902921+0000", 2024-03-20T09:05:44.751 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.89758173600000002 2024-03-20T09:05:44.751 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.751 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.751 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:05:44.751 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:52.903052+0000", 2024-03-20T09:05:44.751 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013116899999999999 2024-03-20T09:05:44.751 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.751 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.751 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 8,11", 2024-03-20T09:05:44.752 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:52.903195+0000", 2024-03-20T09:05:44.752 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00014263599999999999 2024-03-20T09:05:44.752 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.752 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.752 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:44.752 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:52.907145+0000", 2024-03-20T09:05:44.752 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0039502679999999998 2024-03-20T09:05:44.752 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.752 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.752 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:05:44.752 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:52.914888+0000", 2024-03-20T09:05:44.752 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0077423800000000001 2024-03-20T09:05:44.752 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.752 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.753 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:44.753 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:52.914933+0000", 2024-03-20T09:05:44.753 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4875e-05 2024-03-20T09:05:44.753 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.753 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.753 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:05:44.753 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:52.914989+0000", 2024-03-20T09:05:44.753 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6184999999999998e-05 2024-03-20T09:05:44.753 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.753 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.753 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:44.753 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:52.915029+0000", 2024-03-20T09:05:44.753 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.0290999999999998e-05 2024-03-20T09:05:44.753 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.753 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:44.754 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.754 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.754 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.754 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:73620 3.13 3:cd619b87:::benchmark_data_smithi122_119671_object73619:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-03-20T09:05:44.754 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:03:55.976096+0000", 2024-03-20T09:05:44.754 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 108.719266003, 2024-03-20T09:05:44.754 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.92946227000000003, 2024-03-20T09:05:44.754 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:44.754 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:05:44.754 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:44.754 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:44.754 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:44.754 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 73620 2024-03-20T09:05:44.754 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.755 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:44.755 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.755 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:44.755 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:55.976096+0000", 2024-03-20T09:05:44.755 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.755 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.755 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.755 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:44.755 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:55.976096+0000", 2024-03-20T09:05:44.755 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.755 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.755 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.755 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:44.755 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:55.976097+0000", 2024-03-20T09:05:44.756 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3209999999999999e-06 2024-03-20T09:05:44.756 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.756 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.756 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:44.756 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:55.976168+0000", 2024-03-20T09:05:44.756 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.1017000000000001e-05 2024-03-20T09:05:44.756 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.756 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.756 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:44.756 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:55.976189+0000", 2024-03-20T09:05:44.756 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0905000000000001e-05 2024-03-20T09:05:44.756 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.756 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.756 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:44.757 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:55.976196+0000", 2024-03-20T09:05:44.757 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.4699999999999996e-06 2024-03-20T09:05:44.757 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.757 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.757 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.757 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:55.978573+0000", 2024-03-20T09:05:44.757 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.002376796 2024-03-20T09:05:44.757 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.757 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.757 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op must wait for map", 2024-03-20T09:05:44.757 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:55.978580+0000", 2024-03-20T09:05:44.757 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.1729999999999998e-06 2024-03-20T09:05:44.757 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.757 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.758 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.758 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:55.980464+0000", 2024-03-20T09:05:44.758 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0018836409999999999 2024-03-20T09:05:44.758 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.758 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.758 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:05:44.758 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:55.980473+0000", 2024-03-20T09:05:44.758 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.2450000000000004e-06 2024-03-20T09:05:44.758 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.758 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.758 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.758 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.893763+0000", 2024-03-20T09:05:44.758 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.91329008700000003 2024-03-20T09:05:44.759 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.759 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.759 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:05:44.759 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.893888+0000", 2024-03-20T09:05:44.759 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000125158 2024-03-20T09:05:44.759 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.759 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.759 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,7", 2024-03-20T09:05:44.759 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.894041+0000", 2024-03-20T09:05:44.759 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000153072 2024-03-20T09:05:44.759 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.759 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.759 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:05:44.759 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.897345+0000", 2024-03-20T09:05:44.759 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0033032090000000001 2024-03-20T09:05:44.760 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.760 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.760 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:44.760 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.897798+0000", 2024-03-20T09:05:44.760 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00045289199999999999 2024-03-20T09:05:44.760 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.760 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.760 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:44.760 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.905454+0000", 2024-03-20T09:05:44.760 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0076566070000000002 2024-03-20T09:05:44.760 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.760 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.760 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:05:44.760 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.905514+0000", 2024-03-20T09:05:44.761 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0034999999999997e-05 2024-03-20T09:05:44.761 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.761 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.761 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:44.761 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.905558+0000", 2024-03-20T09:05:44.761 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.3642e-05 2024-03-20T09:05:44.761 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.761 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:44.761 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.761 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.761 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.761 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:73630 3.13 3:cb6b078d:::benchmark_data_smithi122_119671_object73629:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-03-20T09:05:44.761 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:03:55.988187+0000", 2024-03-20T09:05:44.761 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 108.70717462499999, 2024-03-20T09:05:44.762 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.92487139900000004, 2024-03-20T09:05:44.762 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:44.762 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:05:44.762 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:44.762 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:44.762 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:44.762 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 73630 2024-03-20T09:05:44.762 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.762 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:44.762 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.762 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:44.762 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:55.988187+0000", 2024-03-20T09:05:44.762 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.762 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.763 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.763 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:44.763 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:55.988187+0000", 2024-03-20T09:05:44.763 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.763 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.763 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.763 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:44.763 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:55.988189+0000", 2024-03-20T09:05:44.763 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7290000000000001e-06 2024-03-20T09:05:44.763 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.763 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.763 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:44.763 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:55.988223+0000", 2024-03-20T09:05:44.763 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.3986000000000003e-05 2024-03-20T09:05:44.764 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.764 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.764 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:44.764 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:55.988236+0000", 2024-03-20T09:05:44.764 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3618000000000001e-05 2024-03-20T09:05:44.764 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.764 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.764 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:44.764 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:55.988245+0000", 2024-03-20T09:05:44.764 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.1649999999999995e-06 2024-03-20T09:05:44.764 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.764 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.764 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.764 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:55.988853+0000", 2024-03-20T09:05:44.765 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00060710500000000004 2024-03-20T09:05:44.765 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.765 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.765 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:05:44.765 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:55.988863+0000", 2024-03-20T09:05:44.765 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0020999999999999e-05 2024-03-20T09:05:44.765 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.765 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.765 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.765 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.895577+0000", 2024-03-20T09:05:44.765 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.90671466000000001 2024-03-20T09:05:44.765 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.766 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.766 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:05:44.766 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.895687+0000", 2024-03-20T09:05:44.766 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00011007599999999999 2024-03-20T09:05:44.766 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.766 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.766 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,7", 2024-03-20T09:05:44.766 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.895793+0000", 2024-03-20T09:05:44.766 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00010533499999999999 2024-03-20T09:05:44.766 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.766 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.766 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:44.767 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.899874+0000", 2024-03-20T09:05:44.767 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0040815859999999999 2024-03-20T09:05:44.767 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.767 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.767 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:05:44.767 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.903311+0000", 2024-03-20T09:05:44.767 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0034369639999999998 2024-03-20T09:05:44.767 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.767 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.767 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:44.767 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.912958+0000", 2024-03-20T09:05:44.767 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0096467219999999999 2024-03-20T09:05:44.767 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.767 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.768 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:05:44.768 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.913015+0000", 2024-03-20T09:05:44.768 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7438000000000001e-05 2024-03-20T09:05:44.768 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.768 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.768 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:44.768 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.913058+0000", 2024-03-20T09:05:44.768 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.2994000000000002e-05 2024-03-20T09:05:44.768 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.768 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:44.768 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.768 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.768 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.768 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:73637 3.13 3:c8df8b7f:::benchmark_data_smithi122_119671_object73636:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-03-20T09:05:44.769 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:03:56.001333+0000", 2024-03-20T09:05:44.769 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 108.694028612, 2024-03-20T09:05:44.769 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.91202702400000002, 2024-03-20T09:05:44.769 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:44.769 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:05:44.769 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:44.769 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:44.769 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:44.769 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 73637 2024-03-20T09:05:44.769 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.769 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:44.769 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.769 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:44.769 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.001333+0000", 2024-03-20T09:05:44.770 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.770 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.770 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.770 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:44.770 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.001333+0000", 2024-03-20T09:05:44.770 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.770 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.770 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.770 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:44.770 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.001334+0000", 2024-03-20T09:05:44.770 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5340000000000001e-06 2024-03-20T09:05:44.770 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.770 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.770 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:44.771 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.001373+0000", 2024-03-20T09:05:44.771 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.8127999999999998e-05 2024-03-20T09:05:44.771 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.771 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.771 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:44.771 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.001384+0000", 2024-03-20T09:05:44.771 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1606e-05 2024-03-20T09:05:44.771 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.771 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.771 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:44.771 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.001391+0000", 2024-03-20T09:05:44.771 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.878e-06 2024-03-20T09:05:44.771 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.771 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.772 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.772 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.001452+0000", 2024-03-20T09:05:44.772 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0887999999999997e-05 2024-03-20T09:05:44.772 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.772 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.772 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:05:44.772 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.001460+0000", 2024-03-20T09:05:44.772 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.4789999999999996e-06 2024-03-20T09:05:44.772 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.772 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.772 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.772 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.896600+0000", 2024-03-20T09:05:44.773 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.89513979600000004 2024-03-20T09:05:44.773 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.773 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.773 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:05:44.773 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.896664+0000", 2024-03-20T09:05:44.773 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.3456999999999999e-05 2024-03-20T09:05:44.773 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.773 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.773 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,7", 2024-03-20T09:05:44.773 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.896753+0000", 2024-03-20T09:05:44.773 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.9393000000000003e-05 2024-03-20T09:05:44.773 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.773 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.773 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:44.774 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.899905+0000", 2024-03-20T09:05:44.774 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0031520570000000002 2024-03-20T09:05:44.774 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.774 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.774 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:05:44.774 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.903367+0000", 2024-03-20T09:05:44.774 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0034620720000000001 2024-03-20T09:05:44.774 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.774 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.774 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:44.774 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.913268+0000", 2024-03-20T09:05:44.774 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0099004130000000003 2024-03-20T09:05:44.774 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.774 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.775 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:05:44.775 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.913320+0000", 2024-03-20T09:05:44.775 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2768e-05 2024-03-20T09:05:44.775 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.775 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.775 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:44.775 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.913360+0000", 2024-03-20T09:05:44.775 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.9555000000000003e-05 2024-03-20T09:05:44.775 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.775 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:44.775 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.775 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.775 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.775 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:73639 3.13 3:c8d5a457:::benchmark_data_smithi122_119671_object73638:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-03-20T09:05:44.776 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:03:56.011068+0000", 2024-03-20T09:05:44.776 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 108.684293115, 2024-03-20T09:05:44.776 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.90240711399999995, 2024-03-20T09:05:44.776 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:44.776 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:05:44.776 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:44.776 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:44.776 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:44.776 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 73639 2024-03-20T09:05:44.776 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.776 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:44.776 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.776 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:44.776 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.011068+0000", 2024-03-20T09:05:44.776 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.777 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.777 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.777 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:44.777 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.011068+0000", 2024-03-20T09:05:44.777 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.777 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.777 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.777 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:44.777 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.011070+0000", 2024-03-20T09:05:44.777 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.364e-06 2024-03-20T09:05:44.777 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.777 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.777 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:44.777 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.011143+0000", 2024-03-20T09:05:44.778 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3151999999999994e-05 2024-03-20T09:05:44.778 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.778 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.778 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:44.778 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.011155+0000", 2024-03-20T09:05:44.778 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1729000000000001e-05 2024-03-20T09:05:44.778 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.778 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.778 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:44.778 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.011162+0000", 2024-03-20T09:05:44.778 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3459999999999996e-06 2024-03-20T09:05:44.778 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.778 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.779 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.779 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.011233+0000", 2024-03-20T09:05:44.779 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.0556999999999995e-05 2024-03-20T09:05:44.779 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.779 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.779 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:05:44.779 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.011240+0000", 2024-03-20T09:05:44.779 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.5850000000000002e-06 2024-03-20T09:05:44.779 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.779 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.779 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.779 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.897826+0000", 2024-03-20T09:05:44.779 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.88658585899999998 2024-03-20T09:05:44.779 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.779 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.780 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:05:44.780 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.897930+0000", 2024-03-20T09:05:44.780 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000103589 2024-03-20T09:05:44.780 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.780 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.780 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,7", 2024-03-20T09:05:44.780 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.898051+0000", 2024-03-20T09:05:44.780 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000120993 2024-03-20T09:05:44.780 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.780 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.780 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:44.780 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.903355+0000", 2024-03-20T09:05:44.780 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0053040489999999999 2024-03-20T09:05:44.780 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.781 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.781 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:05:44.781 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.903372+0000", 2024-03-20T09:05:44.781 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7295e-05 2024-03-20T09:05:44.781 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.781 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.781 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:44.781 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.913397+0000", 2024-03-20T09:05:44.781 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.010024580999999999 2024-03-20T09:05:44.781 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.781 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.781 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:05:44.781 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.913443+0000", 2024-03-20T09:05:44.781 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6671000000000001e-05 2024-03-20T09:05:44.782 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.782 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.782 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:44.782 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.913476+0000", 2024-03-20T09:05:44.782 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.2344000000000001e-05 2024-03-20T09:05:44.782 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.782 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:44.782 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.782 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.782 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.782 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:73646 3.13 3:cd853516:::benchmark_data_smithi122_119671_object73645:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-03-20T09:05:44.782 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:03:56.023590+0000", 2024-03-20T09:05:44.782 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 108.671771456, 2024-03-20T09:05:44.782 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.88999176499999999, 2024-03-20T09:05:44.783 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:44.783 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:05:44.783 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:44.783 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:44.783 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:44.783 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 73646 2024-03-20T09:05:44.783 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.783 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:44.783 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.783 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:44.784 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.023590+0000", 2024-03-20T09:05:44.784 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.784 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.784 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.784 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:44.784 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.023590+0000", 2024-03-20T09:05:44.784 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.784 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.784 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.784 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:44.784 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.023592+0000", 2024-03-20T09:05:44.784 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.502e-06 2024-03-20T09:05:44.784 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.784 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.785 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:44.785 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.023661+0000", 2024-03-20T09:05:44.785 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.9518999999999996e-05 2024-03-20T09:05:44.785 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.785 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.785 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:44.785 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.023676+0000", 2024-03-20T09:05:44.785 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4866e-05 2024-03-20T09:05:44.785 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.785 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.785 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:44.785 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.023684+0000", 2024-03-20T09:05:44.785 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.6120000000000001e-06 2024-03-20T09:05:44.785 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.786 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.786 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.786 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.023746+0000", 2024-03-20T09:05:44.786 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2704000000000001e-05 2024-03-20T09:05:44.786 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.786 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.786 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:05:44.786 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.023755+0000", 2024-03-20T09:05:44.786 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.9840000000000007e-06 2024-03-20T09:05:44.786 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.786 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.786 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.786 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.898930+0000", 2024-03-20T09:05:44.786 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.87517464700000003 2024-03-20T09:05:44.787 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.787 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.787 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:05:44.787 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.899040+0000", 2024-03-20T09:05:44.787 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00011052699999999999 2024-03-20T09:05:44.787 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.787 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.787 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,7", 2024-03-20T09:05:44.787 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.899186+0000", 2024-03-20T09:05:44.787 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000145496 2024-03-20T09:05:44.787 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.787 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.787 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:44.787 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.904006+0000", 2024-03-20T09:05:44.787 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0048198850000000003 2024-03-20T09:05:44.788 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.788 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.788 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:05:44.788 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.906804+0000", 2024-03-20T09:05:44.788 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0027984249999999998 2024-03-20T09:05:44.788 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.788 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.788 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:44.788 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.913508+0000", 2024-03-20T09:05:44.788 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0067034779999999997 2024-03-20T09:05:44.788 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.788 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.788 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:05:44.788 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.913551+0000", 2024-03-20T09:05:44.789 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.3333e-05 2024-03-20T09:05:44.789 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.789 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.789 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:44.789 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.913582+0000", 2024-03-20T09:05:44.789 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.0787000000000002e-05 2024-03-20T09:05:44.789 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.789 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:44.789 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.789 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.789 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.789 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:73664 3.13 3:cd2d8b28:::benchmark_data_smithi122_119671_object73663:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-03-20T09:05:44.789 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:03:56.069312+0000", 2024-03-20T09:05:44.789 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 108.626049181, 2024-03-20T09:05:44.790 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.84438812299999999, 2024-03-20T09:05:44.790 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:44.790 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:05:44.790 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:44.790 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:44.790 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:44.790 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 73664 2024-03-20T09:05:44.790 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.790 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:44.790 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.790 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:44.790 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.069312+0000", 2024-03-20T09:05:44.790 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.790 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.791 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.791 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:44.791 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.069312+0000", 2024-03-20T09:05:44.791 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.791 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.791 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.791 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:44.791 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.069314+0000", 2024-03-20T09:05:44.791 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.486e-06 2024-03-20T09:05:44.791 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.791 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.791 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:44.791 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.069362+0000", 2024-03-20T09:05:44.791 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.7902999999999999e-05 2024-03-20T09:05:44.792 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.792 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.792 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:44.792 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.069375+0000", 2024-03-20T09:05:44.792 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.376e-05 2024-03-20T09:05:44.792 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.792 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.792 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:44.792 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.069384+0000", 2024-03-20T09:05:44.792 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.7669999999999992e-06 2024-03-20T09:05:44.792 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.792 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.792 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.792 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.070659+0000", 2024-03-20T09:05:44.793 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0012747559999999999 2024-03-20T09:05:44.793 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.793 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.793 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:05:44.793 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.070668+0000", 2024-03-20T09:05:44.793 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.3810000000000005e-06 2024-03-20T09:05:44.793 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.793 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.793 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.793 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.899927+0000", 2024-03-20T09:05:44.793 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.82925845600000003 2024-03-20T09:05:44.793 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.793 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.793 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:05:44.793 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.900024+0000", 2024-03-20T09:05:44.794 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.6848999999999994e-05 2024-03-20T09:05:44.794 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.794 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.794 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,7", 2024-03-20T09:05:44.794 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.900133+0000", 2024-03-20T09:05:44.794 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000108924 2024-03-20T09:05:44.794 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.794 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.794 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:44.794 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.904035+0000", 2024-03-20T09:05:44.795 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0039028190000000001 2024-03-20T09:05:44.795 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.795 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.795 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:05:44.795 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.906815+0000", 2024-03-20T09:05:44.795 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.002779335 2024-03-20T09:05:44.795 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.795 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.795 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:44.795 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.913612+0000", 2024-03-20T09:05:44.795 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0067976269999999997 2024-03-20T09:05:44.795 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.795 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.795 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:05:44.796 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.913670+0000", 2024-03-20T09:05:44.796 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7129000000000001e-05 2024-03-20T09:05:44.796 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.796 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.796 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:44.796 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.913700+0000", 2024-03-20T09:05:44.796 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.0930999999999999e-05 2024-03-20T09:05:44.796 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.796 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:44.796 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.796 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.796 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.796 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:73667 3.13 3:c9bd335f:::benchmark_data_smithi122_119671_object73666:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-03-20T09:05:44.796 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:03:56.071968+0000", 2024-03-20T09:05:44.797 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 108.623393647, 2024-03-20T09:05:44.797 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.84184903099999997, 2024-03-20T09:05:44.797 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:44.797 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:05:44.797 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:44.797 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:44.797 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:44.797 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 73667 2024-03-20T09:05:44.797 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.797 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:44.797 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.797 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:44.797 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.071968+0000", 2024-03-20T09:05:44.797 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.798 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.798 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.798 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:44.798 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.071968+0000", 2024-03-20T09:05:44.798 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.798 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.798 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.798 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:44.798 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.071969+0000", 2024-03-20T09:05:44.798 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2440000000000001e-06 2024-03-20T09:05:44.798 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.798 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.798 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:44.798 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.072003+0000", 2024-03-20T09:05:44.799 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.4175999999999999e-05 2024-03-20T09:05:44.799 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.799 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.799 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:44.799 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.072014+0000", 2024-03-20T09:05:44.799 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0879999999999999e-05 2024-03-20T09:05:44.799 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.799 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.799 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:44.799 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.072021+0000", 2024-03-20T09:05:44.799 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.8070000000000002e-06 2024-03-20T09:05:44.799 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.799 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.799 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.800 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.072078+0000", 2024-03-20T09:05:44.800 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7265999999999997e-05 2024-03-20T09:05:44.800 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.800 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.800 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:05:44.800 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.072087+0000", 2024-03-20T09:05:44.800 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.1940000000000004e-06 2024-03-20T09:05:44.800 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.800 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.800 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.800 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.900790+0000", 2024-03-20T09:05:44.800 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.82870236200000003 2024-03-20T09:05:44.800 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.800 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.800 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:05:44.801 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.900903+0000", 2024-03-20T09:05:44.801 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000112991 2024-03-20T09:05:44.801 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.801 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.801 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,7", 2024-03-20T09:05:44.801 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.900991+0000", 2024-03-20T09:05:44.801 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.8227999999999996e-05 2024-03-20T09:05:44.801 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.801 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.801 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:44.801 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.904063+0000", 2024-03-20T09:05:44.801 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0030716480000000002 2024-03-20T09:05:44.801 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.801 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.802 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:05:44.802 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.906820+0000", 2024-03-20T09:05:44.802 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0027573020000000001 2024-03-20T09:05:44.802 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.802 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.802 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:44.802 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.913739+0000", 2024-03-20T09:05:44.802 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0069191399999999998 2024-03-20T09:05:44.802 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.802 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.802 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:05:44.802 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.913784+0000", 2024-03-20T09:05:44.802 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4959000000000001e-05 2024-03-20T09:05:44.802 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.803 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.803 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:44.803 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.913817+0000", 2024-03-20T09:05:44.803 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.2833999999999998e-05 2024-03-20T09:05:44.803 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.803 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:44.803 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.803 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.803 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.803 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:73669 3.13 3:cf2f44af:::benchmark_data_smithi122_119671_object73668:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-03-20T09:05:44.803 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:03:56.073404+0000", 2024-03-20T09:05:44.803 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 108.621957877, 2024-03-20T09:05:44.803 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.84052612800000004, 2024-03-20T09:05:44.803 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:44.804 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:05:44.804 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:44.804 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:44.804 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:44.804 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 73669 2024-03-20T09:05:44.804 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.804 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:44.804 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.804 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:44.804 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.073404+0000", 2024-03-20T09:05:44.804 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.804 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.804 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.804 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:44.804 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.073404+0000", 2024-03-20T09:05:44.805 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.805 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.805 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.805 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:44.805 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.073405+0000", 2024-03-20T09:05:44.805 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2729999999999999e-06 2024-03-20T09:05:44.805 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.805 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.805 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:44.805 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.073436+0000", 2024-03-20T09:05:44.805 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.0586000000000002e-05 2024-03-20T09:05:44.805 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.805 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.805 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:44.806 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.073447+0000", 2024-03-20T09:05:44.806 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1977999999999999e-05 2024-03-20T09:05:44.806 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.806 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.806 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:44.806 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.073456+0000", 2024-03-20T09:05:44.806 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.1349999999999992e-06 2024-03-20T09:05:44.806 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.806 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.806 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.806 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.073505+0000", 2024-03-20T09:05:44.806 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9678000000000001e-05 2024-03-20T09:05:44.806 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.806 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.807 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:05:44.807 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.073512+0000", 2024-03-20T09:05:44.807 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.888e-06 2024-03-20T09:05:44.807 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.807 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.807 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.807 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.904089+0000", 2024-03-20T09:05:44.807 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.83057658400000001 2024-03-20T09:05:44.807 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.807 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.807 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:05:44.807 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.904219+0000", 2024-03-20T09:05:44.807 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013010300000000001 2024-03-20T09:05:44.807 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.808 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.808 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,7", 2024-03-20T09:05:44.808 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.904342+0000", 2024-03-20T09:05:44.808 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012284199999999999 2024-03-20T09:05:44.808 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.808 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.808 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:05:44.808 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.909449+0000", 2024-03-20T09:05:44.808 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.005107604 2024-03-20T09:05:44.808 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.808 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.808 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:44.808 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.909494+0000", 2024-03-20T09:05:44.808 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5062999999999999e-05 2024-03-20T09:05:44.809 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.809 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.809 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:44.809 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.913854+0000", 2024-03-20T09:05:44.809 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0043591309999999996 2024-03-20T09:05:44.809 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.809 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.809 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:05:44.809 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.913898+0000", 2024-03-20T09:05:44.809 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4619000000000001e-05 2024-03-20T09:05:44.809 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.809 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.809 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:44.809 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.913930+0000", 2024-03-20T09:05:44.809 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.1643999999999997e-05 2024-03-20T09:05:44.810 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.810 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:44.810 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.810 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.810 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.810 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:73681 3.13 3:cce4d004:::benchmark_data_smithi122_119671_object73680:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-03-20T09:05:44.810 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:03:56.085105+0000", 2024-03-20T09:05:44.810 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 108.610256459, 2024-03-20T09:05:44.810 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.83404779200000001, 2024-03-20T09:05:44.810 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:44.810 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:05:44.810 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:44.810 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:44.811 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:44.811 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 73681 2024-03-20T09:05:44.811 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.811 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:44.811 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.811 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:44.811 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.085105+0000", 2024-03-20T09:05:44.811 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.811 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.811 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.811 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:44.811 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.085105+0000", 2024-03-20T09:05:44.811 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.811 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.812 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.812 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:44.812 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.085106+0000", 2024-03-20T09:05:44.812 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4050000000000001e-06 2024-03-20T09:05:44.812 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.812 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.812 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:44.812 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.085151+0000", 2024-03-20T09:05:44.812 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4085000000000002e-05 2024-03-20T09:05:44.812 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.812 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.812 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:44.812 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.085165+0000", 2024-03-20T09:05:44.812 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4058e-05 2024-03-20T09:05:44.813 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.813 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.813 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:44.813 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.085172+0000", 2024-03-20T09:05:44.813 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.1930000000000004e-06 2024-03-20T09:05:44.813 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.813 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.813 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.813 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.085245+0000", 2024-03-20T09:05:44.813 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.2886999999999995e-05 2024-03-20T09:05:44.813 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.813 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.813 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:05:44.813 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.085256+0000", 2024-03-20T09:05:44.814 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1528e-05 2024-03-20T09:05:44.814 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.814 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.814 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.814 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.905588+0000", 2024-03-20T09:05:44.814 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.82033186499999999 2024-03-20T09:05:44.814 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.814 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.814 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:05:44.814 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.905704+0000", 2024-03-20T09:05:44.814 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00011563199999999999 2024-03-20T09:05:44.814 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.814 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.814 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,7", 2024-03-20T09:05:44.815 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.905840+0000", 2024-03-20T09:05:44.815 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000136656 2024-03-20T09:05:44.815 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.815 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.815 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:05:44.815 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.909455+0000", 2024-03-20T09:05:44.815 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.003614234 2024-03-20T09:05:44.815 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.815 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.815 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:44.815 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.910214+0000", 2024-03-20T09:05:44.815 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00075964000000000001 2024-03-20T09:05:44.815 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.815 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.815 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:44.816 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.919068+0000", 2024-03-20T09:05:44.816 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0088538579999999992 2024-03-20T09:05:44.816 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.816 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.816 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:05:44.816 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.919112+0000", 2024-03-20T09:05:44.816 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4168999999999997e-05 2024-03-20T09:05:44.816 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.816 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.817 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:44.817 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:03:56.919153+0000", 2024-03-20T09:05:44.817 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.0581999999999999e-05 2024-03-20T09:05:44.817 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.817 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:44.817 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.817 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.817 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.817 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:109480 3.b 3:d0d8f1fe:::benchmark_data_smithi122_119671_object9698:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-03-20T09:05:44.817 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:19.038157+0000", 2024-03-20T09:05:44.817 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 25.657204458999999, 2024-03-20T09:05:44.818 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1400804999999998, 2024-03-20T09:05:44.818 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:44.818 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:05:44.818 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:44.818 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:44.818 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:44.818 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 109480 2024-03-20T09:05:44.818 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.818 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:44.818 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.818 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:44.818 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:19.038157+0000", 2024-03-20T09:05:44.818 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.819 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.819 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.819 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:44.819 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:19.038157+0000", 2024-03-20T09:05:44.819 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.819 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.819 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.819 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:44.819 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:19.038158+0000", 2024-03-20T09:05:44.819 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.7300000000000005e-07 2024-03-20T09:05:44.819 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.819 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.819 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:44.819 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:19.038162+0000", 2024-03-20T09:05:44.819 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.732e-06 2024-03-20T09:05:44.820 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.820 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.820 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:44.820 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:19.038172+0000", 2024-03-20T09:05:44.820 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0519999999999999e-05 2024-03-20T09:05:44.820 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.820 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.820 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:44.820 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:19.038186+0000", 2024-03-20T09:05:44.820 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3648e-05 2024-03-20T09:05:44.820 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.820 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.820 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.820 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:19.038223+0000", 2024-03-20T09:05:44.821 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7032000000000001e-05 2024-03-20T09:05:44.821 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.821 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.821 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:05:44.821 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:19.038229+0000", 2024-03-20T09:05:44.821 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7339999999999996e-06 2024-03-20T09:05:44.821 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.821 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.821 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.821 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:19.965221+0000", 2024-03-20T09:05:44.821 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.92699270600000006 2024-03-20T09:05:44.821 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.821 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.821 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:05:44.822 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:19.965235+0000", 2024-03-20T09:05:44.822 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3523000000000001e-05 2024-03-20T09:05:44.822 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.822 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.822 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.822 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:21.165740+0000", 2024-03-20T09:05:44.822 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2005055360000001 2024-03-20T09:05:44.822 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.822 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.822 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:05:44.822 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:21.167693+0000", 2024-03-20T09:05:44.822 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0019527329999999999 2024-03-20T09:05:44.822 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.823 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.823 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,11", 2024-03-20T09:05:44.823 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:21.167815+0000", 2024-03-20T09:05:44.823 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000121567 2024-03-20T09:05:44.823 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.823 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.823 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:44.823 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:21.176147+0000", 2024-03-20T09:05:44.823 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0083319090000000002 2024-03-20T09:05:44.823 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.823 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.823 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:44.823 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:21.176270+0000", 2024-03-20T09:05:44.823 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012345999999999999 2024-03-20T09:05:44.823 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.824 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.824 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:05:44.824 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:21.176286+0000", 2024-03-20T09:05:44.824 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6039e-05 2024-03-20T09:05:44.824 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.824 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.824 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:05:44.824 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:21.176347+0000", 2024-03-20T09:05:44.824 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1042000000000003e-05 2024-03-20T09:05:44.824 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.824 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.824 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:44.824 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:21.178238+0000", 2024-03-20T09:05:44.825 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0018904460000000001 2024-03-20T09:05:44.825 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.825 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:44.825 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.825 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.825 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.825 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:109529 3.b 3:db62707b:::benchmark_data_smithi122_119671_object9747:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-03-20T09:05:44.825 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:19.605374+0000", 2024-03-20T09:05:44.825 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 25.089987548, 2024-03-20T09:05:44.825 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.58226898, 2024-03-20T09:05:44.825 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:44.825 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:05:44.825 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:44.825 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:44.826 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:44.826 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 109529 2024-03-20T09:05:44.826 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.826 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:44.826 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.826 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:44.826 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:19.605374+0000", 2024-03-20T09:05:44.826 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.826 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.826 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.826 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:44.826 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:19.605374+0000", 2024-03-20T09:05:44.826 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.826 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.827 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.827 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:44.827 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:19.605375+0000", 2024-03-20T09:05:44.827 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2529999999999999e-06 2024-03-20T09:05:44.827 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.827 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.827 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:44.827 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:19.605381+0000", 2024-03-20T09:05:44.827 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6869999999999999e-06 2024-03-20T09:05:44.827 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.827 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.827 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:44.827 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:19.605392+0000", 2024-03-20T09:05:44.827 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1263999999999999e-05 2024-03-20T09:05:44.828 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.828 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.828 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:44.828 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:19.605400+0000", 2024-03-20T09:05:44.828 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.4749999999999996e-06 2024-03-20T09:05:44.828 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.828 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.828 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.828 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:19.605456+0000", 2024-03-20T09:05:44.828 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6675000000000002e-05 2024-03-20T09:05:44.828 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.828 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.828 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:05:44.828 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:19.605465+0000", 2024-03-20T09:05:44.829 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.3869999999999995e-06 2024-03-20T09:05:44.829 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.829 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.829 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.829 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:19.965295+0000", 2024-03-20T09:05:44.829 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.35983058600000001 2024-03-20T09:05:44.829 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.829 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.829 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:05:44.829 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:19.965301+0000", 2024-03-20T09:05:44.829 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8690000000000001e-06 2024-03-20T09:05:44.829 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.829 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.829 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.830 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:21.172815+0000", 2024-03-20T09:05:44.830 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.20751407 2024-03-20T09:05:44.830 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.830 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.830 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:05:44.830 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:21.172925+0000", 2024-03-20T09:05:44.830 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000110066 2024-03-20T09:05:44.830 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.830 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.830 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,11", 2024-03-20T09:05:44.830 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:21.173029+0000", 2024-03-20T09:05:44.830 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000103418 2024-03-20T09:05:44.830 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.831 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.831 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:44.831 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:21.178316+0000", 2024-03-20T09:05:44.831 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0052877130000000003 2024-03-20T09:05:44.831 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.831 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.831 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:05:44.831 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:21.181584+0000", 2024-03-20T09:05:44.831 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0032674470000000001 2024-03-20T09:05:44.831 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.831 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.831 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:44.831 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:21.185522+0000", 2024-03-20T09:05:44.831 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0039379430000000002 2024-03-20T09:05:44.831 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.832 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.832 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:05:44.832 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:21.185589+0000", 2024-03-20T09:05:44.832 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.6784000000000003e-05 2024-03-20T09:05:44.832 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.832 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.832 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:44.832 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:21.187643+0000", 2024-03-20T09:05:44.832 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0020543430000000001 2024-03-20T09:05:44.832 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.832 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:44.832 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.832 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.832 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.833 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:109531 3.b 3:da546898:::benchmark_data_smithi122_119671_object9749:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-03-20T09:05:44.833 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:20.185154+0000", 2024-03-20T09:05:44.833 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 24.510207871999999, 2024-03-20T09:05:44.833 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0163651339999999, 2024-03-20T09:05:44.833 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:44.833 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:05:44.833 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:44.833 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:44.833 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:44.833 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 109531 2024-03-20T09:05:44.833 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.833 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:44.833 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.833 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:44.834 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:20.185154+0000", 2024-03-20T09:05:44.834 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.834 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.834 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.834 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:44.834 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:20.185154+0000", 2024-03-20T09:05:44.834 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:44.834 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.834 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.834 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:44.834 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:20.185155+0000", 2024-03-20T09:05:44.834 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.187e-06 2024-03-20T09:05:44.834 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.834 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.835 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:44.835 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:20.185161+0000", 2024-03-20T09:05:44.835 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0630000000000002e-06 2024-03-20T09:05:44.835 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.835 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.835 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:44.835 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:20.185172+0000", 2024-03-20T09:05:44.835 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0597999999999999e-05 2024-03-20T09:05:44.835 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.835 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.835 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:44.835 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:20.185188+0000", 2024-03-20T09:05:44.835 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6101000000000001e-05 2024-03-20T09:05:44.835 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.836 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.836 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.836 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:20.185254+0000", 2024-03-20T09:05:44.836 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.6290000000000004e-05 2024-03-20T09:05:44.836 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.836 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.836 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:05:44.836 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:20.185266+0000", 2024-03-20T09:05:44.836 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2067e-05 2024-03-20T09:05:44.836 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.836 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.836 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:44.836 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:21.178357+0000", 2024-03-20T09:05:44.836 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.99309064499999999 2024-03-20T09:05:44.836 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.837 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.837 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:05:44.837 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:21.178465+0000", 2024-03-20T09:05:44.837 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00010832699999999999 2024-03-20T09:05:44.837 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.837 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.837 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,11", 2024-03-20T09:05:44.837 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:21.178572+0000", 2024-03-20T09:05:44.837 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000107549 2024-03-20T09:05:44.837 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.837 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.837 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:44.837 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:21.187723+0000", 2024-03-20T09:05:44.837 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0091504459999999996 2024-03-20T09:05:44.838 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.838 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.838 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:05:44.838 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:21.187744+0000", 2024-03-20T09:05:44.838 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0594e-05 2024-03-20T09:05:44.838 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.838 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.838 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:44.838 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:21.199527+0000", 2024-03-20T09:05:44.838 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.011783685 2024-03-20T09:05:44.838 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.838 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.838 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:05:44.838 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:21.199567+0000", 2024-03-20T09:05:44.839 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.9968000000000001e-05 2024-03-20T09:05:44.839 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:44.839 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:44.839 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:44.839 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:21.201519+0000", 2024-03-20T09:05:44.839 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001951614 2024-03-20T09:05:44.839 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.839 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:44.839 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.839 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:44.839 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:44.839 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:05:44.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:44 smithi026 ceph-mon[68309]: osdmap e303: 12 total, 11 up, 10 in 2024-03-20T09:05:45.075 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:44 smithi115 ceph-mon[72476]: osdmap e303: 12 total, 11 up, 10 in 2024-03-20T09:05:45.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:44 smithi122 ceph-mon[73495]: osdmap e303: 12 total, 11 up, 10 in 2024-03-20T09:05:45.458 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-20T09:05:45.784 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:05:46.061 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:45 smithi115 ceph-mon[72476]: pgmap v778: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 27 active+clean, 6 active+undersized; 5.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 110 op/s; 26839/262542 objects degraded (10.223%); 22881/262542 objects misplaced (8.715%); 13 MiB/s, 237 objects/s recovering 2024-03-20T09:05:46.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:45 smithi122 ceph-mon[73495]: pgmap v778: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 27 active+clean, 6 active+undersized; 5.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 110 op/s; 26839/262542 objects degraded (10.223%); 22881/262542 objects misplaced (8.715%); 13 MiB/s, 237 objects/s recovering 2024-03-20T09:05:46.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:45 smithi026 ceph-mon[68309]: pgmap v778: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 27 active+clean, 6 active+undersized; 5.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 110 op/s; 26839/262542 objects degraded (10.223%); 22881/262542 objects misplaced (8.715%); 13 MiB/s, 237 objects/s recovering 2024-03-20T09:05:47.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:46 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 26839/262542 objects degraded (10.223%), 12 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-03-20T09:05:47.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:46 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 26839/262542 objects degraded (10.223%), 12 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-03-20T09:05:47.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:46 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 26839/262542 objects degraded (10.223%), 12 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-03-20T09:05:47.857 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:05:47.857 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [ 2024-03-20T09:05:47.857 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.857 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:112280 3.d 3:ba3929c2:::benchmark_data_smithi122_119671_object12498:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-03-20T09:05:47.857 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:46.336235+0000", 2024-03-20T09:05:47.857 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 1.5186649160000001, 2024-03-20T09:05:47.857 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5186900400000001, 2024-03-20T09:05:47.858 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:47.858 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for rw locks", 2024-03-20T09:05:47.858 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:47.858 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:47.858 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:47.858 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 112280 2024-03-20T09:05:47.858 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.858 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:47.858 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.858 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:47.858 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.336235+0000", 2024-03-20T09:05:47.858 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.859 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.859 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.859 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:47.859 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.336235+0000", 2024-03-20T09:05:47.859 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.859 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.859 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.859 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:47.859 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.336237+0000", 2024-03-20T09:05:47.859 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.812e-06 2024-03-20T09:05:47.859 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.859 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.859 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:47.859 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.336246+0000", 2024-03-20T09:05:47.859 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.3520000000000007e-06 2024-03-20T09:05:47.860 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.860 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.860 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:47.860 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.336261+0000", 2024-03-20T09:05:47.860 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5115e-05 2024-03-20T09:05:47.860 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.860 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.860 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:47.860 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.336270+0000", 2024-03-20T09:05:47.860 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.9150000000000002e-06 2024-03-20T09:05:47.860 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.860 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.860 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:47.860 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.397038+0000", 2024-03-20T09:05:47.860 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.060768084999999999 2024-03-20T09:05:47.861 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.861 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.861 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:05:47.861 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.407396+0000", 2024-03-20T09:05:47.861 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.010358615 2024-03-20T09:05:47.861 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.861 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.861 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:47.861 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.541108+0000", 2024-03-20T09:05:47.861 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.13371170700000001 2024-03-20T09:05:47.861 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.861 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.861 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:05:47.861 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.549471+0000", 2024-03-20T09:05:47.862 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0083625490000000004 2024-03-20T09:05:47.862 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.862 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:47.862 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.862 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.862 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.862 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:112392 3.d 3:be9f5588:::benchmark_data_smithi122_119671_object12610:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-03-20T09:05:47.862 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:47.628268+0000", 2024-03-20T09:05:47.862 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.22663230400000001, 2024-03-20T09:05:47.862 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.22671258, 2024-03-20T09:05:47.862 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:47.862 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for rw locks", 2024-03-20T09:05:47.862 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:47.862 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:47.863 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:47.863 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 112392 2024-03-20T09:05:47.863 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.863 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:47.863 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.863 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:47.863 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.628268+0000", 2024-03-20T09:05:47.863 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.863 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.863 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.863 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:47.863 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.628268+0000", 2024-03-20T09:05:47.863 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.863 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.863 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.864 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:47.864 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.628270+0000", 2024-03-20T09:05:47.864 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.858e-06 2024-03-20T09:05:47.864 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.864 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.864 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:47.864 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.628278+0000", 2024-03-20T09:05:47.864 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7219999999999999e-06 2024-03-20T09:05:47.864 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.864 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.864 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:47.864 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.628293+0000", 2024-03-20T09:05:47.864 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4911e-05 2024-03-20T09:05:47.864 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.865 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.865 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:47.865 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.628302+0000", 2024-03-20T09:05:47.865 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.9979999999999992e-06 2024-03-20T09:05:47.865 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.865 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.865 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:47.865 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.695436+0000", 2024-03-20T09:05:47.865 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.067134073000000002 2024-03-20T09:05:47.865 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.865 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.865 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:05:47.865 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.711757+0000", 2024-03-20T09:05:47.865 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.016321473 2024-03-20T09:05:47.865 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.866 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.866 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:47.866 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.838436+0000", 2024-03-20T09:05:47.866 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.12667895700000001 2024-03-20T09:05:47.866 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.866 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.866 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:05:47.866 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.848598+0000", 2024-03-20T09:05:47.866 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.010161554999999999 2024-03-20T09:05:47.866 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.866 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:47.866 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.866 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.866 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.867 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:112258 3.d 3:b8a7a014:::benchmark_data_smithi122_119671_object12476:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-03-20T09:05:47.867 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:45.886262+0000", 2024-03-20T09:05:47.867 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 1.968638761, 2024-03-20T09:05:47.867 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.968767092, 2024-03-20T09:05:47.867 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:47.867 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for rw locks", 2024-03-20T09:05:47.867 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:47.867 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:47.867 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:47.867 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 112258 2024-03-20T09:05:47.867 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.867 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:47.867 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.868 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:47.868 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.886262+0000", 2024-03-20T09:05:47.868 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.868 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.868 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.868 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:47.868 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.886262+0000", 2024-03-20T09:05:47.868 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.868 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.868 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.868 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:47.868 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.886263+0000", 2024-03-20T09:05:47.868 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.843e-06 2024-03-20T09:05:47.869 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.869 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.869 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:47.869 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.886271+0000", 2024-03-20T09:05:47.869 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7549999999999994e-06 2024-03-20T09:05:47.869 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.869 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.869 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:47.869 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.886287+0000", 2024-03-20T09:05:47.869 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5758999999999999e-05 2024-03-20T09:05:47.869 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.869 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.869 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:47.869 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.886297+0000", 2024-03-20T09:05:47.870 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.7249999999999992e-06 2024-03-20T09:05:47.870 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.870 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.870 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:47.870 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.939161+0000", 2024-03-20T09:05:47.870 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.052864046999999997 2024-03-20T09:05:47.870 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.870 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.870 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:05:47.870 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.947709+0000", 2024-03-20T09:05:47.870 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0085477880000000006 2024-03-20T09:05:47.870 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.870 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.870 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:47.871 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.022608+0000", 2024-03-20T09:05:47.871 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.074899462999999999 2024-03-20T09:05:47.871 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.871 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.871 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:05:47.871 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.032931+0000", 2024-03-20T09:05:47.871 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.010323113 2024-03-20T09:05:47.871 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.871 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:47.871 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.871 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.871 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.871 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:112228 3.d 3:bc81f87f:::benchmark_data_smithi122_119671_object12446:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-03-20T09:05:47.871 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:45.616245+0000", 2024-03-20T09:05:47.871 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 2.238655455, 2024-03-20T09:05:47.872 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2388325180000002, 2024-03-20T09:05:47.872 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:47.872 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for rw locks", 2024-03-20T09:05:47.872 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:47.872 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:47.872 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:47.872 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 112228 2024-03-20T09:05:47.872 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.872 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:47.872 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.872 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:47.872 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.616245+0000", 2024-03-20T09:05:47.872 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.872 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.873 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.873 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:47.873 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.616245+0000", 2024-03-20T09:05:47.873 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.873 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.873 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.873 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:47.873 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.616247+0000", 2024-03-20T09:05:47.873 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.875e-06 2024-03-20T09:05:47.873 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.873 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.873 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:47.873 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.616254+0000", 2024-03-20T09:05:47.877 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.1899999999999998e-06 2024-03-20T09:05:47.878 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.878 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.878 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:47.878 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.616270+0000", 2024-03-20T09:05:47.878 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6512e-05 2024-03-20T09:05:47.878 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.878 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.878 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:47.878 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.616281+0000", 2024-03-20T09:05:47.878 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0219e-05 2024-03-20T09:05:47.878 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.878 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.878 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:47.879 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.761165+0000", 2024-03-20T09:05:47.879 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.144883817 2024-03-20T09:05:47.879 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.879 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.879 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:05:47.879 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.778818+0000", 2024-03-20T09:05:47.879 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.017653800000000001 2024-03-20T09:05:47.879 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.879 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.879 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:47.879 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.890246+0000", 2024-03-20T09:05:47.879 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.111427737 2024-03-20T09:05:47.879 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.879 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.880 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:05:47.880 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.899296+0000", 2024-03-20T09:05:47.880 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0090500019999999997 2024-03-20T09:05:47.880 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.880 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:47.880 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.880 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.880 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.880 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:112291 3.d 3:b2fd23b1:::benchmark_data_smithi122_119671_object12509:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-03-20T09:05:47.880 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:46.469063+0000", 2024-03-20T09:05:47.880 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 1.3858372130000001, 2024-03-20T09:05:47.881 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.386060198, 2024-03-20T09:05:47.881 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:47.881 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for rw locks", 2024-03-20T09:05:47.881 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:47.881 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:47.881 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:47.881 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 112291 2024-03-20T09:05:47.881 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.881 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:47.881 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.881 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:47.881 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.469063+0000", 2024-03-20T09:05:47.881 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.881 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.882 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.882 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:47.882 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.469063+0000", 2024-03-20T09:05:47.882 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.882 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.882 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.882 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:47.882 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.469064+0000", 2024-03-20T09:05:47.882 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2330000000000001e-06 2024-03-20T09:05:47.882 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.882 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.882 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:47.882 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.469071+0000", 2024-03-20T09:05:47.882 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.6440000000000003e-06 2024-03-20T09:05:47.883 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.883 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.883 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:47.883 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.469083+0000", 2024-03-20T09:05:47.883 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2442e-05 2024-03-20T09:05:47.883 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.883 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.883 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:47.883 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.469091+0000", 2024-03-20T09:05:47.883 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3799999999999996e-06 2024-03-20T09:05:47.883 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.883 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.883 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:47.883 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.585737+0000", 2024-03-20T09:05:47.883 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.116645844 2024-03-20T09:05:47.884 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.884 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.884 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:05:47.884 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.596192+0000", 2024-03-20T09:05:47.884 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.010455048 2024-03-20T09:05:47.884 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.884 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.884 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:47.884 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.685973+0000", 2024-03-20T09:05:47.884 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.089780992000000004 2024-03-20T09:05:47.884 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.884 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.884 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:05:47.884 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.694319+0000", 2024-03-20T09:05:47.885 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0083457919999999994 2024-03-20T09:05:47.885 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.885 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:47.885 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.885 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.885 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.885 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:112328 3.d 3:bf93c3cf:::benchmark_data_smithi122_119671_object12546:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-03-20T09:05:47.885 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:46.852113+0000", 2024-03-20T09:05:47.885 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 1.002787235, 2024-03-20T09:05:47.885 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0030555889999999, 2024-03-20T09:05:47.885 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:47.885 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for rw locks", 2024-03-20T09:05:47.885 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:47.885 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:47.885 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:47.886 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 112328 2024-03-20T09:05:47.886 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.886 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:47.886 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.886 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:47.886 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.852113+0000", 2024-03-20T09:05:47.886 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.886 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.886 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.886 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:47.886 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.852113+0000", 2024-03-20T09:05:47.886 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.886 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.886 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.887 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:47.887 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.852115+0000", 2024-03-20T09:05:47.887 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4929999999999999e-06 2024-03-20T09:05:47.887 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.887 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.887 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:47.887 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.852121+0000", 2024-03-20T09:05:47.887 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4670000000000001e-06 2024-03-20T09:05:47.887 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.887 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.887 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:47.887 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.852136+0000", 2024-03-20T09:05:47.887 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4901e-05 2024-03-20T09:05:47.887 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.887 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.852145+0000", 2024-03-20T09:05:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.7979999999999995e-06 2024-03-20T09:05:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.928279+0000", 2024-03-20T09:05:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.076134194000000002 2024-03-20T09:05:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:05:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.941618+0000", 2024-03-20T09:05:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.013338859 2024-03-20T09:05:47.888 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.889 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.889 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:47.889 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.983667+0000", 2024-03-20T09:05:47.889 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.042048726000000002 2024-03-20T09:05:47.889 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.889 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.889 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:05:47.889 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.994183+0000", 2024-03-20T09:05:47.889 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.010516723 2024-03-20T09:05:47.889 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.889 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:47.889 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.889 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.889 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.889 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:112374 3.d 3:b98f606a:::benchmark_data_smithi122_119671_object12592:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-03-20T09:05:47.890 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:47.477723+0000", 2024-03-20T09:05:47.890 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.37717707499999997, 2024-03-20T09:05:47.890 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.37750698599999999, 2024-03-20T09:05:47.890 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:47.890 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for rw locks", 2024-03-20T09:05:47.890 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:47.890 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:47.890 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:47.890 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 112374 2024-03-20T09:05:47.890 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.890 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:47.890 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.890 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:47.890 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.477723+0000", 2024-03-20T09:05:47.891 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.891 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.891 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.891 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:47.891 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.477723+0000", 2024-03-20T09:05:47.891 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.891 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.891 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.891 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:47.891 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.477725+0000", 2024-03-20T09:05:47.891 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.689e-06 2024-03-20T09:05:47.891 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.891 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.891 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:47.892 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.477733+0000", 2024-03-20T09:05:47.892 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.8280000000000003e-06 2024-03-20T09:05:47.892 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.892 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.892 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:47.892 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.477747+0000", 2024-03-20T09:05:47.892 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3981e-05 2024-03-20T09:05:47.892 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.892 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.892 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:47.892 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.477756+0000", 2024-03-20T09:05:47.892 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.7679999999999997e-06 2024-03-20T09:05:47.892 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.892 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.892 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:47.893 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.669175+0000", 2024-03-20T09:05:47.893 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.19141962900000001 2024-03-20T09:05:47.893 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.893 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.893 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:05:47.893 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.681627+0000", 2024-03-20T09:05:47.893 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.012451533000000001 2024-03-20T09:05:47.893 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.893 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.893 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:47.893 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.748323+0000", 2024-03-20T09:05:47.893 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.066696351000000001 2024-03-20T09:05:47.893 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.893 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.894 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:05:47.894 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.756868+0000", 2024-03-20T09:05:47.894 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0085453830000000001 2024-03-20T09:05:47.894 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.894 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:47.894 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.894 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.894 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.894 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:112266 3.d 3:b5e01d99:::benchmark_data_smithi122_119671_object12484:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-03-20T09:05:47.894 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:45.966962+0000", 2024-03-20T09:05:47.894 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 1.8879388029999999, 2024-03-20T09:05:47.894 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8883143420000001, 2024-03-20T09:05:47.894 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:47.894 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for rw locks", 2024-03-20T09:05:47.895 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:47.895 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:47.895 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:47.895 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 112266 2024-03-20T09:05:47.895 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.895 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:47.895 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.895 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:47.895 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.966962+0000", 2024-03-20T09:05:47.895 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.895 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.895 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.895 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:47.895 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.966962+0000", 2024-03-20T09:05:47.895 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.896 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.896 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.896 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:47.896 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.966964+0000", 2024-03-20T09:05:47.896 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0200000000000001e-06 2024-03-20T09:05:47.896 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.896 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.896 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:47.896 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.966973+0000", 2024-03-20T09:05:47.896 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.2599999999999994e-06 2024-03-20T09:05:47.896 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.896 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.896 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:47.896 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.966989+0000", 2024-03-20T09:05:47.897 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6554000000000001e-05 2024-03-20T09:05:47.897 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.897 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.897 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:47.897 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.967000+0000", 2024-03-20T09:05:47.897 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0506000000000001e-05 2024-03-20T09:05:47.897 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.897 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.897 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:47.897 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.104273+0000", 2024-03-20T09:05:47.897 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.13727319700000001 2024-03-20T09:05:47.897 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.897 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.897 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:05:47.897 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.113717+0000", 2024-03-20T09:05:47.898 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0094440610000000001 2024-03-20T09:05:47.898 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.898 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.898 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:47.898 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.187324+0000", 2024-03-20T09:05:47.898 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.073606678999999994 2024-03-20T09:05:47.898 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.898 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.898 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:05:47.898 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.196912+0000", 2024-03-20T09:05:47.898 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0095883540000000003 2024-03-20T09:05:47.898 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.898 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:47.898 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.899 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.899 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.899 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "MOSDPGPull(3.15 e303/239 cost 1049576)", 2024-03-20T09:05:47.899 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:45.976510+0000", 2024-03-20T09:05:47.899 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 1.8783907480000002, 2024-03-20T09:05:47.899 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.878809255, 2024-03-20T09:05:47.899 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:47.899 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "queued for pg", 2024-03-20T09:05:47.899 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:47.899 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.899 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:47.899 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.976510+0000", 2024-03-20T09:05:47.899 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.899 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.899 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.900 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:47.900 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.976510+0000", 2024-03-20T09:05:47.900 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.900 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.900 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.900 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:47.900 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.976511+0000", 2024-03-20T09:05:47.900 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3489999999999999e-06 2024-03-20T09:05:47.900 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.900 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.900 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:47.900 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.976525+0000", 2024-03-20T09:05:47.900 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4260000000000001e-05 2024-03-20T09:05:47.900 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.900 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.901 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:47.901 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.976537+0000", 2024-03-20T09:05:47.901 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1865000000000001e-05 2024-03-20T09:05:47.901 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.901 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.901 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:47.901 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.976545+0000", 2024-03-20T09:05:47.901 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.994e-06 2024-03-20T09:05:47.901 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.901 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:47.901 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.901 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.901 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.901 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:112403 3.d 3:ba126cd0:::benchmark_data_smithi122_119671_object12621:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-03-20T09:05:47.902 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:47.691241+0000", 2024-03-20T09:05:47.902 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.16365948899999999, 2024-03-20T09:05:47.902 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.16410814900000001, 2024-03-20T09:05:47.902 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:47.902 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:05:47.902 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:47.902 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:47.902 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:47.902 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 112403 2024-03-20T09:05:47.902 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.902 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:47.902 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.902 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:47.902 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.691241+0000", 2024-03-20T09:05:47.903 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.903 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.903 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.903 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:47.903 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.691241+0000", 2024-03-20T09:05:47.903 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.903 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.903 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.903 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:47.903 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.691243+0000", 2024-03-20T09:05:47.903 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9130000000000001e-06 2024-03-20T09:05:47.903 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.903 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.903 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:47.904 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.691251+0000", 2024-03-20T09:05:47.904 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.2279999999999998e-06 2024-03-20T09:05:47.904 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.904 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.904 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:47.904 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.691266+0000", 2024-03-20T09:05:47.904 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4661000000000001e-05 2024-03-20T09:05:47.904 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.904 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.904 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:47.904 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.691275+0000", 2024-03-20T09:05:47.904 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.2040000000000003e-06 2024-03-20T09:05:47.904 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.904 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.904 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:47.905 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.849624+0000", 2024-03-20T09:05:47.905 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.15834868399999999 2024-03-20T09:05:47.905 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.905 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:47.905 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.905 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.905 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.905 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:112351 3.d 3:bde9227a:::benchmark_data_smithi122_119671_object12569:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-03-20T09:05:47.905 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:47.052333+0000", 2024-03-20T09:05:47.905 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.80256771299999996, 2024-03-20T09:05:47.905 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.80305206299999998, 2024-03-20T09:05:47.905 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:47.905 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for rw locks", 2024-03-20T09:05:47.905 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:47.906 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:47.906 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:47.906 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 112351 2024-03-20T09:05:47.906 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.906 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:47.906 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.906 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:47.906 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.052333+0000", 2024-03-20T09:05:47.906 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.906 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.906 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.906 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:47.906 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.052333+0000", 2024-03-20T09:05:47.906 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.907 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.907 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.907 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:47.907 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.052334+0000", 2024-03-20T09:05:47.907 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.099e-06 2024-03-20T09:05:47.907 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.907 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.907 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:47.907 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.052338+0000", 2024-03-20T09:05:47.907 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.7079999999999996e-06 2024-03-20T09:05:47.907 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.907 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.907 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:47.907 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.052351+0000", 2024-03-20T09:05:47.907 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2225999999999999e-05 2024-03-20T09:05:47.908 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.908 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.908 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:47.908 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.052358+0000", 2024-03-20T09:05:47.908 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.4580000000000004e-06 2024-03-20T09:05:47.908 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.908 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.908 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:47.908 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.219615+0000", 2024-03-20T09:05:47.908 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.16725672899999999 2024-03-20T09:05:47.908 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.908 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.908 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:05:47.908 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.229422+0000", 2024-03-20T09:05:47.909 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0098074919999999993 2024-03-20T09:05:47.909 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.909 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.909 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:47.909 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.315536+0000", 2024-03-20T09:05:47.909 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.086113442999999998 2024-03-20T09:05:47.909 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.909 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.909 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:05:47.909 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.324473+0000", 2024-03-20T09:05:47.909 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0089373249999999994 2024-03-20T09:05:47.909 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.909 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:47.909 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.910 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.910 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.910 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "MOSDPGPush(3.d 303/275 [PushOp(3:be9f5588:::benchmark_data_smithi122_119671_object12610:head, version: 216'773, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:be9f5588:::benchmark_data_smithi122_119671_object12610:head@216'773, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-20T09:05:47.910 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:47.712342+0000", 2024-03-20T09:05:47.910 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.142558252, 2024-03-20T09:05:47.910 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.14311447899999999, 2024-03-20T09:05:47.910 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:47.910 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:05:47.910 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:47.910 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.910 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:47.910 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.712342+0000", 2024-03-20T09:05:47.910 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.910 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.911 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.911 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:47.911 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.712342+0000", 2024-03-20T09:05:47.911 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.911 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.911 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.911 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:47.911 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.712343+0000", 2024-03-20T09:05:47.911 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.9299999999999997e-07 2024-03-20T09:05:47.911 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.911 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.911 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:47.911 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.712367+0000", 2024-03-20T09:05:47.911 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.4474e-05 2024-03-20T09:05:47.911 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.912 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.912 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:47.912 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.712381+0000", 2024-03-20T09:05:47.912 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4109e-05 2024-03-20T09:05:47.912 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.912 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.912 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:47.912 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.712393+0000", 2024-03-20T09:05:47.912 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2051e-05 2024-03-20T09:05:47.912 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.912 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.912 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:47.912 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.801404+0000", 2024-03-20T09:05:47.912 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.089010479000000003 2024-03-20T09:05:47.913 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.913 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.913 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:05:47.913 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.802236+0000", 2024-03-20T09:05:47.913 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00083185799999999999 2024-03-20T09:05:47.913 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.913 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:47.913 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.913 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.913 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.913 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "MOSDPGPull(3.15 e303/239 cost 1049576)", 2024-03-20T09:05:47.913 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:47.113813+0000", 2024-03-20T09:05:47.913 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.74108695999999996, 2024-03-20T09:05:47.913 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.741675376, 2024-03-20T09:05:47.913 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:47.914 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "queued for pg", 2024-03-20T09:05:47.914 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:47.914 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.914 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:47.914 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.113813+0000", 2024-03-20T09:05:47.914 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.914 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.914 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.914 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:47.914 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.113813+0000", 2024-03-20T09:05:47.914 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.914 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.914 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.914 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:47.915 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.113815+0000", 2024-03-20T09:05:47.915 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.319e-06 2024-03-20T09:05:47.915 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.915 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.915 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:47.915 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.113829+0000", 2024-03-20T09:05:47.915 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.471e-05 2024-03-20T09:05:47.915 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.915 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.915 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:47.915 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.113842+0000", 2024-03-20T09:05:47.915 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2533e-05 2024-03-20T09:05:47.915 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.915 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.915 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:47.916 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.113851+0000", 2024-03-20T09:05:47.916 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.7879999999999995e-06 2024-03-20T09:05:47.916 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.916 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:47.916 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.916 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.916 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.916 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "MOSDPGPull(3.15 e303/239 cost 1049576)", 2024-03-20T09:05:47.916 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:45.560552+0000", 2024-03-20T09:05:47.916 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 2.2943479829999998, 2024-03-20T09:05:47.916 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2949662119999998, 2024-03-20T09:05:47.916 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:47.916 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "queued for pg", 2024-03-20T09:05:47.916 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:47.916 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.917 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:47.917 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.560552+0000", 2024-03-20T09:05:47.917 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.917 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.917 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.917 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:47.917 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.560552+0000", 2024-03-20T09:05:47.917 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.917 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.917 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.917 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:47.917 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.560553+0000", 2024-03-20T09:05:47.917 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.5199999999999996e-07 2024-03-20T09:05:47.917 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.918 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.918 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:47.918 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.560562+0000", 2024-03-20T09:05:47.918 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.0750000000000004e-06 2024-03-20T09:05:47.918 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.918 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.918 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:47.918 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.560570+0000", 2024-03-20T09:05:47.918 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7610000000000007e-06 2024-03-20T09:05:47.918 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.918 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.918 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:47.918 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.560574+0000", 2024-03-20T09:05:47.918 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.1910000000000002e-06 2024-03-20T09:05:47.918 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.919 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:47.919 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.919 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.919 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.919 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "MOSDPGPull(3.15 e303/239 cost 1049576)", 2024-03-20T09:05:47.919 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:46.229175+0000", 2024-03-20T09:05:47.919 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 1.6257251350000002, 2024-03-20T09:05:47.919 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.626371147, 2024-03-20T09:05:47.919 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:47.919 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "queued for pg", 2024-03-20T09:05:47.919 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:47.919 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.919 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:47.919 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.229175+0000", 2024-03-20T09:05:47.920 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.920 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.920 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.920 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:47.920 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.229175+0000", 2024-03-20T09:05:47.920 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.920 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.920 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.920 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:47.920 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.229177+0000", 2024-03-20T09:05:47.920 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.288e-06 2024-03-20T09:05:47.920 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.920 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.920 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:47.920 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.229200+0000", 2024-03-20T09:05:47.921 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3493999999999999e-05 2024-03-20T09:05:47.921 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.921 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.921 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:47.921 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.229212+0000", 2024-03-20T09:05:47.921 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2395000000000001e-05 2024-03-20T09:05:47.921 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.921 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.921 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:47.921 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.229220+0000", 2024-03-20T09:05:47.921 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7519999999999996e-06 2024-03-20T09:05:47.921 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.921 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:47.921 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.921 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.922 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.922 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:112357 3.d 3:bd2cfc63:::benchmark_data_smithi122_119671_object12575:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-03-20T09:05:47.922 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:47.160624+0000", 2024-03-20T09:05:47.922 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.69427591300000002, 2024-03-20T09:05:47.922 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.69495099800000004, 2024-03-20T09:05:47.922 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:47.922 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for rw locks", 2024-03-20T09:05:47.925 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:47.925 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:47.926 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:47.926 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 112357 2024-03-20T09:05:47.926 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.926 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:47.926 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.926 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:47.926 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.160624+0000", 2024-03-20T09:05:47.926 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.926 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.926 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.926 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:47.926 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.160624+0000", 2024-03-20T09:05:47.926 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.926 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.927 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.927 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:47.927 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.160626+0000", 2024-03-20T09:05:47.927 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7799999999999999e-06 2024-03-20T09:05:47.927 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.927 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.927 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:47.927 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.160633+0000", 2024-03-20T09:05:47.927 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.1389999999999998e-06 2024-03-20T09:05:47.927 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.927 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.927 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:47.927 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.160647+0000", 2024-03-20T09:05:47.927 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.395e-05 2024-03-20T09:05:47.927 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.928 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.928 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:47.928 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.160655+0000", 2024-03-20T09:05:47.928 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.1759999999999993e-06 2024-03-20T09:05:47.928 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.928 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.928 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:47.928 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.325359+0000", 2024-03-20T09:05:47.928 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.16470362399999999 2024-03-20T09:05:47.928 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.928 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.928 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:05:47.928 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.334586+0000", 2024-03-20T09:05:47.928 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0092267700000000005 2024-03-20T09:05:47.928 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.929 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.929 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:47.929 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.460264+0000", 2024-03-20T09:05:47.929 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.12567798099999999 2024-03-20T09:05:47.929 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.929 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.929 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:05:47.929 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.470620+0000", 2024-03-20T09:05:47.929 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.010355992 2024-03-20T09:05:47.929 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.929 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:47.929 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.929 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.929 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.930 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:112241 3.d 3:b568d2fc:::benchmark_data_smithi122_119671_object12459:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-03-20T09:05:47.930 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:45.772013+0000", 2024-03-20T09:05:47.930 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 2.082887232, 2024-03-20T09:05:47.930 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0836080350000001, 2024-03-20T09:05:47.930 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:47.930 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for rw locks", 2024-03-20T09:05:47.930 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:47.930 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:47.930 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:47.930 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 112241 2024-03-20T09:05:47.930 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.930 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:47.930 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.930 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:47.930 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.772013+0000", 2024-03-20T09:05:47.931 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.931 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.931 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.931 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:47.931 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.772013+0000", 2024-03-20T09:05:47.931 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.931 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.931 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.931 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:47.931 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.772015+0000", 2024-03-20T09:05:47.931 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.75e-06 2024-03-20T09:05:47.931 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.931 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.931 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:47.932 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.772024+0000", 2024-03-20T09:05:47.932 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.329e-06 2024-03-20T09:05:47.932 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.932 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.932 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:47.932 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.772040+0000", 2024-03-20T09:05:47.932 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5468000000000002e-05 2024-03-20T09:05:47.932 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.932 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.932 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:47.932 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.772048+0000", 2024-03-20T09:05:47.932 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.3869999999999995e-06 2024-03-20T09:05:47.932 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.932 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.933 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:47.933 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.929640+0000", 2024-03-20T09:05:47.933 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.157592435 2024-03-20T09:05:47.933 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.933 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.933 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:05:47.933 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.938159+0000", 2024-03-20T09:05:47.933 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0085187639999999999 2024-03-20T09:05:47.933 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.933 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.933 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:47.933 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.983046+0000", 2024-03-20T09:05:47.933 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.044886641999999997 2024-03-20T09:05:47.933 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.933 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.934 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:05:47.934 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.992423+0000", 2024-03-20T09:05:47.934 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0093772409999999997 2024-03-20T09:05:47.934 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.934 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:47.934 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.934 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.934 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.934 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:112313 3.d 3:bc09e473:::benchmark_data_smithi122_119671_object12531:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-03-20T09:05:47.934 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:46.650129+0000", 2024-03-20T09:05:47.934 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 1.2047718540000001, 2024-03-20T09:05:47.934 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2055381729999999, 2024-03-20T09:05:47.934 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:47.934 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for rw locks", 2024-03-20T09:05:47.935 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:47.935 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:47.935 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:47.935 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 112313 2024-03-20T09:05:47.935 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.935 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:47.935 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.935 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:47.935 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.650129+0000", 2024-03-20T09:05:47.935 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.935 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.935 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.935 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:47.935 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.650129+0000", 2024-03-20T09:05:47.936 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.936 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.936 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.936 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:47.936 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.650129+0000", 2024-03-20T09:05:47.936 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.33e-07 2024-03-20T09:05:47.936 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.936 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.936 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:47.936 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.650134+0000", 2024-03-20T09:05:47.936 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6820000000000002e-06 2024-03-20T09:05:47.936 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.936 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.936 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:47.936 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.650142+0000", 2024-03-20T09:05:47.937 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.8250000000000005e-06 2024-03-20T09:05:47.937 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.937 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.937 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:47.937 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.650147+0000", 2024-03-20T09:05:47.937 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8949999999999997e-06 2024-03-20T09:05:47.937 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.937 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.937 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:47.937 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.785657+0000", 2024-03-20T09:05:47.937 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.13551047399999999 2024-03-20T09:05:47.937 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.937 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.937 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:05:47.938 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.794275+0000", 2024-03-20T09:05:47.938 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0086174609999999999 2024-03-20T09:05:47.938 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.938 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.938 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:47.938 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.830548+0000", 2024-03-20T09:05:47.938 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.036273571999999997 2024-03-20T09:05:47.938 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.938 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.938 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:05:47.938 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.838611+0000", 2024-03-20T09:05:47.938 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0080628179999999994 2024-03-20T09:05:47.938 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.938 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:47.939 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.939 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.939 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.939 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:112339 3.d 3:b6841ffe:::benchmark_data_smithi122_119671_object12557:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-03-20T09:05:47.939 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:46.927772+0000", 2024-03-20T09:05:47.939 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.92712865899999997, 2024-03-20T09:05:47.939 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.92794017399999995, 2024-03-20T09:05:47.939 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:47.939 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for rw locks", 2024-03-20T09:05:47.939 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:47.939 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:47.939 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:47.939 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 112339 2024-03-20T09:05:47.939 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.939 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:47.940 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.940 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:47.940 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.927772+0000", 2024-03-20T09:05:47.940 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.940 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.940 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.940 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:47.940 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.927772+0000", 2024-03-20T09:05:47.940 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.940 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.940 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.940 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:47.940 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.927773+0000", 2024-03-20T09:05:47.940 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.136e-06 2024-03-20T09:05:47.940 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.941 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.941 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:47.941 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.927779+0000", 2024-03-20T09:05:47.941 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8270000000000003e-06 2024-03-20T09:05:47.941 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.941 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.941 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:47.941 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.927791+0000", 2024-03-20T09:05:47.941 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1917e-05 2024-03-20T09:05:47.941 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.941 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.941 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:47.941 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.927797+0000", 2024-03-20T09:05:47.941 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.7939999999999997e-06 2024-03-20T09:05:47.942 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.942 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.942 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:47.942 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.018796+0000", 2024-03-20T09:05:47.942 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.090998692000000006 2024-03-20T09:05:47.942 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.942 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.942 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:05:47.942 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.028772+0000", 2024-03-20T09:05:47.942 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0099763270000000001 2024-03-20T09:05:47.942 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.942 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.942 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:47.942 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.120722+0000", 2024-03-20T09:05:47.942 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.091949912999999994 2024-03-20T09:05:47.943 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.943 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.943 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:05:47.943 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.129082+0000", 2024-03-20T09:05:47.943 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0083599550000000005 2024-03-20T09:05:47.943 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.943 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:47.943 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.943 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.943 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.943 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:112277 3.d 3:b2de22a5:::benchmark_data_smithi122_119671_object12495:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-03-20T09:05:47.943 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:46.322405+0000", 2024-03-20T09:05:47.943 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 1.5324955600000001, 2024-03-20T09:05:47.943 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5333531890000001, 2024-03-20T09:05:47.943 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:47.944 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for rw locks", 2024-03-20T09:05:47.944 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:47.944 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:47.944 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:47.944 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 112277 2024-03-20T09:05:47.944 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.944 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:47.944 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.944 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:47.944 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.322405+0000", 2024-03-20T09:05:47.944 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.944 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.944 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.944 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:47.945 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.322405+0000", 2024-03-20T09:05:47.945 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.945 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.945 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.945 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:47.945 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.322407+0000", 2024-03-20T09:05:47.945 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9400000000000001e-06 2024-03-20T09:05:47.945 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.945 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.945 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:47.945 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.322415+0000", 2024-03-20T09:05:47.945 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.7390000000000005e-06 2024-03-20T09:05:47.945 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.945 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.945 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:47.946 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.322433+0000", 2024-03-20T09:05:47.946 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7762999999999999e-05 2024-03-20T09:05:47.946 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.946 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.946 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:47.946 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.322443+0000", 2024-03-20T09:05:47.946 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.3759999999999997e-06 2024-03-20T09:05:47.946 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.946 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.946 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:47.946 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.387155+0000", 2024-03-20T09:05:47.946 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.064712686000000005 2024-03-20T09:05:47.946 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.946 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.947 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:05:47.947 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.395910+0000", 2024-03-20T09:05:47.947 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0087548460000000002 2024-03-20T09:05:47.947 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.947 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.947 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:47.947 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.497899+0000", 2024-03-20T09:05:47.947 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.10198855 2024-03-20T09:05:47.947 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.947 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.947 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:05:47.947 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.508448+0000", 2024-03-20T09:05:47.947 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.010549055999999999 2024-03-20T09:05:47.947 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.947 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:47.948 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.948 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.948 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.948 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:112388 3.d 3:b366bd14:::benchmark_data_smithi122_119671_object12606:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-03-20T09:05:47.948 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:47.547541+0000", 2024-03-20T09:05:47.948 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.30735922399999999, 2024-03-20T09:05:47.948 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.308262484, 2024-03-20T09:05:47.948 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:47.948 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for rw locks", 2024-03-20T09:05:47.948 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:47.948 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:47.948 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:47.948 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 112388 2024-03-20T09:05:47.948 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.949 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:47.949 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.949 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:47.949 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.547541+0000", 2024-03-20T09:05:47.949 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.949 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.949 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.949 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:47.949 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.547541+0000", 2024-03-20T09:05:47.949 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:47.949 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.949 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.949 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:47.949 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.547542+0000", 2024-03-20T09:05:47.949 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3430000000000001e-06 2024-03-20T09:05:47.950 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.950 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.950 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:47.950 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.547549+0000", 2024-03-20T09:05:47.950 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.7379999999999997e-06 2024-03-20T09:05:47.950 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.950 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.950 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:47.950 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.547563+0000", 2024-03-20T09:05:47.950 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3305999999999999e-05 2024-03-20T09:05:47.950 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.950 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.950 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:47.950 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.547573+0000", 2024-03-20T09:05:47.951 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0720000000000001e-05 2024-03-20T09:05:47.951 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.951 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.951 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:47.951 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.683191+0000", 2024-03-20T09:05:47.951 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.135617707 2024-03-20T09:05:47.951 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.951 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.951 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:05:47.951 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.694410+0000", 2024-03-20T09:05:47.951 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.011218806 2024-03-20T09:05:47.951 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.951 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.951 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:47.951 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.790939+0000", 2024-03-20T09:05:47.952 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.096529134000000003 2024-03-20T09:05:47.952 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:47.952 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:47.952 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:05:47.952 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.800138+0000", 2024-03-20T09:05:47.952 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0091993600000000002 2024-03-20T09:05:47.952 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.952 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:47.952 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.952 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:47.952 INFO:teuthology.orchestra.run.smithi115.stdout: ], 2024-03-20T09:05:47.952 INFO:teuthology.orchestra.run.smithi115.stdout: "num_ops": 21 2024-03-20T09:05:47.952 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:05:48.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:47 smithi115 ceph-mon[72476]: pgmap v779: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 27 active+clean, 6 active+undersized; 5.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 108 op/s; 26773/262062 objects degraded (10.216%); 22805/262062 objects misplaced (8.702%); 10 MiB/s, 191 objects/s recovering 2024-03-20T09:05:48.174 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:05:48.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:48.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:05:48.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:47 smithi026 ceph-mon[68309]: pgmap v779: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 27 active+clean, 6 active+undersized; 5.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 108 op/s; 26773/262062 objects degraded (10.216%); 22805/262062 objects misplaced (8.702%); 10 MiB/s, 191 objects/s recovering 2024-03-20T09:05:48.280 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:05:48.312 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:48.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:47 smithi122 ceph-mon[73495]: pgmap v779: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 27 active+clean, 6 active+undersized; 5.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 108 op/s; 26773/262062 objects degraded (10.216%); 22805/262062 objects misplaced (8.702%); 10 MiB/s, 191 objects/s recovering 2024-03-20T09:05:48.658 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-20T09:05:48.993 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:05:49.263 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:05:49.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:49.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:05:49.371 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:05:49.406 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:49.886 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:05:49.888 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:49 smithi115 ceph-mon[72476]: pgmap v780: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 27 active+clean, 6 active+undersized; 5.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 100 op/s; 26735/261657 objects degraded (10.218%); 22309/261657 objects misplaced (8.526%); 11 MiB/s, 170 objects/s recovering 2024-03-20T09:05:49.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:49.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:05:49.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:49 smithi026 ceph-mon[68309]: pgmap v780: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 27 active+clean, 6 active+undersized; 5.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 100 op/s; 26735/261657 objects degraded (10.218%); 22309/261657 objects misplaced (8.526%); 11 MiB/s, 170 objects/s recovering 2024-03-20T09:05:49.990 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:05:50.015 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:49 smithi122 ceph-mon[73495]: pgmap v780: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 27 active+clean, 6 active+undersized; 5.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 100 op/s; 26735/261657 objects degraded (10.218%); 22309/261657 objects misplaced (8.526%); 11 MiB/s, 170 objects/s recovering 2024-03-20T09:05:50.017 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:50.037 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:05:50.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:50.041 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:05:50.141 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:05:50.175 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:50.987 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:05:50.987 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:05:50.987 INFO:teuthology.orchestra.run.smithi115.stdout: "complaint_time": 30, 2024-03-20T09:05:50.988 INFO:teuthology.orchestra.run.smithi115.stdout: "num_blocked_ops": 0 2024-03-20T09:05:50.988 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:05:51.757 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-20T09:05:51.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:51 smithi026 ceph-mon[68309]: pgmap v781: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 27 active+clean, 6 active+undersized; 5.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 89 op/s; 26701/261156 objects degraded (10.224%); 21879/261156 objects misplaced (8.378%); 12 MiB/s, 197 objects/s recovering 2024-03-20T09:05:52.103 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:05:52.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:51 smithi115 ceph-mon[72476]: pgmap v781: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 27 active+clean, 6 active+undersized; 5.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 89 op/s; 26701/261156 objects degraded (10.224%); 21879/261156 objects misplaced (8.378%); 12 MiB/s, 197 objects/s recovering 2024-03-20T09:05:52.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:51 smithi122 ceph-mon[73495]: pgmap v781: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 27 active+clean, 6 active+undersized; 5.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 89 op/s; 26701/261156 objects degraded (10.224%); 21879/261156 objects misplaced (8.378%); 12 MiB/s, 197 objects/s recovering 2024-03-20T09:05:52.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:52 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 26701/261156 objects degraded (10.224%), 12 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-03-20T09:05:53.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:52 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 26701/261156 objects degraded (10.224%), 12 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-03-20T09:05:53.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:52 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 26701/261156 objects degraded (10.224%), 12 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-03-20T09:05:53.426 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:05:53.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:53.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:05:53.438 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:05:53.532 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:05:53.563 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:53.868 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:53 smithi115 ceph-mon[72476]: pgmap v782: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 27 active+clean, 6 active+undersized; 5.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 95 op/s; 26617/260562 objects degraded (10.215%); 21799/260562 objects misplaced (8.366%); 11 MiB/s, 179 objects/s recovering 2024-03-20T09:05:53.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:53 smithi026 ceph-mon[68309]: pgmap v782: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 27 active+clean, 6 active+undersized; 5.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 95 op/s; 26617/260562 objects degraded (10.215%); 21799/260562 objects misplaced (8.366%); 11 MiB/s, 179 objects/s recovering 2024-03-20T09:05:54.190 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:05:54.191 INFO:teuthology.orchestra.run.smithi115.stdout: "size": 20, 2024-03-20T09:05:54.191 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 600, 2024-03-20T09:05:54.191 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [ 2024-03-20T09:05:54.191 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.191 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "MOSDPGPull(3.15 e279/239 cost 1049576)", 2024-03-20T09:05:54.191 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:04:40.924123+0000", 2024-03-20T09:05:54.191 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 73.262405794000003, 2024-03-20T09:05:54.191 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 37.917802715999997, 2024-03-20T09:05:54.192 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:54.192 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:05:54.192 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:54.192 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.192 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:54.192 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:40.924123+0000", 2024-03-20T09:05:54.192 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.193 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.193 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.193 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:54.193 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:40.924123+0000", 2024-03-20T09:05:54.193 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.193 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.193 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.194 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:54.194 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:40.924124+0000", 2024-03-20T09:05:54.194 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.5199999999999995e-07 2024-03-20T09:05:54.194 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.194 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.194 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:54.195 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:40.924134+0000", 2024-03-20T09:05:54.195 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0125999999999999e-05 2024-03-20T09:05:54.195 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.195 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.195 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:54.195 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:40.924142+0000", 2024-03-20T09:05:54.196 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.6459999999999993e-06 2024-03-20T09:05:54.196 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.196 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.196 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:54.196 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:40.924147+0000", 2024-03-20T09:05:54.196 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0760000000000002e-06 2024-03-20T09:05:54.196 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.197 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.197 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.197 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:18.841463+0000", 2024-03-20T09:05:54.197 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 37.917316503000002 2024-03-20T09:05:54.197 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.197 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.198 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:54.198 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:18.841926+0000", 2024-03-20T09:05:54.198 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00046241300000000001 2024-03-20T09:05:54.198 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.198 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:54.198 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.199 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.199 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.199 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "MOSDPGPull(3.15 e280/239 cost 1049576)", 2024-03-20T09:05:54.199 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:04:41.964353+0000", 2024-03-20T09:05:54.199 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 72.222175954999997, 2024-03-20T09:05:54.199 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 36.877965906, 2024-03-20T09:05:54.200 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:54.200 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:05:54.200 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:54.200 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.200 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:54.200 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:41.964353+0000", 2024-03-20T09:05:54.200 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.200 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.200 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.200 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:54.200 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:41.964353+0000", 2024-03-20T09:05:54.200 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.200 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.201 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.201 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:54.201 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:41.964353+0000", 2024-03-20T09:05:54.201 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.3799999999999996e-07 2024-03-20T09:05:54.201 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.201 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.201 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:54.201 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:41.964365+0000", 2024-03-20T09:05:54.201 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1314e-05 2024-03-20T09:05:54.201 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.201 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.201 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:54.201 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:41.964374+0000", 2024-03-20T09:05:54.201 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.8799999999999997e-06 2024-03-20T09:05:54.202 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.202 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.202 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:54.202 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:41.964381+0000", 2024-03-20T09:05:54.202 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.9040000000000003e-06 2024-03-20T09:05:54.202 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.202 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.202 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.202 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:18.841946+0000", 2024-03-20T09:05:54.202 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 36.877565808 2024-03-20T09:05:54.202 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.203 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.203 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:54.203 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:18.842319+0000", 2024-03-20T09:05:54.203 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00037216199999999999 2024-03-20T09:05:54.203 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.203 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:54.203 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.203 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.203 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.203 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "MOSDPGPull(3.15 e281/239 cost 1049576)", 2024-03-20T09:05:54.203 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:04:44.130689+0000", 2024-03-20T09:05:54.203 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 70.055839805000005, 2024-03-20T09:05:54.203 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 34.712016781000003, 2024-03-20T09:05:54.203 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:54.204 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:05:54.204 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:54.204 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.204 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:54.204 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:44.130689+0000", 2024-03-20T09:05:54.204 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.204 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.204 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.204 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:54.204 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:44.130689+0000", 2024-03-20T09:05:54.204 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.205 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.205 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.205 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:54.205 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:44.130690+0000", 2024-03-20T09:05:54.205 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1119999999999999e-06 2024-03-20T09:05:54.205 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.205 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.205 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:54.205 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:44.130703+0000", 2024-03-20T09:05:54.205 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3066e-05 2024-03-20T09:05:54.205 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.205 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.205 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:54.206 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:44.130714+0000", 2024-03-20T09:05:54.206 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.136e-05 2024-03-20T09:05:54.206 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.206 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.206 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:54.206 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:44.130722+0000", 2024-03-20T09:05:54.206 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3499999999999999e-06 2024-03-20T09:05:54.206 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.206 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.206 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.206 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:18.842335+0000", 2024-03-20T09:05:54.206 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 34.711612926000001 2024-03-20T09:05:54.206 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.206 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.207 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:54.207 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:18.842706+0000", 2024-03-20T09:05:54.207 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00037096699999999999 2024-03-20T09:05:54.207 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.207 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:54.207 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.207 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.207 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.207 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "MOSDPGPull(3.15 e282/239 cost 1049576)", 2024-03-20T09:05:54.207 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:04:44.978300+0000", 2024-03-20T09:05:54.207 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 69.208228554000002, 2024-03-20T09:05:54.207 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 33.864778774999998, 2024-03-20T09:05:54.207 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:54.207 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:05:54.208 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:54.208 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.208 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:54.208 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:44.978300+0000", 2024-03-20T09:05:54.208 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.208 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.208 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.208 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:54.208 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:44.978300+0000", 2024-03-20T09:05:54.208 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.208 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.208 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.208 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:54.208 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:44.978301+0000", 2024-03-20T09:05:54.208 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.5999999999999991e-07 2024-03-20T09:05:54.209 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.209 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.209 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:54.209 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:44.978323+0000", 2024-03-20T09:05:54.209 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1520999999999999e-05 2024-03-20T09:05:54.209 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.209 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.209 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:54.209 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:44.978334+0000", 2024-03-20T09:05:54.209 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1124000000000001e-05 2024-03-20T09:05:54.209 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.209 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.209 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:54.210 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:44.978340+0000", 2024-03-20T09:05:54.210 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.5780000000000004e-06 2024-03-20T09:05:54.210 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.210 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.210 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.210 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:18.842720+0000", 2024-03-20T09:05:54.211 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 33.864379855000003 2024-03-20T09:05:54.211 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.211 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.211 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:54.211 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:18.843079+0000", 2024-03-20T09:05:54.211 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00035873699999999998 2024-03-20T09:05:54.212 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.212 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:54.212 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.212 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.212 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.212 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "MOSDPGPull(3.15 e283/239 cost 1049576)", 2024-03-20T09:05:54.213 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:04:45.947857+0000", 2024-03-20T09:05:54.213 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 68.238671401999994, 2024-03-20T09:05:54.213 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 32.895625455000001, 2024-03-20T09:05:54.213 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:54.213 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:05:54.213 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:54.213 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.213 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:54.213 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:45.947857+0000", 2024-03-20T09:05:54.213 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.214 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.214 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.214 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:54.214 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:45.947857+0000", 2024-03-20T09:05:54.214 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.214 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.215 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.215 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:54.215 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:45.947858+0000", 2024-03-20T09:05:54.215 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.8400000000000002e-07 2024-03-20T09:05:54.215 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.216 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.216 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:54.216 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:45.947871+0000", 2024-03-20T09:05:54.216 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2337000000000001e-05 2024-03-20T09:05:54.216 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.216 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.217 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:54.217 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:45.947883+0000", 2024-03-20T09:05:54.217 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2334000000000001e-05 2024-03-20T09:05:54.217 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.217 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.217 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:54.218 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:45.947890+0000", 2024-03-20T09:05:54.218 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.8059999999999997e-06 2024-03-20T09:05:54.218 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.218 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.218 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.219 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:18.843094+0000", 2024-03-20T09:05:54.219 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 32.895204374999999 2024-03-20T09:05:54.219 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.219 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.219 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:54.219 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:18.843483+0000", 2024-03-20T09:05:54.220 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00038861899999999999 2024-03-20T09:05:54.220 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.220 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:54.220 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.220 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.220 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.220 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:110609 3.d 3:bf8f3962:::benchmark_data_smithi122_119671_object10827:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-03-20T09:05:54.221 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:29.784111+0000", 2024-03-20T09:05:54.221 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 24.402417481000001, 2024-03-20T09:05:54.221 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.636143729, 2024-03-20T09:05:54.221 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:54.221 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:05:54.221 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:54.221 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:54.222 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:54.222 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 110609 2024-03-20T09:05:54.222 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.222 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:54.222 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.222 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:54.222 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:29.784111+0000", 2024-03-20T09:05:54.223 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.223 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.223 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.223 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:54.223 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:29.784111+0000", 2024-03-20T09:05:54.223 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.223 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.224 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.224 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:54.224 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:29.784113+0000", 2024-03-20T09:05:54.224 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7749999999999999e-06 2024-03-20T09:05:54.224 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.224 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.225 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:54.225 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:29.784121+0000", 2024-03-20T09:05:54.225 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7030000000000006e-06 2024-03-20T09:05:54.225 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.225 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.225 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:54.225 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:29.784139+0000", 2024-03-20T09:05:54.225 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8692e-05 2024-03-20T09:05:54.226 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.226 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.226 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:54.226 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:29.784151+0000", 2024-03-20T09:05:54.226 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2004000000000001e-05 2024-03-20T09:05:54.226 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.226 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.227 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.227 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:29.991217+0000", 2024-03-20T09:05:54.227 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.20706619400000001 2024-03-20T09:05:54.227 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.227 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.227 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:05:54.227 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:30.008282+0000", 2024-03-20T09:05:54.227 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.017064032999999999 2024-03-20T09:05:54.228 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.228 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.228 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.228 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:30.079328+0000", 2024-03-20T09:05:54.228 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.071046434000000006 2024-03-20T09:05:54.228 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.228 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.229 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:05:54.229 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:30.088127+0000", 2024-03-20T09:05:54.229 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0087991370000000003 2024-03-20T09:05:54.229 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.229 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.229 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.229 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:32.276233+0000", 2024-03-20T09:05:54.229 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1881055150000002 2024-03-20T09:05:54.230 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.230 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.230 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:05:54.230 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:32.284264+0000", 2024-03-20T09:05:54.230 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0080314740000000003 2024-03-20T09:05:54.230 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.231 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.231 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:05:54.231 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:32.297871+0000", 2024-03-20T09:05:54.231 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.013607068999999999 2024-03-20T09:05:54.231 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.231 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.232 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:05:54.232 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:32.381709+0000", 2024-03-20T09:05:54.232 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.083837562000000004 2024-03-20T09:05:54.232 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.232 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.232 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:54.232 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:32.383916+0000", 2024-03-20T09:05:54.233 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.002207437 2024-03-20T09:05:54.233 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.233 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.233 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:54.233 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:32.388692+0000", 2024-03-20T09:05:54.233 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0047761929999999998 2024-03-20T09:05:54.233 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.233 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.233 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:05:54.234 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:32.395940+0000", 2024-03-20T09:05:54.234 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0072478589999999997 2024-03-20T09:05:54.234 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.234 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.234 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:54.234 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:32.420255+0000", 2024-03-20T09:05:54.234 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.024314648000000001 2024-03-20T09:05:54.234 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.235 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:54.235 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.235 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.235 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.235 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:110617 3.d 3:b1db244d:::benchmark_data_smithi122_119671_object10835:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-03-20T09:05:54.235 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:29.966376+0000", 2024-03-20T09:05:54.235 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 24.220152912, 2024-03-20T09:05:54.235 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6789749970000001, 2024-03-20T09:05:54.235 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:54.236 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:05:54.236 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:54.236 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:54.236 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:54.236 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 110617 2024-03-20T09:05:54.236 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.236 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:54.237 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.237 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:54.237 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:29.966376+0000", 2024-03-20T09:05:54.237 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.237 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.237 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.238 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:54.238 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:29.966376+0000", 2024-03-20T09:05:54.238 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.238 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.238 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.238 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:54.238 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:29.966377+0000", 2024-03-20T09:05:54.239 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1459999999999999e-06 2024-03-20T09:05:54.239 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.239 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.239 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:54.239 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:29.966382+0000", 2024-03-20T09:05:54.239 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9150000000000004e-06 2024-03-20T09:05:54.240 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.240 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.240 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:54.240 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:29.966393+0000", 2024-03-20T09:05:54.240 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1554e-05 2024-03-20T09:05:54.240 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.241 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.241 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:54.241 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:29.966400+0000", 2024-03-20T09:05:54.241 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.9909999999999996e-06 2024-03-20T09:05:54.241 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.241 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.242 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.242 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:30.024450+0000", 2024-03-20T09:05:54.242 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.058050079999999997 2024-03-20T09:05:54.242 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.242 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.242 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:05:54.242 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:30.035590+0000", 2024-03-20T09:05:54.242 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.011139534 2024-03-20T09:05:54.243 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.243 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.243 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.243 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:30.163835+0000", 2024-03-20T09:05:54.243 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.12824516499999999 2024-03-20T09:05:54.243 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.243 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.243 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:05:54.243 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:30.171879+0000", 2024-03-20T09:05:54.243 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0080434619999999995 2024-03-20T09:05:54.243 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.243 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.243 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.243 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:32.500944+0000", 2024-03-20T09:05:54.244 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3290656919999999 2024-03-20T09:05:54.244 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.244 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.244 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:05:54.244 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:32.510540+0000", 2024-03-20T09:05:54.244 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0095956909999999999 2024-03-20T09:05:54.244 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.244 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.244 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:05:54.244 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:32.524020+0000", 2024-03-20T09:05:54.244 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.013480371 2024-03-20T09:05:54.244 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.244 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.244 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:05:54.244 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:32.602455+0000", 2024-03-20T09:05:54.245 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.07843493 2024-03-20T09:05:54.245 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.245 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.245 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:54.245 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:32.616355+0000", 2024-03-20T09:05:54.245 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.013899570999999999 2024-03-20T09:05:54.245 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.245 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.245 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:54.245 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:32.620121+0000", 2024-03-20T09:05:54.245 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0037660480000000001 2024-03-20T09:05:54.245 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.245 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.245 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:05:54.246 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:32.624730+0000", 2024-03-20T09:05:54.246 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0046090740000000003 2024-03-20T09:05:54.246 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.246 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.246 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:54.246 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:32.645351+0000", 2024-03-20T09:05:54.246 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.020620772999999998 2024-03-20T09:05:54.246 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.246 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:54.246 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.246 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.246 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.246 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:110799 3.d 3:b7a5cb18:::benchmark_data_smithi122_119671_object11017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-03-20T09:05:54.246 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:31.047138+0000", 2024-03-20T09:05:54.247 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 23.139390653, 2024-03-20T09:05:54.247 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6280017459999998, 2024-03-20T09:05:54.247 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:54.247 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:05:54.247 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:54.247 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:54.247 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:54.247 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 110799 2024-03-20T09:05:54.247 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.247 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:54.247 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.247 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:54.247 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:31.047138+0000", 2024-03-20T09:05:54.247 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.248 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.248 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.248 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:54.248 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:31.047138+0000", 2024-03-20T09:05:54.248 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.248 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.248 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.248 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:54.248 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:31.047140+0000", 2024-03-20T09:05:54.248 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8199999999999999e-06 2024-03-20T09:05:54.248 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.248 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.248 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:54.248 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:31.047150+0000", 2024-03-20T09:05:54.249 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0013000000000001e-05 2024-03-20T09:05:54.249 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.249 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.249 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:54.249 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:31.047167+0000", 2024-03-20T09:05:54.249 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6782e-05 2024-03-20T09:05:54.249 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.249 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.249 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:54.249 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:31.047177+0000", 2024-03-20T09:05:54.249 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0304e-05 2024-03-20T09:05:54.249 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.249 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.249 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.249 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:31.268079+0000", 2024-03-20T09:05:54.250 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.220902352 2024-03-20T09:05:54.250 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.250 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.250 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:05:54.250 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:31.276859+0000", 2024-03-20T09:05:54.250 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0087801570000000002 2024-03-20T09:05:54.250 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.250 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.250 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.250 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:31.378461+0000", 2024-03-20T09:05:54.250 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.101601551 2024-03-20T09:05:54.250 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.250 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.251 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:05:54.251 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:31.387320+0000", 2024-03-20T09:05:54.251 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0088588550000000005 2024-03-20T09:05:54.251 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.251 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.251 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.251 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:33.516281+0000", 2024-03-20T09:05:54.251 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1289613030000001 2024-03-20T09:05:54.251 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.252 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.252 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:05:54.252 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:33.522830+0000", 2024-03-20T09:05:54.252 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0065488730000000002 2024-03-20T09:05:54.252 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.252 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.252 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:05:54.252 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:33.537323+0000", 2024-03-20T09:05:54.252 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.014493071 2024-03-20T09:05:54.252 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.252 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.252 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:05:54.252 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:33.594479+0000", 2024-03-20T09:05:54.253 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.057156351000000001 2024-03-20T09:05:54.253 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.253 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.253 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:54.253 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:33.646073+0000", 2024-03-20T09:05:54.253 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.051593542999999999 2024-03-20T09:05:54.253 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.254 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.254 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:54.254 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:33.649679+0000", 2024-03-20T09:05:54.254 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0036065569999999998 2024-03-20T09:05:54.254 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.254 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.254 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:05:54.254 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:33.655055+0000", 2024-03-20T09:05:54.254 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0053759410000000004 2024-03-20T09:05:54.255 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.255 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.255 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:54.255 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:33.675140+0000", 2024-03-20T09:05:54.255 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.020084273 2024-03-20T09:05:54.255 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.255 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:54.255 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.255 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.255 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.255 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:110802 3.d 3:b6d849bc:::benchmark_data_smithi122_119671_object11020:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-03-20T09:05:54.255 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:31.056672+0000", 2024-03-20T09:05:54.255 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 23.129856396000001, 2024-03-20T09:05:54.255 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8296267200000003, 2024-03-20T09:05:54.256 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:54.256 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:05:54.256 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:54.256 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:54.256 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:54.256 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 110802 2024-03-20T09:05:54.256 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.256 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:54.256 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.256 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:54.256 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:31.056672+0000", 2024-03-20T09:05:54.256 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.256 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.256 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.257 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:54.257 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:31.056672+0000", 2024-03-20T09:05:54.257 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.257 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.257 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.257 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:54.257 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:31.056674+0000", 2024-03-20T09:05:54.257 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8160000000000001e-06 2024-03-20T09:05:54.257 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.257 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.257 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:54.257 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:31.056680+0000", 2024-03-20T09:05:54.257 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1890000000000004e-06 2024-03-20T09:05:54.257 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.258 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.258 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:54.258 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:31.056695+0000", 2024-03-20T09:05:54.258 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4654000000000001e-05 2024-03-20T09:05:54.258 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.258 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.258 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:54.258 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:31.056704+0000", 2024-03-20T09:05:54.258 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.4360000000000008e-06 2024-03-20T09:05:54.258 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.258 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.258 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.258 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:31.277682+0000", 2024-03-20T09:05:54.258 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.22097818999999999 2024-03-20T09:05:54.259 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.259 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.259 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:05:54.259 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:31.287023+0000", 2024-03-20T09:05:54.259 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0093402250000000006 2024-03-20T09:05:54.259 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.259 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.259 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.259 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:31.422108+0000", 2024-03-20T09:05:54.259 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.135085177 2024-03-20T09:05:54.259 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.259 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.259 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:05:54.259 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:31.436656+0000", 2024-03-20T09:05:54.259 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.014548346 2024-03-20T09:05:54.260 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.260 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.260 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.260 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:33.701744+0000", 2024-03-20T09:05:54.260 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2650876430000002 2024-03-20T09:05:54.260 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.260 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.260 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:05:54.260 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:33.712981+0000", 2024-03-20T09:05:54.260 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.011237294 2024-03-20T09:05:54.260 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.260 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.260 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:05:54.260 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:33.728702+0000", 2024-03-20T09:05:54.261 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.015720359 2024-03-20T09:05:54.261 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.261 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.261 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:05:54.261 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:33.855306+0000", 2024-03-20T09:05:54.261 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.12660460500000001 2024-03-20T09:05:54.261 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.261 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.261 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:54.261 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:33.857732+0000", 2024-03-20T09:05:54.261 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0024262839999999999 2024-03-20T09:05:54.261 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.261 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.261 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:54.261 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:33.861286+0000", 2024-03-20T09:05:54.262 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.003553189 2024-03-20T09:05:54.262 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.262 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.262 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:05:54.262 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:33.866333+0000", 2024-03-20T09:05:54.262 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0050470970000000004 2024-03-20T09:05:54.262 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.262 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.262 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:54.262 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:33.886299+0000", 2024-03-20T09:05:54.262 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.019966215999999998 2024-03-20T09:05:54.262 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.262 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:54.262 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.263 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.263 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.263 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:111779 3.d 3:b1310f92:::benchmark_data_smithi122_119671_object11997:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-03-20T09:05:54.263 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:40.895193+0000", 2024-03-20T09:05:54.263 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 13.291335492, 2024-03-20T09:05:54.263 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.64063267, 2024-03-20T09:05:54.263 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:54.263 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:05:54.263 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:54.263 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:54.263 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:54.263 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 111779 2024-03-20T09:05:54.263 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.263 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:54.264 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.264 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:54.264 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:40.895193+0000", 2024-03-20T09:05:54.264 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.264 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.264 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.264 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:54.264 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:40.895193+0000", 2024-03-20T09:05:54.264 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.264 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.264 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.264 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:54.264 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:40.895195+0000", 2024-03-20T09:05:54.264 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.677e-06 2024-03-20T09:05:54.265 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.265 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.265 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:54.265 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:40.895203+0000", 2024-03-20T09:05:54.265 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.3680000000000002e-06 2024-03-20T09:05:54.265 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.265 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.265 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:54.265 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:40.895218+0000", 2024-03-20T09:05:54.265 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5301000000000001e-05 2024-03-20T09:05:54.265 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.265 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.265 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:54.265 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:40.895227+0000", 2024-03-20T09:05:54.265 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.7159999999999992e-06 2024-03-20T09:05:54.266 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.266 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.266 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.266 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:40.975067+0000", 2024-03-20T09:05:54.266 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.079840016999999999 2024-03-20T09:05:54.266 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.266 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.266 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:05:54.266 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:40.983614+0000", 2024-03-20T09:05:54.266 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0085472489999999998 2024-03-20T09:05:54.266 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.266 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.266 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.266 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.051468+0000", 2024-03-20T09:05:54.267 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.067853940000000001 2024-03-20T09:05:54.267 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.267 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.267 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:05:54.267 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.064377+0000", 2024-03-20T09:05:54.267 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.012908309 2024-03-20T09:05:54.267 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.267 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.267 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.267 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:43.535030+0000", 2024-03-20T09:05:54.267 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.4706534549999999 2024-03-20T09:05:54.267 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.267 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.268 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:54.268 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:43.535826+0000", 2024-03-20T09:05:54.268 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00079563799999999999 2024-03-20T09:05:54.268 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.268 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:54.268 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.268 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.268 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.268 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:111814 3.d 3:b8b10f2b:::benchmark_data_smithi122_119671_object12032:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-03-20T09:05:54.268 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:41.237764+0000", 2024-03-20T09:05:54.268 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 12.948764793, 2024-03-20T09:05:54.268 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6506554740000001, 2024-03-20T09:05:54.268 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:54.268 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:05:54.269 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:54.269 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:54.269 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:54.269 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 111814 2024-03-20T09:05:54.269 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.269 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:54.269 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.269 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:54.269 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.237764+0000", 2024-03-20T09:05:54.269 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.269 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.269 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.269 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:54.269 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.237764+0000", 2024-03-20T09:05:54.270 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.270 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.270 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.270 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:54.270 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.237765+0000", 2024-03-20T09:05:54.270 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5600000000000001e-06 2024-03-20T09:05:54.270 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.270 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.270 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:54.270 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.237773+0000", 2024-03-20T09:05:54.270 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3270000000000003e-06 2024-03-20T09:05:54.270 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.270 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.270 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:54.271 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.237785+0000", 2024-03-20T09:05:54.271 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2788e-05 2024-03-20T09:05:54.271 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.271 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.271 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:54.271 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.237794+0000", 2024-03-20T09:05:54.271 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5280000000000003e-06 2024-03-20T09:05:54.271 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.271 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.271 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.271 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.274715+0000", 2024-03-20T09:05:54.271 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.036921484999999997 2024-03-20T09:05:54.271 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.271 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.272 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:05:54.272 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.284730+0000", 2024-03-20T09:05:54.272 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.010014749999999999 2024-03-20T09:05:54.272 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.272 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.272 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.272 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.431400+0000", 2024-03-20T09:05:54.272 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.14666947899999999 2024-03-20T09:05:54.272 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.272 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.274 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:05:54.274 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.440122+0000", 2024-03-20T09:05:54.274 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0087218049999999991 2024-03-20T09:05:54.274 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.274 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.274 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.274 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:43.887067+0000", 2024-03-20T09:05:54.274 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.4469455490000001 2024-03-20T09:05:54.274 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.274 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.274 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:54.274 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:43.888419+0000", 2024-03-20T09:05:54.275 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001352203 2024-03-20T09:05:54.275 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.275 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:54.275 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.275 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.275 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.275 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:111829 3.d 3:bb84c761:::benchmark_data_smithi122_119671_object12047:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-03-20T09:05:54.275 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:41.307046+0000", 2024-03-20T09:05:54.275 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 12.879482471999999, 2024-03-20T09:05:54.275 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.769093705, 2024-03-20T09:05:54.275 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:54.275 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:05:54.275 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:54.275 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:54.275 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:54.276 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 111829 2024-03-20T09:05:54.276 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.276 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:54.276 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.276 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:54.276 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.307046+0000", 2024-03-20T09:05:54.276 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.276 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.276 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.276 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:54.276 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.307046+0000", 2024-03-20T09:05:54.276 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.276 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.276 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.277 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:54.277 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.307048+0000", 2024-03-20T09:05:54.277 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8109999999999999e-06 2024-03-20T09:05:54.277 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.277 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.277 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:54.277 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.307054+0000", 2024-03-20T09:05:54.277 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.5769999999999999e-06 2024-03-20T09:05:54.277 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.277 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.277 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:54.277 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.307067+0000", 2024-03-20T09:05:54.277 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2813000000000001e-05 2024-03-20T09:05:54.277 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.277 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.278 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:54.278 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.307076+0000", 2024-03-20T09:05:54.278 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.2719999999999997e-06 2024-03-20T09:05:54.278 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.278 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.278 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.278 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.478251+0000", 2024-03-20T09:05:54.278 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.17117516799999999 2024-03-20T09:05:54.278 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.278 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.278 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:05:54.278 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.490660+0000", 2024-03-20T09:05:54.278 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.012409369 2024-03-20T09:05:54.278 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.279 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.279 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.279 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.599037+0000", 2024-03-20T09:05:54.279 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.108377022 2024-03-20T09:05:54.279 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.279 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.279 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:05:54.279 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.614142+0000", 2024-03-20T09:05:54.279 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.015104988999999999 2024-03-20T09:05:54.279 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.279 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.279 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.279 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.075409+0000", 2024-03-20T09:05:54.279 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.461266755 2024-03-20T09:05:54.280 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.280 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.280 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:54.280 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.076140+0000", 2024-03-20T09:05:54.280 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00073092899999999995 2024-03-20T09:05:54.280 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.280 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:54.280 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.280 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.280 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.280 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:111883 3.d 3:b65c8180:::benchmark_data_smithi122_119671_object12101:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-03-20T09:05:54.280 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:41.800850+0000", 2024-03-20T09:05:54.280 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 12.385678724, 2024-03-20T09:05:54.280 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.692717933, 2024-03-20T09:05:54.280 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:54.281 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:05:54.281 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:54.281 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:54.281 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:54.281 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 111883 2024-03-20T09:05:54.281 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.281 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:54.281 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.281 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:54.281 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.800850+0000", 2024-03-20T09:05:54.281 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.281 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.281 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.281 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:54.282 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.800850+0000", 2024-03-20T09:05:54.282 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.282 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.282 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.282 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:54.282 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.800851+0000", 2024-03-20T09:05:54.282 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1179999999999999e-06 2024-03-20T09:05:54.282 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.282 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.282 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:54.282 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.800856+0000", 2024-03-20T09:05:54.282 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9429999999999999e-06 2024-03-20T09:05:54.282 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.282 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.282 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:54.283 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.800867+0000", 2024-03-20T09:05:54.283 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1127e-05 2024-03-20T09:05:54.283 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.283 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.283 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:54.283 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.800872+0000", 2024-03-20T09:05:54.283 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2549999999999997e-06 2024-03-20T09:05:54.283 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.283 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.283 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.283 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.005639+0000", 2024-03-20T09:05:54.283 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.20476703099999999 2024-03-20T09:05:54.283 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.283 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.284 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:05:54.284 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.015359+0000", 2024-03-20T09:05:54.284 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0097197659999999995 2024-03-20T09:05:54.284 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.284 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.284 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.284 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.076999+0000", 2024-03-20T09:05:54.284 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.061640201999999998 2024-03-20T09:05:54.284 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.284 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.284 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:05:54.284 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.087364+0000", 2024-03-20T09:05:54.284 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.010364485 2024-03-20T09:05:54.284 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.284 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.285 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.285 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.350121+0000", 2024-03-20T09:05:54.285 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2627572740000002 2024-03-20T09:05:54.285 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.285 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.285 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:05:54.285 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.361223+0000", 2024-03-20T09:05:54.285 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.011102272 2024-03-20T09:05:54.285 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.285 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.285 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:05:54.285 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.375549+0000", 2024-03-20T09:05:54.285 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.014325495000000001 2024-03-20T09:05:54.285 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.286 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.286 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:54.286 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.459028+0000", 2024-03-20T09:05:54.286 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.083479353000000006 2024-03-20T09:05:54.286 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.286 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.286 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:54.286 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.462162+0000", 2024-03-20T09:05:54.286 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.003133865 2024-03-20T09:05:54.286 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.286 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.286 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:05:54.286 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.470559+0000", 2024-03-20T09:05:54.286 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0083966969999999998 2024-03-20T09:05:54.287 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.287 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.287 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:05:54.287 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.474293+0000", 2024-03-20T09:05:54.287 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0037343250000000001 2024-03-20T09:05:54.287 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.287 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.287 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:54.287 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.493568+0000", 2024-03-20T09:05:54.287 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.019274724999999999 2024-03-20T09:05:54.287 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.287 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:54.287 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.287 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.287 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.288 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:111946 3.d 3:b7750090:::benchmark_data_smithi122_119671_object12164:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-03-20T09:05:54.288 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:42.199835+0000", 2024-03-20T09:05:54.288 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 11.986693218999999, 2024-03-20T09:05:54.288 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6486077139999997, 2024-03-20T09:05:54.288 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:54.288 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:05:54.288 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:54.288 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:54.288 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:54.288 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 111946 2024-03-20T09:05:54.288 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.288 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:54.288 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.288 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:54.289 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.199835+0000", 2024-03-20T09:05:54.289 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.289 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.289 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.289 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:54.289 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.199835+0000", 2024-03-20T09:05:54.289 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.289 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.289 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.289 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:54.289 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.199837+0000", 2024-03-20T09:05:54.289 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5680000000000001e-06 2024-03-20T09:05:54.289 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.289 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.289 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:54.290 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.199843+0000", 2024-03-20T09:05:54.290 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0399999999999998e-06 2024-03-20T09:05:54.290 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.290 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.290 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:54.290 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.199856+0000", 2024-03-20T09:05:54.290 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2880999999999999e-05 2024-03-20T09:05:54.290 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.290 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.290 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:54.290 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.199864+0000", 2024-03-20T09:05:54.290 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.3510000000000002e-06 2024-03-20T09:05:54.290 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.290 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.291 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.291 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.260727+0000", 2024-03-20T09:05:54.291 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.060863029999999999 2024-03-20T09:05:54.291 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.291 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.291 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:05:54.291 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.269017+0000", 2024-03-20T09:05:54.291 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0082901120000000005 2024-03-20T09:05:54.291 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.291 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.291 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.291 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.391398+0000", 2024-03-20T09:05:54.291 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.122380398 2024-03-20T09:05:54.291 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.291 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.292 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:05:54.292 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.402344+0000", 2024-03-20T09:05:54.292 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.010946193999999999 2024-03-20T09:05:54.292 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.292 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.292 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.292 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.750414+0000", 2024-03-20T09:05:54.292 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.348070517 2024-03-20T09:05:54.292 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.292 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.292 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:05:54.292 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.757236+0000", 2024-03-20T09:05:54.292 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0068216149999999996 2024-03-20T09:05:54.292 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.292 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.293 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:05:54.293 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.774062+0000", 2024-03-20T09:05:54.293 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.016825618000000001 2024-03-20T09:05:54.293 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.293 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.293 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:54.293 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.819616+0000", 2024-03-20T09:05:54.293 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.045554465000000002 2024-03-20T09:05:54.293 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.293 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.293 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:54.293 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.823981+0000", 2024-03-20T09:05:54.293 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0043652680000000003 2024-03-20T09:05:54.293 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.294 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.294 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:05:54.294 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.825526+0000", 2024-03-20T09:05:54.294 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0015445159999999999 2024-03-20T09:05:54.294 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.294 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.294 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:05:54.294 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.830369+0000", 2024-03-20T09:05:54.294 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.004843099 2024-03-20T09:05:54.294 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.294 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.294 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:54.294 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.848443+0000", 2024-03-20T09:05:54.294 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.018074041999999998 2024-03-20T09:05:54.294 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.295 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:54.295 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.295 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.295 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.295 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:111951 3.d 3:b592a88e:::benchmark_data_smithi122_119671_object12169:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-03-20T09:05:54.295 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:42.292147+0000", 2024-03-20T09:05:54.295 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 11.894381226, 2024-03-20T09:05:54.295 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6288111750000001, 2024-03-20T09:05:54.295 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:54.295 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:05:54.295 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:54.295 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:54.295 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:54.295 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 111951 2024-03-20T09:05:54.296 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.296 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:54.296 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.296 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:54.296 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.292147+0000", 2024-03-20T09:05:54.296 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.296 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.296 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.296 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:54.296 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.292147+0000", 2024-03-20T09:05:54.296 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.296 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.296 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.296 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:54.296 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.292149+0000", 2024-03-20T09:05:54.297 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.167e-06 2024-03-20T09:05:54.297 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.297 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.297 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:54.297 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.292154+0000", 2024-03-20T09:05:54.297 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7799999999999997e-06 2024-03-20T09:05:54.297 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.297 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.297 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:54.297 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.292167+0000", 2024-03-20T09:05:54.297 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2563e-05 2024-03-20T09:05:54.297 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.297 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.297 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:54.298 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.292174+0000", 2024-03-20T09:05:54.298 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.17e-06 2024-03-20T09:05:54.298 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.298 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.298 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.298 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.443195+0000", 2024-03-20T09:05:54.298 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.15102074500000001 2024-03-20T09:05:54.298 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.298 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.298 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:05:54.298 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.454219+0000", 2024-03-20T09:05:54.298 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.011023982999999999 2024-03-20T09:05:54.298 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.298 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.299 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.299 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.527524+0000", 2024-03-20T09:05:54.299 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.073305495999999998 2024-03-20T09:05:54.299 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.299 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.299 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:05:54.299 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.537965+0000", 2024-03-20T09:05:54.299 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.010440825000000001 2024-03-20T09:05:54.299 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.299 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.299 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.299 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.849818+0000", 2024-03-20T09:05:54.299 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3118533660000002 2024-03-20T09:05:54.299 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.300 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.300 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:05:54.300 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.857807+0000", 2024-03-20T09:05:54.300 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0079888770000000001 2024-03-20T09:05:54.300 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.300 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.300 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:05:54.300 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.870323+0000", 2024-03-20T09:05:54.300 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.012515703 2024-03-20T09:05:54.300 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.300 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.300 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:54.300 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.893190+0000", 2024-03-20T09:05:54.300 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.022867032999999998 2024-03-20T09:05:54.301 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.301 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.301 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:54.301 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.896903+0000", 2024-03-20T09:05:54.301 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0037132089999999999 2024-03-20T09:05:54.301 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.301 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.301 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:05:54.301 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.898348+0000", 2024-03-20T09:05:54.301 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001445078 2024-03-20T09:05:54.301 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.301 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.301 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:05:54.301 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.901907+0000", 2024-03-20T09:05:54.301 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0035584729999999999 2024-03-20T09:05:54.302 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.302 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.302 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:54.302 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.920959+0000", 2024-03-20T09:05:54.302 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.019051707000000001 2024-03-20T09:05:54.302 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.302 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:54.302 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.302 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.302 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.302 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:111955 3.d 3:b9ac9af0:::benchmark_data_smithi122_119671_object12173:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-03-20T09:05:54.302 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:42.486309+0000", 2024-03-20T09:05:54.302 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 11.700219238000001, 2024-03-20T09:05:54.302 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6305225640000001, 2024-03-20T09:05:54.303 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:54.303 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:05:54.303 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:54.303 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:54.303 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:54.303 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 111955 2024-03-20T09:05:54.303 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.303 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:54.303 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.303 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:54.303 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.486309+0000", 2024-03-20T09:05:54.303 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.303 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.303 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.304 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:54.304 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.486309+0000", 2024-03-20T09:05:54.304 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.304 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.304 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.304 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:54.304 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.486311+0000", 2024-03-20T09:05:54.304 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5030000000000001e-06 2024-03-20T09:05:54.304 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.304 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.304 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:54.304 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.486317+0000", 2024-03-20T09:05:54.304 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2079999999999997e-06 2024-03-20T09:05:54.304 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.305 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.305 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:54.305 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.486332+0000", 2024-03-20T09:05:54.305 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4756000000000001e-05 2024-03-20T09:05:54.305 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.305 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.305 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:54.305 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.486342+0000", 2024-03-20T09:05:54.305 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0685e-05 2024-03-20T09:05:54.305 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.305 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.305 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.305 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.538985+0000", 2024-03-20T09:05:54.305 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.052642964 2024-03-20T09:05:54.305 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.306 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.306 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:05:54.306 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.548113+0000", 2024-03-20T09:05:54.306 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0091271119999999997 2024-03-20T09:05:54.306 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.306 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.306 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.306 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.623669+0000", 2024-03-20T09:05:54.306 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.075556847999999996 2024-03-20T09:05:54.306 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.306 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.306 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:05:54.306 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.637242+0000", 2024-03-20T09:05:54.306 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.013572424 2024-03-20T09:05:54.307 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.307 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.307 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.307 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.999138+0000", 2024-03-20T09:05:54.307 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3618961220000001 2024-03-20T09:05:54.307 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.307 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.307 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:05:54.307 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.008896+0000", 2024-03-20T09:05:54.307 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.009758437 2024-03-20T09:05:54.307 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.307 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.307 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:05:54.307 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.024927+0000", 2024-03-20T09:05:54.308 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.016030141000000001 2024-03-20T09:05:54.308 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.308 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.308 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:05:54.308 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.082837+0000", 2024-03-20T09:05:54.308 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.057910573999999999 2024-03-20T09:05:54.308 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.308 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.308 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:54.308 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.085787+0000", 2024-03-20T09:05:54.308 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0029503540000000001 2024-03-20T09:05:54.308 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.308 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.308 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:54.308 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.089540+0000", 2024-03-20T09:05:54.309 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0037526370000000001 2024-03-20T09:05:54.309 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.309 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.309 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:05:54.309 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.096049+0000", 2024-03-20T09:05:54.309 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0065090720000000003 2024-03-20T09:05:54.309 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.309 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.309 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:54.309 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:45.116832+0000", 2024-03-20T09:05:54.309 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.020782727000000001 2024-03-20T09:05:54.309 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.309 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:54.309 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.309 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.310 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.310 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:112047 3.d 3:bd3b44cf:::benchmark_data_smithi122_119671_object12265:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-03-20T09:05:54.310 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:43.453333+0000", 2024-03-20T09:05:54.310 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 10.733195773, 2024-03-20T09:05:54.310 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6901377609999999, 2024-03-20T09:05:54.310 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:54.310 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:05:54.310 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:54.310 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:54.310 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:54.310 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 112047 2024-03-20T09:05:54.310 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.310 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:54.310 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.311 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:54.311 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:43.453333+0000", 2024-03-20T09:05:54.311 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.311 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.311 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.311 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:54.311 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:43.453333+0000", 2024-03-20T09:05:54.311 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.311 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.311 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.311 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:54.311 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:43.453334+0000", 2024-03-20T09:05:54.311 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.9199999999999999e-07 2024-03-20T09:05:54.312 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.312 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.312 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:54.312 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:43.453337+0000", 2024-03-20T09:05:54.312 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.687e-06 2024-03-20T09:05:54.312 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.312 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.312 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:54.312 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:43.453345+0000", 2024-03-20T09:05:54.312 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7740000000000004e-06 2024-03-20T09:05:54.312 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.312 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.312 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:54.312 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:43.453350+0000", 2024-03-20T09:05:54.312 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.7350000000000004e-06 2024-03-20T09:05:54.313 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.313 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.313 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.313 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:43.661327+0000", 2024-03-20T09:05:54.313 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.20797734300000001 2024-03-20T09:05:54.313 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.313 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.313 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:05:54.313 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:43.675055+0000", 2024-03-20T09:05:54.313 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.013727698 2024-03-20T09:05:54.313 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.313 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.313 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.313 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:43.756153+0000", 2024-03-20T09:05:54.314 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.081097797999999999 2024-03-20T09:05:54.314 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.314 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.314 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:05:54.314 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:43.765280+0000", 2024-03-20T09:05:54.314 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0091267640000000008 2024-03-20T09:05:54.314 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.314 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.314 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.314 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.066723+0000", 2024-03-20T09:05:54.314 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3014431910000002 2024-03-20T09:05:54.314 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.314 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.314 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:05:54.314 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.074394+0000", 2024-03-20T09:05:54.315 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0076713559999999998 2024-03-20T09:05:54.315 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.315 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.315 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:05:54.315 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.092087+0000", 2024-03-20T09:05:54.315 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.017692504000000001 2024-03-20T09:05:54.315 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.315 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.315 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:54.315 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.116812+0000", 2024-03-20T09:05:54.315 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.024725047999999999 2024-03-20T09:05:54.315 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.315 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.315 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:05:54.316 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.118715+0000", 2024-03-20T09:05:54.316 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0019038830000000001 2024-03-20T09:05:54.316 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.316 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.316 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:54.316 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.121559+0000", 2024-03-20T09:05:54.316 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0028439020000000001 2024-03-20T09:05:54.316 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.316 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.316 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:05:54.316 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.126915+0000", 2024-03-20T09:05:54.316 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0053557270000000002 2024-03-20T09:05:54.316 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.316 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.316 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:54.317 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.143471+0000", 2024-03-20T09:05:54.317 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.016555459000000002 2024-03-20T09:05:54.317 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.317 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:54.317 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.317 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.317 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.317 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:112392 3.d 3:be9f5588:::benchmark_data_smithi122_119671_object12610:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-03-20T09:05:54.317 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:47.628268+0000", 2024-03-20T09:05:54.317 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 6.5582605249999997, 2024-03-20T09:05:54.317 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6413982219999999, 2024-03-20T09:05:54.317 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:54.317 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:05:54.317 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:54.318 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:54.318 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:54.318 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 112392 2024-03-20T09:05:54.318 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.318 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:54.318 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.318 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:54.318 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.628268+0000", 2024-03-20T09:05:54.318 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.318 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.318 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.318 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:54.318 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.628268+0000", 2024-03-20T09:05:54.318 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.319 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.319 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.319 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:54.319 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.628270+0000", 2024-03-20T09:05:54.319 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.858e-06 2024-03-20T09:05:54.319 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.319 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.319 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:54.319 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.628278+0000", 2024-03-20T09:05:54.319 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7219999999999999e-06 2024-03-20T09:05:54.319 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.319 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.319 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:54.319 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.628293+0000", 2024-03-20T09:05:54.319 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4911e-05 2024-03-20T09:05:54.320 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.320 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.320 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:54.320 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.628302+0000", 2024-03-20T09:05:54.320 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.9979999999999992e-06 2024-03-20T09:05:54.320 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.320 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.320 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.320 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.695436+0000", 2024-03-20T09:05:54.320 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.067134073000000002 2024-03-20T09:05:54.320 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.320 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.320 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:05:54.320 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.711757+0000", 2024-03-20T09:05:54.321 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.016321473 2024-03-20T09:05:54.321 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.321 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.321 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.321 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.838436+0000", 2024-03-20T09:05:54.321 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.12667895700000001 2024-03-20T09:05:54.321 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.321 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.321 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:05:54.321 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.848598+0000", 2024-03-20T09:05:54.321 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.010161554999999999 2024-03-20T09:05:54.321 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.321 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.321 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.321 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.157747+0000", 2024-03-20T09:05:54.322 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.309149385 2024-03-20T09:05:54.322 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.322 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.322 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:05:54.322 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.164763+0000", 2024-03-20T09:05:54.322 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0070162970000000003 2024-03-20T09:05:54.322 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.322 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.322 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:05:54.322 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.176845+0000", 2024-03-20T09:05:54.322 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.012082147999999999 2024-03-20T09:05:54.322 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.322 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.322 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:54.323 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.240665+0000", 2024-03-20T09:05:54.323 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.063819186 2024-03-20T09:05:54.323 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.323 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.323 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:54.323 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.244165+0000", 2024-03-20T09:05:54.323 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0035001580000000002 2024-03-20T09:05:54.323 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.323 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.323 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:05:54.323 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.245837+0000", 2024-03-20T09:05:54.323 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0016719969999999999 2024-03-20T09:05:54.323 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.323 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.323 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:05:54.324 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.250134+0000", 2024-03-20T09:05:54.324 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0042968040000000004 2024-03-20T09:05:54.324 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.324 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.324 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:54.324 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.269666+0000", 2024-03-20T09:05:54.324 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0195327 2024-03-20T09:05:54.324 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.324 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:54.324 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.324 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.324 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.324 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:112641 3.d 3:b5458b0b:::benchmark_data_smithi122_119671_object12859:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-03-20T09:05:54.324 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:49.935175+0000", 2024-03-20T09:05:54.325 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 4.2513537809999997, 2024-03-20T09:05:54.325 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6200009839999998, 2024-03-20T09:05:54.325 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:54.325 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:05:54.325 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:54.325 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:54.325 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:54.325 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 112641 2024-03-20T09:05:54.325 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.325 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:54.325 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.325 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:54.325 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:49.935175+0000", 2024-03-20T09:05:54.325 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.325 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.326 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.326 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:54.326 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:49.935175+0000", 2024-03-20T09:05:54.326 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.326 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.326 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.326 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:54.326 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:49.935176+0000", 2024-03-20T09:05:54.326 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.291e-06 2024-03-20T09:05:54.326 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.326 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.326 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:54.326 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:49.935191+0000", 2024-03-20T09:05:54.326 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4771e-05 2024-03-20T09:05:54.327 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.327 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.327 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:54.327 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:49.935203+0000", 2024-03-20T09:05:54.327 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1925e-05 2024-03-20T09:05:54.327 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.327 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.327 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:54.327 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:49.935211+0000", 2024-03-20T09:05:54.327 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.6230000000000003e-06 2024-03-20T09:05:54.327 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.327 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.327 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.327 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:49.997196+0000", 2024-03-20T09:05:54.328 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.061984730000000002 2024-03-20T09:05:54.328 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.328 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.328 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:05:54.328 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.005537+0000", 2024-03-20T09:05:54.328 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0083406179999999993 2024-03-20T09:05:54.328 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.328 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.328 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.328 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.128230+0000", 2024-03-20T09:05:54.328 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.122693413 2024-03-20T09:05:54.328 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.328 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.328 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:05:54.328 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.136799+0000", 2024-03-20T09:05:54.329 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00856903 2024-03-20T09:05:54.329 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.329 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.329 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.329 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:52.413852+0000", 2024-03-20T09:05:54.329 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2770525780000002 2024-03-20T09:05:54.329 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.329 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.329 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:05:54.329 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:52.425672+0000", 2024-03-20T09:05:54.329 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.011820567000000001 2024-03-20T09:05:54.329 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.329 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.329 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:05:54.330 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:52.441836+0000", 2024-03-20T09:05:54.330 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.016164125000000001 2024-03-20T09:05:54.330 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.330 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.330 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:54.330 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:52.519222+0000", 2024-03-20T09:05:54.330 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.077385307 2024-03-20T09:05:54.330 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.330 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.330 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:54.330 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:52.522523+0000", 2024-03-20T09:05:54.330 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0033015779999999999 2024-03-20T09:05:54.330 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.330 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.330 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:05:54.331 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:52.528863+0000", 2024-03-20T09:05:54.331 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0063395639999999998 2024-03-20T09:05:54.331 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.331 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.331 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:05:54.331 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:52.533690+0000", 2024-03-20T09:05:54.331 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.004826777 2024-03-20T09:05:54.331 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.331 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.331 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:54.331 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:52.555176+0000", 2024-03-20T09:05:54.331 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.021486087000000001 2024-03-20T09:05:54.331 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.331 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:54.332 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.332 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.332 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.332 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:112702 3.d 3:b07597e1:::benchmark_data_smithi122_119671_object12920:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-03-20T09:05:54.332 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:50.657764+0000", 2024-03-20T09:05:54.332 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 3.5287649399999999, 2024-03-20T09:05:54.332 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.7225253120000001, 2024-03-20T09:05:54.332 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:05:54.332 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:05:54.332 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:05:54.332 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:05:54.332 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:05:54.332 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 112702 2024-03-20T09:05:54.332 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.332 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:05:54.333 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.333 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:05:54.333 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.657764+0000", 2024-03-20T09:05:54.333 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.333 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.333 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.333 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:05:54.333 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.657764+0000", 2024-03-20T09:05:54.333 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:05:54.333 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.333 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.333 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:05:54.333 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.657765+0000", 2024-03-20T09:05:54.333 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7060000000000001e-06 2024-03-20T09:05:54.334 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.334 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.334 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:05:54.334 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.657774+0000", 2024-03-20T09:05:54.334 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.7099999999999996e-06 2024-03-20T09:05:54.334 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.334 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.334 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:05:54.334 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.657788+0000", 2024-03-20T09:05:54.334 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4114000000000001e-05 2024-03-20T09:05:54.334 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.334 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.334 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:05:54.334 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.657798+0000", 2024-03-20T09:05:54.334 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.4790000000000002e-06 2024-03-20T09:05:54.335 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.335 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.335 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.335 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.817529+0000", 2024-03-20T09:05:54.335 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.159731015 2024-03-20T09:05:54.335 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.335 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.335 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:05:54.335 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.825842+0000", 2024-03-20T09:05:54.335 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0083134569999999998 2024-03-20T09:05:54.335 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.335 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.335 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.335 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.888052+0000", 2024-03-20T09:05:54.336 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.062210069999999999 2024-03-20T09:05:54.336 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.336 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.336 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:05:54.336 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.897510+0000", 2024-03-20T09:05:54.336 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.009457366 2024-03-20T09:05:54.336 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.336 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.336 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:05:54.336 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:53.244083+0000", 2024-03-20T09:05:54.336 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3465735749999999 2024-03-20T09:05:54.336 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.336 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.336 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:05:54.336 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:53.251105+0000", 2024-03-20T09:05:54.337 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0070215369999999996 2024-03-20T09:05:54.337 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.337 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.337 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:05:54.337 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:53.263052+0000", 2024-03-20T09:05:54.337 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.011947573 2024-03-20T09:05:54.337 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.337 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.337 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:54.337 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:53.276761+0000", 2024-03-20T09:05:54.337 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.013708299 2024-03-20T09:05:54.337 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.337 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.337 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:05:54.337 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:53.279758+0000", 2024-03-20T09:05:54.338 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0029975029999999999 2024-03-20T09:05:54.338 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.338 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.338 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:05:54.338 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:53.359751+0000", 2024-03-20T09:05:54.338 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.079993274000000003 2024-03-20T09:05:54.338 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.338 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.338 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:05:54.338 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:53.364270+0000", 2024-03-20T09:05:54.338 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.004518762 2024-03-20T09:05:54.338 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:05:54.338 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:05:54.338 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:05:54.339 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:53.380289+0000", 2024-03-20T09:05:54.339 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.016018872 2024-03-20T09:05:54.339 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.339 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:54.339 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.339 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:05:54.339 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:05:54.339 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:05:54.340 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:53 smithi122 ceph-mon[73495]: pgmap v782: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 27 active+clean, 6 active+undersized; 5.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 95 op/s; 26617/260562 objects degraded (10.215%); 21799/260562 objects misplaced (8.366%); 11 MiB/s, 179 objects/s recovering 2024-03-20T09:05:54.963 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-20T09:05:55.268 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:05:55.315 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:05:55.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:55.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:05:55.420 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:05:55.451 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:05:55.457 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:55.614 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:05:55.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:55.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:05:55.721 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:05:55.725 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:05:55.725 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:05:55.725 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:05:55.725 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:05:55.726 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:05:55.726 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:05:55.730 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:05:55.750 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:55.753 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:05:55.781 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:05:55.800 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:05:55.820 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:05:55.821 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:05:55.821 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:05:55.821 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:05:55.821 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:05:55.821 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:05:55.821 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-20T09:05:55.821 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:05:55.821 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:05:55.821 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:05:55.822 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:05:55.822 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:05:55.822 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:05:55.846 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:55 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/834617645' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:05:55.884 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:55 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/834617645' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:05:55.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:55 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/834617645' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:05:56.101 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:05:56.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:56.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:05:56.205 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:05:56.235 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:56.359 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:05:56.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:56.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:05:56.375 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:05:56.462 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:05:56.492 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:56.871 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:56 smithi026 ceph-mon[68309]: pgmap v783: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 27 active+clean, 6 active+undersized; 5.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 109 op/s; 26548/259524 objects degraded (10.229%); 20835/259524 objects misplaced (8.028%); 14 MiB/s, 230 objects/s recovering 2024-03-20T09:05:56.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:56 smithi115 ceph-mon[72476]: pgmap v783: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 27 active+clean, 6 active+undersized; 5.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 109 op/s; 26548/259524 objects degraded (10.229%); 20835/259524 objects misplaced (8.028%); 14 MiB/s, 230 objects/s recovering 2024-03-20T09:05:56.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:56 smithi122 ceph-mon[73495]: pgmap v783: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 27 active+clean, 6 active+undersized; 5.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 109 op/s; 26548/259524 objects degraded (10.229%); 20835/259524 objects misplaced (8.028%); 14 MiB/s, 230 objects/s recovering 2024-03-20T09:05:57.172 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:05:57.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:57.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:05:57.276 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:05:57.307 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:57.375 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:05:57.375 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:05:57.380 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:05:57.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:57.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:05:57.484 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:05:57.514 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:57.588 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:05:57.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:57.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:05:57.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:57 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 26548/259524 objects degraded (10.229%), 12 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-03-20T09:05:57.692 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:05:57.720 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:57.740 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:05:57.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:57.748 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:05:57.850 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:05:57.879 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:57.898 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:05:57.900 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:57 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 26548/259524 objects degraded (10.229%), 12 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-03-20T09:05:57.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:57.901 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:05:57.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:57 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 26548/259524 objects degraded (10.229%), 12 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-03-20T09:05:58.002 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:05:58.031 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:58.090 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-20T09:05:58.368 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:05:58.368 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:05:58.403 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:05:58.581 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:05:58.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:58.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:05:58.684 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:05:58.711 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:58.730 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:05:58.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:58.733 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:05:58.834 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:05:58.861 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:58.880 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:05:58.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:58.883 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:05:58.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:58 smithi115 ceph-mon[72476]: pgmap v784: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 27 active+clean, 6 active+undersized; 5.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 87 op/s; 26548/259377 objects degraded (10.235%); 20835/259377 objects misplaced (8.033%); 10 MiB/s, 159 objects/s recovering 2024-03-20T09:05:58.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:58 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:05:58.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:58 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:05:58.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:58 smithi122 ceph-mon[73495]: pgmap v784: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 27 active+clean, 6 active+undersized; 5.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 87 op/s; 26548/259377 objects degraded (10.235%); 20835/259377 objects misplaced (8.033%); 10 MiB/s, 159 objects/s recovering 2024-03-20T09:05:58.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:58 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:05:58.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:58 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:05:58.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:58 smithi026 ceph-mon[68309]: pgmap v784: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 27 active+clean, 6 active+undersized; 5.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 87 op/s; 26548/259377 objects degraded (10.235%); 20835/259377 objects misplaced (8.033%); 10 MiB/s, 159 objects/s recovering 2024-03-20T09:05:58.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:58 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:05:58.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:58 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:05:58.984 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:05:59.011 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:59.034 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:05:59.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:59.037 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:05:59.138 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:05:59.165 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:59.186 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:05:59.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:59.188 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:05:59.289 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:05:59.323 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:59.344 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:05:59.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:59.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:05:59.447 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:05:59.477 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:59.575 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:05:59.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:59.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:05:59.640 INFO:teuthology.orchestra.run.smithi026.stdout:moved 22 / 141 (15.6028%) 2024-03-20T09:05:59.641 INFO:teuthology.orchestra.run.smithi026.stdout:avg 15.6667 2024-03-20T09:05:59.641 INFO:teuthology.orchestra.run.smithi026.stdout:stddev 5.53106 -> 4.28607 (expected baseline 3.73175) 2024-03-20T09:05:59.641 INFO:teuthology.orchestra.run.smithi026.stdout:min osd.0 with 6 -> 16 pgs (0.382979 -> 1.02128 * mean) 2024-03-20T09:05:59.641 INFO:teuthology.orchestra.run.smithi026.stdout:max osd.4 with 21 -> 21 pgs (1.34043 -> 1.34043 * mean) 2024-03-20T09:05:59.641 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:05:59.641 INFO:teuthology.orchestra.run.smithi026.stdout:oload 110 2024-03-20T09:05:59.641 INFO:teuthology.orchestra.run.smithi026.stdout:max_change 3 2024-03-20T09:05:59.641 INFO:teuthology.orchestra.run.smithi026.stdout:max_change_osds 4 2024-03-20T09:05:59.641 INFO:teuthology.orchestra.run.smithi026.stdout:average_utilization 0.0190 2024-03-20T09:05:59.641 INFO:teuthology.orchestra.run.smithi026.stdout:overload_utilization 0.0209 2024-03-20T09:05:59.641 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0 weight 0.4982 -> 1.0000 2024-03-20T09:05:59.642 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7 weight 1.0000 -> 0.6396 2024-03-20T09:05:59.642 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11 weight 1.0000 -> 0.7565 2024-03-20T09:05:59.642 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8 weight 0.6007 -> 0.9606 2024-03-20T09:05:59.680 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:05:59.710 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:05:59.814 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:59 smithi115 ceph-mon[72476]: osdmap e304: 12 total, 11 up, 10 in 2024-03-20T09:05:59.814 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:05:59 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'65536','7':'41914','8':'62957','11':'49579'}"}]: dispatch 2024-03-20T09:05:59.910 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:05:59.911 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:59 smithi026 ceph-mon[68309]: osdmap e304: 12 total, 11 up, 10 in 2024-03-20T09:05:59.911 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:05:59 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'65536','7':'41914','8':'62957','11':'49579'}"}]: dispatch 2024-03-20T09:05:59.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:05:59.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:05:59.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:59 smithi122 ceph-mon[73495]: osdmap e304: 12 total, 11 up, 10 in 2024-03-20T09:05:59.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:05:59 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'65536','7':'41914','8':'62957','11':'49579'}"}]: dispatch 2024-03-20T09:06:00.014 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:06:00.041 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:00.059 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:06:00.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:00.062 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:06:00.163 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:06:00.196 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:00.219 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:06:00.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:00.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:06:00.322 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:06:00.349 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:00.368 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:06:00.369 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:06:00.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:00.371 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:06:00.471 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:06:00.473 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-03-20T09:06:00.473 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight-by-utilization 1000 0.05 2024-03-20T09:06:00.497 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:00.499 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:06:00.499 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:06:00.516 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:06:00.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:00.519 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:06:00.619 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:06:00.652 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:00 smithi122 ceph-mon[73495]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:06:00.652 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:00 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'65536','7':'41914','8':'62957','11':'49579'}"}]': finished 2024-03-20T09:06:00.652 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:00 smithi122 ceph-mon[73495]: osdmap e305: 12 total, 11 up, 10 in 2024-03-20T09:06:00.652 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:00 smithi122 ceph-mon[73495]: pgmap v787: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 27 active+clean, 6 active+undersized; 5.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 113 op/s; 26436/258423 objects degraded (10.230%); 20153/258423 objects misplaced (7.798%); 12 MiB/s, 198 objects/s recovering 2024-03-20T09:06:00.652 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:00 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/281507056' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:06:00.652 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:00 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:06:00.652 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:00 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:06:00.652 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:00 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:06:00.658 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:00.659 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:06:00.659 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:06:00.785 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:06:00.788 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:00 smithi115 ceph-mon[72476]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:06:00.788 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:00 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'65536','7':'41914','8':'62957','11':'49579'}"}]': finished 2024-03-20T09:06:00.788 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:00 smithi115 ceph-mon[72476]: osdmap e305: 12 total, 11 up, 10 in 2024-03-20T09:06:00.788 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:00 smithi115 ceph-mon[72476]: pgmap v787: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 27 active+clean, 6 active+undersized; 5.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 113 op/s; 26436/258423 objects degraded (10.230%); 20153/258423 objects misplaced (7.798%); 12 MiB/s, 198 objects/s recovering 2024-03-20T09:06:00.788 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:00 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/281507056' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:06:00.788 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:00 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:06:00.788 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:00 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:06:00.788 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:00 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:06:00.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:00.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:06:00.890 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:06:00.917 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:00 smithi026 ceph-mon[68309]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:06:00.917 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:00 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'65536','7':'41914','8':'62957','11':'49579'}"}]': finished 2024-03-20T09:06:00.917 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:00 smithi026 ceph-mon[68309]: osdmap e305: 12 total, 11 up, 10 in 2024-03-20T09:06:00.917 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:00 smithi026 ceph-mon[68309]: pgmap v787: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+degraded+remapped, 27 active+clean, 6 active+undersized; 5.3 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 113 op/s; 26436/258423 objects degraded (10.230%); 20153/258423 objects misplaced (7.798%); 12 MiB/s, 198 objects/s recovering 2024-03-20T09:06:00.917 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:00 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/281507056' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:06:00.917 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:00 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:06:00.917 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:00 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:06:00.918 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:00 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:06:00.920 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:00.940 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:06:00.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:00.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:06:01.044 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:06:01.071 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:01.091 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:06:01.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:01.094 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:06:01.195 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:06:01.224 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:01.300 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-20T09:06:01.639 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:06:01.653 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:01 smithi115 ceph-mon[72476]: osdmap e306: 12 total, 11 up, 10 in 2024-03-20T09:06:01.653 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:01 smithi115 ceph-mon[72476]: osdmap e307: 12 total, 11 up, 10 in 2024-03-20T09:06:01.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:01 smithi122 ceph-mon[73495]: osdmap e306: 12 total, 11 up, 10 in 2024-03-20T09:06:01.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:01 smithi122 ceph-mon[73495]: osdmap e307: 12 total, 11 up, 10 in 2024-03-20T09:06:01.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:01 smithi026 ceph-mon[68309]: osdmap e306: 12 total, 11 up, 10 in 2024-03-20T09:06:01.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:01 smithi026 ceph-mon[68309]: osdmap e307: 12 total, 11 up, 10 in 2024-03-20T09:06:02.179 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:06:02.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:02.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:06:02.282 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:06:02.323 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:02.343 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:06:02.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:02.347 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:06:02.448 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:06:02.482 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:02.582 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:06:02.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:02.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:06:02.661 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:02 smithi122 ceph-mon[73495]: 3.a starting backfill to osd.8 from (0'0,237'3876] MIN to 304'7159 2024-03-20T09:06:02.661 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:02 smithi122 ceph-mon[73495]: pgmap v790: 47 pgs: 1 unknown, 2 remapped+peering, 2 peering, 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+undersized+degraded, 1 active+recovering+degraded+remapped, 24 active+clean, 6 active+undersized; 5.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 114 op/s; 23677/258006 objects degraded (9.177%); 17064/258006 objects misplaced (6.614%); 16 MiB/s, 259 objects/s recovering 2024-03-20T09:06:02.661 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:02 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 26436/258423 objects degraded (10.230%), 12 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-03-20T09:06:02.661 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:02 smithi122 ceph-mon[73495]: osdmap e308: 12 total, 11 up, 10 in 2024-03-20T09:06:02.690 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:06:02.695 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:06:02.729 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:02.743 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:02 smithi115 ceph-mon[72476]: 3.a starting backfill to osd.8 from (0'0,237'3876] MIN to 304'7159 2024-03-20T09:06:02.743 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:02 smithi115 ceph-mon[72476]: pgmap v790: 47 pgs: 1 unknown, 2 remapped+peering, 2 peering, 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+undersized+degraded, 1 active+recovering+degraded+remapped, 24 active+clean, 6 active+undersized; 5.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 114 op/s; 23677/258006 objects degraded (9.177%); 17064/258006 objects misplaced (6.614%); 16 MiB/s, 259 objects/s recovering 2024-03-20T09:06:02.743 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:02 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 26436/258423 objects degraded (10.230%), 12 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-03-20T09:06:02.743 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:02 smithi115 ceph-mon[72476]: osdmap e308: 12 total, 11 up, 10 in 2024-03-20T09:06:02.952 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:02 smithi026 ceph-mon[68309]: 3.a starting backfill to osd.8 from (0'0,237'3876] MIN to 304'7159 2024-03-20T09:06:02.953 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:02 smithi026 ceph-mon[68309]: pgmap v790: 47 pgs: 1 unknown, 2 remapped+peering, 2 peering, 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+undersized+degraded, 1 active+recovering+degraded+remapped, 24 active+clean, 6 active+undersized; 5.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 114 op/s; 23677/258006 objects degraded (9.177%); 17064/258006 objects misplaced (6.614%); 16 MiB/s, 259 objects/s recovering 2024-03-20T09:06:02.953 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:02 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 26436/258423 objects degraded (10.230%), 12 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-03-20T09:06:02.953 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:02 smithi026 ceph-mon[68309]: osdmap e308: 12 total, 11 up, 10 in 2024-03-20T09:06:03.810 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:06:03.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:03.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:06:03.818 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:06:03.818 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:06:03.838 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:06:03.861 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:06:03.881 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:06:03.904 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:06:03.915 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:06:03.934 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:06:03.961 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:03.966 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:06:03.996 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:06:04.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:03 smithi115 ceph-mon[72476]: osdmap e309: 12 total, 11 up, 10 in 2024-03-20T09:06:04.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:03 smithi115 ceph-mon[72476]: pgmap v793: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 unknown, 3 remapped+peering, 4 peering, 1 active+clean+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+undersized+degraded, 1 active+recovering+degraded+remapped, 21 active+clean, 5 active+undersized; 5.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 76 op/s; 25035/257508 objects degraded (9.722%); 22788/257508 objects misplaced (8.849%); 27 MiB/s, 424 objects/s recovering 2024-03-20T09:06:04.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:03 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/315371897' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:06:04.183 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:06:04.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:03 smithi122 ceph-mon[73495]: osdmap e309: 12 total, 11 up, 10 in 2024-03-20T09:06:04.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:03 smithi122 ceph-mon[73495]: pgmap v793: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 unknown, 3 remapped+peering, 4 peering, 1 active+clean+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+undersized+degraded, 1 active+recovering+degraded+remapped, 21 active+clean, 5 active+undersized; 5.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 76 op/s; 25035/257508 objects degraded (9.722%); 22788/257508 objects misplaced (8.849%); 27 MiB/s, 424 objects/s recovering 2024-03-20T09:06:04.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:03 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/315371897' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:06:04.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:03 smithi026 ceph-mon[68309]: osdmap e309: 12 total, 11 up, 10 in 2024-03-20T09:06:04.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:03 smithi026 ceph-mon[68309]: pgmap v793: 47 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 unknown, 3 remapped+peering, 4 peering, 1 active+clean+remapped, 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+undersized+degraded, 1 active+recovering+degraded+remapped, 21 active+clean, 5 active+undersized; 5.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 76 op/s; 25035/257508 objects degraded (9.722%); 22788/257508 objects misplaced (8.849%); 27 MiB/s, 424 objects/s recovering 2024-03-20T09:06:04.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:03 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/315371897' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:06:04.222 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:06:04.250 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:06:04.274 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:06:04.303 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:06:04.303 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:06:04.303 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:06:04.303 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:06:04.303 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:06:04.303 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:06:04.303 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-20T09:06:04.303 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:06:04.303 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:06:04.303 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:06:04.304 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:06:04.304 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:06:04.304 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:06:04.315 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:06:04.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:04.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:06:04.421 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:06:04.456 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:04.534 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:06:04.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:04.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:06:04.597 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_ops_in_flight 2024-03-20T09:06:04.639 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:06:04.789 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:04.823 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:06:04.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:04.839 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:06:04.939 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:06:04.970 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:05.018 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:06:05.046 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:06:05.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:05.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:06:05.080 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:06:05.080 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:06:05.152 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:06:05.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:04 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:06:05.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:04 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:06:05.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:04 smithi122 ceph-mon[73495]: osdmap e310: 12 total, 11 up, 10 in 2024-03-20T09:06:05.192 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:05.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:04 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:06:05.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:04 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:06:05.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:04 smithi026 ceph-mon[68309]: osdmap e310: 12 total, 11 up, 10 in 2024-03-20T09:06:05.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:04 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:06:05.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:04 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:06:05.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:04 smithi115 ceph-mon[72476]: osdmap e310: 12 total, 11 up, 10 in 2024-03-20T09:06:05.534 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:06:05.931 INFO:teuthology.orchestra.run.smithi026.stdout:moved 3 / 141 (2.12766%) 2024-03-20T09:06:05.931 INFO:teuthology.orchestra.run.smithi026.stdout:avg 15.6667 2024-03-20T09:06:05.932 INFO:teuthology.orchestra.run.smithi026.stdout:stddev 4.28607 -> 4.61078 (expected baseline 3.73175) 2024-03-20T09:06:05.932 INFO:teuthology.orchestra.run.smithi026.stdout:min osd.7 with 7 -> 6 pgs (0.446808 -> 0.382979 * mean) 2024-03-20T09:06:05.932 INFO:teuthology.orchestra.run.smithi026.stdout:max osd.4 with 21 -> 21 pgs (1.34043 -> 1.34043 * mean) 2024-03-20T09:06:05.932 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:06:05.932 INFO:teuthology.orchestra.run.smithi026.stdout:oload 1000 2024-03-20T09:06:05.932 INFO:teuthology.orchestra.run.smithi026.stdout:max_change 0.05 2024-03-20T09:06:05.932 INFO:teuthology.orchestra.run.smithi026.stdout:max_change_osds 4 2024-03-20T09:06:05.932 INFO:teuthology.orchestra.run.smithi026.stdout:average_utilization 0.0190 2024-03-20T09:06:05.933 INFO:teuthology.orchestra.run.smithi026.stdout:overload_utilization 0.1904 2024-03-20T09:06:05.933 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8 weight 0.9606 -> 1.0000 2024-03-20T09:06:05.933 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1 weight 0.8575 -> 0.9075 2024-03-20T09:06:06.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:05 smithi122 ceph-mon[73495]: pgmap v795: 47 pgs: 1 activating+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 2 remapped+peering, 2 peering, 1 active+clean+remapped, 1 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 20 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 5.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 99 op/s; 33902/256815 objects degraded (13.201%); 33489/256815 objects misplaced (13.040%); 30 MiB/s, 472 objects/s recovering 2024-03-20T09:06:06.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:05 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'59472','8':'65536'}"}]: dispatch 2024-03-20T09:06:06.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:05 smithi026 ceph-mon[68309]: pgmap v795: 47 pgs: 1 activating+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 2 remapped+peering, 2 peering, 1 active+clean+remapped, 1 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 20 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 5.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 99 op/s; 33902/256815 objects degraded (13.201%); 33489/256815 objects misplaced (13.040%); 30 MiB/s, 472 objects/s recovering 2024-03-20T09:06:06.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:05 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'59472','8':'65536'}"}]: dispatch 2024-03-20T09:06:06.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:05 smithi115 ceph-mon[72476]: pgmap v795: 47 pgs: 1 activating+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 2 remapped+peering, 2 peering, 1 active+clean+remapped, 1 active+remapped+backfill_wait, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 20 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 5.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 99 op/s; 33902/256815 objects degraded (13.201%); 33489/256815 objects misplaced (13.040%); 30 MiB/s, 472 objects/s recovering 2024-03-20T09:06:06.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:05 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'59472','8':'65536'}"}]: dispatch 2024-03-20T09:06:07.082 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:06:07.114 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:06 smithi026 ceph-mon[68309]: from='client.25171 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:06:07.115 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:06 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'59472','8':'65536'}"}]': finished 2024-03-20T09:06:07.115 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:06 smithi026 ceph-mon[68309]: osdmap e311: 12 total, 11 up, 10 in 2024-03-20T09:06:07.115 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:06 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 33902/256815 objects degraded (13.201%), 14 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:06:07.115 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:06 smithi026 ceph-mon[68309]: osdmap e312: 12 total, 11 up, 10 in 2024-03-20T09:06:07.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:06 smithi122 ceph-mon[73495]: from='client.25171 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:06:07.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:06 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'59472','8':'65536'}"}]': finished 2024-03-20T09:06:07.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:06 smithi122 ceph-mon[73495]: osdmap e311: 12 total, 11 up, 10 in 2024-03-20T09:06:07.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:06 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 33902/256815 objects degraded (13.201%), 14 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:06:07.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:06 smithi122 ceph-mon[73495]: osdmap e312: 12 total, 11 up, 10 in 2024-03-20T09:06:07.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:06 smithi115 ceph-mon[72476]: from='client.25171 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:06:07.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:06 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'59472','8':'65536'}"}]': finished 2024-03-20T09:06:07.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:06 smithi115 ceph-mon[72476]: osdmap e311: 12 total, 11 up, 10 in 2024-03-20T09:06:07.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:06 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 33902/256815 objects degraded (13.201%), 14 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:06:07.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:06 smithi115 ceph-mon[72476]: osdmap e312: 12 total, 11 up, 10 in 2024-03-20T09:06:07.558 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:06:07.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:07.561 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:06:07.655 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-03-20T09:06:07.656 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd test-reweight-by-utilization 1000 3.0 2024-03-20T09:06:07.662 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:06:07.730 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:07.880 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:06:07.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:07.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:06:07.986 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:06:08.019 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:08.169 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:06:08.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:07 smithi026 ceph-mon[68309]: pgmap v798: 47 pgs: 1 activating+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 22 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 5.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 114 op/s; 34959/256296 objects degraded (13.640%); 46143/256296 objects misplaced (18.004%); 1004/85432 objects unfound (1.175%); 34 MiB/s, 545 objects/s recovering 2024-03-20T09:06:08.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:07 smithi026 ceph-mon[68309]: osdmap e313: 12 total, 11 up, 10 in 2024-03-20T09:06:08.226 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:06:08.374 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:06:08.377 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:07 smithi122 ceph-mon[73495]: pgmap v798: 47 pgs: 1 activating+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 22 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 5.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 114 op/s; 34959/256296 objects degraded (13.640%); 46143/256296 objects misplaced (18.004%); 1004/85432 objects unfound (1.175%); 34 MiB/s, 545 objects/s recovering 2024-03-20T09:06:08.377 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:07 smithi122 ceph-mon[73495]: osdmap e313: 12 total, 11 up, 10 in 2024-03-20T09:06:08.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:08.379 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:06:08.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:07 smithi115 ceph-mon[72476]: pgmap v798: 47 pgs: 1 activating+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 2 active+clean+remapped, 1 active+remapped+backfill_wait, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 22 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 5.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 114 op/s; 34959/256296 objects degraded (13.640%); 46143/256296 objects misplaced (18.004%); 1004/85432 objects unfound (1.175%); 34 MiB/s, 545 objects/s recovering 2024-03-20T09:06:08.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:07 smithi115 ceph-mon[72476]: osdmap e313: 12 total, 11 up, 10 in 2024-03-20T09:06:08.479 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:06:08.510 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:08.932 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:06:08.966 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:06:08.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:08.972 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:06:09.072 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:06:09.105 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:08 smithi026 ceph-mon[68309]: Health check failed: 1004/85432 objects unfound (1.175%) (OBJECT_UNFOUND) 2024-03-20T09:06:09.105 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:08 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2762080527' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:06:09.105 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:08 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:06:09.105 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:08 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:06:09.105 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:08 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:06:09.105 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:08 smithi026 ceph-mon[68309]: osdmap e314: 12 total, 11 up, 10 in 2024-03-20T09:06:09.108 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:09.135 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:06:09.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:09.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:06:09.241 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:06:09.270 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:08 smithi122 ceph-mon[73495]: Health check failed: 1004/85432 objects unfound (1.175%) (OBJECT_UNFOUND) 2024-03-20T09:06:09.270 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:08 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2762080527' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:06:09.270 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:08 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:06:09.270 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:08 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:06:09.270 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:08 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:06:09.270 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:08 smithi122 ceph-mon[73495]: osdmap e314: 12 total, 11 up, 10 in 2024-03-20T09:06:09.273 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:09.300 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:06:09.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:09.307 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:06:09.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:08 smithi115 ceph-mon[72476]: Health check failed: 1004/85432 objects unfound (1.175%) (OBJECT_UNFOUND) 2024-03-20T09:06:09.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:08 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2762080527' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:06:09.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:08 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:06:09.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:08 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:06:09.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:08 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:06:09.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:08 smithi115 ceph-mon[72476]: osdmap e314: 12 total, 11 up, 10 in 2024-03-20T09:06:09.408 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:06:09.441 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:09.643 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:06:09.644 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:06:09.644 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:06:09.644 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-03-20T09:06:09.644 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:06:09.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:09.647 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:06:09.748 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:06:09.785 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:10.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:09 smithi026 ceph-mon[68309]: pgmap v801: 47 pgs: 1 remapped+peering, 1 activating+undersized+remapped, 2 activating+undersized+degraded+remapped, 2 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 21 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 5.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 74 op/s; 39259/255921 objects degraded (15.340%); 37856/255921 objects misplaced (14.792%); 1004/85307 objects unfound (1.177%); 42 MiB/s, 669 objects/s recovering 2024-03-20T09:06:10.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:09 smithi115 ceph-mon[72476]: pgmap v801: 47 pgs: 1 remapped+peering, 1 activating+undersized+remapped, 2 activating+undersized+degraded+remapped, 2 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 21 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 5.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 74 op/s; 39259/255921 objects degraded (15.340%); 37856/255921 objects misplaced (14.792%); 1004/85307 objects unfound (1.177%); 42 MiB/s, 669 objects/s recovering 2024-03-20T09:06:10.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:09 smithi122 ceph-mon[73495]: pgmap v801: 47 pgs: 1 remapped+peering, 1 activating+undersized+remapped, 2 activating+undersized+degraded+remapped, 2 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 21 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 5.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 74 op/s; 39259/255921 objects degraded (15.340%); 37856/255921 objects misplaced (14.792%); 1004/85307 objects unfound (1.177%); 42 MiB/s, 669 objects/s recovering 2024-03-20T09:06:10.476 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_blocked_ops 2024-03-20T09:06:10.832 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:06:11.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:11 smithi026 ceph-mon[68309]: pgmap v802: 47 pgs: 1 peering, 1 remapped+peering, 2 activating+undersized+degraded+remapped, 2 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 21 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 5.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 56 op/s; 39256/255837 objects degraded (15.344%); 36727/255837 objects misplaced (14.356%); 1004/85279 objects unfound (1.177%); 42 MiB/s, 665 objects/s recovering 2024-03-20T09:06:12.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:11 smithi115 ceph-mon[72476]: pgmap v802: 47 pgs: 1 peering, 1 remapped+peering, 2 activating+undersized+degraded+remapped, 2 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 21 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 5.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 56 op/s; 39256/255837 objects degraded (15.344%); 36727/255837 objects misplaced (14.356%); 1004/85279 objects unfound (1.177%); 42 MiB/s, 665 objects/s recovering 2024-03-20T09:06:12.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:11 smithi122 ceph-mon[73495]: pgmap v802: 47 pgs: 1 peering, 1 remapped+peering, 2 activating+undersized+degraded+remapped, 2 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 21 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 5.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 56 op/s; 39256/255837 objects degraded (15.344%); 36727/255837 objects misplaced (14.356%); 1004/85279 objects unfound (1.177%); 42 MiB/s, 665 objects/s recovering 2024-03-20T09:06:12.947 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:06:12.947 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:06:12.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:12 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/210933561' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:06:12.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:12 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 39256/255837 objects degraded (15.344%), 15 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:06:12.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:12 smithi122 ceph-mon[73495]: osdmap e315: 12 total, 11 up, 10 in 2024-03-20T09:06:12.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:12 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/210933561' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:06:12.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:12 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 39256/255837 objects degraded (15.344%), 15 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:06:12.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:12 smithi026 ceph-mon[68309]: osdmap e315: 12 total, 11 up, 10 in 2024-03-20T09:06:13.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:12 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/210933561' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:06:13.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:12 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 39256/255837 objects degraded (15.344%), 15 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:06:13.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:12 smithi115 ceph-mon[72476]: osdmap e315: 12 total, 11 up, 10 in 2024-03-20T09:06:13.688 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_historic_ops 2024-03-20T09:06:13.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:13 smithi122 ceph-mon[73495]: pgmap v804: 47 pgs: 1 active+undersized+degraded+remapped, 1 peering, 3 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 21 active+clean, 3 active+recovering+undersized+remapped, 5 active+undersized; 5.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 52 op/s; 32895/255354 objects degraded (12.882%); 45068/255354 objects misplaced (17.649%); 53 MiB/s, 847 objects/s recovering 2024-03-20T09:06:14.063 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:06:14.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:13 smithi115 ceph-mon[72476]: pgmap v804: 47 pgs: 1 active+undersized+degraded+remapped, 1 peering, 3 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 21 active+clean, 3 active+recovering+undersized+remapped, 5 active+undersized; 5.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 52 op/s; 32895/255354 objects degraded (12.882%); 45068/255354 objects misplaced (17.649%); 53 MiB/s, 847 objects/s recovering 2024-03-20T09:06:14.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:13 smithi026 ceph-mon[68309]: pgmap v804: 47 pgs: 1 active+undersized+degraded+remapped, 1 peering, 3 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 21 active+clean, 3 active+recovering+undersized+remapped, 5 active+undersized; 5.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 52 op/s; 32895/255354 objects degraded (12.882%); 45068/255354 objects misplaced (17.649%); 53 MiB/s, 847 objects/s recovering 2024-03-20T09:06:14.442 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:06:14.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:14.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:06:14.540 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:06:14.541 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:06:14.541 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:06:14.541 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:06:14.541 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:06:14.541 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:06:14.541 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:06:14.541 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:06:14.541 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:06:14.541 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:06:14.541 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:06:14.541 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:06:14.541 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:06:14.542 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:06:14.542 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:06:14.542 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:06:14.542 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-20T09:06:14.542 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:06:14.542 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:06:14.542 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:06:14.542 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:06:14.542 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:06:14.542 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:06:14.554 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:06:14.592 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:15.124 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:14 smithi122 ceph-mon[73495]: Health check cleared: OBJECT_UNFOUND (was: 1004/85279 objects unfound (1.177%)) 2024-03-20T09:06:15.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:14 smithi115 ceph-mon[72476]: Health check cleared: OBJECT_UNFOUND (was: 1004/85279 objects unfound (1.177%)) 2024-03-20T09:06:15.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:14 smithi026 ceph-mon[68309]: Health check cleared: OBJECT_UNFOUND (was: 1004/85279 objects unfound (1.177%)) 2024-03-20T09:06:15.276 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:06:15.277 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:06:15.546 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:06:15.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:15.556 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:06:15.658 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:06:15.690 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:15.794 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:06:15.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:15.798 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:06:15.899 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:06:15.933 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:15.962 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:06:15.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:15.965 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:06:15.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:15 smithi026 ceph-mon[68309]: pgmap v805: 47 pgs: 1 active+undersized+degraded+remapped, 3 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 22 active+clean, 3 active+recovering+undersized+remapped, 5 active+undersized; 5.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 75 op/s; 32845/254544 objects degraded (12.903%); 43783/254544 objects misplaced (17.201%); 51 MiB/s, 823 objects/s recovering 2024-03-20T09:06:16.066 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:06:16.096 INFO:teuthology.orchestra.run.smithi026.stdout:moved 0 / 141 (0%) 2024-03-20T09:06:16.096 INFO:teuthology.orchestra.run.smithi026.stdout:avg 15.6667 2024-03-20T09:06:16.096 INFO:teuthology.orchestra.run.smithi026.stdout:stddev 4.61078 -> 4.61078 (expected baseline 3.73175) 2024-03-20T09:06:16.096 INFO:teuthology.orchestra.run.smithi026.stdout:min osd.7 with 6 -> 6 pgs (0.382979 -> 0.382979 * mean) 2024-03-20T09:06:16.096 INFO:teuthology.orchestra.run.smithi026.stdout:max osd.4 with 21 -> 21 pgs (1.34043 -> 1.34043 * mean) 2024-03-20T09:06:16.096 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:06:16.096 INFO:teuthology.orchestra.run.smithi026.stdout:oload 1000 2024-03-20T09:06:16.096 INFO:teuthology.orchestra.run.smithi026.stdout:max_change 3 2024-03-20T09:06:16.096 INFO:teuthology.orchestra.run.smithi026.stdout:max_change_osds 4 2024-03-20T09:06:16.097 INFO:teuthology.orchestra.run.smithi026.stdout:average_utilization 0.0196 2024-03-20T09:06:16.097 INFO:teuthology.orchestra.run.smithi026.stdout:overload_utilization 0.1957 2024-03-20T09:06:16.097 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1 weight 0.9075 -> 1.0000 2024-03-20T09:06:16.100 INFO:teuthology.orchestra.run.smithi026.stderr:no change 2024-03-20T09:06:16.109 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:16.135 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:06:16.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:16.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:06:16.148 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:06:16.148 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:06:16.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:15 smithi115 ceph-mon[72476]: pgmap v805: 47 pgs: 1 active+undersized+degraded+remapped, 3 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 22 active+clean, 3 active+recovering+undersized+remapped, 5 active+undersized; 5.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 75 op/s; 32845/254544 objects degraded (12.903%); 43783/254544 objects misplaced (17.201%); 51 MiB/s, 823 objects/s recovering 2024-03-20T09:06:16.181 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:15 smithi122 ceph-mon[73495]: pgmap v805: 47 pgs: 1 active+undersized+degraded+remapped, 3 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 22 active+clean, 3 active+recovering+undersized+remapped, 5 active+undersized; 5.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 75 op/s; 32845/254544 objects degraded (12.903%); 43783/254544 objects misplaced (17.201%); 51 MiB/s, 823 objects/s recovering 2024-03-20T09:06:16.240 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:06:16.278 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:16.300 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:06:16.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:16.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:06:16.406 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:06:16.436 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:16.458 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:06:16.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:16.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:06:16.462 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 09:06:16 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T09:06:16.094+0000 7fe5fc38b700 -1 mgr.server reply reply (1) Operation not permitted no change 2024-03-20T09:06:16.488 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:06:16.562 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:06:16.605 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:16.633 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:06:16.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:16.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:06:16.740 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:06:16.777 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:16.858 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:06:16.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:16.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:06:16.893 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-03-20T09:06:16.893 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd test-reweight-by-utilization 1000 3.0 2024-03-20T09:06:16.896 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:16 smithi122 ceph-mon[73495]: from='client.25208 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:06:16.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:16 smithi115 ceph-mon[72476]: from='client.25208 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:06:16.936 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_ops_in_flight 2024-03-20T09:06:16.937 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:06:16.963 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:06:16.989 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:16 smithi026 ceph-mon[68309]: from='client.25208 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:06:17.018 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:17.184 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:06:17.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:17.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:06:17.278 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:06:17.293 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:06:17.318 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:06:17.323 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:17.350 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:06:17.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:17.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:06:17.455 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:06:17.501 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:17.764 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:06:17.765 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:06:17.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:17.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:06:17.871 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:06:17.885 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:17 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 32845/254544 objects degraded (12.903%), 14 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:06:17.886 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:17 smithi122 ceph-mon[73495]: pgmap v806: 47 pgs: 3 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 22 active+clean, 3 active+recovering+undersized+remapped, 5 active+undersized; 5.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 99 op/s; 32845/253302 objects degraded (12.967%); 42465/253302 objects misplaced (16.765%); 38 MiB/s, 605 objects/s recovering 2024-03-20T09:06:17.902 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:18.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:17 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 32845/254544 objects degraded (12.903%), 14 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:06:18.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:17 smithi115 ceph-mon[72476]: pgmap v806: 47 pgs: 3 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 22 active+clean, 3 active+recovering+undersized+remapped, 5 active+undersized; 5.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 99 op/s; 32845/253302 objects degraded (12.967%); 42465/253302 objects misplaced (16.765%); 38 MiB/s, 605 objects/s recovering 2024-03-20T09:06:18.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:17 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 32845/254544 objects degraded (12.903%), 14 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:06:18.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:17 smithi026 ceph-mon[68309]: pgmap v806: 47 pgs: 3 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 22 active+clean, 3 active+recovering+undersized+remapped, 5 active+undersized; 5.2 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 99 op/s; 32845/253302 objects degraded (12.967%); 42465/253302 objects misplaced (16.765%); 38 MiB/s, 605 objects/s recovering 2024-03-20T09:06:18.755 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:06:18.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:18.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:06:18.859 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:06:18.891 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:19.088 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:18 smithi122 ceph-mon[73495]: osdmap e316: 12 total, 11 up, 10 in 2024-03-20T09:06:19.092 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:06:19.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:19.095 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:06:19.196 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:06:19.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:18 smithi026 ceph-mon[68309]: osdmap e316: 12 total, 11 up, 10 in 2024-03-20T09:06:19.231 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:19.389 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:06:19.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:18 smithi115 ceph-mon[72476]: osdmap e316: 12 total, 11 up, 10 in 2024-03-20T09:06:19.423 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:06:19.423 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:06:19.943 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:19 smithi026 ceph-mon[68309]: osdmap e317: 12 total, 11 up, 10 in 2024-03-20T09:06:19.944 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:19 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1697056852' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:06:19.944 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:19 smithi026 ceph-mon[68309]: pgmap v809: 47 pgs: 3 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 22 active+clean, 3 active+recovering+undersized+remapped, 5 active+undersized; 5.1 GiB data, 18 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 148 op/s; 32682/252363 objects degraded (12.950%); 41384/252363 objects misplaced (16.399%); 34 MiB/s, 538 objects/s recovering 2024-03-20T09:06:19.949 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:06:20.115 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:19 smithi122 ceph-mon[73495]: osdmap e317: 12 total, 11 up, 10 in 2024-03-20T09:06:20.115 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:19 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1697056852' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:06:20.115 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:19 smithi122 ceph-mon[73495]: pgmap v809: 47 pgs: 3 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 22 active+clean, 3 active+recovering+undersized+remapped, 5 active+undersized; 5.1 GiB data, 18 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 148 op/s; 32682/252363 objects degraded (12.950%); 41384/252363 objects misplaced (16.399%); 34 MiB/s, 538 objects/s recovering 2024-03-20T09:06:20.153 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_blocked_ops 2024-03-20T09:06:20.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:19 smithi115 ceph-mon[72476]: osdmap e317: 12 total, 11 up, 10 in 2024-03-20T09:06:20.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:19 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1697056852' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:06:20.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:19 smithi115 ceph-mon[72476]: pgmap v809: 47 pgs: 3 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 22 active+clean, 3 active+recovering+undersized+remapped, 5 active+undersized; 5.1 GiB data, 18 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 148 op/s; 32682/252363 objects degraded (12.950%); 41384/252363 objects misplaced (16.399%); 34 MiB/s, 538 objects/s recovering 2024-03-20T09:06:20.491 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:06:20.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:20 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:06:20.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:20 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1697056852' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:06:20.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:20 smithi026 ceph-mon[68309]: osdmap e318: 12 total, 11 up, 10 in 2024-03-20T09:06:21.086 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:20 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:06:21.086 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:20 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1697056852' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:06:21.086 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:20 smithi122 ceph-mon[73495]: osdmap e318: 12 total, 11 up, 10 in 2024-03-20T09:06:21.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:20 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:06:21.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:20 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1697056852' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:06:21.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:20 smithi115 ceph-mon[72476]: osdmap e318: 12 total, 11 up, 10 in 2024-03-20T09:06:21.440 INFO:teuthology.orchestra.run.smithi026.stdout:moved 0 / 141 (0%) 2024-03-20T09:06:21.440 INFO:teuthology.orchestra.run.smithi026.stdout:avg 15.6667 2024-03-20T09:06:21.440 INFO:teuthology.orchestra.run.smithi026.stdout:stddev 4.61078 -> 4.61078 (expected baseline 3.73175) 2024-03-20T09:06:21.441 INFO:teuthology.orchestra.run.smithi026.stdout:min osd.7 with 6 -> 6 pgs (0.382979 -> 0.382979 * mean) 2024-03-20T09:06:21.441 INFO:teuthology.orchestra.run.smithi026.stdout:max osd.4 with 21 -> 21 pgs (1.34043 -> 1.34043 * mean) 2024-03-20T09:06:21.441 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:06:21.441 INFO:teuthology.orchestra.run.smithi026.stdout:oload 1000 2024-03-20T09:06:21.441 INFO:teuthology.orchestra.run.smithi026.stdout:max_change 3 2024-03-20T09:06:21.441 INFO:teuthology.orchestra.run.smithi026.stdout:max_change_osds 4 2024-03-20T09:06:21.441 INFO:teuthology.orchestra.run.smithi026.stdout:average_utilization 0.0196 2024-03-20T09:06:21.441 INFO:teuthology.orchestra.run.smithi026.stdout:overload_utilization 0.1964 2024-03-20T09:06:21.441 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1 weight 0.9075 -> 0.9970 2024-03-20T09:06:21.441 INFO:teuthology.orchestra.run.smithi026.stderr:no change 2024-03-20T09:06:21.720 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 09:06:21 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T09:06:21.296+0000 7fe5fc38b700 -1 mgr.server reply reply (1) Operation not permitted no change 2024-03-20T09:06:22.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:21 smithi122 ceph-mon[73495]: from='client.15315 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:06:22.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:21 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 32682/252363 objects degraded (12.950%), 14 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:06:22.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:21 smithi122 ceph-mon[73495]: osdmap e319: 12 total, 11 up, 10 in 2024-03-20T09:06:22.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:21 smithi026 ceph-mon[68309]: from='client.15315 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:06:22.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:21 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 32682/252363 objects degraded (12.950%), 14 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:06:22.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:21 smithi026 ceph-mon[68309]: osdmap e319: 12 total, 11 up, 10 in 2024-03-20T09:06:22.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:21 smithi115 ceph-mon[72476]: from='client.15315 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:06:22.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:21 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 32682/252363 objects degraded (12.950%), 14 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:06:22.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:21 smithi115 ceph-mon[72476]: osdmap e319: 12 total, 11 up, 10 in 2024-03-20T09:06:22.486 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:06:22.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:22.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:06:22.592 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:06:22.603 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:06:22.604 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:06:22.626 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:22.653 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:06:22.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:22.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:06:22.759 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:06:22.792 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:22.994 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:06:23.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:23.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:06:23.005 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-03-20T09:06:23.006 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight-by-utilization 110 3.0 2024-03-20T09:06:23.077 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:22 smithi026 ceph-mon[68309]: pgmap v811: 47 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 22 active+clean, 3 active+recovering+undersized+remapped, 5 active+undersized; 5.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 167 op/s; 29939/251523 objects degraded (11.903%); 43314/251523 objects misplaced (17.221%); 28 MiB/s, 514 objects/s recovering 2024-03-20T09:06:23.101 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:06:23.138 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:23.223 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:22 smithi122 ceph-mon[73495]: pgmap v811: 47 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 22 active+clean, 3 active+recovering+undersized+remapped, 5 active+undersized; 5.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 167 op/s; 29939/251523 objects degraded (11.903%); 43314/251523 objects misplaced (17.221%); 28 MiB/s, 514 objects/s recovering 2024-03-20T09:06:23.244 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:06:23.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:23.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:06:23.349 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:06:23.350 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:06:23.381 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:23.401 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:06:23.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:23.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:06:23.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:22 smithi115 ceph-mon[72476]: pgmap v811: 47 pgs: 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 22 active+clean, 3 active+recovering+undersized+remapped, 5 active+undersized; 5.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 167 op/s; 29939/251523 objects degraded (11.903%); 43314/251523 objects misplaced (17.221%); 28 MiB/s, 514 objects/s recovering 2024-03-20T09:06:23.505 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:06:23.543 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:23.564 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:06:23.576 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:06:23.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:23.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:06:23.603 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:06:23.619 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:06:23.629 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:06:23.654 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:06:23.675 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:06:23.681 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:06:23.704 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:06:23.715 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:23.725 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:06:23.747 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:06:23.769 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:06:23.793 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:06:23.816 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:06:23.816 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:06:23.816 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:06:23.816 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:06:23.816 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:06:23.816 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:06:23.816 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-20T09:06:23.816 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:06:23.817 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:06:23.817 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:06:23.817 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:06:23.817 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:06:23.817 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:06:24.202 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:23 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1053865548' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:06:24.202 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:23 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:06:24.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:23 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1053865548' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:06:24.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:23 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:06:24.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:23 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1053865548' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:06:24.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:23 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:06:24.965 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:06:25.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:24 smithi026 ceph-mon[68309]: pgmap v813: 47 pgs: 4 active+clean+remapped, 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 22 active+clean, 3 active+recovering+undersized+remapped, 5 active+undersized; 5.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 164 op/s; 29857/250482 objects degraded (11.920%); 40417/250482 objects misplaced (16.136%); 45 MiB/s, 867 objects/s recovering 2024-03-20T09:06:25.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:24 smithi026 ceph-mon[68309]: osdmap e320: 12 total, 11 up, 10 in 2024-03-20T09:06:25.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:24 smithi026 ceph-mon[68309]: 1.14 scrub starts 2024-03-20T09:06:25.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:24 smithi026 ceph-mon[68309]: 1.14 scrub ok 2024-03-20T09:06:25.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:24 smithi115 ceph-mon[72476]: pgmap v813: 47 pgs: 4 active+clean+remapped, 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 22 active+clean, 3 active+recovering+undersized+remapped, 5 active+undersized; 5.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 164 op/s; 29857/250482 objects degraded (11.920%); 40417/250482 objects misplaced (16.136%); 45 MiB/s, 867 objects/s recovering 2024-03-20T09:06:25.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:24 smithi115 ceph-mon[72476]: osdmap e320: 12 total, 11 up, 10 in 2024-03-20T09:06:25.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:24 smithi115 ceph-mon[72476]: 1.14 scrub starts 2024-03-20T09:06:25.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:24 smithi115 ceph-mon[72476]: 1.14 scrub ok 2024-03-20T09:06:25.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:24 smithi122 ceph-mon[73495]: pgmap v813: 47 pgs: 4 active+clean+remapped, 1 active+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 22 active+clean, 3 active+recovering+undersized+remapped, 5 active+undersized; 5.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 164 op/s; 29857/250482 objects degraded (11.920%); 40417/250482 objects misplaced (16.136%); 45 MiB/s, 867 objects/s recovering 2024-03-20T09:06:25.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:24 smithi122 ceph-mon[73495]: osdmap e320: 12 total, 11 up, 10 in 2024-03-20T09:06:25.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:24 smithi122 ceph-mon[73495]: 1.14 scrub starts 2024-03-20T09:06:25.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:24 smithi122 ceph-mon[73495]: 1.14 scrub ok 2024-03-20T09:06:25.502 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:06:25.503 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:06:26.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:25 smithi026 ceph-mon[68309]: osdmap e321: 12 total, 11 up, 10 in 2024-03-20T09:06:26.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:25 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:06:26.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:25 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:06:26.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:25 smithi115 ceph-mon[72476]: osdmap e321: 12 total, 11 up, 10 in 2024-03-20T09:06:26.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:25 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:06:26.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:25 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:06:26.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:25 smithi122 ceph-mon[73495]: osdmap e321: 12 total, 11 up, 10 in 2024-03-20T09:06:26.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:25 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:06:26.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:25 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:06:26.925 INFO:teuthology.orchestra.run.smithi026.stdout:moved 2 / 141 (1.41844%) 2024-03-20T09:06:26.926 INFO:teuthology.orchestra.run.smithi026.stdout:avg 15.6667 2024-03-20T09:06:26.926 INFO:teuthology.orchestra.run.smithi026.stdout:stddev 4.61078 -> 4.51335 (expected baseline 3.73175) 2024-03-20T09:06:26.926 INFO:teuthology.orchestra.run.smithi026.stdout:min osd.7 with 6 -> 6 pgs (0.382979 -> 0.382979 * mean) 2024-03-20T09:06:26.926 INFO:teuthology.orchestra.run.smithi026.stdout:max osd.4 with 21 -> 21 pgs (1.34043 -> 1.34043 * mean) 2024-03-20T09:06:26.926 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:06:26.926 INFO:teuthology.orchestra.run.smithi026.stdout:oload 110 2024-03-20T09:06:26.927 INFO:teuthology.orchestra.run.smithi026.stdout:max_change 3 2024-03-20T09:06:26.927 INFO:teuthology.orchestra.run.smithi026.stdout:max_change_osds 4 2024-03-20T09:06:26.927 INFO:teuthology.orchestra.run.smithi026.stdout:average_utilization 0.0198 2024-03-20T09:06:26.927 INFO:teuthology.orchestra.run.smithi026.stdout:overload_utilization 0.0217 2024-03-20T09:06:26.927 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7 weight 0.6396 -> 0.4265 2024-03-20T09:06:26.927 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11 weight 0.7565 -> 0.6123 2024-03-20T09:06:26.927 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6 weight 1.0000 -> 0.8424 2024-03-20T09:06:26.928 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1 weight 0.9075 -> 0.9710 2024-03-20T09:06:27.085 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_historic_ops 2024-03-20T09:06:27.126 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:06:27.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:27.129 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:06:27.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:26 smithi026 ceph-mon[68309]: pgmap v816: 47 pgs: 1 creating+peering, 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 22 active+clean, 3 active+recovering+undersized+remapped, 5 active+undersized; 5.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 113 op/s; 29830/249960 objects degraded (11.934%); 40073/249960 objects misplaced (16.032%); 32 MiB/s, 593 objects/s recovering 2024-03-20T09:06:27.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:26 smithi026 ceph-mon[68309]: osdmap e322: 12 total, 11 up, 10 in 2024-03-20T09:06:27.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:26 smithi026 ceph-mon[68309]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:06:27.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:26 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'63636','6':'55206','7':'27949','11':'40128'}"}]: dispatch 2024-03-20T09:06:27.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:26 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 29830/249960 objects degraded (11.934%), 12 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:06:27.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:26 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'63636','6':'55206','7':'27949','11':'40128'}"}]': finished 2024-03-20T09:06:27.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:26 smithi026 ceph-mon[68309]: osdmap e323: 12 total, 11 up, 10 in 2024-03-20T09:06:27.229 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:06:27.255 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:26 smithi115 ceph-mon[72476]: pgmap v816: 47 pgs: 1 creating+peering, 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 22 active+clean, 3 active+recovering+undersized+remapped, 5 active+undersized; 5.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 113 op/s; 29830/249960 objects degraded (11.934%); 40073/249960 objects misplaced (16.032%); 32 MiB/s, 593 objects/s recovering 2024-03-20T09:06:27.255 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:26 smithi115 ceph-mon[72476]: osdmap e322: 12 total, 11 up, 10 in 2024-03-20T09:06:27.255 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:26 smithi115 ceph-mon[72476]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:06:27.256 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:26 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'63636','6':'55206','7':'27949','11':'40128'}"}]: dispatch 2024-03-20T09:06:27.256 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:26 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 29830/249960 objects degraded (11.934%), 12 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:06:27.256 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:26 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'63636','6':'55206','7':'27949','11':'40128'}"}]': finished 2024-03-20T09:06:27.256 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:26 smithi115 ceph-mon[72476]: osdmap e323: 12 total, 11 up, 10 in 2024-03-20T09:06:27.258 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:27.280 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:06:27.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:27.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:06:27.384 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:06:27.415 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:27.437 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:06:27.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:27.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:06:27.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:26 smithi122 ceph-mon[73495]: pgmap v816: 47 pgs: 1 creating+peering, 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 22 active+clean, 3 active+recovering+undersized+remapped, 5 active+undersized; 5.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 113 op/s; 29830/249960 objects degraded (11.934%); 40073/249960 objects misplaced (16.032%); 32 MiB/s, 593 objects/s recovering 2024-03-20T09:06:27.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:26 smithi122 ceph-mon[73495]: osdmap e322: 12 total, 11 up, 10 in 2024-03-20T09:06:27.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:26 smithi122 ceph-mon[73495]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:06:27.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'63636','6':'55206','7':'27949','11':'40128'}"}]: dispatch 2024-03-20T09:06:27.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:26 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 29830/249960 objects degraded (11.934%), 12 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:06:27.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:26 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'63636','6':'55206','7':'27949','11':'40128'}"}]': finished 2024-03-20T09:06:27.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:26 smithi122 ceph-mon[73495]: osdmap e323: 12 total, 11 up, 10 in 2024-03-20T09:06:27.505 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:06:27.541 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:06:27.579 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:27.686 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:06:27.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:27.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:06:27.790 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:06:27.821 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:28.621 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:06:28.938 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:06:29.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:28 smithi122 ceph-mon[73495]: pgmap v819: 47 pgs: 1 creating+peering, 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 22 active+clean, 3 active+recovering+undersized+remapped, 5 active+undersized; 5.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 81 op/s; 29830/249507 objects degraded (11.956%); 38018/249507 objects misplaced (15.237%); 26 MiB/s, 416 objects/s recovering 2024-03-20T09:06:29.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:28 smithi122 ceph-mon[73495]: osdmap e324: 12 total, 11 up, 10 in 2024-03-20T09:06:29.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:28 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1692291170' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:06:29.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:28 smithi026 ceph-mon[68309]: pgmap v819: 47 pgs: 1 creating+peering, 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 22 active+clean, 3 active+recovering+undersized+remapped, 5 active+undersized; 5.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 81 op/s; 29830/249507 objects degraded (11.956%); 38018/249507 objects misplaced (15.237%); 26 MiB/s, 416 objects/s recovering 2024-03-20T09:06:29.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:28 smithi026 ceph-mon[68309]: osdmap e324: 12 total, 11 up, 10 in 2024-03-20T09:06:29.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:28 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1692291170' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:06:29.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:28 smithi115 ceph-mon[72476]: pgmap v819: 47 pgs: 1 creating+peering, 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 22 active+clean, 3 active+recovering+undersized+remapped, 5 active+undersized; 5.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 81 op/s; 29830/249507 objects degraded (11.956%); 38018/249507 objects misplaced (15.237%); 26 MiB/s, 416 objects/s recovering 2024-03-20T09:06:29.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:28 smithi115 ceph-mon[72476]: osdmap e324: 12 total, 11 up, 10 in 2024-03-20T09:06:29.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:28 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1692291170' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:06:29.571 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:06:29.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:29.574 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:06:29.675 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:06:29.711 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:29.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:29 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:06:29.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:29 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1692291170' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:06:29.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:29 smithi026 ceph-mon[68309]: osdmap e325: 12 total, 11 up, 10 in 2024-03-20T09:06:30.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:29 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:06:30.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:29 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1692291170' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:06:30.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:29 smithi122 ceph-mon[73495]: osdmap e325: 12 total, 11 up, 10 in 2024-03-20T09:06:30.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:29 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:06:30.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:29 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1692291170' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:06:30.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:29 smithi115 ceph-mon[72476]: osdmap e325: 12 total, 11 up, 10 in 2024-03-20T09:06:31.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:30 smithi122 ceph-mon[73495]: pgmap v822: 47 pgs: 1 creating+peering, 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 22 active+clean, 3 active+recovering+undersized+remapped, 5 active+undersized; 5.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 62 op/s; 29768/249204 objects degraded (11.945%); 37961/249204 objects misplaced (15.233%); 22 MiB/s, 346 objects/s recovering 2024-03-20T09:06:31.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:30 smithi122 ceph-mon[73495]: osdmap e326: 12 total, 11 up, 10 in 2024-03-20T09:06:31.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:30 smithi026 ceph-mon[68309]: pgmap v822: 47 pgs: 1 creating+peering, 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 22 active+clean, 3 active+recovering+undersized+remapped, 5 active+undersized; 5.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 62 op/s; 29768/249204 objects degraded (11.945%); 37961/249204 objects misplaced (15.233%); 22 MiB/s, 346 objects/s recovering 2024-03-20T09:06:31.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:30 smithi026 ceph-mon[68309]: osdmap e326: 12 total, 11 up, 10 in 2024-03-20T09:06:31.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:30 smithi115 ceph-mon[72476]: pgmap v822: 47 pgs: 1 creating+peering, 4 active+clean+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 22 active+clean, 3 active+recovering+undersized+remapped, 5 active+undersized; 5.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 62 op/s; 29768/249204 objects degraded (11.945%); 37961/249204 objects misplaced (15.233%); 22 MiB/s, 346 objects/s recovering 2024-03-20T09:06:31.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:30 smithi115 ceph-mon[72476]: osdmap e326: 12 total, 11 up, 10 in 2024-03-20T09:06:32.037 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:06:32.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:31 smithi122 ceph-mon[73495]: osdmap e327: 12 total, 11 up, 10 in 2024-03-20T09:06:32.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:31 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 29768/249204 objects degraded (11.945%), 12 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:06:32.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:31 smithi026 ceph-mon[68309]: osdmap e327: 12 total, 11 up, 10 in 2024-03-20T09:06:32.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:31 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 29768/249204 objects degraded (11.945%), 12 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:06:32.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:31 smithi115 ceph-mon[72476]: osdmap e327: 12 total, 11 up, 10 in 2024-03-20T09:06:32.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:31 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 29768/249204 objects degraded (11.945%), 12 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:06:33.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:32 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:06:33.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:32 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:06:33.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:32 smithi122 ceph-mon[73495]: pgmap v825: 47 pgs: 4 active+clean+remapped, 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 22 active+clean, 3 active+recovering+undersized+remapped, 5 active+undersized; 5.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 79 op/s; 34861/248553 objects degraded (14.026%); 35309/248553 objects misplaced (14.206%); 12 MiB/s, 193 objects/s recovering 2024-03-20T09:06:33.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:32 smithi122 ceph-mon[73495]: osdmap e328: 12 total, 11 up, 10 in 2024-03-20T09:06:33.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:32 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:06:33.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:32 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:06:33.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:32 smithi026 ceph-mon[68309]: pgmap v825: 47 pgs: 4 active+clean+remapped, 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 22 active+clean, 3 active+recovering+undersized+remapped, 5 active+undersized; 5.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 79 op/s; 34861/248553 objects degraded (14.026%); 35309/248553 objects misplaced (14.206%); 12 MiB/s, 193 objects/s recovering 2024-03-20T09:06:33.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:32 smithi026 ceph-mon[68309]: osdmap e328: 12 total, 11 up, 10 in 2024-03-20T09:06:33.345 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7, 8, 10, 11] out_osds: [9, 3] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11] 2024-03-20T09:06:33.345 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:06:33.345 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-20T09:06:33.345 INFO:tasks.thrashosds.thrasher:Setting osd 8 primary_affinity to 1.000000 2024-03-20T09:06:33.345 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd primary-affinity 8 1 2024-03-20T09:06:33.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:32 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:06:33.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:32 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:06:33.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:32 smithi115 ceph-mon[72476]: pgmap v825: 47 pgs: 4 active+clean+remapped, 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 22 active+clean, 3 active+recovering+undersized+remapped, 5 active+undersized; 5.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 79 op/s; 34861/248553 objects degraded (14.026%); 35309/248553 objects misplaced (14.206%); 12 MiB/s, 193 objects/s recovering 2024-03-20T09:06:33.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:32 smithi115 ceph-mon[72476]: osdmap e328: 12 total, 11 up, 10 in 2024-03-20T09:06:34.600 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:06:34.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:34.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:06:34.707 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:06:34.739 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:35.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:35 smithi122 ceph-mon[73495]: pgmap v827: 47 pgs: 4 active+clean+remapped, 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 23 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 5.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 103 op/s; 34784/247734 objects degraded (14.041%); 33999/247734 objects misplaced (13.724%); 22 MiB/s, 438 objects/s recovering 2024-03-20T09:06:35.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:35 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:06:35.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:35 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:06:35.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:35 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:06:35.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:35 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:06:35.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:35 smithi122 ceph-mon[73495]: osdmap e329: 12 total, 11 up, 10 in 2024-03-20T09:06:35.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:35 smithi026 ceph-mon[68309]: pgmap v827: 47 pgs: 4 active+clean+remapped, 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 23 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 5.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 103 op/s; 34784/247734 objects degraded (14.041%); 33999/247734 objects misplaced (13.724%); 22 MiB/s, 438 objects/s recovering 2024-03-20T09:06:35.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:35 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:06:35.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:35 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:06:35.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:35 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:06:35.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:35 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:06:35.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:35 smithi026 ceph-mon[68309]: osdmap e329: 12 total, 11 up, 10 in 2024-03-20T09:06:36.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:35 smithi115 ceph-mon[72476]: pgmap v827: 47 pgs: 4 active+clean+remapped, 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 23 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 5.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 103 op/s; 34784/247734 objects degraded (14.041%); 33999/247734 objects misplaced (13.724%); 22 MiB/s, 438 objects/s recovering 2024-03-20T09:06:36.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:35 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:06:36.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:35 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:06:36.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:06:36.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:06:36.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:35 smithi115 ceph-mon[72476]: osdmap e329: 12 total, 11 up, 10 in 2024-03-20T09:06:36.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:36 smithi122 ceph-mon[73495]: pgmap v829: 47 pgs: 4 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 24 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 5.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 132 op/s; 34735/246924 objects degraded (14.067%); 32288/246924 objects misplaced (13.076%); 36 MiB/s, 657 objects/s recovering 2024-03-20T09:06:36.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:36 smithi026 ceph-mon[68309]: pgmap v829: 47 pgs: 4 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 24 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 5.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 132 op/s; 34735/246924 objects degraded (14.067%); 32288/246924 objects misplaced (13.076%); 36 MiB/s, 657 objects/s recovering 2024-03-20T09:06:37.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:36 smithi115 ceph-mon[72476]: pgmap v829: 47 pgs: 4 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 24 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 5.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 132 op/s; 34735/246924 objects degraded (14.067%); 32288/246924 objects misplaced (13.076%); 36 MiB/s, 657 objects/s recovering 2024-03-20T09:06:37.949 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:37 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:06:37.949 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:37 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:06:37.949 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:37 smithi122 ceph-mon[73495]: osdmap e330: 12 total, 11 up, 10 in 2024-03-20T09:06:37.949 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:37 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 34735/246924 objects degraded (14.067%), 12 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:06:37.949 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:37 smithi122 ceph-mon[73495]: osdmap e331: 12 total, 11 up, 10 in 2024-03-20T09:06:37.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:37 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:06:37.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:37 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:06:37.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:37 smithi026 ceph-mon[68309]: osdmap e330: 12 total, 11 up, 10 in 2024-03-20T09:06:37.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:37 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 34735/246924 objects degraded (14.067%), 12 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:06:37.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:37 smithi026 ceph-mon[68309]: osdmap e331: 12 total, 11 up, 10 in 2024-03-20T09:06:38.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:37 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:06:38.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:37 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:06:38.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:37 smithi115 ceph-mon[72476]: osdmap e330: 12 total, 11 up, 10 in 2024-03-20T09:06:38.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:37 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 34735/246924 objects degraded (14.067%), 12 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:06:38.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:37 smithi115 ceph-mon[72476]: osdmap e331: 12 total, 11 up, 10 in 2024-03-20T09:06:39.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:38 smithi122 ceph-mon[73495]: pgmap v832: 47 pgs: 4 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 24 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 5.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 109 op/s; 34735/246426 objects degraded (14.096%); 31795/246426 objects misplaced (12.902%); 25 MiB/s, 401 objects/s recovering 2024-03-20T09:06:39.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:38 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:06:39.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:38 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:06:39.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:38 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:06:39.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:38 smithi026 ceph-mon[68309]: pgmap v832: 47 pgs: 4 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 24 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 5.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 109 op/s; 34735/246426 objects degraded (14.096%); 31795/246426 objects misplaced (12.902%); 25 MiB/s, 401 objects/s recovering 2024-03-20T09:06:39.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:38 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:06:39.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:38 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:06:39.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:38 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:06:39.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:38 smithi115 ceph-mon[72476]: pgmap v832: 47 pgs: 4 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 24 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 5.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 109 op/s; 34735/246426 objects degraded (14.096%); 31795/246426 objects misplaced (12.902%); 25 MiB/s, 401 objects/s recovering 2024-03-20T09:06:39.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:38 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:06:39.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:38 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:06:39.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:38 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:06:39.682 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:06:39.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:39.686 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:06:39.787 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:06:39.817 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:39.846 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:06:39.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:39.849 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:06:39.950 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:06:39.978 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:39.997 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:06:40.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:40.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:06:40.101 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:06:40.129 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:41.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:40 smithi122 ceph-mon[73495]: pgmap v833: 47 pgs: 4 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 24 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 5.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 98 op/s; 34648/245958 objects degraded (14.087%); 31717/245958 objects misplaced (12.895%); 23 MiB/s, 371 objects/s recovering 2024-03-20T09:06:41.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:40 smithi026 ceph-mon[68309]: pgmap v833: 47 pgs: 4 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 24 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 5.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 98 op/s; 34648/245958 objects degraded (14.087%); 31717/245958 objects misplaced (12.895%); 23 MiB/s, 371 objects/s recovering 2024-03-20T09:06:41.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:40 smithi115 ceph-mon[72476]: pgmap v833: 47 pgs: 4 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 24 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 5.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 98 op/s; 34648/245958 objects degraded (14.087%); 31717/245958 objects misplaced (12.895%); 23 MiB/s, 371 objects/s recovering 2024-03-20T09:06:41.880 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:06:42.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:41 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 34648/245958 objects degraded (14.087%), 12 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:06:42.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:41 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 34648/245958 objects degraded (14.087%), 12 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:06:42.280 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:06:42.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:42.284 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:06:42.385 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:06:42.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:41 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 34648/245958 objects degraded (14.087%), 12 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:06:42.423 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:42.449 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:06:42.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:42.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:06:42.555 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:06:42.589 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:42.612 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:06:42.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:42.615 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:06:42.717 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:06:42.746 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:42.766 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:06:42.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:42.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:06:42.796 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:06:42.848 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:06:42.849 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:06:42.871 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:06:42.888 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:06:42.899 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:42.917 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:06:42.917 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:06:42.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:42.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:06:42.938 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:42 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:06:42.938 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:42 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:06:42.948 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:06:42.979 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:06:43.005 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:06:43.021 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:06:43.031 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:06:43.047 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:42 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:06:43.047 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:42 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:06:43.047 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:42 smithi115 ceph-mon[72476]: pgmap v834: 47 pgs: 4 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 24 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 5.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 106 op/s; 34604/245148 objects degraded (14.116%); 29576/245148 objects misplaced (12.065%); 28 MiB/s, 451 objects/s recovering 2024-03-20T09:06:43.047 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:42 smithi115 ceph-mon[72476]: osdmap e332: 12 total, 11 up, 10 in 2024-03-20T09:06:43.048 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:42 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3714995871' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:06:43.050 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:43.053 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:06:43.053 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-03-20T09:06:43.053 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-03-20T09:06:43.053 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-03-20T09:06:43.053 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.053 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:118748 2.d 2:b971eba0:::benchmark_data_smithi122_107654_object118747:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-03-20T09:06:43.054 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T08:59:54.363884+0000", 2024-03-20T09:06:43.054 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 408.68536736200002, 2024-03-20T09:06:43.054 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.052185667999999998, 2024-03-20T09:06:43.054 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:06:43.054 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:06:43.054 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:06:43.054 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:06:43.054 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:06:43.054 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 118748 2024-03-20T09:06:43.054 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.054 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:06:43.054 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.054 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:06:43.055 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.363884+0000", 2024-03-20T09:06:43.055 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.055 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.055 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.055 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:06:43.055 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.363884+0000", 2024-03-20T09:06:43.055 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.055 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.055 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.055 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:06:43.055 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.363885+0000", 2024-03-20T09:06:43.055 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6470000000000001e-06 2024-03-20T09:06:43.055 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.055 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.056 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:06:43.056 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.363906+0000", 2024-03-20T09:06:43.056 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0386000000000001e-05 2024-03-20T09:06:43.056 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.056 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.056 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:06:43.056 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.363918+0000", 2024-03-20T09:06:43.056 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2489e-05 2024-03-20T09:06:43.056 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.056 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.056 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:06:43.056 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.363925+0000", 2024-03-20T09:06:43.056 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2690000000000002e-06 2024-03-20T09:06:43.057 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.057 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.057 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:06:43.057 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.363981+0000", 2024-03-20T09:06:43.057 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5197e-05 2024-03-20T09:06:43.057 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.057 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.057 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:06:43.057 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.364115+0000", 2024-03-20T09:06:43.057 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000134702 2024-03-20T09:06:43.057 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.057 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.057 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,4", 2024-03-20T09:06:43.057 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.364206+0000", 2024-03-20T09:06:43.058 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0320000000000003e-05 2024-03-20T09:06:43.058 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.058 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.058 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:06:43.058 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.366952+0000", 2024-03-20T09:06:43.058 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027462049999999998 2024-03-20T09:06:43.058 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.058 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.058 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:06:43.058 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.367308+0000", 2024-03-20T09:06:43.058 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00035595000000000001 2024-03-20T09:06:43.058 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.058 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.059 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:06:43.059 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.415979+0000", 2024-03-20T09:06:43.059 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.048671023000000001 2024-03-20T09:06:43.059 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.059 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.059 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:06:43.059 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.416021+0000", 2024-03-20T09:06:43.059 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2117999999999999e-05 2024-03-20T09:06:43.059 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.059 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.059 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:06:43.059 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T08:59:54.416069+0000", 2024-03-20T09:06:43.059 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8362000000000002e-05 2024-03-20T09:06:43.059 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.060 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:06:43.060 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.060 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.060 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.060 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.24550.0:160661 2.a e163/121 2:5bedbf46:::benchmark_data_smithi122_107654_object41876:head v 163'10135, mlcod=162'10119)", 2024-03-20T09:06:43.060 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:00:49.186957+0000", 2024-03-20T09:06:43.060 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 353.86229357799999, 2024-03-20T09:06:43.060 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.052572612999999997, 2024-03-20T09:06:43.060 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:06:43.060 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:06:43.060 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:06:43.060 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.060 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:06:43.060 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:49.186957+0000", 2024-03-20T09:06:43.060 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.061 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.061 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.061 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:06:43.061 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:49.186957+0000", 2024-03-20T09:06:43.061 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.061 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.061 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.061 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:06:43.061 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:49.186958+0000", 2024-03-20T09:06:43.061 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7999999999999994e-08 2024-03-20T09:06:43.061 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.061 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.061 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:06:43.061 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:49.186961+0000", 2024-03-20T09:06:43.062 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6200000000000001e-06 2024-03-20T09:06:43.062 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.062 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.062 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:06:43.062 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:49.186967+0000", 2024-03-20T09:06:43.062 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4700000000000001e-06 2024-03-20T09:06:43.062 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.062 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.062 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:06:43.062 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:49.186969+0000", 2024-03-20T09:06:43.062 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3599999999999999e-06 2024-03-20T09:06:43.062 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.062 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.063 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:06:43.063 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:49.228922+0000", 2024-03-20T09:06:43.063 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.041952518000000001 2024-03-20T09:06:43.063 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.063 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.064 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:06:43.064 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:49.228937+0000", 2024-03-20T09:06:43.064 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5739e-05 2024-03-20T09:06:43.064 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.064 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.064 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:06:43.064 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:49.239484+0000", 2024-03-20T09:06:43.064 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010546934000000001 2024-03-20T09:06:43.064 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.064 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.065 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:06:43.065 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:49.239530+0000", 2024-03-20T09:06:43.065 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5884000000000003e-05 2024-03-20T09:06:43.066 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.066 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:06:43.066 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.066 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.066 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.066 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.24550.0:160664 2.a e163/121 2:5ccc4994:::benchmark_data_smithi122_107654_object41879:head v 163'10136, mlcod=162'10119)", 2024-03-20T09:06:43.066 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:00:49.186972+0000", 2024-03-20T09:06:43.066 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 353.86227909299998, 2024-03-20T09:06:43.066 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.052633104, 2024-03-20T09:06:43.066 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:06:43.066 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:06:43.067 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:06:43.067 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.067 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:06:43.067 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:49.186972+0000", 2024-03-20T09:06:43.067 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.067 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.067 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.067 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:06:43.067 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:49.186972+0000", 2024-03-20T09:06:43.067 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.067 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.067 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.067 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:06:43.067 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:49.186972+0000", 2024-03-20T09:06:43.068 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9000000000000003e-08 2024-03-20T09:06:43.068 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.068 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.068 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:06:43.068 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:49.186974+0000", 2024-03-20T09:06:43.068 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9829999999999999e-06 2024-03-20T09:06:43.068 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.068 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.068 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:06:43.068 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:49.186977+0000", 2024-03-20T09:06:43.068 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7989999999999998e-06 2024-03-20T09:06:43.068 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.068 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.068 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:06:43.068 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:49.186979+0000", 2024-03-20T09:06:43.069 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0490000000000002e-06 2024-03-20T09:06:43.069 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.069 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.069 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:06:43.069 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:49.232178+0000", 2024-03-20T09:06:43.069 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.045199594000000003 2024-03-20T09:06:43.069 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.069 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.069 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:06:43.070 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:49.232201+0000", 2024-03-20T09:06:43.070 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2932e-05 2024-03-20T09:06:43.070 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.070 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.070 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:06:43.070 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:49.239552+0000", 2024-03-20T09:06:43.070 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0073505230000000003 2024-03-20T09:06:43.070 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.070 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.070 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:06:43.070 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:00:49.239605+0000", 2024-03-20T09:06:43.070 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3134999999999999e-05 2024-03-20T09:06:43.070 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.071 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:06:43.071 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.071 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.071 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.071 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:175761 2.d 2:bf2603ab:::benchmark_data_smithi122_107654_object56976:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-03-20T09:06:43.071 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:01:21.527435+0000", 2024-03-20T09:06:43.071 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 321.52181632399999, 2024-03-20T09:06:43.071 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.217663624, 2024-03-20T09:06:43.071 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:06:43.071 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:06:43.071 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:06:43.071 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:06:43.071 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:06:43.071 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 175761 2024-03-20T09:06:43.072 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.072 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:06:43.072 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.072 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:06:43.072 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.527435+0000", 2024-03-20T09:06:43.072 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.072 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.072 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.072 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:06:43.072 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.527435+0000", 2024-03-20T09:06:43.072 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.072 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.072 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.072 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:06:43.073 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.527436+0000", 2024-03-20T09:06:43.073 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7829999999999999e-06 2024-03-20T09:06:43.073 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.073 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.073 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:06:43.073 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.527442+0000", 2024-03-20T09:06:43.073 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3700000000000003e-06 2024-03-20T09:06:43.073 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.073 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.073 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:06:43.073 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.527458+0000", 2024-03-20T09:06:43.073 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5699999999999999e-05 2024-03-20T09:06:43.073 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.073 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.074 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:06:43.074 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.527466+0000", 2024-03-20T09:06:43.074 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3799999999999994e-06 2024-03-20T09:06:43.074 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.074 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.074 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:06:43.074 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.527523+0000", 2024-03-20T09:06:43.074 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6935000000000003e-05 2024-03-20T09:06:43.074 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.074 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.074 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:06:43.074 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.527634+0000", 2024-03-20T09:06:43.074 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011149099999999999 2024-03-20T09:06:43.075 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.075 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.075 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,4", 2024-03-20T09:06:43.075 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.527711+0000", 2024-03-20T09:06:43.075 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6625999999999994e-05 2024-03-20T09:06:43.075 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.075 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.075 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:06:43.075 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.535121+0000", 2024-03-20T09:06:43.075 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0074103340000000002 2024-03-20T09:06:43.075 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.075 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.075 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:06:43.075 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.535568+0000", 2024-03-20T09:06:43.076 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00044684400000000001 2024-03-20T09:06:43.076 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.076 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.076 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:06:43.076 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.744991+0000", 2024-03-20T09:06:43.076 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20942271900000001 2024-03-20T09:06:43.076 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.076 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.076 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:06:43.076 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.745050+0000", 2024-03-20T09:06:43.076 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9577999999999998e-05 2024-03-20T09:06:43.076 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.076 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.076 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:06:43.077 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.745098+0000", 2024-03-20T09:06:43.077 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7864000000000001e-05 2024-03-20T09:06:43.077 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.077 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:06:43.077 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.077 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.077 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.077 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:175769 2.d 2:b8f49887:::benchmark_data_smithi122_107654_object56984:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-03-20T09:06:43.077 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:01:21.529376+0000", 2024-03-20T09:06:43.077 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 321.51987458799999, 2024-03-20T09:06:43.077 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.21617177300000001, 2024-03-20T09:06:43.077 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:06:43.077 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:06:43.078 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:06:43.078 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:06:43.078 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:06:43.078 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 175769 2024-03-20T09:06:43.078 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.078 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:06:43.078 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.078 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:06:43.078 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.529376+0000", 2024-03-20T09:06:43.078 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.078 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.078 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.078 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:06:43.078 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.529376+0000", 2024-03-20T09:06:43.079 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.079 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.079 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.079 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:06:43.079 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.529378+0000", 2024-03-20T09:06:43.079 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.539e-06 2024-03-20T09:06:43.079 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.079 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.079 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:06:43.079 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.529385+0000", 2024-03-20T09:06:43.079 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5209999999999999e-06 2024-03-20T09:06:43.079 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.079 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.079 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:06:43.080 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.529397+0000", 2024-03-20T09:06:43.080 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2432000000000001e-05 2024-03-20T09:06:43.080 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.080 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.080 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:06:43.080 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.529405+0000", 2024-03-20T09:06:43.080 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1189999999999997e-06 2024-03-20T09:06:43.080 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.080 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.080 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:06:43.080 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.531481+0000", 2024-03-20T09:06:43.080 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0020761740000000001 2024-03-20T09:06:43.080 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.080 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.081 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:06:43.081 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.531589+0000", 2024-03-20T09:06:43.081 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010778299999999999 2024-03-20T09:06:43.081 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.081 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.081 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,4", 2024-03-20T09:06:43.081 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.531665+0000", 2024-03-20T09:06:43.081 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6352000000000004e-05 2024-03-20T09:06:43.081 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.081 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.081 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:06:43.081 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.535246+0000", 2024-03-20T09:06:43.081 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0035810479999999999 2024-03-20T09:06:43.081 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.082 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.082 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:06:43.082 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.535687+0000", 2024-03-20T09:06:43.082 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000441025 2024-03-20T09:06:43.082 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.082 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.082 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:06:43.082 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.745459+0000", 2024-03-20T09:06:43.082 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.209771177 2024-03-20T09:06:43.082 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.082 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.082 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:06:43.082 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.745510+0000", 2024-03-20T09:06:43.083 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1198e-05 2024-03-20T09:06:43.083 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.083 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.083 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:06:43.083 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.745548+0000", 2024-03-20T09:06:43.083 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8405000000000002e-05 2024-03-20T09:06:43.083 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.083 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:06:43.083 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.083 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.083 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.083 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:175828 2.d 2:b98a2b8a:::benchmark_data_smithi122_107654_object57043:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-03-20T09:06:43.083 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:01:21.612159+0000", 2024-03-20T09:06:43.083 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 321.43709237299998, 2024-03-20T09:06:43.084 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.13743612999999999, 2024-03-20T09:06:43.084 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:06:43.084 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:06:43.084 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:06:43.084 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:06:43.084 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:06:43.084 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 175828 2024-03-20T09:06:43.084 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.084 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:06:43.084 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.084 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:06:43.084 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.612159+0000", 2024-03-20T09:06:43.084 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.084 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.085 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.085 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:06:43.085 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.612159+0000", 2024-03-20T09:06:43.085 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.085 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.085 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.085 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:06:43.085 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.612161+0000", 2024-03-20T09:06:43.085 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.838e-06 2024-03-20T09:06:43.085 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.085 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.085 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:06:43.085 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.612167+0000", 2024-03-20T09:06:43.085 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1789999999999996e-06 2024-03-20T09:06:43.086 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.086 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.086 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:06:43.086 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.612179+0000", 2024-03-20T09:06:43.086 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1898e-05 2024-03-20T09:06:43.086 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.086 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.086 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:06:43.086 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.612187+0000", 2024-03-20T09:06:43.086 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5229999999999995e-06 2024-03-20T09:06:43.086 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.086 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.086 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:06:43.087 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.612243+0000", 2024-03-20T09:06:43.087 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5550999999999997e-05 2024-03-20T09:06:43.087 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.087 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.087 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:06:43.087 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.612342+0000", 2024-03-20T09:06:43.087 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9178999999999994e-05 2024-03-20T09:06:43.087 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.087 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.087 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,4", 2024-03-20T09:06:43.087 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.612436+0000", 2024-03-20T09:06:43.087 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3715000000000007e-05 2024-03-20T09:06:43.087 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.087 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.088 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:06:43.088 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.616968+0000", 2024-03-20T09:06:43.088 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0045325039999999997 2024-03-20T09:06:43.088 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.088 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.088 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:06:43.088 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.618085+0000", 2024-03-20T09:06:43.088 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0011170959999999999 2024-03-20T09:06:43.088 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.088 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.088 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:06:43.088 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.749530+0000", 2024-03-20T09:06:43.088 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.13144441900000001 2024-03-20T09:06:43.088 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.089 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.089 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:06:43.089 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.749566+0000", 2024-03-20T09:06:43.089 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6170000000000001e-05 2024-03-20T09:06:43.089 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.089 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.089 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:06:43.089 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.749595+0000", 2024-03-20T09:06:43.089 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9057999999999999e-05 2024-03-20T09:06:43.089 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.089 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:06:43.089 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.089 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.089 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.090 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:175869 2.d 2:b1012f91:::benchmark_data_smithi122_107654_object57084:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-03-20T09:06:43.090 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:01:21.654958+0000", 2024-03-20T09:06:43.090 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 321.39429303499998, 2024-03-20T09:06:43.090 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.11208813099999999, 2024-03-20T09:06:43.090 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:06:43.090 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:06:43.090 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:06:43.090 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:06:43.090 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:06:43.090 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 175869 2024-03-20T09:06:43.090 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.090 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:06:43.090 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.091 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:06:43.091 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.654958+0000", 2024-03-20T09:06:43.091 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.091 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.091 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.091 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:06:43.091 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.654958+0000", 2024-03-20T09:06:43.091 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.091 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.091 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.091 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:06:43.091 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.654959+0000", 2024-03-20T09:06:43.091 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4699999999999999e-06 2024-03-20T09:06:43.091 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.092 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.092 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:06:43.092 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.654966+0000", 2024-03-20T09:06:43.092 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6379999999999998e-06 2024-03-20T09:06:43.092 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.092 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.092 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:06:43.092 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.654979+0000", 2024-03-20T09:06:43.092 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3101e-05 2024-03-20T09:06:43.092 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.092 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.092 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:06:43.092 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.654991+0000", 2024-03-20T09:06:43.092 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1328000000000001e-05 2024-03-20T09:06:43.092 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.093 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.093 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:06:43.093 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.655054+0000", 2024-03-20T09:06:43.093 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3812000000000005e-05 2024-03-20T09:06:43.093 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.093 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.093 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:06:43.093 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.655172+0000", 2024-03-20T09:06:43.093 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011718900000000001 2024-03-20T09:06:43.093 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.093 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.093 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,4", 2024-03-20T09:06:43.093 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.655291+0000", 2024-03-20T09:06:43.094 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000119813 2024-03-20T09:06:43.094 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.094 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.094 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:06:43.094 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.659715+0000", 2024-03-20T09:06:43.094 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0044238719999999997 2024-03-20T09:06:43.094 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.094 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.094 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:06:43.094 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.659858+0000", 2024-03-20T09:06:43.094 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014323100000000001 2024-03-20T09:06:43.094 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.094 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.094 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:06:43.094 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.766945+0000", 2024-03-20T09:06:43.095 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.107086707 2024-03-20T09:06:43.095 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.095 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.095 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:06:43.095 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.767001+0000", 2024-03-20T09:06:43.095 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5701000000000001e-05 2024-03-20T09:06:43.095 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.095 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.095 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:06:43.095 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.767046+0000", 2024-03-20T09:06:43.095 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5269000000000003e-05 2024-03-20T09:06:43.095 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.095 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:06:43.096 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.096 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.096 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.096 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:175907 2.d 2:be16808b:::benchmark_data_smithi122_107654_object57122:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-03-20T09:06:43.096 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:01:21.702038+0000", 2024-03-20T09:06:43.096 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 321.34721297800002, 2024-03-20T09:06:43.096 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.065159905000000004, 2024-03-20T09:06:43.096 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:06:43.096 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:06:43.096 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:06:43.096 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:06:43.096 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:06:43.096 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 175907 2024-03-20T09:06:43.096 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.096 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:06:43.097 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.097 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:06:43.097 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.702038+0000", 2024-03-20T09:06:43.097 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.097 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.097 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.097 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:06:43.097 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.702038+0000", 2024-03-20T09:06:43.097 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.097 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.097 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.097 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:06:43.097 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.702039+0000", 2024-03-20T09:06:43.098 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4249999999999999e-06 2024-03-20T09:06:43.098 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.098 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.098 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:06:43.098 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.702046+0000", 2024-03-20T09:06:43.098 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1789999999999996e-06 2024-03-20T09:06:43.098 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.098 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.098 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:06:43.098 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.702059+0000", 2024-03-20T09:06:43.098 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.382e-05 2024-03-20T09:06:43.098 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.098 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.098 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:06:43.099 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.702067+0000", 2024-03-20T09:06:43.099 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4950000000000002e-06 2024-03-20T09:06:43.099 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.099 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.099 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:06:43.099 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.702124+0000", 2024-03-20T09:06:43.099 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7454999999999998e-05 2024-03-20T09:06:43.099 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.099 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.099 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:06:43.099 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.702249+0000", 2024-03-20T09:06:43.099 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012431599999999999 2024-03-20T09:06:43.099 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.099 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.100 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,4", 2024-03-20T09:06:43.100 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.702367+0000", 2024-03-20T09:06:43.100 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011844400000000001 2024-03-20T09:06:43.100 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.100 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.100 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:06:43.100 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.706782+0000", 2024-03-20T09:06:43.100 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0044149289999999997 2024-03-20T09:06:43.100 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.100 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.100 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:06:43.100 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.706821+0000", 2024-03-20T09:06:43.100 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8451000000000002e-05 2024-03-20T09:06:43.100 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.101 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.101 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:06:43.101 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.767101+0000", 2024-03-20T09:06:43.101 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.060280496000000003 2024-03-20T09:06:43.101 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.101 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.101 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:06:43.101 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.767156+0000", 2024-03-20T09:06:43.101 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4617000000000002e-05 2024-03-20T09:06:43.101 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.101 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.101 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:06:43.101 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.767198+0000", 2024-03-20T09:06:43.101 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2277999999999997e-05 2024-03-20T09:06:43.102 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.102 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:06:43.102 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.102 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.102 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.102 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:175909 2.d 2:b9d345a8:::benchmark_data_smithi122_107654_object57124:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-03-20T09:06:43.102 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:01:21.708400+0000", 2024-03-20T09:06:43.102 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 321.34085095799998, 2024-03-20T09:06:43.102 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.058924399000000002, 2024-03-20T09:06:43.102 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:06:43.102 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:06:43.102 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:06:43.102 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:06:43.102 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:06:43.103 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 175909 2024-03-20T09:06:43.103 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.103 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:06:43.103 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.103 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:06:43.103 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.708400+0000", 2024-03-20T09:06:43.103 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.103 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.103 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.103 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:06:43.103 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.708400+0000", 2024-03-20T09:06:43.103 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.103 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.104 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.104 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:06:43.104 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.708402+0000", 2024-03-20T09:06:43.104 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8589999999999999e-06 2024-03-20T09:06:43.104 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.104 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.104 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:06:43.104 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.708409+0000", 2024-03-20T09:06:43.104 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5710000000000003e-06 2024-03-20T09:06:43.104 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.104 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.104 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:06:43.104 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.708434+0000", 2024-03-20T09:06:43.104 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5049999999999999e-05 2024-03-20T09:06:43.105 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.105 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.105 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:06:43.105 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.708442+0000", 2024-03-20T09:06:43.105 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5390000000000007e-06 2024-03-20T09:06:43.105 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.105 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.105 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:06:43.105 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.708498+0000", 2024-03-20T09:06:43.105 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5535999999999998e-05 2024-03-20T09:06:43.105 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.105 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.105 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:06:43.106 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.708609+0000", 2024-03-20T09:06:43.106 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000111204 2024-03-20T09:06:43.106 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.106 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.106 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,4", 2024-03-20T09:06:43.106 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.708712+0000", 2024-03-20T09:06:43.106 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000103201 2024-03-20T09:06:43.106 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.106 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.106 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:06:43.106 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.713421+0000", 2024-03-20T09:06:43.106 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0047091429999999998 2024-03-20T09:06:43.106 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.107 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.107 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:06:43.107 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.713533+0000", 2024-03-20T09:06:43.107 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000111866 2024-03-20T09:06:43.107 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.107 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.107 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:06:43.107 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.767252+0000", 2024-03-20T09:06:43.107 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.053719252000000002 2024-03-20T09:06:43.107 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.107 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.107 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:06:43.107 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.767286+0000", 2024-03-20T09:06:43.107 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3847000000000003e-05 2024-03-20T09:06:43.108 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.108 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.108 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:06:43.108 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.767324+0000", 2024-03-20T09:06:43.108 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8331000000000003e-05 2024-03-20T09:06:43.108 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.108 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:06:43.108 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.108 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.108 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.108 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:175915 2.d 2:b463c085:::benchmark_data_smithi122_107654_object57130:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-03-20T09:06:43.108 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:01:21.713801+0000", 2024-03-20T09:06:43.108 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 321.33545043700002, 2024-03-20T09:06:43.109 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.053662903999999997, 2024-03-20T09:06:43.109 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:06:43.109 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:06:43.109 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:06:43.109 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:06:43.109 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:06:43.109 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 175915 2024-03-20T09:06:43.109 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.109 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:06:43.109 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.109 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:06:43.109 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.713801+0000", 2024-03-20T09:06:43.109 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.109 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.109 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.110 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:06:43.110 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.713801+0000", 2024-03-20T09:06:43.110 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.110 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.110 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.110 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:06:43.110 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.713802+0000", 2024-03-20T09:06:43.110 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5340000000000001e-06 2024-03-20T09:06:43.110 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.110 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.110 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:06:43.110 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.713808+0000", 2024-03-20T09:06:43.110 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1870000000000002e-06 2024-03-20T09:06:43.111 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.111 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.111 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:06:43.111 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.713821+0000", 2024-03-20T09:06:43.111 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.308e-05 2024-03-20T09:06:43.111 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.111 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.111 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:06:43.111 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.713831+0000", 2024-03-20T09:06:43.111 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.8509999999999994e-06 2024-03-20T09:06:43.111 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.111 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.111 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:06:43.112 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.713887+0000", 2024-03-20T09:06:43.112 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5347999999999999e-05 2024-03-20T09:06:43.112 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.112 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.112 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:06:43.112 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.714004+0000", 2024-03-20T09:06:43.112 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011745 2024-03-20T09:06:43.112 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.112 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.112 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,4", 2024-03-20T09:06:43.112 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.714073+0000", 2024-03-20T09:06:43.112 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9398000000000002e-05 2024-03-20T09:06:43.112 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.112 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.113 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:06:43.113 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.718582+0000", 2024-03-20T09:06:43.113 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0045083959999999996 2024-03-20T09:06:43.113 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.113 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.113 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:06:43.113 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.718698+0000", 2024-03-20T09:06:43.113 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000116519 2024-03-20T09:06:43.113 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.113 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.113 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:06:43.113 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.767367+0000", 2024-03-20T09:06:43.113 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.048668790000000003 2024-03-20T09:06:43.113 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.114 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.114 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:06:43.114 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.767421+0000", 2024-03-20T09:06:43.114 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4143e-05 2024-03-20T09:06:43.114 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.114 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.114 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:06:43.114 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.767464+0000", 2024-03-20T09:06:43.114 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2208e-05 2024-03-20T09:06:43.114 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.114 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:06:43.114 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.114 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.115 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.115 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:175921 2.d 2:b457806e:::benchmark_data_smithi122_107654_object57136:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-03-20T09:06:43.115 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:01:21.719538+0000", 2024-03-20T09:06:43.115 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 321.32971296900001, 2024-03-20T09:06:43.115 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.052064786000000002, 2024-03-20T09:06:43.115 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:06:43.115 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:06:43.115 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:06:43.115 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:06:43.115 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:06:43.115 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 175921 2024-03-20T09:06:43.115 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.115 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:06:43.115 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.116 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:06:43.116 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.719538+0000", 2024-03-20T09:06:43.116 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.116 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.116 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.116 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:06:43.116 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.719538+0000", 2024-03-20T09:06:43.116 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.116 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.116 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.116 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:06:43.116 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.719540+0000", 2024-03-20T09:06:43.116 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1320000000000001e-06 2024-03-20T09:06:43.116 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.116 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.117 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:06:43.117 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.719546+0000", 2024-03-20T09:06:43.117 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5520000000000004e-06 2024-03-20T09:06:43.117 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.117 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.117 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:06:43.117 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.719559+0000", 2024-03-20T09:06:43.117 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3067e-05 2024-03-20T09:06:43.117 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.117 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.117 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:06:43.117 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.719566+0000", 2024-03-20T09:06:43.117 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5190000000000003e-06 2024-03-20T09:06:43.117 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.118 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.118 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:06:43.118 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.722342+0000", 2024-03-20T09:06:43.118 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027753280000000001 2024-03-20T09:06:43.118 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.118 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.118 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:06:43.118 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.722465+0000", 2024-03-20T09:06:43.118 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000123585 2024-03-20T09:06:43.118 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.118 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.118 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,4", 2024-03-20T09:06:43.118 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.722582+0000", 2024-03-20T09:06:43.119 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011691799999999999 2024-03-20T09:06:43.119 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.119 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.119 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:06:43.119 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.731264+0000", 2024-03-20T09:06:43.119 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0086821539999999992 2024-03-20T09:06:43.119 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.119 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.119 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:06:43.119 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.731524+0000", 2024-03-20T09:06:43.119 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00025944500000000002 2024-03-20T09:06:43.119 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.119 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.119 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:06:43.120 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.771498+0000", 2024-03-20T09:06:43.120 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.039974020999999998 2024-03-20T09:06:43.120 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.120 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.120 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:06:43.120 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.771555+0000", 2024-03-20T09:06:43.120 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7666e-05 2024-03-20T09:06:43.120 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.120 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.120 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:06:43.120 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.771603+0000", 2024-03-20T09:06:43.120 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7398999999999998e-05 2024-03-20T09:06:43.120 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.120 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:06:43.121 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.121 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.121 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.121 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:175924 2.d 2:bbf39ba8:::benchmark_data_smithi122_107654_object57139:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-03-20T09:06:43.121 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:01:21.720055+0000", 2024-03-20T09:06:43.121 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 321.32919652700002, 2024-03-20T09:06:43.121 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.051663834999999998, 2024-03-20T09:06:43.121 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:06:43.121 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:06:43.121 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:06:43.121 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:06:43.121 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:06:43.121 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 175924 2024-03-20T09:06:43.121 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.122 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:06:43.122 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.122 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:06:43.122 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.720055+0000", 2024-03-20T09:06:43.122 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.122 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.122 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.122 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:06:43.122 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.720055+0000", 2024-03-20T09:06:43.122 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.122 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.122 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.122 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:06:43.123 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.720055+0000", 2024-03-20T09:06:43.123 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6700000000000003e-07 2024-03-20T09:06:43.123 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.123 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.123 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:06:43.123 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.720060+0000", 2024-03-20T09:06:43.123 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4800000000000003e-06 2024-03-20T09:06:43.123 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.123 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.123 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:06:43.123 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.720070+0000", 2024-03-20T09:06:43.123 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0329999999999999e-05 2024-03-20T09:06:43.123 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.123 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.123 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:06:43.124 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.720077+0000", 2024-03-20T09:06:43.124 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6889999999999998e-06 2024-03-20T09:06:43.124 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.124 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.124 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:06:43.124 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.726675+0000", 2024-03-20T09:06:43.124 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0065983099999999996 2024-03-20T09:06:43.124 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.124 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.124 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:06:43.124 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.726797+0000", 2024-03-20T09:06:43.124 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000121735 2024-03-20T09:06:43.125 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.125 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.125 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,4", 2024-03-20T09:06:43.125 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.727431+0000", 2024-03-20T09:06:43.125 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00063405000000000004 2024-03-20T09:06:43.125 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.125 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.125 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:06:43.125 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.731681+0000", 2024-03-20T09:06:43.125 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0042503339999999997 2024-03-20T09:06:43.125 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.125 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.125 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:06:43.125 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.735093+0000", 2024-03-20T09:06:43.126 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0034115619999999999 2024-03-20T09:06:43.126 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.126 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.126 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:06:43.126 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.771640+0000", 2024-03-20T09:06:43.126 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.036547341999999997 2024-03-20T09:06:43.126 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.126 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.126 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:06:43.126 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.771684+0000", 2024-03-20T09:06:43.126 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3664000000000001e-05 2024-03-20T09:06:43.126 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.126 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.127 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:06:43.127 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:21.771718+0000", 2024-03-20T09:06:43.127 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4572e-05 2024-03-20T09:06:43.127 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.127 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:06:43.127 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.127 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.127 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.127 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:178057 2.d 2:bf3924a1:::benchmark_data_smithi122_107654_object59272:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e173)", 2024-03-20T09:06:43.127 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:01:24.152831+0000", 2024-03-20T09:06:43.127 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 318.89641975199999, 2024-03-20T09:06:43.127 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.049537255000000002, 2024-03-20T09:06:43.127 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:06:43.128 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:06:43.128 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:06:43.128 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:06:43.128 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:06:43.128 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 178057 2024-03-20T09:06:43.128 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.128 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:06:43.128 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.128 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:06:43.128 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.152831+0000", 2024-03-20T09:06:43.128 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.128 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.128 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.128 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:06:43.129 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.152831+0000", 2024-03-20T09:06:43.129 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.129 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.129 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.129 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:06:43.129 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.152832+0000", 2024-03-20T09:06:43.129 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.32e-07 2024-03-20T09:06:43.129 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.129 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.129 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:06:43.130 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.152833+0000", 2024-03-20T09:06:43.130 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7680000000000001e-06 2024-03-20T09:06:43.130 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.130 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.130 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:06:43.130 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.152840+0000", 2024-03-20T09:06:43.130 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0719999999999996e-06 2024-03-20T09:06:43.130 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.130 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.130 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:06:43.130 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.152846+0000", 2024-03-20T09:06:43.130 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9549999999999997e-06 2024-03-20T09:06:43.130 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.131 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.131 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:06:43.131 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.177410+0000", 2024-03-20T09:06:43.131 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.024564981 2024-03-20T09:06:43.131 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.131 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.131 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:06:43.131 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.177531+0000", 2024-03-20T09:06:43.131 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000120067 2024-03-20T09:06:43.131 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.131 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.131 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,4", 2024-03-20T09:06:43.131 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.177661+0000", 2024-03-20T09:06:43.131 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013047300000000001 2024-03-20T09:06:43.132 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.132 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.132 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:06:43.132 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.181029+0000", 2024-03-20T09:06:43.132 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0033682040000000001 2024-03-20T09:06:43.132 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.132 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.132 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:06:43.132 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.186061+0000", 2024-03-20T09:06:43.132 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0050313349999999996 2024-03-20T09:06:43.132 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.132 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.132 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:06:43.132 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.202250+0000", 2024-03-20T09:06:43.133 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016189058999999999 2024-03-20T09:06:43.133 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.133 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.133 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:06:43.133 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.202316+0000", 2024-03-20T09:06:43.133 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6707999999999999e-05 2024-03-20T09:06:43.133 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.133 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.133 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:06:43.133 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.202369+0000", 2024-03-20T09:06:43.133 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2200999999999997e-05 2024-03-20T09:06:43.133 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.133 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:06:43.133 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.134 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.134 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.134 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:178066 2.d 2:b3bdb337:::benchmark_data_smithi122_107654_object59281:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e173)", 2024-03-20T09:06:43.134 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:01:24.152850+0000", 2024-03-20T09:06:43.134 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 318.89640054300003, 2024-03-20T09:06:43.134 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.049667280000000001, 2024-03-20T09:06:43.134 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:06:43.134 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:06:43.134 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:06:43.134 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:06:43.134 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:06:43.134 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 178066 2024-03-20T09:06:43.134 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.134 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:06:43.135 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.135 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:06:43.135 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.152850+0000", 2024-03-20T09:06:43.135 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.135 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.135 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.135 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:06:43.135 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.152850+0000", 2024-03-20T09:06:43.135 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.135 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.135 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.135 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:06:43.135 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.152851+0000", 2024-03-20T09:06:43.135 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8000000000000002e-07 2024-03-20T09:06:43.136 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.136 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.136 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:06:43.136 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.152853+0000", 2024-03-20T09:06:43.136 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7269999999999999e-06 2024-03-20T09:06:43.136 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.136 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.136 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:06:43.136 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.152859+0000", 2024-03-20T09:06:43.136 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.015e-06 2024-03-20T09:06:43.136 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.136 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.136 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:06:43.136 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.152863+0000", 2024-03-20T09:06:43.136 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.301e-06 2024-03-20T09:06:43.137 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.137 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.137 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:06:43.137 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.181200+0000", 2024-03-20T09:06:43.137 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.028337277000000001 2024-03-20T09:06:43.137 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.137 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.137 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:06:43.137 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.181328+0000", 2024-03-20T09:06:43.137 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012754500000000001 2024-03-20T09:06:43.137 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.137 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.137 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,4", 2024-03-20T09:06:43.138 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.181457+0000", 2024-03-20T09:06:43.138 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012943699999999999 2024-03-20T09:06:43.138 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.138 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.138 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:06:43.138 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.186107+0000", 2024-03-20T09:06:43.138 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0046496740000000003 2024-03-20T09:06:43.138 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.138 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.138 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:06:43.138 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.197318+0000", 2024-03-20T09:06:43.138 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011211284 2024-03-20T09:06:43.138 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.138 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.139 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:06:43.139 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.202417+0000", 2024-03-20T09:06:43.139 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.005099045 2024-03-20T09:06:43.139 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.139 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.139 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:06:43.139 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.202481+0000", 2024-03-20T09:06:43.139 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4245000000000006e-05 2024-03-20T09:06:43.139 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.139 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.139 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:06:43.139 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.202518+0000", 2024-03-20T09:06:43.139 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6449999999999998e-05 2024-03-20T09:06:43.139 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.140 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:06:43.140 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.140 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.140 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.140 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:178083 2.d 2:bd298ceb:::benchmark_data_smithi122_107654_object59298:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e173)", 2024-03-20T09:06:43.140 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:01:24.152868+0000", 2024-03-20T09:06:43.140 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 318.89638342199999, 2024-03-20T09:06:43.140 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.049776828000000002, 2024-03-20T09:06:43.140 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:06:43.140 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:06:43.140 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:06:43.140 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:06:43.140 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:06:43.140 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 178083 2024-03-20T09:06:43.140 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.141 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:06:43.141 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.141 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:06:43.141 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.152868+0000", 2024-03-20T09:06:43.141 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.141 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.141 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.141 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:06:43.141 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.152868+0000", 2024-03-20T09:06:43.141 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.141 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.141 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.141 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:06:43.141 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.152868+0000", 2024-03-20T09:06:43.142 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7700000000000001e-07 2024-03-20T09:06:43.142 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.142 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.142 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:06:43.142 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.152870+0000", 2024-03-20T09:06:43.142 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.612e-06 2024-03-20T09:06:43.142 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.142 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.142 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:06:43.142 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.152876+0000", 2024-03-20T09:06:43.142 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0279999999999997e-06 2024-03-20T09:06:43.142 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.142 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.143 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:06:43.143 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.152880+0000", 2024-03-20T09:06:43.143 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.408e-06 2024-03-20T09:06:43.143 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.143 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.143 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:06:43.143 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.186152+0000", 2024-03-20T09:06:43.143 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.033271631000000003 2024-03-20T09:06:43.143 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.143 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.143 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:06:43.143 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.186275+0000", 2024-03-20T09:06:43.143 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000123707 2024-03-20T09:06:43.143 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.144 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.144 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,4", 2024-03-20T09:06:43.144 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.186397+0000", 2024-03-20T09:06:43.144 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012189900000000001 2024-03-20T09:06:43.144 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.144 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.144 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:06:43.144 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.197302+0000", 2024-03-20T09:06:43.144 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010904374999999999 2024-03-20T09:06:43.144 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.144 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.144 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:06:43.144 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.197375+0000", 2024-03-20T09:06:43.145 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3461999999999996e-05 2024-03-20T09:06:43.145 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.145 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.145 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:06:43.145 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.202559+0000", 2024-03-20T09:06:43.145 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0051837039999999999 2024-03-20T09:06:43.145 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.145 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.145 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:06:43.145 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.202609+0000", 2024-03-20T09:06:43.145 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9963999999999998e-05 2024-03-20T09:06:43.145 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.145 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.146 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:06:43.146 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.202644+0000", 2024-03-20T09:06:43.146 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5760999999999999e-05 2024-03-20T09:06:43.146 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.146 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:06:43.146 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.146 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.146 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.146 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:178092 2.d 2:b1c79f15:::benchmark_data_smithi122_107654_object59307:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e173)", 2024-03-20T09:06:43.146 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:01:24.152886+0000", 2024-03-20T09:06:43.146 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 318.89636544500001, 2024-03-20T09:06:43.146 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.049917079000000003, 2024-03-20T09:06:43.146 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:06:43.147 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:06:43.147 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:06:43.147 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:06:43.147 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:06:43.147 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 178092 2024-03-20T09:06:43.147 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.147 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:06:43.147 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.147 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:06:43.147 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.152886+0000", 2024-03-20T09:06:43.147 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.147 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.147 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.147 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:06:43.148 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.152886+0000", 2024-03-20T09:06:43.148 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.148 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.148 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.148 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:06:43.148 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.152886+0000", 2024-03-20T09:06:43.148 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9499999999999998e-07 2024-03-20T09:06:43.148 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.148 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.148 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:06:43.148 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.152888+0000", 2024-03-20T09:06:43.148 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7069999999999999e-06 2024-03-20T09:06:43.148 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.148 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.148 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:06:43.149 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.152894+0000", 2024-03-20T09:06:43.149 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9719999999999997e-06 2024-03-20T09:06:43.149 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.149 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.149 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:06:43.149 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.152898+0000", 2024-03-20T09:06:43.149 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4569999999999998e-06 2024-03-20T09:06:43.149 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.149 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.149 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:06:43.149 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.189998+0000", 2024-03-20T09:06:43.149 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.037099701999999998 2024-03-20T09:06:43.149 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.150 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.150 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:06:43.150 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.190126+0000", 2024-03-20T09:06:43.150 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012846600000000001 2024-03-20T09:06:43.150 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.150 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.150 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,4", 2024-03-20T09:06:43.150 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.190251+0000", 2024-03-20T09:06:43.150 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012473900000000001 2024-03-20T09:06:43.150 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.150 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.150 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:06:43.150 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.197364+0000", 2024-03-20T09:06:43.150 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0071126669999999996 2024-03-20T09:06:43.151 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.151 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.151 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:06:43.151 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.197379+0000", 2024-03-20T09:06:43.151 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5843000000000001e-05 2024-03-20T09:06:43.151 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.151 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.151 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:06:43.151 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.202716+0000", 2024-03-20T09:06:43.151 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0053362679999999999 2024-03-20T09:06:43.151 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.151 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.151 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:06:43.151 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.202765+0000", 2024-03-20T09:06:43.151 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9101999999999998e-05 2024-03-20T09:06:43.152 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.152 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.152 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:06:43.152 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.202803+0000", 2024-03-20T09:06:43.152 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7861000000000002e-05 2024-03-20T09:06:43.152 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.152 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:06:43.152 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.152 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.152 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.152 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:178111 2.d 2:bb252570:::benchmark_data_smithi122_107654_object59326:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e173)", 2024-03-20T09:06:43.152 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:01:24.152904+0000", 2024-03-20T09:06:43.152 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 318.89634713200002, 2024-03-20T09:06:43.152 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.058570822000000002, 2024-03-20T09:06:43.153 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:06:43.153 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:06:43.153 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:06:43.153 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:06:43.153 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:06:43.153 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 178111 2024-03-20T09:06:43.153 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.153 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:06:43.153 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.153 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:06:43.153 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.152904+0000", 2024-03-20T09:06:43.153 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.153 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.153 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.154 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:06:43.154 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.152904+0000", 2024-03-20T09:06:43.154 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.154 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.154 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.154 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:06:43.154 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.152904+0000", 2024-03-20T09:06:43.154 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9400000000000001e-07 2024-03-20T09:06:43.154 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.154 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.154 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:06:43.154 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.152909+0000", 2024-03-20T09:06:43.154 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9350000000000002e-06 2024-03-20T09:06:43.154 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.155 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.155 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:06:43.155 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.152915+0000", 2024-03-20T09:06:43.155 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2400000000000004e-06 2024-03-20T09:06:43.155 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.155 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.155 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:06:43.155 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.152920+0000", 2024-03-20T09:06:43.155 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4190000000000004e-06 2024-03-20T09:06:43.155 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.155 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.155 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:06:43.155 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.193711+0000", 2024-03-20T09:06:43.155 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.040791486000000002 2024-03-20T09:06:43.156 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.156 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.156 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:06:43.156 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.193796+0000", 2024-03-20T09:06:43.156 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5193999999999998e-05 2024-03-20T09:06:43.156 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.156 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.156 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,4", 2024-03-20T09:06:43.156 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.193871+0000", 2024-03-20T09:06:43.156 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4721999999999994e-05 2024-03-20T09:06:43.156 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.156 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.156 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:06:43.156 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.202194+0000", 2024-03-20T09:06:43.157 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0083231139999999995 2024-03-20T09:06:43.157 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.157 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.157 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:06:43.157 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.202382+0000", 2024-03-20T09:06:43.157 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00018747999999999999 2024-03-20T09:06:43.157 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.157 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.157 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:06:43.157 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.211366+0000", 2024-03-20T09:06:43.157 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0089842640000000005 2024-03-20T09:06:43.157 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.157 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.157 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:06:43.157 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.211426+0000", 2024-03-20T09:06:43.158 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9571999999999998e-05 2024-03-20T09:06:43.158 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.158 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.158 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:06:43.158 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.211475+0000", 2024-03-20T09:06:43.158 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9101999999999998e-05 2024-03-20T09:06:43.158 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.158 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:06:43.158 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.158 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.158 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.158 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:178113 2.d 2:baf67913:::benchmark_data_smithi122_107654_object59328:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e173)", 2024-03-20T09:06:43.158 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:01:24.152926+0000", 2024-03-20T09:06:43.158 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 318.89632500599998, 2024-03-20T09:06:43.159 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.058693822, 2024-03-20T09:06:43.159 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:06:43.159 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:06:43.159 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:06:43.159 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:06:43.159 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:06:43.159 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 178113 2024-03-20T09:06:43.159 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.159 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:06:43.159 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.159 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:06:43.159 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.152926+0000", 2024-03-20T09:06:43.159 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.159 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.160 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.160 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:06:43.160 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.152926+0000", 2024-03-20T09:06:43.160 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.160 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.160 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.160 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:06:43.160 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.152926+0000", 2024-03-20T09:06:43.160 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0800000000000001e-07 2024-03-20T09:06:43.160 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.160 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.160 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:06:43.160 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.152928+0000", 2024-03-20T09:06:43.160 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0420000000000001e-06 2024-03-20T09:06:43.160 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.161 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.161 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:06:43.161 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.152934+0000", 2024-03-20T09:06:43.161 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0079999999999999e-06 2024-03-20T09:06:43.161 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.161 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.161 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:06:43.161 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.152940+0000", 2024-03-20T09:06:43.161 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8760000000000002e-06 2024-03-20T09:06:43.161 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.161 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.161 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:06:43.161 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.197411+0000", 2024-03-20T09:06:43.161 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.044470717999999999 2024-03-20T09:06:43.162 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.162 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.162 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:06:43.162 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.197541+0000", 2024-03-20T09:06:43.162 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012976700000000001 2024-03-20T09:06:43.162 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.162 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.162 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,4", 2024-03-20T09:06:43.162 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.197679+0000", 2024-03-20T09:06:43.162 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013859899999999999 2024-03-20T09:06:43.162 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.162 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.162 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:06:43.162 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.202658+0000", 2024-03-20T09:06:43.163 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0049783609999999997 2024-03-20T09:06:43.163 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.163 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.163 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:06:43.163 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.202855+0000", 2024-03-20T09:06:43.163 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00019715600000000001 2024-03-20T09:06:43.163 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.163 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.163 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:06:43.163 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.211531+0000", 2024-03-20T09:06:43.163 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0086757429999999996 2024-03-20T09:06:43.163 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.163 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.163 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:06:43.164 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.211583+0000", 2024-03-20T09:06:43.164 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2024999999999997e-05 2024-03-20T09:06:43.164 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.164 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.164 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:06:43.164 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.211620+0000", 2024-03-20T09:06:43.164 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7218999999999997e-05 2024-03-20T09:06:43.164 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.164 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:06:43.164 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.164 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.164 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.164 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:178137 2.d 2:b8f946e7:::benchmark_data_smithi122_107654_object59352:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-03-20T09:06:43.164 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:01:24.164598+0000", 2024-03-20T09:06:43.164 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 318.88465263199998, 2024-03-20T09:06:43.165 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.057207959000000003, 2024-03-20T09:06:43.165 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:06:43.165 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:06:43.165 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:06:43.165 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:06:43.165 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:06:43.165 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 178137 2024-03-20T09:06:43.165 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.165 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:06:43.165 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.165 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:06:43.165 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.164598+0000", 2024-03-20T09:06:43.165 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.165 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.166 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.166 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:06:43.166 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.164598+0000", 2024-03-20T09:06:43.166 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.166 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.166 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.166 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:06:43.166 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.164600+0000", 2024-03-20T09:06:43.166 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.373e-06 2024-03-20T09:06:43.166 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.166 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.166 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:06:43.166 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.164607+0000", 2024-03-20T09:06:43.166 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8170000000000001e-06 2024-03-20T09:06:43.167 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.167 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.167 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:06:43.167 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.164620+0000", 2024-03-20T09:06:43.167 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3216e-05 2024-03-20T09:06:43.167 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.167 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.167 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:06:43.167 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.164628+0000", 2024-03-20T09:06:43.167 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1130000000000001e-06 2024-03-20T09:06:43.167 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.167 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.167 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:06:43.167 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.202906+0000", 2024-03-20T09:06:43.168 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.038277799000000001 2024-03-20T09:06:43.168 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.168 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.168 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:06:43.168 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.203018+0000", 2024-03-20T09:06:43.168 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011186100000000001 2024-03-20T09:06:43.168 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.168 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.168 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,4", 2024-03-20T09:06:43.168 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.203130+0000", 2024-03-20T09:06:43.168 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000112471 2024-03-20T09:06:43.168 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.168 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.168 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:06:43.169 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.211630+0000", 2024-03-20T09:06:43.169 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0085001989999999999 2024-03-20T09:06:43.169 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.169 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.169 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:06:43.169 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.211685+0000", 2024-03-20T09:06:43.169 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5220000000000003e-05 2024-03-20T09:06:43.169 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.169 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.169 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:06:43.169 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.221698+0000", 2024-03-20T09:06:43.169 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010012539000000001 2024-03-20T09:06:43.169 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.170 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.170 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:06:43.170 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.221753+0000", 2024-03-20T09:06:43.170 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5284000000000001e-05 2024-03-20T09:06:43.170 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.170 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.170 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:06:43.170 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.221806+0000", 2024-03-20T09:06:43.170 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3066999999999999e-05 2024-03-20T09:06:43.170 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.170 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:06:43.170 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.170 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.170 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.171 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:178144 2.d 2:bd620457:::benchmark_data_smithi122_107654_object59359:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-03-20T09:06:43.171 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:01:24.171531+0000", 2024-03-20T09:06:43.171 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 318.877720469, 2024-03-20T09:06:43.171 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.050437404999999998, 2024-03-20T09:06:43.171 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:06:43.171 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:06:43.171 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:06:43.171 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:06:43.171 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:06:43.171 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 178144 2024-03-20T09:06:43.171 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.171 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:06:43.171 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.171 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:06:43.171 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.171531+0000", 2024-03-20T09:06:43.172 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.172 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.172 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.172 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:06:43.172 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.171531+0000", 2024-03-20T09:06:43.172 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:43.172 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.172 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.172 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:06:43.172 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.171532+0000", 2024-03-20T09:06:43.172 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9099999999999991e-07 2024-03-20T09:06:43.172 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.172 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.172 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:06:43.173 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.171537+0000", 2024-03-20T09:06:43.173 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0039999999999999e-06 2024-03-20T09:06:43.173 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.173 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.173 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:06:43.173 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.171547+0000", 2024-03-20T09:06:43.173 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0723999999999999e-05 2024-03-20T09:06:43.173 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.173 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.173 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:06:43.173 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.171555+0000", 2024-03-20T09:06:43.173 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7859999999999997e-06 2024-03-20T09:06:43.173 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.173 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.174 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:06:43.174 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.207672+0000", 2024-03-20T09:06:43.174 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.036117345000000002 2024-03-20T09:06:43.174 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.174 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.174 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:06:43.174 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.207803+0000", 2024-03-20T09:06:43.174 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000131073 2024-03-20T09:06:43.174 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.174 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.174 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,4", 2024-03-20T09:06:43.174 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.207924+0000", 2024-03-20T09:06:43.174 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012050800000000001 2024-03-20T09:06:43.174 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.175 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.175 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:06:43.175 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.216114+0000", 2024-03-20T09:06:43.175 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0081903879999999998 2024-03-20T09:06:43.175 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.175 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.175 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:06:43.175 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.219483+0000", 2024-03-20T09:06:43.175 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003368443 2024-03-20T09:06:43.175 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.175 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.175 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:06:43.175 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.221864+0000", 2024-03-20T09:06:43.175 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002380858 2024-03-20T09:06:43.176 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.176 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.176 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:06:43.176 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.221922+0000", 2024-03-20T09:06:43.176 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8613999999999998e-05 2024-03-20T09:06:43.176 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:43.176 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:43.176 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:06:43.176 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:24.221968+0000", 2024-03-20T09:06:43.176 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5670999999999997e-05 2024-03-20T09:06:43.176 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.176 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:06:43.176 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.176 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:43.176 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:06:43.177 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:06:43.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:43.178 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:06:43.178 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:06:43.179 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:06:43.179 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:06:43.179 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:06:43.179 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:06:43.179 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:06:43.179 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:06:43.179 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:06:43.179 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-20T09:06:43.179 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:06:43.179 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:06:43.180 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:06:43.180 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:06:43.180 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:06:43.180 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:06:43.180 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:06:43.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:06:43.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:42 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:06:43.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:42 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:06:43.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:42 smithi122 ceph-mon[73495]: pgmap v834: 47 pgs: 4 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 24 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 5.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 106 op/s; 34604/245148 objects degraded (14.116%); 29576/245148 objects misplaced (12.065%); 28 MiB/s, 451 objects/s recovering 2024-03-20T09:06:43.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:42 smithi122 ceph-mon[73495]: osdmap e332: 12 total, 11 up, 10 in 2024-03-20T09:06:43.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:42 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3714995871' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:06:43.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:42 smithi026 ceph-mon[68309]: pgmap v834: 47 pgs: 4 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 24 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 5.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 106 op/s; 34604/245148 objects degraded (14.116%); 29576/245148 objects misplaced (12.065%); 28 MiB/s, 451 objects/s recovering 2024-03-20T09:06:43.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:42 smithi026 ceph-mon[68309]: osdmap e332: 12 total, 11 up, 10 in 2024-03-20T09:06:43.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:42 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3714995871' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:06:43.281 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:06:43.309 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:43.332 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:06:43.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:43.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:06:43.435 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:06:43.464 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:43.483 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:06:43.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:43.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:06:43.587 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:06:43.623 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:43.662 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:06:43.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:43.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:06:43.766 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:06:43.794 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:43.817 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:06:43.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:43.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:06:43.890 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_ops_in_flight 2024-03-20T09:06:43.921 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:06:43.962 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:43.986 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:06:43.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:43.989 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:06:44.090 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:06:44.119 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:44.141 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:06:44.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:44.145 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:06:44.187 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:06:44.246 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:06:44.274 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:44.472 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:06:44.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:44.475 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:06:44.576 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:06:44.607 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:44.631 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:06:44.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:44.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:06:44.736 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:06:44.766 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:45.117 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:06:45.119 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:44 smithi122 ceph-mon[73495]: pgmap v836: 47 pgs: 4 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 24 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 5.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 105 op/s; 34582/244596 objects degraded (14.138%); 29030/244596 objects misplaced (11.869%); 29 MiB/s, 458 objects/s recovering 2024-03-20T09:06:45.119 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:44 smithi122 ceph-mon[73495]: osdmap e333: 12 total, 11 up, 10 in 2024-03-20T09:06:45.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:45.120 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:06:45.221 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:06:45.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:44 smithi026 ceph-mon[68309]: pgmap v836: 47 pgs: 4 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 24 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 5.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 105 op/s; 34582/244596 objects degraded (14.138%); 29030/244596 objects misplaced (11.869%); 29 MiB/s, 458 objects/s recovering 2024-03-20T09:06:45.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:44 smithi026 ceph-mon[68309]: osdmap e333: 12 total, 11 up, 10 in 2024-03-20T09:06:45.266 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:45.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:44 smithi115 ceph-mon[72476]: pgmap v836: 47 pgs: 4 active+clean+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 24 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 5.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 105 op/s; 34582/244596 objects degraded (14.138%); 29030/244596 objects misplaced (11.869%); 29 MiB/s, 458 objects/s recovering 2024-03-20T09:06:45.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:44 smithi115 ceph-mon[72476]: osdmap e333: 12 total, 11 up, 10 in 2024-03-20T09:06:45.523 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:06:45.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:45.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:06:45.633 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:06:45.662 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:45.681 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:06:45.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:45.684 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:06:45.785 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:06:45.820 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:45.977 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:06:46.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:45 smithi026 ceph-mon[68309]: osdmap e334: 12 total, 11 up, 10 in 2024-03-20T09:06:46.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:45 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3538961923' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:06:46.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:45 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:06:46.298 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:06:46.298 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:06:46.318 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:45 smithi122 ceph-mon[73495]: osdmap e334: 12 total, 11 up, 10 in 2024-03-20T09:06:46.318 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:45 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3538961923' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:06:46.318 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:45 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:06:46.354 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:06:46.355 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:06:46.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:45 smithi115 ceph-mon[72476]: osdmap e334: 12 total, 11 up, 10 in 2024-03-20T09:06:46.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:45 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3538961923' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:06:46.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:45 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:06:46.672 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:06:46.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:46.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:06:46.776 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:06:46.805 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:46.826 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:06:46.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:46.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:06:46.931 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:06:46.977 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:47.101 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_blocked_ops 2024-03-20T09:06:47.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:46 smithi115 ceph-mon[72476]: pgmap v839: 47 pgs: 4 active+clean+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 24 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 152 op/s; 34465/243216 objects degraded (14.171%); 26823/243216 objects misplaced (11.028%); 46 MiB/s, 754 objects/s recovering 2024-03-20T09:06:47.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:46 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:06:47.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:46 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:06:47.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:46 smithi115 ceph-mon[72476]: osdmap e335: 12 total, 11 up, 10 in 2024-03-20T09:06:47.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:46 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 34465/243216 objects degraded (14.171%), 12 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:06:47.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:46 smithi026 ceph-mon[68309]: pgmap v839: 47 pgs: 4 active+clean+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 24 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 152 op/s; 34465/243216 objects degraded (14.171%); 26823/243216 objects misplaced (11.028%); 46 MiB/s, 754 objects/s recovering 2024-03-20T09:06:47.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:46 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:06:47.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:46 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:06:47.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:46 smithi026 ceph-mon[68309]: osdmap e335: 12 total, 11 up, 10 in 2024-03-20T09:06:47.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:46 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 34465/243216 objects degraded (14.171%), 12 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:06:47.400 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:06:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:46 smithi122 ceph-mon[73495]: pgmap v839: 47 pgs: 4 active+clean+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 24 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 152 op/s; 34465/243216 objects degraded (14.171%); 26823/243216 objects misplaced (11.028%); 46 MiB/s, 754 objects/s recovering 2024-03-20T09:06:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:46 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:06:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:46 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:06:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:46 smithi122 ceph-mon[73495]: osdmap e335: 12 total, 11 up, 10 in 2024-03-20T09:06:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:46 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 34465/243216 objects degraded (14.171%), 12 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:06:47.941 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:06:47.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:47.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:06:47.984 INFO:teuthology.orchestra.run.smithi026.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-03-20T09:06:48.046 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:06:48.073 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:48.097 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:06:48.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:48.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:06:48.202 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:06:48.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:47 smithi026 ceph-mon[68309]: osdmap e336: 12 total, 11 up, 10 in 2024-03-20T09:06:48.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:47 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3773671148' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:06:48.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:47 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:06:48.228 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:47 smithi122 ceph-mon[73495]: osdmap e336: 12 total, 11 up, 10 in 2024-03-20T09:06:48.228 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:47 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3773671148' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:06:48.228 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:47 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:06:48.231 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:48.274 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:06:48.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:48.276 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:06:48.356 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:06:48.377 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:06:48.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:47 smithi115 ceph-mon[72476]: osdmap e336: 12 total, 11 up, 10 in 2024-03-20T09:06:48.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:47 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3773671148' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:06:48.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:47 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:06:48.406 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:48.428 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:06:48.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:48.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:06:48.533 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:06:48.560 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:48.912 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:06:48.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:48.928 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:06:49.029 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:06:49.063 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:49.148 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:48 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:06:49.148 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:48 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:06:49.148 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:48 smithi122 ceph-mon[73495]: pgmap v842: 47 pgs: 1 creating+remapped+peering, 4 active+clean+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 25 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 164 op/s; 31798/242622 objects degraded (13.106%); 25572/242622 objects misplaced (10.540%); 45 MiB/s, 966 objects/s recovering 2024-03-20T09:06:49.148 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:48 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-03-20T09:06:49.148 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:48 smithi122 ceph-mon[73495]: osdmap e337: 12 total, 11 up, 10 in 2024-03-20T09:06:49.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:48 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:06:49.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:48 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:06:49.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:48 smithi115 ceph-mon[72476]: pgmap v842: 47 pgs: 1 creating+remapped+peering, 4 active+clean+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 25 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 164 op/s; 31798/242622 objects degraded (13.106%); 25572/242622 objects misplaced (10.540%); 45 MiB/s, 966 objects/s recovering 2024-03-20T09:06:49.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:48 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-03-20T09:06:49.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:48 smithi115 ceph-mon[72476]: osdmap e337: 12 total, 11 up, 10 in 2024-03-20T09:06:49.450 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:06:49.450 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:06:49.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:48 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:06:49.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:48 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:06:49.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:48 smithi026 ceph-mon[68309]: pgmap v842: 47 pgs: 1 creating+remapped+peering, 4 active+clean+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 25 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 164 op/s; 31798/242622 objects degraded (13.106%); 25572/242622 objects misplaced (10.540%); 45 MiB/s, 966 objects/s recovering 2024-03-20T09:06:49.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:48 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-03-20T09:06:49.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:48 smithi026 ceph-mon[68309]: osdmap e337: 12 total, 11 up, 10 in 2024-03-20T09:06:49.524 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:06:50.317 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:06:50.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:50.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:06:50.365 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_historic_ops 2024-03-20T09:06:50.424 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:06:50.454 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:50.659 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:06:51.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:50 smithi122 ceph-mon[73495]: pgmap v844: 47 pgs: 1 creating+remapped+peering, 4 active+clean+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 25 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.9 GiB data, 18 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 142 op/s; 31691/242055 objects degraded (13.092%); 25391/242055 objects misplaced (10.490%); 39 MiB/s, 839 objects/s recovering 2024-03-20T09:06:51.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:50 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:06:51.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:50 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:06:51.217 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:50 smithi026 ceph-mon[68309]: pgmap v844: 47 pgs: 1 creating+remapped+peering, 4 active+clean+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 25 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.9 GiB data, 18 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 142 op/s; 31691/242055 objects degraded (13.092%); 25391/242055 objects misplaced (10.490%); 39 MiB/s, 839 objects/s recovering 2024-03-20T09:06:51.217 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:50 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:06:51.217 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:50 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:06:51.278 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:06:51.279 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:06:51.305 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:06:51.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:51.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:06:51.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:50 smithi115 ceph-mon[72476]: pgmap v844: 47 pgs: 1 creating+remapped+peering, 4 active+clean+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 25 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.9 GiB data, 18 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 142 op/s; 31691/242055 objects degraded (13.092%); 25391/242055 objects misplaced (10.490%); 39 MiB/s, 839 objects/s recovering 2024-03-20T09:06:51.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:50 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:06:51.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:50 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:06:51.411 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:06:51.443 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:51.497 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:06:51.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:51.500 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:06:51.601 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:06:51.628 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:51.727 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:06:51.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:51.732 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:06:51.833 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:06:51.860 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:51.882 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:06:51.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:51.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:06:51.985 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:06:52.023 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:52.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:51 smithi115 ceph-mon[72476]: osdmap e338: 12 total, 11 up, 10 in 2024-03-20T09:06:52.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:51 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 31691/242055 objects degraded (13.092%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:06:52.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:51 smithi115 ceph-mon[72476]: osdmap e339: 12 total, 11 up, 10 in 2024-03-20T09:06:52.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:51 smithi026 ceph-mon[68309]: osdmap e338: 12 total, 11 up, 10 in 2024-03-20T09:06:52.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:51 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 31691/242055 objects degraded (13.092%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:06:52.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:51 smithi026 ceph-mon[68309]: osdmap e339: 12 total, 11 up, 10 in 2024-03-20T09:06:52.441 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:51 smithi122 ceph-mon[73495]: osdmap e338: 12 total, 11 up, 10 in 2024-03-20T09:06:52.441 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:51 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 31691/242055 objects degraded (13.092%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:06:52.442 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:51 smithi122 ceph-mon[73495]: osdmap e339: 12 total, 11 up, 10 in 2024-03-20T09:06:52.679 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:06:52.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:52.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:06:52.752 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:06:52.752 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:06:52.786 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:06:52.823 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:53.149 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:52 smithi122 ceph-mon[73495]: pgmap v846: 47 pgs: 1 creating+remapped+peering, 4 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 25 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 118 op/s; 28775/241182 objects degraded (11.931%); 26676/241182 objects misplaced (11.061%); 27 MiB/s, 578 objects/s recovering 2024-03-20T09:06:53.328 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:52 smithi026 ceph-mon[68309]: pgmap v846: 47 pgs: 1 creating+remapped+peering, 4 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 25 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 118 op/s; 28775/241182 objects degraded (11.931%); 26676/241182 objects misplaced (11.061%); 27 MiB/s, 578 objects/s recovering 2024-03-20T09:06:53.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:52 smithi115 ceph-mon[72476]: pgmap v846: 47 pgs: 1 creating+remapped+peering, 4 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 25 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.9 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 118 op/s; 28775/241182 objects degraded (11.931%); 26676/241182 objects misplaced (11.061%); 27 MiB/s, 578 objects/s recovering 2024-03-20T09:06:54.627 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:06:54.627 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:06:54.664 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:06:54.703 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:06:54.742 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:06:54.765 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:06:54.789 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:06:54.814 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:06:54.838 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:06:54.862 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:06:54.895 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:06:54.896 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:06:54.896 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:06:54.896 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:06:54.896 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:06:54.896 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:06:54.896 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-20T09:06:54.896 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:06:54.896 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:06:54.897 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:06:54.897 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:06:54.897 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:06:54.897 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:06:55.002 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:06:55.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:55 smithi026 ceph-mon[68309]: pgmap v848: 47 pgs: 5 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 25 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 112 op/s; 28763/240597 objects degraded (11.955%); 28470/240597 objects misplaced (11.833%); 21 MiB/s, 341 objects/s recovering 2024-03-20T09:06:55.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:55 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1280489318' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:06:55.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:55 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2192091430' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:06:55.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:55 smithi115 ceph-mon[72476]: pgmap v848: 47 pgs: 5 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 25 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 112 op/s; 28763/240597 objects degraded (11.955%); 28470/240597 objects misplaced (11.833%); 21 MiB/s, 341 objects/s recovering 2024-03-20T09:06:55.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:55 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1280489318' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:06:55.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:55 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2192091430' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:06:55.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:55 smithi122 ceph-mon[73495]: pgmap v848: 47 pgs: 5 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 25 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 112 op/s; 28763/240597 objects degraded (11.955%); 28470/240597 objects misplaced (11.833%); 21 MiB/s, 341 objects/s recovering 2024-03-20T09:06:55.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:55 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1280489318' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:06:55.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:55 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2192091430' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:06:55.855 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_ops_in_flight 2024-03-20T09:06:55.913 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:06:55.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:55.917 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:06:56.018 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:06:56.057 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:56.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:56 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:06:56.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:56 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2192091430' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:06:56.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:56 smithi122 ceph-mon[73495]: osdmap e340: 12 total, 11 up, 10 in 2024-03-20T09:06:56.202 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:06:56.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:56 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:06:56.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:56 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2192091430' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:06:56.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:56 smithi026 ceph-mon[68309]: osdmap e340: 12 total, 11 up, 10 in 2024-03-20T09:06:56.256 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7, 8, 10, 11] out_osds: [9, 3] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11] 2024-03-20T09:06:56.256 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:06:56.256 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-20T09:06:56.257 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:06:56.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:56 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:06:56.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:56 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2192091430' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:06:56.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:56 smithi115 ceph-mon[72476]: osdmap e340: 12 total, 11 up, 10 in 2024-03-20T09:06:56.469 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:06:56.469 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:06:56.509 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:06:56.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:56.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:06:56.575 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:06:56.614 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:06:56.646 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:56.670 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:06:56.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:56.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:06:56.774 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:06:56.805 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:57.161 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:06:57.163 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:57 smithi122 ceph-mon[73495]: pgmap v850: 47 pgs: 5 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 25 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.9 GiB data, 18 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 141 op/s; 28668/239508 objects degraded (11.970%); 26861/239508 objects misplaced (11.215%); 36 MiB/s, 579 objects/s recovering 2024-03-20T09:06:57.163 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:57 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 28668/239508 objects degraded (11.970%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:06:57.163 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:57 smithi122 ceph-mon[73495]: osdmap e341: 12 total, 11 up, 10 in 2024-03-20T09:06:57.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:57.164 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:06:57.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:57 smithi026 ceph-mon[68309]: pgmap v850: 47 pgs: 5 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 25 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.9 GiB data, 18 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 141 op/s; 28668/239508 objects degraded (11.970%); 26861/239508 objects misplaced (11.215%); 36 MiB/s, 579 objects/s recovering 2024-03-20T09:06:57.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:57 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 28668/239508 objects degraded (11.970%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:06:57.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:57 smithi026 ceph-mon[68309]: osdmap e341: 12 total, 11 up, 10 in 2024-03-20T09:06:57.266 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:06:57.299 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:57.400 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:06:57.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:57.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:06:57.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:57 smithi115 ceph-mon[72476]: pgmap v850: 47 pgs: 5 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 25 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.9 GiB data, 18 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 141 op/s; 28668/239508 objects degraded (11.970%); 26861/239508 objects misplaced (11.215%); 36 MiB/s, 579 objects/s recovering 2024-03-20T09:06:57.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:57 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 28668/239508 objects degraded (11.970%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:06:57.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:57 smithi115 ceph-mon[72476]: osdmap e341: 12 total, 11 up, 10 in 2024-03-20T09:06:57.504 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:06:57.533 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:57.557 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:06:57.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:57.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:06:57.663 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:06:57.693 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:57.913 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:06:58.046 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:06:58.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:58.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:06:58.153 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:06:58.188 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:58.224 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:06:58.225 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-03-20T09:06:58.225 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:58.225 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:120302 3.f 3.9a27822f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-03-20T09:06:58.225 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:06:58.141873+0000", 2024-03-20T09:06:58.225 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.082406965999999998, 2024-03-20T09:06:58.225 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.082452322999999994, 2024-03-20T09:06:58.225 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:06:58.225 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "waiting for peered", 2024-03-20T09:06:58.225 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:06:58.225 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:06:58.225 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:06:58.225 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 120302 2024-03-20T09:06:58.226 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:58.226 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:06:58.226 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:58.226 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:06:58.226 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.141873+0000", 2024-03-20T09:06:58.226 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:58.226 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:58.226 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:58.226 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:06:58.226 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.141873+0000", 2024-03-20T09:06:58.226 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:06:58.226 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:58.226 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:58.227 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:06:58.227 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.141874+0000", 2024-03-20T09:06:58.227 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3319999999999999e-06 2024-03-20T09:06:58.227 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:58.227 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:58.227 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:06:58.227 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.141880+0000", 2024-03-20T09:06:58.227 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3619999999999997e-06 2024-03-20T09:06:58.227 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:58.227 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:58.227 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:06:58.227 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.141892+0000", 2024-03-20T09:06:58.227 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2704000000000001e-05 2024-03-20T09:06:58.227 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:58.228 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:58.228 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:06:58.228 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.141899+0000", 2024-03-20T09:06:58.228 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6919999999999996e-06 2024-03-20T09:06:58.228 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:58.228 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:58.228 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:06:58.228 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.141969+0000", 2024-03-20T09:06:58.228 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9597000000000005e-05 2024-03-20T09:06:58.228 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:06:58.228 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:06:58.228 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:06:58.228 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.141981+0000", 2024-03-20T09:06:58.228 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2658e-05 2024-03-20T09:06:58.229 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:58.229 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:06:58.229 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:58.229 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:06:58.229 INFO:teuthology.orchestra.run.smithi122.stdout: ], 2024-03-20T09:06:58.229 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 1 2024-03-20T09:06:58.229 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:06:58.316 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:06:58.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:58.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:06:58.423 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:06:58.455 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:58.470 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:06:58.559 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:06:58.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:58.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:06:58.598 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:06:58.663 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:06:58.695 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:58.860 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:06:58.862 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:06:58.940 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_blocked_ops 2024-03-20T09:06:58.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:58.947 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:06:58.947 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:06:59.048 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:06:59.082 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:59.105 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:06:59.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:06:59.108 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:06:59.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:59 smithi122 ceph-mon[73495]: pgmap v852: 47 pgs: 5 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 25 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 126 op/s; 28668/238899 objects degraded (12.000%); 25933/238899 objects misplaced (10.855%); 34 MiB/s, 635 objects/s recovering 2024-03-20T09:06:59.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:59 smithi122 ceph-mon[73495]: 3.9 scrub starts 2024-03-20T09:06:59.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:59 smithi122 ceph-mon[73495]: osdmap e342: 12 total, 11 up, 10 in 2024-03-20T09:06:59.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:59 smithi122 ceph-mon[73495]: 1.6 scrub starts 2024-03-20T09:06:59.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:59 smithi122 ceph-mon[73495]: 1.6 scrub ok 2024-03-20T09:06:59.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:06:59 smithi122 ceph-mon[73495]: 3.9 scrub ok 2024-03-20T09:06:59.209 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:06:59.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:59 smithi026 ceph-mon[68309]: pgmap v852: 47 pgs: 5 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 25 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 126 op/s; 28668/238899 objects degraded (12.000%); 25933/238899 objects misplaced (10.855%); 34 MiB/s, 635 objects/s recovering 2024-03-20T09:06:59.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:59 smithi026 ceph-mon[68309]: 3.9 scrub starts 2024-03-20T09:06:59.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:59 smithi026 ceph-mon[68309]: osdmap e342: 12 total, 11 up, 10 in 2024-03-20T09:06:59.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:59 smithi026 ceph-mon[68309]: 1.6 scrub starts 2024-03-20T09:06:59.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:59 smithi026 ceph-mon[68309]: 1.6 scrub ok 2024-03-20T09:06:59.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:06:59 smithi026 ceph-mon[68309]: 3.9 scrub ok 2024-03-20T09:06:59.246 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:06:59.252 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:06:59.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:59 smithi115 ceph-mon[72476]: pgmap v852: 47 pgs: 5 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 25 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 126 op/s; 28668/238899 objects degraded (12.000%); 25933/238899 objects misplaced (10.855%); 34 MiB/s, 635 objects/s recovering 2024-03-20T09:06:59.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:59 smithi115 ceph-mon[72476]: 3.9 scrub starts 2024-03-20T09:06:59.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:59 smithi115 ceph-mon[72476]: osdmap e342: 12 total, 11 up, 10 in 2024-03-20T09:06:59.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:59 smithi115 ceph-mon[72476]: 1.6 scrub starts 2024-03-20T09:06:59.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:59 smithi115 ceph-mon[72476]: 1.6 scrub ok 2024-03-20T09:06:59.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:06:59 smithi115 ceph-mon[72476]: 3.9 scrub ok 2024-03-20T09:06:59.981 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:06:59.982 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":853,"stamp":"2024-03-20T09:06:58.017339+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5219438638,"num_objects":79633,"num_object_clones":0,"num_object_copies":238899,"num_objects_missing_on_primary":5016,"num_objects_missing":5016,"num_objects_degraded":28668,"num_objects_misplaced":25933,"num_objects_unfound":0,"num_objects_dirty":79633,"num_whiteouts":0,"num_read":24,"num_read_kb":21,"num_write":90372,"num_write_kb":4509656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64394,"num_bytes_recovered":4006240153,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163643,"ondisk_log_size":163643,"up":125,"acting":125,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":158,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":18473864,"kb_used_data":16596060,"kb_used_omap":15,"kb_used_meta":1877680,"kb_avail":918977656,"statfs":{"total":959950356480,"available":941033119744,"internally_reserved":0,"allocated":16994365440,"data_stored":16988422107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15948,"internal_metadata":1922744756},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-41811968,"num_objects":-638,"num_object_clones":0,"num_object_copies":-1914,"num_objects_missing_on_primary":-43,"num_objects_missing":-43,"num_objects_degraded":-107,"num_objects_misplaced":25,"num_objects_unfound":0,"num_objects_dirty":-638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":638,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3044,"num_bytes_recovered":161873920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.085069"},"pg_stats":[{"pgid":"3.19","version":"339'5548","reported_seq":19545,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-20T09:06:54.421814+0000","last_change":"2024-03-20T09:05:36.733109+0000","last_active":"2024-03-20T09:06:54.421814+0000","last_peered":"2024-03-20T09:06:54.421814+0000","last_clean":"2024-03-20T09:06:54.421814+0000","last_became_active":"2024-03-20T09:05:36.725732+0000","last_became_peered":"2024-03-20T09:05:36.725732+0000","last_unstale":"2024-03-20T09:06:54.421814+0000","last_undegraded":"2024-03-20T09:06:54.421814+0000","last_fullsized":"2024-03-20T09:06:54.421814+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":5,"last_scrub":"284'4948","last_scrub_stamp":"2024-03-20T09:04:48.131969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:04:48.131969+0000","objects_scrubbed":3082,"log_size":5548,"log_dups_size":0,"ondisk_log_size":5548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":324,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162660352,"num_objects":2482,"num_object_clones":0,"num_object_copies":7446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":600,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3145,"num_bytes_recovered":184090624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,10],"acting":[1,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"20'137","reported_seq":401,"reported_epoch":339,"state":"active+clean+remapped","last_fresh":"2024-03-20T09:06:51.996334+0000","last_change":"2024-03-20T09:06:02.926582+0000","last_active":"2024-03-20T09:06:51.996334+0000","last_peered":"2024-03-20T09:06:51.996334+0000","last_clean":"2024-03-20T09:06:51.996334+0000","last_became_active":"2024-03-20T09:06:02.926039+0000","last_became_peered":"2024-03-20T09:06:02.926039+0000","last_unstale":"2024-03-20T09:06:51.996334+0000","last_undegraded":"2024-03-20T09:06:51.996334+0000","last_fullsized":"2024-03-20T09:06:51.996334+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:05:23.614560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:05:23.614560+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8,4],"avail_no_missing":["2","4","8"],"object_location_counts":[{"shards":"2,4,8","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"339'7558","reported_seq":17008,"reported_epoch":339,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-20T09:06:53.581712+0000","last_change":"2024-03-20T09:06:08.163861+0000","last_active":"2024-03-20T09:06:53.581712+0000","last_peered":"2024-03-20T09:06:53.581712+0000","last_clean":"2024-03-20T09:05:59.443016+0000","last_became_active":"2024-03-20T09:06:08.132694+0000","last_became_peered":"2024-03-20T09:06:08.132694+0000","last_unstale":"2024-03-20T09:06:53.581712+0000","last_undegraded":"2024-03-20T09:06:53.581712+0000","last_fullsized":"2024-03-20T09:06:07.921226+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":7558,"log_dups_size":0,"ondisk_log_size":7558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328073216,"num_objects":5006,"num_object_clones":0,"num_object_copies":15018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4665,"num_objects_unfound":0,"num_objects_dirty":5006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7558,"num_write_kb":402048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4573,"num_bytes_recovered":299696128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4],"avail_no_missing":["6","4","7","8"],"object_location_counts":[{"shards":"4,6","objects":341},{"shards":"4,6,7","objects":4665}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"20'137","reported_seq":713,"reported_epoch":339,"state":"active+undersized","last_fresh":"2024-03-20T09:06:51.995886+0000","last_change":"2024-03-20T09:04:34.736101+0000","last_active":"2024-03-20T09:06:51.995886+0000","last_peered":"2024-03-20T09:06:51.995886+0000","last_clean":"2024-03-20T09:04:29.198571+0000","last_became_active":"2024-03-20T09:04:34.736101+0000","last_became_peered":"2024-03-20T09:04:34.736101+0000","last_unstale":"2024-03-20T09:06:51.995886+0000","last_undegraded":"2024-03-20T09:06:51.995886+0000","last_fullsized":"2024-03-20T09:04:34.733152+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:03:56.726408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:03:56.726408+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:06:54.125750+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":16,"num_write_kb":267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":240422,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"340'7554","reported_seq":23219,"reported_epoch":340,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-20T09:06:55.381942+0000","last_change":"2024-03-20T09:06:00.902304+0000","last_active":"2024-03-20T09:06:55.381942+0000","last_peered":"2024-03-20T09:06:55.381942+0000","last_clean":"2024-03-20T09:05:18.895010+0000","last_became_active":"2024-03-20T09:06:00.863261+0000","last_became_peered":"2024-03-20T09:06:00.863261+0000","last_unstale":"2024-03-20T09:06:55.381942+0000","last_undegraded":"2024-03-20T09:06:55.381942+0000","last_fullsized":"2024-03-20T09:06:00.643399+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":7554,"log_dups_size":0,"ondisk_log_size":7554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334757888,"num_objects":5108,"num_object_clones":0,"num_object_copies":15324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1509,"num_objects_unfound":0,"num_objects_dirty":5108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7554,"num_write_kb":405184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7680,"num_bytes_recovered":503316480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,11],"acting":[4,11],"avail_no_missing":["4","3","11"],"object_location_counts":[{"shards":"3,4,11","objects":1509},{"shards":"4,11","objects":3599}],"blocked_by":[],"up_primary":0,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"20'137","reported_seq":512,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-20T09:06:51.950898+0000","last_change":"2024-03-20T09:03:13.525409+0000","last_active":"2024-03-20T09:06:51.950898+0000","last_peered":"2024-03-20T09:06:51.950898+0000","last_clean":"2024-03-20T09:06:51.950898+0000","last_became_active":"2024-03-20T09:03:07.875459+0000","last_became_peered":"2024-03-20T09:03:07.875459+0000","last_unstale":"2024-03-20T09:06:51.950898+0000","last_undegraded":"2024-03-20T09:06:51.950898+0000","last_fullsized":"2024-03-20T09:06:51.950898+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:03:13.525325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:03:13.525325+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"340'5596","reported_seq":5888,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-20T09:06:56.704952+0000","last_change":"2024-03-20T09:03:30.919404+0000","last_active":"2024-03-20T09:06:56.704952+0000","last_peered":"2024-03-20T09:06:56.704952+0000","last_clean":"2024-03-20T09:06:56.704952+0000","last_became_active":"2024-03-20T09:03:30.916325+0000","last_became_peered":"2024-03-20T09:03:30.916325+0000","last_unstale":"2024-03-20T09:06:56.704952+0000","last_undegraded":"2024-03-20T09:06:56.704952+0000","last_fullsized":"2024-03-20T09:06:56.704952+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":5596,"log_dups_size":0,"ondisk_log_size":5596,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:06:59.865574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163020800,"num_objects":2487,"num_object_clones":0,"num_object_copies":7461,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3831,"num_write_kb":202208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"20'137","reported_seq":683,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-20T09:06:55.001637+0000","last_change":"2024-03-20T09:06:02.927341+0000","last_active":"2024-03-20T09:06:55.001637+0000","last_peered":"2024-03-20T09:06:55.001637+0000","last_clean":"2024-03-20T09:06:55.001637+0000","last_became_active":"2024-03-20T09:06:02.926326+0000","last_became_peered":"2024-03-20T09:06:02.926326+0000","last_unstale":"2024-03-20T09:06:55.001637+0000","last_undegraded":"2024-03-20T09:06:55.001637+0000","last_fullsized":"2024-03-20T09:06:55.001637+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:24.397259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:24.397259+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:06:56.906358+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109226,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":109226,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.6","version":"339'5576","reported_seq":10448,"reported_epoch":339,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-20T09:06:54.949392+0000","last_change":"2024-03-20T09:06:45.380360+0000","last_active":"2024-03-20T09:06:54.949392+0000","last_peered":"2024-03-20T09:06:54.949392+0000","last_clean":"2024-03-20T09:05:59.303157+0000","last_became_active":"2024-03-20T09:06:01.863758+0000","last_became_peered":"2024-03-20T09:06:01.863758+0000","last_unstale":"2024-03-20T09:06:54.949392+0000","last_undegraded":"2024-03-20T09:06:01.677696+0000","last_fullsized":"2024-03-20T09:06:01.644829+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'4943","last_scrub_stamp":"2024-03-20T09:04:46.116541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:04:46.116541+0000","objects_scrubbed":3048,"log_size":5576,"log_dups_size":0,"ondisk_log_size":5576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":317,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158269440,"num_objects":2415,"num_object_clones":0,"num_object_copies":7245,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":186,"num_objects_misplaced":865,"num_objects_unfound":0,"num_objects_dirty":2415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":633,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1590,"num_bytes_recovered":104202240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[1,4],"avail_no_missing":["1","4","7"],"object_location_counts":[{"shards":"1,4","objects":1364},{"shards":"1,4,7","objects":865}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"20'137","reported_seq":582,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-20T09:06:54.997431+0000","last_change":"2024-03-20T09:03:59.909516+0000","last_active":"2024-03-20T09:06:54.997431+0000","last_peered":"2024-03-20T09:06:54.997431+0000","last_clean":"2024-03-20T09:06:54.997431+0000","last_became_active":"2024-03-20T09:03:58.987697+0000","last_became_peered":"2024-03-20T09:03:58.987697+0000","last_unstale":"2024-03-20T09:06:54.997431+0000","last_undegraded":"2024-03-20T09:06:54.997431+0000","last_fullsized":"2024-03-20T09:06:54.997431+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:03:59.909367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:03:59.909367+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.7","version":"340'5509","reported_seq":15259,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-20T09:06:56.597777+0000","last_change":"2024-03-20T09:06:30.957907+0000","last_active":"2024-03-20T09:06:56.597777+0000","last_peered":"2024-03-20T09:06:56.597777+0000","last_clean":"2024-03-20T09:06:56.597777+0000","last_became_active":"2024-03-20T09:06:30.951316+0000","last_became_peered":"2024-03-20T09:06:30.951316+0000","last_unstale":"2024-03-20T09:06:56.597777+0000","last_undegraded":"2024-03-20T09:06:56.597777+0000","last_fullsized":"2024-03-20T09:06:56.597777+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'4179","last_scrub_stamp":"2024-03-20T09:04:00.661196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:04:00.661196+0000","objects_scrubbed":2441,"log_size":5509,"log_dups_size":0,"ondisk_log_size":5509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163971072,"num_objects":2502,"num_object_clones":0,"num_object_copies":7506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1331,"num_write_kb":44544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3137,"num_bytes_recovered":176553984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"20'137","reported_seq":591,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-20T09:06:55.000696+0000","last_change":"2024-03-20T09:04:01.861749+0000","last_active":"2024-03-20T09:06:55.000696+0000","last_peered":"2024-03-20T09:06:55.000696+0000","last_clean":"2024-03-20T09:06:55.000696+0000","last_became_active":"2024-03-20T09:03:58.990557+0000","last_became_peered":"2024-03-20T09:03:58.990557+0000","last_unstale":"2024-03-20T09:06:55.000696+0000","last_undegraded":"2024-03-20T09:06:55.000696+0000","last_fullsized":"2024-03-20T09:06:55.000696+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:04:01.861603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:04:01.861603+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,4],"acting":[10,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.4","version":"339'5473","reported_seq":5770,"reported_epoch":339,"state":"active+undersized+degraded","last_fresh":"2024-03-20T09:06:54.749591+0000","last_change":"2024-03-20T09:04:34.736993+0000","last_active":"2024-03-20T09:06:54.749591+0000","last_peered":"2024-03-20T09:06:54.749591+0000","last_clean":"2024-03-20T09:04:33.698023+0000","last_became_active":"2024-03-20T09:04:34.736993+0000","last_became_peered":"2024-03-20T09:04:34.736993+0000","last_unstale":"2024-03-20T09:06:54.749591+0000","last_undegraded":"2024-03-20T09:04:34.735838+0000","last_fullsized":"2024-03-20T09:04:34.735696+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":199,"log_size":5473,"log_dups_size":0,"ondisk_log_size":5473,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164200448,"num_objects":2505,"num_object_clones":0,"num_object_copies":7515,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2505,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3735,"num_write_kb":199712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":["1","11"],"object_location_counts":[{"shards":"1,11","objects":2505}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"20'137","reported_seq":577,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-20T09:06:51.934080+0000","last_change":"2024-03-20T09:06:30.955732+0000","last_active":"2024-03-20T09:06:51.934080+0000","last_peered":"2024-03-20T09:06:51.934080+0000","last_clean":"2024-03-20T09:06:51.934080+0000","last_became_active":"2024-03-20T09:06:30.955286+0000","last_became_peered":"2024-03-20T09:06:30.955286+0000","last_unstale":"2024-03-20T09:06:51.934080+0000","last_undegraded":"2024-03-20T09:06:51.934080+0000","last_fullsized":"2024-03-20T09:06:51.934080+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:04:46.981050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:04:46.981050+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"340'8446","reported_seq":8901,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-03-20T09:06:56.934736+0000","last_change":"2024-03-20T09:04:57.536709+0000","last_active":"2024-03-20T09:06:56.934736+0000","last_peered":"2024-03-20T09:06:56.934736+0000","last_clean":"2024-03-20T09:06:56.934736+0000","last_became_active":"2024-03-20T09:03:30.921890+0000","last_became_peered":"2024-03-20T09:03:30.921890+0000","last_unstale":"2024-03-20T09:06:56.934736+0000","last_undegraded":"2024-03-20T09:06:56.934736+0000","last_fullsized":"2024-03-20T09:06:56.934736+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"285'7948","last_scrub_stamp":"2024-03-20T09:04:57.536605+0000","last_deep_scrub":"285'7948","last_deep_scrub_stamp":"2024-03-20T09:04:57.536605+0000","last_clean_scrub_stamp":"2024-03-20T09:04:57.536605+0000","objects_scrubbed":3080,"log_size":8446,"log_dups_size":0,"ondisk_log_size":8446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"queued for scrub","scrub_duration":5935,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167903232,"num_objects":2562,"num_object_clones":0,"num_object_copies":7686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1647,"num_write_kb":65920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,10],"acting":[0,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":654,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-20T09:06:55.001331+0000","last_change":"2024-03-20T09:06:02.926003+0000","last_active":"2024-03-20T09:06:55.001331+0000","last_peered":"2024-03-20T09:06:55.001331+0000","last_clean":"2024-03-20T09:06:55.001331+0000","last_became_active":"2024-03-20T09:06:02.925687+0000","last_became_peered":"2024-03-20T09:06:02.925687+0000","last_unstale":"2024-03-20T09:06:55.001331+0000","last_undegraded":"2024-03-20T09:06:55.001331+0000","last_fullsized":"2024-03-20T09:06:55.001331+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:23.383145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:23.383145+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:06:56.906342+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,10],"acting":[11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"339'5515","reported_seq":7134,"reported_epoch":339,"state":"active+undersized+degraded","last_fresh":"2024-03-20T09:06:53.420337+0000","last_change":"2024-03-20T09:04:34.738561+0000","last_active":"2024-03-20T09:06:53.420337+0000","last_peered":"2024-03-20T09:06:53.420337+0000","last_clean":"2024-03-20T09:04:33.645766+0000","last_became_active":"2024-03-20T09:04:34.738561+0000","last_became_peered":"2024-03-20T09:04:34.738561+0000","last_unstale":"2024-03-20T09:06:53.420337+0000","last_undegraded":"2024-03-20T09:04:34.734426+0000","last_fullsized":"2024-03-20T09:04:34.734302+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":5515,"log_dups_size":0,"ondisk_log_size":5515,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165412864,"num_objects":2524,"num_object_clones":0,"num_object_copies":7572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2524,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3740,"num_write_kb":200448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":329,"num_bytes_recovered":21561344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":2524}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"20'137","reported_seq":897,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-03-20T09:06:56.934835+0000","last_change":"2024-03-20T09:06:02.926788+0000","last_active":"2024-03-20T09:06:56.934835+0000","last_peered":"2024-03-20T09:06:56.934835+0000","last_clean":"2024-03-20T09:06:56.934835+0000","last_became_active":"2024-03-20T09:06:02.926378+0000","last_became_peered":"2024-03-20T09:06:02.926378+0000","last_unstale":"2024-03-20T09:06:56.934835+0000","last_undegraded":"2024-03-20T09:06:56.934835+0000","last_fullsized":"2024-03-20T09:06:56.934835+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:03:59.584956+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:03:59.584956+0000","last_clean_scrub_stamp":"2024-03-20T09:03:59.584956+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,8],"acting":[0,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"340'5446","reported_seq":7123,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-20T09:06:56.651381+0000","last_change":"2024-03-20T09:03:30.928051+0000","last_active":"2024-03-20T09:06:56.651381+0000","last_peered":"2024-03-20T09:06:56.651381+0000","last_clean":"2024-03-20T09:06:56.651381+0000","last_became_active":"2024-03-20T09:03:30.926683+0000","last_became_peered":"2024-03-20T09:03:30.926683+0000","last_unstale":"2024-03-20T09:06:56.651381+0000","last_undegraded":"2024-03-20T09:06:56.651381+0000","last_fullsized":"2024-03-20T09:06:56.651381+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":5446,"log_dups_size":0,"ondisk_log_size":5446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:06:59.865589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158334976,"num_objects":2416,"num_object_clones":0,"num_object_copies":7248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3702,"num_write_kb":195776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":343,"num_bytes_recovered":22511616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,1],"acting":[11,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1","version":"20'137","reported_seq":609,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-20T09:06:54.997242+0000","last_change":"2024-03-20T09:06:02.924071+0000","last_active":"2024-03-20T09:06:54.997242+0000","last_peered":"2024-03-20T09:06:54.997242+0000","last_clean":"2024-03-20T09:06:54.997242+0000","last_became_active":"2024-03-20T09:06:02.923630+0000","last_became_peered":"2024-03-20T09:06:02.923630+0000","last_unstale":"2024-03-20T09:06:54.997242+0000","last_undegraded":"2024-03-20T09:06:54.997242+0000","last_fullsized":"2024-03-20T09:06:54.997242+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:04:00.896754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:04:00.896754+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2],"acting":[10,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3","version":"340'5547","reported_seq":5838,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-20T09:06:55.214733+0000","last_change":"2024-03-20T09:03:30.939580+0000","last_active":"2024-03-20T09:06:55.214733+0000","last_peered":"2024-03-20T09:06:55.214733+0000","last_clean":"2024-03-20T09:06:55.214733+0000","last_became_active":"2024-03-20T09:03:30.937809+0000","last_became_peered":"2024-03-20T09:03:30.937809+0000","last_unstale":"2024-03-20T09:06:55.214733+0000","last_undegraded":"2024-03-20T09:06:55.214733+0000","last_fullsized":"2024-03-20T09:06:55.214733+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":5547,"log_dups_size":0,"ondisk_log_size":5547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:06:58.320982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159219712,"num_objects":2429,"num_object_clones":0,"num_object_copies":7287,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3753,"num_write_kb":197856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"20'137","reported_seq":582,"reported_epoch":339,"state":"active+undersized","last_fresh":"2024-03-20T09:06:51.941129+0000","last_change":"2024-03-20T09:04:34.737319+0000","last_active":"2024-03-20T09:06:51.941129+0000","last_peered":"2024-03-20T09:06:51.941129+0000","last_clean":"2024-03-20T09:04:29.191837+0000","last_became_active":"2024-03-20T09:04:34.737319+0000","last_became_peered":"2024-03-20T09:04:34.737319+0000","last_unstale":"2024-03-20T09:06:51.941129+0000","last_undegraded":"2024-03-20T09:06:51.941129+0000","last_fullsized":"2024-03-20T09:04:34.733375+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:21.734957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:21.734957+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:06:53.477637+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":["6","10"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"339'5531","reported_seq":7140,"reported_epoch":339,"state":"active+undersized+degraded","last_fresh":"2024-03-20T09:06:52.174701+0000","last_change":"2024-03-20T09:04:34.737379+0000","last_active":"2024-03-20T09:06:52.174701+0000","last_peered":"2024-03-20T09:06:52.174701+0000","last_clean":"2024-03-20T09:04:33.579072+0000","last_became_active":"2024-03-20T09:04:34.737379+0000","last_became_peered":"2024-03-20T09:04:34.737379+0000","last_unstale":"2024-03-20T09:06:52.174701+0000","last_undegraded":"2024-03-20T09:04:34.733476+0000","last_fullsized":"2024-03-20T09:04:34.733363+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"253'4200","last_scrub_stamp":"2024-03-20T09:04:00.025798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:04:00.025798+0000","objects_scrubbed":2428,"log_size":5531,"log_dups_size":0,"ondisk_log_size":5531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163905536,"num_objects":2501,"num_object_clones":0,"num_object_copies":7503,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2501,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1335,"num_write_kb":45056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,6","objects":2501}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"20'137","reported_seq":465,"reported_epoch":341,"state":"active+clean+remapped","last_fresh":"2024-03-20T09:06:56.934856+0000","last_change":"2024-03-20T09:06:02.922773+0000","last_active":"2024-03-20T09:06:56.934856+0000","last_peered":"2024-03-20T09:06:56.934856+0000","last_clean":"2024-03-20T09:06:56.934856+0000","last_became_active":"2024-03-20T09:06:02.922304+0000","last_became_peered":"2024-03-20T09:06:02.922304+0000","last_unstale":"2024-03-20T09:06:56.934856+0000","last_undegraded":"2024-03-20T09:06:56.934856+0000","last_fullsized":"2024-03-20T09:06:56.934856+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:24.936556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:24.936556+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:06:57.049994+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2,11],"avail_no_missing":["0","2","11"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"340'5447","reported_seq":5774,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-20T09:06:56.610471+0000","last_change":"2024-03-20T09:04:43.349194+0000","last_active":"2024-03-20T09:06:56.610471+0000","last_peered":"2024-03-20T09:06:56.610471+0000","last_clean":"2024-03-20T09:06:56.610471+0000","last_became_active":"2024-03-20T09:03:30.910402+0000","last_became_peered":"2024-03-20T09:03:30.910402+0000","last_unstale":"2024-03-20T09:06:56.610471+0000","last_undegraded":"2024-03-20T09:06:56.610471+0000","last_fullsized":"2024-03-20T09:06:56.610471+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'4850","last_scrub_stamp":"2024-03-20T09:04:43.349014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:04:43.349014+0000","objects_scrubbed":2998,"log_size":5447,"log_dups_size":0,"ondisk_log_size":5447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157351936,"num_objects":2401,"num_object_clones":0,"num_object_copies":7203,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":597,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"20'137","reported_seq":506,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-20T09:06:55.003241+0000","last_change":"2024-03-20T09:06:02.929720+0000","last_active":"2024-03-20T09:06:55.003241+0000","last_peered":"2024-03-20T09:06:55.003241+0000","last_clean":"2024-03-20T09:06:55.003241+0000","last_became_active":"2024-03-20T09:06:02.929364+0000","last_became_peered":"2024-03-20T09:06:02.929364+0000","last_unstale":"2024-03-20T09:06:55.003241+0000","last_undegraded":"2024-03-20T09:06:55.003241+0000","last_fullsized":"2024-03-20T09:06:55.003241+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:03:55.906959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:03:55.906959+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:06:59.297326+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,1],"acting":[4,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"339'11383","reported_seq":22961,"reported_epoch":339,"state":"active+clean+remapped","last_fresh":"2024-03-20T09:06:52.997193+0000","last_change":"2024-03-20T09:06:08.944363+0000","last_active":"2024-03-20T09:06:52.997193+0000","last_peered":"2024-03-20T09:06:52.997193+0000","last_clean":"2024-03-20T09:06:52.997193+0000","last_became_active":"2024-03-20T09:06:08.928778+0000","last_became_peered":"2024-03-20T09:06:08.928778+0000","last_unstale":"2024-03-20T09:06:52.997193+0000","last_undegraded":"2024-03-20T09:06:52.997193+0000","last_fullsized":"2024-03-20T09:06:52.997193+0000","mapping_epoch":313,"log_start":"221'1300","ondisk_log_start":"221'1300","created":204,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"270'9729","last_scrub_stamp":"2024-03-20T09:04:24.105978+0000","last_deep_scrub":"270'9729","last_deep_scrub_stamp":"2024-03-20T09:04:24.105978+0000","last_clean_scrub_stamp":"2024-03-20T09:04:24.105978+0000","objects_scrubbed":5255,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":26,"scrub_schedule":"queued for scrub","scrub_duration":25793,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331808768,"num_objects":5063,"num_object_clones":0,"num_object_copies":15189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5063,"num_objects_unfound":0,"num_objects_dirty":5063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7535,"num_write_kb":403136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5528,"num_bytes_recovered":362283008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10,3],"avail_no_missing":["6","3","10"],"object_location_counts":[{"shards":"3,6,10","objects":5063}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"20'137","reported_seq":589,"reported_epoch":339,"state":"active+undersized","last_fresh":"2024-03-20T09:06:51.941055+0000","last_change":"2024-03-20T09:04:34.737366+0000","last_active":"2024-03-20T09:06:51.941055+0000","last_peered":"2024-03-20T09:06:51.941055+0000","last_clean":"2024-03-20T09:04:29.191733+0000","last_became_active":"2024-03-20T09:04:34.737366+0000","last_became_peered":"2024-03-20T09:04:34.737366+0000","last_unstale":"2024-03-20T09:06:51.941055+0000","last_undegraded":"2024-03-20T09:06:51.941055+0000","last_fullsized":"2024-03-20T09:04:34.733742+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:21.734957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:21.734957+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:06:53.477657+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"339'7406","reported_seq":10202,"reported_epoch":339,"state":"active+recovering+degraded+remapped","last_fresh":"2024-03-20T09:06:53.871398+0000","last_change":"2024-03-20T09:04:47.242963+0000","last_active":"2024-03-20T09:06:53.871398+0000","last_peered":"2024-03-20T09:06:53.871398+0000","last_clean":"2024-03-20T09:04:10.072458+0000","last_became_active":"2024-03-20T09:04:46.620359+0000","last_became_peered":"2024-03-20T09:04:46.620359+0000","last_unstale":"2024-03-20T09:06:53.871398+0000","last_undegraded":"2024-03-20T09:04:46.424109+0000","last_fullsized":"2024-03-20T09:06:53.871398+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'5229","last_scrub_stamp":"2024-03-20T09:04:09.606259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:04:09.606259+0000","objects_scrubbed":5193,"log_size":7406,"log_dups_size":0,"ondisk_log_size":7406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325648408,"num_objects":4970,"num_object_clones":0,"num_object_copies":14910,"num_objects_missing_on_primary":4012,"num_objects_missing":4012,"num_objects_degraded":4970,"num_objects_misplaced":4012,"num_objects_unfound":0,"num_objects_dirty":4970,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7406,"num_write_kb":395969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1218,"num_bytes_recovered":79822848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7,9],"avail_no_missing":["7"],"object_location_counts":[{"shards":"6,7","objects":944},{"shards":"6,7,9","objects":1},{"shards":"7,9","objects":4012}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"20'137","reported_seq":640,"reported_epoch":341,"state":"active+undersized+degraded","last_fresh":"2024-03-20T09:06:56.934768+0000","last_change":"2024-03-20T09:04:34.740608+0000","last_active":"2024-03-20T09:06:56.934768+0000","last_peered":"2024-03-20T09:06:56.934768+0000","last_clean":"2024-03-20T09:04:29.200818+0000","last_became_active":"2024-03-20T09:04:34.740608+0000","last_became_peered":"2024-03-20T09:04:34.740608+0000","last_unstale":"2024-03-20T09:06:56.934768+0000","last_undegraded":"2024-03-20T09:04:34.735008+0000","last_fullsized":"2024-03-20T09:04:34.734889+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:04:04.682759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:04:04.682759+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[{"shards":"0,6","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"340'7471","reported_seq":24825,"reported_epoch":340,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-20T09:06:55.074801+0000","last_change":"2024-03-20T09:06:29.128475+0000","last_active":"2024-03-20T09:06:55.074801+0000","last_peered":"2024-03-20T09:06:55.074801+0000","last_clean":"2024-03-20T09:04:10.062034+0000","last_became_active":"2024-03-20T09:06:29.055447+0000","last_became_peered":"2024-03-20T09:06:29.055447+0000","last_unstale":"2024-03-20T09:06:55.074801+0000","last_undegraded":"2024-03-20T09:06:29.046703+0000","last_fullsized":"2024-03-20T09:06:28.937531+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":7471,"log_dups_size":0,"ondisk_log_size":7471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319356928,"num_objects":4873,"num_object_clones":0,"num_object_copies":14619,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7471,"num_write_kb":395008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8520,"num_bytes_recovered":558366720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,10],"acting":[4,10],"avail_no_missing":["4","9","11"],"object_location_counts":[{"shards":"3,4,9,10,11","objects":1386},{"shards":"3,4,11","objects":912},{"shards":"4,9,10,11","objects":324},{"shards":"4,9,11","objects":1455},{"shards":"4,10","objects":796}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"20'137","reported_seq":417,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-20T09:06:55.002881+0000","last_change":"2024-03-20T09:06:11.654156+0000","last_active":"2024-03-20T09:06:55.002881+0000","last_peered":"2024-03-20T09:06:55.002881+0000","last_clean":"2024-03-20T09:06:55.002881+0000","last_became_active":"2024-03-20T09:06:11.653699+0000","last_became_peered":"2024-03-20T09:06:11.653699+0000","last_unstale":"2024-03-20T09:06:55.002881+0000","last_undegraded":"2024-03-20T09:06:55.002881+0000","last_fullsized":"2024-03-20T09:06:55.002881+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:03:48.928433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:03:48.928433+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:06:58.321030+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":9,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":48085,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"340'7417","reported_seq":19387,"reported_epoch":340,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-20T09:06:56.761857+0000","last_change":"2024-03-20T09:06:01.949187+0000","last_active":"2024-03-20T09:06:56.761857+0000","last_peered":"2024-03-20T09:06:56.761857+0000","last_clean":"2024-03-20T09:05:59.447108+0000","last_became_active":"2024-03-20T09:06:01.895009+0000","last_became_peered":"2024-03-20T09:06:01.895009+0000","last_unstale":"2024-03-20T09:06:56.761857+0000","last_undegraded":"2024-03-20T09:06:01.696128+0000","last_fullsized":"2024-03-20T09:06:01.646709+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":7417,"log_dups_size":0,"ondisk_log_size":7417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320667648,"num_objects":4893,"num_object_clones":0,"num_object_copies":14679,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7417,"num_write_kb":393920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5840,"num_bytes_recovered":345112576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,10],"acting":[11,8],"avail_no_missing":["11","7","8"],"object_location_counts":[{"shards":"8,11","objects":4891}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.12","version":"20'137","reported_seq":839,"reported_epoch":339,"state":"active+undersized","last_fresh":"2024-03-20T09:06:51.933941+0000","last_change":"2024-03-20T09:04:34.751346+0000","last_active":"2024-03-20T09:06:51.933941+0000","last_peered":"2024-03-20T09:06:51.933941+0000","last_clean":"2024-03-20T09:03:59.001853+0000","last_became_active":"2024-03-20T09:04:34.751346+0000","last_became_peered":"2024-03-20T09:04:34.751346+0000","last_unstale":"2024-03-20T09:06:51.933941+0000","last_undegraded":"2024-03-20T09:06:51.933941+0000","last_fullsized":"2024-03-20T09:04:34.735354+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:04:05.319747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:04:05.319747+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"339'5466","reported_seq":18836,"reported_epoch":339,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-20T09:06:53.000286+0000","last_change":"2024-03-20T09:06:43.134508+0000","last_active":"2024-03-20T09:06:53.000286+0000","last_peered":"2024-03-20T09:06:53.000286+0000","last_clean":"2024-03-20T09:05:58.734145+0000","last_became_active":"2024-03-20T09:06:01.859835+0000","last_became_peered":"2024-03-20T09:06:01.859835+0000","last_unstale":"2024-03-20T09:06:53.000286+0000","last_undegraded":"2024-03-20T09:06:01.669030+0000","last_fullsized":"2024-03-20T09:06:01.645488+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":5466,"log_dups_size":0,"ondisk_log_size":5466,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162660352,"num_objects":2482,"num_object_clones":0,"num_object_copies":7446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2482,"num_objects_misplaced":2567,"num_objects_unfound":0,"num_objects_dirty":2482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3722,"num_write_kb":198528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5291,"num_bytes_recovered":344031232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,2],"avail_no_missing":["6","2","9"],"object_location_counts":[{"shards":"2,6","objects":-85},{"shards":"2,6,9","objects":2416}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.13","version":"20'137","reported_seq":575,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-20T09:06:51.934022+0000","last_change":"2024-03-20T09:04:48.832913+0000","last_active":"2024-03-20T09:06:51.934022+0000","last_peered":"2024-03-20T09:06:51.934022+0000","last_clean":"2024-03-20T09:06:51.934022+0000","last_became_active":"2024-03-20T09:04:04.103645+0000","last_became_peered":"2024-03-20T09:04:04.103645+0000","last_unstale":"2024-03-20T09:06:51.934022+0000","last_undegraded":"2024-03-20T09:06:51.934022+0000","last_fullsized":"2024-03-20T09:06:51.934022+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:04:48.832800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:04:48.832800+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"340'5583","reported_seq":9653,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-20T09:06:55.568682+0000","last_change":"2024-03-20T09:03:47.320424+0000","last_active":"2024-03-20T09:06:55.568682+0000","last_peered":"2024-03-20T09:06:55.568682+0000","last_clean":"2024-03-20T09:06:55.568682+0000","last_became_active":"2024-03-20T09:03:46.901072+0000","last_became_peered":"2024-03-20T09:03:46.901072+0000","last_unstale":"2024-03-20T09:06:55.568682+0000","last_undegraded":"2024-03-20T09:06:55.568682+0000","last_fullsized":"2024-03-20T09:06:55.568682+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":5,"last_scrub":"243'3881","last_scrub_stamp":"2024-03-20T09:03:47.320275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:03:47.320275+0000","objects_scrubbed":2110,"log_size":5583,"log_dups_size":0,"ondisk_log_size":5583,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":410,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161415168,"num_objects":2463,"num_object_clones":0,"num_object_copies":7389,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1709,"num_write_kb":65984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,6],"acting":[10,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.10","version":"20'137","reported_seq":369,"reported_epoch":340,"state":"active+undersized","last_fresh":"2024-03-20T09:06:54.997298+0000","last_change":"2024-03-20T09:04:34.735801+0000","last_active":"2024-03-20T09:06:54.997298+0000","last_peered":"2024-03-20T09:06:54.997298+0000","last_clean":"2024-03-20T08:54:29.262652+0000","last_became_active":"2024-03-20T09:04:34.735801+0000","last_became_peered":"2024-03-20T09:04:34.735801+0000","last_unstale":"2024-03-20T09:06:54.997298+0000","last_undegraded":"2024-03-20T09:06:54.997298+0000","last_fullsized":"2024-03-20T09:04:34.732688+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:04:04.319316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:04:04.319316+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:06:59.054409+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":8,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":48084,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":["10","7"],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.12","version":"340'5510","reported_seq":20768,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-03-20T09:06:56.934692+0000","last_change":"2024-03-20T09:06:44.957783+0000","last_active":"2024-03-20T09:06:56.934692+0000","last_peered":"2024-03-20T09:06:56.934692+0000","last_clean":"2024-03-20T09:06:56.934692+0000","last_became_active":"2024-03-20T09:06:44.949696+0000","last_became_peered":"2024-03-20T09:06:44.949696+0000","last_unstale":"2024-03-20T09:06:56.934692+0000","last_undegraded":"2024-03-20T09:06:56.934692+0000","last_fullsized":"2024-03-20T09:06:56.934692+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":5,"last_scrub":"285'4956","last_scrub_stamp":"2024-03-20T09:04:50.135501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:04:50.135501+0000","objects_scrubbed":3023,"log_size":5510,"log_dups_size":0,"ondisk_log_size":5510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":356,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161742848,"num_objects":2468,"num_object_clones":0,"num_object_copies":7404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":555,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3104,"num_bytes_recovered":169148416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"20'137","reported_seq":450,"reported_epoch":341,"state":"active+clean","last_fresh":"2024-03-20T09:06:56.934812+0000","last_change":"2024-03-20T09:06:02.928303+0000","last_active":"2024-03-20T09:06:56.934812+0000","last_peered":"2024-03-20T09:06:56.934812+0000","last_clean":"2024-03-20T09:06:56.934812+0000","last_became_active":"2024-03-20T09:06:02.927902+0000","last_became_peered":"2024-03-20T09:06:02.927902+0000","last_unstale":"2024-03-20T09:06:56.934812+0000","last_undegraded":"2024-03-20T09:06:56.934812+0000","last_fullsized":"2024-03-20T09:06:56.934812+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:04:42.894571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:04:42.894571+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1],"acting":[0,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"340'5526","reported_seq":11302,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-20T09:06:55.247395+0000","last_change":"2024-03-20T09:03:56.895356+0000","last_active":"2024-03-20T09:06:55.247395+0000","last_peered":"2024-03-20T09:06:55.247395+0000","last_clean":"2024-03-20T09:06:55.247395+0000","last_became_active":"2024-03-20T09:03:56.891781+0000","last_became_peered":"2024-03-20T09:03:56.891781+0000","last_unstale":"2024-03-20T09:06:55.247395+0000","last_undegraded":"2024-03-20T09:06:55.247395+0000","last_fullsized":"2024-03-20T09:06:55.247395+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":5526,"log_dups_size":0,"ondisk_log_size":5526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:06:58.321016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166494208,"num_objects":2541,"num_object_clones":0,"num_object_copies":7623,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3733,"num_write_kb":200736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1825,"num_bytes_recovered":119603200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"340'5404","reported_seq":11083,"reported_epoch":340,"state":"active+clean","last_fresh":"2024-03-20T09:06:55.094377+0000","last_change":"2024-03-20T09:03:52.899809+0000","last_active":"2024-03-20T09:06:55.094377+0000","last_peered":"2024-03-20T09:06:55.094377+0000","last_clean":"2024-03-20T09:06:55.094377+0000","last_became_active":"2024-03-20T09:03:52.897604+0000","last_became_peered":"2024-03-20T09:03:52.897604+0000","last_unstale":"2024-03-20T09:06:55.094377+0000","last_undegraded":"2024-03-20T09:06:55.094377+0000","last_fullsized":"2024-03-20T09:06:55.094377+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":5404,"log_dups_size":0,"ondisk_log_size":5404,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:06:58.321002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160595968,"num_objects":2451,"num_object_clones":0,"num_object_copies":7353,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3667,"num_write_kb":195744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1792,"num_bytes_recovered":117440512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,11],"acting":[4,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"340'5627","reported_seq":10539,"reported_epoch":340,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-20T09:06:56.569935+0000","last_change":"2024-03-20T09:06:11.661796+0000","last_active":"2024-03-20T09:06:56.569935+0000","last_peered":"2024-03-20T09:06:56.569935+0000","last_clean":"2024-03-20T09:03:31.866916+0000","last_became_active":"2024-03-20T09:06:11.451333+0000","last_became_peered":"2024-03-20T09:06:11.451333+0000","last_unstale":"2024-03-20T09:06:56.569935+0000","last_undegraded":"2024-03-20T09:06:05.930917+0000","last_fullsized":"2024-03-20T09:06:01.646279+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":5627,"log_dups_size":0,"ondisk_log_size":5627,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168525824,"num_objects":2572,"num_object_clones":0,"num_object_copies":7716,"num_objects_missing_on_primary":1004,"num_objects_missing":1004,"num_objects_degraded":5355,"num_objects_misplaced":2185,"num_objects_unfound":0,"num_objects_dirty":2572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3842,"num_write_kb":205216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1151,"num_bytes_recovered":75333632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,0],"acting":[11,4],"avail_no_missing":["2","3","6"],"object_location_counts":[{"shards":"2,4,6,7,11","objects":195},{"shards":"2,6","objects":1004},{"shards":"4,7,11","objects":1231},{"shards":"4,11","objects":-198},{"shards":"6,7,8,11","objects":1},{"shards":"7,8,11","objects":123}],"blocked_by":[],"up_primary":8,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.14","version":"20'137","reported_seq":651,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-20T09:06:51.995838+0000","last_change":"2024-03-20T09:06:24.048877+0000","last_active":"2024-03-20T09:06:51.995838+0000","last_peered":"2024-03-20T09:06:51.995838+0000","last_clean":"2024-03-20T09:06:51.995838+0000","last_became_active":"2024-03-20T09:04:14.208448+0000","last_became_peered":"2024-03-20T09:04:14.208448+0000","last_unstale":"2024-03-20T09:06:51.995838+0000","last_undegraded":"2024-03-20T09:06:51.995838+0000","last_fullsized":"2024-03-20T09:06:51.995838+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:06:24.048708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:06:24.048708+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:07:52.733927+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"339'5611","reported_seq":17369,"reported_epoch":339,"state":"active+clean+remapped","last_fresh":"2024-03-20T09:06:54.395318+0000","last_change":"2024-03-20T09:06:18.953397+0000","last_active":"2024-03-20T09:06:54.395318+0000","last_peered":"2024-03-20T09:06:54.395318+0000","last_clean":"2024-03-20T09:06:54.395318+0000","last_became_active":"2024-03-20T09:06:18.946699+0000","last_became_peered":"2024-03-20T09:06:18.946699+0000","last_unstale":"2024-03-20T09:06:54.395318+0000","last_undegraded":"2024-03-20T09:06:54.395318+0000","last_fullsized":"2024-03-20T09:06:54.395318+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":5,"last_scrub":"255'4276","last_scrub_stamp":"2024-03-20T09:04:01.842913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:04:01.842913+0000","objects_scrubbed":2457,"log_size":5611,"log_dups_size":0,"ondisk_log_size":5611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":540,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165675008,"num_objects":2528,"num_object_clones":0,"num_object_copies":7584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2528,"num_objects_unfound":0,"num_objects_dirty":2528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1343,"num_write_kb":45248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3164,"num_bytes_recovered":178585600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2,7],"avail_no_missing":["1","2","7"],"object_location_counts":[{"shards":"1,2,7","objects":2528}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"340'5498","reported_seq":19009,"reported_epoch":341,"state":"active+clean+remapped","last_fresh":"2024-03-20T09:06:56.935053+0000","last_change":"2024-03-20T09:06:46.990481+0000","last_active":"2024-03-20T09:06:56.935053+0000","last_peered":"2024-03-20T09:06:56.935053+0000","last_clean":"2024-03-20T09:06:56.935053+0000","last_became_active":"2024-03-20T09:06:46.981557+0000","last_became_peered":"2024-03-20T09:06:46.981557+0000","last_unstale":"2024-03-20T09:06:56.935053+0000","last_undegraded":"2024-03-20T09:06:56.935053+0000","last_fullsized":"2024-03-20T09:06:56.935053+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":5,"last_scrub":"251'4129","last_scrub_stamp":"2024-03-20T09:03:57.865687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:03:57.865687+0000","objects_scrubbed":2325,"log_size":5498,"log_dups_size":0,"ondisk_log_size":5498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166330368,"num_objects":2538,"num_object_clones":0,"num_object_copies":7614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2538,"num_objects_unfound":0,"num_objects_dirty":2538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1377,"num_write_kb":50880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3120,"num_bytes_recovered":173342720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1,6],"avail_no_missing":["0","1","6"],"object_location_counts":[{"shards":"0,1,6","objects":2538}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"339'5418","reported_seq":18998,"reported_epoch":339,"state":"active+clean","last_fresh":"2024-03-20T09:06:54.464508+0000","last_change":"2024-03-20T09:06:24.997013+0000","last_active":"2024-03-20T09:06:54.464508+0000","last_peered":"2024-03-20T09:06:54.464508+0000","last_clean":"2024-03-20T09:06:54.464508+0000","last_became_active":"2024-03-20T09:06:24.988319+0000","last_became_peered":"2024-03-20T09:06:24.988319+0000","last_unstale":"2024-03-20T09:06:54.464508+0000","last_undegraded":"2024-03-20T09:06:54.464508+0000","last_fullsized":"2024-03-20T09:06:54.464508+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":5,"last_scrub":"276'4872","last_scrub_stamp":"2024-03-20T09:04:44.020587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:04:44.020587+0000","objects_scrubbed":2997,"log_size":5418,"log_dups_size":0,"ondisk_log_size":5418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160628736,"num_objects":2451,"num_object_clones":0,"num_object_copies":7353,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":546,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3042,"num_bytes_recovered":170590208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,8],"acting":[1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":5218631704,"num_objects":79631,"num_object_clones":0,"num_object_copies":238893,"num_objects_missing_on_primary":5016,"num_objects_missing":5016,"num_objects_degraded":28667,"num_objects_misplaced":25932,"num_objects_unfound":0,"num_objects_dirty":79631,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":90339,"num_write_kb":4509121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64392,"num_bytes_recovered":4005593088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18956361728,"data_stored":18956353584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160766,"ondisk_log_size":160766,"up":70,"acting":68,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":806934,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":23,"num_read_kb":20,"num_write":33,"num_write_kb":535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":647065,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2007040,"data_stored":1989280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2877,"ondisk_log_size":2877,"up":55,"acting":57,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860547,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2187572,"kb_used_data":2016172,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":91557580,"statfs":{"total":95995035648,"available":93754961920,"internally_reserved":0,"allocated":2064560128,"data_stored":2063972973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":175503812},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":89,"seq":382252089480,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1446580,"kb_used_data":1289068,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":92298572,"statfs":{"total":95995035648,"available":94513737728,"internally_reserved":0,"allocated":1320005632,"data_stored":1319418477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":161282496},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":0,"seq":343597383824,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":70,"seq":300647710868,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1048704,"kb_used_data":905076,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92696448,"statfs":{"total":95995035648,"available":94921162752,"internally_reserved":0,"allocated":926797824,"data_stored":926207117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":147061190},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":60,"seq":257698037913,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2230780,"kb_used_data":1938608,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":91514372,"statfs":{"total":95995035648,"available":93710716928,"internally_reserved":0,"allocated":1985134592,"data_stored":1984543365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":299170250},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332252,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2112368,"kb_used_data":1940964,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":91632784,"statfs":{"total":95995035648,"available":93831970816,"internally_reserved":0,"allocated":1987547136,"data_stored":1986960056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":175503814},"hb_peers":[0,1,2,3,4,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575032,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1648616,"kb_used_data":1459168,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":92096536,"statfs":{"total":95995035648,"available":94306852864,"internally_reserved":0,"allocated":1494188032,"data_stored":1493605024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":193984966},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738559,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1416824,"kb_used_data":1205164,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":92328328,"statfs":{"total":95995035648,"available":94544207872,"internally_reserved":0,"allocated":1234087936,"data_stored":1233500781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411505,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1229112,"kb_used_data":1075752,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":92516040,"statfs":{"total":95995035648,"available":94736424960,"internally_reserved":0,"allocated":1101570048,"data_stored":1100983488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":111669149869,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953642,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2894976,"kb_used_data":2643060,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":90850176,"statfs":{"total":95995035648,"available":93030580224,"internally_reserved":0,"allocated":2706493440,"data_stored":2705902733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,1,2,3,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659271,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2258332,"kb_used_data":2123028,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":91486820,"statfs":{"total":95995035648,"available":93682503680,"internally_reserved":0,"allocated":2173980672,"data_stored":2173328093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1256259584,"data_stored":1256259584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1488650240,"data_stored":1488650240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1099300864,"data_stored":1099300864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1181679616,"data_stored":1181679616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2704605184,"data_stored":2704605184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2174091264,"data_stored":2174091264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1984434176,"data_stored":1984430104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1982337024,"data_stored":1982332952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":924057600,"data_stored":924057600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":782368768,"data_stored":782368768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1318256640,"data_stored":1318256640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2060320768,"data_stored":2060320768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:06:59.985 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pool set .mgr pgp_num 21 2024-03-20T09:07:00.005 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:07:00.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:00.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:07:00.110 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:07:00.145 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:00.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:00 smithi122 ceph-mon[73495]: from='client.15366 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:07:00.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:00 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:07:00.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:00 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:07:00.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:00 smithi122 ceph-mon[73495]: osdmap e343: 12 total, 11 up, 10 in 2024-03-20T09:07:00.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:00 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:07:00.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:00 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:07:00.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:00 smithi026 ceph-mon[68309]: from='client.15366 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:07:00.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:00 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:07:00.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:00 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:07:00.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:00 smithi026 ceph-mon[68309]: osdmap e343: 12 total, 11 up, 10 in 2024-03-20T09:07:00.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:00 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:07:00.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:00 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:07:00.246 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:07:00.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:00.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:07:00.351 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:07:00.377 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:00 smithi115 ceph-mon[72476]: from='client.15366 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:07:00.378 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:00 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:07:00.378 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:00 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:07:00.378 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:00 smithi115 ceph-mon[72476]: osdmap e343: 12 total, 11 up, 10 in 2024-03-20T09:07:00.378 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:00 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:07:00.378 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:00 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:07:00.380 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:00.399 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:07:00.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:00.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:07:00.503 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:07:00.532 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:00.605 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:07:00.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:00.608 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:07:00.709 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:07:00.738 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:01.320 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:01 smithi026 ceph-mon[68309]: pgmap v855: 47 pgs: 5 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 25 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.8 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 111 op/s; 28557/238341 objects degraded (11.982%); 25812/238341 objects misplaced (10.830%); 21 MiB/s, 460 objects/s recovering 2024-03-20T09:07:01.320 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:01 smithi026 ceph-mon[68309]: osdmap e344: 12 total, 11 up, 10 in 2024-03-20T09:07:01.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:01 smithi115 ceph-mon[72476]: pgmap v855: 47 pgs: 5 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 25 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.8 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 111 op/s; 28557/238341 objects degraded (11.982%); 25812/238341 objects misplaced (10.830%); 21 MiB/s, 460 objects/s recovering 2024-03-20T09:07:01.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:01 smithi115 ceph-mon[72476]: osdmap e344: 12 total, 11 up, 10 in 2024-03-20T09:07:01.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:01 smithi122 ceph-mon[73495]: pgmap v855: 47 pgs: 5 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 25 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.8 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 111 op/s; 28557/238341 objects degraded (11.982%); 25812/238341 objects misplaced (10.830%); 21 MiB/s, 460 objects/s recovering 2024-03-20T09:07:01.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:01 smithi122 ceph-mon[73495]: osdmap e344: 12 total, 11 up, 10 in 2024-03-20T09:07:01.665 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:07:01.788 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:07:01.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:01.792 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:07:01.893 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:07:01.923 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:01.948 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:07:01.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:01.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:07:02.053 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:07:02.082 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:02.102 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:07:02.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:02.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:07:02.160 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:07:02.161 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:07:02.161 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-03-20T09:07:02.161 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-03-20T09:07:02.161 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:07:02.206 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:07:02.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:02 smithi026 ceph-mon[68309]: 3.11 scrub starts 2024-03-20T09:07:02.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:02 smithi026 ceph-mon[68309]: osdmap e345: 12 total, 11 up, 10 in 2024-03-20T09:07:02.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:02 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 28557/238341 objects degraded (11.982%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:07:02.235 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:02.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:02 smithi115 ceph-mon[72476]: 3.11 scrub starts 2024-03-20T09:07:02.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:02 smithi115 ceph-mon[72476]: osdmap e345: 12 total, 11 up, 10 in 2024-03-20T09:07:02.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:02 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 28557/238341 objects degraded (11.982%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:07:02.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:02 smithi122 ceph-mon[73495]: 3.11 scrub starts 2024-03-20T09:07:02.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:02 smithi122 ceph-mon[73495]: osdmap e345: 12 total, 11 up, 10 in 2024-03-20T09:07:02.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:02 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 28557/238341 objects degraded (11.982%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:07:02.791 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:07:02.812 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:07:02.835 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:07:02.858 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:07:02.879 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:07:02.901 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:07:02.923 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:07:02.939 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_historic_ops 2024-03-20T09:07:02.944 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:07:02.965 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:07:02.983 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:07:02.985 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:07:02.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:02.986 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:07:03.009 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:07:03.009 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:07:03.009 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:07:03.009 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:07:03.009 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:07:03.009 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:07:03.010 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-20T09:07:03.010 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:07:03.010 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:07:03.010 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:07:03.010 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:07:03.010 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:07:03.010 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:07:03.045 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:07:03.087 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:07:03.113 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:03.137 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:07:03.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:03.139 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:07:03.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:03 smithi122 ceph-mon[73495]: 3.11 scrub ok 2024-03-20T09:07:03.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:03 smithi122 ceph-mon[73495]: pgmap v858: 47 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 24 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 121 op/s; 28351/237435 objects degraded (11.941%); 24209/237435 objects misplaced (10.196%); 26 MiB/s, 575 objects/s recovering 2024-03-20T09:07:03.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:03 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1798908783' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:07:03.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:03 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/829433057' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:07:03.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:03 smithi026 ceph-mon[68309]: 3.11 scrub ok 2024-03-20T09:07:03.245 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:03 smithi026 ceph-mon[68309]: pgmap v858: 47 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 24 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 121 op/s; 28351/237435 objects degraded (11.941%); 24209/237435 objects misplaced (10.196%); 26 MiB/s, 575 objects/s recovering 2024-03-20T09:07:03.246 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:03 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1798908783' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:07:03.246 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:03 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/829433057' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:07:03.246 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:07:03.260 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:07:03.280 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:03.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:03 smithi115 ceph-mon[72476]: 3.11 scrub ok 2024-03-20T09:07:03.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:03 smithi115 ceph-mon[72476]: pgmap v858: 47 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 24 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 121 op/s; 28351/237435 objects degraded (11.941%); 24209/237435 objects misplaced (10.196%); 26 MiB/s, 575 objects/s recovering 2024-03-20T09:07:03.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:03 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1798908783' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:07:03.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:03 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/829433057' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:07:03.628 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:07:03.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:03.630 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:07:03.731 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:07:03.766 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:03.870 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:07:03.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:03.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:07:03.973 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:07:04.001 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:04.023 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:07:04.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:04.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:07:04.127 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:07:04.152 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:04 smithi122 ceph-mon[73495]: 1.1 scrub starts 2024-03-20T09:07:04.152 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:04 smithi122 ceph-mon[73495]: 1.1 scrub ok 2024-03-20T09:07:04.152 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:04 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:07:04.152 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:04 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1798908783' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:07:04.152 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:04 smithi122 ceph-mon[73495]: osdmap e346: 12 total, 11 up, 10 in 2024-03-20T09:07:04.154 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:04.174 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:07:04.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:04.176 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:07:04.277 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:07:04.307 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:04 smithi026 ceph-mon[68309]: 1.1 scrub starts 2024-03-20T09:07:04.307 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:04 smithi026 ceph-mon[68309]: 1.1 scrub ok 2024-03-20T09:07:04.307 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:04 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:07:04.307 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:04 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1798908783' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:07:04.307 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:04 smithi026 ceph-mon[68309]: osdmap e346: 12 total, 11 up, 10 in 2024-03-20T09:07:04.309 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:04.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:04 smithi115 ceph-mon[72476]: 1.1 scrub starts 2024-03-20T09:07:04.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:04 smithi115 ceph-mon[72476]: 1.1 scrub ok 2024-03-20T09:07:04.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:04 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:07:04.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:04 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1798908783' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:07:04.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:04 smithi115 ceph-mon[72476]: osdmap e346: 12 total, 11 up, 10 in 2024-03-20T09:07:05.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:05 smithi026 ceph-mon[68309]: pgmap v860: 47 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 153 op/s; 24198/236697 objects degraded (10.223%); 27298/236697 objects misplaced (11.533%); 22 MiB/s, 563 objects/s recovering 2024-03-20T09:07:05.266 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:07:05.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:05.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:07:05.333 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:07:05.333 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:07:05.363 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:05 smithi122 ceph-mon[73495]: pgmap v860: 47 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 153 op/s; 24198/236697 objects degraded (10.223%); 27298/236697 objects misplaced (11.533%); 22 MiB/s, 563 objects/s recovering 2024-03-20T09:07:05.372 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:07:05.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:05 smithi115 ceph-mon[72476]: pgmap v860: 47 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 153 op/s; 24198/236697 objects degraded (10.223%); 27298/236697 objects misplaced (11.533%); 22 MiB/s, 563 objects/s recovering 2024-03-20T09:07:05.408 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:05.609 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:07:05.609 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:07:05.659 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:07:05.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:05.662 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:07:05.762 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:07:05.799 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:05.823 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:07:05.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:05.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:07:05.927 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:07:05.959 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:05.980 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:07:05.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:05.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:07:06.062 INFO:teuthology.orchestra.run.smithi026.stderr:set pool 1 pgp_num to 21 2024-03-20T09:07:06.084 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:07:06.113 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:06.187 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:07:06.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:06.192 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:07:06.206 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-20T09:07:06.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:06 smithi026 ceph-mon[68309]: 1.5 deep-scrub starts 2024-03-20T09:07:06.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:06 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3386144041' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-03-20T09:07:06.293 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:07:06.321 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:06 smithi115 ceph-mon[72476]: 1.5 deep-scrub starts 2024-03-20T09:07:06.321 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:06 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3386144041' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-03-20T09:07:06.323 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:06.343 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:07:06.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:06.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:07:06.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:06 smithi122 ceph-mon[73495]: 1.5 deep-scrub starts 2024-03-20T09:07:06.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:06 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3386144041' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-03-20T09:07:06.447 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:07:06.473 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:06.494 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:07:06.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:06.497 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:07:06.598 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:07:06.627 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:06.628 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:07:06.647 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:07:06.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:06.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:07:06.752 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:07:06.782 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:07.131 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:07:07.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:07.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:07:07.235 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:07:07.253 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:07:07.264 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:07:07.271 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:07.298 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:07:07.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:07.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:07:07.402 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:07:07.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:07 smithi115 ceph-mon[72476]: pgmap v861: 47 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 168 op/s; 24093/235311 objects degraded (10.239%); 26295/235311 objects misplaced (11.175%); 26 MiB/s, 638 objects/s recovering 2024-03-20T09:07:07.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:07 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3386144041' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-03-20T09:07:07.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:07 smithi115 ceph-mon[72476]: osdmap e347: 12 total, 11 up, 10 in 2024-03-20T09:07:07.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:07 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 24093/235311 objects degraded (10.239%), 8 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:07:07.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:07 smithi115 ceph-mon[72476]: osdmap e348: 12 total, 11 up, 10 in 2024-03-20T09:07:07.426 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:07 smithi122 ceph-mon[73495]: pgmap v861: 47 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 168 op/s; 24093/235311 objects degraded (10.239%); 26295/235311 objects misplaced (11.175%); 26 MiB/s, 638 objects/s recovering 2024-03-20T09:07:07.426 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:07 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3386144041' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-03-20T09:07:07.426 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:07 smithi122 ceph-mon[73495]: osdmap e347: 12 total, 11 up, 10 in 2024-03-20T09:07:07.426 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:07 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 24093/235311 objects degraded (10.239%), 8 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:07:07.427 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:07 smithi122 ceph-mon[73495]: osdmap e348: 12 total, 11 up, 10 in 2024-03-20T09:07:07.428 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:07.447 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:07:07.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:07.450 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:07:07.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:07 smithi026 ceph-mon[68309]: pgmap v861: 47 pgs: 1 active+clean+scrubbing, 5 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 168 op/s; 24093/235311 objects degraded (10.239%); 26295/235311 objects misplaced (11.175%); 26 MiB/s, 638 objects/s recovering 2024-03-20T09:07:07.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:07 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3386144041' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-03-20T09:07:07.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:07 smithi026 ceph-mon[68309]: osdmap e347: 12 total, 11 up, 10 in 2024-03-20T09:07:07.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:07 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 24093/235311 objects degraded (10.239%), 8 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:07:07.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:07 smithi026 ceph-mon[68309]: osdmap e348: 12 total, 11 up, 10 in 2024-03-20T09:07:07.551 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:07:07.577 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:07.596 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:07:07.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:07.599 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:07:07.610 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:07:07.699 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:07:07.726 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:07.745 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:07:07.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:07.748 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:07:07.848 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:07:07.885 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:08.235 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:07:08.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:08.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:07:08.339 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:07:08.365 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:08 smithi026 ceph-mon[68309]: 1.5 deep-scrub ok 2024-03-20T09:07:08.365 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:08 smithi026 ceph-mon[68309]: osdmap e349: 12 total, 11 up, 10 in 2024-03-20T09:07:08.367 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:08.395 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:07:08.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:08.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:07:08.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:08 smithi115 ceph-mon[72476]: 1.5 deep-scrub ok 2024-03-20T09:07:08.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:08 smithi115 ceph-mon[72476]: osdmap e349: 12 total, 11 up, 10 in 2024-03-20T09:07:08.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:08 smithi122 ceph-mon[73495]: 1.5 deep-scrub ok 2024-03-20T09:07:08.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:08 smithi122 ceph-mon[73495]: osdmap e349: 12 total, 11 up, 10 in 2024-03-20T09:07:08.500 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:07:08.527 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:08.545 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:07:08.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:08.548 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:07:08.649 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:07:08.677 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:08.714 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:07:08.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:08.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:07:08.818 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:07:08.846 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:08.873 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:07:08.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:08.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:07:08.978 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:07:09.007 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:09.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:09 smithi115 ceph-mon[72476]: pgmap v864: 47 pgs: 1 creating+peering, 5 active+clean+remapped, 1 active+clean+scrubbing+deep, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 4.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 168 op/s; 24099/234402 objects degraded (10.281%); 26204/234402 objects misplaced (11.179%); 9.7 MiB/s, 348 objects/s recovering 2024-03-20T09:07:09.282 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:09 smithi026 ceph-mon[68309]: pgmap v864: 47 pgs: 1 creating+peering, 5 active+clean+remapped, 1 active+clean+scrubbing+deep, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 4.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 168 op/s; 24099/234402 objects degraded (10.281%); 26204/234402 objects misplaced (11.179%); 9.7 MiB/s, 348 objects/s recovering 2024-03-20T09:07:09.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:09 smithi122 ceph-mon[73495]: pgmap v864: 47 pgs: 1 creating+peering, 5 active+clean+remapped, 1 active+clean+scrubbing+deep, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 4.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 168 op/s; 24099/234402 objects degraded (10.281%); 26204/234402 objects misplaced (11.179%); 9.7 MiB/s, 348 objects/s recovering 2024-03-20T09:07:09.857 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:07:09.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:09.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:07:09.965 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:07:09.998 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:10.381 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:07:10.381 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:07:10.713 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:07:10.847 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:07:10.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:10.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:07:10.955 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:07:10.985 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:11.078 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:07:11.096 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:11 smithi026 ceph-mon[68309]: pgmap v866: 47 pgs: 1 creating+peering, 5 active+clean+remapped, 1 active+clean+scrubbing+deep, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 4.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 160 op/s; 23976/233799 objects degraded (10.255%); 26063/233799 objects misplaced (11.148%); 11 MiB/s, 312 objects/s recovering 2024-03-20T09:07:11.096 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:11 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2918357045' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:07:11.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:11 smithi115 ceph-mon[72476]: pgmap v866: 47 pgs: 1 creating+peering, 5 active+clean+remapped, 1 active+clean+scrubbing+deep, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 4.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 160 op/s; 23976/233799 objects degraded (10.255%); 26063/233799 objects misplaced (11.148%); 11 MiB/s, 312 objects/s recovering 2024-03-20T09:07:11.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:11 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2918357045' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:07:11.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:11 smithi122 ceph-mon[73495]: pgmap v866: 47 pgs: 1 creating+peering, 5 active+clean+remapped, 1 active+clean+scrubbing+deep, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 4.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 160 op/s; 23976/233799 objects degraded (10.255%); 26063/233799 objects misplaced (11.148%); 11 MiB/s, 312 objects/s recovering 2024-03-20T09:07:11.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:11 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2918357045' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:07:12.147 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7, 8, 10, 11] out_osds: [9, 3] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11] 2024-03-20T09:07:12.147 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:07:12.148 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-03-20T09:07:12.148 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:07:12.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:12 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:07:12.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:12 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2918357045' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:07:12.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:12 smithi115 ceph-mon[72476]: osdmap e350: 12 total, 11 up, 10 in 2024-03-20T09:07:12.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:12 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 23976/233799 objects degraded (10.255%), 8 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:07:12.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:12 smithi115 ceph-mon[72476]: osdmap e351: 12 total, 11 up, 10 in 2024-03-20T09:07:12.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:12 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:07:12.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:12 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2918357045' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:07:12.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:12 smithi122 ceph-mon[73495]: osdmap e350: 12 total, 11 up, 10 in 2024-03-20T09:07:12.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:12 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 23976/233799 objects degraded (10.255%), 8 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:07:12.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:12 smithi122 ceph-mon[73495]: osdmap e351: 12 total, 11 up, 10 in 2024-03-20T09:07:12.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:12 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:07:12.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:12 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2918357045' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:07:12.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:12 smithi026 ceph-mon[68309]: osdmap e350: 12 total, 11 up, 10 in 2024-03-20T09:07:12.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:12 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 23976/233799 objects degraded (10.255%), 8 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:07:12.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:12 smithi026 ceph-mon[68309]: osdmap e351: 12 total, 11 up, 10 in 2024-03-20T09:07:13.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:13 smithi115 ceph-mon[72476]: pgmap v868: 47 pgs: 1 creating+peering, 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 27 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 132 op/s; 15817/233076 objects degraded (6.786%); 32079/233076 objects misplaced (13.763%); 3.7 MiB/s, 138 objects/s recovering 2024-03-20T09:07:13.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:13 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:07:13.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:13 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:07:13.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:13 smithi122 ceph-mon[73495]: pgmap v868: 47 pgs: 1 creating+peering, 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 27 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 132 op/s; 15817/233076 objects degraded (6.786%); 32079/233076 objects misplaced (13.763%); 3.7 MiB/s, 138 objects/s recovering 2024-03-20T09:07:13.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:13 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:07:13.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:13 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:07:13.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:13 smithi026 ceph-mon[68309]: pgmap v868: 47 pgs: 1 creating+peering, 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 27 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 132 op/s; 15817/233076 objects degraded (6.786%); 32079/233076 objects misplaced (13.763%); 3.7 MiB/s, 138 objects/s recovering 2024-03-20T09:07:13.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:13 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:07:13.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:13 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:07:14.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:14 smithi115 ceph-mon[72476]: osdmap e352: 12 total, 11 up, 10 in 2024-03-20T09:07:14.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:14 smithi122 ceph-mon[73495]: osdmap e352: 12 total, 11 up, 10 in 2024-03-20T09:07:14.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:14 smithi026 ceph-mon[68309]: osdmap e352: 12 total, 11 up, 10 in 2024-03-20T09:07:15.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:15 smithi115 ceph-mon[72476]: pgmap v871: 47 pgs: 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 130 op/s; 15800/232155 objects degraded (6.806%); 31995/232155 objects misplaced (13.782%); 3.6 MiB/s, 57 objects/s recovering 2024-03-20T09:07:15.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:15 smithi122 ceph-mon[73495]: pgmap v871: 47 pgs: 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 130 op/s; 15800/232155 objects degraded (6.806%); 31995/232155 objects misplaced (13.782%); 3.6 MiB/s, 57 objects/s recovering 2024-03-20T09:07:15.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:15 smithi026 ceph-mon[68309]: pgmap v871: 47 pgs: 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 130 op/s; 15800/232155 objects degraded (6.806%); 31995/232155 objects misplaced (13.782%); 3.6 MiB/s, 57 objects/s recovering 2024-03-20T09:07:15.553 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-20T09:07:15.575 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:07:15.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:15.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:07:15.679 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:07:15.706 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:15.725 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:07:15.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:15.728 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:07:15.829 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:07:15.855 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:15.874 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:07:15.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:15.876 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:07:15.977 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:07:16.012 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:16.034 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:07:16.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:16.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:07:16.139 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:07:16.168 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:17.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:17 smithi115 ceph-mon[72476]: pgmap v872: 47 pgs: 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 153 op/s; 15701/231033 objects degraded (6.796%); 31336/231033 objects misplaced (13.563%); 5.6 MiB/s, 90 objects/s recovering 2024-03-20T09:07:17.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:17 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 15701/231033 objects degraded (6.796%), 7 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:07:17.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:17 smithi115 ceph-mon[72476]: osdmap e353: 12 total, 11 up, 10 in 2024-03-20T09:07:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:17 smithi122 ceph-mon[73495]: pgmap v872: 47 pgs: 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 153 op/s; 15701/231033 objects degraded (6.796%); 31336/231033 objects misplaced (13.563%); 5.6 MiB/s, 90 objects/s recovering 2024-03-20T09:07:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:17 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 15701/231033 objects degraded (6.796%), 7 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:07:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:17 smithi122 ceph-mon[73495]: osdmap e353: 12 total, 11 up, 10 in 2024-03-20T09:07:17.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:17 smithi026 ceph-mon[68309]: pgmap v872: 47 pgs: 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 153 op/s; 15701/231033 objects degraded (6.796%); 31336/231033 objects misplaced (13.563%); 5.6 MiB/s, 90 objects/s recovering 2024-03-20T09:07:17.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:17 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 15701/231033 objects degraded (6.796%), 7 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:07:17.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:17 smithi026 ceph-mon[68309]: osdmap e353: 12 total, 11 up, 10 in 2024-03-20T09:07:17.818 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:07:17.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:17.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:07:17.924 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:07:17.951 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:17.969 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:07:17.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:17.972 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:07:18.073 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:07:18.102 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:18.125 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:07:18.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:18.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:07:18.230 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:07:18.257 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:18.277 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:07:18.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:18.281 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:07:18.382 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:07:18.412 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:18.433 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:07:18.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:18.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:07:18.540 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:07:18.571 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:18.821 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:07:18.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:18.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:07:18.924 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:07:18.952 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:18.970 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:07:18.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:18.974 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:07:19.075 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:07:19.104 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:19.176 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:07:19.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:19.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:07:19.284 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:07:19.308 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:19 smithi122 ceph-mon[73495]: pgmap v874: 47 pgs: 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 156 op/s; 15703/230265 objects degraded (6.820%); 31274/230265 objects misplaced (13.582%); 3.7 MiB/s, 58 objects/s recovering 2024-03-20T09:07:19.310 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:19.330 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:07:19.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:19.333 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:07:19.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:19 smithi115 ceph-mon[72476]: pgmap v874: 47 pgs: 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 156 op/s; 15703/230265 objects degraded (6.820%); 31274/230265 objects misplaced (13.582%); 3.7 MiB/s, 58 objects/s recovering 2024-03-20T09:07:19.434 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:07:19.461 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:19.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:19 smithi026 ceph-mon[68309]: pgmap v874: 47 pgs: 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 156 op/s; 15703/230265 objects degraded (6.820%); 31274/230265 objects misplaced (13.582%); 3.7 MiB/s, 58 objects/s recovering 2024-03-20T09:07:19.480 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:07:19.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:19.483 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:07:19.584 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:07:19.610 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:19.628 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:07:19.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:19.631 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:07:19.701 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:07:19.731 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:07:19.732 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:07:19.758 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:19.777 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:07:19.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:19.780 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:07:19.881 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:07:19.907 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:19.926 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:07:19.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:19.930 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:07:20.031 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:07:20.059 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:20.078 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:07:20.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:20.081 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:07:20.182 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:07:20.214 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:20.831 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:07:20.968 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:07:20.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:20.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:07:21.073 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:07:21.100 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:21.120 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:07:21.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:21.124 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:07:21.204 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:07:21.223 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:07:21.224 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:07:21.244 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:07:21.252 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:21.269 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:07:21.271 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:07:21.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:21.274 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:07:21.292 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:07:21.312 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:07:21.336 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:07:21.356 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:07:21.375 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:07:21.376 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:07:21.397 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:07:21.401 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:21 smithi115 ceph-mon[72476]: pgmap v875: 47 pgs: 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 131 op/s; 15585/229605 objects degraded (6.788%); 31176/229605 objects misplaced (13.578%); 3.0 MiB/s, 48 objects/s recovering 2024-03-20T09:07:21.404 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:21.418 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:07:21.418 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:07:21.418 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:07:21.418 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:07:21.418 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:07:21.418 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:07:21.418 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-20T09:07:21.419 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:07:21.419 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:07:21.419 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:07:21.419 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:07:21.419 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:07:21.419 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:07:21.425 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:07:21.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:21.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:07:21.439 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:21 smithi026 ceph-mon[68309]: pgmap v875: 47 pgs: 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 131 op/s; 15585/229605 objects degraded (6.788%); 31176/229605 objects misplaced (13.578%); 3.0 MiB/s, 48 objects/s recovering 2024-03-20T09:07:21.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:21 smithi122 ceph-mon[73495]: pgmap v875: 47 pgs: 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 131 op/s; 15585/229605 objects degraded (6.788%); 31176/229605 objects misplaced (13.578%); 3.0 MiB/s, 48 objects/s recovering 2024-03-20T09:07:21.528 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:07:21.555 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:21.573 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:07:21.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:21.576 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:07:21.677 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:07:21.705 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:21.724 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:07:21.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:21.727 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:07:21.828 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:07:21.857 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:21.877 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:07:21.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:21.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:07:21.981 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:07:22.006 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:22.025 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:07:22.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:22.027 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:07:22.128 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:07:22.154 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:22 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2401835469' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:07:22.154 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:22 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 15585/229605 objects degraded (6.788%), 7 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:07:22.156 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:22.180 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:07:22.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:22.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:07:22.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:22 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2401835469' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:07:22.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:22 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 15585/229605 objects degraded (6.788%), 7 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:07:22.285 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:07:22.311 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:22 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2401835469' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:07:22.312 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:22 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 15585/229605 objects degraded (6.788%), 7 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:07:22.313 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:22.332 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:07:22.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:22.335 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:07:22.436 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:07:22.464 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:22.484 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:07:22.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:22.487 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:07:22.588 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:07:22.616 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:22.636 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:07:22.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:22.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:07:22.741 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:07:22.766 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:22.785 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:07:22.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:22.788 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:07:22.889 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:07:22.919 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:22.937 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:07:22.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:22.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:07:23.041 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:07:23.068 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:23.091 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:07:23.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:23.094 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:07:23.144 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:07:23.145 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:07:23.145 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-03-20T09:07:23.145 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-03-20T09:07:23.145 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:07:23.194 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:07:23.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:23 smithi026 ceph-mon[68309]: pgmap v876: 47 pgs: 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 129 op/s; 15554/228867 objects degraded (6.796%); 30577/228867 objects misplaced (13.360%); 4.9 MiB/s, 77 objects/s recovering 2024-03-20T09:07:23.221 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:23 smithi115 ceph-mon[72476]: pgmap v876: 47 pgs: 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 129 op/s; 15554/228867 objects degraded (6.796%); 30577/228867 objects misplaced (13.360%); 4.9 MiB/s, 77 objects/s recovering 2024-03-20T09:07:23.223 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:23.246 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:07:23.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:23.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:07:23.335 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:07:23.350 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:07:23.386 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:23.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:23 smithi122 ceph-mon[73495]: pgmap v876: 47 pgs: 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.7 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 129 op/s; 15554/228867 objects degraded (6.796%); 30577/228867 objects misplaced (13.360%); 4.9 MiB/s, 77 objects/s recovering 2024-03-20T09:07:23.788 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-20T09:07:24.195 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:07:24.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:24.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:07:24.213 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:07:24.214 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:07:24.300 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:07:24.329 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:24.402 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:07:24.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:24.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:07:24.507 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:07:24.538 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:25.180 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:07:25.181 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:07:25.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:25 smithi115 ceph-mon[72476]: pgmap v877: 47 pgs: 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 139 op/s; 15539/227958 objects degraded (6.817%); 30486/227958 objects misplaced (13.374%); 4.3 MiB/s, 68 objects/s recovering 2024-03-20T09:07:25.414 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:25 smithi026 ceph-mon[68309]: pgmap v877: 47 pgs: 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 139 op/s; 15539/227958 objects degraded (6.817%); 30486/227958 objects misplaced (13.374%); 4.3 MiB/s, 68 objects/s recovering 2024-03-20T09:07:25.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:25 smithi122 ceph-mon[73495]: pgmap v877: 47 pgs: 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 139 op/s; 15539/227958 objects degraded (6.817%); 30486/227958 objects misplaced (13.374%); 4.3 MiB/s, 68 objects/s recovering 2024-03-20T09:07:25.588 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:07:25.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:25.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:07:25.692 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:07:25.720 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:25.741 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:07:25.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:25.744 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:07:25.845 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:07:25.872 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:25.891 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:07:25.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:25.894 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:07:25.998 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:07:26.024 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:26.044 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:07:26.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:26.048 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:07:26.149 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:07:26.180 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:26.215 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:07:26.235 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:26 smithi026 ceph-mon[68309]: from='client.15402 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:07:26.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:26 smithi115 ceph-mon[72476]: from='client.15402 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:07:26.431 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:07:26.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:26.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:07:26.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:26 smithi122 ceph-mon[73495]: from='client.15402 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:07:26.536 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:07:26.565 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:26.575 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":877,"stamp":"2024-03-20T09:07:23.671654+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4980428846,"num_objects":75986,"num_object_clones":0,"num_object_copies":227958,"num_objects_missing_on_primary":4735,"num_objects_missing":4735,"num_objects_degraded":15539,"num_objects_misplaced":30486,"num_objects_unfound":0,"num_objects_dirty":75986,"num_whiteouts":0,"num_read":24,"num_read_kb":21,"num_write":94019,"num_write_kb":4509656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69800,"num_bytes_recovered":4242759577,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":166990,"ondisk_log_size":166990,"up":125,"acting":128,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":155,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":17474188,"kb_used_data":15547836,"kb_used_omap":15,"kb_used_meta":1926256,"kb_avail":919977332,"statfs":{"total":959950356480,"available":942056787968,"internally_reserved":0,"allocated":15920984064,"data_stored":15914541462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15948,"internal_metadata":1972486580},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-91684864,"num_objects":-1399,"num_object_clones":0,"num_object_copies":-4197,"num_objects_missing_on_primary":-119,"num_objects_missing":-119,"num_objects_degraded":-261,"num_objects_misplaced":-1509,"num_objects_unfound":0,"num_objects_dirty":-1399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1399,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":686,"num_bytes_recovered":44957696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002610"},"pg_stats":[{"pgid":"3.19","version":"353'5657","reported_seq":19682,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-20T09:07:18.776092+0000","last_change":"2024-03-20T09:05:36.733109+0000","last_active":"2024-03-20T09:07:18.776092+0000","last_peered":"2024-03-20T09:07:18.776092+0000","last_clean":"2024-03-20T09:07:18.776092+0000","last_became_active":"2024-03-20T09:05:36.725732+0000","last_became_peered":"2024-03-20T09:05:36.725732+0000","last_unstale":"2024-03-20T09:07:18.776092+0000","last_undegraded":"2024-03-20T09:07:18.776092+0000","last_fullsized":"2024-03-20T09:07:18.776092+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":5,"last_scrub":"284'4948","last_scrub_stamp":"2024-03-20T09:04:48.131969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:04:48.131969+0000","objects_scrubbed":3082,"log_size":5657,"log_dups_size":0,"ondisk_log_size":5657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":324,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155516928,"num_objects":2373,"num_object_clones":0,"num_object_copies":7119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":709,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3145,"num_bytes_recovered":184090624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,10],"acting":[1,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"20'137","reported_seq":430,"reported_epoch":353,"state":"active+clean+remapped","last_fresh":"2024-03-20T09:07:16.986814+0000","last_change":"2024-03-20T09:06:02.926582+0000","last_active":"2024-03-20T09:07:16.986814+0000","last_peered":"2024-03-20T09:07:16.986814+0000","last_clean":"2024-03-20T09:07:16.986814+0000","last_became_active":"2024-03-20T09:06:02.926039+0000","last_became_peered":"2024-03-20T09:06:02.926039+0000","last_unstale":"2024-03-20T09:07:16.986814+0000","last_undegraded":"2024-03-20T09:07:16.986814+0000","last_fullsized":"2024-03-20T09:07:16.986814+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:05:23.614560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:05:23.614560+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8,4],"avail_no_missing":["2","4","8"],"object_location_counts":[{"shards":"2,4,8","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"353'7774","reported_seq":17385,"reported_epoch":353,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-20T09:07:19.168974+0000","last_change":"2024-03-20T09:06:08.163861+0000","last_active":"2024-03-20T09:07:19.168974+0000","last_peered":"2024-03-20T09:07:19.168974+0000","last_clean":"2024-03-20T09:05:59.443016+0000","last_became_active":"2024-03-20T09:06:08.132694+0000","last_became_peered":"2024-03-20T09:06:08.132694+0000","last_unstale":"2024-03-20T09:07:19.168974+0000","last_undegraded":"2024-03-20T09:07:19.168974+0000","last_fullsized":"2024-03-20T09:06:07.921226+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":7774,"log_dups_size":0,"ondisk_log_size":7774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313917440,"num_objects":4790,"num_object_clones":0,"num_object_copies":14370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4599,"num_objects_unfound":0,"num_objects_dirty":4790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7774,"num_write_kb":402048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4639,"num_bytes_recovered":304021504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4],"avail_no_missing":["6","4","7","8"],"object_location_counts":[{"shards":"4,6","objects":191},{"shards":"4,6,7","objects":4599}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"20'137","reported_seq":742,"reported_epoch":353,"state":"active+undersized","last_fresh":"2024-03-20T09:07:16.986865+0000","last_change":"2024-03-20T09:04:34.736101+0000","last_active":"2024-03-20T09:07:16.986865+0000","last_peered":"2024-03-20T09:07:16.986865+0000","last_clean":"2024-03-20T09:04:29.198571+0000","last_became_active":"2024-03-20T09:04:34.736101+0000","last_became_peered":"2024-03-20T09:04:34.736101+0000","last_unstale":"2024-03-20T09:07:16.986865+0000","last_undegraded":"2024-03-20T09:07:16.986865+0000","last_fullsized":"2024-03-20T09:04:34.733152+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:03:56.726408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:03:56.726408+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:07:17.894924+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":16,"num_write_kb":267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":240422,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"353'7786","reported_seq":28201,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-20T09:07:21.629029+0000","last_change":"2024-03-20T09:07:07.963778+0000","last_active":"2024-03-20T09:07:21.629029+0000","last_peered":"2024-03-20T09:07:21.629029+0000","last_clean":"2024-03-20T09:07:21.629029+0000","last_became_active":"2024-03-20T09:07:07.948193+0000","last_became_peered":"2024-03-20T09:07:07.948193+0000","last_unstale":"2024-03-20T09:07:21.629029+0000","last_undegraded":"2024-03-20T09:07:21.629029+0000","last_fullsized":"2024-03-20T09:07:21.629029+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":7786,"log_dups_size":0,"ondisk_log_size":7786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319553536,"num_objects":4876,"num_object_clones":0,"num_object_copies":14628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7786,"num_write_kb":405184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10037,"num_bytes_recovered":602210304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,11],"acting":[0,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"20'137","reported_seq":541,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-20T09:07:16.950321+0000","last_change":"2024-03-20T09:03:13.525409+0000","last_active":"2024-03-20T09:07:16.950321+0000","last_peered":"2024-03-20T09:07:16.950321+0000","last_clean":"2024-03-20T09:07:16.950321+0000","last_became_active":"2024-03-20T09:03:07.875459+0000","last_became_peered":"2024-03-20T09:03:07.875459+0000","last_unstale":"2024-03-20T09:07:16.950321+0000","last_undegraded":"2024-03-20T09:07:16.950321+0000","last_fullsized":"2024-03-20T09:07:16.950321+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:03:13.525325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:03:13.525325+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"353'5718","reported_seq":6037,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-20T09:07:21.265701+0000","last_change":"2024-03-20T09:03:30.919404+0000","last_active":"2024-03-20T09:07:21.265701+0000","last_peered":"2024-03-20T09:07:21.265701+0000","last_clean":"2024-03-20T09:07:21.265701+0000","last_became_active":"2024-03-20T09:03:30.916325+0000","last_became_peered":"2024-03-20T09:03:30.916325+0000","last_unstale":"2024-03-20T09:07:21.265701+0000","last_undegraded":"2024-03-20T09:07:21.265701+0000","last_fullsized":"2024-03-20T09:07:21.265701+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":5718,"log_dups_size":0,"ondisk_log_size":5718,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:07:21.962381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155025408,"num_objects":2365,"num_object_clones":0,"num_object_copies":7095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3953,"num_write_kb":202208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"20'137","reported_seq":710,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-20T09:07:16.989117+0000","last_change":"2024-03-20T09:06:02.927341+0000","last_active":"2024-03-20T09:07:16.989117+0000","last_peered":"2024-03-20T09:07:16.989117+0000","last_clean":"2024-03-20T09:07:16.989117+0000","last_became_active":"2024-03-20T09:06:02.926326+0000","last_became_peered":"2024-03-20T09:06:02.926326+0000","last_unstale":"2024-03-20T09:07:16.989117+0000","last_undegraded":"2024-03-20T09:07:16.989117+0000","last_fullsized":"2024-03-20T09:07:16.989117+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:24.397259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:24.397259+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:07:17.940127+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109226,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":109226,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.6","version":"353'5712","reported_seq":13613,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-20T09:07:21.345319+0000","last_change":"2024-03-20T09:07:01.057258+0000","last_active":"2024-03-20T09:07:21.345319+0000","last_peered":"2024-03-20T09:07:21.345319+0000","last_clean":"2024-03-20T09:07:21.345319+0000","last_became_active":"2024-03-20T09:07:01.049398+0000","last_became_peered":"2024-03-20T09:07:01.049398+0000","last_unstale":"2024-03-20T09:07:21.345319+0000","last_undegraded":"2024-03-20T09:07:21.345319+0000","last_fullsized":"2024-03-20T09:07:21.345319+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'4943","last_scrub_stamp":"2024-03-20T09:04:46.116541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:04:46.116541+0000","objects_scrubbed":3048,"log_size":5712,"log_dups_size":0,"ondisk_log_size":5712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":317,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149356544,"num_objects":2279,"num_object_clones":0,"num_object_copies":6837,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":769,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3089,"num_bytes_recovered":160890880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"20'137","reported_seq":618,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-20T09:07:16.939860+0000","last_change":"2024-03-20T09:07:06.480278+0000","last_active":"2024-03-20T09:07:16.939860+0000","last_peered":"2024-03-20T09:07:16.939860+0000","last_clean":"2024-03-20T09:07:16.939860+0000","last_became_active":"2024-03-20T09:03:58.987697+0000","last_became_peered":"2024-03-20T09:03:58.987697+0000","last_unstale":"2024-03-20T09:07:16.939860+0000","last_undegraded":"2024-03-20T09:07:16.939860+0000","last_fullsized":"2024-03-20T09:07:16.939860+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:07:06.480160+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:07:06.480160+0000","last_clean_scrub_stamp":"2024-03-20T09:07:06.480160+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:08:11.276206+0000","scrub_duration":1503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.7","version":"353'5625","reported_seq":15401,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-20T09:07:20.262517+0000","last_change":"2024-03-20T09:06:30.957907+0000","last_active":"2024-03-20T09:07:20.262517+0000","last_peered":"2024-03-20T09:07:20.262517+0000","last_clean":"2024-03-20T09:07:20.262517+0000","last_became_active":"2024-03-20T09:06:30.951316+0000","last_became_peered":"2024-03-20T09:06:30.951316+0000","last_unstale":"2024-03-20T09:07:20.262517+0000","last_undegraded":"2024-03-20T09:07:20.262517+0000","last_fullsized":"2024-03-20T09:07:20.262517+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'4179","last_scrub_stamp":"2024-03-20T09:04:00.661196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:04:00.661196+0000","objects_scrubbed":2441,"log_size":5625,"log_dups_size":0,"ondisk_log_size":5625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156368896,"num_objects":2386,"num_object_clones":0,"num_object_copies":7158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1447,"num_write_kb":44544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3137,"num_bytes_recovered":176553984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"20'137","reported_seq":626,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-20T09:07:16.939875+0000","last_change":"2024-03-20T09:06:58.034407+0000","last_active":"2024-03-20T09:07:16.939875+0000","last_peered":"2024-03-20T09:07:16.939875+0000","last_clean":"2024-03-20T09:07:16.939875+0000","last_became_active":"2024-03-20T09:03:58.990557+0000","last_became_peered":"2024-03-20T09:03:58.990557+0000","last_unstale":"2024-03-20T09:07:16.939875+0000","last_undegraded":"2024-03-20T09:07:16.939875+0000","last_fullsized":"2024-03-20T09:07:16.939875+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:06:58.034269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:06:58.034269+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:08:02.563069+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,4],"acting":[10,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.4","version":"353'5578","reported_seq":5904,"reported_epoch":353,"state":"active+undersized+degraded","last_fresh":"2024-03-20T09:07:19.262767+0000","last_change":"2024-03-20T09:04:34.736993+0000","last_active":"2024-03-20T09:07:19.262767+0000","last_peered":"2024-03-20T09:07:19.262767+0000","last_clean":"2024-03-20T09:04:33.698023+0000","last_became_active":"2024-03-20T09:04:34.736993+0000","last_became_peered":"2024-03-20T09:04:34.736993+0000","last_unstale":"2024-03-20T09:07:19.262767+0000","last_undegraded":"2024-03-20T09:04:34.735838+0000","last_fullsized":"2024-03-20T09:04:34.735696+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":199,"log_size":5578,"log_dups_size":0,"ondisk_log_size":5578,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:07:21.116228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157319168,"num_objects":2400,"num_object_clones":0,"num_object_copies":7200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2400,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3840,"num_write_kb":199712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":["1","11"],"object_location_counts":[{"shards":"1,11","objects":2400}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"20'137","reported_seq":606,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-20T09:07:16.992835+0000","last_change":"2024-03-20T09:06:30.955732+0000","last_active":"2024-03-20T09:07:16.992835+0000","last_peered":"2024-03-20T09:07:16.992835+0000","last_clean":"2024-03-20T09:07:16.992835+0000","last_became_active":"2024-03-20T09:06:30.955286+0000","last_became_peered":"2024-03-20T09:06:30.955286+0000","last_unstale":"2024-03-20T09:07:16.992835+0000","last_undegraded":"2024-03-20T09:07:16.992835+0000","last_fullsized":"2024-03-20T09:07:16.992835+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:04:46.981050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:04:46.981050+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,8],"acting":[1,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"353'8577","reported_seq":9056,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-20T09:07:21.612717+0000","last_change":"2024-03-20T09:04:57.536709+0000","last_active":"2024-03-20T09:07:21.612717+0000","last_peered":"2024-03-20T09:07:21.612717+0000","last_clean":"2024-03-20T09:07:21.612717+0000","last_became_active":"2024-03-20T09:03:30.921890+0000","last_became_peered":"2024-03-20T09:03:30.921890+0000","last_unstale":"2024-03-20T09:07:21.612717+0000","last_undegraded":"2024-03-20T09:07:21.612717+0000","last_fullsized":"2024-03-20T09:07:21.612717+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"285'7948","last_scrub_stamp":"2024-03-20T09:04:57.536605+0000","last_deep_scrub":"285'7948","last_deep_scrub_stamp":"2024-03-20T09:04:57.536605+0000","last_clean_scrub_stamp":"2024-03-20T09:04:57.536605+0000","objects_scrubbed":3080,"log_size":8577,"log_dups_size":0,"ondisk_log_size":8577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"queued for scrub","scrub_duration":5935,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159318016,"num_objects":2431,"num_object_clones":0,"num_object_copies":7293,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1778,"num_write_kb":65920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,10],"acting":[0,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":681,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-20T09:07:16.988945+0000","last_change":"2024-03-20T09:06:02.926003+0000","last_active":"2024-03-20T09:07:16.988945+0000","last_peered":"2024-03-20T09:07:16.988945+0000","last_clean":"2024-03-20T09:07:16.988945+0000","last_became_active":"2024-03-20T09:06:02.925687+0000","last_became_peered":"2024-03-20T09:06:02.925687+0000","last_unstale":"2024-03-20T09:07:16.988945+0000","last_undegraded":"2024-03-20T09:07:16.988945+0000","last_fullsized":"2024-03-20T09:07:16.988945+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:23.383145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:23.383145+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:07:17.940088+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,10],"acting":[11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"353'5651","reported_seq":7299,"reported_epoch":353,"state":"active+undersized+degraded","last_fresh":"2024-03-20T09:07:18.966485+0000","last_change":"2024-03-20T09:04:34.738561+0000","last_active":"2024-03-20T09:07:18.966485+0000","last_peered":"2024-03-20T09:07:18.966485+0000","last_clean":"2024-03-20T09:04:33.645766+0000","last_became_active":"2024-03-20T09:04:34.738561+0000","last_became_peered":"2024-03-20T09:04:34.738561+0000","last_unstale":"2024-03-20T09:07:18.966485+0000","last_undegraded":"2024-03-20T09:04:34.734426+0000","last_fullsized":"2024-03-20T09:04:34.734302+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":5651,"log_dups_size":0,"ondisk_log_size":5651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:07:22.327214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156499968,"num_objects":2388,"num_object_clones":0,"num_object_copies":7164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3876,"num_write_kb":200448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":329,"num_bytes_recovered":21561344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":2388}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"20'137","reported_seq":922,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-20T09:07:16.966427+0000","last_change":"2024-03-20T09:06:02.926788+0000","last_active":"2024-03-20T09:07:16.966427+0000","last_peered":"2024-03-20T09:07:16.966427+0000","last_clean":"2024-03-20T09:07:16.966427+0000","last_became_active":"2024-03-20T09:06:02.926378+0000","last_became_peered":"2024-03-20T09:06:02.926378+0000","last_unstale":"2024-03-20T09:07:16.966427+0000","last_undegraded":"2024-03-20T09:07:16.966427+0000","last_fullsized":"2024-03-20T09:07:16.966427+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:03:59.584956+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:03:59.584956+0000","last_clean_scrub_stamp":"2024-03-20T09:03:59.584956+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,8],"acting":[0,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"353'5539","reported_seq":7242,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-20T09:07:20.952233+0000","last_change":"2024-03-20T09:03:30.928051+0000","last_active":"2024-03-20T09:07:20.952233+0000","last_peered":"2024-03-20T09:07:20.952233+0000","last_clean":"2024-03-20T09:07:20.952233+0000","last_became_active":"2024-03-20T09:03:30.926683+0000","last_became_peered":"2024-03-20T09:03:30.926683+0000","last_unstale":"2024-03-20T09:07:20.952233+0000","last_undegraded":"2024-03-20T09:07:20.952233+0000","last_fullsized":"2024-03-20T09:07:20.952233+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":5539,"log_dups_size":0,"ondisk_log_size":5539,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:07:21.962398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152240128,"num_objects":2323,"num_object_clones":0,"num_object_copies":6969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3795,"num_write_kb":195776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":343,"num_bytes_recovered":22511616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,1],"acting":[11,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1","version":"20'137","reported_seq":644,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-20T09:07:16.939777+0000","last_change":"2024-03-20T09:07:03.025487+0000","last_active":"2024-03-20T09:07:16.939777+0000","last_peered":"2024-03-20T09:07:16.939777+0000","last_clean":"2024-03-20T09:07:16.939777+0000","last_became_active":"2024-03-20T09:06:02.923630+0000","last_became_peered":"2024-03-20T09:06:02.923630+0000","last_unstale":"2024-03-20T09:07:16.939777+0000","last_undegraded":"2024-03-20T09:07:16.939777+0000","last_fullsized":"2024-03-20T09:07:16.939777+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:07:03.025368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:07:03.025368+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:08:11.370212+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2],"acting":[10,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3","version":"353'5655","reported_seq":5973,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-20T09:07:19.684724+0000","last_change":"2024-03-20T09:03:30.939580+0000","last_active":"2024-03-20T09:07:19.684724+0000","last_peered":"2024-03-20T09:07:19.684724+0000","last_clean":"2024-03-20T09:07:19.684724+0000","last_became_active":"2024-03-20T09:03:30.937809+0000","last_became_peered":"2024-03-20T09:03:30.937809+0000","last_unstale":"2024-03-20T09:07:19.684724+0000","last_undegraded":"2024-03-20T09:07:19.684724+0000","last_fullsized":"2024-03-20T09:07:19.684724+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":5655,"log_dups_size":0,"ondisk_log_size":5655,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:07:21.400702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152141824,"num_objects":2321,"num_object_clones":0,"num_object_copies":6963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3861,"num_write_kb":197856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"20'137","reported_seq":611,"reported_epoch":353,"state":"active+undersized","last_fresh":"2024-03-20T09:07:16.941720+0000","last_change":"2024-03-20T09:04:34.737319+0000","last_active":"2024-03-20T09:07:16.941720+0000","last_peered":"2024-03-20T09:07:16.941720+0000","last_clean":"2024-03-20T09:04:29.191837+0000","last_became_active":"2024-03-20T09:04:34.737319+0000","last_became_peered":"2024-03-20T09:04:34.737319+0000","last_unstale":"2024-03-20T09:07:16.941720+0000","last_undegraded":"2024-03-20T09:07:16.941720+0000","last_fullsized":"2024-03-20T09:04:34.733375+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:21.734957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:21.734957+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:07:17.346042+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":["6","10"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"353'5673","reported_seq":7310,"reported_epoch":353,"state":"active+undersized+degraded","last_fresh":"2024-03-20T09:07:22.765486+0000","last_change":"2024-03-20T09:04:34.737379+0000","last_active":"2024-03-20T09:07:22.765486+0000","last_peered":"2024-03-20T09:07:22.765486+0000","last_clean":"2024-03-20T09:04:33.579072+0000","last_became_active":"2024-03-20T09:04:34.737379+0000","last_became_peered":"2024-03-20T09:04:34.737379+0000","last_unstale":"2024-03-20T09:07:22.765486+0000","last_undegraded":"2024-03-20T09:04:34.733476+0000","last_fullsized":"2024-03-20T09:04:34.733363+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"253'4200","last_scrub_stamp":"2024-03-20T09:04:00.025798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:04:00.025798+0000","objects_scrubbed":2428,"log_size":5673,"log_dups_size":0,"ondisk_log_size":5673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154599424,"num_objects":2359,"num_object_clones":0,"num_object_copies":7077,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2359,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1477,"num_write_kb":45056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,6","objects":2359}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"20'137","reported_seq":490,"reported_epoch":353,"state":"active+clean+remapped","last_fresh":"2024-03-20T09:07:16.966468+0000","last_change":"2024-03-20T09:06:02.922773+0000","last_active":"2024-03-20T09:07:16.966468+0000","last_peered":"2024-03-20T09:07:16.966468+0000","last_clean":"2024-03-20T09:07:16.966468+0000","last_became_active":"2024-03-20T09:06:02.922304+0000","last_became_peered":"2024-03-20T09:06:02.922304+0000","last_unstale":"2024-03-20T09:07:16.966468+0000","last_undegraded":"2024-03-20T09:07:16.966468+0000","last_fullsized":"2024-03-20T09:07:16.966468+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:24.936556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:24.936556+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:07:18.020531+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2,11],"avail_no_missing":["0","2","11"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"353'5557","reported_seq":5942,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-20T09:07:21.411577+0000","last_change":"2024-03-20T09:06:58.210889+0000","last_active":"2024-03-20T09:07:21.411577+0000","last_peered":"2024-03-20T09:07:21.411577+0000","last_clean":"2024-03-20T09:07:21.411577+0000","last_became_active":"2024-03-20T09:03:30.910402+0000","last_became_peered":"2024-03-20T09:03:30.910402+0000","last_unstale":"2024-03-20T09:07:21.411577+0000","last_undegraded":"2024-03-20T09:07:21.411577+0000","last_fullsized":"2024-03-20T09:07:21.411577+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'5452","last_scrub_stamp":"2024-03-20T09:06:58.210664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:06:58.210664+0000","objects_scrubbed":2397,"log_size":5557,"log_dups_size":0,"ondisk_log_size":5557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:08:27.441415+0000","scrub_duration":272,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150142976,"num_objects":2291,"num_object_clones":0,"num_object_copies":6873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":707,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"20'137","reported_seq":533,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-20T09:07:16.952627+0000","last_change":"2024-03-20T09:06:02.929720+0000","last_active":"2024-03-20T09:07:16.952627+0000","last_peered":"2024-03-20T09:07:16.952627+0000","last_clean":"2024-03-20T09:07:16.952627+0000","last_became_active":"2024-03-20T09:06:02.929364+0000","last_became_peered":"2024-03-20T09:06:02.929364+0000","last_unstale":"2024-03-20T09:07:16.952627+0000","last_undegraded":"2024-03-20T09:07:16.952627+0000","last_fullsized":"2024-03-20T09:07:16.952627+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:03:55.906959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:03:55.906959+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:07:21.400756+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,1],"acting":[4,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"353'11619","reported_seq":23225,"reported_epoch":353,"state":"active+clean+remapped","last_fresh":"2024-03-20T09:07:19.261915+0000","last_change":"2024-03-20T09:06:08.944363+0000","last_active":"2024-03-20T09:07:19.261915+0000","last_peered":"2024-03-20T09:07:19.261915+0000","last_clean":"2024-03-20T09:07:19.261915+0000","last_became_active":"2024-03-20T09:06:08.928778+0000","last_became_peered":"2024-03-20T09:06:08.928778+0000","last_unstale":"2024-03-20T09:07:19.261915+0000","last_undegraded":"2024-03-20T09:07:19.261915+0000","last_fullsized":"2024-03-20T09:07:19.261915+0000","mapping_epoch":313,"log_start":"221'1600","ondisk_log_start":"221'1600","created":204,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"270'9729","last_scrub_stamp":"2024-03-20T09:04:24.105978+0000","last_deep_scrub":"270'9729","last_deep_scrub_stamp":"2024-03-20T09:04:24.105978+0000","last_clean_scrub_stamp":"2024-03-20T09:04:24.105978+0000","objects_scrubbed":5255,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":26,"scrub_schedule":"queued for scrub","scrub_duration":25793,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316342272,"num_objects":4827,"num_object_clones":0,"num_object_copies":14481,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4827,"num_objects_unfound":0,"num_objects_dirty":4827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7771,"num_write_kb":403136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5528,"num_bytes_recovered":362283008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10,3],"avail_no_missing":["6","3","10"],"object_location_counts":[{"shards":"3,6,10","objects":4827}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"20'137","reported_seq":618,"reported_epoch":353,"state":"active+undersized","last_fresh":"2024-03-20T09:07:16.941528+0000","last_change":"2024-03-20T09:04:34.737366+0000","last_active":"2024-03-20T09:07:16.941528+0000","last_peered":"2024-03-20T09:07:16.941528+0000","last_clean":"2024-03-20T09:04:29.191733+0000","last_became_active":"2024-03-20T09:04:34.737366+0000","last_became_peered":"2024-03-20T09:04:34.737366+0000","last_unstale":"2024-03-20T09:07:16.941528+0000","last_undegraded":"2024-03-20T09:07:16.941528+0000","last_fullsized":"2024-03-20T09:04:34.733742+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:21.734957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:21.734957+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:07:17.346030+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"353'7611","reported_seq":10844,"reported_epoch":353,"state":"active+recovering+degraded+remapped","last_fresh":"2024-03-20T09:07:19.282921+0000","last_change":"2024-03-20T09:04:47.242963+0000","last_active":"2024-03-20T09:07:19.282921+0000","last_peered":"2024-03-20T09:07:19.282921+0000","last_clean":"2024-03-20T09:04:10.072458+0000","last_became_active":"2024-03-20T09:04:46.620359+0000","last_became_peered":"2024-03-20T09:04:46.620359+0000","last_unstale":"2024-03-20T09:07:19.282921+0000","last_undegraded":"2024-03-20T09:04:46.424109+0000","last_fullsized":"2024-03-20T09:07:19.282921+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'5229","last_scrub_stamp":"2024-03-20T09:04:09.606259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:04:09.606259+0000","objects_scrubbed":5193,"log_size":7611,"log_dups_size":0,"ondisk_log_size":7611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312213528,"num_objects":4765,"num_object_clones":0,"num_object_copies":14295,"num_objects_missing_on_primary":3808,"num_objects_missing":3808,"num_objects_degraded":4765,"num_objects_misplaced":3808,"num_objects_unfound":0,"num_objects_dirty":4765,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7611,"num_write_kb":395969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1423,"num_bytes_recovered":93257728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7,9],"avail_no_missing":["7"],"object_location_counts":[{"shards":"6,7","objects":944},{"shards":"6,7,9","objects":1},{"shards":"7,9","objects":3808}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"20'137","reported_seq":665,"reported_epoch":353,"state":"active+undersized+degraded","last_fresh":"2024-03-20T09:07:16.966554+0000","last_change":"2024-03-20T09:04:34.740608+0000","last_active":"2024-03-20T09:07:16.966554+0000","last_peered":"2024-03-20T09:07:16.966554+0000","last_clean":"2024-03-20T09:04:29.200818+0000","last_became_active":"2024-03-20T09:04:34.740608+0000","last_became_peered":"2024-03-20T09:04:34.740608+0000","last_unstale":"2024-03-20T09:07:16.966554+0000","last_undegraded":"2024-03-20T09:04:34.735008+0000","last_fullsized":"2024-03-20T09:04:34.734889+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:04:04.682759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:04:04.682759+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[{"shards":"0,6","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"353'7707","reported_seq":27398,"reported_epoch":353,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-20T09:07:20.406274+0000","last_change":"2024-03-20T09:07:06.560609+0000","last_active":"2024-03-20T09:07:20.406274+0000","last_peered":"2024-03-20T09:07:20.406274+0000","last_clean":"2024-03-20T09:04:10.062034+0000","last_became_active":"2024-03-20T09:06:29.055447+0000","last_became_peered":"2024-03-20T09:06:29.055447+0000","last_unstale":"2024-03-20T09:07:20.406274+0000","last_undegraded":"2024-03-20T09:07:20.406274+0000","last_fullsized":"2024-03-20T09:06:28.937531+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":7707,"log_dups_size":0,"ondisk_log_size":7707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303890432,"num_objects":4637,"num_object_clones":0,"num_object_copies":13911,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4904,"num_objects_unfound":0,"num_objects_dirty":4637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7707,"num_write_kb":395008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9290,"num_bytes_recovered":608829440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,10],"acting":[4,10],"avail_no_missing":["4","9","11"],"object_location_counts":[{"shards":"3,4,9,10,11","objects":1386},{"shards":"3,4,11","objects":912},{"shards":"4,9,10,11","objects":324},{"shards":"4,9,11","objects":685},{"shards":"4,10","objects":1330}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"20'137","reported_seq":444,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-20T09:07:16.952370+0000","last_change":"2024-03-20T09:06:11.654156+0000","last_active":"2024-03-20T09:07:16.952370+0000","last_peered":"2024-03-20T09:07:16.952370+0000","last_clean":"2024-03-20T09:07:16.952370+0000","last_became_active":"2024-03-20T09:06:11.653699+0000","last_became_peered":"2024-03-20T09:06:11.653699+0000","last_unstale":"2024-03-20T09:07:16.952370+0000","last_undegraded":"2024-03-20T09:07:16.952370+0000","last_fullsized":"2024-03-20T09:07:16.952370+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:03:48.928433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:03:48.928433+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:07:21.400744+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":9,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":48085,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"353'7613","reported_seq":20247,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-20T09:07:21.608952+0000","last_change":"2024-03-20T09:06:59.037447+0000","last_active":"2024-03-20T09:07:21.608952+0000","last_peered":"2024-03-20T09:07:21.608952+0000","last_clean":"2024-03-20T09:07:21.608952+0000","last_became_active":"2024-03-20T09:06:59.023790+0000","last_became_peered":"2024-03-20T09:06:59.023790+0000","last_unstale":"2024-03-20T09:07:21.608952+0000","last_undegraded":"2024-03-20T09:07:21.608952+0000","last_fullsized":"2024-03-20T09:07:21.608952+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":7613,"log_dups_size":0,"ondisk_log_size":7613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307822592,"num_objects":4697,"num_object_clones":0,"num_object_copies":14091,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7613,"num_write_kb":393920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6155,"num_bytes_recovered":345112576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,10],"acting":[11,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.12","version":"20'137","reported_seq":868,"reported_epoch":353,"state":"active+undersized","last_fresh":"2024-03-20T09:07:16.992850+0000","last_change":"2024-03-20T09:04:34.751346+0000","last_active":"2024-03-20T09:07:16.992850+0000","last_peered":"2024-03-20T09:07:16.992850+0000","last_clean":"2024-03-20T09:03:59.001853+0000","last_became_active":"2024-03-20T09:04:34.751346+0000","last_became_peered":"2024-03-20T09:04:34.751346+0000","last_unstale":"2024-03-20T09:07:16.992850+0000","last_undegraded":"2024-03-20T09:07:16.992850+0000","last_fullsized":"2024-03-20T09:04:34.735354+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:04:05.319747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:04:05.319747+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:07:17.114885+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"353'5575","reported_seq":19207,"reported_epoch":353,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-20T09:07:19.295515+0000","last_change":"2024-03-20T09:06:43.134508+0000","last_active":"2024-03-20T09:07:19.295515+0000","last_peered":"2024-03-20T09:07:19.295515+0000","last_clean":"2024-03-20T09:05:58.734145+0000","last_became_active":"2024-03-20T09:06:01.859835+0000","last_became_peered":"2024-03-20T09:06:01.859835+0000","last_unstale":"2024-03-20T09:07:19.295515+0000","last_undegraded":"2024-03-20T09:06:01.669030+0000","last_fullsized":"2024-03-20T09:06:01.645488+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":5575,"log_dups_size":0,"ondisk_log_size":5575,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155516928,"num_objects":2373,"num_object_clones":0,"num_object_copies":7119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2373,"num_objects_misplaced":2450,"num_objects_unfound":0,"num_objects_dirty":2373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3831,"num_write_kb":198528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5408,"num_bytes_recovered":351698944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,2],"avail_no_missing":["6","2","9"],"object_location_counts":[{"shards":"2,6","objects":-77},{"shards":"2,6,9","objects":2299}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.13","version":"20'137","reported_seq":604,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-20T09:07:16.992765+0000","last_change":"2024-03-20T09:04:48.832913+0000","last_active":"2024-03-20T09:07:16.992765+0000","last_peered":"2024-03-20T09:07:16.992765+0000","last_clean":"2024-03-20T09:07:16.992765+0000","last_became_active":"2024-03-20T09:04:04.103645+0000","last_became_peered":"2024-03-20T09:04:04.103645+0000","last_unstale":"2024-03-20T09:07:16.992765+0000","last_undegraded":"2024-03-20T09:07:16.992765+0000","last_fullsized":"2024-03-20T09:07:16.992765+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:04:48.832800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:04:48.832800+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"353'5698","reported_seq":9827,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-20T09:07:20.828294+0000","last_change":"2024-03-20T09:07:01.247863+0000","last_active":"2024-03-20T09:07:20.828294+0000","last_peered":"2024-03-20T09:07:20.828294+0000","last_clean":"2024-03-20T09:07:20.828294+0000","last_became_active":"2024-03-20T09:03:46.901072+0000","last_became_peered":"2024-03-20T09:03:46.901072+0000","last_unstale":"2024-03-20T09:07:20.828294+0000","last_undegraded":"2024-03-20T09:07:20.828294+0000","last_fullsized":"2024-03-20T09:07:20.828294+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":5,"last_scrub":"345'5607","last_scrub_stamp":"2024-03-20T09:07:01.247738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:07:01.247738+0000","objects_scrubbed":2440,"log_size":5698,"log_dups_size":0,"ondisk_log_size":5698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:08:11.482392+0000","scrub_duration":262,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153878528,"num_objects":2348,"num_object_clones":0,"num_object_copies":7044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1824,"num_write_kb":65984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,6],"acting":[10,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.10","version":"20'137","reported_seq":396,"reported_epoch":353,"state":"active+undersized","last_fresh":"2024-03-20T09:07:16.939710+0000","last_change":"2024-03-20T09:04:34.735801+0000","last_active":"2024-03-20T09:07:16.939710+0000","last_peered":"2024-03-20T09:07:16.939710+0000","last_clean":"2024-03-20T08:54:29.262652+0000","last_became_active":"2024-03-20T09:04:34.735801+0000","last_became_peered":"2024-03-20T09:04:34.735801+0000","last_unstale":"2024-03-20T09:07:16.939710+0000","last_undegraded":"2024-03-20T09:07:16.939710+0000","last_fullsized":"2024-03-20T09:04:34.732688+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:04:04.319316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:04:04.319316+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:07:17.031979+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":8,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":48084,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":["10","7"],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.12","version":"353'5635","reported_seq":20917,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-20T09:07:21.723504+0000","last_change":"2024-03-20T09:06:44.957783+0000","last_active":"2024-03-20T09:07:21.723504+0000","last_peered":"2024-03-20T09:07:21.723504+0000","last_clean":"2024-03-20T09:07:21.723504+0000","last_became_active":"2024-03-20T09:06:44.949696+0000","last_became_peered":"2024-03-20T09:06:44.949696+0000","last_unstale":"2024-03-20T09:07:21.723504+0000","last_undegraded":"2024-03-20T09:07:21.723504+0000","last_fullsized":"2024-03-20T09:07:21.723504+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":5,"last_scrub":"285'4956","last_scrub_stamp":"2024-03-20T09:04:50.135501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:04:50.135501+0000","objects_scrubbed":3023,"log_size":5635,"log_dups_size":0,"ondisk_log_size":5635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":356,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153550848,"num_objects":2343,"num_object_clones":0,"num_object_copies":7029,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":680,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3104,"num_bytes_recovered":169148416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"20'137","reported_seq":475,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-20T09:07:16.966425+0000","last_change":"2024-03-20T09:06:02.928303+0000","last_active":"2024-03-20T09:07:16.966425+0000","last_peered":"2024-03-20T09:07:16.966425+0000","last_clean":"2024-03-20T09:07:16.966425+0000","last_became_active":"2024-03-20T09:06:02.927902+0000","last_became_peered":"2024-03-20T09:06:02.927902+0000","last_unstale":"2024-03-20T09:07:16.966425+0000","last_undegraded":"2024-03-20T09:07:16.966425+0000","last_fullsized":"2024-03-20T09:07:16.966425+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:04:42.894571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:04:42.894571+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1],"acting":[0,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"353'5638","reported_seq":11440,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-20T09:07:19.823978+0000","last_change":"2024-03-20T09:03:56.895356+0000","last_active":"2024-03-20T09:07:19.823978+0000","last_peered":"2024-03-20T09:07:19.823978+0000","last_clean":"2024-03-20T09:07:19.823978+0000","last_became_active":"2024-03-20T09:03:56.891781+0000","last_became_peered":"2024-03-20T09:03:56.891781+0000","last_unstale":"2024-03-20T09:07:19.823978+0000","last_undegraded":"2024-03-20T09:07:19.823978+0000","last_fullsized":"2024-03-20T09:07:19.823978+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":5638,"log_dups_size":0,"ondisk_log_size":5638,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:07:21.400731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159154176,"num_objects":2429,"num_object_clones":0,"num_object_copies":7287,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3845,"num_write_kb":200736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1825,"num_bytes_recovered":119603200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"353'5509","reported_seq":11214,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-20T09:07:19.844901+0000","last_change":"2024-03-20T09:03:52.899809+0000","last_active":"2024-03-20T09:07:19.844901+0000","last_peered":"2024-03-20T09:07:19.844901+0000","last_clean":"2024-03-20T09:07:19.844901+0000","last_became_active":"2024-03-20T09:03:52.897604+0000","last_became_peered":"2024-03-20T09:03:52.897604+0000","last_unstale":"2024-03-20T09:07:19.844901+0000","last_undegraded":"2024-03-20T09:07:19.844901+0000","last_fullsized":"2024-03-20T09:07:19.844901+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":5509,"log_dups_size":0,"ondisk_log_size":5509,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:07:21.400718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153714688,"num_objects":2346,"num_object_clones":0,"num_object_copies":7038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3772,"num_write_kb":195744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1792,"num_bytes_recovered":117440512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,11],"acting":[4,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"353'5737","reported_seq":11061,"reported_epoch":353,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-20T09:07:21.751782+0000","last_change":"2024-03-20T09:06:59.467085+0000","last_active":"2024-03-20T09:07:21.751782+0000","last_peered":"2024-03-20T09:07:21.751782+0000","last_clean":"2024-03-20T09:03:31.866916+0000","last_became_active":"2024-03-20T09:06:11.451333+0000","last_became_peered":"2024-03-20T09:06:11.451333+0000","last_unstale":"2024-03-20T09:07:21.751782+0000","last_undegraded":"2024-03-20T09:06:05.930917+0000","last_fullsized":"2024-03-20T09:06:01.646279+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":5737,"log_dups_size":0,"ondisk_log_size":5737,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161316864,"num_objects":2462,"num_object_clones":0,"num_object_copies":7386,"num_objects_missing_on_primary":927,"num_objects_missing":927,"num_objects_degraded":1253,"num_objects_misplaced":5060,"num_objects_unfound":0,"num_objects_dirty":2462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3952,"num_write_kb":205216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1228,"num_bytes_recovered":80379904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,0],"acting":[11,4],"avail_no_missing":["2","3","6"],"object_location_counts":[{"shards":"2,4,6,7,11","objects":85},{"shards":"2,6","objects":927},{"shards":"4,7,11","objects":1231},{"shards":"4,11","objects":-231},{"shards":"6,7,8,11","objects":1},{"shards":"7,8,11","objects":123}],"blocked_by":[],"up_primary":8,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.14","version":"20'137","reported_seq":680,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-20T09:07:16.986877+0000","last_change":"2024-03-20T09:06:24.048877+0000","last_active":"2024-03-20T09:07:16.986877+0000","last_peered":"2024-03-20T09:07:16.986877+0000","last_clean":"2024-03-20T09:07:16.986877+0000","last_became_active":"2024-03-20T09:04:14.208448+0000","last_became_peered":"2024-03-20T09:04:14.208448+0000","last_unstale":"2024-03-20T09:07:16.986877+0000","last_undegraded":"2024-03-20T09:07:16.986877+0000","last_fullsized":"2024-03-20T09:07:16.986877+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:06:24.048708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:06:24.048708+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:07:50.013768+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"353'5726","reported_seq":17512,"reported_epoch":353,"state":"active+clean+remapped","last_fresh":"2024-03-20T09:07:18.696906+0000","last_change":"2024-03-20T09:06:18.953397+0000","last_active":"2024-03-20T09:07:18.696906+0000","last_peered":"2024-03-20T09:07:18.696906+0000","last_clean":"2024-03-20T09:07:18.696906+0000","last_became_active":"2024-03-20T09:06:18.946699+0000","last_became_peered":"2024-03-20T09:06:18.946699+0000","last_unstale":"2024-03-20T09:07:18.696906+0000","last_undegraded":"2024-03-20T09:07:18.696906+0000","last_fullsized":"2024-03-20T09:07:18.696906+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":5,"last_scrub":"255'4276","last_scrub_stamp":"2024-03-20T09:04:01.842913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:04:01.842913+0000","objects_scrubbed":2457,"log_size":5726,"log_dups_size":0,"ondisk_log_size":5726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":540,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158138368,"num_objects":2413,"num_object_clones":0,"num_object_copies":7239,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2413,"num_objects_unfound":0,"num_objects_dirty":2413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1458,"num_write_kb":45248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3164,"num_bytes_recovered":178585600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2,7],"avail_no_missing":["1","2","7"],"object_location_counts":[{"shards":"1,2,7","objects":2413}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"353'5612","reported_seq":19147,"reported_epoch":353,"state":"active+clean+remapped","last_fresh":"2024-03-20T09:07:21.420003+0000","last_change":"2024-03-20T09:06:46.990481+0000","last_active":"2024-03-20T09:07:21.420003+0000","last_peered":"2024-03-20T09:07:21.420003+0000","last_clean":"2024-03-20T09:07:21.420003+0000","last_became_active":"2024-03-20T09:06:46.981557+0000","last_became_peered":"2024-03-20T09:06:46.981557+0000","last_unstale":"2024-03-20T09:07:21.420003+0000","last_undegraded":"2024-03-20T09:07:21.420003+0000","last_fullsized":"2024-03-20T09:07:21.420003+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":5,"last_scrub":"251'4129","last_scrub_stamp":"2024-03-20T09:03:57.865687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:03:57.865687+0000","objects_scrubbed":2325,"log_size":5612,"log_dups_size":0,"ondisk_log_size":5612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158859264,"num_objects":2424,"num_object_clones":0,"num_object_copies":7272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2424,"num_objects_unfound":0,"num_objects_dirty":2424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1491,"num_write_kb":50880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3120,"num_bytes_recovered":173342720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1,6],"avail_no_missing":["0","1","6"],"object_location_counts":[{"shards":"0,1,6","objects":2424}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"353'5531","reported_seq":19139,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-20T09:07:19.861784+0000","last_change":"2024-03-20T09:06:24.997013+0000","last_active":"2024-03-20T09:07:19.861784+0000","last_peered":"2024-03-20T09:07:19.861784+0000","last_clean":"2024-03-20T09:07:19.861784+0000","last_became_active":"2024-03-20T09:06:24.988319+0000","last_became_peered":"2024-03-20T09:06:24.988319+0000","last_unstale":"2024-03-20T09:07:19.861784+0000","last_undegraded":"2024-03-20T09:07:19.861784+0000","last_fullsized":"2024-03-20T09:07:19.861784+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":5,"last_scrub":"276'4872","last_scrub_stamp":"2024-03-20T09:04:44.020587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:04:44.020587+0000","objects_scrubbed":2997,"log_size":5531,"log_dups_size":0,"ondisk_log_size":5531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153223168,"num_objects":2338,"num_object_clones":0,"num_object_copies":7014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":659,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3042,"num_bytes_recovered":170590208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,8],"acting":[1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":4979621912,"num_objects":75984,"num_object_clones":0,"num_object_copies":227952,"num_objects_missing_on_primary":4735,"num_objects_missing":4735,"num_objects_degraded":15538,"num_objects_misplaced":30485,"num_objects_unfound":0,"num_objects_dirty":75984,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":93986,"num_write_kb":4509121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69798,"num_bytes_recovered":4242112512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17678016512,"data_stored":17678008368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":164113,"ondisk_log_size":164113,"up":70,"acting":71,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":806934,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":23,"num_read_kb":20,"num_write":33,"num_write_kb":535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":647065,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2007040,"data_stored":1989280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2877,"ondisk_log_size":2877,"up":55,"acting":57,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860552,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2120720,"kb_used_data":1945160,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":91624432,"statfs":{"total":95995035648,"available":93823418368,"internally_reserved":0,"allocated":1991843840,"data_stored":1991202593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":179763652},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":89,"seq":382252089485,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1430096,"kb_used_data":1276744,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":92315056,"statfs":{"total":95995035648,"available":94530617344,"internally_reserved":0,"allocated":1307385856,"data_stored":1306744609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":157022656},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":0,"seq":343597383829,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":70,"seq":300647710873,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1023768,"kb_used_data":875984,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":92721384,"statfs":{"total":95995035648,"available":94946697216,"internally_reserved":0,"allocated":897007616,"data_stored":896362817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":151321030},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":60,"seq":257698037918,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1509460,"kb_used_data":1217292,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":92235692,"statfs":{"total":95995035648,"available":94449348608,"internally_reserved":0,"allocated":1246507008,"data_stored":1245861689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":299170250},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332257,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2038932,"kb_used_data":1859212,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":91706220,"statfs":{"total":95995035648,"available":93907169280,"internally_reserved":0,"allocated":1903833088,"data_stored":1903187769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":184023494},"hb_peers":[0,1,2,3,4,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575037,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1594512,"kb_used_data":1382856,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":92150640,"statfs":{"total":95995035648,"available":94362255360,"internally_reserved":0,"allocated":1416044544,"data_stored":1415403297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":216725958},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738564,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1531344,"kb_used_data":1319688,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":92213808,"statfs":{"total":95995035648,"available":94426939392,"internally_reserved":0,"allocated":1351360512,"data_stored":1350719265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411511,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1181464,"kb_used_data":1023952,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":92563688,"statfs":{"total":95995035648,"available":94785216512,"internally_reserved":0,"allocated":1048526848,"data_stored":1047882049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161282501},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":111669149874,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953647,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2785560,"kb_used_data":2523920,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":90959592,"statfs":{"total":95995035648,"available":93142622208,"internally_reserved":0,"allocated":2584494080,"data_stored":2583849281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":267909573},"hb_peers":[0,1,2,3,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659271,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2258332,"kb_used_data":2123028,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":91486820,"statfs":{"total":95995035648,"available":93682503680,"internally_reserved":0,"allocated":2173980672,"data_stored":2173328093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1349648384,"data_stored":1349648384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1411842048,"data_stored":1411842048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1044774912,"data_stored":1044774912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1013907456,"data_stored":1013907456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2582446080,"data_stored":2582446080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2174091264,"data_stored":2174091264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1901268992,"data_stored":1901264920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1243353088,"data_stored":1243349016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":893779968,"data_stored":893779968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":769130496,"data_stored":769130496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1305870336,"data_stored":1305870336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1987903488,"data_stored":1987903488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:07:26.579 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-20T09:07:26.579 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pool set .mgr pg_num 31 2024-03-20T09:07:26.580 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:07:26.588 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:07:26.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:26.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:07:26.691 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:07:26.723 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:26.799 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:07:26.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:26.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:07:26.904 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:07:26.935 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:26.959 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:07:26.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:26.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:07:27.064 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:07:27.089 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:27.108 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:07:27.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:27.111 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:07:27.212 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:07:27.238 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:27.258 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:07:27.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:27.261 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:07:27.318 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:07:27.362 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:07:27.385 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:27 smithi122 ceph-mon[73495]: pgmap v878: 47 pgs: 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 143 op/s; 15415/226725 objects degraded (6.799%); 29769/226725 objects misplaced (13.130%); 4.4 MiB/s, 71 objects/s recovering 2024-03-20T09:07:27.386 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:27 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 15415/226725 objects degraded (6.799%), 7 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:07:27.388 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:27.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:27 smithi115 ceph-mon[72476]: pgmap v878: 47 pgs: 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 143 op/s; 15415/226725 objects degraded (6.799%); 29769/226725 objects misplaced (13.130%); 4.4 MiB/s, 71 objects/s recovering 2024-03-20T09:07:27.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:27 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 15415/226725 objects degraded (6.799%), 7 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:07:27.408 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:07:27.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:27.411 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:07:27.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:27 smithi026 ceph-mon[68309]: pgmap v878: 47 pgs: 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 143 op/s; 15415/226725 objects degraded (6.799%); 29769/226725 objects misplaced (13.130%); 4.4 MiB/s, 71 objects/s recovering 2024-03-20T09:07:27.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:27 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 15415/226725 objects degraded (6.799%), 7 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:07:27.512 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:07:27.541 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:27.561 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:07:27.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:27.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:07:27.665 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:07:27.694 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:27.715 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:07:27.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:27.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:07:27.820 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:07:27.848 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:27.871 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:07:27.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:27.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:07:27.975 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:07:28.003 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:28.024 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:07:28.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:28.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:07:28.121 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:07:28.128 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:07:28.155 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:28.176 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:07:28.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:28.179 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:07:28.280 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:07:28.311 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:28.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:28 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:07:28.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:28 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:07:28.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:28 smithi115 ceph-mon[72476]: osdmap e354: 12 total, 11 up, 10 in 2024-03-20T09:07:28.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:28 smithi115 ceph-mon[72476]: from='client.25270 172.21.15.26:0/1862959224' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:07:28.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:28 smithi115 ceph-mon[72476]: from='client.25270 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:07:28.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:28 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:07:28.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:28 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:07:28.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:28 smithi122 ceph-mon[73495]: osdmap e354: 12 total, 11 up, 10 in 2024-03-20T09:07:28.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:28 smithi122 ceph-mon[73495]: from='client.25270 172.21.15.26:0/1862959224' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:07:28.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:28 smithi122 ceph-mon[73495]: from='client.25270 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:07:28.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:28 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:07:28.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:28 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:07:28.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:28 smithi026 ceph-mon[68309]: osdmap e354: 12 total, 11 up, 10 in 2024-03-20T09:07:28.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:28 smithi026 ceph-mon[68309]: from='client.25270 172.21.15.26:0/1862959224' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:07:28.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:28 smithi026 ceph-mon[68309]: from='client.25270 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:07:29.095 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:07:29.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:29 smithi115 ceph-mon[72476]: pgmap v880: 47 pgs: 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 152 op/s; 15427/225696 objects degraded (6.835%); 29679/225696 objects misplaced (13.150%); 4.5 MiB/s, 71 objects/s recovering 2024-03-20T09:07:29.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:29 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:07:29.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:29 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:07:29.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:29 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:07:29.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:29 smithi115 ceph-mon[72476]: from='client.25270 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:07:29.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:29 smithi115 ceph-mon[72476]: osdmap e355: 12 total, 11 up, 10 in 2024-03-20T09:07:29.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:29 smithi122 ceph-mon[73495]: pgmap v880: 47 pgs: 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 152 op/s; 15427/225696 objects degraded (6.835%); 29679/225696 objects misplaced (13.150%); 4.5 MiB/s, 71 objects/s recovering 2024-03-20T09:07:29.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:29 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:07:29.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:29 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:07:29.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:29 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:07:29.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:29 smithi122 ceph-mon[73495]: from='client.25270 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:07:29.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:29 smithi122 ceph-mon[73495]: osdmap e355: 12 total, 11 up, 10 in 2024-03-20T09:07:29.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:29 smithi026 ceph-mon[68309]: pgmap v880: 47 pgs: 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 152 op/s; 15427/225696 objects degraded (6.835%); 29679/225696 objects misplaced (13.150%); 4.5 MiB/s, 71 objects/s recovering 2024-03-20T09:07:29.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:29 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:07:29.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:29 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:07:29.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:29 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:07:29.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:29 smithi026 ceph-mon[68309]: from='client.25270 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:07:29.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:29 smithi026 ceph-mon[68309]: osdmap e355: 12 total, 11 up, 10 in 2024-03-20T09:07:30.536 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:07:30.573 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:07:30.576 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:07:30.597 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:07:30.618 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:07:30.640 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:07:30.666 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:07:30.686 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:07:30.709 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:07:30.732 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:07:30.753 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:07:30.753 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:07:30.753 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:07:30.753 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:07:30.753 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:07:30.754 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:07:30.754 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-20T09:07:30.754 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:07:30.754 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:07:30.754 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:07:30.754 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:07:30.754 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:07:30.754 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:07:31.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:31 smithi115 ceph-mon[72476]: pgmap v882: 47 pgs: 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 152 op/s; 15326/225195 objects degraded (6.806%); 29604/225195 objects misplaced (13.146%); 3.3 MiB/s, 52 objects/s recovering 2024-03-20T09:07:31.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:31 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3819381992' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:07:31.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:31 smithi122 ceph-mon[73495]: pgmap v882: 47 pgs: 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 152 op/s; 15326/225195 objects degraded (6.806%); 29604/225195 objects misplaced (13.146%); 3.3 MiB/s, 52 objects/s recovering 2024-03-20T09:07:31.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:31 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3819381992' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:07:31.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:31 smithi026 ceph-mon[68309]: pgmap v882: 47 pgs: 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 152 op/s; 15326/225195 objects degraded (6.806%); 29604/225195 objects misplaced (13.146%); 3.3 MiB/s, 52 objects/s recovering 2024-03-20T09:07:31.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:31 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3819381992' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:07:31.866 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:07:31.866 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:07:32.160 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:32 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 15326/225195 objects degraded (6.806%), 7 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-03-20T09:07:32.163 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:07:32.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:32.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:07:32.192 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:07:32.193 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:07:32.269 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:07:32.301 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:32.323 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:07:32.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:32.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:07:32.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:32 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 15326/225195 objects degraded (6.806%), 7 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-03-20T09:07:32.432 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:07:32.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:32 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 15326/225195 objects degraded (6.806%), 7 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-03-20T09:07:32.458 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:32.477 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:07:32.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:32.479 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:07:32.580 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:07:32.610 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:32.683 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:07:32.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:32.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:07:32.787 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:07:32.814 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:32.833 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:07:32.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:32.836 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:07:32.937 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:07:32.966 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:33.065 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:07:33.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:33.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:07:33.176 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:07:33.205 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:33.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:33 smithi026 ceph-mon[68309]: pgmap v883: 47 pgs: 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 142 op/s; 15309/224529 objects degraded (6.818%); 28922/224529 objects misplaced (12.881%); 5.8 MiB/s, 92 objects/s recovering 2024-03-20T09:07:33.230 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:07:33.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:33.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:07:33.334 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:07:33.361 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:33.380 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:07:33.382 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:33 smithi122 ceph-mon[73495]: pgmap v883: 47 pgs: 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 142 op/s; 15309/224529 objects degraded (6.818%); 28922/224529 objects misplaced (12.881%); 5.8 MiB/s, 92 objects/s recovering 2024-03-20T09:07:33.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:33.383 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:07:33.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:33 smithi115 ceph-mon[72476]: pgmap v883: 47 pgs: 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 142 op/s; 15309/224529 objects degraded (6.818%); 28922/224529 objects misplaced (12.881%); 5.8 MiB/s, 92 objects/s recovering 2024-03-20T09:07:33.420 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-20T09:07:33.484 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:07:33.514 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:33.645 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:07:33.714 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:07:33.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:33.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:07:33.819 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:07:33.851 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:33.951 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:07:34.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:34.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:07:34.021 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:07:34.122 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:07:34.150 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:34.170 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:07:34.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:34.173 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:07:34.194 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:07:34.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:34 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:07:34.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:34 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:07:34.267 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:07:34.274 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:07:34.300 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:34.320 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:07:34.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:34.322 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:07:34.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:34 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:07:34.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:34 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:07:34.423 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:07:34.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:34 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:07:34.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:34 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:07:34.452 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:34.470 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:07:34.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:34.473 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:07:34.574 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:07:34.602 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:34.624 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:07:34.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:34.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:07:34.729 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:07:34.756 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:34.776 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:07:34.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:34.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:07:34.880 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:07:34.908 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:34.932 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:07:34.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:34.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:07:35.037 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:07:35.064 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:35.083 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:07:35.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:35.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:07:35.140 INFO:teuthology.orchestra.run.smithi026.stderr:set pool 1 pg_num to 31 2024-03-20T09:07:35.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:35 smithi115 ceph-mon[72476]: pgmap v884: 47 pgs: 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 125 op/s; 15296/223707 objects degraded (6.838%); 28842/223707 objects misplaced (12.893%); 3.2 MiB/s, 51 objects/s recovering 2024-03-20T09:07:35.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:35 smithi115 ceph-mon[72476]: osdmap e356: 12 total, 11 up, 10 in 2024-03-20T09:07:35.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:35 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/714983013' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-03-20T09:07:35.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:35 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-03-20T09:07:35.187 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:07:35.219 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:35.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:35 smithi122 ceph-mon[73495]: pgmap v884: 47 pgs: 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 125 op/s; 15296/223707 objects degraded (6.838%); 28842/223707 objects misplaced (12.893%); 3.2 MiB/s, 51 objects/s recovering 2024-03-20T09:07:35.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:35 smithi122 ceph-mon[73495]: osdmap e356: 12 total, 11 up, 10 in 2024-03-20T09:07:35.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:35 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/714983013' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-03-20T09:07:35.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:35 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-03-20T09:07:35.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:35 smithi026 ceph-mon[68309]: pgmap v884: 47 pgs: 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 125 op/s; 15296/223707 objects degraded (6.838%); 28842/223707 objects misplaced (12.893%); 3.2 MiB/s, 51 objects/s recovering 2024-03-20T09:07:35.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:35 smithi026 ceph-mon[68309]: osdmap e356: 12 total, 11 up, 10 in 2024-03-20T09:07:35.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:35 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/714983013' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-03-20T09:07:35.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:35 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]: dispatch 2024-03-20T09:07:35.870 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:07:35.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:35.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:07:35.973 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:07:36.000 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:36.022 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:07:36.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:36.025 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:07:36.126 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:07:36.158 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:36.177 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:07:36.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:36.180 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:07:36.282 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:07:36.306 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:36 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-03-20T09:07:36.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:36 smithi122 ceph-mon[73495]: osdmap e357: 12 total, 11 up, 10 in 2024-03-20T09:07:36.307 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:36 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-20T09:07:36.308 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:36.327 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:07:36.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:36.330 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:07:36.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:36 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-03-20T09:07:36.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:36 smithi115 ceph-mon[72476]: osdmap e357: 12 total, 11 up, 10 in 2024-03-20T09:07:36.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:36 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-20T09:07:36.431 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:07:36.457 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:36.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:36 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-03-20T09:07:36.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:36 smithi026 ceph-mon[68309]: osdmap e357: 12 total, 11 up, 10 in 2024-03-20T09:07:36.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:36 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-20T09:07:36.476 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:07:36.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:36.478 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:07:36.579 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:07:36.609 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:36.737 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:07:36.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:36.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:07:36.845 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:07:36.872 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:36.891 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:07:36.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:36.894 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:07:36.995 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:07:37.023 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:37.045 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:07:37.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:37.053 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:07:37.154 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:07:37.186 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:37.225 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:07:37.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:37.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:07:37.257 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:37 smithi026 ceph-mon[68309]: pgmap v887: 47 pgs: 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.5 GiB data, 17 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 135 op/s; 15183/222615 objects degraded (6.820%); 28003/222615 objects misplaced (12.579%); 6.7 MiB/s, 107 objects/s recovering 2024-03-20T09:07:37.257 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:37 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-03-20T09:07:37.257 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:37 smithi026 ceph-mon[68309]: osdmap e358: 12 total, 11 up, 10 in 2024-03-20T09:07:37.257 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:37 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 15183/222615 objects degraded (6.820%), 7 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-03-20T09:07:37.257 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:37 smithi026 ceph-mon[68309]: osdmap e359: 12 total, 11 up, 10 in 2024-03-20T09:07:37.329 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:07:37.365 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:37.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:37 smithi115 ceph-mon[72476]: pgmap v887: 47 pgs: 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.5 GiB data, 17 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 135 op/s; 15183/222615 objects degraded (6.820%); 28003/222615 objects misplaced (12.579%); 6.7 MiB/s, 107 objects/s recovering 2024-03-20T09:07:37.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:37 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-03-20T09:07:37.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:37 smithi115 ceph-mon[72476]: osdmap e358: 12 total, 11 up, 10 in 2024-03-20T09:07:37.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:37 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 15183/222615 objects degraded (6.820%), 7 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-03-20T09:07:37.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:37 smithi115 ceph-mon[72476]: osdmap e359: 12 total, 11 up, 10 in 2024-03-20T09:07:37.439 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:07:37.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:37.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:07:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:37 smithi122 ceph-mon[73495]: pgmap v887: 47 pgs: 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.5 GiB data, 17 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 135 op/s; 15183/222615 objects degraded (6.820%); 28003/222615 objects misplaced (12.579%); 6.7 MiB/s, 107 objects/s recovering 2024-03-20T09:07:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:37 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-03-20T09:07:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:37 smithi122 ceph-mon[73495]: osdmap e358: 12 total, 11 up, 10 in 2024-03-20T09:07:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:37 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 15183/222615 objects degraded (6.820%), 7 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-03-20T09:07:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:37 smithi122 ceph-mon[73495]: osdmap e359: 12 total, 11 up, 10 in 2024-03-20T09:07:37.543 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:07:37.576 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:37.915 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:07:37.916 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:07:38.114 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:07:38.131 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:07:38.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:38.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:07:38.235 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:07:38.262 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:38.281 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:07:38.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:38.284 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:07:38.385 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:07:38.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:38 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-20T09:07:38.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:38 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-20T09:07:38.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:38 smithi115 ceph-mon[72476]: osdmap e360: 12 total, 11 up, 10 in 2024-03-20T09:07:38.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:38 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2237301774' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:07:38.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:38 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:07:38.409 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:38 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-20T09:07:38.409 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:38 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-20T09:07:38.409 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:38 smithi122 ceph-mon[73495]: osdmap e360: 12 total, 11 up, 10 in 2024-03-20T09:07:38.410 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:38 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2237301774' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:07:38.410 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:38 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:07:38.412 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:38.431 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:07:38.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:38.434 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:07:38.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:38 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-20T09:07:38.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:38 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-20T09:07:38.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:38 smithi026 ceph-mon[68309]: osdmap e360: 12 total, 11 up, 10 in 2024-03-20T09:07:38.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:38 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2237301774' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:07:38.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:38 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:07:38.535 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:07:38.565 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:38.585 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:07:38.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:38.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:07:38.689 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:07:38.694 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-20T09:07:38.720 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:38.959 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:07:39.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:39 smithi115 ceph-mon[72476]: pgmap v890: 57 pgs: 2 peering, 9 unknown, 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 2 active+recovering+undersized+remapped, 6 active+undersized; 4.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 160 op/s; 15196/221784 objects degraded (6.852%); 27944/221784 objects misplaced (12.600%); 6.6 MiB/s, 105 objects/s recovering 2024-03-20T09:07:39.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:39 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:07:39.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:39 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:07:39.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:39 smithi115 ceph-mon[72476]: osdmap e361: 12 total, 11 up, 10 in 2024-03-20T09:07:39.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:39 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:07:39.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:39 smithi122 ceph-mon[73495]: pgmap v890: 57 pgs: 2 peering, 9 unknown, 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 2 active+recovering+undersized+remapped, 6 active+undersized; 4.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 160 op/s; 15196/221784 objects degraded (6.852%); 27944/221784 objects misplaced (12.600%); 6.6 MiB/s, 105 objects/s recovering 2024-03-20T09:07:39.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:39 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:07:39.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:39 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:07:39.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:39 smithi122 ceph-mon[73495]: osdmap e361: 12 total, 11 up, 10 in 2024-03-20T09:07:39.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:39 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:07:39.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:39 smithi026 ceph-mon[68309]: pgmap v890: 57 pgs: 2 peering, 9 unknown, 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 2 active+recovering+undersized+remapped, 6 active+undersized; 4.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 160 op/s; 15196/221784 objects degraded (6.852%); 27944/221784 objects misplaced (12.600%); 6.6 MiB/s, 105 objects/s recovering 2024-03-20T09:07:39.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:39 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:07:39.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:39 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:07:39.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:39 smithi026 ceph-mon[68309]: osdmap e361: 12 total, 11 up, 10 in 2024-03-20T09:07:39.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:39 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:07:40.172 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:07:40.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:40.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:07:40.276 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:07:40.306 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:40.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:40 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:07:40.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:40 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:07:40.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:40 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:07:40.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:40 smithi115 ceph-mon[72476]: osdmap e362: 12 total, 11 up, 10 in 2024-03-20T09:07:40.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:40 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:07:40.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:40 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:07:40.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:40 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:07:40.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:40 smithi122 ceph-mon[73495]: osdmap e362: 12 total, 11 up, 10 in 2024-03-20T09:07:40.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:40 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:07:40.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:40 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:07:40.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:40 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:07:40.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:40 smithi026 ceph-mon[68309]: osdmap e362: 12 total, 11 up, 10 in 2024-03-20T09:07:40.755 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:07:40.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:40.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:07:40.860 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:07:40.887 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:40.910 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:07:40.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:40.913 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:07:41.014 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:07:41.045 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:41.219 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:07:41.247 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:07:41.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:41.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:07:41.272 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:07:41.293 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:07:41.314 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:07:41.337 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:07:41.354 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:07:41.362 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:07:41.389 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:07:41.389 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:41.410 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:07:41.415 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:07:41.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:41.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:07:41.430 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:07:41.451 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:07:41.472 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:07:41.494 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:07:41.494 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:07:41.494 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:07:41.494 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:07:41.494 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:07:41.494 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:07:41.495 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-20T09:07:41.495 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:07:41.495 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:07:41.495 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:07:41.495 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:07:41.495 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:07:41.495 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:07:41.520 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:07:41.547 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:41.566 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:07:41.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:41.568 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:07:41.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:41 smithi115 ceph-mon[72476]: pgmap v893: 57 pgs: 1 remapped+peering, 1 remapped, 2 peering, 7 unknown, 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 2 active+recovering+undersized+remapped, 6 active+undersized; 4.5 GiB data, 17 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 109 op/s; 15096/221274 objects degraded (6.822%); 27868/221274 objects misplaced (12.594%); 754 KiB/s, 11 objects/s recovering 2024-03-20T09:07:41.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:41 smithi115 ceph-mon[72476]: osdmap e363: 12 total, 11 up, 10 in 2024-03-20T09:07:41.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:41 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1789959943' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:07:41.669 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:07:41.695 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:41.715 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:07:41.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:41.717 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:07:41.818 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:07:41.842 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:41 smithi026 ceph-mon[68309]: pgmap v893: 57 pgs: 1 remapped+peering, 1 remapped, 2 peering, 7 unknown, 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 2 active+recovering+undersized+remapped, 6 active+undersized; 4.5 GiB data, 17 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 109 op/s; 15096/221274 objects degraded (6.822%); 27868/221274 objects misplaced (12.594%); 754 KiB/s, 11 objects/s recovering 2024-03-20T09:07:41.842 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:41 smithi026 ceph-mon[68309]: osdmap e363: 12 total, 11 up, 10 in 2024-03-20T09:07:41.842 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:41 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1789959943' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:07:41.849 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:41.908 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:07:41.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:41 smithi122 ceph-mon[73495]: pgmap v893: 57 pgs: 1 remapped+peering, 1 remapped, 2 peering, 7 unknown, 5 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 4 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 2 active+recovering+undersized+remapped, 6 active+undersized; 4.5 GiB data, 17 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 109 op/s; 15096/221274 objects degraded (6.822%); 27868/221274 objects misplaced (12.594%); 754 KiB/s, 11 objects/s recovering 2024-03-20T09:07:41.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:41 smithi122 ceph-mon[73495]: osdmap e363: 12 total, 11 up, 10 in 2024-03-20T09:07:41.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:41 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1789959943' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:07:42.039 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7, 8, 10, 11] out_osds: [9, 3] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11] 2024-03-20T09:07:42.039 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:07:42.039 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 2, 4, 5, 6, 7, 8, 10, 11] 2024-03-20T09:07:42.039 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd out 4 2024-03-20T09:07:42.698 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:07:42.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:42.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:07:42.728 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:07:42.802 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:07:42.828 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:42 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 15096/221274 objects degraded (6.822%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:07:42.828 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:42 smithi115 ceph-mon[72476]: osdmap e364: 12 total, 11 up, 10 in 2024-03-20T09:07:42.830 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:42.842 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:42 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 15096/221274 objects degraded (6.822%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:07:42.843 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:42 smithi026 ceph-mon[68309]: osdmap e364: 12 total, 11 up, 10 in 2024-03-20T09:07:42.850 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:07:42.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:42.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:07:42.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:42 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 15096/221274 objects degraded (6.822%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:07:42.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:42 smithi122 ceph-mon[73495]: osdmap e364: 12 total, 11 up, 10 in 2024-03-20T09:07:42.955 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:07:42.983 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:43.003 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:07:43.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:43.006 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:07:43.107 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:07:43.135 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:43.154 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:07:43.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:43.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:07:43.174 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:07:43.178 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:07:43.178 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:07:43.258 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:07:43.290 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:43.418 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:07:43.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:43.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:07:43.524 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:07:43.551 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:43.570 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:07:43.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:43.572 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:07:43.673 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:07:43.700 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:43.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:43 smithi026 ceph-mon[68309]: pgmap v896: 57 pgs: 4 peering, 1 activating+undersized+remapped, 2 remapped+peering, 1 remapped, 5 active+clean+remapped, 2 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 6 active+undersized; 4.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 2.0 KiB/s rd, 50 KiB/s wr, 106 op/s; 15069/220572 objects degraded (6.832%); 27045/220572 objects misplaced (12.261%); 7.0 MiB/s, 111 objects/s recovering 2024-03-20T09:07:43.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:43 smithi026 ceph-mon[68309]: Health check failed: Reduced data availability: 2 pgs inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-20T09:07:43.721 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:07:43.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:43.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:07:43.782 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:07:43.783 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:07:43.783 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-03-20T09:07:43.783 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-03-20T09:07:43.783 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:07:43.824 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:07:43.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:43 smithi115 ceph-mon[72476]: pgmap v896: 57 pgs: 4 peering, 1 activating+undersized+remapped, 2 remapped+peering, 1 remapped, 5 active+clean+remapped, 2 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 6 active+undersized; 4.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 2.0 KiB/s rd, 50 KiB/s wr, 106 op/s; 15069/220572 objects degraded (6.832%); 27045/220572 objects misplaced (12.261%); 7.0 MiB/s, 111 objects/s recovering 2024-03-20T09:07:43.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:43 smithi115 ceph-mon[72476]: Health check failed: Reduced data availability: 2 pgs inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-20T09:07:43.853 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:43.876 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:07:43.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:43.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:07:43.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:43 smithi122 ceph-mon[73495]: pgmap v896: 57 pgs: 4 peering, 1 activating+undersized+remapped, 2 remapped+peering, 1 remapped, 5 active+clean+remapped, 2 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 6 active+undersized; 4.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 2.0 KiB/s rd, 50 KiB/s wr, 106 op/s; 15069/220572 objects degraded (6.832%); 27045/220572 objects misplaced (12.261%); 7.0 MiB/s, 111 objects/s recovering 2024-03-20T09:07:43.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:43 smithi122 ceph-mon[73495]: Health check failed: Reduced data availability: 2 pgs inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-20T09:07:43.980 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:07:44.007 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:44.027 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:07:44.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:44.030 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:07:44.131 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:07:44.161 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:44.182 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:07:44.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:44.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:07:44.287 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:07:44.313 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:44.332 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:07:44.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:44.335 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:07:44.435 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:07:44.463 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:44.484 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:07:44.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:44.486 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:07:44.587 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:07:44.614 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:44.632 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:07:44.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:44.635 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:07:44.703 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-20T09:07:44.736 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:07:44.762 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:44.781 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:07:44.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:44.783 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:07:44.884 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:07:44.913 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:44.944 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:07:44.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:44.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:07:45.049 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:07:45.078 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:45.098 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:07:45.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:45.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:07:45.179 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:07:45.202 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:07:45.227 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:45.246 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:07:45.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:45.249 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:07:45.350 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:07:45.380 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:45.409 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:07:45.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:45.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:07:45.513 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:07:45.540 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:45.560 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:07:45.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:45.563 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:07:45.664 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:07:45.690 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:45.708 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:07:45.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:45.711 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:07:45.812 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:07:45.838 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:45 smithi026 ceph-mon[68309]: pgmap v898: 57 pgs: 4 peering, 1 activating+undersized+remapped, 2 remapped+peering, 6 active+clean+remapped, 2 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 6 active+undersized; 4.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 2.7 KiB/s rd, 71 KiB/s wr, 149 op/s; 15030/219711 objects degraded (6.841%); 26955/219711 objects misplaced (12.268%); 6.4 MiB/s, 100 objects/s recovering 2024-03-20T09:07:45.840 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:45.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:45 smithi115 ceph-mon[72476]: pgmap v898: 57 pgs: 4 peering, 1 activating+undersized+remapped, 2 remapped+peering, 6 active+clean+remapped, 2 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 6 active+undersized; 4.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 2.7 KiB/s rd, 71 KiB/s wr, 149 op/s; 15030/219711 objects degraded (6.841%); 26955/219711 objects misplaced (12.268%); 6.4 MiB/s, 100 objects/s recovering 2024-03-20T09:07:45.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:45 smithi122 ceph-mon[73495]: pgmap v898: 57 pgs: 4 peering, 1 activating+undersized+remapped, 2 remapped+peering, 6 active+clean+remapped, 2 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 6 active+undersized; 4.5 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 2.7 KiB/s rd, 71 KiB/s wr, 149 op/s; 15030/219711 objects degraded (6.841%); 26955/219711 objects misplaced (12.268%); 6.4 MiB/s, 100 objects/s recovering 2024-03-20T09:07:47.089 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:07:47.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:47.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:07:47.102 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:07:47.196 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:07:47.222 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:47.243 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:07:47.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:47.246 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:07:47.347 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:07:47.382 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:47.480 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:07:47.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:47.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:07:47.584 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:07:47.612 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:47 smithi026 ceph-mon[68309]: pgmap v899: 57 pgs: 4 peering, 2 remapped+peering, 6 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+degraded+remapped, 31 active+clean, 1 active+recovering+undersized+remapped, 6 active+undersized; 4.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 2.2 KiB/s rd, 56 KiB/s wr, 159 op/s; 14908/218514 objects degraded (6.822%); 26487/218514 objects misplaced (12.121%); 6.9 MiB/s, 161 objects/s recovering 2024-03-20T09:07:47.613 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:47 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 14908/218514 objects degraded (6.822%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:07:47.613 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:47 smithi026 ceph-mon[68309]: osdmap e365: 12 total, 11 up, 10 in 2024-03-20T09:07:47.616 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:47.693 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:07:47.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:47.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:07:47.798 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:07:47.826 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:47.900 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:07:47.902 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:47 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mon-a[68286]: 2024-03-20T09:07:47.860+0000 7f683364b700 -1 mon.a@0(leader).osd e365 definitely_dead 0 2024-03-20T09:07:47.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:47.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:07:47.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:47 smithi115 ceph-mon[72476]: pgmap v899: 57 pgs: 4 peering, 2 remapped+peering, 6 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+degraded+remapped, 31 active+clean, 1 active+recovering+undersized+remapped, 6 active+undersized; 4.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 2.2 KiB/s rd, 56 KiB/s wr, 159 op/s; 14908/218514 objects degraded (6.822%); 26487/218514 objects misplaced (12.121%); 6.9 MiB/s, 161 objects/s recovering 2024-03-20T09:07:47.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:47 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 14908/218514 objects degraded (6.822%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:07:47.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:47 smithi115 ceph-mon[72476]: osdmap e365: 12 total, 11 up, 10 in 2024-03-20T09:07:47.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:47 smithi122 ceph-mon[73495]: pgmap v899: 57 pgs: 4 peering, 2 remapped+peering, 6 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+degraded+remapped, 31 active+clean, 1 active+recovering+undersized+remapped, 6 active+undersized; 4.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 2.2 KiB/s rd, 56 KiB/s wr, 159 op/s; 14908/218514 objects degraded (6.822%); 26487/218514 objects misplaced (12.121%); 6.9 MiB/s, 161 objects/s recovering 2024-03-20T09:07:47.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:47 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 14908/218514 objects degraded (6.822%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:07:47.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:47 smithi122 ceph-mon[73495]: osdmap e365: 12 total, 11 up, 10 in 2024-03-20T09:07:48.004 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:07:48.032 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:48.055 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:07:48.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:48.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:07:48.160 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:07:48.188 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:48.211 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:07:48.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:48.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:07:48.316 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:07:48.342 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:48.361 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:07:48.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:48.363 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:07:48.464 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:07:48.490 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:48.509 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:07:48.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:48.512 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:07:48.566 INFO:teuthology.orchestra.run.smithi026.stderr:marked out osd.4. 2024-03-20T09:07:48.613 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:07:48.639 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:48.660 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:07:48.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:48.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:07:48.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:48 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/738390537' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-03-20T09:07:48.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:48 smithi122 ceph-mon[73495]: Client client.admin marked osd.4 out, while it was still marked up 2024-03-20T09:07:48.764 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:07:48.794 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:48.891 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:07:48.891 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:07:48.895 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:48 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/738390537' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-03-20T09:07:48.895 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:48 smithi026 ceph-mon[68309]: Client client.admin marked osd.4 out, while it was still marked up 2024-03-20T09:07:48.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:48.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:07:48.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:48 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/738390537' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-03-20T09:07:48.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:48 smithi115 ceph-mon[72476]: Client client.admin marked osd.4 out, while it was still marked up 2024-03-20T09:07:48.996 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:07:49.025 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:49.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:49 smithi115 ceph-mon[72476]: pgmap v901: 57 pgs: 8 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+degraded+remapped, 35 active+clean, 1 active+recovering+undersized+remapped, 6 active+undersized; 4.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 1.2 KiB/s rd, 30 KiB/s wr, 152 op/s; 14893/217701 objects degraded (6.841%); 26437/217701 objects misplaced (12.144%); 2.5 MiB/s, 85 objects/s recovering 2024-03-20T09:07:49.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:49 smithi115 ceph-mon[72476]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-20T09:07:49.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:49 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/738390537' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-03-20T09:07:49.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:49 smithi115 ceph-mon[72476]: osdmap e366: 12 total, 11 up, 9 in 2024-03-20T09:07:49.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:49 smithi115 ceph-mon[72476]: 3.f scrub starts 2024-03-20T09:07:49.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:49 smithi122 ceph-mon[73495]: pgmap v901: 57 pgs: 8 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+degraded+remapped, 35 active+clean, 1 active+recovering+undersized+remapped, 6 active+undersized; 4.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 1.2 KiB/s rd, 30 KiB/s wr, 152 op/s; 14893/217701 objects degraded (6.841%); 26437/217701 objects misplaced (12.144%); 2.5 MiB/s, 85 objects/s recovering 2024-03-20T09:07:49.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:49 smithi122 ceph-mon[73495]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-20T09:07:49.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:49 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/738390537' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-03-20T09:07:49.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:49 smithi122 ceph-mon[73495]: osdmap e366: 12 total, 11 up, 9 in 2024-03-20T09:07:49.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:49 smithi122 ceph-mon[73495]: 3.f scrub starts 2024-03-20T09:07:49.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:49 smithi026 ceph-mon[68309]: pgmap v901: 57 pgs: 8 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+degraded+remapped, 35 active+clean, 1 active+recovering+undersized+remapped, 6 active+undersized; 4.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 1.2 KiB/s rd, 30 KiB/s wr, 152 op/s; 14893/217701 objects degraded (6.841%); 26437/217701 objects misplaced (12.144%); 2.5 MiB/s, 85 objects/s recovering 2024-03-20T09:07:49.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:49 smithi026 ceph-mon[68309]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-20T09:07:49.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:49 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/738390537' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-03-20T09:07:49.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:49 smithi026 ceph-mon[68309]: osdmap e366: 12 total, 11 up, 9 in 2024-03-20T09:07:49.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:49 smithi026 ceph-mon[68309]: 3.f scrub starts 2024-03-20T09:07:50.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:50 smithi115 ceph-mon[72476]: 1.15 scrub starts 2024-03-20T09:07:50.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:50 smithi115 ceph-mon[72476]: 1.15 scrub ok 2024-03-20T09:07:50.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:50 smithi115 ceph-mon[72476]: 3.f scrub ok 2024-03-20T09:07:50.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:50 smithi115 ceph-mon[72476]: osdmap e367: 12 total, 11 up, 9 in 2024-03-20T09:07:50.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:50 smithi115 ceph-mon[72476]: 3.16 scrub starts 2024-03-20T09:07:50.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:50 smithi115 ceph-mon[72476]: 3.0 scrub starts 2024-03-20T09:07:50.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:50 smithi115 ceph-mon[72476]: 3.0 scrub ok 2024-03-20T09:07:50.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:50 smithi122 ceph-mon[73495]: 1.15 scrub starts 2024-03-20T09:07:50.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:50 smithi122 ceph-mon[73495]: 1.15 scrub ok 2024-03-20T09:07:50.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:50 smithi122 ceph-mon[73495]: 3.f scrub ok 2024-03-20T09:07:50.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:50 smithi122 ceph-mon[73495]: osdmap e367: 12 total, 11 up, 9 in 2024-03-20T09:07:50.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:50 smithi122 ceph-mon[73495]: 3.16 scrub starts 2024-03-20T09:07:50.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:50 smithi122 ceph-mon[73495]: 3.0 scrub starts 2024-03-20T09:07:50.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:50 smithi122 ceph-mon[73495]: 3.0 scrub ok 2024-03-20T09:07:50.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:50 smithi026 ceph-mon[68309]: 1.15 scrub starts 2024-03-20T09:07:50.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:50 smithi026 ceph-mon[68309]: 1.15 scrub ok 2024-03-20T09:07:50.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:50 smithi026 ceph-mon[68309]: 3.f scrub ok 2024-03-20T09:07:50.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:50 smithi026 ceph-mon[68309]: osdmap e367: 12 total, 11 up, 9 in 2024-03-20T09:07:50.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:50 smithi026 ceph-mon[68309]: 3.16 scrub starts 2024-03-20T09:07:50.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:50 smithi026 ceph-mon[68309]: 3.0 scrub starts 2024-03-20T09:07:50.975 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:50 smithi026 ceph-mon[68309]: 3.0 scrub ok 2024-03-20T09:07:51.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:51 smithi115 ceph-mon[72476]: pgmap v904: 57 pgs: 8 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+degraded+remapped, 35 active+clean, 1 active+recovering+undersized+remapped, 6 active+undersized; 4.4 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 115 op/s; 14868/217626 objects degraded (6.832%); 26437/217626 objects misplaced (12.148%); 2.5 MiB/s, 91 objects/s recovering 2024-03-20T09:07:51.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:51 smithi115 ceph-mon[72476]: 1.1b scrub starts 2024-03-20T09:07:51.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:51 smithi115 ceph-mon[72476]: 1.1b scrub ok 2024-03-20T09:07:51.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:51 smithi115 ceph-mon[72476]: 3.16 scrub ok 2024-03-20T09:07:51.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:51 smithi115 ceph-mon[72476]: osdmap e368: 12 total, 11 up, 9 in 2024-03-20T09:07:51.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:51 smithi115 ceph-mon[72476]: 3.a starting backfill to osd.8 from (0'0,310'7203] MIN to 365'8001 2024-03-20T09:07:51.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:51 smithi122 ceph-mon[73495]: pgmap v904: 57 pgs: 8 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+degraded+remapped, 35 active+clean, 1 active+recovering+undersized+remapped, 6 active+undersized; 4.4 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 115 op/s; 14868/217626 objects degraded (6.832%); 26437/217626 objects misplaced (12.148%); 2.5 MiB/s, 91 objects/s recovering 2024-03-20T09:07:51.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:51 smithi122 ceph-mon[73495]: 1.1b scrub starts 2024-03-20T09:07:51.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:51 smithi122 ceph-mon[73495]: 1.1b scrub ok 2024-03-20T09:07:51.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:51 smithi122 ceph-mon[73495]: 3.16 scrub ok 2024-03-20T09:07:51.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:51 smithi122 ceph-mon[73495]: osdmap e368: 12 total, 11 up, 9 in 2024-03-20T09:07:51.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:51 smithi122 ceph-mon[73495]: 3.a starting backfill to osd.8 from (0'0,310'7203] MIN to 365'8001 2024-03-20T09:07:51.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:51 smithi026 ceph-mon[68309]: pgmap v904: 57 pgs: 8 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+degraded+remapped, 35 active+clean, 1 active+recovering+undersized+remapped, 6 active+undersized; 4.4 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 115 op/s; 14868/217626 objects degraded (6.832%); 26437/217626 objects misplaced (12.148%); 2.5 MiB/s, 91 objects/s recovering 2024-03-20T09:07:51.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:51 smithi026 ceph-mon[68309]: 1.1b scrub starts 2024-03-20T09:07:51.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:51 smithi026 ceph-mon[68309]: 1.1b scrub ok 2024-03-20T09:07:51.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:51 smithi026 ceph-mon[68309]: 3.16 scrub ok 2024-03-20T09:07:51.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:51 smithi026 ceph-mon[68309]: osdmap e368: 12 total, 11 up, 9 in 2024-03-20T09:07:51.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:51 smithi026 ceph-mon[68309]: 3.a starting backfill to osd.8 from (0'0,310'7203] MIN to 365'8001 2024-03-20T09:07:52.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:52 smithi115 ceph-mon[72476]: osdmap e369: 12 total, 11 up, 9 in 2024-03-20T09:07:52.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:52 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 14868/217626 objects degraded (6.832%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:07:52.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:52 smithi122 ceph-mon[73495]: osdmap e369: 12 total, 11 up, 9 in 2024-03-20T09:07:52.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:52 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 14868/217626 objects degraded (6.832%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:07:52.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:52 smithi026 ceph-mon[68309]: osdmap e369: 12 total, 11 up, 9 in 2024-03-20T09:07:52.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:52 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 14868/217626 objects degraded (6.832%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:07:53.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:53 smithi115 ceph-mon[72476]: pgmap v907: 57 pgs: 1 peering, 8 remapped+peering, 7 active+clean+remapped, 1 active+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 1 active+recovering+undersized+remapped, 1 active+clean+remapped+scrubbing, 5 active+undersized; 4.4 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 63 op/s; 12524/216924 objects degraded (5.773%); 26027/216924 objects misplaced (11.998%); 3.7 MiB/s, 59 objects/s recovering 2024-03-20T09:07:53.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:53 smithi115 ceph-mon[72476]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-20T09:07:53.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:53 smithi115 ceph-mon[72476]: osdmap e370: 12 total, 11 up, 9 in 2024-03-20T09:07:53.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:53 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:07:53.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:53 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:07:53.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:53 smithi122 ceph-mon[73495]: pgmap v907: 57 pgs: 1 peering, 8 remapped+peering, 7 active+clean+remapped, 1 active+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 1 active+recovering+undersized+remapped, 1 active+clean+remapped+scrubbing, 5 active+undersized; 4.4 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 63 op/s; 12524/216924 objects degraded (5.773%); 26027/216924 objects misplaced (11.998%); 3.7 MiB/s, 59 objects/s recovering 2024-03-20T09:07:53.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:53 smithi122 ceph-mon[73495]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-20T09:07:53.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:53 smithi122 ceph-mon[73495]: osdmap e370: 12 total, 11 up, 9 in 2024-03-20T09:07:53.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:53 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:07:53.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:53 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:07:53.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:53 smithi026 ceph-mon[68309]: pgmap v907: 57 pgs: 1 peering, 8 remapped+peering, 7 active+clean+remapped, 1 active+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 1 active+recovering+undersized+remapped, 1 active+clean+remapped+scrubbing, 5 active+undersized; 4.4 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 63 op/s; 12524/216924 objects degraded (5.773%); 26027/216924 objects misplaced (11.998%); 3.7 MiB/s, 59 objects/s recovering 2024-03-20T09:07:53.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:53 smithi026 ceph-mon[68309]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-20T09:07:53.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:53 smithi026 ceph-mon[68309]: osdmap e370: 12 total, 11 up, 9 in 2024-03-20T09:07:53.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:53 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:07:53.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:53 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:07:54.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:54 smithi115 ceph-mon[72476]: osdmap e371: 12 total, 11 up, 9 in 2024-03-20T09:07:54.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:54 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3603342856' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:07:54.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:54 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:07:54.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:54 smithi122 ceph-mon[73495]: osdmap e371: 12 total, 11 up, 9 in 2024-03-20T09:07:54.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:54 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3603342856' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:07:54.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:54 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:07:54.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:54 smithi026 ceph-mon[68309]: osdmap e371: 12 total, 11 up, 9 in 2024-03-20T09:07:54.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:54 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3603342856' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:07:54.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:54 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:07:55.896 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:55 smithi026 ceph-mon[68309]: pgmap v910: 57 pgs: 1 peering, 8 remapped+peering, 7 active+clean+remapped, 1 active+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 25 active+clean, 2 active+recovering+undersized+remapped, 1 active+clean+remapped+scrubbing, 5 active+undersized; 4.4 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 62 op/s; 12519/216867 objects degraded (5.773%); 28050/216867 objects misplaced (12.934%); 7.6 MiB/s, 121 objects/s recovering 2024-03-20T09:07:55.897 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:55 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:07:55.897 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:55 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:07:55.897 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:55 smithi026 ceph-mon[68309]: osdmap e372: 12 total, 11 up, 9 in 2024-03-20T09:07:55.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:55 smithi115 ceph-mon[72476]: pgmap v910: 57 pgs: 1 peering, 8 remapped+peering, 7 active+clean+remapped, 1 active+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 25 active+clean, 2 active+recovering+undersized+remapped, 1 active+clean+remapped+scrubbing, 5 active+undersized; 4.4 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 62 op/s; 12519/216867 objects degraded (5.773%); 28050/216867 objects misplaced (12.934%); 7.6 MiB/s, 121 objects/s recovering 2024-03-20T09:07:55.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:55 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:07:55.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:55 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:07:55.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:55 smithi115 ceph-mon[72476]: osdmap e372: 12 total, 11 up, 9 in 2024-03-20T09:07:55.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:55 smithi122 ceph-mon[73495]: pgmap v910: 57 pgs: 1 peering, 8 remapped+peering, 7 active+clean+remapped, 1 active+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 25 active+clean, 2 active+recovering+undersized+remapped, 1 active+clean+remapped+scrubbing, 5 active+undersized; 4.4 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 62 op/s; 12519/216867 objects degraded (5.773%); 28050/216867 objects misplaced (12.934%); 7.6 MiB/s, 121 objects/s recovering 2024-03-20T09:07:55.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:55 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:07:55.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:55 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:07:55.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:55 smithi122 ceph-mon[73495]: osdmap e372: 12 total, 11 up, 9 in 2024-03-20T09:07:56.314 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:07:57.110 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:07:57.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:57.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:07:57.227 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:07:57.260 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:57.288 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:07:57.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:57.291 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:07:57.393 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:07:57.456 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:57.621 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:57 smithi026 ceph-mon[68309]: pgmap v912: 57 pgs: 1 active+recovering+remapped, 1 activating+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 peering, 10 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.4 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 19452/216663 objects degraded (8.978%); 42796/216663 objects misplaced (19.752%); 22 MiB/s, 357 objects/s recovering 2024-03-20T09:07:57.622 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:57 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 19452/216663 objects degraded (8.978%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-20T09:07:57.622 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:57 smithi026 ceph-mon[68309]: osdmap e373: 12 total, 11 up, 9 in 2024-03-20T09:07:57.622 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:57 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:07:57.622 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:57 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:07:57.674 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:07:57.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:57.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:07:57.788 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:07:57.819 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:57.842 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:07:57.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:57.845 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:07:57.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:57 smithi115 ceph-mon[72476]: pgmap v912: 57 pgs: 1 active+recovering+remapped, 1 activating+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 peering, 10 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.4 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 19452/216663 objects degraded (8.978%); 42796/216663 objects misplaced (19.752%); 22 MiB/s, 357 objects/s recovering 2024-03-20T09:07:57.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:57 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 19452/216663 objects degraded (8.978%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-20T09:07:57.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:57 smithi115 ceph-mon[72476]: osdmap e373: 12 total, 11 up, 9 in 2024-03-20T09:07:57.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:57 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:07:57.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:57 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:07:57.946 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:07:57.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:57 smithi122 ceph-mon[73495]: pgmap v912: 57 pgs: 1 active+recovering+remapped, 1 activating+undersized+remapped, 1 activating+undersized+degraded+remapped, 1 peering, 10 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 26 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.4 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 19452/216663 objects degraded (8.978%); 42796/216663 objects misplaced (19.752%); 22 MiB/s, 357 objects/s recovering 2024-03-20T09:07:57.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:57 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 19452/216663 objects degraded (8.978%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-20T09:07:57.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:57 smithi122 ceph-mon[73495]: osdmap e373: 12 total, 11 up, 9 in 2024-03-20T09:07:57.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:57 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:07:57.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:57 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:07:58.007 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:58.092 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:07:58.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:58.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:07:58.198 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:07:58.230 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:58.255 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:07:58.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:58.258 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:07:58.359 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:07:58.403 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:58.445 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:07:58.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:58.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:07:58.549 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:07:58.594 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:58.636 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:58 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3754070989' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:07:58.636 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:58 smithi122 ceph-mon[73495]: osdmap e374: 12 total, 11 up, 9 in 2024-03-20T09:07:58.646 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:07:58.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:58.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:07:58.653 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:58 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3754070989' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:07:58.653 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:58 smithi026 ceph-mon[68309]: osdmap e374: 12 total, 11 up, 9 in 2024-03-20T09:07:58.752 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:07:58.782 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:58.804 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:07:58.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:58.807 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:07:58.850 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:07:58.850 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-03-20T09:07:58.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-03-20T09:07:58.851 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:07:58.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.851 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24550.0:85672 2.9 2:984dd526:::benchmark_data_smithi122_107654_object85671:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-03-20T09:07:58.851 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T08:59:07.803371+0000", 2024-03-20T09:07:58.851 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 531.04250346499998, 2024-03-20T09:07:58.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.212404648, 2024-03-20T09:07:58.851 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:07:58.851 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:07:58.851 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:07:58.851 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24550", 2024-03-20T09:07:58.851 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:07:58.851 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 85672 2024-03-20T09:07:58.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.852 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:07:58.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:07:58.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:07.803371+0000", 2024-03-20T09:07:58.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:07:58.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:07.803371+0000", 2024-03-20T09:07:58.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:07:58.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:07.803371+0000", 2024-03-20T09:07:58.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.44e-07 2024-03-20T09:07:58.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.853 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:07:58.853 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:07.803423+0000", 2024-03-20T09:07:58.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1718999999999997e-05 2024-03-20T09:07:58.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.853 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:07:58.853 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:07.803432+0000", 2024-03-20T09:07:58.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.234e-06 2024-03-20T09:07:58.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.853 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:07:58.854 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:07.803439+0000", 2024-03-20T09:07:58.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4060000000000002e-06 2024-03-20T09:07:58.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.854 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.854 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:07.806188+0000", 2024-03-20T09:07:58.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027488159999999998 2024-03-20T09:07:58.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.854 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:07:58.854 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:07.806324+0000", 2024-03-20T09:07:58.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013685300000000001 2024-03-20T09:07:58.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,7", 2024-03-20T09:07:58.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:07.806474+0000", 2024-03-20T09:07:58.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014907399999999999 2024-03-20T09:07:58.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:07:58.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:07.812274+0000", 2024-03-20T09:07:58.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0058007629999999996 2024-03-20T09:07:58.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:07.813091+0000", 2024-03-20T09:07:58.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00081638499999999998 2024-03-20T09:07:58.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.856 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:08.015668+0000", 2024-03-20T09:07:58.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.20257760299999999 2024-03-20T09:07:58.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:07:58.856 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:08.015730+0000", 2024-03-20T09:07:58.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1829999999999996e-05 2024-03-20T09:07:58.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:07:58.856 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T08:59:08.015776+0000", 2024-03-20T09:07:58.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5420999999999998e-05 2024-03-20T09:07:58.857 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.857 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:07:58.857 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.857 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:58467 3.12 3.34c92ed2 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e233)", 2024-03-20T09:07:58.857 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:31.876504+0000", 2024-03-20T09:07:58.857 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 266.96936998899997, 2024-03-20T09:07:58.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0009192840000001, 2024-03-20T09:07:58.857 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:07:58.857 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:07:58.857 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:07:58.857 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:07:58.858 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:07:58.858 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58467 2024-03-20T09:07:58.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.858 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:07:58.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.858 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:07:58.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.876504+0000", 2024-03-20T09:07:58.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.858 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:07:58.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.876504+0000", 2024-03-20T09:07:58.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:07:58.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.876506+0000", 2024-03-20T09:07:58.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5e-06 2024-03-20T09:07:58.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:07:58.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.876580+0000", 2024-03-20T09:07:58.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3909e-05 2024-03-20T09:07:58.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:07:58.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.876597+0000", 2024-03-20T09:07:58.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7629e-05 2024-03-20T09:07:58.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.860 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:07:58.860 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.876620+0000", 2024-03-20T09:07:58.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2354e-05 2024-03-20T09:07:58.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.860 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.860 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.881960+0000", 2024-03-20T09:07:58.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0053402290000000002 2024-03-20T09:07:58.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.860 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:07:58.861 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.881968+0000", 2024-03-20T09:07:58.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6369999999999999e-06 2024-03-20T09:07:58.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.861 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.861 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877405+0000", 2024-03-20T09:07:58.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.99543781899999995 2024-03-20T09:07:58.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.861 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:07:58.861 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877424+0000", 2024-03-20T09:07:58.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8207e-05 2024-03-20T09:07:58.861 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.861 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:07:58.861 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.862 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:58485 3.19 3.ce6b2519 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-20T09:07:58.862 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:31.886747+0000", 2024-03-20T09:07:58.862 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 266.959127217, 2024-03-20T09:07:58.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.99067739200000005, 2024-03-20T09:07:58.862 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:07:58.862 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:07:58.862 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:07:58.862 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:07:58.862 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:07:58.862 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58485 2024-03-20T09:07:58.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.862 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:07:58.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:07:58.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.886747+0000", 2024-03-20T09:07:58.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:07:58.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.886747+0000", 2024-03-20T09:07:58.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:07:58.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.886749+0000", 2024-03-20T09:07:58.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.575e-06 2024-03-20T09:07:58.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:07:58.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.886804+0000", 2024-03-20T09:07:58.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.537e-05 2024-03-20T09:07:58.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:07:58.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.886818+0000", 2024-03-20T09:07:58.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3940999999999999e-05 2024-03-20T09:07:58.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:07:58.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.886827+0000", 2024-03-20T09:07:58.865 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5229999999999995e-06 2024-03-20T09:07:58.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.865 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.865 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.886871+0000", 2024-03-20T09:07:58.865 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4369000000000002e-05 2024-03-20T09:07:58.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.865 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:07:58.865 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.886876+0000", 2024-03-20T09:07:58.865 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2809999999999999e-06 2024-03-20T09:07:58.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.865 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.866 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877412+0000", 2024-03-20T09:07:58.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.99053602699999999 2024-03-20T09:07:58.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.866 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:07:58.866 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877425+0000", 2024-03-20T09:07:58.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2306e-05 2024-03-20T09:07:58.866 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.866 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:07:58.866 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.866 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:58497 3.12 3.8e7b9d92 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-20T09:07:58.866 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:31.892587+0000", 2024-03-20T09:07:58.867 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 266.95328780099999, 2024-03-20T09:07:58.867 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.98492999299999995, 2024-03-20T09:07:58.867 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:07:58.867 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:07:58.867 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:07:58.867 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:07:58.867 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:07:58.867 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58497 2024-03-20T09:07:58.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.867 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:07:58.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.867 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:07:58.867 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.892587+0000", 2024-03-20T09:07:58.867 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:07:58.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.892587+0000", 2024-03-20T09:07:58.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:07:58.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.892588+0000", 2024-03-20T09:07:58.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.265e-06 2024-03-20T09:07:58.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:07:58.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.892665+0000", 2024-03-20T09:07:58.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6978999999999996e-05 2024-03-20T09:07:58.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:07:58.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.892676+0000", 2024-03-20T09:07:58.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0814e-05 2024-03-20T09:07:58.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:07:58.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.892682+0000", 2024-03-20T09:07:58.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3419999999999996e-06 2024-03-20T09:07:58.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.892731+0000", 2024-03-20T09:07:58.870 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9107000000000003e-05 2024-03-20T09:07:58.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.870 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:07:58.870 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.892736+0000", 2024-03-20T09:07:58.870 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0370000000000002e-06 2024-03-20T09:07:58.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.870 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.870 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877505+0000", 2024-03-20T09:07:58.870 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.98476891600000005 2024-03-20T09:07:58.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.871 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:07:58.871 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877517+0000", 2024-03-20T09:07:58.871 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1533e-05 2024-03-20T09:07:58.871 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.871 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:07:58.871 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.871 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:58477 3.18 3:1b48b892:::benchmark_data_smithi122_119671_object58476:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e234)", 2024-03-20T09:07:58.871 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:32.877148+0000", 2024-03-20T09:07:58.871 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 265.96872641700003, 2024-03-20T09:07:58.871 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2590889679999999, 2024-03-20T09:07:58.871 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:07:58.871 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:07:58.872 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:07:58.872 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:07:58.872 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:07:58.872 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58477 2024-03-20T09:07:58.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.872 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:07:58.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.872 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:07:58.872 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877148+0000", 2024-03-20T09:07:58.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.872 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:07:58.873 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877148+0000", 2024-03-20T09:07:58.873 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:07:58.873 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877150+0000", 2024-03-20T09:07:58.873 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6759999999999999e-06 2024-03-20T09:07:58.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:07:58.873 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877312+0000", 2024-03-20T09:07:58.873 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000162342 2024-03-20T09:07:58.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.874 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:07:58.874 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877325+0000", 2024-03-20T09:07:58.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3506000000000001e-05 2024-03-20T09:07:58.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.874 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:07:58.874 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877336+0000", 2024-03-20T09:07:58.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0129000000000001e-05 2024-03-20T09:07:58.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.874 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.874 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878426+0000", 2024-03-20T09:07:58.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010908980000000001 2024-03-20T09:07:58.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:07:58.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878436+0000", 2024-03-20T09:07:58.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8700000000000004e-06 2024-03-20T09:07:58.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.119544+0000", 2024-03-20T09:07:58.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2411071929999999 2024-03-20T09:07:58.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:07:58.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.119724+0000", 2024-03-20T09:07:58.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000180485 2024-03-20T09:07:58.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2,7,8", 2024-03-20T09:07:58.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.119915+0000", 2024-03-20T09:07:58.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019086900000000001 2024-03-20T09:07:58.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.128602+0000", 2024-03-20T09:07:58.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086875939999999999 2024-03-20T09:07:58.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.134430+0000", 2024-03-20T09:07:58.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0058273589999999998 2024-03-20T09:07:58.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.877 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:07:58.877 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.134689+0000", 2024-03-20T09:07:58.877 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025899000000000001 2024-03-20T09:07:58.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.877 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.877 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.134789+0000", 2024-03-20T09:07:58.877 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9790999999999994e-05 2024-03-20T09:07:58.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.877 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:07:58.877 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.134857+0000", 2024-03-20T09:07:58.878 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8856000000000006e-05 2024-03-20T09:07:58.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.878 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:07:58.878 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.136237+0000", 2024-03-20T09:07:58.878 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00137941 2024-03-20T09:07:58.878 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.878 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:07:58.878 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.878 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:58482 3.18 3:1aeed275:::benchmark_data_smithi122_119671_object58481:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e234)", 2024-03-20T09:07:58.878 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:32.877352+0000", 2024-03-20T09:07:58.878 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 265.96852255900001, 2024-03-20T09:07:58.879 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2600809530000001, 2024-03-20T09:07:58.879 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:07:58.879 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:07:58.879 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:07:58.879 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:07:58.879 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:07:58.879 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58482 2024-03-20T09:07:58.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.879 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:07:58.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.879 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:07:58.879 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877352+0000", 2024-03-20T09:07:58.879 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:07:58.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877352+0000", 2024-03-20T09:07:58.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:07:58.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877353+0000", 2024-03-20T09:07:58.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4499999999999996e-07 2024-03-20T09:07:58.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:07:58.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877393+0000", 2024-03-20T09:07:58.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0172000000000001e-05 2024-03-20T09:07:58.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:07:58.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877403+0000", 2024-03-20T09:07:58.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0152999999999999e-05 2024-03-20T09:07:58.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:07:58.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877410+0000", 2024-03-20T09:07:58.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1729999999999998e-06 2024-03-20T09:07:58.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878483+0000", 2024-03-20T09:07:58.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001073035 2024-03-20T09:07:58.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.882 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:07:58.882 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878491+0000", 2024-03-20T09:07:58.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6149999999999999e-06 2024-03-20T09:07:58.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.882 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.882 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.120924+0000", 2024-03-20T09:07:58.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.242433439 2024-03-20T09:07:58.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.882 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:07:58.883 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.121069+0000", 2024-03-20T09:07:58.883 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014456000000000001 2024-03-20T09:07:58.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.883 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2,7,8", 2024-03-20T09:07:58.883 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.121248+0000", 2024-03-20T09:07:58.883 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00017885500000000001 2024-03-20T09:07:58.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.883 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.883 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.134254+0000", 2024-03-20T09:07:58.883 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013006856000000001 2024-03-20T09:07:58.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.884 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.884 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.134471+0000", 2024-03-20T09:07:58.884 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000216076 2024-03-20T09:07:58.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.884 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:07:58.884 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.134697+0000", 2024-03-20T09:07:58.884 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022649599999999999 2024-03-20T09:07:58.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.884 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.884 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.136387+0000", 2024-03-20T09:07:58.884 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001689739 2024-03-20T09:07:58.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.885 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:07:58.885 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.136449+0000", 2024-03-20T09:07:58.885 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2148000000000002e-05 2024-03-20T09:07:58.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.885 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:07:58.885 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.137433+0000", 2024-03-20T09:07:58.885 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098379099999999992 2024-03-20T09:07:58.885 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.885 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:07:58.885 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.886 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:58496 3.18 3:1c5e5fc5:::benchmark_data_smithi122_119671_object58495:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e234)", 2024-03-20T09:07:58.886 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:32.877424+0000", 2024-03-20T09:07:58.886 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 265.96845028299998, 2024-03-20T09:07:58.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2609285780000001, 2024-03-20T09:07:58.886 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:07:58.886 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:07:58.886 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:07:58.886 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:07:58.886 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:07:58.886 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58496 2024-03-20T09:07:58.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.886 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:07:58.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:07:58.887 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877424+0000", 2024-03-20T09:07:58.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.887 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:07:58.887 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877424+0000", 2024-03-20T09:07:58.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.887 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:07:58.887 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877425+0000", 2024-03-20T09:07:58.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0100000000000003e-07 2024-03-20T09:07:58.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.887 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:07:58.888 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877513+0000", 2024-03-20T09:07:58.888 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7820999999999998e-05 2024-03-20T09:07:58.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.888 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:07:58.888 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877521+0000", 2024-03-20T09:07:58.888 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2840000000000006e-06 2024-03-20T09:07:58.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.888 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:07:58.888 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877528+0000", 2024-03-20T09:07:58.888 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.781e-06 2024-03-20T09:07:58.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.889 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.889 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878574+0000", 2024-03-20T09:07:58.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010459829999999999 2024-03-20T09:07:58.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.889 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:07:58.889 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878579+0000", 2024-03-20T09:07:58.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5960000000000003e-06 2024-03-20T09:07:58.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.889 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.889 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.122222+0000", 2024-03-20T09:07:58.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2436422620000001 2024-03-20T09:07:58.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.890 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:07:58.890 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.122376+0000", 2024-03-20T09:07:58.890 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000154528 2024-03-20T09:07:58.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.890 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2,7,8", 2024-03-20T09:07:58.890 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.123115+0000", 2024-03-20T09:07:58.890 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073839299999999995 2024-03-20T09:07:58.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.890 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.890 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.134509+0000", 2024-03-20T09:07:58.890 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011394213 2024-03-20T09:07:58.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.891 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.134586+0000", 2024-03-20T09:07:58.891 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7649000000000001e-05 2024-03-20T09:07:58.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.891 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:07:58.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.134704+0000", 2024-03-20T09:07:58.891 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000117108 2024-03-20T09:07:58.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.891 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.137481+0000", 2024-03-20T09:07:58.892 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00277787 2024-03-20T09:07:58.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.892 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:07:58.892 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.137533+0000", 2024-03-20T09:07:58.892 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1783999999999998e-05 2024-03-20T09:07:58.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.892 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:07:58.892 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.138353+0000", 2024-03-20T09:07:58.892 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00081940500000000005 2024-03-20T09:07:58.892 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.892 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:07:58.892 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.893 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:58502 3.14 3:2edc29ec:::benchmark_data_smithi122_119671_object58501:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e234)", 2024-03-20T09:07:58.893 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:32.877541+0000", 2024-03-20T09:07:58.893 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 265.96833360099998, 2024-03-20T09:07:58.893 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2043195660000001, 2024-03-20T09:07:58.893 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:07:58.893 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:07:58.893 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:07:58.893 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:07:58.893 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:07:58.893 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58502 2024-03-20T09:07:58.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.893 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:07:58.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.894 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:07:58.894 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877541+0000", 2024-03-20T09:07:58.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.894 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:07:58.894 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877541+0000", 2024-03-20T09:07:58.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.894 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:07:58.894 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877541+0000", 2024-03-20T09:07:58.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8800000000000002e-07 2024-03-20T09:07:58.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:07:58.895 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877581+0000", 2024-03-20T09:07:58.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9286000000000003e-05 2024-03-20T09:07:58.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:07:58.895 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877590+0000", 2024-03-20T09:07:58.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7229999999999999e-06 2024-03-20T09:07:58.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:07:58.895 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878533+0000", 2024-03-20T09:07:58.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094220800000000004 2024-03-20T09:07:58.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.896 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.896 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878627+0000", 2024-03-20T09:07:58.896 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4044999999999998e-05 2024-03-20T09:07:58.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.896 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:07:58.896 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878633+0000", 2024-03-20T09:07:58.896 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.251e-06 2024-03-20T09:07:58.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.896 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.896 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.062341+0000", 2024-03-20T09:07:58.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.183707877 2024-03-20T09:07:58.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.897 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:07:58.897 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.062443+0000", 2024-03-20T09:07:58.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101999 2024-03-20T09:07:58.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.897 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4,8,11", 2024-03-20T09:07:58.897 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.062563+0000", 2024-03-20T09:07:58.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011998300000000001 2024-03-20T09:07:58.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.897 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.897 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.078982+0000", 2024-03-20T09:07:58.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016419407 2024-03-20T09:07:58.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.079028+0000", 2024-03-20T09:07:58.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5501999999999999e-05 2024-03-20T09:07:58.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.079073+0000", 2024-03-20T09:07:58.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5225000000000002e-05 2024-03-20T09:07:58.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:07:58.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.080427+0000", 2024-03-20T09:07:58.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001354374 2024-03-20T09:07:58.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:07:58.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.080479+0000", 2024-03-20T09:07:58.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1847e-05 2024-03-20T09:07:58.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:07:58.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.081860+0000", 2024-03-20T09:07:58.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013811509999999999 2024-03-20T09:07:58.899 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.899 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:07:58.900 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.900 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:58506 3.16 3:6b2181e1:::benchmark_data_smithi122_119671_object58505:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e234)", 2024-03-20T09:07:58.900 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:32.878602+0000", 2024-03-20T09:07:58.900 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 265.967272267, 2024-03-20T09:07:58.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.18545392, 2024-03-20T09:07:58.900 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:07:58.900 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:07:58.900 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:07:58.900 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:07:58.900 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:07:58.900 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58506 2024-03-20T09:07:58.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.900 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:07:58.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.901 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:07:58.901 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878602+0000", 2024-03-20T09:07:58.901 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.901 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:07:58.901 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878602+0000", 2024-03-20T09:07:58.901 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.902 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:07:58.902 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878603+0000", 2024-03-20T09:07:58.902 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1699999999999997e-07 2024-03-20T09:07:58.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.902 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:07:58.902 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878659+0000", 2024-03-20T09:07:58.902 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6477999999999997e-05 2024-03-20T09:07:58.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.902 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:07:58.902 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878671+0000", 2024-03-20T09:07:58.902 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1287999999999999e-05 2024-03-20T09:07:58.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.903 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:07:58.903 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878676+0000", 2024-03-20T09:07:58.903 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.801e-06 2024-03-20T09:07:58.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.903 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.903 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878737+0000", 2024-03-20T09:07:58.903 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0763000000000001e-05 2024-03-20T09:07:58.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.903 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:07:58.903 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878747+0000", 2024-03-20T09:07:58.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3700000000000001e-06 2024-03-20T09:07:58.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.042973+0000", 2024-03-20T09:07:58.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.164226663 2024-03-20T09:07:58.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:07:58.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.043149+0000", 2024-03-20T09:07:58.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00017547400000000001 2024-03-20T09:07:58.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5,7", 2024-03-20T09:07:58.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.043262+0000", 2024-03-20T09:07:58.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011314 2024-03-20T09:07:58.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.905 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:07:58.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.046319+0000", 2024-03-20T09:07:58.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0030568449999999999 2024-03-20T09:07:58.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.905 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.060550+0000", 2024-03-20T09:07:58.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014231354999999999 2024-03-20T09:07:58.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.905 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.906 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.062075+0000", 2024-03-20T09:07:58.906 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015245840000000001 2024-03-20T09:07:58.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:07:58.906 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.062136+0000", 2024-03-20T09:07:58.906 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.177e-05 2024-03-20T09:07:58.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:07:58.906 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.064056+0000", 2024-03-20T09:07:58.906 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001919572 2024-03-20T09:07:58.906 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.906 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:07:58.907 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.907 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:58511 3.14 3:2c7b4300:::benchmark_data_smithi122_119671_object58510:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e234)", 2024-03-20T09:07:58.907 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:32.878689+0000", 2024-03-20T09:07:58.907 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 265.96718539800003, 2024-03-20T09:07:58.907 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.230044227, 2024-03-20T09:07:58.907 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:07:58.907 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:07:58.907 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:07:58.907 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:07:58.907 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:07:58.907 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58511 2024-03-20T09:07:58.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.908 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:07:58.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.908 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:07:58.908 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878689+0000", 2024-03-20T09:07:58.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.908 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:07:58.908 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878689+0000", 2024-03-20T09:07:58.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.908 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:07:58.908 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878689+0000", 2024-03-20T09:07:58.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2800000000000002e-07 2024-03-20T09:07:58.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.909 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:07:58.909 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878744+0000", 2024-03-20T09:07:58.909 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4780000000000001e-05 2024-03-20T09:07:58.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.909 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:07:58.909 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878751+0000", 2024-03-20T09:07:58.909 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.516e-06 2024-03-20T09:07:58.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.909 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:07:58.909 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878755+0000", 2024-03-20T09:07:58.910 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2579999999999997e-06 2024-03-20T09:07:58.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.910 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878822+0000", 2024-03-20T09:07:58.910 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7376e-05 2024-03-20T09:07:58.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:07:58.910 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878831+0000", 2024-03-20T09:07:58.910 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8680000000000005e-06 2024-03-20T09:07:58.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.079110+0000", 2024-03-20T09:07:58.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.20027843 2024-03-20T09:07:58.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:07:58.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.079268+0000", 2024-03-20T09:07:58.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015832199999999999 2024-03-20T09:07:58.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4,8,11", 2024-03-20T09:07:58.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.079459+0000", 2024-03-20T09:07:58.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019131000000000001 2024-03-20T09:07:58.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.912 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.083888+0000", 2024-03-20T09:07:58.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0044284270000000004 2024-03-20T09:07:58.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.912 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.084205+0000", 2024-03-20T09:07:58.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00031737099999999999 2024-03-20T09:07:58.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.912 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:07:58.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.086321+0000", 2024-03-20T09:07:58.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002115914 2024-03-20T09:07:58.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.913 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.913 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.107469+0000", 2024-03-20T09:07:58.913 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021148275000000001 2024-03-20T09:07:58.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.913 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:07:58.913 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.107532+0000", 2024-03-20T09:07:58.913 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2669999999999995e-05 2024-03-20T09:07:58.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.913 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:07:58.913 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.108733+0000", 2024-03-20T09:07:58.913 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012012819999999999 2024-03-20T09:07:58.913 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.914 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:07:58.914 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.914 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:58515 3.18 3:1891b56f:::benchmark_data_smithi122_119671_object58514:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e234)", 2024-03-20T09:07:58.914 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:32.878766+0000", 2024-03-20T09:07:58.914 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 265.96710880400002, 2024-03-20T09:07:58.914 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.262981237, 2024-03-20T09:07:58.914 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:07:58.914 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:07:58.914 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:07:58.914 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:07:58.914 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:07:58.915 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58515 2024-03-20T09:07:58.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.915 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:07:58.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.915 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:07:58.915 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878766+0000", 2024-03-20T09:07:58.915 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.915 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:07:58.915 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878766+0000", 2024-03-20T09:07:58.915 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.915 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:07:58.916 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878766+0000", 2024-03-20T09:07:58.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2499999999999995e-07 2024-03-20T09:07:58.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.916 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:07:58.916 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878836+0000", 2024-03-20T09:07:58.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0188e-05 2024-03-20T09:07:58.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.916 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:07:58.916 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878847+0000", 2024-03-20T09:07:58.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1189000000000001e-05 2024-03-20T09:07:58.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:07:58.917 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878855+0000", 2024-03-20T09:07:58.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4719999999999998e-06 2024-03-20T09:07:58.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.917 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878912+0000", 2024-03-20T09:07:58.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7286e-05 2024-03-20T09:07:58.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:07:58.917 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878922+0000", 2024-03-20T09:07:58.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5969999999999998e-06 2024-03-20T09:07:58.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.918 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.918 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.127278+0000", 2024-03-20T09:07:58.918 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.248356488 2024-03-20T09:07:58.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.918 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:07:58.918 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.127425+0000", 2024-03-20T09:07:58.918 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014669100000000001 2024-03-20T09:07:58.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.918 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2,7,8", 2024-03-20T09:07:58.919 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.127605+0000", 2024-03-20T09:07:58.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000180087 2024-03-20T09:07:58.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.919 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.919 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.134547+0000", 2024-03-20T09:07:58.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0069416720000000003 2024-03-20T09:07:58.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.919 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.919 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.134623+0000", 2024-03-20T09:07:58.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6108999999999999e-05 2024-03-20T09:07:58.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:07:58.920 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.134711+0000", 2024-03-20T09:07:58.920 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7955999999999996e-05 2024-03-20T09:07:58.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.920 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.140582+0000", 2024-03-20T09:07:58.920 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0058715759999999999 2024-03-20T09:07:58.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:07:58.920 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.140623+0000", 2024-03-20T09:07:58.920 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0496e-05 2024-03-20T09:07:58.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.921 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:07:58.921 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.141747+0000", 2024-03-20T09:07:58.921 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011239049999999999 2024-03-20T09:07:58.921 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.921 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:07:58.921 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.921 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:109493 3.19 3:9d0cc81e:::benchmark_data_smithi122_119671_object9711:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-03-20T09:07:58.921 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:05:19.098145+0000", 2024-03-20T09:07:58.921 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 159.74772934800001, 2024-03-20T09:07:58.921 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9731130889999999, 2024-03-20T09:07:58.922 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:07:58.922 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:07:58.922 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:07:58.922 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:07:58.922 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:07:58.922 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 109493 2024-03-20T09:07:58.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.922 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:07:58.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.922 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:07:58.922 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.098145+0000", 2024-03-20T09:07:58.922 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:07:58.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.098145+0000", 2024-03-20T09:07:58.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:07:58.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.098146+0000", 2024-03-20T09:07:58.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.176e-06 2024-03-20T09:07:58.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:07:58.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.098152+0000", 2024-03-20T09:07:58.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5180000000000004e-06 2024-03-20T09:07:58.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.924 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:07:58.924 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.098168+0000", 2024-03-20T09:07:58.924 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6727000000000001e-05 2024-03-20T09:07:58.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.924 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:07:58.924 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.098177+0000", 2024-03-20T09:07:58.924 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6950000000000006e-06 2024-03-20T09:07:58.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.924 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.924 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.098239+0000", 2024-03-20T09:07:58.924 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1574000000000004e-05 2024-03-20T09:07:58.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.925 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:07:58.925 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.098246+0000", 2024-03-20T09:07:58.925 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7689999999999996e-06 2024-03-20T09:07:58.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.925 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.925 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.965892+0000", 2024-03-20T09:07:58.925 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.86764582700000004 2024-03-20T09:07:58.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.925 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:07:58.925 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.965906+0000", 2024-03-20T09:07:58.925 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4141000000000001e-05 2024-03-20T09:07:58.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.926 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.926 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.063274+0000", 2024-03-20T09:07:58.926 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.097367703 2024-03-20T09:07:58.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.926 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:07:58.926 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.063457+0000", 2024-03-20T09:07:58.926 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000182726 2024-03-20T09:07:58.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.926 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4,10", 2024-03-20T09:07:58.926 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.063630+0000", 2024-03-20T09:07:58.927 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000172838 2024-03-20T09:07:58.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.927 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.927 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.067484+0000", 2024-03-20T09:07:58.927 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0038543789999999998 2024-03-20T09:07:58.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.927 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:07:58.927 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.068439+0000", 2024-03-20T09:07:58.927 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095458700000000001 2024-03-20T09:07:58.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.927 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.928 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.070204+0000", 2024-03-20T09:07:58.928 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0017656379999999999 2024-03-20T09:07:58.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.928 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:07:58.928 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.070248+0000", 2024-03-20T09:07:58.928 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3988e-05 2024-03-20T09:07:58.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.928 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:07:58.928 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.071258+0000", 2024-03-20T09:07:58.928 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010098030000000001 2024-03-20T09:07:58.928 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.928 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:07:58.928 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.929 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:109510 3.19 3:9fc561c9:::benchmark_data_smithi122_119671_object9728:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-03-20T09:07:58.929 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:05:19.331524+0000", 2024-03-20T09:07:58.929 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 159.514350431, 2024-03-20T09:07:58.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7485444129999999, 2024-03-20T09:07:58.929 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:07:58.929 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:07:58.929 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:07:58.929 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:07:58.929 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:07:58.929 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 109510 2024-03-20T09:07:58.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.929 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:07:58.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:07:58.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.331524+0000", 2024-03-20T09:07:58.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:07:58.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.331524+0000", 2024-03-20T09:07:58.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:07:58.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.331525+0000", 2024-03-20T09:07:58.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.313e-06 2024-03-20T09:07:58.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:07:58.931 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.331530+0000", 2024-03-20T09:07:58.931 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0810000000000001e-06 2024-03-20T09:07:58.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:07:58.931 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.331544+0000", 2024-03-20T09:07:58.931 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3713999999999999e-05 2024-03-20T09:07:58.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:07:58.931 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.331552+0000", 2024-03-20T09:07:58.931 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3180000000000007e-06 2024-03-20T09:07:58.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.932 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.932 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.331626+0000", 2024-03-20T09:07:58.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4109000000000005e-05 2024-03-20T09:07:58.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.932 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:07:58.932 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.331641+0000", 2024-03-20T09:07:58.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4109e-05 2024-03-20T09:07:58.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.932 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.932 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.965939+0000", 2024-03-20T09:07:58.933 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.63429816299999997 2024-03-20T09:07:58.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.933 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:07:58.933 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.965947+0000", 2024-03-20T09:07:58.933 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5459999999999992e-06 2024-03-20T09:07:58.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.933 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.933 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.066002+0000", 2024-03-20T09:07:58.933 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1000546309999999 2024-03-20T09:07:58.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.933 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:07:58.933 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.066142+0000", 2024-03-20T09:07:58.934 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013997400000000001 2024-03-20T09:07:58.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.934 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4,10", 2024-03-20T09:07:58.934 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.066319+0000", 2024-03-20T09:07:58.934 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000177596 2024-03-20T09:07:58.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.934 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:07:58.934 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.068444+0000", 2024-03-20T09:07:58.934 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021245209999999999 2024-03-20T09:07:58.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.935 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.935 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.069381+0000", 2024-03-20T09:07:58.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093740999999999996 2024-03-20T09:07:58.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.935 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.935 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.078968+0000", 2024-03-20T09:07:58.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0095866749999999994 2024-03-20T09:07:58.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.935 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:07:58.935 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.079048+0000", 2024-03-20T09:07:58.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9920999999999996e-05 2024-03-20T09:07:58.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.935 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:07:58.936 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.080068+0000", 2024-03-20T09:07:58.936 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010203320000000001 2024-03-20T09:07:58.936 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.936 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:07:58.936 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.936 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:109521 3.19 3:9f0303bd:::benchmark_data_smithi122_119671_object9739:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-03-20T09:07:58.936 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:05:19.455979+0000", 2024-03-20T09:07:58.936 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 159.389895628, 2024-03-20T09:07:58.936 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.625088366, 2024-03-20T09:07:58.936 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:07:58.936 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:07:58.936 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:07:58.936 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:07:58.937 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:07:58.937 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 109521 2024-03-20T09:07:58.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.937 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:07:58.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.937 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:07:58.937 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.455979+0000", 2024-03-20T09:07:58.937 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.937 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:07:58.937 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.455979+0000", 2024-03-20T09:07:58.937 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.938 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:07:58.938 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.455980+0000", 2024-03-20T09:07:58.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.161e-06 2024-03-20T09:07:58.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.938 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:07:58.938 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.455984+0000", 2024-03-20T09:07:58.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2969999999999997e-06 2024-03-20T09:07:58.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.938 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:07:58.938 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.455995+0000", 2024-03-20T09:07:58.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.064e-05 2024-03-20T09:07:58.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.939 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:07:58.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.456001+0000", 2024-03-20T09:07:58.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9959999999999999e-06 2024-03-20T09:07:58.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.939 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.456054+0000", 2024-03-20T09:07:58.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3532000000000002e-05 2024-03-20T09:07:58.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.939 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:07:58.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.456063+0000", 2024-03-20T09:07:58.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9129999999999992e-06 2024-03-20T09:07:58.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.940 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.940 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.966087+0000", 2024-03-20T09:07:58.940 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.51002333899999996 2024-03-20T09:07:58.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.940 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:07:58.940 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.966096+0000", 2024-03-20T09:07:58.940 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5799999999999998e-06 2024-03-20T09:07:58.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.940 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.940 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.067530+0000", 2024-03-20T09:07:58.941 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1014342690000001 2024-03-20T09:07:58.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.941 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:07:58.941 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.067631+0000", 2024-03-20T09:07:58.941 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000100739 2024-03-20T09:07:58.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.941 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4,10", 2024-03-20T09:07:58.941 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.067716+0000", 2024-03-20T09:07:58.941 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4911000000000001e-05 2024-03-20T09:07:58.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.941 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:07:58.942 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.068516+0000", 2024-03-20T09:07:58.942 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079979400000000005 2024-03-20T09:07:58.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.942 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.942 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.069500+0000", 2024-03-20T09:07:58.942 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098445499999999997 2024-03-20T09:07:58.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.942 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.942 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.080148+0000", 2024-03-20T09:07:58.942 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010647933 2024-03-20T09:07:58.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.942 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:07:58.943 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.080188+0000", 2024-03-20T09:07:58.943 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9891000000000001e-05 2024-03-20T09:07:58.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.943 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:07:58.943 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.081067+0000", 2024-03-20T09:07:58.943 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00087891599999999999 2024-03-20T09:07:58.943 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.943 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:07:58.943 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.943 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:111215 3.19 3:992593d6:::benchmark_data_smithi122_119671_object11433:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-20T09:07:58.943 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:05:36.211371+0000", 2024-03-20T09:07:58.944 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 142.63450319099999, 2024-03-20T09:07:58.944 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.52400578499999995, 2024-03-20T09:07:58.944 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:07:58.944 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:07:58.944 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:07:58.944 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:07:58.944 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:07:58.944 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 111215 2024-03-20T09:07:58.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.944 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:07:58.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.944 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:07:58.944 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.211371+0000", 2024-03-20T09:07:58.944 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.945 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:07:58.945 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.211371+0000", 2024-03-20T09:07:58.945 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.945 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:07:58.945 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.211373+0000", 2024-03-20T09:07:58.945 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4589999999999999e-06 2024-03-20T09:07:58.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.945 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:07:58.945 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.211379+0000", 2024-03-20T09:07:58.946 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4119999999999998e-06 2024-03-20T09:07:58.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.946 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.946 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:07:58.946 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.211396+0000", 2024-03-20T09:07:58.946 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7309999999999999e-05 2024-03-20T09:07:58.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.946 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.946 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:07:58.946 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.211406+0000", 2024-03-20T09:07:58.946 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5780000000000005e-06 2024-03-20T09:07:58.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.946 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.946 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.947 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.211466+0000", 2024-03-20T09:07:58.947 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9957000000000002e-05 2024-03-20T09:07:58.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.947 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:07:58.947 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.211475+0000", 2024-03-20T09:07:58.947 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5200000000000003e-06 2024-03-20T09:07:58.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.947 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.947 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.731345+0000", 2024-03-20T09:07:58.947 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.51986975800000002 2024-03-20T09:07:58.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.948 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:07:58.948 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.731474+0000", 2024-03-20T09:07:58.948 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012923500000000001 2024-03-20T09:07:58.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.948 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4,10", 2024-03-20T09:07:58.948 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.731558+0000", 2024-03-20T09:07:58.948 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3554999999999995e-05 2024-03-20T09:07:58.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.948 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.948 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.733276+0000", 2024-03-20T09:07:58.948 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0017183820000000001 2024-03-20T09:07:58.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.949 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:07:58.949 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.734434+0000", 2024-03-20T09:07:58.949 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011574210000000001 2024-03-20T09:07:58.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.949 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.949 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.735293+0000", 2024-03-20T09:07:58.949 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085930700000000002 2024-03-20T09:07:58.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.949 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:07:58.949 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.735338+0000", 2024-03-20T09:07:58.949 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4855000000000003e-05 2024-03-20T09:07:58.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.950 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:07:58.950 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.735377+0000", 2024-03-20T09:07:58.950 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9035999999999997e-05 2024-03-20T09:07:58.950 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.950 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:07:58.950 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.950 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:111241 3.19 3:9a97667f:::benchmark_data_smithi122_119671_object11459:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-20T09:07:58.950 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:05:36.490833+0000", 2024-03-20T09:07:58.950 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 142.35504093, 2024-03-20T09:07:58.950 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.24470658000000001, 2024-03-20T09:07:58.950 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:07:58.951 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:07:58.951 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:07:58.951 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:07:58.951 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:07:58.951 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 111241 2024-03-20T09:07:58.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.951 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:07:58.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.951 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:07:58.951 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.490833+0000", 2024-03-20T09:07:58.951 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.951 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:07:58.952 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.490833+0000", 2024-03-20T09:07:58.952 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.952 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:07:58.952 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.490834+0000", 2024-03-20T09:07:58.952 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6899999999999996e-07 2024-03-20T09:07:58.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.952 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:07:58.952 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.490838+0000", 2024-03-20T09:07:58.952 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0729999999999998e-06 2024-03-20T09:07:58.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.953 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:07:58.953 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.490849+0000", 2024-03-20T09:07:58.953 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0223e-05 2024-03-20T09:07:58.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.953 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:07:58.953 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.490856+0000", 2024-03-20T09:07:58.953 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7659999999999998e-06 2024-03-20T09:07:58.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.953 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.953 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.490910+0000", 2024-03-20T09:07:58.953 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3309000000000001e-05 2024-03-20T09:07:58.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.954 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:07:58.954 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.490919+0000", 2024-03-20T09:07:58.954 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.952e-06 2024-03-20T09:07:58.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.954 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.954 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.733310+0000", 2024-03-20T09:07:58.954 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.24239117099999999 2024-03-20T09:07:58.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.954 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:07:58.954 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.733395+0000", 2024-03-20T09:07:58.954 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5097000000000002e-05 2024-03-20T09:07:58.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.955 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4,10", 2024-03-20T09:07:58.955 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.733465+0000", 2024-03-20T09:07:58.955 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0050999999999997e-05 2024-03-20T09:07:58.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.955 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.955 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.734561+0000", 2024-03-20T09:07:58.955 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010958039999999999 2024-03-20T09:07:58.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.955 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:07:58.955 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.734587+0000", 2024-03-20T09:07:58.955 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6163999999999999e-05 2024-03-20T09:07:58.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.956 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.956 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.735433+0000", 2024-03-20T09:07:58.956 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084601800000000003 2024-03-20T09:07:58.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.956 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:07:58.956 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.735491+0000", 2024-03-20T09:07:58.956 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8517000000000002e-05 2024-03-20T09:07:58.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.956 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:07:58.956 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.735540+0000", 2024-03-20T09:07:58.956 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8566000000000002e-05 2024-03-20T09:07:58.957 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.957 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:07:58.957 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.957 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:114288 3.18 3.4c1b3578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e311)", 2024-03-20T09:07:58.957 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:06:06.044904+0000", 2024-03-20T09:07:58.957 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 112.800970288, 2024-03-20T09:07:58.957 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.87759928399999998, 2024-03-20T09:07:58.957 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:07:58.957 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:07:58.957 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:07:58.957 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:07:58.957 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:07:58.957 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 114288 2024-03-20T09:07:58.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.958 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:07:58.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.958 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:07:58.958 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:06.044904+0000", 2024-03-20T09:07:58.958 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.958 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:07:58.958 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:06.044904+0000", 2024-03-20T09:07:58.958 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.958 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:07:58.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:06.044905+0000", 2024-03-20T09:07:58.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.339e-06 2024-03-20T09:07:58.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:07:58.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:06.044911+0000", 2024-03-20T09:07:58.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8760000000000002e-06 2024-03-20T09:07:58.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:07:58.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:06.044926+0000", 2024-03-20T09:07:58.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4569999999999999e-05 2024-03-20T09:07:58.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:07:58.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:06.044935+0000", 2024-03-20T09:07:58.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4509999999999998e-06 2024-03-20T09:07:58.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:06.044995+0000", 2024-03-20T09:07:58.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9981000000000001e-05 2024-03-20T09:07:58.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:07:58.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:06.045005+0000", 2024-03-20T09:07:58.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3500000000000003e-06 2024-03-20T09:07:58.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.961 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:06.922494+0000", 2024-03-20T09:07:58.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.87748940499999994 2024-03-20T09:07:58.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:07:58.961 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:06.922503+0000", 2024-03-20T09:07:58.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.312e-06 2024-03-20T09:07:58.961 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.961 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:07:58.961 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.962 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:116322 3.18 3:1e15a469:::benchmark_data_smithi122_119671_object16540:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-03-20T09:07:58.962 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:06:24.334626+0000", 2024-03-20T09:07:58.962 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 94.511248094999999, 2024-03-20T09:07:58.962 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.66543802200000002, 2024-03-20T09:07:58.962 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:07:58.962 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:07:58.962 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:07:58.962 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:07:58.962 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:07:58.962 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 116322 2024-03-20T09:07:58.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.962 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:07:58.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.962 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:07:58.963 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:24.334626+0000", 2024-03-20T09:07:58.963 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.963 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:07:58.963 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:24.334626+0000", 2024-03-20T09:07:58.963 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.964 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:07:58.964 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:24.334628+0000", 2024-03-20T09:07:58.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.409e-06 2024-03-20T09:07:58.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.964 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:07:58.964 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:24.334634+0000", 2024-03-20T09:07:58.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5780000000000004e-06 2024-03-20T09:07:58.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:07:58.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:24.334648+0000", 2024-03-20T09:07:58.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3879999999999999e-05 2024-03-20T09:07:58.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:07:58.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:24.334655+0000", 2024-03-20T09:07:58.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3549999999999999e-06 2024-03-20T09:07:58.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.966 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:24.334717+0000", 2024-03-20T09:07:58.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1235999999999994e-05 2024-03-20T09:07:58.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.966 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:07:58.966 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:24.334728+0000", 2024-03-20T09:07:58.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1374e-05 2024-03-20T09:07:58.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.966 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.966 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:24.995249+0000", 2024-03-20T09:07:58.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.66052049400000001 2024-03-20T09:07:58.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:07:58.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:24.995376+0000", 2024-03-20T09:07:58.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000127093 2024-03-20T09:07:58.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2,8", 2024-03-20T09:07:58.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:24.995497+0000", 2024-03-20T09:07:58.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012109599999999999 2024-03-20T09:07:58.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:24.998800+0000", 2024-03-20T09:07:58.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0033034000000000002 2024-03-20T09:07:58.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:07:58.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:24.998814+0000", 2024-03-20T09:07:58.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3971999999999999e-05 2024-03-20T09:07:58.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:24.999946+0000", 2024-03-20T09:07:58.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011315019999999999 2024-03-20T09:07:58.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:07:58.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:25.000013+0000", 2024-03-20T09:07:58.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7111000000000002e-05 2024-03-20T09:07:58.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:07:58.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:25.000064+0000", 2024-03-20T09:07:58.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1521999999999999e-05 2024-03-20T09:07:58.969 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.969 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:07:58.969 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.969 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:127405 3.19 3:9e576a5f:::benchmark_data_smithi122_119671_object27623:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-03-20T09:07:58.969 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:07:49.241955+0000", 2024-03-20T09:07:58.969 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 9.6039192310000008, 2024-03-20T09:07:58.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.501775818, 2024-03-20T09:07:58.970 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:07:58.970 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:07:58.970 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:07:58.970 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:07:58.970 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:07:58.970 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 127405 2024-03-20T09:07:58.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.970 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:07:58.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:07:58.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:49.241955+0000", 2024-03-20T09:07:58.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:07:58.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:49.241955+0000", 2024-03-20T09:07:58.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:07:58.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:49.241957+0000", 2024-03-20T09:07:58.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6959999999999999e-06 2024-03-20T09:07:58.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:07:58.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:49.241962+0000", 2024-03-20T09:07:58.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4970000000000001e-06 2024-03-20T09:07:58.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:07:58.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:49.241976+0000", 2024-03-20T09:07:58.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3421000000000001e-05 2024-03-20T09:07:58.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:07:58.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:49.241983+0000", 2024-03-20T09:07:58.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5009999999999998e-06 2024-03-20T09:07:58.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:49.242041+0000", 2024-03-20T09:07:58.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7432999999999997e-05 2024-03-20T09:07:58.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:07:58.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:49.242050+0000", 2024-03-20T09:07:58.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.6940000000000007e-06 2024-03-20T09:07:58.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:49.564641+0000", 2024-03-20T09:07:58.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.32259047800000001 2024-03-20T09:07:58.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:07:58.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:49.564656+0000", 2024-03-20T09:07:58.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4891e-05 2024-03-20T09:07:58.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:50.731937+0000", 2024-03-20T09:07:58.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.167281504 2024-03-20T09:07:58.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:07:58.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:50.732042+0000", 2024-03-20T09:07:58.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010509999999999999 2024-03-20T09:07:58.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2,10", 2024-03-20T09:07:58.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:50.732155+0000", 2024-03-20T09:07:58.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011228 2024-03-20T09:07:58.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:07:58.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:50.738251+0000", 2024-03-20T09:07:58.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0060966639999999999 2024-03-20T09:07:58.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:50.741081+0000", 2024-03-20T09:07:58.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028295780000000001 2024-03-20T09:07:58.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:50.741708+0000", 2024-03-20T09:07:58.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00062702899999999998 2024-03-20T09:07:58.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:07:58.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:50.741799+0000", 2024-03-20T09:07:58.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0753000000000003e-05 2024-03-20T09:07:58.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:07:58.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:50.743731+0000", 2024-03-20T09:07:58.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001932299 2024-03-20T09:07:58.977 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.978 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:07:58.978 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.978 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:127415 3.19 3:9809a41d:::benchmark_data_smithi122_119671_object27633:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-03-20T09:07:58.978 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:07:49.853687+0000", 2024-03-20T09:07:58.978 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 8.9921874339999999, 2024-03-20T09:07:58.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.89167427600000004, 2024-03-20T09:07:58.978 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:07:58.978 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:07:58.978 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:07:58.978 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:07:58.978 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:07:58.979 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 127415 2024-03-20T09:07:58.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.979 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:07:58.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:07:58.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:49.853687+0000", 2024-03-20T09:07:58.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:07:58.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:49.853687+0000", 2024-03-20T09:07:58.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:07:58.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:07:58.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:49.853689+0000", 2024-03-20T09:07:58.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.52e-06 2024-03-20T09:07:58.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:07:58.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:49.853701+0000", 2024-03-20T09:07:58.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1824000000000001e-05 2024-03-20T09:07:58.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:07:58.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:49.853741+0000", 2024-03-20T09:07:58.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9725000000000003e-05 2024-03-20T09:07:58.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:07:58.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:49.853749+0000", 2024-03-20T09:07:58.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3289999999999994e-06 2024-03-20T09:07:58.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:49.853825+0000", 2024-03-20T09:07:58.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5434999999999999e-05 2024-03-20T09:07:58.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:07:58.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:49.853834+0000", 2024-03-20T09:07:58.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7210000000000006e-06 2024-03-20T09:07:58.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:07:58.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:50.735898+0000", 2024-03-20T09:07:58.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.88206368400000001 2024-03-20T09:07:58.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:07:58.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:50.736077+0000", 2024-03-20T09:07:58.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000179047 2024-03-20T09:07:58.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2,10", 2024-03-20T09:07:58.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:50.736232+0000", 2024-03-20T09:07:58.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015514599999999999 2024-03-20T09:07:58.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:07:58.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:50.738260+0000", 2024-03-20T09:07:58.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0020276980000000001 2024-03-20T09:07:58.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:50.741131+0000", 2024-03-20T09:07:58.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028708879999999998 2024-03-20T09:07:58.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:07:58.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:50.743806+0000", 2024-03-20T09:07:58.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026751879999999998 2024-03-20T09:07:58.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:07:58.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:50.743877+0000", 2024-03-20T09:07:58.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0698000000000006e-05 2024-03-20T09:07:58.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:07:58.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:07:58.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:07:58.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:50.745361+0000", 2024-03-20T09:07:58.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001484373 2024-03-20T09:07:58.984 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.984 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:07:58.985 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.985 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:07:58.985 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:07:58.985 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:07:58.985 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:07:58.986 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:58 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3754070989' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:07:58.986 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:58 smithi115 ceph-mon[72476]: osdmap e374: 12 total, 11 up, 9 in 2024-03-20T09:07:59.018 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:59.041 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:07:59.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:59.046 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:07:59.075 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:07:59.147 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:07:59.176 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:59.203 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:07:59.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:59.207 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:07:59.308 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:07:59.340 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:59.362 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:07:59.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:59.366 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:07:59.466 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:07:59.500 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:59.526 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:07:59.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:59.532 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:07:59.632 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:07:59.667 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:59.689 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-20T09:07:59.691 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:07:59.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:59.694 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:07:59.781 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:07:59.795 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:07:59.826 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:07:59.849 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:07:59.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:07:59.852 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:07:59.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:07:59 smithi122 ceph-mon[73495]: pgmap v914: 57 pgs: 1 active+recovering+remapped, 1 activating+undersized+remapped, 1 activating+undersized+degraded+remapped, 10 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.4 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 19452/216663 objects degraded (8.978%); 41783/216663 objects misplaced (19.285%); 29 MiB/s, 469 objects/s recovering 2024-03-20T09:07:59.953 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:07:59.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:07:59 smithi026 ceph-mon[68309]: pgmap v914: 57 pgs: 1 active+recovering+remapped, 1 activating+undersized+remapped, 1 activating+undersized+degraded+remapped, 10 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.4 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 19452/216663 objects degraded (8.978%); 41783/216663 objects misplaced (19.285%); 29 MiB/s, 469 objects/s recovering 2024-03-20T09:08:00.026 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:00.067 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:08:00.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:00.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:08:00.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:07:59 smithi115 ceph-mon[72476]: pgmap v914: 57 pgs: 1 active+recovering+remapped, 1 activating+undersized+remapped, 1 activating+undersized+degraded+remapped, 10 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.4 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 19452/216663 objects degraded (8.978%); 41783/216663 objects misplaced (19.285%); 29 MiB/s, 469 objects/s recovering 2024-03-20T09:08:00.181 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:08:00.213 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:00.233 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:08:00.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:00.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:08:00.340 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:08:00.369 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:00.392 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:08:00.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:00.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:08:00.498 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:08:00.548 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:00.639 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:08:00.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:00.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:08:00.745 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:08:00.784 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:00.817 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 8, 10, 11] out_osds: [9, 3, 4] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11] 2024-03-20T09:08:00.817 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:08:00.817 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-03-20T09:08:00.817 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-20T09:08:00.817 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-20T09:08:00.818 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-03-20T09:08:00.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:00 smithi122 ceph-mon[73495]: osdmap e375: 12 total, 11 up, 9 in 2024-03-20T09:08:00.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:00 smithi026 ceph-mon[68309]: osdmap e375: 12 total, 11 up, 9 in 2024-03-20T09:08:01.079 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:08:01.154 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:08:01.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:00 smithi115 ceph-mon[72476]: osdmap e375: 12 total, 11 up, 9 in 2024-03-20T09:08:01.359 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:08:01.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:01.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:08:01.467 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:08:01.542 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:01.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:01 smithi026 ceph-mon[68309]: pgmap v917: 57 pgs: 1 active+undersized+degraded+remapped, 1 active+recovering+remapped, 1 activating+undersized+remapped, 10 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.4 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 7 op/s; 19452/216663 objects degraded (8.978%); 40755/216663 objects misplaced (18.810%); 39 MiB/s, 630 objects/s recovering 2024-03-20T09:08:01.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:01 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:08:01.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:01 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:08:01.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:01 smithi026 ceph-mon[68309]: osdmap e376: 12 total, 11 up, 9 in 2024-03-20T09:08:02.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:01 smithi115 ceph-mon[72476]: pgmap v917: 57 pgs: 1 active+undersized+degraded+remapped, 1 active+recovering+remapped, 1 activating+undersized+remapped, 10 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.4 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 7 op/s; 19452/216663 objects degraded (8.978%); 40755/216663 objects misplaced (18.810%); 39 MiB/s, 630 objects/s recovering 2024-03-20T09:08:02.195 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:01 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:08:02.195 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:01 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:08:02.195 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:01 smithi115 ceph-mon[72476]: osdmap e376: 12 total, 11 up, 9 in 2024-03-20T09:08:02.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:01 smithi122 ceph-mon[73495]: pgmap v917: 57 pgs: 1 active+undersized+degraded+remapped, 1 active+recovering+remapped, 1 activating+undersized+remapped, 10 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.4 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 7 op/s; 19452/216663 objects degraded (8.978%); 40755/216663 objects misplaced (18.810%); 39 MiB/s, 630 objects/s recovering 2024-03-20T09:08:02.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:01 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:08:02.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:01 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:08:02.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:01 smithi122 ceph-mon[73495]: osdmap e376: 12 total, 11 up, 9 in 2024-03-20T09:08:02.213 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:08:02.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:02.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:08:02.323 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:08:02.377 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:02.746 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:08:02.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:02.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:08:02.852 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:08:02.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:02 smithi115 ceph-mon[72476]: pgmap v919: 57 pgs: 1 peering, 1 active+recovering+remapped, 1 active+undersized+degraded+remapped, 10 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.4 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 3 op/s; 19448/216609 objects degraded (8.978%); 37916/216609 objects misplaced (17.504%); 47 MiB/s, 756 objects/s recovering 2024-03-20T09:08:02.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:02 smithi115 ceph-mon[72476]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-20T09:08:02.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:02 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 19448/216609 objects degraded (8.978%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-20T09:08:02.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:02 smithi115 ceph-mon[72476]: osdmap e377: 12 total, 11 up, 9 in 2024-03-20T09:08:02.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:02 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2156492934' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:08:02.911 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:02 smithi026 ceph-mon[68309]: pgmap v919: 57 pgs: 1 peering, 1 active+recovering+remapped, 1 active+undersized+degraded+remapped, 10 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.4 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 3 op/s; 19448/216609 objects degraded (8.978%); 37916/216609 objects misplaced (17.504%); 47 MiB/s, 756 objects/s recovering 2024-03-20T09:08:02.911 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:02 smithi026 ceph-mon[68309]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-20T09:08:02.911 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:02 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 19448/216609 objects degraded (8.978%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-20T09:08:02.912 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:02 smithi026 ceph-mon[68309]: osdmap e377: 12 total, 11 up, 9 in 2024-03-20T09:08:02.912 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:02 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2156492934' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:08:02.914 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:02.986 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:08:02.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:02.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:08:03.011 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:08:03.093 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:08:03.143 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:03.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:02 smithi122 ceph-mon[73495]: pgmap v919: 57 pgs: 1 peering, 1 active+recovering+remapped, 1 active+undersized+degraded+remapped, 10 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.4 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 3 op/s; 19448/216609 objects degraded (8.978%); 37916/216609 objects misplaced (17.504%); 47 MiB/s, 756 objects/s recovering 2024-03-20T09:08:03.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:02 smithi122 ceph-mon[73495]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-20T09:08:03.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:02 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 19448/216609 objects degraded (8.978%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-20T09:08:03.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:02 smithi122 ceph-mon[73495]: osdmap e377: 12 total, 11 up, 9 in 2024-03-20T09:08:03.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:02 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2156492934' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:08:03.308 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:08:03.308 INFO:teuthology.orchestra.run.smithi115.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-20T09:08:03.308 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:08:03.503 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:08:03.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:03.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:08:03.515 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:08:03.516 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:08:03.516 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-03-20T09:08:03.516 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:08:03.615 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:08:03.643 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:03.667 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:08:03.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:03.670 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:08:03.771 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:08:03.803 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:03.826 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:08:03.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:03.830 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:08:03.931 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:08:03.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:03 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:08:03.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:03 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2156492934' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:08:03.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:03 smithi122 ceph-mon[73495]: osdmap e378: 12 total, 11 up, 9 in 2024-03-20T09:08:03.963 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:03 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:08:03.963 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:03 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2156492934' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:08:03.963 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:03 smithi115 ceph-mon[72476]: osdmap e378: 12 total, 11 up, 9 in 2024-03-20T09:08:03.966 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:03.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:03 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:08:03.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:03 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2156492934' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:08:03.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:03 smithi026 ceph-mon[68309]: osdmap e378: 12 total, 11 up, 9 in 2024-03-20T09:08:04.915 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-20T09:08:05.224 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:04 smithi026 ceph-mon[68309]: pgmap v922: 57 pgs: 1 peering, 1 active+recovering+remapped, 1 active+undersized+degraded+remapped, 10 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.4 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 5 op/s; 19446/216591 objects degraded (8.978%); 37327/216591 objects misplaced (17.234%); 48 MiB/s, 775 objects/s recovering 2024-03-20T09:08:05.376 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:08:05.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:04 smithi115 ceph-mon[72476]: pgmap v922: 57 pgs: 1 peering, 1 active+recovering+remapped, 1 active+undersized+degraded+remapped, 10 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.4 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 5 op/s; 19446/216591 objects degraded (8.978%); 37327/216591 objects misplaced (17.234%); 48 MiB/s, 775 objects/s recovering 2024-03-20T09:08:05.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:04 smithi122 ceph-mon[73495]: pgmap v922: 57 pgs: 1 peering, 1 active+recovering+remapped, 1 active+undersized+degraded+remapped, 10 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 28 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.4 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 5 op/s; 19446/216591 objects degraded (8.978%); 37327/216591 objects misplaced (17.234%); 48 MiB/s, 775 objects/s recovering 2024-03-20T09:08:06.109 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:08:06.159 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:05 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:08:06.160 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:05 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:08:06.160 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:05 smithi026 ceph-mon[68309]: osdmap e379: 12 total, 11 up, 9 in 2024-03-20T09:08:06.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:05 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:08:06.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:05 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:08:06.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:05 smithi115 ceph-mon[72476]: osdmap e379: 12 total, 11 up, 9 in 2024-03-20T09:08:06.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:05 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:08:06.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:05 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:08:06.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:05 smithi122 ceph-mon[73495]: osdmap e379: 12 total, 11 up, 9 in 2024-03-20T09:08:06.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:06.575 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:08:06.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:08:06.676 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:08:06.707 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:06.730 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:08:06.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:06.733 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:08:06.835 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:08:06.861 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:06.883 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:08:06.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:06.886 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:08:06.987 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:08:07.036 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:07.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:06 smithi115 ceph-mon[72476]: pgmap v924: 57 pgs: 1 active+recovering+remapped, 10 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 29 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.4 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 9 op/s; 19441/216501 objects degraded (8.980%); 34884/216501 objects misplaced (16.113%); 49 MiB/s, 780 objects/s recovering 2024-03-20T09:08:07.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:06 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 19441/216501 objects degraded (8.980%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-20T09:08:07.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:06 smithi122 ceph-mon[73495]: pgmap v924: 57 pgs: 1 active+recovering+remapped, 10 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 29 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.4 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 9 op/s; 19441/216501 objects degraded (8.980%); 34884/216501 objects misplaced (16.113%); 49 MiB/s, 780 objects/s recovering 2024-03-20T09:08:07.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:06 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 19441/216501 objects degraded (8.980%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-20T09:08:07.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:06 smithi026 ceph-mon[68309]: pgmap v924: 57 pgs: 1 active+recovering+remapped, 10 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 29 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.4 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 9 op/s; 19441/216501 objects degraded (8.980%); 34884/216501 objects misplaced (16.113%); 49 MiB/s, 780 objects/s recovering 2024-03-20T09:08:07.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:06 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 19441/216501 objects degraded (8.980%), 10 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-20T09:08:09.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:08 smithi115 ceph-mon[72476]: pgmap v925: 57 pgs: 1 active+recovering+remapped, 10 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 29 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.4 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 5 op/s; 19441/216501 objects degraded (8.980%); 33428/216501 objects misplaced (15.440%); 44 MiB/s, 734 objects/s recovering 2024-03-20T09:08:09.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:08 smithi122 ceph-mon[73495]: pgmap v925: 57 pgs: 1 active+recovering+remapped, 10 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 29 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.4 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 5 op/s; 19441/216501 objects degraded (8.980%); 33428/216501 objects misplaced (15.440%); 44 MiB/s, 734 objects/s recovering 2024-03-20T09:08:09.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:08 smithi026 ceph-mon[68309]: pgmap v925: 57 pgs: 1 active+recovering+remapped, 10 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 29 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.4 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 5 op/s; 19441/216501 objects degraded (8.980%); 33428/216501 objects misplaced (15.440%); 44 MiB/s, 734 objects/s recovering 2024-03-20T09:08:10.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:09 smithi115 ceph-mon[72476]: osdmap e380: 12 total, 11 up, 9 in 2024-03-20T09:08:10.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:09 smithi122 ceph-mon[73495]: osdmap e380: 12 total, 11 up, 9 in 2024-03-20T09:08:10.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:09 smithi026 ceph-mon[68309]: osdmap e380: 12 total, 11 up, 9 in 2024-03-20T09:08:11.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:11 smithi115 ceph-mon[72476]: pgmap v927: 57 pgs: 1 active+recovering+remapped, 10 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 29 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 5 op/s; 19441/216501 objects degraded (8.980%); 31793/216501 objects misplaced (14.685%); 54 MiB/s, 899 objects/s recovering 2024-03-20T09:08:11.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:11 smithi115 ceph-mon[72476]: osdmap e381: 12 total, 11 up, 9 in 2024-03-20T09:08:11.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:11 smithi122 ceph-mon[73495]: pgmap v927: 57 pgs: 1 active+recovering+remapped, 10 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 29 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 5 op/s; 19441/216501 objects degraded (8.980%); 31793/216501 objects misplaced (14.685%); 54 MiB/s, 899 objects/s recovering 2024-03-20T09:08:11.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:11 smithi122 ceph-mon[73495]: osdmap e381: 12 total, 11 up, 9 in 2024-03-20T09:08:11.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:11 smithi026 ceph-mon[68309]: pgmap v927: 57 pgs: 1 active+recovering+remapped, 10 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 29 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 4.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 5 op/s; 19441/216501 objects degraded (8.980%); 31793/216501 objects misplaced (14.685%); 54 MiB/s, 899 objects/s recovering 2024-03-20T09:08:11.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:11 smithi026 ceph-mon[68309]: osdmap e381: 12 total, 11 up, 9 in 2024-03-20T09:08:11.533 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 8, 10, 11] out_osds: [9, 3, 4] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11] 2024-03-20T09:08:11.534 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:08:11.534 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-20T09:08:11.534 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-20T09:08:12.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:12 smithi115 ceph-mon[72476]: osdmap e382: 12 total, 11 up, 9 in 2024-03-20T09:08:12.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:12 smithi115 ceph-mon[72476]: osdmap e383: 12 total, 11 up, 9 in 2024-03-20T09:08:12.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:12 smithi122 ceph-mon[73495]: osdmap e382: 12 total, 11 up, 9 in 2024-03-20T09:08:12.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:12 smithi122 ceph-mon[73495]: osdmap e383: 12 total, 11 up, 9 in 2024-03-20T09:08:12.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:12 smithi026 ceph-mon[68309]: osdmap e382: 12 total, 11 up, 9 in 2024-03-20T09:08:12.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:12 smithi026 ceph-mon[68309]: osdmap e383: 12 total, 11 up, 9 in 2024-03-20T09:08:13.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:13 smithi115 ceph-mon[72476]: pgmap v930: 57 pgs: 1 active+recovering+remapped, 10 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 30 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 4.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 19441/216501 objects degraded (8.980%); 29189/216501 objects misplaced (13.482%); 54 MiB/s, 1.11k objects/s recovering 2024-03-20T09:08:13.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:13 smithi115 ceph-mon[72476]: osdmap e384: 12 total, 11 up, 9 in 2024-03-20T09:08:13.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:13 smithi122 ceph-mon[73495]: pgmap v930: 57 pgs: 1 active+recovering+remapped, 10 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 30 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 4.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 19441/216501 objects degraded (8.980%); 29189/216501 objects misplaced (13.482%); 54 MiB/s, 1.11k objects/s recovering 2024-03-20T09:08:13.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:13 smithi122 ceph-mon[73495]: osdmap e384: 12 total, 11 up, 9 in 2024-03-20T09:08:13.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:13 smithi026 ceph-mon[68309]: pgmap v930: 57 pgs: 1 active+recovering+remapped, 10 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 30 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 4.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 19441/216501 objects degraded (8.980%); 29189/216501 objects misplaced (13.482%); 54 MiB/s, 1.11k objects/s recovering 2024-03-20T09:08:13.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:13 smithi026 ceph-mon[68309]: osdmap e384: 12 total, 11 up, 9 in 2024-03-20T09:08:13.734 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:08:14.013 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:08:14.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:14.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:08:14.119 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:08:14.351 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:08:14.428 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:08:14.428 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:08:14.428 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-03-20T09:08:14.428 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-03-20T09:08:14.428 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:08:14.657 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:14.753 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:08:14.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:14.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:08:15.041 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:08:15.111 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:15 smithi115 ceph-mon[72476]: pgmap v933: 57 pgs: 1 active+recovering+remapped, 10 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 31 active+clean, 5 active+undersized; 4.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 19441/216501 objects degraded (8.980%); 29094/216501 objects misplaced (13.438%); 35 MiB/s, 1.07k objects/s recovering 2024-03-20T09:08:15.111 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:15 smithi115 ceph-mon[72476]: osdmap e385: 12 total, 11 up, 9 in 2024-03-20T09:08:15.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:15 smithi026 ceph-mon[68309]: pgmap v933: 57 pgs: 1 active+recovering+remapped, 10 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 31 active+clean, 5 active+undersized; 4.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 19441/216501 objects degraded (8.980%); 29094/216501 objects misplaced (13.438%); 35 MiB/s, 1.07k objects/s recovering 2024-03-20T09:08:15.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:15 smithi026 ceph-mon[68309]: osdmap e385: 12 total, 11 up, 9 in 2024-03-20T09:08:15.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:15 smithi122 ceph-mon[73495]: pgmap v933: 57 pgs: 1 active+recovering+remapped, 10 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 31 active+clean, 5 active+undersized; 4.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 19441/216501 objects degraded (8.980%); 29094/216501 objects misplaced (13.438%); 35 MiB/s, 1.07k objects/s recovering 2024-03-20T09:08:15.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:15 smithi122 ceph-mon[73495]: osdmap e385: 12 total, 11 up, 9 in 2024-03-20T09:08:16.202 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:16 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:08:16.203 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:16 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:08:16.203 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:16 smithi026 ceph-mon[68309]: osdmap e386: 12 total, 11 up, 9 in 2024-03-20T09:08:16.203 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:16 smithi026 ceph-mon[68309]: osdmap e387: 12 total, 11 up, 9 in 2024-03-20T09:08:16.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:16 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:08:16.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:16 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:08:16.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:16 smithi115 ceph-mon[72476]: osdmap e386: 12 total, 11 up, 9 in 2024-03-20T09:08:16.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:16 smithi115 ceph-mon[72476]: osdmap e387: 12 total, 11 up, 9 in 2024-03-20T09:08:16.443 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:08:16.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:16 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:08:16.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:16 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:08:16.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:16 smithi122 ceph-mon[73495]: osdmap e386: 12 total, 11 up, 9 in 2024-03-20T09:08:16.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:16 smithi122 ceph-mon[73495]: osdmap e387: 12 total, 11 up, 9 in 2024-03-20T09:08:16.473 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:08:16.498 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:08:16.520 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:08:16.692 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:16.716 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:08:16.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:16.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:08:16.754 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:08:16.779 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:08:16.801 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:08:16.822 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:08:16.824 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:08:16.848 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:08:16.850 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:16.871 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:08:16.872 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:08:16.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:16.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:08:16.893 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:08:16.893 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:08:16.893 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:08:16.893 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:08:16.893 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:08:16.893 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:08:16.893 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-20T09:08:16.893 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:08:16.893 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:08:16.893 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:08:16.894 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:08:16.894 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:08:16.894 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:08:16.976 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:08:17.005 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:17.033 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:08:17.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:17.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:08:17.138 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:08:17.172 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:17 smithi026 ceph-mon[68309]: pgmap v936: 57 pgs: 1 active+recovering+remapped, 1 peering, 11 active+clean+remapped, 1 active+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 31 active+clean, 5 active+undersized; 4.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 12041/216501 objects degraded (5.562%); 31416/216501 objects misplaced (14.511%); 21 MiB/s, 996 objects/s recovering 2024-03-20T09:08:17.172 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:17 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 12041/216501 objects degraded (5.562%), 7 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:08:17.172 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:17 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2172022227' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:08:17.175 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:17.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:17 smithi115 ceph-mon[72476]: pgmap v936: 57 pgs: 1 active+recovering+remapped, 1 peering, 11 active+clean+remapped, 1 active+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 31 active+clean, 5 active+undersized; 4.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 12041/216501 objects degraded (5.562%); 31416/216501 objects misplaced (14.511%); 21 MiB/s, 996 objects/s recovering 2024-03-20T09:08:17.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:17 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 12041/216501 objects degraded (5.562%), 7 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:08:17.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:17 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2172022227' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:08:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:17 smithi122 ceph-mon[73495]: pgmap v936: 57 pgs: 1 active+recovering+remapped, 1 peering, 11 active+clean+remapped, 1 active+undersized+remapped, 4 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 31 active+clean, 5 active+undersized; 4.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 12041/216501 objects degraded (5.562%); 31416/216501 objects misplaced (14.511%); 21 MiB/s, 996 objects/s recovering 2024-03-20T09:08:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:17 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 12041/216501 objects degraded (5.562%), 7 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:08:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:17 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2172022227' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:08:18.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:18 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:08:18.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:18 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2172022227' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:08:18.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:18 smithi115 ceph-mon[72476]: osdmap e388: 12 total, 11 up, 9 in 2024-03-20T09:08:18.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:18 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:08:18.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:18 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2172022227' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:08:18.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:18 smithi122 ceph-mon[73495]: osdmap e388: 12 total, 11 up, 9 in 2024-03-20T09:08:18.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:18 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:08:18.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:18 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2172022227' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:08:18.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:18 smithi026 ceph-mon[68309]: osdmap e388: 12 total, 11 up, 9 in 2024-03-20T09:08:19.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:19 smithi115 ceph-mon[72476]: pgmap v939: 57 pgs: 1 activating+undersized+degraded+remapped, 1 activating+remapped, 1 active+recovering+remapped, 1 peering, 11 active+clean+remapped, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 30 active+clean, 5 active+undersized; 4.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 26 op/s; 20326/216183 objects degraded (9.402%); 36076/216183 objects misplaced (16.688%); 96 MiB/s, 2.76k objects/s recovering 2024-03-20T09:08:19.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:19 smithi122 ceph-mon[73495]: pgmap v939: 57 pgs: 1 activating+undersized+degraded+remapped, 1 activating+remapped, 1 active+recovering+remapped, 1 peering, 11 active+clean+remapped, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 30 active+clean, 5 active+undersized; 4.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 26 op/s; 20326/216183 objects degraded (9.402%); 36076/216183 objects misplaced (16.688%); 96 MiB/s, 2.76k objects/s recovering 2024-03-20T09:08:19.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:19 smithi026 ceph-mon[68309]: pgmap v939: 57 pgs: 1 activating+undersized+degraded+remapped, 1 activating+remapped, 1 active+recovering+remapped, 1 peering, 11 active+clean+remapped, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 30 active+clean, 5 active+undersized; 4.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 26 op/s; 20326/216183 objects degraded (9.402%); 36076/216183 objects misplaced (16.688%); 96 MiB/s, 2.76k objects/s recovering 2024-03-20T09:08:20.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:20 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:08:20.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:20 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:08:20.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:20 smithi115 ceph-mon[72476]: osdmap e389: 12 total, 11 up, 9 in 2024-03-20T09:08:20.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:20 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:08:20.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:20 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:08:20.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:20 smithi122 ceph-mon[73495]: osdmap e389: 12 total, 11 up, 9 in 2024-03-20T09:08:20.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:20 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:08:20.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:20 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:08:20.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:20 smithi026 ceph-mon[68309]: osdmap e389: 12 total, 11 up, 9 in 2024-03-20T09:08:21.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:21 smithi115 ceph-mon[72476]: pgmap v941: 57 pgs: 1 activating+undersized+degraded+remapped, 1 activating+remapped, 1 active+recovering+remapped, 11 active+clean+remapped, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 31 active+clean, 5 active+undersized; 4.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 22 op/s; 20229/216183 objects degraded (9.357%); 35979/216183 objects misplaced (16.643%); 67 MiB/s, 1.73k objects/s recovering 2024-03-20T09:08:21.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:21 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:08:21.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:21 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:08:21.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:21 smithi115 ceph-mon[72476]: osdmap e390: 12 total, 11 up, 9 in 2024-03-20T09:08:21.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:21 smithi122 ceph-mon[73495]: pgmap v941: 57 pgs: 1 activating+undersized+degraded+remapped, 1 activating+remapped, 1 active+recovering+remapped, 11 active+clean+remapped, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 31 active+clean, 5 active+undersized; 4.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 22 op/s; 20229/216183 objects degraded (9.357%); 35979/216183 objects misplaced (16.643%); 67 MiB/s, 1.73k objects/s recovering 2024-03-20T09:08:21.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:21 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:08:21.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:21 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:08:21.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:21 smithi122 ceph-mon[73495]: osdmap e390: 12 total, 11 up, 9 in 2024-03-20T09:08:21.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:21 smithi026 ceph-mon[68309]: pgmap v941: 57 pgs: 1 activating+undersized+degraded+remapped, 1 activating+remapped, 1 active+recovering+remapped, 11 active+clean+remapped, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 31 active+clean, 5 active+undersized; 4.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 22 op/s; 20229/216183 objects degraded (9.357%); 35979/216183 objects misplaced (16.643%); 67 MiB/s, 1.73k objects/s recovering 2024-03-20T09:08:21.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:21 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:08:21.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:21 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:08:21.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:21 smithi026 ceph-mon[68309]: osdmap e390: 12 total, 11 up, 9 in 2024-03-20T09:08:22.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:22 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 20229/216183 objects degraded (9.357%), 7 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-20T09:08:22.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:22 smithi115 ceph-mon[72476]: osdmap e391: 12 total, 11 up, 9 in 2024-03-20T09:08:22.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:22 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 20229/216183 objects degraded (9.357%), 7 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-20T09:08:22.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:22 smithi122 ceph-mon[73495]: osdmap e391: 12 total, 11 up, 9 in 2024-03-20T09:08:22.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:22 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 20229/216183 objects degraded (9.357%), 7 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-20T09:08:22.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:22 smithi026 ceph-mon[68309]: osdmap e391: 12 total, 11 up, 9 in 2024-03-20T09:08:23.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:23 smithi115 ceph-mon[72476]: pgmap v943: 57 pgs: 1 active+recovering+remapped, 1 activating+undersized+degraded+remapped, 1 activating+remapped, 11 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 18 op/s; 20229/216183 objects degraded (9.357%); 34379/216183 objects misplaced (15.903%); 69 MiB/s, 1.64k objects/s recovering 2024-03-20T09:08:23.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:23 smithi122 ceph-mon[73495]: pgmap v943: 57 pgs: 1 active+recovering+remapped, 1 activating+undersized+degraded+remapped, 1 activating+remapped, 11 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 18 op/s; 20229/216183 objects degraded (9.357%); 34379/216183 objects misplaced (15.903%); 69 MiB/s, 1.64k objects/s recovering 2024-03-20T09:08:23.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:23 smithi026 ceph-mon[68309]: pgmap v943: 57 pgs: 1 active+recovering+remapped, 1 activating+undersized+degraded+remapped, 1 activating+remapped, 11 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 18 op/s; 20229/216183 objects degraded (9.357%); 34379/216183 objects misplaced (15.903%); 69 MiB/s, 1.64k objects/s recovering 2024-03-20T09:08:25.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:25 smithi115 ceph-mon[72476]: pgmap v945: 57 pgs: 1 active+recovering+remapped, 1 activating+undersized+degraded+remapped, 1 activating+remapped, 11 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 20229/216183 objects degraded (9.357%); 34379/216183 objects misplaced (15.903%); 14 MiB/s, 218 objects/s recovering 2024-03-20T09:08:25.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:25 smithi122 ceph-mon[73495]: pgmap v945: 57 pgs: 1 active+recovering+remapped, 1 activating+undersized+degraded+remapped, 1 activating+remapped, 11 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 20229/216183 objects degraded (9.357%); 34379/216183 objects misplaced (15.903%); 14 MiB/s, 218 objects/s recovering 2024-03-20T09:08:25.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:25 smithi026 ceph-mon[68309]: pgmap v945: 57 pgs: 1 active+recovering+remapped, 1 activating+undersized+degraded+remapped, 1 activating+remapped, 11 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.4 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 20229/216183 objects degraded (9.357%); 34379/216183 objects misplaced (15.903%); 14 MiB/s, 218 objects/s recovering 2024-03-20T09:08:26.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:26 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:08:26.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:26 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:08:26.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:26 smithi115 ceph-mon[72476]: osdmap e392: 12 total, 11 up, 9 in 2024-03-20T09:08:26.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:26 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:08:26.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:26 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:08:26.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:26 smithi122 ceph-mon[73495]: osdmap e392: 12 total, 11 up, 9 in 2024-03-20T09:08:26.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:26 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:08:26.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:26 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:08:26.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:26 smithi026 ceph-mon[68309]: osdmap e392: 12 total, 11 up, 9 in 2024-03-20T09:08:27.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:27 smithi115 ceph-mon[72476]: pgmap v947: 57 pgs: 1 active+recovering+remapped, 1 activating+undersized+degraded+remapped, 1 activating+remapped, 11 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.4 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 20120/216183 objects degraded (9.307%); 32246/216183 objects misplaced (14.916%); 30 MiB/s, 494 objects/s recovering 2024-03-20T09:08:27.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:27 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 20120/216183 objects degraded (9.307%), 7 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-20T09:08:27.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:27 smithi115 ceph-mon[72476]: osdmap e393: 12 total, 11 up, 9 in 2024-03-20T09:08:27.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:27 smithi122 ceph-mon[73495]: pgmap v947: 57 pgs: 1 active+recovering+remapped, 1 activating+undersized+degraded+remapped, 1 activating+remapped, 11 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.4 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 20120/216183 objects degraded (9.307%); 32246/216183 objects misplaced (14.916%); 30 MiB/s, 494 objects/s recovering 2024-03-20T09:08:27.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:27 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 20120/216183 objects degraded (9.307%), 7 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-20T09:08:27.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:27 smithi122 ceph-mon[73495]: osdmap e393: 12 total, 11 up, 9 in 2024-03-20T09:08:27.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:27 smithi026 ceph-mon[68309]: pgmap v947: 57 pgs: 1 active+recovering+remapped, 1 activating+undersized+degraded+remapped, 1 activating+remapped, 11 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.4 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 20120/216183 objects degraded (9.307%); 32246/216183 objects misplaced (14.916%); 30 MiB/s, 494 objects/s recovering 2024-03-20T09:08:27.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:27 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 20120/216183 objects degraded (9.307%), 7 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-20T09:08:27.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:27 smithi026 ceph-mon[68309]: osdmap e393: 12 total, 11 up, 9 in 2024-03-20T09:08:29.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:29 smithi115 ceph-mon[72476]: pgmap v949: 57 pgs: 1 active+recovering+remapped, 1 activating+undersized+degraded+remapped, 1 activating+remapped, 11 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.4 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 20120/216183 objects degraded (9.307%); 32246/216183 objects misplaced (14.916%); 18 MiB/s, 292 objects/s recovering 2024-03-20T09:08:29.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:29 smithi115 ceph-mon[72476]: osdmap e394: 12 total, 11 up, 9 in 2024-03-20T09:08:29.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:29 smithi122 ceph-mon[73495]: pgmap v949: 57 pgs: 1 active+recovering+remapped, 1 activating+undersized+degraded+remapped, 1 activating+remapped, 11 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.4 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 20120/216183 objects degraded (9.307%); 32246/216183 objects misplaced (14.916%); 18 MiB/s, 292 objects/s recovering 2024-03-20T09:08:29.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:29 smithi122 ceph-mon[73495]: osdmap e394: 12 total, 11 up, 9 in 2024-03-20T09:08:29.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:29 smithi026 ceph-mon[68309]: pgmap v949: 57 pgs: 1 active+recovering+remapped, 1 activating+undersized+degraded+remapped, 1 activating+remapped, 11 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.4 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 20120/216183 objects degraded (9.307%); 32246/216183 objects misplaced (14.916%); 18 MiB/s, 292 objects/s recovering 2024-03-20T09:08:29.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:29 smithi026 ceph-mon[68309]: osdmap e394: 12 total, 11 up, 9 in 2024-03-20T09:08:29.952 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:08:30.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:30 smithi115 ceph-mon[72476]: osdmap e395: 12 total, 11 up, 9 in 2024-03-20T09:08:30.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:30 smithi122 ceph-mon[73495]: osdmap e395: 12 total, 11 up, 9 in 2024-03-20T09:08:30.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:30 smithi026 ceph-mon[68309]: osdmap e395: 12 total, 11 up, 9 in 2024-03-20T09:08:31.132 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:31 smithi122 ceph-mon[73495]: pgmap v952: 57 pgs: 1 active+recovering+remapped, 1 activating+undersized+degraded+remapped, 1 activating+remapped, 11 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.4 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 20010/216183 objects degraded (9.256%); 32136/216183 objects misplaced (14.865%); 23 MiB/s, 379 objects/s recovering 2024-03-20T09:08:31.133 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:31 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:08:31.133 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:31 smithi115 ceph-mon[72476]: pgmap v952: 57 pgs: 1 active+recovering+remapped, 1 activating+undersized+degraded+remapped, 1 activating+remapped, 11 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.4 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 20010/216183 objects degraded (9.256%); 32136/216183 objects misplaced (14.865%); 23 MiB/s, 379 objects/s recovering 2024-03-20T09:08:31.133 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:31 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:08:31.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:31 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:08:31.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:31 smithi115 ceph-mon[72476]: osdmap e396: 12 total, 11 up, 9 in 2024-03-20T09:08:31.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:31 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:08:31.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:31 smithi122 ceph-mon[73495]: osdmap e396: 12 total, 11 up, 9 in 2024-03-20T09:08:31.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:31 smithi026 ceph-mon[68309]: pgmap v952: 57 pgs: 1 active+recovering+remapped, 1 activating+undersized+degraded+remapped, 1 activating+remapped, 11 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.4 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 20010/216183 objects degraded (9.256%); 32136/216183 objects misplaced (14.865%); 23 MiB/s, 379 objects/s recovering 2024-03-20T09:08:31.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:31 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:08:31.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:31 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:08:31.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:31 smithi026 ceph-mon[68309]: osdmap e396: 12 total, 11 up, 9 in 2024-03-20T09:08:32.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:32 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 20010/216183 objects degraded (9.256%), 7 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-20T09:08:32.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:32 smithi115 ceph-mon[72476]: osdmap e397: 12 total, 11 up, 9 in 2024-03-20T09:08:32.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:32 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 20010/216183 objects degraded (9.256%), 7 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-20T09:08:32.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:32 smithi122 ceph-mon[73495]: osdmap e397: 12 total, 11 up, 9 in 2024-03-20T09:08:32.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:32 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 20010/216183 objects degraded (9.256%), 7 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-20T09:08:32.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:32 smithi026 ceph-mon[68309]: osdmap e397: 12 total, 11 up, 9 in 2024-03-20T09:08:33.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:33 smithi115 ceph-mon[72476]: pgmap v954: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 1 activating+remapped, 12 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.4 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 17766/216183 objects degraded (8.218%); 32324/216183 objects misplaced (14.952%); 22 MiB/s, 520 objects/s recovering 2024-03-20T09:08:33.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:33 smithi122 ceph-mon[73495]: pgmap v954: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 1 activating+remapped, 12 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.4 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 17766/216183 objects degraded (8.218%); 32324/216183 objects misplaced (14.952%); 22 MiB/s, 520 objects/s recovering 2024-03-20T09:08:33.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:33 smithi026 ceph-mon[68309]: pgmap v954: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 1 activating+remapped, 12 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.4 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 17766/216183 objects degraded (8.218%); 32324/216183 objects misplaced (14.952%); 22 MiB/s, 520 objects/s recovering 2024-03-20T09:08:35.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:35 smithi115 ceph-mon[72476]: pgmap v956: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 1 activating+remapped, 12 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.4 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 6 op/s; 17749/216081 objects degraded (8.214%); 32324/216081 objects misplaced (14.959%); 18 MiB/s, 439 objects/s recovering 2024-03-20T09:08:35.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:35 smithi122 ceph-mon[73495]: pgmap v956: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 1 activating+remapped, 12 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.4 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 6 op/s; 17749/216081 objects degraded (8.214%); 32324/216081 objects misplaced (14.959%); 18 MiB/s, 439 objects/s recovering 2024-03-20T09:08:35.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:35 smithi026 ceph-mon[68309]: pgmap v956: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 1 activating+remapped, 12 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.4 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 6 op/s; 17749/216081 objects degraded (8.214%); 32324/216081 objects misplaced (14.959%); 18 MiB/s, 439 objects/s recovering 2024-03-20T09:08:37.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:37 smithi115 ceph-mon[72476]: pgmap v957: 57 pgs: 1 active+recovering+remapped, 13 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.4 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 119 op/s; 9364/213807 objects degraded (4.380%); 36527/213807 objects misplaced (17.084%); 25 MiB/s, 531 objects/s recovering 2024-03-20T09:08:37.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:37 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 9364/213807 objects degraded (4.380%), 6 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-20T09:08:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:37 smithi122 ceph-mon[73495]: pgmap v957: 57 pgs: 1 active+recovering+remapped, 13 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.4 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 119 op/s; 9364/213807 objects degraded (4.380%); 36527/213807 objects misplaced (17.084%); 25 MiB/s, 531 objects/s recovering 2024-03-20T09:08:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:37 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 9364/213807 objects degraded (4.380%), 6 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-20T09:08:37.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:37 smithi026 ceph-mon[68309]: pgmap v957: 57 pgs: 1 active+recovering+remapped, 13 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.4 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 119 op/s; 9364/213807 objects degraded (4.380%); 36527/213807 objects misplaced (17.084%); 25 MiB/s, 531 objects/s recovering 2024-03-20T09:08:37.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:37 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 9364/213807 objects degraded (4.380%), 6 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-20T09:08:38.401 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-20T09:08:39.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:39 smithi115 ceph-mon[72476]: pgmap v958: 57 pgs: 1 active+recovering+remapped, 13 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.3 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 111 op/s; 9364/213501 objects degraded (4.386%); 36527/213501 objects misplaced (17.109%); 20 MiB/s, 425 objects/s recovering 2024-03-20T09:08:39.416 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:08:39.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:39.423 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:39 smithi026 ceph-mon[68309]: pgmap v958: 57 pgs: 1 active+recovering+remapped, 13 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.3 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 111 op/s; 9364/213501 objects degraded (4.386%); 36527/213501 objects misplaced (17.109%); 20 MiB/s, 425 objects/s recovering 2024-03-20T09:08:39.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:08:39.431 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:08:39.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:39 smithi122 ceph-mon[73495]: pgmap v958: 57 pgs: 1 active+recovering+remapped, 13 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.3 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 111 op/s; 9364/213501 objects degraded (4.386%); 36527/213501 objects misplaced (17.109%); 20 MiB/s, 425 objects/s recovering 2024-03-20T09:08:39.524 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:08:39.571 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:39.599 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:08:39.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:39.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:08:39.704 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:08:39.732 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:08:39.732 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:08:39.732 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 397, 2024-03-20T09:08:39.732 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "852fbb68-e696-11ee-95c9-87774f69a715", 2024-03-20T09:08:39.733 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-03-20T08:48:35.281088+0000", 2024-03-20T09:08:39.733 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-03-20T09:08:31.958601+0000", 2024-03-20T09:08:39.733 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2024-03-20T09:04:33.695633+0000", 2024-03-20T09:08:39.733 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2024-03-20T09:07:48.551904+0000", 2024-03-20T09:08:39.733 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-20T09:08:39.733 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5806080, 2024-03-20T09:08:39.733 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2024-03-20T09:08:39.733 INFO:teuthology.orchestra.run.smithi026.stdout: "nodeep-scrub", 2024-03-20T09:08:39.733 INFO:teuthology.orchestra.run.smithi026.stdout: "noscrub", 2024-03-20T09:08:39.733 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2024-03-20T09:08:39.733 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2024-03-20T09:08:39.734 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2024-03-20T09:08:39.734 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2024-03-20T09:08:39.734 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:08:39.734 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 26, 2024-03-20T09:08:39.734 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2024-03-20T09:08:39.734 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-20T09:08:39.734 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-20T09:08:39.734 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2024-03-20T09:08:39.734 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 3, 2024-03-20T09:08:39.734 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 12, 2024-03-20T09:08:39.734 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "luminous", 2024-03-20T09:08:39.734 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "jewel", 2024-03-20T09:08:39.735 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "squid", 2024-03-20T09:08:39.735 INFO:teuthology.orchestra.run.smithi026.stdout: "allow_crimson": false, 2024-03-20T09:08:39.735 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2024-03-20T09:08:39.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.735 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2024-03-20T09:08:39.735 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": ".mgr", 2024-03-20T09:08:39.735 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-03-20T08:52:17.442324+0000", 2024-03-20T09:08:39.735 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-03-20T09:08:39.735 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-03-20T09:08:39.735 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-03-20T09:08:39.735 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 3, 2024-03-20T09:08:39.735 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 2, 2024-03-20T09:08:39.735 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-03-20T09:08:39.736 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-03-20T09:08:39.736 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-03-20T09:08:39.736 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-03-20T09:08:39.736 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-20T09:08:39.736 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-03-20T09:08:39.736 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-03-20T09:08:39.736 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 31, 2024-03-20T09:08:39.736 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 31, 2024-03-20T09:08:39.736 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 31, 2024-03-20T09:08:39.736 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 31, 2024-03-20T09:08:39.736 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 31, 2024-03-20T09:08:39.737 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-03-20T09:08:39.737 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-03-20T09:08:39.737 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-03-20T09:08:39.737 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-03-20T09:08:39.737 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-03-20T09:08:39.737 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-03-20T09:08:39.737 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-03-20T09:08:39.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.737 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "360", 2024-03-20T09:08:39.737 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-03-20T09:08:39.737 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-20T09:08:39.738 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "358", 2024-03-20T09:08:39.738 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-03-20T09:08:39.738 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-03-20T09:08:39.738 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-03-20T09:08:39.738 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-03-20T09:08:39.738 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-03-20T09:08:39.738 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-03-20T09:08:39.738 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-03-20T09:08:39.738 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-03-20T09:08:39.738 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-03-20T09:08:39.738 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-03-20T09:08:39.739 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-03-20T09:08:39.739 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-03-20T09:08:39.739 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-03-20T09:08:39.739 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-03-20T09:08:39.739 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-03-20T09:08:39.739 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-20T09:08:39.739 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-20T09:08:39.739 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-20T09:08:39.739 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-03-20T09:08:39.739 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-03-20T09:08:39.739 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-03-20T09:08:39.739 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-03-20T09:08:39.740 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-03-20T09:08:39.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.740 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-03-20T09:08:39.740 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-03-20T09:08:39.740 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-03-20T09:08:39.740 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-03-20T09:08:39.740 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-03-20T09:08:39.740 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-03-20T09:08:39.740 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-03-20T09:08:39.741 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-03-20T09:08:39.741 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-03-20T09:08:39.741 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-03-20T09:08:39.741 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-03-20T09:08:39.741 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2024-03-20T09:08:39.741 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_max": 32, 2024-03-20T09:08:39.741 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2024-03-20T09:08:39.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.741 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-03-20T09:08:39.741 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": {} 2024-03-20T09:08:39.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.742 INFO:teuthology.orchestra.run.smithi026.stdout: "read_balance": { 2024-03-20T09:08:39.742 INFO:teuthology.orchestra.run.smithi026.stdout: "score_type": "Fair distribution", 2024-03-20T09:08:39.742 INFO:teuthology.orchestra.run.smithi026.stdout: "score_acting": 2.7000000476837158, 2024-03-20T09:08:39.742 INFO:teuthology.orchestra.run.smithi026.stdout: "score_stable": 2.7000000476837158, 2024-03-20T09:08:39.742 INFO:teuthology.orchestra.run.smithi026.stdout: "optimal_score": 0.67000001668930054, 2024-03-20T09:08:39.742 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_acting": 1.809999942779541, 2024-03-20T09:08:39.742 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_stable": 1.809999942779541, 2024-03-20T09:08:39.742 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity_weighted": 0.92150616645812988, 2024-03-20T09:08:39.742 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity": 0.92000001668930054, 2024-03-20T09:08:39.742 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-03-20T09:08:39.742 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.743 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 3, 2024-03-20T09:08:39.743 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "unique_pool_1", 2024-03-20T09:08:39.743 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-03-20T09:02:48.375175+0000", 2024-03-20T09:08:39.743 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-03-20T09:08:39.743 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-03-20T09:08:39.743 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-03-20T09:08:39.743 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 3, 2024-03-20T09:08:39.743 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 2, 2024-03-20T09:08:39.743 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-03-20T09:08:39.744 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-03-20T09:08:39.744 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-03-20T09:08:39.744 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-03-20T09:08:39.744 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-20T09:08:39.744 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-03-20T09:08:39.744 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-03-20T09:08:39.744 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 26, 2024-03-20T09:08:39.744 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 26, 2024-03-20T09:08:39.744 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 26, 2024-03-20T09:08:39.744 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 26, 2024-03-20T09:08:39.744 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 26, 2024-03-20T09:08:39.745 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-03-20T09:08:39.745 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-03-20T09:08:39.745 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-03-20T09:08:39.745 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-03-20T09:08:39.745 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-03-20T09:08:39.745 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-03-20T09:08:39.745 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-03-20T09:08:39.745 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.745 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "233", 2024-03-20T09:08:39.745 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-03-20T09:08:39.745 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-20T09:08:39.745 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "231", 2024-03-20T09:08:39.746 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-03-20T09:08:39.746 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-03-20T09:08:39.746 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-03-20T09:08:39.746 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-03-20T09:08:39.746 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-03-20T09:08:39.746 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-03-20T09:08:39.746 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-03-20T09:08:39.746 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-03-20T09:08:39.746 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-03-20T09:08:39.746 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-03-20T09:08:39.746 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-03-20T09:08:39.747 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-03-20T09:08:39.747 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-03-20T09:08:39.747 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-03-20T09:08:39.747 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-03-20T09:08:39.747 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-20T09:08:39.747 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-20T09:08:39.747 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-20T09:08:39.747 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-03-20T09:08:39.747 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-03-20T09:08:39.747 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-03-20T09:08:39.747 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-03-20T09:08:39.747 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-03-20T09:08:39.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.748 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-03-20T09:08:39.748 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-03-20T09:08:39.748 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-03-20T09:08:39.748 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-03-20T09:08:39.748 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-03-20T09:08:39.748 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-03-20T09:08:39.748 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-03-20T09:08:39.748 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-03-20T09:08:39.748 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-03-20T09:08:39.748 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-03-20T09:08:39.748 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-03-20T09:08:39.748 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2024-03-20T09:08:39.749 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-03-20T09:08:39.749 INFO:teuthology.orchestra.run.smithi026.stdout: "rados": {} 2024-03-20T09:08:39.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.749 INFO:teuthology.orchestra.run.smithi026.stdout: "read_balance": { 2024-03-20T09:08:39.749 INFO:teuthology.orchestra.run.smithi026.stdout: "score_type": "Fair distribution", 2024-03-20T09:08:39.749 INFO:teuthology.orchestra.run.smithi026.stdout: "score_acting": 3.2100000381469727, 2024-03-20T09:08:39.749 INFO:teuthology.orchestra.run.smithi026.stdout: "score_stable": 3.2100000381469727, 2024-03-20T09:08:39.749 INFO:teuthology.orchestra.run.smithi026.stdout: "optimal_score": 0.67000001668930054, 2024-03-20T09:08:39.749 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_acting": 2.1500000953674316, 2024-03-20T09:08:39.749 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_stable": 2.1500000953674316, 2024-03-20T09:08:39.749 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity_weighted": 0.92150616645812988, 2024-03-20T09:08:39.749 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity": 0.92000001668930054, 2024-03-20T09:08:39.750 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-03-20T09:08:39.750 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.750 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.750 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:08:39.750 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:08:39.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.750 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-03-20T09:08:39.750 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "ad3fd9c0-316f-4fe5-a63b-d480087c1729", 2024-03-20T09:08:39.750 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:08:39.750 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:08:39.750 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:08:39.750 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:08:39.750 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:08:39.751 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:08:39.754 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 8, 2024-03-20T09:08:39.754 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 385, 2024-03-20T09:08:39.754 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:08:39.754 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:08:39.755 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:08:39.755 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.755 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.755 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2024-03-20T09:08:39.755 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:08:39.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.755 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.755 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2024-03-20T09:08:39.755 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:08:39.755 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.755 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.756 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:08:39.756 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.756 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.756 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2024-03-20T09:08:39.756 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:08:39.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.756 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.756 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2024-03-20T09:08:39.757 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:08:39.757 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.757 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.757 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:08:39.757 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.757 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.757 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.757 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2024-03-20T09:08:39.757 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:08:39.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.757 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.758 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.758 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2024-03-20T09:08:39.758 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:08:39.758 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.758 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.758 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:08:39.758 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.758 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.758 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2024-03-20T09:08:39.758 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:08:39.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.759 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.759 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2024-03-20T09:08:39.759 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:08:39.759 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.759 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.759 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/2139805293", 2024-03-20T09:08:39.759 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/2139805293", 2024-03-20T09:08:39.759 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/2139805293", 2024-03-20T09:08:39.759 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/2139805293", 2024-03-20T09:08:39.760 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:08:39.760 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:08:39.760 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:08:39.760 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.760 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.760 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.760 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-03-20T09:08:39.760 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e85b94f5-2954-47a1-9c20-096f2606eb04", 2024-03-20T09:08:39.760 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:08:39.760 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:08:39.760 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0.97100830078125, 2024-03-20T09:08:39.760 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:08:39.761 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:08:39.761 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:08:39.761 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2024-03-20T09:08:39.761 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 386, 2024-03-20T09:08:39.761 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:08:39.761 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:08:39.761 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:08:39.761 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.761 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.761 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2024-03-20T09:08:39.761 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:08:39.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.762 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.762 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2024-03-20T09:08:39.762 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:08:39.762 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.762 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.762 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:08:39.762 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.762 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.762 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2024-03-20T09:08:39.763 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:08:39.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.763 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.763 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2024-03-20T09:08:39.763 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:08:39.763 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.763 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.763 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:08:39.763 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.764 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.764 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2024-03-20T09:08:39.764 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:08:39.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.764 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.764 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2024-03-20T09:08:39.764 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:08:39.764 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.764 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.764 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:08:39.765 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.765 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.765 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2024-03-20T09:08:39.765 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:08:39.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.765 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.765 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2024-03-20T09:08:39.765 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:08:39.765 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.765 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.766 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/3853355134", 2024-03-20T09:08:39.766 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/3853355134", 2024-03-20T09:08:39.766 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/3853355134", 2024-03-20T09:08:39.766 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/3853355134", 2024-03-20T09:08:39.766 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:08:39.766 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:08:39.766 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:08:39.766 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.766 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-03-20T09:08:39.766 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "dd2b10f0-741c-47a4-b224-ac6bfe2c14ef", 2024-03-20T09:08:39.767 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:08:39.767 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:08:39.767 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:08:39.767 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:08:39.767 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:08:39.767 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:08:39.767 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 18, 2024-03-20T09:08:39.767 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 383, 2024-03-20T09:08:39.767 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:08:39.767 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:08:39.767 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:08:39.767 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.768 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.768 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2024-03-20T09:08:39.768 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:08:39.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.768 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.768 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2024-03-20T09:08:39.768 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:08:39.768 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.768 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.769 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:08:39.769 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.769 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.769 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2024-03-20T09:08:39.769 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:08:39.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.769 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.769 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2024-03-20T09:08:39.769 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:08:39.770 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.770 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.770 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:08:39.770 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.770 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.770 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2024-03-20T09:08:39.770 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:08:39.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.770 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.771 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2024-03-20T09:08:39.771 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:08:39.771 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.771 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.771 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:08:39.771 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.771 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.771 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2024-03-20T09:08:39.771 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:08:39.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.772 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.772 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2024-03-20T09:08:39.772 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:08:39.772 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.772 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.772 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/3993357967", 2024-03-20T09:08:39.772 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/3993357967", 2024-03-20T09:08:39.772 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/3993357967", 2024-03-20T09:08:39.772 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/3993357967", 2024-03-20T09:08:39.773 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:08:39.773 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:08:39.773 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:08:39.773 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.773 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.773 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-03-20T09:08:39.773 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1", 2024-03-20T09:08:39.773 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:08:39.773 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 0, 2024-03-20T09:08:39.773 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-03-20T09:08:39.774 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:08:39.774 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:08:39.774 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:08:39.774 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 26, 2024-03-20T09:08:39.774 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 274, 2024-03-20T09:08:39.774 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:08:39.774 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:08:39.774 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:08:39.774 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.774 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.774 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2024-03-20T09:08:39.775 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:08:39.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.775 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.775 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2024-03-20T09:08:39.775 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:08:39.775 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.775 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.775 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:08:39.775 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.776 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.776 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2024-03-20T09:08:39.776 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:08:39.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.776 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.776 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2024-03-20T09:08:39.776 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:08:39.776 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.776 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.777 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:08:39.777 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.777 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.777 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2024-03-20T09:08:39.777 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:08:39.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.777 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.777 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2024-03-20T09:08:39.778 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:08:39.778 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.778 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.778 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:08:39.778 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.778 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.778 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2024-03-20T09:08:39.778 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:08:39.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.779 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.779 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2024-03-20T09:08:39.779 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:08:39.779 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.779 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.779 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/4110225509", 2024-03-20T09:08:39.779 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/4110225509", 2024-03-20T09:08:39.779 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/4110225509", 2024-03-20T09:08:39.779 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/4110225509", 2024-03-20T09:08:39.779 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:08:39.779 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:08:39.779 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:08:39.780 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.780 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.780 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-03-20T09:08:39.780 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "f14e34d5-9829-42a3-9b25-31a0976c5ef7", 2024-03-20T09:08:39.780 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:08:39.780 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 0, 2024-03-20T09:08:39.780 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-03-20T09:08:39.780 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:08:39.780 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:08:39.780 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:08:39.780 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 32, 2024-03-20T09:08:39.780 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 366, 2024-03-20T09:08:39.781 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:08:39.781 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:08:39.781 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:08:39.781 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.781 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.781 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6800", 2024-03-20T09:08:39.781 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:08:39.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.781 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.781 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6801", 2024-03-20T09:08:39.782 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:08:39.782 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.782 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.782 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:08:39.782 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.782 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.782 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6802", 2024-03-20T09:08:39.782 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:08:39.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.783 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.783 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6803", 2024-03-20T09:08:39.783 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:08:39.783 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.783 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.783 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:08:39.783 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.783 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.784 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6806", 2024-03-20T09:08:39.784 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:08:39.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.784 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.784 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6807", 2024-03-20T09:08:39.784 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:08:39.784 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.784 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.784 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:08:39.784 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.785 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.785 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6804", 2024-03-20T09:08:39.785 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:08:39.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.785 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.785 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6805", 2024-03-20T09:08:39.785 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:08:39.785 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.785 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.785 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6801/477923058", 2024-03-20T09:08:39.786 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6803/477923058", 2024-03-20T09:08:39.786 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6807/477923058", 2024-03-20T09:08:39.786 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6805/477923058", 2024-03-20T09:08:39.786 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:08:39.786 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:08:39.786 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:08:39.786 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.786 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-03-20T09:08:39.786 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "cb431b6a-540d-4318-ae35-d379e1df0185", 2024-03-20T09:08:39.786 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 0, 2024-03-20T09:08:39.787 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:08:39.787 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:08:39.787 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:08:39.787 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:08:39.787 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:08:39.787 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 41, 2024-03-20T09:08:39.787 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 265, 2024-03-20T09:08:39.787 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 274, 2024-03-20T09:08:39.787 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:08:39.787 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:08:39.787 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.788 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.788 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6808", 2024-03-20T09:08:39.788 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:08:39.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.788 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.788 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6809", 2024-03-20T09:08:39.788 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:08:39.788 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.788 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.788 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:08:39.789 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.789 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.789 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6810", 2024-03-20T09:08:39.789 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:08:39.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.789 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.789 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6811", 2024-03-20T09:08:39.789 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:08:39.789 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.789 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.790 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:08:39.790 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.790 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.790 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6814", 2024-03-20T09:08:39.790 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:08:39.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.790 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.790 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6815", 2024-03-20T09:08:39.791 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:08:39.791 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.791 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.791 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:08:39.791 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.791 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.791 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6812", 2024-03-20T09:08:39.791 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:08:39.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.791 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.792 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6813", 2024-03-20T09:08:39.792 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:08:39.792 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.792 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.792 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6809/1339268003", 2024-03-20T09:08:39.792 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6811/1339268003", 2024-03-20T09:08:39.792 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6815/1339268003", 2024-03-20T09:08:39.792 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6813/1339268003", 2024-03-20T09:08:39.792 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:08:39.792 INFO:teuthology.orchestra.run.smithi026.stdout: "exists" 2024-03-20T09:08:39.792 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.793 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2024-03-20T09:08:39.793 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "753ed830-e764-4ff3-9d9b-87e7fa748b81", 2024-03-20T09:08:39.793 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:08:39.793 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:08:39.793 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0.842376708984375, 2024-03-20T09:08:39.793 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:08:39.793 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:08:39.793 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:08:39.793 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 51, 2024-03-20T09:08:39.793 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 394, 2024-03-20T09:08:39.794 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:08:39.794 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:08:39.794 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:08:39.794 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.794 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.794 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6816", 2024-03-20T09:08:39.794 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:08:39.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.794 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.795 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6817", 2024-03-20T09:08:39.795 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:08:39.795 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.795 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.795 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:08:39.795 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.795 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.795 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6818", 2024-03-20T09:08:39.795 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:08:39.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.796 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.796 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6819", 2024-03-20T09:08:39.796 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:08:39.796 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.796 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.796 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:08:39.796 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.796 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.796 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6822", 2024-03-20T09:08:39.797 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:08:39.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.797 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.797 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6823", 2024-03-20T09:08:39.797 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:08:39.797 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.797 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.797 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:08:39.797 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.798 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.798 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6820", 2024-03-20T09:08:39.798 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:08:39.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.798 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.798 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6821", 2024-03-20T09:08:39.798 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:08:39.798 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.798 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.799 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6817/3278301843", 2024-03-20T09:08:39.799 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6819/3278301843", 2024-03-20T09:08:39.799 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6823/3278301843", 2024-03-20T09:08:39.799 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6821/3278301843", 2024-03-20T09:08:39.799 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:08:39.799 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:08:39.799 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:08:39.799 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.799 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2024-03-20T09:08:39.799 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "55bcbd37-7a6f-4ab1-b5a9-91a951444d86", 2024-03-20T09:08:39.799 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:08:39.800 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:08:39.800 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0.4264678955078125, 2024-03-20T09:08:39.800 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 0.058074951171875, 2024-03-20T09:08:39.800 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:08:39.800 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:08:39.800 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 60, 2024-03-20T09:08:39.800 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 274, 2024-03-20T09:08:39.800 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:08:39.800 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:08:39.800 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:08:39.800 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.800 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.801 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6824", 2024-03-20T09:08:39.801 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:08:39.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.801 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.801 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6825", 2024-03-20T09:08:39.801 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:08:39.801 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.801 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.801 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:08:39.801 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.801 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.802 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6826", 2024-03-20T09:08:39.802 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:08:39.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.802 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.802 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6827", 2024-03-20T09:08:39.802 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:08:39.802 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.802 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.802 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:08:39.802 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.803 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.803 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6830", 2024-03-20T09:08:39.803 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:08:39.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.803 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.803 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6831", 2024-03-20T09:08:39.803 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:08:39.803 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.803 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.804 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:08:39.804 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.804 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.804 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6828", 2024-03-20T09:08:39.804 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:08:39.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.804 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.804 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6829", 2024-03-20T09:08:39.804 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:08:39.805 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.805 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.805 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6825/3152424001", 2024-03-20T09:08:39.805 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6827/3152424001", 2024-03-20T09:08:39.805 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6831/3152424001", 2024-03-20T09:08:39.805 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6829/3152424001", 2024-03-20T09:08:39.805 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:08:39.805 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:08:39.805 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:08:39.805 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.806 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 8, 2024-03-20T09:08:39.806 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "bc86248e-6bd0-4369-b8a2-e593bc2a610d", 2024-03-20T09:08:39.806 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:08:39.806 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:08:39.806 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:08:39.806 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:08:39.806 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:08:39.806 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:08:39.806 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 70, 2024-03-20T09:08:39.807 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 380, 2024-03-20T09:08:39.807 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:08:39.807 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:08:39.807 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:08:39.807 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.807 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.807 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6800", 2024-03-20T09:08:39.807 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:08:39.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.808 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.808 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6801", 2024-03-20T09:08:39.808 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:08:39.808 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.808 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.808 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:08:39.808 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.808 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.809 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6802", 2024-03-20T09:08:39.809 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:08:39.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.809 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.809 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6803", 2024-03-20T09:08:39.809 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:08:39.809 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.809 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.809 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:08:39.810 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.810 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.810 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6806", 2024-03-20T09:08:39.810 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:08:39.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.810 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.810 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6807", 2024-03-20T09:08:39.810 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:08:39.810 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.811 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.811 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:08:39.811 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.811 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.811 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6804", 2024-03-20T09:08:39.811 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:08:39.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.811 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.811 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6805", 2024-03-20T09:08:39.811 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:08:39.812 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.812 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.812 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6801/1309426740", 2024-03-20T09:08:39.812 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6803/1309426740", 2024-03-20T09:08:39.812 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6807/1309426740", 2024-03-20T09:08:39.812 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6805/1309426740", 2024-03-20T09:08:39.812 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:08:39.812 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:08:39.812 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:08:39.812 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.812 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 9, 2024-03-20T09:08:39.813 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b20a4aa-032a-4c09-ac75-153c554424e7", 2024-03-20T09:08:39.813 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:08:39.813 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 0, 2024-03-20T09:08:39.813 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-03-20T09:08:39.813 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:08:39.813 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:08:39.813 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:08:39.813 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 80, 2024-03-20T09:08:39.813 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 257, 2024-03-20T09:08:39.813 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:08:39.813 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:08:39.813 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:08:39.814 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.814 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.814 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6808", 2024-03-20T09:08:39.814 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:08:39.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.814 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.814 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6809", 2024-03-20T09:08:39.814 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:08:39.814 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.815 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.815 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:08:39.815 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.815 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.815 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6810", 2024-03-20T09:08:39.815 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:08:39.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.815 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.815 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6811", 2024-03-20T09:08:39.816 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:08:39.816 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.816 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.816 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:08:39.816 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.816 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.816 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6814", 2024-03-20T09:08:39.816 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:08:39.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.816 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.817 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6815", 2024-03-20T09:08:39.817 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:08:39.817 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.817 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.817 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:08:39.817 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.817 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.817 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6812", 2024-03-20T09:08:39.817 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:08:39.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.818 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.818 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6813", 2024-03-20T09:08:39.818 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:08:39.818 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.818 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.818 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6809/2953210076", 2024-03-20T09:08:39.818 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6811/2953210076", 2024-03-20T09:08:39.818 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6815/2953210076", 2024-03-20T09:08:39.819 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6813/2953210076", 2024-03-20T09:08:39.819 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:08:39.819 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:08:39.819 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:08:39.819 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.819 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 10, 2024-03-20T09:08:39.819 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e62c1fb3-5dd1-4d0d-997c-d20633d19361", 2024-03-20T09:08:39.819 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:08:39.819 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:08:39.820 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:08:39.820 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:08:39.820 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:08:39.820 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:08:39.820 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 89, 2024-03-20T09:08:39.820 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 386, 2024-03-20T09:08:39.820 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:08:39.820 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:08:39.820 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:08:39.820 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.820 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.821 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6816", 2024-03-20T09:08:39.821 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:08:39.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.821 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.821 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6817", 2024-03-20T09:08:39.821 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:08:39.821 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.821 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.821 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:08:39.821 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.822 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.822 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6818", 2024-03-20T09:08:39.822 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:08:39.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.822 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.822 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6819", 2024-03-20T09:08:39.822 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:08:39.822 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.822 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.823 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:08:39.823 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.823 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.823 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6822", 2024-03-20T09:08:39.823 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:08:39.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.823 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.823 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6823", 2024-03-20T09:08:39.823 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:08:39.823 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.824 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.824 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:08:39.824 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.824 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.824 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6820", 2024-03-20T09:08:39.824 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:08:39.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.824 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.825 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6821", 2024-03-20T09:08:39.825 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:08:39.825 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.825 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.825 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6817/409394340", 2024-03-20T09:08:39.825 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6819/409394340", 2024-03-20T09:08:39.825 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6823/409394340", 2024-03-20T09:08:39.825 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6821/409394340", 2024-03-20T09:08:39.825 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:08:39.825 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:08:39.825 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:08:39.825 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.826 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 11, 2024-03-20T09:08:39.826 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b83994d5-e5ef-48b0-9758-c26843cec3b6", 2024-03-20T09:08:39.826 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:08:39.826 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:08:39.826 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0.6123046875, 2024-03-20T09:08:39.826 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:08:39.826 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:08:39.826 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:08:39.826 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 96, 2024-03-20T09:08:39.827 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 385, 2024-03-20T09:08:39.827 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:08:39.827 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:08:39.827 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:08:39.827 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.827 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.827 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6824", 2024-03-20T09:08:39.827 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:08:39.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.827 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.827 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6825", 2024-03-20T09:08:39.828 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:08:39.828 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.828 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.828 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:08:39.828 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.828 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.828 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6826", 2024-03-20T09:08:39.828 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:08:39.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.829 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.829 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6827", 2024-03-20T09:08:39.829 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:08:39.829 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.829 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.829 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:08:39.829 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.829 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.829 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6830", 2024-03-20T09:08:39.830 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:08:39.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.830 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.830 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6831", 2024-03-20T09:08:39.830 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:08:39.830 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.830 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.830 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:08:39.830 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:08:39.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.831 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:08:39.831 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6828", 2024-03-20T09:08:39.831 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:08:39.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.831 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:08:39.831 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6829", 2024-03-20T09:08:39.831 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:08:39.831 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.831 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.832 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6825/2704069354", 2024-03-20T09:08:39.832 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6827/2704069354", 2024-03-20T09:08:39.832 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6831/2704069354", 2024-03-20T09:08:39.832 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6829/2704069354", 2024-03-20T09:08:39.832 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:08:39.832 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:08:39.832 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:08:39.832 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.832 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.832 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:08:39.832 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2024-03-20T09:08:39.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.833 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-03-20T09:08:39.833 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:08:39.833 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:08:39.833 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:08:39.833 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:08:39.833 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:08:39.833 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:08:04.860551+0000", 2024-03-20T09:08:39.833 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:08:39.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.833 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-03-20T09:08:39.833 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:08:39.834 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:08:39.834 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:08:39.834 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:08:39.834 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:08:39.834 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:07:33.401607+0000", 2024-03-20T09:08:39.834 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:08:39.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.834 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-03-20T09:08:39.834 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:08:39.834 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:08:39.835 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:08:39.835 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:08:39.835 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:08:39.835 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:08:18.494972+0000", 2024-03-20T09:08:39.835 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:08:39.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.835 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-03-20T09:08:39.835 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:08:39.835 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:08:39.835 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:08:39.835 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:08:39.836 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 35063, 2024-03-20T09:08:39.836 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:08:00.063524+0000", 2024-03-20T09:08:39.836 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:08:39.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.836 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-03-20T09:08:39.836 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:08:39.836 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:08:39.836 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:08:39.836 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:08:39.836 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 65536, 2024-03-20T09:08:39.836 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:08:30.077943+0000", 2024-03-20T09:08:39.837 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:08:39.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.837 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-03-20T09:08:39.837 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:08:39.837 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:08:39.837 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:08:39.837 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:08:39.837 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:08:39.837 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:03:53.609882+0000", 2024-03-20T09:08:39.837 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 273 2024-03-20T09:08:39.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.838 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2024-03-20T09:08:39.838 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:08:39.838 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:08:39.838 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:08:39.838 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:08:39.838 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:08:39.838 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:08:14.465561+0000", 2024-03-20T09:08:39.838 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:08:39.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.838 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2024-03-20T09:08:39.839 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:08:39.839 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:08:39.839 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:08:39.839 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:08:39.839 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:08:39.839 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:07:57.050958+0000", 2024-03-20T09:08:39.839 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:08:39.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.839 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 8, 2024-03-20T09:08:39.839 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:08:39.839 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:08:39.839 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:08:39.840 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:08:39.840 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:08:39.840 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:08:24.980388+0000", 2024-03-20T09:08:39.840 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:08:39.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.840 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 9, 2024-03-20T09:08:39.840 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:08:39.840 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:08:39.840 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:08:39.840 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:08:39.840 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 34736, 2024-03-20T09:08:39.841 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:07:52.842622+0000", 2024-03-20T09:08:39.841 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:08:39.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.841 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 10, 2024-03-20T09:08:39.841 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:08:39.841 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:08:39.841 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:08:39.841 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:08:39.841 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:08:39.841 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:08:19.771648+0000", 2024-03-20T09:08:39.841 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:08:39.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.842 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 11, 2024-03-20T09:08:39.842 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:08:39.842 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:08:39.842 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:08:39.842 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:08:39.842 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:08:39.842 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:07:26.222744+0000", 2024-03-20T09:08:39.842 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:08:39.842 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.842 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:08:39.843 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [], 2024-03-20T09:08:39.843 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2024-03-20T09:08:39.843 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_primaries": [], 2024-03-20T09:08:39.843 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [ 2024-03-20T09:08:39.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.843 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "1.8", 2024-03-20T09:08:39.843 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:08:39.843 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-03-20T09:08:39.843 INFO:teuthology.orchestra.run.smithi026.stdout: 8, 2024-03-20T09:08:39.843 INFO:teuthology.orchestra.run.smithi026.stdout: 4 2024-03-20T09:08:39.843 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.844 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "1.b", 2024-03-20T09:08:39.844 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:08:39.844 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-03-20T09:08:39.844 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-03-20T09:08:39.844 INFO:teuthology.orchestra.run.smithi026.stdout: 11 2024-03-20T09:08:39.844 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.844 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "1.f", 2024-03-20T09:08:39.844 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:08:39.845 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-03-20T09:08:39.845 INFO:teuthology.orchestra.run.smithi026.stdout: 8, 2024-03-20T09:08:39.845 INFO:teuthology.orchestra.run.smithi026.stdout: 4 2024-03-20T09:08:39.845 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.845 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "1.15", 2024-03-20T09:08:39.845 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:08:39.845 INFO:teuthology.orchestra.run.smithi026.stdout: 8, 2024-03-20T09:08:39.845 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-03-20T09:08:39.845 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-03-20T09:08:39.846 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.846 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "1.17", 2024-03-20T09:08:39.846 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:08:39.846 INFO:teuthology.orchestra.run.smithi026.stdout: 8, 2024-03-20T09:08:39.846 INFO:teuthology.orchestra.run.smithi026.stdout: 10, 2024-03-20T09:08:39.846 INFO:teuthology.orchestra.run.smithi026.stdout: 6 2024-03-20T09:08:39.846 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.847 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "1.18", 2024-03-20T09:08:39.847 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:08:39.847 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-03-20T09:08:39.847 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-03-20T09:08:39.847 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-03-20T09:08:39.847 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.847 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.2", 2024-03-20T09:08:39.847 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:08:39.847 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-03-20T09:08:39.848 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-03-20T09:08:39.848 INFO:teuthology.orchestra.run.smithi026.stdout: 4 2024-03-20T09:08:39.848 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.848 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.6", 2024-03-20T09:08:39.848 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:08:39.848 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-03-20T09:08:39.848 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-03-20T09:08:39.848 INFO:teuthology.orchestra.run.smithi026.stdout: 4 2024-03-20T09:08:39.848 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.849 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.8", 2024-03-20T09:08:39.849 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:08:39.849 INFO:teuthology.orchestra.run.smithi026.stdout: 11, 2024-03-20T09:08:39.849 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-03-20T09:08:39.849 INFO:teuthology.orchestra.run.smithi026.stdout: 4 2024-03-20T09:08:39.849 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.849 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.a", 2024-03-20T09:08:39.849 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:08:39.849 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-03-20T09:08:39.849 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-03-20T09:08:39.849 INFO:teuthology.orchestra.run.smithi026.stdout: 4 2024-03-20T09:08:39.850 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.850 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.b", 2024-03-20T09:08:39.850 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:08:39.850 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-03-20T09:08:39.850 INFO:teuthology.orchestra.run.smithi026.stdout: 11 2024-03-20T09:08:39.850 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.850 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.c", 2024-03-20T09:08:39.851 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:08:39.851 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-03-20T09:08:39.851 INFO:teuthology.orchestra.run.smithi026.stdout: 10, 2024-03-20T09:08:39.851 INFO:teuthology.orchestra.run.smithi026.stdout: 3 2024-03-20T09:08:39.851 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.851 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.d", 2024-03-20T09:08:39.851 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:08:39.851 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-03-20T09:08:39.851 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-03-20T09:08:39.851 INFO:teuthology.orchestra.run.smithi026.stdout: 9 2024-03-20T09:08:39.851 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.852 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.e", 2024-03-20T09:08:39.852 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:08:39.852 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-03-20T09:08:39.852 INFO:teuthology.orchestra.run.smithi026.stdout: 10 2024-03-20T09:08:39.852 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.852 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.10", 2024-03-20T09:08:39.852 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:08:39.852 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-03-20T09:08:39.853 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-03-20T09:08:39.853 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-03-20T09:08:39.853 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.853 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.15", 2024-03-20T09:08:39.853 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:08:39.853 INFO:teuthology.orchestra.run.smithi026.stdout: 11, 2024-03-20T09:08:39.853 INFO:teuthology.orchestra.run.smithi026.stdout: 0 2024-03-20T09:08:39.853 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.854 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.16", 2024-03-20T09:08:39.854 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:08:39.854 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-03-20T09:08:39.854 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-03-20T09:08:39.854 INFO:teuthology.orchestra.run.smithi026.stdout: 7 2024-03-20T09:08:39.854 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:08:39.854 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.17", 2024-03-20T09:08:39.854 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:08:39.854 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-03-20T09:08:39.855 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-03-20T09:08:39.855 INFO:teuthology.orchestra.run.smithi026.stdout: 6 2024-03-20T09:08:39.855 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:08:39.855 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.855 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:08:39.855 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2024-03-20T09:08:39.855 INFO:teuthology.orchestra.run.smithi026.stdout: "blocklist": { 2024-03-20T09:08:39.855 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/4291404191": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:08:39.855 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/642012566": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:08:39.855 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6801/1181459999": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:08:39.855 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/3984513952": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:08:39.855 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6800/1654945498": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:08:39.855 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6800/1181459999": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:08:39.856 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/137770981": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:08:39.856 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/1451237762": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:08:39.856 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/3938386631": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:08:39.856 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6801/1654945498": "2024-03-21T08:49:04.596754+0000" 2024-03-20T09:08:39.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.856 INFO:teuthology.orchestra.run.smithi026.stdout: "range_blocklist": {}, 2024-03-20T09:08:39.856 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2024-03-20T09:08:39.856 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2024-03-20T09:08:39.856 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2024-03-20T09:08:39.856 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2024-03-20T09:08:39.856 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2024-03-20T09:08:39.856 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2024-03-20T09:08:39.856 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2024-03-20T09:08:39.857 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:08:39.857 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2024-03-20T09:08:39.857 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2024-03-20T09:08:39.857 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2024-03-20T09:08:39.857 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2024-03-20T09:08:39.857 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {}, 2024-03-20T09:08:39.857 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": { 2024-03-20T09:08:39.857 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_enabled": false, 2024-03-20T09:08:39.857 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_bucket_count": 0, 2024-03-20T09:08:39.857 INFO:teuthology.orchestra.run.smithi026.stdout: "degraded_stretch_mode": 0, 2024-03-20T09:08:39.857 INFO:teuthology.orchestra.run.smithi026.stdout: "recovering_stretch_mode": 0, 2024-03-20T09:08:39.858 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_bucket": 0 2024-03-20T09:08:39.858 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:08:39.858 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:08:39.858 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:40.015 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:08:40.016 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:08:40.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:40 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3353684560' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:08:40.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:40 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:08:40.417 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:40 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3353684560' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:08:40.417 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:40 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:08:40.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:40 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3353684560' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:08:40.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:40 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:08:40.908 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:08:41.107 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:08:41.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:41.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:08:41.125 INFO:tasks.thrashosds.thrasher:j is {'epoch': 397, 'fsid': '852fbb68-e696-11ee-95c9-87774f69a715', 'created': '2024-03-20T08:48:35.281088+0000', 'modified': '2024-03-20T09:08:31.958601+0000', 'last_up_change': '2024-03-20T09:04:33.695633+0000', 'last_in_change': '2024-03-20T09:07:48.551904+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-20T08:52:17.442324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '360', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '358', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.700000047683716, 'score_stable': 2.700000047683716, 'optimal_score': 0.6700000166893005, 'raw_score_acting': 1.809999942779541, 'raw_score_stable': 1.809999942779541, 'primary_affinity_weighted': 0.9215061664581299, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 3, 'pool_name': 'unique_pool_1', 'create_time': '2024-03-20T09:02:48.375175+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '233', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '231', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.2100000381469727, 'score_stable': 3.2100000381469727, 'optimal_score': 0.6700000166893005, 'raw_score_acting': 2.1500000953674316, 'raw_score_stable': 2.1500000953674316, 'primary_affinity_weighted': 0.9215061664581299, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}], 'osds': [{'osd': 0, 'uuid': 'ad3fd9c0-316f-4fe5-a63b-d480087c1729', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 385, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6802', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6803', 'nonce': 2139805293}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6804', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6805', 'nonce': 2139805293}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6808', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6809', 'nonce': 2139805293}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6806', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6807', 'nonce': 2139805293}]}, 'public_addr': '172.21.15.26:6803/2139805293', 'cluster_addr': '172.21.15.26:6805/2139805293', 'heartbeat_back_addr': '172.21.15.26:6809/2139805293', 'heartbeat_front_addr': '172.21.15.26:6807/2139805293', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'e85b94f5-2954-47a1-9c20-096f2606eb04', 'up': 1, 'in': 1, 'weight': 0.97100830078125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 386, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6810', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6811', 'nonce': 3853355134}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6812', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6813', 'nonce': 3853355134}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6816', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6817', 'nonce': 3853355134}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6814', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6815', 'nonce': 3853355134}]}, 'public_addr': '172.21.15.26:6811/3853355134', 'cluster_addr': '172.21.15.26:6813/3853355134', 'heartbeat_back_addr': '172.21.15.26:6817/3853355134', 'heartbeat_front_addr': '172.21.15.26:6815/3853355134', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'dd2b10f0-741c-47a4-b224-ac6bfe2c14ef', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 383, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6818', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6819', 'nonce': 3993357967}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6820', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6821', 'nonce': 3993357967}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6824', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6825', 'nonce': 3993357967}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6822', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6823', 'nonce': 3993357967}]}, 'public_addr': '172.21.15.26:6819/3993357967', 'cluster_addr': '172.21.15.26:6821/3993357967', 'heartbeat_back_addr': '172.21.15.26:6825/3993357967', 'heartbeat_front_addr': '172.21.15.26:6823/3993357967', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 274, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6826', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6827', 'nonce': 4110225509}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6828', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6829', 'nonce': 4110225509}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6832', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6833', 'nonce': 4110225509}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6830', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6831', 'nonce': 4110225509}]}, 'public_addr': '172.21.15.26:6827/4110225509', 'cluster_addr': '172.21.15.26:6829/4110225509', 'heartbeat_back_addr': '172.21.15.26:6833/4110225509', 'heartbeat_front_addr': '172.21.15.26:6831/4110225509', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'f14e34d5-9829-42a3-9b25-31a0976c5ef7', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 366, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6800', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6801', 'nonce': 477923058}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6802', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6803', 'nonce': 477923058}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6806', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6807', 'nonce': 477923058}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6804', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6805', 'nonce': 477923058}]}, 'public_addr': '172.21.15.115:6801/477923058', 'cluster_addr': '172.21.15.115:6803/477923058', 'heartbeat_back_addr': '172.21.15.115:6807/477923058', 'heartbeat_front_addr': '172.21.15.115:6805/477923058', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'cb431b6a-540d-4318-ae35-d379e1df0185', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 265, 'down_at': 274, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6808', 'nonce': 1339268003}, {'type': 'v1', 'addr': '172.21.15.115:6809', 'nonce': 1339268003}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6810', 'nonce': 1339268003}, {'type': 'v1', 'addr': '172.21.15.115:6811', 'nonce': 1339268003}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6814', 'nonce': 1339268003}, {'type': 'v1', 'addr': '172.21.15.115:6815', 'nonce': 1339268003}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6812', 'nonce': 1339268003}, {'type': 'v1', 'addr': '172.21.15.115:6813', 'nonce': 1339268003}]}, 'public_addr': '172.21.15.115:6809/1339268003', 'cluster_addr': '172.21.15.115:6811/1339268003', 'heartbeat_back_addr': '172.21.15.115:6815/1339268003', 'heartbeat_front_addr': '172.21.15.115:6813/1339268003', 'state': ['exists']}, {'osd': 6, 'uuid': '753ed830-e764-4ff3-9d9b-87e7fa748b81', 'up': 1, 'in': 1, 'weight': 0.842376708984375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 394, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6816', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6817', 'nonce': 3278301843}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6818', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6819', 'nonce': 3278301843}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6822', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6823', 'nonce': 3278301843}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6820', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6821', 'nonce': 3278301843}]}, 'public_addr': '172.21.15.115:6817/3278301843', 'cluster_addr': '172.21.15.115:6819/3278301843', 'heartbeat_back_addr': '172.21.15.115:6823/3278301843', 'heartbeat_front_addr': '172.21.15.115:6821/3278301843', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '55bcbd37-7a6f-4ab1-b5a9-91a951444d86', 'up': 1, 'in': 1, 'weight': 0.4264678955078125, 'primary_affinity': 0.058074951171875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 274, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6824', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6825', 'nonce': 3152424001}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6826', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6827', 'nonce': 3152424001}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6830', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6831', 'nonce': 3152424001}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6828', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6829', 'nonce': 3152424001}]}, 'public_addr': '172.21.15.115:6825/3152424001', 'cluster_addr': '172.21.15.115:6827/3152424001', 'heartbeat_back_addr': '172.21.15.115:6831/3152424001', 'heartbeat_front_addr': '172.21.15.115:6829/3152424001', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'bc86248e-6bd0-4369-b8a2-e593bc2a610d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 70, 'up_thru': 380, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6800', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6801', 'nonce': 1309426740}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6802', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6803', 'nonce': 1309426740}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6806', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6807', 'nonce': 1309426740}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6804', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6805', 'nonce': 1309426740}]}, 'public_addr': '172.21.15.122:6801/1309426740', 'cluster_addr': '172.21.15.122:6803/1309426740', 'heartbeat_back_addr': '172.21.15.122:6807/1309426740', 'heartbeat_front_addr': '172.21.15.122:6805/1309426740', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '9b20a4aa-032a-4c09-ac75-153c554424e7', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 257, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6808', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6809', 'nonce': 2953210076}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6810', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6811', 'nonce': 2953210076}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6814', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6815', 'nonce': 2953210076}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6812', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6813', 'nonce': 2953210076}]}, 'public_addr': '172.21.15.122:6809/2953210076', 'cluster_addr': '172.21.15.122:6811/2953210076', 'heartbeat_back_addr': '172.21.15.122:6815/2953210076', 'heartbeat_front_addr': '172.21.15.122:6813/2953210076', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'e62c1fb3-5dd1-4d0d-997c-d20633d19361', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 89, 'up_thru': 386, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6816', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6817', 'nonce': 409394340}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6818', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6819', 'nonce': 409394340}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6822', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6823', 'nonce': 409394340}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6820', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6821', 'nonce': 409394340}]}, 'public_addr': '172.21.15.122:6817/409394340', 'cluster_addr': '172.21.15.122:6819/409394340', 'heartbeat_back_addr': '172.21.15.122:6823/409394340', 'heartbeat_front_addr': '172.21.15.122:6821/409394340', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'b83994d5-e5ef-48b0-9758-c26843cec3b6', 'up': 1, 'in': 1, 'weight': 0.6123046875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 96, 'up_thru': 385, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6824', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6825', 'nonce': 2704069354}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6826', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6827', 'nonce': 2704069354}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6830', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6831', 'nonce': 2704069354}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6828', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6829', 'nonce': 2704069354}]}, 'public_addr': '172.21.15.122:6825/2704069354', 'cluster_addr': '172.21.15.122:6827/2704069354', 'heartbeat_back_addr': '172.21.15.122:6831/2704069354', 'heartbeat_front_addr': '172.21.15.122:6829/2704069354', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:08:04.860551+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:07:33.401607+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:08:18.494972+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 35063, 'last_purged_snaps_scrub': '2024-03-20T09:08:00.063524+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-20T09:08:30.077943+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:03:53.609882+0000', 'dead_epoch': 273}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:08:14.465561+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:07:57.050958+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:08:24.980388+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 34736, 'last_purged_snaps_scrub': '2024-03-20T09:07:52.842622+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:08:19.771648+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:07:26.222744+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '1.8', 'osds': [2, 8, 4]}, {'pgid': '1.b', 'osds': [0, 2, 11]}, {'pgid': '1.f', 'osds': [6, 8, 4]}, {'pgid': '1.15', 'osds': [8, 1, 2]}, {'pgid': '1.17', 'osds': [8, 10, 6]}, {'pgid': '1.18', 'osds': [2, 7, 8]}, {'pgid': '3.2', 'osds': [6, 1, 4]}, {'pgid': '3.6', 'osds': [0, 1, 4]}, {'pgid': '3.8', 'osds': [11, 1, 4]}, {'pgid': '3.a', 'osds': [6, 1, 4]}, {'pgid': '3.b', 'osds': [0, 11]}, {'pgid': '3.c', 'osds': [6, 10, 3]}, {'pgid': '3.d', 'osds': [6, 7, 9]}, {'pgid': '3.e', 'osds': [0, 10]}, {'pgid': '3.10', 'osds': [6, 0, 2]}, {'pgid': '3.15', 'osds': [11, 0]}, {'pgid': '3.16', 'osds': [1, 2, 7]}, {'pgid': '3.17', 'osds': [0, 1, 6]}], 'primary_temp': [], 'blocklist': {'172.21.15.26:0/4291404191': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/642012566': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:6801/1181459999': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/3984513952': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6800/1654945498': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6800/1181459999': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/137770981': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:0/1451237762': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/3938386631': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6801/1654945498': '2024-03-21T08:49:04.596754+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-20T09:08:41.126 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:08:41.215 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:08:41.244 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:41.265 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:08:41.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:41.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:08:41.369 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:08:41.396 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:41.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:41 smithi115 ceph-mon[72476]: pgmap v959: 57 pgs: 1 active+recovering+remapped, 13 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.3 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 126 op/s; 9260/212556 objects degraded (4.356%); 36435/212556 objects misplaced (17.141%); 17 MiB/s, 362 objects/s recovering 2024-03-20T09:08:41.421 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:08:41.422 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:41 smithi122 ceph-mon[73495]: pgmap v959: 57 pgs: 1 active+recovering+remapped, 13 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.3 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 126 op/s; 9260/212556 objects degraded (4.356%); 36435/212556 objects misplaced (17.141%); 17 MiB/s, 362 objects/s recovering 2024-03-20T09:08:41.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:41.423 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:08:41.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:41 smithi026 ceph-mon[68309]: pgmap v959: 57 pgs: 1 active+recovering+remapped, 13 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.3 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 126 op/s; 9260/212556 objects degraded (4.356%); 36435/212556 objects misplaced (17.141%); 17 MiB/s, 362 objects/s recovering 2024-03-20T09:08:41.524 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:08:41.552 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:41.575 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:08:41.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:41.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:08:41.679 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:08:41.707 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:41.730 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:08:41.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:41.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:08:41.836 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:08:41.864 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:41.886 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:08:41.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:41.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:08:41.990 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:08:42.017 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:08:42.018 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:42.037 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:08:42.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:42.040 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:08:42.141 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:08:42.169 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:42.188 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:08:42.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:42.191 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:08:42.292 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:08:42.316 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:42 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:08:42.316 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:42 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:08:42.317 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:42 smithi122 ceph-mon[73495]: osdmap e398: 12 total, 11 up, 9 in 2024-03-20T09:08:42.317 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:42 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:08:42.317 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:42 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:08:42.317 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:42 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:08:42.317 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:42 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 9260/212556 objects degraded (4.356%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-03-20T09:08:42.317 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:42 smithi122 ceph-mon[73495]: osdmap e399: 12 total, 11 up, 9 in 2024-03-20T09:08:42.318 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:42.343 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:08:42.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:42.347 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:08:42.402 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:08:42.447 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:08:42.470 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:08:42.470 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:08:42.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:42 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:08:42.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:42 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:08:42.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:42 smithi026 ceph-mon[68309]: osdmap e398: 12 total, 11 up, 9 in 2024-03-20T09:08:42.475 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:42 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:08:42.475 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:42 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:08:42.475 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:42 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:08:42.475 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:42 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 9260/212556 objects degraded (4.356%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-03-20T09:08:42.475 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:42 smithi026 ceph-mon[68309]: osdmap e399: 12 total, 11 up, 9 in 2024-03-20T09:08:42.491 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:42.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:42 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:08:42.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:42 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:08:42.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:42 smithi115 ceph-mon[72476]: osdmap e398: 12 total, 11 up, 9 in 2024-03-20T09:08:42.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:08:42.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:08:42.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:08:42.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:42 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 9260/212556 objects degraded (4.356%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-03-20T09:08:42.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:42 smithi115 ceph-mon[72476]: osdmap e399: 12 total, 11 up, 9 in 2024-03-20T09:08:42.699 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:08:42.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:42.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:08:42.804 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:08:42.839 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:42.994 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:08:43.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:43.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:08:43.103 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:08:43.141 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:43.353 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-20T09:08:43.426 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:43 smithi026 ceph-mon[68309]: pgmap v961: 57 pgs: 1 active+recovering+remapped, 13 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.3 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 213 op/s; 9212/209931 objects degraded (4.388%); 34448/209931 objects misplaced (16.409%); 14 MiB/s, 219 objects/s recovering 2024-03-20T09:08:43.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:43 smithi115 ceph-mon[72476]: pgmap v961: 57 pgs: 1 active+recovering+remapped, 13 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.3 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 213 op/s; 9212/209931 objects degraded (4.388%); 34448/209931 objects misplaced (16.409%); 14 MiB/s, 219 objects/s recovering 2024-03-20T09:08:43.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:43 smithi122 ceph-mon[73495]: pgmap v961: 57 pgs: 1 active+recovering+remapped, 13 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.3 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 213 op/s; 9212/209931 objects degraded (4.388%); 34448/209931 objects misplaced (16.409%); 14 MiB/s, 219 objects/s recovering 2024-03-20T09:08:43.717 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:08:43.717 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:08:45.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:45 smithi115 ceph-mon[72476]: pgmap v963: 57 pgs: 1 active+recovering+remapped, 13 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.3 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 170 op/s; 9174/209703 objects degraded (4.375%); 34448/209703 objects misplaced (16.427%); 8.5 MiB/s, 135 objects/s recovering 2024-03-20T09:08:45.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:45 smithi122 ceph-mon[73495]: pgmap v963: 57 pgs: 1 active+recovering+remapped, 13 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.3 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 170 op/s; 9174/209703 objects degraded (4.375%); 34448/209703 objects misplaced (16.427%); 8.5 MiB/s, 135 objects/s recovering 2024-03-20T09:08:45.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:45 smithi026 ceph-mon[68309]: pgmap v963: 57 pgs: 1 active+recovering+remapped, 13 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.3 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 170 op/s; 9174/209703 objects degraded (4.375%); 34448/209703 objects misplaced (16.427%); 8.5 MiB/s, 135 objects/s recovering 2024-03-20T09:08:47.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:47 smithi115 ceph-mon[72476]: pgmap v964: 57 pgs: 1 active+recovering+remapped, 13 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.2 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 271 op/s; 9070/206988 objects degraded (4.382%); 32372/206988 objects misplaced (15.640%); 17 MiB/s, 306 objects/s recovering 2024-03-20T09:08:47.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:47 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 9070/206988 objects degraded (4.382%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-03-20T09:08:47.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:47 smithi122 ceph-mon[73495]: pgmap v964: 57 pgs: 1 active+recovering+remapped, 13 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.2 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 271 op/s; 9070/206988 objects degraded (4.382%); 32372/206988 objects misplaced (15.640%); 17 MiB/s, 306 objects/s recovering 2024-03-20T09:08:47.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:47 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 9070/206988 objects degraded (4.382%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-03-20T09:08:47.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:47 smithi026 ceph-mon[68309]: pgmap v964: 57 pgs: 1 active+recovering+remapped, 13 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.2 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 271 op/s; 9070/206988 objects degraded (4.382%); 32372/206988 objects misplaced (15.640%); 17 MiB/s, 306 objects/s recovering 2024-03-20T09:08:47.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:47 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 9070/206988 objects degraded (4.382%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-03-20T09:08:49.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:49 smithi115 ceph-mon[72476]: pgmap v965: 57 pgs: 1 active+recovering+remapped, 13 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.2 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 247 op/s; 9070/206601 objects degraded (4.390%); 32372/206601 objects misplaced (15.669%); 17 MiB/s, 298 objects/s recovering 2024-03-20T09:08:49.671 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:49 smithi026 ceph-mon[68309]: pgmap v965: 57 pgs: 1 active+recovering+remapped, 13 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.2 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 247 op/s; 9070/206601 objects degraded (4.390%); 32372/206601 objects misplaced (15.669%); 17 MiB/s, 298 objects/s recovering 2024-03-20T09:08:49.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:49 smithi122 ceph-mon[73495]: pgmap v965: 57 pgs: 1 active+recovering+remapped, 13 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.2 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 247 op/s; 9070/206601 objects degraded (4.390%); 32372/206601 objects misplaced (15.669%); 17 MiB/s, 298 objects/s recovering 2024-03-20T09:08:51.046 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:08:51.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:51.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:08:51.156 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:08:51.213 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:51.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:51 smithi026 ceph-mon[68309]: pgmap v966: 57 pgs: 1 active+recovering+remapped, 13 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.2 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 181 op/s; 8942/205572 objects degraded (4.350%); 32259/205572 objects misplaced (15.692%); 8.6 MiB/s, 168 objects/s recovering 2024-03-20T09:08:51.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:51 smithi115 ceph-mon[72476]: pgmap v966: 57 pgs: 1 active+recovering+remapped, 13 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.2 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 181 op/s; 8942/205572 objects degraded (4.350%); 32259/205572 objects misplaced (15.692%); 8.6 MiB/s, 168 objects/s recovering 2024-03-20T09:08:51.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:51 smithi122 ceph-mon[73495]: pgmap v966: 57 pgs: 1 active+recovering+remapped, 13 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.2 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 181 op/s; 8942/205572 objects degraded (4.350%); 32259/205572 objects misplaced (15.692%); 8.6 MiB/s, 168 objects/s recovering 2024-03-20T09:08:51.857 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:08:51.881 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:08:51.906 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:08:52.133 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:08:52.160 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:08:52.182 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:08:52.212 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:08:52.217 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:08:52.254 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:08:52.268 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:08:52.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:52.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:08:52.296 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:08:52.321 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:08:52.346 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:08:52.346 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:08:52.346 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:08:52.346 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:08:52.346 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:08:52.346 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:08:52.347 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-20T09:08:52.347 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:08:52.347 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:08:52.347 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:08:52.347 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:08:52.347 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:08:52.347 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:08:52.352 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:52 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3704059968' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:08:52.352 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:52 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8942/205572 objects degraded (4.350%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-03-20T09:08:52.352 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:52 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3343562412' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:08:52.360 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:08:52.374 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:08:52.406 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:52.427 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:08:52.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:52.433 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:08:52.534 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:08:52.569 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:52.627 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:08:52.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:52.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:08:52.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:52 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3704059968' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:08:52.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:52 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8942/205572 objects degraded (4.350%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-03-20T09:08:52.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:52 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3343562412' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:08:52.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:52 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3704059968' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:08:52.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:52 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8942/205572 objects degraded (4.350%), 6 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-03-20T09:08:52.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:52 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3343562412' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:08:52.732 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:08:52.763 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:52.941 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:08:52.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:52.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:08:53.054 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:08:53.084 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:53.434 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:08:53.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:53.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:08:53.440 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:08:53.440 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:08:53.538 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:08:53.565 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:53.585 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:08:53.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:53.589 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:08:53.622 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:53 smithi026 ceph-mon[68309]: pgmap v967: 57 pgs: 1 active+remapped+backfilling, 13 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.1 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 243 op/s; 8908/202614 objects degraded (4.397%); 35276/202614 objects misplaced (17.410%); 13 MiB/s, 302 objects/s recovering 2024-03-20T09:08:53.622 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:53 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:08:53.622 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:53 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3343562412' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:08:53.622 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:53 smithi026 ceph-mon[68309]: osdmap e400: 12 total, 11 up, 9 in 2024-03-20T09:08:53.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:53 smithi115 ceph-mon[72476]: pgmap v967: 57 pgs: 1 active+remapped+backfilling, 13 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.1 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 243 op/s; 8908/202614 objects degraded (4.397%); 35276/202614 objects misplaced (17.410%); 13 MiB/s, 302 objects/s recovering 2024-03-20T09:08:53.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:53 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:08:53.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:53 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3343562412' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:08:53.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:53 smithi115 ceph-mon[72476]: osdmap e400: 12 total, 11 up, 9 in 2024-03-20T09:08:53.690 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:08:53.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:53 smithi122 ceph-mon[73495]: pgmap v967: 57 pgs: 1 active+remapped+backfilling, 13 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.1 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 243 op/s; 8908/202614 objects degraded (4.397%); 35276/202614 objects misplaced (17.410%); 13 MiB/s, 302 objects/s recovering 2024-03-20T09:08:53.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:53 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:08:53.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:53 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3343562412' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:08:53.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:53 smithi122 ceph-mon[73495]: osdmap e400: 12 total, 11 up, 9 in 2024-03-20T09:08:53.722 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:54.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:54 smithi026 ceph-mon[68309]: 1.18 scrub starts 2024-03-20T09:08:54.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:54 smithi026 ceph-mon[68309]: 1.18 scrub ok 2024-03-20T09:08:54.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:54 smithi026 ceph-mon[68309]: 1.1a scrub starts 2024-03-20T09:08:54.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:54 smithi026 ceph-mon[68309]: 1.1a scrub ok 2024-03-20T09:08:54.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:54 smithi115 ceph-mon[72476]: 1.18 scrub starts 2024-03-20T09:08:54.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:54 smithi115 ceph-mon[72476]: 1.18 scrub ok 2024-03-20T09:08:54.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:54 smithi115 ceph-mon[72476]: 1.1a scrub starts 2024-03-20T09:08:54.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:54 smithi115 ceph-mon[72476]: 1.1a scrub ok 2024-03-20T09:08:54.690 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:08:54.693 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:08:54.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:54 smithi122 ceph-mon[73495]: 1.18 scrub starts 2024-03-20T09:08:54.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:54 smithi122 ceph-mon[73495]: 1.18 scrub ok 2024-03-20T09:08:54.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:54 smithi122 ceph-mon[73495]: 1.1a scrub starts 2024-03-20T09:08:54.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:54 smithi122 ceph-mon[73495]: 1.1a scrub ok 2024-03-20T09:08:55.400 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":969,"stamp":"2024-03-20T09:08:53.711254+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4422154769,"num_objects":67465,"num_object_clones":0,"num_object_copies":202395,"num_objects_missing_on_primary":4729,"num_objects_missing":4729,"num_objects_degraded":8875,"num_objects_misplaced":35276,"num_objects_unfound":0,"num_objects_dirty":67465,"num_whiteouts":0,"num_read":30,"num_read_kb":26,"num_write":98669,"num_write_kb":4314014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110299,"num_bytes_recovered":6285234781,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":176419,"ondisk_log_size":176419,"up":150,"acting":161,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":179,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":15873292,"kb_used_data":13921216,"kb_used_omap":14,"kb_used_meta":1951985,"kb_avail":827833076,"statfs":{"total":863955320832,"available":847701069824,"internally_reserved":0,"allocated":14255325184,"data_stored":14247583151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":1998833647},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-159514352,"num_objects":-2436,"num_object_clones":0,"num_object_copies":-7308,"num_objects_missing_on_primary":-252,"num_objects_missing":-252,"num_objects_degraded":-299,"num_objects_misplaced":828,"num_objects_unfound":0,"num_objects_dirty":-2436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2436,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3029,"num_bytes_recovered":135554256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002686"},"pg_stats":[{"pgid":"1.1e","version":"20'137","reported_seq":478,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-03-20T09:08:42.035856+0000","last_change":"2024-03-20T09:07:41.961068+0000","last_active":"2024-03-20T09:08:42.035856+0000","last_peered":"2024-03-20T09:08:42.035856+0000","last_clean":"2024-03-20T09:08:42.035856+0000","last_became_active":"2024-03-20T09:07:41.960647+0000","last_became_peered":"2024-03-20T09:07:41.960647+0000","last_unstale":"2024-03-20T09:08:42.035856+0000","last_undegraded":"2024-03-20T09:08:42.035856+0000","last_fullsized":"2024-03-20T09:08:42.035856+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:03:55.906959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:03:55.906959+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:08:46.968303+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,10],"acting":[8,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"20'137","reported_seq":322,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-03-20T09:08:41.968056+0000","last_change":"2024-03-20T09:07:42.013692+0000","last_active":"2024-03-20T09:08:41.968056+0000","last_peered":"2024-03-20T09:08:41.968056+0000","last_clean":"2024-03-20T09:08:41.968056+0000","last_became_active":"2024-03-20T09:07:42.013245+0000","last_became_peered":"2024-03-20T09:07:42.013245+0000","last_unstale":"2024-03-20T09:08:41.968056+0000","last_undegraded":"2024-03-20T09:08:41.968056+0000","last_fullsized":"2024-03-20T09:08:41.968056+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:03:48.928433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:03:48.928433+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:08:42.956287+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":5,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":24043,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"20'137","reported_seq":765,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-03-20T09:08:42.035614+0000","last_change":"2024-03-20T09:07:41.962095+0000","last_active":"2024-03-20T09:08:42.035614+0000","last_peered":"2024-03-20T09:08:42.035614+0000","last_clean":"2024-03-20T09:08:42.035614+0000","last_became_active":"2024-03-20T09:07:41.961622+0000","last_became_peered":"2024-03-20T09:07:41.961622+0000","last_unstale":"2024-03-20T09:08:42.035614+0000","last_undegraded":"2024-03-20T09:08:42.035614+0000","last_fullsized":"2024-03-20T09:08:42.035614+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:04:04.682759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:04:04.682759+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:08:46.968327+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163840,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.19","version":"20'137","reported_seq":841,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-03-20T09:08:41.967302+0000","last_change":"2024-03-20T09:07:41.963076+0000","last_active":"2024-03-20T09:08:41.967302+0000","last_peered":"2024-03-20T09:08:41.967302+0000","last_clean":"2024-03-20T09:08:41.967302+0000","last_became_active":"2024-03-20T09:07:41.962680+0000","last_became_peered":"2024-03-20T09:07:41.962680+0000","last_unstale":"2024-03-20T09:08:41.967302+0000","last_undegraded":"2024-03-20T09:08:41.967302+0000","last_fullsized":"2024-03-20T09:08:41.967302+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:03:56.726408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:03:56.726408+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":8,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":120211,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"20'137","reported_seq":545,"reported_epoch":400,"state":"active+clean+remapped","last_fresh":"2024-03-20T09:08:52.876579+0000","last_change":"2024-03-20T09:08:52.876636+0000","last_active":"2024-03-20T09:08:52.876579+0000","last_peered":"2024-03-20T09:08:52.876579+0000","last_clean":"2024-03-20T09:08:52.876579+0000","last_became_active":"2024-03-20T09:07:41.965126+0000","last_became_peered":"2024-03-20T09:07:41.965126+0000","last_unstale":"2024-03-20T09:08:52.876579+0000","last_undegraded":"2024-03-20T09:08:52.876579+0000","last_fullsized":"2024-03-20T09:08:52.876579+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:08:52.876490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:08:52.876490+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:10:01.705833+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7,8],"avail_no_missing":["2","7","8"],"object_location_counts":[{"shards":"2,7,8","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1b","version":"20'137","reported_seq":594,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-03-20T09:08:42.035620+0000","last_change":"2024-03-20T09:07:49.974049+0000","last_active":"2024-03-20T09:08:42.035620+0000","last_peered":"2024-03-20T09:08:42.035620+0000","last_clean":"2024-03-20T09:08:42.035620+0000","last_became_active":"2024-03-20T09:07:41.962733+0000","last_became_peered":"2024-03-20T09:07:41.962733+0000","last_unstale":"2024-03-20T09:08:42.035620+0000","last_undegraded":"2024-03-20T09:08:42.035620+0000","last_fullsized":"2024-03-20T09:08:42.035620+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:07:49.973958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:07:49.973958+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:08:54.897032+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.19","version":"399'5908","reported_seq":26326,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-03-20T09:08:49.914083+0000","last_change":"2024-03-20T09:08:16.043717+0000","last_active":"2024-03-20T09:08:49.914083+0000","last_peered":"2024-03-20T09:08:49.914083+0000","last_clean":"2024-03-20T09:08:49.914083+0000","last_became_active":"2024-03-20T09:08:16.036346+0000","last_became_peered":"2024-03-20T09:08:16.036346+0000","last_unstale":"2024-03-20T09:08:49.914083+0000","last_undegraded":"2024-03-20T09:08:49.914083+0000","last_fullsized":"2024-03-20T09:08:49.914083+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":5,"last_scrub":"284'4948","last_scrub_stamp":"2024-03-20T09:04:48.131969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:04:48.131969+0000","objects_scrubbed":3082,"log_size":5908,"log_dups_size":0,"ondisk_log_size":5908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:08:52.923949+0000","scrub_duration":324,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139067392,"num_objects":2122,"num_object_clones":0,"num_object_copies":6366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":960,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6290,"num_bytes_recovered":331808768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,2],"acting":[1,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"20'137","reported_seq":532,"reported_epoch":400,"state":"active+clean+remapped","last_fresh":"2024-03-20T09:08:52.355182+0000","last_change":"2024-03-20T09:07:36.954335+0000","last_active":"2024-03-20T09:08:52.355182+0000","last_peered":"2024-03-20T09:08:52.355182+0000","last_clean":"2024-03-20T09:08:52.355182+0000","last_became_active":"2024-03-20T09:07:36.954026+0000","last_became_peered":"2024-03-20T09:07:36.954026+0000","last_unstale":"2024-03-20T09:08:52.355182+0000","last_undegraded":"2024-03-20T09:08:52.355182+0000","last_fullsized":"2024-03-20T09:08:52.355182+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:05:23.614560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:05:23.614560+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:08:54.875887+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131344,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":100624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8,4],"avail_no_missing":["2","4","8"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"399'8292","reported_seq":29923,"reported_epoch":399,"state":"active+remapped+backfilling","last_fresh":"2024-03-20T09:08:50.290097+0000","last_change":"2024-03-20T09:08:47.663847+0000","last_active":"2024-03-20T09:08:50.290097+0000","last_peered":"2024-03-20T09:08:50.290097+0000","last_clean":"2024-03-20T09:05:59.443016+0000","last_became_active":"2024-03-20T09:07:50.801326+0000","last_became_peered":"2024-03-20T09:07:50.801326+0000","last_unstale":"2024-03-20T09:08:50.290097+0000","last_undegraded":"2024-03-20T09:08:50.290097+0000","last_fullsized":"2024-03-20T09:08:50.290097+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":8292,"log_dups_size":0,"ondisk_log_size":8292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279969792,"num_objects":4272,"num_object_clones":0,"num_object_copies":12816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3837,"num_objects_unfound":0,"num_objects_dirty":4272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8292,"num_write_kb":402048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10598,"num_bytes_recovered":631832576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,8],"acting":[6,1,4],"avail_no_missing":["6","1","4","7"],"object_location_counts":[{"shards":"1,4,6","objects":4272}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"20'137","reported_seq":842,"reported_epoch":400,"state":"active+undersized","last_fresh":"2024-03-20T09:08:52.355203+0000","last_change":"2024-03-20T09:07:36.951344+0000","last_active":"2024-03-20T09:08:52.355203+0000","last_peered":"2024-03-20T09:08:52.355203+0000","last_clean":"2024-03-20T09:04:29.198571+0000","last_became_active":"2024-03-20T09:07:36.951344+0000","last_became_peered":"2024-03-20T09:07:36.951344+0000","last_unstale":"2024-03-20T09:08:52.355203+0000","last_undegraded":"2024-03-20T09:08:52.355203+0000","last_fullsized":"2024-03-20T09:07:36.946346+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:03:56.726408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:03:56.726408+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:08:54.875899+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8,"num_write_kb":133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":120211,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"399'8346","reported_seq":34217,"reported_epoch":399,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-20T09:08:50.697735+0000","last_change":"2024-03-20T09:08:08.228591+0000","last_active":"2024-03-20T09:08:50.697735+0000","last_peered":"2024-03-20T09:08:50.697735+0000","last_clean":"2024-03-20T09:07:48.417829+0000","last_became_active":"2024-03-20T09:07:50.853930+0000","last_became_peered":"2024-03-20T09:07:50.853930+0000","last_unstale":"2024-03-20T09:08:50.697735+0000","last_undegraded":"2024-03-20T09:07:50.636239+0000","last_fullsized":"2024-03-20T09:07:50.567026+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":8346,"log_dups_size":0,"ondisk_log_size":8346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282853376,"num_objects":4316,"num_object_clones":0,"num_object_copies":12948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":1930,"num_objects_unfound":0,"num_objects_dirty":4316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8346,"num_write_kb":405184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12717,"num_bytes_recovered":777846784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,7],"acting":[0,11],"avail_no_missing":["0","3","4","11"],"object_location_counts":[{"shards":"0,4,11","objects":1930},{"shards":"0,11","objects":2376}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"20'137","reported_seq":651,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-03-20T09:08:41.967114+0000","last_change":"2024-03-20T09:07:52.618525+0000","last_active":"2024-03-20T09:08:41.967114+0000","last_peered":"2024-03-20T09:08:41.967114+0000","last_clean":"2024-03-20T09:08:41.967114+0000","last_became_active":"2024-03-20T09:07:52.618107+0000","last_became_peered":"2024-03-20T09:07:52.618107+0000","last_unstale":"2024-03-20T09:08:41.967114+0000","last_undegraded":"2024-03-20T09:08:41.967114+0000","last_fullsized":"2024-03-20T09:08:41.967114+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:03:13.525325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:03:13.525325+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:08:43.117674+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"399'5956","reported_seq":6372,"reported_epoch":399,"state":"active+clean+remapped","last_fresh":"2024-03-20T09:08:49.018720+0000","last_change":"2024-03-20T09:07:50.588770+0000","last_active":"2024-03-20T09:08:49.018720+0000","last_peered":"2024-03-20T09:08:49.018720+0000","last_clean":"2024-03-20T09:08:49.018720+0000","last_became_active":"2024-03-20T09:07:50.576012+0000","last_became_peered":"2024-03-20T09:07:50.576012+0000","last_unstale":"2024-03-20T09:08:49.018720+0000","last_undegraded":"2024-03-20T09:08:49.018720+0000","last_fullsized":"2024-03-20T09:08:49.018720+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":5956,"log_dups_size":0,"ondisk_log_size":5956,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139427840,"num_objects":2127,"num_object_clones":0,"num_object_copies":6381,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2127,"num_objects_unfound":0,"num_objects_dirty":2127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4191,"num_write_kb":202208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1,4],"avail_no_missing":["11","1","4"],"object_location_counts":[{"shards":"1,4,11","objects":2127}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"20'137","reported_seq":806,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-03-20T09:08:42.005670+0000","last_change":"2024-03-20T09:06:02.927341+0000","last_active":"2024-03-20T09:08:42.005670+0000","last_peered":"2024-03-20T09:08:42.005670+0000","last_clean":"2024-03-20T09:08:42.005670+0000","last_became_active":"2024-03-20T09:06:02.926326+0000","last_became_peered":"2024-03-20T09:06:02.926326+0000","last_unstale":"2024-03-20T09:08:42.005670+0000","last_undegraded":"2024-03-20T09:08:42.005670+0000","last_fullsized":"2024-03-20T09:08:42.005670+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:24.397259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:24.397259+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:08:43.653735+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109226,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":109226,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.6","version":"399'5944","reported_seq":13916,"reported_epoch":399,"state":"active+clean+remapped","last_fresh":"2024-03-20T09:08:50.448205+0000","last_change":"2024-03-20T09:08:31.564953+0000","last_active":"2024-03-20T09:08:50.448205+0000","last_peered":"2024-03-20T09:08:50.448205+0000","last_clean":"2024-03-20T09:08:50.448205+0000","last_became_active":"2024-03-20T09:08:30.522411+0000","last_became_peered":"2024-03-20T09:08:30.522411+0000","last_unstale":"2024-03-20T09:08:50.448205+0000","last_undegraded":"2024-03-20T09:08:50.448205+0000","last_fullsized":"2024-03-20T09:08:50.448205+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'4943","last_scrub_stamp":"2024-03-20T09:04:46.116541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:04:46.116541+0000","objects_scrubbed":3048,"log_size":5944,"log_dups_size":0,"ondisk_log_size":5944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":317,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134152192,"num_objects":2047,"num_object_clones":0,"num_object_copies":6141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2047,"num_objects_unfound":0,"num_objects_dirty":2047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1001,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3089,"num_bytes_recovered":160890880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1,4],"avail_no_missing":["0","1","4"],"object_location_counts":[{"shards":"0,1,4","objects":2047}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"20'137","reported_seq":718,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-03-20T09:08:41.967104+0000","last_change":"2024-03-20T09:07:37.003973+0000","last_active":"2024-03-20T09:08:41.967104+0000","last_peered":"2024-03-20T09:08:41.967104+0000","last_clean":"2024-03-20T09:08:41.967104+0000","last_became_active":"2024-03-20T09:07:37.003722+0000","last_became_peered":"2024-03-20T09:07:37.003722+0000","last_unstale":"2024-03-20T09:08:41.967104+0000","last_undegraded":"2024-03-20T09:08:41.967104+0000","last_fullsized":"2024-03-20T09:08:41.967104+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:07:06.480160+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:07:06.480160+0000","last_clean_scrub_stamp":"2024-03-20T09:07:06.480160+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.7","version":"399'5889","reported_seq":22043,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-03-20T09:08:49.303393+0000","last_change":"2024-03-20T09:08:15.050515+0000","last_active":"2024-03-20T09:08:49.303393+0000","last_peered":"2024-03-20T09:08:49.303393+0000","last_clean":"2024-03-20T09:08:49.303393+0000","last_became_active":"2024-03-20T09:08:15.041211+0000","last_became_peered":"2024-03-20T09:08:15.041211+0000","last_unstale":"2024-03-20T09:08:49.303393+0000","last_undegraded":"2024-03-20T09:08:49.303393+0000","last_fullsized":"2024-03-20T09:08:49.303393+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'4179","last_scrub_stamp":"2024-03-20T09:04:00.661196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:04:00.661196+0000","objects_scrubbed":2441,"log_size":5889,"log_dups_size":0,"ondisk_log_size":5889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139067392,"num_objects":2122,"num_object_clones":0,"num_object_copies":6366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1711,"num_write_kb":44544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6274,"num_bytes_recovered":325976064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"20'137","reported_seq":736,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-03-20T09:08:41.967113+0000","last_change":"2024-03-20T09:07:52.593985+0000","last_active":"2024-03-20T09:08:41.967113+0000","last_peered":"2024-03-20T09:08:41.967113+0000","last_clean":"2024-03-20T09:08:41.967113+0000","last_became_active":"2024-03-20T09:07:52.593671+0000","last_became_peered":"2024-03-20T09:07:52.593671+0000","last_unstale":"2024-03-20T09:08:41.967113+0000","last_undegraded":"2024-03-20T09:08:41.967113+0000","last_fullsized":"2024-03-20T09:08:41.967113+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:06:58.034269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:06:58.034269+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.4","version":"399'5856","reported_seq":6274,"reported_epoch":399,"state":"active+undersized+degraded","last_fresh":"2024-03-20T09:08:49.957457+0000","last_change":"2024-03-20T09:04:34.736993+0000","last_active":"2024-03-20T09:08:49.957457+0000","last_peered":"2024-03-20T09:08:49.957457+0000","last_clean":"2024-03-20T09:04:33.698023+0000","last_became_active":"2024-03-20T09:04:34.736993+0000","last_became_peered":"2024-03-20T09:04:34.736993+0000","last_unstale":"2024-03-20T09:08:49.957457+0000","last_undegraded":"2024-03-20T09:04:34.735838+0000","last_fullsized":"2024-03-20T09:04:34.735696+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":199,"log_size":5856,"log_dups_size":0,"ondisk_log_size":5856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:08:50.896708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139100160,"num_objects":2122,"num_object_clones":0,"num_object_copies":6366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4118,"num_write_kb":199712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":["1","11"],"object_location_counts":[{"shards":"1,11","objects":2122}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"20'137","reported_seq":711,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-03-20T09:08:41.967403+0000","last_change":"2024-03-20T09:07:52.603604+0000","last_active":"2024-03-20T09:08:41.967403+0000","last_peered":"2024-03-20T09:08:41.967403+0000","last_clean":"2024-03-20T09:08:41.967403+0000","last_became_active":"2024-03-20T09:07:52.603192+0000","last_became_peered":"2024-03-20T09:07:52.603192+0000","last_unstale":"2024-03-20T09:08:41.967403+0000","last_undegraded":"2024-03-20T09:08:41.967403+0000","last_fullsized":"2024-03-20T09:08:41.967403+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:04:46.981050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:04:46.981050+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,10],"acting":[1,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"399'8870","reported_seq":15810,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-03-20T09:08:50.636231+0000","last_change":"2024-03-20T09:08:15.045902+0000","last_active":"2024-03-20T09:08:50.636231+0000","last_peered":"2024-03-20T09:08:50.636231+0000","last_clean":"2024-03-20T09:08:50.636231+0000","last_became_active":"2024-03-20T09:08:15.038033+0000","last_became_peered":"2024-03-20T09:08:15.038033+0000","last_unstale":"2024-03-20T09:08:50.636231+0000","last_undegraded":"2024-03-20T09:08:50.636231+0000","last_fullsized":"2024-03-20T09:08:50.636231+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"285'7948","last_scrub_stamp":"2024-03-20T09:04:57.536605+0000","last_deep_scrub":"285'7948","last_deep_scrub_stamp":"2024-03-20T09:04:57.536605+0000","last_clean_scrub_stamp":"2024-03-20T09:04:57.536605+0000","objects_scrubbed":3080,"log_size":8870,"log_dups_size":0,"ondisk_log_size":8870,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"queued for scrub","scrub_duration":5935,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140115968,"num_objects":2138,"num_object_clones":0,"num_object_copies":6414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2071,"num_write_kb":65920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3179,"num_bytes_recovered":150798336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,11],"acting":[0,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":777,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-03-20T09:08:42.005316+0000","last_change":"2024-03-20T09:06:02.926003+0000","last_active":"2024-03-20T09:08:42.005316+0000","last_peered":"2024-03-20T09:08:42.005316+0000","last_clean":"2024-03-20T09:08:42.005316+0000","last_became_active":"2024-03-20T09:06:02.925687+0000","last_became_peered":"2024-03-20T09:06:02.925687+0000","last_unstale":"2024-03-20T09:08:42.005316+0000","last_undegraded":"2024-03-20T09:08:42.005316+0000","last_fullsized":"2024-03-20T09:08:42.005316+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:23.383145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:23.383145+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:08:43.653721+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,10],"acting":[11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"399'5933","reported_seq":13938,"reported_epoch":399,"state":"active+clean+remapped","last_fresh":"2024-03-20T09:08:49.994262+0000","last_change":"2024-03-20T09:08:29.097299+0000","last_active":"2024-03-20T09:08:49.994262+0000","last_peered":"2024-03-20T09:08:49.994262+0000","last_clean":"2024-03-20T09:08:49.994262+0000","last_became_active":"2024-03-20T09:08:29.090030+0000","last_became_peered":"2024-03-20T09:08:29.090030+0000","last_unstale":"2024-03-20T09:08:49.994262+0000","last_undegraded":"2024-03-20T09:08:49.994262+0000","last_fullsized":"2024-03-20T09:08:49.994262+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":5933,"log_dups_size":0,"ondisk_log_size":5933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:08:51.223750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138018816,"num_objects":2106,"num_object_clones":0,"num_object_copies":6318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2106,"num_objects_unfound":0,"num_objects_dirty":2106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4158,"num_write_kb":200448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3456,"num_bytes_recovered":168296448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1,4],"avail_no_missing":["6","1","4"],"object_location_counts":[{"shards":"1,4,6","objects":2106}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"20'137","reported_seq":1027,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-03-20T09:08:41.968317+0000","last_change":"2024-03-20T09:07:52.618129+0000","last_active":"2024-03-20T09:08:41.968317+0000","last_peered":"2024-03-20T09:08:41.968317+0000","last_clean":"2024-03-20T09:08:41.968317+0000","last_became_active":"2024-03-20T09:07:52.617726+0000","last_became_peered":"2024-03-20T09:07:52.617726+0000","last_unstale":"2024-03-20T09:08:41.968317+0000","last_undegraded":"2024-03-20T09:08:41.968317+0000","last_fullsized":"2024-03-20T09:08:41.968317+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:03:59.584956+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:03:59.584956+0000","last_clean_scrub_stamp":"2024-03-20T09:03:59.584956+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"399'5797","reported_seq":7622,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-03-20T09:08:49.330487+0000","last_change":"2024-03-20T09:07:50.197027+0000","last_active":"2024-03-20T09:08:49.330487+0000","last_peered":"2024-03-20T09:08:49.330487+0000","last_clean":"2024-03-20T09:08:49.330487+0000","last_became_active":"2024-03-20T09:03:30.926683+0000","last_became_peered":"2024-03-20T09:03:30.926683+0000","last_unstale":"2024-03-20T09:08:49.330487+0000","last_undegraded":"2024-03-20T09:08:49.330487+0000","last_fullsized":"2024-03-20T09:08:49.330487+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"366'5661","last_scrub_stamp":"2024-03-20T09:07:50.196877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:07:50.196877+0000","objects_scrubbed":2239,"log_size":5797,"log_dups_size":0,"ondisk_log_size":5797,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:08:59.629478+0000","scrub_duration":216,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137822208,"num_objects":2103,"num_object_clones":0,"num_object_copies":6309,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":136,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,1],"acting":[11,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1","version":"20'137","reported_seq":740,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-03-20T09:08:41.967473+0000","last_change":"2024-03-20T09:07:03.025487+0000","last_active":"2024-03-20T09:08:41.967473+0000","last_peered":"2024-03-20T09:08:41.967473+0000","last_clean":"2024-03-20T09:08:41.967473+0000","last_became_active":"2024-03-20T09:06:02.923630+0000","last_became_peered":"2024-03-20T09:06:02.923630+0000","last_unstale":"2024-03-20T09:08:41.967473+0000","last_undegraded":"2024-03-20T09:08:41.967473+0000","last_fullsized":"2024-03-20T09:08:41.967473+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:07:03.025368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:07:03.025368+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2],"acting":[10,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3","version":"399'5943","reported_seq":12602,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-03-20T09:08:50.656120+0000","last_change":"2024-03-20T09:08:13.002350+0000","last_active":"2024-03-20T09:08:50.656120+0000","last_peered":"2024-03-20T09:08:50.656120+0000","last_clean":"2024-03-20T09:08:50.656120+0000","last_became_active":"2024-03-20T09:08:12.992488+0000","last_became_peered":"2024-03-20T09:08:12.992488+0000","last_unstale":"2024-03-20T09:08:50.656120+0000","last_undegraded":"2024-03-20T09:08:50.656120+0000","last_fullsized":"2024-03-20T09:08:50.656120+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":5943,"log_dups_size":0,"ondisk_log_size":5943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:08:52.581144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133267456,"num_objects":2033,"num_object_clones":0,"num_object_copies":6099,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4149,"num_write_kb":197856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3119,"num_bytes_recovered":145162240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,11],"acting":[0,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"20'137","reported_seq":709,"reported_epoch":399,"state":"active+undersized","last_fresh":"2024-03-20T09:08:41.967255+0000","last_change":"2024-03-20T09:07:36.949270+0000","last_active":"2024-03-20T09:08:41.967255+0000","last_peered":"2024-03-20T09:08:41.967255+0000","last_clean":"2024-03-20T09:04:29.191837+0000","last_became_active":"2024-03-20T09:07:36.949270+0000","last_became_peered":"2024-03-20T09:07:36.949270+0000","last_unstale":"2024-03-20T09:08:41.967255+0000","last_undegraded":"2024-03-20T09:08:41.967255+0000","last_fullsized":"2024-03-20T09:07:36.946730+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:21.734957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:21.734957+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:08:43.117617+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":["6","10"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"400'5956","reported_seq":7688,"reported_epoch":400,"state":"active+undersized+degraded","last_fresh":"2024-03-20T09:08:52.539101+0000","last_change":"2024-03-20T09:04:34.737379+0000","last_active":"2024-03-20T09:08:52.539101+0000","last_peered":"2024-03-20T09:08:52.539101+0000","last_clean":"2024-03-20T09:04:33.579072+0000","last_became_active":"2024-03-20T09:04:34.737379+0000","last_became_peered":"2024-03-20T09:04:34.737379+0000","last_unstale":"2024-03-20T09:08:52.539101+0000","last_undegraded":"2024-03-20T09:04:34.733476+0000","last_fullsized":"2024-03-20T09:04:34.733363+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"253'4200","last_scrub_stamp":"2024-03-20T09:04:00.025798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:04:00.025798+0000","objects_scrubbed":2428,"log_size":5956,"log_dups_size":0,"ondisk_log_size":5956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136052736,"num_objects":2076,"num_object_clones":0,"num_object_copies":6228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1760,"num_write_kb":45056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,6","objects":2076}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"20'137","reported_seq":591,"reported_epoch":399,"state":"active+clean+remapped","last_fresh":"2024-03-20T09:08:41.971842+0000","last_change":"2024-03-20T09:07:36.950563+0000","last_active":"2024-03-20T09:08:41.971842+0000","last_peered":"2024-03-20T09:08:41.971842+0000","last_clean":"2024-03-20T09:08:41.971842+0000","last_became_active":"2024-03-20T09:07:36.950103+0000","last_became_peered":"2024-03-20T09:07:36.950103+0000","last_unstale":"2024-03-20T09:08:41.971842+0000","last_undegraded":"2024-03-20T09:08:41.971842+0000","last_fullsized":"2024-03-20T09:08:41.971842+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:24.936556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:24.936556+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:08:42.474902+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2,11],"avail_no_missing":["0","2","11"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"399'5851","reported_seq":6328,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-03-20T09:08:51.341095+0000","last_change":"2024-03-20T09:06:58.210889+0000","last_active":"2024-03-20T09:08:51.341095+0000","last_peered":"2024-03-20T09:08:51.341095+0000","last_clean":"2024-03-20T09:08:51.341095+0000","last_became_active":"2024-03-20T09:03:30.910402+0000","last_became_peered":"2024-03-20T09:03:30.910402+0000","last_unstale":"2024-03-20T09:08:51.341095+0000","last_undegraded":"2024-03-20T09:08:51.341095+0000","last_fullsized":"2024-03-20T09:08:51.341095+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'5452","last_scrub_stamp":"2024-03-20T09:06:58.210664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:06:58.210664+0000","objects_scrubbed":2397,"log_size":5851,"log_dups_size":0,"ondisk_log_size":5851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":272,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130875392,"num_objects":1997,"num_object_clones":0,"num_object_copies":5991,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1001,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"20'137","reported_seq":466,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-03-20T09:08:42.035990+0000","last_change":"2024-03-20T09:07:52.604147+0000","last_active":"2024-03-20T09:08:42.035990+0000","last_peered":"2024-03-20T09:08:42.035990+0000","last_clean":"2024-03-20T09:08:42.035990+0000","last_became_active":"2024-03-20T09:07:52.603796+0000","last_became_peered":"2024-03-20T09:07:52.603796+0000","last_unstale":"2024-03-20T09:08:42.035990+0000","last_undegraded":"2024-03-20T09:08:42.035990+0000","last_fullsized":"2024-03-20T09:08:42.035990+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:03:55.906959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:03:55.906959+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:08:46.968350+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,2],"acting":[8,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.c","version":"399'12111","reported_seq":23810,"reported_epoch":399,"state":"active+clean+remapped","last_fresh":"2024-03-20T09:08:50.194123+0000","last_change":"2024-03-20T09:06:08.944363+0000","last_active":"2024-03-20T09:08:50.194123+0000","last_peered":"2024-03-20T09:08:50.194123+0000","last_clean":"2024-03-20T09:08:50.194123+0000","last_became_active":"2024-03-20T09:06:08.928778+0000","last_became_peered":"2024-03-20T09:06:08.928778+0000","last_unstale":"2024-03-20T09:08:50.194123+0000","last_undegraded":"2024-03-20T09:08:50.194123+0000","last_fullsized":"2024-03-20T09:08:50.194123+0000","mapping_epoch":313,"log_start":"224'2100","ondisk_log_start":"224'2100","created":204,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"270'9729","last_scrub_stamp":"2024-03-20T09:04:24.105978+0000","last_deep_scrub":"270'9729","last_deep_scrub_stamp":"2024-03-20T09:04:24.105978+0000","last_clean_scrub_stamp":"2024-03-20T09:04:24.105978+0000","objects_scrubbed":5255,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":26,"scrub_schedule":"queued for scrub","scrub_duration":25793,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284098560,"num_objects":4335,"num_object_clones":0,"num_object_copies":13005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4335,"num_objects_unfound":0,"num_objects_dirty":4335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8263,"num_write_kb":403136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5528,"num_bytes_recovered":362283008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10,3],"avail_no_missing":["6","3","10"],"object_location_counts":[{"shards":"3,6,10","objects":4335}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"20'137","reported_seq":723,"reported_epoch":399,"state":"active+clean+remapped","last_fresh":"2024-03-20T09:08:41.967229+0000","last_change":"2024-03-20T09:07:52.600973+0000","last_active":"2024-03-20T09:08:41.967229+0000","last_peered":"2024-03-20T09:08:41.967229+0000","last_clean":"2024-03-20T09:08:41.967229+0000","last_became_active":"2024-03-20T09:07:52.600508+0000","last_became_peered":"2024-03-20T09:07:52.600508+0000","last_unstale":"2024-03-20T09:08:41.967229+0000","last_undegraded":"2024-03-20T09:08:41.967229+0000","last_fullsized":"2024-03-20T09:08:41.967229+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:21.734957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:21.734957+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:08:43.117648+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8,4],"avail_no_missing":["6","4","8"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"399'8163","reported_seq":14001,"reported_epoch":399,"state":"active+recovering+degraded+remapped","last_fresh":"2024-03-20T09:08:50.241594+0000","last_change":"2024-03-20T09:04:47.242963+0000","last_active":"2024-03-20T09:08:50.241594+0000","last_peered":"2024-03-20T09:08:50.241594+0000","last_clean":"2024-03-20T09:04:10.072458+0000","last_became_active":"2024-03-20T09:04:46.620359+0000","last_became_peered":"2024-03-20T09:04:46.620359+0000","last_unstale":"2024-03-20T09:08:50.241594+0000","last_undegraded":"2024-03-20T09:04:46.424109+0000","last_fullsized":"2024-03-20T09:08:50.241594+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'5229","last_scrub_stamp":"2024-03-20T09:04:09.606259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:04:09.606259+0000","objects_scrubbed":5193,"log_size":8163,"log_dups_size":0,"ondisk_log_size":8163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276037656,"num_objects":4213,"num_object_clones":0,"num_object_copies":12639,"num_objects_missing_on_primary":2557,"num_objects_missing":2557,"num_objects_degraded":4213,"num_objects_misplaced":2557,"num_objects_unfound":0,"num_objects_dirty":4213,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8163,"num_write_kb":395969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2685,"num_bytes_recovered":175964160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7,9],"avail_no_missing":["7"],"object_location_counts":[{"shards":"6,7","objects":944},{"shards":"6,7,9","objects":711},{"shards":"7,9","objects":2557}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"20'137","reported_seq":764,"reported_epoch":399,"state":"active+undersized","last_fresh":"2024-03-20T09:08:41.968240+0000","last_change":"2024-03-20T09:07:36.950317+0000","last_active":"2024-03-20T09:08:41.968240+0000","last_peered":"2024-03-20T09:08:41.968240+0000","last_clean":"2024-03-20T09:04:29.200818+0000","last_became_active":"2024-03-20T09:07:36.950317+0000","last_became_peered":"2024-03-20T09:07:36.950317+0000","last_unstale":"2024-03-20T09:08:41.968240+0000","last_undegraded":"2024-03-20T09:08:41.968240+0000","last_fullsized":"2024-03-20T09:07:36.947977+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:04:04.682759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:04:04.682759+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:08:42.474916+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163840,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"399'8247","reported_seq":34759,"reported_epoch":399,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-20T09:08:50.621895+0000","last_change":"2024-03-20T09:08:31.698000+0000","last_active":"2024-03-20T09:08:50.621895+0000","last_peered":"2024-03-20T09:08:50.621895+0000","last_clean":"2024-03-20T09:04:10.062034+0000","last_became_active":"2024-03-20T09:08:30.542785+0000","last_became_peered":"2024-03-20T09:08:30.542785+0000","last_unstale":"2024-03-20T09:08:50.621895+0000","last_undegraded":"2024-03-20T09:08:15.948124+0000","last_fullsized":"2024-03-20T09:07:50.566231+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":8247,"log_dups_size":0,"ondisk_log_size":8247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268500992,"num_objects":4097,"num_object_clones":0,"num_object_copies":12291,"num_objects_missing_on_primary":1998,"num_objects_missing":1998,"num_objects_degraded":280,"num_objects_misplaced":5801,"num_objects_unfound":0,"num_objects_dirty":4097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8247,"num_write_kb":395008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11581,"num_bytes_recovered":713687040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,8],"acting":[0,10],"avail_no_missing":["4","9","11"],"object_location_counts":[{"shards":"0,3,4,9,10,11","objects":20},{"shards":"0,3,4,10,11","objects":912},{"shards":"0,4,9,10,11","objects":1455},{"shards":"0,10","objects":-277},{"shards":"3,4,9,10,11","objects":1383},{"shards":"4,9,10,11","objects":324}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"20'137","reported_seq":310,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-03-20T09:08:41.967961+0000","last_change":"2024-03-20T09:08:00.708670+0000","last_active":"2024-03-20T09:08:41.967961+0000","last_peered":"2024-03-20T09:08:41.967961+0000","last_clean":"2024-03-20T09:08:41.967961+0000","last_became_active":"2024-03-20T09:08:00.708284+0000","last_became_peered":"2024-03-20T09:08:00.708284+0000","last_unstale":"2024-03-20T09:08:41.967961+0000","last_undegraded":"2024-03-20T09:08:41.967961+0000","last_fullsized":"2024-03-20T09:08:41.967961+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:03:48.928433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:03:48.928433+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:08:44.944329+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13417,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":4,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":24042,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,11],"acting":[1,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"399'8129","reported_seq":20907,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-03-20T09:08:49.425241+0000","last_change":"2024-03-20T09:07:49.396021+0000","last_active":"2024-03-20T09:08:49.425241+0000","last_peered":"2024-03-20T09:08:49.425241+0000","last_clean":"2024-03-20T09:08:49.425241+0000","last_became_active":"2024-03-20T09:06:59.023790+0000","last_became_peered":"2024-03-20T09:06:59.023790+0000","last_unstale":"2024-03-20T09:08:49.425241+0000","last_undegraded":"2024-03-20T09:08:49.425241+0000","last_fullsized":"2024-03-20T09:08:49.425241+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'7842","last_scrub_stamp":"2024-03-20T09:07:49.395885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:07:49.395885+0000","objects_scrubbed":4468,"log_size":8129,"log_dups_size":0,"ondisk_log_size":8129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:08:51.498648+0000","scrub_duration":457,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274006016,"num_objects":4181,"num_object_clones":0,"num_object_copies":12543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8129,"num_write_kb":393920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6155,"num_bytes_recovered":345112576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,10],"acting":[11,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.12","version":"20'137","reported_seq":963,"reported_epoch":399,"state":"active+undersized","last_fresh":"2024-03-20T09:08:41.967296+0000","last_change":"2024-03-20T09:04:34.751346+0000","last_active":"2024-03-20T09:08:41.967296+0000","last_peered":"2024-03-20T09:08:41.967296+0000","last_clean":"2024-03-20T09:03:59.001853+0000","last_became_active":"2024-03-20T09:04:34.751346+0000","last_became_peered":"2024-03-20T09:04:34.751346+0000","last_unstale":"2024-03-20T09:08:41.967296+0000","last_undegraded":"2024-03-20T09:08:41.967296+0000","last_fullsized":"2024-03-20T09:04:34.735354+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:04:05.319747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:04:05.319747+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:08:44.944303+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"399'5851","reported_seq":25351,"reported_epoch":399,"state":"active+clean+remapped","last_fresh":"2024-03-20T09:08:50.053189+0000","last_change":"2024-03-20T09:08:11.991828+0000","last_active":"2024-03-20T09:08:50.053189+0000","last_peered":"2024-03-20T09:08:50.053189+0000","last_clean":"2024-03-20T09:08:50.053189+0000","last_became_active":"2024-03-20T09:08:11.985281+0000","last_became_peered":"2024-03-20T09:08:11.985281+0000","last_unstale":"2024-03-20T09:08:50.053189+0000","last_undegraded":"2024-03-20T09:08:50.053189+0000","last_fullsized":"2024-03-20T09:08:50.053189+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":5851,"log_dups_size":0,"ondisk_log_size":5851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:08:53.260961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137428992,"num_objects":2097,"num_object_clones":0,"num_object_copies":6291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2097,"num_objects_unfound":0,"num_objects_dirty":2097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4107,"num_write_kb":198528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8292,"num_bytes_recovered":495484928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0,2],"avail_no_missing":["6","0","2"],"object_location_counts":[{"shards":"0,2,6","objects":2097}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.13","version":"20'137","reported_seq":700,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-03-20T09:08:41.967250+0000","last_change":"2024-03-20T09:04:48.832913+0000","last_active":"2024-03-20T09:08:41.967250+0000","last_peered":"2024-03-20T09:08:41.967250+0000","last_clean":"2024-03-20T09:08:41.967250+0000","last_became_active":"2024-03-20T09:04:04.103645+0000","last_became_peered":"2024-03-20T09:04:04.103645+0000","last_unstale":"2024-03-20T09:08:41.967250+0000","last_undegraded":"2024-03-20T09:08:41.967250+0000","last_fullsized":"2024-03-20T09:08:41.967250+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:04:48.832800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:04:48.832800+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"399'5944","reported_seq":16458,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-03-20T09:08:51.086682+0000","last_change":"2024-03-20T09:08:16.042688+0000","last_active":"2024-03-20T09:08:51.086682+0000","last_peered":"2024-03-20T09:08:51.086682+0000","last_clean":"2024-03-20T09:08:51.086682+0000","last_became_active":"2024-03-20T09:08:16.036521+0000","last_became_peered":"2024-03-20T09:08:16.036521+0000","last_unstale":"2024-03-20T09:08:51.086682+0000","last_undegraded":"2024-03-20T09:08:51.086682+0000","last_fullsized":"2024-03-20T09:08:51.086682+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":5,"last_scrub":"345'5607","last_scrub_stamp":"2024-03-20T09:07:01.247738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:07:01.247738+0000","objects_scrubbed":2440,"log_size":5944,"log_dups_size":0,"ondisk_log_size":5944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":262,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137756672,"num_objects":2102,"num_object_clones":0,"num_object_copies":6306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2070,"num_write_kb":65984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3141,"num_bytes_recovered":146604032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,0],"acting":[10,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.10","version":"20'137","reported_seq":491,"reported_epoch":399,"state":"active+undersized","last_fresh":"2024-03-20T09:08:41.966865+0000","last_change":"2024-03-20T09:04:34.735801+0000","last_active":"2024-03-20T09:08:41.966865+0000","last_peered":"2024-03-20T09:08:41.966865+0000","last_clean":"2024-03-20T08:54:29.262652+0000","last_became_active":"2024-03-20T09:04:34.735801+0000","last_became_peered":"2024-03-20T09:04:34.735801+0000","last_unstale":"2024-03-20T09:08:41.966865+0000","last_undegraded":"2024-03-20T09:08:41.966865+0000","last_fullsized":"2024-03-20T09:04:34.732688+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:04:04.319316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:04:04.319316+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":8,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":48084,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":["10","7"],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.12","version":"399'5872","reported_seq":21246,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-03-20T09:08:50.533082+0000","last_change":"2024-03-20T09:06:44.957783+0000","last_active":"2024-03-20T09:08:50.533082+0000","last_peered":"2024-03-20T09:08:50.533082+0000","last_clean":"2024-03-20T09:08:50.533082+0000","last_became_active":"2024-03-20T09:06:44.949696+0000","last_became_peered":"2024-03-20T09:06:44.949696+0000","last_unstale":"2024-03-20T09:08:50.533082+0000","last_undegraded":"2024-03-20T09:08:50.533082+0000","last_fullsized":"2024-03-20T09:08:50.533082+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":5,"last_scrub":"285'4956","last_scrub_stamp":"2024-03-20T09:04:50.135501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:04:50.135501+0000","objects_scrubbed":3023,"log_size":5872,"log_dups_size":0,"ondisk_log_size":5872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":356,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138018816,"num_objects":2106,"num_object_clones":0,"num_object_copies":6318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":917,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3104,"num_bytes_recovered":169148416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"20'137","reported_seq":571,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-03-20T09:08:41.969733+0000","last_change":"2024-03-20T09:06:02.928303+0000","last_active":"2024-03-20T09:08:41.969733+0000","last_peered":"2024-03-20T09:08:41.969733+0000","last_clean":"2024-03-20T09:08:41.969733+0000","last_became_active":"2024-03-20T09:06:02.927902+0000","last_became_peered":"2024-03-20T09:06:02.927902+0000","last_unstale":"2024-03-20T09:08:41.969733+0000","last_undegraded":"2024-03-20T09:08:41.969733+0000","last_fullsized":"2024-03-20T09:08:41.969733+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:04:42.894571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:04:42.894571+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1],"acting":[0,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"400'5956","reported_seq":18081,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-20T09:08:52.862920+0000","last_change":"2024-03-20T09:08:12.976469+0000","last_active":"2024-03-20T09:08:52.862920+0000","last_peered":"2024-03-20T09:08:52.862920+0000","last_clean":"2024-03-20T09:08:52.862920+0000","last_became_active":"2024-03-20T09:08:12.968732+0000","last_became_peered":"2024-03-20T09:08:12.968732+0000","last_unstale":"2024-03-20T09:08:52.862920+0000","last_undegraded":"2024-03-20T09:08:52.862920+0000","last_fullsized":"2024-03-20T09:08:52.862920+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":5956,"log_dups_size":0,"ondisk_log_size":5956,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138313728,"num_objects":2111,"num_object_clones":0,"num_object_copies":6333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4163,"num_write_kb":200736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4934,"num_bytes_recovered":268304384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.16","version":"20'137","reported_seq":387,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-03-20T09:08:41.968166+0000","last_change":"2024-03-20T09:07:41.963035+0000","last_active":"2024-03-20T09:08:41.968166+0000","last_peered":"2024-03-20T09:08:41.968166+0000","last_clean":"2024-03-20T09:08:41.968166+0000","last_became_active":"2024-03-20T09:07:41.962567+0000","last_became_peered":"2024-03-20T09:07:41.962567+0000","last_unstale":"2024-03-20T09:08:41.968166+0000","last_undegraded":"2024-03-20T09:08:41.968166+0000","last_fullsized":"2024-03-20T09:08:41.968166+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:06:58.034269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:06:58.034269+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:08:42.956273+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,11],"acting":[1,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"399'5807","reported_seq":17727,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-03-20T09:08:51.550002+0000","last_change":"2024-03-20T09:08:10.016943+0000","last_active":"2024-03-20T09:08:51.550002+0000","last_peered":"2024-03-20T09:08:51.550002+0000","last_clean":"2024-03-20T09:08:51.550002+0000","last_became_active":"2024-03-20T09:08:10.010857+0000","last_became_peered":"2024-03-20T09:08:10.010857+0000","last_unstale":"2024-03-20T09:08:51.550002+0000","last_undegraded":"2024-03-20T09:08:51.550002+0000","last_fullsized":"2024-03-20T09:08:51.550002+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":5807,"log_dups_size":0,"ondisk_log_size":5807,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:08:53.069266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134184960,"num_objects":2048,"num_object_clones":0,"num_object_copies":6144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4070,"num_write_kb":195744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":263192576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.17","version":"20'137","reported_seq":711,"reported_epoch":399,"state":"active+clean+remapped","last_fresh":"2024-03-20T09:08:42.036097+0000","last_change":"2024-03-20T09:07:41.959847+0000","last_active":"2024-03-20T09:08:42.036097+0000","last_peered":"2024-03-20T09:08:42.036097+0000","last_clean":"2024-03-20T09:08:42.036097+0000","last_became_active":"2024-03-20T09:07:41.959382+0000","last_became_peered":"2024-03-20T09:07:41.959382+0000","last_unstale":"2024-03-20T09:08:42.036097+0000","last_undegraded":"2024-03-20T09:08:42.036097+0000","last_fullsized":"2024-03-20T09:08:42.036097+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:21.734957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:21.734957+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:08:46.968265+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10,6],"avail_no_missing":["8","6","10"],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.15","version":"399'6006","reported_seq":14540,"reported_epoch":399,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-20T09:08:49.632125+0000","last_change":"2024-03-20T09:08:11.498334+0000","last_active":"2024-03-20T09:08:49.632125+0000","last_peered":"2024-03-20T09:08:49.632125+0000","last_clean":"2024-03-20T09:03:31.866916+0000","last_became_active":"2024-03-20T09:07:59.224446+0000","last_became_peered":"2024-03-20T09:07:59.224446+0000","last_unstale":"2024-03-20T09:08:49.632125+0000","last_undegraded":"2024-03-20T09:07:54.170652+0000","last_fullsized":"2024-03-20T09:07:50.565002+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":6006,"log_dups_size":0,"ondisk_log_size":6006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143687680,"num_objects":2193,"num_object_clones":0,"num_object_copies":6579,"num_objects_missing_on_primary":174,"num_objects_missing":174,"num_objects_degraded":174,"num_objects_misplaced":4114,"num_objects_unfound":0,"num_objects_dirty":2193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4221,"num_write_kb":205216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1982,"num_bytes_recovered":129728512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,6],"acting":[11,0],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"0,11","objects":664},{"shards":"2,6","objects":174},{"shards":"4,7,11","objects":1231},{"shards":"6,7,8,11","objects":1},{"shards":"7,8,11","objects":123}],"blocked_by":[],"up_primary":8,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.14","version":"20'137","reported_seq":778,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-03-20T09:08:52.354992+0000","last_change":"2024-03-20T09:06:24.048877+0000","last_active":"2024-03-20T09:08:52.354992+0000","last_peered":"2024-03-20T09:08:52.354992+0000","last_clean":"2024-03-20T09:08:52.354992+0000","last_became_active":"2024-03-20T09:04:14.208448+0000","last_became_peered":"2024-03-20T09:04:14.208448+0000","last_unstale":"2024-03-20T09:08:52.354992+0000","last_undegraded":"2024-03-20T09:08:52.354992+0000","last_fullsized":"2024-03-20T09:08:52.354992+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:06:24.048708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:06:24.048708+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"399'5971","reported_seq":17881,"reported_epoch":399,"state":"active+clean+remapped","last_fresh":"2024-03-20T09:08:49.964998+0000","last_change":"2024-03-20T09:07:50.168018+0000","last_active":"2024-03-20T09:08:49.964998+0000","last_peered":"2024-03-20T09:08:49.964998+0000","last_clean":"2024-03-20T09:08:49.964998+0000","last_became_active":"2024-03-20T09:06:18.946699+0000","last_became_peered":"2024-03-20T09:06:18.946699+0000","last_unstale":"2024-03-20T09:08:49.964998+0000","last_undegraded":"2024-03-20T09:08:49.964998+0000","last_fullsized":"2024-03-20T09:08:49.964998+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":5,"last_scrub":"366'5814","last_scrub_stamp":"2024-03-20T09:07:50.167856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:07:50.167856+0000","objects_scrubbed":2325,"log_size":5971,"log_dups_size":0,"ondisk_log_size":5971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:09:07.829071+0000","scrub_duration":242,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142082048,"num_objects":2168,"num_object_clones":0,"num_object_copies":6504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2168,"num_objects_unfound":0,"num_objects_dirty":2168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1703,"num_write_kb":45248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3164,"num_bytes_recovered":178585600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2,7],"avail_no_missing":["1","2","7"],"object_location_counts":[{"shards":"1,2,7","objects":2168}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"20'137","reported_seq":322,"reported_epoch":399,"state":"active+clean+remapped","last_fresh":"2024-03-20T09:08:42.035758+0000","last_change":"2024-03-20T09:07:48.964270+0000","last_active":"2024-03-20T09:08:42.035758+0000","last_peered":"2024-03-20T09:08:42.035758+0000","last_clean":"2024-03-20T09:08:42.035758+0000","last_became_active":"2024-03-20T09:07:41.961768+0000","last_became_peered":"2024-03-20T09:07:41.961768+0000","last_unstale":"2024-03-20T09:08:42.035758+0000","last_undegraded":"2024-03-20T09:08:42.035758+0000","last_fullsized":"2024-03-20T09:08:42.035758+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:07:48.964057+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:07:06.480160+0000","last_clean_scrub_stamp":"2024-03-20T09:07:48.964057+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:08:52.021520+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1,2],"avail_no_missing":["8","1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.17","version":"399'5880","reported_seq":19507,"reported_epoch":399,"state":"active+clean+remapped","last_fresh":"2024-03-20T09:08:50.627638+0000","last_change":"2024-03-20T09:06:46.990481+0000","last_active":"2024-03-20T09:08:50.627638+0000","last_peered":"2024-03-20T09:08:50.627638+0000","last_clean":"2024-03-20T09:08:50.627638+0000","last_became_active":"2024-03-20T09:06:46.981557+0000","last_became_peered":"2024-03-20T09:06:46.981557+0000","last_unstale":"2024-03-20T09:08:50.627638+0000","last_undegraded":"2024-03-20T09:08:50.627638+0000","last_fullsized":"2024-03-20T09:08:50.627638+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":5,"last_scrub":"251'4129","last_scrub_stamp":"2024-03-20T09:03:57.865687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:03:57.865687+0000","objects_scrubbed":2325,"log_size":5880,"log_dups_size":0,"ondisk_log_size":5880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141295616,"num_objects":2156,"num_object_clones":0,"num_object_copies":6468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2156,"num_objects_unfound":0,"num_objects_dirty":2156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1759,"num_write_kb":50880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3120,"num_bytes_recovered":173342720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1,6],"avail_no_missing":["0","1","6"],"object_location_counts":[{"shards":"0,1,6","objects":2156}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1a","version":"20'137","reported_seq":311,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-03-20T09:08:41.967193+0000","last_change":"2024-03-20T09:07:52.605608+0000","last_active":"2024-03-20T09:08:41.967193+0000","last_peered":"2024-03-20T09:08:41.967193+0000","last_clean":"2024-03-20T09:08:41.967193+0000","last_became_active":"2024-03-20T09:07:52.605174+0000","last_became_peered":"2024-03-20T09:07:52.605174+0000","last_unstale":"2024-03-20T09:08:41.967193+0000","last_undegraded":"2024-03-20T09:08:41.967193+0000","last_fullsized":"2024-03-20T09:08:41.967193+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:03:13.525325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:03:13.525325+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:08:46.967125+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":8,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":48084,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"399'5794","reported_seq":19495,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-03-20T09:08:49.991168+0000","last_change":"2024-03-20T09:06:24.997013+0000","last_active":"2024-03-20T09:08:49.991168+0000","last_peered":"2024-03-20T09:08:49.991168+0000","last_clean":"2024-03-20T09:08:49.991168+0000","last_became_active":"2024-03-20T09:06:24.988319+0000","last_became_peered":"2024-03-20T09:06:24.988319+0000","last_unstale":"2024-03-20T09:08:49.991168+0000","last_undegraded":"2024-03-20T09:08:49.991168+0000","last_fullsized":"2024-03-20T09:08:49.991168+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":5,"last_scrub":"276'4872","last_scrub_stamp":"2024-03-20T09:04:44.020587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:04:44.020587+0000","objects_scrubbed":2997,"log_size":5794,"log_dups_size":0,"ondisk_log_size":5794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:08:52.923987+0000","scrub_duration":325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135987200,"num_objects":2075,"num_object_clones":0,"num_object_copies":6225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":922,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3042,"num_bytes_recovered":170590208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,8],"acting":[1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":4421189656,"num_objects":67463,"num_object_clones":0,"num_object_copies":202389,"num_objects_missing_on_primary":4729,"num_objects_missing":4729,"num_objects_degraded":8875,"num_objects_misplaced":35275,"num_objects_unfound":0,"num_objects_dirty":67463,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98628,"num_write_kb":4313345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110298,"num_bytes_recovered":6284640256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16980582400,"data_stored":16980574256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":172172,"ondisk_log_size":172172,"up":68,"acting":73,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":965113,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":29,"num_read_kb":25,"num_write":41,"num_write_kb":669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":594525,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2007040,"data_stored":1989280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4247,"ondisk_log_size":4247,"up":82,"acting":88,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860569,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2275640,"kb_used_data":2082032,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":91469512,"statfs":{"total":95995035648,"available":93664780288,"internally_reserved":0,"allocated":2132000768,"data_stored":2131117432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":198244804},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":89,"seq":382252089503,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1417080,"kb_used_data":1223472,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":92328072,"statfs":{"total":95995035648,"available":94543945728,"internally_reserved":0,"allocated":1252835328,"data_stored":1251951992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":198244800},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":0,"seq":343597383847,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":70,"seq":300647710891,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1044032,"kb_used_data":860152,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":92701120,"statfs":{"total":95995035648,"available":94925946880,"internally_reserved":0,"allocated":880795648,"data_stored":879908760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":188283334},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":60,"seq":257698037936,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1729348,"kb_used_data":1413564,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":92015804,"statfs":{"total":95995035648,"available":94224183296,"internally_reserved":0,"allocated":1447489536,"data_stored":1446598576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":323353034},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332275,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2066300,"kb_used_data":1850484,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":91678852,"statfs":{"total":95995035648,"available":93879144448,"internally_reserved":0,"allocated":1894895616,"data_stored":1894008208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":220985798},"hb_peers":[0,1,2,3,4,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575055,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2059192,"kb_used_data":1767024,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":91685960,"statfs":{"total":95995035648,"available":93886423040,"internally_reserved":0,"allocated":1809432576,"data_stored":1808549240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":299170246},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738575,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1804664,"kb_used_data":1548592,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":91940488,"statfs":{"total":95995035648,"available":94147059712,"internally_reserved":0,"allocated":1585758208,"data_stored":1584874872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411529,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1218704,"kb_used_data":1052868,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":92526448,"statfs":{"total":95995035648,"available":94747082752,"internally_reserved":0,"allocated":1078136832,"data_stored":1077245978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169802181},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":111669149892,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953665,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659271,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2258332,"kb_used_data":2123028,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":91486820,"statfs":{"total":95995035648,"available":93682503680,"internally_reserved":0,"allocated":2173980672,"data_stored":2173328093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1581711360,"data_stored":1581711360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1806041088,"data_stored":1806041088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1074528256,"data_stored":1074528256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":829489152,"data_stored":829489152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1371406336,"data_stored":1371406336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2174091264,"data_stored":2174091264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1883639808,"data_stored":1883635736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1444286464,"data_stored":1444282392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":884146176,"data_stored":884146176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":559874048,"data_stored":559874048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1245904896,"data_stored":1245904896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2125463552,"data_stored":2125463552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:08:55.404 INFO:tasks.thrashosds.thrasher:Setting 1.1a to [0, 2, 11, 6, 4, 9] 2024-03-20T09:08:55.404 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '1.1a', '0', '2', '11', '6', '4', '9'] 2024-03-20T09:08:55.404 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pg-upmap-items 1.1a 0 2 11 6 4 9 2024-03-20T09:08:55.405 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:08:55.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:55.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:08:55.441 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:08:55.506 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:08:55.534 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:55.555 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:08:55.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:55.558 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:08:55.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:55 smithi115 ceph-mon[72476]: pgmap v969: 57 pgs: 1 active+remapped+backfilling, 13 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.1 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 243 op/s; 8875/202395 objects degraded (4.385%); 35276/202395 objects misplaced (17.429%); 13 MiB/s, 302 objects/s recovering 2024-03-20T09:08:55.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:55 smithi115 ceph-mon[72476]: 3.13 scrub starts 2024-03-20T09:08:55.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:55 smithi115 ceph-mon[72476]: 3.13 scrub ok 2024-03-20T09:08:55.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:55 smithi115 ceph-mon[72476]: 1.16 scrub starts 2024-03-20T09:08:55.659 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:08:55.689 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:55 smithi026 ceph-mon[68309]: pgmap v969: 57 pgs: 1 active+remapped+backfilling, 13 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.1 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 243 op/s; 8875/202395 objects degraded (4.385%); 35276/202395 objects misplaced (17.429%); 13 MiB/s, 302 objects/s recovering 2024-03-20T09:08:55.689 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:55 smithi026 ceph-mon[68309]: 3.13 scrub starts 2024-03-20T09:08:55.689 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:55 smithi026 ceph-mon[68309]: 3.13 scrub ok 2024-03-20T09:08:55.689 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:55 smithi026 ceph-mon[68309]: 1.16 scrub starts 2024-03-20T09:08:55.690 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:55.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:55 smithi122 ceph-mon[73495]: pgmap v969: 57 pgs: 1 active+remapped+backfilling, 13 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 5 active+undersized; 4.1 GiB data, 15 GiB used, 789 GiB / 805 GiB avail; 0 B/s wr, 243 op/s; 8875/202395 objects degraded (4.385%); 35276/202395 objects misplaced (17.429%); 13 MiB/s, 302 objects/s recovering 2024-03-20T09:08:55.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:55 smithi122 ceph-mon[73495]: 3.13 scrub starts 2024-03-20T09:08:55.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:55 smithi122 ceph-mon[73495]: 3.13 scrub ok 2024-03-20T09:08:55.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:55 smithi122 ceph-mon[73495]: 1.16 scrub starts 2024-03-20T09:08:55.891 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:08:55.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:55.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:08:55.925 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:08:55.959 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:08:55.974 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:08:55.998 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:08:56.027 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:56.419 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:08:56.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:56.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:08:56.515 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:56 smithi026 ceph-mon[68309]: from='client.25342 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:08:56.515 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:56 smithi026 ceph-mon[68309]: 1.1d scrub starts 2024-03-20T09:08:56.515 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:56 smithi026 ceph-mon[68309]: 1.1d scrub ok 2024-03-20T09:08:56.524 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:08:56.554 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:56.594 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:08:56.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:56.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:08:56.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:56 smithi115 ceph-mon[72476]: from='client.25342 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:08:56.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:56 smithi115 ceph-mon[72476]: 1.1d scrub starts 2024-03-20T09:08:56.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:56 smithi115 ceph-mon[72476]: 1.1d scrub ok 2024-03-20T09:08:56.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:56 smithi122 ceph-mon[73495]: from='client.25342 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:08:56.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:56 smithi122 ceph-mon[73495]: 1.1d scrub starts 2024-03-20T09:08:56.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:56 smithi122 ceph-mon[73495]: 1.1d scrub ok 2024-03-20T09:08:56.697 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:08:56.726 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:56.745 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:08:56.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:56.748 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:08:56.849 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:08:56.878 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:56.899 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:08:56.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:56.901 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:08:57.002 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:08:57.029 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:57.048 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:08:57.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:57.051 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:08:57.128 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:08:57.129 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:08:57.129 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-03-20T09:08:57.129 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:08:57.152 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:08:57.180 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:57.199 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:08:57.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:57.202 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:08:57.303 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:08:57.335 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:57.362 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:08:57.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:57.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:08:57.470 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:08:57.503 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:57.577 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:08:57.583 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:57 smithi026 ceph-mon[68309]: pgmap v970: 57 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 13 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 31 active+clean, 5 active+undersized; 4.1 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 213 op/s; 8694/200544 objects degraded (4.335%); 33758/200544 objects misplaced (16.833%); 14 MiB/s, 291 objects/s recovering 2024-03-20T09:08:57.583 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:57 smithi026 ceph-mon[68309]: 1.8 scrub starts 2024-03-20T09:08:57.583 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:57 smithi026 ceph-mon[68309]: 1.8 scrub ok 2024-03-20T09:08:57.583 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:57 smithi026 ceph-mon[68309]: 1.16 scrub ok 2024-03-20T09:08:57.584 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:57 smithi026 ceph-mon[68309]: 1.d scrub starts 2024-03-20T09:08:57.584 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:57 smithi026 ceph-mon[68309]: 1.d scrub ok 2024-03-20T09:08:57.584 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:57 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8694/200544 objects degraded (4.335%), 6 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:08:57.584 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:57 smithi026 ceph-mon[68309]: osdmap e401: 12 total, 11 up, 9 in 2024-03-20T09:08:57.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:57.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:08:57.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:57 smithi115 ceph-mon[72476]: pgmap v970: 57 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 13 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 31 active+clean, 5 active+undersized; 4.1 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 213 op/s; 8694/200544 objects degraded (4.335%); 33758/200544 objects misplaced (16.833%); 14 MiB/s, 291 objects/s recovering 2024-03-20T09:08:57.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:57 smithi115 ceph-mon[72476]: 1.8 scrub starts 2024-03-20T09:08:57.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:57 smithi115 ceph-mon[72476]: 1.8 scrub ok 2024-03-20T09:08:57.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:57 smithi115 ceph-mon[72476]: 1.16 scrub ok 2024-03-20T09:08:57.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:57 smithi115 ceph-mon[72476]: 1.d scrub starts 2024-03-20T09:08:57.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:57 smithi115 ceph-mon[72476]: 1.d scrub ok 2024-03-20T09:08:57.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:57 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8694/200544 objects degraded (4.335%), 6 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:08:57.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:57 smithi115 ceph-mon[72476]: osdmap e401: 12 total, 11 up, 9 in 2024-03-20T09:08:57.685 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:08:57.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:57 smithi122 ceph-mon[73495]: pgmap v970: 57 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 13 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 31 active+clean, 5 active+undersized; 4.1 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 213 op/s; 8694/200544 objects degraded (4.335%); 33758/200544 objects misplaced (16.833%); 14 MiB/s, 291 objects/s recovering 2024-03-20T09:08:57.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:57 smithi122 ceph-mon[73495]: 1.8 scrub starts 2024-03-20T09:08:57.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:57 smithi122 ceph-mon[73495]: 1.8 scrub ok 2024-03-20T09:08:57.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:57 smithi122 ceph-mon[73495]: 1.16 scrub ok 2024-03-20T09:08:57.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:57 smithi122 ceph-mon[73495]: 1.d scrub starts 2024-03-20T09:08:57.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:57 smithi122 ceph-mon[73495]: 1.d scrub ok 2024-03-20T09:08:57.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:57 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8694/200544 objects degraded (4.335%), 6 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:08:57.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:57 smithi122 ceph-mon[73495]: osdmap e401: 12 total, 11 up, 9 in 2024-03-20T09:08:57.717 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:57.740 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:08:57.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:57.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:08:57.846 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:08:57.874 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:57.893 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:08:57.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:57.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:08:57.997 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:08:58.024 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:58.046 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:08:58.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:58.051 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:08:58.152 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:08:58.182 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:58.203 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:08:58.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:58.210 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:08:58.259 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-20T09:08:58.311 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:08:58.337 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:58.357 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:08:58.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:58.360 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:08:58.461 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:08:58.487 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:58.506 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:08:58.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:58.509 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:08:58.610 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:08:58.637 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:58.658 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:08:58.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:58.661 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:08:58.762 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:08:58.790 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:58.809 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:08:58.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:58.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:08:58.913 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:08:58.945 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:58.974 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:08:58.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:58.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:08:59.080 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:08:59.104 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:58 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:08:59.104 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:58 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:08:59.104 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:58 smithi115 ceph-mon[72476]: pgmap v972: 57 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 13 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 31 active+clean, 5 active+undersized; 4.1 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 248 op/s; 8711/199575 objects degraded (4.365%); 33468/199575 objects misplaced (16.770%); 18 MiB/s, 378 objects/s recovering 2024-03-20T09:08:59.105 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:58 smithi115 ceph-mon[72476]: 3.19 scrub starts 2024-03-20T09:08:59.105 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:58 smithi115 ceph-mon[72476]: osdmap e402: 12 total, 11 up, 9 in 2024-03-20T09:08:59.106 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:59.127 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:08:59.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:59.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:08:59.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:58 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:08:59.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:58 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:08:59.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:58 smithi026 ceph-mon[68309]: pgmap v972: 57 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 13 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 31 active+clean, 5 active+undersized; 4.1 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 248 op/s; 8711/199575 objects degraded (4.365%); 33468/199575 objects misplaced (16.770%); 18 MiB/s, 378 objects/s recovering 2024-03-20T09:08:59.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:58 smithi026 ceph-mon[68309]: 3.19 scrub starts 2024-03-20T09:08:59.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:58 smithi026 ceph-mon[68309]: osdmap e402: 12 total, 11 up, 9 in 2024-03-20T09:08:59.230 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:08:59.256 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:58 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:08:59.256 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:58 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:08:59.256 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:58 smithi122 ceph-mon[73495]: pgmap v972: 57 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 13 active+clean+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 31 active+clean, 5 active+undersized; 4.1 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 248 op/s; 8711/199575 objects degraded (4.365%); 33468/199575 objects misplaced (16.770%); 18 MiB/s, 378 objects/s recovering 2024-03-20T09:08:59.256 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:58 smithi122 ceph-mon[73495]: 3.19 scrub starts 2024-03-20T09:08:59.256 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:58 smithi122 ceph-mon[73495]: osdmap e402: 12 total, 11 up, 9 in 2024-03-20T09:08:59.258 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:59.277 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:08:59.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:59.280 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:08:59.381 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:08:59.409 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:59.463 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:08:59.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:59.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:08:59.502 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:08:59.568 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:08:59.595 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:59.615 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:08:59.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:59.618 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:08:59.719 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:08:59.747 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:59.767 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:08:59.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:59.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:08:59.872 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:08:59.901 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:08:59.927 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:08:59.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:08:59.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:09:00.032 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:09:00.058 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:00.077 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:09:00.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:00.081 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:09:00.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:59 smithi115 ceph-mon[72476]: 3.19 scrub ok 2024-03-20T09:09:00.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:59 smithi115 ceph-mon[72476]: 3.18 scrub starts 2024-03-20T09:09:00.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:08:59 smithi115 ceph-mon[72476]: 3.18 scrub ok 2024-03-20T09:09:00.182 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:09:00.212 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:00.231 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:09:00.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:00.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:09:00.334 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:59 smithi026 ceph-mon[68309]: 3.19 scrub ok 2024-03-20T09:09:00.335 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:59 smithi026 ceph-mon[68309]: 3.18 scrub starts 2024-03-20T09:09:00.335 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:08:59 smithi026 ceph-mon[68309]: 3.18 scrub ok 2024-03-20T09:09:00.336 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:09:00.369 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:00.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:59 smithi122 ceph-mon[73495]: 3.19 scrub ok 2024-03-20T09:09:00.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:59 smithi122 ceph-mon[73495]: 3.18 scrub starts 2024-03-20T09:09:00.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:08:59 smithi122 ceph-mon[73495]: 3.18 scrub ok 2024-03-20T09:09:00.722 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:09:00.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:00.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:09:00.827 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:09:00.828 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:09:00.854 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:00.874 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:09:00.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:00.878 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:09:00.915 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:09:00.939 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:09:00.959 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:09:00.979 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:09:00.982 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:09:01.008 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:09:01.019 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:01.032 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:09:01.058 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:09:01.080 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:09:01.101 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:09:01.124 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:09:01.151 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:09:01.151 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:09:01.151 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:09:01.151 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:09:01.151 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:09:01.151 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:09:01.151 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-20T09:09:01.151 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:09:01.151 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:09:01.152 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:09:01.152 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:09:01.152 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:09:01.152 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:09:01.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:00 smithi122 ceph-mon[73495]: pgmap v974: 57 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 31 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 4.1 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 8605/199140 objects degraded (4.321%); 33218/199140 objects misplaced (16.681%) 2024-03-20T09:09:01.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:00 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2690668483' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:09:01.202 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:00 smithi026 ceph-mon[68309]: pgmap v974: 57 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 31 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 4.1 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 8605/199140 objects degraded (4.321%); 33218/199140 objects misplaced (16.681%) 2024-03-20T09:09:01.202 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:00 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2690668483' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:09:01.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:00 smithi115 ceph-mon[72476]: pgmap v974: 57 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 31 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 4.1 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 8605/199140 objects degraded (4.321%); 33218/199140 objects misplaced (16.681%) 2024-03-20T09:09:01.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:00 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2690668483' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:09:01.874 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:09:01.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:01.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:09:01.979 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:09:02.013 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:02.056 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:09:02.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:02.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:09:02.173 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:09:02.203 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:02.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:01 smithi026 ceph-mon[68309]: 1.13 scrub starts 2024-03-20T09:09:02.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:01 smithi026 ceph-mon[68309]: 1.13 scrub ok 2024-03-20T09:09:02.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:01 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8605/199140 objects degraded (4.321%), 5 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:09:02.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:01 smithi026 ceph-mon[68309]: osdmap e403: 12 total, 11 up, 9 in 2024-03-20T09:09:02.224 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:09:02.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:02.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:09:02.329 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:09:02.363 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:02.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:01 smithi115 ceph-mon[72476]: 1.13 scrub starts 2024-03-20T09:09:02.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:01 smithi115 ceph-mon[72476]: 1.13 scrub ok 2024-03-20T09:09:02.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:01 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8605/199140 objects degraded (4.321%), 5 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:09:02.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:01 smithi115 ceph-mon[72476]: osdmap e403: 12 total, 11 up, 9 in 2024-03-20T09:09:02.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:01 smithi122 ceph-mon[73495]: 1.13 scrub starts 2024-03-20T09:09:02.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:01 smithi122 ceph-mon[73495]: 1.13 scrub ok 2024-03-20T09:09:02.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:01 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8605/199140 objects degraded (4.321%), 5 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:09:02.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:01 smithi122 ceph-mon[73495]: osdmap e403: 12 total, 11 up, 9 in 2024-03-20T09:09:02.491 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:09:02.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:02.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:09:02.520 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:09:02.521 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:09:02.595 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:09:02.624 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:03.000 INFO:teuthology.orchestra.run.smithi026.stderr:set 1.1a pg_upmap_items mapping to [0->2,11->6,4->9] 2024-03-20T09:09:03.278 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:09:03.282 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:02 smithi026 ceph-mon[68309]: pgmap v975: 57 pgs: 1 active+remapped+backfilling, 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 4.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 8566/198096 objects degraded (4.324%); 31314/198096 objects misplaced (15.807%) 2024-03-20T09:09:03.282 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:02 smithi026 ceph-mon[68309]: 1.14 scrub starts 2024-03-20T09:09:03.283 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:02 smithi026 ceph-mon[68309]: 1.14 scrub ok 2024-03-20T09:09:03.283 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:02 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1041327135' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.1a", "id": [0, 2, 11, 6, 4, 9]}]: dispatch 2024-03-20T09:09:03.283 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:02 smithi026 ceph-mon[68309]: 1.3 scrub starts 2024-03-20T09:09:03.283 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:02 smithi026 ceph-mon[68309]: 1.3 scrub ok 2024-03-20T09:09:03.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:03.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:09:03.386 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:09:03.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:02 smithi115 ceph-mon[72476]: pgmap v975: 57 pgs: 1 active+remapped+backfilling, 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 4.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 8566/198096 objects degraded (4.324%); 31314/198096 objects misplaced (15.807%) 2024-03-20T09:09:03.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:02 smithi115 ceph-mon[72476]: 1.14 scrub starts 2024-03-20T09:09:03.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:02 smithi115 ceph-mon[72476]: 1.14 scrub ok 2024-03-20T09:09:03.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:02 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1041327135' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.1a", "id": [0, 2, 11, 6, 4, 9]}]: dispatch 2024-03-20T09:09:03.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:02 smithi115 ceph-mon[72476]: 1.3 scrub starts 2024-03-20T09:09:03.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:02 smithi115 ceph-mon[72476]: 1.3 scrub ok 2024-03-20T09:09:03.416 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:03.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:02 smithi122 ceph-mon[73495]: pgmap v975: 57 pgs: 1 active+remapped+backfilling, 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 4.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 8566/198096 objects degraded (4.324%); 31314/198096 objects misplaced (15.807%) 2024-03-20T09:09:03.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:02 smithi122 ceph-mon[73495]: 1.14 scrub starts 2024-03-20T09:09:03.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:02 smithi122 ceph-mon[73495]: 1.14 scrub ok 2024-03-20T09:09:03.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:02 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1041327135' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.1a", "id": [0, 2, 11, 6, 4, 9]}]: dispatch 2024-03-20T09:09:03.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:02 smithi122 ceph-mon[73495]: 1.3 scrub starts 2024-03-20T09:09:03.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:02 smithi122 ceph-mon[73495]: 1.3 scrub ok 2024-03-20T09:09:03.517 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:09:03.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:03.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:09:03.626 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:09:03.654 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:03.854 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:09:03.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:03.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:09:03.961 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:09:03.987 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:04.009 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:09:04.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:04.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:09:04.113 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:09:04.150 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:04.227 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:09:04.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:04.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:09:04.332 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:09:04.357 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:04 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:09:04.358 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:04 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:09:04.358 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:04 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1041327135' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.1a", "id": [0, 2, 11, 6, 4, 9]}]': finished 2024-03-20T09:09:04.358 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:04 smithi122 ceph-mon[73495]: osdmap e404: 12 total, 11 up, 9 in 2024-03-20T09:09:04.359 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:04.377 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:09:04.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:04.381 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:09:04.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:04 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:09:04.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:04 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:09:04.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:04 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1041327135' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.1a", "id": [0, 2, 11, 6, 4, 9]}]': finished 2024-03-20T09:09:04.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:04 smithi115 ceph-mon[72476]: osdmap e404: 12 total, 11 up, 9 in 2024-03-20T09:09:04.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:04 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:09:04.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:04 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:09:04.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:04 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1041327135' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.1a", "id": [0, 2, 11, 6, 4, 9]}]': finished 2024-03-20T09:09:04.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:04 smithi026 ceph-mon[68309]: osdmap e404: 12 total, 11 up, 9 in 2024-03-20T09:09:04.481 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:09:04.507 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:04.522 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:09:04.529 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:09:04.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:04.531 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:09:04.632 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:09:04.666 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:04.741 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:09:04.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:04.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:09:04.857 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:09:04.883 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:04.902 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:09:04.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:04.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:09:05.009 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:09:05.014 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:09:05.035 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:05.060 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:09:05.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:05.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:09:05.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:05 smithi115 ceph-mon[72476]: pgmap v978: 57 pgs: 1 active+remapped+backfilling, 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 4.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 8558/198039 objects degraded (4.321%); 31314/198039 objects misplaced (15.812%) 2024-03-20T09:09:05.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:05 smithi115 ceph-mon[72476]: osdmap e405: 12 total, 11 up, 9 in 2024-03-20T09:09:05.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:05 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1543678047' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:09:05.164 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:09:05.190 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:05.214 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:09:05.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:05.215 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:09:05.316 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:09:05.342 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:05 smithi122 ceph-mon[73495]: pgmap v978: 57 pgs: 1 active+remapped+backfilling, 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 4.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 8558/198039 objects degraded (4.321%); 31314/198039 objects misplaced (15.812%) 2024-03-20T09:09:05.342 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:05 smithi122 ceph-mon[73495]: osdmap e405: 12 total, 11 up, 9 in 2024-03-20T09:09:05.342 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:05 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1543678047' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:09:05.344 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:05.363 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:09:05.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:05.365 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:09:05.466 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:09:05.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:05 smithi026 ceph-mon[68309]: pgmap v978: 57 pgs: 1 active+remapped+backfilling, 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 32 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 4.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 8558/198039 objects degraded (4.321%); 31314/198039 objects misplaced (15.812%) 2024-03-20T09:09:05.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:05 smithi026 ceph-mon[68309]: osdmap e405: 12 total, 11 up, 9 in 2024-03-20T09:09:05.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:05 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1543678047' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:09:05.494 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:05.514 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:09:05.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:05.518 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:09:05.619 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:09:05.646 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:05.665 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:09:05.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:05.667 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:09:05.769 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:09:05.795 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:05.814 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:09:05.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:05.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:09:05.872 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:09:05.873 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:09:05.917 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:09:05.945 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:05.963 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:09:05.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:05.966 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:09:06.067 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:09:06.099 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:06.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:06 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:09:06.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:06 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1543678047' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:09:06.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:06 smithi115 ceph-mon[72476]: osdmap e406: 12 total, 11 up, 9 in 2024-03-20T09:09:06.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:06 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:09:06.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:06 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1543678047' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:09:06.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:06 smithi122 ceph-mon[73495]: osdmap e406: 12 total, 11 up, 9 in 2024-03-20T09:09:06.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:06 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:09:06.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:06 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1543678047' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:09:06.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:06 smithi026 ceph-mon[68309]: osdmap e406: 12 total, 11 up, 9 in 2024-03-20T09:09:06.247 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:09:06.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:06.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:09:06.353 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:09:06.379 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:06.401 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:09:06.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:06.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:09:06.475 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:09:06.504 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:09:06.534 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:06.558 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:09:06.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:06.560 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:09:06.661 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:09:06.689 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:06.710 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:09:06.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:06.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:09:06.814 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:09:06.841 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:06.861 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:09:06.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:06.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:09:06.951 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-20T09:09:06.965 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:09:06.993 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:07.013 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:09:07.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:07.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:09:07.118 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:09:07.149 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:07.191 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:07 smithi026 ceph-mon[68309]: pgmap v981: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 4.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 72 op/s; 8514/197220 objects degraded (4.317%); 29482/197220 objects misplaced (14.949%); 14 MiB/s, 230 objects/s recovering 2024-03-20T09:09:07.191 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:07 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8514/197220 objects degraded (4.317%), 5 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:09:07.191 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:07 smithi026 ceph-mon[68309]: osdmap e407: 12 total, 11 up, 9 in 2024-03-20T09:09:07.192 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:09:07.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:07.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:09:07.297 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:09:07.328 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:07.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:07 smithi115 ceph-mon[72476]: pgmap v981: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 4.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 72 op/s; 8514/197220 objects degraded (4.317%); 29482/197220 objects misplaced (14.949%); 14 MiB/s, 230 objects/s recovering 2024-03-20T09:09:07.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:07 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8514/197220 objects degraded (4.317%), 5 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:09:07.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:07 smithi115 ceph-mon[72476]: osdmap e407: 12 total, 11 up, 9 in 2024-03-20T09:09:07.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:07 smithi122 ceph-mon[73495]: pgmap v981: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 4.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 72 op/s; 8514/197220 objects degraded (4.317%); 29482/197220 objects misplaced (14.949%); 14 MiB/s, 230 objects/s recovering 2024-03-20T09:09:07.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:07 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8514/197220 objects degraded (4.317%), 5 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:09:07.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:07 smithi122 ceph-mon[73495]: osdmap e407: 12 total, 11 up, 9 in 2024-03-20T09:09:08.245 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:09:08.257 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:09:08.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:08.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:09:08.277 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:09:08.363 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:09:08.392 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:08.518 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:09:08.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:08.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:09:08.626 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:09:08.655 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:08.758 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:09:08.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:08.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:09:08.863 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:09:08.887 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 8, 10, 11] out_osds: [9, 3, 4] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11] 2024-03-20T09:09:08.887 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:09:08.887 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-20T09:09:08.887 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-20T09:09:08.895 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:09.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:09 smithi115 ceph-mon[72476]: pgmap v983: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 4.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 107 op/s; 8514/196581 objects degraded (4.331%); 29315/196581 objects misplaced (14.912%); 13 MiB/s, 215 objects/s recovering 2024-03-20T09:09:09.446 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:09:09.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:09 smithi122 ceph-mon[73495]: pgmap v983: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 4.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 107 op/s; 8514/196581 objects degraded (4.331%); 29315/196581 objects misplaced (14.912%); 13 MiB/s, 215 objects/s recovering 2024-03-20T09:09:09.448 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:09 smithi026 ceph-mon[68309]: pgmap v983: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 4.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 107 op/s; 8514/196581 objects degraded (4.331%); 29315/196581 objects misplaced (14.912%); 13 MiB/s, 215 objects/s recovering 2024-03-20T09:09:09.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:09.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:09:09.472 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:09:09.485 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:09:09.485 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:09:09.504 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:09:09.537 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:09:09.551 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:09:09.561 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:09:09.585 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:09.589 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:09:09.608 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:09:09.614 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:09:09.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:09.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:09:09.644 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:09:09.676 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:09:09.696 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:09:09.717 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:09:09.728 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:09:09.739 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:09:09.739 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:09:09.739 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:09:09.739 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:09:09.739 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:09:09.739 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:09:09.739 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-20T09:09:09.739 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:09:09.739 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:09:09.739 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:09:09.740 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:09:09.740 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:09:09.740 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:09:09.756 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:09.776 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:09:09.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:09.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:09:09.880 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:09:09.906 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:09.926 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:09:09.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:09.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:09:10.030 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:09:10.071 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:10.232 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:09:10.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:10.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:09:10.341 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:09:10.367 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:10 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2728608251' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:09:10.367 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:10 smithi115 ceph-mon[72476]: 3.16 deep-scrub starts 2024-03-20T09:09:10.369 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:10.390 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:09:10.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:10.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:09:10.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:10 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2728608251' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:09:10.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:10 smithi122 ceph-mon[73495]: 3.16 deep-scrub starts 2024-03-20T09:09:10.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:10 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2728608251' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:09:10.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:10 smithi026 ceph-mon[68309]: 3.16 deep-scrub starts 2024-03-20T09:09:10.494 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:09:10.530 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:10.941 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:09:10.941 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:09:11.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:11 smithi026 ceph-mon[68309]: pgmap v984: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 4.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 102 op/s; 8498/196185 objects degraded (4.332%); 28701/196185 objects misplaced (14.630%); 13 MiB/s, 202 objects/s recovering 2024-03-20T09:09:11.233 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:09:11.233 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-03-20T09:09:11.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-03-20T09:09:11.233 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:09:11.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.233 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:30429 3.b 3:d5ec34f7:::benchmark_data_smithi122_119671_object30428:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e221)", 2024-03-20T09:09:11.233 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:16.489633+0000", 2024-03-20T09:09:11.233 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 354.73941858900002, 2024-03-20T09:09:11.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.212205006, 2024-03-20T09:09:11.233 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:09:11.233 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:11.233 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:09:11.233 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:09:11.234 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:09:11.234 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 30429 2024-03-20T09:09:11.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.234 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:09:11.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:09:11.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.489633+0000", 2024-03-20T09:09:11.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:09:11.234 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.489633+0000", 2024-03-20T09:09:11.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:09:11.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.489633+0000", 2024-03-20T09:09:11.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1200000000000003e-07 2024-03-20T09:09:11.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:09:11.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.489702+0000", 2024-03-20T09:09:11.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8428000000000003e-05 2024-03-20T09:09:11.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:09:11.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.489710+0000", 2024-03-20T09:09:11.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2819999999999996e-06 2024-03-20T09:09:11.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:09:11.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.489715+0000", 2024-03-20T09:09:11.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9339999999999997e-06 2024-03-20T09:09:11.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:09:11.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.489757+0000", 2024-03-20T09:09:11.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2243999999999997e-05 2024-03-20T09:09:11.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:09:11.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.489838+0000", 2024-03-20T09:09:11.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0592000000000004e-05 2024-03-20T09:09:11.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4,11", 2024-03-20T09:09:11.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.489915+0000", 2024-03-20T09:09:11.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7416000000000006e-05 2024-03-20T09:09:11.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:09:11.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.490837+0000", 2024-03-20T09:09:11.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00092125900000000001 2024-03-20T09:09:11.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.491272+0000", 2024-03-20T09:09:11.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00043533999999999999 2024-03-20T09:09:11.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.701698+0000", 2024-03-20T09:09:11.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.21042634099999999 2024-03-20T09:09:11.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:09:11.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.701773+0000", 2024-03-20T09:09:11.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5204999999999996e-05 2024-03-20T09:09:11.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:09:11.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.701838+0000", 2024-03-20T09:09:11.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4352999999999999e-05 2024-03-20T09:09:11.239 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.239 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:11.239 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.239 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:30438 3.b 3:dc354d7f:::benchmark_data_smithi122_119671_object30437:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-03-20T09:09:11.239 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:16.492459+0000", 2024-03-20T09:09:11.239 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 354.73659192399998, 2024-03-20T09:09:11.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.20947844500000001, 2024-03-20T09:09:11.239 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:09:11.239 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:11.240 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:09:11.240 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:09:11.240 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:09:11.240 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 30438 2024-03-20T09:09:11.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.240 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:09:11.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:09:11.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.492459+0000", 2024-03-20T09:09:11.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:09:11.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.492459+0000", 2024-03-20T09:09:11.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:09:11.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.492460+0000", 2024-03-20T09:09:11.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8199999999999999e-07 2024-03-20T09:09:11.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:09:11.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.492529+0000", 2024-03-20T09:09:11.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8862999999999994e-05 2024-03-20T09:09:11.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:09:11.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.492536+0000", 2024-03-20T09:09:11.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.883e-06 2024-03-20T09:09:11.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:09:11.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.492540+0000", 2024-03-20T09:09:11.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5120000000000002e-06 2024-03-20T09:09:11.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:09:11.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.492575+0000", 2024-03-20T09:09:11.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4805000000000003e-05 2024-03-20T09:09:11.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:09:11.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.492655+0000", 2024-03-20T09:09:11.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0192999999999996e-05 2024-03-20T09:09:11.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4,11", 2024-03-20T09:09:11.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.492742+0000", 2024-03-20T09:09:11.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6110000000000001e-05 2024-03-20T09:09:11.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:09:11.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.493593+0000", 2024-03-20T09:09:11.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085177500000000004 2024-03-20T09:09:11.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.494017+0000", 2024-03-20T09:09:11.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00042358000000000001 2024-03-20T09:09:11.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.701874+0000", 2024-03-20T09:09:11.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.20785695400000001 2024-03-20T09:09:11.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:09:11.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.701911+0000", 2024-03-20T09:09:11.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6754000000000001e-05 2024-03-20T09:09:11.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:09:11.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.701938+0000", 2024-03-20T09:09:11.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7334e-05 2024-03-20T09:09:11.244 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.245 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:11.245 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.245 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:30444 3.b 3:d164033f:::benchmark_data_smithi122_119671_object30443:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-03-20T09:09:11.245 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:16.495200+0000", 2024-03-20T09:09:11.245 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 354.73385168900001, 2024-03-20T09:09:11.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.207676944, 2024-03-20T09:09:11.245 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:09:11.245 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:11.245 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:09:11.245 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:09:11.245 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:09:11.245 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 30444 2024-03-20T09:09:11.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.246 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:09:11.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:09:11.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.495200+0000", 2024-03-20T09:09:11.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:09:11.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.495200+0000", 2024-03-20T09:09:11.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:09:11.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.495201+0000", 2024-03-20T09:09:11.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1500000000000003e-07 2024-03-20T09:09:11.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:09:11.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.495264+0000", 2024-03-20T09:09:11.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3510999999999996e-05 2024-03-20T09:09:11.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:09:11.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.495278+0000", 2024-03-20T09:09:11.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4165000000000001e-05 2024-03-20T09:09:11.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:09:11.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.495286+0000", 2024-03-20T09:09:11.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1629999999999996e-06 2024-03-20T09:09:11.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:09:11.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.495341+0000", 2024-03-20T09:09:11.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4719999999999998e-05 2024-03-20T09:09:11.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:09:11.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.495479+0000", 2024-03-20T09:09:11.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000137727 2024-03-20T09:09:11.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4,11", 2024-03-20T09:09:11.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.495633+0000", 2024-03-20T09:09:11.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000154454 2024-03-20T09:09:11.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:09:11.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.496410+0000", 2024-03-20T09:09:11.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00077638800000000003 2024-03-20T09:09:11.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.496761+0000", 2024-03-20T09:09:11.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00035142499999999999 2024-03-20T09:09:11.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.702818+0000", 2024-03-20T09:09:11.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.20605689399999999 2024-03-20T09:09:11.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:09:11.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.702852+0000", 2024-03-20T09:09:11.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4322000000000001e-05 2024-03-20T09:09:11.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:09:11.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.702877+0000", 2024-03-20T09:09:11.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4260000000000002e-05 2024-03-20T09:09:11.250 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.251 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:11.251 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.251 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:30527 3.b 3:d31efdba:::benchmark_data_smithi122_119671_object30526:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-03-20T09:09:11.251 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:16.530092+0000", 2024-03-20T09:09:11.251 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 354.69895903499997, 2024-03-20T09:09:11.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.17472232500000001, 2024-03-20T09:09:11.251 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:09:11.251 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:11.251 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:09:11.251 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:09:11.251 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:09:11.251 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 30527 2024-03-20T09:09:11.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.252 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:09:11.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:09:11.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.530092+0000", 2024-03-20T09:09:11.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:09:11.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.530092+0000", 2024-03-20T09:09:11.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:09:11.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.530094+0000", 2024-03-20T09:09:11.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4589999999999999e-06 2024-03-20T09:09:11.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:09:11.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.530179+0000", 2024-03-20T09:09:11.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5641000000000002e-05 2024-03-20T09:09:11.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:09:11.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.530197+0000", 2024-03-20T09:09:11.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7696000000000001e-05 2024-03-20T09:09:11.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:09:11.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.530206+0000", 2024-03-20T09:09:11.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4030000000000007e-06 2024-03-20T09:09:11.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:09:11.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.530302+0000", 2024-03-20T09:09:11.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.6055000000000001e-05 2024-03-20T09:09:11.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:09:11.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.530477+0000", 2024-03-20T09:09:11.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00017556599999999999 2024-03-20T09:09:11.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4,11", 2024-03-20T09:09:11.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.530620+0000", 2024-03-20T09:09:11.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014269900000000001 2024-03-20T09:09:11.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:09:11.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.531881+0000", 2024-03-20T09:09:11.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012607759999999999 2024-03-20T09:09:11.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.532966+0000", 2024-03-20T09:09:11.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001085351 2024-03-20T09:09:11.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.704735+0000", 2024-03-20T09:09:11.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.17176888700000001 2024-03-20T09:09:11.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:09:11.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.704779+0000", 2024-03-20T09:09:11.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3692000000000001e-05 2024-03-20T09:09:11.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:09:11.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.704815+0000", 2024-03-20T09:09:11.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6100000000000003e-05 2024-03-20T09:09:11.257 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.257 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:11.257 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.257 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:30549 3.b 3:d61cac04:::benchmark_data_smithi122_119671_object30548:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-03-20T09:09:11.257 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:16.538518+0000", 2024-03-20T09:09:11.257 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 354.69053378299998, 2024-03-20T09:09:11.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.16638256200000001, 2024-03-20T09:09:11.257 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:09:11.257 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:11.257 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:09:11.257 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:09:11.257 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:09:11.258 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 30549 2024-03-20T09:09:11.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.258 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:09:11.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:09:11.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.538518+0000", 2024-03-20T09:09:11.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:09:11.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.538518+0000", 2024-03-20T09:09:11.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:09:11.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.538518+0000", 2024-03-20T09:09:11.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5700000000000002e-07 2024-03-20T09:09:11.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:09:11.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.538631+0000", 2024-03-20T09:09:11.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011322700000000001 2024-03-20T09:09:11.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:09:11.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.538645+0000", 2024-03-20T09:09:11.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3059e-05 2024-03-20T09:09:11.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:09:11.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.538653+0000", 2024-03-20T09:09:11.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3499999999999997e-06 2024-03-20T09:09:11.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:09:11.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.538707+0000", 2024-03-20T09:09:11.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3696999999999998e-05 2024-03-20T09:09:11.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:09:11.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.538823+0000", 2024-03-20T09:09:11.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000116165 2024-03-20T09:09:11.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4,11", 2024-03-20T09:09:11.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.538910+0000", 2024-03-20T09:09:11.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6883000000000001e-05 2024-03-20T09:09:11.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:09:11.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.539889+0000", 2024-03-20T09:09:11.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0009789899999999999 2024-03-20T09:09:11.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.540509+0000", 2024-03-20T09:09:11.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00062069200000000005 2024-03-20T09:09:11.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.704845+0000", 2024-03-20T09:09:11.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.164336026 2024-03-20T09:09:11.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:09:11.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.704879+0000", 2024-03-20T09:09:11.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3269999999999998e-05 2024-03-20T09:09:11.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:09:11.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.704900+0000", 2024-03-20T09:09:11.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1546e-05 2024-03-20T09:09:11.262 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.263 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:11.263 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.263 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:30565 3.b 3:d3cb9be9:::benchmark_data_smithi122_119671_object30564:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-03-20T09:09:11.263 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:16.545408+0000", 2024-03-20T09:09:11.263 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 354.68364345800001, 2024-03-20T09:09:11.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.16015575800000001, 2024-03-20T09:09:11.263 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:09:11.263 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:11.263 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:09:11.263 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:09:11.263 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:09:11.264 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 30565 2024-03-20T09:09:11.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.264 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:09:11.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:09:11.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.545408+0000", 2024-03-20T09:09:11.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:09:11.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.545408+0000", 2024-03-20T09:09:11.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:09:11.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.545409+0000", 2024-03-20T09:09:11.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2610000000000001e-06 2024-03-20T09:09:11.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:09:11.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.545454+0000", 2024-03-20T09:09:11.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4922999999999997e-05 2024-03-20T09:09:11.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:09:11.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.545467+0000", 2024-03-20T09:09:11.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3254e-05 2024-03-20T09:09:11.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:09:11.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.545475+0000", 2024-03-20T09:09:11.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7540000000000006e-06 2024-03-20T09:09:11.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:09:11.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.545528+0000", 2024-03-20T09:09:11.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2506000000000002e-05 2024-03-20T09:09:11.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:09:11.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.545608+0000", 2024-03-20T09:09:11.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0765000000000003e-05 2024-03-20T09:09:11.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4,11", 2024-03-20T09:09:11.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.545698+0000", 2024-03-20T09:09:11.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0085000000000002e-05 2024-03-20T09:09:11.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:09:11.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.546723+0000", 2024-03-20T09:09:11.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010241460000000001 2024-03-20T09:09:11.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.548465+0000", 2024-03-20T09:09:11.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0017427129999999999 2024-03-20T09:09:11.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.705482+0000", 2024-03-20T09:09:11.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.15701707000000001 2024-03-20T09:09:11.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:09:11.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.705525+0000", 2024-03-20T09:09:11.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3007999999999999e-05 2024-03-20T09:09:11.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:09:11.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.705564+0000", 2024-03-20T09:09:11.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8272999999999998e-05 2024-03-20T09:09:11.269 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.269 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:11.269 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.269 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:30586 3.b 3:d43d94e4:::benchmark_data_smithi122_119671_object30585:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-03-20T09:09:11.269 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:16.554791+0000", 2024-03-20T09:09:11.269 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 354.67426041800002, 2024-03-20T09:09:11.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.15206215300000001, 2024-03-20T09:09:11.269 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:09:11.269 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:11.270 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:09:11.270 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:09:11.270 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:09:11.270 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 30586 2024-03-20T09:09:11.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.270 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:09:11.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:09:11.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.554791+0000", 2024-03-20T09:09:11.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:09:11.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.554791+0000", 2024-03-20T09:09:11.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:09:11.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.554792+0000", 2024-03-20T09:09:11.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1149999999999999e-06 2024-03-20T09:09:11.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:09:11.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.554895+0000", 2024-03-20T09:09:11.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010300400000000001 2024-03-20T09:09:11.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:09:11.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.554915+0000", 2024-03-20T09:09:11.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9919999999999999e-05 2024-03-20T09:09:11.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:09:11.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.554921+0000", 2024-03-20T09:09:11.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9739999999999999e-06 2024-03-20T09:09:11.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:09:11.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.554983+0000", 2024-03-20T09:09:11.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1684000000000001e-05 2024-03-20T09:09:11.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:09:11.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.555136+0000", 2024-03-20T09:09:11.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015296799999999999 2024-03-20T09:09:11.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4,11", 2024-03-20T09:09:11.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.555300+0000", 2024-03-20T09:09:11.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00016395299999999999 2024-03-20T09:09:11.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:09:11.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.556589+0000", 2024-03-20T09:09:11.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001289267 2024-03-20T09:09:11.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.556789+0000", 2024-03-20T09:09:11.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00020009499999999999 2024-03-20T09:09:11.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.706783+0000", 2024-03-20T09:09:11.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.149993919 2024-03-20T09:09:11.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:09:11.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.706827+0000", 2024-03-20T09:09:11.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4379999999999999e-05 2024-03-20T09:09:11.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:09:11.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.706853+0000", 2024-03-20T09:09:11.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5874e-05 2024-03-20T09:09:11.275 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.275 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:11.275 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.275 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:30636 3.b 3:d036ddde:::benchmark_data_smithi122_119671_object30635:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-03-20T09:09:11.275 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:16.575534+0000", 2024-03-20T09:09:11.275 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 354.65351712799998, 2024-03-20T09:09:11.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.13147451499999999, 2024-03-20T09:09:11.276 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:09:11.276 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:11.276 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:09:11.276 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:09:11.276 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:09:11.276 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 30636 2024-03-20T09:09:11.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.276 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:09:11.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:09:11.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.575534+0000", 2024-03-20T09:09:11.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:09:11.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.575534+0000", 2024-03-20T09:09:11.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:09:11.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.575536+0000", 2024-03-20T09:09:11.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.26e-06 2024-03-20T09:09:11.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:09:11.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.575579+0000", 2024-03-20T09:09:11.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3917e-05 2024-03-20T09:09:11.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:09:11.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.575594+0000", 2024-03-20T09:09:11.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4491000000000001e-05 2024-03-20T09:09:11.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:09:11.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.575602+0000", 2024-03-20T09:09:11.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6839999999999996e-06 2024-03-20T09:09:11.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:09:11.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.575664+0000", 2024-03-20T09:09:11.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1917999999999999e-05 2024-03-20T09:09:11.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:09:11.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.575758+0000", 2024-03-20T09:09:11.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4738e-05 2024-03-20T09:09:11.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4,11", 2024-03-20T09:09:11.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.575860+0000", 2024-03-20T09:09:11.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101409 2024-03-20T09:09:11.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:09:11.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.576877+0000", 2024-03-20T09:09:11.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010176650000000001 2024-03-20T09:09:11.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.577980+0000", 2024-03-20T09:09:11.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011023300000000001 2024-03-20T09:09:11.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.706905+0000", 2024-03-20T09:09:11.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.12892548600000001 2024-03-20T09:09:11.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:09:11.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.706964+0000", 2024-03-20T09:09:11.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8996000000000002e-05 2024-03-20T09:09:11.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:09:11.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.707009+0000", 2024-03-20T09:09:11.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4620999999999999e-05 2024-03-20T09:09:11.281 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.281 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:11.281 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.281 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:30640 3.b 3:d2fa5035:::benchmark_data_smithi122_119671_object30639:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-03-20T09:09:11.281 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:16.578122+0000", 2024-03-20T09:09:11.281 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 354.65092958700001, 2024-03-20T09:09:11.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.12898076, 2024-03-20T09:09:11.281 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:09:11.282 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:11.282 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:09:11.282 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:09:11.282 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:09:11.282 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 30640 2024-03-20T09:09:11.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.282 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:09:11.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:09:11.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.578122+0000", 2024-03-20T09:09:11.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:09:11.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.578122+0000", 2024-03-20T09:09:11.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:09:11.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.578123+0000", 2024-03-20T09:09:11.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6499999999999998e-07 2024-03-20T09:09:11.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:09:11.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.578175+0000", 2024-03-20T09:09:11.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2266999999999999e-05 2024-03-20T09:09:11.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:09:11.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.578185+0000", 2024-03-20T09:09:11.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0169e-05 2024-03-20T09:09:11.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:09:11.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.578193+0000", 2024-03-20T09:09:11.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7989999999999994e-06 2024-03-20T09:09:11.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:09:11.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.578234+0000", 2024-03-20T09:09:11.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1443000000000003e-05 2024-03-20T09:09:11.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:09:11.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.578319+0000", 2024-03-20T09:09:11.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5243999999999999e-05 2024-03-20T09:09:11.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4,11", 2024-03-20T09:09:11.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.578417+0000", 2024-03-20T09:09:11.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7907999999999998e-05 2024-03-20T09:09:11.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:09:11.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.579305+0000", 2024-03-20T09:09:11.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00088804400000000005 2024-03-20T09:09:11.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.580466+0000", 2024-03-20T09:09:11.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011601960000000001 2024-03-20T09:09:11.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.707051+0000", 2024-03-20T09:09:11.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.12658492700000001 2024-03-20T09:09:11.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:09:11.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.707081+0000", 2024-03-20T09:09:11.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0311000000000002e-05 2024-03-20T09:09:11.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:09:11.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.707103+0000", 2024-03-20T09:09:11.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1687000000000001e-05 2024-03-20T09:09:11.287 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.287 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:11.287 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.287 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:30643 3.b 3:d35a2676:::benchmark_data_smithi122_119671_object30642:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-03-20T09:09:11.287 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:16.579229+0000", 2024-03-20T09:09:11.287 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 354.64982234899998, 2024-03-20T09:09:11.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.12799962300000001, 2024-03-20T09:09:11.288 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:09:11.288 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:11.288 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:09:11.288 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:09:11.288 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:09:11.288 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 30643 2024-03-20T09:09:11.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.288 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:09:11.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:09:11.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.579229+0000", 2024-03-20T09:09:11.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:09:11.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.579229+0000", 2024-03-20T09:09:11.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:09:11.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.579230+0000", 2024-03-20T09:09:11.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6899999999999997e-07 2024-03-20T09:09:11.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:09:11.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.579325+0000", 2024-03-20T09:09:11.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5246000000000003e-05 2024-03-20T09:09:11.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:09:11.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.579332+0000", 2024-03-20T09:09:11.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0550000000000002e-06 2024-03-20T09:09:11.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:09:11.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.579337+0000", 2024-03-20T09:09:11.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7169999999999999e-06 2024-03-20T09:09:11.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:09:11.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.579377+0000", 2024-03-20T09:09:11.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0370000000000001e-05 2024-03-20T09:09:11.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:09:11.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.579553+0000", 2024-03-20T09:09:11.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000176346 2024-03-20T09:09:11.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4,11", 2024-03-20T09:09:11.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.579694+0000", 2024-03-20T09:09:11.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014056600000000001 2024-03-20T09:09:11.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:09:11.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.580630+0000", 2024-03-20T09:09:11.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093630999999999999 2024-03-20T09:09:11.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.580936+0000", 2024-03-20T09:09:11.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00030534099999999999 2024-03-20T09:09:11.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.707152+0000", 2024-03-20T09:09:11.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.12621663899999999 2024-03-20T09:09:11.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:09:11.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.707206+0000", 2024-03-20T09:09:11.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3359999999999997e-05 2024-03-20T09:09:11.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:09:11.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.707229+0000", 2024-03-20T09:09:11.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3104000000000001e-05 2024-03-20T09:09:11.293 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.293 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:11.293 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.293 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:30690 3.b 3:d7915426:::benchmark_data_smithi122_119671_object30689:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-03-20T09:09:11.293 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:16.602716+0000", 2024-03-20T09:09:11.293 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 354.62633495900002, 2024-03-20T09:09:11.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.104781474, 2024-03-20T09:09:11.293 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:09:11.293 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:11.293 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:09:11.293 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:09:11.294 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:09:11.294 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 30690 2024-03-20T09:09:11.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.294 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:09:11.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:09:11.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.602716+0000", 2024-03-20T09:09:11.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:09:11.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.602716+0000", 2024-03-20T09:09:11.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:09:11.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.602717+0000", 2024-03-20T09:09:11.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0219999999999999e-06 2024-03-20T09:09:11.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:09:11.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.602764+0000", 2024-03-20T09:09:11.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6391999999999999e-05 2024-03-20T09:09:11.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:09:11.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.602776+0000", 2024-03-20T09:09:11.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2594e-05 2024-03-20T09:09:11.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:09:11.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.602786+0000", 2024-03-20T09:09:11.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3810000000000005e-06 2024-03-20T09:09:11.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:09:11.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.602848+0000", 2024-03-20T09:09:11.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2668000000000004e-05 2024-03-20T09:09:11.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:09:11.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.602957+0000", 2024-03-20T09:09:11.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010862900000000001 2024-03-20T09:09:11.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4,11", 2024-03-20T09:09:11.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.603056+0000", 2024-03-20T09:09:11.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9216999999999996e-05 2024-03-20T09:09:11.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:09:11.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.604055+0000", 2024-03-20T09:09:11.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00099879999999999999 2024-03-20T09:09:11.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.604482+0000", 2024-03-20T09:09:11.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00042717500000000002 2024-03-20T09:09:11.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.707443+0000", 2024-03-20T09:09:11.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.10296060999999999 2024-03-20T09:09:11.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:09:11.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.707474+0000", 2024-03-20T09:09:11.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1343000000000001e-05 2024-03-20T09:09:11.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:09:11.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.707498+0000", 2024-03-20T09:09:11.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3643e-05 2024-03-20T09:09:11.299 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.299 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:11.299 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.299 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:30712 3.b 3:d7cf6d7e:::benchmark_data_smithi122_119671_object30711:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-03-20T09:09:11.299 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:16.614438+0000", 2024-03-20T09:09:11.299 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 354.61461322299999, 2024-03-20T09:09:11.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.094310621999999997, 2024-03-20T09:09:11.299 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:09:11.299 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:11.299 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:09:11.299 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:09:11.300 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:09:11.300 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 30712 2024-03-20T09:09:11.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.300 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:09:11.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:09:11.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.614438+0000", 2024-03-20T09:09:11.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:09:11.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.614438+0000", 2024-03-20T09:09:11.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.301 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:09:11.301 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.614439+0000", 2024-03-20T09:09:11.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2899999999999999e-06 2024-03-20T09:09:11.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.301 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:09:11.301 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.614522+0000", 2024-03-20T09:09:11.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2223000000000002e-05 2024-03-20T09:09:11.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.301 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:09:11.301 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.614537+0000", 2024-03-20T09:09:11.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5721000000000001e-05 2024-03-20T09:09:11.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.301 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:09:11.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.614544+0000", 2024-03-20T09:09:11.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6189999999999996e-06 2024-03-20T09:09:11.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:09:11.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.614604+0000", 2024-03-20T09:09:11.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9953e-05 2024-03-20T09:09:11.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:09:11.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.614743+0000", 2024-03-20T09:09:11.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013937500000000001 2024-03-20T09:09:11.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4,11", 2024-03-20T09:09:11.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.614913+0000", 2024-03-20T09:09:11.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00016981900000000001 2024-03-20T09:09:11.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:09:11.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.615940+0000", 2024-03-20T09:09:11.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001027158 2024-03-20T09:09:11.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.616433+0000", 2024-03-20T09:09:11.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049219800000000001 2024-03-20T09:09:11.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.708697+0000", 2024-03-20T09:09:11.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.092264070000000004 2024-03-20T09:09:11.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:09:11.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.708727+0000", 2024-03-20T09:09:11.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0759000000000001e-05 2024-03-20T09:09:11.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:09:11.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.708749+0000", 2024-03-20T09:09:11.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1437000000000002e-05 2024-03-20T09:09:11.304 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.304 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:11.304 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.305 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:30715 3.b 3:d93cbf5b:::benchmark_data_smithi122_119671_object30714:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-03-20T09:09:11.305 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:16.616497+0000", 2024-03-20T09:09:11.305 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 354.61255423799997, 2024-03-20T09:09:11.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.092351005999999999, 2024-03-20T09:09:11.305 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:09:11.305 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:11.305 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:09:11.305 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:09:11.305 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:09:11.305 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 30715 2024-03-20T09:09:11.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.305 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:09:11.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:09:11.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.616497+0000", 2024-03-20T09:09:11.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:09:11.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.616497+0000", 2024-03-20T09:09:11.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:09:11.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.616498+0000", 2024-03-20T09:09:11.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1500000000000004e-07 2024-03-20T09:09:11.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:09:11.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.616610+0000", 2024-03-20T09:09:11.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011233799999999999 2024-03-20T09:09:11.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:09:11.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.616624+0000", 2024-03-20T09:09:11.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.362e-05 2024-03-20T09:09:11.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:09:11.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.616633+0000", 2024-03-20T09:09:11.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7460000000000006e-06 2024-03-20T09:09:11.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:09:11.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.616684+0000", 2024-03-20T09:09:11.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1511999999999997e-05 2024-03-20T09:09:11.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:09:11.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.616833+0000", 2024-03-20T09:09:11.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000149285 2024-03-20T09:09:11.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4,11", 2024-03-20T09:09:11.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.616996+0000", 2024-03-20T09:09:11.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00016253599999999999 2024-03-20T09:09:11.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.309 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:09:11.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.617651+0000", 2024-03-20T09:09:11.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00065477299999999999 2024-03-20T09:09:11.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.309 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.618484+0000", 2024-03-20T09:09:11.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00083334200000000005 2024-03-20T09:09:11.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.309 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.708779+0000", 2024-03-20T09:09:11.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.090295185 2024-03-20T09:09:11.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.309 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:09:11.310 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.708824+0000", 2024-03-20T09:09:11.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4545999999999997e-05 2024-03-20T09:09:11.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:09:11.310 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.708848+0000", 2024-03-20T09:09:11.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4408000000000001e-05 2024-03-20T09:09:11.310 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.310 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:11.310 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.310 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:30718 3.b 3:de862fe7:::benchmark_data_smithi122_119671_object30717:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-03-20T09:09:11.310 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:16.618483+0000", 2024-03-20T09:09:11.310 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 354.61056886300003, 2024-03-20T09:09:11.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.095133886000000001, 2024-03-20T09:09:11.311 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:09:11.311 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:11.311 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:09:11.311 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:09:11.311 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:09:11.311 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 30718 2024-03-20T09:09:11.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.311 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:09:11.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:09:11.311 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.618483+0000", 2024-03-20T09:09:11.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:09:11.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.618483+0000", 2024-03-20T09:09:11.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:09:11.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.618483+0000", 2024-03-20T09:09:11.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0900000000000001e-07 2024-03-20T09:09:11.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:09:11.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.618569+0000", 2024-03-20T09:09:11.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5456000000000003e-05 2024-03-20T09:09:11.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:09:11.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.618579+0000", 2024-03-20T09:09:11.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0601000000000001e-05 2024-03-20T09:09:11.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:09:11.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.618585+0000", 2024-03-20T09:09:11.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1940000000000003e-06 2024-03-20T09:09:11.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:09:11.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.618629+0000", 2024-03-20T09:09:11.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3427000000000003e-05 2024-03-20T09:09:11.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:09:11.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.618708+0000", 2024-03-20T09:09:11.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8668e-05 2024-03-20T09:09:11.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4,11", 2024-03-20T09:09:11.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.618808+0000", 2024-03-20T09:09:11.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000100568 2024-03-20T09:09:11.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:09:11.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.619580+0000", 2024-03-20T09:09:11.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00077188300000000001 2024-03-20T09:09:11.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.620007+0000", 2024-03-20T09:09:11.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00042724100000000002 2024-03-20T09:09:11.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.315 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.713541+0000", 2024-03-20T09:09:11.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.093534095999999997 2024-03-20T09:09:11.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:09:11.315 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.713591+0000", 2024-03-20T09:09:11.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9981000000000001e-05 2024-03-20T09:09:11.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:09:11.315 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:16.713616+0000", 2024-03-20T09:09:11.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5062000000000002e-05 2024-03-20T09:09:11.316 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.316 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:11.316 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.316 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:58474 3.17 3:efc4c91d:::benchmark_data_smithi122_119671_object58473:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e234)", 2024-03-20T09:09:11.316 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:32.876662+0000", 2024-03-20T09:09:11.316 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 338.35238920500001, 2024-03-20T09:09:11.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2026188529999999, 2024-03-20T09:09:11.316 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:09:11.316 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:09:11.316 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:09:11.316 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:09:11.316 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:09:11.316 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58474 2024-03-20T09:09:11.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.317 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:09:11.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:09:11.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.876662+0000", 2024-03-20T09:09:11.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:09:11.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.876662+0000", 2024-03-20T09:09:11.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:09:11.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.876664+0000", 2024-03-20T09:09:11.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5770000000000001e-06 2024-03-20T09:09:11.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:09:11.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.876953+0000", 2024-03-20T09:09:11.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00028972500000000001 2024-03-20T09:09:11.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:09:11.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.876967+0000", 2024-03-20T09:09:11.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.329e-05 2024-03-20T09:09:11.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:09:11.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.876978+0000", 2024-03-20T09:09:11.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1483e-05 2024-03-20T09:09:11.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:09:11.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877044+0000", 2024-03-20T09:09:11.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6137999999999996e-05 2024-03-20T09:09:11.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:09:11.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877055+0000", 2024-03-20T09:09:11.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0224000000000001e-05 2024-03-20T09:09:11.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:09:11.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.066151+0000", 2024-03-20T09:09:11.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.189096543 2024-03-20T09:09:11.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for degraded object", 2024-03-20T09:09:11.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.066932+0000", 2024-03-20T09:09:11.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078113200000000003 2024-03-20T09:09:11.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:09:11.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.079192+0000", 2024-03-20T09:09:11.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012259617 2024-03-20T09:09:11.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:09:11.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.079281+0000", 2024-03-20T09:09:11.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9123999999999996e-05 2024-03-20T09:09:11.321 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.321 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:11.321 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.321 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:69227 3.17 3:ea388972:::benchmark_data_smithi122_119671_object69226:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-03-20T09:09:11.321 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:50.859576+0000", 2024-03-20T09:09:11.321 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 320.36947585799999, 2024-03-20T09:09:11.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.219397547, 2024-03-20T09:09:11.321 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:09:11.321 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:11.321 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:09:11.321 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:09:11.321 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:09:11.322 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 69227 2024-03-20T09:09:11.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.322 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:09:11.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:09:11.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.859576+0000", 2024-03-20T09:09:11.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:09:11.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.859576+0000", 2024-03-20T09:09:11.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:09:11.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.859577+0000", 2024-03-20T09:09:11.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.455e-06 2024-03-20T09:09:11.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:09:11.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.859617+0000", 2024-03-20T09:09:11.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9734999999999998e-05 2024-03-20T09:09:11.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:09:11.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.859629+0000", 2024-03-20T09:09:11.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2424e-05 2024-03-20T09:09:11.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:09:11.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.859637+0000", 2024-03-20T09:09:11.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5399999999999998e-06 2024-03-20T09:09:11.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:09:11.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.859699+0000", 2024-03-20T09:09:11.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2549e-05 2024-03-20T09:09:11.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:09:11.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.859909+0000", 2024-03-20T09:09:11.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00020935999999999999 2024-03-20T09:09:11.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 1,5,6", 2024-03-20T09:09:11.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.860099+0000", 2024-03-20T09:09:11.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000190075 2024-03-20T09:09:11.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:09:11.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.861657+0000", 2024-03-20T09:09:11.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015586179999999999 2024-03-20T09:09:11.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.861707+0000", 2024-03-20T09:09:11.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9619e-05 2024-03-20T09:09:11.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.865102+0000", 2024-03-20T09:09:11.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0033946110000000001 2024-03-20T09:09:11.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:51.078161+0000", 2024-03-20T09:09:11.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.21305988100000001 2024-03-20T09:09:11.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:09:11.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:51.078244+0000", 2024-03-20T09:09:11.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2939e-05 2024-03-20T09:09:11.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:09:11.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:51.078973+0000", 2024-03-20T09:09:11.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00072874099999999998 2024-03-20T09:09:11.327 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.327 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:11.327 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.327 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:69231 3.17 3:eb7da1ac:::benchmark_data_smithi122_119671_object69230:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-03-20T09:09:11.327 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:50.862152+0000", 2024-03-20T09:09:11.327 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 320.366899037, 2024-03-20T09:09:11.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.21958736100000001, 2024-03-20T09:09:11.328 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:09:11.328 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:11.328 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:09:11.328 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:09:11.328 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:09:11.328 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 69231 2024-03-20T09:09:11.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.328 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:09:11.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.328 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:09:11.328 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.862152+0000", 2024-03-20T09:09:11.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.328 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:09:11.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.862152+0000", 2024-03-20T09:09:11.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:09:11.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.862154+0000", 2024-03-20T09:09:11.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2020000000000001e-06 2024-03-20T09:09:11.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:09:11.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.862210+0000", 2024-03-20T09:09:11.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6196999999999997e-05 2024-03-20T09:09:11.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:09:11.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.862226+0000", 2024-03-20T09:09:11.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6480000000000001e-05 2024-03-20T09:09:11.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:09:11.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.862235+0000", 2024-03-20T09:09:11.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3969999999999994e-06 2024-03-20T09:09:11.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:09:11.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.862292+0000", 2024-03-20T09:09:11.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7692000000000003e-05 2024-03-20T09:09:11.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:09:11.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.862457+0000", 2024-03-20T09:09:11.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00016452799999999999 2024-03-20T09:09:11.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 1,5,6", 2024-03-20T09:09:11.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.862664+0000", 2024-03-20T09:09:11.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000207593 2024-03-20T09:09:11.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:09:11.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.863864+0000", 2024-03-20T09:09:11.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011997170000000001 2024-03-20T09:09:11.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.866188+0000", 2024-03-20T09:09:11.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023236160000000001 2024-03-20T09:09:11.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.870836+0000", 2024-03-20T09:09:11.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0046480030000000004 2024-03-20T09:09:11.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:51.080960+0000", 2024-03-20T09:09:11.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.21012456400000001 2024-03-20T09:09:11.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:09:11.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:51.081035+0000", 2024-03-20T09:09:11.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4549999999999996e-05 2024-03-20T09:09:11.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:09:11.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:51.081740+0000", 2024-03-20T09:09:11.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00070482200000000002 2024-03-20T09:09:11.333 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.333 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:11.333 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.334 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:69279 3.17 3:e8652f58:::benchmark_data_smithi122_119671_object69278:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-03-20T09:09:11.334 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:50.915567+0000", 2024-03-20T09:09:11.334 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 320.31348411200003, 2024-03-20T09:09:11.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.16679550100000001, 2024-03-20T09:09:11.334 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:09:11.334 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:11.334 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:09:11.334 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:09:11.334 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:09:11.334 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 69279 2024-03-20T09:09:11.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.335 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:09:11.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:09:11.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.915567+0000", 2024-03-20T09:09:11.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:09:11.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.915567+0000", 2024-03-20T09:09:11.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:09:11.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.915569+0000", 2024-03-20T09:09:11.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5319999999999999e-06 2024-03-20T09:09:11.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:09:11.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.915642+0000", 2024-03-20T09:09:11.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2857999999999999e-05 2024-03-20T09:09:11.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:09:11.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.915657+0000", 2024-03-20T09:09:11.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5319999999999999e-05 2024-03-20T09:09:11.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.337 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:09:11.337 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.915665+0000", 2024-03-20T09:09:11.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7479999999999993e-06 2024-03-20T09:09:11.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.337 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:09:11.337 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.915719+0000", 2024-03-20T09:09:11.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3866000000000002e-05 2024-03-20T09:09:11.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.337 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:09:11.337 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.915892+0000", 2024-03-20T09:09:11.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000173263 2024-03-20T09:09:11.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 1,5,6", 2024-03-20T09:09:11.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.916059+0000", 2024-03-20T09:09:11.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00016670300000000001 2024-03-20T09:09:11.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:09:11.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.920656+0000", 2024-03-20T09:09:11.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0045975870000000002 2024-03-20T09:09:11.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.923937+0000", 2024-03-20T09:09:11.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0032809240000000002 2024-03-20T09:09:11.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.923991+0000", 2024-03-20T09:09:11.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3885000000000003e-05 2024-03-20T09:09:11.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:51.081815+0000", 2024-03-20T09:09:11.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.157824095 2024-03-20T09:09:11.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:09:11.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:51.081873+0000", 2024-03-20T09:09:11.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7850999999999999e-05 2024-03-20T09:09:11.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:09:11.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:51.082363+0000", 2024-03-20T09:09:11.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00048986899999999998 2024-03-20T09:09:11.340 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.340 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:11.340 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.340 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:69323 3.17 3:eec96e78:::benchmark_data_smithi122_119671_object69322:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e245)", 2024-03-20T09:09:11.341 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:50.962444+0000", 2024-03-20T09:09:11.341 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 320.26660724999999, 2024-03-20T09:09:11.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.12040653599999999, 2024-03-20T09:09:11.341 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:09:11.341 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:11.341 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:09:11.341 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:09:11.341 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:09:11.341 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 69323 2024-03-20T09:09:11.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.341 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:09:11.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:09:11.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.962444+0000", 2024-03-20T09:09:11.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:09:11.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.962444+0000", 2024-03-20T09:09:11.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:09:11.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.962446+0000", 2024-03-20T09:09:11.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5889999999999999e-06 2024-03-20T09:09:11.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:09:11.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.962563+0000", 2024-03-20T09:09:11.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000117689 2024-03-20T09:09:11.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:09:11.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.962575+0000", 2024-03-20T09:09:11.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1365e-05 2024-03-20T09:09:11.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:09:11.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.962582+0000", 2024-03-20T09:09:11.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3300000000000001e-06 2024-03-20T09:09:11.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:09:11.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.962640+0000", 2024-03-20T09:09:11.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7880000000000001e-05 2024-03-20T09:09:11.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:09:11.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.962812+0000", 2024-03-20T09:09:11.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00017243200000000001 2024-03-20T09:09:11.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 1,5,6", 2024-03-20T09:09:11.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.962984+0000", 2024-03-20T09:09:11.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000171535 2024-03-20T09:09:11.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:09:11.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.964206+0000", 2024-03-20T09:09:11.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012215819999999999 2024-03-20T09:09:11.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.965820+0000", 2024-03-20T09:09:11.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001614487 2024-03-20T09:09:11.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:50.967503+0000", 2024-03-20T09:09:11.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016829690000000001 2024-03-20T09:09:11.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:51.082426+0000", 2024-03-20T09:09:11.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.11492305999999999 2024-03-20T09:09:11.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:09:11.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:51.082486+0000", 2024-03-20T09:09:11.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9691999999999997e-05 2024-03-20T09:09:11.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:09:11.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:51.082851+0000", 2024-03-20T09:09:11.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00036492600000000001 2024-03-20T09:09:11.346 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.346 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:11.346 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.347 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:105741 3.b 3:d76aa0e8:::benchmark_data_smithi122_119671_object5959:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e287)", 2024-03-20T09:09:11.347 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:05:04.237211+0000", 2024-03-20T09:09:11.347 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 246.99184028400001, 2024-03-20T09:09:11.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.089842662000000004, 2024-03-20T09:09:11.347 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:09:11.347 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:11.347 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:09:11.347 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:09:11.347 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:09:11.347 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105741 2024-03-20T09:09:11.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.347 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:09:11.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:09:11.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:04.237211+0000", 2024-03-20T09:09:11.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.348 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:09:11.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:04.237211+0000", 2024-03-20T09:09:11.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:09:11.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.348 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:09:11.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:04.237212+0000", 2024-03-20T09:09:11.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2200000000000003e-07 2024-03-20T09:09:11.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.348 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:09:11.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:04.237216+0000", 2024-03-20T09:09:11.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6909999999999999e-06 2024-03-20T09:09:11.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:09:11.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:04.237227+0000", 2024-03-20T09:09:11.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1647999999999999e-05 2024-03-20T09:09:11.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:09:11.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:04.237234+0000", 2024-03-20T09:09:11.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8510000000000001e-06 2024-03-20T09:09:11.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:09:11.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:04.237310+0000", 2024-03-20T09:09:11.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5710000000000005e-05 2024-03-20T09:09:11.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:09:11.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:04.237392+0000", 2024-03-20T09:09:11.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2154e-05 2024-03-20T09:09:11.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4,11", 2024-03-20T09:09:11.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:04.237462+0000", 2024-03-20T09:09:11.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9839999999999995e-05 2024-03-20T09:09:11.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:09:11.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:04.239679+0000", 2024-03-20T09:09:11.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0022174460000000001 2024-03-20T09:09:11.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:04.240135+0000", 2024-03-20T09:09:11.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00045618299999999998 2024-03-20T09:09:11.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:11.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:04.326978+0000", 2024-03-20T09:09:11.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.086842266000000001 2024-03-20T09:09:11.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:09:11.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:04.327016+0000", 2024-03-20T09:09:11.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8300999999999998e-05 2024-03-20T09:09:11.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:11.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:11.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:09:11.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:04.327054+0000", 2024-03-20T09:09:11.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7750000000000003e-05 2024-03-20T09:09:11.352 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.352 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:11.352 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.352 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:11.352 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:11.352 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:09:11.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:11 smithi115 ceph-mon[72476]: pgmap v984: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 4.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 102 op/s; 8498/196185 objects degraded (4.332%); 28701/196185 objects misplaced (14.630%); 13 MiB/s, 202 objects/s recovering 2024-03-20T09:09:11.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:11 smithi122 ceph-mon[73495]: pgmap v984: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 4.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 102 op/s; 8498/196185 objects degraded (4.332%); 28701/196185 objects misplaced (14.630%); 13 MiB/s, 202 objects/s recovering 2024-03-20T09:09:11.983 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:09:11.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:11.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:09:12.089 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:09:12.130 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:12.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:12 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8498/196185 objects degraded (4.332%), 5 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:09:12.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:12 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8498/196185 objects degraded (4.332%), 5 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:09:12.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:12 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8498/196185 objects degraded (4.332%), 5 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:09:12.942 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:09:13.147 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:13 smithi026 ceph-mon[68309]: pgmap v985: 57 pgs: 1 active+clean+remapped+scrubbing+deep, 12 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 4.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 136 op/s; 8448/194883 objects degraded (4.335%); 28390/194883 objects misplaced (14.568%); 11 MiB/s, 176 objects/s recovering 2024-03-20T09:09:13.187 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-20T09:09:13.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:13 smithi115 ceph-mon[72476]: pgmap v985: 57 pgs: 1 active+clean+remapped+scrubbing+deep, 12 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 4.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 136 op/s; 8448/194883 objects degraded (4.335%); 28390/194883 objects misplaced (14.568%); 11 MiB/s, 176 objects/s recovering 2024-03-20T09:09:13.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:13 smithi122 ceph-mon[73495]: pgmap v985: 57 pgs: 1 active+clean+remapped+scrubbing+deep, 12 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 4.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 136 op/s; 8448/194883 objects degraded (4.335%); 28390/194883 objects misplaced (14.568%); 11 MiB/s, 176 objects/s recovering 2024-03-20T09:09:13.558 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:09:14.085 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:09:14.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:14.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:09:14.191 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:09:14.217 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:14.236 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:09:14.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:14.238 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:09:14.339 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:09:14.377 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:14.942 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:09:14.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:14.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:09:14.954 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:09:15.048 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:09:15.079 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:15.099 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:09:15.099 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:09:15.099 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 407, 2024-03-20T09:09:15.099 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "852fbb68-e696-11ee-95c9-87774f69a715", 2024-03-20T09:09:15.099 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-03-20T08:48:35.281088+0000", 2024-03-20T09:09:15.100 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-03-20T09:09:06.969357+0000", 2024-03-20T09:09:15.100 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2024-03-20T09:04:33.695633+0000", 2024-03-20T09:09:15.100 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2024-03-20T09:07:48.551904+0000", 2024-03-20T09:09:15.100 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-20T09:09:15.100 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5799936, 2024-03-20T09:09:15.100 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2024-03-20T09:09:15.100 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2024-03-20T09:09:15.100 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2024-03-20T09:09:15.100 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2024-03-20T09:09:15.100 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2024-03-20T09:09:15.100 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:09:15.100 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 26, 2024-03-20T09:09:15.100 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2024-03-20T09:09:15.100 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-20T09:09:15.101 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-20T09:09:15.101 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2024-03-20T09:09:15.101 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 3, 2024-03-20T09:09:15.101 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 12, 2024-03-20T09:09:15.101 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "luminous", 2024-03-20T09:09:15.101 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "jewel", 2024-03-20T09:09:15.101 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "squid", 2024-03-20T09:09:15.101 INFO:teuthology.orchestra.run.smithi026.stdout: "allow_crimson": false, 2024-03-20T09:09:15.101 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2024-03-20T09:09:15.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.101 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2024-03-20T09:09:15.101 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": ".mgr", 2024-03-20T09:09:15.101 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-03-20T08:52:17.442324+0000", 2024-03-20T09:09:15.101 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-03-20T09:09:15.102 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-03-20T09:09:15.102 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-03-20T09:09:15.102 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 3, 2024-03-20T09:09:15.102 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 2, 2024-03-20T09:09:15.102 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-03-20T09:09:15.102 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-03-20T09:09:15.102 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-03-20T09:09:15.102 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-03-20T09:09:15.102 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-20T09:09:15.102 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-03-20T09:09:15.102 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-03-20T09:09:15.102 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 31, 2024-03-20T09:09:15.102 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 31, 2024-03-20T09:09:15.103 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 31, 2024-03-20T09:09:15.103 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 31, 2024-03-20T09:09:15.103 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 31, 2024-03-20T09:09:15.103 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-03-20T09:09:15.103 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-03-20T09:09:15.103 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-03-20T09:09:15.103 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-03-20T09:09:15.103 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-03-20T09:09:15.103 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-03-20T09:09:15.104 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-03-20T09:09:15.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.104 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "360", 2024-03-20T09:09:15.104 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-03-20T09:09:15.104 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-20T09:09:15.104 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "358", 2024-03-20T09:09:15.104 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-03-20T09:09:15.104 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-03-20T09:09:15.104 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-03-20T09:09:15.104 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-03-20T09:09:15.104 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-03-20T09:09:15.104 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-03-20T09:09:15.104 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-03-20T09:09:15.104 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-03-20T09:09:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-03-20T09:09:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-03-20T09:09:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-03-20T09:09:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-03-20T09:09:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-03-20T09:09:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-03-20T09:09:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-03-20T09:09:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-20T09:09:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-20T09:09:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-20T09:09:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-03-20T09:09:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-03-20T09:09:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-03-20T09:09:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-03-20T09:09:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-03-20T09:09:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-03-20T09:09:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-03-20T09:09:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-03-20T09:09:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-03-20T09:09:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-03-20T09:09:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-03-20T09:09:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-03-20T09:09:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-03-20T09:09:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-03-20T09:09:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-03-20T09:09:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-03-20T09:09:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2024-03-20T09:09:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_max": 32, 2024-03-20T09:09:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2024-03-20T09:09:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-03-20T09:09:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": {} 2024-03-20T09:09:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: "read_balance": { 2024-03-20T09:09:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: "score_type": "Fair distribution", 2024-03-20T09:09:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: "score_acting": 2.7000000476837158, 2024-03-20T09:09:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: "score_stable": 2.7000000476837158, 2024-03-20T09:09:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: "optimal_score": 0.67000001668930054, 2024-03-20T09:09:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_acting": 1.809999942779541, 2024-03-20T09:09:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_stable": 1.809999942779541, 2024-03-20T09:09:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity_weighted": 0.92150616645812988, 2024-03-20T09:09:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity": 0.92000001668930054, 2024-03-20T09:09:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-03-20T09:09:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 3, 2024-03-20T09:09:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "unique_pool_1", 2024-03-20T09:09:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-03-20T09:02:48.375175+0000", 2024-03-20T09:09:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-03-20T09:09:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-03-20T09:09:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-03-20T09:09:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 3, 2024-03-20T09:09:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 2, 2024-03-20T09:09:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-03-20T09:09:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-03-20T09:09:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-03-20T09:09:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-03-20T09:09:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-20T09:09:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-03-20T09:09:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-03-20T09:09:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 26, 2024-03-20T09:09:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 26, 2024-03-20T09:09:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 26, 2024-03-20T09:09:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 26, 2024-03-20T09:09:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 26, 2024-03-20T09:09:15.110 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-03-20T09:09:15.110 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-03-20T09:09:15.110 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-03-20T09:09:15.110 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-03-20T09:09:15.110 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-03-20T09:09:15.110 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-03-20T09:09:15.110 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-03-20T09:09:15.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.110 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "233", 2024-03-20T09:09:15.110 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-03-20T09:09:15.110 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-20T09:09:15.110 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "231", 2024-03-20T09:09:15.110 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-03-20T09:09:15.111 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-03-20T09:09:15.111 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-03-20T09:09:15.111 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-03-20T09:09:15.111 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-03-20T09:09:15.111 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-03-20T09:09:15.111 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-03-20T09:09:15.111 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-03-20T09:09:15.111 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-03-20T09:09:15.111 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-03-20T09:09:15.111 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-03-20T09:09:15.111 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-03-20T09:09:15.111 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-03-20T09:09:15.111 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-03-20T09:09:15.111 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-03-20T09:09:15.112 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-20T09:09:15.112 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-20T09:09:15.112 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-20T09:09:15.112 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-03-20T09:09:15.112 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-03-20T09:09:15.112 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-03-20T09:09:15.112 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-03-20T09:09:15.112 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-03-20T09:09:15.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.112 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-03-20T09:09:15.112 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-03-20T09:09:15.112 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-03-20T09:09:15.112 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-03-20T09:09:15.112 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-03-20T09:09:15.113 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-03-20T09:09:15.113 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-03-20T09:09:15.113 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-03-20T09:09:15.113 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-03-20T09:09:15.113 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-03-20T09:09:15.113 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-03-20T09:09:15.113 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2024-03-20T09:09:15.113 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-03-20T09:09:15.113 INFO:teuthology.orchestra.run.smithi026.stdout: "rados": {} 2024-03-20T09:09:15.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.113 INFO:teuthology.orchestra.run.smithi026.stdout: "read_balance": { 2024-03-20T09:09:15.113 INFO:teuthology.orchestra.run.smithi026.stdout: "score_type": "Fair distribution", 2024-03-20T09:09:15.113 INFO:teuthology.orchestra.run.smithi026.stdout: "score_acting": 3.2100000381469727, 2024-03-20T09:09:15.113 INFO:teuthology.orchestra.run.smithi026.stdout: "score_stable": 3.2100000381469727, 2024-03-20T09:09:15.113 INFO:teuthology.orchestra.run.smithi026.stdout: "optimal_score": 0.67000001668930054, 2024-03-20T09:09:15.114 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_acting": 2.1500000953674316, 2024-03-20T09:09:15.114 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_stable": 2.1500000953674316, 2024-03-20T09:09:15.114 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity_weighted": 0.92150616645812988, 2024-03-20T09:09:15.114 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity": 0.92000001668930054, 2024-03-20T09:09:15.114 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-03-20T09:09:15.114 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.114 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.114 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:09:15.114 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:09:15.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.114 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-03-20T09:09:15.114 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "ad3fd9c0-316f-4fe5-a63b-d480087c1729", 2024-03-20T09:09:15.114 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:09:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:09:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:09:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:09:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:09:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:09:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 8, 2024-03-20T09:09:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 385, 2024-03-20T09:09:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:09:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:09:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:09:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.115 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2024-03-20T09:09:15.116 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:09:15.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.116 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.116 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2024-03-20T09:09:15.116 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:09:15.116 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.116 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.116 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:09:15.116 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.116 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.116 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2024-03-20T09:09:15.117 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:09:15.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.117 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.117 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2024-03-20T09:09:15.117 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:09:15.117 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.117 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.117 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:09:15.117 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.117 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.118 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2024-03-20T09:09:15.118 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:09:15.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.118 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.118 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2024-03-20T09:09:15.118 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:09:15.118 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.118 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.118 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:09:15.118 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2024-03-20T09:09:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:09:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2024-03-20T09:09:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:09:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/2139805293", 2024-03-20T09:09:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/2139805293", 2024-03-20T09:09:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/2139805293", 2024-03-20T09:09:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/2139805293", 2024-03-20T09:09:15.119 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:09:15.120 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:09:15.120 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:09:15.120 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.120 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-03-20T09:09:15.120 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e85b94f5-2954-47a1-9c20-096f2606eb04", 2024-03-20T09:09:15.120 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:09:15.120 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:09:15.120 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0.97100830078125, 2024-03-20T09:09:15.120 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:09:15.120 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:09:15.120 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:09:15.120 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2024-03-20T09:09:15.120 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 386, 2024-03-20T09:09:15.120 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:09:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:09:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:09:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2024-03-20T09:09:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:09:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2024-03-20T09:09:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:09:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.122 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:09:15.122 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.122 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.122 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2024-03-20T09:09:15.122 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:09:15.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.122 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.122 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2024-03-20T09:09:15.122 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:09:15.122 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.122 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.122 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:09:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2024-03-20T09:09:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:09:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2024-03-20T09:09:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:09:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:09:15.123 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.124 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.124 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2024-03-20T09:09:15.124 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:09:15.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.124 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.124 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2024-03-20T09:09:15.124 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:09:15.124 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.124 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.124 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/3853355134", 2024-03-20T09:09:15.124 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/3853355134", 2024-03-20T09:09:15.124 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/3853355134", 2024-03-20T09:09:15.125 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/3853355134", 2024-03-20T09:09:15.125 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:09:15.125 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:09:15.125 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:09:15.125 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.125 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-03-20T09:09:15.125 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "dd2b10f0-741c-47a4-b224-ac6bfe2c14ef", 2024-03-20T09:09:15.125 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:09:15.125 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:09:15.125 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:09:15.125 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:09:15.125 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:09:15.125 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:09:15.126 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 18, 2024-03-20T09:09:15.126 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 383, 2024-03-20T09:09:15.126 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:09:15.126 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:09:15.126 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:09:15.126 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.126 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.126 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2024-03-20T09:09:15.126 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:09:15.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.126 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.127 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2024-03-20T09:09:15.127 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:09:15.127 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.127 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.127 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:09:15.127 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.127 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.127 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2024-03-20T09:09:15.127 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:09:15.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.127 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.127 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2024-03-20T09:09:15.128 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:09:15.128 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.128 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.128 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:09:15.128 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.128 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.128 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2024-03-20T09:09:15.128 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:09:15.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.128 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.128 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2024-03-20T09:09:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:09:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:09:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2024-03-20T09:09:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:09:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2024-03-20T09:09:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:09:15.129 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.130 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.130 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/3993357967", 2024-03-20T09:09:15.130 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/3993357967", 2024-03-20T09:09:15.130 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/3993357967", 2024-03-20T09:09:15.130 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/3993357967", 2024-03-20T09:09:15.130 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:09:15.130 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:09:15.130 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:09:15.130 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.130 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-03-20T09:09:15.131 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1", 2024-03-20T09:09:15.131 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:09:15.131 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 0, 2024-03-20T09:09:15.131 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-03-20T09:09:15.131 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:09:15.131 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:09:15.131 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:09:15.131 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 26, 2024-03-20T09:09:15.131 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 274, 2024-03-20T09:09:15.131 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:09:15.131 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:09:15.131 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:09:15.131 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.132 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.132 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2024-03-20T09:09:15.132 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:09:15.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.132 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.132 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2024-03-20T09:09:15.132 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:09:15.132 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.132 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.132 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:09:15.132 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.132 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.133 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2024-03-20T09:09:15.133 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:09:15.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.133 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.133 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2024-03-20T09:09:15.133 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:09:15.133 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.133 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.133 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:09:15.133 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.133 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.134 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2024-03-20T09:09:15.134 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:09:15.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.134 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.134 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2024-03-20T09:09:15.134 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:09:15.134 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.134 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.134 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:09:15.134 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.135 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.135 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2024-03-20T09:09:15.135 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:09:15.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.135 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.135 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2024-03-20T09:09:15.135 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:09:15.135 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.135 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.135 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/4110225509", 2024-03-20T09:09:15.135 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/4110225509", 2024-03-20T09:09:15.135 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/4110225509", 2024-03-20T09:09:15.136 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/4110225509", 2024-03-20T09:09:15.136 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:09:15.136 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:09:15.136 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:09:15.136 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.136 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-03-20T09:09:15.136 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "f14e34d5-9829-42a3-9b25-31a0976c5ef7", 2024-03-20T09:09:15.136 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:09:15.136 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 0, 2024-03-20T09:09:15.136 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-03-20T09:09:15.136 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:09:15.137 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:09:15.137 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:09:15.137 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 32, 2024-03-20T09:09:15.137 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 366, 2024-03-20T09:09:15.137 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:09:15.137 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:09:15.137 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:09:15.137 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.137 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.137 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6800", 2024-03-20T09:09:15.137 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:09:15.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.138 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.138 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6801", 2024-03-20T09:09:15.138 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:09:15.138 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.138 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.138 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:09:15.138 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.138 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.138 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6802", 2024-03-20T09:09:15.138 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:09:15.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.139 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.139 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6803", 2024-03-20T09:09:15.139 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:09:15.139 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.139 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.139 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:09:15.139 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.139 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.139 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6806", 2024-03-20T09:09:15.139 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:09:15.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.139 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.140 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6807", 2024-03-20T09:09:15.140 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:09:15.140 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.140 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.140 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:09:15.140 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.140 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.140 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6804", 2024-03-20T09:09:15.140 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:09:15.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.141 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.141 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6805", 2024-03-20T09:09:15.141 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:09:15.141 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.141 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.141 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6801/477923058", 2024-03-20T09:09:15.141 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6803/477923058", 2024-03-20T09:09:15.141 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6807/477923058", 2024-03-20T09:09:15.141 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6805/477923058", 2024-03-20T09:09:15.141 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:09:15.141 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:09:15.141 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:09:15.142 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.142 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-03-20T09:09:15.142 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "cb431b6a-540d-4318-ae35-d379e1df0185", 2024-03-20T09:09:15.142 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 0, 2024-03-20T09:09:15.142 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:09:15.142 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:09:15.142 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:09:15.142 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:09:15.142 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:09:15.142 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 41, 2024-03-20T09:09:15.142 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 265, 2024-03-20T09:09:15.143 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 274, 2024-03-20T09:09:15.143 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:09:15.143 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:09:15.143 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.143 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.143 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6808", 2024-03-20T09:09:15.143 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:09:15.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.143 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.143 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6809", 2024-03-20T09:09:15.143 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:09:15.143 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.144 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.144 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:09:15.144 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.144 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.144 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6810", 2024-03-20T09:09:15.144 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:09:15.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.144 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.144 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6811", 2024-03-20T09:09:15.144 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:09:15.144 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.145 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.145 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:09:15.145 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.145 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.145 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6814", 2024-03-20T09:09:15.145 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:09:15.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.145 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.145 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6815", 2024-03-20T09:09:15.145 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:09:15.145 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.145 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.146 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:09:15.146 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.146 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.146 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6812", 2024-03-20T09:09:15.146 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:09:15.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.146 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.146 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6813", 2024-03-20T09:09:15.146 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1339268003 2024-03-20T09:09:15.146 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.147 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.147 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6809/1339268003", 2024-03-20T09:09:15.147 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6811/1339268003", 2024-03-20T09:09:15.147 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6815/1339268003", 2024-03-20T09:09:15.147 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6813/1339268003", 2024-03-20T09:09:15.147 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:09:15.147 INFO:teuthology.orchestra.run.smithi026.stdout: "exists" 2024-03-20T09:09:15.147 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.147 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2024-03-20T09:09:15.147 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "753ed830-e764-4ff3-9d9b-87e7fa748b81", 2024-03-20T09:09:15.147 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:09:15.148 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:09:15.148 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0.842376708984375, 2024-03-20T09:09:15.148 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:09:15.148 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:09:15.148 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:09:15.148 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 51, 2024-03-20T09:09:15.148 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 404, 2024-03-20T09:09:15.148 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:09:15.148 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:09:15.148 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:09:15.148 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.148 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.148 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6816", 2024-03-20T09:09:15.149 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:09:15.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.149 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.149 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6817", 2024-03-20T09:09:15.149 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:09:15.149 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.149 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.149 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:09:15.149 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.149 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.149 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6818", 2024-03-20T09:09:15.149 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:09:15.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.150 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.150 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6819", 2024-03-20T09:09:15.150 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:09:15.150 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.150 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.150 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:09:15.150 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.150 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.150 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6822", 2024-03-20T09:09:15.151 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:09:15.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.151 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.151 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6823", 2024-03-20T09:09:15.151 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:09:15.151 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.151 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.151 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:09:15.151 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.151 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.151 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6820", 2024-03-20T09:09:15.152 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:09:15.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.152 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.152 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6821", 2024-03-20T09:09:15.152 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:09:15.152 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.152 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.152 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6817/3278301843", 2024-03-20T09:09:15.152 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6819/3278301843", 2024-03-20T09:09:15.152 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6823/3278301843", 2024-03-20T09:09:15.152 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6821/3278301843", 2024-03-20T09:09:15.152 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:09:15.153 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:09:15.153 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:09:15.153 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.153 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2024-03-20T09:09:15.153 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "55bcbd37-7a6f-4ab1-b5a9-91a951444d86", 2024-03-20T09:09:15.153 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:09:15.153 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:09:15.153 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0.4264678955078125, 2024-03-20T09:09:15.153 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 0.058074951171875, 2024-03-20T09:09:15.153 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:09:15.153 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:09:15.153 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 60, 2024-03-20T09:09:15.154 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 274, 2024-03-20T09:09:15.154 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:09:15.154 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:09:15.154 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:09:15.154 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.154 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.154 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6824", 2024-03-20T09:09:15.154 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:09:15.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.154 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.155 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6825", 2024-03-20T09:09:15.155 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:09:15.155 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.155 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.155 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:09:15.155 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.155 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.155 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6826", 2024-03-20T09:09:15.155 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:09:15.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.155 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.156 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6827", 2024-03-20T09:09:15.156 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:09:15.156 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.157 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.157 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:09:15.157 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.157 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.158 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6830", 2024-03-20T09:09:15.158 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:09:15.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.158 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.158 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6831", 2024-03-20T09:09:15.158 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:09:15.158 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.158 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.158 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:09:15.158 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.159 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.159 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6828", 2024-03-20T09:09:15.159 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:09:15.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.159 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.159 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6829", 2024-03-20T09:09:15.159 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:09:15.159 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.159 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.159 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6825/3152424001", 2024-03-20T09:09:15.159 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6827/3152424001", 2024-03-20T09:09:15.159 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6831/3152424001", 2024-03-20T09:09:15.159 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6829/3152424001", 2024-03-20T09:09:15.160 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:09:15.160 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:09:15.160 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:09:15.160 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.160 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 8, 2024-03-20T09:09:15.160 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "bc86248e-6bd0-4369-b8a2-e593bc2a610d", 2024-03-20T09:09:15.160 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:09:15.160 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:09:15.160 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:09:15.160 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:09:15.160 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:09:15.160 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:09:15.160 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 70, 2024-03-20T09:09:15.161 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 380, 2024-03-20T09:09:15.161 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:09:15.161 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:09:15.161 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:09:15.161 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.161 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.161 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6800", 2024-03-20T09:09:15.161 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:09:15.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.161 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.161 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6801", 2024-03-20T09:09:15.161 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:09:15.162 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.162 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.162 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:09:15.162 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.162 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.162 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6802", 2024-03-20T09:09:15.162 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:09:15.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.162 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.162 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6803", 2024-03-20T09:09:15.163 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:09:15.163 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.163 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.163 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:09:15.163 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.163 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.163 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6806", 2024-03-20T09:09:15.163 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:09:15.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.163 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.163 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6807", 2024-03-20T09:09:15.164 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:09:15.164 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.164 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.164 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:09:15.164 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.164 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.164 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6804", 2024-03-20T09:09:15.164 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:09:15.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.164 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.164 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6805", 2024-03-20T09:09:15.165 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:09:15.165 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.165 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.165 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6801/1309426740", 2024-03-20T09:09:15.165 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6803/1309426740", 2024-03-20T09:09:15.165 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6807/1309426740", 2024-03-20T09:09:15.165 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6805/1309426740", 2024-03-20T09:09:15.165 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:09:15.165 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:09:15.165 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:09:15.165 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.166 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 9, 2024-03-20T09:09:15.166 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b20a4aa-032a-4c09-ac75-153c554424e7", 2024-03-20T09:09:15.166 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:09:15.166 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 0, 2024-03-20T09:09:15.166 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-03-20T09:09:15.166 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:09:15.166 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:09:15.166 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:09:15.166 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 80, 2024-03-20T09:09:15.166 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 257, 2024-03-20T09:09:15.166 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:09:15.166 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:09:15.166 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:09:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6808", 2024-03-20T09:09:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:09:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6809", 2024-03-20T09:09:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:09:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.167 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:09:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6810", 2024-03-20T09:09:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:09:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6811", 2024-03-20T09:09:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:09:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.168 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:09:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6814", 2024-03-20T09:09:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:09:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6815", 2024-03-20T09:09:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:09:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.169 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:09:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6812", 2024-03-20T09:09:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:09:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6813", 2024-03-20T09:09:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:09:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.170 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6809/2953210076", 2024-03-20T09:09:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6811/2953210076", 2024-03-20T09:09:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6815/2953210076", 2024-03-20T09:09:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6813/2953210076", 2024-03-20T09:09:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:09:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:09:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:09:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 10, 2024-03-20T09:09:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e62c1fb3-5dd1-4d0d-997c-d20633d19361", 2024-03-20T09:09:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:09:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:09:15.171 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:09:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:09:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:09:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:09:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 89, 2024-03-20T09:09:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 386, 2024-03-20T09:09:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:09:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:09:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:09:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6816", 2024-03-20T09:09:15.172 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:09:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6817", 2024-03-20T09:09:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:09:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:09:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6818", 2024-03-20T09:09:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:09:15.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6819", 2024-03-20T09:09:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:09:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:09:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6822", 2024-03-20T09:09:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:09:15.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6823", 2024-03-20T09:09:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:09:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:09:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6820", 2024-03-20T09:09:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:09:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.175 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6821", 2024-03-20T09:09:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:09:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6817/409394340", 2024-03-20T09:09:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6819/409394340", 2024-03-20T09:09:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6823/409394340", 2024-03-20T09:09:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6821/409394340", 2024-03-20T09:09:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:09:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:09:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:09:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.176 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 11, 2024-03-20T09:09:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b83994d5-e5ef-48b0-9758-c26843cec3b6", 2024-03-20T09:09:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:09:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:09:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0.6123046875, 2024-03-20T09:09:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:09:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:09:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:09:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 96, 2024-03-20T09:09:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 385, 2024-03-20T09:09:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:09:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:09:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:09:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.177 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6824", 2024-03-20T09:09:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:09:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6825", 2024-03-20T09:09:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:09:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:09:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6826", 2024-03-20T09:09:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:09:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6827", 2024-03-20T09:09:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:09:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:09:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.179 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6830", 2024-03-20T09:09:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:09:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6831", 2024-03-20T09:09:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:09:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:09:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:09:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:09:15.180 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6828", 2024-03-20T09:09:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:09:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:09:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6829", 2024-03-20T09:09:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:09:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6825/2704069354", 2024-03-20T09:09:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6827/2704069354", 2024-03-20T09:09:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6831/2704069354", 2024-03-20T09:09:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6829/2704069354", 2024-03-20T09:09:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:09:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:09:15.181 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:09:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:09:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2024-03-20T09:09:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-03-20T09:09:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:09:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:09:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:09:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:09:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:09:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:08:04.860551+0000", 2024-03-20T09:09:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:09:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-03-20T09:09:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:09:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:09:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:09:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:09:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:09:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:08:57.275643+0000", 2024-03-20T09:09:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:09:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-03-20T09:09:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:09:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:09:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:09:15.183 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:09:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:09:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:08:18.494972+0000", 2024-03-20T09:09:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:09:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-03-20T09:09:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:09:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:09:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:09:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:09:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 35063, 2024-03-20T09:09:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:09:02.812475+0000", 2024-03-20T09:09:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:09:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-03-20T09:09:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:09:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:09:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:09:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:09:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 65536, 2024-03-20T09:09:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:08:30.077943+0000", 2024-03-20T09:09:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:09:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-03-20T09:09:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:09:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:09:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:09:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:09:15.185 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:09:15.186 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:03:53.609882+0000", 2024-03-20T09:09:15.186 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 273 2024-03-20T09:09:15.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.186 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2024-03-20T09:09:15.186 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:09:15.186 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:09:15.186 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:09:15.186 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:09:15.186 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:09:15.186 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:08:14.465561+0000", 2024-03-20T09:09:15.186 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:09:15.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.186 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2024-03-20T09:09:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:09:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:09:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:09:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:09:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:09:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:07:57.050958+0000", 2024-03-20T09:09:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:09:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 8, 2024-03-20T09:09:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:09:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:09:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:09:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:09:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:09:15.187 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:08:24.980388+0000", 2024-03-20T09:09:15.188 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:09:15.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.188 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 9, 2024-03-20T09:09:15.188 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:09:15.188 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:09:15.188 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:09:15.188 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:09:15.188 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 34736, 2024-03-20T09:09:15.188 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:07:52.842622+0000", 2024-03-20T09:09:15.188 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:09:15.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.188 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 10, 2024-03-20T09:09:15.188 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:09:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:09:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:09:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:09:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:09:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:08:19.771648+0000", 2024-03-20T09:09:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:09:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 11, 2024-03-20T09:09:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:09:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:09:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:09:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:09:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:09:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:08:40.300610+0000", 2024-03-20T09:09:15.189 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:09:15.190 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.190 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:09:15.190 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [], 2024-03-20T09:09:15.190 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2024-03-20T09:09:15.190 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_primaries": [], 2024-03-20T09:09:15.190 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [ 2024-03-20T09:09:15.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.190 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "1.8", 2024-03-20T09:09:15.190 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:09:15.190 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-03-20T09:09:15.190 INFO:teuthology.orchestra.run.smithi026.stdout: 8, 2024-03-20T09:09:15.190 INFO:teuthology.orchestra.run.smithi026.stdout: 4 2024-03-20T09:09:15.190 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "1.b", 2024-03-20T09:09:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:09:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-03-20T09:09:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-03-20T09:09:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: 11 2024-03-20T09:09:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "1.f", 2024-03-20T09:09:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:09:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-03-20T09:09:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: 8, 2024-03-20T09:09:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: 4 2024-03-20T09:09:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.192 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "1.15", 2024-03-20T09:09:15.192 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:09:15.192 INFO:teuthology.orchestra.run.smithi026.stdout: 8, 2024-03-20T09:09:15.192 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-03-20T09:09:15.192 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-03-20T09:09:15.192 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.192 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "1.17", 2024-03-20T09:09:15.192 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:09:15.192 INFO:teuthology.orchestra.run.smithi026.stdout: 8, 2024-03-20T09:09:15.192 INFO:teuthology.orchestra.run.smithi026.stdout: 10, 2024-03-20T09:09:15.192 INFO:teuthology.orchestra.run.smithi026.stdout: 6 2024-03-20T09:09:15.192 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "1.18", 2024-03-20T09:09:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:09:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-03-20T09:09:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-03-20T09:09:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-03-20T09:09:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.2", 2024-03-20T09:09:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:09:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-03-20T09:09:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-03-20T09:09:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: 4 2024-03-20T09:09:15.193 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.6", 2024-03-20T09:09:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:09:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-03-20T09:09:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-03-20T09:09:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: 4 2024-03-20T09:09:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.8", 2024-03-20T09:09:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:09:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: 11, 2024-03-20T09:09:15.194 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-03-20T09:09:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: 4 2024-03-20T09:09:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.b", 2024-03-20T09:09:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:09:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-03-20T09:09:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: 11 2024-03-20T09:09:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.c", 2024-03-20T09:09:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:09:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-03-20T09:09:15.195 INFO:teuthology.orchestra.run.smithi026.stdout: 10, 2024-03-20T09:09:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: 3 2024-03-20T09:09:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.d", 2024-03-20T09:09:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:09:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-03-20T09:09:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: 7, 2024-03-20T09:09:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: 9 2024-03-20T09:09:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.196 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.e", 2024-03-20T09:09:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:09:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-03-20T09:09:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: 10 2024-03-20T09:09:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.10", 2024-03-20T09:09:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:09:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-03-20T09:09:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-03-20T09:09:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: 2 2024-03-20T09:09:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.15", 2024-03-20T09:09:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:09:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: 11, 2024-03-20T09:09:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: 0 2024-03-20T09:09:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.16", 2024-03-20T09:09:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:09:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-03-20T09:09:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-03-20T09:09:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: 7 2024-03-20T09:09:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:09:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "3.17", 2024-03-20T09:09:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:09:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-03-20T09:09:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-03-20T09:09:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: 6 2024-03-20T09:09:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:09:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:09:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2024-03-20T09:09:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: "blocklist": { 2024-03-20T09:09:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/4291404191": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:09:15.199 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/642012566": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:09:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6801/1181459999": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:09:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/3984513952": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:09:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6800/1654945498": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:09:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6800/1181459999": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:09:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/137770981": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:09:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/1451237762": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:09:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/3938386631": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:09:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6801/1654945498": "2024-03-21T08:49:04.596754+0000" 2024-03-20T09:09:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "range_blocklist": {}, 2024-03-20T09:09:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2024-03-20T09:09:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2024-03-20T09:09:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2024-03-20T09:09:15.200 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2024-03-20T09:09:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2024-03-20T09:09:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2024-03-20T09:09:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2024-03-20T09:09:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:09:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2024-03-20T09:09:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2024-03-20T09:09:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2024-03-20T09:09:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2024-03-20T09:09:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {}, 2024-03-20T09:09:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": { 2024-03-20T09:09:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_enabled": false, 2024-03-20T09:09:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_bucket_count": 0, 2024-03-20T09:09:15.201 INFO:teuthology.orchestra.run.smithi026.stdout: "degraded_stretch_mode": 0, 2024-03-20T09:09:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: "recovering_stretch_mode": 0, 2024-03-20T09:09:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_bucket": 0 2024-03-20T09:09:15.202 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:09:15.202 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:09:15.206 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:09:15.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:15.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:09:15.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:15 smithi026 ceph-mon[68309]: pgmap v986: 57 pgs: 1 active+clean+remapped+scrubbing+deep, 12 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 4.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 110 op/s; 8421/194733 objects degraded (4.324%); 28390/194733 objects misplaced (14.579%); 8.3 MiB/s, 132 objects/s recovering 2024-03-20T09:09:15.310 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:09:15.356 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:15.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:15 smithi115 ceph-mon[72476]: pgmap v986: 57 pgs: 1 active+clean+remapped+scrubbing+deep, 12 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 4.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 110 op/s; 8421/194733 objects degraded (4.324%); 28390/194733 objects misplaced (14.579%); 8.3 MiB/s, 132 objects/s recovering 2024-03-20T09:09:15.413 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:09:15.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:15.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:09:15.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:15 smithi122 ceph-mon[73495]: pgmap v986: 57 pgs: 1 active+clean+remapped+scrubbing+deep, 12 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 4.0 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 110 op/s; 8421/194733 objects degraded (4.324%); 28390/194733 objects misplaced (14.579%); 8.3 MiB/s, 132 objects/s recovering 2024-03-20T09:09:15.524 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:09:15.551 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:15.570 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:09:15.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:15.572 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:09:15.674 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:09:15.705 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:15.743 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:09:15.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:15.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:09:15.847 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:09:15.874 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:15.893 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:09:15.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:15.896 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:09:15.909 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:09:15.909 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:09:15.909 INFO:teuthology.orchestra.run.smithi115.stdout: "num_ops": 0 2024-03-20T09:09:15.909 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:09:15.997 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:09:16.025 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:16.048 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:09:16.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:16 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1302793380' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:09:16.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:16 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1260975879' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:09:16.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:16 smithi122 ceph-mon[73495]: 3.16 deep-scrub ok 2024-03-20T09:09:16.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:16 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1302793380' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:09:16.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:16 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1260975879' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:09:16.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:16 smithi026 ceph-mon[68309]: 3.16 deep-scrub ok 2024-03-20T09:09:16.225 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:09:16.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:16 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1302793380' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:09:16.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:16 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1260975879' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:09:16.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:16 smithi115 ceph-mon[72476]: 3.16 deep-scrub ok 2024-03-20T09:09:16.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:16.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:09:16.329 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:09:16.360 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:16.613 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:09:16.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:16.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:09:16.718 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:09:16.720 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-20T09:09:16.749 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:16.770 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:09:16.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:16.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:09:16.883 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:09:16.912 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:16.933 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:09:16.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:16.935 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:09:17.003 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:09:17.036 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:09:17.062 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:17.081 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:09:17.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:17.083 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:09:17.184 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:09:17.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:17 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:09:17.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:17 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:09:17.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:17 smithi122 ceph-mon[73495]: pgmap v987: 57 pgs: 1 active+clean+remapped+scrubbing+deep, 12 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 3.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 109 op/s; 8390/193944 objects degraded (4.326%); 27367/193944 objects misplaced (14.111%); 5.2 MiB/s, 83 objects/s recovering 2024-03-20T09:09:17.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:17 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:09:17.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:17 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1302793380' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:09:17.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:17 smithi122 ceph-mon[73495]: osdmap e408: 12 total, 11 up, 9 in 2024-03-20T09:09:17.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:17 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8390/193944 objects degraded (4.326%), 5 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:09:17.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:17 smithi122 ceph-mon[73495]: osdmap e409: 12 total, 11 up, 9 in 2024-03-20T09:09:17.215 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:17.388 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:17 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:09:17.388 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:17 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:09:17.388 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:17 smithi026 ceph-mon[68309]: pgmap v987: 57 pgs: 1 active+clean+remapped+scrubbing+deep, 12 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 3.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 109 op/s; 8390/193944 objects degraded (4.326%); 27367/193944 objects misplaced (14.111%); 5.2 MiB/s, 83 objects/s recovering 2024-03-20T09:09:17.388 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:17 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:09:17.388 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:17 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1302793380' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:09:17.388 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:17 smithi026 ceph-mon[68309]: osdmap e408: 12 total, 11 up, 9 in 2024-03-20T09:09:17.388 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:17 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8390/193944 objects degraded (4.326%), 5 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:09:17.388 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:17 smithi026 ceph-mon[68309]: osdmap e409: 12 total, 11 up, 9 in 2024-03-20T09:09:17.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:17 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:09:17.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:17 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:09:17.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:17 smithi115 ceph-mon[72476]: pgmap v987: 57 pgs: 1 active+clean+remapped+scrubbing+deep, 12 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 3.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 109 op/s; 8390/193944 objects degraded (4.326%); 27367/193944 objects misplaced (14.111%); 5.2 MiB/s, 83 objects/s recovering 2024-03-20T09:09:17.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:17 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:09:17.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:17 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1302793380' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:09:17.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:17 smithi115 ceph-mon[72476]: osdmap e408: 12 total, 11 up, 9 in 2024-03-20T09:09:17.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:17 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8390/193944 objects degraded (4.326%), 5 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:09:17.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:17 smithi115 ceph-mon[72476]: osdmap e409: 12 total, 11 up, 9 in 2024-03-20T09:09:17.421 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:09:17.427 INFO:tasks.thrashosds.thrasher:j is {'epoch': 407, 'fsid': '852fbb68-e696-11ee-95c9-87774f69a715', 'created': '2024-03-20T08:48:35.281088+0000', 'modified': '2024-03-20T09:09:06.969357+0000', 'last_up_change': '2024-03-20T09:04:33.695633+0000', 'last_in_change': '2024-03-20T09:07:48.551904+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-20T08:52:17.442324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '360', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '358', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.700000047683716, 'score_stable': 2.700000047683716, 'optimal_score': 0.6700000166893005, 'raw_score_acting': 1.809999942779541, 'raw_score_stable': 1.809999942779541, 'primary_affinity_weighted': 0.9215061664581299, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 3, 'pool_name': 'unique_pool_1', 'create_time': '2024-03-20T09:02:48.375175+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '233', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '231', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.2100000381469727, 'score_stable': 3.2100000381469727, 'optimal_score': 0.6700000166893005, 'raw_score_acting': 2.1500000953674316, 'raw_score_stable': 2.1500000953674316, 'primary_affinity_weighted': 0.9215061664581299, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}], 'osds': [{'osd': 0, 'uuid': 'ad3fd9c0-316f-4fe5-a63b-d480087c1729', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 385, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6802', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6803', 'nonce': 2139805293}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6804', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6805', 'nonce': 2139805293}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6808', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6809', 'nonce': 2139805293}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6806', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6807', 'nonce': 2139805293}]}, 'public_addr': '172.21.15.26:6803/2139805293', 'cluster_addr': '172.21.15.26:6805/2139805293', 'heartbeat_back_addr': '172.21.15.26:6809/2139805293', 'heartbeat_front_addr': '172.21.15.26:6807/2139805293', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'e85b94f5-2954-47a1-9c20-096f2606eb04', 'up': 1, 'in': 1, 'weight': 0.97100830078125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 386, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6810', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6811', 'nonce': 3853355134}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6812', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6813', 'nonce': 3853355134}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6816', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6817', 'nonce': 3853355134}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6814', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6815', 'nonce': 3853355134}]}, 'public_addr': '172.21.15.26:6811/3853355134', 'cluster_addr': '172.21.15.26:6813/3853355134', 'heartbeat_back_addr': '172.21.15.26:6817/3853355134', 'heartbeat_front_addr': '172.21.15.26:6815/3853355134', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'dd2b10f0-741c-47a4-b224-ac6bfe2c14ef', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 383, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6818', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6819', 'nonce': 3993357967}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6820', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6821', 'nonce': 3993357967}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6824', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6825', 'nonce': 3993357967}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6822', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6823', 'nonce': 3993357967}]}, 'public_addr': '172.21.15.26:6819/3993357967', 'cluster_addr': '172.21.15.26:6821/3993357967', 'heartbeat_back_addr': '172.21.15.26:6825/3993357967', 'heartbeat_front_addr': '172.21.15.26:6823/3993357967', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 274, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6826', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6827', 'nonce': 4110225509}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6828', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6829', 'nonce': 4110225509}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6832', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6833', 'nonce': 4110225509}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6830', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6831', 'nonce': 4110225509}]}, 'public_addr': '172.21.15.26:6827/4110225509', 'cluster_addr': '172.21.15.26:6829/4110225509', 'heartbeat_back_addr': '172.21.15.26:6833/4110225509', 'heartbeat_front_addr': '172.21.15.26:6831/4110225509', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'f14e34d5-9829-42a3-9b25-31a0976c5ef7', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 366, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6800', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6801', 'nonce': 477923058}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6802', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6803', 'nonce': 477923058}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6806', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6807', 'nonce': 477923058}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6804', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6805', 'nonce': 477923058}]}, 'public_addr': '172.21.15.115:6801/477923058', 'cluster_addr': '172.21.15.115:6803/477923058', 'heartbeat_back_addr': '172.21.15.115:6807/477923058', 'heartbeat_front_addr': '172.21.15.115:6805/477923058', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'cb431b6a-540d-4318-ae35-d379e1df0185', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 265, 'down_at': 274, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6808', 'nonce': 1339268003}, {'type': 'v1', 'addr': '172.21.15.115:6809', 'nonce': 1339268003}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6810', 'nonce': 1339268003}, {'type': 'v1', 'addr': '172.21.15.115:6811', 'nonce': 1339268003}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6814', 'nonce': 1339268003}, {'type': 'v1', 'addr': '172.21.15.115:6815', 'nonce': 1339268003}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6812', 'nonce': 1339268003}, {'type': 'v1', 'addr': '172.21.15.115:6813', 'nonce': 1339268003}]}, 'public_addr': '172.21.15.115:6809/1339268003', 'cluster_addr': '172.21.15.115:6811/1339268003', 'heartbeat_back_addr': '172.21.15.115:6815/1339268003', 'heartbeat_front_addr': '172.21.15.115:6813/1339268003', 'state': ['exists']}, {'osd': 6, 'uuid': '753ed830-e764-4ff3-9d9b-87e7fa748b81', 'up': 1, 'in': 1, 'weight': 0.842376708984375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 404, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6816', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6817', 'nonce': 3278301843}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6818', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6819', 'nonce': 3278301843}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6822', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6823', 'nonce': 3278301843}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6820', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6821', 'nonce': 3278301843}]}, 'public_addr': '172.21.15.115:6817/3278301843', 'cluster_addr': '172.21.15.115:6819/3278301843', 'heartbeat_back_addr': '172.21.15.115:6823/3278301843', 'heartbeat_front_addr': '172.21.15.115:6821/3278301843', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '55bcbd37-7a6f-4ab1-b5a9-91a951444d86', 'up': 1, 'in': 1, 'weight': 0.4264678955078125, 'primary_affinity': 0.058074951171875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 274, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6824', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6825', 'nonce': 3152424001}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6826', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6827', 'nonce': 3152424001}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6830', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6831', 'nonce': 3152424001}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6828', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6829', 'nonce': 3152424001}]}, 'public_addr': '172.21.15.115:6825/3152424001', 'cluster_addr': '172.21.15.115:6827/3152424001', 'heartbeat_back_addr': '172.21.15.115:6831/3152424001', 'heartbeat_front_addr': '172.21.15.115:6829/3152424001', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'bc86248e-6bd0-4369-b8a2-e593bc2a610d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 70, 'up_thru': 380, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6800', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6801', 'nonce': 1309426740}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6802', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6803', 'nonce': 1309426740}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6806', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6807', 'nonce': 1309426740}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6804', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6805', 'nonce': 1309426740}]}, 'public_addr': '172.21.15.122:6801/1309426740', 'cluster_addr': '172.21.15.122:6803/1309426740', 'heartbeat_back_addr': '172.21.15.122:6807/1309426740', 'heartbeat_front_addr': '172.21.15.122:6805/1309426740', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '9b20a4aa-032a-4c09-ac75-153c554424e7', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 257, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6808', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6809', 'nonce': 2953210076}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6810', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6811', 'nonce': 2953210076}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6814', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6815', 'nonce': 2953210076}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6812', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6813', 'nonce': 2953210076}]}, 'public_addr': '172.21.15.122:6809/2953210076', 'cluster_addr': '172.21.15.122:6811/2953210076', 'heartbeat_back_addr': '172.21.15.122:6815/2953210076', 'heartbeat_front_addr': '172.21.15.122:6813/2953210076', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'e62c1fb3-5dd1-4d0d-997c-d20633d19361', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 89, 'up_thru': 386, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6816', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6817', 'nonce': 409394340}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6818', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6819', 'nonce': 409394340}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6822', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6823', 'nonce': 409394340}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6820', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6821', 'nonce': 409394340}]}, 'public_addr': '172.21.15.122:6817/409394340', 'cluster_addr': '172.21.15.122:6819/409394340', 'heartbeat_back_addr': '172.21.15.122:6823/409394340', 'heartbeat_front_addr': '172.21.15.122:6821/409394340', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'b83994d5-e5ef-48b0-9758-c26843cec3b6', 'up': 1, 'in': 1, 'weight': 0.6123046875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 96, 'up_thru': 385, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6824', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6825', 'nonce': 2704069354}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6826', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6827', 'nonce': 2704069354}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6830', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6831', 'nonce': 2704069354}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6828', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6829', 'nonce': 2704069354}]}, 'public_addr': '172.21.15.122:6825/2704069354', 'cluster_addr': '172.21.15.122:6827/2704069354', 'heartbeat_back_addr': '172.21.15.122:6831/2704069354', 'heartbeat_front_addr': '172.21.15.122:6829/2704069354', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:08:04.860551+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:08:57.275643+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:08:18.494972+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 35063, 'last_purged_snaps_scrub': '2024-03-20T09:09:02.812475+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-20T09:08:30.077943+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:03:53.609882+0000', 'dead_epoch': 273}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:08:14.465561+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:07:57.050958+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:08:24.980388+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 34736, 'last_purged_snaps_scrub': '2024-03-20T09:07:52.842622+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:08:19.771648+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:08:40.300610+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '1.8', 'osds': [2, 8, 4]}, {'pgid': '1.b', 'osds': [0, 2, 11]}, {'pgid': '1.f', 'osds': [6, 8, 4]}, {'pgid': '1.15', 'osds': [8, 1, 2]}, {'pgid': '1.17', 'osds': [8, 10, 6]}, {'pgid': '1.18', 'osds': [2, 7, 8]}, {'pgid': '3.2', 'osds': [6, 1, 4]}, {'pgid': '3.6', 'osds': [0, 1, 4]}, {'pgid': '3.8', 'osds': [11, 1, 4]}, {'pgid': '3.b', 'osds': [0, 11]}, {'pgid': '3.c', 'osds': [6, 10, 3]}, {'pgid': '3.d', 'osds': [6, 7, 9]}, {'pgid': '3.e', 'osds': [0, 10]}, {'pgid': '3.10', 'osds': [6, 0, 2]}, {'pgid': '3.15', 'osds': [11, 0]}, {'pgid': '3.16', 'osds': [1, 2, 7]}, {'pgid': '3.17', 'osds': [0, 1, 6]}], 'primary_temp': [], 'blocklist': {'172.21.15.26:0/4291404191': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/642012566': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:6801/1181459999': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/3984513952': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6800/1654945498': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6800/1181459999': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/137770981': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:0/1451237762': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/3938386631': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6801/1654945498': '2024-03-21T08:49:04.596754+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-20T09:09:17.427 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:09:17.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:17.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:09:17.529 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:09:17.555 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:17.573 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:09:17.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:17.576 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:09:17.677 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:09:17.703 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:17.721 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:09:17.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:17.724 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:09:17.825 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:09:17.852 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:17.871 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:09:17.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:17.874 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:09:17.975 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:09:18.006 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:18.296 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:09:18.299 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:09:18.321 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:09:18.342 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:09:18.362 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:09:18.382 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:09:18.402 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:09:18.424 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:09:18.448 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:09:18.455 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:09:18.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:18.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:09:18.469 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:09:18.491 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:09:18.518 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:09:18.518 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:09:18.518 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:09:18.518 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:09:18.518 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:09:18.518 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:09:18.518 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-20T09:09:18.518 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:09:18.518 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:09:18.518 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:09:18.519 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:09:18.519 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:09:18.519 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:09:18.559 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:09:18.596 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:18.746 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:09:18.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:18.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:09:18.850 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:09:18.880 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:19.055 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:09:19.056 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:09:19.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:19 smithi026 ceph-mon[68309]: pgmap v990: 57 pgs: 1 active+clean+remapped+scrubbing+deep, 12 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 3.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 116 op/s; 8390/193386 objects degraded (4.338%); 27278/193386 objects misplaced (14.105%); 4.5 MiB/s, 71 objects/s recovering 2024-03-20T09:09:19.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:19 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4002469350' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:09:19.231 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:09:19.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:19.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:09:19.336 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:09:19.367 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:19.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:19 smithi115 ceph-mon[72476]: pgmap v990: 57 pgs: 1 active+clean+remapped+scrubbing+deep, 12 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 3.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 116 op/s; 8390/193386 objects degraded (4.338%); 27278/193386 objects misplaced (14.105%); 4.5 MiB/s, 71 objects/s recovering 2024-03-20T09:09:19.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:19 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4002469350' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:09:19.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:19 smithi122 ceph-mon[73495]: pgmap v990: 57 pgs: 1 active+clean+remapped+scrubbing+deep, 12 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 3.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 116 op/s; 8390/193386 objects degraded (4.338%); 27278/193386 objects misplaced (14.105%); 4.5 MiB/s, 71 objects/s recovering 2024-03-20T09:09:19.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:19 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4002469350' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:09:19.782 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:09:19.783 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:09:19.815 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-20T09:09:19.990 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:09:20.132 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:09:20.217 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:09:20.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:20.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:09:20.322 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:09:20.348 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:20 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:09:20.348 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:20 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:09:20.348 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:20 smithi026 ceph-mon[68309]: osdmap e410: 12 total, 11 up, 9 in 2024-03-20T09:09:20.350 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:20.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:20 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:09:20.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:20 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:09:20.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:20 smithi115 ceph-mon[72476]: osdmap e410: 12 total, 11 up, 9 in 2024-03-20T09:09:20.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:20 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:09:20.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:20 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:09:20.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:20 smithi122 ceph-mon[73495]: osdmap e410: 12 total, 11 up, 9 in 2024-03-20T09:09:20.484 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:09:20.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:20.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:09:20.588 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:09:20.616 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:20.636 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:09:20.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:20.639 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:09:20.736 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:09:20.739 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:09:20.766 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:20.785 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:09:20.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:20.788 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:09:20.888 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:09:20.915 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:20.936 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:09:20.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:20.939 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:09:21.001 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:09:21.004 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:09:21.040 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:09:21.079 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:21.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:21 smithi122 ceph-mon[73495]: pgmap v992: 57 pgs: 1 active+clean+remapped+scrubbing+deep, 12 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 3.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 91 op/s; 8381/193077 objects degraded (4.341%); 26604/193077 objects misplaced (13.779%); 6.7 MiB/s, 106 objects/s recovering 2024-03-20T09:09:21.360 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:21 smithi026 ceph-mon[68309]: pgmap v992: 57 pgs: 1 active+clean+remapped+scrubbing+deep, 12 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 3.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 91 op/s; 8381/193077 objects degraded (4.341%); 26604/193077 objects misplaced (13.779%); 6.7 MiB/s, 106 objects/s recovering 2024-03-20T09:09:21.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:21 smithi115 ceph-mon[72476]: pgmap v992: 57 pgs: 1 active+clean+remapped+scrubbing+deep, 12 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 3.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 91 op/s; 8381/193077 objects degraded (4.341%); 26604/193077 objects misplaced (13.779%); 6.7 MiB/s, 106 objects/s recovering 2024-03-20T09:09:21.784 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:09:22.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:22 smithi115 ceph-mon[72476]: from='client.25427 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:09:22.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:22 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8381/193077 objects degraded (4.341%), 5 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:09:22.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:22 smithi115 ceph-mon[72476]: osdmap e411: 12 total, 11 up, 9 in 2024-03-20T09:09:22.198 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:09:22.198 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:09:22.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:22 smithi122 ceph-mon[73495]: from='client.25427 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:09:22.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:22 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8381/193077 objects degraded (4.341%), 5 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:09:22.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:22 smithi122 ceph-mon[73495]: osdmap e411: 12 total, 11 up, 9 in 2024-03-20T09:09:22.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:22 smithi026 ceph-mon[68309]: from='client.25427 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:09:22.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:22 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8381/193077 objects degraded (4.341%), 5 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:09:22.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:22 smithi026 ceph-mon[68309]: osdmap e411: 12 total, 11 up, 9 in 2024-03-20T09:09:22.725 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":992,"stamp":"2024-03-20T09:09:19.717682+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4218447707,"num_objects":64359,"num_object_clones":0,"num_object_copies":193077,"num_objects_missing_on_primary":3869,"num_objects_missing":3869,"num_objects_degraded":8381,"num_objects_misplaced":26604,"num_objects_unfound":0,"num_objects_dirty":64359,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":97564,"num_write_kb":4113010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110453,"num_bytes_recovered":6349984164,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":181208,"ondisk_log_size":181208,"up":150,"acting":161,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":178,"num_osds":9,"num_per_pool_osds":9,"num_per_pool_omap_osds":9,"kb":843706368,"kb_used":15378328,"kb_used_data":13408188,"kb_used_omap":14,"kb_used_meta":1970033,"kb_avail":828328040,"statfs":{"total":863955320832,"available":848207912960,"internally_reserved":0,"allocated":13729984512,"data_stored":13721969003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14353,"internal_metadata":2017314799},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,0,5,8,13,2,0,1,2,4,12,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-36175872,"num_objects":-552,"num_object_clones":0,"num_object_copies":-1656,"num_objects_missing_on_primary":-154,"num_objects_missing":-154,"num_objects_degraded":-40,"num_objects_misplaced":-1786,"num_objects_unfound":0,"num_objects_dirty":-552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":552,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":642,"num_bytes_recovered":42074112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001374"},"pg_stats":[{"pgid":"1.1e","version":"20'137","reported_seq":496,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-20T09:09:16.050614+0000","last_change":"2024-03-20T09:07:41.961068+0000","last_active":"2024-03-20T09:09:16.050614+0000","last_peered":"2024-03-20T09:09:16.050614+0000","last_clean":"2024-03-20T09:09:16.050614+0000","last_became_active":"2024-03-20T09:07:41.960647+0000","last_became_peered":"2024-03-20T09:07:41.960647+0000","last_unstale":"2024-03-20T09:09:16.050614+0000","last_undegraded":"2024-03-20T09:09:16.050614+0000","last_fullsized":"2024-03-20T09:09:16.050614+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:03:55.906959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:03:55.906959+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,10],"acting":[8,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"20'137","reported_seq":346,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-03-20T09:09:06.974373+0000","last_change":"2024-03-20T09:08:55.968957+0000","last_active":"2024-03-20T09:09:06.974373+0000","last_peered":"2024-03-20T09:09:06.974373+0000","last_clean":"2024-03-20T09:09:06.974373+0000","last_became_active":"2024-03-20T09:07:42.013245+0000","last_became_peered":"2024-03-20T09:07:42.013245+0000","last_unstale":"2024-03-20T09:09:06.974373+0000","last_undegraded":"2024-03-20T09:09:06.974373+0000","last_fullsized":"2024-03-20T09:09:06.974373+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:08:55.968619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:08:55.968619+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:10:18.422830+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"20'137","reported_seq":783,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-20T09:09:16.050570+0000","last_change":"2024-03-20T09:07:41.962095+0000","last_active":"2024-03-20T09:09:16.050570+0000","last_peered":"2024-03-20T09:09:16.050570+0000","last_clean":"2024-03-20T09:09:16.050570+0000","last_became_active":"2024-03-20T09:07:41.961622+0000","last_became_peered":"2024-03-20T09:07:41.961622+0000","last_unstale":"2024-03-20T09:09:16.050570+0000","last_undegraded":"2024-03-20T09:09:16.050570+0000","last_fullsized":"2024-03-20T09:09:16.050570+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:04:04.682759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:04:04.682759+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163840,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.19","version":"20'137","reported_seq":859,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-20T09:09:16.047860+0000","last_change":"2024-03-20T09:07:41.963076+0000","last_active":"2024-03-20T09:09:16.047860+0000","last_peered":"2024-03-20T09:09:16.047860+0000","last_clean":"2024-03-20T09:09:16.047860+0000","last_became_active":"2024-03-20T09:07:41.962680+0000","last_became_peered":"2024-03-20T09:07:41.962680+0000","last_unstale":"2024-03-20T09:09:16.047860+0000","last_undegraded":"2024-03-20T09:09:16.047860+0000","last_fullsized":"2024-03-20T09:09:16.047860+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:03:56.726408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:03:56.726408+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":8,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":120211,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"20'137","reported_seq":563,"reported_epoch":409,"state":"active+clean+remapped","last_fresh":"2024-03-20T09:09:17.080657+0000","last_change":"2024-03-20T09:08:52.876636+0000","last_active":"2024-03-20T09:09:17.080657+0000","last_peered":"2024-03-20T09:09:17.080657+0000","last_clean":"2024-03-20T09:09:17.080657+0000","last_became_active":"2024-03-20T09:07:41.965126+0000","last_became_peered":"2024-03-20T09:07:41.965126+0000","last_unstale":"2024-03-20T09:09:17.080657+0000","last_undegraded":"2024-03-20T09:09:17.080657+0000","last_fullsized":"2024-03-20T09:09:17.080657+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:08:52.876490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:08:52.876490+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:10:01.705833+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7,8],"avail_no_missing":["2","7","8"],"object_location_counts":[{"shards":"2,7,8","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1b","version":"20'137","reported_seq":612,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-20T09:09:16.050452+0000","last_change":"2024-03-20T09:07:49.974049+0000","last_active":"2024-03-20T09:09:16.050452+0000","last_peered":"2024-03-20T09:09:16.050452+0000","last_clean":"2024-03-20T09:09:16.050452+0000","last_became_active":"2024-03-20T09:07:41.962733+0000","last_became_peered":"2024-03-20T09:07:41.962733+0000","last_unstale":"2024-03-20T09:09:16.050452+0000","last_undegraded":"2024-03-20T09:09:16.050452+0000","last_fullsized":"2024-03-20T09:09:16.050452+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:07:49.973958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:07:49.973958+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.19","version":"407'6002","reported_seq":26465,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-03-20T09:09:13.232936+0000","last_change":"2024-03-20T09:08:58.127363+0000","last_active":"2024-03-20T09:09:13.232936+0000","last_peered":"2024-03-20T09:09:13.232936+0000","last_clean":"2024-03-20T09:09:13.232936+0000","last_became_active":"2024-03-20T09:08:16.036346+0000","last_became_peered":"2024-03-20T09:08:16.036346+0000","last_unstale":"2024-03-20T09:09:13.232936+0000","last_undegraded":"2024-03-20T09:09:13.232936+0000","last_fullsized":"2024-03-20T09:09:13.232936+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":5,"last_scrub":"402'5952","last_scrub_stamp":"2024-03-20T09:08:58.127136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:08:58.127136+0000","objects_scrubbed":2081,"log_size":6002,"log_dups_size":0,"ondisk_log_size":6002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:10:27.523713+0000","scrub_duration":234,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132907008,"num_objects":2028,"num_object_clones":0,"num_object_copies":6084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1054,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6290,"num_bytes_recovered":331808768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,2],"acting":[1,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"20'137","reported_seq":559,"reported_epoch":409,"state":"active+clean+remapped","last_fresh":"2024-03-20T09:09:17.080972+0000","last_change":"2024-03-20T09:08:55.831937+0000","last_active":"2024-03-20T09:09:17.080972+0000","last_peered":"2024-03-20T09:09:17.080972+0000","last_clean":"2024-03-20T09:09:17.080972+0000","last_became_active":"2024-03-20T09:07:36.954026+0000","last_became_peered":"2024-03-20T09:07:36.954026+0000","last_unstale":"2024-03-20T09:09:17.080972+0000","last_undegraded":"2024-03-20T09:09:17.080972+0000","last_fullsized":"2024-03-20T09:09:17.080972+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:08:55.831799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:08:55.831799+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:09:56.138789+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8],"acting":[2,8,4],"avail_no_missing":["2","4","8"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"407'8480","reported_seq":36210,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-03-20T09:09:15.305311+0000","last_change":"2024-03-20T09:09:04.027805+0000","last_active":"2024-03-20T09:09:15.305311+0000","last_peered":"2024-03-20T09:09:15.305311+0000","last_clean":"2024-03-20T09:09:15.305311+0000","last_became_active":"2024-03-20T09:09:04.010129+0000","last_became_peered":"2024-03-20T09:09:04.010129+0000","last_unstale":"2024-03-20T09:09:15.305311+0000","last_undegraded":"2024-03-20T09:09:15.305311+0000","last_fullsized":"2024-03-20T09:09:15.305311+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":8480,"log_dups_size":0,"ondisk_log_size":8480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267649024,"num_objects":4084,"num_object_clones":0,"num_object_copies":12252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8480,"num_write_kb":402048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13635,"num_bytes_recovered":830865408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,8],"acting":[6,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"20'137","reported_seq":860,"reported_epoch":409,"state":"active+undersized","last_fresh":"2024-03-20T09:09:17.080765+0000","last_change":"2024-03-20T09:07:36.951344+0000","last_active":"2024-03-20T09:09:17.080765+0000","last_peered":"2024-03-20T09:09:17.080765+0000","last_clean":"2024-03-20T09:04:29.198571+0000","last_became_active":"2024-03-20T09:07:36.951344+0000","last_became_peered":"2024-03-20T09:07:36.951344+0000","last_unstale":"2024-03-20T09:09:17.080765+0000","last_undegraded":"2024-03-20T09:09:17.080765+0000","last_fullsized":"2024-03-20T09:07:36.946346+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:03:56.726408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:03:56.726408+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:09:17.853401+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8,"num_write_kb":133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":120211,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"407'8528","reported_seq":34879,"reported_epoch":407,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-20T09:09:15.812350+0000","last_change":"2024-03-20T09:08:08.228591+0000","last_active":"2024-03-20T09:09:15.812350+0000","last_peered":"2024-03-20T09:09:15.812350+0000","last_clean":"2024-03-20T09:07:48.417829+0000","last_became_active":"2024-03-20T09:07:50.853930+0000","last_became_peered":"2024-03-20T09:07:50.853930+0000","last_unstale":"2024-03-20T09:09:15.812350+0000","last_undegraded":"2024-03-20T09:07:50.636239+0000","last_fullsized":"2024-03-20T09:07:50.567026+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":8528,"log_dups_size":0,"ondisk_log_size":8528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270925824,"num_objects":4134,"num_object_clones":0,"num_object_copies":12402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":1698,"num_objects_unfound":0,"num_objects_dirty":4134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8528,"num_write_kb":405184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12949,"num_bytes_recovered":793051136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,7],"acting":[0,11],"avail_no_missing":["0","3","4","11"],"object_location_counts":[{"shards":"0,4,11","objects":1698},{"shards":"0,11","objects":2426}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"20'137","reported_seq":667,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-03-20T09:09:06.973352+0000","last_change":"2024-03-20T09:07:52.618525+0000","last_active":"2024-03-20T09:09:06.973352+0000","last_peered":"2024-03-20T09:09:06.973352+0000","last_clean":"2024-03-20T09:09:06.973352+0000","last_became_active":"2024-03-20T09:07:52.618107+0000","last_became_peered":"2024-03-20T09:07:52.618107+0000","last_unstale":"2024-03-20T09:09:06.973352+0000","last_undegraded":"2024-03-20T09:09:06.973352+0000","last_fullsized":"2024-03-20T09:09:06.973352+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:03:13.525325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:03:13.525325+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:09:08.508719+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"410'6074","reported_seq":6512,"reported_epoch":410,"state":"active+clean+remapped","last_fresh":"2024-03-20T09:09:19.420544+0000","last_change":"2024-03-20T09:07:50.588770+0000","last_active":"2024-03-20T09:09:19.420544+0000","last_peered":"2024-03-20T09:09:19.420544+0000","last_clean":"2024-03-20T09:09:19.420544+0000","last_became_active":"2024-03-20T09:07:50.576012+0000","last_became_peered":"2024-03-20T09:07:50.576012+0000","last_unstale":"2024-03-20T09:09:19.420544+0000","last_undegraded":"2024-03-20T09:09:19.420544+0000","last_fullsized":"2024-03-20T09:09:19.420544+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":6074,"log_dups_size":0,"ondisk_log_size":6074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:09:21.691399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131694592,"num_objects":2009,"num_object_clones":0,"num_object_copies":6027,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2009,"num_objects_unfound":0,"num_objects_dirty":2009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4309,"num_write_kb":202208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1],"acting":[11,1,4],"avail_no_missing":["11","1","4"],"object_location_counts":[{"shards":"1,4,11","objects":2009}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"20'137","reported_seq":828,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-20T09:09:19.053734+0000","last_change":"2024-03-20T09:06:02.927341+0000","last_active":"2024-03-20T09:09:19.053734+0000","last_peered":"2024-03-20T09:09:19.053734+0000","last_clean":"2024-03-20T09:09:19.053734+0000","last_became_active":"2024-03-20T09:06:02.926326+0000","last_became_peered":"2024-03-20T09:06:02.926326+0000","last_unstale":"2024-03-20T09:09:19.053734+0000","last_undegraded":"2024-03-20T09:09:19.053734+0000","last_fullsized":"2024-03-20T09:09:19.053734+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:24.397259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:24.397259+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:09:21.691374+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109226,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":109226,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.6","version":"407'6028","reported_seq":14017,"reported_epoch":407,"state":"active+clean+remapped","last_fresh":"2024-03-20T09:09:14.972050+0000","last_change":"2024-03-20T09:08:31.564953+0000","last_active":"2024-03-20T09:09:14.972050+0000","last_peered":"2024-03-20T09:09:14.972050+0000","last_clean":"2024-03-20T09:09:14.972050+0000","last_became_active":"2024-03-20T09:08:30.522411+0000","last_became_peered":"2024-03-20T09:08:30.522411+0000","last_unstale":"2024-03-20T09:09:14.972050+0000","last_undegraded":"2024-03-20T09:09:14.972050+0000","last_fullsized":"2024-03-20T09:09:14.972050+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'4943","last_scrub_stamp":"2024-03-20T09:04:46.116541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:04:46.116541+0000","objects_scrubbed":3048,"log_size":6028,"log_dups_size":0,"ondisk_log_size":6028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":317,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128647168,"num_objects":1963,"num_object_clones":0,"num_object_copies":5889,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1963,"num_objects_unfound":0,"num_objects_dirty":1963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1085,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3089,"num_bytes_recovered":160890880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1,4],"avail_no_missing":["0","1","4"],"object_location_counts":[{"shards":"0,1,4","objects":1963}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"20'137","reported_seq":737,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-20T09:09:16.048196+0000","last_change":"2024-03-20T09:07:37.003973+0000","last_active":"2024-03-20T09:09:16.048196+0000","last_peered":"2024-03-20T09:09:16.048196+0000","last_clean":"2024-03-20T09:09:16.048196+0000","last_became_active":"2024-03-20T09:07:37.003722+0000","last_became_peered":"2024-03-20T09:07:37.003722+0000","last_unstale":"2024-03-20T09:09:16.048196+0000","last_undegraded":"2024-03-20T09:09:16.048196+0000","last_fullsized":"2024-03-20T09:09:16.048196+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:07:06.480160+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:07:06.480160+0000","last_clean_scrub_stamp":"2024-03-20T09:07:06.480160+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.7","version":"410'6002","reported_seq":22178,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-20T09:09:19.580719+0000","last_change":"2024-03-20T09:08:15.050515+0000","last_active":"2024-03-20T09:09:19.580719+0000","last_peered":"2024-03-20T09:09:19.580719+0000","last_clean":"2024-03-20T09:09:19.580719+0000","last_became_active":"2024-03-20T09:08:15.041211+0000","last_became_peered":"2024-03-20T09:08:15.041211+0000","last_unstale":"2024-03-20T09:09:19.580719+0000","last_undegraded":"2024-03-20T09:09:19.580719+0000","last_fullsized":"2024-03-20T09:09:19.580719+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"254'4179","last_scrub_stamp":"2024-03-20T09:04:00.661196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:04:00.661196+0000","objects_scrubbed":2441,"log_size":6002,"log_dups_size":0,"ondisk_log_size":6002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131661824,"num_objects":2009,"num_object_clones":0,"num_object_copies":6027,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1824,"num_write_kb":44544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6274,"num_bytes_recovered":325976064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"20'137","reported_seq":754,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-20T09:09:16.048273+0000","last_change":"2024-03-20T09:07:52.593985+0000","last_active":"2024-03-20T09:09:16.048273+0000","last_peered":"2024-03-20T09:09:16.048273+0000","last_clean":"2024-03-20T09:09:16.048273+0000","last_became_active":"2024-03-20T09:07:52.593671+0000","last_became_peered":"2024-03-20T09:07:52.593671+0000","last_unstale":"2024-03-20T09:09:16.048273+0000","last_undegraded":"2024-03-20T09:09:16.048273+0000","last_fullsized":"2024-03-20T09:09:16.048273+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:06:58.034269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:06:58.034269+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.4","version":"407'5945","reported_seq":6380,"reported_epoch":407,"state":"active+undersized+degraded","last_fresh":"2024-03-20T09:09:13.111551+0000","last_change":"2024-03-20T09:04:34.736993+0000","last_active":"2024-03-20T09:09:13.111551+0000","last_peered":"2024-03-20T09:09:13.111551+0000","last_clean":"2024-03-20T09:04:33.698023+0000","last_became_active":"2024-03-20T09:04:34.736993+0000","last_became_peered":"2024-03-20T09:04:34.736993+0000","last_unstale":"2024-03-20T09:09:13.111551+0000","last_undegraded":"2024-03-20T09:04:34.735838+0000","last_fullsized":"2024-03-20T09:04:34.735696+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":199,"log_size":5945,"log_dups_size":0,"ondisk_log_size":5945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:09:13.876232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133267456,"num_objects":2033,"num_object_clones":0,"num_object_copies":6099,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4207,"num_write_kb":199712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":["1","11"],"object_location_counts":[{"shards":"1,11","objects":2033}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"20'137","reported_seq":735,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-03-20T09:09:06.974003+0000","last_change":"2024-03-20T09:09:02.936326+0000","last_active":"2024-03-20T09:09:06.974003+0000","last_peered":"2024-03-20T09:09:06.974003+0000","last_clean":"2024-03-20T09:09:06.974003+0000","last_became_active":"2024-03-20T09:07:52.603192+0000","last_became_peered":"2024-03-20T09:07:52.603192+0000","last_unstale":"2024-03-20T09:09:06.974003+0000","last_undegraded":"2024-03-20T09:09:06.974003+0000","last_fullsized":"2024-03-20T09:09:06.974003+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:09:02.936192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:09:02.936192+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:10:14.716369+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,10],"acting":[1,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"407'8971","reported_seq":15928,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-03-20T09:09:15.811474+0000","last_change":"2024-03-20T09:08:15.045902+0000","last_active":"2024-03-20T09:09:15.811474+0000","last_peered":"2024-03-20T09:09:15.811474+0000","last_clean":"2024-03-20T09:09:15.811474+0000","last_became_active":"2024-03-20T09:08:15.038033+0000","last_became_peered":"2024-03-20T09:08:15.038033+0000","last_unstale":"2024-03-20T09:09:15.811474+0000","last_undegraded":"2024-03-20T09:09:15.811474+0000","last_fullsized":"2024-03-20T09:09:15.811474+0000","mapping_epoch":385,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"285'7948","last_scrub_stamp":"2024-03-20T09:04:57.536605+0000","last_deep_scrub":"285'7948","last_deep_scrub_stamp":"2024-03-20T09:04:57.536605+0000","last_clean_scrub_stamp":"2024-03-20T09:04:57.536605+0000","objects_scrubbed":3080,"log_size":8971,"log_dups_size":0,"ondisk_log_size":8971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"queued for scrub","scrub_duration":5935,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133496832,"num_objects":2037,"num_object_clones":0,"num_object_copies":6111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2172,"num_write_kb":65920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3179,"num_bytes_recovered":150798336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,11],"acting":[0,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":799,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-20T09:09:19.053306+0000","last_change":"2024-03-20T09:06:02.926003+0000","last_active":"2024-03-20T09:09:19.053306+0000","last_peered":"2024-03-20T09:09:19.053306+0000","last_clean":"2024-03-20T09:09:19.053306+0000","last_became_active":"2024-03-20T09:06:02.925687+0000","last_became_peered":"2024-03-20T09:06:02.925687+0000","last_unstale":"2024-03-20T09:09:19.053306+0000","last_undegraded":"2024-03-20T09:09:19.053306+0000","last_fullsized":"2024-03-20T09:09:19.053306+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:23.383145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:23.383145+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:09:21.691348+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,10],"acting":[11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"407'6034","reported_seq":14056,"reported_epoch":407,"state":"active+clean+remapped","last_fresh":"2024-03-20T09:09:15.030028+0000","last_change":"2024-03-20T09:08:29.097299+0000","last_active":"2024-03-20T09:09:15.030028+0000","last_peered":"2024-03-20T09:09:15.030028+0000","last_clean":"2024-03-20T09:09:15.030028+0000","last_became_active":"2024-03-20T09:08:29.090030+0000","last_became_peered":"2024-03-20T09:08:29.090030+0000","last_unstale":"2024-03-20T09:09:15.030028+0000","last_undegraded":"2024-03-20T09:09:15.030028+0000","last_fullsized":"2024-03-20T09:09:15.030028+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":6034,"log_dups_size":0,"ondisk_log_size":6034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:09:17.500634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131399680,"num_objects":2005,"num_object_clones":0,"num_object_copies":6015,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2005,"num_objects_unfound":0,"num_objects_dirty":2005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4259,"num_write_kb":200448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3456,"num_bytes_recovered":168296448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1,4],"avail_no_missing":["6","1","4"],"object_location_counts":[{"shards":"1,4,6","objects":2005}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"20'137","reported_seq":1043,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-03-20T09:09:07.025658+0000","last_change":"2024-03-20T09:07:52.618129+0000","last_active":"2024-03-20T09:09:07.025658+0000","last_peered":"2024-03-20T09:09:07.025658+0000","last_clean":"2024-03-20T09:09:07.025658+0000","last_became_active":"2024-03-20T09:07:52.617726+0000","last_became_peered":"2024-03-20T09:07:52.617726+0000","last_unstale":"2024-03-20T09:09:07.025658+0000","last_undegraded":"2024-03-20T09:09:07.025658+0000","last_fullsized":"2024-03-20T09:09:07.025658+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:03:59.584956+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:03:59.584956+0000","last_clean_scrub_stamp":"2024-03-20T09:03:59.584956+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"410'5921","reported_seq":7769,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-20T09:09:19.236183+0000","last_change":"2024-03-20T09:07:50.197027+0000","last_active":"2024-03-20T09:09:19.236183+0000","last_peered":"2024-03-20T09:09:19.236183+0000","last_clean":"2024-03-20T09:09:19.236183+0000","last_became_active":"2024-03-20T09:03:30.926683+0000","last_became_peered":"2024-03-20T09:03:30.926683+0000","last_unstale":"2024-03-20T09:09:19.236183+0000","last_undegraded":"2024-03-20T09:09:19.236183+0000","last_fullsized":"2024-03-20T09:09:19.236183+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"366'5661","last_scrub_stamp":"2024-03-20T09:07:50.196877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:07:50.196877+0000","objects_scrubbed":2239,"log_size":5921,"log_dups_size":0,"ondisk_log_size":5921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":216,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129695744,"num_objects":1979,"num_object_clones":0,"num_object_copies":5937,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":260,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,1],"acting":[11,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1","version":"20'137","reported_seq":758,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-20T09:09:16.048088+0000","last_change":"2024-03-20T09:07:03.025487+0000","last_active":"2024-03-20T09:09:16.048088+0000","last_peered":"2024-03-20T09:09:16.048088+0000","last_clean":"2024-03-20T09:09:16.048088+0000","last_became_active":"2024-03-20T09:06:02.923630+0000","last_became_peered":"2024-03-20T09:06:02.923630+0000","last_unstale":"2024-03-20T09:09:16.048088+0000","last_undegraded":"2024-03-20T09:09:16.048088+0000","last_fullsized":"2024-03-20T09:09:16.048088+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:07:03.025368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:07:03.025368+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2],"acting":[10,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3","version":"407'6054","reported_seq":12729,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-03-20T09:09:15.829486+0000","last_change":"2024-03-20T09:08:13.002350+0000","last_active":"2024-03-20T09:09:15.829486+0000","last_peered":"2024-03-20T09:09:15.829486+0000","last_clean":"2024-03-20T09:09:15.829486+0000","last_became_active":"2024-03-20T09:08:12.992488+0000","last_became_peered":"2024-03-20T09:08:12.992488+0000","last_unstale":"2024-03-20T09:09:15.829486+0000","last_undegraded":"2024-03-20T09:09:15.829486+0000","last_fullsized":"2024-03-20T09:09:15.829486+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":6054,"log_dups_size":0,"ondisk_log_size":6054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:09:20.387064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125992960,"num_objects":1922,"num_object_clones":0,"num_object_copies":5766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4260,"num_write_kb":197856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3119,"num_bytes_recovered":145162240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,11],"acting":[0,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"20'137","reported_seq":725,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-03-20T09:09:06.973388+0000","last_change":"2024-03-20T09:07:36.949270+0000","last_active":"2024-03-20T09:09:06.973388+0000","last_peered":"2024-03-20T09:09:06.973388+0000","last_clean":"2024-03-20T09:04:29.191837+0000","last_became_active":"2024-03-20T09:07:36.949270+0000","last_became_peered":"2024-03-20T09:07:36.949270+0000","last_unstale":"2024-03-20T09:09:06.973388+0000","last_undegraded":"2024-03-20T09:09:06.973388+0000","last_fullsized":"2024-03-20T09:07:36.946730+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:21.734957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:21.734957+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:09:08.508668+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10],"avail_no_missing":["6","10"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"409'6035","reported_seq":7785,"reported_epoch":409,"state":"active+undersized+degraded","last_fresh":"2024-03-20T09:09:17.491210+0000","last_change":"2024-03-20T09:04:34.737379+0000","last_active":"2024-03-20T09:09:17.491210+0000","last_peered":"2024-03-20T09:09:17.491210+0000","last_clean":"2024-03-20T09:04:33.579072+0000","last_became_active":"2024-03-20T09:04:34.737379+0000","last_became_peered":"2024-03-20T09:04:34.737379+0000","last_unstale":"2024-03-20T09:09:17.491210+0000","last_undegraded":"2024-03-20T09:04:34.733476+0000","last_fullsized":"2024-03-20T09:04:34.733363+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"253'4200","last_scrub_stamp":"2024-03-20T09:04:00.025798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:04:00.025798+0000","objects_scrubbed":2428,"log_size":6035,"log_dups_size":0,"ondisk_log_size":6035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:09:21.929754+0000","scrub_duration":666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130875392,"num_objects":1997,"num_object_clones":0,"num_object_copies":5991,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1839,"num_write_kb":45056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,6","objects":1997}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"20'137","reported_seq":607,"reported_epoch":407,"state":"active+clean+remapped","last_fresh":"2024-03-20T09:09:07.025923+0000","last_change":"2024-03-20T09:07:36.950563+0000","last_active":"2024-03-20T09:09:07.025923+0000","last_peered":"2024-03-20T09:09:07.025923+0000","last_clean":"2024-03-20T09:09:07.025923+0000","last_became_active":"2024-03-20T09:07:36.950103+0000","last_became_peered":"2024-03-20T09:07:36.950103+0000","last_unstale":"2024-03-20T09:09:07.025923+0000","last_undegraded":"2024-03-20T09:09:07.025923+0000","last_fullsized":"2024-03-20T09:09:07.025923+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:24.936556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:24.936556+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:09:09.503140+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2,11],"avail_no_missing":["0","2","11"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"408'5934","reported_seq":6429,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-20T09:09:16.050442+0000","last_change":"2024-03-20T09:06:58.210889+0000","last_active":"2024-03-20T09:09:16.050442+0000","last_peered":"2024-03-20T09:09:16.050442+0000","last_clean":"2024-03-20T09:09:16.050442+0000","last_became_active":"2024-03-20T09:03:30.910402+0000","last_became_peered":"2024-03-20T09:03:30.910402+0000","last_unstale":"2024-03-20T09:09:16.050442+0000","last_undegraded":"2024-03-20T09:09:16.050442+0000","last_fullsized":"2024-03-20T09:09:16.050442+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'5452","last_scrub_stamp":"2024-03-20T09:06:58.210664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:06:58.210664+0000","objects_scrubbed":2397,"log_size":5934,"log_dups_size":0,"ondisk_log_size":5934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":272,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125435904,"num_objects":1914,"num_object_clones":0,"num_object_copies":5742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1084,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"20'137","reported_seq":484,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-20T09:09:16.050717+0000","last_change":"2024-03-20T09:07:52.604147+0000","last_active":"2024-03-20T09:09:16.050717+0000","last_peered":"2024-03-20T09:09:16.050717+0000","last_clean":"2024-03-20T09:09:16.050717+0000","last_became_active":"2024-03-20T09:07:52.603796+0000","last_became_peered":"2024-03-20T09:07:52.603796+0000","last_unstale":"2024-03-20T09:09:16.050717+0000","last_undegraded":"2024-03-20T09:09:16.050717+0000","last_fullsized":"2024-03-20T09:09:16.050717+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:03:55.906959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:03:55.906959+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,2],"acting":[8,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.c","version":"407'12301","reported_seq":24016,"reported_epoch":407,"state":"active+clean+remapped","last_fresh":"2024-03-20T09:09:15.206938+0000","last_change":"2024-03-20T09:06:08.944363+0000","last_active":"2024-03-20T09:09:15.206938+0000","last_peered":"2024-03-20T09:09:15.206938+0000","last_clean":"2024-03-20T09:09:15.206938+0000","last_became_active":"2024-03-20T09:06:08.928778+0000","last_became_peered":"2024-03-20T09:06:08.928778+0000","last_unstale":"2024-03-20T09:09:15.206938+0000","last_undegraded":"2024-03-20T09:09:15.206938+0000","last_fullsized":"2024-03-20T09:09:15.206938+0000","mapping_epoch":313,"log_start":"225'2300","ondisk_log_start":"225'2300","created":204,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"270'9729","last_scrub_stamp":"2024-03-20T09:04:24.105978+0000","last_deep_scrub":"270'9729","last_deep_scrub_stamp":"2024-03-20T09:04:24.105978+0000","last_clean_scrub_stamp":"2024-03-20T09:04:24.105978+0000","objects_scrubbed":5255,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":26,"scrub_schedule":"queued for scrub","scrub_duration":25793,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271646720,"num_objects":4145,"num_object_clones":0,"num_object_copies":12435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4145,"num_objects_unfound":0,"num_objects_dirty":4145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8453,"num_write_kb":403136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5528,"num_bytes_recovered":362283008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10],"acting":[6,10,3],"avail_no_missing":["6","3","10"],"object_location_counts":[{"shards":"3,6,10","objects":4145}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"20'137","reported_seq":739,"reported_epoch":407,"state":"active+clean+remapped","last_fresh":"2024-03-20T09:09:06.973152+0000","last_change":"2024-03-20T09:07:52.600973+0000","last_active":"2024-03-20T09:09:06.973152+0000","last_peered":"2024-03-20T09:09:06.973152+0000","last_clean":"2024-03-20T09:09:06.973152+0000","last_became_active":"2024-03-20T09:07:52.600508+0000","last_became_peered":"2024-03-20T09:07:52.600508+0000","last_unstale":"2024-03-20T09:09:06.973152+0000","last_undegraded":"2024-03-20T09:09:06.973152+0000","last_fullsized":"2024-03-20T09:09:06.973152+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:21.734957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:21.734957+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:09:08.508697+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8,4],"avail_no_missing":["6","4","8"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"407'8332","reported_seq":15223,"reported_epoch":407,"state":"active+recovering+degraded+remapped","last_fresh":"2024-03-20T09:09:15.461214+0000","last_change":"2024-03-20T09:04:47.242963+0000","last_active":"2024-03-20T09:09:15.461214+0000","last_peered":"2024-03-20T09:09:15.461214+0000","last_clean":"2024-03-20T09:04:10.072458+0000","last_became_active":"2024-03-20T09:04:46.620359+0000","last_became_peered":"2024-03-20T09:04:46.620359+0000","last_unstale":"2024-03-20T09:09:15.461214+0000","last_undegraded":"2024-03-20T09:04:46.424109+0000","last_fullsized":"2024-03-20T09:09:15.461214+0000","mapping_epoch":275,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'5229","last_scrub_stamp":"2024-03-20T09:04:09.606259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:04:09.606259+0000","objects_scrubbed":5193,"log_size":8332,"log_dups_size":0,"ondisk_log_size":8332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264962072,"num_objects":4044,"num_object_clones":0,"num_object_copies":12132,"num_objects_missing_on_primary":2039,"num_objects_missing":2039,"num_objects_degraded":4044,"num_objects_misplaced":2039,"num_objects_unfound":0,"num_objects_dirty":4044,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8332,"num_write_kb":395969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3203,"num_bytes_recovered":209911808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7,9],"avail_no_missing":["7"],"object_location_counts":[{"shards":"6,7","objects":944},{"shards":"6,7,9","objects":1060},{"shards":"7,9","objects":2039}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"20'137","reported_seq":780,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-03-20T09:09:07.025864+0000","last_change":"2024-03-20T09:07:36.950317+0000","last_active":"2024-03-20T09:09:07.025864+0000","last_peered":"2024-03-20T09:09:07.025864+0000","last_clean":"2024-03-20T09:04:29.200818+0000","last_became_active":"2024-03-20T09:07:36.950317+0000","last_became_peered":"2024-03-20T09:07:36.950317+0000","last_unstale":"2024-03-20T09:09:07.025864+0000","last_undegraded":"2024-03-20T09:09:07.025864+0000","last_fullsized":"2024-03-20T09:07:36.947977+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:04:04.682759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:04:04.682759+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:09:09.503156+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163840,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"407'8428","reported_seq":35446,"reported_epoch":407,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-20T09:09:16.015985+0000","last_change":"2024-03-20T09:08:31.698000+0000","last_active":"2024-03-20T09:09:16.015985+0000","last_peered":"2024-03-20T09:09:16.015985+0000","last_clean":"2024-03-20T09:04:10.062034+0000","last_became_active":"2024-03-20T09:08:30.542785+0000","last_became_peered":"2024-03-20T09:08:30.542785+0000","last_unstale":"2024-03-20T09:09:16.015985+0000","last_undegraded":"2024-03-20T09:08:15.948124+0000","last_fullsized":"2024-03-20T09:07:50.566231+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":8428,"log_dups_size":0,"ondisk_log_size":8428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256638976,"num_objects":3916,"num_object_clones":0,"num_object_copies":11748,"num_objects_missing_on_primary":1830,"num_objects_missing":1830,"num_objects_degraded":297,"num_objects_misplaced":5455,"num_objects_unfound":0,"num_objects_dirty":3916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8428,"num_write_kb":395008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11742,"num_bytes_recovered":724041728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,8],"acting":[0,10],"avail_no_missing":["4","9","11"],"object_location_counts":[{"shards":"0,3,4,9,10,11","objects":10},{"shards":"0,3,4,10,11","objects":912},{"shards":"0,4,9,10,11","objects":1455},{"shards":"0,10","objects":-297},{"shards":"3,4,9,10,11","objects":1215},{"shards":"4,9,10,11","objects":324}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"20'137","reported_seq":334,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-03-20T09:09:06.974260+0000","last_change":"2024-03-20T09:08:56.938853+0000","last_active":"2024-03-20T09:09:06.974260+0000","last_peered":"2024-03-20T09:09:06.974260+0000","last_clean":"2024-03-20T09:09:06.974260+0000","last_became_active":"2024-03-20T09:08:00.708284+0000","last_became_peered":"2024-03-20T09:08:00.708284+0000","last_unstale":"2024-03-20T09:09:06.974260+0000","last_undegraded":"2024-03-20T09:09:06.974260+0000","last_fullsized":"2024-03-20T09:09:06.974260+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:08:56.938721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:08:56.938721+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:10:22.571428+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,11],"acting":[1,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"410'8347","reported_seq":21147,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-20T09:09:19.412626+0000","last_change":"2024-03-20T09:07:49.396021+0000","last_active":"2024-03-20T09:09:19.412626+0000","last_peered":"2024-03-20T09:09:19.412626+0000","last_clean":"2024-03-20T09:09:19.412626+0000","last_became_active":"2024-03-20T09:06:59.023790+0000","last_became_peered":"2024-03-20T09:06:59.023790+0000","last_unstale":"2024-03-20T09:09:19.412626+0000","last_undegraded":"2024-03-20T09:09:19.412626+0000","last_fullsized":"2024-03-20T09:09:19.412626+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":204,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'7842","last_scrub_stamp":"2024-03-20T09:07:49.395885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:07:49.395885+0000","objects_scrubbed":4468,"log_size":8347,"log_dups_size":0,"ondisk_log_size":8347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":457,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259719168,"num_objects":3963,"num_object_clones":0,"num_object_copies":11889,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8347,"num_write_kb":393920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6155,"num_bytes_recovered":345112576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,10],"acting":[11,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.12","version":"20'137","reported_seq":980,"reported_epoch":407,"state":"active+undersized","last_fresh":"2024-03-20T09:09:06.974743+0000","last_change":"2024-03-20T09:04:34.751346+0000","last_active":"2024-03-20T09:09:06.974743+0000","last_peered":"2024-03-20T09:09:06.974743+0000","last_clean":"2024-03-20T09:03:59.001853+0000","last_became_active":"2024-03-20T09:04:34.751346+0000","last_became_peered":"2024-03-20T09:04:34.751346+0000","last_unstale":"2024-03-20T09:09:06.974743+0000","last_undegraded":"2024-03-20T09:09:06.974743+0000","last_fullsized":"2024-03-20T09:04:34.735354+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:04:05.319747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:04:05.319747+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:09:18.955038+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"407'5950","reported_seq":25466,"reported_epoch":407,"state":"active+clean+remapped","last_fresh":"2024-03-20T09:09:14.839445+0000","last_change":"2024-03-20T09:08:11.991828+0000","last_active":"2024-03-20T09:09:14.839445+0000","last_peered":"2024-03-20T09:09:14.839445+0000","last_clean":"2024-03-20T09:09:14.839445+0000","last_became_active":"2024-03-20T09:08:11.985281+0000","last_became_peered":"2024-03-20T09:08:11.985281+0000","last_unstale":"2024-03-20T09:09:14.839445+0000","last_undegraded":"2024-03-20T09:09:14.839445+0000","last_fullsized":"2024-03-20T09:09:14.839445+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":5950,"log_dups_size":0,"ondisk_log_size":5950,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:09:19.505001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130940928,"num_objects":1998,"num_object_clones":0,"num_object_copies":5994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1998,"num_objects_unfound":0,"num_objects_dirty":1998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4206,"num_write_kb":198528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8292,"num_bytes_recovered":495484928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0,2],"avail_no_missing":["6","0","2"],"object_location_counts":[{"shards":"0,2,6","objects":1998}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.13","version":"20'137","reported_seq":724,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-03-20T09:09:06.973876+0000","last_change":"2024-03-20T09:09:01.952700+0000","last_active":"2024-03-20T09:09:06.973876+0000","last_peered":"2024-03-20T09:09:06.973876+0000","last_clean":"2024-03-20T09:09:06.973876+0000","last_became_active":"2024-03-20T09:04:04.103645+0000","last_became_peered":"2024-03-20T09:04:04.103645+0000","last_unstale":"2024-03-20T09:09:06.973876+0000","last_undegraded":"2024-03-20T09:09:06.973876+0000","last_fullsized":"2024-03-20T09:09:06.973876+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:09:01.952591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:09:01.952591+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:10:08.683018+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"407'6032","reported_seq":16564,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-20T09:09:16.048030+0000","last_change":"2024-03-20T09:08:16.042688+0000","last_active":"2024-03-20T09:09:16.048030+0000","last_peered":"2024-03-20T09:09:16.048030+0000","last_clean":"2024-03-20T09:09:16.048030+0000","last_became_active":"2024-03-20T09:08:16.036521+0000","last_became_peered":"2024-03-20T09:08:16.036521+0000","last_unstale":"2024-03-20T09:09:16.048030+0000","last_undegraded":"2024-03-20T09:09:16.048030+0000","last_fullsized":"2024-03-20T09:09:16.048030+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":5,"last_scrub":"345'5607","last_scrub_stamp":"2024-03-20T09:07:01.247738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:07:01.247738+0000","objects_scrubbed":2440,"log_size":6032,"log_dups_size":0,"ondisk_log_size":6032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":262,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131989504,"num_objects":2014,"num_object_clones":0,"num_object_copies":6042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2158,"num_write_kb":65984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3141,"num_bytes_recovered":146604032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,0],"acting":[10,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.10","version":"20'137","reported_seq":510,"reported_epoch":408,"state":"active+undersized","last_fresh":"2024-03-20T09:09:16.048067+0000","last_change":"2024-03-20T09:04:34.735801+0000","last_active":"2024-03-20T09:09:16.048067+0000","last_peered":"2024-03-20T09:09:16.048067+0000","last_clean":"2024-03-20T08:54:29.262652+0000","last_became_active":"2024-03-20T09:04:34.735801+0000","last_became_peered":"2024-03-20T09:04:34.735801+0000","last_unstale":"2024-03-20T09:09:16.048067+0000","last_undegraded":"2024-03-20T09:09:16.048067+0000","last_fullsized":"2024-03-20T09:04:34.732688+0000","mapping_epoch":274,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:04:04.319316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:04:04.319316+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:09:21.138541+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":8,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":48084,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7],"acting":[10,7],"avail_no_missing":["10","7"],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.12","version":"407'5976","reported_seq":21367,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-03-20T09:09:15.056292+0000","last_change":"2024-03-20T09:06:44.957783+0000","last_active":"2024-03-20T09:09:15.056292+0000","last_peered":"2024-03-20T09:09:15.056292+0000","last_clean":"2024-03-20T09:09:15.056292+0000","last_became_active":"2024-03-20T09:06:44.949696+0000","last_became_peered":"2024-03-20T09:06:44.949696+0000","last_unstale":"2024-03-20T09:09:15.056292+0000","last_undegraded":"2024-03-20T09:09:15.056292+0000","last_fullsized":"2024-03-20T09:09:15.056292+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":5,"last_scrub":"285'4956","last_scrub_stamp":"2024-03-20T09:04:50.135501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:04:50.135501+0000","objects_scrubbed":3023,"log_size":5976,"log_dups_size":0,"ondisk_log_size":5976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":356,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131203072,"num_objects":2002,"num_object_clones":0,"num_object_copies":6006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1021,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3104,"num_bytes_recovered":169148416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"20'137","reported_seq":587,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-03-20T09:09:07.025790+0000","last_change":"2024-03-20T09:06:02.928303+0000","last_active":"2024-03-20T09:09:07.025790+0000","last_peered":"2024-03-20T09:09:07.025790+0000","last_clean":"2024-03-20T09:09:07.025790+0000","last_became_active":"2024-03-20T09:06:02.927902+0000","last_became_peered":"2024-03-20T09:06:02.927902+0000","last_unstale":"2024-03-20T09:09:07.025790+0000","last_undegraded":"2024-03-20T09:09:07.025790+0000","last_fullsized":"2024-03-20T09:09:07.025790+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:04:42.894571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:04:42.894571+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1],"acting":[0,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"409'6036","reported_seq":18207,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-20T09:09:18.127686+0000","last_change":"2024-03-20T09:08:54.067778+0000","last_active":"2024-03-20T09:09:18.127686+0000","last_peered":"2024-03-20T09:09:18.127686+0000","last_clean":"2024-03-20T09:09:18.127686+0000","last_became_active":"2024-03-20T09:08:12.968732+0000","last_became_peered":"2024-03-20T09:08:12.968732+0000","last_unstale":"2024-03-20T09:09:18.127686+0000","last_undegraded":"2024-03-20T09:09:18.127686+0000","last_fullsized":"2024-03-20T09:09:18.127686+0000","mapping_epoch":383,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":384,"parent":"0.0","parent_split_bits":5,"last_scrub":"400'5959","last_scrub_stamp":"2024-03-20T09:08:54.067100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:08:54.067100+0000","objects_scrubbed":2080,"log_size":6036,"log_dups_size":0,"ondisk_log_size":6036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:10:04.035425+0000","scrub_duration":235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131268608,"num_objects":2003,"num_object_clones":0,"num_object_copies":6009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.16","version":"20'137","reported_seq":412,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-03-20T09:09:06.973871+0000","last_change":"2024-03-20T09:08:56.688563+0000","last_active":"2024-03-20T09:09:06.973871+0000","last_peered":"2024-03-20T09:09:06.973871+0000","last_clean":"2024-03-20T09:09:06.973871+0000","last_became_active":"2024-03-20T09:07:41.962567+0000","last_became_peered":"2024-03-20T09:07:41.962567+0000","last_unstale":"2024-03-20T09:09:06.973871+0000","last_undegraded":"2024-03-20T09:09:06.973871+0000","last_fullsized":"2024-03-20T09:09:06.973871+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:08:56.688407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:08:56.688407+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:10:24.478583+0000","scrub_duration":1738,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,11],"acting":[1,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"408'5901","reported_seq":17839,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-20T09:09:16.413510+0000","last_change":"2024-03-20T09:08:10.016943+0000","last_active":"2024-03-20T09:09:16.413510+0000","last_peered":"2024-03-20T09:09:16.413510+0000","last_clean":"2024-03-20T09:09:16.413510+0000","last_became_active":"2024-03-20T09:08:10.010857+0000","last_became_peered":"2024-03-20T09:08:10.010857+0000","last_unstale":"2024-03-20T09:09:16.413510+0000","last_undegraded":"2024-03-20T09:09:16.413510+0000","last_fullsized":"2024-03-20T09:09:16.413510+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":5901,"log_dups_size":0,"ondisk_log_size":5901,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:09:21.098520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128024576,"num_objects":1954,"num_object_clones":0,"num_object_copies":5862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4164,"num_write_kb":195744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":263192576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.17","version":"20'137","reported_seq":729,"reported_epoch":408,"state":"active+clean+remapped","last_fresh":"2024-03-20T09:09:16.050844+0000","last_change":"2024-03-20T09:07:41.959847+0000","last_active":"2024-03-20T09:09:16.050844+0000","last_peered":"2024-03-20T09:09:16.050844+0000","last_clean":"2024-03-20T09:09:16.050844+0000","last_became_active":"2024-03-20T09:07:41.959382+0000","last_became_peered":"2024-03-20T09:07:41.959382+0000","last_unstale":"2024-03-20T09:09:16.050844+0000","last_undegraded":"2024-03-20T09:09:16.050844+0000","last_fullsized":"2024-03-20T09:09:16.050844+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:02:21.734957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:02:21.734957+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10],"acting":[8,10,6],"avail_no_missing":["8","6","10"],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.15","version":"410'6106","reported_seq":19099,"reported_epoch":410,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-20T09:09:19.681213+0000","last_change":"2024-03-20T09:08:58.294847+0000","last_active":"2024-03-20T09:09:19.681213+0000","last_peered":"2024-03-20T09:09:19.681213+0000","last_clean":"2024-03-20T09:03:31.866916+0000","last_became_active":"2024-03-20T09:07:59.224446+0000","last_became_peered":"2024-03-20T09:07:59.224446+0000","last_unstale":"2024-03-20T09:09:19.681213+0000","last_undegraded":"2024-03-20T09:09:19.681213+0000","last_fullsized":"2024-03-20T09:07:50.565002+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:02:48.874894+0000","objects_scrubbed":0,"log_size":6106,"log_dups_size":0,"ondisk_log_size":6106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137134080,"num_objects":2093,"num_object_clones":0,"num_object_copies":6279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1166,"num_objects_unfound":0,"num_objects_dirty":2093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4321,"num_write_kb":205216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3122,"num_bytes_recovered":204439552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,6],"acting":[11,0],"avail_no_missing":["11","2","3"],"object_location_counts":[{"shards":"0,11","objects":1704},{"shards":"4,7,8,11","objects":1},{"shards":"4,7,11","objects":388}],"blocked_by":[],"up_primary":8,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.14","version":"20'137","reported_seq":804,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-20T09:09:17.080834+0000","last_change":"2024-03-20T09:09:01.787351+0000","last_active":"2024-03-20T09:09:17.080834+0000","last_peered":"2024-03-20T09:09:17.080834+0000","last_clean":"2024-03-20T09:09:17.080834+0000","last_became_active":"2024-03-20T09:04:14.208448+0000","last_became_peered":"2024-03-20T09:04:14.208448+0000","last_unstale":"2024-03-20T09:09:17.080834+0000","last_undegraded":"2024-03-20T09:09:17.080834+0000","last_fullsized":"2024-03-20T09:09:17.080834+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:09:01.787206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:09:01.787206+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:10:27.504550+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"407'7989","reported_seq":19996,"reported_epoch":407,"state":"active+clean+remapped+scrubbing+deep","last_fresh":"2024-03-20T09:09:15.044785+0000","last_change":"2024-03-20T09:09:09.903444+0000","last_active":"2024-03-20T09:09:15.044785+0000","last_peered":"2024-03-20T09:09:15.044785+0000","last_clean":"2024-03-20T09:09:15.044785+0000","last_became_active":"2024-03-20T09:06:18.946699+0000","last_became_peered":"2024-03-20T09:06:18.946699+0000","last_unstale":"2024-03-20T09:09:15.044785+0000","last_undegraded":"2024-03-20T09:09:15.044785+0000","last_fullsized":"2024-03-20T09:09:15.044785+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":5,"last_scrub":"366'5814","last_scrub_stamp":"2024-03-20T09:07:50.167856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:07:50.167856+0000","objects_scrubbed":1911,"log_size":7989,"log_dups_size":0,"ondisk_log_size":7989,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"deep scrubbing for 5s","scrub_duration":242,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135069696,"num_objects":2061,"num_object_clones":0,"num_object_copies":6183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2061,"num_objects_unfound":0,"num_objects_dirty":2061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1810,"num_write_kb":45248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3164,"num_bytes_recovered":178585600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2,7],"avail_no_missing":["1","2","7"],"object_location_counts":[{"shards":"1,2,7","objects":2061}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"20'137","reported_seq":340,"reported_epoch":408,"state":"active+clean+remapped","last_fresh":"2024-03-20T09:09:16.050475+0000","last_change":"2024-03-20T09:07:48.964270+0000","last_active":"2024-03-20T09:09:16.050475+0000","last_peered":"2024-03-20T09:09:16.050475+0000","last_clean":"2024-03-20T09:09:16.050475+0000","last_became_active":"2024-03-20T09:07:41.961768+0000","last_became_peered":"2024-03-20T09:07:41.961768+0000","last_unstale":"2024-03-20T09:09:16.050475+0000","last_undegraded":"2024-03-20T09:09:16.050475+0000","last_fullsized":"2024-03-20T09:09:16.050475+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:07:48.964057+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:07:06.480160+0000","last_clean_scrub_stamp":"2024-03-20T09:07:48.964057+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1],"acting":[8,1,2],"avail_no_missing":["8","1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.17","version":"407'5972","reported_seq":19616,"reported_epoch":407,"state":"active+clean+remapped","last_fresh":"2024-03-20T09:09:15.914393+0000","last_change":"2024-03-20T09:06:46.990481+0000","last_active":"2024-03-20T09:09:15.914393+0000","last_peered":"2024-03-20T09:09:15.914393+0000","last_clean":"2024-03-20T09:09:15.914393+0000","last_became_active":"2024-03-20T09:06:46.981557+0000","last_became_peered":"2024-03-20T09:06:46.981557+0000","last_unstale":"2024-03-20T09:09:15.914393+0000","last_undegraded":"2024-03-20T09:09:15.914393+0000","last_fullsized":"2024-03-20T09:09:15.914393+0000","mapping_epoch":335,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":5,"last_scrub":"251'4129","last_scrub_stamp":"2024-03-20T09:03:57.865687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:03:57.865687+0000","objects_scrubbed":2325,"log_size":5972,"log_dups_size":0,"ondisk_log_size":5972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135266304,"num_objects":2064,"num_object_clones":0,"num_object_copies":6192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2064,"num_objects_unfound":0,"num_objects_dirty":2064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1851,"num_write_kb":50880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3120,"num_bytes_recovered":173342720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1,6],"avail_no_missing":["0","1","6"],"object_location_counts":[{"shards":"0,1,6","objects":2064}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1a","version":"20'137","reported_seq":335,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-03-20T09:09:06.974338+0000","last_change":"2024-03-20T09:08:52.989685+0000","last_active":"2024-03-20T09:09:06.974338+0000","last_peered":"2024-03-20T09:09:06.974338+0000","last_clean":"2024-03-20T09:09:06.974338+0000","last_became_active":"2024-03-20T09:07:52.605174+0000","last_became_peered":"2024-03-20T09:07:52.605174+0000","last_unstale":"2024-03-20T09:09:06.974338+0000","last_undegraded":"2024-03-20T09:09:06.974338+0000","last_fullsized":"2024-03-20T09:09:06.974338+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:08:52.989586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:08:52.989586+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:10:12.001540+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"407'5883","reported_seq":19628,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-03-20T09:09:14.586126+0000","last_change":"2024-03-20T09:08:59.172806+0000","last_active":"2024-03-20T09:09:14.586126+0000","last_peered":"2024-03-20T09:09:14.586126+0000","last_clean":"2024-03-20T09:09:14.586126+0000","last_became_active":"2024-03-20T09:06:24.988319+0000","last_became_peered":"2024-03-20T09:06:24.988319+0000","last_unstale":"2024-03-20T09:09:14.586126+0000","last_undegraded":"2024-03-20T09:09:14.586126+0000","last_fullsized":"2024-03-20T09:09:14.586126+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":5,"last_scrub":"402'5834","last_scrub_stamp":"2024-03-20T09:08:59.172617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:02:48.874894+0000","last_clean_scrub_stamp":"2024-03-20T09:08:59.172617+0000","objects_scrubbed":2035,"log_size":5883,"log_dups_size":0,"ondisk_log_size":5883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:10:16.184206+0000","scrub_duration":242,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130154496,"num_objects":1986,"num_object_clones":0,"num_object_copies":5958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1011,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3042,"num_bytes_recovered":170590208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,8],"acting":[1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":4217667608,"num_objects":64357,"num_object_clones":0,"num_object_copies":193071,"num_objects_missing_on_primary":3869,"num_objects_missing":3869,"num_objects_degraded":8381,"num_objects_misplaced":26603,"num_objects_unfound":0,"num_objects_dirty":64357,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":97540,"num_write_kb":4112609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110452,"num_bytes_recovered":6349586432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16145129472,"data_stored":16145121328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":176961,"ondisk_log_size":176961,"up":68,"acting":73,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":780099,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":24,"num_write_kb":401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":397732,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2007040,"data_stored":1989280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4247,"ondisk_log_size":4247,"up":82,"acting":88,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860575,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2198068,"kb_used_data":1996136,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":91547084,"statfs":{"total":95995035648,"available":93744214016,"internally_reserved":0,"allocated":2044043264,"data_stored":2043120274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":206764484},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":89,"seq":382252089508,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1366312,"kb_used_data":1168540,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":92378840,"statfs":{"total":95995035648,"available":94595932160,"internally_reserved":0,"allocated":1196584960,"data_stored":1195665969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":202504640},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":0,"seq":343597383852,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":70,"seq":300647710896,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1186544,"kb_used_data":1002660,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":92558608,"statfs":{"total":95995035648,"available":94780014592,"internally_reserved":0,"allocated":1026723840,"data_stored":1025801297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":188283334},"hb_peers":[0,1,2,3,4,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":60,"seq":257698037941,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1367396,"kb_used_data":1039132,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92377756,"statfs":{"total":95995035648,"available":94594822144,"internally_reserved":0,"allocated":1064071168,"data_stored":1063148502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":336132554},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332280,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2046684,"kb_used_data":1835028,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":91698468,"statfs":{"total":95995035648,"available":93899231232,"internally_reserved":0,"allocated":1879068672,"data_stored":1878149558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":216725958},"hb_peers":[0,1,2,3,4,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575060,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1991576,"kb_used_data":1685520,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":91753576,"statfs":{"total":95995035648,"available":93955661824,"internally_reserved":0,"allocated":1725972480,"data_stored":1725057438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":313391558},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,5,8,13],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738580,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1790168,"kb_used_data":1542416,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":91954984,"statfs":{"total":95995035648,"available":94161903616,"internally_reserved":0,"allocated":1579433984,"data_stored":1578518942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,0,2,0,1,2,4,12,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411534,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1173248,"kb_used_data":1015728,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":92571904,"statfs":{"total":95995035648,"available":94793629696,"internally_reserved":0,"allocated":1040105472,"data_stored":1039178930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161282501},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":111669149897,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953670,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659271,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2258332,"kb_used_data":2123028,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":91486820,"statfs":{"total":95995035648,"available":93682503680,"internally_reserved":0,"allocated":2173980672,"data_stored":2173328093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1577648128,"data_stored":1577648128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1722941440,"data_stored":1722941440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1036845056,"data_stored":1036845056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":816906240,"data_stored":816906240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1073152000,"data_stored":1073152000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2174091264,"data_stored":2174091264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1876299776,"data_stored":1876295704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1059983360,"data_stored":1059979288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1025638400,"data_stored":1025638400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":550502400,"data_stored":550502400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1194000384,"data_stored":1194000384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2037121024,"data_stored":2037121024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:09:22.728 INFO:tasks.thrashosds.thrasher:Setting 1.b to [6, 0, 7, 1, 8, 10] 2024-03-20T09:09:22.728 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '1.b', '6', '0', '7', '1', '8', '10'] 2024-03-20T09:09:22.728 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pg-upmap-items 1.b 6 0 7 1 8 10 2024-03-20T09:09:22.735 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:09:22.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:22.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:09:22.839 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:09:22.869 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:22.947 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-20T09:09:23.252 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:09:23.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:23 smithi115 ceph-mon[72476]: pgmap v993: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 3.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 117 op/s; 8335/191832 objects degraded (4.345%); 26300/191832 objects misplaced (13.710%); 4.3 MiB/s, 68 objects/s recovering 2024-03-20T09:09:23.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:23 smithi122 ceph-mon[73495]: pgmap v993: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 3.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 117 op/s; 8335/191832 objects degraded (4.345%); 26300/191832 objects misplaced (13.710%); 4.3 MiB/s, 68 objects/s recovering 2024-03-20T09:09:23.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:23 smithi026 ceph-mon[68309]: pgmap v993: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 3.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 117 op/s; 8335/191832 objects degraded (4.345%); 26300/191832 objects misplaced (13.710%); 4.3 MiB/s, 68 objects/s recovering 2024-03-20T09:09:24.126 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:24 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:09:24.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:24 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:09:24.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:24 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:09:24.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:24 smithi115 ceph-mon[72476]: osdmap e412: 12 total, 11 up, 9 in 2024-03-20T09:09:24.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:24 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:09:24.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:24 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:09:24.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:24 smithi122 ceph-mon[73495]: osdmap e412: 12 total, 11 up, 9 in 2024-03-20T09:09:24.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:24 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:09:24.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:24 smithi026 ceph-mon[68309]: osdmap e412: 12 total, 11 up, 9 in 2024-03-20T09:09:25.228 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:09:25.228 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:09:25.255 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:25 smithi115 ceph-mon[72476]: pgmap v996: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 3.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 95 op/s; 8309/191673 objects degraded (4.335%); 26300/191673 objects misplaced (13.721%); 4.0 MiB/s, 63 objects/s recovering 2024-03-20T09:09:25.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:25 smithi122 ceph-mon[73495]: pgmap v996: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 3.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 95 op/s; 8309/191673 objects degraded (4.335%); 26300/191673 objects misplaced (13.721%); 4.0 MiB/s, 63 objects/s recovering 2024-03-20T09:09:25.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:25 smithi026 ceph-mon[68309]: pgmap v996: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 3.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 95 op/s; 8309/191673 objects degraded (4.335%); 26300/191673 objects misplaced (13.721%); 4.0 MiB/s, 63 objects/s recovering 2024-03-20T09:09:25.981 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-20T09:09:26.297 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:09:27.123 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:09:27.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:27.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:09:27.229 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:09:27.258 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:27.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:27 smithi115 ceph-mon[72476]: pgmap v997: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 3.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 142 op/s; 8234/190476 objects degraded (4.323%); 25305/190476 objects misplaced (13.285%); 6.9 MiB/s, 110 objects/s recovering 2024-03-20T09:09:27.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:27 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8234/190476 objects degraded (4.323%), 5 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:09:27.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:27 smithi115 ceph-mon[72476]: osdmap e413: 12 total, 11 up, 9 in 2024-03-20T09:09:27.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:27 smithi122 ceph-mon[73495]: pgmap v997: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 3.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 142 op/s; 8234/190476 objects degraded (4.323%); 25305/190476 objects misplaced (13.285%); 6.9 MiB/s, 110 objects/s recovering 2024-03-20T09:09:27.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:27 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8234/190476 objects degraded (4.323%), 5 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:09:27.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:27 smithi122 ceph-mon[73495]: osdmap e413: 12 total, 11 up, 9 in 2024-03-20T09:09:27.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:27 smithi026 ceph-mon[68309]: pgmap v997: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 3.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 142 op/s; 8234/190476 objects degraded (4.323%); 25305/190476 objects misplaced (13.285%); 6.9 MiB/s, 110 objects/s recovering 2024-03-20T09:09:27.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:27 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8234/190476 objects degraded (4.323%), 5 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:09:27.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:27 smithi026 ceph-mon[68309]: osdmap e413: 12 total, 11 up, 9 in 2024-03-20T09:09:28.406 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:09:28.406 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:09:28.710 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:09:28.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:28.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:09:28.752 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:09:28.781 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:09:28.817 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:09:28.849 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:29.147 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-20T09:09:29.199 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:09:29.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:29.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:09:29.252 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:29 smithi026 ceph-mon[68309]: pgmap v999: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 3.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 120 op/s; 8234/189669 objects degraded (4.341%); 25177/189669 objects misplaced (13.274%); 4.3 MiB/s, 68 objects/s recovering 2024-03-20T09:09:29.304 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:09:29.330 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:29.350 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:09:29.351 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:29 smithi122 ceph-mon[73495]: pgmap v999: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 3.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 120 op/s; 8234/189669 objects degraded (4.341%); 25177/189669 objects misplaced (13.274%); 4.3 MiB/s, 68 objects/s recovering 2024-03-20T09:09:29.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:29.352 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:09:29.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:29 smithi115 ceph-mon[72476]: pgmap v999: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 3.9 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 120 op/s; 8234/189669 objects degraded (4.341%); 25177/189669 objects misplaced (13.274%); 4.3 MiB/s, 68 objects/s recovering 2024-03-20T09:09:29.453 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:09:29.482 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:29.483 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:09:29.500 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:09:29.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:29.503 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:09:29.604 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:09:29.640 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:30.102 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:09:30.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:30 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1768010600' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:09:30.412 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:30 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1768010600' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:09:30.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:30 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1768010600' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:09:30.994 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:09:30.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:30.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:09:31.099 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:09:31.131 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:31.274 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:31 smithi115 ceph-mon[72476]: pgmap v1000: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 3.8 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 115 op/s; 8215/189153 objects degraded (4.343%); 24735/189153 objects misplaced (13.077%); 4.4 MiB/s, 106 objects/s recovering 2024-03-20T09:09:31.274 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:31 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:09:31.274 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:31 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1768010600' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:09:31.274 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:31 smithi115 ceph-mon[72476]: osdmap e414: 12 total, 11 up, 9 in 2024-03-20T09:09:31.280 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:09:31.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:31.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:09:31.384 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:09:31.408 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:31 smithi122 ceph-mon[73495]: pgmap v1000: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 3.8 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 115 op/s; 8215/189153 objects degraded (4.343%); 24735/189153 objects misplaced (13.077%); 4.4 MiB/s, 106 objects/s recovering 2024-03-20T09:09:31.409 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:31 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:09:31.409 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:31 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1768010600' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:09:31.409 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:31 smithi122 ceph-mon[73495]: osdmap e414: 12 total, 11 up, 9 in 2024-03-20T09:09:31.411 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:31.430 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:09:31.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:31.434 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:09:31.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:31 smithi026 ceph-mon[68309]: pgmap v1000: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 3.8 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 115 op/s; 8215/189153 objects degraded (4.343%); 24735/189153 objects misplaced (13.077%); 4.4 MiB/s, 106 objects/s recovering 2024-03-20T09:09:31.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:31 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:09:31.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:31 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1768010600' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:09:31.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:31 smithi026 ceph-mon[68309]: osdmap e414: 12 total, 11 up, 9 in 2024-03-20T09:09:31.535 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:09:31.568 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:31.569 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:09:31.570 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:09:31.591 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:09:31.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:31.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:09:31.696 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:09:31.731 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:31.833 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:09:31.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:31.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:09:31.937 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:09:31.966 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:32.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:32 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8215/189153 objects degraded (4.343%), 5 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:09:32.350 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:32 smithi115 ceph-mon[72476]: osdmap e415: 12 total, 11 up, 9 in 2024-03-20T09:09:32.371 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:32 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8215/189153 objects degraded (4.343%), 5 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:09:32.371 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:32 smithi026 ceph-mon[68309]: osdmap e415: 12 total, 11 up, 9 in 2024-03-20T09:09:32.383 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-20T09:09:32.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:32 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8215/189153 objects degraded (4.343%), 5 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:09:32.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:32 smithi122 ceph-mon[73495]: osdmap e415: 12 total, 11 up, 9 in 2024-03-20T09:09:32.675 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:09:32.929 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:09:32.929 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:09:32.929 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:09:32.930 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:09:32.930 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:09:32.949 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:09:32.976 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:09:32.999 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:09:33.019 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:09:33.020 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:09:33.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:33.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:09:33.044 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:09:33.068 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:09:33.068 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:09:33.068 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:09:33.068 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:09:33.068 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:09:33.068 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:09:33.068 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-20T09:09:33.068 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:09:33.068 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:09:33.069 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:09:33.069 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:09:33.069 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:09:33.069 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:09:33.125 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:09:33.153 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:33.193 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:09:33.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:33.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:09:33.225 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:33 smithi115 ceph-mon[72476]: pgmap v1002: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 3.8 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 172 op/s; 8146/187539 objects degraded (4.344%); 24389/187539 objects misplaced (13.005%); 5.5 MiB/s, 122 objects/s recovering 2024-03-20T09:09:33.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:33 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:09:33.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:33 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:09:33.226 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:33 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3477153178' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:09:33.297 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:09:33.323 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:33.342 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:09:33.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:33.345 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:09:33.416 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:33 smithi026 ceph-mon[68309]: pgmap v1002: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 3.8 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 172 op/s; 8146/187539 objects degraded (4.344%); 24389/187539 objects misplaced (13.005%); 5.5 MiB/s, 122 objects/s recovering 2024-03-20T09:09:33.416 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:33 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:09:33.416 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:33 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:09:33.417 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:33 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3477153178' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:09:33.445 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:09:33.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:33 smithi122 ceph-mon[73495]: pgmap v1002: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 3.8 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 172 op/s; 8146/187539 objects degraded (4.344%); 24389/187539 objects misplaced (13.005%); 5.5 MiB/s, 122 objects/s recovering 2024-03-20T09:09:33.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:33 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:09:33.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:33 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:09:33.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:33 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3477153178' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:09:33.476 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:33.599 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:09:33.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:33.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:09:33.704 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:09:33.738 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:33.765 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:09:33.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:33.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:09:33.800 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:09:33.801 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:09:33.870 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:09:33.901 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:33.924 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:09:33.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:33.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:09:34.028 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:09:34.055 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:34.076 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:09:34.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:34.079 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:09:34.110 INFO:teuthology.orchestra.run.smithi026.stderr:set 1.b pg_upmap_items mapping to [6->0,7->1,8->10] 2024-03-20T09:09:34.180 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:09:34.210 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:34.270 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:09:34.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:34 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2020546131' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.b", "id": [6, 0, 7, 1, 8, 10]}]: dispatch 2024-03-20T09:09:34.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:34 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.b", "id": [6, 0, 7, 1, 8, 10]}]: dispatch 2024-03-20T09:09:34.414 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:09:34.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:34.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:09:34.446 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:34 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2020546131' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.b", "id": [6, 0, 7, 1, 8, 10]}]: dispatch 2024-03-20T09:09:34.447 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:34 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.b", "id": [6, 0, 7, 1, 8, 10]}]: dispatch 2024-03-20T09:09:34.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:34 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2020546131' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.b", "id": [6, 0, 7, 1, 8, 10]}]: dispatch 2024-03-20T09:09:34.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:34 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.b", "id": [6, 0, 7, 1, 8, 10]}]: dispatch 2024-03-20T09:09:34.518 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:09:34.548 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:34.747 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:09:34.749 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:09:34.749 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:09:34.749 INFO:teuthology.orchestra.run.smithi115.stdout: "num_ops": 0 2024-03-20T09:09:34.749 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:09:34.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:34.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:09:34.852 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:09:34.883 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:34.897 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:09:34.904 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:09:34.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:34.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:09:35.008 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:09:35.037 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:35.164 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:09:35.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:35.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:09:35.269 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:09:35.301 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:35 smithi026 ceph-mon[68309]: pgmap v1004: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 3.8 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 111 op/s; 8125/187407 objects degraded (4.335%); 24389/187407 objects misplaced (13.014%); 2.7 MiB/s, 84 objects/s recovering 2024-03-20T09:09:35.301 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:35 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.b", "id": [6, 0, 7, 1, 8, 10]}]': finished 2024-03-20T09:09:35.301 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:35 smithi026 ceph-mon[68309]: osdmap e416: 12 total, 11 up, 9 in 2024-03-20T09:09:35.303 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:35.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:35 smithi115 ceph-mon[72476]: pgmap v1004: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 3.8 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 111 op/s; 8125/187407 objects degraded (4.335%); 24389/187407 objects misplaced (13.014%); 2.7 MiB/s, 84 objects/s recovering 2024-03-20T09:09:35.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:35 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.b", "id": [6, 0, 7, 1, 8, 10]}]': finished 2024-03-20T09:09:35.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:35 smithi115 ceph-mon[72476]: osdmap e416: 12 total, 11 up, 9 in 2024-03-20T09:09:35.407 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:09:35.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:35.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:09:35.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:35 smithi122 ceph-mon[73495]: pgmap v1004: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 3.8 GiB data, 15 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 111 op/s; 8125/187407 objects degraded (4.335%); 24389/187407 objects misplaced (13.014%); 2.7 MiB/s, 84 objects/s recovering 2024-03-20T09:09:35.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:35 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.b", "id": [6, 0, 7, 1, 8, 10]}]': finished 2024-03-20T09:09:35.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:35 smithi122 ceph-mon[73495]: osdmap e416: 12 total, 11 up, 9 in 2024-03-20T09:09:35.513 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:09:35.539 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:35.558 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:09:35.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:35.561 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:09:35.622 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-20T09:09:35.662 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:09:35.689 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:35.709 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:09:35.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:35.712 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:09:35.802 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:09:35.813 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:09:35.876 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:35.914 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:09:35.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:35.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:09:35.940 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:09:36.024 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:09:36.055 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:36.398 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:09:36.404 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:09:36.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:36 smithi115 ceph-mon[72476]: osdmap e417: 12 total, 11 up, 9 in 2024-03-20T09:09:36.406 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:36 smithi026 ceph-mon[68309]: osdmap e417: 12 total, 11 up, 9 in 2024-03-20T09:09:36.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:36.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:09:36.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:36 smithi122 ceph-mon[73495]: osdmap e417: 12 total, 11 up, 9 in 2024-03-20T09:09:36.509 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:09:36.542 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:36.650 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:09:36.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:36.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:09:36.755 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:09:36.783 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:37.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:37 smithi026 ceph-mon[68309]: pgmap v1007: 57 pgs: 13 active+clean+remapped, 1 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 5 active+undersized; 3.8 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 147 op/s; 8065/186396 objects degraded (4.327%); 24144/186396 objects misplaced (12.953%); 3.1 MiB/s, 155 objects/s recovering 2024-03-20T09:09:37.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:37 smithi026 ceph-mon[68309]: osdmap e418: 12 total, 11 up, 9 in 2024-03-20T09:09:37.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:37 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8065/186396 objects degraded (4.327%), 5 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:09:37.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:37 smithi026 ceph-mon[68309]: osdmap e419: 12 total, 11 up, 9 in 2024-03-20T09:09:37.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:37 smithi115 ceph-mon[72476]: pgmap v1007: 57 pgs: 13 active+clean+remapped, 1 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 5 active+undersized; 3.8 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 147 op/s; 8065/186396 objects degraded (4.327%); 24144/186396 objects misplaced (12.953%); 3.1 MiB/s, 155 objects/s recovering 2024-03-20T09:09:37.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:37 smithi115 ceph-mon[72476]: osdmap e418: 12 total, 11 up, 9 in 2024-03-20T09:09:37.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:37 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8065/186396 objects degraded (4.327%), 5 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:09:37.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:37 smithi115 ceph-mon[72476]: osdmap e419: 12 total, 11 up, 9 in 2024-03-20T09:09:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:37 smithi122 ceph-mon[73495]: pgmap v1007: 57 pgs: 13 active+clean+remapped, 1 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 33 active+clean, 5 active+undersized; 3.8 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 147 op/s; 8065/186396 objects degraded (4.327%); 24144/186396 objects misplaced (12.953%); 3.1 MiB/s, 155 objects/s recovering 2024-03-20T09:09:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:37 smithi122 ceph-mon[73495]: osdmap e418: 12 total, 11 up, 9 in 2024-03-20T09:09:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:37 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8065/186396 objects degraded (4.327%), 5 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-03-20T09:09:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:37 smithi122 ceph-mon[73495]: osdmap e419: 12 total, 11 up, 9 in 2024-03-20T09:09:37.632 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:09:37.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:37.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:09:37.738 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:09:37.767 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:37.865 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:09:37.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:37.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:09:37.971 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:09:38.004 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:38.124 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:09:38.134 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:09:38.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:38.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:09:38.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:38 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:09:38.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:38 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:09:38.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:38 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2519837509' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:09:38.238 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:09:38.264 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:38.283 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:09:38.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:38.286 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:09:38.387 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:09:38.416 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:38 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:09:38.416 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:38 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:09:38.416 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:38 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2519837509' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:09:38.418 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:38.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:38 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:09:38.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:38 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:09:38.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:38 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2519837509' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:09:38.620 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:09:38.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:38.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:09:38.725 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:09:38.759 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:39.013 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:09:39.047 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:09:39.067 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:09:39.098 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:09:39.118 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:09:39.143 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:09:39.164 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:09:39.185 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:09:39.206 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:09:39.229 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:09:39.255 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:09:39.256 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:09:39.256 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:09:39.256 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:09:39.256 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:09:39.256 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:09:39.256 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-20T09:09:39.256 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:09:39.256 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:09:39.256 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:09:39.256 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:09:39.256 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:09:39.257 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:09:39.380 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:39 smithi026 ceph-mon[68309]: pgmap v1010: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.8 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 144 op/s; 8065/185667 objects degraded (4.344%); 24029/185667 objects misplaced (12.942%); 2.7 MiB/s, 137 objects/s recovering 2024-03-20T09:09:39.380 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:39 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:09:39.380 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:39 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2519837509' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:09:39.380 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:39 smithi026 ceph-mon[68309]: osdmap e420: 12 total, 11 up, 9 in 2024-03-20T09:09:39.380 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:39 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2297615227' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:09:39.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:39 smithi115 ceph-mon[72476]: pgmap v1010: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.8 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 144 op/s; 8065/185667 objects degraded (4.344%); 24029/185667 objects misplaced (12.942%); 2.7 MiB/s, 137 objects/s recovering 2024-03-20T09:09:39.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:39 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:09:39.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:39 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2519837509' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:09:39.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:39 smithi115 ceph-mon[72476]: osdmap e420: 12 total, 11 up, 9 in 2024-03-20T09:09:39.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:39 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2297615227' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:09:39.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:39 smithi122 ceph-mon[73495]: pgmap v1010: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.8 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 144 op/s; 8065/185667 objects degraded (4.344%); 24029/185667 objects misplaced (12.942%); 2.7 MiB/s, 137 objects/s recovering 2024-03-20T09:09:39.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:39 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:09:39.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:39 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2519837509' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:09:39.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:39 smithi122 ceph-mon[73495]: osdmap e420: 12 total, 11 up, 9 in 2024-03-20T09:09:39.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:39 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2297615227' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:09:39.459 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:09:39.459 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:09:39.778 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 8, 10, 11] out_osds: [9, 3, 4] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11] 2024-03-20T09:09:39.778 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:09:39.778 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-03-20T09:09:39.778 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd test-reweight-by-utilization 1000 3.0 2024-03-20T09:09:40.066 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:09:40.067 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:09:40.112 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:09:40.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:40.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:09:40.119 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:09:40.218 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:09:40.244 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:40.262 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:09:40.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:40.265 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:09:40.366 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:09:40.380 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-20T09:09:40.398 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:40.423 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:09:40.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:40.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:09:40.527 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:09:40.558 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:40.596 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:09:40.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:40.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:09:40.673 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:09:40.700 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:09:40.730 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:40.928 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:09:40.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:40.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:09:41.037 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:09:41.067 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:41.383 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:09:41.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:41 smithi115 ceph-mon[72476]: pgmap v1012: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.8 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 112 op/s; 8041/185274 objects degraded (4.340%); 24009/185274 objects misplaced (12.959%); 2.3 MiB/s, 0 objects/s recovering 2024-03-20T09:09:41.412 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:41 smithi026 ceph-mon[68309]: pgmap v1012: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.8 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 112 op/s; 8041/185274 objects degraded (4.340%); 24009/185274 objects misplaced (12.959%); 2.3 MiB/s, 0 objects/s recovering 2024-03-20T09:09:41.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:41 smithi122 ceph-mon[73495]: pgmap v1012: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.8 GiB data, 14 GiB used, 790 GiB / 805 GiB avail; 0 B/s wr, 112 op/s; 8041/185274 objects degraded (4.340%); 24009/185274 objects misplaced (12.959%); 2.3 MiB/s, 0 objects/s recovering 2024-03-20T09:09:41.518 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:09:41.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:41.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:09:41.623 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:09:41.653 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:41.692 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:09:41.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:41.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:09:41.800 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:09:41.826 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:41.844 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:09:41.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:41.847 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:09:41.948 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:09:41.989 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:42.034 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:09:42.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:42.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:09:42.067 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:09:42.073 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:09:42.138 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:09:42.164 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:42.182 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:09:42.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:42.187 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:09:42.288 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:09:42.304 INFO:teuthology.orchestra.run.smithi026.stdout:moved 1 / 171 (0.584795%) 2024-03-20T09:09:42.335 INFO:teuthology.orchestra.run.smithi026.stdout:avg 21.375 2024-03-20T09:09:42.336 INFO:teuthology.orchestra.run.smithi026.stdout:stddev 4.69874 -> 4.36785 (expected baseline 4.32471) 2024-03-20T09:09:42.336 INFO:teuthology.orchestra.run.smithi026.stdout:min osd.7 with 11 -> 12 pgs (0.51462 -> 0.561404 * mean) 2024-03-20T09:09:42.336 INFO:teuthology.orchestra.run.smithi026.stdout:max osd.1 with 24 -> 23 pgs (1.12281 -> 1.07602 * mean) 2024-03-20T09:09:42.336 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:09:42.336 INFO:teuthology.orchestra.run.smithi026.stdout:oload 1000 2024-03-20T09:09:42.336 INFO:teuthology.orchestra.run.smithi026.stdout:max_change 3 2024-03-20T09:09:42.336 INFO:teuthology.orchestra.run.smithi026.stdout:max_change_osds 4 2024-03-20T09:09:42.336 INFO:teuthology.orchestra.run.smithi026.stdout:average_utilization 0.0175 2024-03-20T09:09:42.336 INFO:teuthology.orchestra.run.smithi026.stdout:overload_utilization 0.1746 2024-03-20T09:09:42.336 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7 weight 0.4265 -> 0.5874 2024-03-20T09:09:42.337 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:42.337 INFO:teuthology.orchestra.run.smithi026.stderr:no change 2024-03-20T09:09:42.360 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 09:09:42 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T09:09:42.303+0000 7fe5fc38b700 -1 mgr.server reply reply (1) Operation not permitted no change 2024-03-20T09:09:42.360 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:42 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:09:42.360 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:42 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:09:42.360 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:42 smithi026 ceph-mon[68309]: osdmap e421: 12 total, 11 up, 9 in 2024-03-20T09:09:42.360 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:42 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:09:42.360 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:42 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8041/185274 objects degraded (4.340%), 5 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:09:42.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:42 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:09:42.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:42 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:09:42.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:42 smithi115 ceph-mon[72476]: osdmap e421: 12 total, 11 up, 9 in 2024-03-20T09:09:42.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:09:42.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:42 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8041/185274 objects degraded (4.340%), 5 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:09:42.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:42 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:09:42.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:42 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:09:42.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:42 smithi122 ceph-mon[73495]: osdmap e421: 12 total, 11 up, 9 in 2024-03-20T09:09:42.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:42 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:09:42.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:42 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8041/185274 objects degraded (4.340%), 5 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:09:42.517 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:09:42.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:42.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:09:42.622 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:09:42.652 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:42.817 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:09:42.817 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:09:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:43 smithi115 ceph-mon[72476]: pgmap v1014: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 166 op/s; 7976/183594 objects degraded (4.344%); 23614/183594 objects misplaced (12.862%); 2.4 MiB/s, 1 objects/s recovering 2024-03-20T09:09:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:43 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:09:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:43 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:09:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:43 smithi115 ceph-mon[72476]: from='client.25426 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:09:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:43 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:09:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:43 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:09:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:43 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:09:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:43 smithi115 ceph-mon[72476]: osdmap e422: 12 total, 11 up, 9 in 2024-03-20T09:09:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:43 smithi122 ceph-mon[73495]: pgmap v1014: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 166 op/s; 7976/183594 objects degraded (4.344%); 23614/183594 objects misplaced (12.862%); 2.4 MiB/s, 1 objects/s recovering 2024-03-20T09:09:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:43 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:09:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:43 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:09:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:43 smithi122 ceph-mon[73495]: from='client.25426 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:09:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:43 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:09:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:43 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:09:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:43 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:09:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:43 smithi122 ceph-mon[73495]: osdmap e422: 12 total, 11 up, 9 in 2024-03-20T09:09:43.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:43 smithi026 ceph-mon[68309]: pgmap v1014: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 166 op/s; 7976/183594 objects degraded (4.344%); 23614/183594 objects misplaced (12.862%); 2.4 MiB/s, 1 objects/s recovering 2024-03-20T09:09:43.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:43 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:09:43.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:43 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:09:43.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:43 smithi026 ceph-mon[68309]: from='client.25426 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:09:43.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:43 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:09:43.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:43 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:09:43.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:43 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:09:43.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:43 smithi026 ceph-mon[68309]: osdmap e422: 12 total, 11 up, 9 in 2024-03-20T09:09:43.522 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:09:43.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:43.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:09:43.627 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:09:43.654 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:43.672 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:09:43.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:43.675 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:09:43.684 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_ops_in_flight 2024-03-20T09:09:43.776 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:09:43.811 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:43.964 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:09:43.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:43.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:09:43.974 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:09:43.991 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-03-20T09:09:43.992 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd test-reweight-by-utilization 110 0.05 2024-03-20T09:09:44.067 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:09:44.094 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:44.244 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:09:44.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:44.247 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:09:44.348 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:09:44.384 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:44.457 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:09:44.461 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:09:44.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:44.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:09:44.563 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:09:44.593 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:44.613 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:09:44.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:44.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:09:44.717 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:09:44.747 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:44.870 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:09:44.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:44.873 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:09:44.974 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:09:45.002 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:45.021 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:09:45.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:45.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:09:45.126 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:09:45.145 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:09:45.153 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:45.401 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:09:45.403 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:45 smithi122 ceph-mon[73495]: pgmap v1016: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 124 op/s; 7945/183417 objects degraded (4.332%); 23614/183417 objects misplaced (12.874%); 1.9 MiB/s, 29 objects/s recovering 2024-03-20T09:09:45.403 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:45 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/548435345' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:09:45.403 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:45 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:09:45.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:45.404 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:09:45.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:45 smithi115 ceph-mon[72476]: pgmap v1016: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 124 op/s; 7945/183417 objects degraded (4.332%); 23614/183417 objects misplaced (12.874%); 1.9 MiB/s, 29 objects/s recovering 2024-03-20T09:09:45.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:45 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/548435345' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:09:45.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:45 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:09:45.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:45 smithi026 ceph-mon[68309]: pgmap v1016: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 124 op/s; 7945/183417 objects degraded (4.332%); 23614/183417 objects misplaced (12.874%); 1.9 MiB/s, 29 objects/s recovering 2024-03-20T09:09:45.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:45 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/548435345' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:09:45.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:45 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:09:45.505 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:09:45.511 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:09:45.535 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:45.783 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:09:45.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:45.786 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:09:45.887 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:09:45.916 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:45.952 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:09:45.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:45.955 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:09:46.055 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:09:46.088 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:46.089 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:09:46.090 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:09:46.187 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:09:46.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:46.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:09:46.294 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:09:46.324 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:46.343 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:09:46.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:46.346 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:09:46.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:46 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:09:46.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:46 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:09:46.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:46 smithi115 ceph-mon[72476]: osdmap e423: 12 total, 11 up, 9 in 2024-03-20T09:09:46.447 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:09:46.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:46 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:09:46.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:46 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:09:46.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:46 smithi122 ceph-mon[73495]: osdmap e423: 12 total, 11 up, 9 in 2024-03-20T09:09:46.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:46 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:09:46.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:46 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:09:46.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:46 smithi026 ceph-mon[68309]: osdmap e423: 12 total, 11 up, 9 in 2024-03-20T09:09:46.474 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:46.859 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_blocked_ops 2024-03-20T09:09:46.923 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:09:46.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:46.926 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:09:47.027 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:09:47.054 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:47.074 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:09:47.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:47.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:09:47.178 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:09:47.182 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:09:47.214 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:47.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:47 smithi115 ceph-mon[72476]: pgmap v1018: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 161 op/s; 7884/182361 objects degraded (4.323%); 23346/182361 objects misplaced (12.802%); 3.4 MiB/s, 54 objects/s recovering 2024-03-20T09:09:47.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:47 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 7884/182361 objects degraded (4.323%), 5 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:09:47.427 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:47 smithi122 ceph-mon[73495]: pgmap v1018: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 161 op/s; 7884/182361 objects degraded (4.323%); 23346/182361 objects misplaced (12.802%); 3.4 MiB/s, 54 objects/s recovering 2024-03-20T09:09:47.427 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:47 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 7884/182361 objects degraded (4.323%), 5 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:09:47.463 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:09:47.464 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:47 smithi026 ceph-mon[68309]: pgmap v1018: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 161 op/s; 7884/182361 objects degraded (4.323%); 23346/182361 objects misplaced (12.802%); 3.4 MiB/s, 54 objects/s recovering 2024-03-20T09:09:47.465 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:47 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 7884/182361 objects degraded (4.323%), 5 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:09:47.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:47.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:09:47.567 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:09:47.593 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:47.666 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:09:47.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:47.669 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:09:47.742 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:09:47.764 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:09:47.770 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:09:47.784 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:09:47.796 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:47.808 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:09:47.817 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:09:47.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:47.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:09:47.829 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:09:47.851 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:09:47.877 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:09:47.898 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:09:47.921 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:09:47.922 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:09:47.948 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:47.957 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:09:47.967 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:09:47.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:47.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:09:47.978 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:09:47.978 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:09:47.978 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:09:47.978 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:09:47.978 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:09:47.978 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:09:47.979 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-20T09:09:47.979 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:09:47.979 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:09:47.979 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:09:47.979 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:09:47.979 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:09:47.979 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:09:48.037 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:09:48.070 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:09:48.096 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:48.116 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:09:48.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:48.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:09:48.223 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:09:48.250 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:48.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:48 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1311148444' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:09:48.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:48 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1311148444' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:09:48.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:48 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1311148444' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:09:48.598 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:09:48.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:48.601 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:09:48.702 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:09:48.728 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:48.751 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:09:48.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:48.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:09:48.854 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:09:48.882 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:49.167 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:09:49.209 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:09:49.210 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:09:49.402 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:49 smithi122 ceph-mon[73495]: pgmap v1019: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 143 op/s; 7884/181641 objects degraded (4.340%); 23229/181641 objects misplaced (12.788%); 2.1 MiB/s, 34 objects/s recovering 2024-03-20T09:09:49.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:49 smithi115 ceph-mon[72476]: pgmap v1019: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 143 op/s; 7884/181641 objects degraded (4.340%); 23229/181641 objects misplaced (12.788%); 2.1 MiB/s, 34 objects/s recovering 2024-03-20T09:09:49.416 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:49 smithi026 ceph-mon[68309]: pgmap v1019: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 143 op/s; 7884/181641 objects degraded (4.340%); 23229/181641 objects misplaced (12.788%); 2.1 MiB/s, 34 objects/s recovering 2024-03-20T09:09:49.432 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:09:49.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:49.435 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:09:49.536 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:09:49.568 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:49.641 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:09:49.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:49.644 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:09:49.671 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:09:49.672 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:09:49.672 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-03-20T09:09:49.672 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-03-20T09:09:49.672 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:09:49.745 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:09:49.785 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:49.861 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:09:49.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:49.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:09:49.908 INFO:teuthology.orchestra.run.smithi026.stdout:moved 4 / 171 (2.33918%) 2024-03-20T09:09:49.908 INFO:teuthology.orchestra.run.smithi026.stdout:avg 21.375 2024-03-20T09:09:49.909 INFO:teuthology.orchestra.run.smithi026.stdout:stddev 4.69874 -> 5.10973 (expected baseline 4.32471) 2024-03-20T09:09:49.909 INFO:teuthology.orchestra.run.smithi026.stdout:min osd.7 with 11 -> 11 pgs (0.51462 -> 0.51462 * mean) 2024-03-20T09:09:49.909 INFO:teuthology.orchestra.run.smithi026.stdout:max osd.1 with 24 -> 24 pgs (1.12281 -> 1.12281 * mean) 2024-03-20T09:09:49.909 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:09:49.909 INFO:teuthology.orchestra.run.smithi026.stdout:oload 110 2024-03-20T09:09:49.909 INFO:teuthology.orchestra.run.smithi026.stdout:max_change 0.05 2024-03-20T09:09:49.909 INFO:teuthology.orchestra.run.smithi026.stdout:max_change_osds 4 2024-03-20T09:09:49.909 INFO:teuthology.orchestra.run.smithi026.stdout:average_utilization 0.0173 2024-03-20T09:09:49.909 INFO:teuthology.orchestra.run.smithi026.stdout:overload_utilization 0.0191 2024-03-20T09:09:49.909 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5 weight 1.0000 -> 0.9500 2024-03-20T09:09:49.909 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11 weight 0.6123 -> 0.5623 2024-03-20T09:09:49.909 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7 weight 0.4265 -> 0.4765 2024-03-20T09:09:49.909 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6 weight 0.8424 -> 0.7924 2024-03-20T09:09:49.911 INFO:teuthology.orchestra.run.smithi026.stderr:no change 2024-03-20T09:09:49.965 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:09:49.993 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:50.014 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:09:50.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:50.016 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:09:50.027 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 09:09:49 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T09:09:49.907+0000 7fe5fc38b700 -1 mgr.server reply reply (4) Interrupted system call no change 2024-03-20T09:09:50.118 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:09:50.152 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:50.540 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_historic_ops 2024-03-20T09:09:50.803 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:09:51.005 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:09:51.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:51.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:09:51.010 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-03-20T09:09:51.010 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd test-reweight-by-utilization 1000 0.05 2024-03-20T09:09:51.110 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:09:51.137 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:51.157 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:09:51.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:51.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:09:51.211 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:09:51.241 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:51 smithi026 ceph-mon[68309]: pgmap v1020: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 100 op/s; 7864/181185 objects degraded (4.340%); 23207/181185 objects misplaced (12.808%); 1.5 MiB/s, 23 objects/s recovering 2024-03-20T09:09:51.241 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:51 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:09:51.241 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:51 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:09:51.241 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:51 smithi026 ceph-mon[68309]: from='client.15567 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:09:51.241 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:51 smithi026 ceph-mon[68309]: osdmap e424: 12 total, 11 up, 9 in 2024-03-20T09:09:51.262 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:09:51.294 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:51.349 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:51 smithi122 ceph-mon[73495]: pgmap v1020: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 100 op/s; 7864/181185 objects degraded (4.340%); 23207/181185 objects misplaced (12.808%); 1.5 MiB/s, 23 objects/s recovering 2024-03-20T09:09:51.349 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:51 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:09:51.349 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:51 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:09:51.349 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:51 smithi122 ceph-mon[73495]: from='client.15567 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:09:51.349 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:51 smithi122 ceph-mon[73495]: osdmap e424: 12 total, 11 up, 9 in 2024-03-20T09:09:51.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:51 smithi115 ceph-mon[72476]: pgmap v1020: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 100 op/s; 7864/181185 objects degraded (4.340%); 23207/181185 objects misplaced (12.808%); 1.5 MiB/s, 23 objects/s recovering 2024-03-20T09:09:51.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:51 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:09:51.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:51 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:09:51.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:51 smithi115 ceph-mon[72476]: from='client.15567 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:09:51.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:51 smithi115 ceph-mon[72476]: osdmap e424: 12 total, 11 up, 9 in 2024-03-20T09:09:51.427 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:09:51.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:51.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:09:51.503 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:09:51.532 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:09:51.559 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:51.683 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:09:51.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:51.685 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:09:51.786 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:09:51.817 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:51.968 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:09:51.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:51.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:09:52.072 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:09:52.106 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:52.164 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:09:52.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:52.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:09:52.268 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:09:52.294 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:52.315 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:09:52.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:52.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:09:52.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:52 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 7864/181185 objects degraded (4.340%), 5 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:09:52.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:52 smithi115 ceph-mon[72476]: osdmap e425: 12 total, 11 up, 9 in 2024-03-20T09:09:52.419 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:09:52.445 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:52.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:52 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 7864/181185 objects degraded (4.340%), 5 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:09:52.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:52 smithi122 ceph-mon[73495]: osdmap e425: 12 total, 11 up, 9 in 2024-03-20T09:09:52.464 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:09:52.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:52.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:09:52.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:52 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 7864/181185 objects degraded (4.340%), 5 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:09:52.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:52 smithi026 ceph-mon[68309]: osdmap e425: 12 total, 11 up, 9 in 2024-03-20T09:09:52.567 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:09:52.597 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:52.618 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:09:52.619 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:09:52.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:52.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:09:52.723 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:09:52.754 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:52.772 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:09:52.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:52.775 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:09:52.876 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:09:52.909 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:52.916 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:09:52.916 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-03-20T09:09:52.916 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-03-20T09:09:52.916 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-03-20T09:09:52.916 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.916 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:185562 2.5 2:a002230b:::benchmark_data_smithi122_107654_object66777:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-20T09:09:52.916 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:01:32.624249+0000", 2024-03-20T09:09:52.916 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 500.28828928299998, 2024-03-20T09:09:52.917 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20806407800000001, 2024-03-20T09:09:52.917 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:09:52.917 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:52.917 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:09:52.917 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:09:52.917 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:09:52.917 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 185562 2024-03-20T09:09:52.917 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.917 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:09:52.917 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.917 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:09:52.917 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.624249+0000", 2024-03-20T09:09:52.917 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:52.917 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.918 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.918 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:09:52.918 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.624249+0000", 2024-03-20T09:09:52.918 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:52.918 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.918 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.918 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:09:52.918 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.624250+0000", 2024-03-20T09:09:52.918 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1930000000000001e-06 2024-03-20T09:09:52.918 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.918 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.918 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:09:52.918 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.624256+0000", 2024-03-20T09:09:52.919 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0929999999999999e-06 2024-03-20T09:09:52.919 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.919 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.919 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:09:52.919 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.624269+0000", 2024-03-20T09:09:52.919 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.26e-05 2024-03-20T09:09:52.919 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.919 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.919 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:09:52.919 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.624279+0000", 2024-03-20T09:09:52.919 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0249e-05 2024-03-20T09:09:52.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.920 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.920 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:09:52.920 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.624328+0000", 2024-03-20T09:09:52.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8386000000000001e-05 2024-03-20T09:09:52.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.920 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.920 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:09:52.920 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.624430+0000", 2024-03-20T09:09:52.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010203499999999999 2024-03-20T09:09:52.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.920 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.921 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,4", 2024-03-20T09:09:52.921 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.624528+0000", 2024-03-20T09:09:52.921 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.8733999999999994e-05 2024-03-20T09:09:52.921 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.921 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.921 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:09:52.921 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.634445+0000", 2024-03-20T09:09:52.921 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0099167649999999993 2024-03-20T09:09:52.921 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.921 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.921 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:52.921 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.639754+0000", 2024-03-20T09:09:52.921 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0053083039999999998 2024-03-20T09:09:52.921 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.921 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.922 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:52.922 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.832194+0000", 2024-03-20T09:09:52.922 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19244091899999999 2024-03-20T09:09:52.922 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.922 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.922 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:09:52.922 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.832255+0000", 2024-03-20T09:09:52.922 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1036000000000003e-05 2024-03-20T09:09:52.922 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.922 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.922 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:09:52.922 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.832313+0000", 2024-03-20T09:09:52.923 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7763999999999998e-05 2024-03-20T09:09:52.923 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:52.923 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:09:52.923 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:52.923 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.923 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.923 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:185568 2.5 2:a62ec9a1:::benchmark_data_smithi122_107654_object66783:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-20T09:09:52.923 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:01:32.624392+0000", 2024-03-20T09:09:52.923 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 500.28814650200002, 2024-03-20T09:09:52.923 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.22075340199999999, 2024-03-20T09:09:52.923 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:09:52.923 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:52.923 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:09:52.923 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:09:52.924 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:09:52.924 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 185568 2024-03-20T09:09:52.924 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.924 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:09:52.924 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.924 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:09:52.924 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.624392+0000", 2024-03-20T09:09:52.924 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:52.924 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.924 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.924 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:09:52.924 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.624392+0000", 2024-03-20T09:09:52.924 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:52.924 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.925 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.925 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:09:52.925 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.624393+0000", 2024-03-20T09:09:52.925 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3099999999999996e-07 2024-03-20T09:09:52.925 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.925 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.925 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:09:52.925 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.624398+0000", 2024-03-20T09:09:52.925 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.938e-06 2024-03-20T09:09:52.925 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.925 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.925 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:09:52.925 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.624407+0000", 2024-03-20T09:09:52.925 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5899999999999997e-06 2024-03-20T09:09:52.926 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.926 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:09:52.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.624414+0000", 2024-03-20T09:09:52.926 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0820000000000001e-06 2024-03-20T09:09:52.926 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.926 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:09:52.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.629815+0000", 2024-03-20T09:09:52.926 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0054001700000000001 2024-03-20T09:09:52.926 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.926 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:09:52.927 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.629938+0000", 2024-03-20T09:09:52.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012370299999999999 2024-03-20T09:09:52.927 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.927 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.927 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,4", 2024-03-20T09:09:52.927 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.630056+0000", 2024-03-20T09:09:52.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000117316 2024-03-20T09:09:52.927 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.927 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.927 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:09:52.927 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.634535+0000", 2024-03-20T09:09:52.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0044798099999999999 2024-03-20T09:09:52.928 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.928 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.928 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:52.928 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.639894+0000", 2024-03-20T09:09:52.928 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0053589329999999998 2024-03-20T09:09:52.928 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.928 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.928 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:52.928 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.845027+0000", 2024-03-20T09:09:52.928 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.205132488 2024-03-20T09:09:52.928 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.928 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.928 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:09:52.928 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.845100+0000", 2024-03-20T09:09:52.928 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2849000000000007e-05 2024-03-20T09:09:52.929 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.929 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.929 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:09:52.929 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.845145+0000", 2024-03-20T09:09:52.929 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5692000000000002e-05 2024-03-20T09:09:52.929 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:52.929 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:09:52.929 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:52.929 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.929 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.929 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:185581 2.f 2:f574bb59:::benchmark_data_smithi122_107654_object66796:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-20T09:09:52.929 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:01:32.639940+0000", 2024-03-20T09:09:52.929 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 500.27259889800001, 2024-03-20T09:09:52.929 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.201438917, 2024-03-20T09:09:52.930 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:09:52.930 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:52.930 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:09:52.930 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:09:52.930 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:09:52.930 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 185581 2024-03-20T09:09:52.930 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.930 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:09:52.930 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.930 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:09:52.930 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.639940+0000", 2024-03-20T09:09:52.930 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:52.930 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.931 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.931 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:09:52.931 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.639940+0000", 2024-03-20T09:09:52.931 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:52.931 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.931 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.931 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:09:52.931 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.639941+0000", 2024-03-20T09:09:52.931 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.372e-06 2024-03-20T09:09:52.931 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.931 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.931 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:09:52.931 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.639947+0000", 2024-03-20T09:09:52.931 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9780000000000002e-06 2024-03-20T09:09:52.932 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.932 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.932 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:09:52.932 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.639959+0000", 2024-03-20T09:09:52.932 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2362e-05 2024-03-20T09:09:52.932 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.932 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.932 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:09:52.932 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.639966+0000", 2024-03-20T09:09:52.932 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2049999999999996e-06 2024-03-20T09:09:52.932 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.932 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.932 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:09:52.933 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.640019+0000", 2024-03-20T09:09:52.933 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2485999999999999e-05 2024-03-20T09:09:52.933 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.933 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:09:52.933 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.640150+0000", 2024-03-20T09:09:52.933 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013122000000000001 2024-03-20T09:09:52.933 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.933 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 4,9", 2024-03-20T09:09:52.933 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.640267+0000", 2024-03-20T09:09:52.933 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000116663 2024-03-20T09:09:52.933 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.933 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:09:52.934 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.644329+0000", 2024-03-20T09:09:52.934 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040625749999999997 2024-03-20T09:09:52.934 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.934 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.934 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:52.934 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.647507+0000", 2024-03-20T09:09:52.934 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003177994 2024-03-20T09:09:52.934 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.934 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.934 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:52.934 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.841311+0000", 2024-03-20T09:09:52.934 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19380404900000001 2024-03-20T09:09:52.934 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.935 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.935 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:09:52.935 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.841351+0000", 2024-03-20T09:09:52.935 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9962999999999997e-05 2024-03-20T09:09:52.935 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.935 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.935 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:09:52.935 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.841378+0000", 2024-03-20T09:09:52.935 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.705e-05 2024-03-20T09:09:52.935 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:52.935 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:09:52.935 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:52.935 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.935 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.935 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:185588 2.5 2:a570ae3d:::benchmark_data_smithi122_107654_object66803:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-20T09:09:52.936 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:01:32.649315+0000", 2024-03-20T09:09:52.936 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 500.26322333100001, 2024-03-20T09:09:52.936 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.203669501, 2024-03-20T09:09:52.936 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:09:52.936 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:52.936 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:09:52.936 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:09:52.936 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:09:52.936 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 185588 2024-03-20T09:09:52.936 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.936 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:09:52.936 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.936 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:09:52.937 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.649315+0000", 2024-03-20T09:09:52.937 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:52.937 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.937 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.937 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:09:52.937 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.649315+0000", 2024-03-20T09:09:52.937 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:52.937 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.937 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.937 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:09:52.937 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.649317+0000", 2024-03-20T09:09:52.937 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.446e-06 2024-03-20T09:09:52.937 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.937 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.937 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:09:52.938 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.649324+0000", 2024-03-20T09:09:52.938 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1130000000000003e-06 2024-03-20T09:09:52.938 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.938 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.938 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:09:52.938 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.649339+0000", 2024-03-20T09:09:52.938 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5633000000000001e-05 2024-03-20T09:09:52.938 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.938 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.938 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:09:52.938 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.649348+0000", 2024-03-20T09:09:52.938 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9169999999999995e-06 2024-03-20T09:09:52.938 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.938 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.939 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:09:52.939 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.649405+0000", 2024-03-20T09:09:52.939 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.698e-05 2024-03-20T09:09:52.939 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.939 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.939 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:09:52.939 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.649542+0000", 2024-03-20T09:09:52.939 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000136935 2024-03-20T09:09:52.939 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.939 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.939 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,4", 2024-03-20T09:09:52.939 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.649672+0000", 2024-03-20T09:09:52.939 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013025600000000001 2024-03-20T09:09:52.939 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.939 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.940 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:52.940 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.657929+0000", 2024-03-20T09:09:52.940 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0082568629999999997 2024-03-20T09:09:52.940 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.940 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.940 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:09:52.940 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.657945+0000", 2024-03-20T09:09:52.940 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5486999999999999e-05 2024-03-20T09:09:52.940 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.940 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.940 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:52.940 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.852883+0000", 2024-03-20T09:09:52.940 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19493843199999999 2024-03-20T09:09:52.941 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.941 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.941 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:09:52.941 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.852936+0000", 2024-03-20T09:09:52.941 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3270999999999999e-05 2024-03-20T09:09:52.941 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.941 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.941 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:09:52.941 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.852985+0000", 2024-03-20T09:09:52.941 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8167999999999997e-05 2024-03-20T09:09:52.941 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:52.941 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:09:52.941 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:52.942 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.942 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.942 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:185591 2.5 2:aefd965d:::benchmark_data_smithi122_107654_object66806:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-20T09:09:52.942 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:01:32.658407+0000", 2024-03-20T09:09:52.942 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 500.25413163799999, 2024-03-20T09:09:52.942 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.194743105, 2024-03-20T09:09:52.942 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:09:52.942 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:52.942 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:09:52.942 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:09:52.942 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:09:52.942 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 185591 2024-03-20T09:09:52.942 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.942 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:09:52.943 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.943 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:09:52.943 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.658407+0000", 2024-03-20T09:09:52.943 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:52.943 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.943 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.943 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:09:52.943 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.658407+0000", 2024-03-20T09:09:52.943 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:52.943 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.943 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.943 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:09:52.943 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.658408+0000", 2024-03-20T09:09:52.943 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5e-06 2024-03-20T09:09:52.943 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.944 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.944 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:09:52.944 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.658415+0000", 2024-03-20T09:09:52.944 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0099999999999998e-06 2024-03-20T09:09:52.944 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.944 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.944 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:09:52.944 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.658431+0000", 2024-03-20T09:09:52.944 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5358000000000001e-05 2024-03-20T09:09:52.944 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.944 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.944 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:09:52.944 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.658439+0000", 2024-03-20T09:09:52.944 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.721e-06 2024-03-20T09:09:52.944 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.945 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.945 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:09:52.945 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.658488+0000", 2024-03-20T09:09:52.945 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8650999999999999e-05 2024-03-20T09:09:52.945 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.945 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.945 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:09:52.945 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.658592+0000", 2024-03-20T09:09:52.945 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104246 2024-03-20T09:09:52.945 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.945 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.945 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,4", 2024-03-20T09:09:52.945 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.658672+0000", 2024-03-20T09:09:52.945 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9739000000000003e-05 2024-03-20T09:09:52.946 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.946 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.946 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:09:52.946 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.663815+0000", 2024-03-20T09:09:52.946 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0051429090000000002 2024-03-20T09:09:52.946 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.946 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.946 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:52.946 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.667108+0000", 2024-03-20T09:09:52.946 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0032932349999999998 2024-03-20T09:09:52.946 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.946 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.946 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:52.946 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.853053+0000", 2024-03-20T09:09:52.947 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18594439600000001 2024-03-20T09:09:52.947 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.947 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.947 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:09:52.947 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.853108+0000", 2024-03-20T09:09:52.947 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5331999999999998e-05 2024-03-20T09:09:52.947 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.947 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.947 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:09:52.947 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.853150+0000", 2024-03-20T09:09:52.947 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2008000000000002e-05 2024-03-20T09:09:52.947 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:52.947 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:09:52.947 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:52.947 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.948 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.948 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:185600 2.f 2:f995c3b9:::benchmark_data_smithi122_107654_object66815:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-20T09:09:52.948 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:01:32.658682+0000", 2024-03-20T09:09:52.948 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 500.25385602900002, 2024-03-20T09:09:52.948 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18279983399999999, 2024-03-20T09:09:52.948 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:09:52.948 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:52.948 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:09:52.948 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:09:52.948 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:09:52.948 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 185600 2024-03-20T09:09:52.948 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.948 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:09:52.948 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.949 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:09:52.949 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.658682+0000", 2024-03-20T09:09:52.949 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:52.949 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.949 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.949 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:09:52.949 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.658682+0000", 2024-03-20T09:09:52.949 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:52.949 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.949 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.949 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:09:52.949 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.658683+0000", 2024-03-20T09:09:52.949 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9500000000000003e-07 2024-03-20T09:09:52.949 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.949 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.950 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:09:52.950 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.658686+0000", 2024-03-20T09:09:52.950 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7470000000000001e-06 2024-03-20T09:09:52.950 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.950 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.950 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:09:52.950 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.658692+0000", 2024-03-20T09:09:52.950 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5010000000000001e-06 2024-03-20T09:09:52.950 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.950 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.950 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:09:52.950 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.658698+0000", 2024-03-20T09:09:52.950 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6099999999999997e-06 2024-03-20T09:09:52.950 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.951 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.951 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:09:52.951 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.658760+0000", 2024-03-20T09:09:52.951 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.224e-05 2024-03-20T09:09:52.951 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.951 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.951 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:09:52.951 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.658882+0000", 2024-03-20T09:09:52.951 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000121863 2024-03-20T09:09:52.951 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.951 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.951 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 4,9", 2024-03-20T09:09:52.951 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.659002+0000", 2024-03-20T09:09:52.951 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011997699999999999 2024-03-20T09:09:52.952 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.952 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.952 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:09:52.952 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.667427+0000", 2024-03-20T09:09:52.952 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0084247769999999996 2024-03-20T09:09:52.952 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.952 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.952 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:52.952 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.671513+0000", 2024-03-20T09:09:52.952 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040864990000000004 2024-03-20T09:09:52.952 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.952 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.952 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:52.952 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.841409+0000", 2024-03-20T09:09:52.952 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.169896253 2024-03-20T09:09:52.953 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.953 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.953 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:09:52.953 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.841442+0000", 2024-03-20T09:09:52.953 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2387e-05 2024-03-20T09:09:52.953 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.953 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.953 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:09:52.953 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.841482+0000", 2024-03-20T09:09:52.953 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0485000000000003e-05 2024-03-20T09:09:52.953 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:52.953 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:09:52.953 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:52.954 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.954 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.954 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24550.0:185615 2.5 2:a2e93340:::benchmark_data_smithi122_107654_object66830:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-20T09:09:52.954 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:01:32.681959+0000", 2024-03-20T09:09:52.954 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 500.23057982300003, 2024-03-20T09:09:52.954 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18416463, 2024-03-20T09:09:52.954 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:09:52.954 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:52.954 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:09:52.955 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24550", 2024-03-20T09:09:52.955 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2776182695", 2024-03-20T09:09:52.955 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 185615 2024-03-20T09:09:52.955 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.955 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:09:52.955 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.955 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:09:52.956 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.681959+0000", 2024-03-20T09:09:52.956 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:52.956 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.956 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.956 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:09:52.956 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.681959+0000", 2024-03-20T09:09:52.956 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:52.956 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.956 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.956 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:09:52.956 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.681960+0000", 2024-03-20T09:09:52.956 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6819999999999999e-06 2024-03-20T09:09:52.956 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.956 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.957 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:09:52.957 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.681968+0000", 2024-03-20T09:09:52.957 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.729e-06 2024-03-20T09:09:52.957 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.957 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.957 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:09:52.957 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.681986+0000", 2024-03-20T09:09:52.957 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7961999999999998e-05 2024-03-20T09:09:52.957 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.957 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.957 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:09:52.957 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.681996+0000", 2024-03-20T09:09:52.957 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0285999999999999e-05 2024-03-20T09:09:52.957 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.957 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.958 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:09:52.958 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.682063+0000", 2024-03-20T09:09:52.958 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6970000000000004e-05 2024-03-20T09:09:52.958 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.958 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.958 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:09:52.958 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.682184+0000", 2024-03-20T09:09:52.958 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000120957 2024-03-20T09:09:52.958 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.958 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.958 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,4", 2024-03-20T09:09:52.958 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.682305+0000", 2024-03-20T09:09:52.958 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000120597 2024-03-20T09:09:52.958 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.959 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.959 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:09:52.959 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.687029+0000", 2024-03-20T09:09:52.959 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0047239170000000002 2024-03-20T09:09:52.959 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.959 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.959 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:52.959 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.687609+0000", 2024-03-20T09:09:52.959 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00058059300000000002 2024-03-20T09:09:52.959 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.959 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.959 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:52.959 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.866021+0000", 2024-03-20T09:09:52.959 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.17841179800000001 2024-03-20T09:09:52.960 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.960 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.960 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:09:52.960 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.866081+0000", 2024-03-20T09:09:52.960 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9664000000000003e-05 2024-03-20T09:09:52.960 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.960 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.960 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:09:52.960 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:01:32.866123+0000", 2024-03-20T09:09:52.960 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2475000000000002e-05 2024-03-20T09:09:52.960 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:52.960 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:09:52.960 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:52.960 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.960 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.961 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:58473 3.15 3.375d35d5 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e233)", 2024-03-20T09:09:52.961 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:03:31.876616+0000", 2024-03-20T09:09:52.961 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 381.03592224900001, 2024-03-20T09:09:52.961 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.99934791199999995, 2024-03-20T09:09:52.961 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:09:52.961 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:09:52.961 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:09:52.961 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:09:52.961 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:09:52.961 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 58473 2024-03-20T09:09:52.961 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.961 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:09:52.961 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.961 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:09:52.962 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:31.876616+0000", 2024-03-20T09:09:52.962 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:52.962 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.962 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.962 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:09:52.962 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:31.876616+0000", 2024-03-20T09:09:52.962 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:52.962 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.962 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.962 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:09:52.962 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:31.876618+0000", 2024-03-20T09:09:52.962 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3799999999999999e-06 2024-03-20T09:09:52.962 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.962 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.962 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:09:52.963 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:31.876644+0000", 2024-03-20T09:09:52.963 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6225999999999999e-05 2024-03-20T09:09:52.963 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.963 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.963 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:09:52.963 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:31.876663+0000", 2024-03-20T09:09:52.963 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9228999999999998e-05 2024-03-20T09:09:52.963 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.963 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.963 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:09:52.963 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:31.876673+0000", 2024-03-20T09:09:52.963 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5899999999999997e-06 2024-03-20T09:09:52.963 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.963 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.964 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:09:52.964 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:31.877790+0000", 2024-03-20T09:09:52.964 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0011176510000000001 2024-03-20T09:09:52.964 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.964 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.964 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:09:52.964 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:31.877800+0000", 2024-03-20T09:09:52.964 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5189999999999998e-06 2024-03-20T09:09:52.964 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.964 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.964 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:09:52.964 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:32.875951+0000", 2024-03-20T09:09:52.964 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.99815112900000003 2024-03-20T09:09:52.964 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.964 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.965 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:09:52.965 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:03:32.875964+0000", 2024-03-20T09:09:52.965 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3188e-05 2024-03-20T09:09:52.965 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:52.965 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:09:52.965 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:52.965 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.965 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.965 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:137643 3.15 3:a84eca2e:::benchmark_data_smithi122_119671_object37861:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-03-20T09:09:52.965 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:09:35.283389+0000", 2024-03-20T09:09:52.965 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 17.629149248000001, 2024-03-20T09:09:52.965 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.87423633899999997, 2024-03-20T09:09:52.965 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:09:52.965 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:52.966 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:09:52.966 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:09:52.966 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:09:52.966 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 137643 2024-03-20T09:09:52.966 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.966 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:09:52.966 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.966 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:09:52.966 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:35.283389+0000", 2024-03-20T09:09:52.966 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:52.966 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.966 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.966 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:09:52.966 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:35.283389+0000", 2024-03-20T09:09:52.967 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:52.967 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.967 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.967 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:09:52.967 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:35.283391+0000", 2024-03-20T09:09:52.967 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8870000000000001e-06 2024-03-20T09:09:52.967 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.967 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.967 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:09:52.967 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:35.283399+0000", 2024-03-20T09:09:52.967 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2120000000000003e-06 2024-03-20T09:09:52.967 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.967 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.967 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:09:52.968 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:35.283414+0000", 2024-03-20T09:09:52.968 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5121e-05 2024-03-20T09:09:52.968 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.968 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.968 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:09:52.968 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:35.283425+0000", 2024-03-20T09:09:52.968 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0197000000000001e-05 2024-03-20T09:09:52.968 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.968 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.968 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:09:52.968 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:35.283508+0000", 2024-03-20T09:09:52.968 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3332999999999996e-05 2024-03-20T09:09:52.968 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.968 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.968 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:09:52.969 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:35.283520+0000", 2024-03-20T09:09:52.969 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2140000000000001e-05 2024-03-20T09:09:52.969 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.969 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.969 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:09:52.969 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.152152+0000", 2024-03-20T09:09:52.969 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.86863241700000005 2024-03-20T09:09:52.969 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.969 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.969 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:09:52.969 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.152309+0000", 2024-03-20T09:09:52.969 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000156522 2024-03-20T09:09:52.969 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.969 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.969 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,6", 2024-03-20T09:09:52.970 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.152433+0000", 2024-03-20T09:09:52.970 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012421000000000001 2024-03-20T09:09:52.970 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.970 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.970 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:52.970 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.156255+0000", 2024-03-20T09:09:52.970 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0038215129999999999 2024-03-20T09:09:52.970 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.970 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.970 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:09:52.970 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.157263+0000", 2024-03-20T09:09:52.970 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010086170000000001 2024-03-20T09:09:52.970 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.971 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.971 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:52.971 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.157560+0000", 2024-03-20T09:09:52.971 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000296514 2024-03-20T09:09:52.971 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.971 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.971 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:09:52.971 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.157595+0000", 2024-03-20T09:09:52.971 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4700000000000003e-05 2024-03-20T09:09:52.971 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.971 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.971 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:09:52.971 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.157626+0000", 2024-03-20T09:09:52.971 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0956e-05 2024-03-20T09:09:52.972 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:52.972 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:09:52.972 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:52.972 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.972 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.972 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:137669 3.15 3:aa0bfb94:::benchmark_data_smithi122_119671_object37887:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-03-20T09:09:52.972 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:09:35.444814+0000", 2024-03-20T09:09:52.972 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 17.467724908000001, 2024-03-20T09:09:52.972 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.71353490399999997, 2024-03-20T09:09:52.972 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:09:52.972 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:52.972 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:09:52.972 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:09:52.972 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:09:52.973 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 137669 2024-03-20T09:09:52.973 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.973 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:09:52.973 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.973 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:09:52.973 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:35.444814+0000", 2024-03-20T09:09:52.973 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:52.973 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.973 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.973 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:09:52.973 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:35.444814+0000", 2024-03-20T09:09:52.973 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:52.973 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.973 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.974 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:09:52.974 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:35.444815+0000", 2024-03-20T09:09:52.974 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3999999999999999e-06 2024-03-20T09:09:52.974 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.974 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.974 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:09:52.974 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:35.444821+0000", 2024-03-20T09:09:52.974 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.494e-06 2024-03-20T09:09:52.974 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.974 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.974 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:09:52.974 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:35.444834+0000", 2024-03-20T09:09:52.974 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2507000000000001e-05 2024-03-20T09:09:52.974 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.975 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.975 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:09:52.975 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:35.444843+0000", 2024-03-20T09:09:52.975 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8999999999999995e-06 2024-03-20T09:09:52.975 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.975 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.975 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:09:52.975 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:35.444894+0000", 2024-03-20T09:09:52.975 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1595000000000003e-05 2024-03-20T09:09:52.975 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.975 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.975 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:09:52.975 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:35.444902+0000", 2024-03-20T09:09:52.975 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5700000000000004e-06 2024-03-20T09:09:52.976 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.976 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.976 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:09:52.976 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.154158+0000", 2024-03-20T09:09:52.976 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.70925570500000001 2024-03-20T09:09:52.976 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.976 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.976 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:09:52.976 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.154264+0000", 2024-03-20T09:09:52.976 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105943 2024-03-20T09:09:52.976 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.976 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.976 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,6", 2024-03-20T09:09:52.976 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.154382+0000", 2024-03-20T09:09:52.977 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000118354 2024-03-20T09:09:52.977 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.977 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.977 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:52.977 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.156320+0000", 2024-03-20T09:09:52.977 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001937652 2024-03-20T09:09:52.977 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.977 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.977 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:09:52.977 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.157418+0000", 2024-03-20T09:09:52.977 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00109829 2024-03-20T09:09:52.977 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.977 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.977 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:52.977 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.158297+0000", 2024-03-20T09:09:52.978 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00087904699999999999 2024-03-20T09:09:52.978 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.978 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.978 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:09:52.978 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.158326+0000", 2024-03-20T09:09:52.978 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8941999999999998e-05 2024-03-20T09:09:52.978 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.978 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.978 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:09:52.978 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.158348+0000", 2024-03-20T09:09:52.978 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2504999999999999e-05 2024-03-20T09:09:52.978 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:52.978 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:09:52.979 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:52.979 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.979 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.979 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:137737 3.15 3:aae73d1a:::benchmark_data_smithi122_119671_object37955:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-03-20T09:09:52.979 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:09:36.047768+0000", 2024-03-20T09:09:52.979 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 16.864770677999999, 2024-03-20T09:09:52.979 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.127718947, 2024-03-20T09:09:52.979 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:09:52.979 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:52.979 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:09:52.979 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:09:52.979 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:09:52.979 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 137737 2024-03-20T09:09:52.979 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.979 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:09:52.980 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.980 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:09:52.980 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.047768+0000", 2024-03-20T09:09:52.980 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:52.980 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.980 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.980 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:09:52.980 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.047768+0000", 2024-03-20T09:09:52.980 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:52.980 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.980 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.980 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:09:52.980 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.047769+0000", 2024-03-20T09:09:52.980 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.468e-06 2024-03-20T09:09:52.980 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.981 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.981 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:09:52.981 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.047783+0000", 2024-03-20T09:09:52.981 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3371e-05 2024-03-20T09:09:52.981 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.981 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.981 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:09:52.981 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.047795+0000", 2024-03-20T09:09:52.981 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2571999999999999e-05 2024-03-20T09:09:52.981 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.981 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.981 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:09:52.981 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.047802+0000", 2024-03-20T09:09:52.981 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9809999999999997e-06 2024-03-20T09:09:52.982 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.982 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.982 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:09:52.982 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.047850+0000", 2024-03-20T09:09:52.982 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7957999999999997e-05 2024-03-20T09:09:52.982 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.982 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.982 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:09:52.982 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.047857+0000", 2024-03-20T09:09:52.982 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5350000000000001e-06 2024-03-20T09:09:52.982 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.982 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.982 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:09:52.982 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.155197+0000", 2024-03-20T09:09:52.983 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.107340343 2024-03-20T09:09:52.983 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.983 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.983 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:09:52.983 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.155305+0000", 2024-03-20T09:09:52.983 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000108195 2024-03-20T09:09:52.983 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.983 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.983 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,6", 2024-03-20T09:09:52.983 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.155412+0000", 2024-03-20T09:09:52.983 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000106387 2024-03-20T09:09:52.983 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.983 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.983 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:52.984 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.157304+0000", 2024-03-20T09:09:52.984 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001892842 2024-03-20T09:09:52.984 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.984 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.984 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:09:52.984 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.157434+0000", 2024-03-20T09:09:52.984 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012963300000000001 2024-03-20T09:09:52.984 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.984 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.984 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:52.984 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.175386+0000", 2024-03-20T09:09:52.984 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017951534000000002 2024-03-20T09:09:52.984 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.984 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.984 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:09:52.985 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.175447+0000", 2024-03-20T09:09:52.985 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1581999999999994e-05 2024-03-20T09:09:52.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.985 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.985 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:09:52.985 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.175487+0000", 2024-03-20T09:09:52.985 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9545999999999997e-05 2024-03-20T09:09:52.985 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:52.985 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:09:52.985 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:52.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.985 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.985 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:137744 3.15 3:ae69f1ff:::benchmark_data_smithi122_119671_object37962:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-03-20T09:09:52.985 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:09:36.107530+0000", 2024-03-20T09:09:52.986 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 16.805008475000001, 2024-03-20T09:09:52.986 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.068093603000000003, 2024-03-20T09:09:52.986 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:09:52.986 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:52.986 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:09:52.986 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:09:52.986 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:09:52.986 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 137744 2024-03-20T09:09:52.986 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.986 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:09:52.986 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.986 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:09:52.986 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.107530+0000", 2024-03-20T09:09:52.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:52.987 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.987 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.987 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:09:52.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.107530+0000", 2024-03-20T09:09:52.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:52.987 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.987 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.987 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:09:52.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.107531+0000", 2024-03-20T09:09:52.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1200000000000002e-07 2024-03-20T09:09:52.987 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.987 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.987 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:09:52.988 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.107535+0000", 2024-03-20T09:09:52.988 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1010000000000002e-06 2024-03-20T09:09:52.988 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.988 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:09:52.988 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.107546+0000", 2024-03-20T09:09:52.988 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1097000000000001e-05 2024-03-20T09:09:52.988 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.988 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:09:52.988 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.107552+0000", 2024-03-20T09:09:52.988 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7599999999999999e-06 2024-03-20T09:09:52.988 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.989 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:09:52.989 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.107600+0000", 2024-03-20T09:09:52.989 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8461999999999998e-05 2024-03-20T09:09:52.989 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.989 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.989 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:09:52.989 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.107606+0000", 2024-03-20T09:09:52.989 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7289999999999997e-06 2024-03-20T09:09:52.989 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.989 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.989 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:09:52.989 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.156350+0000", 2024-03-20T09:09:52.989 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.048744005999999999 2024-03-20T09:09:52.989 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.989 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:09:52.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.156420+0000", 2024-03-20T09:09:52.990 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9926999999999996e-05 2024-03-20T09:09:52.990 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.990 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,6", 2024-03-20T09:09:52.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.156486+0000", 2024-03-20T09:09:52.990 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6507999999999994e-05 2024-03-20T09:09:52.990 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.990 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:09:52.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.157438+0000", 2024-03-20T09:09:52.990 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00095189899999999997 2024-03-20T09:09:52.990 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.991 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.991 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:52.991 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.157653+0000", 2024-03-20T09:09:52.991 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00021505499999999999 2024-03-20T09:09:52.991 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.991 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.991 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:52.991 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.175538+0000", 2024-03-20T09:09:52.991 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017884598000000002 2024-03-20T09:09:52.991 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.991 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.991 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:09:52.991 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.175586+0000", 2024-03-20T09:09:52.992 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.782e-05 2024-03-20T09:09:52.992 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.992 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.992 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:09:52.992 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.175624+0000", 2024-03-20T09:09:52.992 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7829e-05 2024-03-20T09:09:52.992 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:52.992 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:09:52.992 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:52.992 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.992 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.992 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:137783 3.15 3:ab296098:::benchmark_data_smithi122_119671_object38001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-03-20T09:09:52.992 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:09:36.234211+0000", 2024-03-20T09:09:52.992 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 16.678327231000001, 2024-03-20T09:09:52.993 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.034928793999999999, 2024-03-20T09:09:52.993 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:09:52.993 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:52.993 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:09:52.993 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:09:52.993 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:09:52.993 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 137783 2024-03-20T09:09:52.993 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.993 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:09:52.993 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.993 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:09:52.993 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.234211+0000", 2024-03-20T09:09:52.993 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:52.993 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.994 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.994 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:09:52.994 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.234211+0000", 2024-03-20T09:09:52.994 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:52.994 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.994 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.994 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:09:52.994 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.234212+0000", 2024-03-20T09:09:52.994 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2399999999999996e-07 2024-03-20T09:09:52.994 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.994 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.994 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:09:52.994 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.234217+0000", 2024-03-20T09:09:52.995 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7090000000000001e-06 2024-03-20T09:09:52.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.995 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.995 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:09:52.995 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.234226+0000", 2024-03-20T09:09:52.995 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1619999999999997e-06 2024-03-20T09:09:52.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.995 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.995 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:09:52.995 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.234232+0000", 2024-03-20T09:09:52.995 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5929999999999996e-06 2024-03-20T09:09:52.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.995 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.995 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:09:52.995 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.234276+0000", 2024-03-20T09:09:52.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4205000000000001e-05 2024-03-20T09:09:52.996 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.996 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:09:52.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.234388+0000", 2024-03-20T09:09:52.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000112657 2024-03-20T09:09:52.996 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.996 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,6", 2024-03-20T09:09:52.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.234460+0000", 2024-03-20T09:09:52.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1687999999999995e-05 2024-03-20T09:09:52.996 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.996 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.997 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:09:52.997 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.235834+0000", 2024-03-20T09:09:52.997 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001373841 2024-03-20T09:09:52.997 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.997 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.997 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:52.997 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.236046+0000", 2024-03-20T09:09:52.997 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00021241399999999999 2024-03-20T09:09:52.997 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.997 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.997 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:52.997 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.269050+0000", 2024-03-20T09:09:52.997 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.033003168999999999 2024-03-20T09:09:52.997 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.997 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.998 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:09:52.998 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.269089+0000", 2024-03-20T09:09:52.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9514999999999997e-05 2024-03-20T09:09:52.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.998 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.998 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:09:52.998 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.269140+0000", 2024-03-20T09:09:52.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0917000000000001e-05 2024-03-20T09:09:52.998 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:52.998 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:09:52.998 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:52.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.998 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.999 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:137995 3.15 3:ac2327e5:::benchmark_data_smithi122_119671_object38213:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-03-20T09:09:52.999 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:09:37.530189+0000", 2024-03-20T09:09:52.999 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 15.382349242, 2024-03-20T09:09:52.999 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.032631987000000001, 2024-03-20T09:09:52.999 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:09:52.999 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:52.999 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:09:52.999 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:09:52.999 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:09:52.999 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 137995 2024-03-20T09:09:52.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:52.999 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:09:52.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:52.999 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:09:53.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:37.530189+0000", 2024-03-20T09:09:53.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:53.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.000 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:09:53.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:37.530189+0000", 2024-03-20T09:09:53.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:53.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.000 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:09:53.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:37.530190+0000", 2024-03-20T09:09:53.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.159e-06 2024-03-20T09:09:53.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.001 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:09:53.001 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:37.530196+0000", 2024-03-20T09:09:53.001 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4519999999999996e-06 2024-03-20T09:09:53.001 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.001 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.001 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:09:53.001 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:37.530208+0000", 2024-03-20T09:09:53.001 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2053e-05 2024-03-20T09:09:53.001 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.001 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.001 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:09:53.001 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:37.530216+0000", 2024-03-20T09:09:53.001 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5750000000000001e-06 2024-03-20T09:09:53.001 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.002 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.002 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:09:53.002 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:37.530267+0000", 2024-03-20T09:09:53.002 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0967999999999997e-05 2024-03-20T09:09:53.002 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.002 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.002 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:09:53.002 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:37.530364+0000", 2024-03-20T09:09:53.002 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7097999999999998e-05 2024-03-20T09:09:53.002 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.002 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.002 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,6", 2024-03-20T09:09:53.002 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:37.530438+0000", 2024-03-20T09:09:53.002 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3196999999999997e-05 2024-03-20T09:09:53.003 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.003 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.003 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:09:53.003 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:37.532041+0000", 2024-03-20T09:09:53.003 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0016033569999999999 2024-03-20T09:09:53.003 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.003 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.003 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:53.003 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:37.532445+0000", 2024-03-20T09:09:53.003 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000403941 2024-03-20T09:09:53.003 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.003 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.003 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:53.003 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:37.562705+0000", 2024-03-20T09:09:53.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.030260298000000001 2024-03-20T09:09:53.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.004 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.004 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:09:53.004 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:37.562767+0000", 2024-03-20T09:09:53.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.135e-05 2024-03-20T09:09:53.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.004 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.004 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:09:53.004 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:37.562821+0000", 2024-03-20T09:09:53.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4539000000000001e-05 2024-03-20T09:09:53.004 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:53.004 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:09:53.004 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:53.005 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.005 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:139511 3.15 3:aa631b2c:::benchmark_data_smithi122_119671_object39729:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e423)", 2024-03-20T09:09:53.005 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:09:49.144468+0000", 2024-03-20T09:09:53.005 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 3.768070866, 2024-03-20T09:09:53.005 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.030954964000000001, 2024-03-20T09:09:53.005 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:09:53.005 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:53.005 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:09:53.005 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:09:53.005 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:09:53.005 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 139511 2024-03-20T09:09:53.005 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.005 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:09:53.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.006 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:09:53.006 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:49.144468+0000", 2024-03-20T09:09:53.006 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:53.006 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.006 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.006 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:09:53.006 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:49.144468+0000", 2024-03-20T09:09:53.006 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:53.006 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.006 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.006 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:09:53.006 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:49.144468+0000", 2024-03-20T09:09:53.006 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4799999999999997e-07 2024-03-20T09:09:53.006 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.007 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.007 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:09:53.007 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:49.144482+0000", 2024-03-20T09:09:53.007 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3200000000000001e-05 2024-03-20T09:09:53.007 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.007 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.007 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:09:53.007 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:49.144492+0000", 2024-03-20T09:09:53.007 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0251999999999999e-05 2024-03-20T09:09:53.007 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.007 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.007 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:09:53.007 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:49.144498+0000", 2024-03-20T09:09:53.007 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8209999999999998e-06 2024-03-20T09:09:53.008 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.008 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.008 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:09:53.008 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:49.144539+0000", 2024-03-20T09:09:53.008 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1637999999999997e-05 2024-03-20T09:09:53.008 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.008 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.008 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:09:53.008 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:49.144614+0000", 2024-03-20T09:09:53.008 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5240000000000005e-05 2024-03-20T09:09:53.008 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.008 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.008 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,6", 2024-03-20T09:09:53.008 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:49.144677+0000", 2024-03-20T09:09:53.008 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2779000000000003e-05 2024-03-20T09:09:53.009 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.009 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.009 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:09:53.009 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:49.146181+0000", 2024-03-20T09:09:53.009 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0015036680000000001 2024-03-20T09:09:53.009 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.009 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.009 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:53.009 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:49.146582+0000", 2024-03-20T09:09:53.009 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00040108100000000002 2024-03-20T09:09:53.009 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.009 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.009 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:53.009 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:49.175349+0000", 2024-03-20T09:09:53.010 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.028767088 2024-03-20T09:09:53.010 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.010 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.010 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:09:53.010 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:49.175389+0000", 2024-03-20T09:09:53.010 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0057e-05 2024-03-20T09:09:53.010 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.010 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.010 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:09:53.010 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:49.175423+0000", 2024-03-20T09:09:53.010 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3392000000000001e-05 2024-03-20T09:09:53.010 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:53.010 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:09:53.010 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:53.011 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.011 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.011 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:139631 3.15 3:aa7cfb69:::benchmark_data_smithi122_119671_object39849:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-03-20T09:09:53.011 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:09:50.167454+0000", 2024-03-20T09:09:53.011 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 2.74508471, 2024-03-20T09:09:53.011 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.050260246000000001, 2024-03-20T09:09:53.011 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:09:53.011 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:53.011 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:09:53.011 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:09:53.011 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:09:53.011 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 139631 2024-03-20T09:09:53.011 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.011 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:09:53.011 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.012 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:09:53.012 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:50.167454+0000", 2024-03-20T09:09:53.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:53.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.012 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.012 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:09:53.012 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:50.167454+0000", 2024-03-20T09:09:53.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:53.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.012 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.012 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:09:53.012 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:50.167455+0000", 2024-03-20T09:09:53.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.091e-06 2024-03-20T09:09:53.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.013 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.013 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:09:53.013 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:50.167460+0000", 2024-03-20T09:09:53.013 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5949999999999998e-06 2024-03-20T09:09:53.013 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.013 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.013 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:09:53.013 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:50.167473+0000", 2024-03-20T09:09:53.013 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2286000000000001e-05 2024-03-20T09:09:53.013 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.013 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.013 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:09:53.013 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:50.167482+0000", 2024-03-20T09:09:53.013 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0589999999999992e-06 2024-03-20T09:09:53.013 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.014 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.014 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:09:53.014 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:50.167534+0000", 2024-03-20T09:09:53.014 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2074999999999999e-05 2024-03-20T09:09:53.014 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.014 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.014 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op must wait for map", 2024-03-20T09:09:53.014 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:50.167538+0000", 2024-03-20T09:09:53.014 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5070000000000002e-06 2024-03-20T09:09:53.014 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.014 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.014 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:09:53.014 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:50.168636+0000", 2024-03-20T09:09:53.014 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001097359 2024-03-20T09:09:53.015 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.015 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.015 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:09:53.015 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:50.168725+0000", 2024-03-20T09:09:53.015 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8964999999999999e-05 2024-03-20T09:09:53.015 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.015 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.015 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,6", 2024-03-20T09:09:53.015 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:50.168796+0000", 2024-03-20T09:09:53.015 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1687999999999995e-05 2024-03-20T09:09:53.015 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.015 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.015 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:09:53.015 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:50.170294+0000", 2024-03-20T09:09:53.016 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014978789999999999 2024-03-20T09:09:53.016 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.016 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.016 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:53.016 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:50.170754+0000", 2024-03-20T09:09:53.016 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00045970400000000002 2024-03-20T09:09:53.016 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.016 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.016 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:53.016 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:50.217640+0000", 2024-03-20T09:09:53.016 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.046885926000000001 2024-03-20T09:09:53.016 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.016 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.016 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:09:53.017 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:50.217680+0000", 2024-03-20T09:09:53.017 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0401000000000002e-05 2024-03-20T09:09:53.017 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.017 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.017 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:09:53.017 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:50.217714+0000", 2024-03-20T09:09:53.017 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3711000000000003e-05 2024-03-20T09:09:53.017 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:53.017 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:09:53.017 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:53.017 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.017 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.017 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:139715 3.15 3:ab175f5e:::benchmark_data_smithi122_119671_object39933:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-03-20T09:09:53.017 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:09:50.985429+0000", 2024-03-20T09:09:53.018 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 1.92710922, 2024-03-20T09:09:53.018 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.007483094, 2024-03-20T09:09:53.018 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:09:53.018 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:53.018 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:09:53.018 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:09:53.018 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:09:53.018 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 139715 2024-03-20T09:09:53.018 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.018 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:09:53.018 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.018 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:09:53.018 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:50.985429+0000", 2024-03-20T09:09:53.018 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:53.018 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.019 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.019 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:09:53.019 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:50.985429+0000", 2024-03-20T09:09:53.019 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:53.019 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.019 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.019 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:09:53.019 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:50.985431+0000", 2024-03-20T09:09:53.019 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.835e-06 2024-03-20T09:09:53.019 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.019 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.019 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:09:53.019 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:50.985437+0000", 2024-03-20T09:09:53.019 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9909999999999999e-06 2024-03-20T09:09:53.020 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.020 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.020 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:09:53.020 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:50.985450+0000", 2024-03-20T09:09:53.020 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2678e-05 2024-03-20T09:09:53.020 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.020 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.020 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:09:53.020 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:50.985458+0000", 2024-03-20T09:09:53.020 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4270000000000008e-06 2024-03-20T09:09:53.020 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.020 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.020 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:09:53.020 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:50.985512+0000", 2024-03-20T09:09:53.020 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3981999999999999e-05 2024-03-20T09:09:53.021 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.021 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.021 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:09:53.021 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:50.985602+0000", 2024-03-20T09:09:53.021 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0019999999999995e-05 2024-03-20T09:09:53.021 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.021 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.021 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,6", 2024-03-20T09:09:53.021 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:50.985677+0000", 2024-03-20T09:09:53.021 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5049000000000006e-05 2024-03-20T09:09:53.021 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.021 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.021 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:09:53.021 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:50.987346+0000", 2024-03-20T09:09:53.022 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001668477 2024-03-20T09:09:53.022 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.022 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.022 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:53.022 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:50.987668+0000", 2024-03-20T09:09:53.022 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00032223100000000003 2024-03-20T09:09:53.022 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.022 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.022 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:53.022 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:50.992847+0000", 2024-03-20T09:09:53.022 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0051790949999999999 2024-03-20T09:09:53.022 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.022 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.022 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:09:53.023 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:50.992881+0000", 2024-03-20T09:09:53.023 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.358e-05 2024-03-20T09:09:53.023 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.023 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.023 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:09:53.023 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:50.992912+0000", 2024-03-20T09:09:53.023 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1729000000000001e-05 2024-03-20T09:09:53.023 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:53.023 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:09:53.023 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:53.023 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.023 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.023 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:139804 3.15 3:aa210460:::benchmark_data_smithi122_119671_object40022:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e425)", 2024-03-20T09:09:53.023 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:09:52.097670+0000", 2024-03-20T09:09:53.024 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.814868606, 2024-03-20T09:09:53.024 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010228584000000001, 2024-03-20T09:09:53.024 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:09:53.024 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:53.024 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:09:53.024 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:09:53.024 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:09:53.024 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 139804 2024-03-20T09:09:53.024 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.024 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:09:53.024 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.024 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:09:53.024 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.097670+0000", 2024-03-20T09:09:53.024 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:53.024 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.025 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.025 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:09:53.025 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.097670+0000", 2024-03-20T09:09:53.025 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:53.025 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.025 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.025 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:09:53.025 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.097670+0000", 2024-03-20T09:09:53.025 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9599999999999999e-07 2024-03-20T09:09:53.025 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.025 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.025 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:09:53.025 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.097674+0000", 2024-03-20T09:09:53.025 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8510000000000001e-06 2024-03-20T09:09:53.025 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.026 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.026 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:09:53.026 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.097684+0000", 2024-03-20T09:09:53.026 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5470000000000002e-06 2024-03-20T09:09:53.026 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.026 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.026 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:09:53.026 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.097691+0000", 2024-03-20T09:09:53.026 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5549999999999997e-06 2024-03-20T09:09:53.026 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.026 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.026 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:09:53.026 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.097734+0000", 2024-03-20T09:09:53.026 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2236e-05 2024-03-20T09:09:53.027 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.027 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.027 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:09:53.027 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.097831+0000", 2024-03-20T09:09:53.027 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7118000000000001e-05 2024-03-20T09:09:53.027 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.027 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.027 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,6", 2024-03-20T09:09:53.027 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.097905+0000", 2024-03-20T09:09:53.027 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4757000000000003e-05 2024-03-20T09:09:53.027 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.027 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.027 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:09:53.027 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.099370+0000", 2024-03-20T09:09:53.028 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0014648409999999999 2024-03-20T09:09:53.028 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.028 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.028 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:53.028 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.099967+0000", 2024-03-20T09:09:53.028 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00059666600000000004 2024-03-20T09:09:53.028 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.028 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.028 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:53.028 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.107840+0000", 2024-03-20T09:09:53.028 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0078727740000000008 2024-03-20T09:09:53.028 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.028 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.028 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:09:53.029 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.107871+0000", 2024-03-20T09:09:53.029 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1579999999999999e-05 2024-03-20T09:09:53.029 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.029 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.029 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:09:53.029 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.107898+0000", 2024-03-20T09:09:53.029 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7063000000000002e-05 2024-03-20T09:09:53.029 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:53.029 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:09:53.029 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:53.029 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.029 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.029 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.24910.0:139661 3.e e425/367 3:7618218b:::benchmark_data_smithi122_119671_object39879:head v 425'8705, mlcod=225'2531)", 2024-03-20T09:09:53.029 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:09:52.275042+0000", 2024-03-20T09:09:53.029 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.63749659199999997, 2024-03-20T09:09:53.030 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0062266530000000004, 2024-03-20T09:09:53.030 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:09:53.030 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:53.030 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:09:53.030 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.030 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:09:53.030 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.275042+0000", 2024-03-20T09:09:53.030 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:53.030 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.030 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.030 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:09:53.030 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.275042+0000", 2024-03-20T09:09:53.030 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:53.031 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.031 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.031 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:09:53.031 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.275043+0000", 2024-03-20T09:09:53.031 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0809999999999999e-06 2024-03-20T09:09:53.031 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.031 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.031 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:09:53.031 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.275052+0000", 2024-03-20T09:09:53.031 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2310000000000002e-06 2024-03-20T09:09:53.031 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.031 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.031 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:09:53.031 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.275063+0000", 2024-03-20T09:09:53.031 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.11e-05 2024-03-20T09:09:53.032 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.032 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.032 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:09:53.032 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.275070+0000", 2024-03-20T09:09:53.032 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1409999999999999e-06 2024-03-20T09:09:53.032 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.032 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.032 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:09:53.032 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.275129+0000", 2024-03-20T09:09:53.032 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8273000000000003e-05 2024-03-20T09:09:53.032 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.032 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.032 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:09:53.032 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.275156+0000", 2024-03-20T09:09:53.033 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7199000000000002e-05 2024-03-20T09:09:53.033 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.033 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.033 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:09:53.033 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.281227+0000", 2024-03-20T09:09:53.033 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0060710649999999996 2024-03-20T09:09:53.033 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.033 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.033 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:09:53.033 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.281268+0000", 2024-03-20T09:09:53.033 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1563000000000002e-05 2024-03-20T09:09:53.033 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:53.033 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:09:53.033 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:53.033 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.034 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.034 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:139899 3.9 3:97665445:::benchmark_data_smithi122_119671_object40117:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e425)", 2024-03-20T09:09:53.034 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:09:52.889414+0000", 2024-03-20T09:09:53.034 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.023124744999999999, 2024-03-20T09:09:53.034 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0065586530000000002, 2024-03-20T09:09:53.034 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:09:53.034 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:09:53.034 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:09:53.034 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:09:53.034 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:09:53.034 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 139899 2024-03-20T09:09:53.034 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.034 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:09:53.034 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.035 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:09:53.035 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.889414+0000", 2024-03-20T09:09:53.035 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:53.035 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.035 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.035 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:09:53.035 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.889414+0000", 2024-03-20T09:09:53.035 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:09:53.035 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.035 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.035 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:09:53.035 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.889415+0000", 2024-03-20T09:09:53.035 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.85e-07 2024-03-20T09:09:53.035 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.036 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.036 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:09:53.036 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.889419+0000", 2024-03-20T09:09:53.036 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6020000000000002e-06 2024-03-20T09:09:53.036 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.036 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.036 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:09:53.036 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.889430+0000", 2024-03-20T09:09:53.036 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0718e-05 2024-03-20T09:09:53.036 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.036 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.036 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:09:53.036 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.889436+0000", 2024-03-20T09:09:53.036 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9359999999999996e-06 2024-03-20T09:09:53.036 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.037 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.037 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:09:53.037 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.889480+0000", 2024-03-20T09:09:53.037 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4475999999999999e-05 2024-03-20T09:09:53.037 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.037 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.037 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:09:53.037 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.889567+0000", 2024-03-20T09:09:53.037 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6841000000000004e-05 2024-03-20T09:09:53.037 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.037 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.037 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 7,11", 2024-03-20T09:09:53.037 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.889637+0000", 2024-03-20T09:09:53.037 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0028000000000001e-05 2024-03-20T09:09:53.038 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.038 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.038 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:53.038 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.891774+0000", 2024-03-20T09:09:53.038 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0021370880000000001 2024-03-20T09:09:53.038 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.038 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.038 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:09:53.038 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.895871+0000", 2024-03-20T09:09:53.038 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00409641 2024-03-20T09:09:53.038 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.038 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.038 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:09:53.038 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.895882+0000", 2024-03-20T09:09:53.039 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1538e-05 2024-03-20T09:09:53.039 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.039 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.039 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:09:53.039 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.895928+0000", 2024-03-20T09:09:53.039 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5331999999999999e-05 2024-03-20T09:09:53.039 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:09:53.039 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:09:53.039 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:09:53.039 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:52.895972+0000", 2024-03-20T09:09:53.039 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4799000000000003e-05 2024-03-20T09:09:53.039 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:53.039 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:09:53.039 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:53.039 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:09:53.040 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:09:53.040 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:09:53.168 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:09:53.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:53 smithi122 ceph-mon[73495]: pgmap v1022: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 131 op/s; 7806/180252 objects degraded (4.331%); 22952/180252 objects misplaced (12.733%); 2.7 MiB/s, 43 objects/s recovering 2024-03-20T09:09:53.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:53 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2955260497' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:09:53.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:53 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:09:53.427 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:53 smithi026 ceph-mon[68309]: pgmap v1022: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 131 op/s; 7806/180252 objects degraded (4.331%); 22952/180252 objects misplaced (12.733%); 2.7 MiB/s, 43 objects/s recovering 2024-03-20T09:09:53.427 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:53 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2955260497' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:09:53.427 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:53 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:09:53.460 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:09:53.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:53.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:09:53.564 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:09:53.593 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:53.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:53 smithi115 ceph-mon[72476]: pgmap v1022: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 131 op/s; 7806/180252 objects degraded (4.331%); 22952/180252 objects misplaced (12.733%); 2.7 MiB/s, 43 objects/s recovering 2024-03-20T09:09:53.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:53 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2955260497' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:09:53.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:53 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:09:53.669 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_ops_in_flight 2024-03-20T09:09:53.942 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:09:54.413 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:54 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:09:54.413 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:54 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:09:54.413 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:54 smithi026 ceph-mon[68309]: osdmap e426: 12 total, 11 up, 9 in 2024-03-20T09:09:54.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:54 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:09:54.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:54 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:09:54.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:54 smithi122 ceph-mon[73495]: osdmap e426: 12 total, 11 up, 9 in 2024-03-20T09:09:54.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:54 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:09:54.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:54 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:09:54.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:54 smithi115 ceph-mon[72476]: osdmap e426: 12 total, 11 up, 9 in 2024-03-20T09:09:55.044 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:09:55.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:55.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:09:55.148 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:09:55.163 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:55 smithi122 ceph-mon[73495]: pgmap v1025: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 110 op/s; 7784/179652 objects degraded (4.333%); 22841/179652 objects misplaced (12.714%); 2.1 MiB/s, 33 objects/s recovering 2024-03-20T09:09:55.177 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:55.198 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:09:55.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:55.200 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:09:55.264 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:55 smithi026 ceph-mon[68309]: pgmap v1025: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 110 op/s; 7784/179652 objects degraded (4.333%); 22841/179652 objects misplaced (12.714%); 2.1 MiB/s, 33 objects/s recovering 2024-03-20T09:09:55.301 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:09:55.328 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:55.347 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:09:55.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:55.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:09:55.453 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:09:55.480 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:55.504 INFO:teuthology.orchestra.run.smithi026.stdout:moved 0 / 171 (0%) 2024-03-20T09:09:55.504 INFO:teuthology.orchestra.run.smithi026.stdout:avg 21.375 2024-03-20T09:09:55.504 INFO:teuthology.orchestra.run.smithi026.stdout:stddev 4.69874 -> 4.69874 (expected baseline 4.32471) 2024-03-20T09:09:55.504 INFO:teuthology.orchestra.run.smithi026.stdout:min osd.7 with 11 -> 11 pgs (0.51462 -> 0.51462 * mean) 2024-03-20T09:09:55.504 INFO:teuthology.orchestra.run.smithi026.stdout:max osd.1 with 24 -> 24 pgs (1.12281 -> 1.12281 * mean) 2024-03-20T09:09:55.504 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:09:55.504 INFO:teuthology.orchestra.run.smithi026.stdout:oload 1000 2024-03-20T09:09:55.505 INFO:teuthology.orchestra.run.smithi026.stdout:max_change 0.05 2024-03-20T09:09:55.505 INFO:teuthology.orchestra.run.smithi026.stdout:max_change_osds 4 2024-03-20T09:09:55.505 INFO:teuthology.orchestra.run.smithi026.stdout:average_utilization 0.0172 2024-03-20T09:09:55.505 INFO:teuthology.orchestra.run.smithi026.stdout:overload_utilization 0.1721 2024-03-20T09:09:55.505 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7 weight 0.4265 -> 0.4765 2024-03-20T09:09:55.507 INFO:teuthology.orchestra.run.smithi026.stderr:no change 2024-03-20T09:09:55.530 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 09:09:55 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T09:09:55.503+0000 7fe5fc38b700 -1 mgr.server reply reply (1) Operation not permitted no change 2024-03-20T09:09:55.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:55 smithi115 ceph-mon[72476]: pgmap v1025: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.7 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 110 op/s; 7784/179652 objects degraded (4.333%); 22841/179652 objects misplaced (12.714%); 2.1 MiB/s, 33 objects/s recovering 2024-03-20T09:09:55.728 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:09:55.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:55.731 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:09:55.832 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:09:55.869 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:55.995 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:09:55.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:55.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:09:56.099 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:09:56.126 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:56.145 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:09:56.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:56.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:09:56.168 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:56 smithi026 ceph-mon[68309]: from='client.15576 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:09:56.250 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:09:56.277 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:56 smithi122 ceph-mon[73495]: from='client.15576 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:09:56.280 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:56.306 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:09:56.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:56.308 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:09:56.337 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:09:56.337 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:09:56.337 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-03-20T09:09:56.337 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:09:56.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:56 smithi115 ceph-mon[72476]: from='client.15576 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:09:56.409 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:09:56.436 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:56.455 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:09:56.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:56.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:09:56.559 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:09:56.590 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:56.940 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:09:56.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:56.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:09:56.953 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-03-20T09:09:56.953 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd test-reweight-by-utilization 1000 1.0 2024-03-20T09:09:57.017 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:09:57.017 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:09:57.017 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:09:57.017 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:09:57.017 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:09:57.017 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:09:57.018 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:09:57.018 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:09:57.043 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:09:57.046 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:09:57.055 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:09:57.074 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:57.080 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:09:57.097 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:09:57.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:57.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:09:57.106 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:09:57.106 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:09:57.106 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:09:57.106 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:09:57.106 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:09:57.106 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:09:57.106 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-20T09:09:57.107 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:09:57.107 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:09:57.107 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:09:57.107 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:09:57.107 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:09:57.107 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:09:57.153 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_blocked_ops 2024-03-20T09:09:57.170 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:57 smithi026 ceph-mon[68309]: pgmap v1026: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.6 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 132 op/s; 7742/178794 objects degraded (4.330%); 22535/178794 objects misplaced (12.604%); 4.1 MiB/s, 65 objects/s recovering 2024-03-20T09:09:57.170 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:57 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4063626330' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:09:57.170 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:57 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 7742/178794 objects degraded (4.330%), 5 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:09:57.170 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:57 smithi026 ceph-mon[68309]: osdmap e427: 12 total, 11 up, 9 in 2024-03-20T09:09:57.202 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:09:57.230 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:57.251 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:09:57.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:57.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:09:57.356 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:09:57.394 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:57.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:57 smithi122 ceph-mon[73495]: pgmap v1026: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.6 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 132 op/s; 7742/178794 objects degraded (4.330%); 22535/178794 objects misplaced (12.604%); 4.1 MiB/s, 65 objects/s recovering 2024-03-20T09:09:57.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:57 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4063626330' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:09:57.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:57 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 7742/178794 objects degraded (4.330%), 5 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:09:57.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:57 smithi122 ceph-mon[73495]: osdmap e427: 12 total, 11 up, 9 in 2024-03-20T09:09:57.470 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:09:57.494 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:09:57.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:57.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:09:57.606 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:09:57.630 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:57 smithi115 ceph-mon[72476]: pgmap v1026: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.6 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 132 op/s; 7742/178794 objects degraded (4.330%); 22535/178794 objects misplaced (12.604%); 4.1 MiB/s, 65 objects/s recovering 2024-03-20T09:09:57.630 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:57 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4063626330' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:09:57.631 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:57 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 7742/178794 objects degraded (4.330%), 5 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:09:57.631 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:57 smithi115 ceph-mon[72476]: osdmap e427: 12 total, 11 up, 9 in 2024-03-20T09:09:57.632 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:57.651 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:09:57.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:57.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:09:57.755 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:09:57.791 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:57.811 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:09:57.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:57.814 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:09:57.915 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:09:57.952 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:09:57.952 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:09:57.953 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:09:57.954 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:09:57.964 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:57.999 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:09:58.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:58.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:09:58.103 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:09:58.131 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:58.175 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:58 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:09:58.175 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:58 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:09:58.232 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:09:58.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:58.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:09:58.338 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:09:58.365 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:58.386 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:09:58.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:58.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:09:58.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:58 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:09:58.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:58 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:09:58.489 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:09:58.521 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:58.547 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:09:58.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:58.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:09:58.653 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:09:58.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:58 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:09:58.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:58 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:09:58.681 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:59.029 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:09:59.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:59.032 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:09:59.133 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:09:59.159 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:59.179 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:09:59.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:59.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:09:59.284 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:09:59.311 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:09:59 smithi122 ceph-mon[73495]: pgmap v1028: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.6 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 117 op/s; 7742/178134 objects degraded (4.346%); 22408/178134 objects misplaced (12.579%); 3.0 MiB/s, 47 objects/s recovering 2024-03-20T09:09:59.314 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:59.344 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:09:59.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:59.347 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:09:59.447 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:09:59.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:09:59 smithi026 ceph-mon[68309]: pgmap v1028: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.6 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 117 op/s; 7742/178134 objects degraded (4.346%); 22408/178134 objects misplaced (12.579%); 3.0 MiB/s, 47 objects/s recovering 2024-03-20T09:09:59.479 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:09:59.634 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:09:59.634 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:09:59.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:09:59 smithi115 ceph-mon[72476]: pgmap v1028: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.6 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 117 op/s; 7742/178134 objects degraded (4.346%); 22408/178134 objects misplaced (12.579%); 3.0 MiB/s, 47 objects/s recovering 2024-03-20T09:09:59.929 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:09:59.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:09:59.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:09:59.954 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:10:00.033 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:10:00.065 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:00.139 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:10:00.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:00.145 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:10:00.180 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:00 smithi026 ceph-mon[68309]: Health detail: HEALTH_WARN noscrub flag(s) set; 1 osds down; Degraded data redundancy: 7742/178134 objects degraded (4.346%), 5 pgs degraded, 9 pgs undersized 2024-03-20T09:10:00.246 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:10:00.274 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:00.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:00 smithi122 ceph-mon[73495]: Health detail: HEALTH_WARN noscrub flag(s) set; 1 osds down; Degraded data redundancy: 7742/178134 objects degraded (4.346%), 5 pgs degraded, 9 pgs undersized 2024-03-20T09:10:00.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:00 smithi122 ceph-mon[73495]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-03-20T09:10:00.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:00 smithi122 ceph-mon[73495]: [WRN] OSD_DOWN: 1 osds down 2024-03-20T09:10:00.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:00 smithi122 ceph-mon[73495]: osd.5 (root=default,host=smithi115) is down 2024-03-20T09:10:00.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:00 smithi122 ceph-mon[73495]: [WRN] PG_DEGRADED: Degraded data redundancy: 7742/178134 objects degraded (4.346%), 5 pgs degraded, 9 pgs undersized 2024-03-20T09:10:00.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:00 smithi122 ceph-mon[73495]: pg 1.7 is stuck undersized for 2m, current state active+undersized, last acting [6,10] 2024-03-20T09:10:00.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:00 smithi122 ceph-mon[73495]: pg 1.9 is stuck undersized for 2m, current state active+undersized, last acting [2,7] 2024-03-20T09:10:00.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:00 smithi122 ceph-mon[73495]: pg 1.c is stuck undersized for 2m, current state active+undersized, last acting [0,6] 2024-03-20T09:10:00.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:00 smithi122 ceph-mon[73495]: pg 1.10 is stuck undersized for 5m, current state active+undersized, last acting [10,7] 2024-03-20T09:10:00.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:00 smithi122 ceph-mon[73495]: pg 1.12 is stuck undersized for 5m, current state active+undersized, last acting [1,6] 2024-03-20T09:10:00.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:00 smithi122 ceph-mon[73495]: pg 3.4 is stuck undersized for 5m, current state active+undersized+degraded, last acting [1,11] 2024-03-20T09:10:00.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:00 smithi122 ceph-mon[73495]: pg 3.5 is stuck undersized for 5m, current state active+undersized+degraded, last acting [2,6] 2024-03-20T09:10:00.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:00 smithi122 ceph-mon[73495]: pg 3.b is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [0,11] 2024-03-20T09:10:00.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:00 smithi122 ceph-mon[73495]: pg 3.d is active+recovering+degraded+remapped, acting [6,7,9] 2024-03-20T09:10:00.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:00 smithi122 ceph-mon[73495]: pg 3.e is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [0,10] 2024-03-20T09:10:00.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:00 smithi026 ceph-mon[68309]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-03-20T09:10:00.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:00 smithi026 ceph-mon[68309]: [WRN] OSD_DOWN: 1 osds down 2024-03-20T09:10:00.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:00 smithi026 ceph-mon[68309]: osd.5 (root=default,host=smithi115) is down 2024-03-20T09:10:00.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:00 smithi026 ceph-mon[68309]: [WRN] PG_DEGRADED: Degraded data redundancy: 7742/178134 objects degraded (4.346%), 5 pgs degraded, 9 pgs undersized 2024-03-20T09:10:00.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:00 smithi026 ceph-mon[68309]: pg 1.7 is stuck undersized for 2m, current state active+undersized, last acting [6,10] 2024-03-20T09:10:00.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:00 smithi026 ceph-mon[68309]: pg 1.9 is stuck undersized for 2m, current state active+undersized, last acting [2,7] 2024-03-20T09:10:00.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:00 smithi026 ceph-mon[68309]: pg 1.c is stuck undersized for 2m, current state active+undersized, last acting [0,6] 2024-03-20T09:10:00.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:00 smithi026 ceph-mon[68309]: pg 1.10 is stuck undersized for 5m, current state active+undersized, last acting [10,7] 2024-03-20T09:10:00.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:00 smithi026 ceph-mon[68309]: pg 1.12 is stuck undersized for 5m, current state active+undersized, last acting [1,6] 2024-03-20T09:10:00.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:00 smithi026 ceph-mon[68309]: pg 3.4 is stuck undersized for 5m, current state active+undersized+degraded, last acting [1,11] 2024-03-20T09:10:00.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:00 smithi026 ceph-mon[68309]: pg 3.5 is stuck undersized for 5m, current state active+undersized+degraded, last acting [2,6] 2024-03-20T09:10:00.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:00 smithi026 ceph-mon[68309]: pg 3.b is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [0,11] 2024-03-20T09:10:00.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:00 smithi026 ceph-mon[68309]: pg 3.d is active+recovering+degraded+remapped, acting [6,7,9] 2024-03-20T09:10:00.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:00 smithi026 ceph-mon[68309]: pg 3.e is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [0,10] 2024-03-20T09:10:00.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:00 smithi115 ceph-mon[72476]: Health detail: HEALTH_WARN noscrub flag(s) set; 1 osds down; Degraded data redundancy: 7742/178134 objects degraded (4.346%), 5 pgs degraded, 9 pgs undersized 2024-03-20T09:10:00.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:00 smithi115 ceph-mon[72476]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-03-20T09:10:00.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:00 smithi115 ceph-mon[72476]: [WRN] OSD_DOWN: 1 osds down 2024-03-20T09:10:00.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:00 smithi115 ceph-mon[72476]: osd.5 (root=default,host=smithi115) is down 2024-03-20T09:10:00.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:00 smithi115 ceph-mon[72476]: [WRN] PG_DEGRADED: Degraded data redundancy: 7742/178134 objects degraded (4.346%), 5 pgs degraded, 9 pgs undersized 2024-03-20T09:10:00.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:00 smithi115 ceph-mon[72476]: pg 1.7 is stuck undersized for 2m, current state active+undersized, last acting [6,10] 2024-03-20T09:10:00.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:00 smithi115 ceph-mon[72476]: pg 1.9 is stuck undersized for 2m, current state active+undersized, last acting [2,7] 2024-03-20T09:10:00.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:00 smithi115 ceph-mon[72476]: pg 1.c is stuck undersized for 2m, current state active+undersized, last acting [0,6] 2024-03-20T09:10:00.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:00 smithi115 ceph-mon[72476]: pg 1.10 is stuck undersized for 5m, current state active+undersized, last acting [10,7] 2024-03-20T09:10:00.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:00 smithi115 ceph-mon[72476]: pg 1.12 is stuck undersized for 5m, current state active+undersized, last acting [1,6] 2024-03-20T09:10:00.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:00 smithi115 ceph-mon[72476]: pg 3.4 is stuck undersized for 5m, current state active+undersized+degraded, last acting [1,11] 2024-03-20T09:10:00.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:00 smithi115 ceph-mon[72476]: pg 3.5 is stuck undersized for 5m, current state active+undersized+degraded, last acting [2,6] 2024-03-20T09:10:00.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:00 smithi115 ceph-mon[72476]: pg 3.b is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [0,11] 2024-03-20T09:10:00.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:00 smithi115 ceph-mon[72476]: pg 3.d is active+recovering+degraded+remapped, acting [6,7,9] 2024-03-20T09:10:00.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:00 smithi115 ceph-mon[72476]: pg 3.e is stuck undersized for 2m, current state active+recovering+undersized+degraded+remapped, last acting [0,10] 2024-03-20T09:10:01.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:01 smithi122 ceph-mon[73495]: pgmap v1029: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.6 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 95 op/s; 7714/177546 objects degraded (4.345%); 22382/177546 objects misplaced (12.606%); 2.0 MiB/s, 31 objects/s recovering 2024-03-20T09:10:01.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:01 smithi026 ceph-mon[68309]: pgmap v1029: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.6 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 95 op/s; 7714/177546 objects degraded (4.345%); 22382/177546 objects misplaced (12.606%); 2.0 MiB/s, 31 objects/s recovering 2024-03-20T09:10:01.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:01 smithi115 ceph-mon[72476]: pgmap v1029: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.6 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 95 op/s; 7714/177546 objects degraded (4.345%); 22382/177546 objects misplaced (12.606%); 2.0 MiB/s, 31 objects/s recovering 2024-03-20T09:10:01.722 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:10:01.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:01.725 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:10:01.743 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_historic_ops 2024-03-20T09:10:01.826 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:10:01.860 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:02.048 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:10:02.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:02 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 7714/177546 objects degraded (4.345%), 5 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:10:02.389 INFO:teuthology.orchestra.run.smithi026.stdout:moved 1 / 171 (0.584795%) 2024-03-20T09:10:02.389 INFO:teuthology.orchestra.run.smithi026.stdout:avg 21.375 2024-03-20T09:10:02.389 INFO:teuthology.orchestra.run.smithi026.stdout:stddev 4.69874 -> 4.36785 (expected baseline 4.32471) 2024-03-20T09:10:02.389 INFO:teuthology.orchestra.run.smithi026.stdout:min osd.7 with 11 -> 12 pgs (0.51462 -> 0.561404 * mean) 2024-03-20T09:10:02.389 INFO:teuthology.orchestra.run.smithi026.stdout:max osd.1 with 24 -> 23 pgs (1.12281 -> 1.07602 * mean) 2024-03-20T09:10:02.389 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:10:02.389 INFO:teuthology.orchestra.run.smithi026.stdout:oload 1000 2024-03-20T09:10:02.389 INFO:teuthology.orchestra.run.smithi026.stdout:max_change 1 2024-03-20T09:10:02.389 INFO:teuthology.orchestra.run.smithi026.stdout:max_change_osds 4 2024-03-20T09:10:02.389 INFO:teuthology.orchestra.run.smithi026.stdout:average_utilization 0.0170 2024-03-20T09:10:02.389 INFO:teuthology.orchestra.run.smithi026.stdout:overload_utilization 0.1697 2024-03-20T09:10:02.390 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7 weight 0.4265 -> 0.5907 2024-03-20T09:10:02.393 INFO:teuthology.orchestra.run.smithi026.stderr:no change 2024-03-20T09:10:02.411 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:10:02.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:02.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:10:02.430 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:10:02.442 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:02 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 7714/177546 objects degraded (4.345%), 5 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:10:02.516 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:10:02.543 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:02.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:02 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 7714/177546 objects degraded (4.345%), 5 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:10:02.720 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 09:10:02 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T09:10:02.387+0000 7fe5fc38b700 -1 mgr.server reply reply (1) Operation not permitted no change 2024-03-20T09:10:02.994 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:10:02.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:02.998 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:10:03.100 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:10:03.130 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:03.152 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:10:03.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:03.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:10:03.199 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:10:03.216 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-03-20T09:10:03.216 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd test-reweight-by-utilization 1000 0.05 2024-03-20T09:10:03.256 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:10:03.289 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:03.312 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:10:03.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:03.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:10:03.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:03 smithi115 ceph-mon[72476]: pgmap v1030: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.6 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 141 op/s; 7632/176127 objects degraded (4.333%); 22042/176127 objects misplaced (12.515%); 3.1 MiB/s, 49 objects/s recovering 2024-03-20T09:10:03.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:03 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:10:03.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:03 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:10:03.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:03 smithi115 ceph-mon[72476]: osdmap e428: 12 total, 11 up, 9 in 2024-03-20T09:10:03.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:03 smithi115 ceph-mon[72476]: from='client.15588 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:10:03.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:03 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4135070429' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:10:03.416 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:10:03.435 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:03 smithi026 ceph-mon[68309]: pgmap v1030: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.6 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 141 op/s; 7632/176127 objects degraded (4.333%); 22042/176127 objects misplaced (12.515%); 3.1 MiB/s, 49 objects/s recovering 2024-03-20T09:10:03.435 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:03 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:10:03.436 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:03 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:10:03.436 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:03 smithi026 ceph-mon[68309]: osdmap e428: 12 total, 11 up, 9 in 2024-03-20T09:10:03.436 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:03 smithi026 ceph-mon[68309]: from='client.15588 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:10:03.436 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:03 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4135070429' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:10:03.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:03 smithi122 ceph-mon[73495]: pgmap v1030: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.6 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 141 op/s; 7632/176127 objects degraded (4.333%); 22042/176127 objects misplaced (12.515%); 3.1 MiB/s, 49 objects/s recovering 2024-03-20T09:10:03.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:03 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:10:03.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:03 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:10:03.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:03 smithi122 ceph-mon[73495]: osdmap e428: 12 total, 11 up, 9 in 2024-03-20T09:10:03.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:03 smithi122 ceph-mon[73495]: from='client.15588 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:10:03.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:03 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4135070429' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:10:03.444 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:03.463 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:10:03.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:03.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:10:03.569 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:10:03.598 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:03.797 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:10:03.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:03.800 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:10:03.901 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:10:03.931 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:04.142 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:10:04.142 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:10:04.179 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:10:04.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:04.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:10:04.283 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:10:04.313 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:04.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:04 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:10:04.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:04 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4135070429' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:10:04.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:04 smithi122 ceph-mon[73495]: osdmap e429: 12 total, 11 up, 9 in 2024-03-20T09:10:04.461 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:10:04.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:04.463 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:10:04.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:04 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:10:04.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:04 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4135070429' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:10:04.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:04 smithi026 ceph-mon[68309]: osdmap e429: 12 total, 11 up, 9 in 2024-03-20T09:10:04.564 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:10:04.588 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:04 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:10:04.589 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:04 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4135070429' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:10:04.589 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:04 smithi115 ceph-mon[72476]: osdmap e429: 12 total, 11 up, 9 in 2024-03-20T09:10:04.590 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:04.615 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:10:04.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:04.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:10:04.719 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:10:04.751 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:04.961 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_ops_in_flight 2024-03-20T09:10:05.003 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:10:05.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:05.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:10:05.052 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:10:05.108 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:10:05.134 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:05.154 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:10:05.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:05.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:10:05.225 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:10:05.258 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:10:05.286 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:05.307 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:05 smithi026 ceph-mon[68309]: pgmap v1033: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.6 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 146 op/s; 7606/175173 objects degraded (4.342%); 21889/175173 objects misplaced (12.496%); 2.0 MiB/s, 32 objects/s recovering 2024-03-20T09:10:05.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:05 smithi115 ceph-mon[72476]: pgmap v1033: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.6 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 146 op/s; 7606/175173 objects degraded (4.342%); 21889/175173 objects misplaced (12.496%); 2.0 MiB/s, 32 objects/s recovering 2024-03-20T09:10:05.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:05 smithi122 ceph-mon[73495]: pgmap v1033: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.6 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 146 op/s; 7606/175173 objects degraded (4.342%); 21889/175173 objects misplaced (12.496%); 2.0 MiB/s, 32 objects/s recovering 2024-03-20T09:10:05.633 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:10:05.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:05.637 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:10:05.738 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:10:05.765 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:05.941 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-20T09:10:05.940+0000 7f9486d9d700 0 --2- 172.21.15.26:0/3800320652 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f948806a7e0 0x7f948810f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-03-20T09:10:05.971 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:10:05.993 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:10:06.017 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:10:06.039 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:10:06.059 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:10:06.083 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:10:06.114 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:10:06.114 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:10:06.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:06.117 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:10:06.140 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:10:06.169 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:10:06.177 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:10:06.198 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:10:06.217 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:10:06.226 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:10:06.226 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:10:06.226 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:10:06.226 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:10:06.226 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:10:06.227 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:10:06.227 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-20T09:10:06.227 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:10:06.227 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:10:06.227 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:10:06.227 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:10:06.227 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:10:06.227 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:10:06.246 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:06.284 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:10:06.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:06.290 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:10:06.391 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:10:06.418 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:06.443 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:10:06.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:06 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/831592004' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:10:06.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:06.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:10:06.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:06 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/831592004' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:10:06.551 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:10:06.577 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:06.597 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:10:06.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:06.600 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:10:06.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:06 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/831592004' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:10:06.700 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:10:06.727 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:06.745 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:10:06.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:06.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:10:06.850 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:10:06.878 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:06.931 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:10:06.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:06.934 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:10:06.983 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:10:06.984 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:10:07.034 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:10:07.060 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:07.079 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:10:07.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:07.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:10:07.184 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:10:07.212 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:07.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:07 smithi026 ceph-mon[68309]: pgmap v1034: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.5 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 168 op/s; 7551/174096 objects degraded (4.337%); 21544/174096 objects misplaced (12.375%); 3.5 MiB/s, 55 objects/s recovering 2024-03-20T09:10:07.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:07 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 7551/174096 objects degraded (4.337%), 5 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:10:07.231 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:10:07.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:07.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:10:07.243 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:10:07.243 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:10:07.243 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-03-20T09:10:07.243 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:10:07.335 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:10:07.363 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:07.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:07 smithi115 ceph-mon[72476]: pgmap v1034: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.5 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 168 op/s; 7551/174096 objects degraded (4.337%); 21544/174096 objects misplaced (12.375%); 3.5 MiB/s, 55 objects/s recovering 2024-03-20T09:10:07.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:07 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 7551/174096 objects degraded (4.337%), 5 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:10:07.563 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:07 smithi122 ceph-mon[73495]: pgmap v1034: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.5 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 168 op/s; 7551/174096 objects degraded (4.337%); 21544/174096 objects misplaced (12.375%); 3.5 MiB/s, 55 objects/s recovering 2024-03-20T09:10:07.563 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:07 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 7551/174096 objects degraded (4.337%), 5 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:10:07.949 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:10:07.986 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_blocked_ops 2024-03-20T09:10:08.013 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:10:08.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:08.017 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:10:08.118 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:10:08.144 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:08.163 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:10:08.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:08.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:10:08.211 INFO:teuthology.orchestra.run.smithi026.stdout:moved 0 / 171 (0%) 2024-03-20T09:10:08.211 INFO:teuthology.orchestra.run.smithi026.stdout:avg 21.375 2024-03-20T09:10:08.211 INFO:teuthology.orchestra.run.smithi026.stdout:stddev 4.69874 -> 4.69874 (expected baseline 4.32471) 2024-03-20T09:10:08.211 INFO:teuthology.orchestra.run.smithi026.stdout:min osd.7 with 11 -> 11 pgs (0.51462 -> 0.51462 * mean) 2024-03-20T09:10:08.211 INFO:teuthology.orchestra.run.smithi026.stdout:max osd.1 with 24 -> 24 pgs (1.12281 -> 1.12281 * mean) 2024-03-20T09:10:08.211 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:10:08.212 INFO:teuthology.orchestra.run.smithi026.stdout:oload 1000 2024-03-20T09:10:08.212 INFO:teuthology.orchestra.run.smithi026.stdout:max_change 0.05 2024-03-20T09:10:08.212 INFO:teuthology.orchestra.run.smithi026.stdout:max_change_osds 4 2024-03-20T09:10:08.212 INFO:teuthology.orchestra.run.smithi026.stdout:average_utilization 0.0168 2024-03-20T09:10:08.212 INFO:teuthology.orchestra.run.smithi026.stdout:overload_utilization 0.1684 2024-03-20T09:10:08.212 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7 weight 0.4265 -> 0.4765 2024-03-20T09:10:08.214 INFO:teuthology.orchestra.run.smithi026.stderr:no change 2024-03-20T09:10:08.267 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:10:08.283 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:10:08.293 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:08.315 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:10:08.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:08.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:10:08.419 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:10:08.448 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:08.471 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 09:10:08 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T09:10:08.210+0000 7fe5fc38b700 -1 mgr.server reply reply (1) Operation not permitted no change 2024-03-20T09:10:08.646 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:10:08.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:08.653 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:10:08.753 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:10:08.780 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:08.801 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:10:08.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:08.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:10:08.906 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:10:08.933 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:08.985 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:10:09.214 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:10:09.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:09.217 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:10:09.318 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:10:09.344 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:09.363 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:10:09.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:09.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:10:09.372 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:09 smithi026 ceph-mon[68309]: pgmap v1035: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.5 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 175 op/s; 7551/173322 objects degraded (4.357%); 21429/173322 objects misplaced (12.364%); 3.8 MiB/s, 61 objects/s recovering 2024-03-20T09:10:09.372 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:09 smithi026 ceph-mon[68309]: from='client.25480 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:10:09.467 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:10:09.492 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:09 smithi122 ceph-mon[73495]: pgmap v1035: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.5 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 175 op/s; 7551/173322 objects degraded (4.357%); 21429/173322 objects misplaced (12.364%); 3.8 MiB/s, 61 objects/s recovering 2024-03-20T09:10:09.493 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:09 smithi122 ceph-mon[73495]: from='client.25480 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:10:09.494 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:09.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:09 smithi115 ceph-mon[72476]: pgmap v1035: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.5 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 175 op/s; 7551/173322 objects degraded (4.357%); 21429/173322 objects misplaced (12.364%); 3.8 MiB/s, 61 objects/s recovering 2024-03-20T09:10:09.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:09 smithi115 ceph-mon[72476]: from='client.25480 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-20T09:10:09.743 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:10:09.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:09.747 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:10:09.848 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:10:09.875 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:10.073 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:10:10.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:10.076 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:10:10.177 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:10:10.206 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:10.226 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:10:10.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:10.228 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:10:10.330 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:10:10.363 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:10.373 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:10:10.375 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:10:10.376 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:10:10.376 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-03-20T09:10:10.376 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-03-20T09:10:10.376 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:10:10.386 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:10:10.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:10.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:10:10.490 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:10:10.518 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:10.590 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:10:10.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:10.593 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:10:10.694 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:10:10.725 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:10.746 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:10:10.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:10.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:10:10.851 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:10:10.878 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:11.128 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:10:11.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:11.131 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:10:11.137 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_historic_ops 2024-03-20T09:10:11.215 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:10:11.233 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:10:11.258 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:11.277 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:10:11.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:11.284 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:10:11.385 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:10:11.411 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:11.434 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:10:11.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:11.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:10:11.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:11 smithi122 ceph-mon[73495]: pgmap v1036: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.5 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 136 op/s; 7527/172860 objects degraded (4.354%); 21411/172860 objects misplaced (12.386%); 2.4 MiB/s, 38 objects/s recovering 2024-03-20T09:10:11.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:11 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1154329767' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:10:11.455 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:10:11.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:11 smithi026 ceph-mon[68309]: pgmap v1036: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.5 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 136 op/s; 7527/172860 objects degraded (4.354%); 21411/172860 objects misplaced (12.386%); 2.4 MiB/s, 38 objects/s recovering 2024-03-20T09:10:11.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:11 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1154329767' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:10:11.538 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:10:11.564 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:11 smithi115 ceph-mon[72476]: pgmap v1036: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.5 GiB data, 14 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 136 op/s; 7527/172860 objects degraded (4.354%); 21411/172860 objects misplaced (12.386%); 2.4 MiB/s, 38 objects/s recovering 2024-03-20T09:10:11.564 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:11 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1154329767' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:10:11.566 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:11.586 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:10:11.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:11.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:10:11.690 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:10:11.717 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:11.769 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:10:11.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:11.772 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:10:11.873 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:10:11.903 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:11.956 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:10:11.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:11.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:10:12.060 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:10:12.090 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:12.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:12 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:10:12.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:12 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1154329767' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:10:12.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:12 smithi122 ceph-mon[73495]: osdmap e430: 12 total, 11 up, 9 in 2024-03-20T09:10:12.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:12 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 7527/172860 objects degraded (4.354%), 5 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:10:12.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:12 smithi122 ceph-mon[73495]: osdmap e431: 12 total, 11 up, 9 in 2024-03-20T09:10:12.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:12 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:10:12.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:12 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1154329767' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:10:12.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:12 smithi026 ceph-mon[68309]: osdmap e430: 12 total, 11 up, 9 in 2024-03-20T09:10:12.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:12 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 7527/172860 objects degraded (4.354%), 5 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:10:12.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:12 smithi026 ceph-mon[68309]: osdmap e431: 12 total, 11 up, 9 in 2024-03-20T09:10:12.640 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:10:12.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:12.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:10:12.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:12 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:10:12.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:12 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1154329767' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:10:12.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:12 smithi115 ceph-mon[72476]: osdmap e430: 12 total, 11 up, 9 in 2024-03-20T09:10:12.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:12 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 7527/172860 objects degraded (4.354%), 5 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:10:12.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:12 smithi115 ceph-mon[72476]: osdmap e431: 12 total, 11 up, 9 in 2024-03-20T09:10:12.744 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:10:12.773 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:12.896 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:10:12.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:12.899 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:10:13.000 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:10:13.027 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:13.228 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:10:13.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:13.232 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:10:13.273 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:10:13.293 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:10:13.314 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:10:13.332 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:10:13.335 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:10:13.355 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:10:13.359 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:13.376 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:10:13.396 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:10:13.412 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:10:13.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:13.414 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:10:13.416 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:10:13.437 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:10:13.457 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:10:13.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:13 smithi026 ceph-mon[68309]: pgmap v1038: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.5 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 135 op/s; 7470/171867 objects degraded (4.346%); 21051/171867 objects misplaced (12.248%); 3.9 MiB/s, 62 objects/s recovering 2024-03-20T09:10:13.479 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:10:13.480 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:10:13.480 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:10:13.480 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:10:13.480 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:10:13.480 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:10:13.480 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-20T09:10:13.480 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:10:13.480 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:10:13.480 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:10:13.480 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:10:13.480 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:10:13.481 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:10:13.515 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:10:13.547 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:13.549 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:10:13.550 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:10:13.564 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:13 smithi122 ceph-mon[73495]: pgmap v1038: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.5 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 135 op/s; 7470/171867 objects degraded (4.346%); 21051/171867 objects misplaced (12.248%); 3.9 MiB/s, 62 objects/s recovering 2024-03-20T09:10:13.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:13 smithi115 ceph-mon[72476]: pgmap v1038: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.5 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 135 op/s; 7470/171867 objects degraded (4.346%); 21051/171867 objects misplaced (12.248%); 3.9 MiB/s, 62 objects/s recovering 2024-03-20T09:10:13.696 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:10:13.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:13.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:10:13.800 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:10:13.832 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:14.285 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:10:14.286 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:10:14.286 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:10:14.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:14.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:10:14.294 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_ops_in_flight 2024-03-20T09:10:14.388 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:10:14.417 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:14.436 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:10:14.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:14.442 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:10:14.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:14 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3979800398' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:10:14.543 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:10:14.567 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:14 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3979800398' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:10:14.568 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:10:14.569 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:14.589 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:10:14.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:14.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:10:14.625 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:10:14.694 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:10:14.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:14 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3979800398' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:10:14.722 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:14.922 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:10:14.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:14.926 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:10:14.943 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:10:15.027 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:10:15.054 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:15.073 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:10:15.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:15.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:10:15.163 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 8, 10, 11] out_osds: [9, 3, 4] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11] 2024-03-20T09:10:15.163 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:10:15.163 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-03-20T09:10:15.163 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd in 3 2024-03-20T09:10:15.176 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:10:15.205 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:15.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:15 smithi115 ceph-mon[72476]: 1.8 scrub starts 2024-03-20T09:10:15.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:15 smithi115 ceph-mon[72476]: 1.8 scrub ok 2024-03-20T09:10:15.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:15 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:10:15.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:15 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:10:15.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:15 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:10:15.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:15 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:10:15.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:15 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:10:15.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:15 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:10:15.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:15 smithi115 ceph-mon[72476]: 1.13 scrub starts 2024-03-20T09:10:15.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:15 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:10:15.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:15 smithi115 ceph-mon[72476]: 1.13 scrub ok 2024-03-20T09:10:15.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:15 smithi122 ceph-mon[73495]: 1.8 scrub starts 2024-03-20T09:10:15.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:15 smithi122 ceph-mon[73495]: 1.8 scrub ok 2024-03-20T09:10:15.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:15 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:10:15.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:15 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:10:15.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:15 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:10:15.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:15 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:10:15.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:15 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:10:15.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:15 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:10:15.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:15 smithi122 ceph-mon[73495]: 1.13 scrub starts 2024-03-20T09:10:15.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:15 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:10:15.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:15 smithi122 ceph-mon[73495]: 1.13 scrub ok 2024-03-20T09:10:15.448 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:15 smithi026 ceph-mon[68309]: 1.8 scrub starts 2024-03-20T09:10:15.448 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:15 smithi026 ceph-mon[68309]: 1.8 scrub ok 2024-03-20T09:10:15.449 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:15 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:10:15.449 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:15 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:10:15.449 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:15 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:10:15.449 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:15 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:10:15.449 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:15 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:10:15.449 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:15 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:10:15.449 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:15 smithi026 ceph-mon[68309]: 1.13 scrub starts 2024-03-20T09:10:15.449 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:15 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:10:15.449 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:15 smithi026 ceph-mon[68309]: 1.13 scrub ok 2024-03-20T09:10:15.555 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:10:15.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:15.558 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:10:15.659 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:10:15.691 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:15.750 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:10:15.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:15.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:10:15.822 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:10:15.854 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:10:15.883 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:16.050 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:10:16.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:16.053 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:10:16.154 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:10:16.181 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:16.199 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:10:16.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:16.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:10:16.287 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:10:16.302 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:10:16.330 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:16.385 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:10:16.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:16.389 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:10:16.490 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:10:16.527 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:16.617 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:16 smithi026 ceph-mon[68309]: 1.18 scrub starts 2024-03-20T09:10:16.618 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:16 smithi026 ceph-mon[68309]: 1.18 scrub ok 2024-03-20T09:10:16.618 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:16 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:10:16.618 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:16 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:10:16.618 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:16 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:10:16.618 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:16 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:10:16.618 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:16 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:10:16.630 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:10:16.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:16.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:10:16.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:16 smithi115 ceph-mon[72476]: 1.18 scrub starts 2024-03-20T09:10:16.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:16 smithi115 ceph-mon[72476]: 1.18 scrub ok 2024-03-20T09:10:16.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:16 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:10:16.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:16 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:10:16.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:16 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:10:16.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:16 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:10:16.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:16 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:10:16.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:16 smithi122 ceph-mon[73495]: 1.18 scrub starts 2024-03-20T09:10:16.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:16 smithi122 ceph-mon[73495]: 1.18 scrub ok 2024-03-20T09:10:16.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:16 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:10:16.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:16 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:10:16.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:16 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:10:16.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:16 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:10:16.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:16 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:10:16.733 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:10:16.733 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:10:16.737 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:10:16.769 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:16.968 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:10:16.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:16.971 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:10:17.072 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:10:17.102 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:17.252 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:10:17.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:17.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:10:17.356 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:10:17.386 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:17 smithi122 ceph-mon[73495]: pgmap v1041: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.5 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 126 op/s; 7414/170277 objects degraded (4.354%); 20559/170277 objects misplaced (12.074%); 6.1 MiB/s, 97 objects/s recovering 2024-03-20T09:10:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:17 smithi122 ceph-mon[73495]: 3.13 scrub starts 2024-03-20T09:10:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:17 smithi122 ceph-mon[73495]: 3.13 scrub ok 2024-03-20T09:10:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:17 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:10:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:17 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:10:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:17 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 7414/170277 objects degraded (4.354%), 5 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:10:17.511 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:17 smithi026 ceph-mon[68309]: pgmap v1041: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.5 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 126 op/s; 7414/170277 objects degraded (4.354%); 20559/170277 objects misplaced (12.074%); 6.1 MiB/s, 97 objects/s recovering 2024-03-20T09:10:17.511 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:17 smithi026 ceph-mon[68309]: 3.13 scrub starts 2024-03-20T09:10:17.511 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:17 smithi026 ceph-mon[68309]: 3.13 scrub ok 2024-03-20T09:10:17.512 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:17 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:10:17.512 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:17 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:10:17.512 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:17 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 7414/170277 objects degraded (4.354%), 5 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:10:17.582 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_blocked_ops 2024-03-20T09:10:17.584 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:10:17.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:17.588 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:10:17.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:17 smithi115 ceph-mon[72476]: pgmap v1041: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.5 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 126 op/s; 7414/170277 objects degraded (4.354%); 20559/170277 objects misplaced (12.074%); 6.1 MiB/s, 97 objects/s recovering 2024-03-20T09:10:17.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:17 smithi115 ceph-mon[72476]: 3.13 scrub starts 2024-03-20T09:10:17.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:17 smithi115 ceph-mon[72476]: 3.13 scrub ok 2024-03-20T09:10:17.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:17 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:10:17.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:17 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:10:17.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:17 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 7414/170277 objects degraded (4.354%), 5 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:10:17.689 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:10:17.720 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:17.743 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:10:17.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:17.747 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:10:17.848 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:10:17.878 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:17.879 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:10:17.902 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:10:17.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:17.907 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:10:18.008 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:10:18.034 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:18.182 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:10:18.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:18.185 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:10:18.287 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:10:18.323 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:18.382 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:10:18.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:18 smithi122 ceph-mon[73495]: osdmap e432: 12 total, 11 up, 9 in 2024-03-20T09:10:18.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:18 smithi122 ceph-mon[73495]: 1.1a scrub starts 2024-03-20T09:10:18.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:18 smithi122 ceph-mon[73495]: 1.1a scrub ok 2024-03-20T09:10:18.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:18 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2110402062' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:10:18.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:18 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:10:18.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:18 smithi115 ceph-mon[72476]: osdmap e432: 12 total, 11 up, 9 in 2024-03-20T09:10:18.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:18 smithi115 ceph-mon[72476]: 1.1a scrub starts 2024-03-20T09:10:18.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:18 smithi115 ceph-mon[72476]: 1.1a scrub ok 2024-03-20T09:10:18.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:18 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2110402062' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:10:18.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:18 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:10:18.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:18 smithi026 ceph-mon[68309]: osdmap e432: 12 total, 11 up, 9 in 2024-03-20T09:10:18.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:18 smithi026 ceph-mon[68309]: 1.1a scrub starts 2024-03-20T09:10:18.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:18 smithi026 ceph-mon[68309]: 1.1a scrub ok 2024-03-20T09:10:18.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:18 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2110402062' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:10:18.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:18 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:10:18.751 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:10:18.775 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:10:18.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:18.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:10:18.880 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:10:18.914 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:19.040 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:18 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mon-a[68286]: 2024-03-20T09:10:18.815+0000 7f683364b700 -1 mon.a@0(leader).osd e433 definitely_dead 0 2024-03-20T09:10:19.040 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:10:19.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:19.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:10:19.144 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:10:19.175 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:19.379 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:19 smithi026 ceph-mon[68309]: pgmap v1043: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.5 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 127 op/s; 7414/169569 objects degraded (4.372%); 20441/169569 objects misplaced (12.055%); 5.4 MiB/s, 86 objects/s recovering 2024-03-20T09:10:19.379 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:19 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:10:19.379 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:19 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:10:19.380 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:19 smithi026 ceph-mon[68309]: osdmap e433: 12 total, 11 up, 9 in 2024-03-20T09:10:19.380 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:19 smithi026 ceph-mon[68309]: 1.3 deep-scrub starts 2024-03-20T09:10:19.380 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:19 smithi026 ceph-mon[68309]: 1.3 deep-scrub ok 2024-03-20T09:10:19.380 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:19 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/929594871' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-03-20T09:10:19.380 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:19 smithi026 ceph-mon[68309]: 3.2 scrub starts 2024-03-20T09:10:19.385 INFO:teuthology.orchestra.run.smithi026.stderr:marked in osd.3. 2024-03-20T09:10:19.573 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:19 smithi122 ceph-mon[73495]: pgmap v1043: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.5 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 127 op/s; 7414/169569 objects degraded (4.372%); 20441/169569 objects misplaced (12.055%); 5.4 MiB/s, 86 objects/s recovering 2024-03-20T09:10:19.574 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:19 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:10:19.574 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:19 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:10:19.574 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:19 smithi122 ceph-mon[73495]: osdmap e433: 12 total, 11 up, 9 in 2024-03-20T09:10:19.574 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:19 smithi122 ceph-mon[73495]: 1.3 deep-scrub starts 2024-03-20T09:10:19.574 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:19 smithi122 ceph-mon[73495]: 1.3 deep-scrub ok 2024-03-20T09:10:19.574 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:19 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/929594871' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-03-20T09:10:19.574 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:19 smithi122 ceph-mon[73495]: 3.2 scrub starts 2024-03-20T09:10:19.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:19 smithi115 ceph-mon[72476]: pgmap v1043: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.5 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 127 op/s; 7414/169569 objects degraded (4.372%); 20441/169569 objects misplaced (12.055%); 5.4 MiB/s, 86 objects/s recovering 2024-03-20T09:10:19.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:19 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:10:19.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:19 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:10:19.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:19 smithi115 ceph-mon[72476]: osdmap e433: 12 total, 11 up, 9 in 2024-03-20T09:10:19.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:19 smithi115 ceph-mon[72476]: 1.3 deep-scrub starts 2024-03-20T09:10:19.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:19 smithi115 ceph-mon[72476]: 1.3 deep-scrub ok 2024-03-20T09:10:19.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:19 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/929594871' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-03-20T09:10:19.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:19 smithi115 ceph-mon[72476]: 3.2 scrub starts 2024-03-20T09:10:19.903 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:10:19.903 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:10:20.016 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-03-20T09:10:20.428 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:10:20.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:20.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:10:20.533 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:10:20.559 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:20 smithi122 ceph-mon[73495]: 3.2 scrub ok 2024-03-20T09:10:20.560 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:20 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/929594871' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-03-20T09:10:20.560 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:20 smithi122 ceph-mon[73495]: osdmap e434: 12 total, 11 up, 10 in 2024-03-20T09:10:20.560 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:20 smithi122 ceph-mon[73495]: 3.9 scrub starts 2024-03-20T09:10:20.560 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:20 smithi122 ceph-mon[73495]: 3.a scrub starts 2024-03-20T09:10:20.562 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:20.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:20 smithi115 ceph-mon[72476]: 3.2 scrub ok 2024-03-20T09:10:20.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:20 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/929594871' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-03-20T09:10:20.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:20 smithi115 ceph-mon[72476]: osdmap e434: 12 total, 11 up, 10 in 2024-03-20T09:10:20.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:20 smithi115 ceph-mon[72476]: 3.9 scrub starts 2024-03-20T09:10:20.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:20 smithi115 ceph-mon[72476]: 3.a scrub starts 2024-03-20T09:10:20.661 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:10:20.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:20.664 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:10:20.668 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_historic_ops 2024-03-20T09:10:20.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:20 smithi026 ceph-mon[68309]: 3.2 scrub ok 2024-03-20T09:10:20.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:20 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/929594871' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-03-20T09:10:20.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:20 smithi026 ceph-mon[68309]: osdmap e434: 12 total, 11 up, 10 in 2024-03-20T09:10:20.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:20 smithi026 ceph-mon[68309]: 3.9 scrub starts 2024-03-20T09:10:20.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:20 smithi026 ceph-mon[68309]: 3.a scrub starts 2024-03-20T09:10:20.765 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:10:20.793 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:20.817 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:10:20.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:20.821 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:10:20.922 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:10:20.948 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:20.966 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:10:20.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:20.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:10:20.982 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:10:21.070 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:10:21.099 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:21.223 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:10:21.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:21.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:10:21.329 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:10:21.358 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:21.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:21 smithi026 ceph-mon[68309]: 1.2 deep-scrub starts 2024-03-20T09:10:21.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:21 smithi026 ceph-mon[68309]: 1.2 deep-scrub ok 2024-03-20T09:10:21.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:21 smithi026 ceph-mon[68309]: pgmap v1046: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.4 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 102 op/s; 7394/169257 objects degraded (4.369%); 20429/169257 objects misplaced (12.070%); 5.4 MiB/s, 86 objects/s recovering 2024-03-20T09:10:21.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:21 smithi026 ceph-mon[68309]: 3.9 scrub ok 2024-03-20T09:10:21.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:21 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:10:21.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:21 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:10:21.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:21 smithi026 ceph-mon[68309]: osdmap e435: 12 total, 11 up, 10 in 2024-03-20T09:10:21.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:21 smithi026 ceph-mon[68309]: 3.a scrub ok 2024-03-20T09:10:21.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:21 smithi026 ceph-mon[68309]: 3.15 scrub starts 2024-03-20T09:10:21.504 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:10:21.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:21.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:10:21.519 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:10:21.598 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:10:21.610 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:10:21.637 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:10:21.638 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:21.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:21 smithi115 ceph-mon[72476]: 1.2 deep-scrub starts 2024-03-20T09:10:21.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:21 smithi115 ceph-mon[72476]: 1.2 deep-scrub ok 2024-03-20T09:10:21.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:21 smithi115 ceph-mon[72476]: pgmap v1046: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.4 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 102 op/s; 7394/169257 objects degraded (4.369%); 20429/169257 objects misplaced (12.070%); 5.4 MiB/s, 86 objects/s recovering 2024-03-20T09:10:21.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:21 smithi115 ceph-mon[72476]: 3.9 scrub ok 2024-03-20T09:10:21.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:21 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:10:21.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:21 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:10:21.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:21 smithi115 ceph-mon[72476]: osdmap e435: 12 total, 11 up, 10 in 2024-03-20T09:10:21.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:21 smithi115 ceph-mon[72476]: 3.a scrub ok 2024-03-20T09:10:21.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:21 smithi115 ceph-mon[72476]: 3.15 scrub starts 2024-03-20T09:10:21.657 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:10:21.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:21.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:10:21.676 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:10:21.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:21 smithi122 ceph-mon[73495]: 1.2 deep-scrub starts 2024-03-20T09:10:21.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:21 smithi122 ceph-mon[73495]: 1.2 deep-scrub ok 2024-03-20T09:10:21.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:21 smithi122 ceph-mon[73495]: pgmap v1046: 57 pgs: 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 1 active+recovering+degraded+remapped, 34 active+clean, 5 active+undersized; 3.4 GiB data, 13 GiB used, 791 GiB / 805 GiB avail; 0 B/s wr, 102 op/s; 7394/169257 objects degraded (4.369%); 20429/169257 objects misplaced (12.070%); 5.4 MiB/s, 86 objects/s recovering 2024-03-20T09:10:21.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:21 smithi122 ceph-mon[73495]: 3.9 scrub ok 2024-03-20T09:10:21.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:21 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:10:21.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:21 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:10:21.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:21 smithi122 ceph-mon[73495]: osdmap e435: 12 total, 11 up, 10 in 2024-03-20T09:10:21.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:21 smithi122 ceph-mon[73495]: 3.a scrub ok 2024-03-20T09:10:21.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:21 smithi122 ceph-mon[73495]: 3.15 scrub starts 2024-03-20T09:10:21.707 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:10:21.742 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:10:21.761 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:10:21.775 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:10:21.789 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:21.800 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:10:21.830 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:10:21.841 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:10:21.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:21.844 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:10:21.870 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:10:21.909 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:10:21.945 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:10:21.960 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:10:21.960 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:10:21.960 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:10:21.960 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:10:21.960 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:10:21.960 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:10:21.960 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-20T09:10:21.961 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:10:21.961 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:10:21.961 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:10:21.961 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:10:21.961 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:10:21.961 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:10:21.968 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:10:21.997 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:22.226 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:10:22.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:22.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:10:22.331 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:10:22.370 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:22.454 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:10:22.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:22.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:10:22.561 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:10:22.592 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:22.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:22 smithi115 ceph-mon[72476]: 3.15 scrub ok 2024-03-20T09:10:22.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:22 smithi115 ceph-mon[72476]: osdmap e436: 12 total, 11 up, 10 in 2024-03-20T09:10:22.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:22 smithi115 ceph-mon[72476]: 3.b starting backfill to osd.3 from (0'0,289'6937] MIN to 433'9089 2024-03-20T09:10:22.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:22 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1352275971' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:10:22.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:22 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 7394/169257 objects degraded (4.369%), 5 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:10:22.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:22 smithi115 ceph-mon[72476]: osdmap e437: 12 total, 11 up, 10 in 2024-03-20T09:10:22.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:22 smithi122 ceph-mon[73495]: 3.15 scrub ok 2024-03-20T09:10:22.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:22 smithi122 ceph-mon[73495]: osdmap e436: 12 total, 11 up, 10 in 2024-03-20T09:10:22.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:22 smithi122 ceph-mon[73495]: 3.b starting backfill to osd.3 from (0'0,289'6937] MIN to 433'9089 2024-03-20T09:10:22.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:22 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1352275971' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:10:22.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:22 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 7394/169257 objects degraded (4.369%), 5 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:10:22.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:22 smithi122 ceph-mon[73495]: osdmap e437: 12 total, 11 up, 10 in 2024-03-20T09:10:22.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:22 smithi026 ceph-mon[68309]: 3.15 scrub ok 2024-03-20T09:10:22.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:22 smithi026 ceph-mon[68309]: osdmap e436: 12 total, 11 up, 10 in 2024-03-20T09:10:22.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:22 smithi026 ceph-mon[68309]: 3.b starting backfill to osd.3 from (0'0,289'6937] MIN to 433'9089 2024-03-20T09:10:22.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:22 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1352275971' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:10:22.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:22 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 7394/169257 objects degraded (4.369%), 5 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:10:22.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:22 smithi026 ceph-mon[68309]: osdmap e437: 12 total, 11 up, 10 in 2024-03-20T09:10:22.943 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:10:22.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:22.946 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:10:23.000 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:10:23.000 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:10:23.047 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:10:23.083 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:23.119 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:10:23.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:23.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:10:23.224 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:10:23.258 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:23.466 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:10:23.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:23.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:10:23.543 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:10:23.543 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-03-20T09:10:23.543 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-03-20T09:10:23.543 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-03-20T09:10:23.543 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.543 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:109470 3.7 3:e4fe868b:::benchmark_data_smithi122_119671_object9688:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-03-20T09:10:23.543 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:05:18.976612+0000", 2024-03-20T09:10:23.543 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 304.54593846300003, 2024-03-20T09:10:23.543 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2614700619999999, 2024-03-20T09:10:23.543 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:10:23.543 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:10:23.544 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:10:23.544 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:10:23.544 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:10:23.544 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 109470 2024-03-20T09:10:23.544 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.544 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:10:23.544 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.544 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:10:23.544 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:18.976612+0000", 2024-03-20T09:10:23.544 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.544 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.544 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.544 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:10:23.544 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:18.976612+0000", 2024-03-20T09:10:23.545 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.545 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.545 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.545 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:10:23.545 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:18.976613+0000", 2024-03-20T09:10:23.545 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.215e-06 2024-03-20T09:10:23.545 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.545 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.545 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:10:23.545 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:18.976619+0000", 2024-03-20T09:10:23.545 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9789999999999998e-06 2024-03-20T09:10:23.545 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.545 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.546 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:10:23.546 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:18.976631+0000", 2024-03-20T09:10:23.546 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2361000000000001e-05 2024-03-20T09:10:23.546 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.546 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.546 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:10:23.546 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:18.976637+0000", 2024-03-20T09:10:23.546 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2779999999999999e-06 2024-03-20T09:10:23.546 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.546 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.546 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.546 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:18.976691+0000", 2024-03-20T09:10:23.546 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3838999999999997e-05 2024-03-20T09:10:23.547 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.547 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.547 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:10:23.547 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:18.976700+0000", 2024-03-20T09:10:23.547 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9020000000000005e-06 2024-03-20T09:10:23.547 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.547 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.547 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.547 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.965395+0000", 2024-03-20T09:10:23.547 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.98869506799999995 2024-03-20T09:10:23.547 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.547 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.547 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:10:23.547 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.965408+0000", 2024-03-20T09:10:23.548 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.273e-05 2024-03-20T09:10:23.548 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.548 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.548 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.548 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.231957+0000", 2024-03-20T09:10:23.548 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.266549559 2024-03-20T09:10:23.548 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.548 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.548 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:10:23.548 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.232118+0000", 2024-03-20T09:10:23.548 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000160545 2024-03-20T09:10:23.548 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.548 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.548 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:10:23.549 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.232261+0000", 2024-03-20T09:10:23.549 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014308699999999999 2024-03-20T09:10:23.549 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.549 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.549 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.549 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.237013+0000", 2024-03-20T09:10:23.549 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0047517230000000002 2024-03-20T09:10:23.549 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.549 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.549 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:10:23.549 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.237027+0000", 2024-03-20T09:10:23.549 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3733e-05 2024-03-20T09:10:23.549 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.550 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.550 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.550 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.237061+0000", 2024-03-20T09:10:23.550 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4660999999999999e-05 2024-03-20T09:10:23.550 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.550 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.550 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:10:23.550 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.237101+0000", 2024-03-20T09:10:23.550 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0173999999999999e-05 2024-03-20T09:10:23.550 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.550 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.550 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:10:23.550 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.238082+0000", 2024-03-20T09:10:23.551 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00098020799999999999 2024-03-20T09:10:23.551 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.551 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:10:23.551 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.551 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.551 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.551 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:109473 3.e 3:7e47c95e:::benchmark_data_smithi122_119671_object9691:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-03-20T09:10:23.551 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:05:19.000617+0000", 2024-03-20T09:10:23.551 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 304.52193318399998, 2024-03-20T09:10:23.551 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1810451529999999, 2024-03-20T09:10:23.551 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:10:23.551 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:10:23.551 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:10:23.552 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:10:23.552 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:10:23.552 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 109473 2024-03-20T09:10:23.552 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.552 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:10:23.552 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.552 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:10:23.552 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.000617+0000", 2024-03-20T09:10:23.552 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.552 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.552 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.552 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:10:23.552 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.000617+0000", 2024-03-20T09:10:23.552 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.552 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.553 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.553 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:10:23.553 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.000618+0000", 2024-03-20T09:10:23.553 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5199999999999995e-07 2024-03-20T09:10:23.553 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.553 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.553 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:10:23.553 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.000622+0000", 2024-03-20T09:10:23.553 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8909999999999997e-06 2024-03-20T09:10:23.553 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.553 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.553 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:10:23.553 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.000632+0000", 2024-03-20T09:10:23.553 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0019e-05 2024-03-20T09:10:23.554 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.554 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.554 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:10:23.554 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.000637+0000", 2024-03-20T09:10:23.554 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.835e-06 2024-03-20T09:10:23.554 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.554 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.554 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.554 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.000696+0000", 2024-03-20T09:10:23.554 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8306e-05 2024-03-20T09:10:23.554 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.554 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.554 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:10:23.554 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.000703+0000", 2024-03-20T09:10:23.555 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7230000000000004e-06 2024-03-20T09:10:23.555 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.555 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.555 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.555 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.164628+0000", 2024-03-20T09:10:23.555 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1639244790000001 2024-03-20T09:10:23.555 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.555 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.555 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:10:23.555 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.164819+0000", 2024-03-20T09:10:23.555 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00019093499999999999 2024-03-20T09:10:23.555 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.555 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.555 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 4,10", 2024-03-20T09:10:23.556 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.165013+0000", 2024-03-20T09:10:23.556 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000193845 2024-03-20T09:10:23.556 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.556 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.556 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.556 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.173194+0000", 2024-03-20T09:10:23.556 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0081816349999999996 2024-03-20T09:10:23.556 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.556 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.556 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.556 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.173265+0000", 2024-03-20T09:10:23.556 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0659000000000002e-05 2024-03-20T09:10:23.556 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.556 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.557 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:10:23.557 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.177354+0000", 2024-03-20T09:10:23.557 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040886100000000003 2024-03-20T09:10:23.557 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.557 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.557 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:10:23.557 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.177423+0000", 2024-03-20T09:10:23.557 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9135999999999997e-05 2024-03-20T09:10:23.557 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.557 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.557 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:10:23.557 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.181662+0000", 2024-03-20T09:10:23.557 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0042392280000000003 2024-03-20T09:10:23.558 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.558 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:10:23.558 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.558 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.558 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.558 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:109486 3.7 3:e5e21489:::benchmark_data_smithi122_119671_object9704:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-03-20T09:10:23.558 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:05:19.069650+0000", 2024-03-20T09:10:23.558 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 304.45290032700001, 2024-03-20T09:10:23.558 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1732460900000001, 2024-03-20T09:10:23.558 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:10:23.558 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:10:23.558 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:10:23.558 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:10:23.558 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:10:23.559 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 109486 2024-03-20T09:10:23.559 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.559 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:10:23.559 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.559 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:10:23.559 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.069650+0000", 2024-03-20T09:10:23.559 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.559 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.559 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.559 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:10:23.559 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.069650+0000", 2024-03-20T09:10:23.559 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.559 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.559 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.559 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:10:23.560 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.069651+0000", 2024-03-20T09:10:23.560 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0800000000000003e-07 2024-03-20T09:10:23.560 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.560 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.560 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:10:23.560 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.069655+0000", 2024-03-20T09:10:23.560 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4530000000000004e-06 2024-03-20T09:10:23.560 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.560 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.560 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:10:23.560 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.069666+0000", 2024-03-20T09:10:23.560 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0779e-05 2024-03-20T09:10:23.560 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.560 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.561 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:10:23.561 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.069672+0000", 2024-03-20T09:10:23.561 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9680000000000003e-06 2024-03-20T09:10:23.561 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.561 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.561 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.561 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.069719+0000", 2024-03-20T09:10:23.561 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7648000000000002e-05 2024-03-20T09:10:23.561 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.561 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.561 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:10:23.561 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.069727+0000", 2024-03-20T09:10:23.561 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8220000000000007e-06 2024-03-20T09:10:23.561 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.561 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.562 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.562 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.965435+0000", 2024-03-20T09:10:23.562 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.89570771199999999 2024-03-20T09:10:23.562 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.562 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.562 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:10:23.562 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.965444+0000", 2024-03-20T09:10:23.562 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5709999999999998e-06 2024-03-20T09:10:23.562 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.562 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.562 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.562 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.235231+0000", 2024-03-20T09:10:23.562 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2697870090000001 2024-03-20T09:10:23.563 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.563 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.563 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:10:23.563 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.235347+0000", 2024-03-20T09:10:23.563 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000116824 2024-03-20T09:10:23.563 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.563 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.563 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:10:23.563 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.235451+0000", 2024-03-20T09:10:23.563 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000103905 2024-03-20T09:10:23.563 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.563 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.563 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.563 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.238223+0000", 2024-03-20T09:10:23.564 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027718970000000002 2024-03-20T09:10:23.564 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.564 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.564 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:10:23.564 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.239892+0000", 2024-03-20T09:10:23.564 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0016691900000000001 2024-03-20T09:10:23.564 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.564 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.564 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.564 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.241770+0000", 2024-03-20T09:10:23.564 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0018779910000000001 2024-03-20T09:10:23.564 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.564 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.564 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:10:23.565 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.241836+0000", 2024-03-20T09:10:23.565 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6020999999999997e-05 2024-03-20T09:10:23.565 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.565 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.565 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:10:23.565 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.242896+0000", 2024-03-20T09:10:23.565 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010593919999999999 2024-03-20T09:10:23.565 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.565 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:10:23.565 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.565 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.565 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.565 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:109496 3.e 3:7c4f1f5a:::benchmark_data_smithi122_119671_object9714:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-03-20T09:10:23.565 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:05:19.128008+0000", 2024-03-20T09:10:23.565 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 304.39454220200003, 2024-03-20T09:10:23.566 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0608756690000001, 2024-03-20T09:10:23.566 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:10:23.566 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:10:23.566 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:10:23.566 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:10:23.566 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:10:23.566 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 109496 2024-03-20T09:10:23.566 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.566 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:10:23.566 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.566 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:10:23.566 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.128008+0000", 2024-03-20T09:10:23.566 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.567 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.567 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.567 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:10:23.567 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.128008+0000", 2024-03-20T09:10:23.567 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.567 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.567 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.567 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:10:23.567 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.128009+0000", 2024-03-20T09:10:23.567 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9500000000000001e-07 2024-03-20T09:10:23.567 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.567 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.567 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:10:23.567 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.128013+0000", 2024-03-20T09:10:23.567 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9709999999999998e-06 2024-03-20T09:10:23.568 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.568 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.568 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:10:23.568 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.128020+0000", 2024-03-20T09:10:23.568 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6650000000000003e-06 2024-03-20T09:10:23.568 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.568 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.568 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:10:23.568 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.128027+0000", 2024-03-20T09:10:23.568 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8170000000000001e-06 2024-03-20T09:10:23.568 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.568 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.568 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.568 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.128074+0000", 2024-03-20T09:10:23.569 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6990000000000002e-05 2024-03-20T09:10:23.569 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.569 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.569 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:10:23.569 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.128082+0000", 2024-03-20T09:10:23.569 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0499999999999992e-06 2024-03-20T09:10:23.569 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.569 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.569 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.569 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.173318+0000", 2024-03-20T09:10:23.569 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0452362719999999 2024-03-20T09:10:23.569 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.569 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.569 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:10:23.569 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.173517+0000", 2024-03-20T09:10:23.570 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000198106 2024-03-20T09:10:23.570 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.570 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.570 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 4,10", 2024-03-20T09:10:23.570 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.173709+0000", 2024-03-20T09:10:23.570 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000192315 2024-03-20T09:10:23.570 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.570 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.570 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.570 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.184748+0000", 2024-03-20T09:10:23.570 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011038773999999999 2024-03-20T09:10:23.570 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.570 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.571 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:10:23.571 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.184763+0000", 2024-03-20T09:10:23.571 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5145e-05 2024-03-20T09:10:23.571 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.571 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.571 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.571 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.184850+0000", 2024-03-20T09:10:23.571 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7090999999999997e-05 2024-03-20T09:10:23.571 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.571 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.571 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:10:23.571 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.184907+0000", 2024-03-20T09:10:23.571 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7281000000000003e-05 2024-03-20T09:10:23.571 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.572 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.572 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:10:23.572 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.188883+0000", 2024-03-20T09:10:23.572 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0039762970000000002 2024-03-20T09:10:23.572 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.572 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:10:23.572 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.572 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.572 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.572 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:109497 3.7 3:e0cb8a2c:::benchmark_data_smithi122_119671_object9715:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-03-20T09:10:23.572 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:05:19.132859+0000", 2024-03-20T09:10:23.572 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 304.38969100200001, 2024-03-20T09:10:23.572 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.116806387, 2024-03-20T09:10:23.572 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:10:23.573 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:10:23.573 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:10:23.573 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:10:23.573 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:10:23.573 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 109497 2024-03-20T09:10:23.573 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.573 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:10:23.573 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.573 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:10:23.573 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.132859+0000", 2024-03-20T09:10:23.573 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.573 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.573 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.573 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:10:23.573 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.132859+0000", 2024-03-20T09:10:23.574 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.574 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.574 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.574 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:10:23.574 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.132860+0000", 2024-03-20T09:10:23.574 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0800000000000004e-07 2024-03-20T09:10:23.574 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.574 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.574 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:10:23.574 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.132864+0000", 2024-03-20T09:10:23.574 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0110000000000002e-06 2024-03-20T09:10:23.574 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.574 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.574 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:10:23.575 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.132874+0000", 2024-03-20T09:10:23.575 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0051000000000001e-05 2024-03-20T09:10:23.575 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.575 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.575 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:10:23.575 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.132879+0000", 2024-03-20T09:10:23.575 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4689999999999996e-06 2024-03-20T09:10:23.575 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.575 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.575 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.575 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.132924+0000", 2024-03-20T09:10:23.575 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4607000000000002e-05 2024-03-20T09:10:23.575 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.575 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.576 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:10:23.576 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.132932+0000", 2024-03-20T09:10:23.576 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7649999999999993e-06 2024-03-20T09:10:23.576 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.576 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.576 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.576 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.965465+0000", 2024-03-20T09:10:23.576 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.83253320200000003 2024-03-20T09:10:23.576 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.576 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.576 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:10:23.576 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.965469+0000", 2024-03-20T09:10:23.576 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5569999999999997e-06 2024-03-20T09:10:23.576 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.577 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.577 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.577 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.238252+0000", 2024-03-20T09:10:23.577 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.272782562 2024-03-20T09:10:23.577 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.577 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.577 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:10:23.577 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.238368+0000", 2024-03-20T09:10:23.577 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000116007 2024-03-20T09:10:23.577 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.577 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.577 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:10:23.577 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.238467+0000", 2024-03-20T09:10:23.577 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9407000000000006e-05 2024-03-20T09:10:23.577 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.578 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.578 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.578 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.243009+0000", 2024-03-20T09:10:23.578 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0045413959999999996 2024-03-20T09:10:23.578 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.578 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.578 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.578 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.243053+0000", 2024-03-20T09:10:23.578 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4314000000000003e-05 2024-03-20T09:10:23.578 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.578 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.578 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:10:23.578 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.248544+0000", 2024-03-20T09:10:23.578 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0054908370000000001 2024-03-20T09:10:23.579 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.579 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.579 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:10:23.579 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.248595+0000", 2024-03-20T09:10:23.579 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1396000000000001e-05 2024-03-20T09:10:23.579 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.579 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.579 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:10:23.579 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.249665+0000", 2024-03-20T09:10:23.579 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001070098 2024-03-20T09:10:23.579 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.579 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:10:23.579 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.579 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.579 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.580 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:109500 3.e 3:77c65ac3:::benchmark_data_smithi122_119671_object9718:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-03-20T09:10:23.580 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:05:19.222091+0000", 2024-03-20T09:10:23.580 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 304.30045856100003, 2024-03-20T09:10:23.580 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.96992302600000002, 2024-03-20T09:10:23.580 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:10:23.580 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:10:23.580 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:10:23.580 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:10:23.580 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:10:23.580 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 109500 2024-03-20T09:10:23.580 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.580 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:10:23.580 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.580 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:10:23.581 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.222091+0000", 2024-03-20T09:10:23.581 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.581 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.581 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.581 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:10:23.581 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.222091+0000", 2024-03-20T09:10:23.581 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.581 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.581 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.581 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:10:23.581 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.222092+0000", 2024-03-20T09:10:23.581 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9999999999999995e-07 2024-03-20T09:10:23.581 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.581 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.582 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:10:23.582 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.222097+0000", 2024-03-20T09:10:23.582 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6099999999999999e-06 2024-03-20T09:10:23.582 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.582 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.582 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:10:23.582 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.222110+0000", 2024-03-20T09:10:23.582 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2886e-05 2024-03-20T09:10:23.582 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.582 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.582 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:10:23.582 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.222116+0000", 2024-03-20T09:10:23.582 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8270000000000003e-06 2024-03-20T09:10:23.582 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.583 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.583 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.583 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.222178+0000", 2024-03-20T09:10:23.583 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1892999999999998e-05 2024-03-20T09:10:23.583 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.583 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.583 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:10:23.583 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.222191+0000", 2024-03-20T09:10:23.583 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3135999999999999e-05 2024-03-20T09:10:23.583 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.583 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.583 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.583 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.181713+0000", 2024-03-20T09:10:23.584 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95952252599999999 2024-03-20T09:10:23.584 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.584 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.584 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:10:23.584 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.181826+0000", 2024-03-20T09:10:23.584 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011270299999999999 2024-03-20T09:10:23.584 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.584 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.584 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 4,10", 2024-03-20T09:10:23.584 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.181928+0000", 2024-03-20T09:10:23.584 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000101811 2024-03-20T09:10:23.584 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.584 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.584 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.585 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.188936+0000", 2024-03-20T09:10:23.585 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0070081040000000002 2024-03-20T09:10:23.585 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.585 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.585 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.585 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.188973+0000", 2024-03-20T09:10:23.585 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6600999999999997e-05 2024-03-20T09:10:23.585 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.585 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.585 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:10:23.585 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.188984+0000", 2024-03-20T09:10:23.585 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1765e-05 2024-03-20T09:10:23.585 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.585 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.586 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:10:23.586 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.189023+0000", 2024-03-20T09:10:23.586 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8507000000000002e-05 2024-03-20T09:10:23.586 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.586 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.586 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:10:23.586 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.192014+0000", 2024-03-20T09:10:23.586 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002991657 2024-03-20T09:10:23.586 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.586 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:10:23.586 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.586 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.586 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.587 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:109502 3.7 3:e43407e3:::benchmark_data_smithi122_119671_object9720:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-03-20T09:10:23.587 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:05:19.242690+0000", 2024-03-20T09:10:23.587 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 304.27985966300002, 2024-03-20T09:10:23.587 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0126284139999999, 2024-03-20T09:10:23.587 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:10:23.587 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:10:23.587 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:10:23.587 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:10:23.587 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:10:23.587 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 109502 2024-03-20T09:10:23.587 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.587 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:10:23.588 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.588 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:10:23.588 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.242690+0000", 2024-03-20T09:10:23.588 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.588 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.588 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.588 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:10:23.588 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.242690+0000", 2024-03-20T09:10:23.588 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.588 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.588 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.588 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:10:23.588 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.242691+0000", 2024-03-20T09:10:23.589 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.23e-07 2024-03-20T09:10:23.589 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.589 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.589 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:10:23.589 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.242695+0000", 2024-03-20T09:10:23.589 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8210000000000003e-06 2024-03-20T09:10:23.589 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.589 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.589 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:10:23.589 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.242706+0000", 2024-03-20T09:10:23.589 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0698e-05 2024-03-20T09:10:23.589 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.589 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.590 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:10:23.590 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.242711+0000", 2024-03-20T09:10:23.590 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6790000000000002e-06 2024-03-20T09:10:23.590 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.590 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.590 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.590 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.242768+0000", 2024-03-20T09:10:23.590 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6591000000000001e-05 2024-03-20T09:10:23.590 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.590 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.590 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:10:23.590 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.242779+0000", 2024-03-20T09:10:23.590 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1226999999999999e-05 2024-03-20T09:10:23.591 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.591 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.591 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.591 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.965484+0000", 2024-03-20T09:10:23.591 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.72270470499999995 2024-03-20T09:10:23.591 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.591 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.591 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:10:23.591 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.965488+0000", 2024-03-20T09:10:23.591 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2220000000000004e-06 2024-03-20T09:10:23.591 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.591 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.591 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.591 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.239928+0000", 2024-03-20T09:10:23.592 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.274439909 2024-03-20T09:10:23.592 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.592 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.592 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:10:23.592 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.240095+0000", 2024-03-20T09:10:23.592 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016686500000000001 2024-03-20T09:10:23.592 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.592 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.592 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:10:23.592 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.240239+0000", 2024-03-20T09:10:23.592 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000143886 2024-03-20T09:10:23.592 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.592 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.592 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.593 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.243093+0000", 2024-03-20T09:10:23.593 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0028540810000000001 2024-03-20T09:10:23.593 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.593 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.593 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:10:23.593 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.249673+0000", 2024-03-20T09:10:23.593 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0065800520000000003 2024-03-20T09:10:23.593 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.593 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.593 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.593 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.254272+0000", 2024-03-20T09:10:23.593 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.004599087 2024-03-20T09:10:23.593 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.593 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.594 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:10:23.594 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.254318+0000", 2024-03-20T09:10:23.594 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5946999999999999e-05 2024-03-20T09:10:23.594 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.594 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.594 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:10:23.594 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.255319+0000", 2024-03-20T09:10:23.594 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010009210000000001 2024-03-20T09:10:23.594 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.594 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:10:23.594 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.594 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.594 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.595 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:109505 3.7 3:e7605f1b:::benchmark_data_smithi122_119671_object9723:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-03-20T09:10:23.595 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:05:19.307126+0000", 2024-03-20T09:10:23.595 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 304.21542393499999, 2024-03-20T09:10:23.595 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.948903968, 2024-03-20T09:10:23.595 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:10:23.595 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:10:23.595 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:10:23.595 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:10:23.595 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:10:23.595 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 109505 2024-03-20T09:10:23.595 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.595 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:10:23.595 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.595 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:10:23.596 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.307126+0000", 2024-03-20T09:10:23.596 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.596 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.596 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.596 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:10:23.596 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.307126+0000", 2024-03-20T09:10:23.596 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.596 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.596 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.596 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:10:23.596 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.307127+0000", 2024-03-20T09:10:23.596 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2900000000000002e-07 2024-03-20T09:10:23.596 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.596 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.597 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:10:23.597 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.307131+0000", 2024-03-20T09:10:23.597 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2220000000000004e-06 2024-03-20T09:10:23.597 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.597 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.597 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:10:23.597 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.307143+0000", 2024-03-20T09:10:23.597 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1657e-05 2024-03-20T09:10:23.597 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.597 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.597 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:10:23.597 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.307149+0000", 2024-03-20T09:10:23.597 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7039999999999999e-06 2024-03-20T09:10:23.597 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.597 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.598 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.598 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.307197+0000", 2024-03-20T09:10:23.598 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8875000000000002e-05 2024-03-20T09:10:23.598 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.598 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.598 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:10:23.598 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.307205+0000", 2024-03-20T09:10:23.598 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6420000000000007e-06 2024-03-20T09:10:23.598 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.598 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.598 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.598 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.965501+0000", 2024-03-20T09:10:23.598 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.65829631099999997 2024-03-20T09:10:23.599 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.599 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.599 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:10:23.599 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.965506+0000", 2024-03-20T09:10:23.599 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8160000000000001e-06 2024-03-20T09:10:23.599 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.599 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.599 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.599 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.243120+0000", 2024-03-20T09:10:23.599 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.277613938 2024-03-20T09:10:23.599 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.599 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.599 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:10:23.599 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.243259+0000", 2024-03-20T09:10:23.599 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013856 2024-03-20T09:10:23.600 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.600 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.600 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:10:23.600 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.243400+0000", 2024-03-20T09:10:23.600 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000141695 2024-03-20T09:10:23.600 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.600 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.600 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.600 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.249730+0000", 2024-03-20T09:10:23.600 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0063300129999999998 2024-03-20T09:10:23.600 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.600 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.600 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:10:23.600 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.249788+0000", 2024-03-20T09:10:23.601 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7887000000000003e-05 2024-03-20T09:10:23.601 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.601 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.601 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.601 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.255363+0000", 2024-03-20T09:10:23.601 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0055745650000000001 2024-03-20T09:10:23.601 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.601 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.601 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:10:23.601 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.255401+0000", 2024-03-20T09:10:23.601 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8290000000000001e-05 2024-03-20T09:10:23.601 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.601 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.601 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:10:23.602 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.256030+0000", 2024-03-20T09:10:23.602 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00062886400000000003 2024-03-20T09:10:23.602 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.602 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:10:23.602 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.602 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.602 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.602 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:109532 3.7 3:e602d675:::benchmark_data_smithi122_119671_object9750:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-03-20T09:10:23.602 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:05:20.189263+0000", 2024-03-20T09:10:23.602 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 303.333287482, 2024-03-20T09:10:23.602 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0674772130000001, 2024-03-20T09:10:23.602 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:10:23.602 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:10:23.602 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:10:23.603 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:10:23.603 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:10:23.603 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 109532 2024-03-20T09:10:23.603 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.603 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:10:23.603 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.603 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:10:23.603 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.189263+0000", 2024-03-20T09:10:23.603 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.603 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.603 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.603 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:10:23.603 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.189263+0000", 2024-03-20T09:10:23.603 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.604 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.604 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.604 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:10:23.604 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.189264+0000", 2024-03-20T09:10:23.604 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4899999999999999e-06 2024-03-20T09:10:23.604 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.604 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.604 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:10:23.604 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.189270+0000", 2024-03-20T09:10:23.604 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7339999999999996e-06 2024-03-20T09:10:23.604 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.604 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.604 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:10:23.604 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.189281+0000", 2024-03-20T09:10:23.605 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.083e-05 2024-03-20T09:10:23.605 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.605 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.605 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:10:23.605 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.189299+0000", 2024-03-20T09:10:23.605 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7915e-05 2024-03-20T09:10:23.605 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.605 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.605 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.605 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.189358+0000", 2024-03-20T09:10:23.605 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9024999999999998e-05 2024-03-20T09:10:23.605 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.605 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.605 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:10:23.606 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.189365+0000", 2024-03-20T09:10:23.606 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8469999999999996e-06 2024-03-20T09:10:23.606 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.606 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.606 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.606 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.245076+0000", 2024-03-20T09:10:23.606 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.055710345 2024-03-20T09:10:23.606 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.606 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.606 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:10:23.606 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.245215+0000", 2024-03-20T09:10:23.606 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000139128 2024-03-20T09:10:23.606 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.606 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.607 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:10:23.607 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.245350+0000", 2024-03-20T09:10:23.607 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013474900000000001 2024-03-20T09:10:23.607 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.607 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.607 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.607 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.249778+0000", 2024-03-20T09:10:23.607 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0044287340000000001 2024-03-20T09:10:23.607 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.607 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.607 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:10:23.607 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.249795+0000", 2024-03-20T09:10:23.607 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6307000000000002e-05 2024-03-20T09:10:23.607 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.608 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.608 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.608 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.256073+0000", 2024-03-20T09:10:23.608 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0062784099999999999 2024-03-20T09:10:23.608 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.608 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.608 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:10:23.608 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.256134+0000", 2024-03-20T09:10:23.608 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0776999999999998e-05 2024-03-20T09:10:23.608 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.608 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.608 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:10:23.608 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.256740+0000", 2024-03-20T09:10:23.608 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00060592199999999995 2024-03-20T09:10:23.609 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.609 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:10:23.609 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.609 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.609 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.609 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:116565 3.e 3.f438448e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-03-20T09:10:23.609 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:06:26.946897+0000", 2024-03-20T09:10:23.609 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 236.575652762, 2024-03-20T09:10:23.609 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.98363714800000002, 2024-03-20T09:10:23.609 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:10:23.609 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:10:23.609 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:10:23.609 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:10:23.609 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:10:23.609 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 116565 2024-03-20T09:10:23.610 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.610 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:10:23.610 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.610 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:10:23.610 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:26.946897+0000", 2024-03-20T09:10:23.610 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.610 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.610 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.610 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:10:23.610 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:26.946897+0000", 2024-03-20T09:10:23.610 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.610 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.610 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.611 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:10:23.611 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:26.946900+0000", 2024-03-20T09:10:23.611 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3190000000000002e-06 2024-03-20T09:10:23.611 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.611 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.611 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:10:23.611 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:26.946912+0000", 2024-03-20T09:10:23.611 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1984000000000001e-05 2024-03-20T09:10:23.611 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.611 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.611 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:10:23.611 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:26.946926+0000", 2024-03-20T09:10:23.611 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.411e-05 2024-03-20T09:10:23.611 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.612 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.612 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:10:23.612 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:26.946935+0000", 2024-03-20T09:10:23.612 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5060000000000001e-06 2024-03-20T09:10:23.612 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.612 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.612 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.612 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:26.947009+0000", 2024-03-20T09:10:23.612 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3498999999999995e-05 2024-03-20T09:10:23.612 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.612 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.612 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:10:23.612 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:26.947017+0000", 2024-03-20T09:10:23.612 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2029999999999992e-06 2024-03-20T09:10:23.613 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.613 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.613 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.613 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:27.930521+0000", 2024-03-20T09:10:23.613 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.98350451000000005 2024-03-20T09:10:23.613 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.613 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.613 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:10:23.613 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:27.930534+0000", 2024-03-20T09:10:23.613 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3016999999999999e-05 2024-03-20T09:10:23.613 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.613 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:10:23.613 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.613 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.614 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.614 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:120302 3.f 3:f441e459:::benchmark_data_smithi122_119671_object20520:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-03-20T09:10:23.614 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:06:58.141873+0000", 2024-03-20T09:10:23.614 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 205.38067707299999, 2024-03-20T09:10:23.614 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.89588768900000004, 2024-03-20T09:10:23.614 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:10:23.614 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:10:23.614 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:10:23.614 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:10:23.614 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:10:23.614 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 120302 2024-03-20T09:10:23.614 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.614 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:10:23.615 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.615 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:10:23.615 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.141873+0000", 2024-03-20T09:10:23.615 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.615 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.615 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.615 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:10:23.615 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.141873+0000", 2024-03-20T09:10:23.615 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.615 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.615 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.615 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:10:23.615 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.141874+0000", 2024-03-20T09:10:23.615 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3319999999999999e-06 2024-03-20T09:10:23.616 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.616 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.616 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:10:23.616 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.141880+0000", 2024-03-20T09:10:23.616 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3619999999999997e-06 2024-03-20T09:10:23.616 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.616 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.616 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:10:23.616 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.141892+0000", 2024-03-20T09:10:23.616 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2704000000000001e-05 2024-03-20T09:10:23.616 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.616 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.616 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:10:23.616 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.141899+0000", 2024-03-20T09:10:23.617 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6919999999999996e-06 2024-03-20T09:10:23.617 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.617 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.617 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.617 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.141969+0000", 2024-03-20T09:10:23.617 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9597000000000005e-05 2024-03-20T09:10:23.617 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.617 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.617 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:10:23.617 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.141981+0000", 2024-03-20T09:10:23.617 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2658e-05 2024-03-20T09:10:23.617 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.617 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.618 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.618 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.033384+0000", 2024-03-20T09:10:23.618 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.891403116 2024-03-20T09:10:23.618 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.618 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.618 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:10:23.618 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.033505+0000", 2024-03-20T09:10:23.618 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000120519 2024-03-20T09:10:23.618 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.618 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.618 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8,10", 2024-03-20T09:10:23.618 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.033580+0000", 2024-03-20T09:10:23.618 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4938999999999995e-05 2024-03-20T09:10:23.619 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.619 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.619 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.619 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.037614+0000", 2024-03-20T09:10:23.619 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040340580000000001 2024-03-20T09:10:23.619 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.619 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.619 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:10:23.619 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.037629+0000", 2024-03-20T09:10:23.619 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4844999999999999e-05 2024-03-20T09:10:23.619 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.619 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.619 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.620 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.037673+0000", 2024-03-20T09:10:23.620 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4428000000000002e-05 2024-03-20T09:10:23.620 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.620 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.620 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:10:23.620 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.037720+0000", 2024-03-20T09:10:23.620 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7015999999999998e-05 2024-03-20T09:10:23.620 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.620 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.620 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:10:23.620 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.037761+0000", 2024-03-20T09:10:23.620 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0423000000000002e-05 2024-03-20T09:10:23.620 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.620 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:10:23.620 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.621 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.621 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.621 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:120312 3.f 3:f1c830a7:::benchmark_data_smithi122_119671_object20530:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-03-20T09:10:23.621 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:06:58.253819+0000", 2024-03-20T09:10:23.621 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 205.268730893, 2024-03-20T09:10:23.621 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.79499205799999995, 2024-03-20T09:10:23.621 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:10:23.621 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:10:23.621 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:10:23.621 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:10:23.621 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:10:23.621 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 120312 2024-03-20T09:10:23.621 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.621 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:10:23.621 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.622 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:10:23.622 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.253819+0000", 2024-03-20T09:10:23.622 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.622 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.622 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.622 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:10:23.622 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.253819+0000", 2024-03-20T09:10:23.622 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.622 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.622 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.622 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:10:23.622 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.253821+0000", 2024-03-20T09:10:23.622 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8989999999999999e-06 2024-03-20T09:10:23.623 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.623 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.623 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:10:23.623 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.253829+0000", 2024-03-20T09:10:23.623 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5610000000000001e-06 2024-03-20T09:10:23.623 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.623 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.623 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:10:23.623 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.253847+0000", 2024-03-20T09:10:23.623 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8601e-05 2024-03-20T09:10:23.623 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.623 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.623 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:10:23.623 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.253856+0000", 2024-03-20T09:10:23.624 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7490000000000004e-06 2024-03-20T09:10:23.624 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.624 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.624 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.624 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.253923+0000", 2024-03-20T09:10:23.624 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6773000000000006e-05 2024-03-20T09:10:23.624 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.624 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.624 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:10:23.624 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.253933+0000", 2024-03-20T09:10:23.624 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0088000000000001e-05 2024-03-20T09:10:23.624 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.624 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.624 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.624 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.037799+0000", 2024-03-20T09:10:23.625 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.78386641400000001 2024-03-20T09:10:23.625 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.625 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.625 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:10:23.625 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.037930+0000", 2024-03-20T09:10:23.625 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013070399999999999 2024-03-20T09:10:23.625 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.625 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.625 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8,10", 2024-03-20T09:10:23.625 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.038055+0000", 2024-03-20T09:10:23.625 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0001248 2024-03-20T09:10:23.625 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.625 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.625 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.626 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.048493+0000", 2024-03-20T09:10:23.626 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010438114 2024-03-20T09:10:23.626 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.626 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.626 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.626 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.048657+0000", 2024-03-20T09:10:23.626 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016442099999999999 2024-03-20T09:10:23.626 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.626 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.626 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:10:23.626 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.048726+0000", 2024-03-20T09:10:23.626 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8707999999999993e-05 2024-03-20T09:10:23.626 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.627 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.627 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:10:23.627 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.048777+0000", 2024-03-20T09:10:23.627 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0837999999999997e-05 2024-03-20T09:10:23.627 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.627 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.627 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:10:23.627 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.048811+0000", 2024-03-20T09:10:23.627 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4387999999999997e-05 2024-03-20T09:10:23.627 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.627 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:10:23.627 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.627 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.627 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.628 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:120314 3.f 3:f5df3aa9:::benchmark_data_smithi122_119671_object20532:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-03-20T09:10:23.628 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:06:58.276268+0000", 2024-03-20T09:10:23.628 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 205.246282302, 2024-03-20T09:10:23.628 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.77584452000000004, 2024-03-20T09:10:23.628 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:10:23.628 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:10:23.628 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:10:23.628 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:10:23.628 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:10:23.628 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 120314 2024-03-20T09:10:23.628 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.628 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:10:23.628 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.628 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:10:23.629 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.276268+0000", 2024-03-20T09:10:23.629 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.629 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.629 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.629 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:10:23.629 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.276268+0000", 2024-03-20T09:10:23.629 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.629 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.629 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.629 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:10:23.629 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.276269+0000", 2024-03-20T09:10:23.629 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.156e-06 2024-03-20T09:10:23.629 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.629 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.629 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:10:23.629 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.276276+0000", 2024-03-20T09:10:23.630 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8539999999999999e-06 2024-03-20T09:10:23.630 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.630 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.630 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:10:23.630 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.276296+0000", 2024-03-20T09:10:23.630 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9981000000000001e-05 2024-03-20T09:10:23.630 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.630 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.630 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:10:23.630 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.276305+0000", 2024-03-20T09:10:23.630 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1759999999999999e-06 2024-03-20T09:10:23.630 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.630 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.630 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.630 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.276383+0000", 2024-03-20T09:10:23.631 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8186e-05 2024-03-20T09:10:23.631 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.631 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.631 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:10:23.631 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.276397+0000", 2024-03-20T09:10:23.631 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3652000000000001e-05 2024-03-20T09:10:23.631 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.631 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.631 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.631 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.040897+0000", 2024-03-20T09:10:23.631 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.76450070299999995 2024-03-20T09:10:23.631 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.631 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.631 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:10:23.631 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.041007+0000", 2024-03-20T09:10:23.631 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000109875 2024-03-20T09:10:23.632 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.632 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.632 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8,10", 2024-03-20T09:10:23.632 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.041117+0000", 2024-03-20T09:10:23.632 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000109988 2024-03-20T09:10:23.632 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.632 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.632 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.632 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.048537+0000", 2024-03-20T09:10:23.632 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0074194059999999999 2024-03-20T09:10:23.632 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.632 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.632 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:10:23.632 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.048821+0000", 2024-03-20T09:10:23.632 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00028427900000000002 2024-03-20T09:10:23.633 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.633 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.633 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.633 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.052033+0000", 2024-03-20T09:10:23.633 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003212421 2024-03-20T09:10:23.633 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.633 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.633 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:10:23.633 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.052077+0000", 2024-03-20T09:10:23.633 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3665000000000003e-05 2024-03-20T09:10:23.633 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.633 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.633 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:10:23.634 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.052112+0000", 2024-03-20T09:10:23.634 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5178000000000001e-05 2024-03-20T09:10:23.634 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.634 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:10:23.634 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.634 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.634 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.634 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:120319 3.f 3:fe67f674:::benchmark_data_smithi122_119671_object20537:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-03-20T09:10:23.634 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:06:58.307021+0000", 2024-03-20T09:10:23.634 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 205.21552860200001, 2024-03-20T09:10:23.634 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.74520134299999996, 2024-03-20T09:10:23.634 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:10:23.634 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:10:23.634 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:10:23.635 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:10:23.635 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:10:23.635 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 120319 2024-03-20T09:10:23.635 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.635 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:10:23.635 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.635 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:10:23.635 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.307021+0000", 2024-03-20T09:10:23.635 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.635 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.635 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.635 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:10:23.635 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.307021+0000", 2024-03-20T09:10:23.635 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.636 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.636 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.636 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:10:23.636 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.307022+0000", 2024-03-20T09:10:23.636 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.051e-06 2024-03-20T09:10:23.636 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.636 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.636 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:10:23.636 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.307028+0000", 2024-03-20T09:10:23.636 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6180000000000003e-06 2024-03-20T09:10:23.636 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.636 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.636 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:10:23.636 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.307040+0000", 2024-03-20T09:10:23.637 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.199e-05 2024-03-20T09:10:23.637 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.637 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.637 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:10:23.637 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.307047+0000", 2024-03-20T09:10:23.637 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4269999999999996e-06 2024-03-20T09:10:23.637 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.637 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.637 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.637 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.307101+0000", 2024-03-20T09:10:23.637 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4954999999999998e-05 2024-03-20T09:10:23.637 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.637 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.637 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:10:23.637 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.307110+0000", 2024-03-20T09:10:23.638 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3049999999999993e-06 2024-03-20T09:10:23.638 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.638 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.638 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.638 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.044684+0000", 2024-03-20T09:10:23.638 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.73757395699999995 2024-03-20T09:10:23.638 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.638 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.638 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:10:23.638 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.044796+0000", 2024-03-20T09:10:23.638 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000112285 2024-03-20T09:10:23.638 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.638 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.639 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8,10", 2024-03-20T09:10:23.639 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.044865+0000", 2024-03-20T09:10:23.639 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8495000000000001e-05 2024-03-20T09:10:23.639 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.639 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.639 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.639 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.048714+0000", 2024-03-20T09:10:23.639 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0038493149999999999 2024-03-20T09:10:23.639 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.639 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.639 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:10:23.639 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.048825+0000", 2024-03-20T09:10:23.639 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000111082 2024-03-20T09:10:23.639 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.639 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.640 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.640 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.052157+0000", 2024-03-20T09:10:23.640 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003331615 2024-03-20T09:10:23.640 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.640 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.640 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:10:23.640 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.052187+0000", 2024-03-20T09:10:23.640 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0076000000000001e-05 2024-03-20T09:10:23.640 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.640 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.640 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:10:23.640 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.052223+0000", 2024-03-20T09:10:23.640 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6171999999999999e-05 2024-03-20T09:10:23.640 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.640 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:10:23.641 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.641 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.641 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.641 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:127398 3.15 3:ade2fd22:::benchmark_data_smithi122_119671_object27616:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-03-20T09:10:23.641 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:07:48.982204+0000", 2024-03-20T09:10:23.641 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 154.54034571700001, 2024-03-20T09:10:23.641 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 10.622496298, 2024-03-20T09:10:23.641 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:10:23.641 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:10:23.641 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:10:23.641 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:10:23.641 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:10:23.641 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 127398 2024-03-20T09:10:23.642 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.642 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:10:23.642 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.642 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:10:23.642 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:48.982204+0000", 2024-03-20T09:10:23.642 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.642 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.642 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.642 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:10:23.642 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:48.982204+0000", 2024-03-20T09:10:23.642 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.642 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.642 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.642 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:10:23.643 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:48.982205+0000", 2024-03-20T09:10:23.643 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6700000000000003e-07 2024-03-20T09:10:23.643 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.643 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.643 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:10:23.643 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:48.982209+0000", 2024-03-20T09:10:23.643 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0629999999999999e-06 2024-03-20T09:10:23.643 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.643 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.643 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:10:23.643 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:48.982218+0000", 2024-03-20T09:10:23.643 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9190000000000005e-06 2024-03-20T09:10:23.643 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.643 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.643 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:10:23.643 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:48.982226+0000", 2024-03-20T09:10:23.644 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5830000000000001e-06 2024-03-20T09:10:23.644 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.644 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.644 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.644 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:48.982274+0000", 2024-03-20T09:10:23.644 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8028999999999997e-05 2024-03-20T09:10:23.644 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.644 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.644 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:10:23.644 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:48.982280+0000", 2024-03-20T09:10:23.644 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6359999999999997e-06 2024-03-20T09:10:23.644 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.644 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.644 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.644 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.568536+0000", 2024-03-20T09:10:23.645 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.58625587300000004 2024-03-20T09:10:23.645 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.645 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.645 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:10:23.645 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.568541+0000", 2024-03-20T09:10:23.645 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0660000000000003e-06 2024-03-20T09:10:23.645 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.645 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.645 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.645 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.540389+0000", 2024-03-20T09:10:23.645 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9718477429999997 2024-03-20T09:10:23.645 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.645 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.645 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:10:23.645 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.544144+0000", 2024-03-20T09:10:23.645 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0037547710000000001 2024-03-20T09:10:23.646 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.646 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.646 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,6,8", 2024-03-20T09:10:23.646 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.549569+0000", 2024-03-20T09:10:23.646 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.005425249 2024-03-20T09:10:23.646 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.646 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.646 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:10:23.646 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.555980+0000", 2024-03-20T09:10:23.646 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0064105960000000002 2024-03-20T09:10:23.646 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.646 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.646 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.646 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.574350+0000", 2024-03-20T09:10:23.647 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018370250000000001 2024-03-20T09:10:23.647 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.647 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.647 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.647 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.575856+0000", 2024-03-20T09:10:23.647 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001506028 2024-03-20T09:10:23.647 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.647 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.647 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.647 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.593136+0000", 2024-03-20T09:10:23.647 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017279952000000001 2024-03-20T09:10:23.647 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.647 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.647 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:10:23.648 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.595989+0000", 2024-03-20T09:10:23.648 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0028537409999999999 2024-03-20T09:10:23.648 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.648 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.648 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:10:23.648 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.604701+0000", 2024-03-20T09:10:23.648 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0087111319999999999 2024-03-20T09:10:23.648 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.648 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:10:23.648 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.648 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.648 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.648 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:127402 3.7 3:e4ec878c:::benchmark_data_smithi122_119671_object27620:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-03-20T09:10:23.648 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:07:49.102069+0000", 2024-03-20T09:10:23.648 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 154.42048136299999, 2024-03-20T09:10:23.648 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.660559699, 2024-03-20T09:10:23.649 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:10:23.649 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:10:23.649 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:10:23.649 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:10:23.649 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:10:23.649 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 127402 2024-03-20T09:10:23.649 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.649 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:10:23.649 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.649 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:10:23.649 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.102069+0000", 2024-03-20T09:10:23.649 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.649 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.649 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.649 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:10:23.650 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.102069+0000", 2024-03-20T09:10:23.650 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.650 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.650 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.650 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:10:23.650 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.102070+0000", 2024-03-20T09:10:23.650 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6160000000000001e-06 2024-03-20T09:10:23.650 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.650 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.650 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:10:23.650 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.102076+0000", 2024-03-20T09:10:23.650 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4600000000000002e-06 2024-03-20T09:10:23.650 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.650 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.650 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:10:23.651 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.102089+0000", 2024-03-20T09:10:23.651 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3628999999999999e-05 2024-03-20T09:10:23.651 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.651 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.651 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:10:23.651 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.102095+0000", 2024-03-20T09:10:23.651 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1059999999999996e-06 2024-03-20T09:10:23.651 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.651 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.651 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.651 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.102137+0000", 2024-03-20T09:10:23.651 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1773000000000001e-05 2024-03-20T09:10:23.651 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.651 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.651 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:10:23.652 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.102145+0000", 2024-03-20T09:10:23.652 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2699999999999999e-06 2024-03-20T09:10:23.652 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.652 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.652 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.652 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.567880+0000", 2024-03-20T09:10:23.652 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.465735385 2024-03-20T09:10:23.652 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.652 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.652 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:10:23.652 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.567893+0000", 2024-03-20T09:10:23.652 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3558e-05 2024-03-20T09:10:23.652 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.652 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.652 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.652 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.746011+0000", 2024-03-20T09:10:23.653 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1781172019999999 2024-03-20T09:10:23.653 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.653 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.653 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:10:23.653 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.746180+0000", 2024-03-20T09:10:23.653 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016975000000000001 2024-03-20T09:10:23.653 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.653 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.653 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,1", 2024-03-20T09:10:23.653 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.746296+0000", 2024-03-20T09:10:23.653 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000115955 2024-03-20T09:10:23.653 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.653 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.653 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:10:23.653 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.754830+0000", 2024-03-20T09:10:23.654 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0085332069999999993 2024-03-20T09:10:23.654 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.654 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.654 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.654 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.754897+0000", 2024-03-20T09:10:23.654 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7121999999999999e-05 2024-03-20T09:10:23.654 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.654 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.654 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.654 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.760396+0000", 2024-03-20T09:10:23.654 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0054992940000000001 2024-03-20T09:10:23.654 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.654 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.654 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:10:23.654 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.760474+0000", 2024-03-20T09:10:23.655 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7558999999999994e-05 2024-03-20T09:10:23.655 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.655 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.655 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:10:23.655 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.762628+0000", 2024-03-20T09:10:23.655 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0021548130000000002 2024-03-20T09:10:23.655 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.655 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:10:23.655 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.655 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.655 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.655 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:127403 3.8 3:1343bae0:::benchmark_data_smithi122_119671_object27621:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-03-20T09:10:23.655 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:07:49.118016+0000", 2024-03-20T09:10:23.655 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 154.404534439, 2024-03-20T09:10:23.655 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4850547279999999, 2024-03-20T09:10:23.656 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:10:23.656 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:10:23.656 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:10:23.656 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:10:23.656 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:10:23.656 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 127403 2024-03-20T09:10:23.656 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.656 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:10:23.656 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.656 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:10:23.656 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.118016+0000", 2024-03-20T09:10:23.656 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.656 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.656 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.657 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:10:23.657 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.118016+0000", 2024-03-20T09:10:23.657 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.657 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.657 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.657 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:10:23.657 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.118017+0000", 2024-03-20T09:10:23.657 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1999999999999999e-06 2024-03-20T09:10:23.657 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.657 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.657 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:10:23.657 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.118023+0000", 2024-03-20T09:10:23.657 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0039999999999996e-06 2024-03-20T09:10:23.657 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.658 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.658 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:10:23.658 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.118036+0000", 2024-03-20T09:10:23.658 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3314e-05 2024-03-20T09:10:23.658 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.658 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.658 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:10:23.658 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.118044+0000", 2024-03-20T09:10:23.658 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8930000000000005e-06 2024-03-20T09:10:23.658 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.658 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.658 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.658 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.118104+0000", 2024-03-20T09:10:23.658 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0387999999999998e-05 2024-03-20T09:10:23.659 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.659 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.659 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:10:23.659 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.118112+0000", 2024-03-20T09:10:23.659 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5279999999999998e-06 2024-03-20T09:10:23.659 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.659 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.659 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.659 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.565999+0000", 2024-03-20T09:10:23.659 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.447887533 2024-03-20T09:10:23.659 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.659 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.659 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:10:23.659 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.566011+0000", 2024-03-20T09:10:23.660 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1258e-05 2024-03-20T09:10:23.660 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.660 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.660 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.660 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.586244+0000", 2024-03-20T09:10:23.660 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.020233699 2024-03-20T09:10:23.660 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.660 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.660 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:10:23.660 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.586399+0000", 2024-03-20T09:10:23.660 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015447400000000001 2024-03-20T09:10:23.660 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.660 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.660 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,4", 2024-03-20T09:10:23.661 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.586546+0000", 2024-03-20T09:10:23.661 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000146627 2024-03-20T09:10:23.661 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.661 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.661 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:10:23.661 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.588915+0000", 2024-03-20T09:10:23.661 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0023696979999999999 2024-03-20T09:10:23.661 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.661 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.661 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.661 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.595655+0000", 2024-03-20T09:10:23.661 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0067396660000000001 2024-03-20T09:10:23.661 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.661 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.661 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.662 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.602897+0000", 2024-03-20T09:10:23.662 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0072418430000000004 2024-03-20T09:10:23.662 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.662 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.662 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:10:23.662 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.602961+0000", 2024-03-20T09:10:23.662 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3827999999999999e-05 2024-03-20T09:10:23.662 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.662 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.662 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:10:23.662 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.603070+0000", 2024-03-20T09:10:23.662 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000109775 2024-03-20T09:10:23.662 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.662 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:10:23.662 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.663 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.663 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.663 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:127406 3.15 3:afb25ec3:::benchmark_data_smithi122_119671_object27624:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-03-20T09:10:23.663 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:07:49.352179+0000", 2024-03-20T09:10:23.663 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 154.17037147799999, 2024-03-20T09:10:23.663 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 10.268763275, 2024-03-20T09:10:23.663 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:10:23.663 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:10:23.663 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:10:23.663 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:10:23.663 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:10:23.663 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 127406 2024-03-20T09:10:23.663 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.663 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:10:23.663 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.664 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:10:23.664 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.352179+0000", 2024-03-20T09:10:23.664 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.664 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.664 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.664 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:10:23.664 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.352179+0000", 2024-03-20T09:10:23.664 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.664 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.664 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.664 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:10:23.664 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.352180+0000", 2024-03-20T09:10:23.664 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.217e-06 2024-03-20T09:10:23.664 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.664 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.665 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:10:23.665 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.352185+0000", 2024-03-20T09:10:23.665 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4049999999999999e-06 2024-03-20T09:10:23.665 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.665 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.665 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:10:23.665 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.352197+0000", 2024-03-20T09:10:23.665 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2018000000000001e-05 2024-03-20T09:10:23.665 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.665 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.665 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:10:23.665 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.352203+0000", 2024-03-20T09:10:23.665 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8989999999999998e-06 2024-03-20T09:10:23.665 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.665 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.666 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.666 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.352248+0000", 2024-03-20T09:10:23.666 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5330000000000001e-05 2024-03-20T09:10:23.666 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.666 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.666 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:10:23.666 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.352255+0000", 2024-03-20T09:10:23.666 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8070000000000002e-06 2024-03-20T09:10:23.666 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.666 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.666 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.666 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.568560+0000", 2024-03-20T09:10:23.666 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.216305003 2024-03-20T09:10:23.667 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.667 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.667 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:10:23.667 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.568564+0000", 2024-03-20T09:10:23.667 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.036e-06 2024-03-20T09:10:23.667 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.667 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.667 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.667 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.576589+0000", 2024-03-20T09:10:23.667 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 10.008024889 2024-03-20T09:10:23.667 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.667 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.667 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:10:23.667 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.580821+0000", 2024-03-20T09:10:23.667 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.004232085 2024-03-20T09:10:23.668 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.668 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.668 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,6,8", 2024-03-20T09:10:23.668 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.586154+0000", 2024-03-20T09:10:23.668 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0053331819999999997 2024-03-20T09:10:23.668 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.668 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.668 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:10:23.668 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.605130+0000", 2024-03-20T09:10:23.668 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018976066 2024-03-20T09:10:23.668 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.668 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.668 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.668 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.606187+0000", 2024-03-20T09:10:23.669 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001056731 2024-03-20T09:10:23.669 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.669 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.669 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.669 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.607462+0000", 2024-03-20T09:10:23.669 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0012742960000000001 2024-03-20T09:10:23.669 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.669 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.669 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.669 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.609070+0000", 2024-03-20T09:10:23.669 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0016087639999999999 2024-03-20T09:10:23.669 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.669 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.669 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:10:23.669 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.611497+0000", 2024-03-20T09:10:23.670 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0024264310000000002 2024-03-20T09:10:23.670 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.670 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.670 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:10:23.670 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.620942+0000", 2024-03-20T09:10:23.670 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0094451159999999999 2024-03-20T09:10:23.670 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.670 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:10:23.670 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.670 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.670 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.670 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:127411 3.7 3:e73aa2a1:::benchmark_data_smithi122_119671_object27629:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-03-20T09:10:23.670 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:07:49.564941+0000", 2024-03-20T09:10:23.670 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 153.957608585, 2024-03-20T09:10:23.671 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.202028329, 2024-03-20T09:10:23.671 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:10:23.671 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:10:23.671 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:10:23.671 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:10:23.671 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:10:23.671 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 127411 2024-03-20T09:10:23.671 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.671 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:10:23.671 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.671 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:10:23.671 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.564941+0000", 2024-03-20T09:10:23.671 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.671 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.672 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.672 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:10:23.672 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.564941+0000", 2024-03-20T09:10:23.672 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.672 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.672 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.672 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:10:23.672 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.564943+0000", 2024-03-20T09:10:23.672 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.367e-06 2024-03-20T09:10:23.672 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.672 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.672 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:10:23.672 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.564950+0000", 2024-03-20T09:10:23.672 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9759999999999998e-06 2024-03-20T09:10:23.672 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.673 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.673 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:10:23.673 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.564963+0000", 2024-03-20T09:10:23.673 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2853e-05 2024-03-20T09:10:23.673 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.673 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.673 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:10:23.673 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.564973+0000", 2024-03-20T09:10:23.673 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9799999999999993e-06 2024-03-20T09:10:23.673 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.673 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.673 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.673 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.567973+0000", 2024-03-20T09:10:23.673 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0030002919999999999 2024-03-20T09:10:23.674 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.674 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.674 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:10:23.674 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.567981+0000", 2024-03-20T09:10:23.674 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5599999999999994e-06 2024-03-20T09:10:23.674 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.674 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.674 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.674 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.752348+0000", 2024-03-20T09:10:23.674 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.184366904 2024-03-20T09:10:23.674 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.674 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.674 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:10:23.674 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.752544+0000", 2024-03-20T09:10:23.675 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000195534 2024-03-20T09:10:23.675 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.675 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.675 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,1", 2024-03-20T09:10:23.675 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.752725+0000", 2024-03-20T09:10:23.675 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000180819 2024-03-20T09:10:23.675 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.675 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.675 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:10:23.675 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.755179+0000", 2024-03-20T09:10:23.675 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002454492 2024-03-20T09:10:23.675 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.675 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.675 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.676 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.756120+0000", 2024-03-20T09:10:23.676 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00094096700000000002 2024-03-20T09:10:23.676 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.676 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.676 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.676 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.764838+0000", 2024-03-20T09:10:23.676 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0087181940000000003 2024-03-20T09:10:23.676 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.676 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.676 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:10:23.676 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.764911+0000", 2024-03-20T09:10:23.676 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2207999999999997e-05 2024-03-20T09:10:23.676 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.676 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.676 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:10:23.677 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.766970+0000", 2024-03-20T09:10:23.677 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002059183 2024-03-20T09:10:23.677 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.677 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:10:23.677 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.677 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.677 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.677 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:127463 3.15 3:aad91459:::benchmark_data_smithi122_119671_object27681:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-03-20T09:10:23.677 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:07:51.028651+0000", 2024-03-20T09:10:23.677 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 152.49389868399999, 2024-03-20T09:10:23.677 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6769376420000004, 2024-03-20T09:10:23.677 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:10:23.677 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:10:23.678 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:10:23.678 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:10:23.678 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:10:23.678 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 127463 2024-03-20T09:10:23.678 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.678 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:10:23.678 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.678 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:10:23.678 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:51.028651+0000", 2024-03-20T09:10:23.678 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.678 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.678 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.678 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:10:23.678 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:51.028651+0000", 2024-03-20T09:10:23.679 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:10:23.679 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.679 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.679 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:10:23.679 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:51.028652+0000", 2024-03-20T09:10:23.679 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1340000000000001e-06 2024-03-20T09:10:23.679 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.679 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.679 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:10:23.679 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:51.028657+0000", 2024-03-20T09:10:23.679 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.814e-06 2024-03-20T09:10:23.679 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.679 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.679 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:10:23.680 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:51.028666+0000", 2024-03-20T09:10:23.680 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1549999999999996e-06 2024-03-20T09:10:23.680 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.680 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.680 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:10:23.680 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:51.028673+0000", 2024-03-20T09:10:23.680 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0959999999999997e-06 2024-03-20T09:10:23.680 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.680 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.680 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:10:23.680 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.621337+0000", 2024-03-20T09:10:23.680 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5926647089999992 2024-03-20T09:10:23.680 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.680 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.680 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:10:23.680 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.625513+0000", 2024-03-20T09:10:23.681 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0041757469999999996 2024-03-20T09:10:23.681 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.681 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.681 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,6,8", 2024-03-20T09:10:23.681 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.631687+0000", 2024-03-20T09:10:23.681 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0061737179999999999 2024-03-20T09:10:23.681 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.681 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.681 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:10:23.681 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.637587+0000", 2024-03-20T09:10:23.681 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00589997 2024-03-20T09:10:23.681 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.681 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.681 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.682 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.639336+0000", 2024-03-20T09:10:23.682 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0017490959999999999 2024-03-20T09:10:23.682 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.682 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.682 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.682 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.641169+0000", 2024-03-20T09:10:23.682 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0018331599999999999 2024-03-20T09:10:23.682 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.682 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.682 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:10:23.682 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.695540+0000", 2024-03-20T09:10:23.682 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.054371052000000003 2024-03-20T09:10:23.683 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.683 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.683 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:10:23.683 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.697681+0000", 2024-03-20T09:10:23.683 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0021405489999999998 2024-03-20T09:10:23.683 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:10:23.683 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:10:23.683 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:10:23.683 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.705589+0000", 2024-03-20T09:10:23.683 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0079084419999999999 2024-03-20T09:10:23.683 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.683 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:10:23.683 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.683 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:10:23.684 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:10:23.684 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:10:23.684 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:10:23.685 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:23 smithi026 ceph-mon[68309]: 1.b scrub starts 2024-03-20T09:10:23.685 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:23 smithi026 ceph-mon[68309]: 1.b scrub ok 2024-03-20T09:10:23.685 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:23 smithi026 ceph-mon[68309]: pgmap v1049: 57 pgs: 1 remapped+peering, 1 peering, 2 unknown, 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 31 active+clean, 5 active+undersized; 3.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 3824/163392 objects degraded (2.340%); 23905/163392 objects misplaced (14.630%) 2024-03-20T09:10:23.685 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:23 smithi026 ceph-mon[68309]: osdmap e438: 12 total, 11 up, 10 in 2024-03-20T09:10:23.686 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:23 smithi026 ceph-mon[68309]: 1.a scrub starts 2024-03-20T09:10:23.686 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:23 smithi026 ceph-mon[68309]: 1.a scrub ok 2024-03-20T09:10:23.686 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:23 smithi115 ceph-mon[72476]: 1.b scrub starts 2024-03-20T09:10:23.686 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:23 smithi115 ceph-mon[72476]: 1.b scrub ok 2024-03-20T09:10:23.686 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:23 smithi115 ceph-mon[72476]: pgmap v1049: 57 pgs: 1 remapped+peering, 1 peering, 2 unknown, 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 31 active+clean, 5 active+undersized; 3.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 3824/163392 objects degraded (2.340%); 23905/163392 objects misplaced (14.630%) 2024-03-20T09:10:23.686 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:23 smithi115 ceph-mon[72476]: osdmap e438: 12 total, 11 up, 10 in 2024-03-20T09:10:23.686 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:23 smithi115 ceph-mon[72476]: 1.a scrub starts 2024-03-20T09:10:23.686 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:23 smithi115 ceph-mon[72476]: 1.a scrub ok 2024-03-20T09:10:23.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:23 smithi122 ceph-mon[73495]: 1.b scrub starts 2024-03-20T09:10:23.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:23 smithi122 ceph-mon[73495]: 1.b scrub ok 2024-03-20T09:10:23.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:23 smithi122 ceph-mon[73495]: pgmap v1049: 57 pgs: 1 remapped+peering, 1 peering, 2 unknown, 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 31 active+clean, 5 active+undersized; 3.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 3824/163392 objects degraded (2.340%); 23905/163392 objects misplaced (14.630%) 2024-03-20T09:10:23.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:23 smithi122 ceph-mon[73495]: osdmap e438: 12 total, 11 up, 10 in 2024-03-20T09:10:23.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:23 smithi122 ceph-mon[73495]: 1.a scrub starts 2024-03-20T09:10:23.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:23 smithi122 ceph-mon[73495]: 1.a scrub ok 2024-03-20T09:10:23.718 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:24.079 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:10:24.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:24.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:10:24.185 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:10:24.226 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:24.274 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-20T09:10:24.355 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:10:24.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:24.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:10:24.460 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:10:24.487 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:24.506 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:10:24.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:24.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:10:24.609 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:10:24.635 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:24.654 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:10:24.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:24.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:10:24.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:24 smithi122 ceph-mon[73495]: 1.17 deep-scrub starts 2024-03-20T09:10:24.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:24 smithi122 ceph-mon[73495]: 1.17 deep-scrub ok 2024-03-20T09:10:24.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:24 smithi122 ceph-mon[73495]: 3.10 scrub starts 2024-03-20T09:10:24.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:24 smithi026 ceph-mon[68309]: 1.17 deep-scrub starts 2024-03-20T09:10:24.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:24 smithi026 ceph-mon[68309]: 1.17 deep-scrub ok 2024-03-20T09:10:24.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:24 smithi026 ceph-mon[68309]: 3.10 scrub starts 2024-03-20T09:10:24.757 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:10:24.776 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:10:24.782 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:24 smithi115 ceph-mon[72476]: 1.17 deep-scrub starts 2024-03-20T09:10:24.782 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:24 smithi115 ceph-mon[72476]: 1.17 deep-scrub ok 2024-03-20T09:10:24.782 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:24 smithi115 ceph-mon[72476]: 3.10 scrub starts 2024-03-20T09:10:24.784 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:24.802 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:10:24.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:24.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:10:24.906 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:10:24.932 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:24.951 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:10:24.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:24.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:10:25.002 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:10:25.017 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 8, 10, 11, 3] out_osds: [9, 4] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11] 2024-03-20T09:10:25.017 INFO:tasks.thrashosds.thrasher:Reviving osd 5 2024-03-20T09:10:25.017 INFO:tasks.cephadm.osd.5:Restarting osd.5 (starting--it wasn't running)... 2024-03-20T09:10:25.017 DEBUG:teuthology.orchestra.run.smithi115:osd.5> sudo journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.5.service 2024-03-20T09:10:25.020 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl start ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.5 2024-03-20T09:10:25.047 INFO:journalctl@ceph.osd.5.smithi115.stdout:-- Logs begin at Wed 2024-03-20 08:37:21 UTC. -- 2024-03-20T09:10:25.054 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:10:25.111 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:25.154 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:10:25.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:25.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:10:25.259 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:10:25.295 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:25.317 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:10:25.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:25.321 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:10:25.407 INFO:journalctl@ceph.osd.5.smithi115.stdout:Mar 20 09:10:25 smithi115 systemd[1]: Starting Ceph osd.5 for 852fbb68-e696-11ee-95c9-87774f69a715... 2024-03-20T09:10:25.421 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:10:25.463 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:10:25.466 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:25.684 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:10:25.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:25.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:10:25.690 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:25 smithi026 ceph-mon[68309]: pgmap v1052: 57 pgs: 1 remapped+peering, 1 peering, 2 unknown, 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 31 active+clean, 5 active+undersized; 3.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 3820/163365 objects degraded (2.338%); 23905/163365 objects misplaced (14.633%) 2024-03-20T09:10:25.690 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:25 smithi026 ceph-mon[68309]: 3.10 scrub ok 2024-03-20T09:10:25.690 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:25 smithi026 ceph-mon[68309]: 1.1e scrub starts 2024-03-20T09:10:25.691 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:25 smithi026 ceph-mon[68309]: 1.1e scrub ok 2024-03-20T09:10:25.691 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:25 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3029025597' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:10:25.691 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:25 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:10:25.691 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:25 smithi026 ceph-mon[68309]: 1.f scrub starts 2024-03-20T09:10:25.691 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:25 smithi026 ceph-mon[68309]: 1.f scrub ok 2024-03-20T09:10:25.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:25 smithi122 ceph-mon[73495]: pgmap v1052: 57 pgs: 1 remapped+peering, 1 peering, 2 unknown, 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 31 active+clean, 5 active+undersized; 3.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 3820/163365 objects degraded (2.338%); 23905/163365 objects misplaced (14.633%) 2024-03-20T09:10:25.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:25 smithi122 ceph-mon[73495]: 3.10 scrub ok 2024-03-20T09:10:25.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:25 smithi122 ceph-mon[73495]: 1.1e scrub starts 2024-03-20T09:10:25.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:25 smithi122 ceph-mon[73495]: 1.1e scrub ok 2024-03-20T09:10:25.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:25 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3029025597' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:10:25.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:25 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:10:25.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:25 smithi122 ceph-mon[73495]: 1.f scrub starts 2024-03-20T09:10:25.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:25 smithi122 ceph-mon[73495]: 1.f scrub ok 2024-03-20T09:10:25.791 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:10:25.826 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:25.884 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:10:25.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:25.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:10:25.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:25 smithi115 ceph-mon[72476]: pgmap v1052: 57 pgs: 1 remapped+peering, 1 peering, 2 unknown, 13 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 31 active+clean, 5 active+undersized; 3.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 3820/163365 objects degraded (2.338%); 23905/163365 objects misplaced (14.633%) 2024-03-20T09:10:25.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:25 smithi115 ceph-mon[72476]: 3.10 scrub ok 2024-03-20T09:10:25.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:25 smithi115 ceph-mon[72476]: 1.1e scrub starts 2024-03-20T09:10:25.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:25 smithi115 ceph-mon[72476]: 1.1e scrub ok 2024-03-20T09:10:25.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:25 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3029025597' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:10:25.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:25 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:10:25.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:25 smithi115 ceph-mon[72476]: 1.f scrub starts 2024-03-20T09:10:25.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:25 smithi115 ceph-mon[72476]: 1.f scrub ok 2024-03-20T09:10:25.991 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:10:26.025 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:26.046 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:10:26.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:26.050 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:10:26.151 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:10:26.184 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:26.205 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:10:26.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:26.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:10:26.310 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:10:26.340 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:26.360 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:10:26.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:26.363 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:10:26.406 INFO:journalctl@ceph.osd.5.smithi115.stdout:Mar 20 09:10:26 smithi115 bash[138949]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-20T09:10:26.407 INFO:journalctl@ceph.osd.5.smithi115.stdout:Mar 20 09:10:26 smithi115 bash[138949]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-20T09:10:26.407 INFO:journalctl@ceph.osd.5.smithi115.stdout:Mar 20 09:10:26 smithi115 bash[138949]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-20T09:10:26.407 INFO:journalctl@ceph.osd.5.smithi115.stdout:Mar 20 09:10:26 smithi115 bash[138949]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-20T09:10:26.407 INFO:journalctl@ceph.osd.5.smithi115.stdout:Mar 20 09:10:26 smithi115 bash[138949]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-03-20T09:10:26.407 INFO:journalctl@ceph.osd.5.smithi115.stdout:Mar 20 09:10:26 smithi115 bash[138949]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-20T09:10:26.407 INFO:journalctl@ceph.osd.5.smithi115.stdout:Mar 20 09:10:26 smithi115 bash[138949]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-20T09:10:26.464 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:10:26.499 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:26.563 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:10:26.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:26.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:10:26.673 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:10:26.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:26 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:10:26.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:26 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:10:26.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:26 smithi122 ceph-mon[73495]: osdmap e439: 12 total, 11 up, 10 in 2024-03-20T09:10:26.703 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:26 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:10:26.703 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:26 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:10:26.704 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:26 smithi026 ceph-mon[68309]: osdmap e439: 12 total, 11 up, 10 in 2024-03-20T09:10:26.708 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:26.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:26 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:10:26.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:26 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:10:26.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:26 smithi115 ceph-mon[72476]: osdmap e439: 12 total, 11 up, 10 in 2024-03-20T09:10:27.198 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:10:27.262 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:10:27.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:27.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:10:27.369 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:10:27.402 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:27.423 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:10:27.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:27.426 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:10:27.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:27 smithi026 ceph-mon[68309]: pgmap v1054: 57 pgs: 1 unknown, 14 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 32 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 3.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 3820/163167 objects degraded (2.341%); 25272/163167 objects misplaced (15.488%) 2024-03-20T09:10:27.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:27 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 3820/163167 objects degraded (2.341%), 4 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:10:27.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:27 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:10:27.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:27 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:10:27.528 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:10:27.556 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:27.575 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:10:27.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:27.578 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:10:27.640 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:27 smithi115 ceph-mon[72476]: pgmap v1054: 57 pgs: 1 unknown, 14 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 32 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 3.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 3820/163167 objects degraded (2.341%); 25272/163167 objects misplaced (15.488%) 2024-03-20T09:10:27.641 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:27 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 3820/163167 objects degraded (2.341%), 4 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:10:27.641 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:27 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:10:27.641 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:27 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:10:27.641 INFO:journalctl@ceph.osd.5.smithi115.stdout:Mar 20 09:10:27 smithi115 ceph-osd[139272]: -- 172.21.15.115:0/228391124 <== mon.2 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5611b8f3f680 con 0x5611b8f09000 2024-03-20T09:10:27.644 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-20T09:10:27.679 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:10:27.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:27 smithi122 ceph-mon[73495]: pgmap v1054: 57 pgs: 1 unknown, 14 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 32 active+clean, 1 active+recovering+undersized+remapped, 5 active+undersized; 3.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 3820/163167 objects degraded (2.341%); 25272/163167 objects misplaced (15.488%) 2024-03-20T09:10:27.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:27 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 3820/163167 objects degraded (2.341%), 4 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-03-20T09:10:27.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:27 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:10:27.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:27 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:10:27.706 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:27.724 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:10:27.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:27.727 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:10:27.828 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:10:27.871 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:27.906 INFO:journalctl@ceph.osd.5.smithi115.stdout:Mar 20 09:10:27 smithi115 bash[139235]: b56afed1a0bb4ca2ebd4dacea77afb80a80675ab6af7695e1b769ee12bf63b46 2024-03-20T09:10:27.906 INFO:journalctl@ceph.osd.5.smithi115.stdout:Mar 20 09:10:27 smithi115 systemd[1]: Started Ceph osd.5 for 852fbb68-e696-11ee-95c9-87774f69a715. 2024-03-20T09:10:27.912 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:10:27.955 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:10:27.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:27.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:10:28.059 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:10:28.092 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:28.113 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:10:28.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:28.117 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:10:28.217 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:10:28.248 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:28.268 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:10:28.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:28.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:10:28.372 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:10:28.396 INFO:journalctl@ceph.osd.5.smithi115.stdout:Mar 20 09:10:28 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-5[139249]: 2024-03-20T09:10:28.076+0000 7f37cbc027c0 -1 Falling back to public interface 2024-03-20T09:10:28.398 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:28.412 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:10:28.418 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:10:28.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:28.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:10:28.522 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:10:28.554 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:28.575 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:10:28.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:28.578 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:10:28.679 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:10:28.749 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:28.769 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:10:28.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:28.772 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:10:28.873 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:10:29.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:29 smithi115 ceph-mon[72476]: pgmap v1055: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 14 active+clean+remapped, 2 active+undersized+degraded, 32 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 3.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 8865/168072 objects degraded (5.275%); 21555/168072 objects misplaced (12.825%) 2024-03-20T09:10:29.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:29 smithi115 ceph-mon[72476]: osdmap e440: 12 total, 11 up, 10 in 2024-03-20T09:10:29.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:29 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:10:29.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:29 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:10:29.296 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:29 smithi026 ceph-mon[68309]: pgmap v1055: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 14 active+clean+remapped, 2 active+undersized+degraded, 32 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 3.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 8865/168072 objects degraded (5.275%); 21555/168072 objects misplaced (12.825%) 2024-03-20T09:10:29.296 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:29 smithi026 ceph-mon[68309]: osdmap e440: 12 total, 11 up, 10 in 2024-03-20T09:10:29.296 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:29 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:10:29.296 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:29 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:10:29.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:29 smithi122 ceph-mon[73495]: pgmap v1055: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 14 active+clean+remapped, 2 active+undersized+degraded, 32 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 3.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 8865/168072 objects degraded (5.275%); 21555/168072 objects misplaced (12.825%) 2024-03-20T09:10:29.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:29 smithi122 ceph-mon[73495]: osdmap e440: 12 total, 11 up, 10 in 2024-03-20T09:10:29.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:29 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:10:29.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:29 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:10:29.887 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:29.999 INFO:teuthology.orchestra.run.smithi115.stderr:no valid command found; 10 closest matches: 2024-03-20T09:10:30.000 INFO:teuthology.orchestra.run.smithi115.stderr:0 2024-03-20T09:10:30.000 INFO:teuthology.orchestra.run.smithi115.stderr:1 2024-03-20T09:10:30.000 INFO:teuthology.orchestra.run.smithi115.stderr:2 2024-03-20T09:10:30.000 INFO:teuthology.orchestra.run.smithi115.stderr:abort 2024-03-20T09:10:30.000 INFO:teuthology.orchestra.run.smithi115.stderr:assert 2024-03-20T09:10:30.000 INFO:teuthology.orchestra.run.smithi115.stderr:config diff 2024-03-20T09:10:30.000 INFO:teuthology.orchestra.run.smithi115.stderr:config diff get 2024-03-20T09:10:30.000 INFO:teuthology.orchestra.run.smithi115.stderr:config get 2024-03-20T09:10:30.000 INFO:teuthology.orchestra.run.smithi115.stderr:config help [] 2024-03-20T09:10:30.000 INFO:teuthology.orchestra.run.smithi115.stderr:config set ... 2024-03-20T09:10:30.000 INFO:teuthology.orchestra.run.smithi115.stderr:admin_socket: invalid command 2024-03-20T09:10:30.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:30 smithi026 ceph-mon[68309]: osdmap e441: 12 total, 11 up, 10 in 2024-03-20T09:10:30.391 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:30 smithi115 ceph-mon[72476]: osdmap e441: 12 total, 11 up, 10 in 2024-03-20T09:10:30.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:30 smithi122 ceph-mon[73495]: osdmap e441: 12 total, 11 up, 10 in 2024-03-20T09:10:30.869 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T09:10:30.870 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-03-20T09:10:31.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:31 smithi115 ceph-mon[72476]: pgmap v1058: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 14 active+clean+remapped, 2 active+undersized+degraded, 32 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 3.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 8865/168072 objects degraded (5.275%); 21555/168072 objects misplaced (12.825%) 2024-03-20T09:10:31.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:31 smithi122 ceph-mon[73495]: pgmap v1058: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 14 active+clean+remapped, 2 active+undersized+degraded, 32 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 3.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 8865/168072 objects degraded (5.275%); 21555/168072 objects misplaced (12.825%) 2024-03-20T09:10:31.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:31 smithi026 ceph-mon[68309]: pgmap v1058: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 14 active+clean+remapped, 2 active+undersized+degraded, 32 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 3.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 8865/168072 objects degraded (5.275%); 21555/168072 objects misplaced (12.825%) 2024-03-20T09:10:31.906 INFO:journalctl@ceph.osd.5.smithi115.stdout:Mar 20 09:10:31 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-5[139249]: 2024-03-20T09:10:31.557+0000 7f37cbc027c0 -1 osd.5 273 log_to_monitors true 2024-03-20T09:10:32.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:32 smithi115 ceph-mon[72476]: from='osd.5 [v2:172.21.15.115:6808/650416648,v1:172.21.15.115:6809/650416648]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-20T09:10:32.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:32 smithi115 ceph-mon[72476]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-20T09:10:32.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:32 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8865/168072 objects degraded (5.275%), 3 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-03-20T09:10:32.407 INFO:journalctl@ceph.osd.5.smithi115.stdout:Mar 20 09:10:32 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-5[139249]: 2024-03-20T09:10:32.162+0000 7f37bd05a700 -1 osd.5 442 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-20T09:10:32.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:32 smithi122 ceph-mon[73495]: from='osd.5 [v2:172.21.15.115:6808/650416648,v1:172.21.15.115:6809/650416648]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-20T09:10:32.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:32 smithi122 ceph-mon[73495]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-20T09:10:32.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:32 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8865/168072 objects degraded (5.275%), 3 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-03-20T09:10:32.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:32 smithi026 ceph-mon[68309]: from='osd.5 [v2:172.21.15.115:6808/650416648,v1:172.21.15.115:6809/650416648]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-20T09:10:32.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:32 smithi026 ceph-mon[68309]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-20T09:10:32.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:32 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8865/168072 objects degraded (5.275%), 3 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-03-20T09:10:33.009 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:10:33.009 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:10:33.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:10:33.009 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPushReply(3.7 440/435 [PushReplyOp(3:e70db2ea:::benchmark_data_smithi122_119671_object67027:head)])", 2024-03-20T09:10:33.009 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:10:28.149316+0000", 2024-03-20T09:10:33.009 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0012155359999999999, 2024-03-20T09:10:33.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00123649, 2024-03-20T09:10:33.009 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:10:33.010 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:10:33.010 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:10:33.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:10:33.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:10:33.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:28.149316+0000", 2024-03-20T09:10:33.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:10:33.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:10:33.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:10:33.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:10:33.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:28.149316+0000", 2024-03-20T09:10:33.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:10:33.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:10:33.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:10:33.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:10:33.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:28.149317+0000", 2024-03-20T09:10:33.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0020000000000001e-06 2024-03-20T09:10:33.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:10:33.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:10:33.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:10:33.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:28.149323+0000", 2024-03-20T09:10:33.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.344e-06 2024-03-20T09:10:33.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:10:33.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:10:33.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:10:33.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:28.149334+0000", 2024-03-20T09:10:33.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1342000000000001e-05 2024-03-20T09:10:33.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:10:33.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:10:33.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:10:33.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:28.149342+0000", 2024-03-20T09:10:33.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.473e-06 2024-03-20T09:10:33.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:10:33.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:10:33.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:10:33.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:28.149669+0000", 2024-03-20T09:10:33.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00032616399999999999 2024-03-20T09:10:33.012 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:10:33.012 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:10:33.012 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:10:33.012 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:10:33.012 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:10:33.012 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 1 2024-03-20T09:10:33.012 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:10:33.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:33 smithi115 ceph-mon[72476]: pgmap v1059: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 14 active+clean+remapped, 2 active+undersized+degraded, 32 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 3.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 8865/168072 objects degraded (5.275%); 20961/168072 objects misplaced (12.471%) 2024-03-20T09:10:33.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:33 smithi115 ceph-mon[72476]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-20T09:10:33.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:33 smithi115 ceph-mon[72476]: osdmap e442: 12 total, 11 up, 10 in 2024-03-20T09:10:33.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:33 smithi115 ceph-mon[72476]: from='osd.5 [v2:172.21.15.115:6808/650416648,v1:172.21.15.115:6809/650416648]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-03-20T09:10:33.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:33 smithi115 ceph-mon[72476]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-03-20T09:10:33.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:33 smithi122 ceph-mon[73495]: pgmap v1059: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 14 active+clean+remapped, 2 active+undersized+degraded, 32 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 3.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 8865/168072 objects degraded (5.275%); 20961/168072 objects misplaced (12.471%) 2024-03-20T09:10:33.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:33 smithi122 ceph-mon[73495]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-20T09:10:33.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:33 smithi122 ceph-mon[73495]: osdmap e442: 12 total, 11 up, 10 in 2024-03-20T09:10:33.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:33 smithi122 ceph-mon[73495]: from='osd.5 [v2:172.21.15.115:6808/650416648,v1:172.21.15.115:6809/650416648]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-03-20T09:10:33.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:33 smithi122 ceph-mon[73495]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-03-20T09:10:33.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:33 smithi026 ceph-mon[68309]: pgmap v1059: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 14 active+clean+remapped, 2 active+undersized+degraded, 32 active+clean, 2 active+recovering+undersized+remapped, 5 active+undersized; 3.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 8865/168072 objects degraded (5.275%); 20961/168072 objects misplaced (12.471%) 2024-03-20T09:10:33.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:33 smithi026 ceph-mon[68309]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-20T09:10:33.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:33 smithi026 ceph-mon[68309]: osdmap e442: 12 total, 11 up, 10 in 2024-03-20T09:10:33.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:33 smithi026 ceph-mon[68309]: from='osd.5 [v2:172.21.15.115:6808/650416648,v1:172.21.15.115:6809/650416648]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-03-20T09:10:33.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:33 smithi026 ceph-mon[68309]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi115", "root=default"]}]: dispatch 2024-03-20T09:10:34.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:34 smithi115 ceph-mon[72476]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-20T09:10:34.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:34 smithi115 ceph-mon[72476]: osd.5 [v2:172.21.15.115:6808/650416648,v1:172.21.15.115:6809/650416648] boot 2024-03-20T09:10:34.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:34 smithi115 ceph-mon[72476]: osdmap e443: 12 total, 12 up, 10 in 2024-03-20T09:10:34.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:34 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T09:10:34.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:34 smithi115 ceph-mon[72476]: osdmap e444: 12 total, 12 up, 10 in 2024-03-20T09:10:34.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:34 smithi122 ceph-mon[73495]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-20T09:10:34.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:34 smithi122 ceph-mon[73495]: osd.5 [v2:172.21.15.115:6808/650416648,v1:172.21.15.115:6809/650416648] boot 2024-03-20T09:10:34.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:34 smithi122 ceph-mon[73495]: osdmap e443: 12 total, 12 up, 10 in 2024-03-20T09:10:34.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:34 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T09:10:34.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:34 smithi122 ceph-mon[73495]: osdmap e444: 12 total, 12 up, 10 in 2024-03-20T09:10:34.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:34 smithi026 ceph-mon[68309]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-20T09:10:34.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:34 smithi026 ceph-mon[68309]: osd.5 [v2:172.21.15.115:6808/650416648,v1:172.21.15.115:6809/650416648] boot 2024-03-20T09:10:34.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:34 smithi026 ceph-mon[68309]: osdmap e443: 12 total, 12 up, 10 in 2024-03-20T09:10:34.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:34 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-20T09:10:34.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:34 smithi026 ceph-mon[68309]: osdmap e444: 12 total, 12 up, 10 in 2024-03-20T09:10:35.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:35 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:10:35.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:35 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:10:35.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:35 smithi115 ceph-mon[72476]: pgmap v1062: 57 pgs: 1 peering, 2 remapped+peering, 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 12 active+clean+remapped, 2 active+undersized+degraded, 32 active+clean, 2 active+recovering+undersized+remapped, 4 active+undersized; 3.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 8865/168072 objects degraded (5.275%); 20960/168072 objects misplaced (12.471%); 6.5 MiB/s, 103 objects/s recovering 2024-03-20T09:10:35.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:35 smithi115 ceph-mon[72476]: osdmap e445: 12 total, 12 up, 10 in 2024-03-20T09:10:35.411 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:10:35.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:35.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:10:35.418 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:35 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:10:35.418 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:35 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:10:35.418 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:35 smithi026 ceph-mon[68309]: pgmap v1062: 57 pgs: 1 peering, 2 remapped+peering, 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 12 active+clean+remapped, 2 active+undersized+degraded, 32 active+clean, 2 active+recovering+undersized+remapped, 4 active+undersized; 3.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 8865/168072 objects degraded (5.275%); 20960/168072 objects misplaced (12.471%); 6.5 MiB/s, 103 objects/s recovering 2024-03-20T09:10:35.418 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:35 smithi026 ceph-mon[68309]: osdmap e445: 12 total, 12 up, 10 in 2024-03-20T09:10:35.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:35 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:10:35.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:35 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:10:35.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:35 smithi122 ceph-mon[73495]: pgmap v1062: 57 pgs: 1 peering, 2 remapped+peering, 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 12 active+clean+remapped, 2 active+undersized+degraded, 32 active+clean, 2 active+recovering+undersized+remapped, 4 active+undersized; 3.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 8865/168072 objects degraded (5.275%); 20960/168072 objects misplaced (12.471%); 6.5 MiB/s, 103 objects/s recovering 2024-03-20T09:10:35.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:35 smithi122 ceph-mon[73495]: osdmap e445: 12 total, 12 up, 10 in 2024-03-20T09:10:35.519 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:10:35.550 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:35.570 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:10:35.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:35.573 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:10:35.674 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:10:35.871 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-20T09:10:36.086 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:36.259 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:10:36.260 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:10:36.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:36.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:10:36.366 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:10:36.396 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:36.599 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:10:36.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:36.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:10:36.703 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:10:36.772 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:37.080 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:10:37.096 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:37 smithi026 ceph-mon[68309]: pgmap v1065: 57 pgs: 1 peering, 6 remapped+peering, 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 5 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded, 34 active+clean, 3 active+recovering+undersized+remapped, 3 active+undersized; 3.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 10980/168072 objects degraded (6.533%); 5970/168072 objects misplaced (3.552%); 13 MiB/s, 241 objects/s recovering 2024-03-20T09:10:37.096 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:37 smithi026 ceph-mon[68309]: osdmap e446: 12 total, 12 up, 10 in 2024-03-20T09:10:37.096 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:37 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:10:37.096 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:37 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:10:37.096 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:37 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 10980/168072 objects degraded (6.533%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-20T09:10:37.096 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:37 smithi026 ceph-mon[68309]: osdmap e447: 12 total, 12 up, 10 in 2024-03-20T09:10:37.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:37.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:10:37.218 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:10:37.253 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:37.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:37 smithi115 ceph-mon[72476]: pgmap v1065: 57 pgs: 1 peering, 6 remapped+peering, 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 5 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded, 34 active+clean, 3 active+recovering+undersized+remapped, 3 active+undersized; 3.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 10980/168072 objects degraded (6.533%); 5970/168072 objects misplaced (3.552%); 13 MiB/s, 241 objects/s recovering 2024-03-20T09:10:37.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:37 smithi115 ceph-mon[72476]: osdmap e446: 12 total, 12 up, 10 in 2024-03-20T09:10:37.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:37 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:10:37.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:37 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:10:37.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:37 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 10980/168072 objects degraded (6.533%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-20T09:10:37.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:37 smithi115 ceph-mon[72476]: osdmap e447: 12 total, 12 up, 10 in 2024-03-20T09:10:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:37 smithi122 ceph-mon[73495]: pgmap v1065: 57 pgs: 1 peering, 6 remapped+peering, 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 5 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded, 34 active+clean, 3 active+recovering+undersized+remapped, 3 active+undersized; 3.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 10980/168072 objects degraded (6.533%); 5970/168072 objects misplaced (3.552%); 13 MiB/s, 241 objects/s recovering 2024-03-20T09:10:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:37 smithi122 ceph-mon[73495]: osdmap e446: 12 total, 12 up, 10 in 2024-03-20T09:10:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:37 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:10:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:37 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:10:37.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:37 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 10980/168072 objects degraded (6.533%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-20T09:10:37.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:37 smithi122 ceph-mon[73495]: osdmap e447: 12 total, 12 up, 10 in 2024-03-20T09:10:38.005 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:10:38.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:38.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:10:38.110 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:10:38.143 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:39.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:39 smithi115 ceph-mon[72476]: pgmap v1068: 57 pgs: 1 peering, 4 remapped+peering, 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 3 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded, 40 active+clean, 3 active+recovering+undersized+remapped, 1 active+undersized; 3.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 10980/168072 objects degraded (6.533%); 5970/168072 objects misplaced (3.552%); 13 MiB/s, 241 objects/s recovering 2024-03-20T09:10:39.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:39 smithi122 ceph-mon[73495]: pgmap v1068: 57 pgs: 1 peering, 4 remapped+peering, 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 3 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded, 40 active+clean, 3 active+recovering+undersized+remapped, 1 active+undersized; 3.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 10980/168072 objects degraded (6.533%); 5970/168072 objects misplaced (3.552%); 13 MiB/s, 241 objects/s recovering 2024-03-20T09:10:39.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:39 smithi026 ceph-mon[68309]: pgmap v1068: 57 pgs: 1 peering, 4 remapped+peering, 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 3 active+clean+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded, 40 active+clean, 3 active+recovering+undersized+remapped, 1 active+undersized; 3.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 10980/168072 objects degraded (6.533%); 5970/168072 objects misplaced (3.552%); 13 MiB/s, 241 objects/s recovering 2024-03-20T09:10:41.300 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:10:41.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:41.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:10:41.405 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:10:41.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:41 smithi115 ceph-mon[72476]: pgmap v1069: 57 pgs: 3 remapped+peering, 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 3 active+clean+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 41 active+clean, 4 active+recovering+undersized+remapped, 1 active+undersized; 3.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 12440/168072 objects degraded (7.402%); 5970/168072 objects misplaced (3.552%); 9.3 MiB/s, 293 objects/s recovering 2024-03-20T09:10:41.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:41 smithi115 ceph-mon[72476]: osdmap e448: 12 total, 12 up, 10 in 2024-03-20T09:10:41.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:41 smithi122 ceph-mon[73495]: pgmap v1069: 57 pgs: 3 remapped+peering, 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 3 active+clean+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 41 active+clean, 4 active+recovering+undersized+remapped, 1 active+undersized; 3.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 12440/168072 objects degraded (7.402%); 5970/168072 objects misplaced (3.552%); 9.3 MiB/s, 293 objects/s recovering 2024-03-20T09:10:41.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:41 smithi122 ceph-mon[73495]: osdmap e448: 12 total, 12 up, 10 in 2024-03-20T09:10:41.434 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:41.454 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:10:41.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:41.457 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:10:41.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:41 smithi026 ceph-mon[68309]: pgmap v1069: 57 pgs: 3 remapped+peering, 1 activating+undersized+degraded+remapped, 1 active+recovering+remapped, 3 active+clean+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 41 active+clean, 4 active+recovering+undersized+remapped, 1 active+undersized; 3.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 12440/168072 objects degraded (7.402%); 5970/168072 objects misplaced (3.552%); 9.3 MiB/s, 293 objects/s recovering 2024-03-20T09:10:41.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:41 smithi026 ceph-mon[68309]: osdmap e448: 12 total, 12 up, 10 in 2024-03-20T09:10:41.557 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:10:41.585 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:41.603 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:10:41.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:41.606 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:10:41.707 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:10:41.832 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:41.970 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:10:41.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:41.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:10:42.076 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:10:42.114 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:42.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:42 smithi115 ceph-mon[72476]: osdmap e449: 12 total, 12 up, 10 in 2024-03-20T09:10:42.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:42 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 12440/168072 objects degraded (7.402%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:10:42.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:42 smithi122 ceph-mon[73495]: osdmap e449: 12 total, 12 up, 10 in 2024-03-20T09:10:42.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:42 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 12440/168072 objects degraded (7.402%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:10:42.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:42 smithi026 ceph-mon[68309]: osdmap e449: 12 total, 12 up, 10 in 2024-03-20T09:10:42.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:42 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 12440/168072 objects degraded (7.402%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:10:42.992 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:10:42.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:42.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:10:43.097 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:10:43.129 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:43.150 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:10:43.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:43.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:10:43.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: pgmap v1072: 57 pgs: 1 active+undersized+degraded+remapped, 1 active+recovering+remapped, 1 remapped+peering, 1 peering, 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 43 active+clean, 5 active+recovering+undersized+remapped; 3.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 0 op/s; 15583/168045 objects degraded (9.273%); 27 MiB/s, 1.61k objects/s recovering 2024-03-20T09:10:43.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: osdmap e450: 12 total, 12 up, 10 in 2024-03-20T09:10:43.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {auth=31,config=2,health=12,kv=55} crc {auth=2764950393,config=3599132250,health=1132862079,kv=2220733292}) 2024-03-20T09:10:43.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=869441548,logm=3597946349}) 2024-03-20T09:10:43.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=615551518}) 2024-03-20T09:10:43.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=944649251}) 2024-03-20T09:10:43.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1631270514}) 2024-03-20T09:10:43.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2639833894}) 2024-03-20T09:10:43.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3146564972}) 2024-03-20T09:10:43.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4195593963}) 2024-03-20T09:10:43.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=135680167}) 2024-03-20T09:10:43.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=573786874}) 2024-03-20T09:10:43.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1443144167}) 2024-03-20T09:10:43.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2951652205}) 2024-03-20T09:10:43.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3558909144}) 2024-03-20T09:10:43.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2119656565}) 2024-03-20T09:10:43.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3335057020}) 2024-03-20T09:10:43.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3968236221}) 2024-03-20T09:10:43.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3482803010}) 2024-03-20T09:10:43.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2449953336}) 2024-03-20T09:10:43.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4101471215}) 2024-03-20T09:10:43.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1246627036}) 2024-03-20T09:10:43.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3750522635}) 2024-03-20T09:10:43.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2455325710}) 2024-03-20T09:10:43.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3202381441}) 2024-03-20T09:10:43.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2888363655}) 2024-03-20T09:10:43.224 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3500283637}) 2024-03-20T09:10:43.224 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4071316231}) 2024-03-20T09:10:43.224 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=660166270}) 2024-03-20T09:10:43.224 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3812600652}) 2024-03-20T09:10:43.224 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3403424939}) 2024-03-20T09:10:43.224 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1638447558}) 2024-03-20T09:10:43.224 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=155806089}) 2024-03-20T09:10:43.224 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3495521351}) 2024-03-20T09:10:43.224 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1970886986}) 2024-03-20T09:10:43.224 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1787527661}) 2024-03-20T09:10:43.224 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3033714217}) 2024-03-20T09:10:43.224 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2344273786}) 2024-03-20T09:10:43.224 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2253571105}) 2024-03-20T09:10:43.224 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3993672329}) 2024-03-20T09:10:43.225 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2626070479}) 2024-03-20T09:10:43.225 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4273892222}) 2024-03-20T09:10:43.225 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=48,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=16,mon_config_key=12} crc {logm=1182473682,mdsmap=3984425268,mgr=427508465,mgr_command_descs=3831600018,mgr_metadata=2702834764,mgrstat=248488440,mon_config_key=930068308}) 2024-03-20T09:10:43.225 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2488951626}) 2024-03-20T09:10:43.225 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1728954769}) 2024-03-20T09:10:43.225 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=66,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=16} crc {mon_config_key=837948195,monmap=3481182261,osd_metadata=1898051788,osd_pg_creating=4016002691,osdmap=1862387825}) 2024-03-20T09:10:43.225 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1953344786}) 2024-03-20T09:10:43.225 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1232679788}) 2024-03-20T09:10:43.225 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3286485756}) 2024-03-20T09:10:43.225 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=84} crc {osdmap=3922860416}) 2024-03-20T09:10:43.255 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:10:43.285 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: pgmap v1072: 57 pgs: 1 active+undersized+degraded+remapped, 1 active+recovering+remapped, 1 remapped+peering, 1 peering, 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 43 active+clean, 5 active+recovering+undersized+remapped; 3.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 0 op/s; 15583/168045 objects degraded (9.273%); 27 MiB/s, 1.61k objects/s recovering 2024-03-20T09:10:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: osdmap e450: 12 total, 12 up, 10 in 2024-03-20T09:10:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {auth=31,config=2,health=12,kv=55} crc {auth=2764950393,config=3599132250,health=1132862079,kv=2220733292}) 2024-03-20T09:10:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=869441548,logm=3597946349}) 2024-03-20T09:10:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=615551518}) 2024-03-20T09:10:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=944649251}) 2024-03-20T09:10:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1631270514}) 2024-03-20T09:10:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2639833894}) 2024-03-20T09:10:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3146564972}) 2024-03-20T09:10:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4195593963}) 2024-03-20T09:10:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=135680167}) 2024-03-20T09:10:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=573786874}) 2024-03-20T09:10:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1443144167}) 2024-03-20T09:10:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2951652205}) 2024-03-20T09:10:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3558909144}) 2024-03-20T09:10:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2119656565}) 2024-03-20T09:10:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3335057020}) 2024-03-20T09:10:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3968236221}) 2024-03-20T09:10:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3482803010}) 2024-03-20T09:10:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2449953336}) 2024-03-20T09:10:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4101471215}) 2024-03-20T09:10:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1246627036}) 2024-03-20T09:10:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3750522635}) 2024-03-20T09:10:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2455325710}) 2024-03-20T09:10:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3202381441}) 2024-03-20T09:10:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2888363655}) 2024-03-20T09:10:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3500283637}) 2024-03-20T09:10:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4071316231}) 2024-03-20T09:10:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=660166270}) 2024-03-20T09:10:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3812600652}) 2024-03-20T09:10:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3403424939}) 2024-03-20T09:10:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1638447558}) 2024-03-20T09:10:43.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=155806089}) 2024-03-20T09:10:43.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3495521351}) 2024-03-20T09:10:43.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1970886986}) 2024-03-20T09:10:43.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1787527661}) 2024-03-20T09:10:43.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3033714217}) 2024-03-20T09:10:43.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2344273786}) 2024-03-20T09:10:43.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2253571105}) 2024-03-20T09:10:43.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3993672329}) 2024-03-20T09:10:43.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2626070479}) 2024-03-20T09:10:43.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4273892222}) 2024-03-20T09:10:43.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=48,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=16,mon_config_key=12} crc {logm=1182473682,mdsmap=3984425268,mgr=427508465,mgr_command_descs=3831600018,mgr_metadata=2702834764,mgrstat=248488440,mon_config_key=930068308}) 2024-03-20T09:10:43.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2488951626}) 2024-03-20T09:10:43.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1728954769}) 2024-03-20T09:10:43.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=66,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=16} crc {mon_config_key=837948195,monmap=3481182261,osd_metadata=1898051788,osd_pg_creating=4016002691,osdmap=1862387825}) 2024-03-20T09:10:43.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1953344786}) 2024-03-20T09:10:43.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1232679788}) 2024-03-20T09:10:43.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3286485756}) 2024-03-20T09:10:43.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=84} crc {osdmap=3922860416}) 2024-03-20T09:10:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: pgmap v1072: 57 pgs: 1 active+undersized+degraded+remapped, 1 active+recovering+remapped, 1 remapped+peering, 1 peering, 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 43 active+clean, 5 active+recovering+undersized+remapped; 3.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 0 op/s; 15583/168045 objects degraded (9.273%); 27 MiB/s, 1.61k objects/s recovering 2024-03-20T09:10:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: osdmap e450: 12 total, 12 up, 10 in 2024-03-20T09:10:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {auth=31,config=2,health=12,kv=55} crc {auth=2764950393,config=3599132250,health=1132862079,kv=2220733292}) 2024-03-20T09:10:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=869441548,logm=3597946349}) 2024-03-20T09:10:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=615551518}) 2024-03-20T09:10:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=944649251}) 2024-03-20T09:10:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1631270514}) 2024-03-20T09:10:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2639833894}) 2024-03-20T09:10:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3146564972}) 2024-03-20T09:10:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4195593963}) 2024-03-20T09:10:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=135680167}) 2024-03-20T09:10:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=573786874}) 2024-03-20T09:10:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1443144167}) 2024-03-20T09:10:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2951652205}) 2024-03-20T09:10:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3558909144}) 2024-03-20T09:10:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2119656565}) 2024-03-20T09:10:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3335057020}) 2024-03-20T09:10:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3968236221}) 2024-03-20T09:10:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3482803010}) 2024-03-20T09:10:43.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2449953336}) 2024-03-20T09:10:43.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4101471215}) 2024-03-20T09:10:43.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1246627036}) 2024-03-20T09:10:43.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3750522635}) 2024-03-20T09:10:43.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2455325710}) 2024-03-20T09:10:43.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3202381441}) 2024-03-20T09:10:43.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2888363655}) 2024-03-20T09:10:43.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3500283637}) 2024-03-20T09:10:43.452 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4071316231}) 2024-03-20T09:10:43.452 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=660166270}) 2024-03-20T09:10:43.452 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3812600652}) 2024-03-20T09:10:43.452 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3403424939}) 2024-03-20T09:10:43.452 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1638447558}) 2024-03-20T09:10:43.452 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=155806089}) 2024-03-20T09:10:43.452 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3495521351}) 2024-03-20T09:10:43.452 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1970886986}) 2024-03-20T09:10:43.453 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1787527661}) 2024-03-20T09:10:43.453 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3033714217}) 2024-03-20T09:10:43.453 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2344273786}) 2024-03-20T09:10:43.453 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2253571105}) 2024-03-20T09:10:43.453 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3993672329}) 2024-03-20T09:10:43.453 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2626070479}) 2024-03-20T09:10:43.454 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4273892222}) 2024-03-20T09:10:43.454 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=48,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=16,mon_config_key=12} crc {logm=1182473682,mdsmap=3984425268,mgr=427508465,mgr_command_descs=3831600018,mgr_metadata=2702834764,mgrstat=248488440,mon_config_key=930068308}) 2024-03-20T09:10:43.454 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2488951626}) 2024-03-20T09:10:43.454 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1728954769}) 2024-03-20T09:10:43.454 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=66,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=16} crc {mon_config_key=837948195,monmap=3481182261,osd_metadata=1898051788,osd_pg_creating=4016002691,osdmap=1862387825}) 2024-03-20T09:10:43.455 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1953344786}) 2024-03-20T09:10:43.455 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1232679788}) 2024-03-20T09:10:43.455 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3286485756}) 2024-03-20T09:10:43.455 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=84} crc {osdmap=3922860416}) 2024-03-20T09:10:43.489 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:10:43.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:43.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:10:43.595 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:10:43.626 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:43.769 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-20T09:10:44.121 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:44 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:10:44.121 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:44 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:10:44.121 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:44 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:10:44.121 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:44 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:10:44.285 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:10:44.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:44.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:10:44.389 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:10:44.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:44 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:10:44.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:44 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:10:44.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:44 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:10:44.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:44 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:10:44.414 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:44 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:10:44.414 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:44 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:10:44.414 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:44 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:10:44.414 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:44 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:10:44.416 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:44.435 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:10:44.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:44.438 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:10:44.539 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:10:44.573 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:44.980 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:10:44.981 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:10:44.981 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:10:44.981 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:10:44.981 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:10:44.981 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:10:44.981 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:10:44.981 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:10:44.981 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:10:44.981 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:10:44.981 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:10:44.981 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:10:44.982 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:10:44.982 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:10:44.982 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:10:44.982 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:10:44.982 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:10:44.982 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:10:44.982 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:10:44.982 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:10:44.982 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:10:44.983 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:10:44.983 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:10:44.983 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:10:45.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:45 smithi115 ceph-mon[72476]: osdmap e451: 12 total, 12 up, 10 in 2024-03-20T09:10:45.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:45 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2767777175' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:10:45.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:45 smithi122 ceph-mon[73495]: osdmap e451: 12 total, 12 up, 10 in 2024-03-20T09:10:45.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:45 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2767777175' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:10:45.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:45 smithi026 ceph-mon[68309]: osdmap e451: 12 total, 12 up, 10 in 2024-03-20T09:10:45.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:45 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2767777175' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:10:45.517 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:10:45.517 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:10:46.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:46 smithi115 ceph-mon[72476]: osdmap e452: 12 total, 12 up, 10 in 2024-03-20T09:10:46.426 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:10:46.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:46 smithi122 ceph-mon[73495]: osdmap e452: 12 total, 12 up, 10 in 2024-03-20T09:10:46.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:46 smithi026 ceph-mon[68309]: osdmap e452: 12 total, 12 up, 10 in 2024-03-20T09:10:46.528 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:10:46.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:46.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:10:46.633 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:10:46.672 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:47.101 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 0 1 2024-03-20T09:10:47.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:47 smithi115 ceph-mon[72476]: pgmap v1077: 57 pgs: 1 active+recovering+remapped, 2 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean, 3 active+recovering+undersized+remapped; 3.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 137 op/s; 7605/166164 objects degraded (4.577%); 4521/166164 objects misplaced (2.721%); 568 KiB/s, 954 objects/s recovering 2024-03-20T09:10:47.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:47 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 7605/166164 objects degraded (4.577%), 4 pgs degraded (PG_DEGRADED) 2024-03-20T09:10:47.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:47 smithi115 ceph-mon[72476]: osdmap e453: 12 total, 12 up, 10 in 2024-03-20T09:10:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:47 smithi122 ceph-mon[73495]: pgmap v1077: 57 pgs: 1 active+recovering+remapped, 2 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean, 3 active+recovering+undersized+remapped; 3.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 137 op/s; 7605/166164 objects degraded (4.577%); 4521/166164 objects misplaced (2.721%); 568 KiB/s, 954 objects/s recovering 2024-03-20T09:10:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:47 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 7605/166164 objects degraded (4.577%), 4 pgs degraded (PG_DEGRADED) 2024-03-20T09:10:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:47 smithi122 ceph-mon[73495]: osdmap e453: 12 total, 12 up, 10 in 2024-03-20T09:10:47.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:47 smithi026 ceph-mon[68309]: pgmap v1077: 57 pgs: 1 active+recovering+remapped, 2 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean, 3 active+recovering+undersized+remapped; 3.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 137 op/s; 7605/166164 objects degraded (4.577%); 4521/166164 objects misplaced (2.721%); 568 KiB/s, 954 objects/s recovering 2024-03-20T09:10:47.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:47 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 7605/166164 objects degraded (4.577%), 4 pgs degraded (PG_DEGRADED) 2024-03-20T09:10:47.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:47 smithi026 ceph-mon[68309]: osdmap e453: 12 total, 12 up, 10 in 2024-03-20T09:10:49.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:49 smithi115 ceph-mon[72476]: pgmap v1079: 57 pgs: 1 active+recovering+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 48 active+clean, 3 active+recovering+undersized+remapped; 3.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 137 op/s; 7605/165732 objects degraded (4.589%); 4521/165732 objects misplaced (2.728%); 467 KiB/s, 661 objects/s recovering 2024-03-20T09:10:49.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:49 smithi122 ceph-mon[73495]: pgmap v1079: 57 pgs: 1 active+recovering+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 48 active+clean, 3 active+recovering+undersized+remapped; 3.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 137 op/s; 7605/165732 objects degraded (4.589%); 4521/165732 objects misplaced (2.728%); 467 KiB/s, 661 objects/s recovering 2024-03-20T09:10:49.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:49 smithi026 ceph-mon[68309]: pgmap v1079: 57 pgs: 1 active+recovering+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 48 active+clean, 3 active+recovering+undersized+remapped; 3.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 137 op/s; 7605/165732 objects degraded (4.589%); 4521/165732 objects misplaced (2.728%); 467 KiB/s, 661 objects/s recovering 2024-03-20T09:10:50.228 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:10:50.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:50.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:10:50.333 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:10:50.360 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:50.378 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:10:50.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:50.381 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:10:50.482 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:10:50.534 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:50.568 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:10:50.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:50.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:10:50.680 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:10:50.718 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:51.074 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:10:51.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:51.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:10:51.181 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:10:51.207 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:51.229 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:10:51.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:51.231 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:10:51.332 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:10:51.365 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:51.389 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:10:51.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:51.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:10:51.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:51 smithi115 ceph-mon[72476]: pgmap v1080: 57 pgs: 1 active+recovering+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 48 active+clean, 3 active+recovering+undersized+remapped; 3.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 149 op/s; 7605/165315 objects degraded (4.600%); 4521/165315 objects misplaced (2.735%); 437 KiB/s, 619 objects/s recovering 2024-03-20T09:10:51.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:51 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:10:51.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:51 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:10:51.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:51 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:10:51.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:51 smithi122 ceph-mon[73495]: pgmap v1080: 57 pgs: 1 active+recovering+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 48 active+clean, 3 active+recovering+undersized+remapped; 3.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 149 op/s; 7605/165315 objects degraded (4.600%); 4521/165315 objects misplaced (2.735%); 437 KiB/s, 619 objects/s recovering 2024-03-20T09:10:51.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:51 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:10:51.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:51 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:10:51.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:51 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:10:51.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:51 smithi026 ceph-mon[68309]: pgmap v1080: 57 pgs: 1 active+recovering+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 48 active+clean, 3 active+recovering+undersized+remapped; 3.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 149 op/s; 7605/165315 objects degraded (4.600%); 4521/165315 objects misplaced (2.735%); 437 KiB/s, 619 objects/s recovering 2024-03-20T09:10:51.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:51 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:10:51.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:51 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:10:51.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:51 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:10:51.496 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:10:51.527 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:52.284 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:10:52.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:52 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 7605/165315 objects degraded (4.600%), 4 pgs degraded (PG_DEGRADED) 2024-03-20T09:10:52.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:52 smithi115 ceph-mon[72476]: osdmap e454: 12 total, 12 up, 10 in 2024-03-20T09:10:52.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:52 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 7605/165315 objects degraded (4.600%), 4 pgs degraded (PG_DEGRADED) 2024-03-20T09:10:52.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:52 smithi122 ceph-mon[73495]: osdmap e454: 12 total, 12 up, 10 in 2024-03-20T09:10:52.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:52 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 7605/165315 objects degraded (4.600%), 4 pgs degraded (PG_DEGRADED) 2024-03-20T09:10:52.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:52 smithi026 ceph-mon[68309]: osdmap e454: 12 total, 12 up, 10 in 2024-03-20T09:10:53.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:53 smithi115 ceph-mon[72476]: pgmap v1081: 57 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 49 active+clean, 3 active+recovering+undersized+remapped; 3.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 223 op/s; 7632/162906 objects degraded (4.685%); 4440/162906 objects misplaced (2.725%); 807 KiB/s, 840 objects/s recovering 2024-03-20T09:10:53.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:53 smithi115 ceph-mon[72476]: osdmap e455: 12 total, 12 up, 10 in 2024-03-20T09:10:53.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:53 smithi122 ceph-mon[73495]: pgmap v1081: 57 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 49 active+clean, 3 active+recovering+undersized+remapped; 3.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 223 op/s; 7632/162906 objects degraded (4.685%); 4440/162906 objects misplaced (2.725%); 807 KiB/s, 840 objects/s recovering 2024-03-20T09:10:53.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:53 smithi122 ceph-mon[73495]: osdmap e455: 12 total, 12 up, 10 in 2024-03-20T09:10:53.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:53 smithi026 ceph-mon[68309]: pgmap v1081: 57 pgs: 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 49 active+clean, 3 active+recovering+undersized+remapped; 3.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 223 op/s; 7632/162906 objects degraded (4.685%); 4440/162906 objects misplaced (2.725%); 807 KiB/s, 840 objects/s recovering 2024-03-20T09:10:53.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:53 smithi026 ceph-mon[68309]: osdmap e455: 12 total, 12 up, 10 in 2024-03-20T09:10:53.949 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:10:53.964 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:10:54.105 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:10:54.112 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:10:54.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:54.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:10:54.216 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:10:54.244 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:54.264 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:10:54.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:54.267 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:10:54.369 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:10:54.398 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:54.420 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:10:54.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:54.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:10:54.524 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:10:54.554 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:54.584 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:10:54.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:54.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:10:54.691 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:10:54.728 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:55.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:55 smithi122 ceph-mon[73495]: pgmap v1084: 57 pgs: 1 peering, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 49 active+clean, 2 active+recovering+undersized+remapped; 3.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 156 op/s; 7632/162696 objects degraded (4.691%); 4440/162696 objects misplaced (2.729%); 523 KiB/s, 442 objects/s recovering 2024-03-20T09:10:55.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:55 smithi026 ceph-mon[68309]: pgmap v1084: 57 pgs: 1 peering, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 49 active+clean, 2 active+recovering+undersized+remapped; 3.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 156 op/s; 7632/162696 objects degraded (4.691%); 4440/162696 objects misplaced (2.729%); 523 KiB/s, 442 objects/s recovering 2024-03-20T09:10:55.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:55 smithi115 ceph-mon[72476]: pgmap v1084: 57 pgs: 1 peering, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 49 active+clean, 2 active+recovering+undersized+remapped; 3.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 156 op/s; 7632/162696 objects degraded (4.691%); 4440/162696 objects misplaced (2.729%); 523 KiB/s, 442 objects/s recovering 2024-03-20T09:10:55.990 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:10:55.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:55.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:10:56.094 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:10:56.122 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:56.142 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:10:56.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:56.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:10:56.246 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:10:56.273 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:56.294 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:10:56.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:56.297 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:10:56.398 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:10:56.424 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:56.444 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:10:56.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:56.447 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:10:56.548 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:10:56.577 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:56.599 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:10:56.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:56.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:10:56.703 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:10:56.729 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:56.748 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:10:56.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:56.752 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:10:56.853 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:10:56.882 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:56.904 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:10:56.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:56.907 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:10:57.008 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:10:57.049 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:57.085 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:10:57.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:10:57.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:10:57.197 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:10:57.228 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:10:57.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:57 smithi122 ceph-mon[73495]: pgmap v1085: 57 pgs: 1 active+recovering+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 49 active+clean, 2 active+recovering+undersized+remapped; 3.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 220 op/s; 6349/160434 objects degraded (3.957%); 3724/160434 objects misplaced (2.321%); 904 KiB/s, 774 objects/s recovering 2024-03-20T09:10:57.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:57 smithi122 ceph-mon[73495]: osdmap e456: 12 total, 12 up, 10 in 2024-03-20T09:10:57.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:57 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:10:57.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:57 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:10:57.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:57 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 6349/160434 objects degraded (3.957%), 3 pgs degraded (PG_DEGRADED) 2024-03-20T09:10:57.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:57 smithi122 ceph-mon[73495]: osdmap e457: 12 total, 12 up, 10 in 2024-03-20T09:10:57.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:57 smithi026 ceph-mon[68309]: pgmap v1085: 57 pgs: 1 active+recovering+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 49 active+clean, 2 active+recovering+undersized+remapped; 3.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 220 op/s; 6349/160434 objects degraded (3.957%); 3724/160434 objects misplaced (2.321%); 904 KiB/s, 774 objects/s recovering 2024-03-20T09:10:57.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:57 smithi026 ceph-mon[68309]: osdmap e456: 12 total, 12 up, 10 in 2024-03-20T09:10:57.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:57 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:10:57.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:57 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:10:57.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:57 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 6349/160434 objects degraded (3.957%), 3 pgs degraded (PG_DEGRADED) 2024-03-20T09:10:57.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:57 smithi026 ceph-mon[68309]: osdmap e457: 12 total, 12 up, 10 in 2024-03-20T09:10:57.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:57 smithi115 ceph-mon[72476]: pgmap v1085: 57 pgs: 1 active+recovering+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 49 active+clean, 2 active+recovering+undersized+remapped; 3.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 220 op/s; 6349/160434 objects degraded (3.957%); 3724/160434 objects misplaced (2.321%); 904 KiB/s, 774 objects/s recovering 2024-03-20T09:10:57.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:57 smithi115 ceph-mon[72476]: osdmap e456: 12 total, 12 up, 10 in 2024-03-20T09:10:57.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:57 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:10:57.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:57 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:10:57.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:57 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 6349/160434 objects degraded (3.957%), 3 pgs degraded (PG_DEGRADED) 2024-03-20T09:10:57.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:57 smithi115 ceph-mon[72476]: osdmap e457: 12 total, 12 up, 10 in 2024-03-20T09:10:59.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:59 smithi115 ceph-mon[72476]: pgmap v1088: 57 pgs: 1 active+recovering+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 50 active+clean, 1 active+recovering+undersized+remapped; 3.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 189 op/s; 6349/159651 objects degraded (3.977%); 3724/159651 objects misplaced (2.333%); 647 KiB/s, 705 objects/s recovering 2024-03-20T09:10:59.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:10:59 smithi115 ceph-mon[72476]: osdmap e458: 12 total, 12 up, 10 in 2024-03-20T09:10:59.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:59 smithi122 ceph-mon[73495]: pgmap v1088: 57 pgs: 1 active+recovering+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 50 active+clean, 1 active+recovering+undersized+remapped; 3.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 189 op/s; 6349/159651 objects degraded (3.977%); 3724/159651 objects misplaced (2.333%); 647 KiB/s, 705 objects/s recovering 2024-03-20T09:10:59.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:10:59 smithi122 ceph-mon[73495]: osdmap e458: 12 total, 12 up, 10 in 2024-03-20T09:10:59.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:59 smithi026 ceph-mon[68309]: pgmap v1088: 57 pgs: 1 active+recovering+remapped, 1 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 50 active+clean, 1 active+recovering+undersized+remapped; 3.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 189 op/s; 6349/159651 objects degraded (3.977%); 3724/159651 objects misplaced (2.333%); 647 KiB/s, 705 objects/s recovering 2024-03-20T09:10:59.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:10:59 smithi026 ceph-mon[68309]: osdmap e458: 12 total, 12 up, 10 in 2024-03-20T09:11:00.153 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:11:00.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:00.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:11:00.258 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:11:00.287 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:00.318 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:11:00.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:00.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:11:00.423 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:11:00.458 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:01.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:01 smithi115 ceph-mon[72476]: pgmap v1090: 57 pgs: 1 active+recovering+remapped, 1 peering, 2 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 50 active+clean, 1 active+recovering+undersized+remapped; 3.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 199 op/s; 3250/159099 objects degraded (2.043%); 4798/159099 objects misplaced (3.016%); 7.1 MiB/s, 728 objects/s recovering 2024-03-20T09:11:01.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:01 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:11:01.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:01 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:11:01.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:01 smithi122 ceph-mon[73495]: pgmap v1090: 57 pgs: 1 active+recovering+remapped, 1 peering, 2 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 50 active+clean, 1 active+recovering+undersized+remapped; 3.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 199 op/s; 3250/159099 objects degraded (2.043%); 4798/159099 objects misplaced (3.016%); 7.1 MiB/s, 728 objects/s recovering 2024-03-20T09:11:01.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:01 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:11:01.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:01 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:11:01.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:01 smithi026 ceph-mon[68309]: pgmap v1090: 57 pgs: 1 active+recovering+remapped, 1 peering, 2 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 50 active+clean, 1 active+recovering+undersized+remapped; 3.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 199 op/s; 3250/159099 objects degraded (2.043%); 4798/159099 objects misplaced (3.016%); 7.1 MiB/s, 728 objects/s recovering 2024-03-20T09:11:01.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:01 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:11:01.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:01 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:11:02.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:02 smithi115 ceph-mon[72476]: osdmap e459: 12 total, 12 up, 10 in 2024-03-20T09:11:02.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:02 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 3250/159099 objects degraded (2.043%), 3 pgs degraded (PG_DEGRADED) 2024-03-20T09:11:02.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:02 smithi122 ceph-mon[73495]: osdmap e459: 12 total, 12 up, 10 in 2024-03-20T09:11:02.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:02 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 3250/159099 objects degraded (2.043%), 3 pgs degraded (PG_DEGRADED) 2024-03-20T09:11:02.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:02 smithi026 ceph-mon[68309]: osdmap e459: 12 total, 12 up, 10 in 2024-03-20T09:11:02.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:02 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 3250/159099 objects degraded (2.043%), 3 pgs degraded (PG_DEGRADED) 2024-03-20T09:11:03.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:03 smithi115 ceph-mon[72476]: pgmap v1092: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 peering, 3 active+recovering+undersized+degraded+remapped, 51 active+clean; 3.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 226 op/s; 306/156645 objects degraded (0.195%); 5221/156645 objects misplaced (3.333%); 7.7 MiB/s, 460 objects/s recovering 2024-03-20T09:11:03.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:03 smithi115 ceph-mon[72476]: osdmap e460: 12 total, 12 up, 10 in 2024-03-20T09:11:03.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:03 smithi122 ceph-mon[73495]: pgmap v1092: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 peering, 3 active+recovering+undersized+degraded+remapped, 51 active+clean; 3.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 226 op/s; 306/156645 objects degraded (0.195%); 5221/156645 objects misplaced (3.333%); 7.7 MiB/s, 460 objects/s recovering 2024-03-20T09:11:03.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:03 smithi122 ceph-mon[73495]: osdmap e460: 12 total, 12 up, 10 in 2024-03-20T09:11:03.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:03 smithi026 ceph-mon[68309]: pgmap v1092: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 peering, 3 active+recovering+undersized+degraded+remapped, 51 active+clean; 3.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 226 op/s; 306/156645 objects degraded (0.195%); 5221/156645 objects misplaced (3.333%); 7.7 MiB/s, 460 objects/s recovering 2024-03-20T09:11:03.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:03 smithi026 ceph-mon[68309]: osdmap e460: 12 total, 12 up, 10 in 2024-03-20T09:11:03.815 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:11:03.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:03.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:11:03.920 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:11:03.947 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:03.968 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:11:03.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:03.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:11:04.072 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:11:04.101 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:04.124 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:11:04.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:04.128 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:11:04.229 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:11:04.259 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:04.713 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:11:04.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:04.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:11:04.817 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:11:04.845 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:04.870 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:11:04.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:04.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:11:04.972 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:11:05.000 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:05.019 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:11:05.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:05.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:11:05.122 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:11:05.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:05 smithi115 ceph-mon[72476]: pgmap v1094: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 53 active+clean; 3.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 206 op/s; 206/155940 objects degraded (0.132%); 5221/155940 objects misplaced (3.348%); 7.2 MiB/s, 382 objects/s recovering 2024-03-20T09:11:05.149 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:05.169 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:11:05.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:05.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:11:05.273 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:11:05.298 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:05 smithi026 ceph-mon[68309]: pgmap v1094: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 53 active+clean; 3.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 206 op/s; 206/155940 objects degraded (0.132%); 5221/155940 objects misplaced (3.348%); 7.2 MiB/s, 382 objects/s recovering 2024-03-20T09:11:05.307 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:05.388 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:11:05.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:05.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:11:05.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:05 smithi122 ceph-mon[73495]: pgmap v1094: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 53 active+clean; 3.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 206 op/s; 206/155940 objects degraded (0.132%); 5221/155940 objects misplaced (3.348%); 7.2 MiB/s, 382 objects/s recovering 2024-03-20T09:11:05.492 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:11:05.519 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:05.539 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:11:05.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:05.541 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:11:05.642 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:11:05.669 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:05.690 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:11:05.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:05.694 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:11:05.795 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:11:05.822 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:05.844 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:11:05.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:05.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:11:05.947 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:11:05.981 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:06.531 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:11:06.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:06.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:11:06.636 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:11:06.661 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:06.681 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:11:06.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:06.684 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:11:06.784 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:11:06.811 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:06.830 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:11:06.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:06.834 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:11:06.935 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:11:06.962 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:06.983 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:11:06.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:06.988 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:11:07.089 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:11:07.118 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:07.139 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:11:07.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:07.142 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:11:07.243 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:11:07.269 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:07 smithi122 ceph-mon[73495]: pgmap v1095: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 53 active+clean; 3.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 201 op/s; 206/154968 objects degraded (0.133%); 4147/154968 objects misplaced (2.676%); 14 MiB/s, 437 objects/s recovering 2024-03-20T09:11:07.269 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:07 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 206/154968 objects degraded (0.133%), 2 pgs degraded (PG_DEGRADED) 2024-03-20T09:11:07.269 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:07 smithi122 ceph-mon[73495]: osdmap e461: 12 total, 12 up, 10 in 2024-03-20T09:11:07.271 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:07.289 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:07 smithi026 ceph-mon[68309]: pgmap v1095: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 53 active+clean; 3.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 201 op/s; 206/154968 objects degraded (0.133%); 4147/154968 objects misplaced (2.676%); 14 MiB/s, 437 objects/s recovering 2024-03-20T09:11:07.289 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:07 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 206/154968 objects degraded (0.133%), 2 pgs degraded (PG_DEGRADED) 2024-03-20T09:11:07.290 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:07 smithi026 ceph-mon[68309]: osdmap e461: 12 total, 12 up, 10 in 2024-03-20T09:11:07.291 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:11:07.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:07.294 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:11:07.395 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:11:07.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:07 smithi115 ceph-mon[72476]: pgmap v1095: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 53 active+clean; 3.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 201 op/s; 206/154968 objects degraded (0.133%); 4147/154968 objects misplaced (2.676%); 14 MiB/s, 437 objects/s recovering 2024-03-20T09:11:07.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:07 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 206/154968 objects degraded (0.133%), 2 pgs degraded (PG_DEGRADED) 2024-03-20T09:11:07.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:07 smithi115 ceph-mon[72476]: osdmap e461: 12 total, 12 up, 10 in 2024-03-20T09:11:07.423 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:07.443 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:11:07.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:07.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:11:07.547 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:11:07.574 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:07.595 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:11:07.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:07.597 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:11:07.698 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:11:07.731 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:08.365 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:11:08.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:08.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:11:08.474 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:11:08.500 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:08.521 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:11:08.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:08.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:11:08.625 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:11:08.652 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:08.673 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:11:08.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:08.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:11:08.776 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:11:08.802 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:08.821 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:11:08.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:08.824 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:11:08.866 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:11:08.906 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:11:08.924 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:11:08.937 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:11:08.951 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:08.963 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:11:08.970 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:11:08.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:08.973 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:11:08.985 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:11:09.006 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:11:09.027 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:11:09.047 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.0 to 1 (10000) 2024-03-20T09:11:09.058 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:11:09.073 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:11:09.088 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:11:09.098 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:09 smithi122 ceph-mon[73495]: pgmap v1097: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 53 active+clean; 3.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 178 op/s; 235/153534 objects degraded (0.153%); 4050/153534 objects misplaced (2.638%); 11 MiB/s, 216 objects/s recovering 2024-03-20T09:11:09.098 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:09 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/912621315' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-20T09:11:09.098 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:09 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1422911259' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:11:09.098 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:09 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/221759183' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:11:09.100 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:09.105 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:11:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:11:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-03-20T09:11:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-03-20T09:11:09.106 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:11:09.118 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:11:09.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:09.121 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:11:09.123 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:11:09.160 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:11:09.183 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:11:09.193 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:11:09.193 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:11:09.193 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:11:09.193 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:11:09.193 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:11:09.193 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:11:09.193 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:11:09.193 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:11:09.193 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:11:09.193 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:11:09.193 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:11:09.193 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:11:09.201 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:11:09.222 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:11:09.250 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:09.268 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:11:09.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:09.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:11:09.372 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:11:09.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:09 smithi115 ceph-mon[72476]: pgmap v1097: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 53 active+clean; 3.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 178 op/s; 235/153534 objects degraded (0.153%); 4050/153534 objects misplaced (2.638%); 11 MiB/s, 216 objects/s recovering 2024-03-20T09:11:09.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:09 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/912621315' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-20T09:11:09.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:09 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1422911259' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:11:09.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:09 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/221759183' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:11:09.399 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:09.418 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:11:09.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:09.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:11:09.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:09 smithi026 ceph-mon[68309]: pgmap v1097: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 53 active+clean; 3.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 178 op/s; 235/153534 objects degraded (0.153%); 4050/153534 objects misplaced (2.638%); 11 MiB/s, 216 objects/s recovering 2024-03-20T09:11:09.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:09 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/912621315' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-20T09:11:09.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:09 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1422911259' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:11:09.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:09 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/221759183' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:11:09.521 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:11:09.549 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:09.568 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:11:09.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:09.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:11:09.673 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:11:09.699 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:09.717 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:11:09.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:09.720 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:11:09.821 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:11:09.853 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:09.881 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:11:09.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:09.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:11:09.986 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:11:10.013 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:10.031 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:11:10.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:10.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:11:10.135 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:11:10.171 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:10 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:11:10.172 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:10 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/912621315' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-20T09:11:10.172 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:10 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/221759183' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:11:10.172 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:10 smithi026 ceph-mon[68309]: osdmap e462: 12 total, 12 up, 10 in 2024-03-20T09:11:10.172 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:10 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:11:10.172 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:10 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:11:10.174 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:10.202 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:11:10.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:10.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:11:10.309 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:11:10.333 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:10 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:11:10.333 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:10 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/912621315' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-20T09:11:10.333 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:10 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/221759183' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:11:10.333 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:10 smithi115 ceph-mon[72476]: osdmap e462: 12 total, 12 up, 10 in 2024-03-20T09:11:10.333 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:10 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:11:10.333 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:10 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:11:10.334 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:10.353 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:11:10.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:10.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:11:10.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:10 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:11:10.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:10 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/912621315' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-20T09:11:10.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:10 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/221759183' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:11:10.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:10 smithi122 ceph-mon[73495]: osdmap e462: 12 total, 12 up, 10 in 2024-03-20T09:11:10.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:10 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:11:10.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:10 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:11:10.457 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:11:10.485 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:10.612 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:11:10.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:10.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:11:10.651 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-20T09:11:10.717 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:11:10.744 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:10.766 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:11:10.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:10.769 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:11:10.870 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:11:10.897 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:10.916 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:11:10.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:10.919 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:11:11.020 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:11:11.046 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:11.065 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:11:11.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:11.067 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:11:11.168 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:11:11.195 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:11.213 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:11:11.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:11.215 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:11:11.310 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:11 smithi026 ceph-mon[68309]: pgmap v1099: 57 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 53 active+clean; 3.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 152 op/s; 154/153114 objects degraded (0.101%); 4050/153114 objects misplaced (2.645%); 9.2 MiB/s, 377 objects/s recovering 2024-03-20T09:11:11.311 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:11 smithi026 ceph-mon[68309]: osdmap e463: 12 total, 12 up, 10 in 2024-03-20T09:11:11.316 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:11:11.342 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:11 smithi122 ceph-mon[73495]: pgmap v1099: 57 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 53 active+clean; 3.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 152 op/s; 154/153114 objects degraded (0.101%); 4050/153114 objects misplaced (2.645%); 9.2 MiB/s, 377 objects/s recovering 2024-03-20T09:11:11.342 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:11 smithi122 ceph-mon[73495]: osdmap e463: 12 total, 12 up, 10 in 2024-03-20T09:11:11.344 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:11.363 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:11:11.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:11.366 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:11:11.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:11 smithi115 ceph-mon[72476]: pgmap v1099: 57 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 53 active+clean; 3.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 152 op/s; 154/153114 objects degraded (0.101%); 4050/153114 objects misplaced (2.645%); 9.2 MiB/s, 377 objects/s recovering 2024-03-20T09:11:11.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:11 smithi115 ceph-mon[72476]: osdmap e463: 12 total, 12 up, 10 in 2024-03-20T09:11:11.467 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:11:11.498 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:11.628 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:11:11.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:11.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:11:11.733 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:11:11.759 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:11.777 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:11:11.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:11.781 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:11:11.882 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:11:11.914 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:12.016 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:11:12.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:12.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:11:12.047 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 1 1 2024-03-20T09:11:12.124 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:11:12.163 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:12.169 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:11:12.194 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:11:12.196 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:12 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 154/153114 objects degraded (0.101%), 1 pg degraded (PG_DEGRADED) 2024-03-20T09:11:12.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:12.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:11:12.299 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:11:12.334 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:12.391 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:11:12.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:12.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:11:12.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:12 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 154/153114 objects degraded (0.101%), 1 pg degraded (PG_DEGRADED) 2024-03-20T09:11:12.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:12 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 154/153114 objects degraded (0.101%), 1 pg degraded (PG_DEGRADED) 2024-03-20T09:11:12.496 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:11:12.523 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:12.544 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:11:12.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:12.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:11:12.649 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:11:12.675 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:12.696 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:11:12.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:12.699 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:11:12.800 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:11:12.826 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:12.845 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:11:12.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:12.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:11:12.949 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:11:12.975 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:12.993 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:11:12.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:12.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:11:13.097 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:11:13.123 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:13.143 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:11:13.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:13.145 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:11:13.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:13 smithi115 ceph-mon[72476]: pgmap v1101: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 1 peering, 53 active+clean; 3.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 217 op/s; 183/151056 objects degraded (0.121%); 3957/151056 objects misplaced (2.620%); 1.4 MiB/s, 280 objects/s recovering 2024-03-20T09:11:13.182 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:11:13.246 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:11:13.273 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:13.291 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:11:13.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:13.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:11:13.356 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:13 smithi026 ceph-mon[68309]: pgmap v1101: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 1 peering, 53 active+clean; 3.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 217 op/s; 183/151056 objects degraded (0.121%); 3957/151056 objects misplaced (2.620%); 1.4 MiB/s, 280 objects/s recovering 2024-03-20T09:11:13.356 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:11:13.395 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:11:13.419 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:13 smithi122 ceph-mon[73495]: pgmap v1101: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 1 peering, 53 active+clean; 3.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 217 op/s; 183/151056 objects degraded (0.121%); 3957/151056 objects misplaced (2.620%); 1.4 MiB/s, 280 objects/s recovering 2024-03-20T09:11:13.421 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:13.440 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:11:13.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:13.443 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:11:13.544 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:11:13.570 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:13.590 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:11:13.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:13.593 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:11:13.640 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:11:13.693 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:11:13.720 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:13.740 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:11:13.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:13.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:11:13.844 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:11:13.871 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:13.890 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:11:13.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:13.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:11:13.993 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:11:14.019 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:14.038 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:11:14.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:14.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:11:14.142 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:11:14.167 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:14.186 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:11:14.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:14.189 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:11:14.244 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:11:14.244 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-03-20T09:11:14.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-03-20T09:11:14.245 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:11:14.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.245 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:132568 3.e 3:7c19ea7f:::benchmark_data_smithi122_119671_object32786:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e399)", 2024-03-20T09:11:14.245 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:08:51.875907+0000", 2024-03-20T09:11:14.245 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 142.364970798, 2024-03-20T09:11:14.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6843376939999999, 2024-03-20T09:11:14.245 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:14.245 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:11:14.245 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:14.245 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:14.245 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:14.245 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 132568 2024-03-20T09:11:14.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.245 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:14.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:14.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:51.875907+0000", 2024-03-20T09:11:14.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:14.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:51.875907+0000", 2024-03-20T09:11:14.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:14.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:51.875909+0000", 2024-03-20T09:11:14.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7209999999999999e-06 2024-03-20T09:11:14.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:14.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:51.875917+0000", 2024-03-20T09:11:14.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1589999999999993e-06 2024-03-20T09:11:14.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:14.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:51.875936+0000", 2024-03-20T09:11:14.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9437999999999999e-05 2024-03-20T09:11:14.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:14.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:51.875948+0000", 2024-03-20T09:11:14.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1211e-05 2024-03-20T09:11:14.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:52.452365+0000", 2024-03-20T09:11:14.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.57641696899999995 2024-03-20T09:11:14.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:11:14.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:52.455279+0000", 2024-03-20T09:11:14.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0029148199999999998 2024-03-20T09:11:14.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:52.908935+0000", 2024-03-20T09:11:14.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.45365597600000002 2024-03-20T09:11:14.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-03-20T09:11:14.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:52.915560+0000", 2024-03-20T09:11:14.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0066243020000000003 2024-03-20T09:11:14.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:54.559156+0000", 2024-03-20T09:11:14.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.643596608 2024-03-20T09:11:14.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:14.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:54.560245+0000", 2024-03-20T09:11:14.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010884899999999999 2024-03-20T09:11:14.251 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.251 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:14.251 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.251 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:133150 3.e 3:7cd88cf8:::benchmark_data_smithi122_119671_object33368:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-03-20T09:11:14.251 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:08:55.406450+0000", 2024-03-20T09:11:14.251 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 138.834428153, 2024-03-20T09:11:14.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5182847509999999, 2024-03-20T09:11:14.251 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:14.251 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:11:14.251 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:14.251 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:14.252 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:14.252 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 133150 2024-03-20T09:11:14.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.252 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:14.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:14.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:55.406450+0000", 2024-03-20T09:11:14.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:14.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:55.406450+0000", 2024-03-20T09:11:14.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:14.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:55.406452+0000", 2024-03-20T09:11:14.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9520000000000001e-06 2024-03-20T09:11:14.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:14.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:55.406461+0000", 2024-03-20T09:11:14.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2210000000000003e-06 2024-03-20T09:11:14.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:14.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:55.406479+0000", 2024-03-20T09:11:14.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7847e-05 2024-03-20T09:11:14.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:14.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:55.406491+0000", 2024-03-20T09:11:14.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2422999999999999e-05 2024-03-20T09:11:14.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:56.023286+0000", 2024-03-20T09:11:14.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.61679463000000001 2024-03-20T09:11:14.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:11:14.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:56.029445+0000", 2024-03-20T09:11:14.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0061590999999999998 2024-03-20T09:11:14.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:56.136964+0000", 2024-03-20T09:11:14.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.107518851 2024-03-20T09:11:14.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-03-20T09:11:14.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:56.143300+0000", 2024-03-20T09:11:14.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0063357079999999998 2024-03-20T09:11:14.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:57.762618+0000", 2024-03-20T09:11:14.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6193179899999999 2024-03-20T09:11:14.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:11:14.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:57.768549+0000", 2024-03-20T09:11:14.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059316070000000002 2024-03-20T09:11:14.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 8,10", 2024-03-20T09:11:14.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:57.777535+0000", 2024-03-20T09:11:14.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0089858079999999996 2024-03-20T09:11:14.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:11:14.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:57.821444+0000", 2024-03-20T09:11:14.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.043908833000000001 2024-03-20T09:11:14.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:14.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:57.844070+0000", 2024-03-20T09:11:14.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022625782000000001 2024-03-20T09:11:14.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:14.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:57.903682+0000", 2024-03-20T09:11:14.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.059612689000000003 2024-03-20T09:11:14.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:11:14.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:57.908240+0000", 2024-03-20T09:11:14.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004557518 2024-03-20T09:11:14.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:14.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:57.924735+0000", 2024-03-20T09:11:14.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016494792000000001 2024-03-20T09:11:14.259 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.259 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:14.259 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.259 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:133169 3.e 3:73dae553:::benchmark_data_smithi122_119671_object33387:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-03-20T09:11:14.259 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:08:55.718396+0000", 2024-03-20T09:11:14.259 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 138.52248184499999, 2024-03-20T09:11:14.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7203790190000001, 2024-03-20T09:11:14.259 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:14.259 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:11:14.259 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:14.259 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:14.259 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:14.260 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 133169 2024-03-20T09:11:14.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.260 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:14.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:14.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:55.718396+0000", 2024-03-20T09:11:14.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:14.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:55.718396+0000", 2024-03-20T09:11:14.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:14.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:55.718397+0000", 2024-03-20T09:11:14.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7000000000000003e-07 2024-03-20T09:11:14.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:14.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:55.718401+0000", 2024-03-20T09:11:14.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.636e-06 2024-03-20T09:11:14.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:14.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:55.718412+0000", 2024-03-20T09:11:14.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1445e-05 2024-03-20T09:11:14.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:14.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:55.718417+0000", 2024-03-20T09:11:14.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1390000000000003e-06 2024-03-20T09:11:14.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:56.150725+0000", 2024-03-20T09:11:14.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.43230735199999998 2024-03-20T09:11:14.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:11:14.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:56.156928+0000", 2024-03-20T09:11:14.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0062037899999999998 2024-03-20T09:11:14.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:56.273733+0000", 2024-03-20T09:11:14.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.11680410200000001 2024-03-20T09:11:14.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-03-20T09:11:14.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:56.279750+0000", 2024-03-20T09:11:14.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0060171419999999996 2024-03-20T09:11:14.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:58.336585+0000", 2024-03-20T09:11:14.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0568356090000002 2024-03-20T09:11:14.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:11:14.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:58.341779+0000", 2024-03-20T09:11:14.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0051935649999999998 2024-03-20T09:11:14.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 8,10", 2024-03-20T09:11:14.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:58.350664+0000", 2024-03-20T09:11:14.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0088851899999999994 2024-03-20T09:11:14.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:11:14.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:58.412643+0000", 2024-03-20T09:11:14.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.061978640000000002 2024-03-20T09:11:14.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:14.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:58.414799+0000", 2024-03-20T09:11:14.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021557960000000002 2024-03-20T09:11:14.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:14.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:58.417060+0000", 2024-03-20T09:11:14.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0022614290000000001 2024-03-20T09:11:14.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:11:14.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:58.420888+0000", 2024-03-20T09:11:14.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0038281119999999998 2024-03-20T09:11:14.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:14.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:58.438775+0000", 2024-03-20T09:11:14.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017887101999999998 2024-03-20T09:11:14.266 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.266 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:14.266 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.266 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:133320 3.e 3:702a03d8:::benchmark_data_smithi122_119671_object33538:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e401)", 2024-03-20T09:11:14.266 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:08:57.398732+0000", 2024-03-20T09:11:14.266 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 136.84214552500001, 2024-03-20T09:11:14.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.43186861, 2024-03-20T09:11:14.267 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:14.267 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:11:14.267 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:14.267 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:14.267 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:14.267 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 133320 2024-03-20T09:11:14.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.267 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:14.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:14.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:57.398732+0000", 2024-03-20T09:11:14.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:14.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:57.398732+0000", 2024-03-20T09:11:14.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:14.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:57.398734+0000", 2024-03-20T09:11:14.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.142e-06 2024-03-20T09:11:14.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:14.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:57.398739+0000", 2024-03-20T09:11:14.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1220000000000003e-06 2024-03-20T09:11:14.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:14.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:57.398748+0000", 2024-03-20T09:11:14.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9390000000000003e-06 2024-03-20T09:11:14.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:14.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:57.398754+0000", 2024-03-20T09:11:14.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2670000000000003e-06 2024-03-20T09:11:14.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:57.454998+0000", 2024-03-20T09:11:14.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.056243687000000001 2024-03-20T09:11:14.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-03-20T09:11:14.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:57.461074+0000", 2024-03-20T09:11:14.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0060766830000000003 2024-03-20T09:11:14.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:59.829820+0000", 2024-03-20T09:11:14.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.368745455 2024-03-20T09:11:14.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:14.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:08:59.830601+0000", 2024-03-20T09:11:14.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078131499999999998 2024-03-20T09:11:14.271 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.271 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:14.271 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.271 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:134600 3.e 3:75516108:::benchmark_data_smithi122_119671_object34818:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e407)", 2024-03-20T09:11:14.271 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:09:10.073552+0000", 2024-03-20T09:11:14.271 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 124.167325846, 2024-03-20T09:11:14.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4112567939999998, 2024-03-20T09:11:14.271 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:14.272 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:11:14.272 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:14.272 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:14.272 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:14.272 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 134600 2024-03-20T09:11:14.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.272 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:14.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:14.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:10.073552+0000", 2024-03-20T09:11:14.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:14.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:10.073552+0000", 2024-03-20T09:11:14.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:14.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:10.073554+0000", 2024-03-20T09:11:14.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5820000000000001e-06 2024-03-20T09:11:14.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:14.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:10.073560+0000", 2024-03-20T09:11:14.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7329999999999997e-06 2024-03-20T09:11:14.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:14.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:10.073576+0000", 2024-03-20T09:11:14.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5506e-05 2024-03-20T09:11:14.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:14.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:10.073584+0000", 2024-03-20T09:11:14.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6879999999999999e-06 2024-03-20T09:11:14.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:10.332521+0000", 2024-03-20T09:11:14.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.25893775899999999 2024-03-20T09:11:14.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:11:14.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:10.338762+0000", 2024-03-20T09:11:14.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0062404610000000001 2024-03-20T09:11:14.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:10.413809+0000", 2024-03-20T09:11:14.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.075046897000000001 2024-03-20T09:11:14.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-03-20T09:11:14.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:10.422046+0000", 2024-03-20T09:11:14.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0082367959999999994 2024-03-20T09:11:14.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.427814+0000", 2024-03-20T09:11:14.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.005768116 2024-03-20T09:11:14.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:11:14.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.433205+0000", 2024-03-20T09:11:14.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0053911059999999997 2024-03-20T09:11:14.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 8,10", 2024-03-20T09:11:14.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.443299+0000", 2024-03-20T09:11:14.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010094324 2024-03-20T09:11:14.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:14.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.460387+0000", 2024-03-20T09:11:14.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017088335 2024-03-20T09:11:14.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:11:14.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.461597+0000", 2024-03-20T09:11:14.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012094219999999999 2024-03-20T09:11:14.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:14.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.463493+0000", 2024-03-20T09:11:14.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018966269999999999 2024-03-20T09:11:14.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:11:14.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.467623+0000", 2024-03-20T09:11:14.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0041296070000000004 2024-03-20T09:11:14.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:14.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.484809+0000", 2024-03-20T09:11:14.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017185835 2024-03-20T09:11:14.279 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.279 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:14.279 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.279 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:134615 3.e 3:7acdec1b:::benchmark_data_smithi122_119671_object34833:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e407)", 2024-03-20T09:11:14.279 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:09:10.280131+0000", 2024-03-20T09:11:14.279 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 123.96074686599999, 2024-03-20T09:11:14.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.407283402, 2024-03-20T09:11:14.279 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:14.279 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:11:14.279 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:14.279 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:14.279 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:14.279 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 134615 2024-03-20T09:11:14.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.280 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:14.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:14.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:10.280131+0000", 2024-03-20T09:11:14.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:14.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:10.280131+0000", 2024-03-20T09:11:14.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:14.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:10.280132+0000", 2024-03-20T09:11:14.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5600000000000004e-07 2024-03-20T09:11:14.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:14.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:10.280136+0000", 2024-03-20T09:11:14.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3660000000000002e-06 2024-03-20T09:11:14.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:14.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:10.280146+0000", 2024-03-20T09:11:14.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5759999999999995e-06 2024-03-20T09:11:14.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:14.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:10.280151+0000", 2024-03-20T09:11:14.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3029999999999999e-06 2024-03-20T09:11:14.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:10.424783+0000", 2024-03-20T09:11:14.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.14463155899999999 2024-03-20T09:11:14.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:11:14.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:10.431112+0000", 2024-03-20T09:11:14.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0063296819999999997 2024-03-20T09:11:14.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:10.480727+0000", 2024-03-20T09:11:14.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.049614453000000003 2024-03-20T09:11:14.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-03-20T09:11:14.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:10.495950+0000", 2024-03-20T09:11:14.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015222879999999999 2024-03-20T09:11:14.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.686667+0000", 2024-03-20T09:11:14.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.190717486 2024-03-20T09:11:14.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:14.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.687415+0000", 2024-03-20T09:11:14.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00074724100000000005 2024-03-20T09:11:14.284 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.284 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:14.284 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.284 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:134697 3.e 3:7e8f6492:::benchmark_data_smithi122_119671_object34915:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e407)", 2024-03-20T09:11:14.284 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:09:10.984429+0000", 2024-03-20T09:11:14.284 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 123.256448686, 2024-03-20T09:11:14.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.614600979, 2024-03-20T09:11:14.284 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:14.285 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:11:14.285 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:14.285 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:14.285 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:14.285 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 134697 2024-03-20T09:11:14.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.285 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:14.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:14.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:10.984429+0000", 2024-03-20T09:11:14.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:14.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:10.984429+0000", 2024-03-20T09:11:14.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:14.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:10.984430+0000", 2024-03-20T09:11:14.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0499999999999999e-06 2024-03-20T09:11:14.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:14.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:10.984435+0000", 2024-03-20T09:11:14.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8099999999999997e-06 2024-03-20T09:11:14.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:14.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:10.984447+0000", 2024-03-20T09:11:14.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1372000000000001e-05 2024-03-20T09:11:14.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:14.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:10.984453+0000", 2024-03-20T09:11:14.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9519999999999997e-06 2024-03-20T09:11:14.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.038150+0000", 2024-03-20T09:11:14.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.053696586999999997 2024-03-20T09:11:14.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:11:14.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.045225+0000", 2024-03-20T09:11:14.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0070748790000000001 2024-03-20T09:11:14.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.197225+0000", 2024-03-20T09:11:14.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.151999785 2024-03-20T09:11:14.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-03-20T09:11:14.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.203206+0000", 2024-03-20T09:11:14.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059810590000000004 2024-03-20T09:11:14.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.598401+0000", 2024-03-20T09:11:14.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3951947439999999 2024-03-20T09:11:14.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:14.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.599030+0000", 2024-03-20T09:11:14.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00062974099999999996 2024-03-20T09:11:14.289 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.289 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:14.289 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.290 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:134710 3.e 3:734c263f:::benchmark_data_smithi122_119671_object34928:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e407)", 2024-03-20T09:11:14.290 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:09:11.011306+0000", 2024-03-20T09:11:14.290 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 123.229571568, 2024-03-20T09:11:14.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.485949272, 2024-03-20T09:11:14.290 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:14.290 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:11:14.290 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:14.290 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:14.290 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:14.290 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 134710 2024-03-20T09:11:14.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.290 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:14.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:14.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.011306+0000", 2024-03-20T09:11:14.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:14.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.011306+0000", 2024-03-20T09:11:14.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:14.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.011308+0000", 2024-03-20T09:11:14.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.782e-06 2024-03-20T09:11:14.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:14.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.011316+0000", 2024-03-20T09:11:14.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4669999999999998e-06 2024-03-20T09:11:14.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:14.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.011332+0000", 2024-03-20T09:11:14.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6444e-05 2024-03-20T09:11:14.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:14.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.011341+0000", 2024-03-20T09:11:14.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2990000000000002e-06 2024-03-20T09:11:14.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.047802+0000", 2024-03-20T09:11:14.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.036460431000000001 2024-03-20T09:11:14.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:11:14.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.055404+0000", 2024-03-20T09:11:14.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0076017089999999999 2024-03-20T09:11:14.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.151675+0000", 2024-03-20T09:11:14.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.096271837999999998 2024-03-20T09:11:14.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-03-20T09:11:14.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.158088+0000", 2024-03-20T09:11:14.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0064125980000000003 2024-03-20T09:11:14.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.496657+0000", 2024-03-20T09:11:14.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3385689350000001 2024-03-20T09:11:14.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:14.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.497256+0000", 2024-03-20T09:11:14.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00059876899999999997 2024-03-20T09:11:14.295 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.295 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:14.295 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.295 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:134719 3.e 3:759e502d:::benchmark_data_smithi122_119671_object34937:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e407)", 2024-03-20T09:11:14.295 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:09:11.227752+0000", 2024-03-20T09:11:14.295 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 123.01312556000001, 2024-03-20T09:11:14.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5587362300000001, 2024-03-20T09:11:14.295 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:14.295 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:11:14.295 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:14.295 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:14.296 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:14.296 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 134719 2024-03-20T09:11:14.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.296 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:14.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:14.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.227752+0000", 2024-03-20T09:11:14.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:14.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.227752+0000", 2024-03-20T09:11:14.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:14.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.227754+0000", 2024-03-20T09:11:14.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.331e-06 2024-03-20T09:11:14.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:14.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.227760+0000", 2024-03-20T09:11:14.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2899999999999999e-06 2024-03-20T09:11:14.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:14.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.227773+0000", 2024-03-20T09:11:14.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2704000000000001e-05 2024-03-20T09:11:14.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:14.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.227780+0000", 2024-03-20T09:11:14.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1160000000000001e-06 2024-03-20T09:11:14.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.538203+0000", 2024-03-20T09:11:14.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.31042275000000003 2024-03-20T09:11:14.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:11:14.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.545210+0000", 2024-03-20T09:11:14.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0070070080000000003 2024-03-20T09:11:14.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.585445+0000", 2024-03-20T09:11:14.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.040235553 2024-03-20T09:11:14.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-03-20T09:11:14.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.592566+0000", 2024-03-20T09:11:14.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0071205230000000001 2024-03-20T09:11:14.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.785936+0000", 2024-03-20T09:11:14.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1933701860000001 2024-03-20T09:11:14.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:14.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.786489+0000", 2024-03-20T09:11:14.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00055276900000000005 2024-03-20T09:11:14.300 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.300 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:14.300 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.300 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:134769 3.e 3:7cb93753:::benchmark_data_smithi122_119671_object34987:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e407)", 2024-03-20T09:11:14.300 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:09:11.626109+0000", 2024-03-20T09:11:14.300 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 122.614768767, 2024-03-20T09:11:14.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4550553989999999, 2024-03-20T09:11:14.301 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:14.301 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:11:14.301 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:14.301 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:14.301 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:14.301 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 134769 2024-03-20T09:11:14.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.301 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:14.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.301 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:14.301 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.626109+0000", 2024-03-20T09:11:14.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.301 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:14.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.626109+0000", 2024-03-20T09:11:14.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:14.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.626111+0000", 2024-03-20T09:11:14.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.029e-06 2024-03-20T09:11:14.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:14.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.626120+0000", 2024-03-20T09:11:14.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4509999999999992e-06 2024-03-20T09:11:14.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:14.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.626139+0000", 2024-03-20T09:11:14.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9505999999999999e-05 2024-03-20T09:11:14.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:14.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.626150+0000", 2024-03-20T09:11:14.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0969e-05 2024-03-20T09:11:14.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.823864+0000", 2024-03-20T09:11:14.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.19771395 2024-03-20T09:11:14.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:11:14.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.831276+0000", 2024-03-20T09:11:14.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074114690000000004 2024-03-20T09:11:14.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.012671+0000", 2024-03-20T09:11:14.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.18139523799999999 2024-03-20T09:11:14.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-03-20T09:11:14.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.019629+0000", 2024-03-20T09:11:14.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0069581679999999998 2024-03-20T09:11:14.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:14.080553+0000", 2024-03-20T09:11:14.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0609242700000001 2024-03-20T09:11:14.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:14.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:14.081165+0000", 2024-03-20T09:11:14.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00061134899999999996 2024-03-20T09:11:14.305 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.305 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:14.305 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.305 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:134833 3.e 3:7ef92c8b:::benchmark_data_smithi122_119671_object35051:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e407)", 2024-03-20T09:11:14.305 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:09:11.901621+0000", 2024-03-20T09:11:14.305 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 122.339256542, 2024-03-20T09:11:14.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4138356139999999, 2024-03-20T09:11:14.306 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:14.306 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:11:14.306 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:14.306 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:14.306 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:14.306 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 134833 2024-03-20T09:11:14.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.306 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:14.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:14.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.901621+0000", 2024-03-20T09:11:14.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:14.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.901621+0000", 2024-03-20T09:11:14.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:14.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.901623+0000", 2024-03-20T09:11:14.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3069999999999999e-06 2024-03-20T09:11:14.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:14.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.901628+0000", 2024-03-20T09:11:14.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2499999999999997e-06 2024-03-20T09:11:14.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:14.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.901639+0000", 2024-03-20T09:11:14.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1489e-05 2024-03-20T09:11:14.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:14.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.901647+0000", 2024-03-20T09:11:14.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3329999999999999e-06 2024-03-20T09:11:14.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.030353+0000", 2024-03-20T09:11:14.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.128706656 2024-03-20T09:11:14.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.309 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:11:14.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.038968+0000", 2024-03-20T09:11:14.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086142660000000006 2024-03-20T09:11:14.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.309 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.112766+0000", 2024-03-20T09:11:14.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.073798323999999998 2024-03-20T09:11:14.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-03-20T09:11:14.310 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.120119+0000", 2024-03-20T09:11:14.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0073532210000000001 2024-03-20T09:11:14.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.310 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:14.314867+0000", 2024-03-20T09:11:14.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1947479040000002 2024-03-20T09:11:14.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:14.311 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:14.315457+0000", 2024-03-20T09:11:14.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00058986399999999995 2024-03-20T09:11:14.311 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.311 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:14.311 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.311 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:134834 3.e 3:7433f64b:::benchmark_data_smithi122_119671_object35052:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e407)", 2024-03-20T09:11:14.311 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:09:11.943566+0000", 2024-03-20T09:11:14.311 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 122.29731233, 2024-03-20T09:11:14.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4728004439999998, 2024-03-20T09:11:14.311 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:14.311 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:11:14.311 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:14.312 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:14.312 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:14.312 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 134834 2024-03-20T09:11:14.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.312 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:14.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:14.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.943566+0000", 2024-03-20T09:11:14.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:14.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.943566+0000", 2024-03-20T09:11:14.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:14.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.943567+0000", 2024-03-20T09:11:14.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4389999999999999e-06 2024-03-20T09:11:14.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:14.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.943573+0000", 2024-03-20T09:11:14.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1600000000000003e-06 2024-03-20T09:11:14.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:14.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.943585+0000", 2024-03-20T09:11:14.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1657999999999999e-05 2024-03-20T09:11:14.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:14.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:11.943592+0000", 2024-03-20T09:11:14.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.827e-06 2024-03-20T09:11:14.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.120998+0000", 2024-03-20T09:11:14.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.17740619899999999 2024-03-20T09:11:14.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:11:14.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.129488+0000", 2024-03-20T09:11:14.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084902840000000007 2024-03-20T09:11:14.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.315 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.180261+0000", 2024-03-20T09:11:14.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.050772738999999997 2024-03-20T09:11:14.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-03-20T09:11:14.315 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.186899+0000", 2024-03-20T09:11:14.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0066383659999999997 2024-03-20T09:11:14.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.315 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:14.413420+0000", 2024-03-20T09:11:14.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2265204440000002 2024-03-20T09:11:14.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.316 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:14.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:14.416366+0000", 2024-03-20T09:11:14.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0029463279999999998 2024-03-20T09:11:14.316 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.316 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:14.316 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.316 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:134894 3.e 3:7b8cf6af:::benchmark_data_smithi122_119671_object35112:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e407)", 2024-03-20T09:11:14.316 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:09:12.456758+0000", 2024-03-20T09:11:14.316 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 121.784119625, 2024-03-20T09:11:14.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5164645669999999, 2024-03-20T09:11:14.316 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:14.316 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:11:14.317 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:14.317 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:14.317 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:14.317 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 134894 2024-03-20T09:11:14.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.317 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:14.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:14.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.456758+0000", 2024-03-20T09:11:14.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:14.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.456758+0000", 2024-03-20T09:11:14.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:14.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.456760+0000", 2024-03-20T09:11:14.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4729999999999999e-06 2024-03-20T09:11:14.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:14.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.456767+0000", 2024-03-20T09:11:14.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3849999999999996e-06 2024-03-20T09:11:14.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:14.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.456783+0000", 2024-03-20T09:11:14.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6030999999999999e-05 2024-03-20T09:11:14.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:14.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.456799+0000", 2024-03-20T09:11:14.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6218999999999998e-05 2024-03-20T09:11:14.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.533976+0000", 2024-03-20T09:11:14.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.077176169000000003 2024-03-20T09:11:14.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:11:14.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.540779+0000", 2024-03-20T09:11:14.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0068032090000000002 2024-03-20T09:11:14.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.614660+0000", 2024-03-20T09:11:14.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.073881079000000002 2024-03-20T09:11:14.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-03-20T09:11:14.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.623179+0000", 2024-03-20T09:11:14.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0085187890000000006 2024-03-20T09:11:14.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:14.972529+0000", 2024-03-20T09:11:14.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.349349809 2024-03-20T09:11:14.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.321 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:14.321 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:14.973223+0000", 2024-03-20T09:11:14.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00069440400000000003 2024-03-20T09:11:14.321 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.321 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:14.321 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.321 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:134894 3.e 3:7b8cf6af:::benchmark_data_smithi122_119671_object35112:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e407)", 2024-03-20T09:11:14.321 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:09:12.654525+0000", 2024-03-20T09:11:14.321 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 121.586352961, 2024-03-20T09:11:14.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.428484965, 2024-03-20T09:11:14.321 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:14.321 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:11:14.321 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:14.322 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:14.322 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:14.322 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 134894 2024-03-20T09:11:14.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.322 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:14.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:14.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.654525+0000", 2024-03-20T09:11:14.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:14.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.654525+0000", 2024-03-20T09:11:14.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:14.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.654525+0000", 2024-03-20T09:11:14.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2000000000000001e-07 2024-03-20T09:11:14.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:14.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.654526+0000", 2024-03-20T09:11:14.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1999999999999999e-06 2024-03-20T09:11:14.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:14.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.654531+0000", 2024-03-20T09:11:14.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0849999999999999e-06 2024-03-20T09:11:14.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:14.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.654533+0000", 2024-03-20T09:11:14.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7860000000000001e-06 2024-03-20T09:11:14.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.172287+0000", 2024-03-20T09:11:14.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.51775417700000004 2024-03-20T09:11:14.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-03-20T09:11:14.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.185001+0000", 2024-03-20T09:11:14.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012713596000000001 2024-03-20T09:11:14.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.022116+0000", 2024-03-20T09:11:14.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8371153609999999 2024-03-20T09:11:14.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:11:14.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.027345+0000", 2024-03-20T09:11:14.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0052283820000000002 2024-03-20T09:11:14.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 8,10", 2024-03-20T09:11:14.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.038153+0000", 2024-03-20T09:11:14.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010808267999999999 2024-03-20T09:11:14.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:14.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.057991+0000", 2024-03-20T09:11:14.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.019838304000000001 2024-03-20T09:11:14.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:14.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.061253+0000", 2024-03-20T09:11:14.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0032616350000000001 2024-03-20T09:11:14.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:11:14.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.062578+0000", 2024-03-20T09:11:14.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013254359999999999 2024-03-20T09:11:14.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:11:14.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.065962+0000", 2024-03-20T09:11:14.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0033838190000000001 2024-03-20T09:11:14.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.328 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:14.328 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.083010+0000", 2024-03-20T09:11:14.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017047696000000001 2024-03-20T09:11:14.328 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.328 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:14.328 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.328 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:134933 3.e 3:7f44148b:::benchmark_data_smithi122_119671_object35151:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e407)", 2024-03-20T09:11:14.329 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:09:12.929365+0000", 2024-03-20T09:11:14.329 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 121.311513136, 2024-03-20T09:11:14.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4526201109999999, 2024-03-20T09:11:14.329 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:14.329 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:11:14.329 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:14.329 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:14.329 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:14.329 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 134933 2024-03-20T09:11:14.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.329 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:14.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:14.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.929365+0000", 2024-03-20T09:11:14.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:14.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.929365+0000", 2024-03-20T09:11:14.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:14.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.929366+0000", 2024-03-20T09:11:14.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2699999999999999e-06 2024-03-20T09:11:14.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:14.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.929373+0000", 2024-03-20T09:11:14.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6470000000000001e-06 2024-03-20T09:11:14.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:14.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.929386+0000", 2024-03-20T09:11:14.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3285000000000001e-05 2024-03-20T09:11:14.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:14.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:12.929393+0000", 2024-03-20T09:11:14.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4070000000000004e-06 2024-03-20T09:11:14.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.185951+0000", 2024-03-20T09:11:14.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.256557743 2024-03-20T09:11:14.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:11:14.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.194899+0000", 2024-03-20T09:11:14.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0089482569999999994 2024-03-20T09:11:14.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.254995+0000", 2024-03-20T09:11:14.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.060095370000000002 2024-03-20T09:11:14.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-03-20T09:11:14.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.263974+0000", 2024-03-20T09:11:14.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0089789029999999999 2024-03-20T09:11:14.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.234242+0000", 2024-03-20T09:11:14.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.970267794 2024-03-20T09:11:14.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:11:14.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.239987+0000", 2024-03-20T09:11:14.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057452459999999999 2024-03-20T09:11:14.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 8,10", 2024-03-20T09:11:14.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.254800+0000", 2024-03-20T09:11:14.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014813152 2024-03-20T09:11:14.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:14.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.346620+0000", 2024-03-20T09:11:14.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.091820339000000001 2024-03-20T09:11:14.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:14.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.350761+0000", 2024-03-20T09:11:14.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0041404120000000004 2024-03-20T09:11:14.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:11:14.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.352992+0000", 2024-03-20T09:11:14.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00223178 2024-03-20T09:11:14.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:11:14.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.362037+0000", 2024-03-20T09:11:14.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0090443329999999999 2024-03-20T09:11:14.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:14.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.381985+0000", 2024-03-20T09:11:14.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.019948173 2024-03-20T09:11:14.335 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.335 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:14.335 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.336 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:134964 3.e 3:7918c111:::benchmark_data_smithi122_119671_object35182:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e407)", 2024-03-20T09:11:14.336 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:09:13.112507+0000", 2024-03-20T09:11:14.336 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 121.128370797, 2024-03-20T09:11:14.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6438591680000001, 2024-03-20T09:11:14.336 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:14.336 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:11:14.336 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:14.336 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:14.336 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:14.336 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 134964 2024-03-20T09:11:14.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.336 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:14.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.337 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:14.337 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.112507+0000", 2024-03-20T09:11:14.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.337 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:14.337 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.112507+0000", 2024-03-20T09:11:14.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:14.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.112508+0000", 2024-03-20T09:11:14.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3179999999999999e-06 2024-03-20T09:11:14.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:14.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.112514+0000", 2024-03-20T09:11:14.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.198e-06 2024-03-20T09:11:14.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:14.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.112526+0000", 2024-03-20T09:11:14.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2418e-05 2024-03-20T09:11:14.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:14.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.112533+0000", 2024-03-20T09:11:14.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3819999999999998e-06 2024-03-20T09:11:14.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.195684+0000", 2024-03-20T09:11:14.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.083150684000000002 2024-03-20T09:11:14.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:11:14.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.202927+0000", 2024-03-20T09:11:14.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0072433030000000004 2024-03-20T09:11:14.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.347453+0000", 2024-03-20T09:11:14.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.14452575500000001 2024-03-20T09:11:14.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-03-20T09:11:14.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.355776+0000", 2024-03-20T09:11:14.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083223250000000002 2024-03-20T09:11:14.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.607122+0000", 2024-03-20T09:11:14.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2513468850000002 2024-03-20T09:11:14.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:11:14.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.612060+0000", 2024-03-20T09:11:14.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049374309999999999 2024-03-20T09:11:14.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 8,10", 2024-03-20T09:11:14.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.620755+0000", 2024-03-20T09:11:14.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086951470000000003 2024-03-20T09:11:14.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:11:14.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.680542+0000", 2024-03-20T09:11:14.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.059786950999999998 2024-03-20T09:11:14.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:14.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.682326+0000", 2024-03-20T09:11:14.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0017835310000000001 2024-03-20T09:11:14.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:14.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.736776+0000", 2024-03-20T09:11:14.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.054450864000000002 2024-03-20T09:11:14.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:11:14.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.740615+0000", 2024-03-20T09:11:14.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0038389230000000002 2024-03-20T09:11:14.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:14.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.756366+0000", 2024-03-20T09:11:14.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015751053000000001 2024-03-20T09:11:14.343 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.343 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:14.343 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.343 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:135007 3.e 3:7da837fd:::benchmark_data_smithi122_119671_object35225:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e407)", 2024-03-20T09:11:14.343 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:09:13.322228+0000", 2024-03-20T09:11:14.343 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 120.918649637, 2024-03-20T09:11:14.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4577662309999999, 2024-03-20T09:11:14.344 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:14.344 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:11:14.344 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:14.344 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:14.344 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:14.344 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 135007 2024-03-20T09:11:14.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.344 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:14.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:14.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.322228+0000", 2024-03-20T09:11:14.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:14.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.322228+0000", 2024-03-20T09:11:14.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:14.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.322229+0000", 2024-03-20T09:11:14.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1069999999999999e-06 2024-03-20T09:11:14.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:14.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.322235+0000", 2024-03-20T09:11:14.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8459999999999996e-06 2024-03-20T09:11:14.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:14.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.322247+0000", 2024-03-20T09:11:14.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2145e-05 2024-03-20T09:11:14.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:14.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.322256+0000", 2024-03-20T09:11:14.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3899999999999993e-06 2024-03-20T09:11:14.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.356520+0000", 2024-03-20T09:11:14.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.034264586 2024-03-20T09:11:14.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:11:14.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.363129+0000", 2024-03-20T09:11:14.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0066088579999999996 2024-03-20T09:11:14.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.405290+0000", 2024-03-20T09:11:14.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.04216027 2024-03-20T09:11:14.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.348 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-03-20T09:11:14.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.412136+0000", 2024-03-20T09:11:14.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0068461479999999998 2024-03-20T09:11:14.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.348 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.659556+0000", 2024-03-20T09:11:14.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2474200299999998 2024-03-20T09:11:14.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:11:14.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.664511+0000", 2024-03-20T09:11:14.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049553310000000003 2024-03-20T09:11:14.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 8,10", 2024-03-20T09:11:14.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.673605+0000", 2024-03-20T09:11:14.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0090943020000000003 2024-03-20T09:11:14.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:11:14.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.756915+0000", 2024-03-20T09:11:14.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.083309262999999995 2024-03-20T09:11:14.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:14.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.758556+0000", 2024-03-20T09:11:14.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001640915 2024-03-20T09:11:14.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:14.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.761312+0000", 2024-03-20T09:11:14.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027568530000000001 2024-03-20T09:11:14.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:11:14.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.764562+0000", 2024-03-20T09:11:14.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0032495710000000001 2024-03-20T09:11:14.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:14.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.779995+0000", 2024-03-20T09:11:14.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015432616 2024-03-20T09:11:14.351 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.351 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:14.352 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.352 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:135015 3.e 3:79a0da03:::benchmark_data_smithi122_119671_object35233:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e407)", 2024-03-20T09:11:14.352 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:09:13.357165+0000", 2024-03-20T09:11:14.352 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 120.883712824, 2024-03-20T09:11:14.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4503571759999998, 2024-03-20T09:11:14.352 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:14.352 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:11:14.352 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:14.352 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:14.352 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:14.352 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 135015 2024-03-20T09:11:14.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.352 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:14.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:14.353 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.357165+0000", 2024-03-20T09:11:14.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:14.353 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.357165+0000", 2024-03-20T09:11:14.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:14.353 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.357167+0000", 2024-03-20T09:11:14.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6509999999999999e-06 2024-03-20T09:11:14.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:14.354 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.357175+0000", 2024-03-20T09:11:14.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1589999999999993e-06 2024-03-20T09:11:14.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.354 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:14.354 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.357191+0000", 2024-03-20T09:11:14.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6294e-05 2024-03-20T09:11:14.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.354 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:14.354 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.357201+0000", 2024-03-20T09:11:14.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4909999999999994e-06 2024-03-20T09:11:14.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.369107+0000", 2024-03-20T09:11:14.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011906745999999999 2024-03-20T09:11:14.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:11:14.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.376193+0000", 2024-03-20T09:11:14.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0070857669999999998 2024-03-20T09:11:14.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.445234+0000", 2024-03-20T09:11:14.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.069041093999999997 2024-03-20T09:11:14.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-03-20T09:11:14.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.451857+0000", 2024-03-20T09:11:14.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0066224839999999997 2024-03-20T09:11:14.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.714813+0000", 2024-03-20T09:11:14.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2629559349999999 2024-03-20T09:11:14.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.357 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:11:14.357 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.719738+0000", 2024-03-20T09:11:14.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049253279999999997 2024-03-20T09:11:14.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.357 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 8,10", 2024-03-20T09:11:14.357 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.728419+0000", 2024-03-20T09:11:14.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086806910000000008 2024-03-20T09:11:14.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.358 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:11:14.358 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.759621+0000", 2024-03-20T09:11:14.358 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031202649999999998 2024-03-20T09:11:14.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.358 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:14.358 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.781656+0000", 2024-03-20T09:11:14.358 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.022034633000000001 2024-03-20T09:11:14.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:14.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.784093+0000", 2024-03-20T09:11:14.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024371639999999999 2024-03-20T09:11:14.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:11:14.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.787249+0000", 2024-03-20T09:11:14.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031559660000000001 2024-03-20T09:11:14.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:14.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.807522+0000", 2024-03-20T09:11:14.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020273123000000001 2024-03-20T09:11:14.360 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.360 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:14.360 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.360 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:135016 3.e 3:74aa0e69:::benchmark_data_smithi122_119671_object35234:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e407)", 2024-03-20T09:11:14.360 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:09:13.539300+0000", 2024-03-20T09:11:14.360 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 120.701578119, 2024-03-20T09:11:14.360 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4694806900000001, 2024-03-20T09:11:14.360 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:14.360 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:11:14.360 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:14.360 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:14.360 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:14.361 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 135016 2024-03-20T09:11:14.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.361 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:14.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.361 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:14.361 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.539300+0000", 2024-03-20T09:11:14.361 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.361 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:14.361 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.539300+0000", 2024-03-20T09:11:14.361 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.361 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:14.361 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.539301+0000", 2024-03-20T09:11:14.362 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8899999999999998e-07 2024-03-20T09:11:14.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.362 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:14.362 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.539306+0000", 2024-03-20T09:11:14.362 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6890000000000003e-06 2024-03-20T09:11:14.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.362 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:14.362 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.539318+0000", 2024-03-20T09:11:14.362 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2067e-05 2024-03-20T09:11:14.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.362 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:14.363 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.539324+0000", 2024-03-20T09:11:14.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5200000000000003e-06 2024-03-20T09:11:14.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.363 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.363 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.622013+0000", 2024-03-20T09:11:14.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.082688481999999994 2024-03-20T09:11:14.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.363 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:11:14.363 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.628757+0000", 2024-03-20T09:11:14.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0067447690000000003 2024-03-20T09:11:14.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.364 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.364 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.686743+0000", 2024-03-20T09:11:14.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.057985842000000003 2024-03-20T09:11:14.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.364 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-03-20T09:11:14.364 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.693080+0000", 2024-03-20T09:11:14.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0063365339999999996 2024-03-20T09:11:14.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.364 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.364 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.855369+0000", 2024-03-20T09:11:14.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.162289473 2024-03-20T09:11:14.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.365 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:11:14.365 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.860245+0000", 2024-03-20T09:11:14.365 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0048759609999999998 2024-03-20T09:11:14.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.365 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 8,10", 2024-03-20T09:11:14.365 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.868891+0000", 2024-03-20T09:11:14.365 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086459889999999998 2024-03-20T09:11:14.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.365 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:14.365 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.934568+0000", 2024-03-20T09:11:14.365 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.065676885000000004 2024-03-20T09:11:14.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.365 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:11:14.366 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.987403+0000", 2024-03-20T09:11:14.366 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.052834816 2024-03-20T09:11:14.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.366 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:14.366 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.989641+0000", 2024-03-20T09:11:14.366 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0022379349999999999 2024-03-20T09:11:14.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.367 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:11:14.367 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:15.993415+0000", 2024-03-20T09:11:14.367 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037746670000000002 2024-03-20T09:11:14.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.367 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:14.367 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:16.008781+0000", 2024-03-20T09:11:14.367 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015365072 2024-03-20T09:11:14.367 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.367 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:14.367 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.367 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:135023 3.e 3:7cd73ad5:::benchmark_data_smithi122_119671_object35241:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e407)", 2024-03-20T09:11:14.368 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:09:13.922986+0000", 2024-03-20T09:11:14.368 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 120.317891641, 2024-03-20T09:11:14.368 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4720135060000001, 2024-03-20T09:11:14.368 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:14.368 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:11:14.368 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:14.368 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:14.368 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:14.368 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 135023 2024-03-20T09:11:14.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.368 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:14.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:14.368 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.922986+0000", 2024-03-20T09:11:14.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.369 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:14.369 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.922986+0000", 2024-03-20T09:11:14.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:14.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.369 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:14.369 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.922988+0000", 2024-03-20T09:11:14.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9309999999999998e-06 2024-03-20T09:11:14.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.369 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:14.369 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.922995+0000", 2024-03-20T09:11:14.370 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3180000000000003e-06 2024-03-20T09:11:14.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.370 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:14.370 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.923010+0000", 2024-03-20T09:11:14.370 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5248999999999999e-05 2024-03-20T09:11:14.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.370 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:14.370 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.923022+0000", 2024-03-20T09:11:14.370 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1995e-05 2024-03-20T09:11:14.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.370 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.371 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.980269+0000", 2024-03-20T09:11:14.371 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.057247601000000002 2024-03-20T09:11:14.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.371 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:11:14.371 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:13.986835+0000", 2024-03-20T09:11:14.371 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065659580000000002 2024-03-20T09:11:14.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.371 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.371 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:14.027420+0000", 2024-03-20T09:11:14.371 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.040584238000000002 2024-03-20T09:11:14.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.372 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-03-20T09:11:14.372 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:14.034194+0000", 2024-03-20T09:11:14.372 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0067748009999999996 2024-03-20T09:11:14.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.372 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:14.372 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:16.303057+0000", 2024-03-20T09:11:14.372 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.268862151 2024-03-20T09:11:14.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.372 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:11:14.373 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:16.307908+0000", 2024-03-20T09:11:14.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0048516779999999999 2024-03-20T09:11:14.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.373 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 8,10", 2024-03-20T09:11:14.373 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:16.316495+0000", 2024-03-20T09:11:14.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0085864889999999992 2024-03-20T09:11:14.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.374 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:11:14.374 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:16.365213+0000", 2024-03-20T09:11:14.374 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.048718251999999997 2024-03-20T09:11:14.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.375 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:14.375 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:16.367477+0000", 2024-03-20T09:11:14.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0022637260000000002 2024-03-20T09:11:14.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.375 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:14.376 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:16.372736+0000", 2024-03-20T09:11:14.376 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0052591340000000004 2024-03-20T09:11:14.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.376 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:11:14.376 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:16.378066+0000", 2024-03-20T09:11:14.377 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0053299719999999997 2024-03-20T09:11:14.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:14.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:14.377 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:14.377 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:09:16.395000+0000", 2024-03-20T09:11:14.377 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016934013000000001 2024-03-20T09:11:14.377 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.378 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:14.378 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.378 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:14.378 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:14.378 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:11:14.378 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:11:14.405 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:14.424 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:11:14.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:14.427 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:11:14.528 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:11:14.561 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:15.314 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:15 smithi026 ceph-mon[68309]: pgmap v1102: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 1 peering, 53 active+clean; 3.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 182 op/s; 183/150252 objects degraded (0.122%); 3957/150252 objects misplaced (2.634%); 646 KiB/s, 233 objects/s recovering 2024-03-20T09:11:15.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:15 smithi115 ceph-mon[72476]: pgmap v1102: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 1 peering, 53 active+clean; 3.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 182 op/s; 183/150252 objects degraded (0.122%); 3957/150252 objects misplaced (2.634%); 646 KiB/s, 233 objects/s recovering 2024-03-20T09:11:15.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:15 smithi122 ceph-mon[73495]: pgmap v1102: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 1 peering, 53 active+clean; 3.1 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 182 op/s; 183/150252 objects degraded (0.122%); 3957/150252 objects misplaced (2.634%); 646 KiB/s, 233 objects/s recovering 2024-03-20T09:11:15.645 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:11:15.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:15.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:11:15.671 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-20T09:11:15.752 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:11:15.786 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:15.920 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:11:15.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:15.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:11:16.024 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:11:16.056 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:16.077 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:11:16.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:16.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:11:16.182 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:11:16.185 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:11:16.185 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:11:16.209 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:16.228 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:11:16.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:16.232 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:11:16.333 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:11:16.359 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:16.377 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:11:16.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:16.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:11:16.481 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:11:16.507 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:16.528 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:11:16.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:16.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:11:16.632 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:11:16.658 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:16.677 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:11:16.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:16.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:11:16.781 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:11:16.809 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:16.828 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:11:16.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:16.831 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:11:16.932 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:11:16.959 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:16.977 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:11:16.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:16.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:11:17.014 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:11:17.019 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.1 to 1 (10000) 2024-03-20T09:11:17.081 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:11:17.113 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:17.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:17 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:11:17.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:17 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:11:17.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:17 smithi115 ceph-mon[72476]: pgmap v1103: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 3.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 192 op/s; 183/148908 objects degraded (0.123%); 3957/148908 objects misplaced (2.657%); 544 KiB/s, 196 objects/s recovering 2024-03-20T09:11:17.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:17 smithi115 ceph-mon[72476]: osdmap e464: 12 total, 12 up, 10 in 2024-03-20T09:11:17.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:17 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2844400359' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-20T09:11:17.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:17 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 183/148908 objects degraded (0.123%), 1 pg degraded (PG_DEGRADED) 2024-03-20T09:11:17.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:17 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2844400359' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-20T09:11:17.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:17 smithi115 ceph-mon[72476]: osdmap e465: 12 total, 12 up, 10 in 2024-03-20T09:11:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:17 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:11:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:17 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:11:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:17 smithi122 ceph-mon[73495]: pgmap v1103: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 3.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 192 op/s; 183/148908 objects degraded (0.123%); 3957/148908 objects misplaced (2.657%); 544 KiB/s, 196 objects/s recovering 2024-03-20T09:11:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:17 smithi122 ceph-mon[73495]: osdmap e464: 12 total, 12 up, 10 in 2024-03-20T09:11:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:17 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2844400359' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-20T09:11:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:17 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 183/148908 objects degraded (0.123%), 1 pg degraded (PG_DEGRADED) 2024-03-20T09:11:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:17 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2844400359' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-20T09:11:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:17 smithi122 ceph-mon[73495]: osdmap e465: 12 total, 12 up, 10 in 2024-03-20T09:11:17.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:17 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:11:17.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:17 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:11:17.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:17 smithi026 ceph-mon[68309]: pgmap v1103: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 3.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 192 op/s; 183/148908 objects degraded (0.123%); 3957/148908 objects misplaced (2.657%); 544 KiB/s, 196 objects/s recovering 2024-03-20T09:11:17.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:17 smithi026 ceph-mon[68309]: osdmap e464: 12 total, 12 up, 10 in 2024-03-20T09:11:17.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:17 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2844400359' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-20T09:11:17.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:17 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 183/148908 objects degraded (0.123%), 1 pg degraded (PG_DEGRADED) 2024-03-20T09:11:17.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:17 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2844400359' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-20T09:11:17.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:17 smithi026 ceph-mon[68309]: osdmap e465: 12 total, 12 up, 10 in 2024-03-20T09:11:17.764 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:11:17.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:17.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:11:17.869 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:11:17.896 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:17.914 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:11:17.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:17.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:11:18.018 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:11:18.044 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:18.063 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:11:18.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:18.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:11:18.169 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:11:18.195 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:18.213 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:11:18.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:18.216 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:11:18.317 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:11:18.327 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 2 1 2024-03-20T09:11:18.343 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:18.361 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:11:18.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:18.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:11:18.465 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:11:18.491 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:18.509 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:11:18.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:18.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:11:18.613 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:11:18.646 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:19.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:19 smithi115 ceph-mon[72476]: pgmap v1106: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 3.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 256 op/s; 217/147114 objects degraded (0.148%); 3852/147114 objects misplaced (2.618%); 1.1 MiB/s, 23 objects/s recovering 2024-03-20T09:11:19.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:19 smithi122 ceph-mon[73495]: pgmap v1106: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 3.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 256 op/s; 217/147114 objects degraded (0.148%); 3852/147114 objects misplaced (2.618%); 1.1 MiB/s, 23 objects/s recovering 2024-03-20T09:11:19.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:19 smithi026 ceph-mon[68309]: pgmap v1106: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 3.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 256 op/s; 217/147114 objects degraded (0.148%); 3852/147114 objects misplaced (2.618%); 1.1 MiB/s, 23 objects/s recovering 2024-03-20T09:11:20.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:20 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:11:20.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:20 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:11:20.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:20 smithi026 ceph-mon[68309]: osdmap e466: 12 total, 12 up, 10 in 2024-03-20T09:11:20.300 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:11:20.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:20.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:11:20.405 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:11:20.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:20 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:11:20.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:20 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:11:20.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:20 smithi115 ceph-mon[72476]: osdmap e466: 12 total, 12 up, 10 in 2024-03-20T09:11:20.433 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:20.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:20 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:11:20.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:20 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:11:20.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:20 smithi122 ceph-mon[73495]: osdmap e466: 12 total, 12 up, 10 in 2024-03-20T09:11:20.452 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:11:20.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:20.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:11:20.556 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:11:20.590 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:20.994 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:11:21.017 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:11:21.046 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:11:21.076 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:11:21.103 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:11:21.124 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:11:21.145 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:11:21.171 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:11:21.197 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:11:21.218 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:11:21.245 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:11:21.270 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:11:21.271 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:11:21.271 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:11:21.271 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:11:21.271 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:11:21.271 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:11:21.271 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:11:21.271 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:11:21.271 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:11:21.272 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:11:21.272 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:11:21.272 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:11:21.272 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:11:21.341 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:21 smithi026 ceph-mon[68309]: pgmap v1108: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 3.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 200 op/s; 217/146634 objects degraded (0.148%); 3852/146634 objects misplaced (2.627%); 715 KiB/s, 11 objects/s recovering 2024-03-20T09:11:21.341 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:21 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3163634887' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:11:21.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:21 smithi115 ceph-mon[72476]: pgmap v1108: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 3.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 200 op/s; 217/146634 objects degraded (0.148%); 3852/146634 objects misplaced (2.627%); 715 KiB/s, 11 objects/s recovering 2024-03-20T09:11:21.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:21 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3163634887' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:11:21.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:21 smithi122 ceph-mon[73495]: pgmap v1108: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 3.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 200 op/s; 217/146634 objects degraded (0.148%); 3852/146634 objects misplaced (2.627%); 715 KiB/s, 11 objects/s recovering 2024-03-20T09:11:21.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:21 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3163634887' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:11:21.744 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:11:21.745 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:11:21.745 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:11:21.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:21.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:11:21.851 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:11:21.879 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:21.898 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:11:21.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:21.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:11:22.004 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:11:22.019 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:11:22.030 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:22.049 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:11:22.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:22.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:11:22.153 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:11:22.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:22 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1443754047' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:11:22.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:22 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 217/146634 objects degraded (0.148%), 1 pg degraded (PG_DEGRADED) 2024-03-20T09:11:22.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:22 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:11:22.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:22 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1443754047' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:11:22.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:22 smithi115 ceph-mon[72476]: osdmap e467: 12 total, 12 up, 10 in 2024-03-20T09:11:22.179 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:22.198 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:11:22.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:22.200 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:11:22.301 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:11:22.327 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:22.345 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:11:22.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:22.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:11:22.369 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:22 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1443754047' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:11:22.369 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:22 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 217/146634 objects degraded (0.148%), 1 pg degraded (PG_DEGRADED) 2024-03-20T09:11:22.369 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:22 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:11:22.369 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:22 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1443754047' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:11:22.369 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:22 smithi026 ceph-mon[68309]: osdmap e467: 12 total, 12 up, 10 in 2024-03-20T09:11:22.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:22 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1443754047' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:11:22.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:22 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 217/146634 objects degraded (0.148%), 1 pg degraded (PG_DEGRADED) 2024-03-20T09:11:22.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:22 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:11:22.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:22 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1443754047' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:11:22.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:22 smithi122 ceph-mon[73495]: osdmap e467: 12 total, 12 up, 10 in 2024-03-20T09:11:22.449 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:11:22.485 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:22.831 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-20T09:11:23.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:23 smithi115 ceph-mon[72476]: pgmap v1109: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.9 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 223 op/s; 243/144876 objects degraded (0.168%); 3757/144876 objects misplaced (2.593%); 1.4 MiB/s, 22 objects/s recovering 2024-03-20T09:11:23.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:23 smithi122 ceph-mon[73495]: pgmap v1109: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.9 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 223 op/s; 243/144876 objects degraded (0.168%); 3757/144876 objects misplaced (2.593%); 1.4 MiB/s, 22 objects/s recovering 2024-03-20T09:11:23.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:23 smithi026 ceph-mon[68309]: pgmap v1109: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.9 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 223 op/s; 243/144876 objects degraded (0.168%); 3757/144876 objects misplaced (2.593%); 1.4 MiB/s, 22 objects/s recovering 2024-03-20T09:11:23.600 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:11:23.651 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:11:23.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:23.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:11:23.756 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:11:23.783 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:23.803 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:11:23.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:23.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:11:23.907 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:11:23.933 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:23.956 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:11:23.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:23.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:11:23.992 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:11:24.059 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:11:24.095 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:24.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:24 smithi026 ceph-mon[68309]: 1.7 scrub starts 2024-03-20T09:11:24.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:24 smithi026 ceph-mon[68309]: 1.7 scrub ok 2024-03-20T09:11:24.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:24 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:11:24.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:24 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:11:24.250 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:11:24.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:24.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:11:24.356 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:11:24.392 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:24.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:24 smithi115 ceph-mon[72476]: 1.7 scrub starts 2024-03-20T09:11:24.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:24 smithi115 ceph-mon[72476]: 1.7 scrub ok 2024-03-20T09:11:24.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:24 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:11:24.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:24 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:11:24.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:24 smithi122 ceph-mon[73495]: 1.7 scrub starts 2024-03-20T09:11:24.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:24 smithi122 ceph-mon[73495]: 1.7 scrub ok 2024-03-20T09:11:24.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:24 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:11:24.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:24 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:11:25.145 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:11:25.173 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:25 smithi026 ceph-mon[68309]: pgmap v1111: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.9 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 166 op/s; 243/144267 objects degraded (0.168%); 3757/144267 objects misplaced (2.604%); 656 KiB/s, 10 objects/s recovering 2024-03-20T09:11:25.173 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:25 smithi026 ceph-mon[68309]: 1.1c scrub starts 2024-03-20T09:11:25.173 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:25 smithi026 ceph-mon[68309]: 1.1c scrub ok 2024-03-20T09:11:25.173 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:25 smithi026 ceph-mon[68309]: osdmap e468: 12 total, 12 up, 10 in 2024-03-20T09:11:25.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:25 smithi115 ceph-mon[72476]: pgmap v1111: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.9 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 166 op/s; 243/144267 objects degraded (0.168%); 3757/144267 objects misplaced (2.604%); 656 KiB/s, 10 objects/s recovering 2024-03-20T09:11:25.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:25 smithi115 ceph-mon[72476]: 1.1c scrub starts 2024-03-20T09:11:25.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:25 smithi115 ceph-mon[72476]: 1.1c scrub ok 2024-03-20T09:11:25.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:25 smithi115 ceph-mon[72476]: osdmap e468: 12 total, 12 up, 10 in 2024-03-20T09:11:25.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:25 smithi122 ceph-mon[73495]: pgmap v1111: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.9 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 166 op/s; 243/144267 objects degraded (0.168%); 3757/144267 objects misplaced (2.604%); 656 KiB/s, 10 objects/s recovering 2024-03-20T09:11:25.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:25 smithi122 ceph-mon[73495]: 1.1c scrub starts 2024-03-20T09:11:25.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:25 smithi122 ceph-mon[73495]: 1.1c scrub ok 2024-03-20T09:11:25.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:25 smithi122 ceph-mon[73495]: osdmap e468: 12 total, 12 up, 10 in 2024-03-20T09:11:25.643 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:11:25.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:25.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:11:25.651 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:11:25.752 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:11:25.784 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:25.860 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:11:25.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:25.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:11:25.967 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:11:25.995 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:26.014 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:11:26.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:26.017 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:11:26.118 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:11:26.154 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:26.281 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:26 smithi026 ceph-mon[68309]: 3.14 scrub starts 2024-03-20T09:11:26.281 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:26 smithi026 ceph-mon[68309]: 3.14 scrub ok 2024-03-20T09:11:26.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:26 smithi115 ceph-mon[72476]: 3.14 scrub starts 2024-03-20T09:11:26.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:26 smithi115 ceph-mon[72476]: 3.14 scrub ok 2024-03-20T09:11:26.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:26 smithi122 ceph-mon[73495]: 3.14 scrub starts 2024-03-20T09:11:26.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:26 smithi122 ceph-mon[73495]: 3.14 scrub ok 2024-03-20T09:11:26.715 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:11:26.715 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:11:26.910 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:11:26.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:26.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:11:27.014 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:11:27.040 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:27.059 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:11:27.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:27.062 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:11:27.163 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:11:27.176 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:27 smithi026 ceph-mon[68309]: 1.9 scrub starts 2024-03-20T09:11:27.176 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:27 smithi026 ceph-mon[68309]: 1.9 scrub ok 2024-03-20T09:11:27.176 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:27 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:11:27.176 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:27 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:11:27.176 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:27 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:11:27.177 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:27 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:11:27.177 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:27 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:11:27.177 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:27 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:11:27.177 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:27 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 243/142914 objects degraded (0.170%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:11:27.177 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:27 smithi026 ceph-mon[68309]: osdmap e469: 12 total, 12 up, 10 in 2024-03-20T09:11:27.177 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:27 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:11:27.177 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:27 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:11:27.177 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:27 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:11:27.177 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:27 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:11:27.177 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:27 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:11:27.189 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:27.208 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:11:27.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:27.211 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:11:27.312 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:11:27.338 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:27.357 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:11:27.359 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:27 smithi115 ceph-mon[72476]: 1.9 scrub starts 2024-03-20T09:11:27.359 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:27 smithi115 ceph-mon[72476]: 1.9 scrub ok 2024-03-20T09:11:27.359 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:27 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:11:27.359 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:27 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:11:27.359 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:27 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:11:27.359 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:27 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:11:27.359 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:27 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:11:27.359 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:27 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:11:27.359 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:27 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 243/142914 objects degraded (0.170%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:11:27.359 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:27 smithi115 ceph-mon[72476]: osdmap e469: 12 total, 12 up, 10 in 2024-03-20T09:11:27.359 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:27 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:11:27.359 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:27 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:11:27.360 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:27 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:11:27.360 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:27 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:11:27.360 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:27 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:11:27.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:27.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:11:27.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:27 smithi122 ceph-mon[73495]: 1.9 scrub starts 2024-03-20T09:11:27.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:27 smithi122 ceph-mon[73495]: 1.9 scrub ok 2024-03-20T09:11:27.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:27 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:11:27.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:27 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:11:27.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:27 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:11:27.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:27 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:11:27.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:27 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:11:27.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:27 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:11:27.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:27 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 243/142914 objects degraded (0.170%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:11:27.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:27 smithi122 ceph-mon[73495]: osdmap e469: 12 total, 12 up, 10 in 2024-03-20T09:11:27.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:27 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:11:27.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:27 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:11:27.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:27 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:11:27.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:27 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:11:27.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:27 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:11:27.462 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:11:27.501 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:27.966 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-20T09:11:27.984 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:11:27.984 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:11:28.103 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.2 to 1 (10000) 2024-03-20T09:11:28.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:28 smithi026 ceph-mon[68309]: 1.14 scrub starts 2024-03-20T09:11:28.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:28 smithi026 ceph-mon[68309]: 1.14 scrub ok 2024-03-20T09:11:28.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:28 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2576180590' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:11:28.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:28 smithi115 ceph-mon[72476]: 1.14 scrub starts 2024-03-20T09:11:28.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:28 smithi115 ceph-mon[72476]: 1.14 scrub ok 2024-03-20T09:11:28.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:28 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2576180590' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:11:28.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:28 smithi122 ceph-mon[73495]: 1.14 scrub starts 2024-03-20T09:11:28.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:28 smithi122 ceph-mon[73495]: 1.14 scrub ok 2024-03-20T09:11:28.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:28 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2576180590' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:11:29.299 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:11:29.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:29.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:11:29.304 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:29 smithi026 ceph-mon[68309]: pgmap v1115: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.9 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 285/140865 objects degraded (0.202%); 3643/140865 objects misplaced (2.586%) 2024-03-20T09:11:29.304 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:29 smithi026 ceph-mon[68309]: 3.13 scrub starts 2024-03-20T09:11:29.304 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:29 smithi026 ceph-mon[68309]: 3.13 scrub ok 2024-03-20T09:11:29.305 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:29 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2576180590' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-20T09:11:29.305 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:29 smithi026 ceph-mon[68309]: osdmap e470: 12 total, 12 up, 10 in 2024-03-20T09:11:29.405 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:11:29.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:29 smithi115 ceph-mon[72476]: pgmap v1115: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.9 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 285/140865 objects degraded (0.202%); 3643/140865 objects misplaced (2.586%) 2024-03-20T09:11:29.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:29 smithi115 ceph-mon[72476]: 3.13 scrub starts 2024-03-20T09:11:29.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:29 smithi115 ceph-mon[72476]: 3.13 scrub ok 2024-03-20T09:11:29.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:29 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2576180590' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-20T09:11:29.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:29 smithi115 ceph-mon[72476]: osdmap e470: 12 total, 12 up, 10 in 2024-03-20T09:11:29.428 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:29 smithi122 ceph-mon[73495]: pgmap v1115: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.9 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 285/140865 objects degraded (0.202%); 3643/140865 objects misplaced (2.586%) 2024-03-20T09:11:29.428 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:29 smithi122 ceph-mon[73495]: 3.13 scrub starts 2024-03-20T09:11:29.428 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:29 smithi122 ceph-mon[73495]: 3.13 scrub ok 2024-03-20T09:11:29.429 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:29 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2576180590' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-20T09:11:29.429 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:29 smithi122 ceph-mon[73495]: osdmap e470: 12 total, 12 up, 10 in 2024-03-20T09:11:29.430 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:29.448 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:11:29.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:29.451 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:11:29.551 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:11:29.583 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:29.838 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:11:29.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:29.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:11:29.942 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:11:29.968 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:29.987 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:11:29.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:29.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:11:30.091 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:11:30.174 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:30.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:30 smithi115 ceph-mon[72476]: 1.1b scrub starts 2024-03-20T09:11:30.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:30 smithi115 ceph-mon[72476]: 1.1b scrub ok 2024-03-20T09:11:30.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:30 smithi122 ceph-mon[73495]: 1.1b scrub starts 2024-03-20T09:11:30.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:30 smithi122 ceph-mon[73495]: 1.1b scrub ok 2024-03-20T09:11:30.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:30 smithi026 ceph-mon[68309]: 1.1b scrub starts 2024-03-20T09:11:30.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:30 smithi026 ceph-mon[68309]: 1.1b scrub ok 2024-03-20T09:11:31.385 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:31 smithi026 ceph-mon[68309]: pgmap v1117: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.9 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 285/140430 objects degraded (0.203%); 3643/140430 objects misplaced (2.594%) 2024-03-20T09:11:31.385 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:31 smithi026 ceph-mon[68309]: 3.15 scrub starts 2024-03-20T09:11:31.385 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:31 smithi026 ceph-mon[68309]: 3.15 scrub ok 2024-03-20T09:11:31.386 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 5 1 2024-03-20T09:11:31.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:31 smithi115 ceph-mon[72476]: pgmap v1117: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.9 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 285/140430 objects degraded (0.203%); 3643/140430 objects misplaced (2.594%) 2024-03-20T09:11:31.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:31 smithi115 ceph-mon[72476]: 3.15 scrub starts 2024-03-20T09:11:31.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:31 smithi115 ceph-mon[72476]: 3.15 scrub ok 2024-03-20T09:11:31.430 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:11:31.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:31.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:11:31.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:31 smithi122 ceph-mon[73495]: pgmap v1117: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.9 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 285/140430 objects degraded (0.203%); 3643/140430 objects misplaced (2.594%) 2024-03-20T09:11:31.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:31 smithi122 ceph-mon[73495]: 3.15 scrub starts 2024-03-20T09:11:31.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:31 smithi122 ceph-mon[73495]: 3.15 scrub ok 2024-03-20T09:11:31.533 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:11:31.560 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:31.578 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:11:31.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:31.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:11:31.683 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:11:31.709 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:31.728 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:11:31.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:31.731 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:11:31.832 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:11:31.859 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:31.878 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:11:31.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:31.881 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:11:31.982 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:11:32.013 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:32.036 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:11:32.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:32.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:11:32.140 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:11:32.166 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:32.185 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:11:32.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:32.187 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:11:32.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:32 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 285/140430 objects degraded (0.203%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:11:32.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:32 smithi026 ceph-mon[68309]: osdmap e471: 12 total, 12 up, 10 in 2024-03-20T09:11:32.271 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:11:32.271 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:11:32.289 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:11:32.292 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:11:32.314 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:32.322 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:11:32.332 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:11:32.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:32.335 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:11:32.341 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:11:32.362 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:11:32.389 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:11:32.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:32 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 285/140430 objects degraded (0.203%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:11:32.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:32 smithi115 ceph-mon[72476]: osdmap e471: 12 total, 12 up, 10 in 2024-03-20T09:11:32.409 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:11:32.430 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:11:32.435 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:11:32.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:32 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 285/140430 objects degraded (0.203%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:11:32.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:32 smithi122 ceph-mon[73495]: osdmap e471: 12 total, 12 up, 10 in 2024-03-20T09:11:32.448 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:11:32.457 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:11:32.462 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:32.481 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:11:32.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:32.485 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:11:32.485 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:11:32.526 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:11:32.530 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:11:32.530 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:11:32.530 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:11:32.531 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:11:32.531 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:11:32.531 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:11:32.531 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:11:32.531 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:11:32.531 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:11:32.531 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:11:32.531 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:11:32.531 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:11:32.586 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:11:32.612 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:32.630 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:11:32.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:32.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:11:32.734 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:11:32.761 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:32.780 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:11:32.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:32.782 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:11:32.883 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:11:32.915 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:32.939 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:11:32.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:32.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:11:33.068 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:11:33.095 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:33.114 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:11:33.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:33.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:11:33.121 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:11:33.218 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:11:33.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:33 smithi026 ceph-mon[68309]: pgmap v1118: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 317/138381 objects degraded (0.229%); 3537/138381 objects misplaced (2.556%) 2024-03-20T09:11:33.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:33 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2502358287' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:11:33.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:33 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2181419044' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:11:33.250 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:33.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:33 smithi115 ceph-mon[72476]: pgmap v1118: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 317/138381 objects degraded (0.229%); 3537/138381 objects misplaced (2.556%) 2024-03-20T09:11:33.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:33 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2502358287' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:11:33.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:33 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2181419044' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:11:33.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:33 smithi122 ceph-mon[73495]: pgmap v1118: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 317/138381 objects degraded (0.229%); 3537/138381 objects misplaced (2.556%) 2024-03-20T09:11:33.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:33 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2502358287' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:11:33.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:33 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2181419044' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:11:34.127 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:34 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:11:34.128 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:34 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2181419044' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:11:34.128 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:34 smithi026 ceph-mon[68309]: osdmap e472: 12 total, 12 up, 10 in 2024-03-20T09:11:34.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:34 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:11:34.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:34 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2181419044' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:11:34.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:34 smithi115 ceph-mon[72476]: osdmap e472: 12 total, 12 up, 10 in 2024-03-20T09:11:34.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:34 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:11:34.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:34 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2181419044' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:11:34.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:34 smithi122 ceph-mon[73495]: osdmap e472: 12 total, 12 up, 10 in 2024-03-20T09:11:34.648 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:11:34.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:34.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:11:34.753 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:11:34.784 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:35.080 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:11:35.086 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:11:35.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:35.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:11:35.191 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:11:35.222 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:35.278 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:11:35.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:35.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:11:35.300 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:11:35.352 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:35 smithi026 ceph-mon[68309]: pgmap v1121: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 183 op/s; 317/137541 objects degraded (0.230%); 3537/137541 objects misplaced (2.572%); 748 KiB/s, 15 objects/s recovering 2024-03-20T09:11:35.384 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:11:35.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:35 smithi115 ceph-mon[72476]: pgmap v1121: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 183 op/s; 317/137541 objects degraded (0.230%); 3537/137541 objects misplaced (2.572%); 748 KiB/s, 15 objects/s recovering 2024-03-20T09:11:35.408 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:35 smithi122 ceph-mon[73495]: pgmap v1121: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 183 op/s; 317/137541 objects degraded (0.230%); 3537/137541 objects misplaced (2.572%); 748 KiB/s, 15 objects/s recovering 2024-03-20T09:11:35.410 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:35.429 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:11:35.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:35.432 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:11:35.533 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:11:35.559 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:35.578 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:11:35.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:35.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:11:35.682 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:11:35.719 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:35.775 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:11:35.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:35.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:11:35.864 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:11:35.864 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-03-20T09:11:35.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-03-20T09:11:35.864 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:11:35.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.865 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:58467 3.12 3.34c92ed2 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e233)", 2024-03-20T09:11:35.865 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:31.876504+0000", 2024-03-20T09:11:35.865 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 483.95982104199999, 2024-03-20T09:11:35.865 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0009192840000001, 2024-03-20T09:11:35.865 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:35.865 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:11:35.865 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:35.865 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:35.865 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:35.865 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58467 2024-03-20T09:11:35.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.865 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:35.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.865 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:35.866 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.876504+0000", 2024-03-20T09:11:35.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.866 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:35.866 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.876504+0000", 2024-03-20T09:11:35.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.866 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:35.866 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.876506+0000", 2024-03-20T09:11:35.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5e-06 2024-03-20T09:11:35.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.867 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:35.867 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.876580+0000", 2024-03-20T09:11:35.867 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3909e-05 2024-03-20T09:11:35.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.867 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:35.867 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.876597+0000", 2024-03-20T09:11:35.867 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7629e-05 2024-03-20T09:11:35.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.867 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:35.867 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.876620+0000", 2024-03-20T09:11:35.867 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2354e-05 2024-03-20T09:11:35.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.881960+0000", 2024-03-20T09:11:35.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0053402290000000002 2024-03-20T09:11:35.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:11:35.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.881968+0000", 2024-03-20T09:11:35.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6369999999999999e-06 2024-03-20T09:11:35.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877405+0000", 2024-03-20T09:11:35.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.99543781899999995 2024-03-20T09:11:35.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:35.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877424+0000", 2024-03-20T09:11:35.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8207e-05 2024-03-20T09:11:35.869 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.869 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:35.869 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.869 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:58485 3.19 3.ce6b2519 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-20T09:11:35.869 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:31.886747+0000", 2024-03-20T09:11:35.869 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 483.94957827000002, 2024-03-20T09:11:35.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.99067739200000005, 2024-03-20T09:11:35.869 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:35.870 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:11:35.870 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:35.870 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:35.870 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:35.870 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58485 2024-03-20T09:11:35.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.870 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:35.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.870 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:35.870 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.886747+0000", 2024-03-20T09:11:35.870 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.871 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:35.871 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.886747+0000", 2024-03-20T09:11:35.871 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.871 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:35.871 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.886749+0000", 2024-03-20T09:11:35.871 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.575e-06 2024-03-20T09:11:35.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.871 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:35.871 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.886804+0000", 2024-03-20T09:11:35.871 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.537e-05 2024-03-20T09:11:35.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.872 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:35.872 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.886818+0000", 2024-03-20T09:11:35.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3940999999999999e-05 2024-03-20T09:11:35.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.872 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:35.872 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.886827+0000", 2024-03-20T09:11:35.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5229999999999995e-06 2024-03-20T09:11:35.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.872 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.872 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.886871+0000", 2024-03-20T09:11:35.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4369000000000002e-05 2024-03-20T09:11:35.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:11:35.873 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.886876+0000", 2024-03-20T09:11:35.873 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2809999999999999e-06 2024-03-20T09:11:35.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.873 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877412+0000", 2024-03-20T09:11:35.873 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.99053602699999999 2024-03-20T09:11:35.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:35.873 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877425+0000", 2024-03-20T09:11:35.873 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2306e-05 2024-03-20T09:11:35.873 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.874 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:35.874 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.874 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:58497 3.12 3.8e7b9d92 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-20T09:11:35.874 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:31.892587+0000", 2024-03-20T09:11:35.874 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 483.943738854, 2024-03-20T09:11:35.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.98492999299999995, 2024-03-20T09:11:35.874 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:35.874 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:11:35.874 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:35.874 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:35.874 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:35.874 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58497 2024-03-20T09:11:35.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.875 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:35.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:35.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.892587+0000", 2024-03-20T09:11:35.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:35.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.892587+0000", 2024-03-20T09:11:35.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:35.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.892588+0000", 2024-03-20T09:11:35.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.265e-06 2024-03-20T09:11:35.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:35.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.892665+0000", 2024-03-20T09:11:35.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6978999999999996e-05 2024-03-20T09:11:35.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:35.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.892676+0000", 2024-03-20T09:11:35.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0814e-05 2024-03-20T09:11:35.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:35.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.892682+0000", 2024-03-20T09:11:35.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3419999999999996e-06 2024-03-20T09:11:35.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.877 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.877 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.892731+0000", 2024-03-20T09:11:35.877 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9107000000000003e-05 2024-03-20T09:11:35.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.877 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:11:35.877 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.892736+0000", 2024-03-20T09:11:35.877 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0370000000000002e-06 2024-03-20T09:11:35.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.877 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.877 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877505+0000", 2024-03-20T09:11:35.877 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.98476891600000005 2024-03-20T09:11:35.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.878 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:35.878 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877517+0000", 2024-03-20T09:11:35.878 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1533e-05 2024-03-20T09:11:35.878 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.878 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:35.878 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.878 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:58477 3.18 3:1b48b892:::benchmark_data_smithi122_119671_object58476:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e234)", 2024-03-20T09:11:35.878 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:32.877148+0000", 2024-03-20T09:11:35.878 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 482.95917746999999, 2024-03-20T09:11:35.879 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2590889679999999, 2024-03-20T09:11:35.879 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:35.879 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:11:35.879 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:35.879 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:35.879 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:35.879 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58477 2024-03-20T09:11:35.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.879 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:35.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.879 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:35.879 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877148+0000", 2024-03-20T09:11:35.879 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:35.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877148+0000", 2024-03-20T09:11:35.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:35.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877150+0000", 2024-03-20T09:11:35.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6759999999999999e-06 2024-03-20T09:11:35.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:35.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877312+0000", 2024-03-20T09:11:35.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000162342 2024-03-20T09:11:35.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:35.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877325+0000", 2024-03-20T09:11:35.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3506000000000001e-05 2024-03-20T09:11:35.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:35.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877336+0000", 2024-03-20T09:11:35.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0129000000000001e-05 2024-03-20T09:11:35.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878426+0000", 2024-03-20T09:11:35.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010908980000000001 2024-03-20T09:11:35.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.882 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:11:35.882 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878436+0000", 2024-03-20T09:11:35.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8700000000000004e-06 2024-03-20T09:11:35.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.882 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.882 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.119544+0000", 2024-03-20T09:11:35.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2411071929999999 2024-03-20T09:11:35.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.882 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:11:35.882 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.119724+0000", 2024-03-20T09:11:35.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000180485 2024-03-20T09:11:35.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.883 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2,7,8", 2024-03-20T09:11:35.883 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.119915+0000", 2024-03-20T09:11:35.883 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019086900000000001 2024-03-20T09:11:35.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.883 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.883 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.128602+0000", 2024-03-20T09:11:35.883 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086875939999999999 2024-03-20T09:11:35.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.883 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.883 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.134430+0000", 2024-03-20T09:11:35.883 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0058273589999999998 2024-03-20T09:11:35.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.884 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:11:35.884 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.134689+0000", 2024-03-20T09:11:35.884 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025899000000000001 2024-03-20T09:11:35.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.884 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.884 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.134789+0000", 2024-03-20T09:11:35.884 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9790999999999994e-05 2024-03-20T09:11:35.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.884 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:11:35.884 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.134857+0000", 2024-03-20T09:11:35.884 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8856000000000006e-05 2024-03-20T09:11:35.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.885 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:35.885 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.136237+0000", 2024-03-20T09:11:35.885 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00137941 2024-03-20T09:11:35.885 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.885 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:35.885 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.885 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:58482 3.18 3:1aeed275:::benchmark_data_smithi122_119671_object58481:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e234)", 2024-03-20T09:11:35.885 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:32.877352+0000", 2024-03-20T09:11:35.885 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 482.95897361200002, 2024-03-20T09:11:35.885 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2600809530000001, 2024-03-20T09:11:35.885 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:35.886 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:11:35.886 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:35.886 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:35.886 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:35.886 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58482 2024-03-20T09:11:35.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.886 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:35.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:35.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877352+0000", 2024-03-20T09:11:35.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:35.887 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877352+0000", 2024-03-20T09:11:35.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.887 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:35.887 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877353+0000", 2024-03-20T09:11:35.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4499999999999996e-07 2024-03-20T09:11:35.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.887 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:35.887 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877393+0000", 2024-03-20T09:11:35.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0172000000000001e-05 2024-03-20T09:11:35.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.887 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:35.888 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877403+0000", 2024-03-20T09:11:35.888 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0152999999999999e-05 2024-03-20T09:11:35.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.888 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:35.888 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877410+0000", 2024-03-20T09:11:35.888 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1729999999999998e-06 2024-03-20T09:11:35.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.888 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.888 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878483+0000", 2024-03-20T09:11:35.888 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001073035 2024-03-20T09:11:35.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.888 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:11:35.889 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878491+0000", 2024-03-20T09:11:35.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6149999999999999e-06 2024-03-20T09:11:35.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.889 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.889 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.120924+0000", 2024-03-20T09:11:35.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.242433439 2024-03-20T09:11:35.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.889 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:11:35.889 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.121069+0000", 2024-03-20T09:11:35.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014456000000000001 2024-03-20T09:11:35.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.889 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2,7,8", 2024-03-20T09:11:35.890 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.121248+0000", 2024-03-20T09:11:35.890 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00017885500000000001 2024-03-20T09:11:35.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.890 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.890 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.134254+0000", 2024-03-20T09:11:35.890 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013006856000000001 2024-03-20T09:11:35.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.890 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.890 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.134471+0000", 2024-03-20T09:11:35.890 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000216076 2024-03-20T09:11:35.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.891 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:11:35.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.134697+0000", 2024-03-20T09:11:35.891 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022649599999999999 2024-03-20T09:11:35.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.891 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.136387+0000", 2024-03-20T09:11:35.891 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001689739 2024-03-20T09:11:35.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.891 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:11:35.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.136449+0000", 2024-03-20T09:11:35.891 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2148000000000002e-05 2024-03-20T09:11:35.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.892 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:35.892 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.137433+0000", 2024-03-20T09:11:35.892 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098379099999999992 2024-03-20T09:11:35.892 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.892 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:35.892 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.892 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:58496 3.18 3:1c5e5fc5:::benchmark_data_smithi122_119671_object58495:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e234)", 2024-03-20T09:11:35.892 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:32.877424+0000", 2024-03-20T09:11:35.892 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 482.958901336, 2024-03-20T09:11:35.892 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2609285780000001, 2024-03-20T09:11:35.892 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:35.892 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:11:35.893 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:35.893 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:35.893 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:35.893 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58496 2024-03-20T09:11:35.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.893 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:35.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.893 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:35.893 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877424+0000", 2024-03-20T09:11:35.893 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.893 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:35.893 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877424+0000", 2024-03-20T09:11:35.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.894 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:35.894 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877425+0000", 2024-03-20T09:11:35.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0100000000000003e-07 2024-03-20T09:11:35.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.894 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:35.894 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877513+0000", 2024-03-20T09:11:35.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7820999999999998e-05 2024-03-20T09:11:35.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:35.895 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877521+0000", 2024-03-20T09:11:35.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2840000000000006e-06 2024-03-20T09:11:35.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:35.895 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877528+0000", 2024-03-20T09:11:35.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.781e-06 2024-03-20T09:11:35.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.895 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878574+0000", 2024-03-20T09:11:35.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010459829999999999 2024-03-20T09:11:35.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.896 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:11:35.896 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878579+0000", 2024-03-20T09:11:35.896 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5960000000000003e-06 2024-03-20T09:11:35.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.896 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.896 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.122222+0000", 2024-03-20T09:11:35.896 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2436422620000001 2024-03-20T09:11:35.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.896 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:11:35.896 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.122376+0000", 2024-03-20T09:11:35.896 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000154528 2024-03-20T09:11:35.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.897 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2,7,8", 2024-03-20T09:11:35.897 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.123115+0000", 2024-03-20T09:11:35.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00073839299999999995 2024-03-20T09:11:35.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.897 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.897 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.134509+0000", 2024-03-20T09:11:35.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011394213 2024-03-20T09:11:35.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.897 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.897 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.134586+0000", 2024-03-20T09:11:35.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7649000000000001e-05 2024-03-20T09:11:35.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:11:35.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.134704+0000", 2024-03-20T09:11:35.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000117108 2024-03-20T09:11:35.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.137481+0000", 2024-03-20T09:11:35.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00277787 2024-03-20T09:11:35.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:11:35.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.137533+0000", 2024-03-20T09:11:35.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1783999999999998e-05 2024-03-20T09:11:35.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:35.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.138353+0000", 2024-03-20T09:11:35.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00081940500000000005 2024-03-20T09:11:35.899 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.899 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:35.899 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.899 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:58502 3.14 3:2edc29ec:::benchmark_data_smithi122_119671_object58501:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e234)", 2024-03-20T09:11:35.899 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:32.877541+0000", 2024-03-20T09:11:35.899 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 482.958784654, 2024-03-20T09:11:35.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2043195660000001, 2024-03-20T09:11:35.899 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:35.900 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:11:35.900 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:35.900 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:35.900 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:35.900 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58502 2024-03-20T09:11:35.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.900 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:35.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:35.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877541+0000", 2024-03-20T09:11:35.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:35.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877541+0000", 2024-03-20T09:11:35.901 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.901 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:35.901 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877541+0000", 2024-03-20T09:11:35.901 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8800000000000002e-07 2024-03-20T09:11:35.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.901 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:35.901 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877581+0000", 2024-03-20T09:11:35.901 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9286000000000003e-05 2024-03-20T09:11:35.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.901 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:35.901 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877590+0000", 2024-03-20T09:11:35.902 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7229999999999999e-06 2024-03-20T09:11:35.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.902 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:35.902 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878533+0000", 2024-03-20T09:11:35.902 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00094220800000000004 2024-03-20T09:11:35.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.902 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.902 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878627+0000", 2024-03-20T09:11:35.902 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4044999999999998e-05 2024-03-20T09:11:35.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.902 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:11:35.903 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878633+0000", 2024-03-20T09:11:35.903 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.251e-06 2024-03-20T09:11:35.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.903 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.903 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.062341+0000", 2024-03-20T09:11:35.903 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.183707877 2024-03-20T09:11:35.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.903 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:11:35.903 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.062443+0000", 2024-03-20T09:11:35.903 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101999 2024-03-20T09:11:35.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.903 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4,8,11", 2024-03-20T09:11:35.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.062563+0000", 2024-03-20T09:11:35.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011998300000000001 2024-03-20T09:11:35.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.078982+0000", 2024-03-20T09:11:35.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016419407 2024-03-20T09:11:35.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.079028+0000", 2024-03-20T09:11:35.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5501999999999999e-05 2024-03-20T09:11:35.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.905 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.079073+0000", 2024-03-20T09:11:35.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5225000000000002e-05 2024-03-20T09:11:35.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.905 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:11:35.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.080427+0000", 2024-03-20T09:11:35.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001354374 2024-03-20T09:11:35.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.905 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:11:35.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.080479+0000", 2024-03-20T09:11:35.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1847e-05 2024-03-20T09:11:35.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:35.906 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.081860+0000", 2024-03-20T09:11:35.906 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013811509999999999 2024-03-20T09:11:35.906 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.906 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:35.906 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.906 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:58506 3.16 3:6b2181e1:::benchmark_data_smithi122_119671_object58505:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e234)", 2024-03-20T09:11:35.906 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:32.878602+0000", 2024-03-20T09:11:35.906 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 482.95772332000001, 2024-03-20T09:11:35.906 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.18545392, 2024-03-20T09:11:35.906 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:35.906 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:11:35.907 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:35.907 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:35.907 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:35.907 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58506 2024-03-20T09:11:35.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.907 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:35.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.907 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:35.907 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878602+0000", 2024-03-20T09:11:35.907 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.907 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:35.907 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878602+0000", 2024-03-20T09:11:35.907 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.908 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:35.908 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878603+0000", 2024-03-20T09:11:35.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1699999999999997e-07 2024-03-20T09:11:35.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.908 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:35.908 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878659+0000", 2024-03-20T09:11:35.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6477999999999997e-05 2024-03-20T09:11:35.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.908 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:35.908 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878671+0000", 2024-03-20T09:11:35.909 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1287999999999999e-05 2024-03-20T09:11:35.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.909 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:35.909 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878676+0000", 2024-03-20T09:11:35.909 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.801e-06 2024-03-20T09:11:35.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.909 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.909 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878737+0000", 2024-03-20T09:11:35.909 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0763000000000001e-05 2024-03-20T09:11:35.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.909 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:11:35.909 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878747+0000", 2024-03-20T09:11:35.910 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3700000000000001e-06 2024-03-20T09:11:35.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.910 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.042973+0000", 2024-03-20T09:11:35.910 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.164226663 2024-03-20T09:11:35.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:11:35.910 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.043149+0000", 2024-03-20T09:11:35.910 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00017547400000000001 2024-03-20T09:11:35.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5,7", 2024-03-20T09:11:35.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.043262+0000", 2024-03-20T09:11:35.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011314 2024-03-20T09:11:35.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:11:35.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.046319+0000", 2024-03-20T09:11:35.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0030568449999999999 2024-03-20T09:11:35.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.060550+0000", 2024-03-20T09:11:35.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014231354999999999 2024-03-20T09:11:35.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.062075+0000", 2024-03-20T09:11:35.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015245840000000001 2024-03-20T09:11:35.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.912 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:11:35.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.062136+0000", 2024-03-20T09:11:35.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.177e-05 2024-03-20T09:11:35.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.912 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:35.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.064056+0000", 2024-03-20T09:11:35.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001919572 2024-03-20T09:11:35.912 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.912 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:35.912 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.913 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:58511 3.14 3:2c7b4300:::benchmark_data_smithi122_119671_object58510:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e234)", 2024-03-20T09:11:35.913 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:32.878689+0000", 2024-03-20T09:11:35.913 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 482.95763645099998, 2024-03-20T09:11:35.913 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.230044227, 2024-03-20T09:11:35.913 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:35.913 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:11:35.913 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:35.913 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:35.913 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:35.913 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58511 2024-03-20T09:11:35.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.913 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:35.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.914 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:35.914 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878689+0000", 2024-03-20T09:11:35.914 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.914 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:35.914 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878689+0000", 2024-03-20T09:11:35.914 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.914 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:35.914 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878689+0000", 2024-03-20T09:11:35.914 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2800000000000002e-07 2024-03-20T09:11:35.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.915 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:35.915 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878744+0000", 2024-03-20T09:11:35.915 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4780000000000001e-05 2024-03-20T09:11:35.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.915 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:35.915 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878751+0000", 2024-03-20T09:11:35.915 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.516e-06 2024-03-20T09:11:35.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.915 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:35.915 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878755+0000", 2024-03-20T09:11:35.915 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2579999999999997e-06 2024-03-20T09:11:35.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.916 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.916 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878822+0000", 2024-03-20T09:11:35.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7376e-05 2024-03-20T09:11:35.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.916 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:11:35.916 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878831+0000", 2024-03-20T09:11:35.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8680000000000005e-06 2024-03-20T09:11:35.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.916 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.916 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.079110+0000", 2024-03-20T09:11:35.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.20027843 2024-03-20T09:11:35.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:11:35.917 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.079268+0000", 2024-03-20T09:11:35.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015832199999999999 2024-03-20T09:11:35.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4,8,11", 2024-03-20T09:11:35.917 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.079459+0000", 2024-03-20T09:11:35.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019131000000000001 2024-03-20T09:11:35.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.917 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.083888+0000", 2024-03-20T09:11:35.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0044284270000000004 2024-03-20T09:11:35.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.918 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.918 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.084205+0000", 2024-03-20T09:11:35.918 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00031737099999999999 2024-03-20T09:11:35.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.918 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:11:35.918 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.086321+0000", 2024-03-20T09:11:35.918 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002115914 2024-03-20T09:11:35.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.918 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.918 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.107469+0000", 2024-03-20T09:11:35.918 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021148275000000001 2024-03-20T09:11:35.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.919 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:11:35.919 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.107532+0000", 2024-03-20T09:11:35.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2669999999999995e-05 2024-03-20T09:11:35.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.919 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:35.919 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.108733+0000", 2024-03-20T09:11:35.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012012819999999999 2024-03-20T09:11:35.919 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.919 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:35.919 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.920 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:58515 3.18 3:1891b56f:::benchmark_data_smithi122_119671_object58514:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e234)", 2024-03-20T09:11:35.920 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:32.878766+0000", 2024-03-20T09:11:35.920 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 482.95755985699998, 2024-03-20T09:11:35.920 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.262981237, 2024-03-20T09:11:35.920 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:35.920 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:11:35.920 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:35.920 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:35.920 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:35.920 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58515 2024-03-20T09:11:35.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.920 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:35.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:35.921 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878766+0000", 2024-03-20T09:11:35.921 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.921 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:35.921 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878766+0000", 2024-03-20T09:11:35.921 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.921 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:35.921 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878766+0000", 2024-03-20T09:11:35.921 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2499999999999995e-07 2024-03-20T09:11:35.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.921 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:35.922 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878836+0000", 2024-03-20T09:11:35.922 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0188e-05 2024-03-20T09:11:35.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.922 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:35.922 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878847+0000", 2024-03-20T09:11:35.922 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1189000000000001e-05 2024-03-20T09:11:35.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.922 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:35.922 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878855+0000", 2024-03-20T09:11:35.922 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4719999999999998e-06 2024-03-20T09:11:35.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878912+0000", 2024-03-20T09:11:35.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7286e-05 2024-03-20T09:11:35.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:11:35.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.878922+0000", 2024-03-20T09:11:35.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5969999999999998e-06 2024-03-20T09:11:35.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.127278+0000", 2024-03-20T09:11:35.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.248356488 2024-03-20T09:11:35.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.924 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:11:35.924 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.127425+0000", 2024-03-20T09:11:35.924 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014669100000000001 2024-03-20T09:11:35.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.924 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2,7,8", 2024-03-20T09:11:35.924 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.127605+0000", 2024-03-20T09:11:35.924 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000180087 2024-03-20T09:11:35.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.924 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.924 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.134547+0000", 2024-03-20T09:11:35.924 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0069416720000000003 2024-03-20T09:11:35.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.925 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.925 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.134623+0000", 2024-03-20T09:11:35.925 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6108999999999999e-05 2024-03-20T09:11:35.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.925 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:11:35.925 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.134711+0000", 2024-03-20T09:11:35.925 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7955999999999996e-05 2024-03-20T09:11:35.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.925 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.925 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.140582+0000", 2024-03-20T09:11:35.925 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0058715759999999999 2024-03-20T09:11:35.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.926 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:11:35.926 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.140623+0000", 2024-03-20T09:11:35.926 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0496e-05 2024-03-20T09:11:35.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.926 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:35.926 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.141747+0000", 2024-03-20T09:11:35.926 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011239049999999999 2024-03-20T09:11:35.926 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.926 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:35.926 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.926 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:109493 3.19 3:9d0cc81e:::benchmark_data_smithi122_119671_object9711:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-03-20T09:11:35.927 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:05:19.098145+0000", 2024-03-20T09:11:35.927 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 376.73818040100002, 2024-03-20T09:11:35.927 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9731130889999999, 2024-03-20T09:11:35.927 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:35.927 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:11:35.927 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:35.927 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:35.927 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:35.927 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 109493 2024-03-20T09:11:35.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.927 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:35.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.927 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:35.927 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.098145+0000", 2024-03-20T09:11:35.927 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.928 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:35.928 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.098145+0000", 2024-03-20T09:11:35.928 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.928 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:35.928 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.098146+0000", 2024-03-20T09:11:35.928 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.176e-06 2024-03-20T09:11:35.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.928 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:35.928 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.098152+0000", 2024-03-20T09:11:35.928 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5180000000000004e-06 2024-03-20T09:11:35.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.929 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:35.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.098168+0000", 2024-03-20T09:11:35.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6727000000000001e-05 2024-03-20T09:11:35.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.929 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:35.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.098177+0000", 2024-03-20T09:11:35.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6950000000000006e-06 2024-03-20T09:11:35.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.929 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.098239+0000", 2024-03-20T09:11:35.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1574000000000004e-05 2024-03-20T09:11:35.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:11:35.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.098246+0000", 2024-03-20T09:11:35.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7689999999999996e-06 2024-03-20T09:11:35.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.965892+0000", 2024-03-20T09:11:35.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.86764582700000004 2024-03-20T09:11:35.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:11:35.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.965906+0000", 2024-03-20T09:11:35.931 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4141000000000001e-05 2024-03-20T09:11:35.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.931 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.063274+0000", 2024-03-20T09:11:35.931 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.097367703 2024-03-20T09:11:35.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:11:35.931 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.063457+0000", 2024-03-20T09:11:35.931 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000182726 2024-03-20T09:11:35.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4,10", 2024-03-20T09:11:35.931 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.063630+0000", 2024-03-20T09:11:35.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000172838 2024-03-20T09:11:35.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.932 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.932 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.067484+0000", 2024-03-20T09:11:35.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0038543789999999998 2024-03-20T09:11:35.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.932 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:11:35.932 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.068439+0000", 2024-03-20T09:11:35.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095458700000000001 2024-03-20T09:11:35.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.932 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.933 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.070204+0000", 2024-03-20T09:11:35.933 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0017656379999999999 2024-03-20T09:11:35.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.933 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:11:35.933 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.070248+0000", 2024-03-20T09:11:35.933 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3988e-05 2024-03-20T09:11:35.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.933 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:35.933 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.071258+0000", 2024-03-20T09:11:35.933 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010098030000000001 2024-03-20T09:11:35.933 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.933 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:35.933 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.934 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:109510 3.19 3:9fc561c9:::benchmark_data_smithi122_119671_object9728:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-03-20T09:11:35.934 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:05:19.331524+0000", 2024-03-20T09:11:35.934 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 376.50480148399998, 2024-03-20T09:11:35.934 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7485444129999999, 2024-03-20T09:11:35.934 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:35.934 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:11:35.934 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:35.934 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:35.934 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:35.934 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 109510 2024-03-20T09:11:35.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.934 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:35.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.935 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:35.935 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.331524+0000", 2024-03-20T09:11:35.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.935 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:35.935 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.331524+0000", 2024-03-20T09:11:35.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.935 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:35.935 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.331525+0000", 2024-03-20T09:11:35.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.313e-06 2024-03-20T09:11:35.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.936 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:35.936 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.331530+0000", 2024-03-20T09:11:35.936 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0810000000000001e-06 2024-03-20T09:11:35.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.936 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:35.936 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.331544+0000", 2024-03-20T09:11:35.936 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3713999999999999e-05 2024-03-20T09:11:35.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.936 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:35.936 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.331552+0000", 2024-03-20T09:11:35.936 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3180000000000007e-06 2024-03-20T09:11:35.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.937 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.937 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.331626+0000", 2024-03-20T09:11:35.937 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4109000000000005e-05 2024-03-20T09:11:35.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.937 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:11:35.937 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.331641+0000", 2024-03-20T09:11:35.937 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4109e-05 2024-03-20T09:11:35.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.937 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.937 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.965939+0000", 2024-03-20T09:11:35.937 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.63429816299999997 2024-03-20T09:11:35.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.938 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:11:35.938 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.965947+0000", 2024-03-20T09:11:35.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5459999999999992e-06 2024-03-20T09:11:35.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.938 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.938 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.066002+0000", 2024-03-20T09:11:35.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1000546309999999 2024-03-20T09:11:35.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.938 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:11:35.938 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.066142+0000", 2024-03-20T09:11:35.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013997400000000001 2024-03-20T09:11:35.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.939 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4,10", 2024-03-20T09:11:35.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.066319+0000", 2024-03-20T09:11:35.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000177596 2024-03-20T09:11:35.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.939 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:11:35.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.068444+0000", 2024-03-20T09:11:35.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021245209999999999 2024-03-20T09:11:35.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.939 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.069381+0000", 2024-03-20T09:11:35.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00093740999999999996 2024-03-20T09:11:35.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.940 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.940 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.078968+0000", 2024-03-20T09:11:35.940 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0095866749999999994 2024-03-20T09:11:35.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.940 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:11:35.940 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.079048+0000", 2024-03-20T09:11:35.940 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9920999999999996e-05 2024-03-20T09:11:35.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.940 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:35.940 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.080068+0000", 2024-03-20T09:11:35.940 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010203320000000001 2024-03-20T09:11:35.940 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.941 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:35.941 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.941 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:109521 3.19 3:9f0303bd:::benchmark_data_smithi122_119671_object9739:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-03-20T09:11:35.941 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:05:19.455979+0000", 2024-03-20T09:11:35.941 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 376.38034668099999, 2024-03-20T09:11:35.941 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.625088366, 2024-03-20T09:11:35.941 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:35.941 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:11:35.941 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:35.941 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:35.941 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:35.941 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 109521 2024-03-20T09:11:35.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.942 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:35.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.942 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:35.942 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.455979+0000", 2024-03-20T09:11:35.942 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.942 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:35.942 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.455979+0000", 2024-03-20T09:11:35.942 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.942 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:35.942 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.455980+0000", 2024-03-20T09:11:35.943 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.161e-06 2024-03-20T09:11:35.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.943 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:35.943 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.455984+0000", 2024-03-20T09:11:35.943 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2969999999999997e-06 2024-03-20T09:11:35.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.943 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:35.943 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.455995+0000", 2024-03-20T09:11:35.943 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.064e-05 2024-03-20T09:11:35.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.943 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:35.944 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.456001+0000", 2024-03-20T09:11:35.944 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9959999999999999e-06 2024-03-20T09:11:35.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.944 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.944 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.456054+0000", 2024-03-20T09:11:35.944 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3532000000000002e-05 2024-03-20T09:11:35.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.944 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:11:35.944 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.456063+0000", 2024-03-20T09:11:35.944 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9129999999999992e-06 2024-03-20T09:11:35.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.944 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.945 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.966087+0000", 2024-03-20T09:11:35.945 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.51002333899999996 2024-03-20T09:11:35.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.945 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:11:35.945 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.966096+0000", 2024-03-20T09:11:35.945 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5799999999999998e-06 2024-03-20T09:11:35.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.945 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.945 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.067530+0000", 2024-03-20T09:11:35.945 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1014342690000001 2024-03-20T09:11:35.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.945 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:11:35.946 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.067631+0000", 2024-03-20T09:11:35.946 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000100739 2024-03-20T09:11:35.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.946 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.946 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4,10", 2024-03-20T09:11:35.946 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.067716+0000", 2024-03-20T09:11:35.946 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4911000000000001e-05 2024-03-20T09:11:35.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.946 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.946 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:11:35.946 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.068516+0000", 2024-03-20T09:11:35.946 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079979400000000005 2024-03-20T09:11:35.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.946 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.947 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.947 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.069500+0000", 2024-03-20T09:11:35.947 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00098445499999999997 2024-03-20T09:11:35.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.947 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.947 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.080148+0000", 2024-03-20T09:11:35.947 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010647933 2024-03-20T09:11:35.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.947 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:11:35.947 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.080188+0000", 2024-03-20T09:11:35.947 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9891000000000001e-05 2024-03-20T09:11:35.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.948 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:35.948 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:21.081067+0000", 2024-03-20T09:11:35.948 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00087891599999999999 2024-03-20T09:11:35.948 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.948 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:35.948 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.948 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:111215 3.19 3:992593d6:::benchmark_data_smithi122_119671_object11433:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-20T09:11:35.948 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:05:36.211371+0000", 2024-03-20T09:11:35.948 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 359.62495424399998, 2024-03-20T09:11:35.948 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.52400578499999995, 2024-03-20T09:11:35.948 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:35.948 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:11:35.949 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:35.949 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:35.949 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:35.949 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 111215 2024-03-20T09:11:35.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.949 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:35.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.949 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:35.949 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.211371+0000", 2024-03-20T09:11:35.949 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.949 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:35.949 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.211371+0000", 2024-03-20T09:11:35.949 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.950 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:35.950 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.211373+0000", 2024-03-20T09:11:35.950 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4589999999999999e-06 2024-03-20T09:11:35.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.950 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:35.950 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.211379+0000", 2024-03-20T09:11:35.950 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4119999999999998e-06 2024-03-20T09:11:35.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.950 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:35.950 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.211396+0000", 2024-03-20T09:11:35.951 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7309999999999999e-05 2024-03-20T09:11:35.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.951 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:35.951 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.211406+0000", 2024-03-20T09:11:35.951 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5780000000000005e-06 2024-03-20T09:11:35.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.951 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.951 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.211466+0000", 2024-03-20T09:11:35.951 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9957000000000002e-05 2024-03-20T09:11:35.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.951 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:11:35.951 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.211475+0000", 2024-03-20T09:11:35.952 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5200000000000003e-06 2024-03-20T09:11:35.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.952 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.952 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.731345+0000", 2024-03-20T09:11:35.952 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.51986975800000002 2024-03-20T09:11:35.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.952 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:11:35.952 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.731474+0000", 2024-03-20T09:11:35.952 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012923500000000001 2024-03-20T09:11:35.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.952 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4,10", 2024-03-20T09:11:35.952 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.731558+0000", 2024-03-20T09:11:35.953 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3554999999999995e-05 2024-03-20T09:11:35.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.953 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.953 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.733276+0000", 2024-03-20T09:11:35.953 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0017183820000000001 2024-03-20T09:11:35.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.953 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:11:35.953 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.734434+0000", 2024-03-20T09:11:35.953 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011574210000000001 2024-03-20T09:11:35.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.953 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.953 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.735293+0000", 2024-03-20T09:11:35.954 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00085930700000000002 2024-03-20T09:11:35.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.954 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:11:35.954 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.735338+0000", 2024-03-20T09:11:35.954 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4855000000000003e-05 2024-03-20T09:11:35.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.954 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:35.954 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.735377+0000", 2024-03-20T09:11:35.954 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9035999999999997e-05 2024-03-20T09:11:35.954 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.954 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:35.954 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.955 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:111241 3.19 3:9a97667f:::benchmark_data_smithi122_119671_object11459:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-03-20T09:11:35.955 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:05:36.490833+0000", 2024-03-20T09:11:35.955 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 359.34549198299999, 2024-03-20T09:11:35.955 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.24470658000000001, 2024-03-20T09:11:35.955 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:35.955 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:11:35.955 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:35.955 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:35.955 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:35.955 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 111241 2024-03-20T09:11:35.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.955 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:35.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.956 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:35.956 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.490833+0000", 2024-03-20T09:11:35.956 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.956 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:35.956 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.490833+0000", 2024-03-20T09:11:35.956 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.956 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:35.956 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.490834+0000", 2024-03-20T09:11:35.956 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6899999999999996e-07 2024-03-20T09:11:35.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.957 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:35.957 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.490838+0000", 2024-03-20T09:11:35.957 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0729999999999998e-06 2024-03-20T09:11:35.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.957 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:35.957 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.490849+0000", 2024-03-20T09:11:35.957 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0223e-05 2024-03-20T09:11:35.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.957 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:35.957 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.490856+0000", 2024-03-20T09:11:35.957 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7659999999999998e-06 2024-03-20T09:11:35.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.958 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.958 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.490910+0000", 2024-03-20T09:11:35.958 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3309000000000001e-05 2024-03-20T09:11:35.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.958 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:11:35.958 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.490919+0000", 2024-03-20T09:11:35.958 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.952e-06 2024-03-20T09:11:35.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.958 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.958 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.733310+0000", 2024-03-20T09:11:35.958 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.24239117099999999 2024-03-20T09:11:35.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:11:35.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.733395+0000", 2024-03-20T09:11:35.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5097000000000002e-05 2024-03-20T09:11:35.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4,10", 2024-03-20T09:11:35.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.733465+0000", 2024-03-20T09:11:35.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0050999999999997e-05 2024-03-20T09:11:35.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.734561+0000", 2024-03-20T09:11:35.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010958039999999999 2024-03-20T09:11:35.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:11:35.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.734587+0000", 2024-03-20T09:11:35.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6163999999999999e-05 2024-03-20T09:11:35.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.735433+0000", 2024-03-20T09:11:35.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084601800000000003 2024-03-20T09:11:35.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:11:35.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.735491+0000", 2024-03-20T09:11:35.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8517000000000002e-05 2024-03-20T09:11:35.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:35.961 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:36.735540+0000", 2024-03-20T09:11:35.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8566000000000002e-05 2024-03-20T09:11:35.961 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.961 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:35.961 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.961 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:114288 3.18 3.4c1b3578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e311)", 2024-03-20T09:11:35.961 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:06:06.044904+0000", 2024-03-20T09:11:35.961 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 329.79142134099999, 2024-03-20T09:11:35.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.87759928399999998, 2024-03-20T09:11:35.961 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:35.962 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:11:35.962 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:35.962 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:35.962 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:35.962 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 114288 2024-03-20T09:11:35.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.962 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:35.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.962 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:35.962 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:06.044904+0000", 2024-03-20T09:11:35.962 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.962 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:35.963 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:06.044904+0000", 2024-03-20T09:11:35.963 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.963 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:35.963 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:06.044905+0000", 2024-03-20T09:11:35.963 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.339e-06 2024-03-20T09:11:35.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.963 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:35.963 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:06.044911+0000", 2024-03-20T09:11:35.963 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8760000000000002e-06 2024-03-20T09:11:35.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.963 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:35.964 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:06.044926+0000", 2024-03-20T09:11:35.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4569999999999999e-05 2024-03-20T09:11:35.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.964 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:35.964 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:06.044935+0000", 2024-03-20T09:11:35.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4509999999999998e-06 2024-03-20T09:11:35.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.964 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.964 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:06.044995+0000", 2024-03-20T09:11:35.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9981000000000001e-05 2024-03-20T09:11:35.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.964 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:11:35.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:06.045005+0000", 2024-03-20T09:11:35.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3500000000000003e-06 2024-03-20T09:11:35.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:06.922494+0000", 2024-03-20T09:11:35.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.87748940499999994 2024-03-20T09:11:35.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:35.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:06.922503+0000", 2024-03-20T09:11:35.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.312e-06 2024-03-20T09:11:35.965 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.965 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:35.965 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.966 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:116322 3.18 3:1e15a469:::benchmark_data_smithi122_119671_object16540:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e320)", 2024-03-20T09:11:35.966 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:06:24.334626+0000", 2024-03-20T09:11:35.966 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 311.501699148, 2024-03-20T09:11:35.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.66543802200000002, 2024-03-20T09:11:35.966 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:35.966 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:11:35.966 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:35.966 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:35.966 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:35.966 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 116322 2024-03-20T09:11:35.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.966 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:35.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:35.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:24.334626+0000", 2024-03-20T09:11:35.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:35.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:24.334626+0000", 2024-03-20T09:11:35.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:35.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:24.334628+0000", 2024-03-20T09:11:35.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.409e-06 2024-03-20T09:11:35.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:35.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:24.334634+0000", 2024-03-20T09:11:35.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5780000000000004e-06 2024-03-20T09:11:35.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:35.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:24.334648+0000", 2024-03-20T09:11:35.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3879999999999999e-05 2024-03-20T09:11:35.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:35.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:24.334655+0000", 2024-03-20T09:11:35.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3549999999999999e-06 2024-03-20T09:11:35.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:24.334717+0000", 2024-03-20T09:11:35.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1235999999999994e-05 2024-03-20T09:11:35.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:11:35.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:24.334728+0000", 2024-03-20T09:11:35.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1374e-05 2024-03-20T09:11:35.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:24.995249+0000", 2024-03-20T09:11:35.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.66052049400000001 2024-03-20T09:11:35.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:11:35.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:24.995376+0000", 2024-03-20T09:11:35.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000127093 2024-03-20T09:11:35.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2,8", 2024-03-20T09:11:35.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:24.995497+0000", 2024-03-20T09:11:35.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012109599999999999 2024-03-20T09:11:35.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:24.998800+0000", 2024-03-20T09:11:35.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0033034000000000002 2024-03-20T09:11:35.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:11:35.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:24.998814+0000", 2024-03-20T09:11:35.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3971999999999999e-05 2024-03-20T09:11:35.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:24.999946+0000", 2024-03-20T09:11:35.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011315019999999999 2024-03-20T09:11:35.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:11:35.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:25.000013+0000", 2024-03-20T09:11:35.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7111000000000002e-05 2024-03-20T09:11:35.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:35.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:25.000064+0000", 2024-03-20T09:11:35.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1521999999999999e-05 2024-03-20T09:11:35.972 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.972 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:35.972 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.972 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:127405 3.19 3:9e576a5f:::benchmark_data_smithi122_119671_object27623:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-03-20T09:11:35.972 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:07:49.241955+0000", 2024-03-20T09:11:35.972 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 226.59437028400001, 2024-03-20T09:11:35.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.501775818, 2024-03-20T09:11:35.973 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:35.973 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:11:35.973 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:35.973 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:35.973 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:35.973 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 127405 2024-03-20T09:11:35.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.973 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:35.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:35.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:49.241955+0000", 2024-03-20T09:11:35.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:35.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:49.241955+0000", 2024-03-20T09:11:35.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:35.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:49.241957+0000", 2024-03-20T09:11:35.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6959999999999999e-06 2024-03-20T09:11:35.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:35.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:49.241962+0000", 2024-03-20T09:11:35.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4970000000000001e-06 2024-03-20T09:11:35.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:35.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:49.241976+0000", 2024-03-20T09:11:35.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3421000000000001e-05 2024-03-20T09:11:35.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:35.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:49.241983+0000", 2024-03-20T09:11:35.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5009999999999998e-06 2024-03-20T09:11:35.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:49.242041+0000", 2024-03-20T09:11:35.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7432999999999997e-05 2024-03-20T09:11:35.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:11:35.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:49.242050+0000", 2024-03-20T09:11:35.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.6940000000000007e-06 2024-03-20T09:11:35.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:49.564641+0000", 2024-03-20T09:11:35.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.32259047800000001 2024-03-20T09:11:35.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:11:35.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:49.564656+0000", 2024-03-20T09:11:35.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4891e-05 2024-03-20T09:11:35.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:50.731937+0000", 2024-03-20T09:11:35.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.167281504 2024-03-20T09:11:35.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:11:35.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:50.732042+0000", 2024-03-20T09:11:35.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010509999999999999 2024-03-20T09:11:35.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.977 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2,10", 2024-03-20T09:11:35.977 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:50.732155+0000", 2024-03-20T09:11:35.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011228 2024-03-20T09:11:35.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:11:35.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:50.738251+0000", 2024-03-20T09:11:35.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0060966639999999999 2024-03-20T09:11:35.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:50.741081+0000", 2024-03-20T09:11:35.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028295780000000001 2024-03-20T09:11:35.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:50.741708+0000", 2024-03-20T09:11:35.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00062702899999999998 2024-03-20T09:11:35.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:11:35.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:50.741799+0000", 2024-03-20T09:11:35.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0753000000000003e-05 2024-03-20T09:11:35.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:35.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:50.743731+0000", 2024-03-20T09:11:35.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001932299 2024-03-20T09:11:35.979 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.979 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:35.979 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.979 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:127415 3.19 3:9809a41d:::benchmark_data_smithi122_119671_object27633:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-03-20T09:11:35.980 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:07:49.853687+0000", 2024-03-20T09:11:35.980 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 225.982638487, 2024-03-20T09:11:35.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.89167427600000004, 2024-03-20T09:11:35.980 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:35.980 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:11:35.980 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:35.980 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:35.980 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:35.980 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 127415 2024-03-20T09:11:35.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.980 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:35.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:35.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:49.853687+0000", 2024-03-20T09:11:35.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:49.853687+0000", 2024-03-20T09:11:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:49.853689+0000", 2024-03-20T09:11:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.52e-06 2024-03-20T09:11:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:49.853701+0000", 2024-03-20T09:11:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1824000000000001e-05 2024-03-20T09:11:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:49.853741+0000", 2024-03-20T09:11:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9725000000000003e-05 2024-03-20T09:11:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:49.853749+0000", 2024-03-20T09:11:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3289999999999994e-06 2024-03-20T09:11:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:49.853825+0000", 2024-03-20T09:11:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5434999999999999e-05 2024-03-20T09:11:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:11:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:49.853834+0000", 2024-03-20T09:11:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7210000000000006e-06 2024-03-20T09:11:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:50.735898+0000", 2024-03-20T09:11:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.88206368400000001 2024-03-20T09:11:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:11:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:50.736077+0000", 2024-03-20T09:11:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000179047 2024-03-20T09:11:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2,10", 2024-03-20T09:11:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:50.736232+0000", 2024-03-20T09:11:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015514599999999999 2024-03-20T09:11:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:11:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:50.738260+0000", 2024-03-20T09:11:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0020276980000000001 2024-03-20T09:11:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:50.741131+0000", 2024-03-20T09:11:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028708879999999998 2024-03-20T09:11:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:50.743806+0000", 2024-03-20T09:11:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026751879999999998 2024-03-20T09:11:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:11:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:50.743877+0000", 2024-03-20T09:11:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0698000000000006e-05 2024-03-20T09:11:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:07:50.745361+0000", 2024-03-20T09:11:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001484373 2024-03-20T09:11:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:143582 3.19 3:9af20452:::benchmark_data_smithi122_119671_object43800:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e449)", 2024-03-20T09:11:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:10:41.276815+0000", 2024-03-20T09:11:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 54.559510207000002, 2024-03-20T09:11:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.88465880600000002, 2024-03-20T09:11:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:11:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:11:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:11:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:11:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:11:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 143582 2024-03-20T09:11:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:11:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:11:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.276815+0000", 2024-03-20T09:11:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:11:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.276815+0000", 2024-03-20T09:11:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:11:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:11:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.276817+0000", 2024-03-20T09:11:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6750000000000001e-06 2024-03-20T09:11:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:11:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.276822+0000", 2024-03-20T09:11:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5650000000000001e-06 2024-03-20T09:11:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:11:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.276836+0000", 2024-03-20T09:11:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3703000000000001e-05 2024-03-20T09:11:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:11:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.276844+0000", 2024-03-20T09:11:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0630000000000006e-06 2024-03-20T09:11:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.276921+0000", 2024-03-20T09:11:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7275999999999997e-05 2024-03-20T09:11:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:11:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.276935+0000", 2024-03-20T09:11:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3916e-05 2024-03-20T09:11:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:11:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.147527+0000", 2024-03-20T09:11:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.87059159600000002 2024-03-20T09:11:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:11:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.147658+0000", 2024-03-20T09:11:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013134999999999999 2024-03-20T09:11:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,10", 2024-03-20T09:11:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.147783+0000", 2024-03-20T09:11:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012454600000000001 2024-03-20T09:11:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:11:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.152392+0000", 2024-03-20T09:11:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0046092980000000004 2024-03-20T09:11:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.152572+0000", 2024-03-20T09:11:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00017944799999999999 2024-03-20T09:11:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:11:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.161292+0000", 2024-03-20T09:11:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0087206660000000002 2024-03-20T09:11:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:11:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.161367+0000", 2024-03-20T09:11:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4252999999999995e-05 2024-03-20T09:11:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:11:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:11:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:11:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.161474+0000", 2024-03-20T09:11:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107451 2024-03-20T09:11:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:11:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:11:35.992 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:11:35.993 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:11:36.024 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:36.155 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:11:36.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:36.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:11:36.260 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:11:36.286 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:36.305 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:11:36.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:36.308 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:11:36.410 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:11:36.437 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:36.456 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:11:36.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:36.458 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:11:36.559 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:11:36.586 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:36.605 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:11:36.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:36.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:11:36.709 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:11:36.735 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:36.753 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:11:36.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:36.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:11:36.856 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:11:36.885 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:37.114 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:11:37.140 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:37 smithi026 ceph-mon[68309]: pgmap v1122: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 211 op/s; 317/135975 objects degraded (0.233%); 3537/135975 objects misplaced (2.601%); 587 KiB/s, 12 objects/s recovering 2024-03-20T09:11:37.141 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:37 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 317/135975 objects degraded (0.233%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:11:37.141 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:37 smithi026 ceph-mon[68309]: osdmap e473: 12 total, 12 up, 10 in 2024-03-20T09:11:37.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:37 smithi115 ceph-mon[72476]: pgmap v1122: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 211 op/s; 317/135975 objects degraded (0.233%); 3537/135975 objects misplaced (2.601%); 587 KiB/s, 12 objects/s recovering 2024-03-20T09:11:37.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:37 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 317/135975 objects degraded (0.233%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:11:37.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:37 smithi115 ceph-mon[72476]: osdmap e473: 12 total, 12 up, 10 in 2024-03-20T09:11:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:37 smithi122 ceph-mon[73495]: pgmap v1122: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 211 op/s; 317/135975 objects degraded (0.233%); 3537/135975 objects misplaced (2.601%); 587 KiB/s, 12 objects/s recovering 2024-03-20T09:11:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:37 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 317/135975 objects degraded (0.233%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:11:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:37 smithi122 ceph-mon[73495]: osdmap e473: 12 total, 12 up, 10 in 2024-03-20T09:11:39.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:39 smithi115 ceph-mon[72476]: pgmap v1124: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.7 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 267 op/s; 362/133569 objects degraded (0.271%); 3418/133569 objects misplaced (2.559%); 810 KiB/s, 12 objects/s recovering 2024-03-20T09:11:39.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:39 smithi122 ceph-mon[73495]: pgmap v1124: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.7 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 267 op/s; 362/133569 objects degraded (0.271%); 3418/133569 objects misplaced (2.559%); 810 KiB/s, 12 objects/s recovering 2024-03-20T09:11:39.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:39 smithi026 ceph-mon[68309]: pgmap v1124: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.7 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 267 op/s; 362/133569 objects degraded (0.271%); 3418/133569 objects misplaced (2.559%); 810 KiB/s, 12 objects/s recovering 2024-03-20T09:11:40.903 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:11:40.921 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:11:40.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:40.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:11:41.026 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:11:41.058 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:41.085 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:11:41.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:41.088 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:11:41.189 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:11:41.215 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:41.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:41 smithi026 ceph-mon[68309]: pgmap v1125: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.7 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 227 op/s; 362/133110 objects degraded (0.272%); 3418/133110 objects misplaced (2.568%); 629 KiB/s, 9 objects/s recovering 2024-03-20T09:11:41.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:41 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:11:41.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:41 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:11:41.233 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:11:41.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:41.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:11:41.337 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:11:41.362 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:41 smithi122 ceph-mon[73495]: pgmap v1125: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.7 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 227 op/s; 362/133110 objects degraded (0.272%); 3418/133110 objects misplaced (2.568%); 629 KiB/s, 9 objects/s recovering 2024-03-20T09:11:41.362 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:41 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:11:41.362 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:41 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:11:41.364 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:41.383 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:11:41.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:41.385 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:11:41.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:41 smithi115 ceph-mon[72476]: pgmap v1125: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.7 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 227 op/s; 362/133110 objects degraded (0.272%); 3418/133110 objects misplaced (2.568%); 629 KiB/s, 9 objects/s recovering 2024-03-20T09:11:41.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:41 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:11:41.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:41 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:11:41.486 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:11:41.515 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:41.534 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:11:41.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:41.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:11:41.638 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:11:41.665 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:41.684 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:11:41.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:41.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:11:41.788 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:11:41.817 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:42.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:42 smithi115 ceph-mon[72476]: 3.9 scrub starts 2024-03-20T09:11:42.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:42 smithi115 ceph-mon[72476]: 3.9 scrub ok 2024-03-20T09:11:42.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:42 smithi115 ceph-mon[72476]: osdmap e474: 12 total, 12 up, 10 in 2024-03-20T09:11:42.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:42 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:11:42.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:42 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:11:42.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:42 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 362/133110 objects degraded (0.272%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:11:42.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:42 smithi115 ceph-mon[72476]: osdmap e475: 12 total, 12 up, 10 in 2024-03-20T09:11:42.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:42 smithi122 ceph-mon[73495]: 3.9 scrub starts 2024-03-20T09:11:42.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:42 smithi122 ceph-mon[73495]: 3.9 scrub ok 2024-03-20T09:11:42.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:42 smithi122 ceph-mon[73495]: osdmap e474: 12 total, 12 up, 10 in 2024-03-20T09:11:42.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:42 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:11:42.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:42 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:11:42.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:42 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 362/133110 objects degraded (0.272%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:11:42.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:42 smithi122 ceph-mon[73495]: osdmap e475: 12 total, 12 up, 10 in 2024-03-20T09:11:42.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:42 smithi026 ceph-mon[68309]: 3.9 scrub starts 2024-03-20T09:11:42.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:42 smithi026 ceph-mon[68309]: 3.9 scrub ok 2024-03-20T09:11:42.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:42 smithi026 ceph-mon[68309]: osdmap e474: 12 total, 12 up, 10 in 2024-03-20T09:11:42.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:42 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:11:42.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:42 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:11:42.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:42 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 362/133110 objects degraded (0.272%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:11:42.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:42 smithi026 ceph-mon[68309]: osdmap e475: 12 total, 12 up, 10 in 2024-03-20T09:11:43.244 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-20T09:11:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:43 smithi115 ceph-mon[72476]: pgmap v1127: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.7 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 275 op/s; 400/130917 objects degraded (0.306%); 3241/130917 objects misplaced (2.476%); 1.2 MiB/s, 18 objects/s recovering 2024-03-20T09:11:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:43 smithi122 ceph-mon[73495]: pgmap v1127: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.7 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 275 op/s; 400/130917 objects degraded (0.306%); 3241/130917 objects misplaced (2.476%); 1.2 MiB/s, 18 objects/s recovering 2024-03-20T09:11:43.448 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:43 smithi026 ceph-mon[68309]: pgmap v1127: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.7 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 275 op/s; 400/130917 objects degraded (0.306%); 3241/130917 objects misplaced (2.476%); 1.2 MiB/s, 18 objects/s recovering 2024-03-20T09:11:43.668 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:11:43.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:43.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:11:43.776 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:11:43.802 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:43.823 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:11:43.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:43.825 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:11:43.927 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:11:43.962 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:44.280 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:11:44.821 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:11:44.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:44.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:11:44.925 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:11:44.956 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:45.085 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:11:45.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:45.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:11:45.163 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.5 to 1 (10000) 2024-03-20T09:11:45.193 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:11:45.220 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:45.239 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:11:45.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:45.242 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:11:45.342 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:11:45.370 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:45.378 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:45 smithi026 ceph-mon[68309]: pgmap v1129: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.7 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 199 op/s; 400/130311 objects degraded (0.307%); 3241/130311 objects misplaced (2.487%); 694 KiB/s, 11 objects/s recovering 2024-03-20T09:11:45.379 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:45 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3854531798' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:11:45.379 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:45 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:11:45.389 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:11:45.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:45.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:11:45.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:45 smithi115 ceph-mon[72476]: pgmap v1129: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.7 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 199 op/s; 400/130311 objects degraded (0.307%); 3241/130311 objects misplaced (2.487%); 694 KiB/s, 11 objects/s recovering 2024-03-20T09:11:45.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:45 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3854531798' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:11:45.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:45 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:11:45.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:45 smithi122 ceph-mon[73495]: pgmap v1129: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.7 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 199 op/s; 400/130311 objects degraded (0.307%); 3241/130311 objects misplaced (2.487%); 694 KiB/s, 11 objects/s recovering 2024-03-20T09:11:45.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:45 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3854531798' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:11:45.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:45 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:11:45.493 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:11:45.532 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:45.791 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:11:45.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:45.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:11:45.896 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:11:45.923 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:45.942 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:11:45.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:45.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:11:46.046 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:11:46.077 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:46.278 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:11:46.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:46.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:11:46.384 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:11:46.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:46 smithi115 ceph-mon[72476]: 3.10 scrub starts 2024-03-20T09:11:46.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:46 smithi115 ceph-mon[72476]: 3.10 scrub ok 2024-03-20T09:11:46.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:46 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-20T09:11:46.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:46 smithi115 ceph-mon[72476]: osdmap e476: 12 total, 12 up, 10 in 2024-03-20T09:11:46.415 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:46 smithi026 ceph-mon[68309]: 3.10 scrub starts 2024-03-20T09:11:46.415 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:46 smithi026 ceph-mon[68309]: 3.10 scrub ok 2024-03-20T09:11:46.415 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:46 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-20T09:11:46.415 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:46 smithi026 ceph-mon[68309]: osdmap e476: 12 total, 12 up, 10 in 2024-03-20T09:11:46.418 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:46.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:46 smithi122 ceph-mon[73495]: 3.10 scrub starts 2024-03-20T09:11:46.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:46 smithi122 ceph-mon[73495]: 3.10 scrub ok 2024-03-20T09:11:46.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:46 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-20T09:11:46.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:46 smithi122 ceph-mon[73495]: osdmap e476: 12 total, 12 up, 10 in 2024-03-20T09:11:46.504 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:11:46.522 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:11:46.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:46.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:11:46.626 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:11:46.653 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:46.672 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:11:46.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:46.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:11:46.776 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:11:46.802 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:46.822 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:11:46.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:46.825 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:11:46.926 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:11:46.957 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:47.239 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:11:47.247 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:11:47.273 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:11:47.296 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:11:47.318 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:11:47.348 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:11:47.377 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:11:47.413 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:11:47.439 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:11:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:47 smithi122 ceph-mon[73495]: 1.0 scrub starts 2024-03-20T09:11:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:47 smithi122 ceph-mon[73495]: 1.0 scrub ok 2024-03-20T09:11:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:47 smithi122 ceph-mon[73495]: pgmap v1131: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.6 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 233 op/s; 400/128904 objects degraded (0.310%); 3241/128904 objects misplaced (2.514%); 779 KiB/s, 12 objects/s recovering 2024-03-20T09:11:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:47 smithi122 ceph-mon[73495]: 3.2 scrub starts 2024-03-20T09:11:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:47 smithi122 ceph-mon[73495]: 3.2 scrub ok 2024-03-20T09:11:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:47 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 400/128904 objects degraded (0.310%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:11:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:47 smithi122 ceph-mon[73495]: osdmap e477: 12 total, 12 up, 10 in 2024-03-20T09:11:47.460 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:11:47.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:47 smithi026 ceph-mon[68309]: 1.0 scrub starts 2024-03-20T09:11:47.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:47 smithi026 ceph-mon[68309]: 1.0 scrub ok 2024-03-20T09:11:47.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:47 smithi026 ceph-mon[68309]: pgmap v1131: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.6 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 233 op/s; 400/128904 objects degraded (0.310%); 3241/128904 objects misplaced (2.514%); 779 KiB/s, 12 objects/s recovering 2024-03-20T09:11:47.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:47 smithi026 ceph-mon[68309]: 3.2 scrub starts 2024-03-20T09:11:47.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:47 smithi026 ceph-mon[68309]: 3.2 scrub ok 2024-03-20T09:11:47.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:47 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 400/128904 objects degraded (0.310%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:11:47.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:47 smithi026 ceph-mon[68309]: osdmap e477: 12 total, 12 up, 10 in 2024-03-20T09:11:47.482 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:11:47.503 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:11:47.505 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:11:47.505 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:11:47.505 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:11:47.505 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:11:47.505 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:11:47.505 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:11:47.505 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:11:47.505 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:11:47.505 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:11:47.505 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:11:47.505 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:11:47.506 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:11:47.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:47 smithi115 ceph-mon[72476]: 1.0 scrub starts 2024-03-20T09:11:47.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:47 smithi115 ceph-mon[72476]: 1.0 scrub ok 2024-03-20T09:11:47.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:47 smithi115 ceph-mon[72476]: pgmap v1131: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.6 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 233 op/s; 400/128904 objects degraded (0.310%); 3241/128904 objects misplaced (2.514%); 779 KiB/s, 12 objects/s recovering 2024-03-20T09:11:47.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:47 smithi115 ceph-mon[72476]: 3.2 scrub starts 2024-03-20T09:11:47.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:47 smithi115 ceph-mon[72476]: 3.2 scrub ok 2024-03-20T09:11:47.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:47 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 400/128904 objects degraded (0.310%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:11:47.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:47 smithi115 ceph-mon[72476]: osdmap e477: 12 total, 12 up, 10 in 2024-03-20T09:11:48.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:48 smithi122 ceph-mon[73495]: 1.4 scrub starts 2024-03-20T09:11:48.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:48 smithi122 ceph-mon[73495]: 1.4 scrub ok 2024-03-20T09:11:48.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:48 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3450713304' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:11:48.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:48 smithi026 ceph-mon[68309]: 1.4 scrub starts 2024-03-20T09:11:48.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:48 smithi026 ceph-mon[68309]: 1.4 scrub ok 2024-03-20T09:11:48.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:48 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3450713304' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:11:48.610 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:11:48.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:48.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:11:48.623 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 6 1 2024-03-20T09:11:48.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:48 smithi115 ceph-mon[72476]: 1.4 scrub starts 2024-03-20T09:11:48.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:48 smithi115 ceph-mon[72476]: 1.4 scrub ok 2024-03-20T09:11:48.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:48 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3450713304' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:11:48.716 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:11:48.742 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:48.762 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:11:48.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:48.764 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:11:48.865 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:11:48.900 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:48.919 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:11:48.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:48.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:11:49.024 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:11:49.050 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:49.071 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:11:49.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:49.073 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:11:49.174 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:11:49.201 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:49.224 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:11:49.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:49.227 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:11:49.328 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:11:49.358 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:49 smithi026 ceph-mon[68309]: pgmap v1133: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.6 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 242 op/s; 446/126546 objects degraded (0.352%); 5746/126546 objects misplaced (4.541%); 1024 KiB/s, 20 objects/s recovering 2024-03-20T09:11:49.361 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:49.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:49 smithi122 ceph-mon[73495]: pgmap v1133: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.6 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 242 op/s; 446/126546 objects degraded (0.352%); 5746/126546 objects misplaced (4.541%); 1024 KiB/s, 20 objects/s recovering 2024-03-20T09:11:49.490 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:11:49.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:49.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:11:49.601 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:11:49.626 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:49 smithi115 ceph-mon[72476]: pgmap v1133: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.6 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 242 op/s; 446/126546 objects degraded (0.352%); 5746/126546 objects misplaced (4.541%); 1024 KiB/s, 20 objects/s recovering 2024-03-20T09:11:49.628 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:49.649 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:11:49.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:49.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:11:49.753 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:11:49.780 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:49.799 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:11:49.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:49.803 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:11:49.904 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:11:49.940 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:50.006 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:11:50.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:50.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:11:50.112 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:11:50.141 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:50.158 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:11:50.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:50.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:11:50.193 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:11:50.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:50 smithi026 ceph-mon[68309]: 3.8 deep-scrub starts 2024-03-20T09:11:50.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:50 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:11:50.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:50 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:11:50.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:50 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:11:50.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:50 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:11:50.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:50 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2741796002' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:11:50.262 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:11:50.289 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:50.310 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:11:50.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:50.315 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:11:50.393 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:11:50.394 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:11:50.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:50 smithi115 ceph-mon[72476]: 3.8 deep-scrub starts 2024-03-20T09:11:50.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:50 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:11:50.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:50 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:11:50.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:50 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:11:50.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:50 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:11:50.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:50 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2741796002' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:11:50.416 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:11:50.444 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:50.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:50 smithi122 ceph-mon[73495]: 3.8 deep-scrub starts 2024-03-20T09:11:50.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:50 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:11:50.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:50 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:11:50.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:50 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:11:50.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:50 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:11:50.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:50 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2741796002' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:11:50.469 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:11:50.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:50.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:11:50.576 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:11:50.620 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:50.711 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:11:51.097 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:11:51.163 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-20T09:11:51.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:51 smithi122 ceph-mon[73495]: pgmap v1134: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 53 active+clean; 2.6 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 214 op/s; 446/125949 objects degraded (0.354%); 5746/125949 objects misplaced (4.562%); 781 KiB/s, 16 objects/s recovering 2024-03-20T09:11:51.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:51 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:11:51.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:51 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2741796002' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:11:51.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:51 smithi122 ceph-mon[73495]: osdmap e478: 12 total, 12 up, 10 in 2024-03-20T09:11:51.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:51 smithi026 ceph-mon[68309]: pgmap v1134: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 53 active+clean; 2.6 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 214 op/s; 446/125949 objects degraded (0.354%); 5746/125949 objects misplaced (4.562%); 781 KiB/s, 16 objects/s recovering 2024-03-20T09:11:51.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:51 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:11:51.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:51 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2741796002' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:11:51.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:51 smithi026 ceph-mon[68309]: osdmap e478: 12 total, 12 up, 10 in 2024-03-20T09:11:51.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:51 smithi115 ceph-mon[72476]: pgmap v1134: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+degraded+remapped, 53 active+clean; 2.6 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 214 op/s; 446/125949 objects degraded (0.354%); 5746/125949 objects misplaced (4.562%); 781 KiB/s, 16 objects/s recovering 2024-03-20T09:11:51.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:51 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:11:51.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:51 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2741796002' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:11:51.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:51 smithi115 ceph-mon[72476]: osdmap e478: 12 total, 12 up, 10 in 2024-03-20T09:11:51.682 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:11:51.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:51.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:11:51.793 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:11:51.821 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:51.843 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:11:51.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:51.846 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:11:51.947 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:11:51.974 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:51.993 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:11:51.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:51.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:11:52.097 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:11:52.124 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:52.145 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:11:52.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:52.148 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:11:52.249 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:11:52.275 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:52 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 446/125949 objects degraded (0.354%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:11:52.275 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:52 smithi115 ceph-mon[72476]: osdmap e479: 12 total, 12 up, 10 in 2024-03-20T09:11:52.277 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:52.297 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:11:52.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:52.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:11:52.402 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:11:52.433 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:11:52.434 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:11:52.434 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:52 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 446/125949 objects degraded (0.354%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:11:52.434 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:52 smithi026 ceph-mon[68309]: osdmap e479: 12 total, 12 up, 10 in 2024-03-20T09:11:52.437 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:52.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:52 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 446/125949 objects degraded (0.354%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:11:52.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:52 smithi122 ceph-mon[73495]: osdmap e479: 12 total, 12 up, 10 in 2024-03-20T09:11:52.465 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:11:52.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:52.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:11:52.570 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:11:52.596 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:52.616 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:11:52.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:52.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:11:52.720 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:11:52.755 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:53.373 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:53 smithi026 ceph-mon[68309]: 3.8 deep-scrub ok 2024-03-20T09:11:53.373 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:53 smithi026 ceph-mon[68309]: pgmap v1136: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 53 active+clean; 2.5 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 300 op/s; 481/123456 objects degraded (0.390%); 5391/123456 objects misplaced (4.367%); 2.5 MiB/s, 44 objects/s recovering 2024-03-20T09:11:53.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:53 smithi122 ceph-mon[73495]: 3.8 deep-scrub ok 2024-03-20T09:11:53.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:53 smithi122 ceph-mon[73495]: pgmap v1136: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 53 active+clean; 2.5 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 300 op/s; 481/123456 objects degraded (0.390%); 5391/123456 objects misplaced (4.367%); 2.5 MiB/s, 44 objects/s recovering 2024-03-20T09:11:53.649 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:11:53.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:53 smithi115 ceph-mon[72476]: 3.8 deep-scrub ok 2024-03-20T09:11:53.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:53 smithi115 ceph-mon[72476]: pgmap v1136: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 53 active+clean; 2.5 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 300 op/s; 481/123456 objects degraded (0.390%); 5391/123456 objects misplaced (4.367%); 2.5 MiB/s, 44 objects/s recovering 2024-03-20T09:11:55.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:55 smithi122 ceph-mon[73495]: pgmap v1138: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 53 active+clean; 2.5 GiB data, 11 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 230 op/s; 481/122664 objects degraded (0.392%); 5391/122664 objects misplaced (4.395%); 1.5 MiB/s, 24 objects/s recovering 2024-03-20T09:11:55.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:55 smithi026 ceph-mon[68309]: pgmap v1138: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 53 active+clean; 2.5 GiB data, 11 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 230 op/s; 481/122664 objects degraded (0.392%); 5391/122664 objects misplaced (4.395%); 1.5 MiB/s, 24 objects/s recovering 2024-03-20T09:11:55.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:55 smithi115 ceph-mon[72476]: pgmap v1138: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 53 active+clean; 2.5 GiB data, 11 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 230 op/s; 481/122664 objects degraded (0.392%); 5391/122664 objects misplaced (4.395%); 1.5 MiB/s, 24 objects/s recovering 2024-03-20T09:11:55.807 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:11:55.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:55.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:11:55.911 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:11:55.938 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:55.956 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:11:55.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:55.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:11:56.060 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:11:56.086 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:56.105 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:11:56.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:56.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:11:56.209 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:11:56.236 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:56.256 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:11:56.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:56.258 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:11:56.359 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:11:56.389 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:56.543 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:11:56.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:56.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:11:56.647 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:11:56.673 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:56.691 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:11:56.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:56.694 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:11:56.795 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:11:56.821 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:56.839 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:11:56.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:56.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:11:56.943 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:11:56.971 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:56.990 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:11:56.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:56.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:11:57.094 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:11:57.134 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:57.372 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:57 smithi026 ceph-mon[68309]: pgmap v1139: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.5 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 481/121233 objects degraded (0.397%); 5391/121233 objects misplaced (4.447%) 2024-03-20T09:11:57.372 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:57 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 481/121233 objects degraded (0.397%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:11:57.389 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:11:57.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:57.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:11:57.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:57 smithi122 ceph-mon[73495]: pgmap v1139: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.5 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 481/121233 objects degraded (0.397%); 5391/121233 objects misplaced (4.447%) 2024-03-20T09:11:57.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:57 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 481/121233 objects degraded (0.397%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:11:57.495 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:11:57.533 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:57.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:57 smithi115 ceph-mon[72476]: pgmap v1139: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.5 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 481/121233 objects degraded (0.397%); 5391/121233 objects misplaced (4.447%) 2024-03-20T09:11:57.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:57 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 481/121233 objects degraded (0.397%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:11:57.701 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:11:57.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:57.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:11:57.807 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:11:57.813 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:11:57.813 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:11:57.813 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:11:57.813 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:11:57.813 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:11:57.813 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:11:57.828 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:11:57.835 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:57.853 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:11:57.854 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:11:57.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:57.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:11:57.893 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:11:57.931 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:11:57.957 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:11:57.969 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:11:57.984 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:58.005 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:11:58.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:58.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:11:58.008 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:11:58.013 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:11:58.013 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:11:58.013 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:11:58.013 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:11:58.013 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:11:58.013 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:11:58.013 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:11:58.013 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:11:58.013 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:11:58.013 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:11:58.013 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:11:58.013 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:11:58.108 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:11:58.135 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:58.154 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:11:58.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:58.156 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:11:58.194 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.6 to 1 (10000) 2024-03-20T09:11:58.257 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:11:58.284 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:58.303 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:11:58.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:58.306 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:11:58.407 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:11:58.432 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:58.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:58 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3509417838' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:11:58.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:58 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1768057205' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-20T09:11:58.452 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:11:58.454 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:58 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3509417838' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:11:58.454 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:58 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1768057205' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-20T09:11:58.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:58.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:11:58.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:58 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3509417838' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:11:58.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:58 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1768057205' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-20T09:11:58.556 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:11:58.583 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:58.601 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:11:58.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:58.604 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:11:58.704 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:11:58.731 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:58.751 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:11:58.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:58.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:11:58.855 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:11:58.884 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:58.907 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:11:58.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:58.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:11:59.011 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:11:59.039 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:59.058 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:11:59.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:59.061 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:11:59.162 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:11:59.189 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:59.211 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:11:59.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:59.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:11:59.315 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:11:59.342 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:59.364 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:11:59.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:59.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:11:59.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:59 smithi122 ceph-mon[73495]: pgmap v1140: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.4 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 505/119259 objects degraded (0.423%); 5125/119259 objects misplaced (4.297%) 2024-03-20T09:11:59.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:59 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1768057205' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-20T09:11:59.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:59 smithi122 ceph-mon[73495]: osdmap e480: 12 total, 12 up, 10 in 2024-03-20T09:11:59.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:59 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:11:59.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:11:59 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:11:59.471 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:11:59.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:59 smithi026 ceph-mon[68309]: pgmap v1140: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.4 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 505/119259 objects degraded (0.423%); 5125/119259 objects misplaced (4.297%) 2024-03-20T09:11:59.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:59 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1768057205' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-20T09:11:59.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:59 smithi026 ceph-mon[68309]: osdmap e480: 12 total, 12 up, 10 in 2024-03-20T09:11:59.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:59 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:11:59.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:11:59 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:11:59.498 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:59.520 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:11:59.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:59.523 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:11:59.624 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:11:59.651 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:59.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:59 smithi115 ceph-mon[72476]: pgmap v1140: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.4 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 505/119259 objects degraded (0.423%); 5125/119259 objects misplaced (4.297%) 2024-03-20T09:11:59.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:59 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1768057205' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-20T09:11:59.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:59 smithi115 ceph-mon[72476]: osdmap e480: 12 total, 12 up, 10 in 2024-03-20T09:11:59.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:59 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:11:59.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:11:59 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:11:59.670 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:11:59.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:59.674 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:11:59.680 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 7 1 2024-03-20T09:11:59.774 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:11:59.801 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:59.821 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:11:59.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:59.824 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:11:59.885 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:11:59.925 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:11:59.944 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:11:59.944 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:11:59.945 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-03-20T09:11:59.945 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-03-20T09:11:59.945 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:11:59.952 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:11:59.971 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:11:59.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:11:59.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:12:00.075 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:12:00.106 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:00.133 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:12:00.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:00.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:12:00.237 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:12:00.264 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:00.285 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:12:00.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:00.288 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:12:00.389 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:12:00.428 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:00 smithi026 ceph-mon[68309]: osdmap e481: 12 total, 12 up, 10 in 2024-03-20T09:12:00.428 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:00 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:12:00.428 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:00 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:12:00.428 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:00 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:12:00.428 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:00 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:12:00.432 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:00.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:00 smithi115 ceph-mon[72476]: osdmap e481: 12 total, 12 up, 10 in 2024-03-20T09:12:00.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:00 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:12:00.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:00 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:12:00.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:00 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:12:00.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:00 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:12:00.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:00 smithi122 ceph-mon[73495]: osdmap e481: 12 total, 12 up, 10 in 2024-03-20T09:12:00.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:00 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:12:00.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:00 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:12:00.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:00 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:12:00.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:00 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:12:01.078 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-20T09:12:01.206 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:12:01.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:01 smithi026 ceph-mon[68309]: pgmap v1143: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.4 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 505/118626 objects degraded (0.426%); 5125/118626 objects misplaced (4.320%) 2024-03-20T09:12:01.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:01 smithi026 ceph-mon[68309]: osdmap e482: 12 total, 12 up, 10 in 2024-03-20T09:12:01.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:01 smithi115 ceph-mon[72476]: pgmap v1143: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.4 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 505/118626 objects degraded (0.426%); 5125/118626 objects misplaced (4.320%) 2024-03-20T09:12:01.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:01 smithi115 ceph-mon[72476]: osdmap e482: 12 total, 12 up, 10 in 2024-03-20T09:12:01.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:01 smithi122 ceph-mon[73495]: pgmap v1143: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.4 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 505/118626 objects degraded (0.426%); 5125/118626 objects misplaced (4.320%) 2024-03-20T09:12:01.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:01 smithi122 ceph-mon[73495]: osdmap e482: 12 total, 12 up, 10 in 2024-03-20T09:12:01.788 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:12:01.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:01.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:12:01.838 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:12:01.893 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:12:01.953 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:01.972 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:12:01.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:01.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:12:02.076 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:12:02.106 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:02.125 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:12:02.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:02.128 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:12:02.229 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:12:02.277 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:02.361 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:12:02.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:02.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:12:02.404 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:02 smithi026 ceph-mon[68309]: osdmap e483: 12 total, 12 up, 10 in 2024-03-20T09:12:02.405 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:02 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 505/118626 objects degraded (0.426%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:12:02.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:02 smithi122 ceph-mon[73495]: osdmap e483: 12 total, 12 up, 10 in 2024-03-20T09:12:02.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:02 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 505/118626 objects degraded (0.426%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:12:02.470 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:12:02.511 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:02.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:02 smithi115 ceph-mon[72476]: osdmap e483: 12 total, 12 up, 10 in 2024-03-20T09:12:02.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:02 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 505/118626 objects degraded (0.426%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:12:03.048 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:12:03.232 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:12:03.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:03 smithi026 ceph-mon[68309]: pgmap v1146: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 53 active+clean; 2.3 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 382 op/s; 657/114666 objects degraded (0.573%); 4437/114666 objects misplaced (3.869%); 6.0 MiB/s, 95 objects/s recovering 2024-03-20T09:12:03.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:03 smithi026 ceph-mon[68309]: osdmap e484: 12 total, 12 up, 10 in 2024-03-20T09:12:03.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:03 smithi026 ceph-mon[68309]: from='client.15735 172.21.15.26:0/3024641477' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:12:03.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:03 smithi115 ceph-mon[72476]: pgmap v1146: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 53 active+clean; 2.3 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 382 op/s; 657/114666 objects degraded (0.573%); 4437/114666 objects misplaced (3.869%); 6.0 MiB/s, 95 objects/s recovering 2024-03-20T09:12:03.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:03 smithi115 ceph-mon[72476]: osdmap e484: 12 total, 12 up, 10 in 2024-03-20T09:12:03.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:03 smithi115 ceph-mon[72476]: from='client.15735 172.21.15.26:0/3024641477' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:12:03.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:03 smithi122 ceph-mon[73495]: pgmap v1146: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 53 active+clean; 2.3 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 382 op/s; 657/114666 objects degraded (0.573%); 4437/114666 objects misplaced (3.869%); 6.0 MiB/s, 95 objects/s recovering 2024-03-20T09:12:03.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:03 smithi122 ceph-mon[73495]: osdmap e484: 12 total, 12 up, 10 in 2024-03-20T09:12:03.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:03 smithi122 ceph-mon[73495]: from='client.15735 172.21.15.26:0/3024641477' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:12:03.822 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:12:04.079 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:12:04.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:04.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:12:04.184 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:12:04.210 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:04.229 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:12:04.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:04.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:12:04.335 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:12:04.372 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:04 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:12:04.372 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:04 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:12:04.372 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:04 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:12:04.372 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:04 smithi026 ceph-mon[68309]: from='client.15735 172.21.15.26:0/3024641477' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:12:04.373 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:04 smithi026 ceph-mon[68309]: osdmap e485: 12 total, 12 up, 10 in 2024-03-20T09:12:04.378 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:04.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:04 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:12:04.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:04 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:12:04.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:04 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:12:04.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:04 smithi115 ceph-mon[72476]: from='client.15735 172.21.15.26:0/3024641477' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:12:04.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:04 smithi115 ceph-mon[72476]: osdmap e485: 12 total, 12 up, 10 in 2024-03-20T09:12:04.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:04 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:12:04.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:04 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:12:04.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:04 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:12:04.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:04 smithi122 ceph-mon[73495]: from='client.15735 172.21.15.26:0/3024641477' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:12:04.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:04 smithi122 ceph-mon[73495]: osdmap e485: 12 total, 12 up, 10 in 2024-03-20T09:12:05.644 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:12:05.651 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:05 smithi026 ceph-mon[68309]: pgmap v1149: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 53 active+clean; 2.3 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 449 op/s; 657/113226 objects degraded (0.580%); 4437/113226 objects misplaced (3.919%); 6.0 MiB/s, 95 objects/s recovering 2024-03-20T09:12:05.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:05.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:12:05.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:05 smithi115 ceph-mon[72476]: pgmap v1149: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 53 active+clean; 2.3 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 449 op/s; 657/113226 objects degraded (0.580%); 4437/113226 objects misplaced (3.919%); 6.0 MiB/s, 95 objects/s recovering 2024-03-20T09:12:05.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:05 smithi122 ceph-mon[73495]: pgmap v1149: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 53 active+clean; 2.3 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 449 op/s; 657/113226 objects degraded (0.580%); 4437/113226 objects misplaced (3.919%); 6.0 MiB/s, 95 objects/s recovering 2024-03-20T09:12:05.753 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:12:05.786 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:05.812 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:12:05.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:05.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:12:05.924 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:12:05.958 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:05.979 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:12:05.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:05.983 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:12:06.084 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:12:06.149 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:06.288 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:12:06.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:06.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:12:06.394 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:12:06.424 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:06.501 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:12:06.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:06.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:12:06.605 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:12:06.644 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:07.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:07 smithi026 ceph-mon[68309]: pgmap v1150: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.3 GiB data, 9.9 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 378 op/s; 657/111666 objects degraded (0.588%); 4437/111666 objects misplaced (3.973%); 4.3 MiB/s, 68 objects/s recovering 2024-03-20T09:12:07.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:07 smithi026 ceph-mon[68309]: osdmap e486: 12 total, 12 up, 10 in 2024-03-20T09:12:07.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:07 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 657/111666 objects degraded (0.588%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:12:07.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:07 smithi026 ceph-mon[68309]: osdmap e487: 12 total, 12 up, 10 in 2024-03-20T09:12:07.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:07 smithi115 ceph-mon[72476]: pgmap v1150: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.3 GiB data, 9.9 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 378 op/s; 657/111666 objects degraded (0.588%); 4437/111666 objects misplaced (3.973%); 4.3 MiB/s, 68 objects/s recovering 2024-03-20T09:12:07.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:07 smithi115 ceph-mon[72476]: osdmap e486: 12 total, 12 up, 10 in 2024-03-20T09:12:07.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:07 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 657/111666 objects degraded (0.588%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:12:07.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:07 smithi115 ceph-mon[72476]: osdmap e487: 12 total, 12 up, 10 in 2024-03-20T09:12:07.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:07 smithi122 ceph-mon[73495]: pgmap v1150: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.3 GiB data, 9.9 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 378 op/s; 657/111666 objects degraded (0.588%); 4437/111666 objects misplaced (3.973%); 4.3 MiB/s, 68 objects/s recovering 2024-03-20T09:12:07.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:07 smithi122 ceph-mon[73495]: osdmap e486: 12 total, 12 up, 10 in 2024-03-20T09:12:07.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:07 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 657/111666 objects degraded (0.588%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:12:07.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:07 smithi122 ceph-mon[73495]: osdmap e487: 12 total, 12 up, 10 in 2024-03-20T09:12:08.054 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:12:08.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:08.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:12:08.161 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:12:08.201 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:08.653 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:12:08.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:08.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:12:08.759 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:12:08.785 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:08.805 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:12:08.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:08.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:12:08.909 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:12:08.955 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:08.993 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:12:08.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:08.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:12:09.091 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:12:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-03-20T09:12:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-03-20T09:12:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:12:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:58477 3.18 3.491d12d8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-20T09:12:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:31.878059+0000", 2024-03-20T09:12:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 517.05259766999995, 2024-03-20T09:12:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.997644431, 2024-03-20T09:12:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:12:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:12:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:12:09.092 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:12:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:12:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58477 2024-03-20T09:12:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:12:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:12:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.878059+0000", 2024-03-20T09:12:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:12:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.878059+0000", 2024-03-20T09:12:09.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:12:09.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.878061+0000", 2024-03-20T09:12:09.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.437e-06 2024-03-20T09:12:09.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:12:09.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.878180+0000", 2024-03-20T09:12:09.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011947699999999999 2024-03-20T09:12:09.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:12:09.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.878191+0000", 2024-03-20T09:12:09.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0373999999999999e-05 2024-03-20T09:12:09.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:12:09.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.878197+0000", 2024-03-20T09:12:09.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.037e-06 2024-03-20T09:12:09.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:12:09.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.884977+0000", 2024-03-20T09:12:09.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0067808499999999997 2024-03-20T09:12:09.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:12:09.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.884986+0000", 2024-03-20T09:12:09.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8340000000000005e-06 2024-03-20T09:12:09.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:12:09.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.875687+0000", 2024-03-20T09:12:09.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.99070090499999997 2024-03-20T09:12:09.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:12:09.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.875704+0000", 2024-03-20T09:12:09.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6517000000000001e-05 2024-03-20T09:12:09.097 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.097 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:12:09.097 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.097 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:58482 3.18 3.ae4b7758 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-20T09:12:09.097 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:31.881753+0000", 2024-03-20T09:12:09.097 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 517.04890354199995, 2024-03-20T09:12:09.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.99399671300000003, 2024-03-20T09:12:09.097 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:12:09.097 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:12:09.097 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:12:09.097 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:12:09.097 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:12:09.098 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58482 2024-03-20T09:12:09.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.098 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:12:09.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:12:09.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.881753+0000", 2024-03-20T09:12:09.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:12:09.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.881753+0000", 2024-03-20T09:12:09.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:12:09.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.881755+0000", 2024-03-20T09:12:09.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7940000000000001e-06 2024-03-20T09:12:09.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:12:09.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.881792+0000", 2024-03-20T09:12:09.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6389e-05 2024-03-20T09:12:09.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:12:09.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.881827+0000", 2024-03-20T09:12:09.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5485999999999999e-05 2024-03-20T09:12:09.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:12:09.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.881838+0000", 2024-03-20T09:12:09.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0784999999999999e-05 2024-03-20T09:12:09.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:12:09.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.885173+0000", 2024-03-20T09:12:09.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003335471 2024-03-20T09:12:09.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:12:09.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.885180+0000", 2024-03-20T09:12:09.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6460000000000004e-06 2024-03-20T09:12:09.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:12:09.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.875740+0000", 2024-03-20T09:12:09.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.990560102 2024-03-20T09:12:09.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:12:09.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.875750+0000", 2024-03-20T09:12:09.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.004e-05 2024-03-20T09:12:09.101 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.101 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:12:09.101 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.102 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:58496 3.18 3.a3fa7a38 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-20T09:12:09.102 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:31.892558+0000", 2024-03-20T09:12:09.102 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 517.03809871999999, 2024-03-20T09:12:09.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.98322797399999995, 2024-03-20T09:12:09.102 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:12:09.102 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:12:09.102 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:12:09.102 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:12:09.102 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:12:09.102 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58496 2024-03-20T09:12:09.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.102 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:12:09.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:12:09.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.892558+0000", 2024-03-20T09:12:09.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:12:09.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.892558+0000", 2024-03-20T09:12:09.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:12:09.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.892560+0000", 2024-03-20T09:12:09.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6220000000000001e-06 2024-03-20T09:12:09.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:12:09.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.892654+0000", 2024-03-20T09:12:09.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4568999999999995e-05 2024-03-20T09:12:09.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:12:09.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.892668+0000", 2024-03-20T09:12:09.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4046000000000001e-05 2024-03-20T09:12:09.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:12:09.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.892677+0000", 2024-03-20T09:12:09.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9150000000000002e-06 2024-03-20T09:12:09.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:12:09.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.892723+0000", 2024-03-20T09:12:09.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6159000000000003e-05 2024-03-20T09:12:09.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:12:09.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.892729+0000", 2024-03-20T09:12:09.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7089999999999999e-06 2024-03-20T09:12:09.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:12:09.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.875779+0000", 2024-03-20T09:12:09.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.98304958099999995 2024-03-20T09:12:09.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:12:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.875786+0000", 2024-03-20T09:12:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3730000000000004e-06 2024-03-20T09:12:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:12:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:58515 3.18 3.f6ad8918 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-20T09:12:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:31.912641+0000", 2024-03-20T09:12:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 517.01801571700003, 2024-03-20T09:12:09.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.96324149800000003, 2024-03-20T09:12:09.107 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:12:09.107 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:12:09.107 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:12:09.107 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:12:09.107 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:12:09.107 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58515 2024-03-20T09:12:09.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.107 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:12:09.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:12:09.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.912641+0000", 2024-03-20T09:12:09.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:12:09.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.912641+0000", 2024-03-20T09:12:09.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:12:09.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.912643+0000", 2024-03-20T09:12:09.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2419999999999999e-06 2024-03-20T09:12:09.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:12:09.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.912706+0000", 2024-03-20T09:12:09.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2766000000000002e-05 2024-03-20T09:12:09.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:12:09.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.912729+0000", 2024-03-20T09:12:09.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2762999999999999e-05 2024-03-20T09:12:09.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:12:09.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.912742+0000", 2024-03-20T09:12:09.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2796000000000001e-05 2024-03-20T09:12:09.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:12:09.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.912813+0000", 2024-03-20T09:12:09.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1703000000000001e-05 2024-03-20T09:12:09.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:12:09.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:31.912821+0000", 2024-03-20T09:12:09.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8140000000000001e-06 2024-03-20T09:12:09.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:12:09.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.875875+0000", 2024-03-20T09:12:09.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.96305376099999995 2024-03-20T09:12:09.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:12:09.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.875883+0000", 2024-03-20T09:12:09.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6529999999999994e-06 2024-03-20T09:12:09.113 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.113 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:12:09.113 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.113 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:103586 3.5 3:a6a77c29:::benchmark_data_smithi122_119671_object3804:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-03-20T09:12:09.113 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:04:56.103690+0000", 2024-03-20T09:12:09.113 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 432.82696711900002, 2024-03-20T09:12:09.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.180922685, 2024-03-20T09:12:09.113 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:12:09.114 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:12:09.114 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:12:09.114 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:12:09.114 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:12:09.114 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 103586 2024-03-20T09:12:09.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.114 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:12:09.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:12:09.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:04:56.103690+0000", 2024-03-20T09:12:09.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:12:09.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:04:56.103690+0000", 2024-03-20T09:12:09.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:12:09.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:04:56.103691+0000", 2024-03-20T09:12:09.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.415e-06 2024-03-20T09:12:09.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:12:09.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:04:56.103697+0000", 2024-03-20T09:12:09.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1600000000000003e-06 2024-03-20T09:12:09.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:12:09.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:04:56.103712+0000", 2024-03-20T09:12:09.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4561e-05 2024-03-20T09:12:09.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:12:09.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:04:56.103722+0000", 2024-03-20T09:12:09.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8949999999999993e-06 2024-03-20T09:12:09.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:12:09.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:04:56.103810+0000", 2024-03-20T09:12:09.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.865e-05 2024-03-20T09:12:09.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:12:09.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:04:56.103912+0000", 2024-03-20T09:12:09.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101508 2024-03-20T09:12:09.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 6", 2024-03-20T09:12:09.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:04:56.103989+0000", 2024-03-20T09:12:09.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7115999999999998e-05 2024-03-20T09:12:09.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:12:09.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:04:56.105304+0000", 2024-03-20T09:12:09.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013148929999999999 2024-03-20T09:12:09.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:12:09.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:04:56.284447+0000", 2024-03-20T09:12:09.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.179142791 2024-03-20T09:12:09.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:12:09.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:04:56.284514+0000", 2024-03-20T09:12:09.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7144000000000006e-05 2024-03-20T09:12:09.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:12:09.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:04:56.284612+0000", 2024-03-20T09:12:09.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8552000000000001e-05 2024-03-20T09:12:09.119 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.119 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:12:09.119 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.120 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:104413 3.5 3:a1670027:::benchmark_data_smithi122_119671_object4631:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-03-20T09:12:09.120 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:05:00.201771+0000", 2024-03-20T09:12:09.120 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 428.72888557800002, 2024-03-20T09:12:09.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.192000482, 2024-03-20T09:12:09.120 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:12:09.120 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:12:09.120 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:12:09.120 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:12:09.120 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:12:09.120 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 104413 2024-03-20T09:12:09.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.121 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:12:09.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:12:09.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:00.201771+0000", 2024-03-20T09:12:09.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:12:09.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:00.201771+0000", 2024-03-20T09:12:09.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:12:09.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:00.201774+0000", 2024-03-20T09:12:09.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1360000000000001e-06 2024-03-20T09:12:09.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:12:09.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:00.201790+0000", 2024-03-20T09:12:09.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5664000000000001e-05 2024-03-20T09:12:09.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:12:09.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:00.201858+0000", 2024-03-20T09:12:09.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7750999999999996e-05 2024-03-20T09:12:09.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:12:09.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:00.201872+0000", 2024-03-20T09:12:09.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4127999999999999e-05 2024-03-20T09:12:09.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:12:09.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:00.201942+0000", 2024-03-20T09:12:09.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9859999999999999e-05 2024-03-20T09:12:09.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:12:09.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:00.202054+0000", 2024-03-20T09:12:09.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000112213 2024-03-20T09:12:09.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 6", 2024-03-20T09:12:09.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:00.202134+0000", 2024-03-20T09:12:09.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0369000000000003e-05 2024-03-20T09:12:09.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:12:09.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:00.203492+0000", 2024-03-20T09:12:09.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001357104 2024-03-20T09:12:09.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:12:09.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:00.393661+0000", 2024-03-20T09:12:09.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.19016908299999999 2024-03-20T09:12:09.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:12:09.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:00.393705+0000", 2024-03-20T09:12:09.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4472999999999999e-05 2024-03-20T09:12:09.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:12:09.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:00.393772+0000", 2024-03-20T09:12:09.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6700999999999997e-05 2024-03-20T09:12:09.126 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.126 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:12:09.126 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.126 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:108793 3.5 3:a0d369f5:::benchmark_data_smithi122_119671_object9011:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-03-20T09:12:09.126 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:05:15.975048+0000", 2024-03-20T09:12:09.126 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 412.95560867, 2024-03-20T09:12:09.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.34163258600000002, 2024-03-20T09:12:09.126 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:12:09.126 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:12:09.126 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:12:09.126 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:12:09.127 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:12:09.127 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 108793 2024-03-20T09:12:09.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.127 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:12:09.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:12:09.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:15.975048+0000", 2024-03-20T09:12:09.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:12:09.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:15.975048+0000", 2024-03-20T09:12:09.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:12:09.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:15.975050+0000", 2024-03-20T09:12:09.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1040000000000001e-06 2024-03-20T09:12:09.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:12:09.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:15.975062+0000", 2024-03-20T09:12:09.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1215e-05 2024-03-20T09:12:09.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:12:09.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:15.975089+0000", 2024-03-20T09:12:09.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7138e-05 2024-03-20T09:12:09.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:12:09.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:15.975103+0000", 2024-03-20T09:12:09.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4800000000000001e-05 2024-03-20T09:12:09.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:12:09.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:15.975174+0000", 2024-03-20T09:12:09.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0711999999999997e-05 2024-03-20T09:12:09.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:12:09.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:15.975288+0000", 2024-03-20T09:12:09.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011381499999999999 2024-03-20T09:12:09.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 6", 2024-03-20T09:12:09.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:15.975378+0000", 2024-03-20T09:12:09.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9722000000000006e-05 2024-03-20T09:12:09.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:12:09.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:15.976885+0000", 2024-03-20T09:12:09.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015074229999999999 2024-03-20T09:12:09.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:12:09.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:16.316567+0000", 2024-03-20T09:12:09.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.33968153000000001 2024-03-20T09:12:09.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:12:09.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:16.316612+0000", 2024-03-20T09:12:09.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5246e-05 2024-03-20T09:12:09.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:12:09.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:16.316681+0000", 2024-03-20T09:12:09.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8880999999999993e-05 2024-03-20T09:12:09.131 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.131 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:12:09.131 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.132 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:108830 3.5 3:a53c2502:::benchmark_data_smithi122_119671_object9048:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-03-20T09:12:09.132 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:05:16.395350+0000", 2024-03-20T09:12:09.132 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 412.535306725, 2024-03-20T09:12:09.132 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.21877197500000001, 2024-03-20T09:12:09.132 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:12:09.132 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:12:09.132 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:12:09.132 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:12:09.132 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:12:09.132 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 108830 2024-03-20T09:12:09.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.132 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:12:09.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.132 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:12:09.132 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:16.395350+0000", 2024-03-20T09:12:09.133 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.133 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:12:09.133 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:16.395350+0000", 2024-03-20T09:12:09.133 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.133 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:12:09.133 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:16.395352+0000", 2024-03-20T09:12:09.133 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.649e-06 2024-03-20T09:12:09.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.133 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:12:09.133 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:16.395360+0000", 2024-03-20T09:12:09.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9330000000000001e-06 2024-03-20T09:12:09.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.134 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:12:09.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:16.395374+0000", 2024-03-20T09:12:09.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4242e-05 2024-03-20T09:12:09.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.134 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:12:09.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:16.395383+0000", 2024-03-20T09:12:09.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7250000000000003e-06 2024-03-20T09:12:09.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.134 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:12:09.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:16.395433+0000", 2024-03-20T09:12:09.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0599000000000001e-05 2024-03-20T09:12:09.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:12:09.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:16.395530+0000", 2024-03-20T09:12:09.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.6222000000000001e-05 2024-03-20T09:12:09.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 6", 2024-03-20T09:12:09.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:16.395625+0000", 2024-03-20T09:12:09.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5713999999999999e-05 2024-03-20T09:12:09.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:12:09.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:16.397100+0000", 2024-03-20T09:12:09.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001475063 2024-03-20T09:12:09.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:12:09.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:16.614010+0000", 2024-03-20T09:12:09.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.21690953099999999 2024-03-20T09:12:09.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:12:09.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:16.614056+0000", 2024-03-20T09:12:09.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6372999999999998e-05 2024-03-20T09:12:09.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:12:09.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:16.614122+0000", 2024-03-20T09:12:09.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5924000000000001e-05 2024-03-20T09:12:09.137 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.137 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:12:09.137 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.137 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:109398 3.5 3:a4d68906:::benchmark_data_smithi122_119671_object9616:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-03-20T09:12:09.137 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:05:18.384322+0000", 2024-03-20T09:12:09.137 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 410.546334833, 2024-03-20T09:12:09.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.485667882, 2024-03-20T09:12:09.137 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:12:09.137 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:12:09.137 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:12:09.137 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:12:09.137 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:12:09.138 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 109398 2024-03-20T09:12:09.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.138 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:12:09.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.138 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:12:09.138 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.384322+0000", 2024-03-20T09:12:09.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.138 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:12:09.138 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.384322+0000", 2024-03-20T09:12:09.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:12:09.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.384324+0000", 2024-03-20T09:12:09.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9089999999999998e-06 2024-03-20T09:12:09.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:12:09.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.384334+0000", 2024-03-20T09:12:09.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.747e-06 2024-03-20T09:12:09.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:12:09.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.384362+0000", 2024-03-20T09:12:09.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8466000000000002e-05 2024-03-20T09:12:09.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:12:09.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.384377+0000", 2024-03-20T09:12:09.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4613999999999999e-05 2024-03-20T09:12:09.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:12:09.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.384533+0000", 2024-03-20T09:12:09.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015572700000000001 2024-03-20T09:12:09.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:12:09.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.384734+0000", 2024-03-20T09:12:09.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000201084 2024-03-20T09:12:09.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 6", 2024-03-20T09:12:09.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.384871+0000", 2024-03-20T09:12:09.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013722499999999999 2024-03-20T09:12:09.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:12:09.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.386260+0000", 2024-03-20T09:12:09.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001388692 2024-03-20T09:12:09.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:12:09.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.869855+0000", 2024-03-20T09:12:09.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.48359528000000002 2024-03-20T09:12:09.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.142 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:12:09.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.869903+0000", 2024-03-20T09:12:09.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7922e-05 2024-03-20T09:12:09.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.142 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:12:09.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.869990+0000", 2024-03-20T09:12:09.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7216e-05 2024-03-20T09:12:09.143 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.143 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:12:09.143 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.143 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:109418 3.5 3:a35c87e7:::benchmark_data_smithi122_119671_object9636:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-03-20T09:12:09.143 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:05:18.426141+0000", 2024-03-20T09:12:09.143 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 410.50451570400003, 2024-03-20T09:12:09.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.44398548999999998, 2024-03-20T09:12:09.143 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:12:09.143 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:12:09.143 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:12:09.144 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:12:09.144 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:12:09.144 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 109418 2024-03-20T09:12:09.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.144 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:12:09.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.144 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:12:09.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.426141+0000", 2024-03-20T09:12:09.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.144 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:12:09.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.426141+0000", 2024-03-20T09:12:09.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.145 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:12:09.145 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.426142+0000", 2024-03-20T09:12:09.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1200000000000001e-06 2024-03-20T09:12:09.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.145 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:12:09.145 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.426147+0000", 2024-03-20T09:12:09.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1889999999999998e-06 2024-03-20T09:12:09.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.145 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:12:09.145 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.426159+0000", 2024-03-20T09:12:09.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1321000000000001e-05 2024-03-20T09:12:09.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:12:09.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.426165+0000", 2024-03-20T09:12:09.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9170000000000003e-06 2024-03-20T09:12:09.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:12:09.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.426216+0000", 2024-03-20T09:12:09.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0930999999999997e-05 2024-03-20T09:12:09.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:12:09.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.426302+0000", 2024-03-20T09:12:09.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5848999999999997e-05 2024-03-20T09:12:09.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 6", 2024-03-20T09:12:09.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.426366+0000", 2024-03-20T09:12:09.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4558000000000001e-05 2024-03-20T09:12:09.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:12:09.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.428663+0000", 2024-03-20T09:12:09.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002296822 2024-03-20T09:12:09.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:12:09.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.870036+0000", 2024-03-20T09:12:09.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.44137311499999998 2024-03-20T09:12:09.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:12:09.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.870073+0000", 2024-03-20T09:12:09.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6913999999999999e-05 2024-03-20T09:12:09.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:12:09.149 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.870127+0000", 2024-03-20T09:12:09.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3754000000000001e-05 2024-03-20T09:12:09.149 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.149 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:12:09.149 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.149 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:109419 3.5 3:a237f191:::benchmark_data_smithi122_119671_object9637:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-03-20T09:12:09.149 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:05:18.426419+0000", 2024-03-20T09:12:09.149 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 410.50423798100002, 2024-03-20T09:12:09.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.443832067, 2024-03-20T09:12:09.149 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:12:09.149 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:12:09.150 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:12:09.150 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:12:09.150 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:12:09.150 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 109419 2024-03-20T09:12:09.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.150 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:12:09.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:12:09.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.426419+0000", 2024-03-20T09:12:09.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:12:09.151 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.426419+0000", 2024-03-20T09:12:09.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.151 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:12:09.151 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.426420+0000", 2024-03-20T09:12:09.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7400000000000003e-07 2024-03-20T09:12:09.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.151 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:12:09.151 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.426423+0000", 2024-03-20T09:12:09.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7919999999999999e-06 2024-03-20T09:12:09.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:12:09.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.426431+0000", 2024-03-20T09:12:09.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0390000000000005e-06 2024-03-20T09:12:09.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:12:09.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.426436+0000", 2024-03-20T09:12:09.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9200000000000003e-06 2024-03-20T09:12:09.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:12:09.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.427571+0000", 2024-03-20T09:12:09.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001134592 2024-03-20T09:12:09.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:12:09.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.427690+0000", 2024-03-20T09:12:09.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119178 2024-03-20T09:12:09.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 6", 2024-03-20T09:12:09.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.427756+0000", 2024-03-20T09:12:09.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6324999999999999e-05 2024-03-20T09:12:09.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:12:09.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.428748+0000", 2024-03-20T09:12:09.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00099116099999999995 2024-03-20T09:12:09.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:12:09.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.870159+0000", 2024-03-20T09:12:09.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.44141159899999999 2024-03-20T09:12:09.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:12:09.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.870192+0000", 2024-03-20T09:12:09.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2906e-05 2024-03-20T09:12:09.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:12:09.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.870251+0000", 2024-03-20T09:12:09.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8881000000000001e-05 2024-03-20T09:12:09.155 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.155 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:12:09.155 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.155 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:109467 3.5 3:a3b46402:::benchmark_data_smithi122_119671_object9685:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-03-20T09:12:09.155 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:05:18.937113+0000", 2024-03-20T09:12:09.155 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 409.99354351800002, 2024-03-20T09:12:09.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.27326526400000001, 2024-03-20T09:12:09.155 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:12:09.155 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:12:09.155 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:12:09.155 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:12:09.156 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:12:09.156 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 109467 2024-03-20T09:12:09.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.156 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:12:09.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.156 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:12:09.156 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.937113+0000", 2024-03-20T09:12:09.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.156 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:12:09.156 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.937113+0000", 2024-03-20T09:12:09.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.157 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:12:09.157 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.937115+0000", 2024-03-20T09:12:09.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4160000000000001e-06 2024-03-20T09:12:09.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.157 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:12:09.157 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.937121+0000", 2024-03-20T09:12:09.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.511e-06 2024-03-20T09:12:09.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.157 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:12:09.157 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.937138+0000", 2024-03-20T09:12:09.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7091e-05 2024-03-20T09:12:09.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:12:09.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.937147+0000", 2024-03-20T09:12:09.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5040000000000002e-06 2024-03-20T09:12:09.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:12:09.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.937210+0000", 2024-03-20T09:12:09.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2796e-05 2024-03-20T09:12:09.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:12:09.159 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.937302+0000", 2024-03-20T09:12:09.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1948999999999997e-05 2024-03-20T09:12:09.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.159 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 6", 2024-03-20T09:12:09.159 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.937372+0000", 2024-03-20T09:12:09.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0282000000000002e-05 2024-03-20T09:12:09.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.159 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:12:09.159 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.938821+0000", 2024-03-20T09:12:09.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001448646 2024-03-20T09:12:09.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:12:09.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.210267+0000", 2024-03-20T09:12:09.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.27144655600000001 2024-03-20T09:12:09.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:12:09.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.210311+0000", 2024-03-20T09:12:09.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3584000000000001e-05 2024-03-20T09:12:09.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:12:09.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.210379+0000", 2024-03-20T09:12:09.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7929000000000006e-05 2024-03-20T09:12:09.161 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.161 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:12:09.161 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.161 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:109469 3.5 3:a1819fce:::benchmark_data_smithi122_119671_object9687:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-03-20T09:12:09.161 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:05:18.956946+0000", 2024-03-20T09:12:09.161 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 409.973711068, 2024-03-20T09:12:09.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.28300225200000001, 2024-03-20T09:12:09.161 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:12:09.161 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:12:09.161 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:12:09.161 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:12:09.161 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:12:09.161 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 109469 2024-03-20T09:12:09.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.162 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:12:09.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.162 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:12:09.162 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.956946+0000", 2024-03-20T09:12:09.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.162 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:12:09.162 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.956946+0000", 2024-03-20T09:12:09.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:12:09.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.956947+0000", 2024-03-20T09:12:09.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4e-07 2024-03-20T09:12:09.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:12:09.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.956951+0000", 2024-03-20T09:12:09.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2429999999999999e-06 2024-03-20T09:12:09.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:12:09.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.956963+0000", 2024-03-20T09:12:09.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1967000000000001e-05 2024-03-20T09:12:09.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:12:09.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.956970+0000", 2024-03-20T09:12:09.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2409999999999998e-06 2024-03-20T09:12:09.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:12:09.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.957015+0000", 2024-03-20T09:12:09.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4705e-05 2024-03-20T09:12:09.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:12:09.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.957096+0000", 2024-03-20T09:12:09.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1509999999999997e-05 2024-03-20T09:12:09.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 6", 2024-03-20T09:12:09.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.957161+0000", 2024-03-20T09:12:09.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4233000000000007e-05 2024-03-20T09:12:09.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:12:09.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:18.958650+0000", 2024-03-20T09:12:09.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014894260000000001 2024-03-20T09:12:09.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:12:09.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.239839+0000", 2024-03-20T09:12:09.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.28118913099999998 2024-03-20T09:12:09.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:12:09.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.239884+0000", 2024-03-20T09:12:09.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4767e-05 2024-03-20T09:12:09.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:12:09.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:19.239948+0000", 2024-03-20T09:12:09.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4189000000000005e-05 2024-03-20T09:12:09.167 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.167 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:12:09.167 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.167 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:112147 3.5 3:a046c6af:::benchmark_data_smithi122_119671_object12365:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-03-20T09:12:09.167 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:05:44.577839+0000", 2024-03-20T09:12:09.167 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 384.35281829199999, 2024-03-20T09:12:09.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.210152642, 2024-03-20T09:12:09.167 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:12:09.167 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:12:09.167 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:12:09.167 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:12:09.168 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:12:09.168 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 112147 2024-03-20T09:12:09.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.168 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:12:09.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.168 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:12:09.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:44.577839+0000", 2024-03-20T09:12:09.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.168 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:12:09.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:44.577839+0000", 2024-03-20T09:12:09.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:12:09.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:44.577841+0000", 2024-03-20T09:12:09.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1799999999999999e-06 2024-03-20T09:12:09.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:12:09.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:44.577851+0000", 2024-03-20T09:12:09.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0583e-05 2024-03-20T09:12:09.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:12:09.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:44.577882+0000", 2024-03-20T09:12:09.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0783e-05 2024-03-20T09:12:09.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:12:09.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:44.577898+0000", 2024-03-20T09:12:09.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6152999999999999e-05 2024-03-20T09:12:09.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:12:09.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:44.578035+0000", 2024-03-20T09:12:09.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013663400000000001 2024-03-20T09:12:09.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:12:09.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:44.578129+0000", 2024-03-20T09:12:09.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3778999999999998e-05 2024-03-20T09:12:09.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 6", 2024-03-20T09:12:09.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:44.578207+0000", 2024-03-20T09:12:09.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8762000000000002e-05 2024-03-20T09:12:09.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:12:09.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:44.579812+0000", 2024-03-20T09:12:09.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0016045199999999999 2024-03-20T09:12:09.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:12:09.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:44.787827+0000", 2024-03-20T09:12:09.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.208014595 2024-03-20T09:12:09.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.172 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:12:09.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:44.787879+0000", 2024-03-20T09:12:09.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2794999999999999e-05 2024-03-20T09:12:09.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.172 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:12:09.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:44.787991+0000", 2024-03-20T09:12:09.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000111858 2024-03-20T09:12:09.172 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.172 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:12:09.172 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.173 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:112852 3.5 3:a610d192:::benchmark_data_smithi122_119671_object13070:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-03-20T09:12:09.173 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:05:52.112767+0000", 2024-03-20T09:12:09.173 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 376.817890116, 2024-03-20T09:12:09.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.18684830499999999, 2024-03-20T09:12:09.173 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:12:09.173 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:12:09.173 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:12:09.173 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:12:09.173 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:12:09.173 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 112852 2024-03-20T09:12:09.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.173 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:12:09.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:12:09.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:52.112767+0000", 2024-03-20T09:12:09.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:12:09.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:52.112767+0000", 2024-03-20T09:12:09.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:12:09.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:52.112769+0000", 2024-03-20T09:12:09.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0509999999999999e-06 2024-03-20T09:12:09.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:12:09.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:52.112776+0000", 2024-03-20T09:12:09.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.092e-06 2024-03-20T09:12:09.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:12:09.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:52.112792+0000", 2024-03-20T09:12:09.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6205000000000002e-05 2024-03-20T09:12:09.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:12:09.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:52.112811+0000", 2024-03-20T09:12:09.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9301e-05 2024-03-20T09:12:09.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.176 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:12:09.176 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:52.112887+0000", 2024-03-20T09:12:09.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5600999999999997e-05 2024-03-20T09:12:09.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.176 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:12:09.176 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:52.112991+0000", 2024-03-20T09:12:09.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010365 2024-03-20T09:12:09.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.176 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 6", 2024-03-20T09:12:09.176 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:52.113069+0000", 2024-03-20T09:12:09.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8434000000000002e-05 2024-03-20T09:12:09.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:12:09.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:52.114813+0000", 2024-03-20T09:12:09.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0017441780000000001 2024-03-20T09:12:09.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:12:09.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:52.299494+0000", 2024-03-20T09:12:09.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.184680288 2024-03-20T09:12:09.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:12:09.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:52.299541+0000", 2024-03-20T09:12:09.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7386999999999999e-05 2024-03-20T09:12:09.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:12:09.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:52.299615+0000", 2024-03-20T09:12:09.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4117999999999997e-05 2024-03-20T09:12:09.178 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.178 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:12:09.178 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.179 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:113187 3.5 3:a4480037:::benchmark_data_smithi122_119671_object13405:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-03-20T09:12:09.179 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:05:54.769977+0000", 2024-03-20T09:12:09.179 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 374.16068027900002, 2024-03-20T09:12:09.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.198150884, 2024-03-20T09:12:09.179 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:12:09.179 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:12:09.179 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:12:09.179 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:12:09.179 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:12:09.179 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 113187 2024-03-20T09:12:09.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.179 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:12:09.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.179 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:12:09.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:54.769977+0000", 2024-03-20T09:12:09.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:12:09.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:54.769977+0000", 2024-03-20T09:12:09.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:12:09.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:54.769978+0000", 2024-03-20T09:12:09.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6509999999999999e-06 2024-03-20T09:12:09.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:12:09.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:54.769985+0000", 2024-03-20T09:12:09.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6499999999999999e-06 2024-03-20T09:12:09.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:12:09.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:54.769998+0000", 2024-03-20T09:12:09.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3149999999999999e-05 2024-03-20T09:12:09.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:12:09.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:54.770005+0000", 2024-03-20T09:12:09.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7050000000000002e-06 2024-03-20T09:12:09.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:12:09.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:54.770062+0000", 2024-03-20T09:12:09.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7352000000000002e-05 2024-03-20T09:12:09.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:12:09.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:54.770153+0000", 2024-03-20T09:12:09.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1224999999999995e-05 2024-03-20T09:12:09.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 6", 2024-03-20T09:12:09.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:54.770225+0000", 2024-03-20T09:12:09.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1953999999999996e-05 2024-03-20T09:12:09.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:12:09.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:54.771951+0000", 2024-03-20T09:12:09.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0017258600000000001 2024-03-20T09:12:09.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:12:09.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:54.967935+0000", 2024-03-20T09:12:09.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.195983662 2024-03-20T09:12:09.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:12:09.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:54.968010+0000", 2024-03-20T09:12:09.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5122000000000003e-05 2024-03-20T09:12:09.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.184 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:12:09.184 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:05:54.968127+0000", 2024-03-20T09:12:09.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000117553 2024-03-20T09:12:09.184 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.184 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:12:09.184 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.184 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:114288 3.18 3:1eacd832:::benchmark_data_smithi122_119671_object14506:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e312)", 2024-03-20T09:12:09.184 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:06:06.923002+0000", 2024-03-20T09:12:09.184 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 362.007654683, 2024-03-20T09:12:09.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1374643129999999, 2024-03-20T09:12:09.184 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:12:09.184 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:12:09.185 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:12:09.185 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:12:09.185 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:12:09.185 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 114288 2024-03-20T09:12:09.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.185 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:12:09.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:12:09.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:06.923002+0000", 2024-03-20T09:12:09.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:12:09.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:06.923002+0000", 2024-03-20T09:12:09.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:12:09.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:06.923004+0000", 2024-03-20T09:12:09.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3039999999999999e-06 2024-03-20T09:12:09.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:12:09.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:06.923009+0000", 2024-03-20T09:12:09.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.113e-06 2024-03-20T09:12:09.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:12:09.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:06.923023+0000", 2024-03-20T09:12:09.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4161999999999999e-05 2024-03-20T09:12:09.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:12:09.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:06.923032+0000", 2024-03-20T09:12:09.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7679999999999997e-06 2024-03-20T09:12:09.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:12:09.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:06.923220+0000", 2024-03-20T09:12:09.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018834 2024-03-20T09:12:09.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:12:09.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:06.923229+0000", 2024-03-20T09:12:09.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1379999999999996e-06 2024-03-20T09:12:09.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:12:09.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:08.052174+0000", 2024-03-20T09:12:09.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.128944873 2024-03-20T09:12:09.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:12:09.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:08.052323+0000", 2024-03-20T09:12:09.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000149059 2024-03-20T09:12:09.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 1,8", 2024-03-20T09:12:09.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:08.052424+0000", 2024-03-20T09:12:09.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010063199999999999 2024-03-20T09:12:09.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:12:09.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:08.055104+0000", 2024-03-20T09:12:09.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026805230000000002 2024-03-20T09:12:09.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:12:09.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:08.056082+0000", 2024-03-20T09:12:09.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00097824200000000004 2024-03-20T09:12:09.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:12:09.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:08.058997+0000", 2024-03-20T09:12:09.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0029141750000000002 2024-03-20T09:12:09.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:12:09.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:08.059076+0000", 2024-03-20T09:12:09.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9017000000000006e-05 2024-03-20T09:12:09.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:12:09.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:08.060467+0000", 2024-03-20T09:12:09.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013909669999999999 2024-03-20T09:12:09.190 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.191 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:12:09.191 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.191 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:120194 3.5 3:a31c061c:::benchmark_data_smithi122_119671_object20412:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-20T09:12:09.191 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:06:57.257179+0000", 2024-03-20T09:12:09.191 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 311.67347749999999, 2024-03-20T09:12:09.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.186814859, 2024-03-20T09:12:09.191 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:12:09.191 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:12:09.191 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:12:09.191 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:12:09.192 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:12:09.192 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 120194 2024-03-20T09:12:09.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.192 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:12:09.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:12:09.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:57.257179+0000", 2024-03-20T09:12:09.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:12:09.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:57.257179+0000", 2024-03-20T09:12:09.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:12:09.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:57.257182+0000", 2024-03-20T09:12:09.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5289999999999998e-06 2024-03-20T09:12:09.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:12:09.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:57.257190+0000", 2024-03-20T09:12:09.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5480000000000002e-06 2024-03-20T09:12:09.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:12:09.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:57.257211+0000", 2024-03-20T09:12:09.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0635000000000001e-05 2024-03-20T09:12:09.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:12:09.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:57.257222+0000", 2024-03-20T09:12:09.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0969e-05 2024-03-20T09:12:09.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:12:09.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:57.257280+0000", 2024-03-20T09:12:09.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7624999999999998e-05 2024-03-20T09:12:09.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.195 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:12:09.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:57.257375+0000", 2024-03-20T09:12:09.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5254000000000007e-05 2024-03-20T09:12:09.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.195 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 6", 2024-03-20T09:12:09.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:57.257452+0000", 2024-03-20T09:12:09.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7000000000000001e-05 2024-03-20T09:12:09.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.195 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:12:09.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:57.259014+0000", 2024-03-20T09:12:09.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015619640000000001 2024-03-20T09:12:09.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.196 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:12:09.196 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:57.443890+0000", 2024-03-20T09:12:09.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.184876234 2024-03-20T09:12:09.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.196 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:12:09.196 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:57.443936+0000", 2024-03-20T09:12:09.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5729000000000002e-05 2024-03-20T09:12:09.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.196 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:12:09.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:57.443994+0000", 2024-03-20T09:12:09.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8372000000000003e-05 2024-03-20T09:12:09.197 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.197 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:12:09.197 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.197 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:120403 3.5 3:a5c54a04:::benchmark_data_smithi122_119671_object20621:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-03-20T09:12:09.197 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:06:59.089041+0000", 2024-03-20T09:12:09.197 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 309.84161604799999, 2024-03-20T09:12:09.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.18535172699999999, 2024-03-20T09:12:09.197 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:12:09.197 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:12:09.197 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:12:09.198 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:12:09.198 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:12:09.198 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 120403 2024-03-20T09:12:09.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.198 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:12:09.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:12:09.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:59.089041+0000", 2024-03-20T09:12:09.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:12:09.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:59.089041+0000", 2024-03-20T09:12:09.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:12:09.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:59.089043+0000", 2024-03-20T09:12:09.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7749999999999999e-06 2024-03-20T09:12:09.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:12:09.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:59.089051+0000", 2024-03-20T09:12:09.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9840000000000001e-06 2024-03-20T09:12:09.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:12:09.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:59.089075+0000", 2024-03-20T09:12:09.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4286000000000001e-05 2024-03-20T09:12:09.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:12:09.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:59.089085+0000", 2024-03-20T09:12:09.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0125e-05 2024-03-20T09:12:09.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:12:09.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:59.089172+0000", 2024-03-20T09:12:09.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6889000000000001e-05 2024-03-20T09:12:09.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:12:09.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:59.089280+0000", 2024-03-20T09:12:09.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000108156 2024-03-20T09:12:09.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 6", 2024-03-20T09:12:09.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:59.089403+0000", 2024-03-20T09:12:09.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000122767 2024-03-20T09:12:09.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:12:09.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:59.091170+0000", 2024-03-20T09:12:09.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0017667029999999999 2024-03-20T09:12:09.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:12:09.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:59.274224+0000", 2024-03-20T09:12:09.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.183054202 2024-03-20T09:12:09.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:12:09.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:59.274289+0000", 2024-03-20T09:12:09.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5204999999999997e-05 2024-03-20T09:12:09.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:12:09.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:59.274393+0000", 2024-03-20T09:12:09.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010363500000000001 2024-03-20T09:12:09.203 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.203 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:12:09.203 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.203 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:120408 3.5 3:a53c071b:::benchmark_data_smithi122_119671_object20626:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-03-20T09:12:09.203 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:06:59.096066+0000", 2024-03-20T09:12:09.203 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 309.834591042, 2024-03-20T09:12:09.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.17844858499999999, 2024-03-20T09:12:09.203 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:12:09.203 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:12:09.203 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:12:09.203 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:12:09.204 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:12:09.204 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 120408 2024-03-20T09:12:09.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.204 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:12:09.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:12:09.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:59.096066+0000", 2024-03-20T09:12:09.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:12:09.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:59.096066+0000", 2024-03-20T09:12:09.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:12:09.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:12:09.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:59.096067+0000", 2024-03-20T09:12:09.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3209999999999999e-06 2024-03-20T09:12:09.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:12:09.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:59.096074+0000", 2024-03-20T09:12:09.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7989999999999996e-06 2024-03-20T09:12:09.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:12:09.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:59.096089+0000", 2024-03-20T09:12:09.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.525e-05 2024-03-20T09:12:09.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:12:09.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:59.096098+0000", 2024-03-20T09:12:09.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9439999999999994e-06 2024-03-20T09:12:09.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:12:09.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:59.096148+0000", 2024-03-20T09:12:09.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9539000000000001e-05 2024-03-20T09:12:09.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:12:09.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:59.096247+0000", 2024-03-20T09:12:09.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8937999999999994e-05 2024-03-20T09:12:09.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 6", 2024-03-20T09:12:09.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:59.096317+0000", 2024-03-20T09:12:09.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0476999999999996e-05 2024-03-20T09:12:09.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:12:09.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:59.098070+0000", 2024-03-20T09:12:09.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001752717 2024-03-20T09:12:09.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:12:09.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:59.274428+0000", 2024-03-20T09:12:09.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.176358285 2024-03-20T09:12:09.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:12:09.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:59.274464+0000", 2024-03-20T09:12:09.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5490000000000001e-05 2024-03-20T09:12:09.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:12:09.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:12:09.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:12:09.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:06:59.274514+0000", 2024-03-20T09:12:09.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0825000000000002e-05 2024-03-20T09:12:09.209 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.209 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:12:09.209 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.209 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:12:09.209 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:12:09.209 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:12:09.209 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:12:09.250 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:09.275 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:12:09.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:09.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:12:09.381 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:12:09.416 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:09.440 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:12:09.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:09.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:12:09.544 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:12:09.570 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:09 smithi122 ceph-mon[73495]: pgmap v1153: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.2 GiB data, 9.7 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 258 op/s; 770/109122 objects degraded (0.706%); 4066/109122 objects misplaced (3.726%); 2.9 MiB/s, 59 objects/s recovering 2024-03-20T09:12:09.572 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:09.592 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:12:09.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:09.595 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:12:09.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:09 smithi115 ceph-mon[72476]: pgmap v1153: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.2 GiB data, 9.7 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 258 op/s; 770/109122 objects degraded (0.706%); 4066/109122 objects misplaced (3.726%); 2.9 MiB/s, 59 objects/s recovering 2024-03-20T09:12:09.678 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:09 smithi026 ceph-mon[68309]: pgmap v1153: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.2 GiB data, 9.7 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 258 op/s; 770/109122 objects degraded (0.706%); 4066/109122 objects misplaced (3.726%); 2.9 MiB/s, 59 objects/s recovering 2024-03-20T09:12:09.696 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:12:09.723 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:09.744 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:12:09.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:09.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:12:09.848 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:12:09.875 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:09.875 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:12:09.894 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:12:09.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:09.899 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:12:10.000 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:12:10.029 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:10.048 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-20T09:12:10.051 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:12:10.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:10.053 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:12:10.154 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:12:10.181 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:10.200 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:12:10.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:10.203 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:12:10.292 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.7 to 1 (10000) 2024-03-20T09:12:10.304 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:12:10.328 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:10 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3228912318' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:12:10.328 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:10 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:12:10.329 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:10.348 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:12:10.350 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:12:10.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:10.353 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:12:10.386 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:12:10.409 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:12:10.433 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:12:10.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:10 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3228912318' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:12:10.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:10 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:12:10.453 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:12:10.457 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:12:10.482 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:10.487 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:12:10.513 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:12:10.535 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:12:10.558 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:12:10.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:10.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:12:10.569 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:12:10.599 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:12:10.627 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:12:10.656 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:12:10.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:10 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3228912318' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:12:10.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:10 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:12:10.657 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:12:10.657 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:12:10.657 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:12:10.657 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:12:10.657 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:12:10.658 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:12:10.658 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:12:10.658 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:12:10.658 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:12:10.658 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:12:10.658 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:12:10.658 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:12:10.665 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:12:10.691 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:10.709 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:12:10.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:10.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:12:10.813 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:12:10.840 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:10.859 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:12:10.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:10.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:12:10.963 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:12:10.989 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:11.008 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:12:11.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:11.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:12:11.112 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:12:11.129 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 8 1 2024-03-20T09:12:11.139 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:11.157 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:12:11.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:11.159 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:12:11.260 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:12:11.288 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:11.307 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:12:11.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:11.310 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:12:11.327 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:12:11.411 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:12:11.438 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:11.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:11 smithi122 ceph-mon[73495]: pgmap v1154: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.2 GiB data, 9.7 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 280 op/s; 770/107928 objects degraded (0.713%); 4066/107928 objects misplaced (3.767%); 2.4 MiB/s, 50 objects/s recovering 2024-03-20T09:12:11.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:11 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-20T09:12:11.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:11 smithi122 ceph-mon[73495]: osdmap e488: 12 total, 12 up, 10 in 2024-03-20T09:12:11.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:11 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1696650409' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:12:11.457 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:12:11.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:11.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:12:11.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:11 smithi026 ceph-mon[68309]: pgmap v1154: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.2 GiB data, 9.7 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 280 op/s; 770/107928 objects degraded (0.713%); 4066/107928 objects misplaced (3.767%); 2.4 MiB/s, 50 objects/s recovering 2024-03-20T09:12:11.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:11 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-20T09:12:11.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:11 smithi026 ceph-mon[68309]: osdmap e488: 12 total, 12 up, 10 in 2024-03-20T09:12:11.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:11 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1696650409' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:12:11.562 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:12:11.588 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:11.606 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:12:11.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:11.609 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:12:11.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:11 smithi115 ceph-mon[72476]: pgmap v1154: 57 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 54 active+clean; 2.2 GiB data, 9.7 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 280 op/s; 770/107928 objects degraded (0.713%); 4066/107928 objects misplaced (3.767%); 2.4 MiB/s, 50 objects/s recovering 2024-03-20T09:12:11.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:11 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-20T09:12:11.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:11 smithi115 ceph-mon[72476]: osdmap e488: 12 total, 12 up, 10 in 2024-03-20T09:12:11.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:11 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1696650409' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:12:11.710 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:12:11.736 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:11.754 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:12:11.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:11.757 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:12:11.858 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:12:11.884 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:11.903 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:12:11.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:11.906 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:12:12.006 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:12:12.032 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:12.047 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:12:12.057 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:12:12.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:12.060 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:12:12.160 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:12:12.186 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:12.204 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:12:12.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:12.207 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:12:12.308 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:12:12.313 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:12 smithi026 ceph-mon[68309]: osdmap e489: 12 total, 12 up, 10 in 2024-03-20T09:12:12.314 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:12 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 770/107928 objects degraded (0.713%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:12:12.356 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:12.432 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:12:12.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:12.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:12:12.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:12 smithi122 ceph-mon[73495]: osdmap e489: 12 total, 12 up, 10 in 2024-03-20T09:12:12.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:12 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 770/107928 objects degraded (0.713%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:12:12.535 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:12:12.572 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:12.597 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:12:12.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:12.600 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:12:12.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:12 smithi115 ceph-mon[72476]: osdmap e489: 12 total, 12 up, 10 in 2024-03-20T09:12:12.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:12 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 770/107928 objects degraded (0.713%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:12:12.701 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:12:12.778 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:13.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:13 smithi115 ceph-mon[72476]: pgmap v1157: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 3 remapped+peering, 2 unknown, 50 active+clean; 2.1 GiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 551 op/s; 3421/102384 objects degraded (3.341%); 1118/102384 objects misplaced (1.092%); 4.5 MiB/s, 93 objects/s recovering 2024-03-20T09:12:13.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:13 smithi115 ceph-mon[72476]: osdmap e490: 12 total, 12 up, 10 in 2024-03-20T09:12:13.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:13 smithi115 ceph-mon[72476]: 3.f starting backfill to osd.7 from (0'0,0'0] MAX to 487'10189 2024-03-20T09:12:13.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:13 smithi122 ceph-mon[73495]: pgmap v1157: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 3 remapped+peering, 2 unknown, 50 active+clean; 2.1 GiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 551 op/s; 3421/102384 objects degraded (3.341%); 1118/102384 objects misplaced (1.092%); 4.5 MiB/s, 93 objects/s recovering 2024-03-20T09:12:13.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:13 smithi122 ceph-mon[73495]: osdmap e490: 12 total, 12 up, 10 in 2024-03-20T09:12:13.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:13 smithi122 ceph-mon[73495]: 3.f starting backfill to osd.7 from (0'0,0'0] MAX to 487'10189 2024-03-20T09:12:13.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:13 smithi026 ceph-mon[68309]: pgmap v1157: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 3 remapped+peering, 2 unknown, 50 active+clean; 2.1 GiB data, 9.1 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 551 op/s; 3421/102384 objects degraded (3.341%); 1118/102384 objects misplaced (1.092%); 4.5 MiB/s, 93 objects/s recovering 2024-03-20T09:12:13.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:13 smithi026 ceph-mon[68309]: osdmap e490: 12 total, 12 up, 10 in 2024-03-20T09:12:13.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:13 smithi026 ceph-mon[68309]: 3.f starting backfill to osd.7 from (0'0,0'0] MAX to 487'10189 2024-03-20T09:12:14.087 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:12:14.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:14 smithi026 ceph-mon[68309]: osdmap e491: 12 total, 12 up, 10 in 2024-03-20T09:12:14.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:14 smithi115 ceph-mon[72476]: osdmap e491: 12 total, 12 up, 10 in 2024-03-20T09:12:14.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:14 smithi122 ceph-mon[73495]: osdmap e491: 12 total, 12 up, 10 in 2024-03-20T09:12:15.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:15 smithi115 ceph-mon[72476]: pgmap v1160: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 4 remapped+peering, 2 unknown, 49 active+clean; 2.1 GiB data, 9.0 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 588 op/s; 3421/100854 objects degraded (3.392%); 1118/100854 objects misplaced (1.109%); 2.2 MiB/s, 47 objects/s recovering 2024-03-20T09:12:15.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:15 smithi115 ceph-mon[72476]: osdmap e492: 12 total, 12 up, 10 in 2024-03-20T09:12:15.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:15 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:12:15.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:15 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:12:15.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:15 smithi122 ceph-mon[73495]: pgmap v1160: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 4 remapped+peering, 2 unknown, 49 active+clean; 2.1 GiB data, 9.0 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 588 op/s; 3421/100854 objects degraded (3.392%); 1118/100854 objects misplaced (1.109%); 2.2 MiB/s, 47 objects/s recovering 2024-03-20T09:12:15.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:15 smithi122 ceph-mon[73495]: osdmap e492: 12 total, 12 up, 10 in 2024-03-20T09:12:15.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:15 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:12:15.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:15 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:12:15.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:15 smithi026 ceph-mon[68309]: pgmap v1160: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 4 remapped+peering, 2 unknown, 49 active+clean; 2.1 GiB data, 9.0 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 588 op/s; 3421/100854 objects degraded (3.392%); 1118/100854 objects misplaced (1.109%); 2.2 MiB/s, 47 objects/s recovering 2024-03-20T09:12:15.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:15 smithi026 ceph-mon[68309]: osdmap e492: 12 total, 12 up, 10 in 2024-03-20T09:12:15.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:15 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:12:15.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:15 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:12:16.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:16 smithi115 ceph-mon[72476]: osdmap e493: 12 total, 12 up, 10 in 2024-03-20T09:12:16.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:16 smithi122 ceph-mon[73495]: osdmap e493: 12 total, 12 up, 10 in 2024-03-20T09:12:16.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:16 smithi026 ceph-mon[68309]: osdmap e493: 12 total, 12 up, 10 in 2024-03-20T09:12:17.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:17 smithi115 ceph-mon[72476]: pgmap v1163: 57 pgs: 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 4 remapped+peering, 2 unknown, 47 active+clean; 2.0 GiB data, 8.8 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 347 op/s; 3423/98217 objects degraded (3.485%); 3989/98217 objects misplaced (4.061%); 3.5 MiB/s, 55 objects/s recovering 2024-03-20T09:12:17.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:17 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 3423/98217 objects degraded (3.485%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:12:17.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:17 smithi122 ceph-mon[73495]: pgmap v1163: 57 pgs: 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 4 remapped+peering, 2 unknown, 47 active+clean; 2.0 GiB data, 8.8 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 347 op/s; 3423/98217 objects degraded (3.485%); 3989/98217 objects misplaced (4.061%); 3.5 MiB/s, 55 objects/s recovering 2024-03-20T09:12:17.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:17 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 3423/98217 objects degraded (3.485%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:12:17.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:17 smithi026 ceph-mon[68309]: pgmap v1163: 57 pgs: 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 4 remapped+peering, 2 unknown, 47 active+clean; 2.0 GiB data, 8.8 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 347 op/s; 3423/98217 objects degraded (3.485%); 3989/98217 objects misplaced (4.061%); 3.5 MiB/s, 55 objects/s recovering 2024-03-20T09:12:17.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:17 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 3423/98217 objects degraded (3.485%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:12:19.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:19 smithi115 ceph-mon[72476]: pgmap v1164: 57 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 50 active+clean; 1.9 GiB data, 8.7 GiB used, 885 GiB / 894 GiB avail; 560 B/s rd, 12 KiB/s wr, 444 op/s; 6760/93792 objects degraded (7.207%); 5291/93792 objects misplaced (5.641%); 9.9 MiB/s, 166 objects/s recovering 2024-03-20T09:12:19.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:19 smithi122 ceph-mon[73495]: pgmap v1164: 57 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 50 active+clean; 1.9 GiB data, 8.7 GiB used, 885 GiB / 894 GiB avail; 560 B/s rd, 12 KiB/s wr, 444 op/s; 6760/93792 objects degraded (7.207%); 5291/93792 objects misplaced (5.641%); 9.9 MiB/s, 166 objects/s recovering 2024-03-20T09:12:19.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:19 smithi026 ceph-mon[68309]: pgmap v1164: 57 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 50 active+clean; 1.9 GiB data, 8.7 GiB used, 885 GiB / 894 GiB avail; 560 B/s rd, 12 KiB/s wr, 444 op/s; 6760/93792 objects degraded (7.207%); 5291/93792 objects misplaced (5.641%); 9.9 MiB/s, 166 objects/s recovering 2024-03-20T09:12:21.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:21 smithi115 ceph-mon[72476]: pgmap v1165: 57 pgs: 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 50 active+clean; 1.9 GiB data, 8.6 GiB used, 885 GiB / 894 GiB avail; 474 B/s rd, 10 KiB/s wr, 463 op/s; 6760/92118 objects degraded (7.338%); 5037/92118 objects misplaced (5.468%); 8.4 MiB/s, 140 objects/s recovering 2024-03-20T09:12:21.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:21 smithi122 ceph-mon[73495]: pgmap v1165: 57 pgs: 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 50 active+clean; 1.9 GiB data, 8.6 GiB used, 885 GiB / 894 GiB avail; 474 B/s rd, 10 KiB/s wr, 463 op/s; 6760/92118 objects degraded (7.338%); 5037/92118 objects misplaced (5.468%); 8.4 MiB/s, 140 objects/s recovering 2024-03-20T09:12:21.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:21 smithi026 ceph-mon[68309]: pgmap v1165: 57 pgs: 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 50 active+clean; 1.9 GiB data, 8.6 GiB used, 885 GiB / 894 GiB avail; 474 B/s rd, 10 KiB/s wr, 463 op/s; 6760/92118 objects degraded (7.338%); 5037/92118 objects misplaced (5.468%); 8.4 MiB/s, 140 objects/s recovering 2024-03-20T09:12:22.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:22 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 6760/92118 objects degraded (7.338%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:12:22.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:22 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 6760/92118 objects degraded (7.338%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:12:22.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:22 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 6760/92118 objects degraded (7.338%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:12:23.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:23 smithi115 ceph-mon[72476]: pgmap v1166: 57 pgs: 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 50 active+clean; 1.7 GiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 383 B/s rd, 8.4 KiB/s wr, 661 op/s; 6985/84990 objects degraded (8.219%); 3668/84990 objects misplaced (4.316%); 16 MiB/s, 267 objects/s recovering 2024-03-20T09:12:23.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:23 smithi122 ceph-mon[73495]: pgmap v1166: 57 pgs: 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 50 active+clean; 1.7 GiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 383 B/s rd, 8.4 KiB/s wr, 661 op/s; 6985/84990 objects degraded (8.219%); 3668/84990 objects misplaced (4.316%); 16 MiB/s, 267 objects/s recovering 2024-03-20T09:12:23.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:23 smithi026 ceph-mon[68309]: pgmap v1166: 57 pgs: 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 50 active+clean; 1.7 GiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 383 B/s rd, 8.4 KiB/s wr, 661 op/s; 6985/84990 objects degraded (8.219%); 3668/84990 objects misplaced (4.316%); 16 MiB/s, 267 objects/s recovering 2024-03-20T09:12:24.466 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:12:24.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:24.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:12:24.579 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:12:24.614 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:24.649 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:12:24.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:24.652 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:12:24.753 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:12:24.783 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:24.805 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:12:24.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:24.812 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:12:24.913 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:12:24.956 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:24.979 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:12:24.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:24.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:12:25.082 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:12:25.139 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:25.233 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:12:25.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:25.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:12:25.323 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:12:25.338 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:12:25.374 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:25.398 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:12:25.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:25.403 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:12:25.438 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:25 smithi026 ceph-mon[68309]: pgmap v1167: 57 pgs: 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 50 active+clean; 1.7 GiB data, 8.1 GiB used, 886 GiB / 894 GiB avail; 324 B/s rd, 7.1 KiB/s wr, 648 op/s; 6985/82452 objects degraded (8.472%); 3668/82452 objects misplaced (4.449%); 13 MiB/s, 225 objects/s recovering 2024-03-20T09:12:25.504 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:12:25.544 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:25.574 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:12:25.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:25.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:12:25.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:25 smithi115 ceph-mon[72476]: pgmap v1167: 57 pgs: 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 50 active+clean; 1.7 GiB data, 8.1 GiB used, 886 GiB / 894 GiB avail; 324 B/s rd, 7.1 KiB/s wr, 648 op/s; 6985/82452 objects degraded (8.472%); 3668/82452 objects misplaced (4.449%); 13 MiB/s, 225 objects/s recovering 2024-03-20T09:12:25.679 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:12:25.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:25 smithi122 ceph-mon[73495]: pgmap v1167: 57 pgs: 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 50 active+clean; 1.7 GiB data, 8.1 GiB used, 886 GiB / 894 GiB avail; 324 B/s rd, 7.1 KiB/s wr, 648 op/s; 6985/82452 objects degraded (8.472%); 3668/82452 objects misplaced (4.449%); 13 MiB/s, 225 objects/s recovering 2024-03-20T09:12:25.709 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:25.738 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:12:25.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:25.741 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:12:25.842 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:12:25.870 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:25.891 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:12:25.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:25.894 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:12:25.996 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:12:26.027 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:26.050 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:12:26.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:26.054 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:12:26.155 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:12:26.186 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:26.207 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:12:26.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:26.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:12:26.312 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:12:26.408 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:27.183 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:12:27.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:27.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:12:27.290 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:12:27.344 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:12:27.344 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:12:27.352 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:27.406 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:12:27.438 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:27 smithi026 ceph-mon[68309]: pgmap v1168: 57 pgs: 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 50 active+clean; 1.6 GiB data, 7.8 GiB used, 886 GiB / 894 GiB avail; 293 B/s rd, 6.4 KiB/s wr, 687 op/s; 6985/77439 objects degraded (9.020%); 3138/77439 objects misplaced (4.052%); 12 MiB/s, 244 objects/s recovering 2024-03-20T09:12:27.438 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:27 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3522991065' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:12:27.438 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:27 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 6985/77439 objects degraded (9.020%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:12:27.438 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:27 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:12:27.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:27 smithi115 ceph-mon[72476]: pgmap v1168: 57 pgs: 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 50 active+clean; 1.6 GiB data, 7.8 GiB used, 886 GiB / 894 GiB avail; 293 B/s rd, 6.4 KiB/s wr, 687 op/s; 6985/77439 objects degraded (9.020%); 3138/77439 objects misplaced (4.052%); 12 MiB/s, 244 objects/s recovering 2024-03-20T09:12:27.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:27 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3522991065' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:12:27.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:27 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 6985/77439 objects degraded (9.020%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:12:27.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:27 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:12:27.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:27 smithi122 ceph-mon[73495]: pgmap v1168: 57 pgs: 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 50 active+clean; 1.6 GiB data, 7.8 GiB used, 886 GiB / 894 GiB avail; 293 B/s rd, 6.4 KiB/s wr, 687 op/s; 6985/77439 objects degraded (9.020%); 3138/77439 objects misplaced (4.052%); 12 MiB/s, 244 objects/s recovering 2024-03-20T09:12:27.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:27 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3522991065' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:12:27.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:27 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 6985/77439 objects degraded (9.020%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:12:27.731 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:27 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:12:28.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:28 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:12:28.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:28 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3522991065' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:12:28.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:28 smithi115 ceph-mon[72476]: osdmap e494: 12 total, 12 up, 10 in 2024-03-20T09:12:28.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:28 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:12:28.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:28 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3522991065' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:12:28.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:28 smithi122 ceph-mon[73495]: osdmap e494: 12 total, 12 up, 10 in 2024-03-20T09:12:28.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:28 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:12:28.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:28 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3522991065' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:12:28.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:28 smithi026 ceph-mon[68309]: osdmap e494: 12 total, 12 up, 10 in 2024-03-20T09:12:29.481 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-20T09:12:29.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:29 smithi115 ceph-mon[72476]: pgmap v1170: 57 pgs: 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 50 active+clean; 1.5 GiB data, 7.5 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 750 op/s; 7325/71262 objects degraded (10.279%); 2788/71262 objects misplaced (3.912%); 9.0 MiB/s, 277 objects/s recovering 2024-03-20T09:12:29.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:29 smithi122 ceph-mon[73495]: pgmap v1170: 57 pgs: 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 50 active+clean; 1.5 GiB data, 7.5 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 750 op/s; 7325/71262 objects degraded (10.279%); 2788/71262 objects misplaced (3.912%); 9.0 MiB/s, 277 objects/s recovering 2024-03-20T09:12:29.724 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:29 smithi026 ceph-mon[68309]: pgmap v1170: 57 pgs: 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 50 active+clean; 1.5 GiB data, 7.5 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 750 op/s; 7325/71262 objects degraded (10.279%); 2788/71262 objects misplaced (3.912%); 9.0 MiB/s, 277 objects/s recovering 2024-03-20T09:12:30.090 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:12:30.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:30.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:12:30.192 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:12:30.242 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:30.267 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:12:30.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:30.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:12:30.373 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:12:30.398 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:12:30.408 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:30.431 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:12:30.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:30.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:12:30.535 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:12:30.571 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:30.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:30 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:12:30.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:30 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:12:30.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:30 smithi115 ceph-mon[72476]: osdmap e495: 12 total, 12 up, 10 in 2024-03-20T09:12:30.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:30 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:12:30.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:30 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:12:30.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:30 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:12:30.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:30 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:12:30.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:30 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:12:30.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:30 smithi026 ceph-mon[68309]: osdmap e495: 12 total, 12 up, 10 in 2024-03-20T09:12:30.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:30 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:12:30.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:30 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:12:30.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:30 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:12:30.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:30 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:12:30.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:30 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:12:30.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:30 smithi122 ceph-mon[73495]: osdmap e495: 12 total, 12 up, 10 in 2024-03-20T09:12:30.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:30 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:12:30.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:30 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:12:30.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:30 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:12:31.888 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:12:31.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:31 smithi115 ceph-mon[72476]: pgmap v1172: 57 pgs: 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 50 active+clean; 1.4 GiB data, 7.3 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 644 op/s; 7325/69513 objects degraded (10.538%); 2557/69513 objects misplaced (3.678%); 2.3 MiB/s, 193 objects/s recovering 2024-03-20T09:12:31.934 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:31 smithi026 ceph-mon[68309]: pgmap v1172: 57 pgs: 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 50 active+clean; 1.4 GiB data, 7.3 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 644 op/s; 7325/69513 objects degraded (10.538%); 2557/69513 objects misplaced (3.678%); 2.3 MiB/s, 193 objects/s recovering 2024-03-20T09:12:31.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:31 smithi122 ceph-mon[73495]: pgmap v1172: 57 pgs: 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 50 active+clean; 1.4 GiB data, 7.3 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 644 op/s; 7325/69513 objects degraded (10.538%); 2557/69513 objects misplaced (3.678%); 2.3 MiB/s, 193 objects/s recovering 2024-03-20T09:12:32.309 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:12:32.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:32.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:12:32.344 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:12:32.424 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:12:32.454 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:32.481 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:12:32.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:32.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:12:32.585 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:12:32.665 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:32.684 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:12:32.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:32.687 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:12:32.788 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:12:32.814 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:32 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 7325/69513 objects degraded (10.538%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:12:32.814 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:32 smithi115 ceph-mon[72476]: osdmap e496: 12 total, 12 up, 10 in 2024-03-20T09:12:32.816 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:32.836 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:12:32.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:32.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:12:32.940 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:12:32.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:32 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 7325/69513 objects degraded (10.538%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:12:32.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:32 smithi122 ceph-mon[73495]: osdmap e496: 12 total, 12 up, 10 in 2024-03-20T09:12:32.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:32 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 7325/69513 objects degraded (10.538%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:12:32.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:32 smithi026 ceph-mon[68309]: osdmap e496: 12 total, 12 up, 10 in 2024-03-20T09:12:32.968 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:32.990 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:12:32.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:32.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:12:33.096 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:12:33.136 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:33.159 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:12:33.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:33.164 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:12:33.265 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:12:33.295 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:33.316 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:12:33.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:33.320 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:12:33.421 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:12:33.476 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:33.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:33 smithi115 ceph-mon[72476]: pgmap v1173: 57 pgs: 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 50 active+clean; 1.3 GiB data, 6.9 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 856 op/s; 7670/61890 objects degraded (12.393%); 2184/61890 objects misplaced (3.529%); 2.5 MiB/s, 427 objects/s recovering 2024-03-20T09:12:33.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:33 smithi115 ceph-mon[72476]: 1.1e scrub starts 2024-03-20T09:12:33.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:33 smithi115 ceph-mon[72476]: 1.1e scrub ok 2024-03-20T09:12:33.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:33 smithi115 ceph-mon[72476]: 3.a scrub starts 2024-03-20T09:12:33.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:33 smithi115 ceph-mon[72476]: 3.a scrub ok 2024-03-20T09:12:33.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:33 smithi115 ceph-mon[72476]: osdmap e497: 12 total, 12 up, 10 in 2024-03-20T09:12:33.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:33 smithi122 ceph-mon[73495]: pgmap v1173: 57 pgs: 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 50 active+clean; 1.3 GiB data, 6.9 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 856 op/s; 7670/61890 objects degraded (12.393%); 2184/61890 objects misplaced (3.529%); 2.5 MiB/s, 427 objects/s recovering 2024-03-20T09:12:33.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:33 smithi122 ceph-mon[73495]: 1.1e scrub starts 2024-03-20T09:12:33.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:33 smithi122 ceph-mon[73495]: 1.1e scrub ok 2024-03-20T09:12:33.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:33 smithi122 ceph-mon[73495]: 3.a scrub starts 2024-03-20T09:12:33.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:33 smithi122 ceph-mon[73495]: 3.a scrub ok 2024-03-20T09:12:33.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:33 smithi122 ceph-mon[73495]: osdmap e497: 12 total, 12 up, 10 in 2024-03-20T09:12:33.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:33 smithi026 ceph-mon[68309]: pgmap v1173: 57 pgs: 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 50 active+clean; 1.3 GiB data, 6.9 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 856 op/s; 7670/61890 objects degraded (12.393%); 2184/61890 objects misplaced (3.529%); 2.5 MiB/s, 427 objects/s recovering 2024-03-20T09:12:33.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:33 smithi026 ceph-mon[68309]: 1.1e scrub starts 2024-03-20T09:12:33.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:33 smithi026 ceph-mon[68309]: 1.1e scrub ok 2024-03-20T09:12:33.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:33 smithi026 ceph-mon[68309]: 3.a scrub starts 2024-03-20T09:12:33.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:33 smithi026 ceph-mon[68309]: 3.a scrub ok 2024-03-20T09:12:33.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:33 smithi026 ceph-mon[68309]: osdmap e497: 12 total, 12 up, 10 in 2024-03-20T09:12:34.756 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:12:34.765 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:12:34.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:34.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:12:34.878 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:12:34.917 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:34.951 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:12:34.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:34.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:12:35.062 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:12:35.100 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:35.125 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:12:35.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:35.130 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:12:35.231 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:12:35.417 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:35.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:35 smithi026 ceph-mon[68309]: pgmap v1176: 57 pgs: 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 50 active+clean; 1.2 GiB data, 6.8 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 657 op/s; 7669/59418 objects degraded (12.907%); 2184/59418 objects misplaced (3.676%); 181 KiB/s, 311 objects/s recovering 2024-03-20T09:12:35.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:35 smithi115 ceph-mon[72476]: pgmap v1176: 57 pgs: 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 50 active+clean; 1.2 GiB data, 6.8 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 657 op/s; 7669/59418 objects degraded (12.907%); 2184/59418 objects misplaced (3.676%); 181 KiB/s, 311 objects/s recovering 2024-03-20T09:12:35.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:35 smithi122 ceph-mon[73495]: pgmap v1176: 57 pgs: 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 50 active+clean; 1.2 GiB data, 6.8 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 657 op/s; 7669/59418 objects degraded (12.907%); 2184/59418 objects misplaced (3.676%); 181 KiB/s, 311 objects/s recovering 2024-03-20T09:12:36.496 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:12:36.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:36.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:12:36.594 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.8 to 1 (10000) 2024-03-20T09:12:36.601 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:12:36.631 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:36.658 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:12:36.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:36.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:12:36.759 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:36 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:12:36.759 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:36 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:12:36.759 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:36 smithi026 ceph-mon[68309]: osdmap e498: 12 total, 12 up, 10 in 2024-03-20T09:12:36.759 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:36 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2512305155' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:12:36.759 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:36 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:12:36.764 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:12:36.791 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:36.814 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:12:36.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:36 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:12:36.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:36 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:12:36.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:36 smithi115 ceph-mon[72476]: osdmap e498: 12 total, 12 up, 10 in 2024-03-20T09:12:36.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:36 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2512305155' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:12:36.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:36 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:12:37.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:37.226 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:12:37.229 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:36 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:12:37.229 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:36 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:12:37.229 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:36 smithi122 ceph-mon[73495]: osdmap e498: 12 total, 12 up, 10 in 2024-03-20T09:12:37.230 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:36 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2512305155' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:12:37.230 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:36 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:12:37.328 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:12:37.615 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:37.634 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:12:37.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:37.639 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:12:37.740 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:12:37.789 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:37.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:37 smithi115 ceph-mon[72476]: pgmap v1178: 57 pgs: 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 50 active+clean; 1.1 GiB data, 6.5 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 823 op/s; 7669/54678 objects degraded (14.026%); 1943/54678 objects misplaced (3.554%); 181 KiB/s, 463 objects/s recovering 2024-03-20T09:12:37.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:37 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-03-20T09:12:37.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:37 smithi115 ceph-mon[72476]: osdmap e499: 12 total, 12 up, 10 in 2024-03-20T09:12:37.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:37 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 7669/54678 objects degraded (14.026%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:12:37.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:37 smithi122 ceph-mon[73495]: pgmap v1178: 57 pgs: 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 50 active+clean; 1.1 GiB data, 6.5 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 823 op/s; 7669/54678 objects degraded (14.026%); 1943/54678 objects misplaced (3.554%); 181 KiB/s, 463 objects/s recovering 2024-03-20T09:12:37.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:37 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-03-20T09:12:37.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:37 smithi122 ceph-mon[73495]: osdmap e499: 12 total, 12 up, 10 in 2024-03-20T09:12:37.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:37 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 7669/54678 objects degraded (14.026%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:12:37.956 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:37 smithi026 ceph-mon[68309]: pgmap v1178: 57 pgs: 1 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 50 active+clean; 1.1 GiB data, 6.5 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 823 op/s; 7669/54678 objects degraded (14.026%); 1943/54678 objects misplaced (3.554%); 181 KiB/s, 463 objects/s recovering 2024-03-20T09:12:37.956 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:37 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-03-20T09:12:37.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:37 smithi026 ceph-mon[68309]: osdmap e499: 12 total, 12 up, 10 in 2024-03-20T09:12:37.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:37 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 7669/54678 objects degraded (14.026%), 6 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:12:38.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:38 smithi115 ceph-mon[72476]: 1.17 scrub starts 2024-03-20T09:12:38.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:38 smithi115 ceph-mon[72476]: 1.17 scrub ok 2024-03-20T09:12:38.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:38 smithi122 ceph-mon[73495]: 1.17 scrub starts 2024-03-20T09:12:38.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:38 smithi122 ceph-mon[73495]: 1.17 scrub ok 2024-03-20T09:12:38.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:38 smithi026 ceph-mon[68309]: 1.17 scrub starts 2024-03-20T09:12:38.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:38 smithi026 ceph-mon[68309]: 1.17 scrub ok 2024-03-20T09:12:39.835 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:39 smithi026 ceph-mon[68309]: pgmap v1180: 57 pgs: 1 active+remapped+backfill_wait, 5 active+recovering+undersized+degraded+remapped, 51 active+clean; 1.0 GiB data, 6.2 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 731 op/s; 2649/49317 objects degraded (5.371%); 2135/49317 objects misplaced (4.329%); 7.0 MiB/s, 323 objects/s recovering 2024-03-20T09:12:39.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:39 smithi115 ceph-mon[72476]: pgmap v1180: 57 pgs: 1 active+remapped+backfill_wait, 5 active+recovering+undersized+degraded+remapped, 51 active+clean; 1.0 GiB data, 6.2 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 731 op/s; 2649/49317 objects degraded (5.371%); 2135/49317 objects misplaced (4.329%); 7.0 MiB/s, 323 objects/s recovering 2024-03-20T09:12:39.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:39 smithi122 ceph-mon[73495]: pgmap v1180: 57 pgs: 1 active+remapped+backfill_wait, 5 active+recovering+undersized+degraded+remapped, 51 active+clean; 1.0 GiB data, 6.2 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 731 op/s; 2649/49317 objects degraded (5.371%); 2135/49317 objects misplaced (4.329%); 7.0 MiB/s, 323 objects/s recovering 2024-03-20T09:12:41.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:41 smithi115 ceph-mon[72476]: pgmap v1181: 57 pgs: 1 active+remapped+backfill_wait, 5 active+recovering+undersized+degraded+remapped, 51 active+clean; 989 MiB data, 6.1 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 608 op/s; 2649/47448 objects degraded (5.583%); 1888/47448 objects misplaced (3.979%); 5.9 MiB/s, 275 objects/s recovering 2024-03-20T09:12:41.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:41 smithi115 ceph-mon[72476]: 1.1b scrub starts 2024-03-20T09:12:41.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:41 smithi115 ceph-mon[72476]: 1.1b scrub ok 2024-03-20T09:12:41.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:41 smithi115 ceph-mon[72476]: 1.f scrub starts 2024-03-20T09:12:41.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:41 smithi115 ceph-mon[72476]: 1.f scrub ok 2024-03-20T09:12:41.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:41 smithi122 ceph-mon[73495]: pgmap v1181: 57 pgs: 1 active+remapped+backfill_wait, 5 active+recovering+undersized+degraded+remapped, 51 active+clean; 989 MiB data, 6.1 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 608 op/s; 2649/47448 objects degraded (5.583%); 1888/47448 objects misplaced (3.979%); 5.9 MiB/s, 275 objects/s recovering 2024-03-20T09:12:41.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:41 smithi122 ceph-mon[73495]: 1.1b scrub starts 2024-03-20T09:12:41.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:41 smithi122 ceph-mon[73495]: 1.1b scrub ok 2024-03-20T09:12:41.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:41 smithi122 ceph-mon[73495]: 1.f scrub starts 2024-03-20T09:12:41.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:41 smithi122 ceph-mon[73495]: 1.f scrub ok 2024-03-20T09:12:41.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:41 smithi026 ceph-mon[68309]: pgmap v1181: 57 pgs: 1 active+remapped+backfill_wait, 5 active+recovering+undersized+degraded+remapped, 51 active+clean; 989 MiB data, 6.1 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 608 op/s; 2649/47448 objects degraded (5.583%); 1888/47448 objects misplaced (3.979%); 5.9 MiB/s, 275 objects/s recovering 2024-03-20T09:12:41.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:41 smithi026 ceph-mon[68309]: 1.1b scrub starts 2024-03-20T09:12:41.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:41 smithi026 ceph-mon[68309]: 1.1b scrub ok 2024-03-20T09:12:41.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:41 smithi026 ceph-mon[68309]: 1.f scrub starts 2024-03-20T09:12:41.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:41 smithi026 ceph-mon[68309]: 1.f scrub ok 2024-03-20T09:12:42.203 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 10 1 2024-03-20T09:12:42.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:42 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 2649/47448 objects degraded (5.583%), 5 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:12:42.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:42 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 2649/47448 objects degraded (5.583%), 5 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:12:42.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:42 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 2649/47448 objects degraded (5.583%), 5 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:12:43.678 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:12:43.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:43.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:12:43.786 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:12:43.866 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:43.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:43 smithi115 ceph-mon[72476]: pgmap v1182: 57 pgs: 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 52 active+clean; 810 MiB data, 5.6 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 855 op/s; 2724/38874 objects degraded (7.007%); 1535/38874 objects misplaced (3.949%); 7.1 MiB/s, 333 objects/s recovering 2024-03-20T09:12:43.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:43 smithi115 ceph-mon[72476]: 1.a scrub starts 2024-03-20T09:12:43.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:43 smithi115 ceph-mon[72476]: 1.a scrub ok 2024-03-20T09:12:43.912 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:12:43.917 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:43 smithi026 ceph-mon[68309]: pgmap v1182: 57 pgs: 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 52 active+clean; 810 MiB data, 5.6 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 855 op/s; 2724/38874 objects degraded (7.007%); 1535/38874 objects misplaced (3.949%); 7.1 MiB/s, 333 objects/s recovering 2024-03-20T09:12:43.918 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:43 smithi026 ceph-mon[68309]: 1.a scrub starts 2024-03-20T09:12:43.918 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:43 smithi026 ceph-mon[68309]: 1.a scrub ok 2024-03-20T09:12:43.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:43.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:12:43.938 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:12:43.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:43 smithi122 ceph-mon[73495]: pgmap v1182: 57 pgs: 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 52 active+clean; 810 MiB data, 5.6 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 855 op/s; 2724/38874 objects degraded (7.007%); 1535/38874 objects misplaced (3.949%); 7.1 MiB/s, 333 objects/s recovering 2024-03-20T09:12:43.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:43 smithi122 ceph-mon[73495]: 1.a scrub starts 2024-03-20T09:12:43.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:43 smithi122 ceph-mon[73495]: 1.a scrub ok 2024-03-20T09:12:43.984 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:12:44.019 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:12:44.031 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:12:44.072 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:12:44.121 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:12:44.158 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:44.170 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:12:44.178 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:12:44.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:44.182 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:12:44.205 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:12:44.257 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:12:44.282 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:12:44.299 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:12:44.313 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:44.344 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:12:44.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:44.350 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:12:44.353 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:12:44.404 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:12:44.451 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:12:44.451 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:12:44.459 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:12:44.460 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:12:44.460 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:12:44.460 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:12:44.460 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:12:44.460 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:12:44.460 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:12:44.460 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:12:44.460 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:12:44.460 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:12:44.460 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:12:44.460 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:12:44.487 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:44.521 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:12:44.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:44.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:12:44.626 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:12:44.665 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:44.688 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:44 smithi026 ceph-mon[68309]: 1.1c scrub starts 2024-03-20T09:12:44.689 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:44 smithi026 ceph-mon[68309]: 1.1c scrub ok 2024-03-20T09:12:44.689 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:44 smithi026 ceph-mon[68309]: 1.7 scrub starts 2024-03-20T09:12:44.689 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:44 smithi026 ceph-mon[68309]: 1.7 scrub ok 2024-03-20T09:12:44.689 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:44 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2450208563' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:12:44.699 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:12:44.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:44.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:12:44.810 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:12:44.842 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:44.866 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:12:44.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:44.871 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:12:44.909 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:44 smithi115 ceph-mon[72476]: 1.1c scrub starts 2024-03-20T09:12:44.909 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:44 smithi115 ceph-mon[72476]: 1.1c scrub ok 2024-03-20T09:12:44.909 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:44 smithi115 ceph-mon[72476]: 1.7 scrub starts 2024-03-20T09:12:44.909 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:44 smithi115 ceph-mon[72476]: 1.7 scrub ok 2024-03-20T09:12:44.909 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:44 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2450208563' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:12:44.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:44 smithi122 ceph-mon[73495]: 1.1c scrub starts 2024-03-20T09:12:44.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:44 smithi122 ceph-mon[73495]: 1.1c scrub ok 2024-03-20T09:12:44.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:44 smithi122 ceph-mon[73495]: 1.7 scrub starts 2024-03-20T09:12:44.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:44 smithi122 ceph-mon[73495]: 1.7 scrub ok 2024-03-20T09:12:44.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:44 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2450208563' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:12:44.971 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:12:45.021 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:45.056 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:12:45.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:45.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:12:45.164 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:12:45.199 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:45.223 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:12:45.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:45.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:12:45.326 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:12:45.388 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:45.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:45 smithi026 ceph-mon[68309]: 3.9 scrub starts 2024-03-20T09:12:45.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:45 smithi026 ceph-mon[68309]: pgmap v1183: 57 pgs: 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 52 active+clean; 738 MiB data, 5.5 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 838 op/s; 2885/35391 objects degraded (8.152%); 1372/35391 objects misplaced (3.877%); 6.9 MiB/s, 258 objects/s recovering 2024-03-20T09:12:45.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:45 smithi026 ceph-mon[68309]: 3.9 scrub ok 2024-03-20T09:12:46.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:45 smithi115 ceph-mon[72476]: 3.9 scrub starts 2024-03-20T09:12:46.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:45 smithi115 ceph-mon[72476]: pgmap v1183: 57 pgs: 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 52 active+clean; 738 MiB data, 5.5 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 838 op/s; 2885/35391 objects degraded (8.152%); 1372/35391 objects misplaced (3.877%); 6.9 MiB/s, 258 objects/s recovering 2024-03-20T09:12:46.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:45 smithi115 ceph-mon[72476]: 3.9 scrub ok 2024-03-20T09:12:46.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:45 smithi122 ceph-mon[73495]: 3.9 scrub starts 2024-03-20T09:12:46.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:45 smithi122 ceph-mon[73495]: pgmap v1183: 57 pgs: 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 52 active+clean; 738 MiB data, 5.5 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 838 op/s; 2885/35391 objects degraded (8.152%); 1372/35391 objects misplaced (3.877%); 6.9 MiB/s, 258 objects/s recovering 2024-03-20T09:12:46.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:45 smithi122 ceph-mon[73495]: 3.9 scrub ok 2024-03-20T09:12:46.685 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:12:46.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:46.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:12:46.794 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:12:46.830 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:46.852 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:12:46.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:46.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:12:46.958 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:12:47.088 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:47.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:47 smithi026 ceph-mon[68309]: pgmap v1184: 57 pgs: 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 52 active+clean; 599 MiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 864 op/s; 2885/28716 objects degraded (10.047%); 1078/28716 objects misplaced (3.754%); 5.7 MiB/s, 211 objects/s recovering 2024-03-20T09:12:47.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:47 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 2885/28716 objects degraded (10.047%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:12:48.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:47 smithi115 ceph-mon[72476]: pgmap v1184: 57 pgs: 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 52 active+clean; 599 MiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 864 op/s; 2885/28716 objects degraded (10.047%); 1078/28716 objects misplaced (3.754%); 5.7 MiB/s, 211 objects/s recovering 2024-03-20T09:12:48.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:47 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 2885/28716 objects degraded (10.047%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:12:48.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:47 smithi122 ceph-mon[73495]: pgmap v1184: 57 pgs: 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 52 active+clean; 599 MiB data, 5.0 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 864 op/s; 2885/28716 objects degraded (10.047%); 1078/28716 objects misplaced (3.754%); 5.7 MiB/s, 211 objects/s recovering 2024-03-20T09:12:48.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:47 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 2885/28716 objects degraded (10.047%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:12:49.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:48 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:12:49.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:48 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:12:49.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:48 smithi115 ceph-mon[72476]: osdmap e500: 12 total, 12 up, 10 in 2024-03-20T09:12:49.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:48 smithi115 ceph-mon[72476]: 3.14 scrub starts 2024-03-20T09:12:49.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:48 smithi115 ceph-mon[72476]: 3.14 scrub ok 2024-03-20T09:12:49.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:48 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:12:49.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:48 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:12:49.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:48 smithi122 ceph-mon[73495]: osdmap e500: 12 total, 12 up, 10 in 2024-03-20T09:12:49.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:48 smithi122 ceph-mon[73495]: 3.14 scrub starts 2024-03-20T09:12:49.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:48 smithi122 ceph-mon[73495]: 3.14 scrub ok 2024-03-20T09:12:49.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:48 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:12:49.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:48 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:12:49.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:48 smithi026 ceph-mon[68309]: osdmap e500: 12 total, 12 up, 10 in 2024-03-20T09:12:49.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:48 smithi026 ceph-mon[68309]: 3.14 scrub starts 2024-03-20T09:12:49.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:48 smithi026 ceph-mon[68309]: 3.14 scrub ok 2024-03-20T09:12:49.923 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:12:49.978 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:49 smithi026 ceph-mon[68309]: pgmap v1186: 57 pgs: 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 52 active+clean; 437 MiB data, 4.6 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 944 op/s; 3046/20937 objects degraded (14.548%); 703/20937 objects misplaced (3.358%); 3.0 MiB/s, 232 objects/s recovering 2024-03-20T09:12:50.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:49 smithi115 ceph-mon[72476]: pgmap v1186: 57 pgs: 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 52 active+clean; 437 MiB data, 4.6 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 944 op/s; 3046/20937 objects degraded (14.548%); 703/20937 objects misplaced (3.358%); 3.0 MiB/s, 232 objects/s recovering 2024-03-20T09:12:50.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:49 smithi122 ceph-mon[73495]: pgmap v1186: 57 pgs: 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 52 active+clean; 437 MiB data, 4.6 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 944 op/s; 3046/20937 objects degraded (14.548%); 703/20937 objects misplaced (3.358%); 3.0 MiB/s, 232 objects/s recovering 2024-03-20T09:12:50.796 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:12:50.796 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:12:50.796 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-03-20T09:12:50.796 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-03-20T09:12:50.796 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:12:50.852 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:12:50.881 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:12:50.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:50.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:12:50.998 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:12:51.066 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:50 smithi026 ceph-mon[68309]: 3.15 scrub starts 2024-03-20T09:12:51.066 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:50 smithi026 ceph-mon[68309]: 3.15 scrub ok 2024-03-20T09:12:51.066 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:50 smithi026 ceph-mon[68309]: pgmap v1187: 57 pgs: 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 52 active+clean; 388 MiB data, 4.4 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 960 op/s; 3046/18603 objects degraded (16.374%); 381/18603 objects misplaced (2.048%); 3.0 MiB/s, 232 objects/s recovering 2024-03-20T09:12:51.068 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:51.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:50 smithi115 ceph-mon[72476]: 3.15 scrub starts 2024-03-20T09:12:51.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:50 smithi115 ceph-mon[72476]: 3.15 scrub ok 2024-03-20T09:12:51.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:50 smithi115 ceph-mon[72476]: pgmap v1187: 57 pgs: 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 52 active+clean; 388 MiB data, 4.4 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 960 op/s; 3046/18603 objects degraded (16.374%); 381/18603 objects misplaced (2.048%); 3.0 MiB/s, 232 objects/s recovering 2024-03-20T09:12:51.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:50 smithi122 ceph-mon[73495]: 3.15 scrub starts 2024-03-20T09:12:51.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:50 smithi122 ceph-mon[73495]: 3.15 scrub ok 2024-03-20T09:12:51.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:50 smithi122 ceph-mon[73495]: pgmap v1187: 57 pgs: 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 52 active+clean; 388 MiB data, 4.4 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 960 op/s; 3046/18603 objects degraded (16.374%); 381/18603 objects misplaced (2.048%); 3.0 MiB/s, 232 objects/s recovering 2024-03-20T09:12:51.780 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:51 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1159548502' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:12:51.780 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:51 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:12:51.780 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:51 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:12:51.780 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:51 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:12:51.930 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:12:52.159 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:51 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1159548502' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:12:52.160 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:51 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:12:52.160 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:51 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:12:52.160 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:51 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:12:52.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:51 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1159548502' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:12:52.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:51 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:12:52.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:51 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:12:52.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:51 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:12:53.138 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Removed 99779 objects 2024-03-20T09:12:53.139 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Clean up completed and total clean up time :500.509 2024-03-20T09:12:53.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:52 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:12:53.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:52 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:12:53.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:52 smithi115 ceph-mon[72476]: osdmap e501: 12 total, 12 up, 10 in 2024-03-20T09:12:53.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:52 smithi115 ceph-mon[72476]: pgmap v1189: 57 pgs: 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 52 active+clean; 188 MiB data, 3.8 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 1.10k op/s; 3046/9021 objects degraded (33.766%); 381/9021 objects misplaced (4.223%); 1.7 MiB/s, 241 objects/s recovering 2024-03-20T09:12:53.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:52 smithi115 ceph-mon[72476]: 1.14 scrub starts 2024-03-20T09:12:53.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:52 smithi115 ceph-mon[72476]: 1.14 scrub ok 2024-03-20T09:12:53.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:52 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 3046/18603 objects degraded (16.374%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:12:53.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:52 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:12:53.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:52 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:12:53.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:52 smithi122 ceph-mon[73495]: osdmap e501: 12 total, 12 up, 10 in 2024-03-20T09:12:53.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:52 smithi122 ceph-mon[73495]: pgmap v1189: 57 pgs: 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 52 active+clean; 188 MiB data, 3.8 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 1.10k op/s; 3046/9021 objects degraded (33.766%); 381/9021 objects misplaced (4.223%); 1.7 MiB/s, 241 objects/s recovering 2024-03-20T09:12:53.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:52 smithi122 ceph-mon[73495]: 1.14 scrub starts 2024-03-20T09:12:53.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:52 smithi122 ceph-mon[73495]: 1.14 scrub ok 2024-03-20T09:12:53.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:52 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 3046/18603 objects degraded (16.374%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:12:53.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:52 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:12:53.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:52 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:12:53.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:52 smithi026 ceph-mon[68309]: osdmap e501: 12 total, 12 up, 10 in 2024-03-20T09:12:53.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:52 smithi026 ceph-mon[68309]: pgmap v1189: 57 pgs: 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 52 active+clean; 188 MiB data, 3.8 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 1.10k op/s; 3046/9021 objects degraded (33.766%); 381/9021 objects misplaced (4.223%); 1.7 MiB/s, 241 objects/s recovering 2024-03-20T09:12:53.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:52 smithi026 ceph-mon[68309]: 1.14 scrub starts 2024-03-20T09:12:53.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:52 smithi026 ceph-mon[68309]: 1.14 scrub ok 2024-03-20T09:12:53.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:52 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 3046/18603 objects degraded (16.374%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:12:53.444 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-03-20T09:12:53.445 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-03-20T09:12:55.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:54 smithi115 ceph-mon[72476]: pgmap v1190: 57 pgs: 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 52 active+clean; 122 MiB data, 3.6 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 951 op/s; 3109/5856 objects degraded (53.091%); 206/5856 objects misplaced (3.518%); 2.5 MiB/s, 297 objects/s recovering 2024-03-20T09:12:55.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:54 smithi122 ceph-mon[73495]: pgmap v1190: 57 pgs: 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 52 active+clean; 122 MiB data, 3.6 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 951 op/s; 3109/5856 objects degraded (53.091%); 206/5856 objects misplaced (3.518%); 2.5 MiB/s, 297 objects/s recovering 2024-03-20T09:12:55.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:54 smithi026 ceph-mon[68309]: pgmap v1190: 57 pgs: 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 52 active+clean; 122 MiB data, 3.6 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 951 op/s; 3109/5856 objects degraded (53.091%); 206/5856 objects misplaced (3.518%); 2.5 MiB/s, 297 objects/s recovering 2024-03-20T09:12:56.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:55 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:12:56.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:55 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:12:56.161 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:12:56.163 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:55 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:12:56.163 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:55 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:12:56.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:56.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:12:56.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:55 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:12:56.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:55 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:12:56.265 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:12:56.319 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:56.340 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:12:56.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:56.344 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:12:56.445 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:12:56.475 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:56.495 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:12:56.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:56.499 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:12:56.600 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:12:56.627 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:56.649 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:12:56.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:56.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:12:56.754 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:12:56.785 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:56.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:56 smithi115 ceph-mon[72476]: pgmap v1191: 57 pgs: 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 52 active+clean; 42 MiB data, 3.3 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 783 op/s; 3109/1986 objects degraded (156.546%); 880 KiB/s, 152 objects/s recovering 2024-03-20T09:12:56.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:56 smithi115 ceph-mon[72476]: osdmap e502: 12 total, 12 up, 10 in 2024-03-20T09:12:57.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:56 smithi122 ceph-mon[73495]: pgmap v1191: 57 pgs: 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 52 active+clean; 42 MiB data, 3.3 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 783 op/s; 3109/1986 objects degraded (156.546%); 880 KiB/s, 152 objects/s recovering 2024-03-20T09:12:57.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:56 smithi122 ceph-mon[73495]: osdmap e502: 12 total, 12 up, 10 in 2024-03-20T09:12:57.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:56 smithi026 ceph-mon[68309]: pgmap v1191: 57 pgs: 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 52 active+clean; 42 MiB data, 3.3 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 783 op/s; 3109/1986 objects degraded (156.546%); 880 KiB/s, 152 objects/s recovering 2024-03-20T09:12:57.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:56 smithi026 ceph-mon[68309]: osdmap e502: 12 total, 12 up, 10 in 2024-03-20T09:12:57.386 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:12:57.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:57.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:12:57.494 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:12:57.527 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:57.549 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:12:57.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:57.552 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:12:57.653 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:12:57.682 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:57.703 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:12:57.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:57.706 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:12:57.807 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:12:57.834 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:57.855 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:12:57.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:57.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:12:57.959 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:12:57.986 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:58.007 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:12:58.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:58.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:12:58.111 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:12:58.136 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:57 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 3109/1986 objects degraded (156.546%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:12:58.136 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:57 smithi115 ceph-mon[72476]: osdmap e503: 12 total, 12 up, 10 in 2024-03-20T09:12:58.138 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:58.158 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:12:58.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:58.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:12:58.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:57 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 3109/1986 objects degraded (156.546%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:12:58.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:57 smithi122 ceph-mon[73495]: osdmap e503: 12 total, 12 up, 10 in 2024-03-20T09:12:58.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:57 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 3109/1986 objects degraded (156.546%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:12:58.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:57 smithi026 ceph-mon[68309]: osdmap e503: 12 total, 12 up, 10 in 2024-03-20T09:12:58.263 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:12:58.290 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:58.309 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:12:58.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:58.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:12:58.413 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:12:58.442 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:59.035 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-20T09:12:59.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:59 smithi115 ceph-mon[72476]: pgmap v1194: 57 pgs: 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 52 active+clean; 456 KiB data, 3.1 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 505 op/s; 3056/3 objects degraded (101866.667%); 1.2 MiB/s, 574 objects/s recovering 2024-03-20T09:12:59.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:59 smithi115 ceph-mon[72476]: osdmap e504: 12 total, 12 up, 10 in 2024-03-20T09:12:59.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:59 smithi115 ceph-mon[72476]: 3.0 scrub starts 2024-03-20T09:12:59.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:12:59 smithi115 ceph-mon[72476]: 3.0 scrub ok 2024-03-20T09:12:59.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:59 smithi122 ceph-mon[73495]: pgmap v1194: 57 pgs: 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 52 active+clean; 456 KiB data, 3.1 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 505 op/s; 3056/3 objects degraded (101866.667%); 1.2 MiB/s, 574 objects/s recovering 2024-03-20T09:12:59.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:59 smithi122 ceph-mon[73495]: osdmap e504: 12 total, 12 up, 10 in 2024-03-20T09:12:59.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:59 smithi122 ceph-mon[73495]: 3.0 scrub starts 2024-03-20T09:12:59.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:12:59 smithi122 ceph-mon[73495]: 3.0 scrub ok 2024-03-20T09:12:59.464 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:59 smithi026 ceph-mon[68309]: pgmap v1194: 57 pgs: 1 active+remapped+backfill_wait, 4 active+recovering+undersized+degraded+remapped, 52 active+clean; 456 KiB data, 3.1 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 505 op/s; 3056/3 objects degraded (101866.667%); 1.2 MiB/s, 574 objects/s recovering 2024-03-20T09:12:59.465 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:59 smithi026 ceph-mon[68309]: osdmap e504: 12 total, 12 up, 10 in 2024-03-20T09:12:59.465 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:59 smithi026 ceph-mon[68309]: 3.0 scrub starts 2024-03-20T09:12:59.465 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:12:59 smithi026 ceph-mon[68309]: 3.0 scrub ok 2024-03-20T09:12:59.493 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:12:59.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:59.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:12:59.601 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:12:59.633 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:59.654 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:12:59.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:59.657 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:12:59.758 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:12:59.788 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:12:59.946 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:12:59.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:12:59.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:13:00.039 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:13:00.053 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:13:00.085 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:00.105 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:13:00.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:00.108 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:13:00.209 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:13:00.236 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:00.255 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:13:00.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:00.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:13:00.325 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:00 smithi026 ceph-mon[68309]: osdmap e505: 12 total, 12 up, 10 in 2024-03-20T09:13:00.325 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:00 smithi026 ceph-mon[68309]: 3.f scrub starts 2024-03-20T09:13:00.325 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:00 smithi026 ceph-mon[68309]: 3.f scrub ok 2024-03-20T09:13:00.359 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:13:00.390 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:00.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:00 smithi115 ceph-mon[72476]: osdmap e505: 12 total, 12 up, 10 in 2024-03-20T09:13:00.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:00 smithi115 ceph-mon[72476]: 3.f scrub starts 2024-03-20T09:13:00.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:00 smithi115 ceph-mon[72476]: 3.f scrub ok 2024-03-20T09:13:00.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:00 smithi122 ceph-mon[73495]: osdmap e505: 12 total, 12 up, 10 in 2024-03-20T09:13:00.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:00 smithi122 ceph-mon[73495]: 3.f scrub starts 2024-03-20T09:13:00.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:00 smithi122 ceph-mon[73495]: 3.f scrub ok 2024-03-20T09:13:00.790 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:13:01.088 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.10 to 1 (10000) 2024-03-20T09:13:01.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:01 smithi026 ceph-mon[68309]: pgmap v1197: 57 pgs: 4 active+recovering+undersized+degraded+remapped, 53 active+clean; 456 KiB data, 3.1 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 172 op/s; 3056/3 objects degraded (101866.667%); 0 B/s, 752 objects/s recovering 2024-03-20T09:13:01.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:01 smithi026 ceph-mon[68309]: 3.13 scrub starts 2024-03-20T09:13:01.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:01 smithi026 ceph-mon[68309]: 3.13 scrub ok 2024-03-20T09:13:01.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:01 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:13:01.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:01 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:13:01.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:01 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3895910169' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-03-20T09:13:01.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:01 smithi115 ceph-mon[72476]: pgmap v1197: 57 pgs: 4 active+recovering+undersized+degraded+remapped, 53 active+clean; 456 KiB data, 3.1 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 172 op/s; 3056/3 objects degraded (101866.667%); 0 B/s, 752 objects/s recovering 2024-03-20T09:13:01.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:01 smithi115 ceph-mon[72476]: 3.13 scrub starts 2024-03-20T09:13:01.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:01 smithi115 ceph-mon[72476]: 3.13 scrub ok 2024-03-20T09:13:01.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:01 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:13:01.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:01 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:13:01.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:01 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3895910169' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-03-20T09:13:01.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:01 smithi122 ceph-mon[73495]: pgmap v1197: 57 pgs: 4 active+recovering+undersized+degraded+remapped, 53 active+clean; 456 KiB data, 3.1 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 172 op/s; 3056/3 objects degraded (101866.667%); 0 B/s, 752 objects/s recovering 2024-03-20T09:13:01.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:01 smithi122 ceph-mon[73495]: 3.13 scrub starts 2024-03-20T09:13:01.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:01 smithi122 ceph-mon[73495]: 3.13 scrub ok 2024-03-20T09:13:01.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:01 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:13:01.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:01 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:13:01.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:01 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3895910169' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-03-20T09:13:01.450 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:13:01.742 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:13:01.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:01.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:13:01.848 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:13:01.880 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:01.902 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:13:01.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:01.905 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:13:01.975 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:13:02.005 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:13:02.031 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:02.051 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:13:02.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:02.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:13:02.156 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:13:02.182 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:02.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:02 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3895910169' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-03-20T09:13:02.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:02 smithi122 ceph-mon[73495]: osdmap e506: 12 total, 12 up, 10 in 2024-03-20T09:13:02.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:02 smithi122 ceph-mon[73495]: 3.5 scrub starts 2024-03-20T09:13:02.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:02 smithi122 ceph-mon[73495]: 3.5 scrub ok 2024-03-20T09:13:02.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:02 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 3056/3 objects degraded (101866.667%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:13:02.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:02 smithi122 ceph-mon[73495]: osdmap e507: 12 total, 12 up, 10 in 2024-03-20T09:13:02.200 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:13:02.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:02.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:13:02.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:02 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3895910169' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-03-20T09:13:02.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:02 smithi026 ceph-mon[68309]: osdmap e506: 12 total, 12 up, 10 in 2024-03-20T09:13:02.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:02 smithi026 ceph-mon[68309]: 3.5 scrub starts 2024-03-20T09:13:02.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:02 smithi026 ceph-mon[68309]: 3.5 scrub ok 2024-03-20T09:13:02.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:02 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 3056/3 objects degraded (101866.667%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:13:02.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:02 smithi026 ceph-mon[68309]: osdmap e507: 12 total, 12 up, 10 in 2024-03-20T09:13:02.304 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:13:02.338 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:02.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:02 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3895910169' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-03-20T09:13:02.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:02 smithi115 ceph-mon[72476]: osdmap e506: 12 total, 12 up, 10 in 2024-03-20T09:13:02.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:02 smithi115 ceph-mon[72476]: 3.5 scrub starts 2024-03-20T09:13:02.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:02 smithi115 ceph-mon[72476]: 3.5 scrub ok 2024-03-20T09:13:02.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:02 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 3056/3 objects degraded (101866.667%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:13:02.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:02 smithi115 ceph-mon[72476]: osdmap e507: 12 total, 12 up, 10 in 2024-03-20T09:13:02.889 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:13:02.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:02.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:13:02.993 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:13:03.023 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:03.045 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:13:03.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:03.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:13:03.149 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:13:03.182 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:03.204 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:13:03.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:03.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:13:03.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:03 smithi026 ceph-mon[68309]: pgmap v1199: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 54 active+clean; 456 KiB data, 3.1 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 40 op/s; 2578/3 objects degraded (85933.333%); 0 B/s, 718 objects/s recovering 2024-03-20T09:13:03.308 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:13:03.338 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:03.358 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:13:03.360 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:03 smithi115 ceph-mon[72476]: pgmap v1199: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 54 active+clean; 456 KiB data, 3.1 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 40 op/s; 2578/3 objects degraded (85933.333%); 0 B/s, 718 objects/s recovering 2024-03-20T09:13:03.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:03.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:13:03.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:03 smithi122 ceph-mon[73495]: pgmap v1199: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 54 active+clean; 456 KiB data, 3.1 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 40 op/s; 2578/3 objects degraded (85933.333%); 0 B/s, 718 objects/s recovering 2024-03-20T09:13:03.461 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:13:03.490 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:03.510 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:13:03.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:03.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:13:03.613 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:13:03.647 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:03.667 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:13:03.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:03.671 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:13:03.772 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:13:03.801 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:04.147 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 11 1 2024-03-20T09:13:04.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:04 smithi115 ceph-mon[72476]: osdmap e508: 12 total, 12 up, 10 in 2024-03-20T09:13:04.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:04 smithi115 ceph-mon[72476]: 1.0 scrub starts 2024-03-20T09:13:04.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:04 smithi115 ceph-mon[72476]: 1.0 scrub ok 2024-03-20T09:13:04.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:04 smithi122 ceph-mon[73495]: osdmap e508: 12 total, 12 up, 10 in 2024-03-20T09:13:04.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:04 smithi122 ceph-mon[73495]: 1.0 scrub starts 2024-03-20T09:13:04.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:04 smithi122 ceph-mon[73495]: 1.0 scrub ok 2024-03-20T09:13:04.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:04 smithi026 ceph-mon[68309]: osdmap e508: 12 total, 12 up, 10 in 2024-03-20T09:13:04.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:04 smithi026 ceph-mon[68309]: 1.0 scrub starts 2024-03-20T09:13:04.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:04 smithi026 ceph-mon[68309]: 1.0 scrub ok 2024-03-20T09:13:05.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:05 smithi115 ceph-mon[72476]: pgmap v1202: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 55 active+clean; 456 KiB data, 3.1 GiB used, 891 GiB / 894 GiB avail; 1660/3 objects degraded (55333.333%); 0 B/s, 742 objects/s recovering 2024-03-20T09:13:05.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:05 smithi115 ceph-mon[72476]: osdmap e509: 12 total, 12 up, 10 in 2024-03-20T09:13:05.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:05 smithi115 ceph-mon[72476]: 1.12 scrub starts 2024-03-20T09:13:05.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:05 smithi115 ceph-mon[72476]: 1.12 scrub ok 2024-03-20T09:13:05.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:05 smithi115 ceph-mon[72476]: 3.18 deep-scrub starts 2024-03-20T09:13:05.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:05 smithi115 ceph-mon[72476]: 3.18 deep-scrub ok 2024-03-20T09:13:05.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:05 smithi122 ceph-mon[73495]: pgmap v1202: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 55 active+clean; 456 KiB data, 3.1 GiB used, 891 GiB / 894 GiB avail; 1660/3 objects degraded (55333.333%); 0 B/s, 742 objects/s recovering 2024-03-20T09:13:05.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:05 smithi122 ceph-mon[73495]: osdmap e509: 12 total, 12 up, 10 in 2024-03-20T09:13:05.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:05 smithi122 ceph-mon[73495]: 1.12 scrub starts 2024-03-20T09:13:05.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:05 smithi122 ceph-mon[73495]: 1.12 scrub ok 2024-03-20T09:13:05.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:05 smithi122 ceph-mon[73495]: 3.18 deep-scrub starts 2024-03-20T09:13:05.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:05 smithi122 ceph-mon[73495]: 3.18 deep-scrub ok 2024-03-20T09:13:05.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:05 smithi026 ceph-mon[68309]: pgmap v1202: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 55 active+clean; 456 KiB data, 3.1 GiB used, 891 GiB / 894 GiB avail; 1660/3 objects degraded (55333.333%); 0 B/s, 742 objects/s recovering 2024-03-20T09:13:05.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:05 smithi026 ceph-mon[68309]: osdmap e509: 12 total, 12 up, 10 in 2024-03-20T09:13:05.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:05 smithi026 ceph-mon[68309]: 1.12 scrub starts 2024-03-20T09:13:05.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:05 smithi026 ceph-mon[68309]: 1.12 scrub ok 2024-03-20T09:13:05.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:05 smithi026 ceph-mon[68309]: 3.18 deep-scrub starts 2024-03-20T09:13:05.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:05 smithi026 ceph-mon[68309]: 3.18 deep-scrub ok 2024-03-20T09:13:06.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:06 smithi115 ceph-mon[72476]: 1.15 scrub starts 2024-03-20T09:13:06.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:06 smithi115 ceph-mon[72476]: 1.15 scrub ok 2024-03-20T09:13:06.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:06 smithi115 ceph-mon[72476]: 1.9 scrub starts 2024-03-20T09:13:06.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:06 smithi115 ceph-mon[72476]: 1.9 scrub ok 2024-03-20T09:13:06.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:06 smithi115 ceph-mon[72476]: 1.1d scrub starts 2024-03-20T09:13:06.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:06 smithi115 ceph-mon[72476]: 1.1d scrub ok 2024-03-20T09:13:06.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:06 smithi115 ceph-mon[72476]: 1.10 scrub starts 2024-03-20T09:13:06.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:06 smithi115 ceph-mon[72476]: 1.10 scrub ok 2024-03-20T09:13:06.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:06 smithi122 ceph-mon[73495]: 1.15 scrub starts 2024-03-20T09:13:06.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:06 smithi122 ceph-mon[73495]: 1.15 scrub ok 2024-03-20T09:13:06.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:06 smithi122 ceph-mon[73495]: 1.9 scrub starts 2024-03-20T09:13:06.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:06 smithi122 ceph-mon[73495]: 1.9 scrub ok 2024-03-20T09:13:06.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:06 smithi122 ceph-mon[73495]: 1.1d scrub starts 2024-03-20T09:13:06.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:06 smithi122 ceph-mon[73495]: 1.1d scrub ok 2024-03-20T09:13:06.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:06 smithi122 ceph-mon[73495]: 1.10 scrub starts 2024-03-20T09:13:06.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:06 smithi122 ceph-mon[73495]: 1.10 scrub ok 2024-03-20T09:13:06.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:06 smithi026 ceph-mon[68309]: 1.15 scrub starts 2024-03-20T09:13:06.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:06 smithi026 ceph-mon[68309]: 1.15 scrub ok 2024-03-20T09:13:06.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:06 smithi026 ceph-mon[68309]: 1.9 scrub starts 2024-03-20T09:13:06.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:06 smithi026 ceph-mon[68309]: 1.9 scrub ok 2024-03-20T09:13:06.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:06 smithi026 ceph-mon[68309]: 1.1d scrub starts 2024-03-20T09:13:06.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:06 smithi026 ceph-mon[68309]: 1.1d scrub ok 2024-03-20T09:13:06.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:06 smithi026 ceph-mon[68309]: 1.10 scrub starts 2024-03-20T09:13:06.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:06 smithi026 ceph-mon[68309]: 1.10 scrub ok 2024-03-20T09:13:07.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:07 smithi115 ceph-mon[72476]: 1.e scrub starts 2024-03-20T09:13:07.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:07 smithi115 ceph-mon[72476]: 1.e scrub ok 2024-03-20T09:13:07.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:07 smithi115 ceph-mon[72476]: pgmap v1204: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 55 active+clean; 456 KiB data, 3.0 GiB used, 891 GiB / 894 GiB avail; 1660/3 objects degraded (55333.333%); 0 B/s, 743 objects/s recovering 2024-03-20T09:13:07.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:07 smithi115 ceph-mon[72476]: 3.4 scrub starts 2024-03-20T09:13:07.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:07 smithi115 ceph-mon[72476]: 3.4 scrub ok 2024-03-20T09:13:07.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:07 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 1660/3 objects degraded (55333.333%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:13:07.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:07 smithi115 ceph-mon[72476]: osdmap e510: 12 total, 12 up, 10 in 2024-03-20T09:13:07.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:07 smithi122 ceph-mon[73495]: 1.e scrub starts 2024-03-20T09:13:07.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:07 smithi122 ceph-mon[73495]: 1.e scrub ok 2024-03-20T09:13:07.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:07 smithi122 ceph-mon[73495]: pgmap v1204: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 55 active+clean; 456 KiB data, 3.0 GiB used, 891 GiB / 894 GiB avail; 1660/3 objects degraded (55333.333%); 0 B/s, 743 objects/s recovering 2024-03-20T09:13:07.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:07 smithi122 ceph-mon[73495]: 3.4 scrub starts 2024-03-20T09:13:07.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:07 smithi122 ceph-mon[73495]: 3.4 scrub ok 2024-03-20T09:13:07.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:07 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 1660/3 objects degraded (55333.333%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:13:07.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:07 smithi122 ceph-mon[73495]: osdmap e510: 12 total, 12 up, 10 in 2024-03-20T09:13:07.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:07 smithi026 ceph-mon[68309]: 1.e scrub starts 2024-03-20T09:13:07.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:07 smithi026 ceph-mon[68309]: 1.e scrub ok 2024-03-20T09:13:07.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:07 smithi026 ceph-mon[68309]: pgmap v1204: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 55 active+clean; 456 KiB data, 3.0 GiB used, 891 GiB / 894 GiB avail; 1660/3 objects degraded (55333.333%); 0 B/s, 743 objects/s recovering 2024-03-20T09:13:07.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:07 smithi026 ceph-mon[68309]: 3.4 scrub starts 2024-03-20T09:13:07.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:07 smithi026 ceph-mon[68309]: 3.4 scrub ok 2024-03-20T09:13:07.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:07 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 1660/3 objects degraded (55333.333%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:13:07.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:07 smithi026 ceph-mon[68309]: osdmap e510: 12 total, 12 up, 10 in 2024-03-20T09:13:08.079 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:13:08.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:08.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:13:08.184 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:13:08.212 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:08.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:08 smithi026 ceph-mon[68309]: 3.17 scrub starts 2024-03-20T09:13:08.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:08 smithi026 ceph-mon[68309]: 3.17 scrub ok 2024-03-20T09:13:08.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:08 smithi026 ceph-mon[68309]: 3.d scrub starts 2024-03-20T09:13:08.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:08 smithi026 ceph-mon[68309]: 3.d scrub ok 2024-03-20T09:13:08.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:08 smithi026 ceph-mon[68309]: osdmap e511: 12 total, 12 up, 10 in 2024-03-20T09:13:08.339 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:13:08.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:08.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:13:08.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:08 smithi115 ceph-mon[72476]: 3.17 scrub starts 2024-03-20T09:13:08.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:08 smithi115 ceph-mon[72476]: 3.17 scrub ok 2024-03-20T09:13:08.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:08 smithi115 ceph-mon[72476]: 3.d scrub starts 2024-03-20T09:13:08.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:08 smithi115 ceph-mon[72476]: 3.d scrub ok 2024-03-20T09:13:08.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:08 smithi115 ceph-mon[72476]: osdmap e511: 12 total, 12 up, 10 in 2024-03-20T09:13:08.443 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:13:08.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:08 smithi122 ceph-mon[73495]: 3.17 scrub starts 2024-03-20T09:13:08.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:08 smithi122 ceph-mon[73495]: 3.17 scrub ok 2024-03-20T09:13:08.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:08 smithi122 ceph-mon[73495]: 3.d scrub starts 2024-03-20T09:13:08.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:08 smithi122 ceph-mon[73495]: 3.d scrub ok 2024-03-20T09:13:08.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:08 smithi122 ceph-mon[73495]: osdmap e511: 12 total, 12 up, 10 in 2024-03-20T09:13:08.469 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:08.487 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:13:08.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:08.490 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:13:08.591 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:13:08.618 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:08.637 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:13:08.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:08.639 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:13:08.740 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:13:08.771 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:08.809 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:13:08.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:08.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:13:08.841 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:13:08.841 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-03-20T09:13:08.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-03-20T09:13:08.841 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:13:08.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.842 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:58474 3.17 3:efc4c91d:::benchmark_data_smithi122_119671_object58473:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e234)", 2024-03-20T09:13:08.842 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:03:32.876662+0000", 2024-03-20T09:13:08.842 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 575.96115281599998, 2024-03-20T09:13:08.842 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2026188529999999, 2024-03-20T09:13:08.842 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:13:08.842 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:13:08.842 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:13:08.842 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:13:08.842 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:13:08.842 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 58474 2024-03-20T09:13:08.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.842 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:13:08.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:13:08.843 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.876662+0000", 2024-03-20T09:13:08.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:13:08.843 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.876662+0000", 2024-03-20T09:13:08.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:13:08.843 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.876664+0000", 2024-03-20T09:13:08.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5770000000000001e-06 2024-03-20T09:13:08.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.844 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:13:08.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.876953+0000", 2024-03-20T09:13:08.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00028972500000000001 2024-03-20T09:13:08.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.844 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:13:08.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.876967+0000", 2024-03-20T09:13:08.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.329e-05 2024-03-20T09:13:08.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.844 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:13:08.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.876978+0000", 2024-03-20T09:13:08.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1483e-05 2024-03-20T09:13:08.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.845 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.845 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877044+0000", 2024-03-20T09:13:08.845 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6137999999999996e-05 2024-03-20T09:13:08.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.845 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:13:08.845 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:32.877055+0000", 2024-03-20T09:13:08.845 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0224000000000001e-05 2024-03-20T09:13:08.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.845 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.845 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.066151+0000", 2024-03-20T09:13:08.845 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.189096543 2024-03-20T09:13:08.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for degraded object", 2024-03-20T09:13:08.846 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.066932+0000", 2024-03-20T09:13:08.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00078113200000000003 2024-03-20T09:13:08.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.846 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.079192+0000", 2024-03-20T09:13:08.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012259617 2024-03-20T09:13:08.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:13:08.847 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:03:34.079281+0000", 2024-03-20T09:13:08.847 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9123999999999996e-05 2024-03-20T09:13:08.847 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.847 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:13:08.847 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.847 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:143564 3.7 3:e7091616:::benchmark_data_smithi122_119671_object43782:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e449)", 2024-03-20T09:13:08.847 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:10:41.154979+0000", 2024-03-20T09:13:08.847 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 147.682835684, 2024-03-20T09:13:08.847 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.009065637, 2024-03-20T09:13:08.847 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:13:08.847 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:13:08.848 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:13:08.848 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:13:08.848 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:13:08.848 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 143564 2024-03-20T09:13:08.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.848 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:13:08.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.848 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:13:08.848 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.154979+0000", 2024-03-20T09:13:08.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.848 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:13:08.848 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.154979+0000", 2024-03-20T09:13:08.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.849 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:13:08.849 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.154981+0000", 2024-03-20T09:13:08.849 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.911e-06 2024-03-20T09:13:08.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.849 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:13:08.849 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.154990+0000", 2024-03-20T09:13:08.849 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.217e-06 2024-03-20T09:13:08.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.849 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:13:08.849 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.155013+0000", 2024-03-20T09:13:08.849 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2602000000000001e-05 2024-03-20T09:13:08.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:13:08.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.155025+0000", 2024-03-20T09:13:08.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.223e-05 2024-03-20T09:13:08.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.155108+0000", 2024-03-20T09:13:08.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3083000000000004e-05 2024-03-20T09:13:08.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:13:08.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.155123+0000", 2024-03-20T09:13:08.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4681000000000001e-05 2024-03-20T09:13:08.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.156747+0000", 2024-03-20T09:13:08.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0016243069999999 2024-03-20T09:13:08.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:13:08.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.156916+0000", 2024-03-20T09:13:08.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00016824299999999999 2024-03-20T09:13:08.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:13:08.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.157043+0000", 2024-03-20T09:13:08.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000127329 2024-03-20T09:13:08.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.161049+0000", 2024-03-20T09:13:08.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040065709999999996 2024-03-20T09:13:08.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:13:08.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.161057+0000", 2024-03-20T09:13:08.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8679999999999999e-06 2024-03-20T09:13:08.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.163980+0000", 2024-03-20T09:13:08.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0029223510000000001 2024-03-20T09:13:08.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.853 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:13:08.853 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.164015+0000", 2024-03-20T09:13:08.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5037999999999999e-05 2024-03-20T09:13:08.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.853 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:13:08.853 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.164045+0000", 2024-03-20T09:13:08.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0205999999999998e-05 2024-03-20T09:13:08.853 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.853 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:13:08.853 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.854 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:143572 3.7 3:e3d5668d:::benchmark_data_smithi122_119671_object43790:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e449)", 2024-03-20T09:13:08.854 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:10:41.197173+0000", 2024-03-20T09:13:08.854 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 147.640642114, 2024-03-20T09:13:08.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.97170473999999996, 2024-03-20T09:13:08.854 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:13:08.854 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:13:08.854 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:13:08.854 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:13:08.854 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:13:08.854 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 143572 2024-03-20T09:13:08.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.854 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:13:08.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.854 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:13:08.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.197173+0000", 2024-03-20T09:13:08.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:13:08.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.197173+0000", 2024-03-20T09:13:08.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:13:08.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.197175+0000", 2024-03-20T09:13:08.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.305e-06 2024-03-20T09:13:08.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:13:08.856 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.197185+0000", 2024-03-20T09:13:08.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9289999999999993e-06 2024-03-20T09:13:08.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:13:08.856 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.197208+0000", 2024-03-20T09:13:08.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2748e-05 2024-03-20T09:13:08.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:13:08.856 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.197220+0000", 2024-03-20T09:13:08.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1907e-05 2024-03-20T09:13:08.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.857 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.197288+0000", 2024-03-20T09:13:08.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8713000000000004e-05 2024-03-20T09:13:08.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.857 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:13:08.857 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.197297+0000", 2024-03-20T09:13:08.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9970000000000004e-06 2024-03-20T09:13:08.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.857 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.857 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.159349+0000", 2024-03-20T09:13:08.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.96205180099999998 2024-03-20T09:13:08.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.858 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:13:08.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.159486+0000", 2024-03-20T09:13:08.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013625700000000001 2024-03-20T09:13:08.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.858 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:13:08.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.159611+0000", 2024-03-20T09:13:08.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012543 2024-03-20T09:13:08.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.858 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.164084+0000", 2024-03-20T09:13:08.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004472696 2024-03-20T09:13:08.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:13:08.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.165538+0000", 2024-03-20T09:13:08.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014541210000000001 2024-03-20T09:13:08.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.168806+0000", 2024-03-20T09:13:08.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0032684020000000001 2024-03-20T09:13:08.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:13:08.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.168846+0000", 2024-03-20T09:13:08.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9968000000000001e-05 2024-03-20T09:13:08.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.860 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:13:08.860 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.168878+0000", 2024-03-20T09:13:08.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1466e-05 2024-03-20T09:13:08.860 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.860 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:13:08.860 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.860 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:143592 3.7 3:e6af9103:::benchmark_data_smithi122_119671_object43810:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e449)", 2024-03-20T09:13:08.860 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:10:41.340924+0000", 2024-03-20T09:13:08.860 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 147.49689121500001, 2024-03-20T09:13:08.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.82804276799999998, 2024-03-20T09:13:08.860 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:13:08.860 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:13:08.861 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:13:08.861 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:13:08.861 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:13:08.861 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 143592 2024-03-20T09:13:08.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.861 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:13:08.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.861 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:13:08.861 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.340924+0000", 2024-03-20T09:13:08.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.861 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:13:08.861 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.340924+0000", 2024-03-20T09:13:08.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.862 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:13:08.862 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.340925+0000", 2024-03-20T09:13:08.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.482e-06 2024-03-20T09:13:08.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.862 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:13:08.862 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.340938+0000", 2024-03-20T09:13:08.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2612999999999999e-05 2024-03-20T09:13:08.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.862 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:13:08.862 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.340954+0000", 2024-03-20T09:13:08.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6184999999999998e-05 2024-03-20T09:13:08.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:13:08.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.340962+0000", 2024-03-20T09:13:08.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8650000000000001e-06 2024-03-20T09:13:08.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.341020+0000", 2024-03-20T09:13:08.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7757999999999998e-05 2024-03-20T09:13:08.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:13:08.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.341028+0000", 2024-03-20T09:13:08.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5869999999999993e-06 2024-03-20T09:13:08.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.161133+0000", 2024-03-20T09:13:08.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.82010472199999995 2024-03-20T09:13:08.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:13:08.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.161206+0000", 2024-03-20T09:13:08.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3403000000000002e-05 2024-03-20T09:13:08.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:13:08.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.161299+0000", 2024-03-20T09:13:08.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2936999999999995e-05 2024-03-20T09:13:08.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.865 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.865 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.164125+0000", 2024-03-20T09:13:08.865 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028253280000000002 2024-03-20T09:13:08.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.865 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:13:08.865 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.165549+0000", 2024-03-20T09:13:08.865 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423913 2024-03-20T09:13:08.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.865 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.865 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.168917+0000", 2024-03-20T09:13:08.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003368692 2024-03-20T09:13:08.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.866 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:13:08.866 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.168944+0000", 2024-03-20T09:13:08.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7228000000000001e-05 2024-03-20T09:13:08.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.866 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:13:08.866 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.168967+0000", 2024-03-20T09:13:08.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2055000000000001e-05 2024-03-20T09:13:08.866 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.866 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:13:08.867 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.867 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:143603 3.7 3:e63a0e26:::benchmark_data_smithi122_119671_object43821:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e449)", 2024-03-20T09:13:08.867 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:10:41.451662+0000", 2024-03-20T09:13:08.867 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 147.38615326799999, 2024-03-20T09:13:08.867 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.71737729100000003, 2024-03-20T09:13:08.867 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:13:08.867 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:13:08.867 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:13:08.867 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:13:08.867 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:13:08.867 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 143603 2024-03-20T09:13:08.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.867 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:13:08.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:13:08.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.451662+0000", 2024-03-20T09:13:08.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:13:08.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.451662+0000", 2024-03-20T09:13:08.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:13:08.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.451664+0000", 2024-03-20T09:13:08.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.933e-06 2024-03-20T09:13:08.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:13:08.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.451672+0000", 2024-03-20T09:13:08.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8270000000000004e-06 2024-03-20T09:13:08.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:13:08.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.451693+0000", 2024-03-20T09:13:08.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0458e-05 2024-03-20T09:13:08.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.869 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:13:08.869 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.451705+0000", 2024-03-20T09:13:08.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1559999999999999e-05 2024-03-20T09:13:08.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.870 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.870 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.451772+0000", 2024-03-20T09:13:08.870 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7799999999999995e-05 2024-03-20T09:13:08.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.870 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:13:08.870 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.451782+0000", 2024-03-20T09:13:08.870 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8609999999999993e-06 2024-03-20T09:13:08.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.870 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.871 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.162652+0000", 2024-03-20T09:13:08.871 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.71086976499999999 2024-03-20T09:13:08.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.871 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:13:08.871 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.162722+0000", 2024-03-20T09:13:08.871 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9754999999999999e-05 2024-03-20T09:13:08.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.871 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:13:08.871 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.162785+0000", 2024-03-20T09:13:08.871 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3094000000000003e-05 2024-03-20T09:13:08.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.871 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.872 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.165646+0000", 2024-03-20T09:13:08.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028609920000000001 2024-03-20T09:13:08.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.872 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:13:08.872 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.167291+0000", 2024-03-20T09:13:08.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001644838 2024-03-20T09:13:08.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.872 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.872 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.168994+0000", 2024-03-20T09:13:08.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0017038559999999999 2024-03-20T09:13:08.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:13:08.873 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.169020+0000", 2024-03-20T09:13:08.873 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5122000000000001e-05 2024-03-20T09:13:08.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:13:08.873 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.169039+0000", 2024-03-20T09:13:08.873 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9429999999999999e-05 2024-03-20T09:13:08.873 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.873 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:13:08.873 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.873 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:143621 3.7 3:e5e3109e:::benchmark_data_smithi122_119671_object43839:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e449)", 2024-03-20T09:13:08.874 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:10:41.769244+0000", 2024-03-20T09:13:08.874 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 147.068571136, 2024-03-20T09:13:08.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.41200099499999998, 2024-03-20T09:13:08.874 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:13:08.874 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:13:08.874 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:13:08.874 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:13:08.874 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:13:08.874 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 143621 2024-03-20T09:13:08.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.874 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:13:08.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.874 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:13:08.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.769244+0000", 2024-03-20T09:13:08.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:13:08.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.769244+0000", 2024-03-20T09:13:08.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:13:08.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.769246+0000", 2024-03-20T09:13:08.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9640000000000002e-06 2024-03-20T09:13:08.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:13:08.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.769254+0000", 2024-03-20T09:13:08.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0649999999999999e-06 2024-03-20T09:13:08.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:13:08.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.769272+0000", 2024-03-20T09:13:08.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8436000000000001e-05 2024-03-20T09:13:08.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:13:08.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.769284+0000", 2024-03-20T09:13:08.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1252e-05 2024-03-20T09:13:08.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.877 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.769364+0000", 2024-03-20T09:13:08.877 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0786999999999997e-05 2024-03-20T09:13:08.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.877 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:13:08.877 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.769381+0000", 2024-03-20T09:13:08.877 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6588000000000001e-05 2024-03-20T09:13:08.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.877 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.877 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.164160+0000", 2024-03-20T09:13:08.877 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.39477866499999997 2024-03-20T09:13:08.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.878 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:13:08.878 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.164250+0000", 2024-03-20T09:13:08.878 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0861999999999998e-05 2024-03-20T09:13:08.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.878 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:13:08.878 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.164342+0000", 2024-03-20T09:13:08.878 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1445000000000003e-05 2024-03-20T09:13:08.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.878 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.878 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.166922+0000", 2024-03-20T09:13:08.878 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0025798319999999998 2024-03-20T09:13:08.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.879 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:13:08.879 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.167297+0000", 2024-03-20T09:13:08.879 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00037566299999999998 2024-03-20T09:13:08.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.879 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.879 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.181135+0000", 2024-03-20T09:13:08.879 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013837649 2024-03-20T09:13:08.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.879 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:13:08.879 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.181197+0000", 2024-03-20T09:13:08.879 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1902999999999993e-05 2024-03-20T09:13:08.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:13:08.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.181245+0000", 2024-03-20T09:13:08.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7883999999999998e-05 2024-03-20T09:13:08.880 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.880 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:13:08.880 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.880 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:163495 3.b 3:d6e891b8:::benchmark_data_smithi122_119671_object63713:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-03-20T09:13:08.880 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:12:06.244771+0000", 2024-03-20T09:13:08.880 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.593043643000001, 2024-03-20T09:13:08.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.82372252499999998, 2024-03-20T09:13:08.881 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:13:08.881 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:13:08.881 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:13:08.881 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:13:08.881 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:13:08.881 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 163495 2024-03-20T09:13:08.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.881 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:13:08.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:13:08.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.244771+0000", 2024-03-20T09:13:08.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.882 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:13:08.882 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.244771+0000", 2024-03-20T09:13:08.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.882 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:13:08.882 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.244773+0000", 2024-03-20T09:13:08.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.314e-06 2024-03-20T09:13:08.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.882 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:13:08.882 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.244778+0000", 2024-03-20T09:13:08.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.967e-06 2024-03-20T09:13:08.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.883 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:13:08.883 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.244788+0000", 2024-03-20T09:13:08.883 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0421000000000001e-05 2024-03-20T09:13:08.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.883 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:13:08.883 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.244811+0000", 2024-03-20T09:13:08.883 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2806000000000002e-05 2024-03-20T09:13:08.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.883 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.883 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.244857+0000", 2024-03-20T09:13:08.884 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5753000000000001e-05 2024-03-20T09:13:08.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.884 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:13:08.884 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.244862+0000", 2024-03-20T09:13:08.884 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7320000000000003e-06 2024-03-20T09:13:08.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.884 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.884 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.060607+0000", 2024-03-20T09:13:08.884 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.81574488999999994 2024-03-20T09:13:08.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.884 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:13:08.884 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.060700+0000", 2024-03-20T09:13:08.885 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2989e-05 2024-03-20T09:13:08.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.885 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:13:08.885 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.060781+0000", 2024-03-20T09:13:08.885 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0813e-05 2024-03-20T09:13:08.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.885 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.885 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.068238+0000", 2024-03-20T09:13:08.885 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074570679999999999 2024-03-20T09:13:08.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.885 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:13:08.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.068250+0000", 2024-03-20T09:13:08.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1725999999999999e-05 2024-03-20T09:13:08.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.068372+0000", 2024-03-20T09:13:08.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000122027 2024-03-20T09:13:08.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:13:08.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.068440+0000", 2024-03-20T09:13:08.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7723000000000002e-05 2024-03-20T09:13:08.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.887 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:13:08.887 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.068494+0000", 2024-03-20T09:13:08.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4296000000000003e-05 2024-03-20T09:13:08.887 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.887 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:13:08.887 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.887 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:163499 3.b 3:d6314864:::benchmark_data_smithi122_119671_object63717:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-03-20T09:13:08.887 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:12:06.250105+0000", 2024-03-20T09:13:08.887 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.587709857, 2024-03-20T09:13:08.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.82241188399999998, 2024-03-20T09:13:08.888 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:13:08.888 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:13:08.888 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:13:08.888 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:13:08.888 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:13:08.888 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 163499 2024-03-20T09:13:08.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.888 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:13:08.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.888 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:13:08.888 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.250105+0000", 2024-03-20T09:13:08.888 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.888 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:13:08.889 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.250105+0000", 2024-03-20T09:13:08.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.889 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:13:08.889 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.250106+0000", 2024-03-20T09:13:08.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2559999999999999e-06 2024-03-20T09:13:08.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.889 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:13:08.889 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.250112+0000", 2024-03-20T09:13:08.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5990000000000001e-06 2024-03-20T09:13:08.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.889 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:13:08.890 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.250127+0000", 2024-03-20T09:13:08.890 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4936000000000001e-05 2024-03-20T09:13:08.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.890 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:13:08.890 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.250136+0000", 2024-03-20T09:13:08.890 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1740000000000006e-06 2024-03-20T09:13:08.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.890 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.890 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.250201+0000", 2024-03-20T09:13:08.890 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5128000000000004e-05 2024-03-20T09:13:08.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.891 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:13:08.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.250212+0000", 2024-03-20T09:13:08.891 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0297e-05 2024-03-20T09:13:08.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.891 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.064203+0000", 2024-03-20T09:13:08.891 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.81399148600000004 2024-03-20T09:13:08.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.891 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:13:08.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.064296+0000", 2024-03-20T09:13:08.891 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2652999999999995e-05 2024-03-20T09:13:08.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.892 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:13:08.892 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.064434+0000", 2024-03-20T09:13:08.892 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000138329 2024-03-20T09:13:08.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.892 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.892 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.068314+0000", 2024-03-20T09:13:08.892 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0038798249999999999 2024-03-20T09:13:08.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.892 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.892 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.068649+0000", 2024-03-20T09:13:08.892 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00033493599999999998 2024-03-20T09:13:08.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.893 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:13:08.893 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.072408+0000", 2024-03-20T09:13:08.893 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037591949999999999 2024-03-20T09:13:08.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.893 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:13:08.893 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.072469+0000", 2024-03-20T09:13:08.893 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1354999999999998e-05 2024-03-20T09:13:08.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.893 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:13:08.893 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.072517+0000", 2024-03-20T09:13:08.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7715e-05 2024-03-20T09:13:08.894 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.894 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:13:08.894 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.894 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:163507 3.b 3:dfa7f600:::benchmark_data_smithi122_119671_object63725:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-03-20T09:13:08.894 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:12:06.258712+0000", 2024-03-20T09:13:08.894 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.579103154999999, 2024-03-20T09:13:08.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.82676451500000003, 2024-03-20T09:13:08.894 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:13:08.894 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:13:08.894 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:13:08.895 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:13:08.895 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:13:08.895 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 163507 2024-03-20T09:13:08.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.895 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:13:08.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:13:08.895 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.258712+0000", 2024-03-20T09:13:08.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:13:08.895 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.258712+0000", 2024-03-20T09:13:08.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.896 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:13:08.896 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.258712+0000", 2024-03-20T09:13:08.896 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8100000000000003e-07 2024-03-20T09:13:08.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.896 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:13:08.896 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.258716+0000", 2024-03-20T09:13:08.896 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.191e-06 2024-03-20T09:13:08.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.896 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:13:08.896 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.258724+0000", 2024-03-20T09:13:08.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5080000000000005e-06 2024-03-20T09:13:08.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.897 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:13:08.897 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.258730+0000", 2024-03-20T09:13:08.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9089999999999997e-06 2024-03-20T09:13:08.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.897 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.897 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.258783+0000", 2024-03-20T09:13:08.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2624999999999998e-05 2024-03-20T09:13:08.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.897 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:13:08.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.258792+0000", 2024-03-20T09:13:08.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8269999999999993e-06 2024-03-20T09:13:08.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.068709+0000", 2024-03-20T09:13:08.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.80991684200000003 2024-03-20T09:13:08.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:13:08.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.068844+0000", 2024-03-20T09:13:08.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013473200000000001 2024-03-20T09:13:08.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:13:08.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.068987+0000", 2024-03-20T09:13:08.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000143356 2024-03-20T09:13:08.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.076876+0000", 2024-03-20T09:13:08.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0078885090000000001 2024-03-20T09:13:08.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.085245+0000", 2024-03-20T09:13:08.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083692100000000002 2024-03-20T09:13:08.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:13:08.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.085367+0000", 2024-03-20T09:13:08.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000122364 2024-03-20T09:13:08.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:13:08.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.085429+0000", 2024-03-20T09:13:08.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1951000000000004e-05 2024-03-20T09:13:08.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:13:08.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.085476+0000", 2024-03-20T09:13:08.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6910000000000003e-05 2024-03-20T09:13:08.901 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.901 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:13:08.901 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.901 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:163508 3.b 3:dabf16ab:::benchmark_data_smithi122_119671_object63726:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-03-20T09:13:08.901 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:12:06.260617+0000", 2024-03-20T09:13:08.901 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.577198373000002, 2024-03-20T09:13:08.901 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.824912125, 2024-03-20T09:13:08.901 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:13:08.901 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:13:08.901 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:13:08.901 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:13:08.901 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:13:08.901 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 163508 2024-03-20T09:13:08.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.902 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:13:08.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.902 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:13:08.902 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.260617+0000", 2024-03-20T09:13:08.902 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.902 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:13:08.902 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.260617+0000", 2024-03-20T09:13:08.902 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.903 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:13:08.903 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.260618+0000", 2024-03-20T09:13:08.903 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.066e-06 2024-03-20T09:13:08.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.903 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:13:08.903 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.260623+0000", 2024-03-20T09:13:08.903 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1959999999999999e-06 2024-03-20T09:13:08.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.903 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:13:08.903 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.260634+0000", 2024-03-20T09:13:08.903 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1353e-05 2024-03-20T09:13:08.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:13:08.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.260641+0000", 2024-03-20T09:13:08.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9380000000000003e-06 2024-03-20T09:13:08.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.260687+0000", 2024-03-20T09:13:08.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5590999999999998e-05 2024-03-20T09:13:08.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:13:08.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.260693+0000", 2024-03-20T09:13:08.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0109999999999997e-06 2024-03-20T09:13:08.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.905 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.072590+0000", 2024-03-20T09:13:08.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.81189731200000004 2024-03-20T09:13:08.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.905 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:13:08.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.072706+0000", 2024-03-20T09:13:08.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011615 2024-03-20T09:13:08.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:13:08.906 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.072855+0000", 2024-03-20T09:13:08.906 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000148398 2024-03-20T09:13:08.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.906 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.076926+0000", 2024-03-20T09:13:08.906 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040710420000000004 2024-03-20T09:13:08.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.906 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.085349+0000", 2024-03-20T09:13:08.907 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084229460000000006 2024-03-20T09:13:08.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.907 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:13:08.907 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.085483+0000", 2024-03-20T09:13:08.907 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013404199999999999 2024-03-20T09:13:08.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.907 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:13:08.907 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.085508+0000", 2024-03-20T09:13:08.907 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5378e-05 2024-03-20T09:13:08.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.907 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:13:08.907 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.085529+0000", 2024-03-20T09:13:08.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0701999999999999e-05 2024-03-20T09:13:08.908 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.908 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:13:08.908 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.908 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:163533 3.b 3:d6f3d966:::benchmark_data_smithi122_119671_object63751:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-03-20T09:13:08.908 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:12:06.364729+0000", 2024-03-20T09:13:08.908 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.473085521000002, 2024-03-20T09:13:08.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.72093737099999999, 2024-03-20T09:13:08.908 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:13:08.908 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:13:08.908 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:13:08.909 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:13:08.909 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:13:08.909 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 163533 2024-03-20T09:13:08.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.909 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:13:08.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.909 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:13:08.909 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.364729+0000", 2024-03-20T09:13:08.909 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.909 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:13:08.909 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.364729+0000", 2024-03-20T09:13:08.909 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:13:08.910 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.364731+0000", 2024-03-20T09:13:08.910 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4699999999999999e-06 2024-03-20T09:13:08.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:13:08.910 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.364740+0000", 2024-03-20T09:13:08.910 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8319999999999995e-06 2024-03-20T09:13:08.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:13:08.910 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.364755+0000", 2024-03-20T09:13:08.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.562e-05 2024-03-20T09:13:08.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:13:08.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.364766+0000", 2024-03-20T09:13:08.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.028e-05 2024-03-20T09:13:08.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.364844+0000", 2024-03-20T09:13:08.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8495e-05 2024-03-20T09:13:08.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.912 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:13:08.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.364855+0000", 2024-03-20T09:13:08.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0869999999999999e-05 2024-03-20T09:13:08.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.912 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.076960+0000", 2024-03-20T09:13:08.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.71210485999999995 2024-03-20T09:13:08.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.912 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:13:08.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.077106+0000", 2024-03-20T09:13:08.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000145916 2024-03-20T09:13:08.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.913 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:13:08.913 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.077250+0000", 2024-03-20T09:13:08.913 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000143783 2024-03-20T09:13:08.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.913 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.913 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.085196+0000", 2024-03-20T09:13:08.913 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0079463390000000002 2024-03-20T09:13:08.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.913 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:13:08.913 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.085535+0000", 2024-03-20T09:13:08.913 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00033950400000000002 2024-03-20T09:13:08.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.914 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.914 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.085572+0000", 2024-03-20T09:13:08.914 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.608e-05 2024-03-20T09:13:08.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.914 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:13:08.914 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.085623+0000", 2024-03-20T09:13:08.914 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1431999999999998e-05 2024-03-20T09:13:08.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.914 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:13:08.914 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.085667+0000", 2024-03-20T09:13:08.915 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3890000000000002e-05 2024-03-20T09:13:08.915 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.915 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:13:08.915 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.915 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:163553 3.b 3:d15539c8:::benchmark_data_smithi122_119671_object63771:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-03-20T09:13:08.915 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:12:06.406918+0000", 2024-03-20T09:13:08.915 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.430897070999997, 2024-03-20T09:13:08.915 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.69573252200000002, 2024-03-20T09:13:08.915 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:13:08.915 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:13:08.915 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:13:08.915 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:13:08.915 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:13:08.916 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 163553 2024-03-20T09:13:08.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.916 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:13:08.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.916 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:13:08.916 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.406918+0000", 2024-03-20T09:13:08.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.916 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:13:08.916 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.406918+0000", 2024-03-20T09:13:08.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.916 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:13:08.916 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.406920+0000", 2024-03-20T09:13:08.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7439999999999999e-06 2024-03-20T09:13:08.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:13:08.917 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.406928+0000", 2024-03-20T09:13:08.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3960000000000006e-06 2024-03-20T09:13:08.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:13:08.917 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.406945+0000", 2024-03-20T09:13:08.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6804999999999999e-05 2024-03-20T09:13:08.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:13:08.918 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.406956+0000", 2024-03-20T09:13:08.918 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0675e-05 2024-03-20T09:13:08.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.918 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.918 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.407023+0000", 2024-03-20T09:13:08.918 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7341000000000005e-05 2024-03-20T09:13:08.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.918 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:13:08.918 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.407030+0000", 2024-03-20T09:13:08.918 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3880000000000002e-06 2024-03-20T09:13:08.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.919 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.919 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.080645+0000", 2024-03-20T09:13:08.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.67361427100000004 2024-03-20T09:13:08.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.919 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:13:08.919 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.080811+0000", 2024-03-20T09:13:08.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00016642800000000001 2024-03-20T09:13:08.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.919 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:13:08.919 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.080970+0000", 2024-03-20T09:13:08.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000158555 2024-03-20T09:13:08.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.920 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.085283+0000", 2024-03-20T09:13:08.920 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043139420000000003 2024-03-20T09:13:08.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.920 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.094146+0000", 2024-03-20T09:13:08.920 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0088626590000000002 2024-03-20T09:13:08.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:13:08.920 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.102539+0000", 2024-03-20T09:13:08.920 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083926090000000005 2024-03-20T09:13:08.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:13:08.921 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.102594+0000", 2024-03-20T09:13:08.921 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4925e-05 2024-03-20T09:13:08.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.921 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:13:08.921 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.102650+0000", 2024-03-20T09:13:08.921 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6783999999999997e-05 2024-03-20T09:13:08.921 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.921 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:13:08.921 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.921 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:163557 3.b 3:d0818a0e:::benchmark_data_smithi122_119671_object63775:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-03-20T09:13:08.921 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:12:06.416389+0000", 2024-03-20T09:13:08.922 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.421425669999998, 2024-03-20T09:13:08.922 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.68649045900000005, 2024-03-20T09:13:08.922 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:13:08.922 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:13:08.922 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:13:08.922 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:13:08.922 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:13:08.922 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 163557 2024-03-20T09:13:08.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.922 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:13:08.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.922 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:13:08.922 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.416389+0000", 2024-03-20T09:13:08.922 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:13:08.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.416389+0000", 2024-03-20T09:13:08.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:13:08.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.416391+0000", 2024-03-20T09:13:08.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7379999999999999e-06 2024-03-20T09:13:08.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:13:08.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.416399+0000", 2024-03-20T09:13:08.924 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4560000000000003e-06 2024-03-20T09:13:08.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.924 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:13:08.924 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.416416+0000", 2024-03-20T09:13:08.924 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6983999999999999e-05 2024-03-20T09:13:08.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.924 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:13:08.924 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.416427+0000", 2024-03-20T09:13:08.924 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1e-05 2024-03-20T09:13:08.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.924 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.924 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.416494+0000", 2024-03-20T09:13:08.925 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7621999999999997e-05 2024-03-20T09:13:08.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.925 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:13:08.925 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.416504+0000", 2024-03-20T09:13:08.925 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3770000000000002e-06 2024-03-20T09:13:08.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.925 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.925 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.085706+0000", 2024-03-20T09:13:08.925 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.669202301 2024-03-20T09:13:08.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.925 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:13:08.925 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.085852+0000", 2024-03-20T09:13:08.926 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000146589 2024-03-20T09:13:08.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.926 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:13:08.926 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.085973+0000", 2024-03-20T09:13:08.926 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000120389 2024-03-20T09:13:08.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.926 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.926 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.094189+0000", 2024-03-20T09:13:08.926 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0082157429999999993 2024-03-20T09:13:08.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.927 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:13:08.927 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.102717+0000", 2024-03-20T09:13:08.927 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0085289349999999996 2024-03-20T09:13:08.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.927 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.927 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.102775+0000", 2024-03-20T09:13:08.927 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7046000000000002e-05 2024-03-20T09:13:08.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.927 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:13:08.927 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.102836+0000", 2024-03-20T09:13:08.927 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1901000000000002e-05 2024-03-20T09:13:08.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.928 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:13:08.928 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.102880+0000", 2024-03-20T09:13:08.928 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3377999999999997e-05 2024-03-20T09:13:08.928 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.928 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:13:08.928 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.928 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:163567 3.b 3:d6a40b92:::benchmark_data_smithi122_119671_object63785:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-03-20T09:13:08.928 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:12:06.454340+0000", 2024-03-20T09:13:08.928 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.383474904000003, 2024-03-20T09:13:08.928 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.64869723599999995, 2024-03-20T09:13:08.929 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:13:08.929 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:13:08.929 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:13:08.929 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:13:08.929 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:13:08.929 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 163567 2024-03-20T09:13:08.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.929 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:13:08.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.929 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:13:08.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.454340+0000", 2024-03-20T09:13:08.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:13:08.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.454340+0000", 2024-03-20T09:13:08.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:13:08.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.454341+0000", 2024-03-20T09:13:08.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.093e-06 2024-03-20T09:13:08.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:13:08.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.454346+0000", 2024-03-20T09:13:08.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3109999999999996e-06 2024-03-20T09:13:08.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:13:08.931 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.454358+0000", 2024-03-20T09:13:08.931 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.189e-05 2024-03-20T09:13:08.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:13:08.931 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.454365+0000", 2024-03-20T09:13:08.931 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.37e-06 2024-03-20T09:13:08.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.931 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.454429+0000", 2024-03-20T09:13:08.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3829000000000002e-05 2024-03-20T09:13:08.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.932 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:13:08.932 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.454441+0000", 2024-03-20T09:13:08.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2357e-05 2024-03-20T09:13:08.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.932 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.932 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.089918+0000", 2024-03-20T09:13:08.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.63547673900000001 2024-03-20T09:13:08.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.932 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:13:08.933 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.090060+0000", 2024-03-20T09:13:08.933 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000142755 2024-03-20T09:13:08.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.933 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:13:08.933 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.090156+0000", 2024-03-20T09:13:08.933 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5086000000000004e-05 2024-03-20T09:13:08.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.933 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.933 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.102520+0000", 2024-03-20T09:13:08.933 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012364828 2024-03-20T09:13:08.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.934 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:13:08.934 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.102724+0000", 2024-03-20T09:13:08.934 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000204117 2024-03-20T09:13:08.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.934 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.934 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.102971+0000", 2024-03-20T09:13:08.934 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00024605900000000002 2024-03-20T09:13:08.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.934 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:13:08.934 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.103016+0000", 2024-03-20T09:13:08.934 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5182000000000002e-05 2024-03-20T09:13:08.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.935 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:13:08.935 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.103037+0000", 2024-03-20T09:13:08.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.162e-05 2024-03-20T09:13:08.935 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.935 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:13:08.935 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.935 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:163570 3.b 3:db3a9564:::benchmark_data_smithi122_119671_object63788:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-03-20T09:13:08.935 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:12:06.463745+0000", 2024-03-20T09:13:08.935 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.374070097999997, 2024-03-20T09:13:08.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.65138066100000003, 2024-03-20T09:13:08.935 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:13:08.936 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:13:08.936 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:13:08.936 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:13:08.936 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:13:08.936 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 163570 2024-03-20T09:13:08.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.936 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:13:08.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.936 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:13:08.936 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.463745+0000", 2024-03-20T09:13:08.936 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.936 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:13:08.936 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.463745+0000", 2024-03-20T09:13:08.937 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.937 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:13:08.937 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.463747+0000", 2024-03-20T09:13:08.937 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.857e-06 2024-03-20T09:13:08.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.937 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:13:08.937 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.463755+0000", 2024-03-20T09:13:08.937 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2810000000000008e-06 2024-03-20T09:13:08.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.937 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:13:08.938 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.463775+0000", 2024-03-20T09:13:08.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9536999999999999e-05 2024-03-20T09:13:08.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.938 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:13:08.938 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.463786+0000", 2024-03-20T09:13:08.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1396000000000001e-05 2024-03-20T09:13:08.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.938 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.938 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.463842+0000", 2024-03-20T09:13:08.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6248000000000001e-05 2024-03-20T09:13:08.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.939 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:13:08.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.463850+0000", 2024-03-20T09:13:08.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9510000000000006e-06 2024-03-20T09:13:08.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.939 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.094218+0000", 2024-03-20T09:13:08.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.63036753700000003 2024-03-20T09:13:08.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.939 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:13:08.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.094319+0000", 2024-03-20T09:13:08.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101702 2024-03-20T09:13:08.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.940 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:13:08.940 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.094428+0000", 2024-03-20T09:13:08.940 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000109015 2024-03-20T09:13:08.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.940 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.940 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.102703+0000", 2024-03-20T09:13:08.940 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0082749 2024-03-20T09:13:08.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.940 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:13:08.940 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.102732+0000", 2024-03-20T09:13:08.940 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8348e-05 2024-03-20T09:13:08.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.940 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.941 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.115019+0000", 2024-03-20T09:13:08.941 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012287752000000001 2024-03-20T09:13:08.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.941 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:13:08.941 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.115076+0000", 2024-03-20T09:13:08.941 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6524999999999998e-05 2024-03-20T09:13:08.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.941 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:13:08.941 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.115126+0000", 2024-03-20T09:13:08.941 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9611999999999999e-05 2024-03-20T09:13:08.941 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.941 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:13:08.941 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.942 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:163600 3.b 3:d30ded93:::benchmark_data_smithi122_119671_object63818:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-03-20T09:13:08.942 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:12:06.524642+0000", 2024-03-20T09:13:08.942 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.313173356, 2024-03-20T09:13:08.942 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.59071154400000003, 2024-03-20T09:13:08.942 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:13:08.942 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:13:08.942 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:13:08.942 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:13:08.942 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:13:08.942 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 163600 2024-03-20T09:13:08.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.942 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:13:08.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.943 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:13:08.943 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.524642+0000", 2024-03-20T09:13:08.943 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.943 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:13:08.943 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.524642+0000", 2024-03-20T09:13:08.943 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.943 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:13:08.943 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.524643+0000", 2024-03-20T09:13:08.943 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.133e-06 2024-03-20T09:13:08.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.944 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:13:08.944 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.524648+0000", 2024-03-20T09:13:08.944 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.887e-06 2024-03-20T09:13:08.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.944 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:13:08.944 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.524658+0000", 2024-03-20T09:13:08.944 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0647e-05 2024-03-20T09:13:08.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.944 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:13:08.944 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.524664+0000", 2024-03-20T09:13:08.944 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3419999999999998e-06 2024-03-20T09:13:08.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.945 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.945 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.524723+0000", 2024-03-20T09:13:08.945 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9837000000000003e-05 2024-03-20T09:13:08.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.945 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:13:08.945 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.524733+0000", 2024-03-20T09:13:08.945 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8500000000000006e-06 2024-03-20T09:13:08.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.945 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.945 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.098249+0000", 2024-03-20T09:13:08.945 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.57351528699999998 2024-03-20T09:13:08.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.946 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:13:08.946 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.098362+0000", 2024-03-20T09:13:08.946 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000113248 2024-03-20T09:13:08.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.946 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.946 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:13:08.946 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.098452+0000", 2024-03-20T09:13:08.946 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0093999999999995e-05 2024-03-20T09:13:08.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.946 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.946 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:13:08.946 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.102736+0000", 2024-03-20T09:13:08.946 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042841750000000003 2024-03-20T09:13:08.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.947 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.947 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.102925+0000", 2024-03-20T09:13:08.947 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018848000000000001 2024-03-20T09:13:08.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.947 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.947 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.115270+0000", 2024-03-20T09:13:08.947 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012345027999999999 2024-03-20T09:13:08.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.947 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:13:08.947 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.115313+0000", 2024-03-20T09:13:08.948 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3257000000000003e-05 2024-03-20T09:13:08.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.948 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:13:08.948 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.115353+0000", 2024-03-20T09:13:08.948 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0278999999999998e-05 2024-03-20T09:13:08.948 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.948 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:13:08.948 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.948 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:163616 3.b 3:df6321bf:::benchmark_data_smithi122_119671_object63834:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-03-20T09:13:08.949 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:12:06.574476+0000", 2024-03-20T09:13:08.949 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.263338953000002, 2024-03-20T09:13:08.949 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.54878080699999998, 2024-03-20T09:13:08.949 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:13:08.949 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:13:08.949 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:13:08.949 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:13:08.949 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:13:08.949 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 163616 2024-03-20T09:13:08.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.949 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:13:08.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.949 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:13:08.949 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.574476+0000", 2024-03-20T09:13:08.949 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.950 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:13:08.950 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.574476+0000", 2024-03-20T09:13:08.950 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.950 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:13:08.950 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.574477+0000", 2024-03-20T09:13:08.950 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4100000000000001e-06 2024-03-20T09:13:08.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.950 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:13:08.950 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.574484+0000", 2024-03-20T09:13:08.951 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3110000000000002e-06 2024-03-20T09:13:08.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.951 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:13:08.951 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.574499+0000", 2024-03-20T09:13:08.951 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5218999999999999e-05 2024-03-20T09:13:08.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.951 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:13:08.951 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.574508+0000", 2024-03-20T09:13:08.951 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3409999999999992e-06 2024-03-20T09:13:08.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.951 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.952 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.574564+0000", 2024-03-20T09:13:08.952 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5970000000000001e-05 2024-03-20T09:13:08.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.952 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:13:08.952 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.574572+0000", 2024-03-20T09:13:08.952 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1640000000000001e-06 2024-03-20T09:13:08.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.952 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.952 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.103055+0000", 2024-03-20T09:13:08.952 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.52848255700000002 2024-03-20T09:13:08.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.952 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:13:08.953 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.103126+0000", 2024-03-20T09:13:08.953 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1009999999999999e-05 2024-03-20T09:13:08.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.953 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:13:08.953 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.103211+0000", 2024-03-20T09:13:08.953 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4796000000000006e-05 2024-03-20T09:13:08.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.953 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:13:08.953 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.115139+0000", 2024-03-20T09:13:08.953 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011927917 2024-03-20T09:13:08.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.115221+0000", 2024-03-20T09:13:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2009999999999996e-05 2024-03-20T09:13:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.123176+0000", 2024-03-20T09:13:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0079552709999999999 2024-03-20T09:13:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:13:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.123210+0000", 2024-03-20T09:13:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4341000000000002e-05 2024-03-20T09:13:08.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:13:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.123257+0000", 2024-03-20T09:13:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6489999999999997e-05 2024-03-20T09:13:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:13:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:163636 3.b 3:ddb46087:::benchmark_data_smithi122_119671_object63854:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-03-20T09:13:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:12:06.606952+0000", 2024-03-20T09:13:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.230863333999999, 2024-03-20T09:13:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.516426465, 2024-03-20T09:13:08.955 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:13:08.956 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:13:08.956 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:13:08.956 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:13:08.956 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:13:08.956 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 163636 2024-03-20T09:13:08.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.956 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:13:08.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.956 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:13:08.956 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.606952+0000", 2024-03-20T09:13:08.956 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:13:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.606952+0000", 2024-03-20T09:13:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:13:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.606953+0000", 2024-03-20T09:13:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.19e-06 2024-03-20T09:13:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:13:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.606959+0000", 2024-03-20T09:13:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8259999999999998e-06 2024-03-20T09:13:08.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:13:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.606971+0000", 2024-03-20T09:13:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2592000000000001e-05 2024-03-20T09:13:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:13:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.606981+0000", 2024-03-20T09:13:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2399999999999996e-06 2024-03-20T09:13:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.958 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.607044+0000", 2024-03-20T09:13:08.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3427000000000001e-05 2024-03-20T09:13:08.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:13:08.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.607054+0000", 2024-03-20T09:13:08.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0339e-05 2024-03-20T09:13:08.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.106740+0000", 2024-03-20T09:13:08.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.49968526000000002 2024-03-20T09:13:08.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:13:08.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.106865+0000", 2024-03-20T09:13:08.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012532600000000001 2024-03-20T09:13:08.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:13:08.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.106982+0000", 2024-03-20T09:13:08.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011677599999999999 2024-03-20T09:13:08.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:13:08.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.119379+0000", 2024-03-20T09:13:08.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01239751 2024-03-20T09:13:08.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.961 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.123116+0000", 2024-03-20T09:13:08.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037364590000000001 2024-03-20T09:13:08.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.961 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.123289+0000", 2024-03-20T09:13:08.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00017384099999999999 2024-03-20T09:13:08.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:13:08.961 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.123336+0000", 2024-03-20T09:13:08.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6400000000000003e-05 2024-03-20T09:13:08.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:13:08.962 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.123378+0000", 2024-03-20T09:13:08.962 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2279e-05 2024-03-20T09:13:08.962 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.962 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:13:08.962 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.962 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:163675 3.b 3:de6813f5:::benchmark_data_smithi122_119671_object63893:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-03-20T09:13:08.962 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:12:06.673840+0000", 2024-03-20T09:13:08.962 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.163975116000003, 2024-03-20T09:13:08.962 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.44968040399999998, 2024-03-20T09:13:08.962 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:13:08.962 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:13:08.962 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:13:08.963 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:13:08.963 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:13:08.963 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 163675 2024-03-20T09:13:08.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.963 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:13:08.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.963 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:13:08.963 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.673840+0000", 2024-03-20T09:13:08.963 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.963 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:13:08.963 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.673840+0000", 2024-03-20T09:13:08.963 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:13:08.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.673841+0000", 2024-03-20T09:13:08.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.342e-06 2024-03-20T09:13:08.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:13:08.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.673847+0000", 2024-03-20T09:13:08.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0780000000000001e-06 2024-03-20T09:13:08.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.966 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:13:08.966 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.673859+0000", 2024-03-20T09:13:08.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.218e-05 2024-03-20T09:13:08.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.966 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:13:08.966 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.673867+0000", 2024-03-20T09:13:08.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.418e-06 2024-03-20T09:13:08.966 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.673946+0000", 2024-03-20T09:13:08.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9443999999999994e-05 2024-03-20T09:13:08.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:13:08.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.673957+0000", 2024-03-20T09:13:08.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1094000000000001e-05 2024-03-20T09:13:08.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.111018+0000", 2024-03-20T09:13:08.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.43706057700000001 2024-03-20T09:13:08.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:13:08.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.111115+0000", 2024-03-20T09:13:08.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.6879999999999994e-05 2024-03-20T09:13:08.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:13:08.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.111216+0000", 2024-03-20T09:13:08.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010155 2024-03-20T09:13:08.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:13:08.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.123068+0000", 2024-03-20T09:13:08.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011851857 2024-03-20T09:13:08.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.123155+0000", 2024-03-20T09:13:08.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6420000000000003e-05 2024-03-20T09:13:08.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.123424+0000", 2024-03-20T09:13:08.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000269542 2024-03-20T09:13:08.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:13:08.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.123479+0000", 2024-03-20T09:13:08.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4295000000000001e-05 2024-03-20T09:13:08.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:13:08.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.123520+0000", 2024-03-20T09:13:08.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1727000000000002e-05 2024-03-20T09:13:08.970 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.970 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:13:08.970 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.970 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:163694 3.b 3:d8316fd6:::benchmark_data_smithi122_119671_object63912:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-03-20T09:13:08.970 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:12:06.697937+0000", 2024-03-20T09:13:08.970 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.139877747, 2024-03-20T09:13:08.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.43027906700000002, 2024-03-20T09:13:08.970 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:13:08.971 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:13:08.971 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:13:08.971 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:13:08.971 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:13:08.971 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 163694 2024-03-20T09:13:08.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.971 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:13:08.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:13:08.971 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.697937+0000", 2024-03-20T09:13:08.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.971 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:13:08.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.697937+0000", 2024-03-20T09:13:08.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:13:08.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:13:08.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.697938+0000", 2024-03-20T09:13:08.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.192e-06 2024-03-20T09:13:08.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:13:08.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.697944+0000", 2024-03-20T09:13:08.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2299999999999999e-06 2024-03-20T09:13:08.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:13:08.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.697955+0000", 2024-03-20T09:13:08.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1688999999999999e-05 2024-03-20T09:13:08.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:13:08.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.697964+0000", 2024-03-20T09:13:08.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6819999999999992e-06 2024-03-20T09:13:08.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.698022+0000", 2024-03-20T09:13:08.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8158999999999997e-05 2024-03-20T09:13:08.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:13:08.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.698033+0000", 2024-03-20T09:13:08.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0851e-05 2024-03-20T09:13:08.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:13:08.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.115397+0000", 2024-03-20T09:13:08.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.41736394900000001 2024-03-20T09:13:08.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:13:08.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.115494+0000", 2024-03-20T09:13:08.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.6581000000000002e-05 2024-03-20T09:13:08.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:13:08.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.115567+0000", 2024-03-20T09:13:08.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3143000000000001e-05 2024-03-20T09:13:08.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.123568+0000", 2024-03-20T09:13:08.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080014479999999995 2024-03-20T09:13:08.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:08.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.128055+0000", 2024-03-20T09:13:08.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004487001 2024-03-20T09:13:08.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:13:08.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.128120+0000", 2024-03-20T09:13:08.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4498000000000005e-05 2024-03-20T09:13:08.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:13:08.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.128169+0000", 2024-03-20T09:13:08.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9221000000000002e-05 2024-03-20T09:13:08.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:13:08.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:13:08.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:13:08.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.128216+0000", 2024-03-20T09:13:08.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7423000000000003e-05 2024-03-20T09:13:08.977 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.977 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:13:08.977 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.977 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:13:08.977 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:13:08.977 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:13:08.977 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:13:09.006 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:09.025 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:13:09.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:09.028 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:13:09.128 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:13:09.156 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:09.174 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:13:09.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:09.178 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:13:09.279 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:13:09.308 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:09.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:09 smithi115 ceph-mon[72476]: 3.b scrub starts 2024-03-20T09:13:09.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:09 smithi115 ceph-mon[72476]: 3.b scrub ok 2024-03-20T09:13:09.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:09 smithi115 ceph-mon[72476]: 3.10 scrub starts 2024-03-20T09:13:09.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:09 smithi115 ceph-mon[72476]: 3.10 scrub ok 2024-03-20T09:13:09.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:09 smithi115 ceph-mon[72476]: 3.6 scrub starts 2024-03-20T09:13:09.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:09 smithi115 ceph-mon[72476]: 3.6 scrub ok 2024-03-20T09:13:09.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:09 smithi115 ceph-mon[72476]: pgmap v1206: 57 pgs: 1 active+undersized+remapped, 56 active+clean; 443 KiB data, 3.0 GiB used, 891 GiB / 894 GiB avail; 0 B/s, 453 objects/s recovering 2024-03-20T09:13:09.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:09 smithi115 ceph-mon[72476]: 3.19 deep-scrub starts 2024-03-20T09:13:09.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:09 smithi115 ceph-mon[72476]: 3.19 deep-scrub ok 2024-03-20T09:13:09.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:09 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2206123188' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:13:09.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:09 smithi122 ceph-mon[73495]: 3.b scrub starts 2024-03-20T09:13:09.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:09 smithi122 ceph-mon[73495]: 3.b scrub ok 2024-03-20T09:13:09.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:09 smithi122 ceph-mon[73495]: 3.10 scrub starts 2024-03-20T09:13:09.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:09 smithi122 ceph-mon[73495]: 3.10 scrub ok 2024-03-20T09:13:09.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:09 smithi122 ceph-mon[73495]: 3.6 scrub starts 2024-03-20T09:13:09.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:09 smithi122 ceph-mon[73495]: 3.6 scrub ok 2024-03-20T09:13:09.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:09 smithi122 ceph-mon[73495]: pgmap v1206: 57 pgs: 1 active+undersized+remapped, 56 active+clean; 443 KiB data, 3.0 GiB used, 891 GiB / 894 GiB avail; 0 B/s, 453 objects/s recovering 2024-03-20T09:13:09.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:09 smithi122 ceph-mon[73495]: 3.19 deep-scrub starts 2024-03-20T09:13:09.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:09 smithi122 ceph-mon[73495]: 3.19 deep-scrub ok 2024-03-20T09:13:09.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:09 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2206123188' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:13:09.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:09 smithi026 ceph-mon[68309]: 3.b scrub starts 2024-03-20T09:13:09.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:09 smithi026 ceph-mon[68309]: 3.b scrub ok 2024-03-20T09:13:09.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:09 smithi026 ceph-mon[68309]: 3.10 scrub starts 2024-03-20T09:13:09.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:09 smithi026 ceph-mon[68309]: 3.10 scrub ok 2024-03-20T09:13:09.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:09 smithi026 ceph-mon[68309]: 3.6 scrub starts 2024-03-20T09:13:09.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:09 smithi026 ceph-mon[68309]: 3.6 scrub ok 2024-03-20T09:13:09.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:09 smithi026 ceph-mon[68309]: pgmap v1206: 57 pgs: 1 active+undersized+remapped, 56 active+clean; 443 KiB data, 3.0 GiB used, 891 GiB / 894 GiB avail; 0 B/s, 453 objects/s recovering 2024-03-20T09:13:09.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:09 smithi026 ceph-mon[68309]: 3.19 deep-scrub starts 2024-03-20T09:13:09.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:09 smithi026 ceph-mon[68309]: 3.19 deep-scrub ok 2024-03-20T09:13:09.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:09 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2206123188' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:13:09.594 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-20T09:13:09.698 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:13:09.698 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:13:09.698 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:13:09.698 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:13:09.698 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:13:09.698 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:13:09.698 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:13:09.698 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:13:09.698 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:13:09.698 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:13:09.699 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:13:09.699 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:13:09.699 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:13:09.699 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:13:09.699 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:13:09.699 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:13:09.699 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:13:09.699 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:13:09.699 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:13:09.699 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:13:09.699 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:13:09.699 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:13:09.699 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:13:09.699 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:13:09.864 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:13:09.960 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:13:09.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:09.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:13:10.068 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:13:10.095 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:10.117 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:13:10.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:10.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:13:10.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:10 smithi026 ceph-mon[68309]: 3.3 deep-scrub starts 2024-03-20T09:13:10.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:10 smithi026 ceph-mon[68309]: 3.3 deep-scrub ok 2024-03-20T09:13:10.221 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:13:10.251 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:10.271 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:13:10.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:10.275 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:13:10.376 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:13:10.402 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:10 smithi115 ceph-mon[72476]: 3.3 deep-scrub starts 2024-03-20T09:13:10.402 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:10 smithi115 ceph-mon[72476]: 3.3 deep-scrub ok 2024-03-20T09:13:10.404 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:10.423 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:13:10.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:10.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:13:10.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:10 smithi122 ceph-mon[73495]: 3.3 deep-scrub starts 2024-03-20T09:13:10.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:10 smithi122 ceph-mon[73495]: 3.3 deep-scrub ok 2024-03-20T09:13:10.527 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:13:10.555 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:10.804 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:13:10.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:10.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:13:10.815 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:13:10.817 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:13:10.908 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:13:10.938 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:11.088 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:13:11.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:11.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:13:11.192 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:13:11.226 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:11.248 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:13:11.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:11.251 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:13:11.352 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:13:11.378 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:11 smithi122 ceph-mon[73495]: 1.c scrub starts 2024-03-20T09:13:11.378 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:11 smithi122 ceph-mon[73495]: 1.c scrub ok 2024-03-20T09:13:11.379 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:11 smithi122 ceph-mon[73495]: pgmap v1208: 57 pgs: 1 active+undersized+remapped, 56 active+clean; 443 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail; 0 B/s, 280 objects/s recovering 2024-03-20T09:13:11.379 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:11 smithi122 ceph-mon[73495]: 3.7 deep-scrub starts 2024-03-20T09:13:11.379 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:11 smithi122 ceph-mon[73495]: 3.7 deep-scrub ok 2024-03-20T09:13:11.379 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:11 smithi122 ceph-mon[73495]: 1.16 scrub starts 2024-03-20T09:13:11.379 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:11 smithi122 ceph-mon[73495]: 1.16 scrub ok 2024-03-20T09:13:11.379 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:11 smithi122 ceph-mon[73495]: 3.c scrub starts 2024-03-20T09:13:11.379 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:11 smithi122 ceph-mon[73495]: 3.c scrub ok 2024-03-20T09:13:11.381 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:11.400 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:13:11.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:11.404 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:13:11.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:11 smithi115 ceph-mon[72476]: 1.c scrub starts 2024-03-20T09:13:11.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:11 smithi115 ceph-mon[72476]: 1.c scrub ok 2024-03-20T09:13:11.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:11 smithi115 ceph-mon[72476]: pgmap v1208: 57 pgs: 1 active+undersized+remapped, 56 active+clean; 443 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail; 0 B/s, 280 objects/s recovering 2024-03-20T09:13:11.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:11 smithi115 ceph-mon[72476]: 3.7 deep-scrub starts 2024-03-20T09:13:11.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:11 smithi115 ceph-mon[72476]: 3.7 deep-scrub ok 2024-03-20T09:13:11.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:11 smithi115 ceph-mon[72476]: 1.16 scrub starts 2024-03-20T09:13:11.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:11 smithi115 ceph-mon[72476]: 1.16 scrub ok 2024-03-20T09:13:11.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:11 smithi115 ceph-mon[72476]: 3.c scrub starts 2024-03-20T09:13:11.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:11 smithi115 ceph-mon[72476]: 3.c scrub ok 2024-03-20T09:13:11.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:11 smithi026 ceph-mon[68309]: 1.c scrub starts 2024-03-20T09:13:11.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:11 smithi026 ceph-mon[68309]: 1.c scrub ok 2024-03-20T09:13:11.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:11 smithi026 ceph-mon[68309]: pgmap v1208: 57 pgs: 1 active+undersized+remapped, 56 active+clean; 443 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail; 0 B/s, 280 objects/s recovering 2024-03-20T09:13:11.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:11 smithi026 ceph-mon[68309]: 3.7 deep-scrub starts 2024-03-20T09:13:11.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:11 smithi026 ceph-mon[68309]: 3.7 deep-scrub ok 2024-03-20T09:13:11.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:11 smithi026 ceph-mon[68309]: 1.16 scrub starts 2024-03-20T09:13:11.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:11 smithi026 ceph-mon[68309]: 1.16 scrub ok 2024-03-20T09:13:11.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:11 smithi026 ceph-mon[68309]: 3.c scrub starts 2024-03-20T09:13:11.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:11 smithi026 ceph-mon[68309]: 3.c scrub ok 2024-03-20T09:13:11.505 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:13:11.532 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:11.684 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:13:11.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:11.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:13:11.788 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:13:11.821 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:11.962 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:13:11.963 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:13:12.277 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:13:12.398 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:12 smithi026 ceph-mon[68309]: 1.5 scrub starts 2024-03-20T09:13:12.398 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:12 smithi026 ceph-mon[68309]: 1.5 scrub ok 2024-03-20T09:13:12.398 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:12 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-03-20T09:13:12.398 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:12 smithi115 ceph-mon[72476]: 1.5 scrub starts 2024-03-20T09:13:12.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:12 smithi115 ceph-mon[72476]: 1.5 scrub ok 2024-03-20T09:13:12.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:12 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-03-20T09:13:12.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:12 smithi122 ceph-mon[73495]: 1.5 scrub starts 2024-03-20T09:13:12.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:12 smithi122 ceph-mon[73495]: 1.5 scrub ok 2024-03-20T09:13:12.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:12 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-03-20T09:13:12.472 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:13:12.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:12.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:13:12.577 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:13:12.611 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:12.632 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:13:12.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:12.635 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:13:12.736 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:13:12.765 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:12.769 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-20T09:13:12.786 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:13:12.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:12.789 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:13:12.891 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:13:12.920 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:12.939 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:13:12.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:12.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:13:13.030 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:13:13.048 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:13:13.076 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:13.169 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:13 smithi026 ceph-mon[68309]: 1.19 scrub starts 2024-03-20T09:13:13.169 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:13 smithi026 ceph-mon[68309]: 1.19 scrub ok 2024-03-20T09:13:13.169 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:13 smithi026 ceph-mon[68309]: 3.2 scrub starts 2024-03-20T09:13:13.169 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:13 smithi026 ceph-mon[68309]: 3.2 scrub ok 2024-03-20T09:13:13.169 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:13 smithi026 ceph-mon[68309]: pgmap v1209: 57 pgs: 57 active+clean; 603 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-03-20T09:13:13.170 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:13 smithi026 ceph-mon[68309]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-03-20T09:13:13.170 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:13 smithi026 ceph-mon[68309]: Cluster is now healthy 2024-03-20T09:13:13.170 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:13 smithi026 ceph-mon[68309]: 3.16 scrub starts 2024-03-20T09:13:13.170 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:13 smithi026 ceph-mon[68309]: 3.16 scrub ok 2024-03-20T09:13:13.170 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:13 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3662501858' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-20T09:13:13.276 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:13:13.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:13.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:13:13.380 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:13:13.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:13 smithi115 ceph-mon[72476]: 1.19 scrub starts 2024-03-20T09:13:13.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:13 smithi115 ceph-mon[72476]: 1.19 scrub ok 2024-03-20T09:13:13.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:13 smithi115 ceph-mon[72476]: 3.2 scrub starts 2024-03-20T09:13:13.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:13 smithi115 ceph-mon[72476]: 3.2 scrub ok 2024-03-20T09:13:13.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:13 smithi115 ceph-mon[72476]: pgmap v1209: 57 pgs: 57 active+clean; 603 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-03-20T09:13:13.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:13 smithi115 ceph-mon[72476]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-03-20T09:13:13.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:13 smithi115 ceph-mon[72476]: Cluster is now healthy 2024-03-20T09:13:13.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:13 smithi115 ceph-mon[72476]: 3.16 scrub starts 2024-03-20T09:13:13.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:13 smithi115 ceph-mon[72476]: 3.16 scrub ok 2024-03-20T09:13:13.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:13 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3662501858' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-20T09:13:13.407 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:13.433 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'unique_pool_1' removed 2024-03-20T09:13:13.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:13 smithi122 ceph-mon[73495]: 1.19 scrub starts 2024-03-20T09:13:13.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:13 smithi122 ceph-mon[73495]: 1.19 scrub ok 2024-03-20T09:13:13.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:13 smithi122 ceph-mon[73495]: 3.2 scrub starts 2024-03-20T09:13:13.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:13 smithi122 ceph-mon[73495]: 3.2 scrub ok 2024-03-20T09:13:13.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:13 smithi122 ceph-mon[73495]: pgmap v1209: 57 pgs: 57 active+clean; 603 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-03-20T09:13:13.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:13 smithi122 ceph-mon[73495]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-03-20T09:13:13.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:13 smithi122 ceph-mon[73495]: Cluster is now healthy 2024-03-20T09:13:13.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:13 smithi122 ceph-mon[73495]: 3.16 scrub starts 2024-03-20T09:13:13.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:13 smithi122 ceph-mon[73495]: 3.16 scrub ok 2024-03-20T09:13:13.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:13 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3662501858' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-20T09:13:13.856 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:13:13.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:13.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:13:13.960 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:13:13.989 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:14.009 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-20T09:13:14.010 INFO:tasks.radosbench:Beginning radosbench... 2024-03-20T09:13:14.010 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-03-20T09:13:14.011 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pool create unique_pool_2 16 2024-03-20T09:13:14.011 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:13:14.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:14.014 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:13:14.115 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:13:14.149 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:14.367 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:14 smithi026 ceph-mon[68309]: 1.6 scrub starts 2024-03-20T09:13:14.367 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:14 smithi026 ceph-mon[68309]: 1.6 scrub ok 2024-03-20T09:13:14.367 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:14 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:13:14.367 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:14 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:13:14.367 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:14 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3662501858' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-20T09:13:14.367 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:14 smithi026 ceph-mon[68309]: osdmap e512: 12 total, 12 up, 10 in 2024-03-20T09:13:14.367 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:14 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:13:14.367 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:14 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:13:14.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:14 smithi115 ceph-mon[72476]: 1.6 scrub starts 2024-03-20T09:13:14.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:14 smithi115 ceph-mon[72476]: 1.6 scrub ok 2024-03-20T09:13:14.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:14 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:13:14.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:14 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:13:14.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:14 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3662501858' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-20T09:13:14.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:14 smithi115 ceph-mon[72476]: osdmap e512: 12 total, 12 up, 10 in 2024-03-20T09:13:14.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:14 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:13:14.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:14 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:13:14.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:14 smithi122 ceph-mon[73495]: 1.6 scrub starts 2024-03-20T09:13:14.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:14 smithi122 ceph-mon[73495]: 1.6 scrub ok 2024-03-20T09:13:14.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:14 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:13:14.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:14 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:13:14.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:14 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3662501858' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-20T09:13:14.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:14 smithi122 ceph-mon[73495]: osdmap e512: 12 total, 12 up, 10 in 2024-03-20T09:13:14.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:14 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:13:14.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:14 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:13:14.700 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:13:14.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:14.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:13:14.803 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:13:14.832 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:14.943 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:13:14.943 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:13:14.955 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:13:14.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:14.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:13:15.034 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:13:15.059 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:13:15.079 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:13:15.085 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:15.104 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:13:15.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:15.107 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:13:15.148 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:13:15.208 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:13:15.213 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:15 smithi115 ceph-mon[72476]: pgmap v1211: 31 pgs: 31 active+clean; 603 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-03-20T09:13:15.213 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:15 smithi115 ceph-mon[72476]: osdmap e513: 12 total, 12 up, 10 in 2024-03-20T09:13:15.213 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:15 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/990194375' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:13:15.237 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:15.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:15 smithi122 ceph-mon[73495]: pgmap v1211: 31 pgs: 31 active+clean; 603 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-03-20T09:13:15.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:15 smithi122 ceph-mon[73495]: osdmap e513: 12 total, 12 up, 10 in 2024-03-20T09:13:15.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:15 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/990194375' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:13:15.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:15 smithi026 ceph-mon[68309]: pgmap v1211: 31 pgs: 31 active+clean; 603 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-03-20T09:13:15.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:15 smithi026 ceph-mon[68309]: osdmap e513: 12 total, 12 up, 10 in 2024-03-20T09:13:15.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:15 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/990194375' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:13:15.577 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-20T09:13:15.585 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:13:15.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:15.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:13:15.688 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:13:15.716 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:15.735 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:13:15.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:15.738 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:13:15.839 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:13:15.843 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:13:15.867 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:16.066 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:13:16.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:16.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:13:16.153 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.11 to 1 (10000) 2024-03-20T09:13:16.169 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:13:16.198 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:16.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:16 smithi115 ceph-mon[72476]: 1.4 scrub starts 2024-03-20T09:13:16.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:16 smithi115 ceph-mon[72476]: 1.4 scrub ok 2024-03-20T09:13:16.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:16 smithi115 ceph-mon[72476]: 1.1 scrub starts 2024-03-20T09:13:16.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:16 smithi115 ceph-mon[72476]: 1.1 scrub ok 2024-03-20T09:13:16.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:16 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:13:16.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:16 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/990194375' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:13:16.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:16 smithi115 ceph-mon[72476]: osdmap e514: 12 total, 12 up, 10 in 2024-03-20T09:13:16.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:16 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3998953466' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-03-20T09:13:16.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:16 smithi122 ceph-mon[73495]: 1.4 scrub starts 2024-03-20T09:13:16.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:16 smithi122 ceph-mon[73495]: 1.4 scrub ok 2024-03-20T09:13:16.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:16 smithi122 ceph-mon[73495]: 1.1 scrub starts 2024-03-20T09:13:16.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:16 smithi122 ceph-mon[73495]: 1.1 scrub ok 2024-03-20T09:13:16.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:16 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:13:16.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:16 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/990194375' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:13:16.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:16 smithi122 ceph-mon[73495]: osdmap e514: 12 total, 12 up, 10 in 2024-03-20T09:13:16.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:16 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3998953466' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-03-20T09:13:16.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:16 smithi026 ceph-mon[68309]: 1.4 scrub starts 2024-03-20T09:13:16.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:16 smithi026 ceph-mon[68309]: 1.4 scrub ok 2024-03-20T09:13:16.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:16 smithi026 ceph-mon[68309]: 1.1 scrub starts 2024-03-20T09:13:16.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:16 smithi026 ceph-mon[68309]: 1.1 scrub ok 2024-03-20T09:13:16.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:16 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:13:16.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:16 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/990194375' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:13:16.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:16 smithi026 ceph-mon[68309]: osdmap e514: 12 total, 12 up, 10 in 2024-03-20T09:13:16.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:16 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3998953466' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-03-20T09:13:17.047 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:13:17.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:17.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:13:17.152 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:13:17.179 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:17.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:17 smithi115 ceph-mon[72476]: pgmap v1214: 31 pgs: 31 active+clean; 603 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-03-20T09:13:17.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:17 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:13:17.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:17 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:13:17.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:17 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3998953466' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-03-20T09:13:17.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:17 smithi115 ceph-mon[72476]: osdmap e515: 12 total, 12 up, 10 in 2024-03-20T09:13:17.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:17 smithi115 ceph-mon[72476]: osdmap e516: 12 total, 12 up, 10 in 2024-03-20T09:13:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:17 smithi122 ceph-mon[73495]: pgmap v1214: 31 pgs: 31 active+clean; 603 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-03-20T09:13:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:17 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:13:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:17 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:13:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:17 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3998953466' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-03-20T09:13:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:17 smithi122 ceph-mon[73495]: osdmap e515: 12 total, 12 up, 10 in 2024-03-20T09:13:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:17 smithi122 ceph-mon[73495]: osdmap e516: 12 total, 12 up, 10 in 2024-03-20T09:13:17.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:17 smithi026 ceph-mon[68309]: pgmap v1214: 31 pgs: 31 active+clean; 603 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-03-20T09:13:17.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:17 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:13:17.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:17 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:13:17.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:17 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3998953466' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-03-20T09:13:17.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:17 smithi026 ceph-mon[68309]: osdmap e515: 12 total, 12 up, 10 in 2024-03-20T09:13:17.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:17 smithi026 ceph-mon[68309]: osdmap e516: 12 total, 12 up, 10 in 2024-03-20T09:13:17.629 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:13:17.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:17.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:13:17.688 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 3 1 2024-03-20T09:13:17.732 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:13:17.765 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:17.786 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:13:17.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:17.789 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:13:17.861 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:13:17.861 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:13:17.890 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:13:17.918 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:18.116 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:13:18.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:18.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:13:18.221 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:13:18.251 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:18.349 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:13:18.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:18.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:13:18.453 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:13:18.485 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:18.506 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:13:18.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:18.509 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:13:18.610 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:13:18.622 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:13:18.624 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:13:18.641 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:18.650 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-20T09:13:18.658 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:13:18.662 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:13:18.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:18.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:13:18.678 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:13:18.697 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:13:18.718 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:13:18.739 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:13:18.759 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:13:18.765 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:13:18.782 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:13:18.797 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:18.805 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:13:18.816 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:13:18.827 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:13:18.848 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:13:18.869 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:13:18.870 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:13:18.870 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:13:18.870 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:13:18.870 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:13:18.870 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:13:18.870 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:13:18.871 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:13:18.871 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:13:18.871 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:13:18.871 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:13:18.871 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:13:18.871 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:13:18.923 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:13:19.364 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:19 smithi026 ceph-mon[68309]: pgmap v1217: 31 pgs: 31 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-03-20T09:13:19.364 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:19 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:13:19.364 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:19 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:13:19.364 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:19 smithi026 ceph-mon[68309]: osdmap e517: 12 total, 12 up, 10 in 2024-03-20T09:13:19.365 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:19 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3657138330' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:13:19.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:19 smithi115 ceph-mon[72476]: pgmap v1217: 31 pgs: 31 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-03-20T09:13:19.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:19 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:13:19.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:19 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:13:19.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:19 smithi115 ceph-mon[72476]: osdmap e517: 12 total, 12 up, 10 in 2024-03-20T09:13:19.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:19 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3657138330' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:13:19.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:19 smithi122 ceph-mon[73495]: pgmap v1217: 31 pgs: 31 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-03-20T09:13:19.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:19 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:13:19.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:19 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:13:19.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:19 smithi122 ceph-mon[73495]: osdmap e517: 12 total, 12 up, 10 in 2024-03-20T09:13:19.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:19 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3657138330' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:13:19.547 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:13:19.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:19.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:13:19.652 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:13:19.681 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:19.701 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:13:19.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:19.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:13:19.806 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:13:19.838 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:19.860 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:13:19.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:19.863 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:13:19.964 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:13:19.994 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:20.121 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:13:20.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:20.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:13:20.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:20 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:13:20.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:20 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:13:20.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:20 smithi122 ceph-mon[73495]: osdmap e518: 12 total, 12 up, 10 in 2024-03-20T09:13:20.225 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:13:20.252 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:20 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:13:20.252 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:20 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:13:20.253 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:20 smithi026 ceph-mon[68309]: osdmap e518: 12 total, 12 up, 10 in 2024-03-20T09:13:20.255 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:20.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:20 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:13:20.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:20 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:13:20.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:20 smithi115 ceph-mon[72476]: osdmap e518: 12 total, 12 up, 10 in 2024-03-20T09:13:20.455 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:13:20.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:20.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:13:20.559 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:13:20.590 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:20.682 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:13:20.682 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:13:20.682 INFO:teuthology.orchestra.run.smithi115.stdout: "num_ops": 0 2024-03-20T09:13:20.682 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:13:20.790 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:13:20.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:20.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:13:20.894 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:13:20.923 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:21.077 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'unique_pool_2' created 2024-03-20T09:13:21.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:21 smithi026 ceph-mon[68309]: pgmap v1220: 31 pgs: 2 peering, 29 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-03-20T09:13:21.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:21 smithi026 ceph-mon[68309]: osdmap e519: 12 total, 12 up, 10 in 2024-03-20T09:13:21.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:21 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3517876861' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-20T09:13:21.273 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:13:21.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:21.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:13:21.378 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:13:21.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:21 smithi115 ceph-mon[72476]: pgmap v1220: 31 pgs: 2 peering, 29 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-03-20T09:13:21.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:21 smithi115 ceph-mon[72476]: osdmap e519: 12 total, 12 up, 10 in 2024-03-20T09:13:21.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:21 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3517876861' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-20T09:13:21.407 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:21.443 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-20T09:13:21.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:21 smithi122 ceph-mon[73495]: pgmap v1220: 31 pgs: 2 peering, 29 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-03-20T09:13:21.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:21 smithi122 ceph-mon[73495]: osdmap e519: 12 total, 12 up, 10 in 2024-03-20T09:13:21.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:21 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3517876861' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-20T09:13:21.506 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:13:21.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:21.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:13:21.611 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:13:21.640 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:21.660 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:13:21.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:21.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:13:21.709 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:13:21.765 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:13:21.798 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:21.820 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:13:21.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:21.823 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:13:21.925 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:13:21.928 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:13:21.937 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-20T09:13:21.936+0000 7f5883fff700 0 --2- 172.21.15.26:0/141653454 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f5884101a30 0x7f5884198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-03-20T09:13:21.955 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:21.980 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:13:21.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:21.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:13:22.084 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:13:22.112 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:22.248 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:22 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3517876861' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-20T09:13:22.248 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:22 smithi115 ceph-mon[72476]: osdmap e520: 12 total, 12 up, 10 in 2024-03-20T09:13:22.248 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:22 smithi115 ceph-mon[72476]: osdmap e521: 12 total, 12 up, 10 in 2024-03-20T09:13:22.348 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:22 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3517876861' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-20T09:13:22.348 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:22 smithi026 ceph-mon[68309]: osdmap e520: 12 total, 12 up, 10 in 2024-03-20T09:13:22.349 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:22 smithi026 ceph-mon[68309]: osdmap e521: 12 total, 12 up, 10 in 2024-03-20T09:13:22.372 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-03-20T09:13:22.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:22 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3517876861' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-20T09:13:22.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:22 smithi122 ceph-mon[73495]: osdmap e520: 12 total, 12 up, 10 in 2024-03-20T09:13:22.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:22 smithi122 ceph-mon[73495]: osdmap e521: 12 total, 12 up, 10 in 2024-03-20T09:13:22.964 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:13:22.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:22.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:13:23.064 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.3 to 1 (10000) 2024-03-20T09:13:23.069 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:13:23.100 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:23.295 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:13:23.303 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:13:23.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:23.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:13:23.323 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:13:23.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:23 smithi115 ceph-mon[72476]: pgmap v1223: 47 pgs: 3 creating+peering, 13 unknown, 2 peering, 29 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-03-20T09:13:23.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:23 smithi115 ceph-mon[72476]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-20T09:13:23.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:23 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/141653454' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-20T09:13:23.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:23 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/141653454' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-20T09:13:23.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:23 smithi115 ceph-mon[72476]: osdmap e522: 12 total, 12 up, 10 in 2024-03-20T09:13:23.408 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:13:23.409 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:23 smithi026 ceph-mon[68309]: pgmap v1223: 47 pgs: 3 creating+peering, 13 unknown, 2 peering, 29 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-03-20T09:13:23.409 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:23 smithi026 ceph-mon[68309]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-20T09:13:23.409 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:23 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/141653454' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-20T09:13:23.409 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:23 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/141653454' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-20T09:13:23.409 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:23 smithi026 ceph-mon[68309]: osdmap e522: 12 total, 12 up, 10 in 2024-03-20T09:13:23.442 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:23.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:23 smithi122 ceph-mon[73495]: pgmap v1223: 47 pgs: 3 creating+peering, 13 unknown, 2 peering, 29 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-03-20T09:13:23.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:23 smithi122 ceph-mon[73495]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-20T09:13:23.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:23 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/141653454' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-20T09:13:23.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:23 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/141653454' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-20T09:13:23.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:23 smithi122 ceph-mon[73495]: osdmap e522: 12 total, 12 up, 10 in 2024-03-20T09:13:23.517 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:13:23.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:23.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:13:23.622 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:13:23.649 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:23.826 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:13:23.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:23.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:13:23.930 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:13:23.958 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:24.113 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:13:24.114 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:13:24.114 INFO:teuthology.orchestra.run.smithi115.stdout: "complaint_time": 30, 2024-03-20T09:13:24.114 INFO:teuthology.orchestra.run.smithi115.stdout: "num_blocked_ops": 0 2024-03-20T09:13:24.114 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:13:24.489 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-20T09:13:24.490 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:13:24.808 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:13:24.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:24.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:13:24.829 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-20T09:13:24.913 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:13:24.941 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:24.946 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:13:24.960 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:13:24.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:24.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:13:25.064 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:13:25.065 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:13:25.095 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:25.115 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:13:25.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:25.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:13:25.142 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:13:25.220 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:13:25.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:25 smithi026 ceph-mon[68309]: pgmap v1226: 47 pgs: 1 creating+activating, 3 creating+peering, 12 unknown, 2 peering, 29 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-03-20T09:13:25.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:25 smithi026 ceph-mon[68309]: osdmap e523: 12 total, 12 up, 10 in 2024-03-20T09:13:25.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:25 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/681809622' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:13:25.251 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:25.355 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:25 smithi115 ceph-mon[72476]: pgmap v1226: 47 pgs: 1 creating+activating, 3 creating+peering, 12 unknown, 2 peering, 29 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-03-20T09:13:25.355 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:25 smithi115 ceph-mon[72476]: osdmap e523: 12 total, 12 up, 10 in 2024-03-20T09:13:25.355 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:25 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/681809622' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:13:25.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:25 smithi122 ceph-mon[73495]: pgmap v1226: 47 pgs: 1 creating+activating, 3 creating+peering, 12 unknown, 2 peering, 29 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-03-20T09:13:25.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:25 smithi122 ceph-mon[73495]: osdmap e523: 12 total, 12 up, 10 in 2024-03-20T09:13:25.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:25 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/681809622' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:13:26.072 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:26 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:13:26.072 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:26 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/681809622' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:13:26.302 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:13:26.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:26.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:13:26.346 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:26 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:13:26.346 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:26 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/681809622' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:13:26.346 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:26 smithi115 ceph-mon[72476]: osdmap e524: 12 total, 12 up, 10 in 2024-03-20T09:13:26.409 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:13:26.436 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:26.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:26 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:13:26.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:26 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/681809622' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:13:26.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:26 smithi122 ceph-mon[73495]: osdmap e524: 12 total, 12 up, 10 in 2024-03-20T09:13:26.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:26 smithi026 ceph-mon[68309]: osdmap e524: 12 total, 12 up, 10 in 2024-03-20T09:13:26.634 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:13:26.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:26.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:13:26.739 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:13:26.772 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:26.793 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:13:26.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:26.796 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:13:26.897 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:13:26.901 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:13:26.901 INFO:teuthology.orchestra.run.smithi115.stdout: "size": 20, 2024-03-20T09:13:26.901 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 600, 2024-03-20T09:13:26.901 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [ 2024-03-20T09:13:26.901 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.901 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "MOSDPGPull(3.15 e279/239 cost 1049576)", 2024-03-20T09:13:26.901 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:04:40.924123+0000", 2024-03-20T09:13:26.901 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 525.97350020600004, 2024-03-20T09:13:26.901 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 37.917802715999997, 2024-03-20T09:13:26.901 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:13:26.901 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:13:26.901 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:13:26.902 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.902 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:13:26.902 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:40.924123+0000", 2024-03-20T09:13:26.902 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:26.902 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.902 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.902 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:13:26.902 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:40.924123+0000", 2024-03-20T09:13:26.902 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:26.902 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.903 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.903 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:13:26.903 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:40.924124+0000", 2024-03-20T09:13:26.903 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.5199999999999995e-07 2024-03-20T09:13:26.903 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.903 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.903 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:13:26.903 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:40.924134+0000", 2024-03-20T09:13:26.903 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0125999999999999e-05 2024-03-20T09:13:26.903 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.903 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.903 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:13:26.903 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:40.924142+0000", 2024-03-20T09:13:26.904 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.6459999999999993e-06 2024-03-20T09:13:26.904 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.904 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.904 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:13:26.904 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:40.924147+0000", 2024-03-20T09:13:26.904 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0760000000000002e-06 2024-03-20T09:13:26.904 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.904 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.904 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.904 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:18.841463+0000", 2024-03-20T09:13:26.904 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 37.917316503000002 2024-03-20T09:13:26.904 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.904 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.905 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:13:26.905 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:18.841926+0000", 2024-03-20T09:13:26.905 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00046241300000000001 2024-03-20T09:13:26.905 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:26.905 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:13:26.905 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:26.905 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.905 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.905 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "MOSDPGPull(3.15 e280/239 cost 1049576)", 2024-03-20T09:13:26.905 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:04:41.964353+0000", 2024-03-20T09:13:26.905 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 524.93327036699998, 2024-03-20T09:13:26.905 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 36.877965906, 2024-03-20T09:13:26.905 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:13:26.905 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:13:26.906 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:13:26.906 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.906 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:13:26.906 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:41.964353+0000", 2024-03-20T09:13:26.906 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:26.906 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.906 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.906 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:13:26.906 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:41.964353+0000", 2024-03-20T09:13:26.906 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:26.906 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.906 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.906 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:13:26.907 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:41.964353+0000", 2024-03-20T09:13:26.907 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.3799999999999996e-07 2024-03-20T09:13:26.907 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.907 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.907 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:13:26.907 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:41.964365+0000", 2024-03-20T09:13:26.907 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1314e-05 2024-03-20T09:13:26.907 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.907 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.907 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:13:26.907 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:41.964374+0000", 2024-03-20T09:13:26.907 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.8799999999999997e-06 2024-03-20T09:13:26.907 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.908 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.908 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:13:26.908 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:41.964381+0000", 2024-03-20T09:13:26.908 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.9040000000000003e-06 2024-03-20T09:13:26.908 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.908 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.908 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.908 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:18.841946+0000", 2024-03-20T09:13:26.908 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 36.877565808 2024-03-20T09:13:26.908 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.908 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.908 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:13:26.908 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:18.842319+0000", 2024-03-20T09:13:26.908 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00037216199999999999 2024-03-20T09:13:26.909 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:26.909 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:13:26.909 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:26.909 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.909 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.909 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "MOSDPGPull(3.15 e281/239 cost 1049576)", 2024-03-20T09:13:26.909 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:04:44.130689+0000", 2024-03-20T09:13:26.909 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 522.76693421699997, 2024-03-20T09:13:26.909 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 34.712016781000003, 2024-03-20T09:13:26.909 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:13:26.909 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:13:26.909 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:13:26.909 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.910 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:13:26.910 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:44.130689+0000", 2024-03-20T09:13:26.910 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:26.910 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.910 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.910 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:13:26.910 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:44.130689+0000", 2024-03-20T09:13:26.910 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:26.910 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.910 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.910 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:13:26.910 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:44.130690+0000", 2024-03-20T09:13:26.910 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1119999999999999e-06 2024-03-20T09:13:26.911 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.911 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.911 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:13:26.911 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:44.130703+0000", 2024-03-20T09:13:26.911 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3066e-05 2024-03-20T09:13:26.911 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.911 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.911 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:13:26.911 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:44.130714+0000", 2024-03-20T09:13:26.911 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.136e-05 2024-03-20T09:13:26.911 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.911 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.911 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:13:26.912 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:44.130722+0000", 2024-03-20T09:13:26.912 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3499999999999999e-06 2024-03-20T09:13:26.912 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.912 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.912 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.912 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:18.842335+0000", 2024-03-20T09:13:26.912 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 34.711612926000001 2024-03-20T09:13:26.912 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.912 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.912 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:13:26.912 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:18.842706+0000", 2024-03-20T09:13:26.912 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00037096699999999999 2024-03-20T09:13:26.912 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:26.912 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:13:26.913 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:26.913 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.913 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.913 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "MOSDPGPull(3.15 e282/239 cost 1049576)", 2024-03-20T09:13:26.913 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:04:44.978300+0000", 2024-03-20T09:13:26.913 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 521.91932296599998, 2024-03-20T09:13:26.913 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 33.864778774999998, 2024-03-20T09:13:26.913 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:13:26.913 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:13:26.913 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:13:26.913 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.913 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:13:26.913 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:44.978300+0000", 2024-03-20T09:13:26.913 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:26.914 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.914 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.914 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:13:26.914 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:44.978300+0000", 2024-03-20T09:13:26.914 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:26.914 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.914 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.914 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:13:26.914 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:44.978301+0000", 2024-03-20T09:13:26.914 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.5999999999999991e-07 2024-03-20T09:13:26.914 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.914 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.914 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:13:26.914 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:44.978323+0000", 2024-03-20T09:13:26.915 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1520999999999999e-05 2024-03-20T09:13:26.915 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.915 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.915 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:13:26.915 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:44.978334+0000", 2024-03-20T09:13:26.915 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1124000000000001e-05 2024-03-20T09:13:26.915 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.915 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.915 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:13:26.915 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:44.978340+0000", 2024-03-20T09:13:26.915 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.5780000000000004e-06 2024-03-20T09:13:26.915 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.915 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.915 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.916 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:18.842720+0000", 2024-03-20T09:13:26.916 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 33.864379855000003 2024-03-20T09:13:26.916 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.916 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.916 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:13:26.916 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:18.843079+0000", 2024-03-20T09:13:26.916 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00035873699999999998 2024-03-20T09:13:26.916 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:26.916 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:13:26.916 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:26.916 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.916 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.916 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "MOSDPGPull(3.15 e283/239 cost 1049576)", 2024-03-20T09:13:26.916 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:04:45.947857+0000", 2024-03-20T09:13:26.917 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 520.94976581399999, 2024-03-20T09:13:26.917 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 32.895625455000001, 2024-03-20T09:13:26.917 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:13:26.917 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:13:26.917 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:13:26.917 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.917 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:13:26.917 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:45.947857+0000", 2024-03-20T09:13:26.917 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:26.917 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.917 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.917 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:13:26.917 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:45.947857+0000", 2024-03-20T09:13:26.918 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:26.918 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.918 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.918 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:13:26.918 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:45.947858+0000", 2024-03-20T09:13:26.918 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.8400000000000002e-07 2024-03-20T09:13:26.918 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.918 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.918 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:13:26.918 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:45.947871+0000", 2024-03-20T09:13:26.918 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2337000000000001e-05 2024-03-20T09:13:26.918 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.918 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.919 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:13:26.919 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:45.947883+0000", 2024-03-20T09:13:26.919 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2334000000000001e-05 2024-03-20T09:13:26.919 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.919 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.919 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:13:26.919 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:04:45.947890+0000", 2024-03-20T09:13:26.919 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.8059999999999997e-06 2024-03-20T09:13:26.919 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.919 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.919 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.919 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:18.843094+0000", 2024-03-20T09:13:26.919 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 32.895204374999999 2024-03-20T09:13:26.919 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.919 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.920 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:13:26.920 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:18.843483+0000", 2024-03-20T09:13:26.920 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00038861899999999999 2024-03-20T09:13:26.920 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:26.920 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:13:26.920 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:26.920 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.920 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.920 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:110609 3.d 3:bf8f3962:::benchmark_data_smithi122_119671_object10827:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-03-20T09:13:26.920 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:29.784111+0000", 2024-03-20T09:13:26.920 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 477.11351189300001, 2024-03-20T09:13:26.920 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.636143729, 2024-03-20T09:13:26.920 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:13:26.920 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:13:26.921 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:13:26.921 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:13:26.921 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:13:26.921 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 110609 2024-03-20T09:13:26.921 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.921 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:13:26.921 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.921 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:13:26.921 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:29.784111+0000", 2024-03-20T09:13:26.921 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:26.921 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.921 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.921 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:13:26.921 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:29.784111+0000", 2024-03-20T09:13:26.922 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:26.922 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.922 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.922 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:13:26.922 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:29.784113+0000", 2024-03-20T09:13:26.922 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7749999999999999e-06 2024-03-20T09:13:26.922 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.922 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.922 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:13:26.922 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:29.784121+0000", 2024-03-20T09:13:26.922 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7030000000000006e-06 2024-03-20T09:13:26.922 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.922 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.923 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:13:26.923 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:29.784139+0000", 2024-03-20T09:13:26.923 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8692e-05 2024-03-20T09:13:26.923 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.923 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.923 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:13:26.923 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:29.784151+0000", 2024-03-20T09:13:26.923 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2004000000000001e-05 2024-03-20T09:13:26.923 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.923 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.923 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.923 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:29.991217+0000", 2024-03-20T09:13:26.923 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.20706619400000001 2024-03-20T09:13:26.923 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.924 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.924 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:13:26.924 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:30.008282+0000", 2024-03-20T09:13:26.924 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.017064032999999999 2024-03-20T09:13:26.924 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.924 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.924 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.924 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:30.079328+0000", 2024-03-20T09:13:26.924 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.071046434000000006 2024-03-20T09:13:26.924 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.924 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.924 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:13:26.924 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:30.088127+0000", 2024-03-20T09:13:26.924 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0087991370000000003 2024-03-20T09:13:26.925 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.925 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.925 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.925 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:32.276233+0000", 2024-03-20T09:13:26.925 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1881055150000002 2024-03-20T09:13:26.925 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.925 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.925 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:13:26.925 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:32.284264+0000", 2024-03-20T09:13:26.925 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0080314740000000003 2024-03-20T09:13:26.925 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.925 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.925 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:13:26.925 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:32.297871+0000", 2024-03-20T09:13:26.926 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.013607068999999999 2024-03-20T09:13:26.926 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.926 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.926 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:13:26.926 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:32.381709+0000", 2024-03-20T09:13:26.926 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.083837562000000004 2024-03-20T09:13:26.926 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.926 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.926 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:26.926 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:32.383916+0000", 2024-03-20T09:13:26.926 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.002207437 2024-03-20T09:13:26.926 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.926 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.926 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:26.927 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:32.388692+0000", 2024-03-20T09:13:26.927 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0047761929999999998 2024-03-20T09:13:26.927 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.927 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.927 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:13:26.927 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:32.395940+0000", 2024-03-20T09:13:26.927 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0072478589999999997 2024-03-20T09:13:26.927 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.927 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.927 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:13:26.927 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:32.420255+0000", 2024-03-20T09:13:26.927 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.024314648000000001 2024-03-20T09:13:26.927 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:26.927 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:13:26.928 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:26.928 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.928 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.928 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:110617 3.d 3:b1db244d:::benchmark_data_smithi122_119671_object10835:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-03-20T09:13:26.928 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:29.966376+0000", 2024-03-20T09:13:26.928 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 476.93124732400003, 2024-03-20T09:13:26.928 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6789749970000001, 2024-03-20T09:13:26.928 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:13:26.928 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:13:26.928 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:13:26.928 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:13:26.928 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:13:26.928 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 110617 2024-03-20T09:13:26.929 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.929 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:13:26.929 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.929 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:13:26.929 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:29.966376+0000", 2024-03-20T09:13:26.929 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:26.929 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.929 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.929 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:13:26.929 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:29.966376+0000", 2024-03-20T09:13:26.929 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:26.929 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.929 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.929 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:13:26.930 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:29.966377+0000", 2024-03-20T09:13:26.930 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1459999999999999e-06 2024-03-20T09:13:26.930 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.930 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.930 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:13:26.930 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:29.966382+0000", 2024-03-20T09:13:26.930 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9150000000000004e-06 2024-03-20T09:13:26.930 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.930 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.930 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:13:26.930 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:29.966393+0000", 2024-03-20T09:13:26.930 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1554e-05 2024-03-20T09:13:26.930 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.931 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.931 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:13:26.931 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:29.966400+0000", 2024-03-20T09:13:26.931 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.9909999999999996e-06 2024-03-20T09:13:26.931 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.931 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.931 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.931 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:30.024450+0000", 2024-03-20T09:13:26.931 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.058050079999999997 2024-03-20T09:13:26.931 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.931 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.931 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:13:26.931 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:30.035590+0000", 2024-03-20T09:13:26.931 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.011139534 2024-03-20T09:13:26.932 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.932 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.932 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.932 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:30.163835+0000", 2024-03-20T09:13:26.932 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.12824516499999999 2024-03-20T09:13:26.932 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.932 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.932 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:13:26.932 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:30.171879+0000", 2024-03-20T09:13:26.932 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0080434619999999995 2024-03-20T09:13:26.932 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.932 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.932 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.933 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:32.500944+0000", 2024-03-20T09:13:26.933 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3290656919999999 2024-03-20T09:13:26.933 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.933 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.933 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:13:26.933 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:32.510540+0000", 2024-03-20T09:13:26.933 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0095956909999999999 2024-03-20T09:13:26.933 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.933 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.933 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:13:26.933 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:32.524020+0000", 2024-03-20T09:13:26.933 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.013480371 2024-03-20T09:13:26.933 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.933 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.934 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:13:26.934 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:32.602455+0000", 2024-03-20T09:13:26.934 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.07843493 2024-03-20T09:13:26.934 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.934 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.934 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:26.934 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:32.616355+0000", 2024-03-20T09:13:26.934 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.013899570999999999 2024-03-20T09:13:26.934 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.934 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.934 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:26.934 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:32.620121+0000", 2024-03-20T09:13:26.934 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0037660480000000001 2024-03-20T09:13:26.934 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.935 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.935 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:13:26.935 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:32.624730+0000", 2024-03-20T09:13:26.935 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0046090740000000003 2024-03-20T09:13:26.935 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.935 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.935 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:13:26.935 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:32.645351+0000", 2024-03-20T09:13:26.935 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.020620772999999998 2024-03-20T09:13:26.935 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:26.935 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:13:26.935 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:26.935 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.935 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.936 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:110802 3.d 3:b6d849bc:::benchmark_data_smithi122_119671_object11020:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-03-20T09:13:26.936 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:31.056672+0000", 2024-03-20T09:13:26.936 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 475.840950808, 2024-03-20T09:13:26.936 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8296267200000003, 2024-03-20T09:13:26.936 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:13:26.936 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:13:26.936 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:13:26.936 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:13:26.936 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:13:26.936 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 110802 2024-03-20T09:13:26.936 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.936 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:13:26.936 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.936 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:13:26.937 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:31.056672+0000", 2024-03-20T09:13:26.937 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:26.937 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.937 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.937 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:13:26.937 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:31.056672+0000", 2024-03-20T09:13:26.937 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:26.937 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.937 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.937 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:13:26.937 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:31.056674+0000", 2024-03-20T09:13:26.937 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8160000000000001e-06 2024-03-20T09:13:26.937 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.937 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.938 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:13:26.938 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:31.056680+0000", 2024-03-20T09:13:26.938 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1890000000000004e-06 2024-03-20T09:13:26.938 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.938 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.938 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:13:26.938 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:31.056695+0000", 2024-03-20T09:13:26.938 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4654000000000001e-05 2024-03-20T09:13:26.938 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.938 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.938 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:13:26.938 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:31.056704+0000", 2024-03-20T09:13:26.938 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.4360000000000008e-06 2024-03-20T09:13:26.938 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.939 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.939 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.939 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:31.277682+0000", 2024-03-20T09:13:26.939 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.22097818999999999 2024-03-20T09:13:26.939 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.939 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.939 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:13:26.939 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:31.287023+0000", 2024-03-20T09:13:26.939 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0093402250000000006 2024-03-20T09:13:26.939 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.939 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.939 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.939 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:31.422108+0000", 2024-03-20T09:13:26.939 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.135085177 2024-03-20T09:13:26.940 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.940 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.940 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:13:26.940 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:31.436656+0000", 2024-03-20T09:13:26.940 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.014548346 2024-03-20T09:13:26.940 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.940 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.940 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.940 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:33.701744+0000", 2024-03-20T09:13:26.940 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2650876430000002 2024-03-20T09:13:26.940 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.940 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.940 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:13:26.940 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:33.712981+0000", 2024-03-20T09:13:26.941 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.011237294 2024-03-20T09:13:26.941 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.941 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.941 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:13:26.941 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:33.728702+0000", 2024-03-20T09:13:26.941 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.015720359 2024-03-20T09:13:26.941 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.941 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.941 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:13:26.941 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:33.855306+0000", 2024-03-20T09:13:26.941 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.12660460500000001 2024-03-20T09:13:26.941 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.941 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.941 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:26.942 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:33.857732+0000", 2024-03-20T09:13:26.942 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0024262839999999999 2024-03-20T09:13:26.942 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.942 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.942 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:26.942 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:33.861286+0000", 2024-03-20T09:13:26.942 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.003553189 2024-03-20T09:13:26.942 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.942 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.942 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:13:26.942 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:33.866333+0000", 2024-03-20T09:13:26.942 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0050470970000000004 2024-03-20T09:13:26.942 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.942 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.943 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:13:26.943 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:33.886299+0000", 2024-03-20T09:13:26.943 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.019966215999999998 2024-03-20T09:13:26.943 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:26.943 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:13:26.943 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:26.943 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.943 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.943 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:111779 3.d 3:b1310f92:::benchmark_data_smithi122_119671_object11997:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-03-20T09:13:26.943 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:40.895193+0000", 2024-03-20T09:13:26.943 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 466.002429904, 2024-03-20T09:13:26.943 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.64063267, 2024-03-20T09:13:26.943 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:13:26.944 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:13:26.944 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:13:26.944 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:13:26.944 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:13:26.944 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 111779 2024-03-20T09:13:26.944 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.944 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:13:26.944 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.944 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:13:26.944 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:40.895193+0000", 2024-03-20T09:13:26.944 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:26.944 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.944 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.945 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:13:26.945 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:40.895193+0000", 2024-03-20T09:13:26.945 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:26.945 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.945 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.945 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:13:26.945 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:40.895195+0000", 2024-03-20T09:13:26.945 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.677e-06 2024-03-20T09:13:26.945 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.945 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.945 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:13:26.945 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:40.895203+0000", 2024-03-20T09:13:26.945 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.3680000000000002e-06 2024-03-20T09:13:26.946 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.946 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.946 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:13:26.946 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:40.895218+0000", 2024-03-20T09:13:26.946 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5301000000000001e-05 2024-03-20T09:13:26.946 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.946 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.946 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:13:26.946 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:40.895227+0000", 2024-03-20T09:13:26.946 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.7159999999999992e-06 2024-03-20T09:13:26.946 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.946 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.947 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.947 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:40.975067+0000", 2024-03-20T09:13:26.947 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.079840016999999999 2024-03-20T09:13:26.947 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.947 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.947 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:13:26.947 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:40.983614+0000", 2024-03-20T09:13:26.947 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0085472489999999998 2024-03-20T09:13:26.947 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.947 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.947 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.947 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.051468+0000", 2024-03-20T09:13:26.947 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.067853940000000001 2024-03-20T09:13:26.948 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.948 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.948 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:13:26.948 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.064377+0000", 2024-03-20T09:13:26.948 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.012908309 2024-03-20T09:13:26.948 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.948 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.948 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.948 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:43.535030+0000", 2024-03-20T09:13:26.948 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.4706534549999999 2024-03-20T09:13:26.948 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.948 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.948 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:13:26.949 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:43.535826+0000", 2024-03-20T09:13:26.949 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00079563799999999999 2024-03-20T09:13:26.949 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:26.949 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:13:26.949 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:26.949 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.949 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.949 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:111814 3.d 3:b8b10f2b:::benchmark_data_smithi122_119671_object12032:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-03-20T09:13:26.949 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:41.237764+0000", 2024-03-20T09:13:26.949 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 465.65985920499998, 2024-03-20T09:13:26.949 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6506554740000001, 2024-03-20T09:13:26.949 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:13:26.949 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:13:26.949 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:13:26.950 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:13:26.950 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:13:26.950 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 111814 2024-03-20T09:13:26.950 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.950 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:13:26.950 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.950 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:13:26.950 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.237764+0000", 2024-03-20T09:13:26.950 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:26.950 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.950 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.950 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:13:26.950 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.237764+0000", 2024-03-20T09:13:26.951 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:26.951 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.951 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.951 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:13:26.951 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.237765+0000", 2024-03-20T09:13:26.951 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5600000000000001e-06 2024-03-20T09:13:26.951 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.951 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.951 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:13:26.951 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.237773+0000", 2024-03-20T09:13:26.951 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3270000000000003e-06 2024-03-20T09:13:26.951 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.951 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.951 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:13:26.951 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.237785+0000", 2024-03-20T09:13:26.952 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2788e-05 2024-03-20T09:13:26.952 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.952 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.952 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:13:26.952 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.237794+0000", 2024-03-20T09:13:26.952 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5280000000000003e-06 2024-03-20T09:13:26.952 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.952 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.952 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.952 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.274715+0000", 2024-03-20T09:13:26.952 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.036921484999999997 2024-03-20T09:13:26.952 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.952 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.952 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:13:26.952 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.284730+0000", 2024-03-20T09:13:26.953 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.010014749999999999 2024-03-20T09:13:26.953 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.953 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.953 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.953 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.431400+0000", 2024-03-20T09:13:26.953 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.14666947899999999 2024-03-20T09:13:26.953 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.953 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.953 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:13:26.953 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.440122+0000", 2024-03-20T09:13:26.953 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0087218049999999991 2024-03-20T09:13:26.953 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.953 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.953 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.953 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:43.887067+0000", 2024-03-20T09:13:26.954 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.4469455490000001 2024-03-20T09:13:26.954 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.954 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.954 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:13:26.954 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:43.888419+0000", 2024-03-20T09:13:26.954 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001352203 2024-03-20T09:13:26.954 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:26.954 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:13:26.954 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:26.954 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.954 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.954 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:111829 3.d 3:bb84c761:::benchmark_data_smithi122_119671_object12047:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-03-20T09:13:26.954 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:41.307046+0000", 2024-03-20T09:13:26.955 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 465.59057688399997, 2024-03-20T09:13:26.955 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.769093705, 2024-03-20T09:13:26.955 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:13:26.955 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:13:26.955 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:13:26.955 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:13:26.955 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:13:26.955 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 111829 2024-03-20T09:13:26.955 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.955 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:13:26.955 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.955 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:13:26.955 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.307046+0000", 2024-03-20T09:13:26.955 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:26.956 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.956 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.956 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:13:26.956 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.307046+0000", 2024-03-20T09:13:26.956 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:26.956 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.956 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.956 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:13:26.956 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.307048+0000", 2024-03-20T09:13:26.956 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8109999999999999e-06 2024-03-20T09:13:26.956 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.956 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.956 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:13:26.956 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.307054+0000", 2024-03-20T09:13:26.957 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.5769999999999999e-06 2024-03-20T09:13:26.957 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.957 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.957 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:13:26.957 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.307067+0000", 2024-03-20T09:13:26.957 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2813000000000001e-05 2024-03-20T09:13:26.957 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.957 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.957 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:13:26.957 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.307076+0000", 2024-03-20T09:13:26.957 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.2719999999999997e-06 2024-03-20T09:13:26.957 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.957 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.957 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.958 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.478251+0000", 2024-03-20T09:13:26.958 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.17117516799999999 2024-03-20T09:13:26.958 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.958 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.958 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:13:26.958 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.490660+0000", 2024-03-20T09:13:26.958 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.012409369 2024-03-20T09:13:26.958 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.958 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.958 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.958 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.599037+0000", 2024-03-20T09:13:26.958 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.108377022 2024-03-20T09:13:26.958 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.959 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.959 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:13:26.959 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.614142+0000", 2024-03-20T09:13:26.959 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.015104988999999999 2024-03-20T09:13:26.959 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.959 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.959 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.959 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.075409+0000", 2024-03-20T09:13:26.959 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.461266755 2024-03-20T09:13:26.959 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.959 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.959 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:13:26.959 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.076140+0000", 2024-03-20T09:13:26.959 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00073092899999999995 2024-03-20T09:13:26.959 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:26.960 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:13:26.960 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:26.960 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.960 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.960 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:111883 3.d 3:b65c8180:::benchmark_data_smithi122_119671_object12101:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-03-20T09:13:26.960 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:41.800850+0000", 2024-03-20T09:13:26.960 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 465.09677313600002, 2024-03-20T09:13:26.960 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.692717933, 2024-03-20T09:13:26.960 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:13:26.960 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:13:26.960 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:13:26.960 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:13:26.960 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:13:26.961 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 111883 2024-03-20T09:13:26.961 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.961 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:13:26.961 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.961 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:13:26.961 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.800850+0000", 2024-03-20T09:13:26.961 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:26.961 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.961 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.961 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:13:26.961 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.800850+0000", 2024-03-20T09:13:26.961 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:26.961 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.961 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.961 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:13:26.962 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.800851+0000", 2024-03-20T09:13:26.962 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1179999999999999e-06 2024-03-20T09:13:26.962 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.962 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.962 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:13:26.962 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.800856+0000", 2024-03-20T09:13:26.962 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9429999999999999e-06 2024-03-20T09:13:26.962 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.962 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.962 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:13:26.962 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.800867+0000", 2024-03-20T09:13:26.962 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1127e-05 2024-03-20T09:13:26.962 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.963 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.963 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:13:26.963 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:41.800872+0000", 2024-03-20T09:13:26.963 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2549999999999997e-06 2024-03-20T09:13:26.963 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.963 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.963 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.963 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.005639+0000", 2024-03-20T09:13:26.963 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.20476703099999999 2024-03-20T09:13:26.963 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.963 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.963 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:13:26.963 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.015359+0000", 2024-03-20T09:13:26.964 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0097197659999999995 2024-03-20T09:13:26.964 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.964 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.964 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.964 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.076999+0000", 2024-03-20T09:13:26.964 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.061640201999999998 2024-03-20T09:13:26.964 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.964 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.964 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:13:26.964 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.087364+0000", 2024-03-20T09:13:26.964 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.010364485 2024-03-20T09:13:26.964 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.964 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.964 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.965 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.350121+0000", 2024-03-20T09:13:26.965 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2627572740000002 2024-03-20T09:13:26.965 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.965 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.965 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:13:26.965 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.361223+0000", 2024-03-20T09:13:26.965 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.011102272 2024-03-20T09:13:26.965 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.965 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.965 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:13:26.965 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.375549+0000", 2024-03-20T09:13:26.965 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.014325495000000001 2024-03-20T09:13:26.965 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.965 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.966 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:26.966 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.459028+0000", 2024-03-20T09:13:26.966 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.083479353000000006 2024-03-20T09:13:26.966 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.966 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.966 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:26.966 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.462162+0000", 2024-03-20T09:13:26.966 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.003133865 2024-03-20T09:13:26.966 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.966 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.966 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:13:26.966 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.470559+0000", 2024-03-20T09:13:26.966 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0083966969999999998 2024-03-20T09:13:26.966 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.967 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.967 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:13:26.967 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.474293+0000", 2024-03-20T09:13:26.967 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0037343250000000001 2024-03-20T09:13:26.967 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.967 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.967 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:13:26.967 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.493568+0000", 2024-03-20T09:13:26.967 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.019274724999999999 2024-03-20T09:13:26.967 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:26.967 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:13:26.967 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:26.967 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.967 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.968 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:111946 3.d 3:b7750090:::benchmark_data_smithi122_119671_object12164:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-03-20T09:13:26.968 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:42.199835+0000", 2024-03-20T09:13:26.968 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 464.69778763099998, 2024-03-20T09:13:26.968 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6486077139999997, 2024-03-20T09:13:26.968 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:13:26.968 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:13:26.968 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:13:26.968 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:13:26.968 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:13:26.968 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 111946 2024-03-20T09:13:26.968 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.968 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:13:26.968 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.969 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:13:26.969 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.199835+0000", 2024-03-20T09:13:26.969 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:26.969 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.969 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.969 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:13:26.969 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.199835+0000", 2024-03-20T09:13:26.969 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:26.969 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.969 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.969 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:13:26.969 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.199837+0000", 2024-03-20T09:13:26.969 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5680000000000001e-06 2024-03-20T09:13:26.969 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.970 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.970 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:13:26.970 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.199843+0000", 2024-03-20T09:13:26.970 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0399999999999998e-06 2024-03-20T09:13:26.970 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.970 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.970 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:13:26.970 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.199856+0000", 2024-03-20T09:13:26.970 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2880999999999999e-05 2024-03-20T09:13:26.970 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.970 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.970 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:13:26.970 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.199864+0000", 2024-03-20T09:13:26.970 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.3510000000000002e-06 2024-03-20T09:13:26.971 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.971 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.971 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.971 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.260727+0000", 2024-03-20T09:13:26.971 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.060863029999999999 2024-03-20T09:13:26.971 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.971 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.971 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:13:26.971 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.269017+0000", 2024-03-20T09:13:26.971 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0082901120000000005 2024-03-20T09:13:26.971 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.971 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.971 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.971 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.391398+0000", 2024-03-20T09:13:26.972 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.122380398 2024-03-20T09:13:26.972 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.972 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.972 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:13:26.972 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:42.402344+0000", 2024-03-20T09:13:26.972 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.010946193999999999 2024-03-20T09:13:26.972 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.972 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.972 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.972 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.750414+0000", 2024-03-20T09:13:26.972 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.348070517 2024-03-20T09:13:26.972 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.972 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.973 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:13:26.973 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.757236+0000", 2024-03-20T09:13:26.973 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0068216149999999996 2024-03-20T09:13:26.973 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.973 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.973 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:13:26.973 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.774062+0000", 2024-03-20T09:13:26.973 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.016825618000000001 2024-03-20T09:13:26.973 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.973 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.973 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:26.973 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.819616+0000", 2024-03-20T09:13:26.973 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.045554465000000002 2024-03-20T09:13:26.973 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.974 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.974 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:26.974 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.823981+0000", 2024-03-20T09:13:26.974 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0043652680000000003 2024-03-20T09:13:26.974 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.974 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.974 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:13:26.974 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.825526+0000", 2024-03-20T09:13:26.974 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0015445159999999999 2024-03-20T09:13:26.974 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.974 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.974 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:13:26.974 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.830369+0000", 2024-03-20T09:13:26.975 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.004843099 2024-03-20T09:13:26.975 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.975 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.975 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:13:26.975 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:44.848443+0000", 2024-03-20T09:13:26.975 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.018074041999999998 2024-03-20T09:13:26.975 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:26.975 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:13:26.975 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:26.975 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.975 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.975 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:112047 3.d 3:bd3b44cf:::benchmark_data_smithi122_119671_object12265:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-03-20T09:13:26.975 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:43.453333+0000", 2024-03-20T09:13:26.976 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 463.444290185, 2024-03-20T09:13:26.976 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6901377609999999, 2024-03-20T09:13:26.976 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:13:26.976 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:13:26.976 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:13:26.976 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:13:26.976 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:13:26.976 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 112047 2024-03-20T09:13:26.976 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.976 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:13:26.976 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.976 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:13:26.976 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:43.453333+0000", 2024-03-20T09:13:26.977 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:26.977 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.977 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.977 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:13:26.977 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:43.453333+0000", 2024-03-20T09:13:26.977 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:26.977 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.977 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.977 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:13:26.977 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:43.453334+0000", 2024-03-20T09:13:26.977 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.9199999999999999e-07 2024-03-20T09:13:26.977 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.977 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.977 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:13:26.978 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:43.453337+0000", 2024-03-20T09:13:26.978 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.687e-06 2024-03-20T09:13:26.978 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.978 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.978 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:13:26.978 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:43.453345+0000", 2024-03-20T09:13:26.978 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7740000000000004e-06 2024-03-20T09:13:26.978 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.978 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.978 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:13:26.978 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:43.453350+0000", 2024-03-20T09:13:26.978 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.7350000000000004e-06 2024-03-20T09:13:26.978 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.979 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.979 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.979 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:43.661327+0000", 2024-03-20T09:13:26.979 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.20797734300000001 2024-03-20T09:13:26.979 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.979 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.979 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:13:26.979 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:43.675055+0000", 2024-03-20T09:13:26.979 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.013727698 2024-03-20T09:13:26.979 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.979 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.979 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.979 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:43.756153+0000", 2024-03-20T09:13:26.979 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.081097797999999999 2024-03-20T09:13:26.980 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.980 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.980 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:13:26.980 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:43.765280+0000", 2024-03-20T09:13:26.980 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0091267640000000008 2024-03-20T09:13:26.980 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.980 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.980 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.980 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.066723+0000", 2024-03-20T09:13:26.980 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3014431910000002 2024-03-20T09:13:26.980 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.980 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.980 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:13:26.980 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.074394+0000", 2024-03-20T09:13:26.981 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0076713559999999998 2024-03-20T09:13:26.981 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.981 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.981 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:13:26.981 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.092087+0000", 2024-03-20T09:13:26.981 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.017692504000000001 2024-03-20T09:13:26.981 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.981 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.981 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:26.981 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.116812+0000", 2024-03-20T09:13:26.981 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.024725047999999999 2024-03-20T09:13:26.981 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.981 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.981 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:13:26.982 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.118715+0000", 2024-03-20T09:13:26.982 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0019038830000000001 2024-03-20T09:13:26.982 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.982 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.982 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:26.982 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.121559+0000", 2024-03-20T09:13:26.982 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0028439020000000001 2024-03-20T09:13:26.982 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.982 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.982 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:13:26.982 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.126915+0000", 2024-03-20T09:13:26.982 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0053557270000000002 2024-03-20T09:13:26.982 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.982 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.983 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:13:26.983 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:46.143471+0000", 2024-03-20T09:13:26.983 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.016555459000000002 2024-03-20T09:13:26.983 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:26.983 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:13:26.983 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:26.983 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.983 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.983 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:112392 3.d 3:be9f5588:::benchmark_data_smithi122_119671_object12610:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-03-20T09:13:26.983 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:47.628268+0000", 2024-03-20T09:13:26.983 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 459.269354937, 2024-03-20T09:13:26.983 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6413982219999999, 2024-03-20T09:13:26.983 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:13:26.983 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:13:26.983 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:13:26.984 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:13:26.984 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:13:26.984 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 112392 2024-03-20T09:13:26.984 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.984 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:13:26.984 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.984 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:13:26.984 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.628268+0000", 2024-03-20T09:13:26.984 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:26.984 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.984 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.984 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:13:26.984 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.628268+0000", 2024-03-20T09:13:26.984 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:26.984 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.985 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.985 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:13:26.985 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.628270+0000", 2024-03-20T09:13:26.985 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.858e-06 2024-03-20T09:13:26.985 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.985 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.985 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:13:26.985 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.628278+0000", 2024-03-20T09:13:26.985 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7219999999999999e-06 2024-03-20T09:13:26.985 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.985 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.985 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:13:26.985 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.628293+0000", 2024-03-20T09:13:26.985 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4911e-05 2024-03-20T09:13:26.986 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.986 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.986 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:13:26.986 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.628302+0000", 2024-03-20T09:13:26.986 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.9979999999999992e-06 2024-03-20T09:13:26.986 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.986 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.986 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.986 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.695436+0000", 2024-03-20T09:13:26.986 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.067134073000000002 2024-03-20T09:13:26.986 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.986 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.986 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:13:26.987 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.711757+0000", 2024-03-20T09:13:26.987 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.016321473 2024-03-20T09:13:26.987 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.987 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.987 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.987 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.838436+0000", 2024-03-20T09:13:26.987 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.12667895700000001 2024-03-20T09:13:26.987 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.987 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.987 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:13:26.987 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:47.848598+0000", 2024-03-20T09:13:26.987 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.010161554999999999 2024-03-20T09:13:26.987 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.988 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.988 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.988 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.157747+0000", 2024-03-20T09:13:26.988 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.309149385 2024-03-20T09:13:26.988 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.988 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.988 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:13:26.988 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.164763+0000", 2024-03-20T09:13:26.988 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0070162970000000003 2024-03-20T09:13:26.988 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.988 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.988 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:13:26.988 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.176845+0000", 2024-03-20T09:13:26.988 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.012082147999999999 2024-03-20T09:13:26.989 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.989 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.989 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:26.989 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.240665+0000", 2024-03-20T09:13:26.989 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.063819186 2024-03-20T09:13:26.989 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.989 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.989 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:26.989 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.244165+0000", 2024-03-20T09:13:26.989 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0035001580000000002 2024-03-20T09:13:26.989 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.989 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.989 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:13:26.990 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.245837+0000", 2024-03-20T09:13:26.990 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0016719969999999999 2024-03-20T09:13:26.990 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.990 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.990 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:13:26.990 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.250134+0000", 2024-03-20T09:13:26.990 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0042968040000000004 2024-03-20T09:13:26.990 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.990 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.990 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:13:26.990 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.269666+0000", 2024-03-20T09:13:26.990 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0195327 2024-03-20T09:13:26.990 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:26.990 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:13:26.991 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:26.991 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.991 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.991 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:112702 3.d 3:b07597e1:::benchmark_data_smithi122_119671_object12920:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-03-20T09:13:26.991 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:50.657764+0000", 2024-03-20T09:13:26.991 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 456.239859352, 2024-03-20T09:13:26.991 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.7225253120000001, 2024-03-20T09:13:26.991 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:13:26.991 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:13:26.991 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:13:26.991 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:13:26.991 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:13:26.991 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 112702 2024-03-20T09:13:26.991 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.992 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:13:26.992 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.992 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:13:26.992 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.657764+0000", 2024-03-20T09:13:26.992 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:26.992 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.992 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.992 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:13:26.992 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.657764+0000", 2024-03-20T09:13:26.992 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:26.992 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.992 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.993 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:13:26.993 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.657765+0000", 2024-03-20T09:13:26.993 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7060000000000001e-06 2024-03-20T09:13:26.993 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.993 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.993 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:13:26.993 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.657774+0000", 2024-03-20T09:13:26.993 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.7099999999999996e-06 2024-03-20T09:13:26.993 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.993 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.993 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:13:26.993 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.657788+0000", 2024-03-20T09:13:26.993 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4114000000000001e-05 2024-03-20T09:13:26.993 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.994 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.994 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:13:26.994 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.657798+0000", 2024-03-20T09:13:26.994 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.4790000000000002e-06 2024-03-20T09:13:26.994 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.994 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.994 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.994 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.817529+0000", 2024-03-20T09:13:26.994 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.159731015 2024-03-20T09:13:26.994 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.994 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.994 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:13:26.994 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.825842+0000", 2024-03-20T09:13:26.995 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0083134569999999998 2024-03-20T09:13:26.995 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.995 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.995 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.995 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.888052+0000", 2024-03-20T09:13:26.995 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.062210069999999999 2024-03-20T09:13:26.995 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.995 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.995 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:13:26.995 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:50.897510+0000", 2024-03-20T09:13:26.995 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.009457366 2024-03-20T09:13:26.995 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.995 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.995 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:26.996 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:53.244083+0000", 2024-03-20T09:13:26.996 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3465735749999999 2024-03-20T09:13:26.996 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.996 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.996 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:13:26.996 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:53.251105+0000", 2024-03-20T09:13:26.996 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0070215369999999996 2024-03-20T09:13:26.996 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.996 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.996 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:13:26.996 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:53.263052+0000", 2024-03-20T09:13:26.996 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.011947573 2024-03-20T09:13:26.996 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.996 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.997 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:26.997 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:53.276761+0000", 2024-03-20T09:13:26.997 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.013708299 2024-03-20T09:13:26.997 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.997 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.997 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:26.997 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:53.279758+0000", 2024-03-20T09:13:26.997 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0029975029999999999 2024-03-20T09:13:26.997 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.997 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.997 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:13:26.997 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:53.359751+0000", 2024-03-20T09:13:26.997 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.079993274000000003 2024-03-20T09:13:26.997 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.998 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.998 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:13:26.998 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:53.364270+0000", 2024-03-20T09:13:26.998 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.004518762 2024-03-20T09:13:26.998 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.998 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.998 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:13:26.998 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:53.380289+0000", 2024-03-20T09:13:26.998 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.016018872 2024-03-20T09:13:26.998 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:26.998 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:13:26.998 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:26.998 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.999 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:26.999 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:112962 3.d 3:b85f98c8:::benchmark_data_smithi122_119671_object13180:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-03-20T09:13:26.999 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:05:53.037678+0000", 2024-03-20T09:13:26.999 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 453.85994484100002, 2024-03-20T09:13:26.999 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.660180172, 2024-03-20T09:13:26.999 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:13:26.999 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:13:26.999 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:13:26.999 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:13:26.999 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:13:26.999 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 112962 2024-03-20T09:13:26.999 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:26.999 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:13:26.999 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.000 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:13:27.000 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:53.037678+0000", 2024-03-20T09:13:27.000 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:27.000 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.000 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.000 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:13:27.000 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:53.037678+0000", 2024-03-20T09:13:27.000 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:27.000 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.000 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.000 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:13:27.000 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:53.037680+0000", 2024-03-20T09:13:27.000 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.629e-06 2024-03-20T09:13:27.001 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.001 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.001 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:13:27.001 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:53.037688+0000", 2024-03-20T09:13:27.001 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5420000000000005e-06 2024-03-20T09:13:27.001 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.001 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.001 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:13:27.001 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:53.037702+0000", 2024-03-20T09:13:27.001 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3448999999999999e-05 2024-03-20T09:13:27.001 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.001 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.001 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:13:27.001 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:53.037710+0000", 2024-03-20T09:13:27.002 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.2910000000000007e-06 2024-03-20T09:13:27.002 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.002 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.002 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:27.002 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:53.097879+0000", 2024-03-20T09:13:27.002 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.060168511000000001 2024-03-20T09:13:27.002 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.002 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.002 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:13:27.002 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:53.106418+0000", 2024-03-20T09:13:27.002 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0085394370000000004 2024-03-20T09:13:27.002 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.002 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.002 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:27.003 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:53.214704+0000", 2024-03-20T09:13:27.003 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.108286407 2024-03-20T09:13:27.003 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.003 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.003 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:13:27.003 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:53.223056+0000", 2024-03-20T09:13:27.003 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0083511000000000002 2024-03-20T09:13:27.003 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.003 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.003 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:27.003 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:55.625356+0000", 2024-03-20T09:13:27.003 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.4023003780000001 2024-03-20T09:13:27.003 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.004 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.004 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:13:27.004 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:55.634143+0000", 2024-03-20T09:13:27.004 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0087872690000000003 2024-03-20T09:13:27.004 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.004 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.004 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:13:27.004 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:55.647874+0000", 2024-03-20T09:13:27.004 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.013730736 2024-03-20T09:13:27.004 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.004 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.004 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:27.004 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:55.662123+0000", 2024-03-20T09:13:27.004 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.014248856000000001 2024-03-20T09:13:27.004 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.005 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.005 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:27.005 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:55.665279+0000", 2024-03-20T09:13:27.005 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0031563609999999999 2024-03-20T09:13:27.005 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.005 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.005 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:13:27.005 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:55.677190+0000", 2024-03-20T09:13:27.005 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.01191126 2024-03-20T09:13:27.005 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.005 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.005 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:13:27.005 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:55.681670+0000", 2024-03-20T09:13:27.006 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0044800400000000002 2024-03-20T09:13:27.006 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.006 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.006 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:13:27.006 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:05:55.697858+0000", 2024-03-20T09:13:27.006 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.016187905999999998 2024-03-20T09:13:27.006 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:27.006 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:13:27.006 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:27.006 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.006 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.006 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:116726 3.d 3:b33c3aa7:::benchmark_data_smithi122_119671_object16944:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-03-20T09:13:27.006 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:06:29.302227+0000", 2024-03-20T09:13:27.007 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 417.59539610500002, 2024-03-20T09:13:27.007 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.0743580370000001, 2024-03-20T09:13:27.007 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:13:27.007 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:13:27.007 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:13:27.007 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:13:27.007 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:13:27.007 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 116726 2024-03-20T09:13:27.007 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.007 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:13:27.007 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.007 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:13:27.007 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:29.302227+0000", 2024-03-20T09:13:27.007 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:27.008 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.008 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.008 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:13:27.008 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:29.302227+0000", 2024-03-20T09:13:27.008 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:27.008 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.008 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.008 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:13:27.008 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:29.302229+0000", 2024-03-20T09:13:27.008 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.787e-06 2024-03-20T09:13:27.008 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.008 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.008 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:13:27.008 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:29.302236+0000", 2024-03-20T09:13:27.009 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.1199999999999996e-06 2024-03-20T09:13:27.009 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.009 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.009 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:13:27.009 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:29.302250+0000", 2024-03-20T09:13:27.009 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4496e-05 2024-03-20T09:13:27.009 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.009 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.009 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:13:27.009 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:29.302259+0000", 2024-03-20T09:13:27.009 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.9660000000000002e-06 2024-03-20T09:13:27.009 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.009 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.009 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:27.010 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:29.525582+0000", 2024-03-20T09:13:27.010 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.22332232299999999 2024-03-20T09:13:27.010 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.010 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.010 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:13:27.010 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:29.530497+0000", 2024-03-20T09:13:27.010 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0049159010000000003 2024-03-20T09:13:27.010 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.010 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.010 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:27.010 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:30.528451+0000", 2024-03-20T09:13:27.010 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.99795322200000003 2024-03-20T09:13:27.010 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.010 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.011 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:13:27.011 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:30.536402+0000", 2024-03-20T09:13:27.011 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0079508159999999994 2024-03-20T09:13:27.011 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.011 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.011 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:27.011 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:32.291338+0000", 2024-03-20T09:13:27.011 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.754936007 2024-03-20T09:13:27.011 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.011 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.011 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:13:27.011 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:32.297789+0000", 2024-03-20T09:13:27.011 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0064514359999999996 2024-03-20T09:13:27.011 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.011 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.012 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:13:27.012 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:32.308929+0000", 2024-03-20T09:13:27.012 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.011140073 2024-03-20T09:13:27.012 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.012 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.012 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:27.012 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:32.320363+0000", 2024-03-20T09:13:27.012 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.011434199000000001 2024-03-20T09:13:27.012 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.012 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.012 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:27.012 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:32.346857+0000", 2024-03-20T09:13:27.012 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.026493439000000001 2024-03-20T09:13:27.012 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.013 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.013 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:13:27.013 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:32.356761+0000", 2024-03-20T09:13:27.013 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0099038579999999998 2024-03-20T09:13:27.013 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.013 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.013 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:13:27.013 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:32.360912+0000", 2024-03-20T09:13:27.013 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0041510189999999997 2024-03-20T09:13:27.013 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.013 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.013 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:13:27.013 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:32.376585+0000", 2024-03-20T09:13:27.014 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.015673375 2024-03-20T09:13:27.014 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:27.014 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:13:27.014 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:27.014 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.014 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.014 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:116882 3.d 3:b2db6632:::benchmark_data_smithi122_119671_object17100:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-03-20T09:13:27.014 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:06:30.627940+0000", 2024-03-20T09:13:27.014 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 416.26968294300002, 2024-03-20T09:13:27.014 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.4965990499999999, 2024-03-20T09:13:27.014 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:13:27.014 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:13:27.014 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:13:27.014 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:13:27.015 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:13:27.015 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 116882 2024-03-20T09:13:27.015 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.015 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:13:27.015 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.015 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:13:27.015 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:30.627940+0000", 2024-03-20T09:13:27.015 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:27.015 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.015 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.015 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:13:27.015 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:30.627940+0000", 2024-03-20T09:13:27.015 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:27.015 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.016 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.016 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:13:27.016 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:30.627941+0000", 2024-03-20T09:13:27.016 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.108e-06 2024-03-20T09:13:27.016 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.016 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.016 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:13:27.016 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:30.627946+0000", 2024-03-20T09:13:27.016 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1730000000000003e-06 2024-03-20T09:13:27.016 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.016 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.016 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:13:27.016 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:30.627956+0000", 2024-03-20T09:13:27.016 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.3700000000000001e-06 2024-03-20T09:13:27.017 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.017 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.017 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:13:27.017 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:30.627963+0000", 2024-03-20T09:13:27.017 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.2099999999999996e-06 2024-03-20T09:13:27.017 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.017 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.017 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:27.017 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:30.683155+0000", 2024-03-20T09:13:27.017 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.055191993000000002 2024-03-20T09:13:27.017 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.017 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.017 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:13:27.017 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:30.686829+0000", 2024-03-20T09:13:27.018 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0036744910000000002 2024-03-20T09:13:27.018 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.018 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.018 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:27.018 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:32.254528+0000", 2024-03-20T09:13:27.018 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.567698408 2024-03-20T09:13:27.018 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.018 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.018 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:13:27.018 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:32.263009+0000", 2024-03-20T09:13:27.018 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0084811279999999992 2024-03-20T09:13:27.018 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.018 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.018 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:27.019 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:34.023576+0000", 2024-03-20T09:13:27.019 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7605671410000001 2024-03-20T09:13:27.019 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.019 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.019 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:13:27.019 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:34.029828+0000", 2024-03-20T09:13:27.019 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0062514850000000002 2024-03-20T09:13:27.019 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.019 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.019 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:13:27.019 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:34.040527+0000", 2024-03-20T09:13:27.019 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.010699426999999999 2024-03-20T09:13:27.019 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.019 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.020 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:27.020 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:34.093780+0000", 2024-03-20T09:13:27.020 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.053252932000000003 2024-03-20T09:13:27.020 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.020 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.020 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:27.020 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:34.096460+0000", 2024-03-20T09:13:27.020 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0026800399999999999 2024-03-20T09:13:27.020 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.020 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.020 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:13:27.020 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:34.104157+0000", 2024-03-20T09:13:27.020 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0076967340000000002 2024-03-20T09:13:27.020 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.021 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.021 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:13:27.021 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:34.108341+0000", 2024-03-20T09:13:27.021 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0041847619999999999 2024-03-20T09:13:27.021 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.021 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.021 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:13:27.021 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:34.124539+0000", 2024-03-20T09:13:27.021 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.016197647999999999 2024-03-20T09:13:27.021 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:27.021 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:13:27.021 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:27.021 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.021 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.021 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:117334 3.d 3:b30e0839:::benchmark_data_smithi122_119671_object17552:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-03-20T09:13:27.022 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:06:34.167666+0000", 2024-03-20T09:13:27.022 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 412.72995681800001, 2024-03-20T09:13:27.022 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.4823171209999999, 2024-03-20T09:13:27.022 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:13:27.022 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:13:27.022 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:13:27.022 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:13:27.022 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:13:27.022 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 117334 2024-03-20T09:13:27.022 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.022 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:13:27.022 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.022 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:13:27.023 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:34.167666+0000", 2024-03-20T09:13:27.023 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:27.023 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.023 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.023 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:13:27.023 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:34.167666+0000", 2024-03-20T09:13:27.023 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:13:27.023 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.023 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.023 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:13:27.023 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:34.167668+0000", 2024-03-20T09:13:27.023 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8500000000000001e-06 2024-03-20T09:13:27.023 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.023 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.023 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:13:27.024 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:34.167675+0000", 2024-03-20T09:13:27.024 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.1400000000000002e-06 2024-03-20T09:13:27.024 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.024 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.024 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:13:27.024 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:34.167689+0000", 2024-03-20T09:13:27.024 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3380999999999999e-05 2024-03-20T09:13:27.024 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.024 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.024 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:13:27.024 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:34.167696+0000", 2024-03-20T09:13:27.024 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.8900000000000007e-06 2024-03-20T09:13:27.024 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.024 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.025 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:27.025 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:34.227323+0000", 2024-03-20T09:13:27.025 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.059626202000000003 2024-03-20T09:13:27.025 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.025 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.025 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:13:27.025 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:34.230971+0000", 2024-03-20T09:13:27.025 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0036479149999999998 2024-03-20T09:13:27.025 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.025 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.025 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:27.025 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:35.892973+0000", 2024-03-20T09:13:27.025 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.662002347 2024-03-20T09:13:27.025 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.025 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.026 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for rw locks", 2024-03-20T09:13:27.026 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:35.900877+0000", 2024-03-20T09:13:27.026 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0079039339999999996 2024-03-20T09:13:27.026 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.026 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.026 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:13:27.026 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:37.565248+0000", 2024-03-20T09:13:27.026 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6643714009999999 2024-03-20T09:13:27.026 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.026 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.026 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:13:27.026 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:37.571314+0000", 2024-03-20T09:13:27.026 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00606618 2024-03-20T09:13:27.027 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.027 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.027 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:13:27.027 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:37.582039+0000", 2024-03-20T09:13:27.027 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.010724925 2024-03-20T09:13:27.027 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.027 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.027 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:27.027 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:37.627634+0000", 2024-03-20T09:13:27.027 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.045594938000000002 2024-03-20T09:13:27.027 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.027 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.027 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:13:27.027 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:37.630928+0000", 2024-03-20T09:13:27.028 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.003293491 2024-03-20T09:13:27.028 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.028 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.028 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:13:27.028 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:37.632287+0000", 2024-03-20T09:13:27.028 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001359704 2024-03-20T09:13:27.028 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.028 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.028 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:13:27.028 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:37.635670+0000", 2024-03-20T09:13:27.028 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0033825109999999999 2024-03-20T09:13:27.028 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:13:27.028 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:13:27.028 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:13:27.028 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:06:37.649983+0000", 2024-03-20T09:13:27.029 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.014313312 2024-03-20T09:13:27.029 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:27.029 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:13:27.029 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:27.029 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:13:27.029 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:13:27.029 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:13:27.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:27.031 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:27.031 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:13:27.031 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:13:27.132 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:13:27.162 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:27.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:27 smithi026 ceph-mon[68309]: pgmap v1229: 47 pgs: 2 active+undersized+remapped, 1 creating+activating, 2 creating+peering, 42 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-03-20T09:13:27.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:27 smithi026 ceph-mon[68309]: osdmap e525: 12 total, 12 up, 10 in 2024-03-20T09:13:27.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:27 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:13:27.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:27 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:13:27.248 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:27 smithi115 ceph-mon[72476]: pgmap v1229: 47 pgs: 2 active+undersized+remapped, 1 creating+activating, 2 creating+peering, 42 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-03-20T09:13:27.248 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:27 smithi115 ceph-mon[72476]: osdmap e525: 12 total, 12 up, 10 in 2024-03-20T09:13:27.248 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:27 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:13:27.248 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:27 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:13:27.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:27 smithi122 ceph-mon[73495]: pgmap v1229: 47 pgs: 2 active+undersized+remapped, 1 creating+activating, 2 creating+peering, 42 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-03-20T09:13:27.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:27 smithi122 ceph-mon[73495]: osdmap e525: 12 total, 12 up, 10 in 2024-03-20T09:13:27.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:27 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:13:27.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:27 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:13:27.590 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:13:27.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:27.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:13:27.622 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-20T09:13:27.694 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:13:27.725 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:27.744 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:13:27.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:27.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:13:27.849 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:13:27.853 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:13:27.881 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:27.886 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:13:27.921 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:13:28.132 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:13:28.135 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:28 smithi026 ceph-mon[68309]: osdmap e526: 12 total, 12 up, 10 in 2024-03-20T09:13:28.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:28.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:13:28.236 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:13:28.269 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:28.291 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:13:28.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:28.294 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:13:28.395 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:13:28.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:28 smithi115 ceph-mon[72476]: osdmap e526: 12 total, 12 up, 10 in 2024-03-20T09:13:28.423 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:28 smithi122 ceph-mon[73495]: osdmap e526: 12 total, 12 up, 10 in 2024-03-20T09:13:28.425 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:28.445 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:13:28.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:28.448 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:13:28.550 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:13:28.578 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:28.598 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:13:28.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:28.601 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:13:28.702 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:13:28.732 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:28.858 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:13:28.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:28.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:13:28.961 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:13:28.990 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:29.092 INFO:teuthology.orchestra.run.smithi026.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-03-20T09:13:29.117 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:29 smithi026 ceph-mon[68309]: pgmap v1232: 47 pgs: 1 peering, 2 active+undersized+remapped, 44 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-03-20T09:13:29.117 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:29 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/747485124' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-20T09:13:29.118 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:29 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3889263186' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:13:29.218 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:29 smithi115 ceph-mon[72476]: pgmap v1232: 47 pgs: 1 peering, 2 active+undersized+remapped, 44 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-03-20T09:13:29.218 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:29 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/747485124' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-20T09:13:29.218 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:29 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3889263186' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:13:29.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:29 smithi122 ceph-mon[73495]: pgmap v1232: 47 pgs: 1 peering, 2 active+undersized+remapped, 44 active+clean; 577 KiB data, 2.7 GiB used, 891 GiB / 894 GiB avail 2024-03-20T09:13:29.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:29 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/747485124' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-20T09:13:29.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:29 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3889263186' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:13:29.466 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:13:29.466 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:13:29.466 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:13:29.466 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:13:29.466 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:13:29.466 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:13:29.467 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:13:29.467 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:13:29.467 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:13:29.467 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:13:29.467 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:13:29.467 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:13:29.467 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:13:29.467 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:13:29.468 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:13:29.468 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:13:29.468 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:13:29.468 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:13:29.468 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:13:29.468 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:13:29.468 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:13:29.468 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:13:29.468 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:13:29.468 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:13:30.314 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:30 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/747485124' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-20T09:13:30.314 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:30 smithi026 ceph-mon[68309]: osdmap e527: 12 total, 12 up, 10 in 2024-03-20T09:13:30.314 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:30 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:13:30.385 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:30 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/747485124' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-20T09:13:30.386 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:30 smithi115 ceph-mon[72476]: osdmap e527: 12 total, 12 up, 10 in 2024-03-20T09:13:30.386 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:30 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:13:30.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:30 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/747485124' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-20T09:13:30.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:30 smithi122 ceph-mon[73495]: osdmap e527: 12 total, 12 up, 10 in 2024-03-20T09:13:30.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:30 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:13:30.640 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:13:30.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:30.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:13:30.661 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:13:30.677 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:13:30.677 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:13:30.744 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:13:30.774 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:30.971 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:13:30.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:30.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:13:31.077 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:13:31.106 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:31.132 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:31 smithi026 ceph-mon[68309]: pgmap v1234: 47 pgs: 1 peering, 2 active+undersized+remapped, 44 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-03-20T09:13:31.132 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:31 smithi026 ceph-mon[68309]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-20T09:13:31.369 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:31 smithi115 ceph-mon[72476]: pgmap v1234: 47 pgs: 1 peering, 2 active+undersized+remapped, 44 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-03-20T09:13:31.369 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:31 smithi115 ceph-mon[72476]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-20T09:13:31.394 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-20T09:13:31.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:31 smithi122 ceph-mon[73495]: pgmap v1234: 47 pgs: 1 peering, 2 active+undersized+remapped, 44 active+clean; 577 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail 2024-03-20T09:13:31.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:31 smithi122 ceph-mon[73495]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-20T09:13:31.453 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:13:31.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:31.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:13:31.540 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:13:31.541 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:13:31.558 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:13:31.586 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:31.606 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:13:31.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:31.609 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:13:31.612 DEBUG:teuthology.orchestra.run.smithi122:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-03-20T09:13:31.653 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-20T09:13:31.653 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-20T09:13:31.694 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:13:31.701 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:hints = 1 2024-03-20T09:13:31.701 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-20T09:13:31.701 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Object prefix: benchmark_data_smithi122_145698 2024-03-20T09:13:31.702 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T09:13:31.702 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 0 0 0 0 0 0 - 0 2024-03-20T09:13:31.710 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:13:31.745 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:32.000 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:13:32.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:32.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:13:32.030 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:13:32.104 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:13:32.155 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:32.292 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:32 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:13:32.293 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:32 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:13:32.293 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:32 smithi115 ceph-mon[72476]: osdmap e528: 12 total, 12 up, 10 in 2024-03-20T09:13:32.293 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:32 smithi115 ceph-mon[72476]: from='client.15831 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:13:32.293 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:32 smithi115 ceph-mon[72476]: osdmap e529: 12 total, 12 up, 10 in 2024-03-20T09:13:32.357 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:32 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:13:32.357 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:32 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:13:32.357 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:32 smithi026 ceph-mon[68309]: osdmap e528: 12 total, 12 up, 10 in 2024-03-20T09:13:32.357 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:32 smithi026 ceph-mon[68309]: from='client.15831 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:13:32.357 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:32 smithi026 ceph-mon[68309]: osdmap e529: 12 total, 12 up, 10 in 2024-03-20T09:13:32.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:32 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:13:32.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:32 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:13:32.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:32 smithi122 ceph-mon[73495]: osdmap e528: 12 total, 12 up, 10 in 2024-03-20T09:13:32.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:32 smithi122 ceph-mon[73495]: from='client.15831 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:13:32.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:32 smithi122 ceph-mon[73495]: osdmap e529: 12 total, 12 up, 10 in 2024-03-20T09:13:32.702 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 1 16 4046 4030 251.862 251.875 0.002601 0.00395808 2024-03-20T09:13:32.826 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":1235,"stamp":"2024-03-20T09:13:31.099060+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4247,"ondisk_log_size":4247,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":148,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":2763092,"kb_used_data":84048,"kb_used_omap":195,"kb_used_meta":2678716,"kb_avail":934688428,"statfs":{"total":959950356480,"available":957120950272,"internally_reserved":0,"allocated":86065152,"data_stored":72521101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":200482,"internal_metadata":2743005406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.311110"},"pg_stats":[{"pgid":"1.1e","version":"20'137","reported_seq":421,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-20T09:13:29.740317+0000","last_change":"2024-03-20T09:13:20.071999+0000","last_active":"2024-03-20T09:13:29.740317+0000","last_peered":"2024-03-20T09:13:29.740317+0000","last_clean":"2024-03-20T09:13:29.740317+0000","last_became_active":"2024-03-20T09:13:20.071655+0000","last_became_peered":"2024-03-20T09:13:20.071655+0000","last_unstale":"2024-03-20T09:13:29.740317+0000","last_undegraded":"2024-03-20T09:13:29.740317+0000","last_fullsized":"2024-03-20T09:13:29.740317+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:12:31.918194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:12:31.918194+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:13:30.418486+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'137","reported_seq":596,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-20T09:13:29.739667+0000","last_change":"2024-03-20T09:13:05.291275+0000","last_active":"2024-03-20T09:13:29.739667+0000","last_peered":"2024-03-20T09:13:29.739667+0000","last_clean":"2024-03-20T09:13:29.739667+0000","last_became_active":"2024-03-20T09:07:42.013245+0000","last_became_peered":"2024-03-20T09:07:42.013245+0000","last_unstale":"2024-03-20T09:13:29.739667+0000","last_undegraded":"2024-03-20T09:13:29.739667+0000","last_fullsized":"2024-03-20T09:13:29.739667+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:05.291136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:05.291136+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:17.362626+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"20'137","reported_seq":1034,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-20T09:13:26.078155+0000","last_change":"2024-03-20T09:12:42.792997+0000","last_active":"2024-03-20T09:13:26.078155+0000","last_peered":"2024-03-20T09:13:26.078155+0000","last_clean":"2024-03-20T09:13:26.078155+0000","last_became_active":"2024-03-20T09:07:41.961622+0000","last_became_peered":"2024-03-20T09:07:41.961622+0000","last_unstale":"2024-03-20T09:13:26.078155+0000","last_undegraded":"2024-03-20T09:13:26.078155+0000","last_fullsized":"2024-03-20T09:13:26.078155+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:12:42.792916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:12:42.792916+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:13:54.742647+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.19","version":"20'137","reported_seq":1103,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-20T09:13:26.076615+0000","last_change":"2024-03-20T09:13:11.366034+0000","last_active":"2024-03-20T09:13:26.076615+0000","last_peered":"2024-03-20T09:13:26.076615+0000","last_clean":"2024-03-20T09:13:26.076615+0000","last_became_active":"2024-03-20T09:07:41.962680+0000","last_became_peered":"2024-03-20T09:07:41.962680+0000","last_unstale":"2024-03-20T09:13:26.076615+0000","last_undegraded":"2024-03-20T09:13:26.076615+0000","last_fullsized":"2024-03-20T09:13:26.076615+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:11.365892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:11.365892+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:32.244898+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"20'137","reported_seq":557,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-20T09:13:26.074580+0000","last_change":"2024-03-20T09:10:35.278055+0000","last_active":"2024-03-20T09:13:26.074580+0000","last_peered":"2024-03-20T09:13:26.074580+0000","last_clean":"2024-03-20T09:13:26.074580+0000","last_became_active":"2024-03-20T09:10:35.277481+0000","last_became_peered":"2024-03-20T09:10:35.277481+0000","last_unstale":"2024-03-20T09:13:26.074580+0000","last_undegraded":"2024-03-20T09:13:26.074580+0000","last_fullsized":"2024-03-20T09:13:26.074580+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:10:14.880546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:10:14.880546+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:13:30.452872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1b","version":"20'137","reported_seq":863,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-20T09:13:26.077702+0000","last_change":"2024-03-20T09:12:39.855212+0000","last_active":"2024-03-20T09:13:26.077702+0000","last_peered":"2024-03-20T09:13:26.077702+0000","last_clean":"2024-03-20T09:13:26.077702+0000","last_became_active":"2024-03-20T09:07:41.962733+0000","last_became_peered":"2024-03-20T09:07:41.962733+0000","last_unstale":"2024-03-20T09:13:26.077702+0000","last_undegraded":"2024-03-20T09:13:26.077702+0000","last_fullsized":"2024-03-20T09:13:26.077702+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:12:39.855105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:12:39.855105+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:13:44.240877+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":24,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-20T09:13:29.739433+0000","last_change":"2024-03-20T09:13:22.063299+0000","last_active":"2024-03-20T09:13:29.739433+0000","last_peered":"2024-03-20T09:13:29.739433+0000","last_clean":"2024-03-20T09:13:29.739433+0000","last_became_active":"2024-03-20T09:13:22.063020+0000","last_became_peered":"2024-03-20T09:13:22.063020+0000","last_unstale":"2024-03-20T09:13:29.739433+0000","last_undegraded":"2024-03-20T09:13:29.739433+0000","last_fullsized":"2024-03-20T09:13:29.739433+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:41.788247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"20'137","reported_seq":556,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-20T09:13:26.074700+0000","last_change":"2024-03-20T09:10:35.277369+0000","last_active":"2024-03-20T09:13:26.074700+0000","last_peered":"2024-03-20T09:13:26.074700+0000","last_clean":"2024-03-20T09:13:26.074700+0000","last_became_active":"2024-03-20T09:10:35.215772+0000","last_became_peered":"2024-03-20T09:10:35.215772+0000","last_unstale":"2024-03-20T09:13:26.074700+0000","last_undegraded":"2024-03-20T09:13:26.074700+0000","last_fullsized":"2024-03-20T09:13:26.074700+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:10:13.914046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:10:13.914046+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:13:30.452852+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":24,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-20T09:13:29.091641+0000","last_change":"2024-03-20T09:13:22.583666+0000","last_active":"2024-03-20T09:13:29.091641+0000","last_peered":"2024-03-20T09:13:29.091641+0000","last_clean":"2024-03-20T09:13:29.091641+0000","last_became_active":"2024-03-20T09:13:22.583317+0000","last_became_peered":"2024-03-20T09:13:22.583317+0000","last_unstale":"2024-03-20T09:13:29.091641+0000","last_undegraded":"2024-03-20T09:13:29.091641+0000","last_fullsized":"2024-03-20T09:13:29.091641+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:29.205626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,8],"acting":[3,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"20'137","reported_seq":1118,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-03-20T09:13:27.298695+0000","last_change":"2024-03-20T09:13:05.129380+0000","last_active":"2024-03-20T09:13:27.298695+0000","last_peered":"2024-03-20T09:13:27.298695+0000","last_clean":"2024-03-20T09:13:27.298695+0000","last_became_active":"2024-03-20T09:10:34.099414+0000","last_became_peered":"2024-03-20T09:10:34.099414+0000","last_unstale":"2024-03-20T09:13:27.298695+0000","last_undegraded":"2024-03-20T09:13:27.298695+0000","last_fullsized":"2024-03-20T09:13:27.298695+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:05.129288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:05.129288+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:16.260612+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":24,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-20T09:13:29.739763+0000","last_change":"2024-03-20T09:13:22.066637+0000","last_active":"2024-03-20T09:13:29.739763+0000","last_peered":"2024-03-20T09:13:29.739763+0000","last_clean":"2024-03-20T09:13:29.739763+0000","last_became_active":"2024-03-20T09:13:22.066107+0000","last_became_peered":"2024-03-20T09:13:22.066107+0000","last_unstale":"2024-03-20T09:13:29.739763+0000","last_undegraded":"2024-03-20T09:13:29.739763+0000","last_fullsized":"2024-03-20T09:13:29.739763+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:40.959850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,11],"acting":[1,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"20'137","reported_seq":925,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-20T09:13:29.739482+0000","last_change":"2024-03-20T09:12:42.370648+0000","last_active":"2024-03-20T09:13:29.739482+0000","last_peered":"2024-03-20T09:13:29.739482+0000","last_clean":"2024-03-20T09:13:29.739482+0000","last_became_active":"2024-03-20T09:07:52.618107+0000","last_became_peered":"2024-03-20T09:07:52.618107+0000","last_unstale":"2024-03-20T09:13:29.739482+0000","last_undegraded":"2024-03-20T09:13:29.739482+0000","last_fullsized":"2024-03-20T09:13:29.739482+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:12:42.370551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:12:42.370551+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:13:56.545080+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":24,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-20T09:13:29.740247+0000","last_change":"2024-03-20T09:13:22.064363+0000","last_active":"2024-03-20T09:13:29.740247+0000","last_peered":"2024-03-20T09:13:29.740247+0000","last_clean":"2024-03-20T09:13:29.740247+0000","last_became_active":"2024-03-20T09:13:22.063975+0000","last_became_peered":"2024-03-20T09:13:22.063975+0000","last_unstale":"2024-03-20T09:13:29.740247+0000","last_undegraded":"2024-03-20T09:13:29.740247+0000","last_fullsized":"2024-03-20T09:13:29.740247+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:45.622073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"20'137","reported_seq":1089,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-20T09:13:29.740030+0000","last_change":"2024-03-20T09:13:14.395564+0000","last_active":"2024-03-20T09:13:29.740030+0000","last_peered":"2024-03-20T09:13:29.740030+0000","last_clean":"2024-03-20T09:13:29.740030+0000","last_became_active":"2024-03-20T09:12:14.306384+0000","last_became_peered":"2024-03-20T09:12:14.306384+0000","last_unstale":"2024-03-20T09:13:29.740030+0000","last_undegraded":"2024-03-20T09:13:29.740030+0000","last_fullsized":"2024-03-20T09:13:29.740030+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:14.395469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:14.395469+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:24.496429+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":20,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-20T09:13:29.091605+0000","last_change":"2024-03-20T09:13:24.064404+0000","last_active":"2024-03-20T09:13:29.091605+0000","last_peered":"2024-03-20T09:13:29.091605+0000","last_clean":"2024-03-20T09:13:29.091605+0000","last_became_active":"2024-03-20T09:13:24.064091+0000","last_became_peered":"2024-03-20T09:13:24.064091+0000","last_unstale":"2024-03-20T09:13:29.091605+0000","last_undegraded":"2024-03-20T09:13:29.091605+0000","last_fullsized":"2024-03-20T09:13:29.091605+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:33.687560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"20'137","reported_seq":981,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-20T09:13:26.076876+0000","last_change":"2024-03-20T09:13:10.343309+0000","last_active":"2024-03-20T09:13:26.076876+0000","last_peered":"2024-03-20T09:13:26.076876+0000","last_clean":"2024-03-20T09:13:26.076876+0000","last_became_active":"2024-03-20T09:07:37.003722+0000","last_became_peered":"2024-03-20T09:07:37.003722+0000","last_unstale":"2024-03-20T09:13:26.076876+0000","last_undegraded":"2024-03-20T09:13:26.076876+0000","last_fullsized":"2024-03-20T09:13:26.076876+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:10.343198+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:07:06.480160+0000","last_clean_scrub_stamp":"2024-03-20T09:13:10.343198+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:33.639911+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":24,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-20T09:13:29.740166+0000","last_change":"2024-03-20T09:13:22.064621+0000","last_active":"2024-03-20T09:13:29.740166+0000","last_peered":"2024-03-20T09:13:29.740166+0000","last_clean":"2024-03-20T09:13:29.740166+0000","last_became_active":"2024-03-20T09:13:22.064275+0000","last_became_peered":"2024-03-20T09:13:22.064275+0000","last_unstale":"2024-03-20T09:13:29.740166+0000","last_undegraded":"2024-03-20T09:13:29.740166+0000","last_fullsized":"2024-03-20T09:13:29.740166+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:40.477405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"20'137","reported_seq":998,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-20T09:13:26.076792+0000","last_change":"2024-03-20T09:13:12.359759+0000","last_active":"2024-03-20T09:13:26.076792+0000","last_peered":"2024-03-20T09:13:26.076792+0000","last_clean":"2024-03-20T09:13:26.076792+0000","last_became_active":"2024-03-20T09:07:52.593671+0000","last_became_peered":"2024-03-20T09:07:52.593671+0000","last_unstale":"2024-03-20T09:13:26.076792+0000","last_undegraded":"2024-03-20T09:13:26.076792+0000","last_fullsized":"2024-03-20T09:13:26.076792+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:12.359617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:12.359617+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:24.685959+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":24,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-20T09:13:29.739525+0000","last_change":"2024-03-20T09:13:22.067664+0000","last_active":"2024-03-20T09:13:29.739525+0000","last_peered":"2024-03-20T09:13:29.739525+0000","last_clean":"2024-03-20T09:13:29.739525+0000","last_became_active":"2024-03-20T09:13:22.067362+0000","last_became_peered":"2024-03-20T09:13:22.067362+0000","last_unstale":"2024-03-20T09:13:29.739525+0000","last_undegraded":"2024-03-20T09:13:29.739525+0000","last_fullsized":"2024-03-20T09:13:29.739525+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:46.912906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"20'137","reported_seq":994,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-20T09:13:29.739670+0000","last_change":"2024-03-20T09:12:02.221970+0000","last_active":"2024-03-20T09:13:29.739670+0000","last_peered":"2024-03-20T09:13:29.739670+0000","last_clean":"2024-03-20T09:13:29.739670+0000","last_became_active":"2024-03-20T09:12:02.221686+0000","last_became_peered":"2024-03-20T09:12:02.221686+0000","last_unstale":"2024-03-20T09:13:29.739670+0000","last_undegraded":"2024-03-20T09:13:29.739670+0000","last_fullsized":"2024-03-20T09:13:29.739670+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:10:18.788771+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:18.788771+0000","last_clean_scrub_stamp":"2024-03-20T09:10:18.788771+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:13:33.104039+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,8],"acting":[1,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":24,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-20T09:13:29.739672+0000","last_change":"2024-03-20T09:13:22.063240+0000","last_active":"2024-03-20T09:13:29.739672+0000","last_peered":"2024-03-20T09:13:29.739672+0000","last_clean":"2024-03-20T09:13:29.739672+0000","last_became_active":"2024-03-20T09:13:22.062992+0000","last_became_peered":"2024-03-20T09:13:22.062992+0000","last_unstale":"2024-03-20T09:13:29.739672+0000","last_undegraded":"2024-03-20T09:13:29.739672+0000","last_fullsized":"2024-03-20T09:13:29.739672+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:36.289001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":1060,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-20T09:13:29.739911+0000","last_change":"2024-03-20T09:13:03.215035+0000","last_active":"2024-03-20T09:13:29.739911+0000","last_peered":"2024-03-20T09:13:29.739911+0000","last_clean":"2024-03-20T09:13:29.739911+0000","last_became_active":"2024-03-20T09:12:14.305989+0000","last_became_peered":"2024-03-20T09:12:14.305989+0000","last_unstale":"2024-03-20T09:13:29.739911+0000","last_undegraded":"2024-03-20T09:13:29.739911+0000","last_fullsized":"2024-03-20T09:13:29.739911+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:03.214902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:03.214902+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:13.441726+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":24,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-20T09:13:29.739911+0000","last_change":"2024-03-20T09:13:22.066405+0000","last_active":"2024-03-20T09:13:29.739911+0000","last_peered":"2024-03-20T09:13:29.739911+0000","last_clean":"2024-03-20T09:13:29.739911+0000","last_became_active":"2024-03-20T09:13:22.065656+0000","last_became_peered":"2024-03-20T09:13:22.065656+0000","last_unstale":"2024-03-20T09:13:29.739911+0000","last_undegraded":"2024-03-20T09:13:29.739911+0000","last_fullsized":"2024-03-20T09:13:29.739911+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:23.500958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,0],"acting":[1,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"20'137","reported_seq":1298,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-20T09:13:26.075900+0000","last_change":"2024-03-20T09:12:14.313935+0000","last_active":"2024-03-20T09:13:26.075900+0000","last_peered":"2024-03-20T09:13:26.075900+0000","last_clean":"2024-03-20T09:13:26.075900+0000","last_became_active":"2024-03-20T09:12:14.313541+0000","last_became_peered":"2024-03-20T09:12:14.313541+0000","last_unstale":"2024-03-20T09:13:26.075900+0000","last_undegraded":"2024-03-20T09:13:26.075900+0000","last_fullsized":"2024-03-20T09:13:26.075900+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:10:19.513511+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:19.513511+0000","last_clean_scrub_stamp":"2024-03-20T09:10:19.513511+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:13:26.632238+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":29,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-20T09:13:29.739485+0000","last_change":"2024-03-20T09:13:24.443379+0000","last_active":"2024-03-20T09:13:29.739485+0000","last_peered":"2024-03-20T09:13:29.739485+0000","last_clean":"2024-03-20T09:13:29.739485+0000","last_became_active":"2024-03-20T09:13:24.443054+0000","last_became_peered":"2024-03-20T09:13:24.443054+0000","last_unstale":"2024-03-20T09:13:29.739485+0000","last_undegraded":"2024-03-20T09:13:29.739485+0000","last_fullsized":"2024-03-20T09:13:29.739485+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:41.617061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"20'137","reported_seq":1002,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-20T09:13:26.076780+0000","last_change":"2024-03-20T09:13:14.428508+0000","last_active":"2024-03-20T09:13:26.076780+0000","last_peered":"2024-03-20T09:13:26.076780+0000","last_clean":"2024-03-20T09:13:26.076780+0000","last_became_active":"2024-03-20T09:06:02.923630+0000","last_became_peered":"2024-03-20T09:06:02.923630+0000","last_unstale":"2024-03-20T09:13:26.076780+0000","last_undegraded":"2024-03-20T09:13:26.076780+0000","last_fullsized":"2024-03-20T09:13:26.076780+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:14.428416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:14.428416+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:27.674669+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2],"acting":[10,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-20T09:13:26.074776+0000","last_change":"2024-03-20T09:13:22.584008+0000","last_active":"2024-03-20T09:13:26.074776+0000","last_peered":"2024-03-20T09:13:26.074776+0000","last_clean":"2024-03-20T09:13:26.074776+0000","last_became_active":"2024-03-20T09:13:22.583529+0000","last_became_peered":"2024-03-20T09:13:22.583529+0000","last_unstale":"2024-03-20T09:13:26.074776+0000","last_undegraded":"2024-03-20T09:13:26.074776+0000","last_fullsized":"2024-03-20T09:13:26.074776+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:31.869773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,8],"acting":[5,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"20'137","reported_seq":998,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-20T09:13:29.739640+0000","last_change":"2024-03-20T09:13:27.088031+0000","last_active":"2024-03-20T09:13:29.739640+0000","last_peered":"2024-03-20T09:13:29.739640+0000","last_clean":"2024-03-20T09:13:29.739640+0000","last_became_active":"2024-03-20T09:13:27.087760+0000","last_became_peered":"2024-03-20T09:13:27.087760+0000","last_unstale":"2024-03-20T09:13:29.739640+0000","last_undegraded":"2024-03-20T09:13:29.739640+0000","last_fullsized":"2024-03-20T09:13:29.739640+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:12:43.355431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:12:43.355431+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:10.456031+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":19,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-20T09:13:26.076674+0000","last_change":"2024-03-20T09:13:22.064771+0000","last_active":"2024-03-20T09:13:26.076674+0000","last_peered":"2024-03-20T09:13:26.076674+0000","last_clean":"2024-03-20T09:13:26.076674+0000","last_became_active":"2024-03-20T09:13:22.064461+0000","last_became_peered":"2024-03-20T09:13:22.064461+0000","last_unstale":"2024-03-20T09:13:26.076674+0000","last_undegraded":"2024-03-20T09:13:26.076674+0000","last_fullsized":"2024-03-20T09:13:26.076674+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:46.459357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,1],"acting":[10,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.b","version":"20'137","reported_seq":866,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-20T09:13:29.740118+0000","last_change":"2024-03-20T09:13:20.070915+0000","last_active":"2024-03-20T09:13:29.740118+0000","last_peered":"2024-03-20T09:13:29.740118+0000","last_clean":"2024-03-20T09:13:29.740118+0000","last_became_active":"2024-03-20T09:13:20.070596+0000","last_became_peered":"2024-03-20T09:13:20.070596+0000","last_unstale":"2024-03-20T09:13:29.740118+0000","last_undegraded":"2024-03-20T09:13:29.740118+0000","last_fullsized":"2024-03-20T09:13:29.740118+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:10:21.687117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:10:21.687117+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-20T09:13:26.075391+0000","last_change":"2024-03-20T09:13:22.065041+0000","last_active":"2024-03-20T09:13:26.075391+0000","last_peered":"2024-03-20T09:13:26.075391+0000","last_clean":"2024-03-20T09:13:26.075391+0000","last_became_active":"2024-03-20T09:13:22.064687+0000","last_became_peered":"2024-03-20T09:13:22.064687+0000","last_unstale":"2024-03-20T09:13:26.075391+0000","last_undegraded":"2024-03-20T09:13:26.075391+0000","last_fullsized":"2024-03-20T09:13:26.075391+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:43.225518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"20'137","reported_seq":526,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-20T09:13:29.091381+0000","last_change":"2024-03-20T09:13:05.598138+0000","last_active":"2024-03-20T09:13:29.091381+0000","last_peered":"2024-03-20T09:13:29.091381+0000","last_clean":"2024-03-20T09:13:29.091381+0000","last_became_active":"2024-03-20T09:12:14.311319+0000","last_became_peered":"2024-03-20T09:12:14.311319+0000","last_unstale":"2024-03-20T09:13:29.091381+0000","last_undegraded":"2024-03-20T09:13:29.091381+0000","last_fullsized":"2024-03-20T09:13:29.091381+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:05.598028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:05.598028+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:07.032921+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":24,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-20T09:13:29.739519+0000","last_change":"2024-03-20T09:13:22.064279+0000","last_active":"2024-03-20T09:13:29.739519+0000","last_peered":"2024-03-20T09:13:29.739519+0000","last_clean":"2024-03-20T09:13:29.739519+0000","last_became_active":"2024-03-20T09:13:22.063927+0000","last_became_peered":"2024-03-20T09:13:22.063927+0000","last_unstale":"2024-03-20T09:13:29.739519+0000","last_undegraded":"2024-03-20T09:13:29.739519+0000","last_fullsized":"2024-03-20T09:13:29.739519+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:37.913884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"20'137","reported_seq":1011,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-20T09:13:29.739502+0000","last_change":"2024-03-20T09:12:40.360273+0000","last_active":"2024-03-20T09:13:29.739502+0000","last_peered":"2024-03-20T09:13:29.739502+0000","last_clean":"2024-03-20T09:13:29.739502+0000","last_became_active":"2024-03-20T09:10:35.119706+0000","last_became_peered":"2024-03-20T09:10:35.119706+0000","last_unstale":"2024-03-20T09:13:29.739502+0000","last_undegraded":"2024-03-20T09:13:29.739502+0000","last_fullsized":"2024-03-20T09:13:29.739502+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:12:40.360146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:12:40.360146+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:13:50.360030+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":29,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-20T09:13:29.740009+0000","last_change":"2024-03-20T09:13:24.065781+0000","last_active":"2024-03-20T09:13:29.740009+0000","last_peered":"2024-03-20T09:13:29.740009+0000","last_clean":"2024-03-20T09:13:29.740009+0000","last_became_active":"2024-03-20T09:13:24.065349+0000","last_became_peered":"2024-03-20T09:13:24.065349+0000","last_unstale":"2024-03-20T09:13:29.740009+0000","last_undegraded":"2024-03-20T09:13:29.740009+0000","last_fullsized":"2024-03-20T09:13:29.740009+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:29.930666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"20'137","reported_seq":1041,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-20T09:13:29.091723+0000","last_change":"2024-03-20T09:13:27.088076+0000","last_active":"2024-03-20T09:13:29.091723+0000","last_peered":"2024-03-20T09:13:29.091723+0000","last_clean":"2024-03-20T09:13:29.091723+0000","last_became_active":"2024-03-20T09:13:27.087686+0000","last_became_peered":"2024-03-20T09:13:27.087686+0000","last_unstale":"2024-03-20T09:13:29.091723+0000","last_undegraded":"2024-03-20T09:13:29.091723+0000","last_fullsized":"2024-03-20T09:13:29.091723+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:09.755064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:09.755064+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:33.274547+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":24,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-20T09:13:29.739730+0000","last_change":"2024-03-20T09:13:22.064047+0000","last_active":"2024-03-20T09:13:29.739730+0000","last_peered":"2024-03-20T09:13:29.739730+0000","last_clean":"2024-03-20T09:13:29.739730+0000","last_became_active":"2024-03-20T09:13:22.063673+0000","last_became_peered":"2024-03-20T09:13:22.063673+0000","last_unstale":"2024-03-20T09:13:29.739730+0000","last_undegraded":"2024-03-20T09:13:29.739730+0000","last_fullsized":"2024-03-20T09:13:29.739730+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:39.125308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"20'137","reported_seq":585,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-20T09:13:29.739551+0000","last_change":"2024-03-20T09:12:14.308242+0000","last_active":"2024-03-20T09:13:29.739551+0000","last_peered":"2024-03-20T09:13:29.739551+0000","last_clean":"2024-03-20T09:13:29.739551+0000","last_became_active":"2024-03-20T09:12:14.307864+0000","last_became_peered":"2024-03-20T09:12:14.307864+0000","last_unstale":"2024-03-20T09:13:29.739551+0000","last_undegraded":"2024-03-20T09:13:29.739551+0000","last_fullsized":"2024-03-20T09:13:29.739551+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:08:56.938721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:08:56.938721+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:13:33.104000+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"20'137","reported_seq":905,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-20T09:13:26.074867+0000","last_change":"2024-03-20T09:13:04.270196+0000","last_active":"2024-03-20T09:13:26.074867+0000","last_peered":"2024-03-20T09:13:26.074867+0000","last_clean":"2024-03-20T09:13:26.074867+0000","last_became_active":"2024-03-20T09:10:34.096758+0000","last_became_peered":"2024-03-20T09:10:34.096758+0000","last_unstale":"2024-03-20T09:13:26.074867+0000","last_undegraded":"2024-03-20T09:13:26.074867+0000","last_fullsized":"2024-03-20T09:13:26.074867+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:04.270076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:04.270076+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:18.468974+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"20'137","reported_seq":974,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-20T09:13:29.739534+0000","last_change":"2024-03-20T09:10:14.741767+0000","last_active":"2024-03-20T09:13:29.739534+0000","last_peered":"2024-03-20T09:13:29.739534+0000","last_clean":"2024-03-20T09:13:29.739534+0000","last_became_active":"2024-03-20T09:04:04.103645+0000","last_became_peered":"2024-03-20T09:04:04.103645+0000","last_unstale":"2024-03-20T09:13:29.739534+0000","last_undegraded":"2024-03-20T09:13:29.739534+0000","last_fullsized":"2024-03-20T09:13:29.739534+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:10:14.741606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:10:14.741606+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:13:33.104057+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"20'137","reported_seq":431,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-20T09:13:26.074776+0000","last_change":"2024-03-20T09:13:05.305611+0000","last_active":"2024-03-20T09:13:26.074776+0000","last_peered":"2024-03-20T09:13:26.074776+0000","last_clean":"2024-03-20T09:13:26.074776+0000","last_became_active":"2024-03-20T09:10:34.097191+0000","last_became_peered":"2024-03-20T09:10:34.097191+0000","last_unstale":"2024-03-20T09:13:26.074776+0000","last_undegraded":"2024-03-20T09:13:26.074776+0000","last_fullsized":"2024-03-20T09:13:26.074776+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:05.305514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:05.305514+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:31.997881+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"20'137","reported_seq":827,"reported_epoch":524,"state":"peering","last_fresh":"2024-03-20T09:13:25.182185+0000","last_change":"2024-03-20T09:13:26.075373+0000","last_active":"2024-03-20T09:13:26.075373+0000","last_peered":"2024-03-20T09:13:25.182185+0000","last_clean":"2024-03-20T09:13:22.060102+0000","last_became_active":"2024-03-20T09:13:25.181878+0000","last_became_peered":"2024-03-20T09:13:25.181878+0000","last_unstale":"2024-03-20T09:13:25.182185+0000","last_undegraded":"2024-03-20T09:13:25.182185+0000","last_fullsized":"2024-03-20T09:13:25.064094+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:04:42.894571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:04:42.894571+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,3],"acting":[0,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3,8],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"20'137","reported_seq":670,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-20T09:13:29.739653+0000","last_change":"2024-03-20T09:13:27.088201+0000","last_active":"2024-03-20T09:13:29.739653+0000","last_peered":"2024-03-20T09:13:29.739653+0000","last_clean":"2024-03-20T09:13:29.739653+0000","last_became_active":"2024-03-20T09:13:27.087760+0000","last_became_peered":"2024-03-20T09:13:27.087760+0000","last_unstale":"2024-03-20T09:13:29.739653+0000","last_undegraded":"2024-03-20T09:13:29.739653+0000","last_fullsized":"2024-03-20T09:13:29.739653+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:10.182025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:10.182025+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:40.149783+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"20'137","reported_seq":985,"reported_epoch":525,"state":"active+clean","last_fresh":"2024-03-20T09:13:26.077942+0000","last_change":"2024-03-20T09:12:36.869949+0000","last_active":"2024-03-20T09:13:26.077942+0000","last_peered":"2024-03-20T09:13:26.077942+0000","last_clean":"2024-03-20T09:13:26.077942+0000","last_became_active":"2024-03-20T09:10:35.095339+0000","last_became_peered":"2024-03-20T09:10:35.095339+0000","last_unstale":"2024-03-20T09:13:26.077942+0000","last_undegraded":"2024-03-20T09:13:26.077942+0000","last_fullsized":"2024-03-20T09:13:26.077942+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:12:36.869848+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:23.640533+0000","last_clean_scrub_stamp":"2024-03-20T09:12:36.869848+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:13:47.865605+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"20'137","reported_seq":1056,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-03-20T09:13:27.298779+0000","last_change":"2024-03-20T09:12:52.043287+0000","last_active":"2024-03-20T09:13:27.298779+0000","last_peered":"2024-03-20T09:13:27.298779+0000","last_clean":"2024-03-20T09:13:27.298779+0000","last_became_active":"2024-03-20T09:04:14.208448+0000","last_became_peered":"2024-03-20T09:04:14.208448+0000","last_unstale":"2024-03-20T09:13:27.298779+0000","last_undegraded":"2024-03-20T09:13:27.298779+0000","last_fullsized":"2024-03-20T09:13:27.298779+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:12:52.043144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:12:52.043144+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:17.347869+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"20'137","reported_seq":323,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-20T09:13:29.091424+0000","last_change":"2024-03-20T09:13:04.628974+0000","last_active":"2024-03-20T09:13:29.091424+0000","last_peered":"2024-03-20T09:13:29.091424+0000","last_clean":"2024-03-20T09:13:29.091424+0000","last_became_active":"2024-03-20T09:12:14.307581+0000","last_became_peered":"2024-03-20T09:12:14.307581+0000","last_unstale":"2024-03-20T09:13:29.091424+0000","last_undegraded":"2024-03-20T09:13:29.091424+0000","last_fullsized":"2024-03-20T09:13:29.091424+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:04.628862+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:07:06.480160+0000","last_clean_scrub_stamp":"2024-03-20T09:13:04.628862+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:10.875071+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"20'137","reported_seq":594,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-20T09:13:29.739639+0000","last_change":"2024-03-20T09:13:20.074838+0000","last_active":"2024-03-20T09:13:29.739639+0000","last_peered":"2024-03-20T09:13:29.739639+0000","last_clean":"2024-03-20T09:13:29.739639+0000","last_became_active":"2024-03-20T09:13:20.074439+0000","last_became_peered":"2024-03-20T09:13:20.074439+0000","last_unstale":"2024-03-20T09:13:29.739639+0000","last_undegraded":"2024-03-20T09:13:29.739639+0000","last_fullsized":"2024-03-20T09:13:29.739639+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:10:17.768353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:10:17.768353+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:13:33.104022+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,11],"acting":[1,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1925120,"data_stored":1914464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4247,"ondisk_log_size":4247,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860625,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":301492,"kb_used_data":3752,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":93443660,"statfs":{"total":95995035648,"available":95686307840,"internally_reserved":0,"allocated":3842048,"data_stored":2487734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":304871876},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":89,"seq":382252089558,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":233440,"kb_used_data":3728,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":93511712,"statfs":{"total":95995035648,"available":95755993088,"internally_reserved":0,"allocated":3817472,"data_stored":2472090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":235207104},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":0,"seq":343597383902,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":70,"seq":300647710946,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":219560,"kb_used_data":3740,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93525592,"statfs":{"total":95995035648,"available":95770206208,"internally_reserved":0,"allocated":3829760,"data_stored":2479925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":220985798},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":60,"seq":257698037992,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":409992,"kb_used_data":3964,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":93335160,"statfs":{"total":95995035648,"available":95575203840,"internally_reserved":0,"allocated":4059136,"data_stored":2697062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":415758794},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332333,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":265728,"kb_used_data":4084,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":93479424,"statfs":{"total":95995035648,"available":95722930176,"internally_reserved":0,"allocated":4182016,"data_stored":2823494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":267909574},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575111,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":345920,"kb_used_data":3764,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93399232,"statfs":{"total":95995035648,"available":95640813568,"internally_reserved":0,"allocated":3854336,"data_stored":2495814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":350353862},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738629,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":372648,"kb_used_data":16604,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93372504,"statfs":{"total":95995035648,"available":95613444096,"internally_reserved":0,"allocated":17002496,"data_stored":15652661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411584,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":233860,"kb_used_data":36080,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":93511292,"statfs":{"total":95995035648,"available":95755563008,"internally_reserved":0,"allocated":36945920,"data_stored":35588054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":202504645},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149948,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":260148,"kb_used_data":4072,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":93485004,"statfs":{"total":95995035648,"available":95728644096,"internally_reserved":0,"allocated":4169728,"data_stored":2815414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":262207943},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953722,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":443,"seq":1902670512164,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":120304,"kb_used_data":4260,"kb_used_omap":181,"kb_used_meta":115850,"kb_avail":93624848,"statfs":{"total":95995035648,"available":95871844352,"internally_reserved":0,"allocated":4362240,"data_stored":3008853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":186131,"internal_metadata":118630637},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:13:32.829 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:13:33.034 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:13:33.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:33.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:13:33.142 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:13:33.193 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:33.225 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:13:33.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:33.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:13:33.331 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:13:33.377 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:33 smithi026 ceph-mon[68309]: pgmap v1236: 47 pgs: 47 active+clean; 769 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail; 34 KiB/s wr, 0 op/s; 56 KiB/s, 0 objects/s recovering 2024-03-20T09:13:33.387 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:33.392 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:13:33.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:33 smithi115 ceph-mon[72476]: pgmap v1236: 47 pgs: 47 active+clean; 769 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail; 34 KiB/s wr, 0 op/s; 56 KiB/s, 0 objects/s recovering 2024-03-20T09:13:33.419 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:13:33.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:33.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:13:33.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:33 smithi122 ceph-mon[73495]: pgmap v1236: 47 pgs: 47 active+clean; 769 KiB data, 2.6 GiB used, 891 GiB / 894 GiB avail; 34 KiB/s wr, 0 op/s; 56 KiB/s, 0 objects/s recovering 2024-03-20T09:13:33.523 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:13:33.567 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:33.702 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 2 16 7112 7096 221.735 191.625 0.00227069 0.00449871 2024-03-20T09:13:33.714 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:13:33.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:33.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:13:33.769 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:13:33.769 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:13:33.819 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:13:33.870 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:34.165 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:13:34.230 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:13:34.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:34 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2718958708' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:13:34.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:34 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:13:34.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:34 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2718958708' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:13:34.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:34 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:13:34.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:34 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2718958708' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:13:34.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:34 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:13:34.527 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-20T09:13:34.702 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 3 16 10233 10217 212.833 195.062 0.00151425 0.00456002 2024-03-20T09:13:34.829 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:13:35.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:35 smithi115 ceph-mon[72476]: pgmap v1238: 47 pgs: 47 active+clean; 8.9 MiB data, 2.6 GiB used, 891 GiB / 894 GiB avail; 1.4 MiB/s wr, 22 op/s; 53 KiB/s, 0 objects/s recovering 2024-03-20T09:13:35.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:35 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:13:35.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:35 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:13:35.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:35 smithi115 ceph-mon[72476]: osdmap e530: 12 total, 12 up, 10 in 2024-03-20T09:13:35.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:35 smithi122 ceph-mon[73495]: pgmap v1238: 47 pgs: 47 active+clean; 8.9 MiB data, 2.6 GiB used, 891 GiB / 894 GiB avail; 1.4 MiB/s wr, 22 op/s; 53 KiB/s, 0 objects/s recovering 2024-03-20T09:13:35.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:35 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:13:35.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:35 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:13:35.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:35 smithi122 ceph-mon[73495]: osdmap e530: 12 total, 12 up, 10 in 2024-03-20T09:13:35.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:35 smithi026 ceph-mon[68309]: pgmap v1238: 47 pgs: 47 active+clean; 8.9 MiB data, 2.6 GiB used, 891 GiB / 894 GiB avail; 1.4 MiB/s wr, 22 op/s; 53 KiB/s, 0 objects/s recovering 2024-03-20T09:13:35.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:35 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:13:35.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:35 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:13:35.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:35 smithi026 ceph-mon[68309]: osdmap e530: 12 total, 12 up, 10 in 2024-03-20T09:13:35.702 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 4 16 13363 13347 208.526 195.625 0.00216895 0.00478854 2024-03-20T09:13:36.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:36 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-03-20T09:13:36.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:36 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-03-20T09:13:36.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:36 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-03-20T09:13:36.702 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 5 16 15171 15155 189.419 113 0.0017325 0.00474349 2024-03-20T09:13:37.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:37 smithi115 ceph-mon[72476]: pgmap v1240: 47 pgs: 47 active+clean; 583 MiB data, 3.9 GiB used, 890 GiB / 894 GiB avail; 97 MiB/s wr, 1.55k op/s; 53 KiB/s, 0 objects/s recovering 2024-03-20T09:13:37.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:37 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-03-20T09:13:37.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:37 smithi115 ceph-mon[72476]: osdmap e531: 12 total, 12 up, 10 in 2024-03-20T09:13:37.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:37 smithi115 ceph-mon[72476]: osdmap e532: 12 total, 12 up, 10 in 2024-03-20T09:13:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:37 smithi122 ceph-mon[73495]: pgmap v1240: 47 pgs: 47 active+clean; 583 MiB data, 3.9 GiB used, 890 GiB / 894 GiB avail; 97 MiB/s wr, 1.55k op/s; 53 KiB/s, 0 objects/s recovering 2024-03-20T09:13:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:37 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-03-20T09:13:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:37 smithi122 ceph-mon[73495]: osdmap e531: 12 total, 12 up, 10 in 2024-03-20T09:13:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:37 smithi122 ceph-mon[73495]: osdmap e532: 12 total, 12 up, 10 in 2024-03-20T09:13:37.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:37 smithi026 ceph-mon[68309]: pgmap v1240: 47 pgs: 47 active+clean; 583 MiB data, 3.9 GiB used, 890 GiB / 894 GiB avail; 97 MiB/s wr, 1.55k op/s; 53 KiB/s, 0 objects/s recovering 2024-03-20T09:13:37.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:37 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-03-20T09:13:37.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:37 smithi026 ceph-mon[68309]: osdmap e531: 12 total, 12 up, 10 in 2024-03-20T09:13:37.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:37 smithi026 ceph-mon[68309]: osdmap e532: 12 total, 12 up, 10 in 2024-03-20T09:13:37.702 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 6 16 15171 15155 157.848 0 - 0.00474349 2024-03-20T09:13:38.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:38 smithi115 ceph-mon[72476]: 1.1e scrub starts 2024-03-20T09:13:38.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:38 smithi115 ceph-mon[72476]: 1.1e scrub ok 2024-03-20T09:13:38.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:38 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:13:38.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:38 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:13:38.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:38 smithi115 ceph-mon[72476]: 1.d scrub starts 2024-03-20T09:13:38.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:38 smithi115 ceph-mon[72476]: 1.d scrub ok 2024-03-20T09:13:38.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:38 smithi115 ceph-mon[72476]: osdmap e533: 12 total, 12 up, 10 in 2024-03-20T09:13:38.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:38 smithi122 ceph-mon[73495]: 1.1e scrub starts 2024-03-20T09:13:38.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:38 smithi122 ceph-mon[73495]: 1.1e scrub ok 2024-03-20T09:13:38.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:38 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:13:38.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:38 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:13:38.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:38 smithi122 ceph-mon[73495]: 1.d scrub starts 2024-03-20T09:13:38.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:38 smithi122 ceph-mon[73495]: 1.d scrub ok 2024-03-20T09:13:38.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:38 smithi122 ceph-mon[73495]: osdmap e533: 12 total, 12 up, 10 in 2024-03-20T09:13:38.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:38 smithi026 ceph-mon[68309]: 1.1e scrub starts 2024-03-20T09:13:38.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:38 smithi026 ceph-mon[68309]: 1.1e scrub ok 2024-03-20T09:13:38.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:38 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:13:38.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:38 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:13:38.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:38 smithi026 ceph-mon[68309]: 1.d scrub starts 2024-03-20T09:13:38.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:38 smithi026 ceph-mon[68309]: 1.d scrub ok 2024-03-20T09:13:38.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:38 smithi026 ceph-mon[68309]: osdmap e533: 12 total, 12 up, 10 in 2024-03-20T09:13:38.472 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:13:38.472 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:13:38.511 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:13:38.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:38.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:13:38.674 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:13:38.702 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 7 16 16185 16169 144.351 31.6875 0.00214185 0.00692247 2024-03-20T09:13:38.711 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:38.743 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:13:38.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:38.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:13:38.848 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:13:38.922 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:39.158 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:39 smithi026 ceph-mon[68309]: pgmap v1243: 47 pgs: 47 active+clean; 751 MiB data, 4.8 GiB used, 889 GiB / 894 GiB avail; 131 MiB/s wr, 2.10k op/s 2024-03-20T09:13:39.158 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:39 smithi026 ceph-mon[68309]: 1.8 scrub starts 2024-03-20T09:13:39.158 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:39 smithi026 ceph-mon[68309]: 1.8 scrub ok 2024-03-20T09:13:39.284 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_ops_in_flight 2024-03-20T09:13:39.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:39 smithi122 ceph-mon[73495]: pgmap v1243: 47 pgs: 47 active+clean; 751 MiB data, 4.8 GiB used, 889 GiB / 894 GiB avail; 131 MiB/s wr, 2.10k op/s 2024-03-20T09:13:39.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:39 smithi122 ceph-mon[73495]: 1.8 scrub starts 2024-03-20T09:13:39.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:39 smithi122 ceph-mon[73495]: 1.8 scrub ok 2024-03-20T09:13:39.655 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:13:39.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:39 smithi115 ceph-mon[72476]: pgmap v1243: 47 pgs: 47 active+clean; 751 MiB data, 4.8 GiB used, 889 GiB / 894 GiB avail; 131 MiB/s wr, 2.10k op/s 2024-03-20T09:13:39.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:39 smithi115 ceph-mon[72476]: 1.8 scrub starts 2024-03-20T09:13:39.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:39 smithi115 ceph-mon[72476]: 1.8 scrub ok 2024-03-20T09:13:39.702 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 8 16 18606 18590 145.22 151.312 0.00761987 0.00688056 2024-03-20T09:13:40.161 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:40 smithi026 ceph-mon[68309]: 1.11 scrub starts 2024-03-20T09:13:40.161 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:40 smithi026 ceph-mon[68309]: 1.11 scrub ok 2024-03-20T09:13:40.161 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:40 smithi026 ceph-mon[68309]: 1.18 scrub starts 2024-03-20T09:13:40.162 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:40 smithi026 ceph-mon[68309]: 1.18 scrub ok 2024-03-20T09:13:40.162 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:40 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2231843913' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:13:40.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:40 smithi122 ceph-mon[73495]: 1.11 scrub starts 2024-03-20T09:13:40.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:40 smithi122 ceph-mon[73495]: 1.11 scrub ok 2024-03-20T09:13:40.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:40 smithi122 ceph-mon[73495]: 1.18 scrub starts 2024-03-20T09:13:40.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:40 smithi122 ceph-mon[73495]: 1.18 scrub ok 2024-03-20T09:13:40.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:40 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2231843913' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:13:40.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:40 smithi115 ceph-mon[72476]: 1.11 scrub starts 2024-03-20T09:13:40.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:40 smithi115 ceph-mon[72476]: 1.11 scrub ok 2024-03-20T09:13:40.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:40 smithi115 ceph-mon[72476]: 1.18 scrub starts 2024-03-20T09:13:40.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:40 smithi115 ceph-mon[72476]: 1.18 scrub ok 2024-03-20T09:13:40.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:40 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2231843913' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:13:40.703 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 9 16 20865 20849 144.771 141.188 0.00182029 0.00690016 2024-03-20T09:13:41.162 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:41 smithi026 ceph-mon[68309]: pgmap v1245: 47 pgs: 47 active+clean; 888 MiB data, 5.1 GiB used, 889 GiB / 894 GiB avail; 155 MiB/s wr, 2.48k op/s 2024-03-20T09:13:41.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:41 smithi122 ceph-mon[73495]: pgmap v1245: 47 pgs: 47 active+clean; 888 MiB data, 5.1 GiB used, 889 GiB / 894 GiB avail; 155 MiB/s wr, 2.48k op/s 2024-03-20T09:13:41.497 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:13:41.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:41.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:13:41.603 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:13:41.632 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:41 smithi115 ceph-mon[72476]: pgmap v1245: 47 pgs: 47 active+clean; 888 MiB data, 5.1 GiB used, 889 GiB / 894 GiB avail; 155 MiB/s wr, 2.48k op/s 2024-03-20T09:13:41.635 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:41.703 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 10 16 22965 22949 143.418 131.25 0.00188685 0.00696731 2024-03-20T09:13:41.764 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:13:41.764 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:13:41.764 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:13:41.764 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:13:41.765 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:13:41.765 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:13:41.765 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:13:41.765 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:13:41.765 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:13:41.765 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:13:41.765 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:13:41.765 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:13:41.765 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:13:41.765 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:13:41.765 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:13:41.766 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:13:41.766 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:13:41.766 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:13:41.766 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:13:41.766 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:13:41.766 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:13:41.766 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:13:41.766 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:13:41.766 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:13:41.771 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:13:41.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:41.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:13:41.926 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:13:42.032 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:42.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:42 smithi122 ceph-mon[73495]: 1.2 scrub starts 2024-03-20T09:13:42.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:42 smithi122 ceph-mon[73495]: 1.2 scrub ok 2024-03-20T09:13:42.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:42 smithi026 ceph-mon[68309]: 1.2 scrub starts 2024-03-20T09:13:42.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:42 smithi026 ceph-mon[68309]: 1.2 scrub ok 2024-03-20T09:13:42.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:42 smithi115 ceph-mon[72476]: 1.2 scrub starts 2024-03-20T09:13:42.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:42 smithi115 ceph-mon[72476]: 1.2 scrub ok 2024-03-20T09:13:42.703 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 11 16 25462 25446 144.566 156.062 0.00455254 0.00691217 2024-03-20T09:13:43.088 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:13:43.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:43.092 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:13:43.179 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:43 smithi026 ceph-mon[68309]: pgmap v1246: 47 pgs: 1 active+recovering+remapped, 46 active+clean; 1.2 GiB data, 6.2 GiB used, 888 GiB / 894 GiB avail; 106 MiB/s wr, 1.70k op/s; 751/58470 objects misplaced (1.284%); 1.8 MiB/s, 28 objects/s recovering 2024-03-20T09:13:43.194 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:13:43.231 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:43 smithi122 ceph-mon[73495]: pgmap v1246: 47 pgs: 1 active+recovering+remapped, 46 active+clean; 1.2 GiB data, 6.2 GiB used, 888 GiB / 894 GiB avail; 106 MiB/s wr, 1.70k op/s; 751/58470 objects misplaced (1.284%); 1.8 MiB/s, 28 objects/s recovering 2024-03-20T09:13:43.548 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:13:43.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:43 smithi115 ceph-mon[72476]: pgmap v1246: 47 pgs: 1 active+recovering+remapped, 46 active+clean; 1.2 GiB data, 6.2 GiB used, 888 GiB / 894 GiB avail; 106 MiB/s wr, 1.70k op/s; 751/58470 objects misplaced (1.284%); 1.8 MiB/s, 28 objects/s recovering 2024-03-20T09:13:43.703 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 12 16 27577 27561 143.534 132.188 0.00373237 0.00695743 2024-03-20T09:13:44.043 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:13:44.357 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:13:44.361 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:13:44.703 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 13 16 28714 28698 137.958 71.0625 0.00223043 0.0071721 2024-03-20T09:13:44.993 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:13:44.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:44.997 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:13:45.098 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:13:45.230 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:45.252 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:13:45.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:45.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:13:45.343 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:13:45.356 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:13:45.392 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:13:45.392 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:13:45.392 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-03-20T09:13:45.392 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:13:45.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:45 smithi115 ceph-mon[72476]: pgmap v1247: 47 pgs: 1 active+recovering+remapped, 46 active+clean; 1.2 GiB data, 6.4 GiB used, 888 GiB / 894 GiB avail; 88 MiB/s wr, 1.41k op/s; 751/60255 objects misplaced (1.246%); 1.4 MiB/s, 22 objects/s recovering 2024-03-20T09:13:45.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:45 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:13:45.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:45 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:13:45.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:45 smithi115 ceph-mon[72476]: from='client.25705 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:13:45.411 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:45 smithi026 ceph-mon[68309]: pgmap v1247: 47 pgs: 1 active+recovering+remapped, 46 active+clean; 1.2 GiB data, 6.4 GiB used, 888 GiB / 894 GiB avail; 88 MiB/s wr, 1.41k op/s; 751/60255 objects misplaced (1.246%); 1.4 MiB/s, 22 objects/s recovering 2024-03-20T09:13:45.411 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:45 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:13:45.411 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:45 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:13:45.411 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:45 smithi026 ceph-mon[68309]: from='client.25705 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:13:45.413 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:45.417 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:45 smithi122 ceph-mon[73495]: pgmap v1247: 47 pgs: 1 active+recovering+remapped, 46 active+clean; 1.2 GiB data, 6.4 GiB used, 888 GiB / 894 GiB avail; 88 MiB/s wr, 1.41k op/s; 751/60255 objects misplaced (1.246%); 1.4 MiB/s, 22 objects/s recovering 2024-03-20T09:13:45.417 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:45 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:13:45.417 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:45 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:13:45.417 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:45 smithi122 ceph-mon[73495]: from='client.25705 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:13:45.703 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 14 16 29768 29752 132.809 65.875 0.00156466 0.00727381 2024-03-20T09:13:46.082 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_blocked_ops 2024-03-20T09:13:46.165 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:13:46.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:46.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:13:46.269 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:13:46.300 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:46.319 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:13:46.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:46.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:13:46.423 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:13:46.431 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:13:46.442 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:46 smithi026 ceph-mon[68309]: 1.1b scrub starts 2024-03-20T09:13:46.442 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:46 smithi026 ceph-mon[68309]: 1.1b scrub ok 2024-03-20T09:13:46.442 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:46 smithi026 ceph-mon[68309]: osdmap e534: 12 total, 12 up, 10 in 2024-03-20T09:13:46.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:46 smithi122 ceph-mon[73495]: 1.1b scrub starts 2024-03-20T09:13:46.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:46 smithi122 ceph-mon[73495]: 1.1b scrub ok 2024-03-20T09:13:46.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:46 smithi122 ceph-mon[73495]: osdmap e534: 12 total, 12 up, 10 in 2024-03-20T09:13:46.452 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:46 smithi115 ceph-mon[72476]: 1.1b scrub starts 2024-03-20T09:13:46.452 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:46 smithi115 ceph-mon[72476]: 1.1b scrub ok 2024-03-20T09:13:46.452 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:46 smithi115 ceph-mon[72476]: osdmap e534: 12 total, 12 up, 10 in 2024-03-20T09:13:46.454 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:46.477 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:13:46.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:46.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:13:46.581 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:13:46.649 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:46.682 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:13:46.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:46.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:13:46.703 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 15 16 30640 30624 127.588 54.5 0.00681489 0.00783323 2024-03-20T09:13:46.792 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:13:46.823 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:47.172 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:13:47.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:47.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:13:47.278 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:13:47.315 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:47.356 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:13:47.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:47.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:13:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:47 smithi122 ceph-mon[73495]: pgmap v1249: 47 pgs: 1 active+recovering+remapped, 46 active+clean; 1.7 GiB data, 7.6 GiB used, 886 GiB / 894 GiB avail; 121 MiB/s wr, 1.94k op/s; 751/82491 objects misplaced (0.910%); 1.3 MiB/s, 21 objects/s recovering 2024-03-20T09:13:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:47 smithi122 ceph-mon[73495]: osdmap e535: 12 total, 12 up, 10 in 2024-03-20T09:13:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:47 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:13:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:47 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:13:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:47 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:13:47.460 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:13:47.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:47 smithi026 ceph-mon[68309]: pgmap v1249: 47 pgs: 1 active+recovering+remapped, 46 active+clean; 1.7 GiB data, 7.6 GiB used, 886 GiB / 894 GiB avail; 121 MiB/s wr, 1.94k op/s; 751/82491 objects misplaced (0.910%); 1.3 MiB/s, 21 objects/s recovering 2024-03-20T09:13:47.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:47 smithi026 ceph-mon[68309]: osdmap e535: 12 total, 12 up, 10 in 2024-03-20T09:13:47.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:47 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:13:47.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:47 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:13:47.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:47 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:13:47.498 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:47.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:47 smithi115 ceph-mon[72476]: pgmap v1249: 47 pgs: 1 active+recovering+remapped, 46 active+clean; 1.7 GiB data, 7.6 GiB used, 886 GiB / 894 GiB avail; 121 MiB/s wr, 1.94k op/s; 751/82491 objects misplaced (0.910%); 1.3 MiB/s, 21 objects/s recovering 2024-03-20T09:13:47.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:47 smithi115 ceph-mon[72476]: osdmap e535: 12 total, 12 up, 10 in 2024-03-20T09:13:47.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:47 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:13:47.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:47 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:13:47.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:47 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:13:47.703 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 16 16 32432 32416 126.613 112 0.00229675 0.00788709 2024-03-20T09:13:48.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:48 smithi026 ceph-mon[68309]: 1.b scrub starts 2024-03-20T09:13:48.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:48 smithi026 ceph-mon[68309]: 1.b scrub ok 2024-03-20T09:13:48.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:48 smithi026 ceph-mon[68309]: 1.1a scrub starts 2024-03-20T09:13:48.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:48 smithi026 ceph-mon[68309]: 1.1a scrub ok 2024-03-20T09:13:48.601 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:48 smithi122 ceph-mon[73495]: 1.b scrub starts 2024-03-20T09:13:48.602 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:48 smithi122 ceph-mon[73495]: 1.b scrub ok 2024-03-20T09:13:48.602 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:48 smithi122 ceph-mon[73495]: 1.1a scrub starts 2024-03-20T09:13:48.602 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:48 smithi122 ceph-mon[73495]: 1.1a scrub ok 2024-03-20T09:13:48.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:48 smithi115 ceph-mon[72476]: 1.b scrub starts 2024-03-20T09:13:48.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:48 smithi115 ceph-mon[72476]: 1.b scrub ok 2024-03-20T09:13:48.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:48 smithi115 ceph-mon[72476]: 1.1a scrub starts 2024-03-20T09:13:48.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:48 smithi115 ceph-mon[72476]: 1.1a scrub ok 2024-03-20T09:13:48.703 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 17 16 34215 34199 125.72 111.438 0.0088227 0.00794935 2024-03-20T09:13:49.051 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:13:49.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:49.057 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:13:49.158 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:13:49.194 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:49.215 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:13:49.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:49.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:13:49.294 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":1247,"stamp":"2024-03-20T09:13:43.793627+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1316749856,"num_objects":20085,"num_object_clones":0,"num_object_copies":60255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":751,"num_objects_unfound":0,"num_objects_dirty":20085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20083,"num_write_kb":1285312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":172,"num_bytes_recovered":11534336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24330,"ondisk_log_size":24330,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":6732104,"kb_used_data":3968432,"kb_used_omap":195,"kb_used_meta":2763388,"kb_avail":930719416,"statfs":{"total":959950356480,"available":953056681984,"internally_reserved":0,"allocated":4063674368,"data_stored":4049808922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":200482,"internal_metadata":2829709534},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":705757184,"num_objects":10769,"num_object_clones":0,"num_object_copies":32307,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":751,"num_objects_unfound":0,"num_objects_dirty":10769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10769,"num_write_kb":689216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":11206656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.659187"},"pg_stats":[{"pgid":"1.1e","version":"20'137","reported_seq":441,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:38.081711+0000","last_change":"2024-03-20T09:13:36.531143+0000","last_active":"2024-03-20T09:13:38.081711+0000","last_peered":"2024-03-20T09:13:38.081711+0000","last_clean":"2024-03-20T09:13:38.081711+0000","last_became_active":"2024-03-20T09:13:20.071655+0000","last_became_peered":"2024-03-20T09:13:20.071655+0000","last_unstale":"2024-03-20T09:13:38.081711+0000","last_undegraded":"2024-03-20T09:13:38.081711+0000","last_fullsized":"2024-03-20T09:13:38.081711+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:36.531003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:36.531003+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:44.358559+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'137","reported_seq":608,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:38.075695+0000","last_change":"2024-03-20T09:13:05.291275+0000","last_active":"2024-03-20T09:13:38.075695+0000","last_peered":"2024-03-20T09:13:38.075695+0000","last_clean":"2024-03-20T09:13:38.075695+0000","last_became_active":"2024-03-20T09:07:42.013245+0000","last_became_peered":"2024-03-20T09:07:42.013245+0000","last_unstale":"2024-03-20T09:13:38.075695+0000","last_undegraded":"2024-03-20T09:13:38.075695+0000","last_fullsized":"2024-03-20T09:13:38.075695+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:05.291136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:05.291136+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:17.362626+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"20'137","reported_seq":1050,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:38.300802+0000","last_change":"2024-03-20T09:12:42.792997+0000","last_active":"2024-03-20T09:13:38.300802+0000","last_peered":"2024-03-20T09:13:38.300802+0000","last_clean":"2024-03-20T09:13:38.300802+0000","last_became_active":"2024-03-20T09:07:41.961622+0000","last_became_peered":"2024-03-20T09:07:41.961622+0000","last_unstale":"2024-03-20T09:13:38.300802+0000","last_undegraded":"2024-03-20T09:13:38.300802+0000","last_fullsized":"2024-03-20T09:13:38.300802+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:12:42.792916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:12:42.792916+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:13:54.742647+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.19","version":"20'137","reported_seq":1119,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:38.069939+0000","last_change":"2024-03-20T09:13:11.366034+0000","last_active":"2024-03-20T09:13:38.069939+0000","last_peered":"2024-03-20T09:13:38.069939+0000","last_clean":"2024-03-20T09:13:38.069939+0000","last_became_active":"2024-03-20T09:07:41.962680+0000","last_became_peered":"2024-03-20T09:07:41.962680+0000","last_unstale":"2024-03-20T09:13:38.069939+0000","last_undegraded":"2024-03-20T09:13:38.069939+0000","last_fullsized":"2024-03-20T09:13:38.069939+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:11.365892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:11.365892+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:32.244898+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"20'137","reported_seq":581,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:39.292618+0000","last_change":"2024-03-20T09:13:39.292648+0000","last_active":"2024-03-20T09:13:39.292618+0000","last_peered":"2024-03-20T09:13:39.292618+0000","last_clean":"2024-03-20T09:13:39.292618+0000","last_became_active":"2024-03-20T09:10:35.277481+0000","last_became_peered":"2024-03-20T09:10:35.277481+0000","last_unstale":"2024-03-20T09:13:39.292618+0000","last_undegraded":"2024-03-20T09:13:39.292618+0000","last_fullsized":"2024-03-20T09:13:39.292618+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:39.292556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:39.292556+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:07.108785+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1b","version":"20'137","reported_seq":879,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:38.300707+0000","last_change":"2024-03-20T09:12:39.855212+0000","last_active":"2024-03-20T09:13:38.300707+0000","last_peered":"2024-03-20T09:13:38.300707+0000","last_clean":"2024-03-20T09:13:38.300707+0000","last_became_active":"2024-03-20T09:07:41.962733+0000","last_became_peered":"2024-03-20T09:07:41.962733+0000","last_unstale":"2024-03-20T09:13:38.300707+0000","last_undegraded":"2024-03-20T09:13:38.300707+0000","last_fullsized":"2024-03-20T09:13:38.300707+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:12:39.855105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:12:39.855105+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:13:44.240877+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"533'1245","reported_seq":1281,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:40.179099+0000","last_change":"2024-03-20T09:13:22.063299+0000","last_active":"2024-03-20T09:13:40.179099+0000","last_peered":"2024-03-20T09:13:40.179099+0000","last_clean":"2024-03-20T09:13:40.179099+0000","last_became_active":"2024-03-20T09:13:22.063020+0000","last_became_peered":"2024-03-20T09:13:22.063020+0000","last_unstale":"2024-03-20T09:13:40.179099+0000","last_undegraded":"2024-03-20T09:13:40.179099+0000","last_fullsized":"2024-03-20T09:13:40.179099+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":1245,"log_dups_size":0,"ondisk_log_size":1245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:41.788247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81592320,"num_objects":1245,"num_object_clones":0,"num_object_copies":3735,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1245,"num_write_kb":79680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"20'137","reported_seq":580,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:38.313587+0000","last_change":"2024-03-20T09:13:38.313619+0000","last_active":"2024-03-20T09:13:38.313587+0000","last_peered":"2024-03-20T09:13:38.313587+0000","last_clean":"2024-03-20T09:13:38.313587+0000","last_became_active":"2024-03-20T09:10:35.215772+0000","last_became_peered":"2024-03-20T09:10:35.215772+0000","last_unstale":"2024-03-20T09:13:38.313587+0000","last_undegraded":"2024-03-20T09:13:38.313587+0000","last_fullsized":"2024-03-20T09:13:38.313587+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:38.313491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:38.313491+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:46.785027+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"533'1180","reported_seq":1216,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:39.983265+0000","last_change":"2024-03-20T09:13:22.583666+0000","last_active":"2024-03-20T09:13:39.983265+0000","last_peered":"2024-03-20T09:13:39.983265+0000","last_clean":"2024-03-20T09:13:39.983265+0000","last_became_active":"2024-03-20T09:13:22.583317+0000","last_became_peered":"2024-03-20T09:13:22.583317+0000","last_unstale":"2024-03-20T09:13:39.983265+0000","last_undegraded":"2024-03-20T09:13:39.983265+0000","last_fullsized":"2024-03-20T09:13:39.983265+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":1180,"log_dups_size":0,"ondisk_log_size":1180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:29.205626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77332480,"num_objects":1180,"num_object_clones":0,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1180,"num_write_kb":75520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,8],"acting":[3,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"20'137","reported_seq":1132,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:38.300346+0000","last_change":"2024-03-20T09:13:05.129380+0000","last_active":"2024-03-20T09:13:38.300346+0000","last_peered":"2024-03-20T09:13:38.300346+0000","last_clean":"2024-03-20T09:13:38.300346+0000","last_became_active":"2024-03-20T09:10:34.099414+0000","last_became_peered":"2024-03-20T09:10:34.099414+0000","last_unstale":"2024-03-20T09:13:38.300346+0000","last_undegraded":"2024-03-20T09:13:38.300346+0000","last_fullsized":"2024-03-20T09:13:38.300346+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:05.129288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:05.129288+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:16.260612+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"533'1218","reported_seq":1600,"reported_epoch":533,"state":"active+recovering+remapped","last_fresh":"2024-03-20T09:13:40.180919+0000","last_change":"2024-03-20T09:13:38.342293+0000","last_active":"2024-03-20T09:13:40.180919+0000","last_peered":"2024-03-20T09:13:40.180919+0000","last_clean":"2024-03-20T09:13:36.131200+0000","last_became_active":"2024-03-20T09:13:38.328883+0000","last_became_peered":"2024-03-20T09:13:38.328883+0000","last_unstale":"2024-03-20T09:13:40.180919+0000","last_undegraded":"2024-03-20T09:13:40.180919+0000","last_fullsized":"2024-03-20T09:13:40.180919+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":1218,"log_dups_size":0,"ondisk_log_size":1218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79822848,"num_objects":1218,"num_object_clones":0,"num_object_copies":3654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":751,"num_objects_unfound":0,"num_objects_dirty":1218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1218,"num_write_kb":77952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":11206656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,11],"acting":[1,3,11],"avail_no_missing":["1","3","11"],"object_location_counts":[{"shards":"1,3,11","objects":467}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"20'137","reported_seq":937,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:38.300530+0000","last_change":"2024-03-20T09:12:42.370648+0000","last_active":"2024-03-20T09:13:38.300530+0000","last_peered":"2024-03-20T09:13:38.300530+0000","last_clean":"2024-03-20T09:13:38.300530+0000","last_became_active":"2024-03-20T09:07:52.618107+0000","last_became_peered":"2024-03-20T09:07:52.618107+0000","last_unstale":"2024-03-20T09:13:38.300530+0000","last_undegraded":"2024-03-20T09:13:38.300530+0000","last_fullsized":"2024-03-20T09:13:38.300530+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:12:42.370551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:12:42.370551+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:13:56.545080+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"533'1223","reported_seq":1259,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:39.757095+0000","last_change":"2024-03-20T09:13:22.064363+0000","last_active":"2024-03-20T09:13:39.757095+0000","last_peered":"2024-03-20T09:13:39.757095+0000","last_clean":"2024-03-20T09:13:39.757095+0000","last_became_active":"2024-03-20T09:13:22.063975+0000","last_became_peered":"2024-03-20T09:13:22.063975+0000","last_unstale":"2024-03-20T09:13:39.757095+0000","last_undegraded":"2024-03-20T09:13:39.757095+0000","last_fullsized":"2024-03-20T09:13:39.757095+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":1223,"log_dups_size":0,"ondisk_log_size":1223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:45.622073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80150528,"num_objects":1223,"num_object_clones":0,"num_object_copies":3669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1223,"num_write_kb":78272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"20'137","reported_seq":1101,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:38.081621+0000","last_change":"2024-03-20T09:13:14.395564+0000","last_active":"2024-03-20T09:13:38.081621+0000","last_peered":"2024-03-20T09:13:38.081621+0000","last_clean":"2024-03-20T09:13:38.081621+0000","last_became_active":"2024-03-20T09:12:14.306384+0000","last_became_peered":"2024-03-20T09:12:14.306384+0000","last_unstale":"2024-03-20T09:13:38.081621+0000","last_undegraded":"2024-03-20T09:13:38.081621+0000","last_fullsized":"2024-03-20T09:13:38.081621+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:14.395469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:14.395469+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:24.496429+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"533'1175","reported_seq":1207,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:39.980637+0000","last_change":"2024-03-20T09:13:24.064404+0000","last_active":"2024-03-20T09:13:39.980637+0000","last_peered":"2024-03-20T09:13:39.980637+0000","last_clean":"2024-03-20T09:13:39.980637+0000","last_became_active":"2024-03-20T09:13:24.064091+0000","last_became_peered":"2024-03-20T09:13:24.064091+0000","last_unstale":"2024-03-20T09:13:39.980637+0000","last_undegraded":"2024-03-20T09:13:39.980637+0000","last_fullsized":"2024-03-20T09:13:39.980637+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":1175,"log_dups_size":0,"ondisk_log_size":1175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:33.687560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77004800,"num_objects":1175,"num_object_clones":0,"num_object_copies":3525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1175,"num_write_kb":75200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"20'137","reported_seq":997,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:38.070103+0000","last_change":"2024-03-20T09:13:10.343309+0000","last_active":"2024-03-20T09:13:38.070103+0000","last_peered":"2024-03-20T09:13:38.070103+0000","last_clean":"2024-03-20T09:13:38.070103+0000","last_became_active":"2024-03-20T09:07:37.003722+0000","last_became_peered":"2024-03-20T09:07:37.003722+0000","last_unstale":"2024-03-20T09:13:38.070103+0000","last_undegraded":"2024-03-20T09:13:38.070103+0000","last_fullsized":"2024-03-20T09:13:38.070103+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:10.343198+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:07:06.480160+0000","last_clean_scrub_stamp":"2024-03-20T09:13:10.343198+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:33.639911+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.3","version":"533'1159","reported_seq":1195,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:39.735991+0000","last_change":"2024-03-20T09:13:22.064621+0000","last_active":"2024-03-20T09:13:39.735991+0000","last_peered":"2024-03-20T09:13:39.735991+0000","last_clean":"2024-03-20T09:13:39.735991+0000","last_became_active":"2024-03-20T09:13:22.064275+0000","last_became_peered":"2024-03-20T09:13:22.064275+0000","last_unstale":"2024-03-20T09:13:39.735991+0000","last_undegraded":"2024-03-20T09:13:39.735991+0000","last_fullsized":"2024-03-20T09:13:39.735991+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":1159,"log_dups_size":0,"ondisk_log_size":1159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:40.477405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75956224,"num_objects":1159,"num_object_clones":0,"num_object_copies":3477,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1159,"num_write_kb":74176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"20'137","reported_seq":1014,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:38.070094+0000","last_change":"2024-03-20T09:13:12.359759+0000","last_active":"2024-03-20T09:13:38.070094+0000","last_peered":"2024-03-20T09:13:38.070094+0000","last_clean":"2024-03-20T09:13:38.070094+0000","last_became_active":"2024-03-20T09:07:52.593671+0000","last_became_peered":"2024-03-20T09:07:52.593671+0000","last_unstale":"2024-03-20T09:13:38.070094+0000","last_undegraded":"2024-03-20T09:13:38.070094+0000","last_fullsized":"2024-03-20T09:13:38.070094+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:12.359617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:12.359617+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:24.685959+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.6","version":"533'1223","reported_seq":1259,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:40.175207+0000","last_change":"2024-03-20T09:13:22.067664+0000","last_active":"2024-03-20T09:13:40.175207+0000","last_peered":"2024-03-20T09:13:40.175207+0000","last_clean":"2024-03-20T09:13:40.175207+0000","last_became_active":"2024-03-20T09:13:22.067362+0000","last_became_peered":"2024-03-20T09:13:22.067362+0000","last_unstale":"2024-03-20T09:13:40.175207+0000","last_undegraded":"2024-03-20T09:13:40.175207+0000","last_fullsized":"2024-03-20T09:13:40.175207+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":1223,"log_dups_size":0,"ondisk_log_size":1223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:46.912906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80150528,"num_objects":1223,"num_object_clones":0,"num_object_copies":3669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1223,"num_write_kb":78272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"20'137","reported_seq":1006,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:38.075484+0000","last_change":"2024-03-20T09:12:02.221970+0000","last_active":"2024-03-20T09:13:38.075484+0000","last_peered":"2024-03-20T09:13:38.075484+0000","last_clean":"2024-03-20T09:13:38.075484+0000","last_became_active":"2024-03-20T09:12:02.221686+0000","last_became_peered":"2024-03-20T09:12:02.221686+0000","last_unstale":"2024-03-20T09:13:38.075484+0000","last_undegraded":"2024-03-20T09:13:38.075484+0000","last_fullsized":"2024-03-20T09:13:38.075484+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:10:18.788771+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:18.788771+0000","last_clean_scrub_stamp":"2024-03-20T09:10:18.788771+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,8],"acting":[1,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"533'1291","reported_seq":1327,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:40.684375+0000","last_change":"2024-03-20T09:13:22.063240+0000","last_active":"2024-03-20T09:13:40.684375+0000","last_peered":"2024-03-20T09:13:40.684375+0000","last_clean":"2024-03-20T09:13:40.684375+0000","last_became_active":"2024-03-20T09:13:22.062992+0000","last_became_peered":"2024-03-20T09:13:22.062992+0000","last_unstale":"2024-03-20T09:13:40.684375+0000","last_undegraded":"2024-03-20T09:13:40.684375+0000","last_fullsized":"2024-03-20T09:13:40.684375+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":1291,"log_dups_size":0,"ondisk_log_size":1291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:36.289001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84606976,"num_objects":1291,"num_object_clones":0,"num_object_copies":3873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1291,"num_write_kb":82624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":1072,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:38.081371+0000","last_change":"2024-03-20T09:13:03.215035+0000","last_active":"2024-03-20T09:13:38.081371+0000","last_peered":"2024-03-20T09:13:38.081371+0000","last_clean":"2024-03-20T09:13:38.081371+0000","last_became_active":"2024-03-20T09:12:14.305989+0000","last_became_peered":"2024-03-20T09:12:14.305989+0000","last_unstale":"2024-03-20T09:13:38.081371+0000","last_undegraded":"2024-03-20T09:13:38.081371+0000","last_fullsized":"2024-03-20T09:13:38.081371+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:03.214902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:03.214902+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:13.441726+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.7","version":"533'1231","reported_seq":1266,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:40.181221+0000","last_change":"2024-03-20T09:13:22.066405+0000","last_active":"2024-03-20T09:13:40.181221+0000","last_peered":"2024-03-20T09:13:40.181221+0000","last_clean":"2024-03-20T09:13:40.181221+0000","last_became_active":"2024-03-20T09:13:22.065656+0000","last_became_peered":"2024-03-20T09:13:22.065656+0000","last_unstale":"2024-03-20T09:13:40.181221+0000","last_undegraded":"2024-03-20T09:13:40.181221+0000","last_fullsized":"2024-03-20T09:13:40.181221+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":1231,"log_dups_size":0,"ondisk_log_size":1231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:23.500958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80674816,"num_objects":1231,"num_object_clones":0,"num_object_copies":3693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1231,"num_write_kb":78784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,0],"acting":[1,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"20'137","reported_seq":1322,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:40.818613+0000","last_change":"2024-03-20T09:13:40.818653+0000","last_active":"2024-03-20T09:13:40.818613+0000","last_peered":"2024-03-20T09:13:40.818613+0000","last_clean":"2024-03-20T09:13:40.818613+0000","last_became_active":"2024-03-20T09:12:14.313541+0000","last_became_peered":"2024-03-20T09:12:14.313541+0000","last_unstale":"2024-03-20T09:13:40.818613+0000","last_undegraded":"2024-03-20T09:13:40.818613+0000","last_fullsized":"2024-03-20T09:13:40.818613+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:40.818542+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:19.513511+0000","last_clean_scrub_stamp":"2024-03-20T09:13:40.818542+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:01.397421+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"533'1194","reported_seq":1234,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:40.175260+0000","last_change":"2024-03-20T09:13:24.443379+0000","last_active":"2024-03-20T09:13:40.175260+0000","last_peered":"2024-03-20T09:13:40.175260+0000","last_clean":"2024-03-20T09:13:40.175260+0000","last_became_active":"2024-03-20T09:13:24.443054+0000","last_became_peered":"2024-03-20T09:13:24.443054+0000","last_unstale":"2024-03-20T09:13:40.175260+0000","last_undegraded":"2024-03-20T09:13:40.175260+0000","last_fullsized":"2024-03-20T09:13:40.175260+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":1194,"log_dups_size":0,"ondisk_log_size":1194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:41.617061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78249984,"num_objects":1194,"num_object_clones":0,"num_object_copies":3582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1194,"num_write_kb":76416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"20'137","reported_seq":1018,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:38.070040+0000","last_change":"2024-03-20T09:13:14.428508+0000","last_active":"2024-03-20T09:13:38.070040+0000","last_peered":"2024-03-20T09:13:38.070040+0000","last_clean":"2024-03-20T09:13:38.070040+0000","last_became_active":"2024-03-20T09:06:02.923630+0000","last_became_peered":"2024-03-20T09:06:02.923630+0000","last_unstale":"2024-03-20T09:13:38.070040+0000","last_undegraded":"2024-03-20T09:13:38.070040+0000","last_fullsized":"2024-03-20T09:13:38.070040+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:14.428416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:14.428416+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:27.674669+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2],"acting":[10,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2","version":"533'1501","reported_seq":1537,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:42.136089+0000","last_change":"2024-03-20T09:13:22.584008+0000","last_active":"2024-03-20T09:13:42.136089+0000","last_peered":"2024-03-20T09:13:42.136089+0000","last_clean":"2024-03-20T09:13:42.136089+0000","last_became_active":"2024-03-20T09:13:22.583529+0000","last_became_peered":"2024-03-20T09:13:22.583529+0000","last_unstale":"2024-03-20T09:13:42.136089+0000","last_undegraded":"2024-03-20T09:13:42.136089+0000","last_fullsized":"2024-03-20T09:13:42.136089+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":1501,"log_dups_size":0,"ondisk_log_size":1501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:36.926945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98369536,"num_objects":1501,"num_object_clones":0,"num_object_copies":4503,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1501,"num_write_kb":96064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,8],"acting":[5,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"20'137","reported_seq":1010,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:38.300692+0000","last_change":"2024-03-20T09:13:27.088031+0000","last_active":"2024-03-20T09:13:38.300692+0000","last_peered":"2024-03-20T09:13:38.300692+0000","last_clean":"2024-03-20T09:13:38.300692+0000","last_became_active":"2024-03-20T09:13:27.087760+0000","last_became_peered":"2024-03-20T09:13:27.087760+0000","last_unstale":"2024-03-20T09:13:38.300692+0000","last_undegraded":"2024-03-20T09:13:38.300692+0000","last_fullsized":"2024-03-20T09:13:38.300692+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:12:43.355431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:12:43.355431+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:10.456031+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"533'1370","reported_seq":1406,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:41.168175+0000","last_change":"2024-03-20T09:13:22.064771+0000","last_active":"2024-03-20T09:13:41.168175+0000","last_peered":"2024-03-20T09:13:41.168175+0000","last_clean":"2024-03-20T09:13:41.168175+0000","last_became_active":"2024-03-20T09:13:22.064461+0000","last_became_peered":"2024-03-20T09:13:22.064461+0000","last_unstale":"2024-03-20T09:13:41.168175+0000","last_undegraded":"2024-03-20T09:13:41.168175+0000","last_fullsized":"2024-03-20T09:13:41.168175+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":1370,"log_dups_size":0,"ondisk_log_size":1370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:25.759321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89784320,"num_objects":1370,"num_object_clones":0,"num_object_copies":4110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1370,"num_write_kb":87680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,1],"acting":[10,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.b","version":"20'137","reported_seq":878,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:38.081547+0000","last_change":"2024-03-20T09:13:20.070915+0000","last_active":"2024-03-20T09:13:38.081547+0000","last_peered":"2024-03-20T09:13:38.081547+0000","last_clean":"2024-03-20T09:13:38.081547+0000","last_became_active":"2024-03-20T09:13:20.070596+0000","last_became_peered":"2024-03-20T09:13:20.070596+0000","last_unstale":"2024-03-20T09:13:38.081547+0000","last_undegraded":"2024-03-20T09:13:38.081547+0000","last_fullsized":"2024-03-20T09:13:38.081547+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:10:21.687117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:10:21.687117+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:13:38.534178+0000","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.b","version":"533'1418","reported_seq":1454,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:41.708099+0000","last_change":"2024-03-20T09:13:22.065041+0000","last_active":"2024-03-20T09:13:41.708099+0000","last_peered":"2024-03-20T09:13:41.708099+0000","last_clean":"2024-03-20T09:13:41.708099+0000","last_became_active":"2024-03-20T09:13:22.064687+0000","last_became_peered":"2024-03-20T09:13:22.064687+0000","last_unstale":"2024-03-20T09:13:41.708099+0000","last_undegraded":"2024-03-20T09:13:41.708099+0000","last_fullsized":"2024-03-20T09:13:41.708099+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":1418,"log_dups_size":0,"ondisk_log_size":1418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:35.394465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92930048,"num_objects":1418,"num_object_clones":0,"num_object_copies":4254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1418,"num_write_kb":90752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"20'137","reported_seq":538,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:38.072440+0000","last_change":"2024-03-20T09:13:05.598138+0000","last_active":"2024-03-20T09:13:38.072440+0000","last_peered":"2024-03-20T09:13:38.072440+0000","last_clean":"2024-03-20T09:13:38.072440+0000","last_became_active":"2024-03-20T09:12:14.311319+0000","last_became_peered":"2024-03-20T09:12:14.311319+0000","last_unstale":"2024-03-20T09:13:38.072440+0000","last_undegraded":"2024-03-20T09:13:38.072440+0000","last_fullsized":"2024-03-20T09:13:38.072440+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:05.598028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:05.598028+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:07.032921+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"533'1244","reported_seq":1280,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:40.666595+0000","last_change":"2024-03-20T09:13:22.064279+0000","last_active":"2024-03-20T09:13:40.666595+0000","last_peered":"2024-03-20T09:13:40.666595+0000","last_clean":"2024-03-20T09:13:40.666595+0000","last_became_active":"2024-03-20T09:13:22.063927+0000","last_became_peered":"2024-03-20T09:13:22.063927+0000","last_unstale":"2024-03-20T09:13:40.666595+0000","last_undegraded":"2024-03-20T09:13:40.666595+0000","last_fullsized":"2024-03-20T09:13:40.666595+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":1244,"log_dups_size":0,"ondisk_log_size":1244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:37.913884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81526784,"num_objects":1244,"num_object_clones":0,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1244,"num_write_kb":79616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"20'137","reported_seq":1023,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:38.300660+0000","last_change":"2024-03-20T09:12:40.360273+0000","last_active":"2024-03-20T09:13:38.300660+0000","last_peered":"2024-03-20T09:13:38.300660+0000","last_clean":"2024-03-20T09:13:38.300660+0000","last_became_active":"2024-03-20T09:10:35.119706+0000","last_became_peered":"2024-03-20T09:10:35.119706+0000","last_unstale":"2024-03-20T09:13:38.300660+0000","last_undegraded":"2024-03-20T09:13:38.300660+0000","last_fullsized":"2024-03-20T09:13:38.300660+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:12:40.360146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:12:40.360146+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:13:50.360030+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"533'1171","reported_seq":1211,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:39.749813+0000","last_change":"2024-03-20T09:13:24.065781+0000","last_active":"2024-03-20T09:13:39.749813+0000","last_peered":"2024-03-20T09:13:39.749813+0000","last_clean":"2024-03-20T09:13:39.749813+0000","last_became_active":"2024-03-20T09:13:24.065349+0000","last_became_peered":"2024-03-20T09:13:24.065349+0000","last_unstale":"2024-03-20T09:13:39.749813+0000","last_undegraded":"2024-03-20T09:13:39.749813+0000","last_fullsized":"2024-03-20T09:13:39.749813+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":1171,"log_dups_size":0,"ondisk_log_size":1171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:29.930666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76742656,"num_objects":1171,"num_object_clones":0,"num_object_copies":3513,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1171,"num_write_kb":74944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"20'137","reported_seq":1053,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:38.081182+0000","last_change":"2024-03-20T09:13:27.088076+0000","last_active":"2024-03-20T09:13:38.081182+0000","last_peered":"2024-03-20T09:13:38.081182+0000","last_clean":"2024-03-20T09:13:38.081182+0000","last_became_active":"2024-03-20T09:13:27.087686+0000","last_became_peered":"2024-03-20T09:13:27.087686+0000","last_unstale":"2024-03-20T09:13:38.081182+0000","last_undegraded":"2024-03-20T09:13:38.081182+0000","last_fullsized":"2024-03-20T09:13:38.081182+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:09.755064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:09.755064+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:33.274547+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"533'1240","reported_seq":1276,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:39.744978+0000","last_change":"2024-03-20T09:13:22.064047+0000","last_active":"2024-03-20T09:13:39.744978+0000","last_peered":"2024-03-20T09:13:39.744978+0000","last_clean":"2024-03-20T09:13:39.744978+0000","last_became_active":"2024-03-20T09:13:22.063673+0000","last_became_peered":"2024-03-20T09:13:22.063673+0000","last_unstale":"2024-03-20T09:13:39.744978+0000","last_undegraded":"2024-03-20T09:13:39.744978+0000","last_fullsized":"2024-03-20T09:13:39.744978+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":1240,"log_dups_size":0,"ondisk_log_size":1240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:39.125308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81264640,"num_objects":1240,"num_object_clones":0,"num_object_copies":3720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1240,"num_write_kb":79360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"20'137","reported_seq":605,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:38.075611+0000","last_change":"2024-03-20T09:13:38.035423+0000","last_active":"2024-03-20T09:13:38.075611+0000","last_peered":"2024-03-20T09:13:38.075611+0000","last_clean":"2024-03-20T09:13:38.075611+0000","last_became_active":"2024-03-20T09:12:14.307864+0000","last_became_peered":"2024-03-20T09:12:14.307864+0000","last_unstale":"2024-03-20T09:13:38.075611+0000","last_undegraded":"2024-03-20T09:13:38.075611+0000","last_fullsized":"2024-03-20T09:13:38.075611+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:38.035304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:38.035304+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:57.965009+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"20'137","reported_seq":921,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:38.074683+0000","last_change":"2024-03-20T09:13:04.270196+0000","last_active":"2024-03-20T09:13:38.074683+0000","last_peered":"2024-03-20T09:13:38.074683+0000","last_clean":"2024-03-20T09:13:38.074683+0000","last_became_active":"2024-03-20T09:10:34.096758+0000","last_became_peered":"2024-03-20T09:10:34.096758+0000","last_unstale":"2024-03-20T09:13:38.074683+0000","last_undegraded":"2024-03-20T09:13:38.074683+0000","last_fullsized":"2024-03-20T09:13:38.074683+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:04.270076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:04.270076+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:18.468974+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"20'137","reported_seq":986,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:38.075468+0000","last_change":"2024-03-20T09:10:14.741767+0000","last_active":"2024-03-20T09:13:38.075468+0000","last_peered":"2024-03-20T09:13:38.075468+0000","last_clean":"2024-03-20T09:13:38.075468+0000","last_became_active":"2024-03-20T09:04:04.103645+0000","last_became_peered":"2024-03-20T09:04:04.103645+0000","last_unstale":"2024-03-20T09:13:38.075468+0000","last_undegraded":"2024-03-20T09:13:38.075468+0000","last_fullsized":"2024-03-20T09:13:38.075468+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:10:14.741606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:10:14.741606+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"20'137","reported_seq":447,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:38.074649+0000","last_change":"2024-03-20T09:13:05.305611+0000","last_active":"2024-03-20T09:13:38.074649+0000","last_peered":"2024-03-20T09:13:38.074649+0000","last_clean":"2024-03-20T09:13:38.074649+0000","last_became_active":"2024-03-20T09:10:34.097191+0000","last_became_peered":"2024-03-20T09:10:34.097191+0000","last_unstale":"2024-03-20T09:13:38.074649+0000","last_undegraded":"2024-03-20T09:13:38.074649+0000","last_fullsized":"2024-03-20T09:13:38.074649+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:05.305514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:05.305514+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:31.997881+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"20'137","reported_seq":858,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:38.808649+0000","last_change":"2024-03-20T09:13:38.808687+0000","last_active":"2024-03-20T09:13:38.808649+0000","last_peered":"2024-03-20T09:13:38.808649+0000","last_clean":"2024-03-20T09:13:38.808649+0000","last_became_active":"2024-03-20T09:13:27.087859+0000","last_became_peered":"2024-03-20T09:13:27.087859+0000","last_unstale":"2024-03-20T09:13:38.808649+0000","last_undegraded":"2024-03-20T09:13:38.808649+0000","last_fullsized":"2024-03-20T09:13:38.808649+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:38.808565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:38.808565+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:51.238334+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,3],"acting":[0,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"20'137","reported_seq":682,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:38.075669+0000","last_change":"2024-03-20T09:13:27.088201+0000","last_active":"2024-03-20T09:13:38.075669+0000","last_peered":"2024-03-20T09:13:38.075669+0000","last_clean":"2024-03-20T09:13:38.075669+0000","last_became_active":"2024-03-20T09:13:27.087760+0000","last_became_peered":"2024-03-20T09:13:27.087760+0000","last_unstale":"2024-03-20T09:13:38.075669+0000","last_undegraded":"2024-03-20T09:13:38.075669+0000","last_fullsized":"2024-03-20T09:13:38.075669+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:10.182025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:10.182025+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:40.149783+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"20'137","reported_seq":1001,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:38.300932+0000","last_change":"2024-03-20T09:12:36.869949+0000","last_active":"2024-03-20T09:13:38.300932+0000","last_peered":"2024-03-20T09:13:38.300932+0000","last_clean":"2024-03-20T09:13:38.300932+0000","last_became_active":"2024-03-20T09:10:35.095339+0000","last_became_peered":"2024-03-20T09:10:35.095339+0000","last_unstale":"2024-03-20T09:13:38.300932+0000","last_undegraded":"2024-03-20T09:13:38.300932+0000","last_fullsized":"2024-03-20T09:13:38.300932+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:12:36.869848+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:23.640533+0000","last_clean_scrub_stamp":"2024-03-20T09:12:36.869848+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:13:47.865605+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"20'137","reported_seq":1070,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:38.300633+0000","last_change":"2024-03-20T09:12:52.043287+0000","last_active":"2024-03-20T09:13:38.300633+0000","last_peered":"2024-03-20T09:13:38.300633+0000","last_clean":"2024-03-20T09:13:38.300633+0000","last_became_active":"2024-03-20T09:04:14.208448+0000","last_became_peered":"2024-03-20T09:04:14.208448+0000","last_unstale":"2024-03-20T09:13:38.300633+0000","last_undegraded":"2024-03-20T09:13:38.300633+0000","last_fullsized":"2024-03-20T09:13:38.300633+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:12:52.043144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:12:52.043144+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:17.347869+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"20'137","reported_seq":335,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:38.072344+0000","last_change":"2024-03-20T09:13:04.628974+0000","last_active":"2024-03-20T09:13:38.072344+0000","last_peered":"2024-03-20T09:13:38.072344+0000","last_clean":"2024-03-20T09:13:38.072344+0000","last_became_active":"2024-03-20T09:12:14.307581+0000","last_became_peered":"2024-03-20T09:12:14.307581+0000","last_unstale":"2024-03-20T09:13:38.072344+0000","last_undegraded":"2024-03-20T09:13:38.072344+0000","last_fullsized":"2024-03-20T09:13:38.072344+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:04.628862+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:07:06.480160+0000","last_clean_scrub_stamp":"2024-03-20T09:13:04.628862+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:10.875071+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"20'137","reported_seq":606,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-20T09:13:38.075299+0000","last_change":"2024-03-20T09:13:20.074838+0000","last_active":"2024-03-20T09:13:38.075299+0000","last_peered":"2024-03-20T09:13:38.075299+0000","last_clean":"2024-03-20T09:13:38.075299+0000","last_became_active":"2024-03-20T09:13:20.074439+0000","last_became_peered":"2024-03-20T09:13:20.074439+0000","last_unstale":"2024-03-20T09:13:38.075299+0000","last_undegraded":"2024-03-20T09:13:38.075299+0000","last_fullsized":"2024-03-20T09:13:38.075299+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:10:17.768353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:10:17.768353+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,11],"acting":[1,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":1316159488,"num_objects":20083,"num_object_clones":0,"num_object_copies":60249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":751,"num_objects_unfound":0,"num_objects_dirty":20083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20083,"num_write_kb":1285312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":11206656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4203610112,"data_stored":4203610112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20083,"ondisk_log_size":20083,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4247,"ondisk_log_size":4247,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860627,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":889092,"kb_used_data":591356,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":92856060,"statfs":{"total":95995035648,"available":95084605440,"internally_reserved":0,"allocated":605548544,"data_stored":604163065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":304871876},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":89,"seq":382252089561,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":583364,"kb_used_data":349500,"kb_used_omap":1,"kb_used_meta":233854,"kb_avail":93161788,"statfs":{"total":95995035648,"available":95397670912,"internally_reserved":0,"allocated":357888000,"data_stored":356502521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":239466944},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":0,"seq":343597383905,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":70,"seq":300647710949,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":415172,"kb_used_data":185468,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":93329980,"statfs":{"total":95995035648,"available":95569899520,"internally_reserved":0,"allocated":189919232,"data_stored":188533753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":235207110},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":60,"seq":257698037994,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":744588,"kb_used_data":334404,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":93000564,"statfs":{"total":95995035648,"available":95232577536,"internally_reserved":0,"allocated":342429696,"data_stored":341040665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":420018634},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332335,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":771016,"kb_used_data":496892,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":92974136,"statfs":{"total":95995035648,"available":95205515264,"internally_reserved":0,"allocated":508817408,"data_stored":507431929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":280689094},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575113,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1106696,"kb_used_data":768700,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":92638456,"statfs":{"total":95995035648,"available":94861778944,"internally_reserved":0,"allocated":787148800,"data_stored":785763321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":346094022},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738632,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":699460,"kb_used_data":343420,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93045692,"statfs":{"total":95995035648,"available":95278788608,"internally_reserved":0,"allocated":351662080,"data_stored":350276601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411587,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":482380,"kb_used_data":284612,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":93262772,"statfs":{"total":95995035648,"available":95501078528,"internally_reserved":0,"allocated":291442688,"data_stored":290053657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":202504645},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149950,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":651844,"kb_used_data":341628,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":93093308,"statfs":{"total":95995035648,"available":95327547392,"internally_reserved":0,"allocated":349827072,"data_stored":348441593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":317651399},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953724,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":443,"seq":1902670512167,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":388492,"kb_used_data":272452,"kb_used_omap":181,"kb_used_meta":115850,"kb_avail":93356660,"statfs":{"total":95995035648,"available":95597219840,"internally_reserved":0,"allocated":278990848,"data_stored":277601817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":186131,"internal_metadata":118630637},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":377880576,"data_stored":377880576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":796721152,"data_stored":796721152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":309460992,"data_stored":309460992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":388562944,"data_stored":388562944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":298516480,"data_stored":298516480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":515244032,"data_stored":515244032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":351862784,"data_stored":351862784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":187957248,"data_stored":187957248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":354091008,"data_stored":354091008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":623312896,"data_stored":623312896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:13:49.296 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph status --format=json 2024-03-20T09:13:49.319 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:13:49.322 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:13:49.349 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:49.358 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:49 smithi122 ceph-mon[73495]: pgmap v1251: 47 pgs: 1 active+recovering+remapped, 46 active+clean; 1.8 GiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 115 MiB/s wr, 1.83k op/s; 751/86643 objects misplaced (0.867%); 1.3 MiB/s, 21 objects/s recovering 2024-03-20T09:13:49.358 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:49 smithi122 ceph-mon[73495]: 1.3 scrub starts 2024-03-20T09:13:49.358 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:49 smithi122 ceph-mon[73495]: 1.3 scrub ok 2024-03-20T09:13:49.371 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:13:49.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:49.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:13:49.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:49 smithi026 ceph-mon[68309]: pgmap v1251: 47 pgs: 1 active+recovering+remapped, 46 active+clean; 1.8 GiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 115 MiB/s wr, 1.83k op/s; 751/86643 objects misplaced (0.867%); 1.3 MiB/s, 21 objects/s recovering 2024-03-20T09:13:49.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:49 smithi026 ceph-mon[68309]: 1.3 scrub starts 2024-03-20T09:13:49.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:49 smithi026 ceph-mon[68309]: 1.3 scrub ok 2024-03-20T09:13:49.478 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:13:49.508 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:49 smithi115 ceph-mon[72476]: pgmap v1251: 47 pgs: 1 active+recovering+remapped, 46 active+clean; 1.8 GiB data, 8.2 GiB used, 886 GiB / 894 GiB avail; 115 MiB/s wr, 1.83k op/s; 751/86643 objects misplaced (0.867%); 1.3 MiB/s, 21 objects/s recovering 2024-03-20T09:13:49.508 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:49 smithi115 ceph-mon[72476]: 1.3 scrub starts 2024-03-20T09:13:49.508 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:49 smithi115 ceph-mon[72476]: 1.3 scrub ok 2024-03-20T09:13:49.511 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:49.534 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:13:49.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:49.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:13:49.639 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:13:49.696 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:13:49.696 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:13:49.696 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-03-20T09:13:49.696 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-03-20T09:13:49.697 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:13:49.703 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 18 16 36236 36220 125.753 126.312 0.0094565 0.00794646 2024-03-20T09:13:49.740 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:49.772 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:13:49.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:49.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:13:49.881 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:13:49.925 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:50.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:50 smithi115 ceph-mon[72476]: 1.17 scrub starts 2024-03-20T09:13:50.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:50 smithi115 ceph-mon[72476]: 1.17 scrub ok 2024-03-20T09:13:50.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:50 smithi115 ceph-mon[72476]: 1.13 scrub starts 2024-03-20T09:13:50.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:50 smithi122 ceph-mon[73495]: 1.17 scrub starts 2024-03-20T09:13:50.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:50 smithi122 ceph-mon[73495]: 1.17 scrub ok 2024-03-20T09:13:50.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:50 smithi122 ceph-mon[73495]: 1.13 scrub starts 2024-03-20T09:13:50.703 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 19 16 38036 38020 125.055 112.5 0.00654541 0.00799227 2024-03-20T09:13:50.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:50 smithi026 ceph-mon[68309]: 1.17 scrub starts 2024-03-20T09:13:50.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:50 smithi026 ceph-mon[68309]: 1.17 scrub ok 2024-03-20T09:13:50.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:50 smithi026 ceph-mon[68309]: 1.13 scrub starts 2024-03-20T09:13:50.797 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:13:50.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:50.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:13:50.906 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:13:50.937 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:50.980 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:13:50.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:50.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:13:51.085 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:13:51.148 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:51.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:51 smithi115 ceph-mon[72476]: pgmap v1252: 47 pgs: 47 active+clean; 1.9 GiB data, 8.5 GiB used, 885 GiB / 894 GiB avail; 95 MiB/s wr, 1.52k op/s; 5.9 MiB/s, 93 objects/s recovering 2024-03-20T09:13:51.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:51 smithi115 ceph-mon[72476]: 1.13 scrub ok 2024-03-20T09:13:51.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:51 smithi122 ceph-mon[73495]: pgmap v1252: 47 pgs: 47 active+clean; 1.9 GiB data, 8.5 GiB used, 885 GiB / 894 GiB avail; 95 MiB/s wr, 1.52k op/s; 5.9 MiB/s, 93 objects/s recovering 2024-03-20T09:13:51.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:51 smithi122 ceph-mon[73495]: 1.13 scrub ok 2024-03-20T09:13:51.703 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:2024-03-20T09:13:51.703810+0000 min lat: 0.00107883 max lat: 2.19794 avg lat: 0.00803881 2024-03-20T09:13:51.703 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T09:13:51.703 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 20 16 39805 39789 124.33 110.562 0.00932026 0.00803881 2024-03-20T09:13:51.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:51 smithi026 ceph-mon[68309]: pgmap v1252: 47 pgs: 47 active+clean; 1.9 GiB data, 8.5 GiB used, 885 GiB / 894 GiB avail; 95 MiB/s wr, 1.52k op/s; 5.9 MiB/s, 93 objects/s recovering 2024-03-20T09:13:51.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:51 smithi026 ceph-mon[68309]: 1.13 scrub ok 2024-03-20T09:13:51.757 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:13:51.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:51.761 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:13:51.781 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_historic_ops 2024-03-20T09:13:51.862 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:13:51.907 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:52.123 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:13:52.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:52.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:13:52.207 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:13:52.228 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:13:52.266 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:52.374 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:13:52.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:52.379 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:13:52.480 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:13:52.521 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:52.550 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:13:52.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:52.552 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:13:52.653 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:13:52.690 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:52.704 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 21 15 41729 41714 124.138 120.312 0.0115009 0.00805112 2024-03-20T09:13:52.939 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:13:53.089 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:13:53.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:53 smithi115 ceph-mon[72476]: pgmap v1253: 47 pgs: 47 active+clean; 2.2 GiB data, 9.6 GiB used, 884 GiB / 894 GiB avail; 130 MiB/s wr, 2.08k op/s; 5.9 MiB/s, 93 objects/s recovering 2024-03-20T09:13:53.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:53 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3327374705' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:13:53.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:53 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:13:53.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:53 smithi122 ceph-mon[73495]: pgmap v1253: 47 pgs: 47 active+clean; 2.2 GiB data, 9.6 GiB used, 884 GiB / 894 GiB avail; 130 MiB/s wr, 2.08k op/s; 5.9 MiB/s, 93 objects/s recovering 2024-03-20T09:13:53.573 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:53 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3327374705' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:13:53.573 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:53 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:13:53.574 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:53 smithi026 ceph-mon[68309]: pgmap v1253: 47 pgs: 47 active+clean; 2.2 GiB data, 9.6 GiB used, 884 GiB / 894 GiB avail; 130 MiB/s wr, 2.08k op/s; 5.9 MiB/s, 93 objects/s recovering 2024-03-20T09:13:53.574 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:53 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3327374705' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:13:53.574 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:53 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:13:53.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:53.575 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:13:53.575 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:13:53.676 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:13:53.703 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 22 16 43304 43288 122.966 98.375 0.175071 0.00811702 2024-03-20T09:13:53.706 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:53.732 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:13:53.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:53.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:13:53.837 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:13:53.880 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:54.094 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:13:54.115 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:13:54.146 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:13:54.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:54 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:13:54.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:54 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:13:54.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:54 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:13:54.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:54 smithi122 ceph-mon[73495]: Cluster is now healthy 2024-03-20T09:13:54.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:54 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:13:54.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:54 smithi122 ceph-mon[73495]: osdmap e536: 12 total, 12 up, 10 in 2024-03-20T09:13:54.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:54 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2854163446' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:13:54.198 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:13:54.198 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:13:54.231 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:13:54.270 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:13:54.303 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:13:54.369 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:13:54.369 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:13:54.378 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:13:54.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:54 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:13:54.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:54 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:13:54.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:54 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:13:54.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:54 smithi115 ceph-mon[72476]: Cluster is now healthy 2024-03-20T09:13:54.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:54 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:13:54.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:54 smithi115 ceph-mon[72476]: osdmap e536: 12 total, 12 up, 10 in 2024-03-20T09:13:54.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:54 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2854163446' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:13:54.416 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:13:54.431 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:13:54.431 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:13:54.431 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:13:54.431 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:13:54.431 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:13:54.431 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:13:54.431 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:13:54.431 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:13:54.432 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:13:54.432 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:13:54.432 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:13:54.432 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:13:54.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:54 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:13:54.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:54 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:13:54.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:54 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:13:54.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:54 smithi026 ceph-mon[68309]: Cluster is now healthy 2024-03-20T09:13:54.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:54 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:13:54.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:54 smithi026 ceph-mon[68309]: osdmap e536: 12 total, 12 up, 10 in 2024-03-20T09:13:54.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:54 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2854163446' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:13:54.704 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 23 16 44729 44713 121.492 89.0625 0.0134059 0.00822552 2024-03-20T09:13:54.740 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:13:54.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:54.743 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:13:54.844 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:13:54.888 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:55.095 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:13:55.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:55.100 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:13:55.202 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:13:55.231 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:55.285 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:13:55.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:55.291 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:13:55.393 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:13:55.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:55 smithi115 ceph-mon[72476]: 1.f scrub starts 2024-03-20T09:13:55.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:55 smithi115 ceph-mon[72476]: 1.f scrub ok 2024-03-20T09:13:55.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:55 smithi115 ceph-mon[72476]: pgmap v1255: 47 pgs: 47 active+clean; 2.3 GiB data, 9.8 GiB used, 884 GiB / 894 GiB avail; 81 MiB/s wr, 1.30k op/s; 5.9 MiB/s, 93 objects/s recovering 2024-03-20T09:13:55.427 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:55.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:55 smithi122 ceph-mon[73495]: 1.f scrub starts 2024-03-20T09:13:55.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:55 smithi122 ceph-mon[73495]: 1.f scrub ok 2024-03-20T09:13:55.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:55 smithi122 ceph-mon[73495]: pgmap v1255: 47 pgs: 47 active+clean; 2.3 GiB data, 9.8 GiB used, 884 GiB / 894 GiB avail; 81 MiB/s wr, 1.30k op/s; 5.9 MiB/s, 93 objects/s recovering 2024-03-20T09:13:55.447 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:55 smithi026 ceph-mon[68309]: 1.f scrub starts 2024-03-20T09:13:55.448 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:55 smithi026 ceph-mon[68309]: 1.f scrub ok 2024-03-20T09:13:55.448 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:55 smithi026 ceph-mon[68309]: pgmap v1255: 47 pgs: 47 active+clean; 2.3 GiB data, 9.8 GiB used, 884 GiB / 894 GiB avail; 81 MiB/s wr, 1.30k op/s; 5.9 MiB/s, 93 objects/s recovering 2024-03-20T09:13:55.450 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:13:55.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:55.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:13:55.456 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:13:55.456 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:13:55.554 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:13:55.594 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:55.704 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 24 16 46520 46504 121.093 111.938 0.00538714 0.00825392 2024-03-20T09:13:56.247 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:13:56.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:56.250 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:13:56.351 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:13:56.382 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:56.408 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:13:56.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:56.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:13:56.512 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:13:56.548 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:56.572 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:13:56.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:56.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:13:56.676 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:13:56.704 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 25 16 48428 48412 121.019 119.25 0.0106219 0.00825811 2024-03-20T09:13:56.712 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:56.736 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:13:56.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:56.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:13:56.840 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:13:56.903 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:56.966 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_ops_in_flight 2024-03-20T09:13:57.105 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:57 smithi026 ceph-mon[68309]: pgmap v1256: 47 pgs: 47 active+clean; 2.7 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 109 MiB/s wr, 1.75k op/s; 4.9 MiB/s, 78 objects/s recovering 2024-03-20T09:13:57.105 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:57 smithi026 ceph-mon[68309]: osdmap e537: 12 total, 12 up, 10 in 2024-03-20T09:13:57.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:57 smithi122 ceph-mon[73495]: pgmap v1256: 47 pgs: 47 active+clean; 2.7 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 109 MiB/s wr, 1.75k op/s; 4.9 MiB/s, 78 objects/s recovering 2024-03-20T09:13:57.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:57 smithi122 ceph-mon[73495]: osdmap e537: 12 total, 12 up, 10 in 2024-03-20T09:13:57.272 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:13:57.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:57.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:13:57.336 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:13:57.376 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:13:57.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:57 smithi115 ceph-mon[72476]: pgmap v1256: 47 pgs: 47 active+clean; 2.7 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 109 MiB/s wr, 1.75k op/s; 4.9 MiB/s, 78 objects/s recovering 2024-03-20T09:13:57.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:57 smithi115 ceph-mon[72476]: osdmap e537: 12 total, 12 up, 10 in 2024-03-20T09:13:57.413 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:57.437 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:13:57.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:57.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:13:57.542 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:13:57.595 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:57.704 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 26 16 50009 49993 120.165 98.8125 0.00455092 0.00831722 2024-03-20T09:13:58.169 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:13:58.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:58.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:13:58.276 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:13:58.361 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:58.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:58 smithi115 ceph-mon[72476]: 1.1c scrub starts 2024-03-20T09:13:58.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:58 smithi115 ceph-mon[72476]: 1.1c scrub ok 2024-03-20T09:13:58.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:58 smithi122 ceph-mon[73495]: 1.1c scrub starts 2024-03-20T09:13:58.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:58 smithi122 ceph-mon[73495]: 1.1c scrub ok 2024-03-20T09:13:58.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:58 smithi026 ceph-mon[68309]: 1.1c scrub starts 2024-03-20T09:13:58.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:58 smithi026 ceph-mon[68309]: 1.1c scrub ok 2024-03-20T09:13:58.494 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:13:58.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:58.499 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:13:58.600 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:13:58.655 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:58.687 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:13:58.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:58.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:13:58.704 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 27 16 51915 51899 120.126 119.125 0.00267738 0.00831921 2024-03-20T09:13:58.791 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:13:58.858 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:59.089 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:13:59.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:59.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:13:59.196 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:13:59.230 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:59.255 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:13:59.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:13:59.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:13:59.358 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:13:59.391 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:59 smithi122 ceph-mon[73495]: 1.a scrub starts 2024-03-20T09:13:59.391 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:59 smithi122 ceph-mon[73495]: 1.a scrub ok 2024-03-20T09:13:59.391 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:13:59 smithi122 ceph-mon[73495]: pgmap v1258: 47 pgs: 47 active+clean; 2.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-03-20T09:13:59.394 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:13:59.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:59 smithi115 ceph-mon[72476]: 1.a scrub starts 2024-03-20T09:13:59.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:59 smithi115 ceph-mon[72476]: 1.a scrub ok 2024-03-20T09:13:59.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:13:59 smithi115 ceph-mon[72476]: pgmap v1258: 47 pgs: 47 active+clean; 2.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-03-20T09:13:59.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:59 smithi026 ceph-mon[68309]: 1.a scrub starts 2024-03-20T09:13:59.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:59 smithi026 ceph-mon[68309]: 1.a scrub ok 2024-03-20T09:13:59.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:13:59 smithi026 ceph-mon[68309]: pgmap v1258: 47 pgs: 47 active+clean; 2.8 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-03-20T09:13:59.704 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 28 16 53629 53613 119.661 107.125 0.0631116 0.00834361 2024-03-20T09:13:59.852 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:13:59.964 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:14:00.150 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:14:00.150 INFO:teuthology.orchestra.run.smithi026.stdout:{"fsid":"852fbb68-e696-11ee-95c9-87774f69a715","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1397,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":538,"num_osds":12,"num_up_osds":12,"osd_up_since":1710925833,"num_in_osds":10,"osd_in_since":1710925819,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":47}],"num_pgs":47,"num_pools":2,"num_objects":48640,"data_bytes":3188130336,"bytes_used":12527480832,"bytes_avail":947422875648,"bytes_total":959950356480,"write_bytes_sec":97595268,"read_op_per_sec":0,"write_op_per_sec":1489},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.26:8443/"}},"servicemap":{"epoch":35,"modified":"2024-03-20T09:13:31.789453+0000","services":{"osd":{"daemons":{"summary":"","6":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-20T09:14:00.150 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:14:00.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:00.154 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:14:00.255 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:14:00.281 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:00.347 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:14:00.422 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:14:00.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:00.428 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:14:00.529 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:14:00.559 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:00.582 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:14:00.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:00.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:14:00.688 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:14:00.704 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 29 16 55078 55062 118.658 90.5625 0.00471842 0.00842312 2024-03-20T09:14:00.720 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:00.747 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:14:00.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:00.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:14:00.844 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:14:00.844 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:14:00.852 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:14:00.885 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:00.910 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:14:00.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:00.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:14:01.015 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:14:01.051 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:01.149 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:14:01.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:01.155 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:14:01.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:01 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:14:01.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:01 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:14:01.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:01 smithi115 ceph-mon[72476]: pgmap v1259: 47 pgs: 47 active+clean; 3.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-03-20T09:14:01.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:01 smithi115 ceph-mon[72476]: osdmap e538: 12 total, 12 up, 10 in 2024-03-20T09:14:01.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:01 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3753941960' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-20T09:14:01.256 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:14:01.289 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:01.311 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:14:01.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:01.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:14:01.416 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:14:01.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:01 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:14:01.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:01 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:14:01.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:01 smithi122 ceph-mon[73495]: pgmap v1259: 47 pgs: 47 active+clean; 3.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-03-20T09:14:01.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:01 smithi122 ceph-mon[73495]: osdmap e538: 12 total, 12 up, 10 in 2024-03-20T09:14:01.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:01 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3753941960' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-20T09:14:01.458 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:01.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:01 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:14:01.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:01 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:14:01.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:01 smithi026 ceph-mon[68309]: pgmap v1259: 47 pgs: 47 active+clean; 3.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-03-20T09:14:01.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:01 smithi026 ceph-mon[68309]: osdmap e538: 12 total, 12 up, 10 in 2024-03-20T09:14:01.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:01 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3753941960' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-20T09:14:01.706 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 30 16 56708 56692 118.092 101.875 0.00709759 0.00846404 2024-03-20T09:14:01.713 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:14:01.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:01.716 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:14:01.722 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_blocked_ops 2024-03-20T09:14:01.817 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:14:01.864 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:01.891 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:14:01.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:01.904 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:14:02.005 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:14:02.064 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:02.134 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:14:02.432 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:14:02.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:02.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:14:02.540 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:14:02.572 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:02.593 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:14:02.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:02.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:14:02.697 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:14:02.706 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 31 16 58454 58438 117.802 109.125 0.00270145 0.00848335 2024-03-20T09:14:02.755 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:03.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:03 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:14:03.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:03 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:14:03.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:03 smithi115 ceph-mon[72476]: pgmap v1261: 47 pgs: 47 active+clean; 3.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 125 MiB/s wr, 1.99k op/s 2024-03-20T09:14:03.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:03 smithi115 ceph-mon[72476]: osdmap e539: 12 total, 12 up, 10 in 2024-03-20T09:14:03.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:03 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:14:03.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:03 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:14:03.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:03 smithi122 ceph-mon[73495]: pgmap v1261: 47 pgs: 47 active+clean; 3.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 125 MiB/s wr, 1.99k op/s 2024-03-20T09:14:03.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:03 smithi122 ceph-mon[73495]: osdmap e539: 12 total, 12 up, 10 in 2024-03-20T09:14:03.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:03 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:14:03.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:03 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:14:03.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:03 smithi026 ceph-mon[68309]: pgmap v1261: 47 pgs: 47 active+clean; 3.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 125 MiB/s wr, 1.99k op/s 2024-03-20T09:14:03.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:03 smithi026 ceph-mon[68309]: osdmap e539: 12 total, 12 up, 10 in 2024-03-20T09:14:03.607 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-03-20T09:14:03.607 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:14:03.706 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 32 16 60129 60113 117.392 104.688 0.0133432 0.00851388 2024-03-20T09:14:04.706 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 33 16 61711 61695 116.831 98.875 0.00885505 0.00855541 2024-03-20T09:14:05.334 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:14:05.356 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:14:05.366 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:05 smithi026 ceph-mon[68309]: pgmap v1263: 47 pgs: 47 active+clean; 3.4 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-03-20T09:14:05.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:05.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:14:05.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:05 smithi115 ceph-mon[72476]: pgmap v1263: 47 pgs: 47 active+clean; 3.4 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-03-20T09:14:05.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:05 smithi122 ceph-mon[73495]: pgmap v1263: 47 pgs: 47 active+clean; 3.4 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-03-20T09:14:05.468 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:14:05.500 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:05.525 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:14:05.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:05.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:14:05.630 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:14:05.659 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:05.662 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:14:05.682 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:14:05.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:05.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:14:05.706 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 34 16 63298 63282 116.312 99.1875 0.0108656 0.00859366 2024-03-20T09:14:05.742 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:14:05.742 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:14:05.787 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:14:05.847 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:06.660 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:14:06.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:06.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:14:06.706 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 35 16 64758 64742 115.596 91.25 0.00799444 0.00864691 2024-03-20T09:14:06.766 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:14:06.839 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:07.049 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_historic_ops 2024-03-20T09:14:07.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:07 smithi115 ceph-mon[72476]: pgmap v1264: 47 pgs: 47 active+clean; 3.7 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 119 MiB/s wr, 1.91k op/s 2024-03-20T09:14:07.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:07 smithi122 ceph-mon[73495]: pgmap v1264: 47 pgs: 47 active+clean; 3.7 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 119 MiB/s wr, 1.91k op/s 2024-03-20T09:14:07.461 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:14:07.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:07 smithi026 ceph-mon[68309]: pgmap v1264: 47 pgs: 47 active+clean; 3.7 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 119 MiB/s wr, 1.91k op/s 2024-03-20T09:14:07.706 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 36 16 66207 66191 114.9 90.5625 0.00287916 0.00869252 2024-03-20T09:14:08.028 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:14:08.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:08.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:14:08.135 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:14:08.167 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:08.193 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:14:08.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:08.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:14:08.204 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:14:08.242 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:14:08.296 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:14:08.298 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:14:08.329 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:08 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:14:08.329 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:08 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:14:08.329 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:08 smithi122 ceph-mon[73495]: osdmap e540: 12 total, 12 up, 10 in 2024-03-20T09:14:08.332 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:08.335 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:14:08.371 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:14:08.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:08 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:14:08.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:08 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:14:08.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:08 smithi115 ceph-mon[72476]: osdmap e540: 12 total, 12 up, 10 in 2024-03-20T09:14:08.407 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:14:08.432 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:14:08.456 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:14:08.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:08 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:14:08.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:08 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:14:08.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:08 smithi026 ceph-mon[68309]: osdmap e540: 12 total, 12 up, 10 in 2024-03-20T09:14:08.496 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:14:08.528 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:14:08.567 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:14:08.600 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:14:08.615 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:14:08.615 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:14:08.615 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:14:08.615 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:14:08.615 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:14:08.615 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:14:08.615 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:14:08.615 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:14:08.615 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:14:08.615 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:14:08.615 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:14:08.615 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:14:08.706 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 37 16 67601 67585 114.149 87.125 0.00338313 0.00875442 2024-03-20T09:14:08.992 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:14:08.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:08.997 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:14:09.098 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:14:09.135 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:09.404 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:09 smithi026 ceph-mon[68309]: pgmap v1266: 47 pgs: 47 active+clean; 3.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-03-20T09:14:09.405 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:09 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3371750604' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:14:09.405 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:09 smithi026 ceph-mon[68309]: 1.e scrub starts 2024-03-20T09:14:09.405 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:09 smithi026 ceph-mon[68309]: 1.e scrub ok 2024-03-20T09:14:09.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:09 smithi115 ceph-mon[72476]: pgmap v1266: 47 pgs: 47 active+clean; 3.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-03-20T09:14:09.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:09 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3371750604' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:14:09.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:09 smithi115 ceph-mon[72476]: 1.e scrub starts 2024-03-20T09:14:09.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:09 smithi115 ceph-mon[72476]: 1.e scrub ok 2024-03-20T09:14:09.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:09 smithi122 ceph-mon[73495]: pgmap v1266: 47 pgs: 47 active+clean; 3.8 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-03-20T09:14:09.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:09 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3371750604' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:14:09.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:09 smithi122 ceph-mon[73495]: 1.e scrub starts 2024-03-20T09:14:09.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:09 smithi122 ceph-mon[73495]: 1.e scrub ok 2024-03-20T09:14:09.707 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 38 16 68711 68695 112.971 69.375 0.00977216 0.00883985 2024-03-20T09:14:09.897 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:14:09.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:09.904 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:14:10.005 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:14:10.036 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:10.076 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:14:10.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:10.079 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:14:10.180 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:14:10.241 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:10.359 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:14:10.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:10.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:14:10.469 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:14:10.501 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:10.529 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:14:10.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:10.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:14:10.606 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:14:10.606 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-03-20T09:14:10.606 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-03-20T09:14:10.607 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-03-20T09:14:10.607 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.607 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.24910.0:142022 3.d e429/275 3:bca4a78c:::benchmark_data_smithi122_119671_object42240:head v 429'8759, mlcod=256'4799)", 2024-03-20T09:14:10.607 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:10:07.207810+0000", 2024-03-20T09:14:10.607 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 243.39544682799999, 2024-03-20T09:14:10.607 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0054674040000000004, 2024-03-20T09:14:10.607 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:10.607 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:14:10.607 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:10.607 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.607 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:10.607 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:10:07.207810+0000", 2024-03-20T09:14:10.607 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.608 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.608 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.608 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:10.608 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:10:07.207810+0000", 2024-03-20T09:14:10.608 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.608 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.608 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.608 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:10.608 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:10:07.207811+0000", 2024-03-20T09:14:10.608 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5179999999999999e-06 2024-03-20T09:14:10.608 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.608 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.608 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:10.608 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:10:07.207822+0000", 2024-03-20T09:14:10.608 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0226e-05 2024-03-20T09:14:10.609 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.609 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.609 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:10.609 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:10:07.207843+0000", 2024-03-20T09:14:10.609 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1141999999999998e-05 2024-03-20T09:14:10.609 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.609 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.609 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:10.609 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:10:07.207851+0000", 2024-03-20T09:14:10.609 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3590000000000008e-06 2024-03-20T09:14:10.609 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.609 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.609 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:10.609 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:10:07.207933+0000", 2024-03-20T09:14:10.609 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2238999999999996e-05 2024-03-20T09:14:10.610 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.610 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.610 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:14:10.610 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:10:07.207962+0000", 2024-03-20T09:14:10.610 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8520000000000001e-05 2024-03-20T09:14:10.610 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.610 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.610 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:14:10.610 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:10:07.213237+0000", 2024-03-20T09:14:10.610 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0052752600000000004 2024-03-20T09:14:10.610 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.610 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.610 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:10.610 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:10:07.213277+0000", 2024-03-20T09:14:10.610 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0139999999999999e-05 2024-03-20T09:14:10.611 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.611 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:10.611 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.611 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.611 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.611 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e462/434 cost 1049576)", 2024-03-20T09:14:10.611 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:09.212827+0000", 2024-03-20T09:14:10.611 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 181.39042968300001, 2024-03-20T09:14:10.611 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00099083599999999993, 2024-03-20T09:14:10.611 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:10.611 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:14:10.611 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:10.611 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.611 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:10.612 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:09.212827+0000", 2024-03-20T09:14:10.612 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.612 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.612 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.612 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:10.612 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:09.212827+0000", 2024-03-20T09:14:10.612 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.612 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.612 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.612 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:10.612 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:09.212828+0000", 2024-03-20T09:14:10.612 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.037e-06 2024-03-20T09:14:10.612 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.612 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.612 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:10.612 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:09.212840+0000", 2024-03-20T09:14:10.613 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1929e-05 2024-03-20T09:14:10.613 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.613 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.613 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:10.613 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:09.212852+0000", 2024-03-20T09:14:10.613 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2148e-05 2024-03-20T09:14:10.613 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.613 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.613 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:10.613 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:09.212861+0000", 2024-03-20T09:14:10.613 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4519999999999997e-06 2024-03-20T09:14:10.613 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.613 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.613 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:10.613 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:09.212951+0000", 2024-03-20T09:14:10.614 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0557999999999996e-05 2024-03-20T09:14:10.614 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.614 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.614 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:10.614 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:09.213818+0000", 2024-03-20T09:14:10.614 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00086671199999999995 2024-03-20T09:14:10.614 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.614 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:10.614 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.614 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.614 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.614 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e465/434 cost 1049576)", 2024-03-20T09:14:10.614 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:17.855044+0000", 2024-03-20T09:14:10.614 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 172.74821317199999, 2024-03-20T09:14:10.614 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00072467199999999999, 2024-03-20T09:14:10.615 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:10.615 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:14:10.615 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:10.615 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.615 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:10.615 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:17.855044+0000", 2024-03-20T09:14:10.615 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.615 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.615 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.615 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:10.615 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:17.855044+0000", 2024-03-20T09:14:10.615 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.615 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.615 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.615 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:10.616 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:17.855046+0000", 2024-03-20T09:14:10.616 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0329999999999998e-06 2024-03-20T09:14:10.616 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.616 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.616 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:10.616 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:17.855068+0000", 2024-03-20T09:14:10.616 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2833e-05 2024-03-20T09:14:10.616 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.616 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.616 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:10.616 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:17.855096+0000", 2024-03-20T09:14:10.616 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7419999999999998e-05 2024-03-20T09:14:10.616 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.617 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.617 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:10.617 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:17.855118+0000", 2024-03-20T09:14:10.617 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1668999999999999e-05 2024-03-20T09:14:10.617 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.617 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.617 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:10.617 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:17.855224+0000", 2024-03-20T09:14:10.617 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010615099999999999 2024-03-20T09:14:10.617 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.617 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.617 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:10.617 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:17.855768+0000", 2024-03-20T09:14:10.617 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00054456600000000002 2024-03-20T09:14:10.618 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.618 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:10.618 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.618 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.618 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.618 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e472/434 cost 1049576)", 2024-03-20T09:14:10.618 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:35.044500+0000", 2024-03-20T09:14:10.618 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 155.55875666, 2024-03-20T09:14:10.618 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000590983, 2024-03-20T09:14:10.618 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:10.618 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:14:10.618 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:10.618 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.618 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:10.619 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.044500+0000", 2024-03-20T09:14:10.619 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.619 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.619 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.619 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:10.619 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.044500+0000", 2024-03-20T09:14:10.619 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.619 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.619 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.619 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:10.619 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.044501+0000", 2024-03-20T09:14:10.619 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9899999999999999e-07 2024-03-20T09:14:10.619 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.619 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.620 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:10.620 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.044513+0000", 2024-03-20T09:14:10.620 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1574e-05 2024-03-20T09:14:10.620 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.620 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.620 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:10.620 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.044538+0000", 2024-03-20T09:14:10.620 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4909000000000002e-05 2024-03-20T09:14:10.620 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.620 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.620 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:10.620 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.044560+0000", 2024-03-20T09:14:10.620 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2742999999999999e-05 2024-03-20T09:14:10.620 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.620 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.620 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:10.621 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.044616+0000", 2024-03-20T09:14:10.621 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5472e-05 2024-03-20T09:14:10.621 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.621 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.621 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:10.621 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.045091+0000", 2024-03-20T09:14:10.621 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00047538600000000001 2024-03-20T09:14:10.621 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.621 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:10.621 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.621 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.621 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.621 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e472/434 cost 1049576)", 2024-03-20T09:14:10.621 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:35.123747+0000", 2024-03-20T09:14:10.621 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 155.47950958000001, 2024-03-20T09:14:10.622 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00073770699999999999, 2024-03-20T09:14:10.622 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:10.622 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:14:10.622 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:10.622 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.622 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:10.622 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.123747+0000", 2024-03-20T09:14:10.622 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.622 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.622 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.622 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:10.622 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.123747+0000", 2024-03-20T09:14:10.622 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.622 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.623 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.623 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:10.623 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.123749+0000", 2024-03-20T09:14:10.623 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5030000000000001e-06 2024-03-20T09:14:10.623 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.623 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.623 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:10.623 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.123768+0000", 2024-03-20T09:14:10.623 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.893e-05 2024-03-20T09:14:10.623 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.623 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.623 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:10.623 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.123791+0000", 2024-03-20T09:14:10.623 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3166000000000001e-05 2024-03-20T09:14:10.623 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.623 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.624 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:10.624 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.123803+0000", 2024-03-20T09:14:10.624 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2517000000000001e-05 2024-03-20T09:14:10.624 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.624 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.624 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:10.624 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.123924+0000", 2024-03-20T09:14:10.624 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000120413 2024-03-20T09:14:10.624 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.624 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.624 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:10.624 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.124485+0000", 2024-03-20T09:14:10.624 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00056117799999999996 2024-03-20T09:14:10.624 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.624 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:10.624 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.625 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.625 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.625 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e472/434 cost 1049576)", 2024-03-20T09:14:10.625 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:35.343418+0000", 2024-03-20T09:14:10.625 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 155.25983927199999, 2024-03-20T09:14:10.625 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00064077999999999997, 2024-03-20T09:14:10.625 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:10.625 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:14:10.625 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:10.625 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.625 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:10.625 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.343418+0000", 2024-03-20T09:14:10.625 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.625 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.625 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.626 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:10.626 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.343418+0000", 2024-03-20T09:14:10.626 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.626 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.626 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.626 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:10.626 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.343419+0000", 2024-03-20T09:14:10.626 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7239999999999999e-06 2024-03-20T09:14:10.626 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.626 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.626 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:10.626 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.343442+0000", 2024-03-20T09:14:10.626 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2883000000000001e-05 2024-03-20T09:14:10.627 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.627 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.627 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:10.627 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.343469+0000", 2024-03-20T09:14:10.627 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7216000000000002e-05 2024-03-20T09:14:10.627 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.627 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.627 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:10.627 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.343484+0000", 2024-03-20T09:14:10.627 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4333e-05 2024-03-20T09:14:10.627 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.627 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.627 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:10.627 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.343591+0000", 2024-03-20T09:14:10.628 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010685 2024-03-20T09:14:10.628 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.628 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.628 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:10.628 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.344058+0000", 2024-03-20T09:14:10.628 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00046777400000000001 2024-03-20T09:14:10.628 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.628 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:10.628 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.628 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.628 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.628 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e472/434 cost 1049576)", 2024-03-20T09:14:10.628 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:36.828898+0000", 2024-03-20T09:14:10.629 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 153.774358557, 2024-03-20T09:14:10.629 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00059100600000000004, 2024-03-20T09:14:10.629 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:10.629 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:14:10.629 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:10.629 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.629 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:10.629 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:36.828898+0000", 2024-03-20T09:14:10.629 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.629 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.629 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.629 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:10.629 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:36.828898+0000", 2024-03-20T09:14:10.629 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.630 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.630 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.630 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:10.630 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:36.828899+0000", 2024-03-20T09:14:10.630 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1060000000000001e-06 2024-03-20T09:14:10.630 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.630 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.630 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:10.630 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:36.828912+0000", 2024-03-20T09:14:10.630 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3091000000000001e-05 2024-03-20T09:14:10.630 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.630 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.630 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:10.630 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:36.828925+0000", 2024-03-20T09:14:10.631 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2819e-05 2024-03-20T09:14:10.631 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.631 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.631 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:10.631 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:36.828950+0000", 2024-03-20T09:14:10.631 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4729999999999999e-05 2024-03-20T09:14:10.631 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.631 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.631 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:10.631 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:36.829001+0000", 2024-03-20T09:14:10.631 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0670000000000001e-05 2024-03-20T09:14:10.631 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.631 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.631 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:10.632 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:36.829489+0000", 2024-03-20T09:14:10.632 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00048859000000000001 2024-03-20T09:14:10.632 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.632 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:10.632 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.632 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.632 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.632 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e473/434 cost 1049576)", 2024-03-20T09:14:10.632 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:38.408069+0000", 2024-03-20T09:14:10.632 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 152.19518753099999, 2024-03-20T09:14:10.632 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00062260899999999999, 2024-03-20T09:14:10.632 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:10.632 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:14:10.632 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:10.632 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.632 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:10.633 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:38.408069+0000", 2024-03-20T09:14:10.633 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.633 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.633 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.633 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:10.633 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:38.408069+0000", 2024-03-20T09:14:10.633 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.633 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.633 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.633 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:10.633 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:38.408070+0000", 2024-03-20T09:14:10.633 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.24e-06 2024-03-20T09:14:10.633 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.633 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.633 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:10.633 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:38.408085+0000", 2024-03-20T09:14:10.634 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4166e-05 2024-03-20T09:14:10.634 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.634 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.634 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:10.634 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:38.408101+0000", 2024-03-20T09:14:10.634 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5991e-05 2024-03-20T09:14:10.634 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.634 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.634 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:10.634 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:38.408110+0000", 2024-03-20T09:14:10.634 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5689999999999994e-06 2024-03-20T09:14:10.634 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.634 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.635 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:10.635 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:38.408171+0000", 2024-03-20T09:14:10.635 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0745999999999997e-05 2024-03-20T09:14:10.635 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.635 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.635 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:10.635 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:38.408692+0000", 2024-03-20T09:14:10.635 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00052089699999999998 2024-03-20T09:14:10.635 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.635 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:10.635 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.635 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.635 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.635 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e473/434 cost 1049576)", 2024-03-20T09:14:10.636 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:39.640113+0000", 2024-03-20T09:14:10.636 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 150.96314412199999, 2024-03-20T09:14:10.636 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000629661, 2024-03-20T09:14:10.636 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:10.636 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:14:10.636 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:10.636 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.636 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:10.636 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:39.640113+0000", 2024-03-20T09:14:10.636 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.636 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.636 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.636 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:10.636 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:39.640113+0000", 2024-03-20T09:14:10.637 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.637 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.637 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.637 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:10.637 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:39.640114+0000", 2024-03-20T09:14:10.637 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.046e-06 2024-03-20T09:14:10.637 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.637 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.637 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:10.637 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:39.640127+0000", 2024-03-20T09:14:10.637 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3074999999999999e-05 2024-03-20T09:14:10.637 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.638 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.638 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:10.638 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:39.640140+0000", 2024-03-20T09:14:10.638 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3112000000000001e-05 2024-03-20T09:14:10.638 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.638 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.638 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:10.638 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:39.640147+0000", 2024-03-20T09:14:10.638 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1069999999999999e-06 2024-03-20T09:14:10.638 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.638 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.638 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:10.638 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:39.640223+0000", 2024-03-20T09:14:10.638 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6462000000000001e-05 2024-03-20T09:14:10.638 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.639 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.639 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:10.639 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:39.640742+0000", 2024-03-20T09:14:10.639 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00051885899999999999 2024-03-20T09:14:10.639 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.639 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:10.639 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.639 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.639 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.639 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e473/434 cost 1049576)", 2024-03-20T09:14:10.639 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:39.681702+0000", 2024-03-20T09:14:10.639 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 150.92155488899999, 2024-03-20T09:14:10.639 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00060000500000000003, 2024-03-20T09:14:10.639 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:10.639 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:14:10.640 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:10.640 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.640 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:10.640 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:39.681702+0000", 2024-03-20T09:14:10.640 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.640 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.640 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.640 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:10.640 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:39.681702+0000", 2024-03-20T09:14:10.640 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.640 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.640 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.640 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:10.640 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:39.681703+0000", 2024-03-20T09:14:10.640 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.08e-06 2024-03-20T09:14:10.641 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.641 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.641 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:10.641 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:39.681713+0000", 2024-03-20T09:14:10.641 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.8349999999999999e-06 2024-03-20T09:14:10.641 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.641 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.641 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:10.641 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:39.681724+0000", 2024-03-20T09:14:10.641 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1022000000000001e-05 2024-03-20T09:14:10.641 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.641 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.641 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:10.641 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:39.681730+0000", 2024-03-20T09:14:10.641 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5880000000000003e-06 2024-03-20T09:14:10.642 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.642 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.642 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:10.642 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:39.681781+0000", 2024-03-20T09:14:10.642 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0532999999999999e-05 2024-03-20T09:14:10.642 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.642 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.642 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:10.642 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:39.682302+0000", 2024-03-20T09:14:10.642 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00052094700000000001 2024-03-20T09:14:10.642 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.642 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:10.642 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.642 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.642 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.643 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e475/434 cost 1049576)", 2024-03-20T09:14:10.643 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:42.972719+0000", 2024-03-20T09:14:10.643 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 147.63053765800001, 2024-03-20T09:14:10.643 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00060214000000000003, 2024-03-20T09:14:10.643 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:10.643 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:14:10.643 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:10.643 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.643 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:10.643 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:42.972719+0000", 2024-03-20T09:14:10.643 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.643 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.643 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.643 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:10.643 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:42.972719+0000", 2024-03-20T09:14:10.643 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.644 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.644 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.644 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:10.644 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:42.972720+0000", 2024-03-20T09:14:10.644 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.035e-06 2024-03-20T09:14:10.644 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.644 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.644 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:10.644 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:42.972731+0000", 2024-03-20T09:14:10.644 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0868e-05 2024-03-20T09:14:10.644 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.644 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.644 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:10.644 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:42.972744+0000", 2024-03-20T09:14:10.644 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3053e-05 2024-03-20T09:14:10.644 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.645 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.645 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:10.645 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:42.972751+0000", 2024-03-20T09:14:10.645 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.126e-06 2024-03-20T09:14:10.645 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.645 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.645 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:10.645 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:42.972816+0000", 2024-03-20T09:14:10.645 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4380000000000004e-05 2024-03-20T09:14:10.645 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.645 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.645 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:10.645 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:42.973321+0000", 2024-03-20T09:14:10.645 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00050567799999999997 2024-03-20T09:14:10.645 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.646 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:10.646 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.646 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.646 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.646 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e475/434 cost 1049576)", 2024-03-20T09:14:10.646 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:43.069398+0000", 2024-03-20T09:14:10.646 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 147.53385918199999, 2024-03-20T09:14:10.646 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00064565900000000003, 2024-03-20T09:14:10.646 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:10.646 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:14:10.646 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:10.646 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.646 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:10.646 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:43.069398+0000", 2024-03-20T09:14:10.647 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.647 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.647 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.647 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:10.647 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:43.069398+0000", 2024-03-20T09:14:10.647 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.647 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.647 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.647 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:10.647 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:43.069399+0000", 2024-03-20T09:14:10.647 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2190000000000001e-06 2024-03-20T09:14:10.647 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.647 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.647 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:10.648 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:43.069411+0000", 2024-03-20T09:14:10.648 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2500999999999999e-05 2024-03-20T09:14:10.648 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.648 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.648 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:10.648 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:43.069424+0000", 2024-03-20T09:14:10.648 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2958e-05 2024-03-20T09:14:10.648 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.648 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.648 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:10.648 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:43.069438+0000", 2024-03-20T09:14:10.648 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3906e-05 2024-03-20T09:14:10.648 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.648 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.648 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:10.649 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:43.069516+0000", 2024-03-20T09:14:10.649 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7899999999999996e-05 2024-03-20T09:14:10.649 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.649 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.649 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:10.649 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:43.070043+0000", 2024-03-20T09:14:10.649 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00052717499999999995 2024-03-20T09:14:10.649 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.649 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:10.649 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.649 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.649 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.649 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e475/434 cost 1049576)", 2024-03-20T09:14:10.649 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:43.132460+0000", 2024-03-20T09:14:10.649 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 147.47079643500001, 2024-03-20T09:14:10.650 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00060327799999999995, 2024-03-20T09:14:10.650 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:10.650 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:14:10.650 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:10.650 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.650 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:10.650 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:43.132460+0000", 2024-03-20T09:14:10.650 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.650 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.650 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.650 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:10.650 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:43.132460+0000", 2024-03-20T09:14:10.650 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.650 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.651 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.651 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:10.651 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:43.132461+0000", 2024-03-20T09:14:10.651 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9100000000000004e-07 2024-03-20T09:14:10.651 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.651 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.651 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:10.651 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:43.132471+0000", 2024-03-20T09:14:10.651 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0508e-05 2024-03-20T09:14:10.651 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.651 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.651 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:10.651 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:43.132480+0000", 2024-03-20T09:14:10.651 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5650000000000002e-06 2024-03-20T09:14:10.651 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.652 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.652 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:10.652 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:43.132485+0000", 2024-03-20T09:14:10.652 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7670000000000003e-06 2024-03-20T09:14:10.652 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.652 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.652 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:10.652 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:43.132542+0000", 2024-03-20T09:14:10.652 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7463000000000002e-05 2024-03-20T09:14:10.652 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.652 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.652 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:10.652 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:43.133064+0000", 2024-03-20T09:14:10.652 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00052138399999999995 2024-03-20T09:14:10.652 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.653 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:10.653 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.653 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.653 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.653 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e475/434 cost 1049576)", 2024-03-20T09:14:10.653 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:44.577581+0000", 2024-03-20T09:14:10.653 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 146.025676014, 2024-03-20T09:14:10.653 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00083124100000000003, 2024-03-20T09:14:10.653 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:10.653 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:14:10.653 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:10.653 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.653 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:10.653 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:44.577581+0000", 2024-03-20T09:14:10.653 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.654 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.654 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.654 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:10.654 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:44.577581+0000", 2024-03-20T09:14:10.654 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.654 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.654 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.654 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:10.654 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:44.577583+0000", 2024-03-20T09:14:10.654 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0030000000000001e-06 2024-03-20T09:14:10.654 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.654 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.654 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:10.654 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:44.577605+0000", 2024-03-20T09:14:10.654 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2198999999999999e-05 2024-03-20T09:14:10.655 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.655 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.655 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:10.655 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:44.577633+0000", 2024-03-20T09:14:10.655 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8019999999999999e-05 2024-03-20T09:14:10.655 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.655 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.655 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:10.655 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:44.577652+0000", 2024-03-20T09:14:10.655 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8516e-05 2024-03-20T09:14:10.655 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.655 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.655 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:10.655 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:44.577760+0000", 2024-03-20T09:14:10.655 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000108724 2024-03-20T09:14:10.656 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.656 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.656 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:10.656 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:44.578412+0000", 2024-03-20T09:14:10.656 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00065177900000000001 2024-03-20T09:14:10.656 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.656 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:10.656 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.656 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.656 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.656 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e476/434 cost 1049576)", 2024-03-20T09:14:10.656 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:46.038141+0000", 2024-03-20T09:14:10.656 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 144.56511600100001, 2024-03-20T09:14:10.656 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002222365, 2024-03-20T09:14:10.656 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:10.657 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:14:10.657 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:10.657 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.657 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:10.657 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.038141+0000", 2024-03-20T09:14:10.657 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.657 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.657 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.657 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:10.657 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.038141+0000", 2024-03-20T09:14:10.657 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.657 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.657 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.658 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:10.658 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.038142+0000", 2024-03-20T09:14:10.658 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0249999999999999e-06 2024-03-20T09:14:10.658 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.658 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.658 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:10.658 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.038154+0000", 2024-03-20T09:14:10.658 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2379e-05 2024-03-20T09:14:10.658 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.658 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.658 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:10.658 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.038168+0000", 2024-03-20T09:14:10.658 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3329000000000001e-05 2024-03-20T09:14:10.658 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.658 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.659 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:10.659 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.038175+0000", 2024-03-20T09:14:10.659 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0790000000000003e-06 2024-03-20T09:14:10.659 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.659 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.659 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:10.659 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.038231+0000", 2024-03-20T09:14:10.659 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6305999999999999e-05 2024-03-20T09:14:10.659 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.659 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.659 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:10.659 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.040363+0000", 2024-03-20T09:14:10.659 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0021322469999999999 2024-03-20T09:14:10.659 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.660 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:10.660 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.660 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.660 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.660 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e476/434 cost 1049576)", 2024-03-20T09:14:10.660 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:46.647115+0000", 2024-03-20T09:14:10.660 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 143.95614144800001, 2024-03-20T09:14:10.660 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00065813999999999998, 2024-03-20T09:14:10.660 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:10.660 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:14:10.660 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:10.660 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.660 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:10.660 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.647115+0000", 2024-03-20T09:14:10.660 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.660 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.661 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.661 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:10.661 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.647115+0000", 2024-03-20T09:14:10.661 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.661 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.661 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.661 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:10.661 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.647117+0000", 2024-03-20T09:14:10.661 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0700000000000001e-06 2024-03-20T09:14:10.661 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.661 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.661 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:10.661 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.647141+0000", 2024-03-20T09:14:10.661 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3346e-05 2024-03-20T09:14:10.661 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.661 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.662 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:10.662 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.647168+0000", 2024-03-20T09:14:10.662 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7029000000000002e-05 2024-03-20T09:14:10.662 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.662 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.662 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:10.662 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.647182+0000", 2024-03-20T09:14:10.662 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3747000000000001e-05 2024-03-20T09:14:10.662 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.662 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.662 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:10.662 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.647306+0000", 2024-03-20T09:14:10.662 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000124766 2024-03-20T09:14:10.662 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.663 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.663 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:10.663 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.647773+0000", 2024-03-20T09:14:10.663 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00046718200000000001 2024-03-20T09:14:10.663 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.663 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:10.663 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.663 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.663 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.663 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e476/434 cost 1049576)", 2024-03-20T09:14:10.663 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:46.916404+0000", 2024-03-20T09:14:10.663 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 143.68685298099999, 2024-03-20T09:14:10.663 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00060503900000000003, 2024-03-20T09:14:10.663 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:10.664 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:14:10.664 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:10.664 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.664 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:10.664 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.916404+0000", 2024-03-20T09:14:10.664 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.664 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.664 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.664 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:10.664 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.916404+0000", 2024-03-20T09:14:10.664 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.664 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.664 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.664 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:10.664 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.916405+0000", 2024-03-20T09:14:10.664 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0499999999999999e-06 2024-03-20T09:14:10.665 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.665 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.665 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:10.665 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.916416+0000", 2024-03-20T09:14:10.665 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0824e-05 2024-03-20T09:14:10.665 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.665 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.665 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:10.665 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.916427+0000", 2024-03-20T09:14:10.665 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0875e-05 2024-03-20T09:14:10.665 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.665 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.665 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:10.665 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.916451+0000", 2024-03-20T09:14:10.665 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4162e-05 2024-03-20T09:14:10.666 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.666 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.666 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:10.666 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.916499+0000", 2024-03-20T09:14:10.666 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8445000000000001e-05 2024-03-20T09:14:10.666 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.666 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.666 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:10.666 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.917009+0000", 2024-03-20T09:14:10.666 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00050968300000000003 2024-03-20T09:14:10.666 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.666 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:10.666 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.666 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.667 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.667 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e476/434 cost 1049576)", 2024-03-20T09:14:10.667 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:47.042968+0000", 2024-03-20T09:14:10.667 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 143.56028913200001, 2024-03-20T09:14:10.667 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00058974899999999998, 2024-03-20T09:14:10.667 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:10.667 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:14:10.667 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:10.667 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.667 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:10.667 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.042968+0000", 2024-03-20T09:14:10.667 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.667 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.668 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.668 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:10.668 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.042968+0000", 2024-03-20T09:14:10.668 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.668 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.668 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.668 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:10.668 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.042970+0000", 2024-03-20T09:14:10.668 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8870000000000001e-06 2024-03-20T09:14:10.668 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.668 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.668 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:10.668 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.042994+0000", 2024-03-20T09:14:10.668 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4083e-05 2024-03-20T09:14:10.668 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.669 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.669 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:10.669 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.043019+0000", 2024-03-20T09:14:10.669 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5386000000000001e-05 2024-03-20T09:14:10.669 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.669 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.669 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:10.669 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.043034+0000", 2024-03-20T09:14:10.669 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.484e-05 2024-03-20T09:14:10.669 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.669 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.669 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:10.669 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.043127+0000", 2024-03-20T09:14:10.669 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3313000000000006e-05 2024-03-20T09:14:10.670 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.670 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.670 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:10.670 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.043557+0000", 2024-03-20T09:14:10.670 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00043023999999999997 2024-03-20T09:14:10.670 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.670 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:10.670 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.670 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.670 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.670 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e477/434 cost 1049576)", 2024-03-20T09:14:10.670 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:47.242004+0000", 2024-03-20T09:14:10.670 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 143.36125292200001, 2024-03-20T09:14:10.670 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00069037099999999995, 2024-03-20T09:14:10.670 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:10.671 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:14:10.671 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:10.671 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.671 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:10.671 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.242004+0000", 2024-03-20T09:14:10.671 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.671 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.671 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.671 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:10.671 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.242004+0000", 2024-03-20T09:14:10.671 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.671 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.671 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.671 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:10.672 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.242006+0000", 2024-03-20T09:14:10.672 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8199999999999999e-06 2024-03-20T09:14:10.672 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.672 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.672 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:10.672 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.242029+0000", 2024-03-20T09:14:10.672 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.317e-05 2024-03-20T09:14:10.672 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.672 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.672 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:10.672 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.242053+0000", 2024-03-20T09:14:10.672 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4113000000000001e-05 2024-03-20T09:14:10.672 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.672 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.672 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:10.672 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.242067+0000", 2024-03-20T09:14:10.673 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3645999999999999e-05 2024-03-20T09:14:10.673 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.673 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.673 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:10.673 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.242187+0000", 2024-03-20T09:14:10.673 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012064499999999999 2024-03-20T09:14:10.673 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.673 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.673 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:10.673 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.242694+0000", 2024-03-20T09:14:10.673 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00050697700000000004 2024-03-20T09:14:10.673 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.673 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:10.673 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.674 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.674 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.674 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e477/434 cost 1049576)", 2024-03-20T09:14:10.674 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:47.245542+0000", 2024-03-20T09:14:10.674 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 143.357714964, 2024-03-20T09:14:10.674 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00065577400000000003, 2024-03-20T09:14:10.674 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:10.674 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:14:10.674 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:10.674 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.674 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:10.674 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.245542+0000", 2024-03-20T09:14:10.674 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.674 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.674 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.675 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:10.675 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.245542+0000", 2024-03-20T09:14:10.675 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:10.675 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.675 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.675 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:10.675 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.245543+0000", 2024-03-20T09:14:10.675 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1930000000000001e-06 2024-03-20T09:14:10.675 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.675 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.675 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:10.675 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.245560+0000", 2024-03-20T09:14:10.675 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.738e-05 2024-03-20T09:14:10.675 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.675 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.675 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:10.676 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.245583+0000", 2024-03-20T09:14:10.676 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3053000000000001e-05 2024-03-20T09:14:10.676 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.676 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.676 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:10.676 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.245596+0000", 2024-03-20T09:14:10.676 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2830000000000001e-05 2024-03-20T09:14:10.676 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.676 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.676 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:10.676 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.245729+0000", 2024-03-20T09:14:10.676 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013229299999999999 2024-03-20T09:14:10.676 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:10.676 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:10.676 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:10.677 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.246198+0000", 2024-03-20T09:14:10.677 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00046902499999999998 2024-03-20T09:14:10.677 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.677 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:10.677 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.677 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:10.677 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:10.677 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:14:10.677 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:14:10.706 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 39 16 70176 70160 112.422 91.5625 0.0131578 0.00889042 2024-03-20T09:14:10.746 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:10.792 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:14:10.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:10.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:14:10.900 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:14:10.970 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:11.161 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:11 smithi026 ceph-mon[68309]: pgmap v1267: 47 pgs: 47 active+clean; 4.0 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-03-20T09:14:11.357 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:14:11.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:11.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:14:11.430 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:11 smithi122 ceph-mon[73495]: pgmap v1267: 47 pgs: 47 active+clean; 4.0 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-03-20T09:14:11.462 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:14:11.469 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_ops_in_flight 2024-03-20T09:14:11.495 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:11 smithi115 ceph-mon[72476]: pgmap v1267: 47 pgs: 47 active+clean; 4.0 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-03-20T09:14:11.498 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:11.524 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:14:11.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:11.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:14:11.628 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:14:11.660 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:11.683 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:14:11.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:11.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:14:11.707 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:2024-03-20T09:14:11.706995+0000 min lat: 0.00107883 max lat: 2.19794 avg lat: 0.00893656 2024-03-20T09:14:11.707 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T09:14:11.707 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 40 16 71602 71586 111.839 89.125 0.00319 0.00893656 2024-03-20T09:14:11.755 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:14:11.787 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:14:11.865 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:14:11.875 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:12.173 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:12 smithi026 ceph-mon[68309]: 1.7 deep-scrub starts 2024-03-20T09:14:12.173 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:12 smithi026 ceph-mon[68309]: 1.7 deep-scrub ok 2024-03-20T09:14:12.173 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:12 smithi026 ceph-mon[68309]: osdmap e541: 12 total, 12 up, 10 in 2024-03-20T09:14:12.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:12 smithi122 ceph-mon[73495]: 1.7 deep-scrub starts 2024-03-20T09:14:12.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:12 smithi122 ceph-mon[73495]: 1.7 deep-scrub ok 2024-03-20T09:14:12.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:12 smithi122 ceph-mon[73495]: osdmap e541: 12 total, 12 up, 10 in 2024-03-20T09:14:12.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:12 smithi115 ceph-mon[72476]: 1.7 deep-scrub starts 2024-03-20T09:14:12.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:12 smithi115 ceph-mon[72476]: 1.7 deep-scrub ok 2024-03-20T09:14:12.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:12 smithi115 ceph-mon[72476]: osdmap e541: 12 total, 12 up, 10 in 2024-03-20T09:14:12.707 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 41 16 72733 72717 110.835 70.6875 0.00371102 0.00901395 2024-03-20T09:14:13.217 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:14:13.255 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:14:13.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:13.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:14:13.364 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:14:13.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:13 smithi122 ceph-mon[73495]: pgmap v1268: 47 pgs: 47 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-03-20T09:14:13.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:13 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1130359537' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:14:13.437 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:13.477 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:13 smithi026 ceph-mon[68309]: pgmap v1268: 47 pgs: 47 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-03-20T09:14:13.477 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:13 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1130359537' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:14:13.482 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:14:13.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:13.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:14:13.588 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:14:13.642 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:13.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:13 smithi115 ceph-mon[72476]: pgmap v1268: 47 pgs: 47 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-03-20T09:14:13.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:13 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1130359537' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:14:13.688 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:14:13.707 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 42 16 73981 73965 110.053 78 0.00545815 0.00908197 2024-03-20T09:14:13.711 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:14:14.081 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:14:14.081 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-03-20T09:14:14.081 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:14.082 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:74491 4.e 4:706edbfe:::benchmark_data_smithi122_145698_object74490:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e542)", 2024-03-20T09:14:14.082 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:14:14.066513+0000", 2024-03-20T09:14:14.082 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.014225369, 2024-03-20T09:14:14.082 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014261022, 2024-03-20T09:14:14.082 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:14.082 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:14:14.082 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:14:14.082 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:14:14.082 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:14:14.082 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 74491 2024-03-20T09:14:14.082 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:14.082 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:14.083 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:14.083 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:14.083 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:14:14.066513+0000", 2024-03-20T09:14:14.083 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:14.083 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:14.083 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:14.083 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:14.083 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:14:14.066513+0000", 2024-03-20T09:14:14.083 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:14.083 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:14.083 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:14.083 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:14.083 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:14:14.066515+0000", 2024-03-20T09:14:14.083 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.934e-06 2024-03-20T09:14:14.084 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:14.084 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:14.084 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:14.084 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:14:14.066536+0000", 2024-03-20T09:14:14.084 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1569e-05 2024-03-20T09:14:14.084 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:14.084 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:14.084 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:14.084 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:14:14.066551+0000", 2024-03-20T09:14:14.084 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4868000000000001e-05 2024-03-20T09:14:14.084 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:14.084 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:14.084 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:14.084 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:14:14.066558+0000", 2024-03-20T09:14:14.084 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1690000000000003e-06 2024-03-20T09:14:14.085 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:14.085 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:14.085 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:14.085 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:14:14.066627+0000", 2024-03-20T09:14:14.085 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8984000000000002e-05 2024-03-20T09:14:14.085 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:14.085 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:14.085 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:14:14.085 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:14:14.066754+0000", 2024-03-20T09:14:14.085 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012673200000000001 2024-03-20T09:14:14.085 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:14.085 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:14.085 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,6", 2024-03-20T09:14:14.085 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:14:14.066889+0000", 2024-03-20T09:14:14.086 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013473700000000001 2024-03-20T09:14:14.086 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:14.086 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:14.086 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:14.086 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:14:14.069271+0000", 2024-03-20T09:14:14.086 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002382229 2024-03-20T09:14:14.086 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:14.086 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:14.086 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:14:14.086 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:14:14.069436+0000", 2024-03-20T09:14:14.086 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016535200000000001 2024-03-20T09:14:14.086 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:14.086 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:14.086 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:14.086 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:14.087 INFO:teuthology.orchestra.run.smithi122.stdout: ], 2024-03-20T09:14:14.087 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 1 2024-03-20T09:14:14.087 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:14:14.205 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:14 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:14:14.205 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:14 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1130359537' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:14:14.205 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:14 smithi026 ceph-mon[68309]: osdmap e542: 12 total, 12 up, 10 in 2024-03-20T09:14:14.205 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:14 smithi026 ceph-mon[68309]: from='client.25729 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:14:14.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:14 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:14:14.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:14 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1130359537' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:14:14.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:14 smithi122 ceph-mon[73495]: osdmap e542: 12 total, 12 up, 10 in 2024-03-20T09:14:14.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:14 smithi122 ceph-mon[73495]: from='client.25729 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:14:14.652 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:14:14.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:14 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:14:14.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:14 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1130359537' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:14:14.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:14 smithi115 ceph-mon[72476]: osdmap e542: 12 total, 12 up, 10 in 2024-03-20T09:14:14.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:14 smithi115 ceph-mon[72476]: from='client.25729 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:14:14.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:14.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:14:14.707 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 43 16 75238 75222 109.321 78.5625 0.00364489 0.0091313 2024-03-20T09:14:14.761 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:14:14.799 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:14.892 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:14:14.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:14.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:14:14.997 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:14:15.056 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:15.414 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:14:15.415 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:15 smithi026 ceph-mon[68309]: pgmap v1271: 47 pgs: 47 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-03-20T09:14:15.437 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:14:15.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:15.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:14:15.471 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":1270,"stamp":"2024-03-20T09:14:13.184981+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4561895968,"num_objects":69602,"num_object_clones":0,"num_object_copies":208806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69600,"num_write_kb":4454400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":923,"num_bytes_recovered":60751872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73847,"ondisk_log_size":73847,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":16504200,"kb_used_data":13457336,"kb_used_omap":195,"kb_used_meta":3046588,"kb_avail":920947320,"statfs":{"total":959950356480,"available":943050055680,"internally_reserved":0,"allocated":13780312064,"data_stored":13766132658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":200482,"internal_metadata":3119706334},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":555810816,"num_objects":8481,"num_object_clones":0,"num_object_copies":25443,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8481,"num_write_kb":542784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.379556"},"pg_stats":[{"pgid":"1.1e","version":"20'137","reported_seq":455,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:07.125516+0000","last_change":"2024-03-20T09:13:36.531143+0000","last_active":"2024-03-20T09:14:07.125516+0000","last_peered":"2024-03-20T09:14:07.125516+0000","last_clean":"2024-03-20T09:14:07.125516+0000","last_became_active":"2024-03-20T09:13:20.071655+0000","last_became_peered":"2024-03-20T09:13:20.071655+0000","last_unstale":"2024-03-20T09:14:07.125516+0000","last_undegraded":"2024-03-20T09:14:07.125516+0000","last_fullsized":"2024-03-20T09:14:07.125516+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:36.531003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:36.531003+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:44.358559+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'137","reported_seq":622,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:07.130595+0000","last_change":"2024-03-20T09:13:05.291275+0000","last_active":"2024-03-20T09:14:07.130595+0000","last_peered":"2024-03-20T09:14:07.130595+0000","last_clean":"2024-03-20T09:14:07.130595+0000","last_became_active":"2024-03-20T09:07:42.013245+0000","last_became_peered":"2024-03-20T09:07:42.013245+0000","last_unstale":"2024-03-20T09:14:07.130595+0000","last_undegraded":"2024-03-20T09:14:07.130595+0000","last_fullsized":"2024-03-20T09:14:07.130595+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:05.291136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:05.291136+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:17.362626+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"20'137","reported_seq":1072,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:07.130950+0000","last_change":"2024-03-20T09:13:56.707873+0000","last_active":"2024-03-20T09:14:07.130950+0000","last_peered":"2024-03-20T09:14:07.130950+0000","last_clean":"2024-03-20T09:14:07.130950+0000","last_became_active":"2024-03-20T09:07:41.961622+0000","last_became_peered":"2024-03-20T09:07:41.961622+0000","last_unstale":"2024-03-20T09:14:07.130950+0000","last_undegraded":"2024-03-20T09:14:07.130950+0000","last_fullsized":"2024-03-20T09:14:07.130950+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:56.707767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:56.707767+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:17.835273+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.19","version":"20'137","reported_seq":1133,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:07.130598+0000","last_change":"2024-03-20T09:13:11.366034+0000","last_active":"2024-03-20T09:14:07.130598+0000","last_peered":"2024-03-20T09:14:07.130598+0000","last_clean":"2024-03-20T09:14:07.130598+0000","last_became_active":"2024-03-20T09:07:41.962680+0000","last_became_peered":"2024-03-20T09:07:41.962680+0000","last_unstale":"2024-03-20T09:14:07.130598+0000","last_undegraded":"2024-03-20T09:14:07.130598+0000","last_fullsized":"2024-03-20T09:14:07.130598+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:11.365892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:11.365892+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:32.244898+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"20'137","reported_seq":597,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-20T09:14:12.093539+0000","last_change":"2024-03-20T09:13:39.292648+0000","last_active":"2024-03-20T09:14:12.093539+0000","last_peered":"2024-03-20T09:14:12.093539+0000","last_clean":"2024-03-20T09:14:12.093539+0000","last_became_active":"2024-03-20T09:10:35.277481+0000","last_became_peered":"2024-03-20T09:10:35.277481+0000","last_unstale":"2024-03-20T09:14:12.093539+0000","last_undegraded":"2024-03-20T09:14:12.093539+0000","last_fullsized":"2024-03-20T09:14:12.093539+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:39.292556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:39.292556+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:07.108785+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1b","version":"20'137","reported_seq":901,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:07.130956+0000","last_change":"2024-03-20T09:13:44.768551+0000","last_active":"2024-03-20T09:14:07.130956+0000","last_peered":"2024-03-20T09:14:07.130956+0000","last_clean":"2024-03-20T09:14:07.130956+0000","last_became_active":"2024-03-20T09:07:41.962733+0000","last_became_peered":"2024-03-20T09:07:41.962733+0000","last_unstale":"2024-03-20T09:14:07.130956+0000","last_undegraded":"2024-03-20T09:14:07.130956+0000","last_fullsized":"2024-03-20T09:14:07.130956+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:44.768424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:44.768424+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:03.198163+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"540'4410","reported_seq":4460,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:10.203135+0000","last_change":"2024-03-20T09:13:22.063299+0000","last_active":"2024-03-20T09:14:10.203135+0000","last_peered":"2024-03-20T09:14:10.203135+0000","last_clean":"2024-03-20T09:14:10.203135+0000","last_became_active":"2024-03-20T09:13:22.063020+0000","last_became_peered":"2024-03-20T09:13:22.063020+0000","last_unstale":"2024-03-20T09:14:10.203135+0000","last_undegraded":"2024-03-20T09:14:10.203135+0000","last_fullsized":"2024-03-20T09:14:10.203135+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":4410,"log_dups_size":0,"ondisk_log_size":4410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:41.788247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289013760,"num_objects":4410,"num_object_clones":0,"num_object_copies":13230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4410,"num_write_kb":282240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"20'137","reported_seq":596,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-20T09:14:12.093552+0000","last_change":"2024-03-20T09:13:38.313619+0000","last_active":"2024-03-20T09:14:12.093552+0000","last_peered":"2024-03-20T09:14:12.093552+0000","last_clean":"2024-03-20T09:14:12.093552+0000","last_became_active":"2024-03-20T09:10:35.215772+0000","last_became_peered":"2024-03-20T09:10:35.215772+0000","last_unstale":"2024-03-20T09:14:12.093552+0000","last_undegraded":"2024-03-20T09:14:12.093552+0000","last_fullsized":"2024-03-20T09:14:12.093552+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:38.313491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:38.313491+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:46.785027+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"540'4321","reported_seq":4371,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:09.988237+0000","last_change":"2024-03-20T09:13:22.583666+0000","last_active":"2024-03-20T09:14:09.988237+0000","last_peered":"2024-03-20T09:14:09.988237+0000","last_clean":"2024-03-20T09:14:09.988237+0000","last_became_active":"2024-03-20T09:13:22.583317+0000","last_became_peered":"2024-03-20T09:13:22.583317+0000","last_unstale":"2024-03-20T09:14:09.988237+0000","last_undegraded":"2024-03-20T09:14:09.988237+0000","last_fullsized":"2024-03-20T09:14:09.988237+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":4321,"log_dups_size":0,"ondisk_log_size":4321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:29.205626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283181056,"num_objects":4321,"num_object_clones":0,"num_object_copies":12963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4321,"num_write_kb":276544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,8],"acting":[3,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"20'137","reported_seq":1146,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:07.126973+0000","last_change":"2024-03-20T09:13:05.129380+0000","last_active":"2024-03-20T09:14:07.126973+0000","last_peered":"2024-03-20T09:14:07.126973+0000","last_clean":"2024-03-20T09:14:07.126973+0000","last_became_active":"2024-03-20T09:10:34.099414+0000","last_became_peered":"2024-03-20T09:10:34.099414+0000","last_unstale":"2024-03-20T09:14:07.126973+0000","last_undegraded":"2024-03-20T09:14:07.126973+0000","last_fullsized":"2024-03-20T09:14:07.126973+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:05.129288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:05.129288+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:16.260612+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"540'4243","reported_seq":6149,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:08.444064+0000","last_change":"2024-03-20T09:13:46.203083+0000","last_active":"2024-03-20T09:14:08.444064+0000","last_peered":"2024-03-20T09:14:08.444064+0000","last_clean":"2024-03-20T09:14:08.444064+0000","last_became_active":"2024-03-20T09:13:46.198893+0000","last_became_peered":"2024-03-20T09:13:46.198893+0000","last_unstale":"2024-03-20T09:14:08.444064+0000","last_undegraded":"2024-03-20T09:14:08.444064+0000","last_fullsized":"2024-03-20T09:14:08.444064+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":4243,"log_dups_size":0,"ondisk_log_size":4243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:41.040830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278069248,"num_objects":4243,"num_object_clones":0,"num_object_copies":12729,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4243,"num_write_kb":271552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":922,"num_bytes_recovered":60424192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,11],"acting":[2,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.a","version":"20'137","reported_seq":960,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:07.127255+0000","last_change":"2024-03-20T09:13:57.530523+0000","last_active":"2024-03-20T09:14:07.127255+0000","last_peered":"2024-03-20T09:14:07.127255+0000","last_clean":"2024-03-20T09:14:07.127255+0000","last_became_active":"2024-03-20T09:07:52.618107+0000","last_became_peered":"2024-03-20T09:07:52.618107+0000","last_unstale":"2024-03-20T09:14:07.127255+0000","last_undegraded":"2024-03-20T09:14:07.127255+0000","last_fullsized":"2024-03-20T09:14:07.127255+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:57.530396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:57.530396+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:10.089743+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"540'4328","reported_seq":4378,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:09.739351+0000","last_change":"2024-03-20T09:13:22.064363+0000","last_active":"2024-03-20T09:14:09.739351+0000","last_peered":"2024-03-20T09:14:09.739351+0000","last_clean":"2024-03-20T09:14:09.739351+0000","last_became_active":"2024-03-20T09:13:22.063975+0000","last_became_peered":"2024-03-20T09:13:22.063975+0000","last_unstale":"2024-03-20T09:14:09.739351+0000","last_undegraded":"2024-03-20T09:14:09.739351+0000","last_fullsized":"2024-03-20T09:14:09.739351+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":4328,"log_dups_size":0,"ondisk_log_size":4328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:45.622073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283639808,"num_objects":4328,"num_object_clones":0,"num_object_copies":12984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4328,"num_write_kb":276992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"20'137","reported_seq":1115,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:07.125552+0000","last_change":"2024-03-20T09:13:14.395564+0000","last_active":"2024-03-20T09:14:07.125552+0000","last_peered":"2024-03-20T09:14:07.125552+0000","last_clean":"2024-03-20T09:14:07.125552+0000","last_became_active":"2024-03-20T09:12:14.306384+0000","last_became_peered":"2024-03-20T09:12:14.306384+0000","last_unstale":"2024-03-20T09:14:07.125552+0000","last_undegraded":"2024-03-20T09:14:07.125552+0000","last_fullsized":"2024-03-20T09:14:07.125552+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:14.395469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:14.395469+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:24.496429+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"540'4308","reported_seq":4353,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:10.002783+0000","last_change":"2024-03-20T09:13:24.064404+0000","last_active":"2024-03-20T09:14:10.002783+0000","last_peered":"2024-03-20T09:14:10.002783+0000","last_clean":"2024-03-20T09:14:10.002783+0000","last_became_active":"2024-03-20T09:13:24.064091+0000","last_became_peered":"2024-03-20T09:13:24.064091+0000","last_unstale":"2024-03-20T09:14:10.002783+0000","last_undegraded":"2024-03-20T09:14:10.002783+0000","last_fullsized":"2024-03-20T09:14:10.002783+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":4308,"log_dups_size":0,"ondisk_log_size":4308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:33.687560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282329088,"num_objects":4308,"num_object_clones":0,"num_object_copies":12924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4308,"num_write_kb":275712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"20'137","reported_seq":1011,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:07.130834+0000","last_change":"2024-03-20T09:13:10.343309+0000","last_active":"2024-03-20T09:14:07.130834+0000","last_peered":"2024-03-20T09:14:07.130834+0000","last_clean":"2024-03-20T09:14:07.130834+0000","last_became_active":"2024-03-20T09:07:37.003722+0000","last_became_peered":"2024-03-20T09:07:37.003722+0000","last_unstale":"2024-03-20T09:14:07.130834+0000","last_undegraded":"2024-03-20T09:14:07.130834+0000","last_fullsized":"2024-03-20T09:14:07.130834+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:10.343198+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:07:06.480160+0000","last_clean_scrub_stamp":"2024-03-20T09:13:10.343198+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:33.639911+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.3","version":"540'4376","reported_seq":4426,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:09.761239+0000","last_change":"2024-03-20T09:13:22.064621+0000","last_active":"2024-03-20T09:14:09.761239+0000","last_peered":"2024-03-20T09:14:09.761239+0000","last_clean":"2024-03-20T09:14:09.761239+0000","last_became_active":"2024-03-20T09:13:22.064275+0000","last_became_peered":"2024-03-20T09:13:22.064275+0000","last_unstale":"2024-03-20T09:14:09.761239+0000","last_undegraded":"2024-03-20T09:14:09.761239+0000","last_fullsized":"2024-03-20T09:14:09.761239+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":4376,"log_dups_size":0,"ondisk_log_size":4376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:40.477405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":286785536,"num_objects":4376,"num_object_clones":0,"num_object_copies":13128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4376,"num_write_kb":280064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"20'137","reported_seq":1028,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:07.131017+0000","last_change":"2024-03-20T09:13:12.359759+0000","last_active":"2024-03-20T09:14:07.131017+0000","last_peered":"2024-03-20T09:14:07.131017+0000","last_clean":"2024-03-20T09:14:07.131017+0000","last_became_active":"2024-03-20T09:07:52.593671+0000","last_became_peered":"2024-03-20T09:07:52.593671+0000","last_unstale":"2024-03-20T09:14:07.131017+0000","last_undegraded":"2024-03-20T09:14:07.131017+0000","last_fullsized":"2024-03-20T09:14:07.131017+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:12.359617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:12.359617+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:24.685959+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.6","version":"540'4375","reported_seq":4425,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:10.203880+0000","last_change":"2024-03-20T09:13:22.067664+0000","last_active":"2024-03-20T09:14:10.203880+0000","last_peered":"2024-03-20T09:14:10.203880+0000","last_clean":"2024-03-20T09:14:10.203880+0000","last_became_active":"2024-03-20T09:13:22.067362+0000","last_became_peered":"2024-03-20T09:13:22.067362+0000","last_unstale":"2024-03-20T09:14:10.203880+0000","last_undegraded":"2024-03-20T09:14:10.203880+0000","last_fullsized":"2024-03-20T09:14:10.203880+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":4375,"log_dups_size":0,"ondisk_log_size":4375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:46.912906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":286720000,"num_objects":4375,"num_object_clones":0,"num_object_copies":13125,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4375,"num_write_kb":280000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"20'137","reported_seq":1028,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:07.130770+0000","last_change":"2024-03-20T09:13:49.033462+0000","last_active":"2024-03-20T09:14:07.130770+0000","last_peered":"2024-03-20T09:14:07.130770+0000","last_clean":"2024-03-20T09:14:07.130770+0000","last_became_active":"2024-03-20T09:12:02.221686+0000","last_became_peered":"2024-03-20T09:12:02.221686+0000","last_unstale":"2024-03-20T09:14:07.130770+0000","last_undegraded":"2024-03-20T09:14:07.130770+0000","last_fullsized":"2024-03-20T09:14:07.130770+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:49.033327+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:18.788771+0000","last_clean_scrub_stamp":"2024-03-20T09:13:49.033327+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:10.707005+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,8],"acting":[1,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"540'4295","reported_seq":4345,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:10.704053+0000","last_change":"2024-03-20T09:13:22.063240+0000","last_active":"2024-03-20T09:14:10.704053+0000","last_peered":"2024-03-20T09:14:10.704053+0000","last_clean":"2024-03-20T09:14:10.704053+0000","last_became_active":"2024-03-20T09:13:22.062992+0000","last_became_peered":"2024-03-20T09:13:22.062992+0000","last_unstale":"2024-03-20T09:14:10.704053+0000","last_undegraded":"2024-03-20T09:14:10.704053+0000","last_fullsized":"2024-03-20T09:14:10.704053+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":4295,"log_dups_size":0,"ondisk_log_size":4295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:36.289001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281477120,"num_objects":4295,"num_object_clones":0,"num_object_copies":12885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4295,"num_write_kb":274880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":1086,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:07.125437+0000","last_change":"2024-03-20T09:13:03.215035+0000","last_active":"2024-03-20T09:14:07.125437+0000","last_peered":"2024-03-20T09:14:07.125437+0000","last_clean":"2024-03-20T09:14:07.125437+0000","last_became_active":"2024-03-20T09:12:14.305989+0000","last_became_peered":"2024-03-20T09:12:14.305989+0000","last_unstale":"2024-03-20T09:14:07.125437+0000","last_undegraded":"2024-03-20T09:14:07.125437+0000","last_fullsized":"2024-03-20T09:14:07.125437+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:03.214902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:03.214902+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:13.441726+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.7","version":"540'4330","reported_seq":4380,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:10.207598+0000","last_change":"2024-03-20T09:13:22.066405+0000","last_active":"2024-03-20T09:14:10.207598+0000","last_peered":"2024-03-20T09:14:10.207598+0000","last_clean":"2024-03-20T09:14:10.207598+0000","last_became_active":"2024-03-20T09:13:22.065656+0000","last_became_peered":"2024-03-20T09:13:22.065656+0000","last_unstale":"2024-03-20T09:14:10.207598+0000","last_undegraded":"2024-03-20T09:14:10.207598+0000","last_fullsized":"2024-03-20T09:14:10.207598+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":4330,"log_dups_size":0,"ondisk_log_size":4330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:23.500958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283770880,"num_objects":4330,"num_object_clones":0,"num_object_copies":12990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4330,"num_write_kb":277120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,0],"acting":[1,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"20'137","reported_seq":1336,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:07.133378+0000","last_change":"2024-03-20T09:13:40.818653+0000","last_active":"2024-03-20T09:14:07.133378+0000","last_peered":"2024-03-20T09:14:07.133378+0000","last_clean":"2024-03-20T09:14:07.133378+0000","last_became_active":"2024-03-20T09:12:14.313541+0000","last_became_peered":"2024-03-20T09:12:14.313541+0000","last_unstale":"2024-03-20T09:14:07.133378+0000","last_undegraded":"2024-03-20T09:14:07.133378+0000","last_fullsized":"2024-03-20T09:14:07.133378+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:40.818542+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:19.513511+0000","last_clean_scrub_stamp":"2024-03-20T09:13:40.818542+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:01.397421+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"540'4300","reported_seq":4355,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:10.180244+0000","last_change":"2024-03-20T09:13:24.443379+0000","last_active":"2024-03-20T09:14:10.180244+0000","last_peered":"2024-03-20T09:14:10.180244+0000","last_clean":"2024-03-20T09:14:10.180244+0000","last_became_active":"2024-03-20T09:13:24.443054+0000","last_became_peered":"2024-03-20T09:13:24.443054+0000","last_unstale":"2024-03-20T09:14:10.180244+0000","last_undegraded":"2024-03-20T09:14:10.180244+0000","last_fullsized":"2024-03-20T09:14:10.180244+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":4300,"log_dups_size":0,"ondisk_log_size":4300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:41.617061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281804800,"num_objects":4300,"num_object_clones":0,"num_object_copies":12900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4300,"num_write_kb":275200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"20'137","reported_seq":1032,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:07.130727+0000","last_change":"2024-03-20T09:13:14.428508+0000","last_active":"2024-03-20T09:14:07.130727+0000","last_peered":"2024-03-20T09:14:07.130727+0000","last_clean":"2024-03-20T09:14:07.130727+0000","last_became_active":"2024-03-20T09:06:02.923630+0000","last_became_peered":"2024-03-20T09:06:02.923630+0000","last_unstale":"2024-03-20T09:14:07.130727+0000","last_undegraded":"2024-03-20T09:14:07.130727+0000","last_fullsized":"2024-03-20T09:14:07.130727+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:14.428416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:14.428416+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:27.674669+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2],"acting":[10,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2","version":"541'4582","reported_seq":4634,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-20T09:14:12.135241+0000","last_change":"2024-03-20T09:13:22.584008+0000","last_active":"2024-03-20T09:14:12.135241+0000","last_peered":"2024-03-20T09:14:12.135241+0000","last_clean":"2024-03-20T09:14:12.135241+0000","last_became_active":"2024-03-20T09:13:22.583529+0000","last_became_peered":"2024-03-20T09:13:22.583529+0000","last_unstale":"2024-03-20T09:14:12.135241+0000","last_undegraded":"2024-03-20T09:14:12.135241+0000","last_fullsized":"2024-03-20T09:14:12.135241+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":4582,"log_dups_size":0,"ondisk_log_size":4582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:36.926945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300285952,"num_objects":4582,"num_object_clones":0,"num_object_copies":13746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4582,"num_write_kb":293248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,8],"acting":[5,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"20'137","reported_seq":1032,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:10.560978+0000","last_change":"2024-03-20T09:14:10.561025+0000","last_active":"2024-03-20T09:14:10.560978+0000","last_peered":"2024-03-20T09:14:10.560978+0000","last_clean":"2024-03-20T09:14:10.560978+0000","last_became_active":"2024-03-20T09:13:27.087760+0000","last_became_peered":"2024-03-20T09:13:27.087760+0000","last_unstale":"2024-03-20T09:14:10.560978+0000","last_undegraded":"2024-03-20T09:14:10.560978+0000","last_fullsized":"2024-03-20T09:14:10.560978+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:14:10.560890+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:14:10.560890+0000","last_clean_scrub_stamp":"2024-03-20T09:14:10.560890+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:11.348848+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"540'4445","reported_seq":4495,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:11.217431+0000","last_change":"2024-03-20T09:13:22.064771+0000","last_active":"2024-03-20T09:14:11.217431+0000","last_peered":"2024-03-20T09:14:11.217431+0000","last_clean":"2024-03-20T09:14:11.217431+0000","last_became_active":"2024-03-20T09:13:22.064461+0000","last_became_peered":"2024-03-20T09:13:22.064461+0000","last_unstale":"2024-03-20T09:14:11.217431+0000","last_undegraded":"2024-03-20T09:14:11.217431+0000","last_fullsized":"2024-03-20T09:14:11.217431+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":4445,"log_dups_size":0,"ondisk_log_size":4445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:25.759321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291307520,"num_objects":4445,"num_object_clones":0,"num_object_copies":13335,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4445,"num_write_kb":284480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,1],"acting":[10,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.b","version":"20'137","reported_seq":900,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:07.125501+0000","last_change":"2024-03-20T09:13:46.614783+0000","last_active":"2024-03-20T09:14:07.125501+0000","last_peered":"2024-03-20T09:14:07.125501+0000","last_clean":"2024-03-20T09:14:07.125501+0000","last_became_active":"2024-03-20T09:13:20.070596+0000","last_became_peered":"2024-03-20T09:13:20.070596+0000","last_unstale":"2024-03-20T09:14:07.125501+0000","last_undegraded":"2024-03-20T09:14:07.125501+0000","last_fullsized":"2024-03-20T09:14:07.125501+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:46.614691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:46.614691+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:46.810325+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.b","version":"540'4345","reported_seq":4395,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:11.689293+0000","last_change":"2024-03-20T09:13:22.065041+0000","last_active":"2024-03-20T09:14:11.689293+0000","last_peered":"2024-03-20T09:14:11.689293+0000","last_clean":"2024-03-20T09:14:11.689293+0000","last_became_active":"2024-03-20T09:13:22.064687+0000","last_became_peered":"2024-03-20T09:13:22.064687+0000","last_unstale":"2024-03-20T09:14:11.689293+0000","last_undegraded":"2024-03-20T09:14:11.689293+0000","last_fullsized":"2024-03-20T09:14:11.689293+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":4345,"log_dups_size":0,"ondisk_log_size":4345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:35.394465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284753920,"num_objects":4345,"num_object_clones":0,"num_object_copies":13035,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4345,"num_write_kb":278080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"20'137","reported_seq":560,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:08.589589+0000","last_change":"2024-03-20T09:14:08.589630+0000","last_active":"2024-03-20T09:14:08.589589+0000","last_peered":"2024-03-20T09:14:08.589589+0000","last_clean":"2024-03-20T09:14:08.589589+0000","last_became_active":"2024-03-20T09:12:14.311319+0000","last_became_peered":"2024-03-20T09:12:14.311319+0000","last_unstale":"2024-03-20T09:14:08.589589+0000","last_undegraded":"2024-03-20T09:14:08.589589+0000","last_fullsized":"2024-03-20T09:14:08.589589+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:14:08.589510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:14:08.589510+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:21.094562+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"540'4349","reported_seq":4398,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:10.704405+0000","last_change":"2024-03-20T09:13:22.064279+0000","last_active":"2024-03-20T09:14:10.704405+0000","last_peered":"2024-03-20T09:14:10.704405+0000","last_clean":"2024-03-20T09:14:10.704405+0000","last_became_active":"2024-03-20T09:13:22.063927+0000","last_became_peered":"2024-03-20T09:13:22.063927+0000","last_unstale":"2024-03-20T09:14:10.704405+0000","last_undegraded":"2024-03-20T09:14:10.704405+0000","last_fullsized":"2024-03-20T09:14:10.704405+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":4349,"log_dups_size":0,"ondisk_log_size":4349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:37.913884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285016064,"num_objects":4349,"num_object_clones":0,"num_object_copies":13047,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4349,"num_write_kb":278336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"20'137","reported_seq":1045,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:07.127290+0000","last_change":"2024-03-20T09:13:53.635081+0000","last_active":"2024-03-20T09:14:07.127290+0000","last_peered":"2024-03-20T09:14:07.127290+0000","last_clean":"2024-03-20T09:14:07.127290+0000","last_became_active":"2024-03-20T09:10:35.119706+0000","last_became_peered":"2024-03-20T09:10:35.119706+0000","last_unstale":"2024-03-20T09:14:07.127290+0000","last_undegraded":"2024-03-20T09:14:07.127290+0000","last_fullsized":"2024-03-20T09:14:07.127290+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:53.634986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:53.634986+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:06.460977+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"540'4273","reported_seq":4328,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:09.760135+0000","last_change":"2024-03-20T09:13:24.065781+0000","last_active":"2024-03-20T09:14:09.760135+0000","last_peered":"2024-03-20T09:14:09.760135+0000","last_clean":"2024-03-20T09:14:09.760135+0000","last_became_active":"2024-03-20T09:13:24.065349+0000","last_became_peered":"2024-03-20T09:13:24.065349+0000","last_unstale":"2024-03-20T09:14:09.760135+0000","last_undegraded":"2024-03-20T09:14:09.760135+0000","last_fullsized":"2024-03-20T09:14:09.760135+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":4273,"log_dups_size":0,"ondisk_log_size":4273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:29.930666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280035328,"num_objects":4273,"num_object_clones":0,"num_object_copies":12819,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4273,"num_write_kb":273472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"20'137","reported_seq":1067,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:07.129809+0000","last_change":"2024-03-20T09:13:27.088076+0000","last_active":"2024-03-20T09:14:07.129809+0000","last_peered":"2024-03-20T09:14:07.129809+0000","last_clean":"2024-03-20T09:14:07.129809+0000","last_became_active":"2024-03-20T09:13:27.087686+0000","last_became_peered":"2024-03-20T09:13:27.087686+0000","last_unstale":"2024-03-20T09:14:07.129809+0000","last_undegraded":"2024-03-20T09:14:07.129809+0000","last_fullsized":"2024-03-20T09:14:07.129809+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:09.755064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:09.755064+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:33.274547+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"540'4320","reported_seq":4370,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:09.744225+0000","last_change":"2024-03-20T09:13:22.064047+0000","last_active":"2024-03-20T09:14:09.744225+0000","last_peered":"2024-03-20T09:14:09.744225+0000","last_clean":"2024-03-20T09:14:09.744225+0000","last_became_active":"2024-03-20T09:13:22.063673+0000","last_became_peered":"2024-03-20T09:13:22.063673+0000","last_unstale":"2024-03-20T09:14:09.744225+0000","last_undegraded":"2024-03-20T09:14:09.744225+0000","last_fullsized":"2024-03-20T09:14:09.744225+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":4320,"log_dups_size":0,"ondisk_log_size":4320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:39.125308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283115520,"num_objects":4320,"num_object_clones":0,"num_object_copies":12960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4320,"num_write_kb":276480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"20'137","reported_seq":619,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:07.130523+0000","last_change":"2024-03-20T09:13:38.035423+0000","last_active":"2024-03-20T09:14:07.130523+0000","last_peered":"2024-03-20T09:14:07.130523+0000","last_clean":"2024-03-20T09:14:07.130523+0000","last_became_active":"2024-03-20T09:12:14.307864+0000","last_became_peered":"2024-03-20T09:12:14.307864+0000","last_unstale":"2024-03-20T09:14:07.130523+0000","last_undegraded":"2024-03-20T09:14:07.130523+0000","last_fullsized":"2024-03-20T09:14:07.130523+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:38.035304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:38.035304+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:57.965009+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"20'137","reported_seq":937,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-20T09:14:12.094164+0000","last_change":"2024-03-20T09:13:04.270196+0000","last_active":"2024-03-20T09:14:12.094164+0000","last_peered":"2024-03-20T09:14:12.094164+0000","last_clean":"2024-03-20T09:14:12.094164+0000","last_became_active":"2024-03-20T09:10:34.096758+0000","last_became_peered":"2024-03-20T09:10:34.096758+0000","last_unstale":"2024-03-20T09:14:12.094164+0000","last_undegraded":"2024-03-20T09:14:12.094164+0000","last_fullsized":"2024-03-20T09:14:12.094164+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:04.270076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:04.270076+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:18.468974+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"20'137","reported_seq":1008,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:07.130672+0000","last_change":"2024-03-20T09:13:50.092495+0000","last_active":"2024-03-20T09:14:07.130672+0000","last_peered":"2024-03-20T09:14:07.130672+0000","last_clean":"2024-03-20T09:14:07.130672+0000","last_became_active":"2024-03-20T09:04:04.103645+0000","last_became_peered":"2024-03-20T09:04:04.103645+0000","last_unstale":"2024-03-20T09:14:07.130672+0000","last_undegraded":"2024-03-20T09:14:07.130672+0000","last_fullsized":"2024-03-20T09:14:07.130672+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:50.092343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:50.092343+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:51.393395+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"20'137","reported_seq":463,"reported_epoch":541,"state":"active+clean","last_fresh":"2024-03-20T09:14:12.093753+0000","last_change":"2024-03-20T09:13:05.305611+0000","last_active":"2024-03-20T09:14:12.093753+0000","last_peered":"2024-03-20T09:14:12.093753+0000","last_clean":"2024-03-20T09:14:12.093753+0000","last_became_active":"2024-03-20T09:10:34.097191+0000","last_became_peered":"2024-03-20T09:10:34.097191+0000","last_unstale":"2024-03-20T09:14:12.093753+0000","last_undegraded":"2024-03-20T09:14:12.093753+0000","last_fullsized":"2024-03-20T09:14:12.093753+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:05.305514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:05.305514+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:31.997881+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"20'137","reported_seq":872,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:07.133502+0000","last_change":"2024-03-20T09:13:38.808687+0000","last_active":"2024-03-20T09:14:07.133502+0000","last_peered":"2024-03-20T09:14:07.133502+0000","last_clean":"2024-03-20T09:14:07.133502+0000","last_became_active":"2024-03-20T09:13:27.087859+0000","last_became_peered":"2024-03-20T09:13:27.087859+0000","last_unstale":"2024-03-20T09:14:07.133502+0000","last_undegraded":"2024-03-20T09:14:07.133502+0000","last_fullsized":"2024-03-20T09:14:07.133502+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:38.808565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:38.808565+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:51.238334+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,3],"acting":[0,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"20'137","reported_seq":697,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:07.130321+0000","last_change":"2024-03-20T09:13:27.088201+0000","last_active":"2024-03-20T09:14:07.130321+0000","last_peered":"2024-03-20T09:14:07.130321+0000","last_clean":"2024-03-20T09:14:07.130321+0000","last_became_active":"2024-03-20T09:13:27.087760+0000","last_became_peered":"2024-03-20T09:13:27.087760+0000","last_unstale":"2024-03-20T09:14:07.130321+0000","last_undegraded":"2024-03-20T09:14:07.130321+0000","last_fullsized":"2024-03-20T09:14:07.130321+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:10.182025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:10.182025+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:40.149783+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"20'137","reported_seq":1023,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:07.131219+0000","last_change":"2024-03-20T09:13:48.832256+0000","last_active":"2024-03-20T09:14:07.131219+0000","last_peered":"2024-03-20T09:14:07.131219+0000","last_clean":"2024-03-20T09:14:07.131219+0000","last_became_active":"2024-03-20T09:10:35.095339+0000","last_became_peered":"2024-03-20T09:10:35.095339+0000","last_unstale":"2024-03-20T09:14:07.131219+0000","last_undegraded":"2024-03-20T09:14:07.131219+0000","last_fullsized":"2024-03-20T09:14:07.131219+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:48.831557+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:23.640533+0000","last_clean_scrub_stamp":"2024-03-20T09:13:48.831557+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:09.728838+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"20'137","reported_seq":1084,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:07.126946+0000","last_change":"2024-03-20T09:12:52.043287+0000","last_active":"2024-03-20T09:14:07.126946+0000","last_peered":"2024-03-20T09:14:07.126946+0000","last_clean":"2024-03-20T09:14:07.126946+0000","last_became_active":"2024-03-20T09:04:14.208448+0000","last_became_peered":"2024-03-20T09:04:14.208448+0000","last_unstale":"2024-03-20T09:14:07.126946+0000","last_undegraded":"2024-03-20T09:14:07.126946+0000","last_fullsized":"2024-03-20T09:14:07.126946+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:12:52.043144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:12:52.043144+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:17.347869+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"20'137","reported_seq":349,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:07.133109+0000","last_change":"2024-03-20T09:13:04.628974+0000","last_active":"2024-03-20T09:14:07.133109+0000","last_peered":"2024-03-20T09:14:07.133109+0000","last_clean":"2024-03-20T09:14:07.133109+0000","last_became_active":"2024-03-20T09:12:14.307581+0000","last_became_peered":"2024-03-20T09:12:14.307581+0000","last_unstale":"2024-03-20T09:14:07.133109+0000","last_undegraded":"2024-03-20T09:14:07.133109+0000","last_fullsized":"2024-03-20T09:14:07.133109+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:04.628862+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:07:06.480160+0000","last_clean_scrub_stamp":"2024-03-20T09:13:04.628862+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:10.875071+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"20'137","reported_seq":628,"reported_epoch":540,"state":"active+clean","last_fresh":"2024-03-20T09:14:07.130213+0000","last_change":"2024-03-20T09:13:47.938675+0000","last_active":"2024-03-20T09:14:07.130213+0000","last_peered":"2024-03-20T09:14:07.130213+0000","last_clean":"2024-03-20T09:14:07.130213+0000","last_became_active":"2024-03-20T09:13:20.074439+0000","last_became_peered":"2024-03-20T09:13:20.074439+0000","last_unstale":"2024-03-20T09:14:07.130213+0000","last_undegraded":"2024-03-20T09:14:07.130213+0000","last_fullsized":"2024-03-20T09:14:07.130213+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:47.938552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:47.938552+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:06.649880+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,11],"acting":[1,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":4561305600,"num_objects":69600,"num_object_clones":0,"num_object_copies":208800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69600,"num_write_kb":4454400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":922,"num_bytes_recovered":60424192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13819576320,"data_stored":13819576320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69600,"ondisk_log_size":69600,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4247,"ondisk_log_size":4247,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860633,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2560792,"kb_used_data":2214480,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":91184360,"statfs":{"total":95995035648,"available":93372784640,"internally_reserved":0,"allocated":2267627520,"data_stored":2266210645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":354613700},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":89,"seq":382252089567,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1374872,"kb_used_data":1127120,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":92370280,"statfs":{"total":95995035648,"available":94587166720,"internally_reserved":0,"allocated":1154170880,"data_stored":1152754005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":253688256},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":0,"seq":343597383911,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":70,"seq":300647710955,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":820632,"kb_used_data":582608,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":92924520,"statfs":{"total":95995035648,"available":95154708480,"internally_reserved":0,"allocated":596590592,"data_stored":595173717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":243726790},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":60,"seq":257698038000,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1577376,"kb_used_data":1131096,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":92167776,"statfs":{"total":95995035648,"available":94379802624,"internally_reserved":0,"allocated":1158242304,"data_stored":1156821877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":456980938},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332341,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1995608,"kb_used_data":1675664,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":91749544,"statfs":{"total":95995035648,"available":93951533056,"internally_reserved":0,"allocated":1715879936,"data_stored":1714463061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":327612870},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575119,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2934936,"kb_used_data":2552528,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":90810216,"statfs":{"total":95995035648,"available":92989661184,"internally_reserved":0,"allocated":2613788672,"data_stored":2612371797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":391576006},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738638,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1543448,"kb_used_data":1142992,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":92201704,"statfs":{"total":95995035648,"available":94414544896,"internally_reserved":0,"allocated":1170423808,"data_stored":1169006933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411592,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1017376,"kb_used_data":805720,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":92727776,"statfs":{"total":95995035648,"available":94953242624,"internally_reserved":0,"allocated":825057280,"data_stored":823636853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149956,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1685464,"kb_used_data":1369680,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":92059688,"statfs":{"total":95995035648,"available":94269120512,"internally_reserved":0,"allocated":1402552320,"data_stored":1401135445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":323353031},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953730,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":443,"seq":1902670512173,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":993696,"kb_used_data":855448,"kb_used_omap":181,"kb_used_meta":138058,"kb_avail":92751456,"statfs":{"total":95995035648,"available":94977490944,"internally_reserved":0,"allocated":875978752,"data_stored":874558325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":186131,"internal_metadata":141371629},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1167785984,"data_stored":1167785984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2629763072,"data_stored":2629763072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":829095936,"data_stored":829095936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1413808128,"data_stored":1413808128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":884867072,"data_stored":884867072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1720582144,"data_stored":1720582144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1161560064,"data_stored":1161560064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":592707584,"data_stored":592707584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1153105920,"data_stored":1153105920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2266300416,"data_stored":2266300416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:14:15.546 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:14:15.578 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:15 smithi115 ceph-mon[72476]: pgmap v1271: 47 pgs: 47 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-03-20T09:14:15.583 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:15.611 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:14:15.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:15.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:14:15.699 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:15 smithi122 ceph-mon[73495]: pgmap v1271: 47 pgs: 47 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-03-20T09:14:15.707 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 44 16 76544 76528 108.691 81.625 0.00299569 0.00919155 2024-03-20T09:14:15.718 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:14:15.728 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_blocked_ops 2024-03-20T09:14:15.746 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:15.767 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:14:15.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:15.771 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:14:15.872 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:14:15.901 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:15.925 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:14:15.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:15.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:14:16.031 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:14:16.063 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:16.071 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:14:16.090 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:14:16.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:16.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:14:16.197 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:14:16.246 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:16.685 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:14:16.707 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 45 16 77775 77759 107.985 76.9375 0.00298546 0.00925594 2024-03-20T09:14:16.905 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:14:16.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:16.909 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:14:17.010 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:14:17.061 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:17.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:17 smithi026 ceph-mon[68309]: pgmap v1272: 47 pgs: 47 active+clean; 4.6 GiB data, 17 GiB used, 878 GiB / 894 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-03-20T09:14:17.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:17 smithi026 ceph-mon[68309]: osdmap e543: 12 total, 12 up, 10 in 2024-03-20T09:14:17.533 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:17 smithi122 ceph-mon[73495]: pgmap v1272: 47 pgs: 47 active+clean; 4.6 GiB data, 17 GiB used, 878 GiB / 894 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-03-20T09:14:17.533 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:17 smithi122 ceph-mon[73495]: osdmap e543: 12 total, 12 up, 10 in 2024-03-20T09:14:17.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:17 smithi115 ceph-mon[72476]: pgmap v1272: 47 pgs: 47 active+clean; 4.6 GiB data, 17 GiB used, 878 GiB / 894 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-03-20T09:14:17.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:17 smithi115 ceph-mon[72476]: osdmap e543: 12 total, 12 up, 10 in 2024-03-20T09:14:17.707 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 46 16 79092 79076 107.427 82.3125 0.0069493 0.00930427 2024-03-20T09:14:17.922 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:14:17.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:17.925 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:14:18.026 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:14:18.056 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:18.079 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:14:18.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:18.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:14:18.183 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:14:18.219 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:18.241 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:14:18.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:18.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:14:18.346 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:14:18.379 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:18.413 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:14:18.417 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:18.418 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:14:18.474 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:14:18.519 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:14:18.548 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:18.567 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:14:18.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:18.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:14:18.671 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:14:18.701 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:18.707 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 47 16 80433 80417 106.925 83.8125 0.0132094 0.009347 2024-03-20T09:14:18.721 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:14:18.722 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:14:18.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:18.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:14:18.827 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:14:18.859 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:18.883 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:14:18.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:18.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:14:18.987 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:14:19.019 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:19.044 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:14:19.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:19.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:14:19.132 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:14:19.132 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:14:19.132 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-03-20T09:14:19.132 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-03-20T09:14:19.132 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:14:19.147 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:14:19.177 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:19.198 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:14:19.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:19.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:14:19.302 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:14:19.319 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:14:19.354 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:19.375 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:14:19.386 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:14:19.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:19.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:14:19.431 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:14:19.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:19 smithi122 ceph-mon[73495]: pgmap v1274: 47 pgs: 47 active+clean; 4.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-03-20T09:14:19.468 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:14:19.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:19 smithi026 ceph-mon[68309]: pgmap v1274: 47 pgs: 47 active+clean; 4.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-03-20T09:14:19.493 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:14:19.514 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:14:19.553 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:14:19.559 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:19.588 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:14:19.624 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:14:19.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:19 smithi115 ceph-mon[72476]: pgmap v1274: 47 pgs: 47 active+clean; 4.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-03-20T09:14:19.662 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:14:19.707 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 48 16 81629 81613 106.254 74.75 0.0401822 0.00940674 2024-03-20T09:14:19.710 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:14:19.755 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:14:19.789 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:14:19.799 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:14:19.799 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:14:19.799 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:14:19.800 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:14:19.800 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:14:19.800 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:14:19.800 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:14:19.800 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:14:19.800 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:14:19.800 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:14:19.800 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:14:19.800 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:14:20.209 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:14:20.233 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:14:20.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:20.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:14:20.353 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:14:20.395 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:20.527 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:14:20.528 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:20 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1328849679' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:14:20.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:20.534 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:14:20.570 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_historic_ops 2024-03-20T09:14:20.637 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:14:20.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:20 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1328849679' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:14:20.692 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:20.710 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 49 16 82901 82885 105.703 79.5 0.0128989 0.00945624 2024-03-20T09:14:20.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:20 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1328849679' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:14:20.729 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:14:20.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:20.736 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:14:20.836 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:14:20.869 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:20.892 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:14:20.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:20.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:14:20.979 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:14:20.997 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:14:21.042 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:21.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:21 smithi122 ceph-mon[73495]: pgmap v1275: 47 pgs: 47 active+clean; 4.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-03-20T09:14:21.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:21 smithi115 ceph-mon[72476]: pgmap v1275: 47 pgs: 47 active+clean; 4.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-03-20T09:14:21.710 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 50 16 84207 84191 105.222 81.625 0.0109219 0.00950002 2024-03-20T09:14:21.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:21 smithi026 ceph-mon[68309]: pgmap v1275: 47 pgs: 47 active+clean; 4.8 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-03-20T09:14:21.798 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:14:21.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:21.802 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:14:21.903 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:14:21.939 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:21.965 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:14:21.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:21.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:14:22.073 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:14:22.138 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:22.608 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:14:22.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:22.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:14:22.711 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 51 16 85498 85482 104.74 80.6875 0.0105804 0.00954331 2024-03-20T09:14:22.715 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:14:22.753 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:23.377 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:14:23.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:23 smithi026 ceph-mon[68309]: pgmap v1276: 47 pgs: 47 active+clean; 5.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-03-20T09:14:23.508 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:14:23.510 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:23 smithi122 ceph-mon[73495]: pgmap v1276: 47 pgs: 47 active+clean; 5.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-03-20T09:14:23.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:23.511 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:14:23.612 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:14:23.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:23 smithi115 ceph-mon[72476]: pgmap v1276: 47 pgs: 47 active+clean; 5.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-03-20T09:14:23.683 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:23.710 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 52 16 86546 86530 103.986 65.5 0.0278252 0.00961246 2024-03-20T09:14:24.275 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:14:24.275 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:14:24.290 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:14:24.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:24 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3115345504' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:14:24.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:24 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3115345504' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:14:24.710 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 53 16 87809 87793 103.513 78.9375 0.0152075 0.00965689 2024-03-20T09:14:24.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:24 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3115345504' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:14:25.133 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_ops_in_flight 2024-03-20T09:14:25.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:25 smithi122 ceph-mon[73495]: pgmap v1277: 47 pgs: 47 active+clean; 5.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-03-20T09:14:25.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:25 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:14:25.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:25 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3115345504' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:14:25.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:25 smithi122 ceph-mon[73495]: osdmap e544: 12 total, 12 up, 10 in 2024-03-20T09:14:25.519 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:25 smithi026 ceph-mon[68309]: pgmap v1277: 47 pgs: 47 active+clean; 5.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-03-20T09:14:25.519 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:25 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:14:25.519 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:25 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3115345504' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:14:25.519 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:25 smithi026 ceph-mon[68309]: osdmap e544: 12 total, 12 up, 10 in 2024-03-20T09:14:25.553 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:14:25.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:25 smithi115 ceph-mon[72476]: pgmap v1277: 47 pgs: 47 active+clean; 5.1 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-03-20T09:14:25.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:25 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:14:25.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:25 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3115345504' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:14:25.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:25 smithi115 ceph-mon[72476]: osdmap e544: 12 total, 12 up, 10 in 2024-03-20T09:14:25.710 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 54 16 89112 89096 103.104 81.4375 0.00383502 0.00969425 2024-03-20T09:14:25.858 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:14:25.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:25.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:14:25.964 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:14:26.047 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:26.135 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:14:26.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:26.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:14:26.213 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:14:26.240 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:14:26.315 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:26.553 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:14:26.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:26.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:14:26.662 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:14:26.677 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:14:26.686 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:14:26.697 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:26.710 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 55 16 90274 90258 102.55 72.625 0.0210483 0.00974758 2024-03-20T09:14:27.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:27 smithi026 ceph-mon[68309]: pgmap v1279: 47 pgs: 47 active+clean; 5.3 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-03-20T09:14:27.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:27 smithi026 ceph-mon[68309]: from='client.25744 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:14:27.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:27 smithi026 ceph-mon[68309]: osdmap e545: 12 total, 12 up, 10 in 2024-03-20T09:14:27.519 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:14:27.561 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:14:27.563 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:27 smithi122 ceph-mon[73495]: pgmap v1279: 47 pgs: 47 active+clean; 5.3 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-03-20T09:14:27.563 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:27 smithi122 ceph-mon[73495]: from='client.25744 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:14:27.563 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:27 smithi122 ceph-mon[73495]: osdmap e545: 12 total, 12 up, 10 in 2024-03-20T09:14:27.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:27.565 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:14:27.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:27 smithi115 ceph-mon[72476]: pgmap v1279: 47 pgs: 47 active+clean; 5.3 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-03-20T09:14:27.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:27 smithi115 ceph-mon[72476]: from='client.25744 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:14:27.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:27 smithi115 ceph-mon[72476]: osdmap e545: 12 total, 12 up, 10 in 2024-03-20T09:14:27.666 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:14:27.710 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 56 16 91460 91444 102.042 74.125 0.0220868 0.00979469 2024-03-20T09:14:27.733 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:27.991 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:14:27.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:27.996 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:14:28.097 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:14:28.129 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":1279,"stamp":"2024-03-20T09:14:25.811235+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5733155360,"num_objects":87474,"num_object_clones":0,"num_object_copies":262422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87472,"num_write_kb":5598208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":923,"num_bytes_recovered":60751872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91719,"ondisk_log_size":91719,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":19680736,"kb_used_data":16590788,"kb_used_omap":195,"kb_used_meta":3089660,"kb_avail":917770784,"statfs":{"total":959950356480,"available":939797282816,"internally_reserved":0,"allocated":16988966912,"data_stored":16974638379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":200482,"internal_metadata":3163812062},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":780599296,"num_objects":11911,"num_object_clones":0,"num_object_copies":35733,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11911,"num_write_kb":762304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.727120"},"pg_stats":[{"pgid":"1.1e","version":"20'137","reported_seq":463,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-20T09:14:24.270493+0000","last_change":"2024-03-20T09:13:36.531143+0000","last_active":"2024-03-20T09:14:24.270493+0000","last_peered":"2024-03-20T09:14:24.270493+0000","last_clean":"2024-03-20T09:14:24.270493+0000","last_became_active":"2024-03-20T09:13:20.071655+0000","last_became_peered":"2024-03-20T09:13:20.071655+0000","last_unstale":"2024-03-20T09:14:24.270493+0000","last_undegraded":"2024-03-20T09:14:24.270493+0000","last_fullsized":"2024-03-20T09:14:24.270493+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:36.531003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:36.531003+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:44.358559+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'137","reported_seq":630,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-20T09:14:24.281557+0000","last_change":"2024-03-20T09:13:05.291275+0000","last_active":"2024-03-20T09:14:24.281557+0000","last_peered":"2024-03-20T09:14:24.281557+0000","last_clean":"2024-03-20T09:14:24.281557+0000","last_became_active":"2024-03-20T09:07:42.013245+0000","last_became_peered":"2024-03-20T09:07:42.013245+0000","last_unstale":"2024-03-20T09:14:24.281557+0000","last_undegraded":"2024-03-20T09:14:24.281557+0000","last_fullsized":"2024-03-20T09:14:24.281557+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:05.291136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:05.291136+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"20'137","reported_seq":1078,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-03-20T09:14:17.094947+0000","last_change":"2024-03-20T09:13:56.707873+0000","last_active":"2024-03-20T09:14:17.094947+0000","last_peered":"2024-03-20T09:14:17.094947+0000","last_clean":"2024-03-20T09:14:17.094947+0000","last_became_active":"2024-03-20T09:07:41.961622+0000","last_became_peered":"2024-03-20T09:07:41.961622+0000","last_unstale":"2024-03-20T09:14:17.094947+0000","last_undegraded":"2024-03-20T09:14:17.094947+0000","last_fullsized":"2024-03-20T09:14:17.094947+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:56.707767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:56.707767+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:17.835273+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.19","version":"20'137","reported_seq":1139,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-03-20T09:14:17.086913+0000","last_change":"2024-03-20T09:13:11.366034+0000","last_active":"2024-03-20T09:14:17.086913+0000","last_peered":"2024-03-20T09:14:17.086913+0000","last_clean":"2024-03-20T09:14:17.086913+0000","last_became_active":"2024-03-20T09:07:41.962680+0000","last_became_peered":"2024-03-20T09:07:41.962680+0000","last_unstale":"2024-03-20T09:14:17.086913+0000","last_undegraded":"2024-03-20T09:14:17.086913+0000","last_fullsized":"2024-03-20T09:14:17.086913+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:11.365892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:11.365892+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:32.244898+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"20'137","reported_seq":601,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-03-20T09:14:17.089860+0000","last_change":"2024-03-20T09:13:39.292648+0000","last_active":"2024-03-20T09:14:17.089860+0000","last_peered":"2024-03-20T09:14:17.089860+0000","last_clean":"2024-03-20T09:14:17.089860+0000","last_became_active":"2024-03-20T09:10:35.277481+0000","last_became_peered":"2024-03-20T09:10:35.277481+0000","last_unstale":"2024-03-20T09:14:17.089860+0000","last_undegraded":"2024-03-20T09:14:17.089860+0000","last_fullsized":"2024-03-20T09:14:17.089860+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:39.292556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:39.292556+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:07.108785+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1b","version":"20'137","reported_seq":907,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-03-20T09:14:17.094722+0000","last_change":"2024-03-20T09:13:44.768551+0000","last_active":"2024-03-20T09:14:17.094722+0000","last_peered":"2024-03-20T09:14:17.094722+0000","last_clean":"2024-03-20T09:14:17.094722+0000","last_became_active":"2024-03-20T09:07:41.962733+0000","last_became_peered":"2024-03-20T09:07:41.962733+0000","last_unstale":"2024-03-20T09:14:17.094722+0000","last_undegraded":"2024-03-20T09:14:17.094722+0000","last_fullsized":"2024-03-20T09:14:17.094722+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:44.768424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:44.768424+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:03.198163+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"544'5593","reported_seq":5651,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-20T09:14:25.214084+0000","last_change":"2024-03-20T09:13:22.063299+0000","last_active":"2024-03-20T09:14:25.214084+0000","last_peered":"2024-03-20T09:14:25.214084+0000","last_clean":"2024-03-20T09:14:25.214084+0000","last_became_active":"2024-03-20T09:13:22.063020+0000","last_became_peered":"2024-03-20T09:13:22.063020+0000","last_unstale":"2024-03-20T09:14:25.214084+0000","last_undegraded":"2024-03-20T09:14:25.214084+0000","last_fullsized":"2024-03-20T09:14:25.214084+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":5593,"log_dups_size":0,"ondisk_log_size":5593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:41.788247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366542848,"num_objects":5593,"num_object_clones":0,"num_object_copies":16779,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5593,"num_write_kb":357952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"20'137","reported_seq":600,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-03-20T09:14:17.089866+0000","last_change":"2024-03-20T09:13:38.313619+0000","last_active":"2024-03-20T09:14:17.089866+0000","last_peered":"2024-03-20T09:14:17.089866+0000","last_clean":"2024-03-20T09:14:17.089866+0000","last_became_active":"2024-03-20T09:10:35.215772+0000","last_became_peered":"2024-03-20T09:10:35.215772+0000","last_unstale":"2024-03-20T09:14:17.089866+0000","last_undegraded":"2024-03-20T09:14:17.089866+0000","last_fullsized":"2024-03-20T09:14:17.089866+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:38.313491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:38.313491+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:46.785027+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"544'5505","reported_seq":5563,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-20T09:14:24.955700+0000","last_change":"2024-03-20T09:13:22.583666+0000","last_active":"2024-03-20T09:14:24.955700+0000","last_peered":"2024-03-20T09:14:24.955700+0000","last_clean":"2024-03-20T09:14:24.955700+0000","last_became_active":"2024-03-20T09:13:22.583317+0000","last_became_peered":"2024-03-20T09:13:22.583317+0000","last_unstale":"2024-03-20T09:14:24.955700+0000","last_undegraded":"2024-03-20T09:14:24.955700+0000","last_fullsized":"2024-03-20T09:14:24.955700+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":5505,"log_dups_size":0,"ondisk_log_size":5505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:29.205626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360775680,"num_objects":5505,"num_object_clones":0,"num_object_copies":16515,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5505,"num_write_kb":352320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,8],"acting":[3,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"20'137","reported_seq":1152,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-03-20T09:14:17.095463+0000","last_change":"2024-03-20T09:13:05.129380+0000","last_active":"2024-03-20T09:14:17.095463+0000","last_peered":"2024-03-20T09:14:17.095463+0000","last_clean":"2024-03-20T09:14:17.095463+0000","last_became_active":"2024-03-20T09:10:34.099414+0000","last_became_peered":"2024-03-20T09:10:34.099414+0000","last_unstale":"2024-03-20T09:14:17.095463+0000","last_undegraded":"2024-03-20T09:14:17.095463+0000","last_fullsized":"2024-03-20T09:14:17.095463+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:05.129288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:05.129288+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"543'5456","reported_seq":7368,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-03-20T09:14:23.476640+0000","last_change":"2024-03-20T09:13:46.203083+0000","last_active":"2024-03-20T09:14:23.476640+0000","last_peered":"2024-03-20T09:14:23.476640+0000","last_clean":"2024-03-20T09:14:23.476640+0000","last_became_active":"2024-03-20T09:13:46.198893+0000","last_became_peered":"2024-03-20T09:13:46.198893+0000","last_unstale":"2024-03-20T09:14:23.476640+0000","last_undegraded":"2024-03-20T09:14:23.476640+0000","last_fullsized":"2024-03-20T09:14:23.476640+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":5456,"log_dups_size":0,"ondisk_log_size":5456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:41.040830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":357564416,"num_objects":5456,"num_object_clones":0,"num_object_copies":16368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5456,"num_write_kb":349184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":922,"num_bytes_recovered":60424192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,11],"acting":[2,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.a","version":"20'137","reported_seq":968,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-20T09:14:24.266578+0000","last_change":"2024-03-20T09:13:57.530523+0000","last_active":"2024-03-20T09:14:24.266578+0000","last_peered":"2024-03-20T09:14:24.266578+0000","last_clean":"2024-03-20T09:14:24.266578+0000","last_became_active":"2024-03-20T09:07:52.618107+0000","last_became_peered":"2024-03-20T09:07:52.618107+0000","last_unstale":"2024-03-20T09:14:24.266578+0000","last_undegraded":"2024-03-20T09:14:24.266578+0000","last_fullsized":"2024-03-20T09:14:24.266578+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:57.530396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:57.530396+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:10.089743+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"544'5494","reported_seq":5552,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-20T09:14:24.754327+0000","last_change":"2024-03-20T09:13:22.064363+0000","last_active":"2024-03-20T09:14:24.754327+0000","last_peered":"2024-03-20T09:14:24.754327+0000","last_clean":"2024-03-20T09:14:24.754327+0000","last_became_active":"2024-03-20T09:13:22.063975+0000","last_became_peered":"2024-03-20T09:13:22.063975+0000","last_unstale":"2024-03-20T09:14:24.754327+0000","last_undegraded":"2024-03-20T09:14:24.754327+0000","last_fullsized":"2024-03-20T09:14:24.754327+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":5494,"log_dups_size":0,"ondisk_log_size":5494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:45.622073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360054784,"num_objects":5494,"num_object_clones":0,"num_object_copies":16482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5494,"num_write_kb":351616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"20'137","reported_seq":1123,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-20T09:14:24.270494+0000","last_change":"2024-03-20T09:13:14.395564+0000","last_active":"2024-03-20T09:14:24.270494+0000","last_peered":"2024-03-20T09:14:24.270494+0000","last_clean":"2024-03-20T09:14:24.270494+0000","last_became_active":"2024-03-20T09:12:14.306384+0000","last_became_peered":"2024-03-20T09:12:14.306384+0000","last_unstale":"2024-03-20T09:14:24.270494+0000","last_undegraded":"2024-03-20T09:14:24.270494+0000","last_fullsized":"2024-03-20T09:14:24.270494+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:14.395469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:14.395469+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:24.496429+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"544'5444","reported_seq":5498,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-20T09:14:25.001034+0000","last_change":"2024-03-20T09:13:24.064404+0000","last_active":"2024-03-20T09:14:25.001034+0000","last_peered":"2024-03-20T09:14:25.001034+0000","last_clean":"2024-03-20T09:14:25.001034+0000","last_became_active":"2024-03-20T09:13:24.064091+0000","last_became_peered":"2024-03-20T09:13:24.064091+0000","last_unstale":"2024-03-20T09:14:25.001034+0000","last_undegraded":"2024-03-20T09:14:25.001034+0000","last_fullsized":"2024-03-20T09:14:25.001034+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":5444,"log_dups_size":0,"ondisk_log_size":5444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:33.687560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356777984,"num_objects":5444,"num_object_clones":0,"num_object_copies":16332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5444,"num_write_kb":348416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"20'137","reported_seq":1017,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-03-20T09:14:17.087151+0000","last_change":"2024-03-20T09:13:10.343309+0000","last_active":"2024-03-20T09:14:17.087151+0000","last_peered":"2024-03-20T09:14:17.087151+0000","last_clean":"2024-03-20T09:14:17.087151+0000","last_became_active":"2024-03-20T09:07:37.003722+0000","last_became_peered":"2024-03-20T09:07:37.003722+0000","last_unstale":"2024-03-20T09:14:17.087151+0000","last_undegraded":"2024-03-20T09:14:17.087151+0000","last_fullsized":"2024-03-20T09:14:17.087151+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:10.343198+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:07:06.480160+0000","last_clean_scrub_stamp":"2024-03-20T09:13:10.343198+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:33.639911+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.3","version":"544'5594","reported_seq":5652,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-20T09:14:24.786854+0000","last_change":"2024-03-20T09:13:22.064621+0000","last_active":"2024-03-20T09:14:24.786854+0000","last_peered":"2024-03-20T09:14:24.786854+0000","last_clean":"2024-03-20T09:14:24.786854+0000","last_became_active":"2024-03-20T09:13:22.064275+0000","last_became_peered":"2024-03-20T09:13:22.064275+0000","last_unstale":"2024-03-20T09:14:24.786854+0000","last_undegraded":"2024-03-20T09:14:24.786854+0000","last_fullsized":"2024-03-20T09:14:24.786854+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":5594,"log_dups_size":0,"ondisk_log_size":5594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:40.477405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366608384,"num_objects":5594,"num_object_clones":0,"num_object_copies":16782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5594,"num_write_kb":358016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"20'137","reported_seq":1034,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-03-20T09:14:17.087486+0000","last_change":"2024-03-20T09:13:12.359759+0000","last_active":"2024-03-20T09:14:17.087486+0000","last_peered":"2024-03-20T09:14:17.087486+0000","last_clean":"2024-03-20T09:14:17.087486+0000","last_became_active":"2024-03-20T09:07:52.593671+0000","last_became_peered":"2024-03-20T09:07:52.593671+0000","last_unstale":"2024-03-20T09:14:17.087486+0000","last_undegraded":"2024-03-20T09:14:17.087486+0000","last_fullsized":"2024-03-20T09:14:17.087486+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:12.359617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:12.359617+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:24.685959+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.6","version":"544'5550","reported_seq":5608,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-20T09:14:25.219826+0000","last_change":"2024-03-20T09:13:22.067664+0000","last_active":"2024-03-20T09:14:25.219826+0000","last_peered":"2024-03-20T09:14:25.219826+0000","last_clean":"2024-03-20T09:14:25.219826+0000","last_became_active":"2024-03-20T09:13:22.067362+0000","last_became_peered":"2024-03-20T09:13:22.067362+0000","last_unstale":"2024-03-20T09:14:25.219826+0000","last_undegraded":"2024-03-20T09:14:25.219826+0000","last_fullsized":"2024-03-20T09:14:25.219826+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":5550,"log_dups_size":0,"ondisk_log_size":5550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:46.912906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363724800,"num_objects":5550,"num_object_clones":0,"num_object_copies":16650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5550,"num_write_kb":355200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"20'137","reported_seq":1037,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-20T09:14:24.277870+0000","last_change":"2024-03-20T09:13:49.033462+0000","last_active":"2024-03-20T09:14:24.277870+0000","last_peered":"2024-03-20T09:14:24.277870+0000","last_clean":"2024-03-20T09:14:24.277870+0000","last_became_active":"2024-03-20T09:12:02.221686+0000","last_became_peered":"2024-03-20T09:12:02.221686+0000","last_unstale":"2024-03-20T09:14:24.277870+0000","last_undegraded":"2024-03-20T09:14:24.277870+0000","last_fullsized":"2024-03-20T09:14:24.277870+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:49.033327+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:18.788771+0000","last_clean_scrub_stamp":"2024-03-20T09:13:49.033327+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:10.707005+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,8],"acting":[1,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"544'5474","reported_seq":5532,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-20T09:14:25.704302+0000","last_change":"2024-03-20T09:13:22.063240+0000","last_active":"2024-03-20T09:14:25.704302+0000","last_peered":"2024-03-20T09:14:25.704302+0000","last_clean":"2024-03-20T09:14:25.704302+0000","last_became_active":"2024-03-20T09:13:22.062992+0000","last_became_peered":"2024-03-20T09:13:22.062992+0000","last_unstale":"2024-03-20T09:14:25.704302+0000","last_undegraded":"2024-03-20T09:14:25.704302+0000","last_fullsized":"2024-03-20T09:14:25.704302+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":5474,"log_dups_size":0,"ondisk_log_size":5474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:36.289001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358744064,"num_objects":5474,"num_object_clones":0,"num_object_copies":16422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5474,"num_write_kb":350336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":1095,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-20T09:14:24.270614+0000","last_change":"2024-03-20T09:13:03.215035+0000","last_active":"2024-03-20T09:14:24.270614+0000","last_peered":"2024-03-20T09:14:24.270614+0000","last_clean":"2024-03-20T09:14:24.270614+0000","last_became_active":"2024-03-20T09:12:14.305989+0000","last_became_peered":"2024-03-20T09:12:14.305989+0000","last_unstale":"2024-03-20T09:14:24.270614+0000","last_undegraded":"2024-03-20T09:14:24.270614+0000","last_fullsized":"2024-03-20T09:14:24.270614+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:03.214902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:03.214902+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.7","version":"544'5596","reported_seq":5654,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-20T09:14:25.209920+0000","last_change":"2024-03-20T09:13:22.066405+0000","last_active":"2024-03-20T09:14:25.209920+0000","last_peered":"2024-03-20T09:14:25.209920+0000","last_clean":"2024-03-20T09:14:25.209920+0000","last_became_active":"2024-03-20T09:13:22.065656+0000","last_became_peered":"2024-03-20T09:13:22.065656+0000","last_unstale":"2024-03-20T09:14:25.209920+0000","last_undegraded":"2024-03-20T09:14:25.209920+0000","last_fullsized":"2024-03-20T09:14:25.209920+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":5596,"log_dups_size":0,"ondisk_log_size":5596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366739456,"num_objects":5596,"num_object_clones":0,"num_object_copies":16788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5596,"num_write_kb":358144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,0],"acting":[1,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"20'137","reported_seq":1342,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-03-20T09:14:17.094195+0000","last_change":"2024-03-20T09:13:40.818653+0000","last_active":"2024-03-20T09:14:17.094195+0000","last_peered":"2024-03-20T09:14:17.094195+0000","last_clean":"2024-03-20T09:14:17.094195+0000","last_became_active":"2024-03-20T09:12:14.313541+0000","last_became_peered":"2024-03-20T09:12:14.313541+0000","last_unstale":"2024-03-20T09:14:17.094195+0000","last_undegraded":"2024-03-20T09:14:17.094195+0000","last_fullsized":"2024-03-20T09:14:17.094195+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:40.818542+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:19.513511+0000","last_clean_scrub_stamp":"2024-03-20T09:13:40.818542+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:01.397421+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"544'5479","reported_seq":5542,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-20T09:14:25.214217+0000","last_change":"2024-03-20T09:13:24.443379+0000","last_active":"2024-03-20T09:14:25.214217+0000","last_peered":"2024-03-20T09:14:25.214217+0000","last_clean":"2024-03-20T09:14:25.214217+0000","last_became_active":"2024-03-20T09:13:24.443054+0000","last_became_peered":"2024-03-20T09:13:24.443054+0000","last_unstale":"2024-03-20T09:14:25.214217+0000","last_undegraded":"2024-03-20T09:14:25.214217+0000","last_fullsized":"2024-03-20T09:14:25.214217+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":5479,"log_dups_size":0,"ondisk_log_size":5479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:41.617061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":359071744,"num_objects":5479,"num_object_clones":0,"num_object_copies":16437,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5479,"num_write_kb":350656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"20'137","reported_seq":1038,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-03-20T09:14:17.087023+0000","last_change":"2024-03-20T09:13:14.428508+0000","last_active":"2024-03-20T09:14:17.087023+0000","last_peered":"2024-03-20T09:14:17.087023+0000","last_clean":"2024-03-20T09:14:17.087023+0000","last_became_active":"2024-03-20T09:06:02.923630+0000","last_became_peered":"2024-03-20T09:06:02.923630+0000","last_unstale":"2024-03-20T09:14:17.087023+0000","last_undegraded":"2024-03-20T09:14:17.087023+0000","last_fullsized":"2024-03-20T09:14:17.087023+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:14.428416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:14.428416+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:27.674669+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2],"acting":[10,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2","version":"543'5390","reported_seq":5446,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-03-20T09:14:22.099253+0000","last_change":"2024-03-20T09:13:22.584008+0000","last_active":"2024-03-20T09:14:22.099253+0000","last_peered":"2024-03-20T09:14:22.099253+0000","last_clean":"2024-03-20T09:14:22.099253+0000","last_became_active":"2024-03-20T09:13:22.583529+0000","last_became_peered":"2024-03-20T09:13:22.583529+0000","last_unstale":"2024-03-20T09:14:22.099253+0000","last_undegraded":"2024-03-20T09:14:22.099253+0000","last_fullsized":"2024-03-20T09:14:22.099253+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":5390,"log_dups_size":0,"ondisk_log_size":5390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:36.926945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353239040,"num_objects":5390,"num_object_clones":0,"num_object_copies":16170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5390,"num_write_kb":344960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,8],"acting":[5,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"20'137","reported_seq":1040,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-20T09:14:24.266658+0000","last_change":"2024-03-20T09:14:10.561025+0000","last_active":"2024-03-20T09:14:24.266658+0000","last_peered":"2024-03-20T09:14:24.266658+0000","last_clean":"2024-03-20T09:14:24.266658+0000","last_became_active":"2024-03-20T09:13:27.087760+0000","last_became_peered":"2024-03-20T09:13:27.087760+0000","last_unstale":"2024-03-20T09:14:24.266658+0000","last_undegraded":"2024-03-20T09:14:24.266658+0000","last_fullsized":"2024-03-20T09:14:24.266658+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:14:10.560890+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:14:10.560890+0000","last_clean_scrub_stamp":"2024-03-20T09:14:10.560890+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:11.348848+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"543'5216","reported_seq":5272,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-03-20T09:14:21.205917+0000","last_change":"2024-03-20T09:13:22.064771+0000","last_active":"2024-03-20T09:14:21.205917+0000","last_peered":"2024-03-20T09:14:21.205917+0000","last_clean":"2024-03-20T09:14:21.205917+0000","last_became_active":"2024-03-20T09:13:22.064461+0000","last_became_peered":"2024-03-20T09:13:22.064461+0000","last_unstale":"2024-03-20T09:14:21.205917+0000","last_undegraded":"2024-03-20T09:14:21.205917+0000","last_fullsized":"2024-03-20T09:14:21.205917+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":5216,"log_dups_size":0,"ondisk_log_size":5216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:25.759321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341835776,"num_objects":5216,"num_object_clones":0,"num_object_copies":15648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5216,"num_write_kb":333824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,1],"acting":[10,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.b","version":"20'137","reported_seq":908,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-20T09:14:24.270685+0000","last_change":"2024-03-20T09:13:46.614783+0000","last_active":"2024-03-20T09:14:24.270685+0000","last_peered":"2024-03-20T09:14:24.270685+0000","last_clean":"2024-03-20T09:14:24.270685+0000","last_became_active":"2024-03-20T09:13:20.070596+0000","last_became_peered":"2024-03-20T09:13:20.070596+0000","last_unstale":"2024-03-20T09:14:24.270685+0000","last_undegraded":"2024-03-20T09:14:24.270685+0000","last_fullsized":"2024-03-20T09:14:24.270685+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:46.614691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:46.614691+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:46.810325+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.b","version":"543'5148","reported_seq":5203,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-03-20T09:14:21.728644+0000","last_change":"2024-03-20T09:13:22.065041+0000","last_active":"2024-03-20T09:14:21.728644+0000","last_peered":"2024-03-20T09:14:21.728644+0000","last_clean":"2024-03-20T09:14:21.728644+0000","last_became_active":"2024-03-20T09:13:22.064687+0000","last_became_peered":"2024-03-20T09:13:22.064687+0000","last_unstale":"2024-03-20T09:14:21.728644+0000","last_undegraded":"2024-03-20T09:14:21.728644+0000","last_fullsized":"2024-03-20T09:14:21.728644+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:35.394465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337379328,"num_objects":5148,"num_object_clones":0,"num_object_copies":15444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5148,"num_write_kb":329472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"20'137","reported_seq":566,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-03-20T09:14:17.085708+0000","last_change":"2024-03-20T09:14:08.589630+0000","last_active":"2024-03-20T09:14:17.085708+0000","last_peered":"2024-03-20T09:14:17.085708+0000","last_clean":"2024-03-20T09:14:17.085708+0000","last_became_active":"2024-03-20T09:12:14.311319+0000","last_became_peered":"2024-03-20T09:12:14.311319+0000","last_unstale":"2024-03-20T09:14:17.085708+0000","last_undegraded":"2024-03-20T09:14:17.085708+0000","last_fullsized":"2024-03-20T09:14:17.085708+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:14:08.589510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:14:08.589510+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:21.094562+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"544'5542","reported_seq":5600,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-20T09:14:25.696357+0000","last_change":"2024-03-20T09:13:22.064279+0000","last_active":"2024-03-20T09:14:25.696357+0000","last_peered":"2024-03-20T09:14:25.696357+0000","last_clean":"2024-03-20T09:14:25.696357+0000","last_became_active":"2024-03-20T09:13:22.063927+0000","last_became_peered":"2024-03-20T09:13:22.063927+0000","last_unstale":"2024-03-20T09:14:25.696357+0000","last_undegraded":"2024-03-20T09:14:25.696357+0000","last_fullsized":"2024-03-20T09:14:25.696357+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":5542,"log_dups_size":0,"ondisk_log_size":5542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:37.913884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363200512,"num_objects":5542,"num_object_clones":0,"num_object_copies":16626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5542,"num_write_kb":354688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"20'137","reported_seq":1053,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-20T09:14:24.266651+0000","last_change":"2024-03-20T09:13:53.635081+0000","last_active":"2024-03-20T09:14:24.266651+0000","last_peered":"2024-03-20T09:14:24.266651+0000","last_clean":"2024-03-20T09:14:24.266651+0000","last_became_active":"2024-03-20T09:10:35.119706+0000","last_became_peered":"2024-03-20T09:10:35.119706+0000","last_unstale":"2024-03-20T09:14:24.266651+0000","last_undegraded":"2024-03-20T09:14:24.266651+0000","last_fullsized":"2024-03-20T09:14:24.266651+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:53.634986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:53.634986+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:06.460977+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"544'5434","reported_seq":5497,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-20T09:14:24.778379+0000","last_change":"2024-03-20T09:13:24.065781+0000","last_active":"2024-03-20T09:14:24.778379+0000","last_peered":"2024-03-20T09:14:24.778379+0000","last_clean":"2024-03-20T09:14:24.778379+0000","last_became_active":"2024-03-20T09:13:24.065349+0000","last_became_peered":"2024-03-20T09:13:24.065349+0000","last_unstale":"2024-03-20T09:14:24.778379+0000","last_undegraded":"2024-03-20T09:14:24.778379+0000","last_fullsized":"2024-03-20T09:14:24.778379+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":5434,"log_dups_size":0,"ondisk_log_size":5434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:29.930666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356122624,"num_objects":5434,"num_object_clones":0,"num_object_copies":16302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5434,"num_write_kb":347776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"20'137","reported_seq":1075,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-20T09:14:24.277041+0000","last_change":"2024-03-20T09:13:27.088076+0000","last_active":"2024-03-20T09:14:24.277041+0000","last_peered":"2024-03-20T09:14:24.277041+0000","last_clean":"2024-03-20T09:14:24.277041+0000","last_became_active":"2024-03-20T09:13:27.087686+0000","last_became_peered":"2024-03-20T09:13:27.087686+0000","last_unstale":"2024-03-20T09:14:24.277041+0000","last_undegraded":"2024-03-20T09:14:24.277041+0000","last_fullsized":"2024-03-20T09:14:24.277041+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:09.755064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:09.755064+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:33.274547+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"544'5557","reported_seq":5615,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-20T09:14:24.790823+0000","last_change":"2024-03-20T09:13:22.064047+0000","last_active":"2024-03-20T09:14:24.790823+0000","last_peered":"2024-03-20T09:14:24.790823+0000","last_clean":"2024-03-20T09:14:24.790823+0000","last_became_active":"2024-03-20T09:13:22.063673+0000","last_became_peered":"2024-03-20T09:13:22.063673+0000","last_unstale":"2024-03-20T09:14:24.790823+0000","last_undegraded":"2024-03-20T09:14:24.790823+0000","last_fullsized":"2024-03-20T09:14:24.790823+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":5557,"log_dups_size":0,"ondisk_log_size":5557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:39.125308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364183552,"num_objects":5557,"num_object_clones":0,"num_object_copies":16671,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5557,"num_write_kb":355648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"20'137","reported_seq":627,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-20T09:14:24.281465+0000","last_change":"2024-03-20T09:13:38.035423+0000","last_active":"2024-03-20T09:14:24.281465+0000","last_peered":"2024-03-20T09:14:24.281465+0000","last_clean":"2024-03-20T09:14:24.281465+0000","last_became_active":"2024-03-20T09:12:14.307864+0000","last_became_peered":"2024-03-20T09:12:14.307864+0000","last_unstale":"2024-03-20T09:14:24.281465+0000","last_undegraded":"2024-03-20T09:14:24.281465+0000","last_fullsized":"2024-03-20T09:14:24.281465+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:38.035304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:38.035304+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:57.965009+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"20'137","reported_seq":941,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-03-20T09:14:17.090005+0000","last_change":"2024-03-20T09:13:04.270196+0000","last_active":"2024-03-20T09:14:17.090005+0000","last_peered":"2024-03-20T09:14:17.090005+0000","last_clean":"2024-03-20T09:14:17.090005+0000","last_became_active":"2024-03-20T09:10:34.096758+0000","last_became_peered":"2024-03-20T09:10:34.096758+0000","last_unstale":"2024-03-20T09:14:17.090005+0000","last_undegraded":"2024-03-20T09:14:17.090005+0000","last_fullsized":"2024-03-20T09:14:17.090005+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:04.270076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:04.270076+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:18.468974+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"20'137","reported_seq":1016,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-20T09:14:24.277707+0000","last_change":"2024-03-20T09:13:50.092495+0000","last_active":"2024-03-20T09:14:24.277707+0000","last_peered":"2024-03-20T09:14:24.277707+0000","last_clean":"2024-03-20T09:14:24.277707+0000","last_became_active":"2024-03-20T09:04:04.103645+0000","last_became_peered":"2024-03-20T09:04:04.103645+0000","last_unstale":"2024-03-20T09:14:24.277707+0000","last_undegraded":"2024-03-20T09:14:24.277707+0000","last_fullsized":"2024-03-20T09:14:24.277707+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:50.092343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:50.092343+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:51.393395+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"20'137","reported_seq":467,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-03-20T09:14:17.089980+0000","last_change":"2024-03-20T09:13:05.305611+0000","last_active":"2024-03-20T09:14:17.089980+0000","last_peered":"2024-03-20T09:14:17.089980+0000","last_clean":"2024-03-20T09:14:17.089980+0000","last_became_active":"2024-03-20T09:10:34.097191+0000","last_became_peered":"2024-03-20T09:10:34.097191+0000","last_unstale":"2024-03-20T09:14:17.089980+0000","last_undegraded":"2024-03-20T09:14:17.089980+0000","last_fullsized":"2024-03-20T09:14:17.089980+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:05.305514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:05.305514+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:31.997881+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"20'137","reported_seq":878,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-03-20T09:14:17.094137+0000","last_change":"2024-03-20T09:13:38.808687+0000","last_active":"2024-03-20T09:14:17.094137+0000","last_peered":"2024-03-20T09:14:17.094137+0000","last_clean":"2024-03-20T09:14:17.094137+0000","last_became_active":"2024-03-20T09:13:27.087859+0000","last_became_peered":"2024-03-20T09:13:27.087859+0000","last_unstale":"2024-03-20T09:14:17.094137+0000","last_undegraded":"2024-03-20T09:14:17.094137+0000","last_fullsized":"2024-03-20T09:14:17.094137+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:38.808565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:38.808565+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:51.238334+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,3],"acting":[0,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"20'137","reported_seq":705,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-20T09:14:24.279792+0000","last_change":"2024-03-20T09:13:27.088201+0000","last_active":"2024-03-20T09:14:24.279792+0000","last_peered":"2024-03-20T09:14:24.279792+0000","last_clean":"2024-03-20T09:14:24.279792+0000","last_became_active":"2024-03-20T09:13:27.087760+0000","last_became_peered":"2024-03-20T09:13:27.087760+0000","last_unstale":"2024-03-20T09:14:24.279792+0000","last_undegraded":"2024-03-20T09:14:24.279792+0000","last_fullsized":"2024-03-20T09:14:24.279792+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:10.182025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:10.182025+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:40.149783+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"20'137","reported_seq":1029,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-03-20T09:14:17.095136+0000","last_change":"2024-03-20T09:13:48.832256+0000","last_active":"2024-03-20T09:14:17.095136+0000","last_peered":"2024-03-20T09:14:17.095136+0000","last_clean":"2024-03-20T09:14:17.095136+0000","last_became_active":"2024-03-20T09:10:35.095339+0000","last_became_peered":"2024-03-20T09:10:35.095339+0000","last_unstale":"2024-03-20T09:14:17.095136+0000","last_undegraded":"2024-03-20T09:14:17.095136+0000","last_fullsized":"2024-03-20T09:14:17.095136+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:48.831557+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:23.640533+0000","last_clean_scrub_stamp":"2024-03-20T09:13:48.831557+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:09.728838+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"20'137","reported_seq":1090,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-03-20T09:14:17.095869+0000","last_change":"2024-03-20T09:12:52.043287+0000","last_active":"2024-03-20T09:14:17.095869+0000","last_peered":"2024-03-20T09:14:17.095869+0000","last_clean":"2024-03-20T09:14:17.095869+0000","last_became_active":"2024-03-20T09:04:14.208448+0000","last_became_peered":"2024-03-20T09:04:14.208448+0000","last_unstale":"2024-03-20T09:14:17.095869+0000","last_undegraded":"2024-03-20T09:14:17.095869+0000","last_fullsized":"2024-03-20T09:14:17.095869+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:12:52.043144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:12:52.043144+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:17.347869+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"20'137","reported_seq":355,"reported_epoch":543,"state":"active+clean","last_fresh":"2024-03-20T09:14:17.085690+0000","last_change":"2024-03-20T09:13:04.628974+0000","last_active":"2024-03-20T09:14:17.085690+0000","last_peered":"2024-03-20T09:14:17.085690+0000","last_clean":"2024-03-20T09:14:17.085690+0000","last_became_active":"2024-03-20T09:12:14.307581+0000","last_became_peered":"2024-03-20T09:12:14.307581+0000","last_unstale":"2024-03-20T09:14:17.085690+0000","last_undegraded":"2024-03-20T09:14:17.085690+0000","last_fullsized":"2024-03-20T09:14:17.085690+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:04.628862+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:07:06.480160+0000","last_clean_scrub_stamp":"2024-03-20T09:13:04.628862+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:14:19.586344+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"20'137","reported_seq":636,"reported_epoch":544,"state":"active+clean","last_fresh":"2024-03-20T09:14:24.280117+0000","last_change":"2024-03-20T09:13:47.938675+0000","last_active":"2024-03-20T09:14:24.280117+0000","last_peered":"2024-03-20T09:14:24.280117+0000","last_clean":"2024-03-20T09:14:24.280117+0000","last_became_active":"2024-03-20T09:13:20.074439+0000","last_became_peered":"2024-03-20T09:13:20.074439+0000","last_unstale":"2024-03-20T09:14:24.280117+0000","last_undegraded":"2024-03-20T09:14:24.280117+0000","last_fullsized":"2024-03-20T09:14:24.280117+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:47.938552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:47.938552+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:06.649880+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,11],"acting":[1,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":5732564992,"num_objects":87472,"num_object_clones":0,"num_object_copies":262416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87472,"num_write_kb":5598208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":922,"num_bytes_recovered":60424192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17092902912,"data_stored":17092902912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87472,"ondisk_log_size":87472,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4247,"ondisk_log_size":4247,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860636,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3151740,"kb_used_data":2791540,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":90593412,"statfs":{"total":95995035648,"available":92767653888,"internally_reserved":0,"allocated":2858536960,"data_stored":2857106528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":368835012},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":89,"seq":382252089569,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1558972,"kb_used_data":1315380,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":92186180,"statfs":{"total":95995035648,"available":94398648320,"internally_reserved":0,"allocated":1346949120,"data_stored":1345518688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":249428416},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":0,"seq":343597383913,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":70,"seq":300647710957,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":909180,"kb_used_data":683636,"kb_used_omap":1,"kb_used_meta":225534,"kb_avail":92835972,"statfs":{"total":95995035648,"available":95064035328,"internally_reserved":0,"allocated":700043264,"data_stored":698612832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":230947270},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":60,"seq":257698038002,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1776324,"kb_used_data":1334204,"kb_used_omap":1,"kb_used_meta":442110,"kb_avail":91968828,"statfs":{"total":95995035648,"available":94176079872,"internally_reserved":0,"allocated":1366224896,"data_stored":1364790912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":452721098},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332344,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2474444,"kb_used_data":2128128,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":91270708,"statfs":{"total":95995035648,"available":93461204992,"internally_reserved":0,"allocated":2179203072,"data_stored":2177768121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":354613702},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575122,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3601292,"kb_used_data":3204992,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":90143860,"statfs":{"total":95995035648,"available":92307312640,"internally_reserved":0,"allocated":3281911808,"data_stored":3280476857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":405797318},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738640,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1739132,"kb_used_data":1346996,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":92006020,"statfs":{"total":95995035648,"available":94214164480,"internally_reserved":0,"allocated":1379323904,"data_stored":1377893472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411595,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1255492,"kb_used_data":1035516,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":92489660,"statfs":{"total":95995035648,"available":94709411840,"internally_reserved":0,"allocated":1060368384,"data_stored":1058934400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149959,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2077196,"kb_used_data":1743360,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":91667956,"statfs":{"total":95995035648,"available":93867986944,"internally_reserved":0,"allocated":1785200640,"data_stored":1783765689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":341834183},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953733,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":443,"seq":1902670512175,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1136964,"kb_used_data":1007036,"kb_used_omap":181,"kb_used_meta":129738,"kb_avail":92608188,"statfs":{"total":95995035648,"available":94830784512,"internally_reserved":0,"allocated":1031204864,"data_stored":1029770880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":186131,"internal_metadata":132851949},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1377959936,"data_stored":1377959936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3305439232,"data_stored":3305439232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1061879808,"data_stored":1061879808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1798438912,"data_stored":1798438912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1039597568,"data_stored":1039597568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2184052736,"data_stored":2184052736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1369636864,"data_stored":1369636864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":696582144,"data_stored":696582144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1363673088,"data_stored":1363673088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2895642624,"data_stored":2895642624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:14:28.131 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-20T09:14:28.132 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:14:28.132 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-03-20T09:14:28.132 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-20T09:14:28.132 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-20T09:14:28.132 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-03-20T09:14:28.133 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:28.385 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:14:28.386 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:14:28.412 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:14:28.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:28.414 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:14:28.519 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:14:28.551 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:28.575 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:14:28.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:28.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:14:28.679 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:14:28.710 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 57 16 92654 92638 101.561 74.625 0.0172231 0.00984171 2024-03-20T09:14:28.732 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:29.009 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:14:29.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:29.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:14:29.192 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:14:29.223 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:29.248 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:14:29.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:29.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:14:29.353 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:14:29.389 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:29.414 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:14:29.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:29.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:14:29.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:29 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:14:29.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:29 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:14:29.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:29 smithi122 ceph-mon[73495]: pgmap v1281: 47 pgs: 47 active+clean; 5.4 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-03-20T09:14:29.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:29 smithi122 ceph-mon[73495]: osdmap e546: 12 total, 12 up, 10 in 2024-03-20T09:14:29.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:29 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:14:29.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:29 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:14:29.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:29 smithi026 ceph-mon[68309]: pgmap v1281: 47 pgs: 47 active+clean; 5.4 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-03-20T09:14:29.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:29 smithi026 ceph-mon[68309]: osdmap e546: 12 total, 12 up, 10 in 2024-03-20T09:14:29.521 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:14:29.551 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:29 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:14:29.551 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:29 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:14:29.551 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:29 smithi115 ceph-mon[72476]: pgmap v1281: 47 pgs: 47 active+clean; 5.4 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-03-20T09:14:29.551 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:29 smithi115 ceph-mon[72476]: osdmap e546: 12 total, 12 up, 10 in 2024-03-20T09:14:29.555 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:29.568 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_blocked_ops 2024-03-20T09:14:29.577 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:14:29.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:29.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:14:29.682 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:14:29.711 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 58 16 93863 93847 101.113 75.5625 0.0154506 0.00988553 2024-03-20T09:14:29.726 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:29.759 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:14:29.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:29.766 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:14:29.855 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:14:29.866 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:14:29.878 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:14:29.912 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:29.932 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:14:29.940 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:14:29.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:29.948 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:14:30.050 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:14:30.081 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:30.108 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:14:30.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:30.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:14:30.214 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:14:30.271 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:30.309 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:14:30.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:30.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:14:30.420 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:14:30.452 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:30.477 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:14:30.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:30.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:14:30.580 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:14:30.644 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:30.711 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 59 16 95005 94989 100.609 71.375 0.0221836 0.00993517 2024-03-20T09:14:30.870 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:14:30.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:30.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:14:30.976 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:14:31.040 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:31.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:31 smithi026 ceph-mon[68309]: pgmap v1283: 47 pgs: 47 active+clean; 5.5 GiB data, 20 GiB used, 875 GiB / 894 GiB avail; 80 MiB/s wr, 1.29k op/s 2024-03-20T09:14:31.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:31 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:14:31.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:31 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:14:31.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:31 smithi026 ceph-mon[68309]: osdmap e547: 12 total, 12 up, 10 in 2024-03-20T09:14:31.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:31 smithi115 ceph-mon[72476]: pgmap v1283: 47 pgs: 47 active+clean; 5.5 GiB data, 20 GiB used, 875 GiB / 894 GiB avail; 80 MiB/s wr, 1.29k op/s 2024-03-20T09:14:31.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:31 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:14:31.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:31 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:14:31.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:31 smithi115 ceph-mon[72476]: osdmap e547: 12 total, 12 up, 10 in 2024-03-20T09:14:31.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:31 smithi122 ceph-mon[73495]: pgmap v1283: 47 pgs: 47 active+clean; 5.5 GiB data, 20 GiB used, 875 GiB / 894 GiB avail; 80 MiB/s wr, 1.29k op/s 2024-03-20T09:14:31.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:31 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:14:31.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:31 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:14:31.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:31 smithi122 ceph-mon[73495]: osdmap e547: 12 total, 12 up, 10 in 2024-03-20T09:14:31.711 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:2024-03-20T09:14:31.711037+0000 min lat: 0.00107883 max lat: 2.19794 avg lat: 0.00998291 2024-03-20T09:14:31.711 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T09:14:31.711 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 60 16 96152 96136 100.127 71.6875 0.0159789 0.00998291 2024-03-20T09:14:31.752 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:14:31.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:31.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:14:31.858 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:14:31.914 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:32.711 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 61 16 97350 97334 99.7126 74.875 0.0142776 0.0100246 2024-03-20T09:14:32.876 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:14:32.876 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:14:32.888 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:14:32.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:32.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:14:32.994 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:14:33.068 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:33.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:33 smithi115 ceph-mon[72476]: pgmap v1285: 47 pgs: 47 active+clean; 5.7 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-03-20T09:14:33.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:33 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:14:33.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:33 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:14:33.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:33 smithi115 ceph-mon[72476]: osdmap e548: 12 total, 12 up, 10 in 2024-03-20T09:14:33.700 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:33 smithi122 ceph-mon[73495]: pgmap v1285: 47 pgs: 47 active+clean; 5.7 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-03-20T09:14:33.700 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:33 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:14:33.700 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:33 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:14:33.700 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:33 smithi122 ceph-mon[73495]: osdmap e548: 12 total, 12 up, 10 in 2024-03-20T09:14:33.711 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 62 16 98459 98443 99.2222 69.3125 0.0150782 0.0100744 2024-03-20T09:14:33.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:33 smithi026 ceph-mon[68309]: pgmap v1285: 47 pgs: 47 active+clean; 5.7 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-03-20T09:14:33.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:33 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:14:33.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:33 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:14:33.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:33 smithi026 ceph-mon[68309]: osdmap e548: 12 total, 12 up, 10 in 2024-03-20T09:14:33.730 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:14:33.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:33.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:14:33.849 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:14:33.883 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:33.904 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:14:33.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:33.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:14:34.010 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:14:34.085 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:34.201 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:14:34.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:34.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:14:34.255 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_historic_ops 2024-03-20T09:14:34.308 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:14:34.335 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:34.356 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:14:34.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:34.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:14:34.461 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:14:34.491 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:34.520 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:14:34.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:34.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:14:34.602 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:14:34.626 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:14:34.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:34 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:14:34.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:34 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:14:34.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:34 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2825896074' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:14:34.678 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:34 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:14:34.678 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:34 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:14:34.678 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:34 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2825896074' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:14:34.687 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:34.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:34 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:14:34.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:34 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:14:34.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:34 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2825896074' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:14:34.711 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 63 16 99484 99468 98.664 64.0625 0.019411 0.010131 2024-03-20T09:14:35.026 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:14:35.026 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:14:35.026 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:14:35.026 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:14:35.026 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:14:35.026 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:14:35.026 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:14:35.026 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:14:35.026 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:14:35.026 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:14:35.027 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:14:35.027 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:14:35.027 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:14:35.027 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:14:35.027 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:14:35.027 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:14:35.027 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:14:35.027 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:14:35.027 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:14:35.027 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:14:35.028 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:14:35.028 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:14:35.028 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:14:35.028 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:14:35.266 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:14:35.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:35.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:14:35.374 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:14:35.413 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:35.546 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:35 smithi026 ceph-mon[68309]: pgmap v1287: 47 pgs: 47 active+clean; 5.8 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-03-20T09:14:35.546 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:35 smithi026 ceph-mon[68309]: osdmap e549: 12 total, 12 up, 10 in 2024-03-20T09:14:35.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:35 smithi115 ceph-mon[72476]: pgmap v1287: 47 pgs: 47 active+clean; 5.8 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-03-20T09:14:35.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:35 smithi115 ceph-mon[72476]: osdmap e549: 12 total, 12 up, 10 in 2024-03-20T09:14:35.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:35 smithi122 ceph-mon[73495]: pgmap v1287: 47 pgs: 47 active+clean; 5.8 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-03-20T09:14:35.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:35 smithi122 ceph-mon[73495]: osdmap e549: 12 total, 12 up, 10 in 2024-03-20T09:14:35.711 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 64 16 100610 100594 98.222 70.375 0.0180102 0.0101771 2024-03-20T09:14:36.067 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:14:36.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:36.071 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:14:36.172 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:14:36.202 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:36.231 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:14:36.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:36.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:14:36.336 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:14:36.379 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:36.711 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 65 16 101707 101691 97.7656 68.5625 0.00375146 0.0102242 2024-03-20T09:14:36.727 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:14:36.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:36.732 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:14:36.833 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:14:36.865 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:36.868 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:14:36.868 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-20T09:14:36.868 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:14:36.886 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:14:36.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:36.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:14:36.992 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:14:37.022 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:37.043 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:14:37.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:37.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:14:37.149 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:14:37.182 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:37.203 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:14:37.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:37.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:14:37.307 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:14:37.378 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:37.396 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:37 smithi122 ceph-mon[73495]: pgmap v1289: 47 pgs: 47 active+clean; 6.1 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-03-20T09:14:37.396 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:37 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3491221176' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:14:37.396 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:37 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:14:37.405 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:14:37.443 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:14:37.444 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-03-20T09:14:37.444 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-03-20T09:14:37.444 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-03-20T09:14:37.444 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.444 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:109470 3.7 3:e4fe868b:::benchmark_data_smithi122_119671_object9688:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-03-20T09:14:37.444 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:05:18.976612+0000", 2024-03-20T09:14:37.444 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 558.461916772, 2024-03-20T09:14:37.444 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2614700619999999, 2024-03-20T09:14:37.444 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:37.444 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:14:37.444 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:14:37.444 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:14:37.444 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:14:37.444 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 109470 2024-03-20T09:14:37.445 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.445 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:37.445 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.445 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:37.445 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:18.976612+0000", 2024-03-20T09:14:37.445 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.445 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.445 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.445 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:37.445 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:18.976612+0000", 2024-03-20T09:14:37.445 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.445 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.445 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.446 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:37.446 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:18.976613+0000", 2024-03-20T09:14:37.446 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.215e-06 2024-03-20T09:14:37.446 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.446 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.446 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:37.446 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:18.976619+0000", 2024-03-20T09:14:37.446 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9789999999999998e-06 2024-03-20T09:14:37.446 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.446 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.446 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:37.446 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:18.976631+0000", 2024-03-20T09:14:37.446 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2361000000000001e-05 2024-03-20T09:14:37.447 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.447 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.447 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:37.447 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:18.976637+0000", 2024-03-20T09:14:37.447 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2779999999999999e-06 2024-03-20T09:14:37.447 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.447 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.447 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.447 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:18.976691+0000", 2024-03-20T09:14:37.447 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3838999999999997e-05 2024-03-20T09:14:37.447 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.447 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.447 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:14:37.448 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:18.976700+0000", 2024-03-20T09:14:37.448 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9020000000000005e-06 2024-03-20T09:14:37.448 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.448 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.448 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.448 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.965395+0000", 2024-03-20T09:14:37.448 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.98869506799999995 2024-03-20T09:14:37.448 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.448 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.448 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:14:37.448 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.965408+0000", 2024-03-20T09:14:37.448 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.273e-05 2024-03-20T09:14:37.448 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.449 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.449 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.449 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.231957+0000", 2024-03-20T09:14:37.449 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.266549559 2024-03-20T09:14:37.449 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.449 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.449 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:14:37.449 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.232118+0000", 2024-03-20T09:14:37.449 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000160545 2024-03-20T09:14:37.449 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.449 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.449 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:14:37.449 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.232261+0000", 2024-03-20T09:14:37.449 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014308699999999999 2024-03-20T09:14:37.449 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.450 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.450 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.450 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.237013+0000", 2024-03-20T09:14:37.450 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0047517230000000002 2024-03-20T09:14:37.450 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.450 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.450 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:14:37.450 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.237027+0000", 2024-03-20T09:14:37.450 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3733e-05 2024-03-20T09:14:37.450 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.450 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.450 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.450 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.237061+0000", 2024-03-20T09:14:37.451 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4660999999999999e-05 2024-03-20T09:14:37.451 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.451 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.451 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:14:37.451 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.237101+0000", 2024-03-20T09:14:37.451 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0173999999999999e-05 2024-03-20T09:14:37.451 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.451 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.451 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:37.451 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.238082+0000", 2024-03-20T09:14:37.451 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00098020799999999999 2024-03-20T09:14:37.451 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.451 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:37.451 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.452 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.452 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.452 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:109473 3.e 3:7e47c95e:::benchmark_data_smithi122_119671_object9691:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-03-20T09:14:37.452 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:05:19.000617+0000", 2024-03-20T09:14:37.452 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 558.437911493, 2024-03-20T09:14:37.452 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1810451529999999, 2024-03-20T09:14:37.452 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:37.452 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:14:37.452 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:14:37.452 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:14:37.452 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:14:37.452 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 109473 2024-03-20T09:14:37.452 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.452 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:37.452 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.453 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:37.453 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.000617+0000", 2024-03-20T09:14:37.453 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.453 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.453 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.453 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:37.453 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.000617+0000", 2024-03-20T09:14:37.453 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.453 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.453 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.453 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:37.453 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.000618+0000", 2024-03-20T09:14:37.453 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5199999999999995e-07 2024-03-20T09:14:37.453 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.454 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.454 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:37.454 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.000622+0000", 2024-03-20T09:14:37.454 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8909999999999997e-06 2024-03-20T09:14:37.454 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.454 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.454 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:37.454 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.000632+0000", 2024-03-20T09:14:37.454 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0019e-05 2024-03-20T09:14:37.454 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.454 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.454 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:37.454 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.000637+0000", 2024-03-20T09:14:37.455 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.835e-06 2024-03-20T09:14:37.455 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.455 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.455 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.455 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.000696+0000", 2024-03-20T09:14:37.455 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8306e-05 2024-03-20T09:14:37.455 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.455 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.455 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:14:37.455 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.000703+0000", 2024-03-20T09:14:37.455 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7230000000000004e-06 2024-03-20T09:14:37.455 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.455 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.455 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.455 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.164628+0000", 2024-03-20T09:14:37.456 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1639244790000001 2024-03-20T09:14:37.456 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.456 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.456 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:14:37.456 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.164819+0000", 2024-03-20T09:14:37.456 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00019093499999999999 2024-03-20T09:14:37.456 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.456 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.456 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 4,10", 2024-03-20T09:14:37.456 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.165013+0000", 2024-03-20T09:14:37.456 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000193845 2024-03-20T09:14:37.456 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.456 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.456 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.456 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.173194+0000", 2024-03-20T09:14:37.457 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0081816349999999996 2024-03-20T09:14:37.457 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.457 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.457 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.457 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.173265+0000", 2024-03-20T09:14:37.457 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0659000000000002e-05 2024-03-20T09:14:37.457 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.457 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.457 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:14:37.457 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.177354+0000", 2024-03-20T09:14:37.457 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040886100000000003 2024-03-20T09:14:37.457 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.457 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.458 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:14:37.458 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.177423+0000", 2024-03-20T09:14:37.458 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9135999999999997e-05 2024-03-20T09:14:37.458 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.458 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.458 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:37.458 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.181662+0000", 2024-03-20T09:14:37.458 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0042392280000000003 2024-03-20T09:14:37.458 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.458 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:37.458 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.458 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.458 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.458 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:109486 3.7 3:e5e21489:::benchmark_data_smithi122_119671_object9704:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-03-20T09:14:37.459 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:05:19.069650+0000", 2024-03-20T09:14:37.459 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 558.36887863599998, 2024-03-20T09:14:37.459 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1732460900000001, 2024-03-20T09:14:37.459 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:37.459 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:14:37.459 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:14:37.459 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:14:37.459 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:14:37.459 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 109486 2024-03-20T09:14:37.459 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.459 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:37.459 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.459 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:37.459 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.069650+0000", 2024-03-20T09:14:37.460 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.460 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.460 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.460 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:37.460 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.069650+0000", 2024-03-20T09:14:37.460 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.460 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.460 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.460 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:37.460 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.069651+0000", 2024-03-20T09:14:37.460 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0800000000000003e-07 2024-03-20T09:14:37.460 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.460 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.460 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:37.460 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.069655+0000", 2024-03-20T09:14:37.461 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4530000000000004e-06 2024-03-20T09:14:37.461 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.461 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.461 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:37.461 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.069666+0000", 2024-03-20T09:14:37.461 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0779e-05 2024-03-20T09:14:37.461 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.461 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.461 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:37.461 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.069672+0000", 2024-03-20T09:14:37.461 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9680000000000003e-06 2024-03-20T09:14:37.461 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.461 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.461 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.462 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.069719+0000", 2024-03-20T09:14:37.462 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7648000000000002e-05 2024-03-20T09:14:37.462 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.462 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.462 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:14:37.462 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.069727+0000", 2024-03-20T09:14:37.462 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8220000000000007e-06 2024-03-20T09:14:37.462 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.462 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.462 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.462 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.965435+0000", 2024-03-20T09:14:37.462 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.89570771199999999 2024-03-20T09:14:37.462 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.463 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.463 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:14:37.463 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.965444+0000", 2024-03-20T09:14:37.463 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5709999999999998e-06 2024-03-20T09:14:37.463 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.463 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.463 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.463 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.235231+0000", 2024-03-20T09:14:37.463 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2697870090000001 2024-03-20T09:14:37.463 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.463 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.463 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:14:37.463 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.235347+0000", 2024-03-20T09:14:37.463 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000116824 2024-03-20T09:14:37.463 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.464 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.464 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:14:37.464 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.235451+0000", 2024-03-20T09:14:37.464 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000103905 2024-03-20T09:14:37.464 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.464 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.464 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.464 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.238223+0000", 2024-03-20T09:14:37.464 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027718970000000002 2024-03-20T09:14:37.464 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.464 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.464 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:14:37.464 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.239892+0000", 2024-03-20T09:14:37.464 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0016691900000000001 2024-03-20T09:14:37.465 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.465 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.465 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.465 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.241770+0000", 2024-03-20T09:14:37.465 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0018779910000000001 2024-03-20T09:14:37.465 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.465 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.465 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:14:37.465 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.241836+0000", 2024-03-20T09:14:37.465 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6020999999999997e-05 2024-03-20T09:14:37.465 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.465 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.465 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:37.465 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.242896+0000", 2024-03-20T09:14:37.466 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010593919999999999 2024-03-20T09:14:37.466 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.466 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:37.466 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.466 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.466 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.466 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:109496 3.e 3:7c4f1f5a:::benchmark_data_smithi122_119671_object9714:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-03-20T09:14:37.466 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:05:19.128008+0000", 2024-03-20T09:14:37.466 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 558.31052051100005, 2024-03-20T09:14:37.466 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0608756690000001, 2024-03-20T09:14:37.466 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:37.466 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:14:37.466 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:14:37.467 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:14:37.467 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:14:37.467 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 109496 2024-03-20T09:14:37.467 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.467 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:37.467 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.467 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:37.467 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.128008+0000", 2024-03-20T09:14:37.467 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.467 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.467 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.467 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:37.467 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.128008+0000", 2024-03-20T09:14:37.467 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.468 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.468 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.468 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:37.468 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.128009+0000", 2024-03-20T09:14:37.468 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9500000000000001e-07 2024-03-20T09:14:37.468 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.468 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.468 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:37.468 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.128013+0000", 2024-03-20T09:14:37.468 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9709999999999998e-06 2024-03-20T09:14:37.468 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.468 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.468 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:37.468 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.128020+0000", 2024-03-20T09:14:37.468 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6650000000000003e-06 2024-03-20T09:14:37.469 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.469 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.469 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:37.469 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.128027+0000", 2024-03-20T09:14:37.469 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8170000000000001e-06 2024-03-20T09:14:37.469 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.469 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.469 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.469 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.128074+0000", 2024-03-20T09:14:37.469 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6990000000000002e-05 2024-03-20T09:14:37.469 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.469 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.469 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:14:37.469 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.128082+0000", 2024-03-20T09:14:37.470 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0499999999999992e-06 2024-03-20T09:14:37.470 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.470 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.470 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.470 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.173318+0000", 2024-03-20T09:14:37.470 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0452362719999999 2024-03-20T09:14:37.470 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.470 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.470 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:14:37.470 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.173517+0000", 2024-03-20T09:14:37.470 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000198106 2024-03-20T09:14:37.470 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.470 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.470 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 4,10", 2024-03-20T09:14:37.471 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.173709+0000", 2024-03-20T09:14:37.471 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000192315 2024-03-20T09:14:37.471 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.471 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.471 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.471 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.184748+0000", 2024-03-20T09:14:37.471 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011038773999999999 2024-03-20T09:14:37.471 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.471 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.471 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:14:37.471 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.184763+0000", 2024-03-20T09:14:37.471 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5145e-05 2024-03-20T09:14:37.471 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.472 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.472 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.472 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.184850+0000", 2024-03-20T09:14:37.472 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7090999999999997e-05 2024-03-20T09:14:37.472 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.472 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.472 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:14:37.472 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.184907+0000", 2024-03-20T09:14:37.472 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7281000000000003e-05 2024-03-20T09:14:37.472 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.472 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.472 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:37.472 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.188883+0000", 2024-03-20T09:14:37.472 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0039762970000000002 2024-03-20T09:14:37.473 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.473 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:37.473 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.473 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.473 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.473 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:109497 3.7 3:e0cb8a2c:::benchmark_data_smithi122_119671_object9715:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-03-20T09:14:37.473 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:05:19.132859+0000", 2024-03-20T09:14:37.473 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 558.30566931099997, 2024-03-20T09:14:37.473 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.116806387, 2024-03-20T09:14:37.473 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:37.473 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:14:37.473 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:14:37.473 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:14:37.473 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:14:37.474 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 109497 2024-03-20T09:14:37.474 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.474 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:37.474 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.474 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:37.474 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.132859+0000", 2024-03-20T09:14:37.474 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.474 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.474 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.474 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:37.474 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.132859+0000", 2024-03-20T09:14:37.474 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.474 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.475 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.475 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:37.475 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.132860+0000", 2024-03-20T09:14:37.475 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0800000000000004e-07 2024-03-20T09:14:37.475 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.475 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.475 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:37.475 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.132864+0000", 2024-03-20T09:14:37.475 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0110000000000002e-06 2024-03-20T09:14:37.475 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.475 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.475 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:37.475 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.132874+0000", 2024-03-20T09:14:37.475 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0051000000000001e-05 2024-03-20T09:14:37.476 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.476 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.477 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:37.477 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.132879+0000", 2024-03-20T09:14:37.477 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4689999999999996e-06 2024-03-20T09:14:37.477 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.477 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.477 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.478 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.132924+0000", 2024-03-20T09:14:37.478 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4607000000000002e-05 2024-03-20T09:14:37.478 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.479 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.479 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:14:37.479 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.132932+0000", 2024-03-20T09:14:37.479 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7649999999999993e-06 2024-03-20T09:14:37.479 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.479 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.479 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.479 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.965465+0000", 2024-03-20T09:14:37.479 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.83253320200000003 2024-03-20T09:14:37.479 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.479 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.479 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:14:37.479 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.965469+0000", 2024-03-20T09:14:37.479 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5569999999999997e-06 2024-03-20T09:14:37.480 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.480 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.480 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.480 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.238252+0000", 2024-03-20T09:14:37.480 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.272782562 2024-03-20T09:14:37.480 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.480 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.480 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:14:37.480 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.238368+0000", 2024-03-20T09:14:37.480 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000116007 2024-03-20T09:14:37.480 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.480 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.480 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:14:37.480 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.238467+0000", 2024-03-20T09:14:37.481 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9407000000000006e-05 2024-03-20T09:14:37.481 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.481 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.481 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.481 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.243009+0000", 2024-03-20T09:14:37.481 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0045413959999999996 2024-03-20T09:14:37.481 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.481 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.481 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.481 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.243053+0000", 2024-03-20T09:14:37.481 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4314000000000003e-05 2024-03-20T09:14:37.481 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.481 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.481 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:14:37.482 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.248544+0000", 2024-03-20T09:14:37.482 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0054908370000000001 2024-03-20T09:14:37.482 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.482 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.482 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:14:37.482 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.248595+0000", 2024-03-20T09:14:37.482 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1396000000000001e-05 2024-03-20T09:14:37.482 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.482 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.482 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:37.482 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.249665+0000", 2024-03-20T09:14:37.482 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001070098 2024-03-20T09:14:37.482 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.482 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:37.483 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.483 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.483 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.483 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:109500 3.e 3:77c65ac3:::benchmark_data_smithi122_119671_object9718:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-03-20T09:14:37.483 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:05:19.222091+0000", 2024-03-20T09:14:37.483 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 558.21643687000005, 2024-03-20T09:14:37.483 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.96992302600000002, 2024-03-20T09:14:37.483 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:37.483 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:14:37.483 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:14:37.483 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:14:37.483 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:14:37.483 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 109500 2024-03-20T09:14:37.483 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.483 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:37.484 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.484 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:37.484 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.222091+0000", 2024-03-20T09:14:37.484 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.484 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.484 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.484 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:37.484 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.222091+0000", 2024-03-20T09:14:37.484 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.484 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.484 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.484 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:37.484 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.222092+0000", 2024-03-20T09:14:37.484 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9999999999999995e-07 2024-03-20T09:14:37.485 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.485 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.485 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:37.485 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.222097+0000", 2024-03-20T09:14:37.485 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6099999999999999e-06 2024-03-20T09:14:37.485 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.485 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.485 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:37.485 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.222110+0000", 2024-03-20T09:14:37.485 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2886e-05 2024-03-20T09:14:37.485 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.485 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.485 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:37.485 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.222116+0000", 2024-03-20T09:14:37.485 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8270000000000003e-06 2024-03-20T09:14:37.486 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.486 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.486 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.486 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.222178+0000", 2024-03-20T09:14:37.486 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1892999999999998e-05 2024-03-20T09:14:37.486 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.486 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.486 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:14:37.486 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.222191+0000", 2024-03-20T09:14:37.486 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3135999999999999e-05 2024-03-20T09:14:37.486 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.486 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.486 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.487 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.181713+0000", 2024-03-20T09:14:37.487 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95952252599999999 2024-03-20T09:14:37.487 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.487 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.487 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:14:37.487 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.181826+0000", 2024-03-20T09:14:37.487 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011270299999999999 2024-03-20T09:14:37.487 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.487 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.487 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 4,10", 2024-03-20T09:14:37.487 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.181928+0000", 2024-03-20T09:14:37.487 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000101811 2024-03-20T09:14:37.487 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.488 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.488 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.488 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.188936+0000", 2024-03-20T09:14:37.488 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0070081040000000002 2024-03-20T09:14:37.488 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.488 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.488 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.488 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.188973+0000", 2024-03-20T09:14:37.488 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6600999999999997e-05 2024-03-20T09:14:37.488 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.488 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.488 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:14:37.488 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.188984+0000", 2024-03-20T09:14:37.488 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1765e-05 2024-03-20T09:14:37.489 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.489 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.489 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:14:37.489 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.189023+0000", 2024-03-20T09:14:37.489 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8507000000000002e-05 2024-03-20T09:14:37.489 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.489 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.489 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:37.489 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.192014+0000", 2024-03-20T09:14:37.489 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002991657 2024-03-20T09:14:37.489 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.489 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:37.489 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.489 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.489 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.490 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:109502 3.7 3:e43407e3:::benchmark_data_smithi122_119671_object9720:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-03-20T09:14:37.490 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:05:19.242690+0000", 2024-03-20T09:14:37.490 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 558.19583797200005, 2024-03-20T09:14:37.490 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0126284139999999, 2024-03-20T09:14:37.490 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:37.490 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:14:37.490 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:14:37.490 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:14:37.490 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:14:37.490 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 109502 2024-03-20T09:14:37.490 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.490 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:37.490 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.490 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:37.491 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.242690+0000", 2024-03-20T09:14:37.491 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.491 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.491 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.491 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:37.491 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.242690+0000", 2024-03-20T09:14:37.491 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.491 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.491 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.491 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:37.491 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.242691+0000", 2024-03-20T09:14:37.491 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.23e-07 2024-03-20T09:14:37.491 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.491 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.491 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:37.492 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.242695+0000", 2024-03-20T09:14:37.492 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8210000000000003e-06 2024-03-20T09:14:37.492 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.492 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.492 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:37.492 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.242706+0000", 2024-03-20T09:14:37.492 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0698e-05 2024-03-20T09:14:37.492 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.492 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.492 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:37.492 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.242711+0000", 2024-03-20T09:14:37.492 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6790000000000002e-06 2024-03-20T09:14:37.492 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.492 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.492 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.493 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.242768+0000", 2024-03-20T09:14:37.493 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6591000000000001e-05 2024-03-20T09:14:37.493 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.493 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.493 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:14:37.493 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.242779+0000", 2024-03-20T09:14:37.493 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1226999999999999e-05 2024-03-20T09:14:37.493 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.493 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.493 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.493 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.965484+0000", 2024-03-20T09:14:37.493 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.72270470499999995 2024-03-20T09:14:37.493 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.493 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.493 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:14:37.494 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.965488+0000", 2024-03-20T09:14:37.494 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2220000000000004e-06 2024-03-20T09:14:37.494 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.494 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.494 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.494 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.239928+0000", 2024-03-20T09:14:37.494 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.274439909 2024-03-20T09:14:37.494 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.494 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.494 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:14:37.494 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.240095+0000", 2024-03-20T09:14:37.494 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016686500000000001 2024-03-20T09:14:37.494 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.494 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.495 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:14:37.495 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.240239+0000", 2024-03-20T09:14:37.495 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000143886 2024-03-20T09:14:37.495 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.495 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.495 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.495 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.243093+0000", 2024-03-20T09:14:37.495 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0028540810000000001 2024-03-20T09:14:37.495 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.495 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.495 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:14:37.495 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.249673+0000", 2024-03-20T09:14:37.495 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0065800520000000003 2024-03-20T09:14:37.495 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.495 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.496 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.496 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.254272+0000", 2024-03-20T09:14:37.496 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.004599087 2024-03-20T09:14:37.496 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.496 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.496 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:14:37.496 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.254318+0000", 2024-03-20T09:14:37.496 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5946999999999999e-05 2024-03-20T09:14:37.496 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.496 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.496 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:37.496 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.255319+0000", 2024-03-20T09:14:37.496 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010009210000000001 2024-03-20T09:14:37.496 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.496 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:37.497 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.497 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.497 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.497 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:109505 3.7 3:e7605f1b:::benchmark_data_smithi122_119671_object9723:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-03-20T09:14:37.497 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:05:19.307126+0000", 2024-03-20T09:14:37.497 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 558.13140224400001, 2024-03-20T09:14:37.497 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.948903968, 2024-03-20T09:14:37.497 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:37.497 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:14:37.497 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:14:37.497 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:14:37.497 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:14:37.497 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 109505 2024-03-20T09:14:37.497 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.498 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:37.498 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.498 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:37.498 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.307126+0000", 2024-03-20T09:14:37.498 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.498 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.498 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.498 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:37.498 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.307126+0000", 2024-03-20T09:14:37.498 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.498 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.498 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.498 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:37.498 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.307127+0000", 2024-03-20T09:14:37.498 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2900000000000002e-07 2024-03-20T09:14:37.499 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.499 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.499 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:37.499 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.307131+0000", 2024-03-20T09:14:37.499 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2220000000000004e-06 2024-03-20T09:14:37.499 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.499 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.499 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:37.499 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.307143+0000", 2024-03-20T09:14:37.499 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1657e-05 2024-03-20T09:14:37.499 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.499 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.499 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:37.499 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.307149+0000", 2024-03-20T09:14:37.500 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7039999999999999e-06 2024-03-20T09:14:37.500 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.500 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.500 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.500 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.307197+0000", 2024-03-20T09:14:37.500 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8875000000000002e-05 2024-03-20T09:14:37.500 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.500 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.500 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:14:37.500 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.307205+0000", 2024-03-20T09:14:37.500 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6420000000000007e-06 2024-03-20T09:14:37.500 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.500 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.500 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.500 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.965501+0000", 2024-03-20T09:14:37.501 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.65829631099999997 2024-03-20T09:14:37.501 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.501 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.501 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:14:37.501 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:19.965506+0000", 2024-03-20T09:14:37.501 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8160000000000001e-06 2024-03-20T09:14:37.501 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.501 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.501 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.501 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.243120+0000", 2024-03-20T09:14:37.501 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.277613938 2024-03-20T09:14:37.501 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.501 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.501 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:14:37.501 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.243259+0000", 2024-03-20T09:14:37.502 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013856 2024-03-20T09:14:37.502 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.502 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.502 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:14:37.502 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.243400+0000", 2024-03-20T09:14:37.502 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000141695 2024-03-20T09:14:37.502 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.502 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.502 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.502 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.249730+0000", 2024-03-20T09:14:37.502 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0063300129999999998 2024-03-20T09:14:37.502 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.502 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.502 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:14:37.503 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.249788+0000", 2024-03-20T09:14:37.503 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7887000000000003e-05 2024-03-20T09:14:37.503 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.503 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.503 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.503 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.255363+0000", 2024-03-20T09:14:37.503 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0055745650000000001 2024-03-20T09:14:37.503 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.503 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.503 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:14:37.503 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.255401+0000", 2024-03-20T09:14:37.503 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8290000000000001e-05 2024-03-20T09:14:37.503 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.503 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.503 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:37.504 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.256030+0000", 2024-03-20T09:14:37.504 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00062886400000000003 2024-03-20T09:14:37.504 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.504 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:37.504 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.504 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.504 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.504 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:109532 3.7 3:e602d675:::benchmark_data_smithi122_119671_object9750:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-03-20T09:14:37.504 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:05:20.189263+0000", 2024-03-20T09:14:37.504 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 557.24926579099997, 2024-03-20T09:14:37.504 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0674772130000001, 2024-03-20T09:14:37.504 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:37.504 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:14:37.504 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:14:37.505 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:14:37.505 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:14:37.505 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 109532 2024-03-20T09:14:37.505 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.505 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:37.505 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.505 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:37.505 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.189263+0000", 2024-03-20T09:14:37.505 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.505 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.505 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.505 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:37.505 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.189263+0000", 2024-03-20T09:14:37.505 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.506 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.506 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.506 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:37.506 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.189264+0000", 2024-03-20T09:14:37.506 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4899999999999999e-06 2024-03-20T09:14:37.506 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.506 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.506 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:37.506 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.189270+0000", 2024-03-20T09:14:37.506 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7339999999999996e-06 2024-03-20T09:14:37.506 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.506 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.506 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:37.506 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.189281+0000", 2024-03-20T09:14:37.507 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.083e-05 2024-03-20T09:14:37.507 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.507 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.507 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:37.507 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.189299+0000", 2024-03-20T09:14:37.507 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7915e-05 2024-03-20T09:14:37.507 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.507 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.507 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.507 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.189358+0000", 2024-03-20T09:14:37.507 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9024999999999998e-05 2024-03-20T09:14:37.507 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.507 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.507 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:14:37.508 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:20.189365+0000", 2024-03-20T09:14:37.508 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8469999999999996e-06 2024-03-20T09:14:37.508 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.508 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.508 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.508 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.245076+0000", 2024-03-20T09:14:37.508 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.055710345 2024-03-20T09:14:37.508 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.508 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.508 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:14:37.508 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.245215+0000", 2024-03-20T09:14:37.508 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000139128 2024-03-20T09:14:37.508 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.508 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.509 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:14:37.509 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.245350+0000", 2024-03-20T09:14:37.509 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013474900000000001 2024-03-20T09:14:37.509 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.509 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.509 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.509 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.249778+0000", 2024-03-20T09:14:37.509 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0044287340000000001 2024-03-20T09:14:37.509 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.509 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.509 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:14:37.509 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.249795+0000", 2024-03-20T09:14:37.509 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6307000000000002e-05 2024-03-20T09:14:37.509 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.509 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.510 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.510 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.256073+0000", 2024-03-20T09:14:37.510 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0062784099999999999 2024-03-20T09:14:37.510 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.510 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.510 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:14:37.510 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.256134+0000", 2024-03-20T09:14:37.510 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0776999999999998e-05 2024-03-20T09:14:37.510 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.510 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.510 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:37.510 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:05:21.256740+0000", 2024-03-20T09:14:37.510 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00060592199999999995 2024-03-20T09:14:37.510 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.511 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:37.511 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.511 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.511 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.511 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:116565 3.e 3.f438448e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-03-20T09:14:37.511 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:06:26.946897+0000", 2024-03-20T09:14:37.511 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 490.49163107099997, 2024-03-20T09:14:37.511 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.98363714800000002, 2024-03-20T09:14:37.511 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:37.511 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:14:37.511 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:14:37.511 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:14:37.511 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:14:37.511 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 116565 2024-03-20T09:14:37.512 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.512 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:37.512 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.512 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:37.512 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:26.946897+0000", 2024-03-20T09:14:37.512 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.512 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.512 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.512 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:37.512 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:26.946897+0000", 2024-03-20T09:14:37.512 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.512 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.512 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.512 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:37.512 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:26.946900+0000", 2024-03-20T09:14:37.513 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3190000000000002e-06 2024-03-20T09:14:37.513 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.513 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.513 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:37.513 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:26.946912+0000", 2024-03-20T09:14:37.513 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1984000000000001e-05 2024-03-20T09:14:37.513 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.513 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.513 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:37.513 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:26.946926+0000", 2024-03-20T09:14:37.513 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.411e-05 2024-03-20T09:14:37.513 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.513 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.513 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:37.513 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:26.946935+0000", 2024-03-20T09:14:37.514 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5060000000000001e-06 2024-03-20T09:14:37.514 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.514 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.514 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.514 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:26.947009+0000", 2024-03-20T09:14:37.514 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3498999999999995e-05 2024-03-20T09:14:37.514 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.514 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.514 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:14:37.514 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:26.947017+0000", 2024-03-20T09:14:37.514 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2029999999999992e-06 2024-03-20T09:14:37.514 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.514 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.514 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.515 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:27.930521+0000", 2024-03-20T09:14:37.515 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.98350451000000005 2024-03-20T09:14:37.515 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.515 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.515 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:37.515 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:27.930534+0000", 2024-03-20T09:14:37.515 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3016999999999999e-05 2024-03-20T09:14:37.515 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.515 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:37.515 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.515 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.515 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.515 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:120302 3.f 3:f441e459:::benchmark_data_smithi122_119671_object20520:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-03-20T09:14:37.515 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:06:58.141873+0000", 2024-03-20T09:14:37.516 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 459.29665538199998, 2024-03-20T09:14:37.516 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.89588768900000004, 2024-03-20T09:14:37.516 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:37.516 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:14:37.516 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:14:37.516 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:14:37.516 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:14:37.516 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 120302 2024-03-20T09:14:37.516 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.516 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:37.516 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.516 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:37.516 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.141873+0000", 2024-03-20T09:14:37.516 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.517 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.517 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.517 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:37.517 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.141873+0000", 2024-03-20T09:14:37.517 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.517 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.517 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.517 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:37.517 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.141874+0000", 2024-03-20T09:14:37.517 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3319999999999999e-06 2024-03-20T09:14:37.517 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.517 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.517 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:37.517 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.141880+0000", 2024-03-20T09:14:37.518 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3619999999999997e-06 2024-03-20T09:14:37.518 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.518 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.518 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:37.518 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.141892+0000", 2024-03-20T09:14:37.518 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2704000000000001e-05 2024-03-20T09:14:37.518 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.518 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.518 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:37.518 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.141899+0000", 2024-03-20T09:14:37.518 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6919999999999996e-06 2024-03-20T09:14:37.518 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.518 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.518 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.519 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.141969+0000", 2024-03-20T09:14:37.519 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9597000000000005e-05 2024-03-20T09:14:37.519 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.519 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.519 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:14:37.519 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.141981+0000", 2024-03-20T09:14:37.519 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2658e-05 2024-03-20T09:14:37.519 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.519 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.519 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.519 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.033384+0000", 2024-03-20T09:14:37.519 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.891403116 2024-03-20T09:14:37.519 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.519 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.519 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:14:37.520 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.033505+0000", 2024-03-20T09:14:37.520 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000120519 2024-03-20T09:14:37.520 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.520 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.520 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8,10", 2024-03-20T09:14:37.520 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.033580+0000", 2024-03-20T09:14:37.520 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4938999999999995e-05 2024-03-20T09:14:37.520 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.520 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.520 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.520 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.037614+0000", 2024-03-20T09:14:37.520 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040340580000000001 2024-03-20T09:14:37.520 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.520 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.520 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:14:37.521 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.037629+0000", 2024-03-20T09:14:37.521 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4844999999999999e-05 2024-03-20T09:14:37.521 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.521 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.521 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.521 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.037673+0000", 2024-03-20T09:14:37.521 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4428000000000002e-05 2024-03-20T09:14:37.521 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.521 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.521 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:14:37.521 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.037720+0000", 2024-03-20T09:14:37.521 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7015999999999998e-05 2024-03-20T09:14:37.521 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.521 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.522 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:37.522 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.037761+0000", 2024-03-20T09:14:37.522 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0423000000000002e-05 2024-03-20T09:14:37.522 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.522 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:37.522 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.522 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.522 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.522 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:120312 3.f 3:f1c830a7:::benchmark_data_smithi122_119671_object20530:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-03-20T09:14:37.522 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:06:58.253819+0000", 2024-03-20T09:14:37.522 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 459.18470920200002, 2024-03-20T09:14:37.522 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.79499205799999995, 2024-03-20T09:14:37.522 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:37.523 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:14:37.523 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:14:37.523 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:14:37.523 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:14:37.523 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 120312 2024-03-20T09:14:37.523 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.523 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:37.523 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.523 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:37.523 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.253819+0000", 2024-03-20T09:14:37.523 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.523 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.523 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.523 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:37.523 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.253819+0000", 2024-03-20T09:14:37.524 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.524 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.524 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.524 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:37.524 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.253821+0000", 2024-03-20T09:14:37.524 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8989999999999999e-06 2024-03-20T09:14:37.524 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.524 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.524 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:37.524 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.253829+0000", 2024-03-20T09:14:37.524 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5610000000000001e-06 2024-03-20T09:14:37.524 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.524 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.524 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:37.524 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.253847+0000", 2024-03-20T09:14:37.525 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8601e-05 2024-03-20T09:14:37.525 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.525 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.525 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:37.525 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.253856+0000", 2024-03-20T09:14:37.525 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7490000000000004e-06 2024-03-20T09:14:37.525 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.525 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.525 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.525 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.253923+0000", 2024-03-20T09:14:37.525 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6773000000000006e-05 2024-03-20T09:14:37.525 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.525 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.526 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:14:37.526 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.253933+0000", 2024-03-20T09:14:37.526 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0088000000000001e-05 2024-03-20T09:14:37.526 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.526 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.526 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.526 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.037799+0000", 2024-03-20T09:14:37.526 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.78386641400000001 2024-03-20T09:14:37.526 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.526 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.526 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:14:37.526 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.037930+0000", 2024-03-20T09:14:37.526 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013070399999999999 2024-03-20T09:14:37.527 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.527 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.527 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8,10", 2024-03-20T09:14:37.527 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.038055+0000", 2024-03-20T09:14:37.527 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0001248 2024-03-20T09:14:37.527 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.527 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.527 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.527 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.048493+0000", 2024-03-20T09:14:37.527 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010438114 2024-03-20T09:14:37.527 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.527 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.527 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.527 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.048657+0000", 2024-03-20T09:14:37.528 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016442099999999999 2024-03-20T09:14:37.528 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.528 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.528 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:14:37.528 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.048726+0000", 2024-03-20T09:14:37.528 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8707999999999993e-05 2024-03-20T09:14:37.528 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.528 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.528 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:14:37.528 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.048777+0000", 2024-03-20T09:14:37.528 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0837999999999997e-05 2024-03-20T09:14:37.528 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.528 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.528 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:37.529 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.048811+0000", 2024-03-20T09:14:37.529 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4387999999999997e-05 2024-03-20T09:14:37.529 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.529 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:37.529 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.529 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.529 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.529 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:120314 3.f 3:f5df3aa9:::benchmark_data_smithi122_119671_object20532:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-03-20T09:14:37.529 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:06:58.276268+0000", 2024-03-20T09:14:37.529 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 459.16226061100002, 2024-03-20T09:14:37.529 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.77584452000000004, 2024-03-20T09:14:37.529 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:37.529 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:14:37.529 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:14:37.530 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:14:37.530 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:14:37.530 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 120314 2024-03-20T09:14:37.530 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.530 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:37.530 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.530 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:37.530 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.276268+0000", 2024-03-20T09:14:37.530 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.530 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.530 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.530 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:37.530 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.276268+0000", 2024-03-20T09:14:37.530 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.531 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.531 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.531 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:37.531 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.276269+0000", 2024-03-20T09:14:37.531 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.156e-06 2024-03-20T09:14:37.531 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.531 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.531 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:37.531 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.276276+0000", 2024-03-20T09:14:37.531 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8539999999999999e-06 2024-03-20T09:14:37.531 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.531 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.531 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:37.531 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.276296+0000", 2024-03-20T09:14:37.532 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9981000000000001e-05 2024-03-20T09:14:37.532 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.532 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.532 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:37.532 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.276305+0000", 2024-03-20T09:14:37.532 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1759999999999999e-06 2024-03-20T09:14:37.532 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.532 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.532 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.532 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.276383+0000", 2024-03-20T09:14:37.532 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8186e-05 2024-03-20T09:14:37.532 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.532 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.532 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:14:37.532 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.276397+0000", 2024-03-20T09:14:37.533 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3652000000000001e-05 2024-03-20T09:14:37.533 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.533 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.533 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.533 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.040897+0000", 2024-03-20T09:14:37.533 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.76450070299999995 2024-03-20T09:14:37.533 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.533 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.533 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:14:37.533 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.041007+0000", 2024-03-20T09:14:37.533 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000109875 2024-03-20T09:14:37.533 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.533 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.533 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8,10", 2024-03-20T09:14:37.533 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.041117+0000", 2024-03-20T09:14:37.534 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000109988 2024-03-20T09:14:37.534 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.534 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.534 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.534 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.048537+0000", 2024-03-20T09:14:37.534 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0074194059999999999 2024-03-20T09:14:37.534 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.534 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.534 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:14:37.534 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.048821+0000", 2024-03-20T09:14:37.534 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00028427900000000002 2024-03-20T09:14:37.534 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.534 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.534 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.535 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.052033+0000", 2024-03-20T09:14:37.535 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003212421 2024-03-20T09:14:37.535 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.535 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.535 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:14:37.535 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.052077+0000", 2024-03-20T09:14:37.535 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3665000000000003e-05 2024-03-20T09:14:37.535 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.535 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.535 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:37.535 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.052112+0000", 2024-03-20T09:14:37.535 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5178000000000001e-05 2024-03-20T09:14:37.535 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.535 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:37.535 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.536 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.536 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.536 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:120319 3.f 3:fe67f674:::benchmark_data_smithi122_119671_object20537:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-03-20T09:14:37.536 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:06:58.307021+0000", 2024-03-20T09:14:37.536 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 459.13150691099997, 2024-03-20T09:14:37.536 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.74520134299999996, 2024-03-20T09:14:37.536 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:37.536 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:14:37.536 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:14:37.536 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:14:37.536 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:14:37.536 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 120319 2024-03-20T09:14:37.536 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.536 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:37.536 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.537 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:37.537 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.307021+0000", 2024-03-20T09:14:37.537 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.537 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.537 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.537 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:37.537 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.307021+0000", 2024-03-20T09:14:37.537 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.537 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.537 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.537 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:37.537 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.307022+0000", 2024-03-20T09:14:37.537 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.051e-06 2024-03-20T09:14:37.537 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.538 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.538 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:37.538 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.307028+0000", 2024-03-20T09:14:37.538 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6180000000000003e-06 2024-03-20T09:14:37.538 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.538 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.538 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:37.538 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.307040+0000", 2024-03-20T09:14:37.538 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.199e-05 2024-03-20T09:14:37.538 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.538 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.538 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:37.538 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.307047+0000", 2024-03-20T09:14:37.538 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4269999999999996e-06 2024-03-20T09:14:37.539 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.539 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.539 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.539 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.307101+0000", 2024-03-20T09:14:37.539 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4954999999999998e-05 2024-03-20T09:14:37.539 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.539 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.539 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:14:37.539 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:58.307110+0000", 2024-03-20T09:14:37.539 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3049999999999993e-06 2024-03-20T09:14:37.539 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.539 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.539 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.539 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.044684+0000", 2024-03-20T09:14:37.539 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.73757395699999995 2024-03-20T09:14:37.540 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.540 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.540 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:14:37.540 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.044796+0000", 2024-03-20T09:14:37.540 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000112285 2024-03-20T09:14:37.540 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.540 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.540 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 8,10", 2024-03-20T09:14:37.540 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.044865+0000", 2024-03-20T09:14:37.540 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8495000000000001e-05 2024-03-20T09:14:37.540 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.540 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.540 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.540 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.048714+0000", 2024-03-20T09:14:37.541 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0038493149999999999 2024-03-20T09:14:37.541 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.541 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.541 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:14:37.541 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.048825+0000", 2024-03-20T09:14:37.541 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000111082 2024-03-20T09:14:37.541 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.541 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.541 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.541 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.052157+0000", 2024-03-20T09:14:37.541 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003331615 2024-03-20T09:14:37.541 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.541 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.541 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:14:37.542 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.052187+0000", 2024-03-20T09:14:37.542 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0076000000000001e-05 2024-03-20T09:14:37.542 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.542 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.542 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:37.542 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:06:59.052223+0000", 2024-03-20T09:14:37.542 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6171999999999999e-05 2024-03-20T09:14:37.542 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.542 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:37.542 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.542 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.542 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.542 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:127398 3.15 3:ade2fd22:::benchmark_data_smithi122_119671_object27616:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-03-20T09:14:37.542 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:07:48.982204+0000", 2024-03-20T09:14:37.543 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 408.456324026, 2024-03-20T09:14:37.543 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 10.622496298, 2024-03-20T09:14:37.543 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:37.543 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:14:37.543 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:14:37.543 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:14:37.543 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:14:37.543 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 127398 2024-03-20T09:14:37.543 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.543 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:37.543 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.543 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:37.543 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:48.982204+0000", 2024-03-20T09:14:37.543 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.543 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.544 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.544 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:37.544 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:48.982204+0000", 2024-03-20T09:14:37.544 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.544 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.544 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.544 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:37.544 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:48.982205+0000", 2024-03-20T09:14:37.544 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6700000000000003e-07 2024-03-20T09:14:37.544 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.544 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.544 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:37.544 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:48.982209+0000", 2024-03-20T09:14:37.544 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0629999999999999e-06 2024-03-20T09:14:37.544 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.545 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.545 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:37.545 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:48.982218+0000", 2024-03-20T09:14:37.545 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9190000000000005e-06 2024-03-20T09:14:37.545 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.545 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.545 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:37.545 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:48.982226+0000", 2024-03-20T09:14:37.545 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5830000000000001e-06 2024-03-20T09:14:37.545 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.545 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.545 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.545 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:48.982274+0000", 2024-03-20T09:14:37.545 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8028999999999997e-05 2024-03-20T09:14:37.545 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.546 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.546 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:14:37.546 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:48.982280+0000", 2024-03-20T09:14:37.546 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6359999999999997e-06 2024-03-20T09:14:37.546 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.546 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.546 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.546 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.568536+0000", 2024-03-20T09:14:37.546 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.58625587300000004 2024-03-20T09:14:37.546 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.546 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.546 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:14:37.546 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.568541+0000", 2024-03-20T09:14:37.547 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0660000000000003e-06 2024-03-20T09:14:37.547 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.547 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.547 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.547 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.540389+0000", 2024-03-20T09:14:37.547 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9718477429999997 2024-03-20T09:14:37.547 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.547 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.547 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:14:37.547 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.544144+0000", 2024-03-20T09:14:37.547 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0037547710000000001 2024-03-20T09:14:37.547 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.547 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.547 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,6,8", 2024-03-20T09:14:37.548 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.549569+0000", 2024-03-20T09:14:37.548 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.005425249 2024-03-20T09:14:37.548 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.548 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.548 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:14:37.548 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.555980+0000", 2024-03-20T09:14:37.548 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0064105960000000002 2024-03-20T09:14:37.548 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.548 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.548 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.548 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.574350+0000", 2024-03-20T09:14:37.548 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018370250000000001 2024-03-20T09:14:37.548 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.548 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.548 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.549 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.575856+0000", 2024-03-20T09:14:37.549 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001506028 2024-03-20T09:14:37.549 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.549 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.549 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.549 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.593136+0000", 2024-03-20T09:14:37.549 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017279952000000001 2024-03-20T09:14:37.549 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.549 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.549 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:14:37.549 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.595989+0000", 2024-03-20T09:14:37.549 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0028537409999999999 2024-03-20T09:14:37.549 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.549 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.550 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:37.550 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.604701+0000", 2024-03-20T09:14:37.550 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0087111319999999999 2024-03-20T09:14:37.550 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.550 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:37.550 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.550 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.550 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.550 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:127402 3.7 3:e4ec878c:::benchmark_data_smithi122_119671_object27620:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-03-20T09:14:37.550 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:07:49.102069+0000", 2024-03-20T09:14:37.550 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 408.33645967199999, 2024-03-20T09:14:37.550 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.660559699, 2024-03-20T09:14:37.550 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:37.550 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:14:37.551 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:14:37.551 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:14:37.551 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:14:37.551 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 127402 2024-03-20T09:14:37.551 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.551 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:37.551 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.551 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:37.551 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.102069+0000", 2024-03-20T09:14:37.551 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.551 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.551 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.551 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:37.551 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.102069+0000", 2024-03-20T09:14:37.552 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.552 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.552 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.552 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:37.552 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.102070+0000", 2024-03-20T09:14:37.552 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6160000000000001e-06 2024-03-20T09:14:37.552 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.552 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.552 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:37.552 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.102076+0000", 2024-03-20T09:14:37.552 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4600000000000002e-06 2024-03-20T09:14:37.552 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.552 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.552 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:37.553 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.102089+0000", 2024-03-20T09:14:37.553 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3628999999999999e-05 2024-03-20T09:14:37.553 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.553 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.553 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:37.553 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.102095+0000", 2024-03-20T09:14:37.553 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1059999999999996e-06 2024-03-20T09:14:37.553 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.553 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.553 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.553 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.102137+0000", 2024-03-20T09:14:37.553 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1773000000000001e-05 2024-03-20T09:14:37.553 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.553 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.553 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:14:37.554 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.102145+0000", 2024-03-20T09:14:37.554 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2699999999999999e-06 2024-03-20T09:14:37.554 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.554 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.554 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.554 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.567880+0000", 2024-03-20T09:14:37.554 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.465735385 2024-03-20T09:14:37.554 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.554 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.554 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:14:37.554 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.567893+0000", 2024-03-20T09:14:37.554 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3558e-05 2024-03-20T09:14:37.554 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.554 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.555 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.555 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.746011+0000", 2024-03-20T09:14:37.555 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1781172019999999 2024-03-20T09:14:37.555 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.555 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.555 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:14:37.555 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.746180+0000", 2024-03-20T09:14:37.555 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016975000000000001 2024-03-20T09:14:37.555 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.555 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.555 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,1", 2024-03-20T09:14:37.555 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.746296+0000", 2024-03-20T09:14:37.555 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000115955 2024-03-20T09:14:37.555 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.556 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.556 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:14:37.556 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.754830+0000", 2024-03-20T09:14:37.556 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0085332069999999993 2024-03-20T09:14:37.556 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.556 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.556 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.556 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.754897+0000", 2024-03-20T09:14:37.556 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7121999999999999e-05 2024-03-20T09:14:37.556 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.556 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.556 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.556 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.760396+0000", 2024-03-20T09:14:37.556 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0054992940000000001 2024-03-20T09:14:37.556 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.557 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.557 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:14:37.557 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.760474+0000", 2024-03-20T09:14:37.557 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7558999999999994e-05 2024-03-20T09:14:37.557 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.557 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.557 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:37.557 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.762628+0000", 2024-03-20T09:14:37.557 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0021548130000000002 2024-03-20T09:14:37.557 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.557 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:37.557 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.557 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.557 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.558 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:127403 3.8 3:1343bae0:::benchmark_data_smithi122_119671_object27621:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-03-20T09:14:37.558 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:07:49.118016+0000", 2024-03-20T09:14:37.558 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 408.320512748, 2024-03-20T09:14:37.558 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4850547279999999, 2024-03-20T09:14:37.558 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:37.558 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:14:37.558 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:14:37.558 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:14:37.558 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:14:37.558 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 127403 2024-03-20T09:14:37.558 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.558 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:37.558 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.558 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:37.559 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.118016+0000", 2024-03-20T09:14:37.559 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.559 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.559 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.559 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:37.559 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.118016+0000", 2024-03-20T09:14:37.559 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.559 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.559 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.559 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:37.559 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.118017+0000", 2024-03-20T09:14:37.559 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1999999999999999e-06 2024-03-20T09:14:37.559 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.559 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.559 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:37.560 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.118023+0000", 2024-03-20T09:14:37.560 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0039999999999996e-06 2024-03-20T09:14:37.560 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.560 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.560 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:37.560 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.118036+0000", 2024-03-20T09:14:37.560 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3314e-05 2024-03-20T09:14:37.560 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.560 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.560 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:37.560 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.118044+0000", 2024-03-20T09:14:37.560 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8930000000000005e-06 2024-03-20T09:14:37.560 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.560 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.560 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.561 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.118104+0000", 2024-03-20T09:14:37.561 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0387999999999998e-05 2024-03-20T09:14:37.561 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.561 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.561 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:14:37.561 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.118112+0000", 2024-03-20T09:14:37.561 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5279999999999998e-06 2024-03-20T09:14:37.561 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.561 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.561 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.561 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.565999+0000", 2024-03-20T09:14:37.561 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.447887533 2024-03-20T09:14:37.561 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.561 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.562 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:14:37.562 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.566011+0000", 2024-03-20T09:14:37.562 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1258e-05 2024-03-20T09:14:37.562 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.562 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.562 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.562 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.586244+0000", 2024-03-20T09:14:37.562 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.020233699 2024-03-20T09:14:37.562 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.562 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.562 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:14:37.562 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.586399+0000", 2024-03-20T09:14:37.562 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015447400000000001 2024-03-20T09:14:37.562 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.563 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.563 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,4", 2024-03-20T09:14:37.563 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.586546+0000", 2024-03-20T09:14:37.563 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000146627 2024-03-20T09:14:37.563 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.563 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.563 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:14:37.563 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.588915+0000", 2024-03-20T09:14:37.563 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0023696979999999999 2024-03-20T09:14:37.563 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.563 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.563 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.563 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.595655+0000", 2024-03-20T09:14:37.563 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0067396660000000001 2024-03-20T09:14:37.563 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.564 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.564 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.564 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.602897+0000", 2024-03-20T09:14:37.564 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0072418430000000004 2024-03-20T09:14:37.564 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.564 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.564 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:14:37.564 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.602961+0000", 2024-03-20T09:14:37.564 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3827999999999999e-05 2024-03-20T09:14:37.564 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.564 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.564 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:37.564 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.603070+0000", 2024-03-20T09:14:37.564 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000109775 2024-03-20T09:14:37.564 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.565 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:37.565 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.565 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.565 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.565 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:127406 3.15 3:afb25ec3:::benchmark_data_smithi122_119671_object27624:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-03-20T09:14:37.565 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:07:49.352179+0000", 2024-03-20T09:14:37.565 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 408.08634978700002, 2024-03-20T09:14:37.565 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 10.268763275, 2024-03-20T09:14:37.565 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:37.565 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:14:37.565 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:14:37.565 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:14:37.565 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:14:37.565 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 127406 2024-03-20T09:14:37.566 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.566 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:37.566 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.566 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:37.566 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.352179+0000", 2024-03-20T09:14:37.566 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.566 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.566 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.566 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:37.566 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.352179+0000", 2024-03-20T09:14:37.566 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.566 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.566 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.566 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:37.567 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.352180+0000", 2024-03-20T09:14:37.567 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.217e-06 2024-03-20T09:14:37.567 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.567 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.567 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:37.567 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.352185+0000", 2024-03-20T09:14:37.567 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4049999999999999e-06 2024-03-20T09:14:37.567 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.567 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.567 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:37.567 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.352197+0000", 2024-03-20T09:14:37.567 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2018000000000001e-05 2024-03-20T09:14:37.567 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.568 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.568 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:37.568 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.352203+0000", 2024-03-20T09:14:37.568 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8989999999999998e-06 2024-03-20T09:14:37.568 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.568 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.568 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.568 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.352248+0000", 2024-03-20T09:14:37.568 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5330000000000001e-05 2024-03-20T09:14:37.568 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.568 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.568 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:14:37.568 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.352255+0000", 2024-03-20T09:14:37.568 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8070000000000002e-06 2024-03-20T09:14:37.568 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.569 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.569 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.569 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.568560+0000", 2024-03-20T09:14:37.569 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.216305003 2024-03-20T09:14:37.569 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.569 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.569 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:14:37.569 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.568564+0000", 2024-03-20T09:14:37.569 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.036e-06 2024-03-20T09:14:37.569 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.569 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.569 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.569 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.576589+0000", 2024-03-20T09:14:37.569 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 10.008024889 2024-03-20T09:14:37.570 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.570 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.570 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:14:37.570 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.580821+0000", 2024-03-20T09:14:37.570 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.004232085 2024-03-20T09:14:37.570 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.570 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.570 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,6,8", 2024-03-20T09:14:37.570 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.586154+0000", 2024-03-20T09:14:37.570 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0053331819999999997 2024-03-20T09:14:37.570 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.570 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.570 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:14:37.570 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.605130+0000", 2024-03-20T09:14:37.571 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.018976066 2024-03-20T09:14:37.571 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.571 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.571 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.571 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.606187+0000", 2024-03-20T09:14:37.571 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001056731 2024-03-20T09:14:37.571 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.571 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.571 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.571 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.607462+0000", 2024-03-20T09:14:37.571 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0012742960000000001 2024-03-20T09:14:37.571 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.571 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.571 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.572 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.609070+0000", 2024-03-20T09:14:37.572 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0016087639999999999 2024-03-20T09:14:37.572 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.572 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.572 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:14:37.572 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.611497+0000", 2024-03-20T09:14:37.572 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0024264310000000002 2024-03-20T09:14:37.572 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.572 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.572 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:37.572 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.620942+0000", 2024-03-20T09:14:37.572 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0094451159999999999 2024-03-20T09:14:37.572 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.572 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:37.572 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.572 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.573 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.573 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:127411 3.7 3:e73aa2a1:::benchmark_data_smithi122_119671_object27629:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e367)", 2024-03-20T09:14:37.573 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:07:49.564941+0000", 2024-03-20T09:14:37.573 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 407.87358689400003, 2024-03-20T09:14:37.573 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.202028329, 2024-03-20T09:14:37.573 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:37.573 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:14:37.573 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:14:37.573 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:14:37.573 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:14:37.573 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 127411 2024-03-20T09:14:37.573 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.573 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:37.573 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.573 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:37.574 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.564941+0000", 2024-03-20T09:14:37.574 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.574 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.574 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.574 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:37.574 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.564941+0000", 2024-03-20T09:14:37.574 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.574 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.574 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.574 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:37.574 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.564943+0000", 2024-03-20T09:14:37.574 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.367e-06 2024-03-20T09:14:37.574 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.574 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.574 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:37.575 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.564950+0000", 2024-03-20T09:14:37.575 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9759999999999998e-06 2024-03-20T09:14:37.575 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.575 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.575 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:37.575 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.564963+0000", 2024-03-20T09:14:37.575 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2853e-05 2024-03-20T09:14:37.575 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.575 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.575 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:37.575 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.564973+0000", 2024-03-20T09:14:37.575 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9799999999999993e-06 2024-03-20T09:14:37.575 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.575 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.575 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.576 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.567973+0000", 2024-03-20T09:14:37.576 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0030002919999999999 2024-03-20T09:14:37.576 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.576 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.576 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:14:37.576 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:49.567981+0000", 2024-03-20T09:14:37.576 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5599999999999994e-06 2024-03-20T09:14:37.576 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.576 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.576 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.576 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.752348+0000", 2024-03-20T09:14:37.576 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.184366904 2024-03-20T09:14:37.576 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.576 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.577 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:14:37.577 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.752544+0000", 2024-03-20T09:14:37.577 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000195534 2024-03-20T09:14:37.577 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.577 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.577 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,1", 2024-03-20T09:14:37.577 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.752725+0000", 2024-03-20T09:14:37.577 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000180819 2024-03-20T09:14:37.577 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.577 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.577 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:14:37.577 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.755179+0000", 2024-03-20T09:14:37.577 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002454492 2024-03-20T09:14:37.577 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.578 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.578 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.578 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.756120+0000", 2024-03-20T09:14:37.578 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00094096700000000002 2024-03-20T09:14:37.578 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.578 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.578 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.578 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.764838+0000", 2024-03-20T09:14:37.578 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0087181940000000003 2024-03-20T09:14:37.578 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.578 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.578 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:14:37.578 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.764911+0000", 2024-03-20T09:14:37.578 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2207999999999997e-05 2024-03-20T09:14:37.579 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.579 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.579 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:37.579 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:50.766970+0000", 2024-03-20T09:14:37.579 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002059183 2024-03-20T09:14:37.579 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.579 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:37.579 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.579 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.579 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.579 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:127463 3.15 3:aad91459:::benchmark_data_smithi122_119671_object27681:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-03-20T09:14:37.579 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:07:51.028651+0000", 2024-03-20T09:14:37.579 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 406.40987699300001, 2024-03-20T09:14:37.579 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6769376420000004, 2024-03-20T09:14:37.580 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:14:37.580 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:14:37.580 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:14:37.580 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:14:37.580 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:14:37.580 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 127463 2024-03-20T09:14:37.580 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.580 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:14:37.580 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.580 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:14:37.580 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:51.028651+0000", 2024-03-20T09:14:37.580 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.580 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.580 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.580 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:14:37.580 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:51.028651+0000", 2024-03-20T09:14:37.581 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:14:37.581 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.581 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.581 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:14:37.581 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:51.028652+0000", 2024-03-20T09:14:37.581 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1340000000000001e-06 2024-03-20T09:14:37.581 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.581 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.581 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:14:37.581 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:51.028657+0000", 2024-03-20T09:14:37.581 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.814e-06 2024-03-20T09:14:37.581 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.581 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.581 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:14:37.582 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:51.028666+0000", 2024-03-20T09:14:37.582 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1549999999999996e-06 2024-03-20T09:14:37.582 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.582 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.582 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:14:37.582 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:51.028673+0000", 2024-03-20T09:14:37.582 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0959999999999997e-06 2024-03-20T09:14:37.582 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.582 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.582 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:14:37.582 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.621337+0000", 2024-03-20T09:14:37.582 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5926647089999992 2024-03-20T09:14:37.582 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.582 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.583 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:14:37.583 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.625513+0000", 2024-03-20T09:14:37.583 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0041757469999999996 2024-03-20T09:14:37.583 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.583 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.583 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,6,8", 2024-03-20T09:14:37.583 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.631687+0000", 2024-03-20T09:14:37.583 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0061737179999999999 2024-03-20T09:14:37.583 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.583 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.583 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:14:37.583 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.637587+0000", 2024-03-20T09:14:37.583 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00589997 2024-03-20T09:14:37.583 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.583 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.583 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.583 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.639336+0000", 2024-03-20T09:14:37.584 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0017490959999999999 2024-03-20T09:14:37.584 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.584 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.584 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.584 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.641169+0000", 2024-03-20T09:14:37.584 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0018331599999999999 2024-03-20T09:14:37.584 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.584 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.584 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:14:37.584 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.695540+0000", 2024-03-20T09:14:37.584 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.054371052000000003 2024-03-20T09:14:37.584 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.584 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.584 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:14:37.584 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.697681+0000", 2024-03-20T09:14:37.585 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0021405489999999998 2024-03-20T09:14:37.585 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:14:37.585 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:14:37.585 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:14:37.585 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:07:59.705589+0000", 2024-03-20T09:14:37.585 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0079084419999999999 2024-03-20T09:14:37.585 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.585 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:37.585 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.585 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:14:37.585 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:14:37.585 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:14:37.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:37 smithi115 ceph-mon[72476]: pgmap v1289: 47 pgs: 47 active+clean; 6.1 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-03-20T09:14:37.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:37 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3491221176' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:14:37.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:37 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:14:37.711 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 66 16 102795 102779 97.3146 68 0.00677846 0.0102716 2024-03-20T09:14:37.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:37 smithi026 ceph-mon[68309]: pgmap v1289: 47 pgs: 47 active+clean; 6.1 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-03-20T09:14:37.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:37 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3491221176' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:14:37.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:37 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:14:38.323 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-20T09:14:38.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:38 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:14:38.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:38 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:14:38.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:38 smithi115 ceph-mon[72476]: osdmap e550: 12 total, 12 up, 10 in 2024-03-20T09:14:38.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:38 smithi115 ceph-mon[72476]: 4.5 scrub starts 2024-03-20T09:14:38.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:38 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:14:38.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:38 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:14:38.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:38 smithi122 ceph-mon[73495]: osdmap e550: 12 total, 12 up, 10 in 2024-03-20T09:14:38.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:38 smithi122 ceph-mon[73495]: 4.5 scrub starts 2024-03-20T09:14:38.711 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 67 16 103880 103864 96.8742 67.8125 0.00429595 0.0103182 2024-03-20T09:14:38.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:38 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:14:38.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:38 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:14:38.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:38 smithi026 ceph-mon[68309]: osdmap e550: 12 total, 12 up, 10 in 2024-03-20T09:14:38.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:38 smithi026 ceph-mon[68309]: 4.5 scrub starts 2024-03-20T09:14:38.763 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:14:39.413 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:14:39.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:39.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:14:39.522 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:14:39.567 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:39.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:39 smithi115 ceph-mon[72476]: pgmap v1291: 47 pgs: 47 active+clean; 6.1 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 62 MiB/s wr, 997 op/s 2024-03-20T09:14:39.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:39 smithi115 ceph-mon[72476]: 1.6 scrub starts 2024-03-20T09:14:39.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:39 smithi115 ceph-mon[72476]: 4.a scrub starts 2024-03-20T09:14:39.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:39 smithi115 ceph-mon[72476]: 1.6 scrub ok 2024-03-20T09:14:39.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:39 smithi115 ceph-mon[72476]: 4.5 scrub ok 2024-03-20T09:14:39.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:39 smithi122 ceph-mon[73495]: pgmap v1291: 47 pgs: 47 active+clean; 6.1 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 62 MiB/s wr, 997 op/s 2024-03-20T09:14:39.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:39 smithi122 ceph-mon[73495]: 1.6 scrub starts 2024-03-20T09:14:39.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:39 smithi122 ceph-mon[73495]: 4.a scrub starts 2024-03-20T09:14:39.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:39 smithi122 ceph-mon[73495]: 1.6 scrub ok 2024-03-20T09:14:39.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:39 smithi122 ceph-mon[73495]: 4.5 scrub ok 2024-03-20T09:14:39.711 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 68 16 104929 104913 96.4137 65.5625 0.0138567 0.0103686 2024-03-20T09:14:39.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:39 smithi026 ceph-mon[68309]: pgmap v1291: 47 pgs: 47 active+clean; 6.1 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 62 MiB/s wr, 997 op/s 2024-03-20T09:14:39.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:39 smithi026 ceph-mon[68309]: 1.6 scrub starts 2024-03-20T09:14:39.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:39 smithi026 ceph-mon[68309]: 4.a scrub starts 2024-03-20T09:14:39.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:39 smithi026 ceph-mon[68309]: 1.6 scrub ok 2024-03-20T09:14:39.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:39 smithi026 ceph-mon[68309]: 4.5 scrub ok 2024-03-20T09:14:39.729 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:14:39.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:39.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:14:39.835 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:14:39.903 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:39.963 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:14:39.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:39.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:14:40.075 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:14:40.106 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:40.231 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:14:40.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:40.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:14:40.339 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:14:40.375 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:40.414 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:14:40.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:40.422 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:14:40.523 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:14:40.583 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:40.619 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:14:40.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:40.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:14:40.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:40 smithi122 ceph-mon[73495]: 4.a scrub ok 2024-03-20T09:14:40.711 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 69 16 106079 106063 96.058 71.875 0.00990319 0.0104059 2024-03-20T09:14:40.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:40 smithi026 ceph-mon[68309]: 4.a scrub ok 2024-03-20T09:14:40.724 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:14:40.731 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:40 smithi115 ceph-mon[72476]: 4.a scrub ok 2024-03-20T09:14:40.768 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:40.803 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:14:40.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:40.809 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:14:40.827 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:14:40.910 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:14:40.995 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:41.021 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:14:41.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:41.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:14:41.137 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:14:41.188 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:41.212 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:14:41.212 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [ 2024-03-20T09:14:41.212 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:14:41.213 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25690.0:106593 4.1 e550/520 4:8c0d707d:::benchmark_data_smithi122_145698_object106592:head v 550'6605, mlcod=550'6603)", 2024-03-20T09:14:41.213 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:14:41.204132+0000", 2024-03-20T09:14:41.213 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.0055672689999999997, 2024-03-20T09:14:41.213 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0055922810000000002, 2024-03-20T09:14:41.213 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:14:41.213 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:14:41.213 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:14:41.213 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:14:41.213 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:14:41.213 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:41.204132+0000", 2024-03-20T09:14:41.213 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:14:41.213 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:14:41.214 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:14:41.214 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:14:41.214 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:41.204132+0000", 2024-03-20T09:14:41.214 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:14:41.214 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:14:41.214 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:14:41.214 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:14:41.214 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:41.204133+0000", 2024-03-20T09:14:41.214 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0529999999999999e-06 2024-03-20T09:14:41.214 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:14:41.214 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:14:41.215 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:14:41.215 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:41.204165+0000", 2024-03-20T09:14:41.215 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.2456000000000002e-05 2024-03-20T09:14:41.215 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:14:41.215 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:14:41.215 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:14:41.215 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:41.204197+0000", 2024-03-20T09:14:41.215 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.1943000000000002e-05 2024-03-20T09:14:41.215 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:14:41.215 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:14:41.215 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:14:41.216 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:41.204206+0000", 2024-03-20T09:14:41.216 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.1609999999999992e-06 2024-03-20T09:14:41.216 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:14:41.216 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:14:41.216 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:14:41.216 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:41.204290+0000", 2024-03-20T09:14:41.216 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.3026000000000001e-05 2024-03-20T09:14:41.216 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:14:41.216 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:14:41.216 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:14:41.216 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:41.204311+0000", 2024-03-20T09:14:41.216 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.122e-05 2024-03-20T09:14:41.217 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:14:41.217 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:14:41.217 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:14:41.217 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:14:41.217 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:14:41.217 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25690.0:106599 4.1 e550/520 4:8a2f9af3:::benchmark_data_smithi122_145698_object106598:head v 550'6606, mlcod=550'6603)", 2024-03-20T09:14:41.217 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:14:41.207747+0000", 2024-03-20T09:14:41.217 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.001952596, 2024-03-20T09:14:41.217 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0020187669999999999, 2024-03-20T09:14:41.217 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:14:41.217 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:14:41.217 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:14:41.218 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:14:41.218 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:14:41.218 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:41.207747+0000", 2024-03-20T09:14:41.218 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:14:41.218 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:14:41.218 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:14:41.218 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:14:41.218 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:41.207747+0000", 2024-03-20T09:14:41.218 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:14:41.218 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:14:41.218 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:14:41.219 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:14:41.219 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:41.207748+0000", 2024-03-20T09:14:41.219 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.5999999999999991e-07 2024-03-20T09:14:41.219 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:14:41.219 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:14:41.219 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:14:41.219 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:41.207779+0000", 2024-03-20T09:14:41.219 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.1072999999999999e-05 2024-03-20T09:14:41.219 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:14:41.219 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:14:41.219 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:14:41.220 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:41.207792+0000", 2024-03-20T09:14:41.220 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3872e-05 2024-03-20T09:14:41.220 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:14:41.220 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:14:41.220 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:14:41.220 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:41.207802+0000", 2024-03-20T09:14:41.220 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.1840000000000005e-06 2024-03-20T09:14:41.220 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:14:41.220 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:14:41.220 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:14:41.220 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:41.207855+0000", 2024-03-20T09:14:41.220 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3452999999999998e-05 2024-03-20T09:14:41.221 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:14:41.221 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:14:41.221 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:14:41.221 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:41.207875+0000", 2024-03-20T09:14:41.221 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0009000000000001e-05 2024-03-20T09:14:41.221 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:14:41.221 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:14:41.221 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:14:41.221 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:14:41.221 INFO:teuthology.orchestra.run.smithi115.stdout: ], 2024-03-20T09:14:41.221 INFO:teuthology.orchestra.run.smithi115.stdout: "num_ops": 2 2024-03-20T09:14:41.221 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:14:41.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:41.223 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:14:41.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:14:41.289 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:14:41.323 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:14:41.352 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:41.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:41 smithi122 ceph-mon[73495]: pgmap v1292: 47 pgs: 47 active+clean; 6.2 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-03-20T09:14:41.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:41 smithi122 ceph-mon[73495]: 1.12 scrub starts 2024-03-20T09:14:41.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:41 smithi122 ceph-mon[73495]: 1.12 scrub ok 2024-03-20T09:14:41.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:41 smithi122 ceph-mon[73495]: 4.e scrub starts 2024-03-20T09:14:41.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:41 smithi122 ceph-mon[73495]: 1.15 scrub starts 2024-03-20T09:14:41.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:41 smithi122 ceph-mon[73495]: 1.15 scrub ok 2024-03-20T09:14:41.714 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 70 16 107139 107123 95.6304 66.25 0.00514851 0.0104519 2024-03-20T09:14:41.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:41 smithi026 ceph-mon[68309]: pgmap v1292: 47 pgs: 47 active+clean; 6.2 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-03-20T09:14:41.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:41 smithi026 ceph-mon[68309]: 1.12 scrub starts 2024-03-20T09:14:41.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:41 smithi026 ceph-mon[68309]: 1.12 scrub ok 2024-03-20T09:14:41.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:41 smithi026 ceph-mon[68309]: 4.e scrub starts 2024-03-20T09:14:41.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:41 smithi026 ceph-mon[68309]: 1.15 scrub starts 2024-03-20T09:14:41.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:41 smithi026 ceph-mon[68309]: 1.15 scrub ok 2024-03-20T09:14:41.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:41 smithi115 ceph-mon[72476]: pgmap v1292: 47 pgs: 47 active+clean; 6.2 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-03-20T09:14:41.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:41 smithi115 ceph-mon[72476]: 1.12 scrub starts 2024-03-20T09:14:41.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:41 smithi115 ceph-mon[72476]: 1.12 scrub ok 2024-03-20T09:14:41.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:41 smithi115 ceph-mon[72476]: 4.e scrub starts 2024-03-20T09:14:41.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:41 smithi115 ceph-mon[72476]: 1.15 scrub starts 2024-03-20T09:14:41.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:41 smithi115 ceph-mon[72476]: 1.15 scrub ok 2024-03-20T09:14:42.001 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:14:42.110 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:14:42.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:42.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:14:42.141 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-20T09:14:42.214 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:14:42.249 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:42.275 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:14:42.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:42.278 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:14:42.379 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:14:42.413 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:42.439 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:14:42.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:42.446 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:14:42.523 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:14:42.547 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:14:42.612 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:42.713 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 71 16 108163 108147 95.1849 64 0.00869712 0.0105013 2024-03-20T09:14:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:43 smithi115 ceph-mon[72476]: pgmap v1293: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-03-20T09:14:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:43 smithi115 ceph-mon[72476]: 4.e scrub ok 2024-03-20T09:14:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:43 smithi115 ceph-mon[72476]: 4.c scrub starts 2024-03-20T09:14:43.429 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:43 smithi026 ceph-mon[68309]: pgmap v1293: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-03-20T09:14:43.429 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:43 smithi026 ceph-mon[68309]: 4.e scrub ok 2024-03-20T09:14:43.429 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:43 smithi026 ceph-mon[68309]: 4.c scrub starts 2024-03-20T09:14:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:43 smithi122 ceph-mon[73495]: pgmap v1293: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.4 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-03-20T09:14:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:43 smithi122 ceph-mon[73495]: 4.e scrub ok 2024-03-20T09:14:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:43 smithi122 ceph-mon[73495]: 4.c scrub starts 2024-03-20T09:14:43.713 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 72 16 109081 109065 94.6597 57.375 0.00524168 0.0105598 2024-03-20T09:14:44.246 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:14:44.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:44.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:14:44.360 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:14:44.392 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 8, 10, 11, 3] out_osds: [9, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 5] 2024-03-20T09:14:44.392 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:14:44.392 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-20T09:14:44.392 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-20T09:14:44.397 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:44.630 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:14:44.713 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 73 16 109976 109960 94.1293 55.9375 0.0038479 0.0106186 2024-03-20T09:14:45.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:45 smithi115 ceph-mon[72476]: 4.b scrub starts 2024-03-20T09:14:45.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:45 smithi115 ceph-mon[72476]: pgmap v1294: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.5 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-03-20T09:14:45.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:45 smithi115 ceph-mon[72476]: 4.c scrub ok 2024-03-20T09:14:45.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:45 smithi122 ceph-mon[73495]: 4.b scrub starts 2024-03-20T09:14:45.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:45 smithi122 ceph-mon[73495]: pgmap v1294: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.5 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-03-20T09:14:45.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:45 smithi122 ceph-mon[73495]: 4.c scrub ok 2024-03-20T09:14:45.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:45 smithi026 ceph-mon[68309]: 4.b scrub starts 2024-03-20T09:14:45.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:45 smithi026 ceph-mon[68309]: pgmap v1294: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.5 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-03-20T09:14:45.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:45 smithi026 ceph-mon[68309]: 4.c scrub ok 2024-03-20T09:14:45.713 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 74 16 110848 110832 93.5938 54.5 0.00640368 0.0106803 2024-03-20T09:14:46.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:46 smithi026 ceph-mon[68309]: 1.1d scrub starts 2024-03-20T09:14:46.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:46 smithi026 ceph-mon[68309]: 1.1d scrub ok 2024-03-20T09:14:46.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:46 smithi026 ceph-mon[68309]: 1.9 scrub starts 2024-03-20T09:14:46.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:46 smithi026 ceph-mon[68309]: 1.9 scrub ok 2024-03-20T09:14:46.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:46 smithi026 ceph-mon[68309]: 1.10 scrub starts 2024-03-20T09:14:46.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:46 smithi026 ceph-mon[68309]: 1.10 scrub ok 2024-03-20T09:14:46.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:46 smithi115 ceph-mon[72476]: 1.1d scrub starts 2024-03-20T09:14:46.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:46 smithi115 ceph-mon[72476]: 1.1d scrub ok 2024-03-20T09:14:46.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:46 smithi115 ceph-mon[72476]: 1.9 scrub starts 2024-03-20T09:14:46.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:46 smithi115 ceph-mon[72476]: 1.9 scrub ok 2024-03-20T09:14:46.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:46 smithi115 ceph-mon[72476]: 1.10 scrub starts 2024-03-20T09:14:46.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:46 smithi115 ceph-mon[72476]: 1.10 scrub ok 2024-03-20T09:14:46.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:46 smithi122 ceph-mon[73495]: 1.1d scrub starts 2024-03-20T09:14:46.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:46 smithi122 ceph-mon[73495]: 1.1d scrub ok 2024-03-20T09:14:46.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:46 smithi122 ceph-mon[73495]: 1.9 scrub starts 2024-03-20T09:14:46.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:46 smithi122 ceph-mon[73495]: 1.9 scrub ok 2024-03-20T09:14:46.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:46 smithi122 ceph-mon[73495]: 1.10 scrub starts 2024-03-20T09:14:46.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:46 smithi122 ceph-mon[73495]: 1.10 scrub ok 2024-03-20T09:14:46.713 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 75 16 111682 111666 93.0408 52.125 0.0091246 0.0107434 2024-03-20T09:14:47.160 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:14:47.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:47.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:14:47.265 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:14:47.299 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:47.333 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:14:47.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:47.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:14:47.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:47 smithi115 ceph-mon[72476]: 1.0 scrub starts 2024-03-20T09:14:47.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:47 smithi115 ceph-mon[72476]: 1.0 scrub ok 2024-03-20T09:14:47.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:47 smithi115 ceph-mon[72476]: pgmap v1295: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.7 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-03-20T09:14:47.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:47 smithi115 ceph-mon[72476]: 1.c scrub starts 2024-03-20T09:14:47.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:47 smithi115 ceph-mon[72476]: 1.c scrub ok 2024-03-20T09:14:47.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:47 smithi115 ceph-mon[72476]: 4.2 scrub starts 2024-03-20T09:14:47.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:47 smithi115 ceph-mon[72476]: 4.b scrub ok 2024-03-20T09:14:47.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:47 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:14:47.439 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:14:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:47 smithi122 ceph-mon[73495]: 1.0 scrub starts 2024-03-20T09:14:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:47 smithi122 ceph-mon[73495]: 1.0 scrub ok 2024-03-20T09:14:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:47 smithi122 ceph-mon[73495]: pgmap v1295: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.7 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-03-20T09:14:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:47 smithi122 ceph-mon[73495]: 1.c scrub starts 2024-03-20T09:14:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:47 smithi122 ceph-mon[73495]: 1.c scrub ok 2024-03-20T09:14:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:47 smithi122 ceph-mon[73495]: 4.2 scrub starts 2024-03-20T09:14:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:47 smithi122 ceph-mon[73495]: 4.b scrub ok 2024-03-20T09:14:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:47 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:14:47.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:47 smithi026 ceph-mon[68309]: 1.0 scrub starts 2024-03-20T09:14:47.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:47 smithi026 ceph-mon[68309]: 1.0 scrub ok 2024-03-20T09:14:47.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:47 smithi026 ceph-mon[68309]: pgmap v1295: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.7 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-03-20T09:14:47.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:47 smithi026 ceph-mon[68309]: 1.c scrub starts 2024-03-20T09:14:47.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:47 smithi026 ceph-mon[68309]: 1.c scrub ok 2024-03-20T09:14:47.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:47 smithi026 ceph-mon[68309]: 4.2 scrub starts 2024-03-20T09:14:47.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:47 smithi026 ceph-mon[68309]: 4.b scrub ok 2024-03-20T09:14:47.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:47 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:14:47.492 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:47.713 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 76 16 112531 112515 92.5148 53.0625 0.00891191 0.0108046 2024-03-20T09:14:47.752 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:14:47.752 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:14:47.752 INFO:teuthology.orchestra.run.smithi115.stdout: "complaint_time": 30, 2024-03-20T09:14:47.752 INFO:teuthology.orchestra.run.smithi115.stdout: "num_blocked_ops": 0 2024-03-20T09:14:47.752 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:14:47.866 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:14:47.893 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:14:47.925 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:14:47.970 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:14:48.006 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:14:48.052 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:14:48.053 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:14:48.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:48.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:14:48.102 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:14:48.150 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:14:48.161 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:14:48.196 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:14:48.196 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:48.254 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:14:48.295 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:14:48.338 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:14:48.356 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:14:48.356 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:14:48.356 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:14:48.356 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:14:48.356 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:14:48.356 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:14:48.356 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:14:48.356 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:14:48.356 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:14:48.356 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:14:48.357 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:14:48.357 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:14:48.384 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:48 smithi026 ceph-mon[68309]: 4.7 scrub starts 2024-03-20T09:14:48.384 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:48 smithi026 ceph-mon[68309]: 4.2 scrub ok 2024-03-20T09:14:48.384 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:48 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1198212119' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:14:48.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:48 smithi115 ceph-mon[72476]: 4.7 scrub starts 2024-03-20T09:14:48.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:48 smithi115 ceph-mon[72476]: 4.2 scrub ok 2024-03-20T09:14:48.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:48 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1198212119' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:14:48.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:48 smithi122 ceph-mon[73495]: 4.7 scrub starts 2024-03-20T09:14:48.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:48 smithi122 ceph-mon[73495]: 4.2 scrub ok 2024-03-20T09:14:48.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:48 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1198212119' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:14:48.713 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 77 16 113357 113341 91.9837 51.625 0.00425458 0.010867 2024-03-20T09:14:48.968 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:14:49.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:49 smithi122 ceph-mon[73495]: 1.1 scrub starts 2024-03-20T09:14:49.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:49 smithi122 ceph-mon[73495]: 1.1 scrub ok 2024-03-20T09:14:49.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:49 smithi122 ceph-mon[73495]: pgmap v1296: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.7 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 62 MiB/s wr, 986 op/s 2024-03-20T09:14:49.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:49 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:14:49.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:49 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:14:49.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:49 smithi122 ceph-mon[73495]: osdmap e551: 12 total, 12 up, 10 in 2024-03-20T09:14:49.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:49 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:14:49.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:49 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:14:49.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:49 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:14:49.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:49 smithi122 ceph-mon[73495]: 4.7 scrub ok 2024-03-20T09:14:49.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:49 smithi026 ceph-mon[68309]: 1.1 scrub starts 2024-03-20T09:14:49.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:49 smithi026 ceph-mon[68309]: 1.1 scrub ok 2024-03-20T09:14:49.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:49 smithi026 ceph-mon[68309]: pgmap v1296: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.7 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 62 MiB/s wr, 986 op/s 2024-03-20T09:14:49.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:49 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:14:49.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:49 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:14:49.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:49 smithi026 ceph-mon[68309]: osdmap e551: 12 total, 12 up, 10 in 2024-03-20T09:14:49.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:49 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:14:49.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:49 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:14:49.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:49 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:14:49.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:49 smithi026 ceph-mon[68309]: 4.7 scrub ok 2024-03-20T09:14:49.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:49 smithi115 ceph-mon[72476]: 1.1 scrub starts 2024-03-20T09:14:49.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:49 smithi115 ceph-mon[72476]: 1.1 scrub ok 2024-03-20T09:14:49.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:49 smithi115 ceph-mon[72476]: pgmap v1296: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.7 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 62 MiB/s wr, 986 op/s 2024-03-20T09:14:49.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:49 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:14:49.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:49 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:14:49.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:49 smithi115 ceph-mon[72476]: osdmap e551: 12 total, 12 up, 10 in 2024-03-20T09:14:49.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:49 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:14:49.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:49 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:14:49.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:49 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:14:49.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:49 smithi115 ceph-mon[72476]: 4.7 scrub ok 2024-03-20T09:14:49.713 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 78 16 114413 114397 91.6506 66 0.0190061 0.0109061 2024-03-20T09:14:49.977 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:14:49.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:49.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:14:49.991 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-20T09:14:50.081 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:14:50.113 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:50.143 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:14:50.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:50.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:14:50.248 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:14:50.347 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:50.367 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:14:50.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:50 smithi115 ceph-mon[72476]: 1.16 scrub starts 2024-03-20T09:14:50.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:50 smithi115 ceph-mon[72476]: 1.16 scrub ok 2024-03-20T09:14:50.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:50 smithi122 ceph-mon[73495]: 1.16 scrub starts 2024-03-20T09:14:50.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:50 smithi122 ceph-mon[73495]: 1.16 scrub ok 2024-03-20T09:14:50.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:50 smithi026 ceph-mon[68309]: 1.16 scrub starts 2024-03-20T09:14:50.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:50 smithi026 ceph-mon[68309]: 1.16 scrub ok 2024-03-20T09:14:50.713 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 79 16 115259 115243 91.1597 52.875 0.0188753 0.0109647 2024-03-20T09:14:51.207 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:14:51.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:51 smithi115 ceph-mon[72476]: pgmap v1298: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.8 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-03-20T09:14:51.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:51 smithi115 ceph-mon[72476]: 1.14 scrub starts 2024-03-20T09:14:51.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:51 smithi115 ceph-mon[72476]: 1.14 scrub ok 2024-03-20T09:14:51.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:51 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:14:51.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:51 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:14:51.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:51 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/270596749' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:14:51.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:51 smithi122 ceph-mon[73495]: pgmap v1298: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.8 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-03-20T09:14:51.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:51 smithi122 ceph-mon[73495]: 1.14 scrub starts 2024-03-20T09:14:51.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:51 smithi122 ceph-mon[73495]: 1.14 scrub ok 2024-03-20T09:14:51.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:51 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:14:51.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:51 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:14:51.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:51 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/270596749' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:14:51.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:51 smithi026 ceph-mon[68309]: pgmap v1298: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.8 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-03-20T09:14:51.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:51 smithi026 ceph-mon[68309]: 1.14 scrub starts 2024-03-20T09:14:51.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:51 smithi026 ceph-mon[68309]: 1.14 scrub ok 2024-03-20T09:14:51.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:51 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:14:51.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:51 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:14:51.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:51 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/270596749' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:14:51.625 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:14:51.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:51.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:14:51.713 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:2024-03-20T09:14:51.713847+0000 min lat: 0.00107883 max lat: 2.19794 avg lat: 0.011038 2024-03-20T09:14:51.713 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T09:14:51.714 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 80 16 115948 115932 90.5585 43.0625 0.0318366 0.011038 2024-03-20T09:14:51.733 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:14:51.824 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:51.854 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:14:51.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:51.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:14:51.959 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:14:51.989 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:52.010 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:14:52.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:52.014 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:14:52.115 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:14:52.144 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:52.294 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:14:52.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:52.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:14:52.401 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:14:52.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:52 smithi115 ceph-mon[72476]: 4.0 scrub starts 2024-03-20T09:14:52.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:52 smithi115 ceph-mon[72476]: 1.4 scrub starts 2024-03-20T09:14:52.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:52 smithi115 ceph-mon[72476]: 1.4 scrub ok 2024-03-20T09:14:52.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:52 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:14:52.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:52 smithi115 ceph-mon[72476]: Cluster is now healthy 2024-03-20T09:14:52.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:52 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/270596749' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:14:52.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:52 smithi115 ceph-mon[72476]: osdmap e552: 12 total, 12 up, 10 in 2024-03-20T09:14:52.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:52 smithi122 ceph-mon[73495]: 4.0 scrub starts 2024-03-20T09:14:52.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:52 smithi122 ceph-mon[73495]: 1.4 scrub starts 2024-03-20T09:14:52.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:52 smithi122 ceph-mon[73495]: 1.4 scrub ok 2024-03-20T09:14:52.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:52 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:14:52.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:52 smithi122 ceph-mon[73495]: Cluster is now healthy 2024-03-20T09:14:52.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:52 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/270596749' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:14:52.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:52 smithi122 ceph-mon[73495]: osdmap e552: 12 total, 12 up, 10 in 2024-03-20T09:14:52.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:52 smithi026 ceph-mon[68309]: 4.0 scrub starts 2024-03-20T09:14:52.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:52 smithi026 ceph-mon[68309]: 1.4 scrub starts 2024-03-20T09:14:52.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:52 smithi026 ceph-mon[68309]: 1.4 scrub ok 2024-03-20T09:14:52.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:52 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:14:52.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:52 smithi026 ceph-mon[68309]: Cluster is now healthy 2024-03-20T09:14:52.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:52 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/270596749' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:14:52.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:52 smithi026 ceph-mon[68309]: osdmap e552: 12 total, 12 up, 10 in 2024-03-20T09:14:52.465 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:52.714 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 81 16 116763 116747 90.0693 50.9375 0.0100502 0.0110973 2024-03-20T09:14:52.996 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:14:52.997 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:14:53.140 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:14:53.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:53.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:14:53.250 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:14:53.278 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:53.302 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:14:53.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:53.308 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:14:53.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:53 smithi115 ceph-mon[72476]: pgmap v1300: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.0 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-03-20T09:14:53.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:53 smithi115 ceph-mon[72476]: 1.8 scrub starts 2024-03-20T09:14:53.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:53 smithi115 ceph-mon[72476]: 1.8 scrub ok 2024-03-20T09:14:53.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:53 smithi115 ceph-mon[72476]: 4.0 scrub ok 2024-03-20T09:14:53.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:53 smithi115 ceph-mon[72476]: 4.4 deep-scrub starts 2024-03-20T09:14:53.408 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:14:53.443 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:53 smithi122 ceph-mon[73495]: pgmap v1300: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.0 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-03-20T09:14:53.443 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:53 smithi122 ceph-mon[73495]: 1.8 scrub starts 2024-03-20T09:14:53.443 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:53 smithi122 ceph-mon[73495]: 1.8 scrub ok 2024-03-20T09:14:53.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:53 smithi122 ceph-mon[73495]: 4.0 scrub ok 2024-03-20T09:14:53.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:53 smithi122 ceph-mon[73495]: 4.4 deep-scrub starts 2024-03-20T09:14:53.444 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:53.467 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:14:53.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:53 smithi026 ceph-mon[68309]: pgmap v1300: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.0 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-03-20T09:14:53.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:53 smithi026 ceph-mon[68309]: 1.8 scrub starts 2024-03-20T09:14:53.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:53 smithi026 ceph-mon[68309]: 1.8 scrub ok 2024-03-20T09:14:53.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:53 smithi026 ceph-mon[68309]: 4.0 scrub ok 2024-03-20T09:14:53.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:53 smithi026 ceph-mon[68309]: 4.4 deep-scrub starts 2024-03-20T09:14:53.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:53.473 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:14:53.574 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:14:53.606 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:53.679 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:14:53.714 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 82 16 117568 117552 89.5845 50.3125 0.0174211 0.0111579 2024-03-20T09:14:53.810 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:14:53.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:53.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:14:53.821 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-20T09:14:53.914 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:14:53.948 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:53.969 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:14:53.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:53.972 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:14:54.073 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:14:54.186 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:54.250 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:14:54.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:54.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:14:54.353 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:14:54.382 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:54.402 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:14:54.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:54.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:14:54.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:54 smithi026 ceph-mon[68309]: 4.d deep-scrub starts 2024-03-20T09:14:54.506 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:14:54.531 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:54 smithi115 ceph-mon[72476]: 4.d deep-scrub starts 2024-03-20T09:14:54.533 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:54.553 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:14:54.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:54.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:14:54.658 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:14:54.687 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:54.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:54 smithi122 ceph-mon[73495]: 4.d deep-scrub starts 2024-03-20T09:14:54.707 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:14:54.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:54.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:14:54.714 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 83 16 118020 118004 88.8455 28.25 0.0104221 0.011245 2024-03-20T09:14:54.736 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:14:54.758 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:14:54.810 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:14:54.863 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:14:54.915 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:54.963 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:14:54.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:54.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:14:54.981 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-20T09:14:54.978+0000 7f92593ec700 0 --2- 172.21.15.26:0/147740188 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f9254101550 0x7f925410f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-03-20T09:14:55.070 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:14:55.102 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:55.118 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:14:55.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:55.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:14:55.222 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:14:55.253 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:55.282 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:14:55.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:55.288 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:14:55.389 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:14:55.406 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:14:55.407 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:14:55.407 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 552, 2024-03-20T09:14:55.407 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "852fbb68-e696-11ee-95c9-87774f69a715", 2024-03-20T09:14:55.407 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-03-20T08:48:35.281088+0000", 2024-03-20T09:14:55.407 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-03-20T09:14:51.167951+0000", 2024-03-20T09:14:55.407 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2024-03-20T09:10:33.084151+0000", 2024-03-20T09:14:55.407 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2024-03-20T09:10:19.369629+0000", 2024-03-20T09:14:55.407 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-20T09:14:55.407 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5799936, 2024-03-20T09:14:55.407 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2024-03-20T09:14:55.408 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2024-03-20T09:14:55.408 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2024-03-20T09:14:55.408 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2024-03-20T09:14:55.408 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2024-03-20T09:14:55.408 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:14:55.408 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 27, 2024-03-20T09:14:55.408 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2024-03-20T09:14:55.408 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-20T09:14:55.408 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-20T09:14:55.409 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2024-03-20T09:14:55.409 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 4, 2024-03-20T09:14:55.409 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 12, 2024-03-20T09:14:55.409 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "luminous", 2024-03-20T09:14:55.409 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "luminous", 2024-03-20T09:14:55.409 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "squid", 2024-03-20T09:14:55.409 INFO:teuthology.orchestra.run.smithi026.stdout: "allow_crimson": false, 2024-03-20T09:14:55.409 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2024-03-20T09:14:55.409 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.409 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2024-03-20T09:14:55.409 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": ".mgr", 2024-03-20T09:14:55.409 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-03-20T08:52:17.442324+0000", 2024-03-20T09:14:55.409 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-03-20T09:14:55.409 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-03-20T09:14:55.410 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-03-20T09:14:55.410 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 3, 2024-03-20T09:14:55.410 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 2, 2024-03-20T09:14:55.410 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-03-20T09:14:55.410 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-03-20T09:14:55.410 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-03-20T09:14:55.410 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-03-20T09:14:55.410 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-20T09:14:55.410 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-03-20T09:14:55.410 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-03-20T09:14:55.410 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 31, 2024-03-20T09:14:55.410 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 31, 2024-03-20T09:14:55.410 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 31, 2024-03-20T09:14:55.411 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 31, 2024-03-20T09:14:55.411 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 31, 2024-03-20T09:14:55.411 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-03-20T09:14:55.411 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-03-20T09:14:55.411 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-03-20T09:14:55.411 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-03-20T09:14:55.411 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-03-20T09:14:55.411 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-03-20T09:14:55.411 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-03-20T09:14:55.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.411 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "360", 2024-03-20T09:14:55.411 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-03-20T09:14:55.411 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-20T09:14:55.411 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "358", 2024-03-20T09:14:55.412 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-03-20T09:14:55.412 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-03-20T09:14:55.412 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-03-20T09:14:55.412 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-03-20T09:14:55.412 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-03-20T09:14:55.412 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-03-20T09:14:55.412 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-03-20T09:14:55.412 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-03-20T09:14:55.412 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-03-20T09:14:55.412 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-03-20T09:14:55.412 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-03-20T09:14:55.412 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-03-20T09:14:55.412 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-03-20T09:14:55.412 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-03-20T09:14:55.413 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-03-20T09:14:55.413 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-20T09:14:55.413 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-20T09:14:55.413 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-20T09:14:55.413 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-03-20T09:14:55.413 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-03-20T09:14:55.413 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-03-20T09:14:55.413 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-03-20T09:14:55.413 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-03-20T09:14:55.413 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.413 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-03-20T09:14:55.413 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-03-20T09:14:55.413 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-03-20T09:14:55.414 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-03-20T09:14:55.414 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-03-20T09:14:55.414 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-03-20T09:14:55.414 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-03-20T09:14:55.414 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-03-20T09:14:55.414 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-03-20T09:14:55.415 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-03-20T09:14:55.415 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-03-20T09:14:55.415 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2024-03-20T09:14:55.415 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_max": 32, 2024-03-20T09:14:55.415 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2024-03-20T09:14:55.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.415 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-03-20T09:14:55.415 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": {} 2024-03-20T09:14:55.415 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.415 INFO:teuthology.orchestra.run.smithi026.stdout: "read_balance": { 2024-03-20T09:14:55.415 INFO:teuthology.orchestra.run.smithi026.stdout: "score_type": "Fair distribution", 2024-03-20T09:14:55.415 INFO:teuthology.orchestra.run.smithi026.stdout: "score_acting": 2.3399999141693115, 2024-03-20T09:14:55.415 INFO:teuthology.orchestra.run.smithi026.stdout: "score_stable": 2.3399999141693115, 2024-03-20T09:14:55.415 INFO:teuthology.orchestra.run.smithi026.stdout: "optimal_score": 0.82999998331069946, 2024-03-20T09:14:55.416 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_acting": 1.940000057220459, 2024-03-20T09:14:55.416 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_stable": 1.940000057220459, 2024-03-20T09:14:55.416 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity_weighted": 0.92150616645812988, 2024-03-20T09:14:55.416 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity": 0.92000001668930054, 2024-03-20T09:14:55.416 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-03-20T09:14:55.416 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.416 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.416 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 4, 2024-03-20T09:14:55.416 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "unique_pool_2", 2024-03-20T09:14:55.416 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-03-20T09:13:20.690041+0000", 2024-03-20T09:14:55.416 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-03-20T09:14:55.416 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-03-20T09:14:55.416 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-03-20T09:14:55.417 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 3, 2024-03-20T09:14:55.417 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 2, 2024-03-20T09:14:55.417 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-03-20T09:14:55.417 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-03-20T09:14:55.417 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-03-20T09:14:55.417 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-03-20T09:14:55.417 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-20T09:14:55.417 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-03-20T09:14:55.417 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-03-20T09:14:55.417 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 16, 2024-03-20T09:14:55.418 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 16, 2024-03-20T09:14:55.418 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 16, 2024-03-20T09:14:55.418 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 16, 2024-03-20T09:14:55.418 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 16, 2024-03-20T09:14:55.418 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-03-20T09:14:55.418 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-03-20T09:14:55.418 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-03-20T09:14:55.418 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-03-20T09:14:55.418 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-03-20T09:14:55.418 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-03-20T09:14:55.418 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-03-20T09:14:55.419 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.419 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "527", 2024-03-20T09:14:55.419 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-03-20T09:14:55.419 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-20T09:14:55.419 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2024-03-20T09:14:55.419 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-03-20T09:14:55.419 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-03-20T09:14:55.419 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-03-20T09:14:55.419 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-03-20T09:14:55.419 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-03-20T09:14:55.419 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-03-20T09:14:55.419 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-03-20T09:14:55.419 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-03-20T09:14:55.419 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-03-20T09:14:55.420 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-03-20T09:14:55.420 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-03-20T09:14:55.420 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-03-20T09:14:55.420 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-03-20T09:14:55.420 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-03-20T09:14:55.420 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-03-20T09:14:55.420 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-20T09:14:55.420 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-20T09:14:55.420 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-20T09:14:55.420 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-03-20T09:14:55.420 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-03-20T09:14:55.420 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-03-20T09:14:55.420 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-03-20T09:14:55.420 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-03-20T09:14:55.421 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.421 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-03-20T09:14:55.421 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-03-20T09:14:55.421 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-03-20T09:14:55.421 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-03-20T09:14:55.421 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-03-20T09:14:55.421 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-03-20T09:14:55.421 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-03-20T09:14:55.421 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-03-20T09:14:55.421 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-03-20T09:14:55.421 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-03-20T09:14:55.421 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-03-20T09:14:55.421 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2024-03-20T09:14:55.421 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-03-20T09:14:55.421 INFO:teuthology.orchestra.run.smithi026.stdout: "rados": {} 2024-03-20T09:14:55.422 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.422 INFO:teuthology.orchestra.run.smithi026.stdout: "read_balance": { 2024-03-20T09:14:55.422 INFO:teuthology.orchestra.run.smithi026.stdout: "score_type": "Fair distribution", 2024-03-20T09:14:55.422 INFO:teuthology.orchestra.run.smithi026.stdout: "score_acting": 3.0099999904632568, 2024-03-20T09:14:55.422 INFO:teuthology.orchestra.run.smithi026.stdout: "score_stable": 3.0099999904632568, 2024-03-20T09:14:55.422 INFO:teuthology.orchestra.run.smithi026.stdout: "optimal_score": 0.82999998331069946, 2024-03-20T09:14:55.422 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_acting": 2.5, 2024-03-20T09:14:55.422 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_stable": 2.5, 2024-03-20T09:14:55.422 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity_weighted": 0.92150616645812988, 2024-03-20T09:14:55.422 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity": 0.92000001668930054, 2024-03-20T09:14:55.422 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-03-20T09:14:55.422 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.422 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.423 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:14:55.423 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:14:55.423 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.423 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-03-20T09:14:55.423 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "ad3fd9c0-316f-4fe5-a63b-d480087c1729", 2024-03-20T09:14:55.423 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:14:55.423 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:14:55.423 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:14:55.423 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:14:55.423 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:14:55.423 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:14:55.423 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 8, 2024-03-20T09:14:55.423 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 525, 2024-03-20T09:14:55.423 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:14:55.424 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:14:55.424 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:14:55.424 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.424 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.424 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2024-03-20T09:14:55.424 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:14:55.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.424 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.424 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.424 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2024-03-20T09:14:55.424 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:14:55.424 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.424 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.424 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.425 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:14:55.425 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.425 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.425 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.425 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2024-03-20T09:14:55.425 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:14:55.425 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.425 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.425 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.425 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2024-03-20T09:14:55.425 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:14:55.425 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.425 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.426 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.426 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:14:55.426 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.426 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.426 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.426 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2024-03-20T09:14:55.426 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:14:55.426 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.426 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.426 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.427 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2024-03-20T09:14:55.427 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:14:55.427 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.427 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.427 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:14:55.427 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.427 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.427 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.427 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2024-03-20T09:14:55.427 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:14:55.427 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.427 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.427 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.428 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2024-03-20T09:14:55.428 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:14:55.428 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.428 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.428 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/2139805293", 2024-03-20T09:14:55.428 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/2139805293", 2024-03-20T09:14:55.428 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/2139805293", 2024-03-20T09:14:55.428 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/2139805293", 2024-03-20T09:14:55.428 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:14:55.428 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:14:55.428 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:14:55.428 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.428 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.429 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.429 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-03-20T09:14:55.429 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e85b94f5-2954-47a1-9c20-096f2606eb04", 2024-03-20T09:14:55.429 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:14:55.429 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:14:55.429 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:14:55.429 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:14:55.429 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:14:55.429 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:14:55.429 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2024-03-20T09:14:55.429 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 533, 2024-03-20T09:14:55.429 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:14:55.429 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:14:55.429 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:14:55.429 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.430 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.430 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.430 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2024-03-20T09:14:55.430 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:14:55.430 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.430 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.430 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.430 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2024-03-20T09:14:55.430 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:14:55.431 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.431 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.431 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.431 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:14:55.431 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.431 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.431 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.431 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2024-03-20T09:14:55.431 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:14:55.431 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.431 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.431 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.431 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2024-03-20T09:14:55.431 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:14:55.431 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.432 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.432 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.432 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:14:55.432 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.432 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.432 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.432 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2024-03-20T09:14:55.432 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:14:55.432 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.432 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.432 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.432 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2024-03-20T09:14:55.432 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:14:55.432 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.433 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.433 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.433 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:14:55.433 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.433 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.433 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.433 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2024-03-20T09:14:55.433 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:14:55.433 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.433 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.433 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.433 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2024-03-20T09:14:55.433 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:14:55.433 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.433 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.434 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.434 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/3853355134", 2024-03-20T09:14:55.434 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/3853355134", 2024-03-20T09:14:55.434 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/3853355134", 2024-03-20T09:14:55.434 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/3853355134", 2024-03-20T09:14:55.434 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:14:55.434 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:14:55.434 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:14:55.434 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.434 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.434 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.434 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-03-20T09:14:55.434 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "dd2b10f0-741c-47a4-b224-ac6bfe2c14ef", 2024-03-20T09:14:55.435 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:14:55.435 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:14:55.435 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:14:55.435 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:14:55.435 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:14:55.435 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:14:55.435 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 18, 2024-03-20T09:14:55.435 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 534, 2024-03-20T09:14:55.435 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:14:55.435 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:14:55.435 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:14:55.435 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.435 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.435 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.435 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2024-03-20T09:14:55.436 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:14:55.436 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.436 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.436 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.436 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2024-03-20T09:14:55.436 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:14:55.436 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.436 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.436 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.436 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:14:55.436 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.436 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.436 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.436 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2024-03-20T09:14:55.437 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:14:55.437 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.437 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.437 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.437 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2024-03-20T09:14:55.437 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:14:55.437 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.437 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.437 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.437 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:14:55.437 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.437 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.437 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.437 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2024-03-20T09:14:55.438 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:14:55.438 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.438 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.438 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.438 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2024-03-20T09:14:55.438 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:14:55.438 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.438 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.438 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.438 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:14:55.438 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.438 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.438 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.438 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2024-03-20T09:14:55.439 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:14:55.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.439 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.439 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.439 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2024-03-20T09:14:55.439 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:14:55.439 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.439 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.439 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.439 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/3993357967", 2024-03-20T09:14:55.439 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/3993357967", 2024-03-20T09:14:55.439 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/3993357967", 2024-03-20T09:14:55.439 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/3993357967", 2024-03-20T09:14:55.439 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:14:55.439 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:14:55.440 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:14:55.440 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.440 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.440 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.440 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-03-20T09:14:55.440 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1", 2024-03-20T09:14:55.440 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:14:55.440 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:14:55.440 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:14:55.440 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:14:55.441 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:14:55.441 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:14:55.441 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 26, 2024-03-20T09:14:55.441 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 525, 2024-03-20T09:14:55.441 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:14:55.441 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:14:55.441 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:14:55.441 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.441 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.441 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.441 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2024-03-20T09:14:55.441 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:14:55.441 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.441 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.442 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.442 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2024-03-20T09:14:55.442 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:14:55.442 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.442 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.442 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.442 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:14:55.442 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.442 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.442 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.442 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2024-03-20T09:14:55.442 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:14:55.442 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.442 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.442 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.443 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2024-03-20T09:14:55.443 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:14:55.443 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.443 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.443 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.443 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:14:55.443 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.443 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.443 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.443 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2024-03-20T09:14:55.443 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:14:55.443 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.443 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.443 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.445 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2024-03-20T09:14:55.445 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:14:55.445 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.445 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.445 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.445 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:14:55.445 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.446 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.446 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.446 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2024-03-20T09:14:55.446 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:14:55.446 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.446 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.446 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.446 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2024-03-20T09:14:55.446 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:14:55.446 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.446 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.446 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.446 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/4110225509", 2024-03-20T09:14:55.446 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/4110225509", 2024-03-20T09:14:55.447 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/4110225509", 2024-03-20T09:14:55.447 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/4110225509", 2024-03-20T09:14:55.447 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:14:55.447 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:14:55.447 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:14:55.447 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.447 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.447 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.447 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-03-20T09:14:55.447 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "f14e34d5-9829-42a3-9b25-31a0976c5ef7", 2024-03-20T09:14:55.447 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:14:55.447 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 0, 2024-03-20T09:14:55.447 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-03-20T09:14:55.447 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:14:55.448 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:14:55.448 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:14:55.448 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 32, 2024-03-20T09:14:55.448 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 366, 2024-03-20T09:14:55.448 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:14:55.448 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:14:55.448 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:14:55.448 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.448 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.448 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.448 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6800", 2024-03-20T09:14:55.448 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:14:55.448 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.448 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.449 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.449 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6801", 2024-03-20T09:14:55.449 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:14:55.449 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.449 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.449 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.449 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:14:55.449 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.449 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.449 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.449 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6802", 2024-03-20T09:14:55.449 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:14:55.449 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.449 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.450 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.450 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6803", 2024-03-20T09:14:55.450 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:14:55.450 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.450 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.450 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.450 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:14:55.450 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.450 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.450 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.450 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6806", 2024-03-20T09:14:55.450 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:14:55.450 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.450 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.451 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.451 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6807", 2024-03-20T09:14:55.451 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:14:55.451 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.451 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.451 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.451 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:14:55.451 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.451 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.451 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.451 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6804", 2024-03-20T09:14:55.451 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:14:55.451 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.451 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.452 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.452 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6805", 2024-03-20T09:14:55.452 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:14:55.452 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.452 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.452 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.452 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6801/477923058", 2024-03-20T09:14:55.452 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6803/477923058", 2024-03-20T09:14:55.452 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6807/477923058", 2024-03-20T09:14:55.452 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6805/477923058", 2024-03-20T09:14:55.452 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:14:55.452 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:14:55.452 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:14:55.452 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.453 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.453 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.453 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-03-20T09:14:55.453 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "cb431b6a-540d-4318-ae35-d379e1df0185", 2024-03-20T09:14:55.453 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:14:55.453 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:14:55.453 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:14:55.453 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:14:55.453 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 41, 2024-03-20T09:14:55.453 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 273, 2024-03-20T09:14:55.453 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 443, 2024-03-20T09:14:55.453 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 520, 2024-03-20T09:14:55.453 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 274, 2024-03-20T09:14:55.453 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:14:55.454 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:14:55.454 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.454 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.454 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.454 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6808", 2024-03-20T09:14:55.454 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:14:55.454 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.454 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.454 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.454 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6809", 2024-03-20T09:14:55.454 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:14:55.454 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.454 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.454 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.455 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:14:55.455 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.455 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.455 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.455 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6810", 2024-03-20T09:14:55.455 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:14:55.455 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.455 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.455 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.455 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6811", 2024-03-20T09:14:55.455 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:14:55.455 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.456 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.456 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.456 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:14:55.456 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.456 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.456 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.456 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6814", 2024-03-20T09:14:55.456 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:14:55.456 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.456 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.456 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.456 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6815", 2024-03-20T09:14:55.456 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:14:55.456 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.457 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.457 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:14:55.457 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.457 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.457 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.457 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6812", 2024-03-20T09:14:55.457 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:14:55.457 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.457 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.457 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.457 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6813", 2024-03-20T09:14:55.457 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:14:55.457 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.458 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.458 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.458 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6809/650416648", 2024-03-20T09:14:55.458 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6811/650416648", 2024-03-20T09:14:55.458 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6815/650416648", 2024-03-20T09:14:55.458 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6813/650416648", 2024-03-20T09:14:55.458 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:14:55.458 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:14:55.458 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:14:55.458 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.458 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.458 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.458 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2024-03-20T09:14:55.458 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "753ed830-e764-4ff3-9d9b-87e7fa748b81", 2024-03-20T09:14:55.459 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:14:55.459 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:14:55.459 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:14:55.459 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:14:55.459 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:14:55.459 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:14:55.459 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 51, 2024-03-20T09:14:55.459 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 525, 2024-03-20T09:14:55.459 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:14:55.459 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:14:55.459 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:14:55.459 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.459 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.459 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.460 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6816", 2024-03-20T09:14:55.460 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:14:55.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.460 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.460 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6817", 2024-03-20T09:14:55.460 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:14:55.460 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.460 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.460 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:14:55.460 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.460 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.460 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6818", 2024-03-20T09:14:55.461 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:14:55.461 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.461 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.461 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6819", 2024-03-20T09:14:55.461 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:14:55.461 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.461 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.461 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.461 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:14:55.461 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.461 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.461 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6822", 2024-03-20T09:14:55.462 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:14:55.462 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.462 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.462 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.462 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6823", 2024-03-20T09:14:55.462 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:14:55.462 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.462 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.462 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.462 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:14:55.462 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.462 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.462 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.462 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6820", 2024-03-20T09:14:55.463 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:14:55.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.463 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.463 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.463 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6821", 2024-03-20T09:14:55.463 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:14:55.463 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.463 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.463 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6817/3278301843", 2024-03-20T09:14:55.463 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6819/3278301843", 2024-03-20T09:14:55.463 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6823/3278301843", 2024-03-20T09:14:55.463 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6821/3278301843", 2024-03-20T09:14:55.464 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:14:55.464 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:14:55.464 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:14:55.464 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.464 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.464 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.464 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2024-03-20T09:14:55.464 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "55bcbd37-7a6f-4ab1-b5a9-91a951444d86", 2024-03-20T09:14:55.464 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:14:55.464 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:14:55.464 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:14:55.464 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 0.058074951171875, 2024-03-20T09:14:55.464 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:14:55.464 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:14:55.464 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 60, 2024-03-20T09:14:55.465 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 517, 2024-03-20T09:14:55.465 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:14:55.465 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:14:55.465 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:14:55.465 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.465 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.465 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.465 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6824", 2024-03-20T09:14:55.465 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:14:55.465 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.465 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.465 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.465 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6825", 2024-03-20T09:14:55.465 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:14:55.465 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.466 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.466 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:14:55.466 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.466 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.466 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6826", 2024-03-20T09:14:55.466 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:14:55.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.466 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.466 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6827", 2024-03-20T09:14:55.466 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:14:55.467 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.467 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.467 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:14:55.467 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.467 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.467 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6830", 2024-03-20T09:14:55.467 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:14:55.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.467 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.467 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6831", 2024-03-20T09:14:55.467 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:14:55.467 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.468 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.468 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.468 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:14:55.468 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.468 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.468 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6828", 2024-03-20T09:14:55.468 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:14:55.468 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.468 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.468 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6829", 2024-03-20T09:14:55.468 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:14:55.468 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.469 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.469 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6825/3152424001", 2024-03-20T09:14:55.469 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6827/3152424001", 2024-03-20T09:14:55.469 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6831/3152424001", 2024-03-20T09:14:55.469 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6829/3152424001", 2024-03-20T09:14:55.469 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:14:55.469 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:14:55.469 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:14:55.469 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.469 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 8, 2024-03-20T09:14:55.469 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "bc86248e-6bd0-4369-b8a2-e593bc2a610d", 2024-03-20T09:14:55.470 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:14:55.470 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:14:55.470 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:14:55.470 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:14:55.470 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:14:55.470 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:14:55.470 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 70, 2024-03-20T09:14:55.470 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 496, 2024-03-20T09:14:55.470 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:14:55.470 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:14:55.470 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:14:55.470 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.470 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.470 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.471 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6800", 2024-03-20T09:14:55.471 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:14:55.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.471 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.471 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.471 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6801", 2024-03-20T09:14:55.471 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:14:55.471 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.471 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.471 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:14:55.471 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.471 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.471 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.472 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6802", 2024-03-20T09:14:55.472 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:14:55.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.472 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.472 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6803", 2024-03-20T09:14:55.472 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:14:55.472 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.472 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.472 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:14:55.472 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.473 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.473 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6806", 2024-03-20T09:14:55.473 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:14:55.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.473 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.473 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6807", 2024-03-20T09:14:55.473 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:14:55.473 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.473 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.473 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:14:55.473 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.474 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.474 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6804", 2024-03-20T09:14:55.474 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:14:55.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.474 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.474 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.474 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6805", 2024-03-20T09:14:55.474 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:14:55.474 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.474 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.474 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6801/1309426740", 2024-03-20T09:14:55.474 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6803/1309426740", 2024-03-20T09:14:55.474 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6807/1309426740", 2024-03-20T09:14:55.475 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6805/1309426740", 2024-03-20T09:14:55.475 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:14:55.475 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:14:55.475 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:14:55.475 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.475 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.475 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.475 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 9, 2024-03-20T09:14:55.475 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b20a4aa-032a-4c09-ac75-153c554424e7", 2024-03-20T09:14:55.475 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:14:55.475 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 0, 2024-03-20T09:14:55.475 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-03-20T09:14:55.475 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:14:55.475 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:14:55.476 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:14:55.476 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 80, 2024-03-20T09:14:55.476 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 257, 2024-03-20T09:14:55.476 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:14:55.476 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:14:55.476 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:14:55.476 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.476 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.476 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6808", 2024-03-20T09:14:55.476 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:14:55.476 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.476 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.476 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.477 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6809", 2024-03-20T09:14:55.477 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:14:55.477 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.477 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.477 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:14:55.477 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.477 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.477 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6810", 2024-03-20T09:14:55.477 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:14:55.477 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.477 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.477 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.478 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6811", 2024-03-20T09:14:55.478 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:14:55.478 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.478 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.478 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:14:55.478 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.478 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.478 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6814", 2024-03-20T09:14:55.478 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:14:55.478 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.478 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.478 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.479 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6815", 2024-03-20T09:14:55.479 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:14:55.479 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.479 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.479 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:14:55.479 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.479 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.479 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6812", 2024-03-20T09:14:55.479 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:14:55.479 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.479 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.479 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.479 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6813", 2024-03-20T09:14:55.480 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:14:55.480 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.480 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.480 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6809/2953210076", 2024-03-20T09:14:55.480 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6811/2953210076", 2024-03-20T09:14:55.480 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6815/2953210076", 2024-03-20T09:14:55.480 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6813/2953210076", 2024-03-20T09:14:55.480 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:14:55.480 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:14:55.480 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:14:55.480 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.480 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.480 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.481 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 10, 2024-03-20T09:14:55.481 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e62c1fb3-5dd1-4d0d-997c-d20633d19361", 2024-03-20T09:14:55.481 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:14:55.481 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:14:55.481 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:14:55.481 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:14:55.481 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:14:55.481 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:14:55.481 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 89, 2024-03-20T09:14:55.481 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 520, 2024-03-20T09:14:55.481 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:14:55.481 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:14:55.481 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:14:55.482 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.482 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.482 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6816", 2024-03-20T09:14:55.482 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:14:55.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.482 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.482 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.482 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6817", 2024-03-20T09:14:55.482 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:14:55.482 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.482 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.482 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.482 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:14:55.483 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.483 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.483 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6818", 2024-03-20T09:14:55.483 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:14:55.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.483 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.483 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.483 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6819", 2024-03-20T09:14:55.483 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:14:55.483 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.483 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.483 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.483 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:14:55.484 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.484 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.484 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6822", 2024-03-20T09:14:55.484 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:14:55.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.484 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.484 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.484 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6823", 2024-03-20T09:14:55.484 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:14:55.484 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.484 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.484 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.484 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:14:55.484 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.485 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.485 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6820", 2024-03-20T09:14:55.485 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:14:55.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.485 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.485 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.485 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6821", 2024-03-20T09:14:55.485 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:14:55.485 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.485 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.485 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.485 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6817/409394340", 2024-03-20T09:14:55.485 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6819/409394340", 2024-03-20T09:14:55.485 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6823/409394340", 2024-03-20T09:14:55.486 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6821/409394340", 2024-03-20T09:14:55.486 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:14:55.486 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:14:55.486 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:14:55.486 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.486 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.486 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.486 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 11, 2024-03-20T09:14:55.486 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b83994d5-e5ef-48b0-9758-c26843cec3b6", 2024-03-20T09:14:55.486 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:14:55.486 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:14:55.486 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:14:55.486 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:14:55.486 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:14:55.487 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:14:55.487 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 96, 2024-03-20T09:14:55.487 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 522, 2024-03-20T09:14:55.487 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:14:55.487 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:14:55.487 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:14:55.487 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.487 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.487 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6824", 2024-03-20T09:14:55.487 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:14:55.487 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.487 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.487 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.487 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6825", 2024-03-20T09:14:55.488 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:14:55.488 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.488 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.488 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:14:55.488 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.488 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.488 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6826", 2024-03-20T09:14:55.488 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:14:55.488 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.488 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.488 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.488 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6827", 2024-03-20T09:14:55.489 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:14:55.489 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.489 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.489 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:14:55.489 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.489 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.489 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6830", 2024-03-20T09:14:55.489 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:14:55.489 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.489 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.489 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.489 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6831", 2024-03-20T09:14:55.489 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:14:55.490 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.490 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.490 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:14:55.490 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:14:55.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.490 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:14:55.490 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6828", 2024-03-20T09:14:55.490 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:14:55.490 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.490 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.490 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:14:55.490 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6829", 2024-03-20T09:14:55.490 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:14:55.491 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.491 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.491 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.491 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6825/2704069354", 2024-03-20T09:14:55.491 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6827/2704069354", 2024-03-20T09:14:55.491 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6831/2704069354", 2024-03-20T09:14:55.491 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6829/2704069354", 2024-03-20T09:14:55.491 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:14:55.491 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:14:55.491 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:14:55.491 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.491 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.491 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:14:55.491 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2024-03-20T09:14:55.491 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.492 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-03-20T09:14:55.492 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:14:55.492 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:14:55.492 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:14:55.492 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:14:55.492 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:14:55.492 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:14:01.793030+0000", 2024-03-20T09:14:55.492 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:14:55.492 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.492 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.492 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-03-20T09:14:55.492 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:14:55.492 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:14:55.492 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:14:55.493 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:14:55.493 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:14:55.493 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:13:37.136739+0000", 2024-03-20T09:14:55.493 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:14:55.493 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.493 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.493 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-03-20T09:14:55.493 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:14:55.493 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:14:55.493 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:14:55.493 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:14:55.493 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:14:55.493 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:14:29.963247+0000", 2024-03-20T09:14:55.493 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:14:55.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.494 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-03-20T09:14:55.494 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:14:55.494 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:14:55.494 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:14:55.494 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:14:55.494 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:14:55.494 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:14:06.590345+0000", 2024-03-20T09:14:55.494 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:14:55.494 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.494 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.494 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-03-20T09:14:55.494 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:14:55.495 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:14:55.495 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:14:55.495 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:14:55.495 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 65536, 2024-03-20T09:14:55.495 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:14:33.513834+0000", 2024-03-20T09:14:55.495 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:14:55.495 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.495 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.495 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-03-20T09:14:55.495 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:14:55.495 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:14:55.495 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:14:55.495 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:14:55.495 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:14:55.495 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:14:27.640318+0000", 2024-03-20T09:14:55.495 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 273 2024-03-20T09:14:55.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.496 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2024-03-20T09:14:55.496 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:14:55.496 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:14:55.496 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:14:55.496 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:14:55.496 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:14:55.496 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:14:32.283694+0000", 2024-03-20T09:14:55.496 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:14:55.496 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.496 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.496 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2024-03-20T09:14:55.496 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:14:55.497 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:14:55.497 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:14:55.497 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:14:55.497 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:14:55.497 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:14:48.028594+0000", 2024-03-20T09:14:55.497 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:14:55.497 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.497 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.497 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 8, 2024-03-20T09:14:55.497 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:14:55.497 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:14:55.497 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:14:55.497 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:14:55.497 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:14:55.497 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:14:50.225887+0000", 2024-03-20T09:14:55.498 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:14:55.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.498 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 9, 2024-03-20T09:14:55.498 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:14:55.498 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:14:55.498 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:14:55.498 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:14:55.498 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 34736, 2024-03-20T09:14:55.498 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:13:53.078437+0000", 2024-03-20T09:14:55.498 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:14:55.498 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.498 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.499 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 10, 2024-03-20T09:14:55.499 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:14:55.499 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:14:55.499 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:14:55.499 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:14:55.499 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:14:55.499 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:13:59.357928+0000", 2024-03-20T09:14:55.499 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:14:55.499 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.499 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.499 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 11, 2024-03-20T09:14:55.499 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:14:55.499 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:14:55.499 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:14:55.500 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:14:55.500 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:14:55.500 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:13:30.485826+0000", 2024-03-20T09:14:55.500 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:14:55.500 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.500 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:14:55.500 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [], 2024-03-20T09:14:55.500 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [ 2024-03-20T09:14:55.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.500 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "4.f", 2024-03-20T09:14:55.500 INFO:teuthology.orchestra.run.smithi026.stdout: "mappings": [ 2024-03-20T09:14:55.500 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:14:55.500 INFO:teuthology.orchestra.run.smithi026.stdout: "from": 1, 2024-03-20T09:14:55.500 INFO:teuthology.orchestra.run.smithi026.stdout: "to": 2 2024-03-20T09:14:55.500 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.501 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:14:55.501 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.501 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:14:55.501 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_primaries": [], 2024-03-20T09:14:55.501 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2024-03-20T09:14:55.501 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2024-03-20T09:14:55.501 INFO:teuthology.orchestra.run.smithi026.stdout: "blocklist": { 2024-03-20T09:14:55.501 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/4291404191": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:14:55.501 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/642012566": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:14:55.501 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6801/1181459999": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:14:55.501 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/3984513952": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:14:55.501 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6800/1654945498": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:14:55.501 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6800/1181459999": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:14:55.501 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/137770981": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:14:55.501 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/1451237762": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:14:55.502 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/3938386631": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:14:55.502 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6801/1654945498": "2024-03-21T08:49:04.596754+0000" 2024-03-20T09:14:55.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.502 INFO:teuthology.orchestra.run.smithi026.stdout: "range_blocklist": {}, 2024-03-20T09:14:55.502 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2024-03-20T09:14:55.502 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2024-03-20T09:14:55.502 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2024-03-20T09:14:55.502 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2024-03-20T09:14:55.502 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2024-03-20T09:14:55.502 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2024-03-20T09:14:55.502 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2024-03-20T09:14:55.502 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.502 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:14:55.502 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2024-03-20T09:14:55.503 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2024-03-20T09:14:55.503 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2024-03-20T09:14:55.503 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2024-03-20T09:14:55.503 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {}, 2024-03-20T09:14:55.503 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": { 2024-03-20T09:14:55.503 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_enabled": false, 2024-03-20T09:14:55.503 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_bucket_count": 0, 2024-03-20T09:14:55.503 INFO:teuthology.orchestra.run.smithi026.stdout: "degraded_stretch_mode": 0, 2024-03-20T09:14:55.503 INFO:teuthology.orchestra.run.smithi026.stdout: "recovering_stretch_mode": 0, 2024-03-20T09:14:55.503 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_bucket": 0 2024-03-20T09:14:55.503 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:14:55.503 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:14:55.504 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:55 smithi115 ceph-mon[72476]: pgmap v1301: 47 pgs: 47 active+clean; 7.0 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 43 MiB/s wr, 690 op/s 2024-03-20T09:14:55.504 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:55 smithi115 ceph-mon[72476]: 4.f deep-scrub starts 2024-03-20T09:14:55.504 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:55 smithi115 ceph-mon[72476]: 1.19 scrub starts 2024-03-20T09:14:55.504 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:55 smithi115 ceph-mon[72476]: 1.19 scrub ok 2024-03-20T09:14:55.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:55.505 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:55.505 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:14:55.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:14:55.606 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:14:55.649 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:55 smithi026 ceph-mon[68309]: pgmap v1301: 47 pgs: 47 active+clean; 7.0 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 43 MiB/s wr, 690 op/s 2024-03-20T09:14:55.649 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:55 smithi026 ceph-mon[68309]: 4.f deep-scrub starts 2024-03-20T09:14:55.649 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:55 smithi026 ceph-mon[68309]: 1.19 scrub starts 2024-03-20T09:14:55.650 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:55 smithi026 ceph-mon[68309]: 1.19 scrub ok 2024-03-20T09:14:55.650 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:55.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:55 smithi122 ceph-mon[73495]: pgmap v1301: 47 pgs: 47 active+clean; 7.0 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 43 MiB/s wr, 690 op/s 2024-03-20T09:14:55.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:55 smithi122 ceph-mon[73495]: 4.f deep-scrub starts 2024-03-20T09:14:55.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:55 smithi122 ceph-mon[73495]: 1.19 scrub starts 2024-03-20T09:14:55.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:55 smithi122 ceph-mon[73495]: 1.19 scrub ok 2024-03-20T09:14:55.714 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 84 16 118492 118476 88.139 29.5 0.0280403 0.0113368 2024-03-20T09:14:56.485 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:56 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1910273533' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:14:56.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:56 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1910273533' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:14:56.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:56 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1910273533' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:14:56.714 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 85 16 118974 118958 87.4565 30.125 0.0925357 0.0114277 2024-03-20T09:14:56.845 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:14:56.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:56.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:14:56.953 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:14:56.980 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:57.002 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:14:57.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:57.006 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:14:57.108 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:14:57.137 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:57.162 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:14:57.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:57.168 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:14:57.270 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:14:57.389 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:57 smithi026 ceph-mon[68309]: pgmap v1302: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 7.2 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 57 MiB/s wr, 908 op/s 2024-03-20T09:14:57.390 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:57.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:57 smithi115 ceph-mon[72476]: pgmap v1302: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 7.2 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 57 MiB/s wr, 908 op/s 2024-03-20T09:14:57.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:57 smithi122 ceph-mon[73495]: pgmap v1302: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 7.2 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 57 MiB/s wr, 908 op/s 2024-03-20T09:14:57.714 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 86 16 119473 119457 86.8023 31.1875 0.0230325 0.0115145 2024-03-20T09:14:58.167 INFO:tasks.thrashosds.thrasher:j is {'epoch': 552, 'fsid': '852fbb68-e696-11ee-95c9-87774f69a715', 'created': '2024-03-20T08:48:35.281088+0000', 'modified': '2024-03-20T09:14:51.167951+0000', 'last_up_change': '2024-03-20T09:10:33.084151+0000', 'last_in_change': '2024-03-20T09:10:19.369629+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-20T08:52:17.442324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '360', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '358', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.3399999141693115, 'score_stable': 2.3399999141693115, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 1.940000057220459, 'raw_score_stable': 1.940000057220459, 'primary_affinity_weighted': 0.9215061664581299, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 4, 'pool_name': 'unique_pool_2', 'create_time': '2024-03-20T09:13:20.690041+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '527', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.009999990463257, 'score_stable': 3.009999990463257, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.9215061664581299, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}], 'osds': [{'osd': 0, 'uuid': 'ad3fd9c0-316f-4fe5-a63b-d480087c1729', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 525, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6802', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6803', 'nonce': 2139805293}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6804', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6805', 'nonce': 2139805293}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6808', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6809', 'nonce': 2139805293}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6806', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6807', 'nonce': 2139805293}]}, 'public_addr': '172.21.15.26:6803/2139805293', 'cluster_addr': '172.21.15.26:6805/2139805293', 'heartbeat_back_addr': '172.21.15.26:6809/2139805293', 'heartbeat_front_addr': '172.21.15.26:6807/2139805293', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'e85b94f5-2954-47a1-9c20-096f2606eb04', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 533, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6810', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6811', 'nonce': 3853355134}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6812', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6813', 'nonce': 3853355134}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6816', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6817', 'nonce': 3853355134}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6814', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6815', 'nonce': 3853355134}]}, 'public_addr': '172.21.15.26:6811/3853355134', 'cluster_addr': '172.21.15.26:6813/3853355134', 'heartbeat_back_addr': '172.21.15.26:6817/3853355134', 'heartbeat_front_addr': '172.21.15.26:6815/3853355134', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'dd2b10f0-741c-47a4-b224-ac6bfe2c14ef', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 534, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6818', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6819', 'nonce': 3993357967}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6820', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6821', 'nonce': 3993357967}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6824', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6825', 'nonce': 3993357967}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6822', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6823', 'nonce': 3993357967}]}, 'public_addr': '172.21.15.26:6819/3993357967', 'cluster_addr': '172.21.15.26:6821/3993357967', 'heartbeat_back_addr': '172.21.15.26:6825/3993357967', 'heartbeat_front_addr': '172.21.15.26:6823/3993357967', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 525, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6826', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6827', 'nonce': 4110225509}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6828', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6829', 'nonce': 4110225509}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6832', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6833', 'nonce': 4110225509}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6830', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6831', 'nonce': 4110225509}]}, 'public_addr': '172.21.15.26:6827/4110225509', 'cluster_addr': '172.21.15.26:6829/4110225509', 'heartbeat_back_addr': '172.21.15.26:6833/4110225509', 'heartbeat_front_addr': '172.21.15.26:6831/4110225509', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'f14e34d5-9829-42a3-9b25-31a0976c5ef7', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 366, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6800', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6801', 'nonce': 477923058}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6802', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6803', 'nonce': 477923058}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6806', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6807', 'nonce': 477923058}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6804', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6805', 'nonce': 477923058}]}, 'public_addr': '172.21.15.115:6801/477923058', 'cluster_addr': '172.21.15.115:6803/477923058', 'heartbeat_back_addr': '172.21.15.115:6807/477923058', 'heartbeat_front_addr': '172.21.15.115:6805/477923058', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'cb431b6a-540d-4318-ae35-d379e1df0185', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 41, 'last_clean_end': 273, 'up_from': 443, 'up_thru': 520, 'down_at': 274, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6808', 'nonce': 650416648}, {'type': 'v1', 'addr': '172.21.15.115:6809', 'nonce': 650416648}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6810', 'nonce': 650416648}, {'type': 'v1', 'addr': '172.21.15.115:6811', 'nonce': 650416648}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6814', 'nonce': 650416648}, {'type': 'v1', 'addr': '172.21.15.115:6815', 'nonce': 650416648}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6812', 'nonce': 650416648}, {'type': 'v1', 'addr': '172.21.15.115:6813', 'nonce': 650416648}]}, 'public_addr': '172.21.15.115:6809/650416648', 'cluster_addr': '172.21.15.115:6811/650416648', 'heartbeat_back_addr': '172.21.15.115:6815/650416648', 'heartbeat_front_addr': '172.21.15.115:6813/650416648', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '753ed830-e764-4ff3-9d9b-87e7fa748b81', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 525, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6816', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6817', 'nonce': 3278301843}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6818', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6819', 'nonce': 3278301843}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6822', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6823', 'nonce': 3278301843}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6820', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6821', 'nonce': 3278301843}]}, 'public_addr': '172.21.15.115:6817/3278301843', 'cluster_addr': '172.21.15.115:6819/3278301843', 'heartbeat_back_addr': '172.21.15.115:6823/3278301843', 'heartbeat_front_addr': '172.21.15.115:6821/3278301843', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '55bcbd37-7a6f-4ab1-b5a9-91a951444d86', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.058074951171875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 517, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6824', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6825', 'nonce': 3152424001}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6826', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6827', 'nonce': 3152424001}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6830', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6831', 'nonce': 3152424001}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6828', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6829', 'nonce': 3152424001}]}, 'public_addr': '172.21.15.115:6825/3152424001', 'cluster_addr': '172.21.15.115:6827/3152424001', 'heartbeat_back_addr': '172.21.15.115:6831/3152424001', 'heartbeat_front_addr': '172.21.15.115:6829/3152424001', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'bc86248e-6bd0-4369-b8a2-e593bc2a610d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 70, 'up_thru': 496, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6800', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6801', 'nonce': 1309426740}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6802', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6803', 'nonce': 1309426740}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6806', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6807', 'nonce': 1309426740}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6804', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6805', 'nonce': 1309426740}]}, 'public_addr': '172.21.15.122:6801/1309426740', 'cluster_addr': '172.21.15.122:6803/1309426740', 'heartbeat_back_addr': '172.21.15.122:6807/1309426740', 'heartbeat_front_addr': '172.21.15.122:6805/1309426740', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '9b20a4aa-032a-4c09-ac75-153c554424e7', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 257, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6808', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6809', 'nonce': 2953210076}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6810', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6811', 'nonce': 2953210076}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6814', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6815', 'nonce': 2953210076}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6812', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6813', 'nonce': 2953210076}]}, 'public_addr': '172.21.15.122:6809/2953210076', 'cluster_addr': '172.21.15.122:6811/2953210076', 'heartbeat_back_addr': '172.21.15.122:6815/2953210076', 'heartbeat_front_addr': '172.21.15.122:6813/2953210076', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'e62c1fb3-5dd1-4d0d-997c-d20633d19361', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 89, 'up_thru': 520, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6816', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6817', 'nonce': 409394340}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6818', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6819', 'nonce': 409394340}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6822', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6823', 'nonce': 409394340}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6820', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6821', 'nonce': 409394340}]}, 'public_addr': '172.21.15.122:6817/409394340', 'cluster_addr': '172.21.15.122:6819/409394340', 'heartbeat_back_addr': '172.21.15.122:6823/409394340', 'heartbeat_front_addr': '172.21.15.122:6821/409394340', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'b83994d5-e5ef-48b0-9758-c26843cec3b6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 96, 'up_thru': 522, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6824', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6825', 'nonce': 2704069354}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6826', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6827', 'nonce': 2704069354}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6830', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6831', 'nonce': 2704069354}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6828', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6829', 'nonce': 2704069354}]}, 'public_addr': '172.21.15.122:6825/2704069354', 'cluster_addr': '172.21.15.122:6827/2704069354', 'heartbeat_back_addr': '172.21.15.122:6831/2704069354', 'heartbeat_front_addr': '172.21.15.122:6829/2704069354', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:14:01.793030+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:13:37.136739+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:14:29.963247+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:14:06.590345+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-20T09:14:33.513834+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:14:27.640318+0000', 'dead_epoch': 273}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:14:32.283694+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:14:48.028594+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:14:50.225887+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 34736, 'last_purged_snaps_scrub': '2024-03-20T09:13:53.078437+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:13:59.357928+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:13:30.485826+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.26:0/4291404191': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/642012566': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:6801/1181459999': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/3984513952': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6800/1654945498': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6800/1181459999': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/137770981': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:0/1451237762': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/3938386631': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6801/1654945498': '2024-03-21T08:49:04.596754+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-20T09:14:58.168 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:14:58.593 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:58 smithi026 ceph-mon[68309]: 4.6 scrub starts 2024-03-20T09:14:58.606 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:14:58.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:58.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:14:58.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:58 smithi115 ceph-mon[72476]: 4.6 scrub starts 2024-03-20T09:14:58.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:58 smithi122 ceph-mon[73495]: 4.6 scrub starts 2024-03-20T09:14:58.714 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 87 16 119855 119839 86.079 23.875 0.0782493 0.0116118 2024-03-20T09:14:58.721 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:14:58.875 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:58.931 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:14:58.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:58.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:14:59.040 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:14:59.069 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:59.090 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:14:59.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:59.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:14:59.194 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:14:59.220 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:59.239 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:14:59.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:59.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:14:59.343 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:14:59.372 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:59.396 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:14:59.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:59.399 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:14:59.500 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:14:59.528 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:14:59.548 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:14:59.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:14:59.554 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:14:59.655 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:14:59.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:59 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:14:59.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:59 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:14:59.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:59 smithi115 ceph-mon[72476]: 1.11 scrub starts 2024-03-20T09:14:59.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:59 smithi115 ceph-mon[72476]: pgmap v1303: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 7.2 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 51 MiB/s wr, 809 op/s 2024-03-20T09:14:59.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:59 smithi115 ceph-mon[72476]: 1.11 scrub ok 2024-03-20T09:14:59.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:59 smithi115 ceph-mon[72476]: osdmap e553: 12 total, 12 up, 10 in 2024-03-20T09:14:59.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:59 smithi115 ceph-mon[72476]: 1.5 scrub starts 2024-03-20T09:14:59.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:14:59 smithi115 ceph-mon[72476]: 1.5 scrub ok 2024-03-20T09:14:59.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:59 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:14:59.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:59 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:14:59.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:59 smithi122 ceph-mon[73495]: 1.11 scrub starts 2024-03-20T09:14:59.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:59 smithi122 ceph-mon[73495]: pgmap v1303: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 7.2 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 51 MiB/s wr, 809 op/s 2024-03-20T09:14:59.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:59 smithi122 ceph-mon[73495]: 1.11 scrub ok 2024-03-20T09:14:59.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:59 smithi122 ceph-mon[73495]: osdmap e553: 12 total, 12 up, 10 in 2024-03-20T09:14:59.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:59 smithi122 ceph-mon[73495]: 1.5 scrub starts 2024-03-20T09:14:59.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:14:59 smithi122 ceph-mon[73495]: 1.5 scrub ok 2024-03-20T09:14:59.714 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 88 16 120215 120199 85.3566 22.5 0.00433556 0.0117044 2024-03-20T09:14:59.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:59 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:14:59.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:59 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:14:59.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:59 smithi026 ceph-mon[68309]: 1.11 scrub starts 2024-03-20T09:14:59.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:59 smithi026 ceph-mon[68309]: pgmap v1303: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 7.2 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 51 MiB/s wr, 809 op/s 2024-03-20T09:14:59.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:59 smithi026 ceph-mon[68309]: 1.11 scrub ok 2024-03-20T09:14:59.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:59 smithi026 ceph-mon[68309]: osdmap e553: 12 total, 12 up, 10 in 2024-03-20T09:14:59.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:59 smithi026 ceph-mon[68309]: 1.5 scrub starts 2024-03-20T09:14:59.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:14:59 smithi026 ceph-mon[68309]: 1.5 scrub ok 2024-03-20T09:14:59.760 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:00.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:00 smithi115 ceph-mon[72476]: 4.9 deep-scrub starts 2024-03-20T09:15:00.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:00 smithi122 ceph-mon[73495]: 4.9 deep-scrub starts 2024-03-20T09:15:00.714 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 89 16 120565 120549 84.6433 21.875 0.00945103 0.011805 2024-03-20T09:15:00.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:00 smithi026 ceph-mon[68309]: 4.9 deep-scrub starts 2024-03-20T09:15:00.942 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:15:00.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:00.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:15:01.052 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:15:01.083 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:01.108 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:15:01.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:01.111 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:15:01.212 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:15:01.239 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:01.258 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:15:01.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:01.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:15:01.364 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:15:01.393 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:01.413 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:15:01.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:01.416 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:15:01.517 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:15:01.655 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:01.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:01 smithi115 ceph-mon[72476]: pgmap v1305: 47 pgs: 4 active+clean+scrubbing+deep, 43 active+clean; 7.3 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 36 MiB/s wr, 571 op/s 2024-03-20T09:15:01.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:01 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1860332053' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:15:01.657 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:01 smithi026 ceph-mon[68309]: pgmap v1305: 47 pgs: 4 active+clean+scrubbing+deep, 43 active+clean; 7.3 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 36 MiB/s wr, 571 op/s 2024-03-20T09:15:01.657 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:01 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1860332053' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:15:01.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:01 smithi122 ceph-mon[73495]: pgmap v1305: 47 pgs: 4 active+clean+scrubbing+deep, 43 active+clean; 7.3 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 36 MiB/s wr, 571 op/s 2024-03-20T09:15:01.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:01 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1860332053' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:15:01.714 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 90 13 120916 120903 83.9487 22.125 0.0608821 0.011904 2024-03-20T09:15:01.844 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:15:01.845 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:15:01.845 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:15:01.845 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:15:01.845 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:15:01.845 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:15:01.845 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:15:01.861 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:15:01.931 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:15:01.933 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:15:01.987 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:15:02.041 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:15:02.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:02.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:15:02.074 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:15:02.078 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:15:02.090 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:15:02.164 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:15:02.164 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:15:02.164 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:15:02.164 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:15:02.164 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:15:02.164 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:15:02.164 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:15:02.164 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:15:02.164 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:15:02.164 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:15:02.164 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:15:02.164 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:15:02.167 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:15:02.195 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:02.214 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:15:02.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:02.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:15:02.318 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:15:02.358 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:02.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:02 smithi115 ceph-mon[72476]: 4.6 scrub ok 2024-03-20T09:15:02.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:02 smithi122 ceph-mon[73495]: 4.6 scrub ok 2024-03-20T09:15:02.715 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Total time run: 90.1093 2024-03-20T09:15:02.715 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Total writes made: 120916 2024-03-20T09:15:02.715 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Write size: 65536 2024-03-20T09:15:02.715 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Object size: 65536 2024-03-20T09:15:02.715 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Bandwidth (MB/sec): 83.8676 2024-03-20T09:15:02.715 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Stddev Bandwidth: 40.8552 2024-03-20T09:15:02.715 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Max bandwidth (MB/sec): 251.875 2024-03-20T09:15:02.715 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Min bandwidth (MB/sec): 0 2024-03-20T09:15:02.715 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Average IOPS: 1341 2024-03-20T09:15:02.715 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Stddev IOPS: 653.683 2024-03-20T09:15:02.715 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Max IOPS: 4030 2024-03-20T09:15:02.715 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Min IOPS: 0 2024-03-20T09:15:02.716 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Average Latency(s): 0.011912 2024-03-20T09:15:02.716 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Stddev Latency(s): 0.0312397 2024-03-20T09:15:02.716 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Max latency(s): 2.19794 2024-03-20T09:15:02.716 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Min latency(s): 0.00107883 2024-03-20T09:15:02.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:02 smithi026 ceph-mon[68309]: 4.6 scrub ok 2024-03-20T09:15:02.754 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Cleaning up (deleting benchmark objects) 2024-03-20T09:15:03.148 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:15:03.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:03.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:15:03.263 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:15:03.291 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:03.312 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:15:03.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:03.316 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:15:03.417 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:15:03.444 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:03.464 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:15:03.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:03.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:15:03.568 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:15:03.596 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:03.617 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:15:03.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:03.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:15:03.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:03 smithi122 ceph-mon[73495]: pgmap v1306: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 7.3 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 36 MiB/s wr, 574 op/s 2024-03-20T09:15:03.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:03 smithi122 ceph-mon[73495]: 1.2 scrub starts 2024-03-20T09:15:03.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:03 smithi122 ceph-mon[73495]: 1.2 scrub ok 2024-03-20T09:15:03.721 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:15:03.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:03 smithi026 ceph-mon[68309]: pgmap v1306: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 7.3 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 36 MiB/s wr, 574 op/s 2024-03-20T09:15:03.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:03 smithi026 ceph-mon[68309]: 1.2 scrub starts 2024-03-20T09:15:03.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:03 smithi026 ceph-mon[68309]: 1.2 scrub ok 2024-03-20T09:15:03.744 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:03 smithi115 ceph-mon[72476]: pgmap v1306: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 7.3 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 36 MiB/s wr, 574 op/s 2024-03-20T09:15:03.744 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:03 smithi115 ceph-mon[72476]: 1.2 scrub starts 2024-03-20T09:15:03.744 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:03 smithi115 ceph-mon[72476]: 1.2 scrub ok 2024-03-20T09:15:03.746 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:03.765 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:15:03.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:03.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:15:03.870 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:15:03.897 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:03.916 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:15:03.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:03.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:15:04.020 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:15:04.046 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:04.066 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:15:04.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:04.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:15:04.109 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:15:04.109 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:15:04.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:04.109 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.25690.0:121467 4.0 e553/522 4:0036a350:::benchmark_data_smithi122_145698_object548:head v 553'7515, mlcod=553'7513)", 2024-03-20T09:15:04.110 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:15:04.108670+0000", 2024-03-20T09:15:04.110 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.000109409, 2024-03-20T09:15:04.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000147167, 2024-03-20T09:15:04.110 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:15:04.110 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:15:04.110 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:15:04.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:04.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:15:04.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:04.108670+0000", 2024-03-20T09:15:04.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:15:04.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:04.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:04.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:15:04.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:04.108670+0000", 2024-03-20T09:15:04.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:15:04.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:04.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:04.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:15:04.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:04.108671+0000", 2024-03-20T09:15:04.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.079e-06 2024-03-20T09:15:04.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:04.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:04.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:15:04.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:04.108678+0000", 2024-03-20T09:15:04.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0380000000000002e-06 2024-03-20T09:15:04.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:04.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:04.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:15:04.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:04.108693+0000", 2024-03-20T09:15:04.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5169e-05 2024-03-20T09:15:04.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:04.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:04.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:15:04.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:04.108702+0000", 2024-03-20T09:15:04.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4300000000000006e-06 2024-03-20T09:15:04.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:04.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:04.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:15:04.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:04.108770+0000", 2024-03-20T09:15:04.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.779e-05 2024-03-20T09:15:04.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:04.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:04.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:15:04.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:04.108789+0000", 2024-03-20T09:15:04.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9919e-05 2024-03-20T09:15:04.113 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:04.113 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:04.113 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:04.113 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:04.113 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:15:04.113 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 1 2024-03-20T09:15:04.113 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:15:04.170 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:15:04.325 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:04.431 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:15:04.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:04.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:15:04.539 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:15:04.672 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:05.623 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:15:05.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:05.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:15:05.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:05 smithi026 ceph-mon[68309]: pgmap v1307: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 7.4 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 33 MiB/s wr, 535 op/s 2024-03-20T09:15:05.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:05 smithi026 ceph-mon[68309]: 4.8 scrub starts 2024-03-20T09:15:05.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:05 smithi026 ceph-mon[68309]: 1.13 scrub starts 2024-03-20T09:15:05.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:05 smithi026 ceph-mon[68309]: 1.13 scrub ok 2024-03-20T09:15:05.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:05 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:15:05.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:05 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:15:05.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:05 smithi026 ceph-mon[68309]: 4.8 scrub ok 2024-03-20T09:15:05.737 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:15:05.765 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:05 smithi122 ceph-mon[73495]: pgmap v1307: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 7.4 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 33 MiB/s wr, 535 op/s 2024-03-20T09:15:05.765 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:05 smithi122 ceph-mon[73495]: 4.8 scrub starts 2024-03-20T09:15:05.765 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:05 smithi122 ceph-mon[73495]: 1.13 scrub starts 2024-03-20T09:15:05.766 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:05 smithi122 ceph-mon[73495]: 1.13 scrub ok 2024-03-20T09:15:05.766 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:05 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:15:05.766 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:05 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:15:05.766 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:05 smithi122 ceph-mon[73495]: 4.8 scrub ok 2024-03-20T09:15:05.767 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:05.788 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:15:05.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:05.793 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:15:05.894 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:15:05.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:05 smithi115 ceph-mon[72476]: pgmap v1307: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 7.4 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 33 MiB/s wr, 535 op/s 2024-03-20T09:15:05.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:05 smithi115 ceph-mon[72476]: 4.8 scrub starts 2024-03-20T09:15:05.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:05 smithi115 ceph-mon[72476]: 1.13 scrub starts 2024-03-20T09:15:05.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:05 smithi115 ceph-mon[72476]: 1.13 scrub ok 2024-03-20T09:15:05.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:05 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:15:05.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:05 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:15:05.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:05 smithi115 ceph-mon[72476]: 4.8 scrub ok 2024-03-20T09:15:06.015 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:06.846 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:06 smithi026 ceph-mon[68309]: osdmap e554: 12 total, 12 up, 10 in 2024-03-20T09:15:06.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:06 smithi115 ceph-mon[72476]: osdmap e554: 12 total, 12 up, 10 in 2024-03-20T09:15:06.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:06 smithi122 ceph-mon[73495]: osdmap e554: 12 total, 12 up, 10 in 2024-03-20T09:15:07.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:07 smithi115 ceph-mon[72476]: pgmap v1309: 47 pgs: 2 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 41 active+clean; 7.3 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 20 MiB/s wr, 375 op/s 2024-03-20T09:15:07.937 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:07 smithi026 ceph-mon[68309]: pgmap v1309: 47 pgs: 2 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 41 active+clean; 7.3 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 20 MiB/s wr, 375 op/s 2024-03-20T09:15:07.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:07 smithi122 ceph-mon[73495]: pgmap v1309: 47 pgs: 2 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 41 active+clean; 7.3 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 20 MiB/s wr, 375 op/s 2024-03-20T09:15:07.976 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-20T09:15:08.293 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:15:08.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:08.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:15:08.397 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:15:08.432 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:08.454 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:15:08.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:08.458 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:15:08.559 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:15:08.587 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:08.619 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:15:08.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:08.622 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:15:08.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:08 smithi122 ceph-mon[73495]: 1.d scrub starts 2024-03-20T09:15:08.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:08 smithi122 ceph-mon[73495]: 1.d scrub ok 2024-03-20T09:15:08.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:08 smithi122 ceph-mon[73495]: 4.3 scrub starts 2024-03-20T09:15:08.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:08 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:15:08.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:08 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:15:08.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:08 smithi122 ceph-mon[73495]: osdmap e555: 12 total, 12 up, 10 in 2024-03-20T09:15:08.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:08 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:15:08.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:08 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:15:08.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:08 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:15:08.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:08 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:15:08.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:08 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:15:08.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:08 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:15:08.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:08 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:15:08.723 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:15:08.724 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:08 smithi026 ceph-mon[68309]: 1.d scrub starts 2024-03-20T09:15:08.724 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:08 smithi026 ceph-mon[68309]: 1.d scrub ok 2024-03-20T09:15:08.724 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:08 smithi026 ceph-mon[68309]: 4.3 scrub starts 2024-03-20T09:15:08.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:08 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:15:08.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:08 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:15:08.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:08 smithi026 ceph-mon[68309]: osdmap e555: 12 total, 12 up, 10 in 2024-03-20T09:15:08.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:08 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:15:08.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:08 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:15:08.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:08 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:15:08.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:08 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:15:08.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:08 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:15:08.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:08 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:15:08.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:08 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:15:08.758 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:08.784 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:15:08.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:08.788 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:15:08.889 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:15:08.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:08 smithi115 ceph-mon[72476]: 1.d scrub starts 2024-03-20T09:15:08.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:08 smithi115 ceph-mon[72476]: 1.d scrub ok 2024-03-20T09:15:08.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:08 smithi115 ceph-mon[72476]: 4.3 scrub starts 2024-03-20T09:15:08.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:08 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:15:08.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:08 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:15:08.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:08 smithi115 ceph-mon[72476]: osdmap e555: 12 total, 12 up, 10 in 2024-03-20T09:15:08.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:08 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:15:08.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:08 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:15:08.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:08 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:15:08.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:08 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:15:08.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:08 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:15:08.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:08 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:15:08.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:08 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:15:08.916 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:08.942 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:15:08.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:08.950 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:15:09.051 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:15:09.078 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:09.099 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:15:09.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:09.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:15:09.204 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:15:09.236 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:09.259 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:15:09.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:09.265 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:15:09.366 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:15:09.396 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:09.424 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:15:09.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:09.427 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:15:09.528 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:15:09.554 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:09.574 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:15:09.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:09.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:15:09.677 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:15:09.703 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:09.724 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:15:09.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:09 smithi026 ceph-mon[68309]: pgmap v1311: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 7.3 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 127 B/s rd, 15 MiB/s wr, 378 op/s 2024-03-20T09:15:09.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:09 smithi026 ceph-mon[68309]: 1.1e scrub starts 2024-03-20T09:15:09.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:09 smithi026 ceph-mon[68309]: 1.1e scrub ok 2024-03-20T09:15:09.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:09 smithi026 ceph-mon[68309]: 4.3 scrub ok 2024-03-20T09:15:09.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:09.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:15:09.828 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:15:09.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:09 smithi115 ceph-mon[72476]: pgmap v1311: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 7.3 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 127 B/s rd, 15 MiB/s wr, 378 op/s 2024-03-20T09:15:09.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:09 smithi115 ceph-mon[72476]: 1.1e scrub starts 2024-03-20T09:15:09.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:09 smithi115 ceph-mon[72476]: 1.1e scrub ok 2024-03-20T09:15:09.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:09 smithi115 ceph-mon[72476]: 4.3 scrub ok 2024-03-20T09:15:09.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:09 smithi122 ceph-mon[73495]: pgmap v1311: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 7.3 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 127 B/s rd, 15 MiB/s wr, 378 op/s 2024-03-20T09:15:09.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:09 smithi122 ceph-mon[73495]: 1.1e scrub starts 2024-03-20T09:15:09.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:09 smithi122 ceph-mon[73495]: 1.1e scrub ok 2024-03-20T09:15:09.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:09 smithi122 ceph-mon[73495]: 4.3 scrub ok 2024-03-20T09:15:09.993 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:10.414 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:15:10.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:10.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:15:10.491 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:15:10.526 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:15:10.554 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:10.573 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:15:10.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:10.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:15:10.678 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:15:10.705 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:15:10.728 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:10 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:15:10.728 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:10 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:15:10.728 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:10 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1263297028' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:15:10.869 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:10.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:10 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:15:10.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:10 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:15:10.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:10 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1263297028' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:15:10.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:10 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:15:10.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:10 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:15:10.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:10 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1263297028' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:15:11.558 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:15:11.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:11.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:15:11.663 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:15:11.842 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:11 smithi026 ceph-mon[68309]: pgmap v1312: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 7.3 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 127 B/s rd, 4.3 MiB/s wr, 226 op/s 2024-03-20T09:15:11.842 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:11 smithi026 ceph-mon[68309]: 1.18 scrub starts 2024-03-20T09:15:11.842 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:11 smithi026 ceph-mon[68309]: 1.18 scrub ok 2024-03-20T09:15:11.842 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:11 smithi026 ceph-mon[68309]: 4.1 scrub starts 2024-03-20T09:15:11.842 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:11 smithi026 ceph-mon[68309]: 1.1b scrub starts 2024-03-20T09:15:11.842 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:11 smithi026 ceph-mon[68309]: 1.1b scrub ok 2024-03-20T09:15:11.842 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:11 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:15:11.842 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:11 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1263297028' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:15:11.842 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:11 smithi026 ceph-mon[68309]: osdmap e556: 12 total, 12 up, 10 in 2024-03-20T09:15:11.848 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:11.899 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:15:11.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:11.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:15:11.908 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:11 smithi115 ceph-mon[72476]: pgmap v1312: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 7.3 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 127 B/s rd, 4.3 MiB/s wr, 226 op/s 2024-03-20T09:15:11.908 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:11 smithi115 ceph-mon[72476]: 1.18 scrub starts 2024-03-20T09:15:11.908 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:11 smithi115 ceph-mon[72476]: 1.18 scrub ok 2024-03-20T09:15:11.908 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:11 smithi115 ceph-mon[72476]: 4.1 scrub starts 2024-03-20T09:15:11.908 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:11 smithi115 ceph-mon[72476]: 1.1b scrub starts 2024-03-20T09:15:11.908 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:11 smithi115 ceph-mon[72476]: 1.1b scrub ok 2024-03-20T09:15:11.908 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:11 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:15:11.909 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:11 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1263297028' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:15:11.909 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:11 smithi115 ceph-mon[72476]: osdmap e556: 12 total, 12 up, 10 in 2024-03-20T09:15:11.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:11 smithi122 ceph-mon[73495]: pgmap v1312: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 7.3 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 127 B/s rd, 4.3 MiB/s wr, 226 op/s 2024-03-20T09:15:11.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:11 smithi122 ceph-mon[73495]: 1.18 scrub starts 2024-03-20T09:15:11.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:11 smithi122 ceph-mon[73495]: 1.18 scrub ok 2024-03-20T09:15:11.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:11 smithi122 ceph-mon[73495]: 4.1 scrub starts 2024-03-20T09:15:11.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:11 smithi122 ceph-mon[73495]: 1.1b scrub starts 2024-03-20T09:15:11.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:11 smithi122 ceph-mon[73495]: 1.1b scrub ok 2024-03-20T09:15:11.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:11 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:15:11.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:11 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1263297028' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:15:11.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:11 smithi122 ceph-mon[73495]: osdmap e556: 12 total, 12 up, 10 in 2024-03-20T09:15:12.005 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:15:12.034 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:12.058 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:15:12.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:12.061 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:15:12.162 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:15:12.188 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:12.207 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:15:12.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:12.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:15:12.311 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:15:12.338 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:12.358 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:15:12.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:12.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:15:12.386 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:15:12.448 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:15:12.461 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:15:12.488 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:12.509 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:15:12.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:12.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:15:12.613 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:15:12.641 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:12.663 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:15:12.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:12.668 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:15:12.770 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:15:12.807 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:12.827 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:15:12.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:12.830 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:15:12.931 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:15:12.960 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:12.981 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:15:12.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:12.985 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:15:13.086 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:15:13.122 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:15:13.212 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:13.370 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:15:13.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:13.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:15:13.484 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:15:13.513 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:13.532 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:15:13.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:13.537 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:15:13.638 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:15:13.650 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:13 smithi026 ceph-mon[68309]: pgmap v1314: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 7.2 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 159 B/s rd, 1.2 MiB/s wr, 394 op/s 2024-03-20T09:15:13.650 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:13 smithi026 ceph-mon[68309]: from='client.25762 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:15:13.666 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:13.685 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:15:13.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:13.688 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:15:13.789 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:15:13.816 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:13.838 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:15:13.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:13.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:15:13.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:13 smithi115 ceph-mon[72476]: pgmap v1314: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 7.2 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 159 B/s rd, 1.2 MiB/s wr, 394 op/s 2024-03-20T09:15:13.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:13 smithi115 ceph-mon[72476]: from='client.25762 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:15:13.942 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:15:13.949 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:13 smithi122 ceph-mon[73495]: pgmap v1314: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 7.2 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 159 B/s rd, 1.2 MiB/s wr, 394 op/s 2024-03-20T09:15:13.949 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:13 smithi122 ceph-mon[73495]: from='client.25762 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:15:13.978 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:13.998 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:15:14.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:14.003 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:15:14.042 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:15:14.074 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":1314,"stamp":"2024-03-20T09:15:11.890935+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7743734328,"num_objects":118154,"num_object_clones":0,"num_object_copies":354462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":118154,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123682,"num_write_kb":7738625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":923,"num_bytes_recovered":60751872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134990,"ondisk_log_size":134990,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":26004020,"kb_used_data":22731028,"kb_used_omap":195,"kb_used_meta":3271484,"kb_avail":911447500,"statfs":{"total":959950356480,"available":933322240000,"internally_reserved":0,"allocated":23276572672,"data_stored":23266941163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":200482,"internal_metadata":3349999838},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,4,2,1,12,14,14,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-149487592,"num_objects":-2280,"num_object_clones":0,"num_object_copies":-6840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2280,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2534,"num_write_kb":8065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.425798"},"pg_stats":[{"pgid":"1.1e","version":"20'137","reported_seq":493,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-20T09:15:08.506823+0000","last_change":"2024-03-20T09:15:08.506869+0000","last_active":"2024-03-20T09:15:08.506823+0000","last_peered":"2024-03-20T09:15:08.506823+0000","last_clean":"2024-03-20T09:15:08.506823+0000","last_became_active":"2024-03-20T09:13:20.071655+0000","last_became_peered":"2024-03-20T09:13:20.071655+0000","last_unstale":"2024-03-20T09:15:08.506823+0000","last_undegraded":"2024-03-20T09:15:08.506823+0000","last_fullsized":"2024-03-20T09:15:08.506823+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:08.506732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:15:08.506732+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:22.387718+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'137","reported_seq":663,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-03-20T09:15:10.624634+0000","last_change":"2024-03-20T09:14:44.782411+0000","last_active":"2024-03-20T09:15:10.624634+0000","last_peered":"2024-03-20T09:15:10.624634+0000","last_clean":"2024-03-20T09:15:10.624634+0000","last_became_active":"2024-03-20T09:07:42.013245+0000","last_became_peered":"2024-03-20T09:07:42.013245+0000","last_unstale":"2024-03-20T09:15:10.624634+0000","last_undegraded":"2024-03-20T09:15:10.624634+0000","last_fullsized":"2024-03-20T09:15:10.624634+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:14:44.782290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:14:44.782290+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:52.058206+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"20'137","reported_seq":1105,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-03-20T09:15:10.654299+0000","last_change":"2024-03-20T09:13:56.707873+0000","last_active":"2024-03-20T09:15:10.654299+0000","last_peered":"2024-03-20T09:15:10.654299+0000","last_clean":"2024-03-20T09:15:10.654299+0000","last_became_active":"2024-03-20T09:07:41.961622+0000","last_became_peered":"2024-03-20T09:07:41.961622+0000","last_unstale":"2024-03-20T09:15:10.654299+0000","last_undegraded":"2024-03-20T09:15:10.654299+0000","last_fullsized":"2024-03-20T09:15:10.654299+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:56.707767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:56.707767+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:17.835273+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.19","version":"20'137","reported_seq":1174,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-03-20T09:15:10.602585+0000","last_change":"2024-03-20T09:14:54.527786+0000","last_active":"2024-03-20T09:15:10.602585+0000","last_peered":"2024-03-20T09:15:10.602585+0000","last_clean":"2024-03-20T09:15:10.602585+0000","last_became_active":"2024-03-20T09:07:41.962680+0000","last_became_peered":"2024-03-20T09:07:41.962680+0000","last_unstale":"2024-03-20T09:15:10.602585+0000","last_undegraded":"2024-03-20T09:15:10.602585+0000","last_fullsized":"2024-03-20T09:15:10.602585+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:14:54.527668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:14:54.527668+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:18.597530+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"20'137","reported_seq":623,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-20T09:15:05.458579+0000","last_change":"2024-03-20T09:13:39.292648+0000","last_active":"2024-03-20T09:15:05.458579+0000","last_peered":"2024-03-20T09:15:05.458579+0000","last_clean":"2024-03-20T09:15:05.458579+0000","last_became_active":"2024-03-20T09:10:35.277481+0000","last_became_peered":"2024-03-20T09:10:35.277481+0000","last_unstale":"2024-03-20T09:15:05.458579+0000","last_undegraded":"2024-03-20T09:15:05.458579+0000","last_fullsized":"2024-03-20T09:15:05.458579+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:39.292556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:39.292556+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:07.108785+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1b","version":"20'137","reported_seq":942,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-03-20T09:15:10.652003+0000","last_change":"2024-03-20T09:15:10.569807+0000","last_active":"2024-03-20T09:15:10.652003+0000","last_peered":"2024-03-20T09:15:10.652003+0000","last_clean":"2024-03-20T09:15:10.652003+0000","last_became_active":"2024-03-20T09:07:41.962733+0000","last_became_peered":"2024-03-20T09:07:41.962733+0000","last_unstale":"2024-03-20T09:15:10.652003+0000","last_undegraded":"2024-03-20T09:15:10.652003+0000","last_fullsized":"2024-03-20T09:15:10.652003+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:10.569712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:15:10.569712+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:32.526084+0000","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"556'9757","reported_seq":9924,"reported_epoch":556,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-20T09:15:11.341800+0000","last_change":"2024-03-20T09:14:53.662387+0000","last_active":"2024-03-20T09:15:11.341800+0000","last_peered":"2024-03-20T09:15:11.341800+0000","last_clean":"2024-03-20T09:15:11.341800+0000","last_became_active":"2024-03-20T09:13:22.063020+0000","last_became_peered":"2024-03-20T09:13:22.063020+0000","last_unstale":"2024-03-20T09:15:11.341800+0000","last_undegraded":"2024-03-20T09:15:11.341800+0000","last_fullsized":"2024-03-20T09:15:11.341800+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":2424,"log_size":9757,"log_dups_size":0,"ondisk_log_size":9757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 17s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":486211608,"num_objects":7420,"num_object_clones":0,"num_object_copies":22260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7420,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7794,"num_write_kb":486785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"20'137","reported_seq":631,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-20T09:15:05.458590+0000","last_change":"2024-03-20T09:14:52.016392+0000","last_active":"2024-03-20T09:15:05.458590+0000","last_peered":"2024-03-20T09:15:05.458590+0000","last_clean":"2024-03-20T09:15:05.458590+0000","last_became_active":"2024-03-20T09:10:35.215772+0000","last_became_peered":"2024-03-20T09:10:35.215772+0000","last_unstale":"2024-03-20T09:15:05.458590+0000","last_undegraded":"2024-03-20T09:15:05.458590+0000","last_fullsized":"2024-03-20T09:15:05.458590+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:14:52.016290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:14:52.016290+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:19.535050+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"555'7682","reported_seq":7839,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-20T09:15:09.989873+0000","last_change":"2024-03-20T09:14:44.134032+0000","last_active":"2024-03-20T09:15:09.989873+0000","last_peered":"2024-03-20T09:15:09.989873+0000","last_clean":"2024-03-20T09:15:09.989873+0000","last_became_active":"2024-03-20T09:13:22.583317+0000","last_became_peered":"2024-03-20T09:13:22.583317+0000","last_unstale":"2024-03-20T09:15:09.989873+0000","last_undegraded":"2024-03-20T09:15:09.989873+0000","last_fullsized":"2024-03-20T09:15:09.989873+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"550'6791","last_scrub_stamp":"2024-03-20T09:14:44.133820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:14:44.133820+0000","objects_scrubbed":6742,"log_size":7682,"log_dups_size":0,"ondisk_log_size":7682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:57.587887+0000","scrub_duration":2011,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":481689600,"num_objects":7350,"num_object_clones":0,"num_object_copies":22050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7682,"num_write_kb":481024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,8],"acting":[3,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"20'137","reported_seq":1185,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-20T09:15:07.497945+0000","last_change":"2024-03-20T09:14:45.086348+0000","last_active":"2024-03-20T09:15:07.497945+0000","last_peered":"2024-03-20T09:15:07.497945+0000","last_clean":"2024-03-20T09:15:07.497945+0000","last_became_active":"2024-03-20T09:10:34.099414+0000","last_became_peered":"2024-03-20T09:10:34.099414+0000","last_unstale":"2024-03-20T09:15:07.497945+0000","last_undegraded":"2024-03-20T09:15:07.497945+0000","last_fullsized":"2024-03-20T09:15:07.497945+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:14:45.086222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:14:45.086222+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:59.410055+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"555'9623","reported_seq":11645,"reported_epoch":555,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-20T09:15:08.610870+0000","last_change":"2024-03-20T09:14:53.987108+0000","last_active":"2024-03-20T09:15:08.610870+0000","last_peered":"2024-03-20T09:15:08.610870+0000","last_clean":"2024-03-20T09:15:08.610870+0000","last_became_active":"2024-03-20T09:13:46.198893+0000","last_became_peered":"2024-03-20T09:13:46.198893+0000","last_unstale":"2024-03-20T09:15:08.610870+0000","last_undegraded":"2024-03-20T09:15:08.610870+0000","last_fullsized":"2024-03-20T09:15:08.610870+0000","mapping_epoch":534,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":2274,"log_size":9623,"log_dups_size":0,"ondisk_log_size":9623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 14s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":495714304,"num_objects":7564,"num_object_clones":0,"num_object_copies":22692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7850,"num_write_kb":493248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":922,"num_bytes_recovered":60424192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,11],"acting":[2,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.a","version":"20'137","reported_seq":992,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-03-20T09:15:10.605773+0000","last_change":"2024-03-20T09:13:57.530523+0000","last_active":"2024-03-20T09:15:10.605773+0000","last_peered":"2024-03-20T09:15:10.605773+0000","last_clean":"2024-03-20T09:15:10.605773+0000","last_became_active":"2024-03-20T09:07:52.618107+0000","last_became_peered":"2024-03-20T09:07:52.618107+0000","last_unstale":"2024-03-20T09:15:10.605773+0000","last_undegraded":"2024-03-20T09:15:10.605773+0000","last_fullsized":"2024-03-20T09:15:10.605773+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:57.530396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:57.530396+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"555'7665","reported_seq":7744,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-20T09:15:09.895521+0000","last_change":"2024-03-20T09:13:22.064363+0000","last_active":"2024-03-20T09:15:09.895521+0000","last_peered":"2024-03-20T09:15:09.895521+0000","last_clean":"2024-03-20T09:15:09.895521+0000","last_became_active":"2024-03-20T09:13:22.063975+0000","last_became_peered":"2024-03-20T09:13:22.063975+0000","last_unstale":"2024-03-20T09:15:09.895521+0000","last_undegraded":"2024-03-20T09:15:09.895521+0000","last_fullsized":"2024-03-20T09:15:09.895521+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":0,"log_size":7665,"log_dups_size":0,"ondisk_log_size":7665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":481361920,"num_objects":7345,"num_object_clones":0,"num_object_copies":22035,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7665,"num_write_kb":480320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"20'137","reported_seq":1154,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-20T09:15:07.500099+0000","last_change":"2024-03-20T09:14:50.495971+0000","last_active":"2024-03-20T09:15:07.500099+0000","last_peered":"2024-03-20T09:15:07.500099+0000","last_clean":"2024-03-20T09:15:07.500099+0000","last_became_active":"2024-03-20T09:12:14.306384+0000","last_became_peered":"2024-03-20T09:12:14.306384+0000","last_unstale":"2024-03-20T09:15:07.500099+0000","last_undegraded":"2024-03-20T09:15:07.500099+0000","last_fullsized":"2024-03-20T09:15:07.500099+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:14:50.495883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:14:50.495883+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:56.403354+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"555'7646","reported_seq":7801,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-20T09:15:10.134651+0000","last_change":"2024-03-20T09:14:52.155033+0000","last_active":"2024-03-20T09:15:10.134651+0000","last_peered":"2024-03-20T09:15:10.134651+0000","last_clean":"2024-03-20T09:15:10.134651+0000","last_became_active":"2024-03-20T09:13:24.064091+0000","last_became_peered":"2024-03-20T09:13:24.064091+0000","last_unstale":"2024-03-20T09:15:10.134651+0000","last_undegraded":"2024-03-20T09:15:10.134651+0000","last_fullsized":"2024-03-20T09:15:10.134651+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"552'7170","last_scrub_stamp":"2024-03-20T09:14:52.154865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:14:52.154865+0000","objects_scrubbed":7129,"log_size":7646,"log_dups_size":0,"ondisk_log_size":7646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:12.538193+0000","scrub_duration":1965,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":478674944,"num_objects":7304,"num_object_clones":0,"num_object_copies":21912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7646,"num_write_kb":478400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"20'137","reported_seq":1051,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-03-20T09:15:10.602628+0000","last_change":"2024-03-20T09:14:58.582184+0000","last_active":"2024-03-20T09:15:10.602628+0000","last_peered":"2024-03-20T09:15:10.602628+0000","last_clean":"2024-03-20T09:15:10.602628+0000","last_became_active":"2024-03-20T09:07:37.003722+0000","last_became_peered":"2024-03-20T09:07:37.003722+0000","last_unstale":"2024-03-20T09:15:10.602628+0000","last_undegraded":"2024-03-20T09:15:10.602628+0000","last_fullsized":"2024-03-20T09:15:10.602628+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:14:58.582047+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:07:06.480160+0000","last_clean_scrub_stamp":"2024-03-20T09:14:58.582047+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:03.684147+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.3","version":"555'7923","reported_seq":8087,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-20T09:15:09.734019+0000","last_change":"2024-03-20T09:15:09.333207+0000","last_active":"2024-03-20T09:15:09.734019+0000","last_peered":"2024-03-20T09:15:09.734019+0000","last_clean":"2024-03-20T09:15:09.734019+0000","last_became_active":"2024-03-20T09:13:22.064275+0000","last_became_peered":"2024-03-20T09:13:22.064275+0000","last_unstale":"2024-03-20T09:15:09.734019+0000","last_undegraded":"2024-03-20T09:15:09.734019+0000","last_fullsized":"2024-03-20T09:15:09.734019+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'7909","last_scrub_stamp":"2024-03-20T09:15:09.333054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:15:09.333054+0000","objects_scrubbed":7634,"log_size":7923,"log_dups_size":0,"ondisk_log_size":7923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:21.566609+0000","scrub_duration":1910,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":498270208,"num_objects":7603,"num_object_clones":0,"num_object_copies":22809,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7923,"num_write_kb":496832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"20'137","reported_seq":1069,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-03-20T09:15:10.603224+0000","last_change":"2024-03-20T09:14:38.386308+0000","last_active":"2024-03-20T09:15:10.603224+0000","last_peered":"2024-03-20T09:15:10.603224+0000","last_clean":"2024-03-20T09:15:10.603224+0000","last_became_active":"2024-03-20T09:07:52.593671+0000","last_became_peered":"2024-03-20T09:07:52.593671+0000","last_unstale":"2024-03-20T09:15:10.603224+0000","last_undegraded":"2024-03-20T09:15:10.603224+0000","last_fullsized":"2024-03-20T09:15:10.603224+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:14:38.386162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:14:38.386162+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:46.342680+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.6","version":"556'7801","reported_seq":7968,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-03-20T09:15:11.328708+0000","last_change":"2024-03-20T09:15:01.280269+0000","last_active":"2024-03-20T09:15:11.328708+0000","last_peered":"2024-03-20T09:15:11.328708+0000","last_clean":"2024-03-20T09:15:11.328708+0000","last_became_active":"2024-03-20T09:13:22.067362+0000","last_became_peered":"2024-03-20T09:13:22.067362+0000","last_unstale":"2024-03-20T09:15:11.328708+0000","last_undegraded":"2024-03-20T09:15:11.328708+0000","last_fullsized":"2024-03-20T09:15:11.328708+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"553'7597","last_scrub_stamp":"2024-03-20T09:15:01.280134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:15:01.280134+0000","objects_scrubbed":7550,"log_size":7801,"log_dups_size":0,"ondisk_log_size":7801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:02.447803+0000","scrub_duration":3618,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485949440,"num_objects":7415,"num_object_clones":0,"num_object_copies":22245,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7801,"num_write_kb":486912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"20'137","reported_seq":1061,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-03-20T09:15:10.631588+0000","last_change":"2024-03-20T09:13:49.033462+0000","last_active":"2024-03-20T09:15:10.631588+0000","last_peered":"2024-03-20T09:15:10.631588+0000","last_clean":"2024-03-20T09:15:10.631588+0000","last_became_active":"2024-03-20T09:12:02.221686+0000","last_became_peered":"2024-03-20T09:12:02.221686+0000","last_unstale":"2024-03-20T09:15:10.631588+0000","last_undegraded":"2024-03-20T09:15:10.631588+0000","last_fullsized":"2024-03-20T09:15:10.631588+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:49.033327+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:18.788771+0000","last_clean_scrub_stamp":"2024-03-20T09:13:49.033327+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:10.707005+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,8],"acting":[1,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"555'7717","reported_seq":7871,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-03-20T09:15:10.605833+0000","last_change":"2024-03-20T09:14:38.547201+0000","last_active":"2024-03-20T09:15:10.605833+0000","last_peered":"2024-03-20T09:15:10.605833+0000","last_clean":"2024-03-20T09:15:10.605833+0000","last_became_active":"2024-03-20T09:13:22.062992+0000","last_became_peered":"2024-03-20T09:13:22.062992+0000","last_unstale":"2024-03-20T09:15:10.605833+0000","last_undegraded":"2024-03-20T09:15:10.605833+0000","last_fullsized":"2024-03-20T09:15:10.605833+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"550'6432","last_scrub_stamp":"2024-03-20T09:14:38.546966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:14:38.546966+0000","objects_scrubbed":6395,"log_size":7717,"log_dups_size":0,"ondisk_log_size":7717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:50.964808+0000","scrub_duration":1128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":480837632,"num_objects":7337,"num_object_clones":0,"num_object_copies":22011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7717,"num_write_kb":481728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":1125,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-20T09:15:07.500044+0000","last_change":"2024-03-20T09:14:45.600449+0000","last_active":"2024-03-20T09:15:07.500044+0000","last_peered":"2024-03-20T09:15:07.500044+0000","last_clean":"2024-03-20T09:15:07.500044+0000","last_became_active":"2024-03-20T09:12:14.305989+0000","last_became_peered":"2024-03-20T09:12:14.305989+0000","last_unstale":"2024-03-20T09:15:07.500044+0000","last_undegraded":"2024-03-20T09:15:07.500044+0000","last_fullsized":"2024-03-20T09:15:07.500044+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:14:45.600372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:14:45.600372+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:03.841360+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.7","version":"556'7769","reported_seq":7929,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-03-20T09:15:11.251988+0000","last_change":"2024-03-20T09:14:48.807600+0000","last_active":"2024-03-20T09:15:11.251988+0000","last_peered":"2024-03-20T09:15:11.251988+0000","last_clean":"2024-03-20T09:15:11.251988+0000","last_became_active":"2024-03-20T09:13:22.065656+0000","last_became_peered":"2024-03-20T09:13:22.065656+0000","last_unstale":"2024-03-20T09:15:11.251988+0000","last_undegraded":"2024-03-20T09:15:11.251988+0000","last_fullsized":"2024-03-20T09:15:11.251988+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"551'7096","last_scrub_stamp":"2024-03-20T09:14:48.807283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:14:48.807283+0000","objects_scrubbed":7044,"log_size":7769,"log_dups_size":0,"ondisk_log_size":7769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:49.433056+0000","scrub_duration":2072,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483590144,"num_objects":7379,"num_object_clones":0,"num_object_copies":22137,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7769,"num_write_kb":484736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,0],"acting":[1,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"20'137","reported_seq":1377,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-03-20T09:15:10.644801+0000","last_change":"2024-03-20T09:15:01.958964+0000","last_active":"2024-03-20T09:15:10.644801+0000","last_peered":"2024-03-20T09:15:10.644801+0000","last_clean":"2024-03-20T09:15:10.644801+0000","last_became_active":"2024-03-20T09:12:14.313541+0000","last_became_peered":"2024-03-20T09:12:14.313541+0000","last_unstale":"2024-03-20T09:15:10.644801+0000","last_undegraded":"2024-03-20T09:15:10.644801+0000","last_fullsized":"2024-03-20T09:15:10.644801+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:01.958839+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:19.513511+0000","last_clean_scrub_stamp":"2024-03-20T09:15:01.958839+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:16.285345+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"556'9544","reported_seq":9731,"reported_epoch":556,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-20T09:15:11.331669+0000","last_change":"2024-03-20T09:14:52.652029+0000","last_active":"2024-03-20T09:15:11.331669+0000","last_peered":"2024-03-20T09:15:11.331669+0000","last_clean":"2024-03-20T09:15:11.331669+0000","last_became_active":"2024-03-20T09:13:24.443054+0000","last_became_peered":"2024-03-20T09:13:24.443054+0000","last_unstale":"2024-03-20T09:15:11.331669+0000","last_undegraded":"2024-03-20T09:15:11.331669+0000","last_fullsized":"2024-03-20T09:15:11.331669+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":2373,"log_size":9544,"log_dups_size":0,"ondisk_log_size":9544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 18s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":476839936,"num_objects":7276,"num_object_clones":0,"num_object_copies":21828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7650,"num_write_kb":477632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"20'137","reported_seq":1073,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-03-20T09:15:10.602702+0000","last_change":"2024-03-20T09:14:47.524723+0000","last_active":"2024-03-20T09:15:10.602702+0000","last_peered":"2024-03-20T09:15:10.602702+0000","last_clean":"2024-03-20T09:15:10.602702+0000","last_became_active":"2024-03-20T09:06:02.923630+0000","last_became_peered":"2024-03-20T09:06:02.923630+0000","last_unstale":"2024-03-20T09:15:10.602702+0000","last_undegraded":"2024-03-20T09:15:10.602702+0000","last_fullsized":"2024-03-20T09:15:10.602702+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:14:47.524588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:14:47.524588+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:59.328110+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2],"acting":[10,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2","version":"554'7723","reported_seq":7880,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-20T09:15:07.143787+0000","last_change":"2024-03-20T09:14:47.368419+0000","last_active":"2024-03-20T09:15:07.143787+0000","last_peered":"2024-03-20T09:15:07.143787+0000","last_clean":"2024-03-20T09:15:07.143787+0000","last_became_active":"2024-03-20T09:13:22.583529+0000","last_became_peered":"2024-03-20T09:13:22.583529+0000","last_unstale":"2024-03-20T09:15:07.143787+0000","last_undegraded":"2024-03-20T09:15:07.143787+0000","last_fullsized":"2024-03-20T09:15:07.143787+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"550'7072","last_scrub_stamp":"2024-03-20T09:14:47.368211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:14:47.368211+0000","objects_scrubbed":7034,"log_size":7723,"log_dups_size":0,"ondisk_log_size":7723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:50.465872+0000","scrub_duration":1228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":489488384,"num_objects":7469,"num_object_clones":0,"num_object_copies":22407,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7723,"num_write_kb":486144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,8],"acting":[5,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"20'137","reported_seq":1065,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-03-20T09:15:10.605890+0000","last_change":"2024-03-20T09:14:10.561025+0000","last_active":"2024-03-20T09:15:10.605890+0000","last_peered":"2024-03-20T09:15:10.605890+0000","last_clean":"2024-03-20T09:15:10.605890+0000","last_became_active":"2024-03-20T09:13:27.087760+0000","last_became_peered":"2024-03-20T09:13:27.087760+0000","last_unstale":"2024-03-20T09:15:10.605890+0000","last_undegraded":"2024-03-20T09:15:10.605890+0000","last_fullsized":"2024-03-20T09:15:10.605890+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:14:10.560890+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:14:10.560890+0000","last_clean_scrub_stamp":"2024-03-20T09:14:10.560890+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:11.348848+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"556'7738","reported_seq":7896,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-03-20T09:15:11.191168+0000","last_change":"2024-03-20T09:14:41.980308+0000","last_active":"2024-03-20T09:15:11.191168+0000","last_peered":"2024-03-20T09:15:11.191168+0000","last_clean":"2024-03-20T09:15:11.191168+0000","last_became_active":"2024-03-20T09:13:22.064461+0000","last_became_peered":"2024-03-20T09:13:22.064461+0000","last_unstale":"2024-03-20T09:15:11.191168+0000","last_undegraded":"2024-03-20T09:15:11.191168+0000","last_fullsized":"2024-03-20T09:15:11.191168+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"550'6707","last_scrub_stamp":"2024-03-20T09:14:41.980111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:14:41.980111+0000","objects_scrubbed":6659,"log_size":7738,"log_dups_size":0,"ondisk_log_size":7738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:42.128922+0000","scrub_duration":1587,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482869248,"num_objects":7368,"num_object_clones":0,"num_object_copies":22104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7738,"num_write_kb":483392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,1],"acting":[10,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.b","version":"20'137","reported_seq":930,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-20T09:15:07.500356+0000","last_change":"2024-03-20T09:13:46.614783+0000","last_active":"2024-03-20T09:15:07.500356+0000","last_peered":"2024-03-20T09:15:07.500356+0000","last_clean":"2024-03-20T09:15:07.500356+0000","last_became_active":"2024-03-20T09:13:20.070596+0000","last_became_peered":"2024-03-20T09:13:20.070596+0000","last_unstale":"2024-03-20T09:15:07.500356+0000","last_undegraded":"2024-03-20T09:15:07.500356+0000","last_fullsized":"2024-03-20T09:15:07.500356+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:46.614691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:46.614691+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.b","version":"556'7650","reported_seq":7807,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-03-20T09:15:11.715388+0000","last_change":"2024-03-20T09:14:46.191729+0000","last_active":"2024-03-20T09:15:11.715388+0000","last_peered":"2024-03-20T09:15:11.715388+0000","last_clean":"2024-03-20T09:15:11.715388+0000","last_became_active":"2024-03-20T09:13:22.064687+0000","last_became_peered":"2024-03-20T09:13:22.064687+0000","last_unstale":"2024-03-20T09:15:11.715388+0000","last_undegraded":"2024-03-20T09:15:11.715388+0000","last_fullsized":"2024-03-20T09:15:11.715388+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"550'6828","last_scrub_stamp":"2024-03-20T09:14:46.191490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:14:46.191490+0000","objects_scrubbed":6762,"log_size":7650,"log_dups_size":0,"ondisk_log_size":7650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:15.502318+0000","scrub_duration":2514,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":476053504,"num_objects":7264,"num_object_clones":0,"num_object_copies":21792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7650,"num_write_kb":477248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"20'137","reported_seq":593,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-03-20T09:15:10.604549+0000","last_change":"2024-03-20T09:14:08.589630+0000","last_active":"2024-03-20T09:15:10.604549+0000","last_peered":"2024-03-20T09:15:10.604549+0000","last_clean":"2024-03-20T09:15:10.604549+0000","last_became_active":"2024-03-20T09:12:14.311319+0000","last_became_peered":"2024-03-20T09:12:14.311319+0000","last_unstale":"2024-03-20T09:15:10.604549+0000","last_undegraded":"2024-03-20T09:15:10.604549+0000","last_fullsized":"2024-03-20T09:15:10.604549+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:14:08.589510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:14:08.589510+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:21.094562+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"555'7679","reported_seq":7833,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-03-20T09:15:10.605663+0000","last_change":"2024-03-20T09:14:39.771481+0000","last_active":"2024-03-20T09:15:10.605663+0000","last_peered":"2024-03-20T09:15:10.605663+0000","last_clean":"2024-03-20T09:15:10.605663+0000","last_became_active":"2024-03-20T09:13:22.063927+0000","last_became_peered":"2024-03-20T09:13:22.063927+0000","last_unstale":"2024-03-20T09:15:10.605663+0000","last_undegraded":"2024-03-20T09:15:10.605663+0000","last_fullsized":"2024-03-20T09:15:10.605663+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"550'6524","last_scrub_stamp":"2024-03-20T09:14:39.771224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:14:39.771224+0000","objects_scrubbed":6484,"log_size":7679,"log_dups_size":0,"ondisk_log_size":7679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:03.494718+0000","scrub_duration":1395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482279424,"num_objects":7359,"num_object_clones":0,"num_object_copies":22077,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7679,"num_write_kb":481216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"20'137","reported_seq":1078,"reported_epoch":556,"state":"active+clean+scrubbing","last_fresh":"2024-03-20T09:15:10.605860+0000","last_change":"2024-03-20T09:15:07.570520+0000","last_active":"2024-03-20T09:15:10.605860+0000","last_peered":"2024-03-20T09:15:10.605860+0000","last_clean":"2024-03-20T09:15:10.605860+0000","last_became_active":"2024-03-20T09:10:35.119706+0000","last_became_peered":"2024-03-20T09:10:35.119706+0000","last_unstale":"2024-03-20T09:15:10.605860+0000","last_undegraded":"2024-03-20T09:15:10.605860+0000","last_fullsized":"2024-03-20T09:15:10.605860+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:53.634986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:53.634986+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"555'9172","reported_seq":9312,"reported_epoch":555,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-20T09:15:09.892307+0000","last_change":"2024-03-20T09:14:58.425816+0000","last_active":"2024-03-20T09:15:09.892307+0000","last_peered":"2024-03-20T09:15:09.892307+0000","last_clean":"2024-03-20T09:15:09.892307+0000","last_became_active":"2024-03-20T09:13:24.065349+0000","last_became_peered":"2024-03-20T09:13:24.065349+0000","last_unstale":"2024-03-20T09:15:09.892307+0000","last_undegraded":"2024-03-20T09:15:09.892307+0000","last_fullsized":"2024-03-20T09:15:09.892307+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:13:21.066370+0000","objects_scrubbed":1774,"log_size":9172,"log_dups_size":0,"ondisk_log_size":9172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 11s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485949440,"num_objects":7415,"num_object_clones":0,"num_object_copies":22245,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7741,"num_write_kb":484992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"20'137","reported_seq":1105,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-20T09:15:07.503432+0000","last_change":"2024-03-20T09:14:46.139707+0000","last_active":"2024-03-20T09:15:07.503432+0000","last_peered":"2024-03-20T09:15:07.503432+0000","last_clean":"2024-03-20T09:15:07.503432+0000","last_became_active":"2024-03-20T09:13:27.087686+0000","last_became_peered":"2024-03-20T09:13:27.087686+0000","last_unstale":"2024-03-20T09:15:07.503432+0000","last_undegraded":"2024-03-20T09:15:07.503432+0000","last_fullsized":"2024-03-20T09:15:07.503432+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:14:46.139576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:14:46.139576+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:08.971745+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"555'7654","reported_seq":7817,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-20T09:15:09.793893+0000","last_change":"2024-03-20T09:15:05.380631+0000","last_active":"2024-03-20T09:15:09.793893+0000","last_peered":"2024-03-20T09:15:09.793893+0000","last_clean":"2024-03-20T09:15:09.793893+0000","last_became_active":"2024-03-20T09:13:22.063673+0000","last_became_peered":"2024-03-20T09:13:22.063673+0000","last_unstale":"2024-03-20T09:15:09.793893+0000","last_undegraded":"2024-03-20T09:15:09.793893+0000","last_fullsized":"2024-03-20T09:15:09.793893+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"553'7531","last_scrub_stamp":"2024-03-20T09:15:05.380547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:15:05.380547+0000","objects_scrubbed":7413,"log_size":7654,"log_dups_size":0,"ondisk_log_size":7654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:25.174578+0000","scrub_duration":962,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":477364224,"num_objects":7284,"num_object_clones":0,"num_object_copies":21852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7654,"num_write_kb":478016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"20'137","reported_seq":660,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-03-20T09:15:10.624502+0000","last_change":"2024-03-20T09:15:07.365971+0000","last_active":"2024-03-20T09:15:10.624502+0000","last_peered":"2024-03-20T09:15:10.624502+0000","last_clean":"2024-03-20T09:15:10.624502+0000","last_became_active":"2024-03-20T09:12:14.307864+0000","last_became_peered":"2024-03-20T09:12:14.307864+0000","last_unstale":"2024-03-20T09:15:10.624502+0000","last_undegraded":"2024-03-20T09:15:10.624502+0000","last_fullsized":"2024-03-20T09:15:10.624502+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:07.365857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:15:07.365857+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:35.457364+0000","scrub_duration":268,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"20'137","reported_seq":971,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-20T09:15:05.458917+0000","last_change":"2024-03-20T09:14:40.236846+0000","last_active":"2024-03-20T09:15:05.458917+0000","last_peered":"2024-03-20T09:15:05.458917+0000","last_clean":"2024-03-20T09:15:05.458917+0000","last_became_active":"2024-03-20T09:10:34.096758+0000","last_became_peered":"2024-03-20T09:10:34.096758+0000","last_unstale":"2024-03-20T09:15:05.458917+0000","last_undegraded":"2024-03-20T09:15:05.458917+0000","last_fullsized":"2024-03-20T09:15:05.458917+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:14:40.236685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:14:40.236685+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:04.722728+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"20'137","reported_seq":1048,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-03-20T09:15:10.631481+0000","last_change":"2024-03-20T09:15:05.139462+0000","last_active":"2024-03-20T09:15:10.631481+0000","last_peered":"2024-03-20T09:15:10.631481+0000","last_clean":"2024-03-20T09:15:10.631481+0000","last_became_active":"2024-03-20T09:04:04.103645+0000","last_became_peered":"2024-03-20T09:04:04.103645+0000","last_unstale":"2024-03-20T09:15:10.631481+0000","last_undegraded":"2024-03-20T09:15:10.631481+0000","last_fullsized":"2024-03-20T09:15:10.631481+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:05.139344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:15:05.139344+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:05.673418+0000","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"20'137","reported_seq":497,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-20T09:15:05.458717+0000","last_change":"2024-03-20T09:14:45.137392+0000","last_active":"2024-03-20T09:15:05.458717+0000","last_peered":"2024-03-20T09:15:05.458717+0000","last_clean":"2024-03-20T09:15:05.458717+0000","last_became_active":"2024-03-20T09:10:34.097191+0000","last_became_peered":"2024-03-20T09:10:34.097191+0000","last_unstale":"2024-03-20T09:15:05.458717+0000","last_undegraded":"2024-03-20T09:15:05.458717+0000","last_fullsized":"2024-03-20T09:15:05.458717+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:14:45.137315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:14:45.137315+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:07.569642+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"20'137","reported_seq":912,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-03-20T09:15:10.644666+0000","last_change":"2024-03-20T09:14:57.929482+0000","last_active":"2024-03-20T09:15:10.644666+0000","last_peered":"2024-03-20T09:15:10.644666+0000","last_clean":"2024-03-20T09:15:10.644666+0000","last_became_active":"2024-03-20T09:13:27.087859+0000","last_became_peered":"2024-03-20T09:13:27.087859+0000","last_unstale":"2024-03-20T09:15:10.644666+0000","last_undegraded":"2024-03-20T09:15:10.644666+0000","last_fullsized":"2024-03-20T09:15:10.644666+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:14:57.929339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:14:57.929339+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:02.614879+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,3],"acting":[0,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"20'137","reported_seq":737,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-03-20T09:15:10.625177+0000","last_change":"2024-03-20T09:14:49.747607+0000","last_active":"2024-03-20T09:15:10.625177+0000","last_peered":"2024-03-20T09:15:10.625177+0000","last_clean":"2024-03-20T09:15:10.625177+0000","last_became_active":"2024-03-20T09:13:27.087760+0000","last_became_peered":"2024-03-20T09:13:27.087760+0000","last_unstale":"2024-03-20T09:15:10.625177+0000","last_undegraded":"2024-03-20T09:15:10.625177+0000","last_fullsized":"2024-03-20T09:15:10.625177+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:14:49.747468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:14:49.747468+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:15.011120+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"20'137","reported_seq":1056,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-03-20T09:15:10.652126+0000","last_change":"2024-03-20T09:13:48.832256+0000","last_active":"2024-03-20T09:15:10.652126+0000","last_peered":"2024-03-20T09:15:10.652126+0000","last_clean":"2024-03-20T09:15:10.652126+0000","last_became_active":"2024-03-20T09:10:35.095339+0000","last_became_peered":"2024-03-20T09:10:35.095339+0000","last_unstale":"2024-03-20T09:15:10.652126+0000","last_undegraded":"2024-03-20T09:15:10.652126+0000","last_fullsized":"2024-03-20T09:15:10.652126+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:48.831557+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:23.640533+0000","last_clean_scrub_stamp":"2024-03-20T09:13:48.831557+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"20'137","reported_seq":1122,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-03-20T09:15:07.498549+0000","last_change":"2024-03-20T09:14:50.041856+0000","last_active":"2024-03-20T09:15:07.498549+0000","last_peered":"2024-03-20T09:15:07.498549+0000","last_clean":"2024-03-20T09:15:07.498549+0000","last_became_active":"2024-03-20T09:04:14.208448+0000","last_became_peered":"2024-03-20T09:04:14.208448+0000","last_unstale":"2024-03-20T09:15:07.498549+0000","last_undegraded":"2024-03-20T09:15:07.498549+0000","last_fullsized":"2024-03-20T09:15:07.498549+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:14:50.041715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:14:50.041715+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:14.138911+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"20'137","reported_seq":390,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-03-20T09:15:10.604494+0000","last_change":"2024-03-20T09:14:40.654041+0000","last_active":"2024-03-20T09:15:10.604494+0000","last_peered":"2024-03-20T09:15:10.604494+0000","last_clean":"2024-03-20T09:15:10.604494+0000","last_became_active":"2024-03-20T09:12:14.307581+0000","last_became_peered":"2024-03-20T09:12:14.307581+0000","last_unstale":"2024-03-20T09:15:10.604494+0000","last_undegraded":"2024-03-20T09:15:10.604494+0000","last_fullsized":"2024-03-20T09:15:10.604494+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:14:40.653909+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:07:06.480160+0000","last_clean_scrub_stamp":"2024-03-20T09:14:40.653909+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:53.456165+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"20'137","reported_seq":660,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-03-20T09:15:10.624161+0000","last_change":"2024-03-20T09:13:47.938675+0000","last_active":"2024-03-20T09:15:10.624161+0000","last_peered":"2024-03-20T09:15:10.624161+0000","last_clean":"2024-03-20T09:15:10.624161+0000","last_became_active":"2024-03-20T09:13:20.074439+0000","last_became_peered":"2024-03-20T09:13:20.074439+0000","last_unstale":"2024-03-20T09:15:10.624161+0000","last_undegraded":"2024-03-20T09:15:10.624161+0000","last_fullsized":"2024-03-20T09:15:10.624161+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:47.938552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:47.938552+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,11],"acting":[1,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":7743143960,"num_objects":118152,"num_object_clones":0,"num_object_copies":354456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":118152,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123682,"num_write_kb":7738625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":922,"num_bytes_recovered":60424192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":23223349248,"data_stored":23223337032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130743,"ondisk_log_size":130743,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4247,"ondisk_log_size":4247,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860645,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4180656,"kb_used_data":3810728,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":89564496,"statfs":{"total":95995035648,"available":91714043904,"internally_reserved":0,"allocated":3902185472,"data_stored":3901203050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":378796484},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":89,"seq":382252089579,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2158636,"kb_used_data":1888676,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":91586516,"statfs":{"total":95995035648,"available":93784592384,"internally_reserved":0,"allocated":1934004224,"data_stored":1933025874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":276429248},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":0,"seq":343597383922,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":70,"seq":300647710967,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1174212,"kb_used_data":944508,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":92570940,"statfs":{"total":95995035648,"available":94792642560,"internally_reserved":0,"allocated":967176192,"data_stored":966255833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":235207110},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":60,"seq":257698038012,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2346052,"kb_used_data":1895428,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":91399100,"statfs":{"total":95995035648,"available":93592678400,"internally_reserved":0,"allocated":1940918272,"data_stored":1939994361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":461240778},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332353,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3184556,"kb_used_data":2820196,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":90560596,"statfs":{"total":95995035648,"available":92734050304,"internally_reserved":0,"allocated":2887880704,"data_stored":2886902354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":373094854},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575131,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4667968,"kb_used_data":4257600,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":89077184,"statfs":{"total":95995035648,"available":91215036416,"internally_reserved":0,"allocated":4359782400,"data_stored":4358857969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":420018630},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,1,0,0,12,8,10,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738650,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2297984,"kb_used_data":1883456,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":91447168,"statfs":{"total":95995035648,"available":93641900032,"internally_reserved":0,"allocated":1928658944,"data_stored":1927734513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411604,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1673004,"kb_used_data":1434732,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":92072148,"statfs":{"total":95995035648,"available":94281879552,"internally_reserved":0,"allocated":1469165568,"data_stored":1468183666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":243726789},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149968,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2734244,"kb_used_data":2369636,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":91010908,"statfs":{"total":95995035648,"available":93195169792,"internally_reserved":0,"allocated":2426507264,"data_stored":2425528914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":373094855},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953742,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":443,"seq":1902670512184,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1586708,"kb_used_data":1426068,"kb_used_omap":181,"kb_used_meta":160266,"kb_avail":92158444,"statfs":{"total":95995035648,"available":94370246656,"internally_reserved":0,"allocated":1460293632,"data_stored":1459254629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":186131,"internal_metadata":164112621},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1921912832,"data_stored":1921908760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4354084864,"data_stored":4354080792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1465319424,"data_stored":1465319424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2418016256,"data_stored":2418016256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1456472064,"data_stored":1456472064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2884173824,"data_stored":2884173824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1937375232,"data_stored":1937375232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":963837952,"data_stored":963837952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1928593408,"data_stored":1928593408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":3893563392,"data_stored":3893559320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:15:14.077 INFO:tasks.thrashosds.thrasher:Setting 1.1 to [6, 11, 5] 2024-03-20T09:15:14.077 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '1.1', '6', '11', '5'] 2024-03-20T09:15:14.077 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pg-upmap 1.1 6 11 5 2024-03-20T09:15:14.103 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:15:14.235 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:14.544 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:15:14.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:14.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:15:14.657 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:15:14.688 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:14.708 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:15:14.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:14.711 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:15:14.812 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:15:14.840 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:14.862 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:15:14.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:14.865 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:15:14.966 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:15:14.992 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:15.012 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:15:15.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:15.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:15:15.116 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:15:15.160 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:15:15.194 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:15.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:15.617 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:15:15.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:15:15.719 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:15:15.746 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:15.770 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:15:15.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:15.774 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:15:15.875 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:15:15.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:15 smithi122 ceph-mon[73495]: pgmap v1315: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 7.2 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 127 B/s rd, 662 KiB/s wr, 314 op/s 2024-03-20T09:15:15.950 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:15 smithi026 ceph-mon[68309]: pgmap v1315: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 7.2 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 127 B/s rd, 662 KiB/s wr, 314 op/s 2024-03-20T09:15:15.953 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:16.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:15 smithi115 ceph-mon[72476]: pgmap v1315: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 7.2 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 127 B/s rd, 662 KiB/s wr, 314 op/s 2024-03-20T09:15:16.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:16 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:15:16.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:16 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:15:16.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:16 smithi026 ceph-mon[68309]: osdmap e557: 12 total, 12 up, 10 in 2024-03-20T09:15:17.022 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:15:17.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:17.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:15:17.148 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:15:17.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:16 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:15:17.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:16 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:15:17.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:16 smithi115 ceph-mon[72476]: osdmap e557: 12 total, 12 up, 10 in 2024-03-20T09:15:17.174 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:17.192 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:15:17.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:17.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:15:17.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:16 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:15:17.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:16 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:15:17.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:16 smithi122 ceph-mon[73495]: osdmap e557: 12 total, 12 up, 10 in 2024-03-20T09:15:17.295 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:15:17.322 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:17.343 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:15:17.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:17.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:15:17.447 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:15:17.474 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:17.495 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:15:17.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:17.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:15:17.599 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:15:17.625 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:17.643 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:15:17.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:17.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:15:17.747 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:15:17.776 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:17.798 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:15:17.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:17.802 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:15:17.903 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:15:17.930 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:17.950 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:15:17.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:17.954 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:15:18.055 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:15:18.081 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:17 smithi122 ceph-mon[73495]: pgmap v1317: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 7.2 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 321 op/s 2024-03-20T09:15:18.081 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:17 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:15:18.081 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:17 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:15:18.083 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:18.106 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:15:18.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:18.110 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:15:18.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:17 smithi115 ceph-mon[72476]: pgmap v1317: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 7.2 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 321 op/s 2024-03-20T09:15:18.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:17 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:15:18.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:17 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:15:18.210 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:15:18.227 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:17 smithi026 ceph-mon[68309]: pgmap v1317: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 7.2 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 0 B/s wr, 321 op/s 2024-03-20T09:15:18.227 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:17 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:15:18.227 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:17 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:15:18.349 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:18.564 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:15:18.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:18.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:15:18.674 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:15:18.700 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:18.719 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:15:18.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:18.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:15:18.824 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:15:18.932 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:19.077 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:15:19.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:19.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:15:19.089 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:15:19.187 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:15:19.216 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:19.238 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:15:19.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:19.242 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:15:19.343 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:15:19.370 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:19.396 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:15:19.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:19.403 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:15:19.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:19 smithi122 ceph-mon[73495]: pgmap v1318: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 7.1 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 401 op/s 2024-03-20T09:15:19.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:19 smithi122 ceph-mon[73495]: osdmap e558: 12 total, 12 up, 10 in 2024-03-20T09:15:19.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:19 smithi026 ceph-mon[68309]: pgmap v1318: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 7.1 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 401 op/s 2024-03-20T09:15:19.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:19 smithi026 ceph-mon[68309]: osdmap e558: 12 total, 12 up, 10 in 2024-03-20T09:15:19.505 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:15:19.529 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:19 smithi115 ceph-mon[72476]: pgmap v1318: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 7.1 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 401 op/s 2024-03-20T09:15:19.529 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:19 smithi115 ceph-mon[72476]: osdmap e558: 12 total, 12 up, 10 in 2024-03-20T09:15:19.531 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:19.549 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:15:19.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:19.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:15:19.653 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:15:19.794 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:20.049 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:15:20.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:20.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:15:20.159 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:15:20.313 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:20.397 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:15:20.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:20.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:15:20.410 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:15:20.410 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:15:20.410 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-03-20T09:15:20.410 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-03-20T09:15:20.410 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:15:20.417 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:15:20.461 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:15:20.501 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:15:20.536 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:15:20.602 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:15:20.646 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:20.737 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:15:20.786 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:15:20.805 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:15:20.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:20.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:15:20.846 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:15:20.926 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:15:20.931 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:15:21.037 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:15:21.096 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:21.123 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:15:21.181 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:15:21.274 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:15:21.354 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:21 smithi026 ceph-mon[68309]: pgmap v1320: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 7.1 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 225 op/s 2024-03-20T09:15:21.355 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:21 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3869071041' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:15:21.356 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:15:21.356 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:15:21.356 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:15:21.357 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:15:21.357 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:15:21.357 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:15:21.357 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:15:21.357 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:15:21.357 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:15:21.357 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:15:21.357 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:15:21.357 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:15:21.357 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:15:21.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:21 smithi115 ceph-mon[72476]: pgmap v1320: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 7.1 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 225 op/s 2024-03-20T09:15:21.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:21 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3869071041' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:15:21.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:21 smithi122 ceph-mon[73495]: pgmap v1320: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 7.1 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 225 op/s 2024-03-20T09:15:21.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:21 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3869071041' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:15:22.166 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:15:22.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:22.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:15:22.276 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:15:22.305 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:22.331 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:15:22.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:22.335 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:15:22.436 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:15:22.463 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:22.485 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:15:22.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:22.489 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:15:22.590 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:15:22.617 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:22.638 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:15:22.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:22.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:15:22.742 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:15:22.771 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:22.791 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:15:22.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:22.795 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:15:22.895 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:15:23.014 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:23.104 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:15:23.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:23.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:15:23.224 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:15:23.250 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:23.269 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:15:23.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:23.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:15:23.373 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:15:23.406 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:23.431 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:15:23.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:23.435 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:15:23.537 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:15:23.563 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:23 smithi122 ceph-mon[73495]: pgmap v1321: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 7.0 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 321 op/s 2024-03-20T09:15:23.565 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:23.585 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:15:23.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:23.588 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:15:23.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:23 smithi115 ceph-mon[72476]: pgmap v1321: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 7.0 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 321 op/s 2024-03-20T09:15:23.689 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:15:23.717 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:23.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:23 smithi026 ceph-mon[68309]: pgmap v1321: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 7.0 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 321 op/s 2024-03-20T09:15:23.738 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:15:23.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:23.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:15:23.842 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:15:23.869 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:23.893 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:15:23.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:23.896 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:15:23.997 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:15:24.024 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:24.043 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:15:24.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:24.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:15:24.147 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:15:24.175 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:24.196 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:15:24.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:24.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:15:24.302 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:15:24.332 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:24.355 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:15:24.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:24.359 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:15:24.460 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:15:24.489 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:24.514 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:15:24.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:24.516 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:15:24.567 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-20T09:15:24.617 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:15:24.642 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:24 smithi115 ceph-mon[72476]: pgmap v1322: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 7.0 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 303 op/s 2024-03-20T09:15:24.645 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:24.664 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:15:24.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:24.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:15:24.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:24 smithi122 ceph-mon[73495]: pgmap v1322: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 7.0 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 303 op/s 2024-03-20T09:15:24.707 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:24 smithi026 ceph-mon[68309]: pgmap v1322: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 7.0 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 303 op/s 2024-03-20T09:15:24.768 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:15:25.001 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:15:25.047 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:25.114 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:15:25.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:25.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:15:25.261 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:15:25.287 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:25.307 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:15:25.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:25.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:15:25.411 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:15:25.439 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:25.467 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:15:25.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:25.470 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:15:25.571 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:15:25.772 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:26.696 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:15:26.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:26.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:15:26.805 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:15:26.833 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:26.852 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:15:26.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:26.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:15:26.959 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:15:27.115 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:27.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:27 smithi115 ceph-mon[72476]: pgmap v1323: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 7.0 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 318 op/s 2024-03-20T09:15:27.306 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:27 smithi026 ceph-mon[68309]: pgmap v1323: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 7.0 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 318 op/s 2024-03-20T09:15:27.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:27 smithi122 ceph-mon[73495]: pgmap v1323: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 7.0 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 318 op/s 2024-03-20T09:15:27.831 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:15:27.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:27.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:15:27.938 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:15:27.965 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:27.988 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:15:27.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:27.992 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:15:28.093 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:15:28.120 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:28.138 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:15:28.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:28.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:15:28.242 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:15:28.408 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:28.467 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:15:28.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:28.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:15:28.577 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:15:28.604 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:28.631 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:15:28.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:28.635 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:15:28.736 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:15:28.764 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:28.789 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:15:28.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:28.796 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:15:28.897 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:15:28.926 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:28.945 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:15:28.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:28.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:15:28.977 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:15:29.049 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:15:29.069 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:15:29.171 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:29.346 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:15:29.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:29.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:15:29.406 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:29 smithi026 ceph-mon[68309]: pgmap v1324: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 6.9 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 313 op/s 2024-03-20T09:15:29.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:29 smithi115 ceph-mon[72476]: pgmap v1324: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 6.9 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 313 op/s 2024-03-20T09:15:29.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:29 smithi122 ceph-mon[73495]: pgmap v1324: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 6.9 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 313 op/s 2024-03-20T09:15:29.454 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:15:29.480 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:29.499 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:15:29.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:29.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:15:29.603 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:15:29.633 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:29.653 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:15:29.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:29.657 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:15:29.759 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:15:29.787 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:29.808 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:15:29.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:29.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:15:29.912 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:15:29.938 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:29.958 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:15:29.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:29.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:15:30.062 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:15:30.093 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:30.115 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:15:30.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:30.119 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:15:30.162 INFO:teuthology.orchestra.run.smithi026.stderr:set 1.1 pg_upmap mapping to [6,11,5] 2024-03-20T09:15:30.171 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:15:30.220 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:15:30.248 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:30.271 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:15:30.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:30.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:15:30.376 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:15:30.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:30 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/804385571' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:15:30.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:30 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:15:30.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:30 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1924175858' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.1", "id": [6, 11, 5]}]: dispatch 2024-03-20T09:15:30.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:30 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.1", "id": [6, 11, 5]}]: dispatch 2024-03-20T09:15:30.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:30 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/804385571' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:15:30.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:30 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:15:30.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:30 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1924175858' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.1", "id": [6, 11, 5]}]: dispatch 2024-03-20T09:15:30.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:30 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.1", "id": [6, 11, 5]}]: dispatch 2024-03-20T09:15:30.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:30 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/804385571' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:15:30.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:30 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:15:30.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:30 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1924175858' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.1", "id": [6, 11, 5]}]: dispatch 2024-03-20T09:15:30.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:30 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.1", "id": [6, 11, 5]}]: dispatch 2024-03-20T09:15:30.465 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:31.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:31 smithi115 ceph-mon[72476]: pgmap v1325: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 6.9 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 309 op/s 2024-03-20T09:15:31.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:31 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:15:31.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:31 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:15:31.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:31 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.1", "id": [6, 11, 5]}]': finished 2024-03-20T09:15:31.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:31 smithi115 ceph-mon[72476]: osdmap e559: 12 total, 12 up, 10 in 2024-03-20T09:15:31.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:31 smithi115 ceph-mon[72476]: 1.f scrub starts 2024-03-20T09:15:31.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:31 smithi122 ceph-mon[73495]: pgmap v1325: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 6.9 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 309 op/s 2024-03-20T09:15:31.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:31 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:15:31.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:31 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:15:31.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:31 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.1", "id": [6, 11, 5]}]': finished 2024-03-20T09:15:31.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:31 smithi122 ceph-mon[73495]: osdmap e559: 12 total, 12 up, 10 in 2024-03-20T09:15:31.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:31 smithi122 ceph-mon[73495]: 1.f scrub starts 2024-03-20T09:15:31.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:31 smithi026 ceph-mon[68309]: pgmap v1325: 47 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 42 active+clean; 6.9 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 309 op/s 2024-03-20T09:15:31.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:31 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:15:31.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:31 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:15:31.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:31 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.1", "id": [6, 11, 5]}]': finished 2024-03-20T09:15:31.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:31 smithi026 ceph-mon[68309]: osdmap e559: 12 total, 12 up, 10 in 2024-03-20T09:15:31.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:31 smithi026 ceph-mon[68309]: 1.f scrub starts 2024-03-20T09:15:32.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:32 smithi122 ceph-mon[73495]: osdmap e560: 12 total, 12 up, 10 in 2024-03-20T09:15:32.475 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:32 smithi026 ceph-mon[68309]: osdmap e560: 12 total, 12 up, 10 in 2024-03-20T09:15:32.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:32 smithi115 ceph-mon[72476]: osdmap e560: 12 total, 12 up, 10 in 2024-03-20T09:15:33.110 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:15:33.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:33.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:15:33.215 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:15:33.238 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:33 smithi026 ceph-mon[68309]: pgmap v1328: 47 pgs: 1 active+clean+scrubbing, 1 unknown, 4 active+clean+scrubbing+deep, 41 active+clean; 6.8 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 337 op/s 2024-03-20T09:15:33.238 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:33 smithi026 ceph-mon[68309]: osdmap e561: 12 total, 12 up, 10 in 2024-03-20T09:15:33.289 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:33.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:33 smithi122 ceph-mon[73495]: pgmap v1328: 47 pgs: 1 active+clean+scrubbing, 1 unknown, 4 active+clean+scrubbing+deep, 41 active+clean; 6.8 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 337 op/s 2024-03-20T09:15:33.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:33 smithi122 ceph-mon[73495]: osdmap e561: 12 total, 12 up, 10 in 2024-03-20T09:15:33.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:33 smithi115 ceph-mon[72476]: pgmap v1328: 47 pgs: 1 active+clean+scrubbing, 1 unknown, 4 active+clean+scrubbing+deep, 41 active+clean; 6.8 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 337 op/s 2024-03-20T09:15:33.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:33 smithi115 ceph-mon[72476]: osdmap e561: 12 total, 12 up, 10 in 2024-03-20T09:15:33.966 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:15:33.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:33.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:15:34.071 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:15:34.147 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:34.194 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:15:34.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:34.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:15:34.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:34 smithi026 ceph-mon[68309]: osdmap e562: 12 total, 12 up, 10 in 2024-03-20T09:15:34.303 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:15:34.338 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:34.388 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:15:34.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:34.392 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:15:34.493 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:15:34.517 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:15:34.517 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:15:34.568 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:34.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:34 smithi115 ceph-mon[72476]: osdmap e562: 12 total, 12 up, 10 in 2024-03-20T09:15:34.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:34 smithi122 ceph-mon[73495]: osdmap e562: 12 total, 12 up, 10 in 2024-03-20T09:15:35.049 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:15:35.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:35.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:15:35.164 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:15:35.193 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:35.229 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:15:35.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:35.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:15:35.335 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:15:35.372 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:35.394 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:15:35.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:35.399 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:15:35.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:35 smithi026 ceph-mon[68309]: pgmap v1331: 47 pgs: 1 remapped+peering, 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 43 active+clean; 6.8 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 418 op/s 2024-03-20T09:15:35.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:35 smithi026 ceph-mon[68309]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-20T09:15:35.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:35 smithi026 ceph-mon[68309]: osdmap e563: 12 total, 12 up, 10 in 2024-03-20T09:15:35.499 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:15:35.524 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:35 smithi115 ceph-mon[72476]: pgmap v1331: 47 pgs: 1 remapped+peering, 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 43 active+clean; 6.8 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 418 op/s 2024-03-20T09:15:35.524 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:35 smithi115 ceph-mon[72476]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-20T09:15:35.524 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:35 smithi115 ceph-mon[72476]: osdmap e563: 12 total, 12 up, 10 in 2024-03-20T09:15:35.526 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:35.546 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:15:35.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:35.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:15:35.650 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:15:35.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:35 smithi122 ceph-mon[73495]: pgmap v1331: 47 pgs: 1 remapped+peering, 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 43 active+clean; 6.8 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 0 B/s wr, 418 op/s 2024-03-20T09:15:35.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:35 smithi122 ceph-mon[73495]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-20T09:15:35.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:35 smithi122 ceph-mon[73495]: osdmap e563: 12 total, 12 up, 10 in 2024-03-20T09:15:35.714 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:35.762 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:15:35.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:35.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:15:35.874 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:15:35.910 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:35.934 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:15:35.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:35.940 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:15:35.947 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:15:35.947 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:15:35.947 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-20T09:15:35.787+0000 7ff225b17700 0 --2- 172.21.15.26:0/240228728 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7ff220109160 0x7ff220115670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-03-20T09:15:35.995 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:15:36.031 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:15:36.040 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:15:36.085 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:15:36.118 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:15:36.137 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:36.166 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:15:36.203 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:15:36.238 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:15:36.282 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:15:36.321 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:15:36.359 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:15:36.391 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:15:36.397 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:15:36.398 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:15:36.398 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:15:36.398 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:15:36.398 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:15:36.398 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:15:36.398 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:15:36.398 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:15:36.398 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:15:36.398 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:15:36.398 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:15:36.398 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:15:36.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:36.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:15:36.408 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:36 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3528136094' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:15:36.408 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:36 smithi026 ceph-mon[68309]: pgmap v1333: 47 pgs: 47 active+clean; 6.6 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 791 op/s 2024-03-20T09:15:36.447 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-20T09:15:36.504 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:15:36.614 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:36.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:36 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3528136094' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:15:36.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:36 smithi115 ceph-mon[72476]: pgmap v1333: 47 pgs: 47 active+clean; 6.6 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 791 op/s 2024-03-20T09:15:36.691 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:15:36.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:36.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:15:36.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:36 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3528136094' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:15:36.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:36 smithi122 ceph-mon[73495]: pgmap v1333: 47 pgs: 47 active+clean; 6.6 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 791 op/s 2024-03-20T09:15:36.798 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:15:36.827 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:36.852 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:15:36.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:36.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:15:36.960 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:15:36.989 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:37.017 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:15:37.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:37.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:15:37.123 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:15:37.160 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:37.189 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:15:37.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:37.192 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:15:37.292 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:15:37.358 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:37.372 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:15:37.401 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:15:37.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:37.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:15:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:37 smithi122 ceph-mon[73495]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-20T09:15:37.513 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:15:37.544 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:37.581 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:15:37.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:37.588 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:15:37.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:37 smithi115 ceph-mon[72476]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-20T09:15:37.690 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:15:37.701 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:37 smithi026 ceph-mon[68309]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-20T09:15:37.721 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:37.745 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:15:37.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:37.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:15:37.802 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:15:37.849 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:15:37.876 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:37.901 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:15:37.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:37.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:15:38.006 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:15:38.039 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:38.063 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:15:38.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:38.073 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:15:38.173 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:15:38.202 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:38.223 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:15:38.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:38.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:15:38.327 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:15:38.409 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:38 smithi026 ceph-mon[68309]: pgmap v1334: 47 pgs: 47 active+clean; 6.5 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 938 op/s 2024-03-20T09:15:38.412 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:38.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:38 smithi122 ceph-mon[73495]: pgmap v1334: 47 pgs: 47 active+clean; 6.5 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 938 op/s 2024-03-20T09:15:38.453 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:15:38.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:38.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:15:38.574 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:15:38.611 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:38 smithi115 ceph-mon[72476]: pgmap v1334: 47 pgs: 47 active+clean; 6.5 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 938 op/s 2024-03-20T09:15:38.614 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:38.642 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:15:38.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:38.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:15:38.747 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:15:38.802 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:38.845 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:15:38.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:38.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:15:38.949 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:15:38.985 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:39.012 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:15:39.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:39.018 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:15:39.055 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:15:39.119 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:15:39.214 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:39.269 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:15:39.616 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:15:39.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:39.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:15:39.721 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:15:39.750 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:39.775 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:15:39.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:39.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:15:39.881 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:15:39.930 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:40.205 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 8, 10, 11, 3] out_osds: [9, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 5] 2024-03-20T09:15:40.206 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:15:40.206 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-20T09:15:40.206 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-20T09:15:40.580 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:40 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:15:40.580 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:40 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:15:40.580 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:40 smithi026 ceph-mon[68309]: osdmap e564: 12 total, 12 up, 10 in 2024-03-20T09:15:40.580 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:40 smithi026 ceph-mon[68309]: pgmap v1336: 47 pgs: 47 active+clean; 6.5 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 719 op/s 2024-03-20T09:15:40.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:40 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:15:40.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:40 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:15:40.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:40 smithi115 ceph-mon[72476]: osdmap e564: 12 total, 12 up, 10 in 2024-03-20T09:15:40.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:40 smithi115 ceph-mon[72476]: pgmap v1336: 47 pgs: 47 active+clean; 6.5 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 719 op/s 2024-03-20T09:15:40.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:40 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:15:40.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:40 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:15:40.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:40 smithi122 ceph-mon[73495]: osdmap e564: 12 total, 12 up, 10 in 2024-03-20T09:15:40.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:40 smithi122 ceph-mon[73495]: pgmap v1336: 47 pgs: 47 active+clean; 6.5 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 0 B/s wr, 719 op/s 2024-03-20T09:15:41.776 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:15:41.776 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:15:41.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.776 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25690.0:139994 4.7 4:e91fc955:::benchmark_data_smithi122_145698_object19075:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e564)", 2024-03-20T09:15:41.776 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:15:41.210207+0000", 2024-03-20T09:15:41.776 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.024866282999999999, 2024-03-20T09:15:41.776 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.024910315999999998, 2024-03-20T09:15:41.776 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:15:41.776 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:15:41.776 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:15:41.777 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25690", 2024-03-20T09:15:41.777 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:15:41.777 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 139994 2024-03-20T09:15:41.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.777 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:15:41.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.777 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:15:41.777 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.210207+0000", 2024-03-20T09:15:41.777 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:15:41.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.777 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:15:41.777 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.210207+0000", 2024-03-20T09:15:41.777 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:15:41.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.778 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:15:41.778 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.210209+0000", 2024-03-20T09:15:41.778 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.889e-06 2024-03-20T09:15:41.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.778 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:15:41.778 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.210215+0000", 2024-03-20T09:15:41.778 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5819999999999998e-06 2024-03-20T09:15:41.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.778 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:15:41.779 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.210229+0000", 2024-03-20T09:15:41.779 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4228e-05 2024-03-20T09:15:41.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.779 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:15:41.779 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.210238+0000", 2024-03-20T09:15:41.779 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6130000000000004e-06 2024-03-20T09:15:41.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.779 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:15:41.779 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.210293+0000", 2024-03-20T09:15:41.779 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5148000000000001e-05 2024-03-20T09:15:41.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.780 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:15:41.780 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.210429+0000", 2024-03-20T09:15:41.780 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000136079 2024-03-20T09:15:41.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.780 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.780 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:15:41.780 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.210544+0000", 2024-03-20T09:15:41.780 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000114338 2024-03-20T09:15:41.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.780 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.780 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:15:41.780 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.220928+0000", 2024-03-20T09:15:41.780 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010384894 2024-03-20T09:15:41.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.781 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:15:41.781 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.229978+0000", 2024-03-20T09:15:41.781 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0090498000000000002 2024-03-20T09:15:41.781 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:41.781 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:41.781 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:41.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.781 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25690.0:139996 4.7 4:ef744818:::benchmark_data_smithi122_145698_object19077:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e564)", 2024-03-20T09:15:41.781 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:15:41.210260+0000", 2024-03-20T09:15:41.781 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.024813214, 2024-03-20T09:15:41.781 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.024964777, 2024-03-20T09:15:41.781 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:15:41.782 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:15:41.782 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:15:41.782 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25690", 2024-03-20T09:15:41.782 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:15:41.782 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 139996 2024-03-20T09:15:41.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.782 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:15:41.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.782 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:15:41.782 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.210260+0000", 2024-03-20T09:15:41.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:15:41.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:15:41.783 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.210260+0000", 2024-03-20T09:15:41.783 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:15:41.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:15:41.783 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.210260+0000", 2024-03-20T09:15:41.783 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0999999999999999e-07 2024-03-20T09:15:41.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:15:41.783 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.210263+0000", 2024-03-20T09:15:41.783 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6630000000000002e-06 2024-03-20T09:15:41.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.784 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:15:41.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.210270+0000", 2024-03-20T09:15:41.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4200000000000001e-06 2024-03-20T09:15:41.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.784 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:15:41.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.210277+0000", 2024-03-20T09:15:41.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5660000000000003e-06 2024-03-20T09:15:41.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.784 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:15:41.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.217031+0000", 2024-03-20T09:15:41.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0067545699999999997 2024-03-20T09:15:41.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.785 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:15:41.785 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.217157+0000", 2024-03-20T09:15:41.785 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012605200000000001 2024-03-20T09:15:41.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.785 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:15:41.785 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.217267+0000", 2024-03-20T09:15:41.785 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010986500000000001 2024-03-20T09:15:41.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.785 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:15:41.786 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.222540+0000", 2024-03-20T09:15:41.786 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0052732409999999997 2024-03-20T09:15:41.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.786 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:15:41.786 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.229984+0000", 2024-03-20T09:15:41.786 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074437519999999997 2024-03-20T09:15:41.786 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:41.786 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:41.786 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:41.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.786 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25690.0:139997 4.4 4:28578021:::benchmark_data_smithi122_145698_object19078:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e564)", 2024-03-20T09:15:41.787 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:15:41.210284+0000", 2024-03-20T09:15:41.787 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.024789130999999999, 2024-03-20T09:15:41.787 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.025031968000000002, 2024-03-20T09:15:41.787 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:15:41.787 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:15:41.787 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:15:41.787 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25690", 2024-03-20T09:15:41.787 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:15:41.787 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 139997 2024-03-20T09:15:41.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.787 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:15:41.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:15:41.788 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.210284+0000", 2024-03-20T09:15:41.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:15:41.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.788 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:15:41.788 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.210284+0000", 2024-03-20T09:15:41.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:15:41.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.788 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:15:41.788 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.210284+0000", 2024-03-20T09:15:41.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3200000000000001e-07 2024-03-20T09:15:41.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.789 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:15:41.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.210286+0000", 2024-03-20T09:15:41.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0839999999999999e-06 2024-03-20T09:15:41.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.789 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:15:41.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.210292+0000", 2024-03-20T09:15:41.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4249999999999997e-06 2024-03-20T09:15:41.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.789 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:15:41.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.210296+0000", 2024-03-20T09:15:41.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.656e-06 2024-03-20T09:15:41.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.790 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:15:41.790 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.210350+0000", 2024-03-20T09:15:41.790 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3615000000000001e-05 2024-03-20T09:15:41.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.790 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:15:41.790 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.210500+0000", 2024-03-20T09:15:41.790 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015045100000000001 2024-03-20T09:15:41.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.790 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2,3", 2024-03-20T09:15:41.790 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.210620+0000", 2024-03-20T09:15:41.790 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011961 2024-03-20T09:15:41.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.791 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:15:41.791 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.234149+0000", 2024-03-20T09:15:41.791 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023529445 2024-03-20T09:15:41.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.791 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:15:41.791 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.234263+0000", 2024-03-20T09:15:41.791 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000114058 2024-03-20T09:15:41.791 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:41.791 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:41.791 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:41.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.792 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25690.0:140003 4.d 4:b0241ad8:::benchmark_data_smithi122_145698_object19084:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e564)", 2024-03-20T09:15:41.792 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:15:41.218958+0000", 2024-03-20T09:15:41.792 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.016115117000000002, 2024-03-20T09:15:41.792 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016448022, 2024-03-20T09:15:41.792 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:15:41.792 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:15:41.792 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:15:41.792 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25690", 2024-03-20T09:15:41.792 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:15:41.792 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 140003 2024-03-20T09:15:41.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.792 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:15:41.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.793 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:15:41.793 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.218958+0000", 2024-03-20T09:15:41.793 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:15:41.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.793 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:15:41.793 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.218958+0000", 2024-03-20T09:15:41.793 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:15:41.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.793 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:15:41.793 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.218959+0000", 2024-03-20T09:15:41.793 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5099999999999999e-06 2024-03-20T09:15:41.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.794 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:15:41.794 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.218964+0000", 2024-03-20T09:15:41.794 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2699999999999998e-06 2024-03-20T09:15:41.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.794 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:15:41.794 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.218976+0000", 2024-03-20T09:15:41.794 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2462e-05 2024-03-20T09:15:41.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.794 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:15:41.794 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.218984+0000", 2024-03-20T09:15:41.794 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1389999999999995e-06 2024-03-20T09:15:41.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.795 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:15:41.795 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.219118+0000", 2024-03-20T09:15:41.795 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133578 2024-03-20T09:15:41.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.795 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:15:41.795 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.219227+0000", 2024-03-20T09:15:41.795 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000109207 2024-03-20T09:15:41.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.795 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:15:41.795 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.219334+0000", 2024-03-20T09:15:41.796 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106649 2024-03-20T09:15:41.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.796 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:15:41.796 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.228548+0000", 2024-03-20T09:15:41.796 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092144940000000002 2024-03-20T09:15:41.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.796 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:15:41.796 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.234991+0000", 2024-03-20T09:15:41.796 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0064431970000000003 2024-03-20T09:15:41.796 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:41.796 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:41.796 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:41.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.797 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25690.0:140010 4.d 4:b974ae18:::benchmark_data_smithi122_145698_object19091:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e564)", 2024-03-20T09:15:41.797 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:15:41.230078+0000", 2024-03-20T09:15:41.797 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0049950530000000002, 2024-03-20T09:15:41.797 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054163939999999997, 2024-03-20T09:15:41.797 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:15:41.797 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:15:41.797 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:15:41.797 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25690", 2024-03-20T09:15:41.797 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:15:41.797 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 140010 2024-03-20T09:15:41.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.797 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:15:41.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.798 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:15:41.798 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.230078+0000", 2024-03-20T09:15:41.798 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:15:41.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.798 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:15:41.798 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.230078+0000", 2024-03-20T09:15:41.798 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:15:41.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.798 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:15:41.798 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.230079+0000", 2024-03-20T09:15:41.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.319e-06 2024-03-20T09:15:41.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:15:41.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.230085+0000", 2024-03-20T09:15:41.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.015e-06 2024-03-20T09:15:41.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:15:41.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.230098+0000", 2024-03-20T09:15:41.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.258e-05 2024-03-20T09:15:41.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:15:41.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.230107+0000", 2024-03-20T09:15:41.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7499999999999992e-06 2024-03-20T09:15:41.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.800 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:15:41.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.230304+0000", 2024-03-20T09:15:41.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019784300000000001 2024-03-20T09:15:41.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.800 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:15:41.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.230416+0000", 2024-03-20T09:15:41.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011173 2024-03-20T09:15:41.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.800 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:15:41.801 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.230526+0000", 2024-03-20T09:15:41.801 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010987700000000001 2024-03-20T09:15:41.801 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:41.801 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:41.801 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:41.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.801 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.25690.0:140013 4.e e564/520 4:727e1831:::benchmark_data_smithi122_145698_object19094:head v 564'8755, mlcod=564'8753)", 2024-03-20T09:15:41.801 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:15:41.230125+0000", 2024-03-20T09:15:41.801 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0049479509999999999, 2024-03-20T09:15:41.801 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054447380000000002, 2024-03-20T09:15:41.801 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:15:41.801 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:15:41.801 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:15:41.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.802 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:15:41.802 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.230125+0000", 2024-03-20T09:15:41.802 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:15:41.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.802 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:15:41.802 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.230125+0000", 2024-03-20T09:15:41.802 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:15:41.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.802 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:15:41.802 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.230126+0000", 2024-03-20T09:15:41.802 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2900000000000003e-07 2024-03-20T09:15:41.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.803 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:15:41.803 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.230132+0000", 2024-03-20T09:15:41.803 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2580000000000001e-06 2024-03-20T09:15:41.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.803 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:15:41.803 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.230141+0000", 2024-03-20T09:15:41.803 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0410000000000003e-06 2024-03-20T09:15:41.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.803 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:15:41.803 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.230147+0000", 2024-03-20T09:15:41.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3300000000000004e-06 2024-03-20T09:15:41.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.804 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:15:41.804 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.230211+0000", 2024-03-20T09:15:41.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4270000000000006e-05 2024-03-20T09:15:41.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.804 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:15:41.804 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.230231+0000", 2024-03-20T09:15:41.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9386999999999999e-05 2024-03-20T09:15:41.804 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:41.804 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:41.804 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:41.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.805 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25690.0:140011 4.d 4:ba4f9d62:::benchmark_data_smithi122_145698_object19092:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e564)", 2024-03-20T09:15:41.805 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:15:41.230276+0000", 2024-03-20T09:15:41.805 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.004796687, 2024-03-20T09:15:41.805 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0053576680000000003, 2024-03-20T09:15:41.805 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:15:41.805 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:15:41.805 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:15:41.805 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25690", 2024-03-20T09:15:41.805 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:15:41.805 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 140011 2024-03-20T09:15:41.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.806 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:15:41.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:15:41.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.230276+0000", 2024-03-20T09:15:41.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:15:41.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:15:41.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.230276+0000", 2024-03-20T09:15:41.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:15:41.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:15:41.807 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.230277+0000", 2024-03-20T09:15:41.807 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-03-20T09:15:41.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.807 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:15:41.807 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.230280+0000", 2024-03-20T09:15:41.807 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3050000000000001e-06 2024-03-20T09:15:41.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.807 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:15:41.807 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.230288+0000", 2024-03-20T09:15:41.807 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1750000000000005e-06 2024-03-20T09:15:41.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.808 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:15:41.808 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.230295+0000", 2024-03-20T09:15:41.808 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2269999999999998e-06 2024-03-20T09:15:41.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.808 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:15:41.808 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.235083+0000", 2024-03-20T09:15:41.808 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0047888419999999998 2024-03-20T09:15:41.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.808 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:15:41.808 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.235168+0000", 2024-03-20T09:15:41.808 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4464999999999999e-05 2024-03-20T09:15:41.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.809 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:15:41.809 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.235245+0000", 2024-03-20T09:15:41.809 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6736999999999994e-05 2024-03-20T09:15:41.809 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:41.809 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:41.809 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:41.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.809 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.25690.0:140014 4.9 e564/522 4:9b551bc3:::benchmark_data_smithi122_145698_object19095:head v 564'12544, mlcod=564'12542)", 2024-03-20T09:15:41.809 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:15:41.234385+0000", 2024-03-20T09:15:41.809 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.00068747000000000003, 2024-03-20T09:15:41.809 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013227149999999999, 2024-03-20T09:15:41.809 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:15:41.809 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:15:41.810 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:15:41.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.810 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:15:41.810 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.234385+0000", 2024-03-20T09:15:41.810 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:15:41.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.810 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:15:41.810 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.234385+0000", 2024-03-20T09:15:41.810 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:15:41.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.810 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:15:41.810 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.234387+0000", 2024-03-20T09:15:41.811 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1069999999999999e-06 2024-03-20T09:15:41.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.811 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:15:41.811 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.234395+0000", 2024-03-20T09:15:41.811 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5189999999999992e-06 2024-03-20T09:15:41.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.811 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:15:41.811 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.234407+0000", 2024-03-20T09:15:41.811 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1693e-05 2024-03-20T09:15:41.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.812 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:15:41.812 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.234416+0000", 2024-03-20T09:15:41.812 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9269999999999994e-06 2024-03-20T09:15:41.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.812 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:15:41.812 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.235051+0000", 2024-03-20T09:15:41.812 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00063499900000000002 2024-03-20T09:15:41.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.812 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:15:41.812 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.235074+0000", 2024-03-20T09:15:41.812 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3380999999999999e-05 2024-03-20T09:15:41.812 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:41.813 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:41.813 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:41.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.813 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.25690.0:140015 4.e e564/520 4:79f88cbd:::benchmark_data_smithi122_145698_object19096:head v 564'8756, mlcod=564'8753)", 2024-03-20T09:15:41.813 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:15:41.234482+0000", 2024-03-20T09:15:41.813 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.00059061699999999999, 2024-03-20T09:15:41.813 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00128637, 2024-03-20T09:15:41.813 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:15:41.813 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:15:41.813 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:15:41.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.813 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:15:41.813 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.234482+0000", 2024-03-20T09:15:41.813 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:15:41.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.814 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:15:41.814 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.234482+0000", 2024-03-20T09:15:41.814 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:15:41.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.814 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:15:41.814 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.234483+0000", 2024-03-20T09:15:41.814 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.13e-07 2024-03-20T09:15:41.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.814 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:15:41.815 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.234489+0000", 2024-03-20T09:15:41.815 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.331e-06 2024-03-20T09:15:41.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.815 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:15:41.815 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.234499+0000", 2024-03-20T09:15:41.815 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7499999999999998e-06 2024-03-20T09:15:41.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.815 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:15:41.815 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.234505+0000", 2024-03-20T09:15:41.815 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1179999999999997e-06 2024-03-20T09:15:41.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.815 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:15:41.816 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.234685+0000", 2024-03-20T09:15:41.816 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018046800000000001 2024-03-20T09:15:41.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.816 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:15:41.816 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.234698+0000", 2024-03-20T09:15:41.816 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2880999999999999e-05 2024-03-20T09:15:41.816 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:41.816 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:41.816 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:41.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.816 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25690.0:139992 4.7 4:eef5f49a:::benchmark_data_smithi122_145698_object19073:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e564)", 2024-03-20T09:15:41.816 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:15:41.204504+0000", 2024-03-20T09:15:41.817 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.030568452999999999, 2024-03-20T09:15:41.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031334395000000001, 2024-03-20T09:15:41.817 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:15:41.817 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:15:41.817 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:15:41.817 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25690", 2024-03-20T09:15:41.817 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:15:41.817 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 139992 2024-03-20T09:15:41.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.817 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:15:41.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:15:41.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.204504+0000", 2024-03-20T09:15:41.818 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:15:41.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.818 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:15:41.818 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.204504+0000", 2024-03-20T09:15:41.818 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:15:41.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.818 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:15:41.818 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.204506+0000", 2024-03-20T09:15:41.818 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.091e-06 2024-03-20T09:15:41.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.819 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:15:41.819 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.204510+0000", 2024-03-20T09:15:41.819 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0430000000000001e-06 2024-03-20T09:15:41.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.819 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:15:41.819 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.204520+0000", 2024-03-20T09:15:41.819 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9359999999999994e-06 2024-03-20T09:15:41.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.819 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:15:41.819 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.204525+0000", 2024-03-20T09:15:41.819 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4029999999999998e-06 2024-03-20T09:15:41.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.820 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:15:41.820 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.204580+0000", 2024-03-20T09:15:41.820 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4957999999999998e-05 2024-03-20T09:15:41.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.820 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:15:41.820 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.204702+0000", 2024-03-20T09:15:41.820 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012188000000000001 2024-03-20T09:15:41.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.820 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:15:41.820 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.204814+0000", 2024-03-20T09:15:41.820 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011202 2024-03-20T09:15:41.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.821 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:15:41.821 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.209040+0000", 2024-03-20T09:15:41.821 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042263860000000004 2024-03-20T09:15:41.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:41.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:41.821 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:15:41.821 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:41.209158+0000", 2024-03-20T09:15:41.821 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000118232 2024-03-20T09:15:41.821 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:41.821 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:41.821 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:41.821 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:41.821 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:15:41.821 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 10 2024-03-20T09:15:41.822 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:15:42.020 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:15:42.031 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:42.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:15:42.132 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:15:42.162 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:42.186 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:15:42.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:42.190 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:15:42.291 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:15:42.348 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:41 smithi026 ceph-mon[68309]: pgmap v1337: 47 pgs: 47 active+clean; 6.3 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 919 op/s 2024-03-20T09:15:42.361 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:42.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:41 smithi115 ceph-mon[72476]: pgmap v1337: 47 pgs: 47 active+clean; 6.3 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 919 op/s 2024-03-20T09:15:42.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:41 smithi122 ceph-mon[73495]: pgmap v1337: 47 pgs: 47 active+clean; 6.3 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 919 op/s 2024-03-20T09:15:42.512 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:15:42.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:42.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:15:42.548 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-20T09:15:42.624 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:15:42.701 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:42.985 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:15:42.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:42.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:15:43.074 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:15:43.084 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:15:43.094 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:15:43.120 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:43.143 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:15:43.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:43.147 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:15:43.156 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:15:43.248 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:15:43.346 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=629151230,config=3599132250,health=179246142,kv=1264228815}) 2024-03-20T09:15:43.346 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {kv=5,logm=95} crc {kv=3475053005,logm=158382774}) 2024-03-20T09:15:43.346 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1497751906}) 2024-03-20T09:15:43.347 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3586660124}) 2024-03-20T09:15:43.347 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3667533063}) 2024-03-20T09:15:43.347 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=69263601}) 2024-03-20T09:15:43.347 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=302591335}) 2024-03-20T09:15:43.347 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2663088715}) 2024-03-20T09:15:43.347 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4191880488}) 2024-03-20T09:15:43.347 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2111152748}) 2024-03-20T09:15:43.347 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4091942309}) 2024-03-20T09:15:43.347 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2055233650}) 2024-03-20T09:15:43.347 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2106457045}) 2024-03-20T09:15:43.347 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3942070632}) 2024-03-20T09:15:43.347 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3279974661}) 2024-03-20T09:15:43.347 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3990014061}) 2024-03-20T09:15:43.347 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=77242084}) 2024-03-20T09:15:43.348 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=164596583}) 2024-03-20T09:15:43.348 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1730248992}) 2024-03-20T09:15:43.348 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=758147730}) 2024-03-20T09:15:43.348 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1466616538}) 2024-03-20T09:15:43.348 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=752526762}) 2024-03-20T09:15:43.348 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1281560372}) 2024-03-20T09:15:43.348 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=338648910}) 2024-03-20T09:15:43.348 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1827788708}) 2024-03-20T09:15:43.348 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=892700008}) 2024-03-20T09:15:43.348 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3142849952}) 2024-03-20T09:15:43.348 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=486814348}) 2024-03-20T09:15:43.348 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3061998553}) 2024-03-20T09:15:43.348 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2443147010}) 2024-03-20T09:15:43.348 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2233510328}) 2024-03-20T09:15:43.349 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1592837106}) 2024-03-20T09:15:43.349 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1558172663}) 2024-03-20T09:15:43.349 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869672621}) 2024-03-20T09:15:43.349 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1706839778}) 2024-03-20T09:15:43.349 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4021484978}) 2024-03-20T09:15:43.349 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3290597120}) 2024-03-20T09:15:43.349 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4178762496}) 2024-03-20T09:15:43.349 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2183112551}) 2024-03-20T09:15:43.349 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=801113066}) 2024-03-20T09:15:43.349 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=687166872}) 2024-03-20T09:15:43.349 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1921137764}) 2024-03-20T09:15:43.349 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3391595682}) 2024-03-20T09:15:43.349 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=573983314}) 2024-03-20T09:15:43.349 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1782107662}) 2024-03-20T09:15:43.350 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1269558609}) 2024-03-20T09:15:43.350 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1743549999}) 2024-03-20T09:15:43.350 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2049346706}) 2024-03-20T09:15:43.350 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=74,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=2} crc {logm=929780676,mdsmap=3984425268,mgr=427508465,mgr_command_descs=3831600018,mgr_metadata=2702834764,mgrstat=751331749}) 2024-03-20T09:15:43.350 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=12,mon_config_key=88} crc {mgrstat=1220086362,mon_config_key=515194570}) 2024-03-20T09:15:43.350 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2670031428}) 2024-03-20T09:15:43.350 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=90,monmap=5,osd_metadata=5} crc {mon_config_key=1308601562,monmap=3481182261,osd_metadata=2990638787}) 2024-03-20T09:15:43.350 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=7,osd_pg_creating=1,osdmap=92} crc {osd_metadata=1401827842,osd_pg_creating=325057191,osdmap=2539980398}) 2024-03-20T09:15:43.350 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=27} crc {osdmap=1846764003}) 2024-03-20T09:15:43.350 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=629151230,config=3599132250,health=179246142,kv=1264228815}) 2024-03-20T09:15:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {kv=5,logm=95} crc {kv=3475053005,logm=158382774}) 2024-03-20T09:15:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1497751906}) 2024-03-20T09:15:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3586660124}) 2024-03-20T09:15:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3667533063}) 2024-03-20T09:15:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=69263601}) 2024-03-20T09:15:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=302591335}) 2024-03-20T09:15:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2663088715}) 2024-03-20T09:15:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4191880488}) 2024-03-20T09:15:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2111152748}) 2024-03-20T09:15:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4091942309}) 2024-03-20T09:15:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2055233650}) 2024-03-20T09:15:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2106457045}) 2024-03-20T09:15:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3942070632}) 2024-03-20T09:15:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3279974661}) 2024-03-20T09:15:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3990014061}) 2024-03-20T09:15:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=77242084}) 2024-03-20T09:15:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=164596583}) 2024-03-20T09:15:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1730248992}) 2024-03-20T09:15:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=758147730}) 2024-03-20T09:15:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1466616538}) 2024-03-20T09:15:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=752526762}) 2024-03-20T09:15:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1281560372}) 2024-03-20T09:15:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=338648910}) 2024-03-20T09:15:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1827788708}) 2024-03-20T09:15:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=892700008}) 2024-03-20T09:15:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3142849952}) 2024-03-20T09:15:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=486814348}) 2024-03-20T09:15:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3061998553}) 2024-03-20T09:15:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2443147010}) 2024-03-20T09:15:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2233510328}) 2024-03-20T09:15:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1592837106}) 2024-03-20T09:15:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1558172663}) 2024-03-20T09:15:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869672621}) 2024-03-20T09:15:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1706839778}) 2024-03-20T09:15:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4021484978}) 2024-03-20T09:15:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3290597120}) 2024-03-20T09:15:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4178762496}) 2024-03-20T09:15:43.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2183112551}) 2024-03-20T09:15:43.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=801113066}) 2024-03-20T09:15:43.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=687166872}) 2024-03-20T09:15:43.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1921137764}) 2024-03-20T09:15:43.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3391595682}) 2024-03-20T09:15:43.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=573983314}) 2024-03-20T09:15:43.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1782107662}) 2024-03-20T09:15:43.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1269558609}) 2024-03-20T09:15:43.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1743549999}) 2024-03-20T09:15:43.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2049346706}) 2024-03-20T09:15:43.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=74,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=2} crc {logm=929780676,mdsmap=3984425268,mgr=427508465,mgr_command_descs=3831600018,mgr_metadata=2702834764,mgrstat=751331749}) 2024-03-20T09:15:43.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=12,mon_config_key=88} crc {mgrstat=1220086362,mon_config_key=515194570}) 2024-03-20T09:15:43.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2670031428}) 2024-03-20T09:15:43.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=90,monmap=5,osd_metadata=5} crc {mon_config_key=1308601562,monmap=3481182261,osd_metadata=2990638787}) 2024-03-20T09:15:43.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=7,osd_pg_creating=1,osdmap=92} crc {osd_metadata=1401827842,osd_pg_creating=325057191,osdmap=2539980398}) 2024-03-20T09:15:43.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=27} crc {osdmap=1846764003}) 2024-03-20T09:15:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=629151230,config=3599132250,health=179246142,kv=1264228815}) 2024-03-20T09:15:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {kv=5,logm=95} crc {kv=3475053005,logm=158382774}) 2024-03-20T09:15:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1497751906}) 2024-03-20T09:15:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3586660124}) 2024-03-20T09:15:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3667533063}) 2024-03-20T09:15:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=69263601}) 2024-03-20T09:15:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=302591335}) 2024-03-20T09:15:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2663088715}) 2024-03-20T09:15:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4191880488}) 2024-03-20T09:15:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2111152748}) 2024-03-20T09:15:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4091942309}) 2024-03-20T09:15:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2055233650}) 2024-03-20T09:15:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2106457045}) 2024-03-20T09:15:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3942070632}) 2024-03-20T09:15:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3279974661}) 2024-03-20T09:15:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3990014061}) 2024-03-20T09:15:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=77242084}) 2024-03-20T09:15:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=164596583}) 2024-03-20T09:15:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1730248992}) 2024-03-20T09:15:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=758147730}) 2024-03-20T09:15:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1466616538}) 2024-03-20T09:15:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=752526762}) 2024-03-20T09:15:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1281560372}) 2024-03-20T09:15:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=338648910}) 2024-03-20T09:15:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1827788708}) 2024-03-20T09:15:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=892700008}) 2024-03-20T09:15:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3142849952}) 2024-03-20T09:15:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=486814348}) 2024-03-20T09:15:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3061998553}) 2024-03-20T09:15:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2443147010}) 2024-03-20T09:15:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2233510328}) 2024-03-20T09:15:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1592837106}) 2024-03-20T09:15:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1558172663}) 2024-03-20T09:15:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1869672621}) 2024-03-20T09:15:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1706839778}) 2024-03-20T09:15:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4021484978}) 2024-03-20T09:15:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3290597120}) 2024-03-20T09:15:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4178762496}) 2024-03-20T09:15:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2183112551}) 2024-03-20T09:15:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=801113066}) 2024-03-20T09:15:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=687166872}) 2024-03-20T09:15:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1921137764}) 2024-03-20T09:15:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3391595682}) 2024-03-20T09:15:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=573983314}) 2024-03-20T09:15:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1782107662}) 2024-03-20T09:15:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1269558609}) 2024-03-20T09:15:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1743549999}) 2024-03-20T09:15:43.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2049346706}) 2024-03-20T09:15:43.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=74,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=2} crc {logm=929780676,mdsmap=3984425268,mgr=427508465,mgr_command_descs=3831600018,mgr_metadata=2702834764,mgrstat=751331749}) 2024-03-20T09:15:43.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=12,mon_config_key=88} crc {mgrstat=1220086362,mon_config_key=515194570}) 2024-03-20T09:15:43.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2670031428}) 2024-03-20T09:15:43.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=90,monmap=5,osd_metadata=5} crc {mon_config_key=1308601562,monmap=3481182261,osd_metadata=2990638787}) 2024-03-20T09:15:43.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=7,osd_pg_creating=1,osdmap=92} crc {osd_metadata=1401827842,osd_pg_creating=325057191,osdmap=2539980398}) 2024-03-20T09:15:43.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=27} crc {osdmap=1846764003}) 2024-03-20T09:15:43.937 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:15:43.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:43.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:15:44.044 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:15:44.073 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:44.093 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:15:44.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:44.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:15:44.197 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:15:44.224 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:44 smithi026 ceph-mon[68309]: pgmap v1338: 47 pgs: 47 active+clean; 6.2 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 906 op/s 2024-03-20T09:15:44.295 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:44.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:44 smithi115 ceph-mon[72476]: pgmap v1338: 47 pgs: 47 active+clean; 6.2 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 906 op/s 2024-03-20T09:15:44.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:44 smithi122 ceph-mon[73495]: pgmap v1338: 47 pgs: 47 active+clean; 6.2 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 906 op/s 2024-03-20T09:15:45.352 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:15:45.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:45 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/944593790' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:15:45.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:45 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:15:45.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:45 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/944593790' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:15:45.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:45 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:15:45.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:45 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/944593790' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:15:45.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:45 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:15:46.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:46 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:15:46.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:46 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:15:46.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:46 smithi115 ceph-mon[72476]: osdmap e565: 12 total, 12 up, 10 in 2024-03-20T09:15:46.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:46 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:15:46.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:46 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:15:46.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:46 smithi115 ceph-mon[72476]: pgmap v1340: 47 pgs: 47 active+clean; 6.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 870 op/s 2024-03-20T09:15:46.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:46 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:15:46.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:46 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:15:46.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:46 smithi122 ceph-mon[73495]: osdmap e565: 12 total, 12 up, 10 in 2024-03-20T09:15:46.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:46 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:15:46.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:46 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:15:46.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:46 smithi122 ceph-mon[73495]: pgmap v1340: 47 pgs: 47 active+clean; 6.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 870 op/s 2024-03-20T09:15:46.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:46 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:15:46.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:46 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:15:46.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:46 smithi026 ceph-mon[68309]: osdmap e565: 12 total, 12 up, 10 in 2024-03-20T09:15:46.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:46 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:15:46.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:46 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:15:46.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:46 smithi026 ceph-mon[68309]: pgmap v1340: 47 pgs: 47 active+clean; 6.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 0 B/s wr, 870 op/s 2024-03-20T09:15:47.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:47 smithi115 ceph-mon[72476]: osdmap e566: 12 total, 12 up, 10 in 2024-03-20T09:15:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:47 smithi122 ceph-mon[73495]: osdmap e566: 12 total, 12 up, 10 in 2024-03-20T09:15:47.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:47 smithi026 ceph-mon[68309]: osdmap e566: 12 total, 12 up, 10 in 2024-03-20T09:15:47.527 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:15:47.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:47.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:15:47.637 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:15:47.665 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:47.692 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:15:47.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:47.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:15:47.797 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:15:47.870 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:48.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:48 smithi026 ceph-mon[68309]: 1.e scrub starts 2024-03-20T09:15:48.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:48 smithi026 ceph-mon[68309]: 1.e scrub ok 2024-03-20T09:15:48.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:48 smithi026 ceph-mon[68309]: pgmap v1342: 47 pgs: 47 active+clean; 6.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 1.06k op/s 2024-03-20T09:15:48.339 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:15:48.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:48.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:15:48.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:48 smithi115 ceph-mon[72476]: 1.e scrub starts 2024-03-20T09:15:48.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:48 smithi115 ceph-mon[72476]: 1.e scrub ok 2024-03-20T09:15:48.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:48 smithi115 ceph-mon[72476]: pgmap v1342: 47 pgs: 47 active+clean; 6.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 1.06k op/s 2024-03-20T09:15:48.445 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:15:48.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:48 smithi122 ceph-mon[73495]: 1.e scrub starts 2024-03-20T09:15:48.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:48 smithi122 ceph-mon[73495]: 1.e scrub ok 2024-03-20T09:15:48.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:48 smithi122 ceph-mon[73495]: pgmap v1342: 47 pgs: 47 active+clean; 6.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 1.06k op/s 2024-03-20T09:15:48.473 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:48.496 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:15:48.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:48.502 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:15:48.603 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:15:48.633 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:48.658 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:15:48.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:48.663 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:15:48.764 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:15:48.791 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:48.813 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:15:48.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:48.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:15:48.918 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:15:49.000 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:49.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:49 smithi122 ceph-mon[73495]: 1.6 scrub starts 2024-03-20T09:15:49.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:49 smithi122 ceph-mon[73495]: 1.6 scrub ok 2024-03-20T09:15:49.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:49 smithi026 ceph-mon[68309]: 1.6 scrub starts 2024-03-20T09:15:49.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:49 smithi026 ceph-mon[68309]: 1.6 scrub ok 2024-03-20T09:15:49.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:49 smithi115 ceph-mon[72476]: 1.6 scrub starts 2024-03-20T09:15:49.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:49 smithi115 ceph-mon[72476]: 1.6 scrub ok 2024-03-20T09:15:50.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:50 smithi026 ceph-mon[68309]: 1.f scrub starts 2024-03-20T09:15:50.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:50 smithi026 ceph-mon[68309]: 4.9 scrub starts 2024-03-20T09:15:50.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:50 smithi026 ceph-mon[68309]: 1.f scrub ok 2024-03-20T09:15:50.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:50 smithi026 ceph-mon[68309]: 4.f scrub starts 2024-03-20T09:15:50.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:50 smithi026 ceph-mon[68309]: pgmap v1343: 47 pgs: 47 active+clean; 6.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 784 op/s 2024-03-20T09:15:50.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:50 smithi115 ceph-mon[72476]: 1.f scrub starts 2024-03-20T09:15:50.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:50 smithi115 ceph-mon[72476]: 4.9 scrub starts 2024-03-20T09:15:50.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:50 smithi115 ceph-mon[72476]: 1.f scrub ok 2024-03-20T09:15:50.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:50 smithi115 ceph-mon[72476]: 4.f scrub starts 2024-03-20T09:15:50.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:50 smithi115 ceph-mon[72476]: pgmap v1343: 47 pgs: 47 active+clean; 6.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 784 op/s 2024-03-20T09:15:50.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:50 smithi122 ceph-mon[73495]: 1.f scrub starts 2024-03-20T09:15:50.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:50 smithi122 ceph-mon[73495]: 4.9 scrub starts 2024-03-20T09:15:50.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:50 smithi122 ceph-mon[73495]: 1.f scrub ok 2024-03-20T09:15:50.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:50 smithi122 ceph-mon[73495]: 4.f scrub starts 2024-03-20T09:15:50.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:50 smithi122 ceph-mon[73495]: pgmap v1343: 47 pgs: 47 active+clean; 6.0 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 784 op/s 2024-03-20T09:15:51.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:51 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:15:51.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:51 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:15:51.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:51 smithi026 ceph-mon[68309]: 4.4 scrub starts 2024-03-20T09:15:51.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:51 smithi026 ceph-mon[68309]: osdmap e567: 12 total, 12 up, 10 in 2024-03-20T09:15:51.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:51 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:15:51.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:51 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:15:51.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:51 smithi115 ceph-mon[72476]: 4.4 scrub starts 2024-03-20T09:15:51.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:51 smithi115 ceph-mon[72476]: osdmap e567: 12 total, 12 up, 10 in 2024-03-20T09:15:51.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:51 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:15:51.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:51 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:15:51.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:51 smithi122 ceph-mon[73495]: 4.4 scrub starts 2024-03-20T09:15:51.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:51 smithi122 ceph-mon[73495]: osdmap e567: 12 total, 12 up, 10 in 2024-03-20T09:15:52.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:52 smithi115 ceph-mon[72476]: 4.9 scrub ok 2024-03-20T09:15:52.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:52 smithi115 ceph-mon[72476]: 4.1 scrub starts 2024-03-20T09:15:52.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:52 smithi115 ceph-mon[72476]: 4.2 scrub starts 2024-03-20T09:15:52.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:52 smithi115 ceph-mon[72476]: 4.d scrub starts 2024-03-20T09:15:52.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:52 smithi115 ceph-mon[72476]: 4.2 scrub ok 2024-03-20T09:15:52.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:52 smithi115 ceph-mon[72476]: 4.f scrub ok 2024-03-20T09:15:52.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:52 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2949325616' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:15:52.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:52 smithi115 ceph-mon[72476]: pgmap v1345: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 796 op/s 2024-03-20T09:15:52.662 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:15:52.673 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:52 smithi026 ceph-mon[68309]: 4.9 scrub ok 2024-03-20T09:15:52.673 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:52 smithi026 ceph-mon[68309]: 4.1 scrub starts 2024-03-20T09:15:52.673 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:52 smithi026 ceph-mon[68309]: 4.2 scrub starts 2024-03-20T09:15:52.673 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:52 smithi026 ceph-mon[68309]: 4.d scrub starts 2024-03-20T09:15:52.673 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:52 smithi026 ceph-mon[68309]: 4.2 scrub ok 2024-03-20T09:15:52.673 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:52 smithi026 ceph-mon[68309]: 4.f scrub ok 2024-03-20T09:15:52.673 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:52 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2949325616' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:15:52.673 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:52 smithi026 ceph-mon[68309]: pgmap v1345: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 796 op/s 2024-03-20T09:15:52.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:52 smithi122 ceph-mon[73495]: 4.9 scrub ok 2024-03-20T09:15:52.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:52 smithi122 ceph-mon[73495]: 4.1 scrub starts 2024-03-20T09:15:52.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:52 smithi122 ceph-mon[73495]: 4.2 scrub starts 2024-03-20T09:15:52.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:52 smithi122 ceph-mon[73495]: 4.d scrub starts 2024-03-20T09:15:52.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:52 smithi122 ceph-mon[73495]: 4.2 scrub ok 2024-03-20T09:15:52.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:52 smithi122 ceph-mon[73495]: 4.f scrub ok 2024-03-20T09:15:52.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:52 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2949325616' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:15:52.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:52 smithi122 ceph-mon[73495]: pgmap v1345: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 796 op/s 2024-03-20T09:15:53.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:53 smithi115 ceph-mon[72476]: 4.1 scrub ok 2024-03-20T09:15:53.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:53 smithi115 ceph-mon[72476]: 1.1a scrub starts 2024-03-20T09:15:53.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:53 smithi115 ceph-mon[72476]: 1.1a scrub ok 2024-03-20T09:15:53.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:53 smithi122 ceph-mon[73495]: 4.1 scrub ok 2024-03-20T09:15:53.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:53 smithi122 ceph-mon[73495]: 1.1a scrub starts 2024-03-20T09:15:53.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:53 smithi122 ceph-mon[73495]: 1.1a scrub ok 2024-03-20T09:15:53.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:53 smithi026 ceph-mon[68309]: 4.1 scrub ok 2024-03-20T09:15:53.724 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:53 smithi026 ceph-mon[68309]: 1.1a scrub starts 2024-03-20T09:15:53.724 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:53 smithi026 ceph-mon[68309]: 1.1a scrub ok 2024-03-20T09:15:53.936 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:15:53.936 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:15:53.968 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:15:54.006 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:15:54.006 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:15:54.006 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-03-20T09:15:54.006 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-03-20T09:15:54.006 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:15:54.019 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:15:54.076 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:15:54.126 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:15:54.163 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:15:54.190 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:15:54.190 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:15:54.190 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 567, 2024-03-20T09:15:54.190 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "852fbb68-e696-11ee-95c9-87774f69a715", 2024-03-20T09:15:54.190 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-03-20T08:48:35.281088+0000", 2024-03-20T09:15:54.190 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-03-20T09:15:50.177264+0000", 2024-03-20T09:15:54.190 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2024-03-20T09:10:33.084151+0000", 2024-03-20T09:15:54.190 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2024-03-20T09:10:19.369629+0000", 2024-03-20T09:15:54.190 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-20T09:15:54.191 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5804032, 2024-03-20T09:15:54.191 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2024-03-20T09:15:54.191 INFO:teuthology.orchestra.run.smithi026.stdout: "nodeep-scrub", 2024-03-20T09:15:54.191 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2024-03-20T09:15:54.191 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2024-03-20T09:15:54.191 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2024-03-20T09:15:54.191 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2024-03-20T09:15:54.191 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:15:54.191 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 27, 2024-03-20T09:15:54.191 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2024-03-20T09:15:54.191 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-20T09:15:54.191 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-20T09:15:54.191 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2024-03-20T09:15:54.191 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 4, 2024-03-20T09:15:54.192 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 12, 2024-03-20T09:15:54.192 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "luminous", 2024-03-20T09:15:54.192 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "luminous", 2024-03-20T09:15:54.192 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "squid", 2024-03-20T09:15:54.192 INFO:teuthology.orchestra.run.smithi026.stdout: "allow_crimson": false, 2024-03-20T09:15:54.192 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2024-03-20T09:15:54.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.192 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2024-03-20T09:15:54.192 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": ".mgr", 2024-03-20T09:15:54.192 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-03-20T08:52:17.442324+0000", 2024-03-20T09:15:54.192 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-03-20T09:15:54.192 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-03-20T09:15:54.192 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-03-20T09:15:54.193 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 3, 2024-03-20T09:15:54.193 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 2, 2024-03-20T09:15:54.193 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-03-20T09:15:54.193 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-03-20T09:15:54.193 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-03-20T09:15:54.193 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-03-20T09:15:54.193 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-20T09:15:54.193 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-03-20T09:15:54.193 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-03-20T09:15:54.193 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 31, 2024-03-20T09:15:54.193 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 31, 2024-03-20T09:15:54.193 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 31, 2024-03-20T09:15:54.193 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 31, 2024-03-20T09:15:54.193 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 31, 2024-03-20T09:15:54.194 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-03-20T09:15:54.194 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-03-20T09:15:54.194 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-03-20T09:15:54.194 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-03-20T09:15:54.194 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-03-20T09:15:54.194 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-03-20T09:15:54.194 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-03-20T09:15:54.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.194 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "360", 2024-03-20T09:15:54.194 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-03-20T09:15:54.194 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-20T09:15:54.195 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "358", 2024-03-20T09:15:54.195 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-03-20T09:15:54.195 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-03-20T09:15:54.195 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-03-20T09:15:54.195 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-03-20T09:15:54.195 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-03-20T09:15:54.195 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-03-20T09:15:54.195 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-03-20T09:15:54.195 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-03-20T09:15:54.195 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-03-20T09:15:54.195 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-03-20T09:15:54.195 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-03-20T09:15:54.195 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-03-20T09:15:54.196 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-03-20T09:15:54.196 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-03-20T09:15:54.196 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-03-20T09:15:54.196 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-20T09:15:54.196 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-20T09:15:54.196 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-20T09:15:54.196 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-03-20T09:15:54.196 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-03-20T09:15:54.196 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-03-20T09:15:54.196 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-03-20T09:15:54.196 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-03-20T09:15:54.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.196 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-03-20T09:15:54.196 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-03-20T09:15:54.197 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-03-20T09:15:54.197 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-03-20T09:15:54.197 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-03-20T09:15:54.197 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-03-20T09:15:54.197 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-03-20T09:15:54.197 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-03-20T09:15:54.197 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-03-20T09:15:54.197 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-03-20T09:15:54.197 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-03-20T09:15:54.197 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2024-03-20T09:15:54.197 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_max": 32, 2024-03-20T09:15:54.197 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2024-03-20T09:15:54.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.198 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-03-20T09:15:54.198 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": {} 2024-03-20T09:15:54.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.198 INFO:teuthology.orchestra.run.smithi026.stdout: "read_balance": { 2024-03-20T09:15:54.198 INFO:teuthology.orchestra.run.smithi026.stdout: "score_type": "Fair distribution", 2024-03-20T09:15:54.198 INFO:teuthology.orchestra.run.smithi026.stdout: "score_acting": 2.3399999141693115, 2024-03-20T09:15:54.198 INFO:teuthology.orchestra.run.smithi026.stdout: "score_stable": 2.3399999141693115, 2024-03-20T09:15:54.198 INFO:teuthology.orchestra.run.smithi026.stdout: "optimal_score": 0.82999998331069946, 2024-03-20T09:15:54.198 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_acting": 1.940000057220459, 2024-03-20T09:15:54.198 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_stable": 1.940000057220459, 2024-03-20T09:15:54.198 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity_weighted": 0.92150616645812988, 2024-03-20T09:15:54.198 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity": 0.92000001668930054, 2024-03-20T09:15:54.198 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-03-20T09:15:54.199 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.199 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 4, 2024-03-20T09:15:54.199 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "unique_pool_2", 2024-03-20T09:15:54.199 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-03-20T09:13:20.690041+0000", 2024-03-20T09:15:54.199 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-03-20T09:15:54.199 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-03-20T09:15:54.199 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-03-20T09:15:54.199 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 3, 2024-03-20T09:15:54.199 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 2, 2024-03-20T09:15:54.199 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-03-20T09:15:54.199 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-03-20T09:15:54.199 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-03-20T09:15:54.199 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-03-20T09:15:54.200 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-20T09:15:54.200 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-03-20T09:15:54.200 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-03-20T09:15:54.200 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 16, 2024-03-20T09:15:54.200 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 16, 2024-03-20T09:15:54.200 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 16, 2024-03-20T09:15:54.200 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 16, 2024-03-20T09:15:54.200 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 16, 2024-03-20T09:15:54.200 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-03-20T09:15:54.200 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-03-20T09:15:54.200 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-03-20T09:15:54.200 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-03-20T09:15:54.200 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-03-20T09:15:54.201 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-03-20T09:15:54.201 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-03-20T09:15:54.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.201 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "527", 2024-03-20T09:15:54.201 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-03-20T09:15:54.201 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-20T09:15:54.201 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2024-03-20T09:15:54.201 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-03-20T09:15:54.201 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-03-20T09:15:54.201 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-03-20T09:15:54.201 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-03-20T09:15:54.201 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-03-20T09:15:54.201 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-03-20T09:15:54.202 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-03-20T09:15:54.202 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-03-20T09:15:54.202 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-03-20T09:15:54.202 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-03-20T09:15:54.202 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-03-20T09:15:54.202 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-03-20T09:15:54.202 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-03-20T09:15:54.202 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-03-20T09:15:54.202 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-03-20T09:15:54.202 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-20T09:15:54.202 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-20T09:15:54.203 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-20T09:15:54.203 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-03-20T09:15:54.203 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-03-20T09:15:54.203 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-03-20T09:15:54.203 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-03-20T09:15:54.203 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-03-20T09:15:54.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.203 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-03-20T09:15:54.203 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-03-20T09:15:54.203 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-03-20T09:15:54.203 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-03-20T09:15:54.203 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-03-20T09:15:54.203 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-03-20T09:15:54.204 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-03-20T09:15:54.204 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-03-20T09:15:54.204 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-03-20T09:15:54.204 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-03-20T09:15:54.204 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-03-20T09:15:54.204 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2024-03-20T09:15:54.204 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-03-20T09:15:54.204 INFO:teuthology.orchestra.run.smithi026.stdout: "rados": {} 2024-03-20T09:15:54.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.204 INFO:teuthology.orchestra.run.smithi026.stdout: "read_balance": { 2024-03-20T09:15:54.204 INFO:teuthology.orchestra.run.smithi026.stdout: "score_type": "Fair distribution", 2024-03-20T09:15:54.204 INFO:teuthology.orchestra.run.smithi026.stdout: "score_acting": 3.0099999904632568, 2024-03-20T09:15:54.204 INFO:teuthology.orchestra.run.smithi026.stdout: "score_stable": 3.0099999904632568, 2024-03-20T09:15:54.204 INFO:teuthology.orchestra.run.smithi026.stdout: "optimal_score": 0.82999998331069946, 2024-03-20T09:15:54.205 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_acting": 2.5, 2024-03-20T09:15:54.205 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_stable": 2.5, 2024-03-20T09:15:54.205 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity_weighted": 0.92150616645812988, 2024-03-20T09:15:54.205 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity": 0.92000001668930054, 2024-03-20T09:15:54.205 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-03-20T09:15:54.205 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.205 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.205 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:15:54.205 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:15:54.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.205 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-03-20T09:15:54.205 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "ad3fd9c0-316f-4fe5-a63b-d480087c1729", 2024-03-20T09:15:54.206 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:15:54.206 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:15:54.206 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:15:54.206 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:15:54.206 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:15:54.206 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:15:54.206 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 8, 2024-03-20T09:15:54.206 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 561, 2024-03-20T09:15:54.206 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:15:54.206 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:15:54.206 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:15:54.206 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.207 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.207 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2024-03-20T09:15:54.207 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:15:54.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.207 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.207 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2024-03-20T09:15:54.207 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:15:54.207 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.207 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.207 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:15:54.207 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.208 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.208 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2024-03-20T09:15:54.208 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:15:54.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.208 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.208 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2024-03-20T09:15:54.208 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:15:54.208 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.208 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.208 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:15:54.209 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.209 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.209 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2024-03-20T09:15:54.209 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:15:54.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.209 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.209 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2024-03-20T09:15:54.209 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:15:54.209 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.209 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.210 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:15:54.210 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.210 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.210 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2024-03-20T09:15:54.210 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:15:54.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.210 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.210 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2024-03-20T09:15:54.210 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:15:54.210 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.211 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.211 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/2139805293", 2024-03-20T09:15:54.211 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/2139805293", 2024-03-20T09:15:54.211 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/2139805293", 2024-03-20T09:15:54.211 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/2139805293", 2024-03-20T09:15:54.211 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:15:54.211 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:15:54.211 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:15:54.211 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.212 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-03-20T09:15:54.212 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e85b94f5-2954-47a1-9c20-096f2606eb04", 2024-03-20T09:15:54.212 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:15:54.212 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:15:54.212 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:15:54.212 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:15:54.212 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:15:54.212 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:15:54.212 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2024-03-20T09:15:54.212 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 533, 2024-03-20T09:15:54.212 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:15:54.212 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:15:54.212 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:15:54.212 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.213 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.213 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2024-03-20T09:15:54.213 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:15:54.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.213 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.213 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2024-03-20T09:15:54.213 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:15:54.213 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.213 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.213 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:15:54.214 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.214 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.214 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2024-03-20T09:15:54.214 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:15:54.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.214 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.214 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2024-03-20T09:15:54.214 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:15:54.214 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.214 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.215 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:15:54.215 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.215 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.215 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2024-03-20T09:15:54.215 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:15:54.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.215 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.215 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2024-03-20T09:15:54.215 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:15:54.215 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.216 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.216 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:15:54.216 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.216 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.216 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2024-03-20T09:15:54.216 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:15:54.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.216 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.216 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2024-03-20T09:15:54.216 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:15:54.217 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.217 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.217 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/3853355134", 2024-03-20T09:15:54.217 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/3853355134", 2024-03-20T09:15:54.217 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/3853355134", 2024-03-20T09:15:54.217 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/3853355134", 2024-03-20T09:15:54.217 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:15:54.217 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:15:54.217 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:15:54.217 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.218 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-03-20T09:15:54.218 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "dd2b10f0-741c-47a4-b224-ac6bfe2c14ef", 2024-03-20T09:15:54.218 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:15:54.218 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:15:54.218 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:15:54.218 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:15:54.218 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:15:54.218 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:15:54.218 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 18, 2024-03-20T09:15:54.218 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 534, 2024-03-20T09:15:54.218 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:15:54.218 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:15:54.219 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:15:54.219 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.219 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.219 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2024-03-20T09:15:54.219 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:15:54.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.219 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.219 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2024-03-20T09:15:54.219 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:15:54.219 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.219 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.220 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:15:54.220 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.220 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.220 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2024-03-20T09:15:54.220 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:15:54.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.220 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.220 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2024-03-20T09:15:54.220 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:15:54.220 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.220 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.221 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:15:54.221 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.221 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.221 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2024-03-20T09:15:54.221 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:15:54.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.221 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.221 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2024-03-20T09:15:54.221 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:15:54.221 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.221 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.222 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:15:54.222 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.222 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.222 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2024-03-20T09:15:54.222 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:15:54.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.222 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.222 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2024-03-20T09:15:54.222 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:15:54.222 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.223 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.223 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/3993357967", 2024-03-20T09:15:54.223 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/3993357967", 2024-03-20T09:15:54.223 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/3993357967", 2024-03-20T09:15:54.223 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/3993357967", 2024-03-20T09:15:54.223 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:15:54.223 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:15:54.223 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:15:54.223 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.223 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-03-20T09:15:54.224 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1", 2024-03-20T09:15:54.224 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:15:54.224 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:15:54.224 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:15:54.224 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:15:54.224 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:15:54.224 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:15:54.224 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 26, 2024-03-20T09:15:54.224 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 525, 2024-03-20T09:15:54.224 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:15:54.224 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:15:54.224 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:15:54.224 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.225 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.225 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2024-03-20T09:15:54.225 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:15:54.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.225 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.225 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2024-03-20T09:15:54.225 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:15:54.225 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.225 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.225 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:15:54.225 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.226 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.226 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2024-03-20T09:15:54.226 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:15:54.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.226 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.226 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2024-03-20T09:15:54.226 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:15:54.226 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.226 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.226 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:15:54.226 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.227 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.227 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2024-03-20T09:15:54.227 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:15:54.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.227 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.227 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2024-03-20T09:15:54.227 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:15:54.227 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.227 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.227 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:15:54.228 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.228 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.228 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2024-03-20T09:15:54.228 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:15:54.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.228 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.228 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2024-03-20T09:15:54.228 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:15:54.228 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.228 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.228 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/4110225509", 2024-03-20T09:15:54.229 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/4110225509", 2024-03-20T09:15:54.229 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/4110225509", 2024-03-20T09:15:54.229 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/4110225509", 2024-03-20T09:15:54.229 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:15:54.229 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:15:54.229 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:15:54.229 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.229 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-03-20T09:15:54.229 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "f14e34d5-9829-42a3-9b25-31a0976c5ef7", 2024-03-20T09:15:54.229 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:15:54.229 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 0, 2024-03-20T09:15:54.229 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-03-20T09:15:54.230 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:15:54.230 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:15:54.230 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:15:54.230 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 32, 2024-03-20T09:15:54.230 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 366, 2024-03-20T09:15:54.230 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:15:54.230 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:15:54.230 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:15:54.230 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.230 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.230 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6800", 2024-03-20T09:15:54.230 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:15:54.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.231 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.231 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6801", 2024-03-20T09:15:54.231 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:15:54.231 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.231 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.231 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:15:54.231 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.231 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.231 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6802", 2024-03-20T09:15:54.231 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:15:54.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.232 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.232 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6803", 2024-03-20T09:15:54.232 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:15:54.232 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.232 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.232 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:15:54.232 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.232 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.232 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6806", 2024-03-20T09:15:54.232 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:15:54.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.233 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.233 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6807", 2024-03-20T09:15:54.233 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:15:54.233 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.233 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.233 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:15:54.233 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.233 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.233 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6804", 2024-03-20T09:15:54.234 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:15:54.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.234 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.234 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6805", 2024-03-20T09:15:54.234 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:15:54.234 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.234 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.234 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6801/477923058", 2024-03-20T09:15:54.234 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6803/477923058", 2024-03-20T09:15:54.234 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6807/477923058", 2024-03-20T09:15:54.234 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6805/477923058", 2024-03-20T09:15:54.235 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:15:54.235 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:15:54.235 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:15:54.235 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.235 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-03-20T09:15:54.235 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "cb431b6a-540d-4318-ae35-d379e1df0185", 2024-03-20T09:15:54.235 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:15:54.235 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:15:54.235 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:15:54.235 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:15:54.235 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 41, 2024-03-20T09:15:54.235 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 273, 2024-03-20T09:15:54.236 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 443, 2024-03-20T09:15:54.236 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 520, 2024-03-20T09:15:54.236 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 274, 2024-03-20T09:15:54.236 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:15:54.236 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:15:54.236 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.236 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.236 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6808", 2024-03-20T09:15:54.236 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:15:54.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.236 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.237 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6809", 2024-03-20T09:15:54.237 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:15:54.237 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.237 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.237 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:15:54.237 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.237 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.237 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6810", 2024-03-20T09:15:54.237 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:15:54.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.237 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.237 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6811", 2024-03-20T09:15:54.238 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:15:54.238 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.238 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.238 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:15:54.238 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.238 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.238 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6814", 2024-03-20T09:15:54.238 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:15:54.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.239 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.239 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6815", 2024-03-20T09:15:54.239 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:15:54.239 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.239 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.239 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:15:54.239 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.239 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.239 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6812", 2024-03-20T09:15:54.239 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:15:54.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.240 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.240 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6813", 2024-03-20T09:15:54.240 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:15:54.240 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.240 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.240 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6809/650416648", 2024-03-20T09:15:54.240 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6811/650416648", 2024-03-20T09:15:54.240 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6815/650416648", 2024-03-20T09:15:54.241 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6813/650416648", 2024-03-20T09:15:54.241 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:15:54.241 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:15:54.241 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:15:54.241 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.241 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2024-03-20T09:15:54.241 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "753ed830-e764-4ff3-9d9b-87e7fa748b81", 2024-03-20T09:15:54.241 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:15:54.241 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:15:54.241 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:15:54.242 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:15:54.242 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:15:54.242 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:15:54.242 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 51, 2024-03-20T09:15:54.242 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 562, 2024-03-20T09:15:54.242 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:15:54.242 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:15:54.242 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:15:54.242 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.242 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.242 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6816", 2024-03-20T09:15:54.243 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:15:54.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.243 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.243 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6817", 2024-03-20T09:15:54.243 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:15:54.243 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.243 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.243 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:15:54.243 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.243 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.243 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6818", 2024-03-20T09:15:54.244 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:15:54.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.244 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.244 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6819", 2024-03-20T09:15:54.244 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:15:54.244 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.244 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.244 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:15:54.244 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.244 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.244 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6822", 2024-03-20T09:15:54.245 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:15:54.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.245 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.245 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6823", 2024-03-20T09:15:54.245 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:15:54.245 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.245 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.245 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:15:54.245 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.246 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.246 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6820", 2024-03-20T09:15:54.246 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:15:54.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.246 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.246 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6821", 2024-03-20T09:15:54.246 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:15:54.246 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.246 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.246 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6817/3278301843", 2024-03-20T09:15:54.247 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6819/3278301843", 2024-03-20T09:15:54.247 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6823/3278301843", 2024-03-20T09:15:54.247 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6821/3278301843", 2024-03-20T09:15:54.247 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:15:54.247 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:15:54.247 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:15:54.247 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.247 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2024-03-20T09:15:54.247 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "55bcbd37-7a6f-4ab1-b5a9-91a951444d86", 2024-03-20T09:15:54.247 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:15:54.247 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:15:54.248 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:15:54.248 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 0.058074951171875, 2024-03-20T09:15:54.248 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:15:54.248 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:15:54.248 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 60, 2024-03-20T09:15:54.248 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 517, 2024-03-20T09:15:54.248 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:15:54.248 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:15:54.248 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:15:54.248 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.248 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.248 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6824", 2024-03-20T09:15:54.249 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:15:54.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.249 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.249 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6825", 2024-03-20T09:15:54.249 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:15:54.249 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.249 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.249 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:15:54.249 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.249 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.249 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6826", 2024-03-20T09:15:54.250 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:15:54.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.250 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.250 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6827", 2024-03-20T09:15:54.250 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:15:54.250 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.250 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.250 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:15:54.250 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.251 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.251 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6830", 2024-03-20T09:15:54.251 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:15:54.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.251 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.251 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6831", 2024-03-20T09:15:54.251 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:15:54.251 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.251 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.251 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:15:54.251 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.252 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.252 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6828", 2024-03-20T09:15:54.252 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:15:54.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.252 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.252 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6829", 2024-03-20T09:15:54.252 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:15:54.252 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.252 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.252 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6825/3152424001", 2024-03-20T09:15:54.252 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6827/3152424001", 2024-03-20T09:15:54.253 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6831/3152424001", 2024-03-20T09:15:54.253 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6829/3152424001", 2024-03-20T09:15:54.253 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:15:54.253 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:15:54.253 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:15:54.253 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.253 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 8, 2024-03-20T09:15:54.253 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "bc86248e-6bd0-4369-b8a2-e593bc2a610d", 2024-03-20T09:15:54.253 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:15:54.253 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:15:54.253 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:15:54.254 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:15:54.254 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:15:54.254 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:15:54.254 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 70, 2024-03-20T09:15:54.254 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 496, 2024-03-20T09:15:54.254 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:15:54.254 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:15:54.254 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:15:54.254 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.254 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.254 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6800", 2024-03-20T09:15:54.255 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:15:54.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.255 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.255 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6801", 2024-03-20T09:15:54.255 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:15:54.255 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.255 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.255 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:15:54.255 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.255 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.256 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6802", 2024-03-20T09:15:54.256 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:15:54.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.256 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.256 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6803", 2024-03-20T09:15:54.256 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:15:54.256 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.256 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.256 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:15:54.256 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.256 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.257 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6806", 2024-03-20T09:15:54.257 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:15:54.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.257 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.257 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6807", 2024-03-20T09:15:54.257 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:15:54.257 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.257 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.257 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:15:54.257 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.258 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.258 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6804", 2024-03-20T09:15:54.258 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:15:54.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.258 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.258 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6805", 2024-03-20T09:15:54.258 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:15:54.258 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.258 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.259 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6801/1309426740", 2024-03-20T09:15:54.259 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6803/1309426740", 2024-03-20T09:15:54.259 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6807/1309426740", 2024-03-20T09:15:54.259 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6805/1309426740", 2024-03-20T09:15:54.259 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:15:54.259 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:15:54.259 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:15:54.259 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.259 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 9, 2024-03-20T09:15:54.259 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b20a4aa-032a-4c09-ac75-153c554424e7", 2024-03-20T09:15:54.259 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:15:54.260 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 0, 2024-03-20T09:15:54.260 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-03-20T09:15:54.260 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:15:54.260 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:15:54.260 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:15:54.260 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 80, 2024-03-20T09:15:54.260 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 257, 2024-03-20T09:15:54.260 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:15:54.260 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:15:54.260 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:15:54.260 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.260 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.261 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6808", 2024-03-20T09:15:54.261 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:15:54.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.261 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.261 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6809", 2024-03-20T09:15:54.261 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:15:54.261 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.261 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.261 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:15:54.261 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.262 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.262 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6810", 2024-03-20T09:15:54.262 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:15:54.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.262 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.262 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6811", 2024-03-20T09:15:54.262 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:15:54.262 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.262 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.262 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:15:54.262 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.263 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.263 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6814", 2024-03-20T09:15:54.263 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:15:54.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.263 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.263 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6815", 2024-03-20T09:15:54.263 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:15:54.263 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.263 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.263 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:15:54.263 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.264 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.264 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6812", 2024-03-20T09:15:54.264 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:15:54.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.264 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.264 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6813", 2024-03-20T09:15:54.264 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:15:54.264 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.264 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.264 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6809/2953210076", 2024-03-20T09:15:54.264 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6811/2953210076", 2024-03-20T09:15:54.264 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6815/2953210076", 2024-03-20T09:15:54.265 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6813/2953210076", 2024-03-20T09:15:54.265 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:15:54.265 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:15:54.265 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:15:54.265 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.265 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 10, 2024-03-20T09:15:54.265 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e62c1fb3-5dd1-4d0d-997c-d20633d19361", 2024-03-20T09:15:54.265 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:15:54.265 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:15:54.265 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:15:54.265 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:15:54.266 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:15:54.266 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:15:54.266 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 89, 2024-03-20T09:15:54.266 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 520, 2024-03-20T09:15:54.266 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:15:54.266 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:15:54.266 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:15:54.266 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.266 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.266 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6816", 2024-03-20T09:15:54.266 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:15:54.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.267 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.267 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6817", 2024-03-20T09:15:54.267 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:15:54.267 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.267 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.267 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:15:54.267 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.267 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.267 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6818", 2024-03-20T09:15:54.267 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:15:54.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.268 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.268 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6819", 2024-03-20T09:15:54.268 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:15:54.268 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.268 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.268 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:15:54.268 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.268 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.268 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6822", 2024-03-20T09:15:54.268 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:15:54.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.269 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.269 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6823", 2024-03-20T09:15:54.269 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:15:54.269 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.269 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.269 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:15:54.269 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.270 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.270 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6820", 2024-03-20T09:15:54.270 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:15:54.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.270 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.270 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6821", 2024-03-20T09:15:54.270 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:15:54.270 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.270 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.270 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6817/409394340", 2024-03-20T09:15:54.270 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6819/409394340", 2024-03-20T09:15:54.271 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6823/409394340", 2024-03-20T09:15:54.271 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6821/409394340", 2024-03-20T09:15:54.271 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:15:54.271 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:15:54.271 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:15:54.271 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.271 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 11, 2024-03-20T09:15:54.271 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b83994d5-e5ef-48b0-9758-c26843cec3b6", 2024-03-20T09:15:54.271 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:15:54.271 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:15:54.271 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:15:54.272 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:15:54.272 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:15:54.272 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:15:54.272 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 96, 2024-03-20T09:15:54.272 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 522, 2024-03-20T09:15:54.272 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:15:54.272 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:15:54.272 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:15:54.272 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.272 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.272 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6824", 2024-03-20T09:15:54.272 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:15:54.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.273 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.273 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6825", 2024-03-20T09:15:54.273 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:15:54.273 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.273 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.273 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:15:54.273 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.273 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.273 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6826", 2024-03-20T09:15:54.273 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:15:54.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.274 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.274 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6827", 2024-03-20T09:15:54.274 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:15:54.274 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.274 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.274 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:15:54.274 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.274 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.274 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6830", 2024-03-20T09:15:54.275 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:15:54.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.275 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.275 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6831", 2024-03-20T09:15:54.275 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:15:54.275 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.275 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.275 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:15:54.275 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:15:54.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.275 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:15:54.276 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6828", 2024-03-20T09:15:54.276 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:15:54.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.276 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:15:54.276 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6829", 2024-03-20T09:15:54.276 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:15:54.276 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.276 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.276 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6825/2704069354", 2024-03-20T09:15:54.276 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6827/2704069354", 2024-03-20T09:15:54.276 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6831/2704069354", 2024-03-20T09:15:54.276 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6829/2704069354", 2024-03-20T09:15:54.277 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:15:54.277 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:15:54.277 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:15:54.277 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.277 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.277 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:15:54.277 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2024-03-20T09:15:54.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.277 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-03-20T09:15:54.277 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:15:54.277 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:15:54.277 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:15:54.277 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:15:54.277 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:15:54.278 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:15:15.507934+0000", 2024-03-20T09:15:54.278 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:15:54.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.278 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-03-20T09:15:54.278 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:15:54.278 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:15:54.278 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:15:54.278 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:15:54.278 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:15:54.278 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:15:07.459472+0000", 2024-03-20T09:15:54.278 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:15:54.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.279 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-03-20T09:15:54.279 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:15:54.279 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:15:54.279 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:15:54.279 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:15:54.279 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:15:54.279 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:15:38.999217+0000", 2024-03-20T09:15:54.279 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:15:54.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.279 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-03-20T09:15:54.279 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:15:54.279 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:15:54.279 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:15:54.280 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:15:54.280 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:15:54.280 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:15:17.274583+0000", 2024-03-20T09:15:54.280 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:15:54.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.280 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-03-20T09:15:54.280 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:15:54.280 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:15:54.280 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:15:54.280 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:15:54.280 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 65536, 2024-03-20T09:15:54.280 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:15:45.509487+0000", 2024-03-20T09:15:54.280 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:15:54.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.281 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-03-20T09:15:54.281 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:15:54.281 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:15:54.281 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:15:54.281 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:15:54.281 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:15:54.281 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:15:49.636448+0000", 2024-03-20T09:15:54.281 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 273 2024-03-20T09:15:54.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.281 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2024-03-20T09:15:54.281 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:15:54.282 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:15:54.282 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:15:54.282 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:15:54.282 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:15:54.282 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:14:32.283694+0000", 2024-03-20T09:15:54.282 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:15:54.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.282 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2024-03-20T09:15:54.282 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:15:54.282 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:15:54.282 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:15:54.282 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:15:54.283 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:15:54.283 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:14:48.028594+0000", 2024-03-20T09:15:54.283 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:15:54.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.283 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 8, 2024-03-20T09:15:54.283 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:15:54.283 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:15:54.283 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:15:54.283 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:15:54.283 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:15:54.283 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:14:50.225887+0000", 2024-03-20T09:15:54.283 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:15:54.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.284 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 9, 2024-03-20T09:15:54.284 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:15:54.284 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:15:54.284 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:15:54.284 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:15:54.284 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 34736, 2024-03-20T09:15:54.284 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:15:05.229610+0000", 2024-03-20T09:15:54.284 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:15:54.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.284 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 10, 2024-03-20T09:15:54.284 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:15:54.285 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:15:54.285 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:15:54.285 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:15:54.285 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:15:54.285 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:15:10.066125+0000", 2024-03-20T09:15:54.285 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:15:54.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.285 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 11, 2024-03-20T09:15:54.285 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:15:54.285 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:15:54.285 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:15:54.285 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:15:54.285 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:15:54.285 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:14:57.699566+0000", 2024-03-20T09:15:54.286 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:15:54.286 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.286 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:15:54.286 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [ 2024-03-20T09:15:54.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.286 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "1.1", 2024-03-20T09:15:54.286 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:15:54.286 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-03-20T09:15:54.286 INFO:teuthology.orchestra.run.smithi026.stdout: 11, 2024-03-20T09:15:54.286 INFO:teuthology.orchestra.run.smithi026.stdout: 5 2024-03-20T09:15:54.286 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.286 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.286 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:15:54.287 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [ 2024-03-20T09:15:54.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.287 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "4.f", 2024-03-20T09:15:54.287 INFO:teuthology.orchestra.run.smithi026.stdout: "mappings": [ 2024-03-20T09:15:54.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:15:54.287 INFO:teuthology.orchestra.run.smithi026.stdout: "from": 1, 2024-03-20T09:15:54.287 INFO:teuthology.orchestra.run.smithi026.stdout: "to": 2 2024-03-20T09:15:54.287 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.287 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:15:54.287 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.287 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:15:54.287 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_primaries": [], 2024-03-20T09:15:54.287 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2024-03-20T09:15:54.287 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2024-03-20T09:15:54.288 INFO:teuthology.orchestra.run.smithi026.stdout: "blocklist": { 2024-03-20T09:15:54.288 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/4291404191": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:15:54.288 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/642012566": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:15:54.288 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6801/1181459999": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:15:54.288 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/3984513952": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:15:54.288 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6800/1654945498": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:15:54.288 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6800/1181459999": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:15:54.288 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/137770981": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:15:54.288 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/1451237762": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:15:54.288 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/3938386631": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:15:54.288 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6801/1654945498": "2024-03-21T08:49:04.596754+0000" 2024-03-20T09:15:54.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.288 INFO:teuthology.orchestra.run.smithi026.stdout: "range_blocklist": {}, 2024-03-20T09:15:54.288 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2024-03-20T09:15:54.288 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2024-03-20T09:15:54.289 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2024-03-20T09:15:54.289 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2024-03-20T09:15:54.289 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2024-03-20T09:15:54.289 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2024-03-20T09:15:54.289 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2024-03-20T09:15:54.289 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:15:54.289 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2024-03-20T09:15:54.289 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2024-03-20T09:15:54.289 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2024-03-20T09:15:54.289 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2024-03-20T09:15:54.289 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {}, 2024-03-20T09:15:54.289 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": { 2024-03-20T09:15:54.289 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_enabled": false, 2024-03-20T09:15:54.289 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_bucket_count": 0, 2024-03-20T09:15:54.290 INFO:teuthology.orchestra.run.smithi026.stdout: "degraded_stretch_mode": 0, 2024-03-20T09:15:54.290 INFO:teuthology.orchestra.run.smithi026.stdout: "recovering_stretch_mode": 0, 2024-03-20T09:15:54.290 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_bucket": 0 2024-03-20T09:15:54.290 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:15:54.290 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:15:54.290 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:15:54.461 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:15:54.508 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:15:54.529 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:54 smithi026 ceph-mon[68309]: 4.4 scrub ok 2024-03-20T09:15:54.529 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:54 smithi026 ceph-mon[68309]: 4.d scrub ok 2024-03-20T09:15:54.529 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:54 smithi026 ceph-mon[68309]: 1.a scrub starts 2024-03-20T09:15:54.529 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:54 smithi026 ceph-mon[68309]: 1.a scrub ok 2024-03-20T09:15:54.529 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:54 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1968445618' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:15:54.551 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:15:54.599 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:15:54.625 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:15:54.625 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:15:54.625 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:15:54.625 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:15:54.625 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:15:54.625 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:15:54.625 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:15:54.625 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:15:54.625 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:15:54.625 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:15:54.625 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:15:54.625 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:15:54.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:54 smithi115 ceph-mon[72476]: 4.4 scrub ok 2024-03-20T09:15:54.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:54 smithi115 ceph-mon[72476]: 4.d scrub ok 2024-03-20T09:15:54.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:54 smithi115 ceph-mon[72476]: 1.a scrub starts 2024-03-20T09:15:54.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:54 smithi115 ceph-mon[72476]: 1.a scrub ok 2024-03-20T09:15:54.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:54 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1968445618' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:15:54.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:54 smithi122 ceph-mon[73495]: 4.4 scrub ok 2024-03-20T09:15:54.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:54 smithi122 ceph-mon[73495]: 4.d scrub ok 2024-03-20T09:15:54.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:54 smithi122 ceph-mon[73495]: 1.a scrub starts 2024-03-20T09:15:54.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:54 smithi122 ceph-mon[73495]: 1.a scrub ok 2024-03-20T09:15:54.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:54 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1968445618' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:15:55.067 INFO:tasks.thrashosds.thrasher:j is {'epoch': 567, 'fsid': '852fbb68-e696-11ee-95c9-87774f69a715', 'created': '2024-03-20T08:48:35.281088+0000', 'modified': '2024-03-20T09:15:50.177264+0000', 'last_up_change': '2024-03-20T09:10:33.084151+0000', 'last_in_change': '2024-03-20T09:10:19.369629+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-20T08:52:17.442324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '360', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '358', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.3399999141693115, 'score_stable': 2.3399999141693115, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 1.940000057220459, 'raw_score_stable': 1.940000057220459, 'primary_affinity_weighted': 0.9215061664581299, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 4, 'pool_name': 'unique_pool_2', 'create_time': '2024-03-20T09:13:20.690041+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '527', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.009999990463257, 'score_stable': 3.009999990463257, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.9215061664581299, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}], 'osds': [{'osd': 0, 'uuid': 'ad3fd9c0-316f-4fe5-a63b-d480087c1729', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 561, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6802', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6803', 'nonce': 2139805293}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6804', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6805', 'nonce': 2139805293}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6808', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6809', 'nonce': 2139805293}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6806', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6807', 'nonce': 2139805293}]}, 'public_addr': '172.21.15.26:6803/2139805293', 'cluster_addr': '172.21.15.26:6805/2139805293', 'heartbeat_back_addr': '172.21.15.26:6809/2139805293', 'heartbeat_front_addr': '172.21.15.26:6807/2139805293', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'e85b94f5-2954-47a1-9c20-096f2606eb04', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 533, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6810', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6811', 'nonce': 3853355134}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6812', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6813', 'nonce': 3853355134}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6816', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6817', 'nonce': 3853355134}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6814', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6815', 'nonce': 3853355134}]}, 'public_addr': '172.21.15.26:6811/3853355134', 'cluster_addr': '172.21.15.26:6813/3853355134', 'heartbeat_back_addr': '172.21.15.26:6817/3853355134', 'heartbeat_front_addr': '172.21.15.26:6815/3853355134', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'dd2b10f0-741c-47a4-b224-ac6bfe2c14ef', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 534, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6818', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6819', 'nonce': 3993357967}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6820', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6821', 'nonce': 3993357967}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6824', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6825', 'nonce': 3993357967}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6822', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6823', 'nonce': 3993357967}]}, 'public_addr': '172.21.15.26:6819/3993357967', 'cluster_addr': '172.21.15.26:6821/3993357967', 'heartbeat_back_addr': '172.21.15.26:6825/3993357967', 'heartbeat_front_addr': '172.21.15.26:6823/3993357967', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 525, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6826', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6827', 'nonce': 4110225509}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6828', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6829', 'nonce': 4110225509}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6832', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6833', 'nonce': 4110225509}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6830', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6831', 'nonce': 4110225509}]}, 'public_addr': '172.21.15.26:6827/4110225509', 'cluster_addr': '172.21.15.26:6829/4110225509', 'heartbeat_back_addr': '172.21.15.26:6833/4110225509', 'heartbeat_front_addr': '172.21.15.26:6831/4110225509', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'f14e34d5-9829-42a3-9b25-31a0976c5ef7', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 366, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6800', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6801', 'nonce': 477923058}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6802', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6803', 'nonce': 477923058}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6806', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6807', 'nonce': 477923058}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6804', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6805', 'nonce': 477923058}]}, 'public_addr': '172.21.15.115:6801/477923058', 'cluster_addr': '172.21.15.115:6803/477923058', 'heartbeat_back_addr': '172.21.15.115:6807/477923058', 'heartbeat_front_addr': '172.21.15.115:6805/477923058', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'cb431b6a-540d-4318-ae35-d379e1df0185', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 41, 'last_clean_end': 273, 'up_from': 443, 'up_thru': 520, 'down_at': 274, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6808', 'nonce': 650416648}, {'type': 'v1', 'addr': '172.21.15.115:6809', 'nonce': 650416648}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6810', 'nonce': 650416648}, {'type': 'v1', 'addr': '172.21.15.115:6811', 'nonce': 650416648}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6814', 'nonce': 650416648}, {'type': 'v1', 'addr': '172.21.15.115:6815', 'nonce': 650416648}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6812', 'nonce': 650416648}, {'type': 'v1', 'addr': '172.21.15.115:6813', 'nonce': 650416648}]}, 'public_addr': '172.21.15.115:6809/650416648', 'cluster_addr': '172.21.15.115:6811/650416648', 'heartbeat_back_addr': '172.21.15.115:6815/650416648', 'heartbeat_front_addr': '172.21.15.115:6813/650416648', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '753ed830-e764-4ff3-9d9b-87e7fa748b81', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 562, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6816', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6817', 'nonce': 3278301843}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6818', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6819', 'nonce': 3278301843}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6822', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6823', 'nonce': 3278301843}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6820', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6821', 'nonce': 3278301843}]}, 'public_addr': '172.21.15.115:6817/3278301843', 'cluster_addr': '172.21.15.115:6819/3278301843', 'heartbeat_back_addr': '172.21.15.115:6823/3278301843', 'heartbeat_front_addr': '172.21.15.115:6821/3278301843', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '55bcbd37-7a6f-4ab1-b5a9-91a951444d86', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.058074951171875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 517, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6824', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6825', 'nonce': 3152424001}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6826', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6827', 'nonce': 3152424001}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6830', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6831', 'nonce': 3152424001}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6828', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6829', 'nonce': 3152424001}]}, 'public_addr': '172.21.15.115:6825/3152424001', 'cluster_addr': '172.21.15.115:6827/3152424001', 'heartbeat_back_addr': '172.21.15.115:6831/3152424001', 'heartbeat_front_addr': '172.21.15.115:6829/3152424001', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'bc86248e-6bd0-4369-b8a2-e593bc2a610d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 70, 'up_thru': 496, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6800', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6801', 'nonce': 1309426740}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6802', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6803', 'nonce': 1309426740}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6806', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6807', 'nonce': 1309426740}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6804', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6805', 'nonce': 1309426740}]}, 'public_addr': '172.21.15.122:6801/1309426740', 'cluster_addr': '172.21.15.122:6803/1309426740', 'heartbeat_back_addr': '172.21.15.122:6807/1309426740', 'heartbeat_front_addr': '172.21.15.122:6805/1309426740', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '9b20a4aa-032a-4c09-ac75-153c554424e7', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 257, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6808', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6809', 'nonce': 2953210076}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6810', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6811', 'nonce': 2953210076}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6814', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6815', 'nonce': 2953210076}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6812', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6813', 'nonce': 2953210076}]}, 'public_addr': '172.21.15.122:6809/2953210076', 'cluster_addr': '172.21.15.122:6811/2953210076', 'heartbeat_back_addr': '172.21.15.122:6815/2953210076', 'heartbeat_front_addr': '172.21.15.122:6813/2953210076', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'e62c1fb3-5dd1-4d0d-997c-d20633d19361', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 89, 'up_thru': 520, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6816', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6817', 'nonce': 409394340}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6818', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6819', 'nonce': 409394340}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6822', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6823', 'nonce': 409394340}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6820', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6821', 'nonce': 409394340}]}, 'public_addr': '172.21.15.122:6817/409394340', 'cluster_addr': '172.21.15.122:6819/409394340', 'heartbeat_back_addr': '172.21.15.122:6823/409394340', 'heartbeat_front_addr': '172.21.15.122:6821/409394340', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'b83994d5-e5ef-48b0-9758-c26843cec3b6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 96, 'up_thru': 522, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6824', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6825', 'nonce': 2704069354}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6826', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6827', 'nonce': 2704069354}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6830', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6831', 'nonce': 2704069354}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6828', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6829', 'nonce': 2704069354}]}, 'public_addr': '172.21.15.122:6825/2704069354', 'cluster_addr': '172.21.15.122:6827/2704069354', 'heartbeat_back_addr': '172.21.15.122:6831/2704069354', 'heartbeat_front_addr': '172.21.15.122:6829/2704069354', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:15:15.507934+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:15:07.459472+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:15:38.999217+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:15:17.274583+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-20T09:15:45.509487+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:15:49.636448+0000', 'dead_epoch': 273}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:14:32.283694+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:14:48.028594+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:14:50.225887+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 34736, 'last_purged_snaps_scrub': '2024-03-20T09:15:05.229610+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:15:10.066125+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:14:57.699566+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.1', 'osds': [6, 11, 5]}], 'pg_upmap_items': [{'pgid': '4.f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.26:0/4291404191': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/642012566': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:6801/1181459999': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/3984513952': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6800/1654945498': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6800/1181459999': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/137770981': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:0/1451237762': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/3938386631': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6801/1654945498': '2024-03-21T08:49:04.596754+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-20T09:15:55.067 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:15:55.475 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:55 smithi026 ceph-mon[68309]: 4.e scrub starts 2024-03-20T09:15:55.476 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:55 smithi026 ceph-mon[68309]: pgmap v1346: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 5.7 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 733 op/s 2024-03-20T09:15:55.476 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:55 smithi026 ceph-mon[68309]: 1.17 scrub starts 2024-03-20T09:15:55.476 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:55 smithi026 ceph-mon[68309]: 1.17 scrub ok 2024-03-20T09:15:55.480 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:15:55.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:55.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:15:55.587 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:15:55.621 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:55 smithi122 ceph-mon[73495]: 4.e scrub starts 2024-03-20T09:15:55.621 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:55 smithi122 ceph-mon[73495]: pgmap v1346: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 5.7 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 733 op/s 2024-03-20T09:15:55.621 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:55 smithi122 ceph-mon[73495]: 1.17 scrub starts 2024-03-20T09:15:55.621 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:55 smithi122 ceph-mon[73495]: 1.17 scrub ok 2024-03-20T09:15:55.623 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:55.650 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:15:55.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:55.655 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:15:55.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:55 smithi115 ceph-mon[72476]: 4.e scrub starts 2024-03-20T09:15:55.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:55 smithi115 ceph-mon[72476]: pgmap v1346: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 5.7 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 733 op/s 2024-03-20T09:15:55.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:55 smithi115 ceph-mon[72476]: 1.17 scrub starts 2024-03-20T09:15:55.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:55 smithi115 ceph-mon[72476]: 1.17 scrub ok 2024-03-20T09:15:55.756 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:15:55.821 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:56.059 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:15:56.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:56.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:15:56.082 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-20T09:15:56.170 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:15:56.209 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:56.254 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:15:56.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:56.258 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:15:56.359 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:15:56.396 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:56.425 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:15:56.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:56.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:15:56.478 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:56 smithi026 ceph-mon[68309]: 1.b scrub starts 2024-03-20T09:15:56.478 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:56 smithi026 ceph-mon[68309]: 1.b scrub ok 2024-03-20T09:15:56.478 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:56 smithi026 ceph-mon[68309]: 4.e scrub ok 2024-03-20T09:15:56.478 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:56 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:15:56.478 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:56 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:15:56.478 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:56 smithi026 ceph-mon[68309]: 1.15 scrub starts 2024-03-20T09:15:56.478 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:56 smithi026 ceph-mon[68309]: 1.15 scrub ok 2024-03-20T09:15:56.478 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:56 smithi026 ceph-mon[68309]: pgmap v1347: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 5.6 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 811 op/s 2024-03-20T09:15:56.529 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:15:56.560 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:15:56.605 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:15:56.646 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:56.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:56 smithi115 ceph-mon[72476]: 1.b scrub starts 2024-03-20T09:15:56.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:56 smithi115 ceph-mon[72476]: 1.b scrub ok 2024-03-20T09:15:56.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:56 smithi115 ceph-mon[72476]: 4.e scrub ok 2024-03-20T09:15:56.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:56 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:15:56.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:56 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:15:56.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:56 smithi115 ceph-mon[72476]: 1.15 scrub starts 2024-03-20T09:15:56.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:56 smithi115 ceph-mon[72476]: 1.15 scrub ok 2024-03-20T09:15:56.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:56 smithi115 ceph-mon[72476]: pgmap v1347: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 5.6 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 811 op/s 2024-03-20T09:15:56.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:56 smithi122 ceph-mon[73495]: 1.b scrub starts 2024-03-20T09:15:56.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:56 smithi122 ceph-mon[73495]: 1.b scrub ok 2024-03-20T09:15:56.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:56 smithi122 ceph-mon[73495]: 4.e scrub ok 2024-03-20T09:15:56.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:56 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:15:56.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:56 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:15:56.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:56 smithi122 ceph-mon[73495]: 1.15 scrub starts 2024-03-20T09:15:56.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:56 smithi122 ceph-mon[73495]: 1.15 scrub ok 2024-03-20T09:15:56.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:56 smithi122 ceph-mon[73495]: pgmap v1347: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 5.6 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 811 op/s 2024-03-20T09:15:56.920 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:15:56.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:56.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:15:57.028 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:15:57.103 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:57.135 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:15:57.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:57.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:15:57.238 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:15:57.266 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:57.290 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:15:57.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:57.295 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:15:57.395 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:15:57.439 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:57.461 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:15:57.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:57.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:15:57.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:57 smithi026 ceph-mon[68309]: 1.3 scrub starts 2024-03-20T09:15:57.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:57 smithi026 ceph-mon[68309]: 1.3 scrub ok 2024-03-20T09:15:57.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:57 smithi026 ceph-mon[68309]: osdmap e568: 12 total, 12 up, 10 in 2024-03-20T09:15:57.564 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:15:57.589 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:57 smithi122 ceph-mon[73495]: 1.3 scrub starts 2024-03-20T09:15:57.589 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:57 smithi122 ceph-mon[73495]: 1.3 scrub ok 2024-03-20T09:15:57.589 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:57 smithi122 ceph-mon[73495]: osdmap e568: 12 total, 12 up, 10 in 2024-03-20T09:15:57.591 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:57.601 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:15:57.615 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:15:57.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:57.622 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:15:57.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:57 smithi115 ceph-mon[72476]: 1.3 scrub starts 2024-03-20T09:15:57.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:57 smithi115 ceph-mon[72476]: 1.3 scrub ok 2024-03-20T09:15:57.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:57 smithi115 ceph-mon[72476]: osdmap e568: 12 total, 12 up, 10 in 2024-03-20T09:15:57.723 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:15:57.752 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:57.775 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:15:57.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:57.781 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:15:57.882 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:15:57.919 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:57.944 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:15:57.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:57.950 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:15:58.050 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:15:58.082 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:58.103 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:15:58.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:58.115 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:15:58.216 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:15:58.244 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:58.273 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:15:58.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:58.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:15:58.377 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:15:58.417 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:58.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:58 smithi026 ceph-mon[68309]: pgmap v1349: 47 pgs: 47 active+clean; 5.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 969 op/s 2024-03-20T09:15:58.529 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:15:58.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:58 smithi115 ceph-mon[72476]: pgmap v1349: 47 pgs: 47 active+clean; 5.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 969 op/s 2024-03-20T09:15:58.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:58 smithi122 ceph-mon[73495]: pgmap v1349: 47 pgs: 47 active+clean; 5.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 969 op/s 2024-03-20T09:15:59.528 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:15:59.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:59.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:15:59.632 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:15:59.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:59 smithi115 ceph-mon[72476]: 1.7 scrub starts 2024-03-20T09:15:59.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:15:59 smithi115 ceph-mon[72476]: 1.7 scrub ok 2024-03-20T09:15:59.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:59 smithi122 ceph-mon[73495]: 1.7 scrub starts 2024-03-20T09:15:59.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:15:59 smithi122 ceph-mon[73495]: 1.7 scrub ok 2024-03-20T09:15:59.700 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:59 smithi026 ceph-mon[68309]: 1.7 scrub starts 2024-03-20T09:15:59.700 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:15:59 smithi026 ceph-mon[68309]: 1.7 scrub ok 2024-03-20T09:15:59.705 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:15:59.843 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:15:59.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:15:59.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:15:59.950 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:16:00.018 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:00.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:00 smithi115 ceph-mon[72476]: 1.4 scrub starts 2024-03-20T09:16:00.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:00 smithi115 ceph-mon[72476]: 1.4 scrub ok 2024-03-20T09:16:00.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:00 smithi115 ceph-mon[72476]: 4.5 scrub starts 2024-03-20T09:16:00.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:00 smithi115 ceph-mon[72476]: 1.9 scrub starts 2024-03-20T09:16:00.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:00 smithi115 ceph-mon[72476]: 1.9 scrub ok 2024-03-20T09:16:00.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:00 smithi115 ceph-mon[72476]: pgmap v1350: 47 pgs: 47 active+clean; 5.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 715 op/s 2024-03-20T09:16:00.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:00 smithi122 ceph-mon[73495]: 1.4 scrub starts 2024-03-20T09:16:00.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:00 smithi122 ceph-mon[73495]: 1.4 scrub ok 2024-03-20T09:16:00.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:00 smithi122 ceph-mon[73495]: 4.5 scrub starts 2024-03-20T09:16:00.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:00 smithi122 ceph-mon[73495]: 1.9 scrub starts 2024-03-20T09:16:00.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:00 smithi122 ceph-mon[73495]: 1.9 scrub ok 2024-03-20T09:16:00.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:00 smithi122 ceph-mon[73495]: pgmap v1350: 47 pgs: 47 active+clean; 5.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 715 op/s 2024-03-20T09:16:00.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:00 smithi026 ceph-mon[68309]: 1.4 scrub starts 2024-03-20T09:16:00.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:00 smithi026 ceph-mon[68309]: 1.4 scrub ok 2024-03-20T09:16:00.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:00 smithi026 ceph-mon[68309]: 4.5 scrub starts 2024-03-20T09:16:00.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:00 smithi026 ceph-mon[68309]: 1.9 scrub starts 2024-03-20T09:16:00.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:00 smithi026 ceph-mon[68309]: 1.9 scrub ok 2024-03-20T09:16:00.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:00 smithi026 ceph-mon[68309]: pgmap v1350: 47 pgs: 47 active+clean; 5.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 715 op/s 2024-03-20T09:16:01.435 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:16:01.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:01.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:16:01.545 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:16:01.574 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:01.574 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:01 smithi122 ceph-mon[73495]: 4.7 scrub starts 2024-03-20T09:16:01.575 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:01 smithi122 ceph-mon[73495]: 4.5 scrub ok 2024-03-20T09:16:01.636 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:16:01.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:01.645 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:16:01.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:01 smithi115 ceph-mon[72476]: 4.7 scrub starts 2024-03-20T09:16:01.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:01 smithi115 ceph-mon[72476]: 4.5 scrub ok 2024-03-20T09:16:01.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:01 smithi026 ceph-mon[68309]: 4.7 scrub starts 2024-03-20T09:16:01.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:01 smithi026 ceph-mon[68309]: 4.5 scrub ok 2024-03-20T09:16:01.746 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:16:01.828 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:01.974 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:16:02.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:02.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:16:02.101 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:16:02.176 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:02.450 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:16:02.468 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:16:02.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:02 smithi026 ceph-mon[68309]: pgmap v1351: 47 pgs: 47 active+clean; 5.3 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 796 op/s 2024-03-20T09:16:02.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:02 smithi115 ceph-mon[72476]: pgmap v1351: 47 pgs: 47 active+clean; 5.3 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 796 op/s 2024-03-20T09:16:02.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:02 smithi122 ceph-mon[73495]: pgmap v1351: 47 pgs: 47 active+clean; 5.3 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 796 op/s 2024-03-20T09:16:03.389 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:16:03.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:03.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:16:03.516 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:16:03.528 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:03 smithi026 ceph-mon[68309]: 4.7 scrub ok 2024-03-20T09:16:03.528 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:03 smithi026 ceph-mon[68309]: from='client.15951 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:16:03.529 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:03 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:16:03.529 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:03 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:16:03.546 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:03.576 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:16:03.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:03.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:16:03.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:03 smithi115 ceph-mon[72476]: 4.7 scrub ok 2024-03-20T09:16:03.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:03 smithi115 ceph-mon[72476]: from='client.15951 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:16:03.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:03 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:16:03.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:03 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:16:03.686 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:16:03.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:03 smithi122 ceph-mon[73495]: 4.7 scrub ok 2024-03-20T09:16:03.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:03 smithi122 ceph-mon[73495]: from='client.15951 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:16:03.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:03 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:16:03.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:03 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:16:03.720 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:03.742 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:16:03.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:03.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:16:03.848 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:16:03.875 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:03.898 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:16:03.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:03.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:16:03.910 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":1351,"stamp":"2024-03-20T09:16:01.939296+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5715526200,"num_objects":87206,"num_object_clones":0,"num_object_copies":261618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87206,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154630,"num_write_kb":7738625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":923,"num_bytes_recovered":60751872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160508,"ondisk_log_size":160508,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":20224984,"kb_used_data":16788624,"kb_used_omap":195,"kb_used_meta":3435324,"kb_avail":917226536,"statfs":{"total":959950356480,"available":939239972864,"internally_reserved":0,"allocated":17191550976,"data_stored":17189008614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":200482,"internal_metadata":3517771998},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-522452992,"num_objects":-7972,"num_object_clones":0,"num_object_copies":-23916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7972,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.011617"},"pg_stats":[{"pgid":"1.1e","version":"20'137","reported_seq":520,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:56.294817+0000","last_change":"2024-03-20T09:15:08.506869+0000","last_active":"2024-03-20T09:15:56.294817+0000","last_peered":"2024-03-20T09:15:56.294817+0000","last_clean":"2024-03-20T09:15:56.294817+0000","last_became_active":"2024-03-20T09:13:20.071655+0000","last_became_peered":"2024-03-20T09:13:20.071655+0000","last_unstale":"2024-03-20T09:15:56.294817+0000","last_undegraded":"2024-03-20T09:15:56.294817+0000","last_fullsized":"2024-03-20T09:15:56.294817+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:08.506732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:15:08.506732+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:22.387718+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'137","reported_seq":687,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:56.292190+0000","last_change":"2024-03-20T09:14:44.782411+0000","last_active":"2024-03-20T09:15:56.292190+0000","last_peered":"2024-03-20T09:15:56.292190+0000","last_clean":"2024-03-20T09:15:56.292190+0000","last_became_active":"2024-03-20T09:07:42.013245+0000","last_became_peered":"2024-03-20T09:07:42.013245+0000","last_unstale":"2024-03-20T09:15:56.292190+0000","last_undegraded":"2024-03-20T09:15:56.292190+0000","last_fullsized":"2024-03-20T09:15:56.292190+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:14:44.782290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:14:44.782290+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"20'137","reported_seq":1129,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:56.300023+0000","last_change":"2024-03-20T09:13:56.707873+0000","last_active":"2024-03-20T09:15:56.300023+0000","last_peered":"2024-03-20T09:15:56.300023+0000","last_clean":"2024-03-20T09:15:56.300023+0000","last_became_active":"2024-03-20T09:07:41.961622+0000","last_became_peered":"2024-03-20T09:07:41.961622+0000","last_unstale":"2024-03-20T09:15:56.300023+0000","last_undegraded":"2024-03-20T09:15:56.300023+0000","last_fullsized":"2024-03-20T09:15:56.300023+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:13:56.707767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:13:56.707767+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.19","version":"20'137","reported_seq":1198,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:56.290321+0000","last_change":"2024-03-20T09:14:54.527786+0000","last_active":"2024-03-20T09:15:56.290321+0000","last_peered":"2024-03-20T09:15:56.290321+0000","last_clean":"2024-03-20T09:15:56.290321+0000","last_became_active":"2024-03-20T09:07:41.962680+0000","last_became_peered":"2024-03-20T09:07:41.962680+0000","last_unstale":"2024-03-20T09:15:56.290321+0000","last_undegraded":"2024-03-20T09:15:56.290321+0000","last_fullsized":"2024-03-20T09:15:56.290321+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:14:54.527668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:14:54.527668+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:18.597530+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"20'137","reported_seq":660,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:56.292530+0000","last_change":"2024-03-20T09:15:10.009033+0000","last_active":"2024-03-20T09:15:56.292530+0000","last_peered":"2024-03-20T09:15:56.292530+0000","last_clean":"2024-03-20T09:15:56.292530+0000","last_became_active":"2024-03-20T09:10:35.277481+0000","last_became_peered":"2024-03-20T09:10:35.277481+0000","last_unstale":"2024-03-20T09:15:56.292530+0000","last_undegraded":"2024-03-20T09:15:56.292530+0000","last_fullsized":"2024-03-20T09:15:56.292530+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:10.008935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:15:10.008935+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:23.911941+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1b","version":"20'137","reported_seq":966,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:56.297877+0000","last_change":"2024-03-20T09:15:10.569807+0000","last_active":"2024-03-20T09:15:56.297877+0000","last_peered":"2024-03-20T09:15:56.297877+0000","last_clean":"2024-03-20T09:15:56.297877+0000","last_became_active":"2024-03-20T09:07:41.962733+0000","last_became_peered":"2024-03-20T09:07:41.962733+0000","last_unstale":"2024-03-20T09:15:56.297877+0000","last_undegraded":"2024-03-20T09:15:56.297877+0000","last_fullsized":"2024-03-20T09:15:56.297877+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:10.569712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:15:10.569712+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:32.526084+0000","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"568'13516","reported_seq":13873,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:57.299007+0000","last_change":"2024-03-20T09:15:53.214593+0000","last_active":"2024-03-20T09:15:57.299007+0000","last_peered":"2024-03-20T09:15:57.299007+0000","last_clean":"2024-03-20T09:15:57.299007+0000","last_became_active":"2024-03-20T09:13:22.063020+0000","last_became_peered":"2024-03-20T09:13:22.063020+0000","last_unstale":"2024-03-20T09:15:57.299007+0000","last_undegraded":"2024-03-20T09:15:57.299007+0000","last_fullsized":"2024-03-20T09:15:57.299007+0000","mapping_epoch":520,"log_start":"538'3500","ondisk_log_start":"538'3500","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'13288","last_scrub_stamp":"2024-03-20T09:15:53.214204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:15:53.214204+0000","objects_scrubbed":5818,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:00.546404+0000","scrub_duration":2512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362872856,"num_objects":5538,"num_object_clones":0,"num_object_copies":16614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5538,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9676,"num_write_kb":486785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"20'137","reported_seq":659,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:56.292544+0000","last_change":"2024-03-20T09:14:52.016392+0000","last_active":"2024-03-20T09:15:56.292544+0000","last_peered":"2024-03-20T09:15:56.292544+0000","last_clean":"2024-03-20T09:15:56.292544+0000","last_became_active":"2024-03-20T09:10:35.215772+0000","last_became_peered":"2024-03-20T09:10:35.215772+0000","last_unstale":"2024-03-20T09:15:56.292544+0000","last_undegraded":"2024-03-20T09:15:56.292544+0000","last_fullsized":"2024-03-20T09:15:56.292544+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:14:52.016290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:14:52.016290+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:19.535050+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"568'9666","reported_seq":9850,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:16:00.511420+0000","last_change":"2024-03-20T09:14:44.134032+0000","last_active":"2024-03-20T09:16:00.511420+0000","last_peered":"2024-03-20T09:16:00.511420+0000","last_clean":"2024-03-20T09:16:00.511420+0000","last_became_active":"2024-03-20T09:13:22.583317+0000","last_became_peered":"2024-03-20T09:13:22.583317+0000","last_unstale":"2024-03-20T09:16:00.511420+0000","last_undegraded":"2024-03-20T09:16:00.511420+0000","last_fullsized":"2024-03-20T09:16:00.511420+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"550'6791","last_scrub_stamp":"2024-03-20T09:14:44.133820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:14:44.133820+0000","objects_scrubbed":6742,"log_size":9666,"log_dups_size":0,"ondisk_log_size":9666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2011,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351666176,"num_objects":5366,"num_object_clones":0,"num_object_copies":16098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9666,"num_write_kb":481024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,8],"acting":[3,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"20'137","reported_seq":1211,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:56.300227+0000","last_change":"2024-03-20T09:14:45.086348+0000","last_active":"2024-03-20T09:15:56.300227+0000","last_peered":"2024-03-20T09:15:56.300227+0000","last_clean":"2024-03-20T09:15:56.300227+0000","last_became_active":"2024-03-20T09:10:34.099414+0000","last_became_peered":"2024-03-20T09:10:34.099414+0000","last_unstale":"2024-03-20T09:15:56.300227+0000","last_undegraded":"2024-03-20T09:15:56.300227+0000","last_fullsized":"2024-03-20T09:15:56.300227+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:14:45.086222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:14:45.086222+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:15:59.410055+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"568'13995","reported_seq":16224,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:59.035864+0000","last_change":"2024-03-20T09:15:51.667711+0000","last_active":"2024-03-20T09:15:59.035864+0000","last_peered":"2024-03-20T09:15:59.035864+0000","last_clean":"2024-03-20T09:15:59.035864+0000","last_became_active":"2024-03-20T09:13:46.198893+0000","last_became_peered":"2024-03-20T09:13:46.198893+0000","last_unstale":"2024-03-20T09:15:59.035864+0000","last_undegraded":"2024-03-20T09:15:59.035864+0000","last_fullsized":"2024-03-20T09:15:59.035864+0000","mapping_epoch":534,"log_start":"539'3902","ondisk_log_start":"539'3902","created":520,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'13610","last_scrub_stamp":"2024-03-20T09:15:51.667432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:15:51.667432+0000","objects_scrubbed":6005,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:01.763807+0000","scrub_duration":2000,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365297664,"num_objects":5574,"num_object_clones":0,"num_object_copies":16722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9840,"num_write_kb":493248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":922,"num_bytes_recovered":60424192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,11],"acting":[2,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.a","version":"20'137","reported_seq":1024,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:56.288530+0000","last_change":"2024-03-20T09:15:53.510427+0000","last_active":"2024-03-20T09:15:56.288530+0000","last_peered":"2024-03-20T09:15:56.288530+0000","last_clean":"2024-03-20T09:15:56.288530+0000","last_became_active":"2024-03-20T09:07:52.618107+0000","last_became_peered":"2024-03-20T09:07:52.618107+0000","last_unstale":"2024-03-20T09:15:56.288530+0000","last_undegraded":"2024-03-20T09:15:56.288530+0000","last_fullsized":"2024-03-20T09:15:56.288530+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:53.510342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:15:53.510342+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:08.887351+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"568'9744","reported_seq":9926,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:59.905822+0000","last_change":"2024-03-20T09:15:51.598949+0000","last_active":"2024-03-20T09:15:59.905822+0000","last_peered":"2024-03-20T09:15:59.905822+0000","last_clean":"2024-03-20T09:15:59.905822+0000","last_became_active":"2024-03-20T09:13:22.063975+0000","last_became_peered":"2024-03-20T09:13:22.063975+0000","last_unstale":"2024-03-20T09:15:59.905822+0000","last_undegraded":"2024-03-20T09:15:59.905822+0000","last_fullsized":"2024-03-20T09:15:59.905822+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'9304","last_scrub_stamp":"2024-03-20T09:15:51.598725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:15:51.598725+0000","objects_scrubbed":5732,"log_size":9744,"log_dups_size":0,"ondisk_log_size":9744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:06.892316+0000","scrub_duration":1055,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345112576,"num_objects":5266,"num_object_clones":0,"num_object_copies":15798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9744,"num_write_kb":480320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"20'137","reported_seq":1188,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:58.623102+0000","last_change":"2024-03-20T09:15:58.623124+0000","last_active":"2024-03-20T09:15:58.623102+0000","last_peered":"2024-03-20T09:15:58.623102+0000","last_clean":"2024-03-20T09:15:58.623102+0000","last_became_active":"2024-03-20T09:12:14.306384+0000","last_became_peered":"2024-03-20T09:12:14.306384+0000","last_unstale":"2024-03-20T09:15:58.623102+0000","last_undegraded":"2024-03-20T09:15:58.623102+0000","last_fullsized":"2024-03-20T09:15:58.623102+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:58.623048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:15:58.623048+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:02.219739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"568'9651","reported_seq":9833,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:16:00.550110+0000","last_change":"2024-03-20T09:14:52.155033+0000","last_active":"2024-03-20T09:16:00.550110+0000","last_peered":"2024-03-20T09:16:00.550110+0000","last_clean":"2024-03-20T09:16:00.550110+0000","last_became_active":"2024-03-20T09:13:24.064091+0000","last_became_peered":"2024-03-20T09:13:24.064091+0000","last_unstale":"2024-03-20T09:16:00.550110+0000","last_undegraded":"2024-03-20T09:16:00.550110+0000","last_fullsized":"2024-03-20T09:16:00.550110+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"552'7170","last_scrub_stamp":"2024-03-20T09:14:52.154865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:14:52.154865+0000","objects_scrubbed":7129,"log_size":9651,"log_dups_size":0,"ondisk_log_size":9651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:12.538193+0000","scrub_duration":1965,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347275264,"num_objects":5299,"num_object_clones":0,"num_object_copies":15897,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9651,"num_write_kb":478400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"20'137","reported_seq":1076,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:56.290446+0000","last_change":"2024-03-20T09:14:58.582184+0000","last_active":"2024-03-20T09:15:56.290446+0000","last_peered":"2024-03-20T09:15:56.290446+0000","last_clean":"2024-03-20T09:15:56.290446+0000","last_became_active":"2024-03-20T09:07:37.003722+0000","last_became_peered":"2024-03-20T09:07:37.003722+0000","last_unstale":"2024-03-20T09:15:56.290446+0000","last_undegraded":"2024-03-20T09:15:56.290446+0000","last_fullsized":"2024-03-20T09:15:56.290446+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:14:58.582047+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:07:06.480160+0000","last_clean_scrub_stamp":"2024-03-20T09:14:58.582047+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:03.684147+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.3","version":"568'9941","reported_seq":10132,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:59.895298+0000","last_change":"2024-03-20T09:15:09.333207+0000","last_active":"2024-03-20T09:15:59.895298+0000","last_peered":"2024-03-20T09:15:59.895298+0000","last_clean":"2024-03-20T09:15:59.895298+0000","last_became_active":"2024-03-20T09:13:22.064275+0000","last_became_peered":"2024-03-20T09:13:22.064275+0000","last_unstale":"2024-03-20T09:15:59.895298+0000","last_undegraded":"2024-03-20T09:15:59.895298+0000","last_fullsized":"2024-03-20T09:15:59.895298+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'7909","last_scrub_stamp":"2024-03-20T09:15:09.333054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:15:09.333054+0000","objects_scrubbed":7634,"log_size":9941,"log_dups_size":0,"ondisk_log_size":9941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:21.566609+0000","scrub_duration":1910,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366018560,"num_objects":5585,"num_object_clones":0,"num_object_copies":16755,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9941,"num_write_kb":496832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"20'137","reported_seq":1101,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:56.290435+0000","last_change":"2024-03-20T09:15:47.574426+0000","last_active":"2024-03-20T09:15:56.290435+0000","last_peered":"2024-03-20T09:15:56.290435+0000","last_clean":"2024-03-20T09:15:56.290435+0000","last_became_active":"2024-03-20T09:07:52.593671+0000","last_became_peered":"2024-03-20T09:07:52.593671+0000","last_unstale":"2024-03-20T09:15:56.290435+0000","last_undegraded":"2024-03-20T09:15:56.290435+0000","last_fullsized":"2024-03-20T09:15:56.290435+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:47.574277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:15:47.574277+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:12.232176+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.6","version":"568'9650","reported_seq":9841,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:57.304076+0000","last_change":"2024-03-20T09:15:01.280269+0000","last_active":"2024-03-20T09:15:57.304076+0000","last_peered":"2024-03-20T09:15:57.304076+0000","last_clean":"2024-03-20T09:15:57.304076+0000","last_became_active":"2024-03-20T09:13:22.067362+0000","last_became_peered":"2024-03-20T09:13:22.067362+0000","last_unstale":"2024-03-20T09:15:57.304076+0000","last_undegraded":"2024-03-20T09:15:57.304076+0000","last_fullsized":"2024-03-20T09:15:57.304076+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"553'7597","last_scrub_stamp":"2024-03-20T09:15:01.280134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:15:01.280134+0000","objects_scrubbed":7550,"log_size":9650,"log_dups_size":0,"ondisk_log_size":9650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:02.447803+0000","scrub_duration":3618,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364773376,"num_objects":5566,"num_object_clones":0,"num_object_copies":16698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9650,"num_write_kb":486912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"20'137","reported_seq":1093,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:56.292712+0000","last_change":"2024-03-20T09:15:55.668210+0000","last_active":"2024-03-20T09:15:56.292712+0000","last_peered":"2024-03-20T09:15:56.292712+0000","last_clean":"2024-03-20T09:15:56.292712+0000","last_became_active":"2024-03-20T09:12:02.221686+0000","last_became_peered":"2024-03-20T09:12:02.221686+0000","last_unstale":"2024-03-20T09:15:56.292712+0000","last_undegraded":"2024-03-20T09:15:56.292712+0000","last_fullsized":"2024-03-20T09:15:56.292712+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:55.668113+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:18.788771+0000","last_clean_scrub_stamp":"2024-03-20T09:15:55.668113+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:12.700115+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,8],"acting":[1,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"568'9640","reported_seq":9881,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:16:00.673503+0000","last_change":"2024-03-20T09:16:00.516288+0000","last_active":"2024-03-20T09:16:00.673503+0000","last_peered":"2024-03-20T09:16:00.673503+0000","last_clean":"2024-03-20T09:16:00.673503+0000","last_became_active":"2024-03-20T09:13:22.062992+0000","last_became_peered":"2024-03-20T09:13:22.062992+0000","last_unstale":"2024-03-20T09:16:00.673503+0000","last_undegraded":"2024-03-20T09:16:00.673503+0000","last_fullsized":"2024-03-20T09:16:00.673503+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'9630","last_scrub_stamp":"2024-03-20T09:16:00.516080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:16:00.516080+0000","objects_scrubbed":5447,"log_size":9640,"log_dups_size":0,"ondisk_log_size":9640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:29.908467+0000","scrub_duration":937,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":354811904,"num_objects":5414,"num_object_clones":0,"num_object_copies":16242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9640,"num_write_kb":481728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":1151,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:56.293695+0000","last_change":"2024-03-20T09:14:45.600449+0000","last_active":"2024-03-20T09:15:56.293695+0000","last_peered":"2024-03-20T09:15:56.293695+0000","last_clean":"2024-03-20T09:15:56.293695+0000","last_became_active":"2024-03-20T09:12:14.305989+0000","last_became_peered":"2024-03-20T09:12:14.305989+0000","last_unstale":"2024-03-20T09:15:56.293695+0000","last_undegraded":"2024-03-20T09:15:56.293695+0000","last_fullsized":"2024-03-20T09:15:56.293695+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:14:45.600372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:14:45.600372+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:03.841360+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.7","version":"568'9622","reported_seq":9807,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:57.298995+0000","last_change":"2024-03-20T09:14:48.807600+0000","last_active":"2024-03-20T09:15:57.298995+0000","last_peered":"2024-03-20T09:15:57.298995+0000","last_clean":"2024-03-20T09:15:57.298995+0000","last_became_active":"2024-03-20T09:13:22.065656+0000","last_became_peered":"2024-03-20T09:13:22.065656+0000","last_unstale":"2024-03-20T09:15:57.298995+0000","last_undegraded":"2024-03-20T09:15:57.298995+0000","last_fullsized":"2024-03-20T09:15:57.298995+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"551'7096","last_scrub_stamp":"2024-03-20T09:14:48.807283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:14:48.807283+0000","objects_scrubbed":7044,"log_size":9622,"log_dups_size":0,"ondisk_log_size":9622,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2072,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362151936,"num_objects":5526,"num_object_clones":0,"num_object_copies":16578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9622,"num_write_kb":484736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,0],"acting":[1,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"20'137","reported_seq":1401,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:56.294146+0000","last_change":"2024-03-20T09:15:01.958964+0000","last_active":"2024-03-20T09:15:56.294146+0000","last_peered":"2024-03-20T09:15:56.294146+0000","last_clean":"2024-03-20T09:15:56.294146+0000","last_became_active":"2024-03-20T09:12:14.313541+0000","last_became_peered":"2024-03-20T09:12:14.313541+0000","last_unstale":"2024-03-20T09:15:56.294146+0000","last_undegraded":"2024-03-20T09:15:56.294146+0000","last_fullsized":"2024-03-20T09:15:56.294146+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:01.958839+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:19.513511+0000","last_clean_scrub_stamp":"2024-03-20T09:15:01.958839+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:16.285345+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"568'13095","reported_seq":13449,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:57.292178+0000","last_change":"2024-03-20T09:15:52.485886+0000","last_active":"2024-03-20T09:15:57.292178+0000","last_peered":"2024-03-20T09:15:57.292178+0000","last_clean":"2024-03-20T09:15:57.292178+0000","last_became_active":"2024-03-20T09:13:24.443054+0000","last_became_peered":"2024-03-20T09:13:24.443054+0000","last_unstale":"2024-03-20T09:15:57.292178+0000","last_undegraded":"2024-03-20T09:15:57.292178+0000","last_fullsized":"2024-03-20T09:15:57.292178+0000","mapping_epoch":522,"log_start":"536'3001","ondisk_log_start":"536'3001","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'12854","last_scrub_stamp":"2024-03-20T09:15:52.485530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:15:52.485530+0000","objects_scrubbed":5795,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:58.805212+0000","scrub_duration":2823,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":359858176,"num_objects":5491,"num_object_clones":0,"num_object_copies":16473,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9435,"num_write_kb":477632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"20'137","reported_seq":554,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:56.288318+0000","last_change":"2024-03-20T09:15:34.225440+0000","last_active":"2024-03-20T09:15:56.288318+0000","last_peered":"2024-03-20T09:15:56.288318+0000","last_clean":"2024-03-20T09:15:56.288318+0000","last_became_active":"2024-03-20T09:15:34.225022+0000","last_became_peered":"2024-03-20T09:15:34.225022+0000","last_unstale":"2024-03-20T09:15:56.288318+0000","last_undegraded":"2024-03-20T09:15:56.288318+0000","last_fullsized":"2024-03-20T09:15:56.288318+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:14:47.524588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:14:47.524588+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:13.568057+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,5],"acting":[6,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"568'9622","reported_seq":9874,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:57.163533+0000","last_change":"2024-03-20T09:15:51.597847+0000","last_active":"2024-03-20T09:15:57.163533+0000","last_peered":"2024-03-20T09:15:57.163533+0000","last_clean":"2024-03-20T09:15:57.163533+0000","last_became_active":"2024-03-20T09:13:22.583529+0000","last_became_peered":"2024-03-20T09:13:22.583529+0000","last_unstale":"2024-03-20T09:15:57.163533+0000","last_undegraded":"2024-03-20T09:15:57.163533+0000","last_fullsized":"2024-03-20T09:15:57.163533+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'9317","last_scrub_stamp":"2024-03-20T09:15:51.597715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:15:51.597715+0000","objects_scrubbed":5894,"log_size":9622,"log_dups_size":0,"ondisk_log_size":9622,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:18.767531+0000","scrub_duration":911,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365035520,"num_objects":5570,"num_object_clones":0,"num_object_copies":16710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9622,"num_write_kb":486144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,8],"acting":[5,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"20'137","reported_seq":1097,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:58.696655+0000","last_change":"2024-03-20T09:15:58.696682+0000","last_active":"2024-03-20T09:15:58.696655+0000","last_peered":"2024-03-20T09:15:58.696655+0000","last_clean":"2024-03-20T09:15:58.696655+0000","last_became_active":"2024-03-20T09:13:27.087760+0000","last_became_peered":"2024-03-20T09:13:27.087760+0000","last_unstale":"2024-03-20T09:15:58.696655+0000","last_undegraded":"2024-03-20T09:15:58.696655+0000","last_fullsized":"2024-03-20T09:15:58.696655+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:58.696597+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:14:10.560890+0000","last_clean_scrub_stamp":"2024-03-20T09:15:58.696597+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:00.568581+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"568'9765","reported_seq":10012,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:16:01.203515+0000","last_change":"2024-03-20T09:15:54.984595+0000","last_active":"2024-03-20T09:16:01.203515+0000","last_peered":"2024-03-20T09:16:01.203515+0000","last_clean":"2024-03-20T09:16:01.203515+0000","last_became_active":"2024-03-20T09:13:22.064461+0000","last_became_peered":"2024-03-20T09:13:22.064461+0000","last_unstale":"2024-03-20T09:16:01.203515+0000","last_undegraded":"2024-03-20T09:16:01.203515+0000","last_fullsized":"2024-03-20T09:16:01.203515+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'9446","last_scrub_stamp":"2024-03-20T09:15:54.984433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:15:54.984433+0000","objects_scrubbed":5711,"log_size":9765,"log_dups_size":0,"ondisk_log_size":9765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:23.039467+0000","scrub_duration":1570,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350027776,"num_objects":5341,"num_object_clones":0,"num_object_copies":16023,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9765,"num_write_kb":483392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,1],"acting":[10,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.b","version":"20'137","reported_seq":965,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:56.295034+0000","last_change":"2024-03-20T09:15:54.613271+0000","last_active":"2024-03-20T09:15:56.295034+0000","last_peered":"2024-03-20T09:15:56.295034+0000","last_clean":"2024-03-20T09:15:56.295034+0000","last_became_active":"2024-03-20T09:13:20.070596+0000","last_became_peered":"2024-03-20T09:13:20.070596+0000","last_unstale":"2024-03-20T09:15:56.295034+0000","last_undegraded":"2024-03-20T09:15:56.295034+0000","last_fullsized":"2024-03-20T09:15:56.295034+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:54.613162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:15:54.613162+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:15.365704+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.b","version":"568'9401","reported_seq":9583,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:56.894106+0000","last_change":"2024-03-20T09:14:46.191729+0000","last_active":"2024-03-20T09:15:56.894106+0000","last_peered":"2024-03-20T09:15:56.894106+0000","last_clean":"2024-03-20T09:15:56.894106+0000","last_became_active":"2024-03-20T09:13:22.064687+0000","last_became_peered":"2024-03-20T09:13:22.064687+0000","last_unstale":"2024-03-20T09:15:56.894106+0000","last_undegraded":"2024-03-20T09:15:56.894106+0000","last_fullsized":"2024-03-20T09:15:56.894106+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"550'6828","last_scrub_stamp":"2024-03-20T09:14:46.191490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:14:46.191490+0000","objects_scrubbed":6762,"log_size":9401,"log_dups_size":0,"ondisk_log_size":9401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:15.502318+0000","scrub_duration":2514,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":361299968,"num_objects":5513,"num_object_clones":0,"num_object_copies":16539,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9401,"num_write_kb":477248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"20'137","reported_seq":625,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:56.309615+0000","last_change":"2024-03-20T09:15:46.735519+0000","last_active":"2024-03-20T09:15:56.309615+0000","last_peered":"2024-03-20T09:15:56.309615+0000","last_clean":"2024-03-20T09:15:56.309615+0000","last_became_active":"2024-03-20T09:12:14.311319+0000","last_became_peered":"2024-03-20T09:12:14.311319+0000","last_unstale":"2024-03-20T09:15:56.309615+0000","last_undegraded":"2024-03-20T09:15:56.309615+0000","last_fullsized":"2024-03-20T09:15:56.309615+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:46.735418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:15:46.735418+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:08.164707+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"568'9643","reported_seq":9822,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:16:00.724843+0000","last_change":"2024-03-20T09:14:39.771481+0000","last_active":"2024-03-20T09:16:00.724843+0000","last_peered":"2024-03-20T09:16:00.724843+0000","last_clean":"2024-03-20T09:16:00.724843+0000","last_became_active":"2024-03-20T09:13:22.063927+0000","last_became_peered":"2024-03-20T09:13:22.063927+0000","last_unstale":"2024-03-20T09:16:00.724843+0000","last_undegraded":"2024-03-20T09:16:00.724843+0000","last_fullsized":"2024-03-20T09:16:00.724843+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"550'6524","last_scrub_stamp":"2024-03-20T09:14:39.771224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:14:39.771224+0000","objects_scrubbed":6484,"log_size":9643,"log_dups_size":0,"ondisk_log_size":9643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:03.494718+0000","scrub_duration":1395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353566720,"num_objects":5395,"num_object_clones":0,"num_object_copies":16185,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9643,"num_write_kb":481216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"20'137","reported_seq":1113,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:56.288549+0000","last_change":"2024-03-20T09:15:48.626548+0000","last_active":"2024-03-20T09:15:56.288549+0000","last_peered":"2024-03-20T09:15:56.288549+0000","last_clean":"2024-03-20T09:15:56.288549+0000","last_became_active":"2024-03-20T09:10:35.119706+0000","last_became_peered":"2024-03-20T09:10:35.119706+0000","last_unstale":"2024-03-20T09:15:56.288549+0000","last_undegraded":"2024-03-20T09:15:56.288549+0000","last_fullsized":"2024-03-20T09:15:56.288549+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:48.626365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:15:48.626365+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:13.935726+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"568'13487","reported_seq":13844,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:59.910429+0000","last_change":"2024-03-20T09:15:50.446152+0000","last_active":"2024-03-20T09:15:59.910429+0000","last_peered":"2024-03-20T09:15:59.910429+0000","last_clean":"2024-03-20T09:15:59.910429+0000","last_became_active":"2024-03-20T09:13:24.065349+0000","last_became_peered":"2024-03-20T09:13:24.065349+0000","last_unstale":"2024-03-20T09:15:59.910429+0000","last_undegraded":"2024-03-20T09:15:59.910429+0000","last_fullsized":"2024-03-20T09:15:59.910429+0000","mapping_epoch":522,"log_start":"538'3400","ondisk_log_start":"538'3400","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'13038","last_scrub_stamp":"2024-03-20T09:15:50.445858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:15:50.445858+0000","objects_scrubbed":5938,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:58.046301+0000","scrub_duration":1870,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":357040128,"num_objects":5448,"num_object_clones":0,"num_object_copies":16344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9708,"num_write_kb":484992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"20'137","reported_seq":1132,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:56.293278+0000","last_change":"2024-03-20T09:14:46.139707+0000","last_active":"2024-03-20T09:15:56.293278+0000","last_peered":"2024-03-20T09:15:56.293278+0000","last_clean":"2024-03-20T09:15:56.293278+0000","last_became_active":"2024-03-20T09:13:27.087686+0000","last_became_peered":"2024-03-20T09:13:27.087686+0000","last_unstale":"2024-03-20T09:15:56.293278+0000","last_undegraded":"2024-03-20T09:15:56.293278+0000","last_fullsized":"2024-03-20T09:15:56.293278+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:14:46.139576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:14:46.139576+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:08.971745+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"568'9626","reported_seq":9815,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:59.929451+0000","last_change":"2024-03-20T09:15:05.380631+0000","last_active":"2024-03-20T09:15:59.929451+0000","last_peered":"2024-03-20T09:15:59.929451+0000","last_clean":"2024-03-20T09:15:59.929451+0000","last_became_active":"2024-03-20T09:13:22.063673+0000","last_became_peered":"2024-03-20T09:13:22.063673+0000","last_unstale":"2024-03-20T09:15:59.929451+0000","last_undegraded":"2024-03-20T09:15:59.929451+0000","last_fullsized":"2024-03-20T09:15:59.929451+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"553'7531","last_scrub_stamp":"2024-03-20T09:15:05.380547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:15:05.380547+0000","objects_scrubbed":7413,"log_size":9626,"log_dups_size":0,"ondisk_log_size":9626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:25.174578+0000","scrub_duration":962,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348127232,"num_objects":5312,"num_object_clones":0,"num_object_copies":15936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9626,"num_write_kb":478016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"20'137","reported_seq":684,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:56.292087+0000","last_change":"2024-03-20T09:15:07.365971+0000","last_active":"2024-03-20T09:15:56.292087+0000","last_peered":"2024-03-20T09:15:56.292087+0000","last_clean":"2024-03-20T09:15:56.292087+0000","last_became_active":"2024-03-20T09:12:14.307864+0000","last_became_peered":"2024-03-20T09:12:14.307864+0000","last_unstale":"2024-03-20T09:15:56.292087+0000","last_undegraded":"2024-03-20T09:15:56.292087+0000","last_fullsized":"2024-03-20T09:15:56.292087+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:07.365857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:15:07.365857+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:35.457364+0000","scrub_duration":268,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"20'137","reported_seq":1000,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:56.292758+0000","last_change":"2024-03-20T09:14:40.236846+0000","last_active":"2024-03-20T09:15:56.292758+0000","last_peered":"2024-03-20T09:15:56.292758+0000","last_clean":"2024-03-20T09:15:56.292758+0000","last_became_active":"2024-03-20T09:10:34.096758+0000","last_became_peered":"2024-03-20T09:10:34.096758+0000","last_unstale":"2024-03-20T09:15:56.292758+0000","last_undegraded":"2024-03-20T09:15:56.292758+0000","last_fullsized":"2024-03-20T09:15:56.292758+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:14:40.236685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:14:40.236685+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:04.722728+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"20'137","reported_seq":1073,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:56.292606+0000","last_change":"2024-03-20T09:15:05.139462+0000","last_active":"2024-03-20T09:15:56.292606+0000","last_peered":"2024-03-20T09:15:56.292606+0000","last_clean":"2024-03-20T09:15:56.292606+0000","last_became_active":"2024-03-20T09:04:04.103645+0000","last_became_peered":"2024-03-20T09:04:04.103645+0000","last_unstale":"2024-03-20T09:15:56.292606+0000","last_undegraded":"2024-03-20T09:15:56.292606+0000","last_fullsized":"2024-03-20T09:15:56.292606+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:05.139344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:15:05.139344+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:05.673418+0000","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"20'137","reported_seq":526,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:56.292655+0000","last_change":"2024-03-20T09:14:45.137392+0000","last_active":"2024-03-20T09:15:56.292655+0000","last_peered":"2024-03-20T09:15:56.292655+0000","last_clean":"2024-03-20T09:15:56.292655+0000","last_became_active":"2024-03-20T09:10:34.097191+0000","last_became_peered":"2024-03-20T09:10:34.097191+0000","last_unstale":"2024-03-20T09:15:56.292655+0000","last_undegraded":"2024-03-20T09:15:56.292655+0000","last_fullsized":"2024-03-20T09:15:56.292655+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:14:45.137315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:14:45.137315+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:07.569642+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"20'137","reported_seq":937,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:56.294023+0000","last_change":"2024-03-20T09:14:57.929482+0000","last_active":"2024-03-20T09:15:56.294023+0000","last_peered":"2024-03-20T09:15:56.294023+0000","last_clean":"2024-03-20T09:15:56.294023+0000","last_became_active":"2024-03-20T09:13:27.087859+0000","last_became_peered":"2024-03-20T09:13:27.087859+0000","last_unstale":"2024-03-20T09:15:56.294023+0000","last_undegraded":"2024-03-20T09:15:56.294023+0000","last_fullsized":"2024-03-20T09:15:56.294023+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:14:57.929339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:14:57.929339+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:02.614879+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,3],"acting":[0,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"20'137","reported_seq":761,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:56.292138+0000","last_change":"2024-03-20T09:14:49.747607+0000","last_active":"2024-03-20T09:15:56.292138+0000","last_peered":"2024-03-20T09:15:56.292138+0000","last_clean":"2024-03-20T09:15:56.292138+0000","last_became_active":"2024-03-20T09:13:27.087760+0000","last_became_peered":"2024-03-20T09:13:27.087760+0000","last_unstale":"2024-03-20T09:15:56.292138+0000","last_undegraded":"2024-03-20T09:15:56.292138+0000","last_fullsized":"2024-03-20T09:15:56.292138+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:14:49.747468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:14:49.747468+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:15.011120+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"20'137","reported_seq":1088,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:56.299315+0000","last_change":"2024-03-20T09:15:54.388457+0000","last_active":"2024-03-20T09:15:56.299315+0000","last_peered":"2024-03-20T09:15:56.299315+0000","last_clean":"2024-03-20T09:15:56.299315+0000","last_became_active":"2024-03-20T09:10:35.095339+0000","last_became_peered":"2024-03-20T09:10:35.095339+0000","last_unstale":"2024-03-20T09:15:56.299315+0000","last_undegraded":"2024-03-20T09:15:56.299315+0000","last_fullsized":"2024-03-20T09:15:56.299315+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:54.388308+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:23.640533+0000","last_clean_scrub_stamp":"2024-03-20T09:15:54.388308+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:09.582019+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"20'137","reported_seq":1149,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:56.304893+0000","last_change":"2024-03-20T09:14:50.041856+0000","last_active":"2024-03-20T09:15:56.304893+0000","last_peered":"2024-03-20T09:15:56.304893+0000","last_clean":"2024-03-20T09:15:56.304893+0000","last_became_active":"2024-03-20T09:04:14.208448+0000","last_became_peered":"2024-03-20T09:04:14.208448+0000","last_unstale":"2024-03-20T09:15:56.304893+0000","last_undegraded":"2024-03-20T09:15:56.304893+0000","last_fullsized":"2024-03-20T09:15:56.304893+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:14:50.041715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:14:50.041715+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:14.138911+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"20'137","reported_seq":422,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:56.309693+0000","last_change":"2024-03-20T09:15:55.844234+0000","last_active":"2024-03-20T09:15:56.309693+0000","last_peered":"2024-03-20T09:15:56.309693+0000","last_clean":"2024-03-20T09:15:56.309693+0000","last_became_active":"2024-03-20T09:12:14.307581+0000","last_became_peered":"2024-03-20T09:12:14.307581+0000","last_unstale":"2024-03-20T09:15:56.309693+0000","last_undegraded":"2024-03-20T09:15:56.309693+0000","last_fullsized":"2024-03-20T09:15:56.309693+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:55.844132+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:07:06.480160+0000","last_clean_scrub_stamp":"2024-03-20T09:15:55.844132+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:03.670100+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"20'137","reported_seq":693,"reported_epoch":568,"state":"active+clean","last_fresh":"2024-03-20T09:15:56.291916+0000","last_change":"2024-03-20T09:15:51.692096+0000","last_active":"2024-03-20T09:15:56.291916+0000","last_peered":"2024-03-20T09:15:56.291916+0000","last_clean":"2024-03-20T09:15:56.291916+0000","last_became_active":"2024-03-20T09:13:20.074439+0000","last_became_peered":"2024-03-20T09:13:20.074439+0000","last_unstale":"2024-03-20T09:15:56.291916+0000","last_undegraded":"2024-03-20T09:15:56.291916+0000","last_fullsized":"2024-03-20T09:15:56.291916+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:51.691974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:15:51.691974+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:56.756163+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,11],"acting":[1,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":5714935832,"num_objects":87204,"num_object_clones":0,"num_object_copies":261612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87204,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154630,"num_write_kb":7738625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":922,"num_bytes_recovered":60424192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17096388608,"data_stored":17096376392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156261,"ondisk_log_size":156261,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4247,"ondisk_log_size":4247,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860655,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3200396,"kb_used_data":2786052,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":90544756,"statfs":{"total":95995035648,"available":92717830144,"internally_reserved":0,"allocated":2852917248,"data_stored":2852665449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":424278468},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":89,"seq":382252089589,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1649800,"kb_used_data":1379840,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":92095352,"statfs":{"total":95995035648,"available":94305640448,"internally_reserved":0,"allocated":1412956160,"data_stored":1412708433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":276429248},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":0,"seq":343597383932,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":70,"seq":300647710977,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":914376,"kb_used_data":684672,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":92830776,"statfs":{"total":95995035648,"available":95058714624,"internally_reserved":0,"allocated":701104128,"data_stored":700856401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":235207110},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":60,"seq":257698038022,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1845840,"kb_used_data":1377352,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":91899312,"statfs":{"total":95995035648,"available":94104895488,"internally_reserved":0,"allocated":1410408448,"data_stored":1410157169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":479721930},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332363,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2437256,"kb_used_data":2054848,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":91307896,"statfs":{"total":95995035648,"available":93499285504,"internally_reserved":0,"allocated":2104164352,"data_stored":2103916625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":391576006},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575140,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3669320,"kb_used_data":3222660,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":90075832,"statfs":{"total":95995035648,"available":92237651968,"internally_reserved":0,"allocated":3300003840,"data_stored":3299752041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":456980934},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738659,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1858680,"kb_used_data":1426288,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":91886472,"statfs":{"total":95995035648,"available":94091747328,"internally_reserved":0,"allocated":1460518912,"data_stored":1460224893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411614,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1344976,"kb_used_data":1056712,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":92400176,"statfs":{"total":95995035648,"available":94617780224,"internally_reserved":0,"allocated":1082073088,"data_stored":1081821809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149978,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2093832,"kb_used_data":1737792,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":91651320,"statfs":{"total":95995035648,"available":93850951680,"internally_reserved":0,"allocated":1779499008,"data_stored":1779251281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":364575175},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953752,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":443,"seq":1902670512194,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1210508,"kb_used_data":1062408,"kb_used_omap":181,"kb_used_meta":147786,"kb_avail":92534644,"statfs":{"total":95995035648,"available":94755475456,"internally_reserved":0,"allocated":1087905792,"data_stored":1087654513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":186131,"internal_metadata":151333101},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1447038976,"data_stored":1447034904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3277066240,"data_stored":3277062168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1077018624,"data_stored":1077018624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1764556800,"data_stored":1764556800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1080492032,"data_stored":1080492032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2100101120,"data_stored":2100101120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1405943808,"data_stored":1405943808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":697630720,"data_stored":697630720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1402208256,"data_stored":1402208256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2844332032,"data_stored":2844327960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:16:03.913 INFO:tasks.thrashosds.thrasher:Setting 1.3 to [4, 0, 5, 9, 3, 11] 2024-03-20T09:16:03.913 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '1.3', '4', '0', '5', '9', '3', '11'] 2024-03-20T09:16:03.913 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pg-upmap-items 1.3 4 0 5 9 3 11 2024-03-20T09:16:04.002 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:16:04.032 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:04.054 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:16:04.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:04.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:16:04.160 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:16:04.255 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:04.324 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:16:04.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:04.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:16:04.402 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:04 smithi026 ceph-mon[68309]: osdmap e569: 12 total, 12 up, 10 in 2024-03-20T09:16:04.402 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:04 smithi026 ceph-mon[68309]: pgmap v1353: 47 pgs: 47 active+clean; 5.2 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 744 op/s 2024-03-20T09:16:04.433 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:16:04.462 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:04.489 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:16:04.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:04.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:16:04.595 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:16:04.624 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:04.652 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:16:04.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:04 smithi115 ceph-mon[72476]: osdmap e569: 12 total, 12 up, 10 in 2024-03-20T09:16:04.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:04 smithi115 ceph-mon[72476]: pgmap v1353: 47 pgs: 47 active+clean; 5.2 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 744 op/s 2024-03-20T09:16:04.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:04.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:16:04.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:04 smithi122 ceph-mon[73495]: osdmap e569: 12 total, 12 up, 10 in 2024-03-20T09:16:04.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:04 smithi122 ceph-mon[73495]: pgmap v1353: 47 pgs: 47 active+clean; 5.2 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 744 op/s 2024-03-20T09:16:04.756 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:16:04.831 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:16:04.848 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:04.861 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:16:04.861 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:16:05.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:05 smithi115 ceph-mon[72476]: 4.c scrub starts 2024-03-20T09:16:05.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:05 smithi115 ceph-mon[72476]: 1.5 scrub starts 2024-03-20T09:16:05.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:05 smithi115 ceph-mon[72476]: 4.a scrub starts 2024-03-20T09:16:05.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:05 smithi115 ceph-mon[72476]: 1.5 scrub ok 2024-03-20T09:16:05.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:05 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3144696399' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:16:05.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:05 smithi122 ceph-mon[73495]: 4.c scrub starts 2024-03-20T09:16:05.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:05 smithi122 ceph-mon[73495]: 1.5 scrub starts 2024-03-20T09:16:05.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:05 smithi122 ceph-mon[73495]: 4.a scrub starts 2024-03-20T09:16:05.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:05 smithi122 ceph-mon[73495]: 1.5 scrub ok 2024-03-20T09:16:05.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:05 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3144696399' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:16:05.727 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:05 smithi026 ceph-mon[68309]: 4.c scrub starts 2024-03-20T09:16:05.727 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:05 smithi026 ceph-mon[68309]: 1.5 scrub starts 2024-03-20T09:16:05.727 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:05 smithi026 ceph-mon[68309]: 4.a scrub starts 2024-03-20T09:16:05.727 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:05 smithi026 ceph-mon[68309]: 1.5 scrub ok 2024-03-20T09:16:05.728 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:05 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3144696399' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:16:05.962 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:16:06.340 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:16:06.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:06.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:16:06.445 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:16:06.553 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:06.572 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:16:06.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:06.575 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:16:06.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:06 smithi115 ceph-mon[72476]: 1.1d scrub starts 2024-03-20T09:16:06.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:06 smithi115 ceph-mon[72476]: 1.1d scrub ok 2024-03-20T09:16:06.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:06 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:16:06.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:06 smithi115 ceph-mon[72476]: Cluster is now healthy 2024-03-20T09:16:06.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:06 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3144696399' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:16:06.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:06 smithi115 ceph-mon[72476]: osdmap e570: 12 total, 12 up, 10 in 2024-03-20T09:16:06.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:06 smithi115 ceph-mon[72476]: 4.c scrub ok 2024-03-20T09:16:06.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:06 smithi115 ceph-mon[72476]: pgmap v1355: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.1 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 790 op/s 2024-03-20T09:16:06.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:06 smithi115 ceph-mon[72476]: 4.a scrub ok 2024-03-20T09:16:06.675 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:16:06.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:06 smithi122 ceph-mon[73495]: 1.1d scrub starts 2024-03-20T09:16:06.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:06 smithi122 ceph-mon[73495]: 1.1d scrub ok 2024-03-20T09:16:06.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:06 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:16:06.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:06 smithi122 ceph-mon[73495]: Cluster is now healthy 2024-03-20T09:16:06.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:06 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3144696399' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:16:06.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:06 smithi122 ceph-mon[73495]: osdmap e570: 12 total, 12 up, 10 in 2024-03-20T09:16:06.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:06 smithi122 ceph-mon[73495]: 4.c scrub ok 2024-03-20T09:16:06.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:06 smithi122 ceph-mon[73495]: pgmap v1355: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.1 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 790 op/s 2024-03-20T09:16:06.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:06 smithi122 ceph-mon[73495]: 4.a scrub ok 2024-03-20T09:16:06.702 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:06.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:06 smithi026 ceph-mon[68309]: 1.1d scrub starts 2024-03-20T09:16:06.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:06 smithi026 ceph-mon[68309]: 1.1d scrub ok 2024-03-20T09:16:06.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:06 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:16:06.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:06 smithi026 ceph-mon[68309]: Cluster is now healthy 2024-03-20T09:16:06.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:06 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3144696399' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:16:06.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:06 smithi026 ceph-mon[68309]: osdmap e570: 12 total, 12 up, 10 in 2024-03-20T09:16:06.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:06 smithi026 ceph-mon[68309]: 4.c scrub ok 2024-03-20T09:16:06.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:06 smithi026 ceph-mon[68309]: pgmap v1355: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.1 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 790 op/s 2024-03-20T09:16:06.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:06 smithi026 ceph-mon[68309]: 4.a scrub ok 2024-03-20T09:16:06.722 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:16:06.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:06.725 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:16:06.825 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:16:06.858 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:06.878 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:16:06.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:06.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:16:06.984 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:16:07.062 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:07.419 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:07 smithi026 ceph-mon[68309]: 4.6 scrub starts 2024-03-20T09:16:07.419 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:07 smithi026 ceph-mon[68309]: 1.11 scrub starts 2024-03-20T09:16:07.420 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:07 smithi026 ceph-mon[68309]: 1.11 scrub ok 2024-03-20T09:16:07.420 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:07 smithi026 ceph-mon[68309]: 1.12 scrub starts 2024-03-20T09:16:07.420 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:07 smithi026 ceph-mon[68309]: 1.12 scrub ok 2024-03-20T09:16:07.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:07 smithi122 ceph-mon[73495]: 4.6 scrub starts 2024-03-20T09:16:07.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:07 smithi122 ceph-mon[73495]: 1.11 scrub starts 2024-03-20T09:16:07.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:07 smithi122 ceph-mon[73495]: 1.11 scrub ok 2024-03-20T09:16:07.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:07 smithi122 ceph-mon[73495]: 1.12 scrub starts 2024-03-20T09:16:07.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:07 smithi122 ceph-mon[73495]: 1.12 scrub ok 2024-03-20T09:16:07.767 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:16:07.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:07.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:16:07.870 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:16:07.902 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:07.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:07 smithi115 ceph-mon[72476]: 4.6 scrub starts 2024-03-20T09:16:07.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:07 smithi115 ceph-mon[72476]: 1.11 scrub starts 2024-03-20T09:16:07.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:07 smithi115 ceph-mon[72476]: 1.11 scrub ok 2024-03-20T09:16:07.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:07 smithi115 ceph-mon[72476]: 1.12 scrub starts 2024-03-20T09:16:07.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:07 smithi115 ceph-mon[72476]: 1.12 scrub ok 2024-03-20T09:16:07.929 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:16:07.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:07.933 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:16:08.034 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:16:08.136 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:08.431 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:08 smithi026 ceph-mon[68309]: 4.6 scrub ok 2024-03-20T09:16:08.431 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:08 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:16:08.431 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:08 smithi026 ceph-mon[68309]: pgmap v1356: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 948 op/s 2024-03-20T09:16:08.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:08 smithi122 ceph-mon[73495]: 4.6 scrub ok 2024-03-20T09:16:08.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:08 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:16:08.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:08 smithi122 ceph-mon[73495]: pgmap v1356: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 948 op/s 2024-03-20T09:16:08.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:08 smithi115 ceph-mon[72476]: 4.6 scrub ok 2024-03-20T09:16:08.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:08 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:16:08.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:08 smithi115 ceph-mon[72476]: pgmap v1356: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 948 op/s 2024-03-20T09:16:09.220 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:16:09.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:09.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:16:09.241 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-20T09:16:09.326 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:16:09.354 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:09.478 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:09 smithi026 ceph-mon[68309]: 1.13 scrub starts 2024-03-20T09:16:09.478 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:09 smithi026 ceph-mon[68309]: 1.13 scrub ok 2024-03-20T09:16:09.540 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:16:09.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:09 smithi115 ceph-mon[72476]: 1.13 scrub starts 2024-03-20T09:16:09.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:09 smithi115 ceph-mon[72476]: 1.13 scrub ok 2024-03-20T09:16:09.700 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:09 smithi122 ceph-mon[73495]: 1.13 scrub starts 2024-03-20T09:16:09.700 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:09 smithi122 ceph-mon[73495]: 1.13 scrub ok 2024-03-20T09:16:09.796 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:16:09.890 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:16:10.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:10 smithi026 ceph-mon[68309]: pgmap v1357: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 696 op/s 2024-03-20T09:16:10.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:10 smithi115 ceph-mon[72476]: pgmap v1357: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 696 op/s 2024-03-20T09:16:10.909 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:16:10.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:10.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:16:10.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:10 smithi122 ceph-mon[73495]: pgmap v1357: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.0 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 696 op/s 2024-03-20T09:16:11.014 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:16:11.056 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:11.082 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:16:11.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:11.091 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:16:11.192 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:16:11.252 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:11.387 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:16:11.436 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:16:11.478 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:11 smithi026 ceph-mon[68309]: 1.1c scrub starts 2024-03-20T09:16:11.479 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:11 smithi026 ceph-mon[68309]: 1.1c scrub ok 2024-03-20T09:16:11.479 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:11 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1774390493' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:16:11.480 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:16:11.533 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:16:11.564 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:16:11.617 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:16:11.657 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:16:11.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:11 smithi115 ceph-mon[72476]: 1.1c scrub starts 2024-03-20T09:16:11.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:11 smithi115 ceph-mon[72476]: 1.1c scrub ok 2024-03-20T09:16:11.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:11 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1774390493' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:16:11.698 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:16:11.741 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:16:11.774 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:16:11.820 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:16:11.859 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:16:11.865 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:16:11.865 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:16:11.866 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:16:11.866 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:16:11.866 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:16:11.866 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:16:11.866 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:16:11.866 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:16:11.866 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:16:11.866 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:16:11.866 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:16:11.866 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:16:11.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:11 smithi122 ceph-mon[73495]: 1.1c scrub starts 2024-03-20T09:16:11.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:11 smithi122 ceph-mon[73495]: 1.1c scrub ok 2024-03-20T09:16:11.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:11 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1774390493' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:16:12.783 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:12 smithi026 ceph-mon[68309]: pgmap v1358: 47 pgs: 47 active+clean; 4.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 693 op/s 2024-03-20T09:16:12.783 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:12 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:16:12.783 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:12 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:16:12.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:12 smithi115 ceph-mon[72476]: pgmap v1358: 47 pgs: 47 active+clean; 4.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 693 op/s 2024-03-20T09:16:12.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:12 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:16:12.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:12 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:16:12.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:12 smithi122 ceph-mon[73495]: pgmap v1358: 47 pgs: 47 active+clean; 4.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 693 op/s 2024-03-20T09:16:12.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:12 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:16:12.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:12 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:16:13.476 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:16:13.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:13.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:16:13.592 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:16:13.623 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:13.644 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:16:13.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:13.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:16:13.749 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:16:13.789 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:13 smithi122 ceph-mon[73495]: 1.0 scrub starts 2024-03-20T09:16:13.789 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:13 smithi122 ceph-mon[73495]: 1.0 scrub ok 2024-03-20T09:16:13.789 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:13 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:16:13.789 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:13 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:16:13.789 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:13 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:16:13.790 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:13.812 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:16:13.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:13.815 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:16:13.822 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:13 smithi026 ceph-mon[68309]: 1.0 scrub starts 2024-03-20T09:16:13.822 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:13 smithi026 ceph-mon[68309]: 1.0 scrub ok 2024-03-20T09:16:13.822 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:13 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:16:13.823 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:13 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:16:13.823 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:13 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:16:13.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:13 smithi115 ceph-mon[72476]: 1.0 scrub starts 2024-03-20T09:16:13.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:13 smithi115 ceph-mon[72476]: 1.0 scrub ok 2024-03-20T09:16:13.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:13 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:16:13.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:13 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:16:13.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:13 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:16:13.916 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:16:14.023 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:14.080 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:16:14.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:14.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:16:14.193 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:16:14.519 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:14.552 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:16:14.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:14.560 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:16:14.660 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:16:14.722 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:14.898 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:16:14.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:14 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:16:14.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:14 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:16:14.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:14 smithi115 ceph-mon[72476]: 1.1 scrub starts 2024-03-20T09:16:14.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:14 smithi115 ceph-mon[72476]: 1.1 scrub ok 2024-03-20T09:16:14.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:14 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:16:14.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:14 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:16:14.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:14 smithi115 ceph-mon[72476]: pgmap v1359: 47 pgs: 47 active+clean; 4.8 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 744 op/s 2024-03-20T09:16:14.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:14 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:16:14.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:14 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:16:14.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:14 smithi122 ceph-mon[73495]: 1.1 scrub starts 2024-03-20T09:16:14.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:14 smithi122 ceph-mon[73495]: 1.1 scrub ok 2024-03-20T09:16:14.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:14 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:16:14.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:14 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:16:14.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:14 smithi122 ceph-mon[73495]: pgmap v1359: 47 pgs: 47 active+clean; 4.8 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 744 op/s 2024-03-20T09:16:14.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:14 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:16:14.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:14 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:16:14.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:14 smithi026 ceph-mon[68309]: 1.1 scrub starts 2024-03-20T09:16:14.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:14 smithi026 ceph-mon[68309]: 1.1 scrub ok 2024-03-20T09:16:14.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:14 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:16:14.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:14 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:16:14.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:14 smithi026 ceph-mon[68309]: pgmap v1359: 47 pgs: 47 active+clean; 4.8 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 744 op/s 2024-03-20T09:16:15.105 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:16:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:16:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25690.0:167043 4.f 4:fa090df0:::benchmark_data_smithi122_145698_object46124:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e571)", 2024-03-20T09:16:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:16:15.094260+0000", 2024-03-20T09:16:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0081570979999999998, 2024-03-20T09:16:15.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0081952199999999996, 2024-03-20T09:16:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:16:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:16:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:16:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25690", 2024-03-20T09:16:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:16:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 167043 2024-03-20T09:16:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:16:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:16:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:15.094260+0000", 2024-03-20T09:16:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:16:15.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:16:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:15.094260+0000", 2024-03-20T09:16:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:16:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:16:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:15.094261+0000", 2024-03-20T09:16:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.344e-06 2024-03-20T09:16:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:16:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:15.094267+0000", 2024-03-20T09:16:15.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2680000000000002e-06 2024-03-20T09:16:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:16:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:15.094278+0000", 2024-03-20T09:16:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1374e-05 2024-03-20T09:16:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:16:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:15.094286+0000", 2024-03-20T09:16:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5650000000000004e-06 2024-03-20T09:16:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:16:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:15.094337+0000", 2024-03-20T09:16:15.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0997e-05 2024-03-20T09:16:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:16:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:15.094450+0000", 2024-03-20T09:16:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011308500000000001 2024-03-20T09:16:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,11", 2024-03-20T09:16:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:15.094559+0000", 2024-03-20T09:16:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000109207 2024-03-20T09:16:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:15.109 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:16:15.110 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 1 2024-03-20T09:16:15.110 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:16:15.841 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:16:15.888 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:15 smithi026 ceph-mon[68309]: 1.c deep-scrub starts 2024-03-20T09:16:15.888 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:15 smithi026 ceph-mon[68309]: 1.c deep-scrub ok 2024-03-20T09:16:15.888 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:15 smithi026 ceph-mon[68309]: osdmap e571: 12 total, 12 up, 10 in 2024-03-20T09:16:15.888 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:15 smithi026 ceph-mon[68309]: 1.16 deep-scrub starts 2024-03-20T09:16:15.889 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:15 smithi026 ceph-mon[68309]: 1.16 deep-scrub ok 2024-03-20T09:16:15.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:15 smithi115 ceph-mon[72476]: 1.c deep-scrub starts 2024-03-20T09:16:15.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:15 smithi115 ceph-mon[72476]: 1.c deep-scrub ok 2024-03-20T09:16:15.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:15 smithi115 ceph-mon[72476]: osdmap e571: 12 total, 12 up, 10 in 2024-03-20T09:16:15.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:15 smithi115 ceph-mon[72476]: 1.16 deep-scrub starts 2024-03-20T09:16:15.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:15 smithi115 ceph-mon[72476]: 1.16 deep-scrub ok 2024-03-20T09:16:15.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:15 smithi122 ceph-mon[73495]: 1.c deep-scrub starts 2024-03-20T09:16:15.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:15 smithi122 ceph-mon[73495]: 1.c deep-scrub ok 2024-03-20T09:16:15.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:15 smithi122 ceph-mon[73495]: osdmap e571: 12 total, 12 up, 10 in 2024-03-20T09:16:15.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:15 smithi122 ceph-mon[73495]: 1.16 deep-scrub starts 2024-03-20T09:16:15.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:15 smithi122 ceph-mon[73495]: 1.16 deep-scrub ok 2024-03-20T09:16:16.576 INFO:teuthology.orchestra.run.smithi026.stderr:set 1.3 pg_upmap_items mapping to [4->0,5->9,3->11] 2024-03-20T09:16:16.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:16 smithi115 ceph-mon[72476]: 1.14 deep-scrub starts 2024-03-20T09:16:16.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:16 smithi115 ceph-mon[72476]: 1.14 deep-scrub ok 2024-03-20T09:16:16.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:16 smithi115 ceph-mon[72476]: pgmap v1361: 47 pgs: 47 active+clean; 4.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 755 op/s 2024-03-20T09:16:16.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:16 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1703528226' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.3", "id": [4, 0, 5, 9, 3, 11]}]: dispatch 2024-03-20T09:16:16.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:16 smithi122 ceph-mon[73495]: 1.14 deep-scrub starts 2024-03-20T09:16:16.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:16 smithi122 ceph-mon[73495]: 1.14 deep-scrub ok 2024-03-20T09:16:16.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:16 smithi122 ceph-mon[73495]: pgmap v1361: 47 pgs: 47 active+clean; 4.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 755 op/s 2024-03-20T09:16:16.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:16 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1703528226' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.3", "id": [4, 0, 5, 9, 3, 11]}]: dispatch 2024-03-20T09:16:16.977 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:16 smithi026 ceph-mon[68309]: 1.14 deep-scrub starts 2024-03-20T09:16:16.977 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:16 smithi026 ceph-mon[68309]: 1.14 deep-scrub ok 2024-03-20T09:16:16.978 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:16 smithi026 ceph-mon[68309]: pgmap v1361: 47 pgs: 47 active+clean; 4.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 755 op/s 2024-03-20T09:16:16.978 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:16 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1703528226' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.3", "id": [4, 0, 5, 9, 3, 11]}]: dispatch 2024-03-20T09:16:17.002 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:16:17.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:17.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:16:17.111 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:16:17.142 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:17.171 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:16:17.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:17.174 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:16:17.275 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:16:17.304 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:17.329 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:16:17.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:17.338 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:16:17.438 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:16:17.549 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:17.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:17 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1703528226' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.3", "id": [4, 0, 5, 9, 3, 11]}]': finished 2024-03-20T09:16:17.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:17 smithi115 ceph-mon[72476]: osdmap e572: 12 total, 12 up, 10 in 2024-03-20T09:16:17.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:17 smithi115 ceph-mon[72476]: 4.0 scrub starts 2024-03-20T09:16:17.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:17 smithi115 ceph-mon[72476]: 1.10 scrub starts 2024-03-20T09:16:17.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:17 smithi115 ceph-mon[72476]: 1.10 scrub ok 2024-03-20T09:16:17.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:17 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1703528226' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.3", "id": [4, 0, 5, 9, 3, 11]}]': finished 2024-03-20T09:16:17.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:17 smithi122 ceph-mon[73495]: osdmap e572: 12 total, 12 up, 10 in 2024-03-20T09:16:17.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:17 smithi122 ceph-mon[73495]: 4.0 scrub starts 2024-03-20T09:16:17.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:17 smithi122 ceph-mon[73495]: 1.10 scrub starts 2024-03-20T09:16:17.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:17 smithi122 ceph-mon[73495]: 1.10 scrub ok 2024-03-20T09:16:17.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:17 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1703528226' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.3", "id": [4, 0, 5, 9, 3, 11]}]': finished 2024-03-20T09:16:17.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:17 smithi026 ceph-mon[68309]: osdmap e572: 12 total, 12 up, 10 in 2024-03-20T09:16:17.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:17 smithi026 ceph-mon[68309]: 4.0 scrub starts 2024-03-20T09:16:17.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:17 smithi026 ceph-mon[68309]: 1.10 scrub starts 2024-03-20T09:16:17.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:17 smithi026 ceph-mon[68309]: 1.10 scrub ok 2024-03-20T09:16:18.435 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:16:18.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:18.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:16:18.541 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:16:18.576 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:18.601 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:16:18.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:18.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:16:18.704 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:16:18.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:18 smithi026 ceph-mon[68309]: pgmap v1363: 47 pgs: 47 active+clean; 4.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 947 op/s 2024-03-20T09:16:18.736 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:18.759 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:16:18.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:18.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:16:18.863 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:16:18.904 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:18.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:18 smithi115 ceph-mon[72476]: pgmap v1363: 47 pgs: 47 active+clean; 4.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 947 op/s 2024-03-20T09:16:18.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:18 smithi122 ceph-mon[73495]: pgmap v1363: 47 pgs: 47 active+clean; 4.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 947 op/s 2024-03-20T09:16:19.011 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:16:19.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:19.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:16:19.118 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:16:19.201 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:19.692 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:16:19.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:19.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:16:19.799 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:16:19.845 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:19 smithi026 ceph-mon[68309]: 4.b scrub starts 2024-03-20T09:16:19.845 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:19 smithi026 ceph-mon[68309]: 4.0 scrub ok 2024-03-20T09:16:19.845 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:19 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:16:19.845 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:19 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:16:19.846 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:19 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:16:19.846 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:19 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:16:19.849 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:19.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:19 smithi115 ceph-mon[72476]: 4.b scrub starts 2024-03-20T09:16:19.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:19 smithi115 ceph-mon[72476]: 4.0 scrub ok 2024-03-20T09:16:19.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:19 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:16:19.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:19 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:16:19.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:19 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:16:19.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:19 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:16:19.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:19 smithi122 ceph-mon[73495]: 4.b scrub starts 2024-03-20T09:16:19.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:19 smithi122 ceph-mon[73495]: 4.0 scrub ok 2024-03-20T09:16:19.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:19 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:16:19.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:19 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:16:19.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:19 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:16:19.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:19 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:16:20.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:20 smithi115 ceph-mon[72476]: 1.2 scrub starts 2024-03-20T09:16:20.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:20 smithi115 ceph-mon[72476]: 1.2 scrub ok 2024-03-20T09:16:20.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:20 smithi115 ceph-mon[72476]: osdmap e573: 12 total, 12 up, 10 in 2024-03-20T09:16:20.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:20 smithi115 ceph-mon[72476]: 4.b scrub ok 2024-03-20T09:16:20.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:20 smithi115 ceph-mon[72476]: pgmap v1365: 47 pgs: 47 active+clean; 4.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 728 op/s 2024-03-20T09:16:20.943 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:20 smithi026 ceph-mon[68309]: 1.2 scrub starts 2024-03-20T09:16:20.943 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:20 smithi026 ceph-mon[68309]: 1.2 scrub ok 2024-03-20T09:16:20.943 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:20 smithi026 ceph-mon[68309]: osdmap e573: 12 total, 12 up, 10 in 2024-03-20T09:16:20.943 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:20 smithi026 ceph-mon[68309]: 4.b scrub ok 2024-03-20T09:16:20.943 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:20 smithi026 ceph-mon[68309]: pgmap v1365: 47 pgs: 47 active+clean; 4.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 728 op/s 2024-03-20T09:16:20.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:20 smithi122 ceph-mon[73495]: 1.2 scrub starts 2024-03-20T09:16:20.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:20 smithi122 ceph-mon[73495]: 1.2 scrub ok 2024-03-20T09:16:20.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:20 smithi122 ceph-mon[73495]: osdmap e573: 12 total, 12 up, 10 in 2024-03-20T09:16:20.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:20 smithi122 ceph-mon[73495]: 4.b scrub ok 2024-03-20T09:16:20.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:20 smithi122 ceph-mon[73495]: pgmap v1365: 47 pgs: 47 active+clean; 4.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 728 op/s 2024-03-20T09:16:20.952 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:16:20.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:20.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:16:20.987 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-20T09:16:21.064 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:16:21.092 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:21.115 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:16:21.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:21.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:16:21.223 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:16:21.302 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:21.352 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:16:21.368 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:16:21.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:21.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:16:21.474 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:16:21.574 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:21.707 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:16:21.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:21.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:16:21.815 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:16:21.870 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:21 smithi026 ceph-mon[68309]: 1.8 scrub starts 2024-03-20T09:16:21.870 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:21 smithi026 ceph-mon[68309]: 1.8 scrub ok 2024-03-20T09:16:21.875 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:21.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:21 smithi115 ceph-mon[72476]: 1.8 scrub starts 2024-03-20T09:16:21.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:21 smithi115 ceph-mon[72476]: 1.8 scrub ok 2024-03-20T09:16:21.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:21 smithi122 ceph-mon[73495]: 1.8 scrub starts 2024-03-20T09:16:21.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:21 smithi122 ceph-mon[73495]: 1.8 scrub ok 2024-03-20T09:16:22.060 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:16:22.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:22.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:16:22.168 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:16:22.259 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:22.577 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:16:22.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:22.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:16:22.620 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:16:22.683 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:16:22.718 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:22.737 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:16:22.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:22.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:16:22.848 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:16:22.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:22 smithi115 ceph-mon[72476]: pgmap v1366: 47 pgs: 47 active+clean; 4.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 843 op/s 2024-03-20T09:16:22.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:22 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/908441303' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:16:22.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:22 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:16:22.944 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:22 smithi026 ceph-mon[68309]: pgmap v1366: 47 pgs: 47 active+clean; 4.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 843 op/s 2024-03-20T09:16:22.944 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:22 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/908441303' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:16:22.944 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:22 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:16:22.944 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:22.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:22 smithi122 ceph-mon[73495]: pgmap v1366: 47 pgs: 47 active+clean; 4.4 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 843 op/s 2024-03-20T09:16:22.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:22 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/908441303' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:16:22.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:22 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:16:22.996 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:16:22.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:22.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:16:23.006 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:16:23.100 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:16:23.141 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:23.175 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:16:23.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:23.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:16:23.282 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:16:23.382 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:23.545 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:16:23.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:23.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:16:23.654 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:16:23.683 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:23.720 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:16:23.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:23 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:16:23.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:23 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:16:23.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:23 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:16:23.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:23 smithi026 ceph-mon[68309]: 4.3 scrub starts 2024-03-20T09:16:23.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:23 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:16:23.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:23 smithi026 ceph-mon[68309]: osdmap e574: 12 total, 12 up, 10 in 2024-03-20T09:16:23.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:23 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:16:23.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:23 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:16:23.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:23.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:16:23.827 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:16:23.858 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:23 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:16:23.858 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:23 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:16:23.858 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:23 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:16:23.858 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:23 smithi115 ceph-mon[72476]: 4.3 scrub starts 2024-03-20T09:16:23.858 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:23 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:16:23.858 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:23 smithi115 ceph-mon[72476]: osdmap e574: 12 total, 12 up, 10 in 2024-03-20T09:16:23.858 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:23 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:16:23.859 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:23 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:16:23.860 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:23.886 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:16:23.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:23.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:16:23.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:23 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:16:23.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:23 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:16:23.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:23 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:16:23.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:23 smithi122 ceph-mon[73495]: 4.3 scrub starts 2024-03-20T09:16:23.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:23 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:16:23.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:23 smithi122 ceph-mon[73495]: osdmap e574: 12 total, 12 up, 10 in 2024-03-20T09:16:23.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:23 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:16:23.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:23 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:16:23.995 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:16:24.058 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:24.095 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:16:24.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:24.102 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:16:24.203 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:16:24.232 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:24.254 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:16:24.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:24.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:16:24.360 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:16:24.388 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:24.414 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:16:24.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:24.416 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:16:24.517 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:16:24.553 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:24.579 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:16:24.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:24.586 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:16:24.627 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:24 smithi026 ceph-mon[68309]: osdmap e575: 12 total, 12 up, 10 in 2024-03-20T09:16:24.627 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:24 smithi026 ceph-mon[68309]: pgmap v1369: 47 pgs: 47 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 639 op/s 2024-03-20T09:16:24.686 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:16:24.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:24 smithi122 ceph-mon[73495]: osdmap e575: 12 total, 12 up, 10 in 2024-03-20T09:16:24.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:24 smithi122 ceph-mon[73495]: pgmap v1369: 47 pgs: 47 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 639 op/s 2024-03-20T09:16:24.720 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:24.730 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 8, 10, 11, 3] out_osds: [9, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 5] 2024-03-20T09:16:24.730 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:16:24.730 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-20T09:16:24.730 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-20T09:16:24.742 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:16:24.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:24.746 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:16:24.846 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:16:24.878 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:24.900 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:16:24.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:24.904 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:16:24.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:24 smithi115 ceph-mon[72476]: osdmap e575: 12 total, 12 up, 10 in 2024-03-20T09:16:24.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:24 smithi115 ceph-mon[72476]: pgmap v1369: 47 pgs: 47 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 639 op/s 2024-03-20T09:16:25.005 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:16:25.091 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:25.585 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:16:25.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:25 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/58043942' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:16:25.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:25 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/58043942' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:16:25.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:25 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/58043942' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:16:26.322 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:16:26.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:26.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:16:26.429 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:16:26.463 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:26.489 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:16:26.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:26.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:16:26.514 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:16:26.514 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:16:26.514 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:16:26.514 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:16:26.514 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:16:26.514 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:16:26.514 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:16:26.514 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:16:26.514 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:16:26.515 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:16:26.515 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:16:26.515 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:16:26.515 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:16:26.515 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:16:26.515 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:16:26.515 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:16:26.515 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:16:26.515 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:16:26.515 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:16:26.515 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:16:26.516 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:16:26.516 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:16:26.516 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:16:26.516 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:16:26.593 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:16:26.625 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:26.658 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:16:26.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:26.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:16:26.765 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:16:26.798 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:26 smithi115 ceph-mon[72476]: pgmap v1370: 47 pgs: 47 active+clean; 4.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 933 op/s 2024-03-20T09:16:26.798 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:26 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:16:26.798 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:26 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:16:26.799 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:26.823 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:16:26.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:26.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:16:26.928 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:16:26.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:26 smithi122 ceph-mon[73495]: pgmap v1370: 47 pgs: 47 active+clean; 4.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 933 op/s 2024-03-20T09:16:26.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:26 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:16:26.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:26 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:16:26.964 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:26.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:26 smithi026 ceph-mon[68309]: pgmap v1370: 47 pgs: 47 active+clean; 4.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 933 op/s 2024-03-20T09:16:26.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:26 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:16:26.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:26 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:16:26.987 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:16:26.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:26.993 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:16:27.095 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:16:27.163 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:27.187 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:16:27.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:27.193 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:16:27.294 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:16:27.394 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:27.439 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:16:27.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:27.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:16:27.577 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:16:27.641 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:27.788 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:16:27.789 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:16:27.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:27.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:16:27.898 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:16:27.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:27 smithi115 ceph-mon[72476]: osdmap e576: 12 total, 12 up, 10 in 2024-03-20T09:16:27.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:27 smithi122 ceph-mon[73495]: osdmap e576: 12 total, 12 up, 10 in 2024-03-20T09:16:27.981 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:27 smithi026 ceph-mon[68309]: osdmap e576: 12 total, 12 up, 10 in 2024-03-20T09:16:27.981 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:28.062 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:16:28.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:28.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:16:28.169 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:16:28.202 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:28.224 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:16:28.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:28.227 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:16:28.328 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:16:28.362 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:28.385 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:16:28.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:28.392 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:16:28.493 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:16:28.522 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:28.544 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:16:28.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:28.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:16:28.651 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:16:28.685 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:28.716 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:16:28.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:28.719 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:16:28.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:28 smithi026 ceph-mon[68309]: pgmap v1372: 47 pgs: 47 active+clean; 4.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 925 op/s 2024-03-20T09:16:28.732 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:16:28.733 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:16:28.820 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:16:28.857 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:28.857 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:28 smithi115 ceph-mon[72476]: pgmap v1372: 47 pgs: 47 active+clean; 4.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 925 op/s 2024-03-20T09:16:28.887 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:16:28.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:28.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:16:28.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:28 smithi122 ceph-mon[73495]: pgmap v1372: 47 pgs: 47 active+clean; 4.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 925 op/s 2024-03-20T09:16:28.991 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:16:29.032 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:29.057 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:16:29.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:29.062 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:16:29.163 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:16:29.206 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:29.231 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:16:29.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:29.234 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:16:29.335 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:16:29.343 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:16:29.370 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:29.391 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:16:29.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:29.394 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:16:29.495 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:16:29.529 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:29.569 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:16:29.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:29.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:16:29.679 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:16:29.727 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:29.838 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:16:30.073 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-20T09:16:30.224 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:30 smithi026 ceph-mon[68309]: pgmap v1373: 47 pgs: 47 active+clean; 4.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 601 op/s 2024-03-20T09:16:30.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:30 smithi115 ceph-mon[72476]: pgmap v1373: 47 pgs: 47 active+clean; 4.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 601 op/s 2024-03-20T09:16:30.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:30 smithi122 ceph-mon[73495]: pgmap v1373: 47 pgs: 47 active+clean; 4.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 601 op/s 2024-03-20T09:16:30.919 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:16:30.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:30.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:16:31.034 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:16:31.069 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:31.093 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:16:31.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:31.096 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:16:31.197 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:16:31.230 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:31.274 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:16:31.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:31.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:16:31.379 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:16:31.431 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:31.456 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:16:31.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:31.460 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:16:31.561 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:16:31.622 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:31.656 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:16:31.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:31.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:16:31.765 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:16:31.818 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:31.946 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:16:31.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:31.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:16:32.055 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:16:32.089 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:32.111 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:16:32.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:32.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:16:32.216 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:32 smithi026 ceph-mon[68309]: pgmap v1374: 47 pgs: 47 active+clean; 3.9 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 815 op/s 2024-03-20T09:16:32.218 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:16:32.301 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:32.369 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:16:32.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:32.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:16:32.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:32 smithi115 ceph-mon[72476]: pgmap v1374: 47 pgs: 47 active+clean; 3.9 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 815 op/s 2024-03-20T09:16:32.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:32 smithi122 ceph-mon[73495]: pgmap v1374: 47 pgs: 47 active+clean; 3.9 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 815 op/s 2024-03-20T09:16:32.479 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:16:32.508 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:32.541 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:16:32.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:32.545 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:16:32.646 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:16:32.679 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:32.700 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:16:32.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:32.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:16:32.804 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:16:32.837 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:32.868 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:16:32.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:32.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:16:32.980 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:16:33.069 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:33.191 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:16:33.233 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:16:33.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:33.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:16:33.370 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:16:33.430 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:33 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2952775959' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:16:33.430 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:33 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:16:33.442 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:33.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:33 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2952775959' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:16:33.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:33 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:16:33.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:33 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2952775959' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:16:33.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:33 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:16:33.817 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:16:33.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:33.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:16:33.935 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:16:33.999 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:34.042 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:16:34.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:34.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:16:34.059 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:16:34.150 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:16:34.183 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:34.224 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:16:34.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:34.230 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:16:34.331 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:16:34.407 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:34.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:34 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:16:34.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:34 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:16:34.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:34 smithi122 ceph-mon[73495]: osdmap e577: 12 total, 12 up, 10 in 2024-03-20T09:16:34.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:34 smithi122 ceph-mon[73495]: pgmap v1376: 47 pgs: 47 active+clean; 3.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 741 op/s 2024-03-20T09:16:34.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:34 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:16:34.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:34 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:16:34.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:34 smithi026 ceph-mon[68309]: osdmap e577: 12 total, 12 up, 10 in 2024-03-20T09:16:34.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:34 smithi026 ceph-mon[68309]: pgmap v1376: 47 pgs: 47 active+clean; 3.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 741 op/s 2024-03-20T09:16:34.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:34 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:16:34.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:34 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:16:34.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:34 smithi115 ceph-mon[72476]: osdmap e577: 12 total, 12 up, 10 in 2024-03-20T09:16:34.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:34 smithi115 ceph-mon[72476]: pgmap v1376: 47 pgs: 47 active+clean; 3.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 741 op/s 2024-03-20T09:16:34.753 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:16:34.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:34.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:16:34.857 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:16:34.891 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:34.914 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:16:34.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:34.919 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:16:35.020 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:16:35.049 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:35.074 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:16:35.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:35.079 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:16:35.180 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:16:35.222 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:35.247 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:16:35.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:35.256 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:16:35.357 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:16:35.384 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:35.405 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:16:35.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:35.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:16:35.511 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:16:35.600 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:35.866 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:16:35.866 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:16:35.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:35.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:16:35.964 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:16:35.964 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:16:35.964 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 577, 2024-03-20T09:16:35.964 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "852fbb68-e696-11ee-95c9-87774f69a715", 2024-03-20T09:16:35.964 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-03-20T08:48:35.281088+0000", 2024-03-20T09:16:35.965 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-03-20T09:16:33.175520+0000", 2024-03-20T09:16:35.965 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2024-03-20T09:10:33.084151+0000", 2024-03-20T09:16:35.965 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2024-03-20T09:10:19.369629+0000", 2024-03-20T09:16:35.965 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-20T09:16:35.965 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5806080, 2024-03-20T09:16:35.965 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2024-03-20T09:16:35.965 INFO:teuthology.orchestra.run.smithi026.stdout: "nodeep-scrub", 2024-03-20T09:16:35.965 INFO:teuthology.orchestra.run.smithi026.stdout: "noscrub", 2024-03-20T09:16:35.965 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2024-03-20T09:16:35.965 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2024-03-20T09:16:35.965 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2024-03-20T09:16:35.965 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2024-03-20T09:16:35.965 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:16:35.966 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 27, 2024-03-20T09:16:35.966 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2024-03-20T09:16:35.966 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-20T09:16:35.966 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-20T09:16:35.966 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2024-03-20T09:16:35.966 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 4, 2024-03-20T09:16:35.966 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 12, 2024-03-20T09:16:35.966 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "luminous", 2024-03-20T09:16:35.966 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "luminous", 2024-03-20T09:16:35.966 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "squid", 2024-03-20T09:16:35.966 INFO:teuthology.orchestra.run.smithi026.stdout: "allow_crimson": false, 2024-03-20T09:16:35.966 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2024-03-20T09:16:35.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:35.966 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2024-03-20T09:16:35.967 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": ".mgr", 2024-03-20T09:16:35.967 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-03-20T08:52:17.442324+0000", 2024-03-20T09:16:35.967 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-03-20T09:16:35.967 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-03-20T09:16:35.967 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-03-20T09:16:35.967 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 3, 2024-03-20T09:16:35.967 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 2, 2024-03-20T09:16:35.967 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-03-20T09:16:35.967 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-03-20T09:16:35.967 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-03-20T09:16:35.967 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-03-20T09:16:35.967 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-20T09:16:35.967 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-03-20T09:16:35.968 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-03-20T09:16:35.968 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 31, 2024-03-20T09:16:35.968 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 31, 2024-03-20T09:16:35.968 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 31, 2024-03-20T09:16:35.968 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 31, 2024-03-20T09:16:35.968 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 31, 2024-03-20T09:16:35.968 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-03-20T09:16:35.968 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-03-20T09:16:35.968 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-03-20T09:16:35.968 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-03-20T09:16:35.968 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-03-20T09:16:35.968 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-03-20T09:16:35.968 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-03-20T09:16:35.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.968 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "360", 2024-03-20T09:16:35.968 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-03-20T09:16:35.969 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-20T09:16:35.969 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "358", 2024-03-20T09:16:35.969 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-03-20T09:16:35.969 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-03-20T09:16:35.969 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-03-20T09:16:35.969 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-03-20T09:16:35.969 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-03-20T09:16:35.969 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-03-20T09:16:35.969 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-03-20T09:16:35.969 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-03-20T09:16:35.969 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-03-20T09:16:35.969 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-03-20T09:16:35.969 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-03-20T09:16:35.970 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-03-20T09:16:35.970 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-03-20T09:16:35.970 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-03-20T09:16:35.970 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-03-20T09:16:35.970 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-20T09:16:35.970 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-20T09:16:35.970 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-20T09:16:35.970 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-03-20T09:16:35.970 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-03-20T09:16:35.970 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-03-20T09:16:35.970 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-03-20T09:16:35.970 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-03-20T09:16:35.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.971 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-03-20T09:16:35.971 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-03-20T09:16:35.971 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-03-20T09:16:35.971 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-03-20T09:16:35.971 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-03-20T09:16:35.971 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-03-20T09:16:35.971 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-03-20T09:16:35.971 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-03-20T09:16:35.971 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-03-20T09:16:35.971 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-03-20T09:16:35.971 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-03-20T09:16:35.971 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2024-03-20T09:16:35.971 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_max": 32, 2024-03-20T09:16:35.971 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2024-03-20T09:16:35.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.972 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-03-20T09:16:35.972 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": {} 2024-03-20T09:16:35.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.972 INFO:teuthology.orchestra.run.smithi026.stdout: "read_balance": { 2024-03-20T09:16:35.972 INFO:teuthology.orchestra.run.smithi026.stdout: "score_type": "Fair distribution", 2024-03-20T09:16:35.972 INFO:teuthology.orchestra.run.smithi026.stdout: "score_acting": 2.3399999141693115, 2024-03-20T09:16:35.972 INFO:teuthology.orchestra.run.smithi026.stdout: "score_stable": 2.3399999141693115, 2024-03-20T09:16:35.972 INFO:teuthology.orchestra.run.smithi026.stdout: "optimal_score": 0.82999998331069946, 2024-03-20T09:16:35.972 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_acting": 1.940000057220459, 2024-03-20T09:16:35.972 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_stable": 1.940000057220459, 2024-03-20T09:16:35.972 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity_weighted": 0.92150616645812988, 2024-03-20T09:16:35.972 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity": 0.92000001668930054, 2024-03-20T09:16:35.972 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-03-20T09:16:35.972 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:35.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:35.973 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 4, 2024-03-20T09:16:35.973 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "unique_pool_2", 2024-03-20T09:16:35.973 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-03-20T09:13:20.690041+0000", 2024-03-20T09:16:35.973 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-03-20T09:16:35.973 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-03-20T09:16:35.973 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-03-20T09:16:35.973 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 3, 2024-03-20T09:16:35.973 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 2, 2024-03-20T09:16:35.973 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-03-20T09:16:35.973 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-03-20T09:16:35.973 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-03-20T09:16:35.974 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-03-20T09:16:35.974 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-20T09:16:35.974 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-03-20T09:16:35.974 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-03-20T09:16:35.974 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 16, 2024-03-20T09:16:35.974 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 16, 2024-03-20T09:16:35.974 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 16, 2024-03-20T09:16:35.974 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 16, 2024-03-20T09:16:35.974 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 16, 2024-03-20T09:16:35.974 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-03-20T09:16:35.974 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-03-20T09:16:35.974 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-03-20T09:16:35.974 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-03-20T09:16:35.974 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-03-20T09:16:35.974 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-03-20T09:16:35.975 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-03-20T09:16:35.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.975 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "527", 2024-03-20T09:16:35.975 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-03-20T09:16:35.975 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-20T09:16:35.975 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2024-03-20T09:16:35.975 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-03-20T09:16:35.975 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-03-20T09:16:35.975 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-03-20T09:16:35.975 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-03-20T09:16:35.975 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-03-20T09:16:35.975 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-03-20T09:16:35.975 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-03-20T09:16:35.976 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-03-20T09:16:35.976 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-03-20T09:16:35.976 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-03-20T09:16:35.976 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-03-20T09:16:35.976 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-03-20T09:16:35.976 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-03-20T09:16:35.976 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-03-20T09:16:35.976 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-03-20T09:16:35.976 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-20T09:16:35.976 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-20T09:16:35.976 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-20T09:16:35.976 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-03-20T09:16:35.976 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-03-20T09:16:35.976 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-03-20T09:16:35.977 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-03-20T09:16:35.977 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-03-20T09:16:35.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.977 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-03-20T09:16:35.977 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-03-20T09:16:35.977 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-03-20T09:16:35.977 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-03-20T09:16:35.977 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-03-20T09:16:35.977 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-03-20T09:16:35.977 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-03-20T09:16:35.977 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-03-20T09:16:35.977 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-03-20T09:16:35.977 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-03-20T09:16:35.977 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-03-20T09:16:35.978 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2024-03-20T09:16:35.978 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-03-20T09:16:35.978 INFO:teuthology.orchestra.run.smithi026.stdout: "rados": {} 2024-03-20T09:16:35.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.978 INFO:teuthology.orchestra.run.smithi026.stdout: "read_balance": { 2024-03-20T09:16:35.978 INFO:teuthology.orchestra.run.smithi026.stdout: "score_type": "Fair distribution", 2024-03-20T09:16:35.978 INFO:teuthology.orchestra.run.smithi026.stdout: "score_acting": 3.0099999904632568, 2024-03-20T09:16:35.978 INFO:teuthology.orchestra.run.smithi026.stdout: "score_stable": 3.0099999904632568, 2024-03-20T09:16:35.978 INFO:teuthology.orchestra.run.smithi026.stdout: "optimal_score": 0.82999998331069946, 2024-03-20T09:16:35.978 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_acting": 2.5, 2024-03-20T09:16:35.978 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_stable": 2.5, 2024-03-20T09:16:35.978 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity_weighted": 0.92150616645812988, 2024-03-20T09:16:35.978 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity": 0.92000001668930054, 2024-03-20T09:16:35.979 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-03-20T09:16:35.979 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:35.979 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:35.979 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:16:35.979 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:16:35.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:35.979 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-03-20T09:16:35.979 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "ad3fd9c0-316f-4fe5-a63b-d480087c1729", 2024-03-20T09:16:35.979 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:16:35.979 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:16:35.979 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:16:35.979 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:16:35.979 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:16:35.979 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:16:35.979 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 8, 2024-03-20T09:16:35.979 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 561, 2024-03-20T09:16:35.980 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:16:35.980 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:16:35.980 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:16:35.980 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:35.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:35.980 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:35.980 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2024-03-20T09:16:35.980 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:16:35.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:35.980 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:35.980 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2024-03-20T09:16:35.980 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:16:35.980 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:16:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2024-03-20T09:16:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:16:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2024-03-20T09:16:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:16:35.981 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:16:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2024-03-20T09:16:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:16:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2024-03-20T09:16:35.982 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:16:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:16:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2024-03-20T09:16:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:16:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2024-03-20T09:16:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:16:35.983 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/2139805293", 2024-03-20T09:16:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/2139805293", 2024-03-20T09:16:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/2139805293", 2024-03-20T09:16:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/2139805293", 2024-03-20T09:16:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:16:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:16:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:16:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-03-20T09:16:35.984 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e85b94f5-2954-47a1-9c20-096f2606eb04", 2024-03-20T09:16:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:16:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:16:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:16:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:16:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:16:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:16:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2024-03-20T09:16:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 533, 2024-03-20T09:16:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:16:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:16:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:16:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:35.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2024-03-20T09:16:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:16:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2024-03-20T09:16:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:16:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:16:35.986 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2024-03-20T09:16:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:16:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2024-03-20T09:16:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:16:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:16:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:35.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2024-03-20T09:16:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:16:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2024-03-20T09:16:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:16:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:16:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:35.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2024-03-20T09:16:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:16:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2024-03-20T09:16:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:16:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.989 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/3853355134", 2024-03-20T09:16:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/3853355134", 2024-03-20T09:16:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/3853355134", 2024-03-20T09:16:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/3853355134", 2024-03-20T09:16:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:16:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:16:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:16:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-03-20T09:16:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "dd2b10f0-741c-47a4-b224-ac6bfe2c14ef", 2024-03-20T09:16:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:16:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:16:35.990 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:16:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:16:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:16:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:16:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 18, 2024-03-20T09:16:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 534, 2024-03-20T09:16:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:16:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:16:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:16:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2024-03-20T09:16:35.991 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:16:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2024-03-20T09:16:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:16:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:16:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:35.992 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2024-03-20T09:16:35.993 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:16:35.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:35.993 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:35.993 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2024-03-20T09:16:35.993 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:16:35.993 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:35.993 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:35.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.993 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:16:35.993 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:35.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:35.993 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:35.993 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2024-03-20T09:16:35.994 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:16:35.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:35.994 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:35.994 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2024-03-20T09:16:35.994 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:16:35.994 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:35.994 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:35.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.994 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:16:35.994 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:35.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:35.994 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:35.994 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2024-03-20T09:16:35.995 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:16:35.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:35.995 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:35.995 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2024-03-20T09:16:35.995 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:16:35.995 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:35.995 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:35.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.995 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/3993357967", 2024-03-20T09:16:35.995 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/3993357967", 2024-03-20T09:16:35.995 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/3993357967", 2024-03-20T09:16:35.995 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/3993357967", 2024-03-20T09:16:35.995 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:16:35.996 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:16:35.996 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:16:35.996 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:35.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:35.996 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-03-20T09:16:35.996 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1", 2024-03-20T09:16:35.996 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:16:35.996 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:16:35.996 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:16:35.996 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:16:35.996 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:16:35.996 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:16:35.996 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 26, 2024-03-20T09:16:35.996 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 525, 2024-03-20T09:16:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:16:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:16:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:16:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2024-03-20T09:16:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:16:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2024-03-20T09:16:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:16:35.997 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:35.998 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:35.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.998 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:16:35.998 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:35.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:35.998 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:35.998 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2024-03-20T09:16:35.998 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:16:35.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:35.998 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:35.998 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2024-03-20T09:16:35.998 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:16:35.998 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:35.999 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:35.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.999 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:16:35.999 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:35.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:35.999 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:35.999 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2024-03-20T09:16:35.999 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:16:35.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:35.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:35.999 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:35.999 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2024-03-20T09:16:35.999 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:16:35.999 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.000 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.000 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:16:36.000 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:36.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.000 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:36.000 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2024-03-20T09:16:36.000 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:16:36.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.000 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:36.000 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2024-03-20T09:16:36.000 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:16:36.000 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.001 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.001 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/4110225509", 2024-03-20T09:16:36.001 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/4110225509", 2024-03-20T09:16:36.001 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/4110225509", 2024-03-20T09:16:36.001 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/4110225509", 2024-03-20T09:16:36.001 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:16:36.001 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:16:36.001 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:16:36.001 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.001 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-03-20T09:16:36.001 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "f14e34d5-9829-42a3-9b25-31a0976c5ef7", 2024-03-20T09:16:36.002 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:16:36.002 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 0, 2024-03-20T09:16:36.002 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-03-20T09:16:36.002 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:16:36.002 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:16:36.002 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:16:36.002 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 32, 2024-03-20T09:16:36.002 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 366, 2024-03-20T09:16:36.002 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:16:36.002 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:16:36.002 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:16:36.002 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:36.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.002 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:36.003 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6800", 2024-03-20T09:16:36.003 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:16:36.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.003 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:36.003 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6801", 2024-03-20T09:16:36.003 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:16:36.003 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.003 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.003 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:16:36.003 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:36.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.003 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:36.004 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6802", 2024-03-20T09:16:36.004 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:16:36.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.004 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:36.004 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6803", 2024-03-20T09:16:36.004 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:16:36.004 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.004 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.004 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:16:36.004 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:36.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.004 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:36.004 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6806", 2024-03-20T09:16:36.004 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:16:36.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.005 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:36.005 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6807", 2024-03-20T09:16:36.005 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:16:36.005 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.005 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.005 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:16:36.005 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:36.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.005 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:36.005 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6804", 2024-03-20T09:16:36.005 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:16:36.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.006 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:36.006 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6805", 2024-03-20T09:16:36.006 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:16:36.006 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.006 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.006 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6801/477923058", 2024-03-20T09:16:36.006 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6803/477923058", 2024-03-20T09:16:36.006 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6807/477923058", 2024-03-20T09:16:36.006 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6805/477923058", 2024-03-20T09:16:36.006 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:16:36.006 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:16:36.007 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:16:36.007 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.007 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-03-20T09:16:36.007 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "cb431b6a-540d-4318-ae35-d379e1df0185", 2024-03-20T09:16:36.007 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:16:36.007 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:16:36.007 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:16:36.007 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:16:36.007 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 41, 2024-03-20T09:16:36.007 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 273, 2024-03-20T09:16:36.007 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 443, 2024-03-20T09:16:36.007 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 520, 2024-03-20T09:16:36.007 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 274, 2024-03-20T09:16:36.008 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:16:36.008 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:16:36.008 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:36.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.008 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:36.008 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6808", 2024-03-20T09:16:36.008 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:16:36.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.008 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:36.008 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6809", 2024-03-20T09:16:36.008 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:16:36.008 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.008 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.009 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:16:36.009 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:36.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.009 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:36.009 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6810", 2024-03-20T09:16:36.009 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:16:36.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.009 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:36.009 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6811", 2024-03-20T09:16:36.009 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:16:36.009 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.009 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.010 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:16:36.010 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:36.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.010 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:36.010 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6814", 2024-03-20T09:16:36.010 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:16:36.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.010 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:36.010 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6815", 2024-03-20T09:16:36.010 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:16:36.010 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.010 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.011 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:16:36.011 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:36.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.011 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:36.011 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6812", 2024-03-20T09:16:36.011 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:16:36.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.011 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:36.011 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6813", 2024-03-20T09:16:36.011 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:16:36.011 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.011 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.012 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6809/650416648", 2024-03-20T09:16:36.012 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6811/650416648", 2024-03-20T09:16:36.012 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6815/650416648", 2024-03-20T09:16:36.012 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6813/650416648", 2024-03-20T09:16:36.012 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:16:36.012 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:16:36.012 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:16:36.012 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.012 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2024-03-20T09:16:36.012 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "753ed830-e764-4ff3-9d9b-87e7fa748b81", 2024-03-20T09:16:36.012 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:16:36.012 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:16:36.013 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:16:36.013 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:16:36.013 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:16:36.013 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:16:36.013 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 51, 2024-03-20T09:16:36.013 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 562, 2024-03-20T09:16:36.013 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:16:36.013 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:16:36.013 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:16:36.013 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:36.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.013 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:36.013 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6816", 2024-03-20T09:16:36.013 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:16:36.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.014 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:36.014 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6817", 2024-03-20T09:16:36.014 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:16:36.014 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.014 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.014 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:16:36.014 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:36.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.014 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:36.014 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6818", 2024-03-20T09:16:36.015 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:16:36.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.015 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:36.015 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6819", 2024-03-20T09:16:36.015 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:16:36.015 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.015 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.015 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:16:36.015 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:36.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.015 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:36.015 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6822", 2024-03-20T09:16:36.016 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:16:36.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.016 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:36.016 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6823", 2024-03-20T09:16:36.016 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:16:36.016 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.016 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.016 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:16:36.016 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:36.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.016 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:36.016 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6820", 2024-03-20T09:16:36.016 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:16:36.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.017 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:36.017 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6821", 2024-03-20T09:16:36.017 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:16:36.017 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.017 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.017 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6817/3278301843", 2024-03-20T09:16:36.017 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6819/3278301843", 2024-03-20T09:16:36.017 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6823/3278301843", 2024-03-20T09:16:36.017 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6821/3278301843", 2024-03-20T09:16:36.017 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:16:36.018 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:16:36.018 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:16:36.018 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.018 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2024-03-20T09:16:36.018 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "55bcbd37-7a6f-4ab1-b5a9-91a951444d86", 2024-03-20T09:16:36.018 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:16:36.018 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:16:36.018 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:16:36.018 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 0.058074951171875, 2024-03-20T09:16:36.018 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:16:36.018 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:16:36.018 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 60, 2024-03-20T09:16:36.019 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 517, 2024-03-20T09:16:36.019 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:16:36.019 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:16:36.019 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:16:36.019 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:36.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.019 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:36.019 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6824", 2024-03-20T09:16:36.019 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:16:36.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.019 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:36.019 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6825", 2024-03-20T09:16:36.019 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:16:36.020 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.020 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.020 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:16:36.020 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:36.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.020 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:36.020 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6826", 2024-03-20T09:16:36.020 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:16:36.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.020 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:36.020 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6827", 2024-03-20T09:16:36.020 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:16:36.020 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.021 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.021 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:16:36.021 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:36.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.021 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:36.021 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6830", 2024-03-20T09:16:36.021 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:16:36.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.021 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:36.021 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6831", 2024-03-20T09:16:36.021 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:16:36.021 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.021 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.022 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:16:36.022 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:36.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.022 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:36.022 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6828", 2024-03-20T09:16:36.022 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:16:36.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.022 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:36.022 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6829", 2024-03-20T09:16:36.022 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:16:36.022 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.022 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.023 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6825/3152424001", 2024-03-20T09:16:36.023 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6827/3152424001", 2024-03-20T09:16:36.023 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6831/3152424001", 2024-03-20T09:16:36.023 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6829/3152424001", 2024-03-20T09:16:36.023 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:16:36.023 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:16:36.023 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:16:36.023 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.023 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 8, 2024-03-20T09:16:36.023 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "bc86248e-6bd0-4369-b8a2-e593bc2a610d", 2024-03-20T09:16:36.023 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:16:36.023 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:16:36.024 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:16:36.024 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:16:36.024 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:16:36.024 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:16:36.024 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 70, 2024-03-20T09:16:36.024 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 496, 2024-03-20T09:16:36.024 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:16:36.024 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:16:36.024 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:16:36.024 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:36.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.024 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:36.024 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6800", 2024-03-20T09:16:36.024 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:16:36.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.025 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:36.025 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6801", 2024-03-20T09:16:36.025 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:16:36.025 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.025 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.025 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:16:36.025 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:36.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.025 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:36.025 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6802", 2024-03-20T09:16:36.025 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:16:36.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.026 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:36.026 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6803", 2024-03-20T09:16:36.026 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:16:36.026 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.026 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.026 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:16:36.026 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:36.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.026 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:36.026 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6806", 2024-03-20T09:16:36.026 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:16:36.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.027 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:36.027 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6807", 2024-03-20T09:16:36.027 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:16:36.027 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.027 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.027 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:16:36.027 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:36.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.027 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:36.027 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6804", 2024-03-20T09:16:36.027 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:16:36.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.028 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:36.028 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6805", 2024-03-20T09:16:36.028 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:16:36.028 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.028 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.028 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6801/1309426740", 2024-03-20T09:16:36.028 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6803/1309426740", 2024-03-20T09:16:36.028 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6807/1309426740", 2024-03-20T09:16:36.028 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6805/1309426740", 2024-03-20T09:16:36.028 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:16:36.028 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:16:36.029 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:16:36.029 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.029 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 9, 2024-03-20T09:16:36.029 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b20a4aa-032a-4c09-ac75-153c554424e7", 2024-03-20T09:16:36.029 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:16:36.029 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 0, 2024-03-20T09:16:36.029 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-03-20T09:16:36.029 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:16:36.029 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:16:36.029 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:16:36.029 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 80, 2024-03-20T09:16:36.029 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 257, 2024-03-20T09:16:36.029 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:16:36.030 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:16:36.030 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:16:36.030 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:36.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.030 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:36.030 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6808", 2024-03-20T09:16:36.030 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:16:36.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.030 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:36.030 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6809", 2024-03-20T09:16:36.030 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:16:36.030 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.030 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.031 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:16:36.031 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:36.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.031 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:36.031 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6810", 2024-03-20T09:16:36.031 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:16:36.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.031 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:36.031 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6811", 2024-03-20T09:16:36.031 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:16:36.031 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.031 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.032 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:16:36.032 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:36.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.032 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:36.032 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6814", 2024-03-20T09:16:36.032 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:16:36.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.032 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:36.032 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6815", 2024-03-20T09:16:36.032 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:16:36.032 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.032 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.033 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:16:36.033 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:36.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.033 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:36.033 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6812", 2024-03-20T09:16:36.033 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:16:36.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.033 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:36.033 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6813", 2024-03-20T09:16:36.033 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:16:36.033 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.033 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.034 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6809/2953210076", 2024-03-20T09:16:36.034 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6811/2953210076", 2024-03-20T09:16:36.034 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6815/2953210076", 2024-03-20T09:16:36.034 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6813/2953210076", 2024-03-20T09:16:36.034 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:16:36.034 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:16:36.034 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:16:36.034 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.034 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 10, 2024-03-20T09:16:36.034 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e62c1fb3-5dd1-4d0d-997c-d20633d19361", 2024-03-20T09:16:36.034 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:16:36.035 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:16:36.035 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:16:36.035 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:16:36.035 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:16:36.035 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:16:36.035 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 89, 2024-03-20T09:16:36.035 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 520, 2024-03-20T09:16:36.035 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:16:36.035 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:16:36.035 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:16:36.035 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:36.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.035 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:36.035 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6816", 2024-03-20T09:16:36.036 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:16:36.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.036 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:36.036 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6817", 2024-03-20T09:16:36.036 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:16:36.036 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.036 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.036 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:16:36.036 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:36.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.036 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:36.037 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6818", 2024-03-20T09:16:36.037 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:16:36.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.037 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:36.037 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6819", 2024-03-20T09:16:36.037 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:16:36.037 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.037 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.037 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:16:36.037 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:36.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.037 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:36.038 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6822", 2024-03-20T09:16:36.038 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:16:36.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.038 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:36.038 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6823", 2024-03-20T09:16:36.038 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:16:36.038 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.038 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.038 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:16:36.038 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:36.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.038 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:36.038 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6820", 2024-03-20T09:16:36.039 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:16:36.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.039 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:36.039 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6821", 2024-03-20T09:16:36.039 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:16:36.039 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.039 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.039 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6817/409394340", 2024-03-20T09:16:36.039 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6819/409394340", 2024-03-20T09:16:36.039 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6823/409394340", 2024-03-20T09:16:36.039 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6821/409394340", 2024-03-20T09:16:36.039 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:16:36.039 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:16:36.040 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:16:36.040 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.040 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 11, 2024-03-20T09:16:36.040 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b83994d5-e5ef-48b0-9758-c26843cec3b6", 2024-03-20T09:16:36.040 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:16:36.040 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:16:36.040 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:16:36.040 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:16:36.040 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:16:36.040 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:16:36.040 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 96, 2024-03-20T09:16:36.040 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 522, 2024-03-20T09:16:36.040 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:16:36.041 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:16:36.041 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:16:36.041 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:36.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.041 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:36.041 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6824", 2024-03-20T09:16:36.041 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:16:36.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.041 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:36.041 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6825", 2024-03-20T09:16:36.041 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:16:36.041 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.041 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.041 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:16:36.042 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:36.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.042 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:36.042 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6826", 2024-03-20T09:16:36.042 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:16:36.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.042 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:36.042 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6827", 2024-03-20T09:16:36.042 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:16:36.042 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.042 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.042 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:16:36.043 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:36.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.043 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:36.043 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6830", 2024-03-20T09:16:36.043 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:16:36.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.043 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:36.043 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6831", 2024-03-20T09:16:36.043 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:16:36.043 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.043 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.043 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:16:36.044 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:16:36.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.044 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:16:36.044 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6828", 2024-03-20T09:16:36.044 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:16:36.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.044 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:16:36.044 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6829", 2024-03-20T09:16:36.044 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:16:36.044 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.044 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.044 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6825/2704069354", 2024-03-20T09:16:36.045 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6827/2704069354", 2024-03-20T09:16:36.045 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6831/2704069354", 2024-03-20T09:16:36.045 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6829/2704069354", 2024-03-20T09:16:36.045 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:16:36.045 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:16:36.045 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:16:36.045 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.045 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.045 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:16:36.045 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2024-03-20T09:16:36.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.045 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-03-20T09:16:36.045 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:16:36.045 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:16:36.045 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:16:36.046 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:16:36.046 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:16:36.046 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:16:22.161960+0000", 2024-03-20T09:16:36.046 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:16:36.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.046 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-03-20T09:16:36.046 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:16:36.046 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:16:36.046 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:16:36.046 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:16:36.046 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:16:36.046 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:16:23.370086+0000", 2024-03-20T09:16:36.047 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:16:36.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.047 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-03-20T09:16:36.047 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:16:36.047 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:16:36.047 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:16:36.047 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:16:36.047 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:16:36.047 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:15:38.999217+0000", 2024-03-20T09:16:36.047 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:16:36.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.047 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-03-20T09:16:36.047 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:16:36.048 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:16:36.048 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:16:36.048 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:16:36.048 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:16:36.048 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:16:26.574552+0000", 2024-03-20T09:16:36.048 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:16:36.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.048 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-03-20T09:16:36.048 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:16:36.048 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:16:36.048 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:16:36.048 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:16:36.048 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 65536, 2024-03-20T09:16:36.048 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:15:45.509487+0000", 2024-03-20T09:16:36.049 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:16:36.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.049 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-03-20T09:16:36.049 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:16:36.049 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:16:36.049 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:16:36.049 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:16:36.049 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:16:36.049 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:15:49.636448+0000", 2024-03-20T09:16:36.049 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 273 2024-03-20T09:16:36.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.049 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2024-03-20T09:16:36.050 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:16:36.050 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:16:36.050 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:16:36.050 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:16:36.050 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:16:36.050 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:15:55.522497+0000", 2024-03-20T09:16:36.050 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:16:36.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.050 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2024-03-20T09:16:36.050 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:16:36.050 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:16:36.050 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:16:36.050 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:16:36.050 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:16:36.050 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:16:02.566768+0000", 2024-03-20T09:16:36.051 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:16:36.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.051 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 8, 2024-03-20T09:16:36.051 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:16:36.051 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:16:36.051 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:16:36.051 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:16:36.051 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:16:36.051 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:16:13.598899+0000", 2024-03-20T09:16:36.051 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:16:36.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.051 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 9, 2024-03-20T09:16:36.051 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:16:36.051 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:16:36.052 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:16:36.052 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:16:36.052 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 34736, 2024-03-20T09:16:36.052 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:16:19.250075+0000", 2024-03-20T09:16:36.052 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:16:36.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.052 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 10, 2024-03-20T09:16:36.052 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:16:36.052 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:16:36.052 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:16:36.052 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:16:36.052 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:16:36.052 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:16:19.152105+0000", 2024-03-20T09:16:36.052 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:16:36.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.053 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 11, 2024-03-20T09:16:36.053 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:16:36.053 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:16:36.053 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:16:36.053 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:16:36.053 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:16:36.053 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:16:13.900747+0000", 2024-03-20T09:16:36.053 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:16:36.053 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.053 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:16:36.053 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [ 2024-03-20T09:16:36.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.053 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "1.1", 2024-03-20T09:16:36.053 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:16:36.053 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-03-20T09:16:36.054 INFO:teuthology.orchestra.run.smithi026.stdout: 11, 2024-03-20T09:16:36.054 INFO:teuthology.orchestra.run.smithi026.stdout: 5 2024-03-20T09:16:36.054 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.054 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.054 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:16:36.054 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [ 2024-03-20T09:16:36.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.054 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "4.f", 2024-03-20T09:16:36.054 INFO:teuthology.orchestra.run.smithi026.stdout: "mappings": [ 2024-03-20T09:16:36.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:16:36.054 INFO:teuthology.orchestra.run.smithi026.stdout: "from": 1, 2024-03-20T09:16:36.054 INFO:teuthology.orchestra.run.smithi026.stdout: "to": 2 2024-03-20T09:16:36.054 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.054 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:16:36.054 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.055 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:16:36.055 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_primaries": [], 2024-03-20T09:16:36.055 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2024-03-20T09:16:36.055 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2024-03-20T09:16:36.055 INFO:teuthology.orchestra.run.smithi026.stdout: "blocklist": { 2024-03-20T09:16:36.055 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/4291404191": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:16:36.055 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/642012566": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:16:36.055 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6801/1181459999": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:16:36.055 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/3984513952": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:16:36.055 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6800/1654945498": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:16:36.055 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6800/1181459999": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:16:36.055 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/137770981": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:16:36.055 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/1451237762": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:16:36.055 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/3938386631": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:16:36.055 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6801/1654945498": "2024-03-21T08:49:04.596754+0000" 2024-03-20T09:16:36.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.056 INFO:teuthology.orchestra.run.smithi026.stdout: "range_blocklist": {}, 2024-03-20T09:16:36.056 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2024-03-20T09:16:36.056 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2024-03-20T09:16:36.056 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2024-03-20T09:16:36.056 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2024-03-20T09:16:36.056 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2024-03-20T09:16:36.056 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2024-03-20T09:16:36.056 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2024-03-20T09:16:36.056 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:16:36.056 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2024-03-20T09:16:36.056 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2024-03-20T09:16:36.056 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2024-03-20T09:16:36.056 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2024-03-20T09:16:36.056 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {}, 2024-03-20T09:16:36.056 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": { 2024-03-20T09:16:36.057 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_enabled": false, 2024-03-20T09:16:36.057 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_bucket_count": 0, 2024-03-20T09:16:36.057 INFO:teuthology.orchestra.run.smithi026.stdout: "degraded_stretch_mode": 0, 2024-03-20T09:16:36.057 INFO:teuthology.orchestra.run.smithi026.stdout: "recovering_stretch_mode": 0, 2024-03-20T09:16:36.057 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_bucket": 0 2024-03-20T09:16:36.057 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:16:36.057 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:16:36.057 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:16:36.085 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:36.105 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:16:36.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:36.108 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:16:36.209 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:16:36.224 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:36 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1156535620' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:16:36.224 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:36 smithi026 ceph-mon[68309]: pgmap v1377: 47 pgs: 47 active+clean; 3.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 858 op/s 2024-03-20T09:16:36.275 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:36.333 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:16:36.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:36.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:16:36.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:36 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1156535620' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:16:36.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:36 smithi115 ceph-mon[72476]: pgmap v1377: 47 pgs: 47 active+clean; 3.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 858 op/s 2024-03-20T09:16:36.436 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:16:36.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:36 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1156535620' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:16:36.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:36 smithi122 ceph-mon[73495]: pgmap v1377: 47 pgs: 47 active+clean; 3.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 858 op/s 2024-03-20T09:16:36.469 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:36.497 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:16:36.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:36.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:16:36.605 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:16:36.633 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:36.654 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:16:36.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:36.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:16:36.674 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:16:36.758 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:16:36.789 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:36.813 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:16:36.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:36.822 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:16:36.923 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:16:36.954 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:36.981 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:16:36.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:36.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:16:37.086 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:16:37.138 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:37.926 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:16:37.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:37.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:16:38.035 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:16:38.086 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:38.280 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:16:38.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:38.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:16:38.390 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:16:38.394 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:38 smithi026 ceph-mon[68309]: pgmap v1378: 47 pgs: 47 active+clean; 3.5 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 824 op/s 2024-03-20T09:16:38.417 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:38.443 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:16:38.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:38.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:16:38.547 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:16:38.598 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:38 smithi122 ceph-mon[73495]: pgmap v1378: 47 pgs: 47 active+clean; 3.5 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 824 op/s 2024-03-20T09:16:38.601 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:38.625 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:16:38.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:38.629 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:16:38.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:38 smithi115 ceph-mon[72476]: pgmap v1378: 47 pgs: 47 active+clean; 3.5 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 824 op/s 2024-03-20T09:16:38.730 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:16:38.760 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:38.785 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:16:38.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:38.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:16:38.862 INFO:tasks.thrashosds.thrasher:j is {'epoch': 577, 'fsid': '852fbb68-e696-11ee-95c9-87774f69a715', 'created': '2024-03-20T08:48:35.281088+0000', 'modified': '2024-03-20T09:16:33.175520+0000', 'last_up_change': '2024-03-20T09:10:33.084151+0000', 'last_in_change': '2024-03-20T09:10:19.369629+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-20T08:52:17.442324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '360', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '358', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.3399999141693115, 'score_stable': 2.3399999141693115, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 1.940000057220459, 'raw_score_stable': 1.940000057220459, 'primary_affinity_weighted': 0.9215061664581299, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 4, 'pool_name': 'unique_pool_2', 'create_time': '2024-03-20T09:13:20.690041+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '527', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.009999990463257, 'score_stable': 3.009999990463257, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.9215061664581299, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}], 'osds': [{'osd': 0, 'uuid': 'ad3fd9c0-316f-4fe5-a63b-d480087c1729', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 561, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6802', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6803', 'nonce': 2139805293}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6804', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6805', 'nonce': 2139805293}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6808', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6809', 'nonce': 2139805293}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6806', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6807', 'nonce': 2139805293}]}, 'public_addr': '172.21.15.26:6803/2139805293', 'cluster_addr': '172.21.15.26:6805/2139805293', 'heartbeat_back_addr': '172.21.15.26:6809/2139805293', 'heartbeat_front_addr': '172.21.15.26:6807/2139805293', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'e85b94f5-2954-47a1-9c20-096f2606eb04', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 533, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6810', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6811', 'nonce': 3853355134}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6812', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6813', 'nonce': 3853355134}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6816', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6817', 'nonce': 3853355134}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6814', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6815', 'nonce': 3853355134}]}, 'public_addr': '172.21.15.26:6811/3853355134', 'cluster_addr': '172.21.15.26:6813/3853355134', 'heartbeat_back_addr': '172.21.15.26:6817/3853355134', 'heartbeat_front_addr': '172.21.15.26:6815/3853355134', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'dd2b10f0-741c-47a4-b224-ac6bfe2c14ef', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 534, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6818', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6819', 'nonce': 3993357967}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6820', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6821', 'nonce': 3993357967}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6824', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6825', 'nonce': 3993357967}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6822', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6823', 'nonce': 3993357967}]}, 'public_addr': '172.21.15.26:6819/3993357967', 'cluster_addr': '172.21.15.26:6821/3993357967', 'heartbeat_back_addr': '172.21.15.26:6825/3993357967', 'heartbeat_front_addr': '172.21.15.26:6823/3993357967', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 525, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6826', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6827', 'nonce': 4110225509}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6828', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6829', 'nonce': 4110225509}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6832', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6833', 'nonce': 4110225509}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6830', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6831', 'nonce': 4110225509}]}, 'public_addr': '172.21.15.26:6827/4110225509', 'cluster_addr': '172.21.15.26:6829/4110225509', 'heartbeat_back_addr': '172.21.15.26:6833/4110225509', 'heartbeat_front_addr': '172.21.15.26:6831/4110225509', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'f14e34d5-9829-42a3-9b25-31a0976c5ef7', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 366, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6800', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6801', 'nonce': 477923058}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6802', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6803', 'nonce': 477923058}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6806', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6807', 'nonce': 477923058}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6804', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6805', 'nonce': 477923058}]}, 'public_addr': '172.21.15.115:6801/477923058', 'cluster_addr': '172.21.15.115:6803/477923058', 'heartbeat_back_addr': '172.21.15.115:6807/477923058', 'heartbeat_front_addr': '172.21.15.115:6805/477923058', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'cb431b6a-540d-4318-ae35-d379e1df0185', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 41, 'last_clean_end': 273, 'up_from': 443, 'up_thru': 520, 'down_at': 274, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6808', 'nonce': 650416648}, {'type': 'v1', 'addr': '172.21.15.115:6809', 'nonce': 650416648}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6810', 'nonce': 650416648}, {'type': 'v1', 'addr': '172.21.15.115:6811', 'nonce': 650416648}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6814', 'nonce': 650416648}, {'type': 'v1', 'addr': '172.21.15.115:6815', 'nonce': 650416648}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6812', 'nonce': 650416648}, {'type': 'v1', 'addr': '172.21.15.115:6813', 'nonce': 650416648}]}, 'public_addr': '172.21.15.115:6809/650416648', 'cluster_addr': '172.21.15.115:6811/650416648', 'heartbeat_back_addr': '172.21.15.115:6815/650416648', 'heartbeat_front_addr': '172.21.15.115:6813/650416648', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '753ed830-e764-4ff3-9d9b-87e7fa748b81', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 562, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6816', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6817', 'nonce': 3278301843}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6818', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6819', 'nonce': 3278301843}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6822', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6823', 'nonce': 3278301843}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6820', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6821', 'nonce': 3278301843}]}, 'public_addr': '172.21.15.115:6817/3278301843', 'cluster_addr': '172.21.15.115:6819/3278301843', 'heartbeat_back_addr': '172.21.15.115:6823/3278301843', 'heartbeat_front_addr': '172.21.15.115:6821/3278301843', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '55bcbd37-7a6f-4ab1-b5a9-91a951444d86', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.058074951171875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 517, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6824', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6825', 'nonce': 3152424001}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6826', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6827', 'nonce': 3152424001}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6830', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6831', 'nonce': 3152424001}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6828', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6829', 'nonce': 3152424001}]}, 'public_addr': '172.21.15.115:6825/3152424001', 'cluster_addr': '172.21.15.115:6827/3152424001', 'heartbeat_back_addr': '172.21.15.115:6831/3152424001', 'heartbeat_front_addr': '172.21.15.115:6829/3152424001', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'bc86248e-6bd0-4369-b8a2-e593bc2a610d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 70, 'up_thru': 496, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6800', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6801', 'nonce': 1309426740}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6802', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6803', 'nonce': 1309426740}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6806', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6807', 'nonce': 1309426740}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6804', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6805', 'nonce': 1309426740}]}, 'public_addr': '172.21.15.122:6801/1309426740', 'cluster_addr': '172.21.15.122:6803/1309426740', 'heartbeat_back_addr': '172.21.15.122:6807/1309426740', 'heartbeat_front_addr': '172.21.15.122:6805/1309426740', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '9b20a4aa-032a-4c09-ac75-153c554424e7', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 257, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6808', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6809', 'nonce': 2953210076}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6810', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6811', 'nonce': 2953210076}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6814', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6815', 'nonce': 2953210076}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6812', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6813', 'nonce': 2953210076}]}, 'public_addr': '172.21.15.122:6809/2953210076', 'cluster_addr': '172.21.15.122:6811/2953210076', 'heartbeat_back_addr': '172.21.15.122:6815/2953210076', 'heartbeat_front_addr': '172.21.15.122:6813/2953210076', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'e62c1fb3-5dd1-4d0d-997c-d20633d19361', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 89, 'up_thru': 520, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6816', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6817', 'nonce': 409394340}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6818', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6819', 'nonce': 409394340}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6822', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6823', 'nonce': 409394340}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6820', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6821', 'nonce': 409394340}]}, 'public_addr': '172.21.15.122:6817/409394340', 'cluster_addr': '172.21.15.122:6819/409394340', 'heartbeat_back_addr': '172.21.15.122:6823/409394340', 'heartbeat_front_addr': '172.21.15.122:6821/409394340', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'b83994d5-e5ef-48b0-9758-c26843cec3b6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 96, 'up_thru': 522, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6824', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6825', 'nonce': 2704069354}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6826', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6827', 'nonce': 2704069354}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6830', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6831', 'nonce': 2704069354}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6828', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6829', 'nonce': 2704069354}]}, 'public_addr': '172.21.15.122:6825/2704069354', 'cluster_addr': '172.21.15.122:6827/2704069354', 'heartbeat_back_addr': '172.21.15.122:6831/2704069354', 'heartbeat_front_addr': '172.21.15.122:6829/2704069354', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:16:22.161960+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:16:23.370086+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:15:38.999217+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:16:26.574552+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-20T09:15:45.509487+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:15:49.636448+0000', 'dead_epoch': 273}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:15:55.522497+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:16:02.566768+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:16:13.598899+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 34736, 'last_purged_snaps_scrub': '2024-03-20T09:16:19.250075+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:16:19.152105+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:16:13.900747+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.1', 'osds': [6, 11, 5]}], 'pg_upmap_items': [{'pgid': '4.f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.26:0/4291404191': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/642012566': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:6801/1181459999': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/3984513952': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6800/1654945498': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6800/1181459999': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/137770981': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:0/1451237762': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/3938386631': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6801/1654945498': '2024-03-21T08:49:04.596754+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-20T09:16:38.862 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:16:38.891 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:16:38.971 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:39.572 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:16:39.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:39.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:16:39.678 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:16:39.706 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:39.738 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:16:39.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:39.743 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:16:39.844 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:16:39.876 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:39.901 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:16:39.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:39.905 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:16:40.006 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:16:40.044 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:40 smithi122 ceph-mon[73495]: pgmap v1379: 47 pgs: 47 active+clean; 3.5 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 819 op/s 2024-03-20T09:16:40.091 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:40 smithi026 ceph-mon[68309]: pgmap v1379: 47 pgs: 47 active+clean; 3.5 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 819 op/s 2024-03-20T09:16:40.098 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:40.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:40 smithi115 ceph-mon[72476]: pgmap v1379: 47 pgs: 47 active+clean; 3.5 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 819 op/s 2024-03-20T09:16:40.799 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:16:40.799 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:16:41.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:41 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3590886948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:16:41.310 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:16:41.310 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:16:41.310 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:16:41.310 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:16:41.310 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:16:41.310 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:16:41.310 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:16:41.311 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:16:41.311 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:16:41.311 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:16:41.333 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:16:41.378 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:16:41.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:41 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3590886948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:16:41.436 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:16:41.436 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:16:41.436 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:16:41.436 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:16:41.436 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:16:41.436 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:16:41.436 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:16:41.436 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:16:41.436 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:16:41.436 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:16:41.436 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:16:41.436 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:16:41.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:41 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3590886948' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:16:42.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:42 smithi115 ceph-mon[72476]: pgmap v1380: 47 pgs: 47 active+clean; 3.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 811 op/s 2024-03-20T09:16:42.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:42 smithi122 ceph-mon[73495]: pgmap v1380: 47 pgs: 47 active+clean; 3.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 811 op/s 2024-03-20T09:16:42.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:42 smithi026 ceph-mon[68309]: pgmap v1380: 47 pgs: 47 active+clean; 3.4 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 811 op/s 2024-03-20T09:16:42.681 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-20T09:16:42.784 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:16:42.784 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:16:42.806 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:16:42.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:42.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:16:42.918 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:16:42.954 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:42.976 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:16:42.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:42.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:16:43.080 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:16:43.113 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:43.134 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:16:43.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:43.137 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:16:43.238 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:16:43.272 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:43.291 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:16:43.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:43.294 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:16:43.395 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:16:43.448 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:44.224 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:44 smithi026 ceph-mon[68309]: pgmap v1381: 47 pgs: 47 active+clean; 3.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 757 op/s 2024-03-20T09:16:44.339 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:16:44.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:44.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:16:44.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:44 smithi115 ceph-mon[72476]: pgmap v1381: 47 pgs: 47 active+clean; 3.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 757 op/s 2024-03-20T09:16:44.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:44 smithi122 ceph-mon[73495]: pgmap v1381: 47 pgs: 47 active+clean; 3.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 757 op/s 2024-03-20T09:16:44.452 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:16:44.485 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:44.511 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:16:44.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:44.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:16:44.619 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:16:44.652 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:44.674 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:16:44.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:44.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:16:44.779 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:16:44.809 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:44.835 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:16:44.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:44.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:16:44.842 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:16:44.938 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:16:44.967 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:44.994 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:16:44.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:44.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:16:45.098 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:16:45.127 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:45.154 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:16:45.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:45.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:16:45.258 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:16:45.295 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:45.327 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:16:45.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:45.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:16:45.431 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:16:45.468 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:45.494 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:16:45.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:45.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:16:45.602 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:16:45.631 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:45.651 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:16:45.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:45.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:16:45.755 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:16:45.825 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:46.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:46 smithi026 ceph-mon[68309]: pgmap v1382: 47 pgs: 47 active+clean; 3.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 875 op/s 2024-03-20T09:16:46.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:46 smithi115 ceph-mon[72476]: pgmap v1382: 47 pgs: 47 active+clean; 3.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 875 op/s 2024-03-20T09:16:46.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:46 smithi122 ceph-mon[73495]: pgmap v1382: 47 pgs: 47 active+clean; 3.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 875 op/s 2024-03-20T09:16:46.903 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:16:46.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:46.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:16:47.009 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:16:47.059 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:16:47.105 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:47.203 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:16:47.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:47.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:16:47.313 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:16:47.341 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:47.362 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:16:47.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:47.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:16:47.367 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:16:47.467 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:16:47.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:47 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4042324' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:16:47.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:47 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:16:47.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:47 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:16:47.494 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:47.527 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:16:47.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:47.530 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:16:47.631 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:16:47.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:47 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4042324' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:16:47.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:47 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:16:47.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:47 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:16:47.660 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:47.685 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:16:47.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:47.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:16:47.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:47 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4042324' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:16:47.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:47 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:16:47.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:47 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:16:47.790 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:16:47.863 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:48.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:48 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:16:48.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:48 smithi115 ceph-mon[72476]: osdmap e578: 12 total, 12 up, 10 in 2024-03-20T09:16:48.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:48 smithi115 ceph-mon[72476]: pgmap v1384: 47 pgs: 47 active+clean; 3.0 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 822 op/s 2024-03-20T09:16:48.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:48 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:16:48.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:48 smithi122 ceph-mon[73495]: osdmap e578: 12 total, 12 up, 10 in 2024-03-20T09:16:48.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:48 smithi122 ceph-mon[73495]: pgmap v1384: 47 pgs: 47 active+clean; 3.0 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 822 op/s 2024-03-20T09:16:48.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:48 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:16:48.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:48 smithi026 ceph-mon[68309]: osdmap e578: 12 total, 12 up, 10 in 2024-03-20T09:16:48.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:48 smithi026 ceph-mon[68309]: pgmap v1384: 47 pgs: 47 active+clean; 3.0 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 822 op/s 2024-03-20T09:16:49.456 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:16:49.472 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:16:49.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:49 smithi115 ceph-mon[72476]: 4.3 scrub starts 2024-03-20T09:16:49.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:49 smithi122 ceph-mon[73495]: 4.3 scrub starts 2024-03-20T09:16:49.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:49 smithi026 ceph-mon[68309]: 4.3 scrub starts 2024-03-20T09:16:50.234 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:16:50.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:50 smithi026 ceph-mon[68309]: from='client.25831 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:16:50.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:50 smithi026 ceph-mon[68309]: 4.3 scrub ok 2024-03-20T09:16:50.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:50 smithi026 ceph-mon[68309]: pgmap v1385: 47 pgs: 47 active+clean; 3.0 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 828 op/s 2024-03-20T09:16:50.798 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":1384,"stamp":"2024-03-20T09:16:47.970722+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3249340984,"num_objects":49575,"num_object_clones":0,"num_object_copies":148725,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49575,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":192261,"num_write_kb":7738625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":923,"num_bytes_recovered":60751872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":164940,"ondisk_log_size":164940,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":13208236,"kb_used_data":9601244,"kb_used_omap":195,"kb_used_meta":3606076,"kb_avail":924243284,"statfs":{"total":959950356480,"available":946425122816,"internally_reserved":0,"allocated":9831673856,"data_stored":9828774144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":200482,"internal_metadata":3692622046},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,4,8,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-539820032,"num_objects":-8237,"num_object_clones":0,"num_object_copies":-24711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8237,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.009407"},"pg_stats":[{"pgid":"1.1e","version":"20'137","reported_seq":538,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-20T09:16:33.201053+0000","last_change":"2024-03-20T09:15:08.506869+0000","last_active":"2024-03-20T09:16:33.201053+0000","last_peered":"2024-03-20T09:16:33.201053+0000","last_clean":"2024-03-20T09:16:33.201053+0000","last_became_active":"2024-03-20T09:13:20.071655+0000","last_became_peered":"2024-03-20T09:13:20.071655+0000","last_unstale":"2024-03-20T09:16:33.201053+0000","last_undegraded":"2024-03-20T09:16:33.201053+0000","last_fullsized":"2024-03-20T09:16:33.201053+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:08.506732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:15:08.506732+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'137","reported_seq":715,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-20T09:16:47.369058+0000","last_change":"2024-03-20T09:16:04.640087+0000","last_active":"2024-03-20T09:16:47.369058+0000","last_peered":"2024-03-20T09:16:47.369058+0000","last_clean":"2024-03-20T09:16:47.369058+0000","last_became_active":"2024-03-20T09:07:42.013245+0000","last_became_peered":"2024-03-20T09:07:42.013245+0000","last_unstale":"2024-03-20T09:16:47.369058+0000","last_undegraded":"2024-03-20T09:16:47.369058+0000","last_fullsized":"2024-03-20T09:16:47.369058+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:16:04.639913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:16:04.639913+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:24.391962+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"20'137","reported_seq":1155,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-20T09:16:33.201031+0000","last_change":"2024-03-20T09:16:10.150919+0000","last_active":"2024-03-20T09:16:33.201031+0000","last_peered":"2024-03-20T09:16:33.201031+0000","last_clean":"2024-03-20T09:16:33.201031+0000","last_became_active":"2024-03-20T09:07:41.961622+0000","last_became_peered":"2024-03-20T09:07:41.961622+0000","last_unstale":"2024-03-20T09:16:33.201031+0000","last_undegraded":"2024-03-20T09:16:33.201031+0000","last_fullsized":"2024-03-20T09:16:33.201031+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:16:10.150777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:16:10.150777+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:13.547241+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.19","version":"20'137","reported_seq":1216,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-20T09:16:33.196364+0000","last_change":"2024-03-20T09:14:54.527786+0000","last_active":"2024-03-20T09:16:33.196364+0000","last_peered":"2024-03-20T09:16:33.196364+0000","last_clean":"2024-03-20T09:16:33.196364+0000","last_became_active":"2024-03-20T09:07:41.962680+0000","last_became_peered":"2024-03-20T09:07:41.962680+0000","last_unstale":"2024-03-20T09:16:33.196364+0000","last_undegraded":"2024-03-20T09:16:33.196364+0000","last_fullsized":"2024-03-20T09:16:33.196364+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:14:54.527668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:14:54.527668+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"20'137","reported_seq":678,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-20T09:16:33.199433+0000","last_change":"2024-03-20T09:15:10.009033+0000","last_active":"2024-03-20T09:16:33.199433+0000","last_peered":"2024-03-20T09:16:33.199433+0000","last_clean":"2024-03-20T09:16:33.199433+0000","last_became_active":"2024-03-20T09:10:35.277481+0000","last_became_peered":"2024-03-20T09:10:35.277481+0000","last_unstale":"2024-03-20T09:16:33.199433+0000","last_undegraded":"2024-03-20T09:16:33.199433+0000","last_fullsized":"2024-03-20T09:16:33.199433+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:10.008935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:15:10.008935+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1b","version":"20'137","reported_seq":984,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-20T09:16:33.200904+0000","last_change":"2024-03-20T09:15:10.569807+0000","last_active":"2024-03-20T09:16:33.200904+0000","last_peered":"2024-03-20T09:16:33.200904+0000","last_clean":"2024-03-20T09:16:33.200904+0000","last_became_active":"2024-03-20T09:07:41.962733+0000","last_became_peered":"2024-03-20T09:07:41.962733+0000","last_unstale":"2024-03-20T09:16:33.200904+0000","last_undegraded":"2024-03-20T09:16:33.200904+0000","last_fullsized":"2024-03-20T09:16:33.200904+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:10.569712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:15:10.569712+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"577'16030","reported_seq":16403,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-20T09:16:47.368909+0000","last_change":"2024-03-20T09:15:53.214593+0000","last_active":"2024-03-20T09:16:47.368909+0000","last_peered":"2024-03-20T09:16:47.368909+0000","last_clean":"2024-03-20T09:16:47.368909+0000","last_became_active":"2024-03-20T09:13:22.063020+0000","last_became_peered":"2024-03-20T09:13:22.063020+0000","last_unstale":"2024-03-20T09:16:47.368909+0000","last_undegraded":"2024-03-20T09:16:47.368909+0000","last_fullsized":"2024-03-20T09:16:47.368909+0000","mapping_epoch":520,"log_start":"547'6000","ondisk_log_start":"547'6000","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'13288","last_scrub_stamp":"2024-03-20T09:15:53.214204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:15:53.214204+0000","objects_scrubbed":5818,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:00.546404+0000","scrub_duration":2512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198115352,"num_objects":3024,"num_object_clones":0,"num_object_copies":9072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3024,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12190,"num_write_kb":486785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"20'137","reported_seq":685,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-20T09:16:33.199457+0000","last_change":"2024-03-20T09:16:20.603395+0000","last_active":"2024-03-20T09:16:33.199457+0000","last_peered":"2024-03-20T09:16:33.199457+0000","last_clean":"2024-03-20T09:16:33.199457+0000","last_became_active":"2024-03-20T09:10:35.215772+0000","last_became_peered":"2024-03-20T09:10:35.215772+0000","last_unstale":"2024-03-20T09:16:33.199457+0000","last_undegraded":"2024-03-20T09:16:33.199457+0000","last_fullsized":"2024-03-20T09:16:33.199457+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:16:20.603281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:16:20.603281+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:27.570145+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"577'11967","reported_seq":12228,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-20T09:16:45.644850+0000","last_change":"2024-03-20T09:16:05.503682+0000","last_active":"2024-03-20T09:16:45.644850+0000","last_peered":"2024-03-20T09:16:45.644850+0000","last_clean":"2024-03-20T09:16:45.644850+0000","last_became_active":"2024-03-20T09:13:22.583317+0000","last_became_peered":"2024-03-20T09:13:22.583317+0000","last_unstale":"2024-03-20T09:16:45.644850+0000","last_undegraded":"2024-03-20T09:16:45.644850+0000","last_fullsized":"2024-03-20T09:16:45.644850+0000","mapping_epoch":520,"log_start":"535'1900","ondisk_log_start":"535'1900","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"570'9920","last_scrub_stamp":"2024-03-20T09:16:05.503555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:16:05.503555+0000","objects_scrubbed":5155,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:26.501339+0000","scrub_duration":1820,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200867840,"num_objects":3065,"num_object_clones":0,"num_object_copies":9195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11967,"num_write_kb":481024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,8],"acting":[3,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"20'137","reported_seq":1237,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-20T09:16:33.212413+0000","last_change":"2024-03-20T09:15:59.752312+0000","last_active":"2024-03-20T09:16:33.212413+0000","last_peered":"2024-03-20T09:16:33.212413+0000","last_clean":"2024-03-20T09:16:33.212413+0000","last_became_active":"2024-03-20T09:10:34.099414+0000","last_became_peered":"2024-03-20T09:10:34.099414+0000","last_unstale":"2024-03-20T09:16:33.212413+0000","last_undegraded":"2024-03-20T09:16:33.212413+0000","last_fullsized":"2024-03-20T09:16:33.212413+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:59.752166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:15:59.752166+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:05.403185+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"577'16262","reported_seq":18511,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-20T09:16:44.058259+0000","last_change":"2024-03-20T09:15:51.667711+0000","last_active":"2024-03-20T09:16:44.058259+0000","last_peered":"2024-03-20T09:16:44.058259+0000","last_clean":"2024-03-20T09:16:44.058259+0000","last_became_active":"2024-03-20T09:13:46.198893+0000","last_became_peered":"2024-03-20T09:13:46.198893+0000","last_unstale":"2024-03-20T09:16:44.058259+0000","last_undegraded":"2024-03-20T09:16:44.058259+0000","last_fullsized":"2024-03-20T09:16:44.058259+0000","mapping_epoch":534,"log_start":"547'6202","ondisk_log_start":"547'6202","created":520,"last_epoch_clean":535,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'13610","last_scrub_stamp":"2024-03-20T09:15:51.667432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:15:51.667432+0000","objects_scrubbed":6005,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:01.763807+0000","scrub_duration":2000,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216727552,"num_objects":3307,"num_object_clones":0,"num_object_copies":9921,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12107,"num_write_kb":493248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":922,"num_bytes_recovered":60424192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,11],"acting":[2,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.a","version":"20'137","reported_seq":1043,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-20T09:16:33.197071+0000","last_change":"2024-03-20T09:15:53.510427+0000","last_active":"2024-03-20T09:16:33.197071+0000","last_peered":"2024-03-20T09:16:33.197071+0000","last_clean":"2024-03-20T09:16:33.197071+0000","last_became_active":"2024-03-20T09:07:52.618107+0000","last_became_peered":"2024-03-20T09:07:52.618107+0000","last_unstale":"2024-03-20T09:16:33.197071+0000","last_undegraded":"2024-03-20T09:16:33.197071+0000","last_fullsized":"2024-03-20T09:16:33.197071+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:53.510342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:15:53.510342+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:08.887351+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"577'11944","reported_seq":12144,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-20T09:16:45.006574+0000","last_change":"2024-03-20T09:15:51.598949+0000","last_active":"2024-03-20T09:16:45.006574+0000","last_peered":"2024-03-20T09:16:45.006574+0000","last_clean":"2024-03-20T09:16:45.006574+0000","last_became_active":"2024-03-20T09:13:22.063975+0000","last_became_peered":"2024-03-20T09:13:22.063975+0000","last_unstale":"2024-03-20T09:16:45.006574+0000","last_undegraded":"2024-03-20T09:16:45.006574+0000","last_fullsized":"2024-03-20T09:16:45.006574+0000","mapping_epoch":520,"log_start":"533'1900","ondisk_log_start":"533'1900","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'9304","last_scrub_stamp":"2024-03-20T09:15:51.598725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:15:51.598725+0000","objects_scrubbed":5732,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:06.892316+0000","scrub_duration":1055,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200933376,"num_objects":3066,"num_object_clones":0,"num_object_copies":9198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11944,"num_write_kb":480320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"20'137","reported_seq":1207,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-20T09:16:33.201071+0000","last_change":"2024-03-20T09:15:58.623124+0000","last_active":"2024-03-20T09:16:33.201071+0000","last_peered":"2024-03-20T09:16:33.201071+0000","last_clean":"2024-03-20T09:16:33.201071+0000","last_became_active":"2024-03-20T09:12:14.306384+0000","last_became_peered":"2024-03-20T09:12:14.306384+0000","last_unstale":"2024-03-20T09:16:33.201071+0000","last_undegraded":"2024-03-20T09:16:33.201071+0000","last_fullsized":"2024-03-20T09:16:33.201071+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:58.623048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:15:58.623048+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:02.219739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"577'11907","reported_seq":12161,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-20T09:16:45.609515+0000","last_change":"2024-03-20T09:16:18.380273+0000","last_active":"2024-03-20T09:16:45.609515+0000","last_peered":"2024-03-20T09:16:45.609515+0000","last_clean":"2024-03-20T09:16:45.609515+0000","last_became_active":"2024-03-20T09:13:24.064091+0000","last_became_peered":"2024-03-20T09:13:24.064091+0000","last_unstale":"2024-03-20T09:16:45.609515+0000","last_undegraded":"2024-03-20T09:16:45.609515+0000","last_fullsized":"2024-03-20T09:16:45.609515+0000","mapping_epoch":522,"log_start":"535'1900","ondisk_log_start":"535'1900","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'10474","last_scrub_stamp":"2024-03-20T09:16:18.380003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:16:18.380003+0000","objects_scrubbed":4519,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:30.515519+0000","scrub_duration":1814,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199426048,"num_objects":3043,"num_object_clones":0,"num_object_copies":9129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11907,"num_write_kb":478400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"20'137","reported_seq":1102,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-20T09:16:33.197606+0000","last_change":"2024-03-20T09:16:04.618419+0000","last_active":"2024-03-20T09:16:33.197606+0000","last_peered":"2024-03-20T09:16:33.197606+0000","last_clean":"2024-03-20T09:16:33.197606+0000","last_became_active":"2024-03-20T09:07:37.003722+0000","last_became_peered":"2024-03-20T09:07:37.003722+0000","last_unstale":"2024-03-20T09:16:33.197606+0000","last_undegraded":"2024-03-20T09:16:33.197606+0000","last_fullsized":"2024-03-20T09:16:33.197606+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:16:04.618325+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:07:06.480160+0000","last_clean_scrub_stamp":"2024-03-20T09:16:04.618325+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:22.080021+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.3","version":"577'12234","reported_seq":12447,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-20T09:16:44.960183+0000","last_change":"2024-03-20T09:16:22.625130+0000","last_active":"2024-03-20T09:16:44.960183+0000","last_peered":"2024-03-20T09:16:44.960183+0000","last_clean":"2024-03-20T09:16:44.960183+0000","last_became_active":"2024-03-20T09:13:22.064275+0000","last_became_peered":"2024-03-20T09:13:22.064275+0000","last_unstale":"2024-03-20T09:16:44.960183+0000","last_undegraded":"2024-03-20T09:16:44.960183+0000","last_fullsized":"2024-03-20T09:16:44.960183+0000","mapping_epoch":520,"log_start":"535'2200","ondisk_log_start":"535'2200","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'7909","last_scrub_stamp":"2024-03-20T09:15:09.333054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:15:09.333054+0000","objects_scrubbed":99,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:45.642404+0000","scrub_duration":1910,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215744512,"num_objects":3292,"num_object_clones":0,"num_object_copies":9876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12234,"num_write_kb":496832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,1],"acting":[11,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"20'137","reported_seq":1119,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-20T09:16:33.197792+0000","last_change":"2024-03-20T09:15:47.574426+0000","last_active":"2024-03-20T09:16:33.197792+0000","last_peered":"2024-03-20T09:16:33.197792+0000","last_clean":"2024-03-20T09:16:33.197792+0000","last_became_active":"2024-03-20T09:07:52.593671+0000","last_became_peered":"2024-03-20T09:07:52.593671+0000","last_unstale":"2024-03-20T09:16:33.197792+0000","last_undegraded":"2024-03-20T09:16:33.197792+0000","last_fullsized":"2024-03-20T09:16:33.197792+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:47.574277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:15:47.574277+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:12.232176+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.6","version":"577'12168","reported_seq":12438,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-20T09:16:47.369091+0000","last_change":"2024-03-20T09:16:06.551173+0000","last_active":"2024-03-20T09:16:47.369091+0000","last_peered":"2024-03-20T09:16:47.369091+0000","last_clean":"2024-03-20T09:16:47.369091+0000","last_became_active":"2024-03-20T09:13:22.067362+0000","last_became_peered":"2024-03-20T09:13:22.067362+0000","last_unstale":"2024-03-20T09:16:47.369091+0000","last_undegraded":"2024-03-20T09:16:47.369091+0000","last_fullsized":"2024-03-20T09:16:47.369091+0000","mapping_epoch":520,"log_start":"535'2100","ondisk_log_start":"535'2100","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"570'10110","last_scrub_stamp":"2024-03-20T09:16:06.551041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:16:06.551041+0000","objects_scrubbed":5111,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:32.144258+0000","scrub_duration":1062,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199753728,"num_objects":3048,"num_object_clones":0,"num_object_copies":9144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12168,"num_write_kb":486912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"20'137","reported_seq":1114,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-20T09:16:47.368853+0000","last_change":"2024-03-20T09:15:55.668210+0000","last_active":"2024-03-20T09:16:47.368853+0000","last_peered":"2024-03-20T09:16:47.368853+0000","last_clean":"2024-03-20T09:16:47.368853+0000","last_became_active":"2024-03-20T09:12:02.221686+0000","last_became_peered":"2024-03-20T09:12:02.221686+0000","last_unstale":"2024-03-20T09:16:47.368853+0000","last_undegraded":"2024-03-20T09:16:47.368853+0000","last_fullsized":"2024-03-20T09:16:47.368853+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:55.668113+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:18.788771+0000","last_clean_scrub_stamp":"2024-03-20T09:15:55.668113+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:12.700115+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,8],"acting":[1,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"577'11883","reported_seq":12140,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-20T09:16:45.764488+0000","last_change":"2024-03-20T09:16:00.516288+0000","last_active":"2024-03-20T09:16:45.764488+0000","last_peered":"2024-03-20T09:16:45.764488+0000","last_clean":"2024-03-20T09:16:45.764488+0000","last_became_active":"2024-03-20T09:13:22.062992+0000","last_became_peered":"2024-03-20T09:13:22.062992+0000","last_unstale":"2024-03-20T09:16:45.764488+0000","last_undegraded":"2024-03-20T09:16:45.764488+0000","last_fullsized":"2024-03-20T09:16:45.764488+0000","mapping_epoch":520,"log_start":"533'1800","ondisk_log_start":"533'1800","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'9630","last_scrub_stamp":"2024-03-20T09:16:00.516080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:16:00.516080+0000","objects_scrubbed":5447,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:29.908467+0000","scrub_duration":937,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207814656,"num_objects":3171,"num_object_clones":0,"num_object_copies":9513,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11883,"num_write_kb":481728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,11],"acting":[6,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":1178,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-20T09:16:33.201412+0000","last_change":"2024-03-20T09:16:12.587235+0000","last_active":"2024-03-20T09:16:33.201412+0000","last_peered":"2024-03-20T09:16:33.201412+0000","last_clean":"2024-03-20T09:16:33.201412+0000","last_became_active":"2024-03-20T09:12:14.305989+0000","last_became_peered":"2024-03-20T09:12:14.305989+0000","last_unstale":"2024-03-20T09:16:33.201412+0000","last_undegraded":"2024-03-20T09:16:33.201412+0000","last_fullsized":"2024-03-20T09:16:33.201412+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:16:12.587129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:16:12.587129+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:21.638063+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.7","version":"577'12104","reported_seq":12368,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-20T09:16:47.369082+0000","last_change":"2024-03-20T09:16:01.692418+0000","last_active":"2024-03-20T09:16:47.369082+0000","last_peered":"2024-03-20T09:16:47.369082+0000","last_clean":"2024-03-20T09:16:47.369082+0000","last_became_active":"2024-03-20T09:13:22.065656+0000","last_became_peered":"2024-03-20T09:13:22.065656+0000","last_unstale":"2024-03-20T09:16:47.369082+0000","last_undegraded":"2024-03-20T09:16:47.369082+0000","last_fullsized":"2024-03-20T09:16:47.369082+0000","mapping_epoch":520,"log_start":"535'2100","ondisk_log_start":"535'2100","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'9867","last_scrub_stamp":"2024-03-20T09:16:01.692239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:16:01.692239+0000","objects_scrubbed":5339,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:08.988153+0000","scrub_duration":2198,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199491584,"num_objects":3044,"num_object_clones":0,"num_object_copies":9132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12104,"num_write_kb":484736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,0],"acting":[1,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"20'137","reported_seq":1427,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-20T09:16:33.222917+0000","last_change":"2024-03-20T09:16:19.056692+0000","last_active":"2024-03-20T09:16:33.222917+0000","last_peered":"2024-03-20T09:16:33.222917+0000","last_clean":"2024-03-20T09:16:33.222917+0000","last_became_active":"2024-03-20T09:12:14.313541+0000","last_became_peered":"2024-03-20T09:12:14.313541+0000","last_unstale":"2024-03-20T09:16:33.222917+0000","last_undegraded":"2024-03-20T09:16:33.222917+0000","last_fullsized":"2024-03-20T09:16:33.222917+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:16:19.056613+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:19.513511+0000","last_clean_scrub_stamp":"2024-03-20T09:16:19.056613+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:23.551840+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"577'15611","reported_seq":15985,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-20T09:16:47.368597+0000","last_change":"2024-03-20T09:15:52.485886+0000","last_active":"2024-03-20T09:16:47.368597+0000","last_peered":"2024-03-20T09:16:47.368597+0000","last_clean":"2024-03-20T09:16:47.368597+0000","last_became_active":"2024-03-20T09:13:24.443054+0000","last_became_peered":"2024-03-20T09:13:24.443054+0000","last_unstale":"2024-03-20T09:16:47.368597+0000","last_undegraded":"2024-03-20T09:16:47.368597+0000","last_fullsized":"2024-03-20T09:16:47.368597+0000","mapping_epoch":522,"log_start":"544'5601","ondisk_log_start":"544'5601","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'12854","last_scrub_stamp":"2024-03-20T09:15:52.485530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:15:52.485530+0000","objects_scrubbed":5795,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:58.805212+0000","scrub_duration":2823,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194969600,"num_objects":2975,"num_object_clones":0,"num_object_copies":8925,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11951,"num_write_kb":477632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"20'137","reported_seq":580,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-20T09:16:33.196759+0000","last_change":"2024-03-20T09:16:13.706013+0000","last_active":"2024-03-20T09:16:33.196759+0000","last_peered":"2024-03-20T09:16:33.196759+0000","last_clean":"2024-03-20T09:16:33.196759+0000","last_became_active":"2024-03-20T09:15:34.225022+0000","last_became_peered":"2024-03-20T09:15:34.225022+0000","last_unstale":"2024-03-20T09:16:33.196759+0000","last_undegraded":"2024-03-20T09:16:33.196759+0000","last_fullsized":"2024-03-20T09:16:33.196759+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:16:13.705862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:16:13.705862+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:25.043793+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,5],"acting":[6,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"577'12187","reported_seq":12457,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-20T09:16:47.179762+0000","last_change":"2024-03-20T09:15:51.597847+0000","last_active":"2024-03-20T09:16:47.179762+0000","last_peered":"2024-03-20T09:16:47.179762+0000","last_clean":"2024-03-20T09:16:47.179762+0000","last_became_active":"2024-03-20T09:13:22.583529+0000","last_became_peered":"2024-03-20T09:13:22.583529+0000","last_unstale":"2024-03-20T09:16:47.179762+0000","last_undegraded":"2024-03-20T09:16:47.179762+0000","last_fullsized":"2024-03-20T09:16:47.179762+0000","mapping_epoch":520,"log_start":"535'2100","ondisk_log_start":"535'2100","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'9317","last_scrub_stamp":"2024-03-20T09:15:51.597715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:15:51.597715+0000","objects_scrubbed":5894,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:18.767531+0000","scrub_duration":911,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196935680,"num_objects":3005,"num_object_clones":0,"num_object_copies":9015,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12187,"num_write_kb":486144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,8],"acting":[5,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"20'137","reported_seq":1115,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-20T09:16:33.197038+0000","last_change":"2024-03-20T09:15:58.696682+0000","last_active":"2024-03-20T09:16:33.197038+0000","last_peered":"2024-03-20T09:16:33.197038+0000","last_clean":"2024-03-20T09:16:33.197038+0000","last_became_active":"2024-03-20T09:13:27.087760+0000","last_became_peered":"2024-03-20T09:13:27.087760+0000","last_unstale":"2024-03-20T09:16:33.197038+0000","last_undegraded":"2024-03-20T09:16:33.197038+0000","last_fullsized":"2024-03-20T09:16:33.197038+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:58.696597+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:14:10.560890+0000","last_clean_scrub_stamp":"2024-03-20T09:15:58.696597+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:00.568581+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"577'12035","reported_seq":12300,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-20T09:16:46.264018+0000","last_change":"2024-03-20T09:15:54.984595+0000","last_active":"2024-03-20T09:16:46.264018+0000","last_peered":"2024-03-20T09:16:46.264018+0000","last_clean":"2024-03-20T09:16:46.264018+0000","last_became_active":"2024-03-20T09:13:22.064461+0000","last_became_peered":"2024-03-20T09:13:22.064461+0000","last_unstale":"2024-03-20T09:16:46.264018+0000","last_undegraded":"2024-03-20T09:16:46.264018+0000","last_fullsized":"2024-03-20T09:16:46.264018+0000","mapping_epoch":520,"log_start":"535'2000","ondisk_log_start":"535'2000","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'9446","last_scrub_stamp":"2024-03-20T09:15:54.984433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:15:54.984433+0000","objects_scrubbed":5711,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:23.039467+0000","scrub_duration":1570,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201261056,"num_objects":3071,"num_object_clones":0,"num_object_copies":9213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12035,"num_write_kb":483392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,1],"acting":[10,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.b","version":"20'137","reported_seq":983,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-20T09:16:33.201215+0000","last_change":"2024-03-20T09:15:54.613271+0000","last_active":"2024-03-20T09:16:33.201215+0000","last_peered":"2024-03-20T09:16:33.201215+0000","last_clean":"2024-03-20T09:16:33.201215+0000","last_became_active":"2024-03-20T09:13:20.070596+0000","last_became_peered":"2024-03-20T09:13:20.070596+0000","last_unstale":"2024-03-20T09:16:33.201215+0000","last_undegraded":"2024-03-20T09:16:33.201215+0000","last_fullsized":"2024-03-20T09:16:33.201215+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:54.613162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:15:54.613162+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:15.365704+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.b","version":"577'11834","reported_seq":12086,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-20T09:16:46.964940+0000","last_change":"2024-03-20T09:16:19.692214+0000","last_active":"2024-03-20T09:16:46.964940+0000","last_peered":"2024-03-20T09:16:46.964940+0000","last_clean":"2024-03-20T09:16:46.964940+0000","last_became_active":"2024-03-20T09:13:22.064687+0000","last_became_peered":"2024-03-20T09:13:22.064687+0000","last_unstale":"2024-03-20T09:16:46.964940+0000","last_undegraded":"2024-03-20T09:16:46.964940+0000","last_fullsized":"2024-03-20T09:16:46.964940+0000","mapping_epoch":520,"log_start":"533'1800","ondisk_log_start":"533'1800","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"573'10496","last_scrub_stamp":"2024-03-20T09:16:19.691908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:16:19.691908+0000","objects_scrubbed":4445,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:43.398249+0000","scrub_duration":1636,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201850880,"num_objects":3080,"num_object_clones":0,"num_object_copies":9240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11834,"num_write_kb":477248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"20'137","reported_seq":644,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-20T09:16:33.211375+0000","last_change":"2024-03-20T09:15:46.735519+0000","last_active":"2024-03-20T09:16:33.211375+0000","last_peered":"2024-03-20T09:16:33.211375+0000","last_clean":"2024-03-20T09:16:33.211375+0000","last_became_active":"2024-03-20T09:12:14.311319+0000","last_became_peered":"2024-03-20T09:12:14.311319+0000","last_unstale":"2024-03-20T09:16:33.211375+0000","last_undegraded":"2024-03-20T09:16:33.211375+0000","last_fullsized":"2024-03-20T09:16:33.211375+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:46.735418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:15:46.735418+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:08.164707+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"577'11928","reported_seq":12186,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-20T09:16:45.774051+0000","last_change":"2024-03-20T09:16:05.995158+0000","last_active":"2024-03-20T09:16:45.774051+0000","last_peered":"2024-03-20T09:16:45.774051+0000","last_clean":"2024-03-20T09:16:45.774051+0000","last_became_active":"2024-03-20T09:13:22.063927+0000","last_became_peered":"2024-03-20T09:13:22.063927+0000","last_unstale":"2024-03-20T09:16:45.774051+0000","last_undegraded":"2024-03-20T09:16:45.774051+0000","last_fullsized":"2024-03-20T09:16:45.774051+0000","mapping_epoch":520,"log_start":"535'1900","ondisk_log_start":"535'1900","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"570'9915","last_scrub_stamp":"2024-03-20T09:16:05.995021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:16:05.995021+0000","objects_scrubbed":5156,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:09.132019+0000","scrub_duration":1382,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203816960,"num_objects":3110,"num_object_clones":0,"num_object_copies":9330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11928,"num_write_kb":481216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"20'137","reported_seq":1131,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-20T09:16:33.196923+0000","last_change":"2024-03-20T09:15:48.626548+0000","last_active":"2024-03-20T09:16:33.196923+0000","last_peered":"2024-03-20T09:16:33.196923+0000","last_clean":"2024-03-20T09:16:33.196923+0000","last_became_active":"2024-03-20T09:10:35.119706+0000","last_became_peered":"2024-03-20T09:10:35.119706+0000","last_unstale":"2024-03-20T09:16:33.196923+0000","last_undegraded":"2024-03-20T09:16:33.196923+0000","last_fullsized":"2024-03-20T09:16:33.196923+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:48.626365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:15:48.626365+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:13.935726+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"577'15714","reported_seq":16089,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-20T09:16:44.975365+0000","last_change":"2024-03-20T09:15:50.446152+0000","last_active":"2024-03-20T09:16:44.975365+0000","last_peered":"2024-03-20T09:16:44.975365+0000","last_clean":"2024-03-20T09:16:44.975365+0000","last_became_active":"2024-03-20T09:13:24.065349+0000","last_became_peered":"2024-03-20T09:13:24.065349+0000","last_unstale":"2024-03-20T09:16:44.975365+0000","last_undegraded":"2024-03-20T09:16:44.975365+0000","last_fullsized":"2024-03-20T09:16:44.975365+0000","mapping_epoch":522,"log_start":"545'5700","ondisk_log_start":"545'5700","created":520,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'13038","last_scrub_stamp":"2024-03-20T09:15:50.445858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:15:50.445858+0000","objects_scrubbed":5938,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:58.046301+0000","scrub_duration":1870,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211091456,"num_objects":3221,"num_object_clones":0,"num_object_copies":9663,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11935,"num_write_kb":484992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,3],"acting":[11,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"570'138","reported_seq":1159,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-20T09:16:33.212175+0000","last_change":"2024-03-20T09:16:13.725190+0000","last_active":"2024-03-20T09:16:33.212175+0000","last_peered":"2024-03-20T09:16:33.212175+0000","last_clean":"2024-03-20T09:16:33.212175+0000","last_became_active":"2024-03-20T09:13:27.087686+0000","last_became_peered":"2024-03-20T09:13:27.087686+0000","last_unstale":"2024-03-20T09:16:33.212175+0000","last_undegraded":"2024-03-20T09:16:33.212175+0000","last_fullsized":"2024-03-20T09:16:33.212175+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"570'138","last_scrub_stamp":"2024-03-20T09:16:13.725035+0000","last_deep_scrub":"570'138","last_deep_scrub_stamp":"2024-03-20T09:16:13.725035+0000","last_clean_scrub_stamp":"2024-03-20T09:16:13.725035+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:23.157908+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"577'11887","reported_seq":12094,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-20T09:16:44.963445+0000","last_change":"2024-03-20T09:15:05.380631+0000","last_active":"2024-03-20T09:16:44.963445+0000","last_peered":"2024-03-20T09:16:44.963445+0000","last_clean":"2024-03-20T09:16:44.963445+0000","last_became_active":"2024-03-20T09:13:22.063673+0000","last_became_peered":"2024-03-20T09:13:22.063673+0000","last_unstale":"2024-03-20T09:16:44.963445+0000","last_undegraded":"2024-03-20T09:16:44.963445+0000","last_fullsized":"2024-03-20T09:16:44.963445+0000","mapping_epoch":520,"log_start":"533'1800","ondisk_log_start":"533'1800","created":520,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":0,"last_scrub":"553'7531","last_scrub_stamp":"2024-03-20T09:15:05.380547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:13:21.066370+0000","last_clean_scrub_stamp":"2024-03-20T09:15:05.380547+0000","objects_scrubbed":7413,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:45.642443+0000","scrub_duration":962,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199950336,"num_objects":3051,"num_object_clones":0,"num_object_copies":9153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11887,"num_write_kb":478016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"20'137","reported_seq":704,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-20T09:16:47.368993+0000","last_change":"2024-03-20T09:15:07.365971+0000","last_active":"2024-03-20T09:16:47.368993+0000","last_peered":"2024-03-20T09:16:47.368993+0000","last_clean":"2024-03-20T09:16:47.368993+0000","last_became_active":"2024-03-20T09:12:14.307864+0000","last_became_peered":"2024-03-20T09:12:14.307864+0000","last_unstale":"2024-03-20T09:16:47.368993+0000","last_undegraded":"2024-03-20T09:16:47.368993+0000","last_fullsized":"2024-03-20T09:16:47.368993+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:07.365857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:15:07.365857+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:49.452418+0000","scrub_duration":268,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"20'137","reported_seq":1026,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-20T09:16:33.199578+0000","last_change":"2024-03-20T09:16:06.498718+0000","last_active":"2024-03-20T09:16:33.199578+0000","last_peered":"2024-03-20T09:16:33.199578+0000","last_clean":"2024-03-20T09:16:33.199578+0000","last_became_active":"2024-03-20T09:10:34.096758+0000","last_became_peered":"2024-03-20T09:10:34.096758+0000","last_unstale":"2024-03-20T09:16:33.199578+0000","last_undegraded":"2024-03-20T09:16:33.199578+0000","last_fullsized":"2024-03-20T09:16:33.199578+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:16:06.498606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:16:06.498606+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:27.510987+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"20'137","reported_seq":1101,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-20T09:16:47.368742+0000","last_change":"2024-03-20T09:16:07.646493+0000","last_active":"2024-03-20T09:16:47.368742+0000","last_peered":"2024-03-20T09:16:47.368742+0000","last_clean":"2024-03-20T09:16:47.368742+0000","last_became_active":"2024-03-20T09:04:04.103645+0000","last_became_peered":"2024-03-20T09:04:04.103645+0000","last_unstale":"2024-03-20T09:16:47.368742+0000","last_undegraded":"2024-03-20T09:16:47.368742+0000","last_fullsized":"2024-03-20T09:16:47.368742+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:16:07.646373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:16:07.646373+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:37.592817+0000","scrub_duration":157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"20'137","reported_seq":552,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-20T09:16:33.199558+0000","last_change":"2024-03-20T09:16:16.579338+0000","last_active":"2024-03-20T09:16:33.199558+0000","last_peered":"2024-03-20T09:16:33.199558+0000","last_clean":"2024-03-20T09:16:33.199558+0000","last_became_active":"2024-03-20T09:10:34.097191+0000","last_became_peered":"2024-03-20T09:10:34.097191+0000","last_unstale":"2024-03-20T09:16:33.199558+0000","last_undegraded":"2024-03-20T09:16:33.199558+0000","last_fullsized":"2024-03-20T09:16:33.199558+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:16:16.579199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:16:16.579199+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:35.191503+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"20'137","reported_seq":963,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-20T09:16:33.213418+0000","last_change":"2024-03-20T09:16:06.090018+0000","last_active":"2024-03-20T09:16:33.213418+0000","last_peered":"2024-03-20T09:16:33.213418+0000","last_clean":"2024-03-20T09:16:33.213418+0000","last_became_active":"2024-03-20T09:13:27.087859+0000","last_became_peered":"2024-03-20T09:13:27.087859+0000","last_unstale":"2024-03-20T09:16:33.213418+0000","last_undegraded":"2024-03-20T09:16:33.213418+0000","last_fullsized":"2024-03-20T09:16:33.213418+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:16:06.089869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:16:06.089869+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:25.886365+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,3],"acting":[0,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"20'137","reported_seq":790,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-20T09:16:47.369202+0000","last_change":"2024-03-20T09:16:15.449041+0000","last_active":"2024-03-20T09:16:47.369202+0000","last_peered":"2024-03-20T09:16:47.369202+0000","last_clean":"2024-03-20T09:16:47.369202+0000","last_became_active":"2024-03-20T09:13:27.087760+0000","last_became_peered":"2024-03-20T09:13:27.087760+0000","last_unstale":"2024-03-20T09:16:47.369202+0000","last_undegraded":"2024-03-20T09:16:47.369202+0000","last_fullsized":"2024-03-20T09:16:47.369202+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:16:15.448920+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:16:15.448920+0000","last_clean_scrub_stamp":"2024-03-20T09:16:15.448920+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:24.424610+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"20'137","reported_seq":1106,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-20T09:16:33.201571+0000","last_change":"2024-03-20T09:15:54.388457+0000","last_active":"2024-03-20T09:16:33.201571+0000","last_peered":"2024-03-20T09:16:33.201571+0000","last_clean":"2024-03-20T09:16:33.201571+0000","last_became_active":"2024-03-20T09:10:35.095339+0000","last_became_peered":"2024-03-20T09:10:35.095339+0000","last_unstale":"2024-03-20T09:16:33.201571+0000","last_undegraded":"2024-03-20T09:16:33.201571+0000","last_fullsized":"2024-03-20T09:16:33.201571+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:54.388308+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:23.640533+0000","last_clean_scrub_stamp":"2024-03-20T09:15:54.388308+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:09.582019+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"20'137","reported_seq":1175,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-20T09:16:33.216030+0000","last_change":"2024-03-20T09:16:15.754498+0000","last_active":"2024-03-20T09:16:33.216030+0000","last_peered":"2024-03-20T09:16:33.216030+0000","last_clean":"2024-03-20T09:16:33.216030+0000","last_became_active":"2024-03-20T09:04:14.208448+0000","last_became_peered":"2024-03-20T09:04:14.208448+0000","last_unstale":"2024-03-20T09:16:33.216030+0000","last_undegraded":"2024-03-20T09:16:33.216030+0000","last_fullsized":"2024-03-20T09:16:33.216030+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:16:15.754362+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:16:15.754362+0000","last_clean_scrub_stamp":"2024-03-20T09:16:15.754362+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:17.375529+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"20'137","reported_seq":440,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-20T09:16:33.207948+0000","last_change":"2024-03-20T09:15:55.844234+0000","last_active":"2024-03-20T09:16:33.207948+0000","last_peered":"2024-03-20T09:16:33.207948+0000","last_clean":"2024-03-20T09:16:33.207948+0000","last_became_active":"2024-03-20T09:12:14.307581+0000","last_became_peered":"2024-03-20T09:12:14.307581+0000","last_unstale":"2024-03-20T09:16:33.207948+0000","last_undegraded":"2024-03-20T09:16:33.207948+0000","last_fullsized":"2024-03-20T09:16:33.207948+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:55.844132+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:07:06.480160+0000","last_clean_scrub_stamp":"2024-03-20T09:15:55.844132+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:17:03.670100+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"20'137","reported_seq":713,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-03-20T09:16:47.368843+0000","last_change":"2024-03-20T09:15:51.692096+0000","last_active":"2024-03-20T09:16:47.368843+0000","last_peered":"2024-03-20T09:16:47.368843+0000","last_clean":"2024-03-20T09:16:47.368843+0000","last_became_active":"2024-03-20T09:13:20.074439+0000","last_became_peered":"2024-03-20T09:13:20.074439+0000","last_unstale":"2024-03-20T09:16:47.368843+0000","last_undegraded":"2024-03-20T09:16:47.368843+0000","last_fullsized":"2024-03-20T09:16:47.368843+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:15:51.691974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:15:51.691974+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:16:56.756163+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,11],"acting":[1,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":3248750616,"num_objects":49573,"num_object_clones":0,"num_object_copies":148719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49573,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":192261,"num_write_kb":7738625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":922,"num_bytes_recovered":60424192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9731518464,"data_stored":9731506248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160692,"ondisk_log_size":160692,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4248,"ondisk_log_size":4248,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860664,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2047096,"kb_used_data":1628592,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":91698056,"statfs":{"total":95995035648,"available":93898809344,"internally_reserved":0,"allocated":1667678208,"data_stored":1667386452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":428538308},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":89,"seq":382252089598,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1093492,"kb_used_data":805484,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":92651660,"statfs":{"total":95995035648,"available":94875299840,"internally_reserved":0,"allocated":824815616,"data_stored":824527932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":294910400},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":0,"seq":343597383941,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":70,"seq":300647710986,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":641908,"kb_used_data":389996,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":93103244,"statfs":{"total":95995035648,"available":95337721856,"internally_reserved":0,"allocated":399355904,"data_stored":399068220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":257948102},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":60,"seq":257698038031,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1286716,"kb_used_data":796020,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":92458436,"statfs":{"total":95995035648,"available":94677438464,"internally_reserved":0,"allocated":815124480,"data_stored":814833244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":502462922},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332372,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1560180,"kb_used_data":1190252,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":92184972,"statfs":{"total":95995035648,"available":94397411328,"internally_reserved":0,"allocated":1218818048,"data_stored":1218530364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":378796486},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575150,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2299576,"kb_used_data":1784880,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":91445576,"statfs":{"total":95995035648,"available":93640269824,"internally_reserved":0,"allocated":1827717120,"data_stored":1827425364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":526645702},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738669,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1221880,"kb_used_data":789424,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":92523272,"statfs":{"total":95995035648,"available":94743830528,"internally_reserved":0,"allocated":808370176,"data_stored":808078420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411623,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":900796,"kb_used_data":621108,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":92844356,"statfs":{"total":95995035648,"available":95072620544,"internally_reserved":0,"allocated":636014592,"data_stored":635723356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":286390725},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149987,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1408948,"kb_used_data":1012460,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":92336204,"statfs":{"total":95995035648,"available":94552272896,"internally_reserved":0,"allocated":1036759040,"data_stored":1036471356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":405797319},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953761,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":443,"seq":1902670512204,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":747644,"kb_used_data":583028,"kb_used_omap":181,"kb_used_meta":164426,"kb_avail":92997508,"statfs":{"total":95995035648,"available":95229448192,"internally_reserved":0,"allocated":597020672,"data_stored":596729436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":186131,"internal_metadata":168372461},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":796921856,"data_stored":796917784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1800671232,"data_stored":1800667160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":632291328,"data_stored":632291328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1018494976,"data_stored":1018494976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":591855616,"data_stored":591855616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1209597952,"data_stored":1209597952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":812449792,"data_stored":812449792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":395313152,"data_stored":395313152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":813432832,"data_stored":813432832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1660489728,"data_stored":1660485656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:16:50.801 INFO:tasks.thrashosds.thrasher:Setting 4.b to [8, 0, 7] 2024-03-20T09:16:50.801 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '4.b', '8', '0', '7'] 2024-03-20T09:16:50.801 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pg-upmap 4.b 8 0 7 2024-03-20T09:16:50.811 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:16:50.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:50.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:16:50.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:50 smithi115 ceph-mon[72476]: from='client.25831 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:16:50.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:50 smithi115 ceph-mon[72476]: 4.3 scrub ok 2024-03-20T09:16:50.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:50 smithi115 ceph-mon[72476]: pgmap v1385: 47 pgs: 47 active+clean; 3.0 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 828 op/s 2024-03-20T09:16:50.918 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:16:50.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:50 smithi122 ceph-mon[73495]: from='client.25831 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:16:50.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:50 smithi122 ceph-mon[73495]: 4.3 scrub ok 2024-03-20T09:16:50.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:50 smithi122 ceph-mon[73495]: pgmap v1385: 47 pgs: 47 active+clean; 3.0 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 828 op/s 2024-03-20T09:16:50.996 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:51.402 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:16:51.402 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:16:51.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:51.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:16:51.505 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:16:51.542 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:51.574 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:16:51.578 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:16:51.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:51.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:16:51.656 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:51 smithi026 ceph-mon[68309]: 1.19 scrub starts 2024-03-20T09:16:51.656 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:51 smithi026 ceph-mon[68309]: 1.19 scrub ok 2024-03-20T09:16:51.683 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:16:51.731 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:51.752 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:16:51.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:51.755 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:16:51.856 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:16:51.885 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:51 smithi122 ceph-mon[73495]: 1.19 scrub starts 2024-03-20T09:16:51.885 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:51 smithi122 ceph-mon[73495]: 1.19 scrub ok 2024-03-20T09:16:51.885 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:51.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:51 smithi115 ceph-mon[72476]: 1.19 scrub starts 2024-03-20T09:16:51.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:51 smithi115 ceph-mon[72476]: 1.19 scrub ok 2024-03-20T09:16:51.908 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:16:51.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:51.910 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:16:52.014 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:16:52.044 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:52.069 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:16:52.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:52.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:16:52.175 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:16:52.214 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:52.236 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:16:52.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:52.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:16:52.345 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:16:52.378 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:52.409 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:16:52.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:52.413 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:16:52.514 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:16:52.541 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:52.566 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:16:52.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:52.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:16:52.670 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:16:52.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:52 smithi122 ceph-mon[73495]: 1.1e scrub starts 2024-03-20T09:16:52.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:52 smithi122 ceph-mon[73495]: 1.1e scrub ok 2024-03-20T09:16:52.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:52 smithi122 ceph-mon[73495]: pgmap v1386: 47 pgs: 47 active+clean; 2.9 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 825 op/s 2024-03-20T09:16:52.701 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:52.725 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:16:52.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:52.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:16:52.830 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:16:52.856 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:52 smithi115 ceph-mon[72476]: 1.1e scrub starts 2024-03-20T09:16:52.856 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:52 smithi115 ceph-mon[72476]: 1.1e scrub ok 2024-03-20T09:16:52.856 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:52 smithi115 ceph-mon[72476]: pgmap v1386: 47 pgs: 47 active+clean; 2.9 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 825 op/s 2024-03-20T09:16:52.858 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:52.884 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:16:52.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:52.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:16:52.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:52 smithi026 ceph-mon[68309]: 1.1e scrub starts 2024-03-20T09:16:52.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:52 smithi026 ceph-mon[68309]: 1.1e scrub ok 2024-03-20T09:16:52.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:52 smithi026 ceph-mon[68309]: pgmap v1386: 47 pgs: 47 active+clean; 2.9 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 825 op/s 2024-03-20T09:16:52.989 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:16:53.026 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:53.059 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:16:53.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:53.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:16:53.166 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:16:53.199 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:53.223 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:16:53.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:53.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:16:53.328 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:16:53.414 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:53.545 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:16:53.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:53.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:16:53.652 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:16:53.686 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:53.712 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:16:53.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:53.716 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:16:53.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:53 smithi026 ceph-mon[68309]: 1.18 scrub starts 2024-03-20T09:16:53.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:53 smithi026 ceph-mon[68309]: 1.18 scrub ok 2024-03-20T09:16:53.817 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:16:53.888 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:53.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:53 smithi115 ceph-mon[72476]: 1.18 scrub starts 2024-03-20T09:16:53.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:53 smithi115 ceph-mon[72476]: 1.18 scrub ok 2024-03-20T09:16:53.936 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:16:53.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:53.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:16:53.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:53 smithi122 ceph-mon[73495]: 1.18 scrub starts 2024-03-20T09:16:53.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:53 smithi122 ceph-mon[73495]: 1.18 scrub ok 2024-03-20T09:16:54.043 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:16:54.071 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:54.106 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:16:54.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:54.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:16:54.214 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:16:54.244 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:54.271 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:16:54.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:54.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:16:54.379 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:16:54.421 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:54.445 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:16:54.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:54.448 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:16:54.549 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:16:54.598 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:54.627 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:16:54.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:54.632 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:16:54.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:54 smithi122 ceph-mon[73495]: 1.1b scrub starts 2024-03-20T09:16:54.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:54 smithi122 ceph-mon[73495]: 1.1b scrub ok 2024-03-20T09:16:54.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:54 smithi122 ceph-mon[73495]: pgmap v1387: 47 pgs: 47 active+clean; 2.8 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 818 op/s 2024-03-20T09:16:54.734 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:16:54.769 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:54.790 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:16:54.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:54.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:16:54.895 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:16:54.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:54 smithi115 ceph-mon[72476]: 1.1b scrub starts 2024-03-20T09:16:54.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:54 smithi115 ceph-mon[72476]: 1.1b scrub ok 2024-03-20T09:16:54.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:54 smithi115 ceph-mon[72476]: pgmap v1387: 47 pgs: 47 active+clean; 2.8 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 818 op/s 2024-03-20T09:16:54.918 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:54 smithi026 ceph-mon[68309]: 1.1b scrub starts 2024-03-20T09:16:54.918 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:54 smithi026 ceph-mon[68309]: 1.1b scrub ok 2024-03-20T09:16:54.918 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:54 smithi026 ceph-mon[68309]: pgmap v1387: 47 pgs: 47 active+clean; 2.8 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 818 op/s 2024-03-20T09:16:54.923 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:54.943 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:16:54.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:54.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:16:55.048 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:16:55.075 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:55.099 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:16:55.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:55.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:16:55.204 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:16:55.236 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:55.259 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:16:55.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:55.262 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:16:55.363 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:16:55.398 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:55.421 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:16:55.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:55.426 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:16:55.527 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:16:55.556 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:55.576 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:16:55.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:55.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:16:55.683 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:16:55.723 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:55.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:55 smithi026 ceph-mon[68309]: 1.d scrub starts 2024-03-20T09:16:55.755 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:16:55.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:55.759 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:16:55.860 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:16:55.884 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:55 smithi122 ceph-mon[73495]: 1.d scrub starts 2024-03-20T09:16:55.887 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:55.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:55 smithi115 ceph-mon[72476]: 1.d scrub starts 2024-03-20T09:16:55.915 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:16:55.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:55.922 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:16:56.023 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:16:56.085 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:56.639 INFO:teuthology.orchestra.run.smithi026.stderr:set 4.b pg_upmap mapping to [8,0,7] 2024-03-20T09:16:56.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:56 smithi115 ceph-mon[72476]: 1.d scrub ok 2024-03-20T09:16:56.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:56 smithi115 ceph-mon[72476]: pgmap v1388: 47 pgs: 47 active+clean; 2.6 GiB data, 12 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 814 op/s 2024-03-20T09:16:56.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:56 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3039460595' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.b", "id": [8, 0, 7]}]: dispatch 2024-03-20T09:16:56.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:56 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.b", "id": [8, 0, 7]}]: dispatch 2024-03-20T09:16:56.928 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:56 smithi026 ceph-mon[68309]: 1.d scrub ok 2024-03-20T09:16:56.928 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:56 smithi026 ceph-mon[68309]: pgmap v1388: 47 pgs: 47 active+clean; 2.6 GiB data, 12 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 814 op/s 2024-03-20T09:16:56.928 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:56 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3039460595' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.b", "id": [8, 0, 7]}]: dispatch 2024-03-20T09:16:56.928 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:56 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.b", "id": [8, 0, 7]}]: dispatch 2024-03-20T09:16:56.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:56 smithi122 ceph-mon[73495]: 1.d scrub ok 2024-03-20T09:16:56.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:56 smithi122 ceph-mon[73495]: pgmap v1388: 47 pgs: 47 active+clean; 2.6 GiB data, 12 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 814 op/s 2024-03-20T09:16:56.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:56 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3039460595' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.b", "id": [8, 0, 7]}]: dispatch 2024-03-20T09:16:56.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:56 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.b", "id": [8, 0, 7]}]: dispatch 2024-03-20T09:16:57.280 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:16:57.280 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:16:57.280 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-03-20T09:16:57.280 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:16:57.563 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:16:57.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:57.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:16:57.674 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:16:57.702 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:57.721 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:16:57.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:57.724 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:16:57.825 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:16:57.852 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:57.863 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:57 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.b", "id": [8, 0, 7]}]': finished 2024-03-20T09:16:57.863 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:57 smithi026 ceph-mon[68309]: osdmap e579: 12 total, 12 up, 10 in 2024-03-20T09:16:57.863 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:57 smithi026 ceph-mon[68309]: osdmap e580: 12 total, 12 up, 10 in 2024-03-20T09:16:57.863 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:57 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:16:57.864 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:57 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:16:57.871 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:16:57.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:57.874 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:16:57.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:57 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.b", "id": [8, 0, 7]}]': finished 2024-03-20T09:16:57.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:57 smithi115 ceph-mon[72476]: osdmap e579: 12 total, 12 up, 10 in 2024-03-20T09:16:57.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:57 smithi115 ceph-mon[72476]: osdmap e580: 12 total, 12 up, 10 in 2024-03-20T09:16:57.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:57 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:16:57.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:57 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:16:57.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:57 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.b", "id": [8, 0, 7]}]': finished 2024-03-20T09:16:57.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:57 smithi122 ceph-mon[73495]: osdmap e579: 12 total, 12 up, 10 in 2024-03-20T09:16:57.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:57 smithi122 ceph-mon[73495]: osdmap e580: 12 total, 12 up, 10 in 2024-03-20T09:16:57.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:57 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:16:57.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:57 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:16:57.975 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:16:58.003 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:58.021 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:16:58.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:58.025 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:16:58.107 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:16:58.127 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:16:58.161 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:58.165 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:16:58.183 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:16:58.184 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:16:58.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:58.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:16:58.209 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:16:58.243 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:16:58.267 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:16:58.287 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:16:58.288 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:16:58.307 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:16:58.313 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:58.327 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:16:58.331 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:16:58.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:58.334 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:16:58.347 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:16:58.367 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:16:58.388 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:16:58.410 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:16:58.411 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:16:58.411 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:16:58.411 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:16:58.411 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:16:58.411 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:16:58.411 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:16:58.411 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:16:58.411 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:16:58.411 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:16:58.412 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:16:58.412 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:16:58.412 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:16:58.434 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:16:58.465 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:58.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:58 smithi115 ceph-mon[72476]: 1.1a scrub starts 2024-03-20T09:16:58.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:58 smithi115 ceph-mon[72476]: 1.1a scrub ok 2024-03-20T09:16:58.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:58 smithi115 ceph-mon[72476]: pgmap v1391: 47 pgs: 1 unknown, 46 active+clean; 2.4 GiB data, 11 GiB used, 883 GiB / 894 GiB avail 2024-03-20T09:16:58.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:58 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/704440796' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:16:58.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:58 smithi115 ceph-mon[72476]: osdmap e581: 12 total, 12 up, 10 in 2024-03-20T09:16:58.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:58 smithi122 ceph-mon[73495]: 1.1a scrub starts 2024-03-20T09:16:58.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:58 smithi122 ceph-mon[73495]: 1.1a scrub ok 2024-03-20T09:16:58.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:58 smithi122 ceph-mon[73495]: pgmap v1391: 47 pgs: 1 unknown, 46 active+clean; 2.4 GiB data, 11 GiB used, 883 GiB / 894 GiB avail 2024-03-20T09:16:58.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:58 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/704440796' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:16:58.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:58 smithi122 ceph-mon[73495]: osdmap e581: 12 total, 12 up, 10 in 2024-03-20T09:16:58.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:58 smithi026 ceph-mon[68309]: 1.1a scrub starts 2024-03-20T09:16:58.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:58 smithi026 ceph-mon[68309]: 1.1a scrub ok 2024-03-20T09:16:58.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:58 smithi026 ceph-mon[68309]: pgmap v1391: 47 pgs: 1 unknown, 46 active+clean; 2.4 GiB data, 11 GiB used, 883 GiB / 894 GiB avail 2024-03-20T09:16:58.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:58 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/704440796' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:16:58.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:58 smithi026 ceph-mon[68309]: osdmap e581: 12 total, 12 up, 10 in 2024-03-20T09:16:59.118 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:16:59.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:59.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:16:59.148 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-20T09:16:59.228 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:16:59.255 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:59.291 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:16:59.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:59.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:16:59.396 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:16:59.425 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:59.450 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:16:59.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:16:59.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:16:59.554 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:16:59.662 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:16:59.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:59 smithi115 ceph-mon[72476]: 4.b starting backfill to osd.7 from (0'0,0'0] MAX to 578'12317 2024-03-20T09:16:59.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:59 smithi115 ceph-mon[72476]: 4.b starting backfill to osd.8 from (0'0,0'0] MAX to 578'12317 2024-03-20T09:16:59.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:59 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:16:59.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:59 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:16:59.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:16:59 smithi115 ceph-mon[72476]: 4.4 scrub starts 2024-03-20T09:16:59.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:59 smithi122 ceph-mon[73495]: 4.b starting backfill to osd.7 from (0'0,0'0] MAX to 578'12317 2024-03-20T09:16:59.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:59 smithi122 ceph-mon[73495]: 4.b starting backfill to osd.8 from (0'0,0'0] MAX to 578'12317 2024-03-20T09:16:59.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:59 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:16:59.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:59 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:16:59.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:16:59 smithi122 ceph-mon[73495]: 4.4 scrub starts 2024-03-20T09:16:59.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:59 smithi026 ceph-mon[68309]: 4.b starting backfill to osd.7 from (0'0,0'0] MAX to 578'12317 2024-03-20T09:16:59.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:59 smithi026 ceph-mon[68309]: 4.b starting backfill to osd.8 from (0'0,0'0] MAX to 578'12317 2024-03-20T09:16:59.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:59 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:16:59.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:59 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:16:59.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:16:59 smithi026 ceph-mon[68309]: 4.4 scrub starts 2024-03-20T09:17:00.353 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:17:00.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:00.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:17:00.460 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:17:00.490 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:00.514 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:17:00.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:00.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:17:00.620 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:17:00.651 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:00.682 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:17:00.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:00.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:17:00.788 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:17:00.824 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:00 smithi115 ceph-mon[72476]: osdmap e582: 12 total, 12 up, 10 in 2024-03-20T09:17:00.824 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:00 smithi115 ceph-mon[72476]: pgmap v1394: 47 pgs: 1 unknown, 46 active+clean; 2.4 GiB data, 11 GiB used, 883 GiB / 894 GiB avail 2024-03-20T09:17:00.824 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:00 smithi115 ceph-mon[72476]: 4.8 scrub starts 2024-03-20T09:17:00.824 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:00 smithi115 ceph-mon[72476]: 4.4 scrub ok 2024-03-20T09:17:00.828 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:00.854 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:17:00.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:00.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:17:00.891 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:00 smithi026 ceph-mon[68309]: osdmap e582: 12 total, 12 up, 10 in 2024-03-20T09:17:00.891 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:00 smithi026 ceph-mon[68309]: pgmap v1394: 47 pgs: 1 unknown, 46 active+clean; 2.4 GiB data, 11 GiB used, 883 GiB / 894 GiB avail 2024-03-20T09:17:00.892 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:00 smithi026 ceph-mon[68309]: 4.8 scrub starts 2024-03-20T09:17:00.892 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:00 smithi026 ceph-mon[68309]: 4.4 scrub ok 2024-03-20T09:17:00.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:00 smithi122 ceph-mon[73495]: osdmap e582: 12 total, 12 up, 10 in 2024-03-20T09:17:00.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:00 smithi122 ceph-mon[73495]: pgmap v1394: 47 pgs: 1 unknown, 46 active+clean; 2.4 GiB data, 11 GiB used, 883 GiB / 894 GiB avail 2024-03-20T09:17:00.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:00 smithi122 ceph-mon[73495]: 4.8 scrub starts 2024-03-20T09:17:00.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:00 smithi122 ceph-mon[73495]: 4.4 scrub ok 2024-03-20T09:17:00.966 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:17:00.998 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:01.033 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:17:01.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:01.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:17:01.137 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:17:01.186 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:17:01.227 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:01.479 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:17:01.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:01.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:17:01.593 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:17:01.623 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:01.646 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:17:01.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:01.649 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:17:01.723 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:17:01.746 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:01 smithi026 ceph-mon[68309]: 4.8 scrub ok 2024-03-20T09:17:01.746 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:01 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3591377618' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:17:01.746 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:01 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:17:01.750 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:17:01.779 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:01.801 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:17:01.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:01.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:17:01.906 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:17:01.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:01 smithi122 ceph-mon[73495]: 4.8 scrub ok 2024-03-20T09:17:01.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:01 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3591377618' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:17:01.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:01 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:17:01.940 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:01.966 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:17:01.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:01.970 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:17:02.071 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:17:02.103 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:02.126 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:17:02.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:02.132 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:17:02.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:01 smithi115 ceph-mon[72476]: 4.8 scrub ok 2024-03-20T09:17:02.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:01 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3591377618' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:17:02.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:01 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:17:02.233 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:17:02.280 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:02.308 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:17:02.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:02.313 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:17:02.414 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:17:02.454 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:02.482 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:17:02.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:02.486 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:17:02.587 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:17:02.622 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:02.627 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:17:02.643 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:17:02.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:02.646 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:17:02.747 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:17:02.777 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:02.805 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:17:02.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:02.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:17:02.890 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 8, 10, 11, 3] out_osds: [9, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 5] 2024-03-20T09:17:02.890 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 0 1 2024-03-20T09:17:02.909 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:17:02.939 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:02.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:02 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:17:02.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:02 smithi122 ceph-mon[73495]: Cluster is now healthy 2024-03-20T09:17:02.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:02 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:17:02.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:02 smithi122 ceph-mon[73495]: osdmap e583: 12 total, 12 up, 10 in 2024-03-20T09:17:02.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:02 smithi122 ceph-mon[73495]: pgmap v1396: 47 pgs: 1 unknown, 46 active+clean; 2.3 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 525 op/s 2024-03-20T09:17:02.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:02 smithi122 ceph-mon[73495]: 4.d scrub starts 2024-03-20T09:17:02.965 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:17:02.969 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:02 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:17:02.969 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:02 smithi115 ceph-mon[72476]: Cluster is now healthy 2024-03-20T09:17:02.969 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:02 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:17:02.969 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:02 smithi115 ceph-mon[72476]: osdmap e583: 12 total, 12 up, 10 in 2024-03-20T09:17:02.969 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:02 smithi115 ceph-mon[72476]: pgmap v1396: 47 pgs: 1 unknown, 46 active+clean; 2.3 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 525 op/s 2024-03-20T09:17:02.969 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:02 smithi115 ceph-mon[72476]: 4.d scrub starts 2024-03-20T09:17:02.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:02.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:17:02.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:02 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:17:02.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:02 smithi026 ceph-mon[68309]: Cluster is now healthy 2024-03-20T09:17:02.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:02 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:17:02.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:02 smithi026 ceph-mon[68309]: osdmap e583: 12 total, 12 up, 10 in 2024-03-20T09:17:02.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:02 smithi026 ceph-mon[68309]: pgmap v1396: 47 pgs: 1 unknown, 46 active+clean; 2.3 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 525 op/s 2024-03-20T09:17:02.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:02 smithi026 ceph-mon[68309]: 4.d scrub starts 2024-03-20T09:17:03.071 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:17:03.144 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:03.438 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:17:03.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:03.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:17:03.545 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:17:03.566 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:17:03.604 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:17:03.649 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:03.764 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:17:03.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:03.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:17:03.872 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:17:03.899 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:03.919 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:17:03.922 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:03.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:17:04.022 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:03 smithi026 ceph-mon[68309]: 1.7 scrub starts 2024-03-20T09:17:04.022 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:03 smithi026 ceph-mon[68309]: 1.7 scrub ok 2024-03-20T09:17:04.022 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:03 smithi026 ceph-mon[68309]: 4.9 scrub starts 2024-03-20T09:17:04.024 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:17:04.078 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:04.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:03 smithi115 ceph-mon[72476]: 1.7 scrub starts 2024-03-20T09:17:04.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:03 smithi115 ceph-mon[72476]: 1.7 scrub ok 2024-03-20T09:17:04.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:03 smithi115 ceph-mon[72476]: 4.9 scrub starts 2024-03-20T09:17:04.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:03 smithi122 ceph-mon[73495]: 1.7 scrub starts 2024-03-20T09:17:04.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:03 smithi122 ceph-mon[73495]: 1.7 scrub ok 2024-03-20T09:17:04.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:03 smithi122 ceph-mon[73495]: 4.9 scrub starts 2024-03-20T09:17:04.246 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:17:04.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:04.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:17:04.350 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:17:04.378 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:04.401 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:17:04.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:04.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:17:04.466 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:17:04.466 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:17:04.508 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:17:04.577 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:04.762 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:04 smithi026 ceph-mon[68309]: 4.f deep-scrub starts 2024-03-20T09:17:04.762 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:04 smithi026 ceph-mon[68309]: 4.d scrub ok 2024-03-20T09:17:04.762 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:04 smithi026 ceph-mon[68309]: 4.9 scrub ok 2024-03-20T09:17:04.762 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:04 smithi026 ceph-mon[68309]: pgmap v1397: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.4 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 78 MiB/s wr, 2.43k op/s; 4350/117846 objects misplaced (3.691%); 2.7 MiB/s, 43 objects/s recovering 2024-03-20T09:17:04.862 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:17:05.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:04 smithi115 ceph-mon[72476]: 4.f deep-scrub starts 2024-03-20T09:17:05.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:04 smithi115 ceph-mon[72476]: 4.d scrub ok 2024-03-20T09:17:05.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:04 smithi115 ceph-mon[72476]: 4.9 scrub ok 2024-03-20T09:17:05.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:04 smithi115 ceph-mon[72476]: pgmap v1397: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.4 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 78 MiB/s wr, 2.43k op/s; 4350/117846 objects misplaced (3.691%); 2.7 MiB/s, 43 objects/s recovering 2024-03-20T09:17:05.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:04 smithi122 ceph-mon[73495]: 4.f deep-scrub starts 2024-03-20T09:17:05.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:04 smithi122 ceph-mon[73495]: 4.d scrub ok 2024-03-20T09:17:05.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:04 smithi122 ceph-mon[73495]: 4.9 scrub ok 2024-03-20T09:17:05.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:04 smithi122 ceph-mon[73495]: pgmap v1397: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 2.4 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 78 MiB/s wr, 2.43k op/s; 4350/117846 objects misplaced (3.691%); 2.7 MiB/s, 43 objects/s recovering 2024-03-20T09:17:05.254 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:17:05.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:05.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:17:05.359 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:17:05.402 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:05.424 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:17:05.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:05.429 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:17:05.530 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:17:05.598 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:05.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:05 smithi026 ceph-mon[68309]: 1.4 scrub starts 2024-03-20T09:17:05.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:05 smithi026 ceph-mon[68309]: 1.4 scrub ok 2024-03-20T09:17:05.982 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:17:05.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:05.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:17:06.089 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:17:06.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:05 smithi115 ceph-mon[72476]: 1.4 scrub starts 2024-03-20T09:17:06.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:05 smithi115 ceph-mon[72476]: 1.4 scrub ok 2024-03-20T09:17:06.175 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:06.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:05 smithi122 ceph-mon[73495]: 1.4 scrub starts 2024-03-20T09:17:06.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:05 smithi122 ceph-mon[73495]: 1.4 scrub ok 2024-03-20T09:17:07.074 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:17:07.075 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:06 smithi026 ceph-mon[68309]: pgmap v1398: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfilling, 45 active+clean; 2.3 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 60 MiB/s wr, 2.12k op/s; 4350/112068 objects misplaced (3.882%); 2.1 MiB/s, 33 objects/s recovering 2024-03-20T09:17:07.083 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:07.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:17:07.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:06 smithi115 ceph-mon[72476]: pgmap v1398: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfilling, 45 active+clean; 2.3 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 60 MiB/s wr, 2.12k op/s; 4350/112068 objects misplaced (3.882%); 2.1 MiB/s, 33 objects/s recovering 2024-03-20T09:17:07.184 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:17:07.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:06 smithi122 ceph-mon[73495]: pgmap v1398: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfilling, 45 active+clean; 2.3 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 60 MiB/s wr, 2.12k op/s; 4350/112068 objects misplaced (3.882%); 2.1 MiB/s, 33 objects/s recovering 2024-03-20T09:17:07.213 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:07.236 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:17:07.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:07.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:17:07.343 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:17:07.425 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:07.813 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:17:07.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:07.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:17:07.924 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:17:07.957 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:07.987 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:17:07.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:07.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:17:08.092 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:17:08.135 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:07 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:17:08.135 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:07 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:17:08.135 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:07 smithi122 ceph-mon[73495]: osdmap e584: 12 total, 12 up, 10 in 2024-03-20T09:17:08.135 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:07 smithi122 ceph-mon[73495]: 4.1 scrub starts 2024-03-20T09:17:08.136 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:08.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:07 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:17:08.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:07 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:17:08.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:07 smithi115 ceph-mon[72476]: osdmap e584: 12 total, 12 up, 10 in 2024-03-20T09:17:08.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:07 smithi115 ceph-mon[72476]: 4.1 scrub starts 2024-03-20T09:17:08.161 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:17:08.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:08.164 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:17:08.224 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:07 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:17:08.224 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:07 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:17:08.224 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:07 smithi026 ceph-mon[68309]: osdmap e584: 12 total, 12 up, 10 in 2024-03-20T09:17:08.224 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:07 smithi026 ceph-mon[68309]: 4.1 scrub starts 2024-03-20T09:17:08.265 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:17:08.291 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:08.317 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:17:08.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:08.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:17:08.425 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:17:08.493 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:08.790 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:08 smithi026 ceph-mon[68309]: 1.9 scrub starts 2024-03-20T09:17:08.790 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:08 smithi026 ceph-mon[68309]: 1.9 scrub ok 2024-03-20T09:17:08.790 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:08 smithi026 ceph-mon[68309]: 4.1 scrub ok 2024-03-20T09:17:08.790 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:08 smithi026 ceph-mon[68309]: pgmap v1400: 47 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing+deep, 45 active+clean; 2.2 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 58 MiB/s wr, 2.23k op/s; 2876/107697 objects misplaced (2.670%); 6.6 MiB/s, 104 objects/s recovering 2024-03-20T09:17:09.161 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:08 smithi115 ceph-mon[72476]: 1.9 scrub starts 2024-03-20T09:17:09.161 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:08 smithi115 ceph-mon[72476]: 1.9 scrub ok 2024-03-20T09:17:09.161 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:08 smithi115 ceph-mon[72476]: 4.1 scrub ok 2024-03-20T09:17:09.161 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:08 smithi115 ceph-mon[72476]: pgmap v1400: 47 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing+deep, 45 active+clean; 2.2 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 58 MiB/s wr, 2.23k op/s; 2876/107697 objects misplaced (2.670%); 6.6 MiB/s, 104 objects/s recovering 2024-03-20T09:17:09.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:08 smithi122 ceph-mon[73495]: 1.9 scrub starts 2024-03-20T09:17:09.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:08 smithi122 ceph-mon[73495]: 1.9 scrub ok 2024-03-20T09:17:09.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:08 smithi122 ceph-mon[73495]: 4.1 scrub ok 2024-03-20T09:17:09.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:08 smithi122 ceph-mon[73495]: pgmap v1400: 47 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing+deep, 45 active+clean; 2.2 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 58 MiB/s wr, 2.23k op/s; 2876/107697 objects misplaced (2.670%); 6.6 MiB/s, 104 objects/s recovering 2024-03-20T09:17:10.784 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:17:10.785 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:17:10.786 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-20T09:17:10.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:10.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:17:10.894 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:17:10.978 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:11.039 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:17:11.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:11.047 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:10 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:17:11.047 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:10 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:17:11.047 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:10 smithi026 ceph-mon[68309]: osdmap e585: 12 total, 12 up, 10 in 2024-03-20T09:17:11.047 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:10 smithi026 ceph-mon[68309]: pgmap v1402: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfilling, 45 active+clean; 2.2 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 58 MiB/s wr, 2.11k op/s; 2876/107070 objects misplaced (2.686%); 6.5 MiB/s, 104 objects/s recovering 2024-03-20T09:17:11.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:17:11.149 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:17:11.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:10 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:17:11.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:10 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:17:11.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:10 smithi115 ceph-mon[72476]: osdmap e585: 12 total, 12 up, 10 in 2024-03-20T09:17:11.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:10 smithi115 ceph-mon[72476]: pgmap v1402: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfilling, 45 active+clean; 2.2 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 58 MiB/s wr, 2.11k op/s; 2876/107070 objects misplaced (2.686%); 6.5 MiB/s, 104 objects/s recovering 2024-03-20T09:17:11.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:10 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:17:11.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:10 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:17:11.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:10 smithi122 ceph-mon[73495]: osdmap e585: 12 total, 12 up, 10 in 2024-03-20T09:17:11.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:10 smithi122 ceph-mon[73495]: pgmap v1402: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfilling, 45 active+clean; 2.2 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 58 MiB/s wr, 2.11k op/s; 2876/107070 objects misplaced (2.686%); 6.5 MiB/s, 104 objects/s recovering 2024-03-20T09:17:11.229 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:11.319 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:17:11.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:11.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:17:11.422 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:17:11.466 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:11.490 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:17:11.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:11.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:17:11.594 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:17:11.692 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:11.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:11 smithi026 ceph-mon[68309]: 1.a scrub starts 2024-03-20T09:17:11.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:11 smithi026 ceph-mon[68309]: 1.a scrub ok 2024-03-20T09:17:11.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:11 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:17:11.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:11 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:17:11.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:11 smithi026 ceph-mon[68309]: osdmap e586: 12 total, 12 up, 10 in 2024-03-20T09:17:11.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:11 smithi026 ceph-mon[68309]: 4.7 scrub starts 2024-03-20T09:17:11.975 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:17:12.054 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:17:12.097 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:17:12.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:11 smithi115 ceph-mon[72476]: 1.a scrub starts 2024-03-20T09:17:12.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:11 smithi115 ceph-mon[72476]: 1.a scrub ok 2024-03-20T09:17:12.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:11 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:17:12.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:11 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:17:12.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:11 smithi115 ceph-mon[72476]: osdmap e586: 12 total, 12 up, 10 in 2024-03-20T09:17:12.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:11 smithi115 ceph-mon[72476]: 4.7 scrub starts 2024-03-20T09:17:12.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:11 smithi122 ceph-mon[73495]: 1.a scrub starts 2024-03-20T09:17:12.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:11 smithi122 ceph-mon[73495]: 1.a scrub ok 2024-03-20T09:17:12.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:11 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:17:12.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:11 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:17:12.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:11 smithi122 ceph-mon[73495]: osdmap e586: 12 total, 12 up, 10 in 2024-03-20T09:17:12.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:11 smithi122 ceph-mon[73495]: 4.7 scrub starts 2024-03-20T09:17:12.339 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:17:12.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:12.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:17:12.364 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:17:12.397 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:17:12.448 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:17:12.455 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:17:12.480 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:12.500 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:17:12.510 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:17:12.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:12.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:17:12.560 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:17:12.613 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:17:12.616 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:17:12.664 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:17:12.711 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:17:12.730 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:12.754 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:17:12.770 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:17:12.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:12.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:17:12.791 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:17:12.798 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:17:12.798 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:17:12.798 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:17:12.798 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:17:12.799 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:17:12.799 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:17:12.799 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:17:12.799 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:17:12.799 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:17:12.799 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:17:12.799 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:17:12.799 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:17:12.881 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:17:12.913 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:12.920 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-20T09:17:12.919+0000 7f7ec37fe700 0 --2- 172.21.15.26:0/1547557664 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7ec41063a0 0x7f7ec410df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-03-20T09:17:12.941 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:17:12.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:12.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:17:12.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:12 smithi026 ceph-mon[68309]: 4.a scrub starts 2024-03-20T09:17:12.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:12 smithi026 ceph-mon[68309]: 4.a scrub ok 2024-03-20T09:17:12.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:12 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3985674771' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:17:12.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:12 smithi026 ceph-mon[68309]: pgmap v1404: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfilling, 45 active+clean; 2.1 GiB data, 9.9 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 585 op/s; 2876/101514 objects misplaced (2.833%); 6.0 MiB/s, 95 objects/s recovering 2024-03-20T09:17:12.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:12 smithi026 ceph-mon[68309]: 4.7 scrub ok 2024-03-20T09:17:13.056 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:17:13.095 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:13.119 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:17:13.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:13.123 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:17:13.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:12 smithi115 ceph-mon[72476]: 4.a scrub starts 2024-03-20T09:17:13.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:12 smithi115 ceph-mon[72476]: 4.a scrub ok 2024-03-20T09:17:13.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:12 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3985674771' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:17:13.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:12 smithi115 ceph-mon[72476]: pgmap v1404: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfilling, 45 active+clean; 2.1 GiB data, 9.9 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 585 op/s; 2876/101514 objects misplaced (2.833%); 6.0 MiB/s, 95 objects/s recovering 2024-03-20T09:17:13.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:12 smithi115 ceph-mon[72476]: 4.7 scrub ok 2024-03-20T09:17:13.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:12 smithi122 ceph-mon[73495]: 4.a scrub starts 2024-03-20T09:17:13.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:12 smithi122 ceph-mon[73495]: 4.a scrub ok 2024-03-20T09:17:13.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:12 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3985674771' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:17:13.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:12 smithi122 ceph-mon[73495]: pgmap v1404: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfilling, 45 active+clean; 2.1 GiB data, 9.9 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 585 op/s; 2876/101514 objects misplaced (2.833%); 6.0 MiB/s, 95 objects/s recovering 2024-03-20T09:17:13.198 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:12 smithi122 ceph-mon[73495]: 4.7 scrub ok 2024-03-20T09:17:13.224 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:17:13.268 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:13.288 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:17:13.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:13.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:17:13.392 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:17:13.419 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:13.440 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:17:13.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:13.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:17:13.545 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:17:13.586 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:13.633 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:17:13.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:13.638 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:17:13.739 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:17:13.841 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:13.905 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:13 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:17:13.906 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:13 smithi026 ceph-mon[68309]: 1.6 scrub starts 2024-03-20T09:17:13.906 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:13 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3840733790' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-20T09:17:13.937 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.0 to 1 (10000) 2024-03-20T09:17:13.937 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:17:13.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:13.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:17:14.060 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:17:14.097 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:14.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:13 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:17:14.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:13 smithi115 ceph-mon[72476]: 1.6 scrub starts 2024-03-20T09:17:14.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:13 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3840733790' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-20T09:17:14.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:13 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:17:14.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:13 smithi122 ceph-mon[73495]: 1.6 scrub starts 2024-03-20T09:17:14.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:13 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3840733790' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-20T09:17:14.650 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:17:14.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:14.659 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:17:14.759 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:17:14.803 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:14.840 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:17:14.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:14.848 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:17:14.949 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:17:14.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:14 smithi026 ceph-mon[68309]: 1.6 scrub ok 2024-03-20T09:17:14.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:14 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3840733790' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-20T09:17:14.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:14 smithi026 ceph-mon[68309]: osdmap e587: 12 total, 12 up, 10 in 2024-03-20T09:17:14.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:14 smithi026 ceph-mon[68309]: pgmap v1406: 47 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing+deep, 45 active+clean; 2.0 GiB data, 9.7 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 550 op/s; 1284/97779 objects misplaced (1.313%); 7.2 MiB/s, 115 objects/s recovering 2024-03-20T09:17:15.004 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:15.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:14 smithi122 ceph-mon[73495]: 1.6 scrub ok 2024-03-20T09:17:15.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:14 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3840733790' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-20T09:17:15.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:14 smithi122 ceph-mon[73495]: osdmap e587: 12 total, 12 up, 10 in 2024-03-20T09:17:15.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:14 smithi122 ceph-mon[73495]: pgmap v1406: 47 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing+deep, 45 active+clean; 2.0 GiB data, 9.7 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 550 op/s; 1284/97779 objects misplaced (1.313%); 7.2 MiB/s, 115 objects/s recovering 2024-03-20T09:17:15.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:14 smithi115 ceph-mon[72476]: 1.6 scrub ok 2024-03-20T09:17:15.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:14 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3840733790' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-20T09:17:15.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:14 smithi115 ceph-mon[72476]: osdmap e587: 12 total, 12 up, 10 in 2024-03-20T09:17:15.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:14 smithi115 ceph-mon[72476]: pgmap v1406: 47 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing+deep, 45 active+clean; 2.0 GiB data, 9.7 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 550 op/s; 1284/97779 objects misplaced (1.313%); 7.2 MiB/s, 115 objects/s recovering 2024-03-20T09:17:15.818 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 1 1 2024-03-20T09:17:16.120 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:15 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:17:16.120 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:15 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:17:16.120 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:15 smithi026 ceph-mon[68309]: 1.3 scrub starts 2024-03-20T09:17:16.120 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:15 smithi026 ceph-mon[68309]: 1.3 scrub ok 2024-03-20T09:17:16.120 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:15 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:17:16.120 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:15 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:17:16.120 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:15 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:17:16.120 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:15 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2815619849' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:17:16.120 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:15 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:17:16.122 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:17:16.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:16.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:17:16.267 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:17:16.295 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:15 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:17:16.295 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:15 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:17:16.296 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:15 smithi122 ceph-mon[73495]: 1.3 scrub starts 2024-03-20T09:17:16.296 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:15 smithi122 ceph-mon[73495]: 1.3 scrub ok 2024-03-20T09:17:16.296 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:15 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:17:16.296 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:15 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:17:16.296 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:15 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:17:16.296 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:15 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2815619849' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:17:16.296 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:15 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:17:16.297 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:16.323 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:17:16.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:16.326 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:17:16.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:15 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:17:16.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:15 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:17:16.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:15 smithi115 ceph-mon[72476]: 1.3 scrub starts 2024-03-20T09:17:16.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:15 smithi115 ceph-mon[72476]: 1.3 scrub ok 2024-03-20T09:17:16.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:15 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:17:16.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:15 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:17:16.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:15 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:17:16.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:15 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2815619849' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:17:16.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:15 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:17:16.427 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:17:16.455 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:16.478 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:17:16.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:16.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:17:16.570 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:17:16.582 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:17:16.664 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:16.726 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:17:17.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:16 smithi026 ceph-mon[68309]: 1.f scrub starts 2024-03-20T09:17:17.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:16 smithi026 ceph-mon[68309]: 1.f scrub ok 2024-03-20T09:17:17.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:16 smithi026 ceph-mon[68309]: pgmap v1407: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfilling, 45 active+clean; 1.9 GiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 770 op/s; 1284/92781 objects misplaced (1.384%); 7.0 MiB/s, 111 objects/s recovering 2024-03-20T09:17:17.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:16 smithi026 ceph-mon[68309]: 1.b scrub starts 2024-03-20T09:17:17.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:16 smithi026 ceph-mon[68309]: 1.b scrub ok 2024-03-20T09:17:17.224 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:16 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:17:17.224 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:16 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:17:17.224 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:16 smithi026 ceph-mon[68309]: osdmap e588: 12 total, 12 up, 10 in 2024-03-20T09:17:17.238 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:17:17.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:17.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:17:17.345 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:17:17.371 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:16 smithi115 ceph-mon[72476]: 1.f scrub starts 2024-03-20T09:17:17.371 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:16 smithi115 ceph-mon[72476]: 1.f scrub ok 2024-03-20T09:17:17.371 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:16 smithi115 ceph-mon[72476]: pgmap v1407: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfilling, 45 active+clean; 1.9 GiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 770 op/s; 1284/92781 objects misplaced (1.384%); 7.0 MiB/s, 111 objects/s recovering 2024-03-20T09:17:17.371 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:16 smithi115 ceph-mon[72476]: 1.b scrub starts 2024-03-20T09:17:17.372 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:16 smithi115 ceph-mon[72476]: 1.b scrub ok 2024-03-20T09:17:17.372 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:16 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:17:17.372 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:16 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:17:17.372 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:16 smithi115 ceph-mon[72476]: osdmap e588: 12 total, 12 up, 10 in 2024-03-20T09:17:17.374 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:17.398 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:17:17.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:17.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:17:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:16 smithi122 ceph-mon[73495]: 1.f scrub starts 2024-03-20T09:17:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:16 smithi122 ceph-mon[73495]: 1.f scrub ok 2024-03-20T09:17:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:16 smithi122 ceph-mon[73495]: pgmap v1407: 47 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfilling, 45 active+clean; 1.9 GiB data, 9.4 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 770 op/s; 1284/92781 objects misplaced (1.384%); 7.0 MiB/s, 111 objects/s recovering 2024-03-20T09:17:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:16 smithi122 ceph-mon[73495]: 1.b scrub starts 2024-03-20T09:17:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:16 smithi122 ceph-mon[73495]: 1.b scrub ok 2024-03-20T09:17:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:16 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:17:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:16 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:17:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:16 smithi122 ceph-mon[73495]: osdmap e588: 12 total, 12 up, 10 in 2024-03-20T09:17:17.502 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:17:17.529 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:17.547 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:17:17.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:17.550 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:17:17.651 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:17:17.679 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:17.699 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:17:17.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:17.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:17:17.803 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:17:17.829 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:17.848 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:17:17.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:17.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:17:17.952 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:17:17.982 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:18.009 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:17:18.009 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-03-20T09:17:18.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-03-20T09:17:18.009 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:17:18.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.009 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:143564 3.7 3:e7091616:::benchmark_data_smithi122_119671_object43782:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e449)", 2024-03-20T09:17:18.009 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:10:41.154979+0000", 2024-03-20T09:17:18.010 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 396.45165322600002, 2024-03-20T09:17:18.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.009065637, 2024-03-20T09:17:18.010 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:17:18.010 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:18.010 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:17:18.010 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:17:18.010 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:17:18.010 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 143564 2024-03-20T09:17:18.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.010 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:17:18.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:17:18.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.154979+0000", 2024-03-20T09:17:18.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:17:18.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.154979+0000", 2024-03-20T09:17:18.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:17:18.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.154981+0000", 2024-03-20T09:17:18.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.911e-06 2024-03-20T09:17:18.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:17:18.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.154990+0000", 2024-03-20T09:17:18.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.217e-06 2024-03-20T09:17:18.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:17:18.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.155013+0000", 2024-03-20T09:17:18.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2602000000000001e-05 2024-03-20T09:17:18.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:17:18.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.155025+0000", 2024-03-20T09:17:18.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.223e-05 2024-03-20T09:17:18.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.155108+0000", 2024-03-20T09:17:18.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3083000000000004e-05 2024-03-20T09:17:18.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:17:18.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.155123+0000", 2024-03-20T09:17:18.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4681000000000001e-05 2024-03-20T09:17:18.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.156747+0000", 2024-03-20T09:17:18.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0016243069999999 2024-03-20T09:17:18.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:17:18.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.156916+0000", 2024-03-20T09:17:18.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00016824299999999999 2024-03-20T09:17:18.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:17:18.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.157043+0000", 2024-03-20T09:17:18.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000127329 2024-03-20T09:17:18.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.161049+0000", 2024-03-20T09:17:18.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040065709999999996 2024-03-20T09:17:18.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:17:18.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.161057+0000", 2024-03-20T09:17:18.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8679999999999999e-06 2024-03-20T09:17:18.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.163980+0000", 2024-03-20T09:17:18.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0029223510000000001 2024-03-20T09:17:18.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:17:18.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.164015+0000", 2024-03-20T09:17:18.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5037999999999999e-05 2024-03-20T09:17:18.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:17:18.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.164045+0000", 2024-03-20T09:17:18.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0205999999999998e-05 2024-03-20T09:17:18.016 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.016 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:17:18.016 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.016 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:143572 3.7 3:e3d5668d:::benchmark_data_smithi122_119671_object43790:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e449)", 2024-03-20T09:17:18.016 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:10:41.197173+0000", 2024-03-20T09:17:18.017 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 396.40945965600002, 2024-03-20T09:17:18.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.97170473999999996, 2024-03-20T09:17:18.017 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:17:18.017 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:18.017 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:17:18.017 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:17:18.017 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:17:18.017 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 143572 2024-03-20T09:17:18.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.017 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:17:18.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:17:18.017 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.197173+0000", 2024-03-20T09:17:18.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:17:18.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.197173+0000", 2024-03-20T09:17:18.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:17:18.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.197175+0000", 2024-03-20T09:17:18.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.305e-06 2024-03-20T09:17:18.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:17:18.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.197185+0000", 2024-03-20T09:17:18.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9289999999999993e-06 2024-03-20T09:17:18.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:17:18.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.197208+0000", 2024-03-20T09:17:18.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2748e-05 2024-03-20T09:17:18.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:17:18.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.197220+0000", 2024-03-20T09:17:18.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1907e-05 2024-03-20T09:17:18.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.197288+0000", 2024-03-20T09:17:18.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8713000000000004e-05 2024-03-20T09:17:18.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:17:18.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.197297+0000", 2024-03-20T09:17:18.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9970000000000004e-06 2024-03-20T09:17:18.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.159349+0000", 2024-03-20T09:17:18.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.96205180099999998 2024-03-20T09:17:18.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:17:18.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.159486+0000", 2024-03-20T09:17:18.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013625700000000001 2024-03-20T09:17:18.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:17:18.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.159611+0000", 2024-03-20T09:17:18.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012543 2024-03-20T09:17:18.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.164084+0000", 2024-03-20T09:17:18.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004472696 2024-03-20T09:17:18.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:17:18.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.165538+0000", 2024-03-20T09:17:18.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014541210000000001 2024-03-20T09:17:18.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.168806+0000", 2024-03-20T09:17:18.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0032684020000000001 2024-03-20T09:17:18.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:17:18.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.168846+0000", 2024-03-20T09:17:18.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9968000000000001e-05 2024-03-20T09:17:18.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:17:18.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.168878+0000", 2024-03-20T09:17:18.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1466e-05 2024-03-20T09:17:18.023 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.023 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:17:18.023 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.023 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:143592 3.7 3:e6af9103:::benchmark_data_smithi122_119671_object43810:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e449)", 2024-03-20T09:17:18.023 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:10:41.340924+0000", 2024-03-20T09:17:18.023 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 396.26570875700003, 2024-03-20T09:17:18.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.82804276799999998, 2024-03-20T09:17:18.023 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:17:18.023 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:18.024 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:17:18.024 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:17:18.024 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:17:18.024 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 143592 2024-03-20T09:17:18.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.024 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:17:18.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:17:18.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.340924+0000", 2024-03-20T09:17:18.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:17:18.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.340924+0000", 2024-03-20T09:17:18.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:17:18.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.340925+0000", 2024-03-20T09:17:18.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.482e-06 2024-03-20T09:17:18.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:17:18.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.340938+0000", 2024-03-20T09:17:18.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2612999999999999e-05 2024-03-20T09:17:18.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:17:18.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.340954+0000", 2024-03-20T09:17:18.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6184999999999998e-05 2024-03-20T09:17:18.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:17:18.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.340962+0000", 2024-03-20T09:17:18.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8650000000000001e-06 2024-03-20T09:17:18.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.341020+0000", 2024-03-20T09:17:18.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7757999999999998e-05 2024-03-20T09:17:18.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:17:18.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.341028+0000", 2024-03-20T09:17:18.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5869999999999993e-06 2024-03-20T09:17:18.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.161133+0000", 2024-03-20T09:17:18.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.82010472199999995 2024-03-20T09:17:18.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:17:18.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.161206+0000", 2024-03-20T09:17:18.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3403000000000002e-05 2024-03-20T09:17:18.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:17:18.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.161299+0000", 2024-03-20T09:17:18.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2936999999999995e-05 2024-03-20T09:17:18.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.164125+0000", 2024-03-20T09:17:18.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028253280000000002 2024-03-20T09:17:18.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:17:18.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.165549+0000", 2024-03-20T09:17:18.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001423913 2024-03-20T09:17:18.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.168917+0000", 2024-03-20T09:17:18.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003368692 2024-03-20T09:17:18.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:17:18.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.168944+0000", 2024-03-20T09:17:18.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7228000000000001e-05 2024-03-20T09:17:18.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:17:18.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.168967+0000", 2024-03-20T09:17:18.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2055000000000001e-05 2024-03-20T09:17:18.030 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.030 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:17:18.030 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.030 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:143603 3.7 3:e63a0e26:::benchmark_data_smithi122_119671_object43821:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e449)", 2024-03-20T09:17:18.030 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:10:41.451662+0000", 2024-03-20T09:17:18.030 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 396.15497081000001, 2024-03-20T09:17:18.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.71737729100000003, 2024-03-20T09:17:18.030 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:17:18.030 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:18.030 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:17:18.031 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:17:18.031 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:17:18.031 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 143603 2024-03-20T09:17:18.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.031 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:17:18.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:17:18.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.451662+0000", 2024-03-20T09:17:18.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:17:18.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.451662+0000", 2024-03-20T09:17:18.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:17:18.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.451664+0000", 2024-03-20T09:17:18.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.933e-06 2024-03-20T09:17:18.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:17:18.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.451672+0000", 2024-03-20T09:17:18.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8270000000000004e-06 2024-03-20T09:17:18.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.032 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:17:18.032 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.451693+0000", 2024-03-20T09:17:18.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0458e-05 2024-03-20T09:17:18.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:17:18.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.451705+0000", 2024-03-20T09:17:18.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1559999999999999e-05 2024-03-20T09:17:18.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.451772+0000", 2024-03-20T09:17:18.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7799999999999995e-05 2024-03-20T09:17:18.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:17:18.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.451782+0000", 2024-03-20T09:17:18.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8609999999999993e-06 2024-03-20T09:17:18.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.162652+0000", 2024-03-20T09:17:18.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.71086976499999999 2024-03-20T09:17:18.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:17:18.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.162722+0000", 2024-03-20T09:17:18.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9754999999999999e-05 2024-03-20T09:17:18.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:17:18.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.162785+0000", 2024-03-20T09:17:18.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3094000000000003e-05 2024-03-20T09:17:18.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.165646+0000", 2024-03-20T09:17:18.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028609920000000001 2024-03-20T09:17:18.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:17:18.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.167291+0000", 2024-03-20T09:17:18.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001644838 2024-03-20T09:17:18.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.168994+0000", 2024-03-20T09:17:18.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0017038559999999999 2024-03-20T09:17:18.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:17:18.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.169020+0000", 2024-03-20T09:17:18.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5122000000000001e-05 2024-03-20T09:17:18.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:17:18.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.169039+0000", 2024-03-20T09:17:18.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9429999999999999e-05 2024-03-20T09:17:18.036 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.036 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:17:18.037 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.037 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:143621 3.7 3:e5e3109e:::benchmark_data_smithi122_119671_object43839:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e449)", 2024-03-20T09:17:18.037 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:10:41.769244+0000", 2024-03-20T09:17:18.037 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 395.83738867800002, 2024-03-20T09:17:18.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.41200099499999998, 2024-03-20T09:17:18.037 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:17:18.037 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:18.037 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:17:18.037 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:17:18.037 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:17:18.037 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 143621 2024-03-20T09:17:18.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.037 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:17:18.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:17:18.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.769244+0000", 2024-03-20T09:17:18.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:17:18.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.769244+0000", 2024-03-20T09:17:18.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:17:18.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.769246+0000", 2024-03-20T09:17:18.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9640000000000002e-06 2024-03-20T09:17:18.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:17:18.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.769254+0000", 2024-03-20T09:17:18.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0649999999999999e-06 2024-03-20T09:17:18.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:17:18.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.769272+0000", 2024-03-20T09:17:18.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8436000000000001e-05 2024-03-20T09:17:18.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:17:18.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.769284+0000", 2024-03-20T09:17:18.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1252e-05 2024-03-20T09:17:18.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.769364+0000", 2024-03-20T09:17:18.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0786999999999997e-05 2024-03-20T09:17:18.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:17:18.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.769381+0000", 2024-03-20T09:17:18.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6588000000000001e-05 2024-03-20T09:17:18.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.164160+0000", 2024-03-20T09:17:18.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.39477866499999997 2024-03-20T09:17:18.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:17:18.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.164250+0000", 2024-03-20T09:17:18.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0861999999999998e-05 2024-03-20T09:17:18.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:17:18.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.164342+0000", 2024-03-20T09:17:18.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1445000000000003e-05 2024-03-20T09:17:18.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.166922+0000", 2024-03-20T09:17:18.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0025798319999999998 2024-03-20T09:17:18.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:17:18.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.167297+0000", 2024-03-20T09:17:18.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00037566299999999998 2024-03-20T09:17:18.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.181135+0000", 2024-03-20T09:17:18.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013837649 2024-03-20T09:17:18.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:17:18.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.181197+0000", 2024-03-20T09:17:18.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1902999999999993e-05 2024-03-20T09:17:18.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:17:18.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.181245+0000", 2024-03-20T09:17:18.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7883999999999998e-05 2024-03-20T09:17:18.043 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.043 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:17:18.043 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.043 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:163495 3.b 3:d6e891b8:::benchmark_data_smithi122_119671_object63713:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-03-20T09:17:18.043 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:12:06.244771+0000", 2024-03-20T09:17:18.044 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 311.36186118500001, 2024-03-20T09:17:18.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.82372252499999998, 2024-03-20T09:17:18.044 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:17:18.044 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:18.044 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:17:18.044 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:17:18.044 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:17:18.044 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 163495 2024-03-20T09:17:18.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.044 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:17:18.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:17:18.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.244771+0000", 2024-03-20T09:17:18.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:17:18.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.244771+0000", 2024-03-20T09:17:18.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:17:18.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.244773+0000", 2024-03-20T09:17:18.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.314e-06 2024-03-20T09:17:18.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:17:18.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.244778+0000", 2024-03-20T09:17:18.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.967e-06 2024-03-20T09:17:18.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:17:18.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.244788+0000", 2024-03-20T09:17:18.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0421000000000001e-05 2024-03-20T09:17:18.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:17:18.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.244811+0000", 2024-03-20T09:17:18.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2806000000000002e-05 2024-03-20T09:17:18.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.244857+0000", 2024-03-20T09:17:18.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5753000000000001e-05 2024-03-20T09:17:18.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:17:18.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.244862+0000", 2024-03-20T09:17:18.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7320000000000003e-06 2024-03-20T09:17:18.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.060607+0000", 2024-03-20T09:17:18.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.81574488999999994 2024-03-20T09:17:18.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:17:18.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.060700+0000", 2024-03-20T09:17:18.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2989e-05 2024-03-20T09:17:18.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:17:18.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.060781+0000", 2024-03-20T09:17:18.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0813e-05 2024-03-20T09:17:18.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.048 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.068238+0000", 2024-03-20T09:17:18.048 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074570679999999999 2024-03-20T09:17:18.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.048 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:17:18.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.068250+0000", 2024-03-20T09:17:18.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1725999999999999e-05 2024-03-20T09:17:18.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.068372+0000", 2024-03-20T09:17:18.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000122027 2024-03-20T09:17:18.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.049 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:17:18.049 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.068440+0000", 2024-03-20T09:17:18.049 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7723000000000002e-05 2024-03-20T09:17:18.049 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.049 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.050 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:17:18.050 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.068494+0000", 2024-03-20T09:17:18.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4296000000000003e-05 2024-03-20T09:17:18.050 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.050 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:17:18.050 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.050 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.050 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:163499 3.b 3:d6314864:::benchmark_data_smithi122_119671_object63717:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-03-20T09:17:18.050 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:12:06.250105+0000", 2024-03-20T09:17:18.050 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 311.35652739900002, 2024-03-20T09:17:18.050 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.82241188399999998, 2024-03-20T09:17:18.050 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:17:18.051 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:18.051 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:17:18.051 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:17:18.051 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:17:18.051 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 163499 2024-03-20T09:17:18.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.051 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:17:18.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:17:18.051 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.250105+0000", 2024-03-20T09:17:18.051 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.051 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.051 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.051 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:17:18.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.250105+0000", 2024-03-20T09:17:18.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:17:18.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.250106+0000", 2024-03-20T09:17:18.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2559999999999999e-06 2024-03-20T09:17:18.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:17:18.052 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.250112+0000", 2024-03-20T09:17:18.052 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5990000000000001e-06 2024-03-20T09:17:18.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.052 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.052 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:17:18.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.250127+0000", 2024-03-20T09:17:18.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4936000000000001e-05 2024-03-20T09:17:18.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:17:18.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.250136+0000", 2024-03-20T09:17:18.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1740000000000006e-06 2024-03-20T09:17:18.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.250201+0000", 2024-03-20T09:17:18.053 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5128000000000004e-05 2024-03-20T09:17:18.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.053 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.053 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:17:18.053 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.250212+0000", 2024-03-20T09:17:18.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0297e-05 2024-03-20T09:17:18.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.064203+0000", 2024-03-20T09:17:18.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.81399148600000004 2024-03-20T09:17:18.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.054 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:17:18.054 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.064296+0000", 2024-03-20T09:17:18.054 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2652999999999995e-05 2024-03-20T09:17:18.054 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:17:18.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.064434+0000", 2024-03-20T09:17:18.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000138329 2024-03-20T09:17:18.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.068314+0000", 2024-03-20T09:17:18.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0038798249999999999 2024-03-20T09:17:18.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.055 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.055 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.068649+0000", 2024-03-20T09:17:18.055 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00033493599999999998 2024-03-20T09:17:18.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:17:18.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.072408+0000", 2024-03-20T09:17:18.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037591949999999999 2024-03-20T09:17:18.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:17:18.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.072469+0000", 2024-03-20T09:17:18.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1354999999999998e-05 2024-03-20T09:17:18.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.056 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:17:18.056 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.072517+0000", 2024-03-20T09:17:18.056 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7715e-05 2024-03-20T09:17:18.057 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.057 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:17:18.057 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.057 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:163507 3.b 3:dfa7f600:::benchmark_data_smithi122_119671_object63725:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-03-20T09:17:18.057 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:12:06.258712+0000", 2024-03-20T09:17:18.057 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 311.34792069700001, 2024-03-20T09:17:18.057 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.82676451500000003, 2024-03-20T09:17:18.057 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:17:18.057 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:18.057 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:17:18.057 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:17:18.057 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:17:18.057 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 163507 2024-03-20T09:17:18.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.058 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:17:18.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:17:18.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.258712+0000", 2024-03-20T09:17:18.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.058 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:17:18.058 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.258712+0000", 2024-03-20T09:17:18.058 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:17:18.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.258712+0000", 2024-03-20T09:17:18.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8100000000000003e-07 2024-03-20T09:17:18.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:17:18.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.258716+0000", 2024-03-20T09:17:18.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.191e-06 2024-03-20T09:17:18.059 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.059 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.059 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:17:18.059 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.258724+0000", 2024-03-20T09:17:18.059 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5080000000000005e-06 2024-03-20T09:17:18.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:17:18.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.258730+0000", 2024-03-20T09:17:18.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9089999999999997e-06 2024-03-20T09:17:18.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.258783+0000", 2024-03-20T09:17:18.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2624999999999998e-05 2024-03-20T09:17:18.060 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.060 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.060 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:17:18.060 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.258792+0000", 2024-03-20T09:17:18.060 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8269999999999993e-06 2024-03-20T09:17:18.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.068709+0000", 2024-03-20T09:17:18.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.80991684200000003 2024-03-20T09:17:18.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:17:18.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.068844+0000", 2024-03-20T09:17:18.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013473200000000001 2024-03-20T09:17:18.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.061 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:17:18.061 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.068987+0000", 2024-03-20T09:17:18.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000143356 2024-03-20T09:17:18.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.076876+0000", 2024-03-20T09:17:18.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0078885090000000001 2024-03-20T09:17:18.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.085245+0000", 2024-03-20T09:17:18.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083692100000000002 2024-03-20T09:17:18.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:17:18.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.085367+0000", 2024-03-20T09:17:18.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000122364 2024-03-20T09:17:18.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:17:18.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.085429+0000", 2024-03-20T09:17:18.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1951000000000004e-05 2024-03-20T09:17:18.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:17:18.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.085476+0000", 2024-03-20T09:17:18.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6910000000000003e-05 2024-03-20T09:17:18.063 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.063 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:17:18.063 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.064 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:163508 3.b 3:dabf16ab:::benchmark_data_smithi122_119671_object63726:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-03-20T09:17:18.064 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:12:06.260617+0000", 2024-03-20T09:17:18.064 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 311.34601591500001, 2024-03-20T09:17:18.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.824912125, 2024-03-20T09:17:18.064 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:17:18.064 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:18.064 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:17:18.064 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:17:18.064 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:17:18.064 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 163508 2024-03-20T09:17:18.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.064 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:17:18.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:17:18.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.260617+0000", 2024-03-20T09:17:18.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:17:18.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.260617+0000", 2024-03-20T09:17:18.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:17:18.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.260618+0000", 2024-03-20T09:17:18.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.066e-06 2024-03-20T09:17:18.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:17:18.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.260623+0000", 2024-03-20T09:17:18.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1959999999999999e-06 2024-03-20T09:17:18.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:17:18.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.260634+0000", 2024-03-20T09:17:18.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1353e-05 2024-03-20T09:17:18.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:17:18.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.260641+0000", 2024-03-20T09:17:18.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9380000000000003e-06 2024-03-20T09:17:18.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.260687+0000", 2024-03-20T09:17:18.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5590999999999998e-05 2024-03-20T09:17:18.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:17:18.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.260693+0000", 2024-03-20T09:17:18.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0109999999999997e-06 2024-03-20T09:17:18.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.072590+0000", 2024-03-20T09:17:18.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.81189731200000004 2024-03-20T09:17:18.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:17:18.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.072706+0000", 2024-03-20T09:17:18.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011615 2024-03-20T09:17:18.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:17:18.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.072855+0000", 2024-03-20T09:17:18.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000148398 2024-03-20T09:17:18.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.076926+0000", 2024-03-20T09:17:18.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040710420000000004 2024-03-20T09:17:18.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.085349+0000", 2024-03-20T09:17:18.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084229460000000006 2024-03-20T09:17:18.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:17:18.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.085483+0000", 2024-03-20T09:17:18.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013404199999999999 2024-03-20T09:17:18.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:17:18.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.085508+0000", 2024-03-20T09:17:18.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5378e-05 2024-03-20T09:17:18.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:17:18.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.085529+0000", 2024-03-20T09:17:18.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0701999999999999e-05 2024-03-20T09:17:18.070 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.070 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:17:18.070 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.070 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:163533 3.b 3:d6f3d966:::benchmark_data_smithi122_119671_object63751:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-03-20T09:17:18.070 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:12:06.364729+0000", 2024-03-20T09:17:18.070 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 311.241903063, 2024-03-20T09:17:18.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.72093737099999999, 2024-03-20T09:17:18.070 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:17:18.070 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:18.070 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:17:18.070 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:17:18.071 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:17:18.071 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 163533 2024-03-20T09:17:18.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.071 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:17:18.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:17:18.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.364729+0000", 2024-03-20T09:17:18.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:17:18.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.364729+0000", 2024-03-20T09:17:18.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:17:18.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.364731+0000", 2024-03-20T09:17:18.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4699999999999999e-06 2024-03-20T09:17:18.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:17:18.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.364740+0000", 2024-03-20T09:17:18.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8319999999999995e-06 2024-03-20T09:17:18.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:17:18.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.364755+0000", 2024-03-20T09:17:18.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.562e-05 2024-03-20T09:17:18.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:17:18.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.364766+0000", 2024-03-20T09:17:18.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.028e-05 2024-03-20T09:17:18.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.364844+0000", 2024-03-20T09:17:18.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8495e-05 2024-03-20T09:17:18.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:17:18.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.364855+0000", 2024-03-20T09:17:18.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0869999999999999e-05 2024-03-20T09:17:18.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.076960+0000", 2024-03-20T09:17:18.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.71210485999999995 2024-03-20T09:17:18.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:17:18.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.077106+0000", 2024-03-20T09:17:18.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000145916 2024-03-20T09:17:18.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:17:18.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.077250+0000", 2024-03-20T09:17:18.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000143783 2024-03-20T09:17:18.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.085196+0000", 2024-03-20T09:17:18.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0079463390000000002 2024-03-20T09:17:18.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:17:18.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.085535+0000", 2024-03-20T09:17:18.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00033950400000000002 2024-03-20T09:17:18.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.085572+0000", 2024-03-20T09:17:18.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.608e-05 2024-03-20T09:17:18.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:17:18.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.085623+0000", 2024-03-20T09:17:18.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1431999999999998e-05 2024-03-20T09:17:18.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.076 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:17:18.076 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.085667+0000", 2024-03-20T09:17:18.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3890000000000002e-05 2024-03-20T09:17:18.076 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.076 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:17:18.076 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.077 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:163553 3.b 3:d15539c8:::benchmark_data_smithi122_119671_object63771:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-03-20T09:17:18.077 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:12:06.406918+0000", 2024-03-20T09:17:18.077 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 311.19971461300003, 2024-03-20T09:17:18.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.69573252200000002, 2024-03-20T09:17:18.077 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:17:18.077 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:18.077 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:17:18.077 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:17:18.077 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:17:18.077 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 163553 2024-03-20T09:17:18.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.078 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:17:18.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:17:18.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.406918+0000", 2024-03-20T09:17:18.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:17:18.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.406918+0000", 2024-03-20T09:17:18.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:17:18.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.406920+0000", 2024-03-20T09:17:18.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7439999999999999e-06 2024-03-20T09:17:18.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:17:18.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.406928+0000", 2024-03-20T09:17:18.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3960000000000006e-06 2024-03-20T09:17:18.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:17:18.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.406945+0000", 2024-03-20T09:17:18.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6804999999999999e-05 2024-03-20T09:17:18.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:17:18.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.406956+0000", 2024-03-20T09:17:18.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0675e-05 2024-03-20T09:17:18.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.407023+0000", 2024-03-20T09:17:18.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7341000000000005e-05 2024-03-20T09:17:18.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:17:18.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.407030+0000", 2024-03-20T09:17:18.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3880000000000002e-06 2024-03-20T09:17:18.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.080645+0000", 2024-03-20T09:17:18.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.67361427100000004 2024-03-20T09:17:18.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:17:18.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.080811+0000", 2024-03-20T09:17:18.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00016642800000000001 2024-03-20T09:17:18.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:17:18.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.080970+0000", 2024-03-20T09:17:18.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000158555 2024-03-20T09:17:18.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.085283+0000", 2024-03-20T09:17:18.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043139420000000003 2024-03-20T09:17:18.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.094146+0000", 2024-03-20T09:17:18.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0088626590000000002 2024-03-20T09:17:18.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:17:18.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.102539+0000", 2024-03-20T09:17:18.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083926090000000005 2024-03-20T09:17:18.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:17:18.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.102594+0000", 2024-03-20T09:17:18.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4925e-05 2024-03-20T09:17:18.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:17:18.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.102650+0000", 2024-03-20T09:17:18.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6783999999999997e-05 2024-03-20T09:17:18.083 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.083 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:17:18.083 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.083 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:163557 3.b 3:d0818a0e:::benchmark_data_smithi122_119671_object63775:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-03-20T09:17:18.083 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:12:06.416389+0000", 2024-03-20T09:17:18.083 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 311.19024321199998, 2024-03-20T09:17:18.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.68649045900000005, 2024-03-20T09:17:18.084 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:17:18.084 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:18.084 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:17:18.084 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:17:18.084 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:17:18.084 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 163557 2024-03-20T09:17:18.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.084 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:17:18.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:17:18.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.416389+0000", 2024-03-20T09:17:18.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:17:18.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.416389+0000", 2024-03-20T09:17:18.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:17:18.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.416391+0000", 2024-03-20T09:17:18.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7379999999999999e-06 2024-03-20T09:17:18.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:17:18.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.416399+0000", 2024-03-20T09:17:18.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4560000000000003e-06 2024-03-20T09:17:18.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:17:18.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.416416+0000", 2024-03-20T09:17:18.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6983999999999999e-05 2024-03-20T09:17:18.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:17:18.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.416427+0000", 2024-03-20T09:17:18.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1e-05 2024-03-20T09:17:18.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.416494+0000", 2024-03-20T09:17:18.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7621999999999997e-05 2024-03-20T09:17:18.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.087 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:17:18.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.416504+0000", 2024-03-20T09:17:18.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3770000000000002e-06 2024-03-20T09:17:18.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.087 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.085706+0000", 2024-03-20T09:17:18.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.669202301 2024-03-20T09:17:18.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.087 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:17:18.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.085852+0000", 2024-03-20T09:17:18.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000146589 2024-03-20T09:17:18.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:17:18.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.085973+0000", 2024-03-20T09:17:18.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000120389 2024-03-20T09:17:18.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.094189+0000", 2024-03-20T09:17:18.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0082157429999999993 2024-03-20T09:17:18.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:17:18.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.102717+0000", 2024-03-20T09:17:18.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0085289349999999996 2024-03-20T09:17:18.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.102775+0000", 2024-03-20T09:17:18.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7046000000000002e-05 2024-03-20T09:17:18.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:17:18.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.102836+0000", 2024-03-20T09:17:18.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1901000000000002e-05 2024-03-20T09:17:18.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:17:18.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.102880+0000", 2024-03-20T09:17:18.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3377999999999997e-05 2024-03-20T09:17:18.090 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.090 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:17:18.090 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.090 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:163567 3.b 3:d6a40b92:::benchmark_data_smithi122_119671_object63785:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-03-20T09:17:18.090 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:12:06.454340+0000", 2024-03-20T09:17:18.090 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 311.15229244599999, 2024-03-20T09:17:18.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.64869723599999995, 2024-03-20T09:17:18.091 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:17:18.091 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:18.091 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:17:18.091 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:17:18.091 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:17:18.091 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 163567 2024-03-20T09:17:18.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.091 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:17:18.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:17:18.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.454340+0000", 2024-03-20T09:17:18.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:17:18.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.454340+0000", 2024-03-20T09:17:18.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:17:18.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.454341+0000", 2024-03-20T09:17:18.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.093e-06 2024-03-20T09:17:18.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:17:18.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.454346+0000", 2024-03-20T09:17:18.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3109999999999996e-06 2024-03-20T09:17:18.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:17:18.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.454358+0000", 2024-03-20T09:17:18.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.189e-05 2024-03-20T09:17:18.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:17:18.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.454365+0000", 2024-03-20T09:17:18.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.37e-06 2024-03-20T09:17:18.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.454429+0000", 2024-03-20T09:17:18.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3829000000000002e-05 2024-03-20T09:17:18.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:17:18.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.454441+0000", 2024-03-20T09:17:18.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2357e-05 2024-03-20T09:17:18.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.089918+0000", 2024-03-20T09:17:18.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.63547673900000001 2024-03-20T09:17:18.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:17:18.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.090060+0000", 2024-03-20T09:17:18.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000142755 2024-03-20T09:17:18.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:17:18.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.090156+0000", 2024-03-20T09:17:18.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5086000000000004e-05 2024-03-20T09:17:18.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.102520+0000", 2024-03-20T09:17:18.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012364828 2024-03-20T09:17:18.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:17:18.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.102724+0000", 2024-03-20T09:17:18.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000204117 2024-03-20T09:17:18.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.102971+0000", 2024-03-20T09:17:18.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00024605900000000002 2024-03-20T09:17:18.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:17:18.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.103016+0000", 2024-03-20T09:17:18.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5182000000000002e-05 2024-03-20T09:17:18.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:17:18.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.103037+0000", 2024-03-20T09:17:18.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.162e-05 2024-03-20T09:17:18.096 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.096 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:17:18.097 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.097 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:163570 3.b 3:db3a9564:::benchmark_data_smithi122_119671_object63788:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-03-20T09:17:18.097 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:12:06.463745+0000", 2024-03-20T09:17:18.097 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 311.14288764000003, 2024-03-20T09:17:18.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.65138066100000003, 2024-03-20T09:17:18.097 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:17:18.097 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:18.097 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:17:18.097 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:17:18.097 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:17:18.097 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 163570 2024-03-20T09:17:18.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.098 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:17:18.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:17:18.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.463745+0000", 2024-03-20T09:17:18.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:17:18.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.463745+0000", 2024-03-20T09:17:18.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:17:18.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.463747+0000", 2024-03-20T09:17:18.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.857e-06 2024-03-20T09:17:18.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:17:18.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.463755+0000", 2024-03-20T09:17:18.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2810000000000008e-06 2024-03-20T09:17:18.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:17:18.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.463775+0000", 2024-03-20T09:17:18.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9536999999999999e-05 2024-03-20T09:17:18.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:17:18.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.463786+0000", 2024-03-20T09:17:18.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1396000000000001e-05 2024-03-20T09:17:18.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.463842+0000", 2024-03-20T09:17:18.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6248000000000001e-05 2024-03-20T09:17:18.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:17:18.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.463850+0000", 2024-03-20T09:17:18.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9510000000000006e-06 2024-03-20T09:17:18.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.094218+0000", 2024-03-20T09:17:18.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.63036753700000003 2024-03-20T09:17:18.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:17:18.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.094319+0000", 2024-03-20T09:17:18.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101702 2024-03-20T09:17:18.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:17:18.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.094428+0000", 2024-03-20T09:17:18.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000109015 2024-03-20T09:17:18.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.102703+0000", 2024-03-20T09:17:18.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0082749 2024-03-20T09:17:18.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:17:18.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.102732+0000", 2024-03-20T09:17:18.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8348e-05 2024-03-20T09:17:18.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.115019+0000", 2024-03-20T09:17:18.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012287752000000001 2024-03-20T09:17:18.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:17:18.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.115076+0000", 2024-03-20T09:17:18.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6524999999999998e-05 2024-03-20T09:17:18.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:17:18.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.115126+0000", 2024-03-20T09:17:18.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9611999999999999e-05 2024-03-20T09:17:18.103 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.103 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:17:18.103 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.103 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:163600 3.b 3:d30ded93:::benchmark_data_smithi122_119671_object63818:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-03-20T09:17:18.103 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:12:06.524642+0000", 2024-03-20T09:17:18.103 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 311.08199089800002, 2024-03-20T09:17:18.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.59071154400000003, 2024-03-20T09:17:18.104 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:17:18.104 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:18.104 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:17:18.104 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:17:18.104 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:17:18.104 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 163600 2024-03-20T09:17:18.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.104 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:17:18.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:17:18.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.524642+0000", 2024-03-20T09:17:18.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:17:18.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.524642+0000", 2024-03-20T09:17:18.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:17:18.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.524643+0000", 2024-03-20T09:17:18.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.133e-06 2024-03-20T09:17:18.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:17:18.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.524648+0000", 2024-03-20T09:17:18.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.887e-06 2024-03-20T09:17:18.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:17:18.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.524658+0000", 2024-03-20T09:17:18.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0647e-05 2024-03-20T09:17:18.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:17:18.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.524664+0000", 2024-03-20T09:17:18.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3419999999999998e-06 2024-03-20T09:17:18.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.524723+0000", 2024-03-20T09:17:18.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9837000000000003e-05 2024-03-20T09:17:18.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:17:18.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.524733+0000", 2024-03-20T09:17:18.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8500000000000006e-06 2024-03-20T09:17:18.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.098249+0000", 2024-03-20T09:17:18.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.57351528699999998 2024-03-20T09:17:18.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:17:18.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.098362+0000", 2024-03-20T09:17:18.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000113248 2024-03-20T09:17:18.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:17:18.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.098452+0000", 2024-03-20T09:17:18.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0093999999999995e-05 2024-03-20T09:17:18.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:17:18.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.102736+0000", 2024-03-20T09:17:18.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042841750000000003 2024-03-20T09:17:18.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.102925+0000", 2024-03-20T09:17:18.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018848000000000001 2024-03-20T09:17:18.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.115270+0000", 2024-03-20T09:17:18.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012345027999999999 2024-03-20T09:17:18.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:17:18.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.115313+0000", 2024-03-20T09:17:18.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3257000000000003e-05 2024-03-20T09:17:18.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:17:18.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.115353+0000", 2024-03-20T09:17:18.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0278999999999998e-05 2024-03-20T09:17:18.109 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.109 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:17:18.110 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.110 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:163616 3.b 3:df6321bf:::benchmark_data_smithi122_119671_object63834:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-03-20T09:17:18.110 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:12:06.574476+0000", 2024-03-20T09:17:18.110 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 311.03215649499998, 2024-03-20T09:17:18.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.54878080699999998, 2024-03-20T09:17:18.110 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:17:18.110 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:18.110 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:17:18.110 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:17:18.110 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:17:18.110 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 163616 2024-03-20T09:17:18.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.111 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:17:18.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:17:18.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.574476+0000", 2024-03-20T09:17:18.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:17:18.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.574476+0000", 2024-03-20T09:17:18.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:17:18.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.574477+0000", 2024-03-20T09:17:18.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4100000000000001e-06 2024-03-20T09:17:18.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:17:18.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.574484+0000", 2024-03-20T09:17:18.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3110000000000002e-06 2024-03-20T09:17:18.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:17:18.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.574499+0000", 2024-03-20T09:17:18.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5218999999999999e-05 2024-03-20T09:17:18.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:17:18.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.574508+0000", 2024-03-20T09:17:18.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3409999999999992e-06 2024-03-20T09:17:18.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.574564+0000", 2024-03-20T09:17:18.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5970000000000001e-05 2024-03-20T09:17:18.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:17:18.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.574572+0000", 2024-03-20T09:17:18.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1640000000000001e-06 2024-03-20T09:17:18.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.103055+0000", 2024-03-20T09:17:18.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.52848255700000002 2024-03-20T09:17:18.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:17:18.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.103126+0000", 2024-03-20T09:17:18.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1009999999999999e-05 2024-03-20T09:17:18.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:17:18.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.103211+0000", 2024-03-20T09:17:18.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4796000000000006e-05 2024-03-20T09:17:18.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:17:18.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.115139+0000", 2024-03-20T09:17:18.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011927917 2024-03-20T09:17:18.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.115221+0000", 2024-03-20T09:17:18.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2009999999999996e-05 2024-03-20T09:17:18.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.123176+0000", 2024-03-20T09:17:18.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0079552709999999999 2024-03-20T09:17:18.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:17:18.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.123210+0000", 2024-03-20T09:17:18.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4341000000000002e-05 2024-03-20T09:17:18.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:17:18.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.123257+0000", 2024-03-20T09:17:18.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6489999999999997e-05 2024-03-20T09:17:18.116 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.116 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:17:18.116 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.116 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:163636 3.b 3:ddb46087:::benchmark_data_smithi122_119671_object63854:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-03-20T09:17:18.116 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:12:06.606952+0000", 2024-03-20T09:17:18.116 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 310.99968087600001, 2024-03-20T09:17:18.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.516426465, 2024-03-20T09:17:18.116 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:17:18.116 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:18.117 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:17:18.117 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:17:18.117 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:17:18.117 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 163636 2024-03-20T09:17:18.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.117 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:17:18.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:17:18.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.606952+0000", 2024-03-20T09:17:18.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:17:18.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.606952+0000", 2024-03-20T09:17:18.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:17:18.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.606953+0000", 2024-03-20T09:17:18.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.19e-06 2024-03-20T09:17:18.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:17:18.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.606959+0000", 2024-03-20T09:17:18.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8259999999999998e-06 2024-03-20T09:17:18.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:17:18.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.606971+0000", 2024-03-20T09:17:18.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2592000000000001e-05 2024-03-20T09:17:18.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:17:18.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.606981+0000", 2024-03-20T09:17:18.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2399999999999996e-06 2024-03-20T09:17:18.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.607044+0000", 2024-03-20T09:17:18.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3427000000000001e-05 2024-03-20T09:17:18.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:17:18.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.607054+0000", 2024-03-20T09:17:18.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0339e-05 2024-03-20T09:17:18.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.106740+0000", 2024-03-20T09:17:18.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.49968526000000002 2024-03-20T09:17:18.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:17:18.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.106865+0000", 2024-03-20T09:17:18.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012532600000000001 2024-03-20T09:17:18.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:17:18.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.106982+0000", 2024-03-20T09:17:18.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011677599999999999 2024-03-20T09:17:18.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:17:18.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.119379+0000", 2024-03-20T09:17:18.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01239751 2024-03-20T09:17:18.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.123116+0000", 2024-03-20T09:17:18.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037364590000000001 2024-03-20T09:17:18.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.123289+0000", 2024-03-20T09:17:18.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00017384099999999999 2024-03-20T09:17:18.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:17:18.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.123336+0000", 2024-03-20T09:17:18.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6400000000000003e-05 2024-03-20T09:17:18.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:17:18.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.123378+0000", 2024-03-20T09:17:18.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2279e-05 2024-03-20T09:17:18.123 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.123 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:17:18.123 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.123 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:163675 3.b 3:de6813f5:::benchmark_data_smithi122_119671_object63893:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-03-20T09:17:18.123 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:12:06.673840+0000", 2024-03-20T09:17:18.123 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 310.93279265799998, 2024-03-20T09:17:18.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.44968040399999998, 2024-03-20T09:17:18.123 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:17:18.123 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:18.123 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:17:18.123 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:17:18.123 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:17:18.124 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 163675 2024-03-20T09:17:18.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.124 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:17:18.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:17:18.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.673840+0000", 2024-03-20T09:17:18.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:17:18.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.673840+0000", 2024-03-20T09:17:18.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:17:18.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.673841+0000", 2024-03-20T09:17:18.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.342e-06 2024-03-20T09:17:18.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:17:18.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.673847+0000", 2024-03-20T09:17:18.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0780000000000001e-06 2024-03-20T09:17:18.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:17:18.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.673859+0000", 2024-03-20T09:17:18.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.218e-05 2024-03-20T09:17:18.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:17:18.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.673867+0000", 2024-03-20T09:17:18.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.418e-06 2024-03-20T09:17:18.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.673946+0000", 2024-03-20T09:17:18.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9443999999999994e-05 2024-03-20T09:17:18.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:17:18.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.673957+0000", 2024-03-20T09:17:18.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1094000000000001e-05 2024-03-20T09:17:18.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.111018+0000", 2024-03-20T09:17:18.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.43706057700000001 2024-03-20T09:17:18.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:17:18.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.111115+0000", 2024-03-20T09:17:18.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.6879999999999994e-05 2024-03-20T09:17:18.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:17:18.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.111216+0000", 2024-03-20T09:17:18.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010155 2024-03-20T09:17:18.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:17:18.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.123068+0000", 2024-03-20T09:17:18.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011851857 2024-03-20T09:17:18.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.123155+0000", 2024-03-20T09:17:18.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6420000000000003e-05 2024-03-20T09:17:18.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.123424+0000", 2024-03-20T09:17:18.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000269542 2024-03-20T09:17:18.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:17:18.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.123479+0000", 2024-03-20T09:17:18.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4295000000000001e-05 2024-03-20T09:17:18.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:17:18.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.123520+0000", 2024-03-20T09:17:18.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1727000000000002e-05 2024-03-20T09:17:18.129 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.129 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:17:18.129 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.130 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:163694 3.b 3:d8316fd6:::benchmark_data_smithi122_119671_object63912:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e486)", 2024-03-20T09:17:18.130 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:12:06.697937+0000", 2024-03-20T09:17:18.130 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 310.90869528899998, 2024-03-20T09:17:18.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.43027906700000002, 2024-03-20T09:17:18.130 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:17:18.130 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:18.130 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:17:18.130 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:17:18.130 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:17:18.130 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 163694 2024-03-20T09:17:18.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.130 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:17:18.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:17:18.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.697937+0000", 2024-03-20T09:17:18.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:17:18.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.697937+0000", 2024-03-20T09:17:18.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:17:18.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.697938+0000", 2024-03-20T09:17:18.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.192e-06 2024-03-20T09:17:18.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.132 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:17:18.132 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.697944+0000", 2024-03-20T09:17:18.132 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2299999999999999e-06 2024-03-20T09:17:18.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.132 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:17:18.132 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.697955+0000", 2024-03-20T09:17:18.132 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1688999999999999e-05 2024-03-20T09:17:18.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.132 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:17:18.132 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.697964+0000", 2024-03-20T09:17:18.133 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6819999999999992e-06 2024-03-20T09:17:18.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.133 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.133 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.698022+0000", 2024-03-20T09:17:18.133 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8158999999999997e-05 2024-03-20T09:17:18.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.133 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:17:18.133 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:06.698033+0000", 2024-03-20T09:17:18.133 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0851e-05 2024-03-20T09:17:18.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.133 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.115397+0000", 2024-03-20T09:17:18.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.41736394900000001 2024-03-20T09:17:18.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.134 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:17:18.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.115494+0000", 2024-03-20T09:17:18.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.6581000000000002e-05 2024-03-20T09:17:18.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.134 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:17:18.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.115567+0000", 2024-03-20T09:17:18.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3143000000000001e-05 2024-03-20T09:17:18.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.123568+0000", 2024-03-20T09:17:18.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080014479999999995 2024-03-20T09:17:18.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.128055+0000", 2024-03-20T09:17:18.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004487001 2024-03-20T09:17:18.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:17:18.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.128120+0000", 2024-03-20T09:17:18.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4498000000000005e-05 2024-03-20T09:17:18.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:17:18.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.128169+0000", 2024-03-20T09:17:18.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9221000000000002e-05 2024-03-20T09:17:18.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:17:18.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:07.128216+0000", 2024-03-20T09:17:18.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7423000000000003e-05 2024-03-20T09:17:18.136 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.136 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:17:18.136 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.137 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25690.0:125029 4.c 4:30baefc8:::benchmark_data_smithi122_145698_object4110:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e556)", 2024-03-20T09:17:18.137 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:15:14.463352+0000", 2024-03-20T09:17:18.137 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 123.143280788, 2024-03-20T09:17:18.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.21636993500000001, 2024-03-20T09:17:18.137 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:17:18.137 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:18.137 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:17:18.137 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25690", 2024-03-20T09:17:18.137 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:17:18.137 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 125029 2024-03-20T09:17:18.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.137 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:17:18.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.137 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:17:18.137 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:14.463352+0000", 2024-03-20T09:17:18.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.138 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:17:18.138 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:14.463352+0000", 2024-03-20T09:17:18.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:17:18.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.138 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:17:18.138 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:14.463354+0000", 2024-03-20T09:17:18.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.835e-06 2024-03-20T09:17:18.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:17:18.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:14.463361+0000", 2024-03-20T09:17:18.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1790000000000002e-06 2024-03-20T09:17:18.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:17:18.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:14.463373+0000", 2024-03-20T09:17:18.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2659e-05 2024-03-20T09:17:18.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:17:18.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:14.463381+0000", 2024-03-20T09:17:18.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6290000000000001e-06 2024-03-20T09:17:18.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:17:18.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:14.654346+0000", 2024-03-20T09:17:18.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.19096487500000001 2024-03-20T09:17:18.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:17:18.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:14.654489+0000", 2024-03-20T09:17:18.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014344100000000001 2024-03-20T09:17:18.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 6,8", 2024-03-20T09:17:18.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:14.654621+0000", 2024-03-20T09:17:18.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000131777 2024-03-20T09:17:18.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:14.662673+0000", 2024-03-20T09:17:18.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080513649999999996 2024-03-20T09:17:18.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:18.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:14.662711+0000", 2024-03-20T09:17:18.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8118000000000003e-05 2024-03-20T09:17:18.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:17:18.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:14.679627+0000", 2024-03-20T09:17:18.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01691645 2024-03-20T09:17:18.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.142 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:17:18.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:14.679674+0000", 2024-03-20T09:17:18.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6570999999999998e-05 2024-03-20T09:17:18.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:17:18.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:17:18.142 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:17:18.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:14.679722+0000", 2024-03-20T09:17:18.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8035999999999999e-05 2024-03-20T09:17:18.142 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.142 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:17:18.142 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.142 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:17:18.143 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:17:18.143 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:17:18.181 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:17:18.182 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:17:18.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:18.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:17:18.285 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:17:18.343 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:18.433 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:17:18.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:18.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:17:18.447 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:18 smithi026 ceph-mon[68309]: osdmap e589: 12 total, 12 up, 10 in 2024-03-20T09:17:18.447 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:18 smithi026 ceph-mon[68309]: 4.f deep-scrub ok 2024-03-20T09:17:18.447 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:18 smithi026 ceph-mon[68309]: pgmap v1410: 47 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing+deep, 45 active+clean; 1.8 GiB data, 9.2 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 690 op/s; 4/89079 objects misplaced (0.004%); 14 MiB/s, 216 objects/s recovering 2024-03-20T09:17:18.447 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:18 smithi026 ceph-mon[68309]: osdmap e590: 12 total, 12 up, 10 in 2024-03-20T09:17:18.543 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:17:18.567 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:18 smithi115 ceph-mon[72476]: osdmap e589: 12 total, 12 up, 10 in 2024-03-20T09:17:18.567 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:18 smithi115 ceph-mon[72476]: 4.f deep-scrub ok 2024-03-20T09:17:18.568 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:18 smithi115 ceph-mon[72476]: pgmap v1410: 47 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing+deep, 45 active+clean; 1.8 GiB data, 9.2 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 690 op/s; 4/89079 objects misplaced (0.004%); 14 MiB/s, 216 objects/s recovering 2024-03-20T09:17:18.568 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:18 smithi115 ceph-mon[72476]: osdmap e590: 12 total, 12 up, 10 in 2024-03-20T09:17:18.569 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:18.591 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:17:18.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:18.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:17:18.698 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:17:18.699 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:18 smithi122 ceph-mon[73495]: osdmap e589: 12 total, 12 up, 10 in 2024-03-20T09:17:18.699 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:18 smithi122 ceph-mon[73495]: 4.f deep-scrub ok 2024-03-20T09:17:18.699 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:18 smithi122 ceph-mon[73495]: pgmap v1410: 47 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing+deep, 45 active+clean; 1.8 GiB data, 9.2 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 690 op/s; 4/89079 objects misplaced (0.004%); 14 MiB/s, 216 objects/s recovering 2024-03-20T09:17:18.699 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:18 smithi122 ceph-mon[73495]: osdmap e590: 12 total, 12 up, 10 in 2024-03-20T09:17:18.743 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:18.768 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:17:18.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:18.772 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:17:18.873 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:17:18.955 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-20T09:17:18.968 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:19.069 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:17:19.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:19.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:17:19.176 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:17:19.244 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:19.270 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:17:19.363 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:17:19.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:19.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:17:19.468 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:17:19.497 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:19.526 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:17:19.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:19.530 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:17:19.631 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:17:19.673 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:19.712 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:17:19.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:19.716 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:17:19.817 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:17:19.826 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:17:19.848 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:19.876 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:17:19.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:19.879 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:17:19.914 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:17:19.979 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:17:20.015 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:20.089 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:20 smithi026 ceph-mon[68309]: pgmap v1412: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.8 GiB data, 9.2 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 516 op/s; 4/88473 objects misplaced (0.005%); 6.3 MiB/s, 101 objects/s recovering 2024-03-20T09:17:20.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:20 smithi122 ceph-mon[73495]: pgmap v1412: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.8 GiB data, 9.2 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 516 op/s; 4/88473 objects misplaced (0.005%); 6.3 MiB/s, 101 objects/s recovering 2024-03-20T09:17:20.372 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:17:20.375 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:20 smithi115 ceph-mon[72476]: pgmap v1412: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 1.8 GiB data, 9.2 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 516 op/s; 4/88473 objects misplaced (0.005%); 6.3 MiB/s, 101 objects/s recovering 2024-03-20T09:17:20.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:20.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:17:20.477 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:17:20.575 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:20.791 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:17:20.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:20.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:17:20.898 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:17:20.954 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:21.006 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:17:21.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:21.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:17:21.115 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:17:21.193 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:21.362 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:17:21.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:21.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:17:21.469 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:17:21.507 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:21.636 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:17:21.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:21.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:17:21.668 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:17:21.669 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:17:21.669 INFO:teuthology.orchestra.run.smithi115.stdout: "num_ops": 0 2024-03-20T09:17:21.669 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:17:21.745 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:17:21.781 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:21.805 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:17:21.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:21.808 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:17:21.909 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:17:21.941 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:21.963 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:17:21.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:21.967 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:17:22.039 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:17:22.058 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.1 to 1 (10000) 2024-03-20T09:17:22.068 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:17:22.101 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:21 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3491147427' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-20T09:17:22.101 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:21 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-20T09:17:22.102 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:22.114 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:17:22.124 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:17:22.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:22.131 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:17:22.175 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:17:22.229 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:17:22.232 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:17:22.265 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:21 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3491147427' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-20T09:17:22.265 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:21 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-20T09:17:22.267 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:22.292 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:17:22.337 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:17:22.383 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:17:22.394 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:17:22.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:22.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:17:22.417 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:17:22.433 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-20T09:17:22.458 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:17:22.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:21 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3491147427' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-20T09:17:22.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:21 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-20T09:17:22.497 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:17:22.501 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:17:22.553 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:17:22.567 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:22.598 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:17:22.601 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:17:22.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:22.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:17:22.638 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:17:22.648 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:17:22.648 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:17:22.648 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:17:22.648 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:17:22.648 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:17:22.648 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:17:22.648 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:17:22.648 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:17:22.648 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:17:22.648 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:17:22.648 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:17:22.649 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:17:22.711 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:17:22.774 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:22.804 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:17:23.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:23 smithi115 ceph-mon[72476]: pgmap v1413: 47 pgs: 47 active+clean; 1.7 GiB data, 8.8 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 549 op/s; 6.3 MiB/s, 101 objects/s recovering 2024-03-20T09:17:23.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:23 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-20T09:17:23.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:23 smithi115 ceph-mon[72476]: osdmap e591: 12 total, 12 up, 10 in 2024-03-20T09:17:23.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:23 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3678460595' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:17:23.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:23 smithi122 ceph-mon[73495]: pgmap v1413: 47 pgs: 47 active+clean; 1.7 GiB data, 8.8 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 549 op/s; 6.3 MiB/s, 101 objects/s recovering 2024-03-20T09:17:23.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:23 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-20T09:17:23.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:23 smithi122 ceph-mon[73495]: osdmap e591: 12 total, 12 up, 10 in 2024-03-20T09:17:23.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:23 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3678460595' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:17:23.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:23 smithi026 ceph-mon[68309]: pgmap v1413: 47 pgs: 47 active+clean; 1.7 GiB data, 8.8 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 549 op/s; 6.3 MiB/s, 101 objects/s recovering 2024-03-20T09:17:23.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:23 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-20T09:17:23.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:23 smithi026 ceph-mon[68309]: osdmap e591: 12 total, 12 up, 10 in 2024-03-20T09:17:23.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:23 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3678460595' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:17:24.870 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:17:24.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:24.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:17:24.979 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:17:25.152 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:25.199 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:17:25.199 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:17:25.208 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:17:25.215 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:25 smithi115 ceph-mon[72476]: pgmap v1415: 47 pgs: 47 active+clean; 1.6 GiB data, 8.7 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 559 op/s; 9.5 KiB/s, 0 objects/s recovering 2024-03-20T09:17:25.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:25.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:17:25.224 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:25 smithi026 ceph-mon[68309]: pgmap v1415: 47 pgs: 47 active+clean; 1.6 GiB data, 8.7 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 559 op/s; 9.5 KiB/s, 0 objects/s recovering 2024-03-20T09:17:25.319 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:17:25.349 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:25.374 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:17:25.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:25.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:17:25.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:25 smithi122 ceph-mon[73495]: pgmap v1415: 47 pgs: 47 active+clean; 1.6 GiB data, 8.7 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 559 op/s; 9.5 KiB/s, 0 objects/s recovering 2024-03-20T09:17:25.478 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:17:25.544 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:26.339 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:26 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:17:26.339 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:26 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:17:26.339 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:26 smithi115 ceph-mon[72476]: osdmap e592: 12 total, 12 up, 10 in 2024-03-20T09:17:26.339 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:26 smithi115 ceph-mon[72476]: pgmap v1417: 47 pgs: 47 active+clean; 1.5 GiB data, 8.3 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 709 op/s; 8.2 KiB/s, 0 objects/s recovering 2024-03-20T09:17:26.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:26 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:17:26.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:26 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:17:26.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:26 smithi122 ceph-mon[73495]: osdmap e592: 12 total, 12 up, 10 in 2024-03-20T09:17:26.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:26 smithi122 ceph-mon[73495]: pgmap v1417: 47 pgs: 47 active+clean; 1.5 GiB data, 8.3 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 709 op/s; 8.2 KiB/s, 0 objects/s recovering 2024-03-20T09:17:26.481 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:26 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:17:26.481 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:26 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:17:26.481 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:26 smithi026 ceph-mon[68309]: osdmap e592: 12 total, 12 up, 10 in 2024-03-20T09:17:26.481 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:26 smithi026 ceph-mon[68309]: pgmap v1417: 47 pgs: 47 active+clean; 1.5 GiB data, 8.3 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 709 op/s; 8.2 KiB/s, 0 objects/s recovering 2024-03-20T09:17:26.699 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-20T09:17:27.068 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:17:27.088 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 2 1 2024-03-20T09:17:27.650 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:17:27.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:27.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:17:27.761 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:17:27.792 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:27.824 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:17:27.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:27.827 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:17:27.928 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:17:27.962 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:27.996 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:17:27.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:28.000 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:17:28.101 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:17:28.135 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:28.144 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:17:28.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:28 smithi115 ceph-mon[72476]: pgmap v1418: 47 pgs: 47 active+clean; 1.4 GiB data, 7.9 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 904 op/s; 8.0 KiB/s, 0 objects/s recovering 2024-03-20T09:17:28.434 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:28 smithi026 ceph-mon[68309]: pgmap v1418: 47 pgs: 47 active+clean; 1.4 GiB data, 7.9 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 904 op/s; 8.0 KiB/s, 0 objects/s recovering 2024-03-20T09:17:28.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:28 smithi122 ceph-mon[73495]: pgmap v1418: 47 pgs: 47 active+clean; 1.4 GiB data, 7.9 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 904 op/s; 8.0 KiB/s, 0 objects/s recovering 2024-03-20T09:17:29.089 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:17:29.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:29.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:17:29.194 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:17:29.284 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:29.425 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:17:29.458 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:17:29.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:29.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:17:29.464 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:29 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/304689950' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:17:29.565 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:17:29.616 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:29.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:29 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/304689950' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:17:29.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:29 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/304689950' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:17:29.707 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:17:29.707 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:17:30.131 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:17:30.287 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:17:30.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:30.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:17:30.399 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:17:30.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:30 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:17:30.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:30 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/304689950' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:17:30.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:30 smithi026 ceph-mon[68309]: osdmap e593: 12 total, 12 up, 10 in 2024-03-20T09:17:30.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:30 smithi026 ceph-mon[68309]: pgmap v1420: 47 pgs: 47 active+clean; 1.3 GiB data, 7.8 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 707 op/s 2024-03-20T09:17:30.487 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:30.507 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-20T09:17:30.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:30 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:17:30.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:30 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/304689950' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:17:30.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:30 smithi122 ceph-mon[73495]: osdmap e593: 12 total, 12 up, 10 in 2024-03-20T09:17:30.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:30 smithi122 ceph-mon[73495]: pgmap v1420: 47 pgs: 47 active+clean; 1.3 GiB data, 7.8 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 707 op/s 2024-03-20T09:17:30.816 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:17:30.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:30 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:17:30.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:30 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/304689950' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:17:30.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:30 smithi115 ceph-mon[72476]: osdmap e593: 12 total, 12 up, 10 in 2024-03-20T09:17:30.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:30 smithi115 ceph-mon[72476]: pgmap v1420: 47 pgs: 47 active+clean; 1.3 GiB data, 7.8 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 707 op/s 2024-03-20T09:17:31.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:31 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:17:31.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:31 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:17:31.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:31 smithi115 ceph-mon[72476]: osdmap e594: 12 total, 12 up, 10 in 2024-03-20T09:17:31.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:31 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:17:31.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:31 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:17:31.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:31 smithi122 ceph-mon[73495]: osdmap e594: 12 total, 12 up, 10 in 2024-03-20T09:17:31.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:31 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:17:31.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:31 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:17:31.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:31 smithi026 ceph-mon[68309]: osdmap e594: 12 total, 12 up, 10 in 2024-03-20T09:17:31.978 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:17:31.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:31.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:17:32.083 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:17:32.119 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:32.145 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:17:32.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:32.149 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:17:32.250 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:17:32.293 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:32.317 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:17:32.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:32.321 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:17:32.422 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:17:32.455 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:32.485 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:17:32.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:32.489 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:17:32.490 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:17:32.589 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:17:32.631 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:32.727 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:32 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:17:32.727 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:32 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:17:32.727 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:32 smithi026 ceph-mon[68309]: pgmap v1422: 47 pgs: 47 active+clean; 1.2 GiB data, 7.3 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.01k op/s 2024-03-20T09:17:32.829 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:32 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:17:32.829 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:32 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:17:32.829 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:32 smithi115 ceph-mon[72476]: pgmap v1422: 47 pgs: 47 active+clean; 1.2 GiB data, 7.3 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.01k op/s 2024-03-20T09:17:32.833 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:17:32.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:32.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:17:32.936 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:17:32.950 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:32 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:17:32.950 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:32 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:17:32.950 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:32 smithi122 ceph-mon[73495]: pgmap v1422: 47 pgs: 47 active+clean; 1.2 GiB data, 7.3 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.01k op/s 2024-03-20T09:17:32.966 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:33.215 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:17:33.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:33.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:17:33.320 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:17:33.398 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:33.447 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:17:33.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:33.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:17:33.495 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:17:33.495 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:17:33.518 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.2 to 1 (10000) 2024-03-20T09:17:33.553 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:17:33.589 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:33.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:33 smithi026 ceph-mon[68309]: osdmap e595: 12 total, 12 up, 10 in 2024-03-20T09:17:33.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:33 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:17:33.724 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:33 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:17:33.724 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:33 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4136752715' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:17:33.724 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:33 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:17:33.879 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:17:33.889 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:33 smithi115 ceph-mon[72476]: osdmap e595: 12 total, 12 up, 10 in 2024-03-20T09:17:33.889 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:33 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:17:33.889 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:33 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:17:33.890 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:33 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4136752715' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:17:33.890 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:33 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:17:33.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:33 smithi122 ceph-mon[73495]: osdmap e595: 12 total, 12 up, 10 in 2024-03-20T09:17:33.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:33 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:17:33.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:33 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:17:33.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:33 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4136752715' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:17:33.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:33 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:17:34.309 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-20T09:17:34.491 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:34 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-20T09:17:34.491 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:34 smithi026 ceph-mon[68309]: osdmap e596: 12 total, 12 up, 10 in 2024-03-20T09:17:34.491 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:34 smithi026 ceph-mon[68309]: pgmap v1425: 47 pgs: 47 active+clean; 1.1 GiB data, 7.1 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 923 op/s 2024-03-20T09:17:34.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:34 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-20T09:17:34.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:34 smithi115 ceph-mon[72476]: osdmap e596: 12 total, 12 up, 10 in 2024-03-20T09:17:34.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:34 smithi115 ceph-mon[72476]: pgmap v1425: 47 pgs: 47 active+clean; 1.1 GiB data, 7.1 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 923 op/s 2024-03-20T09:17:34.673 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:17:34.873 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:17:34.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:34.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:17:34.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:34 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-20T09:17:34.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:34 smithi122 ceph-mon[73495]: osdmap e596: 12 total, 12 up, 10 in 2024-03-20T09:17:34.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:34 smithi122 ceph-mon[73495]: pgmap v1425: 47 pgs: 47 active+clean; 1.1 GiB data, 7.1 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 923 op/s 2024-03-20T09:17:34.976 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:17:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:35.113 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:17:35.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:35.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:17:35.218 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:17:35.252 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:35.975 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:17:36.007 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:17:36.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:36.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:17:36.019 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:17:36.053 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:17:36.111 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:17:36.129 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:17:36.141 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:36.170 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:17:36.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:36.177 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:17:36.202 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:17:36.203 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:17:36.248 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:17:36.278 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:17:36.290 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:17:36.306 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:36 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/640348217' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:17:36.310 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:36.327 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:17:36.363 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:17:36.415 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:17:36.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:35 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/640348217' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:17:36.456 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:17:36.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:35 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/640348217' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:17:36.498 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:17:36.498 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:17:36.498 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:17:36.499 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:17:36.499 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:17:36.499 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:17:36.499 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:17:36.499 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:17:36.499 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:17:36.499 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:17:36.499 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:17:36.499 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:17:36.605 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 5 1 2024-03-20T09:17:36.666 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:17:36.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:36.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:17:36.772 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:17:36.888 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:37.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:37 smithi115 ceph-mon[72476]: pgmap v1426: 47 pgs: 47 active+clean; 977 MiB data, 6.8 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.06k op/s 2024-03-20T09:17:37.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:37 smithi026 ceph-mon[68309]: pgmap v1426: 47 pgs: 47 active+clean; 977 MiB data, 6.8 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.06k op/s 2024-03-20T09:17:37.259 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:17:37.260 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:17:37.260 INFO:teuthology.orchestra.run.smithi115.stdout: "complaint_time": 30, 2024-03-20T09:17:37.260 INFO:teuthology.orchestra.run.smithi115.stdout: "num_blocked_ops": 0 2024-03-20T09:17:37.260 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:17:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:37 smithi122 ceph-mon[73495]: pgmap v1426: 47 pgs: 47 active+clean; 977 MiB data, 6.8 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 1.06k op/s 2024-03-20T09:17:37.675 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:17:37.716 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:17:37.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:37.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:17:37.821 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:17:37.867 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:37.877 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:17:37.973 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:17:37.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:37.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:17:38.090 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-20T09:17:38.091 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:17:38.122 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:38.156 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:17:38.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:38.162 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:17:38.263 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:17:38.301 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:38.328 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:17:38.335 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:38 smithi115 ceph-mon[72476]: pgmap v1427: 47 pgs: 47 active+clean; 863 MiB data, 6.3 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 942 op/s 2024-03-20T09:17:38.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:38.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:17:38.438 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:17:38.442 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:17:38.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:38 smithi122 ceph-mon[73495]: pgmap v1427: 47 pgs: 47 active+clean; 863 MiB data, 6.3 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 942 op/s 2024-03-20T09:17:38.471 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:38.488 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:38 smithi026 ceph-mon[68309]: pgmap v1427: 47 pgs: 47 active+clean; 863 MiB data, 6.3 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 942 op/s 2024-03-20T09:17:38.492 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:17:38.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:38.496 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:17:38.597 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:17:38.630 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:38.679 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:17:38.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:38.682 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:17:38.783 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:17:38.834 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:38.935 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:17:38.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:38.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:17:39.048 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:17:39.088 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:39.252 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:17:39.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:39.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:17:39.363 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:17:39.403 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:39.432 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:17:39.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:39.441 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:17:39.542 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:17:39.572 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:39.605 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:17:39.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:39.610 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:17:39.711 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:17:39.729 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:17:39.764 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:40.141 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:17:40.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:40.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:17:40.247 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:17:40.302 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:40.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:40 smithi115 ceph-mon[72476]: pgmap v1428: 47 pgs: 47 active+clean; 846 MiB data, 6.3 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 720 op/s 2024-03-20T09:17:40.431 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:40 smithi026 ceph-mon[68309]: pgmap v1428: 47 pgs: 47 active+clean; 846 MiB data, 6.3 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 720 op/s 2024-03-20T09:17:40.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:40 smithi122 ceph-mon[73495]: pgmap v1428: 47 pgs: 47 active+clean; 846 MiB data, 6.3 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 720 op/s 2024-03-20T09:17:40.546 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:17:40.546 INFO:teuthology.orchestra.run.smithi115.stdout: "size": 20, 2024-03-20T09:17:40.546 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 600, 2024-03-20T09:17:40.546 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [ 2024-03-20T09:17:40.546 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.546 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.25690.0:109039 4.a 4:598af869:::benchmark_data_smithi122_145698_object109038:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e550)", 2024-03-20T09:17:40.546 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:14:43.615620+0000", 2024-03-20T09:17:40.546 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 176.871502859, 2024-03-20T09:17:40.546 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.077999587999999995, 2024-03-20T09:17:40.546 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:17:40.547 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:40.547 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:17:40.547 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.25690", 2024-03-20T09:17:40.547 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:17:40.547 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 109039 2024-03-20T09:17:40.547 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.547 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:17:40.547 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.547 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:17:40.548 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:43.615620+0000", 2024-03-20T09:17:40.548 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.548 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.548 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.548 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:17:40.548 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:43.615620+0000", 2024-03-20T09:17:40.548 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.548 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.548 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.548 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:17:40.548 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:43.615622+0000", 2024-03-20T09:17:40.548 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.587e-06 2024-03-20T09:17:40.549 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.549 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.549 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:17:40.549 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:43.615681+0000", 2024-03-20T09:17:40.549 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9250999999999999e-05 2024-03-20T09:17:40.549 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.549 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.549 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:17:40.549 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:43.615694+0000", 2024-03-20T09:17:40.549 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3635000000000001e-05 2024-03-20T09:17:40.549 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.549 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.549 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:17:40.550 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:43.615702+0000", 2024-03-20T09:17:40.550 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.695e-06 2024-03-20T09:17:40.550 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.550 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.550 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:17:40.550 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:43.615760+0000", 2024-03-20T09:17:40.550 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8266000000000001e-05 2024-03-20T09:17:40.550 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.550 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.550 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:17:40.550 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:43.615878+0000", 2024-03-20T09:17:40.550 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000117484 2024-03-20T09:17:40.551 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.551 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.551 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,10", 2024-03-20T09:17:40.551 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:43.615983+0000", 2024-03-20T09:17:40.551 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000104895 2024-03-20T09:17:40.551 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.551 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.551 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:17:40.552 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:43.618726+0000", 2024-03-20T09:17:40.552 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0027429770000000002 2024-03-20T09:17:40.552 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.552 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.552 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:40.552 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:43.620058+0000", 2024-03-20T09:17:40.552 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0013323860000000001 2024-03-20T09:17:40.552 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.552 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.553 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:40.553 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:43.693530+0000", 2024-03-20T09:17:40.553 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.073471929000000005 2024-03-20T09:17:40.553 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.553 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.553 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:17:40.553 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:43.693571+0000", 2024-03-20T09:17:40.553 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.0907e-05 2024-03-20T09:17:40.553 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.553 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.553 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:17:40.553 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:43.693620+0000", 2024-03-20T09:17:40.553 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8575999999999997e-05 2024-03-20T09:17:40.554 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.554 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:17:40.554 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.554 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.554 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.554 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.25690.0:110754 4.a 4:5eb6d213:::benchmark_data_smithi122_145698_object110753:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e550)", 2024-03-20T09:17:40.554 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:14:45.510376+0000", 2024-03-20T09:17:40.554 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 174.97674632499999, 2024-03-20T09:17:40.554 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.13335748, 2024-03-20T09:17:40.554 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:17:40.554 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:40.554 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:17:40.554 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.25690", 2024-03-20T09:17:40.554 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:17:40.555 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 110754 2024-03-20T09:17:40.555 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.555 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:17:40.555 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.555 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:17:40.555 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:45.510376+0000", 2024-03-20T09:17:40.555 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.555 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.555 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.555 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:17:40.555 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:45.510376+0000", 2024-03-20T09:17:40.555 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.555 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.555 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.555 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:17:40.556 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:45.510378+0000", 2024-03-20T09:17:40.556 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.316e-06 2024-03-20T09:17:40.556 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.556 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.556 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:17:40.556 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:45.510428+0000", 2024-03-20T09:17:40.556 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0658000000000002e-05 2024-03-20T09:17:40.556 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.556 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.556 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:17:40.556 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:45.510439+0000", 2024-03-20T09:17:40.556 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0929e-05 2024-03-20T09:17:40.556 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.556 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.556 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:17:40.557 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:45.510445+0000", 2024-03-20T09:17:40.557 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5820000000000001e-06 2024-03-20T09:17:40.557 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.557 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.557 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:17:40.557 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:45.510488+0000", 2024-03-20T09:17:40.557 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.3229999999999998e-05 2024-03-20T09:17:40.557 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.557 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.557 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:17:40.557 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:45.510596+0000", 2024-03-20T09:17:40.557 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000107357 2024-03-20T09:17:40.558 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.558 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.558 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,10", 2024-03-20T09:17:40.558 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:45.510695+0000", 2024-03-20T09:17:40.558 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.9376000000000005e-05 2024-03-20T09:17:40.558 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.558 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.558 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:17:40.558 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:45.513572+0000", 2024-03-20T09:17:40.558 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0028768470000000001 2024-03-20T09:17:40.558 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.558 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.558 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:40.559 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:45.513815+0000", 2024-03-20T09:17:40.559 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00024340399999999999 2024-03-20T09:17:40.559 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.559 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.559 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:40.559 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:45.643643+0000", 2024-03-20T09:17:40.559 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.129828155 2024-03-20T09:17:40.559 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.559 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.559 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:17:40.559 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:45.643686+0000", 2024-03-20T09:17:40.559 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.2239e-05 2024-03-20T09:17:40.559 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.559 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.559 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:17:40.559 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:45.643734+0000", 2024-03-20T09:17:40.559 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8387000000000003e-05 2024-03-20T09:17:40.560 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.560 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:17:40.560 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.560 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.560 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.560 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.25690.0:110762 4.a 4:59fddb10:::benchmark_data_smithi122_145698_object110761:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e550)", 2024-03-20T09:17:40.560 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:14:45.523688+0000", 2024-03-20T09:17:40.560 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 174.96343523199999, 2024-03-20T09:17:40.560 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.12015424800000001, 2024-03-20T09:17:40.560 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:17:40.560 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:40.560 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:17:40.560 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.25690", 2024-03-20T09:17:40.560 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:17:40.560 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 110762 2024-03-20T09:17:40.561 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.561 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:17:40.561 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.561 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:17:40.561 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:45.523688+0000", 2024-03-20T09:17:40.561 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.561 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.561 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.561 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:17:40.561 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:45.523688+0000", 2024-03-20T09:17:40.561 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.561 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.562 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.562 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:17:40.562 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:45.523689+0000", 2024-03-20T09:17:40.562 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6440000000000001e-06 2024-03-20T09:17:40.562 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.562 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.562 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:17:40.562 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:45.523732+0000", 2024-03-20T09:17:40.562 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.2722000000000002e-05 2024-03-20T09:17:40.562 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.562 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.562 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:17:40.563 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:45.523744+0000", 2024-03-20T09:17:40.563 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1909999999999999e-05 2024-03-20T09:17:40.563 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.563 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.563 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:17:40.563 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:45.523753+0000", 2024-03-20T09:17:40.563 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.8109999999999992e-06 2024-03-20T09:17:40.563 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.563 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.563 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:17:40.563 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:45.523803+0000", 2024-03-20T09:17:40.563 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0216999999999997e-05 2024-03-20T09:17:40.563 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.563 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.563 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:17:40.564 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:45.523915+0000", 2024-03-20T09:17:40.564 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000111974 2024-03-20T09:17:40.564 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.564 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.564 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,10", 2024-03-20T09:17:40.564 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:45.524016+0000", 2024-03-20T09:17:40.564 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000100818 2024-03-20T09:17:40.564 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.564 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.564 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:17:40.564 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:45.526633+0000", 2024-03-20T09:17:40.564 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0026170389999999998 2024-03-20T09:17:40.564 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.564 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.564 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:40.564 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:45.526866+0000", 2024-03-20T09:17:40.565 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000233074 2024-03-20T09:17:40.565 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.565 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.565 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:40.565 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:45.643773+0000", 2024-03-20T09:17:40.565 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.116906858 2024-03-20T09:17:40.565 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.565 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.565 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:17:40.565 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:45.643810+0000", 2024-03-20T09:17:40.565 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6942e-05 2024-03-20T09:17:40.565 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.565 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.565 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:17:40.565 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:45.643842+0000", 2024-03-20T09:17:40.565 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.2239000000000001e-05 2024-03-20T09:17:40.565 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.566 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:17:40.566 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.566 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.566 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.566 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.25690.0:125863 4.a 4:50987ddb:::benchmark_data_smithi122_145698_object4944:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e557)", 2024-03-20T09:17:40.566 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:15:17.247957+0000", 2024-03-20T09:17:40.566 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 143.23916572900001, 2024-03-20T09:17:40.566 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.091080502999999993, 2024-03-20T09:17:40.566 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:17:40.566 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:40.566 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:17:40.566 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.25690", 2024-03-20T09:17:40.566 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:17:40.566 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 125863 2024-03-20T09:17:40.567 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.567 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:17:40.567 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.567 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:17:40.567 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:17.247957+0000", 2024-03-20T09:17:40.567 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.567 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.567 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.567 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:17:40.567 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:17.247957+0000", 2024-03-20T09:17:40.567 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.567 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.567 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.568 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:17:40.568 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:17.247958+0000", 2024-03-20T09:17:40.568 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1990000000000001e-06 2024-03-20T09:17:40.568 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.568 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.568 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:17:40.568 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:17.247963+0000", 2024-03-20T09:17:40.568 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5900000000000001e-06 2024-03-20T09:17:40.568 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.568 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.568 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:17:40.568 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:17.247974+0000", 2024-03-20T09:17:40.568 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0822000000000001e-05 2024-03-20T09:17:40.568 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.568 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.569 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:17:40.569 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:17.247981+0000", 2024-03-20T09:17:40.569 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.8099999999999998e-06 2024-03-20T09:17:40.569 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.569 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.569 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:17:40.569 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:17.248037+0000", 2024-03-20T09:17:40.569 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5852e-05 2024-03-20T09:17:40.569 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.569 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.569 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:17:40.569 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:17.248120+0000", 2024-03-20T09:17:40.569 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.2305000000000006e-05 2024-03-20T09:17:40.569 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.569 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.570 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,10", 2024-03-20T09:17:40.570 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:17.248195+0000", 2024-03-20T09:17:40.570 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.5334999999999996e-05 2024-03-20T09:17:40.570 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.570 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.570 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:17:40.570 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:17.250972+0000", 2024-03-20T09:17:40.570 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0027774369999999998 2024-03-20T09:17:40.570 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.570 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.570 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:40.570 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:17.251302+0000", 2024-03-20T09:17:40.570 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00032991100000000001 2024-03-20T09:17:40.571 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.571 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.571 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:40.571 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:17.338906+0000", 2024-03-20T09:17:40.571 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.087603613999999996 2024-03-20T09:17:40.571 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.571 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.571 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:17:40.571 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:17.338977+0000", 2024-03-20T09:17:40.571 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.1223999999999994e-05 2024-03-20T09:17:40.572 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.572 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.572 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:17:40.572 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:17.339038+0000", 2024-03-20T09:17:40.572 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0404e-05 2024-03-20T09:17:40.572 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.572 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:17:40.572 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.572 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.572 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.572 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.25690.0:125979 4.a 4:5b2f4de1:::benchmark_data_smithi122_145698_object5060:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e557)", 2024-03-20T09:17:40.572 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:15:17.682650+0000", 2024-03-20T09:17:40.572 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 142.80447290199999, 2024-03-20T09:17:40.572 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.069011876999999999, 2024-03-20T09:17:40.573 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:17:40.573 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:40.573 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:17:40.573 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.25690", 2024-03-20T09:17:40.573 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:17:40.573 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 125979 2024-03-20T09:17:40.573 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.573 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:17:40.573 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.573 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:17:40.573 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:17.682650+0000", 2024-03-20T09:17:40.573 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.573 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.573 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.574 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:17:40.574 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:17.682650+0000", 2024-03-20T09:17:40.574 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.574 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.574 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.574 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:17:40.574 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:17.682651+0000", 2024-03-20T09:17:40.574 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.483e-06 2024-03-20T09:17:40.574 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.574 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.574 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:17:40.574 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:17.682658+0000", 2024-03-20T09:17:40.574 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2879999999999998e-06 2024-03-20T09:17:40.575 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.575 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.575 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:17:40.575 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:17.682672+0000", 2024-03-20T09:17:40.575 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4781e-05 2024-03-20T09:17:40.575 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.575 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.575 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:17:40.575 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:17.682679+0000", 2024-03-20T09:17:40.575 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.0049999999999998e-06 2024-03-20T09:17:40.575 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.575 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.575 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:17:40.575 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:17.682759+0000", 2024-03-20T09:17:40.576 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.9919999999999994e-05 2024-03-20T09:17:40.576 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.576 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.576 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:17:40.576 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:17.682859+0000", 2024-03-20T09:17:40.576 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.9230999999999999e-05 2024-03-20T09:17:40.576 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.576 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.576 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,10", 2024-03-20T09:17:40.576 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:17.682940+0000", 2024-03-20T09:17:40.576 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.1323999999999996e-05 2024-03-20T09:17:40.576 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.576 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.576 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:17:40.576 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:17.685955+0000", 2024-03-20T09:17:40.577 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0030148979999999998 2024-03-20T09:17:40.577 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.577 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.577 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:40.577 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:17.686808+0000", 2024-03-20T09:17:40.577 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00085296299999999999 2024-03-20T09:17:40.577 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.577 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.577 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:40.577 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:17.751554+0000", 2024-03-20T09:17:40.577 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.064746182999999999 2024-03-20T09:17:40.577 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.577 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.577 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:17:40.578 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:17.751611+0000", 2024-03-20T09:17:40.578 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6650000000000001e-05 2024-03-20T09:17:40.578 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.578 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.578 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:17:40.578 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:17.751662+0000", 2024-03-20T09:17:40.578 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1150999999999999e-05 2024-03-20T09:17:40.578 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.578 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:17:40.578 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.578 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.578 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.578 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.25690.0:140184 4.a 4:528495aa:::benchmark_data_smithi122_145698_object19265:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e564)", 2024-03-20T09:17:40.579 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:15:41.406183+0000", 2024-03-20T09:17:40.579 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 119.080939341, 2024-03-20T09:17:40.579 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.072787195999999998, 2024-03-20T09:17:40.579 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:17:40.579 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:40.579 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:17:40.579 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.25690", 2024-03-20T09:17:40.579 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:17:40.579 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 140184 2024-03-20T09:17:40.579 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.579 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:17:40.580 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.580 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:17:40.580 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:41.406183+0000", 2024-03-20T09:17:40.580 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.580 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.580 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.580 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:17:40.580 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:41.406183+0000", 2024-03-20T09:17:40.580 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.581 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.581 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.581 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:17:40.581 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:41.406185+0000", 2024-03-20T09:17:40.581 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3340000000000001e-06 2024-03-20T09:17:40.581 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.581 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.581 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:17:40.581 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:41.406189+0000", 2024-03-20T09:17:40.582 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6949999999999999e-06 2024-03-20T09:17:40.582 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.582 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.582 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:17:40.582 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:41.406199+0000", 2024-03-20T09:17:40.582 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.7119999999999995e-06 2024-03-20T09:17:40.582 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.582 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.583 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:17:40.583 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:41.406204+0000", 2024-03-20T09:17:40.583 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3090000000000003e-06 2024-03-20T09:17:40.583 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.583 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.583 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:17:40.583 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:41.406256+0000", 2024-03-20T09:17:40.583 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1858999999999999e-05 2024-03-20T09:17:40.584 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.584 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.584 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:17:40.584 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:41.406398+0000", 2024-03-20T09:17:40.584 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000142025 2024-03-20T09:17:40.584 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.584 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.584 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,10", 2024-03-20T09:17:40.585 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:41.406519+0000", 2024-03-20T09:17:40.585 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012042 2024-03-20T09:17:40.585 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.585 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.585 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:40.585 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:41.409789+0000", 2024-03-20T09:17:40.585 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0032706689999999999 2024-03-20T09:17:40.585 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.585 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.586 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:17:40.586 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:41.409883+0000", 2024-03-20T09:17:40.586 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.3900000000000006e-05 2024-03-20T09:17:40.586 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.586 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.586 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:40.586 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:41.478794+0000", 2024-03-20T09:17:40.586 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.06891042 2024-03-20T09:17:40.587 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.587 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.587 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:17:40.587 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:41.478917+0000", 2024-03-20T09:17:40.587 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000123538 2024-03-20T09:17:40.587 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.587 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.587 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:17:40.588 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:41.478971+0000", 2024-03-20T09:17:40.588 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3315e-05 2024-03-20T09:17:40.588 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.588 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:17:40.588 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.588 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.588 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.588 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.25690.0:144845 4.a 4:5bc4cea4:::benchmark_data_smithi122_145698_object23926:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e566)", 2024-03-20T09:17:40.588 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:15:46.733630+0000", 2024-03-20T09:17:40.589 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 113.753492478, 2024-03-20T09:17:40.589 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.057307276999999997, 2024-03-20T09:17:40.589 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:17:40.589 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:40.589 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:17:40.589 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.25690", 2024-03-20T09:17:40.589 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:17:40.589 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 144845 2024-03-20T09:17:40.589 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.590 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:17:40.590 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.590 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:17:40.590 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.733630+0000", 2024-03-20T09:17:40.590 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.590 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.590 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.590 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:17:40.591 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.733630+0000", 2024-03-20T09:17:40.591 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.591 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.591 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.591 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:17:40.591 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.733632+0000", 2024-03-20T09:17:40.591 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.378e-06 2024-03-20T09:17:40.591 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.591 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.592 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:17:40.592 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.733637+0000", 2024-03-20T09:17:40.592 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6160000000000001e-06 2024-03-20T09:17:40.592 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.592 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.592 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:17:40.592 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.733653+0000", 2024-03-20T09:17:40.592 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5554e-05 2024-03-20T09:17:40.592 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.593 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.593 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:17:40.593 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.733666+0000", 2024-03-20T09:17:40.593 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2768999999999999e-05 2024-03-20T09:17:40.593 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.593 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.593 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:17:40.593 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.733718+0000", 2024-03-20T09:17:40.593 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1984000000000002e-05 2024-03-20T09:17:40.593 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.594 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.594 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:17:40.594 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.733847+0000", 2024-03-20T09:17:40.594 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012902399999999999 2024-03-20T09:17:40.594 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.594 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.594 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,10", 2024-03-20T09:17:40.594 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.733960+0000", 2024-03-20T09:17:40.594 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00011346 2024-03-20T09:17:40.594 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.594 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.594 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:40.595 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.740960+0000", 2024-03-20T09:17:40.595 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0069997130000000003 2024-03-20T09:17:40.595 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.595 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.595 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:17:40.595 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.744577+0000", 2024-03-20T09:17:40.595 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0036169240000000001 2024-03-20T09:17:40.595 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.595 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.595 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:40.595 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.790884+0000", 2024-03-20T09:17:40.595 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.046307256999999998 2024-03-20T09:17:40.595 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.595 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.595 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:17:40.596 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.790914+0000", 2024-03-20T09:17:40.596 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9754e-05 2024-03-20T09:17:40.596 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.596 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.596 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:17:40.596 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.790938+0000", 2024-03-20T09:17:40.596 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3844000000000001e-05 2024-03-20T09:17:40.596 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.596 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:17:40.596 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.596 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.596 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.596 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.25690.0:144847 4.a 4:52fb51a8:::benchmark_data_smithi122_145698_object23928:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e566)", 2024-03-20T09:17:40.597 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:15:46.737299+0000", 2024-03-20T09:17:40.597 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 113.749824203, 2024-03-20T09:17:40.597 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.066702358000000003, 2024-03-20T09:17:40.597 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:17:40.597 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:40.597 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:17:40.597 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.25690", 2024-03-20T09:17:40.597 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:17:40.597 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 144847 2024-03-20T09:17:40.597 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.597 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:17:40.597 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.598 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:17:40.598 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.737299+0000", 2024-03-20T09:17:40.598 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.598 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.598 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.598 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:17:40.598 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.737299+0000", 2024-03-20T09:17:40.598 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.598 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.598 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.598 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:17:40.599 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.737300+0000", 2024-03-20T09:17:40.599 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0559999999999999e-06 2024-03-20T09:17:40.599 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.599 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.599 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:17:40.599 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.737306+0000", 2024-03-20T09:17:40.599 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.5409999999999997e-06 2024-03-20T09:17:40.599 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.599 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.599 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:17:40.599 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.737316+0000", 2024-03-20T09:17:40.599 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.961e-06 2024-03-20T09:17:40.600 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.600 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.600 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:17:40.600 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.737325+0000", 2024-03-20T09:17:40.600 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.8540000000000003e-06 2024-03-20T09:17:40.600 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.600 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.600 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:17:40.600 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.737996+0000", 2024-03-20T09:17:40.600 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00067138500000000004 2024-03-20T09:17:40.600 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.600 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.600 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:17:40.600 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.738101+0000", 2024-03-20T09:17:40.600 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000104357 2024-03-20T09:17:40.601 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.601 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.601 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,10", 2024-03-20T09:17:40.601 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.738173+0000", 2024-03-20T09:17:40.601 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.2386999999999997e-05 2024-03-20T09:17:40.601 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.601 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.601 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:17:40.601 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.744583+0000", 2024-03-20T09:17:40.601 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.006410336 2024-03-20T09:17:40.601 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.601 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.601 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:40.601 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.746323+0000", 2024-03-20T09:17:40.601 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0017395760000000001 2024-03-20T09:17:40.601 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.602 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.602 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:40.602 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.803886+0000", 2024-03-20T09:17:40.602 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.057563357000000002 2024-03-20T09:17:40.602 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.602 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.602 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:17:40.602 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.803948+0000", 2024-03-20T09:17:40.602 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1212999999999998e-05 2024-03-20T09:17:40.602 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.602 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.602 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:17:40.602 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.804001+0000", 2024-03-20T09:17:40.603 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3334999999999997e-05 2024-03-20T09:17:40.603 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.603 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:17:40.603 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.603 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.603 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.603 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.25690.0:144848 4.a 4:5acadd92:::benchmark_data_smithi122_145698_object23929:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e566)", 2024-03-20T09:17:40.603 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:15:46.737342+0000", 2024-03-20T09:17:40.603 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 113.749780957, 2024-03-20T09:17:40.603 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.073776833, 2024-03-20T09:17:40.603 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:17:40.603 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:40.603 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:17:40.603 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.25690", 2024-03-20T09:17:40.603 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:17:40.604 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 144848 2024-03-20T09:17:40.604 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.604 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:17:40.604 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.604 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:17:40.604 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.737342+0000", 2024-03-20T09:17:40.604 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.604 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.604 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.604 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:17:40.604 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.737342+0000", 2024-03-20T09:17:40.604 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.605 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.605 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.605 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:17:40.605 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.737342+0000", 2024-03-20T09:17:40.605 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.13e-07 2024-03-20T09:17:40.605 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.605 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.605 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:17:40.605 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.737345+0000", 2024-03-20T09:17:40.605 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5790000000000002e-06 2024-03-20T09:17:40.605 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.605 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.605 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:17:40.605 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.737356+0000", 2024-03-20T09:17:40.605 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0607e-05 2024-03-20T09:17:40.606 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.606 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.606 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:17:40.606 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.737362+0000", 2024-03-20T09:17:40.606 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0950000000000001e-06 2024-03-20T09:17:40.606 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.606 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.606 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:17:40.606 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.740984+0000", 2024-03-20T09:17:40.606 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0036223679999999999 2024-03-20T09:17:40.606 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.606 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.606 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:17:40.607 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.741064+0000", 2024-03-20T09:17:40.607 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.9653000000000005e-05 2024-03-20T09:17:40.607 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.607 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.607 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,10", 2024-03-20T09:17:40.607 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.741139+0000", 2024-03-20T09:17:40.607 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.4956999999999994e-05 2024-03-20T09:17:40.607 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.607 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.607 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:17:40.607 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.744759+0000", 2024-03-20T09:17:40.607 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0036207639999999998 2024-03-20T09:17:40.607 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.608 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.608 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:40.608 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.746361+0000", 2024-03-20T09:17:40.608 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00160125 2024-03-20T09:17:40.608 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.608 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.608 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:40.608 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.811052+0000", 2024-03-20T09:17:40.608 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.064691528999999998 2024-03-20T09:17:40.608 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.608 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.608 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:17:40.608 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.811085+0000", 2024-03-20T09:17:40.608 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.2530000000000002e-05 2024-03-20T09:17:40.608 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.609 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.609 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:17:40.609 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:46.811119+0000", 2024-03-20T09:17:40.609 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.3988000000000001e-05 2024-03-20T09:17:40.609 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.609 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:17:40.609 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.609 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.609 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.609 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.25690.0:148010 4.a 4:5417085b:::benchmark_data_smithi122_145698_object27091:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e567)", 2024-03-20T09:17:40.609 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:15:50.726587+0000", 2024-03-20T09:17:40.609 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 109.760535812, 2024-03-20T09:17:40.609 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.065542409999999995, 2024-03-20T09:17:40.610 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:17:40.610 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:40.610 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:17:40.610 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.25690", 2024-03-20T09:17:40.610 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:17:40.610 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 148010 2024-03-20T09:17:40.610 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.610 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:17:40.610 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.610 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:17:40.610 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:50.726587+0000", 2024-03-20T09:17:40.610 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.610 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.611 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.611 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:17:40.611 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:50.726587+0000", 2024-03-20T09:17:40.611 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.611 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.611 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.611 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:17:40.611 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:50.726588+0000", 2024-03-20T09:17:40.611 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3120000000000001e-06 2024-03-20T09:17:40.611 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.611 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.611 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:17:40.611 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:50.726593+0000", 2024-03-20T09:17:40.611 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.7079999999999996e-06 2024-03-20T09:17:40.611 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.612 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.612 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:17:40.612 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:50.726603+0000", 2024-03-20T09:17:40.612 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0049e-05 2024-03-20T09:17:40.612 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.612 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.612 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:17:40.612 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:50.726609+0000", 2024-03-20T09:17:40.612 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4239999999999998e-06 2024-03-20T09:17:40.612 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.612 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.612 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:17:40.612 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:50.726666+0000", 2024-03-20T09:17:40.612 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6100999999999997e-05 2024-03-20T09:17:40.612 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.613 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.613 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:17:40.613 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:50.726798+0000", 2024-03-20T09:17:40.613 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000132242 2024-03-20T09:17:40.613 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.613 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.613 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,10", 2024-03-20T09:17:40.613 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:50.726932+0000", 2024-03-20T09:17:40.613 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013411 2024-03-20T09:17:40.613 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.613 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.614 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:17:40.614 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:50.731202+0000", 2024-03-20T09:17:40.614 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0042705180000000001 2024-03-20T09:17:40.614 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.614 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.614 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:40.614 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:50.731966+0000", 2024-03-20T09:17:40.614 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00076390699999999998 2024-03-20T09:17:40.614 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.614 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.615 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:40.615 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:50.792042+0000", 2024-03-20T09:17:40.615 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.060075438000000002 2024-03-20T09:17:40.615 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.615 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.615 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:17:40.615 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:50.792083+0000", 2024-03-20T09:17:40.615 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.1560999999999997e-05 2024-03-20T09:17:40.615 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.615 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.615 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:17:40.615 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:50.792129+0000", 2024-03-20T09:17:40.615 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.604e-05 2024-03-20T09:17:40.615 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.615 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:17:40.616 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.616 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.616 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.616 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.25690.0:148020 4.a 4:5c8511ca:::benchmark_data_smithi122_145698_object27101:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e567)", 2024-03-20T09:17:40.616 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:15:50.738146+0000", 2024-03-20T09:17:40.616 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 109.748976673, 2024-03-20T09:17:40.616 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.054423449999999998, 2024-03-20T09:17:40.616 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:17:40.616 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:40.616 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:17:40.616 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.25690", 2024-03-20T09:17:40.616 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:17:40.616 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 148020 2024-03-20T09:17:40.616 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.617 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:17:40.617 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.617 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:17:40.617 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:50.738146+0000", 2024-03-20T09:17:40.617 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.617 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.617 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.617 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:17:40.617 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:50.738146+0000", 2024-03-20T09:17:40.617 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.617 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.617 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.617 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:17:40.617 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:50.738147+0000", 2024-03-20T09:17:40.617 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4899999999999995e-07 2024-03-20T09:17:40.618 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.618 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.618 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:17:40.618 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:50.738150+0000", 2024-03-20T09:17:40.618 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.3830000000000001e-06 2024-03-20T09:17:40.618 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.618 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.618 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:17:40.618 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:50.738161+0000", 2024-03-20T09:17:40.618 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0793e-05 2024-03-20T09:17:40.618 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.618 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.619 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:17:40.619 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:50.738167+0000", 2024-03-20T09:17:40.619 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.3740000000000003e-06 2024-03-20T09:17:40.619 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.619 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.619 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:17:40.619 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:50.742760+0000", 2024-03-20T09:17:40.619 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0045924349999999997 2024-03-20T09:17:40.619 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.619 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.619 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:17:40.619 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:50.742891+0000", 2024-03-20T09:17:40.619 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013080200000000001 2024-03-20T09:17:40.619 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.619 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.620 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,10", 2024-03-20T09:17:40.620 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:50.743014+0000", 2024-03-20T09:17:40.620 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012321000000000001 2024-03-20T09:17:40.620 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.620 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.620 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:40.620 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:50.746956+0000", 2024-03-20T09:17:40.620 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0039421650000000001 2024-03-20T09:17:40.620 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.620 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.620 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:17:40.620 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:50.748073+0000", 2024-03-20T09:17:40.621 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001117177 2024-03-20T09:17:40.621 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.621 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.621 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:40.621 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:50.792494+0000", 2024-03-20T09:17:40.621 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.044420482999999997 2024-03-20T09:17:40.621 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.621 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.621 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:17:40.621 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:50.792531+0000", 2024-03-20T09:17:40.621 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7595999999999997e-05 2024-03-20T09:17:40.622 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.622 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.622 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:17:40.622 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:15:50.792570+0000", 2024-03-20T09:17:40.622 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.8383000000000002e-05 2024-03-20T09:17:40.622 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.622 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:17:40.622 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.622 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.622 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.622 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.25690.0:197739 4.a 4:51585674:::benchmark_data_smithi122_145698_object76820:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e578)", 2024-03-20T09:17:40.622 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:16:52.921818+0000", 2024-03-20T09:17:40.622 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 47.565304671, 2024-03-20T09:17:40.623 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.061530990000000001, 2024-03-20T09:17:40.623 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:17:40.623 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:40.623 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:17:40.623 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.25690", 2024-03-20T09:17:40.623 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:17:40.623 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 197739 2024-03-20T09:17:40.623 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.623 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:17:40.623 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.623 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:17:40.623 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.921818+0000", 2024-03-20T09:17:40.623 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.623 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.623 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.624 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:17:40.624 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.921818+0000", 2024-03-20T09:17:40.624 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.624 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.624 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.624 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:17:40.624 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.921819+0000", 2024-03-20T09:17:40.624 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.164e-06 2024-03-20T09:17:40.624 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.624 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.624 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:17:40.624 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.921824+0000", 2024-03-20T09:17:40.624 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8069999999999999e-06 2024-03-20T09:17:40.624 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.625 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.625 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:17:40.625 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.921836+0000", 2024-03-20T09:17:40.625 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1773999999999999e-05 2024-03-20T09:17:40.625 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.625 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.625 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:17:40.625 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.921845+0000", 2024-03-20T09:17:40.625 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.1959999999999997e-06 2024-03-20T09:17:40.625 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.625 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.625 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:17:40.625 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.921909+0000", 2024-03-20T09:17:40.625 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.3811000000000003e-05 2024-03-20T09:17:40.626 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.626 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.626 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:17:40.626 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.922038+0000", 2024-03-20T09:17:40.626 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012944599999999999 2024-03-20T09:17:40.626 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.626 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.626 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,10", 2024-03-20T09:17:40.626 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.922165+0000", 2024-03-20T09:17:40.626 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000126286 2024-03-20T09:17:40.627 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.627 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.627 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:17:40.627 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.939482+0000", 2024-03-20T09:17:40.627 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.017317744999999999 2024-03-20T09:17:40.627 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.627 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.627 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:40.627 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.939529+0000", 2024-03-20T09:17:40.627 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6644000000000003e-05 2024-03-20T09:17:40.627 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.627 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.627 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:40.627 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.983212+0000", 2024-03-20T09:17:40.628 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.043682795000000003 2024-03-20T09:17:40.628 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.628 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.628 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:17:40.628 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.983272+0000", 2024-03-20T09:17:40.628 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0146000000000003e-05 2024-03-20T09:17:40.628 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.628 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.628 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:17:40.628 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.983349+0000", 2024-03-20T09:17:40.628 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7175999999999994e-05 2024-03-20T09:17:40.629 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.629 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:17:40.629 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.629 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.629 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.629 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.25690.0:197742 4.a 4:57b7f2ac:::benchmark_data_smithi122_145698_object76823:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e578)", 2024-03-20T09:17:40.629 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:16:52.926105+0000", 2024-03-20T09:17:40.629 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 47.561017978000002, 2024-03-20T09:17:40.629 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.057385037999999999, 2024-03-20T09:17:40.629 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:17:40.629 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:40.629 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:17:40.629 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.25690", 2024-03-20T09:17:40.629 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:17:40.629 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 197742 2024-03-20T09:17:40.630 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.630 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:17:40.630 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.630 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:17:40.630 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.926105+0000", 2024-03-20T09:17:40.630 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.630 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.630 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.631 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:17:40.631 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.926105+0000", 2024-03-20T09:17:40.631 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.631 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.631 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.631 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:17:40.631 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.926107+0000", 2024-03-20T09:17:40.632 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.854e-06 2024-03-20T09:17:40.632 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.632 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.632 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:17:40.632 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.926114+0000", 2024-03-20T09:17:40.632 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.9070000000000001e-06 2024-03-20T09:17:40.632 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.632 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.632 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:17:40.632 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.926134+0000", 2024-03-20T09:17:40.632 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0152e-05 2024-03-20T09:17:40.632 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.632 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.633 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:17:40.633 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.926143+0000", 2024-03-20T09:17:40.633 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.3980000000000005e-06 2024-03-20T09:17:40.633 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.633 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.633 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:17:40.633 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.926213+0000", 2024-03-20T09:17:40.633 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.0309999999999996e-05 2024-03-20T09:17:40.633 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.633 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.633 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:17:40.633 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.926339+0000", 2024-03-20T09:17:40.633 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000125315 2024-03-20T09:17:40.633 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.633 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.633 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,10", 2024-03-20T09:17:40.634 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.926452+0000", 2024-03-20T09:17:40.634 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000113471 2024-03-20T09:17:40.634 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.634 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.634 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:40.634 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.939573+0000", 2024-03-20T09:17:40.634 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.013120562000000001 2024-03-20T09:17:40.634 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.634 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.634 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:17:40.634 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.939584+0000", 2024-03-20T09:17:40.634 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1674e-05 2024-03-20T09:17:40.634 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.635 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.635 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:40.635 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.983388+0000", 2024-03-20T09:17:40.635 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.043804002000000002 2024-03-20T09:17:40.635 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.635 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.635 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:17:40.635 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.983440+0000", 2024-03-20T09:17:40.635 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1223000000000001e-05 2024-03-20T09:17:40.635 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.635 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.635 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:17:40.635 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.983490+0000", 2024-03-20T09:17:40.635 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0170000000000002e-05 2024-03-20T09:17:40.636 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.636 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:17:40.636 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.636 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.636 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.636 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.25690.0:197739 4.a 4:51585674:::benchmark_data_smithi122_145698_object76820:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e578)", 2024-03-20T09:17:40.636 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:16:52.927610+0000", 2024-03-20T09:17:40.636 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 47.559512720999997, 2024-03-20T09:17:40.636 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.055677506000000002, 2024-03-20T09:17:40.636 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:17:40.636 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for ondisk", 2024-03-20T09:17:40.636 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:17:40.636 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.25690", 2024-03-20T09:17:40.636 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:17:40.637 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 197739 2024-03-20T09:17:40.637 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.637 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:17:40.637 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.637 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:17:40.637 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.927610+0000", 2024-03-20T09:17:40.637 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.637 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.637 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.637 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:17:40.637 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.927610+0000", 2024-03-20T09:17:40.637 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.637 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.638 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.638 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:17:40.638 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.927611+0000", 2024-03-20T09:17:40.638 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.7899999999999999e-07 2024-03-20T09:17:40.638 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.638 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.638 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:17:40.638 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.927613+0000", 2024-03-20T09:17:40.638 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6649999999999999e-06 2024-03-20T09:17:40.638 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.638 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.638 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:17:40.638 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.927624+0000", 2024-03-20T09:17:40.639 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0549e-05 2024-03-20T09:17:40.639 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.639 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.639 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:17:40.639 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.927630+0000", 2024-03-20T09:17:40.639 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.3670000000000002e-06 2024-03-20T09:17:40.639 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.639 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.639 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:17:40.639 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.930042+0000", 2024-03-20T09:17:40.639 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0024121809999999998 2024-03-20T09:17:40.639 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.639 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.639 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for ondisk", 2024-03-20T09:17:40.640 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.930114+0000", 2024-03-20T09:17:40.640 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.1892999999999998e-05 2024-03-20T09:17:40.640 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.640 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.640 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:17:40.640 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.983288+0000", 2024-03-20T09:17:40.640 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.053173372000000003 2024-03-20T09:17:40.640 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.640 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:17:40.640 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.640 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.640 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.640 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.25690.0:197742 4.a 4:57b7f2ac:::benchmark_data_smithi122_145698_object76823:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e578)", 2024-03-20T09:17:40.640 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:16:52.927639+0000", 2024-03-20T09:17:40.641 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 47.559483743000001, 2024-03-20T09:17:40.641 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.055809399000000003, 2024-03-20T09:17:40.641 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:17:40.641 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for ondisk", 2024-03-20T09:17:40.641 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:17:40.641 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.25690", 2024-03-20T09:17:40.641 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:17:40.641 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 197742 2024-03-20T09:17:40.641 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.641 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:17:40.641 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.641 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:17:40.641 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.927639+0000", 2024-03-20T09:17:40.641 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.642 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.642 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.642 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:17:40.642 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.927639+0000", 2024-03-20T09:17:40.642 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.642 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.642 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.642 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:17:40.642 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.927639+0000", 2024-03-20T09:17:40.642 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.1100000000000001e-07 2024-03-20T09:17:40.642 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.642 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.642 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:17:40.643 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.927642+0000", 2024-03-20T09:17:40.643 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5289999999999998e-06 2024-03-20T09:17:40.643 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.643 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.643 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:17:40.643 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.927648+0000", 2024-03-20T09:17:40.643 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2709999999999998e-06 2024-03-20T09:17:40.643 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.643 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.643 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:17:40.643 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.927654+0000", 2024-03-20T09:17:40.643 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4040000000000003e-06 2024-03-20T09:17:40.643 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.643 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.644 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:17:40.644 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.930131+0000", 2024-03-20T09:17:40.644 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0024778199999999999 2024-03-20T09:17:40.644 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.644 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.644 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for ondisk", 2024-03-20T09:17:40.644 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.930164+0000", 2024-03-20T09:17:40.644 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.2490999999999998e-05 2024-03-20T09:17:40.644 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.644 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.644 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:17:40.645 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.983448+0000", 2024-03-20T09:17:40.645 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.053284472999999999 2024-03-20T09:17:40.645 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.645 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:17:40.645 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.645 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.645 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.645 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.25690.0:197744 4.a 4:53ab3216:::benchmark_data_smithi122_145698_object76825:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e578)", 2024-03-20T09:17:40.645 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:16:52.927660+0000", 2024-03-20T09:17:40.645 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 47.559462728, 2024-03-20T09:17:40.645 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.055969155, 2024-03-20T09:17:40.645 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:17:40.646 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:40.646 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:17:40.646 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.25690", 2024-03-20T09:17:40.646 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:17:40.646 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 197744 2024-03-20T09:17:40.646 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.646 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:17:40.646 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.646 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:17:40.646 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.927660+0000", 2024-03-20T09:17:40.646 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.646 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.646 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.647 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:17:40.647 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.927660+0000", 2024-03-20T09:17:40.647 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.647 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.647 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.647 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:17:40.647 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.927661+0000", 2024-03-20T09:17:40.647 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3600000000000004e-07 2024-03-20T09:17:40.647 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.647 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.647 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:17:40.647 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.927663+0000", 2024-03-20T09:17:40.647 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.4839999999999998e-06 2024-03-20T09:17:40.648 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.648 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.648 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:17:40.648 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.927669+0000", 2024-03-20T09:17:40.648 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1029999999999998e-06 2024-03-20T09:17:40.648 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.648 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.648 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:17:40.648 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.927674+0000", 2024-03-20T09:17:40.648 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2780000000000001e-06 2024-03-20T09:17:40.648 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.648 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.648 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:17:40.648 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.930184+0000", 2024-03-20T09:17:40.649 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0025099829999999999 2024-03-20T09:17:40.649 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.649 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.649 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:17:40.649 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.930258+0000", 2024-03-20T09:17:40.649 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3761000000000001e-05 2024-03-20T09:17:40.649 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.649 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.649 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,10", 2024-03-20T09:17:40.649 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.930339+0000", 2024-03-20T09:17:40.649 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.0620999999999999e-05 2024-03-20T09:17:40.649 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.649 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.649 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:17:40.649 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.939589+0000", 2024-03-20T09:17:40.650 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0092503859999999993 2024-03-20T09:17:40.650 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.650 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.650 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:40.650 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.945398+0000", 2024-03-20T09:17:40.650 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0058083350000000004 2024-03-20T09:17:40.650 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.650 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.650 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:40.650 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.983538+0000", 2024-03-20T09:17:40.650 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.038140088000000003 2024-03-20T09:17:40.650 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.650 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.650 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:17:40.651 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.983589+0000", 2024-03-20T09:17:40.651 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1391000000000003e-05 2024-03-20T09:17:40.651 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.651 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.651 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:17:40.651 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.983629+0000", 2024-03-20T09:17:40.651 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.0188999999999998e-05 2024-03-20T09:17:40.651 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.651 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:17:40.651 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.651 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.651 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.651 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.25690.0:197753 4.a 4:5f2ed517:::benchmark_data_smithi122_145698_object76834:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e578)", 2024-03-20T09:17:40.651 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:16:52.931894+0000", 2024-03-20T09:17:40.651 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 47.555229042999997, 2024-03-20T09:17:40.652 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.062578059000000005, 2024-03-20T09:17:40.652 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:17:40.652 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:40.652 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:17:40.652 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.25690", 2024-03-20T09:17:40.652 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:17:40.652 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 197753 2024-03-20T09:17:40.652 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.652 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:17:40.652 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.652 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:17:40.652 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.931894+0000", 2024-03-20T09:17:40.652 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.652 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.653 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.653 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:17:40.653 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.931894+0000", 2024-03-20T09:17:40.653 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.653 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.653 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.653 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:17:40.653 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.931895+0000", 2024-03-20T09:17:40.653 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.263e-06 2024-03-20T09:17:40.653 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.653 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.653 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:17:40.653 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.931901+0000", 2024-03-20T09:17:40.653 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8459999999999996e-06 2024-03-20T09:17:40.653 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.654 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.654 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:17:40.654 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.931912+0000", 2024-03-20T09:17:40.654 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1094000000000001e-05 2024-03-20T09:17:40.654 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.654 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.654 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:17:40.654 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.931920+0000", 2024-03-20T09:17:40.654 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.6969999999999993e-06 2024-03-20T09:17:40.654 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.654 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.654 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:17:40.654 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.934808+0000", 2024-03-20T09:17:40.655 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0028885999999999998 2024-03-20T09:17:40.655 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.655 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.655 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:17:40.655 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.934930+0000", 2024-03-20T09:17:40.655 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000122259 2024-03-20T09:17:40.655 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.655 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.655 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,10", 2024-03-20T09:17:40.655 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.935056+0000", 2024-03-20T09:17:40.655 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012578399999999999 2024-03-20T09:17:40.655 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.655 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.655 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:17:40.655 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.939600+0000", 2024-03-20T09:17:40.656 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0045442039999999996 2024-03-20T09:17:40.656 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.656 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.656 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:40.656 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.945434+0000", 2024-03-20T09:17:40.656 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0058334370000000003 2024-03-20T09:17:40.656 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.656 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.656 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:40.656 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.994362+0000", 2024-03-20T09:17:40.656 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.048927960999999999 2024-03-20T09:17:40.656 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.656 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.656 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:17:40.656 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.994425+0000", 2024-03-20T09:17:40.657 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2921000000000003e-05 2024-03-20T09:17:40.657 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.657 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.657 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:17:40.657 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:52.994472+0000", 2024-03-20T09:17:40.657 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6993000000000002e-05 2024-03-20T09:17:40.657 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.657 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:17:40.657 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.657 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.657 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.657 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.25690.0:198677 4.a 4:568a5e87:::benchmark_data_smithi122_145698_object77758:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e578)", 2024-03-20T09:17:40.657 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:16:54.006509+0000", 2024-03-20T09:17:40.657 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 46.480613734999999, 2024-03-20T09:17:40.657 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.062092557999999999, 2024-03-20T09:17:40.658 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:17:40.658 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:40.658 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:17:40.658 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.25690", 2024-03-20T09:17:40.658 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:17:40.658 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 198677 2024-03-20T09:17:40.658 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.658 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:17:40.658 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.658 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:17:40.658 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:54.006509+0000", 2024-03-20T09:17:40.658 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.658 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.659 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.659 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:17:40.659 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:54.006509+0000", 2024-03-20T09:17:40.659 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.659 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.659 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.659 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:17:40.659 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:54.006510+0000", 2024-03-20T09:17:40.659 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.189e-06 2024-03-20T09:17:40.659 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.659 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.659 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:17:40.659 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:54.006516+0000", 2024-03-20T09:17:40.659 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3890000000000004e-06 2024-03-20T09:17:40.659 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.660 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.660 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:17:40.660 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:54.006527+0000", 2024-03-20T09:17:40.660 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1435e-05 2024-03-20T09:17:40.660 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.660 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.660 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:17:40.660 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:54.006536+0000", 2024-03-20T09:17:40.660 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.704e-06 2024-03-20T09:17:40.660 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.660 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.660 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:17:40.660 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:54.006626+0000", 2024-03-20T09:17:40.660 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.0297000000000006e-05 2024-03-20T09:17:40.660 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.660 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.661 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:17:40.661 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:54.006724+0000", 2024-03-20T09:17:40.661 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.8109000000000005e-05 2024-03-20T09:17:40.661 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.661 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.661 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,10", 2024-03-20T09:17:40.661 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:54.006836+0000", 2024-03-20T09:17:40.661 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000112179 2024-03-20T09:17:40.661 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.661 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.661 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:17:40.661 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:54.010415+0000", 2024-03-20T09:17:40.661 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0035790729999999999 2024-03-20T09:17:40.661 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.661 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.662 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:40.662 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:54.011496+0000", 2024-03-20T09:17:40.662 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00108051 2024-03-20T09:17:40.662 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.662 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.662 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:40.662 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:54.068487+0000", 2024-03-20T09:17:40.662 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.056991211999999999 2024-03-20T09:17:40.662 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.662 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.662 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:17:40.662 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:54.068549+0000", 2024-03-20T09:17:40.662 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1716000000000003e-05 2024-03-20T09:17:40.662 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.663 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.663 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:17:40.663 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:16:54.068602+0000", 2024-03-20T09:17:40.663 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2744999999999997e-05 2024-03-20T09:17:40.663 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.663 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:17:40.663 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.663 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.663 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.663 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.25690.0:210808 4.a 4:50ba7174:::benchmark_data_smithi122_145698_object89889:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e586)", 2024-03-20T09:17:40.663 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:17:13.808087+0000", 2024-03-20T09:17:40.663 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 26.679036, 2024-03-20T09:17:40.663 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.074541115000000005, 2024-03-20T09:17:40.664 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:17:40.664 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:40.664 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:17:40.664 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.25690", 2024-03-20T09:17:40.664 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:17:40.664 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 210808 2024-03-20T09:17:40.664 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.664 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:17:40.664 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.664 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:17:40.664 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:13.808087+0000", 2024-03-20T09:17:40.664 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.664 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.664 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.665 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:17:40.665 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:13.808087+0000", 2024-03-20T09:17:40.665 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.665 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.665 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.665 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:17:40.665 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:13.808088+0000", 2024-03-20T09:17:40.665 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.2299999999999999e-07 2024-03-20T09:17:40.665 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.665 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.665 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:17:40.665 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:13.808092+0000", 2024-03-20T09:17:40.665 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.1099999999999996e-06 2024-03-20T09:17:40.665 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.665 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.666 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:17:40.666 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:13.808101+0000", 2024-03-20T09:17:40.666 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.6870000000000006e-06 2024-03-20T09:17:40.666 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.666 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.666 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:17:40.666 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:13.808107+0000", 2024-03-20T09:17:40.666 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6110000000000001e-06 2024-03-20T09:17:40.666 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.666 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.666 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:17:40.666 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:13.808158+0000", 2024-03-20T09:17:40.666 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0550000000000002e-05 2024-03-20T09:17:40.667 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.667 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.667 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:17:40.667 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:13.808291+0000", 2024-03-20T09:17:40.667 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013353700000000001 2024-03-20T09:17:40.667 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.667 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.667 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,10", 2024-03-20T09:17:40.667 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:13.808416+0000", 2024-03-20T09:17:40.667 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000124976 2024-03-20T09:17:40.667 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.667 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.667 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:17:40.667 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:13.812395+0000", 2024-03-20T09:17:40.668 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0039792500000000001 2024-03-20T09:17:40.668 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.668 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.668 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:40.668 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:13.813497+0000", 2024-03-20T09:17:40.668 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001101964 2024-03-20T09:17:40.668 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.668 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.668 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:40.668 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:13.882506+0000", 2024-03-20T09:17:40.668 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.069008694999999995 2024-03-20T09:17:40.668 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.668 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.668 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:17:40.668 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:13.882573+0000", 2024-03-20T09:17:40.669 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.7187999999999994e-05 2024-03-20T09:17:40.669 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.669 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.669 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:17:40.669 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:13.882628+0000", 2024-03-20T09:17:40.669 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4623999999999997e-05 2024-03-20T09:17:40.669 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.669 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:17:40.669 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.669 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.669 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.669 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.25690.0:212438 4.a 4:52e5a8c6:::benchmark_data_smithi122_145698_object91519:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e587)", 2024-03-20T09:17:40.669 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:17:16.524273+0000", 2024-03-20T09:17:40.669 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 23.962849990999999, 2024-03-20T09:17:40.670 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.055373095999999997, 2024-03-20T09:17:40.670 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:17:40.670 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:17:40.670 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:17:40.670 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.25690", 2024-03-20T09:17:40.670 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:17:40.670 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 212438 2024-03-20T09:17:40.670 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.670 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:17:40.670 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.670 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:17:40.670 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:16.524273+0000", 2024-03-20T09:17:40.670 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.670 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.671 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.671 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:17:40.671 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:16.524273+0000", 2024-03-20T09:17:40.671 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:40.671 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.671 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.671 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:17:40.671 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:16.524274+0000", 2024-03-20T09:17:40.671 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2860000000000001e-06 2024-03-20T09:17:40.671 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.671 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.671 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:17:40.671 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:16.524279+0000", 2024-03-20T09:17:40.672 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8099999999999997e-06 2024-03-20T09:17:40.672 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.672 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.672 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:17:40.672 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:16.524289+0000", 2024-03-20T09:17:40.672 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0338e-05 2024-03-20T09:17:40.672 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.672 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.672 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:17:40.672 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:16.524295+0000", 2024-03-20T09:17:40.672 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6559999999999997e-06 2024-03-20T09:17:40.672 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.673 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.673 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:17:40.673 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:16.524347+0000", 2024-03-20T09:17:40.673 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1981000000000003e-05 2024-03-20T09:17:40.673 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.673 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.673 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:17:40.673 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:16.524469+0000", 2024-03-20T09:17:40.673 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000122395 2024-03-20T09:17:40.673 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.673 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.673 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,10", 2024-03-20T09:17:40.673 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:16.524599+0000", 2024-03-20T09:17:40.673 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012994699999999999 2024-03-20T09:17:40.674 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.674 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.674 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:40.674 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:16.529214+0000", 2024-03-20T09:17:40.674 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0046146809999999998 2024-03-20T09:17:40.674 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.674 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.674 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:17:40.674 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:16.529317+0000", 2024-03-20T09:17:40.674 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000103247 2024-03-20T09:17:40.674 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.674 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.674 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:17:40.674 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:16.551175+0000", 2024-03-20T09:17:40.675 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.021858312000000001 2024-03-20T09:17:40.675 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.675 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.675 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:17:40.675 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:16.551277+0000", 2024-03-20T09:17:40.675 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000101859 2024-03-20T09:17:40.675 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:40.675 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:40.675 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:17:40.675 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:16.579646+0000", 2024-03-20T09:17:40.675 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.028368583999999999 2024-03-20T09:17:40.675 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.675 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:17:40.675 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.675 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:40.675 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:17:40.676 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:17:42.129 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-20T09:17:42.130 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:17:42.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:42.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:17:42.237 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:17:42.260 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:17:42.270 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:42.299 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:17:42.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:42.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:17:42.405 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:17:42.441 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:42.473 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:17:42.473 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:17:42.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:42.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:17:42.580 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:17:42.610 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:42.809 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:17:42.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:42.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:17:42.923 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:17:42.959 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:43.099 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:17:43.111 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.5 to 1 (10000) 2024-03-20T09:17:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:43 smithi115 ceph-mon[72476]: pgmap v1429: 47 pgs: 47 active+clean; 679 MiB data, 5.8 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 854 op/s 2024-03-20T09:17:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:43 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3844706342' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:17:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:43 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:17:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:43 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2631381680' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:17:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:43 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:17:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:43 smithi122 ceph-mon[73495]: pgmap v1429: 47 pgs: 47 active+clean; 679 MiB data, 5.8 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 854 op/s 2024-03-20T09:17:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:43 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3844706342' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:17:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:43 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:17:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:43 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2631381680' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:17:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:43 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:17:43.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:43 smithi026 ceph-mon[68309]: pgmap v1429: 47 pgs: 47 active+clean; 679 MiB data, 5.8 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 854 op/s 2024-03-20T09:17:43.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:43 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3844706342' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:17:43.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:43 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:17:43.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:43 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2631381680' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:17:43.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:43 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:17:43.926 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:17:43.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:43.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:17:44.031 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:17:44.067 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:44.111 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:17:44.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:44.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:17:44.219 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:17:44.272 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:44.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:44 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:17:44.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:44 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:17:44.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:44 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:17:44.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:44 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-20T09:17:44.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:44 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:17:44.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:44 smithi115 ceph-mon[72476]: osdmap e597: 12 total, 12 up, 10 in 2024-03-20T09:17:44.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:44 smithi115 ceph-mon[72476]: pgmap v1431: 47 pgs: 47 active+clean; 596 MiB data, 5.5 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 845 op/s 2024-03-20T09:17:44.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:44 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:17:44.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:44 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:17:44.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:44 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:17:44.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:44 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-20T09:17:44.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:44 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:17:44.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:44 smithi122 ceph-mon[73495]: osdmap e597: 12 total, 12 up, 10 in 2024-03-20T09:17:44.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:44 smithi122 ceph-mon[73495]: pgmap v1431: 47 pgs: 47 active+clean; 596 MiB data, 5.5 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 845 op/s 2024-03-20T09:17:44.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:44 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:17:44.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:44 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:17:44.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:44 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:17:44.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:44 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-20T09:17:44.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:44 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:17:44.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:44 smithi026 ceph-mon[68309]: osdmap e597: 12 total, 12 up, 10 in 2024-03-20T09:17:44.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:44 smithi026 ceph-mon[68309]: pgmap v1431: 47 pgs: 47 active+clean; 596 MiB data, 5.5 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 845 op/s 2024-03-20T09:17:45.128 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:17:45.129 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [ 2024-03-20T09:17:45.129 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:45.129 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25690.0:235769 4.0 e597/522 4:0adf835d:::benchmark_data_smithi122_145698_object114850:head v 597'14551, mlcod=597'14549)", 2024-03-20T09:17:45.129 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:17:45.125899+0000", 2024-03-20T09:17:45.129 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.0010996770000000001, 2024-03-20T09:17:45.129 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001123289, 2024-03-20T09:17:45.129 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:17:45.129 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:17:45.129 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:17:45.129 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:45.129 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:17:45.129 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:45.125899+0000", 2024-03-20T09:17:45.130 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:45.130 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:45.130 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:45.130 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:17:45.130 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:45.125899+0000", 2024-03-20T09:17:45.130 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:45.130 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:45.130 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:45.130 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:17:45.130 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:45.125900+0000", 2024-03-20T09:17:45.130 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.077e-06 2024-03-20T09:17:45.130 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:45.130 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:45.130 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:17:45.131 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:45.125907+0000", 2024-03-20T09:17:45.131 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0750000000000003e-06 2024-03-20T09:17:45.131 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:45.131 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:45.131 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:17:45.131 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:45.125918+0000", 2024-03-20T09:17:45.131 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1977999999999999e-05 2024-03-20T09:17:45.131 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:45.131 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:45.131 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:17:45.131 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:45.125925+0000", 2024-03-20T09:17:45.131 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.8290000000000002e-06 2024-03-20T09:17:45.131 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:45.131 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:45.132 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:17:45.132 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:45.125979+0000", 2024-03-20T09:17:45.132 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4153000000000001e-05 2024-03-20T09:17:45.132 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:45.132 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:45.132 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:17:45.132 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:45.126000+0000", 2024-03-20T09:17:45.132 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0945e-05 2024-03-20T09:17:45.132 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:45.132 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:17:45.132 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:45.132 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:45.132 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:45.132 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25690.0:235763 4.3 e597/520 4:cbade1d3:::benchmark_data_smithi122_145698_object114844:head v 597'15152, mlcod=597'15150)", 2024-03-20T09:17:45.133 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:17:45.126265+0000", 2024-03-20T09:17:45.133 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.00073382500000000002, 2024-03-20T09:17:45.133 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00079723900000000004, 2024-03-20T09:17:45.133 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:17:45.133 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:17:45.133 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:17:45.133 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:45.133 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:17:45.133 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:45.126265+0000", 2024-03-20T09:17:45.133 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:45.133 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:45.133 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:45.133 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:17:45.133 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:45.126265+0000", 2024-03-20T09:17:45.134 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:45.134 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:45.134 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:45.134 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:17:45.134 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:45.126266+0000", 2024-03-20T09:17:45.134 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.8999999999999995e-07 2024-03-20T09:17:45.134 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:45.134 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:45.134 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:17:45.134 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:45.126273+0000", 2024-03-20T09:17:45.134 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.2540000000000003e-06 2024-03-20T09:17:45.134 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:45.134 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:45.134 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:17:45.135 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:45.126289+0000", 2024-03-20T09:17:45.135 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5376e-05 2024-03-20T09:17:45.135 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:45.135 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:45.135 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:17:45.135 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:45.126297+0000", 2024-03-20T09:17:45.135 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7670000000000003e-06 2024-03-20T09:17:45.135 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:45.135 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:45.135 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:17:45.135 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:45.126420+0000", 2024-03-20T09:17:45.135 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000123201 2024-03-20T09:17:45.135 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:45.135 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:45.135 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:17:45.136 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:45.126439+0000", 2024-03-20T09:17:45.136 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9383000000000001e-05 2024-03-20T09:17:45.136 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:45.136 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:17:45.136 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:45.136 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:45.136 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:45.136 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25690.0:235767 4.8 e597/520 4:156d896b:::benchmark_data_smithi122_145698_object114848:head v 597'14621, mlcod=597'14619)", 2024-03-20T09:17:45.136 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:17:45.126318+0000", 2024-03-20T09:17:45.136 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.000680885, 2024-03-20T09:17:45.136 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000779044, 2024-03-20T09:17:45.136 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:17:45.136 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:17:45.136 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:17:45.136 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:45.137 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:17:45.137 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:45.126318+0000", 2024-03-20T09:17:45.137 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:45.137 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:45.137 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:45.137 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:17:45.137 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:45.126318+0000", 2024-03-20T09:17:45.137 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:17:45.137 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:45.137 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:45.137 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:17:45.137 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:45.126319+0000", 2024-03-20T09:17:45.137 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6899999999999998e-07 2024-03-20T09:17:45.137 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:45.137 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:45.138 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:17:45.138 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:45.126323+0000", 2024-03-20T09:17:45.138 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4889999999999997e-06 2024-03-20T09:17:45.138 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:45.138 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:45.138 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:17:45.138 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:45.126330+0000", 2024-03-20T09:17:45.138 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.0160000000000003e-06 2024-03-20T09:17:45.138 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:45.138 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:45.138 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:17:45.138 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:45.126335+0000", 2024-03-20T09:17:45.138 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3489999999999999e-06 2024-03-20T09:17:45.138 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:45.139 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:45.139 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:17:45.139 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:45.126379+0000", 2024-03-20T09:17:45.139 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.3745000000000002e-05 2024-03-20T09:17:45.139 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:17:45.139 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:17:45.139 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:17:45.139 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:17:45.126401+0000", 2024-03-20T09:17:45.139 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1861e-05 2024-03-20T09:17:45.139 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:45.139 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:17:45.139 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:45.139 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:17:45.139 INFO:teuthology.orchestra.run.smithi115.stdout: ], 2024-03-20T09:17:45.140 INFO:teuthology.orchestra.run.smithi115.stdout: "num_ops": 3 2024-03-20T09:17:45.140 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:17:45.256 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:17:45.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:45.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:17:45.282 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 6 1 2024-03-20T09:17:45.358 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:45 smithi026 ceph-mon[68309]: 1.2 scrub starts 2024-03-20T09:17:45.358 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:45 smithi026 ceph-mon[68309]: 1.2 scrub ok 2024-03-20T09:17:45.369 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:17:45.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:45 smithi115 ceph-mon[72476]: 1.2 scrub starts 2024-03-20T09:17:45.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:45 smithi115 ceph-mon[72476]: 1.2 scrub ok 2024-03-20T09:17:45.400 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:45.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:45 smithi122 ceph-mon[73495]: 1.2 scrub starts 2024-03-20T09:17:45.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:45 smithi122 ceph-mon[73495]: 1.2 scrub ok 2024-03-20T09:17:45.460 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:17:45.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:45.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:17:45.565 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:17:45.595 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:46.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:46 smithi115 ceph-mon[72476]: 1.15 scrub starts 2024-03-20T09:17:46.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:46 smithi115 ceph-mon[72476]: 1.15 scrub ok 2024-03-20T09:17:46.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:46 smithi115 ceph-mon[72476]: 1.1d scrub starts 2024-03-20T09:17:46.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:46 smithi115 ceph-mon[72476]: 1.1d scrub ok 2024-03-20T09:17:46.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:46 smithi115 ceph-mon[72476]: pgmap v1432: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 447 MiB data, 5.2 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 846 op/s 2024-03-20T09:17:46.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:46 smithi122 ceph-mon[73495]: 1.15 scrub starts 2024-03-20T09:17:46.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:46 smithi122 ceph-mon[73495]: 1.15 scrub ok 2024-03-20T09:17:46.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:46 smithi122 ceph-mon[73495]: 1.1d scrub starts 2024-03-20T09:17:46.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:46 smithi122 ceph-mon[73495]: 1.1d scrub ok 2024-03-20T09:17:46.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:46 smithi122 ceph-mon[73495]: pgmap v1432: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 447 MiB data, 5.2 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 846 op/s 2024-03-20T09:17:46.452 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:17:46.453 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-20T09:17:46.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:46.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:17:46.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:46 smithi026 ceph-mon[68309]: 1.15 scrub starts 2024-03-20T09:17:46.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:46 smithi026 ceph-mon[68309]: 1.15 scrub ok 2024-03-20T09:17:46.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:46 smithi026 ceph-mon[68309]: 1.1d scrub starts 2024-03-20T09:17:46.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:46 smithi026 ceph-mon[68309]: 1.1d scrub ok 2024-03-20T09:17:46.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:46 smithi026 ceph-mon[68309]: pgmap v1432: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 447 MiB data, 5.2 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 846 op/s 2024-03-20T09:17:46.494 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:17:46.556 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:17:46.585 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:46.617 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:17:46.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:46.629 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:17:46.731 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:17:46.805 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:46.836 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:17:46.837 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:17:46.846 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:17:46.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:46.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:17:46.871 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:17:46.926 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:17:46.951 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:17:46.969 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:17:47.020 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:17:47.044 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:47.064 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:17:47.095 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:17:47.102 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:17:47.160 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:17:47.220 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:17:47.283 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:17:47.325 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:17:47.372 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:17:47.385 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:17:47.385 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:17:47.386 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:17:47.386 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:17:47.386 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:17:47.386 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:17:47.386 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:17:47.386 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:17:47.386 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:17:47.386 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:17:47.386 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:17:47.386 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:17:47.406 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:47 smithi026 ceph-mon[68309]: 1.c scrub starts 2024-03-20T09:17:47.406 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:47 smithi026 ceph-mon[68309]: 1.c scrub ok 2024-03-20T09:17:47.406 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:47 smithi026 ceph-mon[68309]: 1.11 scrub starts 2024-03-20T09:17:47.406 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:47 smithi026 ceph-mon[68309]: 1.11 scrub ok 2024-03-20T09:17:47.406 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:47 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4251563763' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:17:47.406 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:47 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:17:47.407 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:47 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:17:47.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:47 smithi115 ceph-mon[72476]: 1.c scrub starts 2024-03-20T09:17:47.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:47 smithi115 ceph-mon[72476]: 1.c scrub ok 2024-03-20T09:17:47.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:47 smithi115 ceph-mon[72476]: 1.11 scrub starts 2024-03-20T09:17:47.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:47 smithi115 ceph-mon[72476]: 1.11 scrub ok 2024-03-20T09:17:47.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:47 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4251563763' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:17:47.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:47 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:17:47.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:47 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:17:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:47 smithi122 ceph-mon[73495]: 1.c scrub starts 2024-03-20T09:17:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:47 smithi122 ceph-mon[73495]: 1.c scrub ok 2024-03-20T09:17:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:47 smithi122 ceph-mon[73495]: 1.11 scrub starts 2024-03-20T09:17:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:47 smithi122 ceph-mon[73495]: 1.11 scrub ok 2024-03-20T09:17:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:47 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4251563763' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:17:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:47 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:17:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:47 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:17:48.255 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:17:48.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:48.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:17:48.359 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:17:48.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:48 smithi115 ceph-mon[72476]: 1.e scrub starts 2024-03-20T09:17:48.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:48 smithi115 ceph-mon[72476]: 1.e scrub ok 2024-03-20T09:17:48.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:48 smithi115 ceph-mon[72476]: 4.2 scrub starts 2024-03-20T09:17:48.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:48 smithi115 ceph-mon[72476]: 4.2 scrub ok 2024-03-20T09:17:48.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:48 smithi115 ceph-mon[72476]: 1.17 scrub starts 2024-03-20T09:17:48.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:48 smithi115 ceph-mon[72476]: 1.17 scrub ok 2024-03-20T09:17:48.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:48 smithi115 ceph-mon[72476]: osdmap e598: 12 total, 12 up, 10 in 2024-03-20T09:17:48.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:48 smithi115 ceph-mon[72476]: 1.14 scrub starts 2024-03-20T09:17:48.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:48 smithi115 ceph-mon[72476]: 1.14 scrub ok 2024-03-20T09:17:48.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:48 smithi115 ceph-mon[72476]: 4.c scrub starts 2024-03-20T09:17:48.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:48 smithi115 ceph-mon[72476]: pgmap v1434: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 333 MiB data, 4.8 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.02k op/s 2024-03-20T09:17:48.438 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:48 smithi026 ceph-mon[68309]: 1.e scrub starts 2024-03-20T09:17:48.438 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:48 smithi026 ceph-mon[68309]: 1.e scrub ok 2024-03-20T09:17:48.439 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:48 smithi026 ceph-mon[68309]: 4.2 scrub starts 2024-03-20T09:17:48.439 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:48 smithi026 ceph-mon[68309]: 4.2 scrub ok 2024-03-20T09:17:48.439 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:48 smithi026 ceph-mon[68309]: 1.17 scrub starts 2024-03-20T09:17:48.439 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:48 smithi026 ceph-mon[68309]: 1.17 scrub ok 2024-03-20T09:17:48.439 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:48 smithi026 ceph-mon[68309]: osdmap e598: 12 total, 12 up, 10 in 2024-03-20T09:17:48.439 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:48 smithi026 ceph-mon[68309]: 1.14 scrub starts 2024-03-20T09:17:48.439 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:48 smithi026 ceph-mon[68309]: 1.14 scrub ok 2024-03-20T09:17:48.439 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:48 smithi026 ceph-mon[68309]: 4.c scrub starts 2024-03-20T09:17:48.439 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:48 smithi026 ceph-mon[68309]: pgmap v1434: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 333 MiB data, 4.8 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.02k op/s 2024-03-20T09:17:48.442 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:48.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:48 smithi122 ceph-mon[73495]: 1.e scrub starts 2024-03-20T09:17:48.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:48 smithi122 ceph-mon[73495]: 1.e scrub ok 2024-03-20T09:17:48.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:48 smithi122 ceph-mon[73495]: 4.2 scrub starts 2024-03-20T09:17:48.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:48 smithi122 ceph-mon[73495]: 4.2 scrub ok 2024-03-20T09:17:48.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:48 smithi122 ceph-mon[73495]: 1.17 scrub starts 2024-03-20T09:17:48.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:48 smithi122 ceph-mon[73495]: 1.17 scrub ok 2024-03-20T09:17:48.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:48 smithi122 ceph-mon[73495]: osdmap e598: 12 total, 12 up, 10 in 2024-03-20T09:17:48.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:48 smithi122 ceph-mon[73495]: 1.14 scrub starts 2024-03-20T09:17:48.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:48 smithi122 ceph-mon[73495]: 1.14 scrub ok 2024-03-20T09:17:48.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:48 smithi122 ceph-mon[73495]: 4.c scrub starts 2024-03-20T09:17:48.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:48 smithi122 ceph-mon[73495]: pgmap v1434: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 333 MiB data, 4.8 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 1.02k op/s 2024-03-20T09:17:48.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:48.503 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:17:48.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:17:48.604 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:17:48.636 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:49.362 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:17:49.363 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:17:49.446 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:49 smithi026 ceph-mon[68309]: 1.1 scrub starts 2024-03-20T09:17:49.446 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:49 smithi026 ceph-mon[68309]: 1.1 scrub ok 2024-03-20T09:17:49.446 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:49 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:17:49.446 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:49 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:17:49.447 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:49 smithi026 ceph-mon[68309]: 4.c scrub ok 2024-03-20T09:17:49.447 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:49 smithi026 ceph-mon[68309]: 1.12 scrub starts 2024-03-20T09:17:49.447 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:49 smithi026 ceph-mon[68309]: 1.12 scrub ok 2024-03-20T09:17:49.447 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:49 smithi026 ceph-mon[68309]: osdmap e599: 12 total, 12 up, 10 in 2024-03-20T09:17:49.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:49 smithi122 ceph-mon[73495]: 1.1 scrub starts 2024-03-20T09:17:49.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:49 smithi122 ceph-mon[73495]: 1.1 scrub ok 2024-03-20T09:17:49.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:49 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:17:49.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:49 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:17:49.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:49 smithi122 ceph-mon[73495]: 4.c scrub ok 2024-03-20T09:17:49.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:49 smithi122 ceph-mon[73495]: 1.12 scrub starts 2024-03-20T09:17:49.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:49 smithi122 ceph-mon[73495]: 1.12 scrub ok 2024-03-20T09:17:49.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:49 smithi122 ceph-mon[73495]: osdmap e599: 12 total, 12 up, 10 in 2024-03-20T09:17:49.513 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:17:49.607 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:17:49.609 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:49 smithi115 ceph-mon[72476]: 1.1 scrub starts 2024-03-20T09:17:49.609 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:49 smithi115 ceph-mon[72476]: 1.1 scrub ok 2024-03-20T09:17:49.609 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:49 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:17:49.609 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:49 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:17:49.609 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:49 smithi115 ceph-mon[72476]: 4.c scrub ok 2024-03-20T09:17:49.609 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:49 smithi115 ceph-mon[72476]: 1.12 scrub starts 2024-03-20T09:17:49.609 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:49 smithi115 ceph-mon[72476]: 1.12 scrub ok 2024-03-20T09:17:49.610 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:49 smithi115 ceph-mon[72476]: osdmap e599: 12 total, 12 up, 10 in 2024-03-20T09:17:49.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:49.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:17:49.712 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:17:49.746 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:49.774 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:17:49.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:49.782 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:17:49.884 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:17:49.941 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:49.968 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:17:49.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:49.972 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:17:50.073 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:17:50.165 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:50.190 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-20T09:17:50.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:50 smithi122 ceph-mon[73495]: 1.5 scrub starts 2024-03-20T09:17:50.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:50 smithi122 ceph-mon[73495]: 1.5 scrub ok 2024-03-20T09:17:50.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:50 smithi122 ceph-mon[73495]: 1.8 scrub starts 2024-03-20T09:17:50.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:50 smithi122 ceph-mon[73495]: 1.8 scrub ok 2024-03-20T09:17:50.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:50 smithi122 ceph-mon[73495]: 1.0 scrub starts 2024-03-20T09:17:50.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:50 smithi122 ceph-mon[73495]: 1.0 scrub ok 2024-03-20T09:17:50.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:50 smithi122 ceph-mon[73495]: pgmap v1436: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 316 MiB data, 4.7 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 645 op/s 2024-03-20T09:17:50.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:50 smithi026 ceph-mon[68309]: 1.5 scrub starts 2024-03-20T09:17:50.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:50 smithi026 ceph-mon[68309]: 1.5 scrub ok 2024-03-20T09:17:50.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:50 smithi026 ceph-mon[68309]: 1.8 scrub starts 2024-03-20T09:17:50.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:50 smithi026 ceph-mon[68309]: 1.8 scrub ok 2024-03-20T09:17:50.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:50 smithi026 ceph-mon[68309]: 1.0 scrub starts 2024-03-20T09:17:50.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:50 smithi026 ceph-mon[68309]: 1.0 scrub ok 2024-03-20T09:17:50.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:50 smithi026 ceph-mon[68309]: pgmap v1436: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 316 MiB data, 4.7 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 645 op/s 2024-03-20T09:17:50.615 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:17:50.654 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:17:50.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:50 smithi115 ceph-mon[72476]: 1.5 scrub starts 2024-03-20T09:17:50.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:50 smithi115 ceph-mon[72476]: 1.5 scrub ok 2024-03-20T09:17:50.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:50 smithi115 ceph-mon[72476]: 1.8 scrub starts 2024-03-20T09:17:50.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:50 smithi115 ceph-mon[72476]: 1.8 scrub ok 2024-03-20T09:17:50.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:50 smithi115 ceph-mon[72476]: 1.0 scrub starts 2024-03-20T09:17:50.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:50 smithi115 ceph-mon[72476]: 1.0 scrub ok 2024-03-20T09:17:50.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:50 smithi115 ceph-mon[72476]: pgmap v1436: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 316 MiB data, 4.7 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 645 op/s 2024-03-20T09:17:50.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:50.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:17:50.764 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:17:50.841 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:51.109 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:17:51.285 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:51 smithi115 ceph-mon[72476]: 4.e scrub starts 2024-03-20T09:17:51.285 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:51 smithi115 ceph-mon[72476]: 4.e scrub ok 2024-03-20T09:17:51.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:51 smithi122 ceph-mon[73495]: 4.e scrub starts 2024-03-20T09:17:51.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:51 smithi122 ceph-mon[73495]: 4.e scrub ok 2024-03-20T09:17:51.481 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:51 smithi026 ceph-mon[68309]: 4.e scrub starts 2024-03-20T09:17:51.481 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:51 smithi026 ceph-mon[68309]: 4.e scrub ok 2024-03-20T09:17:52.240 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:17:52.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:52.259 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.6 to 1 (10000) 2024-03-20T09:17:52.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:17:52.360 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:17:52.398 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:52.400 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:52 smithi115 ceph-mon[72476]: 4.5 scrub starts 2024-03-20T09:17:52.400 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:52 smithi115 ceph-mon[72476]: 4.5 scrub ok 2024-03-20T09:17:52.400 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:52 smithi115 ceph-mon[72476]: 4.0 scrub starts 2024-03-20T09:17:52.400 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:52 smithi115 ceph-mon[72476]: 1.10 scrub starts 2024-03-20T09:17:52.400 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:52 smithi115 ceph-mon[72476]: 1.10 scrub ok 2024-03-20T09:17:52.400 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:52 smithi115 ceph-mon[72476]: 4.0 scrub ok 2024-03-20T09:17:52.400 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:52 smithi115 ceph-mon[72476]: 1.1c scrub starts 2024-03-20T09:17:52.401 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:52 smithi115 ceph-mon[72476]: 1.1c scrub ok 2024-03-20T09:17:52.401 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:52 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2851887207' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-20T09:17:52.401 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:52 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-20T09:17:52.401 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:52 smithi115 ceph-mon[72476]: pgmap v1437: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 157 MiB data, 4.2 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 878 op/s 2024-03-20T09:17:52.425 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:17:52.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:52.433 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:17:52.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:52 smithi026 ceph-mon[68309]: 4.5 scrub starts 2024-03-20T09:17:52.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:52 smithi026 ceph-mon[68309]: 4.5 scrub ok 2024-03-20T09:17:52.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:52 smithi026 ceph-mon[68309]: 4.0 scrub starts 2024-03-20T09:17:52.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:52 smithi026 ceph-mon[68309]: 1.10 scrub starts 2024-03-20T09:17:52.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:52 smithi026 ceph-mon[68309]: 1.10 scrub ok 2024-03-20T09:17:52.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:52 smithi026 ceph-mon[68309]: 4.0 scrub ok 2024-03-20T09:17:52.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:52 smithi026 ceph-mon[68309]: 1.1c scrub starts 2024-03-20T09:17:52.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:52 smithi026 ceph-mon[68309]: 1.1c scrub ok 2024-03-20T09:17:52.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:52 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2851887207' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-20T09:17:52.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:52 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-20T09:17:52.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:52 smithi026 ceph-mon[68309]: pgmap v1437: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 157 MiB data, 4.2 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 878 op/s 2024-03-20T09:17:52.533 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:17:52.610 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Removed 120916 objects 2024-03-20T09:17:52.610 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Clean up completed and total clean up time :169.856 2024-03-20T09:17:52.616 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:52.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:52 smithi122 ceph-mon[73495]: 4.5 scrub starts 2024-03-20T09:17:52.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:52 smithi122 ceph-mon[73495]: 4.5 scrub ok 2024-03-20T09:17:52.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:52 smithi122 ceph-mon[73495]: 4.0 scrub starts 2024-03-20T09:17:52.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:52 smithi122 ceph-mon[73495]: 1.10 scrub starts 2024-03-20T09:17:52.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:52 smithi122 ceph-mon[73495]: 1.10 scrub ok 2024-03-20T09:17:52.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:52 smithi122 ceph-mon[73495]: 4.0 scrub ok 2024-03-20T09:17:52.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:52 smithi122 ceph-mon[73495]: 1.1c scrub starts 2024-03-20T09:17:52.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:52 smithi122 ceph-mon[73495]: 1.1c scrub ok 2024-03-20T09:17:52.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:52 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2851887207' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-20T09:17:52.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:52 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-20T09:17:52.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:52 smithi122 ceph-mon[73495]: pgmap v1437: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 157 MiB data, 4.2 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 878 op/s 2024-03-20T09:17:52.728 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:17:52.728 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:17:52.768 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:17:52.770 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:52.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:17:52.872 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:17:52.900 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:52.918 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:17:52.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:52.921 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:17:53.022 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:17:53.052 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:53.329 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:53 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-20T09:17:53.329 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:53 smithi115 ceph-mon[72476]: osdmap e600: 12 total, 12 up, 10 in 2024-03-20T09:17:53.329 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:53 smithi115 ceph-mon[72476]: 1.16 scrub starts 2024-03-20T09:17:53.329 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:53 smithi115 ceph-mon[72476]: 1.16 scrub ok 2024-03-20T09:17:53.356 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_ops_in_flight 2024-03-20T09:17:53.400 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:17:53.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:53.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:17:53.415 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 7 1 2024-03-20T09:17:53.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:53 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-20T09:17:53.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:53 smithi026 ceph-mon[68309]: osdmap e600: 12 total, 12 up, 10 in 2024-03-20T09:17:53.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:53 smithi026 ceph-mon[68309]: 1.16 scrub starts 2024-03-20T09:17:53.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:53 smithi026 ceph-mon[68309]: 1.16 scrub ok 2024-03-20T09:17:53.504 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:17:53.530 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:53 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-20T09:17:53.530 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:53 smithi122 ceph-mon[73495]: osdmap e600: 12 total, 12 up, 10 in 2024-03-20T09:17:53.530 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:53 smithi122 ceph-mon[73495]: 1.16 scrub starts 2024-03-20T09:17:53.530 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:53 smithi122 ceph-mon[73495]: 1.16 scrub ok 2024-03-20T09:17:53.532 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:53.543 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:17:53.554 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:17:53.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:53.557 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:17:53.636 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:17:53.659 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:17:53.692 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:53.713 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:17:53.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:53.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:17:53.817 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:17:53.846 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:53.865 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:17:53.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:53.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:17:53.969 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:17:54.000 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:54.011 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-03-20T09:17:54.011 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-03-20T09:17:54.369 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:17:54.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:54 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2393307030' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:17:54.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:54 smithi122 ceph-mon[73495]: pgmap v1439: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 78 MiB data, 4.0 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 783 op/s 2024-03-20T09:17:54.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:54 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2393307030' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:17:54.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:54 smithi026 ceph-mon[68309]: pgmap v1439: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 78 MiB data, 4.0 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 783 op/s 2024-03-20T09:17:54.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:54 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2393307030' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:17:54.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:54 smithi115 ceph-mon[72476]: pgmap v1439: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 78 MiB data, 4.0 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 783 op/s 2024-03-20T09:17:54.952 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:17:54.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:54.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:17:55.058 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:17:55.091 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:55.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:55 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:17:55.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:55 smithi122 ceph-mon[73495]: Cluster is now healthy 2024-03-20T09:17:55.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:55 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2393307030' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:17:55.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:55 smithi122 ceph-mon[73495]: osdmap e601: 12 total, 12 up, 10 in 2024-03-20T09:17:55.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:55 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:17:55.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:55 smithi026 ceph-mon[68309]: Cluster is now healthy 2024-03-20T09:17:55.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:55 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2393307030' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:17:55.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:55 smithi026 ceph-mon[68309]: osdmap e601: 12 total, 12 up, 10 in 2024-03-20T09:17:55.480 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:17:55.480 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:17:55.526 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:17:55.544 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:17:55.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:55.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:17:55.650 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:17:55.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:55 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:17:55.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:55 smithi115 ceph-mon[72476]: Cluster is now healthy 2024-03-20T09:17:55.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:55 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2393307030' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:17:55.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:55 smithi115 ceph-mon[72476]: osdmap e601: 12 total, 12 up, 10 in 2024-03-20T09:17:55.680 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:55.759 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:17:56.093 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_blocked_ops 2024-03-20T09:17:56.129 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:17:56.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:56.131 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:17:56.232 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:17:56.265 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:56.385 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:17:56.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:56 smithi122 ceph-mon[73495]: 4.b scrub starts 2024-03-20T09:17:56.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:56 smithi122 ceph-mon[73495]: 4.b scrub ok 2024-03-20T09:17:56.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:56 smithi122 ceph-mon[73495]: pgmap v1441: 47 pgs: 47 active+clean; 10 MiB data, 3.8 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 666 op/s 2024-03-20T09:17:56.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:56 smithi026 ceph-mon[68309]: 4.b scrub starts 2024-03-20T09:17:56.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:56 smithi026 ceph-mon[68309]: 4.b scrub ok 2024-03-20T09:17:56.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:56 smithi026 ceph-mon[68309]: pgmap v1441: 47 pgs: 47 active+clean; 10 MiB data, 3.8 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 666 op/s 2024-03-20T09:17:56.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:56 smithi115 ceph-mon[72476]: 4.b scrub starts 2024-03-20T09:17:56.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:56 smithi115 ceph-mon[72476]: 4.b scrub ok 2024-03-20T09:17:56.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:56 smithi115 ceph-mon[72476]: pgmap v1441: 47 pgs: 47 active+clean; 10 MiB data, 3.8 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 666 op/s 2024-03-20T09:17:57.540 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:57 smithi122 ceph-mon[73495]: 4.6 scrub starts 2024-03-20T09:17:57.540 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:57 smithi122 ceph-mon[73495]: 4.6 scrub ok 2024-03-20T09:17:57.567 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:17:57.594 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:57 smithi026 ceph-mon[68309]: 4.6 scrub starts 2024-03-20T09:17:57.594 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:57 smithi026 ceph-mon[68309]: 4.6 scrub ok 2024-03-20T09:17:57.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:57 smithi115 ceph-mon[72476]: 4.6 scrub starts 2024-03-20T09:17:57.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:57 smithi115 ceph-mon[72476]: 4.6 scrub ok 2024-03-20T09:17:58.107 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:17:58.107 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:17:58.246 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:17:58.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:58.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:17:58.351 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:17:58.388 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:58.410 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:17:58.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:58.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:17:58.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:58 smithi122 ceph-mon[73495]: 1.13 scrub starts 2024-03-20T09:17:58.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:58 smithi122 ceph-mon[73495]: 1.13 scrub ok 2024-03-20T09:17:58.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:58 smithi122 ceph-mon[73495]: pgmap v1442: 47 pgs: 47 active+clean; 577 KiB data, 3.7 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 631 op/s 2024-03-20T09:17:58.514 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:17:58.542 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:58 smithi115 ceph-mon[72476]: 1.13 scrub starts 2024-03-20T09:17:58.542 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:58 smithi115 ceph-mon[72476]: 1.13 scrub ok 2024-03-20T09:17:58.542 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:58 smithi115 ceph-mon[72476]: pgmap v1442: 47 pgs: 47 active+clean; 577 KiB data, 3.7 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 631 op/s 2024-03-20T09:17:58.544 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:58.564 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:17:58.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:58.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:17:58.657 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:58 smithi026 ceph-mon[68309]: 1.13 scrub starts 2024-03-20T09:17:58.657 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:58 smithi026 ceph-mon[68309]: 1.13 scrub ok 2024-03-20T09:17:58.657 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:58 smithi026 ceph-mon[68309]: pgmap v1442: 47 pgs: 47 active+clean; 577 KiB data, 3.7 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 631 op/s 2024-03-20T09:17:58.668 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:17:58.698 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:58.717 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:17:58.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:58.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:17:58.771 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_historic_ops 2024-03-20T09:17:58.821 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:17:58.847 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:58.867 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:17:58.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:58.870 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:17:58.971 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:17:59.003 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:59.021 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:17:59.033 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:17:59.255 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.7 to 1 (10000) 2024-03-20T09:17:59.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:59.439 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:17:59.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:17:59.504 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:17:59 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3158559863' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:17:59.531 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:17:59 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3158559863' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:17:59.540 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:17:59.572 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:17:59 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3158559863' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:17:59.575 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:59.596 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:17:59.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:59.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:17:59.701 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:17:59.730 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:59.750 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:17:59.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:59.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:17:59.855 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:17:59.883 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:17:59.902 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:17:59.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:17:59.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:17:59.916 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:17:59.916 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:17:59.917 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:17:59.925 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:17:59.945 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:17:59.965 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:17:59.985 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:18:00.006 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:18:00.007 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:18:00.027 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:18:00.036 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:00.047 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:18:00.068 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:18:00.088 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:18:00.089 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:18:00.089 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:18:00.089 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:18:00.089 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:18:00.089 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:18:00.089 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:18:00.090 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:18:00.090 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:18:00.090 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:18:00.090 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:18:00.090 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:18:00.090 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:18:00.235 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:18:00.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:00.238 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:18:00.249 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 8 1 2024-03-20T09:18:00.285 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'unique_pool_2' removed 2024-03-20T09:18:00.339 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:18:00.378 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:00.454 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:18:00.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:00.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:18:00.519 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:00 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3158559863' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-20T09:18:00.519 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:00 smithi122 ceph-mon[73495]: osdmap e602: 12 total, 12 up, 10 in 2024-03-20T09:18:00.519 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:00 smithi122 ceph-mon[73495]: 1.1a scrub starts 2024-03-20T09:18:00.519 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:00 smithi122 ceph-mon[73495]: 1.1a scrub ok 2024-03-20T09:18:00.519 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:00 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2085191004' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-20T09:18:00.519 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:00 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4108622832' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:18:00.519 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:00 smithi122 ceph-mon[73495]: pgmap v1444: 47 pgs: 47 active+clean; 577 KiB data, 3.7 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 319 op/s 2024-03-20T09:18:00.566 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:18:00.591 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:00 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3158559863' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-20T09:18:00.591 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:00 smithi026 ceph-mon[68309]: osdmap e602: 12 total, 12 up, 10 in 2024-03-20T09:18:00.591 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:00 smithi026 ceph-mon[68309]: 1.1a scrub starts 2024-03-20T09:18:00.592 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:00 smithi026 ceph-mon[68309]: 1.1a scrub ok 2024-03-20T09:18:00.592 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:00 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2085191004' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-20T09:18:00.592 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:00 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4108622832' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:18:00.592 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:00 smithi026 ceph-mon[68309]: pgmap v1444: 47 pgs: 47 active+clean; 577 KiB data, 3.7 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 319 op/s 2024-03-20T09:18:00.593 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:00.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:00 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3158559863' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-20T09:18:00.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:00 smithi115 ceph-mon[72476]: osdmap e602: 12 total, 12 up, 10 in 2024-03-20T09:18:00.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:00 smithi115 ceph-mon[72476]: 1.1a scrub starts 2024-03-20T09:18:00.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:00 smithi115 ceph-mon[72476]: 1.1a scrub ok 2024-03-20T09:18:00.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:00 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2085191004' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-20T09:18:00.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:00 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4108622832' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:18:00.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:00 smithi115 ceph-mon[72476]: pgmap v1444: 47 pgs: 47 active+clean; 577 KiB data, 3.7 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 319 op/s 2024-03-20T09:18:00.778 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:18:00.779 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-03-20T09:18:00.779 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-03-20T09:18:00.779 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-03-20T09:18:00.779 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.779 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.24910.0:137643 3.15 3:a84eca2e:::benchmark_data_smithi122_119671_object37861:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-03-20T09:18:00.779 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:09:35.283389+0000", 2024-03-20T09:18:00.779 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 505.491748884, 2024-03-20T09:18:00.779 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.87423633899999997, 2024-03-20T09:18:00.779 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:00.779 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:00.779 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:00.779 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.24910", 2024-03-20T09:18:00.780 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:18:00.780 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 137643 2024-03-20T09:18:00.780 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.780 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:00.780 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.780 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:00.780 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:35.283389+0000", 2024-03-20T09:18:00.780 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.780 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.780 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.780 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:00.780 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:35.283389+0000", 2024-03-20T09:18:00.780 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.781 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.781 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.781 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:00.781 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:35.283391+0000", 2024-03-20T09:18:00.781 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8870000000000001e-06 2024-03-20T09:18:00.781 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.781 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.781 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:00.781 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:35.283399+0000", 2024-03-20T09:18:00.781 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2120000000000003e-06 2024-03-20T09:18:00.781 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.781 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.781 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:00.781 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:35.283414+0000", 2024-03-20T09:18:00.782 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5121e-05 2024-03-20T09:18:00.782 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.782 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.782 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:00.782 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:35.283425+0000", 2024-03-20T09:18:00.782 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0197000000000001e-05 2024-03-20T09:18:00.782 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.782 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.782 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.782 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:35.283508+0000", 2024-03-20T09:18:00.782 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3332999999999996e-05 2024-03-20T09:18:00.782 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.782 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.783 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:18:00.783 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:35.283520+0000", 2024-03-20T09:18:00.783 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2140000000000001e-05 2024-03-20T09:18:00.783 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.783 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.783 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.783 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.152152+0000", 2024-03-20T09:18:00.783 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.86863241700000005 2024-03-20T09:18:00.783 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.783 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.783 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:00.783 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.152309+0000", 2024-03-20T09:18:00.783 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000156522 2024-03-20T09:18:00.783 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.783 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.784 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,6", 2024-03-20T09:18:00.784 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.152433+0000", 2024-03-20T09:18:00.784 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012421000000000001 2024-03-20T09:18:00.784 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.784 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.784 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:00.784 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.156255+0000", 2024-03-20T09:18:00.784 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0038215129999999999 2024-03-20T09:18:00.784 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.784 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.784 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:18:00.784 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.157263+0000", 2024-03-20T09:18:00.784 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010086170000000001 2024-03-20T09:18:00.784 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.785 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.785 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:00.785 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.157560+0000", 2024-03-20T09:18:00.785 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000296514 2024-03-20T09:18:00.785 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.785 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.785 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:00.785 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.157595+0000", 2024-03-20T09:18:00.785 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4700000000000003e-05 2024-03-20T09:18:00.785 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.785 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.785 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:00.785 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:09:36.157626+0000", 2024-03-20T09:18:00.785 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0956e-05 2024-03-20T09:18:00.786 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.786 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:00.786 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.786 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.786 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.786 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212811 4.b 4.11862d5b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:18:00.786 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.225559+0000", 2024-03-20T09:18:00.786 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 43.54957924, 2024-03-20T09:18:00.786 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.040956411, 2024-03-20T09:18:00.786 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:00.786 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:18:00.786 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:00.786 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:18:00.787 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:18:00.787 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212811 2024-03-20T09:18:00.787 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.787 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:00.787 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.787 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:00.787 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.225559+0000", 2024-03-20T09:18:00.787 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.787 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.787 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.787 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:00.787 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.225559+0000", 2024-03-20T09:18:00.787 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.787 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.788 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.788 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:00.788 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.225560+0000", 2024-03-20T09:18:00.788 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.42e-06 2024-03-20T09:18:00.788 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.788 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.788 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:00.788 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.225566+0000", 2024-03-20T09:18:00.788 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3480000000000003e-06 2024-03-20T09:18:00.788 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.788 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.788 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:00.788 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.225578+0000", 2024-03-20T09:18:00.788 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2313000000000001e-05 2024-03-20T09:18:00.788 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.789 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.789 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:00.789 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.225586+0000", 2024-03-20T09:18:00.789 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3049999999999993e-06 2024-03-20T09:18:00.789 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.789 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.789 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.789 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.225638+0000", 2024-03-20T09:18:00.789 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1431000000000003e-05 2024-03-20T09:18:00.789 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.789 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.789 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:18:00.789 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.225644+0000", 2024-03-20T09:18:00.789 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0909999999999998e-06 2024-03-20T09:18:00.790 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.790 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.790 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.790 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.266501+0000", 2024-03-20T09:18:00.790 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0408575849999999 2024-03-20T09:18:00.790 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.790 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.790 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:00.790 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.266515+0000", 2024-03-20T09:18:00.790 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3918e-05 2024-03-20T09:18:00.790 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.790 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:00.790 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.791 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.791 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.791 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212818 4.b 4.8a8d05db (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:18:00.791 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.249767+0000", 2024-03-20T09:18:00.791 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 43.525370936000002, 2024-03-20T09:18:00.791 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0170676489999999, 2024-03-20T09:18:00.791 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:00.791 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:18:00.791 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:00.791 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:18:00.791 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:18:00.791 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212818 2024-03-20T09:18:00.791 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.791 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:00.792 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.792 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:00.792 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.249767+0000", 2024-03-20T09:18:00.792 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.792 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.792 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.792 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:00.792 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.249767+0000", 2024-03-20T09:18:00.792 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.792 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.792 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.792 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:00.792 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.249769+0000", 2024-03-20T09:18:00.792 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5239999999999999e-06 2024-03-20T09:18:00.792 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.793 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.793 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:00.793 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.249775+0000", 2024-03-20T09:18:00.793 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3709999999999996e-06 2024-03-20T09:18:00.793 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.793 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.793 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:00.793 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.249789+0000", 2024-03-20T09:18:00.793 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4204e-05 2024-03-20T09:18:00.793 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.793 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.793 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:00.794 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.249797+0000", 2024-03-20T09:18:00.794 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7400000000000004e-06 2024-03-20T09:18:00.794 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.794 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.794 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.794 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.249847+0000", 2024-03-20T09:18:00.794 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9585e-05 2024-03-20T09:18:00.794 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.794 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.794 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:18:00.794 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.249855+0000", 2024-03-20T09:18:00.794 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0029999999999994e-06 2024-03-20T09:18:00.794 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.794 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.795 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.795 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.266829+0000", 2024-03-20T09:18:00.795 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.016974319 2024-03-20T09:18:00.795 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.795 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.795 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:00.795 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.266835+0000", 2024-03-20T09:18:00.795 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9030000000000001e-06 2024-03-20T09:18:00.795 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.795 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:00.795 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.795 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.795 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.795 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212844 4.b 4.5ff7566b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:18:00.796 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.291365+0000", 2024-03-20T09:18:00.796 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 43.483772623999997, 2024-03-20T09:18:00.796 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.97549757100000001, 2024-03-20T09:18:00.796 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:00.796 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:18:00.796 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:00.796 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:18:00.796 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:18:00.796 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212844 2024-03-20T09:18:00.796 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.796 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:00.796 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.796 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:00.796 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.291365+0000", 2024-03-20T09:18:00.797 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.797 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.797 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.797 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:00.797 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.291365+0000", 2024-03-20T09:18:00.797 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.797 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.797 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.797 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:00.797 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.291367+0000", 2024-03-20T09:18:00.797 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.283e-06 2024-03-20T09:18:00.797 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.797 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.797 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:00.797 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.291373+0000", 2024-03-20T09:18:00.798 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8719999999999999e-06 2024-03-20T09:18:00.798 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.798 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.798 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:00.798 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.291384+0000", 2024-03-20T09:18:00.798 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1895999999999999e-05 2024-03-20T09:18:00.798 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.798 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.798 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:00.798 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.291391+0000", 2024-03-20T09:18:00.798 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9999999999999999e-06 2024-03-20T09:18:00.798 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.798 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.798 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.799 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.291442+0000", 2024-03-20T09:18:00.799 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0389000000000002e-05 2024-03-20T09:18:00.799 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.799 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.799 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:18:00.799 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.291452+0000", 2024-03-20T09:18:00.799 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9899999999999992e-06 2024-03-20T09:18:00.799 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.799 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.799 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.799 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.266856+0000", 2024-03-20T09:18:00.799 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.97540415000000003 2024-03-20T09:18:00.799 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.799 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.800 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:00.800 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.266863+0000", 2024-03-20T09:18:00.800 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9909999999999996e-06 2024-03-20T09:18:00.800 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.800 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:00.800 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.800 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.800 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.800 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212811 4.b 4:dab46188:::benchmark_data_smithi122_145698_object91892:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:18:00.800 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.302808+0000", 2024-03-20T09:18:00.800 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 43.472330542000002, 2024-03-20T09:18:00.800 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.97602456199999998, 2024-03-20T09:18:00.800 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:00.800 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:00.801 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:00.801 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:18:00.801 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:18:00.801 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212811 2024-03-20T09:18:00.801 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.801 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:00.801 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.801 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:00.801 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302808+0000", 2024-03-20T09:18:00.801 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.801 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.801 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.801 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:00.801 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302808+0000", 2024-03-20T09:18:00.802 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.802 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.802 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.802 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:00.802 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302809+0000", 2024-03-20T09:18:00.802 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.094e-06 2024-03-20T09:18:00.802 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.802 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.802 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:00.802 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302813+0000", 2024-03-20T09:18:00.802 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6140000000000002e-06 2024-03-20T09:18:00.802 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.802 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.802 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:00.803 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302821+0000", 2024-03-20T09:18:00.803 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6559999999999992e-06 2024-03-20T09:18:00.803 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.803 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.803 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:00.803 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302827+0000", 2024-03-20T09:18:00.803 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.55e-06 2024-03-20T09:18:00.803 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.803 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.803 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.803 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302871+0000", 2024-03-20T09:18:00.803 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3659000000000003e-05 2024-03-20T09:18:00.803 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.803 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.804 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:18:00.804 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302878+0000", 2024-03-20T09:18:00.804 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7320000000000001e-06 2024-03-20T09:18:00.804 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.804 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.804 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.804 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.266903+0000", 2024-03-20T09:18:00.804 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.96402543299999999 2024-03-20T09:18:00.804 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.804 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.804 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:00.804 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.267005+0000", 2024-03-20T09:18:00.804 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010216400000000001 2024-03-20T09:18:00.804 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.804 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.805 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:18:00.805 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.267087+0000", 2024-03-20T09:18:00.805 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1893999999999999e-05 2024-03-20T09:18:00.805 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.805 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.805 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:00.805 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.274657+0000", 2024-03-20T09:18:00.805 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0075700979999999999 2024-03-20T09:18:00.805 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.805 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.805 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:18:00.805 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.278672+0000", 2024-03-20T09:18:00.805 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040150840000000004 2024-03-20T09:18:00.805 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.805 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.806 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:00.806 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.278772+0000", 2024-03-20T09:18:00.806 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9944000000000004e-05 2024-03-20T09:18:00.806 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.806 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.806 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:00.806 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.278807+0000", 2024-03-20T09:18:00.806 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4996999999999997e-05 2024-03-20T09:18:00.806 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.806 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.806 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:00.806 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.278832+0000", 2024-03-20T09:18:00.806 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4643000000000001e-05 2024-03-20T09:18:00.807 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.807 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:00.807 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.807 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.807 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.807 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212818 4.b 4:dba0b151:::benchmark_data_smithi122_145698_object91899:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:18:00.807 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.302840+0000", 2024-03-20T09:18:00.807 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 43.472297677999997, 2024-03-20T09:18:00.807 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.98232673100000001, 2024-03-20T09:18:00.807 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:00.807 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:00.807 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:00.807 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:18:00.807 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:18:00.808 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212818 2024-03-20T09:18:00.808 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.808 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:00.808 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.808 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:00.808 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302840+0000", 2024-03-20T09:18:00.808 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.808 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.808 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.808 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:00.808 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302840+0000", 2024-03-20T09:18:00.808 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.808 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.808 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.809 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:00.809 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302841+0000", 2024-03-20T09:18:00.809 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8900000000000001e-07 2024-03-20T09:18:00.809 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.809 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.809 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:00.809 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302842+0000", 2024-03-20T09:18:00.809 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6449999999999999e-06 2024-03-20T09:18:00.809 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.809 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.809 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:00.809 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302847+0000", 2024-03-20T09:18:00.809 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6609999999999999e-06 2024-03-20T09:18:00.809 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.810 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.810 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:00.810 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302851+0000", 2024-03-20T09:18:00.810 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7529999999999999e-06 2024-03-20T09:18:00.810 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.810 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.810 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.810 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302914+0000", 2024-03-20T09:18:00.810 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2842000000000006e-05 2024-03-20T09:18:00.810 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.810 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.810 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:18:00.810 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302925+0000", 2024-03-20T09:18:00.810 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1732e-05 2024-03-20T09:18:00.810 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.811 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.811 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.811 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.269166+0000", 2024-03-20T09:18:00.811 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.96624085900000001 2024-03-20T09:18:00.811 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.811 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.811 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:00.811 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.269235+0000", 2024-03-20T09:18:00.811 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8522000000000006e-05 2024-03-20T09:18:00.811 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.811 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.811 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:18:00.811 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.269317+0000", 2024-03-20T09:18:00.811 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2165999999999999e-05 2024-03-20T09:18:00.812 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.812 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.812 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:00.812 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.274708+0000", 2024-03-20T09:18:00.812 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0053907690000000001 2024-03-20T09:18:00.812 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.812 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.812 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:18:00.812 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.278679+0000", 2024-03-20T09:18:00.812 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0039710659999999997 2024-03-20T09:18:00.812 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.812 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.812 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:00.813 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.285056+0000", 2024-03-20T09:18:00.813 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0063770950000000002 2024-03-20T09:18:00.813 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.813 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.813 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:00.813 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.285113+0000", 2024-03-20T09:18:00.813 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7479000000000003e-05 2024-03-20T09:18:00.813 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.813 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.813 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:00.813 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.285167+0000", 2024-03-20T09:18:00.813 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3853000000000001e-05 2024-03-20T09:18:00.813 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.813 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:00.814 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.814 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.814 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.814 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212844 4.b 4:d66aeffa:::benchmark_data_smithi122_145698_object91925:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:18:00.814 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.302862+0000", 2024-03-20T09:18:00.814 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 43.472275633000002, 2024-03-20T09:18:00.814 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.98801962399999999, 2024-03-20T09:18:00.814 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:00.814 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:00.814 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:00.814 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:18:00.814 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:18:00.814 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212844 2024-03-20T09:18:00.814 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.815 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:00.815 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.815 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:00.815 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302862+0000", 2024-03-20T09:18:00.815 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.815 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.815 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.815 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:00.815 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302862+0000", 2024-03-20T09:18:00.815 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.815 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.815 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.815 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:00.815 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302863+0000", 2024-03-20T09:18:00.816 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4699999999999998e-07 2024-03-20T09:18:00.816 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.816 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.816 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:00.816 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302864+0000", 2024-03-20T09:18:00.816 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4330000000000001e-06 2024-03-20T09:18:00.816 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.816 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.816 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:00.816 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302868+0000", 2024-03-20T09:18:00.816 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6260000000000001e-06 2024-03-20T09:18:00.816 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.816 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.816 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:00.817 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302871+0000", 2024-03-20T09:18:00.817 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0929999999999999e-06 2024-03-20T09:18:00.817 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.817 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.817 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.817 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302948+0000", 2024-03-20T09:18:00.817 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7279000000000003e-05 2024-03-20T09:18:00.817 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.817 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.817 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:18:00.817 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302952+0000", 2024-03-20T09:18:00.817 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5750000000000001e-06 2024-03-20T09:18:00.817 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.817 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.818 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.818 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.270936+0000", 2024-03-20T09:18:00.818 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.96798445499999997 2024-03-20T09:18:00.818 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.818 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.818 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:00.818 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.271002+0000", 2024-03-20T09:18:00.818 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6129000000000003e-05 2024-03-20T09:18:00.818 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.818 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.818 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:18:00.818 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.271080+0000", 2024-03-20T09:18:00.818 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7349999999999996e-05 2024-03-20T09:18:00.819 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.819 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.819 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:18:00.819 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.278687+0000", 2024-03-20T09:18:00.819 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0076070679999999998 2024-03-20T09:18:00.819 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.819 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.819 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:00.819 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.278729+0000", 2024-03-20T09:18:00.819 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2608000000000003e-05 2024-03-20T09:18:00.819 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.819 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.819 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:00.819 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.290783+0000", 2024-03-20T09:18:00.820 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012053632 2024-03-20T09:18:00.820 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.820 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.820 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:00.820 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.290833+0000", 2024-03-20T09:18:00.820 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0235000000000003e-05 2024-03-20T09:18:00.820 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.820 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.820 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:00.820 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.290882+0000", 2024-03-20T09:18:00.820 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8894000000000003e-05 2024-03-20T09:18:00.820 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.820 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:00.821 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.821 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.821 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.821 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212851 4.b 4:deb6b4bb:::benchmark_data_smithi122_145698_object91932:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:18:00.821 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.302879+0000", 2024-03-20T09:18:00.821 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 43.472258998000001, 2024-03-20T09:18:00.821 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.98818408400000002, 2024-03-20T09:18:00.821 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:00.821 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:00.821 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:00.821 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:18:00.821 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:18:00.821 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212851 2024-03-20T09:18:00.821 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.821 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:00.822 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.822 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:00.822 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302879+0000", 2024-03-20T09:18:00.822 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.822 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.822 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.822 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:00.822 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302879+0000", 2024-03-20T09:18:00.822 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.822 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.822 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.822 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:00.822 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302879+0000", 2024-03-20T09:18:00.823 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.23e-07 2024-03-20T09:18:00.823 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.823 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.823 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:00.823 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302881+0000", 2024-03-20T09:18:00.823 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3209999999999999e-06 2024-03-20T09:18:00.823 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.823 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.823 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:00.823 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302884+0000", 2024-03-20T09:18:00.823 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4470000000000002e-06 2024-03-20T09:18:00.823 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.823 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.823 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:00.823 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302887+0000", 2024-03-20T09:18:00.824 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.287e-06 2024-03-20T09:18:00.824 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.824 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.824 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.824 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302966+0000", 2024-03-20T09:18:00.824 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8929000000000003e-05 2024-03-20T09:18:00.824 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.824 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.824 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:18:00.824 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302970+0000", 2024-03-20T09:18:00.824 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6320000000000001e-06 2024-03-20T09:18:00.824 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.824 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.824 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.824 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.272683+0000", 2024-03-20T09:18:00.825 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.96971318299999998 2024-03-20T09:18:00.825 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.825 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.825 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:00.825 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.272746+0000", 2024-03-20T09:18:00.825 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2973999999999997e-05 2024-03-20T09:18:00.825 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.825 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.825 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:18:00.825 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.272821+0000", 2024-03-20T09:18:00.825 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4542999999999994e-05 2024-03-20T09:18:00.825 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.825 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.825 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:18:00.826 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.278841+0000", 2024-03-20T09:18:00.826 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0060203790000000002 2024-03-20T09:18:00.826 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.826 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.826 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:00.826 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.278871+0000", 2024-03-20T09:18:00.826 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9556999999999998e-05 2024-03-20T09:18:00.826 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.826 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.826 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:00.826 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.290988+0000", 2024-03-20T09:18:00.826 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012117681999999999 2024-03-20T09:18:00.826 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.827 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.827 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:00.827 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.291033+0000", 2024-03-20T09:18:00.827 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5232999999999999e-05 2024-03-20T09:18:00.827 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.827 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.827 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:00.827 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.291063+0000", 2024-03-20T09:18:00.827 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9694000000000001e-05 2024-03-20T09:18:00.827 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.827 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:00.827 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.827 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.827 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.828 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212872 4.b 4:d3c1b7c6:::benchmark_data_smithi122_145698_object91953:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:18:00.828 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.325422+0000", 2024-03-20T09:18:00.828 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 43.449716363999997, 2024-03-20T09:18:00.828 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.96576648700000001, 2024-03-20T09:18:00.828 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:00.828 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:00.828 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:00.828 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:18:00.828 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:18:00.828 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212872 2024-03-20T09:18:00.828 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.828 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:00.828 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.829 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:00.829 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.325422+0000", 2024-03-20T09:18:00.829 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.829 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.829 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.829 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:00.829 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.325422+0000", 2024-03-20T09:18:00.829 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.829 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.829 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.829 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:00.829 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.325424+0000", 2024-03-20T09:18:00.829 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.88e-06 2024-03-20T09:18:00.829 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.829 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.830 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:00.830 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.325439+0000", 2024-03-20T09:18:00.830 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5359e-05 2024-03-20T09:18:00.830 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.830 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.830 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:00.830 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.325453+0000", 2024-03-20T09:18:00.830 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.398e-05 2024-03-20T09:18:00.830 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.830 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.830 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:00.830 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.325462+0000", 2024-03-20T09:18:00.830 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5170000000000005e-06 2024-03-20T09:18:00.830 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.831 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.831 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.831 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.325525+0000", 2024-03-20T09:18:00.831 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2822000000000003e-05 2024-03-20T09:18:00.831 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.831 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.831 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:18:00.831 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.325536+0000", 2024-03-20T09:18:00.831 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0387999999999999e-05 2024-03-20T09:18:00.831 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.831 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.831 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.831 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.274737+0000", 2024-03-20T09:18:00.831 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.949201136 2024-03-20T09:18:00.832 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.832 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.832 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:00.832 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.274812+0000", 2024-03-20T09:18:00.832 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5683e-05 2024-03-20T09:18:00.832 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.832 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.832 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:18:00.832 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.274895+0000", 2024-03-20T09:18:00.832 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2427000000000002e-05 2024-03-20T09:18:00.832 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.832 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.832 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:18:00.832 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.278845+0000", 2024-03-20T09:18:00.833 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0039496699999999997 2024-03-20T09:18:00.833 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.833 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.833 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:00.833 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.278899+0000", 2024-03-20T09:18:00.833 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4008999999999997e-05 2024-03-20T09:18:00.833 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.833 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.833 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:00.833 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.291126+0000", 2024-03-20T09:18:00.833 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012227269000000001 2024-03-20T09:18:00.833 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.833 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.833 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:00.833 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.291158+0000", 2024-03-20T09:18:00.834 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2459000000000002e-05 2024-03-20T09:18:00.834 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.834 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.834 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:00.834 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.291188+0000", 2024-03-20T09:18:00.834 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9887999999999999e-05 2024-03-20T09:18:00.834 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.834 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:00.834 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.834 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.834 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.834 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212882 4.b 4:d9263e6a:::benchmark_data_smithi122_145698_object91963:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:18:00.834 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.339019+0000", 2024-03-20T09:18:00.834 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 43.436119091999998, 2024-03-20T09:18:00.835 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95779115400000003, 2024-03-20T09:18:00.835 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:00.835 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:00.835 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:00.835 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:18:00.835 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:18:00.835 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212882 2024-03-20T09:18:00.835 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.835 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:00.835 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.835 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:00.835 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.339019+0000", 2024-03-20T09:18:00.835 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.835 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.836 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.836 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:00.836 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.339019+0000", 2024-03-20T09:18:00.836 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.836 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.836 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.836 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:00.836 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.339020+0000", 2024-03-20T09:18:00.836 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.192e-06 2024-03-20T09:18:00.836 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.836 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.836 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:00.836 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.339026+0000", 2024-03-20T09:18:00.837 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.558e-06 2024-03-20T09:18:00.837 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.837 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.837 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:00.837 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.339038+0000", 2024-03-20T09:18:00.837 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2242000000000001e-05 2024-03-20T09:18:00.837 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.837 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.837 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:00.837 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.339045+0000", 2024-03-20T09:18:00.837 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0899999999999999e-06 2024-03-20T09:18:00.837 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.837 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.838 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.838 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.339095+0000", 2024-03-20T09:18:00.838 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0056000000000003e-05 2024-03-20T09:18:00.838 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.838 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.838 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:18:00.838 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.339105+0000", 2024-03-20T09:18:00.838 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9059999999999997e-06 2024-03-20T09:18:00.838 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.838 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.838 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.838 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.276635+0000", 2024-03-20T09:18:00.838 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.93753048299999997 2024-03-20T09:18:00.838 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.839 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.839 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:00.839 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.276708+0000", 2024-03-20T09:18:00.839 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2668000000000003e-05 2024-03-20T09:18:00.839 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.839 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.839 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:18:00.839 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.276796+0000", 2024-03-20T09:18:00.839 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7843000000000006e-05 2024-03-20T09:18:00.839 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.839 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.839 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:18:00.839 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.278909+0000", 2024-03-20T09:18:00.839 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0021127950000000002 2024-03-20T09:18:00.840 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.840 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.840 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:00.840 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.278957+0000", 2024-03-20T09:18:00.840 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8223000000000002e-05 2024-03-20T09:18:00.840 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.840 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.840 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:00.840 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.296712+0000", 2024-03-20T09:18:00.840 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017754789999999999 2024-03-20T09:18:00.840 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.840 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.840 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:00.840 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.296764+0000", 2024-03-20T09:18:00.840 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2324999999999998e-05 2024-03-20T09:18:00.841 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.841 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.841 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:00.841 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.296810+0000", 2024-03-20T09:18:00.841 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5982999999999997e-05 2024-03-20T09:18:00.841 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.841 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:00.841 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.841 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.841 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.841 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212892 4.b 4:d92eb486:::benchmark_data_smithi122_145698_object91973:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:18:00.841 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.353878+0000", 2024-03-20T09:18:00.841 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 43.421259839000001, 2024-03-20T09:18:00.841 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.94304838899999999, 2024-03-20T09:18:00.841 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:00.842 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:00.842 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:00.842 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:18:00.842 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:18:00.842 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212892 2024-03-20T09:18:00.842 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.842 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:00.842 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.842 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:00.842 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.353878+0000", 2024-03-20T09:18:00.842 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.842 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.842 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.842 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:00.843 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.353878+0000", 2024-03-20T09:18:00.843 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.843 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.843 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.843 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:00.843 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.353880+0000", 2024-03-20T09:18:00.843 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.528e-06 2024-03-20T09:18:00.843 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.843 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.843 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:00.843 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.353887+0000", 2024-03-20T09:18:00.843 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0330000000000003e-06 2024-03-20T09:18:00.843 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.843 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.843 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:00.844 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.353901+0000", 2024-03-20T09:18:00.844 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4188999999999999e-05 2024-03-20T09:18:00.844 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.844 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.844 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:00.844 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.353909+0000", 2024-03-20T09:18:00.844 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3780000000000001e-06 2024-03-20T09:18:00.844 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.844 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.844 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.844 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.353970+0000", 2024-03-20T09:18:00.844 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0522000000000001e-05 2024-03-20T09:18:00.844 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.844 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.845 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:18:00.845 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.353979+0000", 2024-03-20T09:18:00.845 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0459999999999994e-06 2024-03-20T09:18:00.845 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.845 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.845 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.845 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.278984+0000", 2024-03-20T09:18:00.845 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.92500549300000001 2024-03-20T09:18:00.845 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.845 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.845 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:00.845 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.279054+0000", 2024-03-20T09:18:00.845 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9393999999999993e-05 2024-03-20T09:18:00.845 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.845 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.846 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:18:00.846 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.279120+0000", 2024-03-20T09:18:00.846 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6508999999999996e-05 2024-03-20T09:18:00.846 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.846 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.846 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:00.846 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.285226+0000", 2024-03-20T09:18:00.846 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0061055240000000002 2024-03-20T09:18:00.846 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.846 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.846 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:18:00.846 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.287457+0000", 2024-03-20T09:18:00.846 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0022312439999999998 2024-03-20T09:18:00.846 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.847 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.847 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:00.847 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.296854+0000", 2024-03-20T09:18:00.847 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0093965170000000001 2024-03-20T09:18:00.847 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.847 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.847 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:00.847 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.296886+0000", 2024-03-20T09:18:00.847 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1952999999999997e-05 2024-03-20T09:18:00.847 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.847 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.847 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:00.847 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.296927+0000", 2024-03-20T09:18:00.847 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1059000000000001e-05 2024-03-20T09:18:00.848 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.848 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:00.848 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.848 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.848 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.848 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212895 4.b 4:de658af1:::benchmark_data_smithi122_145698_object91976:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:18:00.848 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.354410+0000", 2024-03-20T09:18:00.848 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 43.420728443000002, 2024-03-20T09:18:00.848 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.94266090199999997, 2024-03-20T09:18:00.848 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:00.848 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:00.848 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:00.848 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:18:00.848 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:18:00.849 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212895 2024-03-20T09:18:00.849 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.849 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:00.849 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.849 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:00.849 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.354410+0000", 2024-03-20T09:18:00.849 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.849 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.849 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.849 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:00.849 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.354410+0000", 2024-03-20T09:18:00.849 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.849 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.849 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.849 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:00.849 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.354411+0000", 2024-03-20T09:18:00.850 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.33e-07 2024-03-20T09:18:00.850 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.850 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.850 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:00.850 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.354416+0000", 2024-03-20T09:18:00.850 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0579999999999997e-06 2024-03-20T09:18:00.850 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.850 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.850 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:00.850 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.354429+0000", 2024-03-20T09:18:00.850 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3370000000000001e-05 2024-03-20T09:18:00.850 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.850 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.850 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:00.851 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.354436+0000", 2024-03-20T09:18:00.851 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4220000000000003e-06 2024-03-20T09:18:00.851 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.851 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.851 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.851 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.354485+0000", 2024-03-20T09:18:00.851 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8176000000000001e-05 2024-03-20T09:18:00.851 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.851 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.851 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:18:00.851 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.354492+0000", 2024-03-20T09:18:00.851 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0729999999999999e-06 2024-03-20T09:18:00.851 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.851 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.851 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.852 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.280876+0000", 2024-03-20T09:18:00.852 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.92638386399999995 2024-03-20T09:18:00.852 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.852 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.852 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:00.852 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.280985+0000", 2024-03-20T09:18:00.852 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000109072 2024-03-20T09:18:00.852 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.852 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.852 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:18:00.852 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.281057+0000", 2024-03-20T09:18:00.852 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2485999999999997e-05 2024-03-20T09:18:00.852 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.852 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.853 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:00.853 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.285294+0000", 2024-03-20T09:18:00.853 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0042366859999999999 2024-03-20T09:18:00.853 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.853 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.853 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:18:00.853 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.287461+0000", 2024-03-20T09:18:00.853 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0021671709999999999 2024-03-20T09:18:00.853 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.853 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.853 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:00.853 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.296989+0000", 2024-03-20T09:18:00.853 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0095281020000000001 2024-03-20T09:18:00.853 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.854 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.854 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:00.854 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.297030+0000", 2024-03-20T09:18:00.854 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1399000000000001e-05 2024-03-20T09:18:00.854 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.854 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.854 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:00.854 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.297071+0000", 2024-03-20T09:18:00.854 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0089999999999997e-05 2024-03-20T09:18:00.854 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.854 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:00.854 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.854 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.854 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.855 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212904 4.b 4:dcc51503:::benchmark_data_smithi122_145698_object91985:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:18:00.855 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.363857+0000", 2024-03-20T09:18:00.855 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 43.411281496999997, 2024-03-20T09:18:00.855 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.93336363, 2024-03-20T09:18:00.855 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:00.855 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:00.855 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:00.855 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:18:00.855 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:18:00.855 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212904 2024-03-20T09:18:00.855 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.855 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:00.855 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.855 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:00.856 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.363857+0000", 2024-03-20T09:18:00.856 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.856 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.856 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.856 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:00.856 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.363857+0000", 2024-03-20T09:18:00.856 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.856 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.856 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.856 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:00.856 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.363858+0000", 2024-03-20T09:18:00.856 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4500000000000001e-06 2024-03-20T09:18:00.856 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.856 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.856 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:00.857 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.363868+0000", 2024-03-20T09:18:00.857 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6339999999999996e-06 2024-03-20T09:18:00.857 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.857 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.857 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:00.857 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.363882+0000", 2024-03-20T09:18:00.857 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4800000000000001e-05 2024-03-20T09:18:00.857 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.857 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.857 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:00.857 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.363892+0000", 2024-03-20T09:18:00.857 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6949999999999995e-06 2024-03-20T09:18:00.857 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.857 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.857 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.858 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.363952+0000", 2024-03-20T09:18:00.858 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9534000000000003e-05 2024-03-20T09:18:00.858 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.858 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.858 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:18:00.858 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.363962+0000", 2024-03-20T09:18:00.858 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0399e-05 2024-03-20T09:18:00.858 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.858 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.858 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.858 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.282952+0000", 2024-03-20T09:18:00.858 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.91899036999999995 2024-03-20T09:18:00.858 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.858 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.858 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:00.859 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.283061+0000", 2024-03-20T09:18:00.859 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000108918 2024-03-20T09:18:00.859 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.859 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.859 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:18:00.859 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.283184+0000", 2024-03-20T09:18:00.859 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000123022 2024-03-20T09:18:00.859 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.859 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.859 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:18:00.859 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.287465+0000", 2024-03-20T09:18:00.859 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0042803260000000001 2024-03-20T09:18:00.859 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.859 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.859 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:00.860 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.290686+0000", 2024-03-20T09:18:00.860 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0032214570000000001 2024-03-20T09:18:00.860 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.860 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.860 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:00.860 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.297127+0000", 2024-03-20T09:18:00.860 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0064408030000000002 2024-03-20T09:18:00.860 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.860 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.860 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:00.860 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.297182+0000", 2024-03-20T09:18:00.860 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4962e-05 2024-03-20T09:18:00.860 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.860 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.860 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:00.861 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.297220+0000", 2024-03-20T09:18:00.861 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8260000000000003e-05 2024-03-20T09:18:00.861 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.861 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:00.861 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.861 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.861 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.861 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212920 4.b 4:df609520:::benchmark_data_smithi122_145698_object92001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:18:00.861 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.384067+0000", 2024-03-20T09:18:00.861 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 43.391070724999999, 2024-03-20T09:18:00.861 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.93677413600000004, 2024-03-20T09:18:00.861 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:00.861 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:00.861 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:00.861 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:18:00.861 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:18:00.862 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212920 2024-03-20T09:18:00.862 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.862 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:00.862 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.862 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:00.862 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.384067+0000", 2024-03-20T09:18:00.862 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.862 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.862 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.862 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:00.862 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.384067+0000", 2024-03-20T09:18:00.862 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.862 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.862 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.863 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:00.863 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.384069+0000", 2024-03-20T09:18:00.863 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.854e-06 2024-03-20T09:18:00.863 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.863 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.863 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:00.863 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.384077+0000", 2024-03-20T09:18:00.863 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.464e-06 2024-03-20T09:18:00.863 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.863 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.863 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:00.863 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.384094+0000", 2024-03-20T09:18:00.863 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7396000000000001e-05 2024-03-20T09:18:00.863 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.863 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.863 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:00.864 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.384104+0000", 2024-03-20T09:18:00.864 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9490000000000008e-06 2024-03-20T09:18:00.864 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.864 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.864 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.864 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.384186+0000", 2024-03-20T09:18:00.864 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1691000000000001e-05 2024-03-20T09:18:00.864 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.864 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.864 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:18:00.864 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.384197+0000", 2024-03-20T09:18:00.864 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0913e-05 2024-03-20T09:18:00.864 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.865 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.865 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.865 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.285329+0000", 2024-03-20T09:18:00.865 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.90113206499999998 2024-03-20T09:18:00.865 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.865 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.865 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:00.865 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.285405+0000", 2024-03-20T09:18:00.865 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6494000000000003e-05 2024-03-20T09:18:00.865 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.865 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.865 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:18:00.865 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.285477+0000", 2024-03-20T09:18:00.865 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1557999999999995e-05 2024-03-20T09:18:00.865 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.865 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.866 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:00.866 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.290733+0000", 2024-03-20T09:18:00.866 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0052559299999999998 2024-03-20T09:18:00.866 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.866 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.866 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:18:00.866 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.290897+0000", 2024-03-20T09:18:00.866 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016413199999999999 2024-03-20T09:18:00.866 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.866 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.866 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:00.866 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.320755+0000", 2024-03-20T09:18:00.866 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.029858264999999998 2024-03-20T09:18:00.866 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.867 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.867 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:00.867 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.320801+0000", 2024-03-20T09:18:00.867 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5895e-05 2024-03-20T09:18:00.867 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.867 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.867 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:00.867 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.320841+0000", 2024-03-20T09:18:00.867 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.053e-05 2024-03-20T09:18:00.867 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.867 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:00.867 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.867 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.867 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.868 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212932 4.b 4:d9fbd8b0:::benchmark_data_smithi122_145698_object92013:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:18:00.868 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.407231+0000", 2024-03-20T09:18:00.868 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 43.367906929, 2024-03-20T09:18:00.868 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.91372579799999998, 2024-03-20T09:18:00.868 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:00.868 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:00.868 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:00.868 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:18:00.868 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:18:00.868 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212932 2024-03-20T09:18:00.868 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.868 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:00.868 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.868 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:00.869 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.407231+0000", 2024-03-20T09:18:00.869 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.869 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.869 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.869 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:00.869 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.407231+0000", 2024-03-20T09:18:00.869 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.869 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.869 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.869 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:00.869 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.407233+0000", 2024-03-20T09:18:00.869 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.624e-06 2024-03-20T09:18:00.869 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.869 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.869 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:00.869 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.407239+0000", 2024-03-20T09:18:00.870 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4049999999999997e-06 2024-03-20T09:18:00.870 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.870 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.870 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:00.870 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.407253+0000", 2024-03-20T09:18:00.870 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3497e-05 2024-03-20T09:18:00.870 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.870 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.870 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:00.870 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.407260+0000", 2024-03-20T09:18:00.870 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7209999999999994e-06 2024-03-20T09:18:00.870 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.870 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.870 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.871 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.407324+0000", 2024-03-20T09:18:00.871 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3841000000000001e-05 2024-03-20T09:18:00.871 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.871 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.871 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:18:00.871 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.407335+0000", 2024-03-20T09:18:00.871 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.111e-05 2024-03-20T09:18:00.871 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.871 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.871 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.871 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.287507+0000", 2024-03-20T09:18:00.871 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.88017162100000002 2024-03-20T09:18:00.871 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.871 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.871 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:00.872 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.287619+0000", 2024-03-20T09:18:00.872 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000111566 2024-03-20T09:18:00.872 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.872 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.872 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:18:00.872 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.287738+0000", 2024-03-20T09:18:00.872 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000119543 2024-03-20T09:18:00.872 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.872 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.872 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:00.872 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.290939+0000", 2024-03-20T09:18:00.872 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003200984 2024-03-20T09:18:00.872 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.872 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.872 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:18:00.873 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.291076+0000", 2024-03-20T09:18:00.873 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013691899999999999 2024-03-20T09:18:00.873 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.873 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.873 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:00.873 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.320890+0000", 2024-03-20T09:18:00.873 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.029814245999999999 2024-03-20T09:18:00.873 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.873 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.873 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:00.873 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.320922+0000", 2024-03-20T09:18:00.873 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2128999999999997e-05 2024-03-20T09:18:00.873 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.873 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.873 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:00.874 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.320957+0000", 2024-03-20T09:18:00.874 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4591999999999997e-05 2024-03-20T09:18:00.874 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.874 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:00.874 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.874 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.874 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.874 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212934 4.b 4:d00108c3:::benchmark_data_smithi122_145698_object92015:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:18:00.874 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.412231+0000", 2024-03-20T09:18:00.874 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 43.362907495999998, 2024-03-20T09:18:00.874 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.90885735899999998, 2024-03-20T09:18:00.874 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:00.874 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:00.875 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:00.875 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:18:00.875 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:18:00.875 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212934 2024-03-20T09:18:00.875 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.875 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:00.875 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.875 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:00.875 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.412231+0000", 2024-03-20T09:18:00.875 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.875 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.875 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.875 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:00.876 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.412231+0000", 2024-03-20T09:18:00.876 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.876 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.876 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.876 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:00.876 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.412232+0000", 2024-03-20T09:18:00.876 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9399999999999993e-07 2024-03-20T09:18:00.876 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.876 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.876 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:00.876 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.412240+0000", 2024-03-20T09:18:00.876 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8410000000000006e-06 2024-03-20T09:18:00.876 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.876 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.877 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:00.877 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.412251+0000", 2024-03-20T09:18:00.877 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1097999999999999e-05 2024-03-20T09:18:00.877 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.877 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.877 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:00.877 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.412258+0000", 2024-03-20T09:18:00.877 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5459999999999997e-06 2024-03-20T09:18:00.877 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.877 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.877 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.877 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.412309+0000", 2024-03-20T09:18:00.877 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1066000000000002e-05 2024-03-20T09:18:00.877 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.878 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.878 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:18:00.878 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.412316+0000", 2024-03-20T09:18:00.878 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1029999999999996e-06 2024-03-20T09:18:00.878 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.878 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.878 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.878 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.291236+0000", 2024-03-20T09:18:00.878 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.87891978299999995 2024-03-20T09:18:00.878 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.878 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.878 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:00.878 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.291334+0000", 2024-03-20T09:18:00.878 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7737000000000003e-05 2024-03-20T09:18:00.879 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.879 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.879 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:18:00.879 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.291401+0000", 2024-03-20T09:18:00.879 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7621999999999997e-05 2024-03-20T09:18:00.879 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.879 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.879 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:18:00.879 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.297085+0000", 2024-03-20T09:18:00.879 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.005683207 2024-03-20T09:18:00.879 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.879 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.879 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:00.879 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.297277+0000", 2024-03-20T09:18:00.879 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00019204000000000001 2024-03-20T09:18:00.880 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.880 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.880 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:00.880 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.321001+0000", 2024-03-20T09:18:00.880 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.023724881999999999 2024-03-20T09:18:00.880 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.880 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.880 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:00.880 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.321048+0000", 2024-03-20T09:18:00.880 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6252999999999999e-05 2024-03-20T09:18:00.880 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.880 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.880 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:00.880 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.321088+0000", 2024-03-20T09:18:00.881 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0187e-05 2024-03-20T09:18:00.881 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.881 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:00.881 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.881 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.881 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.881 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212946 4.b 4:d41502f2:::benchmark_data_smithi122_145698_object92027:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:18:00.881 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.427488+0000", 2024-03-20T09:18:00.881 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 43.347650324, 2024-03-20T09:18:00.881 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.89371687399999999, 2024-03-20T09:18:00.881 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:00.881 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:00.881 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:00.881 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:18:00.882 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:18:00.882 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212946 2024-03-20T09:18:00.882 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.882 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:00.882 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.882 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:00.882 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.427488+0000", 2024-03-20T09:18:00.882 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.882 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.882 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.882 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:00.882 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.427488+0000", 2024-03-20T09:18:00.882 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.882 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.883 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.883 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:00.883 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.427489+0000", 2024-03-20T09:18:00.883 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.601e-06 2024-03-20T09:18:00.883 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.883 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.883 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:00.883 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.427494+0000", 2024-03-20T09:18:00.883 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8980000000000004e-06 2024-03-20T09:18:00.883 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.883 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.883 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:00.883 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.427506+0000", 2024-03-20T09:18:00.883 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2071e-05 2024-03-20T09:18:00.884 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.884 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.884 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:00.884 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.427513+0000", 2024-03-20T09:18:00.884 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3690000000000003e-06 2024-03-20T09:18:00.884 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.884 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.884 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.884 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.427559+0000", 2024-03-20T09:18:00.884 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.621e-05 2024-03-20T09:18:00.884 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.884 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.884 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:18:00.884 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.427566+0000", 2024-03-20T09:18:00.884 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4379999999999998e-06 2024-03-20T09:18:00.885 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.885 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.885 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.885 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.293406+0000", 2024-03-20T09:18:00.885 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.86583917600000004 2024-03-20T09:18:00.885 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.885 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.885 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:00.885 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.293519+0000", 2024-03-20T09:18:00.885 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000113708 2024-03-20T09:18:00.885 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.885 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.885 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:18:00.885 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.293641+0000", 2024-03-20T09:18:00.886 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000121506 2024-03-20T09:18:00.886 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.886 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.886 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:18:00.886 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.297090+0000", 2024-03-20T09:18:00.886 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0034492540000000001 2024-03-20T09:18:00.886 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.886 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.886 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:00.886 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.297330+0000", 2024-03-20T09:18:00.886 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000240263 2024-03-20T09:18:00.886 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.886 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.886 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:00.887 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.321130+0000", 2024-03-20T09:18:00.887 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.023799415000000001 2024-03-20T09:18:00.887 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.887 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.887 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:00.887 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.321174+0000", 2024-03-20T09:18:00.887 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3903000000000003e-05 2024-03-20T09:18:00.887 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.887 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.887 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:00.887 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.321205+0000", 2024-03-20T09:18:00.887 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1062000000000002e-05 2024-03-20T09:18:00.887 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.888 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:00.888 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.888 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.888 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.888 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212960 4.b 4:d30e7857:::benchmark_data_smithi122_145698_object92041:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:18:00.888 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.451448+0000", 2024-03-20T09:18:00.888 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 43.323689666999996, 2024-03-20T09:18:00.888 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.86988812999999998, 2024-03-20T09:18:00.888 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:00.888 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:00.888 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:00.888 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:18:00.888 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:18:00.888 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212960 2024-03-20T09:18:00.889 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.889 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:00.889 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.889 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:00.889 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.451448+0000", 2024-03-20T09:18:00.889 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.889 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.889 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.889 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:00.889 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.451448+0000", 2024-03-20T09:18:00.889 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.889 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.889 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.889 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:00.889 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.451450+0000", 2024-03-20T09:18:00.890 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5829999999999999e-06 2024-03-20T09:18:00.890 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.890 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.890 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:00.890 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.451456+0000", 2024-03-20T09:18:00.890 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3389999999999998e-06 2024-03-20T09:18:00.890 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.890 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.890 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:00.890 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.451470+0000", 2024-03-20T09:18:00.890 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3434e-05 2024-03-20T09:18:00.890 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.890 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.890 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:00.891 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.451477+0000", 2024-03-20T09:18:00.891 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7340000000000008e-06 2024-03-20T09:18:00.891 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.891 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.891 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.891 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.451536+0000", 2024-03-20T09:18:00.891 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8105e-05 2024-03-20T09:18:00.891 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.891 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.891 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:18:00.891 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.451544+0000", 2024-03-20T09:18:00.891 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1799999999999996e-06 2024-03-20T09:18:00.891 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.891 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.892 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.892 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.297377+0000", 2024-03-20T09:18:00.892 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.84583357000000003 2024-03-20T09:18:00.892 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.892 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.892 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:00.892 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.297452+0000", 2024-03-20T09:18:00.892 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5074999999999995e-05 2024-03-20T09:18:00.892 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.892 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.892 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:18:00.892 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.297523+0000", 2024-03-20T09:18:00.892 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0479000000000001e-05 2024-03-20T09:18:00.892 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.893 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.893 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:18:00.893 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.304096+0000", 2024-03-20T09:18:00.893 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0065731899999999996 2024-03-20T09:18:00.893 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.893 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.893 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:00.893 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.304136+0000", 2024-03-20T09:18:00.893 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9736e-05 2024-03-20T09:18:00.893 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.893 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.893 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:00.893 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.321247+0000", 2024-03-20T09:18:00.893 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017111428000000001 2024-03-20T09:18:00.894 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.894 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.894 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:00.894 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.321305+0000", 2024-03-20T09:18:00.894 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7930999999999998e-05 2024-03-20T09:18:00.894 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.894 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.894 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:00.894 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.321337+0000", 2024-03-20T09:18:00.894 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1346000000000001e-05 2024-03-20T09:18:00.894 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.894 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:00.894 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.894 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.895 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.895 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212965 4.b 4:d1de5eee:::benchmark_data_smithi122_145698_object92046:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:18:00.895 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.460885+0000", 2024-03-20T09:18:00.895 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 43.314252772000003, 2024-03-20T09:18:00.895 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.86057768800000001, 2024-03-20T09:18:00.895 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:00.895 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:00.895 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:00.895 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:18:00.895 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:18:00.895 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212965 2024-03-20T09:18:00.895 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.895 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:00.895 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.896 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:00.896 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.460885+0000", 2024-03-20T09:18:00.896 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.896 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.896 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.896 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:00.896 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.460885+0000", 2024-03-20T09:18:00.896 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.896 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.896 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.896 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:00.896 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.460886+0000", 2024-03-20T09:18:00.896 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.096e-06 2024-03-20T09:18:00.896 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.897 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.897 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:00.897 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.460892+0000", 2024-03-20T09:18:00.897 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4539999999999998e-06 2024-03-20T09:18:00.897 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.897 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.897 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:00.897 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.460904+0000", 2024-03-20T09:18:00.897 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1807999999999999e-05 2024-03-20T09:18:00.897 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.897 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.897 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:00.897 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.460911+0000", 2024-03-20T09:18:00.898 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1010000000000003e-06 2024-03-20T09:18:00.898 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.898 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.898 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.898 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.460968+0000", 2024-03-20T09:18:00.898 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6895000000000003e-05 2024-03-20T09:18:00.898 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.898 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.898 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:18:00.898 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.460974+0000", 2024-03-20T09:18:00.898 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8220000000000001e-06 2024-03-20T09:18:00.898 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.898 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.899 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.899 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.299491+0000", 2024-03-20T09:18:00.899 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.83851649699999997 2024-03-20T09:18:00.899 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.899 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.899 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:00.899 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.299569+0000", 2024-03-20T09:18:00.899 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8438999999999999e-05 2024-03-20T09:18:00.899 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.899 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.899 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:18:00.899 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.299641+0000", 2024-03-20T09:18:00.899 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1711999999999994e-05 2024-03-20T09:18:00.899 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.900 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.900 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:18:00.900 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.304103+0000", 2024-03-20T09:18:00.900 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0044617800000000003 2024-03-20T09:18:00.900 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.900 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.900 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:00.900 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.304881+0000", 2024-03-20T09:18:00.900 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00077851000000000005 2024-03-20T09:18:00.900 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.900 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.900 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:00.900 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.321377+0000", 2024-03-20T09:18:00.901 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016495841000000001 2024-03-20T09:18:00.901 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.901 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.901 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:00.901 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.321426+0000", 2024-03-20T09:18:00.901 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8973e-05 2024-03-20T09:18:00.901 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.901 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.901 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:00.901 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.321463+0000", 2024-03-20T09:18:00.901 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.676e-05 2024-03-20T09:18:00.901 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.901 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:00.901 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.901 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.902 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.902 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212979 4.b 4:d162c53c:::benchmark_data_smithi122_145698_object92060:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:18:00.902 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.530256+0000", 2024-03-20T09:18:00.902 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 43.244882103000002, 2024-03-20T09:18:00.902 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.79561833800000004, 2024-03-20T09:18:00.902 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:00.902 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:00.902 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:00.902 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:18:00.902 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:18:00.902 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212979 2024-03-20T09:18:00.902 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.902 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:00.902 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.903 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:00.903 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.530256+0000", 2024-03-20T09:18:00.903 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.903 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.903 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.903 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:00.903 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.530256+0000", 2024-03-20T09:18:00.903 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:00.903 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.903 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.903 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:00.903 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.530258+0000", 2024-03-20T09:18:00.903 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.615e-06 2024-03-20T09:18:00.903 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.904 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.904 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:00.904 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.530274+0000", 2024-03-20T09:18:00.904 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6748999999999998e-05 2024-03-20T09:18:00.904 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.904 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.904 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:00.904 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.530300+0000", 2024-03-20T09:18:00.904 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5465000000000001e-05 2024-03-20T09:18:00.904 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.904 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.904 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:00.904 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.530309+0000", 2024-03-20T09:18:00.904 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7800000000000006e-06 2024-03-20T09:18:00.905 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.905 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.905 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.905 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.530372+0000", 2024-03-20T09:18:00.905 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3748e-05 2024-03-20T09:18:00.905 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.905 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.905 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:18:00.905 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.530382+0000", 2024-03-20T09:18:00.905 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4809999999999995e-06 2024-03-20T09:18:00.905 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.905 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.905 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:00.905 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.301707+0000", 2024-03-20T09:18:00.905 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.77132535499999999 2024-03-20T09:18:00.906 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.906 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.906 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:00.906 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.301835+0000", 2024-03-20T09:18:00.906 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012744000000000001 2024-03-20T09:18:00.906 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.906 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.906 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:18:00.906 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.301965+0000", 2024-03-20T09:18:00.906 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013019 2024-03-20T09:18:00.906 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.906 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.906 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:18:00.906 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.304214+0000", 2024-03-20T09:18:00.907 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0022491830000000001 2024-03-20T09:18:00.907 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.907 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.907 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:00.907 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.304972+0000", 2024-03-20T09:18:00.907 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00075838700000000002 2024-03-20T09:18:00.907 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.907 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.907 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:00.907 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.325752+0000", 2024-03-20T09:18:00.907 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.020779410000000002 2024-03-20T09:18:00.907 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.907 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.907 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:00.908 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.325808+0000", 2024-03-20T09:18:00.908 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6017999999999998e-05 2024-03-20T09:18:00.908 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:00.908 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:00.908 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:00.908 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.325874+0000", 2024-03-20T09:18:00.908 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6517e-05 2024-03-20T09:18:00.908 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.908 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:00.908 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.908 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:00.908 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:00.908 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:18:01.071 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:18:01.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:01.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:18:01.176 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:18:01.204 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:01.261 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:18:01.419 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:01 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2085191004' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-20T09:18:01.420 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:01 smithi122 ceph-mon[73495]: osdmap e603: 12 total, 12 up, 10 in 2024-03-20T09:18:01.446 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_ops_in_flight 2024-03-20T09:18:01.453 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:18:01.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:01.455 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:18:01.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:01 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2085191004' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-20T09:18:01.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:01 smithi026 ceph-mon[68309]: osdmap e603: 12 total, 12 up, 10 in 2024-03-20T09:18:01.556 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:18:01.582 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:01.609 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:18:01.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:01.612 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:18:01.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:01 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2085191004' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-20T09:18:01.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:01 smithi115 ceph-mon[72476]: osdmap e603: 12 total, 12 up, 10 in 2024-03-20T09:18:01.712 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:18:01.724 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:18:01.740 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:01.941 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:18:01.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:01.944 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:18:02.045 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:18:02.077 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:02.098 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:18:02.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:02.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:18:02.202 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:18:02.232 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:02.355 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:02 smithi026 ceph-mon[68309]: pgmap v1446: 31 pgs: 31 active+clean; 577 KiB data, 3.7 GiB used, 890 GiB / 894 GiB avail 2024-03-20T09:18:02.430 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:18:02.431 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:02 smithi122 ceph-mon[73495]: pgmap v1446: 31 pgs: 31 active+clean; 577 KiB data, 3.7 GiB used, 890 GiB / 894 GiB avail 2024-03-20T09:18:02.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:02.432 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:18:02.533 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:18:02.564 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:02 smithi115 ceph-mon[72476]: pgmap v1446: 31 pgs: 31 active+clean; 577 KiB data, 3.7 GiB used, 890 GiB / 894 GiB avail 2024-03-20T09:18:02.567 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:02.588 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:18:02.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:02.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:18:02.692 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:18:02.725 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:03.031 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-20T09:18:03.032 INFO:tasks.radosbench:Beginning radosbench... 2024-03-20T09:18:03.032 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-03-20T09:18:03.032 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pool create unique_pool_3 16 2024-03-20T09:18:03.097 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:18:03.335 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:18:03.335 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:18:03.335 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-03-20T09:18:03.335 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:18:03.925 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:03 smithi122 ceph-mon[73495]: 1.18 scrub starts 2024-03-20T09:18:03.925 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:03 smithi122 ceph-mon[73495]: 1.18 scrub ok 2024-03-20T09:18:03.925 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:18:03.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:03.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:18:03.930 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:03 smithi026 ceph-mon[68309]: 1.18 scrub starts 2024-03-20T09:18:03.930 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:03 smithi026 ceph-mon[68309]: 1.18 scrub ok 2024-03-20T09:18:03.949 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_blocked_ops 2024-03-20T09:18:04.031 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:18:04.060 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:04.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:03 smithi115 ceph-mon[72476]: 1.18 scrub starts 2024-03-20T09:18:04.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:03 smithi115 ceph-mon[72476]: 1.18 scrub ok 2024-03-20T09:18:04.209 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:18:04.611 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:18:04.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:04.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:18:04.715 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:18:04.743 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:04.942 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:18:04.944 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:04 smithi122 ceph-mon[73495]: pgmap v1447: 31 pgs: 31 active+clean; 577 KiB data, 3.7 GiB used, 890 GiB / 894 GiB avail 2024-03-20T09:18:04.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:04.945 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:18:04.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:04 smithi026 ceph-mon[68309]: pgmap v1447: 31 pgs: 31 active+clean; 577 KiB data, 3.7 GiB used, 890 GiB / 894 GiB avail 2024-03-20T09:18:05.045 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:18:05.077 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:05.099 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:18:05.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:05.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:18:05.108 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:18:05.110 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:18:05.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:04 smithi115 ceph-mon[72476]: pgmap v1447: 31 pgs: 31 active+clean; 577 KiB data, 3.7 GiB used, 890 GiB / 894 GiB avail 2024-03-20T09:18:05.203 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:18:05.232 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:05.384 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:18:05.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:05.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:18:05.487 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:18:05.521 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:05.542 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:18:05.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:05.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:18:05.646 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:18:05.675 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:05.690 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:18:05.691 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.8 to 1 (10000) 2024-03-20T09:18:05.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:05 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:18:05.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:05 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:18:05.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:05 smithi115 ceph-mon[72476]: 1.d scrub starts 2024-03-20T09:18:05.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:05 smithi115 ceph-mon[72476]: 1.d scrub ok 2024-03-20T09:18:05.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:05 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2615153356' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:18:05.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:05 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2243476498' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:18:05.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:05 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:18:05.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:05 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:18:05.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:05 smithi122 ceph-mon[73495]: 1.d scrub starts 2024-03-20T09:18:05.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:05 smithi122 ceph-mon[73495]: 1.d scrub ok 2024-03-20T09:18:05.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:05 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2615153356' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:18:05.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:05 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2243476498' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:18:05.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:05 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:18:05.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:05 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:18:05.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:05 smithi026 ceph-mon[68309]: 1.d scrub starts 2024-03-20T09:18:05.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:05 smithi026 ceph-mon[68309]: 1.d scrub ok 2024-03-20T09:18:05.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:05 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2615153356' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:18:05.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:05 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2243476498' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:18:05.989 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:18:05.989 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:18:05.989 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-03-20T09:18:05.989 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-03-20T09:18:05.989 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:18:06.598 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_historic_ops 2024-03-20T09:18:06.860 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:18:06.882 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:06 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:18:06.882 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:06 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2615153356' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-03-20T09:18:06.882 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:06 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2243476498' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:18:06.882 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:06 smithi026 ceph-mon[68309]: osdmap e604: 12 total, 12 up, 10 in 2024-03-20T09:18:06.882 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:06 smithi026 ceph-mon[68309]: 1.1e scrub starts 2024-03-20T09:18:06.882 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:06 smithi026 ceph-mon[68309]: 1.1e scrub ok 2024-03-20T09:18:06.882 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:06 smithi026 ceph-mon[68309]: pgmap v1449: 31 pgs: 31 active+clean; 577 KiB data, 3.8 GiB used, 890 GiB / 894 GiB avail 2024-03-20T09:18:06.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:06 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:18:06.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:06 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2615153356' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-03-20T09:18:06.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:06 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2243476498' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:18:06.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:06 smithi122 ceph-mon[73495]: osdmap e604: 12 total, 12 up, 10 in 2024-03-20T09:18:06.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:06 smithi122 ceph-mon[73495]: 1.1e scrub starts 2024-03-20T09:18:06.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:06 smithi122 ceph-mon[73495]: 1.1e scrub ok 2024-03-20T09:18:06.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:06 smithi122 ceph-mon[73495]: pgmap v1449: 31 pgs: 31 active+clean; 577 KiB data, 3.8 GiB used, 890 GiB / 894 GiB avail 2024-03-20T09:18:07.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:06 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:18:07.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:06 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2615153356' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-03-20T09:18:07.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:06 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2243476498' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:18:07.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:06 smithi115 ceph-mon[72476]: osdmap e604: 12 total, 12 up, 10 in 2024-03-20T09:18:07.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:06 smithi115 ceph-mon[72476]: 1.1e scrub starts 2024-03-20T09:18:07.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:06 smithi115 ceph-mon[72476]: 1.1e scrub ok 2024-03-20T09:18:07.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:06 smithi115 ceph-mon[72476]: pgmap v1449: 31 pgs: 31 active+clean; 577 KiB data, 3.8 GiB used, 890 GiB / 894 GiB avail 2024-03-20T09:18:07.515 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 10 1 2024-03-20T09:18:07.741 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:18:07.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:07.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:18:07.845 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:18:07.878 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:07.899 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:18:07.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:07.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:18:08.003 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:18:08.033 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:08.545 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:18:08.546 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-03-20T09:18:08.546 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-03-20T09:18:08.546 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-03-20T09:18:08.546 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.546 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.24910.0:142022 3.d e429/275 3:bca4a78c:::benchmark_data_smithi122_119671_object42240:head v 429'8759, mlcod=256'4799)", 2024-03-20T09:18:08.546 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:10:07.207810+0000", 2024-03-20T09:18:08.546 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 481.33566535400001, 2024-03-20T09:18:08.546 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0054674040000000004, 2024-03-20T09:18:08.546 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:08.547 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:08.547 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:08.547 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.549 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:08.550 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:10:07.207810+0000", 2024-03-20T09:18:08.550 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.550 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.550 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.550 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:08.550 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:10:07.207810+0000", 2024-03-20T09:18:08.550 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.550 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.550 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.550 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:08.550 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:10:07.207811+0000", 2024-03-20T09:18:08.550 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5179999999999999e-06 2024-03-20T09:18:08.550 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.551 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.551 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:08.551 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:10:07.207822+0000", 2024-03-20T09:18:08.551 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0226e-05 2024-03-20T09:18:08.551 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.551 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.551 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:08.551 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:10:07.207843+0000", 2024-03-20T09:18:08.551 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1141999999999998e-05 2024-03-20T09:18:08.551 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.551 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.551 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:08.551 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:10:07.207851+0000", 2024-03-20T09:18:08.551 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3590000000000008e-06 2024-03-20T09:18:08.551 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.552 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.552 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:08.552 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:10:07.207933+0000", 2024-03-20T09:18:08.552 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2238999999999996e-05 2024-03-20T09:18:08.552 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.552 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.552 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:08.552 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:10:07.207962+0000", 2024-03-20T09:18:08.552 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8520000000000001e-05 2024-03-20T09:18:08.552 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.552 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.553 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:08.553 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:10:07.213237+0000", 2024-03-20T09:18:08.553 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0052752600000000004 2024-03-20T09:18:08.553 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.553 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.553 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:08.553 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:10:07.213277+0000", 2024-03-20T09:18:08.553 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0139999999999999e-05 2024-03-20T09:18:08.553 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.553 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:08.553 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.553 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.554 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.554 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e462/434 cost 1049576)", 2024-03-20T09:18:08.554 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:09.212827+0000", 2024-03-20T09:18:08.554 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 419.330648209, 2024-03-20T09:18:08.554 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00099083599999999993, 2024-03-20T09:18:08.554 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:08.554 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:18:08.554 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:08.554 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.554 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:08.554 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:09.212827+0000", 2024-03-20T09:18:08.554 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.554 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.554 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.555 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:08.555 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:09.212827+0000", 2024-03-20T09:18:08.555 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.555 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.555 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.555 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:08.555 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:09.212828+0000", 2024-03-20T09:18:08.555 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.037e-06 2024-03-20T09:18:08.555 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.555 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.555 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:08.555 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:09.212840+0000", 2024-03-20T09:18:08.555 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1929e-05 2024-03-20T09:18:08.555 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.556 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.556 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:08.556 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:09.212852+0000", 2024-03-20T09:18:08.556 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2148e-05 2024-03-20T09:18:08.556 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.556 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.556 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:08.556 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:09.212861+0000", 2024-03-20T09:18:08.556 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4519999999999997e-06 2024-03-20T09:18:08.556 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.556 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.556 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:08.556 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:09.212951+0000", 2024-03-20T09:18:08.556 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0557999999999996e-05 2024-03-20T09:18:08.557 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.557 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.557 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:08.557 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:09.213818+0000", 2024-03-20T09:18:08.557 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00086671199999999995 2024-03-20T09:18:08.557 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.557 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:08.557 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.557 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.557 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.557 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e465/434 cost 1049576)", 2024-03-20T09:18:08.557 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:17.855044+0000", 2024-03-20T09:18:08.557 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 410.68843169799999, 2024-03-20T09:18:08.557 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00072467199999999999, 2024-03-20T09:18:08.558 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:08.558 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:18:08.558 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:08.558 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.558 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:08.558 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:17.855044+0000", 2024-03-20T09:18:08.558 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.558 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.558 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.558 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:08.558 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:17.855044+0000", 2024-03-20T09:18:08.558 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.558 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.559 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.559 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:08.559 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:17.855046+0000", 2024-03-20T09:18:08.559 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0329999999999998e-06 2024-03-20T09:18:08.559 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.559 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.559 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:08.559 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:17.855068+0000", 2024-03-20T09:18:08.559 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2833e-05 2024-03-20T09:18:08.559 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.559 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.559 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:08.559 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:17.855096+0000", 2024-03-20T09:18:08.559 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7419999999999998e-05 2024-03-20T09:18:08.559 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.559 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.560 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:08.560 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:17.855118+0000", 2024-03-20T09:18:08.560 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1668999999999999e-05 2024-03-20T09:18:08.560 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.560 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.560 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:08.560 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:17.855224+0000", 2024-03-20T09:18:08.560 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010615099999999999 2024-03-20T09:18:08.560 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.560 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.560 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:08.560 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:17.855768+0000", 2024-03-20T09:18:08.560 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00054456600000000002 2024-03-20T09:18:08.561 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.561 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:08.561 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.561 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.561 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.561 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e472/434 cost 1049576)", 2024-03-20T09:18:08.561 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:35.044500+0000", 2024-03-20T09:18:08.561 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 393.498975186, 2024-03-20T09:18:08.561 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000590983, 2024-03-20T09:18:08.561 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:08.561 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:18:08.561 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:08.561 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.561 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:08.562 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.044500+0000", 2024-03-20T09:18:08.562 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.562 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.562 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.562 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:08.562 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.044500+0000", 2024-03-20T09:18:08.562 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.562 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.562 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.562 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:08.562 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.044501+0000", 2024-03-20T09:18:08.562 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9899999999999999e-07 2024-03-20T09:18:08.563 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.563 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.563 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:08.563 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.044513+0000", 2024-03-20T09:18:08.563 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1574e-05 2024-03-20T09:18:08.563 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.563 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.563 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:08.563 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.044538+0000", 2024-03-20T09:18:08.563 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4909000000000002e-05 2024-03-20T09:18:08.563 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.563 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.564 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:08.564 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.044560+0000", 2024-03-20T09:18:08.564 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2742999999999999e-05 2024-03-20T09:18:08.564 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.564 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.564 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:08.564 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.044616+0000", 2024-03-20T09:18:08.564 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5472e-05 2024-03-20T09:18:08.564 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.564 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.564 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:08.564 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.045091+0000", 2024-03-20T09:18:08.564 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00047538600000000001 2024-03-20T09:18:08.564 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.565 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:08.565 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.565 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.565 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.565 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e472/434 cost 1049576)", 2024-03-20T09:18:08.565 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:35.123747+0000", 2024-03-20T09:18:08.565 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 393.41972810599998, 2024-03-20T09:18:08.565 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00073770699999999999, 2024-03-20T09:18:08.565 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:08.565 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:18:08.565 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:08.565 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.565 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:08.565 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.123747+0000", 2024-03-20T09:18:08.566 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.566 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.566 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.566 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:08.566 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.123747+0000", 2024-03-20T09:18:08.566 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.566 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.566 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.566 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:08.566 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.123749+0000", 2024-03-20T09:18:08.566 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5030000000000001e-06 2024-03-20T09:18:08.566 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.566 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.566 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:08.567 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.123768+0000", 2024-03-20T09:18:08.567 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.893e-05 2024-03-20T09:18:08.567 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.567 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.567 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:08.567 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.123791+0000", 2024-03-20T09:18:08.567 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3166000000000001e-05 2024-03-20T09:18:08.567 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.567 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.567 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:08.567 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.123803+0000", 2024-03-20T09:18:08.567 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2517000000000001e-05 2024-03-20T09:18:08.567 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.567 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.568 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:08.568 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.123924+0000", 2024-03-20T09:18:08.568 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000120413 2024-03-20T09:18:08.568 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.568 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.568 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:08.568 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.124485+0000", 2024-03-20T09:18:08.568 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00056117799999999996 2024-03-20T09:18:08.568 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.568 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:08.568 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.568 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.568 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.568 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e472/434 cost 1049576)", 2024-03-20T09:18:08.569 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:35.343418+0000", 2024-03-20T09:18:08.569 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 393.20005779799999, 2024-03-20T09:18:08.569 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00064077999999999997, 2024-03-20T09:18:08.569 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:08.569 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:18:08.569 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:08.569 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.569 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:08.569 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.343418+0000", 2024-03-20T09:18:08.569 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.569 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.569 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.569 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:08.569 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.343418+0000", 2024-03-20T09:18:08.570 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.570 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.570 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.570 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:08.570 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.343419+0000", 2024-03-20T09:18:08.570 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7239999999999999e-06 2024-03-20T09:18:08.570 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.570 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.570 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:08.570 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.343442+0000", 2024-03-20T09:18:08.571 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2883000000000001e-05 2024-03-20T09:18:08.571 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.571 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.571 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:08.571 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.343469+0000", 2024-03-20T09:18:08.571 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7216000000000002e-05 2024-03-20T09:18:08.571 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.571 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.571 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:08.571 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.343484+0000", 2024-03-20T09:18:08.571 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4333e-05 2024-03-20T09:18:08.571 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.571 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.571 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:08.571 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.343591+0000", 2024-03-20T09:18:08.572 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010685 2024-03-20T09:18:08.572 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.572 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.572 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:08.572 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:35.344058+0000", 2024-03-20T09:18:08.572 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00046777400000000001 2024-03-20T09:18:08.572 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.572 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:08.572 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.572 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.572 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.572 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e472/434 cost 1049576)", 2024-03-20T09:18:08.572 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:36.828898+0000", 2024-03-20T09:18:08.573 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 391.71457708299999, 2024-03-20T09:18:08.573 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00059100600000000004, 2024-03-20T09:18:08.573 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:08.573 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:18:08.573 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:08.573 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.573 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:08.573 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:36.828898+0000", 2024-03-20T09:18:08.573 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.573 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.573 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.573 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:08.573 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:36.828898+0000", 2024-03-20T09:18:08.573 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.574 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.574 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.574 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:08.574 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:36.828899+0000", 2024-03-20T09:18:08.574 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1060000000000001e-06 2024-03-20T09:18:08.574 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.574 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.574 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:08.574 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:36.828912+0000", 2024-03-20T09:18:08.574 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3091000000000001e-05 2024-03-20T09:18:08.574 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.574 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.574 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:08.574 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:36.828925+0000", 2024-03-20T09:18:08.574 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2819e-05 2024-03-20T09:18:08.575 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.575 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.575 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:08.575 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:36.828950+0000", 2024-03-20T09:18:08.575 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4729999999999999e-05 2024-03-20T09:18:08.575 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.575 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.575 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:08.575 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:36.829001+0000", 2024-03-20T09:18:08.575 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0670000000000001e-05 2024-03-20T09:18:08.575 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.575 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.575 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:08.575 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:36.829489+0000", 2024-03-20T09:18:08.576 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00048859000000000001 2024-03-20T09:18:08.576 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.576 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:08.576 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.576 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.576 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.576 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e473/434 cost 1049576)", 2024-03-20T09:18:08.576 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:38.408069+0000", 2024-03-20T09:18:08.576 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 390.13540605700001, 2024-03-20T09:18:08.576 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00062260899999999999, 2024-03-20T09:18:08.576 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:08.576 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:18:08.576 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:08.576 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.577 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:08.577 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:38.408069+0000", 2024-03-20T09:18:08.577 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.577 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.577 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.577 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:08.577 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:38.408069+0000", 2024-03-20T09:18:08.577 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.577 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.577 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.578 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:08.578 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:38.408070+0000", 2024-03-20T09:18:08.578 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.24e-06 2024-03-20T09:18:08.578 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.578 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.578 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:08.578 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:38.408085+0000", 2024-03-20T09:18:08.578 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4166e-05 2024-03-20T09:18:08.578 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.578 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.578 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:08.578 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:38.408101+0000", 2024-03-20T09:18:08.578 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5991e-05 2024-03-20T09:18:08.578 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.579 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.579 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:08.579 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:38.408110+0000", 2024-03-20T09:18:08.579 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5689999999999994e-06 2024-03-20T09:18:08.579 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.579 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.579 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:08.579 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:38.408171+0000", 2024-03-20T09:18:08.579 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0745999999999997e-05 2024-03-20T09:18:08.579 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.579 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.579 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:08.579 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:38.408692+0000", 2024-03-20T09:18:08.579 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00052089699999999998 2024-03-20T09:18:08.580 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.580 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:08.580 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.580 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.580 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.580 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e473/434 cost 1049576)", 2024-03-20T09:18:08.580 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:39.640113+0000", 2024-03-20T09:18:08.580 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 388.90336264799998, 2024-03-20T09:18:08.580 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000629661, 2024-03-20T09:18:08.580 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:08.580 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:18:08.580 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:08.580 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.580 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:08.580 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:39.640113+0000", 2024-03-20T09:18:08.581 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.581 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.581 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.581 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:08.581 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:39.640113+0000", 2024-03-20T09:18:08.581 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.581 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.581 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.581 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:08.581 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:39.640114+0000", 2024-03-20T09:18:08.581 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.046e-06 2024-03-20T09:18:08.581 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.581 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.581 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:08.582 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:39.640127+0000", 2024-03-20T09:18:08.582 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3074999999999999e-05 2024-03-20T09:18:08.582 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.582 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.582 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:08.582 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:39.640140+0000", 2024-03-20T09:18:08.582 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3112000000000001e-05 2024-03-20T09:18:08.582 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.582 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.582 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:08.582 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:39.640147+0000", 2024-03-20T09:18:08.582 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1069999999999999e-06 2024-03-20T09:18:08.582 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.582 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.582 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:08.583 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:39.640223+0000", 2024-03-20T09:18:08.583 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6462000000000001e-05 2024-03-20T09:18:08.583 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.583 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.583 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:08.583 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:39.640742+0000", 2024-03-20T09:18:08.583 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00051885899999999999 2024-03-20T09:18:08.583 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.583 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:08.583 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.583 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.583 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.583 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e473/434 cost 1049576)", 2024-03-20T09:18:08.583 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:39.681702+0000", 2024-03-20T09:18:08.584 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 388.86177341500002, 2024-03-20T09:18:08.584 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00060000500000000003, 2024-03-20T09:18:08.584 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:08.584 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:18:08.584 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:08.584 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.584 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:08.584 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:39.681702+0000", 2024-03-20T09:18:08.584 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.584 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.584 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.584 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:08.584 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:39.681702+0000", 2024-03-20T09:18:08.584 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.584 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.585 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.585 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:08.585 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:39.681703+0000", 2024-03-20T09:18:08.585 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.08e-06 2024-03-20T09:18:08.585 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.585 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.585 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:08.585 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:39.681713+0000", 2024-03-20T09:18:08.585 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.8349999999999999e-06 2024-03-20T09:18:08.585 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.585 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.585 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:08.585 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:39.681724+0000", 2024-03-20T09:18:08.585 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1022000000000001e-05 2024-03-20T09:18:08.585 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.586 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.586 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:08.586 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:39.681730+0000", 2024-03-20T09:18:08.586 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5880000000000003e-06 2024-03-20T09:18:08.586 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.586 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.586 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:08.586 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:39.681781+0000", 2024-03-20T09:18:08.586 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0532999999999999e-05 2024-03-20T09:18:08.586 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.586 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.586 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:08.586 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:39.682302+0000", 2024-03-20T09:18:08.586 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00052094700000000001 2024-03-20T09:18:08.587 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.587 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:08.587 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.587 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.587 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.587 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e475/434 cost 1049576)", 2024-03-20T09:18:08.587 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:42.972719+0000", 2024-03-20T09:18:08.587 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 385.570756184, 2024-03-20T09:18:08.587 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00060214000000000003, 2024-03-20T09:18:08.587 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:08.587 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:18:08.587 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:08.587 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.587 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:08.588 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:42.972719+0000", 2024-03-20T09:18:08.588 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.588 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.588 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.588 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:08.588 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:42.972719+0000", 2024-03-20T09:18:08.588 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.588 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.588 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.588 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:08.588 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:42.972720+0000", 2024-03-20T09:18:08.588 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.035e-06 2024-03-20T09:18:08.588 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.588 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.588 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:08.589 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:42.972731+0000", 2024-03-20T09:18:08.589 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0868e-05 2024-03-20T09:18:08.589 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.589 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.589 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:08.589 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:42.972744+0000", 2024-03-20T09:18:08.589 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3053e-05 2024-03-20T09:18:08.589 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.589 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.589 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:08.589 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:42.972751+0000", 2024-03-20T09:18:08.589 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.126e-06 2024-03-20T09:18:08.589 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.589 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.590 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:08.590 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:42.972816+0000", 2024-03-20T09:18:08.590 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4380000000000004e-05 2024-03-20T09:18:08.590 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.590 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.590 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:08.590 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:42.973321+0000", 2024-03-20T09:18:08.590 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00050567799999999997 2024-03-20T09:18:08.590 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.590 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:08.590 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.590 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.590 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.590 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e475/434 cost 1049576)", 2024-03-20T09:18:08.590 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:43.069398+0000", 2024-03-20T09:18:08.591 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 385.47407770799998, 2024-03-20T09:18:08.591 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00064565900000000003, 2024-03-20T09:18:08.591 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:08.591 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:18:08.591 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:08.591 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.591 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:08.591 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:43.069398+0000", 2024-03-20T09:18:08.591 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.591 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.591 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.591 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:08.591 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:43.069398+0000", 2024-03-20T09:18:08.591 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.592 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.592 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.592 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:08.592 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:43.069399+0000", 2024-03-20T09:18:08.592 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2190000000000001e-06 2024-03-20T09:18:08.592 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.592 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.592 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:08.592 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:43.069411+0000", 2024-03-20T09:18:08.592 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2500999999999999e-05 2024-03-20T09:18:08.592 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.592 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.592 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:08.592 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:43.069424+0000", 2024-03-20T09:18:08.592 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2958e-05 2024-03-20T09:18:08.593 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.593 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.593 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:08.593 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:43.069438+0000", 2024-03-20T09:18:08.593 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3906e-05 2024-03-20T09:18:08.593 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.593 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.593 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:08.593 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:43.069516+0000", 2024-03-20T09:18:08.593 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7899999999999996e-05 2024-03-20T09:18:08.593 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.593 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.593 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:08.593 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:43.070043+0000", 2024-03-20T09:18:08.594 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00052717499999999995 2024-03-20T09:18:08.594 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.594 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:08.594 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.594 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.594 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.594 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e475/434 cost 1049576)", 2024-03-20T09:18:08.594 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:43.132460+0000", 2024-03-20T09:18:08.594 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 385.41101496099998, 2024-03-20T09:18:08.594 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00060327799999999995, 2024-03-20T09:18:08.594 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:08.594 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:18:08.594 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:08.594 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.595 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:08.595 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:43.132460+0000", 2024-03-20T09:18:08.595 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.595 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.595 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.595 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:08.595 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:43.132460+0000", 2024-03-20T09:18:08.595 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.595 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.595 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.595 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:08.595 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:43.132461+0000", 2024-03-20T09:18:08.595 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9100000000000004e-07 2024-03-20T09:18:08.595 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.595 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.596 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:08.596 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:43.132471+0000", 2024-03-20T09:18:08.596 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0508e-05 2024-03-20T09:18:08.596 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.596 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.596 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:08.596 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:43.132480+0000", 2024-03-20T09:18:08.596 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5650000000000002e-06 2024-03-20T09:18:08.596 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.596 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.596 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:08.596 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:43.132485+0000", 2024-03-20T09:18:08.596 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7670000000000003e-06 2024-03-20T09:18:08.596 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.597 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.597 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:08.597 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:43.132542+0000", 2024-03-20T09:18:08.597 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7463000000000002e-05 2024-03-20T09:18:08.597 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.597 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.597 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:08.597 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:43.133064+0000", 2024-03-20T09:18:08.597 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00052138399999999995 2024-03-20T09:18:08.597 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.597 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:08.597 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.597 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.597 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.597 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e475/434 cost 1049576)", 2024-03-20T09:18:08.598 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:44.577581+0000", 2024-03-20T09:18:08.598 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 383.96589454000002, 2024-03-20T09:18:08.598 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00083124100000000003, 2024-03-20T09:18:08.598 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:08.598 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:18:08.598 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:08.598 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.598 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:08.598 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:44.577581+0000", 2024-03-20T09:18:08.598 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.598 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.598 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.598 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:08.598 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:44.577581+0000", 2024-03-20T09:18:08.599 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.599 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.599 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.599 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:08.599 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:44.577583+0000", 2024-03-20T09:18:08.599 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0030000000000001e-06 2024-03-20T09:18:08.599 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.599 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.599 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:08.599 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:44.577605+0000", 2024-03-20T09:18:08.599 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2198999999999999e-05 2024-03-20T09:18:08.599 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.599 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.599 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:08.600 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:44.577633+0000", 2024-03-20T09:18:08.600 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8019999999999999e-05 2024-03-20T09:18:08.600 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.600 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.600 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:08.600 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:44.577652+0000", 2024-03-20T09:18:08.600 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8516e-05 2024-03-20T09:18:08.600 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.600 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.600 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:08.600 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:44.577760+0000", 2024-03-20T09:18:08.600 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000108724 2024-03-20T09:18:08.600 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.600 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.601 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:08.601 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:44.578412+0000", 2024-03-20T09:18:08.601 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00065177900000000001 2024-03-20T09:18:08.601 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.601 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:08.601 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.601 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.601 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.601 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e476/434 cost 1049576)", 2024-03-20T09:18:08.601 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:46.038141+0000", 2024-03-20T09:18:08.601 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 382.505334527, 2024-03-20T09:18:08.601 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002222365, 2024-03-20T09:18:08.601 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:08.601 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:18:08.601 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:08.602 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.602 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:08.602 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.038141+0000", 2024-03-20T09:18:08.602 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.602 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.602 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.602 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:08.602 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.038141+0000", 2024-03-20T09:18:08.602 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.602 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.602 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.602 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:08.602 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.038142+0000", 2024-03-20T09:18:08.603 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0249999999999999e-06 2024-03-20T09:18:08.603 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.603 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.603 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:08.603 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.038154+0000", 2024-03-20T09:18:08.603 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2379e-05 2024-03-20T09:18:08.603 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.603 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.603 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:08.603 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.038168+0000", 2024-03-20T09:18:08.603 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3329000000000001e-05 2024-03-20T09:18:08.603 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.603 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.603 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:08.603 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.038175+0000", 2024-03-20T09:18:08.604 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0790000000000003e-06 2024-03-20T09:18:08.604 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.604 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.604 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:08.604 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.038231+0000", 2024-03-20T09:18:08.604 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6305999999999999e-05 2024-03-20T09:18:08.604 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.604 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.604 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:08.604 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.040363+0000", 2024-03-20T09:18:08.604 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0021322469999999999 2024-03-20T09:18:08.604 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.604 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:08.604 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.605 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.605 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.605 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e476/434 cost 1049576)", 2024-03-20T09:18:08.605 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:46.647115+0000", 2024-03-20T09:18:08.605 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 381.89635997400001, 2024-03-20T09:18:08.605 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00065813999999999998, 2024-03-20T09:18:08.605 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:08.605 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:18:08.605 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:08.605 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.605 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:08.605 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.647115+0000", 2024-03-20T09:18:08.605 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.605 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.605 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.606 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:08.606 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.647115+0000", 2024-03-20T09:18:08.606 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.606 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.606 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.606 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:08.606 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.647117+0000", 2024-03-20T09:18:08.606 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0700000000000001e-06 2024-03-20T09:18:08.606 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.606 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.606 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:08.606 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.647141+0000", 2024-03-20T09:18:08.606 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3346e-05 2024-03-20T09:18:08.606 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.607 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.607 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:08.607 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.647168+0000", 2024-03-20T09:18:08.607 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7029000000000002e-05 2024-03-20T09:18:08.607 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.607 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.607 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:08.607 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.647182+0000", 2024-03-20T09:18:08.607 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3747000000000001e-05 2024-03-20T09:18:08.607 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.607 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.607 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:08.607 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.647306+0000", 2024-03-20T09:18:08.607 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000124766 2024-03-20T09:18:08.608 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.608 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.608 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:08.608 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.647773+0000", 2024-03-20T09:18:08.608 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00046718200000000001 2024-03-20T09:18:08.608 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.608 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:08.608 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.608 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.608 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.608 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e476/434 cost 1049576)", 2024-03-20T09:18:08.608 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:46.916404+0000", 2024-03-20T09:18:08.608 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 381.62707150699998, 2024-03-20T09:18:08.608 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00060503900000000003, 2024-03-20T09:18:08.608 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:08.609 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:18:08.609 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:08.609 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.609 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:08.609 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.916404+0000", 2024-03-20T09:18:08.609 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.609 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.609 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.609 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:08.609 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.916404+0000", 2024-03-20T09:18:08.609 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.609 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.609 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.609 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:08.610 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.916405+0000", 2024-03-20T09:18:08.610 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0499999999999999e-06 2024-03-20T09:18:08.610 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.610 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.610 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:08.610 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.916416+0000", 2024-03-20T09:18:08.610 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0824e-05 2024-03-20T09:18:08.610 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.610 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.610 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:08.610 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.916427+0000", 2024-03-20T09:18:08.610 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0875e-05 2024-03-20T09:18:08.610 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.611 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.611 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:08.611 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.916451+0000", 2024-03-20T09:18:08.611 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4162e-05 2024-03-20T09:18:08.611 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.611 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.611 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:08.611 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.916499+0000", 2024-03-20T09:18:08.611 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8445000000000001e-05 2024-03-20T09:18:08.611 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.611 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.611 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:08.611 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:46.917009+0000", 2024-03-20T09:18:08.611 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00050968300000000003 2024-03-20T09:18:08.612 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.612 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:08.612 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.612 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.612 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.612 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e476/434 cost 1049576)", 2024-03-20T09:18:08.612 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:47.042968+0000", 2024-03-20T09:18:08.612 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 381.500507658, 2024-03-20T09:18:08.612 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00058974899999999998, 2024-03-20T09:18:08.612 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:08.612 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:18:08.612 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:08.612 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.612 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:08.613 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.042968+0000", 2024-03-20T09:18:08.613 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.613 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.613 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.613 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:08.613 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.042968+0000", 2024-03-20T09:18:08.613 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.613 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.613 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.613 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:08.613 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.042970+0000", 2024-03-20T09:18:08.613 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8870000000000001e-06 2024-03-20T09:18:08.613 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.613 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.614 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:08.614 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.042994+0000", 2024-03-20T09:18:08.614 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4083e-05 2024-03-20T09:18:08.614 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.614 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.614 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:08.614 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.043019+0000", 2024-03-20T09:18:08.614 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5386000000000001e-05 2024-03-20T09:18:08.614 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.614 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.614 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:08.614 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.043034+0000", 2024-03-20T09:18:08.614 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.484e-05 2024-03-20T09:18:08.614 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.615 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.615 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:08.615 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.043127+0000", 2024-03-20T09:18:08.615 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3313000000000006e-05 2024-03-20T09:18:08.615 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.615 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.615 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:08.615 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.043557+0000", 2024-03-20T09:18:08.615 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00043023999999999997 2024-03-20T09:18:08.615 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.615 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:08.615 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.615 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.615 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.616 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e477/434 cost 1049576)", 2024-03-20T09:18:08.616 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:47.242004+0000", 2024-03-20T09:18:08.616 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 381.30147144799997, 2024-03-20T09:18:08.616 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00069037099999999995, 2024-03-20T09:18:08.616 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:08.616 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:18:08.616 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:08.616 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.616 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:08.616 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.242004+0000", 2024-03-20T09:18:08.616 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.616 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.616 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.616 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:08.616 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.242004+0000", 2024-03-20T09:18:08.617 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.617 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.617 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.617 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:08.617 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.242006+0000", 2024-03-20T09:18:08.617 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8199999999999999e-06 2024-03-20T09:18:08.617 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.617 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.617 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:08.617 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.242029+0000", 2024-03-20T09:18:08.617 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.317e-05 2024-03-20T09:18:08.617 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.617 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.617 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:08.618 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.242053+0000", 2024-03-20T09:18:08.618 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4113000000000001e-05 2024-03-20T09:18:08.618 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.618 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.618 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:08.618 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.242067+0000", 2024-03-20T09:18:08.618 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3645999999999999e-05 2024-03-20T09:18:08.618 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.618 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.618 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:08.618 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.242187+0000", 2024-03-20T09:18:08.618 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012064499999999999 2024-03-20T09:18:08.618 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.618 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.618 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:08.619 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.242694+0000", 2024-03-20T09:18:08.619 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00050697700000000004 2024-03-20T09:18:08.619 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.619 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:08.619 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.619 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.619 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.619 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDPGPull(3.e e477/434 cost 1049576)", 2024-03-20T09:18:08.619 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:11:47.245542+0000", 2024-03-20T09:18:08.619 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 381.29793348999999, 2024-03-20T09:18:08.619 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00065577400000000003, 2024-03-20T09:18:08.619 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:08.619 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:18:08.619 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:08.619 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.620 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:08.620 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.245542+0000", 2024-03-20T09:18:08.620 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.620 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.620 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.620 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:08.620 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.245542+0000", 2024-03-20T09:18:08.620 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:08.620 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.620 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.620 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:08.620 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.245543+0000", 2024-03-20T09:18:08.620 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1930000000000001e-06 2024-03-20T09:18:08.621 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.621 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.621 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:08.621 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.245560+0000", 2024-03-20T09:18:08.621 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.738e-05 2024-03-20T09:18:08.621 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.621 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.621 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:08.621 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.245583+0000", 2024-03-20T09:18:08.621 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3053000000000001e-05 2024-03-20T09:18:08.621 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.621 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.621 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:08.621 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.245596+0000", 2024-03-20T09:18:08.621 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2830000000000001e-05 2024-03-20T09:18:08.622 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.622 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.622 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:08.622 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.245729+0000", 2024-03-20T09:18:08.622 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013229299999999999 2024-03-20T09:18:08.622 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:08.622 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:08.622 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:08.622 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:11:47.246198+0000", 2024-03-20T09:18:08.622 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00046902499999999998 2024-03-20T09:18:08.622 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.622 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:08.622 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.623 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:08.623 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:08.623 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:18:08.683 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:18:08.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:08.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:18:08.788 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:18:08.818 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:09.171 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:18:09.172 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:18:09.172 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:18:09.172 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:18:09.172 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:18:09.191 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:18:09.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:09 smithi122 ceph-mon[73495]: pgmap v1450: 31 pgs: 31 active+clean; 577 KiB data, 3.7 GiB used, 890 GiB / 894 GiB avail 2024-03-20T09:18:09.211 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:18:09.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:09 smithi026 ceph-mon[68309]: pgmap v1450: 31 pgs: 31 active+clean; 577 KiB data, 3.7 GiB used, 890 GiB / 894 GiB avail 2024-03-20T09:18:09.232 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:18:09.245 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_ops_in_flight 2024-03-20T09:18:09.252 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:18:09.266 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:18:09.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:09.269 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:18:09.272 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:18:09.291 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:18:09.312 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:18:09.314 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:18:09.314 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:18:09.314 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:18:09.314 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:18:09.314 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:18:09.314 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:18:09.314 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:18:09.314 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:18:09.314 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:18:09.314 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:18:09.315 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:18:09.315 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:18:09.370 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:18:09.402 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:09.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:09 smithi115 ceph-mon[72476]: pgmap v1450: 31 pgs: 31 active+clean; 577 KiB data, 3.7 GiB used, 890 GiB / 894 GiB avail 2024-03-20T09:18:09.523 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:18:09.563 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:18:09.615 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:18:10.070 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'unique_pool_3' created 2024-03-20T09:18:10.325 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:10 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3907053419' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:18:10.325 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:10 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/278867636' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-20T09:18:10.385 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:10 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3907053419' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:18:10.386 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:10 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/278867636' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-20T09:18:10.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:10 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3907053419' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:18:10.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:10 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/278867636' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-20T09:18:11.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:11 smithi122 ceph-mon[73495]: pgmap v1451: 31 pgs: 31 active+clean; 577 KiB data, 3.7 GiB used, 890 GiB / 894 GiB avail 2024-03-20T09:18:11.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:11 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/278867636' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-20T09:18:11.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:11 smithi122 ceph-mon[73495]: osdmap e605: 12 total, 12 up, 10 in 2024-03-20T09:18:11.228 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:18:11.228 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:18:11.342 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:11 smithi026 ceph-mon[68309]: pgmap v1451: 31 pgs: 31 active+clean; 577 KiB data, 3.7 GiB used, 890 GiB / 894 GiB avail 2024-03-20T09:18:11.342 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:11 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/278867636' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-20T09:18:11.342 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:11 smithi026 ceph-mon[68309]: osdmap e605: 12 total, 12 up, 10 in 2024-03-20T09:18:11.353 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:18:11.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:11.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:18:11.379 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-03-20T09:18:11.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:11 smithi115 ceph-mon[72476]: pgmap v1451: 31 pgs: 31 active+clean; 577 KiB data, 3.7 GiB used, 890 GiB / 894 GiB avail 2024-03-20T09:18:11.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:11 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/278867636' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-20T09:18:11.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:11 smithi115 ceph-mon[72476]: osdmap e605: 12 total, 12 up, 10 in 2024-03-20T09:18:11.458 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:18:11.490 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:11.512 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:18:11.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:11.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:18:11.616 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:18:11.646 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:11.734 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:18:12.092 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_blocked_ops 2024-03-20T09:18:12.094 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:18:12.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:12.097 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:18:12.198 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:18:12.229 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:12.327 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:18:12.330 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:12 smithi026 ceph-mon[68309]: osdmap e606: 12 total, 12 up, 10 in 2024-03-20T09:18:12.330 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:12 smithi026 ceph-mon[68309]: pgmap v1454: 47 pgs: 5 creating+peering, 8 unknown, 34 active+clean; 577 KiB data, 3.7 GiB used, 890 GiB / 894 GiB avail 2024-03-20T09:18:12.358 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:18:12.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:12 smithi115 ceph-mon[72476]: osdmap e606: 12 total, 12 up, 10 in 2024-03-20T09:18:12.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:12 smithi115 ceph-mon[72476]: pgmap v1454: 47 pgs: 5 creating+peering, 8 unknown, 34 active+clean; 577 KiB data, 3.7 GiB used, 890 GiB / 894 GiB avail 2024-03-20T09:18:12.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:12 smithi122 ceph-mon[73495]: osdmap e606: 12 total, 12 up, 10 in 2024-03-20T09:18:12.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:12 smithi122 ceph-mon[73495]: pgmap v1454: 47 pgs: 5 creating+peering, 8 unknown, 34 active+clean; 577 KiB data, 3.7 GiB used, 890 GiB / 894 GiB avail 2024-03-20T09:18:12.661 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:18:13.181 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:18:13.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:13.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:18:13.248 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.10 to 1 (10000) 2024-03-20T09:18:13.285 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:18:13.314 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:13.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:13 smithi115 ceph-mon[72476]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-20T09:18:13.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:13 smithi115 ceph-mon[72476]: osdmap e607: 12 total, 12 up, 10 in 2024-03-20T09:18:13.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:13 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2893773427' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-03-20T09:18:13.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:13 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-03-20T09:18:13.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:13 smithi122 ceph-mon[73495]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-20T09:18:13.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:13 smithi122 ceph-mon[73495]: osdmap e607: 12 total, 12 up, 10 in 2024-03-20T09:18:13.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:13 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2893773427' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-03-20T09:18:13.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:13 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-03-20T09:18:13.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:13 smithi026 ceph-mon[68309]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-20T09:18:13.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:13 smithi026 ceph-mon[68309]: osdmap e607: 12 total, 12 up, 10 in 2024-03-20T09:18:13.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:13 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2893773427' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-03-20T09:18:13.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:13 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-03-20T09:18:13.461 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:18:13.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:13.464 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:18:13.565 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:18:13.595 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:13.745 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:18:13.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:13.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:18:13.850 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:18:13.878 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:13.977 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:18:13.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:13.979 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:18:13.992 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:18:13.992 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:18:14.080 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:18:14.112 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:14.133 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:18:14.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:14.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:18:14.217 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 11 1 2024-03-20T09:18:14.237 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:18:14.267 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:14.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:14 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-03-20T09:18:14.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:14 smithi115 ceph-mon[72476]: osdmap e608: 12 total, 12 up, 10 in 2024-03-20T09:18:14.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:14 smithi115 ceph-mon[72476]: pgmap v1457: 47 pgs: 5 creating+peering, 2 unknown, 40 active+clean; 577 KiB data, 3.7 GiB used, 890 GiB / 894 GiB avail 2024-03-20T09:18:14.452 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:18:14.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:14 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-03-20T09:18:14.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:14 smithi026 ceph-mon[68309]: osdmap e608: 12 total, 12 up, 10 in 2024-03-20T09:18:14.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:14 smithi026 ceph-mon[68309]: pgmap v1457: 47 pgs: 5 creating+peering, 2 unknown, 40 active+clean; 577 KiB data, 3.7 GiB used, 890 GiB / 894 GiB avail 2024-03-20T09:18:14.644 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:14 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-03-20T09:18:14.645 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:14 smithi122 ceph-mon[73495]: osdmap e608: 12 total, 12 up, 10 in 2024-03-20T09:18:14.645 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:14 smithi122 ceph-mon[73495]: pgmap v1457: 47 pgs: 5 creating+peering, 2 unknown, 40 active+clean; 577 KiB data, 3.7 GiB used, 890 GiB / 894 GiB avail 2024-03-20T09:18:14.668 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_historic_ops 2024-03-20T09:18:14.714 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:18:14.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:14.717 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:18:14.818 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:18:14.845 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:14.864 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:18:14.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:14.867 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:18:14.968 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:18:14.974 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:18:14.995 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:15.194 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:18:15.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:15.198 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:18:15.299 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:18:15.331 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:15.353 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:18:15.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:15.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:18:15.457 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:18:15.487 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:15.507 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:18:15.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:15.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:18:15.517 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:18:15.551 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:18:15.612 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:18:15.642 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:15.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:15 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/811312938' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:18:15.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:15 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/811312938' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:18:15.924 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:15 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/811312938' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:18:16.693 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:18:16.696 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:16 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:18:16.696 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:16 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/811312938' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:18:16.696 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:16 smithi026 ceph-mon[68309]: osdmap e609: 12 total, 12 up, 10 in 2024-03-20T09:18:16.696 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:16 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:18:16.696 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:16 smithi026 ceph-mon[68309]: pgmap v1459: 47 pgs: 47 active+clean; 577 KiB data, 3.7 GiB used, 890 GiB / 894 GiB avail 2024-03-20T09:18:16.696 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:16 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:18:16.697 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:16 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:18:16.697 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:16 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:18:16.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:16 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:18:16.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:16 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/811312938' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:18:16.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:16 smithi122 ceph-mon[73495]: osdmap e609: 12 total, 12 up, 10 in 2024-03-20T09:18:16.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:16 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:18:16.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:16 smithi122 ceph-mon[73495]: pgmap v1459: 47 pgs: 47 active+clean; 577 KiB data, 3.7 GiB used, 890 GiB / 894 GiB avail 2024-03-20T09:18:16.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:16 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:18:16.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:16 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:18:16.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:16 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:18:16.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:16.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:18:16.705 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:18:16.724 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:18:16.744 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:18:16.763 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:18:16.766 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:18:16.766 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:18:16.783 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:18:16.799 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:18:16.803 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:18:16.823 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:18:16.830 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:16.846 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:18:16.865 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:18:16.885 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:18:16.905 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:18:16.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:16 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:18:16.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:16 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/811312938' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:18:16.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:16 smithi115 ceph-mon[72476]: osdmap e609: 12 total, 12 up, 10 in 2024-03-20T09:18:16.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:16 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:18:16.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:16 smithi115 ceph-mon[72476]: pgmap v1459: 47 pgs: 47 active+clean; 577 KiB data, 3.7 GiB used, 890 GiB / 894 GiB avail 2024-03-20T09:18:16.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:16 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:18:16.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:16 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:18:16.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:16 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:18:16.925 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:18:16.927 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:18:16.927 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:18:16.927 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:18:16.927 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:18:16.927 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:18:16.927 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:18:16.927 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:18:16.927 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:18:16.927 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:18:16.927 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:18:16.927 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:18:16.927 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:18:17.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:17 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/945758604' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:18:17.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:17 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/945758604' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:18:17.955 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:17 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/945758604' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:18:18.324 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_ops_in_flight 2024-03-20T09:18:18.589 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:18:18.672 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:18 smithi026 ceph-mon[68309]: pgmap v1460: 47 pgs: 47 active+clean; 577 KiB data, 3.7 GiB used, 890 GiB / 894 GiB avail 2024-03-20T09:18:18.672 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:18 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1565028065' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-20T09:18:18.773 INFO:teuthology.orchestra.run.smithi026.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-03-20T09:18:18.830 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:18 smithi122 ceph-mon[73495]: pgmap v1460: 47 pgs: 47 active+clean; 577 KiB data, 3.7 GiB used, 890 GiB / 894 GiB avail 2024-03-20T09:18:18.830 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:18 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1565028065' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-20T09:18:18.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:18 smithi115 ceph-mon[72476]: pgmap v1460: 47 pgs: 47 active+clean; 577 KiB data, 3.7 GiB used, 890 GiB / 894 GiB avail 2024-03-20T09:18:18.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:18 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1565028065' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-20T09:18:18.981 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:18:18.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:18.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:18:19.085 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:18:19.116 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:19.466 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:18:19.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:19.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:18:19.572 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:18:19.602 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:19.701 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:18:19.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:19.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:18:19.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:19 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1565028065' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-20T09:18:19.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:19 smithi026 ceph-mon[68309]: osdmap e610: 12 total, 12 up, 10 in 2024-03-20T09:18:19.732 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:19 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1565028065' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-20T09:18:19.732 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:19 smithi122 ceph-mon[73495]: osdmap e610: 12 total, 12 up, 10 in 2024-03-20T09:18:19.807 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:18:19.835 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:19 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1565028065' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-20T09:18:19.835 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:19 smithi115 ceph-mon[72476]: osdmap e610: 12 total, 12 up, 10 in 2024-03-20T09:18:19.837 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:19.858 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:18:19.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:19.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:18:19.962 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:18:19.995 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:20.016 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:18:20.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:20.020 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:18:20.121 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:18:20.151 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:20.226 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:18:20.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:20.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:18:20.287 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:18:20.287 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:18:20.287 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-03-20T09:18:20.287 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:18:20.331 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:18:20.358 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:20.496 DEBUG:teuthology.orchestra.run.smithi122:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-03-20T09:18:20.498 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-20T09:18:20.498 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-20T09:18:20.620 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:hints = 1 2024-03-20T09:18:20.620 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-20T09:18:20.620 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Object prefix: benchmark_data_smithi122_157894 2024-03-20T09:18:20.621 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T09:18:20.621 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 0 0 0 0 0 0 - 0 2024-03-20T09:18:20.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:20 smithi115 ceph-mon[72476]: pgmap v1462: 47 pgs: 47 active+clean; 577 KiB data, 3.7 GiB used, 890 GiB / 894 GiB avail 2024-03-20T09:18:20.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:20 smithi122 ceph-mon[73495]: pgmap v1462: 47 pgs: 47 active+clean; 577 KiB data, 3.7 GiB used, 890 GiB / 894 GiB avail 2024-03-20T09:18:20.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:20 smithi026 ceph-mon[68309]: pgmap v1462: 47 pgs: 47 active+clean; 577 KiB data, 3.7 GiB used, 890 GiB / 894 GiB avail 2024-03-20T09:18:21.039 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_blocked_ops 2024-03-20T09:18:21.108 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:18:21.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:21.112 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:18:21.213 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:18:21.258 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:21.285 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:18:21.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:21.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:18:21.390 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:18:21.436 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:21.466 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:18:21.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:21.469 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:18:21.480 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:18:21.570 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:18:21.598 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.11 to 1 (10000) 2024-03-20T09:18:21.612 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:21.620 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 1 15 3708 3693 230.801 230.812 0.00474894 0.00432139 2024-03-20T09:18:21.640 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:18:21.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:21.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:18:21.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:21 smithi122 ceph-mon[73495]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-20T09:18:21.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:21 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2004942919' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-03-20T09:18:21.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:21 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-03-20T09:18:21.745 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:18:21.792 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:21.819 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:18:21.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:21.823 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:18:21.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:21 smithi115 ceph-mon[72476]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-20T09:18:21.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:21 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2004942919' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-03-20T09:18:21.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:21 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-03-20T09:18:21.924 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:18:21.963 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:21.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:21 smithi026 ceph-mon[68309]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-20T09:18:21.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:21 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2004942919' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-03-20T09:18:21.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:21 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-03-20T09:18:21.989 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:18:21.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:21.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:18:22.094 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:18:22.143 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:22.249 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:18:22.402 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:18:22.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:22.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:18:22.458 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 3 1 2024-03-20T09:18:22.507 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:18:22.545 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:22.584 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:18:22.591 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:22 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:18:22.591 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:22 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:18:22.591 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:22 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:18:22.591 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:22 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-03-20T09:18:22.591 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:22 smithi026 ceph-mon[68309]: osdmap e611: 12 total, 12 up, 10 in 2024-03-20T09:18:22.591 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:22 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:18:22.591 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:22 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:18:22.591 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:22 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:18:22.592 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:22 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:18:22.592 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:22 smithi026 ceph-mon[68309]: pgmap v1464: 47 pgs: 47 active+clean; 47 MiB data, 3.8 GiB used, 890 GiB / 894 GiB avail; 7.3 MiB/s wr, 116 op/s 2024-03-20T09:18:22.621 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 2 16 7455 7439 232.449 234.125 0.00186033 0.00426732 2024-03-20T09:18:22.761 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:18:22.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:22 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:18:22.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:22 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:18:22.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:22 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:18:22.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:22 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-03-20T09:18:22.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:22 smithi115 ceph-mon[72476]: osdmap e611: 12 total, 12 up, 10 in 2024-03-20T09:18:22.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:22 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:18:22.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:22 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:18:22.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:22 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:18:22.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:22 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:18:22.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:22 smithi115 ceph-mon[72476]: pgmap v1464: 47 pgs: 47 active+clean; 47 MiB data, 3.8 GiB used, 890 GiB / 894 GiB avail; 7.3 MiB/s wr, 116 op/s 2024-03-20T09:18:22.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:22 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:18:22.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:22 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:18:22.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:22 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:18:22.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:22 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-03-20T09:18:22.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:22 smithi122 ceph-mon[73495]: osdmap e611: 12 total, 12 up, 10 in 2024-03-20T09:18:22.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:22 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:18:22.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:22 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:18:22.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:22 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:18:22.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:22 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:18:22.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:22 smithi122 ceph-mon[73495]: pgmap v1464: 47 pgs: 47 active+clean; 47 MiB data, 3.8 GiB used, 890 GiB / 894 GiB avail; 7.3 MiB/s wr, 116 op/s 2024-03-20T09:18:23.621 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 3 16 9851 9835 204.878 149.75 0.0036059 0.00487363 2024-03-20T09:18:23.662 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:23 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:18:23.662 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:23 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:18:23.702 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:18:23.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:23.706 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:18:23.807 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:18:23.837 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:23.876 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:18:23.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:23.885 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:18:23.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:23 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:18:23.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:23 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:18:23.986 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:18:24.042 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:23 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:18:24.043 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:23 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:18:24.045 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:24.092 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:18:24.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:24.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:18:24.155 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:18:24.155 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:18:24.204 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:18:24.266 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:24.621 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 4 15 12251 12236 191.171 150.062 0.00655976 0.00522541 2024-03-20T09:18:24.847 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:18:24.847 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:18:24.847 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-03-20T09:18:24.847 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-03-20T09:18:24.848 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:18:24.874 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:24 smithi122 ceph-mon[73495]: osdmap e612: 12 total, 12 up, 10 in 2024-03-20T09:18:24.874 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:24 smithi122 ceph-mon[73495]: pgmap v1466: 47 pgs: 47 active+clean; 184 MiB data, 4.1 GiB used, 890 GiB / 894 GiB avail; 30 MiB/s wr, 487 op/s 2024-03-20T09:18:24.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:24 smithi026 ceph-mon[68309]: osdmap e612: 12 total, 12 up, 10 in 2024-03-20T09:18:24.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:24 smithi026 ceph-mon[68309]: pgmap v1466: 47 pgs: 47 active+clean; 184 MiB data, 4.1 GiB used, 890 GiB / 894 GiB avail; 30 MiB/s wr, 487 op/s 2024-03-20T09:18:25.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:24 smithi115 ceph-mon[72476]: osdmap e612: 12 total, 12 up, 10 in 2024-03-20T09:18:25.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:24 smithi115 ceph-mon[72476]: pgmap v1466: 47 pgs: 47 active+clean; 184 MiB data, 4.1 GiB used, 890 GiB / 894 GiB avail; 30 MiB/s wr, 487 op/s 2024-03-20T09:18:25.621 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 5 16 15436 15420 192.713 199 0.00313985 0.0051837 2024-03-20T09:18:26.621 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 6 16 18467 18451 192.165 189.438 0.00382618 0.00519929 2024-03-20T09:18:26.702 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_historic_ops 2024-03-20T09:18:26.731 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:18:26.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:26.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:18:26.840 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:18:26.877 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:26.909 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:18:26.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:26.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:18:26.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:26 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:18:26.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:26 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:18:26.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:26 smithi122 ceph-mon[73495]: osdmap e613: 12 total, 12 up, 10 in 2024-03-20T09:18:26.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:26 smithi122 ceph-mon[73495]: pgmap v1468: 47 pgs: 47 active+clean; 564 MiB data, 4.9 GiB used, 889 GiB / 894 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-03-20T09:18:26.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:26 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:18:26.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:26 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:18:26.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:26 smithi026 ceph-mon[68309]: osdmap e613: 12 total, 12 up, 10 in 2024-03-20T09:18:26.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:26 smithi026 ceph-mon[68309]: pgmap v1468: 47 pgs: 47 active+clean; 564 MiB data, 4.9 GiB used, 889 GiB / 894 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-03-20T09:18:27.015 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:18:27.043 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:18:27.044 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:27.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:26 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:18:27.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:26 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:18:27.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:26 smithi115 ceph-mon[72476]: osdmap e613: 12 total, 12 up, 10 in 2024-03-20T09:18:27.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:26 smithi115 ceph-mon[72476]: pgmap v1468: 47 pgs: 47 active+clean; 564 MiB data, 4.9 GiB used, 889 GiB / 894 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-03-20T09:18:27.622 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 7 16 20845 20829 185.944 148.625 0.00419774 0.00537366 2024-03-20T09:18:27.997 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:18:28.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:28.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:18:28.104 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:18:28.132 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:28.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:28 smithi122 ceph-mon[73495]: pgmap v1469: 47 pgs: 47 active+clean; 1.0 GiB data, 6.7 GiB used, 887 GiB / 894 GiB avail; 161 MiB/s wr, 2.57k op/s 2024-03-20T09:18:28.622 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 8 16 23117 23101 180.448 142 0.00173481 0.00549783 2024-03-20T09:18:28.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:28 smithi115 ceph-mon[72476]: pgmap v1469: 47 pgs: 47 active+clean; 1.0 GiB data, 6.7 GiB used, 887 GiB / 894 GiB avail; 161 MiB/s wr, 2.57k op/s 2024-03-20T09:18:28.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:28 smithi026 ceph-mon[68309]: pgmap v1469: 47 pgs: 47 active+clean; 1.0 GiB data, 6.7 GiB used, 887 GiB / 894 GiB avail; 161 MiB/s wr, 2.57k op/s 2024-03-20T09:18:29.310 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:18:29.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:29.320 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:18:29.421 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:18:29.457 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:29.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:29 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:18:29.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:29 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:18:29.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:29 smithi026 ceph-mon[68309]: osdmap e614: 12 total, 12 up, 10 in 2024-03-20T09:18:29.476 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:18:29.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:29.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:18:29.580 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:18:29.604 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:29 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:18:29.604 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:29 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:18:29.605 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:29 smithi115 ceph-mon[72476]: osdmap e614: 12 total, 12 up, 10 in 2024-03-20T09:18:29.606 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:29.622 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 9 16 24836 24820 172.334 107.438 0.0016391 0.00572373 2024-03-20T09:18:29.625 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:18:29.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:29.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:18:29.629 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:29 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:18:29.629 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:29 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:18:29.629 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:29 smithi122 ceph-mon[73495]: osdmap e614: 12 total, 12 up, 10 in 2024-03-20T09:18:29.730 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:18:29.776 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:29.800 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:18:29.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:29.806 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:18:29.906 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:18:29.954 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:30.399 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:18:30.399 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-03-20T09:18:30.399 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-03-20T09:18:30.399 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-03-20T09:18:30.400 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.400 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:126534 4.9 4:9db3ebef:::benchmark_data_smithi122_145698_object5615:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e558)", 2024-03-20T09:18:30.400 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:15:19.415329+0000", 2024-03-20T09:18:30.400 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 190.648150595, 2024-03-20T09:18:30.400 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18024907000000001, 2024-03-20T09:18:30.400 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:30.400 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:30.400 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:30.400 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:18:30.400 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:18:30.400 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 126534 2024-03-20T09:18:30.400 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.401 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:30.401 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.401 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:30.401 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.415329+0000", 2024-03-20T09:18:30.401 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.401 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.401 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.401 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:30.401 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.415329+0000", 2024-03-20T09:18:30.401 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.402 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.402 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.402 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:30.402 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.415329+0000", 2024-03-20T09:18:30.402 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.51e-07 2024-03-20T09:18:30.402 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.402 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.402 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:30.402 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.415332+0000", 2024-03-20T09:18:30.402 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7140000000000002e-06 2024-03-20T09:18:30.402 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.402 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.403 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:30.403 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.415341+0000", 2024-03-20T09:18:30.403 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2529999999999993e-06 2024-03-20T09:18:30.403 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.403 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.403 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:30.403 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.415346+0000", 2024-03-20T09:18:30.403 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7910000000000004e-06 2024-03-20T09:18:30.403 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.403 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.403 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:30.403 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.444078+0000", 2024-03-20T09:18:30.403 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.028731982 2024-03-20T09:18:30.404 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.404 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.404 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:30.404 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.444168+0000", 2024-03-20T09:18:30.404 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9967000000000001e-05 2024-03-20T09:18:30.404 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.404 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.404 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,3", 2024-03-20T09:18:30.404 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.444253+0000", 2024-03-20T09:18:30.404 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4370999999999996e-05 2024-03-20T09:18:30.404 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.404 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.404 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:18:30.405 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.448686+0000", 2024-03-20T09:18:30.405 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0044338169999999996 2024-03-20T09:18:30.405 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.405 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.405 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.405 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.510358+0000", 2024-03-20T09:18:30.405 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.061671436000000003 2024-03-20T09:18:30.405 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.405 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.405 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.405 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.595470+0000", 2024-03-20T09:18:30.405 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.085112159000000007 2024-03-20T09:18:30.405 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.406 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.406 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:30.406 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.595521+0000", 2024-03-20T09:18:30.406 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0773999999999999e-05 2024-03-20T09:18:30.406 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.406 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.406 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:30.407 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.595578+0000", 2024-03-20T09:18:30.407 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7255e-05 2024-03-20T09:18:30.407 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.407 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:30.407 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.407 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.407 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.407 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:126626 4.9 4:981472c5:::benchmark_data_smithi122_145698_object5707:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e558)", 2024-03-20T09:18:30.407 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:15:19.785873+0000", 2024-03-20T09:18:30.407 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 190.277606392, 2024-03-20T09:18:30.408 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.22007701700000001, 2024-03-20T09:18:30.408 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:30.408 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:30.408 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:30.408 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:18:30.408 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:18:30.408 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 126626 2024-03-20T09:18:30.408 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.408 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:30.408 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.408 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:30.408 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.785873+0000", 2024-03-20T09:18:30.408 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.408 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.409 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.409 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:30.409 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.785873+0000", 2024-03-20T09:18:30.409 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.409 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.409 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.409 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:30.409 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.785873+0000", 2024-03-20T09:18:30.409 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6600000000000003e-07 2024-03-20T09:18:30.409 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.409 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.409 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:30.409 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.785875+0000", 2024-03-20T09:18:30.409 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5e-06 2024-03-20T09:18:30.410 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.410 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.410 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:30.410 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.785878+0000", 2024-03-20T09:18:30.410 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.067e-06 2024-03-20T09:18:30.410 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.410 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.410 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:30.410 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.785881+0000", 2024-03-20T09:18:30.410 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7669999999999999e-06 2024-03-20T09:18:30.410 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.410 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.411 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:30.411 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.858640+0000", 2024-03-20T09:18:30.411 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.072759708000000006 2024-03-20T09:18:30.411 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.411 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.411 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:30.411 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.858776+0000", 2024-03-20T09:18:30.411 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013547599999999999 2024-03-20T09:18:30.411 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.411 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.411 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,3", 2024-03-20T09:18:30.411 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.858864+0000", 2024-03-20T09:18:30.411 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8318000000000004e-05 2024-03-20T09:18:30.411 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.412 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.412 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:18:30.412 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.862399+0000", 2024-03-20T09:18:30.412 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0035345509999999999 2024-03-20T09:18:30.412 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.412 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.412 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.412 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.921710+0000", 2024-03-20T09:18:30.412 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.059311244999999999 2024-03-20T09:18:30.412 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.412 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.412 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.412 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.005858+0000", 2024-03-20T09:18:30.412 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.084148223999999994 2024-03-20T09:18:30.413 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.413 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.413 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:30.413 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.005896+0000", 2024-03-20T09:18:30.413 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7706000000000001e-05 2024-03-20T09:18:30.413 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.413 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.413 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:30.413 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.005950+0000", 2024-03-20T09:18:30.413 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4188999999999999e-05 2024-03-20T09:18:30.413 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.413 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:30.414 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.414 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.414 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.414 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:126849 4.9 4:9229c909:::benchmark_data_smithi122_145698_object5930:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e558)", 2024-03-20T09:18:30.414 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:15:20.540259+0000", 2024-03-20T09:18:30.414 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 189.52322081899999, 2024-03-20T09:18:30.414 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.201706686, 2024-03-20T09:18:30.414 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:30.414 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:30.414 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:30.414 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:18:30.414 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:18:30.414 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 126849 2024-03-20T09:18:30.414 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.415 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:30.415 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.415 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:30.415 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.540259+0000", 2024-03-20T09:18:30.415 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.415 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.415 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.415 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:30.415 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.540259+0000", 2024-03-20T09:18:30.415 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.415 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.415 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.415 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:30.416 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.540260+0000", 2024-03-20T09:18:30.416 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8169999999999999e-06 2024-03-20T09:18:30.416 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.416 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.416 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:30.416 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.540267+0000", 2024-03-20T09:18:30.416 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0310000000000004e-06 2024-03-20T09:18:30.416 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.416 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.416 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:30.416 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.540290+0000", 2024-03-20T09:18:30.416 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3688000000000001e-05 2024-03-20T09:18:30.416 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.416 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.417 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:30.417 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.540299+0000", 2024-03-20T09:18:30.417 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4829999999999999e-06 2024-03-20T09:18:30.417 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.417 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.417 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:30.417 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.612395+0000", 2024-03-20T09:18:30.417 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.072096439999999998 2024-03-20T09:18:30.417 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.417 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.417 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:30.417 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.612531+0000", 2024-03-20T09:18:30.417 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013567000000000001 2024-03-20T09:18:30.417 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.418 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.418 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,3", 2024-03-20T09:18:30.418 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.612673+0000", 2024-03-20T09:18:30.418 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014193499999999999 2024-03-20T09:18:30.418 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.418 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.418 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:18:30.418 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.626044+0000", 2024-03-20T09:18:30.418 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013371450999999999 2024-03-20T09:18:30.418 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.418 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.418 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.418 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.737807+0000", 2024-03-20T09:18:30.419 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.111762402 2024-03-20T09:18:30.419 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.419 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.419 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.419 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.741860+0000", 2024-03-20T09:18:30.419 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040529449999999996 2024-03-20T09:18:30.419 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.419 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.419 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:30.419 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.741914+0000", 2024-03-20T09:18:30.419 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4236e-05 2024-03-20T09:18:30.419 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.419 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.419 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:30.420 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.741965+0000", 2024-03-20T09:18:30.420 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1588000000000001e-05 2024-03-20T09:18:30.420 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.420 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:30.420 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.420 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.420 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.420 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:126854 4.9 4:934a0522:::benchmark_data_smithi122_145698_object5935:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e558)", 2024-03-20T09:18:30.420 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:15:20.554985+0000", 2024-03-20T09:18:30.420 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 189.50849435699999, 2024-03-20T09:18:30.420 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18721818900000001, 2024-03-20T09:18:30.420 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:30.420 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:30.420 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:30.421 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:18:30.421 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:18:30.421 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 126854 2024-03-20T09:18:30.421 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.421 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:30.421 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.421 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:30.421 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.554985+0000", 2024-03-20T09:18:30.421 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.421 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.421 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.421 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:30.421 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.554985+0000", 2024-03-20T09:18:30.421 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.422 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.422 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.422 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:30.422 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.554987+0000", 2024-03-20T09:18:30.422 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5009999999999999e-06 2024-03-20T09:18:30.422 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.422 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.422 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:30.422 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.554993+0000", 2024-03-20T09:18:30.422 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9259999999999997e-06 2024-03-20T09:18:30.422 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.422 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.422 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:30.422 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.555006+0000", 2024-03-20T09:18:30.423 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.327e-05 2024-03-20T09:18:30.423 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.423 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.423 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:30.423 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.555014+0000", 2024-03-20T09:18:30.423 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5699999999999993e-06 2024-03-20T09:18:30.423 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.423 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.423 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:30.423 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.616851+0000", 2024-03-20T09:18:30.423 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.061836968999999999 2024-03-20T09:18:30.423 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.423 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.424 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:30.424 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.616986+0000", 2024-03-20T09:18:30.424 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000134203 2024-03-20T09:18:30.424 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.424 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.424 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,3", 2024-03-20T09:18:30.424 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.617130+0000", 2024-03-20T09:18:30.424 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014442599999999999 2024-03-20T09:18:30.424 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.424 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.424 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:18:30.424 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.626053+0000", 2024-03-20T09:18:30.424 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0089233920000000005 2024-03-20T09:18:30.424 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.425 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.425 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.425 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.737846+0000", 2024-03-20T09:18:30.425 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.11179288800000001 2024-03-20T09:18:30.425 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.425 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.425 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.425 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.742083+0000", 2024-03-20T09:18:30.425 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0042367350000000002 2024-03-20T09:18:30.425 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.425 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.425 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:30.425 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.742142+0000", 2024-03-20T09:18:30.425 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9302999999999998e-05 2024-03-20T09:18:30.425 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.426 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.426 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:30.426 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.742203+0000", 2024-03-20T09:18:30.426 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1005999999999998e-05 2024-03-20T09:18:30.426 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.426 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:30.426 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.426 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.426 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.426 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25966.0:8055 5.8 5:107d6b13:::benchmark_data_smithi122_157894_object8054:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e611)", 2024-03-20T09:18:30.426 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:18:22.889240+0000", 2024-03-20T09:18:30.426 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 7.1742395339999998, 2024-03-20T09:18:30.426 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20490609900000001, 2024-03-20T09:18:30.427 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:30.427 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:30.427 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:30.427 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25966", 2024-03-20T09:18:30.427 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:18:30.427 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 8055 2024-03-20T09:18:30.427 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.427 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:30.427 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.427 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:30.427 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.889240+0000", 2024-03-20T09:18:30.427 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.427 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.427 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.428 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:30.428 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.889240+0000", 2024-03-20T09:18:30.428 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.428 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.428 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.428 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:30.428 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.889241+0000", 2024-03-20T09:18:30.428 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3900000000000004e-07 2024-03-20T09:18:30.428 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.428 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.428 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:30.428 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.889262+0000", 2024-03-20T09:18:30.428 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1622999999999999e-05 2024-03-20T09:18:30.428 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.429 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.429 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:30.429 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.889273+0000", 2024-03-20T09:18:30.429 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1019000000000001e-05 2024-03-20T09:18:30.429 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.429 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.429 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:30.429 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.889281+0000", 2024-03-20T09:18:30.429 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2919999999999998e-06 2024-03-20T09:18:30.429 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.429 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.429 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:30.429 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.889355+0000", 2024-03-20T09:18:30.429 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4365999999999999e-05 2024-03-20T09:18:30.429 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.430 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.430 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:30.430 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.889474+0000", 2024-03-20T09:18:30.430 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000118649 2024-03-20T09:18:30.430 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.430 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.430 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:18:30.430 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.889613+0000", 2024-03-20T09:18:30.430 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013953500000000001 2024-03-20T09:18:30.430 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.430 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.430 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.430 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.890819+0000", 2024-03-20T09:18:30.431 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001205383 2024-03-20T09:18:30.431 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.431 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.431 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:18:30.431 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.890848+0000", 2024-03-20T09:18:30.431 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9437999999999998e-05 2024-03-20T09:18:30.431 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.431 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.431 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.431 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:23.094078+0000", 2024-03-20T09:18:30.431 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20323011299999999 2024-03-20T09:18:30.431 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.431 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.431 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:30.432 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:23.094116+0000", 2024-03-20T09:18:30.432 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7308000000000003e-05 2024-03-20T09:18:30.432 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.432 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.432 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:30.432 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:23.094146+0000", 2024-03-20T09:18:30.432 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0534000000000003e-05 2024-03-20T09:18:30.432 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.432 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:30.432 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.432 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.432 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.432 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25966.0:8056 5.8 5:1472a471:::benchmark_data_smithi122_157894_object8055:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e611)", 2024-03-20T09:18:30.432 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:18:22.889438+0000", 2024-03-20T09:18:30.433 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 7.1740412090000003, 2024-03-20T09:18:30.433 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20678607900000001, 2024-03-20T09:18:30.433 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:30.433 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:30.433 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:30.433 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25966", 2024-03-20T09:18:30.433 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:18:30.433 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 8056 2024-03-20T09:18:30.433 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.433 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:30.433 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.433 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:30.433 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.889438+0000", 2024-03-20T09:18:30.434 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.434 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.434 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.434 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:30.434 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.889438+0000", 2024-03-20T09:18:30.434 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.434 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.434 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.434 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:30.434 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.889439+0000", 2024-03-20T09:18:30.434 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3e-07 2024-03-20T09:18:30.434 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.435 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.435 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:30.435 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.889458+0000", 2024-03-20T09:18:30.435 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8669999999999999e-05 2024-03-20T09:18:30.435 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.435 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.435 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:30.435 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.889466+0000", 2024-03-20T09:18:30.435 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8030000000000003e-06 2024-03-20T09:18:30.435 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.435 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.435 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:30.435 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.889473+0000", 2024-03-20T09:18:30.436 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5769999999999999e-06 2024-03-20T09:18:30.436 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.436 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.436 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:30.436 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.890109+0000", 2024-03-20T09:18:30.436 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00063546700000000002 2024-03-20T09:18:30.436 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.436 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.436 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:30.436 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.890217+0000", 2024-03-20T09:18:30.436 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000108218 2024-03-20T09:18:30.436 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.437 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.437 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:18:30.437 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.890374+0000", 2024-03-20T09:18:30.437 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015692000000000001 2024-03-20T09:18:30.437 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.437 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.437 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.437 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.891591+0000", 2024-03-20T09:18:30.437 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0012171700000000001 2024-03-20T09:18:30.437 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.437 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.437 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:18:30.438 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.891604+0000", 2024-03-20T09:18:30.438 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2771e-05 2024-03-20T09:18:30.438 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.438 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.438 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.438 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:23.096163+0000", 2024-03-20T09:18:30.438 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20455932299999999 2024-03-20T09:18:30.438 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.438 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.438 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:30.438 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:23.096196+0000", 2024-03-20T09:18:30.438 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2950000000000001e-05 2024-03-20T09:18:30.438 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.439 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.439 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:30.439 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:23.096224+0000", 2024-03-20T09:18:30.439 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8479999999999998e-05 2024-03-20T09:18:30.439 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.439 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:30.439 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.439 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.439 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.439 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25966.0:8064 5.8 5:148c550b:::benchmark_data_smithi122_157894_object8063:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e611)", 2024-03-20T09:18:30.439 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:18:22.890824+0000", 2024-03-20T09:18:30.439 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 7.1726559200000004, 2024-03-20T09:18:30.440 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20594858299999999, 2024-03-20T09:18:30.440 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:30.440 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:30.440 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:30.440 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25966", 2024-03-20T09:18:30.440 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:18:30.440 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 8064 2024-03-20T09:18:30.440 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.440 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:30.440 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.440 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:30.440 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.890824+0000", 2024-03-20T09:18:30.441 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.441 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.441 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.441 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:30.441 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.890824+0000", 2024-03-20T09:18:30.441 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.441 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.441 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.441 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:30.441 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.890824+0000", 2024-03-20T09:18:30.441 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1999999999999998e-07 2024-03-20T09:18:30.441 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.441 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.442 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:30.442 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.890843+0000", 2024-03-20T09:18:30.442 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8899e-05 2024-03-20T09:18:30.442 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.442 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.442 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:30.442 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.890852+0000", 2024-03-20T09:18:30.442 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1910000000000006e-06 2024-03-20T09:18:30.442 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.442 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.442 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:30.442 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.890858+0000", 2024-03-20T09:18:30.442 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3820000000000003e-06 2024-03-20T09:18:30.443 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.443 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.443 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:30.443 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.890910+0000", 2024-03-20T09:18:30.443 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2061000000000002e-05 2024-03-20T09:18:30.443 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.443 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.443 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:30.443 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.891049+0000", 2024-03-20T09:18:30.443 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000139216 2024-03-20T09:18:30.443 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.443 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.444 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:18:30.444 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.891210+0000", 2024-03-20T09:18:30.444 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000160431 2024-03-20T09:18:30.444 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.444 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.444 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:18:30.444 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.891671+0000", 2024-03-20T09:18:30.444 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00046143999999999997 2024-03-20T09:18:30.444 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.444 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.444 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.444 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.891923+0000", 2024-03-20T09:18:30.444 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00025236099999999999 2024-03-20T09:18:30.444 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.445 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.445 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.445 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:23.096713+0000", 2024-03-20T09:18:30.445 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.204790106 2024-03-20T09:18:30.445 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.445 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.445 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:30.445 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:23.096746+0000", 2024-03-20T09:18:30.445 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2150999999999998e-05 2024-03-20T09:18:30.445 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.445 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.445 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:30.445 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:23.096772+0000", 2024-03-20T09:18:30.446 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6525000000000001e-05 2024-03-20T09:18:30.446 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.446 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:30.446 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.446 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.446 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.446 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25966.0:8069 5.8 5:16fb6b90:::benchmark_data_smithi122_157894_object8068:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e611)", 2024-03-20T09:18:30.446 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:18:22.893982+0000", 2024-03-20T09:18:30.446 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 7.1694970549999999, 2024-03-20T09:18:30.446 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.202890184, 2024-03-20T09:18:30.446 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:30.446 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:30.446 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:30.446 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25966", 2024-03-20T09:18:30.447 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:18:30.447 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 8069 2024-03-20T09:18:30.447 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.447 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:30.447 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.447 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:30.447 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.893982+0000", 2024-03-20T09:18:30.447 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.447 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.447 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.447 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:30.447 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.893982+0000", 2024-03-20T09:18:30.447 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.447 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.448 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.448 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:30.448 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.893984+0000", 2024-03-20T09:18:30.448 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2190000000000001e-06 2024-03-20T09:18:30.448 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.448 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.448 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:30.448 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.894003+0000", 2024-03-20T09:18:30.448 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9434000000000001e-05 2024-03-20T09:18:30.448 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.448 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.448 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:30.448 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.894014+0000", 2024-03-20T09:18:30.448 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0865e-05 2024-03-20T09:18:30.448 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.449 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.449 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:30.449 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.894019+0000", 2024-03-20T09:18:30.449 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4149999999999998e-06 2024-03-20T09:18:30.449 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.449 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.449 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:30.449 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.894071+0000", 2024-03-20T09:18:30.449 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1891999999999997e-05 2024-03-20T09:18:30.449 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.449 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.449 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:30.449 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.894178+0000", 2024-03-20T09:18:30.450 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000106554 2024-03-20T09:18:30.450 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.450 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.450 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:18:30.450 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.894263+0000", 2024-03-20T09:18:30.450 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5650999999999997e-05 2024-03-20T09:18:30.450 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.450 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.450 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:18:30.450 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.894793+0000", 2024-03-20T09:18:30.450 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00052971799999999999 2024-03-20T09:18:30.450 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.450 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.451 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.451 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.894975+0000", 2024-03-20T09:18:30.451 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00018218999999999999 2024-03-20T09:18:30.451 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.451 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.451 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.451 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:23.096813+0000", 2024-03-20T09:18:30.451 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.201838142 2024-03-20T09:18:30.451 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.451 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.451 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:30.451 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:23.096848+0000", 2024-03-20T09:18:30.451 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4097000000000002e-05 2024-03-20T09:18:30.452 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.452 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.452 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:30.452 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:23.096873+0000", 2024-03-20T09:18:30.452 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5007e-05 2024-03-20T09:18:30.452 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.452 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:30.452 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.452 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.452 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.452 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25966.0:8107 5.8 5:1953cd43:::benchmark_data_smithi122_157894_object8106:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e611)", 2024-03-20T09:18:30.453 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:18:22.905355+0000", 2024-03-20T09:18:30.453 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 7.1581249629999997, 2024-03-20T09:18:30.453 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.191647285, 2024-03-20T09:18:30.453 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:30.453 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:30.453 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:30.453 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25966", 2024-03-20T09:18:30.453 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:18:30.453 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 8107 2024-03-20T09:18:30.453 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.453 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:30.453 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.453 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:30.454 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.905355+0000", 2024-03-20T09:18:30.454 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.454 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.454 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.454 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:30.454 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.905355+0000", 2024-03-20T09:18:30.454 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.454 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.454 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.454 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:30.454 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.905355+0000", 2024-03-20T09:18:30.454 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2299999999999999e-07 2024-03-20T09:18:30.455 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.455 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.455 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:30.455 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.905375+0000", 2024-03-20T09:18:30.455 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9882e-05 2024-03-20T09:18:30.455 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.455 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.455 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:30.455 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.905385+0000", 2024-03-20T09:18:30.455 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.8279999999999998e-06 2024-03-20T09:18:30.455 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.455 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.455 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:30.456 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.905392+0000", 2024-03-20T09:18:30.456 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.776e-06 2024-03-20T09:18:30.456 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.456 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.456 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:30.456 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.905448+0000", 2024-03-20T09:18:30.456 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5915000000000003e-05 2024-03-20T09:18:30.456 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.456 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.456 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:30.456 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.905551+0000", 2024-03-20T09:18:30.456 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000102946 2024-03-20T09:18:30.456 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.457 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.457 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:18:30.457 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.905672+0000", 2024-03-20T09:18:30.457 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000121659 2024-03-20T09:18:30.457 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.457 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.457 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:18:30.457 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.906318+0000", 2024-03-20T09:18:30.457 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00064511899999999999 2024-03-20T09:18:30.457 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.457 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.457 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.457 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.906945+0000", 2024-03-20T09:18:30.458 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00062755799999999996 2024-03-20T09:18:30.458 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.458 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.458 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.458 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:23.096911+0000", 2024-03-20T09:18:30.458 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18996613800000001 2024-03-20T09:18:30.458 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.458 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.458 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:30.458 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:23.096965+0000", 2024-03-20T09:18:30.458 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4048999999999997e-05 2024-03-20T09:18:30.458 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.458 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.459 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:30.459 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:23.097002+0000", 2024-03-20T09:18:30.459 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6492000000000002e-05 2024-03-20T09:18:30.459 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.459 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:30.459 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.459 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.459 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.459 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25966.0:8131 5.8 5:13c8b554:::benchmark_data_smithi122_157894_object8130:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e611)", 2024-03-20T09:18:30.459 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:18:22.914326+0000", 2024-03-20T09:18:30.459 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 7.149153417, 2024-03-20T09:18:30.459 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18332132400000001, 2024-03-20T09:18:30.459 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:30.459 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:30.460 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:30.460 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25966", 2024-03-20T09:18:30.460 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:18:30.460 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 8131 2024-03-20T09:18:30.460 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.460 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:30.460 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.460 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:30.460 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.914326+0000", 2024-03-20T09:18:30.460 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.460 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.460 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.460 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:30.461 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.914326+0000", 2024-03-20T09:18:30.461 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.461 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.461 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.461 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:30.461 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.914328+0000", 2024-03-20T09:18:30.461 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5990000000000001e-06 2024-03-20T09:18:30.461 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.461 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.461 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:30.461 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.914346+0000", 2024-03-20T09:18:30.461 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8206000000000002e-05 2024-03-20T09:18:30.461 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.461 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.462 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:30.462 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.914357+0000", 2024-03-20T09:18:30.462 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1011e-05 2024-03-20T09:18:30.462 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.462 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.462 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:30.462 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.914363+0000", 2024-03-20T09:18:30.462 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8109999999999999e-06 2024-03-20T09:18:30.462 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.462 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.462 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:30.462 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.914414+0000", 2024-03-20T09:18:30.463 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1737999999999998e-05 2024-03-20T09:18:30.463 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.463 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.463 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:30.463 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.914519+0000", 2024-03-20T09:18:30.463 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010500799999999999 2024-03-20T09:18:30.463 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.463 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.463 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:18:30.463 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.914610+0000", 2024-03-20T09:18:30.463 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0563999999999996e-05 2024-03-20T09:18:30.463 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.463 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.463 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:18:30.463 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.915125+0000", 2024-03-20T09:18:30.464 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00051455299999999995 2024-03-20T09:18:30.464 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.464 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.464 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.464 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.915410+0000", 2024-03-20T09:18:30.464 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00028549599999999999 2024-03-20T09:18:30.464 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.464 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.464 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.464 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:23.097590+0000", 2024-03-20T09:18:30.464 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18218016000000001 2024-03-20T09:18:30.464 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.464 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.464 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:30.465 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:23.097621+0000", 2024-03-20T09:18:30.465 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1278000000000001e-05 2024-03-20T09:18:30.465 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.465 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.465 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:30.465 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:23.097647+0000", 2024-03-20T09:18:30.465 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5899999999999999e-05 2024-03-20T09:18:30.465 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.465 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:30.465 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.465 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.465 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.465 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25966.0:13791 5.8 5:1176840d:::benchmark_data_smithi122_157894_object13790:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e612)", 2024-03-20T09:18:30.465 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:18:25.041960+0000", 2024-03-20T09:18:30.466 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.021519176, 2024-03-20T09:18:30.466 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20611988000000001, 2024-03-20T09:18:30.466 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:30.466 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:30.466 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:30.466 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25966", 2024-03-20T09:18:30.466 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:18:30.466 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 13791 2024-03-20T09:18:30.466 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.466 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:30.466 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.466 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:30.466 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.041960+0000", 2024-03-20T09:18:30.467 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.467 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.467 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.467 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:30.467 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.041960+0000", 2024-03-20T09:18:30.467 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.467 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.467 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.467 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:30.467 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.041961+0000", 2024-03-20T09:18:30.467 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5499999999999998e-07 2024-03-20T09:18:30.467 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.467 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.467 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:30.467 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.041985+0000", 2024-03-20T09:18:30.468 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3558999999999999e-05 2024-03-20T09:18:30.468 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.468 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.468 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:30.468 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.041995+0000", 2024-03-20T09:18:30.468 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0203999999999999e-05 2024-03-20T09:18:30.468 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.468 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.468 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:30.468 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.042001+0000", 2024-03-20T09:18:30.468 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.806e-06 2024-03-20T09:18:30.468 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.468 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.468 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:30.468 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.042047+0000", 2024-03-20T09:18:30.468 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6749999999999998e-05 2024-03-20T09:18:30.469 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.469 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.469 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:30.469 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.042154+0000", 2024-03-20T09:18:30.469 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000107056 2024-03-20T09:18:30.469 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.469 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.469 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:18:30.469 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.042288+0000", 2024-03-20T09:18:30.469 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000133585 2024-03-20T09:18:30.469 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.469 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.469 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:18:30.469 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.043075+0000", 2024-03-20T09:18:30.469 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00078733399999999999 2024-03-20T09:18:30.469 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.470 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.470 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.470 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.043819+0000", 2024-03-20T09:18:30.470 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00074377000000000004 2024-03-20T09:18:30.470 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.470 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.470 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.470 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.248009+0000", 2024-03-20T09:18:30.470 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20418971699999999 2024-03-20T09:18:30.470 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.470 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.470 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:30.470 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.248048+0000", 2024-03-20T09:18:30.470 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9567000000000003e-05 2024-03-20T09:18:30.470 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.471 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.471 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:30.471 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.248080+0000", 2024-03-20T09:18:30.471 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1877e-05 2024-03-20T09:18:30.471 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.471 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:30.471 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.471 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.471 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.471 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25966.0:13819 5.8 5:18a40a82:::benchmark_data_smithi122_157894_object13818:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e612)", 2024-03-20T09:18:30.471 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:18:25.049685+0000", 2024-03-20T09:18:30.471 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.0137940790000002, 2024-03-20T09:18:30.471 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20001260600000001, 2024-03-20T09:18:30.471 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:30.472 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:30.472 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:30.472 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25966", 2024-03-20T09:18:30.472 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:18:30.472 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 13819 2024-03-20T09:18:30.472 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.472 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:30.472 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.472 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:30.472 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.049685+0000", 2024-03-20T09:18:30.472 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.472 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.472 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.472 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:30.472 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.049685+0000", 2024-03-20T09:18:30.473 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.473 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.473 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.473 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:30.473 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.049687+0000", 2024-03-20T09:18:30.473 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3599999999999999e-06 2024-03-20T09:18:30.473 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.473 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.473 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:30.473 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.049708+0000", 2024-03-20T09:18:30.473 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1336e-05 2024-03-20T09:18:30.473 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.473 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.473 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:30.473 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.049722+0000", 2024-03-20T09:18:30.473 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4079e-05 2024-03-20T09:18:30.474 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.474 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.474 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:30.474 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.049731+0000", 2024-03-20T09:18:30.474 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.551e-06 2024-03-20T09:18:30.474 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.474 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.474 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:30.474 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.049796+0000", 2024-03-20T09:18:30.474 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5054000000000004e-05 2024-03-20T09:18:30.474 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.474 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.474 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:30.474 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.049887+0000", 2024-03-20T09:18:30.475 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0759000000000003e-05 2024-03-20T09:18:30.475 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.475 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.475 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:18:30.475 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.050018+0000", 2024-03-20T09:18:30.475 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013103400000000001 2024-03-20T09:18:30.475 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.475 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.475 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:18:30.475 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.050762+0000", 2024-03-20T09:18:30.475 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00074455400000000003 2024-03-20T09:18:30.475 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.475 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.475 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.475 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.052010+0000", 2024-03-20T09:18:30.475 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001248187 2024-03-20T09:18:30.476 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.476 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.476 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.476 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.249631+0000", 2024-03-20T09:18:30.476 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19762068699999999 2024-03-20T09:18:30.476 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.476 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.476 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:30.476 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.249668+0000", 2024-03-20T09:18:30.476 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6575000000000001e-05 2024-03-20T09:18:30.476 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.476 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.476 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:30.476 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.249698+0000", 2024-03-20T09:18:30.476 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0429999999999998e-05 2024-03-20T09:18:30.477 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.477 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:30.477 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.477 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.477 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.477 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25966.0:13852 5.8 5:18b7fd45:::benchmark_data_smithi122_157894_object13851:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e612)", 2024-03-20T09:18:30.477 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:18:25.058256+0000", 2024-03-20T09:18:30.477 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.0052237220000002, 2024-03-20T09:18:30.477 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.192201436, 2024-03-20T09:18:30.477 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:30.477 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:30.477 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:30.477 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25966", 2024-03-20T09:18:30.477 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:18:30.477 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 13852 2024-03-20T09:18:30.477 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.478 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:30.478 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.478 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:30.478 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.058256+0000", 2024-03-20T09:18:30.478 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.478 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.478 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.478 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:30.478 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.058256+0000", 2024-03-20T09:18:30.478 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.478 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.478 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.478 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:30.478 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.058257+0000", 2024-03-20T09:18:30.479 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5659999999999999e-06 2024-03-20T09:18:30.479 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.479 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.479 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:30.479 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.058287+0000", 2024-03-20T09:18:30.479 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9682000000000001e-05 2024-03-20T09:18:30.479 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.479 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.479 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:30.479 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.058298+0000", 2024-03-20T09:18:30.479 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1371e-05 2024-03-20T09:18:30.479 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.479 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.479 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:30.479 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.058308+0000", 2024-03-20T09:18:30.480 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4450000000000002e-06 2024-03-20T09:18:30.480 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.480 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.480 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:30.480 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.058362+0000", 2024-03-20T09:18:30.480 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4412e-05 2024-03-20T09:18:30.480 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.480 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.480 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:30.480 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.058469+0000", 2024-03-20T09:18:30.480 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000107172 2024-03-20T09:18:30.480 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.480 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.480 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:18:30.480 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.058557+0000", 2024-03-20T09:18:30.480 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7272000000000001e-05 2024-03-20T09:18:30.481 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.481 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.481 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:18:30.481 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.059806+0000", 2024-03-20T09:18:30.481 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0012496600000000001 2024-03-20T09:18:30.481 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.481 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.481 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.481 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.060193+0000", 2024-03-20T09:18:30.481 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000386957 2024-03-20T09:18:30.481 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.481 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.481 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.481 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.250396+0000", 2024-03-20T09:18:30.481 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.190202338 2024-03-20T09:18:30.481 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.482 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.482 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:30.482 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.250431+0000", 2024-03-20T09:18:30.482 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5351000000000001e-05 2024-03-20T09:18:30.482 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.482 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.482 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:30.482 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.250457+0000", 2024-03-20T09:18:30.482 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6210000000000001e-05 2024-03-20T09:18:30.482 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.482 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:30.482 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.482 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.482 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.483 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25966.0:13853 5.8 5:150a1528:::benchmark_data_smithi122_157894_object13852:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e612)", 2024-03-20T09:18:30.483 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:18:25.058382+0000", 2024-03-20T09:18:30.483 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.0050975710000003, 2024-03-20T09:18:30.483 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.192191791, 2024-03-20T09:18:30.483 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:30.483 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:30.483 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:30.483 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25966", 2024-03-20T09:18:30.483 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:18:30.483 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 13853 2024-03-20T09:18:30.483 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.483 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:30.483 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.484 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:30.484 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.058382+0000", 2024-03-20T09:18:30.484 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.484 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.484 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.484 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:30.484 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.058382+0000", 2024-03-20T09:18:30.484 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.484 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.484 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.484 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:30.484 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.058383+0000", 2024-03-20T09:18:30.485 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7899999999999997e-07 2024-03-20T09:18:30.485 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.485 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.485 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:30.485 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.058402+0000", 2024-03-20T09:18:30.485 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9055999999999998e-05 2024-03-20T09:18:30.485 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.485 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.485 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:30.485 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.058409+0000", 2024-03-20T09:18:30.485 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4869999999999996e-06 2024-03-20T09:18:30.485 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.485 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.485 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:30.486 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.058414+0000", 2024-03-20T09:18:30.486 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5909999999999998e-06 2024-03-20T09:18:30.486 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.486 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.486 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:30.486 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.059125+0000", 2024-03-20T09:18:30.486 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00071087199999999998 2024-03-20T09:18:30.486 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.486 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.486 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:30.486 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.059227+0000", 2024-03-20T09:18:30.486 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010202800000000001 2024-03-20T09:18:30.486 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.487 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.487 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:18:30.487 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.059341+0000", 2024-03-20T09:18:30.487 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000114436 2024-03-20T09:18:30.487 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.487 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.487 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:18:30.487 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.059945+0000", 2024-03-20T09:18:30.487 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00060406800000000003 2024-03-20T09:18:30.487 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.487 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.487 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.487 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.060413+0000", 2024-03-20T09:18:30.487 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00046826799999999998 2024-03-20T09:18:30.488 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.488 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.488 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.488 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.250500+0000", 2024-03-20T09:18:30.488 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19008615700000001 2024-03-20T09:18:30.488 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.488 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.488 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:30.488 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.250536+0000", 2024-03-20T09:18:30.488 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6531e-05 2024-03-20T09:18:30.488 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.488 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.488 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:30.489 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.250574+0000", 2024-03-20T09:18:30.489 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7518000000000002e-05 2024-03-20T09:18:30.489 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.489 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:30.489 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.489 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.489 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.489 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25966.0:13877 5.8 5:1fec032c:::benchmark_data_smithi122_157894_object13876:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e612)", 2024-03-20T09:18:30.489 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:18:25.065548+0000", 2024-03-20T09:18:30.489 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 4.9979315660000001, 2024-03-20T09:18:30.489 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18574700699999999, 2024-03-20T09:18:30.489 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:30.489 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:30.489 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:30.490 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25966", 2024-03-20T09:18:30.490 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:18:30.490 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 13877 2024-03-20T09:18:30.490 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.490 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:30.490 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.490 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:30.490 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.065548+0000", 2024-03-20T09:18:30.490 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.490 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.490 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.490 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:30.490 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.065548+0000", 2024-03-20T09:18:30.491 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.491 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.491 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.491 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:30.491 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.065549+0000", 2024-03-20T09:18:30.491 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3500000000000005e-07 2024-03-20T09:18:30.491 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.491 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.491 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:30.491 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.065566+0000", 2024-03-20T09:18:30.491 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7555e-05 2024-03-20T09:18:30.491 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.491 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.491 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:30.491 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.065579+0000", 2024-03-20T09:18:30.492 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2877999999999999e-05 2024-03-20T09:18:30.492 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.492 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.492 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:30.492 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.065587+0000", 2024-03-20T09:18:30.492 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2069999999999995e-06 2024-03-20T09:18:30.492 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.492 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.492 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:30.492 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.065645+0000", 2024-03-20T09:18:30.492 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7138000000000001e-05 2024-03-20T09:18:30.492 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.492 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.492 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:30.492 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.065774+0000", 2024-03-20T09:18:30.493 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000129438 2024-03-20T09:18:30.493 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.493 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.493 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:18:30.493 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.065898+0000", 2024-03-20T09:18:30.493 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000124511 2024-03-20T09:18:30.493 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.493 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.493 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:18:30.493 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.066836+0000", 2024-03-20T09:18:30.493 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00093743899999999998 2024-03-20T09:18:30.493 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.493 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.493 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.493 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.066910+0000", 2024-03-20T09:18:30.494 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3763999999999994e-05 2024-03-20T09:18:30.494 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.494 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.494 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.494 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.251232+0000", 2024-03-20T09:18:30.494 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18432262399999999 2024-03-20T09:18:30.494 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.494 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.494 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:30.494 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.251265+0000", 2024-03-20T09:18:30.494 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2441999999999999e-05 2024-03-20T09:18:30.494 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.494 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.495 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:30.495 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.251295+0000", 2024-03-20T09:18:30.495 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0176e-05 2024-03-20T09:18:30.495 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.495 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:30.495 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.495 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.495 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.495 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25966.0:13882 5.8 5:16b3ea0d:::benchmark_data_smithi122_157894_object13881:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e612)", 2024-03-20T09:18:30.495 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:18:25.067500+0000", 2024-03-20T09:18:30.495 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 4.9959791530000004, 2024-03-20T09:18:30.495 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18392726500000001, 2024-03-20T09:18:30.495 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:30.495 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:30.495 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:30.495 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25966", 2024-03-20T09:18:30.496 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:18:30.496 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 13882 2024-03-20T09:18:30.496 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.496 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:30.496 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.496 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:30.496 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.067500+0000", 2024-03-20T09:18:30.496 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.496 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.496 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.496 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:30.496 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.067500+0000", 2024-03-20T09:18:30.496 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.496 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.496 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.496 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:30.497 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.067501+0000", 2024-03-20T09:18:30.497 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7100000000000011e-07 2024-03-20T09:18:30.497 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.497 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.497 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:30.497 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.067518+0000", 2024-03-20T09:18:30.497 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6793e-05 2024-03-20T09:18:30.497 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.497 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.497 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:30.497 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.067529+0000", 2024-03-20T09:18:30.497 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0701e-05 2024-03-20T09:18:30.497 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.497 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.497 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:30.498 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.067536+0000", 2024-03-20T09:18:30.498 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1579999999999999e-06 2024-03-20T09:18:30.498 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.498 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.498 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:30.498 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.067598+0000", 2024-03-20T09:18:30.498 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2018000000000002e-05 2024-03-20T09:18:30.498 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.498 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.498 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:30.498 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.067740+0000", 2024-03-20T09:18:30.498 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000142297 2024-03-20T09:18:30.498 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.498 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.499 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:18:30.499 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.067890+0000", 2024-03-20T09:18:30.499 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014984800000000001 2024-03-20T09:18:30.499 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.499 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.499 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:18:30.499 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.069124+0000", 2024-03-20T09:18:30.499 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001233428 2024-03-20T09:18:30.499 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.499 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.499 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.499 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.069170+0000", 2024-03-20T09:18:30.499 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6626999999999999e-05 2024-03-20T09:18:30.499 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.499 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.499 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.500 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.251337+0000", 2024-03-20T09:18:30.500 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18216716399999999 2024-03-20T09:18:30.500 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.500 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.500 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:30.500 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.251384+0000", 2024-03-20T09:18:30.500 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6492000000000001e-05 2024-03-20T09:18:30.500 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.500 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.500 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:30.500 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.251428+0000", 2024-03-20T09:18:30.500 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3767999999999998e-05 2024-03-20T09:18:30.500 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.500 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:30.500 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.500 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.501 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.501 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25966.0:13885 5.8 5:1f12e1ca:::benchmark_data_smithi122_157894_object13884:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e612)", 2024-03-20T09:18:30.501 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:18:25.067560+0000", 2024-03-20T09:18:30.501 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 4.9959195799999998, 2024-03-20T09:18:30.501 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.184273932, 2024-03-20T09:18:30.501 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:30.501 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:30.501 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:30.501 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25966", 2024-03-20T09:18:30.501 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:18:30.501 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 13885 2024-03-20T09:18:30.501 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.501 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:30.502 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.502 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:30.502 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.067560+0000", 2024-03-20T09:18:30.502 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.502 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.502 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.502 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:30.502 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.067560+0000", 2024-03-20T09:18:30.502 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.502 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.502 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.502 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:30.502 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.067560+0000", 2024-03-20T09:18:30.502 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4400000000000001e-07 2024-03-20T09:18:30.502 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.503 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.503 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:30.503 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.067579+0000", 2024-03-20T09:18:30.503 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.825e-05 2024-03-20T09:18:30.503 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.503 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.503 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:30.503 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.067584+0000", 2024-03-20T09:18:30.503 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3199999999999999e-06 2024-03-20T09:18:30.503 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.503 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.503 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:30.503 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.067588+0000", 2024-03-20T09:18:30.503 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.619e-06 2024-03-20T09:18:30.503 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.503 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.504 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:30.504 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.068540+0000", 2024-03-20T09:18:30.504 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00095254400000000005 2024-03-20T09:18:30.504 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.504 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.504 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:30.504 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.068632+0000", 2024-03-20T09:18:30.504 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1689999999999998e-05 2024-03-20T09:18:30.504 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.504 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.504 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:18:30.504 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.068720+0000", 2024-03-20T09:18:30.504 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8245999999999995e-05 2024-03-20T09:18:30.504 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.504 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.505 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:18:30.505 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.069269+0000", 2024-03-20T09:18:30.505 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00054908799999999998 2024-03-20T09:18:30.505 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.505 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.505 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.505 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.069563+0000", 2024-03-20T09:18:30.505 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00029382200000000001 2024-03-20T09:18:30.505 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.505 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.505 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.505 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.251733+0000", 2024-03-20T09:18:30.505 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18216969899999999 2024-03-20T09:18:30.505 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.505 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.506 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:30.506 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.251791+0000", 2024-03-20T09:18:30.506 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8529000000000002e-05 2024-03-20T09:18:30.506 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.506 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.506 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:30.506 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.251834+0000", 2024-03-20T09:18:30.506 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2681e-05 2024-03-20T09:18:30.506 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.506 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:30.506 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.506 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.506 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.506 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25966.0:13922 5.8 5:196ac634:::benchmark_data_smithi122_157894_object13921:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e612)", 2024-03-20T09:18:30.507 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:18:25.078201+0000", 2024-03-20T09:18:30.507 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 4.9852787420000002, 2024-03-20T09:18:30.507 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.173775388, 2024-03-20T09:18:30.507 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:30.507 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:30.507 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:30.507 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25966", 2024-03-20T09:18:30.507 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:18:30.507 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 13922 2024-03-20T09:18:30.507 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.507 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:30.507 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.507 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:30.508 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.078201+0000", 2024-03-20T09:18:30.508 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.508 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.508 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.508 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:30.508 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.078201+0000", 2024-03-20T09:18:30.508 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.508 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.508 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.508 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:30.508 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.078202+0000", 2024-03-20T09:18:30.508 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1999999999999998e-07 2024-03-20T09:18:30.508 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.508 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.508 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:30.509 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.078220+0000", 2024-03-20T09:18:30.509 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8593e-05 2024-03-20T09:18:30.509 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.509 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.509 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:30.509 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.078230+0000", 2024-03-20T09:18:30.509 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0301e-05 2024-03-20T09:18:30.509 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.509 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.509 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:30.509 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.078236+0000", 2024-03-20T09:18:30.509 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7520000000000002e-06 2024-03-20T09:18:30.509 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.509 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.509 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:30.510 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.078300+0000", 2024-03-20T09:18:30.510 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4055999999999998e-05 2024-03-20T09:18:30.510 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.510 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.510 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:30.510 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.078391+0000", 2024-03-20T09:18:30.510 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1079e-05 2024-03-20T09:18:30.510 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.510 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.510 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:18:30.510 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.078483+0000", 2024-03-20T09:18:30.510 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1664999999999997e-05 2024-03-20T09:18:30.510 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.510 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.511 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:18:30.511 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.079220+0000", 2024-03-20T09:18:30.511 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000737088 2024-03-20T09:18:30.511 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.511 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.511 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.511 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.079400+0000", 2024-03-20T09:18:30.511 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00017947199999999999 2024-03-20T09:18:30.511 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.511 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.511 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.511 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.251882+0000", 2024-03-20T09:18:30.511 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.17248198300000001 2024-03-20T09:18:30.512 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.512 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.512 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:30.512 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.251936+0000", 2024-03-20T09:18:30.512 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4494999999999999e-05 2024-03-20T09:18:30.512 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.512 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.512 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:30.512 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.251976+0000", 2024-03-20T09:18:30.512 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0083999999999998e-05 2024-03-20T09:18:30.512 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.512 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:30.512 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.512 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.512 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.513 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25966.0:24831 5.8 5:1389c878:::benchmark_data_smithi122_157894_object24830:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e614)", 2024-03-20T09:18:30.513 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:18:29.510386+0000", 2024-03-20T09:18:30.513 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.55309355699999996, 2024-03-20T09:18:30.513 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19539772599999999, 2024-03-20T09:18:30.513 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:30.513 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:30.513 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:30.513 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25966", 2024-03-20T09:18:30.513 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:18:30.513 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 24831 2024-03-20T09:18:30.513 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.513 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:30.513 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.513 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:30.513 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.510386+0000", 2024-03-20T09:18:30.514 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.514 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.514 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.514 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:30.514 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.510386+0000", 2024-03-20T09:18:30.514 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.514 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.514 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.514 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:30.514 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.510388+0000", 2024-03-20T09:18:30.514 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6649999999999999e-06 2024-03-20T09:18:30.514 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.514 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.515 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:30.515 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.510407+0000", 2024-03-20T09:18:30.515 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9822000000000001e-05 2024-03-20T09:18:30.515 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.515 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.515 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:30.515 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.510419+0000", 2024-03-20T09:18:30.515 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1714e-05 2024-03-20T09:18:30.515 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.515 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.515 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:30.515 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.510425+0000", 2024-03-20T09:18:30.515 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.947e-06 2024-03-20T09:18:30.515 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.516 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.516 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:30.516 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.510470+0000", 2024-03-20T09:18:30.516 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5386000000000002e-05 2024-03-20T09:18:30.516 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.516 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.516 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:30.516 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.510595+0000", 2024-03-20T09:18:30.516 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000124583 2024-03-20T09:18:30.516 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.516 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.516 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:18:30.516 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.510692+0000", 2024-03-20T09:18:30.516 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7442999999999995e-05 2024-03-20T09:18:30.517 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.517 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.517 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:18:30.517 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.511628+0000", 2024-03-20T09:18:30.517 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00093502899999999998 2024-03-20T09:18:30.517 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.517 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.517 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.517 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.511809+0000", 2024-03-20T09:18:30.517 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00018124100000000001 2024-03-20T09:18:30.517 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.517 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.517 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.517 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.705668+0000", 2024-03-20T09:18:30.518 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19385886899999999 2024-03-20T09:18:30.518 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.518 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.518 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:30.518 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.705743+0000", 2024-03-20T09:18:30.518 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5062999999999996e-05 2024-03-20T09:18:30.518 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.518 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.518 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:30.518 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.705784+0000", 2024-03-20T09:18:30.518 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0964000000000003e-05 2024-03-20T09:18:30.519 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.519 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:30.519 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.519 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.519 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.519 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25966.0:24835 5.8 5:1d182003:::benchmark_data_smithi122_157894_object24834:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e614)", 2024-03-20T09:18:30.519 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:18:29.513133+0000", 2024-03-20T09:18:30.519 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.55034638899999999, 2024-03-20T09:18:30.519 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.192806691, 2024-03-20T09:18:30.519 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:18:30.519 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:30.519 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:18:30.519 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25966", 2024-03-20T09:18:30.520 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:18:30.520 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 24835 2024-03-20T09:18:30.520 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.520 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:18:30.520 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.520 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:18:30.520 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.513133+0000", 2024-03-20T09:18:30.520 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.520 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.520 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.520 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:18:30.520 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.513133+0000", 2024-03-20T09:18:30.520 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:18:30.520 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.520 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.521 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:18:30.521 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.513134+0000", 2024-03-20T09:18:30.521 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7899999999999997e-07 2024-03-20T09:18:30.521 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.521 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.521 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:18:30.521 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.513152+0000", 2024-03-20T09:18:30.521 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8337000000000001e-05 2024-03-20T09:18:30.521 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.521 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.521 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:18:30.521 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.513163+0000", 2024-03-20T09:18:30.521 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0903e-05 2024-03-20T09:18:30.522 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.522 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.522 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:18:30.522 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.513169+0000", 2024-03-20T09:18:30.522 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0990000000000004e-06 2024-03-20T09:18:30.522 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.522 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.522 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:18:30.522 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.513205+0000", 2024-03-20T09:18:30.522 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5839e-05 2024-03-20T09:18:30.522 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.522 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.522 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:18:30.522 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.513293+0000", 2024-03-20T09:18:30.523 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8311000000000002e-05 2024-03-20T09:18:30.523 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.523 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.523 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:18:30.523 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.513384+0000", 2024-03-20T09:18:30.523 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0925000000000001e-05 2024-03-20T09:18:30.523 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.523 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.523 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:18:30.523 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.514050+0000", 2024-03-20T09:18:30.523 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00066575499999999997 2024-03-20T09:18:30.523 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.523 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.523 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.524 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.514266+0000", 2024-03-20T09:18:30.524 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000215999 2024-03-20T09:18:30.524 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.524 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.524 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:30.524 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.705839+0000", 2024-03-20T09:18:30.524 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19157292000000001 2024-03-20T09:18:30.524 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.524 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.524 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:18:30.524 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.705898+0000", 2024-03-20T09:18:30.524 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8554000000000002e-05 2024-03-20T09:18:30.524 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:18:30.524 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:18:30.525 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:18:30.525 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.705940+0000", 2024-03-20T09:18:30.525 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.227e-05 2024-03-20T09:18:30.525 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.525 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:30.525 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.525 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:18:30.525 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:18:30.525 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:18:30.622 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 10 16 27069 27053 169.056 139.562 0.00309615 0.00591009 2024-03-20T09:18:30.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:30 smithi115 ceph-mon[72476]: pgmap v1471: 47 pgs: 47 active+clean; 1.1 GiB data, 6.9 GiB used, 887 GiB / 894 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-03-20T09:18:30.664 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:30 smithi026 ceph-mon[68309]: pgmap v1471: 47 pgs: 47 active+clean; 1.1 GiB data, 6.9 GiB used, 887 GiB / 894 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-03-20T09:18:30.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:30 smithi122 ceph-mon[73495]: pgmap v1471: 47 pgs: 47 active+clean; 1.1 GiB data, 6.9 GiB used, 887 GiB / 894 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-03-20T09:18:31.417 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:18:31.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:31.421 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:18:31.440 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-20T09:18:31.522 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:18:31.582 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:31.622 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 11 16 29688 29672 168.567 163.688 0.0028765 0.00592754 2024-03-20T09:18:31.637 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:18:31.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:31.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:18:31.744 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:18:31.791 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:31.834 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:18:32.563 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:18:32.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:32.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:18:32.622 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 12 16 32218 32202 167.696 158.125 0.00395063 0.00595692 2024-03-20T09:18:32.667 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:18:32.694 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:32.713 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:18:32.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:32.715 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:18:32.816 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:18:32.855 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:32.883 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:18:32.884 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:18:32.884 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:18:32.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:32.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:18:32.928 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:18:32.966 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:18:32.989 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:18:33.002 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:18:33.043 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:18:33.051 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:33.094 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:18:33.128 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:18:33.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:33 smithi115 ceph-mon[72476]: pgmap v1472: 47 pgs: 47 active+clean; 1.5 GiB data, 8.0 GiB used, 886 GiB / 894 GiB avail; 168 MiB/s wr, 2.68k op/s 2024-03-20T09:18:33.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:33 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3887150056' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:18:33.170 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:18:33.213 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:18:33.219 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:18:33.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:33.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:18:33.247 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:18:33.286 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:18:33.295 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:18:33.295 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:18:33.295 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:18:33.295 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:18:33.295 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:18:33.295 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:18:33.295 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:18:33.295 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:18:33.295 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:18:33.295 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:18:33.296 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:18:33.296 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:18:33.324 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:18:33.327 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:33 smithi026 ceph-mon[68309]: pgmap v1472: 47 pgs: 47 active+clean; 1.5 GiB data, 8.0 GiB used, 886 GiB / 894 GiB avail; 168 MiB/s wr, 2.68k op/s 2024-03-20T09:18:33.327 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:33 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3887150056' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:18:33.354 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:33 smithi122 ceph-mon[73495]: pgmap v1472: 47 pgs: 47 active+clean; 1.5 GiB data, 8.0 GiB used, 886 GiB / 894 GiB avail; 168 MiB/s wr, 2.68k op/s 2024-03-20T09:18:33.354 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:33 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3887150056' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:18:33.357 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:33.382 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:18:33.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:33.386 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:18:33.487 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:18:33.532 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:33.622 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 13 16 33922 33906 162.988 106.5 0.00382013 0.00613046 2024-03-20T09:18:34.110 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.3 to 1 (10000) 2024-03-20T09:18:34.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:34 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3864835843' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-20T09:18:34.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:34 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3864835843' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-20T09:18:34.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:34 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3864835843' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-20T09:18:34.622 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 14 15 36313 36298 162.024 149.5 0.00525082 0.00616758 2024-03-20T09:18:35.130 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:18:35.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:35.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:18:35.239 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:18:35.281 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:35.393 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:35 smithi026 ceph-mon[68309]: pgmap v1473: 47 pgs: 47 active+clean; 1.7 GiB data, 8.8 GiB used, 885 GiB / 894 GiB avail; 168 MiB/s wr, 2.69k op/s 2024-03-20T09:18:35.393 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:35 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3864835843' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-20T09:18:35.393 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:35 smithi026 ceph-mon[68309]: osdmap e615: 12 total, 12 up, 10 in 2024-03-20T09:18:35.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:35 smithi115 ceph-mon[72476]: pgmap v1473: 47 pgs: 47 active+clean; 1.7 GiB data, 8.8 GiB used, 885 GiB / 894 GiB avail; 168 MiB/s wr, 2.69k op/s 2024-03-20T09:18:35.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:35 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3864835843' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-20T09:18:35.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:35 smithi115 ceph-mon[72476]: osdmap e615: 12 total, 12 up, 10 in 2024-03-20T09:18:35.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:35 smithi122 ceph-mon[73495]: pgmap v1473: 47 pgs: 47 active+clean; 1.7 GiB data, 8.8 GiB used, 885 GiB / 894 GiB avail; 168 MiB/s wr, 2.69k op/s 2024-03-20T09:18:35.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:35 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3864835843' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-20T09:18:35.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:35 smithi122 ceph-mon[73495]: osdmap e615: 12 total, 12 up, 10 in 2024-03-20T09:18:35.622 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 15 16 38529 38513 160.451 138.438 0.00757763 0.0062278 2024-03-20T09:18:36.112 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:18:36.138 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:36 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3549735794' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:18:36.139 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:36 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:18:36.344 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:18:36.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:36.348 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:36 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3549735794' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:18:36.349 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:36 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:18:36.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:18:36.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:36 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3549735794' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:18:36.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:36 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:18:36.450 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:18:36.485 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:36.509 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:18:36.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:36.513 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:18:36.577 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-20T09:18:36.578 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:18:36.614 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:18:36.622 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 16 16 40661 40645 158.75 133.25 0.00836935 0.00629553 2024-03-20T09:18:36.646 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:36.669 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:18:36.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:36.674 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:18:36.775 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:18:36.818 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:37.091 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:18:37.285 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:18:37.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:37.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:18:37.371 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:37 smithi026 ceph-mon[68309]: pgmap v1475: 47 pgs: 47 active+clean; 2.0 GiB data, 9.6 GiB used, 884 GiB / 894 GiB avail; 129 MiB/s wr, 2.06k op/s 2024-03-20T09:18:37.371 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:37 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:18:37.371 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:37 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:18:37.371 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:37 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:18:37.371 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:37 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:18:37.371 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:37 smithi026 ceph-mon[68309]: osdmap e616: 12 total, 12 up, 10 in 2024-03-20T09:18:37.393 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:18:37.395 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:18:37.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:37 smithi115 ceph-mon[72476]: pgmap v1475: 47 pgs: 47 active+clean; 2.0 GiB data, 9.6 GiB used, 884 GiB / 894 GiB avail; 129 MiB/s wr, 2.06k op/s 2024-03-20T09:18:37.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:37 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:18:37.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:37 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:18:37.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:37 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:18:37.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:37 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:18:37.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:37 smithi115 ceph-mon[72476]: osdmap e616: 12 total, 12 up, 10 in 2024-03-20T09:18:37.428 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:37 smithi122 ceph-mon[73495]: pgmap v1475: 47 pgs: 47 active+clean; 2.0 GiB data, 9.6 GiB used, 884 GiB / 894 GiB avail; 129 MiB/s wr, 2.06k op/s 2024-03-20T09:18:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:37 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:18:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:37 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:18:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:37 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:18:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:37 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:18:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:37 smithi122 ceph-mon[73495]: osdmap e616: 12 total, 12 up, 10 in 2024-03-20T09:18:37.622 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 17 15 42689 42674 156.871 126.812 0.00961987 0.00636992 2024-03-20T09:18:37.787 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:18:37.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:37.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:18:37.892 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:18:37.938 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:37.965 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:18:37.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:37.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:18:37.980 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:18:37.980 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:18:38.071 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:18:38.101 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:38.131 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:18:38.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:38.135 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:18:38.236 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:18:38.293 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:38.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:38 smithi122 ceph-mon[73495]: 1.4 scrub starts 2024-03-20T09:18:38.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:38 smithi122 ceph-mon[73495]: 1.4 scrub ok 2024-03-20T09:18:38.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:38 smithi122 ceph-mon[73495]: osdmap e617: 12 total, 12 up, 10 in 2024-03-20T09:18:38.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:38 smithi122 ceph-mon[73495]: 1.9 scrub starts 2024-03-20T09:18:38.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:38 smithi122 ceph-mon[73495]: 1.9 scrub ok 2024-03-20T09:18:38.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:38 smithi122 ceph-mon[73495]: pgmap v1478: 47 pgs: 47 active+clean; 2.4 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 159 MiB/s wr, 2.54k op/s 2024-03-20T09:18:38.470 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:18:38.622 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 18 16 44583 44567 154.729 118.312 0.00265332 0.00645874 2024-03-20T09:18:38.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:38 smithi115 ceph-mon[72476]: 1.4 scrub starts 2024-03-20T09:18:38.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:38 smithi115 ceph-mon[72476]: 1.4 scrub ok 2024-03-20T09:18:38.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:38 smithi115 ceph-mon[72476]: osdmap e617: 12 total, 12 up, 10 in 2024-03-20T09:18:38.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:38 smithi115 ceph-mon[72476]: 1.9 scrub starts 2024-03-20T09:18:38.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:38 smithi115 ceph-mon[72476]: 1.9 scrub ok 2024-03-20T09:18:38.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:38 smithi115 ceph-mon[72476]: pgmap v1478: 47 pgs: 47 active+clean; 2.4 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 159 MiB/s wr, 2.54k op/s 2024-03-20T09:18:38.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:38 smithi026 ceph-mon[68309]: 1.4 scrub starts 2024-03-20T09:18:38.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:38 smithi026 ceph-mon[68309]: 1.4 scrub ok 2024-03-20T09:18:38.724 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:38 smithi026 ceph-mon[68309]: osdmap e617: 12 total, 12 up, 10 in 2024-03-20T09:18:38.724 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:38 smithi026 ceph-mon[68309]: 1.9 scrub starts 2024-03-20T09:18:38.724 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:38 smithi026 ceph-mon[68309]: 1.9 scrub ok 2024-03-20T09:18:38.724 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:38 smithi026 ceph-mon[68309]: pgmap v1478: 47 pgs: 47 active+clean; 2.4 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 159 MiB/s wr, 2.54k op/s 2024-03-20T09:18:38.923 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-20T09:18:38.961 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:18:38.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:38.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:18:39.016 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:18:39.066 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:18:39.150 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:39.299 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:18:39.623 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 19 16 46505 46489 152.907 120.125 0.00322302 0.00653517 2024-03-20T09:18:39.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:39 smithi026 ceph-mon[68309]: 1.b scrub starts 2024-03-20T09:18:39.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:39 smithi026 ceph-mon[68309]: 1.b scrub ok 2024-03-20T09:18:39.909 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:39 smithi115 ceph-mon[72476]: 1.b scrub starts 2024-03-20T09:18:39.909 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:39 smithi115 ceph-mon[72476]: 1.b scrub ok 2024-03-20T09:18:39.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:39 smithi122 ceph-mon[73495]: 1.b scrub starts 2024-03-20T09:18:39.948 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:39 smithi122 ceph-mon[73495]: 1.b scrub ok 2024-03-20T09:18:40.623 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:2024-03-20T09:18:40.622845+0000 min lat: 0.0010551 max lat: 0.247256 avg lat: 0.00660406 2024-03-20T09:18:40.623 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T09:18:40.623 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 20 16 48448 48432 151.333 121.438 0.00522533 0.00660406 2024-03-20T09:18:40.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:40 smithi115 ceph-mon[72476]: 1.1b scrub starts 2024-03-20T09:18:40.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:40 smithi115 ceph-mon[72476]: 1.1b scrub ok 2024-03-20T09:18:40.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:40 smithi115 ceph-mon[72476]: 1.7 scrub starts 2024-03-20T09:18:40.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:40 smithi115 ceph-mon[72476]: 1.7 scrub ok 2024-03-20T09:18:40.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:40 smithi115 ceph-mon[72476]: 1.19 scrub starts 2024-03-20T09:18:40.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:40 smithi115 ceph-mon[72476]: pgmap v1479: 47 pgs: 47 active+clean; 2.5 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 128 MiB/s wr, 2.05k op/s 2024-03-20T09:18:40.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:40 smithi115 ceph-mon[72476]: 1.19 scrub ok 2024-03-20T09:18:40.931 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:18:40.933 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:18:40.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:40 smithi122 ceph-mon[73495]: 1.1b scrub starts 2024-03-20T09:18:40.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:40 smithi122 ceph-mon[73495]: 1.1b scrub ok 2024-03-20T09:18:40.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:40 smithi122 ceph-mon[73495]: 1.7 scrub starts 2024-03-20T09:18:40.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:40 smithi122 ceph-mon[73495]: 1.7 scrub ok 2024-03-20T09:18:40.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:40 smithi122 ceph-mon[73495]: 1.19 scrub starts 2024-03-20T09:18:40.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:40 smithi122 ceph-mon[73495]: pgmap v1479: 47 pgs: 47 active+clean; 2.5 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 128 MiB/s wr, 2.05k op/s 2024-03-20T09:18:40.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:40 smithi122 ceph-mon[73495]: 1.19 scrub ok 2024-03-20T09:18:40.955 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:40 smithi026 ceph-mon[68309]: 1.1b scrub starts 2024-03-20T09:18:40.956 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:40 smithi026 ceph-mon[68309]: 1.1b scrub ok 2024-03-20T09:18:40.956 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:40 smithi026 ceph-mon[68309]: 1.7 scrub starts 2024-03-20T09:18:40.956 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:40 smithi026 ceph-mon[68309]: 1.7 scrub ok 2024-03-20T09:18:40.956 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:40 smithi026 ceph-mon[68309]: 1.19 scrub starts 2024-03-20T09:18:40.956 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:40 smithi026 ceph-mon[68309]: pgmap v1479: 47 pgs: 47 active+clean; 2.5 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 128 MiB/s wr, 2.05k op/s 2024-03-20T09:18:40.956 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:40 smithi026 ceph-mon[68309]: 1.19 scrub ok 2024-03-20T09:18:41.623 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 21 15 49686 49671 147.814 77.4375 0.00654124 0.00676061 2024-03-20T09:18:41.869 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:41 smithi115 ceph-mon[72476]: from='client.16134 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:18:41.869 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:41 smithi115 ceph-mon[72476]: 1.6 scrub starts 2024-03-20T09:18:41.870 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:41 smithi115 ceph-mon[72476]: 1.6 scrub ok 2024-03-20T09:18:41.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:41 smithi122 ceph-mon[73495]: from='client.16134 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:18:41.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:41 smithi122 ceph-mon[73495]: 1.6 scrub starts 2024-03-20T09:18:41.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:41 smithi122 ceph-mon[73495]: 1.6 scrub ok 2024-03-20T09:18:41.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:41 smithi026 ceph-mon[68309]: from='client.16134 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:18:41.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:41 smithi026 ceph-mon[68309]: 1.6 scrub starts 2024-03-20T09:18:41.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:41 smithi026 ceph-mon[68309]: 1.6 scrub ok 2024-03-20T09:18:42.239 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:18:42.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:42.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:18:42.346 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:18:42.404 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:42.527 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:18:42.533 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:42 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:18:42.533 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:42 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:18:42.533 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:42 smithi026 ceph-mon[68309]: pgmap v1480: 47 pgs: 47 active+clean; 2.8 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 135 MiB/s wr, 2.17k op/s 2024-03-20T09:18:42.533 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:42 smithi026 ceph-mon[68309]: osdmap e618: 12 total, 12 up, 10 in 2024-03-20T09:18:42.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:42.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:18:42.623 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 22 16 51682 51666 146.762 124.688 0.00812021 0.00680972 2024-03-20T09:18:42.635 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:18:42.647 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:18:42.647 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:18:42.684 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:42.807 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":1479,"stamp":"2024-03-20T09:18:40.016247+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2682651168,"num_objects":40927,"num_object_clones":0,"num_object_copies":122781,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40925,"num_write_kb":2619200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45173,"ondisk_log_size":45173,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":11763984,"kb_used_data":7748064,"kb_used_omap":195,"kb_used_meta":4015612,"kb_avail":925687536,"statfs":{"total":959950356480,"available":947904036864,"internally_reserved":0,"allocated":7934017536,"data_stored":7930906590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":200482,"internal_metadata":4111986910},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":805109760,"num_objects":12285,"num_object_clones":0,"num_object_copies":36855,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12285,"num_write_kb":786240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.003297"},"pg_stats":[{"pgid":"1.1e","version":"20'137","reported_seq":631,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-03-20T09:18:34.091112+0000","last_change":"2024-03-20T09:18:05.872016+0000","last_active":"2024-03-20T09:18:34.091112+0000","last_peered":"2024-03-20T09:18:34.091112+0000","last_clean":"2024-03-20T09:18:34.091112+0000","last_became_active":"2024-03-20T09:13:20.071655+0000","last_became_peered":"2024-03-20T09:13:20.071655+0000","last_unstale":"2024-03-20T09:18:34.091112+0000","last_undegraded":"2024-03-20T09:18:34.091112+0000","last_fullsized":"2024-03-20T09:18:34.091112+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:18:05.871879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:18:05.871879+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:34.328518+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'137","reported_seq":802,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-03-20T09:18:37.258091+0000","last_change":"2024-03-20T09:17:45.497854+0000","last_active":"2024-03-20T09:18:37.258091+0000","last_peered":"2024-03-20T09:18:37.258091+0000","last_clean":"2024-03-20T09:18:37.258091+0000","last_became_active":"2024-03-20T09:07:42.013245+0000","last_became_peered":"2024-03-20T09:07:42.013245+0000","last_unstale":"2024-03-20T09:18:37.258091+0000","last_undegraded":"2024-03-20T09:18:37.258091+0000","last_fullsized":"2024-03-20T09:18:37.258091+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:45.497703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:17:45.497703+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:06.263941+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"20'137","reported_seq":1242,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-03-20T09:18:36.088392+0000","last_change":"2024-03-20T09:17:50.948714+0000","last_active":"2024-03-20T09:18:36.088392+0000","last_peered":"2024-03-20T09:18:36.088392+0000","last_clean":"2024-03-20T09:18:36.088392+0000","last_became_active":"2024-03-20T09:07:41.961622+0000","last_became_peered":"2024-03-20T09:07:41.961622+0000","last_unstale":"2024-03-20T09:18:36.088392+0000","last_undegraded":"2024-03-20T09:18:36.088392+0000","last_fullsized":"2024-03-20T09:18:36.088392+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:50.948579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:17:50.948579+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:07.838518+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.19","version":"20'137","reported_seq":1303,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-03-20T09:18:36.087706+0000","last_change":"2024-03-20T09:16:50.515193+0000","last_active":"2024-03-20T09:18:36.087706+0000","last_peered":"2024-03-20T09:18:36.087706+0000","last_clean":"2024-03-20T09:18:36.087706+0000","last_became_active":"2024-03-20T09:07:41.962680+0000","last_became_peered":"2024-03-20T09:07:41.962680+0000","last_unstale":"2024-03-20T09:18:36.087706+0000","last_undegraded":"2024-03-20T09:18:36.087706+0000","last_fullsized":"2024-03-20T09:18:36.087706+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:16:50.515057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:16:50.515057+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"20'137","reported_seq":773,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-03-20T09:18:36.083224+0000","last_change":"2024-03-20T09:18:02.925001+0000","last_active":"2024-03-20T09:18:36.083224+0000","last_peered":"2024-03-20T09:18:36.083224+0000","last_clean":"2024-03-20T09:18:36.083224+0000","last_became_active":"2024-03-20T09:10:35.277481+0000","last_became_peered":"2024-03-20T09:10:35.277481+0000","last_unstale":"2024-03-20T09:18:36.083224+0000","last_undegraded":"2024-03-20T09:18:36.083224+0000","last_fullsized":"2024-03-20T09:18:36.083224+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:18:02.924847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:18:02.924847+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:10.438683+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1b","version":"20'137","reported_seq":1071,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-03-20T09:18:36.088368+0000","last_change":"2024-03-20T09:16:53.286382+0000","last_active":"2024-03-20T09:18:36.088368+0000","last_peered":"2024-03-20T09:18:36.088368+0000","last_clean":"2024-03-20T09:18:36.088368+0000","last_became_active":"2024-03-20T09:07:41.962733+0000","last_became_peered":"2024-03-20T09:07:41.962733+0000","last_unstale":"2024-03-20T09:18:36.088368+0000","last_undegraded":"2024-03-20T09:18:36.088368+0000","last_fullsized":"2024-03-20T09:18:36.088368+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:16:53.286245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:16:53.286245+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:18:37.050850+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.c","version":"617'2713","reported_seq":2746,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-03-20T09:18:37.569129+0000","last_change":"2024-03-20T09:18:11.083767+0000","last_active":"2024-03-20T09:18:37.569129+0000","last_peered":"2024-03-20T09:18:37.569129+0000","last_clean":"2024-03-20T09:18:37.569129+0000","last_became_active":"2024-03-20T09:18:11.083335+0000","last_became_peered":"2024-03-20T09:18:11.083335+0000","last_unstale":"2024-03-20T09:18:37.569129+0000","last_undegraded":"2024-03-20T09:18:37.569129+0000","last_fullsized":"2024-03-20T09:18:37.569129+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":2713,"log_dups_size":0,"ondisk_log_size":2713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:14.815713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177799168,"num_objects":2713,"num_object_clones":0,"num_object_copies":8139,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2713,"num_write_kb":173632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"20'137","reported_seq":772,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-03-20T09:18:36.083232+0000","last_change":"2024-03-20T09:17:48.865046+0000","last_active":"2024-03-20T09:18:36.083232+0000","last_peered":"2024-03-20T09:18:36.083232+0000","last_clean":"2024-03-20T09:18:36.083232+0000","last_became_active":"2024-03-20T09:10:35.215772+0000","last_became_peered":"2024-03-20T09:10:35.215772+0000","last_unstale":"2024-03-20T09:18:36.083232+0000","last_undegraded":"2024-03-20T09:18:36.083232+0000","last_fullsized":"2024-03-20T09:18:36.083232+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:48.864925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:17:48.864925+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:13.258032+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"615'2321","reported_seq":2351,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-03-20T09:18:35.155084+0000","last_change":"2024-03-20T09:18:11.523337+0000","last_active":"2024-03-20T09:18:35.155084+0000","last_peered":"2024-03-20T09:18:35.155084+0000","last_clean":"2024-03-20T09:18:35.155084+0000","last_became_active":"2024-03-20T09:18:11.523040+0000","last_became_peered":"2024-03-20T09:18:11.523040+0000","last_unstale":"2024-03-20T09:18:35.155084+0000","last_undegraded":"2024-03-20T09:18:35.155084+0000","last_fullsized":"2024-03-20T09:18:35.155084+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":2321,"log_dups_size":0,"ondisk_log_size":2321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:39.162942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152109056,"num_objects":2321,"num_object_clones":0,"num_object_copies":6963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2321,"num_write_kb":148544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"20'137","reported_seq":1334,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-03-20T09:18:37.405925+0000","last_change":"2024-03-20T09:18:37.405968+0000","last_active":"2024-03-20T09:18:37.405925+0000","last_peered":"2024-03-20T09:18:37.405925+0000","last_clean":"2024-03-20T09:18:37.405925+0000","last_became_active":"2024-03-20T09:10:34.099414+0000","last_became_peered":"2024-03-20T09:10:34.099414+0000","last_unstale":"2024-03-20T09:18:37.405925+0000","last_undegraded":"2024-03-20T09:18:37.405925+0000","last_fullsized":"2024-03-20T09:18:37.405925+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:18:37.405832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:18:37.405832+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:44.985164+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"616'2592","reported_seq":2624,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-03-20T09:18:36.783519+0000","last_change":"2024-03-20T09:18:11.081562+0000","last_active":"2024-03-20T09:18:36.783519+0000","last_peered":"2024-03-20T09:18:36.783519+0000","last_clean":"2024-03-20T09:18:36.783519+0000","last_became_active":"2024-03-20T09:18:11.081322+0000","last_became_peered":"2024-03-20T09:18:11.081322+0000","last_unstale":"2024-03-20T09:18:36.783519+0000","last_undegraded":"2024-03-20T09:18:36.783519+0000","last_fullsized":"2024-03-20T09:18:36.783519+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":2592,"log_dups_size":0,"ondisk_log_size":2592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:38.316484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169869312,"num_objects":2592,"num_object_clones":0,"num_object_copies":7776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2592,"num_write_kb":165888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.a","version":"20'137","reported_seq":1128,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-03-20T09:18:34.089796+0000","last_change":"2024-03-20T09:17:10.382831+0000","last_active":"2024-03-20T09:18:34.089796+0000","last_peered":"2024-03-20T09:18:34.089796+0000","last_clean":"2024-03-20T09:18:34.089796+0000","last_became_active":"2024-03-20T09:07:52.618107+0000","last_became_peered":"2024-03-20T09:07:52.618107+0000","last_unstale":"2024-03-20T09:18:34.089796+0000","last_undegraded":"2024-03-20T09:18:34.089796+0000","last_fullsized":"2024-03-20T09:18:34.089796+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:10.382727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:17:10.382727+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:18:36.610236+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"616'2526","reported_seq":2558,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-03-20T09:18:36.270085+0000","last_change":"2024-03-20T09:18:11.081321+0000","last_active":"2024-03-20T09:18:36.270085+0000","last_peered":"2024-03-20T09:18:36.270085+0000","last_clean":"2024-03-20T09:18:36.270085+0000","last_became_active":"2024-03-20T09:18:11.080988+0000","last_became_peered":"2024-03-20T09:18:11.080988+0000","last_unstale":"2024-03-20T09:18:36.270085+0000","last_undegraded":"2024-03-20T09:18:36.270085+0000","last_fullsized":"2024-03-20T09:18:36.270085+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":2526,"log_dups_size":0,"ondisk_log_size":2526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:30.376445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165543936,"num_objects":2526,"num_object_clones":0,"num_object_copies":7578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2526,"num_write_kb":161664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,0],"acting":[10,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"20'137","reported_seq":1291,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-03-20T09:18:34.091201+0000","last_change":"2024-03-20T09:17:05.200111+0000","last_active":"2024-03-20T09:18:34.091201+0000","last_peered":"2024-03-20T09:18:34.091201+0000","last_clean":"2024-03-20T09:18:34.091201+0000","last_became_active":"2024-03-20T09:12:14.306384+0000","last_became_peered":"2024-03-20T09:12:14.306384+0000","last_unstale":"2024-03-20T09:18:34.091201+0000","last_undegraded":"2024-03-20T09:18:34.091201+0000","last_fullsized":"2024-03-20T09:18:34.091201+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:05.199980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:17:05.199980+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"615'2416","reported_seq":2445,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-03-20T09:18:35.761879+0000","last_change":"2024-03-20T09:18:11.523500+0000","last_active":"2024-03-20T09:18:35.761879+0000","last_peered":"2024-03-20T09:18:35.761879+0000","last_clean":"2024-03-20T09:18:35.761879+0000","last_became_active":"2024-03-20T09:18:11.522986+0000","last_became_peered":"2024-03-20T09:18:11.522986+0000","last_unstale":"2024-03-20T09:18:35.761879+0000","last_undegraded":"2024-03-20T09:18:35.761879+0000","last_fullsized":"2024-03-20T09:18:35.761879+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":2416,"log_dups_size":0,"ondisk_log_size":2416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:22.129530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158334976,"num_objects":2416,"num_object_clones":0,"num_object_copies":7248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2416,"num_write_kb":154624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"20'137","reported_seq":1189,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-03-20T09:18:36.088569+0000","last_change":"2024-03-20T09:17:48.424808+0000","last_active":"2024-03-20T09:18:36.088569+0000","last_peered":"2024-03-20T09:18:36.088569+0000","last_clean":"2024-03-20T09:18:36.088569+0000","last_became_active":"2024-03-20T09:07:37.003722+0000","last_became_peered":"2024-03-20T09:07:37.003722+0000","last_unstale":"2024-03-20T09:18:36.088569+0000","last_undegraded":"2024-03-20T09:18:36.088569+0000","last_fullsized":"2024-03-20T09:18:36.088569+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:48.424717+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:07:06.480160+0000","last_clean_scrub_stamp":"2024-03-20T09:17:48.424717+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:02.571050+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.2","version":"615'2437","reported_seq":2467,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-03-20T09:18:35.837669+0000","last_change":"2024-03-20T09:18:11.083520+0000","last_active":"2024-03-20T09:18:35.837669+0000","last_peered":"2024-03-20T09:18:35.837669+0000","last_clean":"2024-03-20T09:18:35.837669+0000","last_became_active":"2024-03-20T09:18:11.083136+0000","last_became_peered":"2024-03-20T09:18:11.083136+0000","last_unstale":"2024-03-20T09:18:35.837669+0000","last_undegraded":"2024-03-20T09:18:35.837669+0000","last_fullsized":"2024-03-20T09:18:35.837669+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":2437,"log_dups_size":0,"ondisk_log_size":2437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:27.386223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159711232,"num_objects":2437,"num_object_clones":0,"num_object_copies":7311,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2437,"num_write_kb":155968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"20'137","reported_seq":1206,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-03-20T09:18:36.087853+0000","last_change":"2024-03-20T09:17:13.283154+0000","last_active":"2024-03-20T09:18:36.087853+0000","last_peered":"2024-03-20T09:18:36.087853+0000","last_clean":"2024-03-20T09:18:36.087853+0000","last_became_active":"2024-03-20T09:07:52.593671+0000","last_became_peered":"2024-03-20T09:07:52.593671+0000","last_unstale":"2024-03-20T09:18:36.087853+0000","last_undegraded":"2024-03-20T09:18:36.087853+0000","last_fullsized":"2024-03-20T09:18:36.087853+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:13.283059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:17:13.283059+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:18:38.560211+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.7","version":"616'2668","reported_seq":2700,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-03-20T09:18:37.201641+0000","last_change":"2024-03-20T09:18:11.080196+0000","last_active":"2024-03-20T09:18:37.201641+0000","last_peered":"2024-03-20T09:18:37.201641+0000","last_clean":"2024-03-20T09:18:37.201641+0000","last_became_active":"2024-03-20T09:18:11.079858+0000","last_became_peered":"2024-03-20T09:18:11.079858+0000","last_unstale":"2024-03-20T09:18:37.201641+0000","last_undegraded":"2024-03-20T09:18:37.201641+0000","last_fullsized":"2024-03-20T09:18:37.201641+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":2668,"log_dups_size":0,"ondisk_log_size":2668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:31.794901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174850048,"num_objects":2668,"num_object_clones":0,"num_object_copies":8004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2668,"num_write_kb":170752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"20'137","reported_seq":1201,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-03-20T09:18:37.257749+0000","last_change":"2024-03-20T09:17:15.398442+0000","last_active":"2024-03-20T09:18:37.257749+0000","last_peered":"2024-03-20T09:18:37.257749+0000","last_clean":"2024-03-20T09:18:37.257749+0000","last_became_active":"2024-03-20T09:12:02.221686+0000","last_became_peered":"2024-03-20T09:12:02.221686+0000","last_unstale":"2024-03-20T09:18:37.257749+0000","last_undegraded":"2024-03-20T09:18:37.257749+0000","last_fullsized":"2024-03-20T09:18:37.257749+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:15.398330+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:18.788771+0000","last_clean_scrub_stamp":"2024-03-20T09:17:15.398330+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:18:43.919476+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,8],"acting":[1,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"616'2520","reported_seq":2552,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-03-20T09:18:36.277601+0000","last_change":"2024-03-20T09:18:11.078580+0000","last_active":"2024-03-20T09:18:36.277601+0000","last_peered":"2024-03-20T09:18:36.277601+0000","last_clean":"2024-03-20T09:18:36.277601+0000","last_became_active":"2024-03-20T09:18:11.078275+0000","last_became_peered":"2024-03-20T09:18:11.078275+0000","last_unstale":"2024-03-20T09:18:36.277601+0000","last_undegraded":"2024-03-20T09:18:36.277601+0000","last_fullsized":"2024-03-20T09:18:36.277601+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":2520,"log_dups_size":0,"ondisk_log_size":2520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:37.063979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165150720,"num_objects":2520,"num_object_clones":0,"num_object_copies":7560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2520,"num_write_kb":161280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":1263,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-03-20T09:18:34.091135+0000","last_change":"2024-03-20T09:17:48.991381+0000","last_active":"2024-03-20T09:18:34.091135+0000","last_peered":"2024-03-20T09:18:34.091135+0000","last_clean":"2024-03-20T09:18:34.091135+0000","last_became_active":"2024-03-20T09:12:14.305989+0000","last_became_peered":"2024-03-20T09:12:14.305989+0000","last_unstale":"2024-03-20T09:18:34.091135+0000","last_undegraded":"2024-03-20T09:18:34.091135+0000","last_fullsized":"2024-03-20T09:18:34.091135+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:48.991260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:17:48.991260+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:01.289300+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.6","version":"617'2808","reported_seq":2840,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-03-20T09:18:39.401955+0000","last_change":"2024-03-20T09:18:11.523259+0000","last_active":"2024-03-20T09:18:39.401955+0000","last_peered":"2024-03-20T09:18:39.401955+0000","last_clean":"2024-03-20T09:18:39.401955+0000","last_became_active":"2024-03-20T09:18:11.522922+0000","last_became_peered":"2024-03-20T09:18:11.522922+0000","last_unstale":"2024-03-20T09:18:39.401955+0000","last_undegraded":"2024-03-20T09:18:39.401955+0000","last_fullsized":"2024-03-20T09:18:39.401955+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":2808,"log_dups_size":0,"ondisk_log_size":2808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:32.564244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184025088,"num_objects":2808,"num_object_clones":0,"num_object_copies":8424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2808,"num_write_kb":179712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"20'137","reported_seq":1513,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-03-20T09:18:36.089423+0000","last_change":"2024-03-20T09:17:44.468318+0000","last_active":"2024-03-20T09:18:36.089423+0000","last_peered":"2024-03-20T09:18:36.089423+0000","last_clean":"2024-03-20T09:18:36.089423+0000","last_became_active":"2024-03-20T09:12:14.313541+0000","last_became_peered":"2024-03-20T09:12:14.313541+0000","last_unstale":"2024-03-20T09:18:36.089423+0000","last_undegraded":"2024-03-20T09:18:36.089423+0000","last_fullsized":"2024-03-20T09:18:36.089423+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:44.468179+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:19.513511+0000","last_clean_scrub_stamp":"2024-03-20T09:17:44.468179+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:18:58.972637+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"616'2588","reported_seq":2618,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-03-20T09:18:37.045121+0000","last_change":"2024-03-20T09:18:11.082142+0000","last_active":"2024-03-20T09:18:37.045121+0000","last_peered":"2024-03-20T09:18:37.045121+0000","last_clean":"2024-03-20T09:18:37.045121+0000","last_became_active":"2024-03-20T09:18:11.081786+0000","last_became_peered":"2024-03-20T09:18:11.081786+0000","last_unstale":"2024-03-20T09:18:37.045121+0000","last_undegraded":"2024-03-20T09:18:37.045121+0000","last_fullsized":"2024-03-20T09:18:37.045121+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":2588,"log_dups_size":0,"ondisk_log_size":2588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:16.733507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169607168,"num_objects":2588,"num_object_clones":0,"num_object_copies":7764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2588,"num_write_kb":165632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"20'137","reported_seq":665,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-03-20T09:18:34.090003+0000","last_change":"2024-03-20T09:17:47.350252+0000","last_active":"2024-03-20T09:18:34.090003+0000","last_peered":"2024-03-20T09:18:34.090003+0000","last_clean":"2024-03-20T09:18:34.090003+0000","last_became_active":"2024-03-20T09:15:34.225022+0000","last_became_peered":"2024-03-20T09:15:34.225022+0000","last_unstale":"2024-03-20T09:18:34.090003+0000","last_undegraded":"2024-03-20T09:18:34.090003+0000","last_fullsized":"2024-03-20T09:18:34.090003+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:47.350114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:17:47.350114+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:00.915901+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,5],"acting":[6,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"616'2581","reported_seq":2611,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-03-20T09:18:37.025694+0000","last_change":"2024-03-20T09:18:11.081759+0000","last_active":"2024-03-20T09:18:37.025694+0000","last_peered":"2024-03-20T09:18:37.025694+0000","last_clean":"2024-03-20T09:18:37.025694+0000","last_became_active":"2024-03-20T09:18:11.081425+0000","last_became_peered":"2024-03-20T09:18:11.081425+0000","last_unstale":"2024-03-20T09:18:37.025694+0000","last_undegraded":"2024-03-20T09:18:37.025694+0000","last_fullsized":"2024-03-20T09:18:37.025694+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":2581,"log_dups_size":0,"ondisk_log_size":2581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:22.317679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169148416,"num_objects":2581,"num_object_clones":0,"num_object_copies":7743,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2581,"num_write_kb":165184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"20'137","reported_seq":1200,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-03-20T09:18:34.089888+0000","last_change":"2024-03-20T09:17:02.462532+0000","last_active":"2024-03-20T09:18:34.089888+0000","last_peered":"2024-03-20T09:18:34.089888+0000","last_clean":"2024-03-20T09:18:34.089888+0000","last_became_active":"2024-03-20T09:13:27.087760+0000","last_became_peered":"2024-03-20T09:13:27.087760+0000","last_unstale":"2024-03-20T09:18:34.089888+0000","last_undegraded":"2024-03-20T09:18:34.089888+0000","last_fullsized":"2024-03-20T09:18:34.089888+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:02.462423+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:14:10.560890+0000","last_clean_scrub_stamp":"2024-03-20T09:17:02.462423+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:18:36.610193+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"616'2570","reported_seq":2602,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-03-20T09:18:37.205174+0000","last_change":"2024-03-20T09:18:11.079851+0000","last_active":"2024-03-20T09:18:37.205174+0000","last_peered":"2024-03-20T09:18:37.205174+0000","last_clean":"2024-03-20T09:18:37.205174+0000","last_became_active":"2024-03-20T09:18:11.079589+0000","last_became_peered":"2024-03-20T09:18:11.079589+0000","last_unstale":"2024-03-20T09:18:37.205174+0000","last_undegraded":"2024-03-20T09:18:37.205174+0000","last_fullsized":"2024-03-20T09:18:37.205174+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":2570,"log_dups_size":0,"ondisk_log_size":2570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:36.062370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168427520,"num_objects":2570,"num_object_clones":0,"num_object_copies":7710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2570,"num_write_kb":164480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,2],"acting":[5,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"20'137","reported_seq":1068,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-03-20T09:18:34.091096+0000","last_change":"2024-03-20T09:17:16.172485+0000","last_active":"2024-03-20T09:18:34.091096+0000","last_peered":"2024-03-20T09:18:34.091096+0000","last_clean":"2024-03-20T09:18:34.091096+0000","last_became_active":"2024-03-20T09:13:20.070596+0000","last_became_peered":"2024-03-20T09:13:20.070596+0000","last_unstale":"2024-03-20T09:18:34.091096+0000","last_undegraded":"2024-03-20T09:18:34.091096+0000","last_fullsized":"2024-03-20T09:18:34.091096+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:16.172375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:17:16.172375+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:18:36.303263+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.a","version":"617'2950","reported_seq":2984,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-03-20T09:18:39.370823+0000","last_change":"2024-03-20T09:18:11.079740+0000","last_active":"2024-03-20T09:18:39.370823+0000","last_peered":"2024-03-20T09:18:39.370823+0000","last_clean":"2024-03-20T09:18:39.370823+0000","last_became_active":"2024-03-20T09:18:11.079361+0000","last_became_peered":"2024-03-20T09:18:11.079361+0000","last_unstale":"2024-03-20T09:18:39.370823+0000","last_undegraded":"2024-03-20T09:18:39.370823+0000","last_fullsized":"2024-03-20T09:18:39.370823+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":2950,"log_dups_size":0,"ondisk_log_size":2950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:28.144588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193331200,"num_objects":2950,"num_object_clones":0,"num_object_copies":8850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2950,"num_write_kb":188800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,10],"acting":[2,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"20'137","reported_seq":727,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-03-20T09:18:34.092282+0000","last_change":"2024-03-20T09:17:46.760738+0000","last_active":"2024-03-20T09:18:34.092282+0000","last_peered":"2024-03-20T09:18:34.092282+0000","last_clean":"2024-03-20T09:18:34.092282+0000","last_became_active":"2024-03-20T09:12:14.311319+0000","last_became_peered":"2024-03-20T09:12:14.311319+0000","last_unstale":"2024-03-20T09:18:34.092282+0000","last_undegraded":"2024-03-20T09:18:34.092282+0000","last_fullsized":"2024-03-20T09:18:34.092282+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:46.760611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:17:46.760611+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:18:47.142571+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"616'2540","reported_seq":2571,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-03-20T09:18:37.018026+0000","last_change":"2024-03-20T09:18:11.523380+0000","last_active":"2024-03-20T09:18:37.018026+0000","last_peered":"2024-03-20T09:18:37.018026+0000","last_clean":"2024-03-20T09:18:37.018026+0000","last_became_active":"2024-03-20T09:18:11.522906+0000","last_became_peered":"2024-03-20T09:18:11.522906+0000","last_unstale":"2024-03-20T09:18:37.018026+0000","last_undegraded":"2024-03-20T09:18:37.018026+0000","last_fullsized":"2024-03-20T09:18:37.018026+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":2540,"log_dups_size":0,"ondisk_log_size":2540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:19.027570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166461440,"num_objects":2540,"num_object_clones":0,"num_object_copies":7620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2540,"num_write_kb":162560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"20'137","reported_seq":1216,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-03-20T09:18:34.089979+0000","last_change":"2024-03-20T09:17:15.472439+0000","last_active":"2024-03-20T09:18:34.089979+0000","last_peered":"2024-03-20T09:18:34.089979+0000","last_clean":"2024-03-20T09:18:34.089979+0000","last_became_active":"2024-03-20T09:10:35.119706+0000","last_became_peered":"2024-03-20T09:10:35.119706+0000","last_unstale":"2024-03-20T09:18:34.089979+0000","last_undegraded":"2024-03-20T09:18:34.089979+0000","last_fullsized":"2024-03-20T09:18:34.089979+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:15.472340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:17:15.472340+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:18:36.610218+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"615'2243","reported_seq":2273,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-03-20T09:18:35.147747+0000","last_change":"2024-03-20T09:18:11.080513+0000","last_active":"2024-03-20T09:18:35.147747+0000","last_peered":"2024-03-20T09:18:35.147747+0000","last_clean":"2024-03-20T09:18:35.147747+0000","last_became_active":"2024-03-20T09:18:11.080212+0000","last_became_peered":"2024-03-20T09:18:11.080212+0000","last_unstale":"2024-03-20T09:18:35.147747+0000","last_undegraded":"2024-03-20T09:18:35.147747+0000","last_fullsized":"2024-03-20T09:18:35.147747+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":2243,"log_dups_size":0,"ondisk_log_size":2243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:14.576641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146997248,"num_objects":2243,"num_object_clones":0,"num_object_copies":6729,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2243,"num_write_kb":143552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"570'138","reported_seq":1243,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-03-20T09:18:34.092456+0000","last_change":"2024-03-20T09:17:45.816003+0000","last_active":"2024-03-20T09:18:34.092456+0000","last_peered":"2024-03-20T09:18:34.092456+0000","last_clean":"2024-03-20T09:18:34.092456+0000","last_became_active":"2024-03-20T09:13:27.087686+0000","last_became_peered":"2024-03-20T09:13:27.087686+0000","last_unstale":"2024-03-20T09:18:34.092456+0000","last_undegraded":"2024-03-20T09:18:34.092456+0000","last_fullsized":"2024-03-20T09:18:34.092456+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"570'138","last_scrub_stamp":"2024-03-20T09:17:45.815855+0000","last_deep_scrub":"570'138","last_deep_scrub_stamp":"2024-03-20T09:16:13.725035+0000","last_clean_scrub_stamp":"2024-03-20T09:17:45.815855+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:05.315376+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"615'2452","reported_seq":2482,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-03-20T09:18:35.837851+0000","last_change":"2024-03-20T09:18:11.523430+0000","last_active":"2024-03-20T09:18:35.837851+0000","last_peered":"2024-03-20T09:18:35.837851+0000","last_clean":"2024-03-20T09:18:35.837851+0000","last_became_active":"2024-03-20T09:18:11.522844+0000","last_became_peered":"2024-03-20T09:18:11.522844+0000","last_unstale":"2024-03-20T09:18:35.837851+0000","last_undegraded":"2024-03-20T09:18:35.837851+0000","last_fullsized":"2024-03-20T09:18:35.837851+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":2452,"log_dups_size":0,"ondisk_log_size":2452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:10.081321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160694272,"num_objects":2452,"num_object_clones":0,"num_object_copies":7356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2452,"num_write_kb":156928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,11],"acting":[6,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.d","version":"20'137","reported_seq":799,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-03-20T09:18:37.257975+0000","last_change":"2024-03-20T09:18:05.353742+0000","last_active":"2024-03-20T09:18:37.257975+0000","last_peered":"2024-03-20T09:18:37.257975+0000","last_clean":"2024-03-20T09:18:37.257975+0000","last_became_active":"2024-03-20T09:12:14.307864+0000","last_became_peered":"2024-03-20T09:12:14.307864+0000","last_unstale":"2024-03-20T09:18:37.257975+0000","last_undegraded":"2024-03-20T09:18:37.257975+0000","last_fullsized":"2024-03-20T09:18:37.257975+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:18:05.353559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:18:05.353559+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:07.761487+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"20'137","reported_seq":1113,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-03-20T09:18:36.083347+0000","last_change":"2024-03-20T09:17:47.874749+0000","last_active":"2024-03-20T09:18:36.083347+0000","last_peered":"2024-03-20T09:18:36.083347+0000","last_clean":"2024-03-20T09:18:36.083347+0000","last_became_active":"2024-03-20T09:10:34.096758+0000","last_became_peered":"2024-03-20T09:10:34.096758+0000","last_unstale":"2024-03-20T09:18:36.083347+0000","last_undegraded":"2024-03-20T09:18:36.083347+0000","last_fullsized":"2024-03-20T09:18:36.083347+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:47.874609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:17:47.874609+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:11.136312+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"20'137","reported_seq":1188,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-03-20T09:18:37.257578+0000","last_change":"2024-03-20T09:17:57.414516+0000","last_active":"2024-03-20T09:18:37.257578+0000","last_peered":"2024-03-20T09:18:37.257578+0000","last_clean":"2024-03-20T09:18:37.257578+0000","last_became_active":"2024-03-20T09:04:04.103645+0000","last_became_peered":"2024-03-20T09:04:04.103645+0000","last_unstale":"2024-03-20T09:18:37.257578+0000","last_undegraded":"2024-03-20T09:18:37.257578+0000","last_fullsized":"2024-03-20T09:18:37.257578+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:57.414409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:17:57.414409+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:26.874738+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"20'137","reported_seq":639,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-03-20T09:18:36.083346+0000","last_change":"2024-03-20T09:17:50.805976+0000","last_active":"2024-03-20T09:18:36.083346+0000","last_peered":"2024-03-20T09:18:36.083346+0000","last_clean":"2024-03-20T09:18:36.083346+0000","last_became_active":"2024-03-20T09:10:34.097191+0000","last_became_peered":"2024-03-20T09:10:34.097191+0000","last_unstale":"2024-03-20T09:18:36.083346+0000","last_undegraded":"2024-03-20T09:18:36.083346+0000","last_fullsized":"2024-03-20T09:18:36.083346+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:50.805845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:17:50.805845+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:00.311287+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"20'137","reported_seq":1049,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-03-20T09:18:36.087213+0000","last_change":"2024-03-20T09:17:46.485538+0000","last_active":"2024-03-20T09:18:36.087213+0000","last_peered":"2024-03-20T09:18:36.087213+0000","last_clean":"2024-03-20T09:18:36.087213+0000","last_became_active":"2024-03-20T09:13:27.087859+0000","last_became_peered":"2024-03-20T09:13:27.087859+0000","last_unstale":"2024-03-20T09:18:36.087213+0000","last_undegraded":"2024-03-20T09:18:36.087213+0000","last_fullsized":"2024-03-20T09:18:36.087213+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:46.485413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:17:46.485413+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:13.166557+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,3],"acting":[0,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"20'137","reported_seq":877,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-03-20T09:18:37.257841+0000","last_change":"2024-03-20T09:17:52.461386+0000","last_active":"2024-03-20T09:18:37.257841+0000","last_peered":"2024-03-20T09:18:37.257841+0000","last_clean":"2024-03-20T09:18:37.257841+0000","last_became_active":"2024-03-20T09:13:27.087760+0000","last_became_peered":"2024-03-20T09:13:27.087760+0000","last_unstale":"2024-03-20T09:18:37.257841+0000","last_undegraded":"2024-03-20T09:18:37.257841+0000","last_fullsized":"2024-03-20T09:18:37.257841+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:52.461261+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:16:15.448920+0000","last_clean_scrub_stamp":"2024-03-20T09:17:52.461261+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:16.137279+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"20'137","reported_seq":1193,"reported_epoch":616,"state":"active+clean","last_fresh":"2024-03-20T09:18:36.088709+0000","last_change":"2024-03-20T09:17:46.961152+0000","last_active":"2024-03-20T09:18:36.088709+0000","last_peered":"2024-03-20T09:18:36.088709+0000","last_clean":"2024-03-20T09:18:36.088709+0000","last_became_active":"2024-03-20T09:10:35.095339+0000","last_became_peered":"2024-03-20T09:10:35.095339+0000","last_unstale":"2024-03-20T09:18:36.088709+0000","last_undegraded":"2024-03-20T09:18:36.088709+0000","last_fullsized":"2024-03-20T09:18:36.088709+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:46.961059+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:23.640533+0000","last_clean_scrub_stamp":"2024-03-20T09:17:46.961059+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:11.863478+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"20'137","reported_seq":1264,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-03-20T09:18:37.255639+0000","last_change":"2024-03-20T09:17:47.414530+0000","last_active":"2024-03-20T09:18:37.255639+0000","last_peered":"2024-03-20T09:18:37.255639+0000","last_clean":"2024-03-20T09:18:37.255639+0000","last_became_active":"2024-03-20T09:04:14.208448+0000","last_became_peered":"2024-03-20T09:04:14.208448+0000","last_unstale":"2024-03-20T09:18:37.255639+0000","last_undegraded":"2024-03-20T09:18:37.255639+0000","last_fullsized":"2024-03-20T09:18:37.255639+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:47.414381+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:16:15.754362+0000","last_clean_scrub_stamp":"2024-03-20T09:17:47.414381+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:08.321085+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"20'137","reported_seq":524,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-03-20T09:18:34.091607+0000","last_change":"2024-03-20T09:17:44.727578+0000","last_active":"2024-03-20T09:18:34.091607+0000","last_peered":"2024-03-20T09:18:34.091607+0000","last_clean":"2024-03-20T09:18:34.091607+0000","last_became_active":"2024-03-20T09:12:14.307581+0000","last_became_peered":"2024-03-20T09:12:14.307581+0000","last_unstale":"2024-03-20T09:18:34.091607+0000","last_undegraded":"2024-03-20T09:18:34.091607+0000","last_fullsized":"2024-03-20T09:18:34.091607+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:44.727492+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:07:06.480160+0000","last_clean_scrub_stamp":"2024-03-20T09:17:44.727492+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:18:46.002339+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"20'137","reported_seq":808,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-03-20T09:18:37.257646+0000","last_change":"2024-03-20T09:17:59.332278+0000","last_active":"2024-03-20T09:18:37.257646+0000","last_peered":"2024-03-20T09:18:37.257646+0000","last_clean":"2024-03-20T09:18:37.257646+0000","last_became_active":"2024-03-20T09:13:20.074439+0000","last_became_peered":"2024-03-20T09:13:20.074439+0000","last_unstale":"2024-03-20T09:18:37.257646+0000","last_undegraded":"2024-03-20T09:18:37.257646+0000","last_fullsized":"2024-03-20T09:18:37.257646+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:59.332142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:17:59.332142+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:28.719063+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,11],"acting":[1,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":2682060800,"num_objects":40925,"num_object_clones":0,"num_object_copies":122775,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40925,"num_write_kb":2619200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8066891776,"data_stored":8066891776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40925,"ondisk_log_size":40925,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4248,"ondisk_log_size":4248,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860686,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1083308,"kb_used_data":582880,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":92661844,"statfs":{"total":95995035648,"available":94885728256,"internally_reserved":0,"allocated":596869120,"data_stored":596561780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":512424388},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":89,"seq":382252089620,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1313772,"kb_used_data":945248,"kb_used_omap":1,"kb_used_meta":368510,"kb_avail":92431380,"statfs":{"total":95995035648,"available":94649733120,"internally_reserved":0,"allocated":967933952,"data_stored":967626612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":377354688},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":0,"seq":343597383965,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":70,"seq":300647711008,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":594728,"kb_used_data":320608,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":93150424,"statfs":{"total":95995035648,"available":95386034176,"internally_reserved":0,"allocated":328302592,"data_stored":327995252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":280689094},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":60,"seq":257698038053,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1276084,"kb_used_data":763176,"kb_used_omap":1,"kb_used_meta":512894,"kb_avail":92469068,"statfs":{"total":95995035648,"available":94688325632,"internally_reserved":0,"allocated":781492224,"data_stored":781181332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":525203914},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332394,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1185896,"kb_used_data":775712,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92559256,"statfs":{"total":95995035648,"available":94780678144,"internally_reserved":0,"allocated":794329088,"data_stored":794021748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":420018630},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575172,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1226632,"kb_used_data":681784,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":92518520,"statfs":{"total":95995035648,"available":94738964480,"internally_reserved":0,"allocated":698146816,"data_stored":697830496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":557906374},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738691,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1753784,"kb_used_data":1285292,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":91991368,"statfs":{"total":95995035648,"available":94199160832,"internally_reserved":0,"allocated":1316139008,"data_stored":1315827203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411646,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1237000,"kb_used_data":917056,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92508152,"statfs":{"total":95995035648,"available":94728347648,"internally_reserved":0,"allocated":939065344,"data_stored":938745472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":327612869},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150009,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":563308,"kb_used_data":148960,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":93181844,"statfs":{"total":95995035648,"available":95418208256,"internally_reserved":0,"allocated":152535040,"data_stored":152227700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":424278471},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953783,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":443,"seq":1902670512226,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1529472,"kb_used_data":1327348,"kb_used_omap":181,"kb_used_meta":201930,"kb_avail":92215680,"statfs":{"total":95995035648,"available":94428856320,"internally_reserved":0,"allocated":1359204352,"data_stored":1358888995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":186131,"internal_metadata":206776557},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1359216640,"data_stored":1359216640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":707002368,"data_stored":707002368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":944242688,"data_stored":944242688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":158400512,"data_stored":158400512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1363542016,"data_stored":1363542016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":804323328,"data_stored":804323328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":797245440,"data_stored":797245440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":341901312,"data_stored":341901312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":986972160,"data_stored":986972160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":604045312,"data_stored":604045312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:18:42.810 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:18:42.899 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:18:42.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:42 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:18:42.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:42 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:18:42.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:42 smithi115 ceph-mon[72476]: pgmap v1480: 47 pgs: 47 active+clean; 2.8 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 135 MiB/s wr, 2.17k op/s 2024-03-20T09:18:42.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:42 smithi115 ceph-mon[72476]: osdmap e618: 12 total, 12 up, 10 in 2024-03-20T09:18:42.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:42.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:18:42.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:42 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:18:42.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:42 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:18:42.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:42 smithi122 ceph-mon[73495]: pgmap v1480: 47 pgs: 47 active+clean; 2.8 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 135 MiB/s wr, 2.17k op/s 2024-03-20T09:18:42.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:42 smithi122 ceph-mon[73495]: osdmap e618: 12 total, 12 up, 10 in 2024-03-20T09:18:43.006 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:18:43.049 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:43.159 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:18:43.377 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-20T09:18:43.411 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:18:43.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:43.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:18:43.516 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:18:43.551 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:43.576 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:18:43.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:43.580 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:18:43.623 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 23 16 53112 53096 144.267 89.375 0.00465241 0.00692517 2024-03-20T09:18:43.680 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:18:43.731 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:43.808 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:18:44.146 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:18:44.623 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 24 16 55006 54990 143.188 118.375 0.00260971 0.00697709 2024-03-20T09:18:44.798 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:18:44.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:44.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:18:44.818 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:44 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:18:44.818 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:44 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:18:44.818 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:44 smithi115 ceph-mon[72476]: 1.f scrub starts 2024-03-20T09:18:44.818 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:44 smithi115 ceph-mon[72476]: osdmap e619: 12 total, 12 up, 10 in 2024-03-20T09:18:44.818 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:44 smithi115 ceph-mon[72476]: 1.f scrub ok 2024-03-20T09:18:44.818 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:44 smithi115 ceph-mon[72476]: pgmap v1483: 47 pgs: 47 active+clean; 3.0 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-03-20T09:18:44.822 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:44 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:18:44.822 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:44 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:18:44.822 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:44 smithi026 ceph-mon[68309]: 1.f scrub starts 2024-03-20T09:18:44.822 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:44 smithi026 ceph-mon[68309]: osdmap e619: 12 total, 12 up, 10 in 2024-03-20T09:18:44.822 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:44 smithi026 ceph-mon[68309]: 1.f scrub ok 2024-03-20T09:18:44.822 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:44 smithi026 ceph-mon[68309]: pgmap v1483: 47 pgs: 47 active+clean; 3.0 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-03-20T09:18:44.904 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:18:44.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:44 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:18:44.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:44 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:18:44.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:44 smithi122 ceph-mon[73495]: 1.f scrub starts 2024-03-20T09:18:44.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:44 smithi122 ceph-mon[73495]: osdmap e619: 12 total, 12 up, 10 in 2024-03-20T09:18:44.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:44 smithi122 ceph-mon[73495]: 1.f scrub ok 2024-03-20T09:18:44.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:44 smithi122 ceph-mon[73495]: pgmap v1483: 47 pgs: 47 active+clean; 3.0 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-03-20T09:18:44.953 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:45.164 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:18:45.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:45.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:18:45.268 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:18:45.295 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:45.346 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:18:45.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:45.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:18:45.450 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:18:45.521 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:18:45.534 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:45.574 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:18:45.623 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 25 16 56446 56430 141.06 90 0.00244315 0.00707984 2024-03-20T09:18:45.628 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:18:45.634 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:18:45.672 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:18:45.711 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:18:45.755 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:18:45.789 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:18:45.842 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:18:45.854 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:18:45.888 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:18:45.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:45 smithi115 ceph-mon[72476]: 1.a scrub starts 2024-03-20T09:18:45.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:45 smithi115 ceph-mon[72476]: 1.a scrub ok 2024-03-20T09:18:45.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:45 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3670607790' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:18:45.913 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:18:45.913 INFO:teuthology.orchestra.run.smithi115.stdout: "size": 20, 2024-03-20T09:18:45.913 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 600, 2024-03-20T09:18:45.913 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [ 2024-03-20T09:18:45.913 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.913 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "MOSDPGRecoveryDelete(3.2 e447,444 3:47fea1fd:::benchmark_data_smithi122_119671_object26434:head,362'5756)", 2024-03-20T09:18:45.913 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:10:37.805969+0000", 2024-03-20T09:18:45.913 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 488.10326724200002, 2024-03-20T09:18:45.913 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.10664080400000001, 2024-03-20T09:18:45.913 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:18:45.913 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:18:45.913 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:18:45.913 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.914 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:18:45.914 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:37.805969+0000", 2024-03-20T09:18:45.914 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:45.914 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.914 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.914 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:18:45.914 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:37.805969+0000", 2024-03-20T09:18:45.914 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:45.914 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.914 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.914 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:18:45.914 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:37.805970+0000", 2024-03-20T09:18:45.914 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5199999999999997e-07 2024-03-20T09:18:45.915 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.915 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.915 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:18:45.915 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:37.805973+0000", 2024-03-20T09:18:45.915 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.878e-06 2024-03-20T09:18:45.915 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.915 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.915 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:18:45.915 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:37.912467+0000", 2024-03-20T09:18:45.915 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.10649381400000001 2024-03-20T09:18:45.915 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.915 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.915 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:18:45.915 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:37.912494+0000", 2024-03-20T09:18:45.916 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6780000000000001e-05 2024-03-20T09:18:45.916 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.916 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.916 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:18:45.916 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:37.912548+0000", 2024-03-20T09:18:45.916 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4431000000000001e-05 2024-03-20T09:18:45.916 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.916 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.916 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:18:45.916 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:37.912557+0000", 2024-03-20T09:18:45.916 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.7579999999999998e-06 2024-03-20T09:18:45.916 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.916 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.916 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:18:45.917 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:37.912610+0000", 2024-03-20T09:18:45.917 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2590999999999998e-05 2024-03-20T09:18:45.917 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:45.917 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:18:45.917 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:45.917 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.917 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.917 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "MOSDPGRecoveryDelete(3.4 e447,444 3:24fd9fa0:::benchmark_data_smithi122_119671_object1710:head,284'4900)", 2024-03-20T09:18:45.917 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:10:39.199845+0000", 2024-03-20T09:18:45.917 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 486.70939135999998, 2024-03-20T09:18:45.917 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.124788118, 2024-03-20T09:18:45.917 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:18:45.918 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:18:45.918 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:18:45.918 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.918 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:18:45.918 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:39.199845+0000", 2024-03-20T09:18:45.918 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:45.918 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.918 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.918 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:18:45.918 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:39.199845+0000", 2024-03-20T09:18:45.918 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:45.918 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.918 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.919 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:18:45.919 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:39.199845+0000", 2024-03-20T09:18:45.919 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5000000000000003e-07 2024-03-20T09:18:45.919 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.919 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.919 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:18:45.919 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:39.199849+0000", 2024-03-20T09:18:45.919 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6950000000000002e-06 2024-03-20T09:18:45.919 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.919 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.919 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:18:45.919 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:39.324485+0000", 2024-03-20T09:18:45.919 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.124635656 2024-03-20T09:18:45.919 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.920 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.920 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:18:45.920 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:39.324500+0000", 2024-03-20T09:18:45.920 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4796e-05 2024-03-20T09:18:45.920 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.920 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.920 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:18:45.920 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:39.324570+0000", 2024-03-20T09:18:45.920 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.0035000000000003e-05 2024-03-20T09:18:45.920 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.920 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.920 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:18:45.920 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:39.324578+0000", 2024-03-20T09:18:45.920 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5709999999999998e-06 2024-03-20T09:18:45.921 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.921 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.921 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:18:45.921 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:39.324633+0000", 2024-03-20T09:18:45.921 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4815000000000003e-05 2024-03-20T09:18:45.921 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:45.921 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:18:45.921 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:45.921 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.921 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.921 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.24910.0:143615 3.5 3:a5aec83c:::benchmark_data_smithi122_119671_object43833:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e449)", 2024-03-20T09:18:45.921 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:10:41.618190+0000", 2024-03-20T09:18:45.921 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 484.29104608099999, 2024-03-20T09:18:45.922 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54554617800000005, 2024-03-20T09:18:45.922 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:18:45.922 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:45.922 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:18:45.922 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.24910", 2024-03-20T09:18:45.922 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:18:45.922 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 143615 2024-03-20T09:18:45.922 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.922 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:18:45.922 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.922 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:18:45.922 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:41.618190+0000", 2024-03-20T09:18:45.922 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:45.923 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.923 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.923 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:18:45.923 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:41.618190+0000", 2024-03-20T09:18:45.923 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:45.923 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.923 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.923 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:18:45.923 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:41.618191+0000", 2024-03-20T09:18:45.923 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.249e-06 2024-03-20T09:18:45.923 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.923 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.924 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:18:45.924 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:41.618198+0000", 2024-03-20T09:18:45.924 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2570000000000004e-06 2024-03-20T09:18:45.924 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.924 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.924 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:18:45.924 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:41.618209+0000", 2024-03-20T09:18:45.924 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1052e-05 2024-03-20T09:18:45.924 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.924 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.924 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:18:45.924 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:41.618217+0000", 2024-03-20T09:18:45.924 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.9209999999999992e-06 2024-03-20T09:18:45.924 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.925 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.925 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:18:45.925 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:41.618271+0000", 2024-03-20T09:18:45.925 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4453000000000002e-05 2024-03-20T09:18:45.925 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.925 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.925 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:18:45.925 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:41.618278+0000", 2024-03-20T09:18:45.925 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.7950000000000002e-06 2024-03-20T09:18:45.925 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.925 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.925 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:18:45.925 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:42.155475+0000", 2024-03-20T09:18:45.926 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.53719698299999996 2024-03-20T09:18:45.926 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.926 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.926 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:18:45.926 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:42.155563+0000", 2024-03-20T09:18:45.926 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.7811000000000003e-05 2024-03-20T09:18:45.926 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.926 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.926 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,6", 2024-03-20T09:18:45.926 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:42.155641+0000", 2024-03-20T09:18:45.926 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.8677999999999994e-05 2024-03-20T09:18:45.926 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.926 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.927 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:45.927 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:42.160290+0000", 2024-03-20T09:18:45.927 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0046481680000000003 2024-03-20T09:18:45.927 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.927 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.927 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:18:45.927 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:42.160305+0000", 2024-03-20T09:18:45.927 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5483999999999999e-05 2024-03-20T09:18:45.927 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.927 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.927 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:45.927 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:42.163448+0000", 2024-03-20T09:18:45.927 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0031426840000000002 2024-03-20T09:18:45.927 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.928 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.928 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:18:45.928 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:42.163509+0000", 2024-03-20T09:18:45.928 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0959999999999999e-05 2024-03-20T09:18:45.928 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.928 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.928 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:18:45.928 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:42.163736+0000", 2024-03-20T09:18:45.928 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00022768300000000001 2024-03-20T09:18:45.928 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:45.928 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:18:45.928 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:45.928 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.928 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.929 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24910.0:143643 3.16 e450/444 3:6e30d992:::benchmark_data_smithi122_119671_object43861:head v 450'8417, mlcod=285'5106)", 2024-03-20T09:18:45.929 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:10:42.187214+0000", 2024-03-20T09:18:45.929 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 483.722022477, 2024-03-20T09:18:45.929 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.077928490000000003, 2024-03-20T09:18:45.929 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:18:45.929 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:45.929 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:18:45.929 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.929 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:18:45.929 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:42.187214+0000", 2024-03-20T09:18:45.929 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:45.929 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.929 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.929 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:18:45.930 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:42.187214+0000", 2024-03-20T09:18:45.930 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:45.930 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.930 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.930 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:18:45.930 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:42.187215+0000", 2024-03-20T09:18:45.930 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.0400000000000005e-07 2024-03-20T09:18:45.930 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.930 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.930 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:18:45.930 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:42.187222+0000", 2024-03-20T09:18:45.930 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.8310000000000001e-06 2024-03-20T09:18:45.930 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.930 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.931 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:18:45.931 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:42.187232+0000", 2024-03-20T09:18:45.931 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.7289999999999995e-06 2024-03-20T09:18:45.931 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.931 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.931 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:18:45.931 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:42.187244+0000", 2024-03-20T09:18:45.931 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1674999999999999e-05 2024-03-20T09:18:45.931 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.931 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.931 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:18:45.931 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:42.187297+0000", 2024-03-20T09:18:45.931 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3582999999999998e-05 2024-03-20T09:18:45.932 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.932 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.932 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:18:45.932 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:42.187327+0000", 2024-03-20T09:18:45.932 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9524999999999999e-05 2024-03-20T09:18:45.932 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.932 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.932 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:18:45.932 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:42.265093+0000", 2024-03-20T09:18:45.932 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.077766207000000004 2024-03-20T09:18:45.932 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.932 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.932 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:18:45.933 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:42.265142+0000", 2024-03-20T09:18:45.933 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9135999999999998e-05 2024-03-20T09:18:45.933 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:45.933 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:18:45.933 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:45.933 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.933 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.933 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.24910.0:143645 3.16 e450/444 3:6c3eaf40:::benchmark_data_smithi122_119671_object43863:head v 450'8418, mlcod=285'5106)", 2024-03-20T09:18:45.933 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:10:42.193636+0000", 2024-03-20T09:18:45.933 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 483.71560072900002, 2024-03-20T09:18:45.933 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.071539695, 2024-03-20T09:18:45.933 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:18:45.933 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:45.933 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:18:45.933 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.934 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:18:45.934 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:42.193636+0000", 2024-03-20T09:18:45.934 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:45.934 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.934 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.934 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:18:45.934 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:42.193636+0000", 2024-03-20T09:18:45.934 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:45.934 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.934 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.934 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:18:45.934 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:42.193636+0000", 2024-03-20T09:18:45.934 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.4099999999999997e-07 2024-03-20T09:18:45.934 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.935 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.935 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:18:45.935 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:42.193645+0000", 2024-03-20T09:18:45.935 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.6799999999999999e-06 2024-03-20T09:18:45.935 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.935 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.935 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:18:45.935 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:42.193655+0000", 2024-03-20T09:18:45.935 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0163000000000001e-05 2024-03-20T09:18:45.935 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.935 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.935 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:18:45.935 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:42.193662+0000", 2024-03-20T09:18:45.935 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.314e-06 2024-03-20T09:18:45.935 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.936 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.936 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:18:45.936 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:42.193720+0000", 2024-03-20T09:18:45.936 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8526999999999997e-05 2024-03-20T09:18:45.936 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.936 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.936 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:18:45.936 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:42.193747+0000", 2024-03-20T09:18:45.936 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.7370000000000001e-05 2024-03-20T09:18:45.936 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.936 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.936 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:18:45.936 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:42.265146+0000", 2024-03-20T09:18:45.936 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.071398270999999999 2024-03-20T09:18:45.936 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.937 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.937 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:18:45.937 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:10:42.265175+0000", 2024-03-20T09:18:45.937 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9529000000000001e-05 2024-03-20T09:18:45.937 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:45.937 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:18:45.937 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:45.937 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.937 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.937 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.25690.0:27568 4.2 4:4972ced8:::benchmark_data_smithi122_145698_object27567:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e533)", 2024-03-20T09:18:45.937 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:13:43.694271+0000", 2024-03-20T09:18:45.937 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 302.214965538, 2024-03-20T09:18:45.937 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.082637444000000004, 2024-03-20T09:18:45.937 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:18:45.938 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:45.938 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:18:45.938 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.25690", 2024-03-20T09:18:45.938 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:18:45.938 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 27568 2024-03-20T09:18:45.938 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.938 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:18:45.938 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.938 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:18:45.938 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:43.694271+0000", 2024-03-20T09:18:45.938 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:45.938 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.938 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.939 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:18:45.939 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:43.694271+0000", 2024-03-20T09:18:45.939 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:45.939 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.939 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.939 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:18:45.939 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:43.694272+0000", 2024-03-20T09:18:45.939 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5149999999999999e-06 2024-03-20T09:18:45.939 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.939 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.939 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:18:45.939 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:43.694321+0000", 2024-03-20T09:18:45.940 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9259999999999999e-05 2024-03-20T09:18:45.940 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.940 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.940 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:18:45.940 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:43.694333+0000", 2024-03-20T09:18:45.940 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1867e-05 2024-03-20T09:18:45.940 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.940 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.940 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:18:45.940 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:43.694339+0000", 2024-03-20T09:18:45.940 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9900000000000002e-06 2024-03-20T09:18:45.940 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.940 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.940 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:18:45.940 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:43.694393+0000", 2024-03-20T09:18:45.940 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3745999999999997e-05 2024-03-20T09:18:45.941 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.941 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.941 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:18:45.941 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:43.694526+0000", 2024-03-20T09:18:45.941 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013332400000000001 2024-03-20T09:18:45.941 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.941 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.941 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 8,11", 2024-03-20T09:18:45.941 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:43.694686+0000", 2024-03-20T09:18:45.941 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000159877 2024-03-20T09:18:45.941 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.941 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.941 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:18:45.941 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:43.695599+0000", 2024-03-20T09:18:45.941 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00091252399999999995 2024-03-20T09:18:45.942 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.942 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.942 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:45.942 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:43.696017+0000", 2024-03-20T09:18:45.942 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00041821600000000001 2024-03-20T09:18:45.942 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.942 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.942 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:45.942 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:43.776799+0000", 2024-03-20T09:18:45.942 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.080781804999999998 2024-03-20T09:18:45.942 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.942 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.942 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:18:45.943 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:43.776858+0000", 2024-03-20T09:18:45.943 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8791e-05 2024-03-20T09:18:45.943 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.943 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.943 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:18:45.943 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:43.776908+0000", 2024-03-20T09:18:45.943 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0528999999999997e-05 2024-03-20T09:18:45.943 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:45.943 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:18:45.943 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:45.943 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.943 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.943 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.25690.0:39150 4.2 4:421621a2:::benchmark_data_smithi122_145698_object39149:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e535)", 2024-03-20T09:18:45.943 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:13:51.238859+0000", 2024-03-20T09:18:45.943 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 294.67037725099999, 2024-03-20T09:18:45.943 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.16292329999999999, 2024-03-20T09:18:45.944 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:18:45.944 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:45.944 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:18:45.944 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.25690", 2024-03-20T09:18:45.944 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:18:45.944 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 39150 2024-03-20T09:18:45.944 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.944 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:18:45.944 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.944 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:18:45.944 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:51.238859+0000", 2024-03-20T09:18:45.944 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:45.944 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.944 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.944 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:18:45.944 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:51.238859+0000", 2024-03-20T09:18:45.945 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:45.945 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.945 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.945 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:18:45.945 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:51.238860+0000", 2024-03-20T09:18:45.945 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.02e-07 2024-03-20T09:18:45.945 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.945 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.945 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:18:45.945 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:51.238928+0000", 2024-03-20T09:18:45.945 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.8524999999999999e-05 2024-03-20T09:18:45.945 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.945 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.945 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:18:45.946 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:51.238937+0000", 2024-03-20T09:18:45.946 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.8210000000000008e-06 2024-03-20T09:18:45.946 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.946 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.946 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:18:45.946 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:51.238943+0000", 2024-03-20T09:18:45.946 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4419999999999997e-06 2024-03-20T09:18:45.946 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.946 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.946 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:18:45.946 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:51.238980+0000", 2024-03-20T09:18:45.946 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7731000000000002e-05 2024-03-20T09:18:45.946 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.947 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.947 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:18:45.947 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:51.239066+0000", 2024-03-20T09:18:45.947 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5338000000000002e-05 2024-03-20T09:18:45.947 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.947 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.947 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 8,11", 2024-03-20T09:18:45.947 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:51.239149+0000", 2024-03-20T09:18:45.947 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.3255000000000002e-05 2024-03-20T09:18:45.947 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.947 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.947 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:18:45.947 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:51.240188+0000", 2024-03-20T09:18:45.948 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0010385730000000001 2024-03-20T09:18:45.948 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.948 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.948 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:45.948 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:51.240534+0000", 2024-03-20T09:18:45.948 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00034686 2024-03-20T09:18:45.948 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.948 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.948 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:45.948 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:51.401661+0000", 2024-03-20T09:18:45.948 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.161126299 2024-03-20T09:18:45.948 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.948 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.948 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:18:45.948 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:51.401727+0000", 2024-03-20T09:18:45.948 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.6493000000000002e-05 2024-03-20T09:18:45.949 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.949 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.949 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:18:45.949 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:51.401782+0000", 2024-03-20T09:18:45.949 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5061e-05 2024-03-20T09:18:45.949 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:45.949 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:18:45.949 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:45.949 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.949 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.949 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.25690.0:39158 4.2 4:411283cb:::benchmark_data_smithi122_145698_object39157:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e535)", 2024-03-20T09:18:45.949 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:13:51.246242+0000", 2024-03-20T09:18:45.949 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 294.66299394399999, 2024-03-20T09:18:45.949 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.156398223, 2024-03-20T09:18:45.949 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:18:45.950 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:45.950 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:18:45.950 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.25690", 2024-03-20T09:18:45.950 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:18:45.950 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 39158 2024-03-20T09:18:45.950 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.950 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:18:45.950 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.950 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:18:45.950 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:51.246242+0000", 2024-03-20T09:18:45.950 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:45.950 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.950 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.950 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:18:45.951 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:51.246242+0000", 2024-03-20T09:18:45.951 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:45.951 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.951 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.951 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:18:45.951 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:51.246243+0000", 2024-03-20T09:18:45.951 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.2600000000000001e-07 2024-03-20T09:18:45.951 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.951 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.951 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:18:45.951 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:51.246303+0000", 2024-03-20T09:18:45.951 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9348000000000002e-05 2024-03-20T09:18:45.951 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.951 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.951 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:18:45.952 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:51.246314+0000", 2024-03-20T09:18:45.952 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1024e-05 2024-03-20T09:18:45.952 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.952 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.952 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:18:45.952 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:51.246319+0000", 2024-03-20T09:18:45.952 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5709999999999997e-06 2024-03-20T09:18:45.952 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.952 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.952 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:18:45.952 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:51.246361+0000", 2024-03-20T09:18:45.952 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.1885999999999998e-05 2024-03-20T09:18:45.952 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.952 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.952 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:18:45.952 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:51.246456+0000", 2024-03-20T09:18:45.953 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.5161999999999995e-05 2024-03-20T09:18:45.953 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.953 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.953 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 8,11", 2024-03-20T09:18:45.953 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:51.246550+0000", 2024-03-20T09:18:45.953 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.3325999999999994e-05 2024-03-20T09:18:45.953 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.953 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.953 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:45.953 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:51.247622+0000", 2024-03-20T09:18:45.953 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0010724700000000001 2024-03-20T09:18:45.953 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.953 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.953 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:18:45.954 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:51.247631+0000", 2024-03-20T09:18:45.954 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.7029999999999995e-06 2024-03-20T09:18:45.954 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.954 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.954 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:45.954 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:51.402564+0000", 2024-03-20T09:18:45.954 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.15493327100000001 2024-03-20T09:18:45.954 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.954 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.954 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:18:45.954 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:51.402600+0000", 2024-03-20T09:18:45.954 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6359000000000002e-05 2024-03-20T09:18:45.954 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.955 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.955 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:18:45.955 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:51.402641+0000", 2024-03-20T09:18:45.955 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.0176999999999998e-05 2024-03-20T09:18:45.955 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:45.955 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:18:45.955 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:45.955 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.955 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.955 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.25690.0:43783 4.2 4:4defcf21:::benchmark_data_smithi122_145698_object43782:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e536)", 2024-03-20T09:18:45.955 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:13:53.934519+0000", 2024-03-20T09:18:45.955 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 291.97471700099999, 2024-03-20T09:18:45.955 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.174223394, 2024-03-20T09:18:45.955 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:18:45.955 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:45.955 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:18:45.956 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.25690", 2024-03-20T09:18:45.956 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:18:45.956 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 43783 2024-03-20T09:18:45.956 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.956 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:18:45.956 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.956 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:18:45.956 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:53.934519+0000", 2024-03-20T09:18:45.956 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:45.956 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.956 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.956 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:18:45.956 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:53.934519+0000", 2024-03-20T09:18:45.956 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:45.956 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.956 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.957 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:18:45.957 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:53.934520+0000", 2024-03-20T09:18:45.957 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1170000000000001e-06 2024-03-20T09:18:45.957 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.957 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.957 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:18:45.957 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:53.934549+0000", 2024-03-20T09:18:45.957 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8793999999999999e-05 2024-03-20T09:18:45.957 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.957 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.957 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:18:45.957 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:53.934561+0000", 2024-03-20T09:18:45.957 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1654000000000001e-05 2024-03-20T09:18:45.957 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.957 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.958 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:18:45.958 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:53.934566+0000", 2024-03-20T09:18:45.958 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3870000000000003e-06 2024-03-20T09:18:45.958 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.958 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.958 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:18:45.958 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:53.934653+0000", 2024-03-20T09:18:45.958 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.7293999999999995e-05 2024-03-20T09:18:45.958 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.958 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.958 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:18:45.958 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:53.934802+0000", 2024-03-20T09:18:45.958 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00014854700000000001 2024-03-20T09:18:45.958 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.959 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.959 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 8,11", 2024-03-20T09:18:45.959 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:53.934956+0000", 2024-03-20T09:18:45.959 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00015373700000000001 2024-03-20T09:18:45.959 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.959 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.959 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:18:45.959 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:53.936357+0000", 2024-03-20T09:18:45.959 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001401717 2024-03-20T09:18:45.959 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.959 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.959 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:45.959 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:53.936731+0000", 2024-03-20T09:18:45.959 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00037317399999999999 2024-03-20T09:18:45.960 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.960 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.960 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:45.960 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:54.108632+0000", 2024-03-20T09:18:45.960 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.17190133399999999 2024-03-20T09:18:45.960 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.960 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.960 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:18:45.960 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:54.108692+0000", 2024-03-20T09:18:45.960 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0158000000000002e-05 2024-03-20T09:18:45.960 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.960 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.960 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:18:45.960 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:13:54.108743+0000", 2024-03-20T09:18:45.960 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0481e-05 2024-03-20T09:18:45.960 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:45.960 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:18:45.961 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:45.961 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.961 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.961 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25690.0:111820 4.b e550/520 4:dbadf309:::benchmark_data_smithi122_145698_object111819:head v 550'6868, mlcod=550'6866)", 2024-03-20T09:18:45.961 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:14:46.866373+0000", 2024-03-20T09:18:45.961 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 239.04286344100001, 2024-03-20T09:18:45.961 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.067778783999999995, 2024-03-20T09:18:45.961 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:18:45.961 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:45.961 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:18:45.961 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.961 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:18:45.961 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.866373+0000", 2024-03-20T09:18:45.961 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:45.961 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.961 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.962 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:18:45.962 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.866373+0000", 2024-03-20T09:18:45.962 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:45.962 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.962 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.962 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:18:45.962 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.866374+0000", 2024-03-20T09:18:45.962 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.4300000000000002e-07 2024-03-20T09:18:45.962 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.962 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.962 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:18:45.962 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.866419+0000", 2024-03-20T09:18:45.962 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5445999999999998e-05 2024-03-20T09:18:45.962 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.963 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.963 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:18:45.963 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.866429+0000", 2024-03-20T09:18:45.963 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0419e-05 2024-03-20T09:18:45.963 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.963 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.963 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:18:45.963 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.866436+0000", 2024-03-20T09:18:45.963 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.8290000000000002e-06 2024-03-20T09:18:45.963 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.963 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.963 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:18:45.963 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.866480+0000", 2024-03-20T09:18:45.963 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.3995000000000001e-05 2024-03-20T09:18:45.963 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.963 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.964 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:18:45.964 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.866493+0000", 2024-03-20T09:18:45.964 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3179e-05 2024-03-20T09:18:45.964 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.964 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.964 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:18:45.964 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.934098+0000", 2024-03-20T09:18:45.964 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.067604832000000004 2024-03-20T09:18:45.964 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.964 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.964 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:18:45.964 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.934152+0000", 2024-03-20T09:18:45.964 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3341000000000003e-05 2024-03-20T09:18:45.964 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:45.965 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:18:45.965 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:45.965 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.965 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.965 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.25690.0:111826 4.2 4:4b348dcc:::benchmark_data_smithi122_145698_object111825:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e550)", 2024-03-20T09:18:45.965 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:14:46.867441+0000", 2024-03-20T09:18:45.965 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 239.04179493000001, 2024-03-20T09:18:45.965 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.088850539000000006, 2024-03-20T09:18:45.965 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:18:45.965 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:45.965 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:18:45.965 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.25690", 2024-03-20T09:18:45.965 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:18:45.965 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 111826 2024-03-20T09:18:45.966 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.966 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:18:45.966 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.966 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:18:45.966 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.867441+0000", 2024-03-20T09:18:45.966 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:45.966 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.966 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.966 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:18:45.966 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.867441+0000", 2024-03-20T09:18:45.966 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:45.966 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.967 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.967 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:18:45.967 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.867443+0000", 2024-03-20T09:18:45.967 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3939999999999999e-06 2024-03-20T09:18:45.967 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.967 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.967 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:18:45.967 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.867499+0000", 2024-03-20T09:18:45.967 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6433e-05 2024-03-20T09:18:45.967 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.967 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.967 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:18:45.967 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.867512+0000", 2024-03-20T09:18:45.967 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2578999999999999e-05 2024-03-20T09:18:45.968 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.968 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.968 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:18:45.968 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.867519+0000", 2024-03-20T09:18:45.968 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.4120000000000004e-06 2024-03-20T09:18:45.968 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.968 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.968 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:18:45.968 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.938949+0000", 2024-03-20T09:18:45.968 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.071430265000000007 2024-03-20T09:18:45.968 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.968 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.968 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:18:45.969 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.939100+0000", 2024-03-20T09:18:45.969 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00015061299999999999 2024-03-20T09:18:45.969 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.969 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.969 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 8,11", 2024-03-20T09:18:45.969 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.939204+0000", 2024-03-20T09:18:45.969 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000104424 2024-03-20T09:18:45.969 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.969 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.969 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:45.969 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.949416+0000", 2024-03-20T09:18:45.969 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.010211932 2024-03-20T09:18:45.969 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.970 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.970 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:18:45.970 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.949432+0000", 2024-03-20T09:18:45.970 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5542000000000001e-05 2024-03-20T09:18:45.970 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.970 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.970 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:45.970 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.956153+0000", 2024-03-20T09:18:45.970 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0067214340000000001 2024-03-20T09:18:45.970 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.970 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.970 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:18:45.971 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.956233+0000", 2024-03-20T09:18:45.971 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.0084000000000001e-05 2024-03-20T09:18:45.971 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.971 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.971 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:18:45.971 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.956292+0000", 2024-03-20T09:18:45.971 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8427000000000002e-05 2024-03-20T09:18:45.971 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:45.971 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:18:45.971 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:45.971 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.971 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.971 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.25690.0:111828 4.2 4:4e6842c7:::benchmark_data_smithi122_145698_object111827:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e550)", 2024-03-20T09:18:45.972 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:14:46.868836+0000", 2024-03-20T09:18:45.972 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 239.04040053200001, 2024-03-20T09:18:45.972 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.087648169999999997, 2024-03-20T09:18:45.972 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:18:45.972 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:45.972 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:18:45.972 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.25690", 2024-03-20T09:18:45.972 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:18:45.972 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 111828 2024-03-20T09:18:45.972 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.972 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:18:45.972 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.973 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:18:45.973 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.868836+0000", 2024-03-20T09:18:45.973 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:45.973 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.973 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.973 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:18:45.973 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.868836+0000", 2024-03-20T09:18:45.973 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:45.973 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.973 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.973 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:18:45.973 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.868837+0000", 2024-03-20T09:18:45.973 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.0100000000000004e-07 2024-03-20T09:18:45.974 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.974 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.974 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:18:45.974 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.868904+0000", 2024-03-20T09:18:45.974 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.7708999999999998e-05 2024-03-20T09:18:45.974 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.974 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.974 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:18:45.974 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.868913+0000", 2024-03-20T09:18:45.974 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.4989999999999994e-06 2024-03-20T09:18:45.974 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.974 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.974 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:18:45.975 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.868918+0000", 2024-03-20T09:18:45.975 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0669999999999999e-06 2024-03-20T09:18:45.975 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.975 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.975 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:18:45.975 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.942896+0000", 2024-03-20T09:18:45.975 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.073978349999999998 2024-03-20T09:18:45.975 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.975 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.975 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:18:45.975 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.943011+0000", 2024-03-20T09:18:45.975 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000114768 2024-03-20T09:18:45.975 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.975 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.975 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 8,11", 2024-03-20T09:18:45.975 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.943143+0000", 2024-03-20T09:18:45.976 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000131827 2024-03-20T09:18:45.976 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.976 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.976 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:45.976 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.949461+0000", 2024-03-20T09:18:45.976 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0063179789999999996 2024-03-20T09:18:45.976 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.976 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.976 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:18:45.976 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.952939+0000", 2024-03-20T09:18:45.976 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.003478362 2024-03-20T09:18:45.976 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.976 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.976 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:45.976 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.956367+0000", 2024-03-20T09:18:45.977 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0034275159999999998 2024-03-20T09:18:45.977 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.977 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.977 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:18:45.977 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.956425+0000", 2024-03-20T09:18:45.977 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8251999999999997e-05 2024-03-20T09:18:45.977 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.977 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.977 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:18:45.977 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.956484+0000", 2024-03-20T09:18:45.977 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9039999999999997e-05 2024-03-20T09:18:45.977 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:45.977 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:18:45.977 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:45.977 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.977 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.978 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25690.0:111824 4.b e550/520 4:d74810e1:::benchmark_data_smithi122_145698_object111823:head v 550'6870, mlcod=550'6866)", 2024-03-20T09:18:45.978 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:14:46.870375+0000", 2024-03-20T09:18:45.978 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 239.03886144200001, 2024-03-20T09:18:45.978 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.073606780999999996, 2024-03-20T09:18:45.978 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:18:45.978 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:45.978 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:18:45.978 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.978 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:18:45.978 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.870375+0000", 2024-03-20T09:18:45.978 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:45.978 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.978 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.978 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:18:45.978 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.870375+0000", 2024-03-20T09:18:45.979 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:45.979 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.979 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.979 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:18:45.979 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.870375+0000", 2024-03-20T09:18:45.979 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.06e-07 2024-03-20T09:18:45.979 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.979 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.979 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:18:45.979 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.870425+0000", 2024-03-20T09:18:45.979 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9672999999999997e-05 2024-03-20T09:18:45.979 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.979 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.979 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:18:45.979 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.934360+0000", 2024-03-20T09:18:45.979 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.063935211000000006 2024-03-20T09:18:45.980 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.980 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.980 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:18:45.980 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.934370+0000", 2024-03-20T09:18:45.980 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0079e-05 2024-03-20T09:18:45.980 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.980 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.980 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:18:45.980 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.934506+0000", 2024-03-20T09:18:45.980 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013579799999999999 2024-03-20T09:18:45.980 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.980 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.980 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:18:45.980 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.934526+0000", 2024-03-20T09:18:45.980 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9457e-05 2024-03-20T09:18:45.980 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.981 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.981 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:18:45.981 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.943919+0000", 2024-03-20T09:18:45.981 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0093935089999999995 2024-03-20T09:18:45.981 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.981 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.981 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:18:45.981 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.943982+0000", 2024-03-20T09:18:45.981 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2447999999999996e-05 2024-03-20T09:18:45.981 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:45.981 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:18:45.981 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:45.981 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.981 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.981 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25690.0:111830 4.1 e550/520 4:84a72058:::benchmark_data_smithi122_145698_object111829:head v 550'6948, mlcod=550'6946)", 2024-03-20T09:18:45.981 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:14:46.876564+0000", 2024-03-20T09:18:45.982 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 239.03267250499999, 2024-03-20T09:18:45.982 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.068788345000000001, 2024-03-20T09:18:45.982 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:18:45.982 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:45.982 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:18:45.982 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.982 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:18:45.982 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.876564+0000", 2024-03-20T09:18:45.982 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:45.982 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.982 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.982 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:18:45.982 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.876564+0000", 2024-03-20T09:18:45.982 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:45.982 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.983 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.983 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:18:45.983 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.876565+0000", 2024-03-20T09:18:45.983 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.2699999999999998e-07 2024-03-20T09:18:45.983 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.983 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.983 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:18:45.983 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.876624+0000", 2024-03-20T09:18:45.983 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9005000000000002e-05 2024-03-20T09:18:45.983 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.983 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.983 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:18:45.983 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.934324+0000", 2024-03-20T09:18:45.983 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.057700728999999999 2024-03-20T09:18:45.983 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.983 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.984 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:18:45.984 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.934332+0000", 2024-03-20T09:18:45.984 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.8699999999999992e-06 2024-03-20T09:18:45.984 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.984 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.984 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:18:45.984 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.934609+0000", 2024-03-20T09:18:45.984 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00027687399999999998 2024-03-20T09:18:45.984 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.984 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.984 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:18:45.984 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.934629+0000", 2024-03-20T09:18:45.984 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9882e-05 2024-03-20T09:18:45.984 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.984 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.985 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:18:45.985 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.945303+0000", 2024-03-20T09:18:45.985 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.010674395 2024-03-20T09:18:45.985 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.985 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.985 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:18:45.985 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.945352+0000", 2024-03-20T09:18:45.985 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8662999999999998e-05 2024-03-20T09:18:45.985 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:45.985 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:18:45.985 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:45.985 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.985 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.985 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.25690.0:111833 4.2 4:4f20c26a:::benchmark_data_smithi122_145698_object111832:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e550)", 2024-03-20T09:18:45.985 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:14:46.880152+0000", 2024-03-20T09:18:45.985 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 239.029084028, 2024-03-20T09:18:45.986 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.076576116999999999, 2024-03-20T09:18:45.986 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:18:45.986 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:45.986 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:18:45.986 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.25690", 2024-03-20T09:18:45.986 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:18:45.986 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 111833 2024-03-20T09:18:45.986 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.986 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:18:45.986 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.986 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:18:45.986 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.880152+0000", 2024-03-20T09:18:45.986 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:45.986 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.986 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.987 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:18:45.987 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.880152+0000", 2024-03-20T09:18:45.987 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:45.987 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.987 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.987 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:18:45.987 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.880154+0000", 2024-03-20T09:18:45.987 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.486e-06 2024-03-20T09:18:45.987 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.987 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.987 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:18:45.987 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.880342+0000", 2024-03-20T09:18:45.987 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00018790199999999999 2024-03-20T09:18:45.987 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.987 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.988 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:18:45.988 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.934214+0000", 2024-03-20T09:18:45.988 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.053872241000000001 2024-03-20T09:18:45.988 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.988 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.988 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:18:45.988 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.934226+0000", 2024-03-20T09:18:45.988 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2598e-05 2024-03-20T09:18:45.988 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.988 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.988 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:18:45.988 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.946089+0000", 2024-03-20T09:18:45.988 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.011862167 2024-03-20T09:18:45.988 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.988 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.989 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:18:45.989 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.946239+0000", 2024-03-20T09:18:45.989 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00015076299999999999 2024-03-20T09:18:45.989 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.989 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.989 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 8,11", 2024-03-20T09:18:45.989 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.946385+0000", 2024-03-20T09:18:45.989 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00014608899999999999 2024-03-20T09:18:45.989 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.989 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.989 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:18:45.989 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.956309+0000", 2024-03-20T09:18:45.989 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.009923879 2024-03-20T09:18:45.989 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.989 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.990 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:45.990 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.956543+0000", 2024-03-20T09:18:45.990 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00023369499999999999 2024-03-20T09:18:45.990 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.990 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.990 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:45.990 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.956615+0000", 2024-03-20T09:18:45.990 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.1799999999999997e-05 2024-03-20T09:18:45.990 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.990 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.990 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:18:45.990 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.956676+0000", 2024-03-20T09:18:45.990 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1199999999999997e-05 2024-03-20T09:18:45.990 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.990 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.990 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:18:45.991 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:14:46.956728+0000", 2024-03-20T09:18:45.991 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2296999999999998e-05 2024-03-20T09:18:45.991 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:45.991 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:18:45.991 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:45.991 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.991 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.991 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.25966.0:25887 5.f 5:f7c302b7:::benchmark_data_smithi122_157894_object25886:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e614)", 2024-03-20T09:18:45.991 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:18:30.064057+0000", 2024-03-20T09:18:45.991 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 15.845179503000001, 2024-03-20T09:18:45.991 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.094367248000000001, 2024-03-20T09:18:45.991 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:18:45.991 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:45.991 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:18:45.991 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.25966", 2024-03-20T09:18:45.991 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:18:45.992 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 25887 2024-03-20T09:18:45.992 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.992 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:18:45.992 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.992 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:18:45.992 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.064057+0000", 2024-03-20T09:18:45.992 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:45.992 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.992 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.992 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:18:45.992 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.064057+0000", 2024-03-20T09:18:45.992 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:45.992 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.992 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.992 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:18:45.993 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.064058+0000", 2024-03-20T09:18:45.993 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0729999999999999e-06 2024-03-20T09:18:45.993 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.993 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.993 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:18:45.993 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.064115+0000", 2024-03-20T09:18:45.993 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6943e-05 2024-03-20T09:18:45.993 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.993 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.993 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:18:45.993 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.064126+0000", 2024-03-20T09:18:45.993 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1112e-05 2024-03-20T09:18:45.993 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.994 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.994 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:18:45.994 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.064133+0000", 2024-03-20T09:18:45.994 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.5660000000000001e-06 2024-03-20T09:18:45.994 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.994 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.994 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:18:45.994 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.064203+0000", 2024-03-20T09:18:45.994 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.9837000000000002e-05 2024-03-20T09:18:45.994 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.994 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.994 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:18:45.995 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.064321+0000", 2024-03-20T09:18:45.995 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000117528 2024-03-20T09:18:45.995 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.995 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.995 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,11", 2024-03-20T09:18:45.995 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.064463+0000", 2024-03-20T09:18:45.995 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00014268199999999999 2024-03-20T09:18:45.995 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.995 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.995 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:18:45.995 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.065593+0000", 2024-03-20T09:18:45.995 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001129148 2024-03-20T09:18:45.996 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.996 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.996 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:45.996 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.066813+0000", 2024-03-20T09:18:45.996 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001220669 2024-03-20T09:18:45.996 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.996 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.996 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:45.996 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.158317+0000", 2024-03-20T09:18:45.996 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.091503765000000001 2024-03-20T09:18:45.996 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.996 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.996 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:18:45.996 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.158382+0000", 2024-03-20T09:18:45.996 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4783000000000006e-05 2024-03-20T09:18:45.997 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.997 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.997 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:18:45.997 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.158424+0000", 2024-03-20T09:18:45.997 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.2141999999999997e-05 2024-03-20T09:18:45.997 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:45.997 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:18:45.997 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:45.997 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.997 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.997 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.25966.0:25897 5.f 5:f3d972d2:::benchmark_data_smithi122_157894_object25896:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e614)", 2024-03-20T09:18:45.997 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:18:30.068486+0000", 2024-03-20T09:18:45.997 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 15.840749961, 2024-03-20T09:18:45.997 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.090079832999999998, 2024-03-20T09:18:45.997 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:18:45.998 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:45.998 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:18:45.998 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.25966", 2024-03-20T09:18:45.998 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:18:45.998 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 25897 2024-03-20T09:18:45.998 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.998 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:18:45.998 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.998 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:18:45.998 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.068486+0000", 2024-03-20T09:18:45.998 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:45.998 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.998 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.998 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:18:45.998 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.068486+0000", 2024-03-20T09:18:45.999 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:45.999 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.999 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.999 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:18:45.999 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.068488+0000", 2024-03-20T09:18:45.999 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.235e-06 2024-03-20T09:18:45.999 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.999 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.999 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:18:45.999 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.068544+0000", 2024-03-20T09:18:45.999 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6439e-05 2024-03-20T09:18:45.999 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:45.999 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:45.999 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:18:45.999 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.068559+0000", 2024-03-20T09:18:45.999 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5347000000000001e-05 2024-03-20T09:18:46.000 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.000 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.000 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:18:46.000 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.068568+0000", 2024-03-20T09:18:46.000 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.7029999999999995e-06 2024-03-20T09:18:46.000 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.000 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.000 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:18:46.000 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.068627+0000", 2024-03-20T09:18:46.000 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8628000000000002e-05 2024-03-20T09:18:46.000 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.000 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.000 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:18:46.000 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.068732+0000", 2024-03-20T09:18:46.000 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000105176 2024-03-20T09:18:46.000 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.001 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.001 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,11", 2024-03-20T09:18:46.001 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.068829+0000", 2024-03-20T09:18:46.001 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.7017999999999998e-05 2024-03-20T09:18:46.001 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.001 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.001 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:18:46.001 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.070718+0000", 2024-03-20T09:18:46.001 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00188933 2024-03-20T09:18:46.001 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.001 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.001 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:46.001 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.070884+0000", 2024-03-20T09:18:46.001 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00016575100000000001 2024-03-20T09:18:46.001 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.002 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.002 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:46.002 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.158475+0000", 2024-03-20T09:18:46.002 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.087590977 2024-03-20T09:18:46.002 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.002 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.002 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:18:46.002 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.158530+0000", 2024-03-20T09:18:46.002 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5498000000000003e-05 2024-03-20T09:18:46.002 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.002 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.002 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:18:46.002 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.158566+0000", 2024-03-20T09:18:46.002 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.5731000000000001e-05 2024-03-20T09:18:46.003 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:46.003 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:18:46.003 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:46.003 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.003 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.003 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.25966.0:25903 5.f 5:f3e92c11:::benchmark_data_smithi122_157894_object25902:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e614)", 2024-03-20T09:18:46.003 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:18:30.070997+0000", 2024-03-20T09:18:46.003 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 15.838239142999999, 2024-03-20T09:18:46.003 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.087695511000000004, 2024-03-20T09:18:46.003 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:18:46.003 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:46.003 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:18:46.003 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.25966", 2024-03-20T09:18:46.004 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:18:46.004 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 25903 2024-03-20T09:18:46.004 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.004 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:18:46.004 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.004 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:18:46.004 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.070997+0000", 2024-03-20T09:18:46.004 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:46.004 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.004 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.004 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:18:46.004 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.070997+0000", 2024-03-20T09:18:46.004 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:46.004 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.004 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.004 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:18:46.005 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.070998+0000", 2024-03-20T09:18:46.005 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.9299999999999996e-07 2024-03-20T09:18:46.005 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.005 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.005 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:18:46.005 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.071017+0000", 2024-03-20T09:18:46.005 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9043e-05 2024-03-20T09:18:46.005 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.005 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.005 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:18:46.005 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.071028+0000", 2024-03-20T09:18:46.005 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0643e-05 2024-03-20T09:18:46.005 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.005 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.005 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:18:46.006 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.071033+0000", 2024-03-20T09:18:46.006 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5400000000000003e-06 2024-03-20T09:18:46.006 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.006 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.006 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:18:46.006 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.071080+0000", 2024-03-20T09:18:46.006 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6887e-05 2024-03-20T09:18:46.006 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.006 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.006 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:18:46.006 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.071213+0000", 2024-03-20T09:18:46.006 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013303800000000001 2024-03-20T09:18:46.006 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.006 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.007 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,11", 2024-03-20T09:18:46.007 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.071355+0000", 2024-03-20T09:18:46.007 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000142161 2024-03-20T09:18:46.007 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.007 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.007 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:46.007 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.073158+0000", 2024-03-20T09:18:46.007 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0018030870000000001 2024-03-20T09:18:46.007 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.007 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.007 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:18:46.007 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.073264+0000", 2024-03-20T09:18:46.007 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000105967 2024-03-20T09:18:46.007 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.008 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.008 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:46.008 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.158611+0000", 2024-03-20T09:18:46.008 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.085346303999999998 2024-03-20T09:18:46.008 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.008 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.008 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:18:46.008 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.158657+0000", 2024-03-20T09:18:46.008 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6303e-05 2024-03-20T09:18:46.008 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.008 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.008 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:18:46.008 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.158693+0000", 2024-03-20T09:18:46.008 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.5645000000000002e-05 2024-03-20T09:18:46.008 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:46.009 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:18:46.009 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:46.009 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.009 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.009 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.25966.0:25913 5.f 5:fef1fa3b:::benchmark_data_smithi122_157894_object25912:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e614)", 2024-03-20T09:18:46.009 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:18:30.079706+0000", 2024-03-20T09:18:46.009 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 15.829530571999999, 2024-03-20T09:18:46.009 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.079084529000000001, 2024-03-20T09:18:46.009 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:18:46.009 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:46.009 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:18:46.009 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.25966", 2024-03-20T09:18:46.009 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:18:46.009 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 25913 2024-03-20T09:18:46.009 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.010 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:18:46.010 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.010 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:18:46.010 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.079706+0000", 2024-03-20T09:18:46.010 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:46.010 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.010 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.010 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:18:46.010 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.079706+0000", 2024-03-20T09:18:46.010 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:46.010 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.010 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.010 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:18:46.011 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.079707+0000", 2024-03-20T09:18:46.011 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1379999999999999e-06 2024-03-20T09:18:46.011 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.011 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.011 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:18:46.011 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.079752+0000", 2024-03-20T09:18:46.011 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5175e-05 2024-03-20T09:18:46.011 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.011 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.011 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:18:46.011 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.079763+0000", 2024-03-20T09:18:46.011 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0539e-05 2024-03-20T09:18:46.011 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.011 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.011 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:18:46.012 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.079768+0000", 2024-03-20T09:18:46.012 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6550000000000001e-06 2024-03-20T09:18:46.012 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.012 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.012 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:18:46.012 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.079810+0000", 2024-03-20T09:18:46.012 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.1921000000000001e-05 2024-03-20T09:18:46.012 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.012 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.012 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:18:46.012 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.079901+0000", 2024-03-20T09:18:46.012 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.0836999999999998e-05 2024-03-20T09:18:46.012 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.012 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.012 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,11", 2024-03-20T09:18:46.012 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.079990+0000", 2024-03-20T09:18:46.012 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.8739000000000005e-05 2024-03-20T09:18:46.013 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.013 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.013 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:18:46.013 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.080947+0000", 2024-03-20T09:18:46.013 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00095694300000000003 2024-03-20T09:18:46.013 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.013 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.013 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:46.013 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.081254+0000", 2024-03-20T09:18:46.013 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00030785900000000001 2024-03-20T09:18:46.013 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.013 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.013 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:46.013 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.158733+0000", 2024-03-20T09:18:46.013 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.077478456000000001 2024-03-20T09:18:46.013 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.014 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.014 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:18:46.014 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.158771+0000", 2024-03-20T09:18:46.014 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.8226999999999998e-05 2024-03-20T09:18:46.014 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.014 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.014 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:18:46.014 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.158790+0000", 2024-03-20T09:18:46.014 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9040000000000001e-05 2024-03-20T09:18:46.014 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:46.014 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:18:46.014 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:46.015 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.015 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.015 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.25966.0:25916 5.f 5:f626f375:::benchmark_data_smithi122_157894_object25915:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e614)", 2024-03-20T09:18:46.015 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:18:30.083220+0000", 2024-03-20T09:18:46.015 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 15.826016568, 2024-03-20T09:18:46.015 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.075637637999999993, 2024-03-20T09:18:46.015 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:18:46.015 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:18:46.015 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:18:46.015 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.25966", 2024-03-20T09:18:46.015 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:18:46.015 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 25916 2024-03-20T09:18:46.015 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.015 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:18:46.017 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.017 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:18:46.017 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.083220+0000", 2024-03-20T09:18:46.017 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:46.018 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.018 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.018 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:18:46.018 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.083220+0000", 2024-03-20T09:18:46.018 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:18:46.018 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.018 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.018 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:18:46.018 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.083220+0000", 2024-03-20T09:18:46.018 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.8999999999999996e-07 2024-03-20T09:18:46.018 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.018 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.018 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:18:46.018 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.083274+0000", 2024-03-20T09:18:46.018 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3949000000000001e-05 2024-03-20T09:18:46.019 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.019 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.019 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:18:46.019 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.083284+0000", 2024-03-20T09:18:46.019 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.013e-05 2024-03-20T09:18:46.019 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.019 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.019 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:18:46.019 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.083290+0000", 2024-03-20T09:18:46.019 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4e-06 2024-03-20T09:18:46.019 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.019 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.019 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:18:46.019 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.083338+0000", 2024-03-20T09:18:46.020 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8032999999999999e-05 2024-03-20T09:18:46.020 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.020 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.020 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:18:46.020 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.083459+0000", 2024-03-20T09:18:46.020 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012145400000000001 2024-03-20T09:18:46.020 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.020 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.020 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,11", 2024-03-20T09:18:46.020 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.083601+0000", 2024-03-20T09:18:46.020 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000142027 2024-03-20T09:18:46.020 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.020 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.020 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:18:46.021 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.084214+0000", 2024-03-20T09:18:46.021 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00061220299999999997 2024-03-20T09:18:46.021 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.021 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.021 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:46.021 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.084584+0000", 2024-03-20T09:18:46.021 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00037049199999999999 2024-03-20T09:18:46.021 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.021 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.021 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:46.021 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.158814+0000", 2024-03-20T09:18:46.021 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.074229798999999999 2024-03-20T09:18:46.021 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.022 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.022 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:18:46.022 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.158839+0000", 2024-03-20T09:18:46.022 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5247000000000001e-05 2024-03-20T09:18:46.022 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:18:46.022 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:18:46.022 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:18:46.022 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:30.158857+0000", 2024-03-20T09:18:46.022 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8213999999999999e-05 2024-03-20T09:18:46.022 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:46.022 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:18:46.022 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:46.022 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:18:46.022 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:18:46.022 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:18:46.023 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:45 smithi122 ceph-mon[73495]: 1.a scrub starts 2024-03-20T09:18:46.023 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:45 smithi122 ceph-mon[73495]: 1.a scrub ok 2024-03-20T09:18:46.023 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:45 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3670607790' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:18:46.024 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:18:46.024 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:18:46.024 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:18:46.024 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:18:46.024 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:18:46.024 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:18:46.024 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:18:46.024 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:18:46.024 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:18:46.024 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:18:46.024 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:18:46.024 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:18:46.025 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:18:46.025 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:45 smithi026 ceph-mon[68309]: 1.a scrub starts 2024-03-20T09:18:46.025 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:45 smithi026 ceph-mon[68309]: 1.a scrub ok 2024-03-20T09:18:46.025 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:45 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3670607790' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:18:46.623 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 26 16 58055 58039 139.502 100.562 0.00625539 0.00716316 2024-03-20T09:18:46.660 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-20T09:18:46.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:46 smithi115 ceph-mon[72476]: 1.3 scrub starts 2024-03-20T09:18:46.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:46 smithi115 ceph-mon[72476]: 1.3 scrub ok 2024-03-20T09:18:46.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:46 smithi115 ceph-mon[72476]: pgmap v1484: 47 pgs: 47 active+clean; 3.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 103 MiB/s wr, 1.64k op/s 2024-03-20T09:18:46.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:46 smithi122 ceph-mon[73495]: 1.3 scrub starts 2024-03-20T09:18:46.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:46 smithi122 ceph-mon[73495]: 1.3 scrub ok 2024-03-20T09:18:46.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:46 smithi122 ceph-mon[73495]: pgmap v1484: 47 pgs: 47 active+clean; 3.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 103 MiB/s wr, 1.64k op/s 2024-03-20T09:18:46.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:46 smithi026 ceph-mon[68309]: 1.3 scrub starts 2024-03-20T09:18:46.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:46 smithi026 ceph-mon[68309]: 1.3 scrub ok 2024-03-20T09:18:46.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:46 smithi026 ceph-mon[68309]: pgmap v1484: 47 pgs: 47 active+clean; 3.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 103 MiB/s wr, 1.64k op/s 2024-03-20T09:18:47.623 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 27 15 59505 59490 137.693 90.6875 0.00668313 0.00725837 2024-03-20T09:18:47.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:47 smithi115 ceph-mon[72476]: 1.15 scrub starts 2024-03-20T09:18:47.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:47 smithi115 ceph-mon[72476]: 1.15 scrub ok 2024-03-20T09:18:47.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:47 smithi115 ceph-mon[72476]: osdmap e620: 12 total, 12 up, 10 in 2024-03-20T09:18:47.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:47 smithi122 ceph-mon[73495]: 1.15 scrub starts 2024-03-20T09:18:47.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:47 smithi122 ceph-mon[73495]: 1.15 scrub ok 2024-03-20T09:18:47.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:47 smithi122 ceph-mon[73495]: osdmap e620: 12 total, 12 up, 10 in 2024-03-20T09:18:47.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:47 smithi026 ceph-mon[68309]: 1.15 scrub starts 2024-03-20T09:18:47.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:47 smithi026 ceph-mon[68309]: 1.15 scrub ok 2024-03-20T09:18:47.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:47 smithi026 ceph-mon[68309]: osdmap e620: 12 total, 12 up, 10 in 2024-03-20T09:18:48.411 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:18:48.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:48.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:18:48.518 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:18:48.552 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:48.578 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:18:48.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:48.582 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:18:48.623 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 28 16 61215 61199 136.59 106.812 0.00352848 0.00731747 2024-03-20T09:18:48.683 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:18:48.716 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:48.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:48 smithi026 ceph-mon[68309]: pgmap v1486: 47 pgs: 47 active+clean; 3.5 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 119 MiB/s wr, 1.90k op/s 2024-03-20T09:18:48.748 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:18:48.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:48.753 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:18:48.853 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:18:48.889 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:48 smithi115 ceph-mon[72476]: pgmap v1486: 47 pgs: 47 active+clean; 3.5 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 119 MiB/s wr, 1.90k op/s 2024-03-20T09:18:48.892 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:48.924 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:18:48.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:48.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:18:48.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:48 smithi122 ceph-mon[73495]: pgmap v1486: 47 pgs: 47 active+clean; 3.5 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 119 MiB/s wr, 1.90k op/s 2024-03-20T09:18:49.029 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:18:49.065 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:49.089 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:18:49.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:49.093 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:18:49.194 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:18:49.220 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:49.238 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:18:49.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:49.241 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:18:49.342 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:18:49.374 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:49.396 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:18:49.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:49.399 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:18:49.500 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:18:49.540 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:49.578 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:18:49.580 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:18:49.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:49.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:18:49.624 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 29 16 62692 62676 135.063 92.3125 0.00970444 0.00739981 2024-03-20T09:18:49.692 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:18:49.743 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:49.770 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:18:49.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:49.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:18:49.875 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:18:49.905 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:49.928 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:18:49.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:49.931 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:18:50.033 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:18:50.099 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:50.171 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:18:50.182 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:18:50.624 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 30 16 64263 64247 133.834 98.1875 0.00464709 0.00746795 2024-03-20T09:18:50.882 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:18:51.079 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:18:51.184 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:51 smithi026 ceph-mon[68309]: pgmap v1487: 47 pgs: 47 active+clean; 3.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-03-20T09:18:51.184 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:51 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/172798651' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:18:51.184 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:51 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:18:51.184 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:51 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:18:51.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:51 smithi115 ceph-mon[72476]: pgmap v1487: 47 pgs: 47 active+clean; 3.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-03-20T09:18:51.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:51 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/172798651' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:18:51.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:51 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:18:51.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:51 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:18:51.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:51 smithi122 ceph-mon[73495]: pgmap v1487: 47 pgs: 47 active+clean; 3.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-03-20T09:18:51.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:51 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/172798651' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:18:51.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:51 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:18:51.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:51 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:18:51.625 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 31 16 65744 65728 132.497 92.5625 0.0105318 0.00754303 2024-03-20T09:18:52.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:52 smithi115 ceph-mon[72476]: from='client.26002 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:18:52.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:52 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:18:52.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:52 smithi115 ceph-mon[72476]: Cluster is now healthy 2024-03-20T09:18:52.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:52 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/172798651' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:18:52.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:52 smithi115 ceph-mon[72476]: osdmap e621: 12 total, 12 up, 10 in 2024-03-20T09:18:52.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:52 smithi122 ceph-mon[73495]: from='client.26002 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:18:52.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:52 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:18:52.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:52 smithi122 ceph-mon[73495]: Cluster is now healthy 2024-03-20T09:18:52.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:52 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/172798651' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:18:52.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:52 smithi122 ceph-mon[73495]: osdmap e621: 12 total, 12 up, 10 in 2024-03-20T09:18:52.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:52 smithi026 ceph-mon[68309]: from='client.26002 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:18:52.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:52 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:18:52.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:52 smithi026 ceph-mon[68309]: Cluster is now healthy 2024-03-20T09:18:52.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:52 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/172798651' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:18:52.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:52 smithi026 ceph-mon[68309]: osdmap e621: 12 total, 12 up, 10 in 2024-03-20T09:18:52.625 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 32 16 67324 67308 131.442 98.75 0.00662706 0.00760366 2024-03-20T09:18:53.214 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:18:53.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:53.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:18:53.322 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:18:53.332 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:53 smithi026 ceph-mon[68309]: pgmap v1489: 47 pgs: 47 active+clean; 3.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 102 MiB/s wr, 1.62k op/s 2024-03-20T09:18:53.333 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:53 smithi026 ceph-mon[68309]: osdmap e622: 12 total, 12 up, 10 in 2024-03-20T09:18:53.375 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:53.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:53 smithi115 ceph-mon[72476]: pgmap v1489: 47 pgs: 47 active+clean; 3.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 102 MiB/s wr, 1.62k op/s 2024-03-20T09:18:53.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:53 smithi115 ceph-mon[72476]: osdmap e622: 12 total, 12 up, 10 in 2024-03-20T09:18:53.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:53 smithi122 ceph-mon[73495]: pgmap v1489: 47 pgs: 47 active+clean; 3.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 102 MiB/s wr, 1.62k op/s 2024-03-20T09:18:53.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:53 smithi122 ceph-mon[73495]: osdmap e622: 12 total, 12 up, 10 in 2024-03-20T09:18:53.625 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 33 16 69013 68997 130.657 105.562 0.00425783 0.00765005 2024-03-20T09:18:53.797 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":1487,"stamp":"2024-03-20T09:18:50.018547+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3817734688,"num_objects":58247,"num_object_clones":0,"num_object_copies":174741,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58245,"num_write_kb":3727680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62493,"ondisk_log_size":62493,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":15242740,"kb_used_data":11071312,"kb_used_omap":195,"kb_used_meta":4171132,"kb_avail":922208780,"statfs":{"total":959950356480,"available":944341790720,"internally_reserved":0,"allocated":11337023488,"data_stored":11333751302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":200482,"internal_metadata":4271239390},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":629604352,"num_objects":9607,"num_object_clones":0,"num_object_copies":28821,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9607,"num_write_kb":614848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.761813"},"pg_stats":[{"pgid":"1.1e","version":"20'137","reported_seq":639,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-20T09:18:43.546956+0000","last_change":"2024-03-20T09:18:05.872016+0000","last_active":"2024-03-20T09:18:43.546956+0000","last_peered":"2024-03-20T09:18:43.546956+0000","last_clean":"2024-03-20T09:18:43.546956+0000","last_became_active":"2024-03-20T09:13:20.071655+0000","last_became_peered":"2024-03-20T09:13:20.071655+0000","last_unstale":"2024-03-20T09:18:43.546956+0000","last_undegraded":"2024-03-20T09:18:43.546956+0000","last_fullsized":"2024-03-20T09:18:43.546956+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:18:05.871879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:18:05.871879+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:34.328518+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'137","reported_seq":808,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-03-20T09:18:47.269234+0000","last_change":"2024-03-20T09:17:45.497854+0000","last_active":"2024-03-20T09:18:47.269234+0000","last_peered":"2024-03-20T09:18:47.269234+0000","last_clean":"2024-03-20T09:18:47.269234+0000","last_became_active":"2024-03-20T09:07:42.013245+0000","last_became_peered":"2024-03-20T09:07:42.013245+0000","last_unstale":"2024-03-20T09:18:47.269234+0000","last_undegraded":"2024-03-20T09:18:47.269234+0000","last_fullsized":"2024-03-20T09:18:47.269234+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:45.497703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:17:45.497703+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:06.263941+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"20'137","reported_seq":1248,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-20T09:18:43.552338+0000","last_change":"2024-03-20T09:17:50.948714+0000","last_active":"2024-03-20T09:18:43.552338+0000","last_peered":"2024-03-20T09:18:43.552338+0000","last_clean":"2024-03-20T09:18:43.552338+0000","last_became_active":"2024-03-20T09:07:41.961622+0000","last_became_peered":"2024-03-20T09:07:41.961622+0000","last_unstale":"2024-03-20T09:18:43.552338+0000","last_undegraded":"2024-03-20T09:18:43.552338+0000","last_fullsized":"2024-03-20T09:18:43.552338+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:50.948579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:17:50.948579+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:07.838518+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.19","version":"20'137","reported_seq":1317,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-20T09:18:43.544532+0000","last_change":"2024-03-20T09:18:40.053646+0000","last_active":"2024-03-20T09:18:43.544532+0000","last_peered":"2024-03-20T09:18:43.544532+0000","last_clean":"2024-03-20T09:18:43.544532+0000","last_became_active":"2024-03-20T09:07:41.962680+0000","last_became_peered":"2024-03-20T09:07:41.962680+0000","last_unstale":"2024-03-20T09:18:43.544532+0000","last_undegraded":"2024-03-20T09:18:43.544532+0000","last_fullsized":"2024-03-20T09:18:43.544532+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:18:40.053497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:18:40.053497+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:20:04.818734+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"20'137","reported_seq":779,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-20T09:18:43.554504+0000","last_change":"2024-03-20T09:18:02.925001+0000","last_active":"2024-03-20T09:18:43.554504+0000","last_peered":"2024-03-20T09:18:43.554504+0000","last_clean":"2024-03-20T09:18:43.554504+0000","last_became_active":"2024-03-20T09:10:35.277481+0000","last_became_peered":"2024-03-20T09:10:35.277481+0000","last_unstale":"2024-03-20T09:18:43.554504+0000","last_undegraded":"2024-03-20T09:18:43.554504+0000","last_fullsized":"2024-03-20T09:18:43.554504+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:18:02.924847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:18:02.924847+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:10.438683+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1b","version":"20'137","reported_seq":1085,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-20T09:18:43.552213+0000","last_change":"2024-03-20T09:18:39.119126+0000","last_active":"2024-03-20T09:18:43.552213+0000","last_peered":"2024-03-20T09:18:43.552213+0000","last_clean":"2024-03-20T09:18:43.552213+0000","last_became_active":"2024-03-20T09:07:41.962733+0000","last_became_peered":"2024-03-20T09:07:41.962733+0000","last_unstale":"2024-03-20T09:18:43.552213+0000","last_undegraded":"2024-03-20T09:18:43.552213+0000","last_fullsized":"2024-03-20T09:18:43.552213+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:18:39.119006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:18:39.119006+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:43.420972+0000","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.c","version":"620'3726","reported_seq":3766,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-03-20T09:18:47.561866+0000","last_change":"2024-03-20T09:18:11.083767+0000","last_active":"2024-03-20T09:18:47.561866+0000","last_peered":"2024-03-20T09:18:47.561866+0000","last_clean":"2024-03-20T09:18:47.561866+0000","last_became_active":"2024-03-20T09:18:11.083335+0000","last_became_peered":"2024-03-20T09:18:11.083335+0000","last_unstale":"2024-03-20T09:18:47.561866+0000","last_undegraded":"2024-03-20T09:18:47.561866+0000","last_fullsized":"2024-03-20T09:18:47.561866+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":3726,"log_dups_size":0,"ondisk_log_size":3726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:14.815713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244187136,"num_objects":3726,"num_object_clones":0,"num_object_copies":11178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3726,"num_write_kb":238464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"20'137","reported_seq":778,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-20T09:18:43.556596+0000","last_change":"2024-03-20T09:17:48.865046+0000","last_active":"2024-03-20T09:18:43.556596+0000","last_peered":"2024-03-20T09:18:43.556596+0000","last_clean":"2024-03-20T09:18:43.556596+0000","last_became_active":"2024-03-20T09:10:35.215772+0000","last_became_peered":"2024-03-20T09:10:35.215772+0000","last_unstale":"2024-03-20T09:18:43.556596+0000","last_undegraded":"2024-03-20T09:18:43.556596+0000","last_fullsized":"2024-03-20T09:18:43.556596+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:48.864925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:17:48.864925+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:13.258032+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"619'3541","reported_seq":3579,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-20T09:18:45.156185+0000","last_change":"2024-03-20T09:18:11.523337+0000","last_active":"2024-03-20T09:18:45.156185+0000","last_peered":"2024-03-20T09:18:45.156185+0000","last_clean":"2024-03-20T09:18:45.156185+0000","last_became_active":"2024-03-20T09:18:11.523040+0000","last_became_peered":"2024-03-20T09:18:11.523040+0000","last_unstale":"2024-03-20T09:18:45.156185+0000","last_undegraded":"2024-03-20T09:18:45.156185+0000","last_fullsized":"2024-03-20T09:18:45.156185+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":3541,"log_dups_size":0,"ondisk_log_size":3541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:39.162942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232062976,"num_objects":3541,"num_object_clones":0,"num_object_copies":10623,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3541,"num_write_kb":226624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"20'137","reported_seq":1340,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-03-20T09:18:47.264243+0000","last_change":"2024-03-20T09:18:37.405968+0000","last_active":"2024-03-20T09:18:47.264243+0000","last_peered":"2024-03-20T09:18:47.264243+0000","last_clean":"2024-03-20T09:18:47.264243+0000","last_became_active":"2024-03-20T09:10:34.099414+0000","last_became_peered":"2024-03-20T09:10:34.099414+0000","last_unstale":"2024-03-20T09:18:47.264243+0000","last_undegraded":"2024-03-20T09:18:47.264243+0000","last_fullsized":"2024-03-20T09:18:47.264243+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:18:37.405832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:18:37.405832+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:44.985164+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"619'3658","reported_seq":3696,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-20T09:18:46.779831+0000","last_change":"2024-03-20T09:18:11.081562+0000","last_active":"2024-03-20T09:18:46.779831+0000","last_peered":"2024-03-20T09:18:46.779831+0000","last_clean":"2024-03-20T09:18:46.779831+0000","last_became_active":"2024-03-20T09:18:11.081322+0000","last_became_peered":"2024-03-20T09:18:11.081322+0000","last_unstale":"2024-03-20T09:18:46.779831+0000","last_undegraded":"2024-03-20T09:18:46.779831+0000","last_fullsized":"2024-03-20T09:18:46.779831+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":3658,"log_dups_size":0,"ondisk_log_size":3658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:38.316484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239730688,"num_objects":3658,"num_object_clones":0,"num_object_copies":10974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3658,"num_write_kb":234112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.a","version":"20'137","reported_seq":1144,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-20T09:18:44.538605+0000","last_change":"2024-03-20T09:18:44.538665+0000","last_active":"2024-03-20T09:18:44.538605+0000","last_peered":"2024-03-20T09:18:44.538605+0000","last_clean":"2024-03-20T09:18:44.538605+0000","last_became_active":"2024-03-20T09:07:52.618107+0000","last_became_peered":"2024-03-20T09:07:52.618107+0000","last_unstale":"2024-03-20T09:18:44.538605+0000","last_undegraded":"2024-03-20T09:18:44.538605+0000","last_fullsized":"2024-03-20T09:18:44.538605+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:18:44.538510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:18:44.538510+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:57.058581+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"619'3627","reported_seq":3665,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-20T09:18:46.288575+0000","last_change":"2024-03-20T09:18:11.081321+0000","last_active":"2024-03-20T09:18:46.288575+0000","last_peered":"2024-03-20T09:18:46.288575+0000","last_clean":"2024-03-20T09:18:46.288575+0000","last_became_active":"2024-03-20T09:18:11.080988+0000","last_became_peered":"2024-03-20T09:18:11.080988+0000","last_unstale":"2024-03-20T09:18:46.288575+0000","last_undegraded":"2024-03-20T09:18:46.288575+0000","last_fullsized":"2024-03-20T09:18:46.288575+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":3627,"log_dups_size":0,"ondisk_log_size":3627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:30.376445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237699072,"num_objects":3627,"num_object_clones":0,"num_object_copies":10881,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3627,"num_write_kb":232128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,0],"acting":[10,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"20'137","reported_seq":1308,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-20T09:18:43.546894+0000","last_change":"2024-03-20T09:18:36.914682+0000","last_active":"2024-03-20T09:18:43.546894+0000","last_peered":"2024-03-20T09:18:43.546894+0000","last_clean":"2024-03-20T09:18:43.546894+0000","last_became_active":"2024-03-20T09:12:14.306384+0000","last_became_peered":"2024-03-20T09:12:14.306384+0000","last_unstale":"2024-03-20T09:18:43.546894+0000","last_undegraded":"2024-03-20T09:18:43.546894+0000","last_fullsized":"2024-03-20T09:18:43.546894+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:18:36.914556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:18:36.914556+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:57.995921+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"619'3519","reported_seq":3556,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-20T09:18:45.793166+0000","last_change":"2024-03-20T09:18:11.523500+0000","last_active":"2024-03-20T09:18:45.793166+0000","last_peered":"2024-03-20T09:18:45.793166+0000","last_clean":"2024-03-20T09:18:45.793166+0000","last_became_active":"2024-03-20T09:18:11.522986+0000","last_became_peered":"2024-03-20T09:18:11.522986+0000","last_unstale":"2024-03-20T09:18:45.793166+0000","last_undegraded":"2024-03-20T09:18:45.793166+0000","last_fullsized":"2024-03-20T09:18:45.793166+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":3519,"log_dups_size":0,"ondisk_log_size":3519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:22.129530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230621184,"num_objects":3519,"num_object_clones":0,"num_object_copies":10557,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3519,"num_write_kb":225216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"20'137","reported_seq":1195,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-20T09:18:43.544678+0000","last_change":"2024-03-20T09:17:48.424808+0000","last_active":"2024-03-20T09:18:43.544678+0000","last_peered":"2024-03-20T09:18:43.544678+0000","last_clean":"2024-03-20T09:18:43.544678+0000","last_became_active":"2024-03-20T09:07:37.003722+0000","last_became_peered":"2024-03-20T09:07:37.003722+0000","last_unstale":"2024-03-20T09:18:43.544678+0000","last_undegraded":"2024-03-20T09:18:43.544678+0000","last_fullsized":"2024-03-20T09:18:43.544678+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:48.424717+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:07:06.480160+0000","last_clean_scrub_stamp":"2024-03-20T09:17:48.424717+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:02.571050+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.2","version":"619'3562","reported_seq":3600,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-20T09:18:45.831151+0000","last_change":"2024-03-20T09:18:11.083520+0000","last_active":"2024-03-20T09:18:45.831151+0000","last_peered":"2024-03-20T09:18:45.831151+0000","last_clean":"2024-03-20T09:18:45.831151+0000","last_became_active":"2024-03-20T09:18:11.083136+0000","last_became_peered":"2024-03-20T09:18:11.083136+0000","last_unstale":"2024-03-20T09:18:45.831151+0000","last_undegraded":"2024-03-20T09:18:45.831151+0000","last_fullsized":"2024-03-20T09:18:45.831151+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":3562,"log_dups_size":0,"ondisk_log_size":3562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:27.386223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233439232,"num_objects":3562,"num_object_clones":0,"num_object_copies":10686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3562,"num_write_kb":227968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"20'137","reported_seq":1220,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-20T09:18:43.544807+0000","last_change":"2024-03-20T09:18:41.078380+0000","last_active":"2024-03-20T09:18:43.544807+0000","last_peered":"2024-03-20T09:18:43.544807+0000","last_clean":"2024-03-20T09:18:43.544807+0000","last_became_active":"2024-03-20T09:07:52.593671+0000","last_became_peered":"2024-03-20T09:07:52.593671+0000","last_unstale":"2024-03-20T09:18:43.544807+0000","last_undegraded":"2024-03-20T09:18:43.544807+0000","last_fullsized":"2024-03-20T09:18:43.544807+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:18:41.078246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:18:41.078246+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:20:00.918926+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.7","version":"619'3735","reported_seq":3772,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-20T09:18:47.193224+0000","last_change":"2024-03-20T09:18:11.080196+0000","last_active":"2024-03-20T09:18:47.193224+0000","last_peered":"2024-03-20T09:18:47.193224+0000","last_clean":"2024-03-20T09:18:47.193224+0000","last_became_active":"2024-03-20T09:18:11.079858+0000","last_became_peered":"2024-03-20T09:18:11.079858+0000","last_unstale":"2024-03-20T09:18:47.193224+0000","last_undegraded":"2024-03-20T09:18:47.193224+0000","last_fullsized":"2024-03-20T09:18:47.193224+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":3735,"log_dups_size":0,"ondisk_log_size":3735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:31.794901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244776960,"num_objects":3735,"num_object_clones":0,"num_object_copies":11205,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3735,"num_write_kb":239040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"20'137","reported_seq":1215,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-03-20T09:18:47.268284+0000","last_change":"2024-03-20T09:18:45.511064+0000","last_active":"2024-03-20T09:18:47.268284+0000","last_peered":"2024-03-20T09:18:47.268284+0000","last_clean":"2024-03-20T09:18:47.268284+0000","last_became_active":"2024-03-20T09:12:02.221686+0000","last_became_peered":"2024-03-20T09:12:02.221686+0000","last_unstale":"2024-03-20T09:18:47.268284+0000","last_undegraded":"2024-03-20T09:18:47.268284+0000","last_fullsized":"2024-03-20T09:18:47.268284+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:18:45.510938+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:18.788771+0000","last_clean_scrub_stamp":"2024-03-20T09:18:45.510938+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:20:05.681444+0000","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,8],"acting":[1,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"619'3611","reported_seq":3649,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-20T09:18:46.283367+0000","last_change":"2024-03-20T09:18:11.078580+0000","last_active":"2024-03-20T09:18:46.283367+0000","last_peered":"2024-03-20T09:18:46.283367+0000","last_clean":"2024-03-20T09:18:46.283367+0000","last_became_active":"2024-03-20T09:18:11.078275+0000","last_became_peered":"2024-03-20T09:18:11.078275+0000","last_unstale":"2024-03-20T09:18:46.283367+0000","last_undegraded":"2024-03-20T09:18:46.283367+0000","last_fullsized":"2024-03-20T09:18:46.283367+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":3611,"log_dups_size":0,"ondisk_log_size":3611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:37.063979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236650496,"num_objects":3611,"num_object_clones":0,"num_object_copies":10833,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3611,"num_write_kb":231104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":1271,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-20T09:18:43.546834+0000","last_change":"2024-03-20T09:17:48.991381+0000","last_active":"2024-03-20T09:18:43.546834+0000","last_peered":"2024-03-20T09:18:43.546834+0000","last_clean":"2024-03-20T09:18:43.546834+0000","last_became_active":"2024-03-20T09:12:14.305989+0000","last_became_peered":"2024-03-20T09:12:14.305989+0000","last_unstale":"2024-03-20T09:18:43.546834+0000","last_undegraded":"2024-03-20T09:18:43.546834+0000","last_fullsized":"2024-03-20T09:18:43.546834+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:48.991260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:17:48.991260+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:01.289300+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.6","version":"620'3802","reported_seq":3842,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-03-20T09:18:49.406290+0000","last_change":"2024-03-20T09:18:11.523259+0000","last_active":"2024-03-20T09:18:49.406290+0000","last_peered":"2024-03-20T09:18:49.406290+0000","last_clean":"2024-03-20T09:18:49.406290+0000","last_became_active":"2024-03-20T09:18:11.522922+0000","last_became_peered":"2024-03-20T09:18:11.522922+0000","last_unstale":"2024-03-20T09:18:49.406290+0000","last_undegraded":"2024-03-20T09:18:49.406290+0000","last_fullsized":"2024-03-20T09:18:49.406290+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":3802,"log_dups_size":0,"ondisk_log_size":3802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:32.564244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249167872,"num_objects":3802,"num_object_clones":0,"num_object_copies":11406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3802,"num_write_kb":243328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"20'137","reported_seq":1519,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-20T09:18:43.548409+0000","last_change":"2024-03-20T09:17:44.468318+0000","last_active":"2024-03-20T09:18:43.548409+0000","last_peered":"2024-03-20T09:18:43.548409+0000","last_clean":"2024-03-20T09:18:43.548409+0000","last_became_active":"2024-03-20T09:12:14.313541+0000","last_became_peered":"2024-03-20T09:12:14.313541+0000","last_unstale":"2024-03-20T09:18:43.548409+0000","last_undegraded":"2024-03-20T09:18:43.548409+0000","last_fullsized":"2024-03-20T09:18:43.548409+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:44.468179+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:19.513511+0000","last_clean_scrub_stamp":"2024-03-20T09:17:44.468179+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:18:58.972637+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"619'3609","reported_seq":3646,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-20T09:18:47.108330+0000","last_change":"2024-03-20T09:18:11.082142+0000","last_active":"2024-03-20T09:18:47.108330+0000","last_peered":"2024-03-20T09:18:47.108330+0000","last_clean":"2024-03-20T09:18:47.108330+0000","last_became_active":"2024-03-20T09:18:11.081786+0000","last_became_peered":"2024-03-20T09:18:11.081786+0000","last_unstale":"2024-03-20T09:18:47.108330+0000","last_undegraded":"2024-03-20T09:18:47.108330+0000","last_fullsized":"2024-03-20T09:18:47.108330+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":3609,"log_dups_size":0,"ondisk_log_size":3609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:16.733507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236519424,"num_objects":3609,"num_object_clones":0,"num_object_copies":10827,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3609,"num_write_kb":230976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"20'137","reported_seq":673,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-20T09:18:43.550785+0000","last_change":"2024-03-20T09:17:47.350252+0000","last_active":"2024-03-20T09:18:43.550785+0000","last_peered":"2024-03-20T09:18:43.550785+0000","last_clean":"2024-03-20T09:18:43.550785+0000","last_became_active":"2024-03-20T09:15:34.225022+0000","last_became_peered":"2024-03-20T09:15:34.225022+0000","last_unstale":"2024-03-20T09:18:43.550785+0000","last_undegraded":"2024-03-20T09:18:43.550785+0000","last_fullsized":"2024-03-20T09:18:43.550785+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:47.350114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:17:47.350114+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:00.915901+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,5],"acting":[6,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"619'3672","reported_seq":3709,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-20T09:18:46.880733+0000","last_change":"2024-03-20T09:18:11.081759+0000","last_active":"2024-03-20T09:18:46.880733+0000","last_peered":"2024-03-20T09:18:46.880733+0000","last_clean":"2024-03-20T09:18:46.880733+0000","last_became_active":"2024-03-20T09:18:11.081425+0000","last_became_peered":"2024-03-20T09:18:11.081425+0000","last_unstale":"2024-03-20T09:18:46.880733+0000","last_undegraded":"2024-03-20T09:18:46.880733+0000","last_fullsized":"2024-03-20T09:18:46.880733+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":3672,"log_dups_size":0,"ondisk_log_size":3672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:22.317679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240648192,"num_objects":3672,"num_object_clones":0,"num_object_copies":11016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3672,"num_write_kb":235008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"20'137","reported_seq":1216,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-20T09:18:43.582947+0000","last_change":"2024-03-20T09:18:39.685223+0000","last_active":"2024-03-20T09:18:43.582947+0000","last_peered":"2024-03-20T09:18:43.582947+0000","last_clean":"2024-03-20T09:18:43.582947+0000","last_became_active":"2024-03-20T09:13:27.087760+0000","last_became_peered":"2024-03-20T09:13:27.087760+0000","last_unstale":"2024-03-20T09:18:43.582947+0000","last_undegraded":"2024-03-20T09:18:43.582947+0000","last_fullsized":"2024-03-20T09:18:43.582947+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:18:39.685133+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:14:10.560890+0000","last_clean_scrub_stamp":"2024-03-20T09:18:39.685133+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:50.414787+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"619'3620","reported_seq":3658,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-20T09:18:47.209364+0000","last_change":"2024-03-20T09:18:11.079851+0000","last_active":"2024-03-20T09:18:47.209364+0000","last_peered":"2024-03-20T09:18:47.209364+0000","last_clean":"2024-03-20T09:18:47.209364+0000","last_became_active":"2024-03-20T09:18:11.079589+0000","last_became_peered":"2024-03-20T09:18:11.079589+0000","last_unstale":"2024-03-20T09:18:47.209364+0000","last_undegraded":"2024-03-20T09:18:47.209364+0000","last_fullsized":"2024-03-20T09:18:47.209364+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":3620,"log_dups_size":0,"ondisk_log_size":3620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:36.062370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237240320,"num_objects":3620,"num_object_clones":0,"num_object_copies":10860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3620,"num_write_kb":231680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,2],"acting":[5,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"20'137","reported_seq":1084,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-20T09:18:43.548727+0000","last_change":"2024-03-20T09:18:38.995228+0000","last_active":"2024-03-20T09:18:43.548727+0000","last_peered":"2024-03-20T09:18:43.548727+0000","last_clean":"2024-03-20T09:18:43.548727+0000","last_became_active":"2024-03-20T09:13:20.070596+0000","last_became_peered":"2024-03-20T09:13:20.070596+0000","last_unstale":"2024-03-20T09:18:43.548727+0000","last_undegraded":"2024-03-20T09:18:43.548727+0000","last_fullsized":"2024-03-20T09:18:43.548727+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:18:38.995049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:18:38.995049+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:52.947693+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.a","version":"620'4008","reported_seq":4047,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-03-20T09:18:49.413139+0000","last_change":"2024-03-20T09:18:11.079740+0000","last_active":"2024-03-20T09:18:49.413139+0000","last_peered":"2024-03-20T09:18:49.413139+0000","last_clean":"2024-03-20T09:18:49.413139+0000","last_became_active":"2024-03-20T09:18:11.079361+0000","last_became_peered":"2024-03-20T09:18:11.079361+0000","last_unstale":"2024-03-20T09:18:49.413139+0000","last_undegraded":"2024-03-20T09:18:49.413139+0000","last_fullsized":"2024-03-20T09:18:49.413139+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":4008,"log_dups_size":0,"ondisk_log_size":4008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:28.144588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262668288,"num_objects":4008,"num_object_clones":0,"num_object_copies":12024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4008,"num_write_kb":256512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,10],"acting":[2,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"20'137","reported_seq":735,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-20T09:18:43.556427+0000","last_change":"2024-03-20T09:17:46.760738+0000","last_active":"2024-03-20T09:18:43.556427+0000","last_peered":"2024-03-20T09:18:43.556427+0000","last_clean":"2024-03-20T09:18:43.556427+0000","last_became_active":"2024-03-20T09:12:14.311319+0000","last_became_peered":"2024-03-20T09:12:14.311319+0000","last_unstale":"2024-03-20T09:18:43.556427+0000","last_undegraded":"2024-03-20T09:18:43.556427+0000","last_fullsized":"2024-03-20T09:18:43.556427+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:46.760611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:17:46.760611+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:18:47.142571+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"619'3597","reported_seq":3632,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-20T09:18:47.045787+0000","last_change":"2024-03-20T09:18:11.523380+0000","last_active":"2024-03-20T09:18:47.045787+0000","last_peered":"2024-03-20T09:18:47.045787+0000","last_clean":"2024-03-20T09:18:47.045787+0000","last_became_active":"2024-03-20T09:18:11.522906+0000","last_became_peered":"2024-03-20T09:18:11.522906+0000","last_unstale":"2024-03-20T09:18:47.045787+0000","last_undegraded":"2024-03-20T09:18:47.045787+0000","last_fullsized":"2024-03-20T09:18:47.045787+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":3597,"log_dups_size":0,"ondisk_log_size":3597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:19.027570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235732992,"num_objects":3597,"num_object_clones":0,"num_object_copies":10791,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3597,"num_write_kb":230208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"20'137","reported_seq":1232,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-20T09:18:43.648768+0000","last_change":"2024-03-20T09:18:43.648807+0000","last_active":"2024-03-20T09:18:43.648768+0000","last_peered":"2024-03-20T09:18:43.648768+0000","last_clean":"2024-03-20T09:18:43.648768+0000","last_became_active":"2024-03-20T09:10:35.119706+0000","last_became_peered":"2024-03-20T09:10:35.119706+0000","last_unstale":"2024-03-20T09:18:43.648768+0000","last_undegraded":"2024-03-20T09:18:43.648768+0000","last_fullsized":"2024-03-20T09:18:43.648768+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:18:43.648689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:18:43.648689+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:20:09.695726+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"619'3391","reported_seq":3429,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-20T09:18:45.145491+0000","last_change":"2024-03-20T09:18:11.080513+0000","last_active":"2024-03-20T09:18:45.145491+0000","last_peered":"2024-03-20T09:18:45.145491+0000","last_clean":"2024-03-20T09:18:45.145491+0000","last_became_active":"2024-03-20T09:18:11.080212+0000","last_became_peered":"2024-03-20T09:18:11.080212+0000","last_unstale":"2024-03-20T09:18:45.145491+0000","last_undegraded":"2024-03-20T09:18:45.145491+0000","last_fullsized":"2024-03-20T09:18:45.145491+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":3391,"log_dups_size":0,"ondisk_log_size":3391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:14.576641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222232576,"num_objects":3391,"num_object_clones":0,"num_object_copies":10173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3391,"num_write_kb":217024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"570'138","reported_seq":1251,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-20T09:18:43.549401+0000","last_change":"2024-03-20T09:17:45.816003+0000","last_active":"2024-03-20T09:18:43.549401+0000","last_peered":"2024-03-20T09:18:43.549401+0000","last_clean":"2024-03-20T09:18:43.549401+0000","last_became_active":"2024-03-20T09:13:27.087686+0000","last_became_peered":"2024-03-20T09:13:27.087686+0000","last_unstale":"2024-03-20T09:18:43.549401+0000","last_undegraded":"2024-03-20T09:18:43.549401+0000","last_fullsized":"2024-03-20T09:18:43.549401+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"570'138","last_scrub_stamp":"2024-03-20T09:17:45.815855+0000","last_deep_scrub":"570'138","last_deep_scrub_stamp":"2024-03-20T09:16:13.725035+0000","last_clean_scrub_stamp":"2024-03-20T09:17:45.815855+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:05.315376+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"619'3567","reported_seq":3605,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-20T09:18:45.839021+0000","last_change":"2024-03-20T09:18:11.523430+0000","last_active":"2024-03-20T09:18:45.839021+0000","last_peered":"2024-03-20T09:18:45.839021+0000","last_clean":"2024-03-20T09:18:45.839021+0000","last_became_active":"2024-03-20T09:18:11.522844+0000","last_became_peered":"2024-03-20T09:18:11.522844+0000","last_unstale":"2024-03-20T09:18:45.839021+0000","last_undegraded":"2024-03-20T09:18:45.839021+0000","last_fullsized":"2024-03-20T09:18:45.839021+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":3567,"log_dups_size":0,"ondisk_log_size":3567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:10.081321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233766912,"num_objects":3567,"num_object_clones":0,"num_object_copies":10701,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3567,"num_write_kb":228288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,11],"acting":[6,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.d","version":"20'137","reported_seq":805,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-03-20T09:18:47.268467+0000","last_change":"2024-03-20T09:18:05.353742+0000","last_active":"2024-03-20T09:18:47.268467+0000","last_peered":"2024-03-20T09:18:47.268467+0000","last_clean":"2024-03-20T09:18:47.268467+0000","last_became_active":"2024-03-20T09:12:14.307864+0000","last_became_peered":"2024-03-20T09:12:14.307864+0000","last_unstale":"2024-03-20T09:18:47.268467+0000","last_undegraded":"2024-03-20T09:18:47.268467+0000","last_fullsized":"2024-03-20T09:18:47.268467+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:18:05.353559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:18:05.353559+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:07.761487+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"20'137","reported_seq":1119,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-20T09:18:43.557999+0000","last_change":"2024-03-20T09:17:47.874749+0000","last_active":"2024-03-20T09:18:43.557999+0000","last_peered":"2024-03-20T09:18:43.557999+0000","last_clean":"2024-03-20T09:18:43.557999+0000","last_became_active":"2024-03-20T09:10:34.096758+0000","last_became_peered":"2024-03-20T09:10:34.096758+0000","last_unstale":"2024-03-20T09:18:43.557999+0000","last_undegraded":"2024-03-20T09:18:43.557999+0000","last_fullsized":"2024-03-20T09:18:43.557999+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:47.874609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:17:47.874609+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:11.136312+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"20'137","reported_seq":1194,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-03-20T09:18:47.268155+0000","last_change":"2024-03-20T09:17:57.414516+0000","last_active":"2024-03-20T09:18:47.268155+0000","last_peered":"2024-03-20T09:18:47.268155+0000","last_clean":"2024-03-20T09:18:47.268155+0000","last_became_active":"2024-03-20T09:04:04.103645+0000","last_became_peered":"2024-03-20T09:04:04.103645+0000","last_unstale":"2024-03-20T09:18:47.268155+0000","last_undegraded":"2024-03-20T09:18:47.268155+0000","last_fullsized":"2024-03-20T09:18:47.268155+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:57.414409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:17:57.414409+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:26.874738+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"20'137","reported_seq":645,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-20T09:18:43.556695+0000","last_change":"2024-03-20T09:17:50.805976+0000","last_active":"2024-03-20T09:18:43.556695+0000","last_peered":"2024-03-20T09:18:43.556695+0000","last_clean":"2024-03-20T09:18:43.556695+0000","last_became_active":"2024-03-20T09:10:34.097191+0000","last_became_peered":"2024-03-20T09:10:34.097191+0000","last_unstale":"2024-03-20T09:18:43.556695+0000","last_undegraded":"2024-03-20T09:18:43.556695+0000","last_fullsized":"2024-03-20T09:18:43.556695+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:50.805845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:17:50.805845+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:00.311287+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"20'137","reported_seq":1055,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-20T09:18:43.548380+0000","last_change":"2024-03-20T09:17:46.485538+0000","last_active":"2024-03-20T09:18:43.548380+0000","last_peered":"2024-03-20T09:18:43.548380+0000","last_clean":"2024-03-20T09:18:43.548380+0000","last_became_active":"2024-03-20T09:13:27.087859+0000","last_became_peered":"2024-03-20T09:13:27.087859+0000","last_unstale":"2024-03-20T09:18:43.548380+0000","last_undegraded":"2024-03-20T09:18:43.548380+0000","last_fullsized":"2024-03-20T09:18:43.548380+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:46.485413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:17:46.485413+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:13.166557+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,3],"acting":[0,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"20'137","reported_seq":883,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-03-20T09:18:47.268247+0000","last_change":"2024-03-20T09:17:52.461386+0000","last_active":"2024-03-20T09:18:47.268247+0000","last_peered":"2024-03-20T09:18:47.268247+0000","last_clean":"2024-03-20T09:18:47.268247+0000","last_became_active":"2024-03-20T09:13:27.087760+0000","last_became_peered":"2024-03-20T09:13:27.087760+0000","last_unstale":"2024-03-20T09:18:47.268247+0000","last_undegraded":"2024-03-20T09:18:47.268247+0000","last_fullsized":"2024-03-20T09:18:47.268247+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:52.461261+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:16:15.448920+0000","last_clean_scrub_stamp":"2024-03-20T09:17:52.461261+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:16.137279+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"20'137","reported_seq":1199,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-20T09:18:43.552336+0000","last_change":"2024-03-20T09:17:46.961152+0000","last_active":"2024-03-20T09:18:43.552336+0000","last_peered":"2024-03-20T09:18:43.552336+0000","last_clean":"2024-03-20T09:18:43.552336+0000","last_became_active":"2024-03-20T09:10:35.095339+0000","last_became_peered":"2024-03-20T09:10:35.095339+0000","last_unstale":"2024-03-20T09:18:43.552336+0000","last_undegraded":"2024-03-20T09:18:43.552336+0000","last_fullsized":"2024-03-20T09:18:43.552336+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:46.961059+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:23.640533+0000","last_clean_scrub_stamp":"2024-03-20T09:17:46.961059+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:11.863478+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"20'137","reported_seq":1270,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-03-20T09:18:47.265939+0000","last_change":"2024-03-20T09:17:47.414530+0000","last_active":"2024-03-20T09:18:47.265939+0000","last_peered":"2024-03-20T09:18:47.265939+0000","last_clean":"2024-03-20T09:18:47.265939+0000","last_became_active":"2024-03-20T09:04:14.208448+0000","last_became_peered":"2024-03-20T09:04:14.208448+0000","last_unstale":"2024-03-20T09:18:47.265939+0000","last_undegraded":"2024-03-20T09:18:47.265939+0000","last_fullsized":"2024-03-20T09:18:47.265939+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:47.414381+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:16:15.754362+0000","last_clean_scrub_stamp":"2024-03-20T09:17:47.414381+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:08.321085+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"20'137","reported_seq":532,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-03-20T09:18:43.557373+0000","last_change":"2024-03-20T09:17:44.727578+0000","last_active":"2024-03-20T09:18:43.557373+0000","last_peered":"2024-03-20T09:18:43.557373+0000","last_clean":"2024-03-20T09:18:43.557373+0000","last_became_active":"2024-03-20T09:12:14.307581+0000","last_became_peered":"2024-03-20T09:12:14.307581+0000","last_unstale":"2024-03-20T09:18:43.557373+0000","last_undegraded":"2024-03-20T09:18:43.557373+0000","last_fullsized":"2024-03-20T09:18:43.557373+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:44.727492+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:07:06.480160+0000","last_clean_scrub_stamp":"2024-03-20T09:17:44.727492+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:18:46.002339+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"20'137","reported_seq":814,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-03-20T09:18:47.268008+0000","last_change":"2024-03-20T09:17:59.332278+0000","last_active":"2024-03-20T09:18:47.268008+0000","last_peered":"2024-03-20T09:18:47.268008+0000","last_clean":"2024-03-20T09:18:47.268008+0000","last_became_active":"2024-03-20T09:13:20.074439+0000","last_became_peered":"2024-03-20T09:13:20.074439+0000","last_unstale":"2024-03-20T09:18:47.268008+0000","last_undegraded":"2024-03-20T09:18:47.268008+0000","last_fullsized":"2024-03-20T09:18:47.268008+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:59.332142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:17:59.332142+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:28.719063+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,11],"acting":[1,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":3817144320,"num_objects":58245,"num_object_clones":0,"num_object_copies":174735,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58245,"num_write_kb":3727680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11459493888,"data_stored":11459493888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58245,"ondisk_log_size":58245,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4248,"ondisk_log_size":4248,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860688,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1380892,"kb_used_data":880464,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":92364260,"statfs":{"total":95995035648,"available":94581002240,"internally_reserved":0,"allocated":901595136,"data_stored":901269886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":512424388},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":89,"seq":382252089622,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1753496,"kb_used_data":1375248,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":91991656,"statfs":{"total":95995035648,"available":94199455744,"internally_reserved":0,"allocated":1408253952,"data_stored":1407928702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":387316160},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":0,"seq":343597383967,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":70,"seq":300647711010,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":783192,"kb_used_data":463248,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92961960,"statfs":{"total":95995035648,"available":95193047040,"internally_reserved":0,"allocated":474365952,"data_stored":474040702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":327612870},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":60,"seq":257698038055,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1634976,"kb_used_data":1120664,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":92110176,"statfs":{"total":95995035648,"available":94320820224,"internally_reserved":0,"allocated":1147559936,"data_stored":1147231134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":526645706},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332396,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1552856,"kb_used_data":1124624,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":92192296,"statfs":{"total":95995035648,"available":94404911104,"internally_reserved":0,"allocated":1151614976,"data_stored":1151289726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":438499782},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575174,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1508008,"kb_used_data":949276,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":92237144,"statfs":{"total":95995035648,"available":94450835456,"internally_reserved":0,"allocated":972058624,"data_stored":971728859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":572127686},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738693,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2330456,"kb_used_data":1848080,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":91414696,"statfs":{"total":95995035648,"available":93608648704,"internally_reserved":0,"allocated":1892433920,"data_stored":1892108670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":493943237},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411648,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1555632,"kb_used_data":1217636,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":92189520,"statfs":{"total":95995035648,"available":94402068480,"internally_reserved":0,"allocated":1246859264,"data_stored":1246525947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150011,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":639136,"kb_used_data":220624,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":93106016,"statfs":{"total":95995035648,"available":95340560384,"internally_reserved":0,"allocated":225918976,"data_stored":225593726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":428538311},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953785,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":443,"seq":1902670512228,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2104096,"kb_used_data":1871448,"kb_used_omap":181,"kb_used_meta":232458,"kb_avail":91641056,"statfs":{"total":95995035648,"available":93840441344,"internally_reserved":0,"allocated":1916362752,"data_stored":1916033950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":186131,"internal_metadata":238037229},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1914568704,"data_stored":1914568704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":976551936,"data_stored":976551936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1276772352,"data_stored":1276772352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":230621184,"data_stored":230621184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1922039808,"data_stored":1922039808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1162543104,"data_stored":1162543104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1161887744,"data_stored":1161887744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":483721216,"data_stored":483721216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1420492800,"data_stored":1420492800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":910295040,"data_stored":910295040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:18:53.800 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-20T09:18:53.800 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:18:53.800 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-20T09:18:53.801 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-20T09:18:53.942 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:18:53.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:53.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:18:54.052 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:18:54.078 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:54.098 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:18:54.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:54.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:18:54.202 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:18:54.238 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:54.265 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:18:54.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:54.268 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:18:54.368 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:18:54.394 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:54 smithi026 ceph-mon[68309]: 1.e deep-scrub starts 2024-03-20T09:18:54.394 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:54 smithi026 ceph-mon[68309]: 1.e deep-scrub ok 2024-03-20T09:18:54.394 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:54 smithi026 ceph-mon[68309]: pgmap v1491: 47 pgs: 47 active+clean; 4.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-03-20T09:18:54.395 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:54 smithi122 ceph-mon[73495]: 1.e deep-scrub starts 2024-03-20T09:18:54.395 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:54 smithi122 ceph-mon[73495]: 1.e deep-scrub ok 2024-03-20T09:18:54.395 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:54 smithi122 ceph-mon[73495]: pgmap v1491: 47 pgs: 47 active+clean; 4.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-03-20T09:18:54.398 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:54.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:54 smithi115 ceph-mon[72476]: 1.e deep-scrub starts 2024-03-20T09:18:54.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:54 smithi115 ceph-mon[72476]: 1.e deep-scrub ok 2024-03-20T09:18:54.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:54 smithi115 ceph-mon[72476]: pgmap v1491: 47 pgs: 47 active+clean; 4.0 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-03-20T09:18:54.422 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:18:54.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:54.426 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:18:54.527 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:18:54.558 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:54.580 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:18:54.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:54.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:18:54.625 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 34 16 70319 70303 129.215 81.625 0.00282708 0.00771784 2024-03-20T09:18:54.685 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:18:54.714 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:54.748 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:18:54.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:54.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:18:54.812 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:18:54.824 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:18:54.863 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:18:54.926 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:55.090 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:18:55.090 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:18:55.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.090 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:70838 5.5 5.54bb6fd5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e622)", 2024-03-20T09:18:55.090 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:18:55.086002+0000", 2024-03-20T09:18:55.090 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.00017865000000000001, 2024-03-20T09:18:55.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000218376, 2024-03-20T09:18:55.090 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:18:55.090 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "queued for pg", 2024-03-20T09:18:55.090 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:18:55.090 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:18:55.091 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:18:55.091 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 70838 2024-03-20T09:18:55.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.091 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:18:55.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:18:55.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.086002+0000", 2024-03-20T09:18:55.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:18:55.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:18:55.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.086002+0000", 2024-03-20T09:18:55.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:18:55.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:18:55.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.086003+0000", 2024-03-20T09:18:55.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.48e-06 2024-03-20T09:18:55.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:18:55.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.086051+0000", 2024-03-20T09:18:55.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8006000000000001e-05 2024-03-20T09:18:55.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:18:55.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.086067+0000", 2024-03-20T09:18:55.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5622999999999999e-05 2024-03-20T09:18:55.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:18:55.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.086074+0000", 2024-03-20T09:18:55.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6290000000000001e-06 2024-03-20T09:18:55.093 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:18:55.093 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:18:55.093 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:18:55.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.093 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.25966.0:70837 5.2 e622/605)", 2024-03-20T09:18:55.093 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:18:55.083859+0000", 2024-03-20T09:18:55.093 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0023217709999999998, 2024-03-20T09:18:55.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024329109999999998, 2024-03-20T09:18:55.093 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:18:55.094 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "queued for pg", 2024-03-20T09:18:55.094 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:18:55.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:18:55.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.083859+0000", 2024-03-20T09:18:55.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:18:55.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.094 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:18:55.094 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.083859+0000", 2024-03-20T09:18:55.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:18:55.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:18:55.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.083859+0000", 2024-03-20T09:18:55.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9800000000000003e-07 2024-03-20T09:18:55.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:18:55.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.086229+0000", 2024-03-20T09:18:55.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0023696640000000001 2024-03-20T09:18:55.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:18:55.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.086240+0000", 2024-03-20T09:18:55.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1177e-05 2024-03-20T09:18:55.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:18:55.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.086247+0000", 2024-03-20T09:18:55.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7859999999999999e-06 2024-03-20T09:18:55.096 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:18:55.096 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:18:55.096 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:18:55.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.096 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop_reply(client.25966.0:70826 5.5 e622/605)", 2024-03-20T09:18:55.096 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:18:55.086292+0000", 2024-03-20T09:18:55.096 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4294967295.9998889, 2024-03-20T09:18:55.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5766000000000001e-05, 2024-03-20T09:18:55.096 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:18:55.097 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "queued for pg", 2024-03-20T09:18:55.097 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:18:55.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:18:55.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.086292+0000", 2024-03-20T09:18:55.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:18:55.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:18:55.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.086292+0000", 2024-03-20T09:18:55.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:18:55.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:18:55.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.086292+0000", 2024-03-20T09:18:55.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3799999999999997e-07 2024-03-20T09:18:55.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:18:55.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.086296+0000", 2024-03-20T09:18:55.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8460000000000001e-06 2024-03-20T09:18:55.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:18:55.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.086306+0000", 2024-03-20T09:18:55.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.008e-05 2024-03-20T09:18:55.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:18:55.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.086313+0000", 2024-03-20T09:18:55.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2119999999999997e-06 2024-03-20T09:18:55.099 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:18:55.099 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:18:55.099 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:18:55.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.099 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:70826 5.5 5:a9460f61:::benchmark_data_smithi122_157894_object70825:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e622)", 2024-03-20T09:18:55.099 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:18:55.076936+0000", 2024-03-20T09:18:55.099 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0092447629999999996, 2024-03-20T09:18:55.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.009477961, 2024-03-20T09:18:55.099 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:18:55.099 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:18:55.099 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:18:55.099 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:18:55.100 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:18:55.100 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 70826 2024-03-20T09:18:55.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.100 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:18:55.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:18:55.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.076936+0000", 2024-03-20T09:18:55.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:18:55.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:18:55.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.076936+0000", 2024-03-20T09:18:55.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:18:55.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:18:55.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.076936+0000", 2024-03-20T09:18:55.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2e-07 2024-03-20T09:18:55.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:18:55.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.076970+0000", 2024-03-20T09:18:55.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.358e-05 2024-03-20T09:18:55.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:18:55.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.076977+0000", 2024-03-20T09:18:55.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2350000000000002e-06 2024-03-20T09:18:55.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:18:55.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.076983+0000", 2024-03-20T09:18:55.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5389999999999998e-06 2024-03-20T09:18:55.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:18:55.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.079414+0000", 2024-03-20T09:18:55.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024316339999999998 2024-03-20T09:18:55.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:18:55.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.079506+0000", 2024-03-20T09:18:55.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1640999999999999e-05 2024-03-20T09:18:55.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 1,6", 2024-03-20T09:18:55.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.079593+0000", 2024-03-20T09:18:55.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7245999999999998e-05 2024-03-20T09:18:55.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:18:55.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.083291+0000", 2024-03-20T09:18:55.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036980440000000002 2024-03-20T09:18:55.103 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:18:55.103 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:18:55.103 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:18:55.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.104 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.25966.0:70828 5.2 e622/605 5:4a0a6700:::benchmark_data_smithi122_157894_object70827:head v 622'4455, mlcod=622'4452)", 2024-03-20T09:18:55.104 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:18:55.080331+0000", 2024-03-20T09:18:55.104 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0058494979999999999, 2024-03-20T09:18:55.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0061768420000000001, 2024-03-20T09:18:55.104 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:18:55.104 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:18:55.104 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:18:55.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:18:55.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.080331+0000", 2024-03-20T09:18:55.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:18:55.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:18:55.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.080331+0000", 2024-03-20T09:18:55.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:18:55.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:18:55.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.080332+0000", 2024-03-20T09:18:55.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0079999999999999e-06 2024-03-20T09:18:55.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:18:55.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.080364+0000", 2024-03-20T09:18:55.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2369999999999997e-05 2024-03-20T09:18:55.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:18:55.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.080406+0000", 2024-03-20T09:18:55.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1291000000000002e-05 2024-03-20T09:18:55.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:18:55.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.080420+0000", 2024-03-20T09:18:55.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3971999999999999e-05 2024-03-20T09:18:55.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:18:55.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.080703+0000", 2024-03-20T09:18:55.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00028317000000000001 2024-03-20T09:18:55.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:18:55.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.080724+0000", 2024-03-20T09:18:55.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1273000000000001e-05 2024-03-20T09:18:55.107 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:18:55.107 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:18:55.107 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:18:55.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.107 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:70830 5.b 5:dfd775ed:::benchmark_data_smithi122_157894_object70829:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e622)", 2024-03-20T09:18:55.107 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:18:55.080502+0000", 2024-03-20T09:18:55.107 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0056780399999999996, 2024-03-20T09:18:55.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0060715420000000001, 2024-03-20T09:18:55.107 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:18:55.107 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:18:55.107 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:18:55.108 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:18:55.108 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:18:55.108 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 70830 2024-03-20T09:18:55.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.108 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:18:55.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:18:55.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.080502+0000", 2024-03-20T09:18:55.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:18:55.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:18:55.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.080502+0000", 2024-03-20T09:18:55.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:18:55.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:18:55.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.080504+0000", 2024-03-20T09:18:55.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3e-06 2024-03-20T09:18:55.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:18:55.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.080586+0000", 2024-03-20T09:18:55.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1955999999999999e-05 2024-03-20T09:18:55.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:18:55.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.080598+0000", 2024-03-20T09:18:55.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2274e-05 2024-03-20T09:18:55.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:18:55.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.080606+0000", 2024-03-20T09:18:55.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.824e-06 2024-03-20T09:18:55.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:18:55.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.080663+0000", 2024-03-20T09:18:55.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7326e-05 2024-03-20T09:18:55.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:18:55.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.080811+0000", 2024-03-20T09:18:55.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014796399999999999 2024-03-20T09:18:55.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5,7", 2024-03-20T09:18:55.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.080955+0000", 2024-03-20T09:18:55.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014381800000000001 2024-03-20T09:18:55.111 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:18:55.111 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:18:55.111 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:18:55.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.111 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.25966.0:70829 5.0 e622/605 5:00faa73b:::benchmark_data_smithi122_157894_object70828:head v 622'4466, mlcod=622'4464)", 2024-03-20T09:18:55.111 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:18:55.081824+0000", 2024-03-20T09:18:55.111 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0043565080000000003, 2024-03-20T09:18:55.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0048242399999999996, 2024-03-20T09:18:55.111 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:18:55.111 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:18:55.111 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:18:55.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:18:55.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.081824+0000", 2024-03-20T09:18:55.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:18:55.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:18:55.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.081824+0000", 2024-03-20T09:18:55.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:18:55.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:18:55.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.081824+0000", 2024-03-20T09:18:55.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.39e-07 2024-03-20T09:18:55.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:18:55.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.081861+0000", 2024-03-20T09:18:55.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6905e-05 2024-03-20T09:18:55.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:18:55.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.081869+0000", 2024-03-20T09:18:55.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0900000000000005e-06 2024-03-20T09:18:55.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:18:55.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.081875+0000", 2024-03-20T09:18:55.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2900000000000002e-06 2024-03-20T09:18:55.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:18:55.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.082199+0000", 2024-03-20T09:18:55.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00032430700000000002 2024-03-20T09:18:55.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:18:55.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.082217+0000", 2024-03-20T09:18:55.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8349e-05 2024-03-20T09:18:55.114 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:18:55.114 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:18:55.114 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:18:55.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.114 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.25966.0:70836 5.8 e622/605 5:1fb6c81b:::benchmark_data_smithi122_157894_object70835:head v 622'4371, mlcod=622'4368)", 2024-03-20T09:18:55.115 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:18:55.082223+0000", 2024-03-20T09:18:55.115 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0039573990000000003, 2024-03-20T09:18:55.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0044852980000000004, 2024-03-20T09:18:55.115 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:18:55.115 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:18:55.115 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:18:55.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:18:55.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.082223+0000", 2024-03-20T09:18:55.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:18:55.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:18:55.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.082223+0000", 2024-03-20T09:18:55.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:18:55.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:18:55.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.082223+0000", 2024-03-20T09:18:55.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7999999999999996e-07 2024-03-20T09:18:55.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:18:55.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.082253+0000", 2024-03-20T09:18:55.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9933e-05 2024-03-20T09:18:55.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:18:55.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.082273+0000", 2024-03-20T09:18:55.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9728000000000002e-05 2024-03-20T09:18:55.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:18:55.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.082279+0000", 2024-03-20T09:18:55.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1120000000000001e-06 2024-03-20T09:18:55.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:18:55.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.082352+0000", 2024-03-20T09:18:55.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2967999999999997e-05 2024-03-20T09:18:55.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:18:55.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.082370+0000", 2024-03-20T09:18:55.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7727000000000002e-05 2024-03-20T09:18:55.118 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:18:55.118 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:18:55.118 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:18:55.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.119 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.25966.0:70834 5.2 e622/605 5:417f1156:::benchmark_data_smithi122_157894_object70833:head v 622'4456, mlcod=622'4453)", 2024-03-20T09:18:55.119 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:18:55.083130+0000", 2024-03-20T09:18:55.119 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0030508110000000001, 2024-03-20T09:18:55.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036399610000000002, 2024-03-20T09:18:55.119 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:18:55.119 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:18:55.119 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:18:55.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:18:55.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.083130+0000", 2024-03-20T09:18:55.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:18:55.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:18:55.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.083130+0000", 2024-03-20T09:18:55.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:18:55.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:18:55.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.083130+0000", 2024-03-20T09:18:55.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8400000000000004e-07 2024-03-20T09:18:55.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:18:55.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.083165+0000", 2024-03-20T09:18:55.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4737999999999998e-05 2024-03-20T09:18:55.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:18:55.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.083175+0000", 2024-03-20T09:18:55.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0489000000000001e-05 2024-03-20T09:18:55.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:18:55.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.083183+0000", 2024-03-20T09:18:55.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4070000000000004e-06 2024-03-20T09:18:55.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:18:55.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.083910+0000", 2024-03-20T09:18:55.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00072712500000000002 2024-03-20T09:18:55.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:18:55.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.083931+0000", 2024-03-20T09:18:55.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1514000000000001e-05 2024-03-20T09:18:55.122 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:18:55.122 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:18:55.122 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:18:55.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.122 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:70833 5.3 5:cbc6ec55:::benchmark_data_smithi122_157894_object70832:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e622)", 2024-03-20T09:18:55.122 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:18:55.083532+0000", 2024-03-20T09:18:55.122 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0026479759999999998, 2024-03-20T09:18:55.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0033100040000000001, 2024-03-20T09:18:55.122 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:18:55.122 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:18:55.122 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:18:55.123 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:18:55.123 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:18:55.123 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 70833 2024-03-20T09:18:55.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.123 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:18:55.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:18:55.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.083532+0000", 2024-03-20T09:18:55.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:18:55.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:18:55.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.083532+0000", 2024-03-20T09:18:55.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:18:55.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:18:55.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.083539+0000", 2024-03-20T09:18:55.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3040000000000001e-06 2024-03-20T09:18:55.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:18:55.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.083578+0000", 2024-03-20T09:18:55.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9567999999999998e-05 2024-03-20T09:18:55.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:18:55.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.083599+0000", 2024-03-20T09:18:55.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0265e-05 2024-03-20T09:18:55.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:18:55.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.083606+0000", 2024-03-20T09:18:55.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6369999999999999e-06 2024-03-20T09:18:55.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:18:55.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.083734+0000", 2024-03-20T09:18:55.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000128002 2024-03-20T09:18:55.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:18:55.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.083887+0000", 2024-03-20T09:18:55.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000152607 2024-03-20T09:18:55.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:18:55.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:18:55.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5,6", 2024-03-20T09:18:55.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.084019+0000", 2024-03-20T09:18:55.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000131705 2024-03-20T09:18:55.126 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:18:55.126 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:18:55.126 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:18:55.126 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:18:55.126 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:18:55.126 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 10 2024-03-20T09:18:55.126 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:18:55.625 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 35 16 71641 71625 127.884 82.625 0.00830405 0.00781574 2024-03-20T09:18:56.293 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:18:56.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:56.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:18:56.332 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:18:56.399 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:18:56.443 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:56.474 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:18:56.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:56.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:18:56.578 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:18:56.625 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 36 16 73006 72990 126.701 85.3125 0.00763658 0.00788758 2024-03-20T09:18:56.632 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:57.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:57 smithi115 ceph-mon[72476]: pgmap v1492: 47 pgs: 47 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 87 MiB/s wr, 1.40k op/s 2024-03-20T09:18:57.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:57 smithi122 ceph-mon[73495]: pgmap v1492: 47 pgs: 47 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 87 MiB/s wr, 1.40k op/s 2024-03-20T09:18:57.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:57 smithi026 ceph-mon[68309]: pgmap v1492: 47 pgs: 47 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 87 MiB/s wr, 1.40k op/s 2024-03-20T09:18:57.528 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:18:57.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:57.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:18:57.557 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-20T09:18:57.626 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 37 16 74586 74570 125.946 98.75 0.00499532 0.00793576 2024-03-20T09:18:57.637 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:18:57.702 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:57.778 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:18:57.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:57.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:18:57.888 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:18:57.925 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:57.947 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:18:57.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:57.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:18:58.053 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:18:58.107 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:58.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:18:58 smithi026 ceph-mon[68309]: pgmap v1493: 47 pgs: 47 active+clean; 4.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-03-20T09:18:58.626 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 38 16 76086 76070 125.098 93.75 0.0171715 0.00798936 2024-03-20T09:18:58.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:18:58 smithi115 ceph-mon[72476]: pgmap v1493: 47 pgs: 47 active+clean; 4.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-03-20T09:18:58.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:18:58 smithi122 ceph-mon[73495]: pgmap v1493: 47 pgs: 47 active+clean; 4.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-03-20T09:18:58.977 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:18:58.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:58.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:18:59.082 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:18:59.145 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:59.281 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:18:59.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:59.288 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:18:59.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:18:59.390 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:18:59.425 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:59.448 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:18:59.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:59.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:18:59.552 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:18:59.610 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:59.625 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 39 16 77525 77509 124.196 89.9375 0.00839737 0.00804745 2024-03-20T09:18:59.639 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:18:59.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:59.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:18:59.745 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:18:59.775 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:59.798 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:18:59.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:59.801 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:18:59.902 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:18:59.928 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:18:59.955 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:18:59.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:18:59.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:19:00.059 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:19:00.113 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:00.626 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:2024-03-20T09:19:00.625913+0000 min lat: 0.0010551 max lat: 0.247256 avg lat: 0.00812371 2024-03-20T09:19:00.626 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T09:19:00.626 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 40 16 78772 78756 123.04 77.9375 0.0134265 0.00812371 2024-03-20T09:19:00.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:00 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:19:00.692 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:00 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:19:00.693 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:00 smithi115 ceph-mon[72476]: osdmap e623: 12 total, 12 up, 10 in 2024-03-20T09:19:00.693 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:00 smithi115 ceph-mon[72476]: pgmap v1495: 47 pgs: 47 active+clean; 4.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 87 MiB/s wr, 1.40k op/s 2024-03-20T09:19:00.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:00 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:19:00.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:00 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:19:00.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:00 smithi122 ceph-mon[73495]: osdmap e623: 12 total, 12 up, 10 in 2024-03-20T09:19:00.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:00 smithi122 ceph-mon[73495]: pgmap v1495: 47 pgs: 47 active+clean; 4.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 87 MiB/s wr, 1.40k op/s 2024-03-20T09:19:00.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:00 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:19:00.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:00 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:19:00.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:00 smithi026 ceph-mon[68309]: osdmap e623: 12 total, 12 up, 10 in 2024-03-20T09:19:00.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:00 smithi026 ceph-mon[68309]: pgmap v1495: 47 pgs: 47 active+clean; 4.5 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 87 MiB/s wr, 1.40k op/s 2024-03-20T09:19:01.303 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:01 smithi026 ceph-mon[68309]: 1.2 scrub starts 2024-03-20T09:19:01.303 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:01 smithi026 ceph-mon[68309]: 1.2 scrub ok 2024-03-20T09:19:01.626 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 41 16 80327 80311 122.409 97.1875 0.00746116 0.00816505 2024-03-20T09:19:01.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:01 smithi115 ceph-mon[72476]: 1.2 scrub starts 2024-03-20T09:19:01.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:01 smithi115 ceph-mon[72476]: 1.2 scrub ok 2024-03-20T09:19:01.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:01 smithi122 ceph-mon[73495]: 1.2 scrub starts 2024-03-20T09:19:01.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:01 smithi122 ceph-mon[73495]: 1.2 scrub ok 2024-03-20T09:19:02.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:02 smithi026 ceph-mon[68309]: pgmap v1496: 47 pgs: 47 active+clean; 4.7 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-03-20T09:19:02.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:02 smithi026 ceph-mon[68309]: osdmap e624: 12 total, 12 up, 10 in 2024-03-20T09:19:02.626 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 42 16 81794 81778 121.678 91.6875 0.0122176 0.00821454 2024-03-20T09:19:02.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:02 smithi115 ceph-mon[72476]: pgmap v1496: 47 pgs: 47 active+clean; 4.7 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-03-20T09:19:02.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:02 smithi115 ceph-mon[72476]: osdmap e624: 12 total, 12 up, 10 in 2024-03-20T09:19:02.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:02 smithi122 ceph-mon[73495]: pgmap v1496: 47 pgs: 47 active+clean; 4.7 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-03-20T09:19:02.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:02 smithi122 ceph-mon[73495]: osdmap e624: 12 total, 12 up, 10 in 2024-03-20T09:19:03.504 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:19:03.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:03.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:19:03.532 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:19:03.592 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:19:03.622 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:19:03.626 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 43 16 83201 83185 120.893 87.9375 0.0121104 0.00826726 2024-03-20T09:19:03.637 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:19:03.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:03 smithi115 ceph-mon[72476]: 1.1 scrub starts 2024-03-20T09:19:03.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:03 smithi115 ceph-mon[72476]: 1.1 scrub ok 2024-03-20T09:19:03.668 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:03.672 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:19:03.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:03 smithi122 ceph-mon[73495]: 1.1 scrub starts 2024-03-20T09:19:03.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:03 smithi122 ceph-mon[73495]: 1.1 scrub ok 2024-03-20T09:19:03.706 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:19:03.706 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:19:03.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:03.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:19:03.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:03 smithi026 ceph-mon[68309]: 1.1 scrub starts 2024-03-20T09:19:03.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:03 smithi026 ceph-mon[68309]: 1.1 scrub ok 2024-03-20T09:19:03.748 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:19:03.786 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:19:03.812 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:19:03.826 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:19:03.841 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:19:03.842 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:19:03.842 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 624, 2024-03-20T09:19:03.842 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "852fbb68-e696-11ee-95c9-87774f69a715", 2024-03-20T09:19:03.842 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-03-20T08:48:35.281088+0000", 2024-03-20T09:19:03.842 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-03-20T09:19:02.257250+0000", 2024-03-20T09:19:03.842 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2024-03-20T09:10:33.084151+0000", 2024-03-20T09:19:03.842 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2024-03-20T09:10:19.369629+0000", 2024-03-20T09:19:03.842 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-20T09:19:03.843 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5799936, 2024-03-20T09:19:03.843 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2024-03-20T09:19:03.843 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2024-03-20T09:19:03.843 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2024-03-20T09:19:03.843 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2024-03-20T09:19:03.843 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2024-03-20T09:19:03.843 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:19:03.843 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 27, 2024-03-20T09:19:03.843 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2024-03-20T09:19:03.843 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-20T09:19:03.843 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-20T09:19:03.843 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2024-03-20T09:19:03.844 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 5, 2024-03-20T09:19:03.844 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 12, 2024-03-20T09:19:03.844 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "luminous", 2024-03-20T09:19:03.844 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "luminous", 2024-03-20T09:19:03.844 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "squid", 2024-03-20T09:19:03.844 INFO:teuthology.orchestra.run.smithi026.stdout: "allow_crimson": false, 2024-03-20T09:19:03.844 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2024-03-20T09:19:03.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.844 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2024-03-20T09:19:03.844 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": ".mgr", 2024-03-20T09:19:03.844 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-03-20T08:52:17.442324+0000", 2024-03-20T09:19:03.844 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-03-20T09:19:03.844 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-03-20T09:19:03.844 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-03-20T09:19:03.845 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 3, 2024-03-20T09:19:03.845 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 2, 2024-03-20T09:19:03.845 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-03-20T09:19:03.845 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-03-20T09:19:03.845 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-03-20T09:19:03.845 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-03-20T09:19:03.845 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-20T09:19:03.846 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-03-20T09:19:03.846 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-03-20T09:19:03.846 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 31, 2024-03-20T09:19:03.846 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 31, 2024-03-20T09:19:03.846 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 31, 2024-03-20T09:19:03.846 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 31, 2024-03-20T09:19:03.846 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 31, 2024-03-20T09:19:03.846 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-03-20T09:19:03.846 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-03-20T09:19:03.846 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-03-20T09:19:03.846 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-03-20T09:19:03.846 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-03-20T09:19:03.846 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-03-20T09:19:03.847 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-03-20T09:19:03.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.847 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "360", 2024-03-20T09:19:03.847 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-03-20T09:19:03.847 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-20T09:19:03.847 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "358", 2024-03-20T09:19:03.847 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-03-20T09:19:03.847 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-03-20T09:19:03.847 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-03-20T09:19:03.847 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-03-20T09:19:03.847 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-03-20T09:19:03.847 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-03-20T09:19:03.847 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-03-20T09:19:03.848 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-03-20T09:19:03.848 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-03-20T09:19:03.848 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-03-20T09:19:03.848 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-03-20T09:19:03.848 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-03-20T09:19:03.848 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-03-20T09:19:03.848 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-03-20T09:19:03.848 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-03-20T09:19:03.849 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-20T09:19:03.849 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-20T09:19:03.849 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-20T09:19:03.849 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-03-20T09:19:03.849 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-03-20T09:19:03.849 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-03-20T09:19:03.849 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-03-20T09:19:03.849 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-03-20T09:19:03.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.849 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-03-20T09:19:03.850 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-03-20T09:19:03.850 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-03-20T09:19:03.850 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-03-20T09:19:03.850 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-03-20T09:19:03.850 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-03-20T09:19:03.850 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-03-20T09:19:03.850 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-03-20T09:19:03.850 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-03-20T09:19:03.850 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-03-20T09:19:03.850 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-03-20T09:19:03.850 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2024-03-20T09:19:03.850 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_max": 32, 2024-03-20T09:19:03.850 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2024-03-20T09:19:03.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.851 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-03-20T09:19:03.851 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": {} 2024-03-20T09:19:03.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.852 INFO:teuthology.orchestra.run.smithi026.stdout: "read_balance": { 2024-03-20T09:19:03.852 INFO:teuthology.orchestra.run.smithi026.stdout: "score_type": "Fair distribution", 2024-03-20T09:19:03.852 INFO:teuthology.orchestra.run.smithi026.stdout: "score_acting": 2.3399999141693115, 2024-03-20T09:19:03.852 INFO:teuthology.orchestra.run.smithi026.stdout: "score_stable": 2.3399999141693115, 2024-03-20T09:19:03.852 INFO:teuthology.orchestra.run.smithi026.stdout: "optimal_score": 0.82999998331069946, 2024-03-20T09:19:03.852 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_acting": 1.940000057220459, 2024-03-20T09:19:03.852 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_stable": 1.940000057220459, 2024-03-20T09:19:03.852 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity_weighted": 0.92150616645812988, 2024-03-20T09:19:03.852 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity": 0.92000001668930054, 2024-03-20T09:19:03.852 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-03-20T09:19:03.852 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.852 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 5, 2024-03-20T09:19:03.853 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "unique_pool_3", 2024-03-20T09:19:03.853 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-03-20T09:18:09.411082+0000", 2024-03-20T09:19:03.853 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-03-20T09:19:03.853 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-03-20T09:19:03.853 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-03-20T09:19:03.853 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 3, 2024-03-20T09:19:03.853 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 2, 2024-03-20T09:19:03.853 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-03-20T09:19:03.853 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-03-20T09:19:03.853 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-03-20T09:19:03.853 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-03-20T09:19:03.853 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-20T09:19:03.853 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-03-20T09:19:03.853 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-03-20T09:19:03.854 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 16, 2024-03-20T09:19:03.854 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 16, 2024-03-20T09:19:03.854 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 16, 2024-03-20T09:19:03.854 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 16, 2024-03-20T09:19:03.854 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 16, 2024-03-20T09:19:03.854 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-03-20T09:19:03.854 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-03-20T09:19:03.854 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-03-20T09:19:03.854 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-03-20T09:19:03.854 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-03-20T09:19:03.854 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-03-20T09:19:03.854 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-03-20T09:19:03.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.855 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "610", 2024-03-20T09:19:03.855 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-03-20T09:19:03.855 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-20T09:19:03.855 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2024-03-20T09:19:03.855 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-03-20T09:19:03.855 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-03-20T09:19:03.855 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-03-20T09:19:03.855 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-03-20T09:19:03.855 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-03-20T09:19:03.855 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-03-20T09:19:03.855 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-03-20T09:19:03.855 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-03-20T09:19:03.855 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-03-20T09:19:03.855 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-03-20T09:19:03.855 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-03-20T09:19:03.856 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-03-20T09:19:03.856 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-03-20T09:19:03.856 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-03-20T09:19:03.856 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-03-20T09:19:03.856 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-20T09:19:03.856 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-20T09:19:03.856 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-20T09:19:03.856 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-03-20T09:19:03.856 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-03-20T09:19:03.856 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-03-20T09:19:03.856 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-03-20T09:19:03.856 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-03-20T09:19:03.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.856 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-03-20T09:19:03.856 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-03-20T09:19:03.857 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-03-20T09:19:03.857 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-03-20T09:19:03.858 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-03-20T09:19:03.858 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-03-20T09:19:03.858 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-03-20T09:19:03.858 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-03-20T09:19:03.859 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-03-20T09:19:03.859 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-03-20T09:19:03.859 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-03-20T09:19:03.859 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2024-03-20T09:19:03.859 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-03-20T09:19:03.859 INFO:teuthology.orchestra.run.smithi026.stdout: "rados": {} 2024-03-20T09:19:03.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.860 INFO:teuthology.orchestra.run.smithi026.stdout: "read_balance": { 2024-03-20T09:19:03.860 INFO:teuthology.orchestra.run.smithi026.stdout: "score_type": "Fair distribution", 2024-03-20T09:19:03.860 INFO:teuthology.orchestra.run.smithi026.stdout: "score_acting": 2.2699999809265137, 2024-03-20T09:19:03.860 INFO:teuthology.orchestra.run.smithi026.stdout: "score_stable": 2.2699999809265137, 2024-03-20T09:19:03.860 INFO:teuthology.orchestra.run.smithi026.stdout: "optimal_score": 0.82999998331069946, 2024-03-20T09:19:03.860 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_acting": 1.8799999952316284, 2024-03-20T09:19:03.861 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_stable": 1.8799999952316284, 2024-03-20T09:19:03.861 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity_weighted": 0.92150616645812988, 2024-03-20T09:19:03.861 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity": 0.92000001668930054, 2024-03-20T09:19:03.861 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-03-20T09:19:03.861 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.861 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.861 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:19:03.862 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:19:03.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.862 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-03-20T09:19:03.862 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "ad3fd9c0-316f-4fe5-a63b-d480087c1729", 2024-03-20T09:19:03.862 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:19:03.862 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:19:03.862 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:19:03.862 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:19:03.863 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:19:03.863 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:19:03.863 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 8, 2024-03-20T09:19:03.863 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 605, 2024-03-20T09:19:03.863 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:19:03.863 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:19:03.863 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:19:03.863 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.863 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.863 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2024-03-20T09:19:03.863 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:19:03.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.864 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.864 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2024-03-20T09:19:03.864 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:19:03.864 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.864 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.864 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:19:03.864 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.864 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.864 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2024-03-20T09:19:03.864 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:19:03.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.864 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.865 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2024-03-20T09:19:03.865 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:19:03.865 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.865 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.865 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:19:03.865 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.866 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.866 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2024-03-20T09:19:03.866 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:19:03.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.866 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.866 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2024-03-20T09:19:03.866 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:19:03.867 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.867 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.867 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:19:03.867 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.867 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.867 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2024-03-20T09:19:03.867 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:19:03.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.867 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.867 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2024-03-20T09:19:03.868 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:19:03.868 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.868 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.868 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/2139805293", 2024-03-20T09:19:03.868 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/2139805293", 2024-03-20T09:19:03.868 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/2139805293", 2024-03-20T09:19:03.868 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/2139805293", 2024-03-20T09:19:03.868 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:19:03.868 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:19:03.868 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:19:03.868 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.869 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-03-20T09:19:03.869 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e85b94f5-2954-47a1-9c20-096f2606eb04", 2024-03-20T09:19:03.869 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:19:03.869 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:19:03.869 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:19:03.869 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:19:03.870 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:19:03.870 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:19:03.870 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2024-03-20T09:19:03.870 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 605, 2024-03-20T09:19:03.870 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:19:03.870 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:19:03.871 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:19:03.871 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.871 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.872 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2024-03-20T09:19:03.872 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:19:03.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.872 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.872 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2024-03-20T09:19:03.872 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:19:03.873 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.873 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.873 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:19:03.873 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.874 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.874 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2024-03-20T09:19:03.874 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:19:03.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.874 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.875 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2024-03-20T09:19:03.875 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:19:03.875 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.875 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.875 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:19:03.876 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.876 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.876 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2024-03-20T09:19:03.876 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:19:03.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.877 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.877 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2024-03-20T09:19:03.877 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:19:03.877 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.877 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.878 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:19:03.878 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.878 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.878 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2024-03-20T09:19:03.878 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:19:03.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.878 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.878 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2024-03-20T09:19:03.879 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:19:03.879 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.879 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.879 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/3853355134", 2024-03-20T09:19:03.879 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/3853355134", 2024-03-20T09:19:03.879 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/3853355134", 2024-03-20T09:19:03.880 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/3853355134", 2024-03-20T09:19:03.880 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:19:03.880 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:19:03.880 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:19:03.880 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.880 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-03-20T09:19:03.880 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "dd2b10f0-741c-47a4-b224-ac6bfe2c14ef", 2024-03-20T09:19:03.880 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:19:03.880 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:19:03.880 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:19:03.880 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:19:03.880 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:19:03.881 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:19:03.881 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 18, 2024-03-20T09:19:03.881 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 605, 2024-03-20T09:19:03.881 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:19:03.881 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:19:03.881 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:19:03.881 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.881 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.881 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2024-03-20T09:19:03.881 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:19:03.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.881 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.882 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2024-03-20T09:19:03.882 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:19:03.882 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.882 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.882 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:19:03.882 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.883 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.883 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2024-03-20T09:19:03.883 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:19:03.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.884 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.884 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2024-03-20T09:19:03.884 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:19:03.884 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.884 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.885 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:19:03.885 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.885 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.885 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2024-03-20T09:19:03.885 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:19:03.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.885 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.885 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2024-03-20T09:19:03.885 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:19:03.886 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.886 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.886 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:19:03.886 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.886 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.886 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2024-03-20T09:19:03.886 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:19:03.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.886 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.886 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2024-03-20T09:19:03.887 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:19:03.887 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.887 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.887 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/3993357967", 2024-03-20T09:19:03.887 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/3993357967", 2024-03-20T09:19:03.887 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/3993357967", 2024-03-20T09:19:03.887 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/3993357967", 2024-03-20T09:19:03.887 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:19:03.887 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:19:03.887 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:19:03.887 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.888 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-03-20T09:19:03.888 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1", 2024-03-20T09:19:03.888 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:19:03.888 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:19:03.888 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:19:03.888 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:19:03.888 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:19:03.888 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:19:03.888 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 26, 2024-03-20T09:19:03.888 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 605, 2024-03-20T09:19:03.888 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:19:03.888 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:19:03.888 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:19:03.889 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.889 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.889 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2024-03-20T09:19:03.889 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:19:03.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.889 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.889 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2024-03-20T09:19:03.889 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:19:03.889 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.889 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.890 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:19:03.890 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.890 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.890 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2024-03-20T09:19:03.890 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:19:03.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.891 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.891 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2024-03-20T09:19:03.891 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:19:03.891 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.891 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.891 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:19:03.891 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.891 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.891 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2024-03-20T09:19:03.891 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:19:03.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.892 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.892 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2024-03-20T09:19:03.892 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:19:03.892 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.892 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.892 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:19:03.892 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.892 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.892 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2024-03-20T09:19:03.892 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:19:03.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.892 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.893 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2024-03-20T09:19:03.893 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:19:03.893 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.893 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.893 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/4110225509", 2024-03-20T09:19:03.893 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/4110225509", 2024-03-20T09:19:03.893 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/4110225509", 2024-03-20T09:19:03.893 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/4110225509", 2024-03-20T09:19:03.893 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:19:03.893 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:19:03.893 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:19:03.893 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.894 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-03-20T09:19:03.894 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "f14e34d5-9829-42a3-9b25-31a0976c5ef7", 2024-03-20T09:19:03.894 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:19:03.894 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 0, 2024-03-20T09:19:03.894 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-03-20T09:19:03.894 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:19:03.894 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:19:03.894 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:19:03.894 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 32, 2024-03-20T09:19:03.894 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 366, 2024-03-20T09:19:03.894 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:19:03.894 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:19:03.895 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:19:03.895 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.895 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.895 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6800", 2024-03-20T09:19:03.895 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:19:03.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.896 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.896 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6801", 2024-03-20T09:19:03.896 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:19:03.896 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.896 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.896 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:19:03.896 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.897 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.897 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6802", 2024-03-20T09:19:03.897 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:19:03.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.897 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.897 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6803", 2024-03-20T09:19:03.897 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:19:03.897 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.898 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.898 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:19:03.898 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.899 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.899 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6806", 2024-03-20T09:19:03.899 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:19:03.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.899 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.900 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6807", 2024-03-20T09:19:03.900 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:19:03.900 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.900 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.900 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:19:03.901 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.901 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.901 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6804", 2024-03-20T09:19:03.901 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:19:03.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.902 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.902 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6805", 2024-03-20T09:19:03.902 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:19:03.902 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.902 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.903 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6801/477923058", 2024-03-20T09:19:03.903 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6803/477923058", 2024-03-20T09:19:03.903 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6807/477923058", 2024-03-20T09:19:03.903 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6805/477923058", 2024-03-20T09:19:03.903 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:19:03.903 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:19:03.904 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:19:03.904 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.904 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-03-20T09:19:03.904 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "cb431b6a-540d-4318-ae35-d379e1df0185", 2024-03-20T09:19:03.904 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:19:03.905 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:19:03.905 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:19:03.905 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:19:03.905 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 41, 2024-03-20T09:19:03.905 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 273, 2024-03-20T09:19:03.905 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 443, 2024-03-20T09:19:03.905 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 605, 2024-03-20T09:19:03.906 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 274, 2024-03-20T09:19:03.906 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:19:03.906 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:19:03.906 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.906 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.907 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6808", 2024-03-20T09:19:03.907 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:19:03.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.907 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.907 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6809", 2024-03-20T09:19:03.908 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:19:03.908 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.908 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.908 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:19:03.908 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.909 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.909 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6810", 2024-03-20T09:19:03.909 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:19:03.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.910 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.910 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6811", 2024-03-20T09:19:03.910 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:19:03.910 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.910 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.910 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:19:03.911 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.911 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.911 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6814", 2024-03-20T09:19:03.911 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:19:03.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.912 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.912 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6815", 2024-03-20T09:19:03.912 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:19:03.912 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.912 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.913 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:19:03.913 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.913 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.913 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6812", 2024-03-20T09:19:03.913 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:19:03.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.914 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.914 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6813", 2024-03-20T09:19:03.914 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:19:03.914 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.914 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.915 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6809/650416648", 2024-03-20T09:19:03.915 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6811/650416648", 2024-03-20T09:19:03.915 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6815/650416648", 2024-03-20T09:19:03.915 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6813/650416648", 2024-03-20T09:19:03.915 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:19:03.915 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:19:03.915 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:19:03.915 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.916 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2024-03-20T09:19:03.916 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "753ed830-e764-4ff3-9d9b-87e7fa748b81", 2024-03-20T09:19:03.916 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:19:03.916 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:19:03.916 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:19:03.916 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:19:03.916 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:19:03.916 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:19:03.916 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 51, 2024-03-20T09:19:03.916 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 605, 2024-03-20T09:19:03.916 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:19:03.917 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:19:03.917 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:19:03.917 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.917 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.917 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6816", 2024-03-20T09:19:03.917 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:19:03.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.917 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.917 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6817", 2024-03-20T09:19:03.917 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:19:03.917 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.917 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.918 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:19:03.918 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.918 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.918 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6818", 2024-03-20T09:19:03.918 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:19:03.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.918 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.918 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6819", 2024-03-20T09:19:03.918 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:19:03.918 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.918 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.919 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:19:03.919 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.919 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.919 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6822", 2024-03-20T09:19:03.919 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:19:03.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.919 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.919 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6823", 2024-03-20T09:19:03.919 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:19:03.919 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.919 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.920 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:19:03.920 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.920 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.920 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6820", 2024-03-20T09:19:03.920 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:19:03.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.920 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.921 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6821", 2024-03-20T09:19:03.921 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:19:03.921 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.921 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.921 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6817/3278301843", 2024-03-20T09:19:03.921 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6819/3278301843", 2024-03-20T09:19:03.921 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6823/3278301843", 2024-03-20T09:19:03.921 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6821/3278301843", 2024-03-20T09:19:03.921 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:19:03.921 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:19:03.921 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:19:03.921 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.922 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2024-03-20T09:19:03.922 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "55bcbd37-7a6f-4ab1-b5a9-91a951444d86", 2024-03-20T09:19:03.922 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:19:03.922 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:19:03.922 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:19:03.922 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 0.058074951171875, 2024-03-20T09:19:03.922 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:19:03.922 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:19:03.922 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 60, 2024-03-20T09:19:03.922 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 517, 2024-03-20T09:19:03.922 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:19:03.922 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:19:03.923 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:19:03.923 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.923 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.923 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6824", 2024-03-20T09:19:03.923 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:19:03.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.923 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.923 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6825", 2024-03-20T09:19:03.923 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:19:03.923 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.923 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.923 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:19:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6826", 2024-03-20T09:19:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:19:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6827", 2024-03-20T09:19:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:19:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:19:03.924 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6830", 2024-03-20T09:19:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:19:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6831", 2024-03-20T09:19:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:19:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:19:03.925 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.926 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.926 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6828", 2024-03-20T09:19:03.926 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:19:03.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.926 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.926 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6829", 2024-03-20T09:19:03.926 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:19:03.926 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6825/3152424001", 2024-03-20T09:19:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6827/3152424001", 2024-03-20T09:19:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6831/3152424001", 2024-03-20T09:19:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6829/3152424001", 2024-03-20T09:19:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:19:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:19:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:19:03.927 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 8, 2024-03-20T09:19:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "bc86248e-6bd0-4369-b8a2-e593bc2a610d", 2024-03-20T09:19:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:19:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:19:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:19:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:19:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:19:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:19:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 70, 2024-03-20T09:19:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 605, 2024-03-20T09:19:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:19:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:19:03.928 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:19:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6800", 2024-03-20T09:19:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:19:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6801", 2024-03-20T09:19:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:19:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:19:03.929 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6802", 2024-03-20T09:19:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:19:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6803", 2024-03-20T09:19:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:19:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.930 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:19:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6806", 2024-03-20T09:19:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:19:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6807", 2024-03-20T09:19:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:19:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.931 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:19:03.932 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.932 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.932 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6804", 2024-03-20T09:19:03.932 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:19:03.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.932 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.932 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6805", 2024-03-20T09:19:03.932 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:19:03.932 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.932 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.932 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6801/1309426740", 2024-03-20T09:19:03.932 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6803/1309426740", 2024-03-20T09:19:03.933 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6807/1309426740", 2024-03-20T09:19:03.933 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6805/1309426740", 2024-03-20T09:19:03.933 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:19:03.933 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:19:03.933 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:19:03.933 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.933 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 9, 2024-03-20T09:19:03.933 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b20a4aa-032a-4c09-ac75-153c554424e7", 2024-03-20T09:19:03.933 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:19:03.933 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 0, 2024-03-20T09:19:03.933 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-03-20T09:19:03.933 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:19:03.934 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:19:03.934 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:19:03.934 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 80, 2024-03-20T09:19:03.934 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 257, 2024-03-20T09:19:03.934 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:19:03.934 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:19:03.934 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:19:03.934 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.934 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.934 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6808", 2024-03-20T09:19:03.934 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:19:03.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.935 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.935 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6809", 2024-03-20T09:19:03.935 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:19:03.935 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.935 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.935 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:19:03.935 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.935 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.935 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6810", 2024-03-20T09:19:03.935 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:19:03.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.935 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.936 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6811", 2024-03-20T09:19:03.936 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:19:03.936 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.936 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.936 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:19:03.936 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.936 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.936 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6814", 2024-03-20T09:19:03.936 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:19:03.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.936 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.937 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6815", 2024-03-20T09:19:03.937 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:19:03.937 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.937 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.937 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:19:03.937 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.937 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.937 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6812", 2024-03-20T09:19:03.937 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:19:03.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.937 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.938 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6813", 2024-03-20T09:19:03.938 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:19:03.938 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.938 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.938 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6809/2953210076", 2024-03-20T09:19:03.938 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6811/2953210076", 2024-03-20T09:19:03.938 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6815/2953210076", 2024-03-20T09:19:03.938 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6813/2953210076", 2024-03-20T09:19:03.938 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:19:03.938 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:19:03.938 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:19:03.938 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.939 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 10, 2024-03-20T09:19:03.939 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e62c1fb3-5dd1-4d0d-997c-d20633d19361", 2024-03-20T09:19:03.939 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:19:03.939 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:19:03.939 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:19:03.939 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:19:03.939 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:19:03.939 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:19:03.939 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 89, 2024-03-20T09:19:03.939 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 605, 2024-03-20T09:19:03.939 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:19:03.939 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:19:03.940 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:19:03.940 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.940 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.940 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6816", 2024-03-20T09:19:03.940 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:19:03.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.940 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.940 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6817", 2024-03-20T09:19:03.940 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:19:03.940 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.940 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.941 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:19:03.941 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.941 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.941 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6818", 2024-03-20T09:19:03.941 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:19:03.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.941 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.941 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6819", 2024-03-20T09:19:03.941 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:19:03.941 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.941 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.942 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:19:03.942 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.942 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.942 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6822", 2024-03-20T09:19:03.942 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:19:03.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.942 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.942 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6823", 2024-03-20T09:19:03.942 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:19:03.942 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.942 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.943 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:19:03.943 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.943 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.943 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6820", 2024-03-20T09:19:03.943 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:19:03.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.943 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.943 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6821", 2024-03-20T09:19:03.943 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:19:03.943 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.943 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.943 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6817/409394340", 2024-03-20T09:19:03.944 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6819/409394340", 2024-03-20T09:19:03.944 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6823/409394340", 2024-03-20T09:19:03.944 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6821/409394340", 2024-03-20T09:19:03.944 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:19:03.944 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:19:03.944 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:19:03.944 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.944 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 11, 2024-03-20T09:19:03.944 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b83994d5-e5ef-48b0-9758-c26843cec3b6", 2024-03-20T09:19:03.944 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:19:03.944 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:19:03.944 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:19:03.945 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:19:03.945 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:19:03.945 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:19:03.945 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 96, 2024-03-20T09:19:03.945 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 605, 2024-03-20T09:19:03.945 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:19:03.945 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:19:03.945 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:19:03.945 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.945 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.945 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6824", 2024-03-20T09:19:03.945 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:19:03.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.946 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.946 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6825", 2024-03-20T09:19:03.946 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:19:03.946 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.946 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.946 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:19:03.946 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.946 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.946 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.946 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6826", 2024-03-20T09:19:03.946 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:19:03.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.946 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.946 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.947 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6827", 2024-03-20T09:19:03.947 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:19:03.947 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.947 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.947 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:19:03.947 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.947 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.947 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6830", 2024-03-20T09:19:03.947 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:19:03.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.947 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.947 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6831", 2024-03-20T09:19:03.948 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:19:03.948 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.948 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.948 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:19:03.948 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:19:03.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.948 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:19:03.948 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6828", 2024-03-20T09:19:03.948 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:19:03.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.948 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:19:03.948 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6829", 2024-03-20T09:19:03.949 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:19:03.949 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.949 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.949 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6825/2704069354", 2024-03-20T09:19:03.949 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6827/2704069354", 2024-03-20T09:19:03.949 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6831/2704069354", 2024-03-20T09:19:03.949 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6829/2704069354", 2024-03-20T09:19:03.949 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:19:03.949 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:19:03.949 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:19:03.949 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.949 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.949 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:19:03.949 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2024-03-20T09:19:03.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.950 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-03-20T09:19:03.950 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:19:03.950 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:19:03.950 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:19:03.950 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:19:03.950 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:19:03.950 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:17:46.912184+0000", 2024-03-20T09:19:03.950 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:19:03.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.950 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-03-20T09:19:03.950 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:19:03.950 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:19:03.950 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:19:03.951 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:19:03.951 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:19:03.951 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:17:42.613848+0000", 2024-03-20T09:19:03.951 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:19:03.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.951 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-03-20T09:19:03.951 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:19:03.951 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:19:03.951 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:19:03.951 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:19:03.951 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:19:03.951 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:18:28.075721+0000", 2024-03-20T09:19:03.951 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:19:03.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.952 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-03-20T09:19:03.952 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:19:03.952 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:19:03.952 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:19:03.952 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:19:03.952 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:19:03.952 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:18:50.415345+0000", 2024-03-20T09:19:03.952 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:19:03.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.952 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-03-20T09:19:03.952 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:19:03.952 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:19:03.953 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:19:03.953 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:19:03.953 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 65536, 2024-03-20T09:19:03.953 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:18:36.060981+0000", 2024-03-20T09:19:03.953 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:19:03.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.953 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-03-20T09:19:03.953 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:19:03.953 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:19:03.953 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:19:03.953 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:19:03.953 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:19:03.953 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:18:04.798647+0000", 2024-03-20T09:19:03.953 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 273 2024-03-20T09:19:03.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.954 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2024-03-20T09:19:03.954 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:19:03.954 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:19:03.954 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:19:03.954 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:19:03.954 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:19:03.954 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:18:25.255640+0000", 2024-03-20T09:19:03.954 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:19:03.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.954 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2024-03-20T09:19:03.954 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:19:03.955 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:19:03.955 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:19:03.955 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:19:03.955 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:19:03.955 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:18:22.820436+0000", 2024-03-20T09:19:03.955 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:19:03.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.955 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 8, 2024-03-20T09:19:03.955 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:19:03.955 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:19:03.955 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:19:03.955 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:19:03.955 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:19:03.955 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:18:41.755936+0000", 2024-03-20T09:19:03.956 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:19:03.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.956 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 9, 2024-03-20T09:19:03.956 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:19:03.956 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:19:03.956 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:19:03.956 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:19:03.956 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 34736, 2024-03-20T09:19:03.956 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:17:47.602111+0000", 2024-03-20T09:19:03.956 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:19:03.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.956 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 10, 2024-03-20T09:19:03.957 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:19:03.957 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:19:03.957 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:19:03.957 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:19:03.957 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:19:03.957 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:18:43.404674+0000", 2024-03-20T09:19:03.957 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:19:03.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.957 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 11, 2024-03-20T09:19:03.957 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:19:03.957 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:19:03.957 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:19:03.957 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:19:03.957 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:19:03.958 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:18:59.066448+0000", 2024-03-20T09:19:03.958 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:19:03.958 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.958 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:19:03.958 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [ 2024-03-20T09:19:03.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:03.958 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "1.1", 2024-03-20T09:19:03.958 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:19:03.958 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-03-20T09:19:03.958 INFO:teuthology.orchestra.run.smithi026.stdout: 11, 2024-03-20T09:19:03.958 INFO:teuthology.orchestra.run.smithi026.stdout: 5 2024-03-20T09:19:03.958 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:03.958 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.958 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:19:03.958 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2024-03-20T09:19:03.959 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_primaries": [], 2024-03-20T09:19:03.959 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2024-03-20T09:19:03.959 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2024-03-20T09:19:03.959 INFO:teuthology.orchestra.run.smithi026.stdout: "blocklist": { 2024-03-20T09:19:03.959 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/4291404191": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:19:03.959 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/642012566": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:19:03.959 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6801/1181459999": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:19:03.959 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/3984513952": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:19:03.959 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6800/1654945498": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:19:03.959 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6800/1181459999": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:19:03.959 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/137770981": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:19:03.959 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/1451237762": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:19:03.959 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/3938386631": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:19:03.959 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6801/1654945498": "2024-03-21T08:49:04.596754+0000" 2024-03-20T09:19:03.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.960 INFO:teuthology.orchestra.run.smithi026.stdout: "range_blocklist": {}, 2024-03-20T09:19:03.960 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2024-03-20T09:19:03.960 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2024-03-20T09:19:03.960 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2024-03-20T09:19:03.960 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2024-03-20T09:19:03.960 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2024-03-20T09:19:03.960 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2024-03-20T09:19:03.960 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2024-03-20T09:19:03.960 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:03.960 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2024-03-20T09:19:03.960 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2024-03-20T09:19:03.960 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2024-03-20T09:19:03.960 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2024-03-20T09:19:03.960 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {}, 2024-03-20T09:19:03.961 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": { 2024-03-20T09:19:03.961 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_enabled": false, 2024-03-20T09:19:03.961 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_bucket_count": 0, 2024-03-20T09:19:03.961 INFO:teuthology.orchestra.run.smithi026.stdout: "degraded_stretch_mode": 0, 2024-03-20T09:19:03.961 INFO:teuthology.orchestra.run.smithi026.stdout: "recovering_stretch_mode": 0, 2024-03-20T09:19:03.961 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_bucket": 0 2024-03-20T09:19:03.961 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:03.961 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:19:03.962 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:19:03.963 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:19:03.963 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:19:03.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:03.963 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:03.963 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:19:03.963 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:19:03.970 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:19:03.981 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:19:03.981 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:19:03.981 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:19:03.981 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:19:03.981 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:19:03.981 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:19:03.981 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:19:03.981 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:19:03.981 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:19:03.982 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:19:03.982 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:19:03.982 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:19:04.064 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:19:04.102 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:04.133 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:19:04.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:04.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:19:04.240 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:19:04.272 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:04.296 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:19:04.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:04.298 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:19:04.400 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:19:04.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:04 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1979364805' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:19:04.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:04 smithi115 ceph-mon[72476]: 1.5 deep-scrub starts 2024-03-20T09:19:04.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:04 smithi115 ceph-mon[72476]: 1.5 deep-scrub ok 2024-03-20T09:19:04.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:04 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2264206566' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:19:04.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:04 smithi115 ceph-mon[72476]: pgmap v1498: 47 pgs: 47 active+clean; 4.9 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-03-20T09:19:04.460 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:04.494 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:19:04.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:04.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:19:04.598 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:19:04.626 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 44 16 84580 84564 120.104 86.1875 0.00815254 0.0083219 2024-03-20T09:19:04.630 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:04.661 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:19:04.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:04.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:19:04.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:04 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1979364805' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:19:04.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:04 smithi122 ceph-mon[73495]: 1.5 deep-scrub starts 2024-03-20T09:19:04.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:04 smithi122 ceph-mon[73495]: 1.5 deep-scrub ok 2024-03-20T09:19:04.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:04 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2264206566' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:19:04.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:04 smithi122 ceph-mon[73495]: pgmap v1498: 47 pgs: 47 active+clean; 4.9 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-03-20T09:19:04.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:04 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1979364805' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:19:04.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:04 smithi026 ceph-mon[68309]: 1.5 deep-scrub starts 2024-03-20T09:19:04.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:04 smithi026 ceph-mon[68309]: 1.5 deep-scrub ok 2024-03-20T09:19:04.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:04 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2264206566' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:19:04.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:04 smithi026 ceph-mon[68309]: pgmap v1498: 47 pgs: 47 active+clean; 4.9 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-03-20T09:19:04.766 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:19:04.798 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:04.829 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:19:04.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:04.834 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:19:04.935 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:19:04.977 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:05.006 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:19:05.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:05.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:19:05.112 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:19:05.164 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:05.329 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:05 smithi115 ceph-mon[72476]: 1.10 scrub starts 2024-03-20T09:19:05.329 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:05 smithi115 ceph-mon[72476]: 1.10 scrub ok 2024-03-20T09:19:05.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:05 smithi026 ceph-mon[68309]: 1.10 scrub starts 2024-03-20T09:19:05.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:05 smithi026 ceph-mon[68309]: 1.10 scrub ok 2024-03-20T09:19:05.626 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 45 16 86001 85985 119.408 88.8125 0.00348521 0.00836643 2024-03-20T09:19:05.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:05 smithi122 ceph-mon[73495]: 1.10 scrub starts 2024-03-20T09:19:05.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:05 smithi122 ceph-mon[73495]: 1.10 scrub ok 2024-03-20T09:19:05.935 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:19:05.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:05.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:19:06.045 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:19:06.084 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:06.111 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:19:06.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:06.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:19:06.216 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:19:06.254 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:06.283 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:19:06.285 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:19:06.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:06.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:19:06.330 INFO:tasks.thrashosds.thrasher:j is {'epoch': 624, 'fsid': '852fbb68-e696-11ee-95c9-87774f69a715', 'created': '2024-03-20T08:48:35.281088+0000', 'modified': '2024-03-20T09:19:02.257250+0000', 'last_up_change': '2024-03-20T09:10:33.084151+0000', 'last_in_change': '2024-03-20T09:10:19.369629+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 5, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-20T08:52:17.442324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '360', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '358', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.3399999141693115, 'score_stable': 2.3399999141693115, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 1.940000057220459, 'raw_score_stable': 1.940000057220459, 'primary_affinity_weighted': 0.9215061664581299, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 5, 'pool_name': 'unique_pool_3', 'create_time': '2024-03-20T09:18:09.411082+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '610', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.2699999809265137, 'score_stable': 2.2699999809265137, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9215061664581299, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}], 'osds': [{'osd': 0, 'uuid': 'ad3fd9c0-316f-4fe5-a63b-d480087c1729', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 605, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6802', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6803', 'nonce': 2139805293}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6804', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6805', 'nonce': 2139805293}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6808', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6809', 'nonce': 2139805293}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6806', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6807', 'nonce': 2139805293}]}, 'public_addr': '172.21.15.26:6803/2139805293', 'cluster_addr': '172.21.15.26:6805/2139805293', 'heartbeat_back_addr': '172.21.15.26:6809/2139805293', 'heartbeat_front_addr': '172.21.15.26:6807/2139805293', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'e85b94f5-2954-47a1-9c20-096f2606eb04', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 605, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6810', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6811', 'nonce': 3853355134}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6812', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6813', 'nonce': 3853355134}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6816', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6817', 'nonce': 3853355134}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6814', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6815', 'nonce': 3853355134}]}, 'public_addr': '172.21.15.26:6811/3853355134', 'cluster_addr': '172.21.15.26:6813/3853355134', 'heartbeat_back_addr': '172.21.15.26:6817/3853355134', 'heartbeat_front_addr': '172.21.15.26:6815/3853355134', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'dd2b10f0-741c-47a4-b224-ac6bfe2c14ef', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 605, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6818', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6819', 'nonce': 3993357967}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6820', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6821', 'nonce': 3993357967}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6824', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6825', 'nonce': 3993357967}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6822', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6823', 'nonce': 3993357967}]}, 'public_addr': '172.21.15.26:6819/3993357967', 'cluster_addr': '172.21.15.26:6821/3993357967', 'heartbeat_back_addr': '172.21.15.26:6825/3993357967', 'heartbeat_front_addr': '172.21.15.26:6823/3993357967', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 605, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6826', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6827', 'nonce': 4110225509}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6828', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6829', 'nonce': 4110225509}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6832', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6833', 'nonce': 4110225509}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6830', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6831', 'nonce': 4110225509}]}, 'public_addr': '172.21.15.26:6827/4110225509', 'cluster_addr': '172.21.15.26:6829/4110225509', 'heartbeat_back_addr': '172.21.15.26:6833/4110225509', 'heartbeat_front_addr': '172.21.15.26:6831/4110225509', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'f14e34d5-9829-42a3-9b25-31a0976c5ef7', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 366, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6800', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6801', 'nonce': 477923058}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6802', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6803', 'nonce': 477923058}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6806', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6807', 'nonce': 477923058}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6804', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6805', 'nonce': 477923058}]}, 'public_addr': '172.21.15.115:6801/477923058', 'cluster_addr': '172.21.15.115:6803/477923058', 'heartbeat_back_addr': '172.21.15.115:6807/477923058', 'heartbeat_front_addr': '172.21.15.115:6805/477923058', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'cb431b6a-540d-4318-ae35-d379e1df0185', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 41, 'last_clean_end': 273, 'up_from': 443, 'up_thru': 605, 'down_at': 274, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6808', 'nonce': 650416648}, {'type': 'v1', 'addr': '172.21.15.115:6809', 'nonce': 650416648}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6810', 'nonce': 650416648}, {'type': 'v1', 'addr': '172.21.15.115:6811', 'nonce': 650416648}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6814', 'nonce': 650416648}, {'type': 'v1', 'addr': '172.21.15.115:6815', 'nonce': 650416648}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6812', 'nonce': 650416648}, {'type': 'v1', 'addr': '172.21.15.115:6813', 'nonce': 650416648}]}, 'public_addr': '172.21.15.115:6809/650416648', 'cluster_addr': '172.21.15.115:6811/650416648', 'heartbeat_back_addr': '172.21.15.115:6815/650416648', 'heartbeat_front_addr': '172.21.15.115:6813/650416648', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '753ed830-e764-4ff3-9d9b-87e7fa748b81', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 605, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6816', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6817', 'nonce': 3278301843}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6818', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6819', 'nonce': 3278301843}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6822', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6823', 'nonce': 3278301843}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6820', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6821', 'nonce': 3278301843}]}, 'public_addr': '172.21.15.115:6817/3278301843', 'cluster_addr': '172.21.15.115:6819/3278301843', 'heartbeat_back_addr': '172.21.15.115:6823/3278301843', 'heartbeat_front_addr': '172.21.15.115:6821/3278301843', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '55bcbd37-7a6f-4ab1-b5a9-91a951444d86', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.058074951171875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 517, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6824', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6825', 'nonce': 3152424001}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6826', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6827', 'nonce': 3152424001}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6830', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6831', 'nonce': 3152424001}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6828', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6829', 'nonce': 3152424001}]}, 'public_addr': '172.21.15.115:6825/3152424001', 'cluster_addr': '172.21.15.115:6827/3152424001', 'heartbeat_back_addr': '172.21.15.115:6831/3152424001', 'heartbeat_front_addr': '172.21.15.115:6829/3152424001', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'bc86248e-6bd0-4369-b8a2-e593bc2a610d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 70, 'up_thru': 605, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6800', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6801', 'nonce': 1309426740}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6802', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6803', 'nonce': 1309426740}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6806', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6807', 'nonce': 1309426740}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6804', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6805', 'nonce': 1309426740}]}, 'public_addr': '172.21.15.122:6801/1309426740', 'cluster_addr': '172.21.15.122:6803/1309426740', 'heartbeat_back_addr': '172.21.15.122:6807/1309426740', 'heartbeat_front_addr': '172.21.15.122:6805/1309426740', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '9b20a4aa-032a-4c09-ac75-153c554424e7', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 257, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6808', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6809', 'nonce': 2953210076}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6810', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6811', 'nonce': 2953210076}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6814', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6815', 'nonce': 2953210076}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6812', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6813', 'nonce': 2953210076}]}, 'public_addr': '172.21.15.122:6809/2953210076', 'cluster_addr': '172.21.15.122:6811/2953210076', 'heartbeat_back_addr': '172.21.15.122:6815/2953210076', 'heartbeat_front_addr': '172.21.15.122:6813/2953210076', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'e62c1fb3-5dd1-4d0d-997c-d20633d19361', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 89, 'up_thru': 605, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6816', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6817', 'nonce': 409394340}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6818', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6819', 'nonce': 409394340}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6822', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6823', 'nonce': 409394340}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6820', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6821', 'nonce': 409394340}]}, 'public_addr': '172.21.15.122:6817/409394340', 'cluster_addr': '172.21.15.122:6819/409394340', 'heartbeat_back_addr': '172.21.15.122:6823/409394340', 'heartbeat_front_addr': '172.21.15.122:6821/409394340', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'b83994d5-e5ef-48b0-9758-c26843cec3b6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 96, 'up_thru': 605, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6824', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6825', 'nonce': 2704069354}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6826', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6827', 'nonce': 2704069354}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6830', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6831', 'nonce': 2704069354}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6828', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6829', 'nonce': 2704069354}]}, 'public_addr': '172.21.15.122:6825/2704069354', 'cluster_addr': '172.21.15.122:6827/2704069354', 'heartbeat_back_addr': '172.21.15.122:6831/2704069354', 'heartbeat_front_addr': '172.21.15.122:6829/2704069354', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:17:46.912184+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:17:42.613848+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:18:28.075721+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:18:50.415345+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-20T09:18:36.060981+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:18:04.798647+0000', 'dead_epoch': 273}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:18:25.255640+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:18:22.820436+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:18:41.755936+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 34736, 'last_purged_snaps_scrub': '2024-03-20T09:17:47.602111+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:18:43.404674+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:18:59.066448+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.1', 'osds': [6, 11, 5]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.26:0/4291404191': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/642012566': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:6801/1181459999': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/3984513952': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6800/1654945498': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6800/1181459999': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/137770981': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:0/1451237762': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/3938386631': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6801/1654945498': '2024-03-21T08:49:04.596754+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-20T09:19:06.330 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:19:06.354 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:06 smithi115 ceph-mon[72476]: 1.0 scrub starts 2024-03-20T09:19:06.354 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:06 smithi115 ceph-mon[72476]: 1.0 scrub ok 2024-03-20T09:19:06.355 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:06 smithi115 ceph-mon[72476]: pgmap v1499: 47 pgs: 47 active+clean; 5.1 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-03-20T09:19:06.387 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:19:06.462 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:06.589 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:19:06.595 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:06 smithi026 ceph-mon[68309]: 1.0 scrub starts 2024-03-20T09:19:06.595 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:06 smithi026 ceph-mon[68309]: 1.0 scrub ok 2024-03-20T09:19:06.596 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:06 smithi026 ceph-mon[68309]: pgmap v1499: 47 pgs: 47 active+clean; 5.1 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-03-20T09:19:06.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:06.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:19:06.626 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 46 16 87229 87213 118.481 76.75 0.00323611 0.00843595 2024-03-20T09:19:06.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:06 smithi122 ceph-mon[73495]: 1.0 scrub starts 2024-03-20T09:19:06.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:06 smithi122 ceph-mon[73495]: 1.0 scrub ok 2024-03-20T09:19:06.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:06 smithi122 ceph-mon[73495]: pgmap v1499: 47 pgs: 47 active+clean; 5.1 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-03-20T09:19:06.698 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:19:06.760 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:07.627 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 47 16 88532 88516 117.693 81.4375 0.00320383 0.00848548 2024-03-20T09:19:07.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:07 smithi115 ceph-mon[72476]: 1.c scrub starts 2024-03-20T09:19:07.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:07 smithi115 ceph-mon[72476]: 1.c scrub ok 2024-03-20T09:19:07.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:07 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:19:07.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:07 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:19:07.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:07 smithi115 ceph-mon[72476]: osdmap e625: 12 total, 12 up, 10 in 2024-03-20T09:19:07.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:07 smithi115 ceph-mon[72476]: osdmap e626: 12 total, 12 up, 10 in 2024-03-20T09:19:07.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:07 smithi122 ceph-mon[73495]: 1.c scrub starts 2024-03-20T09:19:07.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:07 smithi122 ceph-mon[73495]: 1.c scrub ok 2024-03-20T09:19:07.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:07 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:19:07.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:07 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:19:07.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:07 smithi122 ceph-mon[73495]: osdmap e625: 12 total, 12 up, 10 in 2024-03-20T09:19:07.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:07 smithi122 ceph-mon[73495]: osdmap e626: 12 total, 12 up, 10 in 2024-03-20T09:19:07.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:07 smithi026 ceph-mon[68309]: 1.c scrub starts 2024-03-20T09:19:07.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:07 smithi026 ceph-mon[68309]: 1.c scrub ok 2024-03-20T09:19:07.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:07 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:19:07.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:07 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:19:07.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:07 smithi026 ceph-mon[68309]: osdmap e625: 12 total, 12 up, 10 in 2024-03-20T09:19:07.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:07 smithi026 ceph-mon[68309]: osdmap e626: 12 total, 12 up, 10 in 2024-03-20T09:19:07.929 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:19:08.238 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:19:08.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:08.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:19:08.294 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:19:08.352 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:19:08.396 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:08.531 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:19:08.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:08.538 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:08 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:19:08.538 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:08 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:19:08.538 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:08 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3802813824' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:19:08.538 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:08 smithi026 ceph-mon[68309]: pgmap v1502: 47 pgs: 47 active+clean; 5.3 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 100 MiB/s wr, 1.61k op/s 2024-03-20T09:19:08.538 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:08 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:19:08.538 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:08 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3802813824' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:19:08.538 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:08 smithi026 ceph-mon[68309]: osdmap e627: 12 total, 12 up, 10 in 2024-03-20T09:19:08.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:19:08.626 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 48 16 89699 89683 116.76 72.9375 0.0125449 0.00856127 2024-03-20T09:19:08.640 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:19:08.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:08 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:19:08.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:08 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:19:08.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:08 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3802813824' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:19:08.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:08 smithi115 ceph-mon[72476]: pgmap v1502: 47 pgs: 47 active+clean; 5.3 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 100 MiB/s wr, 1.61k op/s 2024-03-20T09:19:08.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:08 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:19:08.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:08 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3802813824' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:19:08.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:08 smithi115 ceph-mon[72476]: osdmap e627: 12 total, 12 up, 10 in 2024-03-20T09:19:08.670 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:08 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:19:08.670 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:08 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:19:08.670 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:08 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3802813824' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:19:08.670 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:08 smithi122 ceph-mon[73495]: pgmap v1502: 47 pgs: 47 active+clean; 5.3 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 100 MiB/s wr, 1.61k op/s 2024-03-20T09:19:08.670 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:08 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:19:08.671 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:08 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3802813824' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:19:08.671 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:08 smithi122 ceph-mon[73495]: osdmap e627: 12 total, 12 up, 10 in 2024-03-20T09:19:08.673 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:08.696 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:19:08.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:08.701 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:19:08.802 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:19:08.861 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:09.085 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:19:09.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:09.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:19:09.190 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:19:09.240 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:09.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:09 smithi026 ceph-mon[68309]: 1.1c scrub starts 2024-03-20T09:19:09.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:09 smithi026 ceph-mon[68309]: 1.1c scrub ok 2024-03-20T09:19:09.626 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 49 16 91108 91092 116.174 88.0625 0.00906066 0.00860392 2024-03-20T09:19:09.649 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:19:09.652 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:19:09.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:09.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:09 smithi115 ceph-mon[72476]: 1.1c scrub starts 2024-03-20T09:19:09.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:09 smithi115 ceph-mon[72476]: 1.1c scrub ok 2024-03-20T09:19:09.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:19:09.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:09 smithi122 ceph-mon[73495]: 1.1c scrub starts 2024-03-20T09:19:09.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:09 smithi122 ceph-mon[73495]: 1.1c scrub ok 2024-03-20T09:19:09.759 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:19:09.802 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:09.823 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:19:09.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:09.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:19:09.930 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:19:09.972 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:10.001 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:19:10.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:10.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:19:10.108 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:19:10.179 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:10.243 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:19:10.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:10.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:19:10.349 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:19:10.364 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:19:10.380 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:10.402 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:19:10.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:10.405 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:19:10.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:10 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:19:10.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:10 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:19:10.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:10 smithi026 ceph-mon[68309]: pgmap v1504: 47 pgs: 47 active+clean; 5.3 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-03-20T09:19:10.472 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:19:10.472 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:19:10.506 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:19:10.538 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:10.562 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:19:10.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:10.567 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:19:10.627 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 50 16 92358 92342 115.413 78.125 0.0063547 0.00866084 2024-03-20T09:19:10.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:10 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:19:10.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:10 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:19:10.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:10 smithi115 ceph-mon[72476]: pgmap v1504: 47 pgs: 47 active+clean; 5.3 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-03-20T09:19:10.667 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:19:10.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:10 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:19:10.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:10 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:19:10.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:10 smithi122 ceph-mon[73495]: pgmap v1504: 47 pgs: 47 active+clean; 5.3 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-03-20T09:19:10.701 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:10.726 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:19:10.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:10.730 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:19:10.831 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:19:10.901 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:11.627 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 51 15 93685 93670 114.778 83 0.00371865 0.00870801 2024-03-20T09:19:11.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:11 smithi115 ceph-mon[72476]: osdmap e628: 12 total, 12 up, 10 in 2024-03-20T09:19:11.693 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:19:11.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:11 smithi122 ceph-mon[73495]: osdmap e628: 12 total, 12 up, 10 in 2024-03-20T09:19:11.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:11 smithi026 ceph-mon[68309]: osdmap e628: 12 total, 12 up, 10 in 2024-03-20T09:19:12.279 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:19:12.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:12.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:19:12.387 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:19:12.429 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:12.458 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:19:12.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:12.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:19:12.566 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:19:12.589 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:12 smithi026 ceph-mon[68309]: pgmap v1506: 47 pgs: 47 active+clean; 5.5 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-03-20T09:19:12.617 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:12.626 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 52 16 94879 94863 114.004 74.5625 0.00751717 0.00876788 2024-03-20T09:19:12.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:12 smithi115 ceph-mon[72476]: pgmap v1506: 47 pgs: 47 active+clean; 5.5 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-03-20T09:19:12.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:12 smithi122 ceph-mon[73495]: pgmap v1506: 47 pgs: 47 active+clean; 5.5 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-03-20T09:19:13.522 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-20T09:19:13.627 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 53 16 96156 96140 113.359 79.8125 0.00689115 0.00881706 2024-03-20T09:19:13.794 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:19:13.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:13.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:19:13.905 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:19:13.935 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:13.961 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:19:13.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:13.964 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:19:14.065 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:19:14.102 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:14.127 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:19:14.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:14.135 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:19:14.236 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:19:14.286 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:14.424 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:19:14.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:14.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:19:14.532 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:19:14.569 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:14.598 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:19:14.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:14.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:19:14.627 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 54 16 97417 97401 112.719 78.8125 0.0154245 0.00886737 2024-03-20T09:19:14.706 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:19:14.769 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:15.381 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:15 smithi026 ceph-mon[68309]: pgmap v1507: 47 pgs: 47 active+clean; 5.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-03-20T09:19:15.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:15 smithi115 ceph-mon[72476]: pgmap v1507: 47 pgs: 47 active+clean; 5.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-03-20T09:19:15.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:15 smithi122 ceph-mon[73495]: pgmap v1507: 47 pgs: 47 active+clean; 5.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-03-20T09:19:15.627 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 55 16 98669 98653 112.092 78.25 0.00927145 0.00891678 2024-03-20T09:19:16.040 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:19:16.040 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:19:16.040 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:19:16.076 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:19:16.086 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:19:16.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:16.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:19:16.126 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:19:16.187 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:19:16.187 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:19:16.192 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:19:16.211 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:19:16.248 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:16.251 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:19:16.287 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:19:16.314 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:19:16.344 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:19:16.352 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:19:16.352 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:19:16.352 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:19:16.352 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:19:16.352 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:19:16.352 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:19:16.352 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:19:16.352 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:19:16.352 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:19:16.352 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:19:16.352 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:19:16.352 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:19:16.385 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:16 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3296407829' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:19:16.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:16 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3296407829' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:19:16.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:16 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3296407829' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:19:16.627 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 56 16 99872 99856 111.433 75.1875 0.0169384 0.0089693 2024-03-20T09:19:17.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:17 smithi115 ceph-mon[72476]: pgmap v1508: 47 pgs: 47 active+clean; 5.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-03-20T09:19:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:17 smithi122 ceph-mon[73495]: pgmap v1508: 47 pgs: 47 active+clean; 5.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-03-20T09:19:17.464 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:19:17.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:17 smithi026 ceph-mon[68309]: pgmap v1508: 47 pgs: 47 active+clean; 5.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-03-20T09:19:17.489 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:19:17.627 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 57 16 101101 101085 110.826 76.8125 0.013106 0.00901844 2024-03-20T09:19:17.821 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:19:17.838 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:19:17.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:17.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:19:17.944 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:19:17.977 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:18.000 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:19:18.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:18.004 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:19:18.105 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:19:18.152 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:18.235 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:19:18.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:18.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:19:18.285 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:19:18.290 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":1508,"stamp":"2024-03-20T09:19:16.027106+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6279070240,"num_objects":95804,"num_object_clones":0,"num_object_copies":287412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95802,"num_write_kb":6131328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100050,"ondisk_log_size":100050,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":22453904,"kb_used_data":18156080,"kb_used_omap":195,"kb_used_meta":4297468,"kb_avail":914997616,"statfs":{"total":959950356480,"available":936957558784,"internally_reserved":0,"allocated":18591825920,"data_stored":18588158914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":200482,"internal_metadata":4400607454},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":597622784,"num_objects":9119,"num_object_clones":0,"num_object_copies":27357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9119,"num_write_kb":583616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001955"},"pg_stats":[{"pgid":"1.1e","version":"20'137","reported_seq":657,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:10.402945+0000","last_change":"2024-03-20T09:18:05.872016+0000","last_active":"2024-03-20T09:19:10.402945+0000","last_peered":"2024-03-20T09:19:10.402945+0000","last_clean":"2024-03-20T09:19:10.402945+0000","last_became_active":"2024-03-20T09:13:20.071655+0000","last_became_peered":"2024-03-20T09:13:20.071655+0000","last_unstale":"2024-03-20T09:19:10.402945+0000","last_undegraded":"2024-03-20T09:19:10.402945+0000","last_fullsized":"2024-03-20T09:19:10.402945+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:18:05.871879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:18:05.871879+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:34.328518+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'137","reported_seq":824,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:10.397208+0000","last_change":"2024-03-20T09:17:45.497854+0000","last_active":"2024-03-20T09:19:10.397208+0000","last_peered":"2024-03-20T09:19:10.397208+0000","last_clean":"2024-03-20T09:19:10.397208+0000","last_became_active":"2024-03-20T09:07:42.013245+0000","last_became_peered":"2024-03-20T09:07:42.013245+0000","last_unstale":"2024-03-20T09:19:10.397208+0000","last_undegraded":"2024-03-20T09:19:10.397208+0000","last_fullsized":"2024-03-20T09:19:10.397208+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:45.497703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:17:45.497703+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"20'137","reported_seq":1274,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:10.413426+0000","last_change":"2024-03-20T09:19:08.132903+0000","last_active":"2024-03-20T09:19:10.413426+0000","last_peered":"2024-03-20T09:19:10.413426+0000","last_clean":"2024-03-20T09:19:10.413426+0000","last_became_active":"2024-03-20T09:07:41.961622+0000","last_became_peered":"2024-03-20T09:07:41.961622+0000","last_unstale":"2024-03-20T09:19:10.413426+0000","last_undegraded":"2024-03-20T09:19:10.413426+0000","last_fullsized":"2024-03-20T09:19:10.413426+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:19:08.132770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:19:08.132770+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:20:19.752464+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.19","version":"20'137","reported_seq":1335,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:10.393865+0000","last_change":"2024-03-20T09:18:40.053646+0000","last_active":"2024-03-20T09:19:10.393865+0000","last_peered":"2024-03-20T09:19:10.393865+0000","last_clean":"2024-03-20T09:19:10.393865+0000","last_became_active":"2024-03-20T09:07:41.962680+0000","last_became_peered":"2024-03-20T09:07:41.962680+0000","last_unstale":"2024-03-20T09:19:10.393865+0000","last_undegraded":"2024-03-20T09:19:10.393865+0000","last_fullsized":"2024-03-20T09:19:10.393865+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:18:40.053497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:18:40.053497+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:20:04.818734+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"20'137","reported_seq":797,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:10.396530+0000","last_change":"2024-03-20T09:18:02.925001+0000","last_active":"2024-03-20T09:19:10.396530+0000","last_peered":"2024-03-20T09:19:10.396530+0000","last_clean":"2024-03-20T09:19:10.396530+0000","last_became_active":"2024-03-20T09:10:35.277481+0000","last_became_peered":"2024-03-20T09:10:35.277481+0000","last_unstale":"2024-03-20T09:19:10.396530+0000","last_undegraded":"2024-03-20T09:19:10.396530+0000","last_fullsized":"2024-03-20T09:19:10.396530+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:18:02.924847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:18:02.924847+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:10.438683+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1b","version":"20'137","reported_seq":1103,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:10.413435+0000","last_change":"2024-03-20T09:18:39.119126+0000","last_active":"2024-03-20T09:19:10.413435+0000","last_peered":"2024-03-20T09:19:10.413435+0000","last_clean":"2024-03-20T09:19:10.413435+0000","last_became_active":"2024-03-20T09:07:41.962733+0000","last_became_peered":"2024-03-20T09:07:41.962733+0000","last_unstale":"2024-03-20T09:19:10.413435+0000","last_undegraded":"2024-03-20T09:19:10.413435+0000","last_fullsized":"2024-03-20T09:19:10.413435+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:18:39.119006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:18:39.119006+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:43.420972+0000","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"5.c","version":"628'5921","reported_seq":5976,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:12.597306+0000","last_change":"2024-03-20T09:18:11.083767+0000","last_active":"2024-03-20T09:19:12.597306+0000","last_peered":"2024-03-20T09:19:12.597306+0000","last_clean":"2024-03-20T09:19:12.597306+0000","last_became_active":"2024-03-20T09:18:11.083335+0000","last_became_peered":"2024-03-20T09:18:11.083335+0000","last_unstale":"2024-03-20T09:19:12.597306+0000","last_undegraded":"2024-03-20T09:19:12.597306+0000","last_fullsized":"2024-03-20T09:19:12.597306+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":5921,"log_dups_size":0,"ondisk_log_size":5921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:14.815713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388038656,"num_objects":5921,"num_object_clones":0,"num_object_copies":17763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5921,"num_write_kb":378944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"20'137","reported_seq":797,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:10.396599+0000","last_change":"2024-03-20T09:17:48.865046+0000","last_active":"2024-03-20T09:19:10.396599+0000","last_peered":"2024-03-20T09:19:10.396599+0000","last_clean":"2024-03-20T09:19:10.396599+0000","last_became_active":"2024-03-20T09:10:35.215772+0000","last_became_peered":"2024-03-20T09:10:35.215772+0000","last_unstale":"2024-03-20T09:19:10.396599+0000","last_undegraded":"2024-03-20T09:19:10.396599+0000","last_fullsized":"2024-03-20T09:19:10.396599+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:48.864925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:17:48.864925+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:13.258032+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"628'6202","reported_seq":6258,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:15.128867+0000","last_change":"2024-03-20T09:18:11.523337+0000","last_active":"2024-03-20T09:19:15.128867+0000","last_peered":"2024-03-20T09:19:15.128867+0000","last_clean":"2024-03-20T09:19:15.128867+0000","last_became_active":"2024-03-20T09:18:11.523040+0000","last_became_peered":"2024-03-20T09:18:11.523040+0000","last_unstale":"2024-03-20T09:19:15.128867+0000","last_undegraded":"2024-03-20T09:19:15.128867+0000","last_fullsized":"2024-03-20T09:19:15.128867+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":6202,"log_dups_size":0,"ondisk_log_size":6202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:39.162942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406454272,"num_objects":6202,"num_object_clones":0,"num_object_copies":18606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6202,"num_write_kb":396928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"20'137","reported_seq":1356,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:10.404666+0000","last_change":"2024-03-20T09:18:37.405968+0000","last_active":"2024-03-20T09:19:10.404666+0000","last_peered":"2024-03-20T09:19:10.404666+0000","last_clean":"2024-03-20T09:19:10.404666+0000","last_became_active":"2024-03-20T09:10:34.099414+0000","last_became_peered":"2024-03-20T09:10:34.099414+0000","last_unstale":"2024-03-20T09:19:10.404666+0000","last_undegraded":"2024-03-20T09:19:10.404666+0000","last_fullsized":"2024-03-20T09:19:10.404666+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:18:37.405832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:18:37.405832+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:44.985164+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"628'5868","reported_seq":5920,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:11.681444+0000","last_change":"2024-03-20T09:18:11.081562+0000","last_active":"2024-03-20T09:19:11.681444+0000","last_peered":"2024-03-20T09:19:11.681444+0000","last_clean":"2024-03-20T09:19:11.681444+0000","last_became_active":"2024-03-20T09:18:11.081322+0000","last_became_peered":"2024-03-20T09:18:11.081322+0000","last_unstale":"2024-03-20T09:19:11.681444+0000","last_undegraded":"2024-03-20T09:19:11.681444+0000","last_fullsized":"2024-03-20T09:19:11.681444+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":5868,"log_dups_size":0,"ondisk_log_size":5868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:38.316484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384565248,"num_objects":5868,"num_object_clones":0,"num_object_copies":17604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5868,"num_write_kb":375552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.a","version":"20'137","reported_seq":1162,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:10.402982+0000","last_change":"2024-03-20T09:18:44.538665+0000","last_active":"2024-03-20T09:19:10.402982+0000","last_peered":"2024-03-20T09:19:10.402982+0000","last_clean":"2024-03-20T09:19:10.402982+0000","last_became_active":"2024-03-20T09:07:52.618107+0000","last_became_peered":"2024-03-20T09:07:52.618107+0000","last_unstale":"2024-03-20T09:19:10.402982+0000","last_undegraded":"2024-03-20T09:19:10.402982+0000","last_fullsized":"2024-03-20T09:19:10.402982+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:18:44.538510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:18:44.538510+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:57.058581+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"628'5811","reported_seq":5867,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:11.236995+0000","last_change":"2024-03-20T09:18:11.081321+0000","last_active":"2024-03-20T09:19:11.236995+0000","last_peered":"2024-03-20T09:19:11.236995+0000","last_clean":"2024-03-20T09:19:11.236995+0000","last_became_active":"2024-03-20T09:18:11.080988+0000","last_became_peered":"2024-03-20T09:18:11.080988+0000","last_unstale":"2024-03-20T09:19:11.236995+0000","last_undegraded":"2024-03-20T09:19:11.236995+0000","last_fullsized":"2024-03-20T09:19:11.236995+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":5811,"log_dups_size":0,"ondisk_log_size":5811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:30.376445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380829696,"num_objects":5811,"num_object_clones":0,"num_object_copies":17433,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5811,"num_write_kb":371904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,0],"acting":[10,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"20'137","reported_seq":1326,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:10.402895+0000","last_change":"2024-03-20T09:18:36.914682+0000","last_active":"2024-03-20T09:19:10.402895+0000","last_peered":"2024-03-20T09:19:10.402895+0000","last_clean":"2024-03-20T09:19:10.402895+0000","last_became_active":"2024-03-20T09:12:14.306384+0000","last_became_peered":"2024-03-20T09:12:14.306384+0000","last_unstale":"2024-03-20T09:19:10.402895+0000","last_undegraded":"2024-03-20T09:19:10.402895+0000","last_fullsized":"2024-03-20T09:19:10.402895+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:18:36.914556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:18:36.914556+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:57.995921+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"628'6128","reported_seq":6183,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:15.778853+0000","last_change":"2024-03-20T09:18:11.523500+0000","last_active":"2024-03-20T09:19:15.778853+0000","last_peered":"2024-03-20T09:19:15.778853+0000","last_clean":"2024-03-20T09:19:15.778853+0000","last_became_active":"2024-03-20T09:18:11.522986+0000","last_became_peered":"2024-03-20T09:18:11.522986+0000","last_unstale":"2024-03-20T09:19:15.778853+0000","last_undegraded":"2024-03-20T09:19:15.778853+0000","last_fullsized":"2024-03-20T09:19:15.778853+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":6128,"log_dups_size":0,"ondisk_log_size":6128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:22.129530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401604608,"num_objects":6128,"num_object_clones":0,"num_object_copies":18384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6128,"num_write_kb":392192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"20'137","reported_seq":1221,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:10.394003+0000","last_change":"2024-03-20T09:19:03.805348+0000","last_active":"2024-03-20T09:19:10.394003+0000","last_peered":"2024-03-20T09:19:10.394003+0000","last_clean":"2024-03-20T09:19:10.394003+0000","last_became_active":"2024-03-20T09:07:37.003722+0000","last_became_peered":"2024-03-20T09:07:37.003722+0000","last_unstale":"2024-03-20T09:19:10.394003+0000","last_undegraded":"2024-03-20T09:19:10.394003+0000","last_fullsized":"2024-03-20T09:19:10.394003+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:19:03.805185+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:19:03.805185+0000","last_clean_scrub_stamp":"2024-03-20T09:19:03.805185+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:20:14.287929+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.2","version":"628'6229","reported_seq":6285,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:15.828590+0000","last_change":"2024-03-20T09:18:11.083520+0000","last_active":"2024-03-20T09:19:15.828590+0000","last_peered":"2024-03-20T09:19:15.828590+0000","last_clean":"2024-03-20T09:19:15.828590+0000","last_became_active":"2024-03-20T09:18:11.083136+0000","last_became_peered":"2024-03-20T09:18:11.083136+0000","last_unstale":"2024-03-20T09:19:15.828590+0000","last_undegraded":"2024-03-20T09:19:15.828590+0000","last_fullsized":"2024-03-20T09:19:15.828590+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":6229,"log_dups_size":0,"ondisk_log_size":6229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:27.386223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408223744,"num_objects":6229,"num_object_clones":0,"num_object_copies":18687,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6229,"num_write_kb":398656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"20'137","reported_seq":1238,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:10.394121+0000","last_change":"2024-03-20T09:18:41.078380+0000","last_active":"2024-03-20T09:19:10.394121+0000","last_peered":"2024-03-20T09:19:10.394121+0000","last_clean":"2024-03-20T09:19:10.394121+0000","last_became_active":"2024-03-20T09:07:52.593671+0000","last_became_peered":"2024-03-20T09:07:52.593671+0000","last_unstale":"2024-03-20T09:19:10.394121+0000","last_undegraded":"2024-03-20T09:19:10.394121+0000","last_fullsized":"2024-03-20T09:19:10.394121+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:18:41.078246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:18:41.078246+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:20:00.918926+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.7","version":"628'5872","reported_seq":5928,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:12.211345+0000","last_change":"2024-03-20T09:18:11.080196+0000","last_active":"2024-03-20T09:19:12.211345+0000","last_peered":"2024-03-20T09:19:12.211345+0000","last_clean":"2024-03-20T09:19:12.211345+0000","last_became_active":"2024-03-20T09:18:11.079858+0000","last_became_peered":"2024-03-20T09:18:11.079858+0000","last_unstale":"2024-03-20T09:19:12.211345+0000","last_undegraded":"2024-03-20T09:19:12.211345+0000","last_fullsized":"2024-03-20T09:19:12.211345+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":5872,"log_dups_size":0,"ondisk_log_size":5872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:31.794901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384827392,"num_objects":5872,"num_object_clones":0,"num_object_copies":17616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5872,"num_write_kb":375808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"20'137","reported_seq":1231,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:10.398203+0000","last_change":"2024-03-20T09:18:45.511064+0000","last_active":"2024-03-20T09:19:10.398203+0000","last_peered":"2024-03-20T09:19:10.398203+0000","last_clean":"2024-03-20T09:19:10.398203+0000","last_became_active":"2024-03-20T09:12:02.221686+0000","last_became_peered":"2024-03-20T09:12:02.221686+0000","last_unstale":"2024-03-20T09:19:10.398203+0000","last_undegraded":"2024-03-20T09:19:10.398203+0000","last_fullsized":"2024-03-20T09:19:10.398203+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:18:45.510938+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:18.788771+0000","last_clean_scrub_stamp":"2024-03-20T09:18:45.510938+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:20:05.681444+0000","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,8],"acting":[1,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"628'5898","reported_seq":5954,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:11.304472+0000","last_change":"2024-03-20T09:18:11.078580+0000","last_active":"2024-03-20T09:19:11.304472+0000","last_peered":"2024-03-20T09:19:11.304472+0000","last_clean":"2024-03-20T09:19:11.304472+0000","last_became_active":"2024-03-20T09:18:11.078275+0000","last_became_peered":"2024-03-20T09:18:11.078275+0000","last_unstale":"2024-03-20T09:19:11.304472+0000","last_undegraded":"2024-03-20T09:19:11.304472+0000","last_fullsized":"2024-03-20T09:19:11.304472+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":5898,"log_dups_size":0,"ondisk_log_size":5898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:37.063979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386531328,"num_objects":5898,"num_object_clones":0,"num_object_copies":17694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5898,"num_write_kb":377472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,5],"acting":[10,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":1297,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:10.402735+0000","last_change":"2024-03-20T09:19:04.808957+0000","last_active":"2024-03-20T09:19:10.402735+0000","last_peered":"2024-03-20T09:19:10.402735+0000","last_clean":"2024-03-20T09:19:10.402735+0000","last_became_active":"2024-03-20T09:12:14.305989+0000","last_became_peered":"2024-03-20T09:12:14.305989+0000","last_unstale":"2024-03-20T09:19:10.402735+0000","last_undegraded":"2024-03-20T09:19:10.402735+0000","last_fullsized":"2024-03-20T09:19:10.402735+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:19:04.808852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:19:04.808852+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:20:07.412098+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.6","version":"628'5975","reported_seq":6031,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:14.422548+0000","last_change":"2024-03-20T09:18:11.523259+0000","last_active":"2024-03-20T09:19:14.422548+0000","last_peered":"2024-03-20T09:19:14.422548+0000","last_clean":"2024-03-20T09:19:14.422548+0000","last_became_active":"2024-03-20T09:18:11.522922+0000","last_became_peered":"2024-03-20T09:18:11.522922+0000","last_unstale":"2024-03-20T09:19:14.422548+0000","last_undegraded":"2024-03-20T09:19:14.422548+0000","last_fullsized":"2024-03-20T09:19:14.422548+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":5975,"log_dups_size":0,"ondisk_log_size":5975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:32.564244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391577600,"num_objects":5975,"num_object_clones":0,"num_object_copies":17925,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5975,"num_write_kb":382400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"20'137","reported_seq":1546,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:10.398801+0000","last_change":"2024-03-20T09:19:00.698914+0000","last_active":"2024-03-20T09:19:10.398801+0000","last_peered":"2024-03-20T09:19:10.398801+0000","last_clean":"2024-03-20T09:19:10.398801+0000","last_became_active":"2024-03-20T09:12:14.313541+0000","last_became_peered":"2024-03-20T09:12:14.313541+0000","last_unstale":"2024-03-20T09:19:10.398801+0000","last_undegraded":"2024-03-20T09:19:10.398801+0000","last_fullsized":"2024-03-20T09:19:10.398801+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:19:00.698788+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:19.513511+0000","last_clean_scrub_stamp":"2024-03-20T09:19:00.698788+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:20:26.588084+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"628'5884","reported_seq":5939,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:12.125469+0000","last_change":"2024-03-20T09:18:11.082142+0000","last_active":"2024-03-20T09:19:12.125469+0000","last_peered":"2024-03-20T09:19:12.125469+0000","last_clean":"2024-03-20T09:19:12.125469+0000","last_became_active":"2024-03-20T09:18:11.081786+0000","last_became_peered":"2024-03-20T09:18:11.081786+0000","last_unstale":"2024-03-20T09:19:12.125469+0000","last_undegraded":"2024-03-20T09:19:12.125469+0000","last_fullsized":"2024-03-20T09:19:12.125469+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":5884,"log_dups_size":0,"ondisk_log_size":5884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:16.733507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385613824,"num_objects":5884,"num_object_clones":0,"num_object_copies":17652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5884,"num_write_kb":376576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"20'137","reported_seq":699,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:10.401239+0000","last_change":"2024-03-20T09:19:02.318444+0000","last_active":"2024-03-20T09:19:10.401239+0000","last_peered":"2024-03-20T09:19:10.401239+0000","last_clean":"2024-03-20T09:19:10.401239+0000","last_became_active":"2024-03-20T09:15:34.225022+0000","last_became_peered":"2024-03-20T09:15:34.225022+0000","last_unstale":"2024-03-20T09:19:10.401239+0000","last_undegraded":"2024-03-20T09:19:10.401239+0000","last_fullsized":"2024-03-20T09:19:10.401239+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:19:02.318328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:19:02.318328+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:20:11.456575+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,5],"acting":[6,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"628'5927","reported_seq":5982,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:12.115727+0000","last_change":"2024-03-20T09:18:11.081759+0000","last_active":"2024-03-20T09:19:12.115727+0000","last_peered":"2024-03-20T09:19:12.115727+0000","last_clean":"2024-03-20T09:19:12.115727+0000","last_became_active":"2024-03-20T09:18:11.081425+0000","last_became_peered":"2024-03-20T09:18:11.081425+0000","last_unstale":"2024-03-20T09:19:12.115727+0000","last_undegraded":"2024-03-20T09:19:12.115727+0000","last_fullsized":"2024-03-20T09:19:12.115727+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":5927,"log_dups_size":0,"ondisk_log_size":5927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:22.317679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388431872,"num_objects":5927,"num_object_clones":0,"num_object_copies":17781,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5927,"num_write_kb":379328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"20'137","reported_seq":1235,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:10.404205+0000","last_change":"2024-03-20T09:18:39.685223+0000","last_active":"2024-03-20T09:19:10.404205+0000","last_peered":"2024-03-20T09:19:10.404205+0000","last_clean":"2024-03-20T09:19:10.404205+0000","last_became_active":"2024-03-20T09:13:27.087760+0000","last_became_peered":"2024-03-20T09:13:27.087760+0000","last_unstale":"2024-03-20T09:19:10.404205+0000","last_undegraded":"2024-03-20T09:19:10.404205+0000","last_fullsized":"2024-03-20T09:19:10.404205+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:18:39.685133+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:14:10.560890+0000","last_clean_scrub_stamp":"2024-03-20T09:18:39.685133+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:50.414787+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"628'5859","reported_seq":5915,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:12.231344+0000","last_change":"2024-03-20T09:18:11.079851+0000","last_active":"2024-03-20T09:19:12.231344+0000","last_peered":"2024-03-20T09:19:12.231344+0000","last_clean":"2024-03-20T09:19:12.231344+0000","last_became_active":"2024-03-20T09:18:11.079589+0000","last_became_peered":"2024-03-20T09:18:11.079589+0000","last_unstale":"2024-03-20T09:19:12.231344+0000","last_undegraded":"2024-03-20T09:19:12.231344+0000","last_fullsized":"2024-03-20T09:19:12.231344+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":5859,"log_dups_size":0,"ondisk_log_size":5859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:36.062370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383975424,"num_objects":5859,"num_object_clones":0,"num_object_copies":17577,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5859,"num_write_kb":374976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,2],"acting":[5,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"20'137","reported_seq":1102,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:10.402770+0000","last_change":"2024-03-20T09:18:38.995228+0000","last_active":"2024-03-20T09:19:10.402770+0000","last_peered":"2024-03-20T09:19:10.402770+0000","last_clean":"2024-03-20T09:19:10.402770+0000","last_became_active":"2024-03-20T09:13:20.070596+0000","last_became_peered":"2024-03-20T09:13:20.070596+0000","last_unstale":"2024-03-20T09:19:10.402770+0000","last_undegraded":"2024-03-20T09:19:10.402770+0000","last_fullsized":"2024-03-20T09:19:10.402770+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:18:38.995049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:18:38.995049+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:52.947693+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.a","version":"628'6162","reported_seq":6218,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:14.385354+0000","last_change":"2024-03-20T09:18:11.079740+0000","last_active":"2024-03-20T09:19:14.385354+0000","last_peered":"2024-03-20T09:19:14.385354+0000","last_clean":"2024-03-20T09:19:14.385354+0000","last_became_active":"2024-03-20T09:18:11.079361+0000","last_became_peered":"2024-03-20T09:18:11.079361+0000","last_unstale":"2024-03-20T09:19:14.385354+0000","last_undegraded":"2024-03-20T09:19:14.385354+0000","last_fullsized":"2024-03-20T09:19:14.385354+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":6162,"log_dups_size":0,"ondisk_log_size":6162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:28.144588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403832832,"num_objects":6162,"num_object_clones":0,"num_object_copies":18486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6162,"num_write_kb":394368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,10],"acting":[2,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"20'137","reported_seq":762,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:10.395870+0000","last_change":"2024-03-20T09:18:52.574163+0000","last_active":"2024-03-20T09:19:10.395870+0000","last_peered":"2024-03-20T09:19:10.395870+0000","last_clean":"2024-03-20T09:19:10.395870+0000","last_became_active":"2024-03-20T09:12:14.311319+0000","last_became_peered":"2024-03-20T09:12:14.311319+0000","last_unstale":"2024-03-20T09:19:10.395870+0000","last_undegraded":"2024-03-20T09:19:10.395870+0000","last_fullsized":"2024-03-20T09:19:10.395870+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:18:52.573886+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:18:52.573886+0000","last_clean_scrub_stamp":"2024-03-20T09:18:52.573886+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:20:06.529061+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"628'5771","reported_seq":5826,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:12.115384+0000","last_change":"2024-03-20T09:18:11.523380+0000","last_active":"2024-03-20T09:19:12.115384+0000","last_peered":"2024-03-20T09:19:12.115384+0000","last_clean":"2024-03-20T09:19:12.115384+0000","last_became_active":"2024-03-20T09:18:11.522906+0000","last_became_peered":"2024-03-20T09:18:11.522906+0000","last_unstale":"2024-03-20T09:19:12.115384+0000","last_undegraded":"2024-03-20T09:19:12.115384+0000","last_fullsized":"2024-03-20T09:19:12.115384+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":5771,"log_dups_size":0,"ondisk_log_size":5771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:19.027570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378208256,"num_objects":5771,"num_object_clones":0,"num_object_copies":17313,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5771,"num_write_kb":369344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"20'137","reported_seq":1250,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:10.404041+0000","last_change":"2024-03-20T09:18:43.648807+0000","last_active":"2024-03-20T09:19:10.404041+0000","last_peered":"2024-03-20T09:19:10.404041+0000","last_clean":"2024-03-20T09:19:10.404041+0000","last_became_active":"2024-03-20T09:10:35.119706+0000","last_became_peered":"2024-03-20T09:10:35.119706+0000","last_unstale":"2024-03-20T09:19:10.404041+0000","last_undegraded":"2024-03-20T09:19:10.404041+0000","last_fullsized":"2024-03-20T09:19:10.404041+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:18:43.648689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:18:43.648689+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:20:09.695726+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"628'6052","reported_seq":6108,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:15.160581+0000","last_change":"2024-03-20T09:18:11.080513+0000","last_active":"2024-03-20T09:19:15.160581+0000","last_peered":"2024-03-20T09:19:15.160581+0000","last_clean":"2024-03-20T09:19:15.160581+0000","last_became_active":"2024-03-20T09:18:11.080212+0000","last_became_peered":"2024-03-20T09:18:11.080212+0000","last_unstale":"2024-03-20T09:19:15.160581+0000","last_undegraded":"2024-03-20T09:19:15.160581+0000","last_fullsized":"2024-03-20T09:19:15.160581+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":6052,"log_dups_size":0,"ondisk_log_size":6052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:19.675169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":396623872,"num_objects":6052,"num_object_clones":0,"num_object_copies":18156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6052,"num_write_kb":387328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"570'138","reported_seq":1277,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:10.404474+0000","last_change":"2024-03-20T09:19:05.957351+0000","last_active":"2024-03-20T09:19:10.404474+0000","last_peered":"2024-03-20T09:19:10.404474+0000","last_clean":"2024-03-20T09:19:10.404474+0000","last_became_active":"2024-03-20T09:13:27.087686+0000","last_became_peered":"2024-03-20T09:13:27.087686+0000","last_unstale":"2024-03-20T09:19:10.404474+0000","last_undegraded":"2024-03-20T09:19:10.404474+0000","last_fullsized":"2024-03-20T09:19:10.404474+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"570'138","last_scrub_stamp":"2024-03-20T09:19:05.957217+0000","last_deep_scrub":"570'138","last_deep_scrub_stamp":"2024-03-20T09:16:13.725035+0000","last_clean_scrub_stamp":"2024-03-20T09:19:05.957217+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:20:23.781512+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"628'6243","reported_seq":6299,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:15.814592+0000","last_change":"2024-03-20T09:18:11.523430+0000","last_active":"2024-03-20T09:19:15.814592+0000","last_peered":"2024-03-20T09:19:15.814592+0000","last_clean":"2024-03-20T09:19:15.814592+0000","last_became_active":"2024-03-20T09:18:11.522844+0000","last_became_peered":"2024-03-20T09:18:11.522844+0000","last_unstale":"2024-03-20T09:19:15.814592+0000","last_undegraded":"2024-03-20T09:19:15.814592+0000","last_fullsized":"2024-03-20T09:19:15.814592+0000","mapping_epoch":605,"log_start":"0'0","ondisk_log_start":"0'0","created":605,"last_epoch_clean":606,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:18:10.060876+0000","last_clean_scrub_stamp":"2024-03-20T09:18:10.060876+0000","objects_scrubbed":0,"log_size":6243,"log_dups_size":0,"ondisk_log_size":6243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:18.305583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409141248,"num_objects":6243,"num_object_clones":0,"num_object_copies":18729,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6243,"num_write_kb":399552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,11],"acting":[6,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.d","version":"20'137","reported_seq":822,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:10.397039+0000","last_change":"2024-03-20T09:18:05.353742+0000","last_active":"2024-03-20T09:19:10.397039+0000","last_peered":"2024-03-20T09:19:10.397039+0000","last_clean":"2024-03-20T09:19:10.397039+0000","last_became_active":"2024-03-20T09:12:14.307864+0000","last_became_peered":"2024-03-20T09:12:14.307864+0000","last_unstale":"2024-03-20T09:19:10.397039+0000","last_undegraded":"2024-03-20T09:19:10.397039+0000","last_fullsized":"2024-03-20T09:19:10.397039+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:18:05.353559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:18:05.353559+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"20'137","reported_seq":1137,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:10.396597+0000","last_change":"2024-03-20T09:17:47.874749+0000","last_active":"2024-03-20T09:19:10.396597+0000","last_peered":"2024-03-20T09:19:10.396597+0000","last_clean":"2024-03-20T09:19:10.396597+0000","last_became_active":"2024-03-20T09:10:34.096758+0000","last_became_peered":"2024-03-20T09:10:34.096758+0000","last_unstale":"2024-03-20T09:19:10.396597+0000","last_undegraded":"2024-03-20T09:19:10.396597+0000","last_fullsized":"2024-03-20T09:19:10.396597+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:47.874609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:17:47.874609+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:11.136312+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"20'137","reported_seq":1210,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:10.398103+0000","last_change":"2024-03-20T09:17:57.414516+0000","last_active":"2024-03-20T09:19:10.398103+0000","last_peered":"2024-03-20T09:19:10.398103+0000","last_clean":"2024-03-20T09:19:10.398103+0000","last_became_active":"2024-03-20T09:04:04.103645+0000","last_became_peered":"2024-03-20T09:04:04.103645+0000","last_unstale":"2024-03-20T09:19:10.398103+0000","last_undegraded":"2024-03-20T09:19:10.398103+0000","last_fullsized":"2024-03-20T09:19:10.398103+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:57.414409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:17:57.414409+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:26.874738+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"20'137","reported_seq":671,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:10.396660+0000","last_change":"2024-03-20T09:19:04.231877+0000","last_active":"2024-03-20T09:19:10.396660+0000","last_peered":"2024-03-20T09:19:10.396660+0000","last_clean":"2024-03-20T09:19:10.396660+0000","last_became_active":"2024-03-20T09:10:34.097191+0000","last_became_peered":"2024-03-20T09:10:34.097191+0000","last_unstale":"2024-03-20T09:19:10.396660+0000","last_undegraded":"2024-03-20T09:19:10.396660+0000","last_fullsized":"2024-03-20T09:19:10.396660+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:19:04.231763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:19:04.231763+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:20:22.482474+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"20'137","reported_seq":1073,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:10.398445+0000","last_change":"2024-03-20T09:17:46.485538+0000","last_active":"2024-03-20T09:19:10.398445+0000","last_peered":"2024-03-20T09:19:10.398445+0000","last_clean":"2024-03-20T09:19:10.398445+0000","last_became_active":"2024-03-20T09:13:27.087859+0000","last_became_peered":"2024-03-20T09:13:27.087859+0000","last_unstale":"2024-03-20T09:19:10.398445+0000","last_undegraded":"2024-03-20T09:19:10.398445+0000","last_fullsized":"2024-03-20T09:19:10.398445+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:46.485413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:17:46.485413+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:13.166557+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,3],"acting":[0,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"20'137","reported_seq":899,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:10.396892+0000","last_change":"2024-03-20T09:17:52.461386+0000","last_active":"2024-03-20T09:19:10.396892+0000","last_peered":"2024-03-20T09:19:10.396892+0000","last_clean":"2024-03-20T09:19:10.396892+0000","last_became_active":"2024-03-20T09:13:27.087760+0000","last_became_peered":"2024-03-20T09:13:27.087760+0000","last_unstale":"2024-03-20T09:19:10.396892+0000","last_undegraded":"2024-03-20T09:19:10.396892+0000","last_fullsized":"2024-03-20T09:19:10.396892+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:52.461261+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:16:15.448920+0000","last_clean_scrub_stamp":"2024-03-20T09:17:52.461261+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:16.137279+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"20'137","reported_seq":1217,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:10.413703+0000","last_change":"2024-03-20T09:17:46.961152+0000","last_active":"2024-03-20T09:19:10.413703+0000","last_peered":"2024-03-20T09:19:10.413703+0000","last_clean":"2024-03-20T09:19:10.413703+0000","last_became_active":"2024-03-20T09:10:35.095339+0000","last_became_peered":"2024-03-20T09:10:35.095339+0000","last_unstale":"2024-03-20T09:19:10.413703+0000","last_undegraded":"2024-03-20T09:19:10.413703+0000","last_fullsized":"2024-03-20T09:19:10.413703+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:46.961059+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:23.640533+0000","last_clean_scrub_stamp":"2024-03-20T09:17:46.961059+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:11.863478+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"20'137","reported_seq":1286,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:10.405495+0000","last_change":"2024-03-20T09:17:47.414530+0000","last_active":"2024-03-20T09:19:10.405495+0000","last_peered":"2024-03-20T09:19:10.405495+0000","last_clean":"2024-03-20T09:19:10.405495+0000","last_became_active":"2024-03-20T09:04:14.208448+0000","last_became_peered":"2024-03-20T09:04:14.208448+0000","last_unstale":"2024-03-20T09:19:10.405495+0000","last_undegraded":"2024-03-20T09:19:10.405495+0000","last_fullsized":"2024-03-20T09:19:10.405495+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":268,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:47.414381+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:16:15.754362+0000","last_clean_scrub_stamp":"2024-03-20T09:17:47.414381+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:13.468347+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"20'137","reported_seq":558,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:10.396290+0000","last_change":"2024-03-20T09:18:46.645254+0000","last_active":"2024-03-20T09:19:10.396290+0000","last_peered":"2024-03-20T09:19:10.396290+0000","last_clean":"2024-03-20T09:19:10.396290+0000","last_became_active":"2024-03-20T09:12:14.307581+0000","last_became_peered":"2024-03-20T09:12:14.307581+0000","last_unstale":"2024-03-20T09:19:10.396290+0000","last_undegraded":"2024-03-20T09:19:10.396290+0000","last_fullsized":"2024-03-20T09:19:10.396290+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:18:46.645121+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:07:06.480160+0000","last_clean_scrub_stamp":"2024-03-20T09:18:46.645121+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:47.563118+0000","scrub_duration":137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"20'137","reported_seq":830,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-03-20T09:19:10.398317+0000","last_change":"2024-03-20T09:17:59.332278+0000","last_active":"2024-03-20T09:19:10.398317+0000","last_peered":"2024-03-20T09:19:10.398317+0000","last_clean":"2024-03-20T09:19:10.398317+0000","last_became_active":"2024-03-20T09:13:20.074439+0000","last_became_peered":"2024-03-20T09:13:20.074439+0000","last_unstale":"2024-03-20T09:19:10.398317+0000","last_undegraded":"2024-03-20T09:19:10.398317+0000","last_fullsized":"2024-03-20T09:19:10.398317+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:17:59.332142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:17:59.332142+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:19:28.719063+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,11],"acting":[1,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":6278479872,"num_objects":95802,"num_object_clones":0,"num_object_copies":287406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95802,"num_write_kb":6131328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18723831808,"data_stored":18723831808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95802,"ondisk_log_size":95802,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4248,"ondisk_log_size":4248,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860694,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2071940,"kb_used_data":1563196,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":91673212,"statfs":{"total":95995035648,"available":93873369088,"internally_reserved":0,"allocated":1600712704,"data_stored":1600347069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":520944068},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":89,"seq":382252089628,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2629640,"kb_used_data":2233340,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":91115512,"statfs":{"total":95995035648,"available":93302284288,"internally_reserved":0,"allocated":2286940160,"data_stored":2286574525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":405797312},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":0,"seq":343597383972,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":70,"seq":300647711015,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1091080,"kb_used_data":753084,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":92654072,"statfs":{"total":95995035648,"available":94877769728,"internally_reserved":0,"allocated":771158016,"data_stored":770792381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":346094022},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":60,"seq":257698038060,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2415568,"kb_used_data":1838788,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":91329584,"statfs":{"total":95995035648,"available":93521494016,"internally_reserved":0,"allocated":1882918912,"data_stored":1882549725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":590608842},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332402,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2403652,"kb_used_data":1975356,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":91341500,"statfs":{"total":95995035648,"available":93533696000,"internally_reserved":0,"allocated":2022764544,"data_stored":2022398909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":438499782},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575179,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2082820,"kb_used_data":1510204,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":91662332,"statfs":{"total":95995035648,"available":93862227968,"internally_reserved":0,"allocated":1546448896,"data_stored":1546083261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":586348998},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738698,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3492932,"kb_used_data":2992508,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":90252220,"statfs":{"total":95995035648,"available":92418273280,"internally_reserved":0,"allocated":3064328192,"data_stored":3063962557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411653,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2266700,"kb_used_data":1924548,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":91478452,"statfs":{"total":95995035648,"available":93673934848,"internally_reserved":0,"allocated":1970737152,"data_stored":1970367965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150017,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":799300,"kb_used_data":389116,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92945852,"statfs":{"total":95995035648,"available":95176552448,"internally_reserved":0,"allocated":398454784,"data_stored":398089149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":420018631},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953791,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":443,"seq":1902670512233,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3200272,"kb_used_data":2975940,"kb_used_omap":181,"kb_used_meta":224138,"kb_avail":90544880,"statfs":{"total":95995035648,"available":92717957120,"internally_reserved":0,"allocated":3047362560,"data_stored":3046993373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":186131,"internal_metadata":229517549},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3078750208,"data_stored":3078750208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1549729792,"data_stored":1549729792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1993146368,"data_stored":1993146368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":401604608,"data_stored":401604608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3084451840,"data_stored":3084451840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2036531200,"data_stored":2036531200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1894973440,"data_stored":1894973440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":776732672,"data_stored":776732672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2299133952,"data_stored":2299133952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1608777728,"data_stored":1608777728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:19:18.292 INFO:tasks.thrashosds.thrasher:Setting 5.c to [5, 11, 3] 2024-03-20T09:19:18.292 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '5.c', '5', '11', '3'] 2024-03-20T09:19:18.293 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pg-upmap 5.c 5 11 3 2024-03-20T09:19:18.346 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:19:18.385 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:18.410 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:19:18.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:18.413 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:19:18.476 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:18 smithi026 ceph-mon[68309]: from='client.26020 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:19:18.476 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:18 smithi026 ceph-mon[68309]: pgmap v1509: 47 pgs: 47 active+clean; 6.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-03-20T09:19:18.513 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:19:18.540 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:18 smithi122 ceph-mon[73495]: from='client.26020 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:19:18.541 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:18 smithi122 ceph-mon[73495]: pgmap v1509: 47 pgs: 47 active+clean; 6.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-03-20T09:19:18.543 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:18.565 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:19:18.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:18.570 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:19:18.627 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 58 16 102267 102251 110.171 72.875 0.0169738 0.00907272 2024-03-20T09:19:18.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:18 smithi115 ceph-mon[72476]: from='client.26020 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:19:18.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:18 smithi115 ceph-mon[72476]: pgmap v1509: 47 pgs: 47 active+clean; 6.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-03-20T09:19:18.671 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:19:18.707 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:18.735 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:19:18.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:18.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:19:18.840 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:19:18.888 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:18.915 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:19:18.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:18.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:19:19.020 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:19:19.056 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:19.083 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:19:19.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:19.089 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:19:19.190 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:19:19.239 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:19.281 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:19:19.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:19.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:19:19.386 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:19:19.446 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:19.627 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 59 15 103468 103453 109.577 75.125 0.0165858 0.0091222 2024-03-20T09:19:19.672 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:19:19.709 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:19:19.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:19.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:19:19.819 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:19:19.942 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:20.630 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:2024-03-20T09:19:20.627537+0000 min lat: 0.0010551 max lat: 0.247256 avg lat: 0.00916975 2024-03-20T09:19:20.630 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T09:19:20.630 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 60 16 104678 104662 109.01 75.5625 0.0234022 0.00916975 2024-03-20T09:19:21.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:21 smithi115 ceph-mon[72476]: pgmap v1510: 47 pgs: 47 active+clean; 6.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-03-20T09:19:21.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:21 smithi122 ceph-mon[73495]: pgmap v1510: 47 pgs: 47 active+clean; 6.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-03-20T09:19:21.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:21 smithi026 ceph-mon[68309]: pgmap v1510: 47 pgs: 47 active+clean; 6.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-03-20T09:19:21.627 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 61 16 105884 105868 108.459 75.375 0.011754 0.00921579 2024-03-20T09:19:21.838 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:19:22.117 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:22 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:19:22.128 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:19:22.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:22.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:19:22.235 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:19:22.273 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:22.306 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:19:22.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:22.312 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:19:22.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:22 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:19:22.413 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:19:22.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:22 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:19:22.449 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:22.471 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:19:22.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:22.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:19:22.577 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:19:22.615 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:22.627 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 62 16 106896 106880 107.73 63.25 0.00458932 0.00927333 2024-03-20T09:19:22.642 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:19:22.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:22.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:19:22.751 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:19:22.792 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:22.829 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:19:22.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:22.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:19:22.936 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:19:22.992 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:23.024 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:19:23.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:23.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:19:23.121 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:23 smithi115 ceph-mon[72476]: pgmap v1511: 47 pgs: 47 active+clean; 6.3 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-03-20T09:19:23.121 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:23 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:19:23.121 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:23 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:19:23.121 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:23 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:19:23.132 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:19:23.179 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:23.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:23 smithi122 ceph-mon[73495]: pgmap v1511: 47 pgs: 47 active+clean; 6.3 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-03-20T09:19:23.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:23 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:19:23.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:23 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:19:23.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:23 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:19:23.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:23 smithi026 ceph-mon[68309]: pgmap v1511: 47 pgs: 47 active+clean; 6.3 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-03-20T09:19:23.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:23 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:19:23.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:23 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:19:23.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:23 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:19:23.628 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 63 16 108093 108077 107.207 74.8125 0.00537432 0.00932242 2024-03-20T09:19:23.938 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:19:24.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:24 smithi115 ceph-mon[72476]: pgmap v1512: 47 pgs: 47 active+clean; 6.4 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-03-20T09:19:24.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:24 smithi122 ceph-mon[73495]: pgmap v1512: 47 pgs: 47 active+clean; 6.4 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-03-20T09:19:24.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:24 smithi026 ceph-mon[68309]: pgmap v1512: 47 pgs: 47 active+clean; 6.4 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-03-20T09:19:24.565 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:19:24.585 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:19:24.586 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-03-20T09:19:24.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-03-20T09:19:24.586 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:19:24.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.586 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:160216 3.e 3:7442c5bb:::benchmark_data_smithi122_119671_object60434:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e479)", 2024-03-20T09:19:24.586 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:11:57.703382+0000", 2024-03-20T09:19:24.586 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 446.87780869900001, 2024-03-20T09:19:24.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6642757779999999, 2024-03-20T09:19:24.586 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:19:24.586 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:19:24.586 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:19:24.586 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:19:24.587 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:19:24.587 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 160216 2024-03-20T09:19:24.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.587 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:19:24.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:19:24.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:11:57.703382+0000", 2024-03-20T09:19:24.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:19:24.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:11:57.703382+0000", 2024-03-20T09:19:24.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:19:24.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:11:57.703385+0000", 2024-03-20T09:19:24.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3479999999999998e-06 2024-03-20T09:19:24.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:19:24.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:11:57.703393+0000", 2024-03-20T09:19:24.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6039999999999993e-06 2024-03-20T09:19:24.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:19:24.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:11:57.703409+0000", 2024-03-20T09:19:24.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5812999999999999e-05 2024-03-20T09:19:24.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:19:24.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:11:57.703421+0000", 2024-03-20T09:19:24.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1715e-05 2024-03-20T09:19:24.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:19:24.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:11:57.823965+0000", 2024-03-20T09:19:24.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.1205444 2024-03-20T09:19:24.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:19:24.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:11:57.829762+0000", 2024-03-20T09:19:24.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057967720000000004 2024-03-20T09:19:24.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:19:24.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:11:57.864209+0000", 2024-03-20T09:19:24.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.034446919999999999 2024-03-20T09:19:24.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-03-20T09:19:24.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:11:57.867154+0000", 2024-03-20T09:19:24.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0029456069999999998 2024-03-20T09:19:24.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:19:24.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:00.317171+0000", 2024-03-20T09:19:24.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4500168009999999 2024-03-20T09:19:24.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:19:24.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:00.321096+0000", 2024-03-20T09:19:24.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0039244689999999999 2024-03-20T09:19:24.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,10", 2024-03-20T09:19:24.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:00.326694+0000", 2024-03-20T09:19:24.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0055978699999999996 2024-03-20T09:19:24.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:19:24.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:00.356385+0000", 2024-03-20T09:19:24.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.029691827 2024-03-20T09:19:24.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:19:24.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:00.357254+0000", 2024-03-20T09:19:24.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00086866099999999996 2024-03-20T09:19:24.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:19:24.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:00.358324+0000", 2024-03-20T09:19:24.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010702979999999999 2024-03-20T09:19:24.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:19:24.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:00.360171+0000", 2024-03-20T09:19:24.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018465770000000001 2024-03-20T09:19:24.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:19:24.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:00.367658+0000", 2024-03-20T09:19:24.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074870960000000004 2024-03-20T09:19:24.594 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.594 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:24.594 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.594 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:160218 3.e 3:7e6916b2:::benchmark_data_smithi122_119671_object60436:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e479)", 2024-03-20T09:19:24.594 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:11:57.770387+0000", 2024-03-20T09:19:24.594 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 446.81080348699999, 2024-03-20T09:19:24.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4117168279999999, 2024-03-20T09:19:24.594 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:19:24.594 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:19:24.594 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:19:24.594 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:19:24.595 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:19:24.595 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 160218 2024-03-20T09:19:24.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.595 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:19:24.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:19:24.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:11:57.770387+0000", 2024-03-20T09:19:24.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:19:24.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:11:57.770387+0000", 2024-03-20T09:19:24.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:19:24.596 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:11:57.770389+0000", 2024-03-20T09:19:24.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.438e-06 2024-03-20T09:19:24.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:19:24.596 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:11:57.770395+0000", 2024-03-20T09:19:24.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5370000000000003e-06 2024-03-20T09:19:24.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:19:24.596 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:11:57.770410+0000", 2024-03-20T09:19:24.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4772e-05 2024-03-20T09:19:24.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:19:24.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:11:57.770422+0000", 2024-03-20T09:19:24.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1579e-05 2024-03-20T09:19:24.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:19:24.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:11:57.884879+0000", 2024-03-20T09:19:24.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.114456791 2024-03-20T09:19:24.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:19:24.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:11:57.887871+0000", 2024-03-20T09:19:24.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0029920440000000001 2024-03-20T09:19:24.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:19:24.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:11:57.929847+0000", 2024-03-20T09:19:24.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.041976397999999998 2024-03-20T09:19:24.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for rw locks", 2024-03-20T09:19:24.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:11:57.932785+0000", 2024-03-20T09:19:24.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002937537 2024-03-20T09:19:24.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:19:24.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:01.146876+0000", 2024-03-20T09:19:24.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.214091078 2024-03-20T09:19:24.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:19:24.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:01.150269+0000", 2024-03-20T09:19:24.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0033931949999999999 2024-03-20T09:19:24.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,10", 2024-03-20T09:19:24.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:01.155464+0000", 2024-03-20T09:19:24.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0051955650000000001 2024-03-20T09:19:24.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:19:24.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:01.171649+0000", 2024-03-20T09:19:24.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016185113000000001 2024-03-20T09:19:24.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:19:24.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:01.172527+0000", 2024-03-20T09:19:24.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00087785200000000004 2024-03-20T09:19:24.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:19:24.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:01.173634+0000", 2024-03-20T09:19:24.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001106625 2024-03-20T09:19:24.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:19:24.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:01.175173+0000", 2024-03-20T09:19:24.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015394409999999999 2024-03-20T09:19:24.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:19:24.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:12:01.182104+0000", 2024-03-20T09:19:24.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0069308629999999998 2024-03-20T09:19:24.601 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.601 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:24.601 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.601 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:71592 5.3 5:ca0a7735:::benchmark_data_smithi122_157894_object71591:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e622)", 2024-03-20T09:19:24.601 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:18:55.549391+0000", 2024-03-20T09:19:24.601 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 29.031799540000002, 2024-03-20T09:19:24.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.064318555999999999, 2024-03-20T09:19:24.602 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:19:24.602 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:19:24.602 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:19:24.602 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:19:24.602 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:19:24.602 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 71592 2024-03-20T09:19:24.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.602 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:19:24.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:19:24.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.549391+0000", 2024-03-20T09:19:24.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:19:24.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.549391+0000", 2024-03-20T09:19:24.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:19:24.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.549392+0000", 2024-03-20T09:19:24.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0920000000000001e-06 2024-03-20T09:19:24.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:19:24.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.549527+0000", 2024-03-20T09:19:24.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000134246 2024-03-20T09:19:24.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:19:24.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.549539+0000", 2024-03-20T09:19:24.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.222e-05 2024-03-20T09:19:24.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:19:24.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.549547+0000", 2024-03-20T09:19:24.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0399999999999993e-06 2024-03-20T09:19:24.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:19:24.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.549599+0000", 2024-03-20T09:19:24.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2355999999999998e-05 2024-03-20T09:19:24.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:19:24.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.549718+0000", 2024-03-20T09:19:24.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000118548 2024-03-20T09:19:24.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5,6", 2024-03-20T09:19:24.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.549810+0000", 2024-03-20T09:19:24.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2502000000000003e-05 2024-03-20T09:19:24.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:19:24.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.553946+0000", 2024-03-20T09:19:24.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0041353199999999996 2024-03-20T09:19:24.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:19:24.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.610573+0000", 2024-03-20T09:19:24.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.056627291000000003 2024-03-20T09:19:24.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:19:24.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.613601+0000", 2024-03-20T09:19:24.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003028279 2024-03-20T09:19:24.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.606 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:19:24.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.613661+0000", 2024-03-20T09:19:24.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0158000000000002e-05 2024-03-20T09:19:24.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:19:24.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.613710+0000", 2024-03-20T09:19:24.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8504000000000002e-05 2024-03-20T09:19:24.607 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.607 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:24.607 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.607 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:71601 5.3 5:c59faa39:::benchmark_data_smithi122_157894_object71600:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e622)", 2024-03-20T09:19:24.607 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:18:55.556322+0000", 2024-03-20T09:19:24.607 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 29.024868913999999, 2024-03-20T09:19:24.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.065477341999999994, 2024-03-20T09:19:24.608 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:19:24.608 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:19:24.608 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:19:24.608 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:19:24.608 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:19:24.608 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 71601 2024-03-20T09:19:24.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.608 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:19:24.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:19:24.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.556322+0000", 2024-03-20T09:19:24.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:19:24.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.556322+0000", 2024-03-20T09:19:24.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:19:24.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.556323+0000", 2024-03-20T09:19:24.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3060000000000001e-06 2024-03-20T09:19:24.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:19:24.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.556453+0000", 2024-03-20T09:19:24.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012920799999999999 2024-03-20T09:19:24.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:19:24.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.609466+0000", 2024-03-20T09:19:24.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.053013871999999997 2024-03-20T09:19:24.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:19:24.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.609475+0000", 2024-03-20T09:19:24.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7050000000000005e-06 2024-03-20T09:19:24.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:19:24.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.610609+0000", 2024-03-20T09:19:24.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011335620000000001 2024-03-20T09:19:24.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:19:24.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.610766+0000", 2024-03-20T09:19:24.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015706699999999999 2024-03-20T09:19:24.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5,6", 2024-03-20T09:19:24.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.610922+0000", 2024-03-20T09:19:24.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015589099999999999 2024-03-20T09:19:24.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:19:24.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.619052+0000", 2024-03-20T09:19:24.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0081304210000000005 2024-03-20T09:19:24.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:19:24.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.619109+0000", 2024-03-20T09:19:24.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6737999999999998e-05 2024-03-20T09:19:24.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:19:24.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.621713+0000", 2024-03-20T09:19:24.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026046239999999998 2024-03-20T09:19:24.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:19:24.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.621762+0000", 2024-03-20T09:19:24.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8479000000000001e-05 2024-03-20T09:19:24.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:19:24.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:55.621799+0000", 2024-03-20T09:19:24.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7469000000000003e-05 2024-03-20T09:19:24.613 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.613 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:24.613 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.613 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:78362 5.b 5:d15a6fb8:::benchmark_data_smithi122_157894_object78361:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e623)", 2024-03-20T09:19:24.613 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:19:00.277505+0000", 2024-03-20T09:19:24.613 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 24.303685764000001, 2024-03-20T09:19:24.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.057718553999999998, 2024-03-20T09:19:24.613 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:19:24.614 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:19:24.614 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:19:24.614 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:19:24.614 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:19:24.614 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 78362 2024-03-20T09:19:24.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.614 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:19:24.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:19:24.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:00.277505+0000", 2024-03-20T09:19:24.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:19:24.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:00.277505+0000", 2024-03-20T09:19:24.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:19:24.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:00.277507+0000", 2024-03-20T09:19:24.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3459999999999999e-06 2024-03-20T09:19:24.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:19:24.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:00.277571+0000", 2024-03-20T09:19:24.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4078999999999994e-05 2024-03-20T09:19:24.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:19:24.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:00.277582+0000", 2024-03-20T09:19:24.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1195999999999999e-05 2024-03-20T09:19:24.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:19:24.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:00.277589+0000", 2024-03-20T09:19:24.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6990000000000003e-06 2024-03-20T09:19:24.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:19:24.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:00.277642+0000", 2024-03-20T09:19:24.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2269999999999999e-05 2024-03-20T09:19:24.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:19:24.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:00.277781+0000", 2024-03-20T09:19:24.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000139216 2024-03-20T09:19:24.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5,7", 2024-03-20T09:19:24.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:00.277932+0000", 2024-03-20T09:19:24.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015058899999999999 2024-03-20T09:19:24.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:19:24.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:00.282419+0000", 2024-03-20T09:19:24.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0044875790000000002 2024-03-20T09:19:24.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:19:24.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:00.283915+0000", 2024-03-20T09:19:24.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001496067 2024-03-20T09:19:24.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:19:24.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:00.335102+0000", 2024-03-20T09:19:24.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.051187040000000003 2024-03-20T09:19:24.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:19:24.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:00.335171+0000", 2024-03-20T09:19:24.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8423000000000005e-05 2024-03-20T09:19:24.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:19:24.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:00.335224+0000", 2024-03-20T09:19:24.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3050000000000002e-05 2024-03-20T09:19:24.619 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.619 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:24.619 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.619 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:78374 5.3 5:c2a4560c:::benchmark_data_smithi122_157894_object78373:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e623)", 2024-03-20T09:19:24.619 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:19:00.285521+0000", 2024-03-20T09:19:24.619 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 24.295669451999999, 2024-03-20T09:19:24.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.059496985000000002, 2024-03-20T09:19:24.619 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:19:24.619 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:19:24.619 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:19:24.620 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:19:24.620 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:19:24.620 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 78374 2024-03-20T09:19:24.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.620 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:19:24.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:19:24.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:00.285521+0000", 2024-03-20T09:19:24.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:19:24.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:00.285521+0000", 2024-03-20T09:19:24.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:19:24.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:00.285522+0000", 2024-03-20T09:19:24.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2499999999999995e-07 2024-03-20T09:19:24.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:19:24.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:00.285656+0000", 2024-03-20T09:19:24.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133521 2024-03-20T09:19:24.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:19:24.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:00.285668+0000", 2024-03-20T09:19:24.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1970999999999999e-05 2024-03-20T09:19:24.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:19:24.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:00.285675+0000", 2024-03-20T09:19:24.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2910000000000002e-06 2024-03-20T09:19:24.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:19:24.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:00.336068+0000", 2024-03-20T09:19:24.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.050393457000000003 2024-03-20T09:19:24.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:19:24.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:00.336491+0000", 2024-03-20T09:19:24.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00042296400000000001 2024-03-20T09:19:24.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5,6", 2024-03-20T09:19:24.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:00.336616+0000", 2024-03-20T09:19:24.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012459299999999999 2024-03-20T09:19:24.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:19:24.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:00.341899+0000", 2024-03-20T09:19:24.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0052833960000000001 2024-03-20T09:19:24.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:19:24.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:00.344768+0000", 2024-03-20T09:19:24.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028683649999999999 2024-03-20T09:19:24.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:19:24.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:00.344917+0000", 2024-03-20T09:19:24.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014900300000000001 2024-03-20T09:19:24.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:19:24.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:00.344975+0000", 2024-03-20T09:19:24.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8449999999999998e-05 2024-03-20T09:19:24.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:19:24.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:00.345018+0000", 2024-03-20T09:19:24.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3448999999999997e-05 2024-03-20T09:19:24.625 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.625 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:24.625 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.625 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:99334 5.5 5:ac687fb7:::benchmark_data_smithi122_157894_object99333:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e628)", 2024-03-20T09:19:24.625 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:19:16.137325+0000", 2024-03-20T09:19:24.625 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 8.4438657880000001, 2024-03-20T09:19:24.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.050154008, 2024-03-20T09:19:24.625 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:19:24.625 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:19:24.625 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:19:24.626 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:19:24.626 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:19:24.626 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 99334 2024-03-20T09:19:24.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.626 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:19:24.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:19:24.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:16.137325+0000", 2024-03-20T09:19:24.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:19:24.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:16.137325+0000", 2024-03-20T09:19:24.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:19:24.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:16.137326+0000", 2024-03-20T09:19:24.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6300000000000004e-07 2024-03-20T09:19:24.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:19:24.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:16.137390+0000", 2024-03-20T09:19:24.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4386000000000003e-05 2024-03-20T09:19:24.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:19:24.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:16.137399+0000", 2024-03-20T09:19:24.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6999999999999997e-06 2024-03-20T09:19:24.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:19:24.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:16.137404+0000", 2024-03-20T09:19:24.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0719999999999999e-06 2024-03-20T09:19:24.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:19:24.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:16.137454+0000", 2024-03-20T09:19:24.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9561000000000002e-05 2024-03-20T09:19:24.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:19:24.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:16.137594+0000", 2024-03-20T09:19:24.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014038500000000001 2024-03-20T09:19:24.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 1,6", 2024-03-20T09:19:24.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:16.137790+0000", 2024-03-20T09:19:24.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000195594 2024-03-20T09:19:24.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:19:24.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:16.143759+0000", 2024-03-20T09:19:24.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059695310000000001 2024-03-20T09:19:24.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:19:24.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:16.144144+0000", 2024-03-20T09:19:24.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00038438200000000001 2024-03-20T09:19:24.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:19:24.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:16.187370+0000", 2024-03-20T09:19:24.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.043226613999999997 2024-03-20T09:19:24.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:19:24.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:16.187430+0000", 2024-03-20T09:19:24.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9449e-05 2024-03-20T09:19:24.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:19:24.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:16.187479+0000", 2024-03-20T09:19:24.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9471000000000001e-05 2024-03-20T09:19:24.631 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.631 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:24.631 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.631 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.25966.0:99351 5.0 e628/605 5:0dc97e4b:::benchmark_data_smithi122_157894_object99350:head v 628'6195, mlcod=628'6192)", 2024-03-20T09:19:24.631 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:19:16.151374+0000", 2024-03-20T09:19:24.631 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 8.4298166269999992, 2024-03-20T09:19:24.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.054295715000000001, 2024-03-20T09:19:24.631 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:19:24.632 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:19:24.632 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:19:24.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:19:24.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:16.151374+0000", 2024-03-20T09:19:24.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:19:24.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:16.151374+0000", 2024-03-20T09:19:24.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:19:24.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:16.151375+0000", 2024-03-20T09:19:24.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0900000000000001e-07 2024-03-20T09:19:24.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:19:24.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:16.151475+0000", 2024-03-20T09:19:24.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000100083 2024-03-20T09:19:24.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:19:24.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:16.151483+0000", 2024-03-20T09:19:24.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2789999999999998e-06 2024-03-20T09:19:24.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:19:24.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:16.151497+0000", 2024-03-20T09:19:24.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4195e-05 2024-03-20T09:19:24.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:19:24.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:16.151543+0000", 2024-03-20T09:19:24.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5433999999999999e-05 2024-03-20T09:19:24.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:19:24.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:16.151560+0000", 2024-03-20T09:19:24.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7085999999999999e-05 2024-03-20T09:19:24.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:19:24.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:16.205621+0000", 2024-03-20T09:19:24.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.054060902000000001 2024-03-20T09:19:24.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:19:24.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:16.205670+0000", 2024-03-20T09:19:24.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9126999999999999e-05 2024-03-20T09:19:24.635 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.635 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:24.635 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.636 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.25966.0:99347 5.8 e628/605 5:1b247a9c:::benchmark_data_smithi122_157894_object99346:head v 628'6141, mlcod=628'6136)", 2024-03-20T09:19:24.636 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:19:16.153607+0000", 2024-03-20T09:19:24.636 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 8.4275842010000002, 2024-03-20T09:19:24.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.057340749000000003, 2024-03-20T09:19:24.636 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:19:24.636 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:19:24.636 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:19:24.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:19:24.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:16.153607+0000", 2024-03-20T09:19:24.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:19:24.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:16.153607+0000", 2024-03-20T09:19:24.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:19:24.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:16.153607+0000", 2024-03-20T09:19:24.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1600000000000001e-07 2024-03-20T09:19:24.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:19:24.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:16.153662+0000", 2024-03-20T09:19:24.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4756999999999998e-05 2024-03-20T09:19:24.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:19:24.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:16.186918+0000", 2024-03-20T09:19:24.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.033256015999999999 2024-03-20T09:19:24.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:19:24.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:16.186929+0000", 2024-03-20T09:19:24.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0552e-05 2024-03-20T09:19:24.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:19:24.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:16.189498+0000", 2024-03-20T09:19:24.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0025687700000000002 2024-03-20T09:19:24.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:19:24.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:16.189516+0000", 2024-03-20T09:19:24.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:24.640 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 64 16 109138 109122 106.552 65.3125 0.00627539 0.00938095 2024-03-20T09:19:24.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:19:24.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8923000000000002e-05 2024-03-20T09:19:24.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:19:24.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:16.210899+0000", 2024-03-20T09:19:24.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021382526999999998 2024-03-20T09:19:24.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:19:24.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:16.210947+0000", 2024-03-20T09:19:24.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8488000000000001e-05 2024-03-20T09:19:24.642 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.642 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:24.642 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.642 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.25966.0:104891 5.8 e628/605 5:1193296d:::benchmark_data_smithi122_157894_object104890:head v 628'6497, mlcod=628'6495)", 2024-03-20T09:19:24.642 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:19:20.795746+0000", 2024-03-20T09:19:24.642 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 3.7854445490000002, 2024-03-20T09:19:24.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.055355878999999997, 2024-03-20T09:19:24.643 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:19:24.643 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:19:24.643 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:19:24.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:19:24.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:20.795746+0000", 2024-03-20T09:19:24.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:19:24.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:20.795746+0000", 2024-03-20T09:19:24.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:19:24.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:20.795747+0000", 2024-03-20T09:19:24.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5099999999999999e-07 2024-03-20T09:19:24.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:19:24.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:20.795839+0000", 2024-03-20T09:19:24.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1540999999999997e-05 2024-03-20T09:19:24.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.644 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:19:24.644 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:20.795848+0000", 2024-03-20T09:19:24.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2569999999999996e-06 2024-03-20T09:19:24.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:19:24.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:20.795854+0000", 2024-03-20T09:19:24.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.015e-06 2024-03-20T09:19:24.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:19:24.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:20.831202+0000", 2024-03-20T09:19:24.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.035348116999999998 2024-03-20T09:19:24.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:19:24.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:20.831224+0000", 2024-03-20T09:19:24.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2008e-05 2024-03-20T09:19:24.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:19:24.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:20.851049+0000", 2024-03-20T09:19:24.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.019825296999999999 2024-03-20T09:19:24.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:19:24.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:20.851102+0000", 2024-03-20T09:19:24.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2992999999999999e-05 2024-03-20T09:19:24.646 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.646 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:24.647 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.647 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.25966.0:104892 5.0 e628/605 5:0b9a2c00:::benchmark_data_smithi122_157894_object104891:head v 628'6524, mlcod=628'6521)", 2024-03-20T09:19:24.647 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:19:20.799521+0000", 2024-03-20T09:19:24.647 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 3.7816695060000001, 2024-03-20T09:19:24.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.051615823999999998, 2024-03-20T09:19:24.647 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:19:24.647 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:19:24.647 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:19:24.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:19:24.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:20.799521+0000", 2024-03-20T09:19:24.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:19:24.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:20.799521+0000", 2024-03-20T09:19:24.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:19:24.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:20.799523+0000", 2024-03-20T09:19:24.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.393e-06 2024-03-20T09:19:24.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:19:24.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:20.799580+0000", 2024-03-20T09:19:24.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7256000000000002e-05 2024-03-20T09:19:24.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:19:24.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:20.831072+0000", 2024-03-20T09:19:24.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031491926000000003 2024-03-20T09:19:24.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:19:24.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:20.831097+0000", 2024-03-20T09:19:24.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.529e-05 2024-03-20T09:19:24.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:19:24.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:20.831627+0000", 2024-03-20T09:19:24.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00053010100000000005 2024-03-20T09:19:24.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:19:24.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:20.831648+0000", 2024-03-20T09:19:24.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0805000000000002e-05 2024-03-20T09:19:24.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:19:24.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:20.851105+0000", 2024-03-20T09:19:24.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.019457131999999999 2024-03-20T09:19:24.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:19:24.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:20.851137+0000", 2024-03-20T09:19:24.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1921000000000002e-05 2024-03-20T09:19:24.651 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.651 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:24.651 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.651 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:105727 5.3 5:c16c4d6f:::benchmark_data_smithi122_157894_object105726:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e628)", 2024-03-20T09:19:24.651 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:19:21.466973+0000", 2024-03-20T09:19:24.652 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 3.1142184130000001, 2024-03-20T09:19:24.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.060685498999999997, 2024-03-20T09:19:24.652 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:19:24.652 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:19:24.652 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:19:24.652 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:19:24.652 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:19:24.652 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105727 2024-03-20T09:19:24.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.652 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:19:24.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:19:24.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.466973+0000", 2024-03-20T09:19:24.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:19:24.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.466973+0000", 2024-03-20T09:19:24.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:19:24.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.466974+0000", 2024-03-20T09:19:24.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1880000000000001e-06 2024-03-20T09:19:24.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:19:24.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.467036+0000", 2024-03-20T09:19:24.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2428999999999995e-05 2024-03-20T09:19:24.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:19:24.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.467047+0000", 2024-03-20T09:19:24.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0428000000000001e-05 2024-03-20T09:19:24.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:19:24.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.467053+0000", 2024-03-20T09:19:24.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6449999999999999e-06 2024-03-20T09:19:24.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:19:24.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.467102+0000", 2024-03-20T09:19:24.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8986000000000002e-05 2024-03-20T09:19:24.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:19:24.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.467238+0000", 2024-03-20T09:19:24.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013573 2024-03-20T09:19:24.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5,6", 2024-03-20T09:19:24.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.467372+0000", 2024-03-20T09:19:24.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013419800000000001 2024-03-20T09:19:24.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:19:24.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.470976+0000", 2024-03-20T09:19:24.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036040450000000002 2024-03-20T09:19:24.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:19:24.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.527521+0000", 2024-03-20T09:19:24.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.056544481000000001 2024-03-20T09:19:24.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:19:24.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.527556+0000", 2024-03-20T09:19:24.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5351000000000001e-05 2024-03-20T09:19:24.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:19:24.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.527614+0000", 2024-03-20T09:19:24.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8099000000000001e-05 2024-03-20T09:19:24.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:19:24.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.527658+0000", 2024-03-20T09:19:24.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3918999999999997e-05 2024-03-20T09:19:24.657 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.657 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:24.657 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.658 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:105732 5.b 5:d3cbcdef:::benchmark_data_smithi122_157894_object105731:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e628)", 2024-03-20T09:19:24.658 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:19:21.474215+0000", 2024-03-20T09:19:24.658 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 3.1069755140000002, 2024-03-20T09:19:24.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.053536963, 2024-03-20T09:19:24.658 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:19:24.658 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:19:24.658 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:19:24.658 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:19:24.658 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:19:24.658 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105732 2024-03-20T09:19:24.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.658 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:19:24.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:19:24.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.474215+0000", 2024-03-20T09:19:24.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:19:24.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.474215+0000", 2024-03-20T09:19:24.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:19:24.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.474217+0000", 2024-03-20T09:19:24.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6950000000000001e-06 2024-03-20T09:19:24.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:19:24.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.474265+0000", 2024-03-20T09:19:24.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8340999999999997e-05 2024-03-20T09:19:24.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:19:24.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.474278+0000", 2024-03-20T09:19:24.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2481e-05 2024-03-20T09:19:24.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:19:24.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.474286+0000", 2024-03-20T09:19:24.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6920000000000002e-06 2024-03-20T09:19:24.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:19:24.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.474364+0000", 2024-03-20T09:19:24.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8422000000000002e-05 2024-03-20T09:19:24.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:19:24.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.474498+0000", 2024-03-20T09:19:24.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013393199999999999 2024-03-20T09:19:24.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5,7", 2024-03-20T09:19:24.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.474633+0000", 2024-03-20T09:19:24.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000135186 2024-03-20T09:19:24.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:19:24.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.527516+0000", 2024-03-20T09:19:24.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.052883183 2024-03-20T09:19:24.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:19:24.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.527655+0000", 2024-03-20T09:19:24.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013837699999999999 2024-03-20T09:19:24.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:19:24.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.527666+0000", 2024-03-20T09:19:24.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1297999999999999e-05 2024-03-20T09:19:24.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:19:24.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.527714+0000", 2024-03-20T09:19:24.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8040000000000001e-05 2024-03-20T09:19:24.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:19:24.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.527752+0000", 2024-03-20T09:19:24.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8315999999999997e-05 2024-03-20T09:19:24.663 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.663 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:24.664 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.664 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:105733 5.3 5:c36493ae:::benchmark_data_smithi122_157894_object105732:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e628)", 2024-03-20T09:19:24.664 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:19:21.474308+0000", 2024-03-20T09:19:24.664 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 3.1068834110000001, 2024-03-20T09:19:24.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.054467623999999999, 2024-03-20T09:19:24.664 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:19:24.664 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:19:24.664 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:19:24.664 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:19:24.664 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:19:24.664 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 105733 2024-03-20T09:19:24.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.665 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:19:24.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:19:24.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.474308+0000", 2024-03-20T09:19:24.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:19:24.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.474308+0000", 2024-03-20T09:19:24.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:19:24.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.474308+0000", 2024-03-20T09:19:24.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8899999999999999e-07 2024-03-20T09:19:24.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:19:24.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.474378+0000", 2024-03-20T09:19:24.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0202000000000003e-05 2024-03-20T09:19:24.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:19:24.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.474388+0000", 2024-03-20T09:19:24.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2410000000000001e-06 2024-03-20T09:19:24.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:19:24.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.474395+0000", 2024-03-20T09:19:24.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4769999999999997e-06 2024-03-20T09:19:24.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:19:24.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.523785+0000", 2024-03-20T09:19:24.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.049390073 2024-03-20T09:19:24.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:19:24.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.523935+0000", 2024-03-20T09:19:24.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015025699999999999 2024-03-20T09:19:24.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 5,6", 2024-03-20T09:19:24.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.524076+0000", 2024-03-20T09:19:24.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000140338 2024-03-20T09:19:24.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:19:24.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.527854+0000", 2024-03-20T09:19:24.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0037780410000000002 2024-03-20T09:19:24.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:19:24.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.528378+0000", 2024-03-20T09:19:24.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00052388599999999999 2024-03-20T09:19:24.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:19:24.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.528686+0000", 2024-03-20T09:19:24.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00030789400000000002 2024-03-20T09:19:24.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:19:24.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.528739+0000", 2024-03-20T09:19:24.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3016999999999997e-05 2024-03-20T09:19:24.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:19:24.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.528775+0000", 2024-03-20T09:19:24.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6609000000000001e-05 2024-03-20T09:19:24.669 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.670 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:24.670 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.670 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop_reply(client.25966.0:105727 5.3 e628/605 ondisk, result = 0)", 2024-03-20T09:19:24.670 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:19:21.474777+0000", 2024-03-20T09:19:24.670 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 3.1064141869999999, 2024-03-20T09:19:24.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.052752403000000003, 2024-03-20T09:19:24.670 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:19:24.670 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:19:24.670 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:19:24.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:19:24.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.474777+0000", 2024-03-20T09:19:24.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:19:24.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.474777+0000", 2024-03-20T09:19:24.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:19:24.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.474777+0000", 2024-03-20T09:19:24.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0500000000000003e-07 2024-03-20T09:19:24.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:19:24.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.474781+0000", 2024-03-20T09:19:24.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8449999999999996e-06 2024-03-20T09:19:24.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:19:24.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.524770+0000", 2024-03-20T09:19:24.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.049988632999999998 2024-03-20T09:19:24.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:19:24.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.524779+0000", 2024-03-20T09:19:24.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8300000000000002e-06 2024-03-20T09:19:24.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:19:24.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.527506+0000", 2024-03-20T09:19:24.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002727602 2024-03-20T09:19:24.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:19:24.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.527514+0000", 2024-03-20T09:19:24.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8460000000000008e-06 2024-03-20T09:19:24.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:19:24.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.527529+0000", 2024-03-20T09:19:24.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5041999999999999e-05 2024-03-20T09:19:24.673 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.673 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:24.674 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.674 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.25966.0:105735 5.0 e628/605 5:094aa658:::benchmark_data_smithi122_157894_object105734:head v 628'6581, mlcod=628'6578)", 2024-03-20T09:19:24.674 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:19:21.474822+0000", 2024-03-20T09:19:24.674 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 3.1063691480000002, 2024-03-20T09:19:24.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.056749888999999998, 2024-03-20T09:19:24.674 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:19:24.674 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:19:24.674 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:19:24.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:19:24.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.474822+0000", 2024-03-20T09:19:24.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:19:24.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.474822+0000", 2024-03-20T09:19:24.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:19:24.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.474822+0000", 2024-03-20T09:19:24.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.82e-07 2024-03-20T09:19:24.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:19:24.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.474921+0000", 2024-03-20T09:19:24.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.8906999999999993e-05 2024-03-20T09:19:24.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:19:24.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.521647+0000", 2024-03-20T09:19:24.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046725714000000002 2024-03-20T09:19:24.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:19:24.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.521657+0000", 2024-03-20T09:19:24.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.6129999999999993e-06 2024-03-20T09:19:24.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:19:24.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.521821+0000", 2024-03-20T09:19:24.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00016463099999999999 2024-03-20T09:19:24.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:19:24.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.521844+0000", 2024-03-20T09:19:24.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2708e-05 2024-03-20T09:19:24.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:19:24.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.531520+0000", 2024-03-20T09:19:24.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0096762540000000005 2024-03-20T09:19:24.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:19:24.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.531572+0000", 2024-03-20T09:19:24.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1480000000000002e-05 2024-03-20T09:19:24.678 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.678 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:24.678 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.678 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop_reply(client.25966.0:105727 5.3 e628/605 ondisk, result = 0)", 2024-03-20T09:19:24.678 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:19:21.474888+0000", 2024-03-20T09:19:24.678 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 3.1063024810000002, 2024-03-20T09:19:24.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.052774902999999998, 2024-03-20T09:19:24.678 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:19:24.678 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:19:24.678 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:19:24.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:19:24.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.474888+0000", 2024-03-20T09:19:24.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:19:24.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.474888+0000", 2024-03-20T09:19:24.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:19:24.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.474889+0000", 2024-03-20T09:19:24.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4200000000000001e-07 2024-03-20T09:19:24.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:19:24.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.474893+0000", 2024-03-20T09:19:24.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.065e-06 2024-03-20T09:19:24.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:19:24.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.521466+0000", 2024-03-20T09:19:24.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046572716 2024-03-20T09:19:24.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:19:24.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.521478+0000", 2024-03-20T09:19:24.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2577e-05 2024-03-20T09:19:24.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:19:24.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.527545+0000", 2024-03-20T09:19:24.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0060669269999999997 2024-03-20T09:19:24.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:19:24.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.527551+0000", 2024-03-20T09:19:24.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8579999999999996e-06 2024-03-20T09:19:24.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:19:24.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:21.527663+0000", 2024-03-20T09:19:24.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000112318 2024-03-20T09:19:24.682 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.682 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:24.682 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.682 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.25966.0:107902 5.0 e628/605 5:0e8564a7:::benchmark_data_smithi122_157894_object107901:head v 628'6701, mlcod=628'6698)", 2024-03-20T09:19:24.682 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:19:23.456887+0000", 2024-03-20T09:19:24.682 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 1.1243041540000001, 2024-03-20T09:19:24.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.054054464000000003, 2024-03-20T09:19:24.682 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:19:24.682 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:19:24.682 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:19:24.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:19:24.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:23.456887+0000", 2024-03-20T09:19:24.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:19:24.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:23.456887+0000", 2024-03-20T09:19:24.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:19:24.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:23.456887+0000", 2024-03-20T09:19:24.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8999999999999996e-07 2024-03-20T09:19:24.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:19:24.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:23.456920+0000", 2024-03-20T09:19:24.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2583999999999998e-05 2024-03-20T09:19:24.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:19:24.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:23.456928+0000", 2024-03-20T09:19:24.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3280000000000006e-06 2024-03-20T09:19:24.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:19:24.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:23.456935+0000", 2024-03-20T09:19:24.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4849999999999997e-06 2024-03-20T09:19:24.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:19:24.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:23.457018+0000", 2024-03-20T09:19:24.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3203999999999998e-05 2024-03-20T09:19:24.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:19:24.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:23.457036+0000", 2024-03-20T09:19:24.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8167000000000001e-05 2024-03-20T09:19:24.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:19:24.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:23.510895+0000", 2024-03-20T09:19:24.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.053859356999999997 2024-03-20T09:19:24.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:19:24.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:23.510941+0000", 2024-03-20T09:19:24.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5748999999999999e-05 2024-03-20T09:19:24.686 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.686 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:24.686 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.686 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.25966.0:107907 5.8 e628/605 5:14391ff1:::benchmark_data_smithi122_157894_object107906:head v 628'6672, mlcod=628'6670)", 2024-03-20T09:19:24.686 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:19:23.460610+0000", 2024-03-20T09:19:24.686 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 1.120581429, 2024-03-20T09:19:24.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.058858534999999997, 2024-03-20T09:19:24.687 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:19:24.687 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:19:24.687 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:19:24.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:19:24.687 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:23.460610+0000", 2024-03-20T09:19:24.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:19:24.687 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:23.460610+0000", 2024-03-20T09:19:24.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:19:24.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:23.460610+0000", 2024-03-20T09:19:24.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.23e-07 2024-03-20T09:19:24.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:19:24.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:23.460666+0000", 2024-03-20T09:19:24.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5340000000000002e-05 2024-03-20T09:19:24.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:19:24.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:23.460677+0000", 2024-03-20T09:19:24.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1071e-05 2024-03-20T09:19:24.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:19:24.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:23.460684+0000", 2024-03-20T09:19:24.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9519999999999997e-06 2024-03-20T09:19:24.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:19:24.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:23.507302+0000", 2024-03-20T09:19:24.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046618865000000002 2024-03-20T09:19:24.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:19:24.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:23.507329+0000", 2024-03-20T09:19:24.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6084999999999998e-05 2024-03-20T09:19:24.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:19:24.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:23.519425+0000", 2024-03-20T09:19:24.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012096842999999999 2024-03-20T09:19:24.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:19:24.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:23.519468+0000", 2024-03-20T09:19:24.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2656e-05 2024-03-20T09:19:24.690 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.690 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:24.690 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.691 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:107913 5.5 5:abe6d6d5:::benchmark_data_smithi122_157894_object107912:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e628)", 2024-03-20T09:19:24.691 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:19:23.466482+0000", 2024-03-20T09:19:24.691 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 1.114709146, 2024-03-20T09:19:24.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.061554900000000003, 2024-03-20T09:19:24.691 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:19:24.691 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:19:24.691 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:19:24.691 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:19:24.691 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:19:24.691 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 107913 2024-03-20T09:19:24.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.691 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:19:24.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:19:24.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:23.466482+0000", 2024-03-20T09:19:24.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:19:24.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:23.466482+0000", 2024-03-20T09:19:24.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:19:24.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:19:24.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:23.466483+0000", 2024-03-20T09:19:24.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.609e-06 2024-03-20T09:19:24.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:19:24.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:23.466532+0000", 2024-03-20T09:19:24.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8810999999999997e-05 2024-03-20T09:19:24.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:19:24.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:23.507264+0000", 2024-03-20T09:19:24.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.040731478000000002 2024-03-20T09:19:24.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:19:24.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:23.507280+0000", 2024-03-20T09:19:24.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6138e-05 2024-03-20T09:19:24.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:19:24.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:23.507336+0000", 2024-03-20T09:19:24.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5797999999999997e-05 2024-03-20T09:19:24.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:19:24.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:23.507473+0000", 2024-03-20T09:19:24.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013698900000000001 2024-03-20T09:19:24.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 1,6", 2024-03-20T09:19:24.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:23.507617+0000", 2024-03-20T09:19:24.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014411 2024-03-20T09:19:24.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:19:24.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:23.518333+0000", 2024-03-20T09:19:24.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010716274 2024-03-20T09:19:24.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:19:24.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:23.527916+0000", 2024-03-20T09:19:24.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0095827489999999998 2024-03-20T09:19:24.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:19:24.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:23.527933+0000", 2024-03-20T09:19:24.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7e-05 2024-03-20T09:19:24.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:19:24.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:23.527987+0000", 2024-03-20T09:19:24.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4144000000000002e-05 2024-03-20T09:19:24.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:19:24.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:19:24.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:19:24.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:23.528037+0000", 2024-03-20T09:19:24.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9799999999999998e-05 2024-03-20T09:19:24.696 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.696 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:24.697 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.697 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:19:24.697 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:19:24.697 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:19:24.741 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:19:24.772 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:24.797 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:19:24.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:24.800 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:19:24.901 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:19:24.956 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:25.097 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:19:25.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:25.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:19:25.201 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:19:25.294 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:25.354 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:19:25.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:25.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:19:25.464 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:19:25.493 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:25.525 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:19:25.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:25.528 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:19:25.628 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 65 16 110324 110308 106.053 74.125 0.0112722 0.0094246 2024-03-20T09:19:25.629 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:19:25.675 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:25.702 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:19:25.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:25.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:19:25.807 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:19:25.840 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:25.875 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:19:25.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:25.881 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:19:25.956 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-20T09:19:25.982 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:19:26.035 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:26.066 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:19:26.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:26.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:19:26.175 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:19:26.256 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:26.369 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:19:26.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:26.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:19:26.475 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:19:26.515 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:26.549 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:19:26.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:26.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:19:26.628 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 66 15 111373 111358 105.441 65.625 0.0137026 0.00947934 2024-03-20T09:19:26.654 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:19:26.684 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:26.711 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:19:26.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:26.714 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:19:26.815 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:19:26.857 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:26.884 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:19:26.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:26.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:19:26.988 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:19:27.039 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:27.290 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:19:27.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:27 smithi115 ceph-mon[72476]: pgmap v1513: 47 pgs: 47 active+clean; 6.6 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-03-20T09:19:27.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:27 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1384065152' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:19:27.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:27 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2016553987' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.c", "id": [5, 11, 3]}]: dispatch 2024-03-20T09:19:27.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:27 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.c", "id": [5, 11, 3]}]: dispatch 2024-03-20T09:19:27.436 INFO:teuthology.orchestra.run.smithi026.stderr:set 5.c pg_upmap mapping to [5,11,3] 2024-03-20T09:19:27.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:27 smithi122 ceph-mon[73495]: pgmap v1513: 47 pgs: 47 active+clean; 6.6 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-03-20T09:19:27.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:27 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1384065152' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:19:27.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:27 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2016553987' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.c", "id": [5, 11, 3]}]: dispatch 2024-03-20T09:19:27.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:27 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.c", "id": [5, 11, 3]}]: dispatch 2024-03-20T09:19:27.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:27 smithi026 ceph-mon[68309]: pgmap v1513: 47 pgs: 47 active+clean; 6.6 GiB data, 24 GiB used, 870 GiB / 894 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-03-20T09:19:27.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:27 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1384065152' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:19:27.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:27 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2016553987' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.c", "id": [5, 11, 3]}]: dispatch 2024-03-20T09:19:27.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:27 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.c", "id": [5, 11, 3]}]: dispatch 2024-03-20T09:19:27.628 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 67 16 112059 112043 104.506 42.8125 0.00359565 0.00950497 2024-03-20T09:19:27.713 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:19:27.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:27.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:19:27.817 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:19:27.844 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:27.864 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:19:27.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:27.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:19:27.968 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:19:28.000 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:28.021 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:19:28.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:28.024 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:19:28.125 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:19:28.162 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:28.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:28 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:19:28.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:28 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1384065152' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:19:28.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:28 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.c", "id": [5, 11, 3]}]': finished 2024-03-20T09:19:28.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:28 smithi115 ceph-mon[72476]: osdmap e629: 12 total, 12 up, 10 in 2024-03-20T09:19:28.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:28 smithi115 ceph-mon[72476]: osdmap e630: 12 total, 12 up, 10 in 2024-03-20T09:19:28.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:28 smithi115 ceph-mon[72476]: pgmap v1516: 47 pgs: 1 unknown, 46 active+clean; 6.3 GiB data, 25 GiB used, 870 GiB / 894 GiB avail; 29 MiB/s wr, 468 op/s 2024-03-20T09:19:28.434 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:28 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:19:28.434 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:28 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1384065152' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:19:28.434 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:28 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.c", "id": [5, 11, 3]}]': finished 2024-03-20T09:19:28.434 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:28 smithi026 ceph-mon[68309]: osdmap e629: 12 total, 12 up, 10 in 2024-03-20T09:19:28.434 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:28 smithi026 ceph-mon[68309]: osdmap e630: 12 total, 12 up, 10 in 2024-03-20T09:19:28.434 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:28 smithi026 ceph-mon[68309]: pgmap v1516: 47 pgs: 1 unknown, 46 active+clean; 6.3 GiB data, 25 GiB used, 870 GiB / 894 GiB avail; 29 MiB/s wr, 468 op/s 2024-03-20T09:19:28.441 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:19:28.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:28.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:19:28.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:28 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:19:28.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:28 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1384065152' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:19:28.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:28 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.c", "id": [5, 11, 3]}]': finished 2024-03-20T09:19:28.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:28 smithi122 ceph-mon[73495]: osdmap e629: 12 total, 12 up, 10 in 2024-03-20T09:19:28.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:28 smithi122 ceph-mon[73495]: osdmap e630: 12 total, 12 up, 10 in 2024-03-20T09:19:28.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:28 smithi122 ceph-mon[73495]: pgmap v1516: 47 pgs: 1 unknown, 46 active+clean; 6.3 GiB data, 25 GiB used, 870 GiB / 894 GiB avail; 29 MiB/s wr, 468 op/s 2024-03-20T09:19:28.547 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:19:28.575 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:28.628 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 68 16 112059 112043 102.969 0 - 0.00950497 2024-03-20T09:19:28.629 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:19:28.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:28.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:19:28.733 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:19:28.760 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:28.780 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:19:28.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:28.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:19:28.886 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:19:28.914 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:28.940 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:19:28.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:28.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:19:29.044 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:19:29.077 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:29.106 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:19:29.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:29.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:19:29.215 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:19:29.244 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:29.274 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:19:29.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:29.281 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:19:29.382 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:19:29.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:29 smithi115 ceph-mon[72476]: osdmap e631: 12 total, 12 up, 10 in 2024-03-20T09:19:29.455 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:29.532 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:19:29.543 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:29 smithi026 ceph-mon[68309]: osdmap e631: 12 total, 12 up, 10 in 2024-03-20T09:19:29.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:29.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:19:29.628 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 69 16 112843 112827 102.187 24.5 0.00492875 0.00978104 2024-03-20T09:19:29.646 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:19:29.682 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:29.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:29 smithi122 ceph-mon[73495]: osdmap e631: 12 total, 12 up, 10 in 2024-03-20T09:19:29.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:29.717 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:19:29.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:19:29.818 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:19:29.854 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:29.884 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:19:29.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:29.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:19:29.988 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:19:30.034 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:30.127 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:19:30.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:30.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:19:30.166 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:19:30.232 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:19:30.268 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:30.302 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:19:30.305 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:30 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:19:30.305 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:30 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:19:30.305 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:30 smithi026 ceph-mon[68309]: pgmap v1518: 47 pgs: 1 unknown, 46 active+clean; 6.4 GiB data, 25 GiB used, 869 GiB / 894 GiB avail 2024-03-20T09:19:30.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:30.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:19:30.407 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:19:30.441 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:30.471 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:19:30.478 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:30.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:19:30.579 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:19:30.616 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:30 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:19:30.616 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:30 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:19:30.616 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:30 smithi122 ceph-mon[73495]: pgmap v1518: 47 pgs: 1 unknown, 46 active+clean; 6.4 GiB data, 25 GiB used, 869 GiB / 894 GiB avail 2024-03-20T09:19:30.617 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:30.629 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 70 16 113771 113755 101.554 58 0.00545156 0.00984232 2024-03-20T09:19:30.643 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:19:30.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:30.646 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:19:30.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:30 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:19:30.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:30 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:19:30.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:30 smithi115 ceph-mon[72476]: pgmap v1518: 47 pgs: 1 unknown, 46 active+clean; 6.4 GiB data, 25 GiB used, 869 GiB / 894 GiB avail 2024-03-20T09:19:30.747 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:19:30.825 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:30.864 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:19:30.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:30.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:19:30.905 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:19:30.937 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:19:30.971 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:19:31.002 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:31.025 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:19:31.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:31.030 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:19:31.131 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:19:31.172 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:31.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:31 smithi026 ceph-mon[68309]: osdmap e632: 12 total, 12 up, 10 in 2024-03-20T09:19:31.630 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 71 16 114634 114618 100.882 53.9375 0.00547536 0.00990809 2024-03-20T09:19:31.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:31 smithi115 ceph-mon[72476]: osdmap e632: 12 total, 12 up, 10 in 2024-03-20T09:19:31.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:31 smithi122 ceph-mon[73495]: osdmap e632: 12 total, 12 up, 10 in 2024-03-20T09:19:32.478 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:32 smithi026 ceph-mon[68309]: pgmap v1520: 47 pgs: 1 unknown, 46 active+clean; 6.5 GiB data, 25 GiB used, 869 GiB / 894 GiB avail 2024-03-20T09:19:32.631 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 72 16 115541 115525 100.268 56.6875 0.00982664 0.00996673 2024-03-20T09:19:32.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:32 smithi115 ceph-mon[72476]: pgmap v1520: 47 pgs: 1 unknown, 46 active+clean; 6.5 GiB data, 25 GiB used, 869 GiB / 894 GiB avail 2024-03-20T09:19:32.699 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:32 smithi122 ceph-mon[73495]: pgmap v1520: 47 pgs: 1 unknown, 46 active+clean; 6.5 GiB data, 25 GiB used, 869 GiB / 894 GiB avail 2024-03-20T09:19:33.630 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 73 16 116412 116396 99.6402 54.4375 0.016508 0.0100314 2024-03-20T09:19:33.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:33 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:19:33.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:33 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:19:33.700 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:33 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:19:33.700 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:33 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:19:33.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:33 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:19:33.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:33 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:19:34.091 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:19:34.137 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:19:34.184 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:19:34.223 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:19:34.257 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:19:34.323 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:19:34.324 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:19:34.331 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:19:34.370 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:19:34.404 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:19:34.442 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:19:34.489 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:19:34.526 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:19:34.527 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:19:34.527 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:19:34.527 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:19:34.527 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:19:34.527 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:19:34.527 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:19:34.527 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:19:34.527 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:19:34.527 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:19:34.527 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:19:34.527 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:19:34.631 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 74 16 117292 117276 99.0369 55 0.00753156 0.0100899 2024-03-20T09:19:34.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:34 smithi115 ceph-mon[72476]: osdmap e633: 12 total, 12 up, 10 in 2024-03-20T09:19:34.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:34 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3261016616' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:19:34.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:34 smithi115 ceph-mon[72476]: pgmap v1522: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.0 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 108 MiB/s wr, 1.73k op/s; 21003/342885 objects misplaced (6.125%); 405 KiB/s, 6 objects/s recovering 2024-03-20T09:19:34.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:34 smithi122 ceph-mon[73495]: osdmap e633: 12 total, 12 up, 10 in 2024-03-20T09:19:34.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:34 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3261016616' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:19:34.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:34 smithi122 ceph-mon[73495]: pgmap v1522: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.0 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 108 MiB/s wr, 1.73k op/s; 21003/342885 objects misplaced (6.125%); 405 KiB/s, 6 objects/s recovering 2024-03-20T09:19:34.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:34 smithi026 ceph-mon[68309]: osdmap e633: 12 total, 12 up, 10 in 2024-03-20T09:19:34.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:34 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3261016616' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:19:34.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:34 smithi026 ceph-mon[68309]: pgmap v1522: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.0 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 108 MiB/s wr, 1.73k op/s; 21003/342885 objects misplaced (6.125%); 405 KiB/s, 6 objects/s recovering 2024-03-20T09:19:35.188 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 8, 10, 11, 3] out_osds: [9, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 5] 2024-03-20T09:19:35.188 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:19:35.188 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-03-20T09:19:35.189 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd in 4 2024-03-20T09:19:35.631 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 75 16 118146 118130 98.4281 53.375 0.0053562 0.0101537 2024-03-20T09:19:36.347 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:19:36.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:36.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:19:36.452 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:19:36.505 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:36.614 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:36 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:19:36.614 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:36 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:19:36.614 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:36 smithi026 ceph-mon[68309]: osdmap e634: 12 total, 12 up, 10 in 2024-03-20T09:19:36.614 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:36 smithi026 ceph-mon[68309]: pgmap v1524: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.1 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 123 MiB/s wr, 1.97k op/s; 21003/348618 objects misplaced (6.025%); 405 KiB/s, 6 objects/s recovering 2024-03-20T09:19:36.631 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 76 16 119029 119013 97.8591 55.1875 0.00904014 0.0102093 2024-03-20T09:19:36.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:36 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:19:36.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:36 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:19:36.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:36 smithi115 ceph-mon[72476]: osdmap e634: 12 total, 12 up, 10 in 2024-03-20T09:19:36.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:36 smithi115 ceph-mon[72476]: pgmap v1524: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.1 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 123 MiB/s wr, 1.97k op/s; 21003/348618 objects misplaced (6.025%); 405 KiB/s, 6 objects/s recovering 2024-03-20T09:19:36.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:36 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:19:36.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:36 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:19:36.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:36 smithi122 ceph-mon[73495]: osdmap e634: 12 total, 12 up, 10 in 2024-03-20T09:19:36.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:36 smithi122 ceph-mon[73495]: pgmap v1524: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.1 GiB data, 25 GiB used, 869 GiB / 894 GiB avail; 123 MiB/s wr, 1.97k op/s; 21003/348618 objects misplaced (6.025%); 405 KiB/s, 6 objects/s recovering 2024-03-20T09:19:37.029 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:19:37.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:37.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:19:37.136 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:19:37.165 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:37.631 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 77 16 119708 119692 97.1393 42.4375 0.00465612 0.0102739 2024-03-20T09:19:37.641 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:19:37.641 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:19:38.631 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 78 16 120412 120396 96.458 44 0.0121156 0.0103632 2024-03-20T09:19:38.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:38 smithi115 ceph-mon[72476]: pgmap v1525: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.2 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 112 MiB/s wr, 1.79k op/s; 20832/355956 objects misplaced (5.852%); 785 KiB/s, 12 objects/s recovering 2024-03-20T09:19:38.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:38 smithi122 ceph-mon[73495]: pgmap v1525: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.2 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 112 MiB/s wr, 1.79k op/s; 20832/355956 objects misplaced (5.852%); 785 KiB/s, 12 objects/s recovering 2024-03-20T09:19:38.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:38 smithi026 ceph-mon[68309]: pgmap v1525: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.2 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 112 MiB/s wr, 1.79k op/s; 20832/355956 objects misplaced (5.852%); 785 KiB/s, 12 objects/s recovering 2024-03-20T09:19:39.631 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 79 16 121361 121345 95.9878 59.3125 0.078139 0.0104136 2024-03-20T09:19:40.631 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:2024-03-20T09:19:40.631364+0000 min lat: 0.0010551 max lat: 1.69788 avg lat: 0.0104616 2024-03-20T09:19:40.631 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T09:19:40.632 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 80 16 122329 122313 95.5442 60.5 0.00913574 0.0104616 2024-03-20T09:19:41.042 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:19:41.054 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-20T09:19:41.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:41.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:19:41.156 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:19:41.194 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:41.228 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:19:41.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:41.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:19:41.336 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:19:41.367 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:41 smithi122 ceph-mon[73495]: pgmap v1526: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.3 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 102 MiB/s wr, 1.64k op/s; 20832/357504 objects misplaced (5.827%); 759 KiB/s, 11 objects/s recovering 2024-03-20T09:19:41.369 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:41.375 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:19:41.391 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:19:41.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:41.394 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:19:41.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:41 smithi115 ceph-mon[72476]: pgmap v1526: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.3 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 102 MiB/s wr, 1.64k op/s; 20832/357504 objects misplaced (5.827%); 759 KiB/s, 11 objects/s recovering 2024-03-20T09:19:41.479 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:41 smithi026 ceph-mon[68309]: pgmap v1526: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.3 GiB data, 26 GiB used, 868 GiB / 894 GiB avail; 102 MiB/s wr, 1.64k op/s; 20832/357504 objects misplaced (5.827%); 759 KiB/s, 11 objects/s recovering 2024-03-20T09:19:41.494 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:19:41.526 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:41.559 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:19:41.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:41.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:19:41.631 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 81 16 123148 123132 94.9965 51.1875 0.0730001 0.0105226 2024-03-20T09:19:41.663 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:19:41.699 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:41.724 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:19:41.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:41.728 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:19:41.829 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:19:41.858 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:41.879 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:19:41.882 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:41.883 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:19:41.984 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:19:42.017 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:42.056 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:19:42.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:42.061 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:19:42.148 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:19:42.162 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:19:42.190 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:42.216 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:19:42.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:42.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:19:42.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:42 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/15620008' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:19:42.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:42 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:19:42.300 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:19:42.320 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:19:42.351 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:42 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/15620008' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:19:42.351 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:42 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:19:42.353 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:42.376 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:19:42.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:42.380 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:19:42.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:42 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/15620008' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:19:42.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:42 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:19:42.481 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:19:42.578 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:42.631 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 82 16 123952 123936 94.4508 50.25 0.00856461 0.0105817 2024-03-20T09:19:43.134 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:43 smithi026 ceph-mon[68309]: pgmap v1527: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.4 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 50 MiB/s wr, 799 op/s; 20832/363762 objects misplaced (5.727%); 419 KiB/s, 6 objects/s recovering 2024-03-20T09:19:43.134 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:43 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:19:43.134 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:43 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:19:43.135 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:43 smithi026 ceph-mon[68309]: osdmap e635: 12 total, 12 up, 10 in 2024-03-20T09:19:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:43 smithi115 ceph-mon[72476]: pgmap v1527: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.4 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 50 MiB/s wr, 799 op/s; 20832/363762 objects misplaced (5.727%); 419 KiB/s, 6 objects/s recovering 2024-03-20T09:19:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:43 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:19:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:43 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:19:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:43 smithi115 ceph-mon[72476]: osdmap e635: 12 total, 12 up, 10 in 2024-03-20T09:19:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:43 smithi122 ceph-mon[73495]: pgmap v1527: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.4 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 50 MiB/s wr, 799 op/s; 20832/363762 objects misplaced (5.727%); 419 KiB/s, 6 objects/s recovering 2024-03-20T09:19:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:43 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:19:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:43 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:19:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:43 smithi122 ceph-mon[73495]: osdmap e635: 12 total, 12 up, 10 in 2024-03-20T09:19:43.632 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 83 16 124815 124799 93.9626 53.9375 0.00523182 0.0106308 2024-03-20T09:19:44.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:44 smithi122 ceph-mon[73495]: pgmap v1529: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.5 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 53 MiB/s wr, 848 op/s; 20639/368373 objects misplaced (5.603%); 893 KiB/s, 13 objects/s recovering 2024-03-20T09:19:44.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:44 smithi026 ceph-mon[68309]: pgmap v1529: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.5 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 53 MiB/s wr, 848 op/s; 20639/368373 objects misplaced (5.603%); 893 KiB/s, 13 objects/s recovering 2024-03-20T09:19:44.632 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 84 16 125732 125716 93.5263 57.3125 0.010089 0.0106867 2024-03-20T09:19:44.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:44 smithi115 ceph-mon[72476]: pgmap v1529: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 7.5 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 53 MiB/s wr, 848 op/s; 20639/368373 objects misplaced (5.603%); 893 KiB/s, 13 objects/s recovering 2024-03-20T09:19:44.890 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:19:44.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:44.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:19:44.997 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:19:45.063 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:45.335 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:19:45.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:45.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:19:45.437 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:19:45.444 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:19:45.459 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:19:45.476 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:45.498 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:19:45.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:45.502 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:19:45.603 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:19:45.632 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 85 16 126576 126560 93.0465 52.75 0.0138917 0.010742 2024-03-20T09:19:45.648 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:45 smithi115 ceph-mon[72476]: 5.9 scrub starts 2024-03-20T09:19:45.650 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:45.686 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:19:45.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:45.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:19:45.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:45 smithi122 ceph-mon[73495]: 5.9 scrub starts 2024-03-20T09:19:45.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:45 smithi026 ceph-mon[68309]: 5.9 scrub starts 2024-03-20T09:19:45.796 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:19:45.858 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:46.632 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 86 16 127314 127298 92.5009 46.125 0.120516 0.0107996 2024-03-20T09:19:46.646 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:19:46.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:46 smithi115 ceph-mon[72476]: pgmap v1530: 47 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 45 active+clean; 7.6 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 53 MiB/s wr, 843 op/s; 20639/373932 objects misplaced (5.519%); 774 KiB/s, 12 objects/s recovering 2024-03-20T09:19:46.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:46 smithi115 ceph-mon[72476]: 5.2 scrub starts 2024-03-20T09:19:46.686 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:46 smithi026 ceph-mon[68309]: pgmap v1530: 47 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 45 active+clean; 7.6 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 53 MiB/s wr, 843 op/s; 20639/373932 objects misplaced (5.519%); 774 KiB/s, 12 objects/s recovering 2024-03-20T09:19:46.686 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:46 smithi026 ceph-mon[68309]: 5.2 scrub starts 2024-03-20T09:19:46.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:46 smithi122 ceph-mon[73495]: pgmap v1530: 47 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 45 active+clean; 7.6 GiB data, 27 GiB used, 867 GiB / 894 GiB avail; 53 MiB/s wr, 843 op/s; 20639/373932 objects misplaced (5.519%); 774 KiB/s, 12 objects/s recovering 2024-03-20T09:19:46.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:46 smithi122 ceph-mon[73495]: 5.2 scrub starts 2024-03-20T09:19:47.040 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:19:47.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:47.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:19:47.149 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:19:47.210 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:47.242 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:19:47.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:47.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:19:47.348 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:19:47.409 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:47.632 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 87 16 127998 127982 91.9291 42.75 0.0201475 0.010873 2024-03-20T09:19:47.660 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:47 smithi115 ceph-mon[72476]: 1.17 scrub starts 2024-03-20T09:19:47.660 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:47 smithi115 ceph-mon[72476]: 1.17 scrub ok 2024-03-20T09:19:47.660 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:47 smithi115 ceph-mon[72476]: 1.14 scrub starts 2024-03-20T09:19:47.660 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:47 smithi115 ceph-mon[72476]: 1.14 scrub ok 2024-03-20T09:19:47.660 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:47 smithi115 ceph-mon[72476]: 5.9 scrub ok 2024-03-20T09:19:47.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:47 smithi122 ceph-mon[73495]: 1.17 scrub starts 2024-03-20T09:19:47.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:47 smithi122 ceph-mon[73495]: 1.17 scrub ok 2024-03-20T09:19:47.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:47 smithi122 ceph-mon[73495]: 1.14 scrub starts 2024-03-20T09:19:47.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:47 smithi122 ceph-mon[73495]: 1.14 scrub ok 2024-03-20T09:19:47.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:47 smithi122 ceph-mon[73495]: 5.9 scrub ok 2024-03-20T09:19:47.724 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:47 smithi026 ceph-mon[68309]: 1.17 scrub starts 2024-03-20T09:19:47.724 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:47 smithi026 ceph-mon[68309]: 1.17 scrub ok 2024-03-20T09:19:47.724 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:47 smithi026 ceph-mon[68309]: 1.14 scrub starts 2024-03-20T09:19:47.724 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:47 smithi026 ceph-mon[68309]: 1.14 scrub ok 2024-03-20T09:19:47.724 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:47 smithi026 ceph-mon[68309]: 5.9 scrub ok 2024-03-20T09:19:48.085 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:19:48.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:48.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:19:48.192 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:19:48.255 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:48.632 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 88 16 128747 128731 91.4164 46.8125 0.0082762 0.0109336 2024-03-20T09:19:48.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:48 smithi115 ceph-mon[72476]: 5.a scrub starts 2024-03-20T09:19:48.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:48 smithi115 ceph-mon[72476]: 1.1d scrub starts 2024-03-20T09:19:48.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:48 smithi115 ceph-mon[72476]: 1.1d scrub ok 2024-03-20T09:19:48.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:48 smithi115 ceph-mon[72476]: 5.2 scrub ok 2024-03-20T09:19:48.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:48 smithi115 ceph-mon[72476]: pgmap v1531: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 45 active+clean; 7.8 GiB data, 28 GiB used, 866 GiB / 894 GiB avail; 52 MiB/s wr, 836 op/s; 20453/381048 objects misplaced (5.368%); 806 KiB/s, 12 objects/s recovering 2024-03-20T09:19:48.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:48 smithi122 ceph-mon[73495]: 5.a scrub starts 2024-03-20T09:19:48.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:48 smithi122 ceph-mon[73495]: 1.1d scrub starts 2024-03-20T09:19:48.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:48 smithi122 ceph-mon[73495]: 1.1d scrub ok 2024-03-20T09:19:48.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:48 smithi122 ceph-mon[73495]: 5.2 scrub ok 2024-03-20T09:19:48.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:48 smithi122 ceph-mon[73495]: pgmap v1531: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 45 active+clean; 7.8 GiB data, 28 GiB used, 866 GiB / 894 GiB avail; 52 MiB/s wr, 836 op/s; 20453/381048 objects misplaced (5.368%); 806 KiB/s, 12 objects/s recovering 2024-03-20T09:19:48.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:48 smithi026 ceph-mon[68309]: 5.a scrub starts 2024-03-20T09:19:48.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:48 smithi026 ceph-mon[68309]: 1.1d scrub starts 2024-03-20T09:19:48.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:48 smithi026 ceph-mon[68309]: 1.1d scrub ok 2024-03-20T09:19:48.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:48 smithi026 ceph-mon[68309]: 5.2 scrub ok 2024-03-20T09:19:48.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:48 smithi026 ceph-mon[68309]: pgmap v1531: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 45 active+clean; 7.8 GiB data, 28 GiB used, 866 GiB / 894 GiB avail; 52 MiB/s wr, 836 op/s; 20453/381048 objects misplaced (5.368%); 806 KiB/s, 12 objects/s recovering 2024-03-20T09:19:49.238 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:19:49.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:49.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:19:49.343 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:19:49.371 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:49.394 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:19:49.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:49.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:19:49.499 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:19:49.532 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:49.554 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:19:49.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:49.557 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:19:49.633 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 89 16 129397 129381 90.8457 40.625 0.00573249 0.0110022 2024-03-20T09:19:49.647 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:49 smithi026 ceph-mon[68309]: 5.6 scrub starts 2024-03-20T09:19:49.647 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:49 smithi026 ceph-mon[68309]: 5.0 scrub starts 2024-03-20T09:19:49.647 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:49 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mon-a[68286]: 2024-03-20T09:19:49.642+0000 7f683364b700 -1 mon.a@0(leader).osd e635 definitely_dead 0 2024-03-20T09:19:49.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:49 smithi115 ceph-mon[72476]: 5.6 scrub starts 2024-03-20T09:19:49.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:49 smithi115 ceph-mon[72476]: 5.0 scrub starts 2024-03-20T09:19:49.658 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:19:49.687 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:49.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:49 smithi122 ceph-mon[73495]: 5.6 scrub starts 2024-03-20T09:19:49.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:49 smithi122 ceph-mon[73495]: 5.0 scrub starts 2024-03-20T09:19:49.713 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:19:49.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:49.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:19:49.820 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:19:49.859 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:49.898 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:19:49.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:49.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:19:50.005 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:19:50.035 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:50.059 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:19:50.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:50.064 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:19:50.165 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:19:50.221 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:50.397 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:50 smithi026 ceph-mon[68309]: 1.d scrub starts 2024-03-20T09:19:50.397 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:50 smithi026 ceph-mon[68309]: 1.d scrub ok 2024-03-20T09:19:50.398 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:50 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/407535123' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-03-20T09:19:50.398 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:50 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-03-20T09:19:50.398 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:50 smithi026 ceph-mon[68309]: 5.a scrub ok 2024-03-20T09:19:50.398 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:50 smithi026 ceph-mon[68309]: pgmap v1532: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+clean+scrubbing, 43 active+clean; 7.8 GiB data, 28 GiB used, 866 GiB / 894 GiB avail; 52 MiB/s wr, 829 op/s; 20453/382401 objects misplaced (5.349%); 806 KiB/s, 12 objects/s recovering 2024-03-20T09:19:50.398 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:50 smithi026 ceph-mon[68309]: 5.0 scrub ok 2024-03-20T09:19:50.399 INFO:teuthology.orchestra.run.smithi026.stderr:marked in osd.4. 2024-03-20T09:19:50.632 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 90 16 129984 129968 90.2439 36.6875 0.00405268 0.0110518 2024-03-20T09:19:50.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:50 smithi115 ceph-mon[72476]: 1.d scrub starts 2024-03-20T09:19:50.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:50 smithi115 ceph-mon[72476]: 1.d scrub ok 2024-03-20T09:19:50.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:50 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/407535123' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-03-20T09:19:50.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:50 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-03-20T09:19:50.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:50 smithi115 ceph-mon[72476]: 5.a scrub ok 2024-03-20T09:19:50.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:50 smithi115 ceph-mon[72476]: pgmap v1532: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+clean+scrubbing, 43 active+clean; 7.8 GiB data, 28 GiB used, 866 GiB / 894 GiB avail; 52 MiB/s wr, 829 op/s; 20453/382401 objects misplaced (5.349%); 806 KiB/s, 12 objects/s recovering 2024-03-20T09:19:50.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:50 smithi115 ceph-mon[72476]: 5.0 scrub ok 2024-03-20T09:19:50.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:50 smithi122 ceph-mon[73495]: 1.d scrub starts 2024-03-20T09:19:50.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:50 smithi122 ceph-mon[73495]: 1.d scrub ok 2024-03-20T09:19:50.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:50 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/407535123' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-03-20T09:19:50.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:50 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-03-20T09:19:50.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:50 smithi122 ceph-mon[73495]: 5.a scrub ok 2024-03-20T09:19:50.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:50 smithi122 ceph-mon[73495]: pgmap v1532: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+clean+scrubbing, 43 active+clean; 7.8 GiB data, 28 GiB used, 866 GiB / 894 GiB avail; 52 MiB/s wr, 829 op/s; 20453/382401 objects misplaced (5.349%); 806 KiB/s, 12 objects/s recovering 2024-03-20T09:19:50.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:50 smithi122 ceph-mon[73495]: 5.0 scrub ok 2024-03-20T09:19:51.097 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:19:51.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:51.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:19:51.203 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:19:51.220 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:19:51.230 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:51.251 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:19:51.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:51.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:19:51.355 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:19:51.399 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:51 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-03-20T09:19:51.399 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:51 smithi026 ceph-mon[68309]: osdmap e636: 12 total, 12 up, 11 in 2024-03-20T09:19:51.399 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:51 smithi026 ceph-mon[68309]: 5.6 scrub ok 2024-03-20T09:19:51.399 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:51 smithi026 ceph-mon[68309]: osdmap e637: 12 total, 12 up, 11 in 2024-03-20T09:19:51.409 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:51.452 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:19:51.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:51.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:19:51.559 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:19:51.589 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:51.632 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 91 16 129984 129968 89.2522 0 - 0.0110518 2024-03-20T09:19:51.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:51 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-03-20T09:19:51.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:51 smithi115 ceph-mon[72476]: osdmap e636: 12 total, 12 up, 11 in 2024-03-20T09:19:51.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:51 smithi115 ceph-mon[72476]: 5.6 scrub ok 2024-03-20T09:19:51.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:51 smithi115 ceph-mon[72476]: osdmap e637: 12 total, 12 up, 11 in 2024-03-20T09:19:51.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:51 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-03-20T09:19:51.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:51 smithi122 ceph-mon[73495]: osdmap e636: 12 total, 12 up, 11 in 2024-03-20T09:19:51.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:51 smithi122 ceph-mon[73495]: 5.6 scrub ok 2024-03-20T09:19:51.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:51 smithi122 ceph-mon[73495]: osdmap e637: 12 total, 12 up, 11 in 2024-03-20T09:19:51.931 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:19:51.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:51.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:19:51.963 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-03-20T09:19:52.035 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:19:52.063 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:52.086 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:19:52.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:52.091 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:19:52.192 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:19:52.217 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:52.237 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:19:52.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:52.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:19:52.341 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:19:52.374 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:52.476 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:19:52.476 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:19:52.632 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 92 16 129984 129968 88.2821 0 - 0.0110518 2024-03-20T09:19:52.642 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:19:52.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:52.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:19:52.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:52 smithi115 ceph-mon[72476]: 5.e scrub starts 2024-03-20T09:19:52.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:52 smithi115 ceph-mon[72476]: 1.16 scrub starts 2024-03-20T09:19:52.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:52 smithi115 ceph-mon[72476]: 1.9 scrub starts 2024-03-20T09:19:52.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:52 smithi115 ceph-mon[72476]: 1.16 scrub ok 2024-03-20T09:19:52.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:52 smithi115 ceph-mon[72476]: 1.9 scrub ok 2024-03-20T09:19:52.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:52 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:19:52.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:52 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:19:52.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:52 smithi115 ceph-mon[72476]: 1.15 scrub starts 2024-03-20T09:19:52.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:52 smithi115 ceph-mon[72476]: 1.15 scrub ok 2024-03-20T09:19:52.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:52 smithi115 ceph-mon[72476]: 1.7 scrub starts 2024-03-20T09:19:52.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:52 smithi115 ceph-mon[72476]: pgmap v1535: 47 pgs: 5 unknown, 3 peering, 1 active+recovering+undersized+remapped, 3 active+clean+scrubbing, 35 active+clean; 7.4 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 20453/363051 objects misplaced (5.634%); 496 KiB/s, 7 objects/s recovering 2024-03-20T09:19:52.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:52 smithi115 ceph-mon[72476]: 1.7 scrub ok 2024-03-20T09:19:52.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:52 smithi115 ceph-mon[72476]: osdmap e638: 12 total, 12 up, 11 in 2024-03-20T09:19:52.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:52 smithi026 ceph-mon[68309]: 5.e scrub starts 2024-03-20T09:19:52.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:52 smithi026 ceph-mon[68309]: 1.16 scrub starts 2024-03-20T09:19:52.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:52 smithi026 ceph-mon[68309]: 1.9 scrub starts 2024-03-20T09:19:52.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:52 smithi026 ceph-mon[68309]: 1.16 scrub ok 2024-03-20T09:19:52.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:52 smithi026 ceph-mon[68309]: 1.9 scrub ok 2024-03-20T09:19:52.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:52 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:19:52.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:52 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:19:52.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:52 smithi026 ceph-mon[68309]: 1.15 scrub starts 2024-03-20T09:19:52.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:52 smithi026 ceph-mon[68309]: 1.15 scrub ok 2024-03-20T09:19:52.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:52 smithi026 ceph-mon[68309]: 1.7 scrub starts 2024-03-20T09:19:52.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:52 smithi026 ceph-mon[68309]: pgmap v1535: 47 pgs: 5 unknown, 3 peering, 1 active+recovering+undersized+remapped, 3 active+clean+scrubbing, 35 active+clean; 7.4 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 20453/363051 objects misplaced (5.634%); 496 KiB/s, 7 objects/s recovering 2024-03-20T09:19:52.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:52 smithi026 ceph-mon[68309]: 1.7 scrub ok 2024-03-20T09:19:52.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:52 smithi026 ceph-mon[68309]: osdmap e638: 12 total, 12 up, 11 in 2024-03-20T09:19:52.746 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:19:52.771 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:52 smithi122 ceph-mon[73495]: 5.e scrub starts 2024-03-20T09:19:52.771 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:52 smithi122 ceph-mon[73495]: 1.16 scrub starts 2024-03-20T09:19:52.771 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:52 smithi122 ceph-mon[73495]: 1.9 scrub starts 2024-03-20T09:19:52.771 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:52 smithi122 ceph-mon[73495]: 1.16 scrub ok 2024-03-20T09:19:52.771 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:52 smithi122 ceph-mon[73495]: 1.9 scrub ok 2024-03-20T09:19:52.771 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:52 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:19:52.771 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:52 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:19:52.771 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:52 smithi122 ceph-mon[73495]: 1.15 scrub starts 2024-03-20T09:19:52.771 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:52 smithi122 ceph-mon[73495]: 1.15 scrub ok 2024-03-20T09:19:52.771 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:52 smithi122 ceph-mon[73495]: 1.7 scrub starts 2024-03-20T09:19:52.771 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:52 smithi122 ceph-mon[73495]: pgmap v1535: 47 pgs: 5 unknown, 3 peering, 1 active+recovering+undersized+remapped, 3 active+clean+scrubbing, 35 active+clean; 7.4 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 20453/363051 objects misplaced (5.634%); 496 KiB/s, 7 objects/s recovering 2024-03-20T09:19:52.771 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:52 smithi122 ceph-mon[73495]: 1.7 scrub ok 2024-03-20T09:19:52.771 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:52 smithi122 ceph-mon[73495]: osdmap e638: 12 total, 12 up, 11 in 2024-03-20T09:19:52.772 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:52.794 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:19:52.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:52.796 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:19:52.897 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:19:52.923 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:52.946 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:19:52.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:52.948 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:19:53.049 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:19:53.081 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:53.282 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:19:53.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:53.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:19:53.316 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-20T09:19:53.387 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:19:53.414 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:53.434 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:19:53.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:53.437 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:19:53.538 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:19:53.564 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:53.585 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:19:53.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:53.588 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:19:53.633 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Total time run: 92.2991 2024-03-20T09:19:53.633 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Total writes made: 129984 2024-03-20T09:19:53.633 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Write size: 65536 2024-03-20T09:19:53.633 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Object size: 65536 2024-03-20T09:19:53.633 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Bandwidth (MB/sec): 88.0182 2024-03-20T09:19:53.633 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Stddev Bandwidth: 43.5209 2024-03-20T09:19:53.633 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Max bandwidth (MB/sec): 234.125 2024-03-20T09:19:53.633 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Min bandwidth (MB/sec): 0 2024-03-20T09:19:53.633 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Average IOPS: 1408 2024-03-20T09:19:53.633 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Stddev IOPS: 696.334 2024-03-20T09:19:53.633 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Max IOPS: 3746 2024-03-20T09:19:53.634 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Min IOPS: 0 2024-03-20T09:19:53.634 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Average Latency(s): 0.0113517 2024-03-20T09:19:53.634 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Stddev Latency(s): 0.0359118 2024-03-20T09:19:53.634 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Max latency(s): 2.49928 2024-03-20T09:19:53.634 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Min latency(s): 0.0010551 2024-03-20T09:19:53.639 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Cleaning up (deleting benchmark objects) 2024-03-20T09:19:53.689 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:19:53.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:53 smithi122 ceph-mon[73495]: 5.e scrub ok 2024-03-20T09:19:53.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:53 smithi122 ceph-mon[73495]: 1.13 scrub starts 2024-03-20T09:19:53.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:53 smithi122 ceph-mon[73495]: 5.4 scrub starts 2024-03-20T09:19:53.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:53 smithi122 ceph-mon[73495]: 1.13 scrub ok 2024-03-20T09:19:53.722 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:53.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:53 smithi026 ceph-mon[68309]: 5.e scrub ok 2024-03-20T09:19:53.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:53 smithi026 ceph-mon[68309]: 1.13 scrub starts 2024-03-20T09:19:53.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:53 smithi026 ceph-mon[68309]: 5.4 scrub starts 2024-03-20T09:19:53.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:53 smithi026 ceph-mon[68309]: 1.13 scrub ok 2024-03-20T09:19:53.747 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:19:53.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:53.750 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:19:53.850 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:19:53.908 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:53 smithi115 ceph-mon[72476]: 5.e scrub ok 2024-03-20T09:19:53.909 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:53 smithi115 ceph-mon[72476]: 1.13 scrub starts 2024-03-20T09:19:53.909 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:53 smithi115 ceph-mon[72476]: 5.4 scrub starts 2024-03-20T09:19:53.909 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:53 smithi115 ceph-mon[72476]: 1.13 scrub ok 2024-03-20T09:19:53.956 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:53.985 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:19:53.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:53.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:19:54.094 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:19:54.135 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:54.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:54 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:19:54.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:54 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:19:54.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:54 smithi026 ceph-mon[68309]: 1.1b scrub starts 2024-03-20T09:19:54.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:54 smithi026 ceph-mon[68309]: 5.4 scrub ok 2024-03-20T09:19:54.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:54 smithi026 ceph-mon[68309]: 1.1b scrub ok 2024-03-20T09:19:54.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:54 smithi026 ceph-mon[68309]: osdmap e639: 12 total, 12 up, 11 in 2024-03-20T09:19:54.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:54 smithi026 ceph-mon[68309]: pgmap v1538: 47 pgs: 1 activating+undersized+remapped, 2 active+undersized+remapped, 1 active+recovering+undersized+remapped, 3 remapped+peering, 3 unknown, 3 peering, 3 active+clean+scrubbing, 31 active+clean; 7.4 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 20184/365478 objects misplaced (5.523%); 1.4 MiB/s, 22 objects/s recovering 2024-03-20T09:19:54.712 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:19:54.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:54.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:19:54.827 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:19:54.863 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:54 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:19:54.863 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:54 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:19:54.863 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:54 smithi122 ceph-mon[73495]: 1.1b scrub starts 2024-03-20T09:19:54.863 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:54 smithi122 ceph-mon[73495]: 5.4 scrub ok 2024-03-20T09:19:54.863 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:54 smithi122 ceph-mon[73495]: 1.1b scrub ok 2024-03-20T09:19:54.863 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:54 smithi122 ceph-mon[73495]: osdmap e639: 12 total, 12 up, 11 in 2024-03-20T09:19:54.863 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:54 smithi122 ceph-mon[73495]: pgmap v1538: 47 pgs: 1 activating+undersized+remapped, 2 active+undersized+remapped, 1 active+recovering+undersized+remapped, 3 remapped+peering, 3 unknown, 3 peering, 3 active+clean+scrubbing, 31 active+clean; 7.4 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 20184/365478 objects misplaced (5.523%); 1.4 MiB/s, 22 objects/s recovering 2024-03-20T09:19:54.865 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:54.889 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:19:54.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:54.894 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:19:54.914 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:54 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:19:54.914 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:54 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:19:54.914 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:54 smithi115 ceph-mon[72476]: 1.1b scrub starts 2024-03-20T09:19:54.914 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:54 smithi115 ceph-mon[72476]: 5.4 scrub ok 2024-03-20T09:19:54.914 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:54 smithi115 ceph-mon[72476]: 1.1b scrub ok 2024-03-20T09:19:54.914 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:54 smithi115 ceph-mon[72476]: osdmap e639: 12 total, 12 up, 11 in 2024-03-20T09:19:54.914 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:54 smithi115 ceph-mon[72476]: pgmap v1538: 47 pgs: 1 activating+undersized+remapped, 2 active+undersized+remapped, 1 active+recovering+undersized+remapped, 3 remapped+peering, 3 unknown, 3 peering, 3 active+clean+scrubbing, 31 active+clean; 7.4 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 20184/365478 objects misplaced (5.523%); 1.4 MiB/s, 22 objects/s recovering 2024-03-20T09:19:54.994 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:19:55.022 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:55.047 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:19:55.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:55.050 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:19:55.151 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:19:55.205 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:55.893 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:19:55.910 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:55 smithi115 ceph-mon[72476]: osdmap e640: 12 total, 12 up, 11 in 2024-03-20T09:19:55.921 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:19:55.949 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:55 smithi122 ceph-mon[73495]: osdmap e640: 12 total, 12 up, 11 in 2024-03-20T09:19:55.964 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:19:55.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:55 smithi026 ceph-mon[68309]: osdmap e640: 12 total, 12 up, 11 in 2024-03-20T09:19:56.012 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:19:56.130 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:19:56.130 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:19:56.130 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:19:56.131 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:19:56.161 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:19:56.195 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:19:56.229 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:19:56.261 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:19:56.263 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:19:56.263 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:19:56.264 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:19:56.264 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:19:56.264 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:19:56.264 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:19:56.264 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:19:56.264 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:19:56.264 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:19:56.264 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:19:56.264 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:19:56.264 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:19:56.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:56 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2474021474' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:19:56.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:56 smithi026 ceph-mon[68309]: pgmap v1540: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovering+undersized+remapped, 3 remapped+peering, 2 peering, 1 active+clean+scrubbing, 38 active+clean; 7.9 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 219 B/s rd, 122 MiB/s wr, 2.05k op/s; 3/388536 objects degraded (0.001%); 28136/388536 objects misplaced (7.242%); 2.3 MiB/s, 36 objects/s recovering 2024-03-20T09:19:56.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:56 smithi026 ceph-mon[68309]: 1.f scrub starts 2024-03-20T09:19:56.908 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:56 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2474021474' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:19:56.908 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:56 smithi115 ceph-mon[72476]: pgmap v1540: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovering+undersized+remapped, 3 remapped+peering, 2 peering, 1 active+clean+scrubbing, 38 active+clean; 7.9 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 219 B/s rd, 122 MiB/s wr, 2.05k op/s; 3/388536 objects degraded (0.001%); 28136/388536 objects misplaced (7.242%); 2.3 MiB/s, 36 objects/s recovering 2024-03-20T09:19:56.908 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:56 smithi115 ceph-mon[72476]: 1.f scrub starts 2024-03-20T09:19:56.949 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:56 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2474021474' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:19:56.949 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:56 smithi122 ceph-mon[73495]: pgmap v1540: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovering+undersized+remapped, 3 remapped+peering, 2 peering, 1 active+clean+scrubbing, 38 active+clean; 7.9 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 219 B/s rd, 122 MiB/s wr, 2.05k op/s; 3/388536 objects degraded (0.001%); 28136/388536 objects misplaced (7.242%); 2.3 MiB/s, 36 objects/s recovering 2024-03-20T09:19:56.949 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:56 smithi122 ceph-mon[73495]: 1.f scrub starts 2024-03-20T09:19:56.964 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 8, 10, 11, 3, 4] out_osds: [9] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 5] 2024-03-20T09:19:56.964 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:19:56.964 INFO:tasks.thrashosds.thrasher:Adding osd 9 2024-03-20T09:19:56.964 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd in 9 2024-03-20T09:19:57.910 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:57 smithi115 ceph-mon[72476]: 1.f scrub ok 2024-03-20T09:19:57.910 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:57 smithi115 ceph-mon[72476]: Health check failed: Degraded data redundancy: 3/388536 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-03-20T09:19:57.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:57 smithi122 ceph-mon[73495]: 1.f scrub ok 2024-03-20T09:19:57.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:57 smithi122 ceph-mon[73495]: Health check failed: Degraded data redundancy: 3/388536 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-03-20T09:19:57.973 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:57 smithi026 ceph-mon[68309]: 1.f scrub ok 2024-03-20T09:19:57.974 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:57 smithi026 ceph-mon[68309]: Health check failed: Degraded data redundancy: 3/388536 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-03-20T09:19:58.909 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:19:58 smithi115 ceph-mon[72476]: pgmap v1541: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 42 active+clean; 7.8 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 170 B/s rd, 93 MiB/s wr, 1.77k op/s; 8249/384990 objects degraded (2.143%); 43866/384990 objects misplaced (11.394%); 4.9 MiB/s, 79 objects/s recovering 2024-03-20T09:19:58.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:19:58 smithi122 ceph-mon[73495]: pgmap v1541: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 42 active+clean; 7.8 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 170 B/s rd, 93 MiB/s wr, 1.77k op/s; 8249/384990 objects degraded (2.143%); 43866/384990 objects misplaced (11.394%); 4.9 MiB/s, 79 objects/s recovering 2024-03-20T09:19:58.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:19:58 smithi026 ceph-mon[68309]: pgmap v1541: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 42 active+clean; 7.8 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 170 B/s rd, 93 MiB/s wr, 1.77k op/s; 8249/384990 objects degraded (2.143%); 43866/384990 objects misplaced (11.394%); 4.9 MiB/s, 79 objects/s recovering 2024-03-20T09:19:59.408 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:19:59.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:59.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:19:59.513 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:19:59.599 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:59.644 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:19:59.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:59.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:19:59.747 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:19:59.787 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:19:59.820 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:19:59.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:19:59.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:19:59.927 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:19:59.977 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:00.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:00 smithi115 ceph-mon[72476]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Degraded data redundancy: 8249/384990 objects degraded (2.143%), 3 pgs degraded 2024-03-20T09:20:00.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:00 smithi115 ceph-mon[72476]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-20T09:20:00.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:00 smithi115 ceph-mon[72476]: [WRN] PG_DEGRADED: Degraded data redundancy: 8249/384990 objects degraded (2.143%), 3 pgs degraded 2024-03-20T09:20:00.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:00 smithi115 ceph-mon[72476]: pg 5.1 is active+recovering+undersized+degraded+remapped, acting [3,7] 2024-03-20T09:20:00.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:00 smithi115 ceph-mon[72476]: pg 5.5 is active+recovery_wait+undersized+degraded+remapped, acting [0,1] 2024-03-20T09:20:00.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:00 smithi115 ceph-mon[72476]: pg 5.f is active+recovering+undersized+degraded+remapped, acting [5,11] 2024-03-20T09:20:00.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:00 smithi122 ceph-mon[73495]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Degraded data redundancy: 8249/384990 objects degraded (2.143%), 3 pgs degraded 2024-03-20T09:20:00.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:00 smithi122 ceph-mon[73495]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-20T09:20:00.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:00 smithi122 ceph-mon[73495]: [WRN] PG_DEGRADED: Degraded data redundancy: 8249/384990 objects degraded (2.143%), 3 pgs degraded 2024-03-20T09:20:00.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:00 smithi122 ceph-mon[73495]: pg 5.1 is active+recovering+undersized+degraded+remapped, acting [3,7] 2024-03-20T09:20:00.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:00 smithi122 ceph-mon[73495]: pg 5.5 is active+recovery_wait+undersized+degraded+remapped, acting [0,1] 2024-03-20T09:20:00.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:00 smithi122 ceph-mon[73495]: pg 5.f is active+recovering+undersized+degraded+remapped, acting [5,11] 2024-03-20T09:20:00.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:00 smithi026 ceph-mon[68309]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; Degraded data redundancy: 8249/384990 objects degraded (2.143%), 3 pgs degraded 2024-03-20T09:20:00.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:00 smithi026 ceph-mon[68309]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-20T09:20:00.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:00 smithi026 ceph-mon[68309]: [WRN] PG_DEGRADED: Degraded data redundancy: 8249/384990 objects degraded (2.143%), 3 pgs degraded 2024-03-20T09:20:00.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:00 smithi026 ceph-mon[68309]: pg 5.1 is active+recovering+undersized+degraded+remapped, acting [3,7] 2024-03-20T09:20:00.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:00 smithi026 ceph-mon[68309]: pg 5.5 is active+recovery_wait+undersized+degraded+remapped, acting [0,1] 2024-03-20T09:20:00.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:00 smithi026 ceph-mon[68309]: pg 5.f is active+recovering+undersized+degraded+remapped, acting [5,11] 2024-03-20T09:20:01.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:01 smithi115 ceph-mon[72476]: pgmap v1542: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 42 active+clean; 7.8 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 133 B/s rd, 66 MiB/s wr, 1.33k op/s; 8249/383778 objects degraded (2.149%); 43866/383778 objects misplaced (11.430%); 3.1 MiB/s, 50 objects/s recovering 2024-03-20T09:20:01.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:01 smithi122 ceph-mon[73495]: pgmap v1542: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 42 active+clean; 7.8 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 133 B/s rd, 66 MiB/s wr, 1.33k op/s; 8249/383778 objects degraded (2.149%); 43866/383778 objects misplaced (11.430%); 3.1 MiB/s, 50 objects/s recovering 2024-03-20T09:20:01.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:01 smithi026 ceph-mon[68309]: pgmap v1542: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 42 active+clean; 7.8 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 133 B/s rd, 66 MiB/s wr, 1.33k op/s; 8249/383778 objects degraded (2.149%); 43866/383778 objects misplaced (11.430%); 3.1 MiB/s, 50 objects/s recovering 2024-03-20T09:20:02.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:02 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8359/379005 objects degraded (2.206%), 4 pgs degraded (PG_DEGRADED) 2024-03-20T09:20:02.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:02 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8359/379005 objects degraded (2.206%), 4 pgs degraded (PG_DEGRADED) 2024-03-20T09:20:02.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:02 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8359/379005 objects degraded (2.206%), 4 pgs degraded (PG_DEGRADED) 2024-03-20T09:20:03.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:03 smithi115 ceph-mon[72476]: pgmap v1543: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 42 active+clean; 7.7 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 119 B/s rd, 59 MiB/s wr, 1.37k op/s; 8359/379005 objects degraded (2.206%); 43491/379005 objects misplaced (11.475%); 4.7 MiB/s, 75 objects/s recovering 2024-03-20T09:20:03.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:03 smithi122 ceph-mon[73495]: pgmap v1543: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 42 active+clean; 7.7 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 119 B/s rd, 59 MiB/s wr, 1.37k op/s; 8359/379005 objects degraded (2.206%); 43491/379005 objects misplaced (11.475%); 4.7 MiB/s, 75 objects/s recovering 2024-03-20T09:20:03.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:03 smithi026 ceph-mon[68309]: pgmap v1543: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 42 active+clean; 7.7 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 119 B/s rd, 59 MiB/s wr, 1.37k op/s; 8359/379005 objects degraded (2.206%); 43491/379005 objects misplaced (11.475%); 4.7 MiB/s, 75 objects/s recovering 2024-03-20T09:20:04.177 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:20:04.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:04.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:20:04.281 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:20:04.325 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:04.367 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:20:04.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:04.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:20:04.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:04 smithi115 ceph-mon[72476]: pgmap v1544: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 7.6 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 102 B/s rd, 51 MiB/s wr, 1.29k op/s; 8431/375576 objects degraded (2.245%); 43071/375576 objects misplaced (11.468%); 5.2 MiB/s, 83 objects/s recovering 2024-03-20T09:20:04.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:04 smithi122 ceph-mon[73495]: pgmap v1544: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 7.6 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 102 B/s rd, 51 MiB/s wr, 1.29k op/s; 8431/375576 objects degraded (2.245%); 43071/375576 objects misplaced (11.468%); 5.2 MiB/s, 83 objects/s recovering 2024-03-20T09:20:04.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:04 smithi026 ceph-mon[68309]: pgmap v1544: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 7.6 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 102 B/s rd, 51 MiB/s wr, 1.29k op/s; 8431/375576 objects degraded (2.245%); 43071/375576 objects misplaced (11.468%); 5.2 MiB/s, 83 objects/s recovering 2024-03-20T09:20:04.473 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:20:04.501 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:04.526 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:20:04.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:04.529 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:20:04.630 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:20:04.684 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:05.349 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:20:05.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:05.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:20:05.375 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:20:05.378 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:20:05.474 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:20:05.508 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:05.533 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:20:05.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:05.538 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:20:05.640 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:20:05.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:05 smithi115 ceph-mon[72476]: 1.a scrub starts 2024-03-20T09:20:05.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:05 smithi115 ceph-mon[72476]: 1.a scrub ok 2024-03-20T09:20:05.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:05 smithi122 ceph-mon[73495]: 1.a scrub starts 2024-03-20T09:20:05.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:05 smithi122 ceph-mon[73495]: 1.a scrub ok 2024-03-20T09:20:05.714 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:05.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:05 smithi026 ceph-mon[68309]: 1.a scrub starts 2024-03-20T09:20:05.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:05 smithi026 ceph-mon[68309]: 1.a scrub ok 2024-03-20T09:20:05.754 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:20:05.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:05.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:20:05.858 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:20:05.887 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:05.913 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:20:05.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:05.917 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:20:06.018 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:20:06.089 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:06.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:06.121 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:20:06.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:20:06.222 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:20:06.289 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:06.325 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:20:06.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:06.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:20:06.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:06 smithi115 ceph-mon[72476]: pgmap v1545: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 7.6 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 88 B/s rd, 44 MiB/s wr, 1.23k op/s; 8503/371544 objects degraded (2.289%); 42882/371544 objects misplaced (11.542%); 5.1 MiB/s, 82 objects/s recovering 2024-03-20T09:20:06.430 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:20:06.479 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:06.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:06 smithi122 ceph-mon[73495]: pgmap v1545: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 7.6 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 88 B/s rd, 44 MiB/s wr, 1.23k op/s; 8503/371544 objects degraded (2.289%); 42882/371544 objects misplaced (11.542%); 5.1 MiB/s, 82 objects/s recovering 2024-03-20T09:20:06.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:06 smithi026 ceph-mon[68309]: pgmap v1545: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 7.6 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 88 B/s rd, 44 MiB/s wr, 1.23k op/s; 8503/371544 objects degraded (2.289%); 42882/371544 objects misplaced (11.542%); 5.1 MiB/s, 82 objects/s recovering 2024-03-20T09:20:07.238 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:20:07.346 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:20:07.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:07.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:20:07.455 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:20:07.516 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:07.546 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:20:07.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:07.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:20:07.608 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:07 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8503/371544 objects degraded (2.289%), 4 pgs degraded (PG_DEGRADED) 2024-03-20T09:20:07.656 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:20:07.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:07 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8503/371544 objects degraded (2.289%), 4 pgs degraded (PG_DEGRADED) 2024-03-20T09:20:07.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:07 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8503/371544 objects degraded (2.289%), 4 pgs degraded (PG_DEGRADED) 2024-03-20T09:20:07.686 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:07.709 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:20:07.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:07.713 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:20:07.814 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:20:07.881 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:08.256 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:20:08.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:08.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:20:08.361 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:20:08.395 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:08.417 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:20:08.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:08.423 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:20:08.524 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:20:08.552 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:08.573 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:20:08.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:08.577 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:20:08.660 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:08 smithi115 ceph-mon[72476]: pgmap v1546: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 7.5 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 53 KiB/s wr, 618 op/s; 8647/366315 objects degraded (2.361%); 42264/366315 objects misplaced (11.538%); 6.2 MiB/s, 99 objects/s recovering 2024-03-20T09:20:08.664 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:08 smithi026 ceph-mon[68309]: pgmap v1546: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 7.5 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 53 KiB/s wr, 618 op/s; 8647/366315 objects degraded (2.361%); 42264/366315 objects misplaced (11.538%); 6.2 MiB/s, 99 objects/s recovering 2024-03-20T09:20:08.678 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:20:08.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:08 smithi122 ceph-mon[73495]: pgmap v1546: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 7.5 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 53 KiB/s wr, 618 op/s; 8647/366315 objects degraded (2.361%); 42264/366315 objects misplaced (11.538%); 6.2 MiB/s, 99 objects/s recovering 2024-03-20T09:20:08.726 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:09.369 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:20:09.660 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:09 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3095814504' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:20:09.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:09 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3095814504' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:20:09.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:09 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3095814504' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:20:10.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:10 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:20:10.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:10 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3095814504' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:20:10.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:10 smithi115 ceph-mon[72476]: osdmap e641: 12 total, 12 up, 11 in 2024-03-20T09:20:10.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:10 smithi115 ceph-mon[72476]: pgmap v1548: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 7.4 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 618 op/s; 8647/365220 objects degraded (2.368%); 42264/365220 objects misplaced (11.572%); 5.5 MiB/s, 89 objects/s recovering 2024-03-20T09:20:10.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:10 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:20:10.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:10 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3095814504' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:20:10.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:10 smithi122 ceph-mon[73495]: osdmap e641: 12 total, 12 up, 11 in 2024-03-20T09:20:10.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:10 smithi122 ceph-mon[73495]: pgmap v1548: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 7.4 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 618 op/s; 8647/365220 objects degraded (2.368%); 42264/365220 objects misplaced (11.572%); 5.5 MiB/s, 89 objects/s recovering 2024-03-20T09:20:10.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:10 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:20:10.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:10 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3095814504' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:20:10.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:10 smithi026 ceph-mon[68309]: osdmap e641: 12 total, 12 up, 11 in 2024-03-20T09:20:10.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:10 smithi026 ceph-mon[68309]: pgmap v1548: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 7.4 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 618 op/s; 8647/365220 objects degraded (2.368%); 42264/365220 objects misplaced (11.572%); 5.5 MiB/s, 89 objects/s recovering 2024-03-20T09:20:11.007 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:20:11.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:11.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:20:11.112 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:20:11.154 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:11.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:11 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:20:11.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:11 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:20:11.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:11 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:20:11.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:11 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:20:11.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:11 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:20:11.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:11 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:20:12.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:12 smithi115 ceph-mon[72476]: osdmap e642: 12 total, 12 up, 11 in 2024-03-20T09:20:12.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:12 smithi115 ceph-mon[72476]: pgmap v1550: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 7.4 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 578 op/s; 8689/361680 objects degraded (2.402%); 41908/361680 objects misplaced (11.587%); 5.9 MiB/s, 94 objects/s recovering 2024-03-20T09:20:12.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:12 smithi115 ceph-mon[72476]: 1.1 deep-scrub starts 2024-03-20T09:20:12.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:12 smithi115 ceph-mon[72476]: 1.1 deep-scrub ok 2024-03-20T09:20:12.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:12 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8647/365220 objects degraded (2.368%), 4 pgs degraded (PG_DEGRADED) 2024-03-20T09:20:12.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:12 smithi122 ceph-mon[73495]: osdmap e642: 12 total, 12 up, 11 in 2024-03-20T09:20:12.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:12 smithi122 ceph-mon[73495]: pgmap v1550: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 7.4 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 578 op/s; 8689/361680 objects degraded (2.402%); 41908/361680 objects misplaced (11.587%); 5.9 MiB/s, 94 objects/s recovering 2024-03-20T09:20:12.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:12 smithi122 ceph-mon[73495]: 1.1 deep-scrub starts 2024-03-20T09:20:12.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:12 smithi122 ceph-mon[73495]: 1.1 deep-scrub ok 2024-03-20T09:20:12.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:12 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8647/365220 objects degraded (2.368%), 4 pgs degraded (PG_DEGRADED) 2024-03-20T09:20:12.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:12 smithi026 ceph-mon[68309]: osdmap e642: 12 total, 12 up, 11 in 2024-03-20T09:20:12.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:12 smithi026 ceph-mon[68309]: pgmap v1550: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 7.4 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 578 op/s; 8689/361680 objects degraded (2.402%); 41908/361680 objects misplaced (11.587%); 5.9 MiB/s, 94 objects/s recovering 2024-03-20T09:20:12.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:12 smithi026 ceph-mon[68309]: 1.1 deep-scrub starts 2024-03-20T09:20:12.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:12 smithi026 ceph-mon[68309]: 1.1 deep-scrub ok 2024-03-20T09:20:12.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:12 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8647/365220 objects degraded (2.368%), 4 pgs degraded (PG_DEGRADED) 2024-03-20T09:20:12.811 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:20:12.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:12.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:20:12.916 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:20:12.976 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:14.908 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:14 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:20:14.909 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:14 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:20:14.909 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:14 smithi115 ceph-mon[72476]: osdmap e643: 12 total, 12 up, 11 in 2024-03-20T09:20:14.909 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:14 smithi115 ceph-mon[72476]: pgmap v1552: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 7.3 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 392 op/s; 8700/359247 objects degraded (2.422%); 41377/359247 objects misplaced (11.518%); 6.1 MiB/s, 97 objects/s recovering 2024-03-20T09:20:14.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:14 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:20:14.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:14 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:20:14.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:14 smithi122 ceph-mon[73495]: osdmap e643: 12 total, 12 up, 11 in 2024-03-20T09:20:14.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:14 smithi122 ceph-mon[73495]: pgmap v1552: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 7.3 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 392 op/s; 8700/359247 objects degraded (2.422%); 41377/359247 objects misplaced (11.518%); 6.1 MiB/s, 97 objects/s recovering 2024-03-20T09:20:14.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:14 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:20:14.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:14 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:20:14.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:14 smithi026 ceph-mon[68309]: osdmap e643: 12 total, 12 up, 11 in 2024-03-20T09:20:14.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:14 smithi026 ceph-mon[68309]: pgmap v1552: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 7.3 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 392 op/s; 8700/359247 objects degraded (2.422%); 41377/359247 objects misplaced (11.518%); 6.1 MiB/s, 97 objects/s recovering 2024-03-20T09:20:15.975 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:20:15.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:15.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:20:16.082 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:20:16.112 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:16.131 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:20:16.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:16.135 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:20:16.236 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:20:16.274 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:17.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:17 smithi115 ceph-mon[72476]: pgmap v1553: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 7.3 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 487 op/s; 8700/356511 objects degraded (2.440%); 41223/356511 objects misplaced (11.563%); 6.9 MiB/s, 110 objects/s recovering 2024-03-20T09:20:17.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:17 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:20:17.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:17 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:20:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:17 smithi122 ceph-mon[73495]: pgmap v1553: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 7.3 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 487 op/s; 8700/356511 objects degraded (2.440%); 41223/356511 objects misplaced (11.563%); 6.9 MiB/s, 110 objects/s recovering 2024-03-20T09:20:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:17 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:20:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:17 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:20:17.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:17 smithi026 ceph-mon[68309]: pgmap v1553: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 7.3 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 487 op/s; 8700/356511 objects degraded (2.440%); 41223/356511 objects misplaced (11.563%); 6.9 MiB/s, 110 objects/s recovering 2024-03-20T09:20:17.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:17 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:20:17.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:17 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:20:18.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:18 smithi115 ceph-mon[72476]: osdmap e644: 12 total, 12 up, 11 in 2024-03-20T09:20:18.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:18 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8700/356511 objects degraded (2.440%), 4 pgs degraded (PG_DEGRADED) 2024-03-20T09:20:18.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:18 smithi122 ceph-mon[73495]: osdmap e644: 12 total, 12 up, 11 in 2024-03-20T09:20:18.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:18 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8700/356511 objects degraded (2.440%), 4 pgs degraded (PG_DEGRADED) 2024-03-20T09:20:18.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:18 smithi026 ceph-mon[68309]: osdmap e644: 12 total, 12 up, 11 in 2024-03-20T09:20:18.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:18 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8700/356511 objects degraded (2.440%), 4 pgs degraded (PG_DEGRADED) 2024-03-20T09:20:18.883 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:20:18.883 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-03-20T09:20:18.883 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-03-20T09:20:18.883 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:20:18.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.883 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.24910.0:143582 3.19 3:9af20452:::benchmark_data_smithi122_119671_object43800:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e449)", 2024-03-20T09:20:18.884 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:10:41.276815+0000", 2024-03-20T09:20:18.884 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 577.60276204100001, 2024-03-20T09:20:18.884 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.88465880600000002, 2024-03-20T09:20:18.884 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:20:18.884 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:20:18.884 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:20:18.884 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.24910", 2024-03-20T09:20:18.884 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/375549932", 2024-03-20T09:20:18.884 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 143582 2024-03-20T09:20:18.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.884 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:20:18.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.884 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:20:18.885 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.276815+0000", 2024-03-20T09:20:18.885 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:18.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.885 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:20:18.885 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.276815+0000", 2024-03-20T09:20:18.885 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:18.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.885 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:20:18.885 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.276817+0000", 2024-03-20T09:20:18.885 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6750000000000001e-06 2024-03-20T09:20:18.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:20:18.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.276822+0000", 2024-03-20T09:20:18.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5650000000000001e-06 2024-03-20T09:20:18.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:20:18.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.276836+0000", 2024-03-20T09:20:18.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3703000000000001e-05 2024-03-20T09:20:18.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:20:18.887 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.276844+0000", 2024-03-20T09:20:18.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0630000000000006e-06 2024-03-20T09:20:18.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.887 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:20:18.887 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.276921+0000", 2024-03-20T09:20:18.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7275999999999997e-05 2024-03-20T09:20:18.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.887 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:20:18.888 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:41.276935+0000", 2024-03-20T09:20:18.888 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3916e-05 2024-03-20T09:20:18.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.888 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:20:18.888 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.147527+0000", 2024-03-20T09:20:18.888 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.87059159600000002 2024-03-20T09:20:18.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.888 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:20:18.888 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.147658+0000", 2024-03-20T09:20:18.888 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013134999999999999 2024-03-20T09:20:18.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.889 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,10", 2024-03-20T09:20:18.889 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.147783+0000", 2024-03-20T09:20:18.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012454600000000001 2024-03-20T09:20:18.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.889 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:20:18.889 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.152392+0000", 2024-03-20T09:20:18.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0046092980000000004 2024-03-20T09:20:18.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.889 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:18.889 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.152572+0000", 2024-03-20T09:20:18.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00017944799999999999 2024-03-20T09:20:18.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.890 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:18.890 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.161292+0000", 2024-03-20T09:20:18.890 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0087206660000000002 2024-03-20T09:20:18.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.890 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:20:18.890 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.161367+0000", 2024-03-20T09:20:18.891 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4252999999999995e-05 2024-03-20T09:20:18.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.891 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:20:18.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:10:42.161474+0000", 2024-03-20T09:20:18.891 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107451 2024-03-20T09:20:18.892 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:18.892 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:20:18.892 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:18.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.892 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25690.0:28590 4.f 4:febd55ef:::benchmark_data_smithi122_145698_object28589:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e533)", 2024-03-20T09:20:18.892 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:13:44.542655+0000", 2024-03-20T09:20:18.892 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 394.33692230899999, 2024-03-20T09:20:18.892 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.28540777299999998, 2024-03-20T09:20:18.892 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:20:18.892 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:20:18.892 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:20:18.892 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25690", 2024-03-20T09:20:18.893 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:20:18.893 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 28590 2024-03-20T09:20:18.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.893 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:20:18.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.893 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:20:18.893 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.542655+0000", 2024-03-20T09:20:18.893 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:18.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.893 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:20:18.893 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.542655+0000", 2024-03-20T09:20:18.893 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:18.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.894 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:20:18.894 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.542656+0000", 2024-03-20T09:20:18.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1799999999999999e-06 2024-03-20T09:20:18.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.894 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:20:18.894 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.542747+0000", 2024-03-20T09:20:18.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0464999999999995e-05 2024-03-20T09:20:18.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:20:18.895 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.542758+0000", 2024-03-20T09:20:18.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1738e-05 2024-03-20T09:20:18.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:20:18.895 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.542766+0000", 2024-03-20T09:20:18.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.841e-06 2024-03-20T09:20:18.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:20:18.895 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.543580+0000", 2024-03-20T09:20:18.896 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00081360700000000005 2024-03-20T09:20:18.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.896 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:20:18.896 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.543665+0000", 2024-03-20T09:20:18.896 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5067000000000004e-05 2024-03-20T09:20:18.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.897 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2,3,11", 2024-03-20T09:20:18.897 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.543758+0000", 2024-03-20T09:20:18.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3574999999999998e-05 2024-03-20T09:20:18.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.897 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:18.897 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.546513+0000", 2024-03-20T09:20:18.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027545590000000002 2024-03-20T09:20:18.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:20:18.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.546530+0000", 2024-03-20T09:20:18.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.668e-05 2024-03-20T09:20:18.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:18.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.553277+0000", 2024-03-20T09:20:18.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0067469619999999996 2024-03-20T09:20:18.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:18.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.827775+0000", 2024-03-20T09:20:18.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.27449864899999998 2024-03-20T09:20:18.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:20:18.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.827884+0000", 2024-03-20T09:20:18.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010904900000000001 2024-03-20T09:20:18.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:20:18.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.828063+0000", 2024-03-20T09:20:18.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00017840099999999999 2024-03-20T09:20:18.900 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:18.900 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:20:18.901 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:18.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.901 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25690.0:28602 4.f 4:f275a134:::benchmark_data_smithi122_145698_object28601:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e533)", 2024-03-20T09:20:18.901 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:13:44.548479+0000", 2024-03-20T09:20:18.901 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 394.33109838899998, 2024-03-20T09:20:18.901 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.28009240099999999, 2024-03-20T09:20:18.901 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:20:18.901 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:20:18.901 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:20:18.902 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25690", 2024-03-20T09:20:18.902 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:20:18.902 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 28602 2024-03-20T09:20:18.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.902 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:20:18.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.902 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:20:18.902 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.548479+0000", 2024-03-20T09:20:18.902 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:18.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.903 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:20:18.903 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.548479+0000", 2024-03-20T09:20:18.903 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:18.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.903 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:20:18.903 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.548480+0000", 2024-03-20T09:20:18.903 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.097e-06 2024-03-20T09:20:18.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.903 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:20:18.903 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.548522+0000", 2024-03-20T09:20:18.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2083999999999999e-05 2024-03-20T09:20:18.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:20:18.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.548532+0000", 2024-03-20T09:20:18.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.6399999999999992e-06 2024-03-20T09:20:18.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:20:18.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.548539+0000", 2024-03-20T09:20:18.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1980000000000004e-06 2024-03-20T09:20:18.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.905 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:20:18.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.548594+0000", 2024-03-20T09:20:18.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5081999999999999e-05 2024-03-20T09:20:18.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.905 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:20:18.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.548759+0000", 2024-03-20T09:20:18.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00016553 2024-03-20T09:20:18.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2,3,11", 2024-03-20T09:20:18.906 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.548956+0000", 2024-03-20T09:20:18.906 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000196894 2024-03-20T09:20:18.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:18.906 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.553082+0000", 2024-03-20T09:20:18.906 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0041252110000000002 2024-03-20T09:20:18.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:20:18.907 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.553098+0000", 2024-03-20T09:20:18.907 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6521e-05 2024-03-20T09:20:18.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.907 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:18.907 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.553315+0000", 2024-03-20T09:20:18.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00021694499999999999 2024-03-20T09:20:18.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.908 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:18.908 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.828380+0000", 2024-03-20T09:20:18.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.275064486 2024-03-20T09:20:18.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.908 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:20:18.908 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.828446+0000", 2024-03-20T09:20:18.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6200999999999999e-05 2024-03-20T09:20:18.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.909 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:20:18.909 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.828571+0000", 2024-03-20T09:20:18.909 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012551199999999999 2024-03-20T09:20:18.909 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:18.909 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:20:18.909 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:18.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.909 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25690.0:28618 4.4 4:22aad26f:::benchmark_data_smithi122_145698_object28617:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e533)", 2024-03-20T09:20:18.909 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:13:44.553231+0000", 2024-03-20T09:20:18.909 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 394.32634594500001, 2024-03-20T09:20:18.910 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.27470114400000001, 2024-03-20T09:20:18.910 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:20:18.910 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:20:18.910 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:20:18.910 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25690", 2024-03-20T09:20:18.910 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:20:18.910 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 28618 2024-03-20T09:20:18.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.910 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:20:18.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:20:18.910 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.553231+0000", 2024-03-20T09:20:18.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:18.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:20:18.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.553231+0000", 2024-03-20T09:20:18.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:18.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.911 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:20:18.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.553232+0000", 2024-03-20T09:20:18.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.51e-07 2024-03-20T09:20:18.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.912 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:20:18.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.553263+0000", 2024-03-20T09:20:18.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1077000000000001e-05 2024-03-20T09:20:18.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.912 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:20:18.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.553268+0000", 2024-03-20T09:20:18.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2650000000000004e-06 2024-03-20T09:20:18.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.913 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:20:18.913 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.553272+0000", 2024-03-20T09:20:18.913 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5549999999999998e-06 2024-03-20T09:20:18.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.913 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:20:18.913 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.553359+0000", 2024-03-20T09:20:18.913 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7311999999999993e-05 2024-03-20T09:20:18.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.913 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:20:18.914 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.553482+0000", 2024-03-20T09:20:18.914 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012334800000000001 2024-03-20T09:20:18.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.914 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2,3", 2024-03-20T09:20:18.914 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.553621+0000", 2024-03-20T09:20:18.914 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013908099999999999 2024-03-20T09:20:18.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.915 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:20:18.915 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.555504+0000", 2024-03-20T09:20:18.915 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001882597 2024-03-20T09:20:18.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.915 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:18.915 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.557975+0000", 2024-03-20T09:20:18.915 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024711849999999999 2024-03-20T09:20:18.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.916 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:18.916 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.757400+0000", 2024-03-20T09:20:18.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.19942442099999999 2024-03-20T09:20:18.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.916 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:20:18.916 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.827879+0000", 2024-03-20T09:20:18.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.070479374999999997 2024-03-20T09:20:18.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:20:18.917 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.827932+0000", 2024-03-20T09:20:18.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3477000000000003e-05 2024-03-20T09:20:18.917 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:18.917 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:20:18.918 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:18.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.918 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25690.0:28619 4.4 4:2cfa0406:::benchmark_data_smithi122_145698_object28618:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e533)", 2024-03-20T09:20:18.918 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:13:44.553282+0000", 2024-03-20T09:20:18.918 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 394.32629515799999, 2024-03-20T09:20:18.918 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.27638705899999999, 2024-03-20T09:20:18.918 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:20:18.918 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:20:18.919 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:20:18.919 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25690", 2024-03-20T09:20:18.919 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:20:18.919 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 28619 2024-03-20T09:20:18.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.919 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:20:18.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.919 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:20:18.919 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.553282+0000", 2024-03-20T09:20:18.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:18.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:20:18.920 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.553282+0000", 2024-03-20T09:20:18.920 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:18.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:20:18.920 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.553282+0000", 2024-03-20T09:20:18.921 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.15e-07 2024-03-20T09:20:18.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.921 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:20:18.921 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.553309+0000", 2024-03-20T09:20:18.921 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6400000000000001e-05 2024-03-20T09:20:18.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.921 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:20:18.922 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.553315+0000", 2024-03-20T09:20:18.922 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8549999999999998e-06 2024-03-20T09:20:18.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.922 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:20:18.922 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.553318+0000", 2024-03-20T09:20:18.922 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6449999999999998e-06 2024-03-20T09:20:18.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.922 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:20:18.922 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.554569+0000", 2024-03-20T09:20:18.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001250814 2024-03-20T09:20:18.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:20:18.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.554651+0000", 2024-03-20T09:20:18.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2084999999999998e-05 2024-03-20T09:20:18.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2,3", 2024-03-20T09:20:18.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.554732+0000", 2024-03-20T09:20:18.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0970000000000006e-05 2024-03-20T09:20:18.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.924 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:20:18.924 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.555508+0000", 2024-03-20T09:20:18.924 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00077546899999999998 2024-03-20T09:20:18.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.924 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:18.924 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.559185+0000", 2024-03-20T09:20:18.924 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003677105 2024-03-20T09:20:18.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.925 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:18.925 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.829577+0000", 2024-03-20T09:20:18.925 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.27039222299999999 2024-03-20T09:20:18.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.925 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:20:18.925 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.829632+0000", 2024-03-20T09:20:18.925 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5050000000000003e-05 2024-03-20T09:20:18.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.926 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:20:18.926 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.829669+0000", 2024-03-20T09:20:18.926 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7128000000000001e-05 2024-03-20T09:20:18.926 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:18.926 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:20:18.926 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:18.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.926 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25690.0:28620 4.f 4:fe995c48:::benchmark_data_smithi122_145698_object28619:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e533)", 2024-03-20T09:20:18.927 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:13:44.553325+0000", 2024-03-20T09:20:18.927 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 394.32625236899997, 2024-03-20T09:20:18.927 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.27548288199999998, 2024-03-20T09:20:18.927 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:20:18.927 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:20:18.927 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:20:18.927 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25690", 2024-03-20T09:20:18.927 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:20:18.927 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 28620 2024-03-20T09:20:18.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.927 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:20:18.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.928 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:20:18.928 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.553325+0000", 2024-03-20T09:20:18.928 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:18.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.928 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:20:18.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.553325+0000", 2024-03-20T09:20:18.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:18.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.929 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:20:18.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.553325+0000", 2024-03-20T09:20:18.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.96e-07 2024-03-20T09:20:18.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.929 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:20:18.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.553364+0000", 2024-03-20T09:20:18.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9329999999999998e-05 2024-03-20T09:20:18.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:20:18.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.553369+0000", 2024-03-20T09:20:18.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6110000000000004e-06 2024-03-20T09:20:18.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:20:18.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.553373+0000", 2024-03-20T09:20:18.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6890000000000002e-06 2024-03-20T09:20:18.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:20:18.931 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.555121+0000", 2024-03-20T09:20:18.931 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0017484009999999999 2024-03-20T09:20:18.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:20:18.931 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.555268+0000", 2024-03-20T09:20:18.931 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014718000000000001 2024-03-20T09:20:18.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2,3,11", 2024-03-20T09:20:18.932 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.555440+0000", 2024-03-20T09:20:18.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00017122599999999999 2024-03-20T09:20:18.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.932 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:18.932 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.558853+0000", 2024-03-20T09:20:18.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034131019999999999 2024-03-20T09:20:18.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.932 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:18.933 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.559756+0000", 2024-03-20T09:20:18.933 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00090339900000000004 2024-03-20T09:20:18.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.933 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:20:18.933 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.559769+0000", 2024-03-20T09:20:18.933 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2649999999999999e-05 2024-03-20T09:20:18.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.934 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:18.934 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.828627+0000", 2024-03-20T09:20:18.934 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.26885790599999998 2024-03-20T09:20:18.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.934 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:20:18.934 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.828677+0000", 2024-03-20T09:20:18.934 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0627999999999997e-05 2024-03-20T09:20:18.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.935 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:20:18.935 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:13:44.828808+0000", 2024-03-20T09:20:18.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000130464 2024-03-20T09:20:18.935 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:18.935 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:20:18.936 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:18.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.936 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25690.0:121130 4.4 4:2cfeb77e:::benchmark_data_smithi122_145698_object211:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e553)", 2024-03-20T09:20:18.936 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:15:03.248084+0000", 2024-03-20T09:20:18.936 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 315.63149300800001, 2024-03-20T09:20:18.936 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.27476531100000001, 2024-03-20T09:20:18.936 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:20:18.936 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:20:18.936 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:20:18.936 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25690", 2024-03-20T09:20:18.937 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:20:18.937 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 121130 2024-03-20T09:20:18.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.937 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:20:18.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.937 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:20:18.937 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:03.248084+0000", 2024-03-20T09:20:18.937 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:18.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.937 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:20:18.937 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:03.248084+0000", 2024-03-20T09:20:18.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:18.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.938 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:20:18.938 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:03.248085+0000", 2024-03-20T09:20:18.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.72e-07 2024-03-20T09:20:18.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.939 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:20:18.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:03.248087+0000", 2024-03-20T09:20:18.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1519999999999999e-06 2024-03-20T09:20:18.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.939 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:20:18.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:03.248093+0000", 2024-03-20T09:20:18.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7599999999999999e-06 2024-03-20T09:20:18.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.940 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:20:18.940 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:03.248098+0000", 2024-03-20T09:20:18.940 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0289999999999996e-06 2024-03-20T09:20:18.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.940 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:20:18.940 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:03.345904+0000", 2024-03-20T09:20:18.940 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.097806131000000004 2024-03-20T09:20:18.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.940 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:20:18.940 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:03.346025+0000", 2024-03-20T09:20:18.940 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000121459 2024-03-20T09:20:18.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.941 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2,3", 2024-03-20T09:20:18.941 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:03.346135+0000", 2024-03-20T09:20:18.941 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000109545 2024-03-20T09:20:18.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.941 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:20:18.941 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:03.362639+0000", 2024-03-20T09:20:18.941 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016504168999999999 2024-03-20T09:20:18.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.942 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:18.942 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:03.374110+0000", 2024-03-20T09:20:18.942 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011470797 2024-03-20T09:20:18.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.942 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:18.942 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:03.522727+0000", 2024-03-20T09:20:18.942 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.14861739299999999 2024-03-20T09:20:18.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.942 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:20:18.943 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:03.522784+0000", 2024-03-20T09:20:18.943 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7290999999999997e-05 2024-03-20T09:20:18.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.943 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:20:18.943 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:03.522850+0000", 2024-03-20T09:20:18.943 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5213000000000001e-05 2024-03-20T09:20:18.943 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:18.943 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:20:18.943 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:18.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.944 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25690.0:123858 4.d 4:b1c30147:::benchmark_data_smithi122_145698_object2939:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e556)", 2024-03-20T09:20:18.944 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:15:10.860406+0000", 2024-03-20T09:20:18.944 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 308.01917146, 2024-03-20T09:20:18.944 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.318607582, 2024-03-20T09:20:18.944 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:20:18.944 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:20:18.944 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:20:18.944 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25690", 2024-03-20T09:20:18.944 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:20:18.944 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 123858 2024-03-20T09:20:18.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.945 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:20:18.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.945 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:20:18.945 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:10.860406+0000", 2024-03-20T09:20:18.945 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:18.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.946 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.946 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:20:18.946 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:10.860406+0000", 2024-03-20T09:20:18.946 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:18.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.947 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:20:18.947 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:10.860406+0000", 2024-03-20T09:20:18.947 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2599999999999998e-07 2024-03-20T09:20:18.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.948 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:20:18.948 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:10.860409+0000", 2024-03-20T09:20:18.948 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5840000000000002e-06 2024-03-20T09:20:18.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.949 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:20:18.949 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:10.860416+0000", 2024-03-20T09:20:18.949 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0790000000000003e-06 2024-03-20T09:20:18.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.950 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:20:18.950 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:10.860421+0000", 2024-03-20T09:20:18.950 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1919999999999996e-06 2024-03-20T09:20:18.950 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.951 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:20:18.951 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:10.862435+0000", 2024-03-20T09:20:18.951 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002013632 2024-03-20T09:20:18.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.951 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:20:18.952 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:10.862562+0000", 2024-03-20T09:20:18.952 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000127439 2024-03-20T09:20:18.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.952 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:20:18.952 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:10.862684+0000", 2024-03-20T09:20:18.952 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012149 2024-03-20T09:20:18.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.953 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:20:18.953 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:11.021121+0000", 2024-03-20T09:20:18.953 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.15843700399999999 2024-03-20T09:20:18.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.954 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.954 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:18.954 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:11.178808+0000", 2024-03-20T09:20:18.954 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.157687723 2024-03-20T09:20:18.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.955 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:18.955 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:11.178879+0000", 2024-03-20T09:20:18.955 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0723999999999996e-05 2024-03-20T09:20:18.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.956 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:20:18.956 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:11.178940+0000", 2024-03-20T09:20:18.956 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1397999999999997e-05 2024-03-20T09:20:18.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.956 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:20:18.957 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:11.179013+0000", 2024-03-20T09:20:18.957 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2891000000000004e-05 2024-03-20T09:20:18.957 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:18.957 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:20:18.957 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:18.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.957 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25690.0:123861 4.d 4:bb42626e:::benchmark_data_smithi122_145698_object2942:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e556)", 2024-03-20T09:20:18.957 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:15:10.872764+0000", 2024-03-20T09:20:18.957 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 308.00681323700002, 2024-03-20T09:20:18.957 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.31091403099999998, 2024-03-20T09:20:18.958 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:20:18.958 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:20:18.958 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:20:18.958 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25690", 2024-03-20T09:20:18.958 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:20:18.958 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 123861 2024-03-20T09:20:18.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.958 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:20:18.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.958 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:20:18.958 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:10.872764+0000", 2024-03-20T09:20:18.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:18.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:20:18.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:10.872764+0000", 2024-03-20T09:20:18.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:18.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.959 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:20:18.959 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:10.872766+0000", 2024-03-20T09:20:18.959 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9360000000000002e-06 2024-03-20T09:20:18.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:20:18.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:10.872774+0000", 2024-03-20T09:20:18.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6340000000000001e-06 2024-03-20T09:20:18.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.960 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:20:18.960 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:10.872787+0000", 2024-03-20T09:20:18.960 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3763e-05 2024-03-20T09:20:18.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:20:18.961 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:10.872803+0000", 2024-03-20T09:20:18.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5990000000000001e-05 2024-03-20T09:20:18.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.961 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:20:18.961 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:11.015706+0000", 2024-03-20T09:20:18.961 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.14290265899999999 2024-03-20T09:20:18.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.962 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:20:18.962 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:11.015836+0000", 2024-03-20T09:20:18.962 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013034599999999999 2024-03-20T09:20:18.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.962 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:20:18.962 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:11.015950+0000", 2024-03-20T09:20:18.962 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000113891 2024-03-20T09:20:18.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.963 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:20:18.963 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:11.178832+0000", 2024-03-20T09:20:18.963 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.162882095 2024-03-20T09:20:18.963 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.963 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:18.963 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:11.181589+0000", 2024-03-20T09:20:18.963 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002756723 2024-03-20T09:20:18.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.964 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:18.964 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:11.183579+0000", 2024-03-20T09:20:18.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0019901570000000002 2024-03-20T09:20:18.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.964 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.964 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:20:18.964 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:11.183628+0000", 2024-03-20T09:20:18.964 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8337000000000002e-05 2024-03-20T09:20:18.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.965 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:20:18.965 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:11.183678+0000", 2024-03-20T09:20:18.965 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0500000000000001e-05 2024-03-20T09:20:18.965 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:18.965 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:20:18.965 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:18.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.965 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.966 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop_reply(client.25690.0:123858 4.d e556/520 ondisk, result = 0)", 2024-03-20T09:20:18.966 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:15:10.877080+0000", 2024-03-20T09:20:18.966 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 308.00249681600002, 2024-03-20T09:20:18.966 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.30175287299999998, 2024-03-20T09:20:18.966 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:20:18.966 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:20:18.966 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:20:18.966 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.966 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:20:18.966 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:10.877080+0000", 2024-03-20T09:20:18.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:18.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:20:18.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:10.877080+0000", 2024-03-20T09:20:18.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:18.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.967 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.967 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:20:18.967 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:10.877081+0000", 2024-03-20T09:20:18.967 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6e-07 2024-03-20T09:20:18.967 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:20:18.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:10.877083+0000", 2024-03-20T09:20:18.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4590000000000001e-06 2024-03-20T09:20:18.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.968 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.968 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:20:18.968 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:10.877089+0000", 2024-03-20T09:20:18.968 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9050000000000002e-06 2024-03-20T09:20:18.968 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:20:18.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:10.877094+0000", 2024-03-20T09:20:18.969 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5770000000000004e-06 2024-03-20T09:20:18.969 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.969 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.969 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:20:18.969 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:11.178783+0000", 2024-03-20T09:20:18.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.30168958200000001 2024-03-20T09:20:18.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:20:18.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:11.178792+0000", 2024-03-20T09:20:18.970 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3820000000000004e-06 2024-03-20T09:20:18.970 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.970 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.970 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:20:18.970 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:11.178833+0000", 2024-03-20T09:20:18.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1708000000000001e-05 2024-03-20T09:20:18.971 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:18.971 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:20:18.971 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:18.971 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.971 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.971 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop_reply(client.25690.0:123858 4.d e556/520 ondisk, result = 0)", 2024-03-20T09:20:18.971 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:15:10.879963+0000", 2024-03-20T09:20:18.971 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 307.99961386500001, 2024-03-20T09:20:18.971 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.29905547199999999, 2024-03-20T09:20:18.971 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:20:18.972 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:20:18.972 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:20:18.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:20:18.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:10.879963+0000", 2024-03-20T09:20:18.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:18.972 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.972 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.972 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:20:18.972 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:10.879963+0000", 2024-03-20T09:20:18.972 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:18.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:20:18.973 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:10.879965+0000", 2024-03-20T09:20:18.973 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-03-20T09:20:18.973 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.973 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.973 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:20:18.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:10.879970+0000", 2024-03-20T09:20:18.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4580000000000001e-06 2024-03-20T09:20:18.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:20:18.974 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:10.879982+0000", 2024-03-20T09:20:18.974 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.203e-05 2024-03-20T09:20:18.974 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.974 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.974 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:20:18.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:10.879993+0000", 2024-03-20T09:20:18.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0481e-05 2024-03-20T09:20:18.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:20:18.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:11.178865+0000", 2024-03-20T09:20:18.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.29887211200000002 2024-03-20T09:20:18.975 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.975 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.975 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:20:18.975 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:11.178873+0000", 2024-03-20T09:20:18.975 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0620000000000001e-06 2024-03-20T09:20:18.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.976 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:20:18.976 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:11.179019+0000", 2024-03-20T09:20:18.976 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014608899999999999 2024-03-20T09:20:18.976 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:18.976 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:20:18.976 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:18.976 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.976 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.976 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25690.0:125301 4.4 4:2c7f1584:::benchmark_data_smithi122_145698_object4382:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e556)", 2024-03-20T09:20:18.976 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:15:15.464834+0000", 2024-03-20T09:20:18.977 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 303.41474282199999, 2024-03-20T09:20:18.977 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.30628249400000002, 2024-03-20T09:20:18.977 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:20:18.977 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:20:18.977 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:20:18.977 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25690", 2024-03-20T09:20:18.977 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:20:18.977 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 125301 2024-03-20T09:20:18.977 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.977 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:20:18.977 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:20:18.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:15.464834+0000", 2024-03-20T09:20:18.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:18.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:20:18.978 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:15.464834+0000", 2024-03-20T09:20:18.978 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:18.978 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.978 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.978 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:20:18.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:15.464836+0000", 2024-03-20T09:20:18.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.88e-06 2024-03-20T09:20:18.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:20:18.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:15.464843+0000", 2024-03-20T09:20:18.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5479999999999998e-06 2024-03-20T09:20:18.979 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.979 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.979 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:20:18.979 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:15.464862+0000", 2024-03-20T09:20:18.979 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9279e-05 2024-03-20T09:20:18.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:20:18.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:15.464872+0000", 2024-03-20T09:20:18.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3889999999999994e-06 2024-03-20T09:20:18.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.980 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:20:18.980 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:15.632250+0000", 2024-03-20T09:20:18.980 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.16737804100000001 2024-03-20T09:20:18.980 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.980 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:20:18.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:15.632377+0000", 2024-03-20T09:20:18.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012715900000000001 2024-03-20T09:20:18.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2,3", 2024-03-20T09:20:18.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:15.632504+0000", 2024-03-20T09:20:18.981 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012750000000000001 2024-03-20T09:20:18.981 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.981 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.981 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:20:18.981 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:15.656020+0000", 2024-03-20T09:20:18.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023515906999999999 2024-03-20T09:20:18.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:18.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:15.675791+0000", 2024-03-20T09:20:18.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.019771060999999999 2024-03-20T09:20:18.982 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.982 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.982 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:18.982 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:15.771020+0000", 2024-03-20T09:20:18.982 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.095228526999999993 2024-03-20T09:20:18.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:20:18.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:15.771063+0000", 2024-03-20T09:20:18.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3362999999999998e-05 2024-03-20T09:20:18.983 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.983 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.983 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:20:18.983 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:15.771117+0000", 2024-03-20T09:20:18.983 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.384e-05 2024-03-20T09:20:18.984 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:18.984 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:20:18.984 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:18.984 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.984 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.984 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25690.0:126544 4.d 4:b5d7aed4:::benchmark_data_smithi122_145698_object5625:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e558)", 2024-03-20T09:20:18.984 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:15:19.467823+0000", 2024-03-20T09:20:18.984 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 299.41175414700001, 2024-03-20T09:20:18.984 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.24939164899999999, 2024-03-20T09:20:18.984 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:20:18.984 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:20:18.984 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:20:18.985 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25690", 2024-03-20T09:20:18.985 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:20:18.985 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 126544 2024-03-20T09:20:18.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.985 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:20:18.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:20:18.985 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.467823+0000", 2024-03-20T09:20:18.985 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:18.985 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.985 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.985 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:20:18.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.467823+0000", 2024-03-20T09:20:18.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:18.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:20:18.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.467824+0000", 2024-03-20T09:20:18.986 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3400000000000001e-06 2024-03-20T09:20:18.986 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.986 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.986 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:20:18.986 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.467830+0000", 2024-03-20T09:20:18.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9379999999999997e-06 2024-03-20T09:20:18.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:20:18.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.467842+0000", 2024-03-20T09:20:18.987 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1800000000000001e-05 2024-03-20T09:20:18.987 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.987 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.987 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:20:18.987 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.467850+0000", 2024-03-20T09:20:18.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6199999999999999e-06 2024-03-20T09:20:18.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:20:18.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.468043+0000", 2024-03-20T09:20:18.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019302999999999999 2024-03-20T09:20:18.988 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.988 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.988 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:20:18.988 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.468175+0000", 2024-03-20T09:20:18.988 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013174700000000001 2024-03-20T09:20:18.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:20:18.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.468294+0000", 2024-03-20T09:20:18.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119055 2024-03-20T09:20:18.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.989 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:18.989 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.688332+0000", 2024-03-20T09:20:18.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.220038386 2024-03-20T09:20:18.989 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:18.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.688413+0000", 2024-03-20T09:20:18.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0517000000000002e-05 2024-03-20T09:20:18.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:20:18.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.717078+0000", 2024-03-20T09:20:18.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.028665963999999999 2024-03-20T09:20:18.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:20:18.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.717129+0000", 2024-03-20T09:20:18.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0963e-05 2024-03-20T09:20:18.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:20:18.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.717215+0000", 2024-03-20T09:20:18.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5289000000000003e-05 2024-03-20T09:20:18.992 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:18.992 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:20:18.992 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:18.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.993 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25690.0:126551 4.d 4:b26cf088:::benchmark_data_smithi122_145698_object5632:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e558)", 2024-03-20T09:20:18.993 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:15:19.473142+0000", 2024-03-20T09:20:18.993 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 299.40643557499999, 2024-03-20T09:20:18.993 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.24417783100000001, 2024-03-20T09:20:18.993 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:20:18.993 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:20:18.993 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:20:18.993 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25690", 2024-03-20T09:20:18.993 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:20:18.993 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 126551 2024-03-20T09:20:18.993 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.994 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:20:18.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:20:18.994 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.473142+0000", 2024-03-20T09:20:18.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:18.994 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:20:18.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.473142+0000", 2024-03-20T09:20:18.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:18.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:20:18.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.473143+0000", 2024-03-20T09:20:18.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.621e-06 2024-03-20T09:20:18.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:20:18.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.473150+0000", 2024-03-20T09:20:18.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2609999999999999e-06 2024-03-20T09:20:18.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:20:18.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.473163+0000", 2024-03-20T09:20:18.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3071000000000001e-05 2024-03-20T09:20:18.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:20:18.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.473172+0000", 2024-03-20T09:20:18.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1579999999999994e-06 2024-03-20T09:20:18.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:20:18.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.473250+0000", 2024-03-20T09:20:18.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8422000000000002e-05 2024-03-20T09:20:18.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:20:18.997 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.473386+0000", 2024-03-20T09:20:18.997 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000136095 2024-03-20T09:20:18.997 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:20:18.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.473530+0000", 2024-03-20T09:20:18.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014371800000000001 2024-03-20T09:20:18.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:18.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.688376+0000", 2024-03-20T09:20:18.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.21484555799999999 2024-03-20T09:20:18.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:18.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.688489+0000", 2024-03-20T09:20:18.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000113319 2024-03-20T09:20:18.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:18.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:18.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:20:18.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.717224+0000", 2024-03-20T09:20:18.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.028735139999999999 2024-03-20T09:20:19.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:20:19.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.717267+0000", 2024-03-20T09:20:19.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2846999999999998e-05 2024-03-20T09:20:19.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:20:19.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.717319+0000", 2024-03-20T09:20:19.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2621000000000003e-05 2024-03-20T09:20:19.000 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:19.000 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:20:19.001 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:19.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.001 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.001 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25690.0:126654 4.4 4:223a64ab:::benchmark_data_smithi122_145698_object5735:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e558)", 2024-03-20T09:20:19.001 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:15:19.908168+0000", 2024-03-20T09:20:19.001 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 298.971409064, 2024-03-20T09:20:19.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.24208463599999999, 2024-03-20T09:20:19.001 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:20:19.001 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:20:19.002 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:20:19.002 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25690", 2024-03-20T09:20:19.002 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:20:19.002 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 126654 2024-03-20T09:20:19.002 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.002 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:20:19.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.002 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:20:19.002 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.908168+0000", 2024-03-20T09:20:19.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:19.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:20:19.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.908168+0000", 2024-03-20T09:20:19.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:19.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.003 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.003 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:20:19.003 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.908169+0000", 2024-03-20T09:20:19.003 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8100000000000003e-07 2024-03-20T09:20:19.003 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:20:19.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.908172+0000", 2024-03-20T09:20:19.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8899999999999999e-06 2024-03-20T09:20:19.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:20:19.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.908180+0000", 2024-03-20T09:20:19.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7639999999999994e-06 2024-03-20T09:20:19.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:20:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.908187+0000", 2024-03-20T09:20:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1129999999999997e-06 2024-03-20T09:20:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:20:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:20.030969+0000", 2024-03-20T09:20:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.122782668 2024-03-20T09:20:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:20:19.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:20.031094+0000", 2024-03-20T09:20:19.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012471300000000001 2024-03-20T09:20:19.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2,3", 2024-03-20T09:20:19.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:20.031230+0000", 2024-03-20T09:20:19.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000136093 2024-03-20T09:20:19.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:20:19.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:20.056493+0000", 2024-03-20T09:20:19.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.025263418999999999 2024-03-20T09:20:19.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:19.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:20.124310+0000", 2024-03-20T09:20:19.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.067816321999999998 2024-03-20T09:20:19.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.007 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:19.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:20.150180+0000", 2024-03-20T09:20:19.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.025870305 2024-03-20T09:20:19.007 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:20:19.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:20.150220+0000", 2024-03-20T09:20:19.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0082e-05 2024-03-20T09:20:19.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:20:19.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:20.150253+0000", 2024-03-20T09:20:19.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2685999999999998e-05 2024-03-20T09:20:19.008 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:19.008 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:20:19.008 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:19.009 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.009 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25690.0:193276 4.d 4:be441e64:::benchmark_data_smithi122_145698_object72357:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e577)", 2024-03-20T09:20:19.009 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:16:47.259402+0000", 2024-03-20T09:20:19.009 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 211.62017525900001, 2024-03-20T09:20:19.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.22729255800000001, 2024-03-20T09:20:19.009 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:20:19.009 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:20:19.009 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:20:19.009 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25690", 2024-03-20T09:20:19.009 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:20:19.009 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 193276 2024-03-20T09:20:19.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.010 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:20:19.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:20:19.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.259402+0000", 2024-03-20T09:20:19.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:19.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.010 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.010 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:20:19.010 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.259402+0000", 2024-03-20T09:20:19.010 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:19.010 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:20:19.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.259403+0000", 2024-03-20T09:20:19.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.850000000000001e-07 2024-03-20T09:20:19.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:20:19.011 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.259406+0000", 2024-03-20T09:20:19.011 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4910000000000001e-06 2024-03-20T09:20:19.011 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.011 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.011 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:20:19.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.259415+0000", 2024-03-20T09:20:19.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1969999999999996e-06 2024-03-20T09:20:19.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.012 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.012 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:20:19.012 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.259419+0000", 2024-03-20T09:20:19.012 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4599999999999996e-06 2024-03-20T09:20:19.012 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:20:19.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.259532+0000", 2024-03-20T09:20:19.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011336999999999999 2024-03-20T09:20:19.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:20:19.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.259621+0000", 2024-03-20T09:20:19.013 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8494999999999999e-05 2024-03-20T09:20:19.013 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.013 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.013 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:20:19.013 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.259690+0000", 2024-03-20T09:20:19.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9060000000000006e-05 2024-03-20T09:20:19.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:20:19.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.263563+0000", 2024-03-20T09:20:19.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0038733019999999999 2024-03-20T09:20:19.014 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.014 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.014 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:19.014 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.271366+0000", 2024-03-20T09:20:19.014 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0078022710000000004 2024-03-20T09:20:19.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:19.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.486575+0000", 2024-03-20T09:20:19.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.21520989600000001 2024-03-20T09:20:19.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.015 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.015 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:20:19.015 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.486642+0000", 2024-03-20T09:20:19.015 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6699000000000006e-05 2024-03-20T09:20:19.015 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.016 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:20:19.016 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.486695+0000", 2024-03-20T09:20:19.016 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2332e-05 2024-03-20T09:20:19.016 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:19.016 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:20:19.016 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:19.016 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.016 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.016 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25690.0:193278 4.7 4:e3c89a70:::benchmark_data_smithi122_145698_object72359:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e577)", 2024-03-20T09:20:19.016 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:16:47.259432+0000", 2024-03-20T09:20:19.016 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 211.62014544799999, 2024-03-20T09:20:19.017 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.22728831699999999, 2024-03-20T09:20:19.017 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:20:19.017 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:20:19.017 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:20:19.017 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25690", 2024-03-20T09:20:19.017 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:20:19.017 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 193278 2024-03-20T09:20:19.017 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.017 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:20:19.017 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.017 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:20:19.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.259432+0000", 2024-03-20T09:20:19.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:19.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:20:19.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.259432+0000", 2024-03-20T09:20:19.018 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:19.018 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.018 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.018 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:20:19.018 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.259432+0000", 2024-03-20T09:20:19.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9999999999999999e-07 2024-03-20T09:20:19.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:20:19.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.259434+0000", 2024-03-20T09:20:19.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.668e-06 2024-03-20T09:20:19.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.019 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.019 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:20:19.019 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.259438+0000", 2024-03-20T09:20:19.019 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8210000000000003e-06 2024-03-20T09:20:19.019 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:20:19.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.259441+0000", 2024-03-20T09:20:19.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.303e-06 2024-03-20T09:20:19.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.020 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:20:19.020 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.259484+0000", 2024-03-20T09:20:19.020 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3477999999999999e-05 2024-03-20T09:20:19.020 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.020 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:20:19.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.259611+0000", 2024-03-20T09:20:19.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012649100000000001 2024-03-20T09:20:19.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:20:19.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.260317+0000", 2024-03-20T09:20:19.021 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00070576000000000005 2024-03-20T09:20:19.021 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.021 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.021 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:19.021 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.270908+0000", 2024-03-20T09:20:19.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010591402999999999 2024-03-20T09:20:19.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.022 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:20:19.022 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.270920+0000", 2024-03-20T09:20:19.022 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2152e-05 2024-03-20T09:20:19.022 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.022 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:19.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.486618+0000", 2024-03-20T09:20:19.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.215697474 2024-03-20T09:20:19.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.023 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:20:19.023 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.486673+0000", 2024-03-20T09:20:19.023 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5232000000000002e-05 2024-03-20T09:20:19.023 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.023 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.024 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:20:19.024 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.486720+0000", 2024-03-20T09:20:19.024 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7234999999999998e-05 2024-03-20T09:20:19.024 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:19.024 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:20:19.024 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:19.024 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.024 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.024 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25690.0:193285 4.7 4:e21ffae8:::benchmark_data_smithi122_145698_object72366:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e577)", 2024-03-20T09:20:19.024 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:16:47.271303+0000", 2024-03-20T09:20:19.024 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 211.608273943, 2024-03-20T09:20:19.025 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.22892721099999999, 2024-03-20T09:20:19.025 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:20:19.025 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:20:19.025 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:20:19.025 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25690", 2024-03-20T09:20:19.025 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:20:19.025 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 193285 2024-03-20T09:20:19.025 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.025 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:20:19.025 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.025 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:20:19.025 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.271303+0000", 2024-03-20T09:20:19.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:19.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:20:19.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.271303+0000", 2024-03-20T09:20:19.026 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:19.026 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.026 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.026 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:20:19.026 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.271304+0000", 2024-03-20T09:20:19.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.01e-07 2024-03-20T09:20:19.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:20:19.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.271306+0000", 2024-03-20T09:20:19.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0930000000000001e-06 2024-03-20T09:20:19.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.027 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.027 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:20:19.027 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.271311+0000", 2024-03-20T09:20:19.027 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5260000000000001e-06 2024-03-20T09:20:19.027 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:20:19.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.271318+0000", 2024-03-20T09:20:19.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2999999999999998e-06 2024-03-20T09:20:19.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.028 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:20:19.028 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.277243+0000", 2024-03-20T09:20:19.028 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059252139999999998 2024-03-20T09:20:19.028 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.028 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:20:19.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.277371+0000", 2024-03-20T09:20:19.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012820900000000001 2024-03-20T09:20:19.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:20:19.029 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.277489+0000", 2024-03-20T09:20:19.029 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000118274 2024-03-20T09:20:19.029 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.029 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.029 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:20:19.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.282092+0000", 2024-03-20T09:20:19.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0046029240000000004 2024-03-20T09:20:19.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:19.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.288997+0000", 2024-03-20T09:20:19.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0069050020000000004 2024-03-20T09:20:19.030 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.030 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.030 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:19.030 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.500116+0000", 2024-03-20T09:20:19.030 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.21111924300000001 2024-03-20T09:20:19.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:20:19.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.500174+0000", 2024-03-20T09:20:19.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7049000000000002e-05 2024-03-20T09:20:19.031 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.031 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.031 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:20:19.031 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.500230+0000", 2024-03-20T09:20:19.031 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6975999999999998e-05 2024-03-20T09:20:19.031 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:19.031 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:20:19.032 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:19.032 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.032 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.032 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:24818 5.c 5:33dfc16c:::benchmark_data_smithi122_157894_object24817:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e614)", 2024-03-20T09:20:19.032 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:18:29.504995+0000", 2024-03-20T09:20:19.032 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 109.374581932, 2024-03-20T09:20:19.032 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.200133061, 2024-03-20T09:20:19.032 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:20:19.032 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:20:19.032 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:20:19.032 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:20:19.033 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:20:19.033 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 24818 2024-03-20T09:20:19.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.033 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:20:19.033 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.033 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:20:19.033 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.504995+0000", 2024-03-20T09:20:19.033 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:19.033 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:20:19.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.504995+0000", 2024-03-20T09:20:19.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:19.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.034 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.034 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:20:19.034 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.504997+0000", 2024-03-20T09:20:19.034 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3859999999999999e-06 2024-03-20T09:20:19.034 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:20:19.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.505060+0000", 2024-03-20T09:20:19.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3336999999999993e-05 2024-03-20T09:20:19.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.035 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.035 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:20:19.035 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.505076+0000", 2024-03-20T09:20:19.035 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5803000000000001e-05 2024-03-20T09:20:19.035 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:20:19.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.505083+0000", 2024-03-20T09:20:19.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4000000000000003e-06 2024-03-20T09:20:19.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:20:19.036 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.505140+0000", 2024-03-20T09:20:19.036 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6399e-05 2024-03-20T09:20:19.036 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.036 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.036 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:20:19.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.505306+0000", 2024-03-20T09:20:19.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00016615600000000001 2024-03-20T09:20:19.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,6", 2024-03-20T09:20:19.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.505500+0000", 2024-03-20T09:20:19.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000194549 2024-03-20T09:20:19.037 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.037 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.037 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:19.037 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.506968+0000", 2024-03-20T09:20:19.037 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014676800000000001 2024-03-20T09:20:19.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:20:19.038 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.507039+0000", 2024-03-20T09:20:19.038 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0671000000000002e-05 2024-03-20T09:20:19.038 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.038 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.038 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:19.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.704989+0000", 2024-03-20T09:20:19.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.19795005700000001 2024-03-20T09:20:19.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.039 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:20:19.039 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.705070+0000", 2024-03-20T09:20:19.039 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1539000000000007e-05 2024-03-20T09:20:19.039 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.039 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.040 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:20:19.040 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.705128+0000", 2024-03-20T09:20:19.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8084000000000002e-05 2024-03-20T09:20:19.040 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:19.040 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:20:19.040 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:19.040 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.040 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.040 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:24822 5.c 5:3d14bba3:::benchmark_data_smithi122_157894_object24821:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e614)", 2024-03-20T09:20:19.040 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:18:29.507337+0000", 2024-03-20T09:20:19.040 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 109.372240418, 2024-03-20T09:20:19.040 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.19799285699999999, 2024-03-20T09:20:19.041 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:20:19.041 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:20:19.041 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:20:19.041 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:20:19.041 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:20:19.041 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 24822 2024-03-20T09:20:19.041 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.041 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:20:19.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.041 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:20:19.041 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.507337+0000", 2024-03-20T09:20:19.041 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:19.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:20:19.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.507337+0000", 2024-03-20T09:20:19.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:20:19.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.042 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.042 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:20:19.042 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.507338+0000", 2024-03-20T09:20:19.042 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2399999999999996e-07 2024-03-20T09:20:19.042 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:20:19.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.507395+0000", 2024-03-20T09:20:19.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6928000000000001e-05 2024-03-20T09:20:19.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.043 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.043 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:20:19.043 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.507405+0000", 2024-03-20T09:20:19.043 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0346999999999999e-05 2024-03-20T09:20:19.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:20:19.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.507411+0000", 2024-03-20T09:20:19.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6110000000000001e-06 2024-03-20T09:20:19.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.044 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.044 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:20:19.044 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.507458+0000", 2024-03-20T09:20:19.044 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7744999999999998e-05 2024-03-20T09:20:19.044 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:20:19.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.507577+0000", 2024-03-20T09:20:19.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000118675 2024-03-20T09:20:19.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.045 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,6", 2024-03-20T09:20:19.045 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.507664+0000", 2024-03-20T09:20:19.045 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7194000000000006e-05 2024-03-20T09:20:19.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.045 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:20:19.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.508424+0000", 2024-03-20T09:20:19.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00075975900000000004 2024-03-20T09:20:19.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:19.046 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.508709+0000", 2024-03-20T09:20:19.046 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000284746 2024-03-20T09:20:19.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.046 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.046 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:20:19.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.705229+0000", 2024-03-20T09:20:19.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.19652067000000001 2024-03-20T09:20:19.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:20:19.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.705284+0000", 2024-03-20T09:20:19.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4960999999999998e-05 2024-03-20T09:20:19.047 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:20:19.047 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:20:19.047 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:20:19.047 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.705330+0000", 2024-03-20T09:20:19.047 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5296999999999997e-05 2024-03-20T09:20:19.048 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:19.048 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:20:19.048 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:19.048 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:20:19.048 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:20:19.048 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:20:19.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:19 smithi026 ceph-mon[68309]: pgmap v1555: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 7.2 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 504 op/s; 8697/352791 objects degraded (2.465%); 40442/352791 objects misplaced (11.463%); 10 MiB/s, 164 objects/s recovering 2024-03-20T09:20:19.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:19 smithi115 ceph-mon[72476]: pgmap v1555: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 7.2 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 504 op/s; 8697/352791 objects degraded (2.465%); 40442/352791 objects misplaced (11.463%); 10 MiB/s, 164 objects/s recovering 2024-03-20T09:20:19.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:19 smithi122 ceph-mon[73495]: pgmap v1555: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 7.2 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 504 op/s; 8697/352791 objects degraded (2.465%); 40442/352791 objects misplaced (11.463%); 10 MiB/s, 164 objects/s recovering 2024-03-20T09:20:20.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:20 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:20:20.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:20 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:20:20.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:20 smithi115 ceph-mon[72476]: pgmap v1556: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 7.2 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 404 op/s; 8697/351960 objects degraded (2.471%); 40442/351960 objects misplaced (11.491%); 7.1 MiB/s, 114 objects/s recovering 2024-03-20T09:20:20.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:20 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:20:20.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:20 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:20:20.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:20 smithi122 ceph-mon[73495]: pgmap v1556: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 7.2 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 404 op/s; 8697/351960 objects degraded (2.471%); 40442/351960 objects misplaced (11.491%); 7.1 MiB/s, 114 objects/s recovering 2024-03-20T09:20:20.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:20 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:20:20.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:20 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:20:20.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:20 smithi026 ceph-mon[68309]: pgmap v1556: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 7.2 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 404 op/s; 8697/351960 objects degraded (2.471%); 40442/351960 objects misplaced (11.491%); 7.1 MiB/s, 114 objects/s recovering 2024-03-20T09:20:20.859 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:20:20.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:20.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:20:20.963 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:20:21.001 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:21.106 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:20:21.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:21.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:20:21.114 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:20:21.152 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:21 smithi026 ceph-mon[68309]: osdmap e645: 12 total, 12 up, 11 in 2024-03-20T09:20:21.211 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:20:21.239 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:21.259 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:20:21.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:21.264 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:20:21.365 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:20:21.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:21 smithi115 ceph-mon[72476]: osdmap e645: 12 total, 12 up, 11 in 2024-03-20T09:20:21.409 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:21.446 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:20:21.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:21 smithi122 ceph-mon[73495]: osdmap e645: 12 total, 12 up, 11 in 2024-03-20T09:20:21.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:21.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:20:21.554 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:20:21.584 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:21.607 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:20:21.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:21.610 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:20:21.711 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:20:21.748 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:22.147 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:22 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mon-a[68286]: 2024-03-20T09:20:22.137+0000 7f683364b700 -1 mon.a@0(leader).osd e645 definitely_dead 0 2024-03-20T09:20:22.182 INFO:teuthology.orchestra.run.smithi026.stderr:marked in osd.9. 2024-03-20T09:20:22.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:22 smithi115 ceph-mon[72476]: pgmap v1558: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 7.1 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 448 op/s; 8697/348471 objects degraded (2.496%); 40119/348471 objects misplaced (11.513%); 7.5 MiB/s, 121 objects/s recovering 2024-03-20T09:20:22.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:22 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/518058675' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["9"]}]: dispatch 2024-03-20T09:20:22.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:22 smithi122 ceph-mon[73495]: pgmap v1558: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 7.1 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 448 op/s; 8697/348471 objects degraded (2.496%); 40119/348471 objects misplaced (11.513%); 7.5 MiB/s, 121 objects/s recovering 2024-03-20T09:20:22.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:22 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/518058675' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["9"]}]: dispatch 2024-03-20T09:20:22.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:22 smithi026 ceph-mon[68309]: pgmap v1558: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 7.1 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 448 op/s; 8697/348471 objects degraded (2.496%); 40119/348471 objects misplaced (11.513%); 7.5 MiB/s, 121 objects/s recovering 2024-03-20T09:20:22.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:22 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/518058675' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["9"]}]: dispatch 2024-03-20T09:20:22.622 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:20:23.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:23 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/518058675' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["9"]}]': finished 2024-03-20T09:20:23.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:23 smithi122 ceph-mon[73495]: osdmap e646: 12 total, 12 up, 12 in 2024-03-20T09:20:23.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:23 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8697/348471 objects degraded (2.496%), 4 pgs degraded (PG_DEGRADED) 2024-03-20T09:20:23.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:23 smithi122 ceph-mon[73495]: osdmap e647: 12 total, 12 up, 12 in 2024-03-20T09:20:23.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:23 smithi122 ceph-mon[73495]: 1.1c scrub starts 2024-03-20T09:20:23.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:23 smithi122 ceph-mon[73495]: 1.1c scrub ok 2024-03-20T09:20:23.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:23 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:20:23.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:23 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/518058675' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["9"]}]': finished 2024-03-20T09:20:23.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:23 smithi026 ceph-mon[68309]: osdmap e646: 12 total, 12 up, 12 in 2024-03-20T09:20:23.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:23 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8697/348471 objects degraded (2.496%), 4 pgs degraded (PG_DEGRADED) 2024-03-20T09:20:23.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:23 smithi026 ceph-mon[68309]: osdmap e647: 12 total, 12 up, 12 in 2024-03-20T09:20:23.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:23 smithi026 ceph-mon[68309]: 1.1c scrub starts 2024-03-20T09:20:23.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:23 smithi026 ceph-mon[68309]: 1.1c scrub ok 2024-03-20T09:20:23.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:23 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:20:23.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:23 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/518058675' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["9"]}]': finished 2024-03-20T09:20:23.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:23 smithi115 ceph-mon[72476]: osdmap e646: 12 total, 12 up, 12 in 2024-03-20T09:20:23.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:23 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8697/348471 objects degraded (2.496%), 4 pgs degraded (PG_DEGRADED) 2024-03-20T09:20:23.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:23 smithi115 ceph-mon[72476]: osdmap e647: 12 total, 12 up, 12 in 2024-03-20T09:20:23.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:23 smithi115 ceph-mon[72476]: 1.1c scrub starts 2024-03-20T09:20:23.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:23 smithi115 ceph-mon[72476]: 1.1c scrub ok 2024-03-20T09:20:23.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:23 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:20:24.154 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-20T09:20:24.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:24 smithi026 ceph-mon[68309]: osdmap e648: 12 total, 12 up, 12 in 2024-03-20T09:20:24.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:24 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:24.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:24 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:24.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:24 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:24.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:24 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:20:24.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:24 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:20:24.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:24 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:20:24.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:24 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:24.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:24 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:24.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:24 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:24.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:24 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:24.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:24 smithi115 ceph-mon[72476]: osdmap e648: 12 total, 12 up, 12 in 2024-03-20T09:20:24.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:24 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:24.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:24 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:24.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:24 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:24.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:24 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:20:24.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:24 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:20:24.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:24 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:20:24.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:24 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:24.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:24 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:24.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:24 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:24.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:24 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:24.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:24 smithi122 ceph-mon[73495]: osdmap e648: 12 total, 12 up, 12 in 2024-03-20T09:20:24.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:24 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:24.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:24 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:24.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:24 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:24.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:24 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:20:24.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:24 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:20:24.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:24 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:20:24.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:24 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:24.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:24 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:24.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:24 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:24.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:24 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:25.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:25 smithi115 ceph-mon[72476]: osdmap e649: 12 total, 12 up, 12 in 2024-03-20T09:20:25.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:25 smithi122 ceph-mon[73495]: osdmap e649: 12 total, 12 up, 12 in 2024-03-20T09:20:25.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:25 smithi026 ceph-mon[68309]: osdmap e649: 12 total, 12 up, 12 in 2024-03-20T09:20:26.430 INFO:tasks.thrashosds.thrasher:Added osd 9 2024-03-20T09:20:26.539 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:20:26.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:26.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:20:26.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:26 smithi115 ceph-mon[72476]: osdmap e650: 12 total, 12 up, 12 in 2024-03-20T09:20:26.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:26 smithi115 ceph-mon[72476]: pgmap v1565: 47 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 remapped+peering, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 40 active+clean; 7.0 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 447 op/s; 8697/343239 objects degraded (2.534%); 39365/343239 objects misplaced (11.469%); 7.1 MiB/s, 112 objects/s recovering 2024-03-20T09:20:26.658 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:20:26.692 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:26.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:26 smithi122 ceph-mon[73495]: osdmap e650: 12 total, 12 up, 12 in 2024-03-20T09:20:26.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:26 smithi122 ceph-mon[73495]: pgmap v1565: 47 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 remapped+peering, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 40 active+clean; 7.0 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 447 op/s; 8697/343239 objects degraded (2.534%); 39365/343239 objects misplaced (11.469%); 7.1 MiB/s, 112 objects/s recovering 2024-03-20T09:20:26.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:26 smithi026 ceph-mon[68309]: osdmap e650: 12 total, 12 up, 12 in 2024-03-20T09:20:26.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:26 smithi026 ceph-mon[68309]: pgmap v1565: 47 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 remapped+peering, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 40 active+clean; 7.0 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 447 op/s; 8697/343239 objects degraded (2.534%); 39365/343239 objects misplaced (11.469%); 7.1 MiB/s, 112 objects/s recovering 2024-03-20T09:20:26.726 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:20:26.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:26.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:20:26.835 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:20:26.863 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:26.885 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:20:26.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:26.888 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:20:26.989 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:20:27.055 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:27.111 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:20:27.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:27.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:20:27.119 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:20:27.221 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:20:27.250 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:27.279 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:20:27.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:27.282 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:20:27.383 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:20:27.437 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:27.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:27 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8697/343239 objects degraded (2.534%), 4 pgs degraded (PG_DEGRADED) 2024-03-20T09:20:27.477 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:20:27.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:27.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:20:27.587 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:20:27.632 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:27.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:27 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8697/343239 objects degraded (2.534%), 4 pgs degraded (PG_DEGRADED) 2024-03-20T09:20:27.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:27 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8697/343239 objects degraded (2.534%), 4 pgs degraded (PG_DEGRADED) 2024-03-20T09:20:28.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:28 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:20:28.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:28 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:20:28.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:28 smithi115 ceph-mon[72476]: osdmap e651: 12 total, 12 up, 12 in 2024-03-20T09:20:28.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:28 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:28.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:28 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:28.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:28 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:20:28.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:28 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:20:28.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:28 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:28.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:28 smithi115 ceph-mon[72476]: pgmap v1567: 47 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 6.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 464 op/s; 8695/339414 objects degraded (2.562%); 38650/339414 objects misplaced (11.387%); 7.9 MiB/s, 126 objects/s recovering 2024-03-20T09:20:28.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:28 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:20:28.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:28 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:20:28.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:28 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:20:28.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:28 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:28.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:28 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:20:28.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:28 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:20:28.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:28 smithi122 ceph-mon[73495]: osdmap e651: 12 total, 12 up, 12 in 2024-03-20T09:20:28.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:28 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:28.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:28 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:28.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:28 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:20:28.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:28 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:20:28.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:28 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:28.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:28 smithi122 ceph-mon[73495]: pgmap v1567: 47 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 6.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 464 op/s; 8695/339414 objects degraded (2.562%); 38650/339414 objects misplaced (11.387%); 7.9 MiB/s, 126 objects/s recovering 2024-03-20T09:20:28.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:28 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:20:28.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:28 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:20:28.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:28 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:20:28.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:28 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:28.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:28 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:20:28.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:28 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:20:28.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:28 smithi026 ceph-mon[68309]: osdmap e651: 12 total, 12 up, 12 in 2024-03-20T09:20:28.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:28 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:28.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:28 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:28.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:28 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:20:28.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:28 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:20:28.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:28 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:28.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:28 smithi026 ceph-mon[68309]: pgmap v1567: 47 pgs: 1 active+recovering+undersized+remapped, 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 6.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 464 op/s; 8695/339414 objects degraded (2.562%); 38650/339414 objects misplaced (11.387%); 7.9 MiB/s, 126 objects/s recovering 2024-03-20T09:20:28.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:28 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:20:28.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:28 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:20:28.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:28 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:20:28.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:28 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:29.905 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:20:29.906 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:20:29.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:29.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:20:30.017 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:20:30.058 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:30.111 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:20:30.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:30.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:20:30.215 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:20:30.250 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:30.273 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:20:30.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:30.276 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:20:30.377 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:20:30.408 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:30.430 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:20:30.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:30.434 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:20:30.535 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:20:30.576 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:31.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:31 smithi115 ceph-mon[72476]: pgmap v1568: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 415 op/s; 8695/338553 objects degraded (2.568%); 38650/338553 objects misplaced (11.416%); 6.2 MiB/s, 99 objects/s recovering 2024-03-20T09:20:31.431 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 8, 10, 11, 3, 4, 9] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 5] 2024-03-20T09:20:31.431 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 0 1 2024-03-20T09:20:31.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:31 smithi122 ceph-mon[73495]: pgmap v1568: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 415 op/s; 8695/338553 objects degraded (2.568%); 38650/338553 objects misplaced (11.416%); 6.2 MiB/s, 99 objects/s recovering 2024-03-20T09:20:31.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:31 smithi026 ceph-mon[68309]: pgmap v1568: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 415 op/s; 8695/338553 objects degraded (2.568%); 38650/338553 objects misplaced (11.416%); 6.2 MiB/s, 99 objects/s recovering 2024-03-20T09:20:33.139 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:33 smithi026 ceph-mon[68309]: pgmap v1569: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 468 op/s; 8695/335163 objects degraded (2.594%); 38317/335163 objects misplaced (11.432%); 7.5 MiB/s, 120 objects/s recovering 2024-03-20T09:20:33.139 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:33 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8695/335163 objects degraded (2.594%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:20:33.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:33 smithi115 ceph-mon[72476]: pgmap v1569: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 468 op/s; 8695/335163 objects degraded (2.594%); 38317/335163 objects misplaced (11.432%); 7.5 MiB/s, 120 objects/s recovering 2024-03-20T09:20:33.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:33 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8695/335163 objects degraded (2.594%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:20:33.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:33 smithi122 ceph-mon[73495]: pgmap v1569: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 468 op/s; 8695/335163 objects degraded (2.594%); 38317/335163 objects misplaced (11.432%); 7.5 MiB/s, 120 objects/s recovering 2024-03-20T09:20:33.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:33 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8695/335163 objects degraded (2.594%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:20:34.146 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:20:34.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:34.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:20:34.250 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:20:34.290 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:34.765 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:20:34.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:34.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:20:34.875 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:20:34.906 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:34.929 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:20:34.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:34.934 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:20:35.035 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:20:35.078 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:35.153 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:35 smithi122 ceph-mon[73495]: pgmap v1570: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 453 op/s; 8694/332607 objects degraded (2.614%); 37694/332607 objects misplaced (11.333%); 8.9 MiB/s, 142 objects/s recovering 2024-03-20T09:20:35.153 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:35 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:20:35.153 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:35 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:20:35.153 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:35 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:20:35.153 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:35 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:20:35.153 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:35 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:20:35.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:35 smithi115 ceph-mon[72476]: pgmap v1570: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 453 op/s; 8694/332607 objects degraded (2.614%); 37694/332607 objects misplaced (11.333%); 8.9 MiB/s, 142 objects/s recovering 2024-03-20T09:20:35.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:35 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:20:35.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:35 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:20:35.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:20:35.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:20:35.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:20:35.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:20:35.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:20:35.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:20:35.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:35 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:35.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:35 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:20:35.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:35 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:20:35.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:35 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:20:35.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:35 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:35.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:35 smithi026 ceph-mon[68309]: pgmap v1570: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 453 op/s; 8694/332607 objects degraded (2.614%); 37694/332607 objects misplaced (11.333%); 8.9 MiB/s, 142 objects/s recovering 2024-03-20T09:20:35.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:35 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:20:35.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:35 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:20:35.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:35 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:20:35.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:35 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:20:35.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:35 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:20:35.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:35 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:20:35.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:35 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:20:35.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:35 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:20:35.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:35 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:35.743 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:20:35.787 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:20:35.830 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:20:35.862 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:20:35.896 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:20:35.937 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:20:35.965 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:20:35.991 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:20:36.025 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:20:36.052 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:20:36.087 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:20:36.132 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:20:36.216 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:20:36.217 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:20:36.217 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:20:36.217 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:20:36.217 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:20:36.217 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:20:36.217 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:20:36.218 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:20:36.218 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:20:36.218 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:20:36.219 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:20:36.219 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:20:36.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:36 smithi122 ceph-mon[73495]: osdmap e652: 12 total, 12 up, 12 in 2024-03-20T09:20:36.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:36 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3286923894' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:20:36.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:36 smithi122 ceph-mon[73495]: pgmap v1572: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 478 op/s; 8694/329508 objects degraded (2.638%); 37554/329508 objects misplaced (11.397%); 7.5 MiB/s, 119 objects/s recovering 2024-03-20T09:20:36.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:36 smithi026 ceph-mon[68309]: osdmap e652: 12 total, 12 up, 12 in 2024-03-20T09:20:36.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:36 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3286923894' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:20:36.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:36 smithi026 ceph-mon[68309]: pgmap v1572: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 478 op/s; 8694/329508 objects degraded (2.638%); 37554/329508 objects misplaced (11.397%); 7.5 MiB/s, 119 objects/s recovering 2024-03-20T09:20:36.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:36 smithi115 ceph-mon[72476]: osdmap e652: 12 total, 12 up, 12 in 2024-03-20T09:20:36.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:36 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3286923894' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:20:36.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:36 smithi115 ceph-mon[72476]: pgmap v1572: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 478 op/s; 8694/329508 objects degraded (2.638%); 37554/329508 objects misplaced (11.397%); 7.5 MiB/s, 119 objects/s recovering 2024-03-20T09:20:36.943 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:20:36.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:36.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:20:37.049 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:20:37.086 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:37.109 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:20:37.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:37.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:20:37.214 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:20:37.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:37 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:20:37.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:37 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:20:37.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:37 smithi026 ceph-mon[68309]: osdmap e653: 12 total, 12 up, 12 in 2024-03-20T09:20:37.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:37 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:20:37.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:37 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:20:37.287 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:37.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:37 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:20:37.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:37 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:20:37.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:37 smithi115 ceph-mon[72476]: osdmap e653: 12 total, 12 up, 12 in 2024-03-20T09:20:37.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:37 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:20:37.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:37 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:20:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:37 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:20:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:37 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:20:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:37 smithi122 ceph-mon[73495]: osdmap e653: 12 total, 12 up, 12 in 2024-03-20T09:20:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:37 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:20:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:37 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:20:38.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:38 smithi115 ceph-mon[72476]: osdmap e654: 12 total, 12 up, 12 in 2024-03-20T09:20:38.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:38 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8694/329508 objects degraded (2.638%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:20:38.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:38 smithi115 ceph-mon[72476]: pgmap v1575: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 537 op/s; 8693/325488 objects degraded (2.671%); 36789/325488 objects misplaced (11.303%); 9.6 MiB/s, 154 objects/s recovering 2024-03-20T09:20:38.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:38 smithi122 ceph-mon[73495]: osdmap e654: 12 total, 12 up, 12 in 2024-03-20T09:20:38.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:38 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8694/329508 objects degraded (2.638%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:20:38.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:38 smithi122 ceph-mon[73495]: pgmap v1575: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 537 op/s; 8693/325488 objects degraded (2.671%); 36789/325488 objects misplaced (11.303%); 9.6 MiB/s, 154 objects/s recovering 2024-03-20T09:20:38.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:38 smithi026 ceph-mon[68309]: osdmap e654: 12 total, 12 up, 12 in 2024-03-20T09:20:38.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:38 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8694/329508 objects degraded (2.638%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:20:38.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:38 smithi026 ceph-mon[68309]: pgmap v1575: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 537 op/s; 8693/325488 objects degraded (2.671%); 36789/325488 objects misplaced (11.303%); 9.6 MiB/s, 154 objects/s recovering 2024-03-20T09:20:40.671 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:20:40.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:40.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:20:40.775 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:20:40.809 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:40.846 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:20:40.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:40.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:20:40.856 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:20:40.872 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:20:40.872 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:20:40.953 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:20:40.987 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:41.019 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:20:41.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:41.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:20:41.124 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:20:41.156 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:41.175 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:41 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:20:41.175 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:41 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:20:41.175 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:41 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:20:41.175 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:41 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:20:41.175 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:41 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:20:41.175 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:41 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:20:41.175 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:41 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:41.176 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:41 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1208771001' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:20:41.182 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:20:41.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:41.187 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:20:41.223 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:20:41.287 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:20:41.364 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:41.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:41 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:20:41.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:41 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:20:41.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:41 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:20:41.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:41 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:20:41.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:41 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:20:41.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:41 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:20:41.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:41 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:41.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:41 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1208771001' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:20:41.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:41 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:20:41.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:41 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:20:41.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:41 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:20:41.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:41 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:20:41.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:41 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:20:41.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:41 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:20:41.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:41 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:20:41.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:41 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1208771001' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:20:41.815 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:20:41.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:41.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:20:41.920 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:20:41.954 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:42.137 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:42 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:20:42.137 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:42 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1208771001' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:20:42.137 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:42 smithi026 ceph-mon[68309]: osdmap e655: 12 total, 12 up, 12 in 2024-03-20T09:20:42.137 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:42 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:20:42.137 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:42 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:20:42.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:42 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:20:42.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:42 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1208771001' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:20:42.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:42 smithi115 ceph-mon[72476]: osdmap e655: 12 total, 12 up, 12 in 2024-03-20T09:20:42.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:42 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:20:42.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:42 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:20:42.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:42 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:20:42.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:42 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1208771001' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:20:42.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:42 smithi122 ceph-mon[73495]: osdmap e655: 12 total, 12 up, 12 in 2024-03-20T09:20:42.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:42 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:20:42.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:42 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:20:42.609 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:20:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: pgmap v1578: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 466 op/s; 8693/321117 objects degraded (2.707%); 36435/321117 objects misplaced (11.346%); 8.6 MiB/s, 138 objects/s recovering 2024-03-20T09:20:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: osdmap e656: 12 total, 12 up, 12 in 2024-03-20T09:20:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8693/321117 objects degraded (2.707%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:20:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {auth=34,config=2,health=12,kv=52} crc {auth=170454887,config=3599132250,health=1190912573,kv=2495135322}) 2024-03-20T09:20:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {kv=7,logm=93} crc {kv=1870336269,logm=1993169924}) 2024-03-20T09:20:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2159133334}) 2024-03-20T09:20:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3943270548}) 2024-03-20T09:20:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2703014015}) 2024-03-20T09:20:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3977679037}) 2024-03-20T09:20:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=799309673}) 2024-03-20T09:20:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4195593963}) 2024-03-20T09:20:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=135680167}) 2024-03-20T09:20:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=573786874}) 2024-03-20T09:20:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1443144167}) 2024-03-20T09:20:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2951652205}) 2024-03-20T09:20:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3558909144}) 2024-03-20T09:20:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2119656565}) 2024-03-20T09:20:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3335057020}) 2024-03-20T09:20:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3968236221}) 2024-03-20T09:20:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3482803010}) 2024-03-20T09:20:43.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2449953336}) 2024-03-20T09:20:43.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4101471215}) 2024-03-20T09:20:43.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2176968650}) 2024-03-20T09:20:43.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2338138399}) 2024-03-20T09:20:43.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3575153853}) 2024-03-20T09:20:43.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=578572690}) 2024-03-20T09:20:43.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2256096027}) 2024-03-20T09:20:43.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3637797144}) 2024-03-20T09:20:43.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3667709284}) 2024-03-20T09:20:43.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1074788228}) 2024-03-20T09:20:43.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=984165686}) 2024-03-20T09:20:43.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3755196664}) 2024-03-20T09:20:43.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3218025423}) 2024-03-20T09:20:43.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2304118738}) 2024-03-20T09:20:43.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=724641487}) 2024-03-20T09:20:43.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=73046581}) 2024-03-20T09:20:43.412 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3563660832}) 2024-03-20T09:20:43.412 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=824458794}) 2024-03-20T09:20:43.412 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2380665462}) 2024-03-20T09:20:43.412 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3366750161}) 2024-03-20T09:20:43.412 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1977419419}) 2024-03-20T09:20:43.412 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487428298}) 2024-03-20T09:20:43.412 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1730616147}) 2024-03-20T09:20:43.412 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2078997020}) 2024-03-20T09:20:43.412 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3257053252}) 2024-03-20T09:20:43.413 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1274471828}) 2024-03-20T09:20:43.413 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3929322627}) 2024-03-20T09:20:43.413 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1427203055}) 2024-03-20T09:20:43.413 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=725539873}) 2024-03-20T09:20:43.413 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2753406058}) 2024-03-20T09:20:43.413 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3891111196}) 2024-03-20T09:20:43.413 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2167432268}) 2024-03-20T09:20:43.413 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1545849901}) 2024-03-20T09:20:43.414 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1599325490}) 2024-03-20T09:20:43.414 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2255691456}) 2024-03-20T09:20:43.414 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=619342452}) 2024-03-20T09:20:43.414 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=36480989}) 2024-03-20T09:20:43.414 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2285080519}) 2024-03-20T09:20:43.414 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=803019023}) 2024-03-20T09:20:43.414 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2735740705}) 2024-03-20T09:20:43.415 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=41,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=11,mon_config_key=24} crc {logm=1276869795,mdsmap=3984425268,mgr=427508465,mgr_command_descs=3831600018,mgr_metadata=2702834764,mgrstat=418878718,mon_config_key=3477185178}) 2024-03-20T09:20:43.415 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1637696379}) 2024-03-20T09:20:43.415 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2656284593}) 2024-03-20T09:20:43.415 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=54,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=28} crc {mon_config_key=611748137,monmap=3481182261,osd_metadata=1898051788,osd_pg_creating=4224505912,osdmap=915835400}) 2024-03-20T09:20:43.415 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=81} crc {osdmap=2454376909}) 2024-03-20T09:20:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: pgmap v1578: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 466 op/s; 8693/321117 objects degraded (2.707%); 36435/321117 objects misplaced (11.346%); 8.6 MiB/s, 138 objects/s recovering 2024-03-20T09:20:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: osdmap e656: 12 total, 12 up, 12 in 2024-03-20T09:20:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8693/321117 objects degraded (2.707%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:20:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {auth=34,config=2,health=12,kv=52} crc {auth=170454887,config=3599132250,health=1190912573,kv=2495135322}) 2024-03-20T09:20:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {kv=7,logm=93} crc {kv=1870336269,logm=1993169924}) 2024-03-20T09:20:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2159133334}) 2024-03-20T09:20:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3943270548}) 2024-03-20T09:20:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2703014015}) 2024-03-20T09:20:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3977679037}) 2024-03-20T09:20:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=799309673}) 2024-03-20T09:20:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4195593963}) 2024-03-20T09:20:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=135680167}) 2024-03-20T09:20:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=573786874}) 2024-03-20T09:20:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1443144167}) 2024-03-20T09:20:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2951652205}) 2024-03-20T09:20:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3558909144}) 2024-03-20T09:20:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2119656565}) 2024-03-20T09:20:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3335057020}) 2024-03-20T09:20:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3968236221}) 2024-03-20T09:20:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3482803010}) 2024-03-20T09:20:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2449953336}) 2024-03-20T09:20:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4101471215}) 2024-03-20T09:20:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2176968650}) 2024-03-20T09:20:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2338138399}) 2024-03-20T09:20:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3575153853}) 2024-03-20T09:20:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=578572690}) 2024-03-20T09:20:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2256096027}) 2024-03-20T09:20:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3637797144}) 2024-03-20T09:20:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3667709284}) 2024-03-20T09:20:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1074788228}) 2024-03-20T09:20:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=984165686}) 2024-03-20T09:20:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3755196664}) 2024-03-20T09:20:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3218025423}) 2024-03-20T09:20:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2304118738}) 2024-03-20T09:20:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=724641487}) 2024-03-20T09:20:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=73046581}) 2024-03-20T09:20:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3563660832}) 2024-03-20T09:20:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=824458794}) 2024-03-20T09:20:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2380665462}) 2024-03-20T09:20:43.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3366750161}) 2024-03-20T09:20:43.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1977419419}) 2024-03-20T09:20:43.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487428298}) 2024-03-20T09:20:43.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1730616147}) 2024-03-20T09:20:43.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2078997020}) 2024-03-20T09:20:43.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3257053252}) 2024-03-20T09:20:43.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1274471828}) 2024-03-20T09:20:43.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3929322627}) 2024-03-20T09:20:43.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1427203055}) 2024-03-20T09:20:43.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=725539873}) 2024-03-20T09:20:43.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2753406058}) 2024-03-20T09:20:43.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3891111196}) 2024-03-20T09:20:43.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2167432268}) 2024-03-20T09:20:43.451 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1545849901}) 2024-03-20T09:20:43.452 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1599325490}) 2024-03-20T09:20:43.452 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2255691456}) 2024-03-20T09:20:43.452 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=619342452}) 2024-03-20T09:20:43.452 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=36480989}) 2024-03-20T09:20:43.452 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2285080519}) 2024-03-20T09:20:43.452 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=803019023}) 2024-03-20T09:20:43.452 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2735740705}) 2024-03-20T09:20:43.452 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=41,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=11,mon_config_key=24} crc {logm=1276869795,mdsmap=3984425268,mgr=427508465,mgr_command_descs=3831600018,mgr_metadata=2702834764,mgrstat=418878718,mon_config_key=3477185178}) 2024-03-20T09:20:43.452 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1637696379}) 2024-03-20T09:20:43.452 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2656284593}) 2024-03-20T09:20:43.452 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=54,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=28} crc {mon_config_key=611748137,monmap=3481182261,osd_metadata=1898051788,osd_pg_creating=4224505912,osdmap=915835400}) 2024-03-20T09:20:43.452 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=81} crc {osdmap=2454376909}) 2024-03-20T09:20:43.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: pgmap v1578: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 466 op/s; 8693/321117 objects degraded (2.707%); 36435/321117 objects misplaced (11.346%); 8.6 MiB/s, 138 objects/s recovering 2024-03-20T09:20:43.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: osdmap e656: 12 total, 12 up, 12 in 2024-03-20T09:20:43.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8693/321117 objects degraded (2.707%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:20:43.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {auth=34,config=2,health=12,kv=52} crc {auth=170454887,config=3599132250,health=1190912573,kv=2495135322}) 2024-03-20T09:20:43.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {kv=7,logm=93} crc {kv=1870336269,logm=1993169924}) 2024-03-20T09:20:43.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2159133334}) 2024-03-20T09:20:43.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3943270548}) 2024-03-20T09:20:43.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2703014015}) 2024-03-20T09:20:43.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3977679037}) 2024-03-20T09:20:43.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=799309673}) 2024-03-20T09:20:43.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4195593963}) 2024-03-20T09:20:43.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=135680167}) 2024-03-20T09:20:43.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=573786874}) 2024-03-20T09:20:43.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1443144167}) 2024-03-20T09:20:43.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2951652205}) 2024-03-20T09:20:43.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3558909144}) 2024-03-20T09:20:43.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2119656565}) 2024-03-20T09:20:43.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3335057020}) 2024-03-20T09:20:43.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3968236221}) 2024-03-20T09:20:43.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3482803010}) 2024-03-20T09:20:43.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2449953336}) 2024-03-20T09:20:43.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4101471215}) 2024-03-20T09:20:43.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2176968650}) 2024-03-20T09:20:43.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2338138399}) 2024-03-20T09:20:43.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3575153853}) 2024-03-20T09:20:43.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=578572690}) 2024-03-20T09:20:43.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2256096027}) 2024-03-20T09:20:43.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3637797144}) 2024-03-20T09:20:43.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3667709284}) 2024-03-20T09:20:43.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1074788228}) 2024-03-20T09:20:43.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=984165686}) 2024-03-20T09:20:43.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3755196664}) 2024-03-20T09:20:43.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3218025423}) 2024-03-20T09:20:43.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2304118738}) 2024-03-20T09:20:43.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=724641487}) 2024-03-20T09:20:43.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=73046581}) 2024-03-20T09:20:43.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3563660832}) 2024-03-20T09:20:43.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=824458794}) 2024-03-20T09:20:43.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2380665462}) 2024-03-20T09:20:43.475 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3366750161}) 2024-03-20T09:20:43.475 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1977419419}) 2024-03-20T09:20:43.475 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=487428298}) 2024-03-20T09:20:43.475 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1730616147}) 2024-03-20T09:20:43.475 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2078997020}) 2024-03-20T09:20:43.475 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3257053252}) 2024-03-20T09:20:43.475 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1274471828}) 2024-03-20T09:20:43.475 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3929322627}) 2024-03-20T09:20:43.475 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1427203055}) 2024-03-20T09:20:43.475 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=725539873}) 2024-03-20T09:20:43.475 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2753406058}) 2024-03-20T09:20:43.475 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3891111196}) 2024-03-20T09:20:43.475 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2167432268}) 2024-03-20T09:20:43.475 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1545849901}) 2024-03-20T09:20:43.475 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1599325490}) 2024-03-20T09:20:43.476 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2255691456}) 2024-03-20T09:20:43.476 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=619342452}) 2024-03-20T09:20:43.476 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=36480989}) 2024-03-20T09:20:43.476 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2285080519}) 2024-03-20T09:20:43.476 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=803019023}) 2024-03-20T09:20:43.476 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2735740705}) 2024-03-20T09:20:43.476 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=41,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=11,mon_config_key=24} crc {logm=1276869795,mdsmap=3984425268,mgr=427508465,mgr_command_descs=3831600018,mgr_metadata=2702834764,mgrstat=418878718,mon_config_key=3477185178}) 2024-03-20T09:20:43.476 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1637696379}) 2024-03-20T09:20:43.476 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2656284593}) 2024-03-20T09:20:43.476 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=54,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=28} crc {mon_config_key=611748137,monmap=3481182261,osd_metadata=1898051788,osd_pg_creating=4224505912,osdmap=915835400}) 2024-03-20T09:20:43.476 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=81} crc {osdmap=2454376909}) 2024-03-20T09:20:43.821 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:20:43.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:43.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:20:43.927 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:20:43.985 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:44.016 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:20:44.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:44.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:20:44.127 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:20:44.156 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:44.177 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:20:44.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:44.183 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:20:44.284 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:20:44.333 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:44.364 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:20:44.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:44.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:20:44.469 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:20:44.532 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:44.567 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:20:44.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:44.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:20:44.671 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:20:44.699 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:44.723 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:20:44.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:44.731 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:20:44.832 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:20:44.877 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:45.140 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:45 smithi122 ceph-mon[73495]: pgmap v1580: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 464 op/s; 8692/318678 objects degraded (2.728%); 35824/318678 objects misplaced (11.241%); 8.6 MiB/s, 137 objects/s recovering 2024-03-20T09:20:45.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:45 smithi115 ceph-mon[72476]: pgmap v1580: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 464 op/s; 8692/318678 objects degraded (2.728%); 35824/318678 objects misplaced (11.241%); 8.6 MiB/s, 137 objects/s recovering 2024-03-20T09:20:45.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:45 smithi026 ceph-mon[68309]: pgmap v1580: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 464 op/s; 8692/318678 objects degraded (2.728%); 35824/318678 objects misplaced (11.241%); 8.6 MiB/s, 137 objects/s recovering 2024-03-20T09:20:47.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:47 smithi115 ceph-mon[72476]: pgmap v1581: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 398 op/s; 8692/315924 objects degraded (2.751%); 35671/315924 objects misplaced (11.291%); 6.3 MiB/s, 101 objects/s recovering 2024-03-20T09:20:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:47 smithi122 ceph-mon[73495]: pgmap v1581: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 398 op/s; 8692/315924 objects degraded (2.751%); 35671/315924 objects misplaced (11.291%); 6.3 MiB/s, 101 objects/s recovering 2024-03-20T09:20:47.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:47 smithi026 ceph-mon[68309]: pgmap v1581: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 398 op/s; 8692/315924 objects degraded (2.751%); 35671/315924 objects misplaced (11.291%); 6.3 MiB/s, 101 objects/s recovering 2024-03-20T09:20:48.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:48 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8692/315924 objects degraded (2.751%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:20:48.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:48 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8692/315924 objects degraded (2.751%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:20:48.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:48 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8692/315924 objects degraded (2.751%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:20:49.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:49 smithi122 ceph-mon[73495]: pgmap v1582: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 526 op/s; 8691/311952 objects degraded (2.786%); 34880/311952 objects misplaced (11.181%); 10 MiB/s, 163 objects/s recovering 2024-03-20T09:20:49.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:49 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:20:49.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:49 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:20:49.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:49 smithi026 ceph-mon[68309]: pgmap v1582: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 526 op/s; 8691/311952 objects degraded (2.786%); 34880/311952 objects misplaced (11.181%); 10 MiB/s, 163 objects/s recovering 2024-03-20T09:20:49.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:49 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:20:49.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:49 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:20:49.670 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:49 smithi115 ceph-mon[72476]: pgmap v1582: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 526 op/s; 8691/311952 objects degraded (2.786%); 34880/311952 objects misplaced (11.181%); 10 MiB/s, 163 objects/s recovering 2024-03-20T09:20:49.670 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:49 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:20:49.670 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:49 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:20:50.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:50 smithi122 ceph-mon[73495]: osdmap e657: 12 total, 12 up, 12 in 2024-03-20T09:20:50.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:50 smithi122 ceph-mon[73495]: pgmap v1584: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 416 op/s; 8691/311112 objects degraded (2.794%); 34880/311112 objects misplaced (11.211%); 7.5 MiB/s, 119 objects/s recovering 2024-03-20T09:20:50.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:50 smithi026 ceph-mon[68309]: osdmap e657: 12 total, 12 up, 12 in 2024-03-20T09:20:50.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:50 smithi026 ceph-mon[68309]: pgmap v1584: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 416 op/s; 8691/311112 objects degraded (2.794%); 34880/311112 objects misplaced (11.211%); 7.5 MiB/s, 119 objects/s recovering 2024-03-20T09:20:50.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:50 smithi115 ceph-mon[72476]: osdmap e657: 12 total, 12 up, 12 in 2024-03-20T09:20:50.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:50 smithi115 ceph-mon[72476]: pgmap v1584: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 416 op/s; 8691/311112 objects degraded (2.794%); 34880/311112 objects misplaced (11.211%); 7.5 MiB/s, 119 objects/s recovering 2024-03-20T09:20:51.038 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:20:51.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:51.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:20:51.080 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-20T09:20:51.144 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:20:51.175 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:51.195 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:20:51.198 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:20:51.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:51.200 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:20:51.301 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:20:51.358 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:51.389 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:20:51.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:51.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:20:51.498 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:20:51.531 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:51.553 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:20:51.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:51.557 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:20:51.580 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:20:51.624 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:20:51.646 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:20:51.658 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:20:51.709 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:53.079 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:20:53.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:53.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:20:53.189 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:20:53.235 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:53.262 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:20:53.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:53.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:20:53.368 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:20:53.393 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:53 smithi026 ceph-mon[68309]: pgmap v1585: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 458 op/s; 8691/307578 objects degraded (2.826%); 34522/307578 objects misplaced (11.224%); 8.3 MiB/s, 133 objects/s recovering 2024-03-20T09:20:53.394 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:53 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8691/307578 objects degraded (2.826%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:20:53.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:53 smithi115 ceph-mon[72476]: pgmap v1585: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 458 op/s; 8691/307578 objects degraded (2.826%); 34522/307578 objects misplaced (11.224%); 8.3 MiB/s, 133 objects/s recovering 2024-03-20T09:20:53.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:53 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8691/307578 objects degraded (2.826%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:20:53.415 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:53.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:53 smithi122 ceph-mon[73495]: pgmap v1585: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 458 op/s; 8691/307578 objects degraded (2.826%); 34522/307578 objects misplaced (11.224%); 8.3 MiB/s, 133 objects/s recovering 2024-03-20T09:20:53.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:53 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8691/307578 objects degraded (2.826%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:20:53.453 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:20:53.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:53.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:20:53.566 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:20:53.602 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:53.630 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:20:53.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:53.636 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:20:53.737 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:20:53.767 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:53.795 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:20:53.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:53.799 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:20:53.900 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:20:53.929 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:53.953 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:20:53.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:53.956 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:20:54.057 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:20:54.092 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:54.119 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:20:54.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:54.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:20:54.228 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:20:54.292 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:54.444 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:20:54.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:54.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:20:54.555 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:20:54.614 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:55.158 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.0 to 1 (10000) 2024-03-20T09:20:55.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:55 smithi115 ceph-mon[72476]: pgmap v1586: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 453 op/s; 8690/305079 objects degraded (2.848%); 33902/305079 objects misplaced (11.113%); 8.2 MiB/s, 131 objects/s recovering 2024-03-20T09:20:55.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:55 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/267901058' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-20T09:20:55.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:55 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-20T09:20:55.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:55 smithi122 ceph-mon[73495]: pgmap v1586: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 453 op/s; 8690/305079 objects degraded (2.848%); 33902/305079 objects misplaced (11.113%); 8.2 MiB/s, 131 objects/s recovering 2024-03-20T09:20:55.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:55 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/267901058' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-20T09:20:55.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:55 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-20T09:20:55.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:55 smithi026 ceph-mon[68309]: pgmap v1586: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 453 op/s; 8690/305079 objects degraded (2.848%); 33902/305079 objects misplaced (11.113%); 8.2 MiB/s, 131 objects/s recovering 2024-03-20T09:20:55.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:55 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/267901058' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-20T09:20:55.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:55 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-20T09:20:56.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:56 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-20T09:20:56.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:56 smithi115 ceph-mon[72476]: osdmap e658: 12 total, 12 up, 12 in 2024-03-20T09:20:56.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:56 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-20T09:20:56.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:56 smithi122 ceph-mon[73495]: osdmap e658: 12 total, 12 up, 12 in 2024-03-20T09:20:56.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:56 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-20T09:20:56.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:56 smithi026 ceph-mon[68309]: osdmap e658: 12 total, 12 up, 12 in 2024-03-20T09:20:57.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:57 smithi115 ceph-mon[72476]: pgmap v1588: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 397 op/s; 8690/302415 objects degraded (2.874%); 33902/302415 objects misplaced (11.210%); 5.2 MiB/s, 83 objects/s recovering 2024-03-20T09:20:57.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:57 smithi122 ceph-mon[73495]: pgmap v1588: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 397 op/s; 8690/302415 objects degraded (2.874%); 33902/302415 objects misplaced (11.210%); 5.2 MiB/s, 83 objects/s recovering 2024-03-20T09:20:57.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:57 smithi026 ceph-mon[68309]: pgmap v1588: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 397 op/s; 8690/302415 objects degraded (2.874%); 33902/302415 objects misplaced (11.210%); 5.2 MiB/s, 83 objects/s recovering 2024-03-20T09:20:58.224 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:20:58.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:58.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:20:58.329 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:20:58.393 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:58.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:58 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8690/302415 objects degraded (2.874%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:20:58.421 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:58 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8690/302415 objects degraded (2.874%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:20:58.441 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:20:58.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:58.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:20:58.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:58 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8690/302415 objects degraded (2.874%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:20:58.545 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:20:58.577 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:58.597 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:20:58.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:58.602 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:20:58.703 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:20:58.733 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:58.759 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:20:58.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:58.765 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:20:58.866 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:20:58.881 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 1 1 2024-03-20T09:20:58.899 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:58.924 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:20:58.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:58.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:20:59.028 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:20:59.057 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:59.079 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:20:59.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:59.083 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:20:59.162 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:20:59 smithi026 ceph-mon[68309]: pgmap v1589: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 526 op/s; 8689/297888 objects degraded (2.917%); 32933/297888 objects misplaced (11.055%); 9.3 MiB/s, 149 objects/s recovering 2024-03-20T09:20:59.184 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:20:59.220 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:59.254 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:20:59.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:59.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:20:59.358 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:20:59.389 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:59.421 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:20:59.423 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:20:59 smithi115 ceph-mon[72476]: pgmap v1589: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 526 op/s; 8689/297888 objects degraded (2.917%); 32933/297888 objects misplaced (11.055%); 9.3 MiB/s, 149 objects/s recovering 2024-03-20T09:20:59.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:59.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:20:59.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:20:59 smithi122 ceph-mon[73495]: pgmap v1589: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 526 op/s; 8689/297888 objects degraded (2.917%); 32933/297888 objects misplaced (11.055%); 9.3 MiB/s, 149 objects/s recovering 2024-03-20T09:20:59.525 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:20:59.557 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:20:59.590 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:20:59.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:20:59.594 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:20:59.694 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:20:59.738 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:01.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:01 smithi122 ceph-mon[73495]: pgmap v1590: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 471 op/s; 8689/296973 objects degraded (2.926%); 32933/296973 objects misplaced (11.090%); 8.3 MiB/s, 132 objects/s recovering 2024-03-20T09:21:01.475 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:01 smithi026 ceph-mon[68309]: pgmap v1590: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 471 op/s; 8689/296973 objects degraded (2.926%); 32933/296973 objects misplaced (11.090%); 8.3 MiB/s, 132 objects/s recovering 2024-03-20T09:21:01.668 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:01 smithi115 ceph-mon[72476]: pgmap v1590: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 471 op/s; 8689/296973 objects degraded (2.926%); 32933/296973 objects misplaced (11.090%); 8.3 MiB/s, 132 objects/s recovering 2024-03-20T09:21:02.452 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:02 smithi122 ceph-mon[73495]: pgmap v1591: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 474 op/s; 8689/293325 objects degraded (2.962%); 32577/293325 objects misplaced (11.106%); 8.3 MiB/s, 132 objects/s recovering 2024-03-20T09:21:02.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:02 smithi026 ceph-mon[68309]: pgmap v1591: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 474 op/s; 8689/293325 objects degraded (2.962%); 32577/293325 objects misplaced (11.106%); 8.3 MiB/s, 132 objects/s recovering 2024-03-20T09:21:02.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:02 smithi115 ceph-mon[72476]: pgmap v1591: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 6.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 474 op/s; 8689/293325 objects degraded (2.962%); 32577/293325 objects misplaced (11.106%); 8.3 MiB/s, 132 objects/s recovering 2024-03-20T09:21:03.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:03 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8689/293325 objects degraded (2.962%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:21:03.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:03 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8689/293325 objects degraded (2.962%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:21:03.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:03 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8689/293325 objects degraded (2.962%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:21:03.714 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:21:03.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:03.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:21:03.822 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:21:03.854 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:03.884 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:21:03.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:03.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:21:03.991 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:21:04.020 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:04.044 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:21:04.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:04.047 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:21:04.148 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:21:04.179 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:04.202 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:21:04.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:04.209 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:21:04.310 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:21:04.355 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:04.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:04 smithi122 ceph-mon[73495]: pgmap v1592: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 483 op/s; 8684/290580 objects degraded (2.989%); 31995/290580 objects misplaced (11.011%); 8.2 MiB/s, 131 objects/s recovering 2024-03-20T09:21:04.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:04 smithi026 ceph-mon[68309]: pgmap v1592: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 483 op/s; 8684/290580 objects degraded (2.989%); 31995/290580 objects misplaced (11.011%); 8.2 MiB/s, 131 objects/s recovering 2024-03-20T09:21:04.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:04 smithi115 ceph-mon[72476]: pgmap v1592: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 483 op/s; 8684/290580 objects degraded (2.989%); 31995/290580 objects misplaced (11.011%); 8.2 MiB/s, 131 objects/s recovering 2024-03-20T09:21:07.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:07 smithi115 ceph-mon[72476]: pgmap v1593: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 501 op/s; 8684/287772 objects degraded (3.018%); 31995/287772 objects misplaced (11.118%); 7.5 MiB/s, 120 objects/s recovering 2024-03-20T09:21:07.432 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:07 smithi026 ceph-mon[68309]: pgmap v1593: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 501 op/s; 8684/287772 objects degraded (3.018%); 31995/287772 objects misplaced (11.118%); 7.5 MiB/s, 120 objects/s recovering 2024-03-20T09:21:07.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:07 smithi122 ceph-mon[73495]: pgmap v1593: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 501 op/s; 8684/287772 objects degraded (3.018%); 31995/287772 objects misplaced (11.118%); 7.5 MiB/s, 120 objects/s recovering 2024-03-20T09:21:07.541 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:21:07.541 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:21:07.541 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:21:07.541 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:21:07.541 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:21:07.541 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:21:07.546 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:21:07.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:07.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:21:07.569 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:21:07.622 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:21:07.649 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:21:07.650 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:21:07.677 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:07.694 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:21:07.703 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:21:07.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:07.705 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:21:07.734 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:21:07.787 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:21:07.802 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:21:07.802 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:21:07.802 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:21:07.802 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:21:07.802 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:21:07.802 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:21:07.802 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:21:07.802 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:21:07.803 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:21:07.803 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:21:07.803 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:21:07.803 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:21:07.806 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:21:07.842 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:21:07.852 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:07.879 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:21:07.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:07.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:21:07.985 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:21:08.015 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:08.038 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:21:08.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:08.041 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:21:08.054 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:21:08.054 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:21:08.054 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-03-20T09:21:08.054 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-03-20T09:21:08.054 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:21:08.142 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:21:08.160 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:21:08.171 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:08.197 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:21:08.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:08.200 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:21:08.301 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:21:08.351 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:08.388 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:21:08.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:08.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:21:08.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:08 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1295698336' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:21:08.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:08 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8684/287772 objects degraded (3.018%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:21:08.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:08 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/370772064' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:21:08.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:08 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:21:08.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:08 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1295698336' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:21:08.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:08 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8684/287772 objects degraded (3.018%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:21:08.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:08 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/370772064' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:21:08.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:08 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:21:08.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:08 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1295698336' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:21:08.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:08 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8684/287772 objects degraded (3.018%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:21:08.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:08 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/370772064' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:21:08.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:08 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:21:08.496 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:21:08.527 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:08.551 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:21:08.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:08.556 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:21:08.657 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:21:08.687 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:08.708 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:21:08.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:08.714 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:21:08.815 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:21:08.861 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:08.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:08.914 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:21:08.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:21:09.015 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:21:09.057 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:09.091 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:21:09.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:09.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:21:09.200 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:21:09.270 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:09.305 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:21:09.310 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:09 smithi026 ceph-mon[68309]: pgmap v1594: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 538 op/s; 8684/283029 objects degraded (3.068%); 30999/283029 objects misplaced (10.953%); 10 MiB/s, 165 objects/s recovering 2024-03-20T09:21:09.311 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:09 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:21:09.311 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:09 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:21:09.311 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:09 smithi026 ceph-mon[68309]: osdmap e659: 12 total, 12 up, 12 in 2024-03-20T09:21:09.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:09.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:21:09.413 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:21:09.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:09 smithi122 ceph-mon[73495]: pgmap v1594: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 538 op/s; 8684/283029 objects degraded (3.068%); 30999/283029 objects misplaced (10.953%); 10 MiB/s, 165 objects/s recovering 2024-03-20T09:21:09.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:09 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:21:09.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:09 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:21:09.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:09 smithi122 ceph-mon[73495]: osdmap e659: 12 total, 12 up, 12 in 2024-03-20T09:21:09.441 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:09.473 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:21:09.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:09.477 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:21:09.578 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:21:09.627 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:09.661 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:09 smithi115 ceph-mon[72476]: pgmap v1594: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 538 op/s; 8684/283029 objects degraded (3.068%); 30999/283029 objects misplaced (10.953%); 10 MiB/s, 165 objects/s recovering 2024-03-20T09:21:09.661 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:09 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:21:09.661 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:09 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:21:09.661 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:09 smithi115 ceph-mon[72476]: osdmap e659: 12 total, 12 up, 12 in 2024-03-20T09:21:09.885 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:21:09.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:09.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:21:09.993 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:21:10.035 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:11.083 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:21:11.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:11 smithi122 ceph-mon[73495]: pgmap v1596: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 496 op/s; 8684/282081 objects degraded (3.079%); 30999/282081 objects misplaced (10.989%); 8.3 MiB/s, 132 objects/s recovering 2024-03-20T09:21:11.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:11 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:21:11.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:11 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:21:11.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:11 smithi026 ceph-mon[68309]: pgmap v1596: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 496 op/s; 8684/282081 objects degraded (3.079%); 30999/282081 objects misplaced (10.989%); 8.3 MiB/s, 132 objects/s recovering 2024-03-20T09:21:11.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:11 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:21:11.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:11 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:21:11.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:11 smithi115 ceph-mon[72476]: pgmap v1596: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 496 op/s; 8684/282081 objects degraded (3.079%); 30999/282081 objects misplaced (10.989%); 8.3 MiB/s, 132 objects/s recovering 2024-03-20T09:21:11.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:11 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:21:11.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:11 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:21:11.923 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:21:12.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:12 smithi026 ceph-mon[68309]: osdmap e660: 12 total, 12 up, 12 in 2024-03-20T09:21:12.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:12 smithi122 ceph-mon[73495]: osdmap e660: 12 total, 12 up, 12 in 2024-03-20T09:21:12.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:12 smithi115 ceph-mon[72476]: osdmap e660: 12 total, 12 up, 12 in 2024-03-20T09:21:13.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:13 smithi122 ceph-mon[73495]: pgmap v1598: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 498 op/s; 8684/278601 objects degraded (3.117%); 30626/278601 objects misplaced (10.993%); 8.2 MiB/s, 131 objects/s recovering 2024-03-20T09:21:13.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:13 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8684/278601 objects degraded (3.117%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:21:13.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:13 smithi026 ceph-mon[68309]: pgmap v1598: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 498 op/s; 8684/278601 objects degraded (3.117%); 30626/278601 objects misplaced (10.993%); 8.2 MiB/s, 131 objects/s recovering 2024-03-20T09:21:13.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:13 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8684/278601 objects degraded (3.117%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:21:13.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:13 smithi115 ceph-mon[72476]: pgmap v1598: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 498 op/s; 8684/278601 objects degraded (3.117%); 30626/278601 objects misplaced (10.993%); 8.2 MiB/s, 131 objects/s recovering 2024-03-20T09:21:13.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:13 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8684/278601 objects degraded (3.117%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:21:14.207 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:21:14.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:14.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:21:14.311 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:21:14.345 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:14.368 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:21:14.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:14.374 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:21:14.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:14 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:21:14.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:14 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:21:14.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:14 smithi026 ceph-mon[68309]: osdmap e661: 12 total, 12 up, 12 in 2024-03-20T09:21:14.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:14 smithi026 ceph-mon[68309]: pgmap v1600: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 398 op/s; 8683/275856 objects degraded (3.148%); 30005/275856 objects misplaced (10.877%); 7.1 MiB/s, 113 objects/s recovering 2024-03-20T09:21:14.475 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:21:14.501 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:14 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:21:14.502 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:14 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:21:14.502 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:14 smithi122 ceph-mon[73495]: osdmap e661: 12 total, 12 up, 12 in 2024-03-20T09:21:14.502 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:14 smithi122 ceph-mon[73495]: pgmap v1600: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 398 op/s; 8683/275856 objects degraded (3.148%); 30005/275856 objects misplaced (10.877%); 7.1 MiB/s, 113 objects/s recovering 2024-03-20T09:21:14.503 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:14.525 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:21:14.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:14.528 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:21:14.629 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:21:14.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:14 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:21:14.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:14 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:21:14.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:14 smithi115 ceph-mon[72476]: osdmap e661: 12 total, 12 up, 12 in 2024-03-20T09:21:14.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:14 smithi115 ceph-mon[72476]: pgmap v1600: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 398 op/s; 8683/275856 objects degraded (3.148%); 30005/275856 objects misplaced (10.877%); 7.1 MiB/s, 113 objects/s recovering 2024-03-20T09:21:14.673 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:14.704 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:21:14.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:14.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:21:14.808 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:21:14.858 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:14.908 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:21:14.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:14.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:21:14.959 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-20T09:21:15.015 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:21:15.045 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:15.066 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:21:15.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:15.069 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:21:15.169 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:21:15.204 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:15.242 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:21:15.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:15.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:21:15.354 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:21:15.410 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:15.460 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:21:15.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:15.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:21:15.565 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:21:15.640 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:15.693 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:21:15.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:15.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:21:15.797 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:21:15.826 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:15.854 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:21:15.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:15.860 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:21:15.961 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:21:16.029 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:21:16.029 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:21:16.064 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:16.114 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:21:16.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:16.119 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:21:16.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:21:16.221 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:21:16.270 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:16.300 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:21:16.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:16.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:21:16.404 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:21:16.470 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:16.500 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:21:16.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:16.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:21:16.604 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:21:16.678 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:16.839 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:21:16.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:16.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:21:16.949 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:21:16.982 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:17.008 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:21:17.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:17.012 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:21:17.113 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:21:17.163 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:17 smithi122 ceph-mon[73495]: pgmap v1601: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 422 op/s; 8683/272967 objects degraded (3.181%); 30005/272967 objects misplaced (10.992%); 5.3 MiB/s, 85 objects/s recovering 2024-03-20T09:21:17.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:17 smithi026 ceph-mon[68309]: pgmap v1601: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 422 op/s; 8683/272967 objects degraded (3.181%); 30005/272967 objects misplaced (10.992%); 5.3 MiB/s, 85 objects/s recovering 2024-03-20T09:21:17.664 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:17 smithi115 ceph-mon[72476]: pgmap v1601: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 422 op/s; 8683/272967 objects degraded (3.181%); 30005/272967 objects misplaced (10.992%); 5.3 MiB/s, 85 objects/s recovering 2024-03-20T09:21:18.443 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:21:18.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:18 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8683/272967 objects degraded (3.181%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:21:18.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:18.450 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:18 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8683/272967 objects degraded (3.181%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:21:18.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:21:18.552 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:21:18.584 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:18.608 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:21:18.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:18.612 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:21:18.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:18 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8683/272967 objects degraded (3.181%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:21:18.713 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:21:18.744 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:18.769 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:21:18.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:18.772 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:21:18.873 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:21:18.938 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:18.973 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:21:18.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:18.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:21:19.084 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:21:19.112 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:19.141 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:21:19.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:19.147 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:21:19.248 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:21:19.294 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:19 smithi026 ceph-mon[68309]: pgmap v1602: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 587 op/s; 8682/267978 objects degraded (3.240%); 29036/267978 objects misplaced (10.835%); 10 MiB/s, 166 objects/s recovering 2024-03-20T09:21:19.300 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:19.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:19 smithi115 ceph-mon[72476]: pgmap v1602: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 587 op/s; 8682/267978 objects degraded (3.240%); 29036/267978 objects misplaced (10.835%); 10 MiB/s, 166 objects/s recovering 2024-03-20T09:21:19.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:19 smithi122 ceph-mon[73495]: pgmap v1602: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 587 op/s; 8682/267978 objects degraded (3.240%); 29036/267978 objects misplaced (10.835%); 10 MiB/s, 166 objects/s recovering 2024-03-20T09:21:21.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:21 smithi122 ceph-mon[73495]: pgmap v1603: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 487 op/s; 8682/266874 objects degraded (3.253%); 29036/266874 objects misplaced (10.880%); 8.3 MiB/s, 132 objects/s recovering 2024-03-20T09:21:21.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:21 smithi026 ceph-mon[68309]: pgmap v1603: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 487 op/s; 8682/266874 objects degraded (3.253%); 29036/266874 objects misplaced (10.880%); 8.3 MiB/s, 132 objects/s recovering 2024-03-20T09:21:21.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:21 smithi115 ceph-mon[72476]: pgmap v1603: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 487 op/s; 8682/266874 objects degraded (3.253%); 29036/266874 objects misplaced (10.880%); 8.3 MiB/s, 132 objects/s recovering 2024-03-20T09:21:22.184 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.1 to 1 (10000) 2024-03-20T09:21:22.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:22 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/381126837' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-20T09:21:22.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:22 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/381126837' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-20T09:21:22.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:22 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/381126837' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-20T09:21:23.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:23 smithi122 ceph-mon[73495]: pgmap v1604: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 529 op/s; 8682/262698 objects degraded (3.305%); 28690/262698 objects misplaced (10.921%); 8.1 MiB/s, 130 objects/s recovering 2024-03-20T09:21:23.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:23 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/381126837' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-20T09:21:23.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:23 smithi122 ceph-mon[73495]: osdmap e662: 12 total, 12 up, 12 in 2024-03-20T09:21:23.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:23 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8682/262698 objects degraded (3.305%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:21:23.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:23 smithi026 ceph-mon[68309]: pgmap v1604: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 529 op/s; 8682/262698 objects degraded (3.305%); 28690/262698 objects misplaced (10.921%); 8.1 MiB/s, 130 objects/s recovering 2024-03-20T09:21:23.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:23 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/381126837' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-20T09:21:23.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:23 smithi026 ceph-mon[68309]: osdmap e662: 12 total, 12 up, 12 in 2024-03-20T09:21:23.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:23 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8682/262698 objects degraded (3.305%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:21:23.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:23 smithi115 ceph-mon[72476]: pgmap v1604: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 529 op/s; 8682/262698 objects degraded (3.305%); 28690/262698 objects misplaced (10.921%); 8.1 MiB/s, 130 objects/s recovering 2024-03-20T09:21:23.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:23 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/381126837' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-20T09:21:23.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:23 smithi115 ceph-mon[72476]: osdmap e662: 12 total, 12 up, 12 in 2024-03-20T09:21:23.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:23 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8682/262698 objects degraded (3.305%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:21:24.194 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:24 smithi026 ceph-mon[68309]: pgmap v1606: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 537 op/s; 8679/259737 objects degraded (3.341%); 28039/259737 objects misplaced (10.795%); 8.2 MiB/s, 131 objects/s recovering 2024-03-20T09:21:24.662 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:24 smithi115 ceph-mon[72476]: pgmap v1606: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 537 op/s; 8679/259737 objects degraded (3.341%); 28039/259737 objects misplaced (10.795%); 8.2 MiB/s, 131 objects/s recovering 2024-03-20T09:21:24.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:24 smithi122 ceph-mon[73495]: pgmap v1606: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 537 op/s; 8679/259737 objects degraded (3.341%); 28039/259737 objects misplaced (10.795%); 8.2 MiB/s, 131 objects/s recovering 2024-03-20T09:21:26.299 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:21:26.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:26.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:21:26.408 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:21:26.462 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:26.500 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:21:26.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:26.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:21:26.604 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:21:26.665 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:26.795 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:21:26.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:26.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:21:26.907 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:21:26.952 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:27.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:27 smithi026 ceph-mon[68309]: pgmap v1607: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 540 op/s; 8679/256743 objects degraded (3.380%); 28039/256743 objects misplaced (10.921%); 8.2 MiB/s, 131 objects/s recovering 2024-03-20T09:21:27.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:27 smithi122 ceph-mon[73495]: pgmap v1607: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 540 op/s; 8679/256743 objects degraded (3.380%); 28039/256743 objects misplaced (10.921%); 8.2 MiB/s, 131 objects/s recovering 2024-03-20T09:21:27.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:27 smithi115 ceph-mon[72476]: pgmap v1607: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 540 op/s; 8679/256743 objects degraded (3.380%); 28039/256743 objects misplaced (10.921%); 8.2 MiB/s, 131 objects/s recovering 2024-03-20T09:21:28.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:28 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:21:28.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:28 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:21:28.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:28 smithi122 ceph-mon[73495]: osdmap e663: 12 total, 12 up, 12 in 2024-03-20T09:21:28.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:28 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8679/256743 objects degraded (3.380%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:21:28.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:28 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:21:28.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:28 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:21:28.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:28 smithi026 ceph-mon[68309]: osdmap e663: 12 total, 12 up, 12 in 2024-03-20T09:21:28.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:28 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8679/256743 objects degraded (3.380%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:21:28.664 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 2 1 2024-03-20T09:21:28.665 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:28 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:21:28.665 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:28 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:21:28.665 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:28 smithi115 ceph-mon[72476]: osdmap e663: 12 total, 12 up, 12 in 2024-03-20T09:21:28.665 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:28 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8679/256743 objects degraded (3.380%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:21:29.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:29 smithi026 ceph-mon[68309]: pgmap v1609: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 547 op/s; 8692/253737 objects degraded (3.426%); 27492/253737 objects misplaced (10.835%); 9.3 MiB/s, 149 objects/s recovering 2024-03-20T09:21:29.652 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:21:29.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:29.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:21:29.660 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:29 smithi115 ceph-mon[72476]: pgmap v1609: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 547 op/s; 8692/253737 objects degraded (3.426%); 27492/253737 objects misplaced (10.835%); 9.3 MiB/s, 149 objects/s recovering 2024-03-20T09:21:29.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:29 smithi122 ceph-mon[73495]: pgmap v1609: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.2 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 547 op/s; 8692/253737 objects degraded (3.426%); 27492/253737 objects misplaced (10.835%); 9.3 MiB/s, 149 objects/s recovering 2024-03-20T09:21:29.760 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:21:29.823 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:29.875 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:21:29.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:29.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:21:29.987 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:21:30.022 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:30.049 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:21:30.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:30.056 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:21:30.157 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:21:30.192 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:30.218 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:21:30.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:30.224 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:21:30.326 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:21:30.354 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:30 smithi122 ceph-mon[73495]: pgmap v1610: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 494 op/s; 8689/250815 objects degraded (3.464%); 26982/250815 objects misplaced (10.758%); 7.9 MiB/s, 127 objects/s recovering 2024-03-20T09:21:30.357 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:30.377 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:21:30.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:30.382 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:21:30.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:30 smithi026 ceph-mon[68309]: pgmap v1610: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 494 op/s; 8689/250815 objects degraded (3.464%); 26982/250815 objects misplaced (10.758%); 7.9 MiB/s, 127 objects/s recovering 2024-03-20T09:21:30.483 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:21:30.527 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:30.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:30 smithi115 ceph-mon[72476]: pgmap v1610: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 494 op/s; 8689/250815 objects degraded (3.464%); 26982/250815 objects misplaced (10.758%); 7.9 MiB/s, 127 objects/s recovering 2024-03-20T09:21:31.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:31 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1360236627' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:21:31.726 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:31 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1360236627' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:21:31.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:31 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1360236627' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:21:32.595 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:21:32.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:32.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:21:32.726 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:32 smithi026 ceph-mon[68309]: pgmap v1611: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 531 op/s; 8697/246885 objects degraded (3.523%); 26580/246885 objects misplaced (10.766%); 8.9 MiB/s, 142 objects/s recovering 2024-03-20T09:21:32.726 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:32 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8689/250815 objects degraded (3.464%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:21:32.800 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:21:32.858 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:32.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:32 smithi115 ceph-mon[72476]: pgmap v1611: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 531 op/s; 8697/246885 objects degraded (3.523%); 26580/246885 objects misplaced (10.766%); 8.9 MiB/s, 142 objects/s recovering 2024-03-20T09:21:32.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:32 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8689/250815 objects degraded (3.464%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:21:32.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:32 smithi122 ceph-mon[73495]: pgmap v1611: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 531 op/s; 8697/246885 objects degraded (3.523%); 26580/246885 objects misplaced (10.766%); 8.9 MiB/s, 142 objects/s recovering 2024-03-20T09:21:32.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:32 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8689/250815 objects degraded (3.464%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:21:33.006 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:21:33.006 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:21:33.006 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:21:33.006 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:21:33.006 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:21:33.007 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:21:33.007 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:21:33.007 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:21:33.007 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:21:33.007 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:21:33.007 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:21:33.007 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:21:33.007 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:21:33.007 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:21:33.007 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:21:33.007 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:21:33.008 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:21:33.008 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:21:33.008 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:21:33.008 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:21:33.008 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:21:33.008 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:21:33.008 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:21:33.008 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:21:35.208 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:21:35.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:35 smithi122 ceph-mon[73495]: pgmap v1612: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 534 op/s; 8724/243702 objects degraded (3.580%); 25910/243702 objects misplaced (10.632%); 8.8 MiB/s, 141 objects/s recovering 2024-03-20T09:21:35.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:35 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3391951479' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:21:35.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:35 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:21:35.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:35 smithi026 ceph-mon[68309]: pgmap v1612: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 534 op/s; 8724/243702 objects degraded (3.580%); 25910/243702 objects misplaced (10.632%); 8.8 MiB/s, 141 objects/s recovering 2024-03-20T09:21:35.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:35 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3391951479' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:21:35.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:35 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:21:35.664 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:35 smithi115 ceph-mon[72476]: pgmap v1612: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 5.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 534 op/s; 8724/243702 objects degraded (3.580%); 25910/243702 objects misplaced (10.632%); 8.8 MiB/s, 141 objects/s recovering 2024-03-20T09:21:35.664 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:35 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3391951479' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:21:35.664 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:35 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:21:36.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:36 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:21:36.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:36 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:21:36.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:36 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:21:36.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:36 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:21:36.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:36 smithi122 ceph-mon[73495]: osdmap e664: 12 total, 12 up, 12 in 2024-03-20T09:21:36.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:36 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:21:36.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:36 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:21:36.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:36 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:21:36.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:36 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:21:36.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:36 smithi026 ceph-mon[68309]: osdmap e664: 12 total, 12 up, 12 in 2024-03-20T09:21:36.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:36 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:21:36.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:36 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:21:36.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:36 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:21:36.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:36 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:21:36.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:36 smithi115 ceph-mon[72476]: osdmap e664: 12 total, 12 up, 12 in 2024-03-20T09:21:37.346 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:21:37.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:37.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:21:37.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:37 smithi122 ceph-mon[73495]: pgmap v1614: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 4.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 525 op/s; 8724/240630 objects degraded (3.625%); 25910/240630 objects misplaced (10.768%); 7.1 MiB/s, 115 objects/s recovering 2024-03-20T09:21:37.453 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:21:37.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:37 smithi026 ceph-mon[68309]: pgmap v1614: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 4.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 525 op/s; 8724/240630 objects degraded (3.625%); 25910/240630 objects misplaced (10.768%); 7.1 MiB/s, 115 objects/s recovering 2024-03-20T09:21:37.512 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:21:37.531 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:37.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:37 smithi115 ceph-mon[72476]: pgmap v1614: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 4.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 525 op/s; 8724/240630 objects degraded (3.625%); 25910/240630 objects misplaced (10.768%); 7.1 MiB/s, 115 objects/s recovering 2024-03-20T09:21:38.194 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:38 smithi026 ceph-mon[68309]: 1.14 scrub starts 2024-03-20T09:21:38.194 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:38 smithi026 ceph-mon[68309]: 1.14 scrub ok 2024-03-20T09:21:38.194 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:38 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8724/240630 objects degraded (3.625%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:21:38.194 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:38 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:21:38.194 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:38 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:21:38.194 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:38 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:21:38.194 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:38 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:21:38.194 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:38 smithi026 ceph-mon[68309]: pgmap v1615: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 543 op/s; 8741/237408 objects degraded (3.682%); 25364/237408 objects misplaced (10.684%); 8.7 MiB/s, 140 objects/s recovering 2024-03-20T09:21:38.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:38 smithi122 ceph-mon[73495]: 1.14 scrub starts 2024-03-20T09:21:38.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:38 smithi122 ceph-mon[73495]: 1.14 scrub ok 2024-03-20T09:21:38.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:38 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8724/240630 objects degraded (3.625%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:21:38.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:38 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:21:38.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:38 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:21:38.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:38 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:21:38.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:38 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:21:38.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:38 smithi122 ceph-mon[73495]: pgmap v1615: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 543 op/s; 8741/237408 objects degraded (3.682%); 25364/237408 objects misplaced (10.684%); 8.7 MiB/s, 140 objects/s recovering 2024-03-20T09:21:38.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:38 smithi115 ceph-mon[72476]: 1.14 scrub starts 2024-03-20T09:21:38.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:38 smithi115 ceph-mon[72476]: 1.14 scrub ok 2024-03-20T09:21:38.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:38 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8724/240630 objects degraded (3.625%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:21:38.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:38 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:21:38.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:38 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:21:38.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:38 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:21:38.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:38 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:21:38.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:38 smithi115 ceph-mon[72476]: pgmap v1615: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 543 op/s; 8741/237408 objects degraded (3.682%); 25364/237408 objects misplaced (10.684%); 8.7 MiB/s, 140 objects/s recovering 2024-03-20T09:21:39.505 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:21:39.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:39 smithi115 ceph-mon[72476]: osdmap e665: 12 total, 12 up, 12 in 2024-03-20T09:21:39.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:39 smithi122 ceph-mon[73495]: osdmap e665: 12 total, 12 up, 12 in 2024-03-20T09:21:39.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:39 smithi026 ceph-mon[68309]: osdmap e665: 12 total, 12 up, 12 in 2024-03-20T09:21:40.662 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:40 smithi115 ceph-mon[72476]: 5.a scrub starts 2024-03-20T09:21:40.662 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:40 smithi115 ceph-mon[72476]: pgmap v1617: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 526 op/s; 8736/234246 objects degraded (3.729%); 24904/234246 objects misplaced (10.632%); 7.7 MiB/s, 125 objects/s recovering 2024-03-20T09:21:40.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:40 smithi122 ceph-mon[73495]: 5.a scrub starts 2024-03-20T09:21:40.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:40 smithi122 ceph-mon[73495]: pgmap v1617: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 526 op/s; 8736/234246 objects degraded (3.729%); 24904/234246 objects misplaced (10.632%); 7.7 MiB/s, 125 objects/s recovering 2024-03-20T09:21:40.726 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:40 smithi026 ceph-mon[68309]: 5.a scrub starts 2024-03-20T09:21:40.726 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:40 smithi026 ceph-mon[68309]: pgmap v1617: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 526 op/s; 8736/234246 objects degraded (3.729%); 24904/234246 objects misplaced (10.632%); 7.7 MiB/s, 125 objects/s recovering 2024-03-20T09:21:41.662 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:41 smithi115 ceph-mon[72476]: 1.9 scrub starts 2024-03-20T09:21:41.662 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:41 smithi115 ceph-mon[72476]: 1.9 scrub ok 2024-03-20T09:21:41.662 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:41 smithi115 ceph-mon[72476]: 5.9 scrub starts 2024-03-20T09:21:41.662 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:41 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:21:41.662 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:41 smithi115 ceph-mon[72476]: 5.a scrub ok 2024-03-20T09:21:41.663 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:41 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:21:41.663 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:41 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:21:41.663 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:41 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:21:41.663 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:41 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:21:41.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:41 smithi122 ceph-mon[73495]: 1.9 scrub starts 2024-03-20T09:21:41.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:41 smithi122 ceph-mon[73495]: 1.9 scrub ok 2024-03-20T09:21:41.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:41 smithi122 ceph-mon[73495]: 5.9 scrub starts 2024-03-20T09:21:41.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:41 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:21:41.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:41 smithi122 ceph-mon[73495]: 5.a scrub ok 2024-03-20T09:21:41.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:41 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:21:41.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:41 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:21:41.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:41 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:21:41.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:41 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:21:41.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:41 smithi026 ceph-mon[68309]: 1.9 scrub starts 2024-03-20T09:21:41.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:41 smithi026 ceph-mon[68309]: 1.9 scrub ok 2024-03-20T09:21:41.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:41 smithi026 ceph-mon[68309]: 5.9 scrub starts 2024-03-20T09:21:41.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:41 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:21:41.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:41 smithi026 ceph-mon[68309]: 5.a scrub ok 2024-03-20T09:21:41.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:41 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:21:41.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:41 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:21:41.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:41 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:21:41.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:41 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:21:41.846 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:21:41.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:41.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:21:41.959 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:21:41.989 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:42.014 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:21:42.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:42.021 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:21:42.122 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:21:42.175 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:42.217 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:21:42.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:42.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:21:42.320 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:21:42.377 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:42.401 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:42 smithi122 ceph-mon[73495]: 5.9 scrub ok 2024-03-20T09:21:42.401 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:42 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:21:42.402 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:42 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:21:42.402 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:42 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:21:42.402 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:42 smithi122 ceph-mon[73495]: pgmap v1618: 47 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 558 op/s; 8754/230280 objects degraded (3.801%); 24511/230280 objects misplaced (10.644%); 8.2 MiB/s, 132 objects/s recovering 2024-03-20T09:21:42.402 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:42 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8736/234246 objects degraded (3.729%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:21:42.477 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:42 smithi026 ceph-mon[68309]: 5.9 scrub ok 2024-03-20T09:21:42.477 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:42 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:21:42.477 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:42 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:21:42.477 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:42 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:21:42.477 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:42 smithi026 ceph-mon[68309]: pgmap v1618: 47 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 558 op/s; 8754/230280 objects degraded (3.801%); 24511/230280 objects misplaced (10.644%); 8.2 MiB/s, 132 objects/s recovering 2024-03-20T09:21:42.477 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:42 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8736/234246 objects degraded (3.729%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:21:42.504 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:21:42.504 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:21:42.661 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:42 smithi115 ceph-mon[72476]: 5.9 scrub ok 2024-03-20T09:21:42.661 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:21:42.661 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:42 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:21:42.661 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:42 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:21:42.661 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:42 smithi115 ceph-mon[72476]: pgmap v1618: 47 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 558 op/s; 8754/230280 objects degraded (3.801%); 24511/230280 objects misplaced (10.644%); 8.2 MiB/s, 132 objects/s recovering 2024-03-20T09:21:42.662 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:42 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8736/234246 objects degraded (3.729%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:21:43.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:43 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:21:43.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:43 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:21:43.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:43 smithi122 ceph-mon[73495]: osdmap e666: 12 total, 12 up, 12 in 2024-03-20T09:21:43.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:43 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:21:43.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:43 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:21:43.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:43 smithi026 ceph-mon[68309]: osdmap e666: 12 total, 12 up, 12 in 2024-03-20T09:21:43.910 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:43 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:21:43.910 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:43 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:21:43.910 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:43 smithi115 ceph-mon[72476]: osdmap e666: 12 total, 12 up, 12 in 2024-03-20T09:21:44.298 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:21:44.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:44 smithi026 ceph-mon[68309]: pgmap v1620: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 555 op/s; 8772/227295 objects degraded (3.859%); 23860/227295 objects misplaced (10.497%); 11 MiB/s, 171 objects/s recovering 2024-03-20T09:21:44.908 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:44 smithi115 ceph-mon[72476]: pgmap v1620: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 555 op/s; 8772/227295 objects degraded (3.859%); 23860/227295 objects misplaced (10.497%); 11 MiB/s, 171 objects/s recovering 2024-03-20T09:21:44.947 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:44 smithi122 ceph-mon[73495]: pgmap v1620: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 555 op/s; 8772/227295 objects degraded (3.859%); 23860/227295 objects misplaced (10.497%); 11 MiB/s, 171 objects/s recovering 2024-03-20T09:21:45.994 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-20T09:21:46.167 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:21:46.167 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:21:46.167 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:21:46.167 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:21:46.168 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:21:46.168 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:21:46.168 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:21:46.454 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:21:46.454 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:21:46.454 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:21:46.454 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:21:46.454 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:21:46.454 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:21:46.454 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:21:46.552 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:21:46.574 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:21:46.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:46.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:21:46.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:21:46.660 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:21:46.660 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:21:46.660 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:21:46.660 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:21:46.660 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:21:46.660 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:21:46.687 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:21:46.738 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:47.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:47 smithi122 ceph-mon[73495]: 5.e scrub starts 2024-03-20T09:21:47.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:47 smithi026 ceph-mon[68309]: 5.e scrub starts 2024-03-20T09:21:47.660 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:47 smithi115 ceph-mon[72476]: 5.e scrub starts 2024-03-20T09:21:47.912 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:21:47.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:47.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:21:47.944 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:21:48.018 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:21:48.068 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:48.116 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:21:48.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:48.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:21:48.181 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:48 smithi026 ceph-mon[68309]: 5.2 scrub starts 2024-03-20T09:21:48.181 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:48 smithi026 ceph-mon[68309]: 5.e scrub ok 2024-03-20T09:21:48.181 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:48 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8772/227295 objects degraded (3.859%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:21:48.219 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:21:48.262 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:48.299 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:21:48.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:48.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:21:48.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:48 smithi115 ceph-mon[72476]: 5.2 scrub starts 2024-03-20T09:21:48.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:48 smithi115 ceph-mon[72476]: 5.e scrub ok 2024-03-20T09:21:48.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:48 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8772/227295 objects degraded (3.859%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:21:48.414 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:21:48.440 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:48 smithi122 ceph-mon[73495]: 5.2 scrub starts 2024-03-20T09:21:48.441 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:48 smithi122 ceph-mon[73495]: 5.e scrub ok 2024-03-20T09:21:48.441 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:48 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8772/227295 objects degraded (3.859%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:21:48.441 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:48.468 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:21:48.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:48.472 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:21:48.573 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:21:48.634 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:48.906 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:21:48.907 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:21:48.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:48.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:21:49.014 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:21:49.053 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:49.082 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:21:49.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:49.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:21:49.187 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:21:49.191 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:49 smithi026 ceph-mon[68309]: 1.7 scrub starts 2024-03-20T09:21:49.191 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:49 smithi026 ceph-mon[68309]: 1.7 scrub ok 2024-03-20T09:21:49.192 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:49 smithi026 ceph-mon[68309]: 5.2 scrub ok 2024-03-20T09:21:49.192 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:49 smithi026 ceph-mon[68309]: pgmap v1622: 47 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 495 op/s; 8859/220842 objects degraded (4.011%); 23270/220842 objects misplaced (10.537%); 7.6 MiB/s, 121 objects/s recovering 2024-03-20T09:21:49.192 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:49 smithi026 ceph-mon[68309]: 1.1b scrub starts 2024-03-20T09:21:49.192 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:49 smithi026 ceph-mon[68309]: 1.1b scrub ok 2024-03-20T09:21:49.220 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:49.250 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:21:49.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:49.257 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:21:49.358 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:21:49.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:49 smithi115 ceph-mon[72476]: 1.7 scrub starts 2024-03-20T09:21:49.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:49 smithi115 ceph-mon[72476]: 1.7 scrub ok 2024-03-20T09:21:49.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:49 smithi115 ceph-mon[72476]: 5.2 scrub ok 2024-03-20T09:21:49.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:49 smithi115 ceph-mon[72476]: pgmap v1622: 47 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 495 op/s; 8859/220842 objects degraded (4.011%); 23270/220842 objects misplaced (10.537%); 7.6 MiB/s, 121 objects/s recovering 2024-03-20T09:21:49.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:49 smithi115 ceph-mon[72476]: 1.1b scrub starts 2024-03-20T09:21:49.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:49 smithi115 ceph-mon[72476]: 1.1b scrub ok 2024-03-20T09:21:49.406 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:49.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:49 smithi122 ceph-mon[73495]: 1.7 scrub starts 2024-03-20T09:21:49.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:49 smithi122 ceph-mon[73495]: 1.7 scrub ok 2024-03-20T09:21:49.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:49 smithi122 ceph-mon[73495]: 5.2 scrub ok 2024-03-20T09:21:49.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:49 smithi122 ceph-mon[73495]: pgmap v1622: 47 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 495 op/s; 8859/220842 objects degraded (4.011%); 23270/220842 objects misplaced (10.537%); 7.6 MiB/s, 121 objects/s recovering 2024-03-20T09:21:49.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:49 smithi122 ceph-mon[73495]: 1.1b scrub starts 2024-03-20T09:21:49.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:49 smithi122 ceph-mon[73495]: 1.1b scrub ok 2024-03-20T09:21:49.457 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:21:49.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:49.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:21:49.566 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:21:49.640 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:49.672 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:21:49.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:49.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:21:49.782 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:21:49.826 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:49.884 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:21:49.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:49.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:21:49.995 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:21:50.029 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:50.054 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:21:50.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:50.059 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:21:50.160 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:21:50.192 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:50.222 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:21:50.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:50.226 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:21:50.229 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.2 to 1 (10000) 2024-03-20T09:21:50.266 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:50 smithi026 ceph-mon[68309]: 1.f scrub starts 2024-03-20T09:21:50.266 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:50 smithi026 ceph-mon[68309]: 1.f scrub ok 2024-03-20T09:21:50.266 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:50 smithi026 ceph-mon[68309]: 5.6 scrub starts 2024-03-20T09:21:50.266 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:50 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:21:50.266 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:50 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:21:50.266 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:50 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3698050809' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:21:50.266 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:50 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:21:50.326 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:21:50.373 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:50 smithi122 ceph-mon[73495]: 1.f scrub starts 2024-03-20T09:21:50.374 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:50 smithi122 ceph-mon[73495]: 1.f scrub ok 2024-03-20T09:21:50.374 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:50 smithi122 ceph-mon[73495]: 5.6 scrub starts 2024-03-20T09:21:50.374 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:50 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:21:50.374 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:50 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:21:50.374 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:50 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3698050809' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:21:50.374 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:50 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:21:50.376 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:50.387 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:21:50.404 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:21:50.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:50.410 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:21:50.430 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:21:50.468 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:21:50.511 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:21:50.512 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:21:50.542 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:50.568 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:21:50.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:50.575 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:21:50.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:50 smithi115 ceph-mon[72476]: 1.f scrub starts 2024-03-20T09:21:50.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:50 smithi115 ceph-mon[72476]: 1.f scrub ok 2024-03-20T09:21:50.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:50 smithi115 ceph-mon[72476]: 5.6 scrub starts 2024-03-20T09:21:50.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:50 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:21:50.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:50 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:21:50.660 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:50 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3698050809' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:21:50.660 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:50 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:21:50.675 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:21:50.727 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:50.784 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:21:50.826 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:21:50.878 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:21:50.918 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:21:50.962 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:21:50.998 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:21:51.044 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:21:51.089 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:21:51.132 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:21:51.132 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:21:51.132 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:21:51.133 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:21:51.133 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:21:51.133 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:21:51.133 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:21:51.133 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:21:51.133 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:21:51.133 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:21:51.133 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:21:51.133 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:21:51.350 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:51 smithi026 ceph-mon[68309]: 5.6 scrub ok 2024-03-20T09:21:51.350 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:51 smithi026 ceph-mon[68309]: pgmap v1623: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 552 op/s; 8857/217680 objects degraded (4.069%); 22754/217680 objects misplaced (10.453%); 8.4 MiB/s, 136 objects/s recovering 2024-03-20T09:21:51.350 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:51 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-20T09:21:51.351 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:51 smithi026 ceph-mon[68309]: osdmap e667: 12 total, 12 up, 12 in 2024-03-20T09:21:51.351 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:51 smithi026 ceph-mon[68309]: 1.17 scrub starts 2024-03-20T09:21:51.351 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:51 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4020960354' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:21:51.351 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:51 smithi026 ceph-mon[68309]: 1.17 scrub ok 2024-03-20T09:21:51.660 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:51 smithi115 ceph-mon[72476]: 5.6 scrub ok 2024-03-20T09:21:51.660 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:51 smithi115 ceph-mon[72476]: pgmap v1623: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 552 op/s; 8857/217680 objects degraded (4.069%); 22754/217680 objects misplaced (10.453%); 8.4 MiB/s, 136 objects/s recovering 2024-03-20T09:21:51.660 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:51 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-20T09:21:51.660 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:51 smithi115 ceph-mon[72476]: osdmap e667: 12 total, 12 up, 12 in 2024-03-20T09:21:51.660 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:51 smithi115 ceph-mon[72476]: 1.17 scrub starts 2024-03-20T09:21:51.660 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:51 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4020960354' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:21:51.661 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:51 smithi115 ceph-mon[72476]: 1.17 scrub ok 2024-03-20T09:21:51.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:51 smithi122 ceph-mon[73495]: 5.6 scrub ok 2024-03-20T09:21:51.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:51 smithi122 ceph-mon[73495]: pgmap v1623: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 40 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 552 op/s; 8857/217680 objects degraded (4.069%); 22754/217680 objects misplaced (10.453%); 8.4 MiB/s, 136 objects/s recovering 2024-03-20T09:21:51.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:51 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-20T09:21:51.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:51 smithi122 ceph-mon[73495]: osdmap e667: 12 total, 12 up, 12 in 2024-03-20T09:21:51.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:51 smithi122 ceph-mon[73495]: 1.17 scrub starts 2024-03-20T09:21:51.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:51 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4020960354' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:21:51.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:51 smithi122 ceph-mon[73495]: 1.17 scrub ok 2024-03-20T09:21:52.233 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:21:52.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:52.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:21:52.346 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:21:52.407 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:52.437 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:21:52.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:52.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:21:52.544 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:21:52.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:52 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:21:52.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:52 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:21:52.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:52 smithi115 ceph-mon[72476]: pgmap v1625: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 574 op/s; 8879/213537 objects degraded (4.158%); 22333/213537 objects misplaced (10.459%); 8.9 MiB/s, 142 objects/s recovering 2024-03-20T09:21:52.691 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:52 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:21:52.691 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:52 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:21:52.691 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:52 smithi026 ceph-mon[68309]: pgmap v1625: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 574 op/s; 8879/213537 objects degraded (4.158%); 22333/213537 objects misplaced (10.459%); 8.9 MiB/s, 142 objects/s recovering 2024-03-20T09:21:52.692 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:52.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:52 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:21:52.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:52 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:21:52.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:52 smithi122 ceph-mon[73495]: pgmap v1625: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 574 op/s; 8879/213537 objects degraded (4.158%); 22333/213537 objects misplaced (10.459%); 8.9 MiB/s, 142 objects/s recovering 2024-03-20T09:21:53.561 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:21:53.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:53.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:21:53.568 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:53 smithi026 ceph-mon[68309]: osdmap e668: 12 total, 12 up, 12 in 2024-03-20T09:21:53.568 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:53 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8879/213537 objects degraded (4.158%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:21:53.660 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:53 smithi115 ceph-mon[72476]: osdmap e668: 12 total, 12 up, 12 in 2024-03-20T09:21:53.660 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:53 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8879/213537 objects degraded (4.158%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:21:53.669 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:21:53.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:53 smithi122 ceph-mon[73495]: osdmap e668: 12 total, 12 up, 12 in 2024-03-20T09:21:53.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:53 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8879/213537 objects degraded (4.158%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:21:53.700 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:53.724 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:21:53.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:53.727 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:21:53.828 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:21:53.881 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:54.661 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:54 smithi115 ceph-mon[72476]: 1.1c scrub starts 2024-03-20T09:21:54.662 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:54 smithi115 ceph-mon[72476]: 1.1c scrub ok 2024-03-20T09:21:54.662 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:54 smithi115 ceph-mon[72476]: pgmap v1627: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 571 op/s; 8903/210360 objects degraded (4.232%); 21636/210360 objects misplaced (10.285%); 11 MiB/s, 174 objects/s recovering 2024-03-20T09:21:54.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:54 smithi122 ceph-mon[73495]: 1.1c scrub starts 2024-03-20T09:21:54.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:54 smithi122 ceph-mon[73495]: 1.1c scrub ok 2024-03-20T09:21:54.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:54 smithi122 ceph-mon[73495]: pgmap v1627: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 571 op/s; 8903/210360 objects degraded (4.232%); 21636/210360 objects misplaced (10.285%); 11 MiB/s, 174 objects/s recovering 2024-03-20T09:21:54.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:54 smithi026 ceph-mon[68309]: 1.1c scrub starts 2024-03-20T09:21:54.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:54 smithi026 ceph-mon[68309]: 1.1c scrub ok 2024-03-20T09:21:54.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:54 smithi026 ceph-mon[68309]: pgmap v1627: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 41 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 571 op/s; 8903/210360 objects degraded (4.232%); 21636/210360 objects misplaced (10.285%); 11 MiB/s, 174 objects/s recovering 2024-03-20T09:21:55.392 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:55 smithi026 ceph-mon[68309]: 1.a scrub starts 2024-03-20T09:21:55.392 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:55 smithi026 ceph-mon[68309]: 1.a scrub ok 2024-03-20T09:21:55.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:55 smithi115 ceph-mon[72476]: 1.a scrub starts 2024-03-20T09:21:55.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:55 smithi115 ceph-mon[72476]: 1.a scrub ok 2024-03-20T09:21:55.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:55 smithi122 ceph-mon[73495]: 1.a scrub starts 2024-03-20T09:21:55.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:55 smithi122 ceph-mon[73495]: 1.a scrub ok 2024-03-20T09:21:56.050 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:21:56.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:21:56.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:21:56.160 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:21:56.211 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:21:56.404 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:56 smithi026 ceph-mon[68309]: pgmap v1628: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 573 op/s; 8903/207081 objects degraded (4.299%); 21636/207081 objects misplaced (10.448%); 7.0 MiB/s, 111 objects/s recovering 2024-03-20T09:21:56.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:56 smithi115 ceph-mon[72476]: pgmap v1628: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 573 op/s; 8903/207081 objects degraded (4.299%); 21636/207081 objects misplaced (10.448%); 7.0 MiB/s, 111 objects/s recovering 2024-03-20T09:21:56.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:56 smithi122 ceph-mon[73495]: pgmap v1628: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 573 op/s; 8903/207081 objects degraded (4.299%); 21636/207081 objects misplaced (10.448%); 7.0 MiB/s, 111 objects/s recovering 2024-03-20T09:21:57.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:57 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8903/207081 objects degraded (4.299%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:21:57.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:57 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8903/207081 objects degraded (4.299%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:21:57.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:57 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8903/207081 objects degraded (4.299%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:21:58.416 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:58 smithi026 ceph-mon[68309]: 1.1 scrub starts 2024-03-20T09:21:58.416 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:58 smithi026 ceph-mon[68309]: 1.1 scrub ok 2024-03-20T09:21:58.416 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:21:58 smithi026 ceph-mon[68309]: pgmap v1629: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 573 op/s; 8949/203919 objects degraded (4.389%); 21006/203919 objects misplaced (10.301%); 10 MiB/s, 162 objects/s recovering 2024-03-20T09:21:58.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:58 smithi122 ceph-mon[73495]: 1.1 scrub starts 2024-03-20T09:21:58.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:58 smithi122 ceph-mon[73495]: 1.1 scrub ok 2024-03-20T09:21:58.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:21:58 smithi122 ceph-mon[73495]: pgmap v1629: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 573 op/s; 8949/203919 objects degraded (4.389%); 21006/203919 objects misplaced (10.301%); 10 MiB/s, 162 objects/s recovering 2024-03-20T09:21:58.908 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:58 smithi115 ceph-mon[72476]: 1.1 scrub starts 2024-03-20T09:21:58.909 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:58 smithi115 ceph-mon[72476]: 1.1 scrub ok 2024-03-20T09:21:58.909 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:21:58 smithi115 ceph-mon[72476]: pgmap v1629: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 573 op/s; 8949/203919 objects degraded (4.389%); 21006/203919 objects misplaced (10.301%); 10 MiB/s, 162 objects/s recovering 2024-03-20T09:22:00.291 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:22:00.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:00.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:22:00.339 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 5 1 2024-03-20T09:22:00.403 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:22:00.439 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:00.465 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:22:00.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:00.469 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:22:00.570 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:22:00.609 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:00.648 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:22:00.653 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:00.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:22:00.746 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:22:00.746 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:22:00.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:22:00.746 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.25966.0:191738 5.c e668/630 5:3453ca91:::benchmark_data_smithi122_157894_object61751:head v 668'12062, mlcod=620'3884)", 2024-03-20T09:22:00.747 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:21:55.093255+0000", 2024-03-20T09:22:00.747 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.011117531, 2024-03-20T09:22:00.747 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011148936, 2024-03-20T09:22:00.747 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:22:00.747 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:22:00.747 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:22:00.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:22:00.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:22:00.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:55.093255+0000", 2024-03-20T09:22:00.747 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:22:00.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:22:00.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:22:00.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:22:00.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:55.093255+0000", 2024-03-20T09:22:00.748 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:22:00.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:22:00.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:22:00.748 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:22:00.748 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:55.093256+0000", 2024-03-20T09:22:00.748 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4900000000000004e-07 2024-03-20T09:22:00.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:22:00.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:22:00.748 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:22:00.748 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:55.093278+0000", 2024-03-20T09:22:00.748 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1219000000000001e-05 2024-03-20T09:22:00.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:22:00.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:22:00.749 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:22:00.749 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:55.093291+0000", 2024-03-20T09:22:00.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3484e-05 2024-03-20T09:22:00.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:22:00.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:22:00.749 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:22:00.749 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:55.093299+0000", 2024-03-20T09:22:00.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1880000000000002e-06 2024-03-20T09:22:00.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:22:00.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:22:00.749 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:22:00.749 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:55.093368+0000", 2024-03-20T09:22:00.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8267000000000002e-05 2024-03-20T09:22:00.750 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:22:00.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:22:00.750 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:22:00.750 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:55.093395+0000", 2024-03-20T09:22:00.750 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7274e-05 2024-03-20T09:22:00.750 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:22:00.750 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:22:00.750 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:22:00.750 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:22:00.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:22:00.750 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:191890 5.1 5:8591f054:::benchmark_data_smithi122_157894_object61903:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e668)", 2024-03-20T09:22:00.750 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:21:55.101817+0000", 2024-03-20T09:22:00.750 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0025560019999999999, 2024-03-20T09:22:00.750 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026594370000000002, 2024-03-20T09:22:00.751 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:22:00.751 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:22:00.751 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:22:00.751 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:22:00.751 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:22:00.751 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 191890 2024-03-20T09:22:00.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:22:00.751 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:22:00.751 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:22:00.751 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:22:00.751 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:55.101817+0000", 2024-03-20T09:22:00.751 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:22:00.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:22:00.751 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:22:00.752 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:22:00.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:55.101817+0000", 2024-03-20T09:22:00.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:22:00.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:22:00.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:22:00.752 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:22:00.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:55.101818+0000", 2024-03-20T09:22:00.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3990000000000001e-06 2024-03-20T09:22:00.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:22:00.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:22:00.752 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:22:00.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:55.101825+0000", 2024-03-20T09:22:00.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1500000000000004e-06 2024-03-20T09:22:00.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:22:00.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:22:00.753 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:22:00.753 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:55.101841+0000", 2024-03-20T09:22:00.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6365e-05 2024-03-20T09:22:00.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:22:00.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:22:00.753 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:22:00.753 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:55.101850+0000", 2024-03-20T09:22:00.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9670000000000007e-06 2024-03-20T09:22:00.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:22:00.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:22:00.753 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:22:00.753 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:55.101910+0000", 2024-03-20T09:22:00.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9824999999999997e-05 2024-03-20T09:22:00.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:22:00.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:22:00.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:22:00.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:55.102024+0000", 2024-03-20T09:22:00.754 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011409900000000001 2024-03-20T09:22:00.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:22:00.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:22:00.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4,7", 2024-03-20T09:22:00.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:55.102177+0000", 2024-03-20T09:22:00.754 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015296799999999999 2024-03-20T09:22:00.754 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:22:00.754 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:22:00.754 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:22:00.754 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:22:00.754 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:22:00.755 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 2 2024-03-20T09:22:00.755 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:22:00.755 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:22:00.786 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:00.810 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:22:00.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:00.816 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:22:00.917 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:22:00.974 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:01.021 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:22:01.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:01.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:22:01.136 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:22:01.187 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:01.235 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:22:01.237 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:01 smithi115 ceph-mon[72476]: pgmap v1630: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 570 op/s; 8945/200760 objects degraded (4.456%); 20462/200760 objects misplaced (10.192%); 9.4 MiB/s, 150 objects/s recovering 2024-03-20T09:22:01.238 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:01 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:22:01.238 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:01 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:22:01.238 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:01 smithi115 ceph-mon[72476]: osdmap e669: 12 total, 12 up, 12 in 2024-03-20T09:22:01.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:01.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:22:01.340 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:22:01.382 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:01 smithi026 ceph-mon[68309]: pgmap v1630: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 570 op/s; 8945/200760 objects degraded (4.456%); 20462/200760 objects misplaced (10.192%); 9.4 MiB/s, 150 objects/s recovering 2024-03-20T09:22:01.382 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:01 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:22:01.383 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:01 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:22:01.383 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:01 smithi026 ceph-mon[68309]: osdmap e669: 12 total, 12 up, 12 in 2024-03-20T09:22:01.386 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:01.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:01 smithi122 ceph-mon[73495]: pgmap v1630: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 570 op/s; 8945/200760 objects degraded (4.456%); 20462/200760 objects misplaced (10.192%); 9.4 MiB/s, 150 objects/s recovering 2024-03-20T09:22:01.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:01 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:22:01.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:01 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:22:01.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:01 smithi122 ceph-mon[73495]: osdmap e669: 12 total, 12 up, 12 in 2024-03-20T09:22:01.465 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:22:01.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:01.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:22:01.527 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-20T09:22:01.573 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:22:01.666 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:01.750 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:22:01.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:01.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:22:01.860 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:22:01.925 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:01.977 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:22:01.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:01.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:22:02.006 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:22:02.084 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:22:02.197 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:22:02.219 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:02.269 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:22:02.407 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:22:02.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:02.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:22:02.478 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:02 smithi026 ceph-mon[68309]: pgmap v1632: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 4.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 577 op/s; 8948/196440 objects degraded (4.555%); 20029/196440 objects misplaced (10.196%); 9.6 MiB/s, 153 objects/s recovering 2024-03-20T09:22:02.512 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:22:02.543 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:02 smithi122 ceph-mon[73495]: pgmap v1632: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 4.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 577 op/s; 8948/196440 objects degraded (4.555%); 20029/196440 objects misplaced (10.196%); 9.6 MiB/s, 153 objects/s recovering 2024-03-20T09:22:02.546 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:02.576 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:22:02.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:02.579 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:22:02.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:02 smithi115 ceph-mon[72476]: pgmap v1632: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 4.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 577 op/s; 8948/196440 objects degraded (4.555%); 20029/196440 objects misplaced (10.196%); 9.6 MiB/s, 153 objects/s recovering 2024-03-20T09:22:02.680 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:22:02.765 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:02.806 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:22:02.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:02.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:22:02.913 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:22:02.946 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:02.971 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:22:02.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:02.977 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:22:03.078 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:22:03.107 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:03.131 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:22:03.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:03.135 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:22:03.235 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:22:03.300 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:03.382 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:22:03.663 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:03 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8948/196440 objects degraded (4.555%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:22:03.663 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:03 smithi115 ceph-mon[72476]: osdmap e670: 12 total, 12 up, 12 in 2024-03-20T09:22:03.663 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:03 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2314885030' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:22:03.664 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:03 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:22:03.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:03 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8948/196440 objects degraded (4.555%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:22:03.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:03 smithi122 ceph-mon[73495]: osdmap e670: 12 total, 12 up, 12 in 2024-03-20T09:22:03.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:03 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2314885030' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:22:03.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:03 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:22:03.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:03 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8948/196440 objects degraded (4.555%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:22:03.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:03 smithi026 ceph-mon[68309]: osdmap e670: 12 total, 12 up, 12 in 2024-03-20T09:22:03.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:03 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2314885030' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:22:03.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:03 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:22:04.662 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:04 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:22:04.662 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:04 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:22:04.663 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:04 smithi115 ceph-mon[72476]: osdmap e671: 12 total, 12 up, 12 in 2024-03-20T09:22:04.663 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:04 smithi115 ceph-mon[72476]: pgmap v1635: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 593 op/s; 8947/193224 objects degraded (4.630%); 19309/193224 objects misplaced (9.993%); 10 MiB/s, 161 objects/s recovering 2024-03-20T09:22:04.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:04 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:22:04.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:04 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:22:04.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:04 smithi122 ceph-mon[73495]: osdmap e671: 12 total, 12 up, 12 in 2024-03-20T09:22:04.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:04 smithi122 ceph-mon[73495]: pgmap v1635: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 593 op/s; 8947/193224 objects degraded (4.630%); 19309/193224 objects misplaced (9.993%); 10 MiB/s, 161 objects/s recovering 2024-03-20T09:22:04.724 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:04 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:22:04.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:04 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:22:04.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:04 smithi026 ceph-mon[68309]: osdmap e671: 12 total, 12 up, 12 in 2024-03-20T09:22:04.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:04 smithi026 ceph-mon[68309]: pgmap v1635: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 593 op/s; 8947/193224 objects degraded (4.630%); 19309/193224 objects misplaced (9.993%); 10 MiB/s, 161 objects/s recovering 2024-03-20T09:22:05.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:05 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:22:05.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:05 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:22:05.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:05 smithi115 ceph-mon[72476]: osdmap e672: 12 total, 12 up, 12 in 2024-03-20T09:22:05.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:05 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:22:05.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:05 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:22:05.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:05 smithi122 ceph-mon[73495]: osdmap e672: 12 total, 12 up, 12 in 2024-03-20T09:22:05.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:05 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:22:05.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:05 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:22:05.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:05 smithi026 ceph-mon[68309]: osdmap e672: 12 total, 12 up, 12 in 2024-03-20T09:22:05.994 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:22:06.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:06.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:22:06.107 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:22:06.179 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:06.214 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:22:06.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:06.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:22:06.318 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:22:06.374 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:06.666 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:06 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:22:06.666 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:06 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:22:06.666 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:06 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:22:06.667 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:06 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:22:06.667 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:06 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:22:06.667 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:06 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:22:06.667 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:06 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:22:06.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:06 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:22:06.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:06 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:22:06.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:06 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:22:06.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:06 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:22:06.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:06 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:22:06.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:06 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:22:06.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:06 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:22:06.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:06 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:22:06.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:06 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:22:06.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:06 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:22:06.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:06 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:22:06.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:06 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:22:06.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:06 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:22:06.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:06 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:22:07.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:07 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8947/190185 objects degraded (4.704%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:22:07.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:07 smithi122 ceph-mon[73495]: osdmap e673: 12 total, 12 up, 12 in 2024-03-20T09:22:07.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:07 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8947/190185 objects degraded (4.704%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:22:07.726 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:07 smithi026 ceph-mon[68309]: osdmap e673: 12 total, 12 up, 12 in 2024-03-20T09:22:07.909 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:07 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8947/190185 objects degraded (4.704%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:22:07.909 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:07 smithi115 ceph-mon[72476]: osdmap e673: 12 total, 12 up, 12 in 2024-03-20T09:22:08.451 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:22:08.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:08.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:22:08.557 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:22:08.603 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:08.687 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:22:08.691 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:08 smithi026 ceph-mon[68309]: pgmap v1639: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 558 op/s; 9037/186774 objects degraded (4.838%); 18633/186774 objects misplaced (9.976%); 10 MiB/s, 162 objects/s recovering 2024-03-20T09:22:08.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:08 smithi122 ceph-mon[73495]: pgmap v1639: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 558 op/s; 9037/186774 objects degraded (4.838%); 18633/186774 objects misplaced (9.976%); 10 MiB/s, 162 objects/s recovering 2024-03-20T09:22:08.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:08.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:22:08.808 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:22:08.839 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:08.867 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:22:08.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:08.870 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:22:08.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:08 smithi115 ceph-mon[72476]: pgmap v1639: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 558 op/s; 9037/186774 objects degraded (4.838%); 18633/186774 objects misplaced (9.976%); 10 MiB/s, 162 objects/s recovering 2024-03-20T09:22:08.971 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:22:09.020 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:10.001 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:22:10.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:10.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:22:10.110 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:22:10.164 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:10.208 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:22:10.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:10.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:22:10.313 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:22:10.345 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:10.371 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:22:10.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:10.376 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:22:10.477 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:22:10.531 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:10.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:10.591 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:22:10.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:22:10.692 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:22:10.748 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:10.792 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:22:10.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:10.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:22:10.901 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:22:10.962 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:11.001 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:22:11.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:11.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:22:11.110 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:22:11.165 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:11.175 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:22:11.177 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:22:11.177 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:22:11.178 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-03-20T09:22:11.178 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-03-20T09:22:11.178 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:22:11.195 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:22:11.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:11.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:22:11.210 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:22:11.274 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:10 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:22:11.274 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:10 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:22:11.274 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:10 smithi026 ceph-mon[68309]: osdmap e674: 12 total, 12 up, 12 in 2024-03-20T09:22:11.274 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:10 smithi026 ceph-mon[68309]: pgmap v1641: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 559 op/s; 9033/183150 objects degraded (4.932%); 18054/183150 objects misplaced (9.857%); 8.1 MiB/s, 130 objects/s recovering 2024-03-20T09:22:11.300 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:22:11.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:10 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:22:11.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:10 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:22:11.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:10 smithi115 ceph-mon[72476]: osdmap e674: 12 total, 12 up, 12 in 2024-03-20T09:22:11.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:10 smithi115 ceph-mon[72476]: pgmap v1641: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 559 op/s; 9033/183150 objects degraded (4.932%); 18054/183150 objects misplaced (9.857%); 8.1 MiB/s, 130 objects/s recovering 2024-03-20T09:22:11.408 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:11.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:10 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:22:11.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:10 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:22:11.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:10 smithi122 ceph-mon[73495]: osdmap e674: 12 total, 12 up, 12 in 2024-03-20T09:22:11.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:10 smithi122 ceph-mon[73495]: pgmap v1641: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.7 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 559 op/s; 9033/183150 objects degraded (4.932%); 18054/183150 objects misplaced (9.857%); 8.1 MiB/s, 130 objects/s recovering 2024-03-20T09:22:12.034 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.5 to 1 (10000) 2024-03-20T09:22:12.414 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:12 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1825704647' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:22:12.414 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:12 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:22:12.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:12 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1825704647' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:22:12.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:12 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:22:12.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:12 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1825704647' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:22:12.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:12 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:22:13.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:13 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-20T09:22:13.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:13 smithi115 ceph-mon[72476]: osdmap e675: 12 total, 12 up, 12 in 2024-03-20T09:22:13.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:13 smithi115 ceph-mon[72476]: pgmap v1643: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 647 op/s; 9048/178524 objects degraded (5.068%); 17601/178524 objects misplaced (9.859%); 13 MiB/s, 203 objects/s recovering 2024-03-20T09:22:13.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:13 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 9033/183150 objects degraded (4.932%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:22:13.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:13 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-20T09:22:13.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:13 smithi122 ceph-mon[73495]: osdmap e675: 12 total, 12 up, 12 in 2024-03-20T09:22:13.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:13 smithi122 ceph-mon[73495]: pgmap v1643: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 647 op/s; 9048/178524 objects degraded (5.068%); 17601/178524 objects misplaced (9.859%); 13 MiB/s, 203 objects/s recovering 2024-03-20T09:22:13.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:13 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 9033/183150 objects degraded (4.932%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:22:13.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:13 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-20T09:22:13.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:13 smithi026 ceph-mon[68309]: osdmap e675: 12 total, 12 up, 12 in 2024-03-20T09:22:13.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:13 smithi026 ceph-mon[68309]: pgmap v1643: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 647 op/s; 9048/178524 objects degraded (5.068%); 17601/178524 objects misplaced (9.859%); 13 MiB/s, 203 objects/s recovering 2024-03-20T09:22:13.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:13 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 9033/183150 objects degraded (4.932%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:22:15.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:15 smithi122 ceph-mon[73495]: pgmap v1644: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 626 op/s; 9048/175158 objects degraded (5.166%); 16909/175158 objects misplaced (9.654%); 10 MiB/s, 166 objects/s recovering 2024-03-20T09:22:15.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:15 smithi026 ceph-mon[68309]: pgmap v1644: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 626 op/s; 9048/175158 objects degraded (5.166%); 16909/175158 objects misplaced (9.654%); 10 MiB/s, 166 objects/s recovering 2024-03-20T09:22:15.550 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:22:15.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:15.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:22:15.611 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-20T09:22:15.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:15 smithi115 ceph-mon[72476]: pgmap v1644: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 626 op/s; 9048/175158 objects degraded (5.166%); 16909/175158 objects misplaced (9.654%); 10 MiB/s, 166 objects/s recovering 2024-03-20T09:22:15.661 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:22:15.689 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:15.716 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:22:15.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:15.725 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:22:15.826 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:22:15.881 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:15.932 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:22:15.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:15.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:22:16.039 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:22:16.081 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:16.215 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:22:16.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:16.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:22:16.325 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:22:16.398 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:16.436 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:22:16.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:16.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:22:16.440 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 6 1 2024-03-20T09:22:16.540 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:22:16.594 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:16.623 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:22:16.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:16.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:22:16.731 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:22:16.794 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:16.835 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:22:16.911 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:22:16.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:16.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:22:17.016 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:22:17.077 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:17.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:17 smithi122 ceph-mon[73495]: pgmap v1645: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 630 op/s; 9048/171645 objects degraded (5.271%); 16909/171645 objects misplaced (9.851%); 7.5 MiB/s, 121 objects/s recovering 2024-03-20T09:22:17.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:17 smithi026 ceph-mon[68309]: pgmap v1645: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 630 op/s; 9048/171645 objects degraded (5.271%); 16909/171645 objects misplaced (9.851%); 7.5 MiB/s, 121 objects/s recovering 2024-03-20T09:22:17.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:17 smithi115 ceph-mon[72476]: pgmap v1645: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 630 op/s; 9048/171645 objects degraded (5.271%); 16909/171645 objects misplaced (9.851%); 7.5 MiB/s, 121 objects/s recovering 2024-03-20T09:22:18.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:18 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 9048/171645 objects degraded (5.271%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:22:18.660 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:18 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 9048/171645 objects degraded (5.271%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:22:18.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:18 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 9048/171645 objects degraded (5.271%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:22:19.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:19 smithi026 ceph-mon[68309]: pgmap v1646: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 607 op/s; 9090/168264 objects degraded (5.402%); 16255/168264 objects misplaced (9.660%); 11 MiB/s, 170 objects/s recovering 2024-03-20T09:22:19.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:19 smithi115 ceph-mon[72476]: pgmap v1646: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 607 op/s; 9090/168264 objects degraded (5.402%); 16255/168264 objects misplaced (9.660%); 11 MiB/s, 170 objects/s recovering 2024-03-20T09:22:19.682 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:22:19.690 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:22:19.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:19.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:22:19.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:19 smithi122 ceph-mon[73495]: pgmap v1646: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 607 op/s; 9090/168264 objects degraded (5.402%); 16255/168264 objects misplaced (9.660%); 11 MiB/s, 170 objects/s recovering 2024-03-20T09:22:19.753 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:22:19.795 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:22:19.823 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:19.847 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:22:19.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:19.851 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:22:19.952 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:22:19.992 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:20.016 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:22:20.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:20.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:22:20.125 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:22:20.209 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:20.284 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:22:20.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:20.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:22:20.395 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:22:20.429 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:20.450 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:22:20.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:20.454 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:22:20.554 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:22:20.620 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:20 smithi115 ceph-mon[72476]: pgmap v1647: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 597 op/s; 9080/165216 objects degraded (5.496%); 15720/165216 objects misplaced (9.515%); 9.8 MiB/s, 157 objects/s recovering 2024-03-20T09:22:20.622 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:20.655 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:22:20.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:20.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:22:20.701 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:20 smithi122 ceph-mon[73495]: pgmap v1647: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 597 op/s; 9080/165216 objects degraded (5.496%); 15720/165216 objects misplaced (9.515%); 9.8 MiB/s, 157 objects/s recovering 2024-03-20T09:22:20.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:20 smithi026 ceph-mon[68309]: pgmap v1647: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 597 op/s; 9080/165216 objects degraded (5.496%); 15720/165216 objects misplaced (9.515%); 9.8 MiB/s, 157 objects/s recovering 2024-03-20T09:22:20.768 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:22:20.830 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:20.870 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:22:20.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:20.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:22:20.980 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:22:21.011 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:21.031 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:22:21.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:21.035 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:22:21.137 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:22:21.202 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:21.262 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:22:21.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:21.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:22:21.379 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:22:21.420 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:21.438 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:22:21.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:21.442 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:22:21.543 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:22:21.574 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:21.600 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:22:21.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:21.606 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:22:21.707 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:22:21.735 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:21.760 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:22:21.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:21.764 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:22:21.864 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:22:21.896 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:21.922 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:22:21.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:21.925 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:22:22.026 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:22:22.071 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:23.350 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:22:23.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:23.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:22:23.361 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:23 smithi026 ceph-mon[68309]: pgmap v1648: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 604 op/s; 9143/160173 objects degraded (5.708%); 15255/160173 objects misplaced (9.524%); 9.4 MiB/s, 152 objects/s recovering 2024-03-20T09:22:23.361 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:23 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 9143/160173 objects degraded (5.708%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:22:23.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:23 smithi122 ceph-mon[73495]: pgmap v1648: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 604 op/s; 9143/160173 objects degraded (5.708%); 15255/160173 objects misplaced (9.524%); 9.4 MiB/s, 152 objects/s recovering 2024-03-20T09:22:23.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:23 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 9143/160173 objects degraded (5.708%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:22:23.462 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:22:23.498 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:23.529 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:22:23.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:23.532 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:22:23.633 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:22:23.662 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:23 smithi115 ceph-mon[72476]: pgmap v1648: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 604 op/s; 9143/160173 objects degraded (5.708%); 15255/160173 objects misplaced (9.524%); 9.4 MiB/s, 152 objects/s recovering 2024-03-20T09:22:23.662 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:23 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 9143/160173 objects degraded (5.708%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:22:23.699 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:23.733 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:22:23.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:23.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:22:23.837 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:22:23.896 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:23.950 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:22:23.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:23.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:22:24.054 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:22:24.133 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:24.165 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:22:24.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:24.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:22:24.275 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:22:24.350 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:24.378 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:22:24.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:24.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:22:24.493 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:22:24.556 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:24.600 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:22:24.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:24.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:22:24.703 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:22:24.752 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:25.205 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:25 smithi026 ceph-mon[68309]: pgmap v1649: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 639 op/s; 9228/155484 objects degraded (5.935%); 14489/155484 objects misplaced (9.319%); 9.6 MiB/s, 154 objects/s recovering 2024-03-20T09:22:25.667 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:25 smithi115 ceph-mon[72476]: pgmap v1649: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 639 op/s; 9228/155484 objects degraded (5.935%); 14489/155484 objects misplaced (9.319%); 9.6 MiB/s, 154 objects/s recovering 2024-03-20T09:22:25.699 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:25 smithi122 ceph-mon[73495]: pgmap v1649: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 639 op/s; 9228/155484 objects degraded (5.935%); 14489/155484 objects misplaced (9.319%); 9.6 MiB/s, 154 objects/s recovering 2024-03-20T09:22:27.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:27 smithi026 ceph-mon[68309]: pgmap v1650: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 612 op/s; 9228/153087 objects degraded (6.028%); 14489/153087 objects misplaced (9.465%); 7.8 MiB/s, 126 objects/s recovering 2024-03-20T09:22:27.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:27 smithi115 ceph-mon[72476]: pgmap v1650: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 612 op/s; 9228/153087 objects degraded (6.028%); 14489/153087 objects misplaced (9.465%); 7.8 MiB/s, 126 objects/s recovering 2024-03-20T09:22:27.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:27 smithi122 ceph-mon[73495]: pgmap v1650: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 612 op/s; 9228/153087 objects degraded (6.028%); 14489/153087 objects misplaced (9.465%); 7.8 MiB/s, 126 objects/s recovering 2024-03-20T09:22:27.735 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:22:27.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:27.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:22:27.844 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:22:27.877 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:27.900 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:22:27.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:27.903 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:22:28.004 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:22:28.032 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:28.064 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:22:28.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:28.067 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:22:28.168 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:22:28.246 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:28.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:28 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 9228/153087 objects degraded (6.028%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:22:28.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:28 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 9228/153087 objects degraded (6.028%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:22:28.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:28 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 9228/153087 objects degraded (6.028%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:22:29.231 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:29 smithi026 ceph-mon[68309]: pgmap v1651: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 614 op/s; 9360/149505 objects degraded (6.261%); 13787/149505 objects misplaced (9.222%); 11 MiB/s, 173 objects/s recovering 2024-03-20T09:22:29.231 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:29 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/42797798' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:22:29.244 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:22:29.273 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:22:29.306 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:22:29.341 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:22:29.371 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:22:29.416 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:22:29.430 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:22:29.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:29.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:22:29.454 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:22:29.540 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:22:29.541 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:22:29.568 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:22:29.569 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:29 smithi122 ceph-mon[73495]: pgmap v1651: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 614 op/s; 9360/149505 objects degraded (6.261%); 13787/149505 objects misplaced (9.222%); 11 MiB/s, 173 objects/s recovering 2024-03-20T09:22:29.570 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:29 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/42797798' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:22:29.571 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:29.596 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:22:29.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:29.599 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:22:29.612 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:22:29.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:29 smithi115 ceph-mon[72476]: pgmap v1651: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 614 op/s; 9360/149505 objects degraded (6.261%); 13787/149505 objects misplaced (9.222%); 11 MiB/s, 173 objects/s recovering 2024-03-20T09:22:29.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:29 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/42797798' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:22:29.660 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:22:29.694 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:22:29.699 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:22:29.740 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:22:29.740 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:22:29.740 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:22:29.740 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:22:29.740 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:22:29.740 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:22:29.740 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:22:29.740 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:22:29.740 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:22:29.740 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:22:29.740 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:22:29.740 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:22:29.758 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:30.239 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.6 to 1 (10000) 2024-03-20T09:22:30.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:30 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3366531043' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-20T09:22:30.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:30 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-20T09:22:30.660 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:30 smithi115 ceph-mon[72476]: pgmap v1652: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 628 op/s; 9359/145602 objects degraded (6.428%); 13130/145602 objects misplaced (9.018%); 8.7 MiB/s, 141 objects/s recovering 2024-03-20T09:22:30.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:30 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3366531043' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-20T09:22:30.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:30 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-20T09:22:30.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:30 smithi122 ceph-mon[73495]: pgmap v1652: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 628 op/s; 9359/145602 objects degraded (6.428%); 13130/145602 objects misplaced (9.018%); 8.7 MiB/s, 141 objects/s recovering 2024-03-20T09:22:30.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:30 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3366531043' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-20T09:22:30.726 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:30 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-20T09:22:30.726 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:30 smithi026 ceph-mon[68309]: pgmap v1652: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 628 op/s; 9359/145602 objects degraded (6.428%); 13130/145602 objects misplaced (9.018%); 8.7 MiB/s, 141 objects/s recovering 2024-03-20T09:22:31.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:31 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-20T09:22:31.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:31 smithi026 ceph-mon[68309]: osdmap e676: 12 total, 12 up, 12 in 2024-03-20T09:22:31.662 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:31 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-20T09:22:31.662 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:31 smithi115 ceph-mon[72476]: osdmap e676: 12 total, 12 up, 12 in 2024-03-20T09:22:31.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:31 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-20T09:22:31.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:31 smithi122 ceph-mon[73495]: osdmap e676: 12 total, 12 up, 12 in 2024-03-20T09:22:32.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:32 smithi115 ceph-mon[72476]: pgmap v1654: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 651 op/s; 9393/140616 objects degraded (6.680%); 12616/140616 objects misplaced (8.972%); 9.9 MiB/s, 158 objects/s recovering 2024-03-20T09:22:32.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:32 smithi122 ceph-mon[73495]: pgmap v1654: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 651 op/s; 9393/140616 objects degraded (6.680%); 12616/140616 objects misplaced (8.972%); 9.9 MiB/s, 158 objects/s recovering 2024-03-20T09:22:32.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:32 smithi026 ceph-mon[68309]: pgmap v1654: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 651 op/s; 9393/140616 objects degraded (6.680%); 12616/140616 objects misplaced (8.972%); 9.9 MiB/s, 158 objects/s recovering 2024-03-20T09:22:33.030 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:22:33.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:33.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:22:33.144 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:22:33.221 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:33.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:33 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:22:33.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:33 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:22:33.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:33 smithi026 ceph-mon[68309]: osdmap e677: 12 total, 12 up, 12 in 2024-03-20T09:22:33.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:33 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 9393/140616 objects degraded (6.680%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:22:33.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:33 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:22:33.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:33 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:22:33.487 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:22:33.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:33.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:22:33.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:33 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:22:33.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:33 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:22:33.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:33 smithi115 ceph-mon[72476]: osdmap e677: 12 total, 12 up, 12 in 2024-03-20T09:22:33.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:33 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 9393/140616 objects degraded (6.680%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:22:33.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:33 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:22:33.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:33 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:22:33.667 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:22:33.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:33 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:22:33.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:33 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:22:33.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:33 smithi122 ceph-mon[73495]: osdmap e677: 12 total, 12 up, 12 in 2024-03-20T09:22:33.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:33 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 9393/140616 objects degraded (6.680%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:22:33.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:33 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:22:33.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:33 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:22:33.718 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:34.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:34 smithi115 ceph-mon[72476]: osdmap e678: 12 total, 12 up, 12 in 2024-03-20T09:22:34.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:34 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1085347740' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:22:34.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:34 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:22:34.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:34 smithi115 ceph-mon[72476]: pgmap v1657: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 742 op/s; 9422/136140 objects degraded (6.921%); 11756/136140 objects misplaced (8.635%); 11 MiB/s, 183 objects/s recovering 2024-03-20T09:22:34.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:34 smithi122 ceph-mon[73495]: osdmap e678: 12 total, 12 up, 12 in 2024-03-20T09:22:34.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:34 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1085347740' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:22:34.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:34 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:22:34.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:34 smithi122 ceph-mon[73495]: pgmap v1657: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 742 op/s; 9422/136140 objects degraded (6.921%); 11756/136140 objects misplaced (8.635%); 11 MiB/s, 183 objects/s recovering 2024-03-20T09:22:34.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:34 smithi026 ceph-mon[68309]: osdmap e678: 12 total, 12 up, 12 in 2024-03-20T09:22:34.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:34 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1085347740' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:22:34.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:34 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:22:34.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:34 smithi026 ceph-mon[68309]: pgmap v1657: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 742 op/s; 9422/136140 objects degraded (6.921%); 11756/136140 objects misplaced (8.635%); 11 MiB/s, 183 objects/s recovering 2024-03-20T09:22:35.283 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 7 1 2024-03-20T09:22:35.343 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:22:35.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:35.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:22:35.454 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:22:35.503 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:35.570 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:22:35.573 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:35 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:22:35.573 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:35 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:22:35.573 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:35 smithi115 ceph-mon[72476]: osdmap e679: 12 total, 12 up, 12 in 2024-03-20T09:22:35.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:35.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:22:35.598 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:22:35.598 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:22:35.625 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:35 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:22:35.625 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:35 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:22:35.625 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:35 smithi026 ceph-mon[68309]: osdmap e679: 12 total, 12 up, 12 in 2024-03-20T09:22:35.637 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:22:35.675 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:22:35.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:35 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:22:35.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:35 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:22:35.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:35 smithi122 ceph-mon[73495]: osdmap e679: 12 total, 12 up, 12 in 2024-03-20T09:22:35.714 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:35.752 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:22:35.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:35.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:22:35.863 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:22:35.902 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:35.930 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:22:35.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:35.933 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:22:36.034 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:22:36.098 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:36.161 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:22:36.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:36.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:22:36.272 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:22:36.302 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:36.328 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:22:36.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:36.333 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:22:36.433 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:22:36.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:36 smithi026 ceph-mon[68309]: pgmap v1659: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 658 op/s; 9422/133968 objects degraded (7.033%); 11756/133968 objects misplaced (8.775%); 9.1 MiB/s, 149 objects/s recovering 2024-03-20T09:22:36.504 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:36.554 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:22:36.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:36.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:22:36.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:36 smithi115 ceph-mon[72476]: pgmap v1659: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 658 op/s; 9422/133968 objects degraded (7.033%); 11756/133968 objects misplaced (8.775%); 9.1 MiB/s, 149 objects/s recovering 2024-03-20T09:22:36.662 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:22:36.688 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:36 smithi122 ceph-mon[73495]: pgmap v1659: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 658 op/s; 9422/133968 objects degraded (7.033%); 11756/133968 objects misplaced (8.775%); 9.1 MiB/s, 149 objects/s recovering 2024-03-20T09:22:36.690 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:36.714 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:22:36.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:36.718 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:22:36.819 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:22:36.876 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:36.946 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:22:37.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:37.100 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:22:37.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:22:37.132 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-20T09:22:37.137 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:22:37.202 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:22:37.229 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:37.250 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:22:37.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:37.255 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:22:37.357 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:22:37.387 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:37.411 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:22:37.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:37.418 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:22:37.519 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:22:37.548 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:37 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 9422/133968 objects degraded (7.033%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:22:37.548 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:37 smithi122 ceph-mon[73495]: osdmap e680: 12 total, 12 up, 12 in 2024-03-20T09:22:37.550 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:37.570 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:22:37.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:37.573 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:22:37.580 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:22:37.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:37 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 9422/133968 objects degraded (7.033%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:22:37.660 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:37 smithi115 ceph-mon[72476]: osdmap e680: 12 total, 12 up, 12 in 2024-03-20T09:22:37.674 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:22:37.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:37 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 9422/133968 objects degraded (7.033%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:22:37.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:37 smithi026 ceph-mon[68309]: osdmap e680: 12 total, 12 up, 12 in 2024-03-20T09:22:37.719 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:38.094 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:22:38.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:38.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:22:38.199 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:22:38.267 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:38.608 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:22:38.623 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:22:38.632 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:22:38.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:38.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:22:38.677 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:38 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:22:38.678 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:38 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:22:38.678 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:38 smithi026 ceph-mon[68309]: pgmap v1661: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 568 op/s; 9446/130632 objects degraded (7.231%); 11020/130632 objects misplaced (8.436%); 12 MiB/s, 189 objects/s recovering 2024-03-20T09:22:38.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:38 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:22:38.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:38 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:22:38.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:38 smithi122 ceph-mon[73495]: pgmap v1661: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 568 op/s; 9446/130632 objects degraded (7.231%); 11020/130632 objects misplaced (8.436%); 12 MiB/s, 189 objects/s recovering 2024-03-20T09:22:38.744 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:22:38.822 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:38.899 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:22:38.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:38.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:22:38.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:38 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:22:38.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:38 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:22:38.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:38 smithi115 ceph-mon[72476]: pgmap v1661: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 568 op/s; 9446/130632 objects degraded (7.231%); 11020/130632 objects misplaced (8.436%); 12 MiB/s, 189 objects/s recovering 2024-03-20T09:22:39.006 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:22:39.071 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:41.343 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:22:41.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:41.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:22:41.410 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:41 smithi115 ceph-mon[72476]: pgmap v1662: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 553 op/s; 9440/127032 objects degraded (7.431%); 10480/127032 objects misplaced (8.250%); 10 MiB/s, 165 objects/s recovering 2024-03-20T09:22:41.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:41 smithi122 ceph-mon[73495]: pgmap v1662: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 553 op/s; 9440/127032 objects degraded (7.431%); 10480/127032 objects misplaced (8.250%); 10 MiB/s, 165 objects/s recovering 2024-03-20T09:22:41.450 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:22:41.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:41 smithi026 ceph-mon[68309]: pgmap v1662: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.6 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 553 op/s; 9440/127032 objects degraded (7.431%); 10480/127032 objects misplaced (8.250%); 10 MiB/s, 165 objects/s recovering 2024-03-20T09:22:41.482 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:41.508 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:22:41.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:41.515 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:22:41.616 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:22:41.660 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:42.709 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:22:42.710 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:22:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:43 smithi115 ceph-mon[72476]: pgmap v1663: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 579 op/s; 9440/122220 objects degraded (7.724%); 9921/122220 objects misplaced (8.117%); 11 MiB/s, 182 objects/s recovering 2024-03-20T09:22:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:43 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 9440/122220 objects degraded (7.724%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:22:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:43 smithi115 ceph-mon[72476]: osdmap e681: 12 total, 12 up, 12 in 2024-03-20T09:22:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:43 smithi122 ceph-mon[73495]: pgmap v1663: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 579 op/s; 9440/122220 objects degraded (7.724%); 9921/122220 objects misplaced (8.117%); 11 MiB/s, 182 objects/s recovering 2024-03-20T09:22:43.521 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:43 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 9440/122220 objects degraded (7.724%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:22:43.521 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:43 smithi122 ceph-mon[73495]: osdmap e681: 12 total, 12 up, 12 in 2024-03-20T09:22:43.522 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:43 smithi026 ceph-mon[68309]: pgmap v1663: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 579 op/s; 9440/122220 objects degraded (7.724%); 9921/122220 objects misplaced (8.117%); 11 MiB/s, 182 objects/s recovering 2024-03-20T09:22:43.522 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:43 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 9440/122220 objects degraded (7.724%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:22:43.522 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:43 smithi026 ceph-mon[68309]: osdmap e681: 12 total, 12 up, 12 in 2024-03-20T09:22:43.610 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-20T09:22:44.024 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:22:44.257 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:22:44.267 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:44.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:22:44.368 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:22:44.400 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:44.422 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:22:44.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:44.427 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:22:44.528 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:22:44.599 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:44.620 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:22:44.643 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:22:44.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:44.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:22:44.669 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:22:44.695 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:22:44.744 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:22:44.748 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:22:44.776 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:44.802 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:22:44.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:44.805 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:22:44.857 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:22:44.857 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:22:44.857 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:22:44.865 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:22:44.908 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:22:44.911 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:22:44.940 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:44.947 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:22:44.964 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:22:44.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:44.966 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:22:44.986 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:22:45.023 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:22:45.065 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:22:45.067 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:22:45.071 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:22:45.071 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:22:45.071 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:22:45.071 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:22:45.071 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:22:45.071 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:22:45.071 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:22:45.071 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:22:45.071 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:22:45.071 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:22:45.072 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:22:45.072 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:22:45.103 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:45.130 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:22:45.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:45.133 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:22:45.207 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.7 to 1 (10000) 2024-03-20T09:22:45.234 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:22:45.288 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:45.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:45 smithi115 ceph-mon[72476]: pgmap v1665: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 689 op/s; 9473/117405 objects degraded (8.069%); 9104/117405 objects misplaced (7.754%); 14 MiB/s, 227 objects/s recovering 2024-03-20T09:22:45.453 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:45 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3901008212' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:22:45.454 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:45 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:22:45.454 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:45 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3138544669' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:22:45.455 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:45 smithi122 ceph-mon[73495]: pgmap v1665: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 689 op/s; 9473/117405 objects degraded (8.069%); 9104/117405 objects misplaced (7.754%); 14 MiB/s, 227 objects/s recovering 2024-03-20T09:22:45.455 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:45 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3901008212' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:22:45.456 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:45 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:22:45.456 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:45 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3138544669' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:22:45.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:45 smithi026 ceph-mon[68309]: pgmap v1665: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 689 op/s; 9473/117405 objects degraded (8.069%); 9104/117405 objects misplaced (7.754%); 14 MiB/s, 227 objects/s recovering 2024-03-20T09:22:45.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:45 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3901008212' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:22:45.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:45 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:22:45.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:45 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3138544669' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:22:46.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:46 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-20T09:22:46.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:46 smithi026 ceph-mon[68309]: osdmap e682: 12 total, 12 up, 12 in 2024-03-20T09:22:46.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:46 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:22:46.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:46 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:22:46.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:46 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-20T09:22:46.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:46 smithi115 ceph-mon[72476]: osdmap e682: 12 total, 12 up, 12 in 2024-03-20T09:22:46.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:46 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:22:46.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:46 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:22:46.688 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:22:46.688 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:22:46.688 INFO:teuthology.orchestra.run.smithi115.stdout: "complaint_time": 30, 2024-03-20T09:22:46.688 INFO:teuthology.orchestra.run.smithi115.stdout: "num_blocked_ops": 0 2024-03-20T09:22:46.688 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:22:46.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:46 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-20T09:22:46.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:46 smithi122 ceph-mon[73495]: osdmap e682: 12 total, 12 up, 12 in 2024-03-20T09:22:46.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:46 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:22:46.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:46 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:22:47.411 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:47 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:22:47.412 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:47 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:22:47.412 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:47 smithi115 ceph-mon[72476]: pgmap v1667: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 669 op/s; 9473/114561 objects degraded (8.269%); 8982/114561 objects misplaced (7.840%); 9.5 MiB/s, 153 objects/s recovering 2024-03-20T09:22:47.412 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:47 smithi115 ceph-mon[72476]: osdmap e683: 12 total, 12 up, 12 in 2024-03-20T09:22:47.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:47 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:22:47.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:47 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:22:47.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:47 smithi122 ceph-mon[73495]: pgmap v1667: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 669 op/s; 9473/114561 objects degraded (8.269%); 8982/114561 objects misplaced (7.840%); 9.5 MiB/s, 153 objects/s recovering 2024-03-20T09:22:47.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:47 smithi122 ceph-mon[73495]: osdmap e683: 12 total, 12 up, 12 in 2024-03-20T09:22:47.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:47 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:22:47.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:47 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:22:47.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:47 smithi026 ceph-mon[68309]: pgmap v1667: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 669 op/s; 9473/114561 objects degraded (8.269%); 8982/114561 objects misplaced (7.840%); 9.5 MiB/s, 153 objects/s recovering 2024-03-20T09:22:47.725 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:47 smithi026 ceph-mon[68309]: osdmap e683: 12 total, 12 up, 12 in 2024-03-20T09:22:48.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:48 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:22:48.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:48 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:22:48.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:48 smithi115 ceph-mon[72476]: osdmap e684: 12 total, 12 up, 12 in 2024-03-20T09:22:48.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:48 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 9473/114561 objects degraded (8.269%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:22:48.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:48 smithi115 ceph-mon[72476]: osdmap e685: 12 total, 12 up, 12 in 2024-03-20T09:22:48.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:48 smithi115 ceph-mon[72476]: pgmap v1671: 47 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 453 op/s; 9456/111957 objects degraded (8.446%); 8457/111957 objects misplaced (7.554%); 9.1 MiB/s, 165 objects/s recovering 2024-03-20T09:22:48.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:48 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:22:48.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:48 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:22:48.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:48 smithi122 ceph-mon[73495]: osdmap e684: 12 total, 12 up, 12 in 2024-03-20T09:22:48.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:48 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 9473/114561 objects degraded (8.269%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:22:48.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:48 smithi122 ceph-mon[73495]: osdmap e685: 12 total, 12 up, 12 in 2024-03-20T09:22:48.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:48 smithi122 ceph-mon[73495]: pgmap v1671: 47 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 453 op/s; 9456/111957 objects degraded (8.446%); 8457/111957 objects misplaced (7.554%); 9.1 MiB/s, 165 objects/s recovering 2024-03-20T09:22:48.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:48 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:22:48.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:48 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:22:48.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:48 smithi026 ceph-mon[68309]: osdmap e684: 12 total, 12 up, 12 in 2024-03-20T09:22:48.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:48 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 9473/114561 objects degraded (8.269%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-20T09:22:48.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:48 smithi026 ceph-mon[68309]: osdmap e685: 12 total, 12 up, 12 in 2024-03-20T09:22:48.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:48 smithi026 ceph-mon[68309]: pgmap v1671: 47 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 453 op/s; 9456/111957 objects degraded (8.446%); 8457/111957 objects misplaced (7.554%); 9.1 MiB/s, 165 objects/s recovering 2024-03-20T09:22:49.759 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 8 1 2024-03-20T09:22:49.784 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:22:49.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:49.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:22:49.894 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-20T09:22:49.896 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:22:49.969 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:50.056 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:22:50.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:50.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:22:50.164 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:22:50.264 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:50.292 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:22:50.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:50.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:22:50.368 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:22:50.397 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:22:50.444 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:50.823 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:22:50.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:50.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:22:50.933 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:22:51.005 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:51.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:51 smithi115 ceph-mon[72476]: pgmap v1672: 47 pgs: 1 active+recovering+undersized+remapped, 1 peering, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 562 op/s; 7111/108657 objects degraded (6.544%); 9707/108657 objects misplaced (8.934%); 13 MiB/s, 227 objects/s recovering 2024-03-20T09:22:51.409 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:22:51.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:51 smithi026 ceph-mon[68309]: pgmap v1672: 47 pgs: 1 active+recovering+undersized+remapped, 1 peering, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 562 op/s; 7111/108657 objects degraded (6.544%); 9707/108657 objects misplaced (8.934%); 13 MiB/s, 227 objects/s recovering 2024-03-20T09:22:51.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:51 smithi122 ceph-mon[73495]: pgmap v1672: 47 pgs: 1 active+recovering+undersized+remapped, 1 peering, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.2 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 562 op/s; 7111/108657 objects degraded (6.544%); 9707/108657 objects misplaced (8.934%); 13 MiB/s, 227 objects/s recovering 2024-03-20T09:22:51.769 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:22:51.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:51.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:22:51.875 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:22:51.905 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:51.925 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:22:51.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:51.929 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:22:52.030 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:22:52.085 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:52.228 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:22:52.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:52 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1093179750' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:22:52.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:52 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:22:52.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:52 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1093179750' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:22:52.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:52 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:22:52.650 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:22:52.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:52.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:22:52.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:52 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1093179750' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:22:52.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:52 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:22:52.760 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:22:52.788 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:22:52.810 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:53.105 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:22:53.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:53.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:22:53.210 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:22:53.266 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:53.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:53 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:22:53.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:53 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:22:53.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:53 smithi115 ceph-mon[72476]: pgmap v1673: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 525 op/s; 7111/105096 objects degraded (6.766%); 9707/105096 objects misplaced (9.236%); 11 MiB/s, 186 objects/s recovering 2024-03-20T09:22:53.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:53 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:22:53.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:53 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:22:53.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:53 smithi115 ceph-mon[72476]: osdmap e686: 12 total, 12 up, 12 in 2024-03-20T09:22:53.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:53 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 7111/105096 objects degraded (6.766%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-20T09:22:53.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:53 smithi115 ceph-mon[72476]: osdmap e687: 12 total, 12 up, 12 in 2024-03-20T09:22:53.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:53 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:22:53.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:53 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:22:53.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:53 smithi026 ceph-mon[68309]: pgmap v1673: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 525 op/s; 7111/105096 objects degraded (6.766%); 9707/105096 objects misplaced (9.236%); 11 MiB/s, 186 objects/s recovering 2024-03-20T09:22:53.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:53 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:22:53.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:53 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:22:53.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:53 smithi026 ceph-mon[68309]: osdmap e686: 12 total, 12 up, 12 in 2024-03-20T09:22:53.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:53 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 7111/105096 objects degraded (6.766%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-20T09:22:53.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:53 smithi026 ceph-mon[68309]: osdmap e687: 12 total, 12 up, 12 in 2024-03-20T09:22:53.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:53 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:22:53.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:53 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:22:53.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:53 smithi122 ceph-mon[73495]: pgmap v1673: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 525 op/s; 7111/105096 objects degraded (6.766%); 9707/105096 objects misplaced (9.236%); 11 MiB/s, 186 objects/s recovering 2024-03-20T09:22:53.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:53 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:22:53.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:53 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:22:53.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:53 smithi122 ceph-mon[73495]: osdmap e686: 12 total, 12 up, 12 in 2024-03-20T09:22:53.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:53 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 7111/105096 objects degraded (6.766%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-20T09:22:53.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:53 smithi122 ceph-mon[73495]: osdmap e687: 12 total, 12 up, 12 in 2024-03-20T09:22:54.040 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:22:54.150 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:22:54.150 INFO:teuthology.orchestra.run.smithi115.stdout: "size": 20, 2024-03-20T09:22:54.150 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 600, 2024-03-20T09:22:54.150 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [ 2024-03-20T09:22:54.150 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.150 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:133737 5.1 e640/637 5:882b2c30:::benchmark_data_smithi122_157894_object3750:head v 640'8271, mlcod=611'228)", 2024-03-20T09:22:54.150 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:19:59.599626+0000", 2024-03-20T09:22:54.150 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 174.546012711, 2024-03-20T09:22:54.150 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.087846301000000002, 2024-03-20T09:22:54.151 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:22:54.151 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:22:54.151 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:22:54.151 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.151 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:22:54.151 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:19:59.599626+0000", 2024-03-20T09:22:54.151 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.151 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.151 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.151 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:22:54.151 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:19:59.599626+0000", 2024-03-20T09:22:54.151 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.151 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.151 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.152 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:22:54.152 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:19:59.599627+0000", 2024-03-20T09:22:54.152 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.2900000000000002e-07 2024-03-20T09:22:54.152 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.152 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.152 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:22:54.152 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:19:59.599635+0000", 2024-03-20T09:22:54.152 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.2649999999999999e-06 2024-03-20T09:22:54.152 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.152 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.152 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:22:54.152 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:19:59.599647+0000", 2024-03-20T09:22:54.153 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.199e-05 2024-03-20T09:22:54.153 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.153 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.153 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:22:54.153 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:19:59.599653+0000", 2024-03-20T09:22:54.153 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1970000000000001e-06 2024-03-20T09:22:54.153 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.153 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.153 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:22:54.153 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:19:59.623436+0000", 2024-03-20T09:22:54.153 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.023783420999999999 2024-03-20T09:22:54.153 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.154 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.154 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:22:54.154 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:19:59.623463+0000", 2024-03-20T09:22:54.154 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6812e-05 2024-03-20T09:22:54.154 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.154 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.154 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:22:54.154 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:19:59.687422+0000", 2024-03-20T09:22:54.154 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.063959489999999994 2024-03-20T09:22:54.154 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.154 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.154 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:22:54.155 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:19:59.687473+0000", 2024-03-20T09:22:54.155 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0297000000000003e-05 2024-03-20T09:22:54.155 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.155 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:22:54.155 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.155 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.155 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.155 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:192903 5.5 e668/637 5:ad82fe62:::benchmark_data_smithi122_157894_object62916:head v 668'12075, mlcod=620'3813)", 2024-03-20T09:22:54.155 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:21:56.998657+0000", 2024-03-20T09:22:54.155 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 57.146981906000001, 2024-03-20T09:22:54.155 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.087689238000000003, 2024-03-20T09:22:54.155 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:22:54.155 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:22:54.156 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:22:54.156 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.156 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:22:54.156 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:21:56.998657+0000", 2024-03-20T09:22:54.156 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.156 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.156 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.156 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:22:54.156 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:21:56.998657+0000", 2024-03-20T09:22:54.156 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.156 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.156 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.156 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:22:54.157 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:21:56.998658+0000", 2024-03-20T09:22:54.157 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7099999999999997e-07 2024-03-20T09:22:54.157 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.157 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.157 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:22:54.157 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:21:56.998662+0000", 2024-03-20T09:22:54.157 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6129999999999997e-06 2024-03-20T09:22:54.157 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.157 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.157 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:22:54.157 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:21:56.998669+0000", 2024-03-20T09:22:54.157 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.0679999999999999e-06 2024-03-20T09:22:54.158 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.158 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.158 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:22:54.158 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:21:56.998674+0000", 2024-03-20T09:22:54.158 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5279999999999997e-06 2024-03-20T09:22:54.158 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.158 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.158 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:22:54.158 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:21:57.017425+0000", 2024-03-20T09:22:54.158 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.018751196000000001 2024-03-20T09:22:54.158 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.159 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.159 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:22:54.159 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:21:57.017450+0000", 2024-03-20T09:22:54.159 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.4752999999999998e-05 2024-03-20T09:22:54.159 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.159 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.159 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:22:54.159 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:21:57.086306+0000", 2024-03-20T09:22:54.159 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.068855792999999998 2024-03-20T09:22:54.159 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.159 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.159 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:22:54.160 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:21:57.086346+0000", 2024-03-20T09:22:54.160 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.0915999999999999e-05 2024-03-20T09:22:54.160 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.160 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:22:54.160 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.160 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.160 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.160 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:192913 5.5 e668/637 5:a5ea0c0b:::benchmark_data_smithi122_157894_object62926:head v 668'12076, mlcod=620'3813)", 2024-03-20T09:22:54.160 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:21:56.999874+0000", 2024-03-20T09:22:54.160 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 57.145765482999998, 2024-03-20T09:22:54.160 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.086503127999999999, 2024-03-20T09:22:54.160 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:22:54.160 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:22:54.160 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:22:54.161 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.161 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:22:54.161 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:21:56.999874+0000", 2024-03-20T09:22:54.161 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.161 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.161 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.161 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:22:54.161 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:21:56.999874+0000", 2024-03-20T09:22:54.161 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.161 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.161 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.161 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:22:54.161 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:21:56.999874+0000", 2024-03-20T09:22:54.161 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7000000000000004e-07 2024-03-20T09:22:54.162 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.162 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.162 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:22:54.162 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:21:56.999880+0000", 2024-03-20T09:22:54.162 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0360000000000003e-06 2024-03-20T09:22:54.162 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.162 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.162 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:22:54.162 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:21:56.999890+0000", 2024-03-20T09:22:54.162 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.3470000000000005e-06 2024-03-20T09:22:54.162 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.162 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.162 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:22:54.163 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:21:56.999896+0000", 2024-03-20T09:22:54.163 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4320000000000004e-06 2024-03-20T09:22:54.163 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.163 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.163 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:22:54.163 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:21:57.027807+0000", 2024-03-20T09:22:54.163 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.027911021000000001 2024-03-20T09:22:54.163 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.163 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.163 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:22:54.163 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:21:57.027835+0000", 2024-03-20T09:22:54.163 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.7637999999999999e-05 2024-03-20T09:22:54.163 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.164 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.164 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:22:54.164 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:21:57.086353+0000", 2024-03-20T09:22:54.164 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.058517793999999998 2024-03-20T09:22:54.164 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.164 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.164 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:22:54.164 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:21:57.086377+0000", 2024-03-20T09:22:54.164 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.4090000000000001e-05 2024-03-20T09:22:54.164 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.164 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:22:54.164 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.164 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.165 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.165 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:198670 5.5 e672/637 5:a8531100:::benchmark_data_smithi122_157894_object68683:head v 672'12430, mlcod=622'4145)", 2024-03-20T09:22:54.165 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:06.959336+0000", 2024-03-20T09:22:54.165 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 47.186303508000002, 2024-03-20T09:22:54.165 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.081508116000000005, 2024-03-20T09:22:54.165 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:22:54.165 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:22:54.165 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:22:54.165 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.165 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:22:54.165 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:06.959336+0000", 2024-03-20T09:22:54.165 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.165 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.166 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.166 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:22:54.166 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:06.959336+0000", 2024-03-20T09:22:54.166 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.166 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.166 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.166 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:22:54.166 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:06.959337+0000", 2024-03-20T09:22:54.166 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.8299999999999995e-07 2024-03-20T09:22:54.166 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.166 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.166 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:22:54.166 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:06.959345+0000", 2024-03-20T09:22:54.167 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.2009999999999999e-06 2024-03-20T09:22:54.167 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.167 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.167 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:22:54.167 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:06.959360+0000", 2024-03-20T09:22:54.167 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4822e-05 2024-03-20T09:22:54.167 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.167 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.167 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:22:54.167 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:06.959367+0000", 2024-03-20T09:22:54.167 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.9529999999999999e-06 2024-03-20T09:22:54.167 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.167 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.167 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:22:54.168 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:06.959416+0000", 2024-03-20T09:22:54.168 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8415999999999999e-05 2024-03-20T09:22:54.168 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.168 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.168 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:22:54.168 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:06.959441+0000", 2024-03-20T09:22:54.168 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5267999999999999e-05 2024-03-20T09:22:54.168 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.168 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.168 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:22:54.168 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:07.040794+0000", 2024-03-20T09:22:54.168 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.081353236999999995 2024-03-20T09:22:54.168 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.168 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.169 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:22:54.169 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:07.040844+0000", 2024-03-20T09:22:54.169 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9336000000000003e-05 2024-03-20T09:22:54.169 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.169 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:22:54.169 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.169 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.169 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.169 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:198672 5.5 e672/637 5:a4e43c2c:::benchmark_data_smithi122_157894_object68685:head v 672'12431, mlcod=622'4145)", 2024-03-20T09:22:54.169 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:06.959983+0000", 2024-03-20T09:22:54.169 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 47.185656447, 2024-03-20T09:22:54.169 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.080900867000000001, 2024-03-20T09:22:54.169 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:22:54.169 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:22:54.170 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:22:54.170 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.170 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:22:54.170 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:06.959983+0000", 2024-03-20T09:22:54.170 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.170 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.170 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.170 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:22:54.170 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:06.959983+0000", 2024-03-20T09:22:54.170 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.170 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.170 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.170 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:22:54.171 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:06.959983+0000", 2024-03-20T09:22:54.171 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2399999999999998e-07 2024-03-20T09:22:54.171 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.171 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.171 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:22:54.171 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:06.959989+0000", 2024-03-20T09:22:54.171 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1859999999999997e-06 2024-03-20T09:22:54.171 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.171 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.171 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:22:54.171 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:06.960002+0000", 2024-03-20T09:22:54.171 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2122e-05 2024-03-20T09:22:54.171 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.171 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.172 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:22:54.172 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:06.960008+0000", 2024-03-20T09:22:54.172 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.3999999999999997e-06 2024-03-20T09:22:54.172 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.172 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.172 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:22:54.172 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:06.970210+0000", 2024-03-20T09:22:54.172 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.01020195 2024-03-20T09:22:54.172 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.172 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.172 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:22:54.172 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:06.970234+0000", 2024-03-20T09:22:54.172 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3804000000000001e-05 2024-03-20T09:22:54.172 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.172 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.173 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:22:54.173 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:07.040850+0000", 2024-03-20T09:22:54.173 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.070616590000000007 2024-03-20T09:22:54.173 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.173 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.173 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:22:54.173 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:07.040884+0000", 2024-03-20T09:22:54.173 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.3290999999999997e-05 2024-03-20T09:22:54.173 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.173 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:22:54.173 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.173 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.173 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.173 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:207723 5.5 e675/637 5:a514770a:::benchmark_data_smithi122_157894_object77736:head v 675'13000, mlcod=623'4797)", 2024-03-20T09:22:54.174 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:21.435668+0000", 2024-03-20T09:22:54.174 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 32.709971275000001, 2024-03-20T09:22:54.174 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.082899156000000002, 2024-03-20T09:22:54.174 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:22:54.174 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:22:54.174 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:22:54.174 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.174 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:22:54.174 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:21.435668+0000", 2024-03-20T09:22:54.174 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.174 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.174 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.174 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:22:54.174 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:21.435668+0000", 2024-03-20T09:22:54.175 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.175 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.175 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.175 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:22:54.175 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:21.435669+0000", 2024-03-20T09:22:54.175 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.4099999999999997e-07 2024-03-20T09:22:54.175 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.175 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.175 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:22:54.175 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:21.435677+0000", 2024-03-20T09:22:54.175 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.1000000000000004e-06 2024-03-20T09:22:54.175 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.175 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.175 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:22:54.176 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:21.435690+0000", 2024-03-20T09:22:54.176 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3494e-05 2024-03-20T09:22:54.176 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.176 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.176 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:22:54.176 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:21.435699+0000", 2024-03-20T09:22:54.176 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.276e-06 2024-03-20T09:22:54.176 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.176 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.176 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:22:54.176 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:21.448547+0000", 2024-03-20T09:22:54.176 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.012848515 2024-03-20T09:22:54.176 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.176 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.177 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:22:54.177 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:21.448574+0000", 2024-03-20T09:22:54.177 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.686e-05 2024-03-20T09:22:54.177 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.177 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.177 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:22:54.177 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:21.518525+0000", 2024-03-20T09:22:54.177 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.069950940000000003 2024-03-20T09:22:54.177 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.177 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.177 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:22:54.178 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:21.518567+0000", 2024-03-20T09:22:54.178 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.2030000000000002e-05 2024-03-20T09:22:54.178 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.178 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:22:54.178 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.178 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.178 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.178 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:208841 5.5 e675/637 5:a3eee9ed:::benchmark_data_smithi122_157894_object78854:head v 675'13076, mlcod=623'4831)", 2024-03-20T09:22:54.178 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:23.178048+0000", 2024-03-20T09:22:54.178 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 30.967591203000001, 2024-03-20T09:22:54.178 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.089451347, 2024-03-20T09:22:54.178 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:22:54.178 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:22:54.178 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:22:54.179 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.179 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:22:54.179 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178048+0000", 2024-03-20T09:22:54.179 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.179 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.179 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.179 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:22:54.179 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178048+0000", 2024-03-20T09:22:54.179 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.179 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.179 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.179 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:22:54.179 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178049+0000", 2024-03-20T09:22:54.180 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.153e-06 2024-03-20T09:22:54.180 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.180 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.180 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:22:54.180 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178059+0000", 2024-03-20T09:22:54.180 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0169e-05 2024-03-20T09:22:54.180 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.180 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.180 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:22:54.180 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178076+0000", 2024-03-20T09:22:54.180 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7096e-05 2024-03-20T09:22:54.180 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.180 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.180 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:22:54.181 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178086+0000", 2024-03-20T09:22:54.181 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.6320000000000003e-06 2024-03-20T09:22:54.181 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.181 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.181 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:22:54.181 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178186+0000", 2024-03-20T09:22:54.181 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000100503 2024-03-20T09:22:54.181 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.181 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.181 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:22:54.181 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178215+0000", 2024-03-20T09:22:54.181 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8164e-05 2024-03-20T09:22:54.181 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.181 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.182 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:22:54.182 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.267448+0000", 2024-03-20T09:22:54.182 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.089233281999999997 2024-03-20T09:22:54.182 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.182 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.182 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:22:54.182 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.267499+0000", 2024-03-20T09:22:54.182 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1347999999999997e-05 2024-03-20T09:22:54.182 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.182 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:22:54.182 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.182 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.182 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.183 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:208845 5.5 e675/637 5:ad73ae47:::benchmark_data_smithi122_157894_object78858:head v 675'13077, mlcod=623'4831)", 2024-03-20T09:22:54.183 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:23.178111+0000", 2024-03-20T09:22:54.183 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 30.967528112, 2024-03-20T09:22:54.183 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.089416893999999997, 2024-03-20T09:22:54.183 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:22:54.183 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:22:54.183 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:22:54.183 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.183 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:22:54.183 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178111+0000", 2024-03-20T09:22:54.183 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.183 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.183 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.184 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:22:54.184 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178111+0000", 2024-03-20T09:22:54.184 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.184 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.184 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.184 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:22:54.184 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178111+0000", 2024-03-20T09:22:54.184 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8999999999999998e-07 2024-03-20T09:22:54.184 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.184 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.184 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:22:54.184 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178115+0000", 2024-03-20T09:22:54.184 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7799999999999998e-06 2024-03-20T09:22:54.184 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.185 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.185 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:22:54.185 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178121+0000", 2024-03-20T09:22:54.185 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4219999999999999e-06 2024-03-20T09:22:54.185 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.185 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.185 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:22:54.185 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178125+0000", 2024-03-20T09:22:54.185 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.0740000000000003e-06 2024-03-20T09:22:54.185 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.185 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.185 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:22:54.185 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.194801+0000", 2024-03-20T09:22:54.185 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.016675964000000001 2024-03-20T09:22:54.185 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.186 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.186 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:22:54.186 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.194818+0000", 2024-03-20T09:22:54.186 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7150999999999999e-05 2024-03-20T09:22:54.186 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.186 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.186 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:22:54.186 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.267503+0000", 2024-03-20T09:22:54.186 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.072685657000000001 2024-03-20T09:22:54.186 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.186 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.186 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:22:54.186 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.267528+0000", 2024-03-20T09:22:54.187 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.4556e-05 2024-03-20T09:22:54.187 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.187 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:22:54.187 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.187 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.187 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.187 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:223322 5.5 e682/637 5:a0cca45d:::benchmark_data_smithi122_157894_object93335:head v 682'14007, mlcod=627'5694)", 2024-03-20T09:22:54.187 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:46.235324+0000", 2024-03-20T09:22:54.187 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 7.910315185, 2024-03-20T09:22:54.187 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.088264128999999997, 2024-03-20T09:22:54.187 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:22:54.187 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:22:54.187 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:22:54.187 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.188 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:22:54.188 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235324+0000", 2024-03-20T09:22:54.188 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.188 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.188 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.188 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:22:54.188 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235324+0000", 2024-03-20T09:22:54.188 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.188 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.188 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.188 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:22:54.188 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235324+0000", 2024-03-20T09:22:54.188 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.9700000000000002e-07 2024-03-20T09:22:54.188 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.189 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.189 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:22:54.189 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235329+0000", 2024-03-20T09:22:54.189 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.899e-06 2024-03-20T09:22:54.189 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.189 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.189 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:22:54.189 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235336+0000", 2024-03-20T09:22:54.189 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.9739999999999996e-06 2024-03-20T09:22:54.189 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.189 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.189 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:22:54.189 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235341+0000", 2024-03-20T09:22:54.189 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6999999999999999e-06 2024-03-20T09:22:54.189 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.190 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.190 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:22:54.190 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235536+0000", 2024-03-20T09:22:54.190 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00019468400000000001 2024-03-20T09:22:54.190 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.190 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.190 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:22:54.190 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235559+0000", 2024-03-20T09:22:54.190 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3881000000000001e-05 2024-03-20T09:22:54.190 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.190 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.190 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:22:54.190 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.323553+0000", 2024-03-20T09:22:54.190 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.087993431999999996 2024-03-20T09:22:54.191 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.191 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.191 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:22:54.191 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.323588+0000", 2024-03-20T09:22:54.191 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.5162e-05 2024-03-20T09:22:54.191 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.191 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:22:54.191 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.191 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.191 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.191 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:223326 5.5 e682/637 5:a90b760b:::benchmark_data_smithi122_157894_object93339:head v 682'14008, mlcod=627'5694)", 2024-03-20T09:22:54.191 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:46.235363+0000", 2024-03-20T09:22:54.191 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 7.9102766110000005, 2024-03-20T09:22:54.192 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.088261112000000003, 2024-03-20T09:22:54.192 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:22:54.192 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:22:54.192 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:22:54.192 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.192 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:22:54.192 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235363+0000", 2024-03-20T09:22:54.192 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.192 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.192 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.192 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:22:54.192 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235363+0000", 2024-03-20T09:22:54.192 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.192 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.193 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.193 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:22:54.193 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235363+0000", 2024-03-20T09:22:54.193 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8099999999999999e-07 2024-03-20T09:22:54.193 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.193 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.193 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:22:54.193 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235366+0000", 2024-03-20T09:22:54.193 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.061e-06 2024-03-20T09:22:54.193 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.193 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.193 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:22:54.193 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235371+0000", 2024-03-20T09:22:54.194 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2979999999999999e-06 2024-03-20T09:22:54.194 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.194 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.194 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:22:54.194 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235375+0000", 2024-03-20T09:22:54.194 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.8619999999999996e-06 2024-03-20T09:22:54.194 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.194 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.194 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:22:54.194 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.248068+0000", 2024-03-20T09:22:54.194 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.012692554 2024-03-20T09:22:54.194 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.194 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.195 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:22:54.195 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.248090+0000", 2024-03-20T09:22:54.195 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2747000000000001e-05 2024-03-20T09:22:54.195 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.195 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.195 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:22:54.195 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.323592+0000", 2024-03-20T09:22:54.195 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.075502001999999999 2024-03-20T09:22:54.195 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.195 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.195 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:22:54.195 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.323624+0000", 2024-03-20T09:22:54.195 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.1306999999999997e-05 2024-03-20T09:22:54.196 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.196 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:22:54.196 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.196 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.196 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.196 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:223337 5.5 e682/637 5:a415780d:::benchmark_data_smithi122_157894_object93350:head v 682'14009, mlcod=627'5694)", 2024-03-20T09:22:54.196 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:46.235380+0000", 2024-03-20T09:22:54.196 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 7.9102586380000002, 2024-03-20T09:22:54.196 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.088268737999999999, 2024-03-20T09:22:54.196 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:22:54.196 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:22:54.196 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:22:54.196 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.196 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:22:54.197 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235380+0000", 2024-03-20T09:22:54.197 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.197 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.197 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.197 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:22:54.197 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235380+0000", 2024-03-20T09:22:54.197 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.197 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.197 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.197 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:22:54.197 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235381+0000", 2024-03-20T09:22:54.197 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6999999999999999e-07 2024-03-20T09:22:54.197 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.197 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.198 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:22:54.198 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235387+0000", 2024-03-20T09:22:54.198 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1650000000000003e-06 2024-03-20T09:22:54.198 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.198 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.198 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:22:54.198 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235392+0000", 2024-03-20T09:22:54.198 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8420000000000004e-06 2024-03-20T09:22:54.198 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.198 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.198 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:22:54.198 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235396+0000", 2024-03-20T09:22:54.198 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.8929999999999998e-06 2024-03-20T09:22:54.199 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.199 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.199 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:22:54.199 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.260711+0000", 2024-03-20T09:22:54.199 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.025315500000000001 2024-03-20T09:22:54.199 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.199 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.199 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:22:54.199 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.260729+0000", 2024-03-20T09:22:54.199 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7963000000000001e-05 2024-03-20T09:22:54.199 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.199 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.199 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:22:54.199 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.323627+0000", 2024-03-20T09:22:54.199 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.062897733999999997 2024-03-20T09:22:54.200 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.200 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.200 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:22:54.200 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.323649+0000", 2024-03-20T09:22:54.200 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2470999999999999e-05 2024-03-20T09:22:54.200 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.200 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:22:54.200 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.200 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.200 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.200 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:223338 5.5 e682/637 5:a1f81b44:::benchmark_data_smithi122_157894_object93351:head v 682'14010, mlcod=627'5694)", 2024-03-20T09:22:54.200 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:46.236584+0000", 2024-03-20T09:22:54.200 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 7.9090553520000002, 2024-03-20T09:22:54.200 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.087091848999999999, 2024-03-20T09:22:54.201 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:22:54.201 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:22:54.201 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:22:54.201 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.201 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:22:54.201 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.236584+0000", 2024-03-20T09:22:54.201 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.201 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.201 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.201 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:22:54.201 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.236584+0000", 2024-03-20T09:22:54.201 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.201 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.201 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.202 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:22:54.202 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.236585+0000", 2024-03-20T09:22:54.202 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.2099999999999995e-07 2024-03-20T09:22:54.202 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.202 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.202 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:22:54.202 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.236593+0000", 2024-03-20T09:22:54.202 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.2979999999999992e-06 2024-03-20T09:22:54.202 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.202 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.202 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:22:54.202 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.236607+0000", 2024-03-20T09:22:54.202 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3902e-05 2024-03-20T09:22:54.202 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.203 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.203 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:22:54.203 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.236615+0000", 2024-03-20T09:22:54.203 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.1300000000000001e-06 2024-03-20T09:22:54.203 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.203 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.203 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:22:54.203 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.270998+0000", 2024-03-20T09:22:54.203 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.034383220999999999 2024-03-20T09:22:54.203 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.203 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.203 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:22:54.203 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.271027+0000", 2024-03-20T09:22:54.203 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8649e-05 2024-03-20T09:22:54.204 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.204 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.204 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:22:54.204 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.323653+0000", 2024-03-20T09:22:54.204 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.052625838000000001 2024-03-20T09:22:54.204 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.204 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.204 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:22:54.204 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.323676+0000", 2024-03-20T09:22:54.204 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2889999999999999e-05 2024-03-20T09:22:54.204 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.204 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:22:54.204 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.204 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.204 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.205 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:223309 5.5 e682/637 5:a4c63307:::benchmark_data_smithi122_157894_object93322:head v 682'14011, mlcod=627'5694)", 2024-03-20T09:22:54.205 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:46.248926+0000", 2024-03-20T09:22:54.205 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 7.896712945, 2024-03-20T09:22:54.205 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.094989770000000001, 2024-03-20T09:22:54.205 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:22:54.205 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:22:54.205 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:22:54.205 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.205 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:22:54.205 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.248926+0000", 2024-03-20T09:22:54.205 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.205 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.205 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.205 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:22:54.206 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.248926+0000", 2024-03-20T09:22:54.206 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.206 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.206 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.206 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:22:54.206 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.248927+0000", 2024-03-20T09:22:54.206 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0440000000000001e-06 2024-03-20T09:22:54.206 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.206 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.206 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:22:54.206 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.248938+0000", 2024-03-20T09:22:54.206 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0560000000000001e-05 2024-03-20T09:22:54.206 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.206 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.207 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:22:54.207 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.248952+0000", 2024-03-20T09:22:54.207 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4297000000000001e-05 2024-03-20T09:22:54.207 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.207 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.207 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:22:54.207 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.248964+0000", 2024-03-20T09:22:54.207 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1912e-05 2024-03-20T09:22:54.207 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.207 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.207 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:22:54.207 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.281776+0000", 2024-03-20T09:22:54.207 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.032812458000000003 2024-03-20T09:22:54.207 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.208 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.208 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:22:54.208 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.281805+0000", 2024-03-20T09:22:54.208 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8765e-05 2024-03-20T09:22:54.208 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.208 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.208 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:22:54.208 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.343870+0000", 2024-03-20T09:22:54.208 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.062064805000000001 2024-03-20T09:22:54.208 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.208 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.208 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:22:54.208 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.343916+0000", 2024-03-20T09:22:54.209 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5929e-05 2024-03-20T09:22:54.209 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.209 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:22:54.209 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.209 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.209 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.209 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:223346 5.5 e682/637 5:a014a865:::benchmark_data_smithi122_157894_object93359:head v 682'14012, mlcod=627'5694)", 2024-03-20T09:22:54.209 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:46.254399+0000", 2024-03-20T09:22:54.209 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 7.8912396989999998, 2024-03-20T09:22:54.209 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.089556495, 2024-03-20T09:22:54.209 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:22:54.209 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:22:54.209 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:22:54.209 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.210 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:22:54.210 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.254399+0000", 2024-03-20T09:22:54.210 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.210 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.210 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.210 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:22:54.210 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.254399+0000", 2024-03-20T09:22:54.210 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.210 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.210 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.210 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:22:54.210 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.254400+0000", 2024-03-20T09:22:54.210 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9599999999999999e-07 2024-03-20T09:22:54.210 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.211 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.211 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:22:54.211 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.254405+0000", 2024-03-20T09:22:54.211 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.011e-06 2024-03-20T09:22:54.211 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.211 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.211 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:22:54.211 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.254415+0000", 2024-03-20T09:22:54.211 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.9790000000000005e-06 2024-03-20T09:22:54.211 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.211 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.211 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:22:54.211 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.254420+0000", 2024-03-20T09:22:54.211 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2340000000000002e-06 2024-03-20T09:22:54.212 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.212 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.212 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:22:54.212 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.292675+0000", 2024-03-20T09:22:54.212 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.038254945999999998 2024-03-20T09:22:54.212 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.212 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.212 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:22:54.212 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.292700+0000", 2024-03-20T09:22:54.212 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5173000000000001e-05 2024-03-20T09:22:54.212 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.212 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.212 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:22:54.212 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.343920+0000", 2024-03-20T09:22:54.213 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.051219555 2024-03-20T09:22:54.213 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.213 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.213 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:22:54.213 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.343956+0000", 2024-03-20T09:22:54.213 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6001000000000003e-05 2024-03-20T09:22:54.213 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.213 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:22:54.213 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.213 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.213 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.213 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "MOSDPGPush(5.5 682/637 [PushOp(5:a2e10a16:::benchmark_data_smithi122_157894_object93370:head, version: 628'5825, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:a2e10a16:::benchmark_data_smithi122_157894_object93370:head@628'5825, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-20T09:22:54.213 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:46.279959+0000", 2024-03-20T09:22:54.213 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 7.8656800440000003, 2024-03-20T09:22:54.214 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.078914829000000006, 2024-03-20T09:22:54.214 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:22:54.214 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:22:54.214 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:22:54.214 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.214 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:22:54.214 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.279959+0000", 2024-03-20T09:22:54.214 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.214 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.214 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.214 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:22:54.214 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.279959+0000", 2024-03-20T09:22:54.214 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.214 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.215 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.215 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:22:54.215 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.279960+0000", 2024-03-20T09:22:54.215 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5000000000000001e-07 2024-03-20T09:22:54.215 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.215 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.215 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:22:54.215 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.280020+0000", 2024-03-20T09:22:54.215 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0139000000000001e-05 2024-03-20T09:22:54.215 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.215 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.215 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:22:54.215 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.280040+0000", 2024-03-20T09:22:54.215 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9996999999999998e-05 2024-03-20T09:22:54.215 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.216 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.216 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:22:54.216 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.280062+0000", 2024-03-20T09:22:54.216 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2016000000000001e-05 2024-03-20T09:22:54.216 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.216 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.216 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:22:54.216 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.355287+0000", 2024-03-20T09:22:54.216 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.075224869999999999 2024-03-20T09:22:54.216 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.216 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.216 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:22:54.216 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.355303+0000", 2024-03-20T09:22:54.216 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6127e-05 2024-03-20T09:22:54.217 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.217 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.217 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:22:54.217 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.358874+0000", 2024-03-20T09:22:54.217 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0035708300000000001 2024-03-20T09:22:54.217 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.217 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:22:54.217 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.217 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.217 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.217 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:223777 5.f e685/637 5:febd000c:::benchmark_data_smithi122_157894_object93790:head v 685'13980, mlcod=628'5880)", 2024-03-20T09:22:54.217 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:47.826061+0000", 2024-03-20T09:22:54.217 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 6.319578516, 2024-03-20T09:22:54.217 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.078648409000000002, 2024-03-20T09:22:54.218 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:22:54.218 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:22:54.218 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:22:54.218 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.218 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:22:54.218 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.826061+0000", 2024-03-20T09:22:54.218 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.218 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.218 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.218 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:22:54.218 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.826061+0000", 2024-03-20T09:22:54.218 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.218 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.218 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.219 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:22:54.219 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.826061+0000", 2024-03-20T09:22:54.219 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.03e-07 2024-03-20T09:22:54.219 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.219 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.219 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:22:54.219 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.826066+0000", 2024-03-20T09:22:54.219 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4110000000000004e-06 2024-03-20T09:22:54.219 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.219 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.219 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:22:54.219 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.826072+0000", 2024-03-20T09:22:54.219 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1430000000000003e-06 2024-03-20T09:22:54.219 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.220 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.220 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:22:54.220 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.826077+0000", 2024-03-20T09:22:54.220 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.2930000000000002e-06 2024-03-20T09:22:54.220 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.220 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.220 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:22:54.220 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.869483+0000", 2024-03-20T09:22:54.220 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.043406004999999998 2024-03-20T09:22:54.220 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.220 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.220 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:22:54.220 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.869505+0000", 2024-03-20T09:22:54.221 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2821000000000001e-05 2024-03-20T09:22:54.221 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.221 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.221 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:22:54.221 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.904658+0000", 2024-03-20T09:22:54.221 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.035152950000000002 2024-03-20T09:22:54.221 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.221 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.221 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:22:54.221 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.904709+0000", 2024-03-20T09:22:54.221 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0583e-05 2024-03-20T09:22:54.221 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.221 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:22:54.222 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.222 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.222 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.222 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:223799 5.f e685/637 5:f4174b5e:::benchmark_data_smithi122_157894_object93812:head v 685'13981, mlcod=628'5880)", 2024-03-20T09:22:54.222 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:47.826083+0000", 2024-03-20T09:22:54.222 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 6.3195563259999998, 2024-03-20T09:22:54.222 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.078665499, 2024-03-20T09:22:54.222 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:22:54.222 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:22:54.222 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:22:54.222 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.222 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:22:54.222 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.826083+0000", 2024-03-20T09:22:54.222 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.223 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.223 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.223 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:22:54.223 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.826083+0000", 2024-03-20T09:22:54.223 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.223 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.223 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.223 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:22:54.223 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.826083+0000", 2024-03-20T09:22:54.223 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.4999999999999999e-07 2024-03-20T09:22:54.223 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.223 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.223 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:22:54.223 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.826086+0000", 2024-03-20T09:22:54.224 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.3050000000000001e-06 2024-03-20T09:22:54.224 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.224 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.224 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:22:54.224 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.826092+0000", 2024-03-20T09:22:54.224 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8440000000000003e-06 2024-03-20T09:22:54.224 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.224 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.224 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:22:54.224 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.826097+0000", 2024-03-20T09:22:54.224 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5510000000000001e-06 2024-03-20T09:22:54.224 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.224 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.225 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:22:54.225 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.879519+0000", 2024-03-20T09:22:54.225 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.053422284 2024-03-20T09:22:54.225 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.225 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.225 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:22:54.225 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.879543+0000", 2024-03-20T09:22:54.225 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.4158000000000001e-05 2024-03-20T09:22:54.225 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.225 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.225 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:22:54.225 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.904716+0000", 2024-03-20T09:22:54.225 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.025173017999999998 2024-03-20T09:22:54.225 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.226 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.226 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:22:54.226 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.904748+0000", 2024-03-20T09:22:54.226 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.2088999999999997e-05 2024-03-20T09:22:54.226 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.226 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:22:54.226 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.226 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.226 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.226 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:223816 5.f e685/637 5:fd8e4678:::benchmark_data_smithi122_157894_object93829:head v 685'13982, mlcod=628'5880)", 2024-03-20T09:22:54.226 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:47.826104+0000", 2024-03-20T09:22:54.226 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 6.3195354220000004, 2024-03-20T09:22:54.226 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.084716275999999993, 2024-03-20T09:22:54.226 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:22:54.227 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:22:54.227 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:22:54.227 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.227 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:22:54.227 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.826104+0000", 2024-03-20T09:22:54.227 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.227 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.227 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.227 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:22:54.227 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.826104+0000", 2024-03-20T09:22:54.227 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.227 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.227 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.227 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:22:54.227 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.826104+0000", 2024-03-20T09:22:54.228 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0599999999999999e-07 2024-03-20T09:22:54.228 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.228 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.228 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:22:54.228 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.826110+0000", 2024-03-20T09:22:54.228 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6350000000000003e-06 2024-03-20T09:22:54.228 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.228 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.228 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:22:54.228 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.826115+0000", 2024-03-20T09:22:54.228 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9109999999999998e-06 2024-03-20T09:22:54.228 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.228 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.228 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:22:54.229 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.826120+0000", 2024-03-20T09:22:54.229 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4490000000000001e-06 2024-03-20T09:22:54.229 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.229 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.229 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:22:54.229 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.886066+0000", 2024-03-20T09:22:54.229 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.059945750999999999 2024-03-20T09:22:54.229 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.229 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.229 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:22:54.229 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.886103+0000", 2024-03-20T09:22:54.229 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7614999999999998e-05 2024-03-20T09:22:54.229 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.229 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.230 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:22:54.230 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.910770+0000", 2024-03-20T09:22:54.230 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.024666535999999999 2024-03-20T09:22:54.230 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.230 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.230 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:22:54.230 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.910820+0000", 2024-03-20T09:22:54.230 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0173000000000002e-05 2024-03-20T09:22:54.230 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.230 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:22:54.230 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.230 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.230 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.230 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "MOSDPGPush(5.f 685/637 [PushOp(5:f1254502:::benchmark_data_smithi122_157894_object94700:head, version: 628'5882, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:f1254502:::benchmark_data_smithi122_157894_object94700:head@628'5882, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-20T09:22:54.231 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:47.826130+0000", 2024-03-20T09:22:54.231 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 6.3195095200000004, 2024-03-20T09:22:54.231 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.088800740000000003, 2024-03-20T09:22:54.231 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:22:54.231 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:22:54.231 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:22:54.231 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.231 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:22:54.231 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.826130+0000", 2024-03-20T09:22:54.231 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.231 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.231 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.231 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:22:54.231 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.826130+0000", 2024-03-20T09:22:54.231 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.232 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.232 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.232 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:22:54.232 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.826130+0000", 2024-03-20T09:22:54.232 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.0800000000000001e-07 2024-03-20T09:22:54.232 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.232 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.232 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:22:54.232 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.826161+0000", 2024-03-20T09:22:54.232 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.1158999999999998e-05 2024-03-20T09:22:54.232 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.232 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.232 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:22:54.232 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.826187+0000", 2024-03-20T09:22:54.233 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5978000000000001e-05 2024-03-20T09:22:54.233 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.233 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.233 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:22:54.233 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.826204+0000", 2024-03-20T09:22:54.233 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6501e-05 2024-03-20T09:22:54.233 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.233 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.233 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:22:54.233 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.910972+0000", 2024-03-20T09:22:54.233 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.084768204999999999 2024-03-20T09:22:54.233 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.233 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.234 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:22:54.234 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.910987+0000", 2024-03-20T09:22:54.234 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5277999999999998e-05 2024-03-20T09:22:54.234 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.234 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.234 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:22:54.234 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.914930+0000", 2024-03-20T09:22:54.234 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0039433109999999997 2024-03-20T09:22:54.234 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.234 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:22:54.234 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.234 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.234 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.235 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:224041 5.5 e685/637 5:a0d414bc:::benchmark_data_smithi122_157894_object94054:head v 685'14057, mlcod=628'5888)", 2024-03-20T09:22:54.235 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:48.165764+0000", 2024-03-20T09:22:54.235 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 5.9798751909999996, 2024-03-20T09:22:54.235 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.084626451000000005, 2024-03-20T09:22:54.235 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:22:54.235 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:22:54.235 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:22:54.235 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.235 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:22:54.235 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:48.165764+0000", 2024-03-20T09:22:54.235 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.235 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.235 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.235 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:22:54.236 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:48.165764+0000", 2024-03-20T09:22:54.236 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:54.236 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.236 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.236 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:22:54.236 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:48.165764+0000", 2024-03-20T09:22:54.236 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.36e-07 2024-03-20T09:22:54.236 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.236 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.236 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:22:54.236 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:48.165768+0000", 2024-03-20T09:22:54.236 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.5980000000000001e-06 2024-03-20T09:22:54.236 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.237 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.237 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:22:54.237 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:48.165773+0000", 2024-03-20T09:22:54.237 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.826e-06 2024-03-20T09:22:54.237 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.237 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.237 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:22:54.237 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:48.165777+0000", 2024-03-20T09:22:54.237 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.9829999999999998e-06 2024-03-20T09:22:54.237 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.237 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.237 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:22:54.237 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:48.188256+0000", 2024-03-20T09:22:54.237 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.022479760000000001 2024-03-20T09:22:54.237 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.238 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.238 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:22:54.238 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:48.188277+0000", 2024-03-20T09:22:54.238 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0965999999999999e-05 2024-03-20T09:22:54.238 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.238 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.238 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:22:54.238 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:48.250334+0000", 2024-03-20T09:22:54.238 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.062056300000000002 2024-03-20T09:22:54.238 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:54.238 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:54.238 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:22:54.238 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:48.250390+0000", 2024-03-20T09:22:54.239 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6781999999999999e-05 2024-03-20T09:22:54.239 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.239 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:22:54.239 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.239 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:54.239 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:22:54.239 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:22:54.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:54.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:22:54.341 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:22:54.396 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:54.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:54 smithi026 ceph-mon[68309]: pgmap v1676: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 625 op/s; 7213/100248 objects degraded (7.195%); 8800/100248 objects misplaced (8.778%); 11 MiB/s, 177 objects/s recovering 2024-03-20T09:22:54.558 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:54 smithi115 ceph-mon[72476]: pgmap v1676: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 625 op/s; 7213/100248 objects degraded (7.195%); 8800/100248 objects misplaced (8.778%); 11 MiB/s, 177 objects/s recovering 2024-03-20T09:22:54.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:54 smithi122 ceph-mon[73495]: pgmap v1676: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 625 op/s; 7213/100248 objects degraded (7.195%); 8800/100248 objects misplaced (8.778%); 11 MiB/s, 177 objects/s recovering 2024-03-20T09:22:54.960 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-20T09:22:54.984 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:22:54.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:54.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:22:55.091 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:22:55.180 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:55.223 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:22:55.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:55.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:22:55.332 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:22:55.436 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:55.447 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:22:56.507 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:22:56.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:56.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:22:56.615 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:22:56.692 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:56.921 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:22:56.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:56.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:22:57.031 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:22:57.194 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:57.200 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:57 smithi026 ceph-mon[68309]: pgmap v1677: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 610 op/s; 7213/97305 objects degraded (7.413%); 8597/97305 objects misplaced (8.835%); 9.5 MiB/s, 152 objects/s recovering 2024-03-20T09:22:57.200 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:57 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:22:57.200 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:57 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:22:57.460 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:57 smithi115 ceph-mon[72476]: pgmap v1677: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 610 op/s; 7213/97305 objects degraded (7.413%); 8597/97305 objects misplaced (8.835%); 9.5 MiB/s, 152 objects/s recovering 2024-03-20T09:22:57.460 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:57 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:22:57.461 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:57 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:22:57.618 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:22:57.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:57 smithi122 ceph-mon[73495]: pgmap v1677: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 610 op/s; 7213/97305 objects degraded (7.413%); 8597/97305 objects misplaced (8.835%); 9.5 MiB/s, 152 objects/s recovering 2024-03-20T09:22:57.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:57 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:22:57.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:57 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:22:58.310 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:58 smithi026 ceph-mon[68309]: osdmap e688: 12 total, 12 up, 12 in 2024-03-20T09:22:58.311 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:58 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 7213/97305 objects degraded (7.413%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-20T09:22:58.311 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:58 smithi026 ceph-mon[68309]: osdmap e689: 12 total, 12 up, 12 in 2024-03-20T09:22:58.550 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:58 smithi115 ceph-mon[72476]: osdmap e688: 12 total, 12 up, 12 in 2024-03-20T09:22:58.550 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:58 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 7213/97305 objects degraded (7.413%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-20T09:22:58.550 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:58 smithi115 ceph-mon[72476]: osdmap e689: 12 total, 12 up, 12 in 2024-03-20T09:22:58.564 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:22:58.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:22:58.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:22:58.684 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:22:58.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:58 smithi122 ceph-mon[73495]: osdmap e688: 12 total, 12 up, 12 in 2024-03-20T09:22:58.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:58 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 7213/97305 objects degraded (7.413%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-20T09:22:58.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:58 smithi122 ceph-mon[73495]: osdmap e689: 12 total, 12 up, 12 in 2024-03-20T09:22:58.751 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:22:58.974 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:22:59.234 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.8 to 1 (10000) 2024-03-20T09:22:59.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:59 smithi026 ceph-mon[68309]: pgmap v1680: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 611 op/s; 7241/94248 objects degraded (7.683%); 8394/94248 objects misplaced (8.906%); 8.5 MiB/s, 137 objects/s recovering 2024-03-20T09:22:59.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:59 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/875225097' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:22:59.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:22:59 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:22:59.511 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:22:59.511 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [ 2024-03-20T09:22:59.512 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.512 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.25966.0:231351 5.9 5:9562aab7:::benchmark_data_smithi122_157894_object101364:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e690)", 2024-03-20T09:22:59.512 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:59.479066+0000", 2024-03-20T09:22:59.512 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.023220574000000001, 2024-03-20T09:22:59.512 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.023257888000000001, 2024-03-20T09:22:59.512 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:22:59.512 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:22:59.512 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:22:59.512 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.25966", 2024-03-20T09:22:59.512 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:22:59.512 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 231351 2024-03-20T09:22:59.513 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:59.513 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:22:59.513 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.513 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:22:59.513 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.479066+0000", 2024-03-20T09:22:59.513 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:59.513 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:59.513 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.513 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:22:59.513 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.479066+0000", 2024-03-20T09:22:59.513 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:59.513 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:59.513 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.514 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:22:59.514 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.479076+0000", 2024-03-20T09:22:59.514 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.4909999999999994e-06 2024-03-20T09:22:59.514 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:59.514 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.514 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:22:59.514 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.479083+0000", 2024-03-20T09:22:59.514 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.4919999999999996e-06 2024-03-20T09:22:59.514 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:59.514 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.514 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:22:59.514 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.479103+0000", 2024-03-20T09:22:59.515 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9589000000000002e-05 2024-03-20T09:22:59.515 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:59.515 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.515 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:22:59.515 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.479112+0000", 2024-03-20T09:22:59.515 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.5389999999999996e-06 2024-03-20T09:22:59.515 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:59.515 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.515 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:22:59.515 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.479192+0000", 2024-03-20T09:22:59.515 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.9569999999999999e-05 2024-03-20T09:22:59.515 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:59.515 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.516 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:22:59.516 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.479324+0000", 2024-03-20T09:22:59.516 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013181799999999999 2024-03-20T09:22:59.516 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:59.516 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.516 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,11", 2024-03-20T09:22:59.516 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.479442+0000", 2024-03-20T09:22:59.516 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00011859299999999999 2024-03-20T09:22:59.516 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:59.516 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.516 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:22:59.516 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.489695+0000", 2024-03-20T09:22:59.516 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.010252908999999999 2024-03-20T09:22:59.516 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:59.517 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:22:59.517 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:59.517 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:59.517 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.517 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.25966.0:231353 5.9 5:94a3c301:::benchmark_data_smithi122_157894_object101366:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e690)", 2024-03-20T09:22:59.517 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:59.479138+0000", 2024-03-20T09:22:59.517 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.023148561000000002, 2024-03-20T09:22:59.517 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.023275548, 2024-03-20T09:22:59.517 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:22:59.517 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:22:59.517 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:22:59.517 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.25966", 2024-03-20T09:22:59.517 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:22:59.517 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 231353 2024-03-20T09:22:59.518 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:59.518 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:22:59.518 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.518 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:22:59.518 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.479138+0000", 2024-03-20T09:22:59.518 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:59.518 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:59.518 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.518 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:22:59.518 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.479138+0000", 2024-03-20T09:22:59.518 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:59.518 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:59.519 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.519 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:22:59.519 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.479139+0000", 2024-03-20T09:22:59.519 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2699999999999999e-07 2024-03-20T09:22:59.519 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:59.519 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.519 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:22:59.519 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.479142+0000", 2024-03-20T09:22:59.519 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.033e-06 2024-03-20T09:22:59.519 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:59.519 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.519 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:22:59.519 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.479148+0000", 2024-03-20T09:22:59.520 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.6760000000000001e-06 2024-03-20T09:22:59.520 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:59.520 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.520 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:22:59.520 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.479154+0000", 2024-03-20T09:22:59.520 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2730000000000002e-06 2024-03-20T09:22:59.520 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:59.520 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.520 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:22:59.520 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.484377+0000", 2024-03-20T09:22:59.520 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0052234919999999997 2024-03-20T09:22:59.520 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:59.520 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.520 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:22:59.521 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.484515+0000", 2024-03-20T09:22:59.521 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013808200000000001 2024-03-20T09:22:59.521 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:59.521 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.521 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,11", 2024-03-20T09:22:59.521 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.484654+0000", 2024-03-20T09:22:59.521 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000138947 2024-03-20T09:22:59.521 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:59.521 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.521 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:22:59.521 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.499509+0000", 2024-03-20T09:22:59.521 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.014855278 2024-03-20T09:22:59.521 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:59.522 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:22:59.522 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:59.522 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:59.522 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.522 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:231358 5.3 e690/605 5:c2862c4e:::benchmark_data_smithi122_157894_object101371:head v 690'14515, mlcod=690'14513)", 2024-03-20T09:22:59.522 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:59.483989+0000", 2024-03-20T09:22:59.522 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.018297668999999999, 2024-03-20T09:22:59.522 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.018507281, 2024-03-20T09:22:59.522 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:22:59.522 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:22:59.522 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:22:59.522 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.522 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:22:59.523 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.483989+0000", 2024-03-20T09:22:59.523 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:59.523 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:59.523 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.523 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:22:59.523 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.483989+0000", 2024-03-20T09:22:59.523 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:59.523 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:59.523 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.523 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:22:59.523 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.483990+0000", 2024-03-20T09:22:59.523 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2699999999999999e-06 2024-03-20T09:22:59.523 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:59.523 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.524 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:22:59.524 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.484000+0000", 2024-03-20T09:22:59.524 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.7540000000000001e-06 2024-03-20T09:22:59.524 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:59.524 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.524 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:22:59.524 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.484020+0000", 2024-03-20T09:22:59.524 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9995000000000001e-05 2024-03-20T09:22:59.524 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:59.524 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.524 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:22:59.524 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.484029+0000", 2024-03-20T09:22:59.524 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.3279999999999995e-06 2024-03-20T09:22:59.524 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:59.525 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.525 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:22:59.525 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.484103+0000", 2024-03-20T09:22:59.525 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3708000000000006e-05 2024-03-20T09:22:59.525 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:59.525 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.525 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:22:59.525 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.484131+0000", 2024-03-20T09:22:59.525 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.7419999999999998e-05 2024-03-20T09:22:59.525 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:59.525 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:22:59.525 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:59.525 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:59.525 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.526 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:231365 5.3 e690/605 5:cd4fc376:::benchmark_data_smithi122_157894_object101378:head v 690'14516, mlcod=690'14513)", 2024-03-20T09:22:59.526 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:59.499372+0000", 2024-03-20T09:22:59.526 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.0029147180000000002, 2024-03-20T09:22:59.526 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0031836989999999999, 2024-03-20T09:22:59.526 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:22:59.526 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:22:59.526 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:22:59.526 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.526 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:22:59.526 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.499372+0000", 2024-03-20T09:22:59.526 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:59.526 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:59.526 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.527 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:22:59.527 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.499372+0000", 2024-03-20T09:22:59.527 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:22:59.527 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:59.527 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.527 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:22:59.527 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.499373+0000", 2024-03-20T09:22:59.527 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1680000000000001e-06 2024-03-20T09:22:59.527 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:59.527 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.527 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:22:59.527 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.499383+0000", 2024-03-20T09:22:59.527 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0241000000000001e-05 2024-03-20T09:22:59.528 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:59.528 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.528 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:22:59.528 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.499399+0000", 2024-03-20T09:22:59.528 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5311999999999998e-05 2024-03-20T09:22:59.528 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:59.528 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.528 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:22:59.528 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.499408+0000", 2024-03-20T09:22:59.528 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.4460000000000007e-06 2024-03-20T09:22:59.528 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:59.528 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.528 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:22:59.528 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.499547+0000", 2024-03-20T09:22:59.529 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000139312 2024-03-20T09:22:59.529 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:22:59.529 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:22:59.529 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:22:59.529 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.499576+0000", 2024-03-20T09:22:59.529 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8489000000000001e-05 2024-03-20T09:22:59.529 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:59.529 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:22:59.529 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:59.529 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:22:59.529 INFO:teuthology.orchestra.run.smithi115.stdout: ], 2024-03-20T09:22:59.529 INFO:teuthology.orchestra.run.smithi115.stdout: "num_ops": 4 2024-03-20T09:22:59.529 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:22:59.530 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:59 smithi115 ceph-mon[72476]: pgmap v1680: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 611 op/s; 7241/94248 objects degraded (7.683%); 8394/94248 objects misplaced (8.906%); 8.5 MiB/s, 137 objects/s recovering 2024-03-20T09:22:59.530 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:59 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/875225097' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:22:59.530 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:22:59 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:22:59.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:59 smithi122 ceph-mon[73495]: pgmap v1680: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 611 op/s; 7241/94248 objects degraded (7.683%); 8394/94248 objects misplaced (8.906%); 8.5 MiB/s, 137 objects/s recovering 2024-03-20T09:22:59.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:59 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/875225097' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:22:59.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:22:59 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:23:00.279 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-20T09:23:00.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:00 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-03-20T09:23:00.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:00 smithi115 ceph-mon[72476]: osdmap e690: 12 total, 12 up, 12 in 2024-03-20T09:23:00.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:00 smithi115 ceph-mon[72476]: pgmap v1682: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 578 op/s; 7348/89826 objects degraded (8.180%); 7397/89826 objects misplaced (8.235%); 9.0 MiB/s, 145 objects/s recovering 2024-03-20T09:23:00.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:00 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-03-20T09:23:00.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:00 smithi122 ceph-mon[73495]: osdmap e690: 12 total, 12 up, 12 in 2024-03-20T09:23:00.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:00 smithi122 ceph-mon[73495]: pgmap v1682: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 578 op/s; 7348/89826 objects degraded (8.180%); 7397/89826 objects misplaced (8.235%); 9.0 MiB/s, 145 objects/s recovering 2024-03-20T09:23:00.715 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:23:00.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:00 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-03-20T09:23:00.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:00 smithi026 ceph-mon[68309]: osdmap e690: 12 total, 12 up, 12 in 2024-03-20T09:23:00.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:00 smithi026 ceph-mon[68309]: pgmap v1682: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.8 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 578 op/s; 7348/89826 objects degraded (8.180%); 7397/89826 objects misplaced (8.235%); 9.0 MiB/s, 145 objects/s recovering 2024-03-20T09:23:01.551 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:23:01.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:01.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:23:01.658 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:23:01.723 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:02.115 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:01 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:23:02.115 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:01 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:23:02.115 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:01 smithi026 ceph-mon[68309]: osdmap e691: 12 total, 12 up, 12 in 2024-03-20T09:23:02.157 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 10 1 2024-03-20T09:23:02.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:01 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:23:02.158 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:01 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:23:02.158 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:01 smithi115 ceph-mon[72476]: osdmap e691: 12 total, 12 up, 12 in 2024-03-20T09:23:02.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:01 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:23:02.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:01 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:23:02.197 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:01 smithi122 ceph-mon[73495]: osdmap e691: 12 total, 12 up, 12 in 2024-03-20T09:23:02.198 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:23:02.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:02.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:23:02.305 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:23:02.380 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:02.598 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:23:02.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:02.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:23:02.639 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:23:02.642 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:23:02.704 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:23:02.756 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:02.942 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:23:02.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:02.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:23:03.049 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:23:03.126 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:02 smithi115 ceph-mon[72476]: pgmap v1684: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 666 op/s; 7348/85596 objects degraded (8.585%); 7397/85596 objects misplaced (8.642%); 8.5 MiB/s, 136 objects/s recovering 2024-03-20T09:23:03.127 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:02 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 7348/89826 objects degraded (8.180%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-20T09:23:03.131 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:02 smithi026 ceph-mon[68309]: pgmap v1684: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 666 op/s; 7348/85596 objects degraded (8.585%); 7397/85596 objects misplaced (8.642%); 8.5 MiB/s, 136 objects/s recovering 2024-03-20T09:23:03.131 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:02 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 7348/89826 objects degraded (8.180%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-20T09:23:03.132 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:03.140 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:23:03.141 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:23:03.141 INFO:teuthology.orchestra.run.smithi115.stdout: "complaint_time": 30, 2024-03-20T09:23:03.141 INFO:teuthology.orchestra.run.smithi115.stdout: "num_blocked_ops": 0 2024-03-20T09:23:03.141 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:23:03.190 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:23:03.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:02 smithi122 ceph-mon[73495]: pgmap v1684: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 666 op/s; 7348/85596 objects degraded (8.585%); 7397/85596 objects misplaced (8.642%); 8.5 MiB/s, 136 objects/s recovering 2024-03-20T09:23:03.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:02 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 7348/89826 objects degraded (8.180%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-20T09:23:03.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:03.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:23:03.299 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:23:03.329 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:03.352 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:23:03.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:03.356 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:23:03.457 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:23:03.529 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:03.571 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:23:03.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:03.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:23:03.684 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:23:03.756 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:03.899 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:23:03.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:03.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:23:03.929 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-20T09:23:04.010 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:23:04.038 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:04.063 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:23:04.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:04.066 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:23:04.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:03 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1159449407' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:23:04.167 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:23:04.168 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:03 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1159449407' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:23:04.195 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:03 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1159449407' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:23:04.197 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:04.220 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:23:04.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:04.223 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:23:04.325 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:23:04.387 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:04.428 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:23:04.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:04.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:23:04.448 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:23:04.542 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:23:04.626 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:04.923 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:04 smithi026 ceph-mon[68309]: pgmap v1685: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 724 op/s; 7496/80688 objects degraded (9.290%); 6380/80688 objects misplaced (7.907%); 10 MiB/s, 168 objects/s recovering 2024-03-20T09:23:05.161 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:04 smithi115 ceph-mon[72476]: pgmap v1685: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 724 op/s; 7496/80688 objects degraded (9.290%); 6380/80688 objects misplaced (7.907%); 10 MiB/s, 168 objects/s recovering 2024-03-20T09:23:05.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:04 smithi122 ceph-mon[73495]: pgmap v1685: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 724 op/s; 7496/80688 objects degraded (9.290%); 6380/80688 objects misplaced (7.907%); 10 MiB/s, 168 objects/s recovering 2024-03-20T09:23:05.249 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:23:05.250 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:23:05.250 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:23:05.250 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:23:05.250 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:23:05.250 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:23:05.250 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:23:05.250 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:23:05.250 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:23:05.250 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:23:05.250 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:23:05.250 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:23:05.251 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:23:05.251 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:23:05.251 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:23:05.251 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:23:05.251 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:23:05.251 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:23:05.251 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:23:05.251 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:23:05.251 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:23:05.251 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:23:05.252 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:23:05.252 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:23:07.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:07 smithi122 ceph-mon[73495]: pgmap v1686: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 689 op/s; 7496/77691 objects degraded (9.648%); 6171/77691 objects misplaced (7.943%); 9.1 MiB/s, 146 objects/s recovering 2024-03-20T09:23:07.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:07 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:23:07.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:07 smithi026 ceph-mon[68309]: pgmap v1686: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 689 op/s; 7496/77691 objects degraded (9.648%); 6171/77691 objects misplaced (7.943%); 9.1 MiB/s, 146 objects/s recovering 2024-03-20T09:23:07.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:07 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:23:07.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:07 smithi115 ceph-mon[72476]: pgmap v1686: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 689 op/s; 7496/77691 objects degraded (9.648%); 6171/77691 objects misplaced (7.943%); 9.1 MiB/s, 146 objects/s recovering 2024-03-20T09:23:07.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:07 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:23:07.941 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:23:07.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:07.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:23:08.049 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:23:08.114 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:08.390 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:08 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 7496/77691 objects degraded (9.648%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-20T09:23:08.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:08 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 7496/77691 objects degraded (9.648%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-20T09:23:08.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:08 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 7496/77691 objects degraded (9.648%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-20T09:23:08.942 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:23:08.942 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:23:09.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:09 smithi026 ceph-mon[68309]: pgmap v1687: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 589 op/s; 7515/74685 objects degraded (10.062%); 5974/74685 objects misplaced (7.999%); 7.4 MiB/s, 119 objects/s recovering 2024-03-20T09:23:09.227 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:09 smithi115 ceph-mon[72476]: pgmap v1687: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 589 op/s; 7515/74685 objects degraded (10.062%); 5974/74685 objects misplaced (7.999%); 7.4 MiB/s, 119 objects/s recovering 2024-03-20T09:23:09.403 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:23:09.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:09.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:23:09.513 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:23:09.546 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:09 smithi122 ceph-mon[73495]: pgmap v1687: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 589 op/s; 7515/74685 objects degraded (10.062%); 5974/74685 objects misplaced (7.999%); 7.4 MiB/s, 119 objects/s recovering 2024-03-20T09:23:09.546 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:09.566 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:23:09.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:09.570 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:23:09.671 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:23:09.723 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:09.745 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-20T09:23:09.926 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:23:10.093 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:23:10.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:10.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:23:10.200 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:23:10.229 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:10.256 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:23:10.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:10.258 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:23:10.259 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:23:10.360 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:23:10.394 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:10.419 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:23:10.426 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:10.427 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:23:10.528 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:23:10.564 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:10.586 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:23:10.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:10.590 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:23:10.691 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:23:10.735 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:11.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:11 smithi026 ceph-mon[68309]: pgmap v1688: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 647 op/s; 7652/70386 objects degraded (10.871%); 4863/70386 objects misplaced (6.909%); 8.7 MiB/s, 140 objects/s recovering 2024-03-20T09:23:11.661 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:11 smithi115 ceph-mon[72476]: pgmap v1688: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 647 op/s; 7652/70386 objects degraded (10.871%); 4863/70386 objects misplaced (6.909%); 8.7 MiB/s, 140 objects/s recovering 2024-03-20T09:23:11.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:11 smithi122 ceph-mon[73495]: pgmap v1688: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 647 op/s; 7652/70386 objects degraded (10.871%); 4863/70386 objects misplaced (6.909%); 8.7 MiB/s, 140 objects/s recovering 2024-03-20T09:23:12.430 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:23:12.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:12.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:23:12.535 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:23:12.559 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:23:12.559 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:23:12.568 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:12.599 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:23:12.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:12.602 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:23:12.702 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:23:12.750 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:23:12.752 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:12.814 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:23:12.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:12.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:23:12.926 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:23:12.998 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:13.252 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.10 to 1 (10000) 2024-03-20T09:23:13.274 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:23:13.290 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:13 smithi026 ceph-mon[68309]: pgmap v1689: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 646 op/s; 7652/66252 objects degraded (11.550%); 4863/66252 objects misplaced (7.340%); 8.4 MiB/s, 136 objects/s recovering 2024-03-20T09:23:13.290 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:13 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 7652/66252 objects degraded (11.550%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-20T09:23:13.290 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:13 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2856212429' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-03-20T09:23:13.290 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:13 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-03-20T09:23:13.290 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:13 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1812549711' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:23:13.290 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:13 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:23:13.348 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:13 smithi115 ceph-mon[72476]: pgmap v1689: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 646 op/s; 7652/66252 objects degraded (11.550%); 4863/66252 objects misplaced (7.340%); 8.4 MiB/s, 136 objects/s recovering 2024-03-20T09:23:13.349 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:13 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 7652/66252 objects degraded (11.550%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-20T09:23:13.349 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:13 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2856212429' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-03-20T09:23:13.349 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:13 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-03-20T09:23:13.349 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:13 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1812549711' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:23:13.349 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:13 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:23:13.389 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-20T09:23:13.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:13 smithi122 ceph-mon[73495]: pgmap v1689: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 646 op/s; 7652/66252 objects degraded (11.550%); 4863/66252 objects misplaced (7.340%); 8.4 MiB/s, 136 objects/s recovering 2024-03-20T09:23:13.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:13 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 7652/66252 objects degraded (11.550%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-20T09:23:13.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:13 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2856212429' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-03-20T09:23:13.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:13 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-03-20T09:23:13.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:13 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1812549711' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:23:13.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:13 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:23:13.860 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:23:14.068 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:23:14.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:14.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:23:14.172 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:23:14.230 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:14.391 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:23:14.397 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:14 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:23:14.397 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:14 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:23:14.397 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:14 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:23:14.397 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:14 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-03-20T09:23:14.397 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:14 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:23:14.397 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:14 smithi026 ceph-mon[68309]: osdmap e692: 12 total, 12 up, 12 in 2024-03-20T09:23:14.397 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:14 smithi026 ceph-mon[68309]: 1.1b scrub starts 2024-03-20T09:23:14.397 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:14 smithi026 ceph-mon[68309]: pgmap v1691: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 650 op/s; 7804/61146 objects degraded (12.763%); 3715/61146 objects misplaced (6.076%); 9.0 MiB/s, 144 objects/s recovering 2024-03-20T09:23:14.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:14.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:23:14.499 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:23:14.533 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:14.551 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:14 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:23:14.551 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:14 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:23:14.551 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:14 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:23:14.552 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:14 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-03-20T09:23:14.552 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:14 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:23:14.552 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:14 smithi115 ceph-mon[72476]: osdmap e692: 12 total, 12 up, 12 in 2024-03-20T09:23:14.552 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:14 smithi115 ceph-mon[72476]: 1.1b scrub starts 2024-03-20T09:23:14.552 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:14 smithi115 ceph-mon[72476]: pgmap v1691: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 650 op/s; 7804/61146 objects degraded (12.763%); 3715/61146 objects misplaced (6.076%); 9.0 MiB/s, 144 objects/s recovering 2024-03-20T09:23:14.598 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:23:14.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:14.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:23:14.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:14 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:23:14.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:14 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:23:14.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:14 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:23:14.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:14 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-03-20T09:23:14.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:14 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:23:14.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:14 smithi122 ceph-mon[73495]: osdmap e692: 12 total, 12 up, 12 in 2024-03-20T09:23:14.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:14 smithi122 ceph-mon[73495]: 1.1b scrub starts 2024-03-20T09:23:14.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:14 smithi122 ceph-mon[73495]: pgmap v1691: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 650 op/s; 7804/61146 objects degraded (12.763%); 3715/61146 objects misplaced (6.076%); 9.0 MiB/s, 144 objects/s recovering 2024-03-20T09:23:14.702 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:23:14.728 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:14.746 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:23:14.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:14.749 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:23:14.850 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:23:14.880 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:15.009 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:23:15.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:15.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:23:15.116 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:23:15.146 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:15.298 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:23:15.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:15.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:23:15.409 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:23:15.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:15 smithi026 ceph-mon[68309]: 1.1b scrub ok 2024-03-20T09:23:15.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:15 smithi026 ceph-mon[68309]: 1.9 scrub starts 2024-03-20T09:23:15.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:15 smithi026 ceph-mon[68309]: 1.9 scrub ok 2024-03-20T09:23:15.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:15 smithi026 ceph-mon[68309]: osdmap e693: 12 total, 12 up, 12 in 2024-03-20T09:23:15.484 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:15.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:15 smithi115 ceph-mon[72476]: 1.1b scrub ok 2024-03-20T09:23:15.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:15 smithi115 ceph-mon[72476]: 1.9 scrub starts 2024-03-20T09:23:15.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:15 smithi115 ceph-mon[72476]: 1.9 scrub ok 2024-03-20T09:23:15.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:15 smithi115 ceph-mon[72476]: osdmap e693: 12 total, 12 up, 12 in 2024-03-20T09:23:15.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:15 smithi122 ceph-mon[73495]: 1.1b scrub ok 2024-03-20T09:23:15.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:15 smithi122 ceph-mon[73495]: 1.9 scrub starts 2024-03-20T09:23:15.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:15 smithi122 ceph-mon[73495]: 1.9 scrub ok 2024-03-20T09:23:15.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:15 smithi122 ceph-mon[73495]: osdmap e693: 12 total, 12 up, 12 in 2024-03-20T09:23:15.705 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:23:15.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:15.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:23:15.818 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:23:15.872 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:15.964 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:23:15.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:15.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:23:16.070 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:23:16.105 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:16.131 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:23:16.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:16.138 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:23:16.239 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:23:16.262 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:16 smithi115 ceph-mon[72476]: 1.14 scrub starts 2024-03-20T09:23:16.262 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:16 smithi115 ceph-mon[72476]: 1.14 scrub ok 2024-03-20T09:23:16.262 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:16 smithi115 ceph-mon[72476]: osdmap e694: 12 total, 12 up, 12 in 2024-03-20T09:23:16.262 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:16 smithi115 ceph-mon[72476]: pgmap v1694: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 659 op/s; 7804/58509 objects degraded (13.338%); 3455/58509 objects misplaced (5.905%); 8.0 MiB/s, 129 objects/s recovering 2024-03-20T09:23:16.269 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:16.294 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:23:16.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:16.297 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:23:16.398 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:23:16.423 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:16 smithi122 ceph-mon[73495]: 1.14 scrub starts 2024-03-20T09:23:16.424 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:16 smithi122 ceph-mon[73495]: 1.14 scrub ok 2024-03-20T09:23:16.424 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:16 smithi122 ceph-mon[73495]: osdmap e694: 12 total, 12 up, 12 in 2024-03-20T09:23:16.424 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:16 smithi122 ceph-mon[73495]: pgmap v1694: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 659 op/s; 7804/58509 objects degraded (13.338%); 3455/58509 objects misplaced (5.905%); 8.0 MiB/s, 129 objects/s recovering 2024-03-20T09:23:16.425 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:16.441 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:23:16.441 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:23:16.446 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:23:16.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:16.450 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:23:16.551 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:23:16.600 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:16 smithi026 ceph-mon[68309]: 1.14 scrub starts 2024-03-20T09:23:16.600 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:16 smithi026 ceph-mon[68309]: 1.14 scrub ok 2024-03-20T09:23:16.600 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:16 smithi026 ceph-mon[68309]: osdmap e694: 12 total, 12 up, 12 in 2024-03-20T09:23:16.600 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:16 smithi026 ceph-mon[68309]: pgmap v1694: 47 pgs: 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 659 op/s; 7804/58509 objects degraded (13.338%); 3455/58509 objects misplaced (5.905%); 8.0 MiB/s, 129 objects/s recovering 2024-03-20T09:23:16.603 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:17.258 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-20T09:23:17.320 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:23:17.552 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:17 smithi026 ceph-mon[68309]: 5.a scrub starts 2024-03-20T09:23:17.552 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:17 smithi026 ceph-mon[68309]: 5.a scrub ok 2024-03-20T09:23:17.599 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 11 1 2024-03-20T09:23:17.662 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:17 smithi115 ceph-mon[72476]: 5.a scrub starts 2024-03-20T09:23:17.662 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:17 smithi115 ceph-mon[72476]: 5.a scrub ok 2024-03-20T09:23:17.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:17 smithi122 ceph-mon[73495]: 5.a scrub starts 2024-03-20T09:23:17.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:17 smithi122 ceph-mon[73495]: 5.a scrub ok 2024-03-20T09:23:17.733 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:23:18.011 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:23:18.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:18.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:23:18.116 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:23:18.179 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:18.274 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:18 smithi026 ceph-mon[68309]: 5.6 scrub starts 2024-03-20T09:23:18.274 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:18 smithi026 ceph-mon[68309]: 5.6 scrub ok 2024-03-20T09:23:18.274 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:18 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 7804/58509 objects degraded (13.338%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-20T09:23:18.275 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:18 smithi026 ceph-mon[68309]: osdmap e695: 12 total, 12 up, 12 in 2024-03-20T09:23:18.275 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:18 smithi026 ceph-mon[68309]: pgmap v1696: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 560 op/s; 7819/55464 objects degraded (14.097%); 3157/55464 objects misplaced (5.692%); 11 MiB/s, 174 objects/s recovering 2024-03-20T09:23:18.354 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:23:18.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:18.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:23:18.459 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:23:18.525 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:18.566 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:23:18.578 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:18 smithi115 ceph-mon[72476]: 5.6 scrub starts 2024-03-20T09:23:18.578 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:18 smithi115 ceph-mon[72476]: 5.6 scrub ok 2024-03-20T09:23:18.578 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:18 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 7804/58509 objects degraded (13.338%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-20T09:23:18.578 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:18 smithi115 ceph-mon[72476]: osdmap e695: 12 total, 12 up, 12 in 2024-03-20T09:23:18.579 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:18 smithi115 ceph-mon[72476]: pgmap v1696: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 560 op/s; 7819/55464 objects degraded (14.097%); 3157/55464 objects misplaced (5.692%); 11 MiB/s, 174 objects/s recovering 2024-03-20T09:23:18.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:18.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:23:18.680 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:23:18.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:18 smithi122 ceph-mon[73495]: 5.6 scrub starts 2024-03-20T09:23:18.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:18 smithi122 ceph-mon[73495]: 5.6 scrub ok 2024-03-20T09:23:18.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:18 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 7804/58509 objects degraded (13.338%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-20T09:23:18.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:18 smithi122 ceph-mon[73495]: osdmap e695: 12 total, 12 up, 12 in 2024-03-20T09:23:18.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:18 smithi122 ceph-mon[73495]: pgmap v1696: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 560 op/s; 7819/55464 objects degraded (14.097%); 3157/55464 objects misplaced (5.692%); 11 MiB/s, 174 objects/s recovering 2024-03-20T09:23:18.709 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:18.734 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:23:18.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:18.735 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:23:18.836 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:23:18.866 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:18.891 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:23:18.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:18.895 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:23:18.996 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:23:19.050 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:20.512 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:23:20.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:20.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:23:20.545 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:23:20.618 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:23:20.689 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:21.297 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:21 smithi026 ceph-mon[68309]: 5.9 scrub starts 2024-03-20T09:23:21.297 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:21 smithi026 ceph-mon[68309]: 5.9 scrub ok 2024-03-20T09:23:21.297 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:21 smithi026 ceph-mon[68309]: pgmap v1697: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 492 op/s; 7861/52287 objects degraded (15.034%); 2676/52287 objects misplaced (5.118%); 9.9 MiB/s, 159 objects/s recovering 2024-03-20T09:23:21.377 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:23:21.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:21.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:23:21.490 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:23:21.526 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:21.636 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:23:21.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:21.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:23:21.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:21 smithi115 ceph-mon[72476]: 5.9 scrub starts 2024-03-20T09:23:21.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:21 smithi115 ceph-mon[72476]: 5.9 scrub ok 2024-03-20T09:23:21.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:21 smithi115 ceph-mon[72476]: pgmap v1697: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 492 op/s; 7861/52287 objects degraded (15.034%); 2676/52287 objects misplaced (5.118%); 9.9 MiB/s, 159 objects/s recovering 2024-03-20T09:23:21.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:21 smithi122 ceph-mon[73495]: 5.9 scrub starts 2024-03-20T09:23:21.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:21 smithi122 ceph-mon[73495]: 5.9 scrub ok 2024-03-20T09:23:21.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:21 smithi122 ceph-mon[73495]: pgmap v1697: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 492 op/s; 7861/52287 objects degraded (15.034%); 2676/52287 objects misplaced (5.118%); 9.9 MiB/s, 159 objects/s recovering 2024-03-20T09:23:21.742 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:23:21.777 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:21.801 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:23:21.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:21.808 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:23:21.910 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:23:21.944 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:22.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:22.123 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:23:22.123 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:23:22.224 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:23:22.292 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:22.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:22 smithi026 ceph-mon[68309]: 5.e scrub starts 2024-03-20T09:23:22.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:22 smithi115 ceph-mon[72476]: 5.e scrub starts 2024-03-20T09:23:22.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:22 smithi122 ceph-mon[73495]: 5.e scrub starts 2024-03-20T09:23:22.732 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:23:22.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:22.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:23:22.836 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:23:22.879 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:23.348 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:23:23.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:23.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:23:23.422 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:23:23.422 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:23:23.452 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:23:23.475 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:23 smithi026 ceph-mon[68309]: 1.19 scrub starts 2024-03-20T09:23:23.475 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:23 smithi026 ceph-mon[68309]: 1.19 scrub ok 2024-03-20T09:23:23.475 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:23 smithi026 ceph-mon[68309]: 5.e scrub ok 2024-03-20T09:23:23.475 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:23 smithi026 ceph-mon[68309]: pgmap v1698: 47 pgs: 1 active+clean+scrubbing, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 991 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 574 op/s; 7861/47544 objects degraded (16.534%); 2676/47544 objects misplaced (5.628%); 7.6 MiB/s, 121 objects/s recovering 2024-03-20T09:23:23.475 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:23 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 7861/47544 objects degraded (16.534%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-20T09:23:23.503 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:23.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:23 smithi115 ceph-mon[72476]: 1.19 scrub starts 2024-03-20T09:23:23.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:23 smithi115 ceph-mon[72476]: 1.19 scrub ok 2024-03-20T09:23:23.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:23 smithi115 ceph-mon[72476]: 5.e scrub ok 2024-03-20T09:23:23.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:23 smithi115 ceph-mon[72476]: pgmap v1698: 47 pgs: 1 active+clean+scrubbing, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 991 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 574 op/s; 7861/47544 objects degraded (16.534%); 2676/47544 objects misplaced (5.628%); 7.6 MiB/s, 121 objects/s recovering 2024-03-20T09:23:23.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:23 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 7861/47544 objects degraded (16.534%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-20T09:23:23.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:23 smithi122 ceph-mon[73495]: 1.19 scrub starts 2024-03-20T09:23:23.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:23 smithi122 ceph-mon[73495]: 1.19 scrub ok 2024-03-20T09:23:23.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:23 smithi122 ceph-mon[73495]: 5.e scrub ok 2024-03-20T09:23:23.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:23 smithi122 ceph-mon[73495]: pgmap v1698: 47 pgs: 1 active+clean+scrubbing, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 991 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 574 op/s; 7861/47544 objects degraded (16.534%); 2676/47544 objects misplaced (5.628%); 7.6 MiB/s, 121 objects/s recovering 2024-03-20T09:23:23.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:23 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 7861/47544 objects degraded (16.534%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-20T09:23:24.262 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_ops_in_flight 2024-03-20T09:23:24.595 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:23:25.173 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:23:25.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:25.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:23:25.278 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:23:25.327 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:25.362 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:23:25.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:25.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:23:25.467 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:23:25.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:25 smithi026 ceph-mon[68309]: 5.2 scrub starts 2024-03-20T09:23:25.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:25 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:23:25.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:25 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:23:25.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:25 smithi026 ceph-mon[68309]: 5.2 scrub ok 2024-03-20T09:23:25.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:25 smithi026 ceph-mon[68309]: pgmap v1699: 47 pgs: 1 active+clean+scrubbing, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 875 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 699 op/s; 8004/41994 objects degraded (19.060%); 2265/41994 objects misplaced (5.394%); 6.7 MiB/s, 108 objects/s recovering 2024-03-20T09:23:25.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:25 smithi026 ceph-mon[68309]: osdmap e696: 12 total, 12 up, 12 in 2024-03-20T09:23:25.548 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:25 smithi115 ceph-mon[72476]: 5.2 scrub starts 2024-03-20T09:23:25.549 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:25 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:23:25.549 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:25 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:23:25.549 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:25 smithi115 ceph-mon[72476]: 5.2 scrub ok 2024-03-20T09:23:25.549 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:25 smithi115 ceph-mon[72476]: pgmap v1699: 47 pgs: 1 active+clean+scrubbing, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 875 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 699 op/s; 8004/41994 objects degraded (19.060%); 2265/41994 objects misplaced (5.394%); 6.7 MiB/s, 108 objects/s recovering 2024-03-20T09:23:25.549 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:25 smithi115 ceph-mon[72476]: osdmap e696: 12 total, 12 up, 12 in 2024-03-20T09:23:25.554 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:25.602 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:23:25.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:25.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:23:25.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:25 smithi122 ceph-mon[73495]: 5.2 scrub starts 2024-03-20T09:23:25.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:25 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:23:25.698 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:25 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:23:25.699 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:25 smithi122 ceph-mon[73495]: 5.2 scrub ok 2024-03-20T09:23:25.699 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:25 smithi122 ceph-mon[73495]: pgmap v1699: 47 pgs: 1 active+clean+scrubbing, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 875 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 699 op/s; 8004/41994 objects degraded (19.060%); 2265/41994 objects misplaced (5.394%); 6.7 MiB/s, 108 objects/s recovering 2024-03-20T09:23:25.699 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:25 smithi122 ceph-mon[73495]: osdmap e696: 12 total, 12 up, 12 in 2024-03-20T09:23:25.711 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:23:25.755 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:25.836 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:23:25.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:25.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:23:25.943 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:23:26.010 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:26.062 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:23:26.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:26.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:23:26.166 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:23:26.226 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:26.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:26 smithi122 ceph-mon[73495]: pgmap v1701: 47 pgs: 1 active+clean+scrubbing, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 806 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 696 op/s; 8004/38673 objects degraded (20.697%); 1997/38673 objects misplaced (5.164%); 8.7 MiB/s, 140 objects/s recovering 2024-03-20T09:23:26.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:26 smithi026 ceph-mon[68309]: pgmap v1701: 47 pgs: 1 active+clean+scrubbing, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 806 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 696 op/s; 8004/38673 objects degraded (20.697%); 1997/38673 objects misplaced (5.164%); 8.7 MiB/s, 140 objects/s recovering 2024-03-20T09:23:26.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:26 smithi115 ceph-mon[72476]: pgmap v1701: 47 pgs: 1 active+clean+scrubbing, 3 active+recovering+undersized+degraded+remapped, 43 active+clean; 806 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 696 op/s; 8004/38673 objects degraded (20.697%); 1997/38673 objects misplaced (5.164%); 8.7 MiB/s, 140 objects/s recovering 2024-03-20T09:23:27.118 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:23:27.119 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:23:27.319 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:23:27.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:27 smithi122 ceph-mon[73495]: 1.6 scrub starts 2024-03-20T09:23:27.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:27 smithi122 ceph-mon[73495]: 1.6 scrub ok 2024-03-20T09:23:27.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:27 smithi122 ceph-mon[73495]: 1.7 scrub starts 2024-03-20T09:23:27.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:27 smithi122 ceph-mon[73495]: 1.7 scrub ok 2024-03-20T09:23:27.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:27 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2691706981' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:23:27.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:27 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2732367344' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:23:27.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:27 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:23:27.607 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:23:27.607 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:23:27.608 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:23:27.608 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:23:27.608 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:23:27.608 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:23:27.608 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:23:27.608 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:23:27.608 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:23:27.608 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:23:27.608 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:23:27.609 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:23:27.609 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:23:27.609 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:23:27.609 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:23:27.609 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:23:27.609 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:23:27.609 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:23:27.609 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:23:27.610 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:23:27.610 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:23:27.610 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:23:27.610 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:23:27.610 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:23:27.656 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:27 smithi115 ceph-mon[72476]: 1.6 scrub starts 2024-03-20T09:23:27.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:27 smithi115 ceph-mon[72476]: 1.6 scrub ok 2024-03-20T09:23:27.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:27 smithi115 ceph-mon[72476]: 1.7 scrub starts 2024-03-20T09:23:27.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:27 smithi115 ceph-mon[72476]: 1.7 scrub ok 2024-03-20T09:23:27.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:27 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2691706981' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:23:27.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:27 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2732367344' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:23:27.657 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:27 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:23:27.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:27 smithi026 ceph-mon[68309]: 1.6 scrub starts 2024-03-20T09:23:27.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:27 smithi026 ceph-mon[68309]: 1.6 scrub ok 2024-03-20T09:23:27.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:27 smithi026 ceph-mon[68309]: 1.7 scrub starts 2024-03-20T09:23:27.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:27 smithi026 ceph-mon[68309]: 1.7 scrub ok 2024-03-20T09:23:27.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:27 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2691706981' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:23:27.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:27 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2732367344' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:23:27.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:27 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:23:27.923 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_blocked_ops 2024-03-20T09:23:28.277 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:23:28.290 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:23:28.322 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:23:28.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:28.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:23:28.426 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:23:28.474 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:28.584 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:28 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:23:28.584 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:28 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:23:28.584 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:28 smithi122 ceph-mon[73495]: osdmap e697: 12 total, 12 up, 12 in 2024-03-20T09:23:28.584 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:28 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8004/38673 objects degraded (20.697%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-20T09:23:28.585 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:28 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:23:28.585 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:28 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:23:28.585 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:28 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:23:28.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:28 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:23:28.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:28 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:23:28.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:28 smithi115 ceph-mon[72476]: osdmap e697: 12 total, 12 up, 12 in 2024-03-20T09:23:28.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:28 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8004/38673 objects degraded (20.697%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-20T09:23:28.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:28 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:23:28.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:28 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:23:28.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:28 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:23:28.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:28 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:23:28.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:28 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:23:28.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:28 smithi026 ceph-mon[68309]: osdmap e697: 12 total, 12 up, 12 in 2024-03-20T09:23:28.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:28 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8004/38673 objects degraded (20.697%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-20T09:23:28.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:28 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:23:28.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:28 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:23:28.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:28 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:23:29.129 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:23:29.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:29.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:23:29.236 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:23:29.275 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:29.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:29 smithi026 ceph-mon[68309]: 1.5 scrub starts 2024-03-20T09:23:29.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:29 smithi026 ceph-mon[68309]: 1.5 scrub ok 2024-03-20T09:23:29.656 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:29 smithi122 ceph-mon[73495]: 1.5 scrub starts 2024-03-20T09:23:29.656 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:29 smithi122 ceph-mon[73495]: 1.5 scrub ok 2024-03-20T09:23:29.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:29 smithi115 ceph-mon[72476]: 1.5 scrub starts 2024-03-20T09:23:29.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:29 smithi115 ceph-mon[72476]: 1.5 scrub ok 2024-03-20T09:23:30.087 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:23:30.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:30.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:23:30.197 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:23:30.247 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:30.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:30 smithi122 ceph-mon[73495]: 1.f scrub starts 2024-03-20T09:23:30.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:30 smithi122 ceph-mon[73495]: 1.f scrub ok 2024-03-20T09:23:30.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:30 smithi122 ceph-mon[73495]: pgmap v1704: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 633 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 716 op/s; 8181/30342 objects degraded (26.963%); 1237/30342 objects misplaced (4.077%); 8.7 MiB/s, 139 objects/s recovering 2024-03-20T09:23:30.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:30 smithi026 ceph-mon[68309]: 1.f scrub starts 2024-03-20T09:23:30.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:30 smithi026 ceph-mon[68309]: 1.f scrub ok 2024-03-20T09:23:30.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:30 smithi026 ceph-mon[68309]: pgmap v1704: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 633 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 716 op/s; 8181/30342 objects degraded (26.963%); 1237/30342 objects misplaced (4.077%); 8.7 MiB/s, 139 objects/s recovering 2024-03-20T09:23:30.910 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:30 smithi115 ceph-mon[72476]: 1.f scrub starts 2024-03-20T09:23:30.910 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:30 smithi115 ceph-mon[72476]: 1.f scrub ok 2024-03-20T09:23:30.910 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:30 smithi115 ceph-mon[72476]: pgmap v1704: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 633 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 716 op/s; 8181/30342 objects degraded (26.963%); 1237/30342 objects misplaced (4.077%); 8.7 MiB/s, 139 objects/s recovering 2024-03-20T09:23:31.415 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:31 smithi026 ceph-mon[68309]: 1.1 scrub starts 2024-03-20T09:23:31.500 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:23:31.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:31.505 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:23:31.605 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:23:31.671 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:31 smithi115 ceph-mon[72476]: 1.1 scrub starts 2024-03-20T09:23:31.674 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:31.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:31 smithi122 ceph-mon[73495]: 1.1 scrub starts 2024-03-20T09:23:31.714 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:23:31.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:31.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:23:31.810 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:23:31.810 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:23:31.810 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-03-20T09:23:31.810 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-03-20T09:23:31.810 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:23:31.820 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:23:31.870 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:31.949 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:23:31.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:31.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:23:32.055 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:23:32.095 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:32.136 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:23:32.209 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:23:32.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:32.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:23:32.328 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:23:32.375 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:32.410 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:23:32.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:32.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:23:32.436 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:32 smithi026 ceph-mon[68309]: 1.1 scrub ok 2024-03-20T09:23:32.436 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:32 smithi026 ceph-mon[68309]: 1.a scrub starts 2024-03-20T09:23:32.436 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:32 smithi026 ceph-mon[68309]: 1.a scrub ok 2024-03-20T09:23:32.436 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:32 smithi026 ceph-mon[68309]: pgmap v1705: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 531 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 687 op/s; 8181/25488 objects degraded (32.097%); 1237/25488 objects misplaced (4.853%); 6.6 MiB/s, 106 objects/s recovering 2024-03-20T09:23:32.436 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:32 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8181/30342 objects degraded (26.963%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-20T09:23:32.516 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:23:32.578 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:32.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:32 smithi115 ceph-mon[72476]: 1.1 scrub ok 2024-03-20T09:23:32.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:32 smithi115 ceph-mon[72476]: 1.a scrub starts 2024-03-20T09:23:32.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:32 smithi115 ceph-mon[72476]: 1.a scrub ok 2024-03-20T09:23:32.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:32 smithi115 ceph-mon[72476]: pgmap v1705: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 531 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 687 op/s; 8181/25488 objects degraded (32.097%); 1237/25488 objects misplaced (4.853%); 6.6 MiB/s, 106 objects/s recovering 2024-03-20T09:23:32.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:32 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8181/30342 objects degraded (26.963%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-20T09:23:32.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:32 smithi122 ceph-mon[73495]: 1.1 scrub ok 2024-03-20T09:23:32.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:32 smithi122 ceph-mon[73495]: 1.a scrub starts 2024-03-20T09:23:32.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:32 smithi122 ceph-mon[73495]: 1.a scrub ok 2024-03-20T09:23:32.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:32 smithi122 ceph-mon[73495]: pgmap v1705: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 531 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 687 op/s; 8181/25488 objects degraded (32.097%); 1237/25488 objects misplaced (4.853%); 6.6 MiB/s, 106 objects/s recovering 2024-03-20T09:23:32.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:32 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8181/30342 objects degraded (26.963%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-20T09:23:32.708 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:23:32.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:32.712 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:23:32.732 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_historic_ops 2024-03-20T09:23:32.813 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:23:32.875 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:32.902 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:23:32.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:32.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:23:33.012 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:23:33.049 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:23:33.050 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:33.103 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:23:33.273 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:23:33.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:33.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:23:33.377 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:23:33.416 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:33.439 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:33 smithi026 ceph-mon[68309]: 5.0 scrub starts 2024-03-20T09:23:33.439 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:33 smithi026 ceph-mon[68309]: 5.0 scrub ok 2024-03-20T09:23:33.439 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:33 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:23:33.439 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:33 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:23:33.439 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:33 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:23:33.439 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:33 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:23:33.439 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:33 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:23:33.446 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:23:33.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:33.453 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:23:33.554 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:23:33.602 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:33.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:33 smithi122 ceph-mon[73495]: 5.0 scrub starts 2024-03-20T09:23:33.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:33 smithi122 ceph-mon[73495]: 5.0 scrub ok 2024-03-20T09:23:33.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:33 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:23:33.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:33 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:23:33.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:33 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:23:33.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:33 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:23:33.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:33 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:23:33.910 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:33 smithi115 ceph-mon[72476]: 5.0 scrub starts 2024-03-20T09:23:33.910 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:33 smithi115 ceph-mon[72476]: 5.0 scrub ok 2024-03-20T09:23:33.910 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:33 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:23:33.910 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:33 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:23:33.910 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:33 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:23:33.910 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:33 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:23:33.910 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:33 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:23:34.367 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:23:34.372 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:23:34.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:34.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:23:34.477 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:23:34.526 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:34.568 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:23:34.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:34.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:23:34.676 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:23:34.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:34 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:23:34.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:34 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:23:34.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:34 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:23:34.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:34 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:23:34.697 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:34 smithi122 ceph-mon[73495]: pgmap v1706: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 417 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 739 op/s; 8348/20004 objects degraded (41.732%); 764/20004 objects misplaced (3.819%); 7.3 MiB/s, 116 objects/s recovering 2024-03-20T09:23:34.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:34 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:23:34.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:34 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:23:34.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:34 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:23:34.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:34 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:23:34.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:34 smithi026 ceph-mon[68309]: pgmap v1706: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 417 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 739 op/s; 8348/20004 objects degraded (41.732%); 764/20004 objects misplaced (3.819%); 7.3 MiB/s, 116 objects/s recovering 2024-03-20T09:23:34.727 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:34.861 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:23:34.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:34.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:23:34.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:34 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:23:34.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:34 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:23:34.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:34 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:23:34.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:34 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:23:34.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:34 smithi115 ceph-mon[72476]: pgmap v1706: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 417 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 739 op/s; 8348/20004 objects degraded (41.732%); 764/20004 objects misplaced (3.819%); 7.3 MiB/s, 116 objects/s recovering 2024-03-20T09:23:34.969 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:23:35.015 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:35.046 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:23:35.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:35.050 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:23:35.082 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:23:35.082 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-03-20T09:23:35.083 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-03-20T09:23:35.083 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-03-20T09:23:35.083 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.083 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212811 4.b 4.11862d5b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:23:35.083 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.225559+0000", 2024-03-20T09:23:35.083 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 377.85412933499998, 2024-03-20T09:23:35.083 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.040956411, 2024-03-20T09:23:35.083 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:23:35.083 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:23:35.083 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:23:35.083 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:23:35.083 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:23:35.084 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212811 2024-03-20T09:23:35.084 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.084 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:23:35.084 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.084 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:23:35.084 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.225559+0000", 2024-03-20T09:23:35.084 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.084 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.084 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.084 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:23:35.084 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.225559+0000", 2024-03-20T09:23:35.084 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.085 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.085 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.085 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:23:35.085 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.225560+0000", 2024-03-20T09:23:35.085 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.42e-06 2024-03-20T09:23:35.085 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.085 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.085 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:23:35.085 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.225566+0000", 2024-03-20T09:23:35.085 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3480000000000003e-06 2024-03-20T09:23:35.085 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.085 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.085 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:23:35.086 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.225578+0000", 2024-03-20T09:23:35.086 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2313000000000001e-05 2024-03-20T09:23:35.086 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.086 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.086 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:23:35.086 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.225586+0000", 2024-03-20T09:23:35.086 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3049999999999993e-06 2024-03-20T09:23:35.086 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.086 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.086 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.086 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.225638+0000", 2024-03-20T09:23:35.086 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1431000000000003e-05 2024-03-20T09:23:35.087 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.087 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.087 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:23:35.087 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.225644+0000", 2024-03-20T09:23:35.087 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0909999999999998e-06 2024-03-20T09:23:35.087 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.087 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.088 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.088 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.266501+0000", 2024-03-20T09:23:35.088 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0408575849999999 2024-03-20T09:23:35.088 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.088 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.088 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:23:35.088 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.266515+0000", 2024-03-20T09:23:35.088 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3918e-05 2024-03-20T09:23:35.088 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.088 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:23:35.088 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.088 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.088 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.089 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212818 4.b 4.8a8d05db (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:23:35.089 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.249767+0000", 2024-03-20T09:23:35.089 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 377.82992103100003, 2024-03-20T09:23:35.089 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0170676489999999, 2024-03-20T09:23:35.089 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:23:35.089 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:23:35.089 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:23:35.089 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:23:35.089 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:23:35.089 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212818 2024-03-20T09:23:35.089 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.089 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:23:35.089 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.089 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:23:35.090 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.249767+0000", 2024-03-20T09:23:35.090 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.090 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.090 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.090 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:23:35.090 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.249767+0000", 2024-03-20T09:23:35.090 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.090 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.090 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.090 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:23:35.090 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.249769+0000", 2024-03-20T09:23:35.090 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5239999999999999e-06 2024-03-20T09:23:35.090 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.091 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.091 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:23:35.091 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.249775+0000", 2024-03-20T09:23:35.091 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3709999999999996e-06 2024-03-20T09:23:35.091 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.091 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.091 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:23:35.091 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.249789+0000", 2024-03-20T09:23:35.091 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4204e-05 2024-03-20T09:23:35.091 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.091 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.091 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:23:35.091 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.249797+0000", 2024-03-20T09:23:35.091 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7400000000000004e-06 2024-03-20T09:23:35.092 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.092 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.092 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.092 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.249847+0000", 2024-03-20T09:23:35.092 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9585e-05 2024-03-20T09:23:35.092 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.092 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.092 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:23:35.092 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.249855+0000", 2024-03-20T09:23:35.092 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0029999999999994e-06 2024-03-20T09:23:35.092 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.092 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.092 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.093 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.266829+0000", 2024-03-20T09:23:35.093 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.016974319 2024-03-20T09:23:35.093 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.093 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.093 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:23:35.093 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.266835+0000", 2024-03-20T09:23:35.093 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9030000000000001e-06 2024-03-20T09:23:35.093 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.093 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:23:35.093 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.093 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.093 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.093 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212844 4.b 4.5ff7566b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:23:35.094 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.291365+0000", 2024-03-20T09:23:35.094 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 377.78832271900001, 2024-03-20T09:23:35.094 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.97549757100000001, 2024-03-20T09:23:35.094 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:23:35.094 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:23:35.094 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:23:35.094 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:23:35.094 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:23:35.094 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212844 2024-03-20T09:23:35.094 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.094 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:23:35.094 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.094 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:23:35.094 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.291365+0000", 2024-03-20T09:23:35.095 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.095 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.095 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.095 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:23:35.095 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.291365+0000", 2024-03-20T09:23:35.095 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.095 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.095 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.095 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:23:35.095 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.291367+0000", 2024-03-20T09:23:35.095 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.283e-06 2024-03-20T09:23:35.095 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.095 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.095 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:23:35.096 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.291373+0000", 2024-03-20T09:23:35.096 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8719999999999999e-06 2024-03-20T09:23:35.096 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.096 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.096 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:23:35.096 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.291384+0000", 2024-03-20T09:23:35.096 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1895999999999999e-05 2024-03-20T09:23:35.096 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.096 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.096 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:23:35.096 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.291391+0000", 2024-03-20T09:23:35.096 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9999999999999999e-06 2024-03-20T09:23:35.096 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.096 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.096 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.097 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.291442+0000", 2024-03-20T09:23:35.097 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0389000000000002e-05 2024-03-20T09:23:35.097 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.097 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.097 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:23:35.097 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.291452+0000", 2024-03-20T09:23:35.097 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9899999999999992e-06 2024-03-20T09:23:35.097 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.097 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.097 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.097 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.266856+0000", 2024-03-20T09:23:35.097 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.97540415000000003 2024-03-20T09:23:35.097 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.098 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.098 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:23:35.098 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.266863+0000", 2024-03-20T09:23:35.098 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9909999999999996e-06 2024-03-20T09:23:35.098 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.098 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:23:35.098 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.098 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.098 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.098 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212811 4.b 4:dab46188:::benchmark_data_smithi122_145698_object91892:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:23:35.098 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.302808+0000", 2024-03-20T09:23:35.098 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 377.77688063699998, 2024-03-20T09:23:35.098 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.97602456199999998, 2024-03-20T09:23:35.099 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:23:35.099 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:23:35.099 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:23:35.099 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:23:35.099 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:23:35.099 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212811 2024-03-20T09:23:35.099 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.099 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:23:35.099 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.099 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:23:35.099 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302808+0000", 2024-03-20T09:23:35.099 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.099 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.099 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.100 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:23:35.100 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302808+0000", 2024-03-20T09:23:35.100 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.100 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.100 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.100 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:23:35.100 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302809+0000", 2024-03-20T09:23:35.100 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.094e-06 2024-03-20T09:23:35.100 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.100 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.100 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:23:35.100 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302813+0000", 2024-03-20T09:23:35.100 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6140000000000002e-06 2024-03-20T09:23:35.100 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.100 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.101 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:23:35.101 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302821+0000", 2024-03-20T09:23:35.101 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6559999999999992e-06 2024-03-20T09:23:35.101 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.101 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.101 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:23:35.101 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302827+0000", 2024-03-20T09:23:35.101 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.55e-06 2024-03-20T09:23:35.101 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.101 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.101 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.101 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302871+0000", 2024-03-20T09:23:35.101 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3659000000000003e-05 2024-03-20T09:23:35.102 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.102 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.102 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:23:35.102 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302878+0000", 2024-03-20T09:23:35.102 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7320000000000001e-06 2024-03-20T09:23:35.102 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.102 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.102 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.102 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.266903+0000", 2024-03-20T09:23:35.102 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.96402543299999999 2024-03-20T09:23:35.102 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.102 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.102 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:23:35.102 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.267005+0000", 2024-03-20T09:23:35.103 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010216400000000001 2024-03-20T09:23:35.103 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.103 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.103 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:23:35.103 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.267087+0000", 2024-03-20T09:23:35.103 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1893999999999999e-05 2024-03-20T09:23:35.103 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.103 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.103 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:23:35.103 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.274657+0000", 2024-03-20T09:23:35.103 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0075700979999999999 2024-03-20T09:23:35.103 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.103 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.103 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:23:35.104 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.278672+0000", 2024-03-20T09:23:35.104 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040150840000000004 2024-03-20T09:23:35.104 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.104 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.104 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:23:35.104 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.278772+0000", 2024-03-20T09:23:35.104 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9944000000000004e-05 2024-03-20T09:23:35.104 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.104 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.104 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:23:35.104 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.278807+0000", 2024-03-20T09:23:35.104 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4996999999999997e-05 2024-03-20T09:23:35.104 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.104 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.104 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:23:35.105 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.278832+0000", 2024-03-20T09:23:35.105 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4643000000000001e-05 2024-03-20T09:23:35.105 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.105 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:23:35.105 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.105 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.105 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.105 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212818 4.b 4:dba0b151:::benchmark_data_smithi122_145698_object91899:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:23:35.105 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.302840+0000", 2024-03-20T09:23:35.105 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 377.77684777299999, 2024-03-20T09:23:35.105 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.98232673100000001, 2024-03-20T09:23:35.105 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:23:35.105 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:23:35.105 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:23:35.106 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:23:35.106 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:23:35.106 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212818 2024-03-20T09:23:35.106 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.106 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:23:35.106 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.106 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:23:35.106 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302840+0000", 2024-03-20T09:23:35.106 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.106 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.106 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.106 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:23:35.106 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302840+0000", 2024-03-20T09:23:35.106 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.107 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.107 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.107 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:23:35.107 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302841+0000", 2024-03-20T09:23:35.107 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8900000000000001e-07 2024-03-20T09:23:35.107 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.107 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.107 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:23:35.107 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302842+0000", 2024-03-20T09:23:35.107 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6449999999999999e-06 2024-03-20T09:23:35.107 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.107 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.107 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:23:35.107 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302847+0000", 2024-03-20T09:23:35.107 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6609999999999999e-06 2024-03-20T09:23:35.108 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.108 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.108 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:23:35.108 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302851+0000", 2024-03-20T09:23:35.108 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7529999999999999e-06 2024-03-20T09:23:35.108 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.108 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.108 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.108 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302914+0000", 2024-03-20T09:23:35.108 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2842000000000006e-05 2024-03-20T09:23:35.108 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.108 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.108 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:23:35.108 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302925+0000", 2024-03-20T09:23:35.109 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1732e-05 2024-03-20T09:23:35.109 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.109 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.109 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.109 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.269166+0000", 2024-03-20T09:23:35.109 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.96624085900000001 2024-03-20T09:23:35.109 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.109 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.109 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:23:35.109 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.269235+0000", 2024-03-20T09:23:35.109 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8522000000000006e-05 2024-03-20T09:23:35.109 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.109 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.109 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:23:35.109 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.269317+0000", 2024-03-20T09:23:35.110 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2165999999999999e-05 2024-03-20T09:23:35.110 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.110 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.110 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:23:35.110 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.274708+0000", 2024-03-20T09:23:35.110 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0053907690000000001 2024-03-20T09:23:35.110 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.110 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.110 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:23:35.110 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.278679+0000", 2024-03-20T09:23:35.110 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0039710659999999997 2024-03-20T09:23:35.110 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.110 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.110 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:23:35.111 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.285056+0000", 2024-03-20T09:23:35.111 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0063770950000000002 2024-03-20T09:23:35.111 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.111 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.111 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:23:35.111 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.285113+0000", 2024-03-20T09:23:35.111 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7479000000000003e-05 2024-03-20T09:23:35.111 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.111 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.111 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:23:35.111 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.285167+0000", 2024-03-20T09:23:35.111 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3853000000000001e-05 2024-03-20T09:23:35.111 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.111 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:23:35.112 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.112 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.112 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.112 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212844 4.b 4:d66aeffa:::benchmark_data_smithi122_145698_object91925:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:23:35.112 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.302862+0000", 2024-03-20T09:23:35.112 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 377.77682572800001, 2024-03-20T09:23:35.112 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.98801962399999999, 2024-03-20T09:23:35.112 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:23:35.112 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:23:35.112 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:23:35.112 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:23:35.112 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:23:35.112 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212844 2024-03-20T09:23:35.113 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.113 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:23:35.113 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.113 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:23:35.113 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302862+0000", 2024-03-20T09:23:35.113 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.113 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.113 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.113 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:23:35.113 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302862+0000", 2024-03-20T09:23:35.113 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.113 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.113 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.113 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:23:35.114 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302863+0000", 2024-03-20T09:23:35.114 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4699999999999998e-07 2024-03-20T09:23:35.114 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.114 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.114 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:23:35.114 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302864+0000", 2024-03-20T09:23:35.114 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4330000000000001e-06 2024-03-20T09:23:35.114 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.114 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.114 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:23:35.114 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302868+0000", 2024-03-20T09:23:35.114 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6260000000000001e-06 2024-03-20T09:23:35.114 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.115 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.115 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:23:35.115 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302871+0000", 2024-03-20T09:23:35.115 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0929999999999999e-06 2024-03-20T09:23:35.115 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.115 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.115 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.115 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302948+0000", 2024-03-20T09:23:35.115 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7279000000000003e-05 2024-03-20T09:23:35.115 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.115 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.115 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:23:35.115 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302952+0000", 2024-03-20T09:23:35.115 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5750000000000001e-06 2024-03-20T09:23:35.116 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.116 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.116 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.116 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.270936+0000", 2024-03-20T09:23:35.116 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.96798445499999997 2024-03-20T09:23:35.116 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.116 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.116 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:23:35.116 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.271002+0000", 2024-03-20T09:23:35.116 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6129000000000003e-05 2024-03-20T09:23:35.116 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.116 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.116 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:23:35.117 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.271080+0000", 2024-03-20T09:23:35.117 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7349999999999996e-05 2024-03-20T09:23:35.117 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.117 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.117 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:23:35.117 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.278687+0000", 2024-03-20T09:23:35.117 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0076070679999999998 2024-03-20T09:23:35.117 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.117 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.117 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:23:35.117 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.278729+0000", 2024-03-20T09:23:35.117 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2608000000000003e-05 2024-03-20T09:23:35.117 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.117 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.118 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:23:35.118 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.290783+0000", 2024-03-20T09:23:35.118 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012053632 2024-03-20T09:23:35.118 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.118 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.118 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:23:35.118 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.290833+0000", 2024-03-20T09:23:35.118 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0235000000000003e-05 2024-03-20T09:23:35.118 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.118 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.118 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:23:35.118 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.290882+0000", 2024-03-20T09:23:35.118 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8894000000000003e-05 2024-03-20T09:23:35.118 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.119 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:23:35.119 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.119 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.119 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.119 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212851 4.b 4:deb6b4bb:::benchmark_data_smithi122_145698_object91932:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:23:35.119 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.302879+0000", 2024-03-20T09:23:35.119 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 377.776809093, 2024-03-20T09:23:35.119 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.98818408400000002, 2024-03-20T09:23:35.119 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:23:35.119 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:23:35.119 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:23:35.119 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:23:35.119 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:23:35.119 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212851 2024-03-20T09:23:35.120 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.120 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:23:35.120 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.120 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:23:35.120 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302879+0000", 2024-03-20T09:23:35.120 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.120 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.120 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.120 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:23:35.120 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302879+0000", 2024-03-20T09:23:35.120 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.120 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.120 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.120 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:23:35.121 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302879+0000", 2024-03-20T09:23:35.121 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.23e-07 2024-03-20T09:23:35.121 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.121 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.121 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:23:35.121 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302881+0000", 2024-03-20T09:23:35.121 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3209999999999999e-06 2024-03-20T09:23:35.121 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.121 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.121 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:23:35.121 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302884+0000", 2024-03-20T09:23:35.121 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4470000000000002e-06 2024-03-20T09:23:35.121 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.121 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.121 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:23:35.122 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302887+0000", 2024-03-20T09:23:35.122 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.287e-06 2024-03-20T09:23:35.122 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.122 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.122 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.122 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302966+0000", 2024-03-20T09:23:35.122 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8929000000000003e-05 2024-03-20T09:23:35.122 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.122 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.122 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:23:35.122 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.302970+0000", 2024-03-20T09:23:35.122 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6320000000000001e-06 2024-03-20T09:23:35.122 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.122 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.123 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.123 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.272683+0000", 2024-03-20T09:23:35.123 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.96971318299999998 2024-03-20T09:23:35.123 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.123 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.123 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:23:35.123 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.272746+0000", 2024-03-20T09:23:35.123 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2973999999999997e-05 2024-03-20T09:23:35.123 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.123 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.123 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:23:35.123 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.272821+0000", 2024-03-20T09:23:35.123 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4542999999999994e-05 2024-03-20T09:23:35.123 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.124 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.124 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:23:35.124 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.278841+0000", 2024-03-20T09:23:35.124 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0060203790000000002 2024-03-20T09:23:35.124 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.124 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.124 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:23:35.124 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.278871+0000", 2024-03-20T09:23:35.124 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9556999999999998e-05 2024-03-20T09:23:35.124 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.124 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.124 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:23:35.124 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.290988+0000", 2024-03-20T09:23:35.124 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012117681999999999 2024-03-20T09:23:35.125 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.125 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.125 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:23:35.125 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.291033+0000", 2024-03-20T09:23:35.125 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5232999999999999e-05 2024-03-20T09:23:35.125 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.125 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.125 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:23:35.125 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.291063+0000", 2024-03-20T09:23:35.125 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9694000000000001e-05 2024-03-20T09:23:35.125 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.125 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:23:35.125 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.125 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.126 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.126 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212872 4.b 4:d3c1b7c6:::benchmark_data_smithi122_145698_object91953:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:23:35.126 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.325422+0000", 2024-03-20T09:23:35.126 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 377.75426645900001, 2024-03-20T09:23:35.126 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.96576648700000001, 2024-03-20T09:23:35.126 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:23:35.126 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:23:35.126 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:23:35.126 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:23:35.126 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:23:35.126 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212872 2024-03-20T09:23:35.126 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.126 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:23:35.127 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.127 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:23:35.127 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.325422+0000", 2024-03-20T09:23:35.127 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.127 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.127 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.127 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:23:35.127 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.325422+0000", 2024-03-20T09:23:35.127 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.127 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.127 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.127 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:23:35.127 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.325424+0000", 2024-03-20T09:23:35.127 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.88e-06 2024-03-20T09:23:35.128 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.128 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.128 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:23:35.128 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.325439+0000", 2024-03-20T09:23:35.128 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5359e-05 2024-03-20T09:23:35.128 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.128 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.128 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:23:35.128 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.325453+0000", 2024-03-20T09:23:35.128 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.398e-05 2024-03-20T09:23:35.128 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.128 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.128 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:23:35.128 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.325462+0000", 2024-03-20T09:23:35.129 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5170000000000005e-06 2024-03-20T09:23:35.129 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.129 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.129 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.129 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.325525+0000", 2024-03-20T09:23:35.129 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2822000000000003e-05 2024-03-20T09:23:35.129 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.129 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.129 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:23:35.129 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.325536+0000", 2024-03-20T09:23:35.129 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0387999999999999e-05 2024-03-20T09:23:35.129 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.129 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.129 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.129 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.274737+0000", 2024-03-20T09:23:35.130 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.949201136 2024-03-20T09:23:35.130 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.130 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.130 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:23:35.130 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.274812+0000", 2024-03-20T09:23:35.130 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5683e-05 2024-03-20T09:23:35.130 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.130 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.130 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:23:35.130 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.274895+0000", 2024-03-20T09:23:35.130 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2427000000000002e-05 2024-03-20T09:23:35.130 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.130 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.130 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:23:35.131 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.278845+0000", 2024-03-20T09:23:35.131 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0039496699999999997 2024-03-20T09:23:35.131 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.131 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.131 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:23:35.131 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.278899+0000", 2024-03-20T09:23:35.131 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4008999999999997e-05 2024-03-20T09:23:35.131 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.131 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.131 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:23:35.131 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.291126+0000", 2024-03-20T09:23:35.131 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012227269000000001 2024-03-20T09:23:35.131 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.131 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.132 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:23:35.132 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.291158+0000", 2024-03-20T09:23:35.132 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2459000000000002e-05 2024-03-20T09:23:35.132 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.132 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.132 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:23:35.132 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.291188+0000", 2024-03-20T09:23:35.132 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9887999999999999e-05 2024-03-20T09:23:35.132 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.132 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:23:35.132 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.132 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.132 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.132 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212882 4.b 4:d9263e6a:::benchmark_data_smithi122_145698_object91963:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:23:35.133 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.339019+0000", 2024-03-20T09:23:35.133 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 377.74066918699998, 2024-03-20T09:23:35.133 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.95779115400000003, 2024-03-20T09:23:35.133 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:23:35.133 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:23:35.133 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:23:35.133 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:23:35.133 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:23:35.133 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212882 2024-03-20T09:23:35.133 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.133 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:23:35.133 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.133 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:23:35.133 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.339019+0000", 2024-03-20T09:23:35.133 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.134 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.134 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.134 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:23:35.134 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.339019+0000", 2024-03-20T09:23:35.134 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.134 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.134 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.134 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:23:35.134 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.339020+0000", 2024-03-20T09:23:35.134 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.192e-06 2024-03-20T09:23:35.134 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.134 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.134 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:23:35.135 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.339026+0000", 2024-03-20T09:23:35.135 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.558e-06 2024-03-20T09:23:35.135 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.135 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.135 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:23:35.135 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.339038+0000", 2024-03-20T09:23:35.135 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2242000000000001e-05 2024-03-20T09:23:35.135 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.135 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.135 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:23:35.135 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.339045+0000", 2024-03-20T09:23:35.135 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0899999999999999e-06 2024-03-20T09:23:35.135 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.135 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.136 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.136 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.339095+0000", 2024-03-20T09:23:35.136 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0056000000000003e-05 2024-03-20T09:23:35.136 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.136 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.136 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:23:35.136 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.339105+0000", 2024-03-20T09:23:35.136 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9059999999999997e-06 2024-03-20T09:23:35.136 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.136 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.136 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.136 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.276635+0000", 2024-03-20T09:23:35.136 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.93753048299999997 2024-03-20T09:23:35.136 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.136 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.137 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:23:35.137 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.276708+0000", 2024-03-20T09:23:35.137 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2668000000000003e-05 2024-03-20T09:23:35.137 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.137 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.137 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:23:35.137 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.276796+0000", 2024-03-20T09:23:35.137 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7843000000000006e-05 2024-03-20T09:23:35.137 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.137 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.137 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:23:35.137 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.278909+0000", 2024-03-20T09:23:35.137 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0021127950000000002 2024-03-20T09:23:35.137 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.138 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.138 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:23:35.138 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.278957+0000", 2024-03-20T09:23:35.138 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8223000000000002e-05 2024-03-20T09:23:35.138 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.138 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.138 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:23:35.138 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.296712+0000", 2024-03-20T09:23:35.138 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017754789999999999 2024-03-20T09:23:35.138 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.138 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.138 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:23:35.139 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.296764+0000", 2024-03-20T09:23:35.139 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2324999999999998e-05 2024-03-20T09:23:35.139 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.139 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.139 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:23:35.139 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.296810+0000", 2024-03-20T09:23:35.139 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5982999999999997e-05 2024-03-20T09:23:35.139 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.139 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:23:35.139 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.139 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.139 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.139 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212892 4.b 4:d92eb486:::benchmark_data_smithi122_145698_object91973:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:23:35.139 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.353878+0000", 2024-03-20T09:23:35.139 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 377.72580993399998, 2024-03-20T09:23:35.140 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.94304838899999999, 2024-03-20T09:23:35.140 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:23:35.140 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:23:35.140 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:23:35.140 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:23:35.140 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:23:35.140 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212892 2024-03-20T09:23:35.140 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.140 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:23:35.140 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.140 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:23:35.140 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.353878+0000", 2024-03-20T09:23:35.140 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.140 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.141 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.141 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:23:35.141 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.353878+0000", 2024-03-20T09:23:35.141 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.141 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.141 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.141 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:23:35.141 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.353880+0000", 2024-03-20T09:23:35.141 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.528e-06 2024-03-20T09:23:35.141 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.141 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.141 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:23:35.141 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.353887+0000", 2024-03-20T09:23:35.141 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0330000000000003e-06 2024-03-20T09:23:35.142 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.142 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.142 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:23:35.142 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.353901+0000", 2024-03-20T09:23:35.142 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4188999999999999e-05 2024-03-20T09:23:35.142 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.142 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.142 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:23:35.142 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.353909+0000", 2024-03-20T09:23:35.142 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3780000000000001e-06 2024-03-20T09:23:35.142 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.142 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.142 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.142 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.353970+0000", 2024-03-20T09:23:35.143 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0522000000000001e-05 2024-03-20T09:23:35.143 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.143 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.143 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:23:35.143 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.353979+0000", 2024-03-20T09:23:35.143 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0459999999999994e-06 2024-03-20T09:23:35.143 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.143 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.143 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.143 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.278984+0000", 2024-03-20T09:23:35.143 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.92500549300000001 2024-03-20T09:23:35.143 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.143 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.143 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:23:35.143 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.279054+0000", 2024-03-20T09:23:35.144 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9393999999999993e-05 2024-03-20T09:23:35.144 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.144 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.144 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:23:35.144 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.279120+0000", 2024-03-20T09:23:35.144 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6508999999999996e-05 2024-03-20T09:23:35.144 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.144 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.144 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:23:35.144 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.285226+0000", 2024-03-20T09:23:35.144 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0061055240000000002 2024-03-20T09:23:35.144 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.144 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.144 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:23:35.145 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.287457+0000", 2024-03-20T09:23:35.145 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0022312439999999998 2024-03-20T09:23:35.145 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.145 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.145 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:23:35.145 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.296854+0000", 2024-03-20T09:23:35.145 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0093965170000000001 2024-03-20T09:23:35.145 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.145 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.145 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:23:35.145 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.296886+0000", 2024-03-20T09:23:35.145 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1952999999999997e-05 2024-03-20T09:23:35.146 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.146 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.146 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:23:35.146 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.296927+0000", 2024-03-20T09:23:35.146 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1059000000000001e-05 2024-03-20T09:23:35.146 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.146 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:23:35.146 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.146 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.146 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.146 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212895 4.b 4:de658af1:::benchmark_data_smithi122_145698_object91976:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:23:35.147 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.354410+0000", 2024-03-20T09:23:35.147 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 377.725278538, 2024-03-20T09:23:35.147 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.94266090199999997, 2024-03-20T09:23:35.147 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:23:35.147 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:23:35.147 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:23:35.147 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:23:35.147 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:23:35.147 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212895 2024-03-20T09:23:35.147 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.147 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:23:35.148 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.148 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:23:35.148 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.354410+0000", 2024-03-20T09:23:35.148 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.148 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.148 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.148 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:23:35.148 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.354410+0000", 2024-03-20T09:23:35.148 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.148 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.148 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.148 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:23:35.148 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.354411+0000", 2024-03-20T09:23:35.148 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.33e-07 2024-03-20T09:23:35.149 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.149 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.149 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:23:35.149 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.354416+0000", 2024-03-20T09:23:35.149 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0579999999999997e-06 2024-03-20T09:23:35.149 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.149 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.149 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:23:35.149 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.354429+0000", 2024-03-20T09:23:35.149 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3370000000000001e-05 2024-03-20T09:23:35.149 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.149 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.149 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:23:35.149 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.354436+0000", 2024-03-20T09:23:35.149 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4220000000000003e-06 2024-03-20T09:23:35.150 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.150 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.150 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.150 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.354485+0000", 2024-03-20T09:23:35.150 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8176000000000001e-05 2024-03-20T09:23:35.150 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.150 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.150 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:23:35.150 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.354492+0000", 2024-03-20T09:23:35.150 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0729999999999999e-06 2024-03-20T09:23:35.150 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.150 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.150 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.151 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.280876+0000", 2024-03-20T09:23:35.151 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.92638386399999995 2024-03-20T09:23:35.151 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.151 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.151 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:23:35.151 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.280985+0000", 2024-03-20T09:23:35.151 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000109072 2024-03-20T09:23:35.151 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.151 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.151 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:23:35.151 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.281057+0000", 2024-03-20T09:23:35.151 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2485999999999997e-05 2024-03-20T09:23:35.151 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.151 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.152 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:23:35.152 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.285294+0000", 2024-03-20T09:23:35.152 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0042366859999999999 2024-03-20T09:23:35.152 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.152 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.152 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:23:35.152 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.287461+0000", 2024-03-20T09:23:35.152 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0021671709999999999 2024-03-20T09:23:35.152 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.152 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.152 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:23:35.152 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.296989+0000", 2024-03-20T09:23:35.152 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0095281020000000001 2024-03-20T09:23:35.152 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.153 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.153 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:23:35.153 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.297030+0000", 2024-03-20T09:23:35.153 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1399000000000001e-05 2024-03-20T09:23:35.153 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.153 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.153 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:23:35.153 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.297071+0000", 2024-03-20T09:23:35.153 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0089999999999997e-05 2024-03-20T09:23:35.153 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.153 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:23:35.153 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.153 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.153 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.154 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212904 4.b 4:dcc51503:::benchmark_data_smithi122_145698_object91985:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:23:35.154 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.363857+0000", 2024-03-20T09:23:35.154 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 377.71583159199997, 2024-03-20T09:23:35.154 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.93336363, 2024-03-20T09:23:35.154 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:23:35.154 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:23:35.154 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:23:35.154 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:23:35.154 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:23:35.154 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212904 2024-03-20T09:23:35.154 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.154 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:23:35.154 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.155 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:23:35.155 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.363857+0000", 2024-03-20T09:23:35.155 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.155 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.155 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.155 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:23:35.155 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.363857+0000", 2024-03-20T09:23:35.155 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.155 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.155 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.155 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:23:35.155 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.363858+0000", 2024-03-20T09:23:35.155 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4500000000000001e-06 2024-03-20T09:23:35.155 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.155 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.156 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:23:35.156 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.363868+0000", 2024-03-20T09:23:35.156 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6339999999999996e-06 2024-03-20T09:23:35.156 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.156 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.156 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:23:35.156 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.363882+0000", 2024-03-20T09:23:35.156 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4800000000000001e-05 2024-03-20T09:23:35.156 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.156 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.156 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:23:35.156 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.363892+0000", 2024-03-20T09:23:35.156 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6949999999999995e-06 2024-03-20T09:23:35.156 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.157 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.157 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.157 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.363952+0000", 2024-03-20T09:23:35.157 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9534000000000003e-05 2024-03-20T09:23:35.157 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.157 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.157 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:23:35.157 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.363962+0000", 2024-03-20T09:23:35.157 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0399e-05 2024-03-20T09:23:35.157 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.157 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.157 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.157 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.282952+0000", 2024-03-20T09:23:35.157 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.91899036999999995 2024-03-20T09:23:35.158 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.158 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.158 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:23:35.158 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.283061+0000", 2024-03-20T09:23:35.158 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000108918 2024-03-20T09:23:35.158 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.158 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.158 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:23:35.158 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.283184+0000", 2024-03-20T09:23:35.158 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000123022 2024-03-20T09:23:35.158 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.158 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.158 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:23:35.158 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.287465+0000", 2024-03-20T09:23:35.159 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0042803260000000001 2024-03-20T09:23:35.159 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.159 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.159 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:23:35.159 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.290686+0000", 2024-03-20T09:23:35.159 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0032214570000000001 2024-03-20T09:23:35.159 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.159 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.159 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:23:35.159 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.297127+0000", 2024-03-20T09:23:35.159 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0064408030000000002 2024-03-20T09:23:35.159 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.159 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.159 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:23:35.160 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.297182+0000", 2024-03-20T09:23:35.160 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4962e-05 2024-03-20T09:23:35.160 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.160 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.160 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:23:35.160 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.297220+0000", 2024-03-20T09:23:35.160 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8260000000000003e-05 2024-03-20T09:23:35.160 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.160 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:23:35.160 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.160 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.160 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.160 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212920 4.b 4:df609520:::benchmark_data_smithi122_145698_object92001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:23:35.160 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.384067+0000", 2024-03-20T09:23:35.161 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 377.69562081999999, 2024-03-20T09:23:35.161 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.93677413600000004, 2024-03-20T09:23:35.161 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:23:35.161 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:23:35.161 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:23:35.161 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:23:35.161 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:23:35.161 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212920 2024-03-20T09:23:35.161 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.161 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:23:35.161 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.161 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:23:35.161 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.384067+0000", 2024-03-20T09:23:35.161 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.162 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.162 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.162 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:23:35.162 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.384067+0000", 2024-03-20T09:23:35.162 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.162 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.162 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.162 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:23:35.162 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.384069+0000", 2024-03-20T09:23:35.162 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.854e-06 2024-03-20T09:23:35.162 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.162 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.162 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:23:35.162 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.384077+0000", 2024-03-20T09:23:35.163 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.464e-06 2024-03-20T09:23:35.163 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.163 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.163 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:23:35.163 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.384094+0000", 2024-03-20T09:23:35.163 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7396000000000001e-05 2024-03-20T09:23:35.163 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.163 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.163 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:23:35.163 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.384104+0000", 2024-03-20T09:23:35.163 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9490000000000008e-06 2024-03-20T09:23:35.163 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.163 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.163 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.163 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.384186+0000", 2024-03-20T09:23:35.164 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1691000000000001e-05 2024-03-20T09:23:35.164 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.164 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.164 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:23:35.164 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.384197+0000", 2024-03-20T09:23:35.164 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0913e-05 2024-03-20T09:23:35.164 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.164 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.164 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.164 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.285329+0000", 2024-03-20T09:23:35.164 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.90113206499999998 2024-03-20T09:23:35.164 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.164 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.164 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:23:35.165 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.285405+0000", 2024-03-20T09:23:35.165 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6494000000000003e-05 2024-03-20T09:23:35.165 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.165 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.165 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:23:35.165 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.285477+0000", 2024-03-20T09:23:35.165 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1557999999999995e-05 2024-03-20T09:23:35.165 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.165 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.165 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:23:35.165 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.290733+0000", 2024-03-20T09:23:35.165 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0052559299999999998 2024-03-20T09:23:35.165 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.165 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.165 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:23:35.166 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.290897+0000", 2024-03-20T09:23:35.166 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016413199999999999 2024-03-20T09:23:35.166 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.166 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.166 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:23:35.166 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.320755+0000", 2024-03-20T09:23:35.166 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.029858264999999998 2024-03-20T09:23:35.166 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.166 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.166 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:23:35.166 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.320801+0000", 2024-03-20T09:23:35.166 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5895e-05 2024-03-20T09:23:35.166 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.167 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.167 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:23:35.167 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.320841+0000", 2024-03-20T09:23:35.167 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.053e-05 2024-03-20T09:23:35.167 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.167 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:23:35.167 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.167 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.167 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.167 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212932 4.b 4:d9fbd8b0:::benchmark_data_smithi122_145698_object92013:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:23:35.167 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.407231+0000", 2024-03-20T09:23:35.167 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 377.67245702399998, 2024-03-20T09:23:35.167 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.91372579799999998, 2024-03-20T09:23:35.168 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:23:35.168 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:23:35.168 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:23:35.168 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:23:35.168 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:23:35.168 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212932 2024-03-20T09:23:35.168 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.168 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:23:35.168 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.168 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:23:35.168 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.407231+0000", 2024-03-20T09:23:35.168 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.168 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.168 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.168 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:23:35.169 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.407231+0000", 2024-03-20T09:23:35.169 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.169 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.169 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.169 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:23:35.169 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.407233+0000", 2024-03-20T09:23:35.169 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.624e-06 2024-03-20T09:23:35.169 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.169 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.169 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:23:35.169 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.407239+0000", 2024-03-20T09:23:35.169 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4049999999999997e-06 2024-03-20T09:23:35.169 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.169 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.170 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:23:35.170 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.407253+0000", 2024-03-20T09:23:35.170 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3497e-05 2024-03-20T09:23:35.170 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.170 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.170 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:23:35.170 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.407260+0000", 2024-03-20T09:23:35.170 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7209999999999994e-06 2024-03-20T09:23:35.170 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.170 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.170 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.170 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.407324+0000", 2024-03-20T09:23:35.170 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3841000000000001e-05 2024-03-20T09:23:35.170 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.171 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.171 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:23:35.171 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.407335+0000", 2024-03-20T09:23:35.171 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.111e-05 2024-03-20T09:23:35.171 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.171 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.171 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.171 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.287507+0000", 2024-03-20T09:23:35.171 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.88017162100000002 2024-03-20T09:23:35.171 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.171 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.171 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:23:35.171 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.287619+0000", 2024-03-20T09:23:35.171 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000111566 2024-03-20T09:23:35.172 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.172 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.172 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:23:35.172 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.287738+0000", 2024-03-20T09:23:35.172 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000119543 2024-03-20T09:23:35.172 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.172 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.172 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:23:35.172 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.290939+0000", 2024-03-20T09:23:35.172 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003200984 2024-03-20T09:23:35.172 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.172 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.172 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:23:35.172 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.291076+0000", 2024-03-20T09:23:35.172 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013691899999999999 2024-03-20T09:23:35.173 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.173 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.173 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:23:35.173 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.320890+0000", 2024-03-20T09:23:35.173 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.029814245999999999 2024-03-20T09:23:35.173 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.173 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.173 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:23:35.173 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.320922+0000", 2024-03-20T09:23:35.173 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2128999999999997e-05 2024-03-20T09:23:35.173 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.173 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.173 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:23:35.174 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.320957+0000", 2024-03-20T09:23:35.174 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4591999999999997e-05 2024-03-20T09:23:35.174 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.174 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:23:35.174 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.174 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.174 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.174 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212934 4.b 4:d00108c3:::benchmark_data_smithi122_145698_object92015:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:23:35.174 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.412231+0000", 2024-03-20T09:23:35.174 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 377.66745759100002, 2024-03-20T09:23:35.174 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.90885735899999998, 2024-03-20T09:23:35.174 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:23:35.174 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:23:35.174 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:23:35.175 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:23:35.175 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:23:35.175 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212934 2024-03-20T09:23:35.175 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.175 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:23:35.175 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.175 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:23:35.175 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.412231+0000", 2024-03-20T09:23:35.175 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.175 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.175 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.175 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:23:35.175 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.412231+0000", 2024-03-20T09:23:35.175 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.176 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.176 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.176 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:23:35.176 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.412232+0000", 2024-03-20T09:23:35.176 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9399999999999993e-07 2024-03-20T09:23:35.176 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.176 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.176 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:23:35.176 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.412240+0000", 2024-03-20T09:23:35.176 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8410000000000006e-06 2024-03-20T09:23:35.176 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.176 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.176 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:23:35.176 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.412251+0000", 2024-03-20T09:23:35.176 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1097999999999999e-05 2024-03-20T09:23:35.177 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.177 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.177 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:23:35.177 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.412258+0000", 2024-03-20T09:23:35.177 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5459999999999997e-06 2024-03-20T09:23:35.177 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.177 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.177 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.177 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.412309+0000", 2024-03-20T09:23:35.177 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1066000000000002e-05 2024-03-20T09:23:35.177 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.177 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.177 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:23:35.177 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.412316+0000", 2024-03-20T09:23:35.178 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1029999999999996e-06 2024-03-20T09:23:35.178 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.178 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.178 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.178 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.291236+0000", 2024-03-20T09:23:35.178 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.87891978299999995 2024-03-20T09:23:35.178 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.178 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.178 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:23:35.178 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.291334+0000", 2024-03-20T09:23:35.178 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7737000000000003e-05 2024-03-20T09:23:35.178 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.178 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.179 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:23:35.179 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.291401+0000", 2024-03-20T09:23:35.179 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7621999999999997e-05 2024-03-20T09:23:35.179 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.179 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.179 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:23:35.179 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.297085+0000", 2024-03-20T09:23:35.179 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.005683207 2024-03-20T09:23:35.179 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.179 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.179 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:23:35.179 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.297277+0000", 2024-03-20T09:23:35.179 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00019204000000000001 2024-03-20T09:23:35.180 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.180 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.180 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:23:35.180 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.321001+0000", 2024-03-20T09:23:35.180 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.023724881999999999 2024-03-20T09:23:35.180 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.180 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.180 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:23:35.180 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.321048+0000", 2024-03-20T09:23:35.180 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6252999999999999e-05 2024-03-20T09:23:35.180 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.180 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.180 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:23:35.180 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.321088+0000", 2024-03-20T09:23:35.181 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0187e-05 2024-03-20T09:23:35.181 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.181 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:23:35.181 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.181 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.181 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.181 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212946 4.b 4:d41502f2:::benchmark_data_smithi122_145698_object92027:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:23:35.181 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.427488+0000", 2024-03-20T09:23:35.181 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 377.652200419, 2024-03-20T09:23:35.181 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.89371687399999999, 2024-03-20T09:23:35.181 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:23:35.181 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:23:35.181 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:23:35.181 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:23:35.182 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:23:35.182 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212946 2024-03-20T09:23:35.182 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.182 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:23:35.182 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.182 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:23:35.182 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.427488+0000", 2024-03-20T09:23:35.182 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.182 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.182 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.182 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:23:35.182 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.427488+0000", 2024-03-20T09:23:35.182 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.183 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.183 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.183 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:23:35.183 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.427489+0000", 2024-03-20T09:23:35.183 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.601e-06 2024-03-20T09:23:35.183 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.183 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.183 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:23:35.183 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.427494+0000", 2024-03-20T09:23:35.183 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8980000000000004e-06 2024-03-20T09:23:35.183 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.183 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.183 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:23:35.184 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.427506+0000", 2024-03-20T09:23:35.184 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2071e-05 2024-03-20T09:23:35.184 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.184 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.184 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:23:35.184 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.427513+0000", 2024-03-20T09:23:35.184 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3690000000000003e-06 2024-03-20T09:23:35.184 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.184 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.184 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.184 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.427559+0000", 2024-03-20T09:23:35.184 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.621e-05 2024-03-20T09:23:35.184 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.184 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.185 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:23:35.185 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.427566+0000", 2024-03-20T09:23:35.185 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4379999999999998e-06 2024-03-20T09:23:35.185 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.185 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.185 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.185 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.293406+0000", 2024-03-20T09:23:35.185 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.86583917600000004 2024-03-20T09:23:35.185 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.185 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.185 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:23:35.185 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.293519+0000", 2024-03-20T09:23:35.185 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000113708 2024-03-20T09:23:35.186 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.186 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.186 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:23:35.186 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.293641+0000", 2024-03-20T09:23:35.186 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000121506 2024-03-20T09:23:35.186 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.186 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.186 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:23:35.186 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.297090+0000", 2024-03-20T09:23:35.186 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0034492540000000001 2024-03-20T09:23:35.186 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.186 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.186 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:23:35.186 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.297330+0000", 2024-03-20T09:23:35.187 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000240263 2024-03-20T09:23:35.187 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.187 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.187 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:23:35.187 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.321130+0000", 2024-03-20T09:23:35.187 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.023799415000000001 2024-03-20T09:23:35.187 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.187 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.187 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:23:35.187 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.321174+0000", 2024-03-20T09:23:35.187 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3903000000000003e-05 2024-03-20T09:23:35.187 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.187 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.188 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:23:35.188 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.321205+0000", 2024-03-20T09:23:35.188 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1062000000000002e-05 2024-03-20T09:23:35.188 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.188 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:23:35.188 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.188 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.188 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.188 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212960 4.b 4:d30e7857:::benchmark_data_smithi122_145698_object92041:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:23:35.188 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.451448+0000", 2024-03-20T09:23:35.188 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 377.62823976200002, 2024-03-20T09:23:35.188 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.86988812999999998, 2024-03-20T09:23:35.188 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:23:35.188 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:23:35.189 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:23:35.189 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:23:35.189 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:23:35.189 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212960 2024-03-20T09:23:35.189 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.189 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:23:35.189 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.189 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:23:35.189 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.451448+0000", 2024-03-20T09:23:35.189 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.189 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.189 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.189 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:23:35.189 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.451448+0000", 2024-03-20T09:23:35.190 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.190 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.190 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.190 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:23:35.190 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.451450+0000", 2024-03-20T09:23:35.190 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5829999999999999e-06 2024-03-20T09:23:35.190 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.190 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.190 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:23:35.190 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.451456+0000", 2024-03-20T09:23:35.190 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3389999999999998e-06 2024-03-20T09:23:35.190 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.190 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.191 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:23:35.191 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.451470+0000", 2024-03-20T09:23:35.191 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3434e-05 2024-03-20T09:23:35.191 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.191 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.191 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:23:35.191 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.451477+0000", 2024-03-20T09:23:35.191 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7340000000000008e-06 2024-03-20T09:23:35.191 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.191 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.191 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.191 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.451536+0000", 2024-03-20T09:23:35.191 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8105e-05 2024-03-20T09:23:35.191 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.192 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.192 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:23:35.192 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.451544+0000", 2024-03-20T09:23:35.192 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1799999999999996e-06 2024-03-20T09:23:35.192 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.192 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.192 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.192 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.297377+0000", 2024-03-20T09:23:35.192 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.84583357000000003 2024-03-20T09:23:35.192 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.192 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.192 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:23:35.192 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.297452+0000", 2024-03-20T09:23:35.192 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5074999999999995e-05 2024-03-20T09:23:35.193 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.193 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.193 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:23:35.193 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.297523+0000", 2024-03-20T09:23:35.193 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0479000000000001e-05 2024-03-20T09:23:35.193 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.193 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.193 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:23:35.193 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.304096+0000", 2024-03-20T09:23:35.193 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0065731899999999996 2024-03-20T09:23:35.193 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.193 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.193 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:23:35.193 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.304136+0000", 2024-03-20T09:23:35.194 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9736e-05 2024-03-20T09:23:35.194 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.194 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.194 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:23:35.194 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.321247+0000", 2024-03-20T09:23:35.194 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.017111428000000001 2024-03-20T09:23:35.194 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.194 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.194 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:23:35.194 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.321305+0000", 2024-03-20T09:23:35.194 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7930999999999998e-05 2024-03-20T09:23:35.194 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.194 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.194 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:23:35.195 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.321337+0000", 2024-03-20T09:23:35.195 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1346000000000001e-05 2024-03-20T09:23:35.195 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.195 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:23:35.195 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.195 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.195 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.195 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212965 4.b 4:d1de5eee:::benchmark_data_smithi122_145698_object92046:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:23:35.195 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.460885+0000", 2024-03-20T09:23:35.195 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 377.618802867, 2024-03-20T09:23:35.195 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.86057768800000001, 2024-03-20T09:23:35.195 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:23:35.195 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:23:35.195 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:23:35.196 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:23:35.196 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:23:35.196 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212965 2024-03-20T09:23:35.196 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.196 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:23:35.196 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.196 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:23:35.196 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.460885+0000", 2024-03-20T09:23:35.196 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.196 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.196 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.196 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:23:35.196 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.460885+0000", 2024-03-20T09:23:35.196 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.197 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.197 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.197 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:23:35.197 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.460886+0000", 2024-03-20T09:23:35.197 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.096e-06 2024-03-20T09:23:35.197 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.197 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.197 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:23:35.197 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.460892+0000", 2024-03-20T09:23:35.197 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4539999999999998e-06 2024-03-20T09:23:35.197 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.197 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.197 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:23:35.197 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.460904+0000", 2024-03-20T09:23:35.198 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1807999999999999e-05 2024-03-20T09:23:35.198 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.198 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.198 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:23:35.198 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.460911+0000", 2024-03-20T09:23:35.198 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1010000000000003e-06 2024-03-20T09:23:35.198 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.198 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.198 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.198 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.460968+0000", 2024-03-20T09:23:35.198 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6895000000000003e-05 2024-03-20T09:23:35.198 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.198 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.199 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:23:35.199 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.460974+0000", 2024-03-20T09:23:35.199 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8220000000000001e-06 2024-03-20T09:23:35.199 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.199 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.199 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.199 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.299491+0000", 2024-03-20T09:23:35.199 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.83851649699999997 2024-03-20T09:23:35.199 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.199 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.199 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:23:35.199 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.299569+0000", 2024-03-20T09:23:35.199 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8438999999999999e-05 2024-03-20T09:23:35.199 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.199 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.200 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:23:35.200 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.299641+0000", 2024-03-20T09:23:35.200 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1711999999999994e-05 2024-03-20T09:23:35.200 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.200 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.200 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:23:35.200 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.304103+0000", 2024-03-20T09:23:35.200 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0044617800000000003 2024-03-20T09:23:35.200 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.200 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.200 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:23:35.200 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.304881+0000", 2024-03-20T09:23:35.200 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00077851000000000005 2024-03-20T09:23:35.200 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.200 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.201 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:23:35.201 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.321377+0000", 2024-03-20T09:23:35.201 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016495841000000001 2024-03-20T09:23:35.201 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.201 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.201 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:23:35.201 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.321426+0000", 2024-03-20T09:23:35.201 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8973e-05 2024-03-20T09:23:35.201 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.201 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.201 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:23:35.201 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.321463+0000", 2024-03-20T09:23:35.201 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.676e-05 2024-03-20T09:23:35.201 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.201 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:23:35.202 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.202 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.202 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.202 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:212979 4.b 4:d162c53c:::benchmark_data_smithi122_145698_object92060:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e589)", 2024-03-20T09:23:35.202 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:17:17.530256+0000", 2024-03-20T09:23:35.202 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 377.54943219799998, 2024-03-20T09:23:35.202 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.79561833800000004, 2024-03-20T09:23:35.202 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:23:35.202 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:23:35.202 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:23:35.202 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:23:35.202 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:23:35.202 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 212979 2024-03-20T09:23:35.203 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.203 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:23:35.203 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.203 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:23:35.203 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.530256+0000", 2024-03-20T09:23:35.203 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.203 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.203 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.203 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:23:35.203 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.530256+0000", 2024-03-20T09:23:35.203 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.203 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.203 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.203 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:23:35.204 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.530258+0000", 2024-03-20T09:23:35.204 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.615e-06 2024-03-20T09:23:35.204 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.204 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.204 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:23:35.204 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.530274+0000", 2024-03-20T09:23:35.204 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6748999999999998e-05 2024-03-20T09:23:35.204 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.204 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.204 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:23:35.204 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.530300+0000", 2024-03-20T09:23:35.204 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5465000000000001e-05 2024-03-20T09:23:35.204 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.204 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.204 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:23:35.205 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.530309+0000", 2024-03-20T09:23:35.205 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7800000000000006e-06 2024-03-20T09:23:35.205 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.205 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.205 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.205 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.530372+0000", 2024-03-20T09:23:35.205 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3748e-05 2024-03-20T09:23:35.205 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.205 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.205 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:23:35.205 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:17.530382+0000", 2024-03-20T09:23:35.205 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4809999999999995e-06 2024-03-20T09:23:35.205 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.205 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.206 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.206 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.301707+0000", 2024-03-20T09:23:35.206 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.77132535499999999 2024-03-20T09:23:35.206 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.206 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.206 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:23:35.206 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.301835+0000", 2024-03-20T09:23:35.206 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012744000000000001 2024-03-20T09:23:35.206 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.206 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.206 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,7", 2024-03-20T09:23:35.206 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.301965+0000", 2024-03-20T09:23:35.206 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013019 2024-03-20T09:23:35.207 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.207 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.207 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:23:35.207 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.304214+0000", 2024-03-20T09:23:35.207 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0022491830000000001 2024-03-20T09:23:35.207 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.207 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.207 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:23:35.207 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.304972+0000", 2024-03-20T09:23:35.207 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00075838700000000002 2024-03-20T09:23:35.207 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.207 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.207 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:23:35.207 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.325752+0000", 2024-03-20T09:23:35.208 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.020779410000000002 2024-03-20T09:23:35.208 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.208 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.208 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:23:35.208 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.325808+0000", 2024-03-20T09:23:35.208 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6017999999999998e-05 2024-03-20T09:23:35.208 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.208 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.208 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:23:35.208 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:17:18.325874+0000", 2024-03-20T09:23:35.208 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6517e-05 2024-03-20T09:23:35.208 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.208 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:23:35.209 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.209 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.209 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.209 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25966.0:251632 5.e 5:7b048145:::benchmark_data_smithi122_157894_object121645:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e697)", 2024-03-20T09:23:35.209 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:23:29.872206+0000", 2024-03-20T09:23:35.209 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.207481928, 2024-03-20T09:23:35.209 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.111465986, 2024-03-20T09:23:35.209 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:23:35.209 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:23:35.209 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:23:35.209 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25966", 2024-03-20T09:23:35.209 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:23:35.210 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 251632 2024-03-20T09:23:35.210 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.210 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:23:35.210 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.210 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:23:35.210 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:23:29.872206+0000", 2024-03-20T09:23:35.210 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.210 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.210 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.210 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:23:35.210 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:23:29.872206+0000", 2024-03-20T09:23:35.210 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:23:35.210 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.210 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.211 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:23:35.211 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:23:29.872208+0000", 2024-03-20T09:23:35.211 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7e-06 2024-03-20T09:23:35.211 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.211 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.211 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:23:35.211 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:23:29.872215+0000", 2024-03-20T09:23:35.211 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6510000000000004e-06 2024-03-20T09:23:35.211 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.211 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.211 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:23:35.211 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:23:29.872232+0000", 2024-03-20T09:23:35.211 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7694999999999999e-05 2024-03-20T09:23:35.211 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.211 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.212 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:23:35.212 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:23:29.872242+0000", 2024-03-20T09:23:35.212 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3579999999999992e-06 2024-03-20T09:23:35.212 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.212 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.212 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:23:35.212 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:23:29.872386+0000", 2024-03-20T09:23:35.212 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014478200000000001 2024-03-20T09:23:35.212 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.212 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.212 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:23:35.212 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:23:29.872515+0000", 2024-03-20T09:23:35.212 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012851299999999999 2024-03-20T09:23:35.212 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.213 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.213 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,2", 2024-03-20T09:23:35.213 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:23:29.872607+0000", 2024-03-20T09:23:35.213 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2221999999999999e-05 2024-03-20T09:23:35.213 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.213 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.213 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:23:35.213 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:23:29.876650+0000", 2024-03-20T09:23:35.213 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040430450000000003 2024-03-20T09:23:35.213 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.213 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.213 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:23:35.213 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:23:29.881395+0000", 2024-03-20T09:23:35.213 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.004745099 2024-03-20T09:23:35.214 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.214 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.214 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:23:35.214 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:23:29.983565+0000", 2024-03-20T09:23:35.214 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.10217003099999999 2024-03-20T09:23:35.214 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.214 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.214 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:23:35.214 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:23:29.983625+0000", 2024-03-20T09:23:35.214 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9929999999999997e-05 2024-03-20T09:23:35.214 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:23:35.214 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:23:35.214 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:23:35.214 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:23:29.983672+0000", 2024-03-20T09:23:35.215 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6959999999999998e-05 2024-03-20T09:23:35.215 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.215 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:23:35.215 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.215 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:23:35.215 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:23:35.215 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:23:35.215 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:23:35.251 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:35.354 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:23:35.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:35.359 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:23:35.460 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:23:35.489 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:35.758 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.11 to 1 (10000) 2024-03-20T09:23:35.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:35 smithi026 ceph-mon[68309]: 5.4 scrub starts 2024-03-20T09:23:35.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:35 smithi026 ceph-mon[68309]: 5.4 scrub ok 2024-03-20T09:23:35.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:35 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/988890477' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-03-20T09:23:36.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:35 smithi115 ceph-mon[72476]: 5.4 scrub starts 2024-03-20T09:23:36.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:35 smithi115 ceph-mon[72476]: 5.4 scrub ok 2024-03-20T09:23:36.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:35 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/988890477' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-03-20T09:23:36.195 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:35 smithi122 ceph-mon[73495]: 5.4 scrub starts 2024-03-20T09:23:36.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:35 smithi122 ceph-mon[73495]: 5.4 scrub ok 2024-03-20T09:23:36.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:35 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/988890477' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-03-20T09:23:36.343 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:23:36.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:36.347 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:23:36.370 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_ops_in_flight 2024-03-20T09:23:36.448 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:23:36.494 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:36.738 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:23:36.969 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:36 smithi026 ceph-mon[68309]: 1.17 scrub starts 2024-03-20T09:23:36.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:36 smithi026 ceph-mon[68309]: 1.17 scrub ok 2024-03-20T09:23:36.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:36 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:23:36.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:36 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:23:36.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:36 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/988890477' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-03-20T09:23:36.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:36 smithi026 ceph-mon[68309]: osdmap e698: 12 total, 12 up, 12 in 2024-03-20T09:23:36.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:36 smithi026 ceph-mon[68309]: pgmap v1708: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 347 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 754 op/s; 8034/16653 objects degraded (48.244%); 758/16653 objects misplaced (4.552%); 6.3 MiB/s, 136 objects/s recovering 2024-03-20T09:23:37.006 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:36 smithi122 ceph-mon[73495]: 1.17 scrub starts 2024-03-20T09:23:37.006 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:36 smithi122 ceph-mon[73495]: 1.17 scrub ok 2024-03-20T09:23:37.006 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:36 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:23:37.007 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:36 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:23:37.007 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:36 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/988890477' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-03-20T09:23:37.007 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:36 smithi122 ceph-mon[73495]: osdmap e698: 12 total, 12 up, 12 in 2024-03-20T09:23:37.007 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:36 smithi122 ceph-mon[73495]: pgmap v1708: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 347 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 754 op/s; 8034/16653 objects degraded (48.244%); 758/16653 objects misplaced (4.552%); 6.3 MiB/s, 136 objects/s recovering 2024-03-20T09:23:37.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:36 smithi115 ceph-mon[72476]: 1.17 scrub starts 2024-03-20T09:23:37.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:36 smithi115 ceph-mon[72476]: 1.17 scrub ok 2024-03-20T09:23:37.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:36 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:23:37.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:36 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:23:37.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:36 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/988890477' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-03-20T09:23:37.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:36 smithi115 ceph-mon[72476]: osdmap e698: 12 total, 12 up, 12 in 2024-03-20T09:23:37.157 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:36 smithi115 ceph-mon[72476]: pgmap v1708: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 347 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 754 op/s; 8034/16653 objects degraded (48.244%); 758/16653 objects misplaced (4.552%); 6.3 MiB/s, 136 objects/s recovering 2024-03-20T09:23:37.840 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:23:37.881 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:23:37.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:37.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:23:37.888 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 3 1 2024-03-20T09:23:37.988 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:23:38.021 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:37 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8034/16653 objects degraded (48.244%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-20T09:23:38.021 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:37 smithi122 ceph-mon[73495]: osdmap e699: 12 total, 12 up, 12 in 2024-03-20T09:23:38.024 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:38.130 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:23:38.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:38.140 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:23:38.153 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:37 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8034/16653 objects degraded (48.244%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-20T09:23:38.153 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:37 smithi026 ceph-mon[68309]: osdmap e699: 12 total, 12 up, 12 in 2024-03-20T09:23:38.160 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:37 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8034/16653 objects degraded (48.244%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-20T09:23:38.160 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:37 smithi115 ceph-mon[72476]: osdmap e699: 12 total, 12 up, 12 in 2024-03-20T09:23:38.240 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:23:38.274 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:38.426 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:23:38.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:38.434 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:23:38.535 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:23:38.600 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:38.633 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:23:38.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:38.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:23:38.742 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:23:38.782 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:23:38.783 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:23:38.783 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-03-20T09:23:38.783 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:23:38.792 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:38.817 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:23:38.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:38.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:23:38.871 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:23:38.895 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:23:38.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:38 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:23:38.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:38 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:23:38.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:38 smithi115 ceph-mon[72476]: pgmap v1710: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 260 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 744 op/s; 8061/12465 objects degraded (64.669%); 515/12465 objects misplaced (4.132%); 4.1 MiB/s, 105 objects/s recovering 2024-03-20T09:23:38.918 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:23:38.930 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:23:38.945 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:23:38.961 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:38.973 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:23:38.995 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:23:38.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:38.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:23:38.999 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:23:39.024 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:23:39.047 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:23:39.100 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:23:39.123 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:23:39.124 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:23:39.124 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:23:39.131 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:38 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:23:39.132 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:38 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:23:39.132 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:38 smithi026 ceph-mon[68309]: pgmap v1710: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 260 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 744 op/s; 8061/12465 objects degraded (64.669%); 515/12465 objects misplaced (4.132%); 4.1 MiB/s, 105 objects/s recovering 2024-03-20T09:23:39.134 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:39.139 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:23:39.141 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:23:39.141 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:23:39.141 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:23:39.141 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:23:39.141 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:23:39.141 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:23:39.141 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:23:39.141 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:23:39.141 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:23:39.141 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:23:39.141 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:23:39.141 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:23:39.146 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:38 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:23:39.146 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:38 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:23:39.146 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:38 smithi122 ceph-mon[73495]: pgmap v1710: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 260 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 744 op/s; 8061/12465 objects degraded (64.669%); 515/12465 objects misplaced (4.132%); 4.1 MiB/s, 105 objects/s recovering 2024-03-20T09:23:39.478 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_blocked_ops 2024-03-20T09:23:39.740 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:23:40.159 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:39 smithi115 ceph-mon[72476]: 1.1c scrub starts 2024-03-20T09:23:40.159 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:39 smithi115 ceph-mon[72476]: 1.1c scrub ok 2024-03-20T09:23:40.159 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:39 smithi115 ceph-mon[72476]: osdmap e700: 12 total, 12 up, 12 in 2024-03-20T09:23:40.159 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:39 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2301560382' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:23:40.195 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:39 smithi122 ceph-mon[73495]: 1.1c scrub starts 2024-03-20T09:23:40.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:39 smithi122 ceph-mon[73495]: 1.1c scrub ok 2024-03-20T09:23:40.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:39 smithi122 ceph-mon[73495]: osdmap e700: 12 total, 12 up, 12 in 2024-03-20T09:23:40.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:39 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2301560382' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:23:40.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:39 smithi026 ceph-mon[68309]: 1.1c scrub starts 2024-03-20T09:23:40.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:39 smithi026 ceph-mon[68309]: 1.1c scrub ok 2024-03-20T09:23:40.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:39 smithi026 ceph-mon[68309]: osdmap e700: 12 total, 12 up, 12 in 2024-03-20T09:23:40.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:39 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2301560382' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:23:41.086 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:40 smithi122 ceph-mon[73495]: osdmap e701: 12 total, 12 up, 12 in 2024-03-20T09:23:41.086 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:40 smithi122 ceph-mon[73495]: pgmap v1713: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 184 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 688 op/s; 8211/8823 objects degraded (93.064%); 260/8823 objects misplaced (2.947%); 4.5 MiB/s, 72 objects/s recovering 2024-03-20T09:23:41.160 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:40 smithi115 ceph-mon[72476]: osdmap e701: 12 total, 12 up, 12 in 2024-03-20T09:23:41.160 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:40 smithi115 ceph-mon[72476]: pgmap v1713: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 184 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 688 op/s; 8211/8823 objects degraded (93.064%); 260/8823 objects misplaced (2.947%); 4.5 MiB/s, 72 objects/s recovering 2024-03-20T09:23:41.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:40 smithi026 ceph-mon[68309]: osdmap e701: 12 total, 12 up, 12 in 2024-03-20T09:23:41.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:40 smithi026 ceph-mon[68309]: pgmap v1713: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 184 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 688 op/s; 8211/8823 objects degraded (93.064%); 260/8823 objects misplaced (2.947%); 4.5 MiB/s, 72 objects/s recovering 2024-03-20T09:23:41.340 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:23:41.388 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:23:41.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:41.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:23:41.502 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:23:41.561 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:41.595 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:23:41.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:41.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:23:41.705 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:23:41.708 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:23:41.708 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:23:41.744 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:41.770 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:23:41.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:41.774 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:23:41.875 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:23:41.878 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:41 smithi026 ceph-mon[68309]: 1.15 deep-scrub starts 2024-03-20T09:23:41.878 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:41 smithi026 ceph-mon[68309]: 1.15 deep-scrub ok 2024-03-20T09:23:41.917 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:41.937 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:23:41.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:41.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:23:41.957 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Removed 129984 objects 2024-03-20T09:23:41.957 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Clean up completed and total clean up time :228.319 2024-03-20T09:23:42.041 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:23:42.066 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:41 smithi122 ceph-mon[73495]: 1.15 deep-scrub starts 2024-03-20T09:23:42.066 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:41 smithi122 ceph-mon[73495]: 1.15 deep-scrub ok 2024-03-20T09:23:42.068 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:42.087 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:23:42.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:42.090 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:23:42.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:41 smithi115 ceph-mon[72476]: 1.15 deep-scrub starts 2024-03-20T09:23:42.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:41 smithi115 ceph-mon[72476]: 1.15 deep-scrub ok 2024-03-20T09:23:42.191 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:23:42.219 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:42.240 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:23:42.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:42.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:23:42.344 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:23:42.372 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:43.088 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:42 smithi122 ceph-mon[73495]: pgmap v1714: 47 pgs: 1 active+clean+scrubbing+deep, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 83 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 704 op/s; 7953/3963 objects degraded (200.681%); 260/3963 objects misplaced (6.561%); 3.3 MiB/s, 96 objects/s recovering 2024-03-20T09:23:43.089 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:42 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 8211/8823 objects degraded (93.064%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-20T09:23:43.116 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_historic_ops 2024-03-20T09:23:43.121 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:23:43.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:43.125 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:23:43.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:42 smithi115 ceph-mon[72476]: pgmap v1714: 47 pgs: 1 active+clean+scrubbing+deep, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 83 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 704 op/s; 7953/3963 objects degraded (200.681%); 260/3963 objects misplaced (6.561%); 3.3 MiB/s, 96 objects/s recovering 2024-03-20T09:23:43.155 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:42 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 8211/8823 objects degraded (93.064%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-20T09:23:43.219 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:42 smithi026 ceph-mon[68309]: pgmap v1714: 47 pgs: 1 active+clean+scrubbing+deep, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 83 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 704 op/s; 7953/3963 objects degraded (200.681%); 260/3963 objects misplaced (6.561%); 3.3 MiB/s, 96 objects/s recovering 2024-03-20T09:23:43.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:42 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 8211/8823 objects degraded (93.064%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-20T09:23:43.226 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:23:43.255 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:43.276 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:23:43.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:43.280 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:23:43.381 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:23:43.393 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:23:43.408 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:43.430 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:23:43.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:43.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:23:43.534 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:23:43.559 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:43.658 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:23:43.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:43.662 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:23:43.763 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:23:43.796 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:43.876 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-03-20T09:23:43.876 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-03-20T09:23:43.947 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:23:43.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:43.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:23:44.052 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:23:44.078 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:44.100 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:23:44.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:44.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:23:44.205 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:23:44.240 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:44.391 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:23:44.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:44.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:23:44.479 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:23:44.495 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:23:44.523 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:44.542 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:23:44.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:44.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:23:44.646 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:23:44.675 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:44.694 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:23:44.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:44.697 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:23:44.798 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:23:44.834 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:44.935 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:23:44.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:44.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:23:45.040 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:23:45.069 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:45.209 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:23:45.218 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:23:45.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:45.222 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:23:45.259 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:23:45.259 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:23:45.323 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:23:45.351 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:45.469 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:45 smithi026 ceph-mon[68309]: pgmap v1715: 47 pgs: 1 active+clean+scrubbing+deep, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 20 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 670 op/s; 7400/951 objects degraded (778.128%); 5.1 MiB/s, 205 objects/s recovering 2024-03-20T09:23:45.469 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:45 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2989376079' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:23:45.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:45 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:23:45.499 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:23:45.500 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:45 smithi122 ceph-mon[73495]: pgmap v1715: 47 pgs: 1 active+clean+scrubbing+deep, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 20 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 670 op/s; 7400/951 objects degraded (778.128%); 5.1 MiB/s, 205 objects/s recovering 2024-03-20T09:23:45.501 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:45 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2989376079' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:23:45.501 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:45 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:23:45.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:45.502 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:23:45.603 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:23:45.632 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:45.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:45 smithi115 ceph-mon[72476]: pgmap v1715: 47 pgs: 1 active+clean+scrubbing+deep, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 20 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 670 op/s; 7400/951 objects degraded (778.128%); 5.1 MiB/s, 205 objects/s recovering 2024-03-20T09:23:45.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:45 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2989376079' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:23:45.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:45 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:23:45.922 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_ops_in_flight 2024-03-20T09:23:46.180 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:23:46.258 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:23:46.469 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:46 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:23:46.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:46 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:23:46.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:46 smithi026 ceph-mon[68309]: osdmap e702: 12 total, 12 up, 12 in 2024-03-20T09:23:46.487 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:46 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:23:46.487 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:46 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:23:46.487 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:46 smithi122 ceph-mon[73495]: osdmap e702: 12 total, 12 up, 12 in 2024-03-20T09:23:46.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:46 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:23:46.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:46 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:23:46.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:46 smithi115 ceph-mon[72476]: osdmap e702: 12 total, 12 up, 12 in 2024-03-20T09:23:47.445 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:47 smithi122 ceph-mon[73495]: pgmap v1717: 47 pgs: 1 active+clean+scrubbing+deep, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.6 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 395 op/s; 7400/105 objects degraded (7047.619%); 2.0 MiB/s, 145 objects/s recovering 2024-03-20T09:23:47.445 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:47 smithi122 ceph-mon[73495]: osdmap e703: 12 total, 12 up, 12 in 2024-03-20T09:23:47.469 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:47 smithi026 ceph-mon[68309]: pgmap v1717: 47 pgs: 1 active+clean+scrubbing+deep, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.6 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 395 op/s; 7400/105 objects degraded (7047.619%); 2.0 MiB/s, 145 objects/s recovering 2024-03-20T09:23:47.469 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:47 smithi026 ceph-mon[68309]: osdmap e703: 12 total, 12 up, 12 in 2024-03-20T09:23:47.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:47 smithi115 ceph-mon[72476]: pgmap v1717: 47 pgs: 1 active+clean+scrubbing+deep, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.6 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 395 op/s; 7400/105 objects degraded (7047.619%); 2.0 MiB/s, 145 objects/s recovering 2024-03-20T09:23:47.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:47 smithi115 ceph-mon[72476]: osdmap e703: 12 total, 12 up, 12 in 2024-03-20T09:23:48.027 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:23:48.027 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:23:48.354 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:48 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 7400/105 objects degraded (7047.619%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:23:48.469 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:48 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 7400/105 objects degraded (7047.619%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:23:48.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:48 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 7400/105 objects degraded (7047.619%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:23:48.661 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_blocked_ops 2024-03-20T09:23:48.923 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:23:49.445 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:49 smithi122 ceph-mon[73495]: pgmap v1719: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 367 op/s; 6770/6 objects degraded (112833.333%); 1.8 MiB/s, 212 objects/s recovering 2024-03-20T09:23:49.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:49 smithi026 ceph-mon[68309]: pgmap v1719: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 367 op/s; 6770/6 objects degraded (112833.333%); 1.8 MiB/s, 212 objects/s recovering 2024-03-20T09:23:49.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:49 smithi115 ceph-mon[72476]: pgmap v1719: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 367 op/s; 6770/6 objects degraded (112833.333%); 1.8 MiB/s, 212 objects/s recovering 2024-03-20T09:23:50.316 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:23:50.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:50.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:23:50.421 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:23:50.458 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:50.600 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:23:50.601 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:23:50.723 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:23:50.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:50.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:23:50.850 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:23:50.880 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:50.953 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:23:50.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:50.968 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:23:51.069 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:23:51.111 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:51.276 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_historic_ops 2024-03-20T09:23:51.302 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:51 smithi026 ceph-mon[68309]: pgmap v1720: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 164 op/s; 6399/6 objects degraded (106650.000%); 1.8 MiB/s, 226 objects/s recovering 2024-03-20T09:23:51.535 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:23:51.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:51 smithi115 ceph-mon[72476]: pgmap v1720: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 164 op/s; 6399/6 objects degraded (106650.000%); 1.8 MiB/s, 226 objects/s recovering 2024-03-20T09:23:51.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:51 smithi122 ceph-mon[73495]: pgmap v1720: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 164 op/s; 6399/6 objects degraded (106650.000%); 1.8 MiB/s, 226 objects/s recovering 2024-03-20T09:23:51.961 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:23:51.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:51.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:23:52.065 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:23:52.092 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:52.111 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:23:52.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:52.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:23:52.214 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:23:52.241 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:52.260 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:23:52.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:52.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:23:52.302 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:23:52.364 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:23:52.391 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:52.430 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:23:52.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:52.436 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:23:52.537 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:23:52.564 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:52.582 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:23:52.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:52.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:23:52.686 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:23:52.712 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:52.730 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:23:52.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:52.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:23:52.834 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:23:52.862 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:53.061 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:23:53.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:53.069 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:23:53.170 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:23:53.203 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:53.222 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:23:53.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:53.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:23:53.325 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:23:53.329 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:23:53.329 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:23:53.354 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:53 smithi122 ceph-mon[73495]: pgmap v1721: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 39 op/s; 6399/6 objects degraded (106650.000%); 0 B/s, 125 objects/s recovering 2024-03-20T09:23:53.354 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:53 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 6399/6 objects degraded (106650.000%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:23:53.354 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:53 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4288024276' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-20T09:23:53.354 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:53 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-20T09:23:53.354 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:53 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/409241004' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-20T09:23:53.354 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:53 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-20T09:23:53.356 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:53.377 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.3 to 1 (10000) 2024-03-20T09:23:53.384 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'unique_pool_3' removed 2024-03-20T09:23:53.558 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:23:53.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:53.561 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:23:53.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:53 smithi115 ceph-mon[72476]: pgmap v1721: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 39 op/s; 6399/6 objects degraded (106650.000%); 0 B/s, 125 objects/s recovering 2024-03-20T09:23:53.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:53 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 6399/6 objects degraded (106650.000%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:23:53.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:53 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4288024276' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-20T09:23:53.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:53 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-20T09:23:53.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:53 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/409241004' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-20T09:23:53.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:53 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-20T09:23:53.662 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:23:53.690 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:53.710 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:23:53.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:53.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:23:53.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:53 smithi026 ceph-mon[68309]: pgmap v1721: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 39 op/s; 6399/6 objects degraded (106650.000%); 0 B/s, 125 objects/s recovering 2024-03-20T09:23:53.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:53 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 6399/6 objects degraded (106650.000%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-20T09:23:53.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:53 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4288024276' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-20T09:23:53.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:53 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-20T09:23:53.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:53 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/409241004' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-20T09:23:53.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:53 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-20T09:23:53.813 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:23:53.851 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:23:53.852 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:54.003 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:23:54.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:54.007 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:23:54.018 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_ops_in_flight 2024-03-20T09:23:54.107 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:23:54.134 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:54.153 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:23:54.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:54.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:23:54.257 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:23:54.284 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:54.302 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:23:54.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:54.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:23:54.315 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:23:54.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:54 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-20T09:23:54.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:54 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-20T09:23:54.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:54 smithi115 ceph-mon[72476]: osdmap e704: 12 total, 12 up, 12 in 2024-03-20T09:23:54.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:54 smithi115 ceph-mon[72476]: pgmap v1723: 31 pgs: 31 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:23:54.405 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:23:54.432 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:54.450 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:23:54.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:54.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:23:54.554 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:23:54.580 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:54 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-20T09:23:54.581 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:54 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-20T09:23:54.581 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:54 smithi026 ceph-mon[68309]: osdmap e704: 12 total, 12 up, 12 in 2024-03-20T09:23:54.581 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:54 smithi026 ceph-mon[68309]: pgmap v1723: 31 pgs: 31 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:23:54.583 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:54.637 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:23:54.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:54.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:23:54.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:54 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-20T09:23:54.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:54 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-20T09:23:54.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:54 smithi122 ceph-mon[73495]: osdmap e704: 12 total, 12 up, 12 in 2024-03-20T09:23:54.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:54 smithi122 ceph-mon[73495]: pgmap v1723: 31 pgs: 31 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:23:54.741 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:23:54.767 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:54.785 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:23:54.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:54.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:23:54.889 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:23:54.916 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:54.934 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:23:54.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:54.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:23:55.038 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:23:55.064 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:55.082 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:23:55.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:55.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:23:55.186 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:23:55.214 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:55.462 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:23:55.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:55.466 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:23:55.567 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:23:55.597 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:55.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:55 smithi115 ceph-mon[72476]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6399/6 objects degraded (106650.000%), 1 pg degraded, 1 pg undersized) 2024-03-20T09:23:55.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:55 smithi122 ceph-mon[73495]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6399/6 objects degraded (106650.000%), 1 pg degraded, 1 pg undersized) 2024-03-20T09:23:55.695 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:23:55.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:55.699 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:23:55.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:55 smithi026 ceph-mon[68309]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6399/6 objects degraded (106650.000%), 1 pg degraded, 1 pg undersized) 2024-03-20T09:23:55.785 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:23:55.800 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:23:55.833 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:55.854 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:23:55.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:55.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:23:55.960 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:23:55.964 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:23:55.964 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:23:55.990 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:56.010 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:23:56.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:56.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:23:56.114 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:23:56.144 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:56.182 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:23:56.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:56.186 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:23:56.287 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:23:56.317 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:56.337 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:23:56.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:56.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:23:56.441 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:23:56.467 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:56.591 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:56 smithi122 ceph-mon[73495]: pgmap v1724: 31 pgs: 31 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:23:56.600 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:23:56.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:56.603 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:23:56.617 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_blocked_ops 2024-03-20T09:23:56.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:56 smithi115 ceph-mon[72476]: pgmap v1724: 31 pgs: 31 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:23:56.705 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:23:56.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:56 smithi026 ceph-mon[68309]: pgmap v1724: 31 pgs: 31 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:23:56.734 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:56.753 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:23:56.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:56.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:23:56.857 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:23:56.886 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:56.905 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:23:56.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:56.908 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:23:56.919 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:23:57.009 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:23:57.039 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:57.059 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:23:57.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:57.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:23:57.167 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:23:57.205 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:57.231 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:23:57.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:57.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:23:57.336 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:23:57.368 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:57.390 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:23:57.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:57.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:23:57.494 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:23:57.525 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:57.818 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:23:57.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:23:57.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:23:57.850 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-20T09:23:57.851 INFO:tasks.radosbench:Beginning radosbench... 2024-03-20T09:23:57.851 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-03-20T09:23:57.851 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pool create unique_pool_4 16 2024-03-20T09:23:57.923 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:23:57.962 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:23:58.445 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:23:58.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:58 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:23:58.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:58 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:23:58.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:58 smithi122 ceph-mon[73495]: osdmap e705: 12 total, 12 up, 12 in 2024-03-20T09:23:58.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:58 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:23:58.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:58 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:23:58.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:58 smithi122 ceph-mon[73495]: pgmap v1726: 31 pgs: 31 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:23:58.465 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:23:58.469 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:58 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:23:58.469 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:58 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:23:58.469 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:58 smithi026 ceph-mon[68309]: osdmap e705: 12 total, 12 up, 12 in 2024-03-20T09:23:58.469 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:58 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:23:58.469 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:58 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:23:58.469 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:58 smithi026 ceph-mon[68309]: pgmap v1726: 31 pgs: 31 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:23:58.485 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:23:58.506 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:23:58.526 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:23:58.547 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:23:58.568 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:23:58.589 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:23:58.609 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:23:58.630 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:23:58.649 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:23:58.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:58 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:23:58.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:58 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:23:58.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:58 smithi115 ceph-mon[72476]: osdmap e705: 12 total, 12 up, 12 in 2024-03-20T09:23:58.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:58 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:23:58.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:58 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:23:58.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:58 smithi115 ceph-mon[72476]: pgmap v1726: 31 pgs: 31 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:23:58.670 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:23:58.671 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:23:58.671 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:23:58.671 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:23:58.671 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:23:58.671 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:23:58.671 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:23:58.671 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:23:58.671 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:23:58.671 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:23:58.671 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:23:58.671 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:23:58.671 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:23:58.705 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:23:58.705 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:23:58.705 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-03-20T09:23:58.705 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-03-20T09:23:58.705 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:23:58.975 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 4 1 2024-03-20T09:23:59.381 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:23:59.441 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_historic_ops 2024-03-20T09:23:59.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:59 smithi115 ceph-mon[72476]: osdmap e706: 12 total, 12 up, 12 in 2024-03-20T09:23:59.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:59 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2053808880' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:23:59.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:59 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:23:59.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:59 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:23:59.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:23:59 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4001021004' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:23:59.668 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:59 smithi026 ceph-mon[68309]: osdmap e706: 12 total, 12 up, 12 in 2024-03-20T09:23:59.669 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:59 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2053808880' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:23:59.669 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:59 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:23:59.669 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:59 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:23:59.669 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:23:59 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4001021004' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:23:59.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:59 smithi122 ceph-mon[73495]: osdmap e706: 12 total, 12 up, 12 in 2024-03-20T09:23:59.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:59 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2053808880' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:23:59.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:59 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:23:59.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:59 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:23:59.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:23:59 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4001021004' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:23:59.739 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:23:59.836 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:24:00.616 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:00 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:24:00.617 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:00 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4001021004' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:24:00.617 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:00 smithi122 ceph-mon[73495]: osdmap e707: 12 total, 12 up, 12 in 2024-03-20T09:24:00.617 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:00 smithi122 ceph-mon[73495]: pgmap v1729: 31 pgs: 31 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:24:00.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:00 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:24:00.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:00 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4001021004' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:24:00.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:00 smithi115 ceph-mon[72476]: osdmap e707: 12 total, 12 up, 12 in 2024-03-20T09:24:00.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:00 smithi115 ceph-mon[72476]: pgmap v1729: 31 pgs: 31 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:24:00.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:00 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:24:00.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:00 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4001021004' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:24:00.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:00 smithi026 ceph-mon[68309]: osdmap e707: 12 total, 12 up, 12 in 2024-03-20T09:24:00.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:00 smithi026 ceph-mon[68309]: pgmap v1729: 31 pgs: 31 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:24:00.884 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:24:01.314 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:24:01.318 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:01.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:24:01.419 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:24:01.448 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:01.501 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:24:01.503 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:24:01.503 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-03-20T09:24:01.503 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-03-20T09:24:01.503 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-03-20T09:24:01.503 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.503 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:126534 4.9 4:9db3ebef:::benchmark_data_smithi122_145698_object5615:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e558)", 2024-03-20T09:24:01.503 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:15:19.415329+0000", 2024-03-20T09:24:01.503 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 522.086379943, 2024-03-20T09:24:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18024907000000001, 2024-03-20T09:24:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:24:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:24:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:24:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:24:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:24:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 126534 2024-03-20T09:24:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:24:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:24:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.415329+0000", 2024-03-20T09:24:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.504 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:24:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.415329+0000", 2024-03-20T09:24:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:24:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.415329+0000", 2024-03-20T09:24:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.51e-07 2024-03-20T09:24:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:24:01.505 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.415332+0000", 2024-03-20T09:24:01.506 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7140000000000002e-06 2024-03-20T09:24:01.506 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.506 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.506 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:24:01.506 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.415341+0000", 2024-03-20T09:24:01.506 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2529999999999993e-06 2024-03-20T09:24:01.506 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.506 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.506 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:24:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.415346+0000", 2024-03-20T09:24:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7910000000000004e-06 2024-03-20T09:24:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:24:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.444078+0000", 2024-03-20T09:24:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.028731982 2024-03-20T09:24:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:24:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.444168+0000", 2024-03-20T09:24:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9967000000000001e-05 2024-03-20T09:24:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.507 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,3", 2024-03-20T09:24:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.444253+0000", 2024-03-20T09:24:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4370999999999996e-05 2024-03-20T09:24:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:24:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.448686+0000", 2024-03-20T09:24:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0044338169999999996 2024-03-20T09:24:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.510358+0000", 2024-03-20T09:24:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.061671436000000003 2024-03-20T09:24:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.595470+0000", 2024-03-20T09:24:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.085112159000000007 2024-03-20T09:24:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:24:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.595521+0000", 2024-03-20T09:24:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0773999999999999e-05 2024-03-20T09:24:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:24:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.595578+0000", 2024-03-20T09:24:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7255e-05 2024-03-20T09:24:01.510 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.510 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:24:01.510 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.510 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.510 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.510 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:126626 4.9 4:981472c5:::benchmark_data_smithi122_145698_object5707:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e558)", 2024-03-20T09:24:01.510 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:15:19.785873+0000", 2024-03-20T09:24:01.510 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 521.71583573999999, 2024-03-20T09:24:01.510 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.22007701700000001, 2024-03-20T09:24:01.510 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:24:01.510 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:24:01.510 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:24:01.510 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:24:01.510 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:24:01.511 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 126626 2024-03-20T09:24:01.511 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.511 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:24:01.511 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.511 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:24:01.511 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.785873+0000", 2024-03-20T09:24:01.511 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.511 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.511 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.511 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:24:01.512 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.785873+0000", 2024-03-20T09:24:01.512 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.512 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.512 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.512 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:24:01.512 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.785873+0000", 2024-03-20T09:24:01.512 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6600000000000003e-07 2024-03-20T09:24:01.512 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.513 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.513 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:24:01.513 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.785875+0000", 2024-03-20T09:24:01.513 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5e-06 2024-03-20T09:24:01.513 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.513 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.513 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:24:01.513 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.785878+0000", 2024-03-20T09:24:01.514 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.067e-06 2024-03-20T09:24:01.514 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.514 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.514 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:24:01.514 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.785881+0000", 2024-03-20T09:24:01.514 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7669999999999999e-06 2024-03-20T09:24:01.514 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.514 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.514 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:24:01.514 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.858640+0000", 2024-03-20T09:24:01.514 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.072759708000000006 2024-03-20T09:24:01.515 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.515 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.515 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:24:01.515 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.858776+0000", 2024-03-20T09:24:01.515 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013547599999999999 2024-03-20T09:24:01.515 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.515 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.515 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,3", 2024-03-20T09:24:01.515 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.858864+0000", 2024-03-20T09:24:01.515 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8318000000000004e-05 2024-03-20T09:24:01.515 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.515 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.516 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:24:01.516 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.862399+0000", 2024-03-20T09:24:01.516 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0035345509999999999 2024-03-20T09:24:01.516 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.516 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.516 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.516 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:19.921710+0000", 2024-03-20T09:24:01.516 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.059311244999999999 2024-03-20T09:24:01.516 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.516 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.516 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.516 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.005858+0000", 2024-03-20T09:24:01.516 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.084148223999999994 2024-03-20T09:24:01.517 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.517 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.517 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:24:01.517 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.005896+0000", 2024-03-20T09:24:01.517 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7706000000000001e-05 2024-03-20T09:24:01.517 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.518 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.518 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:24:01.518 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.005950+0000", 2024-03-20T09:24:01.518 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4188999999999999e-05 2024-03-20T09:24:01.518 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.518 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:24:01.518 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.518 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.518 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.518 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:126849 4.9 4:9229c909:::benchmark_data_smithi122_145698_object5930:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e558)", 2024-03-20T09:24:01.518 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:15:20.540259+0000", 2024-03-20T09:24:01.518 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 520.96145016699995, 2024-03-20T09:24:01.519 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.201706686, 2024-03-20T09:24:01.519 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:24:01.519 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:24:01.519 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:24:01.519 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:24:01.519 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:24:01.520 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 126849 2024-03-20T09:24:01.520 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.520 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:24:01.520 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.520 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:24:01.521 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.540259+0000", 2024-03-20T09:24:01.521 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.521 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.521 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.521 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:24:01.521 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.540259+0000", 2024-03-20T09:24:01.521 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.521 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.521 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.521 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:24:01.521 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.540260+0000", 2024-03-20T09:24:01.521 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8169999999999999e-06 2024-03-20T09:24:01.521 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.521 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.522 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:24:01.522 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.540267+0000", 2024-03-20T09:24:01.522 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0310000000000004e-06 2024-03-20T09:24:01.522 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.522 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.522 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:24:01.522 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.540290+0000", 2024-03-20T09:24:01.522 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3688000000000001e-05 2024-03-20T09:24:01.522 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.522 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.522 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:24:01.522 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.540299+0000", 2024-03-20T09:24:01.522 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4829999999999999e-06 2024-03-20T09:24:01.523 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.523 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.523 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:24:01.523 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.612395+0000", 2024-03-20T09:24:01.523 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.072096439999999998 2024-03-20T09:24:01.523 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.523 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.523 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:24:01.523 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.612531+0000", 2024-03-20T09:24:01.523 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013567000000000001 2024-03-20T09:24:01.523 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.523 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.523 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,3", 2024-03-20T09:24:01.523 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.612673+0000", 2024-03-20T09:24:01.524 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014193499999999999 2024-03-20T09:24:01.524 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.524 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.524 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:24:01.524 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.626044+0000", 2024-03-20T09:24:01.524 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.013371450999999999 2024-03-20T09:24:01.524 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.524 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.524 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.524 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.737807+0000", 2024-03-20T09:24:01.524 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.111762402 2024-03-20T09:24:01.524 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.524 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.524 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.525 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.741860+0000", 2024-03-20T09:24:01.525 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040529449999999996 2024-03-20T09:24:01.525 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.525 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.525 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:24:01.525 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.741914+0000", 2024-03-20T09:24:01.525 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4236e-05 2024-03-20T09:24:01.525 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.525 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.525 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:24:01.525 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.741965+0000", 2024-03-20T09:24:01.525 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1588000000000001e-05 2024-03-20T09:24:01.525 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.525 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:24:01.526 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.526 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.526 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.526 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25690.0:126854 4.9 4:934a0522:::benchmark_data_smithi122_145698_object5935:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e558)", 2024-03-20T09:24:01.526 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:15:20.554985+0000", 2024-03-20T09:24:01.526 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 520.94672370499995, 2024-03-20T09:24:01.526 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18721818900000001, 2024-03-20T09:24:01.526 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:24:01.526 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:24:01.526 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:24:01.526 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25690", 2024-03-20T09:24:01.526 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:24:01.526 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 126854 2024-03-20T09:24:01.526 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.527 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:24:01.527 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.527 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:24:01.527 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.554985+0000", 2024-03-20T09:24:01.527 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.527 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.527 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.527 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:24:01.527 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.554985+0000", 2024-03-20T09:24:01.527 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.527 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.527 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.527 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:24:01.527 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.554987+0000", 2024-03-20T09:24:01.528 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5009999999999999e-06 2024-03-20T09:24:01.528 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.528 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.528 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:24:01.528 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.554993+0000", 2024-03-20T09:24:01.528 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9259999999999997e-06 2024-03-20T09:24:01.528 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.528 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.528 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:24:01.528 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.555006+0000", 2024-03-20T09:24:01.528 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.327e-05 2024-03-20T09:24:01.528 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.528 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.528 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:24:01.529 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.555014+0000", 2024-03-20T09:24:01.529 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5699999999999993e-06 2024-03-20T09:24:01.529 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.529 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.529 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:24:01.529 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.616851+0000", 2024-03-20T09:24:01.529 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.061836968999999999 2024-03-20T09:24:01.529 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.529 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.529 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:24:01.529 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.616986+0000", 2024-03-20T09:24:01.529 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000134203 2024-03-20T09:24:01.529 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.529 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.530 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,3", 2024-03-20T09:24:01.530 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.617130+0000", 2024-03-20T09:24:01.530 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014442599999999999 2024-03-20T09:24:01.530 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.530 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.530 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:24:01.530 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.626053+0000", 2024-03-20T09:24:01.530 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0089233920000000005 2024-03-20T09:24:01.530 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.531 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.531 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.531 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.737846+0000", 2024-03-20T09:24:01.531 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.11179288800000001 2024-03-20T09:24:01.531 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.531 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.531 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.531 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.742083+0000", 2024-03-20T09:24:01.531 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0042367350000000002 2024-03-20T09:24:01.531 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.531 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.531 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:24:01.531 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.742142+0000", 2024-03-20T09:24:01.532 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9302999999999998e-05 2024-03-20T09:24:01.532 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.532 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.532 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:24:01.532 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:15:20.742203+0000", 2024-03-20T09:24:01.532 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1005999999999998e-05 2024-03-20T09:24:01.532 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.532 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:24:01.532 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.532 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.532 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.532 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25966.0:8055 5.8 5:107d6b13:::benchmark_data_smithi122_157894_object8054:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e611)", 2024-03-20T09:24:01.532 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:18:22.889240+0000", 2024-03-20T09:24:01.532 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 338.61246888199997, 2024-03-20T09:24:01.532 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20490609900000001, 2024-03-20T09:24:01.533 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:24:01.533 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:24:01.533 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:24:01.533 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25966", 2024-03-20T09:24:01.533 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:24:01.533 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 8055 2024-03-20T09:24:01.533 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.533 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:24:01.533 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.533 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:24:01.533 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.889240+0000", 2024-03-20T09:24:01.533 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.533 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.533 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.533 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:24:01.534 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.889240+0000", 2024-03-20T09:24:01.534 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.534 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.534 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.534 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:24:01.534 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.889241+0000", 2024-03-20T09:24:01.534 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3900000000000004e-07 2024-03-20T09:24:01.534 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.534 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.534 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:24:01.534 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.889262+0000", 2024-03-20T09:24:01.534 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1622999999999999e-05 2024-03-20T09:24:01.535 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.535 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.535 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:24:01.535 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.889273+0000", 2024-03-20T09:24:01.535 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1019000000000001e-05 2024-03-20T09:24:01.535 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.535 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.535 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:24:01.535 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.889281+0000", 2024-03-20T09:24:01.535 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2919999999999998e-06 2024-03-20T09:24:01.535 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.535 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.535 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:24:01.535 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.889355+0000", 2024-03-20T09:24:01.536 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4365999999999999e-05 2024-03-20T09:24:01.536 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.536 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.536 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:24:01.536 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.889474+0000", 2024-03-20T09:24:01.536 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000118649 2024-03-20T09:24:01.536 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.536 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.536 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:24:01.536 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.889613+0000", 2024-03-20T09:24:01.536 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013953500000000001 2024-03-20T09:24:01.536 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.536 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.536 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.537 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.890819+0000", 2024-03-20T09:24:01.537 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001205383 2024-03-20T09:24:01.537 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.537 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.537 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:24:01.537 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.890848+0000", 2024-03-20T09:24:01.537 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9437999999999998e-05 2024-03-20T09:24:01.537 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.537 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.537 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.537 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:23.094078+0000", 2024-03-20T09:24:01.537 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20323011299999999 2024-03-20T09:24:01.537 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.537 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.537 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:24:01.538 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:23.094116+0000", 2024-03-20T09:24:01.538 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7308000000000003e-05 2024-03-20T09:24:01.538 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.538 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.538 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:24:01.538 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:23.094146+0000", 2024-03-20T09:24:01.538 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0534000000000003e-05 2024-03-20T09:24:01.538 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.538 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:24:01.538 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.538 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.538 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.538 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25966.0:8056 5.8 5:1472a471:::benchmark_data_smithi122_157894_object8055:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e611)", 2024-03-20T09:24:01.539 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:18:22.889438+0000", 2024-03-20T09:24:01.539 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 338.61227055699999, 2024-03-20T09:24:01.539 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20678607900000001, 2024-03-20T09:24:01.539 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:24:01.539 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:24:01.539 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:24:01.539 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25966", 2024-03-20T09:24:01.539 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:24:01.539 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 8056 2024-03-20T09:24:01.539 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.539 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:24:01.539 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.539 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:24:01.539 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.889438+0000", 2024-03-20T09:24:01.539 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.540 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.540 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.540 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:24:01.540 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.889438+0000", 2024-03-20T09:24:01.540 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.540 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.540 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.540 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:24:01.540 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.889439+0000", 2024-03-20T09:24:01.540 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3e-07 2024-03-20T09:24:01.540 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.540 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.540 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:24:01.540 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.889458+0000", 2024-03-20T09:24:01.540 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8669999999999999e-05 2024-03-20T09:24:01.541 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.541 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.541 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:24:01.541 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.889466+0000", 2024-03-20T09:24:01.541 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8030000000000003e-06 2024-03-20T09:24:01.541 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.541 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.541 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:24:01.541 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.889473+0000", 2024-03-20T09:24:01.541 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5769999999999999e-06 2024-03-20T09:24:01.541 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.541 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.541 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:24:01.541 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.890109+0000", 2024-03-20T09:24:01.542 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00063546700000000002 2024-03-20T09:24:01.542 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.542 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.542 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:24:01.542 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.890217+0000", 2024-03-20T09:24:01.542 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000108218 2024-03-20T09:24:01.542 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.542 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.542 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:24:01.542 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.890374+0000", 2024-03-20T09:24:01.542 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015692000000000001 2024-03-20T09:24:01.542 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.542 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.543 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.543 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.891591+0000", 2024-03-20T09:24:01.543 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0012171700000000001 2024-03-20T09:24:01.543 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.543 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.543 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:24:01.543 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.891604+0000", 2024-03-20T09:24:01.543 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2771e-05 2024-03-20T09:24:01.543 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.543 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.543 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.543 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:23.096163+0000", 2024-03-20T09:24:01.543 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20455932299999999 2024-03-20T09:24:01.543 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.544 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.544 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:24:01.544 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:23.096196+0000", 2024-03-20T09:24:01.544 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2950000000000001e-05 2024-03-20T09:24:01.544 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.544 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.544 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:24:01.544 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:23.096224+0000", 2024-03-20T09:24:01.544 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8479999999999998e-05 2024-03-20T09:24:01.544 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.544 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:24:01.544 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.544 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.544 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.545 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25966.0:8064 5.8 5:148c550b:::benchmark_data_smithi122_157894_object8063:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e611)", 2024-03-20T09:24:01.545 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:18:22.890824+0000", 2024-03-20T09:24:01.545 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 338.610885268, 2024-03-20T09:24:01.545 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20594858299999999, 2024-03-20T09:24:01.545 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:24:01.545 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:24:01.545 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:24:01.545 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25966", 2024-03-20T09:24:01.545 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:24:01.545 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 8064 2024-03-20T09:24:01.545 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.545 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:24:01.545 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.545 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:24:01.546 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.890824+0000", 2024-03-20T09:24:01.546 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.546 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.546 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.546 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:24:01.546 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.890824+0000", 2024-03-20T09:24:01.546 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.546 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.546 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.546 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:24:01.546 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.890824+0000", 2024-03-20T09:24:01.546 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1999999999999998e-07 2024-03-20T09:24:01.546 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.546 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.547 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:24:01.547 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.890843+0000", 2024-03-20T09:24:01.547 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8899e-05 2024-03-20T09:24:01.547 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.547 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.547 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:24:01.547 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.890852+0000", 2024-03-20T09:24:01.547 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1910000000000006e-06 2024-03-20T09:24:01.547 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.547 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.547 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:24:01.547 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.890858+0000", 2024-03-20T09:24:01.547 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3820000000000003e-06 2024-03-20T09:24:01.547 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.548 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.548 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:24:01.548 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.890910+0000", 2024-03-20T09:24:01.548 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2061000000000002e-05 2024-03-20T09:24:01.548 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.548 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.548 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:24:01.548 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.891049+0000", 2024-03-20T09:24:01.548 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000139216 2024-03-20T09:24:01.548 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.548 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.548 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:24:01.548 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.891210+0000", 2024-03-20T09:24:01.548 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000160431 2024-03-20T09:24:01.548 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.549 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.549 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:24:01.549 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.891671+0000", 2024-03-20T09:24:01.549 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00046143999999999997 2024-03-20T09:24:01.549 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.549 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.549 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.549 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.891923+0000", 2024-03-20T09:24:01.549 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00025236099999999999 2024-03-20T09:24:01.549 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.549 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.549 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.549 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:23.096713+0000", 2024-03-20T09:24:01.549 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.204790106 2024-03-20T09:24:01.550 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.550 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.550 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:24:01.550 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:23.096746+0000", 2024-03-20T09:24:01.550 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2150999999999998e-05 2024-03-20T09:24:01.550 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.550 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.550 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:24:01.550 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:23.096772+0000", 2024-03-20T09:24:01.550 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6525000000000001e-05 2024-03-20T09:24:01.550 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.550 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:24:01.550 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.550 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.551 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.551 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25966.0:8069 5.8 5:16fb6b90:::benchmark_data_smithi122_157894_object8068:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e611)", 2024-03-20T09:24:01.551 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:18:22.893982+0000", 2024-03-20T09:24:01.551 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 338.60772640300002, 2024-03-20T09:24:01.551 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.202890184, 2024-03-20T09:24:01.551 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:24:01.551 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:24:01.551 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:24:01.551 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25966", 2024-03-20T09:24:01.551 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:24:01.551 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 8069 2024-03-20T09:24:01.551 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.551 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:24:01.551 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.552 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:24:01.552 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.893982+0000", 2024-03-20T09:24:01.552 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.552 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.552 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.552 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:24:01.552 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.893982+0000", 2024-03-20T09:24:01.552 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.552 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.552 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.552 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:24:01.552 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.893984+0000", 2024-03-20T09:24:01.552 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2190000000000001e-06 2024-03-20T09:24:01.552 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.552 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.553 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:24:01.553 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.894003+0000", 2024-03-20T09:24:01.553 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9434000000000001e-05 2024-03-20T09:24:01.553 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.553 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.553 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:24:01.553 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.894014+0000", 2024-03-20T09:24:01.553 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0865e-05 2024-03-20T09:24:01.553 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.553 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.553 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:24:01.553 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.894019+0000", 2024-03-20T09:24:01.553 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4149999999999998e-06 2024-03-20T09:24:01.553 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.553 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.554 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:24:01.554 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.894071+0000", 2024-03-20T09:24:01.554 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1891999999999997e-05 2024-03-20T09:24:01.554 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.554 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.554 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:24:01.554 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.894178+0000", 2024-03-20T09:24:01.554 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000106554 2024-03-20T09:24:01.554 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.554 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.554 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:24:01.554 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.894263+0000", 2024-03-20T09:24:01.554 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5650999999999997e-05 2024-03-20T09:24:01.554 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.555 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.555 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:24:01.555 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.894793+0000", 2024-03-20T09:24:01.555 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00052971799999999999 2024-03-20T09:24:01.555 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.555 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.555 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.555 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.894975+0000", 2024-03-20T09:24:01.555 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00018218999999999999 2024-03-20T09:24:01.555 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.555 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.556 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.556 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:23.096813+0000", 2024-03-20T09:24:01.556 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.201838142 2024-03-20T09:24:01.556 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.556 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.556 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:24:01.556 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:23.096848+0000", 2024-03-20T09:24:01.556 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4097000000000002e-05 2024-03-20T09:24:01.556 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.556 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.556 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:24:01.556 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:23.096873+0000", 2024-03-20T09:24:01.556 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5007e-05 2024-03-20T09:24:01.556 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.557 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:24:01.557 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.557 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.557 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.557 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25966.0:8107 5.8 5:1953cd43:::benchmark_data_smithi122_157894_object8106:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e611)", 2024-03-20T09:24:01.557 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:18:22.905355+0000", 2024-03-20T09:24:01.557 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 338.59635431100003, 2024-03-20T09:24:01.557 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.191647285, 2024-03-20T09:24:01.557 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:24:01.557 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:24:01.557 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:24:01.557 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25966", 2024-03-20T09:24:01.557 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:24:01.557 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 8107 2024-03-20T09:24:01.558 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.558 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:24:01.558 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.558 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:24:01.558 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.905355+0000", 2024-03-20T09:24:01.558 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.558 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.558 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.558 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:24:01.558 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.905355+0000", 2024-03-20T09:24:01.558 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.558 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.558 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.558 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:24:01.559 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.905355+0000", 2024-03-20T09:24:01.559 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2299999999999999e-07 2024-03-20T09:24:01.559 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.559 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.559 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:24:01.559 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.905375+0000", 2024-03-20T09:24:01.559 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9882e-05 2024-03-20T09:24:01.559 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.559 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.559 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:24:01.559 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.905385+0000", 2024-03-20T09:24:01.559 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.8279999999999998e-06 2024-03-20T09:24:01.559 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.559 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.560 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:24:01.560 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.905392+0000", 2024-03-20T09:24:01.560 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.776e-06 2024-03-20T09:24:01.560 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.560 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.560 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:24:01.560 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.905448+0000", 2024-03-20T09:24:01.560 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5915000000000003e-05 2024-03-20T09:24:01.560 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.560 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.560 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:24:01.560 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.905551+0000", 2024-03-20T09:24:01.560 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000102946 2024-03-20T09:24:01.560 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.561 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.561 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:24:01.561 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.905672+0000", 2024-03-20T09:24:01.561 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000121659 2024-03-20T09:24:01.561 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.561 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.561 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:24:01.561 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.906318+0000", 2024-03-20T09:24:01.561 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00064511899999999999 2024-03-20T09:24:01.561 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.561 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.561 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.561 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.906945+0000", 2024-03-20T09:24:01.561 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00062755799999999996 2024-03-20T09:24:01.562 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.562 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.562 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.562 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:23.096911+0000", 2024-03-20T09:24:01.562 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18996613800000001 2024-03-20T09:24:01.562 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.562 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.562 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:24:01.562 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:23.096965+0000", 2024-03-20T09:24:01.562 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4048999999999997e-05 2024-03-20T09:24:01.562 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.562 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.562 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:24:01.562 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:23.097002+0000", 2024-03-20T09:24:01.563 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6492000000000002e-05 2024-03-20T09:24:01.563 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.563 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:24:01.563 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.563 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.563 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.563 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25966.0:8131 5.8 5:13c8b554:::benchmark_data_smithi122_157894_object8130:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e611)", 2024-03-20T09:24:01.563 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:18:22.914326+0000", 2024-03-20T09:24:01.563 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 338.58738276499997, 2024-03-20T09:24:01.563 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18332132400000001, 2024-03-20T09:24:01.563 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:24:01.563 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:24:01.563 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:24:01.563 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25966", 2024-03-20T09:24:01.563 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:24:01.564 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 8131 2024-03-20T09:24:01.564 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.564 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:24:01.564 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.564 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:24:01.564 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.914326+0000", 2024-03-20T09:24:01.566 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.566 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.566 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.566 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:24:01.566 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.914326+0000", 2024-03-20T09:24:01.566 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.566 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.566 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.566 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:24:01.567 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.914328+0000", 2024-03-20T09:24:01.567 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5990000000000001e-06 2024-03-20T09:24:01.567 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.567 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.567 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:24:01.567 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.914346+0000", 2024-03-20T09:24:01.567 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8206000000000002e-05 2024-03-20T09:24:01.567 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.567 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.567 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:24:01.567 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.914357+0000", 2024-03-20T09:24:01.568 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1011e-05 2024-03-20T09:24:01.568 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.568 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.568 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:24:01.568 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.914363+0000", 2024-03-20T09:24:01.568 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8109999999999999e-06 2024-03-20T09:24:01.568 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.568 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.568 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:24:01.568 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.914414+0000", 2024-03-20T09:24:01.568 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1737999999999998e-05 2024-03-20T09:24:01.568 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.568 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.568 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:24:01.568 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.914519+0000", 2024-03-20T09:24:01.569 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010500799999999999 2024-03-20T09:24:01.569 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.569 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.569 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:24:01.569 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.914610+0000", 2024-03-20T09:24:01.569 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0563999999999996e-05 2024-03-20T09:24:01.569 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.569 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.569 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:24:01.569 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.915125+0000", 2024-03-20T09:24:01.569 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00051455299999999995 2024-03-20T09:24:01.569 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.569 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.569 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.570 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:22.915410+0000", 2024-03-20T09:24:01.570 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00028549599999999999 2024-03-20T09:24:01.570 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.570 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.570 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.570 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:23.097590+0000", 2024-03-20T09:24:01.570 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18218016000000001 2024-03-20T09:24:01.570 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.570 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.570 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:24:01.570 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:23.097621+0000", 2024-03-20T09:24:01.570 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1278000000000001e-05 2024-03-20T09:24:01.570 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.571 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.571 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:24:01.571 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:23.097647+0000", 2024-03-20T09:24:01.571 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5899999999999999e-05 2024-03-20T09:24:01.571 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.571 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:24:01.571 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.571 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.571 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.571 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25966.0:13791 5.8 5:1176840d:::benchmark_data_smithi122_157894_object13790:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e612)", 2024-03-20T09:24:01.571 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:18:25.041960+0000", 2024-03-20T09:24:01.571 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 336.45974852400002, 2024-03-20T09:24:01.571 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20611988000000001, 2024-03-20T09:24:01.571 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:24:01.571 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:24:01.572 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:24:01.572 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25966", 2024-03-20T09:24:01.572 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:24:01.572 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 13791 2024-03-20T09:24:01.572 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.572 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:24:01.572 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.572 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:24:01.572 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.041960+0000", 2024-03-20T09:24:01.572 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.572 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.572 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.572 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:24:01.572 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.041960+0000", 2024-03-20T09:24:01.572 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.573 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.573 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.573 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:24:01.573 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.041961+0000", 2024-03-20T09:24:01.573 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5499999999999998e-07 2024-03-20T09:24:01.573 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.573 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.573 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:24:01.573 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.041985+0000", 2024-03-20T09:24:01.573 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3558999999999999e-05 2024-03-20T09:24:01.573 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.573 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.573 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:24:01.573 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.041995+0000", 2024-03-20T09:24:01.574 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0203999999999999e-05 2024-03-20T09:24:01.574 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.574 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.574 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:24:01.574 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.042001+0000", 2024-03-20T09:24:01.574 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.806e-06 2024-03-20T09:24:01.574 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.574 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.574 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:24:01.574 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.042047+0000", 2024-03-20T09:24:01.574 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6749999999999998e-05 2024-03-20T09:24:01.574 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.574 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.574 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:24:01.574 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.042154+0000", 2024-03-20T09:24:01.575 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000107056 2024-03-20T09:24:01.575 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.575 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.575 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:24:01.575 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.042288+0000", 2024-03-20T09:24:01.575 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000133585 2024-03-20T09:24:01.575 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.575 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.575 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:24:01.575 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.043075+0000", 2024-03-20T09:24:01.575 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00078733399999999999 2024-03-20T09:24:01.575 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.575 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.575 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.576 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.043819+0000", 2024-03-20T09:24:01.576 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00074377000000000004 2024-03-20T09:24:01.576 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.576 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.576 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.576 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.248009+0000", 2024-03-20T09:24:01.576 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20418971699999999 2024-03-20T09:24:01.576 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.576 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.576 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:24:01.576 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.248048+0000", 2024-03-20T09:24:01.576 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9567000000000003e-05 2024-03-20T09:24:01.576 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.576 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.576 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:24:01.577 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.248080+0000", 2024-03-20T09:24:01.577 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1877e-05 2024-03-20T09:24:01.577 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.577 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:24:01.577 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.577 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.577 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.578 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25966.0:13819 5.8 5:18a40a82:::benchmark_data_smithi122_157894_object13818:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e612)", 2024-03-20T09:24:01.578 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:18:25.049685+0000", 2024-03-20T09:24:01.578 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 336.45202342699997, 2024-03-20T09:24:01.578 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20001260600000001, 2024-03-20T09:24:01.578 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:24:01.578 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:24:01.578 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:24:01.578 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25966", 2024-03-20T09:24:01.578 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:24:01.578 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 13819 2024-03-20T09:24:01.578 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.578 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:24:01.578 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.579 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:24:01.579 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.049685+0000", 2024-03-20T09:24:01.579 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.579 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.579 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.579 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:24:01.579 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.049685+0000", 2024-03-20T09:24:01.579 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.579 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.579 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.579 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:24:01.579 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.049687+0000", 2024-03-20T09:24:01.579 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3599999999999999e-06 2024-03-20T09:24:01.579 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.579 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.580 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:24:01.580 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.049708+0000", 2024-03-20T09:24:01.580 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1336e-05 2024-03-20T09:24:01.580 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.580 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.580 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:24:01.580 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.049722+0000", 2024-03-20T09:24:01.580 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4079e-05 2024-03-20T09:24:01.580 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.580 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.580 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:24:01.580 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.049731+0000", 2024-03-20T09:24:01.580 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.551e-06 2024-03-20T09:24:01.580 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.581 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.581 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:24:01.581 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.049796+0000", 2024-03-20T09:24:01.581 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5054000000000004e-05 2024-03-20T09:24:01.581 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.581 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.581 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:24:01.581 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.049887+0000", 2024-03-20T09:24:01.581 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0759000000000003e-05 2024-03-20T09:24:01.581 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.581 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.581 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:24:01.581 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.050018+0000", 2024-03-20T09:24:01.581 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013103400000000001 2024-03-20T09:24:01.582 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.582 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.582 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:24:01.582 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.050762+0000", 2024-03-20T09:24:01.582 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00074455400000000003 2024-03-20T09:24:01.582 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.582 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.582 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.582 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.052010+0000", 2024-03-20T09:24:01.582 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001248187 2024-03-20T09:24:01.582 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.582 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.582 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.582 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.249631+0000", 2024-03-20T09:24:01.583 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19762068699999999 2024-03-20T09:24:01.583 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.583 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.583 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:24:01.583 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.249668+0000", 2024-03-20T09:24:01.583 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6575000000000001e-05 2024-03-20T09:24:01.583 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.583 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.583 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:24:01.583 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.249698+0000", 2024-03-20T09:24:01.583 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0429999999999998e-05 2024-03-20T09:24:01.583 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.583 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:24:01.583 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.584 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.584 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.584 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25966.0:13852 5.8 5:18b7fd45:::benchmark_data_smithi122_157894_object13851:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e612)", 2024-03-20T09:24:01.584 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:18:25.058256+0000", 2024-03-20T09:24:01.584 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 336.44345306999998, 2024-03-20T09:24:01.584 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.192201436, 2024-03-20T09:24:01.584 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:24:01.584 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:24:01.584 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:24:01.584 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25966", 2024-03-20T09:24:01.584 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:24:01.584 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 13852 2024-03-20T09:24:01.584 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.584 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:24:01.585 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.585 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:24:01.585 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.058256+0000", 2024-03-20T09:24:01.585 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.585 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.585 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.585 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:24:01.585 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.058256+0000", 2024-03-20T09:24:01.585 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.585 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.585 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.585 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:24:01.585 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.058257+0000", 2024-03-20T09:24:01.585 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5659999999999999e-06 2024-03-20T09:24:01.585 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.586 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.586 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:24:01.586 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.058287+0000", 2024-03-20T09:24:01.586 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9682000000000001e-05 2024-03-20T09:24:01.586 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.586 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.586 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:24:01.586 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.058298+0000", 2024-03-20T09:24:01.586 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1371e-05 2024-03-20T09:24:01.586 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.586 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.586 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:24:01.586 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.058308+0000", 2024-03-20T09:24:01.586 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4450000000000002e-06 2024-03-20T09:24:01.587 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.587 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.587 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:24:01.587 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.058362+0000", 2024-03-20T09:24:01.587 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4412e-05 2024-03-20T09:24:01.587 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.587 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.587 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:24:01.587 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.058469+0000", 2024-03-20T09:24:01.587 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000107172 2024-03-20T09:24:01.587 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.587 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.587 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:24:01.587 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.058557+0000", 2024-03-20T09:24:01.588 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7272000000000001e-05 2024-03-20T09:24:01.588 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.588 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.588 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:24:01.588 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.059806+0000", 2024-03-20T09:24:01.588 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0012496600000000001 2024-03-20T09:24:01.588 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.588 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.588 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.588 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.060193+0000", 2024-03-20T09:24:01.588 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000386957 2024-03-20T09:24:01.588 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.588 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.588 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.588 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.250396+0000", 2024-03-20T09:24:01.589 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.190202338 2024-03-20T09:24:01.589 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.589 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.589 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:24:01.589 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.250431+0000", 2024-03-20T09:24:01.589 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5351000000000001e-05 2024-03-20T09:24:01.589 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.589 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.589 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:24:01.589 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.250457+0000", 2024-03-20T09:24:01.589 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6210000000000001e-05 2024-03-20T09:24:01.589 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.589 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:24:01.589 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.589 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.590 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.590 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25966.0:13853 5.8 5:150a1528:::benchmark_data_smithi122_157894_object13852:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e612)", 2024-03-20T09:24:01.590 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:18:25.058382+0000", 2024-03-20T09:24:01.590 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 336.44332691900001, 2024-03-20T09:24:01.590 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.192191791, 2024-03-20T09:24:01.590 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:24:01.590 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:24:01.590 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:24:01.590 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25966", 2024-03-20T09:24:01.590 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:24:01.590 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 13853 2024-03-20T09:24:01.590 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.590 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:24:01.591 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.591 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:24:01.591 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.058382+0000", 2024-03-20T09:24:01.591 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.591 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.591 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.591 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:24:01.591 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.058382+0000", 2024-03-20T09:24:01.591 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.591 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.591 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.591 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:24:01.591 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.058383+0000", 2024-03-20T09:24:01.591 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7899999999999997e-07 2024-03-20T09:24:01.592 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.592 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.592 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:24:01.592 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.058402+0000", 2024-03-20T09:24:01.592 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9055999999999998e-05 2024-03-20T09:24:01.592 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.592 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.592 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:24:01.592 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.058409+0000", 2024-03-20T09:24:01.592 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4869999999999996e-06 2024-03-20T09:24:01.593 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.593 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.593 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:24:01.593 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.058414+0000", 2024-03-20T09:24:01.593 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5909999999999998e-06 2024-03-20T09:24:01.593 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.593 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.593 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:24:01.593 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.059125+0000", 2024-03-20T09:24:01.593 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00071087199999999998 2024-03-20T09:24:01.593 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.593 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.593 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:24:01.593 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.059227+0000", 2024-03-20T09:24:01.594 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010202800000000001 2024-03-20T09:24:01.594 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.594 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.594 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:24:01.594 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.059341+0000", 2024-03-20T09:24:01.594 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000114436 2024-03-20T09:24:01.594 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.594 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.594 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:24:01.594 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.059945+0000", 2024-03-20T09:24:01.594 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00060406800000000003 2024-03-20T09:24:01.594 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.594 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.594 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.595 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.060413+0000", 2024-03-20T09:24:01.595 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00046826799999999998 2024-03-20T09:24:01.595 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.595 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.595 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.595 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.250500+0000", 2024-03-20T09:24:01.595 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19008615700000001 2024-03-20T09:24:01.595 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.595 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.595 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:24:01.595 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.250536+0000", 2024-03-20T09:24:01.595 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6531e-05 2024-03-20T09:24:01.595 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.595 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.596 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:24:01.596 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.250574+0000", 2024-03-20T09:24:01.596 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7518000000000002e-05 2024-03-20T09:24:01.596 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.596 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:24:01.596 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.596 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.596 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.596 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25966.0:13877 5.8 5:1fec032c:::benchmark_data_smithi122_157894_object13876:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e612)", 2024-03-20T09:24:01.596 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:18:25.065548+0000", 2024-03-20T09:24:01.596 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 336.43616091400003, 2024-03-20T09:24:01.596 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18574700699999999, 2024-03-20T09:24:01.596 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:24:01.596 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:24:01.596 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:24:01.597 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25966", 2024-03-20T09:24:01.597 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:24:01.597 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 13877 2024-03-20T09:24:01.597 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.597 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:24:01.597 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.597 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:24:01.597 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.065548+0000", 2024-03-20T09:24:01.597 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.597 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.597 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.597 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:24:01.598 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.065548+0000", 2024-03-20T09:24:01.598 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.598 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.598 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.598 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:24:01.598 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.065549+0000", 2024-03-20T09:24:01.598 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3500000000000005e-07 2024-03-20T09:24:01.598 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.598 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.599 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:24:01.599 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.065566+0000", 2024-03-20T09:24:01.599 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7555e-05 2024-03-20T09:24:01.599 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.599 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.599 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:24:01.599 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.065579+0000", 2024-03-20T09:24:01.599 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2877999999999999e-05 2024-03-20T09:24:01.599 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.599 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.599 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:24:01.599 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.065587+0000", 2024-03-20T09:24:01.599 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2069999999999995e-06 2024-03-20T09:24:01.599 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.599 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.600 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:24:01.600 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.065645+0000", 2024-03-20T09:24:01.600 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7138000000000001e-05 2024-03-20T09:24:01.600 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.600 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.600 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:24:01.600 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.065774+0000", 2024-03-20T09:24:01.600 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000129438 2024-03-20T09:24:01.600 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.600 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.600 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:24:01.600 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.065898+0000", 2024-03-20T09:24:01.600 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000124511 2024-03-20T09:24:01.600 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.601 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.601 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:24:01.601 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.066836+0000", 2024-03-20T09:24:01.601 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00093743899999999998 2024-03-20T09:24:01.601 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.601 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.601 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.601 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.066910+0000", 2024-03-20T09:24:01.601 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3763999999999994e-05 2024-03-20T09:24:01.601 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.601 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.601 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.601 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.251232+0000", 2024-03-20T09:24:01.601 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18432262399999999 2024-03-20T09:24:01.601 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.602 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.602 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:24:01.602 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.251265+0000", 2024-03-20T09:24:01.602 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2441999999999999e-05 2024-03-20T09:24:01.602 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.602 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.602 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:24:01.602 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.251295+0000", 2024-03-20T09:24:01.602 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0176e-05 2024-03-20T09:24:01.602 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.602 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:24:01.602 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.602 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.602 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.603 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25966.0:13882 5.8 5:16b3ea0d:::benchmark_data_smithi122_157894_object13881:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e612)", 2024-03-20T09:24:01.603 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:18:25.067500+0000", 2024-03-20T09:24:01.603 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 336.434208501, 2024-03-20T09:24:01.603 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18392726500000001, 2024-03-20T09:24:01.603 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:24:01.603 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:24:01.603 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:24:01.603 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25966", 2024-03-20T09:24:01.603 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:24:01.603 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 13882 2024-03-20T09:24:01.603 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.603 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:24:01.603 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.603 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:24:01.603 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.067500+0000", 2024-03-20T09:24:01.604 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.604 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.604 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.604 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:24:01.604 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.067500+0000", 2024-03-20T09:24:01.604 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.604 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.604 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.604 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:24:01.604 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.067501+0000", 2024-03-20T09:24:01.604 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7100000000000011e-07 2024-03-20T09:24:01.604 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.604 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.604 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:24:01.605 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.067518+0000", 2024-03-20T09:24:01.605 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6793e-05 2024-03-20T09:24:01.605 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.605 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.605 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:24:01.605 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.067529+0000", 2024-03-20T09:24:01.605 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0701e-05 2024-03-20T09:24:01.605 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.605 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.605 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:24:01.605 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.067536+0000", 2024-03-20T09:24:01.605 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1579999999999999e-06 2024-03-20T09:24:01.605 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.605 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.606 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:24:01.606 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.067598+0000", 2024-03-20T09:24:01.606 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2018000000000002e-05 2024-03-20T09:24:01.606 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.606 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.606 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:24:01.606 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.067740+0000", 2024-03-20T09:24:01.606 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000142297 2024-03-20T09:24:01.606 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.606 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.606 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:24:01.606 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.067890+0000", 2024-03-20T09:24:01.606 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014984800000000001 2024-03-20T09:24:01.606 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.607 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.607 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:24:01.607 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.069124+0000", 2024-03-20T09:24:01.607 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001233428 2024-03-20T09:24:01.607 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.607 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.607 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.607 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.069170+0000", 2024-03-20T09:24:01.607 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6626999999999999e-05 2024-03-20T09:24:01.607 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.607 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.607 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.607 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.251337+0000", 2024-03-20T09:24:01.607 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18216716399999999 2024-03-20T09:24:01.607 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.608 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.608 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:24:01.608 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.251384+0000", 2024-03-20T09:24:01.608 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6492000000000001e-05 2024-03-20T09:24:01.608 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.608 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.608 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:24:01.608 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.251428+0000", 2024-03-20T09:24:01.608 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3767999999999998e-05 2024-03-20T09:24:01.608 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.608 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:24:01.608 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.608 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.608 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.609 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25966.0:13885 5.8 5:1f12e1ca:::benchmark_data_smithi122_157894_object13884:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e612)", 2024-03-20T09:24:01.609 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:18:25.067560+0000", 2024-03-20T09:24:01.609 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 336.43414892800001, 2024-03-20T09:24:01.609 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.184273932, 2024-03-20T09:24:01.609 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:24:01.609 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:24:01.609 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:24:01.609 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25966", 2024-03-20T09:24:01.609 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:24:01.609 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 13885 2024-03-20T09:24:01.609 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.609 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:24:01.609 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.609 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:24:01.610 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.067560+0000", 2024-03-20T09:24:01.610 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.610 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.610 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.610 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:24:01.610 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.067560+0000", 2024-03-20T09:24:01.610 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.610 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.610 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.610 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:24:01.610 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.067560+0000", 2024-03-20T09:24:01.610 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4400000000000001e-07 2024-03-20T09:24:01.610 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.610 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.611 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:24:01.611 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.067579+0000", 2024-03-20T09:24:01.611 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.825e-05 2024-03-20T09:24:01.611 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.611 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.611 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:24:01.611 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.067584+0000", 2024-03-20T09:24:01.611 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3199999999999999e-06 2024-03-20T09:24:01.611 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.611 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.611 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:24:01.611 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.067588+0000", 2024-03-20T09:24:01.611 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.619e-06 2024-03-20T09:24:01.611 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.612 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.612 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:24:01.612 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.068540+0000", 2024-03-20T09:24:01.612 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00095254400000000005 2024-03-20T09:24:01.612 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.612 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.612 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:24:01.612 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.068632+0000", 2024-03-20T09:24:01.612 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1689999999999998e-05 2024-03-20T09:24:01.612 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.612 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.612 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:24:01.612 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.068720+0000", 2024-03-20T09:24:01.612 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8245999999999995e-05 2024-03-20T09:24:01.612 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.613 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.613 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:24:01.613 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.069269+0000", 2024-03-20T09:24:01.613 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00054908799999999998 2024-03-20T09:24:01.613 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.613 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.613 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.613 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.069563+0000", 2024-03-20T09:24:01.613 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00029382200000000001 2024-03-20T09:24:01.613 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.613 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.613 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.613 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.251733+0000", 2024-03-20T09:24:01.613 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18216969899999999 2024-03-20T09:24:01.614 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.614 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.614 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:24:01.614 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.251791+0000", 2024-03-20T09:24:01.614 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8529000000000002e-05 2024-03-20T09:24:01.614 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.614 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.614 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:24:01.614 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.251834+0000", 2024-03-20T09:24:01.614 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2681e-05 2024-03-20T09:24:01.614 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.614 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:24:01.614 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.614 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.615 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.615 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25966.0:13922 5.8 5:196ac634:::benchmark_data_smithi122_157894_object13921:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e612)", 2024-03-20T09:24:01.615 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:18:25.078201+0000", 2024-03-20T09:24:01.615 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 336.42350808999998, 2024-03-20T09:24:01.615 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.173775388, 2024-03-20T09:24:01.615 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:24:01.615 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:24:01.615 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:24:01.615 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25966", 2024-03-20T09:24:01.615 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:24:01.615 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 13922 2024-03-20T09:24:01.615 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.615 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:24:01.615 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.616 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:24:01.616 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.078201+0000", 2024-03-20T09:24:01.616 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.616 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.616 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.616 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:24:01.616 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.078201+0000", 2024-03-20T09:24:01.616 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.616 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.616 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.616 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:24:01.616 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.078202+0000", 2024-03-20T09:24:01.616 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1999999999999998e-07 2024-03-20T09:24:01.616 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.616 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.617 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:24:01.617 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.078220+0000", 2024-03-20T09:24:01.617 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8593e-05 2024-03-20T09:24:01.617 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.617 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.617 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:24:01.617 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.078230+0000", 2024-03-20T09:24:01.617 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0301e-05 2024-03-20T09:24:01.617 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.617 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.617 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:24:01.617 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.078236+0000", 2024-03-20T09:24:01.617 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7520000000000002e-06 2024-03-20T09:24:01.617 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.618 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.618 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:24:01.618 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.078300+0000", 2024-03-20T09:24:01.618 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4055999999999998e-05 2024-03-20T09:24:01.618 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.618 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.618 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:24:01.618 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.078391+0000", 2024-03-20T09:24:01.618 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1079e-05 2024-03-20T09:24:01.618 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.618 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.618 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:24:01.618 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.078483+0000", 2024-03-20T09:24:01.618 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1664999999999997e-05 2024-03-20T09:24:01.619 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.619 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.619 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:24:01.619 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.079220+0000", 2024-03-20T09:24:01.619 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000737088 2024-03-20T09:24:01.619 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.619 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.619 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.619 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.079400+0000", 2024-03-20T09:24:01.619 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00017947199999999999 2024-03-20T09:24:01.619 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.619 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.619 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.619 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.251882+0000", 2024-03-20T09:24:01.620 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.17248198300000001 2024-03-20T09:24:01.620 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.620 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.620 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:24:01.620 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.251936+0000", 2024-03-20T09:24:01.620 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4494999999999999e-05 2024-03-20T09:24:01.620 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.620 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.620 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:24:01.620 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:25.251976+0000", 2024-03-20T09:24:01.620 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0083999999999998e-05 2024-03-20T09:24:01.620 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.620 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:24:01.620 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.620 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.621 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.621 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25966.0:24831 5.8 5:1389c878:::benchmark_data_smithi122_157894_object24830:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e614)", 2024-03-20T09:24:01.621 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:18:29.510386+0000", 2024-03-20T09:24:01.621 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 331.991322905, 2024-03-20T09:24:01.621 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19539772599999999, 2024-03-20T09:24:01.621 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:24:01.621 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:24:01.621 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:24:01.621 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25966", 2024-03-20T09:24:01.621 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:24:01.621 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 24831 2024-03-20T09:24:01.621 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.621 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:24:01.621 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.622 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:24:01.622 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.510386+0000", 2024-03-20T09:24:01.622 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.622 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.622 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.622 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:24:01.622 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.510386+0000", 2024-03-20T09:24:01.622 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.622 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.622 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.622 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:24:01.622 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.510388+0000", 2024-03-20T09:24:01.622 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6649999999999999e-06 2024-03-20T09:24:01.622 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.623 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.623 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:24:01.623 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.510407+0000", 2024-03-20T09:24:01.623 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9822000000000001e-05 2024-03-20T09:24:01.623 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.623 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.623 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:24:01.623 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.510419+0000", 2024-03-20T09:24:01.623 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1714e-05 2024-03-20T09:24:01.623 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.623 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.623 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:24:01.623 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.510425+0000", 2024-03-20T09:24:01.623 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.947e-06 2024-03-20T09:24:01.623 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.624 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.624 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:24:01.624 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.510470+0000", 2024-03-20T09:24:01.624 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5386000000000002e-05 2024-03-20T09:24:01.624 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.624 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.624 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:24:01.624 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.510595+0000", 2024-03-20T09:24:01.624 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000124583 2024-03-20T09:24:01.624 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.624 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.624 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:24:01.624 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.510692+0000", 2024-03-20T09:24:01.624 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7442999999999995e-05 2024-03-20T09:24:01.625 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.625 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.625 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:24:01.625 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.511628+0000", 2024-03-20T09:24:01.625 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00093502899999999998 2024-03-20T09:24:01.625 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.625 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.625 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.625 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.511809+0000", 2024-03-20T09:24:01.625 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00018124100000000001 2024-03-20T09:24:01.625 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.625 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.625 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.625 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.705668+0000", 2024-03-20T09:24:01.626 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19385886899999999 2024-03-20T09:24:01.626 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.626 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.626 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:24:01.626 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.705743+0000", 2024-03-20T09:24:01.626 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5062999999999996e-05 2024-03-20T09:24:01.626 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.626 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.626 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:24:01.626 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.705784+0000", 2024-03-20T09:24:01.626 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0964000000000003e-05 2024-03-20T09:24:01.626 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.626 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:24:01.626 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.627 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.627 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.627 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.25966.0:24835 5.8 5:1d182003:::benchmark_data_smithi122_157894_object24834:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e614)", 2024-03-20T09:24:01.627 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:18:29.513133+0000", 2024-03-20T09:24:01.627 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 331.98857573700002, 2024-03-20T09:24:01.627 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.192806691, 2024-03-20T09:24:01.627 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:24:01.627 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:24:01.627 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:24:01.627 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.25966", 2024-03-20T09:24:01.627 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:24:01.627 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 24835 2024-03-20T09:24:01.627 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.627 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:24:01.628 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.628 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:24:01.628 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.513133+0000", 2024-03-20T09:24:01.628 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.628 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.628 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.628 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:24:01.628 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.513133+0000", 2024-03-20T09:24:01.628 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:24:01.628 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.628 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.628 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:24:01.628 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.513134+0000", 2024-03-20T09:24:01.628 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7899999999999997e-07 2024-03-20T09:24:01.629 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.629 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.629 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:24:01.629 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.513152+0000", 2024-03-20T09:24:01.629 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8337000000000001e-05 2024-03-20T09:24:01.629 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.629 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.629 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:24:01.629 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.513163+0000", 2024-03-20T09:24:01.629 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0903e-05 2024-03-20T09:24:01.629 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.629 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.629 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:24:01.629 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.513169+0000", 2024-03-20T09:24:01.629 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0990000000000004e-06 2024-03-20T09:24:01.630 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.630 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.630 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:24:01.630 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.513205+0000", 2024-03-20T09:24:01.630 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5839e-05 2024-03-20T09:24:01.630 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.630 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.630 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:24:01.630 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.513293+0000", 2024-03-20T09:24:01.630 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8311000000000002e-05 2024-03-20T09:24:01.630 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.630 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.630 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:24:01.630 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.513384+0000", 2024-03-20T09:24:01.631 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0925000000000001e-05 2024-03-20T09:24:01.631 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.631 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.631 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:24:01.631 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.514050+0000", 2024-03-20T09:24:01.631 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00066575499999999997 2024-03-20T09:24:01.631 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.631 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.631 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.631 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.514266+0000", 2024-03-20T09:24:01.631 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000215999 2024-03-20T09:24:01.631 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.631 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.631 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:24:01.632 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.705839+0000", 2024-03-20T09:24:01.632 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19157292000000001 2024-03-20T09:24:01.632 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.632 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.632 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:24:01.632 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.705898+0000", 2024-03-20T09:24:01.632 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8554000000000002e-05 2024-03-20T09:24:01.632 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:24:01.632 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:24:01.632 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:24:01.632 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:18:29.705940+0000", 2024-03-20T09:24:01.632 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.227e-05 2024-03-20T09:24:01.632 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.632 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:24:01.632 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.633 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:24:01.633 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:24:01.633 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:24:01.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:01.634 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:24:01.693 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:24:01.735 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:24:01.770 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:01.790 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:24:01.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:01.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:24:01.895 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:24:01.924 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:01.944 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:24:01.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:01.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:24:02.048 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:24:02.078 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:02.202 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:24:02.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:02.205 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:24:02.214 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-20T09:24:02.306 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:24:02.333 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:02.351 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:24:02.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:02.354 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:24:02.455 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:24:02.482 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:02.485 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:24:02.492 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:24:02.500 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:24:02.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:02.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:24:02.606 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:24:02.633 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:02.689 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:02 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:24:02.689 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:02 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:24:02.690 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:02 smithi026 ceph-mon[68309]: osdmap e708: 12 total, 12 up, 12 in 2024-03-20T09:24:02.690 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:02 smithi026 ceph-mon[68309]: pgmap v1731: 31 pgs: 31 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:24:02.690 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:02 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/992075166' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-20T09:24:02.690 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:02 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/992075166' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-20T09:24:02.690 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:02 smithi026 ceph-mon[68309]: osdmap e709: 12 total, 12 up, 12 in 2024-03-20T09:24:02.731 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:24:02.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:02.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:24:02.803 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:02 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:24:02.803 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:02 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:24:02.804 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:02 smithi115 ceph-mon[72476]: osdmap e708: 12 total, 12 up, 12 in 2024-03-20T09:24:02.804 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:02 smithi115 ceph-mon[72476]: pgmap v1731: 31 pgs: 31 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:24:02.804 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:02 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/992075166' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-20T09:24:02.804 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:02 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/992075166' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-20T09:24:02.804 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:02 smithi115 ceph-mon[72476]: osdmap e709: 12 total, 12 up, 12 in 2024-03-20T09:24:02.833 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'unique_pool_4' created 2024-03-20T09:24:02.836 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:24:02.864 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:02.945 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:02 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:24:02.945 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:02 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:24:02.945 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:02 smithi122 ceph-mon[73495]: osdmap e708: 12 total, 12 up, 12 in 2024-03-20T09:24:02.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:02 smithi122 ceph-mon[73495]: pgmap v1731: 31 pgs: 31 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:24:02.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:02 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/992075166' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-20T09:24:02.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:02 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/992075166' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-20T09:24:02.946 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:02 smithi122 ceph-mon[73495]: osdmap e709: 12 total, 12 up, 12 in 2024-03-20T09:24:03.056 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:24:03.063 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:24:03.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:03.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:24:03.170 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:24:03.198 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:04.403 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-03-20T09:24:04.404 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:24:04.469 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:04 smithi026 ceph-mon[68309]: osdmap e710: 12 total, 12 up, 12 in 2024-03-20T09:24:04.469 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:04 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:24:04.469 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:04 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:24:04.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:04 smithi026 ceph-mon[68309]: pgmap v1734: 47 pgs: 4 creating+peering, 10 unknown, 33 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:24:04.518 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:24:04.518 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:24:04.518 INFO:teuthology.orchestra.run.smithi115.stdout: "num_ops": 0 2024-03-20T09:24:04.518 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:24:04.650 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:24:04.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:04 smithi115 ceph-mon[72476]: osdmap e710: 12 total, 12 up, 12 in 2024-03-20T09:24:04.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:04 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:24:04.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:04 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:24:04.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:04 smithi115 ceph-mon[72476]: pgmap v1734: 47 pgs: 4 creating+peering, 10 unknown, 33 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:24:04.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:04.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:24:04.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:04 smithi122 ceph-mon[73495]: osdmap e710: 12 total, 12 up, 12 in 2024-03-20T09:24:04.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:04 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:24:04.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:04 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:24:04.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:04 smithi122 ceph-mon[73495]: pgmap v1734: 47 pgs: 4 creating+peering, 10 unknown, 33 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:24:04.757 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:24:04.791 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:04.812 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:24:04.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:04.815 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:24:04.916 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:24:04.944 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:05.215 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-20T09:24:05.481 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:24:05.519 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.4 to 1 (10000) 2024-03-20T09:24:05.611 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:24:05.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:05.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:24:05.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:05 smithi115 ceph-mon[72476]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-20T09:24:05.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:05 smithi115 ceph-mon[72476]: osdmap e711: 12 total, 12 up, 12 in 2024-03-20T09:24:05.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:05 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/444067695' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-20T09:24:05.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:05 smithi122 ceph-mon[73495]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-20T09:24:05.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:05 smithi122 ceph-mon[73495]: osdmap e711: 12 total, 12 up, 12 in 2024-03-20T09:24:05.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:05 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/444067695' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-20T09:24:05.715 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:24:05.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:05 smithi026 ceph-mon[68309]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-20T09:24:05.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:05 smithi026 ceph-mon[68309]: osdmap e711: 12 total, 12 up, 12 in 2024-03-20T09:24:05.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:05 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/444067695' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-20T09:24:05.748 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:05.769 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:24:05.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:05.772 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:24:05.873 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:24:05.906 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:06.403 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 9 1 2024-03-20T09:24:06.637 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:06 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/444067695' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-20T09:24:06.637 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:06 smithi026 ceph-mon[68309]: osdmap e712: 12 total, 12 up, 12 in 2024-03-20T09:24:06.637 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:06 smithi026 ceph-mon[68309]: pgmap v1737: 47 pgs: 4 creating+peering, 1 unknown, 42 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:24:06.637 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:24:06.647 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:06 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/444067695' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-20T09:24:06.647 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:06 smithi115 ceph-mon[72476]: osdmap e712: 12 total, 12 up, 12 in 2024-03-20T09:24:06.647 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:06 smithi115 ceph-mon[72476]: pgmap v1737: 47 pgs: 4 creating+peering, 1 unknown, 42 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:24:06.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:06.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:24:06.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:06 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/444067695' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-20T09:24:06.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:06 smithi122 ceph-mon[73495]: osdmap e712: 12 total, 12 up, 12 in 2024-03-20T09:24:06.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:06 smithi122 ceph-mon[73495]: pgmap v1737: 47 pgs: 4 creating+peering, 1 unknown, 42 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:24:06.759 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:24:06.788 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:06.862 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:24:06.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:06.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:24:06.966 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:24:06.999 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:07.020 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:24:07.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:07.023 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:24:07.124 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:24:07.154 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:24:07.155 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:07.179 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:24:07.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:07.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:24:07.199 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:24:07.199 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:24:07.199 INFO:teuthology.orchestra.run.smithi115.stdout: "complaint_time": 30, 2024-03-20T09:24:07.199 INFO:teuthology.orchestra.run.smithi115.stdout: "num_blocked_ops": 0 2024-03-20T09:24:07.199 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:24:07.203 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:24:07.284 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:24:07.314 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:07.375 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:24:07.405 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:07 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3939398957' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:24:07.405 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:07 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:24:07.405 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:07 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3939398957' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:24:07.405 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:07 smithi026 ceph-mon[68309]: osdmap e713: 12 total, 12 up, 12 in 2024-03-20T09:24:07.514 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:24:07.515 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:07 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3939398957' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:24:07.515 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:07 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:24:07.515 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:07 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3939398957' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:24:07.515 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:07 smithi115 ceph-mon[72476]: osdmap e713: 12 total, 12 up, 12 in 2024-03-20T09:24:07.515 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:07.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:24:07.617 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:24:07.644 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:07.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:07 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3939398957' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:24:07.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:07 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:24:07.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:07 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3939398957' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:24:07.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:07 smithi122 ceph-mon[73495]: osdmap e713: 12 total, 12 up, 12 in 2024-03-20T09:24:07.894 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:24:07.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:07.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:24:07.913 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-20T09:24:07.998 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:24:08.023 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:08.043 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:24:08.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:08.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:24:08.147 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:24:08.180 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:08.218 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:24:08.469 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:08 smithi026 ceph-mon[68309]: pgmap v1739: 47 pgs: 1 creating+peering, 46 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:24:08.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:08 smithi115 ceph-mon[72476]: pgmap v1739: 47 pgs: 1 creating+peering, 46 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:24:08.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:08 smithi122 ceph-mon[73495]: pgmap v1739: 47 pgs: 1 creating+peering, 46 active+clean; 577 KiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:24:08.833 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:24:08.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:08.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:24:08.938 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:24:08.970 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:08.991 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:24:08.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:08.994 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:24:09.095 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:24:09.126 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:09.251 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:24:09.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:09.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:24:09.358 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:24:09.386 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:09.406 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:24:09.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:09.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:24:09.510 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:24:09.541 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:09.561 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:24:09.567 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:24:09.567 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:24:09.567 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:24:09.567 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:24:09.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:09.568 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:24:09.585 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:24:09.605 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:24:09.626 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:24:09.635 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:09 smithi115 ceph-mon[72476]: 1.11 scrub starts 2024-03-20T09:24:09.636 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:09 smithi115 ceph-mon[72476]: 1.11 scrub ok 2024-03-20T09:24:09.646 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:24:09.666 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:24:09.668 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:24:09.687 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:24:09.694 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:09 smithi026 ceph-mon[68309]: 1.11 scrub starts 2024-03-20T09:24:09.694 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:09 smithi026 ceph-mon[68309]: 1.11 scrub ok 2024-03-20T09:24:09.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:09 smithi122 ceph-mon[73495]: 1.11 scrub starts 2024-03-20T09:24:09.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:09 smithi122 ceph-mon[73495]: 1.11 scrub ok 2024-03-20T09:24:09.697 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:09.708 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:24:09.729 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:24:09.730 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:24:09.730 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:24:09.730 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:24:09.730 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:24:09.730 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:24:09.730 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:24:09.730 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:24:09.731 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:24:09.731 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:24:09.731 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:24:09.731 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:24:09.731 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:24:09.821 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:24:09.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:09.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:24:09.910 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:24:09.910 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:24:09.925 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:24:09.958 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:09.978 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:24:09.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:09.981 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:24:10.082 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:24:10.113 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:10.132 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:24:10.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:10.135 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:24:10.237 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:24:10.266 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:10.464 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:24:10.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:10.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:24:10.569 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:24:10.575 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:10 smithi115 ceph-mon[72476]: 1.1e scrub starts 2024-03-20T09:24:10.575 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:10 smithi115 ceph-mon[72476]: 1.1e scrub ok 2024-03-20T09:24:10.575 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:10 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3001489303' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:24:10.575 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:10 smithi115 ceph-mon[72476]: pgmap v1740: 47 pgs: 47 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:24:10.601 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:10.605 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-20T09:24:10.621 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:24:10.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:10.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:24:10.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:10 smithi122 ceph-mon[73495]: 1.1e scrub starts 2024-03-20T09:24:10.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:10 smithi122 ceph-mon[73495]: 1.1e scrub ok 2024-03-20T09:24:10.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:10 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3001489303' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:24:10.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:10 smithi122 ceph-mon[73495]: pgmap v1740: 47 pgs: 47 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:24:10.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:10 smithi026 ceph-mon[68309]: 1.1e scrub starts 2024-03-20T09:24:10.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:10 smithi026 ceph-mon[68309]: 1.1e scrub ok 2024-03-20T09:24:10.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:10 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3001489303' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:24:10.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:10 smithi026 ceph-mon[68309]: pgmap v1740: 47 pgs: 47 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:24:10.724 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:24:10.751 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:10.769 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:24:10.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:10.772 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:24:10.873 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:24:10.899 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:10.917 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:24:10.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:10.920 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:24:11.021 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:24:11.049 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:11.207 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:24:11.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:11.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:24:11.261 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:24:11.312 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:24:11.339 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:11.358 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:24:11.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:11.361 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:24:11.462 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:24:11.469 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:11 smithi026 ceph-mon[68309]: 1.18 scrub starts 2024-03-20T09:24:11.469 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:11 smithi026 ceph-mon[68309]: 1.18 scrub ok 2024-03-20T09:24:11.469 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:11 smithi026 ceph-mon[68309]: 1.b scrub starts 2024-03-20T09:24:11.469 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:11 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:24:11.469 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:11 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:24:11.469 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:11 smithi026 ceph-mon[68309]: osdmap e714: 12 total, 12 up, 12 in 2024-03-20T09:24:11.469 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:11 smithi026 ceph-mon[68309]: 1.b scrub ok 2024-03-20T09:24:11.469 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:11 smithi026 ceph-mon[68309]: 1.e scrub starts 2024-03-20T09:24:11.469 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:11 smithi026 ceph-mon[68309]: 1.e scrub ok 2024-03-20T09:24:11.488 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:11.506 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:24:11.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:11.509 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:24:11.610 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:24:11.641 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:11.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:11 smithi115 ceph-mon[72476]: 1.18 scrub starts 2024-03-20T09:24:11.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:11 smithi115 ceph-mon[72476]: 1.18 scrub ok 2024-03-20T09:24:11.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:11 smithi115 ceph-mon[72476]: 1.b scrub starts 2024-03-20T09:24:11.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:11 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:24:11.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:11 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:24:11.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:11 smithi115 ceph-mon[72476]: osdmap e714: 12 total, 12 up, 12 in 2024-03-20T09:24:11.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:11 smithi115 ceph-mon[72476]: 1.b scrub ok 2024-03-20T09:24:11.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:11 smithi115 ceph-mon[72476]: 1.e scrub starts 2024-03-20T09:24:11.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:11 smithi115 ceph-mon[72476]: 1.e scrub ok 2024-03-20T09:24:11.662 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:24:11.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:11.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:24:11.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:11 smithi122 ceph-mon[73495]: 1.18 scrub starts 2024-03-20T09:24:11.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:11 smithi122 ceph-mon[73495]: 1.18 scrub ok 2024-03-20T09:24:11.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:11 smithi122 ceph-mon[73495]: 1.b scrub starts 2024-03-20T09:24:11.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:11 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:24:11.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:11 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:24:11.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:11 smithi122 ceph-mon[73495]: osdmap e714: 12 total, 12 up, 12 in 2024-03-20T09:24:11.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:11 smithi122 ceph-mon[73495]: 1.b scrub ok 2024-03-20T09:24:11.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:11 smithi122 ceph-mon[73495]: 1.e scrub starts 2024-03-20T09:24:11.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:11 smithi122 ceph-mon[73495]: 1.e scrub ok 2024-03-20T09:24:11.766 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:24:11.793 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:11.811 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:24:11.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:11.814 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:24:11.857 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:24:11.878 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:24:11.915 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:24:11.942 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:11.960 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:24:11.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:11.963 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:24:12.064 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:24:12.090 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:12.108 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:24:12.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:12.111 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:24:12.211 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:24:12.242 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:12.263 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:24:12.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:12.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:24:12.367 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:24:12.377 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.9 to 1 (10000) 2024-03-20T09:24:12.380 INFO:teuthology.orchestra.run.smithi026.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-03-20T09:24:12.394 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:12.413 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:24:12.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:12.415 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:24:12.493 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:24:12.516 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:24:12.545 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:12 smithi115 ceph-mon[72476]: 1.3 scrub starts 2024-03-20T09:24:12.545 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:12 smithi115 ceph-mon[72476]: 1.3 scrub ok 2024-03-20T09:24:12.546 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:12 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1563803588' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-03-20T09:24:12.546 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:12 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3728822612' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-20T09:24:12.546 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:12 smithi115 ceph-mon[72476]: pgmap v1742: 47 pgs: 47 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:24:12.546 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:12 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1563803588' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-03-20T09:24:12.546 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:12 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3728822612' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-20T09:24:12.546 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:12 smithi115 ceph-mon[72476]: osdmap e715: 12 total, 12 up, 12 in 2024-03-20T09:24:12.547 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:12.567 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:24:12.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:12.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:24:12.672 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:24:12.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:12 smithi122 ceph-mon[73495]: 1.3 scrub starts 2024-03-20T09:24:12.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:12 smithi122 ceph-mon[73495]: 1.3 scrub ok 2024-03-20T09:24:12.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:12 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1563803588' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-03-20T09:24:12.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:12 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3728822612' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-20T09:24:12.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:12 smithi122 ceph-mon[73495]: pgmap v1742: 47 pgs: 47 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:24:12.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:12 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1563803588' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-03-20T09:24:12.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:12 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3728822612' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-20T09:24:12.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:12 smithi122 ceph-mon[73495]: osdmap e715: 12 total, 12 up, 12 in 2024-03-20T09:24:12.701 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:12.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:12 smithi026 ceph-mon[68309]: 1.3 scrub starts 2024-03-20T09:24:12.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:12 smithi026 ceph-mon[68309]: 1.3 scrub ok 2024-03-20T09:24:12.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:12 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1563803588' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-03-20T09:24:12.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:12 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3728822612' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-20T09:24:12.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:12 smithi026 ceph-mon[68309]: pgmap v1742: 47 pgs: 47 active+clean; 577 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:24:12.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:12 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1563803588' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-03-20T09:24:12.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:12 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3728822612' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-20T09:24:12.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:12 smithi026 ceph-mon[68309]: osdmap e715: 12 total, 12 up, 12 in 2024-03-20T09:24:12.720 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:24:12.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:12.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:24:12.825 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:24:12.852 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:12.870 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:24:12.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:12.874 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:24:12.975 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:24:13.005 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:13.619 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:13 smithi026 ceph-mon[68309]: 1.1d scrub starts 2024-03-20T09:24:13.619 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:13 smithi026 ceph-mon[68309]: 1.1d scrub ok 2024-03-20T09:24:13.619 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:13 smithi026 ceph-mon[68309]: 1.4 scrub starts 2024-03-20T09:24:13.619 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:13 smithi026 ceph-mon[68309]: 1.4 scrub ok 2024-03-20T09:24:13.619 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:13 smithi026 ceph-mon[68309]: osdmap e716: 12 total, 12 up, 12 in 2024-03-20T09:24:13.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:13 smithi122 ceph-mon[73495]: 1.1d scrub starts 2024-03-20T09:24:13.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:13 smithi122 ceph-mon[73495]: 1.1d scrub ok 2024-03-20T09:24:13.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:13 smithi122 ceph-mon[73495]: 1.4 scrub starts 2024-03-20T09:24:13.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:13 smithi122 ceph-mon[73495]: 1.4 scrub ok 2024-03-20T09:24:13.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:13 smithi122 ceph-mon[73495]: osdmap e716: 12 total, 12 up, 12 in 2024-03-20T09:24:13.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:13 smithi115 ceph-mon[72476]: 1.1d scrub starts 2024-03-20T09:24:13.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:13 smithi115 ceph-mon[72476]: 1.1d scrub ok 2024-03-20T09:24:13.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:13 smithi115 ceph-mon[72476]: 1.4 scrub starts 2024-03-20T09:24:13.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:13 smithi115 ceph-mon[72476]: 1.4 scrub ok 2024-03-20T09:24:13.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:13 smithi115 ceph-mon[72476]: osdmap e716: 12 total, 12 up, 12 in 2024-03-20T09:24:14.025 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-20T09:24:14.026 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:24:14.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:14 smithi122 ceph-mon[73495]: 1.12 scrub starts 2024-03-20T09:24:14.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:14 smithi122 ceph-mon[73495]: 1.12 scrub ok 2024-03-20T09:24:14.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:14 smithi122 ceph-mon[73495]: 1.0 scrub starts 2024-03-20T09:24:14.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:14 smithi122 ceph-mon[73495]: 1.0 scrub ok 2024-03-20T09:24:14.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:14 smithi122 ceph-mon[73495]: 1.2 scrub starts 2024-03-20T09:24:14.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:14 smithi122 ceph-mon[73495]: 1.2 scrub ok 2024-03-20T09:24:14.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:14 smithi122 ceph-mon[73495]: pgmap v1745: 47 pgs: 1 remapped+peering, 46 active+clean; 603 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s; 7.8 KiB/s, 0 objects/s recovering 2024-03-20T09:24:14.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:14 smithi122 ceph-mon[73495]: osdmap e717: 12 total, 12 up, 12 in 2024-03-20T09:24:14.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:14 smithi026 ceph-mon[68309]: 1.12 scrub starts 2024-03-20T09:24:14.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:14 smithi026 ceph-mon[68309]: 1.12 scrub ok 2024-03-20T09:24:14.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:14 smithi026 ceph-mon[68309]: 1.0 scrub starts 2024-03-20T09:24:14.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:14 smithi026 ceph-mon[68309]: 1.0 scrub ok 2024-03-20T09:24:14.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:14 smithi026 ceph-mon[68309]: 1.2 scrub starts 2024-03-20T09:24:14.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:14 smithi026 ceph-mon[68309]: 1.2 scrub ok 2024-03-20T09:24:14.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:14 smithi026 ceph-mon[68309]: pgmap v1745: 47 pgs: 1 remapped+peering, 46 active+clean; 603 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s; 7.8 KiB/s, 0 objects/s recovering 2024-03-20T09:24:14.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:14 smithi026 ceph-mon[68309]: osdmap e717: 12 total, 12 up, 12 in 2024-03-20T09:24:14.904 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:14 smithi115 ceph-mon[72476]: 1.12 scrub starts 2024-03-20T09:24:14.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:14 smithi115 ceph-mon[72476]: 1.12 scrub ok 2024-03-20T09:24:14.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:14 smithi115 ceph-mon[72476]: 1.0 scrub starts 2024-03-20T09:24:14.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:14 smithi115 ceph-mon[72476]: 1.0 scrub ok 2024-03-20T09:24:14.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:14 smithi115 ceph-mon[72476]: 1.2 scrub starts 2024-03-20T09:24:14.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:14 smithi115 ceph-mon[72476]: 1.2 scrub ok 2024-03-20T09:24:14.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:14 smithi115 ceph-mon[72476]: pgmap v1745: 47 pgs: 1 remapped+peering, 46 active+clean; 603 KiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s; 7.8 KiB/s, 0 objects/s recovering 2024-03-20T09:24:14.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:14 smithi115 ceph-mon[72476]: osdmap e717: 12 total, 12 up, 12 in 2024-03-20T09:24:15.254 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:24:15.255 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:24:15.258 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:24:15.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:15.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:24:15.303 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:24:15.303 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:24:15.365 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:24:15.398 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:15.419 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:24:15.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:15.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:24:15.523 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:24:15.553 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:15.573 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:24:15.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:15.576 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:24:15.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:15 smithi115 ceph-mon[72476]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-20T09:24:15.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:15 smithi115 ceph-mon[72476]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-20T09:24:15.677 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:24:15.681 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:15 smithi026 ceph-mon[68309]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-20T09:24:15.681 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:15 smithi026 ceph-mon[68309]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-20T09:24:15.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:15 smithi122 ceph-mon[73495]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-20T09:24:15.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:15 smithi122 ceph-mon[73495]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-20T09:24:15.710 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:15.731 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:24:15.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:15.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:24:15.799 DEBUG:teuthology.orchestra.run.smithi122:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-03-20T09:24:15.801 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-20T09:24:15.802 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-20T09:24:15.835 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:24:15.855 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:hints = 1 2024-03-20T09:24:15.855 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-20T09:24:15.855 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Object prefix: benchmark_data_smithi122_168655 2024-03-20T09:24:15.855 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T09:24:15.855 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 0 0 0 0 0 0 - 0 2024-03-20T09:24:15.867 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:15.894 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:24:15.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:15.897 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:24:15.998 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:24:16.024 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:16.042 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:24:16.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:16.045 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:24:16.146 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:24:16.172 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:16.190 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:24:16.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:16.193 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:24:16.294 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:24:16.352 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:16.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:16 smithi122 ceph-mon[73495]: 1.13 scrub starts 2024-03-20T09:24:16.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:16 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:24:16.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:16 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:24:16.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:16 smithi122 ceph-mon[73495]: 1.13 scrub ok 2024-03-20T09:24:16.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:16 smithi122 ceph-mon[73495]: osdmap e718: 12 total, 12 up, 12 in 2024-03-20T09:24:16.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:16 smithi122 ceph-mon[73495]: 1.1a scrub starts 2024-03-20T09:24:16.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:16 smithi122 ceph-mon[73495]: 1.1a scrub ok 2024-03-20T09:24:16.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:16 smithi122 ceph-mon[73495]: pgmap v1748: 47 pgs: 1 remapped+peering, 46 active+clean; 3.8 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 865 KiB/s wr, 16 op/s; 12 KiB/s, 0 objects/s recovering 2024-03-20T09:24:16.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:16 smithi026 ceph-mon[68309]: 1.13 scrub starts 2024-03-20T09:24:16.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:16 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:24:16.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:16 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:24:16.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:16 smithi026 ceph-mon[68309]: 1.13 scrub ok 2024-03-20T09:24:16.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:16 smithi026 ceph-mon[68309]: osdmap e718: 12 total, 12 up, 12 in 2024-03-20T09:24:16.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:16 smithi026 ceph-mon[68309]: 1.1a scrub starts 2024-03-20T09:24:16.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:16 smithi026 ceph-mon[68309]: 1.1a scrub ok 2024-03-20T09:24:16.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:16 smithi026 ceph-mon[68309]: pgmap v1748: 47 pgs: 1 remapped+peering, 46 active+clean; 3.8 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 865 KiB/s wr, 16 op/s; 12 KiB/s, 0 objects/s recovering 2024-03-20T09:24:16.855 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 1 16 2672 2656 165.991 166 0.00474546 0.00600858 2024-03-20T09:24:16.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:16 smithi115 ceph-mon[72476]: 1.13 scrub starts 2024-03-20T09:24:16.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:16 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:24:16.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:16 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:24:16.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:16 smithi115 ceph-mon[72476]: 1.13 scrub ok 2024-03-20T09:24:16.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:16 smithi115 ceph-mon[72476]: osdmap e718: 12 total, 12 up, 12 in 2024-03-20T09:24:16.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:16 smithi115 ceph-mon[72476]: 1.1a scrub starts 2024-03-20T09:24:16.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:16 smithi115 ceph-mon[72476]: 1.1a scrub ok 2024-03-20T09:24:16.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:16 smithi115 ceph-mon[72476]: pgmap v1748: 47 pgs: 1 remapped+peering, 46 active+clean; 3.8 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 865 KiB/s wr, 16 op/s; 12 KiB/s, 0 objects/s recovering 2024-03-20T09:24:17.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:17 smithi122 ceph-mon[73495]: 1.8 scrub starts 2024-03-20T09:24:17.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:17 smithi122 ceph-mon[73495]: 1.8 scrub ok 2024-03-20T09:24:17.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:17 smithi122 ceph-mon[73495]: osdmap e719: 12 total, 12 up, 12 in 2024-03-20T09:24:17.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:17 smithi122 ceph-mon[73495]: 1.d scrub starts 2024-03-20T09:24:17.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:17 smithi122 ceph-mon[73495]: 1.d scrub ok 2024-03-20T09:24:17.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:17 smithi026 ceph-mon[68309]: 1.8 scrub starts 2024-03-20T09:24:17.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:17 smithi026 ceph-mon[68309]: 1.8 scrub ok 2024-03-20T09:24:17.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:17 smithi026 ceph-mon[68309]: osdmap e719: 12 total, 12 up, 12 in 2024-03-20T09:24:17.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:17 smithi026 ceph-mon[68309]: 1.d scrub starts 2024-03-20T09:24:17.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:17 smithi026 ceph-mon[68309]: 1.d scrub ok 2024-03-20T09:24:17.855 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 2 16 6331 6315 197.33 228.688 0.00388923 0.00505935 2024-03-20T09:24:17.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:17 smithi115 ceph-mon[72476]: 1.8 scrub starts 2024-03-20T09:24:17.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:17 smithi115 ceph-mon[72476]: 1.8 scrub ok 2024-03-20T09:24:17.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:17 smithi115 ceph-mon[72476]: osdmap e719: 12 total, 12 up, 12 in 2024-03-20T09:24:17.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:17 smithi115 ceph-mon[72476]: 1.d scrub starts 2024-03-20T09:24:17.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:17 smithi115 ceph-mon[72476]: 1.d scrub ok 2024-03-20T09:24:18.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:18 smithi122 ceph-mon[73495]: 1.10 scrub starts 2024-03-20T09:24:18.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:18 smithi122 ceph-mon[73495]: 1.10 scrub ok 2024-03-20T09:24:18.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:18 smithi122 ceph-mon[73495]: 1.a scrub starts 2024-03-20T09:24:18.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:18 smithi122 ceph-mon[73495]: 1.a scrub ok 2024-03-20T09:24:18.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:18 smithi122 ceph-mon[73495]: pgmap v1750: 47 pgs: 47 active+clean; 97 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s wr, 322 op/s 2024-03-20T09:24:18.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:18 smithi026 ceph-mon[68309]: 1.10 scrub starts 2024-03-20T09:24:18.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:18 smithi026 ceph-mon[68309]: 1.10 scrub ok 2024-03-20T09:24:18.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:18 smithi026 ceph-mon[68309]: 1.a scrub starts 2024-03-20T09:24:18.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:18 smithi026 ceph-mon[68309]: 1.a scrub ok 2024-03-20T09:24:18.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:18 smithi026 ceph-mon[68309]: pgmap v1750: 47 pgs: 47 active+clean; 97 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s wr, 322 op/s 2024-03-20T09:24:18.855 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 3 16 10159 10143 211.298 239.25 0.00145458 0.00472469 2024-03-20T09:24:18.904 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:18 smithi115 ceph-mon[72476]: 1.10 scrub starts 2024-03-20T09:24:18.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:18 smithi115 ceph-mon[72476]: 1.10 scrub ok 2024-03-20T09:24:18.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:18 smithi115 ceph-mon[72476]: 1.a scrub starts 2024-03-20T09:24:18.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:18 smithi115 ceph-mon[72476]: 1.a scrub ok 2024-03-20T09:24:18.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:18 smithi115 ceph-mon[72476]: pgmap v1750: 47 pgs: 47 active+clean; 97 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s wr, 322 op/s 2024-03-20T09:24:19.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:19 smithi026 ceph-mon[68309]: 1.c scrub starts 2024-03-20T09:24:19.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:19 smithi026 ceph-mon[68309]: 1.c scrub ok 2024-03-20T09:24:19.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:19 smithi026 ceph-mon[68309]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-20T09:24:19.855 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 4 16 13284 13268 207.298 195.312 0.00332043 0.0048177 2024-03-20T09:24:19.904 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:19 smithi115 ceph-mon[72476]: 1.c scrub starts 2024-03-20T09:24:19.904 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:19 smithi115 ceph-mon[72476]: 1.c scrub ok 2024-03-20T09:24:19.904 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:19 smithi115 ceph-mon[72476]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-20T09:24:19.945 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:19 smithi122 ceph-mon[73495]: 1.c scrub starts 2024-03-20T09:24:19.945 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:19 smithi122 ceph-mon[73495]: 1.c scrub ok 2024-03-20T09:24:19.945 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:19 smithi122 ceph-mon[73495]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-20T09:24:20.704 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:20 smithi026 ceph-mon[68309]: 1.16 scrub starts 2024-03-20T09:24:20.704 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:20 smithi026 ceph-mon[68309]: 1.16 scrub ok 2024-03-20T09:24:20.705 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:20 smithi026 ceph-mon[68309]: pgmap v1751: 47 pgs: 47 active+clean; 417 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.11k op/s 2024-03-20T09:24:20.855 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 5 16 15431 15415 192.673 134.188 0.00306651 0.0051846 2024-03-20T09:24:20.904 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:20 smithi115 ceph-mon[72476]: 1.16 scrub starts 2024-03-20T09:24:20.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:20 smithi115 ceph-mon[72476]: 1.16 scrub ok 2024-03-20T09:24:20.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:20 smithi115 ceph-mon[72476]: pgmap v1751: 47 pgs: 47 active+clean; 417 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.11k op/s 2024-03-20T09:24:20.944 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:20 smithi122 ceph-mon[73495]: 1.16 scrub starts 2024-03-20T09:24:20.945 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:20 smithi122 ceph-mon[73495]: 1.16 scrub ok 2024-03-20T09:24:20.945 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:20 smithi122 ceph-mon[73495]: pgmap v1751: 47 pgs: 47 active+clean; 417 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.11k op/s 2024-03-20T09:24:21.036 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:24:21.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:21.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:24:21.142 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:24:21.183 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:21.482 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:24:21.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:21.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:24:21.531 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-20T09:24:21.589 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:24:21.628 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:21.669 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:24:21.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:21.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:24:21.782 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:24:21.820 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:21.854 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:24:21.855 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 6 16 18514 18498 192.673 192.688 0.00498353 0.0051841 2024-03-20T09:24:21.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:21.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:24:21.957 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:24:22.027 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:22.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:22 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:24:22.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:22 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:24:22.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:22 smithi026 ceph-mon[68309]: osdmap e720: 12 total, 12 up, 12 in 2024-03-20T09:24:22.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:22 smithi026 ceph-mon[68309]: pgmap v1753: 47 pgs: 47 active+clean; 784 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 116 MiB/s wr, 1.86k op/s 2024-03-20T09:24:22.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:22 smithi026 ceph-mon[68309]: osdmap e721: 12 total, 12 up, 12 in 2024-03-20T09:24:22.857 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 7 16 21462 21446 191.442 184.25 0.00621666 0.00521825 2024-03-20T09:24:22.904 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:22 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:24:22.904 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:22 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:24:22.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:22 smithi115 ceph-mon[72476]: osdmap e720: 12 total, 12 up, 12 in 2024-03-20T09:24:22.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:22 smithi115 ceph-mon[72476]: pgmap v1753: 47 pgs: 47 active+clean; 784 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 116 MiB/s wr, 1.86k op/s 2024-03-20T09:24:22.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:22 smithi115 ceph-mon[72476]: osdmap e721: 12 total, 12 up, 12 in 2024-03-20T09:24:22.944 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:22 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:24:22.945 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:22 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:24:22.945 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:22 smithi122 ceph-mon[73495]: osdmap e720: 12 total, 12 up, 12 in 2024-03-20T09:24:22.945 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:22 smithi122 ceph-mon[73495]: pgmap v1753: 47 pgs: 47 active+clean; 784 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 116 MiB/s wr, 1.86k op/s 2024-03-20T09:24:22.945 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:22 smithi122 ceph-mon[73495]: osdmap e721: 12 total, 12 up, 12 in 2024-03-20T09:24:23.857 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 8 16 23120 23104 180.464 103.625 0.00209913 0.00521022 2024-03-20T09:24:24.857 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 9 16 25047 25031 173.793 120.438 0.00479032 0.00574879 2024-03-20T09:24:24.942 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:24:24.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:24.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:24:24.957 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:24:25.051 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:24:25.083 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:25.108 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:24:25.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:25.113 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:24:25.214 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:24:25.215 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:25 smithi026 ceph-mon[68309]: pgmap v1755: 47 pgs: 47 active+clean; 1.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 143 MiB/s wr, 2.29k op/s 2024-03-20T09:24:25.215 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:25 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4168018017' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:24:25.232 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:24:25.259 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:25.284 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:24:25.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:25.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:24:25.389 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:24:25.430 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:25.445 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:25 smithi122 ceph-mon[73495]: pgmap v1755: 47 pgs: 47 active+clean; 1.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 143 MiB/s wr, 2.29k op/s 2024-03-20T09:24:25.445 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:25 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4168018017' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:24:25.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:25 smithi115 ceph-mon[72476]: pgmap v1755: 47 pgs: 47 active+clean; 1.1 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 143 MiB/s wr, 2.29k op/s 2024-03-20T09:24:25.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:25 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4168018017' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:24:25.656 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:24:25.686 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:24:25.742 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:24:25.816 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:24:25.857 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 10 16 27132 27116 169.445 130.312 0.00650746 0.00589691 2024-03-20T09:24:25.862 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:24:25.909 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:24:25.938 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:24:25.949 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:24:25.960 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:24:25.989 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:24:26.026 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:24:26.054 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:24:26.077 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:24:26.097 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:24:26.098 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:24:26.098 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:24:26.098 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:24:26.099 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:24:26.099 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:24:26.099 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:24:26.099 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:24:26.099 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:24:26.099 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:24:26.099 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:24:26.099 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:24:26.099 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:24:26.257 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:26 smithi026 ceph-mon[68309]: 1.1b scrub starts 2024-03-20T09:24:26.258 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:26 smithi026 ceph-mon[68309]: 1.1b scrub ok 2024-03-20T09:24:26.258 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:26 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:24:26.258 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:26 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:24:26.258 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:26 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:24:26.258 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:26 smithi026 ceph-mon[68309]: Cluster is now healthy 2024-03-20T09:24:26.258 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:26 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4168018017' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:24:26.258 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:26 smithi026 ceph-mon[68309]: osdmap e722: 12 total, 12 up, 12 in 2024-03-20T09:24:26.258 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:26 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1112969059' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:24:26.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:26 smithi115 ceph-mon[72476]: 1.1b scrub starts 2024-03-20T09:24:26.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:26 smithi115 ceph-mon[72476]: 1.1b scrub ok 2024-03-20T09:24:26.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:26 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:24:26.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:26 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:24:26.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:26 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:24:26.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:26 smithi115 ceph-mon[72476]: Cluster is now healthy 2024-03-20T09:24:26.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:26 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4168018017' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:24:26.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:26 smithi115 ceph-mon[72476]: osdmap e722: 12 total, 12 up, 12 in 2024-03-20T09:24:26.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:26 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1112969059' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:24:26.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:26 smithi122 ceph-mon[73495]: 1.1b scrub starts 2024-03-20T09:24:26.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:26 smithi122 ceph-mon[73495]: 1.1b scrub ok 2024-03-20T09:24:26.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:26 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:24:26.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:26 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:24:26.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:26 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:24:26.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:26 smithi122 ceph-mon[73495]: Cluster is now healthy 2024-03-20T09:24:26.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:26 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4168018017' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:24:26.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:26 smithi122 ceph-mon[73495]: osdmap e722: 12 total, 12 up, 12 in 2024-03-20T09:24:26.696 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:26 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1112969059' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:24:26.857 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 11 16 27896 27880 158.381 47.75 0.00199607 0.00598077 2024-03-20T09:24:27.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:27 smithi115 ceph-mon[72476]: from='client.16380 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:24:27.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:27 smithi115 ceph-mon[72476]: pgmap v1757: 47 pgs: 47 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 184 MiB/s wr, 2.94k op/s 2024-03-20T09:24:27.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:27 smithi122 ceph-mon[73495]: from='client.16380 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:24:27.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:27 smithi122 ceph-mon[73495]: pgmap v1757: 47 pgs: 47 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 184 MiB/s wr, 2.94k op/s 2024-03-20T09:24:27.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:27 smithi026 ceph-mon[68309]: from='client.16380 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:24:27.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:27 smithi026 ceph-mon[68309]: pgmap v1757: 47 pgs: 47 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 184 MiB/s wr, 2.94k op/s 2024-03-20T09:24:27.789 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:24:27.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:27.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:24:27.857 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 12 16 27896 27880 145.183 0 - 0.00598077 2024-03-20T09:24:27.896 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:24:27.928 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:27.949 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:24:27.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:27.952 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:24:28.053 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:24:28.091 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:28.442 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:24:28.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:28.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:24:28.547 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:24:28.580 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:28.602 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:24:28.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:28.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:24:28.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:28 smithi115 ceph-mon[72476]: osdmap e723: 12 total, 12 up, 12 in 2024-03-20T09:24:28.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:28 smithi115 ceph-mon[72476]: pgmap v1759: 47 pgs: 47 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 141 MiB/s wr, 2.25k op/s 2024-03-20T09:24:28.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:28 smithi122 ceph-mon[73495]: osdmap e723: 12 total, 12 up, 12 in 2024-03-20T09:24:28.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:28 smithi122 ceph-mon[73495]: pgmap v1759: 47 pgs: 47 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 141 MiB/s wr, 2.25k op/s 2024-03-20T09:24:28.705 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:24:28.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:28 smithi026 ceph-mon[68309]: osdmap e723: 12 total, 12 up, 12 in 2024-03-20T09:24:28.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:28 smithi026 ceph-mon[68309]: pgmap v1759: 47 pgs: 47 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 141 MiB/s wr, 2.25k op/s 2024-03-20T09:24:28.738 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:28.759 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:24:28.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:28.763 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:24:28.857 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 13 16 27896 27880 134.015 0 - 0.00598077 2024-03-20T09:24:28.864 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:24:28.896 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:28.917 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:24:28.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:28.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:24:29.021 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:24:29.051 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:29.071 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:24:29.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:29.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:24:29.103 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":1756,"stamp":"2024-03-20T09:24:25.222353+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1359675936,"num_objects":20740,"num_object_clones":0,"num_object_copies":62220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20738,"num_write_kb":1327232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24986,"ondisk_log_size":24986,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":11040876,"kb_used_data":5227184,"kb_used_omap":198,"kb_used_meta":5813369,"kb_avail":1113900948,"statfs":{"total":1151940427776,"available":1140634570752,"internally_reserved":0,"allocated":5352636416,"data_stored":5346230474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":203677,"internal_metadata":5952889955},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1258225664,"num_objects":19199,"num_object_clones":0,"num_object_copies":57597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19199,"num_write_kb":1228736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.066056"},"pg_stats":[{"pgid":"1.1e","version":"20'137","reported_seq":850,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-03-20T09:24:16.443195+0000","last_change":"2024-03-20T09:24:09.407025+0000","last_active":"2024-03-20T09:24:16.443195+0000","last_peered":"2024-03-20T09:24:16.443195+0000","last_clean":"2024-03-20T09:24:16.443195+0000","last_became_active":"2024-03-20T09:13:20.071655+0000","last_became_peered":"2024-03-20T09:13:20.071655+0000","last_unstale":"2024-03-20T09:24:16.443195+0000","last_undegraded":"2024-03-20T09:24:16.443195+0000","last_fullsized":"2024-03-20T09:24:16.443195+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:09.406870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:09.406870+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:30.492927+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'137","reported_seq":1027,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-03-20T09:24:22.396079+0000","last_change":"2024-03-20T09:24:12.250854+0000","last_active":"2024-03-20T09:24:22.396079+0000","last_peered":"2024-03-20T09:24:22.396079+0000","last_clean":"2024-03-20T09:24:22.396079+0000","last_became_active":"2024-03-20T09:07:42.013245+0000","last_became_peered":"2024-03-20T09:07:42.013245+0000","last_unstale":"2024-03-20T09:24:22.396079+0000","last_undegraded":"2024-03-20T09:24:22.396079+0000","last_fullsized":"2024-03-20T09:24:22.396079+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:12.250743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:12.250743+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:15.406212+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"20'137","reported_seq":1483,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-03-20T09:24:21.464753+0000","last_change":"2024-03-20T09:23:37.923720+0000","last_active":"2024-03-20T09:24:21.464753+0000","last_peered":"2024-03-20T09:24:21.464753+0000","last_clean":"2024-03-20T09:24:21.464753+0000","last_became_active":"2024-03-20T09:07:41.961622+0000","last_became_peered":"2024-03-20T09:07:41.961622+0000","last_unstale":"2024-03-20T09:24:21.464753+0000","last_undegraded":"2024-03-20T09:24:21.464753+0000","last_fullsized":"2024-03-20T09:24:21.464753+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:23:37.923573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:23:37.923573+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:24:41.793072+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.19","version":"20'137","reported_seq":1529,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-03-20T09:24:21.460025+0000","last_change":"2024-03-20T09:23:21.491579+0000","last_active":"2024-03-20T09:24:21.460025+0000","last_peered":"2024-03-20T09:24:21.460025+0000","last_clean":"2024-03-20T09:24:21.460025+0000","last_became_active":"2024-03-20T09:07:41.962680+0000","last_became_peered":"2024-03-20T09:07:41.962680+0000","last_unstale":"2024-03-20T09:24:21.460025+0000","last_undegraded":"2024-03-20T09:24:21.460025+0000","last_fullsized":"2024-03-20T09:24:21.460025+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:23:21.491452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:23:21.491452+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:24:39.536603+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"20'137","reported_seq":993,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-03-20T09:24:22.397545+0000","last_change":"2024-03-20T09:24:10.164173+0000","last_active":"2024-03-20T09:24:22.397545+0000","last_peered":"2024-03-20T09:24:22.397545+0000","last_clean":"2024-03-20T09:24:22.397545+0000","last_became_active":"2024-03-20T09:10:35.277481+0000","last_became_peered":"2024-03-20T09:10:35.277481+0000","last_unstale":"2024-03-20T09:24:22.397545+0000","last_undegraded":"2024-03-20T09:24:22.397545+0000","last_fullsized":"2024-03-20T09:24:22.397545+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:10.164023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:10.164023+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:13.879087+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1b","version":"20'137","reported_seq":1321,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-03-20T09:24:21.464721+0000","last_change":"2024-03-20T09:24:16.445704+0000","last_active":"2024-03-20T09:24:21.464721+0000","last_peered":"2024-03-20T09:24:21.464721+0000","last_clean":"2024-03-20T09:24:21.464721+0000","last_became_active":"2024-03-20T09:24:16.445287+0000","last_became_peered":"2024-03-20T09:24:16.445287+0000","last_unstale":"2024-03-20T09:24:21.464721+0000","last_undegraded":"2024-03-20T09:24:21.464721+0000","last_fullsized":"2024-03-20T09:24:21.464721+0000","mapping_epoch":718,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:23:13.878408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:23:13.878408+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"6.f","version":"721'1448","reported_seq":1481,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-03-20T09:24:23.244269+0000","last_change":"2024-03-20T09:24:04.030788+0000","last_active":"2024-03-20T09:24:23.244269+0000","last_peered":"2024-03-20T09:24:23.244269+0000","last_clean":"2024-03-20T09:24:23.244269+0000","last_became_active":"2024-03-20T09:24:04.030514+0000","last_became_peered":"2024-03-20T09:24:04.030514+0000","last_unstale":"2024-03-20T09:24:23.244269+0000","last_undegraded":"2024-03-20T09:24:23.244269+0000","last_fullsized":"2024-03-20T09:24:23.244269+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":1448,"log_dups_size":0,"ondisk_log_size":1448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:10.180034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94896128,"num_objects":1448,"num_object_clones":0,"num_object_copies":4344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1448,"num_write_kb":92672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"20'137","reported_seq":993,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-03-20T09:24:22.397569+0000","last_change":"2024-03-20T09:24:16.103983+0000","last_active":"2024-03-20T09:24:22.397569+0000","last_peered":"2024-03-20T09:24:22.397569+0000","last_clean":"2024-03-20T09:24:22.397569+0000","last_became_active":"2024-03-20T09:10:35.215772+0000","last_became_peered":"2024-03-20T09:10:35.215772+0000","last_unstale":"2024-03-20T09:24:22.397569+0000","last_undegraded":"2024-03-20T09:24:22.397569+0000","last_fullsized":"2024-03-20T09:24:22.397569+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:16.103860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:16.103860+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:39.371790+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"719'968","reported_seq":995,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-03-20T09:24:21.008724+0000","last_change":"2024-03-20T09:24:03.382661+0000","last_active":"2024-03-20T09:24:21.008724+0000","last_peered":"2024-03-20T09:24:21.008724+0000","last_clean":"2024-03-20T09:24:21.008724+0000","last_became_active":"2024-03-20T09:24:03.382302+0000","last_became_peered":"2024-03-20T09:24:03.382302+0000","last_unstale":"2024-03-20T09:24:21.008724+0000","last_undegraded":"2024-03-20T09:24:21.008724+0000","last_fullsized":"2024-03-20T09:24:21.008724+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":968,"log_dups_size":0,"ondisk_log_size":968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:08.851284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63438848,"num_objects":968,"num_object_clones":0,"num_object_copies":2904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":968,"num_write_kb":61952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"20'137","reported_seq":1568,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-03-20T09:24:22.398177+0000","last_change":"2024-03-20T09:23:14.062062+0000","last_active":"2024-03-20T09:24:22.398177+0000","last_peered":"2024-03-20T09:24:22.398177+0000","last_clean":"2024-03-20T09:24:22.398177+0000","last_became_active":"2024-03-20T09:20:25.341750+0000","last_became_peered":"2024-03-20T09:20:25.341750+0000","last_unstale":"2024-03-20T09:24:22.398177+0000","last_undegraded":"2024-03-20T09:24:22.398177+0000","last_fullsized":"2024-03-20T09:24:22.398177+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:23:14.061933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:23:14.061933+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:24:35.066639+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,2],"acting":[9,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.d","version":"721'1370","reported_seq":1404,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-03-20T09:24:22.978385+0000","last_change":"2024-03-20T09:24:03.382043+0000","last_active":"2024-03-20T09:24:22.978385+0000","last_peered":"2024-03-20T09:24:22.978385+0000","last_clean":"2024-03-20T09:24:22.978385+0000","last_became_active":"2024-03-20T09:24:03.381679+0000","last_became_peered":"2024-03-20T09:24:03.381679+0000","last_unstale":"2024-03-20T09:24:22.978385+0000","last_undegraded":"2024-03-20T09:24:22.978385+0000","last_fullsized":"2024-03-20T09:24:22.978385+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":1370,"log_dups_size":0,"ondisk_log_size":1370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:08.923045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89784320,"num_objects":1370,"num_object_clones":0,"num_object_copies":4110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1370,"num_write_kb":87680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"20'137","reported_seq":266,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-03-20T09:24:22.397850+0000","last_change":"2024-03-20T09:24:17.164244+0000","last_active":"2024-03-20T09:24:22.397850+0000","last_peered":"2024-03-20T09:24:22.397850+0000","last_clean":"2024-03-20T09:24:22.397850+0000","last_became_active":"2024-03-20T09:24:16.444232+0000","last_became_peered":"2024-03-20T09:24:16.444232+0000","last_unstale":"2024-03-20T09:24:22.397850+0000","last_undegraded":"2024-03-20T09:24:22.397850+0000","last_fullsized":"2024-03-20T09:24:22.397850+0000","mapping_epoch":718,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:17.164131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:17.164131+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:45.596474+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.3","version":"719'1011","reported_seq":1040,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-03-20T09:24:21.061868+0000","last_change":"2024-03-20T09:24:03.379783+0000","last_active":"2024-03-20T09:24:21.061868+0000","last_peered":"2024-03-20T09:24:21.061868+0000","last_clean":"2024-03-20T09:24:21.061868+0000","last_became_active":"2024-03-20T09:24:03.379444+0000","last_became_peered":"2024-03-20T09:24:03.379444+0000","last_unstale":"2024-03-20T09:24:21.061868+0000","last_undegraded":"2024-03-20T09:24:21.061868+0000","last_fullsized":"2024-03-20T09:24:21.061868+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":1011,"log_dups_size":0,"ondisk_log_size":1011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:12.568809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66256896,"num_objects":1011,"num_object_clones":0,"num_object_copies":3033,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1011,"num_write_kb":64704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,11],"acting":[6,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"20'137","reported_seq":1518,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-03-20T09:24:16.443077+0000","last_change":"2024-03-20T09:24:12.357887+0000","last_active":"2024-03-20T09:24:16.443077+0000","last_peered":"2024-03-20T09:24:16.443077+0000","last_clean":"2024-03-20T09:24:16.443077+0000","last_became_active":"2024-03-20T09:12:14.306384+0000","last_became_peered":"2024-03-20T09:12:14.306384+0000","last_unstale":"2024-03-20T09:24:16.443077+0000","last_undegraded":"2024-03-20T09:24:16.443077+0000","last_fullsized":"2024-03-20T09:24:16.443077+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:12.357798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:12.357798+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:33.220906+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"719'981","reported_seq":1011,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-03-20T09:24:21.008864+0000","last_change":"2024-03-20T09:24:03.382618+0000","last_active":"2024-03-20T09:24:21.008864+0000","last_peered":"2024-03-20T09:24:21.008864+0000","last_clean":"2024-03-20T09:24:21.008864+0000","last_became_active":"2024-03-20T09:24:03.382365+0000","last_became_peered":"2024-03-20T09:24:03.382365+0000","last_unstale":"2024-03-20T09:24:21.008864+0000","last_undegraded":"2024-03-20T09:24:21.008864+0000","last_fullsized":"2024-03-20T09:24:21.008864+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":981,"log_dups_size":0,"ondisk_log_size":981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:12.805985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64290816,"num_objects":981,"num_object_clones":0,"num_object_copies":2943,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":981,"num_write_kb":62784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"20'137","reported_seq":1416,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-03-20T09:24:21.460099+0000","last_change":"2024-03-20T09:23:28.458409+0000","last_active":"2024-03-20T09:24:21.460099+0000","last_peered":"2024-03-20T09:24:21.460099+0000","last_clean":"2024-03-20T09:24:21.460099+0000","last_became_active":"2024-03-20T09:07:37.003722+0000","last_became_peered":"2024-03-20T09:07:37.003722+0000","last_unstale":"2024-03-20T09:24:21.460099+0000","last_undegraded":"2024-03-20T09:24:21.460099+0000","last_fullsized":"2024-03-20T09:24:21.460099+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:23:28.458329+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:19:03.805185+0000","last_clean_scrub_stamp":"2024-03-20T09:23:28.458329+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:24:53.387396+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"720'1217","reported_seq":1249,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-03-20T09:24:21.882525+0000","last_change":"2024-03-20T09:24:04.031214+0000","last_active":"2024-03-20T09:24:21.882525+0000","last_peered":"2024-03-20T09:24:21.882525+0000","last_clean":"2024-03-20T09:24:21.882525+0000","last_became_active":"2024-03-20T09:24:04.030836+0000","last_became_peered":"2024-03-20T09:24:04.030836+0000","last_unstale":"2024-03-20T09:24:21.882525+0000","last_undegraded":"2024-03-20T09:24:21.882525+0000","last_fullsized":"2024-03-20T09:24:21.882525+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":1217,"log_dups_size":0,"ondisk_log_size":1217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:18.782944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79757312,"num_objects":1217,"num_object_clones":0,"num_object_copies":3651,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1217,"num_write_kb":77888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"20'137","reported_seq":1451,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-03-20T09:24:21.460111+0000","last_change":"2024-03-20T09:24:16.442851+0000","last_active":"2024-03-20T09:24:21.460111+0000","last_peered":"2024-03-20T09:24:21.460111+0000","last_clean":"2024-03-20T09:24:21.460111+0000","last_became_active":"2024-03-20T09:24:16.442503+0000","last_became_peered":"2024-03-20T09:24:16.442503+0000","last_unstale":"2024-03-20T09:24:21.460111+0000","last_undegraded":"2024-03-20T09:24:21.460111+0000","last_fullsized":"2024-03-20T09:24:21.460111+0000","mapping_epoch":718,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:23:25.461105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:23:25.461105+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:24:27.814348+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.4","version":"721'1465","reported_seq":1498,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-03-20T09:24:24.559592+0000","last_change":"2024-03-20T09:24:03.379935+0000","last_active":"2024-03-20T09:24:24.559592+0000","last_peered":"2024-03-20T09:24:24.559592+0000","last_clean":"2024-03-20T09:24:24.559592+0000","last_became_active":"2024-03-20T09:24:03.379675+0000","last_became_peered":"2024-03-20T09:24:03.379675+0000","last_unstale":"2024-03-20T09:24:24.559592+0000","last_undegraded":"2024-03-20T09:24:24.559592+0000","last_fullsized":"2024-03-20T09:24:24.559592+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":1465,"log_dups_size":0,"ondisk_log_size":1465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:02.502245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96010240,"num_objects":1465,"num_object_clones":0,"num_object_copies":4395,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1465,"num_write_kb":93760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"20'137","reported_seq":1435,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-03-20T09:24:22.395832+0000","last_change":"2024-03-20T09:24:11.318300+0000","last_active":"2024-03-20T09:24:22.395832+0000","last_peered":"2024-03-20T09:24:22.395832+0000","last_clean":"2024-03-20T09:24:22.395832+0000","last_became_active":"2024-03-20T09:19:54.499210+0000","last_became_peered":"2024-03-20T09:19:54.499210+0000","last_unstale":"2024-03-20T09:24:22.395832+0000","last_undegraded":"2024-03-20T09:24:22.395832+0000","last_fullsized":"2024-03-20T09:24:22.395832+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:11.318184+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:18.788771+0000","last_clean_scrub_stamp":"2024-03-20T09:24:11.318184+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:14.024707+0000","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"721'1398","reported_seq":1431,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-03-20T09:24:22.986896+0000","last_change":"2024-03-20T09:24:03.380357+0000","last_active":"2024-03-20T09:24:22.986896+0000","last_peered":"2024-03-20T09:24:22.986896+0000","last_clean":"2024-03-20T09:24:22.986896+0000","last_became_active":"2024-03-20T09:24:03.380050+0000","last_became_peered":"2024-03-20T09:24:03.380050+0000","last_unstale":"2024-03-20T09:24:22.986896+0000","last_undegraded":"2024-03-20T09:24:22.986896+0000","last_fullsized":"2024-03-20T09:24:22.986896+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":1398,"log_dups_size":0,"ondisk_log_size":1398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:16.929930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91619328,"num_objects":1398,"num_object_clones":0,"num_object_copies":4194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1398,"num_write_kb":89472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":1490,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-03-20T09:24:16.443004+0000","last_change":"2024-03-20T09:24:13.361385+0000","last_active":"2024-03-20T09:24:16.443004+0000","last_peered":"2024-03-20T09:24:16.443004+0000","last_clean":"2024-03-20T09:24:16.443004+0000","last_became_active":"2024-03-20T09:12:14.305989+0000","last_became_peered":"2024-03-20T09:12:14.305989+0000","last_unstale":"2024-03-20T09:24:16.443004+0000","last_undegraded":"2024-03-20T09:24:16.443004+0000","last_fullsized":"2024-03-20T09:24:16.443004+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:13.361279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:13.361279+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:24.465879+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.5","version":"719'1012","reported_seq":1041,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-03-20T09:24:21.079932+0000","last_change":"2024-03-20T09:24:04.031175+0000","last_active":"2024-03-20T09:24:21.079932+0000","last_peered":"2024-03-20T09:24:21.079932+0000","last_clean":"2024-03-20T09:24:21.079932+0000","last_became_active":"2024-03-20T09:24:04.030820+0000","last_became_peered":"2024-03-20T09:24:04.030820+0000","last_unstale":"2024-03-20T09:24:21.079932+0000","last_undegraded":"2024-03-20T09:24:21.079932+0000","last_fullsized":"2024-03-20T09:24:21.079932+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":1012,"log_dups_size":0,"ondisk_log_size":1012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:26.900507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66322432,"num_objects":1012,"num_object_clones":0,"num_object_copies":3036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1012,"num_write_kb":64768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"20'137","reported_seq":1750,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-03-20T09:24:22.397707+0000","last_change":"2024-03-20T09:24:13.945929+0000","last_active":"2024-03-20T09:24:22.397707+0000","last_peered":"2024-03-20T09:24:22.397707+0000","last_clean":"2024-03-20T09:24:22.397707+0000","last_became_active":"2024-03-20T09:19:54.492879+0000","last_became_peered":"2024-03-20T09:19:54.492879+0000","last_unstale":"2024-03-20T09:24:22.397707+0000","last_undegraded":"2024-03-20T09:24:22.397707+0000","last_fullsized":"2024-03-20T09:24:22.397707+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:13.945829+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:19.513511+0000","last_clean_scrub_stamp":"2024-03-20T09:24:13.945829+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:29.762487+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"721'1606","reported_seq":1640,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-03-20T09:24:24.887863+0000","last_change":"2024-03-20T09:24:03.378882+0000","last_active":"2024-03-20T09:24:24.887863+0000","last_peered":"2024-03-20T09:24:24.887863+0000","last_clean":"2024-03-20T09:24:24.887863+0000","last_became_active":"2024-03-20T09:24:03.378429+0000","last_became_peered":"2024-03-20T09:24:03.378429+0000","last_unstale":"2024-03-20T09:24:24.887863+0000","last_undegraded":"2024-03-20T09:24:24.887863+0000","last_fullsized":"2024-03-20T09:24:24.887863+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":1606,"log_dups_size":0,"ondisk_log_size":1606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:26.886749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105250816,"num_objects":1606,"num_object_clones":0,"num_object_copies":4818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1606,"num_write_kb":102784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"20'137","reported_seq":907,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-03-20T09:24:16.441169+0000","last_change":"2024-03-20T09:23:30.467662+0000","last_active":"2024-03-20T09:24:16.441169+0000","last_peered":"2024-03-20T09:24:16.441169+0000","last_clean":"2024-03-20T09:24:16.441169+0000","last_became_active":"2024-03-20T09:15:34.225022+0000","last_became_peered":"2024-03-20T09:15:34.225022+0000","last_unstale":"2024-03-20T09:24:16.441169+0000","last_undegraded":"2024-03-20T09:24:16.441169+0000","last_fullsized":"2024-03-20T09:24:16.441169+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:23:30.467535+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:20:12.169658+0000","last_clean_scrub_stamp":"2024-03-20T09:23:30.467535+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:24:58.406075+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,5],"acting":[6,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"721'1407","reported_seq":1440,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-03-20T09:24:23.395935+0000","last_change":"2024-03-20T09:24:03.379949+0000","last_active":"2024-03-20T09:24:23.395935+0000","last_peered":"2024-03-20T09:24:23.395935+0000","last_clean":"2024-03-20T09:24:23.395935+0000","last_became_active":"2024-03-20T09:24:03.379597+0000","last_became_peered":"2024-03-20T09:24:03.379597+0000","last_unstale":"2024-03-20T09:24:23.395935+0000","last_undegraded":"2024-03-20T09:24:23.395935+0000","last_fullsized":"2024-03-20T09:24:23.395935+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":1407,"log_dups_size":0,"ondisk_log_size":1407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:31.975923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92209152,"num_objects":1407,"num_object_clones":0,"num_object_copies":4221,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1407,"num_write_kb":90048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"20'137","reported_seq":1442,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-03-20T09:24:16.442809+0000","last_change":"2024-03-20T09:23:25.611676+0000","last_active":"2024-03-20T09:24:16.442809+0000","last_peered":"2024-03-20T09:24:16.442809+0000","last_clean":"2024-03-20T09:24:16.442809+0000","last_became_active":"2024-03-20T09:13:27.087760+0000","last_became_peered":"2024-03-20T09:13:27.087760+0000","last_unstale":"2024-03-20T09:24:16.442809+0000","last_undegraded":"2024-03-20T09:24:16.442809+0000","last_fullsized":"2024-03-20T09:24:16.442809+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:23:25.611535+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:14:10.560890+0000","last_clean_scrub_stamp":"2024-03-20T09:23:25.611535+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:24:53.993207+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"721'1648","reported_seq":1682,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-03-20T09:24:24.902557+0000","last_change":"2024-03-20T09:24:03.381105+0000","last_active":"2024-03-20T09:24:24.902557+0000","last_peered":"2024-03-20T09:24:24.902557+0000","last_clean":"2024-03-20T09:24:24.902557+0000","last_became_active":"2024-03-20T09:24:03.380630+0000","last_became_peered":"2024-03-20T09:24:03.380630+0000","last_unstale":"2024-03-20T09:24:24.902557+0000","last_undegraded":"2024-03-20T09:24:24.902557+0000","last_fullsized":"2024-03-20T09:24:24.902557+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":1648,"log_dups_size":0,"ondisk_log_size":1648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:11.318851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108003328,"num_objects":1648,"num_object_clones":0,"num_object_copies":4944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1648,"num_write_kb":105472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"20'137","reported_seq":1294,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-03-20T09:24:16.443360+0000","last_change":"2024-03-20T09:24:10.480839+0000","last_active":"2024-03-20T09:24:16.443360+0000","last_peered":"2024-03-20T09:24:16.443360+0000","last_clean":"2024-03-20T09:24:16.443360+0000","last_became_active":"2024-03-20T09:13:20.070596+0000","last_became_peered":"2024-03-20T09:13:20.070596+0000","last_unstale":"2024-03-20T09:24:16.443360+0000","last_undegraded":"2024-03-20T09:24:16.443360+0000","last_fullsized":"2024-03-20T09:24:16.443360+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:10.480739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:10.480739+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:10.692605+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.9","version":"721'1364","reported_seq":1402,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-03-20T09:24:23.312240+0000","last_change":"2024-03-20T09:24:13.382970+0000","last_active":"2024-03-20T09:24:23.312240+0000","last_peered":"2024-03-20T09:24:23.312240+0000","last_clean":"2024-03-20T09:24:23.312240+0000","last_became_active":"2024-03-20T09:24:13.382746+0000","last_became_peered":"2024-03-20T09:24:13.382746+0000","last_unstale":"2024-03-20T09:24:23.312240+0000","last_undegraded":"2024-03-20T09:24:23.312240+0000","last_fullsized":"2024-03-20T09:24:23.312240+0000","mapping_epoch":715,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":716,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":1364,"log_dups_size":0,"ondisk_log_size":1364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:24.455949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89391104,"num_objects":1364,"num_object_clones":0,"num_object_copies":4092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1364,"num_write_kb":87296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"20'137","reported_seq":719,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-03-20T09:24:16.441416+0000","last_change":"2024-03-20T09:24:10.821907+0000","last_active":"2024-03-20T09:24:16.441416+0000","last_peered":"2024-03-20T09:24:16.441416+0000","last_clean":"2024-03-20T09:24:16.441416+0000","last_became_active":"2024-03-20T09:19:54.506134+0000","last_became_peered":"2024-03-20T09:19:54.506134+0000","last_unstale":"2024-03-20T09:24:16.441416+0000","last_undegraded":"2024-03-20T09:24:16.441416+0000","last_fullsized":"2024-03-20T09:24:16.441416+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:10.821728+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:18:52.573886+0000","last_clean_scrub_stamp":"2024-03-20T09:24:10.821728+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:23.388764+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"719'899","reported_seq":929,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-03-20T09:24:20.270810+0000","last_change":"2024-03-20T09:24:03.379403+0000","last_active":"2024-03-20T09:24:20.270810+0000","last_peered":"2024-03-20T09:24:20.270810+0000","last_clean":"2024-03-20T09:24:20.270810+0000","last_became_active":"2024-03-20T09:24:03.379073+0000","last_became_peered":"2024-03-20T09:24:03.379073+0000","last_unstale":"2024-03-20T09:24:20.270810+0000","last_undegraded":"2024-03-20T09:24:20.270810+0000","last_fullsized":"2024-03-20T09:24:20.270810+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":899,"log_dups_size":0,"ondisk_log_size":899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:21.840916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58916864,"num_objects":899,"num_object_clones":0,"num_object_copies":2697,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":899,"num_write_kb":57536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,3],"acting":[11,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.f","version":"20'137","reported_seq":1467,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-03-20T09:24:16.442752+0000","last_change":"2024-03-20T09:23:28.597118+0000","last_active":"2024-03-20T09:24:16.442752+0000","last_peered":"2024-03-20T09:24:16.442752+0000","last_clean":"2024-03-20T09:24:16.442752+0000","last_became_active":"2024-03-20T09:19:54.524397+0000","last_became_peered":"2024-03-20T09:19:54.524397+0000","last_unstale":"2024-03-20T09:24:16.442752+0000","last_undegraded":"2024-03-20T09:24:16.442752+0000","last_fullsized":"2024-03-20T09:24:16.442752+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:23:28.597000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:23:28.597000+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:24:45.500274+0000","scrub_duration":154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"721'1618","reported_seq":1649,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-03-20T09:24:24.897842+0000","last_change":"2024-03-20T09:24:03.379324+0000","last_active":"2024-03-20T09:24:24.897842+0000","last_peered":"2024-03-20T09:24:24.897842+0000","last_clean":"2024-03-20T09:24:24.897842+0000","last_became_active":"2024-03-20T09:24:03.378961+0000","last_became_peered":"2024-03-20T09:24:03.378961+0000","last_unstale":"2024-03-20T09:24:24.897842+0000","last_undegraded":"2024-03-20T09:24:24.897842+0000","last_fullsized":"2024-03-20T09:24:24.897842+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":1618,"log_dups_size":0,"ondisk_log_size":1618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:30.598154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106037248,"num_objects":1618,"num_object_clones":0,"num_object_copies":4854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1618,"num_write_kb":103552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"570'138","reported_seq":1472,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-03-20T09:24:22.394582+0000","last_change":"2024-03-20T09:24:18.167726+0000","last_active":"2024-03-20T09:24:22.394582+0000","last_peered":"2024-03-20T09:24:22.394582+0000","last_clean":"2024-03-20T09:24:22.394582+0000","last_became_active":"2024-03-20T09:13:27.087686+0000","last_became_peered":"2024-03-20T09:13:27.087686+0000","last_unstale":"2024-03-20T09:24:22.394582+0000","last_undegraded":"2024-03-20T09:24:22.394582+0000","last_fullsized":"2024-03-20T09:24:22.394582+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"570'138","last_scrub_stamp":"2024-03-20T09:24:18.167603+0000","last_deep_scrub":"570'138","last_deep_scrub_stamp":"2024-03-20T09:16:13.725035+0000","last_clean_scrub_stamp":"2024-03-20T09:24:18.167603+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:45.806376+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"721'1326","reported_seq":1359,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-03-20T09:24:22.981626+0000","last_change":"2024-03-20T09:24:03.381846+0000","last_active":"2024-03-20T09:24:22.981626+0000","last_peered":"2024-03-20T09:24:22.981626+0000","last_clean":"2024-03-20T09:24:22.981626+0000","last_became_active":"2024-03-20T09:24:03.381499+0000","last_became_peered":"2024-03-20T09:24:03.381499+0000","last_unstale":"2024-03-20T09:24:22.981626+0000","last_undegraded":"2024-03-20T09:24:22.981626+0000","last_fullsized":"2024-03-20T09:24:22.981626+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":1326,"log_dups_size":0,"ondisk_log_size":1326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:28.474992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86900736,"num_objects":1326,"num_object_clones":0,"num_object_copies":3978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1326,"num_write_kb":84864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"20'137","reported_seq":1025,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-03-20T09:24:16.958522+0000","last_change":"2024-03-20T09:24:16.958522+0000","last_active":"2024-03-20T09:24:16.958522+0000","last_peered":"2024-03-20T09:24:16.958522+0000","last_clean":"2024-03-20T09:24:16.958522+0000","last_became_active":"2024-03-20T09:19:54.502529+0000","last_became_peered":"2024-03-20T09:19:54.502529+0000","last_unstale":"2024-03-20T09:24:16.958522+0000","last_undegraded":"2024-03-20T09:24:16.958522+0000","last_fullsized":"2024-03-20T09:24:16.958522+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:16.958439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:16.958439+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:31.241580+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"20'137","reported_seq":1333,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-03-20T09:24:22.397690+0000","last_change":"2024-03-20T09:24:13.190086+0000","last_active":"2024-03-20T09:24:22.397690+0000","last_peered":"2024-03-20T09:24:22.397690+0000","last_clean":"2024-03-20T09:24:22.397690+0000","last_became_active":"2024-03-20T09:10:34.096758+0000","last_became_peered":"2024-03-20T09:10:34.096758+0000","last_unstale":"2024-03-20T09:24:22.397690+0000","last_undegraded":"2024-03-20T09:24:22.397690+0000","last_fullsized":"2024-03-20T09:24:22.397690+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:13.189949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:13.189949+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:28.168111+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"20'137","reported_seq":1413,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-03-20T09:24:22.395715+0000","last_change":"2024-03-20T09:24:15.377223+0000","last_active":"2024-03-20T09:24:22.395715+0000","last_peered":"2024-03-20T09:24:22.395715+0000","last_clean":"2024-03-20T09:24:22.395715+0000","last_became_active":"2024-03-20T09:04:04.103645+0000","last_became_peered":"2024-03-20T09:04:04.103645+0000","last_unstale":"2024-03-20T09:24:22.395715+0000","last_undegraded":"2024-03-20T09:24:22.395715+0000","last_fullsized":"2024-03-20T09:24:22.395715+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:15.377106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:15.377106+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:23.936408+0000","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"20'137","reported_seq":867,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-03-20T09:24:22.397651+0000","last_change":"2024-03-20T09:24:17.124668+0000","last_active":"2024-03-20T09:24:22.397651+0000","last_peered":"2024-03-20T09:24:22.397651+0000","last_clean":"2024-03-20T09:24:22.397651+0000","last_became_active":"2024-03-20T09:10:34.097191+0000","last_became_peered":"2024-03-20T09:10:34.097191+0000","last_unstale":"2024-03-20T09:24:22.397651+0000","last_undegraded":"2024-03-20T09:24:22.397651+0000","last_fullsized":"2024-03-20T09:24:22.397651+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:17.124502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:17.124502+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:26.158088+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"20'137","reported_seq":1277,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-03-20T09:24:22.397483+0000","last_change":"2024-03-20T09:24:08.964670+0000","last_active":"2024-03-20T09:24:22.397483+0000","last_peered":"2024-03-20T09:24:22.397483+0000","last_clean":"2024-03-20T09:24:22.397483+0000","last_became_active":"2024-03-20T09:20:25.342982+0000","last_became_peered":"2024-03-20T09:20:25.342982+0000","last_unstale":"2024-03-20T09:24:22.397483+0000","last_undegraded":"2024-03-20T09:24:22.397483+0000","last_fullsized":"2024-03-20T09:24:22.397483+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:08.964577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:08.964577+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:09.518528+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"20'137","reported_seq":1102,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-03-20T09:24:22.396309+0000","last_change":"2024-03-20T09:24:19.259161+0000","last_active":"2024-03-20T09:24:22.396309+0000","last_peered":"2024-03-20T09:24:22.396309+0000","last_clean":"2024-03-20T09:24:22.396309+0000","last_became_active":"2024-03-20T09:13:27.087760+0000","last_became_peered":"2024-03-20T09:13:27.087760+0000","last_unstale":"2024-03-20T09:24:22.396309+0000","last_undegraded":"2024-03-20T09:24:22.396309+0000","last_fullsized":"2024-03-20T09:24:22.396309+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:19.259031+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:16:15.448920+0000","last_clean_scrub_stamp":"2024-03-20T09:24:19.259031+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:22.996346+0000","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"20'137","reported_seq":1426,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-03-20T09:24:21.464886+0000","last_change":"2024-03-20T09:23:34.995066+0000","last_active":"2024-03-20T09:24:21.464886+0000","last_peered":"2024-03-20T09:24:21.464886+0000","last_clean":"2024-03-20T09:24:21.464886+0000","last_became_active":"2024-03-20T09:10:35.095339+0000","last_became_peered":"2024-03-20T09:10:35.095339+0000","last_unstale":"2024-03-20T09:24:21.464886+0000","last_undegraded":"2024-03-20T09:24:21.464886+0000","last_fullsized":"2024-03-20T09:24:21.464886+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:23:34.994901+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:23.640533+0000","last_clean_scrub_stamp":"2024-03-20T09:23:34.994901+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:24:47.732179+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"20'137","reported_seq":1505,"reported_epoch":721,"state":"active+clean","last_fresh":"2024-03-20T09:24:22.396829+0000","last_change":"2024-03-20T09:23:14.976861+0000","last_active":"2024-03-20T09:24:22.396829+0000","last_peered":"2024-03-20T09:24:22.396829+0000","last_clean":"2024-03-20T09:24:22.396829+0000","last_became_active":"2024-03-20T09:20:25.349082+0000","last_became_peered":"2024-03-20T09:20:25.349082+0000","last_unstale":"2024-03-20T09:24:22.396829+0000","last_undegraded":"2024-03-20T09:24:22.396829+0000","last_fullsized":"2024-03-20T09:24:22.396829+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:23:14.976736+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:16:15.754362+0000","last_clean_scrub_stamp":"2024-03-20T09:23:14.976736+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:24:26.212194+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"20'137","reported_seq":758,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-03-20T09:24:16.443705+0000","last_change":"2024-03-20T09:23:41.251626+0000","last_active":"2024-03-20T09:24:16.443705+0000","last_peered":"2024-03-20T09:24:16.443705+0000","last_clean":"2024-03-20T09:24:16.443705+0000","last_became_active":"2024-03-20T09:12:14.307581+0000","last_became_peered":"2024-03-20T09:12:14.307581+0000","last_unstale":"2024-03-20T09:24:16.443705+0000","last_undegraded":"2024-03-20T09:24:16.443705+0000","last_fullsized":"2024-03-20T09:24:16.443705+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:23:41.251546+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:23:41.251546+0000","last_clean_scrub_stamp":"2024-03-20T09:23:41.251546+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:24:58.292060+0000","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"20'137","reported_seq":1024,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-03-20T09:24:16.441608+0000","last_change":"2024-03-20T09:24:15.799821+0000","last_active":"2024-03-20T09:24:16.441608+0000","last_peered":"2024-03-20T09:24:16.441608+0000","last_clean":"2024-03-20T09:24:16.441608+0000","last_became_active":"2024-03-20T09:19:54.502391+0000","last_became_peered":"2024-03-20T09:19:54.502391+0000","last_unstale":"2024-03-20T09:24:16.441608+0000","last_undegraded":"2024-03-20T09:24:16.441608+0000","last_fullsized":"2024-03-20T09:24:16.441608+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:15.799637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:15.799637+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:30.958859+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":1359085568,"num_objects":20738,"num_object_clones":0,"num_object_copies":62214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20738,"num_write_kb":1327232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4043374592,"data_stored":4043374592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20738,"ondisk_log_size":20738,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4248,"ondisk_log_size":4248,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860755,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":735508,"kb_used_data":94856,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":93009644,"statfs":{"total":95995035648,"available":95241875456,"internally_reserved":0,"allocated":97132544,"data_stored":96603341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":656013764},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":89,"seq":382252089690,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":678032,"kb_used_data":209544,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93067120,"statfs":{"total":95995035648,"available":95300730880,"internally_reserved":0,"allocated":214573056,"data_stored":214043853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":479721920},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":80,"seq":343597384034,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":465824,"kb_used_data":312468,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":93279328,"statfs":{"total":95995035648,"available":95518031872,"internally_reserved":0,"allocated":319967232,"data_stored":319433564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":157022656},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":70,"seq":300647711077,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":479504,"kb_used_data":127624,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93265648,"statfs":{"total":95995035648,"available":95504023552,"internally_reserved":0,"allocated":130686976,"data_stored":130157773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":360315334},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":60,"seq":257698038122,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":959960,"kb_used_data":333200,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":92785192,"statfs":{"total":95995035648,"available":95012036608,"internally_reserved":0,"allocated":341196800,"data_stored":340664045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":641792458},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332463,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":744912,"kb_used_data":236168,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93000240,"statfs":{"total":95995035648,"available":95232245760,"internally_reserved":0,"allocated":241836032,"data_stored":241306829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":520944070},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575242,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1314992,"kb_used_data":692384,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":92430160,"statfs":{"total":95995035648,"available":94648483840,"internally_reserved":0,"allocated":709001216,"data_stored":708463033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":637532614},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738760,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1237932,"kb_used_data":588960,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":92507220,"statfs":{"total":95995035648,"available":94727393280,"internally_reserved":0,"allocated":603095040,"data_stored":602556857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411715,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1217336,"kb_used_data":807144,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92527816,"statfs":{"total":95995035648,"available":94748483584,"internally_reserved":0,"allocated":826515456,"data_stored":825973721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":420018629},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150079,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1268328,"kb_used_data":759584,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":92476824,"statfs":{"total":95995035648,"available":94696267776,"internally_reserved":0,"allocated":777814016,"data_stored":777275833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":520944071},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953852,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":716624,"kb_used_data":243976,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93028528,"statfs":{"total":95995035648,"available":95261212672,"internally_reserved":0,"allocated":249831424,"data_stored":249302221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":443,"seq":1902670512295,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1221924,"kb_used_data":821276,"kb_used_omap":181,"kb_used_meta":400458,"kb_avail":92523228,"statfs":{"total":95995035648,"available":94743785472,"internally_reserved":0,"allocated":840986624,"data_stored":840449404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":186131,"internal_metadata":410069229},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":274464768,"data_stored":274464768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":395575296,"data_stored":395575296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":365821952,"data_stored":365821952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":826671104,"data_stored":826671104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":262799360,"data_stored":262799360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":448069632,"data_stored":448069632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":267190272,"data_stored":267190272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":342294528,"data_stored":342294528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":152895488,"data_stored":152895488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":367329280,"data_stored":367329280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":222232576,"data_stored":222232576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":118030336,"data_stored":118030336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:24:29.106 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:24:29.175 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:24:29.204 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:29.354 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:24:29.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:29.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:24:29.459 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:24:29.488 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:29.508 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:24:29.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:29.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:24:29.612 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:24:29.643 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:29.656 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:24:29.663 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:24:29.663 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:24:29.663 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-03-20T09:24:29.663 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-03-20T09:24:29.663 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:24:29.664 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:24:29.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:29.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:24:29.768 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:24:29.801 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:29.816 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:24:29.823 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:24:29.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:29.826 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:24:29.857 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 14 16 27896 27880 124.443 0 - 0.00598077 2024-03-20T09:24:29.927 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:24:29.957 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:30.060 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:24:30.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:30.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:24:30.164 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:24:30.194 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:30.330 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:24:30.447 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:24:30.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:30.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:24:30.551 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:24:30.583 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:30.604 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:24:30.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:30.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:24:30.708 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:24:30.737 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:30.812 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:24:30.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:30.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:24:30.857 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 15 16 27896 27880 116.147 0 - 0.00598077 2024-03-20T09:24:30.860 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:24:30.917 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:24:30.945 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:31.094 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:24:31.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:31.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:24:31.198 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:24:31.231 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:31.253 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:24:31.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:31.256 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:24:31.357 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:24:31.384 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:31 smithi026 ceph-mon[68309]: pgmap v1760: 47 pgs: 47 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 122 MiB/s wr, 1.95k op/s 2024-03-20T09:24:31.387 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:31.637 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-20T09:24:31.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:31 smithi115 ceph-mon[72476]: pgmap v1760: 47 pgs: 47 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 122 MiB/s wr, 1.95k op/s 2024-03-20T09:24:31.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:31 smithi122 ceph-mon[73495]: pgmap v1760: 47 pgs: 47 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 122 MiB/s wr, 1.95k op/s 2024-03-20T09:24:31.858 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 16 16 27896 27880 108.888 0 - 0.00598077 2024-03-20T09:24:32.300 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:32 smithi026 ceph-mon[68309]: 1.14 scrub starts 2024-03-20T09:24:32.300 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:32 smithi026 ceph-mon[68309]: 1.14 scrub ok 2024-03-20T09:24:32.333 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:24:32.338 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:24:32.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:32.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:24:32.443 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:24:32.473 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:32.546 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:24:32.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:32.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:24:32.650 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:24:32.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:32 smithi115 ceph-mon[72476]: 1.14 scrub starts 2024-03-20T09:24:32.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:32 smithi115 ceph-mon[72476]: 1.14 scrub ok 2024-03-20T09:24:32.682 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:32.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:32 smithi122 ceph-mon[73495]: 1.14 scrub starts 2024-03-20T09:24:32.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:32 smithi122 ceph-mon[73495]: 1.14 scrub ok 2024-03-20T09:24:32.703 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:24:32.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:32.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:24:32.807 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:24:32.841 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:32.858 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 17 16 27896 27880 102.483 0 - 0.00598077 2024-03-20T09:24:32.943 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:24:32.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:32.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:24:33.047 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:24:33.078 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:33.088 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:24:33.090 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:24:33.468 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:33 smithi026 ceph-mon[68309]: 1.6 deep-scrub starts 2024-03-20T09:24:33.469 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:33 smithi026 ceph-mon[68309]: 1.6 deep-scrub ok 2024-03-20T09:24:33.469 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:33 smithi026 ceph-mon[68309]: pgmap v1761: 47 pgs: 47 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-03-20T09:24:33.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:33 smithi115 ceph-mon[72476]: 1.6 deep-scrub starts 2024-03-20T09:24:33.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:33 smithi115 ceph-mon[72476]: 1.6 deep-scrub ok 2024-03-20T09:24:33.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:33 smithi115 ceph-mon[72476]: pgmap v1761: 47 pgs: 47 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-03-20T09:24:33.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:33 smithi122 ceph-mon[73495]: 1.6 deep-scrub starts 2024-03-20T09:24:33.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:33 smithi122 ceph-mon[73495]: 1.6 deep-scrub ok 2024-03-20T09:24:33.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:33 smithi122 ceph-mon[73495]: pgmap v1761: 47 pgs: 47 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-03-20T09:24:33.858 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 18 16 27896 27880 96.7893 0 - 0.00598077 2024-03-20T09:24:34.330 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:24:34.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:34.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:24:34.351 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:24:34.364 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:34 smithi026 ceph-mon[68309]: from='client.16386 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:24:34.364 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:34 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:24:34.435 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:24:34.461 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:34.479 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:24:34.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:34 smithi122 ceph-mon[73495]: from='client.16386 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:24:34.482 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:34 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:24:34.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:34.483 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:24:34.584 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:24:34.607 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:34 smithi115 ceph-mon[72476]: from='client.16386 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:24:34.608 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:34 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:24:34.610 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:34.636 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:24:34.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:34.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:24:34.740 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:24:34.767 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:34.858 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 19 16 27896 27880 91.6953 0 - 0.00598077 2024-03-20T09:24:34.979 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":1761,"stamp":"2024-03-20T09:24:32.159916+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1828782624,"num_objects":27898,"num_object_clones":0,"num_object_copies":83694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27896,"num_write_kb":1785344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32144,"ondisk_log_size":32144,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":12752248,"kb_used_data":6876064,"kb_used_omap":198,"kb_used_meta":5875833,"kb_avail":1112189576,"statfs":{"total":1151940427776,"available":1138882125824,"internally_reserved":0,"allocated":7041089536,"data_stored":7034521954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":203677,"internal_metadata":6016853091},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,16,9],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":663617536,"num_objects":10126,"num_object_clones":0,"num_object_copies":30378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10126,"num_write_kb":648064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001782"},"pg_stats":[{"pgid":"1.1e","version":"20'137","reported_seq":858,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.584343+0000","last_change":"2024-03-20T09:24:09.407025+0000","last_active":"2024-03-20T09:24:27.584343+0000","last_peered":"2024-03-20T09:24:27.584343+0000","last_clean":"2024-03-20T09:24:27.584343+0000","last_became_active":"2024-03-20T09:13:20.071655+0000","last_became_peered":"2024-03-20T09:13:20.071655+0000","last_unstale":"2024-03-20T09:24:27.584343+0000","last_undegraded":"2024-03-20T09:24:27.584343+0000","last_fullsized":"2024-03-20T09:24:27.584343+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:09.406870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:09.406870+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:30.492927+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'137","reported_seq":1031,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.584148+0000","last_change":"2024-03-20T09:24:12.250854+0000","last_active":"2024-03-20T09:24:27.584148+0000","last_peered":"2024-03-20T09:24:27.584148+0000","last_clean":"2024-03-20T09:24:27.584148+0000","last_became_active":"2024-03-20T09:07:42.013245+0000","last_became_peered":"2024-03-20T09:07:42.013245+0000","last_unstale":"2024-03-20T09:24:27.584148+0000","last_undegraded":"2024-03-20T09:24:27.584148+0000","last_fullsized":"2024-03-20T09:24:27.584148+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:12.250743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:12.250743+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:15.406212+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"20'137","reported_seq":1489,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.584245+0000","last_change":"2024-03-20T09:23:37.923720+0000","last_active":"2024-03-20T09:24:27.584245+0000","last_peered":"2024-03-20T09:24:27.584245+0000","last_clean":"2024-03-20T09:24:27.584245+0000","last_became_active":"2024-03-20T09:07:41.961622+0000","last_became_peered":"2024-03-20T09:07:41.961622+0000","last_unstale":"2024-03-20T09:24:27.584245+0000","last_undegraded":"2024-03-20T09:24:27.584245+0000","last_fullsized":"2024-03-20T09:24:27.584245+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:23:37.923573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:23:37.923573+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:24:41.793072+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.19","version":"20'137","reported_seq":1535,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.380371+0000","last_change":"2024-03-20T09:23:21.491579+0000","last_active":"2024-03-20T09:24:27.380371+0000","last_peered":"2024-03-20T09:24:27.380371+0000","last_clean":"2024-03-20T09:24:27.380371+0000","last_became_active":"2024-03-20T09:07:41.962680+0000","last_became_peered":"2024-03-20T09:07:41.962680+0000","last_unstale":"2024-03-20T09:24:27.380371+0000","last_undegraded":"2024-03-20T09:24:27.380371+0000","last_fullsized":"2024-03-20T09:24:27.380371+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:23:21.491452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:23:21.491452+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:24:39.536603+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"20'137","reported_seq":997,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.583972+0000","last_change":"2024-03-20T09:24:10.164173+0000","last_active":"2024-03-20T09:24:27.583972+0000","last_peered":"2024-03-20T09:24:27.583972+0000","last_clean":"2024-03-20T09:24:27.583972+0000","last_became_active":"2024-03-20T09:10:35.277481+0000","last_became_peered":"2024-03-20T09:10:35.277481+0000","last_unstale":"2024-03-20T09:24:27.583972+0000","last_undegraded":"2024-03-20T09:24:27.583972+0000","last_fullsized":"2024-03-20T09:24:27.583972+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:10.164023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:10.164023+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:13.879087+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1b","version":"20'137","reported_seq":1335,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.584252+0000","last_change":"2024-03-20T09:24:25.023050+0000","last_active":"2024-03-20T09:24:27.584252+0000","last_peered":"2024-03-20T09:24:27.584252+0000","last_clean":"2024-03-20T09:24:27.584252+0000","last_became_active":"2024-03-20T09:24:16.445287+0000","last_became_peered":"2024-03-20T09:24:16.445287+0000","last_unstale":"2024-03-20T09:24:27.584252+0000","last_undegraded":"2024-03-20T09:24:27.584252+0000","last_fullsized":"2024-03-20T09:24:27.584252+0000","mapping_epoch":718,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:25.022894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:25.022894+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:28.182388+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"6.f","version":"722'1796","reported_seq":1833,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.380991+0000","last_change":"2024-03-20T09:24:04.030788+0000","last_active":"2024-03-20T09:24:27.380991+0000","last_peered":"2024-03-20T09:24:27.380991+0000","last_clean":"2024-03-20T09:24:27.380991+0000","last_became_active":"2024-03-20T09:24:04.030514+0000","last_became_peered":"2024-03-20T09:24:04.030514+0000","last_unstale":"2024-03-20T09:24:27.380991+0000","last_undegraded":"2024-03-20T09:24:27.380991+0000","last_fullsized":"2024-03-20T09:24:27.380991+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":1796,"log_dups_size":0,"ondisk_log_size":1796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:10.180034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117702656,"num_objects":1796,"num_object_clones":0,"num_object_copies":5388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1796,"num_write_kb":114944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"20'137","reported_seq":997,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.583138+0000","last_change":"2024-03-20T09:24:16.103983+0000","last_active":"2024-03-20T09:24:27.583138+0000","last_peered":"2024-03-20T09:24:27.583138+0000","last_clean":"2024-03-20T09:24:27.583138+0000","last_became_active":"2024-03-20T09:10:35.215772+0000","last_became_peered":"2024-03-20T09:10:35.215772+0000","last_unstale":"2024-03-20T09:24:27.583138+0000","last_undegraded":"2024-03-20T09:24:27.583138+0000","last_fullsized":"2024-03-20T09:24:27.583138+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:16.103860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:16.103860+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:39.371790+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"722'1721","reported_seq":1759,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.380506+0000","last_change":"2024-03-20T09:24:03.382661+0000","last_active":"2024-03-20T09:24:27.380506+0000","last_peered":"2024-03-20T09:24:27.380506+0000","last_clean":"2024-03-20T09:24:27.380506+0000","last_became_active":"2024-03-20T09:24:03.382302+0000","last_became_peered":"2024-03-20T09:24:03.382302+0000","last_unstale":"2024-03-20T09:24:27.380506+0000","last_undegraded":"2024-03-20T09:24:27.380506+0000","last_fullsized":"2024-03-20T09:24:27.380506+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":1721,"log_dups_size":0,"ondisk_log_size":1721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:08.851284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112787456,"num_objects":1721,"num_object_clones":0,"num_object_copies":5163,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1721,"num_write_kb":110144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"20'137","reported_seq":1572,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.584266+0000","last_change":"2024-03-20T09:23:14.062062+0000","last_active":"2024-03-20T09:24:27.584266+0000","last_peered":"2024-03-20T09:24:27.584266+0000","last_clean":"2024-03-20T09:24:27.584266+0000","last_became_active":"2024-03-20T09:20:25.341750+0000","last_became_peered":"2024-03-20T09:20:25.341750+0000","last_unstale":"2024-03-20T09:24:27.584266+0000","last_undegraded":"2024-03-20T09:24:27.584266+0000","last_fullsized":"2024-03-20T09:24:27.584266+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:23:14.061933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:23:14.061933+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:24:35.066639+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,2],"acting":[9,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.d","version":"722'1738","reported_seq":1776,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.583036+0000","last_change":"2024-03-20T09:24:03.382043+0000","last_active":"2024-03-20T09:24:27.583036+0000","last_peered":"2024-03-20T09:24:27.583036+0000","last_clean":"2024-03-20T09:24:27.583036+0000","last_became_active":"2024-03-20T09:24:03.381679+0000","last_became_peered":"2024-03-20T09:24:03.381679+0000","last_unstale":"2024-03-20T09:24:27.583036+0000","last_undegraded":"2024-03-20T09:24:27.583036+0000","last_fullsized":"2024-03-20T09:24:27.583036+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":1738,"log_dups_size":0,"ondisk_log_size":1738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:08.923045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113901568,"num_objects":1738,"num_object_clones":0,"num_object_copies":5214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1738,"num_write_kb":111232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"20'137","reported_seq":270,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.584112+0000","last_change":"2024-03-20T09:24:17.164244+0000","last_active":"2024-03-20T09:24:27.584112+0000","last_peered":"2024-03-20T09:24:27.584112+0000","last_clean":"2024-03-20T09:24:27.584112+0000","last_became_active":"2024-03-20T09:24:16.444232+0000","last_became_peered":"2024-03-20T09:24:16.444232+0000","last_unstale":"2024-03-20T09:24:27.584112+0000","last_undegraded":"2024-03-20T09:24:27.584112+0000","last_fullsized":"2024-03-20T09:24:27.584112+0000","mapping_epoch":718,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:17.164131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:17.164131+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:45.596474+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.3","version":"722'1702","reported_seq":1732,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.583612+0000","last_change":"2024-03-20T09:24:03.379783+0000","last_active":"2024-03-20T09:24:27.583612+0000","last_peered":"2024-03-20T09:24:27.583612+0000","last_clean":"2024-03-20T09:24:27.583612+0000","last_became_active":"2024-03-20T09:24:03.379444+0000","last_became_peered":"2024-03-20T09:24:03.379444+0000","last_unstale":"2024-03-20T09:24:27.583612+0000","last_undegraded":"2024-03-20T09:24:27.583612+0000","last_fullsized":"2024-03-20T09:24:27.583612+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":1702,"log_dups_size":0,"ondisk_log_size":1702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:12.568809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111542272,"num_objects":1702,"num_object_clones":0,"num_object_copies":5106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1702,"num_write_kb":108928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,11],"acting":[6,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"20'137","reported_seq":1526,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.584106+0000","last_change":"2024-03-20T09:24:12.357887+0000","last_active":"2024-03-20T09:24:27.584106+0000","last_peered":"2024-03-20T09:24:27.584106+0000","last_clean":"2024-03-20T09:24:27.584106+0000","last_became_active":"2024-03-20T09:12:14.306384+0000","last_became_peered":"2024-03-20T09:12:14.306384+0000","last_unstale":"2024-03-20T09:24:27.584106+0000","last_undegraded":"2024-03-20T09:24:27.584106+0000","last_fullsized":"2024-03-20T09:24:27.584106+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:12.357798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:12.357798+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:33.220906+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"722'1727","reported_seq":1765,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.380428+0000","last_change":"2024-03-20T09:24:03.382618+0000","last_active":"2024-03-20T09:24:27.380428+0000","last_peered":"2024-03-20T09:24:27.380428+0000","last_clean":"2024-03-20T09:24:27.380428+0000","last_became_active":"2024-03-20T09:24:03.382365+0000","last_became_peered":"2024-03-20T09:24:03.382365+0000","last_unstale":"2024-03-20T09:24:27.380428+0000","last_undegraded":"2024-03-20T09:24:27.380428+0000","last_fullsized":"2024-03-20T09:24:27.380428+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":1727,"log_dups_size":0,"ondisk_log_size":1727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:12.805985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113180672,"num_objects":1727,"num_object_clones":0,"num_object_copies":5181,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1727,"num_write_kb":110528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"20'137","reported_seq":1422,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.380418+0000","last_change":"2024-03-20T09:23:28.458409+0000","last_active":"2024-03-20T09:24:27.380418+0000","last_peered":"2024-03-20T09:24:27.380418+0000","last_clean":"2024-03-20T09:24:27.380418+0000","last_became_active":"2024-03-20T09:07:37.003722+0000","last_became_peered":"2024-03-20T09:07:37.003722+0000","last_unstale":"2024-03-20T09:24:27.380418+0000","last_undegraded":"2024-03-20T09:24:27.380418+0000","last_fullsized":"2024-03-20T09:24:27.380418+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:23:28.458329+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:19:03.805185+0000","last_clean_scrub_stamp":"2024-03-20T09:23:28.458329+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:24:53.387396+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"722'1793","reported_seq":1831,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.584301+0000","last_change":"2024-03-20T09:24:04.031214+0000","last_active":"2024-03-20T09:24:27.584301+0000","last_peered":"2024-03-20T09:24:27.584301+0000","last_clean":"2024-03-20T09:24:27.584301+0000","last_became_active":"2024-03-20T09:24:04.030836+0000","last_became_peered":"2024-03-20T09:24:04.030836+0000","last_unstale":"2024-03-20T09:24:27.584301+0000","last_undegraded":"2024-03-20T09:24:27.584301+0000","last_fullsized":"2024-03-20T09:24:27.584301+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":1793,"log_dups_size":0,"ondisk_log_size":1793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:18.782944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117506048,"num_objects":1793,"num_object_clones":0,"num_object_copies":5379,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1793,"num_write_kb":114752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"20'137","reported_seq":1465,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:31.549689+0000","last_change":"2024-03-20T09:24:31.549717+0000","last_active":"2024-03-20T09:24:31.549689+0000","last_peered":"2024-03-20T09:24:31.549689+0000","last_clean":"2024-03-20T09:24:31.549689+0000","last_became_active":"2024-03-20T09:24:16.442503+0000","last_became_peered":"2024-03-20T09:24:16.442503+0000","last_unstale":"2024-03-20T09:24:31.549689+0000","last_undegraded":"2024-03-20T09:24:31.549689+0000","last_fullsized":"2024-03-20T09:24:31.549689+0000","mapping_epoch":718,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:31.549603+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:24:31.549603+0000","last_clean_scrub_stamp":"2024-03-20T09:24:31.549603+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:47.943434+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.4","version":"722'1672","reported_seq":1709,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.584255+0000","last_change":"2024-03-20T09:24:03.379935+0000","last_active":"2024-03-20T09:24:27.584255+0000","last_peered":"2024-03-20T09:24:27.584255+0000","last_clean":"2024-03-20T09:24:27.584255+0000","last_became_active":"2024-03-20T09:24:03.379675+0000","last_became_peered":"2024-03-20T09:24:03.379675+0000","last_unstale":"2024-03-20T09:24:27.584255+0000","last_undegraded":"2024-03-20T09:24:27.584255+0000","last_fullsized":"2024-03-20T09:24:27.584255+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":1672,"log_dups_size":0,"ondisk_log_size":1672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:02.502245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109576192,"num_objects":1672,"num_object_clones":0,"num_object_copies":5016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1672,"num_write_kb":107008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"20'137","reported_seq":1439,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.584138+0000","last_change":"2024-03-20T09:24:11.318300+0000","last_active":"2024-03-20T09:24:27.584138+0000","last_peered":"2024-03-20T09:24:27.584138+0000","last_clean":"2024-03-20T09:24:27.584138+0000","last_became_active":"2024-03-20T09:19:54.499210+0000","last_became_peered":"2024-03-20T09:19:54.499210+0000","last_unstale":"2024-03-20T09:24:27.584138+0000","last_undegraded":"2024-03-20T09:24:27.584138+0000","last_fullsized":"2024-03-20T09:24:27.584138+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:11.318184+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:18.788771+0000","last_clean_scrub_stamp":"2024-03-20T09:24:11.318184+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:14.024707+0000","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"722'1776","reported_seq":1814,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.583051+0000","last_change":"2024-03-20T09:24:03.380357+0000","last_active":"2024-03-20T09:24:27.583051+0000","last_peered":"2024-03-20T09:24:27.583051+0000","last_clean":"2024-03-20T09:24:27.583051+0000","last_became_active":"2024-03-20T09:24:03.380050+0000","last_became_peered":"2024-03-20T09:24:03.380050+0000","last_unstale":"2024-03-20T09:24:27.583051+0000","last_undegraded":"2024-03-20T09:24:27.583051+0000","last_fullsized":"2024-03-20T09:24:27.583051+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":1776,"log_dups_size":0,"ondisk_log_size":1776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:16.929930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116391936,"num_objects":1776,"num_object_clones":0,"num_object_copies":5328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1776,"num_write_kb":113664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":1498,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.584120+0000","last_change":"2024-03-20T09:24:13.361385+0000","last_active":"2024-03-20T09:24:27.584120+0000","last_peered":"2024-03-20T09:24:27.584120+0000","last_clean":"2024-03-20T09:24:27.584120+0000","last_became_active":"2024-03-20T09:12:14.305989+0000","last_became_peered":"2024-03-20T09:12:14.305989+0000","last_unstale":"2024-03-20T09:24:27.584120+0000","last_undegraded":"2024-03-20T09:24:27.584120+0000","last_fullsized":"2024-03-20T09:24:27.584120+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:13.361279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:13.361279+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:24.465879+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.5","version":"722'1763","reported_seq":1791,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.583717+0000","last_change":"2024-03-20T09:24:04.031175+0000","last_active":"2024-03-20T09:24:27.583717+0000","last_peered":"2024-03-20T09:24:27.583717+0000","last_clean":"2024-03-20T09:24:27.583717+0000","last_became_active":"2024-03-20T09:24:04.030820+0000","last_became_peered":"2024-03-20T09:24:04.030820+0000","last_unstale":"2024-03-20T09:24:27.583717+0000","last_undegraded":"2024-03-20T09:24:27.583717+0000","last_fullsized":"2024-03-20T09:24:27.583717+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":1763,"log_dups_size":0,"ondisk_log_size":1763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:26.900507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115539968,"num_objects":1763,"num_object_clones":0,"num_object_copies":5289,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1763,"num_write_kb":112832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"20'137","reported_seq":1754,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.681350+0000","last_change":"2024-03-20T09:24:13.945929+0000","last_active":"2024-03-20T09:24:27.681350+0000","last_peered":"2024-03-20T09:24:27.681350+0000","last_clean":"2024-03-20T09:24:27.681350+0000","last_became_active":"2024-03-20T09:19:54.492879+0000","last_became_peered":"2024-03-20T09:19:54.492879+0000","last_unstale":"2024-03-20T09:24:27.681350+0000","last_undegraded":"2024-03-20T09:24:27.681350+0000","last_fullsized":"2024-03-20T09:24:27.681350+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:13.945829+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:19.513511+0000","last_clean_scrub_stamp":"2024-03-20T09:24:13.945829+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:29.762487+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"722'1767","reported_seq":1805,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.583776+0000","last_change":"2024-03-20T09:24:03.378882+0000","last_active":"2024-03-20T09:24:27.583776+0000","last_peered":"2024-03-20T09:24:27.583776+0000","last_clean":"2024-03-20T09:24:27.583776+0000","last_became_active":"2024-03-20T09:24:03.378429+0000","last_became_peered":"2024-03-20T09:24:03.378429+0000","last_unstale":"2024-03-20T09:24:27.583776+0000","last_undegraded":"2024-03-20T09:24:27.583776+0000","last_fullsized":"2024-03-20T09:24:27.583776+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":1767,"log_dups_size":0,"ondisk_log_size":1767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:26.886749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115802112,"num_objects":1767,"num_object_clones":0,"num_object_copies":5301,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1767,"num_write_kb":113088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"20'137","reported_seq":915,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.583235+0000","last_change":"2024-03-20T09:23:30.467662+0000","last_active":"2024-03-20T09:24:27.583235+0000","last_peered":"2024-03-20T09:24:27.583235+0000","last_clean":"2024-03-20T09:24:27.583235+0000","last_became_active":"2024-03-20T09:15:34.225022+0000","last_became_peered":"2024-03-20T09:15:34.225022+0000","last_unstale":"2024-03-20T09:24:27.583235+0000","last_undegraded":"2024-03-20T09:24:27.583235+0000","last_fullsized":"2024-03-20T09:24:27.583235+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:23:30.467535+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:20:12.169658+0000","last_clean_scrub_stamp":"2024-03-20T09:23:30.467535+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:24:58.406075+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,5],"acting":[6,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"722'1714","reported_seq":1751,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.681177+0000","last_change":"2024-03-20T09:24:03.379949+0000","last_active":"2024-03-20T09:24:27.681177+0000","last_peered":"2024-03-20T09:24:27.681177+0000","last_clean":"2024-03-20T09:24:27.681177+0000","last_became_active":"2024-03-20T09:24:03.379597+0000","last_became_peered":"2024-03-20T09:24:03.379597+0000","last_unstale":"2024-03-20T09:24:27.681177+0000","last_undegraded":"2024-03-20T09:24:27.681177+0000","last_fullsized":"2024-03-20T09:24:27.681177+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":1714,"log_dups_size":0,"ondisk_log_size":1714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:31.975923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112328704,"num_objects":1714,"num_object_clones":0,"num_object_copies":5142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1714,"num_write_kb":109696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"20'137","reported_seq":1450,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.583534+0000","last_change":"2024-03-20T09:23:25.611676+0000","last_active":"2024-03-20T09:24:27.583534+0000","last_peered":"2024-03-20T09:24:27.583534+0000","last_clean":"2024-03-20T09:24:27.583534+0000","last_became_active":"2024-03-20T09:13:27.087760+0000","last_became_peered":"2024-03-20T09:13:27.087760+0000","last_unstale":"2024-03-20T09:24:27.583534+0000","last_undegraded":"2024-03-20T09:24:27.583534+0000","last_fullsized":"2024-03-20T09:24:27.583534+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:23:25.611535+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:14:10.560890+0000","last_clean_scrub_stamp":"2024-03-20T09:23:25.611535+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:24:53.993207+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"722'1833","reported_seq":1871,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.584024+0000","last_change":"2024-03-20T09:24:03.381105+0000","last_active":"2024-03-20T09:24:27.584024+0000","last_peered":"2024-03-20T09:24:27.584024+0000","last_clean":"2024-03-20T09:24:27.584024+0000","last_became_active":"2024-03-20T09:24:03.380630+0000","last_became_peered":"2024-03-20T09:24:03.380630+0000","last_unstale":"2024-03-20T09:24:27.584024+0000","last_undegraded":"2024-03-20T09:24:27.584024+0000","last_fullsized":"2024-03-20T09:24:27.584024+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":1833,"log_dups_size":0,"ondisk_log_size":1833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:11.318851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120127488,"num_objects":1833,"num_object_clones":0,"num_object_copies":5499,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1833,"num_write_kb":117312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"20'137","reported_seq":1302,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.584084+0000","last_change":"2024-03-20T09:24:10.480839+0000","last_active":"2024-03-20T09:24:27.584084+0000","last_peered":"2024-03-20T09:24:27.584084+0000","last_clean":"2024-03-20T09:24:27.584084+0000","last_became_active":"2024-03-20T09:13:20.070596+0000","last_became_peered":"2024-03-20T09:13:20.070596+0000","last_unstale":"2024-03-20T09:24:27.584084+0000","last_undegraded":"2024-03-20T09:24:27.584084+0000","last_fullsized":"2024-03-20T09:24:27.584084+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:10.480739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:10.480739+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:10.692605+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.9","version":"722'1672","reported_seq":1714,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.681174+0000","last_change":"2024-03-20T09:24:13.382970+0000","last_active":"2024-03-20T09:24:27.681174+0000","last_peered":"2024-03-20T09:24:27.681174+0000","last_clean":"2024-03-20T09:24:27.681174+0000","last_became_active":"2024-03-20T09:24:13.382746+0000","last_became_peered":"2024-03-20T09:24:13.382746+0000","last_unstale":"2024-03-20T09:24:27.681174+0000","last_undegraded":"2024-03-20T09:24:27.681174+0000","last_fullsized":"2024-03-20T09:24:27.681174+0000","mapping_epoch":715,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":716,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":1672,"log_dups_size":0,"ondisk_log_size":1672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:24.455949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109576192,"num_objects":1672,"num_object_clones":0,"num_object_copies":5016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1672,"num_write_kb":107008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"20'137","reported_seq":727,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.380623+0000","last_change":"2024-03-20T09:24:10.821907+0000","last_active":"2024-03-20T09:24:27.380623+0000","last_peered":"2024-03-20T09:24:27.380623+0000","last_clean":"2024-03-20T09:24:27.380623+0000","last_became_active":"2024-03-20T09:19:54.506134+0000","last_became_peered":"2024-03-20T09:19:54.506134+0000","last_unstale":"2024-03-20T09:24:27.380623+0000","last_undegraded":"2024-03-20T09:24:27.380623+0000","last_fullsized":"2024-03-20T09:24:27.380623+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:10.821728+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:18:52.573886+0000","last_clean_scrub_stamp":"2024-03-20T09:24:10.821728+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:23.388764+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"722'1726","reported_seq":1764,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.583989+0000","last_change":"2024-03-20T09:24:03.379403+0000","last_active":"2024-03-20T09:24:27.583989+0000","last_peered":"2024-03-20T09:24:27.583989+0000","last_clean":"2024-03-20T09:24:27.583989+0000","last_became_active":"2024-03-20T09:24:03.379073+0000","last_became_peered":"2024-03-20T09:24:03.379073+0000","last_unstale":"2024-03-20T09:24:27.583989+0000","last_undegraded":"2024-03-20T09:24:27.583989+0000","last_fullsized":"2024-03-20T09:24:27.583989+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":1726,"log_dups_size":0,"ondisk_log_size":1726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:21.840916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113115136,"num_objects":1726,"num_object_clones":0,"num_object_copies":5178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1726,"num_write_kb":110464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,3],"acting":[11,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.f","version":"20'137","reported_seq":1475,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.583452+0000","last_change":"2024-03-20T09:23:28.597118+0000","last_active":"2024-03-20T09:24:27.583452+0000","last_peered":"2024-03-20T09:24:27.583452+0000","last_clean":"2024-03-20T09:24:27.583452+0000","last_became_active":"2024-03-20T09:19:54.524397+0000","last_became_peered":"2024-03-20T09:19:54.524397+0000","last_unstale":"2024-03-20T09:24:27.583452+0000","last_undegraded":"2024-03-20T09:24:27.583452+0000","last_fullsized":"2024-03-20T09:24:27.583452+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:23:28.597000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:23:28.597000+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:24:45.500274+0000","scrub_duration":154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"722'1779","reported_seq":1817,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.583783+0000","last_change":"2024-03-20T09:24:03.379324+0000","last_active":"2024-03-20T09:24:27.583783+0000","last_peered":"2024-03-20T09:24:27.583783+0000","last_clean":"2024-03-20T09:24:27.583783+0000","last_became_active":"2024-03-20T09:24:03.378961+0000","last_became_peered":"2024-03-20T09:24:03.378961+0000","last_unstale":"2024-03-20T09:24:27.583783+0000","last_undegraded":"2024-03-20T09:24:27.583783+0000","last_fullsized":"2024-03-20T09:24:27.583783+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":1779,"log_dups_size":0,"ondisk_log_size":1779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:30.598154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116588544,"num_objects":1779,"num_object_clones":0,"num_object_copies":5337,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1779,"num_write_kb":113856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"570'138","reported_seq":1476,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.584137+0000","last_change":"2024-03-20T09:24:18.167726+0000","last_active":"2024-03-20T09:24:27.584137+0000","last_peered":"2024-03-20T09:24:27.584137+0000","last_clean":"2024-03-20T09:24:27.584137+0000","last_became_active":"2024-03-20T09:13:27.087686+0000","last_became_peered":"2024-03-20T09:13:27.087686+0000","last_unstale":"2024-03-20T09:24:27.584137+0000","last_undegraded":"2024-03-20T09:24:27.584137+0000","last_fullsized":"2024-03-20T09:24:27.584137+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"570'138","last_scrub_stamp":"2024-03-20T09:24:18.167603+0000","last_deep_scrub":"570'138","last_deep_scrub_stamp":"2024-03-20T09:16:13.725035+0000","last_clean_scrub_stamp":"2024-03-20T09:24:18.167603+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:45.806376+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"722'1717","reported_seq":1755,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.583422+0000","last_change":"2024-03-20T09:24:03.381846+0000","last_active":"2024-03-20T09:24:27.583422+0000","last_peered":"2024-03-20T09:24:27.583422+0000","last_clean":"2024-03-20T09:24:27.583422+0000","last_became_active":"2024-03-20T09:24:03.381499+0000","last_became_peered":"2024-03-20T09:24:03.381499+0000","last_unstale":"2024-03-20T09:24:27.583422+0000","last_undegraded":"2024-03-20T09:24:27.583422+0000","last_fullsized":"2024-03-20T09:24:27.583422+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":1717,"log_dups_size":0,"ondisk_log_size":1717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:28.474992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112525312,"num_objects":1717,"num_object_clones":0,"num_object_copies":5151,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1717,"num_write_kb":109888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"20'137","reported_seq":1033,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.380499+0000","last_change":"2024-03-20T09:24:16.958522+0000","last_active":"2024-03-20T09:24:27.380499+0000","last_peered":"2024-03-20T09:24:27.380499+0000","last_clean":"2024-03-20T09:24:27.380499+0000","last_became_active":"2024-03-20T09:19:54.502529+0000","last_became_peered":"2024-03-20T09:19:54.502529+0000","last_unstale":"2024-03-20T09:24:27.380499+0000","last_undegraded":"2024-03-20T09:24:27.380499+0000","last_fullsized":"2024-03-20T09:24:27.380499+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:16.958439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:16.958439+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:31.241580+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"20'137","reported_seq":1337,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.583533+0000","last_change":"2024-03-20T09:24:13.190086+0000","last_active":"2024-03-20T09:24:27.583533+0000","last_peered":"2024-03-20T09:24:27.583533+0000","last_clean":"2024-03-20T09:24:27.583533+0000","last_became_active":"2024-03-20T09:10:34.096758+0000","last_became_peered":"2024-03-20T09:10:34.096758+0000","last_unstale":"2024-03-20T09:24:27.583533+0000","last_undegraded":"2024-03-20T09:24:27.583533+0000","last_fullsized":"2024-03-20T09:24:27.583533+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:13.189949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:13.189949+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:28.168111+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"20'137","reported_seq":1417,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.584125+0000","last_change":"2024-03-20T09:24:15.377223+0000","last_active":"2024-03-20T09:24:27.584125+0000","last_peered":"2024-03-20T09:24:27.584125+0000","last_clean":"2024-03-20T09:24:27.584125+0000","last_became_active":"2024-03-20T09:04:04.103645+0000","last_became_peered":"2024-03-20T09:04:04.103645+0000","last_unstale":"2024-03-20T09:24:27.584125+0000","last_undegraded":"2024-03-20T09:24:27.584125+0000","last_fullsized":"2024-03-20T09:24:27.584125+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:15.377106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:15.377106+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:23.936408+0000","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"20'137","reported_seq":871,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.583282+0000","last_change":"2024-03-20T09:24:17.124668+0000","last_active":"2024-03-20T09:24:27.583282+0000","last_peered":"2024-03-20T09:24:27.583282+0000","last_clean":"2024-03-20T09:24:27.583282+0000","last_became_active":"2024-03-20T09:10:34.097191+0000","last_became_peered":"2024-03-20T09:10:34.097191+0000","last_unstale":"2024-03-20T09:24:27.583282+0000","last_undegraded":"2024-03-20T09:24:27.583282+0000","last_fullsized":"2024-03-20T09:24:27.583282+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:17.124502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:17.124502+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:26.158088+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"20'137","reported_seq":1281,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.681334+0000","last_change":"2024-03-20T09:24:08.964670+0000","last_active":"2024-03-20T09:24:27.681334+0000","last_peered":"2024-03-20T09:24:27.681334+0000","last_clean":"2024-03-20T09:24:27.681334+0000","last_became_active":"2024-03-20T09:20:25.342982+0000","last_became_peered":"2024-03-20T09:20:25.342982+0000","last_unstale":"2024-03-20T09:24:27.681334+0000","last_undegraded":"2024-03-20T09:24:27.681334+0000","last_fullsized":"2024-03-20T09:24:27.681334+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:08.964577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:08.964577+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:09.518528+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"20'137","reported_seq":1106,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.583972+0000","last_change":"2024-03-20T09:24:19.259161+0000","last_active":"2024-03-20T09:24:27.583972+0000","last_peered":"2024-03-20T09:24:27.583972+0000","last_clean":"2024-03-20T09:24:27.583972+0000","last_became_active":"2024-03-20T09:13:27.087760+0000","last_became_peered":"2024-03-20T09:13:27.087760+0000","last_unstale":"2024-03-20T09:24:27.583972+0000","last_undegraded":"2024-03-20T09:24:27.583972+0000","last_fullsized":"2024-03-20T09:24:27.583972+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:19.259031+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:16:15.448920+0000","last_clean_scrub_stamp":"2024-03-20T09:24:19.259031+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:22.996346+0000","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"20'137","reported_seq":1432,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.584469+0000","last_change":"2024-03-20T09:23:34.995066+0000","last_active":"2024-03-20T09:24:27.584469+0000","last_peered":"2024-03-20T09:24:27.584469+0000","last_clean":"2024-03-20T09:24:27.584469+0000","last_became_active":"2024-03-20T09:10:35.095339+0000","last_became_peered":"2024-03-20T09:10:35.095339+0000","last_unstale":"2024-03-20T09:24:27.584469+0000","last_undegraded":"2024-03-20T09:24:27.584469+0000","last_fullsized":"2024-03-20T09:24:27.584469+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:23:34.994901+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:23.640533+0000","last_clean_scrub_stamp":"2024-03-20T09:23:34.994901+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:24:47.732179+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"20'137","reported_seq":1509,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.380787+0000","last_change":"2024-03-20T09:23:14.976861+0000","last_active":"2024-03-20T09:24:27.380787+0000","last_peered":"2024-03-20T09:24:27.380787+0000","last_clean":"2024-03-20T09:24:27.380787+0000","last_became_active":"2024-03-20T09:20:25.349082+0000","last_became_peered":"2024-03-20T09:20:25.349082+0000","last_unstale":"2024-03-20T09:24:27.380787+0000","last_undegraded":"2024-03-20T09:24:27.380787+0000","last_fullsized":"2024-03-20T09:24:27.380787+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:23:14.976736+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:16:15.754362+0000","last_clean_scrub_stamp":"2024-03-20T09:23:14.976736+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"20'137","reported_seq":766,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.442203+0000","last_change":"2024-03-20T09:23:41.251626+0000","last_active":"2024-03-20T09:24:27.442203+0000","last_peered":"2024-03-20T09:24:27.442203+0000","last_clean":"2024-03-20T09:24:27.442203+0000","last_became_active":"2024-03-20T09:12:14.307581+0000","last_became_peered":"2024-03-20T09:12:14.307581+0000","last_unstale":"2024-03-20T09:24:27.442203+0000","last_undegraded":"2024-03-20T09:24:27.442203+0000","last_fullsized":"2024-03-20T09:24:27.442203+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:23:41.251546+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:23:41.251546+0000","last_clean_scrub_stamp":"2024-03-20T09:23:41.251546+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:24:58.292060+0000","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"20'137","reported_seq":1032,"reported_epoch":723,"state":"active+clean","last_fresh":"2024-03-20T09:24:27.380583+0000","last_change":"2024-03-20T09:24:15.799821+0000","last_active":"2024-03-20T09:24:27.380583+0000","last_peered":"2024-03-20T09:24:27.380583+0000","last_clean":"2024-03-20T09:24:27.380583+0000","last_became_active":"2024-03-20T09:19:54.502391+0000","last_became_peered":"2024-03-20T09:19:54.502391+0000","last_unstale":"2024-03-20T09:24:27.380583+0000","last_undegraded":"2024-03-20T09:24:27.380583+0000","last_fullsized":"2024-03-20T09:24:27.380583+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:15.799637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:15.799637+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:30.958859+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":1828192256,"num_objects":27896,"num_object_clones":0,"num_object_copies":83688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27896,"num_write_kb":1785344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5483528192,"data_stored":5483528192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27896,"ondisk_log_size":27896,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4248,"ondisk_log_size":4248,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860758,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":869828,"kb_used_data":220856,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":92875324,"statfs":{"total":95995035648,"available":95104331776,"internally_reserved":0,"allocated":226156544,"data_stored":225609381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":664533444},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":89,"seq":382252089692,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":802564,"kb_used_data":342392,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":92942588,"statfs":{"total":95995035648,"available":95173210112,"internally_reserved":0,"allocated":350609408,"data_stored":350062245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":471202240},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":80,"seq":343597384035,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":612660,"kb_used_data":450988,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93132492,"statfs":{"total":95995035648,"available":95367671808,"internally_reserved":0,"allocated":461811712,"data_stored":461269064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":165542336},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":70,"seq":300647711079,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":579272,"kb_used_data":223224,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93165880,"statfs":{"total":95995035648,"available":95401861120,"internally_reserved":0,"allocated":228581376,"data_stored":228034213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":364575174},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":60,"seq":257698038124,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1200460,"kb_used_data":559808,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":92544692,"statfs":{"total":95995035648,"available":94765764608,"internally_reserved":0,"allocated":573243392,"data_stored":572692677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":656013770},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332465,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":991432,"kb_used_data":450744,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":92753720,"statfs":{"total":95995035648,"available":94979809280,"internally_reserved":0,"allocated":461561856,"data_stored":461014693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":553646534},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,16,9],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575243,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1380932,"kb_used_data":754168,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":92364220,"statfs":{"total":95995035648,"available":94580961280,"internally_reserved":0,"allocated":772268032,"data_stored":771720869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":641792454},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738761,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1306568,"kb_used_data":653432,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":92438584,"statfs":{"total":95995035648,"available":94657110016,"internally_reserved":0,"allocated":669114368,"data_stored":668567205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411716,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1303564,"kb_used_data":889216,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92441588,"statfs":{"total":95995035648,"available":94660186112,"internally_reserved":0,"allocated":910557184,"data_stored":910006469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150080,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1393668,"kb_used_data":897400,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":92351484,"statfs":{"total":95995035648,"available":94567919616,"internally_reserved":0,"allocated":918937600,"data_stored":918390437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":508164551},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953854,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":917380,"kb_used_data":448888,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":92827772,"statfs":{"total":95995035648,"available":95055638528,"internally_reserved":0,"allocated":459661312,"data_stored":459114149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":443,"seq":1902670512296,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1393920,"kb_used_data":984948,"kb_used_omap":181,"kb_used_meta":408778,"kb_avail":92351232,"statfs":{"total":95995035648,"available":94567661568,"internally_reserved":0,"allocated":1008586752,"data_stored":1008040552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":186131,"internal_metadata":418588909},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":334430208,"data_stored":334430208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":452853760,"data_stored":452853760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":455999488,"data_stored":455999488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":917110784,"data_stored":917110784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":458162176,"data_stored":458162176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":572522496,"data_stored":572522496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":459735040,"data_stored":459735040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":571539456,"data_stored":571539456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":227082240,"data_stored":227082240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":460324864,"data_stored":460324864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":349110272,"data_stored":349110272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":224657408,"data_stored":224657408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:24:34.982 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-20T09:24:34.982 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:24:34.982 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-20T09:24:34.982 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-20T09:24:35.019 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:24:35.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:35.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:24:35.122 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:24:35.150 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:35.276 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:24:35.277 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:35 smithi115 ceph-mon[72476]: pgmap v1762: 47 pgs: 47 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 800 op/s 2024-03-20T09:24:35.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:35.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:24:35.379 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:24:35.418 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:35.568 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:24:35.570 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:35 smithi026 ceph-mon[68309]: pgmap v1762: 47 pgs: 47 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 800 op/s 2024-03-20T09:24:35.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:35.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:24:35.672 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:24:35.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:35 smithi122 ceph-mon[73495]: pgmap v1762: 47 pgs: 47 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 800 op/s 2024-03-20T09:24:35.700 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:35.825 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:24:35.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:35.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:24:35.858 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:2024-03-20T09:24:35.860846+0000 min lat: 0.00104258 max lat: 0.607386 avg lat: 0.00598077 2024-03-20T09:24:35.858 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T09:24:35.858 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 20 16 27896 27880 87.1107 0 - 0.00598077 2024-03-20T09:24:35.929 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:24:35.959 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:36.034 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:24:36.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:36.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:24:36.065 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:24:36.139 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:24:36.173 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:36.179 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:24:36.179 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-03-20T09:24:36.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-03-20T09:24:36.179 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:24:36.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.179 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:132717 5.5 5:a85622dd:::benchmark_data_smithi122_157894_object2730:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e640)", 2024-03-20T09:24:36.179 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:19:57.993260+0000", 2024-03-20T09:24:36.180 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 278.16804210999999, 2024-03-20T09:24:36.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3269052480000001, 2024-03-20T09:24:36.180 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:24:36.180 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:24:36.180 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:24:36.180 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:24:36.180 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:24:36.180 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 132717 2024-03-20T09:24:36.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.180 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:24:36.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:24:36.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:57.993260+0000", 2024-03-20T09:24:36.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:24:36.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:57.993260+0000", 2024-03-20T09:24:36.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:24:36.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:57.993260+0000", 2024-03-20T09:24:36.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8500000000000002e-07 2024-03-20T09:24:36.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:24:36.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:57.993266+0000", 2024-03-20T09:24:36.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.93e-06 2024-03-20T09:24:36.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:24:36.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:57.993273+0000", 2024-03-20T09:24:36.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2579999999999998e-06 2024-03-20T09:24:36.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:24:36.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:57.993278+0000", 2024-03-20T09:24:36.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2499999999999997e-06 2024-03-20T09:24:36.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:57.999577+0000", 2024-03-20T09:24:36.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0062984790000000001 2024-03-20T09:24:36.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:24:36.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:57.999634+0000", 2024-03-20T09:24:36.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7414999999999998e-05 2024-03-20T09:24:36.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:00.320150+0000", 2024-03-20T09:24:36.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3205160149999999 2024-03-20T09:24:36.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.184 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:24:36.184 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:00.320165+0000", 2024-03-20T09:24:36.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4516e-05 2024-03-20T09:24:36.184 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.184 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:36.184 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.184 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:132729 5.5 5:a5fcbedb:::benchmark_data_smithi122_157894_object2742:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e640)", 2024-03-20T09:24:36.184 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:19:57.993290+0000", 2024-03-20T09:24:36.184 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 278.168011949, 2024-03-20T09:24:36.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3448420080000001, 2024-03-20T09:24:36.185 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:24:36.185 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:24:36.185 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:24:36.185 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:24:36.185 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:24:36.185 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 132729 2024-03-20T09:24:36.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.185 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:24:36.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:24:36.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:57.993290+0000", 2024-03-20T09:24:36.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:24:36.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:57.993290+0000", 2024-03-20T09:24:36.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:24:36.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:57.993290+0000", 2024-03-20T09:24:36.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4200000000000001e-07 2024-03-20T09:24:36.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:24:36.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:57.993292+0000", 2024-03-20T09:24:36.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.193e-06 2024-03-20T09:24:36.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:24:36.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:57.993300+0000", 2024-03-20T09:24:36.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.165e-06 2024-03-20T09:24:36.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:24:36.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:57.993305+0000", 2024-03-20T09:24:36.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5849999999999999e-06 2024-03-20T09:24:36.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:57.999662+0000", 2024-03-20T09:24:36.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0063571749999999996 2024-03-20T09:24:36.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:24:36.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:57.999726+0000", 2024-03-20T09:24:36.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3644000000000003e-05 2024-03-20T09:24:36.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:00.338117+0000", 2024-03-20T09:24:36.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3383908249999998 2024-03-20T09:24:36.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:24:36.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:00.338132+0000", 2024-03-20T09:24:36.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4979e-05 2024-03-20T09:24:36.190 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.190 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:36.190 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.190 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:134917 5.5 5:af4c6f6b:::benchmark_data_smithi122_157894_object4930:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e640)", 2024-03-20T09:24:36.190 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:20:01.472901+0000", 2024-03-20T09:24:36.190 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 274.68840053600002, 2024-03-20T09:24:36.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.608398652, 2024-03-20T09:24:36.191 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:24:36.191 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:24:36.191 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:24:36.191 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:24:36.191 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:24:36.191 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 134917 2024-03-20T09:24:36.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.191 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:24:36.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:24:36.191 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:01.472901+0000", 2024-03-20T09:24:36.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:24:36.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:01.472901+0000", 2024-03-20T09:24:36.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:24:36.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:01.472902+0000", 2024-03-20T09:24:36.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4400000000000001e-07 2024-03-20T09:24:36.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:24:36.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:01.472903+0000", 2024-03-20T09:24:36.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6589999999999999e-06 2024-03-20T09:24:36.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:24:36.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:01.472909+0000", 2024-03-20T09:24:36.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9159999999999998e-06 2024-03-20T09:24:36.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:24:36.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:01.472914+0000", 2024-03-20T09:24:36.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3680000000000004e-06 2024-03-20T09:24:36.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:01.472970+0000", 2024-03-20T09:24:36.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6706999999999998e-05 2024-03-20T09:24:36.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:24:36.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:01.473027+0000", 2024-03-20T09:24:36.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6439e-05 2024-03-20T09:24:36.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:04.081293+0000", 2024-03-20T09:24:36.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6082658990000001 2024-03-20T09:24:36.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:24:36.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:04.081300+0000", 2024-03-20T09:24:36.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3200000000000002e-06 2024-03-20T09:24:36.195 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.195 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:36.195 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.195 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:136502 5.5 5:aca98ce8:::benchmark_data_smithi122_157894_object6515:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e640)", 2024-03-20T09:24:36.195 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:20:04.032052+0000", 2024-03-20T09:24:36.195 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 272.12924970199998, 2024-03-20T09:24:36.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7508287569999998, 2024-03-20T09:24:36.195 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:24:36.195 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:24:36.195 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:24:36.195 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:24:36.196 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:24:36.196 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 136502 2024-03-20T09:24:36.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.196 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:24:36.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.196 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:24:36.196 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:04.032052+0000", 2024-03-20T09:24:36.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.196 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:24:36.196 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:04.032052+0000", 2024-03-20T09:24:36.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:24:36.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:04.032052+0000", 2024-03-20T09:24:36.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2899999999999999e-07 2024-03-20T09:24:36.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:24:36.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:04.032054+0000", 2024-03-20T09:24:36.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.866e-06 2024-03-20T09:24:36.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:24:36.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:04.032060+0000", 2024-03-20T09:24:36.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2340000000000002e-06 2024-03-20T09:24:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:24:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:04.032064+0000", 2024-03-20T09:24:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6310000000000002e-06 2024-03-20T09:24:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:04.032200+0000", 2024-03-20T09:24:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000135432 2024-03-20T09:24:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:24:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:04.032271+0000", 2024-03-20T09:24:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1408999999999993e-05 2024-03-20T09:24:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.782870+0000", 2024-03-20T09:24:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7505991829999998 2024-03-20T09:24:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:24:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.782881+0000", 2024-03-20T09:24:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0573e-05 2024-03-20T09:24:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.199 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:36.200 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.200 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:138116 5.5 5:a2509137:::benchmark_data_smithi122_157894_object8129:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e640)", 2024-03-20T09:24:36.200 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:20:06.707844+0000", 2024-03-20T09:24:36.200 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 269.45345823000002, 2024-03-20T09:24:36.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8833028590000005, 2024-03-20T09:24:36.200 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:24:36.200 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:24:36.200 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:24:36.200 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:24:36.200 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:24:36.200 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 138116 2024-03-20T09:24:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:24:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:24:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.707844+0000", 2024-03-20T09:24:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:24:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.707844+0000", 2024-03-20T09:24:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:24:36.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.707844+0000", 2024-03-20T09:24:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9400000000000001e-07 2024-03-20T09:24:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:24:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.707846+0000", 2024-03-20T09:24:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2199999999999999e-06 2024-03-20T09:24:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:24:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.707854+0000", 2024-03-20T09:24:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8450000000000003e-06 2024-03-20T09:24:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:24:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.707859+0000", 2024-03-20T09:24:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3669999999999996e-06 2024-03-20T09:24:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.708051+0000", 2024-03-20T09:24:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000192098 2024-03-20T09:24:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:24:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.708133+0000", 2024-03-20T09:24:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1102999999999999e-05 2024-03-20T09:24:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:12.591128+0000", 2024-03-20T09:24:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8829951319999996 2024-03-20T09:24:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:24:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:12.591146+0000", 2024-03-20T09:24:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8700000000000001e-05 2024-03-20T09:24:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.204 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:138118 5.5 5:a21f0c93:::benchmark_data_smithi122_157894_object8131:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e640)", 2024-03-20T09:24:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:20:06.707869+0000", 2024-03-20T09:24:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 269.453433045, 2024-03-20T09:24:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8948337950000003, 2024-03-20T09:24:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:24:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:24:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:24:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:24:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:24:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 138118 2024-03-20T09:24:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:24:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:24:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.707869+0000", 2024-03-20T09:24:36.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:24:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.707869+0000", 2024-03-20T09:24:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:24:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.707869+0000", 2024-03-20T09:24:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5000000000000003e-07 2024-03-20T09:24:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:24:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.707871+0000", 2024-03-20T09:24:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1900000000000002e-06 2024-03-20T09:24:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:24:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.707886+0000", 2024-03-20T09:24:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4849e-05 2024-03-20T09:24:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:24:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.707894+0000", 2024-03-20T09:24:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8509999999999999e-06 2024-03-20T09:24:36.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.708173+0000", 2024-03-20T09:24:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000278566 2024-03-20T09:24:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:24:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.708224+0000", 2024-03-20T09:24:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1570999999999998e-05 2024-03-20T09:24:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:12.602687+0000", 2024-03-20T09:24:36.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944628400000001 2024-03-20T09:24:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:24:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:12.602703+0000", 2024-03-20T09:24:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5378000000000001e-05 2024-03-20T09:24:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:138125 5.5 5:acdf3955:::benchmark_data_smithi122_157894_object8138:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e640)", 2024-03-20T09:24:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:20:06.707929+0000", 2024-03-20T09:24:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 269.45337260600002, 2024-03-20T09:24:36.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9079474379999999, 2024-03-20T09:24:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:24:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:24:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:24:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:24:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:24:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 138125 2024-03-20T09:24:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:24:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:24:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.707929+0000", 2024-03-20T09:24:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:24:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.707929+0000", 2024-03-20T09:24:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:24:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.707930+0000", 2024-03-20T09:24:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5400000000000002e-07 2024-03-20T09:24:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:24:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.707932+0000", 2024-03-20T09:24:36.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1859999999999999e-06 2024-03-20T09:24:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:24:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.707939+0000", 2024-03-20T09:24:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1119999999999998e-06 2024-03-20T09:24:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:24:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.707944+0000", 2024-03-20T09:24:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8609999999999997e-06 2024-03-20T09:24:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.708250+0000", 2024-03-20T09:24:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00030575399999999999 2024-03-20T09:24:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:24:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.708342+0000", 2024-03-20T09:24:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2390000000000001e-05 2024-03-20T09:24:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:12.615860+0000", 2024-03-20T09:24:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9075176769999995 2024-03-20T09:24:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:24:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:12.615877+0000", 2024-03-20T09:24:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7003999999999998e-05 2024-03-20T09:24:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:177371 5.5 5:a1b6b191:::benchmark_data_smithi122_157894_object47384:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e663)", 2024-03-20T09:24:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:21:28.979500+0000", 2024-03-20T09:24:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 187.181801291, 2024-03-20T09:24:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8943260209999999, 2024-03-20T09:24:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:24:36.214 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:24:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:24:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:24:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:24:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 177371 2024-03-20T09:24:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:24:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:24:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:28.979500+0000", 2024-03-20T09:24:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:24:36.215 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:28.979500+0000", 2024-03-20T09:24:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:24:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:28.979501+0000", 2024-03-20T09:24:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2000000000000001e-07 2024-03-20T09:24:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:24:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:28.979503+0000", 2024-03-20T09:24:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8950000000000001e-06 2024-03-20T09:24:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.216 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:24:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:28.979508+0000", 2024-03-20T09:24:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5729999999999998e-06 2024-03-20T09:24:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:24:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:28.979513+0000", 2024-03-20T09:24:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509999999999997e-06 2024-03-20T09:24:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:28.995185+0000", 2024-03-20T09:24:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015672063999999999 2024-03-20T09:24:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.217 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:24:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:28.995248+0000", 2024-03-20T09:24:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3153999999999999e-05 2024-03-20T09:24:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:30.873811+0000", 2024-03-20T09:24:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8785628060000001 2024-03-20T09:24:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:24:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:30.873827+0000", 2024-03-20T09:24:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5258e-05 2024-03-20T09:24:36.218 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:191814 5.5 5:a9b30849:::benchmark_data_smithi122_157894_object61827:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e668)", 2024-03-20T09:24:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:21:54.975493+0000", 2024-03-20T09:24:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 161.18580830299999, 2024-03-20T09:24:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3338137570000002, 2024-03-20T09:24:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:24:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:24:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:24:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:24:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:24:36.219 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 191814 2024-03-20T09:24:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:24:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:24:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:54.975493+0000", 2024-03-20T09:24:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:24:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:54.975493+0000", 2024-03-20T09:24:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.220 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:24:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:54.975494+0000", 2024-03-20T09:24:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2899999999999999e-07 2024-03-20T09:24:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:24:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:54.975496+0000", 2024-03-20T09:24:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.12e-06 2024-03-20T09:24:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:24:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:54.975501+0000", 2024-03-20T09:24:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3859999999999998e-06 2024-03-20T09:24:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.221 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:24:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:54.975506+0000", 2024-03-20T09:24:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5290000000000002e-06 2024-03-20T09:24:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:54.979081+0000", 2024-03-20T09:24:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035750059999999999 2024-03-20T09:24:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:24:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:54.979159+0000", 2024-03-20T09:24:36.222 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8078999999999996e-05 2024-03-20T09:24:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.309291+0000", 2024-03-20T09:24:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.330132366 2024-03-20T09:24:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:24:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.309307+0000", 2024-03-20T09:24:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5942000000000001e-05 2024-03-20T09:24:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.223 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:192991 5.5 5:a2123005:::benchmark_data_smithi122_157894_object63004:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e668)", 2024-03-20T09:24:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:21:57.197478+0000", 2024-03-20T09:24:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 158.963823985, 2024-03-20T09:24:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1310714370000001, 2024-03-20T09:24:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:24:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:24:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:24:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:24:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:24:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 192991 2024-03-20T09:24:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:24:36.224 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:24:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197478+0000", 2024-03-20T09:24:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:24:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197478+0000", 2024-03-20T09:24:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:24:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197478+0000", 2024-03-20T09:24:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7300000000000002e-07 2024-03-20T09:24:36.225 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:24:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197480+0000", 2024-03-20T09:24:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.277e-06 2024-03-20T09:24:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:24:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197487+0000", 2024-03-20T09:24:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9160000000000004e-06 2024-03-20T09:24:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:24:36.226 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197492+0000", 2024-03-20T09:24:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6739999999999996e-06 2024-03-20T09:24:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.204290+0000", 2024-03-20T09:24:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0067975529999999996 2024-03-20T09:24:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:24:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.204348+0000", 2024-03-20T09:24:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8020999999999999e-05 2024-03-20T09:24:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.227 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:01.328540+0000", 2024-03-20T09:24:36.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1241922119999996 2024-03-20T09:24:36.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.228 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:24:36.228 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:01.328549+0000", 2024-03-20T09:24:36.228 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4110000000000002e-06 2024-03-20T09:24:36.228 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.228 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:36.228 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.228 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.228 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.228 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:192996 5.5 5:a7785dc6:::benchmark_data_smithi122_157894_object63009:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e668)", 2024-03-20T09:24:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:21:57.197499+0000", 2024-03-20T09:24:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 158.96380318600001, 2024-03-20T09:24:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1424198329999999, 2024-03-20T09:24:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:24:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:24:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:24:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:24:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:24:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 192996 2024-03-20T09:24:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:24:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:24:36.229 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197499+0000", 2024-03-20T09:24:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:24:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197499+0000", 2024-03-20T09:24:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:24:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197499+0000", 2024-03-20T09:24:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3000000000000001e-07 2024-03-20T09:24:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.230 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:24:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197501+0000", 2024-03-20T09:24:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2910000000000002e-06 2024-03-20T09:24:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:24:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197507+0000", 2024-03-20T09:24:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8300000000000001e-06 2024-03-20T09:24:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:24:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197512+0000", 2024-03-20T09:24:36.231 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8250000000000004e-06 2024-03-20T09:24:36.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.204365+0000", 2024-03-20T09:24:36.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0068533769999999999 2024-03-20T09:24:36.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:24:36.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.204408+0000", 2024-03-20T09:24:36.232 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3028999999999997e-05 2024-03-20T09:24:36.232 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.232 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.232 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.232 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:01.339901+0000", 2024-03-20T09:24:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1354924190000002 2024-03-20T09:24:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:24:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:01.339918+0000", 2024-03-20T09:24:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7632e-05 2024-03-20T09:24:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:193014 5.5 5:ac49b0df:::benchmark_data_smithi122_157894_object63027:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e668)", 2024-03-20T09:24:36.233 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:21:57.197518+0000", 2024-03-20T09:24:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 158.96378408000001, 2024-03-20T09:24:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1533976619999997, 2024-03-20T09:24:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:24:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:24:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:24:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:24:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:24:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 193014 2024-03-20T09:24:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:24:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.234 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:24:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197518+0000", 2024-03-20T09:24:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:24:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197518+0000", 2024-03-20T09:24:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:24:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197518+0000", 2024-03-20T09:24:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6399999999999998e-07 2024-03-20T09:24:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.235 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:24:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197520+0000", 2024-03-20T09:24:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0839999999999999e-06 2024-03-20T09:24:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:24:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197526+0000", 2024-03-20T09:24:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4129999999999997e-06 2024-03-20T09:24:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:24:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197530+0000", 2024-03-20T09:24:36.236 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5600000000000004e-06 2024-03-20T09:24:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.204429+0000", 2024-03-20T09:24:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006899105 2024-03-20T09:24:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:24:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.204486+0000", 2024-03-20T09:24:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7281000000000003e-05 2024-03-20T09:24:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.237 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:01.350900+0000", 2024-03-20T09:24:36.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1464130609999996 2024-03-20T09:24:36.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.238 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:24:36.238 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:01.350915+0000", 2024-03-20T09:24:36.238 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5794000000000002e-05 2024-03-20T09:24:36.238 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.238 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:36.238 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.238 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.238 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.238 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:193025 5.5 5:a75663e7:::benchmark_data_smithi122_157894_object63038:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e668)", 2024-03-20T09:24:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:21:57.197554+0000", 2024-03-20T09:24:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 158.96374816599999, 2024-03-20T09:24:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1646879749999997, 2024-03-20T09:24:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:24:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:24:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:24:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:24:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:24:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 193025 2024-03-20T09:24:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:24:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:24:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197554+0000", 2024-03-20T09:24:36.239 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:24:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197554+0000", 2024-03-20T09:24:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:24:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197554+0000", 2024-03-20T09:24:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1199999999999999e-07 2024-03-20T09:24:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.240 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:24:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197557+0000", 2024-03-20T09:24:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.898e-06 2024-03-20T09:24:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:24:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197562+0000", 2024-03-20T09:24:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0440000000000003e-06 2024-03-20T09:24:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:24:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197566+0000", 2024-03-20T09:24:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5369999999999999e-06 2024-03-20T09:24:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.241 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.204505+0000", 2024-03-20T09:24:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0069383500000000002 2024-03-20T09:24:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:24:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.204546+0000", 2024-03-20T09:24:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699000000000002e-05 2024-03-20T09:24:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:01.362230+0000", 2024-03-20T09:24:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1576838379999996 2024-03-20T09:24:36.242 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:24:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:01.362242+0000", 2024-03-20T09:24:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1297e-05 2024-03-20T09:24:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:193028 5.5 5:a0a14a7f:::benchmark_data_smithi122_157894_object63041:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e668)", 2024-03-20T09:24:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:21:57.197572+0000", 2024-03-20T09:24:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 158.96372964299999, 2024-03-20T09:24:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1764598690000003, 2024-03-20T09:24:36.243 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:24:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:24:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:24:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:24:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:24:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 193028 2024-03-20T09:24:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:24:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:24:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197572+0000", 2024-03-20T09:24:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:24:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197572+0000", 2024-03-20T09:24:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:24:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197573+0000", 2024-03-20T09:24:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9699999999999996e-07 2024-03-20T09:24:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:24:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197575+0000", 2024-03-20T09:24:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9089999999999998e-06 2024-03-20T09:24:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:24:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197581+0000", 2024-03-20T09:24:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2110000000000003e-06 2024-03-20T09:24:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:24:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197586+0000", 2024-03-20T09:24:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1120000000000004e-06 2024-03-20T09:24:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.204566+0000", 2024-03-20T09:24:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0069799229999999999 2024-03-20T09:24:36.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:24:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.204609+0000", 2024-03-20T09:24:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2979000000000003e-05 2024-03-20T09:24:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:01.374020+0000", 2024-03-20T09:24:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1694113149999996 2024-03-20T09:24:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:24:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:01.374032+0000", 2024-03-20T09:24:36.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1922999999999999e-05 2024-03-20T09:24:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:195329 5.5 5:afea5365:::benchmark_data_smithi122_157894_object65342:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e669)", 2024-03-20T09:24:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:22:01.285838+0000", 2024-03-20T09:24:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 154.87546393599999, 2024-03-20T09:24:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0307289530000001, 2024-03-20T09:24:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:24:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:24:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:24:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:24:36.248 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:24:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 195329 2024-03-20T09:24:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:24:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:24:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:01.285838+0000", 2024-03-20T09:24:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:24:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:01.285838+0000", 2024-03-20T09:24:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:24:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:01.285838+0000", 2024-03-20T09:24:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8999999999999997e-07 2024-03-20T09:24:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:24:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:01.285841+0000", 2024-03-20T09:24:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5519999999999999e-06 2024-03-20T09:24:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:24:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:01.285849+0000", 2024-03-20T09:24:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0600000000000008e-06 2024-03-20T09:24:36.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:24:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:01.285855+0000", 2024-03-20T09:24:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0209999999999996e-06 2024-03-20T09:24:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:01.289987+0000", 2024-03-20T09:24:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0041315759999999997 2024-03-20T09:24:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:24:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:01.290045+0000", 2024-03-20T09:24:36.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8044000000000002e-05 2024-03-20T09:24:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:03.316558+0000", 2024-03-20T09:24:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0265130240000002 2024-03-20T09:24:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:24:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:03.316567+0000", 2024-03-20T09:24:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1859999999999998e-06 2024-03-20T09:24:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:197729 5.5 5:aa48ba13:::benchmark_data_smithi122_157894_object67742:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e672)", 2024-03-20T09:24:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:22:05.320420+0000", 2024-03-20T09:24:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 150.84088156799999, 2024-03-20T09:24:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8930668800000001, 2024-03-20T09:24:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:24:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:24:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:24:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:24:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:24:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 197729 2024-03-20T09:24:36.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:24:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:24:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:05.320420+0000", 2024-03-20T09:24:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:24:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:05.320420+0000", 2024-03-20T09:24:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:24:36.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:05.320421+0000", 2024-03-20T09:24:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4600000000000004e-07 2024-03-20T09:24:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:24:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:05.320425+0000", 2024-03-20T09:24:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9879999999999998e-06 2024-03-20T09:24:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:24:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:05.320433+0000", 2024-03-20T09:24:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7060000000000004e-06 2024-03-20T09:24:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:24:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:05.320439+0000", 2024-03-20T09:24:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8369999999999999e-06 2024-03-20T09:24:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:05.323263+0000", 2024-03-20T09:24:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028235460000000001 2024-03-20T09:24:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:24:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:05.323349+0000", 2024-03-20T09:24:36.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6416999999999996e-05 2024-03-20T09:24:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.213468+0000", 2024-03-20T09:24:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8901190510000001 2024-03-20T09:24:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:24:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.213487+0000", 2024-03-20T09:24:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8589000000000001e-05 2024-03-20T09:24:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:36.257 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:197732 5.5 5:a103aab1:::benchmark_data_smithi122_157894_object67745:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e672)", 2024-03-20T09:24:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:22:05.320454+0000", 2024-03-20T09:24:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 150.84084797700001, 2024-03-20T09:24:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9028468630000002, 2024-03-20T09:24:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:24:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:24:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:24:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:24:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:24:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 197732 2024-03-20T09:24:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.258 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:24:36.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:24:36.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:05.320454+0000", 2024-03-20T09:24:36.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:24:36.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:05.320454+0000", 2024-03-20T09:24:36.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:24:36.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:05.320454+0000", 2024-03-20T09:24:36.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2899999999999999e-07 2024-03-20T09:24:36.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:24:36.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:05.320457+0000", 2024-03-20T09:24:36.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3209999999999999e-06 2024-03-20T09:24:36.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:24:36.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:05.320462+0000", 2024-03-20T09:24:36.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7320000000000003e-06 2024-03-20T09:24:36.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:24:36.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:05.320467+0000", 2024-03-20T09:24:36.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6670000000000004e-06 2024-03-20T09:24:36.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:05.323380+0000", 2024-03-20T09:24:36.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0029125919999999999 2024-03-20T09:24:36.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:24:36.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:05.323448+0000", 2024-03-20T09:24:36.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8388999999999999e-05 2024-03-20T09:24:36.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.223285+0000", 2024-03-20T09:24:36.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8998367869999999 2024-03-20T09:24:36.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:24:36.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.223301+0000", 2024-03-20T09:24:36.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5945999999999999e-05 2024-03-20T09:24:36.262 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.262 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:36.262 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.262 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:199405 5.5 5:a1324d3e:::benchmark_data_smithi122_157894_object69418:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e673)", 2024-03-20T09:24:36.262 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:22:08.168607+0000", 2024-03-20T09:24:36.262 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 147.992694314, 2024-03-20T09:24:36.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.352534253, 2024-03-20T09:24:36.263 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:24:36.263 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:24:36.263 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:24:36.263 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:24:36.263 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:24:36.263 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 199405 2024-03-20T09:24:36.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.263 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:24:36.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:24:36.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.168607+0000", 2024-03-20T09:24:36.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:24:36.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.168607+0000", 2024-03-20T09:24:36.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:24:36.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.168608+0000", 2024-03-20T09:24:36.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1299999999999999e-07 2024-03-20T09:24:36.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:24:36.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.168609+0000", 2024-03-20T09:24:36.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.483e-06 2024-03-20T09:24:36.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:24:36.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.168612+0000", 2024-03-20T09:24:36.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2940000000000001e-06 2024-03-20T09:24:36.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:24:36.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.168616+0000", 2024-03-20T09:24:36.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.269e-06 2024-03-20T09:24:36.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.178863+0000", 2024-03-20T09:24:36.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010247439000000001 2024-03-20T09:24:36.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:24:36.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.178937+0000", 2024-03-20T09:24:36.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3949999999999995e-05 2024-03-20T09:24:36.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:10.521129+0000", 2024-03-20T09:24:36.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3421915129999999 2024-03-20T09:24:36.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:24:36.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:10.521142+0000", 2024-03-20T09:24:36.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3091999999999999e-05 2024-03-20T09:24:36.267 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.267 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:36.267 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.267 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:199411 5.5 5:a791d230:::benchmark_data_smithi122_157894_object69424:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e673)", 2024-03-20T09:24:36.267 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:22:08.168619+0000", 2024-03-20T09:24:36.267 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 147.992682839, 2024-03-20T09:24:36.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3647598240000001, 2024-03-20T09:24:36.267 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:24:36.267 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:24:36.267 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:24:36.268 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:24:36.268 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:24:36.268 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 199411 2024-03-20T09:24:36.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.268 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:24:36.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:24:36.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.168619+0000", 2024-03-20T09:24:36.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:24:36.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.168619+0000", 2024-03-20T09:24:36.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:24:36.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.168619+0000", 2024-03-20T09:24:36.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.36e-07 2024-03-20T09:24:36.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:24:36.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.168623+0000", 2024-03-20T09:24:36.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9740000000000004e-06 2024-03-20T09:24:36.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:24:36.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.168627+0000", 2024-03-20T09:24:36.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4089999999999999e-06 2024-03-20T09:24:36.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:24:36.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.168629+0000", 2024-03-20T09:24:36.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7499999999999999e-06 2024-03-20T09:24:36.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.178966+0000", 2024-03-20T09:24:36.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010337162 2024-03-20T09:24:36.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:24:36.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.179037+0000", 2024-03-20T09:24:36.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0805999999999999e-05 2024-03-20T09:24:36.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:10.533363+0000", 2024-03-20T09:24:36.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3543253019999999 2024-03-20T09:24:36.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:24:36.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:10.533379+0000", 2024-03-20T09:24:36.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6184999999999998e-05 2024-03-20T09:24:36.272 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.272 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:36.272 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.272 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:222768 5.5 5:acab01bd:::benchmark_data_smithi122_157894_object92781:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e682)", 2024-03-20T09:24:36.272 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:22:45.242872+0000", 2024-03-20T09:24:36.272 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 110.918429289, 2024-03-20T09:24:36.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0710168700000002, 2024-03-20T09:24:36.272 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:24:36.272 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:24:36.272 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:24:36.272 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:24:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:24:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 222768 2024-03-20T09:24:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:24:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:24:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:45.242872+0000", 2024-03-20T09:24:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:24:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:45.242872+0000", 2024-03-20T09:24:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:24:36.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:24:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:45.242873+0000", 2024-03-20T09:24:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4000000000000002e-07 2024-03-20T09:24:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:24:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:45.242876+0000", 2024-03-20T09:24:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8650000000000001e-06 2024-03-20T09:24:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:24:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:45.242883+0000", 2024-03-20T09:24:36.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0099999999999998e-06 2024-03-20T09:24:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:24:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:45.242889+0000", 2024-03-20T09:24:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9429999999999997e-06 2024-03-20T09:24:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:45.249232+0000", 2024-03-20T09:24:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006343347 2024-03-20T09:24:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:24:36.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:45.249329+0000", 2024-03-20T09:24:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7199000000000002e-05 2024-03-20T09:24:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:24:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:49.313872+0000", 2024-03-20T09:24:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0645422570000003 2024-03-20T09:24:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:24:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:49.313889+0000", 2024-03-20T09:24:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7808999999999998e-05 2024-03-20T09:24:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.276 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:36.277 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:36.277 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:24:36.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:36.278 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:24:36.278 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:24:36.279 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:24:36.335 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:36 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2516198713' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:24:36.381 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:24:36.411 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:36.431 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:24:36.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:36 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2516198713' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:24:36.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:36.434 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:24:36.468 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:36 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2516198713' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:24:36.535 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:24:36.564 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:36.583 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:24:36.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:36.586 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:24:36.687 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:24:36.716 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:36.858 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 21 16 27896 27880 82.9625 0 - 0.00598077 2024-03-20T09:24:37.066 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:24:37.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:37.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:24:37.170 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:24:37.203 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:37.224 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:24:37.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:37.228 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:24:37.329 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:24:37.347 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:37 smithi026 ceph-mon[68309]: pgmap v1763: 47 pgs: 47 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 356 op/s 2024-03-20T09:24:37.347 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:37 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:24:37.347 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:37 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2516198713' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:24:37.347 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:37 smithi026 ceph-mon[68309]: osdmap e724: 12 total, 12 up, 12 in 2024-03-20T09:24:37.358 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:37.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:37 smithi115 ceph-mon[72476]: pgmap v1763: 47 pgs: 47 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 356 op/s 2024-03-20T09:24:37.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:37 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:24:37.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:37 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2516198713' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:24:37.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:37 smithi115 ceph-mon[72476]: osdmap e724: 12 total, 12 up, 12 in 2024-03-20T09:24:37.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:37 smithi122 ceph-mon[73495]: pgmap v1763: 47 pgs: 47 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s wr, 356 op/s 2024-03-20T09:24:37.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:37 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:24:37.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:37 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2516198713' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:24:37.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:37 smithi122 ceph-mon[73495]: osdmap e724: 12 total, 12 up, 12 in 2024-03-20T09:24:37.858 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 22 16 27896 27880 79.1915 0 - 0.00598077 2024-03-20T09:24:38.327 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-20T09:24:38.602 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:38 smithi026 ceph-mon[68309]: osdmap e725: 12 total, 12 up, 12 in 2024-03-20T09:24:38.603 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:38 smithi026 ceph-mon[68309]: pgmap v1766: 47 pgs: 47 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s wr, 24 op/s 2024-03-20T09:24:38.639 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:24:38.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:38.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:24:38.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:38 smithi115 ceph-mon[72476]: osdmap e725: 12 total, 12 up, 12 in 2024-03-20T09:24:38.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:38 smithi115 ceph-mon[72476]: pgmap v1766: 47 pgs: 47 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s wr, 24 op/s 2024-03-20T09:24:38.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:38 smithi122 ceph-mon[73495]: osdmap e725: 12 total, 12 up, 12 in 2024-03-20T09:24:38.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:38 smithi122 ceph-mon[73495]: pgmap v1766: 47 pgs: 47 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s wr, 24 op/s 2024-03-20T09:24:38.750 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:24:38.811 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:38.840 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:24:38.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:38.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:24:38.858 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 23 16 30149 30133 81.8699 11.7344 0.0071054 0.0122101 2024-03-20T09:24:38.946 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:24:38.978 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:39.001 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:24:39.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:39.004 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:24:39.105 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:24:39.152 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:39.183 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:24:39.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:39.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:24:39.288 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:24:39.319 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:39.688 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:24:39.858 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 24 16 32377 32361 84.2601 139.25 0.00426519 0.0118635 2024-03-20T09:24:39.953 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:39 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:24:39.954 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:39 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:24:40.154 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:39 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:24:40.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:39 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:24:40.194 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:39 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:24:40.195 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:39 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:24:40.859 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 25 16 34753 34737 86.829 148.5 0.00579565 0.0115124 2024-03-20T09:24:41.154 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:40 smithi115 ceph-mon[72476]: pgmap v1767: 47 pgs: 47 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 184 op/s 2024-03-20T09:24:41.194 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:40 smithi122 ceph-mon[73495]: pgmap v1767: 47 pgs: 47 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 184 op/s 2024-03-20T09:24:41.219 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:40 smithi026 ceph-mon[68309]: pgmap v1767: 47 pgs: 47 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s wr, 184 op/s 2024-03-20T09:24:41.227 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:24:41.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:41.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:24:41.332 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:24:41.376 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:41.404 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:24:41.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:41.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:24:41.509 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:24:41.550 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:41.556 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:24:41.577 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:24:41.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:41.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:24:41.596 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:24:41.646 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:24:41.682 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:24:41.690 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:24:41.734 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:24:41.779 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:24:41.819 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:24:41.856 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:24:41.858 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 26 16 36776 36760 88.3519 126.438 0.00802748 0.0113138 2024-03-20T09:24:41.886 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:41.908 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:24:41.968 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:24:41.996 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:24:42.031 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:24:42.047 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:24:42.047 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:24:42.047 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:24:42.048 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:24:42.048 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:24:42.048 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:24:42.048 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:24:42.048 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:24:42.048 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:24:42.048 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:24:42.048 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:24:42.048 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:24:42.056 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:24:42.056 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:24:42.056 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 725, 2024-03-20T09:24:42.056 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "852fbb68-e696-11ee-95c9-87774f69a715", 2024-03-20T09:24:42.056 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-03-20T08:48:35.281088+0000", 2024-03-20T09:24:42.057 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-03-20T09:24:37.380000+0000", 2024-03-20T09:24:42.057 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2024-03-20T09:10:33.084151+0000", 2024-03-20T09:24:42.057 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2024-03-20T09:20:22.151285+0000", 2024-03-20T09:24:42.057 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-20T09:24:42.057 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5801984, 2024-03-20T09:24:42.057 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2024-03-20T09:24:42.057 INFO:teuthology.orchestra.run.smithi026.stdout: "noscrub", 2024-03-20T09:24:42.057 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2024-03-20T09:24:42.057 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2024-03-20T09:24:42.057 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2024-03-20T09:24:42.057 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2024-03-20T09:24:42.057 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:24:42.058 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 27, 2024-03-20T09:24:42.058 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2024-03-20T09:24:42.058 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-20T09:24:42.058 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-20T09:24:42.058 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2024-03-20T09:24:42.058 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 6, 2024-03-20T09:24:42.058 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 12, 2024-03-20T09:24:42.058 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "luminous", 2024-03-20T09:24:42.058 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "luminous", 2024-03-20T09:24:42.058 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "squid", 2024-03-20T09:24:42.058 INFO:teuthology.orchestra.run.smithi026.stdout: "allow_crimson": false, 2024-03-20T09:24:42.058 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2024-03-20T09:24:42.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.059 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2024-03-20T09:24:42.059 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": ".mgr", 2024-03-20T09:24:42.059 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-03-20T08:52:17.442324+0000", 2024-03-20T09:24:42.059 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-03-20T09:24:42.059 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-03-20T09:24:42.059 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-03-20T09:24:42.059 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 3, 2024-03-20T09:24:42.059 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 2, 2024-03-20T09:24:42.059 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-03-20T09:24:42.059 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-03-20T09:24:42.059 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-03-20T09:24:42.059 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-03-20T09:24:42.059 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-20T09:24:42.060 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-03-20T09:24:42.060 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-03-20T09:24:42.060 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 31, 2024-03-20T09:24:42.060 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 31, 2024-03-20T09:24:42.060 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 31, 2024-03-20T09:24:42.060 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 31, 2024-03-20T09:24:42.060 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 31, 2024-03-20T09:24:42.060 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-03-20T09:24:42.060 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-03-20T09:24:42.060 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-03-20T09:24:42.060 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-03-20T09:24:42.060 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-03-20T09:24:42.060 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-03-20T09:24:42.060 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-03-20T09:24:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "360", 2024-03-20T09:24:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-03-20T09:24:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-20T09:24:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "358", 2024-03-20T09:24:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-03-20T09:24:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-03-20T09:24:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-03-20T09:24:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-03-20T09:24:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-03-20T09:24:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-03-20T09:24:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-03-20T09:24:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-03-20T09:24:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-03-20T09:24:42.061 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-03-20T09:24:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-03-20T09:24:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-03-20T09:24:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-03-20T09:24:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-03-20T09:24:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-03-20T09:24:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-20T09:24:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-20T09:24:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-20T09:24:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-03-20T09:24:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-03-20T09:24:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-03-20T09:24:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-03-20T09:24:42.062 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-03-20T09:24:42.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.063 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-03-20T09:24:42.063 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-03-20T09:24:42.063 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-03-20T09:24:42.063 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-03-20T09:24:42.063 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-03-20T09:24:42.063 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-03-20T09:24:42.063 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-03-20T09:24:42.064 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-03-20T09:24:42.064 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-03-20T09:24:42.064 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-03-20T09:24:42.064 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-03-20T09:24:42.064 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2024-03-20T09:24:42.064 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_max": 32, 2024-03-20T09:24:42.064 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2024-03-20T09:24:42.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.064 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-03-20T09:24:42.064 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": {} 2024-03-20T09:24:42.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.064 INFO:teuthology.orchestra.run.smithi026.stdout: "read_balance": { 2024-03-20T09:24:42.064 INFO:teuthology.orchestra.run.smithi026.stdout: "score_type": "Fair distribution", 2024-03-20T09:24:42.065 INFO:teuthology.orchestra.run.smithi026.stdout: "score_acting": 1.5499999523162842, 2024-03-20T09:24:42.065 INFO:teuthology.orchestra.run.smithi026.stdout: "score_stable": 1.5499999523162842, 2024-03-20T09:24:42.065 INFO:teuthology.orchestra.run.smithi026.stdout: "optimal_score": 1, 2024-03-20T09:24:42.065 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_acting": 1.5499999523162842, 2024-03-20T09:24:42.065 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_stable": 1.5499999523162842, 2024-03-20T09:24:42.065 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity_weighted": 0.92150616645812988, 2024-03-20T09:24:42.065 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity": 0.92000001668930054, 2024-03-20T09:24:42.065 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-03-20T09:24:42.065 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.065 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 6, 2024-03-20T09:24:42.065 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "unique_pool_4", 2024-03-20T09:24:42.065 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-03-20T09:24:02.337365+0000", 2024-03-20T09:24:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-03-20T09:24:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-03-20T09:24:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-03-20T09:24:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 3, 2024-03-20T09:24:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 2, 2024-03-20T09:24:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-03-20T09:24:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-03-20T09:24:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-03-20T09:24:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-03-20T09:24:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-20T09:24:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-03-20T09:24:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-03-20T09:24:42.066 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 16, 2024-03-20T09:24:42.067 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 16, 2024-03-20T09:24:42.067 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 16, 2024-03-20T09:24:42.067 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 16, 2024-03-20T09:24:42.067 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 16, 2024-03-20T09:24:42.067 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-03-20T09:24:42.067 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-03-20T09:24:42.067 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-03-20T09:24:42.067 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-03-20T09:24:42.068 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-03-20T09:24:42.068 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-03-20T09:24:42.068 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-03-20T09:24:42.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.068 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "715", 2024-03-20T09:24:42.068 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-03-20T09:24:42.068 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-20T09:24:42.068 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2024-03-20T09:24:42.068 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-03-20T09:24:42.068 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-03-20T09:24:42.068 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-03-20T09:24:42.068 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-03-20T09:24:42.068 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-03-20T09:24:42.068 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-03-20T09:24:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-03-20T09:24:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-03-20T09:24:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-03-20T09:24:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-03-20T09:24:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-03-20T09:24:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-03-20T09:24:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-03-20T09:24:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-03-20T09:24:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-03-20T09:24:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-20T09:24:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-20T09:24:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-20T09:24:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-03-20T09:24:42.069 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-03-20T09:24:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-03-20T09:24:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-03-20T09:24:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-03-20T09:24:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-03-20T09:24:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-03-20T09:24:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-03-20T09:24:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-03-20T09:24:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-03-20T09:24:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-03-20T09:24:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-03-20T09:24:42.070 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-03-20T09:24:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-03-20T09:24:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-03-20T09:24:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-03-20T09:24:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2024-03-20T09:24:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-03-20T09:24:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: "rados": {} 2024-03-20T09:24:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: "read_balance": { 2024-03-20T09:24:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: "score_type": "Fair distribution", 2024-03-20T09:24:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: "score_acting": 2.25, 2024-03-20T09:24:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: "score_stable": 2.25, 2024-03-20T09:24:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: "optimal_score": 1, 2024-03-20T09:24:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_acting": 2.25, 2024-03-20T09:24:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_stable": 2.25, 2024-03-20T09:24:42.071 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity_weighted": 0.92150616645812988, 2024-03-20T09:24:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity": 0.92000001668930054, 2024-03-20T09:24:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-03-20T09:24:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:24:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:24:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-03-20T09:24:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "ad3fd9c0-316f-4fe5-a63b-d480087c1729", 2024-03-20T09:24:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:24:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:24:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:24:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:24:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:24:42.072 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:24:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 8, 2024-03-20T09:24:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 717, 2024-03-20T09:24:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:24:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:24:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:24:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2024-03-20T09:24:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:24:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.073 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2024-03-20T09:24:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:24:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:24:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2024-03-20T09:24:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:24:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2024-03-20T09:24:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:24:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:24:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2024-03-20T09:24:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:24:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2024-03-20T09:24:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:24:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:24:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2024-03-20T09:24:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:24:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.077 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.077 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2024-03-20T09:24:42.077 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:24:42.077 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.077 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.077 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/2139805293", 2024-03-20T09:24:42.077 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/2139805293", 2024-03-20T09:24:42.077 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/2139805293", 2024-03-20T09:24:42.077 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/2139805293", 2024-03-20T09:24:42.077 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:24:42.077 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:24:42.077 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:24:42.077 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-03-20T09:24:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e85b94f5-2954-47a1-9c20-096f2606eb04", 2024-03-20T09:24:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:24:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:24:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:24:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:24:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:24:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:24:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2024-03-20T09:24:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 709, 2024-03-20T09:24:42.078 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:24:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:24:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:24:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2024-03-20T09:24:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:24:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2024-03-20T09:24:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:24:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:24:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2024-03-20T09:24:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:24:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2024-03-20T09:24:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:24:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:24:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2024-03-20T09:24:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:24:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2024-03-20T09:24:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:24:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:24:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2024-03-20T09:24:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:24:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2024-03-20T09:24:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:24:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/3853355134", 2024-03-20T09:24:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/3853355134", 2024-03-20T09:24:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/3853355134", 2024-03-20T09:24:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/3853355134", 2024-03-20T09:24:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:24:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:24:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:24:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-03-20T09:24:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "dd2b10f0-741c-47a4-b224-ac6bfe2c14ef", 2024-03-20T09:24:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:24:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:24:42.083 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:24:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:24:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:24:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:24:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 18, 2024-03-20T09:24:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 709, 2024-03-20T09:24:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:24:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:24:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:24:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.084 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2024-03-20T09:24:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:24:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2024-03-20T09:24:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:24:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:24:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.085 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2024-03-20T09:24:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:24:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2024-03-20T09:24:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:24:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:24:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.086 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2024-03-20T09:24:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:24:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2024-03-20T09:24:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:24:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:24:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.087 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2024-03-20T09:24:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:24:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2024-03-20T09:24:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:24:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/3993357967", 2024-03-20T09:24:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/3993357967", 2024-03-20T09:24:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/3993357967", 2024-03-20T09:24:42.088 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/3993357967", 2024-03-20T09:24:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:24:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:24:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:24:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-03-20T09:24:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1", 2024-03-20T09:24:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:24:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:24:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:24:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:24:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:24:42.089 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:24:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 26, 2024-03-20T09:24:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 709, 2024-03-20T09:24:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:24:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:24:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:24:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2024-03-20T09:24:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:24:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.090 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2024-03-20T09:24:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:24:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:24:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2024-03-20T09:24:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:24:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.091 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2024-03-20T09:24:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:24:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:24:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2024-03-20T09:24:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:24:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.092 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2024-03-20T09:24:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:24:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:24:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2024-03-20T09:24:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:24:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.093 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2024-03-20T09:24:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:24:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/4110225509", 2024-03-20T09:24:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/4110225509", 2024-03-20T09:24:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/4110225509", 2024-03-20T09:24:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/4110225509", 2024-03-20T09:24:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:24:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:24:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:24:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-03-20T09:24:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "f14e34d5-9829-42a3-9b25-31a0976c5ef7", 2024-03-20T09:24:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:24:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:24:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:24:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:24:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:24:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:24:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 32, 2024-03-20T09:24:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 709, 2024-03-20T09:24:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:24:42.095 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:24:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:24:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6800", 2024-03-20T09:24:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:24:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6801", 2024-03-20T09:24:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:24:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.096 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:24:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6802", 2024-03-20T09:24:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:24:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6803", 2024-03-20T09:24:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:24:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.097 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:24:42.098 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.098 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.098 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6806", 2024-03-20T09:24:42.098 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:24:42.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.098 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.098 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6807", 2024-03-20T09:24:42.101 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:24:42.101 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.101 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.101 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:24:42.101 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.102 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.102 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6804", 2024-03-20T09:24:42.102 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:24:42.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.102 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.102 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6805", 2024-03-20T09:24:42.102 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:24:42.102 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.102 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.102 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6801/477923058", 2024-03-20T09:24:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6803/477923058", 2024-03-20T09:24:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6807/477923058", 2024-03-20T09:24:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6805/477923058", 2024-03-20T09:24:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:24:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:24:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:24:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-03-20T09:24:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "cb431b6a-540d-4318-ae35-d379e1df0185", 2024-03-20T09:24:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:24:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:24:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:24:42.103 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:24:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 41, 2024-03-20T09:24:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 273, 2024-03-20T09:24:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 443, 2024-03-20T09:24:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 709, 2024-03-20T09:24:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 274, 2024-03-20T09:24:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:24:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:24:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6808", 2024-03-20T09:24:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:24:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6809", 2024-03-20T09:24:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:24:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:24:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6810", 2024-03-20T09:24:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:24:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.106 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.106 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6811", 2024-03-20T09:24:42.106 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:24:42.106 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.106 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.106 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:24:42.106 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.106 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.106 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6814", 2024-03-20T09:24:42.106 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:24:42.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.107 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.107 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6815", 2024-03-20T09:24:42.107 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:24:42.107 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.107 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.107 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:24:42.107 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.107 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.107 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6812", 2024-03-20T09:24:42.107 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:24:42.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.108 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.108 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6813", 2024-03-20T09:24:42.108 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:24:42.108 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.108 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.108 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6809/650416648", 2024-03-20T09:24:42.108 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6811/650416648", 2024-03-20T09:24:42.108 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6815/650416648", 2024-03-20T09:24:42.108 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6813/650416648", 2024-03-20T09:24:42.108 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:24:42.108 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:24:42.108 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:24:42.108 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.109 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2024-03-20T09:24:42.109 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "753ed830-e764-4ff3-9d9b-87e7fa748b81", 2024-03-20T09:24:42.109 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:24:42.109 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:24:42.109 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:24:42.109 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:24:42.109 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:24:42.109 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:24:42.109 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 51, 2024-03-20T09:24:42.109 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 709, 2024-03-20T09:24:42.109 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:24:42.109 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:24:42.110 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:24:42.110 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.110 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.110 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6816", 2024-03-20T09:24:42.110 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:24:42.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.110 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.110 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6817", 2024-03-20T09:24:42.110 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:24:42.110 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.110 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.111 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:24:42.111 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.111 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.111 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6818", 2024-03-20T09:24:42.111 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:24:42.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.111 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.111 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6819", 2024-03-20T09:24:42.111 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:24:42.111 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.111 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.111 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:24:42.112 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.112 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.112 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6822", 2024-03-20T09:24:42.112 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:24:42.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.112 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.112 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6823", 2024-03-20T09:24:42.112 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:24:42.112 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.112 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.112 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:24:42.113 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.113 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.113 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6820", 2024-03-20T09:24:42.113 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:24:42.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.113 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.113 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6821", 2024-03-20T09:24:42.113 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:24:42.113 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.113 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.114 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6817/3278301843", 2024-03-20T09:24:42.114 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6819/3278301843", 2024-03-20T09:24:42.114 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6823/3278301843", 2024-03-20T09:24:42.114 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6821/3278301843", 2024-03-20T09:24:42.114 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:24:42.114 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:24:42.114 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:24:42.114 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.114 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2024-03-20T09:24:42.114 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "55bcbd37-7a6f-4ab1-b5a9-91a951444d86", 2024-03-20T09:24:42.114 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:24:42.115 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:24:42.115 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:24:42.115 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 0.058074951171875, 2024-03-20T09:24:42.115 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:24:42.115 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:24:42.115 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 60, 2024-03-20T09:24:42.115 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 517, 2024-03-20T09:24:42.115 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:24:42.115 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:24:42.115 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:24:42.115 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.115 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.115 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6824", 2024-03-20T09:24:42.116 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:24:42.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.116 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.116 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6825", 2024-03-20T09:24:42.116 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:24:42.116 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.116 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.116 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:24:42.116 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.116 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.116 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6826", 2024-03-20T09:24:42.117 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:24:42.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.117 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.117 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6827", 2024-03-20T09:24:42.117 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:24:42.117 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.117 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.117 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:24:42.117 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.117 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.117 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6830", 2024-03-20T09:24:42.118 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:24:42.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.118 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.118 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6831", 2024-03-20T09:24:42.118 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:24:42.118 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.118 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.118 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:24:42.118 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.118 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.119 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6828", 2024-03-20T09:24:42.119 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:24:42.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.119 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.119 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6829", 2024-03-20T09:24:42.119 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:24:42.119 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.119 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.119 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6825/3152424001", 2024-03-20T09:24:42.119 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6827/3152424001", 2024-03-20T09:24:42.119 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6831/3152424001", 2024-03-20T09:24:42.119 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6829/3152424001", 2024-03-20T09:24:42.120 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:24:42.120 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:24:42.120 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:24:42.120 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.120 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 8, 2024-03-20T09:24:42.120 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "bc86248e-6bd0-4369-b8a2-e593bc2a610d", 2024-03-20T09:24:42.120 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:24:42.120 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:24:42.120 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:24:42.120 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:24:42.120 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:24:42.120 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:24:42.120 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 70, 2024-03-20T09:24:42.120 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 718, 2024-03-20T09:24:42.121 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:24:42.121 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:24:42.121 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:24:42.121 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.121 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.121 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6800", 2024-03-20T09:24:42.121 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:24:42.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.121 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.121 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6801", 2024-03-20T09:24:42.121 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:24:42.121 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.121 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.122 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:24:42.122 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.122 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.122 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6802", 2024-03-20T09:24:42.122 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:24:42.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.122 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.122 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6803", 2024-03-20T09:24:42.122 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:24:42.122 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.122 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:24:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6806", 2024-03-20T09:24:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:24:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6807", 2024-03-20T09:24:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:24:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.123 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.124 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:24:42.124 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.124 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.124 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6804", 2024-03-20T09:24:42.124 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:24:42.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.124 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.124 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6805", 2024-03-20T09:24:42.124 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:24:42.124 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6801/1309426740", 2024-03-20T09:24:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6803/1309426740", 2024-03-20T09:24:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6807/1309426740", 2024-03-20T09:24:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6805/1309426740", 2024-03-20T09:24:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:24:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:24:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:24:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 9, 2024-03-20T09:24:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b20a4aa-032a-4c09-ac75-153c554424e7", 2024-03-20T09:24:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:24:42.125 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:24:42.126 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:24:42.126 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:24:42.126 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:24:42.126 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:24:42.126 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 80, 2024-03-20T09:24:42.126 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 718, 2024-03-20T09:24:42.126 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:24:42.126 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:24:42.126 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:24:42.126 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.126 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.126 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6808", 2024-03-20T09:24:42.126 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:24:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6809", 2024-03-20T09:24:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:24:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:24:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6810", 2024-03-20T09:24:42.127 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:24:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6811", 2024-03-20T09:24:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:24:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:24:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6814", 2024-03-20T09:24:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:24:42.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6815", 2024-03-20T09:24:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:24:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:24:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6812", 2024-03-20T09:24:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:24:42.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6813", 2024-03-20T09:24:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:24:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6809/2953210076", 2024-03-20T09:24:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6811/2953210076", 2024-03-20T09:24:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6815/2953210076", 2024-03-20T09:24:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6813/2953210076", 2024-03-20T09:24:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:24:42.130 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:24:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:24:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 10, 2024-03-20T09:24:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e62c1fb3-5dd1-4d0d-997c-d20633d19361", 2024-03-20T09:24:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:24:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:24:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:24:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:24:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:24:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:24:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 89, 2024-03-20T09:24:42.131 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 718, 2024-03-20T09:24:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:24:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:24:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:24:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6816", 2024-03-20T09:24:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:24:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6817", 2024-03-20T09:24:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:24:42.132 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:24:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6818", 2024-03-20T09:24:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:24:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6819", 2024-03-20T09:24:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:24:42.133 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:24:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6822", 2024-03-20T09:24:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:24:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6823", 2024-03-20T09:24:42.134 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:24:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:24:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6820", 2024-03-20T09:24:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:24:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6821", 2024-03-20T09:24:42.135 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:24:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6817/409394340", 2024-03-20T09:24:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6819/409394340", 2024-03-20T09:24:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6823/409394340", 2024-03-20T09:24:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6821/409394340", 2024-03-20T09:24:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:24:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:24:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:24:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.136 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 11, 2024-03-20T09:24:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b83994d5-e5ef-48b0-9758-c26843cec3b6", 2024-03-20T09:24:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:24:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:24:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:24:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:24:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:24:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:24:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 96, 2024-03-20T09:24:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 709, 2024-03-20T09:24:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:24:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:24:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:24:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6824", 2024-03-20T09:24:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:24:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6825", 2024-03-20T09:24:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:24:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:24:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6826", 2024-03-20T09:24:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:24:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6827", 2024-03-20T09:24:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:24:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:24:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6830", 2024-03-20T09:24:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:24:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6831", 2024-03-20T09:24:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:24:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:24:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:24:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.140 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:24:42.141 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6828", 2024-03-20T09:24:42.141 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:24:42.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.141 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:24:42.141 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6829", 2024-03-20T09:24:42.141 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:24:42.141 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.141 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.141 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6825/2704069354", 2024-03-20T09:24:42.141 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6827/2704069354", 2024-03-20T09:24:42.141 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6831/2704069354", 2024-03-20T09:24:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6829/2704069354", 2024-03-20T09:24:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:24:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:24:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:24:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:24:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2024-03-20T09:24:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-03-20T09:24:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:24:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:24:42.142 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:24:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:24:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:24:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:24:03.669839+0000", 2024-03-20T09:24:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:24:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-03-20T09:24:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:24:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:24:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:24:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:24:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:24:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:24:20.926175+0000", 2024-03-20T09:24:42.143 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:24:42.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.144 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-03-20T09:24:42.144 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:24:42.144 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:24:42.144 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:24:42.144 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:24:42.144 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:24:42.144 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:23:23.655318+0000", 2024-03-20T09:24:42.144 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:24:42.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.144 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-03-20T09:24:42.145 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:24:42.145 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:24:42.145 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:24:42.145 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:24:42.145 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:24:42.145 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:23:57.178292+0000", 2024-03-20T09:24:42.145 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:24:42.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-03-20T09:24:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:24:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:24:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:24:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:24:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:24:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:24:01.493001+0000", 2024-03-20T09:24:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:24:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-03-20T09:24:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:24:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:24:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:24:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:24:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:24:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:23:35.692424+0000", 2024-03-20T09:24:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 273 2024-03-20T09:24:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2024-03-20T09:24:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:24:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:24:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:24:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:24:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:24:42.147 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:24:15.357950+0000", 2024-03-20T09:24:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:24:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2024-03-20T09:24:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:24:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:24:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:24:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:24:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:24:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:23:37.970653+0000", 2024-03-20T09:24:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:24:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.148 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 8, 2024-03-20T09:24:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:24:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:24:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:24:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:24:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:24:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:23:57.578904+0000", 2024-03-20T09:24:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:24:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 9, 2024-03-20T09:24:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:24:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:24:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:24:42.149 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:24:42.150 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:24:42.150 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:24:25.102505+0000", 2024-03-20T09:24:42.150 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:24:42.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.150 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 10, 2024-03-20T09:24:42.150 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:24:42.150 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:24:42.150 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:24:42.150 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:24:42.150 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:24:42.150 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:23:58.556278+0000", 2024-03-20T09:24:42.151 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:24:42.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.151 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 11, 2024-03-20T09:24:42.151 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:24:42.151 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:24:42.151 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:24:42.151 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:24:42.151 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:24:42.151 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:24:10.379081+0000", 2024-03-20T09:24:42.151 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:24:42.151 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.151 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:24:42.152 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [ 2024-03-20T09:24:42.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:24:42.152 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "1.1", 2024-03-20T09:24:42.152 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:24:42.152 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-03-20T09:24:42.152 INFO:teuthology.orchestra.run.smithi026.stdout: 11, 2024-03-20T09:24:42.152 INFO:teuthology.orchestra.run.smithi026.stdout: 5 2024-03-20T09:24:42.152 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:24:42.152 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.152 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:24:42.152 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2024-03-20T09:24:42.152 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_primaries": [], 2024-03-20T09:24:42.152 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2024-03-20T09:24:42.152 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2024-03-20T09:24:42.152 INFO:teuthology.orchestra.run.smithi026.stdout: "blocklist": { 2024-03-20T09:24:42.152 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/4291404191": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:24:42.153 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/642012566": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:24:42.153 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6801/1181459999": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:24:42.153 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/3984513952": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:24:42.153 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6800/1654945498": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:24:42.153 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6800/1181459999": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:24:42.153 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/137770981": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:24:42.153 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/1451237762": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:24:42.153 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/3938386631": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:24:42.153 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6801/1654945498": "2024-03-21T08:49:04.596754+0000" 2024-03-20T09:24:42.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.153 INFO:teuthology.orchestra.run.smithi026.stdout: "range_blocklist": {}, 2024-03-20T09:24:42.153 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2024-03-20T09:24:42.154 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2024-03-20T09:24:42.154 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2024-03-20T09:24:42.154 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2024-03-20T09:24:42.154 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2024-03-20T09:24:42.154 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2024-03-20T09:24:42.154 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2024-03-20T09:24:42.154 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:24:42.154 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2024-03-20T09:24:42.154 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2024-03-20T09:24:42.154 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2024-03-20T09:24:42.154 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2024-03-20T09:24:42.154 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {}, 2024-03-20T09:24:42.154 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": { 2024-03-20T09:24:42.155 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_enabled": false, 2024-03-20T09:24:42.155 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_bucket_count": 0, 2024-03-20T09:24:42.155 INFO:teuthology.orchestra.run.smithi026.stdout: "degraded_stretch_mode": 0, 2024-03-20T09:24:42.155 INFO:teuthology.orchestra.run.smithi026.stdout: "recovering_stretch_mode": 0, 2024-03-20T09:24:42.155 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_bucket": 0 2024-03-20T09:24:42.155 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:24:42.155 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:24:42.156 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:41 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3611984773' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:24:42.156 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:41 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3611984773' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:24:42.194 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:41 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3611984773' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:24:42.459 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:24:42.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:42.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:24:42.564 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:24:42.590 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:42.611 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:24:42.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:42.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:24:42.717 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:24:42.759 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:42.849 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:24:42.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:42.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:24:42.858 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 27 16 38480 38464 89.0237 106.5 0.0116165 0.0112281 2024-03-20T09:24:42.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:42 smithi115 ceph-mon[72476]: from='client.16398 172.21.15.26:0/2354368266' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:24:42.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:42 smithi115 ceph-mon[72476]: pgmap v1768: 47 pgs: 47 active+clean; 2.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 514 op/s 2024-03-20T09:24:42.957 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:24:43.006 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:43.068 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:24:43.071 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:42 smithi026 ceph-mon[68309]: from='client.16398 172.21.15.26:0/2354368266' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:24:43.071 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:42 smithi026 ceph-mon[68309]: pgmap v1768: 47 pgs: 47 active+clean; 2.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 514 op/s 2024-03-20T09:24:43.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:43.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:24:43.194 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:24:43.195 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:42 smithi122 ceph-mon[73495]: from='client.16398 172.21.15.26:0/2354368266' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:24:43.195 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:42 smithi122 ceph-mon[73495]: pgmap v1768: 47 pgs: 47 active+clean; 2.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 514 op/s 2024-03-20T09:24:43.246 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:43.859 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 28 16 40262 40246 89.8216 111.375 0.00457295 0.0111289 2024-03-20T09:24:44.859 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 29 16 42170 42154 90.836 119.25 0.00638819 0.0110041 2024-03-20T09:24:45.066 INFO:tasks.thrashosds.thrasher:j is {'epoch': 725, 'fsid': '852fbb68-e696-11ee-95c9-87774f69a715', 'created': '2024-03-20T08:48:35.281088+0000', 'modified': '2024-03-20T09:24:37.380000+0000', 'last_up_change': '2024-03-20T09:10:33.084151+0000', 'last_in_change': '2024-03-20T09:20:22.151285+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 6, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-20T08:52:17.442324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '360', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '358', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5499999523162842, 'score_stable': 1.5499999523162842, 'optimal_score': 1, 'raw_score_acting': 1.5499999523162842, 'raw_score_stable': 1.5499999523162842, 'primary_affinity_weighted': 0.9215061664581299, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 6, 'pool_name': 'unique_pool_4', 'create_time': '2024-03-20T09:24:02.337365+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '715', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9215061664581299, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}], 'osds': [{'osd': 0, 'uuid': 'ad3fd9c0-316f-4fe5-a63b-d480087c1729', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 717, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6802', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6803', 'nonce': 2139805293}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6804', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6805', 'nonce': 2139805293}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6808', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6809', 'nonce': 2139805293}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6806', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6807', 'nonce': 2139805293}]}, 'public_addr': '172.21.15.26:6803/2139805293', 'cluster_addr': '172.21.15.26:6805/2139805293', 'heartbeat_back_addr': '172.21.15.26:6809/2139805293', 'heartbeat_front_addr': '172.21.15.26:6807/2139805293', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'e85b94f5-2954-47a1-9c20-096f2606eb04', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 709, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6810', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6811', 'nonce': 3853355134}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6812', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6813', 'nonce': 3853355134}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6816', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6817', 'nonce': 3853355134}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6814', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6815', 'nonce': 3853355134}]}, 'public_addr': '172.21.15.26:6811/3853355134', 'cluster_addr': '172.21.15.26:6813/3853355134', 'heartbeat_back_addr': '172.21.15.26:6817/3853355134', 'heartbeat_front_addr': '172.21.15.26:6815/3853355134', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'dd2b10f0-741c-47a4-b224-ac6bfe2c14ef', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 709, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6818', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6819', 'nonce': 3993357967}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6820', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6821', 'nonce': 3993357967}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6824', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6825', 'nonce': 3993357967}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6822', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6823', 'nonce': 3993357967}]}, 'public_addr': '172.21.15.26:6819/3993357967', 'cluster_addr': '172.21.15.26:6821/3993357967', 'heartbeat_back_addr': '172.21.15.26:6825/3993357967', 'heartbeat_front_addr': '172.21.15.26:6823/3993357967', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 709, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6826', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6827', 'nonce': 4110225509}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6828', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6829', 'nonce': 4110225509}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6832', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6833', 'nonce': 4110225509}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6830', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6831', 'nonce': 4110225509}]}, 'public_addr': '172.21.15.26:6827/4110225509', 'cluster_addr': '172.21.15.26:6829/4110225509', 'heartbeat_back_addr': '172.21.15.26:6833/4110225509', 'heartbeat_front_addr': '172.21.15.26:6831/4110225509', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'f14e34d5-9829-42a3-9b25-31a0976c5ef7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 709, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6800', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6801', 'nonce': 477923058}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6802', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6803', 'nonce': 477923058}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6806', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6807', 'nonce': 477923058}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6804', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6805', 'nonce': 477923058}]}, 'public_addr': '172.21.15.115:6801/477923058', 'cluster_addr': '172.21.15.115:6803/477923058', 'heartbeat_back_addr': '172.21.15.115:6807/477923058', 'heartbeat_front_addr': '172.21.15.115:6805/477923058', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'cb431b6a-540d-4318-ae35-d379e1df0185', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 41, 'last_clean_end': 273, 'up_from': 443, 'up_thru': 709, 'down_at': 274, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6808', 'nonce': 650416648}, {'type': 'v1', 'addr': '172.21.15.115:6809', 'nonce': 650416648}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6810', 'nonce': 650416648}, {'type': 'v1', 'addr': '172.21.15.115:6811', 'nonce': 650416648}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6814', 'nonce': 650416648}, {'type': 'v1', 'addr': '172.21.15.115:6815', 'nonce': 650416648}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6812', 'nonce': 650416648}, {'type': 'v1', 'addr': '172.21.15.115:6813', 'nonce': 650416648}]}, 'public_addr': '172.21.15.115:6809/650416648', 'cluster_addr': '172.21.15.115:6811/650416648', 'heartbeat_back_addr': '172.21.15.115:6815/650416648', 'heartbeat_front_addr': '172.21.15.115:6813/650416648', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '753ed830-e764-4ff3-9d9b-87e7fa748b81', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 709, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6816', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6817', 'nonce': 3278301843}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6818', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6819', 'nonce': 3278301843}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6822', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6823', 'nonce': 3278301843}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6820', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6821', 'nonce': 3278301843}]}, 'public_addr': '172.21.15.115:6817/3278301843', 'cluster_addr': '172.21.15.115:6819/3278301843', 'heartbeat_back_addr': '172.21.15.115:6823/3278301843', 'heartbeat_front_addr': '172.21.15.115:6821/3278301843', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '55bcbd37-7a6f-4ab1-b5a9-91a951444d86', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.058074951171875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 517, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6824', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6825', 'nonce': 3152424001}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6826', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6827', 'nonce': 3152424001}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6830', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6831', 'nonce': 3152424001}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6828', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6829', 'nonce': 3152424001}]}, 'public_addr': '172.21.15.115:6825/3152424001', 'cluster_addr': '172.21.15.115:6827/3152424001', 'heartbeat_back_addr': '172.21.15.115:6831/3152424001', 'heartbeat_front_addr': '172.21.15.115:6829/3152424001', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'bc86248e-6bd0-4369-b8a2-e593bc2a610d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 70, 'up_thru': 718, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6800', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6801', 'nonce': 1309426740}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6802', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6803', 'nonce': 1309426740}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6806', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6807', 'nonce': 1309426740}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6804', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6805', 'nonce': 1309426740}]}, 'public_addr': '172.21.15.122:6801/1309426740', 'cluster_addr': '172.21.15.122:6803/1309426740', 'heartbeat_back_addr': '172.21.15.122:6807/1309426740', 'heartbeat_front_addr': '172.21.15.122:6805/1309426740', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '9b20a4aa-032a-4c09-ac75-153c554424e7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 718, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6808', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6809', 'nonce': 2953210076}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6810', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6811', 'nonce': 2953210076}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6814', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6815', 'nonce': 2953210076}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6812', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6813', 'nonce': 2953210076}]}, 'public_addr': '172.21.15.122:6809/2953210076', 'cluster_addr': '172.21.15.122:6811/2953210076', 'heartbeat_back_addr': '172.21.15.122:6815/2953210076', 'heartbeat_front_addr': '172.21.15.122:6813/2953210076', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'e62c1fb3-5dd1-4d0d-997c-d20633d19361', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 89, 'up_thru': 718, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6816', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6817', 'nonce': 409394340}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6818', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6819', 'nonce': 409394340}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6822', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6823', 'nonce': 409394340}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6820', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6821', 'nonce': 409394340}]}, 'public_addr': '172.21.15.122:6817/409394340', 'cluster_addr': '172.21.15.122:6819/409394340', 'heartbeat_back_addr': '172.21.15.122:6823/409394340', 'heartbeat_front_addr': '172.21.15.122:6821/409394340', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'b83994d5-e5ef-48b0-9758-c26843cec3b6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 96, 'up_thru': 709, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6824', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6825', 'nonce': 2704069354}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6826', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6827', 'nonce': 2704069354}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6830', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6831', 'nonce': 2704069354}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6828', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6829', 'nonce': 2704069354}]}, 'public_addr': '172.21.15.122:6825/2704069354', 'cluster_addr': '172.21.15.122:6827/2704069354', 'heartbeat_back_addr': '172.21.15.122:6831/2704069354', 'heartbeat_front_addr': '172.21.15.122:6829/2704069354', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:24:03.669839+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:24:20.926175+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:23:23.655318+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:23:57.178292+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:24:01.493001+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:23:35.692424+0000', 'dead_epoch': 273}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:24:15.357950+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:23:37.970653+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:23:57.578904+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:24:25.102505+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:23:58.556278+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:24:10.379081+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.1', 'osds': [6, 11, 5]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.26:0/4291404191': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/642012566': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:6801/1181459999': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/3984513952': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6800/1654945498': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6800/1181459999': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/137770981': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:0/1451237762': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/3938386631': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6801/1654945498': '2024-03-21T08:49:04.596754+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-20T09:24:45.066 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-03-20T09:24:45.311 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:24:45.383 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:24:45.469 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:45 smithi026 ceph-mon[68309]: pgmap v1769: 47 pgs: 47 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-03-20T09:24:45.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:45 smithi115 ceph-mon[72476]: pgmap v1769: 47 pgs: 47 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-03-20T09:24:45.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:45 smithi122 ceph-mon[73495]: pgmap v1769: 47 pgs: 47 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-03-20T09:24:45.859 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 30 16 43870 43854 91.3495 106.25 0.00784082 0.0109428 2024-03-20T09:24:46.859 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 31 16 46008 45992 92.7128 133.625 0.00322637 0.010782 2024-03-20T09:24:47.032 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:24:47.468 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:47 smithi026 ceph-mon[68309]: pgmap v1770: 47 pgs: 47 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.35k op/s 2024-03-20T09:24:47.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:47 smithi115 ceph-mon[72476]: pgmap v1770: 47 pgs: 47 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.35k op/s 2024-03-20T09:24:47.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:47 smithi122 ceph-mon[73495]: pgmap v1770: 47 pgs: 47 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.35k op/s 2024-03-20T09:24:47.859 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 32 16 47461 47445 92.653 90.8125 0.00248485 0.0107628 2024-03-20T09:24:47.985 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:24:47.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:47.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:24:48.091 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:24:48.152 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:48.568 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:24:48.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:48.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:24:48.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:48 smithi115 ceph-mon[72476]: pgmap v1771: 47 pgs: 47 active+clean; 2.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.45k op/s 2024-03-20T09:24:48.675 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:24:48.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:48 smithi122 ceph-mon[73495]: pgmap v1771: 47 pgs: 47 active+clean; 2.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.45k op/s 2024-03-20T09:24:48.698 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:48 smithi026 ceph-mon[68309]: pgmap v1771: 47 pgs: 47 active+clean; 2.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.45k op/s 2024-03-20T09:24:48.716 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:48.742 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:24:48.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:48.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:24:48.846 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:24:48.859 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 33 16 49351 49335 93.4244 118.125 0.00255668 0.0106752 2024-03-20T09:24:48.874 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:48.893 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:24:48.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:48.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:24:48.997 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:24:49.048 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:49.293 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:24:49.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:49.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:24:49.398 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:24:49.424 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:49.442 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:24:49.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:49.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:24:49.546 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:24:49.585 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:49.616 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:24:49.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:49.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:24:49.721 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:24:49.771 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:49.796 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:24:49.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:49.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:24:49.859 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 34 16 50769 50753 93.2831 88.625 0.00252902 0.010716 2024-03-20T09:24:49.900 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:24:49.972 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:50.012 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:24:50.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:50.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:24:50.067 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 8, 10, 11, 3, 4, 9] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 5] 2024-03-20T09:24:50.068 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:24:50.068 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-20T09:24:50.068 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-20T09:24:50.118 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:24:50.149 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:50.171 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:24:50.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:50.178 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:24:50.279 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:24:50.346 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:24:50.365 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:24:50.365 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:24:50.515 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:50.859 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 35 16 52230 52214 93.2267 91.3125 0.0126435 0.010722 2024-03-20T09:24:51.056 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:24:51.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:51 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:24:51.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:51 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:24:51.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:51 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:24:51.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:51 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:24:51.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:51 smithi115 ceph-mon[72476]: pgmap v1772: 47 pgs: 47 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 101 MiB/s wr, 1.62k op/s 2024-03-20T09:24:51.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:51 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2810888397' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:24:51.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:51 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:24:51.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:51 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:24:51.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:51 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:24:51.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:51 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:24:51.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:51 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:24:51.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:51 smithi122 ceph-mon[73495]: pgmap v1772: 47 pgs: 47 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 101 MiB/s wr, 1.62k op/s 2024-03-20T09:24:51.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:51 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2810888397' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:24:51.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:51 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:24:51.468 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:51 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:24:51.468 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:51 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:24:51.468 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:51 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:24:51.468 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:51 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:24:51.468 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:51 smithi026 ceph-mon[68309]: pgmap v1772: 47 pgs: 47 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 101 MiB/s wr, 1.62k op/s 2024-03-20T09:24:51.469 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:51 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2810888397' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:24:51.469 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:51 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:24:51.807 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:24:51.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:51.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:24:51.859 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 36 16 54028 54012 93.7583 112.375 0.00744199 0.0106614 2024-03-20T09:24:51.912 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:24:51.969 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:52.143 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:24:52.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:52.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:24:52.152 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:52 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:24:52.152 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:52 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:24:52.152 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:52 smithi026 ceph-mon[68309]: osdmap e726: 12 total, 12 up, 12 in 2024-03-20T09:24:52.152 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:52 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:24:52.152 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:52 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:24:52.152 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:52 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:24:52.252 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:24:52.285 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:52 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:24:52.285 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:52 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:24:52.285 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:52 smithi122 ceph-mon[73495]: osdmap e726: 12 total, 12 up, 12 in 2024-03-20T09:24:52.285 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:52 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:24:52.285 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:52 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:24:52.285 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:52 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:24:52.286 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:52.307 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:24:52.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:52.310 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:24:52.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:52 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:24:52.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:52 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:24:52.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:52 smithi115 ceph-mon[72476]: osdmap e726: 12 total, 12 up, 12 in 2024-03-20T09:24:52.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:52 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:24:52.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:52 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:24:52.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:52 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:24:52.411 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:24:52.437 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:52.456 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:24:52.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:52.459 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:24:52.560 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:24:52.614 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:52.773 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:24:52.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:52.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:24:52.859 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 37 16 55414 55398 93.5653 86.625 0.0130617 0.010684 2024-03-20T09:24:52.880 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:24:52.925 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:52.952 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:24:52.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:52.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:24:53.058 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:24:53.084 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:53 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:24:53.084 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:53 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:24:53.084 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:53 smithi026 ceph-mon[68309]: pgmap v1774: 47 pgs: 47 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 116 MiB/s wr, 1.85k op/s 2024-03-20T09:24:53.084 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:53 smithi026 ceph-mon[68309]: osdmap e727: 12 total, 12 up, 12 in 2024-03-20T09:24:53.086 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:53.191 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:24:53.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:53.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:24:53.242 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-20T09:24:53.300 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:24:53.328 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:53 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:24:53.328 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:53 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:24:53.328 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:53 smithi122 ceph-mon[73495]: pgmap v1774: 47 pgs: 47 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 116 MiB/s wr, 1.85k op/s 2024-03-20T09:24:53.328 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:53 smithi122 ceph-mon[73495]: osdmap e727: 12 total, 12 up, 12 in 2024-03-20T09:24:53.331 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:53.353 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:24:53.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:53.357 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:24:53.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:53 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:24:53.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:53 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:24:53.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:53 smithi115 ceph-mon[72476]: pgmap v1774: 47 pgs: 47 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 116 MiB/s wr, 1.85k op/s 2024-03-20T09:24:53.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:53 smithi115 ceph-mon[72476]: osdmap e727: 12 total, 12 up, 12 in 2024-03-20T09:24:53.458 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:24:53.529 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:53.548 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:24:53.859 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 38 16 56910 56894 93.5634 93.5 0.00670715 0.0106837 2024-03-20T09:24:54.510 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:24:54.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:54.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:24:54.624 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:24:54.674 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:54.860 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 39 16 58333 58317 93.4446 88.9375 0.00401162 0.0106966 2024-03-20T09:24:55.336 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:24:55.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:55.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:24:55.441 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:24:55.490 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:55 smithi026 ceph-mon[68309]: pgmap v1776: 47 pgs: 47 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.51k op/s 2024-03-20T09:24:55.494 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:55.595 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:24:55.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:55.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:24:55.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:55 smithi115 ceph-mon[72476]: pgmap v1776: 47 pgs: 47 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.51k op/s 2024-03-20T09:24:55.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:55 smithi122 ceph-mon[73495]: pgmap v1776: 47 pgs: 47 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.51k op/s 2024-03-20T09:24:55.701 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:24:55.749 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:55.786 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:24:55.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:55.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:24:55.860 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:2024-03-20T09:24:55.862702+0000 min lat: 0.00104258 max lat: 11.7007 avg lat: 0.0107012 2024-03-20T09:24:55.860 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T09:24:55.860 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 40 16 59805 59789 93.4083 92 0.0082903 0.0107012 2024-03-20T09:24:55.903 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:24:55.939 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:55.968 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:24:55.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:55.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:24:56.072 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:24:56.112 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:56.141 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:24:56.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:56.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:24:56.249 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:24:56.298 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:56.328 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:24:56.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:56.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:24:56.433 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:24:56.481 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:56.584 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:24:56.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:56.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:24:56.644 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:24:56.694 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:24:56.729 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:56.757 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:24:56.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:56.760 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:24:56.860 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 41 16 61475 61459 93.6755 104.375 0.00592548 0.010671 2024-03-20T09:24:56.861 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:24:56.896 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:56.920 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:24:56.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:56.924 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:24:57.025 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:24:57.062 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:57.094 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:24:57.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:57.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:24:57.200 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:24:57.233 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:57.256 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:24:57.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:57.260 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:24:57.361 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:24:57.362 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:24:57.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:57 smithi115 ceph-mon[72476]: pgmap v1777: 47 pgs: 47 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 110 MiB/s wr, 1.77k op/s 2024-03-20T09:24:57.418 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:57 smithi026 ceph-mon[68309]: pgmap v1777: 47 pgs: 47 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 110 MiB/s wr, 1.77k op/s 2024-03-20T09:24:57.430 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:57.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:57 smithi122 ceph-mon[73495]: pgmap v1777: 47 pgs: 47 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 110 MiB/s wr, 1.77k op/s 2024-03-20T09:24:57.716 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:24:57.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:57.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:24:57.828 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:24:57.860 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 42 16 63040 63024 93.7738 97.8125 0.010783 0.0106594 2024-03-20T09:24:57.873 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:58.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:58 smithi115 ceph-mon[72476]: pgmap v1778: 47 pgs: 47 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.51k op/s 2024-03-20T09:24:58.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:58 smithi122 ceph-mon[73495]: pgmap v1778: 47 pgs: 47 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.51k op/s 2024-03-20T09:24:58.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:58 smithi026 ceph-mon[68309]: pgmap v1778: 47 pgs: 47 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 95 MiB/s wr, 1.51k op/s 2024-03-20T09:24:58.727 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:24:58.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:24:58.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:24:58.833 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:24:58.860 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 43 16 64451 64435 93.6436 88.1875 0.00497376 0.0106606 2024-03-20T09:24:58.875 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:24:59.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:59 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:24:59.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:24:59 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:24:59.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:59 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:24:59.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:24:59 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:24:59.860 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 44 16 66115 66099 93.8787 104 0.00648536 0.0106481 2024-03-20T09:24:59.904 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:59 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:24:59.904 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:24:59 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:25:00.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:00 smithi122 ceph-mon[73495]: osdmap e728: 12 total, 12 up, 12 in 2024-03-20T09:25:00.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:00 smithi122 ceph-mon[73495]: pgmap v1780: 47 pgs: 47 active+clean; 3.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-03-20T09:25:00.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:00 smithi026 ceph-mon[68309]: osdmap e728: 12 total, 12 up, 12 in 2024-03-20T09:25:00.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:00 smithi026 ceph-mon[68309]: pgmap v1780: 47 pgs: 47 active+clean; 3.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-03-20T09:25:00.813 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:25:00.814 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:25:00.814 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:25:00.847 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:25:00.860 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 45 16 67760 67744 94.0771 102.812 0.007328 0.0106254 2024-03-20T09:25:00.892 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:25:00.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:00 smithi115 ceph-mon[72476]: osdmap e728: 12 total, 12 up, 12 in 2024-03-20T09:25:00.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:00 smithi115 ceph-mon[72476]: pgmap v1780: 47 pgs: 47 active+clean; 3.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-03-20T09:25:00.938 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:25:00.992 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:25:01.036 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:25:01.055 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:25:01.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:01.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:25:01.075 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:25:01.117 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:25:01.153 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:25:01.169 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:25:01.200 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:25:01.206 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:25:01.206 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:25:01.206 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:25:01.206 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:25:01.206 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:25:01.206 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:25:01.207 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:25:01.207 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:25:01.207 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:25:01.207 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:25:01.207 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:25:01.207 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:25:01.219 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:01.259 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:25:01.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:01.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:25:01.364 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:25:01.409 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:01.439 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:25:01.439 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:01 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/81794679' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:25:01.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:01.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:25:01.546 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:25:01.621 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:01.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:01 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/81794679' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:25:01.662 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:25:01.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:01.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:25:01.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:01 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/81794679' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:25:01.771 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:25:01.799 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:01.821 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:25:01.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:01.825 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:25:01.860 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 46 16 69278 69262 94.0942 94.875 0.00298206 0.0106234 2024-03-20T09:25:01.926 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:25:01.956 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:01.983 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:25:01.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:01.986 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:25:02.087 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:25:02.121 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:02.151 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:25:02.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:02.155 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:25:02.256 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:25:02.298 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:02.318 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:25:02.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:02.322 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:25:02.325 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:25:02.325 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:25:02.325 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 728, 2024-03-20T09:25:02.325 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "852fbb68-e696-11ee-95c9-87774f69a715", 2024-03-20T09:25:02.325 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-03-20T08:48:35.281088+0000", 2024-03-20T09:25:02.325 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-03-20T09:24:59.403265+0000", 2024-03-20T09:25:02.326 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2024-03-20T09:10:33.084151+0000", 2024-03-20T09:25:02.326 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2024-03-20T09:20:22.151285+0000", 2024-03-20T09:25:02.326 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-20T09:25:02.326 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5806080, 2024-03-20T09:25:02.326 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2024-03-20T09:25:02.326 INFO:teuthology.orchestra.run.smithi026.stdout: "nodeep-scrub", 2024-03-20T09:25:02.326 INFO:teuthology.orchestra.run.smithi026.stdout: "noscrub", 2024-03-20T09:25:02.326 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2024-03-20T09:25:02.326 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2024-03-20T09:25:02.326 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2024-03-20T09:25:02.326 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2024-03-20T09:25:02.326 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:25:02.326 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 27, 2024-03-20T09:25:02.327 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2024-03-20T09:25:02.327 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-20T09:25:02.327 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-20T09:25:02.327 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2024-03-20T09:25:02.327 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 6, 2024-03-20T09:25:02.327 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 12, 2024-03-20T09:25:02.327 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "luminous", 2024-03-20T09:25:02.327 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "luminous", 2024-03-20T09:25:02.327 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "squid", 2024-03-20T09:25:02.327 INFO:teuthology.orchestra.run.smithi026.stdout: "allow_crimson": false, 2024-03-20T09:25:02.327 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2024-03-20T09:25:02.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.327 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2024-03-20T09:25:02.328 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": ".mgr", 2024-03-20T09:25:02.328 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-03-20T08:52:17.442324+0000", 2024-03-20T09:25:02.328 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-03-20T09:25:02.328 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-03-20T09:25:02.328 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-03-20T09:25:02.328 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 3, 2024-03-20T09:25:02.328 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 2, 2024-03-20T09:25:02.328 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-03-20T09:25:02.328 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-03-20T09:25:02.328 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-03-20T09:25:02.328 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-03-20T09:25:02.328 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-20T09:25:02.328 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-03-20T09:25:02.328 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-03-20T09:25:02.329 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 31, 2024-03-20T09:25:02.329 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 31, 2024-03-20T09:25:02.329 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 31, 2024-03-20T09:25:02.329 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 31, 2024-03-20T09:25:02.329 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 31, 2024-03-20T09:25:02.329 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-03-20T09:25:02.329 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-03-20T09:25:02.329 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-03-20T09:25:02.329 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-03-20T09:25:02.329 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-03-20T09:25:02.329 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-03-20T09:25:02.329 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-03-20T09:25:02.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.330 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "360", 2024-03-20T09:25:02.330 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-03-20T09:25:02.330 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-20T09:25:02.330 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "358", 2024-03-20T09:25:02.330 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-03-20T09:25:02.330 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-03-20T09:25:02.330 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-03-20T09:25:02.330 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-03-20T09:25:02.330 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-03-20T09:25:02.330 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-03-20T09:25:02.330 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-03-20T09:25:02.330 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-03-20T09:25:02.331 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-03-20T09:25:02.331 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-03-20T09:25:02.331 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-03-20T09:25:02.331 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-03-20T09:25:02.331 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-03-20T09:25:02.331 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-03-20T09:25:02.331 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-03-20T09:25:02.331 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-20T09:25:02.331 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-20T09:25:02.331 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-20T09:25:02.331 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-03-20T09:25:02.331 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-03-20T09:25:02.331 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-03-20T09:25:02.332 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-03-20T09:25:02.332 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-03-20T09:25:02.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.332 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-03-20T09:25:02.332 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-03-20T09:25:02.332 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-03-20T09:25:02.332 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-03-20T09:25:02.332 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-03-20T09:25:02.332 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-03-20T09:25:02.332 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-03-20T09:25:02.332 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-03-20T09:25:02.332 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-03-20T09:25:02.332 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-03-20T09:25:02.332 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-03-20T09:25:02.333 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2024-03-20T09:25:02.333 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_max": 32, 2024-03-20T09:25:02.333 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2024-03-20T09:25:02.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.333 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-03-20T09:25:02.333 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": {} 2024-03-20T09:25:02.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.333 INFO:teuthology.orchestra.run.smithi026.stdout: "read_balance": { 2024-03-20T09:25:02.333 INFO:teuthology.orchestra.run.smithi026.stdout: "score_type": "Fair distribution", 2024-03-20T09:25:02.333 INFO:teuthology.orchestra.run.smithi026.stdout: "score_acting": 1.5499999523162842, 2024-03-20T09:25:02.333 INFO:teuthology.orchestra.run.smithi026.stdout: "score_stable": 1.5499999523162842, 2024-03-20T09:25:02.333 INFO:teuthology.orchestra.run.smithi026.stdout: "optimal_score": 1, 2024-03-20T09:25:02.333 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_acting": 1.5499999523162842, 2024-03-20T09:25:02.334 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_stable": 1.5499999523162842, 2024-03-20T09:25:02.334 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity_weighted": 0.92150616645812988, 2024-03-20T09:25:02.334 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity": 0.92000001668930054, 2024-03-20T09:25:02.334 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-03-20T09:25:02.334 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.334 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 6, 2024-03-20T09:25:02.334 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "unique_pool_4", 2024-03-20T09:25:02.334 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-03-20T09:24:02.337365+0000", 2024-03-20T09:25:02.334 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-03-20T09:25:02.334 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-03-20T09:25:02.334 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-03-20T09:25:02.334 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 3, 2024-03-20T09:25:02.335 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 2, 2024-03-20T09:25:02.335 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-03-20T09:25:02.335 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-03-20T09:25:02.335 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-03-20T09:25:02.335 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-03-20T09:25:02.335 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-20T09:25:02.335 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-03-20T09:25:02.335 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-03-20T09:25:02.335 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 16, 2024-03-20T09:25:02.335 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 16, 2024-03-20T09:25:02.335 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 16, 2024-03-20T09:25:02.335 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 16, 2024-03-20T09:25:02.335 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 16, 2024-03-20T09:25:02.335 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-03-20T09:25:02.335 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-03-20T09:25:02.336 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-03-20T09:25:02.336 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-03-20T09:25:02.336 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-03-20T09:25:02.336 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-03-20T09:25:02.336 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-03-20T09:25:02.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.336 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "715", 2024-03-20T09:25:02.336 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-03-20T09:25:02.336 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-20T09:25:02.336 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2024-03-20T09:25:02.336 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-03-20T09:25:02.336 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-03-20T09:25:02.336 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-03-20T09:25:02.336 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-03-20T09:25:02.337 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-03-20T09:25:02.337 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-03-20T09:25:02.337 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-03-20T09:25:02.337 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-03-20T09:25:02.337 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-03-20T09:25:02.337 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-03-20T09:25:02.337 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-03-20T09:25:02.337 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-03-20T09:25:02.337 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-03-20T09:25:02.337 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-03-20T09:25:02.337 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-03-20T09:25:02.337 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-20T09:25:02.337 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-20T09:25:02.337 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-20T09:25:02.337 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-03-20T09:25:02.338 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-03-20T09:25:02.338 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-03-20T09:25:02.338 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-03-20T09:25:02.338 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-03-20T09:25:02.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.338 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-03-20T09:25:02.338 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-03-20T09:25:02.338 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-03-20T09:25:02.338 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-03-20T09:25:02.338 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-03-20T09:25:02.338 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-03-20T09:25:02.338 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-03-20T09:25:02.338 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-03-20T09:25:02.338 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-03-20T09:25:02.339 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-03-20T09:25:02.339 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-03-20T09:25:02.339 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2024-03-20T09:25:02.339 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-03-20T09:25:02.339 INFO:teuthology.orchestra.run.smithi026.stdout: "rados": {} 2024-03-20T09:25:02.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.339 INFO:teuthology.orchestra.run.smithi026.stdout: "read_balance": { 2024-03-20T09:25:02.339 INFO:teuthology.orchestra.run.smithi026.stdout: "score_type": "Fair distribution", 2024-03-20T09:25:02.339 INFO:teuthology.orchestra.run.smithi026.stdout: "score_acting": 2.25, 2024-03-20T09:25:02.339 INFO:teuthology.orchestra.run.smithi026.stdout: "score_stable": 2.25, 2024-03-20T09:25:02.339 INFO:teuthology.orchestra.run.smithi026.stdout: "optimal_score": 1, 2024-03-20T09:25:02.339 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_acting": 2.25, 2024-03-20T09:25:02.339 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_stable": 2.25, 2024-03-20T09:25:02.339 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity_weighted": 0.92150616645812988, 2024-03-20T09:25:02.340 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity": 0.92000001668930054, 2024-03-20T09:25:02.340 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-03-20T09:25:02.340 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.340 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.340 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:25:02.340 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:25:02.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.340 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-03-20T09:25:02.340 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "ad3fd9c0-316f-4fe5-a63b-d480087c1729", 2024-03-20T09:25:02.340 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:25:02.340 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:25:02.340 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:25:02.340 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:25:02.340 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:25:02.341 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:25:02.341 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 8, 2024-03-20T09:25:02.341 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 717, 2024-03-20T09:25:02.341 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:25:02.341 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:25:02.341 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:25:02.341 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.341 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.341 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2024-03-20T09:25:02.341 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:25:02.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.341 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.342 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2024-03-20T09:25:02.342 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:25:02.342 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.342 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.342 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:25:02.342 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.342 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.342 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2024-03-20T09:25:02.342 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:25:02.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.342 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.343 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2024-03-20T09:25:02.343 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:25:02.343 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.343 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.343 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:25:02.343 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.343 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.343 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2024-03-20T09:25:02.343 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:25:02.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.343 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.343 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2024-03-20T09:25:02.344 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:25:02.344 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.344 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.344 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:25:02.344 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.344 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.344 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2024-03-20T09:25:02.344 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:25:02.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.344 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.344 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2024-03-20T09:25:02.345 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:25:02.345 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.345 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.345 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/2139805293", 2024-03-20T09:25:02.345 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/2139805293", 2024-03-20T09:25:02.345 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/2139805293", 2024-03-20T09:25:02.345 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/2139805293", 2024-03-20T09:25:02.345 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:25:02.345 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:25:02.345 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:25:02.345 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.345 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-03-20T09:25:02.346 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e85b94f5-2954-47a1-9c20-096f2606eb04", 2024-03-20T09:25:02.346 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:25:02.346 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:25:02.346 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:25:02.346 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:25:02.346 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:25:02.346 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:25:02.346 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2024-03-20T09:25:02.346 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 709, 2024-03-20T09:25:02.346 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:25:02.346 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:25:02.346 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:25:02.346 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.346 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.347 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2024-03-20T09:25:02.347 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:25:02.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.347 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.347 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2024-03-20T09:25:02.347 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:25:02.347 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.347 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.347 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:25:02.347 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.348 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.348 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2024-03-20T09:25:02.348 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:25:02.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.348 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.348 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2024-03-20T09:25:02.348 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:25:02.348 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.348 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.348 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:25:02.348 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.348 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.349 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2024-03-20T09:25:02.349 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:25:02.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.349 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.349 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2024-03-20T09:25:02.349 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:25:02.349 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.349 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.349 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:25:02.349 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.349 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.349 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2024-03-20T09:25:02.350 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:25:02.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.350 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.350 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2024-03-20T09:25:02.350 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:25:02.350 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.350 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.350 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/3853355134", 2024-03-20T09:25:02.350 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/3853355134", 2024-03-20T09:25:02.350 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/3853355134", 2024-03-20T09:25:02.350 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/3853355134", 2024-03-20T09:25:02.350 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:25:02.351 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:25:02.351 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:25:02.351 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.351 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-03-20T09:25:02.351 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "dd2b10f0-741c-47a4-b224-ac6bfe2c14ef", 2024-03-20T09:25:02.351 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:25:02.351 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:25:02.351 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:25:02.351 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:25:02.351 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:25:02.351 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:25:02.351 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 18, 2024-03-20T09:25:02.352 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 709, 2024-03-20T09:25:02.352 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:25:02.352 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:25:02.352 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:25:02.352 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.352 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.352 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2024-03-20T09:25:02.352 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:25:02.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.352 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.352 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2024-03-20T09:25:02.352 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:25:02.352 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.353 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.353 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:25:02.353 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.353 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.353 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2024-03-20T09:25:02.353 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:25:02.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.353 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.353 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2024-03-20T09:25:02.353 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:25:02.353 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.353 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.354 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:25:02.354 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.354 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.354 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2024-03-20T09:25:02.354 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:25:02.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.354 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.354 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2024-03-20T09:25:02.354 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:25:02.354 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.354 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.355 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:25:02.355 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.355 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.355 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2024-03-20T09:25:02.355 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:25:02.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.355 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.355 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2024-03-20T09:25:02.355 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:25:02.355 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.355 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.356 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/3993357967", 2024-03-20T09:25:02.356 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/3993357967", 2024-03-20T09:25:02.356 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/3993357967", 2024-03-20T09:25:02.356 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/3993357967", 2024-03-20T09:25:02.356 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:25:02.356 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:25:02.356 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:25:02.356 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.356 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-03-20T09:25:02.356 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1", 2024-03-20T09:25:02.356 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:25:02.356 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:25:02.356 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:25:02.357 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:25:02.357 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:25:02.357 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:25:02.357 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 26, 2024-03-20T09:25:02.357 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 709, 2024-03-20T09:25:02.357 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:25:02.357 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:25:02.357 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:25:02.357 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.357 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.357 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2024-03-20T09:25:02.357 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:25:02.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.357 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.358 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2024-03-20T09:25:02.358 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:25:02.358 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.358 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.358 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:25:02.358 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.358 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.358 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2024-03-20T09:25:02.358 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:25:02.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.358 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.359 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2024-03-20T09:25:02.359 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:25:02.359 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.359 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.359 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:25:02.359 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.359 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.359 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2024-03-20T09:25:02.359 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:25:02.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.360 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.360 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2024-03-20T09:25:02.360 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:25:02.360 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.360 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.360 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:25:02.360 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.360 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.360 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2024-03-20T09:25:02.360 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:25:02.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.361 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.361 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2024-03-20T09:25:02.361 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:25:02.361 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.361 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.361 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/4110225509", 2024-03-20T09:25:02.361 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/4110225509", 2024-03-20T09:25:02.361 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/4110225509", 2024-03-20T09:25:02.361 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/4110225509", 2024-03-20T09:25:02.361 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:25:02.361 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:25:02.361 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:25:02.362 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.362 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-03-20T09:25:02.362 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "f14e34d5-9829-42a3-9b25-31a0976c5ef7", 2024-03-20T09:25:02.362 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:25:02.362 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:25:02.362 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:25:02.362 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:25:02.362 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:25:02.362 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:25:02.362 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 32, 2024-03-20T09:25:02.362 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 709, 2024-03-20T09:25:02.362 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:25:02.362 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:25:02.363 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:25:02.363 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.363 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.363 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6800", 2024-03-20T09:25:02.363 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:25:02.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.363 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.363 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6801", 2024-03-20T09:25:02.363 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:25:02.363 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.363 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.364 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:25:02.364 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.364 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.364 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6802", 2024-03-20T09:25:02.364 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:25:02.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.364 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.364 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6803", 2024-03-20T09:25:02.364 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:25:02.364 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.364 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.365 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:25:02.365 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.365 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.365 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6806", 2024-03-20T09:25:02.365 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:25:02.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.365 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.365 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6807", 2024-03-20T09:25:02.365 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:25:02.365 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.365 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.366 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:25:02.366 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.366 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.366 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6804", 2024-03-20T09:25:02.366 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:25:02.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.366 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.366 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6805", 2024-03-20T09:25:02.366 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:25:02.366 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.366 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.367 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6801/477923058", 2024-03-20T09:25:02.367 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6803/477923058", 2024-03-20T09:25:02.367 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6807/477923058", 2024-03-20T09:25:02.367 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6805/477923058", 2024-03-20T09:25:02.367 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:25:02.367 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:25:02.367 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:25:02.367 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.367 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-03-20T09:25:02.367 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "cb431b6a-540d-4318-ae35-d379e1df0185", 2024-03-20T09:25:02.367 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:25:02.367 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:25:02.367 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:25:02.368 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:25:02.368 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 41, 2024-03-20T09:25:02.368 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 273, 2024-03-20T09:25:02.368 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 443, 2024-03-20T09:25:02.368 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 709, 2024-03-20T09:25:02.368 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 274, 2024-03-20T09:25:02.368 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:25:02.368 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:25:02.368 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.368 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.368 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6808", 2024-03-20T09:25:02.368 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:25:02.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.369 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.369 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6809", 2024-03-20T09:25:02.369 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:25:02.369 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.369 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.369 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:25:02.369 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.369 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.369 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6810", 2024-03-20T09:25:02.369 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:25:02.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.369 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.370 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6811", 2024-03-20T09:25:02.370 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:25:02.370 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.370 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.370 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:25:02.370 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.370 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.370 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6814", 2024-03-20T09:25:02.370 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:25:02.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.370 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.371 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6815", 2024-03-20T09:25:02.371 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:25:02.371 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.371 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.371 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:25:02.371 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.371 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.371 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6812", 2024-03-20T09:25:02.371 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:25:02.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.371 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.371 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6813", 2024-03-20T09:25:02.372 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:25:02.372 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.372 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.372 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6809/650416648", 2024-03-20T09:25:02.372 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6811/650416648", 2024-03-20T09:25:02.372 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6815/650416648", 2024-03-20T09:25:02.372 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6813/650416648", 2024-03-20T09:25:02.372 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:25:02.372 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:25:02.372 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:25:02.372 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.373 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2024-03-20T09:25:02.373 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "753ed830-e764-4ff3-9d9b-87e7fa748b81", 2024-03-20T09:25:02.373 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:25:02.373 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:25:02.373 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:25:02.373 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:25:02.373 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:25:02.373 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:25:02.373 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 51, 2024-03-20T09:25:02.373 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 709, 2024-03-20T09:25:02.373 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:25:02.373 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:25:02.374 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:25:02.374 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.374 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.374 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6816", 2024-03-20T09:25:02.374 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:25:02.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.374 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.374 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6817", 2024-03-20T09:25:02.375 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:25:02.375 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.375 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.375 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:25:02.375 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.375 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.375 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6818", 2024-03-20T09:25:02.375 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:25:02.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.375 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.375 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6819", 2024-03-20T09:25:02.375 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:25:02.375 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.376 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.376 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:25:02.376 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.376 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.376 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6822", 2024-03-20T09:25:02.376 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:25:02.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.376 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.376 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6823", 2024-03-20T09:25:02.376 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:25:02.376 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.377 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.377 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:25:02.377 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.377 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.377 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6820", 2024-03-20T09:25:02.377 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:25:02.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.377 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.377 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6821", 2024-03-20T09:25:02.377 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:25:02.377 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.377 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.378 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6817/3278301843", 2024-03-20T09:25:02.378 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6819/3278301843", 2024-03-20T09:25:02.378 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6823/3278301843", 2024-03-20T09:25:02.378 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6821/3278301843", 2024-03-20T09:25:02.378 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:25:02.378 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:25:02.378 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:25:02.378 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.378 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2024-03-20T09:25:02.378 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "55bcbd37-7a6f-4ab1-b5a9-91a951444d86", 2024-03-20T09:25:02.378 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:25:02.379 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:25:02.379 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:25:02.379 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 0.058074951171875, 2024-03-20T09:25:02.379 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:25:02.379 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:25:02.379 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 60, 2024-03-20T09:25:02.379 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 517, 2024-03-20T09:25:02.379 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:25:02.379 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:25:02.379 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:25:02.379 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.379 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.379 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.379 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6824", 2024-03-20T09:25:02.379 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:25:02.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.380 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.380 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6825", 2024-03-20T09:25:02.380 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:25:02.380 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.380 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.380 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:25:02.380 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.380 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.380 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6826", 2024-03-20T09:25:02.380 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:25:02.381 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.381 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.381 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.381 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6827", 2024-03-20T09:25:02.381 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:25:02.381 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.381 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.381 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.381 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:25:02.381 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.381 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.381 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.381 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6830", 2024-03-20T09:25:02.381 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:25:02.382 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.382 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.382 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.382 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6831", 2024-03-20T09:25:02.382 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:25:02.382 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.382 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.382 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.382 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:25:02.382 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.382 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.382 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.382 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6828", 2024-03-20T09:25:02.382 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:25:02.383 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.383 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.383 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.383 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6829", 2024-03-20T09:25:02.383 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:25:02.383 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.383 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.383 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.383 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6825/3152424001", 2024-03-20T09:25:02.383 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6827/3152424001", 2024-03-20T09:25:02.383 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6831/3152424001", 2024-03-20T09:25:02.383 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6829/3152424001", 2024-03-20T09:25:02.383 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:25:02.383 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:25:02.383 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:25:02.384 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.384 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.384 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.384 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 8, 2024-03-20T09:25:02.384 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "bc86248e-6bd0-4369-b8a2-e593bc2a610d", 2024-03-20T09:25:02.384 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:25:02.384 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:25:02.384 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:25:02.384 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:25:02.384 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:25:02.384 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:25:02.384 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 70, 2024-03-20T09:25:02.384 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 718, 2024-03-20T09:25:02.384 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:25:02.385 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:25:02.385 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:25:02.385 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.385 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.385 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.385 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6800", 2024-03-20T09:25:02.385 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:25:02.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.385 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.385 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.385 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6801", 2024-03-20T09:25:02.385 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:25:02.385 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.385 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.385 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.386 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:25:02.386 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.386 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.386 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.386 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6802", 2024-03-20T09:25:02.386 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:25:02.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.386 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.386 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.386 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6803", 2024-03-20T09:25:02.386 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:25:02.386 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.386 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.386 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.387 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:25:02.387 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.387 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.387 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.387 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6806", 2024-03-20T09:25:02.387 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:25:02.387 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.387 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.387 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.387 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6807", 2024-03-20T09:25:02.387 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:25:02.387 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.387 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.387 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.387 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:25:02.388 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.388 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.388 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.388 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6804", 2024-03-20T09:25:02.388 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:25:02.388 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.388 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.388 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.388 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6805", 2024-03-20T09:25:02.388 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:25:02.388 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.388 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.388 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.388 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6801/1309426740", 2024-03-20T09:25:02.388 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6803/1309426740", 2024-03-20T09:25:02.389 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6807/1309426740", 2024-03-20T09:25:02.389 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6805/1309426740", 2024-03-20T09:25:02.389 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:25:02.389 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:25:02.389 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:25:02.389 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.389 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.389 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.389 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 9, 2024-03-20T09:25:02.389 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b20a4aa-032a-4c09-ac75-153c554424e7", 2024-03-20T09:25:02.389 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:25:02.389 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:25:02.389 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:25:02.389 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:25:02.389 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:25:02.390 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:25:02.390 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 80, 2024-03-20T09:25:02.390 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 718, 2024-03-20T09:25:02.390 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:25:02.390 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:25:02.390 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:25:02.390 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.390 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.390 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.390 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6808", 2024-03-20T09:25:02.390 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:25:02.390 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.390 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.390 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.390 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6809", 2024-03-20T09:25:02.391 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:25:02.391 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.391 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.391 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.391 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:25:02.391 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.391 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.391 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.391 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6810", 2024-03-20T09:25:02.391 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:25:02.391 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.391 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.391 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.391 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6811", 2024-03-20T09:25:02.391 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:25:02.392 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.392 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.392 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.392 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:25:02.392 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.392 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.392 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.392 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6814", 2024-03-20T09:25:02.392 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:25:02.392 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.392 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.392 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.392 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6815", 2024-03-20T09:25:02.392 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:25:02.393 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.393 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.393 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.393 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:25:02.393 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.393 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.393 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.393 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6812", 2024-03-20T09:25:02.393 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:25:02.393 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.393 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.393 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.393 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6813", 2024-03-20T09:25:02.393 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:25:02.393 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.394 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.394 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.394 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6809/2953210076", 2024-03-20T09:25:02.394 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6811/2953210076", 2024-03-20T09:25:02.394 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6815/2953210076", 2024-03-20T09:25:02.394 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6813/2953210076", 2024-03-20T09:25:02.394 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:25:02.394 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:25:02.394 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:25:02.394 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.394 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.394 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.394 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 10, 2024-03-20T09:25:02.395 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e62c1fb3-5dd1-4d0d-997c-d20633d19361", 2024-03-20T09:25:02.395 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:25:02.395 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:25:02.395 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:25:02.395 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:25:02.395 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:25:02.395 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:25:02.395 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 89, 2024-03-20T09:25:02.395 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 718, 2024-03-20T09:25:02.395 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:25:02.395 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:25:02.395 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:25:02.395 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.395 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.396 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.396 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6816", 2024-03-20T09:25:02.396 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:25:02.396 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.396 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.396 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.396 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6817", 2024-03-20T09:25:02.396 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:25:02.396 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.396 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.396 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.396 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:25:02.396 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.396 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.397 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.397 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6818", 2024-03-20T09:25:02.397 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:25:02.397 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.397 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.397 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.397 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6819", 2024-03-20T09:25:02.397 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:25:02.397 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.397 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.397 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.397 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:25:02.397 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.397 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.398 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.398 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6822", 2024-03-20T09:25:02.398 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:25:02.398 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.398 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.398 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.398 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6823", 2024-03-20T09:25:02.398 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:25:02.398 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.398 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.399 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.399 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:25:02.399 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.399 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.399 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.399 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6820", 2024-03-20T09:25:02.399 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:25:02.399 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.399 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.399 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.399 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6821", 2024-03-20T09:25:02.399 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:25:02.399 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.399 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.399 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.399 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6817/409394340", 2024-03-20T09:25:02.400 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6819/409394340", 2024-03-20T09:25:02.400 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6823/409394340", 2024-03-20T09:25:02.400 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6821/409394340", 2024-03-20T09:25:02.400 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:25:02.400 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:25:02.400 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:25:02.400 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.400 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.400 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.400 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 11, 2024-03-20T09:25:02.400 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b83994d5-e5ef-48b0-9758-c26843cec3b6", 2024-03-20T09:25:02.400 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:25:02.400 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:25:02.400 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:25:02.400 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:25:02.401 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:25:02.401 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:25:02.401 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 96, 2024-03-20T09:25:02.401 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 709, 2024-03-20T09:25:02.401 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:25:02.401 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:25:02.401 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:25:02.401 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.401 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.401 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.401 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6824", 2024-03-20T09:25:02.401 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:25:02.401 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.401 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.402 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.402 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6825", 2024-03-20T09:25:02.402 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:25:02.402 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.402 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.402 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.402 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:25:02.402 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.402 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.402 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.402 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6826", 2024-03-20T09:25:02.402 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:25:02.402 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.402 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.402 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.403 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6827", 2024-03-20T09:25:02.403 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:25:02.403 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.403 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.403 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.403 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:25:02.403 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.403 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.403 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.403 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6830", 2024-03-20T09:25:02.403 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:25:02.403 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.403 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.403 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.403 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6831", 2024-03-20T09:25:02.403 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:25:02.403 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.404 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.404 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.404 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:25:02.404 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:25:02.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.404 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:25:02.404 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6828", 2024-03-20T09:25:02.404 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:25:02.404 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.404 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.404 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:25:02.404 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6829", 2024-03-20T09:25:02.404 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:25:02.404 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.404 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.404 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.405 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6825/2704069354", 2024-03-20T09:25:02.405 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6827/2704069354", 2024-03-20T09:25:02.405 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6831/2704069354", 2024-03-20T09:25:02.405 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6829/2704069354", 2024-03-20T09:25:02.405 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:25:02.405 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:25:02.405 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:25:02.405 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.405 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.405 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:25:02.405 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2024-03-20T09:25:02.405 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.405 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-03-20T09:25:02.405 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:25:02.406 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:25:02.406 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:25:02.406 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:25:02.406 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:25:02.406 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:24:03.669839+0000", 2024-03-20T09:25:02.406 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:25:02.406 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.406 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.406 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-03-20T09:25:02.406 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:25:02.406 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:25:02.406 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:25:02.406 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:25:02.406 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:25:02.407 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:24:20.926175+0000", 2024-03-20T09:25:02.407 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:25:02.407 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.407 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.407 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-03-20T09:25:02.407 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:25:02.407 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:25:02.407 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:25:02.407 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:25:02.407 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:25:02.407 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:24:52.112478+0000", 2024-03-20T09:25:02.407 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:25:02.407 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.407 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.408 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-03-20T09:25:02.408 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:25:02.408 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:25:02.408 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:25:02.408 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:25:02.408 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:25:02.408 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:23:57.178292+0000", 2024-03-20T09:25:02.408 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:25:02.408 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.408 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.408 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-03-20T09:25:02.408 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:25:02.408 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:25:02.408 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:25:02.408 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:25:02.409 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:25:02.409 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:24:01.493001+0000", 2024-03-20T09:25:02.409 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:25:02.409 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.409 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.409 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-03-20T09:25:02.409 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:25:02.409 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:25:02.409 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:25:02.409 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:25:02.409 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:25:02.409 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:24:58.674822+0000", 2024-03-20T09:25:02.409 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 273 2024-03-20T09:25:02.409 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.410 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2024-03-20T09:25:02.410 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:25:02.410 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:25:02.410 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:25:02.410 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:25:02.410 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:25:02.410 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:24:15.357950+0000", 2024-03-20T09:25:02.410 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:25:02.410 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.410 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.410 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2024-03-20T09:25:02.410 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:25:02.410 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:25:02.411 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:25:02.411 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:25:02.411 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:25:02.411 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:24:50.165406+0000", 2024-03-20T09:25:02.411 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:25:02.411 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.411 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.411 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 8, 2024-03-20T09:25:02.411 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:25:02.411 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:25:02.411 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:25:02.411 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:25:02.411 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:25:02.411 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:23:57.578904+0000", 2024-03-20T09:25:02.412 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:25:02.412 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.412 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.412 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 9, 2024-03-20T09:25:02.412 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:25:02.412 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:25:02.412 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:25:02.412 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:25:02.412 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:25:02.412 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:24:25.102505+0000", 2024-03-20T09:25:02.412 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:25:02.412 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.412 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.412 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 10, 2024-03-20T09:25:02.413 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:25:02.413 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:25:02.413 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:25:02.413 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:25:02.413 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:25:02.413 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:23:58.556278+0000", 2024-03-20T09:25:02.413 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:25:02.413 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.413 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.413 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 11, 2024-03-20T09:25:02.413 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:25:02.413 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:25:02.413 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:25:02.413 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:25:02.414 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:25:02.414 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:24:10.379081+0000", 2024-03-20T09:25:02.414 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:25:02.414 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.414 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:25:02.414 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [ 2024-03-20T09:25:02.414 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:02.414 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "1.1", 2024-03-20T09:25:02.414 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:25:02.414 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-03-20T09:25:02.414 INFO:teuthology.orchestra.run.smithi026.stdout: 11, 2024-03-20T09:25:02.414 INFO:teuthology.orchestra.run.smithi026.stdout: 5 2024-03-20T09:25:02.414 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:02.414 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.415 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:25:02.415 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2024-03-20T09:25:02.415 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_primaries": [], 2024-03-20T09:25:02.415 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2024-03-20T09:25:02.415 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2024-03-20T09:25:02.415 INFO:teuthology.orchestra.run.smithi026.stdout: "blocklist": { 2024-03-20T09:25:02.415 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/4291404191": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:25:02.415 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/642012566": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:25:02.415 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6801/1181459999": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:25:02.415 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/3984513952": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:25:02.415 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6800/1654945498": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:25:02.415 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6800/1181459999": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:25:02.415 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/137770981": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:25:02.415 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/1451237762": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:25:02.415 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/3938386631": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:25:02.416 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6801/1654945498": "2024-03-21T08:49:04.596754+0000" 2024-03-20T09:25:02.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.416 INFO:teuthology.orchestra.run.smithi026.stdout: "range_blocklist": {}, 2024-03-20T09:25:02.416 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2024-03-20T09:25:02.416 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2024-03-20T09:25:02.416 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2024-03-20T09:25:02.416 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2024-03-20T09:25:02.416 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2024-03-20T09:25:02.416 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2024-03-20T09:25:02.416 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2024-03-20T09:25:02.416 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.416 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:02.416 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2024-03-20T09:25:02.416 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2024-03-20T09:25:02.416 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2024-03-20T09:25:02.417 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2024-03-20T09:25:02.417 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {}, 2024-03-20T09:25:02.417 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": { 2024-03-20T09:25:02.417 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_enabled": false, 2024-03-20T09:25:02.417 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_bucket_count": 0, 2024-03-20T09:25:02.417 INFO:teuthology.orchestra.run.smithi026.stdout: "degraded_stretch_mode": 0, 2024-03-20T09:25:02.417 INFO:teuthology.orchestra.run.smithi026.stdout: "recovering_stretch_mode": 0, 2024-03-20T09:25:02.417 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_bucket": 0 2024-03-20T09:25:02.417 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:02.417 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:25:02.424 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:25:02.443 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:02 smithi026 ceph-mon[68309]: pgmap v1781: 47 pgs: 47 active+clean; 4.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s 2024-03-20T09:25:02.496 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:02.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:02 smithi122 ceph-mon[73495]: pgmap v1781: 47 pgs: 47 active+clean; 4.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s 2024-03-20T09:25:02.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:02 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/619222703' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:25:02.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:02 smithi122 ceph-mon[73495]: osdmap e729: 12 total, 12 up, 12 in 2024-03-20T09:25:02.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:02 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/619222703' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:25:02.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:02 smithi026 ceph-mon[68309]: osdmap e729: 12 total, 12 up, 12 in 2024-03-20T09:25:02.860 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 47 16 70755 70739 94.0562 92.3125 0.0088379 0.0106279 2024-03-20T09:25:02.904 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:02 smithi115 ceph-mon[72476]: pgmap v1781: 47 pgs: 47 active+clean; 4.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s 2024-03-20T09:25:02.904 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:02 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/619222703' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:25:02.904 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:02 smithi115 ceph-mon[72476]: osdmap e729: 12 total, 12 up, 12 in 2024-03-20T09:25:03.860 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 48 16 72292 72276 94.0978 96.0625 0.00313389 0.0106224 2024-03-20T09:25:04.468 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:25:04.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:04.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:25:04.575 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:25:04.575 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:25:04.575 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-03-20T09:25:04.575 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-03-20T09:25:04.575 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:25:04.585 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:25:04.695 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:04.791 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:25:04.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:04.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:25:04.860 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 49 16 73746 73730 94.0319 90.875 0.00665231 0.0106309 2024-03-20T09:25:04.909 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:25:04.968 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:05.404 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:05 smithi026 ceph-mon[68309]: pgmap v1783: 47 pgs: 47 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-03-20T09:25:05.441 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-20T09:25:05.653 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:05 smithi115 ceph-mon[72476]: pgmap v1783: 47 pgs: 47 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-03-20T09:25:05.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:05 smithi122 ceph-mon[73495]: pgmap v1783: 47 pgs: 47 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-03-20T09:25:05.703 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:25:05.743 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:25:05.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:05.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:25:05.759 INFO:tasks.thrashosds.thrasher:j is {'epoch': 728, 'fsid': '852fbb68-e696-11ee-95c9-87774f69a715', 'created': '2024-03-20T08:48:35.281088+0000', 'modified': '2024-03-20T09:24:59.403265+0000', 'last_up_change': '2024-03-20T09:10:33.084151+0000', 'last_in_change': '2024-03-20T09:20:22.151285+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 6, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-20T08:52:17.442324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '360', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '358', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5499999523162842, 'score_stable': 1.5499999523162842, 'optimal_score': 1, 'raw_score_acting': 1.5499999523162842, 'raw_score_stable': 1.5499999523162842, 'primary_affinity_weighted': 0.9215061664581299, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 6, 'pool_name': 'unique_pool_4', 'create_time': '2024-03-20T09:24:02.337365+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '715', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9215061664581299, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}], 'osds': [{'osd': 0, 'uuid': 'ad3fd9c0-316f-4fe5-a63b-d480087c1729', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 717, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6802', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6803', 'nonce': 2139805293}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6804', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6805', 'nonce': 2139805293}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6808', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6809', 'nonce': 2139805293}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6806', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6807', 'nonce': 2139805293}]}, 'public_addr': '172.21.15.26:6803/2139805293', 'cluster_addr': '172.21.15.26:6805/2139805293', 'heartbeat_back_addr': '172.21.15.26:6809/2139805293', 'heartbeat_front_addr': '172.21.15.26:6807/2139805293', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'e85b94f5-2954-47a1-9c20-096f2606eb04', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 709, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6810', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6811', 'nonce': 3853355134}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6812', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6813', 'nonce': 3853355134}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6816', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6817', 'nonce': 3853355134}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6814', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6815', 'nonce': 3853355134}]}, 'public_addr': '172.21.15.26:6811/3853355134', 'cluster_addr': '172.21.15.26:6813/3853355134', 'heartbeat_back_addr': '172.21.15.26:6817/3853355134', 'heartbeat_front_addr': '172.21.15.26:6815/3853355134', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'dd2b10f0-741c-47a4-b224-ac6bfe2c14ef', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 709, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6818', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6819', 'nonce': 3993357967}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6820', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6821', 'nonce': 3993357967}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6824', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6825', 'nonce': 3993357967}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6822', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6823', 'nonce': 3993357967}]}, 'public_addr': '172.21.15.26:6819/3993357967', 'cluster_addr': '172.21.15.26:6821/3993357967', 'heartbeat_back_addr': '172.21.15.26:6825/3993357967', 'heartbeat_front_addr': '172.21.15.26:6823/3993357967', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 709, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6826', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6827', 'nonce': 4110225509}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6828', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6829', 'nonce': 4110225509}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6832', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6833', 'nonce': 4110225509}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6830', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6831', 'nonce': 4110225509}]}, 'public_addr': '172.21.15.26:6827/4110225509', 'cluster_addr': '172.21.15.26:6829/4110225509', 'heartbeat_back_addr': '172.21.15.26:6833/4110225509', 'heartbeat_front_addr': '172.21.15.26:6831/4110225509', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'f14e34d5-9829-42a3-9b25-31a0976c5ef7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 709, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6800', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6801', 'nonce': 477923058}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6802', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6803', 'nonce': 477923058}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6806', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6807', 'nonce': 477923058}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6804', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6805', 'nonce': 477923058}]}, 'public_addr': '172.21.15.115:6801/477923058', 'cluster_addr': '172.21.15.115:6803/477923058', 'heartbeat_back_addr': '172.21.15.115:6807/477923058', 'heartbeat_front_addr': '172.21.15.115:6805/477923058', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'cb431b6a-540d-4318-ae35-d379e1df0185', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 41, 'last_clean_end': 273, 'up_from': 443, 'up_thru': 709, 'down_at': 274, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6808', 'nonce': 650416648}, {'type': 'v1', 'addr': '172.21.15.115:6809', 'nonce': 650416648}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6810', 'nonce': 650416648}, {'type': 'v1', 'addr': '172.21.15.115:6811', 'nonce': 650416648}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6814', 'nonce': 650416648}, {'type': 'v1', 'addr': '172.21.15.115:6815', 'nonce': 650416648}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6812', 'nonce': 650416648}, {'type': 'v1', 'addr': '172.21.15.115:6813', 'nonce': 650416648}]}, 'public_addr': '172.21.15.115:6809/650416648', 'cluster_addr': '172.21.15.115:6811/650416648', 'heartbeat_back_addr': '172.21.15.115:6815/650416648', 'heartbeat_front_addr': '172.21.15.115:6813/650416648', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '753ed830-e764-4ff3-9d9b-87e7fa748b81', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 709, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6816', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6817', 'nonce': 3278301843}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6818', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6819', 'nonce': 3278301843}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6822', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6823', 'nonce': 3278301843}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6820', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6821', 'nonce': 3278301843}]}, 'public_addr': '172.21.15.115:6817/3278301843', 'cluster_addr': '172.21.15.115:6819/3278301843', 'heartbeat_back_addr': '172.21.15.115:6823/3278301843', 'heartbeat_front_addr': '172.21.15.115:6821/3278301843', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '55bcbd37-7a6f-4ab1-b5a9-91a951444d86', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.058074951171875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 517, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6824', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6825', 'nonce': 3152424001}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6826', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6827', 'nonce': 3152424001}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6830', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6831', 'nonce': 3152424001}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6828', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6829', 'nonce': 3152424001}]}, 'public_addr': '172.21.15.115:6825/3152424001', 'cluster_addr': '172.21.15.115:6827/3152424001', 'heartbeat_back_addr': '172.21.15.115:6831/3152424001', 'heartbeat_front_addr': '172.21.15.115:6829/3152424001', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'bc86248e-6bd0-4369-b8a2-e593bc2a610d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 70, 'up_thru': 718, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6800', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6801', 'nonce': 1309426740}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6802', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6803', 'nonce': 1309426740}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6806', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6807', 'nonce': 1309426740}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6804', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6805', 'nonce': 1309426740}]}, 'public_addr': '172.21.15.122:6801/1309426740', 'cluster_addr': '172.21.15.122:6803/1309426740', 'heartbeat_back_addr': '172.21.15.122:6807/1309426740', 'heartbeat_front_addr': '172.21.15.122:6805/1309426740', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '9b20a4aa-032a-4c09-ac75-153c554424e7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 718, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6808', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6809', 'nonce': 2953210076}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6810', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6811', 'nonce': 2953210076}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6814', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6815', 'nonce': 2953210076}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6812', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6813', 'nonce': 2953210076}]}, 'public_addr': '172.21.15.122:6809/2953210076', 'cluster_addr': '172.21.15.122:6811/2953210076', 'heartbeat_back_addr': '172.21.15.122:6815/2953210076', 'heartbeat_front_addr': '172.21.15.122:6813/2953210076', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'e62c1fb3-5dd1-4d0d-997c-d20633d19361', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 89, 'up_thru': 718, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6816', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6817', 'nonce': 409394340}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6818', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6819', 'nonce': 409394340}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6822', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6823', 'nonce': 409394340}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6820', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6821', 'nonce': 409394340}]}, 'public_addr': '172.21.15.122:6817/409394340', 'cluster_addr': '172.21.15.122:6819/409394340', 'heartbeat_back_addr': '172.21.15.122:6823/409394340', 'heartbeat_front_addr': '172.21.15.122:6821/409394340', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'b83994d5-e5ef-48b0-9758-c26843cec3b6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 96, 'up_thru': 709, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6824', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6825', 'nonce': 2704069354}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6826', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6827', 'nonce': 2704069354}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6830', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6831', 'nonce': 2704069354}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6828', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6829', 'nonce': 2704069354}]}, 'public_addr': '172.21.15.122:6825/2704069354', 'cluster_addr': '172.21.15.122:6827/2704069354', 'heartbeat_back_addr': '172.21.15.122:6831/2704069354', 'heartbeat_front_addr': '172.21.15.122:6829/2704069354', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:24:03.669839+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:24:20.926175+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:24:52.112478+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:23:57.178292+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:24:01.493001+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:24:58.674822+0000', 'dead_epoch': 273}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:24:15.357950+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:24:50.165406+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:23:57.578904+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:24:25.102505+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:23:58.556278+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:24:10.379081+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.1', 'osds': [6, 11, 5]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.26:0/4291404191': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/642012566': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:6801/1181459999': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/3984513952': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6800/1654945498': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6800/1181459999': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/137770981': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:0/1451237762': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/3938386631': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6801/1654945498': '2024-03-21T08:49:04.596754+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-20T09:25:05.759 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:25:05.848 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:25:05.860 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 50 16 75202 75186 93.9711 91 0.00634639 0.0106374 2024-03-20T09:25:05.878 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:05.900 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:25:05.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:05.903 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:25:05.984 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:25:06.004 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:25:06.158 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:06.195 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:25:06.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:06.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:25:06.301 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:25:06.343 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:06.370 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:25:06.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:06.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:25:06.477 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:25:06.504 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:06.526 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:25:06.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:06.529 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:25:06.630 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:25:06.661 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:06.687 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:25:06.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:06.690 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:25:06.791 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:25:06.858 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:06.860 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 51 16 76471 76455 93.6836 79.3125 0.0033588 0.0106701 2024-03-20T09:25:06.897 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:25:06.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:06.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:25:07.017 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:25:07.046 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:07.069 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:25:07.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:07.072 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:25:07.173 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:25:07.225 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:07.243 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:25:07.252 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:25:07.272 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:25:07.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:07.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:25:07.382 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:25:07.425 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:07.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:07 smithi122 ceph-mon[73495]: pgmap v1784: 47 pgs: 47 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 106 MiB/s wr, 1.69k op/s 2024-03-20T09:25:07.463 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:25:07.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:07.469 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:07 smithi026 ceph-mon[68309]: pgmap v1784: 47 pgs: 47 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 106 MiB/s wr, 1.69k op/s 2024-03-20T09:25:07.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:25:07.571 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:25:07.607 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:07 smithi115 ceph-mon[72476]: pgmap v1784: 47 pgs: 47 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 106 MiB/s wr, 1.69k op/s 2024-03-20T09:25:07.611 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:07.643 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:25:07.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:07.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:25:07.749 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:25:07.806 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:07.860 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 52 16 77812 77796 93.4937 83.8125 0.00755494 0.0106923 2024-03-20T09:25:08.860 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 53 16 79281 79265 93.4618 91.8125 0.00355535 0.0106928 2024-03-20T09:25:08.980 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:25:08.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:08.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:25:09.087 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:25:09.133 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:09.162 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:25:09.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:09.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:25:09.269 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:25:09.301 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:09.324 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:25:09.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:09.327 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:25:09.403 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:09 smithi115 ceph-mon[72476]: pgmap v1785: 47 pgs: 47 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 99 MiB/s wr, 1.58k op/s 2024-03-20T09:25:09.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:09 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:25:09.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:09 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:25:09.428 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:25:09.486 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:09.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:09 smithi122 ceph-mon[73495]: pgmap v1785: 47 pgs: 47 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 99 MiB/s wr, 1.58k op/s 2024-03-20T09:25:09.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:09 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:25:09.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:09 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:25:09.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:09 smithi026 ceph-mon[68309]: pgmap v1785: 47 pgs: 47 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 99 MiB/s wr, 1.58k op/s 2024-03-20T09:25:09.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:09 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:25:09.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:09 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:25:09.861 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 54 16 80701 80685 93.3744 88.75 0.00726563 0.0107055 2024-03-20T09:25:10.623 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:10 smithi026 ceph-mon[68309]: osdmap e730: 12 total, 12 up, 12 in 2024-03-20T09:25:10.653 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:10 smithi115 ceph-mon[72476]: osdmap e730: 12 total, 12 up, 12 in 2024-03-20T09:25:10.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:10 smithi122 ceph-mon[73495]: osdmap e730: 12 total, 12 up, 12 in 2024-03-20T09:25:10.861 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 55 16 81980 81964 93.13 79.9375 0.0106488 0.010734 2024-03-20T09:25:11.301 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:25:11.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:11.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:25:11.409 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:25:11.461 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:11.484 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:25:11.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:11.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:25:11.590 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:25:11.650 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:25:11.653 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:11 smithi115 ceph-mon[72476]: pgmap v1787: 47 pgs: 47 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.51k op/s 2024-03-20T09:25:11.656 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:11 smithi026 ceph-mon[68309]: pgmap v1787: 47 pgs: 47 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.51k op/s 2024-03-20T09:25:11.669 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:11.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:11 smithi122 ceph-mon[73495]: pgmap v1787: 47 pgs: 47 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.51k op/s 2024-03-20T09:25:11.861 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 56 16 83328 83312 92.9713 84.25 0.0100383 0.0107516 2024-03-20T09:25:12.542 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:25:12.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:12.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:25:12.650 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:25:12.681 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:12.703 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:25:12.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:12.706 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:25:12.807 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:25:12.837 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:12.861 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 57 16 84609 84593 92.7447 80.0625 0.015496 0.0107784 2024-03-20T09:25:12.861 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:25:12.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:12.865 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:25:12.966 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:25:13.053 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:13.068 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:25:13.083 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:25:13.426 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:25:13.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:13.434 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:13 smithi026 ceph-mon[68309]: pgmap v1788: 47 pgs: 47 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-03-20T09:25:13.435 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:13 smithi026 ceph-mon[68309]: osdmap e731: 12 total, 12 up, 12 in 2024-03-20T09:25:13.435 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:13 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1768750629' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:25:13.435 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:13 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:25:13.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:25:13.435 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:25:13.537 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:25:13.611 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:13.653 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:13 smithi115 ceph-mon[72476]: pgmap v1788: 47 pgs: 47 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-03-20T09:25:13.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:13 smithi115 ceph-mon[72476]: osdmap e731: 12 total, 12 up, 12 in 2024-03-20T09:25:13.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:13 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1768750629' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:25:13.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:13 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:25:13.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:13 smithi122 ceph-mon[73495]: pgmap v1788: 47 pgs: 47 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-03-20T09:25:13.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:13 smithi122 ceph-mon[73495]: osdmap e731: 12 total, 12 up, 12 in 2024-03-20T09:25:13.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:13 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1768750629' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:25:13.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:13 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:25:13.861 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 58 16 85769 85753 92.3956 72.5 0.00873735 0.0108192 2024-03-20T09:25:14.542 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:14 smithi026 ceph-mon[68309]: from='client.26305 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:25:14.542 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:14 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:25:14.542 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:14 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:25:14.542 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:14 smithi026 ceph-mon[68309]: osdmap e732: 12 total, 12 up, 12 in 2024-03-20T09:25:14.542 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:14 smithi026 ceph-mon[68309]: 6.2 scrub starts 2024-03-20T09:25:14.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:14 smithi115 ceph-mon[72476]: from='client.26305 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:25:14.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:14 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:25:14.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:14 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:25:14.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:14 smithi115 ceph-mon[72476]: osdmap e732: 12 total, 12 up, 12 in 2024-03-20T09:25:14.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:14 smithi115 ceph-mon[72476]: 6.2 scrub starts 2024-03-20T09:25:14.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:14 smithi122 ceph-mon[73495]: from='client.26305 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:25:14.915 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:14 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:25:14.915 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:14 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:25:14.915 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:14 smithi122 ceph-mon[73495]: osdmap e732: 12 total, 12 up, 12 in 2024-03-20T09:25:14.915 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:14 smithi122 ceph-mon[73495]: 6.2 scrub starts 2024-03-20T09:25:14.917 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 59 16 86957 86941 92.088 74.25 0.0106125 0.0108545 2024-03-20T09:25:15.299 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":1789,"stamp":"2024-03-20T09:25:12.404205+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5274141216,"num_objects":80470,"num_object_clones":0,"num_object_copies":241410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80468,"num_write_kb":5149952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84716,"ondisk_log_size":84716,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":22747740,"kb_used_data":16534252,"kb_used_omap":198,"kb_used_meta":6213177,"kb_avail":1102194084,"statfs":{"total":1151940427776,"available":1128646742016,"internally_reserved":0,"allocated":16931074048,"data_stored":16924142901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":203677,"internal_metadata":6362293347},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,5,7,6,7],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":761528320,"num_objects":11620,"num_object_clones":0,"num_object_copies":34860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11620,"num_write_kb":743680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.230129"},"pg_stats":[{"pgid":"1.1e","version":"20'137","reported_seq":872,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-03-20T09:25:09.260085+0000","last_change":"2024-03-20T09:24:09.407025+0000","last_active":"2024-03-20T09:25:09.260085+0000","last_peered":"2024-03-20T09:25:09.260085+0000","last_clean":"2024-03-20T09:25:09.260085+0000","last_became_active":"2024-03-20T09:13:20.071655+0000","last_became_peered":"2024-03-20T09:13:20.071655+0000","last_unstale":"2024-03-20T09:25:09.260085+0000","last_undegraded":"2024-03-20T09:25:09.260085+0000","last_fullsized":"2024-03-20T09:25:09.260085+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:09.406870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:09.406870+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:30.492927+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'137","reported_seq":1045,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-03-20T09:25:09.264568+0000","last_change":"2024-03-20T09:24:12.250854+0000","last_active":"2024-03-20T09:25:09.264568+0000","last_peered":"2024-03-20T09:25:09.264568+0000","last_clean":"2024-03-20T09:25:09.264568+0000","last_became_active":"2024-03-20T09:07:42.013245+0000","last_became_peered":"2024-03-20T09:07:42.013245+0000","last_unstale":"2024-03-20T09:25:09.264568+0000","last_undegraded":"2024-03-20T09:25:09.264568+0000","last_fullsized":"2024-03-20T09:25:09.264568+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:12.250743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:12.250743+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:15.406212+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"20'137","reported_seq":1504,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-03-20T09:25:09.260472+0000","last_change":"2024-03-20T09:23:37.923720+0000","last_active":"2024-03-20T09:25:09.260472+0000","last_peered":"2024-03-20T09:25:09.260472+0000","last_clean":"2024-03-20T09:25:09.260472+0000","last_became_active":"2024-03-20T09:07:41.961622+0000","last_became_peered":"2024-03-20T09:07:41.961622+0000","last_unstale":"2024-03-20T09:25:09.260472+0000","last_undegraded":"2024-03-20T09:25:09.260472+0000","last_fullsized":"2024-03-20T09:25:09.260472+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:23:37.923573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:23:37.923573+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:14.030936+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.19","version":"20'137","reported_seq":1550,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-03-20T09:25:09.259322+0000","last_change":"2024-03-20T09:23:21.491579+0000","last_active":"2024-03-20T09:25:09.259322+0000","last_peered":"2024-03-20T09:25:09.259322+0000","last_clean":"2024-03-20T09:25:09.259322+0000","last_became_active":"2024-03-20T09:07:41.962680+0000","last_became_peered":"2024-03-20T09:07:41.962680+0000","last_unstale":"2024-03-20T09:25:09.259322+0000","last_undegraded":"2024-03-20T09:25:09.259322+0000","last_fullsized":"2024-03-20T09:25:09.259322+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:23:21.491452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:23:21.491452+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:13.656570+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"20'137","reported_seq":1010,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-03-20T09:25:02.403245+0000","last_change":"2024-03-20T09:24:10.164173+0000","last_active":"2024-03-20T09:25:02.403245+0000","last_peered":"2024-03-20T09:25:02.403245+0000","last_clean":"2024-03-20T09:25:02.403245+0000","last_became_active":"2024-03-20T09:10:35.277481+0000","last_became_peered":"2024-03-20T09:10:35.277481+0000","last_unstale":"2024-03-20T09:25:02.403245+0000","last_undegraded":"2024-03-20T09:25:02.403245+0000","last_fullsized":"2024-03-20T09:25:02.403245+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:10.164023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:10.164023+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:13.879087+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1b","version":"20'137","reported_seq":1350,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-03-20T09:25:09.260456+0000","last_change":"2024-03-20T09:24:25.023050+0000","last_active":"2024-03-20T09:25:09.260456+0000","last_peered":"2024-03-20T09:25:09.260456+0000","last_clean":"2024-03-20T09:25:09.260456+0000","last_became_active":"2024-03-20T09:24:16.445287+0000","last_became_peered":"2024-03-20T09:24:16.445287+0000","last_unstale":"2024-03-20T09:25:09.260456+0000","last_undegraded":"2024-03-20T09:25:09.260456+0000","last_fullsized":"2024-03-20T09:25:09.260456+0000","mapping_epoch":718,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:25.022894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:25.022894+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:28.182388+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"6.f","version":"729'4951","reported_seq":5000,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-03-20T09:25:08.229323+0000","last_change":"2024-03-20T09:24:04.030788+0000","last_active":"2024-03-20T09:25:08.229323+0000","last_peered":"2024-03-20T09:25:08.229323+0000","last_clean":"2024-03-20T09:25:08.229323+0000","last_became_active":"2024-03-20T09:24:04.030514+0000","last_became_peered":"2024-03-20T09:24:04.030514+0000","last_unstale":"2024-03-20T09:25:08.229323+0000","last_undegraded":"2024-03-20T09:25:08.229323+0000","last_fullsized":"2024-03-20T09:25:08.229323+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":4951,"log_dups_size":0,"ondisk_log_size":4951,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:10.180034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324468736,"num_objects":4951,"num_object_clones":0,"num_object_copies":14853,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4951,"num_write_kb":316864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"20'137","reported_seq":1009,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-03-20T09:25:02.403304+0000","last_change":"2024-03-20T09:24:16.103983+0000","last_active":"2024-03-20T09:25:02.403304+0000","last_peered":"2024-03-20T09:25:02.403304+0000","last_clean":"2024-03-20T09:25:02.403304+0000","last_became_active":"2024-03-20T09:10:35.215772+0000","last_became_peered":"2024-03-20T09:10:35.215772+0000","last_unstale":"2024-03-20T09:25:02.403304+0000","last_undegraded":"2024-03-20T09:25:02.403304+0000","last_fullsized":"2024-03-20T09:25:02.403304+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:16.103860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:16.103860+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:39.371790+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"730'5048","reported_seq":5100,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-03-20T09:25:11.008796+0000","last_change":"2024-03-20T09:24:03.382661+0000","last_active":"2024-03-20T09:25:11.008796+0000","last_peered":"2024-03-20T09:25:11.008796+0000","last_clean":"2024-03-20T09:25:11.008796+0000","last_became_active":"2024-03-20T09:24:03.382302+0000","last_became_peered":"2024-03-20T09:24:03.382302+0000","last_unstale":"2024-03-20T09:25:11.008796+0000","last_undegraded":"2024-03-20T09:25:11.008796+0000","last_fullsized":"2024-03-20T09:25:11.008796+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":5048,"log_dups_size":0,"ondisk_log_size":5048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330825728,"num_objects":5048,"num_object_clones":0,"num_object_copies":15144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5048,"num_write_kb":323072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"20'137","reported_seq":1585,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-03-20T09:25:02.410040+0000","last_change":"2024-03-20T09:23:14.062062+0000","last_active":"2024-03-20T09:25:02.410040+0000","last_peered":"2024-03-20T09:25:02.410040+0000","last_clean":"2024-03-20T09:25:02.410040+0000","last_became_active":"2024-03-20T09:20:25.341750+0000","last_became_peered":"2024-03-20T09:20:25.341750+0000","last_unstale":"2024-03-20T09:25:02.410040+0000","last_undegraded":"2024-03-20T09:25:02.410040+0000","last_fullsized":"2024-03-20T09:25:02.410040+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:23:14.061933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:23:14.061933+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,2],"acting":[9,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.d","version":"729'4888","reported_seq":4938,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-03-20T09:25:07.993320+0000","last_change":"2024-03-20T09:24:03.382043+0000","last_active":"2024-03-20T09:25:07.993320+0000","last_peered":"2024-03-20T09:25:07.993320+0000","last_clean":"2024-03-20T09:25:07.993320+0000","last_became_active":"2024-03-20T09:24:03.381679+0000","last_became_peered":"2024-03-20T09:24:03.381679+0000","last_unstale":"2024-03-20T09:25:07.993320+0000","last_undegraded":"2024-03-20T09:25:07.993320+0000","last_fullsized":"2024-03-20T09:25:07.993320+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":4888,"log_dups_size":0,"ondisk_log_size":4888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:08.923045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320339968,"num_objects":4888,"num_object_clones":0,"num_object_copies":14664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4888,"num_write_kb":312832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"20'137","reported_seq":282,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-03-20T09:25:02.408697+0000","last_change":"2024-03-20T09:24:17.164244+0000","last_active":"2024-03-20T09:25:02.408697+0000","last_peered":"2024-03-20T09:25:02.408697+0000","last_clean":"2024-03-20T09:25:02.408697+0000","last_became_active":"2024-03-20T09:24:16.444232+0000","last_became_peered":"2024-03-20T09:24:16.444232+0000","last_unstale":"2024-03-20T09:25:02.408697+0000","last_undegraded":"2024-03-20T09:25:02.408697+0000","last_fullsized":"2024-03-20T09:25:02.408697+0000","mapping_epoch":718,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:17.164131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:17.164131+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:45.596474+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.3","version":"730'5122","reported_seq":5175,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-03-20T09:25:11.093565+0000","last_change":"2024-03-20T09:24:37.949063+0000","last_active":"2024-03-20T09:25:11.093565+0000","last_peered":"2024-03-20T09:25:11.093565+0000","last_clean":"2024-03-20T09:25:11.093565+0000","last_became_active":"2024-03-20T09:24:03.379444+0000","last_became_peered":"2024-03-20T09:24:03.379444+0000","last_unstale":"2024-03-20T09:25:11.093565+0000","last_undegraded":"2024-03-20T09:25:11.093565+0000","last_fullsized":"2024-03-20T09:25:11.093565+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":5122,"log_dups_size":0,"ondisk_log_size":5122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:12.568809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335675392,"num_objects":5122,"num_object_clones":0,"num_object_copies":15366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5122,"num_write_kb":327808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,11],"acting":[6,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"20'137","reported_seq":1541,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-03-20T09:25:09.260091+0000","last_change":"2024-03-20T09:24:12.357887+0000","last_active":"2024-03-20T09:25:09.260091+0000","last_peered":"2024-03-20T09:25:09.260091+0000","last_clean":"2024-03-20T09:25:09.260091+0000","last_became_active":"2024-03-20T09:12:14.306384+0000","last_became_peered":"2024-03-20T09:12:14.306384+0000","last_unstale":"2024-03-20T09:25:09.260091+0000","last_undegraded":"2024-03-20T09:25:09.260091+0000","last_fullsized":"2024-03-20T09:25:09.260091+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:12.357798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:12.357798+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:33.220906+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"730'5160","reported_seq":5212,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-03-20T09:25:10.992244+0000","last_change":"2024-03-20T09:24:03.382618+0000","last_active":"2024-03-20T09:25:10.992244+0000","last_peered":"2024-03-20T09:25:10.992244+0000","last_clean":"2024-03-20T09:25:10.992244+0000","last_became_active":"2024-03-20T09:24:03.382365+0000","last_became_peered":"2024-03-20T09:24:03.382365+0000","last_unstale":"2024-03-20T09:25:10.992244+0000","last_undegraded":"2024-03-20T09:25:10.992244+0000","last_fullsized":"2024-03-20T09:25:10.992244+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":5160,"log_dups_size":0,"ondisk_log_size":5160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:12.805985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338165760,"num_objects":5160,"num_object_clones":0,"num_object_copies":15480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5160,"num_write_kb":330240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"20'137","reported_seq":1436,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-03-20T09:25:09.259301+0000","last_change":"2024-03-20T09:23:28.458409+0000","last_active":"2024-03-20T09:25:09.259301+0000","last_peered":"2024-03-20T09:25:09.259301+0000","last_clean":"2024-03-20T09:25:09.259301+0000","last_became_active":"2024-03-20T09:07:37.003722+0000","last_became_peered":"2024-03-20T09:07:37.003722+0000","last_unstale":"2024-03-20T09:25:09.259301+0000","last_undegraded":"2024-03-20T09:25:09.259301+0000","last_fullsized":"2024-03-20T09:25:09.259301+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:23:28.458329+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:19:03.805185+0000","last_clean_scrub_stamp":"2024-03-20T09:23:28.458329+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:12.633646+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"730'5169","reported_seq":5221,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-03-20T09:25:11.443108+0000","last_change":"2024-03-20T09:24:04.031214+0000","last_active":"2024-03-20T09:25:11.443108+0000","last_peered":"2024-03-20T09:25:11.443108+0000","last_clean":"2024-03-20T09:25:11.443108+0000","last_became_active":"2024-03-20T09:24:04.030836+0000","last_became_peered":"2024-03-20T09:24:04.030836+0000","last_unstale":"2024-03-20T09:25:11.443108+0000","last_undegraded":"2024-03-20T09:25:11.443108+0000","last_fullsized":"2024-03-20T09:25:11.443108+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":5169,"log_dups_size":0,"ondisk_log_size":5169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:18.782944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338755584,"num_objects":5169,"num_object_clones":0,"num_object_copies":15507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5169,"num_write_kb":330816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"20'137","reported_seq":1479,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-03-20T09:25:09.258075+0000","last_change":"2024-03-20T09:24:31.549717+0000","last_active":"2024-03-20T09:25:09.258075+0000","last_peered":"2024-03-20T09:25:09.258075+0000","last_clean":"2024-03-20T09:25:09.258075+0000","last_became_active":"2024-03-20T09:24:16.442503+0000","last_became_peered":"2024-03-20T09:24:16.442503+0000","last_unstale":"2024-03-20T09:25:09.258075+0000","last_undegraded":"2024-03-20T09:25:09.258075+0000","last_fullsized":"2024-03-20T09:25:09.258075+0000","mapping_epoch":718,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:31.549603+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:24:31.549603+0000","last_clean_scrub_stamp":"2024-03-20T09:24:31.549603+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:47.943434+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.4","version":"730'5054","reported_seq":5105,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-03-20T09:25:09.598695+0000","last_change":"2024-03-20T09:24:03.379935+0000","last_active":"2024-03-20T09:25:09.598695+0000","last_peered":"2024-03-20T09:25:09.598695+0000","last_clean":"2024-03-20T09:25:09.598695+0000","last_became_active":"2024-03-20T09:24:03.379675+0000","last_became_peered":"2024-03-20T09:24:03.379675+0000","last_unstale":"2024-03-20T09:25:09.598695+0000","last_undegraded":"2024-03-20T09:25:09.598695+0000","last_fullsized":"2024-03-20T09:25:09.598695+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":5054,"log_dups_size":0,"ondisk_log_size":5054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:11.152543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331218944,"num_objects":5054,"num_object_clones":0,"num_object_copies":15162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5054,"num_write_kb":323456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"20'137","reported_seq":1454,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-03-20T09:25:09.265234+0000","last_change":"2024-03-20T09:24:11.318300+0000","last_active":"2024-03-20T09:25:09.265234+0000","last_peered":"2024-03-20T09:25:09.265234+0000","last_clean":"2024-03-20T09:25:09.265234+0000","last_became_active":"2024-03-20T09:19:54.499210+0000","last_became_peered":"2024-03-20T09:19:54.499210+0000","last_unstale":"2024-03-20T09:25:09.265234+0000","last_undegraded":"2024-03-20T09:25:09.265234+0000","last_fullsized":"2024-03-20T09:25:09.265234+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:11.318184+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:18.788771+0000","last_clean_scrub_stamp":"2024-03-20T09:24:11.318184+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:14.024707+0000","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"729'4945","reported_seq":4995,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-03-20T09:25:07.970394+0000","last_change":"2024-03-20T09:24:03.380357+0000","last_active":"2024-03-20T09:25:07.970394+0000","last_peered":"2024-03-20T09:25:07.970394+0000","last_clean":"2024-03-20T09:25:07.970394+0000","last_became_active":"2024-03-20T09:24:03.380050+0000","last_became_peered":"2024-03-20T09:24:03.380050+0000","last_unstale":"2024-03-20T09:25:07.970394+0000","last_undegraded":"2024-03-20T09:25:07.970394+0000","last_fullsized":"2024-03-20T09:25:07.970394+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":4945,"log_dups_size":0,"ondisk_log_size":4945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:16.929930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324075520,"num_objects":4945,"num_object_clones":0,"num_object_copies":14835,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4945,"num_write_kb":316480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":1512,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-03-20T09:25:09.260016+0000","last_change":"2024-03-20T09:24:13.361385+0000","last_active":"2024-03-20T09:25:09.260016+0000","last_peered":"2024-03-20T09:25:09.260016+0000","last_clean":"2024-03-20T09:25:09.260016+0000","last_became_active":"2024-03-20T09:12:14.305989+0000","last_became_peered":"2024-03-20T09:12:14.305989+0000","last_unstale":"2024-03-20T09:25:09.260016+0000","last_undegraded":"2024-03-20T09:25:09.260016+0000","last_fullsized":"2024-03-20T09:25:09.260016+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:13.361279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:13.361279+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:24.465879+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.5","version":"730'5119","reported_seq":5170,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-03-20T09:25:11.093411+0000","last_change":"2024-03-20T09:24:04.031175+0000","last_active":"2024-03-20T09:25:11.093411+0000","last_peered":"2024-03-20T09:25:11.093411+0000","last_clean":"2024-03-20T09:25:11.093411+0000","last_became_active":"2024-03-20T09:24:04.030820+0000","last_became_peered":"2024-03-20T09:24:04.030820+0000","last_unstale":"2024-03-20T09:25:11.093411+0000","last_undegraded":"2024-03-20T09:25:11.093411+0000","last_fullsized":"2024-03-20T09:25:11.093411+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":5119,"log_dups_size":0,"ondisk_log_size":5119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:26.900507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335478784,"num_objects":5119,"num_object_clones":0,"num_object_copies":15357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5119,"num_write_kb":327616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"20'137","reported_seq":1766,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-03-20T09:25:02.399930+0000","last_change":"2024-03-20T09:24:13.945929+0000","last_active":"2024-03-20T09:25:02.399930+0000","last_peered":"2024-03-20T09:25:02.399930+0000","last_clean":"2024-03-20T09:25:02.399930+0000","last_became_active":"2024-03-20T09:19:54.492879+0000","last_became_peered":"2024-03-20T09:19:54.492879+0000","last_unstale":"2024-03-20T09:25:02.399930+0000","last_undegraded":"2024-03-20T09:25:02.399930+0000","last_fullsized":"2024-03-20T09:25:02.399930+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:13.945829+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:19.513511+0000","last_clean_scrub_stamp":"2024-03-20T09:24:13.945829+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:29.762487+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"730'5130","reported_seq":5180,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-03-20T09:25:09.921404+0000","last_change":"2024-03-20T09:24:03.378882+0000","last_active":"2024-03-20T09:25:09.921404+0000","last_peered":"2024-03-20T09:25:09.921404+0000","last_clean":"2024-03-20T09:25:09.921404+0000","last_became_active":"2024-03-20T09:24:03.378429+0000","last_became_peered":"2024-03-20T09:24:03.378429+0000","last_unstale":"2024-03-20T09:25:09.921404+0000","last_undegraded":"2024-03-20T09:25:09.921404+0000","last_fullsized":"2024-03-20T09:25:09.921404+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":5130,"log_dups_size":0,"ondisk_log_size":5130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:26.886749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336199680,"num_objects":5130,"num_object_clones":0,"num_object_copies":15390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5130,"num_write_kb":328320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"20'137","reported_seq":929,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-03-20T09:25:09.260558+0000","last_change":"2024-03-20T09:23:30.467662+0000","last_active":"2024-03-20T09:25:09.260558+0000","last_peered":"2024-03-20T09:25:09.260558+0000","last_clean":"2024-03-20T09:25:09.260558+0000","last_became_active":"2024-03-20T09:15:34.225022+0000","last_became_peered":"2024-03-20T09:15:34.225022+0000","last_unstale":"2024-03-20T09:25:09.260558+0000","last_undegraded":"2024-03-20T09:25:09.260558+0000","last_fullsized":"2024-03-20T09:25:09.260558+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:23:30.467535+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:20:12.169658+0000","last_clean_scrub_stamp":"2024-03-20T09:23:30.467535+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:12.919895+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,5],"acting":[6,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"729'4858","reported_seq":4905,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-03-20T09:25:08.816651+0000","last_change":"2024-03-20T09:24:03.379949+0000","last_active":"2024-03-20T09:25:08.816651+0000","last_peered":"2024-03-20T09:25:08.816651+0000","last_clean":"2024-03-20T09:25:08.816651+0000","last_became_active":"2024-03-20T09:24:03.379597+0000","last_became_peered":"2024-03-20T09:24:03.379597+0000","last_unstale":"2024-03-20T09:25:08.816651+0000","last_undegraded":"2024-03-20T09:25:08.816651+0000","last_fullsized":"2024-03-20T09:25:08.816651+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":4858,"log_dups_size":0,"ondisk_log_size":4858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:31.975923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318373888,"num_objects":4858,"num_object_clones":0,"num_object_copies":14574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4858,"num_write_kb":310912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"20'137","reported_seq":1464,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-03-20T09:25:09.260759+0000","last_change":"2024-03-20T09:23:25.611676+0000","last_active":"2024-03-20T09:25:09.260759+0000","last_peered":"2024-03-20T09:25:09.260759+0000","last_clean":"2024-03-20T09:25:09.260759+0000","last_became_active":"2024-03-20T09:13:27.087760+0000","last_became_peered":"2024-03-20T09:13:27.087760+0000","last_unstale":"2024-03-20T09:25:09.260759+0000","last_undegraded":"2024-03-20T09:25:09.260759+0000","last_fullsized":"2024-03-20T09:25:09.260759+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:23:25.611535+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:14:10.560890+0000","last_clean_scrub_stamp":"2024-03-20T09:23:25.611535+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:12.919875+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"730'5225","reported_seq":5276,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-03-20T09:25:09.924789+0000","last_change":"2024-03-20T09:24:03.381105+0000","last_active":"2024-03-20T09:25:09.924789+0000","last_peered":"2024-03-20T09:25:09.924789+0000","last_clean":"2024-03-20T09:25:09.924789+0000","last_became_active":"2024-03-20T09:24:03.380630+0000","last_became_peered":"2024-03-20T09:24:03.380630+0000","last_unstale":"2024-03-20T09:25:09.924789+0000","last_undegraded":"2024-03-20T09:25:09.924789+0000","last_fullsized":"2024-03-20T09:25:09.924789+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":5225,"log_dups_size":0,"ondisk_log_size":5225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:11.318851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342425600,"num_objects":5225,"num_object_clones":0,"num_object_copies":15675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5225,"num_write_kb":334400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"20'137","reported_seq":1317,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-03-20T09:25:09.260185+0000","last_change":"2024-03-20T09:24:10.480839+0000","last_active":"2024-03-20T09:25:09.260185+0000","last_peered":"2024-03-20T09:25:09.260185+0000","last_clean":"2024-03-20T09:25:09.260185+0000","last_became_active":"2024-03-20T09:13:20.070596+0000","last_became_peered":"2024-03-20T09:13:20.070596+0000","last_unstale":"2024-03-20T09:25:09.260185+0000","last_undegraded":"2024-03-20T09:25:09.260185+0000","last_fullsized":"2024-03-20T09:25:09.260185+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:10.480739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:10.480739+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:10.692605+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.9","version":"729'4979","reported_seq":5032,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-03-20T09:25:08.816247+0000","last_change":"2024-03-20T09:24:13.382970+0000","last_active":"2024-03-20T09:25:08.816247+0000","last_peered":"2024-03-20T09:25:08.816247+0000","last_clean":"2024-03-20T09:25:08.816247+0000","last_became_active":"2024-03-20T09:24:13.382746+0000","last_became_peered":"2024-03-20T09:24:13.382746+0000","last_unstale":"2024-03-20T09:25:08.816247+0000","last_undegraded":"2024-03-20T09:25:08.816247+0000","last_fullsized":"2024-03-20T09:25:08.816247+0000","mapping_epoch":715,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":716,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":4979,"log_dups_size":0,"ondisk_log_size":4979,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:24.455949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326303744,"num_objects":4979,"num_object_clones":0,"num_object_copies":14937,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4979,"num_write_kb":318656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"20'137","reported_seq":741,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-03-20T09:25:09.259524+0000","last_change":"2024-03-20T09:24:10.821907+0000","last_active":"2024-03-20T09:25:09.259524+0000","last_peered":"2024-03-20T09:25:09.259524+0000","last_clean":"2024-03-20T09:25:09.259524+0000","last_became_active":"2024-03-20T09:19:54.506134+0000","last_became_peered":"2024-03-20T09:19:54.506134+0000","last_unstale":"2024-03-20T09:25:09.259524+0000","last_undegraded":"2024-03-20T09:25:09.259524+0000","last_fullsized":"2024-03-20T09:25:09.259524+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:10.821728+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:18:52.573886+0000","last_clean_scrub_stamp":"2024-03-20T09:24:10.821728+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:23.388764+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"730'4987","reported_seq":5039,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-03-20T09:25:10.382386+0000","last_change":"2024-03-20T09:24:03.379403+0000","last_active":"2024-03-20T09:25:10.382386+0000","last_peered":"2024-03-20T09:25:10.382386+0000","last_clean":"2024-03-20T09:25:10.382386+0000","last_became_active":"2024-03-20T09:24:03.379073+0000","last_became_peered":"2024-03-20T09:24:03.379073+0000","last_unstale":"2024-03-20T09:25:10.382386+0000","last_undegraded":"2024-03-20T09:25:10.382386+0000","last_fullsized":"2024-03-20T09:25:10.382386+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":4987,"log_dups_size":0,"ondisk_log_size":4987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:21.840916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326828032,"num_objects":4987,"num_object_clones":0,"num_object_copies":14961,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4987,"num_write_kb":319168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,3],"acting":[11,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.f","version":"20'137","reported_seq":1489,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-03-20T09:25:09.260662+0000","last_change":"2024-03-20T09:23:28.597118+0000","last_active":"2024-03-20T09:25:09.260662+0000","last_peered":"2024-03-20T09:25:09.260662+0000","last_clean":"2024-03-20T09:25:09.260662+0000","last_became_active":"2024-03-20T09:19:54.524397+0000","last_became_peered":"2024-03-20T09:19:54.524397+0000","last_unstale":"2024-03-20T09:25:09.260662+0000","last_undegraded":"2024-03-20T09:25:09.260662+0000","last_fullsized":"2024-03-20T09:25:09.260662+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:23:28.597000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:23:28.597000+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:12.919855+0000","scrub_duration":154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"730'5055","reported_seq":5105,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-03-20T09:25:09.914604+0000","last_change":"2024-03-20T09:24:03.379324+0000","last_active":"2024-03-20T09:25:09.914604+0000","last_peered":"2024-03-20T09:25:09.914604+0000","last_clean":"2024-03-20T09:25:09.914604+0000","last_became_active":"2024-03-20T09:24:03.378961+0000","last_became_peered":"2024-03-20T09:24:03.378961+0000","last_unstale":"2024-03-20T09:25:09.914604+0000","last_undegraded":"2024-03-20T09:25:09.914604+0000","last_fullsized":"2024-03-20T09:25:09.914604+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":5055,"log_dups_size":0,"ondisk_log_size":5055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:30.598154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331284480,"num_objects":5055,"num_object_clones":0,"num_object_copies":15165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5055,"num_write_kb":323520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"570'138","reported_seq":1490,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-03-20T09:25:09.268136+0000","last_change":"2024-03-20T09:24:18.167726+0000","last_active":"2024-03-20T09:25:09.268136+0000","last_peered":"2024-03-20T09:25:09.268136+0000","last_clean":"2024-03-20T09:25:09.268136+0000","last_became_active":"2024-03-20T09:13:27.087686+0000","last_became_peered":"2024-03-20T09:13:27.087686+0000","last_unstale":"2024-03-20T09:25:09.268136+0000","last_undegraded":"2024-03-20T09:25:09.268136+0000","last_fullsized":"2024-03-20T09:25:09.268136+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"570'138","last_scrub_stamp":"2024-03-20T09:24:18.167603+0000","last_deep_scrub":"570'138","last_deep_scrub_stamp":"2024-03-20T09:16:13.725035+0000","last_clean_scrub_stamp":"2024-03-20T09:24:18.167603+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:45.806376+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"729'4778","reported_seq":4828,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-03-20T09:25:07.996757+0000","last_change":"2024-03-20T09:24:03.381846+0000","last_active":"2024-03-20T09:25:07.996757+0000","last_peered":"2024-03-20T09:25:07.996757+0000","last_clean":"2024-03-20T09:25:07.996757+0000","last_became_active":"2024-03-20T09:24:03.381499+0000","last_became_peered":"2024-03-20T09:24:03.381499+0000","last_unstale":"2024-03-20T09:25:07.996757+0000","last_undegraded":"2024-03-20T09:25:07.996757+0000","last_fullsized":"2024-03-20T09:25:07.996757+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:24:02.368833+0000","objects_scrubbed":0,"log_size":4778,"log_dups_size":0,"ondisk_log_size":4778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:28.474992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313131008,"num_objects":4778,"num_object_clones":0,"num_object_copies":14334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4778,"num_write_kb":305792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"20'137","reported_seq":1047,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-03-20T09:25:09.259409+0000","last_change":"2024-03-20T09:24:16.958522+0000","last_active":"2024-03-20T09:25:09.259409+0000","last_peered":"2024-03-20T09:25:09.259409+0000","last_clean":"2024-03-20T09:25:09.259409+0000","last_became_active":"2024-03-20T09:19:54.502529+0000","last_became_peered":"2024-03-20T09:19:54.502529+0000","last_unstale":"2024-03-20T09:25:09.259409+0000","last_undegraded":"2024-03-20T09:25:09.259409+0000","last_fullsized":"2024-03-20T09:25:09.259409+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:16.958439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:16.958439+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:31.241580+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"20'137","reported_seq":1350,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-03-20T09:25:02.403272+0000","last_change":"2024-03-20T09:24:13.190086+0000","last_active":"2024-03-20T09:25:02.403272+0000","last_peered":"2024-03-20T09:25:02.403272+0000","last_clean":"2024-03-20T09:25:02.403272+0000","last_became_active":"2024-03-20T09:10:34.096758+0000","last_became_peered":"2024-03-20T09:10:34.096758+0000","last_unstale":"2024-03-20T09:25:02.403272+0000","last_undegraded":"2024-03-20T09:25:02.403272+0000","last_fullsized":"2024-03-20T09:25:02.403272+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:13.189949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:13.189949+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:28.168111+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"20'137","reported_seq":1432,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-03-20T09:25:09.264902+0000","last_change":"2024-03-20T09:24:15.377223+0000","last_active":"2024-03-20T09:25:09.264902+0000","last_peered":"2024-03-20T09:25:09.264902+0000","last_clean":"2024-03-20T09:25:09.264902+0000","last_became_active":"2024-03-20T09:04:04.103645+0000","last_became_peered":"2024-03-20T09:04:04.103645+0000","last_unstale":"2024-03-20T09:25:09.264902+0000","last_undegraded":"2024-03-20T09:25:09.264902+0000","last_fullsized":"2024-03-20T09:25:09.264902+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:15.377106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:15.377106+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:23.936408+0000","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"20'137","reported_seq":884,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-03-20T09:25:02.403402+0000","last_change":"2024-03-20T09:24:17.124668+0000","last_active":"2024-03-20T09:25:02.403402+0000","last_peered":"2024-03-20T09:25:02.403402+0000","last_clean":"2024-03-20T09:25:02.403402+0000","last_became_active":"2024-03-20T09:10:34.097191+0000","last_became_peered":"2024-03-20T09:10:34.097191+0000","last_unstale":"2024-03-20T09:25:02.403402+0000","last_undegraded":"2024-03-20T09:25:02.403402+0000","last_fullsized":"2024-03-20T09:25:02.403402+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:17.124502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:17.124502+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:26.158088+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"20'137","reported_seq":1294,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-03-20T09:25:02.400138+0000","last_change":"2024-03-20T09:24:08.964670+0000","last_active":"2024-03-20T09:25:02.400138+0000","last_peered":"2024-03-20T09:25:02.400138+0000","last_clean":"2024-03-20T09:25:02.400138+0000","last_became_active":"2024-03-20T09:20:25.342982+0000","last_became_peered":"2024-03-20T09:20:25.342982+0000","last_unstale":"2024-03-20T09:25:02.400138+0000","last_undegraded":"2024-03-20T09:25:02.400138+0000","last_fullsized":"2024-03-20T09:25:02.400138+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:08.964577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:08.964577+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:09.518528+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"20'137","reported_seq":1120,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-03-20T09:25:09.266911+0000","last_change":"2024-03-20T09:24:19.259161+0000","last_active":"2024-03-20T09:25:09.266911+0000","last_peered":"2024-03-20T09:25:09.266911+0000","last_clean":"2024-03-20T09:25:09.266911+0000","last_became_active":"2024-03-20T09:13:27.087760+0000","last_became_peered":"2024-03-20T09:13:27.087760+0000","last_unstale":"2024-03-20T09:25:09.266911+0000","last_undegraded":"2024-03-20T09:25:09.266911+0000","last_fullsized":"2024-03-20T09:25:09.266911+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:19.259031+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:16:15.448920+0000","last_clean_scrub_stamp":"2024-03-20T09:24:19.259031+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:22.996346+0000","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"20'137","reported_seq":1447,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-03-20T09:25:09.260698+0000","last_change":"2024-03-20T09:23:34.995066+0000","last_active":"2024-03-20T09:25:09.260698+0000","last_peered":"2024-03-20T09:25:09.260698+0000","last_clean":"2024-03-20T09:25:09.260698+0000","last_became_active":"2024-03-20T09:10:35.095339+0000","last_became_peered":"2024-03-20T09:10:35.095339+0000","last_unstale":"2024-03-20T09:25:09.260698+0000","last_undegraded":"2024-03-20T09:25:09.260698+0000","last_fullsized":"2024-03-20T09:25:09.260698+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:23:34.994901+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:23.640533+0000","last_clean_scrub_stamp":"2024-03-20T09:23:34.994901+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:14.030973+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"20'137","reported_seq":1530,"reported_epoch":729,"state":"active+clean","last_fresh":"2024-03-20T09:25:02.397381+0000","last_change":"2024-03-20T09:24:31.104958+0000","last_active":"2024-03-20T09:25:02.397381+0000","last_peered":"2024-03-20T09:25:02.397381+0000","last_clean":"2024-03-20T09:25:02.397381+0000","last_became_active":"2024-03-20T09:20:25.349082+0000","last_became_peered":"2024-03-20T09:20:25.349082+0000","last_unstale":"2024-03-20T09:25:02.397381+0000","last_undegraded":"2024-03-20T09:25:02.397381+0000","last_fullsized":"2024-03-20T09:25:02.397381+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:31.104823+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:16:15.754362+0000","last_clean_scrub_stamp":"2024-03-20T09:24:31.104823+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:45.582333+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"20'137","reported_seq":780,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-03-20T09:25:09.261332+0000","last_change":"2024-03-20T09:23:41.251626+0000","last_active":"2024-03-20T09:25:09.261332+0000","last_peered":"2024-03-20T09:25:09.261332+0000","last_clean":"2024-03-20T09:25:09.261332+0000","last_became_active":"2024-03-20T09:12:14.307581+0000","last_became_peered":"2024-03-20T09:12:14.307581+0000","last_unstale":"2024-03-20T09:25:09.261332+0000","last_undegraded":"2024-03-20T09:25:09.261332+0000","last_fullsized":"2024-03-20T09:25:09.261332+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:23:41.251546+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:23:41.251546+0000","last_clean_scrub_stamp":"2024-03-20T09:23:41.251546+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"20'137","reported_seq":1047,"reported_epoch":730,"state":"active+clean","last_fresh":"2024-03-20T09:25:09.259953+0000","last_change":"2024-03-20T09:24:15.799821+0000","last_active":"2024-03-20T09:25:09.259953+0000","last_peered":"2024-03-20T09:25:09.259953+0000","last_clean":"2024-03-20T09:25:09.259953+0000","last_became_active":"2024-03-20T09:19:54.502391+0000","last_became_peered":"2024-03-20T09:19:54.502391+0000","last_unstale":"2024-03-20T09:25:09.259953+0000","last_undegraded":"2024-03-20T09:25:09.259953+0000","last_fullsized":"2024-03-20T09:25:09.259953+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:15.799637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:15.799637+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:30.958859+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":5273550848,"num_objects":80468,"num_object_clones":0,"num_object_copies":241404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80468,"num_write_kb":5149952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15851323392,"data_stored":15851323392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80468,"ondisk_log_size":80468,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4248,"ondisk_log_size":4248,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860766,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1282136,"kb_used_data":641484,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":92463016,"statfs":{"total":95995035648,"available":94682128384,"internally_reserved":0,"allocated":656879616,"data_stored":656301048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":656013764},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":89,"seq":382252089700,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1471124,"kb_used_data":984588,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":92274028,"statfs":{"total":95995035648,"available":94488604672,"internally_reserved":0,"allocated":1008218112,"data_stored":1007639544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":498203072},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":80,"seq":343597384043,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1491528,"kb_used_data":1239616,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":92253624,"statfs":{"total":95995035648,"available":94467710976,"internally_reserved":0,"allocated":1269366784,"data_stored":1268792681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":257948096},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":70,"seq":300647711087,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1029268,"kb_used_data":659340,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":92715884,"statfs":{"total":95995035648,"available":94941065216,"internally_reserved":0,"allocated":675164160,"data_stored":674585592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":378796486},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":60,"seq":257698038132,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2288160,"kb_used_data":1621140,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":91456992,"statfs":{"total":95995035648,"available":93651959808,"internally_reserved":0,"allocated":1660047360,"data_stored":1659465240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":683014602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332473,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1851348,"kb_used_data":1314828,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":91893804,"statfs":{"total":95995035648,"available":94099255296,"internally_reserved":0,"allocated":1346383872,"data_stored":1345805304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":549386694},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575251,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2156360,"kb_used_data":1503232,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":91588792,"statfs":{"total":95995035648,"available":93786923008,"internally_reserved":0,"allocated":1539309568,"data_stored":1538735465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":668793286},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738769,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1842632,"kb_used_data":1189504,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":91902520,"statfs":{"total":95995035648,"available":94108180480,"internally_reserved":0,"allocated":1218052096,"data_stored":1217477993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411724,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2020112,"kb_used_data":1596040,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":91725040,"statfs":{"total":95995035648,"available":93926440960,"internally_reserved":0,"allocated":1634344960,"data_stored":1633767305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150088,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3168664,"kb_used_data":2573836,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":90576488,"statfs":{"total":95995035648,"available":92750323712,"internally_reserved":0,"allocated":2635608064,"data_stored":2635029496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":609089991},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953862,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1796632,"kb_used_data":1310092,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":91948520,"statfs":{"total":95995035648,"available":94155284480,"internally_reserved":0,"allocated":1341534208,"data_stored":1340955640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":498203077},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":443,"seq":1902670512304,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2349776,"kb_used_data":1900552,"kb_used_omap":181,"kb_used_meta":449034,"kb_avail":91395376,"statfs":{"total":95995035648,"available":93588865024,"internally_reserved":0,"allocated":1946165248,"data_stored":1945587593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":186131,"internal_metadata":459811053},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":962658304,"data_stored":962658304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1311571968,"data_stored":1311571968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1276575744,"data_stored":1276575744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2660958208,"data_stored":2660958208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1351221248,"data_stored":1351221248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1608187904,"data_stored":1608187904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1349320704,"data_stored":1349320704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1688207360,"data_stored":1688207360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":678297600,"data_stored":678297600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1284964352,"data_stored":1284964352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1020723200,"data_stored":1020723200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":658636800,"data_stored":658636800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:25:15.302 INFO:tasks.thrashosds.thrasher:Setting 1.1c to [0, 9, 6, 2, 5, 7] 2024-03-20T09:25:15.302 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '1.1c', '0', '9', '6', '2', '5', '7'] 2024-03-20T09:25:15.302 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pg-upmap-items 1.1c 0 9 6 2 5 7 2024-03-20T09:25:15.393 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:25:15.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:15.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:25:15.501 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:25:15.535 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:15 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:25:15.536 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:15 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:25:15.536 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:15 smithi115 ceph-mon[72476]: 1.f scrub starts 2024-03-20T09:25:15.536 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:15 smithi115 ceph-mon[72476]: 1.1c scrub starts 2024-03-20T09:25:15.536 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:15 smithi115 ceph-mon[72476]: 1.f scrub ok 2024-03-20T09:25:15.536 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:15 smithi115 ceph-mon[72476]: 1.1c scrub ok 2024-03-20T09:25:15.536 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:15 smithi115 ceph-mon[72476]: pgmap v1791: 47 pgs: 47 active+clean; 5.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s 2024-03-20T09:25:15.536 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:15 smithi115 ceph-mon[72476]: 1.9 scrub starts 2024-03-20T09:25:15.536 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:15 smithi115 ceph-mon[72476]: 1.9 scrub ok 2024-03-20T09:25:15.536 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:15 smithi115 ceph-mon[72476]: 1.15 scrub starts 2024-03-20T09:25:15.536 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:15 smithi115 ceph-mon[72476]: osdmap e733: 12 total, 12 up, 12 in 2024-03-20T09:25:15.536 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:15 smithi115 ceph-mon[72476]: 6.c scrub starts 2024-03-20T09:25:15.537 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:15 smithi115 ceph-mon[72476]: 6.2 scrub ok 2024-03-20T09:25:15.538 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:15.564 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:25:15.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:15.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:25:15.669 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:25:15.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:15 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:25:15.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:15 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:25:15.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:15 smithi122 ceph-mon[73495]: 1.f scrub starts 2024-03-20T09:25:15.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:15 smithi122 ceph-mon[73495]: 1.1c scrub starts 2024-03-20T09:25:15.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:15 smithi122 ceph-mon[73495]: 1.f scrub ok 2024-03-20T09:25:15.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:15 smithi122 ceph-mon[73495]: 1.1c scrub ok 2024-03-20T09:25:15.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:15 smithi122 ceph-mon[73495]: pgmap v1791: 47 pgs: 47 active+clean; 5.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s 2024-03-20T09:25:15.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:15 smithi122 ceph-mon[73495]: 1.9 scrub starts 2024-03-20T09:25:15.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:15 smithi122 ceph-mon[73495]: 1.9 scrub ok 2024-03-20T09:25:15.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:15 smithi122 ceph-mon[73495]: 1.15 scrub starts 2024-03-20T09:25:15.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:15 smithi122 ceph-mon[73495]: osdmap e733: 12 total, 12 up, 12 in 2024-03-20T09:25:15.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:15 smithi122 ceph-mon[73495]: 6.c scrub starts 2024-03-20T09:25:15.695 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:15 smithi122 ceph-mon[73495]: 6.2 scrub ok 2024-03-20T09:25:15.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:15 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:25:15.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:15 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:25:15.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:15 smithi026 ceph-mon[68309]: 1.f scrub starts 2024-03-20T09:25:15.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:15 smithi026 ceph-mon[68309]: 1.1c scrub starts 2024-03-20T09:25:15.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:15 smithi026 ceph-mon[68309]: 1.f scrub ok 2024-03-20T09:25:15.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:15 smithi026 ceph-mon[68309]: 1.1c scrub ok 2024-03-20T09:25:15.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:15 smithi026 ceph-mon[68309]: pgmap v1791: 47 pgs: 47 active+clean; 5.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s 2024-03-20T09:25:15.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:15 smithi026 ceph-mon[68309]: 1.9 scrub starts 2024-03-20T09:25:15.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:15 smithi026 ceph-mon[68309]: 1.9 scrub ok 2024-03-20T09:25:15.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:15 smithi026 ceph-mon[68309]: 1.15 scrub starts 2024-03-20T09:25:15.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:15 smithi026 ceph-mon[68309]: osdmap e733: 12 total, 12 up, 12 in 2024-03-20T09:25:15.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:15 smithi026 ceph-mon[68309]: 6.c scrub starts 2024-03-20T09:25:15.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:15 smithi026 ceph-mon[68309]: 6.2 scrub ok 2024-03-20T09:25:15.713 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:15.785 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:25:15.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:15.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:25:15.861 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:2024-03-20T09:25:15.864368+0000 min lat: 0.00104258 max lat: 11.7007 avg lat: 0.0108924 2024-03-20T09:25:15.861 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T09:25:15.861 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 60 16 88124 88108 91.7687 72.9375 0.00540271 0.0108924 2024-03-20T09:25:15.894 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:25:15.945 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:16.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:16 smithi115 ceph-mon[72476]: 1.15 scrub ok 2024-03-20T09:25:16.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:16 smithi115 ceph-mon[72476]: pgmap v1793: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-03-20T09:25:16.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:16 smithi122 ceph-mon[73495]: 1.15 scrub ok 2024-03-20T09:25:16.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:16 smithi122 ceph-mon[73495]: pgmap v1793: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-03-20T09:25:16.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:16 smithi026 ceph-mon[68309]: 1.15 scrub ok 2024-03-20T09:25:16.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:16 smithi026 ceph-mon[68309]: pgmap v1793: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s wr, 1.41k op/s 2024-03-20T09:25:16.861 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 61 16 89446 89430 91.6187 82.625 0.00715377 0.0109109 2024-03-20T09:25:16.919 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:25:17.022 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:25:17.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:17.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:25:17.128 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:25:17.157 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:17.178 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:25:17.180 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:17.181 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:25:17.282 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:25:17.341 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:17.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:17 smithi122 ceph-mon[73495]: 1.19 scrub starts 2024-03-20T09:25:17.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:17 smithi122 ceph-mon[73495]: 1.11 scrub starts 2024-03-20T09:25:17.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:17 smithi122 ceph-mon[73495]: 1.19 scrub ok 2024-03-20T09:25:17.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:17 smithi122 ceph-mon[73495]: 1.11 scrub ok 2024-03-20T09:25:17.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:17 smithi122 ceph-mon[73495]: 6.c scrub ok 2024-03-20T09:25:17.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:17 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:25:17.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:17 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:25:17.653 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:17 smithi115 ceph-mon[72476]: 1.19 scrub starts 2024-03-20T09:25:17.653 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:17 smithi115 ceph-mon[72476]: 1.11 scrub starts 2024-03-20T09:25:17.653 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:17 smithi115 ceph-mon[72476]: 1.19 scrub ok 2024-03-20T09:25:17.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:17 smithi115 ceph-mon[72476]: 1.11 scrub ok 2024-03-20T09:25:17.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:17 smithi115 ceph-mon[72476]: 6.c scrub ok 2024-03-20T09:25:17.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:17 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:25:17.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:17 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:25:17.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:17 smithi026 ceph-mon[68309]: 1.19 scrub starts 2024-03-20T09:25:17.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:17 smithi026 ceph-mon[68309]: 1.11 scrub starts 2024-03-20T09:25:17.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:17 smithi026 ceph-mon[68309]: 1.19 scrub ok 2024-03-20T09:25:17.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:17 smithi026 ceph-mon[68309]: 1.11 scrub ok 2024-03-20T09:25:17.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:17 smithi026 ceph-mon[68309]: 6.c scrub ok 2024-03-20T09:25:17.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:17 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:25:17.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:17 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:25:17.861 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 62 16 90636 90620 91.3405 74.375 0.0133036 0.0109441 2024-03-20T09:25:18.512 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:25:18.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:18.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:25:18.621 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:25:18.662 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:18 smithi115 ceph-mon[72476]: 1.7 scrub starts 2024-03-20T09:25:18.662 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:18 smithi115 ceph-mon[72476]: 1.b scrub starts 2024-03-20T09:25:18.662 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:18 smithi115 ceph-mon[72476]: 1.7 scrub ok 2024-03-20T09:25:18.662 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:18 smithi115 ceph-mon[72476]: 1.3 scrub starts 2024-03-20T09:25:18.662 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:18 smithi115 ceph-mon[72476]: 1.b scrub ok 2024-03-20T09:25:18.662 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:18 smithi115 ceph-mon[72476]: 1.3 scrub ok 2024-03-20T09:25:18.662 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:18 smithi115 ceph-mon[72476]: osdmap e734: 12 total, 12 up, 12 in 2024-03-20T09:25:18.662 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:18 smithi115 ceph-mon[72476]: osdmap e735: 12 total, 12 up, 12 in 2024-03-20T09:25:18.662 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:18 smithi115 ceph-mon[72476]: pgmap v1796: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-03-20T09:25:18.669 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:18 smithi026 ceph-mon[68309]: 1.7 scrub starts 2024-03-20T09:25:18.670 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:18 smithi026 ceph-mon[68309]: 1.b scrub starts 2024-03-20T09:25:18.670 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:18 smithi026 ceph-mon[68309]: 1.7 scrub ok 2024-03-20T09:25:18.670 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:18 smithi026 ceph-mon[68309]: 1.3 scrub starts 2024-03-20T09:25:18.670 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:18 smithi026 ceph-mon[68309]: 1.b scrub ok 2024-03-20T09:25:18.670 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:18 smithi026 ceph-mon[68309]: 1.3 scrub ok 2024-03-20T09:25:18.670 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:18 smithi026 ceph-mon[68309]: osdmap e734: 12 total, 12 up, 12 in 2024-03-20T09:25:18.670 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:18 smithi026 ceph-mon[68309]: osdmap e735: 12 total, 12 up, 12 in 2024-03-20T09:25:18.670 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:18 smithi026 ceph-mon[68309]: pgmap v1796: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-03-20T09:25:18.671 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:18.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:18 smithi122 ceph-mon[73495]: 1.7 scrub starts 2024-03-20T09:25:18.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:18 smithi122 ceph-mon[73495]: 1.b scrub starts 2024-03-20T09:25:18.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:18 smithi122 ceph-mon[73495]: 1.7 scrub ok 2024-03-20T09:25:18.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:18 smithi122 ceph-mon[73495]: 1.3 scrub starts 2024-03-20T09:25:18.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:18 smithi122 ceph-mon[73495]: 1.b scrub ok 2024-03-20T09:25:18.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:18 smithi122 ceph-mon[73495]: 1.3 scrub ok 2024-03-20T09:25:18.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:18 smithi122 ceph-mon[73495]: osdmap e734: 12 total, 12 up, 12 in 2024-03-20T09:25:18.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:18 smithi122 ceph-mon[73495]: osdmap e735: 12 total, 12 up, 12 in 2024-03-20T09:25:18.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:18 smithi122 ceph-mon[73495]: pgmap v1796: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-03-20T09:25:18.861 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 63 16 91844 91828 91.089 75.5 0.0100304 0.0109738 2024-03-20T09:25:19.553 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:25:19.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:19.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:25:19.653 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:25:19.653 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-03-20T09:25:19.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-03-20T09:25:19.653 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:25:19.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.653 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25690.0:126654 4.4 4:223a64ab:::benchmark_data_smithi122_145698_object5735:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e558)", 2024-03-20T09:25:19.653 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:15:19.908168+0000", 2024-03-20T09:25:19.653 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 599.74372308299996, 2024-03-20T09:25:19.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.24208463599999999, 2024-03-20T09:25:19.654 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:25:19.654 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:25:19.654 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:25:19.654 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25690", 2024-03-20T09:25:19.654 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:25:19.654 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 126654 2024-03-20T09:25:19.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.654 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:25:19.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:25:19.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.908168+0000", 2024-03-20T09:25:19.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:25:19.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.908168+0000", 2024-03-20T09:25:19.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:25:19.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.908169+0000", 2024-03-20T09:25:19.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8100000000000003e-07 2024-03-20T09:25:19.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:25:19.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.908172+0000", 2024-03-20T09:25:19.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8899999999999999e-06 2024-03-20T09:25:19.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:25:19.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.908180+0000", 2024-03-20T09:25:19.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7639999999999994e-06 2024-03-20T09:25:19.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:25:19.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:19.908187+0000", 2024-03-20T09:25:19.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1129999999999997e-06 2024-03-20T09:25:19.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:25:19.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:20.030969+0000", 2024-03-20T09:25:19.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.122782668 2024-03-20T09:25:19.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:25:19.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:20.031094+0000", 2024-03-20T09:25:19.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012471300000000001 2024-03-20T09:25:19.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2,3", 2024-03-20T09:25:19.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:20.031230+0000", 2024-03-20T09:25:19.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000136093 2024-03-20T09:25:19.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.658 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:25:19.658 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:20.056493+0000", 2024-03-20T09:25:19.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.025263418999999999 2024-03-20T09:25:19.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.658 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:25:19.658 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:20.124310+0000", 2024-03-20T09:25:19.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.067816321999999998 2024-03-20T09:25:19.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.658 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:25:19.658 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:20.150180+0000", 2024-03-20T09:25:19.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.025870305 2024-03-20T09:25:19.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:25:19.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:20.150220+0000", 2024-03-20T09:25:19.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0082e-05 2024-03-20T09:25:19.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:25:19.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:15:20.150253+0000", 2024-03-20T09:25:19.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2685999999999998e-05 2024-03-20T09:25:19.659 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.659 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:19.659 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.660 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25690.0:193276 4.d 4:be441e64:::benchmark_data_smithi122_145698_object72357:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e577)", 2024-03-20T09:25:19.660 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:16:47.259402+0000", 2024-03-20T09:25:19.660 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 512.39248927799997, 2024-03-20T09:25:19.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.22729255800000001, 2024-03-20T09:25:19.660 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:25:19.660 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:25:19.660 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:25:19.660 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25690", 2024-03-20T09:25:19.660 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:25:19.660 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 193276 2024-03-20T09:25:19.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.660 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:25:19.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:25:19.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.259402+0000", 2024-03-20T09:25:19.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:25:19.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.259402+0000", 2024-03-20T09:25:19.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:25:19.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.259403+0000", 2024-03-20T09:25:19.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.850000000000001e-07 2024-03-20T09:25:19.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:25:19.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.259406+0000", 2024-03-20T09:25:19.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4910000000000001e-06 2024-03-20T09:25:19.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:25:19.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.259415+0000", 2024-03-20T09:25:19.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1969999999999996e-06 2024-03-20T09:25:19.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:25:19.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.259419+0000", 2024-03-20T09:25:19.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4599999999999996e-06 2024-03-20T09:25:19.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:25:19.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.259532+0000", 2024-03-20T09:25:19.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011336999999999999 2024-03-20T09:25:19.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:25:19.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.259621+0000", 2024-03-20T09:25:19.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8494999999999999e-05 2024-03-20T09:25:19.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:25:19.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.259690+0000", 2024-03-20T09:25:19.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9060000000000006e-05 2024-03-20T09:25:19.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:25:19.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.263563+0000", 2024-03-20T09:25:19.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0038733019999999999 2024-03-20T09:25:19.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:25:19.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.271366+0000", 2024-03-20T09:25:19.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0078022710000000004 2024-03-20T09:25:19.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:25:19.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.486575+0000", 2024-03-20T09:25:19.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.21520989600000001 2024-03-20T09:25:19.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:25:19.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.486642+0000", 2024-03-20T09:25:19.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6699000000000006e-05 2024-03-20T09:25:19.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:25:19.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.486695+0000", 2024-03-20T09:25:19.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2332e-05 2024-03-20T09:25:19.665 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.665 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:19.665 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.666 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25690.0:193278 4.7 4:e3c89a70:::benchmark_data_smithi122_145698_object72359:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e577)", 2024-03-20T09:25:19.666 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:16:47.259432+0000", 2024-03-20T09:25:19.666 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 512.39245946699998, 2024-03-20T09:25:19.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.22728831699999999, 2024-03-20T09:25:19.666 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:25:19.666 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:25:19.666 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:25:19.666 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25690", 2024-03-20T09:25:19.666 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:25:19.666 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 193278 2024-03-20T09:25:19.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.666 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:25:19.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:25:19.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.259432+0000", 2024-03-20T09:25:19.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:25:19.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.259432+0000", 2024-03-20T09:25:19.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:25:19.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.259432+0000", 2024-03-20T09:25:19.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9999999999999999e-07 2024-03-20T09:25:19.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:25:19.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.259434+0000", 2024-03-20T09:25:19.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.668e-06 2024-03-20T09:25:19.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:25:19.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.259438+0000", 2024-03-20T09:25:19.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8210000000000003e-06 2024-03-20T09:25:19.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:25:19.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.259441+0000", 2024-03-20T09:25:19.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.303e-06 2024-03-20T09:25:19.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:25:19.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.259484+0000", 2024-03-20T09:25:19.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3477999999999999e-05 2024-03-20T09:25:19.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:25:19.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.259611+0000", 2024-03-20T09:25:19.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012649100000000001 2024-03-20T09:25:19.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:25:19.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.260317+0000", 2024-03-20T09:25:19.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00070576000000000005 2024-03-20T09:25:19.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:25:19.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.270908+0000", 2024-03-20T09:25:19.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010591402999999999 2024-03-20T09:25:19.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:25:19.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.270920+0000", 2024-03-20T09:25:19.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2152e-05 2024-03-20T09:25:19.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:25:19.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.486618+0000", 2024-03-20T09:25:19.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.215697474 2024-03-20T09:25:19.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:25:19.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.486673+0000", 2024-03-20T09:25:19.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5232000000000002e-05 2024-03-20T09:25:19.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:25:19.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.486720+0000", 2024-03-20T09:25:19.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7234999999999998e-05 2024-03-20T09:25:19.671 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.671 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:19.671 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.672 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25690.0:193285 4.7 4:e21ffae8:::benchmark_data_smithi122_145698_object72366:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e577)", 2024-03-20T09:25:19.672 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:16:47.271303+0000", 2024-03-20T09:25:19.672 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 512.38058796200005, 2024-03-20T09:25:19.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.22892721099999999, 2024-03-20T09:25:19.672 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:25:19.672 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:25:19.672 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:25:19.672 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25690", 2024-03-20T09:25:19.672 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/4165420082", 2024-03-20T09:25:19.672 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 193285 2024-03-20T09:25:19.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.672 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:25:19.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:25:19.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.271303+0000", 2024-03-20T09:25:19.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:25:19.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.271303+0000", 2024-03-20T09:25:19.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:25:19.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.271304+0000", 2024-03-20T09:25:19.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.01e-07 2024-03-20T09:25:19.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:25:19.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.271306+0000", 2024-03-20T09:25:19.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0930000000000001e-06 2024-03-20T09:25:19.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:25:19.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.271311+0000", 2024-03-20T09:25:19.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5260000000000001e-06 2024-03-20T09:25:19.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:25:19.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.271318+0000", 2024-03-20T09:25:19.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2999999999999998e-06 2024-03-20T09:25:19.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:25:19.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.277243+0000", 2024-03-20T09:25:19.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059252139999999998 2024-03-20T09:25:19.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:25:19.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.277371+0000", 2024-03-20T09:25:19.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012820900000000001 2024-03-20T09:25:19.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,10", 2024-03-20T09:25:19.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.277489+0000", 2024-03-20T09:25:19.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000118274 2024-03-20T09:25:19.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:25:19.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.282092+0000", 2024-03-20T09:25:19.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0046029240000000004 2024-03-20T09:25:19.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:25:19.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.288997+0000", 2024-03-20T09:25:19.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0069050020000000004 2024-03-20T09:25:19.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:25:19.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.500116+0000", 2024-03-20T09:25:19.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.21111924300000001 2024-03-20T09:25:19.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:25:19.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.500174+0000", 2024-03-20T09:25:19.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7049000000000002e-05 2024-03-20T09:25:19.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:25:19.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:16:47.500230+0000", 2024-03-20T09:25:19.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6975999999999998e-05 2024-03-20T09:25:19.677 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.677 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:19.677 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.677 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:24818 5.c 5:33dfc16c:::benchmark_data_smithi122_157894_object24817:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e614)", 2024-03-20T09:25:19.677 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:18:29.504995+0000", 2024-03-20T09:25:19.678 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 410.14689595099998, 2024-03-20T09:25:19.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.200133061, 2024-03-20T09:25:19.678 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:25:19.678 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:25:19.678 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:25:19.678 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:25:19.678 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:25:19.678 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 24818 2024-03-20T09:25:19.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.678 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:25:19.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:25:19.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.504995+0000", 2024-03-20T09:25:19.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:25:19.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.504995+0000", 2024-03-20T09:25:19.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:25:19.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.504997+0000", 2024-03-20T09:25:19.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3859999999999999e-06 2024-03-20T09:25:19.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:25:19.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.505060+0000", 2024-03-20T09:25:19.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3336999999999993e-05 2024-03-20T09:25:19.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:25:19.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.505076+0000", 2024-03-20T09:25:19.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5803000000000001e-05 2024-03-20T09:25:19.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:25:19.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.505083+0000", 2024-03-20T09:25:19.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4000000000000003e-06 2024-03-20T09:25:19.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:25:19.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.505140+0000", 2024-03-20T09:25:19.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6399e-05 2024-03-20T09:25:19.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:25:19.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.505306+0000", 2024-03-20T09:25:19.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00016615600000000001 2024-03-20T09:25:19.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,6", 2024-03-20T09:25:19.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.505500+0000", 2024-03-20T09:25:19.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000194549 2024-03-20T09:25:19.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:25:19.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.506968+0000", 2024-03-20T09:25:19.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014676800000000001 2024-03-20T09:25:19.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:25:19.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.507039+0000", 2024-03-20T09:25:19.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0671000000000002e-05 2024-03-20T09:25:19.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:25:19.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.704989+0000", 2024-03-20T09:25:19.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.19795005700000001 2024-03-20T09:25:19.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:25:19.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.705070+0000", 2024-03-20T09:25:19.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1539000000000007e-05 2024-03-20T09:25:19.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:25:19.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.705128+0000", 2024-03-20T09:25:19.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8084000000000002e-05 2024-03-20T09:25:19.683 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.683 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:19.683 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.684 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:24822 5.c 5:3d14bba3:::benchmark_data_smithi122_157894_object24821:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e614)", 2024-03-20T09:25:19.684 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:18:29.507337+0000", 2024-03-20T09:25:19.684 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 410.14455443700001, 2024-03-20T09:25:19.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.19799285699999999, 2024-03-20T09:25:19.684 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:25:19.684 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:25:19.684 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:25:19.684 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:25:19.684 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:25:19.684 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 24822 2024-03-20T09:25:19.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.684 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:25:19.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:25:19.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.507337+0000", 2024-03-20T09:25:19.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:25:19.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.507337+0000", 2024-03-20T09:25:19.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:25:19.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.507338+0000", 2024-03-20T09:25:19.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2399999999999996e-07 2024-03-20T09:25:19.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:25:19.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.507395+0000", 2024-03-20T09:25:19.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6928000000000001e-05 2024-03-20T09:25:19.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:25:19.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.507405+0000", 2024-03-20T09:25:19.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0346999999999999e-05 2024-03-20T09:25:19.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:25:19.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.507411+0000", 2024-03-20T09:25:19.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6110000000000001e-06 2024-03-20T09:25:19.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:25:19.687 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.507458+0000", 2024-03-20T09:25:19.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7744999999999998e-05 2024-03-20T09:25:19.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:25:19.687 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.507577+0000", 2024-03-20T09:25:19.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000118675 2024-03-20T09:25:19.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,6", 2024-03-20T09:25:19.687 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.507664+0000", 2024-03-20T09:25:19.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7194000000000006e-05 2024-03-20T09:25:19.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:25:19.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.508424+0000", 2024-03-20T09:25:19.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00075975900000000004 2024-03-20T09:25:19.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:25:19.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.508709+0000", 2024-03-20T09:25:19.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000284746 2024-03-20T09:25:19.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.688 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:25:19.688 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.705229+0000", 2024-03-20T09:25:19.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.19652067000000001 2024-03-20T09:25:19.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:25:19.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.705284+0000", 2024-03-20T09:25:19.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4960999999999998e-05 2024-03-20T09:25:19.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:25:19.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:18:29.705330+0000", 2024-03-20T09:25:19.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5296999999999997e-05 2024-03-20T09:25:19.689 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.689 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:19.689 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.689 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26216.0:34970 6.4 6:2cad1c41:::benchmark_data_smithi122_168655_object34969:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e725)", 2024-03-20T09:25:19.690 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:24:40.950275+0000", 2024-03-20T09:25:19.690 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 38.701616641999998, 2024-03-20T09:25:19.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.059995195000000001, 2024-03-20T09:25:19.690 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:25:19.690 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:25:19.690 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:25:19.690 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26216", 2024-03-20T09:25:19.690 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:25:19.690 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 34970 2024-03-20T09:25:19.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.690 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:25:19.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:25:19.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:24:40.950275+0000", 2024-03-20T09:25:19.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:25:19.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:24:40.950275+0000", 2024-03-20T09:25:19.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:25:19.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:24:40.950276+0000", 2024-03-20T09:25:19.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2869999999999999e-06 2024-03-20T09:25:19.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:25:19.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:24:40.950335+0000", 2024-03-20T09:25:19.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9104999999999997e-05 2024-03-20T09:25:19.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:25:19.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:24:40.950347+0000", 2024-03-20T09:25:19.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1973999999999999e-05 2024-03-20T09:25:19.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:25:19.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:24:40.950355+0000", 2024-03-20T09:25:19.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6210000000000004e-06 2024-03-20T09:25:19.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:25:19.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:24:40.950410+0000", 2024-03-20T09:25:19.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5702000000000003e-05 2024-03-20T09:25:19.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:25:19.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:24:40.950540+0000", 2024-03-20T09:25:19.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129553 2024-03-20T09:25:19.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,5", 2024-03-20T09:25:19.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:24:40.950669+0000", 2024-03-20T09:25:19.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000129333 2024-03-20T09:25:19.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:25:19.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:24:40.951970+0000", 2024-03-20T09:25:19.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013012379999999999 2024-03-20T09:25:19.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:25:19.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:24:40.952748+0000", 2024-03-20T09:25:19.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000777701 2024-03-20T09:25:19.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:25:19.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:24:41.010140+0000", 2024-03-20T09:25:19.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.057391728000000003 2024-03-20T09:25:19.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:25:19.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:24:41.010219+0000", 2024-03-20T09:25:19.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9138e-05 2024-03-20T09:25:19.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:25:19.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:24:41.010270+0000", 2024-03-20T09:25:19.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0815000000000001e-05 2024-03-20T09:25:19.695 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.695 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:19.695 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.696 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26216.0:34978 6.4 6:27d5dada:::benchmark_data_smithi122_168655_object34977:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e725)", 2024-03-20T09:25:19.696 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:24:40.952746+0000", 2024-03-20T09:25:19.696 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 38.699145676000001, 2024-03-20T09:25:19.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.059980076, 2024-03-20T09:25:19.696 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:25:19.696 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:25:19.696 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:25:19.696 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26216", 2024-03-20T09:25:19.696 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:25:19.696 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 34978 2024-03-20T09:25:19.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.696 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:25:19.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:25:19.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:24:40.952746+0000", 2024-03-20T09:25:19.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:25:19.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:24:40.952746+0000", 2024-03-20T09:25:19.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:25:19.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:24:40.952746+0000", 2024-03-20T09:25:19.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1699999999999997e-07 2024-03-20T09:25:19.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:25:19.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:24:40.952805+0000", 2024-03-20T09:25:19.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8133000000000001e-05 2024-03-20T09:25:19.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:25:19.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:24:40.952816+0000", 2024-03-20T09:25:19.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0893999999999999e-05 2024-03-20T09:25:19.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:25:19.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:24:40.952823+0000", 2024-03-20T09:25:19.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6669999999999996e-06 2024-03-20T09:25:19.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.699 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:25:19.699 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:24:40.952870+0000", 2024-03-20T09:25:19.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7023e-05 2024-03-20T09:25:19.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.699 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:25:19.699 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:24:40.953008+0000", 2024-03-20T09:25:19.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000137768 2024-03-20T09:25:19.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.699 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,5", 2024-03-20T09:25:19.699 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:24:40.953131+0000", 2024-03-20T09:25:19.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012266999999999999 2024-03-20T09:25:19.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:25:19.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:24:40.954323+0000", 2024-03-20T09:25:19.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001192425 2024-03-20T09:25:19.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:25:19.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:24:40.956334+0000", 2024-03-20T09:25:19.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0020106640000000001 2024-03-20T09:25:19.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:25:19.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:24:41.012627+0000", 2024-03-20T09:25:19.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.056293298999999998 2024-03-20T09:25:19.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:25:19.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:24:41.012684+0000", 2024-03-20T09:25:19.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.66e-05 2024-03-20T09:25:19.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:25:19.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:24:41.012726+0000", 2024-03-20T09:25:19.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2015999999999999e-05 2024-03-20T09:25:19.701 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.701 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:19.701 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.701 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26216.0:85173 6.4 6:21c98fce:::benchmark_data_smithi122_168655_object85172:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e731)", 2024-03-20T09:25:19.701 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:25:13.305531+0000", 2024-03-20T09:25:19.702 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 6.3463600160000002, 2024-03-20T09:25:19.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.074655041000000005, 2024-03-20T09:25:19.702 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:25:19.702 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:25:19.702 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:25:19.702 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26216", 2024-03-20T09:25:19.702 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:25:19.702 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 85173 2024-03-20T09:25:19.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.702 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:25:19.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:25:19.702 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:13.305531+0000", 2024-03-20T09:25:19.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:25:19.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:13.305531+0000", 2024-03-20T09:25:19.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:25:19.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:13.305537+0000", 2024-03-20T09:25:19.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8329999999999999e-06 2024-03-20T09:25:19.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:25:19.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:13.305563+0000", 2024-03-20T09:25:19.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5947000000000001e-05 2024-03-20T09:25:19.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:25:19.704 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:13.305583+0000", 2024-03-20T09:25:19.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9930000000000001e-05 2024-03-20T09:25:19.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:25:19.704 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:13.305591+0000", 2024-03-20T09:25:19.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4840000000000004e-06 2024-03-20T09:25:19.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:25:19.704 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:13.305647+0000", 2024-03-20T09:25:19.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5306999999999997e-05 2024-03-20T09:25:19.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.705 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:25:19.705 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:13.305819+0000", 2024-03-20T09:25:19.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00017199000000000001 2024-03-20T09:25:19.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.705 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,5", 2024-03-20T09:25:19.705 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:13.305935+0000", 2024-03-20T09:25:19.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000116048 2024-03-20T09:25:19.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.705 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:25:19.706 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:13.308558+0000", 2024-03-20T09:25:19.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026237109999999999 2024-03-20T09:25:19.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:25:19.706 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:13.310383+0000", 2024-03-20T09:25:19.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018248579999999999 2024-03-20T09:25:19.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:25:19.706 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:13.380067+0000", 2024-03-20T09:25:19.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.069684120000000002 2024-03-20T09:25:19.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.707 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:25:19.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:13.380137+0000", 2024-03-20T09:25:19.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9301000000000006e-05 2024-03-20T09:25:19.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.707 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:25:19.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:13.380186+0000", 2024-03-20T09:25:19.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9512000000000003e-05 2024-03-20T09:25:19.707 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.707 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:19.707 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.707 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26216.0:92127 6.4 6:218a10af:::benchmark_data_smithi122_168655_object92126:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e735)", 2024-03-20T09:25:19.707 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:25:19.118194+0000", 2024-03-20T09:25:19.708 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.53369743800000002, 2024-03-20T09:25:19.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.104795362, 2024-03-20T09:25:19.708 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:25:19.708 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:25:19.708 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:25:19.708 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26216", 2024-03-20T09:25:19.708 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:25:19.708 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92127 2024-03-20T09:25:19.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.708 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:25:19.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.708 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:25:19.708 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.118194+0000", 2024-03-20T09:25:19.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:25:19.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.118194+0000", 2024-03-20T09:25:19.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:25:19.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.118195+0000", 2024-03-20T09:25:19.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.573e-06 2024-03-20T09:25:19.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:25:19.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.118228+0000", 2024-03-20T09:25:19.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2341000000000001e-05 2024-03-20T09:25:19.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:25:19.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.118240+0000", 2024-03-20T09:25:19.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2469e-05 2024-03-20T09:25:19.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:25:19.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.118248+0000", 2024-03-20T09:25:19.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5859999999999999e-06 2024-03-20T09:25:19.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:25:19.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.118583+0000", 2024-03-20T09:25:19.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00033515900000000002 2024-03-20T09:25:19.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:25:19.711 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.118677+0000", 2024-03-20T09:25:19.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3877999999999998e-05 2024-03-20T09:25:19.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,5", 2024-03-20T09:25:19.711 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.118788+0000", 2024-03-20T09:25:19.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011106499999999999 2024-03-20T09:25:19.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:25:19.712 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.123253+0000", 2024-03-20T09:25:19.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0044652260000000001 2024-03-20T09:25:19.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.712 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:25:19.712 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.123301+0000", 2024-03-20T09:25:19.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7629000000000001e-05 2024-03-20T09:25:19.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.712 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:25:19.712 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.222856+0000", 2024-03-20T09:25:19.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.099555313000000006 2024-03-20T09:25:19.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.713 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:25:19.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.222921+0000", 2024-03-20T09:25:19.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4956999999999995e-05 2024-03-20T09:25:19.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.713 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:25:19.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.222989+0000", 2024-03-20T09:25:19.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8165999999999998e-05 2024-03-20T09:25:19.713 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.713 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:19.713 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.713 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26216.0:92140 6.4 6:2566ec9a:::benchmark_data_smithi122_168655_object92139:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e735)", 2024-03-20T09:25:19.713 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:25:19.135833+0000", 2024-03-20T09:25:19.714 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.51605788500000005, 2024-03-20T09:25:19.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.093407693, 2024-03-20T09:25:19.714 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:25:19.714 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:25:19.714 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:25:19.714 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26216", 2024-03-20T09:25:19.714 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:25:19.714 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92140 2024-03-20T09:25:19.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.714 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:25:19.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:25:19.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.135833+0000", 2024-03-20T09:25:19.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:25:19.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.135833+0000", 2024-03-20T09:25:19.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:25:19.715 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.135835+0000", 2024-03-20T09:25:19.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.437e-06 2024-03-20T09:25:19.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.715 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:25:19.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.135871+0000", 2024-03-20T09:25:19.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6149999999999998e-05 2024-03-20T09:25:19.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:25:19.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.135884+0000", 2024-03-20T09:25:19.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3067e-05 2024-03-20T09:25:19.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:25:19.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.135893+0000", 2024-03-20T09:25:19.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0990000000000005e-06 2024-03-20T09:25:19.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:25:19.717 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.219545+0000", 2024-03-20T09:25:19.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.083652150999999994 2024-03-20T09:25:19.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.717 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:25:19.717 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.219702+0000", 2024-03-20T09:25:19.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015715400000000001 2024-03-20T09:25:19.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.717 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,5", 2024-03-20T09:25:19.717 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.219863+0000", 2024-03-20T09:25:19.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000160123 2024-03-20T09:25:19.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:25:19.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.223040+0000", 2024-03-20T09:25:19.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031777289999999998 2024-03-20T09:25:19.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:25:19.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.229039+0000", 2024-03-20T09:25:19.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0059983750000000002 2024-03-20T09:25:19.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:25:19.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.229127+0000", 2024-03-20T09:25:19.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8380000000000004e-05 2024-03-20T09:25:19.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.719 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:25:19.719 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.229187+0000", 2024-03-20T09:25:19.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0282999999999999e-05 2024-03-20T09:25:19.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.719 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:25:19.719 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.229241+0000", 2024-03-20T09:25:19.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3745000000000001e-05 2024-03-20T09:25:19.719 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.719 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:19.719 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.720 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26216.0:92142 6.4 6:244d4b3c:::benchmark_data_smithi122_168655_object92141:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e735)", 2024-03-20T09:25:19.720 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:25:19.135911+0000", 2024-03-20T09:25:19.720 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.51598039900000003, 2024-03-20T09:25:19.720 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.093497942000000001, 2024-03-20T09:25:19.720 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:25:19.720 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:25:19.720 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:25:19.720 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26216", 2024-03-20T09:25:19.720 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:25:19.720 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92142 2024-03-20T09:25:19.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.720 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:25:19.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.720 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:25:19.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.135911+0000", 2024-03-20T09:25:19.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:25:19.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.135911+0000", 2024-03-20T09:25:19.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:25:19.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.135911+0000", 2024-03-20T09:25:19.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8999999999999996e-07 2024-03-20T09:25:19.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:25:19.722 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.135954+0000", 2024-03-20T09:25:19.722 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2265000000000003e-05 2024-03-20T09:25:19.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.722 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:25:19.722 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.135963+0000", 2024-03-20T09:25:19.722 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0129999999999999e-06 2024-03-20T09:25:19.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.722 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:25:19.722 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.135970+0000", 2024-03-20T09:25:19.722 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8369999999999999e-06 2024-03-20T09:25:19.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.723 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:25:19.723 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.223074+0000", 2024-03-20T09:25:19.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.087104138999999997 2024-03-20T09:25:19.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.723 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:25:19.723 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.223241+0000", 2024-03-20T09:25:19.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000166835 2024-03-20T09:25:19.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.723 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,5", 2024-03-20T09:25:19.723 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.223393+0000", 2024-03-20T09:25:19.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000152423 2024-03-20T09:25:19.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.724 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:25:19.724 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.229110+0000", 2024-03-20T09:25:19.724 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057167980000000004 2024-03-20T09:25:19.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.724 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:25:19.724 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.229249+0000", 2024-03-20T09:25:19.724 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013882400000000001 2024-03-20T09:25:19.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.724 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:25:19.724 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.229294+0000", 2024-03-20T09:25:19.724 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5352000000000002e-05 2024-03-20T09:25:19.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:25:19.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.229349+0000", 2024-03-20T09:25:19.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4969000000000002e-05 2024-03-20T09:25:19.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:25:19.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.229409+0000", 2024-03-20T09:25:19.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9896999999999999e-05 2024-03-20T09:25:19.725 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.725 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:19.725 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.725 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop_reply(client.26216.0:92127 6.4 e735/709 ondisk, result = 0)", 2024-03-20T09:25:19.726 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:25:19.136397+0000", 2024-03-20T09:25:19.726 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.51549379699999998, 2024-03-20T09:25:19.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.086598340999999995, 2024-03-20T09:25:19.726 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:25:19.726 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:25:19.726 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:25:19.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.726 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:25:19.726 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.136397+0000", 2024-03-20T09:25:19.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.726 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:25:19.726 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.136397+0000", 2024-03-20T09:25:19.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.727 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:25:19.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.136398+0000", 2024-03-20T09:25:19.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6300000000000005e-07 2024-03-20T09:25:19.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.727 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:25:19.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.136402+0000", 2024-03-20T09:25:19.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0459999999999999e-06 2024-03-20T09:25:19.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.727 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:25:19.728 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.136411+0000", 2024-03-20T09:25:19.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5290000000000008e-06 2024-03-20T09:25:19.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.728 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:25:19.728 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.136417+0000", 2024-03-20T09:25:19.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.584e-06 2024-03-20T09:25:19.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.728 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:25:19.728 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.222837+0000", 2024-03-20T09:25:19.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.086419781000000001 2024-03-20T09:25:19.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.729 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:25:19.729 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.222845+0000", 2024-03-20T09:25:19.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0209999999999999e-06 2024-03-20T09:25:19.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.729 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:25:19.729 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.222996+0000", 2024-03-20T09:25:19.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000150817 2024-03-20T09:25:19.729 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.729 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:19.729 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.729 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26216.0:92143 6.b e735/709 6:d29f4b97:::benchmark_data_smithi122_168655_object92142:head v 735'5715, mlcod=735'5712)", 2024-03-20T09:25:19.729 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:25:19.136984+0000", 2024-03-20T09:25:19.730 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.51490717699999999, 2024-03-20T09:25:19.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.081925438000000003, 2024-03-20T09:25:19.730 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:25:19.730 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:25:19.730 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:25:19.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:25:19.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.136984+0000", 2024-03-20T09:25:19.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:25:19.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.136984+0000", 2024-03-20T09:25:19.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:25:19.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.136985+0000", 2024-03-20T09:25:19.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7499999999999999e-07 2024-03-20T09:25:19.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:25:19.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.137009+0000", 2024-03-20T09:25:19.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4139999999999999e-05 2024-03-20T09:25:19.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:25:19.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.137020+0000", 2024-03-20T09:25:19.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1046000000000001e-05 2024-03-20T09:25:19.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:25:19.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.137028+0000", 2024-03-20T09:25:19.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5170000000000002e-06 2024-03-20T09:25:19.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:25:19.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.137104+0000", 2024-03-20T09:25:19.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6914000000000003e-05 2024-03-20T09:25:19.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:25:19.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.137126+0000", 2024-03-20T09:25:19.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1696999999999999e-05 2024-03-20T09:25:19.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.733 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:25:19.733 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.218859+0000", 2024-03-20T09:25:19.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.081733282000000004 2024-03-20T09:25:19.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.733 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:25:19.733 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.218910+0000", 2024-03-20T09:25:19.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0067e-05 2024-03-20T09:25:19.733 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.733 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:19.733 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.734 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26216.0:92151 6.e e735/709 6:72a8fe95:::benchmark_data_smithi122_168655_object92150:head v 735'5711, mlcod=735'5709)", 2024-03-20T09:25:19.734 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:25:19.140297+0000", 2024-03-20T09:25:19.734 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.511593992, 2024-03-20T09:25:19.734 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.089045299999999994, 2024-03-20T09:25:19.734 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:25:19.734 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:25:19.734 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:25:19.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.734 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:25:19.734 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.140297+0000", 2024-03-20T09:25:19.734 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.735 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:25:19.735 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.140297+0000", 2024-03-20T09:25:19.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.735 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:25:19.735 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.140298+0000", 2024-03-20T09:25:19.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4900000000000004e-07 2024-03-20T09:25:19.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.735 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:25:19.735 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.140405+0000", 2024-03-20T09:25:19.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106938 2024-03-20T09:25:19.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:25:19.736 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.140417+0000", 2024-03-20T09:25:19.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1824000000000001e-05 2024-03-20T09:25:19.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:25:19.736 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.140425+0000", 2024-03-20T09:25:19.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4989999999999994e-06 2024-03-20T09:25:19.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:25:19.736 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.218892+0000", 2024-03-20T09:25:19.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.078466514000000001 2024-03-20T09:25:19.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:25:19.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.218917+0000", 2024-03-20T09:25:19.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4956e-05 2024-03-20T09:25:19.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:25:19.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.229298+0000", 2024-03-20T09:25:19.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010380587 2024-03-20T09:25:19.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:25:19.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.229343+0000", 2024-03-20T09:25:19.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5033000000000001e-05 2024-03-20T09:25:19.737 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.738 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:19.738 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.738 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26216.0:92152 6.e e735/709 6:75dfc8b0:::benchmark_data_smithi122_168655_object92151:head v 735'5712, mlcod=735'5709)", 2024-03-20T09:25:19.738 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:25:19.140521+0000", 2024-03-20T09:25:19.738 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.51137041100000002, 2024-03-20T09:25:19.738 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.088866429999999996, 2024-03-20T09:25:19.738 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:25:19.738 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:25:19.738 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:25:19.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.738 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:25:19.738 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.140521+0000", 2024-03-20T09:25:19.739 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.739 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:25:19.739 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.140521+0000", 2024-03-20T09:25:19.739 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.739 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:25:19.739 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.140521+0000", 2024-03-20T09:25:19.739 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7599999999999997e-07 2024-03-20T09:25:19.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.739 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:25:19.739 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.140553+0000", 2024-03-20T09:25:19.740 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1822000000000001e-05 2024-03-20T09:25:19.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.740 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:25:19.740 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.140562+0000", 2024-03-20T09:25:19.740 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3899999999999993e-06 2024-03-20T09:25:19.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.740 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:25:19.740 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.140567+0000", 2024-03-20T09:25:19.740 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.631e-06 2024-03-20T09:25:19.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.740 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:25:19.740 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.221875+0000", 2024-03-20T09:25:19.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.081307573999999994 2024-03-20T09:25:19.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.741 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:25:19.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.221894+0000", 2024-03-20T09:25:19.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9117999999999999e-05 2024-03-20T09:25:19.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.741 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:25:19.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.229360+0000", 2024-03-20T09:25:19.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074660849999999999 2024-03-20T09:25:19.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.741 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:25:19.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.229387+0000", 2024-03-20T09:25:19.742 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7334e-05 2024-03-20T09:25:19.742 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.742 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:19.742 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.742 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26216.0:92157 6.d e735/709 6:b062b6e7:::benchmark_data_smithi122_168655_object92156:head v 735'5755, mlcod=735'5753)", 2024-03-20T09:25:19.742 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:25:19.147985+0000", 2024-03-20T09:25:19.742 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.50390579800000002, 2024-03-20T09:25:19.742 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.085196573999999997, 2024-03-20T09:25:19.742 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:25:19.742 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:25:19.742 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:25:19.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:25:19.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.147985+0000", 2024-03-20T09:25:19.743 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:25:19.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.147985+0000", 2024-03-20T09:25:19.743 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:25:19.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.147987+0000", 2024-03-20T09:25:19.743 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.26e-06 2024-03-20T09:25:19.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.744 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:25:19.744 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.148049+0000", 2024-03-20T09:25:19.744 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2176999999999998e-05 2024-03-20T09:25:19.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.744 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:25:19.744 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.218855+0000", 2024-03-20T09:25:19.744 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.070805882000000001 2024-03-20T09:25:19.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.744 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:25:19.744 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.218867+0000", 2024-03-20T09:25:19.744 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2291e-05 2024-03-20T09:25:19.745 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.745 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.745 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:25:19.745 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.218926+0000", 2024-03-20T09:25:19.745 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9262999999999998e-05 2024-03-20T09:25:19.745 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.745 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.745 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:25:19.745 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.218941+0000", 2024-03-20T09:25:19.745 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5118e-05 2024-03-20T09:25:19.745 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.745 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.745 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:25:19.746 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.233150+0000", 2024-03-20T09:25:19.746 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014208114000000001 2024-03-20T09:25:19.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.746 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:25:19.746 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.233182+0000", 2024-03-20T09:25:19.746 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2468999999999997e-05 2024-03-20T09:25:19.746 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.746 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:19.746 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.746 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "replica_scrub(pg: 6.e,from:0'0,to:733'5415,epoch:735/709,start:6:75af52b4:::benchmark_data_smithi122_168655_object76383:0,end:6:75cc0eb0:::benchmark_data_smithi122_168655_object13478:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-03-20T09:25:19.747 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:25:19.148161+0000", 2024-03-20T09:25:19.747 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.50373049999999997, 2024-03-20T09:25:19.747 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.076424214000000004, 2024-03-20T09:25:19.747 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:25:19.747 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:25:19.747 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:25:19.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:25:19.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.148161+0000", 2024-03-20T09:25:19.747 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:25:19.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.148161+0000", 2024-03-20T09:25:19.748 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.748 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:25:19.748 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.148162+0000", 2024-03-20T09:25:19.748 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1299999999999998e-07 2024-03-20T09:25:19.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.748 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:25:19.748 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.148169+0000", 2024-03-20T09:25:19.748 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7500000000000003e-06 2024-03-20T09:25:19.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.748 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:25:19.749 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.218873+0000", 2024-03-20T09:25:19.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.070703567999999994 2024-03-20T09:25:19.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.749 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:25:19.749 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.218889+0000", 2024-03-20T09:25:19.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.596e-05 2024-03-20T09:25:19.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.749 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:25:19.749 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.224543+0000", 2024-03-20T09:25:19.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056543030000000003 2024-03-20T09:25:19.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.750 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:25:19.750 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.224557+0000", 2024-03-20T09:25:19.750 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4137e-05 2024-03-20T09:25:19.750 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.750 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:25:19.750 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.224585+0000", 2024-03-20T09:25:19.750 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7583e-05 2024-03-20T09:25:19.750 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.750 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:19.750 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.750 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.750 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26216.0:92537 6.4 6:28355bd5:::benchmark_data_smithi122_168655_object92536:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e736)", 2024-03-20T09:25:19.751 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:25:19.556008+0000", 2024-03-20T09:25:19.751 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.095883204999999999, 2024-03-20T09:25:19.751 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.032110922, 2024-03-20T09:25:19.751 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:25:19.751 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:25:19.751 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:25:19.751 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26216", 2024-03-20T09:25:19.751 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:25:19.751 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92537 2024-03-20T09:25:19.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.751 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:25:19.751 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.751 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:25:19.751 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.556008+0000", 2024-03-20T09:25:19.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.752 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:25:19.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.556008+0000", 2024-03-20T09:25:19.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.752 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:25:19.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.556009+0000", 2024-03-20T09:25:19.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.243e-06 2024-03-20T09:25:19.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.752 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:25:19.753 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.556036+0000", 2024-03-20T09:25:19.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6611e-05 2024-03-20T09:25:19.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.753 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:25:19.753 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.556045+0000", 2024-03-20T09:25:19.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6549999999999993e-06 2024-03-20T09:25:19.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.753 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:25:19.753 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.556051+0000", 2024-03-20T09:25:19.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0870000000000003e-06 2024-03-20T09:25:19.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.753 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:25:19.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.567181+0000", 2024-03-20T09:25:19.754 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011129994000000001 2024-03-20T09:25:19.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:25:19.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.567285+0000", 2024-03-20T09:25:19.754 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000104233 2024-03-20T09:25:19.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,5", 2024-03-20T09:25:19.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.567372+0000", 2024-03-20T09:25:19.754 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7511999999999998e-05 2024-03-20T09:25:19.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.755 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:25:19.755 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.582173+0000", 2024-03-20T09:25:19.755 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014800786 2024-03-20T09:25:19.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.755 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:25:19.755 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.582215+0000", 2024-03-20T09:25:19.755 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.142e-05 2024-03-20T09:25:19.755 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.755 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:25:19.755 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.587964+0000", 2024-03-20T09:25:19.755 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057492259999999996 2024-03-20T09:25:19.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.756 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:25:19.756 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.588047+0000", 2024-03-20T09:25:19.756 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3424999999999995e-05 2024-03-20T09:25:19.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.756 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:25:19.756 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.588119+0000", 2024-03-20T09:25:19.756 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1730000000000006e-05 2024-03-20T09:25:19.756 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.756 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:19.756 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.756 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.757 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26216.0:92572 6.4 6:225a4a45:::benchmark_data_smithi122_168655_object92571:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e736)", 2024-03-20T09:25:19.757 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:25:19.594063+0000", 2024-03-20T09:25:19.757 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.057828427000000002, 2024-03-20T09:25:19.757 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.028376775999999999, 2024-03-20T09:25:19.757 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:25:19.757 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:25:19.757 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:25:19.757 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26216", 2024-03-20T09:25:19.757 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:25:19.757 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92572 2024-03-20T09:25:19.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.757 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:25:19.757 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.757 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:25:19.758 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.594063+0000", 2024-03-20T09:25:19.758 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.758 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:25:19.758 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.594063+0000", 2024-03-20T09:25:19.758 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:25:19.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.758 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:25:19.758 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.594065+0000", 2024-03-20T09:25:19.758 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.79e-06 2024-03-20T09:25:19.758 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.758 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.759 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:25:19.759 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.594106+0000", 2024-03-20T09:25:19.759 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1718000000000003e-05 2024-03-20T09:25:19.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.759 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.759 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:25:19.759 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.594121+0000", 2024-03-20T09:25:19.759 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:25:19.761 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4799e-05 2024-03-20T09:25:19.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.761 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:25:19.761 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.594130+0000", 2024-03-20T09:25:19.761 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5280000000000003e-06 2024-03-20T09:25:19.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.761 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:25:19.761 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.595541+0000", 2024-03-20T09:25:19.761 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014110640000000001 2024-03-20T09:25:19.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.761 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:25:19.762 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.595705+0000", 2024-03-20T09:25:19.762 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00016405900000000001 2024-03-20T09:25:19.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.762 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,5", 2024-03-20T09:25:19.762 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.595875+0000", 2024-03-20T09:25:19.762 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000170218 2024-03-20T09:25:19.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.762 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:25:19.762 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.600622+0000", 2024-03-20T09:25:19.762 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004747121 2024-03-20T09:25:19.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.763 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:25:19.763 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.604139+0000", 2024-03-20T09:25:19.763 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035169530000000002 2024-03-20T09:25:19.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.763 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:25:19.763 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.622330+0000", 2024-03-20T09:25:19.763 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018191368999999999 2024-03-20T09:25:19.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.763 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:25:19.763 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.622383+0000", 2024-03-20T09:25:19.763 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2670000000000002e-05 2024-03-20T09:25:19.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:25:19.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:25:19.764 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:25:19.764 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.622440+0000", 2024-03-20T09:25:19.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6487000000000003e-05 2024-03-20T09:25:19.764 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.764 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:19.764 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.764 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:25:19.764 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:25:19.764 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:25:19.764 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:19 smithi026 ceph-mon[68309]: 1.1d scrub starts 2024-03-20T09:25:19.764 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:19 smithi026 ceph-mon[68309]: 1.1d scrub ok 2024-03-20T09:25:19.764 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:19 smithi026 ceph-mon[68309]: 6.d scrub starts 2024-03-20T09:25:19.764 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:19 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:25:19.765 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:19 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:25:19.765 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:19 smithi026 ceph-mon[68309]: 6.e scrub starts 2024-03-20T09:25:19.788 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:19 smithi122 ceph-mon[73495]: 1.1d scrub starts 2024-03-20T09:25:19.788 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:19 smithi122 ceph-mon[73495]: 1.1d scrub ok 2024-03-20T09:25:19.788 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:19 smithi122 ceph-mon[73495]: 6.d scrub starts 2024-03-20T09:25:19.788 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:19 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:25:19.788 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:19 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:25:19.789 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:19 smithi122 ceph-mon[73495]: 6.e scrub starts 2024-03-20T09:25:19.790 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:19.814 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:25:19.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:19.819 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:25:19.861 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 64 16 92855 92839 90.653 63.1875 0.00673947 0.011027 2024-03-20T09:25:19.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:19 smithi115 ceph-mon[72476]: 1.1d scrub starts 2024-03-20T09:25:19.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:19 smithi115 ceph-mon[72476]: 1.1d scrub ok 2024-03-20T09:25:19.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:19 smithi115 ceph-mon[72476]: 6.d scrub starts 2024-03-20T09:25:19.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:19 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:25:19.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:19 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:25:19.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:19 smithi115 ceph-mon[72476]: 6.e scrub starts 2024-03-20T09:25:19.920 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:25:19.999 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:20.212 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:25:20.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:20.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:25:20.320 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:25:20.394 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:20.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:20 smithi026 ceph-mon[68309]: 6.4 scrub starts 2024-03-20T09:25:20.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:20 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:25:20.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:20 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:25:20.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:20 smithi026 ceph-mon[68309]: osdmap e736: 12 total, 12 up, 12 in 2024-03-20T09:25:20.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:20 smithi026 ceph-mon[68309]: 6.d scrub ok 2024-03-20T09:25:20.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:20 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1808613971' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:25:20.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:20 smithi026 ceph-mon[68309]: pgmap v1798: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-03-20T09:25:20.861 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 65 16 93900 93884 90.263 65.3125 0.0106454 0.0110746 2024-03-20T09:25:20.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:20 smithi115 ceph-mon[72476]: 6.4 scrub starts 2024-03-20T09:25:20.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:20 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:25:20.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:20 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:25:20.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:20 smithi115 ceph-mon[72476]: osdmap e736: 12 total, 12 up, 12 in 2024-03-20T09:25:20.904 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:20 smithi115 ceph-mon[72476]: 6.d scrub ok 2024-03-20T09:25:20.904 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:20 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1808613971' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:25:20.904 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:20 smithi115 ceph-mon[72476]: pgmap v1798: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-03-20T09:25:20.914 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:25:20.914 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:25:20.914 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:25:20.914 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:25:20.915 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:25:20.915 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:25:20.915 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:25:20.915 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:25:20.915 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:25:20.915 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:25:20.915 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:25:20.915 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:25:20.915 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:25:20.915 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:25:20.915 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:25:20.916 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:25:20.916 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:25:20.916 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:25:20.916 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:25:20.916 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:25:20.916 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:25:20.916 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:25:20.916 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:25:20.916 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:25:20.943 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:20 smithi122 ceph-mon[73495]: 6.4 scrub starts 2024-03-20T09:25:20.944 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:20 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:25:20.944 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:20 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:25:20.944 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:20 smithi122 ceph-mon[73495]: osdmap e736: 12 total, 12 up, 12 in 2024-03-20T09:25:20.944 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:20 smithi122 ceph-mon[73495]: 6.d scrub ok 2024-03-20T09:25:20.944 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:20 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1808613971' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:25:20.944 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:20 smithi122 ceph-mon[73495]: pgmap v1798: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-03-20T09:25:21.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:21 smithi026 ceph-mon[68309]: 6.e scrub ok 2024-03-20T09:25:21.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:21 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:25:21.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:21 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:25:21.799 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:25:21.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:21.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:25:21.861 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 66 16 95158 95142 90.0866 78.625 0.0104508 0.0110965 2024-03-20T09:25:21.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:21 smithi115 ceph-mon[72476]: 6.e scrub ok 2024-03-20T09:25:21.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:21 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:25:21.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:21 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:25:21.906 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:25:21.943 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:21 smithi122 ceph-mon[73495]: 6.e scrub ok 2024-03-20T09:25:21.944 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:21 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:25:21.944 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:21 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:25:21.965 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:22.134 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:25:22.138 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:25:22.145 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:25:22.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:22.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:25:22.185 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-20T09:25:22.262 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:25:22.291 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:22.314 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:25:22.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:22.316 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:25:22.417 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:25:22.534 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:22.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:22 smithi122 ceph-mon[73495]: 1.1 scrub starts 2024-03-20T09:25:22.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:22 smithi122 ceph-mon[73495]: 1.1 scrub ok 2024-03-20T09:25:22.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:22 smithi122 ceph-mon[73495]: 1.17 scrub starts 2024-03-20T09:25:22.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:22 smithi122 ceph-mon[73495]: 6.4 scrub ok 2024-03-20T09:25:22.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:22 smithi122 ceph-mon[73495]: 1.17 scrub ok 2024-03-20T09:25:22.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:22 smithi122 ceph-mon[73495]: 1.18 scrub starts 2024-03-20T09:25:22.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:22 smithi122 ceph-mon[73495]: 1.18 scrub ok 2024-03-20T09:25:22.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:22 smithi122 ceph-mon[73495]: osdmap e737: 12 total, 12 up, 12 in 2024-03-20T09:25:22.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:22 smithi122 ceph-mon[73495]: pgmap v1800: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-03-20T09:25:22.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:22 smithi026 ceph-mon[68309]: 1.1 scrub starts 2024-03-20T09:25:22.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:22 smithi026 ceph-mon[68309]: 1.1 scrub ok 2024-03-20T09:25:22.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:22 smithi026 ceph-mon[68309]: 1.17 scrub starts 2024-03-20T09:25:22.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:22 smithi026 ceph-mon[68309]: 6.4 scrub ok 2024-03-20T09:25:22.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:22 smithi026 ceph-mon[68309]: 1.17 scrub ok 2024-03-20T09:25:22.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:22 smithi026 ceph-mon[68309]: 1.18 scrub starts 2024-03-20T09:25:22.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:22 smithi026 ceph-mon[68309]: 1.18 scrub ok 2024-03-20T09:25:22.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:22 smithi026 ceph-mon[68309]: osdmap e737: 12 total, 12 up, 12 in 2024-03-20T09:25:22.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:22 smithi026 ceph-mon[68309]: pgmap v1800: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-03-20T09:25:22.861 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 67 16 96302 96286 89.8091 71.5 0.0213364 0.0111309 2024-03-20T09:25:22.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:22 smithi115 ceph-mon[72476]: 1.1 scrub starts 2024-03-20T09:25:22.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:22 smithi115 ceph-mon[72476]: 1.1 scrub ok 2024-03-20T09:25:22.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:22 smithi115 ceph-mon[72476]: 1.17 scrub starts 2024-03-20T09:25:22.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:22 smithi115 ceph-mon[72476]: 6.4 scrub ok 2024-03-20T09:25:22.904 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:22 smithi115 ceph-mon[72476]: 1.17 scrub ok 2024-03-20T09:25:22.904 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:22 smithi115 ceph-mon[72476]: 1.18 scrub starts 2024-03-20T09:25:22.904 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:22 smithi115 ceph-mon[72476]: 1.18 scrub ok 2024-03-20T09:25:22.904 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:22 smithi115 ceph-mon[72476]: osdmap e737: 12 total, 12 up, 12 in 2024-03-20T09:25:22.904 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:22 smithi115 ceph-mon[72476]: pgmap v1800: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-03-20T09:25:23.817 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:25:23.861 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 68 16 97490 97474 89.5802 74.25 0.0150463 0.0111593 2024-03-20T09:25:24.218 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:23 smithi026 ceph-mon[68309]: 6.1 scrub starts 2024-03-20T09:25:24.218 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:23 smithi026 ceph-mon[68309]: 6.7 scrub starts 2024-03-20T09:25:24.218 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:23 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:25:24.218 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:23 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:25:24.218 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:23 smithi026 ceph-mon[68309]: osdmap e738: 12 total, 12 up, 12 in 2024-03-20T09:25:24.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:23 smithi115 ceph-mon[72476]: 6.1 scrub starts 2024-03-20T09:25:24.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:23 smithi115 ceph-mon[72476]: 6.7 scrub starts 2024-03-20T09:25:24.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:23 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:25:24.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:23 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:25:24.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:23 smithi115 ceph-mon[72476]: osdmap e738: 12 total, 12 up, 12 in 2024-03-20T09:25:24.443 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:23 smithi122 ceph-mon[73495]: 6.1 scrub starts 2024-03-20T09:25:24.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:23 smithi122 ceph-mon[73495]: 6.7 scrub starts 2024-03-20T09:25:24.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:23 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:25:24.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:23 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:25:24.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:23 smithi122 ceph-mon[73495]: osdmap e738: 12 total, 12 up, 12 in 2024-03-20T09:25:24.862 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 69 16 98579 98563 89.2683 68.0625 0.0095404 0.011198 2024-03-20T09:25:25.026 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:25:25.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:25.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:25:25.131 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:25:25.194 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:25.217 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:24 smithi026 ceph-mon[68309]: 1.e scrub starts 2024-03-20T09:25:25.218 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:24 smithi026 ceph-mon[68309]: 1.e scrub ok 2024-03-20T09:25:25.218 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:24 smithi026 ceph-mon[68309]: 6.3 scrub starts 2024-03-20T09:25:25.218 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:24 smithi026 ceph-mon[68309]: 6.1 scrub ok 2024-03-20T09:25:25.218 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:24 smithi026 ceph-mon[68309]: pgmap v1802: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-03-20T09:25:25.218 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:24 smithi026 ceph-mon[68309]: 6.7 scrub ok 2024-03-20T09:25:25.403 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:24 smithi115 ceph-mon[72476]: 1.e scrub starts 2024-03-20T09:25:25.403 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:24 smithi115 ceph-mon[72476]: 1.e scrub ok 2024-03-20T09:25:25.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:24 smithi115 ceph-mon[72476]: 6.3 scrub starts 2024-03-20T09:25:25.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:24 smithi115 ceph-mon[72476]: 6.1 scrub ok 2024-03-20T09:25:25.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:24 smithi115 ceph-mon[72476]: pgmap v1802: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-03-20T09:25:25.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:24 smithi115 ceph-mon[72476]: 6.7 scrub ok 2024-03-20T09:25:25.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:24 smithi122 ceph-mon[73495]: 1.e scrub starts 2024-03-20T09:25:25.445 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:24 smithi122 ceph-mon[73495]: 1.e scrub ok 2024-03-20T09:25:25.445 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:24 smithi122 ceph-mon[73495]: 6.3 scrub starts 2024-03-20T09:25:25.445 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:24 smithi122 ceph-mon[73495]: 6.1 scrub ok 2024-03-20T09:25:25.445 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:24 smithi122 ceph-mon[73495]: pgmap v1802: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.15k op/s 2024-03-20T09:25:25.445 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:24 smithi122 ceph-mon[73495]: 6.7 scrub ok 2024-03-20T09:25:25.672 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:25:25.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:25.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:25:25.776 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:25:25.839 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:25.861 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 70 16 99652 99636 88.951 67.0625 0.0173917 0.0112373 2024-03-20T09:25:25.941 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:25:25.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:25.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:25:26.049 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:25:26.115 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:26.218 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:25 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:25:26.218 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:25 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:25:26.218 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:25 smithi026 ceph-mon[68309]: osdmap e739: 12 total, 12 up, 12 in 2024-03-20T09:25:26.219 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:25 smithi026 ceph-mon[68309]: 1.16 scrub starts 2024-03-20T09:25:26.219 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:25 smithi026 ceph-mon[68309]: 1.16 scrub ok 2024-03-20T09:25:26.219 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:25 smithi026 ceph-mon[68309]: 6.3 scrub ok 2024-03-20T09:25:26.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:25 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:25:26.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:25 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:25:26.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:25 smithi115 ceph-mon[72476]: osdmap e739: 12 total, 12 up, 12 in 2024-03-20T09:25:26.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:25 smithi115 ceph-mon[72476]: 1.16 scrub starts 2024-03-20T09:25:26.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:25 smithi115 ceph-mon[72476]: 1.16 scrub ok 2024-03-20T09:25:26.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:25 smithi115 ceph-mon[72476]: 6.3 scrub ok 2024-03-20T09:25:26.443 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:25 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:25:26.443 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:25 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:25:26.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:25 smithi122 ceph-mon[73495]: osdmap e739: 12 total, 12 up, 12 in 2024-03-20T09:25:26.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:25 smithi122 ceph-mon[73495]: 1.16 scrub starts 2024-03-20T09:25:26.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:25 smithi122 ceph-mon[73495]: 1.16 scrub ok 2024-03-20T09:25:26.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:25 smithi122 ceph-mon[73495]: 6.3 scrub ok 2024-03-20T09:25:26.862 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 71 16 100676 100660 88.5996 64 0.00574895 0.0112827 2024-03-20T09:25:27.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:26 smithi115 ceph-mon[72476]: 6.9 scrub starts 2024-03-20T09:25:27.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:26 smithi115 ceph-mon[72476]: 6.f scrub starts 2024-03-20T09:25:27.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:26 smithi115 ceph-mon[72476]: pgmap v1804: 47 pgs: 47 active+clean; 6.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-03-20T09:25:27.443 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:26 smithi122 ceph-mon[73495]: 6.9 scrub starts 2024-03-20T09:25:27.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:26 smithi122 ceph-mon[73495]: 6.f scrub starts 2024-03-20T09:25:27.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:26 smithi122 ceph-mon[73495]: pgmap v1804: 47 pgs: 47 active+clean; 6.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-03-20T09:25:27.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:26 smithi026 ceph-mon[68309]: 6.9 scrub starts 2024-03-20T09:25:27.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:26 smithi026 ceph-mon[68309]: 6.f scrub starts 2024-03-20T09:25:27.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:26 smithi026 ceph-mon[68309]: pgmap v1804: 47 pgs: 47 active+clean; 6.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-03-20T09:25:27.592 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:25:27.862 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 72 16 101654 101638 88.2179 61.125 0.0126211 0.0113318 2024-03-20T09:25:28.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:28 smithi122 ceph-mon[73495]: 6.6 scrub starts 2024-03-20T09:25:28.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:28 smithi122 ceph-mon[73495]: pgmap v1805: 47 pgs: 47 active+clean; 6.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-03-20T09:25:28.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:28 smithi026 ceph-mon[68309]: 6.6 scrub starts 2024-03-20T09:25:28.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:28 smithi026 ceph-mon[68309]: pgmap v1805: 47 pgs: 47 active+clean; 6.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-03-20T09:25:28.862 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 73 16 102694 102678 87.8998 65 0.0103484 0.0113727 2024-03-20T09:25:28.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:28 smithi115 ceph-mon[72476]: 6.6 scrub starts 2024-03-20T09:25:28.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:28 smithi115 ceph-mon[72476]: pgmap v1805: 47 pgs: 47 active+clean; 6.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-03-20T09:25:29.489 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:25:29.524 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:25:29.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:29.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:25:29.629 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:25:29.658 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:29.683 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:25:29.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:29.688 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:25:29.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:29 smithi026 ceph-mon[68309]: 6.9 scrub ok 2024-03-20T09:25:29.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:29 smithi026 ceph-mon[68309]: 6.f scrub ok 2024-03-20T09:25:29.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:29 smithi026 ceph-mon[68309]: 6.8 scrub starts 2024-03-20T09:25:29.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:29 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2928208311' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:25:29.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:29 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:25:29.788 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:25:29.862 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 74 16 103653 103637 87.5219 59.9375 0.0125035 0.0114206 2024-03-20T09:25:29.867 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:29.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:29 smithi115 ceph-mon[72476]: 6.9 scrub ok 2024-03-20T09:25:29.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:29 smithi115 ceph-mon[72476]: 6.f scrub ok 2024-03-20T09:25:29.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:29 smithi115 ceph-mon[72476]: 6.8 scrub starts 2024-03-20T09:25:29.904 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:29 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2928208311' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:25:29.904 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:29 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:25:29.944 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:29 smithi122 ceph-mon[73495]: 6.9 scrub ok 2024-03-20T09:25:29.945 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:29 smithi122 ceph-mon[73495]: 6.f scrub ok 2024-03-20T09:25:29.945 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:29 smithi122 ceph-mon[73495]: 6.8 scrub starts 2024-03-20T09:25:29.945 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:29 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2928208311' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:25:29.945 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:29 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:25:29.949 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:25:29.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:29.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:25:30.061 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:25:30.130 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:30.158 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:25:30.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:30.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:25:30.268 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:25:30.329 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:30.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:30 smithi026 ceph-mon[68309]: 1.13 scrub starts 2024-03-20T09:25:30.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:30 smithi026 ceph-mon[68309]: 1.13 scrub ok 2024-03-20T09:25:30.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:30 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:25:30.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:30 smithi026 ceph-mon[68309]: Cluster is now healthy 2024-03-20T09:25:30.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:30 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:25:30.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:30 smithi026 ceph-mon[68309]: osdmap e740: 12 total, 12 up, 12 in 2024-03-20T09:25:30.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:30 smithi026 ceph-mon[68309]: 1.5 scrub starts 2024-03-20T09:25:30.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:30 smithi026 ceph-mon[68309]: 6.6 scrub ok 2024-03-20T09:25:30.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:30 smithi026 ceph-mon[68309]: 1.5 scrub ok 2024-03-20T09:25:30.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:30 smithi026 ceph-mon[68309]: pgmap v1807: 47 pgs: 47 active+clean; 6.2 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-03-20T09:25:30.862 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 75 16 104815 104799 87.3232 72.625 0.0213282 0.0114469 2024-03-20T09:25:30.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:30 smithi115 ceph-mon[72476]: 1.13 scrub starts 2024-03-20T09:25:30.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:30 smithi115 ceph-mon[72476]: 1.13 scrub ok 2024-03-20T09:25:30.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:30 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:25:30.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:30 smithi115 ceph-mon[72476]: Cluster is now healthy 2024-03-20T09:25:30.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:30 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:25:30.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:30 smithi115 ceph-mon[72476]: osdmap e740: 12 total, 12 up, 12 in 2024-03-20T09:25:30.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:30 smithi115 ceph-mon[72476]: 1.5 scrub starts 2024-03-20T09:25:30.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:30 smithi115 ceph-mon[72476]: 6.6 scrub ok 2024-03-20T09:25:30.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:30 smithi115 ceph-mon[72476]: 1.5 scrub ok 2024-03-20T09:25:30.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:30 smithi115 ceph-mon[72476]: pgmap v1807: 47 pgs: 47 active+clean; 6.2 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-03-20T09:25:30.907 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:25:30.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:30.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:25:30.943 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:30 smithi122 ceph-mon[73495]: 1.13 scrub starts 2024-03-20T09:25:30.944 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:30 smithi122 ceph-mon[73495]: 1.13 scrub ok 2024-03-20T09:25:30.944 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:30 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:25:30.944 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:30 smithi122 ceph-mon[73495]: Cluster is now healthy 2024-03-20T09:25:30.944 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:30 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:25:30.944 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:30 smithi122 ceph-mon[73495]: osdmap e740: 12 total, 12 up, 12 in 2024-03-20T09:25:30.944 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:30 smithi122 ceph-mon[73495]: 1.5 scrub starts 2024-03-20T09:25:30.944 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:30 smithi122 ceph-mon[73495]: 6.6 scrub ok 2024-03-20T09:25:30.944 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:30 smithi122 ceph-mon[73495]: 1.5 scrub ok 2024-03-20T09:25:30.944 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:30 smithi122 ceph-mon[73495]: pgmap v1807: 47 pgs: 47 active+clean; 6.2 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-03-20T09:25:31.013 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:25:31.054 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:31.056 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:25:31.093 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:25:31.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:31.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:25:31.201 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:25:31.233 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:31.256 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:25:31.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:31.258 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:25:31.360 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:25:31.398 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:31.426 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:25:31.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:31.429 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:25:31.496 INFO:teuthology.orchestra.run.smithi026.stderr:set 1.1c pg_upmap_items mapping to [0->9,6->2,5->7] 2024-03-20T09:25:31.529 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:25:31.556 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:31.577 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:25:31.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:31.580 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:25:31.681 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:25:31.711 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:31.730 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:25:31.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:31.733 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:25:31.834 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:25:31.862 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 76 16 105873 105857 87.0442 66.125 0.0132644 0.0114838 2024-03-20T09:25:31.881 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:31 smithi026 ceph-mon[68309]: 6.8 scrub ok 2024-03-20T09:25:31.881 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:31 smithi026 ceph-mon[68309]: 1.2 scrub starts 2024-03-20T09:25:31.881 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:31 smithi026 ceph-mon[68309]: 1.2 scrub ok 2024-03-20T09:25:31.881 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:31 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/848438569' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.1c", "id": [0, 9, 6, 2, 5, 7]}]: dispatch 2024-03-20T09:25:31.889 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:31.904 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:31 smithi115 ceph-mon[72476]: 6.8 scrub ok 2024-03-20T09:25:31.904 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:31 smithi115 ceph-mon[72476]: 1.2 scrub starts 2024-03-20T09:25:31.904 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:31 smithi115 ceph-mon[72476]: 1.2 scrub ok 2024-03-20T09:25:31.904 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:31 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/848438569' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.1c", "id": [0, 9, 6, 2, 5, 7]}]: dispatch 2024-03-20T09:25:31.943 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:31 smithi122 ceph-mon[73495]: 6.8 scrub ok 2024-03-20T09:25:31.943 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:31 smithi122 ceph-mon[73495]: 1.2 scrub starts 2024-03-20T09:25:31.944 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:31 smithi122 ceph-mon[73495]: 1.2 scrub ok 2024-03-20T09:25:31.944 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:31 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/848438569' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.1c", "id": [0, 9, 6, 2, 5, 7]}]: dispatch 2024-03-20T09:25:31.960 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:25:31.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:31.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:25:32.066 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:25:32.126 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:32.157 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:25:32.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:32.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:25:32.266 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:25:32.295 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:32.323 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:25:32.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:32.329 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:25:32.430 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:25:32.480 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:32.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:32 smithi122 ceph-mon[73495]: 6.5 scrub starts 2024-03-20T09:25:32.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:32 smithi122 ceph-mon[73495]: 1.0 scrub starts 2024-03-20T09:25:32.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:32 smithi122 ceph-mon[73495]: 1.0 scrub ok 2024-03-20T09:25:32.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:32 smithi122 ceph-mon[73495]: 1.10 scrub starts 2024-03-20T09:25:32.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:32 smithi122 ceph-mon[73495]: 1.10 scrub ok 2024-03-20T09:25:32.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:32 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/848438569' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.1c", "id": [0, 9, 6, 2, 5, 7]}]': finished 2024-03-20T09:25:32.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:32 smithi122 ceph-mon[73495]: osdmap e741: 12 total, 12 up, 12 in 2024-03-20T09:25:32.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:32 smithi122 ceph-mon[73495]: pgmap v1809: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-03-20T09:25:32.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:32 smithi122 ceph-mon[73495]: 6.5 scrub ok 2024-03-20T09:25:32.694 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:32 smithi122 ceph-mon[73495]: osdmap e742: 12 total, 12 up, 12 in 2024-03-20T09:25:32.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:32 smithi026 ceph-mon[68309]: 6.5 scrub starts 2024-03-20T09:25:32.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:32 smithi026 ceph-mon[68309]: 1.0 scrub starts 2024-03-20T09:25:32.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:32 smithi026 ceph-mon[68309]: 1.0 scrub ok 2024-03-20T09:25:32.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:32 smithi026 ceph-mon[68309]: 1.10 scrub starts 2024-03-20T09:25:32.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:32 smithi026 ceph-mon[68309]: 1.10 scrub ok 2024-03-20T09:25:32.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:32 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/848438569' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.1c", "id": [0, 9, 6, 2, 5, 7]}]': finished 2024-03-20T09:25:32.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:32 smithi026 ceph-mon[68309]: osdmap e741: 12 total, 12 up, 12 in 2024-03-20T09:25:32.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:32 smithi026 ceph-mon[68309]: pgmap v1809: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-03-20T09:25:32.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:32 smithi026 ceph-mon[68309]: 6.5 scrub ok 2024-03-20T09:25:32.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:32 smithi026 ceph-mon[68309]: osdmap e742: 12 total, 12 up, 12 in 2024-03-20T09:25:32.767 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:25:32.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:32.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:25:32.862 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 77 16 106991 106975 86.8212 69.875 0.0160584 0.0115143 2024-03-20T09:25:32.869 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:25:32.897 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:32.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:32 smithi115 ceph-mon[72476]: 6.5 scrub starts 2024-03-20T09:25:32.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:32 smithi115 ceph-mon[72476]: 1.0 scrub starts 2024-03-20T09:25:32.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:32 smithi115 ceph-mon[72476]: 1.0 scrub ok 2024-03-20T09:25:32.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:32 smithi115 ceph-mon[72476]: 1.10 scrub starts 2024-03-20T09:25:32.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:32 smithi115 ceph-mon[72476]: 1.10 scrub ok 2024-03-20T09:25:32.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:32 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/848438569' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.1c", "id": [0, 9, 6, 2, 5, 7]}]': finished 2024-03-20T09:25:32.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:32 smithi115 ceph-mon[72476]: osdmap e741: 12 total, 12 up, 12 in 2024-03-20T09:25:32.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:32 smithi115 ceph-mon[72476]: pgmap v1809: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-03-20T09:25:32.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:32 smithi115 ceph-mon[72476]: 6.5 scrub ok 2024-03-20T09:25:32.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:32 smithi115 ceph-mon[72476]: osdmap e742: 12 total, 12 up, 12 in 2024-03-20T09:25:32.917 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:25:32.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:32.921 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:25:33.022 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:25:33.072 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:33.108 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:25:33.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:33.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:25:33.217 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:25:33.246 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:33.269 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:25:33.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:33.271 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:25:33.372 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:25:33.428 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:33.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:33 smithi026 ceph-mon[68309]: 1.1a scrub starts 2024-03-20T09:25:33.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:33 smithi026 ceph-mon[68309]: 1.1a scrub ok 2024-03-20T09:25:33.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:33 smithi026 ceph-mon[68309]: osdmap e743: 12 total, 12 up, 12 in 2024-03-20T09:25:33.862 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 78 16 108078 108062 86.579 67.9375 0.0135505 0.0115459 2024-03-20T09:25:33.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:33 smithi115 ceph-mon[72476]: 1.1a scrub starts 2024-03-20T09:25:33.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:33 smithi115 ceph-mon[72476]: 1.1a scrub ok 2024-03-20T09:25:33.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:33 smithi115 ceph-mon[72476]: osdmap e743: 12 total, 12 up, 12 in 2024-03-20T09:25:33.943 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:33 smithi122 ceph-mon[73495]: 1.1a scrub starts 2024-03-20T09:25:33.944 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:33 smithi122 ceph-mon[73495]: 1.1a scrub ok 2024-03-20T09:25:33.944 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:33 smithi122 ceph-mon[73495]: osdmap e743: 12 total, 12 up, 12 in 2024-03-20T09:25:34.317 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:25:34.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:34.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:25:34.427 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:25:34.429 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:25:34.429 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:25:34.460 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:34.483 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:25:34.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:34.491 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:25:34.592 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:25:34.655 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:34.681 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:25:34.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:34.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:25:34.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:34 smithi026 ceph-mon[68309]: 1.1e scrub starts 2024-03-20T09:25:34.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:34 smithi026 ceph-mon[68309]: 1.1e scrub ok 2024-03-20T09:25:34.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:34 smithi026 ceph-mon[68309]: pgmap v1812: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-03-20T09:25:34.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:34 smithi026 ceph-mon[68309]: osdmap e744: 12 total, 12 up, 12 in 2024-03-20T09:25:34.788 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:25:34.814 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:25:34.845 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:34.862 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 79 16 108984 108968 86.1998 56.625 0.0169224 0.011591 2024-03-20T09:25:34.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:34 smithi115 ceph-mon[72476]: 1.1e scrub starts 2024-03-20T09:25:34.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:34 smithi115 ceph-mon[72476]: 1.1e scrub ok 2024-03-20T09:25:34.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:34 smithi115 ceph-mon[72476]: pgmap v1812: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-03-20T09:25:34.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:34 smithi115 ceph-mon[72476]: osdmap e744: 12 total, 12 up, 12 in 2024-03-20T09:25:34.943 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:34 smithi122 ceph-mon[73495]: 1.1e scrub starts 2024-03-20T09:25:34.943 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:34 smithi122 ceph-mon[73495]: 1.1e scrub ok 2024-03-20T09:25:34.944 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:34 smithi122 ceph-mon[73495]: pgmap v1812: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.4 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-03-20T09:25:34.944 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:34 smithi122 ceph-mon[73495]: osdmap e744: 12 total, 12 up, 12 in 2024-03-20T09:25:35.478 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-20T09:25:35.531 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:25:35.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:35.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:25:35.575 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:35 smithi026 ceph-mon[68309]: 1.4 scrub starts 2024-03-20T09:25:35.575 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:35 smithi026 ceph-mon[68309]: 1.4 scrub ok 2024-03-20T09:25:35.575 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:35 smithi026 ceph-mon[68309]: 1.12 scrub starts 2024-03-20T09:25:35.575 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:35 smithi026 ceph-mon[68309]: 1.12 scrub ok 2024-03-20T09:25:35.575 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:35 smithi026 ceph-mon[68309]: 6.b scrub starts 2024-03-20T09:25:35.575 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:35 smithi026 ceph-mon[68309]: 1.d scrub starts 2024-03-20T09:25:35.575 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:35 smithi026 ceph-mon[68309]: 1.d scrub ok 2024-03-20T09:25:35.575 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:35 smithi026 ceph-mon[68309]: osdmap e745: 12 total, 12 up, 12 in 2024-03-20T09:25:35.636 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:25:35.640 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:25:35.758 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:35.833 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:25:35.862 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:2024-03-20T09:25:35.865949+0000 min lat: 0.00104258 max lat: 11.7007 avg lat: 0.0116481 2024-03-20T09:25:35.862 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T09:25:35.862 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 80 16 109880 109864 85.8222 56 0.0167813 0.0116481 2024-03-20T09:25:35.877 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:25:35.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:35 smithi115 ceph-mon[72476]: 1.4 scrub starts 2024-03-20T09:25:35.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:35 smithi115 ceph-mon[72476]: 1.4 scrub ok 2024-03-20T09:25:35.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:35 smithi115 ceph-mon[72476]: 1.12 scrub starts 2024-03-20T09:25:35.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:35 smithi115 ceph-mon[72476]: 1.12 scrub ok 2024-03-20T09:25:35.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:35 smithi115 ceph-mon[72476]: 6.b scrub starts 2024-03-20T09:25:35.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:35 smithi115 ceph-mon[72476]: 1.d scrub starts 2024-03-20T09:25:35.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:35 smithi115 ceph-mon[72476]: 1.d scrub ok 2024-03-20T09:25:35.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:35 smithi115 ceph-mon[72476]: osdmap e745: 12 total, 12 up, 12 in 2024-03-20T09:25:35.930 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:25:35.943 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:35 smithi122 ceph-mon[73495]: 1.4 scrub starts 2024-03-20T09:25:35.943 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:35 smithi122 ceph-mon[73495]: 1.4 scrub ok 2024-03-20T09:25:35.943 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:35 smithi122 ceph-mon[73495]: 1.12 scrub starts 2024-03-20T09:25:35.943 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:35 smithi122 ceph-mon[73495]: 1.12 scrub ok 2024-03-20T09:25:35.943 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:35 smithi122 ceph-mon[73495]: 6.b scrub starts 2024-03-20T09:25:35.943 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:35 smithi122 ceph-mon[73495]: 1.d scrub starts 2024-03-20T09:25:35.944 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:35 smithi122 ceph-mon[73495]: 1.d scrub ok 2024-03-20T09:25:35.944 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:35 smithi122 ceph-mon[73495]: osdmap e745: 12 total, 12 up, 12 in 2024-03-20T09:25:35.947 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:25:35.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:35.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:25:36.054 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:25:36.055 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:25:36.090 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:36.115 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:25:36.131 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:25:36.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:36.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:25:36.167 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:25:36.201 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:25:36.241 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:25:36.244 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:25:36.281 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:36.287 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:25:36.307 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:25:36.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:36.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:25:36.322 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:25:36.356 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:25:36.397 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:25:36.409 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:25:36.409 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:25:36.410 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:25:36.410 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:25:36.410 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:25:36.410 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:25:36.410 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:25:36.410 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:25:36.410 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:25:36.410 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:25:36.410 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:25:36.410 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:25:36.413 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:25:36.513 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:36.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:36 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4035829651' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:25:36.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:36 smithi026 ceph-mon[68309]: pgmap v1815: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 967 op/s 2024-03-20T09:25:36.862 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 81 16 110813 110797 85.4826 58.3125 0.0126865 0.0116939 2024-03-20T09:25:36.898 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:25:36.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:36.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:25:36.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:36 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4035829651' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:25:36.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:36 smithi115 ceph-mon[72476]: pgmap v1815: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 967 op/s 2024-03-20T09:25:36.943 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:36 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4035829651' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:25:36.943 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:36 smithi122 ceph-mon[73495]: pgmap v1815: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 967 op/s 2024-03-20T09:25:37.006 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:25:37.058 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:37.358 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 8, 10, 11, 3, 4, 9] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 5] 2024-03-20T09:25:37.358 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:25:37.359 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_4 2024-03-20T09:25:37.359 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:25:37.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:37 smithi026 ceph-mon[68309]: 6.0 scrub starts 2024-03-20T09:25:37.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:37 smithi026 ceph-mon[68309]: 6.b scrub ok 2024-03-20T09:25:37.862 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 82 16 111803 111787 85.1946 61.875 0.00792416 0.0117339 2024-03-20T09:25:37.905 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:37 smithi115 ceph-mon[72476]: 6.0 scrub starts 2024-03-20T09:25:37.906 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:37 smithi115 ceph-mon[72476]: 6.b scrub ok 2024-03-20T09:25:37.943 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:37 smithi122 ceph-mon[73495]: 6.0 scrub starts 2024-03-20T09:25:37.943 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:37 smithi122 ceph-mon[73495]: 6.b scrub ok 2024-03-20T09:25:38.344 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:25:38.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:38.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:25:38.449 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:25:38.511 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:38.540 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:25:38.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:38.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:25:38.650 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:25:38.679 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:38.701 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:25:38.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:38.704 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:25:38.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:38 smithi026 ceph-mon[68309]: 1.1b scrub starts 2024-03-20T09:25:38.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:38 smithi026 ceph-mon[68309]: 1.1b scrub ok 2024-03-20T09:25:38.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:38 smithi026 ceph-mon[68309]: 6.a scrub starts 2024-03-20T09:25:38.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:38 smithi026 ceph-mon[68309]: pgmap v1816: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.7 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 880 op/s 2024-03-20T09:25:38.805 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:25:38.831 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:38 smithi122 ceph-mon[73495]: 1.1b scrub starts 2024-03-20T09:25:38.831 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:38 smithi122 ceph-mon[73495]: 1.1b scrub ok 2024-03-20T09:25:38.831 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:38 smithi122 ceph-mon[73495]: 6.a scrub starts 2024-03-20T09:25:38.831 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:38 smithi122 ceph-mon[73495]: pgmap v1816: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.7 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 880 op/s 2024-03-20T09:25:38.833 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:38.855 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:25:38.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:38.860 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:25:38.862 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 83 16 112801 112785 84.9196 62.375 0.00957767 0.0117722 2024-03-20T09:25:38.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:38 smithi115 ceph-mon[72476]: 1.1b scrub starts 2024-03-20T09:25:38.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:38 smithi115 ceph-mon[72476]: 1.1b scrub ok 2024-03-20T09:25:38.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:38 smithi115 ceph-mon[72476]: 6.a scrub starts 2024-03-20T09:25:38.907 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:38 smithi115 ceph-mon[72476]: pgmap v1816: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.7 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 880 op/s 2024-03-20T09:25:38.960 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:25:39.020 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:39.577 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:39 smithi026 ceph-mon[68309]: 6.0 scrub ok 2024-03-20T09:25:39.589 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:25:39.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:39.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:25:39.694 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:25:39.723 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:39.751 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:25:39.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:39.754 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:25:39.855 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:25:39.863 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 84 16 113915 113899 84.7375 69.625 0.0116657 0.0117979 2024-03-20T09:25:39.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:39 smithi115 ceph-mon[72476]: 6.0 scrub ok 2024-03-20T09:25:39.904 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:39.941 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:25:39.943 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:39 smithi122 ceph-mon[73495]: 6.0 scrub ok 2024-03-20T09:25:39.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:39.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:25:40.049 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:25:40.103 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:40.137 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:25:40.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:40.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:25:40.245 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:25:40.276 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:40.305 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:25:40.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:40.309 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:25:40.410 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:25:40.410 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:25:40.442 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:25:40.515 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:40.697 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:40 smithi026 ceph-mon[68309]: 6.a scrub ok 2024-03-20T09:25:40.697 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:40 smithi026 ceph-mon[68309]: pgmap v1817: 47 pgs: 47 active+clean; 6.8 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-03-20T09:25:40.724 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:25:40.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:40.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:25:40.834 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:25:40.862 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 85 16 114967 114951 84.514 65.75 0.0232844 0.0118286 2024-03-20T09:25:40.895 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:40.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:40 smithi115 ceph-mon[72476]: 6.a scrub ok 2024-03-20T09:25:40.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:40 smithi115 ceph-mon[72476]: pgmap v1817: 47 pgs: 47 active+clean; 6.8 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-03-20T09:25:40.944 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:40 smithi122 ceph-mon[73495]: 6.a scrub ok 2024-03-20T09:25:40.945 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:40 smithi122 ceph-mon[73495]: pgmap v1817: 47 pgs: 47 active+clean; 6.8 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-03-20T09:25:40.945 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:25:40.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:40.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:25:41.055 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:25:41.114 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:41.208 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:25:41.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:41.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:25:41.313 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:25:41.370 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:41.862 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 86 16 116019 116003 84.2958 65.75 0.0227925 0.0118587 2024-03-20T09:25:42.061 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:25:42.282 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:41 smithi026 ceph-mon[68309]: 1.8 scrub starts 2024-03-20T09:25:42.282 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:41 smithi026 ceph-mon[68309]: 1.8 scrub ok 2024-03-20T09:25:42.403 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:41 smithi115 ceph-mon[72476]: 1.8 scrub starts 2024-03-20T09:25:42.403 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:41 smithi115 ceph-mon[72476]: 1.8 scrub ok 2024-03-20T09:25:42.443 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:41 smithi122 ceph-mon[73495]: 1.8 scrub starts 2024-03-20T09:25:42.443 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:41 smithi122 ceph-mon[73495]: 1.8 scrub ok 2024-03-20T09:25:42.861 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:25:42.862 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 87 16 116971 116955 84.0107 59.5 0.0045498 0.0118993 2024-03-20T09:25:42.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:42.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:25:42.967 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:25:43.021 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:25:43.032 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:43.218 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:25:43.218 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:25:43.218 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: pgmap v1818: 47 pgs: 47 active+clean; 7.0 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.13k op/s 2024-03-20T09:25:43.218 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3839756201' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:25:43.219 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=1561764331,config=3599132250,health=3897446751,kv=3231526347}) 2024-03-20T09:25:43.219 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=388519642,logm=3362736358}) 2024-03-20T09:25:43.219 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1204434820}) 2024-03-20T09:25:43.219 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3541692793}) 2024-03-20T09:25:43.219 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1964017561}) 2024-03-20T09:25:43.219 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2473188776}) 2024-03-20T09:25:43.219 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2806681621}) 2024-03-20T09:25:43.219 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1002146227}) 2024-03-20T09:25:43.219 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=545049612}) 2024-03-20T09:25:43.219 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3933622711}) 2024-03-20T09:25:43.219 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2779631701}) 2024-03-20T09:25:43.219 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2759205561}) 2024-03-20T09:25:43.219 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1211326430}) 2024-03-20T09:25:43.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4108955081}) 2024-03-20T09:25:43.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3050465882}) 2024-03-20T09:25:43.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=972069925}) 2024-03-20T09:25:43.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1868893567}) 2024-03-20T09:25:43.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1605852469}) 2024-03-20T09:25:43.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3437494691}) 2024-03-20T09:25:43.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1476831712}) 2024-03-20T09:25:43.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1260788891}) 2024-03-20T09:25:43.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3021576420}) 2024-03-20T09:25:43.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3445947165}) 2024-03-20T09:25:43.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=382021765}) 2024-03-20T09:25:43.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=509867862}) 2024-03-20T09:25:43.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=886000126}) 2024-03-20T09:25:43.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3098687427}) 2024-03-20T09:25:43.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=389619492}) 2024-03-20T09:25:43.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3915131717}) 2024-03-20T09:25:43.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3378176393}) 2024-03-20T09:25:43.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=197649875}) 2024-03-20T09:25:43.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1254230482}) 2024-03-20T09:25:43.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1367524310}) 2024-03-20T09:25:43.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4057723315}) 2024-03-20T09:25:43.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2859428419}) 2024-03-20T09:25:43.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3880318998}) 2024-03-20T09:25:43.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3354611385}) 2024-03-20T09:25:43.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2812176016}) 2024-03-20T09:25:43.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1567715193}) 2024-03-20T09:25:43.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2645052133}) 2024-03-20T09:25:43.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2790227957}) 2024-03-20T09:25:43.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2727571723}) 2024-03-20T09:25:43.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=704814851}) 2024-03-20T09:25:43.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2589810568}) 2024-03-20T09:25:43.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1523314725}) 2024-03-20T09:25:43.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1656046966}) 2024-03-20T09:25:43.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401835930}) 2024-03-20T09:25:43.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1281124935}) 2024-03-20T09:25:43.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=492269322}) 2024-03-20T09:25:43.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4094514829}) 2024-03-20T09:25:43.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3816885347}) 2024-03-20T09:25:43.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1788763514}) 2024-03-20T09:25:43.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1853715137}) 2024-03-20T09:25:43.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1555668148}) 2024-03-20T09:25:43.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1710521867}) 2024-03-20T09:25:43.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1770273924}) 2024-03-20T09:25:43.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=434960087}) 2024-03-20T09:25:43.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=471637986}) 2024-03-20T09:25:43.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=386615153}) 2024-03-20T09:25:43.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=970537150}) 2024-03-20T09:25:43.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3156759238}) 2024-03-20T09:25:43.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3470384788}) 2024-03-20T09:25:43.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=919568145}) 2024-03-20T09:25:43.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=203418181}) 2024-03-20T09:25:43.223 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2140213959}) 2024-03-20T09:25:43.224 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=33,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=33} crc {logm=2563005543,mdsmap=3984425268,mgr=427508465,mgr_command_descs=3831600018,mgr_metadata=2702834764,mgrstat=1584661828,mon_config_key=4278569734}) 2024-03-20T09:25:43.224 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=323793648}) 2024-03-20T09:25:43.224 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1893836861}) 2024-03-20T09:25:43.224 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=45,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=37} crc {mon_config_key=4053473882,monmap=3481182261,osd_metadata=1898051788,osd_pg_creating=3928164913,osdmap=2653751743}) 2024-03-20T09:25:43.224 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=80} crc {osdmap=3817897769}) 2024-03-20T09:25:43.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:25:43.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:25:43.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: pgmap v1818: 47 pgs: 47 active+clean; 7.0 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.13k op/s 2024-03-20T09:25:43.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3839756201' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:25:43.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=1561764331,config=3599132250,health=3897446751,kv=3231526347}) 2024-03-20T09:25:43.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=388519642,logm=3362736358}) 2024-03-20T09:25:43.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1204434820}) 2024-03-20T09:25:43.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3541692793}) 2024-03-20T09:25:43.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1964017561}) 2024-03-20T09:25:43.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2473188776}) 2024-03-20T09:25:43.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2806681621}) 2024-03-20T09:25:43.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1002146227}) 2024-03-20T09:25:43.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=545049612}) 2024-03-20T09:25:43.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3933622711}) 2024-03-20T09:25:43.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2779631701}) 2024-03-20T09:25:43.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2759205561}) 2024-03-20T09:25:43.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1211326430}) 2024-03-20T09:25:43.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4108955081}) 2024-03-20T09:25:43.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3050465882}) 2024-03-20T09:25:43.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=972069925}) 2024-03-20T09:25:43.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1868893567}) 2024-03-20T09:25:43.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1605852469}) 2024-03-20T09:25:43.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3437494691}) 2024-03-20T09:25:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1476831712}) 2024-03-20T09:25:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1260788891}) 2024-03-20T09:25:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3021576420}) 2024-03-20T09:25:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3445947165}) 2024-03-20T09:25:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=382021765}) 2024-03-20T09:25:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=509867862}) 2024-03-20T09:25:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=886000126}) 2024-03-20T09:25:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3098687427}) 2024-03-20T09:25:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=389619492}) 2024-03-20T09:25:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3915131717}) 2024-03-20T09:25:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3378176393}) 2024-03-20T09:25:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=197649875}) 2024-03-20T09:25:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1254230482}) 2024-03-20T09:25:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1367524310}) 2024-03-20T09:25:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4057723315}) 2024-03-20T09:25:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2859428419}) 2024-03-20T09:25:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3880318998}) 2024-03-20T09:25:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3354611385}) 2024-03-20T09:25:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2812176016}) 2024-03-20T09:25:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1567715193}) 2024-03-20T09:25:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2645052133}) 2024-03-20T09:25:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2790227957}) 2024-03-20T09:25:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2727571723}) 2024-03-20T09:25:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=704814851}) 2024-03-20T09:25:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2589810568}) 2024-03-20T09:25:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1523314725}) 2024-03-20T09:25:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1656046966}) 2024-03-20T09:25:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401835930}) 2024-03-20T09:25:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1281124935}) 2024-03-20T09:25:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=492269322}) 2024-03-20T09:25:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4094514829}) 2024-03-20T09:25:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3816885347}) 2024-03-20T09:25:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1788763514}) 2024-03-20T09:25:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1853715137}) 2024-03-20T09:25:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1555668148}) 2024-03-20T09:25:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1710521867}) 2024-03-20T09:25:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1770273924}) 2024-03-20T09:25:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=434960087}) 2024-03-20T09:25:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=471637986}) 2024-03-20T09:25:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=386615153}) 2024-03-20T09:25:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=970537150}) 2024-03-20T09:25:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3156759238}) 2024-03-20T09:25:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3470384788}) 2024-03-20T09:25:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=919568145}) 2024-03-20T09:25:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=203418181}) 2024-03-20T09:25:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2140213959}) 2024-03-20T09:25:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=33,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=33} crc {logm=2563005543,mdsmap=3984425268,mgr=427508465,mgr_command_descs=3831600018,mgr_metadata=2702834764,mgrstat=1584661828,mon_config_key=4278569734}) 2024-03-20T09:25:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=323793648}) 2024-03-20T09:25:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1893836861}) 2024-03-20T09:25:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=45,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=37} crc {mon_config_key=4053473882,monmap=3481182261,osd_metadata=1898051788,osd_pg_creating=3928164913,osdmap=2653751743}) 2024-03-20T09:25:43.409 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=80} crc {osdmap=3817897769}) 2024-03-20T09:25:43.445 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:25:43.445 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:25:43.445 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: pgmap v1818: 47 pgs: 47 active+clean; 7.0 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.13k op/s 2024-03-20T09:25:43.445 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3839756201' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:25:43.445 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=1561764331,config=3599132250,health=3897446751,kv=3231526347}) 2024-03-20T09:25:43.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=388519642,logm=3362736358}) 2024-03-20T09:25:43.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1204434820}) 2024-03-20T09:25:43.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3541692793}) 2024-03-20T09:25:43.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1964017561}) 2024-03-20T09:25:43.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2473188776}) 2024-03-20T09:25:43.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2806681621}) 2024-03-20T09:25:43.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1002146227}) 2024-03-20T09:25:43.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=545049612}) 2024-03-20T09:25:43.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3933622711}) 2024-03-20T09:25:43.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2779631701}) 2024-03-20T09:25:43.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2759205561}) 2024-03-20T09:25:43.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1211326430}) 2024-03-20T09:25:43.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4108955081}) 2024-03-20T09:25:43.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3050465882}) 2024-03-20T09:25:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=972069925}) 2024-03-20T09:25:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1868893567}) 2024-03-20T09:25:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1605852469}) 2024-03-20T09:25:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3437494691}) 2024-03-20T09:25:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1476831712}) 2024-03-20T09:25:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1260788891}) 2024-03-20T09:25:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3021576420}) 2024-03-20T09:25:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3445947165}) 2024-03-20T09:25:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=382021765}) 2024-03-20T09:25:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=509867862}) 2024-03-20T09:25:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=886000126}) 2024-03-20T09:25:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3098687427}) 2024-03-20T09:25:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=389619492}) 2024-03-20T09:25:43.447 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3915131717}) 2024-03-20T09:25:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3378176393}) 2024-03-20T09:25:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=197649875}) 2024-03-20T09:25:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1254230482}) 2024-03-20T09:25:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1367524310}) 2024-03-20T09:25:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4057723315}) 2024-03-20T09:25:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2859428419}) 2024-03-20T09:25:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3880318998}) 2024-03-20T09:25:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3354611385}) 2024-03-20T09:25:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2812176016}) 2024-03-20T09:25:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1567715193}) 2024-03-20T09:25:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2645052133}) 2024-03-20T09:25:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2790227957}) 2024-03-20T09:25:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2727571723}) 2024-03-20T09:25:43.448 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=704814851}) 2024-03-20T09:25:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2589810568}) 2024-03-20T09:25:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1523314725}) 2024-03-20T09:25:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1656046966}) 2024-03-20T09:25:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=401835930}) 2024-03-20T09:25:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1281124935}) 2024-03-20T09:25:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=492269322}) 2024-03-20T09:25:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4094514829}) 2024-03-20T09:25:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3816885347}) 2024-03-20T09:25:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1788763514}) 2024-03-20T09:25:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1853715137}) 2024-03-20T09:25:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1555668148}) 2024-03-20T09:25:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1710521867}) 2024-03-20T09:25:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1770273924}) 2024-03-20T09:25:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=434960087}) 2024-03-20T09:25:43.449 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=471637986}) 2024-03-20T09:25:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=386615153}) 2024-03-20T09:25:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=970537150}) 2024-03-20T09:25:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3156759238}) 2024-03-20T09:25:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3470384788}) 2024-03-20T09:25:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=919568145}) 2024-03-20T09:25:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=203418181}) 2024-03-20T09:25:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2140213959}) 2024-03-20T09:25:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=33,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=33} crc {logm=2563005543,mdsmap=3984425268,mgr=427508465,mgr_command_descs=3831600018,mgr_metadata=2702834764,mgrstat=1584661828,mon_config_key=4278569734}) 2024-03-20T09:25:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=323793648}) 2024-03-20T09:25:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1893836861}) 2024-03-20T09:25:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=45,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=37} crc {mon_config_key=4053473882,monmap=3481182261,osd_metadata=1898051788,osd_pg_creating=3928164913,osdmap=2653751743}) 2024-03-20T09:25:43.450 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=80} crc {osdmap=3817897769}) 2024-03-20T09:25:43.863 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 88 16 118059 118043 83.8287 68 0.00906287 0.0119253 2024-03-20T09:25:44.213 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:25:44.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:44.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:25:44.256 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:43 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:25:44.256 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:43 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3839756201' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:25:44.256 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:43 smithi026 ceph-mon[68309]: osdmap e746: 12 total, 12 up, 12 in 2024-03-20T09:25:44.324 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:25:44.380 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:44.403 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:43 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:25:44.403 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:43 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3839756201' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:25:44.403 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:43 smithi115 ceph-mon[72476]: osdmap e746: 12 total, 12 up, 12 in 2024-03-20T09:25:44.427 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:25:44.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:44.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:25:44.443 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:43 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:25:44.443 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:43 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3839756201' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:25:44.443 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:43 smithi122 ceph-mon[73495]: osdmap e746: 12 total, 12 up, 12 in 2024-03-20T09:25:44.532 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:25:44.571 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:44.597 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:25:44.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:44.600 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:25:44.701 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:25:44.748 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:44.863 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 89 16 119118 119102 83.6305 66.1875 0.0166385 0.0119534 2024-03-20T09:25:44.976 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:25:44.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:44.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:25:45.088 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:25:45.119 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:45.151 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:25:45.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:45.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:25:45.217 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:45 smithi026 ceph-mon[68309]: pgmap v1820: 47 pgs: 47 active+clean; 7.0 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 952 op/s 2024-03-20T09:25:45.258 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:25:45.321 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:45.400 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:25:45.422 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:45 smithi115 ceph-mon[72476]: pgmap v1820: 47 pgs: 47 active+clean; 7.0 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 952 op/s 2024-03-20T09:25:45.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:45.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:25:45.443 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:45 smithi122 ceph-mon[73495]: pgmap v1820: 47 pgs: 47 active+clean; 7.0 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 952 op/s 2024-03-20T09:25:45.524 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:25:45.559 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:45.588 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:25:45.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:45.592 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:25:45.693 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:25:45.762 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:45.863 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 90 7 120106 120099 83.3935 62.3125 0.0130265 0.0119829 2024-03-20T09:25:46.123 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:25:46.129 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:25:46.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:46.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:25:46.167 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:25:46.167 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:25:46.167 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-03-20T09:25:46.167 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-03-20T09:25:46.167 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:25:46.233 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:25:46.257 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:25:46.261 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:46.280 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:25:46.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:46.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:25:46.364 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:25:46.365 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:25:46.384 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:25:46.418 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:46.863 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 91 1 120106 120105 82.4812 0.375 0.1372 0.0119912 2024-03-20T09:25:47.370 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:25:47.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:47.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:25:47.477 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:25:47.508 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:47 smithi115 ceph-mon[72476]: pgmap v1821: 47 pgs: 47 active+clean; 7.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-03-20T09:25:47.510 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:47.532 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:25:47.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:47.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:25:47.578 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:47 smithi026 ceph-mon[68309]: pgmap v1821: 47 pgs: 47 active+clean; 7.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-03-20T09:25:47.637 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:25:47.667 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:47.688 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:25:47.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:47.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:25:47.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:47 smithi122 ceph-mon[73495]: pgmap v1821: 47 pgs: 47 active+clean; 7.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-03-20T09:25:47.793 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:25:47.825 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:47.846 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:25:47.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:47.849 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:25:47.863 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Total time run: 91.5437 2024-03-20T09:25:47.863 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Total writes made: 120106 2024-03-20T09:25:47.863 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Write size: 65536 2024-03-20T09:25:47.863 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Object size: 65536 2024-03-20T09:25:47.863 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Bandwidth (MB/sec): 82.0004 2024-03-20T09:25:47.864 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Stddev Bandwidth: 48.5344 2024-03-20T09:25:47.864 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Max bandwidth (MB/sec): 239.25 2024-03-20T09:25:47.864 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Min bandwidth (MB/sec): 0 2024-03-20T09:25:47.864 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Average IOPS: 1312 2024-03-20T09:25:47.864 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Stddev IOPS: 776.562 2024-03-20T09:25:47.864 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Max IOPS: 3828 2024-03-20T09:25:47.864 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Min IOPS: 0 2024-03-20T09:25:47.864 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Average Latency(s): 0.0120041 2024-03-20T09:25:47.864 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Stddev Latency(s): 0.135323 2024-03-20T09:25:47.864 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Max latency(s): 11.7007 2024-03-20T09:25:47.864 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Min latency(s): 0.00104258 2024-03-20T09:25:47.869 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Cleaning up (deleting benchmark objects) 2024-03-20T09:25:47.950 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:25:47.986 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:48.017 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:25:48.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:48.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:25:48.122 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:25:48.152 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:48.175 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:25:48.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:48.179 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:25:48.253 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-20T09:25:48.280 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:25:48.317 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:48.349 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:25:48.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:48.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:25:48.443 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:48 smithi122 ceph-mon[73495]: from='client.16443 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:25:48.453 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:25:48.468 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:48 smithi026 ceph-mon[68309]: from='client.16443 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:25:48.515 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:48.539 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:25:48.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:48.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:25:48.543 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:48 smithi115 ceph-mon[72476]: from='client.16443 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:25:48.578 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":1821,"stamp":"2024-03-20T09:25:46.203689+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7759594016,"num_objects":118395,"num_object_clones":0,"num_object_copies":355185,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":118395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":118393,"num_write_kb":7577152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122641,"ondisk_log_size":122641,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":29783080,"kb_used_data":23443268,"kb_used_omap":198,"kb_used_meta":6339513,"kb_avail":1095158744,"statfs":{"total":1151940427776,"available":1121442553856,"internally_reserved":0,"allocated":24005906432,"data_stored":24002554396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":203677,"internal_metadata":6491661411},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,2,2,11,5,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":662306816,"num_objects":10106,"num_object_clones":0,"num_object_copies":30318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10106,"num_write_kb":646784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.007987"},"pg_stats":[{"pgid":"1.1e","version":"20'137","reported_seq":913,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-03-20T09:25:42.999254+0000","last_change":"2024-03-20T09:25:33.135367+0000","last_active":"2024-03-20T09:25:42.999254+0000","last_peered":"2024-03-20T09:25:42.999254+0000","last_clean":"2024-03-20T09:25:42.999254+0000","last_became_active":"2024-03-20T09:13:20.071655+0000","last_became_peered":"2024-03-20T09:13:20.071655+0000","last_unstale":"2024-03-20T09:25:42.999254+0000","last_undegraded":"2024-03-20T09:25:42.999254+0000","last_fullsized":"2024-03-20T09:25:42.999254+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:33.135222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:25:33.135222+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:27:02.620727+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'137","reported_seq":1086,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-03-20T09:25:43.002682+0000","last_change":"2024-03-20T09:25:18.228015+0000","last_active":"2024-03-20T09:25:43.002682+0000","last_peered":"2024-03-20T09:25:43.002682+0000","last_clean":"2024-03-20T09:25:43.002682+0000","last_became_active":"2024-03-20T09:07:42.013245+0000","last_became_peered":"2024-03-20T09:07:42.013245+0000","last_unstale":"2024-03-20T09:25:43.002682+0000","last_undegraded":"2024-03-20T09:25:43.002682+0000","last_fullsized":"2024-03-20T09:25:43.002682+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:18.227887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:25:18.227887+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:35.117677+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"20'137","reported_seq":1551,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-03-20T09:25:35.483272+0000","last_change":"2024-03-20T09:25:35.483272+0000","last_active":"2024-03-20T09:25:35.483272+0000","last_peered":"2024-03-20T09:25:35.483272+0000","last_clean":"2024-03-20T09:25:35.483272+0000","last_became_active":"2024-03-20T09:25:35.482959+0000","last_became_peered":"2024-03-20T09:25:35.482959+0000","last_unstale":"2024-03-20T09:25:35.483272+0000","last_undegraded":"2024-03-20T09:25:35.483272+0000","last_fullsized":"2024-03-20T09:25:35.483272+0000","mapping_epoch":744,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":745,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:14.081976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:25:14.081976+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:34.701023+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,1],"acting":[8,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.19","version":"20'137","reported_seq":1588,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-03-20T09:25:35.449732+0000","last_change":"2024-03-20T09:25:15.799832+0000","last_active":"2024-03-20T09:25:35.449732+0000","last_peered":"2024-03-20T09:25:35.449732+0000","last_clean":"2024-03-20T09:25:35.449732+0000","last_became_active":"2024-03-20T09:07:41.962680+0000","last_became_peered":"2024-03-20T09:07:41.962680+0000","last_unstale":"2024-03-20T09:25:35.449732+0000","last_undegraded":"2024-03-20T09:25:35.449732+0000","last_fullsized":"2024-03-20T09:25:35.449732+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:15.799703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:25:15.799703+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:36.932837+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"20'137","reported_seq":1050,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-03-20T09:25:35.452370+0000","last_change":"2024-03-20T09:25:21.295358+0000","last_active":"2024-03-20T09:25:35.452370+0000","last_peered":"2024-03-20T09:25:35.452370+0000","last_clean":"2024-03-20T09:25:35.452370+0000","last_became_active":"2024-03-20T09:10:35.277481+0000","last_became_peered":"2024-03-20T09:10:35.277481+0000","last_unstale":"2024-03-20T09:25:35.452370+0000","last_undegraded":"2024-03-20T09:25:35.452370+0000","last_fullsized":"2024-03-20T09:25:35.452370+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:21.295237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:25:21.295237+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:46.456323+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1b","version":"20'137","reported_seq":1388,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-03-20T09:25:37.132789+0000","last_change":"2024-03-20T09:25:37.132818+0000","last_active":"2024-03-20T09:25:37.132789+0000","last_peered":"2024-03-20T09:25:37.132789+0000","last_clean":"2024-03-20T09:25:37.132789+0000","last_became_active":"2024-03-20T09:24:16.445287+0000","last_became_peered":"2024-03-20T09:24:16.445287+0000","last_unstale":"2024-03-20T09:25:37.132789+0000","last_undegraded":"2024-03-20T09:25:37.132789+0000","last_fullsized":"2024-03-20T09:25:37.132789+0000","mapping_epoch":718,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:37.132721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:25:37.132721+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:50.243021+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"6.f","version":"746'7399","reported_seq":7554,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-03-20T09:25:43.229088+0000","last_change":"2024-03-20T09:25:27.830528+0000","last_active":"2024-03-20T09:25:43.229088+0000","last_peered":"2024-03-20T09:25:43.229088+0000","last_clean":"2024-03-20T09:25:43.229088+0000","last_became_active":"2024-03-20T09:24:04.030514+0000","last_became_peered":"2024-03-20T09:24:04.030514+0000","last_unstale":"2024-03-20T09:25:43.229088+0000","last_undegraded":"2024-03-20T09:25:43.229088+0000","last_fullsized":"2024-03-20T09:25:43.229088+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"739'6380","last_scrub_stamp":"2024-03-20T09:25:27.830146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:27.830146+0000","objects_scrubbed":6341,"log_size":7399,"log_dups_size":0,"ondisk_log_size":7399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:45.839673+0000","scrub_duration":1684,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484900864,"num_objects":7399,"num_object_clones":0,"num_object_copies":22197,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7399,"num_write_kb":473536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"20'137","reported_seq":1050,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-03-20T09:25:41.351717+0000","last_change":"2024-03-20T09:25:41.351717+0000","last_active":"2024-03-20T09:25:41.351717+0000","last_peered":"2024-03-20T09:25:41.351717+0000","last_clean":"2024-03-20T09:25:41.351717+0000","last_became_active":"2024-03-20T09:10:35.215772+0000","last_became_peered":"2024-03-20T09:10:35.215772+0000","last_unstale":"2024-03-20T09:25:41.351717+0000","last_undegraded":"2024-03-20T09:25:41.351717+0000","last_fullsized":"2024-03-20T09:25:41.351717+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:41.351598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:25:41.351598+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:47.469807+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"746'7369","reported_seq":7519,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-03-20T09:25:45.449249+0000","last_change":"2024-03-20T09:25:20.220593+0000","last_active":"2024-03-20T09:25:45.449249+0000","last_peered":"2024-03-20T09:25:45.449249+0000","last_clean":"2024-03-20T09:25:45.449249+0000","last_became_active":"2024-03-20T09:24:03.382302+0000","last_became_peered":"2024-03-20T09:24:03.382302+0000","last_unstale":"2024-03-20T09:25:45.449249+0000","last_undegraded":"2024-03-20T09:25:45.449249+0000","last_fullsized":"2024-03-20T09:25:45.449249+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"736'5773","last_scrub_stamp":"2024-03-20T09:25:20.220414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:20.220414+0000","objects_scrubbed":5729,"log_size":7369,"log_dups_size":0,"ondisk_log_size":7369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:27.281436+0000","scrub_duration":1658,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482934784,"num_objects":7369,"num_object_clones":0,"num_object_copies":22107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7369,"num_write_kb":471616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"20'137","reported_seq":1627,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-03-20T09:25:43.000825+0000","last_change":"2024-03-20T09:25:14.298374+0000","last_active":"2024-03-20T09:25:43.000825+0000","last_peered":"2024-03-20T09:25:43.000825+0000","last_clean":"2024-03-20T09:25:43.000825+0000","last_became_active":"2024-03-20T09:20:25.341750+0000","last_became_peered":"2024-03-20T09:20:25.341750+0000","last_unstale":"2024-03-20T09:25:43.000825+0000","last_undegraded":"2024-03-20T09:25:43.000825+0000","last_fullsized":"2024-03-20T09:25:43.000825+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:14.298275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:25:14.298275+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:28.207301+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,2],"acting":[9,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.d","version":"745'7240","reported_seq":7387,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-03-20T09:25:42.056738+0000","last_change":"2024-03-20T09:25:19.682021+0000","last_active":"2024-03-20T09:25:42.056738+0000","last_peered":"2024-03-20T09:25:42.056738+0000","last_clean":"2024-03-20T09:25:42.056738+0000","last_became_active":"2024-03-20T09:24:03.381679+0000","last_became_peered":"2024-03-20T09:24:03.381679+0000","last_unstale":"2024-03-20T09:25:42.056738+0000","last_undegraded":"2024-03-20T09:25:42.056738+0000","last_fullsized":"2024-03-20T09:25:42.056738+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"736'5792","last_scrub_stamp":"2024-03-20T09:25:19.681737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:19.681737+0000","objects_scrubbed":5741,"log_size":7240,"log_dups_size":0,"ondisk_log_size":7240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:30.845927+0000","scrub_duration":1457,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":474480640,"num_objects":7240,"num_object_clones":0,"num_object_copies":21720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7240,"num_write_kb":463360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"20'137","reported_seq":316,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-03-20T09:25:43.000911+0000","last_change":"2024-03-20T09:24:17.164244+0000","last_active":"2024-03-20T09:25:43.000911+0000","last_peered":"2024-03-20T09:25:43.000911+0000","last_clean":"2024-03-20T09:25:43.000911+0000","last_became_active":"2024-03-20T09:24:16.444232+0000","last_became_peered":"2024-03-20T09:24:16.444232+0000","last_unstale":"2024-03-20T09:25:43.000911+0000","last_undegraded":"2024-03-20T09:25:43.000911+0000","last_fullsized":"2024-03-20T09:25:43.000911+0000","mapping_epoch":718,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:17.164131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:17.164131+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:45.596474+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.3","version":"746'7460","reported_seq":7614,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-03-20T09:25:45.858417+0000","last_change":"2024-03-20T09:25:25.350161+0000","last_active":"2024-03-20T09:25:45.858417+0000","last_peered":"2024-03-20T09:25:45.858417+0000","last_clean":"2024-03-20T09:25:45.858417+0000","last_became_active":"2024-03-20T09:24:03.379444+0000","last_became_peered":"2024-03-20T09:24:03.379444+0000","last_unstale":"2024-03-20T09:25:45.858417+0000","last_undegraded":"2024-03-20T09:25:45.858417+0000","last_fullsized":"2024-03-20T09:25:45.858417+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"739'6169","last_scrub_stamp":"2024-03-20T09:25:25.349835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:25.349835+0000","objects_scrubbed":6123,"log_size":7460,"log_dups_size":0,"ondisk_log_size":7460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:48.751413+0000","scrub_duration":1411,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":488898560,"num_objects":7460,"num_object_clones":0,"num_object_copies":22380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7460,"num_write_kb":477440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,11],"acting":[6,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"20'137","reported_seq":1581,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-03-20T09:25:42.999059+0000","last_change":"2024-03-20T09:25:34.098294+0000","last_active":"2024-03-20T09:25:42.999059+0000","last_peered":"2024-03-20T09:25:42.999059+0000","last_clean":"2024-03-20T09:25:42.999059+0000","last_became_active":"2024-03-20T09:12:14.306384+0000","last_became_peered":"2024-03-20T09:12:14.306384+0000","last_unstale":"2024-03-20T09:25:42.999059+0000","last_undegraded":"2024-03-20T09:25:42.999059+0000","last_fullsized":"2024-03-20T09:25:42.999059+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:34.098199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:25:34.098199+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:48.569329+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"746'7505","reported_seq":7652,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-03-20T09:25:45.466221+0000","last_change":"2024-03-20T09:25:14.990095+0000","last_active":"2024-03-20T09:25:45.466221+0000","last_peered":"2024-03-20T09:25:45.466221+0000","last_clean":"2024-03-20T09:25:45.466221+0000","last_became_active":"2024-03-20T09:24:03.382365+0000","last_became_peered":"2024-03-20T09:24:03.382365+0000","last_unstale":"2024-03-20T09:25:45.466221+0000","last_undegraded":"2024-03-20T09:25:45.466221+0000","last_fullsized":"2024-03-20T09:25:45.466221+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"733'5454","last_scrub_stamp":"2024-03-20T09:25:14.989721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:14.989721+0000","objects_scrubbed":5409,"log_size":7505,"log_dups_size":0,"ondisk_log_size":7505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:21.444399+0000","scrub_duration":1387,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":491847680,"num_objects":7505,"num_object_clones":0,"num_object_copies":22515,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7505,"num_write_kb":480320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"20'137","reported_seq":1474,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-03-20T09:25:35.449744+0000","last_change":"2024-03-20T09:25:29.690984+0000","last_active":"2024-03-20T09:25:35.449744+0000","last_peered":"2024-03-20T09:25:35.449744+0000","last_clean":"2024-03-20T09:25:35.449744+0000","last_became_active":"2024-03-20T09:07:37.003722+0000","last_became_peered":"2024-03-20T09:07:37.003722+0000","last_unstale":"2024-03-20T09:25:35.449744+0000","last_undegraded":"2024-03-20T09:25:35.449744+0000","last_fullsized":"2024-03-20T09:25:35.449744+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:29.690849+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:19:03.805185+0000","last_clean_scrub_stamp":"2024-03-20T09:25:29.690849+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:49.197136+0000","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"745'7218","reported_seq":7368,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-03-20T09:25:41.424730+0000","last_change":"2024-03-20T09:25:24.081645+0000","last_active":"2024-03-20T09:25:41.424730+0000","last_peered":"2024-03-20T09:25:41.424730+0000","last_clean":"2024-03-20T09:25:41.424730+0000","last_became_active":"2024-03-20T09:24:04.030836+0000","last_became_peered":"2024-03-20T09:24:04.030836+0000","last_unstale":"2024-03-20T09:25:41.424730+0000","last_undegraded":"2024-03-20T09:25:41.424730+0000","last_fullsized":"2024-03-20T09:25:41.424730+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"738'6080","last_scrub_stamp":"2024-03-20T09:25:24.081378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:24.081378+0000","objects_scrubbed":6048,"log_size":7218,"log_dups_size":0,"ondisk_log_size":7218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:45.173644+0000","scrub_duration":958,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":473038848,"num_objects":7218,"num_object_clones":0,"num_object_copies":21654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7218,"num_write_kb":461952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"20'137","reported_seq":1509,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-03-20T09:25:35.449723+0000","last_change":"2024-03-20T09:24:31.549717+0000","last_active":"2024-03-20T09:25:35.449723+0000","last_peered":"2024-03-20T09:25:35.449723+0000","last_clean":"2024-03-20T09:25:35.449723+0000","last_became_active":"2024-03-20T09:24:16.442503+0000","last_became_peered":"2024-03-20T09:24:16.442503+0000","last_unstale":"2024-03-20T09:25:35.449723+0000","last_undegraded":"2024-03-20T09:25:35.449723+0000","last_fullsized":"2024-03-20T09:25:35.449723+0000","mapping_epoch":718,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:31.549603+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:24:31.549603+0000","last_clean_scrub_stamp":"2024-03-20T09:24:31.549603+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:47.943434+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.4","version":"746'7517","reported_seq":7668,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-03-20T09:25:44.626307+0000","last_change":"2024-03-20T09:25:21.190820+0000","last_active":"2024-03-20T09:25:44.626307+0000","last_peered":"2024-03-20T09:25:44.626307+0000","last_clean":"2024-03-20T09:25:44.626307+0000","last_became_active":"2024-03-20T09:24:03.379675+0000","last_became_peered":"2024-03-20T09:24:03.379675+0000","last_unstale":"2024-03-20T09:25:44.626307+0000","last_undegraded":"2024-03-20T09:25:44.626307+0000","last_fullsized":"2024-03-20T09:25:44.626307+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"736'5943","last_scrub_stamp":"2024-03-20T09:25:21.190503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:21.190503+0000","objects_scrubbed":5872,"log_size":7517,"log_dups_size":0,"ondisk_log_size":7517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:30.464714+0000","scrub_duration":2104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":492634112,"num_objects":7517,"num_object_clones":0,"num_object_copies":22551,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7517,"num_write_kb":481088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"20'137","reported_seq":1494,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-03-20T09:25:43.003328+0000","last_change":"2024-03-20T09:25:17.292883+0000","last_active":"2024-03-20T09:25:43.003328+0000","last_peered":"2024-03-20T09:25:43.003328+0000","last_clean":"2024-03-20T09:25:43.003328+0000","last_became_active":"2024-03-20T09:19:54.499210+0000","last_became_peered":"2024-03-20T09:19:54.499210+0000","last_unstale":"2024-03-20T09:25:43.003328+0000","last_undegraded":"2024-03-20T09:25:43.003328+0000","last_fullsized":"2024-03-20T09:25:43.003328+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:17.292735+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:18.788771+0000","last_clean_scrub_stamp":"2024-03-20T09:25:17.292735+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:46.814695+0000","scrub_duration":214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"745'7459","reported_seq":7611,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-03-20T09:25:42.027148+0000","last_change":"2024-03-20T09:25:24.693533+0000","last_active":"2024-03-20T09:25:42.027148+0000","last_peered":"2024-03-20T09:25:42.027148+0000","last_clean":"2024-03-20T09:25:42.027148+0000","last_became_active":"2024-03-20T09:24:03.380050+0000","last_became_peered":"2024-03-20T09:24:03.380050+0000","last_unstale":"2024-03-20T09:25:42.027148+0000","last_undegraded":"2024-03-20T09:25:42.027148+0000","last_fullsized":"2024-03-20T09:25:42.027148+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"738'6270","last_scrub_stamp":"2024-03-20T09:25:24.693262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:24.693262+0000","objects_scrubbed":6219,"log_size":7459,"log_dups_size":0,"ondisk_log_size":7459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:40.507377+0000","scrub_duration":1532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":488833024,"num_objects":7459,"num_object_clones":0,"num_object_copies":22377,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7459,"num_write_kb":477376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":1552,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-03-20T09:25:43.000227+0000","last_change":"2024-03-20T09:25:31.128237+0000","last_active":"2024-03-20T09:25:43.000227+0000","last_peered":"2024-03-20T09:25:43.000227+0000","last_clean":"2024-03-20T09:25:43.000227+0000","last_became_active":"2024-03-20T09:12:14.305989+0000","last_became_peered":"2024-03-20T09:12:14.305989+0000","last_unstale":"2024-03-20T09:25:43.000227+0000","last_undegraded":"2024-03-20T09:25:43.000227+0000","last_fullsized":"2024-03-20T09:25:43.000227+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:31.128133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:25:31.128133+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:33.962116+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.5","version":"746'7500","reported_seq":7659,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-03-20T09:25:45.834581+0000","last_change":"2024-03-20T09:25:32.364834+0000","last_active":"2024-03-20T09:25:45.834581+0000","last_peered":"2024-03-20T09:25:45.834581+0000","last_clean":"2024-03-20T09:25:45.834581+0000","last_became_active":"2024-03-20T09:24:04.030820+0000","last_became_peered":"2024-03-20T09:24:04.030820+0000","last_unstale":"2024-03-20T09:25:45.834581+0000","last_undegraded":"2024-03-20T09:25:45.834581+0000","last_fullsized":"2024-03-20T09:25:45.834581+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"741'6653","last_scrub_stamp":"2024-03-20T09:25:32.364492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:32.364492+0000","objects_scrubbed":6600,"log_size":7500,"log_dups_size":0,"ondisk_log_size":7500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:56.726232+0000","scrub_duration":1518,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":491520000,"num_objects":7500,"num_object_clones":0,"num_object_copies":22500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7500,"num_write_kb":480000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"20'137","reported_seq":1809,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-03-20T09:25:43.000755+0000","last_change":"2024-03-20T09:25:29.866424+0000","last_active":"2024-03-20T09:25:43.000755+0000","last_peered":"2024-03-20T09:25:43.000755+0000","last_clean":"2024-03-20T09:25:43.000755+0000","last_became_active":"2024-03-20T09:19:54.492879+0000","last_became_peered":"2024-03-20T09:19:54.492879+0000","last_unstale":"2024-03-20T09:25:43.000755+0000","last_undegraded":"2024-03-20T09:25:43.000755+0000","last_fullsized":"2024-03-20T09:25:43.000755+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:29.866281+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:19.513511+0000","last_clean_scrub_stamp":"2024-03-20T09:25:29.866281+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:50.346246+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"746'7547","reported_seq":7702,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-03-20T09:25:44.962829+0000","last_change":"2024-03-20T09:25:29.595209+0000","last_active":"2024-03-20T09:25:44.962829+0000","last_peered":"2024-03-20T09:25:44.962829+0000","last_clean":"2024-03-20T09:25:44.962829+0000","last_became_active":"2024-03-20T09:24:03.378429+0000","last_became_peered":"2024-03-20T09:24:03.378429+0000","last_unstale":"2024-03-20T09:25:44.962829+0000","last_undegraded":"2024-03-20T09:25:44.962829+0000","last_fullsized":"2024-03-20T09:25:44.962829+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'6530","last_scrub_stamp":"2024-03-20T09:25:29.595052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:29.595052+0000","objects_scrubbed":6463,"log_size":7547,"log_dups_size":0,"ondisk_log_size":7547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:57.562548+0000","scrub_duration":2087,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":494600192,"num_objects":7547,"num_object_clones":0,"num_object_copies":22641,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7547,"num_write_kb":483008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"20'137","reported_seq":969,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-03-20T09:25:42.995557+0000","last_change":"2024-03-20T09:25:20.892620+0000","last_active":"2024-03-20T09:25:42.995557+0000","last_peered":"2024-03-20T09:25:42.995557+0000","last_clean":"2024-03-20T09:25:42.995557+0000","last_became_active":"2024-03-20T09:15:34.225022+0000","last_became_peered":"2024-03-20T09:15:34.225022+0000","last_unstale":"2024-03-20T09:25:42.995557+0000","last_undegraded":"2024-03-20T09:25:42.995557+0000","last_fullsized":"2024-03-20T09:25:42.995557+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:20.892503+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:20:12.169658+0000","last_clean_scrub_stamp":"2024-03-20T09:25:20.892503+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:50.174655+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,5],"acting":[6,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"746'7304","reported_seq":7464,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-03-20T09:25:43.741386+0000","last_change":"2024-03-20T09:25:38.185259+0000","last_active":"2024-03-20T09:25:43.741386+0000","last_peered":"2024-03-20T09:25:43.741386+0000","last_clean":"2024-03-20T09:25:43.741386+0000","last_became_active":"2024-03-20T09:24:03.379597+0000","last_became_peered":"2024-03-20T09:24:03.379597+0000","last_unstale":"2024-03-20T09:25:43.741386+0000","last_undegraded":"2024-03-20T09:25:43.741386+0000","last_fullsized":"2024-03-20T09:25:43.741386+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"745'6949","last_scrub_stamp":"2024-03-20T09:25:38.184935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:38.184935+0000","objects_scrubbed":6887,"log_size":7304,"log_dups_size":0,"ondisk_log_size":7304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:27:07.941673+0000","scrub_duration":2351,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":478674944,"num_objects":7304,"num_object_clones":0,"num_object_copies":21912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7304,"num_write_kb":467456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"20'137","reported_seq":1505,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-03-20T09:25:42.995817+0000","last_change":"2024-03-20T09:25:16.978831+0000","last_active":"2024-03-20T09:25:42.995817+0000","last_peered":"2024-03-20T09:25:42.995817+0000","last_clean":"2024-03-20T09:25:42.995817+0000","last_became_active":"2024-03-20T09:13:27.087760+0000","last_became_peered":"2024-03-20T09:13:27.087760+0000","last_unstale":"2024-03-20T09:25:42.995817+0000","last_undegraded":"2024-03-20T09:25:42.995817+0000","last_fullsized":"2024-03-20T09:25:42.995817+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:16.978732+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:14:10.560890+0000","last_clean_scrub_stamp":"2024-03-20T09:25:16.978732+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:25.771597+0000","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"746'7683","reported_seq":7832,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-03-20T09:25:44.941206+0000","last_change":"2024-03-20T09:25:16.245567+0000","last_active":"2024-03-20T09:25:44.941206+0000","last_peered":"2024-03-20T09:25:44.941206+0000","last_clean":"2024-03-20T09:25:44.941206+0000","last_became_active":"2024-03-20T09:24:03.380630+0000","last_became_peered":"2024-03-20T09:24:03.380630+0000","last_unstale":"2024-03-20T09:25:44.941206+0000","last_undegraded":"2024-03-20T09:25:44.941206+0000","last_fullsized":"2024-03-20T09:25:44.941206+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"733'5704","last_scrub_stamp":"2024-03-20T09:25:16.245344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:16.245344+0000","objects_scrubbed":5644,"log_size":7683,"log_dups_size":0,"ondisk_log_size":7683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:40.292691+0000","scrub_duration":1702,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503513088,"num_objects":7683,"num_object_clones":0,"num_object_copies":23049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7683,"num_write_kb":491712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"20'137","reported_seq":1357,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-03-20T09:25:43.000023+0000","last_change":"2024-03-20T09:25:17.104512+0000","last_active":"2024-03-20T09:25:43.000023+0000","last_peered":"2024-03-20T09:25:43.000023+0000","last_clean":"2024-03-20T09:25:43.000023+0000","last_became_active":"2024-03-20T09:13:20.070596+0000","last_became_peered":"2024-03-20T09:13:20.070596+0000","last_unstale":"2024-03-20T09:25:43.000023+0000","last_undegraded":"2024-03-20T09:25:43.000023+0000","last_fullsized":"2024-03-20T09:25:43.000023+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:17.104330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:25:17.104330+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:23.942913+0000","scrub_duration":172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.9","version":"746'7279","reported_seq":7437,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-03-20T09:25:43.828713+0000","last_change":"2024-03-20T09:25:27.604402+0000","last_active":"2024-03-20T09:25:43.828713+0000","last_peered":"2024-03-20T09:25:43.828713+0000","last_clean":"2024-03-20T09:25:43.828713+0000","last_became_active":"2024-03-20T09:24:13.382746+0000","last_became_peered":"2024-03-20T09:24:13.382746+0000","last_unstale":"2024-03-20T09:25:43.828713+0000","last_undegraded":"2024-03-20T09:25:43.828713+0000","last_fullsized":"2024-03-20T09:25:43.828713+0000","mapping_epoch":715,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":716,"parent":"0.0","parent_split_bits":0,"last_scrub":"739'6312","last_scrub_stamp":"2024-03-20T09:25:27.604184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:27.604184+0000","objects_scrubbed":6258,"log_size":7279,"log_dups_size":0,"ondisk_log_size":7279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:49.818555+0000","scrub_duration":1892,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":477036544,"num_objects":7279,"num_object_clones":0,"num_object_copies":21837,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7279,"num_write_kb":465856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"20'137","reported_seq":781,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-03-20T09:25:43.002361+0000","last_change":"2024-03-20T09:25:23.627331+0000","last_active":"2024-03-20T09:25:43.002361+0000","last_peered":"2024-03-20T09:25:43.002361+0000","last_clean":"2024-03-20T09:25:43.002361+0000","last_became_active":"2024-03-20T09:19:54.506134+0000","last_became_peered":"2024-03-20T09:19:54.506134+0000","last_unstale":"2024-03-20T09:25:43.002361+0000","last_undegraded":"2024-03-20T09:25:43.002361+0000","last_fullsized":"2024-03-20T09:25:43.002361+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:23.627211+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:18:52.573886+0000","last_clean_scrub_stamp":"2024-03-20T09:25:23.627211+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:40.689080+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"746'7361","reported_seq":7517,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-03-20T09:25:45.363942+0000","last_change":"2024-03-20T09:25:29.608740+0000","last_active":"2024-03-20T09:25:45.363942+0000","last_peered":"2024-03-20T09:25:45.363942+0000","last_clean":"2024-03-20T09:25:45.363942+0000","last_became_active":"2024-03-20T09:24:03.379073+0000","last_became_peered":"2024-03-20T09:24:03.379073+0000","last_unstale":"2024-03-20T09:25:45.363942+0000","last_undegraded":"2024-03-20T09:25:45.363942+0000","last_fullsized":"2024-03-20T09:25:45.363942+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'6323","last_scrub_stamp":"2024-03-20T09:25:29.608640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:29.608640+0000","objects_scrubbed":6269,"log_size":7361,"log_dups_size":0,"ondisk_log_size":7361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:39.047255+0000","scrub_duration":1480,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482410496,"num_objects":7361,"num_object_clones":0,"num_object_copies":22083,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7361,"num_write_kb":471104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,3],"acting":[11,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.f","version":"20'137","reported_seq":1529,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-03-20T09:25:42.995712+0000","last_change":"2024-03-20T09:25:14.068325+0000","last_active":"2024-03-20T09:25:42.995712+0000","last_peered":"2024-03-20T09:25:42.995712+0000","last_clean":"2024-03-20T09:25:42.995712+0000","last_became_active":"2024-03-20T09:19:54.524397+0000","last_became_peered":"2024-03-20T09:19:54.524397+0000","last_unstale":"2024-03-20T09:25:42.995712+0000","last_undegraded":"2024-03-20T09:25:42.995712+0000","last_fullsized":"2024-03-20T09:25:42.995712+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:14.068242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:25:14.068242+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:30.699628+0000","scrub_duration":165,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"746'7430","reported_seq":7589,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-03-20T09:25:44.894986+0000","last_change":"2024-03-20T09:25:36.618305+0000","last_active":"2024-03-20T09:25:44.894986+0000","last_peered":"2024-03-20T09:25:44.894986+0000","last_clean":"2024-03-20T09:25:44.894986+0000","last_became_active":"2024-03-20T09:24:03.378961+0000","last_became_peered":"2024-03-20T09:24:03.378961+0000","last_unstale":"2024-03-20T09:25:44.894986+0000","last_undegraded":"2024-03-20T09:25:44.894986+0000","last_fullsized":"2024-03-20T09:25:44.894986+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"745'6858","last_scrub_stamp":"2024-03-20T09:25:36.617966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:36.617966+0000","objects_scrubbed":6792,"log_size":7430,"log_dups_size":0,"ondisk_log_size":7430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:42.059314+0000","scrub_duration":2094,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":486932480,"num_objects":7430,"num_object_clones":0,"num_object_copies":22290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7430,"num_write_kb":475520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"570'138","reported_seq":1522,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-03-20T09:25:42.997438+0000","last_change":"2024-03-20T09:24:18.167726+0000","last_active":"2024-03-20T09:25:42.997438+0000","last_peered":"2024-03-20T09:25:42.997438+0000","last_clean":"2024-03-20T09:25:42.997438+0000","last_became_active":"2024-03-20T09:13:27.087686+0000","last_became_peered":"2024-03-20T09:13:27.087686+0000","last_unstale":"2024-03-20T09:25:42.997438+0000","last_undegraded":"2024-03-20T09:25:42.997438+0000","last_fullsized":"2024-03-20T09:25:42.997438+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"570'138","last_scrub_stamp":"2024-03-20T09:24:18.167603+0000","last_deep_scrub":"570'138","last_deep_scrub_stamp":"2024-03-20T09:16:13.725035+0000","last_clean_scrub_stamp":"2024-03-20T09:24:18.167603+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:45.806376+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"745'7122","reported_seq":7281,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-03-20T09:25:42.061033+0000","last_change":"2024-03-20T09:25:39.076376+0000","last_active":"2024-03-20T09:25:42.061033+0000","last_peered":"2024-03-20T09:25:42.061033+0000","last_clean":"2024-03-20T09:25:42.061033+0000","last_became_active":"2024-03-20T09:24:03.381499+0000","last_became_peered":"2024-03-20T09:24:03.381499+0000","last_unstale":"2024-03-20T09:25:42.061033+0000","last_undegraded":"2024-03-20T09:25:42.061033+0000","last_fullsized":"2024-03-20T09:25:42.061033+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"745'6934","last_scrub_stamp":"2024-03-20T09:25:39.076136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:39.076136+0000","objects_scrubbed":6878,"log_size":7122,"log_dups_size":0,"ondisk_log_size":7122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:51.503424+0000","scrub_duration":1735,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":466747392,"num_objects":7122,"num_object_clones":0,"num_object_copies":21366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7122,"num_write_kb":455808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"20'137","reported_seq":1087,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-03-20T09:25:43.001557+0000","last_change":"2024-03-20T09:25:34.697040+0000","last_active":"2024-03-20T09:25:43.001557+0000","last_peered":"2024-03-20T09:25:43.001557+0000","last_clean":"2024-03-20T09:25:43.001557+0000","last_became_active":"2024-03-20T09:19:54.502529+0000","last_became_peered":"2024-03-20T09:19:54.502529+0000","last_unstale":"2024-03-20T09:25:43.001557+0000","last_undegraded":"2024-03-20T09:25:43.001557+0000","last_fullsized":"2024-03-20T09:25:43.001557+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:34.696897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:25:34.696897+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:38.529617+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"20'137","reported_seq":1390,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-03-20T09:25:35.452754+0000","last_change":"2024-03-20T09:25:34.404637+0000","last_active":"2024-03-20T09:25:35.452754+0000","last_peered":"2024-03-20T09:25:35.452754+0000","last_clean":"2024-03-20T09:25:35.452754+0000","last_became_active":"2024-03-20T09:10:34.096758+0000","last_became_peered":"2024-03-20T09:10:34.096758+0000","last_unstale":"2024-03-20T09:25:35.452754+0000","last_undegraded":"2024-03-20T09:25:35.452754+0000","last_fullsized":"2024-03-20T09:25:35.452754+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:34.404553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:25:34.404553+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:39.709307+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"20'137","reported_seq":1472,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-03-20T09:25:43.003205+0000","last_change":"2024-03-20T09:25:29.370179+0000","last_active":"2024-03-20T09:25:43.003205+0000","last_peered":"2024-03-20T09:25:43.003205+0000","last_clean":"2024-03-20T09:25:43.003205+0000","last_became_active":"2024-03-20T09:04:04.103645+0000","last_became_peered":"2024-03-20T09:04:04.103645+0000","last_unstale":"2024-03-20T09:25:43.003205+0000","last_undegraded":"2024-03-20T09:25:43.003205+0000","last_fullsized":"2024-03-20T09:25:43.003205+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:29.370054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:25:29.370054+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:48.588708+0000","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"20'137","reported_seq":924,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-03-20T09:25:35.452578+0000","last_change":"2024-03-20T09:25:31.378207+0000","last_active":"2024-03-20T09:25:35.452578+0000","last_peered":"2024-03-20T09:25:35.452578+0000","last_clean":"2024-03-20T09:25:35.452578+0000","last_became_active":"2024-03-20T09:10:34.097191+0000","last_became_peered":"2024-03-20T09:10:34.097191+0000","last_unstale":"2024-03-20T09:25:35.452578+0000","last_undegraded":"2024-03-20T09:25:35.452578+0000","last_fullsized":"2024-03-20T09:25:35.452578+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:31.378069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:25:31.378069+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:44.137526+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"20'137","reported_seq":1336,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-03-20T09:25:43.000592+0000","last_change":"2024-03-20T09:25:15.809993+0000","last_active":"2024-03-20T09:25:43.000592+0000","last_peered":"2024-03-20T09:25:43.000592+0000","last_clean":"2024-03-20T09:25:43.000592+0000","last_became_active":"2024-03-20T09:20:25.342982+0000","last_became_peered":"2024-03-20T09:20:25.342982+0000","last_unstale":"2024-03-20T09:25:43.000592+0000","last_undegraded":"2024-03-20T09:25:43.000592+0000","last_fullsized":"2024-03-20T09:25:43.000592+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:15.809834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:25:15.809834+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:41.493422+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"20'137","reported_seq":1160,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-03-20T09:25:43.002633+0000","last_change":"2024-03-20T09:25:25.209569+0000","last_active":"2024-03-20T09:25:43.002633+0000","last_peered":"2024-03-20T09:25:43.002633+0000","last_clean":"2024-03-20T09:25:43.002633+0000","last_became_active":"2024-03-20T09:13:27.087760+0000","last_became_peered":"2024-03-20T09:13:27.087760+0000","last_unstale":"2024-03-20T09:25:43.002633+0000","last_undegraded":"2024-03-20T09:25:43.002633+0000","last_fullsized":"2024-03-20T09:25:43.002633+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:25.209421+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:16:15.448920+0000","last_clean_scrub_stamp":"2024-03-20T09:25:25.209421+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:28.948596+0000","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"20'137","reported_seq":1485,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-03-20T09:25:35.452124+0000","last_change":"2024-03-20T09:25:21.212015+0000","last_active":"2024-03-20T09:25:35.452124+0000","last_peered":"2024-03-20T09:25:35.452124+0000","last_clean":"2024-03-20T09:25:35.452124+0000","last_became_active":"2024-03-20T09:10:35.095339+0000","last_became_peered":"2024-03-20T09:10:35.095339+0000","last_unstale":"2024-03-20T09:25:35.452124+0000","last_undegraded":"2024-03-20T09:25:35.452124+0000","last_fullsized":"2024-03-20T09:25:35.452124+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:21.211866+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:23.640533+0000","last_clean_scrub_stamp":"2024-03-20T09:25:21.211866+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:39.508162+0000","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"20'137","reported_seq":1564,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-03-20T09:25:43.003508+0000","last_change":"2024-03-20T09:24:31.104958+0000","last_active":"2024-03-20T09:25:43.003508+0000","last_peered":"2024-03-20T09:25:43.003508+0000","last_clean":"2024-03-20T09:25:43.003508+0000","last_became_active":"2024-03-20T09:20:25.349082+0000","last_became_peered":"2024-03-20T09:20:25.349082+0000","last_unstale":"2024-03-20T09:25:43.003508+0000","last_undegraded":"2024-03-20T09:25:43.003508+0000","last_fullsized":"2024-03-20T09:25:43.003508+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:31.104823+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:16:15.754362+0000","last_clean_scrub_stamp":"2024-03-20T09:24:31.104823+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:25:45.582333+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"20'137","reported_seq":818,"reported_epoch":745,"state":"active+clean","last_fresh":"2024-03-20T09:25:35.450433+0000","last_change":"2024-03-20T09:25:14.486015+0000","last_active":"2024-03-20T09:25:35.450433+0000","last_peered":"2024-03-20T09:25:35.450433+0000","last_clean":"2024-03-20T09:25:35.450433+0000","last_became_active":"2024-03-20T09:12:14.307581+0000","last_became_peered":"2024-03-20T09:12:14.307581+0000","last_unstale":"2024-03-20T09:25:35.450433+0000","last_undegraded":"2024-03-20T09:25:35.450433+0000","last_fullsized":"2024-03-20T09:25:35.450433+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:14.485904+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:23:41.251546+0000","last_clean_scrub_stamp":"2024-03-20T09:25:14.485904+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:34.770293+0000","scrub_duration":178,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"20'137","reported_seq":1087,"reported_epoch":746,"state":"active+clean","last_fresh":"2024-03-20T09:25:43.002117+0000","last_change":"2024-03-20T09:25:32.603971+0000","last_active":"2024-03-20T09:25:43.002117+0000","last_peered":"2024-03-20T09:25:43.002117+0000","last_clean":"2024-03-20T09:25:43.002117+0000","last_became_active":"2024-03-20T09:19:54.502391+0000","last_became_peered":"2024-03-20T09:19:54.502391+0000","last_unstale":"2024-03-20T09:25:43.002117+0000","last_undegraded":"2024-03-20T09:25:43.002117+0000","last_fullsized":"2024-03-20T09:25:43.002117+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:32.603840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:25:32.603840+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:26:56.771138+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":7759003648,"num_objects":118393,"num_object_clones":0,"num_object_copies":355179,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":118393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":118393,"num_write_kb":7577152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":23175888896,"data_stored":23175888896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118393,"ondisk_log_size":118393,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4248,"ondisk_log_size":4248,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860773,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1607256,"kb_used_data":944400,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":92137896,"statfs":{"total":95995035648,"available":94349205504,"internally_reserved":0,"allocated":967065600,"data_stored":966784564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":678754756},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":89,"seq":382252089706,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1869260,"kb_used_data":1391044,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":91875892,"statfs":{"total":95995035648,"available":94080913408,"internally_reserved":0,"allocated":1424429056,"data_stored":1424152456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":489683392},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":80,"seq":343597384050,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2117388,"kb_used_data":1861316,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":91627764,"statfs":{"total":95995035648,"available":93826830336,"internally_reserved":0,"allocated":1905987584,"data_stored":1905710984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":262207936},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":70,"seq":300647711093,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1285772,"kb_used_data":915844,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":92459380,"statfs":{"total":95995035648,"available":94678405120,"internally_reserved":0,"allocated":937824256,"data_stored":937547656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":378796486},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":60,"seq":257698038138,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2969876,"kb_used_data":2280652,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":90775276,"statfs":{"total":95995035648,"available":92953882624,"internally_reserved":0,"allocated":2335387648,"data_stored":2335107496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":705755594},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332480,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2478296,"kb_used_data":1923728,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":91266856,"statfs":{"total":95995035648,"available":93457260544,"internally_reserved":0,"allocated":1969897472,"data_stored":1969616436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":567867846},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575258,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2703832,"kb_used_data":2050704,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":91041320,"statfs":{"total":95995035648,"available":93226311680,"internally_reserved":0,"allocated":2099920896,"data_stored":2099639860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":668793286},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738776,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2232268,"kb_used_data":1569412,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":91512884,"statfs":{"total":95995035648,"available":93709193216,"internally_reserved":0,"allocated":1607077888,"data_stored":1606801288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411731,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2598548,"kb_used_data":2156428,"kb_used_omap":1,"kb_used_meta":442110,"kb_avail":91146604,"statfs":{"total":95995035648,"available":93334122496,"internally_reserved":0,"allocated":2208182272,"data_stored":2207902120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":452721093},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150095,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4447452,"kb_used_data":3826256,"kb_used_omap":1,"kb_used_meta":621182,"kb_avail":89297700,"statfs":{"total":95995035648,"available":91440844800,"internally_reserved":0,"allocated":3918086144,"data_stored":3917805108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":636090823},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,4,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953869,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2416920,"kb_used_data":1912336,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":91328232,"statfs":{"total":95995035648,"available":93520109568,"internally_reserved":0,"allocated":1958232064,"data_stored":1957951028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":443,"seq":1902670512311,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3056212,"kb_used_data":2611148,"kb_used_omap":181,"kb_used_meta":444874,"kb_avail":90688940,"statfs":{"total":95995035648,"available":92865474560,"internally_reserved":0,"allocated":2673815552,"data_stored":2673535400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":186131,"internal_metadata":455551213},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1429143552,"data_stored":1429143552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1942814720,"data_stored":1942814720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1916665856,"data_stored":1916665856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3933339648,"data_stored":3933339648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1973420032,"data_stored":1973420032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2402746368,"data_stored":2402746368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1968832512,"data_stored":1968832512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2355101696,"data_stored":2355101696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":940965888,"data_stored":940965888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1919156224,"data_stored":1919156224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1424293888,"data_stored":1424293888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":969408512,"data_stored":969408512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:25:48.581 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-20T09:25:48.581 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pool set unique_pool_4 pg_num 26 2024-03-20T09:25:48.644 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:25:48.765 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:49.052 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:25:49.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:49.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:25:49.161 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:25:49.233 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:49.469 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:49 smithi026 ceph-mon[68309]: pgmap v1822: 47 pgs: 47 active+clean; 7.3 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 983 op/s 2024-03-20T09:25:49.524 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:25:49.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:49.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:25:49.544 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:25:49.604 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:25:49.630 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:25:49.636 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:25:49.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:49 smithi115 ceph-mon[72476]: pgmap v1822: 47 pgs: 47 active+clean; 7.3 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 983 op/s 2024-03-20T09:25:49.663 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:49 smithi122 ceph-mon[73495]: pgmap v1822: 47 pgs: 47 active+clean; 7.3 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 983 op/s 2024-03-20T09:25:49.665 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:49.687 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:25:49.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:49.691 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:25:49.792 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:25:49.819 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:49.840 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:25:49.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:49.843 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:25:49.944 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:25:49.973 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:49.996 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:25:49.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:49.999 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:25:50.100 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:25:50.132 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:50.156 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:25:50.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:50.159 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:25:50.260 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:25:50.297 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:50.327 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:25:50.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:50.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:25:50.434 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:25:50.497 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:50.522 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:25:50.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:50.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:25:50.629 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:25:50.661 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:50.688 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:25:50.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:50.691 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:25:50.792 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:25:50.841 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:50.982 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:25:50.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:50.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:25:51.087 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:25:51.135 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:51.164 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:25:51.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:51.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:25:51.271 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:25:51.304 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:51.329 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:25:51.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:51.335 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:25:51.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:51 smithi115 ceph-mon[72476]: pgmap v1823: 47 pgs: 47 active+clean; 7.3 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 911 op/s 2024-03-20T09:25:51.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:51 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:25:51.435 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:25:51.465 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:51.489 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:25:51.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:51.494 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:25:51.595 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:25:51.655 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:51.688 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:25:51.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:51 smithi122 ceph-mon[73495]: pgmap v1823: 47 pgs: 47 active+clean; 7.3 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 911 op/s 2024-03-20T09:25:51.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:51 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:25:51.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:51.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:25:51.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:51 smithi026 ceph-mon[68309]: pgmap v1823: 47 pgs: 47 active+clean; 7.3 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 911 op/s 2024-03-20T09:25:51.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:51 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:25:51.795 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:25:51.843 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:51.873 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:25:51.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:51.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:25:51.982 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:25:52.010 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:52.032 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:25:52.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:52.035 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:25:52.136 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:25:52.167 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:52.191 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:25:52.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:52.195 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:25:52.296 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:25:52.360 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:52.390 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:25:52.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:52.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:25:52.496 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:25:52.527 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:52.550 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:25:52.556 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:52 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:25:52.556 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:52 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:25:52.556 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:52 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:25:52.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:52.557 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:25:52.653 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:52 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:25:52.653 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:52 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:25:52.653 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:52 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:25:52.657 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:25:52.688 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:52 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:25:52.688 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:52 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:25:52.688 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:52 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:25:52.737 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:53.653 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:53 smithi026 ceph-mon[68309]: pgmap v1824: 47 pgs: 47 active+clean; 7.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 35 MiB/s wr, 834 op/s 2024-03-20T09:25:53.653 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:53 smithi115 ceph-mon[72476]: pgmap v1824: 47 pgs: 47 active+clean; 7.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 35 MiB/s wr, 834 op/s 2024-03-20T09:25:53.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:53 smithi122 ceph-mon[73495]: pgmap v1824: 47 pgs: 47 active+clean; 7.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 35 MiB/s wr, 834 op/s 2024-03-20T09:25:54.361 INFO:teuthology.orchestra.run.smithi026.stderr:set pool 6 pg_num to 26 2024-03-20T09:25:54.653 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:54 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/354741418' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-03-20T09:25:54.653 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:54 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-03-20T09:25:54.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:54 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/354741418' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-03-20T09:25:54.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:54 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-03-20T09:25:54.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:54 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/354741418' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-03-20T09:25:54.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:54 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]: dispatch 2024-03-20T09:25:55.259 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:25:55.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:55.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:25:55.379 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:25:55.434 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:55 smithi026 ceph-mon[68309]: pgmap v1825: 47 pgs: 47 active+clean; 7.1 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 91 B/s rd, 31 MiB/s wr, 833 op/s 2024-03-20T09:25:55.434 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:55 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-03-20T09:25:55.434 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:55 smithi026 ceph-mon[68309]: osdmap e747: 12 total, 12 up, 12 in 2024-03-20T09:25:55.434 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:55 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:25:55.434 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:55 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:25:55.438 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:55.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:55 smithi115 ceph-mon[72476]: pgmap v1825: 47 pgs: 47 active+clean; 7.1 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 91 B/s rd, 31 MiB/s wr, 833 op/s 2024-03-20T09:25:55.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:55 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-03-20T09:25:55.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:55 smithi115 ceph-mon[72476]: osdmap e747: 12 total, 12 up, 12 in 2024-03-20T09:25:55.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:55 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:25:55.655 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:55 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:25:55.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:55 smithi122 ceph-mon[73495]: pgmap v1825: 47 pgs: 47 active+clean; 7.1 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 91 B/s rd, 31 MiB/s wr, 833 op/s 2024-03-20T09:25:55.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:55 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-03-20T09:25:55.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:55 smithi122 ceph-mon[73495]: osdmap e747: 12 total, 12 up, 12 in 2024-03-20T09:25:55.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:55 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:25:55.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:55 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:25:56.119 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:25:56.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:56.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:25:56.229 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:25:56.266 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:56.290 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:25:56.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:56.296 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:25:56.397 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:25:56.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:56 smithi026 ceph-mon[68309]: osdmap e748: 12 total, 12 up, 12 in 2024-03-20T09:25:56.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:56 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-20T09:25:56.474 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:56 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-20T09:25:56.540 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:56.601 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:25:56.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:56.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:25:56.607 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:56 smithi115 ceph-mon[72476]: osdmap e748: 12 total, 12 up, 12 in 2024-03-20T09:25:56.607 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:56 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-20T09:25:56.607 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:56 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-20T09:25:56.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:56 smithi122 ceph-mon[73495]: osdmap e748: 12 total, 12 up, 12 in 2024-03-20T09:25:56.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:56 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-20T09:25:56.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:56 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-20T09:25:56.707 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:25:56.753 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:56.772 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:25:56.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:56.775 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:25:56.877 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:25:56.903 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:56.922 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:25:56.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:56.925 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:25:57.026 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:25:57.056 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:57.370 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:25:57.401 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:25:57.440 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:25:57.491 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:25:57.506 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:25:57.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:57.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:25:57.532 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:25:57.578 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:25:57.622 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:25:57.624 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:25:57.653 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:57 smithi115 ceph-mon[72476]: pgmap v1828: 47 pgs: 47 active+clean; 6.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 5.9 MiB/s wr, 925 op/s 2024-03-20T09:25:57.653 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:57 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-20T09:25:57.653 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:57 smithi115 ceph-mon[72476]: osdmap e749: 12 total, 12 up, 12 in 2024-03-20T09:25:57.665 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:25:57.687 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:57 smithi026 ceph-mon[68309]: pgmap v1828: 47 pgs: 47 active+clean; 6.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 5.9 MiB/s wr, 925 op/s 2024-03-20T09:25:57.687 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:57 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-20T09:25:57.687 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:57 smithi026 ceph-mon[68309]: osdmap e749: 12 total, 12 up, 12 in 2024-03-20T09:25:57.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:57 smithi122 ceph-mon[73495]: pgmap v1828: 47 pgs: 47 active+clean; 6.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 5.9 MiB/s wr, 925 op/s 2024-03-20T09:25:57.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:57 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-20T09:25:57.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:57 smithi122 ceph-mon[73495]: osdmap e749: 12 total, 12 up, 12 in 2024-03-20T09:25:57.714 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:57.714 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:25:57.742 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:25:57.768 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:25:57.790 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:25:57.793 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:25:57.793 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:25:57.793 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:25:57.793 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:25:57.793 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:25:57.794 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:25:57.794 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:25:57.794 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:25:57.794 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:25:57.794 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:25:57.794 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:25:57.794 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:25:58.059 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:25:58.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:58.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:25:58.167 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:25:58.212 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:58.241 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:25:58.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:58.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:25:58.344 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:58 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1874724856' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:25:58.344 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:58 smithi026 ceph-mon[68309]: osdmap e750: 12 total, 12 up, 12 in 2024-03-20T09:25:58.344 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:58 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/895304118' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:25:58.344 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:58 smithi026 ceph-mon[68309]: pgmap v1831: 57 pgs: 2 peering, 10 unknown, 45 active+clean; 6.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s 2024-03-20T09:25:58.344 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:58 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-20T09:25:58.345 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:58 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-20T09:25:58.347 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:25:58.387 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:58.412 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:25:58.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:58.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:25:58.461 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:25:58.517 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:25:58.543 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:58.562 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:25:58.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:58.566 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:25:58.653 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:58 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1874724856' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:25:58.653 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:58 smithi115 ceph-mon[72476]: osdmap e750: 12 total, 12 up, 12 in 2024-03-20T09:25:58.653 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:58 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/895304118' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:25:58.653 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:58 smithi115 ceph-mon[72476]: pgmap v1831: 57 pgs: 2 peering, 10 unknown, 45 active+clean; 6.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s 2024-03-20T09:25:58.653 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:58 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-20T09:25:58.653 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:58 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-20T09:25:58.667 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:25:58.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:58 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1874724856' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:25:58.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:58 smithi122 ceph-mon[73495]: osdmap e750: 12 total, 12 up, 12 in 2024-03-20T09:25:58.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:58 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/895304118' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:25:58.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:58 smithi122 ceph-mon[73495]: pgmap v1831: 57 pgs: 2 peering, 10 unknown, 45 active+clean; 6.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.05k op/s 2024-03-20T09:25:58.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:58 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-20T09:25:58.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:58 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-20T09:25:58.693 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:58.712 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:25:58.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:58.715 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:25:58.816 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:25:58.846 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:59.296 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:25:59.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:59.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:25:59.400 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:25:59.433 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:59.458 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:25:59.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:59.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:25:59.541 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:59 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:25:59.541 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:59 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:25:59.541 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:59 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:25:59.541 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:59 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/895304118' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:25:59.542 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:59 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-20T09:25:59.542 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:25:59 smithi026 ceph-mon[68309]: osdmap e751: 12 total, 12 up, 12 in 2024-03-20T09:25:59.562 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:25:59.592 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:59 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:25:59.592 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:59 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:25:59.592 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:59 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:25:59.592 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:59 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/895304118' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:25:59.592 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:59 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-20T09:25:59.593 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:25:59 smithi122 ceph-mon[73495]: osdmap e751: 12 total, 12 up, 12 in 2024-03-20T09:25:59.594 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:59.616 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:25:59.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:59.620 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:25:59.653 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:59 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:25:59.653 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:59 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:25:59.653 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:59 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:25:59.653 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:59 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/895304118' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:25:59.653 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:59 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-20T09:25:59.653 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:25:59 smithi115 ceph-mon[72476]: osdmap e751: 12 total, 12 up, 12 in 2024-03-20T09:25:59.721 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:25:59.750 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:59.771 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:25:59.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:59.774 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:25:59.803 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:25:59.803 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:25:59.875 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:25:59.903 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:25:59.924 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:25:59.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:25:59.927 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:26:00.028 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:26:00.055 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:00.075 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:26:00.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:00.078 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:26:00.179 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:26:00.206 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:00.225 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:26:00.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:00.228 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:26:00.329 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:26:00.362 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:00.510 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:26:00.564 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:26:00.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:00.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:26:00.597 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-20T09:26:00.687 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:26:00.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:00 smithi122 ceph-mon[73495]: osdmap e752: 12 total, 12 up, 12 in 2024-03-20T09:26:00.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:00 smithi122 ceph-mon[73495]: pgmap v1834: 57 pgs: 4 remapped+peering, 2 peering, 6 unknown, 45 active+clean; 6.8 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 681 op/s 2024-03-20T09:26:00.712 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:00 smithi115 ceph-mon[72476]: osdmap e752: 12 total, 12 up, 12 in 2024-03-20T09:26:00.712 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:00 smithi115 ceph-mon[72476]: pgmap v1834: 57 pgs: 4 remapped+peering, 2 peering, 6 unknown, 45 active+clean; 6.8 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 681 op/s 2024-03-20T09:26:00.714 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:00.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:00 smithi026 ceph-mon[68309]: osdmap e752: 12 total, 12 up, 12 in 2024-03-20T09:26:00.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:00 smithi026 ceph-mon[68309]: pgmap v1834: 57 pgs: 4 remapped+peering, 2 peering, 6 unknown, 45 active+clean; 6.8 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 681 op/s 2024-03-20T09:26:00.733 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:26:00.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:00.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:26:00.837 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:26:00.865 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:00.884 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:26:00.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:00.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:26:00.989 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:26:01.017 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:01.039 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:26:01.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:01.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:26:01.144 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:26:01.173 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:01.196 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:26:01.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:01.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:26:01.207 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:26:01.292 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:26:01.301 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:26:01.322 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:26:01.422 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:01.602 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:26:01.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:01.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:26:01.706 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:01 smithi026 ceph-mon[68309]: osdmap e753: 12 total, 12 up, 12 in 2024-03-20T09:26:01.707 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:26:01.733 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:01 smithi122 ceph-mon[73495]: osdmap e753: 12 total, 12 up, 12 in 2024-03-20T09:26:01.735 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:01.760 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:26:01.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:01.764 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:26:01.866 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:26:01.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:01 smithi115 ceph-mon[72476]: osdmap e753: 12 total, 12 up, 12 in 2024-03-20T09:26:01.898 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:01.924 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:26:01.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:01.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:26:02.029 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:26:02.055 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:02.079 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:26:02.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:02.082 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:26:02.183 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:26:02.214 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:02.235 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:26:02.238 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:02.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:26:02.339 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:26:02.370 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:02.396 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:26:02.399 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:02.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:26:02.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:02 smithi026 ceph-mon[68309]: pgmap v1836: 57 pgs: 3 active+recovering+undersized+remapped, 4 remapped+peering, 3 unknown, 47 active+clean; 6.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16872/308985 objects misplaced (5.460%); 926 KiB/s, 15 objects/s recovering 2024-03-20T09:26:02.499 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:26:02.521 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 8, 10, 11, 3, 4, 9] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 5] 2024-03-20T09:26:02.521 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:26:02.522 INFO:tasks.thrashosds.thrasher:Killing osd 10, live_osds are [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 5] 2024-03-20T09:26:02.522 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-20T09:26:02.522 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.10 2024-03-20T09:26:02.591 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:02.714 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:26:02.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:02.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:26:02.831 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:26:02.885 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:02 smithi122 ceph-mon[73495]: pgmap v1836: 57 pgs: 3 active+recovering+undersized+remapped, 4 remapped+peering, 3 unknown, 47 active+clean; 6.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16872/308985 objects misplaced (5.460%); 926 KiB/s, 15 objects/s recovering 2024-03-20T09:26:02.885 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:26:02 smithi122 systemd[1]: Stopping Ceph osd.10 for 852fbb68-e696-11ee-95c9-87774f69a715... 2024-03-20T09:26:02.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:02 smithi115 ceph-mon[72476]: pgmap v1836: 57 pgs: 3 active+recovering+undersized+remapped, 4 remapped+peering, 3 unknown, 47 active+clean; 6.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16872/308985 objects misplaced (5.460%); 926 KiB/s, 15 objects/s recovering 2024-03-20T09:26:02.965 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:03.080 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:26:03.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:03.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:26:03.193 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:26:02 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[94184]: 2024-03-20T09:26:02.887+0000 7f2cf1d46700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-20T09:26:03.193 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:26:02 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[94184]: 2024-03-20T09:26:02.887+0000 7f2cf1d46700 -1 osd.10 753 *** Got signal Terminated *** 2024-03-20T09:26:03.193 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:26:02 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[94184]: 2024-03-20T09:26:02.887+0000 7f2cf1d46700 -1 osd.10 753 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-20T09:26:03.195 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:26:03.304 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:03.502 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:26:03.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:03.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:26:03.609 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:26:03.639 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:03.662 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:26:03.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:03.666 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:26:03.766 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:26:03.792 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:03 smithi115 ceph-mon[72476]: osd.10 marked itself down and dead 2024-03-20T09:26:03.794 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:03.814 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:26:03.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:03.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:26:03.899 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:03 smithi026 ceph-mon[68309]: osd.10 marked itself down and dead 2024-03-20T09:26:03.920 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:26:03.942 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:26:03.943 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:03 smithi122 ceph-mon[73495]: osd.10 marked itself down and dead 2024-03-20T09:26:03.947 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:03.968 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:26:03.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:03.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:26:04.072 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:26:04.100 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:04.121 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:26:04.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:04.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:26:04.225 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:26:04.255 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:04.353 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:26:04.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:04.356 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:26:04.457 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:26:04.485 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:04.511 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:26:04.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:04.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:26:04.578 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:04 smithi122 ceph-mon[73495]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-20T09:26:04.578 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:04 smithi122 ceph-mon[73495]: osdmap e754: 12 total, 11 up, 12 in 2024-03-20T09:26:04.578 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:04 smithi122 ceph-mon[73495]: pgmap v1838: 57 pgs: 1 remapped, 1 peering, 2 stale+active+clean, 5 active+recovering+undersized+remapped, 4 remapped+peering, 44 active+clean; 6.7 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 887 B/s rd, 23 KiB/s wr, 414 op/s; 30246/328647 objects misplaced (9.203%); 2.7 MiB/s, 44 objects/s recovering 2024-03-20T09:26:04.578 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:04 smithi122 ceph-mon[73495]: osdmap e755: 12 total, 11 up, 12 in 2024-03-20T09:26:04.617 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:26:04.662 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:04.686 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:26:04.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:04.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:26:04.790 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:26:04.821 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:04.852 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:26:04.854 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:26:04 smithi122 bash[171559]: ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10 2024-03-20T09:26:04.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:04.855 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:26:04.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:04 smithi115 ceph-mon[72476]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-20T09:26:04.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:04 smithi115 ceph-mon[72476]: osdmap e754: 12 total, 11 up, 12 in 2024-03-20T09:26:04.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:04 smithi115 ceph-mon[72476]: pgmap v1838: 57 pgs: 1 remapped, 1 peering, 2 stale+active+clean, 5 active+recovering+undersized+remapped, 4 remapped+peering, 44 active+clean; 6.7 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 887 B/s rd, 23 KiB/s wr, 414 op/s; 30246/328647 objects misplaced (9.203%); 2.7 MiB/s, 44 objects/s recovering 2024-03-20T09:26:04.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:04 smithi115 ceph-mon[72476]: osdmap e755: 12 total, 11 up, 12 in 2024-03-20T09:26:04.956 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:26:04.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:04 smithi026 ceph-mon[68309]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-20T09:26:04.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:04 smithi026 ceph-mon[68309]: osdmap e754: 12 total, 11 up, 12 in 2024-03-20T09:26:04.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:04 smithi026 ceph-mon[68309]: pgmap v1838: 57 pgs: 1 remapped, 1 peering, 2 stale+active+clean, 5 active+recovering+undersized+remapped, 4 remapped+peering, 44 active+clean; 6.7 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 887 B/s rd, 23 KiB/s wr, 414 op/s; 30246/328647 objects misplaced (9.203%); 2.7 MiB/s, 44 objects/s recovering 2024-03-20T09:26:04.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:04 smithi026 ceph-mon[68309]: osdmap e755: 12 total, 11 up, 12 in 2024-03-20T09:26:04.985 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:05.010 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:26:05.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:05.014 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:26:05.116 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:26:05.164 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:05.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:05 smithi122 ceph-mon[73495]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-20T09:26:05.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:05 smithi122 ceph-mon[73495]: osdmap e756: 12 total, 11 up, 12 in 2024-03-20T09:26:05.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:05 smithi115 ceph-mon[72476]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-20T09:26:05.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:05 smithi115 ceph-mon[72476]: osdmap e756: 12 total, 11 up, 12 in 2024-03-20T09:26:05.920 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:26:05.923 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:05 smithi026 ceph-mon[68309]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-20T09:26:05.923 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:05 smithi026 ceph-mon[68309]: osdmap e756: 12 total, 11 up, 12 in 2024-03-20T09:26:05.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:05.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:26:06.025 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:26:06.053 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:06.074 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:26:06.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:06.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:26:06.178 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:26:06.208 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:06.233 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:26:06.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:06.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:26:06.338 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:26:06.395 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:06.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:06 smithi122 ceph-mon[73495]: pgmap v1841: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 remapped, 1 peering, 2 stale+active+clean, 2 active+undersized, 6 active+recovering+undersized+remapped, 1 remapped+peering, 42 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 888 B/s rd, 40 MiB/s wr, 1.01k op/s; 3692/326298 objects degraded (1.131%); 46686/326298 objects misplaced (14.308%); 14 MiB/s, 218 objects/s recovering 2024-03-20T09:26:06.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:06 smithi026 ceph-mon[68309]: pgmap v1841: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 remapped, 1 peering, 2 stale+active+clean, 2 active+undersized, 6 active+recovering+undersized+remapped, 1 remapped+peering, 42 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 888 B/s rd, 40 MiB/s wr, 1.01k op/s; 3692/326298 objects degraded (1.131%); 46686/326298 objects misplaced (14.308%); 14 MiB/s, 218 objects/s recovering 2024-03-20T09:26:06.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:06 smithi115 ceph-mon[72476]: pgmap v1841: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 remapped, 1 peering, 2 stale+active+clean, 2 active+undersized, 6 active+recovering+undersized+remapped, 1 remapped+peering, 42 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 888 B/s rd, 40 MiB/s wr, 1.01k op/s; 3692/326298 objects degraded (1.131%); 46686/326298 objects misplaced (14.308%); 14 MiB/s, 218 objects/s recovering 2024-03-20T09:26:07.102 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.10.service' 2024-03-20T09:26:07.161 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-20T09:26:07.161 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-20T09:26:07.161 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.10 2024-03-20T09:26:07.161 INFO:tasks.thrashosds.thrasher:remote for osd 10 is ubuntu@smithi122.front.sepia.ceph.com 2024-03-20T09:26:07.162 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -n osd.10 --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-10 --journal-path /var/lib/ceph/osd/ceph-10/journal --op list-pgs 2024-03-20T09:26:07.388 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/osd.10/config 2024-03-20T09:26:07.605 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:26:07.643 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:26:07.673 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:26:07.702 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:26:07.729 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:26:07.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:07 smithi115 ceph-mon[72476]: Health check failed: Degraded data redundancy: 3692/326298 objects degraded (1.131%), 2 pgs degraded (PG_DEGRADED) 2024-03-20T09:26:07.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:07 smithi115 ceph-mon[72476]: osdmap e757: 12 total, 11 up, 12 in 2024-03-20T09:26:07.918 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:07 smithi122 ceph-mon[73495]: Health check failed: Degraded data redundancy: 3692/326298 objects degraded (1.131%), 2 pgs degraded (PG_DEGRADED) 2024-03-20T09:26:07.919 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:07 smithi122 ceph-mon[73495]: osdmap e757: 12 total, 11 up, 12 in 2024-03-20T09:26:07.967 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:07 smithi026 ceph-mon[68309]: Health check failed: Degraded data redundancy: 3692/326298 objects degraded (1.131%), 2 pgs degraded (PG_DEGRADED) 2024-03-20T09:26:07.967 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:07 smithi026 ceph-mon[68309]: osdmap e757: 12 total, 11 up, 12 in 2024-03-20T09:26:08.619 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:08 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3608759293' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:26:08.619 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:08 smithi122 ceph-mon[73495]: pgmap v1843: 57 pgs: 1 active+undersized+degraded, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 1 remapped, 1 peering, 1 stale+active+clean, 4 active+undersized, 5 active+recovering+undersized+remapped, 1 remapped+peering, 40 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 49 MiB/s wr, 1.07k op/s; 10247/326025 objects degraded (3.143%); 43006/326025 objects misplaced (13.191%); 17 MiB/s, 267 objects/s recovering 2024-03-20T09:26:08.682 INFO:teuthology.orchestra.run.smithi122.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-20T09:26:08.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:08 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3608759293' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:26:08.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:08 smithi115 ceph-mon[72476]: pgmap v1843: 57 pgs: 1 active+undersized+degraded, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 1 remapped, 1 peering, 1 stale+active+clean, 4 active+undersized, 5 active+recovering+undersized+remapped, 1 remapped+peering, 40 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 49 MiB/s wr, 1.07k op/s; 10247/326025 objects degraded (3.143%); 43006/326025 objects misplaced (13.191%); 17 MiB/s, 267 objects/s recovering 2024-03-20T09:26:08.967 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:08 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3608759293' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:26:08.967 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:08 smithi026 ceph-mon[68309]: pgmap v1843: 57 pgs: 1 active+undersized+degraded, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 1 remapped, 1 peering, 1 stale+active+clean, 4 active+undersized, 5 active+recovering+undersized+remapped, 1 remapped+peering, 40 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 49 MiB/s wr, 1.07k op/s; 10247/326025 objects degraded (3.143%); 43006/326025 objects misplaced (13.191%); 17 MiB/s, 267 objects/s recovering 2024-03-20T09:26:08.972 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:26:08.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:08.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:26:09.031 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:26:09.086 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:26:09.118 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:26:09.118 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:26:09.120 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:09.146 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:26:09.166 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:26:09.203 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:26:09.203 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:26:09.203 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:26:09.203 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:26:09.204 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:26:09.204 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:26:09.204 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:26:09.204 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:26:09.204 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:26:09.204 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:26:09.204 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:26:09.204 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-03-20T09:26:09.204 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:26:09.246 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:26:09.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:09.249 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:26:09.350 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:26:09.379 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:09.401 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:26:09.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:09.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:26:09.506 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:26:09.537 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:09.558 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:26:09.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:09.561 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:26:09.662 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:26:09.712 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:10.467 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:10 smithi026 ceph-mon[68309]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-20T09:26:10.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:10 smithi115 ceph-mon[72476]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-20T09:26:10.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:10 smithi122 ceph-mon[73495]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-20T09:26:11.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:11 smithi115 ceph-mon[72476]: pgmap v1844: 57 pgs: 1 activating+degraded+remapped, 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 5 active+recovering+undersized+remapped, 37 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 11 KiB/s wr, 192 op/s; 38012/325194 objects degraded (11.689%); 48442/325194 objects misplaced (14.896%); 24 MiB/s, 377 objects/s recovering 2024-03-20T09:26:11.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:11 smithi122 ceph-mon[73495]: pgmap v1844: 57 pgs: 1 activating+degraded+remapped, 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 5 active+recovering+undersized+remapped, 37 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 11 KiB/s wr, 192 op/s; 38012/325194 objects degraded (11.689%); 48442/325194 objects misplaced (14.896%); 24 MiB/s, 377 objects/s recovering 2024-03-20T09:26:11.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:11 smithi026 ceph-mon[68309]: pgmap v1844: 57 pgs: 1 activating+degraded+remapped, 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 5 active+recovering+undersized+remapped, 37 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 11 KiB/s wr, 192 op/s; 38012/325194 objects degraded (11.689%); 48442/325194 objects misplaced (14.896%); 24 MiB/s, 377 objects/s recovering 2024-03-20T09:26:12.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:12 smithi115 ceph-mon[72476]: pgmap v1845: 57 pgs: 1 activating+degraded+remapped, 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 5 active+recovering+undersized+remapped, 37 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 264 B/s rd, 8.5 KiB/s wr, 149 op/s; 37769/325194 objects degraded (11.614%); 47009/325194 objects misplaced (14.456%); 28 MiB/s, 446 objects/s recovering 2024-03-20T09:26:12.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:12 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 37769/325194 objects degraded (11.614%), 8 pgs degraded (PG_DEGRADED) 2024-03-20T09:26:12.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:12 smithi122 ceph-mon[73495]: pgmap v1845: 57 pgs: 1 activating+degraded+remapped, 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 5 active+recovering+undersized+remapped, 37 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 264 B/s rd, 8.5 KiB/s wr, 149 op/s; 37769/325194 objects degraded (11.614%); 47009/325194 objects misplaced (14.456%); 28 MiB/s, 446 objects/s recovering 2024-03-20T09:26:12.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:12 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 37769/325194 objects degraded (11.614%), 8 pgs degraded (PG_DEGRADED) 2024-03-20T09:26:12.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:12 smithi026 ceph-mon[68309]: pgmap v1845: 57 pgs: 1 activating+degraded+remapped, 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 5 active+recovering+undersized+remapped, 37 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 264 B/s rd, 8.5 KiB/s wr, 149 op/s; 37769/325194 objects degraded (11.614%); 47009/325194 objects misplaced (14.456%); 28 MiB/s, 446 objects/s recovering 2024-03-20T09:26:12.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:12 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 37769/325194 objects degraded (11.614%), 8 pgs degraded (PG_DEGRADED) 2024-03-20T09:26:13.423 INFO:teuthology.orchestra.run.smithi122.stdout:6.1 2024-03-20T09:26:13.423 INFO:teuthology.orchestra.run.smithi122.stdout:6.d 2024-03-20T09:26:13.423 INFO:teuthology.orchestra.run.smithi122.stdout:1.7 2024-03-20T09:26:13.423 INFO:teuthology.orchestra.run.smithi122.stdout:1.13 2024-03-20T09:26:13.423 INFO:teuthology.orchestra.run.smithi122.stdout:6.17 2024-03-20T09:26:13.423 INFO:teuthology.orchestra.run.smithi122.stdout:1.10 2024-03-20T09:26:13.423 INFO:teuthology.orchestra.run.smithi122.stdout:1.17 2024-03-20T09:26:13.423 INFO:teuthology.orchestra.run.smithi122.stdout:1.19 2024-03-20T09:26:13.423 INFO:teuthology.orchestra.run.smithi122.stdout:6.19 2024-03-20T09:26:13.423 INFO:teuthology.orchestra.run.smithi122.stdout:6.11 2024-03-20T09:26:13.424 INFO:teuthology.orchestra.run.smithi122.stdout:1.5 2024-03-20T09:26:13.424 INFO:teuthology.orchestra.run.smithi122.stdout:6.f 2024-03-20T09:26:13.424 INFO:teuthology.orchestra.run.smithi122.stdout:1.6 2024-03-20T09:26:13.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:13 smithi122 ceph-mon[73495]: osdmap e758: 12 total, 11 up, 12 in 2024-03-20T09:26:13.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:13 smithi026 ceph-mon[68309]: osdmap e758: 12 total, 11 up, 12 in 2024-03-20T09:26:13.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:13 smithi115 ceph-mon[72476]: osdmap e758: 12 total, 11 up, 12 in 2024-03-20T09:26:14.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:14 smithi122 ceph-mon[73495]: pgmap v1847: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 5 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 37 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 8.2 KiB/s wr, 49 op/s; 37757/325140 objects degraded (11.613%); 46529/325140 objects misplaced (14.310%); 21 MiB/s, 336 objects/s recovering 2024-03-20T09:26:14.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:14 smithi026 ceph-mon[68309]: pgmap v1847: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 5 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 37 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 8.2 KiB/s wr, 49 op/s; 37757/325140 objects degraded (11.613%); 46529/325140 objects misplaced (14.310%); 21 MiB/s, 336 objects/s recovering 2024-03-20T09:26:14.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:14 smithi115 ceph-mon[72476]: pgmap v1847: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 5 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 37 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 8.2 KiB/s wr, 49 op/s; 37757/325140 objects degraded (11.613%); 46529/325140 objects misplaced (14.310%); 21 MiB/s, 336 objects/s recovering 2024-03-20T09:26:17.274 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:17 smithi026 ceph-mon[68309]: pgmap v1848: 57 pgs: 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 5 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 37 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 233 B/s rd, 7.5 KiB/s wr, 39 op/s; 37757/325011 objects degraded (11.617%); 43929/325011 objects misplaced (13.516%); 32 MiB/s, 514 objects/s recovering 2024-03-20T09:26:17.564 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:26:17.564 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:26:17.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:26:17.564 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPushReply(6.16 759/752 [PushReplyOp(6:6c7df7d2:::benchmark_data_smithi122_168655_object44110:head)])", 2024-03-20T09:26:17.564 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:26:17.556161+0000", 2024-03-20T09:26:17.565 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0019936379999999998, 2024-03-20T09:26:17.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0020261889999999999, 2024-03-20T09:26:17.565 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:26:17.565 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:26:17.565 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:26:17.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:26:17.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:26:17.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:17.556161+0000", 2024-03-20T09:26:17.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:26:17.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:26:17.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:26:17.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:26:17.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:17.556161+0000", 2024-03-20T09:26:17.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:26:17.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:26:17.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:26:17.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:26:17.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:17.556162+0000", 2024-03-20T09:26:17.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.536e-06 2024-03-20T09:26:17.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:26:17.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:26:17.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:26:17.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:17.556170+0000", 2024-03-20T09:26:17.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7680000000000008e-06 2024-03-20T09:26:17.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:26:17.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:26:17.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:26:17.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:17.556186+0000", 2024-03-20T09:26:17.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5500999999999999e-05 2024-03-20T09:26:17.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:26:17.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:26:17.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:26:17.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:17.556196+0000", 2024-03-20T09:26:17.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0441999999999999e-05 2024-03-20T09:26:17.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:26:17.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:26:17.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:26:17.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:17.556260+0000", 2024-03-20T09:26:17.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4282000000000006e-05 2024-03-20T09:26:17.568 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:26:17.568 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:26:17.568 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:26:17.568 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:26:17.568 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:26:17.568 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 1 2024-03-20T09:26:17.568 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:26:17.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:17 smithi115 ceph-mon[72476]: pgmap v1848: 57 pgs: 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 5 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 37 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 233 B/s rd, 7.5 KiB/s wr, 39 op/s; 37757/325011 objects degraded (11.617%); 43929/325011 objects misplaced (13.516%); 32 MiB/s, 514 objects/s recovering 2024-03-20T09:26:17.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:17 smithi122 ceph-mon[73495]: pgmap v1848: 57 pgs: 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 5 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 37 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 233 B/s rd, 7.5 KiB/s wr, 39 op/s; 37757/325011 objects degraded (11.617%); 43929/325011 objects misplaced (13.516%); 32 MiB/s, 514 objects/s recovering 2024-03-20T09:26:18.436 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:26:18.437 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:26:18.571 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:18 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 37757/325011 objects degraded (11.617%), 8 pgs degraded (PG_DEGRADED) 2024-03-20T09:26:18.571 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:18 smithi026 ceph-mon[68309]: osdmap e759: 12 total, 11 up, 12 in 2024-03-20T09:26:18.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:18 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 37757/325011 objects degraded (11.617%), 8 pgs degraded (PG_DEGRADED) 2024-03-20T09:26:18.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:18 smithi115 ceph-mon[72476]: osdmap e759: 12 total, 11 up, 12 in 2024-03-20T09:26:18.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:18 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 37757/325011 objects degraded (11.617%), 8 pgs degraded (PG_DEGRADED) 2024-03-20T09:26:18.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:18 smithi122 ceph-mon[73495]: osdmap e759: 12 total, 11 up, 12 in 2024-03-20T09:26:19.030 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:26:19.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:19.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:26:19.063 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-20T09:26:19.138 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:26:19.170 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:19.192 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:26:19.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:19.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:26:19.287 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:19 smithi026 ceph-mon[68309]: pgmap v1850: 57 pgs: 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 5 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 37 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 9 op/s; 37496/324966 objects degraded (11.538%); 43423/324966 objects misplaced (13.362%); 29 MiB/s, 457 objects/s recovering 2024-03-20T09:26:19.287 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:19 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1285882210' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:26:19.299 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:26:19.367 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:19.391 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:26:19.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:19.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:26:19.468 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:26:19.496 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:26:19.524 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:19.653 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:19 smithi115 ceph-mon[72476]: pgmap v1850: 57 pgs: 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 5 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 37 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 9 op/s; 37496/324966 objects degraded (11.538%); 43423/324966 objects misplaced (13.362%); 29 MiB/s, 457 objects/s recovering 2024-03-20T09:26:19.653 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:19 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1285882210' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:26:19.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:19 smithi122 ceph-mon[73495]: pgmap v1850: 57 pgs: 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 5 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 37 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 9 op/s; 37496/324966 objects degraded (11.538%); 43423/324966 objects misplaced (13.362%); 29 MiB/s, 457 objects/s recovering 2024-03-20T09:26:19.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:19 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1285882210' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:26:19.784 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:26:20.437 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:26:20.670 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -n osd.10 --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-10 --journal-path /var/lib/ceph/osd/ceph-10/journal --op export --pgid 6.11 --file /var/log/ceph/exp.6.11.10 2024-03-20T09:26:20.696 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:26:20.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:20.699 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:26:20.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:20 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:26:20.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:20 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1285882210' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:26:20.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:20 smithi026 ceph-mon[68309]: osdmap e760: 12 total, 11 up, 12 in 2024-03-20T09:26:20.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:20 smithi026 ceph-mon[68309]: pgmap v1852: 57 pgs: 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 5 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 37 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 61 op/s; 37431/323748 objects degraded (11.562%); 42090/323748 objects misplaced (13.001%); 26 MiB/s, 416 objects/s recovering 2024-03-20T09:26:20.722 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:20 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:26:20.722 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:20 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1285882210' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:26:20.722 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:20 smithi122 ceph-mon[73495]: osdmap e760: 12 total, 11 up, 12 in 2024-03-20T09:26:20.722 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:20 smithi122 ceph-mon[73495]: pgmap v1852: 57 pgs: 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 5 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 37 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 61 op/s; 37431/323748 objects degraded (11.562%); 42090/323748 objects misplaced (13.001%); 26 MiB/s, 416 objects/s recovering 2024-03-20T09:26:20.800 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:26:20.870 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:20.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:20 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:26:20.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:20 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1285882210' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:26:20.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:20 smithi115 ceph-mon[72476]: osdmap e760: 12 total, 11 up, 12 in 2024-03-20T09:26:20.903 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:20 smithi115 ceph-mon[72476]: pgmap v1852: 57 pgs: 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 5 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 37 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 61 op/s; 37431/323748 objects degraded (11.562%); 42090/323748 objects misplaced (13.001%); 26 MiB/s, 416 objects/s recovering 2024-03-20T09:26:20.920 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/osd.10/config 2024-03-20T09:26:23.152 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:22 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:26:23.152 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:22 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:26:23.152 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:22 smithi115 ceph-mon[72476]: osdmap e761: 12 total, 11 up, 12 in 2024-03-20T09:26:23.153 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:22 smithi115 ceph-mon[72476]: pgmap v1854: 57 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 5 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 37 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 134 op/s; 37017/322584 objects degraded (11.475%); 40126/322584 objects misplaced (12.439%); 27 MiB/s, 437 objects/s recovering 2024-03-20T09:26:23.153 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:22 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 37431/323748 objects degraded (11.562%), 8 pgs degraded (PG_DEGRADED) 2024-03-20T09:26:23.153 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:22 smithi115 ceph-mon[72476]: osdmap e762: 12 total, 11 up, 12 in 2024-03-20T09:26:23.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:22 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:26:23.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:22 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:26:23.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:22 smithi122 ceph-mon[73495]: osdmap e761: 12 total, 11 up, 12 in 2024-03-20T09:26:23.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:22 smithi122 ceph-mon[73495]: pgmap v1854: 57 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 5 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 37 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 134 op/s; 37017/322584 objects degraded (11.475%); 40126/322584 objects misplaced (12.439%); 27 MiB/s, 437 objects/s recovering 2024-03-20T09:26:23.193 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:22 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 37431/323748 objects degraded (11.562%), 8 pgs degraded (PG_DEGRADED) 2024-03-20T09:26:23.193 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:22 smithi122 ceph-mon[73495]: osdmap e762: 12 total, 11 up, 12 in 2024-03-20T09:26:23.216 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:22 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:26:23.217 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:22 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:26:23.217 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:22 smithi026 ceph-mon[68309]: osdmap e761: 12 total, 11 up, 12 in 2024-03-20T09:26:23.217 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:22 smithi026 ceph-mon[68309]: pgmap v1854: 57 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 5 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 37 active+clean; 6.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 134 op/s; 37017/322584 objects degraded (11.475%); 40126/322584 objects misplaced (12.439%); 27 MiB/s, 437 objects/s recovering 2024-03-20T09:26:23.217 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:22 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 37431/323748 objects degraded (11.562%), 8 pgs degraded (PG_DEGRADED) 2024-03-20T09:26:23.217 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:22 smithi026 ceph-mon[68309]: osdmap e762: 12 total, 11 up, 12 in 2024-03-20T09:26:24.535 INFO:teuthology.orchestra.run.smithi122.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-20T09:26:25.494 INFO:teuthology.orchestra.run.smithi122.stderr:Exporting 6.11 info 6.11( v 750'8141 (0'0,750'8141] local-lis/les=749/750 n=3415 ec=749/709 lis/c=749/749 les/c/f=750/750/0 sis=752) 2024-03-20T09:26:25.522 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88001f58:::benchmark_data_smithi122_168655_object105647:head# 2024-03-20T09:26:25.522 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88003a0f:::benchmark_data_smithi122_168655_object102923:head# 2024-03-20T09:26:25.523 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8801c79a:::benchmark_data_smithi122_168655_object74084:head# 2024-03-20T09:26:25.525 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88029288:::benchmark_data_smithi122_168655_object25976:head# 2024-03-20T09:26:25.525 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88048665:::benchmark_data_smithi122_168655_object11931:head# 2024-03-20T09:26:25.526 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88060d12:::benchmark_data_smithi122_168655_object111021:head# 2024-03-20T09:26:25.527 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8806b372:::benchmark_data_smithi122_168655_object67061:head# 2024-03-20T09:26:25.527 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:880736d4:::benchmark_data_smithi122_168655_object100489:head# 2024-03-20T09:26:25.528 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88074123:::benchmark_data_smithi122_168655_object61859:head# 2024-03-20T09:26:25.528 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8808ce3e:::benchmark_data_smithi122_168655_object34287:head# 2024-03-20T09:26:25.529 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8808dd3e:::benchmark_data_smithi122_168655_object39639:head# 2024-03-20T09:26:25.529 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88091bbc:::benchmark_data_smithi122_168655_object64846:head# 2024-03-20T09:26:25.530 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:880a375e:::benchmark_data_smithi122_168655_object77901:head# 2024-03-20T09:26:25.531 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:880a6b2c:::benchmark_data_smithi122_168655_object85921:head# 2024-03-20T09:26:25.532 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:880c9488:::benchmark_data_smithi122_168655_object47484:head# 2024-03-20T09:26:25.532 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:880ce0b9:::benchmark_data_smithi122_168655_object12047:head# 2024-03-20T09:26:25.532 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:880d2781:::benchmark_data_smithi122_168655_object51513:head# 2024-03-20T09:26:25.533 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:880d86b3:::benchmark_data_smithi122_168655_object52945:head# 2024-03-20T09:26:25.533 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:880dc9f5:::benchmark_data_smithi122_168655_object77800:head# 2024-03-20T09:26:25.538 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:880fb963:::benchmark_data_smithi122_168655_object98428:head# 2024-03-20T09:26:25.539 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88107f9f:::benchmark_data_smithi122_168655_object64041:head# 2024-03-20T09:26:25.539 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:881154e2:::benchmark_data_smithi122_168655_object34882:head# 2024-03-20T09:26:25.539 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88116cb5:::benchmark_data_smithi122_168655_object104784:head# 2024-03-20T09:26:25.539 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88126768:::benchmark_data_smithi122_168655_object84758:head# 2024-03-20T09:26:25.539 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88132006:::benchmark_data_smithi122_168655_object10336:head# 2024-03-20T09:26:25.539 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88136845:::benchmark_data_smithi122_168655_object20999:head# 2024-03-20T09:26:25.539 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88143b6d:::benchmark_data_smithi122_168655_object39556:head# 2024-03-20T09:26:25.540 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:881539c5:::benchmark_data_smithi122_168655_object15565:head# 2024-03-20T09:26:25.540 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8816b79a:::benchmark_data_smithi122_168655_object10867:head# 2024-03-20T09:26:25.540 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8816dc11:::benchmark_data_smithi122_168655_object20087:head# 2024-03-20T09:26:25.541 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8817a2bc:::benchmark_data_smithi122_168655_object90553:head# 2024-03-20T09:26:25.541 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:881800a5:::benchmark_data_smithi122_168655_object114273:head# 2024-03-20T09:26:25.542 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88182ae7:::benchmark_data_smithi122_168655_object113046:head# 2024-03-20T09:26:25.542 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8818b705:::benchmark_data_smithi122_168655_object91184:head# 2024-03-20T09:26:25.543 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8819b100:::benchmark_data_smithi122_168655_object50092:head# 2024-03-20T09:26:25.543 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:881a0c87:::benchmark_data_smithi122_168655_object56147:head# 2024-03-20T09:26:25.544 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:881a1596:::benchmark_data_smithi122_168655_object106064:head# 2024-03-20T09:26:25.544 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:881a5bed:::benchmark_data_smithi122_168655_object61578:head# 2024-03-20T09:26:25.544 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:881b608e:::benchmark_data_smithi122_168655_object38450:head# 2024-03-20T09:26:25.545 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:881ceaf1:::benchmark_data_smithi122_168655_object53854:head# 2024-03-20T09:26:25.545 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:881d3f59:::benchmark_data_smithi122_168655_object40802:head# 2024-03-20T09:26:25.546 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:881d5f19:::benchmark_data_smithi122_168655_object105803:head# 2024-03-20T09:26:25.546 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:881f4741:::benchmark_data_smithi122_168655_object92803:head# 2024-03-20T09:26:25.547 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88200d6e:::benchmark_data_smithi122_168655_object62948:head# 2024-03-20T09:26:25.547 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:882067da:::benchmark_data_smithi122_168655_object96005:head# 2024-03-20T09:26:25.547 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88211499:::benchmark_data_smithi122_168655_object40617:head# 2024-03-20T09:26:25.548 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8821a1f7:::benchmark_data_smithi122_168655_object24910:head# 2024-03-20T09:26:25.549 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8821a4dc:::benchmark_data_smithi122_168655_object70680:head# 2024-03-20T09:26:25.549 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8821bdb6:::benchmark_data_smithi122_168655_object107498:head# 2024-03-20T09:26:25.549 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88225d3d:::benchmark_data_smithi122_168655_object47675:head# 2024-03-20T09:26:25.550 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8822feee:::benchmark_data_smithi122_168655_object91493:head# 2024-03-20T09:26:25.550 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88239937:::benchmark_data_smithi122_168655_object43818:head# 2024-03-20T09:26:25.551 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8823d844:::benchmark_data_smithi122_168655_object40477:head# 2024-03-20T09:26:25.551 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8824f22b:::benchmark_data_smithi122_168655_object77122:head# 2024-03-20T09:26:25.552 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88274cda:::benchmark_data_smithi122_168655_object111124:head# 2024-03-20T09:26:25.552 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8828cf1f:::benchmark_data_smithi122_168655_object61435:head# 2024-03-20T09:26:25.553 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8828e455:::benchmark_data_smithi122_168655_object91274:head# 2024-03-20T09:26:25.553 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88292533:::benchmark_data_smithi122_168655_object17167:head# 2024-03-20T09:26:25.553 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88294f34:::benchmark_data_smithi122_168655_object52424:head# 2024-03-20T09:26:25.554 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8829b8c8:::benchmark_data_smithi122_168655_object27428:head# 2024-03-20T09:26:25.554 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8829baa8:::benchmark_data_smithi122_168655_object74127:head# 2024-03-20T09:26:25.555 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8829cd76:::benchmark_data_smithi122_168655_object73607:head# 2024-03-20T09:26:25.555 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:882a888b:::benchmark_data_smithi122_168655_object36826:head# 2024-03-20T09:26:25.555 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:882ad8cb:::benchmark_data_smithi122_168655_object19171:head# 2024-03-20T09:26:25.556 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:882bc5f4:::benchmark_data_smithi122_168655_object110053:head# 2024-03-20T09:26:25.556 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:882cb397:::benchmark_data_smithi122_168655_object114283:head# 2024-03-20T09:26:25.557 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:882d96cb:::benchmark_data_smithi122_168655_object80386:head# 2024-03-20T09:26:25.557 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:882df894:::benchmark_data_smithi122_168655_object23123:head# 2024-03-20T09:26:25.558 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:882e659e:::benchmark_data_smithi122_168655_object85776:head# 2024-03-20T09:26:25.558 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:882e725d:::benchmark_data_smithi122_168655_object38275:head# 2024-03-20T09:26:25.558 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:882ef242:::benchmark_data_smithi122_168655_object119455:head# 2024-03-20T09:26:25.559 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:882f08ee:::benchmark_data_smithi122_168655_object78591:head# 2024-03-20T09:26:25.559 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:882f0de8:::benchmark_data_smithi122_168655_object67896:head# 2024-03-20T09:26:25.560 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:882fa11f:::benchmark_data_smithi122_168655_object95064:head# 2024-03-20T09:26:25.560 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:882fb2e8:::benchmark_data_smithi122_168655_object111743:head# 2024-03-20T09:26:25.561 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88311fa2:::benchmark_data_smithi122_168655_object117822:head# 2024-03-20T09:26:25.561 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8831d676:::benchmark_data_smithi122_168655_object59434:head# 2024-03-20T09:26:25.562 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8832b473:::benchmark_data_smithi122_168655_object103727:head# 2024-03-20T09:26:25.562 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8832f41c:::benchmark_data_smithi122_168655_object54881:head# 2024-03-20T09:26:25.562 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8833aa26:::benchmark_data_smithi122_168655_object58989:head# 2024-03-20T09:26:25.563 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88341e4f:::benchmark_data_smithi122_168655_object91682:head# 2024-03-20T09:26:25.563 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8834bbeb:::benchmark_data_smithi122_168655_object79908:head# 2024-03-20T09:26:25.564 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8834be0a:::benchmark_data_smithi122_168655_object80840:head# 2024-03-20T09:26:25.564 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:883534ec:::benchmark_data_smithi122_168655_object70603:head# 2024-03-20T09:26:25.567 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88354f5e:::benchmark_data_smithi122_168655_object69305:head# 2024-03-20T09:26:25.567 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8836028d:::benchmark_data_smithi122_168655_object76457:head# 2024-03-20T09:26:25.568 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88360aca:::benchmark_data_smithi122_168655_object57011:head# 2024-03-20T09:26:25.568 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8837b11b:::benchmark_data_smithi122_168655_object97659:head# 2024-03-20T09:26:25.569 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8837d9a7:::benchmark_data_smithi122_168655_object39050:head# 2024-03-20T09:26:25.569 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88380a1b:::benchmark_data_smithi122_168655_object41965:head# 2024-03-20T09:26:25.569 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88386f61:::benchmark_data_smithi122_168655_object11918:head# 2024-03-20T09:26:25.570 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8838be91:::benchmark_data_smithi122_168655_object65281:head# 2024-03-20T09:26:25.570 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8838d1e9:::benchmark_data_smithi122_168655_object94632:head# 2024-03-20T09:26:25.571 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:883a797a:::benchmark_data_smithi122_168655_object119310:head# 2024-03-20T09:26:25.572 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:883b16bb:::benchmark_data_smithi122_168655_object52397:head# 2024-03-20T09:26:25.572 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:883b70fc:::benchmark_data_smithi122_168655_object102065:head# 2024-03-20T09:26:25.572 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:883c2239:::benchmark_data_smithi122_168655_object20360:head# 2024-03-20T09:26:25.572 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:883d7e8b:::benchmark_data_smithi122_168655_object65604:head# 2024-03-20T09:26:25.573 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:883dea46:::benchmark_data_smithi122_168655_object73243:head# 2024-03-20T09:26:25.573 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:883e8fa7:::benchmark_data_smithi122_168655_object46834:head# 2024-03-20T09:26:25.574 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:883f1bd4:::benchmark_data_smithi122_168655_object14604:head# 2024-03-20T09:26:25.574 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:883fab19:::benchmark_data_smithi122_168655_object43292:head# 2024-03-20T09:26:25.575 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88406623:::benchmark_data_smithi122_168655_object108261:head# 2024-03-20T09:26:25.575 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8841475e:::benchmark_data_smithi122_168655_object61773:head# 2024-03-20T09:26:25.575 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88415062:::benchmark_data_smithi122_168655_object118381:head# 2024-03-20T09:26:25.576 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8841e0c7:::benchmark_data_smithi122_168655_object59547:head# 2024-03-20T09:26:25.576 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88433bc4:::benchmark_data_smithi122_168655_object14603:head# 2024-03-20T09:26:25.577 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:884391f9:::benchmark_data_smithi122_168655_object79179:head# 2024-03-20T09:26:25.577 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8843b65d:::benchmark_data_smithi122_168655_object77162:head# 2024-03-20T09:26:25.578 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88443f53:::benchmark_data_smithi122_168655_object45395:head# 2024-03-20T09:26:25.578 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8844f493:::benchmark_data_smithi122_168655_object81326:head# 2024-03-20T09:26:25.578 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88458ff0:::benchmark_data_smithi122_168655_object22706:head# 2024-03-20T09:26:25.579 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88461fe1:::benchmark_data_smithi122_168655_object66627:head# 2024-03-20T09:26:25.579 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88462022:::benchmark_data_smithi122_168655_object19045:head# 2024-03-20T09:26:25.580 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88465414:::benchmark_data_smithi122_168655_object119476:head# 2024-03-20T09:26:25.580 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:884663fb:::benchmark_data_smithi122_168655_object71326:head# 2024-03-20T09:26:25.581 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8846ea3c:::benchmark_data_smithi122_168655_object57898:head# 2024-03-20T09:26:25.581 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:884735a7:::benchmark_data_smithi122_168655_object108606:head# 2024-03-20T09:26:25.582 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88476198:::benchmark_data_smithi122_168655_object72105:head# 2024-03-20T09:26:25.582 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8847d914:::benchmark_data_smithi122_168655_object119988:head# 2024-03-20T09:26:25.583 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:884862b1:::benchmark_data_smithi122_168655_object24215:head# 2024-03-20T09:26:25.583 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8848878f:::benchmark_data_smithi122_168655_object97670:head# 2024-03-20T09:26:25.584 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:884a06d5:::benchmark_data_smithi122_168655_object41298:head# 2024-03-20T09:26:25.584 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:884a1224:::benchmark_data_smithi122_168655_object64005:head# 2024-03-20T09:26:25.584 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:884b9b72:::benchmark_data_smithi122_168655_object63613:head# 2024-03-20T09:26:25.585 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:884be9db:::benchmark_data_smithi122_168655_object110993:head# 2024-03-20T09:26:25.585 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:884c0283:::benchmark_data_smithi122_168655_object48572:head# 2024-03-20T09:26:25.585 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:884ccc63:::benchmark_data_smithi122_168655_object90288:head# 2024-03-20T09:26:25.586 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:884ccdc9:::benchmark_data_smithi122_168655_object53067:head# 2024-03-20T09:26:25.586 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:884d1c36:::benchmark_data_smithi122_168655_object115821:head# 2024-03-20T09:26:25.587 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:884d9495:::benchmark_data_smithi122_168655_object28061:head# 2024-03-20T09:26:25.587 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:884ea10b:::benchmark_data_smithi122_168655_object77879:head# 2024-03-20T09:26:25.587 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:884ec59d:::benchmark_data_smithi122_168655_object95895:head# 2024-03-20T09:26:25.588 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:884f9c2d:::benchmark_data_smithi122_168655_object15494:head# 2024-03-20T09:26:25.588 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88501f3c:::benchmark_data_smithi122_168655_object13872:head# 2024-03-20T09:26:25.588 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88502139:::benchmark_data_smithi122_168655_object110411:head# 2024-03-20T09:26:25.589 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8850515e:::benchmark_data_smithi122_168655_object10612:head# 2024-03-20T09:26:25.589 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88508257:::benchmark_data_smithi122_168655_object38630:head# 2024-03-20T09:26:25.590 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88510083:::benchmark_data_smithi122_168655_object32227:head# 2024-03-20T09:26:25.590 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8852cded:::benchmark_data_smithi122_168655_object11637:head# 2024-03-20T09:26:25.591 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:885416a4:::benchmark_data_smithi122_168655_object37679:head# 2024-03-20T09:26:25.591 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88543ae6:::benchmark_data_smithi122_168655_object42184:head# 2024-03-20T09:26:25.591 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8854b4e5:::benchmark_data_smithi122_168655_object13087:head# 2024-03-20T09:26:25.592 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:885537a1:::benchmark_data_smithi122_168655_object84179:head# 2024-03-20T09:26:25.592 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88553d38:::benchmark_data_smithi122_168655_object37231:head# 2024-03-20T09:26:25.593 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8856f567:::benchmark_data_smithi122_168655_object37649:head# 2024-03-20T09:26:25.593 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8856f85a:::benchmark_data_smithi122_168655_object95801:head# 2024-03-20T09:26:25.593 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88572300:::benchmark_data_smithi122_168655_object26444:head# 2024-03-20T09:26:25.594 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8857fa1d:::benchmark_data_smithi122_168655_object66981:head# 2024-03-20T09:26:25.595 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8858dc20:::benchmark_data_smithi122_168655_object19939:head# 2024-03-20T09:26:25.596 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8858f0be:::benchmark_data_smithi122_168655_object71418:head# 2024-03-20T09:26:25.596 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:885aa3d6:::benchmark_data_smithi122_168655_object101521:head# 2024-03-20T09:26:25.596 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:885b3835:::benchmark_data_smithi122_168655_object112928:head# 2024-03-20T09:26:25.596 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:885b3ea4:::benchmark_data_smithi122_168655_object84801:head# 2024-03-20T09:26:25.597 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:885b4dd8:::benchmark_data_smithi122_168655_object18809:head# 2024-03-20T09:26:25.597 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:885ce753:::benchmark_data_smithi122_168655_object24947:head# 2024-03-20T09:26:25.598 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:885d0d54:::benchmark_data_smithi122_168655_object53853:head# 2024-03-20T09:26:25.598 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:885e8ddd:::benchmark_data_smithi122_168655_object47264:head# 2024-03-20T09:26:25.599 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:885f608f:::benchmark_data_smithi122_168655_object86412:head# 2024-03-20T09:26:25.599 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8860a8aa:::benchmark_data_smithi122_168655_object60309:head# 2024-03-20T09:26:25.599 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8860c5f5:::benchmark_data_smithi122_168655_object54976:head# 2024-03-20T09:26:25.600 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8860c7c0:::benchmark_data_smithi122_168655_object64848:head# 2024-03-20T09:26:25.600 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8861e879:::benchmark_data_smithi122_168655_object86409:head# 2024-03-20T09:26:25.601 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88621fd5:::benchmark_data_smithi122_168655_object65287:head# 2024-03-20T09:26:25.601 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8862f3bf:::benchmark_data_smithi122_168655_object78201:head# 2024-03-20T09:26:25.601 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:886393e2:::benchmark_data_smithi122_168655_object99338:head# 2024-03-20T09:26:25.602 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:886395e8:::benchmark_data_smithi122_168655_object73431:head# 2024-03-20T09:26:25.602 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8863d963:::benchmark_data_smithi122_168655_object53336:head# 2024-03-20T09:26:25.603 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88643d8c:::benchmark_data_smithi122_168655_object39485:head# 2024-03-20T09:26:25.603 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8865c41e:::benchmark_data_smithi122_168655_object51888:head# 2024-03-20T09:26:25.603 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:886696ad:::benchmark_data_smithi122_168655_object46787:head# 2024-03-20T09:26:25.604 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:886832cc:::benchmark_data_smithi122_168655_object16742:head# 2024-03-20T09:26:25.604 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8869b5c3:::benchmark_data_smithi122_168655_object47891:head# 2024-03-20T09:26:25.605 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:886a3423:::benchmark_data_smithi122_168655_object65982:head# 2024-03-20T09:26:25.605 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:886a7ead:::benchmark_data_smithi122_168655_object61482:head# 2024-03-20T09:26:25.606 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:886acca1:::benchmark_data_smithi122_168655_object85597:head# 2024-03-20T09:26:25.606 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:886c9e0e:::benchmark_data_smithi122_168655_object100486:head# 2024-03-20T09:26:25.607 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:886cf3fd:::benchmark_data_smithi122_168655_object57190:head# 2024-03-20T09:26:25.607 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:886d46bb:::benchmark_data_smithi122_168655_object110317:head# 2024-03-20T09:26:25.607 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:886dae55:::benchmark_data_smithi122_168655_object109298:head# 2024-03-20T09:26:25.608 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:886dc807:::benchmark_data_smithi122_168655_object81195:head# 2024-03-20T09:26:25.608 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:886e1de8:::benchmark_data_smithi122_168655_object30623:head# 2024-03-20T09:26:25.608 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:886fb059:::benchmark_data_smithi122_168655_object17665:head# 2024-03-20T09:26:25.609 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:887307cc:::benchmark_data_smithi122_168655_object94718:head# 2024-03-20T09:26:25.609 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8874645c:::benchmark_data_smithi122_168655_object61483:head# 2024-03-20T09:26:25.610 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88747712:::benchmark_data_smithi122_168655_object47132:head# 2024-03-20T09:26:25.610 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88750b68:::benchmark_data_smithi122_168655_object40444:head# 2024-03-20T09:26:25.611 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8875170b:::benchmark_data_smithi122_168655_object46490:head# 2024-03-20T09:26:25.611 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8875907a:::benchmark_data_smithi122_168655_object16117:head# 2024-03-20T09:26:25.611 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8876e23d:::benchmark_data_smithi122_168655_object107909:head# 2024-03-20T09:26:25.612 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:887701e4:::benchmark_data_smithi122_168655_object78941:head# 2024-03-20T09:26:25.612 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8877df59:::benchmark_data_smithi122_168655_object69916:head# 2024-03-20T09:26:25.613 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:887927a4:::benchmark_data_smithi122_168655_object72558:head# 2024-03-20T09:26:25.613 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8879a6e8:::benchmark_data_smithi122_168655_object80596:head# 2024-03-20T09:26:25.613 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8879ca77:::benchmark_data_smithi122_168655_object72075:head# 2024-03-20T09:26:25.614 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:887ae17e:::benchmark_data_smithi122_168655_object11178:head# 2024-03-20T09:26:25.615 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:887c2734:::benchmark_data_smithi122_168655_object79001:head# 2024-03-20T09:26:25.615 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:887c7392:::benchmark_data_smithi122_168655_object94975:head# 2024-03-20T09:26:25.616 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:887cb092:::benchmark_data_smithi122_168655_object25974:head# 2024-03-20T09:26:25.616 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:887cb747:::benchmark_data_smithi122_168655_object50522:head# 2024-03-20T09:26:25.616 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:887d6dc2:::benchmark_data_smithi122_168655_object17904:head# 2024-03-20T09:26:25.617 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:887de3ec:::benchmark_data_smithi122_168655_object52249:head# 2024-03-20T09:26:25.617 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:887e6e9c:::benchmark_data_smithi122_168655_object88571:head# 2024-03-20T09:26:25.618 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:887e9633:::benchmark_data_smithi122_168655_object85966:head# 2024-03-20T09:26:25.618 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:887e9916:::benchmark_data_smithi122_168655_object76580:head# 2024-03-20T09:26:25.618 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:887f4d8d:::benchmark_data_smithi122_168655_object51794:head# 2024-03-20T09:26:25.619 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88800cfe:::benchmark_data_smithi122_168655_object109579:head# 2024-03-20T09:26:25.619 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88805713:::benchmark_data_smithi122_168655_object47942:head# 2024-03-20T09:26:25.620 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8880778d:::benchmark_data_smithi122_168655_object22042:head# 2024-03-20T09:26:25.620 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88807c66:::benchmark_data_smithi122_168655_object87481:head# 2024-03-20T09:26:25.621 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88811cd5:::benchmark_data_smithi122_168655_object11062:head# 2024-03-20T09:26:25.621 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88818e7d:::benchmark_data_smithi122_168655_object84677:head# 2024-03-20T09:26:25.622 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8881e017:::benchmark_data_smithi122_168655_object87136:head# 2024-03-20T09:26:25.622 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8881e4df:::benchmark_data_smithi122_168655_object72354:head# 2024-03-20T09:26:25.622 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88824c5f:::benchmark_data_smithi122_168655_object90500:head# 2024-03-20T09:26:25.623 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88824d70:::benchmark_data_smithi122_168655_object44504:head# 2024-03-20T09:26:25.623 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8883d215:::benchmark_data_smithi122_168655_object49440:head# 2024-03-20T09:26:25.624 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88849785:::benchmark_data_smithi122_168655_object11533:head# 2024-03-20T09:26:25.624 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8884a4b9:::benchmark_data_smithi122_168655_object108169:head# 2024-03-20T09:26:25.624 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8885d57c:::benchmark_data_smithi122_168655_object58457:head# 2024-03-20T09:26:25.625 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8886bdd2:::benchmark_data_smithi122_168655_object68823:head# 2024-03-20T09:26:25.625 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8886bed5:::benchmark_data_smithi122_168655_object100895:head# 2024-03-20T09:26:25.626 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8888745e:::benchmark_data_smithi122_168655_object27145:head# 2024-03-20T09:26:25.626 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8889f1da:::benchmark_data_smithi122_168655_object18114:head# 2024-03-20T09:26:25.626 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:888a0a1d:::benchmark_data_smithi122_168655_object80177:head# 2024-03-20T09:26:25.627 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:888aa02d:::benchmark_data_smithi122_168655_object89558:head# 2024-03-20T09:26:25.627 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:888aef90:::benchmark_data_smithi122_168655_object40868:head# 2024-03-20T09:26:25.628 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:888b1e89:::benchmark_data_smithi122_168655_object35743:head# 2024-03-20T09:26:25.628 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:888bd1b6:::benchmark_data_smithi122_168655_object95556:head# 2024-03-20T09:26:25.629 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:888c91f0:::benchmark_data_smithi122_168655_object106356:head# 2024-03-20T09:26:25.629 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:888d29e1:::benchmark_data_smithi122_168655_object49289:head# 2024-03-20T09:26:25.630 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:888d5a84:::benchmark_data_smithi122_168655_object111900:head# 2024-03-20T09:26:25.630 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:888db302:::benchmark_data_smithi122_168655_object12006:head# 2024-03-20T09:26:25.630 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:888ec125:::benchmark_data_smithi122_168655_object15357:head# 2024-03-20T09:26:25.631 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:888eebb9:::benchmark_data_smithi122_168655_object14647:head# 2024-03-20T09:26:25.632 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:888fcc93:::benchmark_data_smithi122_168655_object32470:head# 2024-03-20T09:26:25.632 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:888ff0a4:::benchmark_data_smithi122_168655_object79181:head# 2024-03-20T09:26:25.632 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8890b4a0:::benchmark_data_smithi122_168655_object38120:head# 2024-03-20T09:26:25.633 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88919c20:::benchmark_data_smithi122_168655_object22908:head# 2024-03-20T09:26:25.633 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8891d764:::benchmark_data_smithi122_168655_object98251:head# 2024-03-20T09:26:25.634 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8894088d:::benchmark_data_smithi122_168655_object95855:head# 2024-03-20T09:26:25.634 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88941c0a:::benchmark_data_smithi122_168655_object22089:head# 2024-03-20T09:26:25.635 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88942542:::benchmark_data_smithi122_168655_object95652:head# 2024-03-20T09:26:25.636 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8894cd21:::benchmark_data_smithi122_168655_object36693:head# 2024-03-20T09:26:25.637 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8895c1a5:::benchmark_data_smithi122_168655_object25339:head# 2024-03-20T09:26:25.637 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8895cf78:::benchmark_data_smithi122_168655_object62779:head# 2024-03-20T09:26:25.638 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8896b039:::benchmark_data_smithi122_168655_object58637:head# 2024-03-20T09:26:25.638 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8896b832:::benchmark_data_smithi122_168655_object57655:head# 2024-03-20T09:26:25.639 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8897608d:::benchmark_data_smithi122_168655_object78007:head# 2024-03-20T09:26:25.639 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8897d4af:::benchmark_data_smithi122_168655_object104876:head# 2024-03-20T09:26:25.640 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8898162e:::benchmark_data_smithi122_168655_object28196:head# 2024-03-20T09:26:25.640 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88982a13:::benchmark_data_smithi122_168655_object43829:head# 2024-03-20T09:26:25.641 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8898651b:::benchmark_data_smithi122_168655_object10207:head# 2024-03-20T09:26:25.641 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8898a69b:::benchmark_data_smithi122_168655_object69596:head# 2024-03-20T09:26:25.641 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88996a6d:::benchmark_data_smithi122_168655_object21014:head# 2024-03-20T09:26:25.642 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8899bdd6:::benchmark_data_smithi122_168655_object17574:head# 2024-03-20T09:26:25.642 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:889ad81d:::benchmark_data_smithi122_168655_object50282:head# 2024-03-20T09:26:25.642 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:889b01ad:::benchmark_data_smithi122_168655_object32880:head# 2024-03-20T09:26:25.643 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:889b2ef2:::benchmark_data_smithi122_168655_object30401:head# 2024-03-20T09:26:25.643 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:889b4b97:::benchmark_data_smithi122_168655_object109466:head# 2024-03-20T09:26:25.644 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:889e3371:::benchmark_data_smithi122_168655_object116270:head# 2024-03-20T09:26:25.644 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:889ee76a:::benchmark_data_smithi122_168655_object81104:head# 2024-03-20T09:26:25.644 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:889f96a4:::benchmark_data_smithi122_168655_object36712:head# 2024-03-20T09:26:25.645 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:889fccc5:::benchmark_data_smithi122_168655_object26762:head# 2024-03-20T09:26:25.645 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88a10059:::benchmark_data_smithi122_168655_object102118:head# 2024-03-20T09:26:25.646 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88a14c06:::benchmark_data_smithi122_168655_object104095:head# 2024-03-20T09:26:25.646 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88a2099e:::benchmark_data_smithi122_168655_object11248:head# 2024-03-20T09:26:25.647 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88a250fe:::benchmark_data_smithi122_168655_object85664:head# 2024-03-20T09:26:25.647 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88a4a4ce:::benchmark_data_smithi122_168655_object110583:head# 2024-03-20T09:26:25.648 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88a4c93e:::benchmark_data_smithi122_168655_object20723:head# 2024-03-20T09:26:25.648 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88a59fb4:::benchmark_data_smithi122_168655_object113893:head# 2024-03-20T09:26:25.648 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88a5f150:::benchmark_data_smithi122_168655_object111121:head# 2024-03-20T09:26:25.649 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88a608f3:::benchmark_data_smithi122_168655_object104284:head# 2024-03-20T09:26:25.649 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88a656b1:::benchmark_data_smithi122_168655_object72249:head# 2024-03-20T09:26:25.650 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88a6848d:::benchmark_data_smithi122_168655_object44088:head# 2024-03-20T09:26:25.651 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88a735c1:::benchmark_data_smithi122_168655_object43216:head# 2024-03-20T09:26:25.651 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88a74dc4:::benchmark_data_smithi122_168655_object114175:head# 2024-03-20T09:26:25.652 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88a7ff5a:::benchmark_data_smithi122_168655_object24872:head# 2024-03-20T09:26:25.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:25 smithi115 ceph-mon[72476]: pgmap v1856: 57 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 5 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 37 active+clean; 6.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 179 op/s; 36898/321726 objects degraded (11.469%); 39624/321726 objects misplaced (12.316%); 27 MiB/s, 440 objects/s recovering 2024-03-20T09:26:25.652 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88a800d0:::benchmark_data_smithi122_168655_object62472:head# 2024-03-20T09:26:25.653 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88a81b2a:::benchmark_data_smithi122_168655_object11977:head# 2024-03-20T09:26:25.653 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88a91158:::benchmark_data_smithi122_168655_object103984:head# 2024-03-20T09:26:25.653 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88a99677:::benchmark_data_smithi122_168655_object104019:head# 2024-03-20T09:26:25.654 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88aa5b17:::benchmark_data_smithi122_168655_object109146:head# 2024-03-20T09:26:25.654 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88aa64f9:::benchmark_data_smithi122_168655_object43406:head# 2024-03-20T09:26:25.654 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88aa6a34:::benchmark_data_smithi122_168655_object80481:head# 2024-03-20T09:26:25.655 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88abe8f1:::benchmark_data_smithi122_168655_object80627:head# 2024-03-20T09:26:25.655 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88ac31a2:::benchmark_data_smithi122_168655_object91529:head# 2024-03-20T09:26:25.656 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88acac35:::benchmark_data_smithi122_168655_object99553:head# 2024-03-20T09:26:25.656 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88acaebe:::benchmark_data_smithi122_168655_object36657:head# 2024-03-20T09:26:25.657 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88ad1253:::benchmark_data_smithi122_168655_object66926:head# 2024-03-20T09:26:25.657 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88af2a61:::benchmark_data_smithi122_168655_object90584:head# 2024-03-20T09:26:25.658 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88af8f4b:::benchmark_data_smithi122_168655_object16392:head# 2024-03-20T09:26:25.658 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88afa6c7:::benchmark_data_smithi122_168655_object39928:head# 2024-03-20T09:26:25.659 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88b16084:::benchmark_data_smithi122_168655_object106782:head# 2024-03-20T09:26:25.659 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88b1a0c8:::benchmark_data_smithi122_168655_object26182:head# 2024-03-20T09:26:25.660 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88b1cd39:::benchmark_data_smithi122_168655_object36741:head# 2024-03-20T09:26:25.660 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88b4273c:::benchmark_data_smithi122_168655_object32232:head# 2024-03-20T09:26:25.660 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88b58c0e:::benchmark_data_smithi122_168655_object56298:head# 2024-03-20T09:26:25.660 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88b5f03c:::benchmark_data_smithi122_168655_object50242:head# 2024-03-20T09:26:25.661 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88b5f52d:::benchmark_data_smithi122_168655_object65511:head# 2024-03-20T09:26:25.662 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88b688d2:::benchmark_data_smithi122_168655_object18872:head# 2024-03-20T09:26:25.662 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88b6e339:::benchmark_data_smithi122_168655_object59774:head# 2024-03-20T09:26:25.662 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88b73c09:::benchmark_data_smithi122_168655_object32410:head# 2024-03-20T09:26:25.663 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88b78ee7:::benchmark_data_smithi122_168655_object83106:head# 2024-03-20T09:26:25.663 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88b7d817:::benchmark_data_smithi122_168655_object59282:head# 2024-03-20T09:26:25.664 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88b7ecf0:::benchmark_data_smithi122_168655_object114220:head# 2024-03-20T09:26:25.664 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88b8646a:::benchmark_data_smithi122_168655_object70491:head# 2024-03-20T09:26:25.665 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88b9b476:::benchmark_data_smithi122_168655_object72578:head# 2024-03-20T09:26:25.665 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88ba015e:::benchmark_data_smithi122_168655_object51312:head# 2024-03-20T09:26:25.666 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88ba60f1:::benchmark_data_smithi122_168655_object56294:head# 2024-03-20T09:26:25.666 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88babf45:::benchmark_data_smithi122_168655_object12108:head# 2024-03-20T09:26:25.666 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88bac103:::benchmark_data_smithi122_168655_object48673:head# 2024-03-20T09:26:25.667 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88bb6ec4:::benchmark_data_smithi122_168655_object64222:head# 2024-03-20T09:26:25.667 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88bbdeb6:::benchmark_data_smithi122_168655_object16758:head# 2024-03-20T09:26:25.668 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88bc3830:::benchmark_data_smithi122_168655_object24679:head# 2024-03-20T09:26:25.668 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88bc686c:::benchmark_data_smithi122_168655_object108454:head# 2024-03-20T09:26:25.668 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88bcd0b8:::benchmark_data_smithi122_168655_object54171:head# 2024-03-20T09:26:25.669 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88bd44dd:::benchmark_data_smithi122_168655_object19879:head# 2024-03-20T09:26:25.669 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88bdd1f2:::benchmark_data_smithi122_168655_object45319:head# 2024-03-20T09:26:25.670 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88be4fa9:::benchmark_data_smithi122_168655_object19697:head# 2024-03-20T09:26:25.670 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88be535d:::benchmark_data_smithi122_168655_object77792:head# 2024-03-20T09:26:25.671 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88bf32f6:::benchmark_data_smithi122_168655_object43891:head# 2024-03-20T09:26:25.671 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88bf405a:::benchmark_data_smithi122_168655_object32420:head# 2024-03-20T09:26:25.672 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88bf42ef:::benchmark_data_smithi122_168655_object14520:head# 2024-03-20T09:26:25.673 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88bfa053:::benchmark_data_smithi122_168655_object87270:head# 2024-03-20T09:26:25.673 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88bfbbd0:::benchmark_data_smithi122_168655_object65040:head# 2024-03-20T09:26:25.673 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88bfcc7b:::benchmark_data_smithi122_168655_object60927:head# 2024-03-20T09:26:25.674 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88bfee75:::benchmark_data_smithi122_168655_object46356:head# 2024-03-20T09:26:25.674 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88c03e92:::benchmark_data_smithi122_168655_object18579:head# 2024-03-20T09:26:25.675 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88c110e4:::benchmark_data_smithi122_168655_object56666:head# 2024-03-20T09:26:25.675 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88c14280:::benchmark_data_smithi122_168655_object78407:head# 2024-03-20T09:26:25.675 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88c19968:::benchmark_data_smithi122_168655_object117346:head# 2024-03-20T09:26:25.676 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88c2249f:::benchmark_data_smithi122_168655_object38764:head# 2024-03-20T09:26:25.676 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88c282ba:::benchmark_data_smithi122_168655_object83809:head# 2024-03-20T09:26:25.677 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88c29376:::benchmark_data_smithi122_168655_object83979:head# 2024-03-20T09:26:25.677 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88c3c109:::benchmark_data_smithi122_168655_object105496:head# 2024-03-20T09:26:25.678 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88c51405:::benchmark_data_smithi122_168655_object65832:head# 2024-03-20T09:26:25.678 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88c5d99b:::benchmark_data_smithi122_168655_object97810:head# 2024-03-20T09:26:25.678 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88c5e6b6:::benchmark_data_smithi122_168655_object93492:head# 2024-03-20T09:26:25.679 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88c62e95:::benchmark_data_smithi122_168655_object52885:head# 2024-03-20T09:26:25.679 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88c63bbe:::benchmark_data_smithi122_168655_object83140:head# 2024-03-20T09:26:25.679 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88c642e4:::benchmark_data_smithi122_168655_object60340:head# 2024-03-20T09:26:25.680 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88c78667:::benchmark_data_smithi122_168655_object25208:head# 2024-03-20T09:26:25.680 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88c7c652:::benchmark_data_smithi122_168655_object31189:head# 2024-03-20T09:26:25.681 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88c7da07:::benchmark_data_smithi122_168655_object26433:head# 2024-03-20T09:26:25.681 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88c90c28:::benchmark_data_smithi122_168655_object63839:head# 2024-03-20T09:26:25.681 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88c94a64:::benchmark_data_smithi122_168655_object99812:head# 2024-03-20T09:26:25.682 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88c95942:::benchmark_data_smithi122_168655_object114493:head# 2024-03-20T09:26:25.682 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88c9939c:::benchmark_data_smithi122_168655_object94967:head# 2024-03-20T09:26:25.683 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88c9c6b4:::benchmark_data_smithi122_168655_object83618:head# 2024-03-20T09:26:25.683 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88ca2606:::benchmark_data_smithi122_168655_object112848:head# 2024-03-20T09:26:25.684 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88ca272a:::benchmark_data_smithi122_168655_object91359:head# 2024-03-20T09:26:25.684 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88ca9c67:::benchmark_data_smithi122_168655_object17522:head# 2024-03-20T09:26:25.685 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88cb8e78:::benchmark_data_smithi122_168655_object41773:head# 2024-03-20T09:26:25.685 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88cbd4f5:::benchmark_data_smithi122_168655_object49388:head# 2024-03-20T09:26:25.685 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88cc44a6:::benchmark_data_smithi122_168655_object115796:head# 2024-03-20T09:26:25.686 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88ccd6c0:::benchmark_data_smithi122_168655_object58284:head# 2024-03-20T09:26:25.686 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88cda85b:::benchmark_data_smithi122_168655_object58544:head# 2024-03-20T09:26:25.687 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88cdeda5:::benchmark_data_smithi122_168655_object118147:head# 2024-03-20T09:26:25.687 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88cec7a3:::benchmark_data_smithi122_168655_object81384:head# 2024-03-20T09:26:25.687 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88cfcc59:::benchmark_data_smithi122_168655_object82095:head# 2024-03-20T09:26:25.688 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88d0177f:::benchmark_data_smithi122_168655_object86085:head# 2024-03-20T09:26:25.688 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88d028b3:::benchmark_data_smithi122_168655_object50956:head# 2024-03-20T09:26:25.688 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88d05851:::benchmark_data_smithi122_168655_object118920:head# 2024-03-20T09:26:25.689 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88d10284:::benchmark_data_smithi122_168655_object113009:head# 2024-03-20T09:26:25.689 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88d14930:::benchmark_data_smithi122_168655_object119418:head# 2024-03-20T09:26:25.690 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88d19bf4:::benchmark_data_smithi122_168655_object38351:head# 2024-03-20T09:26:25.690 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88d25f73:::benchmark_data_smithi122_168655_object52482:head# 2024-03-20T09:26:25.691 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88d2f276:::benchmark_data_smithi122_168655_object65577:head# 2024-03-20T09:26:25.691 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88d3320e:::benchmark_data_smithi122_168655_object34814:head# 2024-03-20T09:26:25.691 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88d3b5c5:::benchmark_data_smithi122_168655_object77126:head# 2024-03-20T09:26:25.692 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88d532c9:::benchmark_data_smithi122_168655_object92324:head# 2024-03-20T09:26:25.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:25 smithi122 ceph-mon[73495]: pgmap v1856: 57 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 5 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 37 active+clean; 6.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 179 op/s; 36898/321726 objects degraded (11.469%); 39624/321726 objects misplaced (12.316%); 27 MiB/s, 440 objects/s recovering 2024-03-20T09:26:25.692 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88d63628:::benchmark_data_smithi122_168655_object47535:head# 2024-03-20T09:26:25.693 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88d68c37:::benchmark_data_smithi122_168655_object32396:head# 2024-03-20T09:26:25.693 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88d6d2ed:::benchmark_data_smithi122_168655_object30551:head# 2024-03-20T09:26:25.694 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88d72e6a:::benchmark_data_smithi122_168655_object18402:head# 2024-03-20T09:26:25.694 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88d76447:::benchmark_data_smithi122_168655_object83017:head# 2024-03-20T09:26:25.695 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88d91954:::benchmark_data_smithi122_168655_object47820:head# 2024-03-20T09:26:25.695 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88d97974:::benchmark_data_smithi122_168655_object34384:head# 2024-03-20T09:26:25.695 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88dce609:::benchmark_data_smithi122_168655_object67027:head# 2024-03-20T09:26:25.695 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88dd3891:::benchmark_data_smithi122_168655_object51280:head# 2024-03-20T09:26:25.696 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88de717b:::benchmark_data_smithi122_168655_object59052:head# 2024-03-20T09:26:25.696 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88df99a0:::benchmark_data_smithi122_168655_object77403:head# 2024-03-20T09:26:25.697 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88e24ee0:::benchmark_data_smithi122_168655_object17824:head# 2024-03-20T09:26:25.697 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88e2e3ba:::benchmark_data_smithi122_168655_object86149:head# 2024-03-20T09:26:25.697 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88e330da:::benchmark_data_smithi122_168655_object96372:head# 2024-03-20T09:26:25.698 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88e3e51f:::benchmark_data_smithi122_168655_object19200:head# 2024-03-20T09:26:25.698 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88e3ea6c:::benchmark_data_smithi122_168655_object92838:head# 2024-03-20T09:26:25.699 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88e497bc:::benchmark_data_smithi122_168655_object80642:head# 2024-03-20T09:26:25.699 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88e5aa78:::benchmark_data_smithi122_168655_object51084:head# 2024-03-20T09:26:25.699 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88e5f31b:::benchmark_data_smithi122_168655_object88601:head# 2024-03-20T09:26:25.700 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88e6179a:::benchmark_data_smithi122_168655_object114992:head# 2024-03-20T09:26:25.700 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88e6c27f:::benchmark_data_smithi122_168655_object101428:head# 2024-03-20T09:26:25.701 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88e8d155:::benchmark_data_smithi122_168655_object85278:head# 2024-03-20T09:26:25.701 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88ea28e6:::benchmark_data_smithi122_168655_object39237:head# 2024-03-20T09:26:25.701 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88ea3bb3:::benchmark_data_smithi122_168655_object103464:head# 2024-03-20T09:26:25.702 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88ea9424:::benchmark_data_smithi122_168655_object30513:head# 2024-03-20T09:26:25.702 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88ebec7b:::benchmark_data_smithi122_168655_object103506:head# 2024-03-20T09:26:25.703 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88ec8df7:::benchmark_data_smithi122_168655_object22764:head# 2024-03-20T09:26:25.703 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88ecba51:::benchmark_data_smithi122_168655_object56346:head# 2024-03-20T09:26:25.703 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88ecdf8c:::benchmark_data_smithi122_168655_object69131:head# 2024-03-20T09:26:25.704 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88ee4931:::benchmark_data_smithi122_168655_object41074:head# 2024-03-20T09:26:25.704 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88eec399:::benchmark_data_smithi122_168655_object116634:head# 2024-03-20T09:26:25.705 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88f03544:::benchmark_data_smithi122_168655_object50734:head# 2024-03-20T09:26:25.705 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88f0e5b9:::benchmark_data_smithi122_168655_object21181:head# 2024-03-20T09:26:25.706 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88f1544c:::benchmark_data_smithi122_168655_object96458:head# 2024-03-20T09:26:25.706 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88f22bbd:::benchmark_data_smithi122_168655_object32252:head# 2024-03-20T09:26:25.707 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88f25c6d:::benchmark_data_smithi122_168655_object42129:head# 2024-03-20T09:26:25.707 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88f2f35d:::benchmark_data_smithi122_168655_object32607:head# 2024-03-20T09:26:25.708 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88f369a6:::benchmark_data_smithi122_168655_object37019:head# 2024-03-20T09:26:25.708 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88f3faf3:::benchmark_data_smithi122_168655_object58320:head# 2024-03-20T09:26:25.708 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88f4123a:::benchmark_data_smithi122_168655_object61575:head# 2024-03-20T09:26:25.708 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88f59e3c:::benchmark_data_smithi122_168655_object52448:head# 2024-03-20T09:26:25.708 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88f5ad95:::benchmark_data_smithi122_168655_object93319:head# 2024-03-20T09:26:25.709 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88f67a36:::benchmark_data_smithi122_168655_object107818:head# 2024-03-20T09:26:25.710 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88f6d522:::benchmark_data_smithi122_168655_object88274:head# 2024-03-20T09:26:25.710 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88f787cc:::benchmark_data_smithi122_168655_object57182:head# 2024-03-20T09:26:25.710 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88f78a28:::benchmark_data_smithi122_168655_object65059:head# 2024-03-20T09:26:25.711 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88f7edd5:::benchmark_data_smithi122_168655_object43158:head# 2024-03-20T09:26:25.711 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88f82f39:::benchmark_data_smithi122_168655_object114359:head# 2024-03-20T09:26:25.711 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88f8e6ef:::benchmark_data_smithi122_168655_object90345:head# 2024-03-20T09:26:25.712 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88f8f4fc:::benchmark_data_smithi122_168655_object25591:head# 2024-03-20T09:26:25.712 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88f938e5:::benchmark_data_smithi122_168655_object48269:head# 2024-03-20T09:26:25.713 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88fa033d:::benchmark_data_smithi122_168655_object76122:head# 2024-03-20T09:26:25.713 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88fa0d21:::benchmark_data_smithi122_168655_object37237:head# 2024-03-20T09:26:25.713 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88fba34a:::benchmark_data_smithi122_168655_object18511:head# 2024-03-20T09:26:25.714 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88fc9694:::benchmark_data_smithi122_168655_object21499:head# 2024-03-20T09:26:25.714 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88fcc539:::benchmark_data_smithi122_168655_object99829:head# 2024-03-20T09:26:25.715 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88fd24d4:::benchmark_data_smithi122_168655_object100318:head# 2024-03-20T09:26:25.715 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88fd3a6b:::benchmark_data_smithi122_168655_object120031:head# 2024-03-20T09:26:25.715 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88fd9fe7:::benchmark_data_smithi122_168655_object76389:head# 2024-03-20T09:26:25.716 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88fdb55f:::benchmark_data_smithi122_168655_object110675:head# 2024-03-20T09:26:25.716 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88fe9e52:::benchmark_data_smithi122_168655_object37009:head# 2024-03-20T09:26:25.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:25 smithi026 ceph-mon[68309]: pgmap v1856: 57 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 5 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 37 active+clean; 6.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 179 op/s; 36898/321726 objects degraded (11.469%); 39624/321726 objects misplaced (12.316%); 27 MiB/s, 440 objects/s recovering 2024-03-20T09:26:25.717 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88fec790:::benchmark_data_smithi122_168655_object90454:head# 2024-03-20T09:26:25.718 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88ffcbef:::benchmark_data_smithi122_168655_object115090:head# 2024-03-20T09:26:25.718 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:88ffd1f2:::benchmark_data_smithi122_168655_object53137:head# 2024-03-20T09:26:25.719 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8900ee84:::benchmark_data_smithi122_168655_object72292:head# 2024-03-20T09:26:25.719 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:890118c9:::benchmark_data_smithi122_168655_object102512:head# 2024-03-20T09:26:25.719 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8902bb0b:::benchmark_data_smithi122_168655_object51324:head# 2024-03-20T09:26:25.719 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8902e836:::benchmark_data_smithi122_168655_object11615:head# 2024-03-20T09:26:25.720 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8902efef:::benchmark_data_smithi122_168655_object74513:head# 2024-03-20T09:26:25.720 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8903d12d:::benchmark_data_smithi122_168655_object65022:head# 2024-03-20T09:26:25.720 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8903d983:::benchmark_data_smithi122_168655_object15239:head# 2024-03-20T09:26:25.722 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:890444ee:::benchmark_data_smithi122_168655_object66054:head# 2024-03-20T09:26:25.722 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8904b8ec:::benchmark_data_smithi122_168655_object60315:head# 2024-03-20T09:26:25.722 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89055d0e:::benchmark_data_smithi122_168655_object79386:head# 2024-03-20T09:26:25.723 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89059a59:::benchmark_data_smithi122_168655_object56077:head# 2024-03-20T09:26:25.723 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8905b131:::benchmark_data_smithi122_168655_object95560:head# 2024-03-20T09:26:25.724 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89066abe:::benchmark_data_smithi122_168655_object67454:head# 2024-03-20T09:26:25.724 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:890671c9:::benchmark_data_smithi122_168655_object92925:head# 2024-03-20T09:26:25.725 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8907afab:::benchmark_data_smithi122_168655_object93024:head# 2024-03-20T09:26:25.725 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:890897f4:::benchmark_data_smithi122_168655_object24800:head# 2024-03-20T09:26:25.725 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8908fca5:::benchmark_data_smithi122_168655_object22873:head# 2024-03-20T09:26:25.726 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:890999a6:::benchmark_data_smithi122_168655_object76590:head# 2024-03-20T09:26:25.726 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:890af352:::benchmark_data_smithi122_168655_object87956:head# 2024-03-20T09:26:25.727 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:890b84d1:::benchmark_data_smithi122_168655_object68813:head# 2024-03-20T09:26:25.727 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:890bae07:::benchmark_data_smithi122_168655_object82800:head# 2024-03-20T09:26:25.727 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:890bc44b:::benchmark_data_smithi122_168655_object57001:head# 2024-03-20T09:26:25.728 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:890cb922:::benchmark_data_smithi122_168655_object52678:head# 2024-03-20T09:26:25.728 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:890d3bd3:::benchmark_data_smithi122_168655_object63349:head# 2024-03-20T09:26:25.729 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:890d527e:::benchmark_data_smithi122_168655_object19413:head# 2024-03-20T09:26:25.729 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:890da661:::benchmark_data_smithi122_168655_object75002:head# 2024-03-20T09:26:25.730 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:890e7eba:::benchmark_data_smithi122_168655_object82956:head# 2024-03-20T09:26:25.730 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:890f53d8:::benchmark_data_smithi122_168655_object100573:head# 2024-03-20T09:26:25.731 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8910665a:::benchmark_data_smithi122_168655_object37035:head# 2024-03-20T09:26:25.731 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8910b122:::benchmark_data_smithi122_168655_object96846:head# 2024-03-20T09:26:25.731 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8910feac:::benchmark_data_smithi122_168655_object38837:head# 2024-03-20T09:26:25.732 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89126172:::benchmark_data_smithi122_168655_object93422:head# 2024-03-20T09:26:25.732 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8913036c:::benchmark_data_smithi122_168655_object58013:head# 2024-03-20T09:26:25.733 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89136ce6:::benchmark_data_smithi122_168655_object51075:head# 2024-03-20T09:26:25.733 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8913732a:::benchmark_data_smithi122_168655_object25312:head# 2024-03-20T09:26:25.734 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89155932:::benchmark_data_smithi122_168655_object101093:head# 2024-03-20T09:26:25.734 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:891588e9:::benchmark_data_smithi122_168655_object119063:head# 2024-03-20T09:26:25.734 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89160be6:::benchmark_data_smithi122_168655_object66417:head# 2024-03-20T09:26:25.735 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8916b2ca:::benchmark_data_smithi122_168655_object89946:head# 2024-03-20T09:26:25.735 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8916d8b3:::benchmark_data_smithi122_168655_object111864:head# 2024-03-20T09:26:25.736 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89174b0f:::benchmark_data_smithi122_168655_object30696:head# 2024-03-20T09:26:25.736 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8917605b:::benchmark_data_smithi122_168655_object55636:head# 2024-03-20T09:26:25.737 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:891999bc:::benchmark_data_smithi122_168655_object34491:head# 2024-03-20T09:26:25.737 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8919df99:::benchmark_data_smithi122_168655_object49356:head# 2024-03-20T09:26:25.737 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:891acf64:::benchmark_data_smithi122_168655_object42028:head# 2024-03-20T09:26:25.738 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:891b35c6:::benchmark_data_smithi122_168655_object26959:head# 2024-03-20T09:26:25.738 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:891b904e:::benchmark_data_smithi122_168655_object60542:head# 2024-03-20T09:26:25.742 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:891b91c9:::benchmark_data_smithi122_168655_object38850:head# 2024-03-20T09:26:25.742 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:891bfbdc:::benchmark_data_smithi122_168655_object20666:head# 2024-03-20T09:26:25.742 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:891c292f:::benchmark_data_smithi122_168655_object13031:head# 2024-03-20T09:26:25.742 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:891c4c15:::benchmark_data_smithi122_168655_object82373:head# 2024-03-20T09:26:25.743 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:891cd8fb:::benchmark_data_smithi122_168655_object54225:head# 2024-03-20T09:26:25.743 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:891ce610:::benchmark_data_smithi122_168655_object100133:head# 2024-03-20T09:26:25.743 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:891d41b9:::benchmark_data_smithi122_168655_object97229:head# 2024-03-20T09:26:25.743 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:891dd116:::benchmark_data_smithi122_168655_object104939:head# 2024-03-20T09:26:25.743 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:891dde73:::benchmark_data_smithi122_168655_object59736:head# 2024-03-20T09:26:25.743 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:891e0d4f:::benchmark_data_smithi122_168655_object27580:head# 2024-03-20T09:26:25.743 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:891eb862:::benchmark_data_smithi122_168655_object17444:head# 2024-03-20T09:26:25.744 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:891efac0:::benchmark_data_smithi122_168655_object48020:head# 2024-03-20T09:26:25.744 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89204ef5:::benchmark_data_smithi122_168655_object110966:head# 2024-03-20T09:26:25.744 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:892188c8:::benchmark_data_smithi122_168655_object37140:head# 2024-03-20T09:26:25.745 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8921bb06:::benchmark_data_smithi122_168655_object31946:head# 2024-03-20T09:26:25.745 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89230b21:::benchmark_data_smithi122_168655_object54233:head# 2024-03-20T09:26:25.746 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8923d0f5:::benchmark_data_smithi122_168655_object88484:head# 2024-03-20T09:26:25.746 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8924442e:::benchmark_data_smithi122_168655_object66687:head# 2024-03-20T09:26:25.747 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89244548:::benchmark_data_smithi122_168655_object109851:head# 2024-03-20T09:26:25.747 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89268792:::benchmark_data_smithi122_168655_object38201:head# 2024-03-20T09:26:25.748 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8926e4e8:::benchmark_data_smithi122_168655_object62380:head# 2024-03-20T09:26:25.748 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89271163:::benchmark_data_smithi122_168655_object88381:head# 2024-03-20T09:26:25.748 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89272344:::benchmark_data_smithi122_168655_object72452:head# 2024-03-20T09:26:25.749 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89272b04:::benchmark_data_smithi122_168655_object28644:head# 2024-03-20T09:26:25.749 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89272bc0:::benchmark_data_smithi122_168655_object107119:head# 2024-03-20T09:26:25.750 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:892771c4:::benchmark_data_smithi122_168655_object54988:head# 2024-03-20T09:26:25.750 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8927cdd0:::benchmark_data_smithi122_168655_object63397:head# 2024-03-20T09:26:25.751 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:892805ac:::benchmark_data_smithi122_168655_object49198:head# 2024-03-20T09:26:25.751 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:892877f6:::benchmark_data_smithi122_168655_object56685:head# 2024-03-20T09:26:25.751 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89299e27:::benchmark_data_smithi122_168655_object44409:head# 2024-03-20T09:26:25.752 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8929f5e0:::benchmark_data_smithi122_168655_object86269:head# 2024-03-20T09:26:25.752 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:892a1d8f:::benchmark_data_smithi122_168655_object49284:head# 2024-03-20T09:26:25.753 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:892a704d:::benchmark_data_smithi122_168655_object30094:head# 2024-03-20T09:26:25.753 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:892a77a6:::benchmark_data_smithi122_168655_object88420:head# 2024-03-20T09:26:25.754 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:892b3dd1:::benchmark_data_smithi122_168655_object21458:head# 2024-03-20T09:26:25.754 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:892b57cb:::benchmark_data_smithi122_168655_object91219:head# 2024-03-20T09:26:25.754 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:892bc847:::benchmark_data_smithi122_168655_object102488:head# 2024-03-20T09:26:25.755 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:892bdf7e:::benchmark_data_smithi122_168655_object78152:head# 2024-03-20T09:26:25.755 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:892c0f8e:::benchmark_data_smithi122_168655_object47615:head# 2024-03-20T09:26:25.756 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:892d5662:::benchmark_data_smithi122_168655_object41382:head# 2024-03-20T09:26:25.756 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:892f2971:::benchmark_data_smithi122_168655_object74186:head# 2024-03-20T09:26:25.757 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:892f7009:::benchmark_data_smithi122_168655_object38543:head# 2024-03-20T09:26:25.757 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:892f9de7:::benchmark_data_smithi122_168655_object11735:head# 2024-03-20T09:26:25.757 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:892fe1e6:::benchmark_data_smithi122_168655_object80246:head# 2024-03-20T09:26:25.758 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8930ce7e:::benchmark_data_smithi122_168655_object40218:head# 2024-03-20T09:26:25.758 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89314294:::benchmark_data_smithi122_168655_object24377:head# 2024-03-20T09:26:25.759 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89316ac7:::benchmark_data_smithi122_168655_object74515:head# 2024-03-20T09:26:25.759 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8931a8a6:::benchmark_data_smithi122_168655_object22187:head# 2024-03-20T09:26:25.759 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8931b287:::benchmark_data_smithi122_168655_object74708:head# 2024-03-20T09:26:25.760 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89323c0b:::benchmark_data_smithi122_168655_object88199:head# 2024-03-20T09:26:25.760 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89329a1f:::benchmark_data_smithi122_168655_object72909:head# 2024-03-20T09:26:25.761 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8932a07a:::benchmark_data_smithi122_168655_object106729:head# 2024-03-20T09:26:25.761 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8935012d:::benchmark_data_smithi122_168655_object50385:head# 2024-03-20T09:26:25.762 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89350509:::benchmark_data_smithi122_168655_object78266:head# 2024-03-20T09:26:25.762 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8936010e:::benchmark_data_smithi122_168655_object28333:head# 2024-03-20T09:26:25.763 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8936ba1d:::benchmark_data_smithi122_168655_object36510:head# 2024-03-20T09:26:25.763 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8936e6ee:::benchmark_data_smithi122_168655_object79343:head# 2024-03-20T09:26:25.763 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89395074:::benchmark_data_smithi122_168655_object106627:head# 2024-03-20T09:26:25.764 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8939e16d:::benchmark_data_smithi122_168655_object107530:head# 2024-03-20T09:26:25.764 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:893a4162:::benchmark_data_smithi122_168655_object114036:head# 2024-03-20T09:26:25.765 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:893a9077:::benchmark_data_smithi122_168655_object69618:head# 2024-03-20T09:26:25.765 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:893aa760:::benchmark_data_smithi122_168655_object31438:head# 2024-03-20T09:26:25.765 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:893ae793:::benchmark_data_smithi122_168655_object73478:head# 2024-03-20T09:26:25.766 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:893af018:::benchmark_data_smithi122_168655_object37351:head# 2024-03-20T09:26:25.766 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:893b0479:::benchmark_data_smithi122_168655_object52543:head# 2024-03-20T09:26:25.766 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:893faa27:::benchmark_data_smithi122_168655_object12179:head# 2024-03-20T09:26:25.767 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:894054a5:::benchmark_data_smithi122_168655_object101400:head# 2024-03-20T09:26:25.768 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89406669:::benchmark_data_smithi122_168655_object65334:head# 2024-03-20T09:26:25.768 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8940a29a:::benchmark_data_smithi122_168655_object52110:head# 2024-03-20T09:26:25.768 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8941675f:::benchmark_data_smithi122_168655_object112272:head# 2024-03-20T09:26:25.768 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8941a8c4:::benchmark_data_smithi122_168655_object104495:head# 2024-03-20T09:26:25.769 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:894238c1:::benchmark_data_smithi122_168655_object21412:head# 2024-03-20T09:26:25.769 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8942c42c:::benchmark_data_smithi122_168655_object114304:head# 2024-03-20T09:26:25.770 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8942feaf:::benchmark_data_smithi122_168655_object18685:head# 2024-03-20T09:26:25.771 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8943e8be:::benchmark_data_smithi122_168655_object111780:head# 2024-03-20T09:26:25.771 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89442336:::benchmark_data_smithi122_168655_object90665:head# 2024-03-20T09:26:25.772 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8944aa9e:::benchmark_data_smithi122_168655_object22735:head# 2024-03-20T09:26:25.772 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8944b687:::benchmark_data_smithi122_168655_object114850:head# 2024-03-20T09:26:25.773 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8944bde4:::benchmark_data_smithi122_168655_object108617:head# 2024-03-20T09:26:25.773 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8944ce40:::benchmark_data_smithi122_168655_object66272:head# 2024-03-20T09:26:25.774 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:894645ff:::benchmark_data_smithi122_168655_object21891:head# 2024-03-20T09:26:25.774 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89473e18:::benchmark_data_smithi122_168655_object105399:head# 2024-03-20T09:26:25.775 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8947a1f7:::benchmark_data_smithi122_168655_object103030:head# 2024-03-20T09:26:25.775 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89481f9b:::benchmark_data_smithi122_168655_object55020:head# 2024-03-20T09:26:25.775 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8948aa04:::benchmark_data_smithi122_168655_object104170:head# 2024-03-20T09:26:25.776 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:894972f2:::benchmark_data_smithi122_168655_object24636:head# 2024-03-20T09:26:25.776 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8949b983:::benchmark_data_smithi122_168655_object80502:head# 2024-03-20T09:26:25.776 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8949d1d3:::benchmark_data_smithi122_168655_object73255:head# 2024-03-20T09:26:25.777 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:894a544c:::benchmark_data_smithi122_168655_object67056:head# 2024-03-20T09:26:25.777 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:894b1042:::benchmark_data_smithi122_168655_object114196:head# 2024-03-20T09:26:25.778 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:894bbbcb:::benchmark_data_smithi122_168655_object119194:head# 2024-03-20T09:26:25.778 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:894dfdf4:::benchmark_data_smithi122_168655_object16081:head# 2024-03-20T09:26:25.779 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:894e0181:::benchmark_data_smithi122_168655_object88138:head# 2024-03-20T09:26:25.779 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:894e0394:::benchmark_data_smithi122_168655_object53811:head# 2024-03-20T09:26:25.779 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:894e2c78:::benchmark_data_smithi122_168655_object31399:head# 2024-03-20T09:26:25.780 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:894e631a:::benchmark_data_smithi122_168655_object74405:head# 2024-03-20T09:26:25.780 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:894fc83b:::benchmark_data_smithi122_168655_object99561:head# 2024-03-20T09:26:25.780 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:894fdac8:::benchmark_data_smithi122_168655_object104715:head# 2024-03-20T09:26:25.781 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8950dad5:::benchmark_data_smithi122_168655_object33592:head# 2024-03-20T09:26:25.781 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8951057b:::benchmark_data_smithi122_168655_object20328:head# 2024-03-20T09:26:25.782 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:895233fc:::benchmark_data_smithi122_168655_object64808:head# 2024-03-20T09:26:25.782 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89528050:::benchmark_data_smithi122_168655_object112836:head# 2024-03-20T09:26:25.783 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89528ca1:::benchmark_data_smithi122_168655_object108198:head# 2024-03-20T09:26:25.783 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8952a6da:::benchmark_data_smithi122_168655_object35803:head# 2024-03-20T09:26:25.783 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89541353:::benchmark_data_smithi122_168655_object72647:head# 2024-03-20T09:26:25.784 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8955724f:::benchmark_data_smithi122_168655_object78013:head# 2024-03-20T09:26:25.784 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:895678d5:::benchmark_data_smithi122_168655_object97848:head# 2024-03-20T09:26:25.785 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8956f430:::benchmark_data_smithi122_168655_object46434:head# 2024-03-20T09:26:25.785 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89574ae2:::benchmark_data_smithi122_168655_object14034:head# 2024-03-20T09:26:25.786 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89576d5d:::benchmark_data_smithi122_168655_object66076:head# 2024-03-20T09:26:25.786 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89578252:::benchmark_data_smithi122_168655_object37529:head# 2024-03-20T09:26:25.787 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89583f27:::benchmark_data_smithi122_168655_object69244:head# 2024-03-20T09:26:25.787 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89588683:::benchmark_data_smithi122_168655_object22556:head# 2024-03-20T09:26:25.787 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89592cea:::benchmark_data_smithi122_168655_object15467:head# 2024-03-20T09:26:25.788 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89597e2b:::benchmark_data_smithi122_168655_object13275:head# 2024-03-20T09:26:25.788 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:895acb0b:::benchmark_data_smithi122_168655_object44950:head# 2024-03-20T09:26:25.789 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:895ba273:::benchmark_data_smithi122_168655_object64159:head# 2024-03-20T09:26:25.792 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:895be990:::benchmark_data_smithi122_168655_object45219:head# 2024-03-20T09:26:25.792 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:895c169c:::benchmark_data_smithi122_168655_object98981:head# 2024-03-20T09:26:25.795 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:895cc6cd:::benchmark_data_smithi122_168655_object104601:head# 2024-03-20T09:26:25.795 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:895d5502:::benchmark_data_smithi122_168655_object36500:head# 2024-03-20T09:26:25.796 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:895ebbb0:::benchmark_data_smithi122_168655_object10232:head# 2024-03-20T09:26:25.796 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8960e5fd:::benchmark_data_smithi122_168655_object85161:head# 2024-03-20T09:26:25.797 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8960ed73:::benchmark_data_smithi122_168655_object50905:head# 2024-03-20T09:26:25.797 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89611063:::benchmark_data_smithi122_168655_object106600:head# 2024-03-20T09:26:25.797 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89614099:::benchmark_data_smithi122_168655_object109284:head# 2024-03-20T09:26:25.798 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8961b6b5:::benchmark_data_smithi122_168655_object105466:head# 2024-03-20T09:26:25.798 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8961d411:::benchmark_data_smithi122_168655_object28236:head# 2024-03-20T09:26:25.798 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:896263ef:::benchmark_data_smithi122_168655_object113925:head# 2024-03-20T09:26:25.799 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8962c1ad:::benchmark_data_smithi122_168655_object58738:head# 2024-03-20T09:26:25.799 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:896311ed:::benchmark_data_smithi122_168655_object12394:head# 2024-03-20T09:26:25.799 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89632fa2:::benchmark_data_smithi122_168655_object103950:head# 2024-03-20T09:26:25.800 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8963915d:::benchmark_data_smithi122_168655_object40769:head# 2024-03-20T09:26:25.800 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89643b2c:::benchmark_data_smithi122_168655_object91852:head# 2024-03-20T09:26:25.801 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8964570d:::benchmark_data_smithi122_168655_object47265:head# 2024-03-20T09:26:25.801 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8965148c:::benchmark_data_smithi122_168655_object90889:head# 2024-03-20T09:26:25.801 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89659f5b:::benchmark_data_smithi122_168655_object84468:head# 2024-03-20T09:26:25.802 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8966bd19:::benchmark_data_smithi122_168655_object80378:head# 2024-03-20T09:26:25.802 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8966fff9:::benchmark_data_smithi122_168655_object102182:head# 2024-03-20T09:26:25.802 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8968d578:::benchmark_data_smithi122_168655_object19345:head# 2024-03-20T09:26:25.803 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:896916b9:::benchmark_data_smithi122_168655_object44047:head# 2024-03-20T09:26:25.803 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8969d465:::benchmark_data_smithi122_168655_object35445:head# 2024-03-20T09:26:25.803 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8969e15c:::benchmark_data_smithi122_168655_object108050:head# 2024-03-20T09:26:25.804 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:896a38f2:::benchmark_data_smithi122_168655_object17445:head# 2024-03-20T09:26:25.804 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:896aba59:::benchmark_data_smithi122_168655_object80178:head# 2024-03-20T09:26:25.804 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:896aeb7d:::benchmark_data_smithi122_168655_object69517:head# 2024-03-20T09:26:25.805 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:896b7786:::benchmark_data_smithi122_168655_object69850:head# 2024-03-20T09:26:25.805 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:896cc4be:::benchmark_data_smithi122_168655_object71003:head# 2024-03-20T09:26:25.806 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:896d572a:::benchmark_data_smithi122_168655_object82760:head# 2024-03-20T09:26:25.806 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:896dc0f5:::benchmark_data_smithi122_168655_object26655:head# 2024-03-20T09:26:25.807 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:896e3c67:::benchmark_data_smithi122_168655_object37339:head# 2024-03-20T09:26:25.807 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:896ef21e:::benchmark_data_smithi122_168655_object109121:head# 2024-03-20T09:26:25.808 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:896fc77d:::benchmark_data_smithi122_168655_object113458:head# 2024-03-20T09:26:25.808 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89705e21:::benchmark_data_smithi122_168655_object42785:head# 2024-03-20T09:26:25.808 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8970712e:::benchmark_data_smithi122_168655_object92574:head# 2024-03-20T09:26:25.809 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89708d88:::benchmark_data_smithi122_168655_object36754:head# 2024-03-20T09:26:25.809 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8971bca5:::benchmark_data_smithi122_168655_object24917:head# 2024-03-20T09:26:25.810 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8971e1ae:::benchmark_data_smithi122_168655_object117015:head# 2024-03-20T09:26:25.810 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89733dfc:::benchmark_data_smithi122_168655_object11331:head# 2024-03-20T09:26:25.811 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89735650:::benchmark_data_smithi122_168655_object47289:head# 2024-03-20T09:26:25.811 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8973a53e:::benchmark_data_smithi122_168655_object108013:head# 2024-03-20T09:26:25.812 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89744ddd:::benchmark_data_smithi122_168655_object84982:head# 2024-03-20T09:26:25.813 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8974c126:::benchmark_data_smithi122_168655_object52891:head# 2024-03-20T09:26:25.813 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:897677c3:::benchmark_data_smithi122_168655_object104697:head# 2024-03-20T09:26:25.814 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:897740d2:::benchmark_data_smithi122_168655_object107229:head# 2024-03-20T09:26:25.814 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8977e654:::benchmark_data_smithi122_168655_object22967:head# 2024-03-20T09:26:25.814 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8977fc31:::benchmark_data_smithi122_168655_object90429:head# 2024-03-20T09:26:25.815 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8978098d:::benchmark_data_smithi122_168655_object66858:head# 2024-03-20T09:26:25.815 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89792994:::benchmark_data_smithi122_168655_object95861:head# 2024-03-20T09:26:25.816 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8979538d:::benchmark_data_smithi122_168655_object22317:head# 2024-03-20T09:26:25.816 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8979f352:::benchmark_data_smithi122_168655_object104219:head# 2024-03-20T09:26:25.816 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:897a68f7:::benchmark_data_smithi122_168655_object90130:head# 2024-03-20T09:26:25.817 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:897b502e:::benchmark_data_smithi122_168655_object69477:head# 2024-03-20T09:26:25.817 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:897bf96d:::benchmark_data_smithi122_168655_object26626:head# 2024-03-20T09:26:25.818 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:897c760b:::benchmark_data_smithi122_168655_object33336:head# 2024-03-20T09:26:25.818 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:897cece5:::benchmark_data_smithi122_168655_object83756:head# 2024-03-20T09:26:25.818 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:897d067a:::benchmark_data_smithi122_168655_object90589:head# 2024-03-20T09:26:25.819 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:897f581f:::benchmark_data_smithi122_168655_object27505:head# 2024-03-20T09:26:25.819 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:897f9b71:::benchmark_data_smithi122_168655_object101243:head# 2024-03-20T09:26:25.820 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:897fbf6f:::benchmark_data_smithi122_168655_object99162:head# 2024-03-20T09:26:25.820 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:897fce37:::benchmark_data_smithi122_168655_object83469:head# 2024-03-20T09:26:25.820 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:897fe4dc:::benchmark_data_smithi122_168655_object54386:head# 2024-03-20T09:26:25.821 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89804a8b:::benchmark_data_smithi122_168655_object61408:head# 2024-03-20T09:26:25.821 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89809b35:::benchmark_data_smithi122_168655_object78588:head# 2024-03-20T09:26:25.822 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8980e2d8:::benchmark_data_smithi122_168655_object52835:head# 2024-03-20T09:26:25.822 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:898168a4:::benchmark_data_smithi122_168655_object107892:head# 2024-03-20T09:26:25.822 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8981b468:::benchmark_data_smithi122_168655_object25670:head# 2024-03-20T09:26:25.823 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8981c1f1:::benchmark_data_smithi122_168655_object55882:head# 2024-03-20T09:26:25.823 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8982aa81:::benchmark_data_smithi122_168655_object113949:head# 2024-03-20T09:26:25.824 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:898375d0:::benchmark_data_smithi122_168655_object69356:head# 2024-03-20T09:26:25.824 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89841a3a:::benchmark_data_smithi122_168655_object70018:head# 2024-03-20T09:26:25.824 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89842888:::benchmark_data_smithi122_168655_object76325:head# 2024-03-20T09:26:25.825 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89849a2b:::benchmark_data_smithi122_168655_object54210:head# 2024-03-20T09:26:25.825 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8984ba29:::benchmark_data_smithi122_168655_object94833:head# 2024-03-20T09:26:25.826 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8984d298:::benchmark_data_smithi122_168655_object103192:head# 2024-03-20T09:26:25.826 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89851f94:::benchmark_data_smithi122_168655_object99015:head# 2024-03-20T09:26:25.827 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8986187c:::benchmark_data_smithi122_168655_object73249:head# 2024-03-20T09:26:25.827 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89862e52:::benchmark_data_smithi122_168655_object111786:head# 2024-03-20T09:26:25.828 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8986bd44:::benchmark_data_smithi122_168655_object23454:head# 2024-03-20T09:26:25.828 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8986f9c5:::benchmark_data_smithi122_168655_object81316:head# 2024-03-20T09:26:25.829 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:898992a3:::benchmark_data_smithi122_168655_object30050:head# 2024-03-20T09:26:25.829 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:898a4bf5:::benchmark_data_smithi122_168655_object104051:head# 2024-03-20T09:26:25.829 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:898a9a79:::benchmark_data_smithi122_168655_object59285:head# 2024-03-20T09:26:25.830 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:898ac151:::benchmark_data_smithi122_168655_object77972:head# 2024-03-20T09:26:25.830 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:898b15d9:::benchmark_data_smithi122_168655_object72970:head# 2024-03-20T09:26:25.831 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:898b9023:::benchmark_data_smithi122_168655_object82720:head# 2024-03-20T09:26:25.831 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:898c2cc9:::benchmark_data_smithi122_168655_object90225:head# 2024-03-20T09:26:25.832 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:898c458e:::benchmark_data_smithi122_168655_object12338:head# 2024-03-20T09:26:25.832 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:898cb27f:::benchmark_data_smithi122_168655_object68528:head# 2024-03-20T09:26:25.833 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:898cc31e:::benchmark_data_smithi122_168655_object13687:head# 2024-03-20T09:26:25.833 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89917d09:::benchmark_data_smithi122_168655_object48927:head# 2024-03-20T09:26:25.833 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8991af4c:::benchmark_data_smithi122_168655_object44017:head# 2024-03-20T09:26:25.834 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8991bc48:::benchmark_data_smithi122_168655_object49416:head# 2024-03-20T09:26:25.834 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8992568a:::benchmark_data_smithi122_168655_object73989:head# 2024-03-20T09:26:25.835 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8993188f:::benchmark_data_smithi122_168655_object53863:head# 2024-03-20T09:26:25.835 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:899377f4:::benchmark_data_smithi122_168655_object56589:head# 2024-03-20T09:26:25.835 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:899398a0:::benchmark_data_smithi122_168655_object16602:head# 2024-03-20T09:26:25.836 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8993d76c:::benchmark_data_smithi122_168655_object50530:head# 2024-03-20T09:26:25.836 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8994c2bc:::benchmark_data_smithi122_168655_object109799:head# 2024-03-20T09:26:25.837 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89963f18:::benchmark_data_smithi122_168655_object73531:head# 2024-03-20T09:26:25.837 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:899706f6:::benchmark_data_smithi122_168655_object42764:head# 2024-03-20T09:26:25.837 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:899707a6:::benchmark_data_smithi122_168655_object110189:head# 2024-03-20T09:26:25.838 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:899769f4:::benchmark_data_smithi122_168655_object88521:head# 2024-03-20T09:26:25.838 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8997b632:::benchmark_data_smithi122_168655_object33668:head# 2024-03-20T09:26:25.839 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:899901a4:::benchmark_data_smithi122_168655_object70270:head# 2024-03-20T09:26:25.839 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:899947e3:::benchmark_data_smithi122_168655_object94488:head# 2024-03-20T09:26:25.840 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8999804c:::benchmark_data_smithi122_168655_object29826:head# 2024-03-20T09:26:25.840 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8999d672:::benchmark_data_smithi122_168655_object59896:head# 2024-03-20T09:26:25.841 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:899b0483:::benchmark_data_smithi122_168655_object53308:head# 2024-03-20T09:26:25.841 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:899b20ec:::benchmark_data_smithi122_168655_object84822:head# 2024-03-20T09:26:25.841 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:899b3b90:::benchmark_data_smithi122_168655_object86394:head# 2024-03-20T09:26:25.842 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:899bbbb5:::benchmark_data_smithi122_168655_object47296:head# 2024-03-20T09:26:25.842 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:899c4357:::benchmark_data_smithi122_168655_object62708:head# 2024-03-20T09:26:25.843 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:899ce2b4:::benchmark_data_smithi122_168655_object89361:head# 2024-03-20T09:26:25.843 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:899dad31:::benchmark_data_smithi122_168655_object93946:head# 2024-03-20T09:26:25.844 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:899e013b:::benchmark_data_smithi122_168655_object113079:head# 2024-03-20T09:26:25.844 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:899e399a:::benchmark_data_smithi122_168655_object65538:head# 2024-03-20T09:26:25.844 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:899f4d0b:::benchmark_data_smithi122_168655_object57369:head# 2024-03-20T09:26:25.845 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89a06275:::benchmark_data_smithi122_168655_object57901:head# 2024-03-20T09:26:25.845 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89a0d521:::benchmark_data_smithi122_168655_object115079:head# 2024-03-20T09:26:25.846 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89a130d5:::benchmark_data_smithi122_168655_object94923:head# 2024-03-20T09:26:25.846 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89a1314b:::benchmark_data_smithi122_168655_object77371:head# 2024-03-20T09:26:25.846 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89a3b341:::benchmark_data_smithi122_168655_object91647:head# 2024-03-20T09:26:25.847 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89a3bf4b:::benchmark_data_smithi122_168655_object98869:head# 2024-03-20T09:26:25.847 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89a501c1:::benchmark_data_smithi122_168655_object16849:head# 2024-03-20T09:26:25.848 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89a67fba:::benchmark_data_smithi122_168655_object120064:head# 2024-03-20T09:26:25.848 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89a6bb45:::benchmark_data_smithi122_168655_object37444:head# 2024-03-20T09:26:25.848 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89a7b97e:::benchmark_data_smithi122_168655_object103039:head# 2024-03-20T09:26:25.849 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89a82fc9:::benchmark_data_smithi122_168655_object13155:head# 2024-03-20T09:26:25.849 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89a8489f:::benchmark_data_smithi122_168655_object51923:head# 2024-03-20T09:26:25.850 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89a853ac:::benchmark_data_smithi122_168655_object105414:head# 2024-03-20T09:26:25.850 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89a8b6b6:::benchmark_data_smithi122_168655_object93876:head# 2024-03-20T09:26:25.850 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89a8be3e:::benchmark_data_smithi122_168655_object32656:head# 2024-03-20T09:26:25.851 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89a94f6a:::benchmark_data_smithi122_168655_object79522:head# 2024-03-20T09:26:25.851 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89aa7e37:::benchmark_data_smithi122_168655_object10325:head# 2024-03-20T09:26:25.852 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89aae8ca:::benchmark_data_smithi122_168655_object68730:head# 2024-03-20T09:26:25.852 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89abc733:::benchmark_data_smithi122_168655_object13099:head# 2024-03-20T09:26:25.853 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89abe253:::benchmark_data_smithi122_168655_object61577:head# 2024-03-20T09:26:25.853 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89adff06:::benchmark_data_smithi122_168655_object34810:head# 2024-03-20T09:26:25.854 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89ae28ac:::benchmark_data_smithi122_168655_object47033:head# 2024-03-20T09:26:25.854 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89ae30e7:::benchmark_data_smithi122_168655_object48451:head# 2024-03-20T09:26:25.854 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89af0106:::benchmark_data_smithi122_168655_object99106:head# 2024-03-20T09:26:25.855 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89af4475:::benchmark_data_smithi122_168655_object73436:head# 2024-03-20T09:26:25.855 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89b08e33:::benchmark_data_smithi122_168655_object75577:head# 2024-03-20T09:26:25.856 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89b0e219:::benchmark_data_smithi122_168655_object105333:head# 2024-03-20T09:26:25.856 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89b10e92:::benchmark_data_smithi122_168655_object47135:head# 2024-03-20T09:26:25.856 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89b135f9:::benchmark_data_smithi122_168655_object90993:head# 2024-03-20T09:26:25.857 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89b15146:::benchmark_data_smithi122_168655_object64661:head# 2024-03-20T09:26:25.857 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89b18708:::benchmark_data_smithi122_168655_object13375:head# 2024-03-20T09:26:25.858 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89b43535:::benchmark_data_smithi122_168655_object11024:head# 2024-03-20T09:26:25.858 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89b47b4b:::benchmark_data_smithi122_168655_object66096:head# 2024-03-20T09:26:25.858 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89b4cb66:::benchmark_data_smithi122_168655_object61185:head# 2024-03-20T09:26:25.859 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89b57194:::benchmark_data_smithi122_168655_object84578:head# 2024-03-20T09:26:25.859 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89b62232:::benchmark_data_smithi122_168655_object38522:head# 2024-03-20T09:26:25.860 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89b62e32:::benchmark_data_smithi122_168655_object78868:head# 2024-03-20T09:26:25.860 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89b6601f:::benchmark_data_smithi122_168655_object45768:head# 2024-03-20T09:26:25.861 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89b70b1a:::benchmark_data_smithi122_168655_object105296:head# 2024-03-20T09:26:25.861 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89b7b346:::benchmark_data_smithi122_168655_object110884:head# 2024-03-20T09:26:25.861 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89b7c803:::benchmark_data_smithi122_168655_object61475:head# 2024-03-20T09:26:25.862 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89b862f2:::benchmark_data_smithi122_168655_object58755:head# 2024-03-20T09:26:25.862 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89b88b21:::benchmark_data_smithi122_168655_object113400:head# 2024-03-20T09:26:25.863 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89b8d33a:::benchmark_data_smithi122_168655_object88309:head# 2024-03-20T09:26:25.863 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89b8e852:::benchmark_data_smithi122_168655_object60030:head# 2024-03-20T09:26:25.864 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89b9310c:::benchmark_data_smithi122_168655_object82103:head# 2024-03-20T09:26:25.864 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89b93207:::benchmark_data_smithi122_168655_object65631:head# 2024-03-20T09:26:25.866 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89b9b385:::benchmark_data_smithi122_168655_object25367:head# 2024-03-20T09:26:25.867 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89bb639c:::benchmark_data_smithi122_168655_object62701:head# 2024-03-20T09:26:25.867 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89bb6ffb:::benchmark_data_smithi122_168655_object83974:head# 2024-03-20T09:26:25.867 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89bceff5:::benchmark_data_smithi122_168655_object88261:head# 2024-03-20T09:26:25.868 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89bd7790:::benchmark_data_smithi122_168655_object45650:head# 2024-03-20T09:26:25.868 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89bda29d:::benchmark_data_smithi122_168655_object82399:head# 2024-03-20T09:26:25.868 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89bdfbf0:::benchmark_data_smithi122_168655_object89584:head# 2024-03-20T09:26:25.869 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89be8815:::benchmark_data_smithi122_168655_object111719:head# 2024-03-20T09:26:25.869 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89be9629:::benchmark_data_smithi122_168655_object112470:head# 2024-03-20T09:26:25.870 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89bea5e8:::benchmark_data_smithi122_168655_object35075:head# 2024-03-20T09:26:25.870 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89bebbc9:::benchmark_data_smithi122_168655_object35792:head# 2024-03-20T09:26:25.870 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89c076aa:::benchmark_data_smithi122_168655_object66295:head# 2024-03-20T09:26:25.871 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89c17fd2:::benchmark_data_smithi122_168655_object43621:head# 2024-03-20T09:26:25.871 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89c19889:::benchmark_data_smithi122_168655_object51245:head# 2024-03-20T09:26:25.872 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89c28638:::benchmark_data_smithi122_168655_object15934:head# 2024-03-20T09:26:25.872 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89c2a0d5:::benchmark_data_smithi122_168655_object30171:head# 2024-03-20T09:26:25.873 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89c2ff6f:::benchmark_data_smithi122_168655_object19472:head# 2024-03-20T09:26:25.873 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89c50957:::benchmark_data_smithi122_168655_object41439:head# 2024-03-20T09:26:25.874 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89c54439:::benchmark_data_smithi122_168655_object14309:head# 2024-03-20T09:26:25.874 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89c5a1a3:::benchmark_data_smithi122_168655_object98469:head# 2024-03-20T09:26:25.874 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89c6ee31:::benchmark_data_smithi122_168655_object93590:head# 2024-03-20T09:26:25.875 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89c82771:::benchmark_data_smithi122_168655_object88000:head# 2024-03-20T09:26:25.875 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89c8e982:::benchmark_data_smithi122_168655_object104132:head# 2024-03-20T09:26:25.876 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89c95a74:::benchmark_data_smithi122_168655_object99476:head# 2024-03-20T09:26:25.876 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89c97012:::benchmark_data_smithi122_168655_object40150:head# 2024-03-20T09:26:25.877 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89c97cb7:::benchmark_data_smithi122_168655_object51700:head# 2024-03-20T09:26:25.877 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89c9d475:::benchmark_data_smithi122_168655_object63950:head# 2024-03-20T09:26:25.877 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89ca1136:::benchmark_data_smithi122_168655_object28451:head# 2024-03-20T09:26:25.878 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89ca480a:::benchmark_data_smithi122_168655_object96416:head# 2024-03-20T09:26:25.878 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89ca66ef:::benchmark_data_smithi122_168655_object70023:head# 2024-03-20T09:26:25.879 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89caf004:::benchmark_data_smithi122_168655_object115778:head# 2024-03-20T09:26:25.879 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89cb8d00:::benchmark_data_smithi122_168655_object50099:head# 2024-03-20T09:26:25.880 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89cb9135:::benchmark_data_smithi122_168655_object119965:head# 2024-03-20T09:26:25.880 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89cbd671:::benchmark_data_smithi122_168655_object78371:head# 2024-03-20T09:26:25.880 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89cc0843:::benchmark_data_smithi122_168655_object94318:head# 2024-03-20T09:26:25.881 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89cc3940:::benchmark_data_smithi122_168655_object55318:head# 2024-03-20T09:26:25.881 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89cc6871:::benchmark_data_smithi122_168655_object35093:head# 2024-03-20T09:26:25.882 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89ccb185:::benchmark_data_smithi122_168655_object40908:head# 2024-03-20T09:26:25.882 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89ccddce:::benchmark_data_smithi122_168655_object28930:head# 2024-03-20T09:26:25.882 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89cdcb31:::benchmark_data_smithi122_168655_object110308:head# 2024-03-20T09:26:25.883 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89cdefba:::benchmark_data_smithi122_168655_object108958:head# 2024-03-20T09:26:25.883 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89cea8f1:::benchmark_data_smithi122_168655_object35363:head# 2024-03-20T09:26:25.884 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89cec6f6:::benchmark_data_smithi122_168655_object46685:head# 2024-03-20T09:26:25.884 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89cf199f:::benchmark_data_smithi122_168655_object25488:head# 2024-03-20T09:26:25.884 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89cf19db:::benchmark_data_smithi122_168655_object24509:head# 2024-03-20T09:26:25.885 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89d01c59:::benchmark_data_smithi122_168655_object90877:head# 2024-03-20T09:26:25.885 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89d03c6c:::benchmark_data_smithi122_168655_object81857:head# 2024-03-20T09:26:25.886 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89d0542c:::benchmark_data_smithi122_168655_object43716:head# 2024-03-20T09:26:25.886 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89d089d7:::benchmark_data_smithi122_168655_object66017:head# 2024-03-20T09:26:25.887 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89d121eb:::benchmark_data_smithi122_168655_object108384:head# 2024-03-20T09:26:25.887 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89d158bd:::benchmark_data_smithi122_168655_object29361:head# 2024-03-20T09:26:25.888 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89d1684c:::benchmark_data_smithi122_168655_object112195:head# 2024-03-20T09:26:25.888 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89d2f271:::benchmark_data_smithi122_168655_object108400:head# 2024-03-20T09:26:25.888 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89d3f7d0:::benchmark_data_smithi122_168655_object75681:head# 2024-03-20T09:26:25.889 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89d4d942:::benchmark_data_smithi122_168655_object45563:head# 2024-03-20T09:26:25.890 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89d554c2:::benchmark_data_smithi122_168655_object98130:head# 2024-03-20T09:26:25.891 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89d72667:::benchmark_data_smithi122_168655_object116860:head# 2024-03-20T09:26:25.891 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89d816f5:::benchmark_data_smithi122_168655_object51939:head# 2024-03-20T09:26:25.892 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89db0855:::benchmark_data_smithi122_168655_object103969:head# 2024-03-20T09:26:25.892 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89db17fc:::benchmark_data_smithi122_168655_object34258:head# 2024-03-20T09:26:25.893 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89db50b3:::benchmark_data_smithi122_168655_object42649:head# 2024-03-20T09:26:25.894 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89dbc268:::benchmark_data_smithi122_168655_object71000:head# 2024-03-20T09:26:25.894 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89dd5759:::benchmark_data_smithi122_168655_object50691:head# 2024-03-20T09:26:25.894 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89dd7e8f:::benchmark_data_smithi122_168655_object57187:head# 2024-03-20T09:26:25.895 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89ddeac5:::benchmark_data_smithi122_168655_object19731:head# 2024-03-20T09:26:25.895 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89de1cf2:::benchmark_data_smithi122_168655_object108102:head# 2024-03-20T09:26:25.896 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89de6f64:::benchmark_data_smithi122_168655_object63894:head# 2024-03-20T09:26:25.896 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89df0060:::benchmark_data_smithi122_168655_object96651:head# 2024-03-20T09:26:25.896 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89e05545:::benchmark_data_smithi122_168655_object105345:head# 2024-03-20T09:26:25.897 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89e0c888:::benchmark_data_smithi122_168655_object44779:head# 2024-03-20T09:26:25.897 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89e10554:::benchmark_data_smithi122_168655_object114097:head# 2024-03-20T09:26:25.898 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89e10802:::benchmark_data_smithi122_168655_object75165:head# 2024-03-20T09:26:25.898 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89e18faf:::benchmark_data_smithi122_168655_object53599:head# 2024-03-20T09:26:25.898 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89e1b921:::benchmark_data_smithi122_168655_object51894:head# 2024-03-20T09:26:25.899 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89e2011d:::benchmark_data_smithi122_168655_object50585:head# 2024-03-20T09:26:25.899 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89e2cfb9:::benchmark_data_smithi122_168655_object99968:head# 2024-03-20T09:26:25.900 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89e2e6d7:::benchmark_data_smithi122_168655_object86522:head# 2024-03-20T09:26:25.900 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89e2fd25:::benchmark_data_smithi122_168655_object102400:head# 2024-03-20T09:26:25.903 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89e34ab7:::benchmark_data_smithi122_168655_object40817:head# 2024-03-20T09:26:25.903 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89e3509a:::benchmark_data_smithi122_168655_object28972:head# 2024-03-20T09:26:25.903 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89e35428:::benchmark_data_smithi122_168655_object31174:head# 2024-03-20T09:26:25.904 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89e36a18:::benchmark_data_smithi122_168655_object91967:head# 2024-03-20T09:26:25.904 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89e44d91:::benchmark_data_smithi122_168655_object70051:head# 2024-03-20T09:26:25.905 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89e490fa:::benchmark_data_smithi122_168655_object10660:head# 2024-03-20T09:26:25.905 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89e5b798:::benchmark_data_smithi122_168655_object12278:head# 2024-03-20T09:26:25.906 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89e70ad8:::benchmark_data_smithi122_168655_object21782:head# 2024-03-20T09:26:25.906 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89e7ae8f:::benchmark_data_smithi122_168655_object101361:head# 2024-03-20T09:26:25.907 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89e8ea4a:::benchmark_data_smithi122_168655_object71026:head# 2024-03-20T09:26:25.907 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89e92fb8:::benchmark_data_smithi122_168655_object81849:head# 2024-03-20T09:26:25.908 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89e9fa61:::benchmark_data_smithi122_168655_object26375:head# 2024-03-20T09:26:25.908 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89eb1c91:::benchmark_data_smithi122_168655_object37132:head# 2024-03-20T09:26:25.908 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89eb9bec:::benchmark_data_smithi122_168655_object78333:head# 2024-03-20T09:26:25.909 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89ec3faa:::benchmark_data_smithi122_168655_object100123:head# 2024-03-20T09:26:25.909 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89ec8fb9:::benchmark_data_smithi122_168655_object68970:head# 2024-03-20T09:26:25.910 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89ed086e:::benchmark_data_smithi122_168655_object43526:head# 2024-03-20T09:26:25.910 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89ed0beb:::benchmark_data_smithi122_168655_object25853:head# 2024-03-20T09:26:25.911 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89ed6b33:::benchmark_data_smithi122_168655_object39812:head# 2024-03-20T09:26:25.911 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89ed75eb:::benchmark_data_smithi122_168655_object56237:head# 2024-03-20T09:26:25.912 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89ee1961:::benchmark_data_smithi122_168655_object77145:head# 2024-03-20T09:26:25.912 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89ee3ba4:::benchmark_data_smithi122_168655_object27266:head# 2024-03-20T09:26:25.912 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89efa379:::benchmark_data_smithi122_168655_object23757:head# 2024-03-20T09:26:25.913 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89f0cf9a:::benchmark_data_smithi122_168655_object98381:head# 2024-03-20T09:26:25.913 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89f0fff3:::benchmark_data_smithi122_168655_object53701:head# 2024-03-20T09:26:25.914 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89f10b96:::benchmark_data_smithi122_168655_object111119:head# 2024-03-20T09:26:25.914 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89f156e6:::benchmark_data_smithi122_168655_object78843:head# 2024-03-20T09:26:25.915 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89f19164:::benchmark_data_smithi122_168655_object25025:head# 2024-03-20T09:26:25.915 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89f2254c:::benchmark_data_smithi122_168655_object67825:head# 2024-03-20T09:26:25.915 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89f2b891:::benchmark_data_smithi122_168655_object67248:head# 2024-03-20T09:26:25.916 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89f2e07d:::benchmark_data_smithi122_168655_object46947:head# 2024-03-20T09:26:25.918 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89f328d1:::benchmark_data_smithi122_168655_object73809:head# 2024-03-20T09:26:25.919 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89f3dca5:::benchmark_data_smithi122_168655_object71017:head# 2024-03-20T09:26:25.919 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89f3f1c6:::benchmark_data_smithi122_168655_object59842:head# 2024-03-20T09:26:25.919 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89f43af3:::benchmark_data_smithi122_168655_object105114:head# 2024-03-20T09:26:25.920 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89f45328:::benchmark_data_smithi122_168655_object109243:head# 2024-03-20T09:26:25.920 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89f488be:::benchmark_data_smithi122_168655_object100043:head# 2024-03-20T09:26:25.921 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89f51367:::benchmark_data_smithi122_168655_object23162:head# 2024-03-20T09:26:25.921 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89f69163:::benchmark_data_smithi122_168655_object58543:head# 2024-03-20T09:26:25.922 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89f95d1e:::benchmark_data_smithi122_168655_object30636:head# 2024-03-20T09:26:25.922 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89fa3eb5:::benchmark_data_smithi122_168655_object14597:head# 2024-03-20T09:26:25.922 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89fa9794:::benchmark_data_smithi122_168655_object17385:head# 2024-03-20T09:26:25.923 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89fb6277:::benchmark_data_smithi122_168655_object15279:head# 2024-03-20T09:26:25.923 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89fe29fa:::benchmark_data_smithi122_168655_object17798:head# 2024-03-20T09:26:25.923 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89fe8cf5:::benchmark_data_smithi122_168655_object19694:head# 2024-03-20T09:26:25.924 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89febfee:::benchmark_data_smithi122_168655_object87690:head# 2024-03-20T09:26:25.924 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89ff1478:::benchmark_data_smithi122_168655_object48258:head# 2024-03-20T09:26:25.925 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89ffcf04:::benchmark_data_smithi122_168655_object31437:head# 2024-03-20T09:26:25.925 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:89ffdcd8:::benchmark_data_smithi122_168655_object107372:head# 2024-03-20T09:26:25.926 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a01543f:::benchmark_data_smithi122_168655_object105182:head# 2024-03-20T09:26:25.926 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a017857:::benchmark_data_smithi122_168655_object118481:head# 2024-03-20T09:26:25.926 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a01ac64:::benchmark_data_smithi122_168655_object95607:head# 2024-03-20T09:26:25.927 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a01b2b9:::benchmark_data_smithi122_168655_object25293:head# 2024-03-20T09:26:25.928 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a01b6a9:::benchmark_data_smithi122_168655_object95936:head# 2024-03-20T09:26:25.928 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a027842:::benchmark_data_smithi122_168655_object66269:head# 2024-03-20T09:26:25.928 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a034003:::benchmark_data_smithi122_168655_object72466:head# 2024-03-20T09:26:25.929 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a03aac2:::benchmark_data_smithi122_168655_object95518:head# 2024-03-20T09:26:25.929 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a04df79:::benchmark_data_smithi122_168655_object32781:head# 2024-03-20T09:26:25.930 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a064855:::benchmark_data_smithi122_168655_object14866:head# 2024-03-20T09:26:25.930 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a06f249:::benchmark_data_smithi122_168655_object97669:head# 2024-03-20T09:26:25.931 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a073682:::benchmark_data_smithi122_168655_object35635:head# 2024-03-20T09:26:25.931 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a07679b:::benchmark_data_smithi122_168655_object115874:head# 2024-03-20T09:26:25.931 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a077773:::benchmark_data_smithi122_168655_object24676:head# 2024-03-20T09:26:25.932 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a087acf:::benchmark_data_smithi122_168655_object94646:head# 2024-03-20T09:26:25.932 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a08d1b1:::benchmark_data_smithi122_168655_object112601:head# 2024-03-20T09:26:25.933 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a0a153f:::benchmark_data_smithi122_168655_object12367:head# 2024-03-20T09:26:25.933 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a0a7ec1:::benchmark_data_smithi122_168655_object57118:head# 2024-03-20T09:26:25.934 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a0a97e3:::benchmark_data_smithi122_168655_object58508:head# 2024-03-20T09:26:25.934 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a0b6366:::benchmark_data_smithi122_168655_object19848:head# 2024-03-20T09:26:25.934 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a0c0e18:::benchmark_data_smithi122_168655_object94419:head# 2024-03-20T09:26:25.935 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a0c3331:::benchmark_data_smithi122_168655_object23378:head# 2024-03-20T09:26:25.935 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a0cd8fa:::benchmark_data_smithi122_168655_object47928:head# 2024-03-20T09:26:25.936 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a0ea762:::benchmark_data_smithi122_168655_object86994:head# 2024-03-20T09:26:25.936 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a0ede90:::benchmark_data_smithi122_168655_object10929:head# 2024-03-20T09:26:25.937 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a0f9030:::benchmark_data_smithi122_168655_object113521:head# 2024-03-20T09:26:25.937 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a0fda02:::benchmark_data_smithi122_168655_object43725:head# 2024-03-20T09:26:25.938 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a115a30:::benchmark_data_smithi122_168655_object16965:head# 2024-03-20T09:26:25.938 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a119678:::benchmark_data_smithi122_168655_object103591:head# 2024-03-20T09:26:25.938 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a1248a0:::benchmark_data_smithi122_168655_object37468:head# 2024-03-20T09:26:25.939 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a133f4f:::benchmark_data_smithi122_168655_object58967:head# 2024-03-20T09:26:25.939 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a13bbf0:::benchmark_data_smithi122_168655_object18125:head# 2024-03-20T09:26:25.940 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a145a07:::benchmark_data_smithi122_168655_object66996:head# 2024-03-20T09:26:25.940 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a14c55d:::benchmark_data_smithi122_168655_object112672:head# 2024-03-20T09:26:25.940 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a15010b:::benchmark_data_smithi122_168655_object84872:head# 2024-03-20T09:26:25.941 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a151286:::benchmark_data_smithi122_168655_object18087:head# 2024-03-20T09:26:25.941 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a161efb:::benchmark_data_smithi122_168655_object68109:head# 2024-03-20T09:26:25.942 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a183b7a:::benchmark_data_smithi122_168655_object11493:head# 2024-03-20T09:26:25.942 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a186f7e:::benchmark_data_smithi122_168655_object89114:head# 2024-03-20T09:26:25.943 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a18ccfd:::benchmark_data_smithi122_168655_object64453:head# 2024-03-20T09:26:25.943 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a195342:::benchmark_data_smithi122_168655_object99990:head# 2024-03-20T09:26:25.944 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a1b3dc3:::benchmark_data_smithi122_168655_object114080:head# 2024-03-20T09:26:25.944 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a1c2801:::benchmark_data_smithi122_168655_object55644:head# 2024-03-20T09:26:25.944 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a1c4a6b:::benchmark_data_smithi122_168655_object39482:head# 2024-03-20T09:26:25.945 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a1d5ccd:::benchmark_data_smithi122_168655_object18281:head# 2024-03-20T09:26:25.945 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a1dad76:::benchmark_data_smithi122_168655_object41220:head# 2024-03-20T09:26:25.945 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a1dc88f:::benchmark_data_smithi122_168655_object36914:head# 2024-03-20T09:26:25.946 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a1dd735:::benchmark_data_smithi122_168655_object32240:head# 2024-03-20T09:26:25.946 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a1de307:::benchmark_data_smithi122_168655_object51388:head# 2024-03-20T09:26:25.947 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a1de334:::benchmark_data_smithi122_168655_object69994:head# 2024-03-20T09:26:25.947 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a1ec5a4:::benchmark_data_smithi122_168655_object38898:head# 2024-03-20T09:26:25.948 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a1f4286:::benchmark_data_smithi122_168655_object79406:head# 2024-03-20T09:26:25.948 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a2045b6:::benchmark_data_smithi122_168655_object17398:head# 2024-03-20T09:26:25.949 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a2079fd:::benchmark_data_smithi122_168655_object92633:head# 2024-03-20T09:26:25.949 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a218011:::benchmark_data_smithi122_168655_object99028:head# 2024-03-20T09:26:25.949 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a22033d:::benchmark_data_smithi122_168655_object92000:head# 2024-03-20T09:26:25.950 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a22081c:::benchmark_data_smithi122_168655_object62344:head# 2024-03-20T09:26:25.950 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a230219:::benchmark_data_smithi122_168655_object59037:head# 2024-03-20T09:26:25.951 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a2353d0:::benchmark_data_smithi122_168655_object92444:head# 2024-03-20T09:26:25.951 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a23dc1f:::benchmark_data_smithi122_168655_object42967:head# 2024-03-20T09:26:25.952 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a246aa8:::benchmark_data_smithi122_168655_object104458:head# 2024-03-20T09:26:25.952 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a24a678:::benchmark_data_smithi122_168655_object40060:head# 2024-03-20T09:26:25.952 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a25ca05:::benchmark_data_smithi122_168655_object72400:head# 2024-03-20T09:26:25.953 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a267c21:::benchmark_data_smithi122_168655_object91483:head# 2024-03-20T09:26:25.953 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a268770:::benchmark_data_smithi122_168655_object56440:head# 2024-03-20T09:26:25.954 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a26cd6c:::benchmark_data_smithi122_168655_object43521:head# 2024-03-20T09:26:25.954 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a2711b1:::benchmark_data_smithi122_168655_object21303:head# 2024-03-20T09:26:25.954 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a278cd0:::benchmark_data_smithi122_168655_object21274:head# 2024-03-20T09:26:25.955 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a280193:::benchmark_data_smithi122_168655_object70409:head# 2024-03-20T09:26:25.955 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a295069:::benchmark_data_smithi122_168655_object96061:head# 2024-03-20T09:26:25.956 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a2a2b36:::benchmark_data_smithi122_168655_object110438:head# 2024-03-20T09:26:25.956 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a2a74d8:::benchmark_data_smithi122_168655_object79746:head# 2024-03-20T09:26:25.957 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a2a9956:::benchmark_data_smithi122_168655_object32363:head# 2024-03-20T09:26:25.957 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a2b46e5:::benchmark_data_smithi122_168655_object118862:head# 2024-03-20T09:26:25.957 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a2d02e0:::benchmark_data_smithi122_168655_object97282:head# 2024-03-20T09:26:25.958 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a2df037:::benchmark_data_smithi122_168655_object24529:head# 2024-03-20T09:26:25.958 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a308090:::benchmark_data_smithi122_168655_object33274:head# 2024-03-20T09:26:25.959 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a309199:::benchmark_data_smithi122_168655_object47332:head# 2024-03-20T09:26:25.959 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a32a8e8:::benchmark_data_smithi122_168655_object19620:head# 2024-03-20T09:26:25.959 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a32b4ba:::benchmark_data_smithi122_168655_object109625:head# 2024-03-20T09:26:25.962 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a332341:::benchmark_data_smithi122_168655_object51870:head# 2024-03-20T09:26:25.962 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a33eb7a:::benchmark_data_smithi122_168655_object103119:head# 2024-03-20T09:26:25.963 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a3417a1:::benchmark_data_smithi122_168655_object17304:head# 2024-03-20T09:26:25.963 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a341c79:::benchmark_data_smithi122_168655_object76560:head# 2024-03-20T09:26:25.963 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a344b7c:::benchmark_data_smithi122_168655_object65782:head# 2024-03-20T09:26:25.964 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a3453c7:::benchmark_data_smithi122_168655_object76113:head# 2024-03-20T09:26:25.964 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a347c04:::benchmark_data_smithi122_168655_object91405:head# 2024-03-20T09:26:25.965 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a34c2f7:::benchmark_data_smithi122_168655_object32915:head# 2024-03-20T09:26:25.965 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a356fa6:::benchmark_data_smithi122_168655_object16206:head# 2024-03-20T09:26:25.966 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a35ad41:::benchmark_data_smithi122_168655_object12545:head# 2024-03-20T09:26:25.966 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a35be32:::benchmark_data_smithi122_168655_object27899:head# 2024-03-20T09:26:25.966 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a367cba:::benchmark_data_smithi122_168655_object33768:head# 2024-03-20T09:26:25.967 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a36d783:::benchmark_data_smithi122_168655_object58631:head# 2024-03-20T09:26:25.968 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a376737:::benchmark_data_smithi122_168655_object85839:head# 2024-03-20T09:26:25.969 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a3806fa:::benchmark_data_smithi122_168655_object35651:head# 2024-03-20T09:26:25.970 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a385d6e:::benchmark_data_smithi122_168655_object16268:head# 2024-03-20T09:26:25.970 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a387cc1:::benchmark_data_smithi122_168655_object47038:head# 2024-03-20T09:26:25.970 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a395afb:::benchmark_data_smithi122_168655_object38149:head# 2024-03-20T09:26:25.971 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a3999a6:::benchmark_data_smithi122_168655_object33852:head# 2024-03-20T09:26:25.971 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a3add29:::benchmark_data_smithi122_168655_object24813:head# 2024-03-20T09:26:25.972 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a3f1dfe:::benchmark_data_smithi122_168655_object83977:head# 2024-03-20T09:26:25.972 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a3fce8d:::benchmark_data_smithi122_168655_object27235:head# 2024-03-20T09:26:25.972 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a4035bb:::benchmark_data_smithi122_168655_object100447:head# 2024-03-20T09:26:25.974 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a40cdaf:::benchmark_data_smithi122_168655_object105313:head# 2024-03-20T09:26:25.974 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a41da0c:::benchmark_data_smithi122_168655_object72243:head# 2024-03-20T09:26:25.974 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a4316d3:::benchmark_data_smithi122_168655_object119353:head# 2024-03-20T09:26:25.974 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a43bd5d:::benchmark_data_smithi122_168655_object104098:head# 2024-03-20T09:26:25.975 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a458e28:::benchmark_data_smithi122_168655_object60985:head# 2024-03-20T09:26:25.975 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a46365e:::benchmark_data_smithi122_168655_object46753:head# 2024-03-20T09:26:25.975 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a46f601:::benchmark_data_smithi122_168655_object88519:head# 2024-03-20T09:26:25.976 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a46fed6:::benchmark_data_smithi122_168655_object109509:head# 2024-03-20T09:26:25.976 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a474401:::benchmark_data_smithi122_168655_object49609:head# 2024-03-20T09:26:25.977 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a488e17:::benchmark_data_smithi122_168655_object113909:head# 2024-03-20T09:26:25.977 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a49079f:::benchmark_data_smithi122_168655_object15486:head# 2024-03-20T09:26:25.977 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a496b46:::benchmark_data_smithi122_168655_object29112:head# 2024-03-20T09:26:25.978 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a49fd00:::benchmark_data_smithi122_168655_object30880:head# 2024-03-20T09:26:25.978 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a4a03ff:::benchmark_data_smithi122_168655_object40839:head# 2024-03-20T09:26:25.979 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a4a5c71:::benchmark_data_smithi122_168655_object64382:head# 2024-03-20T09:26:25.979 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a4a97dd:::benchmark_data_smithi122_168655_object21855:head# 2024-03-20T09:26:25.979 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a4ac5b3:::benchmark_data_smithi122_168655_object27046:head# 2024-03-20T09:26:25.980 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a4af637:::benchmark_data_smithi122_168655_object88832:head# 2024-03-20T09:26:25.980 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a4b3398:::benchmark_data_smithi122_168655_object106647:head# 2024-03-20T09:26:25.980 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a4b94c1:::benchmark_data_smithi122_168655_object79240:head# 2024-03-20T09:26:25.981 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a4c1d1b:::benchmark_data_smithi122_168655_object77216:head# 2024-03-20T09:26:25.981 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a4cbcb4:::benchmark_data_smithi122_168655_object79487:head# 2024-03-20T09:26:25.982 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a4d5259:::benchmark_data_smithi122_168655_object32499:head# 2024-03-20T09:26:25.982 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a4e1454:::benchmark_data_smithi122_168655_object68196:head# 2024-03-20T09:26:25.982 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a4f8879:::benchmark_data_smithi122_168655_object80032:head# 2024-03-20T09:26:25.983 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a50f821:::benchmark_data_smithi122_168655_object45311:head# 2024-03-20T09:26:25.983 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a512dc2:::benchmark_data_smithi122_168655_object96359:head# 2024-03-20T09:26:25.984 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a51ce3e:::benchmark_data_smithi122_168655_object65155:head# 2024-03-20T09:26:25.984 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a528b89:::benchmark_data_smithi122_168655_object65109:head# 2024-03-20T09:26:25.985 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a52be47:::benchmark_data_smithi122_168655_object30314:head# 2024-03-20T09:26:25.985 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a52ed3b:::benchmark_data_smithi122_168655_object66646:head# 2024-03-20T09:26:25.987 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a53d650:::benchmark_data_smithi122_168655_object67241:head# 2024-03-20T09:26:25.987 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a5496e4:::benchmark_data_smithi122_168655_object45533:head# 2024-03-20T09:26:25.988 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a5515ab:::benchmark_data_smithi122_168655_object110570:head# 2024-03-20T09:26:25.988 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a557e12:::benchmark_data_smithi122_168655_object11343:head# 2024-03-20T09:26:25.989 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a576788:::benchmark_data_smithi122_168655_object32293:head# 2024-03-20T09:26:25.989 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a57ae52:::benchmark_data_smithi122_168655_object24665:head# 2024-03-20T09:26:25.989 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a5b3130:::benchmark_data_smithi122_168655_object115047:head# 2024-03-20T09:26:25.990 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a5bd901:::benchmark_data_smithi122_168655_object69484:head# 2024-03-20T09:26:25.990 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a5d1bab:::benchmark_data_smithi122_168655_object43056:head# 2024-03-20T09:26:25.991 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a5ee9a1:::benchmark_data_smithi122_168655_object92531:head# 2024-03-20T09:26:25.991 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a60ab00:::benchmark_data_smithi122_168655_object74959:head# 2024-03-20T09:26:25.991 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a60ce63:::benchmark_data_smithi122_168655_object105360:head# 2024-03-20T09:26:25.992 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a6170cd:::benchmark_data_smithi122_168655_object63324:head# 2024-03-20T09:26:25.992 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a6188fc:::benchmark_data_smithi122_168655_object103892:head# 2024-03-20T09:26:25.993 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a62e695:::benchmark_data_smithi122_168655_object97427:head# 2024-03-20T09:26:25.993 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a637c0b:::benchmark_data_smithi122_168655_object59869:head# 2024-03-20T09:26:25.994 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a63ef5e:::benchmark_data_smithi122_168655_object59993:head# 2024-03-20T09:26:25.994 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a645d4c:::benchmark_data_smithi122_168655_object60216:head# 2024-03-20T09:26:25.995 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a652d1c:::benchmark_data_smithi122_168655_object50981:head# 2024-03-20T09:26:25.995 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a67b8b6:::benchmark_data_smithi122_168655_object114811:head# 2024-03-20T09:26:25.995 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a67bde5:::benchmark_data_smithi122_168655_object93133:head# 2024-03-20T09:26:25.996 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a67d5c2:::benchmark_data_smithi122_168655_object89984:head# 2024-03-20T09:26:25.996 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a689dcc:::benchmark_data_smithi122_168655_object109281:head# 2024-03-20T09:26:25.997 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a68c83d:::benchmark_data_smithi122_168655_object49951:head# 2024-03-20T09:26:25.998 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a698802:::benchmark_data_smithi122_168655_object63186:head# 2024-03-20T09:26:25.998 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a69bf26:::benchmark_data_smithi122_168655_object28574:head# 2024-03-20T09:26:25.998 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a6a2807:::benchmark_data_smithi122_168655_object56924:head# 2024-03-20T09:26:25.999 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a6a4397:::benchmark_data_smithi122_168655_object55109:head# 2024-03-20T09:26:25.999 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a6ac4b3:::benchmark_data_smithi122_168655_object87898:head# 2024-03-20T09:26:26.000 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a6cdd7c:::benchmark_data_smithi122_168655_object24752:head# 2024-03-20T09:26:26.000 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a6d3fb3:::benchmark_data_smithi122_168655_object45736:head# 2024-03-20T09:26:26.001 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a6e5bd6:::benchmark_data_smithi122_168655_object72156:head# 2024-03-20T09:26:26.001 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a6ef1a8:::benchmark_data_smithi122_168655_object98334:head# 2024-03-20T09:26:26.002 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a6f385b:::benchmark_data_smithi122_168655_object61584:head# 2024-03-20T09:26:26.002 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a6f8203:::benchmark_data_smithi122_168655_object91904:head# 2024-03-20T09:26:26.002 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a6fd74a:::benchmark_data_smithi122_168655_object76609:head# 2024-03-20T09:26:26.003 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a6fd8e2:::benchmark_data_smithi122_168655_object105809:head# 2024-03-20T09:26:26.003 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a7000f4:::benchmark_data_smithi122_168655_object111809:head# 2024-03-20T09:26:26.004 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a7008f7:::benchmark_data_smithi122_168655_object79261:head# 2024-03-20T09:26:26.004 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a708b8c:::benchmark_data_smithi122_168655_object109244:head# 2024-03-20T09:26:26.004 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a736f95:::benchmark_data_smithi122_168655_object92016:head# 2024-03-20T09:26:26.005 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a746788:::benchmark_data_smithi122_168655_object68432:head# 2024-03-20T09:26:26.005 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a74f2e6:::benchmark_data_smithi122_168655_object107558:head# 2024-03-20T09:26:26.008 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a760ef2:::benchmark_data_smithi122_168655_object15733:head# 2024-03-20T09:26:26.008 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a76f4a0:::benchmark_data_smithi122_168655_object22793:head# 2024-03-20T09:26:26.009 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a781415:::benchmark_data_smithi122_168655_object114252:head# 2024-03-20T09:26:26.009 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a79e76a:::benchmark_data_smithi122_168655_object94586:head# 2024-03-20T09:26:26.010 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a7ab687:::benchmark_data_smithi122_168655_object23434:head# 2024-03-20T09:26:26.010 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a7b9b6f:::benchmark_data_smithi122_168655_object67600:head# 2024-03-20T09:26:26.010 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a7ca337:::benchmark_data_smithi122_168655_object77748:head# 2024-03-20T09:26:26.011 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a7cbcf7:::benchmark_data_smithi122_168655_object17188:head# 2024-03-20T09:26:26.011 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a7d4e7b:::benchmark_data_smithi122_168655_object66513:head# 2024-03-20T09:26:26.012 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a7d7bbb:::benchmark_data_smithi122_168655_object40222:head# 2024-03-20T09:26:26.012 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a7dd137:::benchmark_data_smithi122_168655_object24159:head# 2024-03-20T09:26:26.012 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a7df4e9:::benchmark_data_smithi122_168655_object24162:head# 2024-03-20T09:26:26.013 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a7ed153:::benchmark_data_smithi122_168655_object104033:head# 2024-03-20T09:26:26.013 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a7f087b:::benchmark_data_smithi122_168655_object33456:head# 2024-03-20T09:26:26.014 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a7f99b8:::benchmark_data_smithi122_168655_object21732:head# 2024-03-20T09:26:26.014 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a7fb170:::benchmark_data_smithi122_168655_object81724:head# 2024-03-20T09:26:26.015 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a805ab0:::benchmark_data_smithi122_168655_object55127:head# 2024-03-20T09:26:26.015 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a82bcda:::benchmark_data_smithi122_168655_object45012:head# 2024-03-20T09:26:26.016 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a832cc2:::benchmark_data_smithi122_168655_object31491:head# 2024-03-20T09:26:26.016 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a83da97:::benchmark_data_smithi122_168655_object31857:head# 2024-03-20T09:26:26.017 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a842c43:::benchmark_data_smithi122_168655_object43038:head# 2024-03-20T09:26:26.017 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a84c5f4:::benchmark_data_smithi122_168655_object24812:head# 2024-03-20T09:26:26.017 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a84ff2b:::benchmark_data_smithi122_168655_object28873:head# 2024-03-20T09:26:26.018 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a8509db:::benchmark_data_smithi122_168655_object44488:head# 2024-03-20T09:26:26.018 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a852ddf:::benchmark_data_smithi122_168655_object36540:head# 2024-03-20T09:26:26.019 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a85b04f:::benchmark_data_smithi122_168655_object85330:head# 2024-03-20T09:26:26.019 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a8650bd:::benchmark_data_smithi122_168655_object72912:head# 2024-03-20T09:26:26.019 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a876793:::benchmark_data_smithi122_168655_object63858:head# 2024-03-20T09:26:26.020 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a87e696:::benchmark_data_smithi122_168655_object14931:head# 2024-03-20T09:26:26.020 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a883b09:::benchmark_data_smithi122_168655_object51276:head# 2024-03-20T09:26:26.021 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a88563c:::benchmark_data_smithi122_168655_object86786:head# 2024-03-20T09:26:26.021 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a88ac4a:::benchmark_data_smithi122_168655_object41600:head# 2024-03-20T09:26:26.022 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a8a2f8d:::benchmark_data_smithi122_168655_object88280:head# 2024-03-20T09:26:26.022 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a8a436e:::benchmark_data_smithi122_168655_object78631:head# 2024-03-20T09:26:26.022 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a8a462d:::benchmark_data_smithi122_168655_object111559:head# 2024-03-20T09:26:26.023 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a8a5b2f:::benchmark_data_smithi122_168655_object37871:head# 2024-03-20T09:26:26.023 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a8a7bb5:::benchmark_data_smithi122_168655_object52760:head# 2024-03-20T09:26:26.023 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a8aaec0:::benchmark_data_smithi122_168655_object51439:head# 2024-03-20T09:26:26.024 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a8c38cc:::benchmark_data_smithi122_168655_object64341:head# 2024-03-20T09:26:26.024 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a8d54a8:::benchmark_data_smithi122_168655_object81913:head# 2024-03-20T09:26:26.025 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a8d90ca:::benchmark_data_smithi122_168655_object24500:head# 2024-03-20T09:26:26.025 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a8ec2ae:::benchmark_data_smithi122_168655_object24777:head# 2024-03-20T09:26:26.025 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a903f1d:::benchmark_data_smithi122_168655_object74320:head# 2024-03-20T09:26:26.026 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a90d884:::benchmark_data_smithi122_168655_object73196:head# 2024-03-20T09:26:26.026 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a91c812:::benchmark_data_smithi122_168655_object98170:head# 2024-03-20T09:26:26.027 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a91fb1c:::benchmark_data_smithi122_168655_object116800:head# 2024-03-20T09:26:26.027 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a93b974:::benchmark_data_smithi122_168655_object58029:head# 2024-03-20T09:26:26.028 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a94443a:::benchmark_data_smithi122_168655_object88648:head# 2024-03-20T09:26:26.028 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a9526ac:::benchmark_data_smithi122_168655_object52784:head# 2024-03-20T09:26:26.029 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a953edd:::benchmark_data_smithi122_168655_object62766:head# 2024-03-20T09:26:26.029 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a953fe3:::benchmark_data_smithi122_168655_object79742:head# 2024-03-20T09:26:26.029 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a95ed76:::benchmark_data_smithi122_168655_object59570:head# 2024-03-20T09:26:26.030 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a96d984:::benchmark_data_smithi122_168655_object50884:head# 2024-03-20T09:26:26.030 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a9780c5:::benchmark_data_smithi122_168655_object25082:head# 2024-03-20T09:26:26.031 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a97f90f:::benchmark_data_smithi122_168655_object57031:head# 2024-03-20T09:26:26.031 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a98a5c7:::benchmark_data_smithi122_168655_object45053:head# 2024-03-20T09:26:26.032 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a99bdf5:::benchmark_data_smithi122_168655_object56906:head# 2024-03-20T09:26:26.032 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a9bbe18:::benchmark_data_smithi122_168655_object35496:head# 2024-03-20T09:26:26.032 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a9c2712:::benchmark_data_smithi122_168655_object48089:head# 2024-03-20T09:26:26.033 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a9c8757:::benchmark_data_smithi122_168655_object37157:head# 2024-03-20T09:26:26.033 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a9c9880:::benchmark_data_smithi122_168655_object112785:head# 2024-03-20T09:26:26.034 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a9d3744:::benchmark_data_smithi122_168655_object17261:head# 2024-03-20T09:26:26.034 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a9ee42e:::benchmark_data_smithi122_168655_object61493:head# 2024-03-20T09:26:26.035 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a9fa33b:::benchmark_data_smithi122_168655_object98969:head# 2024-03-20T09:26:26.035 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8a9fd13d:::benchmark_data_smithi122_168655_object93076:head# 2024-03-20T09:26:26.035 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aa0109b:::benchmark_data_smithi122_168655_object118105:head# 2024-03-20T09:26:26.036 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aa0eff0:::benchmark_data_smithi122_168655_object84776:head# 2024-03-20T09:26:26.036 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aa1a41c:::benchmark_data_smithi122_168655_object23916:head# 2024-03-20T09:26:26.037 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aa1a98d:::benchmark_data_smithi122_168655_object64383:head# 2024-03-20T09:26:26.037 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aa20e13:::benchmark_data_smithi122_168655_object84879:head# 2024-03-20T09:26:26.038 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aa297e3:::benchmark_data_smithi122_168655_object71479:head# 2024-03-20T09:26:26.038 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aa29c2c:::benchmark_data_smithi122_168655_object33318:head# 2024-03-20T09:26:26.039 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aa30274:::benchmark_data_smithi122_168655_object111191:head# 2024-03-20T09:26:26.039 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aa32de7:::benchmark_data_smithi122_168655_object110769:head# 2024-03-20T09:26:26.040 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aa37cc9:::benchmark_data_smithi122_168655_object98501:head# 2024-03-20T09:26:26.040 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aa3b1ef:::benchmark_data_smithi122_168655_object65607:head# 2024-03-20T09:26:26.041 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aa3f85e:::benchmark_data_smithi122_168655_object28560:head# 2024-03-20T09:26:26.041 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aa42dde:::benchmark_data_smithi122_168655_object78235:head# 2024-03-20T09:26:26.041 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aa4d1aa:::benchmark_data_smithi122_168655_object111736:head# 2024-03-20T09:26:26.042 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aa63903:::benchmark_data_smithi122_168655_object97765:head# 2024-03-20T09:26:26.042 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aa74a9c:::benchmark_data_smithi122_168655_object39717:head# 2024-03-20T09:26:26.043 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aa7f578:::benchmark_data_smithi122_168655_object99600:head# 2024-03-20T09:26:26.043 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aa8c3c7:::benchmark_data_smithi122_168655_object99915:head# 2024-03-20T09:26:26.044 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aa98a6e:::benchmark_data_smithi122_168655_object46767:head# 2024-03-20T09:26:26.044 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aaa6d72:::benchmark_data_smithi122_168655_object71993:head# 2024-03-20T09:26:26.044 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aac2484:::benchmark_data_smithi122_168655_object99969:head# 2024-03-20T09:26:26.045 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aac72e3:::benchmark_data_smithi122_168655_object95071:head# 2024-03-20T09:26:26.045 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aac8963:::benchmark_data_smithi122_168655_object96824:head# 2024-03-20T09:26:26.046 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aacd278:::benchmark_data_smithi122_168655_object65957:head# 2024-03-20T09:26:26.046 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aad4dd6:::benchmark_data_smithi122_168655_object44070:head# 2024-03-20T09:26:26.047 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aadebd3:::benchmark_data_smithi122_168655_object21019:head# 2024-03-20T09:26:26.047 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aae8e6e:::benchmark_data_smithi122_168655_object63974:head# 2024-03-20T09:26:26.047 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aae934d:::benchmark_data_smithi122_168655_object39059:head# 2024-03-20T09:26:26.048 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aaf04b7:::benchmark_data_smithi122_168655_object37347:head# 2024-03-20T09:26:26.048 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aaf6a90:::benchmark_data_smithi122_168655_object113988:head# 2024-03-20T09:26:26.049 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aaf9008:::benchmark_data_smithi122_168655_object76317:head# 2024-03-20T09:26:26.049 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aafbb11:::benchmark_data_smithi122_168655_object87924:head# 2024-03-20T09:26:26.050 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ab01391:::benchmark_data_smithi122_168655_object14130:head# 2024-03-20T09:26:26.050 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ab032ab:::benchmark_data_smithi122_168655_object87050:head# 2024-03-20T09:26:26.051 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ab06af1:::benchmark_data_smithi122_168655_object70996:head# 2024-03-20T09:26:26.051 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ab0a2af:::benchmark_data_smithi122_168655_object15004:head# 2024-03-20T09:26:26.051 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ab21620:::benchmark_data_smithi122_168655_object72847:head# 2024-03-20T09:26:26.052 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ab30278:::benchmark_data_smithi122_168655_object60576:head# 2024-03-20T09:26:26.052 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ab40779:::benchmark_data_smithi122_168655_object73726:head# 2024-03-20T09:26:26.052 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ab4732e:::benchmark_data_smithi122_168655_object89981:head# 2024-03-20T09:26:26.053 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ab4f961:::benchmark_data_smithi122_168655_object96871:head# 2024-03-20T09:26:26.054 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ab65f64:::benchmark_data_smithi122_168655_object58492:head# 2024-03-20T09:26:26.055 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ab8154f:::benchmark_data_smithi122_168655_object111397:head# 2024-03-20T09:26:26.055 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ab8eddf:::benchmark_data_smithi122_168655_object51062:head# 2024-03-20T09:26:26.056 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ab9ea30:::benchmark_data_smithi122_168655_object65437:head# 2024-03-20T09:26:26.056 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8abbc74d:::benchmark_data_smithi122_168655_object104481:head# 2024-03-20T09:26:26.056 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8abc9ff7:::benchmark_data_smithi122_168655_object115132:head# 2024-03-20T09:26:26.057 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8abce136:::benchmark_data_smithi122_168655_object48633:head# 2024-03-20T09:26:26.058 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8abe2195:::benchmark_data_smithi122_168655_object30633:head# 2024-03-20T09:26:26.058 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8abfb813:::benchmark_data_smithi122_168655_object25462:head# 2024-03-20T09:26:26.058 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8abfe3bc:::benchmark_data_smithi122_168655_object54403:head# 2024-03-20T09:26:26.059 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8abfeb9d:::benchmark_data_smithi122_168655_object60892:head# 2024-03-20T09:26:26.059 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ac0730d:::benchmark_data_smithi122_168655_object43669:head# 2024-03-20T09:26:26.060 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ac1123b:::benchmark_data_smithi122_168655_object51365:head# 2024-03-20T09:26:26.060 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ac13aa6:::benchmark_data_smithi122_168655_object89751:head# 2024-03-20T09:26:26.061 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ac1ae49:::benchmark_data_smithi122_168655_object79026:head# 2024-03-20T09:26:26.061 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ac1bb27:::benchmark_data_smithi122_168655_object73659:head# 2024-03-20T09:26:26.062 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ac20f79:::benchmark_data_smithi122_168655_object81017:head# 2024-03-20T09:26:26.062 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ac2e56f:::benchmark_data_smithi122_168655_object37373:head# 2024-03-20T09:26:26.063 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ac37f1e:::benchmark_data_smithi122_168655_object113131:head# 2024-03-20T09:26:26.063 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ac3ca41:::benchmark_data_smithi122_168655_object99018:head# 2024-03-20T09:26:26.063 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ac3ecaf:::benchmark_data_smithi122_168655_object109002:head# 2024-03-20T09:26:26.064 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ac44dff:::benchmark_data_smithi122_168655_object48603:head# 2024-03-20T09:26:26.065 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ac454ff:::benchmark_data_smithi122_168655_object80986:head# 2024-03-20T09:26:26.065 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ac4aab0:::benchmark_data_smithi122_168655_object86296:head# 2024-03-20T09:26:26.066 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ac4b5a3:::benchmark_data_smithi122_168655_object75699:head# 2024-03-20T09:26:26.066 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ac7fa0b:::benchmark_data_smithi122_168655_object28281:head# 2024-03-20T09:26:26.066 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ac8610a:::benchmark_data_smithi122_168655_object33761:head# 2024-03-20T09:26:26.067 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ac93939:::benchmark_data_smithi122_168655_object116290:head# 2024-03-20T09:26:26.067 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aca5530:::benchmark_data_smithi122_168655_object104719:head# 2024-03-20T09:26:26.068 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aca58d1:::benchmark_data_smithi122_168655_object85951:head# 2024-03-20T09:26:26.068 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aca7d73:::benchmark_data_smithi122_168655_object60357:head# 2024-03-20T09:26:26.069 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8acab157:::benchmark_data_smithi122_168655_object74995:head# 2024-03-20T09:26:26.069 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8acc31ef:::benchmark_data_smithi122_168655_object35611:head# 2024-03-20T09:26:26.069 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8acc3264:::benchmark_data_smithi122_168655_object54732:head# 2024-03-20T09:26:26.070 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8acd5f27:::benchmark_data_smithi122_168655_object30284:head# 2024-03-20T09:26:26.070 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8acd7a10:::benchmark_data_smithi122_168655_object69673:head# 2024-03-20T09:26:26.071 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ace0f9d:::benchmark_data_smithi122_168655_object71180:head# 2024-03-20T09:26:26.071 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ace1450:::benchmark_data_smithi122_168655_object76226:head# 2024-03-20T09:26:26.071 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8acec1fd:::benchmark_data_smithi122_168655_object71187:head# 2024-03-20T09:26:26.072 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8acef77a:::benchmark_data_smithi122_168655_object34919:head# 2024-03-20T09:26:26.072 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8acfab8a:::benchmark_data_smithi122_168655_object62944:head# 2024-03-20T09:26:26.073 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ad0a171:::benchmark_data_smithi122_168655_object57193:head# 2024-03-20T09:26:26.073 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ad10efe:::benchmark_data_smithi122_168655_object44674:head# 2024-03-20T09:26:26.074 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ad24917:::benchmark_data_smithi122_168655_object79576:head# 2024-03-20T09:26:26.074 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ad249ab:::benchmark_data_smithi122_168655_object106213:head# 2024-03-20T09:26:26.074 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ad28ec5:::benchmark_data_smithi122_168655_object112030:head# 2024-03-20T09:26:26.075 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ad293b5:::benchmark_data_smithi122_168655_object68932:head# 2024-03-20T09:26:26.075 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ad4e286:::benchmark_data_smithi122_168655_object83055:head# 2024-03-20T09:26:26.076 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ad625f4:::benchmark_data_smithi122_168655_object35667:head# 2024-03-20T09:26:26.076 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ad66853:::benchmark_data_smithi122_168655_object66855:head# 2024-03-20T09:26:26.076 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ad7161d:::benchmark_data_smithi122_168655_object104076:head# 2024-03-20T09:26:26.078 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ad77720:::benchmark_data_smithi122_168655_object101511:head# 2024-03-20T09:26:26.078 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ad7c5d9:::benchmark_data_smithi122_168655_object76462:head# 2024-03-20T09:26:26.079 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ad7d689:::benchmark_data_smithi122_168655_object71759:head# 2024-03-20T09:26:26.079 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ad7ffce:::benchmark_data_smithi122_168655_object117865:head# 2024-03-20T09:26:26.080 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ad8297b:::benchmark_data_smithi122_168655_object117685:head# 2024-03-20T09:26:26.081 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ad873b7:::benchmark_data_smithi122_168655_object106132:head# 2024-03-20T09:26:26.081 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ad9bce3:::benchmark_data_smithi122_168655_object45140:head# 2024-03-20T09:26:26.081 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ada081e:::benchmark_data_smithi122_168655_object11713:head# 2024-03-20T09:26:26.082 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ada0c56:::benchmark_data_smithi122_168655_object64935:head# 2024-03-20T09:26:26.082 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ada58d7:::benchmark_data_smithi122_168655_object63991:head# 2024-03-20T09:26:26.082 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8adb29ec:::benchmark_data_smithi122_168655_object30648:head# 2024-03-20T09:26:26.083 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8adc3333:::benchmark_data_smithi122_168655_object116414:head# 2024-03-20T09:26:26.083 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8adc49e9:::benchmark_data_smithi122_168655_object96451:head# 2024-03-20T09:26:26.084 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8adc716d:::benchmark_data_smithi122_168655_object30522:head# 2024-03-20T09:26:26.084 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8adcf96d:::benchmark_data_smithi122_168655_object61160:head# 2024-03-20T09:26:26.085 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8add55d2:::benchmark_data_smithi122_168655_object117035:head# 2024-03-20T09:26:26.085 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8adda190:::benchmark_data_smithi122_168655_object89287:head# 2024-03-20T09:26:26.085 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8adda2f7:::benchmark_data_smithi122_168655_object103815:head# 2024-03-20T09:26:26.086 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8adee73b:::benchmark_data_smithi122_168655_object92512:head# 2024-03-20T09:26:26.086 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ae13940:::benchmark_data_smithi122_168655_object101028:head# 2024-03-20T09:26:26.087 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ae1c21e:::benchmark_data_smithi122_168655_object69600:head# 2024-03-20T09:26:26.087 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ae23023:::benchmark_data_smithi122_168655_object12613:head# 2024-03-20T09:26:26.088 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ae25511:::benchmark_data_smithi122_168655_object49327:head# 2024-03-20T09:26:26.088 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ae3f17e:::benchmark_data_smithi122_168655_object116744:head# 2024-03-20T09:26:26.089 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ae45adb:::benchmark_data_smithi122_168655_object32717:head# 2024-03-20T09:26:26.089 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ae493b7:::benchmark_data_smithi122_168655_object80808:head# 2024-03-20T09:26:26.089 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ae65de3:::benchmark_data_smithi122_168655_object45624:head# 2024-03-20T09:26:26.090 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ae695df:::benchmark_data_smithi122_168655_object17278:head# 2024-03-20T09:26:26.090 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ae7772e:::benchmark_data_smithi122_168655_object33790:head# 2024-03-20T09:26:26.091 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ae7a267:::benchmark_data_smithi122_168655_object56648:head# 2024-03-20T09:26:26.091 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ae7a866:::benchmark_data_smithi122_168655_object76210:head# 2024-03-20T09:26:26.092 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ae82470:::benchmark_data_smithi122_168655_object17684:head# 2024-03-20T09:26:26.092 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ae8b83a:::benchmark_data_smithi122_168655_object113721:head# 2024-03-20T09:26:26.092 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ae98e7c:::benchmark_data_smithi122_168655_object99518:head# 2024-03-20T09:26:26.093 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aea1abb:::benchmark_data_smithi122_168655_object72081:head# 2024-03-20T09:26:26.093 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aeaaf2a:::benchmark_data_smithi122_168655_object40200:head# 2024-03-20T09:26:26.094 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aeb0053:::benchmark_data_smithi122_168655_object75126:head# 2024-03-20T09:26:26.094 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aeb2aa0:::benchmark_data_smithi122_168655_object115200:head# 2024-03-20T09:26:26.094 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aebee31:::benchmark_data_smithi122_168655_object100648:head# 2024-03-20T09:26:26.095 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aec4c43:::benchmark_data_smithi122_168655_object52101:head# 2024-03-20T09:26:26.095 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aec8ee3:::benchmark_data_smithi122_168655_object54849:head# 2024-03-20T09:26:26.096 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aee1f2d:::benchmark_data_smithi122_168655_object79595:head# 2024-03-20T09:26:26.096 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8aee38bf:::benchmark_data_smithi122_168655_object118055:head# 2024-03-20T09:26:26.097 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8af07aad:::benchmark_data_smithi122_168655_object112971:head# 2024-03-20T09:26:26.097 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8af0aa92:::benchmark_data_smithi122_168655_object87272:head# 2024-03-20T09:26:26.098 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8af23494:::benchmark_data_smithi122_168655_object60210:head# 2024-03-20T09:26:26.098 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8af26e92:::benchmark_data_smithi122_168655_object98697:head# 2024-03-20T09:26:26.099 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8af281da:::benchmark_data_smithi122_168655_object71822:head# 2024-03-20T09:26:26.099 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8af2d0d1:::benchmark_data_smithi122_168655_object99289:head# 2024-03-20T09:26:26.100 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8af42c7e:::benchmark_data_smithi122_168655_object23015:head# 2024-03-20T09:26:26.100 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8af66966:::benchmark_data_smithi122_168655_object77002:head# 2024-03-20T09:26:26.100 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8af66eb8:::benchmark_data_smithi122_168655_object46901:head# 2024-03-20T09:26:26.101 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8afa160e:::benchmark_data_smithi122_168655_object12708:head# 2024-03-20T09:26:26.101 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8afa21c2:::benchmark_data_smithi122_168655_object40847:head# 2024-03-20T09:26:26.102 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8afa7612:::benchmark_data_smithi122_168655_object62257:head# 2024-03-20T09:26:26.102 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8afb4eae:::benchmark_data_smithi122_168655_object55434:head# 2024-03-20T09:26:26.103 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8afb74b1:::benchmark_data_smithi122_168655_object47147:head# 2024-03-20T09:26:26.103 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8afca9de:::benchmark_data_smithi122_168655_object20347:head# 2024-03-20T09:26:26.103 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8afcb108:::benchmark_data_smithi122_168655_object67368:head# 2024-03-20T09:26:26.104 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8afcdeb5:::benchmark_data_smithi122_168655_object43633:head# 2024-03-20T09:26:26.104 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8afd4d89:::benchmark_data_smithi122_168655_object88387:head# 2024-03-20T09:26:26.105 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8afe1965:::benchmark_data_smithi122_168655_object67904:head# 2024-03-20T09:26:26.105 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8afebfa6:::benchmark_data_smithi122_168655_object38084:head# 2024-03-20T09:26:26.106 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b009f6c:::benchmark_data_smithi122_168655_object81008:head# 2024-03-20T09:26:26.106 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b0120ef:::benchmark_data_smithi122_168655_object115256:head# 2024-03-20T09:26:26.107 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b01ab71:::benchmark_data_smithi122_168655_object66247:head# 2024-03-20T09:26:26.107 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b02b388:::benchmark_data_smithi122_168655_object88140:head# 2024-03-20T09:26:26.108 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b02cfa1:::benchmark_data_smithi122_168655_object101136:head# 2024-03-20T09:26:26.108 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b02d9b3:::benchmark_data_smithi122_168655_object57927:head# 2024-03-20T09:26:26.109 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b03bab3:::benchmark_data_smithi122_168655_object31984:head# 2024-03-20T09:26:26.109 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b0465bc:::benchmark_data_smithi122_168655_object37145:head# 2024-03-20T09:26:26.109 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b04ade3:::benchmark_data_smithi122_168655_object79200:head# 2024-03-20T09:26:26.110 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b04db01:::benchmark_data_smithi122_168655_object60671:head# 2024-03-20T09:26:26.110 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b0514eb:::benchmark_data_smithi122_168655_object24554:head# 2024-03-20T09:26:26.111 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b0614d4:::benchmark_data_smithi122_168655_object70575:head# 2024-03-20T09:26:26.111 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b0773f8:::benchmark_data_smithi122_168655_object115948:head# 2024-03-20T09:26:26.112 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b07af7a:::benchmark_data_smithi122_168655_object48832:head# 2024-03-20T09:26:26.112 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b0843ad:::benchmark_data_smithi122_168655_object119531:head# 2024-03-20T09:26:26.113 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b0866f5:::benchmark_data_smithi122_168655_object14289:head# 2024-03-20T09:26:26.113 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b08ead3:::benchmark_data_smithi122_168655_object105155:head# 2024-03-20T09:26:26.113 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b0977e9:::benchmark_data_smithi122_168655_object119736:head# 2024-03-20T09:26:26.114 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b0b53d6:::benchmark_data_smithi122_168655_object35648:head# 2024-03-20T09:26:26.114 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b0c351a:::benchmark_data_smithi122_168655_object99856:head# 2024-03-20T09:26:26.115 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b0cb76e:::benchmark_data_smithi122_168655_object113951:head# 2024-03-20T09:26:26.115 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b0ceacc:::benchmark_data_smithi122_168655_object99272:head# 2024-03-20T09:26:26.116 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b0cf419:::benchmark_data_smithi122_168655_object102489:head# 2024-03-20T09:26:26.116 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b0d22d9:::benchmark_data_smithi122_168655_object116359:head# 2024-03-20T09:26:26.116 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b0e690d:::benchmark_data_smithi122_168655_object54759:head# 2024-03-20T09:26:26.117 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b1134b9:::benchmark_data_smithi122_168655_object93577:head# 2024-03-20T09:26:26.117 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b1196ad:::benchmark_data_smithi122_168655_object87827:head# 2024-03-20T09:26:26.118 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b12645f:::benchmark_data_smithi122_168655_object92817:head# 2024-03-20T09:26:26.118 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b12cd78:::benchmark_data_smithi122_168655_object34602:head# 2024-03-20T09:26:26.118 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b134f65:::benchmark_data_smithi122_168655_object97512:head# 2024-03-20T09:26:26.119 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b13549f:::benchmark_data_smithi122_168655_object59935:head# 2024-03-20T09:26:26.119 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b151349:::benchmark_data_smithi122_168655_object84819:head# 2024-03-20T09:26:26.119 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b1566ef:::benchmark_data_smithi122_168655_object65529:head# 2024-03-20T09:26:26.120 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b159a75:::benchmark_data_smithi122_168655_object49413:head# 2024-03-20T09:26:26.120 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b16c0c8:::benchmark_data_smithi122_168655_object103920:head# 2024-03-20T09:26:26.121 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b16fb49:::benchmark_data_smithi122_168655_object95208:head# 2024-03-20T09:26:26.121 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b1762ec:::benchmark_data_smithi122_168655_object40320:head# 2024-03-20T09:26:26.122 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b17a74b:::benchmark_data_smithi122_168655_object47426:head# 2024-03-20T09:26:26.122 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b17d4bd:::benchmark_data_smithi122_168655_object66891:head# 2024-03-20T09:26:26.122 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b18214d:::benchmark_data_smithi122_168655_object81506:head# 2024-03-20T09:26:26.123 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b18703f:::benchmark_data_smithi122_168655_object118453:head# 2024-03-20T09:26:26.123 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b18b08a:::benchmark_data_smithi122_168655_object82592:head# 2024-03-20T09:26:26.124 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b19939a:::benchmark_data_smithi122_168655_object56607:head# 2024-03-20T09:26:26.124 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b1a23f7:::benchmark_data_smithi122_168655_object115509:head# 2024-03-20T09:26:26.124 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b1acbfe:::benchmark_data_smithi122_168655_object100987:head# 2024-03-20T09:26:26.125 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b1b081e:::benchmark_data_smithi122_168655_object37196:head# 2024-03-20T09:26:26.125 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b1cd86e:::benchmark_data_smithi122_168655_object73477:head# 2024-03-20T09:26:26.126 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b1d0303:::benchmark_data_smithi122_168655_object115882:head# 2024-03-20T09:26:26.126 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b1d5678:::benchmark_data_smithi122_168655_object63800:head# 2024-03-20T09:26:26.128 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b1de169:::benchmark_data_smithi122_168655_object13093:head# 2024-03-20T09:26:26.129 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b1df4e0:::benchmark_data_smithi122_168655_object19586:head# 2024-03-20T09:26:26.129 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b1e35a1:::benchmark_data_smithi122_168655_object96321:head# 2024-03-20T09:26:26.130 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b1ff329:::benchmark_data_smithi122_168655_object64943:head# 2024-03-20T09:26:26.130 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b203139:::benchmark_data_smithi122_168655_object117143:head# 2024-03-20T09:26:26.131 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b20c9d5:::benchmark_data_smithi122_168655_object90901:head# 2024-03-20T09:26:26.131 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b21e460:::benchmark_data_smithi122_168655_object73938:head# 2024-03-20T09:26:26.132 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b21e81d:::benchmark_data_smithi122_168655_object46443:head# 2024-03-20T09:26:26.132 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b223f2c:::benchmark_data_smithi122_168655_object90532:head# 2024-03-20T09:26:26.133 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b240e31:::benchmark_data_smithi122_168655_object111954:head# 2024-03-20T09:26:26.133 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b24127e:::benchmark_data_smithi122_168655_object10256:head# 2024-03-20T09:26:26.134 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b246f69:::benchmark_data_smithi122_168655_object103128:head# 2024-03-20T09:26:26.134 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b254ea2:::benchmark_data_smithi122_168655_object106825:head# 2024-03-20T09:26:26.134 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b259cb5:::benchmark_data_smithi122_168655_object16566:head# 2024-03-20T09:26:26.135 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b25b9b5:::benchmark_data_smithi122_168655_object98634:head# 2024-03-20T09:26:26.135 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b272b06:::benchmark_data_smithi122_168655_object16077:head# 2024-03-20T09:26:26.136 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b28bb34:::benchmark_data_smithi122_168655_object66499:head# 2024-03-20T09:26:26.136 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b28ee6e:::benchmark_data_smithi122_168655_object53269:head# 2024-03-20T09:26:26.137 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b2a2c59:::benchmark_data_smithi122_168655_object52198:head# 2024-03-20T09:26:26.137 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b2b769b:::benchmark_data_smithi122_168655_object67130:head# 2024-03-20T09:26:26.137 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b2d29b8:::benchmark_data_smithi122_168655_object62718:head# 2024-03-20T09:26:26.138 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b2ee83f:::benchmark_data_smithi122_168655_object28523:head# 2024-03-20T09:26:26.138 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b2fc286:::benchmark_data_smithi122_168655_object15623:head# 2024-03-20T09:26:26.139 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b30507e:::benchmark_data_smithi122_168655_object51634:head# 2024-03-20T09:26:26.139 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b306be2:::benchmark_data_smithi122_168655_object31395:head# 2024-03-20T09:26:26.140 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b31211a:::benchmark_data_smithi122_168655_object104690:head# 2024-03-20T09:26:26.142 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b322e97:::benchmark_data_smithi122_168655_object18252:head# 2024-03-20T09:26:26.143 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b32658e:::benchmark_data_smithi122_168655_object32888:head# 2024-03-20T09:26:26.143 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b32ac82:::benchmark_data_smithi122_168655_object99612:head# 2024-03-20T09:26:26.144 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b32c069:::benchmark_data_smithi122_168655_object64628:head# 2024-03-20T09:26:26.144 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b32d5be:::benchmark_data_smithi122_168655_object57815:head# 2024-03-20T09:26:26.145 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b3332fd:::benchmark_data_smithi122_168655_object59330:head# 2024-03-20T09:26:26.145 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b341f4a:::benchmark_data_smithi122_168655_object43670:head# 2024-03-20T09:26:26.146 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b34b7f4:::benchmark_data_smithi122_168655_object112141:head# 2024-03-20T09:26:26.146 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b34c117:::benchmark_data_smithi122_168655_object68018:head# 2024-03-20T09:26:26.147 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b3500d3:::benchmark_data_smithi122_168655_object50436:head# 2024-03-20T09:26:26.147 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b35ac89:::benchmark_data_smithi122_168655_object83454:head# 2024-03-20T09:26:26.148 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b366345:::benchmark_data_smithi122_168655_object95460:head# 2024-03-20T09:26:26.148 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b37e7cd:::benchmark_data_smithi122_168655_object45084:head# 2024-03-20T09:26:26.149 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b38f71a:::benchmark_data_smithi122_168655_object97311:head# 2024-03-20T09:26:26.150 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b391733:::benchmark_data_smithi122_168655_object70066:head# 2024-03-20T09:26:26.150 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b395783:::benchmark_data_smithi122_168655_object41582:head# 2024-03-20T09:26:26.150 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b39c830:::benchmark_data_smithi122_168655_object66798:head# 2024-03-20T09:26:26.153 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b3a5851:::benchmark_data_smithi122_168655_object85942:head# 2024-03-20T09:26:26.155 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b3a5dd0:::benchmark_data_smithi122_168655_object51982:head# 2024-03-20T09:26:26.155 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b3aa4d6:::benchmark_data_smithi122_168655_object84240:head# 2024-03-20T09:26:26.155 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b3b806b:::benchmark_data_smithi122_168655_object98854:head# 2024-03-20T09:26:26.155 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b3bb45a:::benchmark_data_smithi122_168655_object96711:head# 2024-03-20T09:26:26.155 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b3c31ab:::benchmark_data_smithi122_168655_object46098:head# 2024-03-20T09:26:26.155 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b3c73bf:::benchmark_data_smithi122_168655_object87887:head# 2024-03-20T09:26:26.155 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b3cbc87:::benchmark_data_smithi122_168655_object113822:head# 2024-03-20T09:26:26.156 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b3e2399:::benchmark_data_smithi122_168655_object46984:head# 2024-03-20T09:26:26.156 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b3e3ec4:::benchmark_data_smithi122_168655_object40014:head# 2024-03-20T09:26:26.156 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b3e9f96:::benchmark_data_smithi122_168655_object101788:head# 2024-03-20T09:26:26.157 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b3f8664:::benchmark_data_smithi122_168655_object19645:head# 2024-03-20T09:26:26.157 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b3f9bbb:::benchmark_data_smithi122_168655_object13542:head# 2024-03-20T09:26:26.157 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b403155:::benchmark_data_smithi122_168655_object59280:head# 2024-03-20T09:26:26.158 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b411567:::benchmark_data_smithi122_168655_object59169:head# 2024-03-20T09:26:26.158 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b41b715:::benchmark_data_smithi122_168655_object47817:head# 2024-03-20T09:26:26.159 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b41fcee:::benchmark_data_smithi122_168655_object71828:head# 2024-03-20T09:26:26.159 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b42116c:::benchmark_data_smithi122_168655_object57985:head# 2024-03-20T09:26:26.160 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b4273b2:::benchmark_data_smithi122_168655_object82840:head# 2024-03-20T09:26:26.160 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b4326a3:::benchmark_data_smithi122_168655_object101501:head# 2024-03-20T09:26:26.161 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b43605a:::benchmark_data_smithi122_168655_object96573:head# 2024-03-20T09:26:26.161 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b43809d:::benchmark_data_smithi122_168655_object115300:head# 2024-03-20T09:26:26.161 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b4430ca:::benchmark_data_smithi122_168655_object57585:head# 2024-03-20T09:26:26.162 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b4463ec:::benchmark_data_smithi122_168655_object109889:head# 2024-03-20T09:26:26.162 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b4473d6:::benchmark_data_smithi122_168655_object54740:head# 2024-03-20T09:26:26.163 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b44c23e:::benchmark_data_smithi122_168655_object57160:head# 2024-03-20T09:26:26.163 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b45629d:::benchmark_data_smithi122_168655_object78856:head# 2024-03-20T09:26:26.164 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b465054:::benchmark_data_smithi122_168655_object65655:head# 2024-03-20T09:26:26.164 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b46664b:::benchmark_data_smithi122_168655_object24638:head# 2024-03-20T09:26:26.165 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b47b004:::benchmark_data_smithi122_168655_object54224:head# 2024-03-20T09:26:26.165 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b497114:::benchmark_data_smithi122_168655_object101724:head# 2024-03-20T09:26:26.165 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b49c2e4:::benchmark_data_smithi122_168655_object74848:head# 2024-03-20T09:26:26.166 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b4a083e:::benchmark_data_smithi122_168655_object36647:head# 2024-03-20T09:26:26.166 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b4a74e6:::benchmark_data_smithi122_168655_object40602:head# 2024-03-20T09:26:26.167 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b4b6606:::benchmark_data_smithi122_168655_object49208:head# 2024-03-20T09:26:26.167 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b4bc68e:::benchmark_data_smithi122_168655_object97887:head# 2024-03-20T09:26:26.168 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b4d62f6:::benchmark_data_smithi122_168655_object48290:head# 2024-03-20T09:26:26.168 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b4f956d:::benchmark_data_smithi122_168655_object81964:head# 2024-03-20T09:26:26.168 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b5142b4:::benchmark_data_smithi122_168655_object98019:head# 2024-03-20T09:26:26.169 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b5142cc:::benchmark_data_smithi122_168655_object46120:head# 2024-03-20T09:26:26.169 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b516daf:::benchmark_data_smithi122_168655_object62029:head# 2024-03-20T09:26:26.170 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b51c9e0:::benchmark_data_smithi122_168655_object60238:head# 2024-03-20T09:26:26.170 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b5210f9:::benchmark_data_smithi122_168655_object64624:head# 2024-03-20T09:26:26.171 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b567194:::benchmark_data_smithi122_168655_object46316:head# 2024-03-20T09:26:26.171 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b567d20:::benchmark_data_smithi122_168655_object65406:head# 2024-03-20T09:26:26.172 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b569269:::benchmark_data_smithi122_168655_object27034:head# 2024-03-20T09:26:26.172 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b576b33:::benchmark_data_smithi122_168655_object72424:head# 2024-03-20T09:26:26.173 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b587a12:::benchmark_data_smithi122_168655_object74317:head# 2024-03-20T09:26:26.173 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b589176:::benchmark_data_smithi122_168655_object26891:head# 2024-03-20T09:26:26.174 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b591ee6:::benchmark_data_smithi122_168655_object51912:head# 2024-03-20T09:26:26.174 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b598143:::benchmark_data_smithi122_168655_object25558:head# 2024-03-20T09:26:26.175 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b59d23e:::benchmark_data_smithi122_168655_object37089:head# 2024-03-20T09:26:26.175 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b59dbb1:::benchmark_data_smithi122_168655_object13222:head# 2024-03-20T09:26:26.175 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b5a1153:::benchmark_data_smithi122_168655_object14696:head# 2024-03-20T09:26:26.176 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b5a7c62:::benchmark_data_smithi122_168655_object104770:head# 2024-03-20T09:26:26.177 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b5b414b:::benchmark_data_smithi122_168655_object115804:head# 2024-03-20T09:26:26.178 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b5b4be6:::benchmark_data_smithi122_168655_object14772:head# 2024-03-20T09:26:26.178 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b5b54d0:::benchmark_data_smithi122_168655_object43202:head# 2024-03-20T09:26:26.178 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b5b5d4c:::benchmark_data_smithi122_168655_object50036:head# 2024-03-20T09:26:26.179 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b5c258f:::benchmark_data_smithi122_168655_object65682:head# 2024-03-20T09:26:26.179 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b5cf573:::benchmark_data_smithi122_168655_object45004:head# 2024-03-20T09:26:26.180 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b5d051a:::benchmark_data_smithi122_168655_object111836:head# 2024-03-20T09:26:26.180 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b5d33e4:::benchmark_data_smithi122_168655_object73424:head# 2024-03-20T09:26:26.180 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b5da02c:::benchmark_data_smithi122_168655_object100460:head# 2024-03-20T09:26:26.181 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b5e9bfb:::benchmark_data_smithi122_168655_object50688:head# 2024-03-20T09:26:26.181 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b5edce5:::benchmark_data_smithi122_168655_object72829:head# 2024-03-20T09:26:26.182 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b5eea46:::benchmark_data_smithi122_168655_object111921:head# 2024-03-20T09:26:26.182 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b5f17c2:::benchmark_data_smithi122_168655_object41543:head# 2024-03-20T09:26:26.183 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b5f80dd:::benchmark_data_smithi122_168655_object107342:head# 2024-03-20T09:26:26.183 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b5faeba:::benchmark_data_smithi122_168655_object82711:head# 2024-03-20T09:26:26.183 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b60a133:::benchmark_data_smithi122_168655_object105953:head# 2024-03-20T09:26:26.184 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b61dc60:::benchmark_data_smithi122_168655_object42972:head# 2024-03-20T09:26:26.184 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b622aa8:::benchmark_data_smithi122_168655_object101834:head# 2024-03-20T09:26:26.185 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b62413d:::benchmark_data_smithi122_168655_object46282:head# 2024-03-20T09:26:26.185 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b627875:::benchmark_data_smithi122_168655_object26497:head# 2024-03-20T09:26:26.186 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b631d44:::benchmark_data_smithi122_168655_object117413:head# 2024-03-20T09:26:26.186 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b648f03:::benchmark_data_smithi122_168655_object113707:head# 2024-03-20T09:26:26.187 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b66c158:::benchmark_data_smithi122_168655_object19827:head# 2024-03-20T09:26:26.187 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b66cb32:::benchmark_data_smithi122_168655_object56533:head# 2024-03-20T09:26:26.188 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b671978:::benchmark_data_smithi122_168655_object10691:head# 2024-03-20T09:26:26.188 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b675926:::benchmark_data_smithi122_168655_object92217:head# 2024-03-20T09:26:26.189 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b67cada:::benchmark_data_smithi122_168655_object14610:head# 2024-03-20T09:26:26.189 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b686d01:::benchmark_data_smithi122_168655_object85533:head# 2024-03-20T09:26:26.189 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b68c7ac:::benchmark_data_smithi122_168655_object58451:head# 2024-03-20T09:26:26.190 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b693259:::benchmark_data_smithi122_168655_object78642:head# 2024-03-20T09:26:26.191 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b697d82:::benchmark_data_smithi122_168655_object49519:head# 2024-03-20T09:26:26.191 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b6a2fbc:::benchmark_data_smithi122_168655_object18661:head# 2024-03-20T09:26:26.191 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b6a5ede:::benchmark_data_smithi122_168655_object102239:head# 2024-03-20T09:26:26.192 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b6d301f:::benchmark_data_smithi122_168655_object92715:head# 2024-03-20T09:26:26.192 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b6d5ced:::benchmark_data_smithi122_168655_object85005:head# 2024-03-20T09:26:26.193 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b6d7c0d:::benchmark_data_smithi122_168655_object119202:head# 2024-03-20T09:26:26.193 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b6da525:::benchmark_data_smithi122_168655_object56331:head# 2024-03-20T09:26:26.193 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b6e179f:::benchmark_data_smithi122_168655_object27381:head# 2024-03-20T09:26:26.194 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b6e874d:::benchmark_data_smithi122_168655_object30997:head# 2024-03-20T09:26:26.194 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b6ea6c4:::benchmark_data_smithi122_168655_object109658:head# 2024-03-20T09:26:26.195 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b6efad3:::benchmark_data_smithi122_168655_object90205:head# 2024-03-20T09:26:26.195 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b6f59bd:::benchmark_data_smithi122_168655_object56603:head# 2024-03-20T09:26:26.196 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b6f5cf1:::benchmark_data_smithi122_168655_object19359:head# 2024-03-20T09:26:26.196 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b70162b:::benchmark_data_smithi122_168655_object14983:head# 2024-03-20T09:26:26.197 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b705d33:::benchmark_data_smithi122_168655_object22128:head# 2024-03-20T09:26:26.197 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b7296c5:::benchmark_data_smithi122_168655_object114825:head# 2024-03-20T09:26:26.198 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b729d68:::benchmark_data_smithi122_168655_object38633:head# 2024-03-20T09:26:26.198 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b72e0dd:::benchmark_data_smithi122_168655_object42254:head# 2024-03-20T09:26:26.199 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b72e3c6:::benchmark_data_smithi122_168655_object34733:head# 2024-03-20T09:26:26.199 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b730ace:::benchmark_data_smithi122_168655_object91637:head# 2024-03-20T09:26:26.199 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b73d0d7:::benchmark_data_smithi122_168655_object84624:head# 2024-03-20T09:26:26.200 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b747cf6:::benchmark_data_smithi122_168655_object118736:head# 2024-03-20T09:26:26.200 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b7481f0:::benchmark_data_smithi122_168655_object42805:head# 2024-03-20T09:26:26.200 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b75a513:::benchmark_data_smithi122_168655_object78869:head# 2024-03-20T09:26:26.201 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b75f86c:::benchmark_data_smithi122_168655_object81856:head# 2024-03-20T09:26:26.201 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b769b6f:::benchmark_data_smithi122_168655_object107791:head# 2024-03-20T09:26:26.202 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b7703e4:::benchmark_data_smithi122_168655_object53085:head# 2024-03-20T09:26:26.202 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b77b0d4:::benchmark_data_smithi122_168655_object18460:head# 2024-03-20T09:26:26.203 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b77dee5:::benchmark_data_smithi122_168655_object65348:head# 2024-03-20T09:26:26.203 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b783dec:::benchmark_data_smithi122_168655_object76121:head# 2024-03-20T09:26:26.204 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b7914ca:::benchmark_data_smithi122_168655_object82288:head# 2024-03-20T09:26:26.204 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b793d78:::benchmark_data_smithi122_168655_object111002:head# 2024-03-20T09:26:26.205 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b7a2cc6:::benchmark_data_smithi122_168655_object77173:head# 2024-03-20T09:26:26.205 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b7a400e:::benchmark_data_smithi122_168655_object61093:head# 2024-03-20T09:26:26.206 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b7b465a:::benchmark_data_smithi122_168655_object21863:head# 2024-03-20T09:26:26.206 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b7b72b5:::benchmark_data_smithi122_168655_object12976:head# 2024-03-20T09:26:26.206 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b7b7ba9:::benchmark_data_smithi122_168655_object98327:head# 2024-03-20T09:26:26.207 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b7b944a:::benchmark_data_smithi122_168655_object23477:head# 2024-03-20T09:26:26.207 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b7cddb5:::benchmark_data_smithi122_168655_object116594:head# 2024-03-20T09:26:26.208 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b7d3405:::benchmark_data_smithi122_168655_object48494:head# 2024-03-20T09:26:26.209 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b7de3a9:::benchmark_data_smithi122_168655_object25602:head# 2024-03-20T09:26:26.209 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b7dfa5c:::benchmark_data_smithi122_168655_object112925:head# 2024-03-20T09:26:26.209 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b7e208b:::benchmark_data_smithi122_168655_object119267:head# 2024-03-20T09:26:26.210 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b7e4489:::benchmark_data_smithi122_168655_object32769:head# 2024-03-20T09:26:26.210 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b7e5581:::benchmark_data_smithi122_168655_object101194:head# 2024-03-20T09:26:26.211 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b7f18d8:::benchmark_data_smithi122_168655_object39622:head# 2024-03-20T09:26:26.211 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b7ff37d:::benchmark_data_smithi122_168655_object91771:head# 2024-03-20T09:26:26.212 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b803c95:::benchmark_data_smithi122_168655_object35505:head# 2024-03-20T09:26:26.212 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b8086c2:::benchmark_data_smithi122_168655_object71483:head# 2024-03-20T09:26:26.212 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b809699:::benchmark_data_smithi122_168655_object96289:head# 2024-03-20T09:26:26.213 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b812613:::benchmark_data_smithi122_168655_object96941:head# 2024-03-20T09:26:26.213 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b813617:::benchmark_data_smithi122_168655_object37331:head# 2024-03-20T09:26:26.214 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b8200c5:::benchmark_data_smithi122_168655_object11105:head# 2024-03-20T09:26:26.214 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b82fe61:::benchmark_data_smithi122_168655_object94958:head# 2024-03-20T09:26:26.214 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b835238:::benchmark_data_smithi122_168655_object44319:head# 2024-03-20T09:26:26.215 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b8429e7:::benchmark_data_smithi122_168655_object35899:head# 2024-03-20T09:26:26.215 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b84c24e:::benchmark_data_smithi122_168655_object112044:head# 2024-03-20T09:26:26.215 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b857c1a:::benchmark_data_smithi122_168655_object101841:head# 2024-03-20T09:26:26.216 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b864ef5:::benchmark_data_smithi122_168655_object108390:head# 2024-03-20T09:26:26.216 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b871083:::benchmark_data_smithi122_168655_object77025:head# 2024-03-20T09:26:26.217 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b875561:::benchmark_data_smithi122_168655_object113333:head# 2024-03-20T09:26:26.217 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b87cb58:::benchmark_data_smithi122_168655_object83992:head# 2024-03-20T09:26:26.218 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b886717:::benchmark_data_smithi122_168655_object54627:head# 2024-03-20T09:26:26.218 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b895b98:::benchmark_data_smithi122_168655_object86002:head# 2024-03-20T09:26:26.219 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b896d6a:::benchmark_data_smithi122_168655_object33667:head# 2024-03-20T09:26:26.219 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b8a13d3:::benchmark_data_smithi122_168655_object89759:head# 2024-03-20T09:26:26.219 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b8a6833:::benchmark_data_smithi122_168655_object57277:head# 2024-03-20T09:26:26.220 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b8a7751:::benchmark_data_smithi122_168655_object45382:head# 2024-03-20T09:26:26.220 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b8ab429:::benchmark_data_smithi122_168655_object94880:head# 2024-03-20T09:26:26.221 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b8ae4cc:::benchmark_data_smithi122_168655_object37550:head# 2024-03-20T09:26:26.221 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b8b3a65:::benchmark_data_smithi122_168655_object31511:head# 2024-03-20T09:26:26.222 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b8cd04d:::benchmark_data_smithi122_168655_object90428:head# 2024-03-20T09:26:26.222 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b8de3be:::benchmark_data_smithi122_168655_object88885:head# 2024-03-20T09:26:26.223 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b8debf4:::benchmark_data_smithi122_168655_object71760:head# 2024-03-20T09:26:26.223 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b8dfbc8:::benchmark_data_smithi122_168655_object81574:head# 2024-03-20T09:26:26.223 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b8e6349:::benchmark_data_smithi122_168655_object92845:head# 2024-03-20T09:26:26.224 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b8fd667:::benchmark_data_smithi122_168655_object81963:head# 2024-03-20T09:26:26.224 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b9009f8:::benchmark_data_smithi122_168655_object24472:head# 2024-03-20T09:26:26.225 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b9014d6:::benchmark_data_smithi122_168655_object20479:head# 2024-03-20T09:26:26.225 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b90c647:::benchmark_data_smithi122_168655_object13577:head# 2024-03-20T09:26:26.226 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b90ff94:::benchmark_data_smithi122_168655_object101688:head# 2024-03-20T09:26:26.226 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b93e9fb:::benchmark_data_smithi122_168655_object60438:head# 2024-03-20T09:26:26.226 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b942276:::benchmark_data_smithi122_168655_object35819:head# 2024-03-20T09:26:26.227 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b9426ff:::benchmark_data_smithi122_168655_object28229:head# 2024-03-20T09:26:26.227 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b94c8b9:::benchmark_data_smithi122_168655_object64226:head# 2024-03-20T09:26:26.228 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b94e790:::benchmark_data_smithi122_168655_object64683:head# 2024-03-20T09:26:26.228 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b94ec25:::benchmark_data_smithi122_168655_object32577:head# 2024-03-20T09:26:26.228 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b9503d7:::benchmark_data_smithi122_168655_object10557:head# 2024-03-20T09:26:26.229 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b97fe28:::benchmark_data_smithi122_168655_object117529:head# 2024-03-20T09:26:26.229 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b980ed1:::benchmark_data_smithi122_168655_object79759:head# 2024-03-20T09:26:26.230 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b982ecc:::benchmark_data_smithi122_168655_object21382:head# 2024-03-20T09:26:26.230 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b985733:::benchmark_data_smithi122_168655_object118280:head# 2024-03-20T09:26:26.231 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b985b93:::benchmark_data_smithi122_168655_object76500:head# 2024-03-20T09:26:26.231 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b9a06ad:::benchmark_data_smithi122_168655_object99756:head# 2024-03-20T09:26:26.232 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b9b43ca:::benchmark_data_smithi122_168655_object89260:head# 2024-03-20T09:26:26.232 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b9c5d38:::benchmark_data_smithi122_168655_object115557:head# 2024-03-20T09:26:26.233 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b9c613f:::benchmark_data_smithi122_168655_object60341:head# 2024-03-20T09:26:26.233 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b9c69b4:::benchmark_data_smithi122_168655_object63931:head# 2024-03-20T09:26:26.234 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b9cb16e:::benchmark_data_smithi122_168655_object106401:head# 2024-03-20T09:26:26.234 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b9f3dcd:::benchmark_data_smithi122_168655_object54748:head# 2024-03-20T09:26:26.234 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b9f7fb9:::benchmark_data_smithi122_168655_object102804:head# 2024-03-20T09:26:26.235 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8b9fb8b1:::benchmark_data_smithi122_168655_object33717:head# 2024-03-20T09:26:26.235 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ba109b9:::benchmark_data_smithi122_168655_object66039:head# 2024-03-20T09:26:26.236 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ba218bb:::benchmark_data_smithi122_168655_object45596:head# 2024-03-20T09:26:26.236 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ba297bc:::benchmark_data_smithi122_168655_object11056:head# 2024-03-20T09:26:26.237 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ba36dd0:::benchmark_data_smithi122_168655_object79189:head# 2024-03-20T09:26:26.237 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ba3fc29:::benchmark_data_smithi122_168655_object45805:head# 2024-03-20T09:26:26.237 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ba41de1:::benchmark_data_smithi122_168655_object43106:head# 2024-03-20T09:26:26.238 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ba45883:::benchmark_data_smithi122_168655_object84843:head# 2024-03-20T09:26:26.238 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ba50a0f:::benchmark_data_smithi122_168655_object82141:head# 2024-03-20T09:26:26.239 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ba56311:::benchmark_data_smithi122_168655_object116219:head# 2024-03-20T09:26:26.239 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ba5a26d:::benchmark_data_smithi122_168655_object112172:head# 2024-03-20T09:26:26.240 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ba6e03e:::benchmark_data_smithi122_168655_object111010:head# 2024-03-20T09:26:26.240 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ba7839f:::benchmark_data_smithi122_168655_object16805:head# 2024-03-20T09:26:26.240 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ba844d7:::benchmark_data_smithi122_168655_object109045:head# 2024-03-20T09:26:26.241 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ba870ba:::benchmark_data_smithi122_168655_object52199:head# 2024-03-20T09:26:26.241 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ba8ce14:::benchmark_data_smithi122_168655_object87522:head# 2024-03-20T09:26:26.242 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8baa292e:::benchmark_data_smithi122_168655_object89360:head# 2024-03-20T09:26:26.242 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8baacebb:::benchmark_data_smithi122_168655_object54120:head# 2024-03-20T09:26:26.243 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8babc2a1:::benchmark_data_smithi122_168655_object10318:head# 2024-03-20T09:26:26.243 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8babc908:::benchmark_data_smithi122_168655_object119863:head# 2024-03-20T09:26:26.244 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8babcc99:::benchmark_data_smithi122_168655_object108199:head# 2024-03-20T09:26:26.244 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bac5c97:::benchmark_data_smithi122_168655_object107555:head# 2024-03-20T09:26:26.245 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bada75d:::benchmark_data_smithi122_168655_object115683:head# 2024-03-20T09:26:26.245 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bae0b80:::benchmark_data_smithi122_168655_object43803:head# 2024-03-20T09:26:26.246 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bafa2b4:::benchmark_data_smithi122_168655_object115946:head# 2024-03-20T09:26:26.246 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bafbb5a:::benchmark_data_smithi122_168655_object116723:head# 2024-03-20T09:26:26.246 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bb09e30:::benchmark_data_smithi122_168655_object54520:head# 2024-03-20T09:26:26.247 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bb1218b:::benchmark_data_smithi122_168655_object60659:head# 2024-03-20T09:26:26.248 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bb131ec:::benchmark_data_smithi122_168655_object86426:head# 2024-03-20T09:26:26.248 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bb19b26:::benchmark_data_smithi122_168655_object108220:head# 2024-03-20T09:26:26.248 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bb3149d:::benchmark_data_smithi122_168655_object105811:head# 2024-03-20T09:26:26.249 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bb3cc27:::benchmark_data_smithi122_168655_object48054:head# 2024-03-20T09:26:26.249 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bb4fb40:::benchmark_data_smithi122_168655_object23503:head# 2024-03-20T09:26:26.251 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bb5259e:::benchmark_data_smithi122_168655_object60140:head# 2024-03-20T09:26:26.251 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bb760cc:::benchmark_data_smithi122_168655_object77444:head# 2024-03-20T09:26:26.251 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bb80e35:::benchmark_data_smithi122_168655_object61170:head# 2024-03-20T09:26:26.251 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bb85c82:::benchmark_data_smithi122_168655_object32513:head# 2024-03-20T09:26:26.252 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bb875f7:::benchmark_data_smithi122_168655_object77609:head# 2024-03-20T09:26:26.253 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bb9545c:::benchmark_data_smithi122_168655_object83310:head# 2024-03-20T09:26:26.253 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bbac5b3:::benchmark_data_smithi122_168655_object79912:head# 2024-03-20T09:26:26.253 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bbacf4d:::benchmark_data_smithi122_168655_object98660:head# 2024-03-20T09:26:26.254 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bbb6fb5:::benchmark_data_smithi122_168655_object87644:head# 2024-03-20T09:26:26.254 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bbbdb48:::benchmark_data_smithi122_168655_object35400:head# 2024-03-20T09:26:26.254 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bbc18af:::benchmark_data_smithi122_168655_object42823:head# 2024-03-20T09:26:26.255 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bbc5bc9:::benchmark_data_smithi122_168655_object13445:head# 2024-03-20T09:26:26.255 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bbc5f4b:::benchmark_data_smithi122_168655_object47665:head# 2024-03-20T09:26:26.256 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bbc74e2:::benchmark_data_smithi122_168655_object84687:head# 2024-03-20T09:26:26.256 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bbd36d6:::benchmark_data_smithi122_168655_object90179:head# 2024-03-20T09:26:26.257 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bbd4738:::benchmark_data_smithi122_168655_object82074:head# 2024-03-20T09:26:26.257 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bbdaa23:::benchmark_data_smithi122_168655_object31858:head# 2024-03-20T09:26:26.257 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bbe0823:::benchmark_data_smithi122_168655_object11680:head# 2024-03-20T09:26:26.258 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bbe2877:::benchmark_data_smithi122_168655_object73238:head# 2024-03-20T09:26:26.258 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bbe68ba:::benchmark_data_smithi122_168655_object98611:head# 2024-03-20T09:26:26.259 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bbe7dc3:::benchmark_data_smithi122_168655_object74752:head# 2024-03-20T09:26:26.259 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bbe8013:::benchmark_data_smithi122_168655_object25913:head# 2024-03-20T09:26:26.260 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bbf85fe:::benchmark_data_smithi122_168655_object95084:head# 2024-03-20T09:26:26.260 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bc028ae:::benchmark_data_smithi122_168655_object72333:head# 2024-03-20T09:26:26.260 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bc03430:::benchmark_data_smithi122_168655_object116804:head# 2024-03-20T09:26:26.261 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bc06957:::benchmark_data_smithi122_168655_object26089:head# 2024-03-20T09:26:26.261 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bc2d068:::benchmark_data_smithi122_168655_object69828:head# 2024-03-20T09:26:26.262 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bc381ee:::benchmark_data_smithi122_168655_object96187:head# 2024-03-20T09:26:26.262 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bc3a1ce:::benchmark_data_smithi122_168655_object49477:head# 2024-03-20T09:26:26.262 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bc3aedc:::benchmark_data_smithi122_168655_object35733:head# 2024-03-20T09:26:26.263 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bc405ec:::benchmark_data_smithi122_168655_object14949:head# 2024-03-20T09:26:26.263 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bc4f732:::benchmark_data_smithi122_168655_object73530:head# 2024-03-20T09:26:26.264 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bc53a96:::benchmark_data_smithi122_168655_object47057:head# 2024-03-20T09:26:26.264 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bc5fffa:::benchmark_data_smithi122_168655_object107792:head# 2024-03-20T09:26:26.264 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bc605a1:::benchmark_data_smithi122_168655_object109377:head# 2024-03-20T09:26:26.265 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bc7ba2d:::benchmark_data_smithi122_168655_object98712:head# 2024-03-20T09:26:26.265 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bc898bf:::benchmark_data_smithi122_168655_object79452:head# 2024-03-20T09:26:26.266 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bc8f701:::benchmark_data_smithi122_168655_object99767:head# 2024-03-20T09:26:26.266 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bc9f7d2:::benchmark_data_smithi122_168655_object65191:head# 2024-03-20T09:26:26.267 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bcad6ee:::benchmark_data_smithi122_168655_object103474:head# 2024-03-20T09:26:26.267 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bcc26e9:::benchmark_data_smithi122_168655_object16178:head# 2024-03-20T09:26:26.268 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bcc5f8b:::benchmark_data_smithi122_168655_object28824:head# 2024-03-20T09:26:26.268 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bcc9995:::benchmark_data_smithi122_168655_object56971:head# 2024-03-20T09:26:26.268 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bce0a60:::benchmark_data_smithi122_168655_object99928:head# 2024-03-20T09:26:26.269 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bcfc52f:::benchmark_data_smithi122_168655_object116701:head# 2024-03-20T09:26:26.269 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bd096a2:::benchmark_data_smithi122_168655_object14324:head# 2024-03-20T09:26:26.270 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bd16bb8:::benchmark_data_smithi122_168655_object20394:head# 2024-03-20T09:26:26.271 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bd24f09:::benchmark_data_smithi122_168655_object43708:head# 2024-03-20T09:26:26.271 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bd2af63:::benchmark_data_smithi122_168655_object117631:head# 2024-03-20T09:26:26.271 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bd2f3c5:::benchmark_data_smithi122_168655_object107519:head# 2024-03-20T09:26:26.272 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bd349fa:::benchmark_data_smithi122_168655_object88504:head# 2024-03-20T09:26:26.272 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bd4195b:::benchmark_data_smithi122_168655_object101102:head# 2024-03-20T09:26:26.273 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bd4d256:::benchmark_data_smithi122_168655_object32667:head# 2024-03-20T09:26:26.273 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bd5a3cf:::benchmark_data_smithi122_168655_object108877:head# 2024-03-20T09:26:26.273 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bd675d8:::benchmark_data_smithi122_168655_object71604:head# 2024-03-20T09:26:26.274 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bd725d0:::benchmark_data_smithi122_168655_object92823:head# 2024-03-20T09:26:26.274 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bd7eb80:::benchmark_data_smithi122_168655_object21991:head# 2024-03-20T09:26:26.275 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bd847b4:::benchmark_data_smithi122_168655_object73408:head# 2024-03-20T09:26:26.275 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bd88c10:::benchmark_data_smithi122_168655_object21644:head# 2024-03-20T09:26:26.276 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bd895b8:::benchmark_data_smithi122_168655_object81709:head# 2024-03-20T09:26:26.276 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bd9026e:::benchmark_data_smithi122_168655_object30297:head# 2024-03-20T09:26:26.277 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bd92eb7:::benchmark_data_smithi122_168655_object100528:head# 2024-03-20T09:26:26.277 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bd9cbad:::benchmark_data_smithi122_168655_object91535:head# 2024-03-20T09:26:26.277 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bda7688:::benchmark_data_smithi122_168655_object91022:head# 2024-03-20T09:26:26.278 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bdb929a:::benchmark_data_smithi122_168655_object14968:head# 2024-03-20T09:26:26.278 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bdb9b7d:::benchmark_data_smithi122_168655_object22987:head# 2024-03-20T09:26:26.279 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bdc6048:::benchmark_data_smithi122_168655_object48216:head# 2024-03-20T09:26:26.279 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bdc83e9:::benchmark_data_smithi122_168655_object104282:head# 2024-03-20T09:26:26.280 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bdce4f1:::benchmark_data_smithi122_168655_object74867:head# 2024-03-20T09:26:26.280 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bdd8efb:::benchmark_data_smithi122_168655_object64403:head# 2024-03-20T09:26:26.281 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bde0854:::benchmark_data_smithi122_168655_object38636:head# 2024-03-20T09:26:26.281 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bdee487:::benchmark_data_smithi122_168655_object48508:head# 2024-03-20T09:26:26.281 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bdf84d1:::benchmark_data_smithi122_168655_object115001:head# 2024-03-20T09:26:26.282 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8be02296:::benchmark_data_smithi122_168655_object69870:head# 2024-03-20T09:26:26.282 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8be10bbf:::benchmark_data_smithi122_168655_object31700:head# 2024-03-20T09:26:26.283 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8be1c1dc:::benchmark_data_smithi122_168655_object93393:head# 2024-03-20T09:26:26.283 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8be200f1:::benchmark_data_smithi122_168655_object73862:head# 2024-03-20T09:26:26.284 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8be301df:::benchmark_data_smithi122_168655_object36044:head# 2024-03-20T09:26:26.284 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8be3a990:::benchmark_data_smithi122_168655_object116822:head# 2024-03-20T09:26:26.285 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8be42230:::benchmark_data_smithi122_168655_object110776:head# 2024-03-20T09:26:26.285 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8be4fd23:::benchmark_data_smithi122_168655_object115259:head# 2024-03-20T09:26:26.286 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8be578a5:::benchmark_data_smithi122_168655_object79802:head# 2024-03-20T09:26:26.286 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8be61712:::benchmark_data_smithi122_168655_object56506:head# 2024-03-20T09:26:26.287 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8be643b1:::benchmark_data_smithi122_168655_object17533:head# 2024-03-20T09:26:26.287 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8be64dc0:::benchmark_data_smithi122_168655_object100501:head# 2024-03-20T09:26:26.287 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8be816e8:::benchmark_data_smithi122_168655_object105104:head# 2024-03-20T09:26:26.288 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8be8d0e9:::benchmark_data_smithi122_168655_object42342:head# 2024-03-20T09:26:26.288 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8be9c306:::benchmark_data_smithi122_168655_object43164:head# 2024-03-20T09:26:26.289 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bea6c35:::benchmark_data_smithi122_168655_object34864:head# 2024-03-20T09:26:26.289 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8beaec41:::benchmark_data_smithi122_168655_object18641:head# 2024-03-20T09:26:26.290 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8beb2add:::benchmark_data_smithi122_168655_object22514:head# 2024-03-20T09:26:26.290 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bed0a18:::benchmark_data_smithi122_168655_object14667:head# 2024-03-20T09:26:26.290 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bed4a3f:::benchmark_data_smithi122_168655_object11385:head# 2024-03-20T09:26:26.291 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bedb7de:::benchmark_data_smithi122_168655_object37448:head# 2024-03-20T09:26:26.291 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bee8f09:::benchmark_data_smithi122_168655_object65337:head# 2024-03-20T09:26:26.292 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bef1b07:::benchmark_data_smithi122_168655_object39460:head# 2024-03-20T09:26:26.292 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bef27b9:::benchmark_data_smithi122_168655_object40344:head# 2024-03-20T09:26:26.293 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bf1034b:::benchmark_data_smithi122_168655_object27960:head# 2024-03-20T09:26:26.293 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bf146b2:::benchmark_data_smithi122_168655_object31471:head# 2024-03-20T09:26:26.294 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bf1488f:::benchmark_data_smithi122_168655_object97973:head# 2024-03-20T09:26:26.294 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bf15aea:::benchmark_data_smithi122_168655_object36090:head# 2024-03-20T09:26:26.294 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bf1e45b:::benchmark_data_smithi122_168655_object112896:head# 2024-03-20T09:26:26.295 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bf27f02:::benchmark_data_smithi122_168655_object19015:head# 2024-03-20T09:26:26.295 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bf2b085:::benchmark_data_smithi122_168655_object103009:head# 2024-03-20T09:26:26.295 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bf32ed1:::benchmark_data_smithi122_168655_object19265:head# 2024-03-20T09:26:26.296 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bf39ac0:::benchmark_data_smithi122_168655_object29019:head# 2024-03-20T09:26:26.296 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bf43726:::benchmark_data_smithi122_168655_object104806:head# 2024-03-20T09:26:26.297 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bf596aa:::benchmark_data_smithi122_168655_object30002:head# 2024-03-20T09:26:26.297 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bf5c378:::benchmark_data_smithi122_168655_object87451:head# 2024-03-20T09:26:26.298 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bf6b9a3:::benchmark_data_smithi122_168655_object68886:head# 2024-03-20T09:26:26.298 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bf80589:::benchmark_data_smithi122_168655_object62392:head# 2024-03-20T09:26:26.298 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bf92ea6:::benchmark_data_smithi122_168655_object114826:head# 2024-03-20T09:26:26.299 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bf9336a:::benchmark_data_smithi122_168655_object118287:head# 2024-03-20T09:26:26.299 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bf945d9:::benchmark_data_smithi122_168655_object51005:head# 2024-03-20T09:26:26.300 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bfa13c7:::benchmark_data_smithi122_168655_object12834:head# 2024-03-20T09:26:26.301 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bfa4ba1:::benchmark_data_smithi122_168655_object119538:head# 2024-03-20T09:26:26.301 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bfb35d9:::benchmark_data_smithi122_168655_object102576:head# 2024-03-20T09:26:26.302 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bfc781f:::benchmark_data_smithi122_168655_object28594:head# 2024-03-20T09:26:26.302 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bfcc4f0:::benchmark_data_smithi122_168655_object48029:head# 2024-03-20T09:26:26.302 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bfd6638:::benchmark_data_smithi122_168655_object92255:head# 2024-03-20T09:26:26.303 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bfe8d23:::benchmark_data_smithi122_168655_object64557:head# 2024-03-20T09:26:26.303 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bfec4ae:::benchmark_data_smithi122_168655_object82360:head# 2024-03-20T09:26:26.304 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8bff76cf:::benchmark_data_smithi122_168655_object33423:head# 2024-03-20T09:26:26.305 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c00da0b:::benchmark_data_smithi122_168655_object58348:head# 2024-03-20T09:26:26.305 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c00f804:::benchmark_data_smithi122_168655_object106364:head# 2024-03-20T09:26:26.305 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c018cb4:::benchmark_data_smithi122_168655_object92210:head# 2024-03-20T09:26:26.306 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c032046:::benchmark_data_smithi122_168655_object38385:head# 2024-03-20T09:26:26.306 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c035801:::benchmark_data_smithi122_168655_object90090:head# 2024-03-20T09:26:26.307 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c0477da:::benchmark_data_smithi122_168655_object95195:head# 2024-03-20T09:26:26.307 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c04f66c:::benchmark_data_smithi122_168655_object117745:head# 2024-03-20T09:26:26.307 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c0522cd:::benchmark_data_smithi122_168655_object114897:head# 2024-03-20T09:26:26.308 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c060a9d:::benchmark_data_smithi122_168655_object26202:head# 2024-03-20T09:26:26.308 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c074460:::benchmark_data_smithi122_168655_object95503:head# 2024-03-20T09:26:26.309 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c0777dd:::benchmark_data_smithi122_168655_object69069:head# 2024-03-20T09:26:26.309 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c079c07:::benchmark_data_smithi122_168655_object93398:head# 2024-03-20T09:26:26.309 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c09d1a8:::benchmark_data_smithi122_168655_object114529:head# 2024-03-20T09:26:26.310 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c0a5001:::benchmark_data_smithi122_168655_object116524:head# 2024-03-20T09:26:26.310 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c0b9db9:::benchmark_data_smithi122_168655_object92953:head# 2024-03-20T09:26:26.311 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c0bd41f:::benchmark_data_smithi122_168655_object117881:head# 2024-03-20T09:26:26.311 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c0be8b2:::benchmark_data_smithi122_168655_object116781:head# 2024-03-20T09:26:26.312 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c0d36b0:::benchmark_data_smithi122_168655_object61002:head# 2024-03-20T09:26:26.312 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c0d9702:::benchmark_data_smithi122_168655_object39940:head# 2024-03-20T09:26:26.313 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c0dd536:::benchmark_data_smithi122_168655_object46019:head# 2024-03-20T09:26:26.313 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c0df1f2:::benchmark_data_smithi122_168655_object32823:head# 2024-03-20T09:26:26.314 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c101af0:::benchmark_data_smithi122_168655_object51134:head# 2024-03-20T09:26:26.314 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c106a23:::benchmark_data_smithi122_168655_object112386:head# 2024-03-20T09:26:26.315 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c1088c8:::benchmark_data_smithi122_168655_object71203:head# 2024-03-20T09:26:26.315 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c108cc8:::benchmark_data_smithi122_168655_object15907:head# 2024-03-20T09:26:26.316 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c11d0fa:::benchmark_data_smithi122_168655_object92238:head# 2024-03-20T09:26:26.316 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c11d61c:::benchmark_data_smithi122_168655_object30482:head# 2024-03-20T09:26:26.316 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c125aca:::benchmark_data_smithi122_168655_object118292:head# 2024-03-20T09:26:26.317 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c128765:::benchmark_data_smithi122_168655_object15633:head# 2024-03-20T09:26:26.317 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c142e2f:::benchmark_data_smithi122_168655_object82967:head# 2024-03-20T09:26:26.318 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c168f3c:::benchmark_data_smithi122_168655_object116958:head# 2024-03-20T09:26:26.319 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c172a88:::benchmark_data_smithi122_168655_object56660:head# 2024-03-20T09:26:26.320 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c1761f5:::benchmark_data_smithi122_168655_object54053:head# 2024-03-20T09:26:26.320 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c17a827:::benchmark_data_smithi122_168655_object27329:head# 2024-03-20T09:26:26.320 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c17d03f:::benchmark_data_smithi122_168655_object66143:head# 2024-03-20T09:26:26.321 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c18858c:::benchmark_data_smithi122_168655_object46238:head# 2024-03-20T09:26:26.321 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c197507:::benchmark_data_smithi122_168655_object99645:head# 2024-03-20T09:26:26.322 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c1999a8:::benchmark_data_smithi122_168655_object23076:head# 2024-03-20T09:26:26.322 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c1a13e9:::benchmark_data_smithi122_168655_object25643:head# 2024-03-20T09:26:26.323 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c1a1d9a:::benchmark_data_smithi122_168655_object82670:head# 2024-03-20T09:26:26.323 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c1b0880:::benchmark_data_smithi122_168655_object15806:head# 2024-03-20T09:26:26.323 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c1b4f17:::benchmark_data_smithi122_168655_object109835:head# 2024-03-20T09:26:26.324 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c1bf649:::benchmark_data_smithi122_168655_object12541:head# 2024-03-20T09:26:26.324 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c1c0713:::benchmark_data_smithi122_168655_object13196:head# 2024-03-20T09:26:26.325 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c1df08b:::benchmark_data_smithi122_168655_object76745:head# 2024-03-20T09:26:26.325 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c1e402c:::benchmark_data_smithi122_168655_object62884:head# 2024-03-20T09:26:26.325 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c20238a:::benchmark_data_smithi122_168655_object22409:head# 2024-03-20T09:26:26.326 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c2242a8:::benchmark_data_smithi122_168655_object12417:head# 2024-03-20T09:26:26.326 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c22c9a0:::benchmark_data_smithi122_168655_object32157:head# 2024-03-20T09:26:26.327 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c230328:::benchmark_data_smithi122_168655_object24972:head# 2024-03-20T09:26:26.327 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c24236e:::benchmark_data_smithi122_168655_object20742:head# 2024-03-20T09:26:26.328 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c243081:::benchmark_data_smithi122_168655_object88123:head# 2024-03-20T09:26:26.328 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c2433b2:::benchmark_data_smithi122_168655_object99637:head# 2024-03-20T09:26:26.329 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c24d33e:::benchmark_data_smithi122_168655_object44040:head# 2024-03-20T09:26:26.329 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c252f6d:::benchmark_data_smithi122_168655_object11193:head# 2024-03-20T09:26:26.329 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c25ab2a:::benchmark_data_smithi122_168655_object79264:head# 2024-03-20T09:26:26.330 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c269bd1:::benchmark_data_smithi122_168655_object15769:head# 2024-03-20T09:26:26.330 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c2777c3:::benchmark_data_smithi122_168655_object34671:head# 2024-03-20T09:26:26.331 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c27b46d:::benchmark_data_smithi122_168655_object35486:head# 2024-03-20T09:26:26.331 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c294af3:::benchmark_data_smithi122_168655_object90772:head# 2024-03-20T09:26:26.332 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c2a3f54:::benchmark_data_smithi122_168655_object88531:head# 2024-03-20T09:26:26.332 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c2a9d01:::benchmark_data_smithi122_168655_object83406:head# 2024-03-20T09:26:26.332 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c2b670f:::benchmark_data_smithi122_168655_object100734:head# 2024-03-20T09:26:26.333 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c2c72de:::benchmark_data_smithi122_168655_object92447:head# 2024-03-20T09:26:26.333 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c2c8326:::benchmark_data_smithi122_168655_object96690:head# 2024-03-20T09:26:26.334 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c2e29f1:::benchmark_data_smithi122_168655_object48359:head# 2024-03-20T09:26:26.334 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c2f4804:::benchmark_data_smithi122_168655_object39214:head# 2024-03-20T09:26:26.335 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c304bd4:::benchmark_data_smithi122_168655_object71320:head# 2024-03-20T09:26:26.335 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c314de6:::benchmark_data_smithi122_168655_object65948:head# 2024-03-20T09:26:26.335 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c31e4d2:::benchmark_data_smithi122_168655_object89940:head# 2024-03-20T09:26:26.336 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c32642a:::benchmark_data_smithi122_168655_object115428:head# 2024-03-20T09:26:26.336 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c32931e:::benchmark_data_smithi122_168655_object118224:head# 2024-03-20T09:26:26.337 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c329b97:::benchmark_data_smithi122_168655_object79271:head# 2024-03-20T09:26:26.338 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c32a623:::benchmark_data_smithi122_168655_object18666:head# 2024-03-20T09:26:26.338 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c33870b:::benchmark_data_smithi122_168655_object30690:head# 2024-03-20T09:26:26.338 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c346f1e:::benchmark_data_smithi122_168655_object38520:head# 2024-03-20T09:26:26.339 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c349fed:::benchmark_data_smithi122_168655_object67829:head# 2024-03-20T09:26:26.339 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c355a95:::benchmark_data_smithi122_168655_object51086:head# 2024-03-20T09:26:26.340 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c35ef7f:::benchmark_data_smithi122_168655_object112657:head# 2024-03-20T09:26:26.340 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c3819d3:::benchmark_data_smithi122_168655_object98828:head# 2024-03-20T09:26:26.341 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c3dc17d:::benchmark_data_smithi122_168655_object87042:head# 2024-03-20T09:26:26.341 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c3e5c08:::benchmark_data_smithi122_168655_object62389:head# 2024-03-20T09:26:26.341 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c3e90b7:::benchmark_data_smithi122_168655_object73905:head# 2024-03-20T09:26:26.342 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c3eaa1d:::benchmark_data_smithi122_168655_object65526:head# 2024-03-20T09:26:26.342 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c3ec89e:::benchmark_data_smithi122_168655_object40181:head# 2024-03-20T09:26:26.343 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c3f7101:::benchmark_data_smithi122_168655_object119257:head# 2024-03-20T09:26:26.343 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c3f878a:::benchmark_data_smithi122_168655_object32230:head# 2024-03-20T09:26:26.343 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c417821:::benchmark_data_smithi122_168655_object29222:head# 2024-03-20T09:26:26.344 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c426cf1:::benchmark_data_smithi122_168655_object51905:head# 2024-03-20T09:26:26.344 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c430e2a:::benchmark_data_smithi122_168655_object119840:head# 2024-03-20T09:26:26.345 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c4596ce:::benchmark_data_smithi122_168655_object38267:head# 2024-03-20T09:26:26.345 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c46171e:::benchmark_data_smithi122_168655_object77252:head# 2024-03-20T09:26:26.346 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c4671a8:::benchmark_data_smithi122_168655_object87509:head# 2024-03-20T09:26:26.346 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c47d455:::benchmark_data_smithi122_168655_object65854:head# 2024-03-20T09:26:26.347 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c47de3c:::benchmark_data_smithi122_168655_object77187:head# 2024-03-20T09:26:26.347 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c47e108:::benchmark_data_smithi122_168655_object51606:head# 2024-03-20T09:26:26.348 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c485653:::benchmark_data_smithi122_168655_object119412:head# 2024-03-20T09:26:26.348 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c48a84d:::benchmark_data_smithi122_168655_object54196:head# 2024-03-20T09:26:26.348 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c48d342:::benchmark_data_smithi122_168655_object30191:head# 2024-03-20T09:26:26.349 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c4a2a41:::benchmark_data_smithi122_168655_object89659:head# 2024-03-20T09:26:26.349 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c4a57b1:::benchmark_data_smithi122_168655_object36479:head# 2024-03-20T09:26:26.350 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c4a928b:::benchmark_data_smithi122_168655_object115031:head# 2024-03-20T09:26:26.350 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c4b3a6c:::benchmark_data_smithi122_168655_object94493:head# 2024-03-20T09:26:26.350 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c4b4642:::benchmark_data_smithi122_168655_object11455:head# 2024-03-20T09:26:26.351 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c4b54e6:::benchmark_data_smithi122_168655_object41950:head# 2024-03-20T09:26:26.351 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c4c8328:::benchmark_data_smithi122_168655_object33401:head# 2024-03-20T09:26:26.352 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c4cd5c1:::benchmark_data_smithi122_168655_object82590:head# 2024-03-20T09:26:26.352 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c4d3d0e:::benchmark_data_smithi122_168655_object79421:head# 2024-03-20T09:26:26.353 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c4e069d:::benchmark_data_smithi122_168655_object40747:head# 2024-03-20T09:26:26.353 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c4e7e84:::benchmark_data_smithi122_168655_object106354:head# 2024-03-20T09:26:26.354 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c4eb805:::benchmark_data_smithi122_168655_object78614:head# 2024-03-20T09:26:26.354 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c4f3c40:::benchmark_data_smithi122_168655_object94554:head# 2024-03-20T09:26:26.355 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c502125:::benchmark_data_smithi122_168655_object81995:head# 2024-03-20T09:26:26.355 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c505f52:::benchmark_data_smithi122_168655_object106054:head# 2024-03-20T09:26:26.356 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c50ed0a:::benchmark_data_smithi122_168655_object117092:head# 2024-03-20T09:26:26.356 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c517ee2:::benchmark_data_smithi122_168655_object22287:head# 2024-03-20T09:26:26.356 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c51af99:::benchmark_data_smithi122_168655_object77250:head# 2024-03-20T09:26:26.357 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c54b1d3:::benchmark_data_smithi122_168655_object24782:head# 2024-03-20T09:26:26.357 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c55434c:::benchmark_data_smithi122_168655_object43878:head# 2024-03-20T09:26:26.358 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c5639e6:::benchmark_data_smithi122_168655_object101151:head# 2024-03-20T09:26:26.358 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c573bc1:::benchmark_data_smithi122_168655_object43352:head# 2024-03-20T09:26:26.359 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c57f03b:::benchmark_data_smithi122_168655_object77771:head# 2024-03-20T09:26:26.359 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c57f9e9:::benchmark_data_smithi122_168655_object107968:head# 2024-03-20T09:26:26.360 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c584ba1:::benchmark_data_smithi122_168655_object114213:head# 2024-03-20T09:26:26.360 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c589c99:::benchmark_data_smithi122_168655_object32478:head# 2024-03-20T09:26:26.360 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c590620:::benchmark_data_smithi122_168655_object118543:head# 2024-03-20T09:26:26.361 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c59d6f1:::benchmark_data_smithi122_168655_object27571:head# 2024-03-20T09:26:26.361 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c5a85b8:::benchmark_data_smithi122_168655_object12534:head# 2024-03-20T09:26:26.361 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c5b7d0b:::benchmark_data_smithi122_168655_object83787:head# 2024-03-20T09:26:26.362 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c5c60ac:::benchmark_data_smithi122_168655_object113865:head# 2024-03-20T09:26:26.362 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c5cac18:::benchmark_data_smithi122_168655_object10244:head# 2024-03-20T09:26:26.363 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c5d5101:::benchmark_data_smithi122_168655_object56543:head# 2024-03-20T09:26:26.363 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c5d5892:::benchmark_data_smithi122_168655_object119157:head# 2024-03-20T09:26:26.364 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c5dcae5:::benchmark_data_smithi122_168655_object71747:head# 2024-03-20T09:26:26.364 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c5e16a8:::benchmark_data_smithi122_168655_object69116:head# 2024-03-20T09:26:26.364 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c5ef8ed:::benchmark_data_smithi122_168655_object92371:head# 2024-03-20T09:26:26.365 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c5f87b0:::benchmark_data_smithi122_168655_object94592:head# 2024-03-20T09:26:26.365 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c6006cd:::benchmark_data_smithi122_168655_object106611:head# 2024-03-20T09:26:26.366 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c607208:::benchmark_data_smithi122_168655_object62296:head# 2024-03-20T09:26:26.366 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c60bf75:::benchmark_data_smithi122_168655_object33598:head# 2024-03-20T09:26:26.366 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c6144f2:::benchmark_data_smithi122_168655_object109423:head# 2024-03-20T09:26:26.367 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c61b72e:::benchmark_data_smithi122_168655_object26165:head# 2024-03-20T09:26:26.367 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c61da00:::benchmark_data_smithi122_168655_object74169:head# 2024-03-20T09:26:26.368 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c61fe6a:::benchmark_data_smithi122_168655_object61820:head# 2024-03-20T09:26:26.368 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c623924:::benchmark_data_smithi122_168655_object18367:head# 2024-03-20T09:26:26.368 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c62f19b:::benchmark_data_smithi122_168655_object29421:head# 2024-03-20T09:26:26.369 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c63211e:::benchmark_data_smithi122_168655_object21494:head# 2024-03-20T09:26:26.369 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c6513f6:::benchmark_data_smithi122_168655_object13019:head# 2024-03-20T09:26:26.370 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c65a7bc:::benchmark_data_smithi122_168655_object14126:head# 2024-03-20T09:26:26.370 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c673352:::benchmark_data_smithi122_168655_object89901:head# 2024-03-20T09:26:26.370 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c675893:::benchmark_data_smithi122_168655_object116533:head# 2024-03-20T09:26:26.371 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c67eb1a:::benchmark_data_smithi122_168655_object25183:head# 2024-03-20T09:26:26.371 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c68c29c:::benchmark_data_smithi122_168655_object109818:head# 2024-03-20T09:26:26.372 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c6a42dd:::benchmark_data_smithi122_168655_object116388:head# 2024-03-20T09:26:26.372 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c6ae187:::benchmark_data_smithi122_168655_object13597:head# 2024-03-20T09:26:26.373 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c6b93ca:::benchmark_data_smithi122_168655_object30518:head# 2024-03-20T09:26:26.373 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c6b9d76:::benchmark_data_smithi122_168655_object71885:head# 2024-03-20T09:26:26.373 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c6d3e5e:::benchmark_data_smithi122_168655_object81997:head# 2024-03-20T09:26:26.374 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c6eb18f:::benchmark_data_smithi122_168655_object50275:head# 2024-03-20T09:26:26.374 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c6ee293:::benchmark_data_smithi122_168655_object64460:head# 2024-03-20T09:26:26.375 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c700f1e:::benchmark_data_smithi122_168655_object73827:head# 2024-03-20T09:26:26.375 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c7014ab:::benchmark_data_smithi122_168655_object21270:head# 2024-03-20T09:26:26.376 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c703cf3:::benchmark_data_smithi122_168655_object35596:head# 2024-03-20T09:26:26.376 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c7126fb:::benchmark_data_smithi122_168655_object92678:head# 2024-03-20T09:26:26.376 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c7134e2:::benchmark_data_smithi122_168655_object115637:head# 2024-03-20T09:26:26.377 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c724aa1:::benchmark_data_smithi122_168655_object24423:head# 2024-03-20T09:26:26.377 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c72bb98:::benchmark_data_smithi122_168655_object108026:head# 2024-03-20T09:26:26.377 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c733970:::benchmark_data_smithi122_168655_object109802:head# 2024-03-20T09:26:26.378 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c73f296:::benchmark_data_smithi122_168655_object81338:head# 2024-03-20T09:26:26.378 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c741aeb:::benchmark_data_smithi122_168655_object22333:head# 2024-03-20T09:26:26.379 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c742568:::benchmark_data_smithi122_168655_object106202:head# 2024-03-20T09:26:26.379 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c744027:::benchmark_data_smithi122_168655_object30054:head# 2024-03-20T09:26:26.379 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c7450f8:::benchmark_data_smithi122_168655_object11904:head# 2024-03-20T09:26:26.380 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c747284:::benchmark_data_smithi122_168655_object36305:head# 2024-03-20T09:26:26.380 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c74cfbf:::benchmark_data_smithi122_168655_object60138:head# 2024-03-20T09:26:26.381 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c758d2c:::benchmark_data_smithi122_168655_object62038:head# 2024-03-20T09:26:26.381 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c76cda5:::benchmark_data_smithi122_168655_object76702:head# 2024-03-20T09:26:26.382 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c76f11e:::benchmark_data_smithi122_168655_object22938:head# 2024-03-20T09:26:26.382 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c77f3db:::benchmark_data_smithi122_168655_object82987:head# 2024-03-20T09:26:26.382 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c79a461:::benchmark_data_smithi122_168655_object11706:head# 2024-03-20T09:26:26.383 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c79ea48:::benchmark_data_smithi122_168655_object105622:head# 2024-03-20T09:26:26.383 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c7ad6c0:::benchmark_data_smithi122_168655_object85294:head# 2024-03-20T09:26:26.384 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c7b191b:::benchmark_data_smithi122_168655_object66388:head# 2024-03-20T09:26:26.384 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c7c33a0:::benchmark_data_smithi122_168655_object117420:head# 2024-03-20T09:26:26.385 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c7c4da7:::benchmark_data_smithi122_168655_object93167:head# 2024-03-20T09:26:26.385 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c7c5a0b:::benchmark_data_smithi122_168655_object31760:head# 2024-03-20T09:26:26.386 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c7cd173:::benchmark_data_smithi122_168655_object21436:head# 2024-03-20T09:26:26.386 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c7d7107:::benchmark_data_smithi122_168655_object38264:head# 2024-03-20T09:26:26.386 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c7d8216:::benchmark_data_smithi122_168655_object100810:head# 2024-03-20T09:26:26.387 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c7ebcc5:::benchmark_data_smithi122_168655_object17770:head# 2024-03-20T09:26:26.387 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c7ec004:::benchmark_data_smithi122_168655_object29608:head# 2024-03-20T09:26:26.388 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c802d43:::benchmark_data_smithi122_168655_object108016:head# 2024-03-20T09:26:26.388 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c8163cc:::benchmark_data_smithi122_168655_object74852:head# 2024-03-20T09:26:26.388 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c816bda:::benchmark_data_smithi122_168655_object101314:head# 2024-03-20T09:26:26.389 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c822b8a:::benchmark_data_smithi122_168655_object14567:head# 2024-03-20T09:26:26.389 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c832453:::benchmark_data_smithi122_168655_object58141:head# 2024-03-20T09:26:26.390 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c8337b0:::benchmark_data_smithi122_168655_object63693:head# 2024-03-20T09:26:26.390 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c83f1b1:::benchmark_data_smithi122_168655_object92105:head# 2024-03-20T09:26:26.391 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c844f9d:::benchmark_data_smithi122_168655_object43651:head# 2024-03-20T09:26:26.391 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c84c2d0:::benchmark_data_smithi122_168655_object34576:head# 2024-03-20T09:26:26.391 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c867174:::benchmark_data_smithi122_168655_object118473:head# 2024-03-20T09:26:26.392 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c86f7d1:::benchmark_data_smithi122_168655_object24842:head# 2024-03-20T09:26:26.392 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c89b88b:::benchmark_data_smithi122_168655_object119926:head# 2024-03-20T09:26:26.393 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c8a19c7:::benchmark_data_smithi122_168655_object87135:head# 2024-03-20T09:26:26.393 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c8aabff:::benchmark_data_smithi122_168655_object86993:head# 2024-03-20T09:26:26.393 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c8af9c6:::benchmark_data_smithi122_168655_object26666:head# 2024-03-20T09:26:26.394 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c8b3715:::benchmark_data_smithi122_168655_object102448:head# 2024-03-20T09:26:26.394 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c8c007b:::benchmark_data_smithi122_168655_object25050:head# 2024-03-20T09:26:26.395 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c8c3cac:::benchmark_data_smithi122_168655_object41358:head# 2024-03-20T09:26:26.395 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c8d98c4:::benchmark_data_smithi122_168655_object92197:head# 2024-03-20T09:26:26.396 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c8dcebb:::benchmark_data_smithi122_168655_object93681:head# 2024-03-20T09:26:26.396 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c8ec6b3:::benchmark_data_smithi122_168655_object75953:head# 2024-03-20T09:26:26.396 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c8ee51a:::benchmark_data_smithi122_168655_object90603:head# 2024-03-20T09:26:26.397 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c8f1736:::benchmark_data_smithi122_168655_object119879:head# 2024-03-20T09:26:26.397 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c9074fc:::benchmark_data_smithi122_168655_object34488:head# 2024-03-20T09:26:26.398 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c90b349:::benchmark_data_smithi122_168655_object47919:head# 2024-03-20T09:26:26.398 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c90c2fa:::benchmark_data_smithi122_168655_object12118:head# 2024-03-20T09:26:26.399 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c91abe6:::benchmark_data_smithi122_168655_object106846:head# 2024-03-20T09:26:26.399 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c922db5:::benchmark_data_smithi122_168655_object50456:head# 2024-03-20T09:26:26.400 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c928bce:::benchmark_data_smithi122_168655_object93390:head# 2024-03-20T09:26:26.400 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c9342ba:::benchmark_data_smithi122_168655_object111890:head# 2024-03-20T09:26:26.401 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c93ac23:::benchmark_data_smithi122_168655_object93385:head# 2024-03-20T09:26:26.401 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c93e2ee:::benchmark_data_smithi122_168655_object103777:head# 2024-03-20T09:26:26.401 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c94655b:::benchmark_data_smithi122_168655_object114227:head# 2024-03-20T09:26:26.402 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c949c00:::benchmark_data_smithi122_168655_object32217:head# 2024-03-20T09:26:26.402 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c963847:::benchmark_data_smithi122_168655_object46654:head# 2024-03-20T09:26:26.403 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c967272:::benchmark_data_smithi122_168655_object46979:head# 2024-03-20T09:26:26.403 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c96729e:::benchmark_data_smithi122_168655_object98597:head# 2024-03-20T09:26:26.403 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c970a45:::benchmark_data_smithi122_168655_object92237:head# 2024-03-20T09:26:26.404 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c9726af:::benchmark_data_smithi122_168655_object56841:head# 2024-03-20T09:26:26.404 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c983b3a:::benchmark_data_smithi122_168655_object82286:head# 2024-03-20T09:26:26.405 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c9959ae:::benchmark_data_smithi122_168655_object29209:head# 2024-03-20T09:26:26.405 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c99c251:::benchmark_data_smithi122_168655_object78295:head# 2024-03-20T09:26:26.406 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c9ae215:::benchmark_data_smithi122_168655_object74685:head# 2024-03-20T09:26:26.406 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c9b8efc:::benchmark_data_smithi122_168655_object17779:head# 2024-03-20T09:26:26.407 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c9c039f:::benchmark_data_smithi122_168655_object74773:head# 2024-03-20T09:26:26.407 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c9c7eaf:::benchmark_data_smithi122_168655_object39219:head# 2024-03-20T09:26:26.407 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c9d28fd:::benchmark_data_smithi122_168655_object93824:head# 2024-03-20T09:26:26.408 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c9d2c3a:::benchmark_data_smithi122_168655_object25932:head# 2024-03-20T09:26:26.408 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c9e0abe:::benchmark_data_smithi122_168655_object109374:head# 2024-03-20T09:26:26.409 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c9e964d:::benchmark_data_smithi122_168655_object116122:head# 2024-03-20T09:26:26.409 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c9f365a:::benchmark_data_smithi122_168655_object119018:head# 2024-03-20T09:26:26.410 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c9fd935:::benchmark_data_smithi122_168655_object54787:head# 2024-03-20T09:26:26.410 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8c9ffe5a:::benchmark_data_smithi122_168655_object23264:head# 2024-03-20T09:26:26.410 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ca133a4:::benchmark_data_smithi122_168655_object48771:head# 2024-03-20T09:26:26.411 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ca2c1ff:::benchmark_data_smithi122_168655_object12193:head# 2024-03-20T09:26:26.411 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ca2f613:::benchmark_data_smithi122_168655_object28521:head# 2024-03-20T09:26:26.412 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ca355f8:::benchmark_data_smithi122_168655_object108492:head# 2024-03-20T09:26:26.412 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ca3e590:::benchmark_data_smithi122_168655_object107698:head# 2024-03-20T09:26:26.413 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ca49537:::benchmark_data_smithi122_168655_object53140:head# 2024-03-20T09:26:26.413 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ca528fb:::benchmark_data_smithi122_168655_object16381:head# 2024-03-20T09:26:26.414 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ca5df76:::benchmark_data_smithi122_168655_object99396:head# 2024-03-20T09:26:26.414 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ca70ddf:::benchmark_data_smithi122_168655_object11943:head# 2024-03-20T09:26:26.414 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ca71a5d:::benchmark_data_smithi122_168655_object109505:head# 2024-03-20T09:26:26.415 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ca72d7d:::benchmark_data_smithi122_168655_object62650:head# 2024-03-20T09:26:26.415 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ca75e46:::benchmark_data_smithi122_168655_object85667:head# 2024-03-20T09:26:26.416 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ca89fde:::benchmark_data_smithi122_168655_object51788:head# 2024-03-20T09:26:26.416 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ca9b5c7:::benchmark_data_smithi122_168655_object88871:head# 2024-03-20T09:26:26.417 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8caa2fc9:::benchmark_data_smithi122_168655_object64717:head# 2024-03-20T09:26:26.417 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8caa450d:::benchmark_data_smithi122_168655_object119124:head# 2024-03-20T09:26:26.417 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8caa61c5:::benchmark_data_smithi122_168655_object75617:head# 2024-03-20T09:26:26.418 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cac19b1:::benchmark_data_smithi122_168655_object110984:head# 2024-03-20T09:26:26.418 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cac1b5d:::benchmark_data_smithi122_168655_object56109:head# 2024-03-20T09:26:26.419 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cac5346:::benchmark_data_smithi122_168655_object105316:head# 2024-03-20T09:26:26.419 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cac9366:::benchmark_data_smithi122_168655_object34268:head# 2024-03-20T09:26:26.419 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cacf04b:::benchmark_data_smithi122_168655_object117779:head# 2024-03-20T09:26:26.420 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cad2635:::benchmark_data_smithi122_168655_object89686:head# 2024-03-20T09:26:26.420 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cadae8b:::benchmark_data_smithi122_168655_object116376:head# 2024-03-20T09:26:26.421 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cae0b9b:::benchmark_data_smithi122_168655_object39445:head# 2024-03-20T09:26:26.421 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8caecc46:::benchmark_data_smithi122_168655_object57194:head# 2024-03-20T09:26:26.422 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8caf1e6a:::benchmark_data_smithi122_168655_object27593:head# 2024-03-20T09:26:26.422 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cafd825:::benchmark_data_smithi122_168655_object63997:head# 2024-03-20T09:26:26.423 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cb06df0:::benchmark_data_smithi122_168655_object94751:head# 2024-03-20T09:26:26.423 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cb089c6:::benchmark_data_smithi122_168655_object49797:head# 2024-03-20T09:26:26.423 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cb11d8d:::benchmark_data_smithi122_168655_object118609:head# 2024-03-20T09:26:26.424 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cb1e5e5:::benchmark_data_smithi122_168655_object90927:head# 2024-03-20T09:26:26.424 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cb20d43:::benchmark_data_smithi122_168655_object99825:head# 2024-03-20T09:26:26.425 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cb21853:::benchmark_data_smithi122_168655_object33270:head# 2024-03-20T09:26:26.425 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cb288b1:::benchmark_data_smithi122_168655_object24906:head# 2024-03-20T09:26:26.425 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cb2e951:::benchmark_data_smithi122_168655_object94184:head# 2024-03-20T09:26:26.426 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cb3d8ad:::benchmark_data_smithi122_168655_object43659:head# 2024-03-20T09:26:26.426 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cb40243:::benchmark_data_smithi122_168655_object88783:head# 2024-03-20T09:26:26.427 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cb421e9:::benchmark_data_smithi122_168655_object113743:head# 2024-03-20T09:26:26.427 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cb44e40:::benchmark_data_smithi122_168655_object40033:head# 2024-03-20T09:26:26.427 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cb6317e:::benchmark_data_smithi122_168655_object99845:head# 2024-03-20T09:26:26.428 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cb66926:::benchmark_data_smithi122_168655_object72428:head# 2024-03-20T09:26:26.428 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cb6f5b1:::benchmark_data_smithi122_168655_object64012:head# 2024-03-20T09:26:26.429 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cb72d66:::benchmark_data_smithi122_168655_object91398:head# 2024-03-20T09:26:26.429 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cb78691:::benchmark_data_smithi122_168655_object70777:head# 2024-03-20T09:26:26.430 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cb806c2:::benchmark_data_smithi122_168655_object44298:head# 2024-03-20T09:26:26.430 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cb862f7:::benchmark_data_smithi122_168655_object111049:head# 2024-03-20T09:26:26.430 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cb8e7f0:::benchmark_data_smithi122_168655_object51931:head# 2024-03-20T09:26:26.431 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cb913f5:::benchmark_data_smithi122_168655_object100111:head# 2024-03-20T09:26:26.431 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cb9398b:::benchmark_data_smithi122_168655_object62258:head# 2024-03-20T09:26:26.432 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cb961d0:::benchmark_data_smithi122_168655_object22755:head# 2024-03-20T09:26:26.432 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cb967ce:::benchmark_data_smithi122_168655_object34147:head# 2024-03-20T09:26:26.433 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cb9b1f6:::benchmark_data_smithi122_168655_object89270:head# 2024-03-20T09:26:26.433 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cba38a7:::benchmark_data_smithi122_168655_object79023:head# 2024-03-20T09:26:26.433 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cba565f:::benchmark_data_smithi122_168655_object61317:head# 2024-03-20T09:26:26.434 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cbafe85:::benchmark_data_smithi122_168655_object94266:head# 2024-03-20T09:26:26.434 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cbbeaef:::benchmark_data_smithi122_168655_object65199:head# 2024-03-20T09:26:26.435 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cbc2a7d:::benchmark_data_smithi122_168655_object38088:head# 2024-03-20T09:26:26.435 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cbd3189:::benchmark_data_smithi122_168655_object37622:head# 2024-03-20T09:26:26.435 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cbf28fd:::benchmark_data_smithi122_168655_object74785:head# 2024-03-20T09:26:26.436 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cbfbbc9:::benchmark_data_smithi122_168655_object19153:head# 2024-03-20T09:26:26.436 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cc08dae:::benchmark_data_smithi122_168655_object41357:head# 2024-03-20T09:26:26.437 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cc2567d:::benchmark_data_smithi122_168655_object12406:head# 2024-03-20T09:26:26.437 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cc2b575:::benchmark_data_smithi122_168655_object101011:head# 2024-03-20T09:26:26.438 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cc34e3f:::benchmark_data_smithi122_168655_object13343:head# 2024-03-20T09:26:26.438 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cc356c5:::benchmark_data_smithi122_168655_object42820:head# 2024-03-20T09:26:26.439 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cc450cf:::benchmark_data_smithi122_168655_object22710:head# 2024-03-20T09:26:26.439 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cc458ac:::benchmark_data_smithi122_168655_object92334:head# 2024-03-20T09:26:26.440 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cc4cb0e:::benchmark_data_smithi122_168655_object110182:head# 2024-03-20T09:26:26.440 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cc5701d:::benchmark_data_smithi122_168655_object70354:head# 2024-03-20T09:26:26.440 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cc5aea9:::benchmark_data_smithi122_168655_object13288:head# 2024-03-20T09:26:26.441 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cc67d00:::benchmark_data_smithi122_168655_object20577:head# 2024-03-20T09:26:26.441 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cc6cb37:::benchmark_data_smithi122_168655_object84281:head# 2024-03-20T09:26:26.442 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cc8c163:::benchmark_data_smithi122_168655_object76647:head# 2024-03-20T09:26:26.442 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cc936f3:::benchmark_data_smithi122_168655_object108759:head# 2024-03-20T09:26:26.442 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cc9eacf:::benchmark_data_smithi122_168655_object23408:head# 2024-03-20T09:26:26.443 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ccbd68e:::benchmark_data_smithi122_168655_object20097:head# 2024-03-20T09:26:26.443 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ccd0d1c:::benchmark_data_smithi122_168655_object99839:head# 2024-03-20T09:26:26.444 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ccdb98c:::benchmark_data_smithi122_168655_object92130:head# 2024-03-20T09:26:26.444 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ccdba71:::benchmark_data_smithi122_168655_object16341:head# 2024-03-20T09:26:26.445 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cced362:::benchmark_data_smithi122_168655_object91978:head# 2024-03-20T09:26:26.445 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ccfafd5:::benchmark_data_smithi122_168655_object47230:head# 2024-03-20T09:26:26.445 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ccfdf0a:::benchmark_data_smithi122_168655_object50217:head# 2024-03-20T09:26:26.446 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cd04d0f:::benchmark_data_smithi122_168655_object18865:head# 2024-03-20T09:26:26.446 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cd10891:::benchmark_data_smithi122_168655_object36620:head# 2024-03-20T09:26:26.447 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cd175ec:::benchmark_data_smithi122_168655_object115190:head# 2024-03-20T09:26:26.447 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cd17d27:::benchmark_data_smithi122_168655_object68549:head# 2024-03-20T09:26:26.448 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cd19909:::benchmark_data_smithi122_168655_object15941:head# 2024-03-20T09:26:26.448 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cd1c59c:::benchmark_data_smithi122_168655_object31840:head# 2024-03-20T09:26:26.448 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cd2633a:::benchmark_data_smithi122_168655_object13154:head# 2024-03-20T09:26:26.449 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cd2b274:::benchmark_data_smithi122_168655_object119865:head# 2024-03-20T09:26:26.449 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cd2bae3:::benchmark_data_smithi122_168655_object85437:head# 2024-03-20T09:26:26.450 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cd2dc6d:::benchmark_data_smithi122_168655_object21976:head# 2024-03-20T09:26:26.450 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cd32fc0:::benchmark_data_smithi122_168655_object66634:head# 2024-03-20T09:26:26.451 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cd3d0ea:::benchmark_data_smithi122_168655_object31385:head# 2024-03-20T09:26:26.451 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cd53125:::benchmark_data_smithi122_168655_object29242:head# 2024-03-20T09:26:26.452 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cd5ada6:::benchmark_data_smithi122_168655_object67231:head# 2024-03-20T09:26:26.452 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cd5f4c5:::benchmark_data_smithi122_168655_object23126:head# 2024-03-20T09:26:26.453 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cd68750:::benchmark_data_smithi122_168655_object81238:head# 2024-03-20T09:26:26.453 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cd6e07b:::benchmark_data_smithi122_168655_object65495:head# 2024-03-20T09:26:26.453 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cd75128:::benchmark_data_smithi122_168655_object44988:head# 2024-03-20T09:26:26.454 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cd8b6a9:::benchmark_data_smithi122_168655_object43497:head# 2024-03-20T09:26:26.454 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cd94604:::benchmark_data_smithi122_168655_object57268:head# 2024-03-20T09:26:26.454 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cd964bf:::benchmark_data_smithi122_168655_object10876:head# 2024-03-20T09:26:26.455 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cd9c9bf:::benchmark_data_smithi122_168655_object65045:head# 2024-03-20T09:26:26.455 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cda8110:::benchmark_data_smithi122_168655_object82905:head# 2024-03-20T09:26:26.455 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cdb4af9:::benchmark_data_smithi122_168655_object33027:head# 2024-03-20T09:26:26.456 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cdc6a4d:::benchmark_data_smithi122_168655_object19866:head# 2024-03-20T09:26:26.456 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cdd09d2:::benchmark_data_smithi122_168655_object102710:head# 2024-03-20T09:26:26.458 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cddcc29:::benchmark_data_smithi122_168655_object14908:head# 2024-03-20T09:26:26.458 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cdde164:::benchmark_data_smithi122_168655_object38082:head# 2024-03-20T09:26:26.458 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cde4164:::benchmark_data_smithi122_168655_object36369:head# 2024-03-20T09:26:26.458 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cdf7d79:::benchmark_data_smithi122_168655_object18566:head# 2024-03-20T09:26:26.459 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ce070e2:::benchmark_data_smithi122_168655_object116729:head# 2024-03-20T09:26:26.459 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ce2f3d6:::benchmark_data_smithi122_168655_object42038:head# 2024-03-20T09:26:26.460 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ce3f9a1:::benchmark_data_smithi122_168655_object99195:head# 2024-03-20T09:26:26.460 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ce4df3c:::benchmark_data_smithi122_168655_object28622:head# 2024-03-20T09:26:26.461 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ce53bae:::benchmark_data_smithi122_168655_object77456:head# 2024-03-20T09:26:26.461 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ce5e0b8:::benchmark_data_smithi122_168655_object39985:head# 2024-03-20T09:26:26.462 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ce68386:::benchmark_data_smithi122_168655_object39184:head# 2024-03-20T09:26:26.462 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ce6c692:::benchmark_data_smithi122_168655_object50085:head# 2024-03-20T09:26:26.462 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ce84e1f:::benchmark_data_smithi122_168655_object111110:head# 2024-03-20T09:26:26.463 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ce86092:::benchmark_data_smithi122_168655_object56163:head# 2024-03-20T09:26:26.463 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ce94264:::benchmark_data_smithi122_168655_object13363:head# 2024-03-20T09:26:26.464 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ce9cc23:::benchmark_data_smithi122_168655_object49908:head# 2024-03-20T09:26:26.464 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cea1cf7:::benchmark_data_smithi122_168655_object21952:head# 2024-03-20T09:26:26.465 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cea9f4d:::benchmark_data_smithi122_168655_object76826:head# 2024-03-20T09:26:26.465 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cead519:::benchmark_data_smithi122_168655_object115269:head# 2024-03-20T09:26:26.465 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cead676:::benchmark_data_smithi122_168655_object68990:head# 2024-03-20T09:26:26.466 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ceba882:::benchmark_data_smithi122_168655_object63873:head# 2024-03-20T09:26:26.467 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cec0a3d:::benchmark_data_smithi122_168655_object12642:head# 2024-03-20T09:26:26.467 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cec91ba:::benchmark_data_smithi122_168655_object83155:head# 2024-03-20T09:26:26.468 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cedb6ec:::benchmark_data_smithi122_168655_object55804:head# 2024-03-20T09:26:26.468 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cee8f01:::benchmark_data_smithi122_168655_object102352:head# 2024-03-20T09:26:26.468 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ceefb3f:::benchmark_data_smithi122_168655_object108318:head# 2024-03-20T09:26:26.469 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cef531a:::benchmark_data_smithi122_168655_object59937:head# 2024-03-20T09:26:26.469 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cf0081a:::benchmark_data_smithi122_168655_object47093:head# 2024-03-20T09:26:26.470 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cf091ca:::benchmark_data_smithi122_168655_object28945:head# 2024-03-20T09:26:26.470 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cf3a915:::benchmark_data_smithi122_168655_object44828:head# 2024-03-20T09:26:26.471 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cf3cb5e:::benchmark_data_smithi122_168655_object115057:head# 2024-03-20T09:26:26.471 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cf5145b:::benchmark_data_smithi122_168655_object83715:head# 2024-03-20T09:26:26.472 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cf59ab7:::benchmark_data_smithi122_168655_object65283:head# 2024-03-20T09:26:26.472 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cf59d76:::benchmark_data_smithi122_168655_object51557:head# 2024-03-20T09:26:26.473 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cf6711c:::benchmark_data_smithi122_168655_object27425:head# 2024-03-20T09:26:26.473 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cf67314:::benchmark_data_smithi122_168655_object35629:head# 2024-03-20T09:26:26.473 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cf69b42:::benchmark_data_smithi122_168655_object14220:head# 2024-03-20T09:26:26.474 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cf7159f:::benchmark_data_smithi122_168655_object25724:head# 2024-03-20T09:26:26.474 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cf7338a:::benchmark_data_smithi122_168655_object33548:head# 2024-03-20T09:26:26.475 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cf73cf7:::benchmark_data_smithi122_168655_object40513:head# 2024-03-20T09:26:26.475 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cf7433d:::benchmark_data_smithi122_168655_object119792:head# 2024-03-20T09:26:26.475 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cf75098:::benchmark_data_smithi122_168655_object40647:head# 2024-03-20T09:26:26.476 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cf83233:::benchmark_data_smithi122_168655_object58389:head# 2024-03-20T09:26:26.476 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cf86fff:::benchmark_data_smithi122_168655_object108433:head# 2024-03-20T09:26:26.476 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cf8fd65:::benchmark_data_smithi122_168655_object112961:head# 2024-03-20T09:26:26.477 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cfadeb2:::benchmark_data_smithi122_168655_object113471:head# 2024-03-20T09:26:26.477 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cfb502d:::benchmark_data_smithi122_168655_object82119:head# 2024-03-20T09:26:26.478 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cfba573:::benchmark_data_smithi122_168655_object52713:head# 2024-03-20T09:26:26.478 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cfba6f8:::benchmark_data_smithi122_168655_object38236:head# 2024-03-20T09:26:26.479 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cfbdaa6:::benchmark_data_smithi122_168655_object48593:head# 2024-03-20T09:26:26.479 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cfc1ca1:::benchmark_data_smithi122_168655_object17097:head# 2024-03-20T09:26:26.479 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cfd5597:::benchmark_data_smithi122_168655_object73034:head# 2024-03-20T09:26:26.480 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cfe8ca9:::benchmark_data_smithi122_168655_object60488:head# 2024-03-20T09:26:26.480 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cfeb728:::benchmark_data_smithi122_168655_object99624:head# 2024-03-20T09:26:26.481 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cff2d65:::benchmark_data_smithi122_168655_object69267:head# 2024-03-20T09:26:26.481 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cff4270:::benchmark_data_smithi122_168655_object91385:head# 2024-03-20T09:26:26.482 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8cffaa98:::benchmark_data_smithi122_168655_object57798:head# 2024-03-20T09:26:26.482 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d00e726:::benchmark_data_smithi122_168655_object101797:head# 2024-03-20T09:26:26.483 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d01a20b:::benchmark_data_smithi122_168655_object118644:head# 2024-03-20T09:26:26.483 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d04ada5:::benchmark_data_smithi122_168655_object30169:head# 2024-03-20T09:26:26.484 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d0562d1:::benchmark_data_smithi122_168655_object85901:head# 2024-03-20T09:26:26.484 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d057f5f:::benchmark_data_smithi122_168655_object17402:head# 2024-03-20T09:26:26.487 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d0597f4:::benchmark_data_smithi122_168655_object87129:head# 2024-03-20T09:26:26.487 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d077078:::benchmark_data_smithi122_168655_object18332:head# 2024-03-20T09:26:26.487 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d07a7be:::benchmark_data_smithi122_168655_object23535:head# 2024-03-20T09:26:26.488 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d0838a3:::benchmark_data_smithi122_168655_object63330:head# 2024-03-20T09:26:26.488 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d0921df:::benchmark_data_smithi122_168655_object109213:head# 2024-03-20T09:26:26.489 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d09d55d:::benchmark_data_smithi122_168655_object85418:head# 2024-03-20T09:26:26.489 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d09dbd4:::benchmark_data_smithi122_168655_object101669:head# 2024-03-20T09:26:26.490 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d0a7fd3:::benchmark_data_smithi122_168655_object73872:head# 2024-03-20T09:26:26.490 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d0acb39:::benchmark_data_smithi122_168655_object12499:head# 2024-03-20T09:26:26.490 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d0afe4b:::benchmark_data_smithi122_168655_object49744:head# 2024-03-20T09:26:26.491 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d0f1c0d:::benchmark_data_smithi122_168655_object81210:head# 2024-03-20T09:26:26.492 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d0f2ea1:::benchmark_data_smithi122_168655_object110879:head# 2024-03-20T09:26:26.492 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d0f393c:::benchmark_data_smithi122_168655_object111218:head# 2024-03-20T09:26:26.493 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d107193:::benchmark_data_smithi122_168655_object114658:head# 2024-03-20T09:26:26.493 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d10d51c:::benchmark_data_smithi122_168655_object82006:head# 2024-03-20T09:26:26.494 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d1375bd:::benchmark_data_smithi122_168655_object115751:head# 2024-03-20T09:26:26.494 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d13dbd4:::benchmark_data_smithi122_168655_object82049:head# 2024-03-20T09:26:26.495 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d1431bd:::benchmark_data_smithi122_168655_object99971:head# 2024-03-20T09:26:26.495 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d1437a8:::benchmark_data_smithi122_168655_object66128:head# 2024-03-20T09:26:26.495 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d145e5f:::benchmark_data_smithi122_168655_object53101:head# 2024-03-20T09:26:26.496 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d1471dc:::benchmark_data_smithi122_168655_object29558:head# 2024-03-20T09:26:26.496 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d14e2db:::benchmark_data_smithi122_168655_object48191:head# 2024-03-20T09:26:26.497 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d152845:::benchmark_data_smithi122_168655_object116862:head# 2024-03-20T09:26:26.498 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d1528ce:::benchmark_data_smithi122_168655_object21968:head# 2024-03-20T09:26:26.498 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d152944:::benchmark_data_smithi122_168655_object61683:head# 2024-03-20T09:26:26.499 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d15aa08:::benchmark_data_smithi122_168655_object16275:head# 2024-03-20T09:26:26.499 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d15aaf2:::benchmark_data_smithi122_168655_object10675:head# 2024-03-20T09:26:26.500 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d16b5f8:::benchmark_data_smithi122_168655_object25741:head# 2024-03-20T09:26:26.501 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d16db24:::benchmark_data_smithi122_168655_object42160:head# 2024-03-20T09:26:26.501 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d178b90:::benchmark_data_smithi122_168655_object65397:head# 2024-03-20T09:26:26.502 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d17fea3:::benchmark_data_smithi122_168655_object31981:head# 2024-03-20T09:26:26.502 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d1879a1:::benchmark_data_smithi122_168655_object31245:head# 2024-03-20T09:26:26.503 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d1892bb:::benchmark_data_smithi122_168655_object77883:head# 2024-03-20T09:26:26.503 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d18bf2b:::benchmark_data_smithi122_168655_object75458:head# 2024-03-20T09:26:26.503 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d192cc0:::benchmark_data_smithi122_168655_object85508:head# 2024-03-20T09:26:26.504 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d1a92bd:::benchmark_data_smithi122_168655_object61988:head# 2024-03-20T09:26:26.504 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d1b1066:::benchmark_data_smithi122_168655_object99639:head# 2024-03-20T09:26:26.505 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d1b811f:::benchmark_data_smithi122_168655_object42897:head# 2024-03-20T09:26:26.505 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d1cd03e:::benchmark_data_smithi122_168655_object71856:head# 2024-03-20T09:26:26.506 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d1d5939:::benchmark_data_smithi122_168655_object93190:head# 2024-03-20T09:26:26.506 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d1d6dd1:::benchmark_data_smithi122_168655_object32021:head# 2024-03-20T09:26:26.507 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d1d9d97:::benchmark_data_smithi122_168655_object32928:head# 2024-03-20T09:26:26.507 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d1de94e:::benchmark_data_smithi122_168655_object85091:head# 2024-03-20T09:26:26.508 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d1e62b0:::benchmark_data_smithi122_168655_object86963:head# 2024-03-20T09:26:26.508 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d1e8786:::benchmark_data_smithi122_168655_object107231:head# 2024-03-20T09:26:26.509 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d1e9e77:::benchmark_data_smithi122_168655_object14460:head# 2024-03-20T09:26:26.509 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d1f56a7:::benchmark_data_smithi122_168655_object16960:head# 2024-03-20T09:26:26.509 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d20b6bb:::benchmark_data_smithi122_168655_object45263:head# 2024-03-20T09:26:26.510 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d226fe5:::benchmark_data_smithi122_168655_object34132:head# 2024-03-20T09:26:26.510 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d236263:::benchmark_data_smithi122_168655_object73908:head# 2024-03-20T09:26:26.511 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d23e70e:::benchmark_data_smithi122_168655_object45510:head# 2024-03-20T09:26:26.511 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d23ff26:::benchmark_data_smithi122_168655_object73910:head# 2024-03-20T09:26:26.511 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d241790:::benchmark_data_smithi122_168655_object68727:head# 2024-03-20T09:26:26.512 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d24f1b3:::benchmark_data_smithi122_168655_object50277:head# 2024-03-20T09:26:26.512 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d252232:::benchmark_data_smithi122_168655_object62201:head# 2024-03-20T09:26:26.512 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d272cb6:::benchmark_data_smithi122_168655_object42830:head# 2024-03-20T09:26:26.513 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d276b39:::benchmark_data_smithi122_168655_object110695:head# 2024-03-20T09:26:26.513 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d27a7e8:::benchmark_data_smithi122_168655_object105164:head# 2024-03-20T09:26:26.514 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d283556:::benchmark_data_smithi122_168655_object55377:head# 2024-03-20T09:26:26.514 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d28402d:::benchmark_data_smithi122_168655_object99942:head# 2024-03-20T09:26:26.515 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d2a3758:::benchmark_data_smithi122_168655_object64110:head# 2024-03-20T09:26:26.515 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d2ad066:::benchmark_data_smithi122_168655_object51659:head# 2024-03-20T09:26:26.516 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d2b8a03:::benchmark_data_smithi122_168655_object108801:head# 2024-03-20T09:26:26.516 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d2bd4f2:::benchmark_data_smithi122_168655_object100090:head# 2024-03-20T09:26:26.516 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d2beee3:::benchmark_data_smithi122_168655_object113922:head# 2024-03-20T09:26:26.517 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d2d95ad:::benchmark_data_smithi122_168655_object105717:head# 2024-03-20T09:26:26.517 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d2e3000:::benchmark_data_smithi122_168655_object51850:head# 2024-03-20T09:26:26.518 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d2e68b6:::benchmark_data_smithi122_168655_object113098:head# 2024-03-20T09:26:26.518 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d2eafbc:::benchmark_data_smithi122_168655_object30172:head# 2024-03-20T09:26:26.519 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d2f0b86:::benchmark_data_smithi122_168655_object90805:head# 2024-03-20T09:26:26.519 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d2f65aa:::benchmark_data_smithi122_168655_object50407:head# 2024-03-20T09:26:26.520 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d310f65:::benchmark_data_smithi122_168655_object78262:head# 2024-03-20T09:26:26.520 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d316119:::benchmark_data_smithi122_168655_object108684:head# 2024-03-20T09:26:26.520 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d31c993:::benchmark_data_smithi122_168655_object110265:head# 2024-03-20T09:26:26.521 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d32037b:::benchmark_data_smithi122_168655_object12927:head# 2024-03-20T09:26:26.521 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d32e12f:::benchmark_data_smithi122_168655_object95100:head# 2024-03-20T09:26:26.522 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d33049c:::benchmark_data_smithi122_168655_object61156:head# 2024-03-20T09:26:26.522 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d35fc29:::benchmark_data_smithi122_168655_object92614:head# 2024-03-20T09:26:26.523 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d360086:::benchmark_data_smithi122_168655_object63538:head# 2024-03-20T09:26:26.523 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d375ef9:::benchmark_data_smithi122_168655_object57823:head# 2024-03-20T09:26:26.524 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d37ee9a:::benchmark_data_smithi122_168655_object82503:head# 2024-03-20T09:26:26.524 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d387aea:::benchmark_data_smithi122_168655_object32539:head# 2024-03-20T09:26:26.525 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d392f2d:::benchmark_data_smithi122_168655_object51432:head# 2024-03-20T09:26:26.525 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d395e95:::benchmark_data_smithi122_168655_object36244:head# 2024-03-20T09:26:26.526 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d39d55b:::benchmark_data_smithi122_168655_object58996:head# 2024-03-20T09:26:26.526 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d3ab0b1:::benchmark_data_smithi122_168655_object30632:head# 2024-03-20T09:26:26.526 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d3c3ffc:::benchmark_data_smithi122_168655_object72383:head# 2024-03-20T09:26:26.527 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d3c5d36:::benchmark_data_smithi122_168655_object117921:head# 2024-03-20T09:26:26.527 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d3d42fd:::benchmark_data_smithi122_168655_object13353:head# 2024-03-20T09:26:26.528 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d3d4672:::benchmark_data_smithi122_168655_object56998:head# 2024-03-20T09:26:26.528 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d3e7d89:::benchmark_data_smithi122_168655_object106283:head# 2024-03-20T09:26:26.529 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d3f11dd:::benchmark_data_smithi122_168655_object78725:head# 2024-03-20T09:26:26.529 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d3f69f0:::benchmark_data_smithi122_168655_object49723:head# 2024-03-20T09:26:26.529 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d402ad1:::benchmark_data_smithi122_168655_object64076:head# 2024-03-20T09:26:26.530 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d41b19b:::benchmark_data_smithi122_168655_object35134:head# 2024-03-20T09:26:26.530 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d41fa2f:::benchmark_data_smithi122_168655_object55452:head# 2024-03-20T09:26:26.531 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d4249f8:::benchmark_data_smithi122_168655_object98091:head# 2024-03-20T09:26:26.531 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d42675f:::benchmark_data_smithi122_168655_object116273:head# 2024-03-20T09:26:26.531 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d42dde8:::benchmark_data_smithi122_168655_object52955:head# 2024-03-20T09:26:26.532 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d432441:::benchmark_data_smithi122_168655_object110473:head# 2024-03-20T09:26:26.532 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d43b9a4:::benchmark_data_smithi122_168655_object24350:head# 2024-03-20T09:26:26.533 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d448a03:::benchmark_data_smithi122_168655_object10351:head# 2024-03-20T09:26:26.533 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d449f3e:::benchmark_data_smithi122_168655_object106289:head# 2024-03-20T09:26:26.533 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d44c278:::benchmark_data_smithi122_168655_object35353:head# 2024-03-20T09:26:26.534 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d44e9ce:::benchmark_data_smithi122_168655_object75691:head# 2024-03-20T09:26:26.534 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d46022d:::benchmark_data_smithi122_168655_object44677:head# 2024-03-20T09:26:26.535 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d46bc30:::benchmark_data_smithi122_168655_object39279:head# 2024-03-20T09:26:26.535 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d46c117:::benchmark_data_smithi122_168655_object99373:head# 2024-03-20T09:26:26.536 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d476e3c:::benchmark_data_smithi122_168655_object83151:head# 2024-03-20T09:26:26.536 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d48c1a4:::benchmark_data_smithi122_168655_object67814:head# 2024-03-20T09:26:26.537 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d4be015:::benchmark_data_smithi122_168655_object93596:head# 2024-03-20T09:26:26.537 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d4bf8c0:::benchmark_data_smithi122_168655_object72819:head# 2024-03-20T09:26:26.537 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d4ca743:::benchmark_data_smithi122_168655_object37644:head# 2024-03-20T09:26:26.538 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d4cda70:::benchmark_data_smithi122_168655_object52146:head# 2024-03-20T09:26:26.539 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d4d2e52:::benchmark_data_smithi122_168655_object14086:head# 2024-03-20T09:26:26.539 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d4d5086:::benchmark_data_smithi122_168655_object115962:head# 2024-03-20T09:26:26.540 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d4dffbf:::benchmark_data_smithi122_168655_object17934:head# 2024-03-20T09:26:26.540 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d4e46b8:::benchmark_data_smithi122_168655_object90229:head# 2024-03-20T09:26:26.540 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d506513:::benchmark_data_smithi122_168655_object109063:head# 2024-03-20T09:26:26.541 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d513bd8:::benchmark_data_smithi122_168655_object44131:head# 2024-03-20T09:26:26.541 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d52020b:::benchmark_data_smithi122_168655_object56137:head# 2024-03-20T09:26:26.542 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d52703b:::benchmark_data_smithi122_168655_object80477:head# 2024-03-20T09:26:26.542 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d52f543:::benchmark_data_smithi122_168655_object104320:head# 2024-03-20T09:26:26.542 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d533f65:::benchmark_data_smithi122_168655_object118745:head# 2024-03-20T09:26:26.543 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d53f5f8:::benchmark_data_smithi122_168655_object96922:head# 2024-03-20T09:26:26.543 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d54544c:::benchmark_data_smithi122_168655_object82491:head# 2024-03-20T09:26:26.544 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d55958b:::benchmark_data_smithi122_168655_object85724:head# 2024-03-20T09:26:26.544 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d564a97:::benchmark_data_smithi122_168655_object69542:head# 2024-03-20T09:26:26.545 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d5671fb:::benchmark_data_smithi122_168655_object43257:head# 2024-03-20T09:26:26.545 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d57bdf3:::benchmark_data_smithi122_168655_object103393:head# 2024-03-20T09:26:26.546 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d57ef9a:::benchmark_data_smithi122_168655_object114121:head# 2024-03-20T09:26:26.546 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d59941c:::benchmark_data_smithi122_168655_object16590:head# 2024-03-20T09:26:26.546 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d5c51a8:::benchmark_data_smithi122_168655_object14910:head# 2024-03-20T09:26:26.547 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d5c72cb:::benchmark_data_smithi122_168655_object70451:head# 2024-03-20T09:26:26.547 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d5c8d88:::benchmark_data_smithi122_168655_object75431:head# 2024-03-20T09:26:26.548 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d5c95b3:::benchmark_data_smithi122_168655_object107217:head# 2024-03-20T09:26:26.548 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d5ce0c9:::benchmark_data_smithi122_168655_object12924:head# 2024-03-20T09:26:26.549 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d5ce98c:::benchmark_data_smithi122_168655_object33688:head# 2024-03-20T09:26:26.549 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d5d3e4c:::benchmark_data_smithi122_168655_object52414:head# 2024-03-20T09:26:26.550 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d5d93fc:::benchmark_data_smithi122_168655_object64250:head# 2024-03-20T09:26:26.550 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d5fe941:::benchmark_data_smithi122_168655_object103758:head# 2024-03-20T09:26:26.551 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d60a06d:::benchmark_data_smithi122_168655_object52127:head# 2024-03-20T09:26:26.551 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d60a7a9:::benchmark_data_smithi122_168655_object18147:head# 2024-03-20T09:26:26.552 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d614985:::benchmark_data_smithi122_168655_object62979:head# 2024-03-20T09:26:26.552 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d616605:::benchmark_data_smithi122_168655_object65141:head# 2024-03-20T09:26:26.553 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d624e24:::benchmark_data_smithi122_168655_object65363:head# 2024-03-20T09:26:26.553 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d62f6a8:::benchmark_data_smithi122_168655_object63255:head# 2024-03-20T09:26:26.554 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d63abf0:::benchmark_data_smithi122_168655_object97687:head# 2024-03-20T09:26:26.554 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d63fa20:::benchmark_data_smithi122_168655_object79440:head# 2024-03-20T09:26:26.555 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d642389:::benchmark_data_smithi122_168655_object24288:head# 2024-03-20T09:26:26.555 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d643c58:::benchmark_data_smithi122_168655_object56717:head# 2024-03-20T09:26:26.555 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d647218:::benchmark_data_smithi122_168655_object94873:head# 2024-03-20T09:26:26.556 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d64f2ff:::benchmark_data_smithi122_168655_object117379:head# 2024-03-20T09:26:26.557 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d650da9:::benchmark_data_smithi122_168655_object17922:head# 2024-03-20T09:26:26.557 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d659275:::benchmark_data_smithi122_168655_object109078:head# 2024-03-20T09:26:26.558 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d6609e1:::benchmark_data_smithi122_168655_object77374:head# 2024-03-20T09:26:26.558 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d6626b5:::benchmark_data_smithi122_168655_object28225:head# 2024-03-20T09:26:26.559 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d66cd01:::benchmark_data_smithi122_168655_object51267:head# 2024-03-20T09:26:26.559 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d66fb6f:::benchmark_data_smithi122_168655_object65391:head# 2024-03-20T09:26:26.560 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d67c5e2:::benchmark_data_smithi122_168655_object54309:head# 2024-03-20T09:26:26.560 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d67ce47:::benchmark_data_smithi122_168655_object88608:head# 2024-03-20T09:26:26.560 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d67ce95:::benchmark_data_smithi122_168655_object11106:head# 2024-03-20T09:26:26.561 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d68d0ff:::benchmark_data_smithi122_168655_object76954:head# 2024-03-20T09:26:26.561 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d6910c8:::benchmark_data_smithi122_168655_object94365:head# 2024-03-20T09:26:26.562 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d6b1f5f:::benchmark_data_smithi122_168655_object23903:head# 2024-03-20T09:26:26.562 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d6bb96a:::benchmark_data_smithi122_168655_object87492:head# 2024-03-20T09:26:26.563 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d6d77db:::benchmark_data_smithi122_168655_object114725:head# 2024-03-20T09:26:26.563 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d6e02e8:::benchmark_data_smithi122_168655_object109807:head# 2024-03-20T09:26:26.563 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d6e68c3:::benchmark_data_smithi122_168655_object93225:head# 2024-03-20T09:26:26.564 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d6e7213:::benchmark_data_smithi122_168655_object48142:head# 2024-03-20T09:26:26.564 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d6f0aee:::benchmark_data_smithi122_168655_object67559:head# 2024-03-20T09:26:26.564 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d6fa619:::benchmark_data_smithi122_168655_object51405:head# 2024-03-20T09:26:26.565 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d6fc662:::benchmark_data_smithi122_168655_object23412:head# 2024-03-20T09:26:26.565 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d7104a4:::benchmark_data_smithi122_168655_object53231:head# 2024-03-20T09:26:26.566 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d72e549:::benchmark_data_smithi122_168655_object106207:head# 2024-03-20T09:26:26.566 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d75f039:::benchmark_data_smithi122_168655_object64265:head# 2024-03-20T09:26:26.567 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d772e40:::benchmark_data_smithi122_168655_object102664:head# 2024-03-20T09:26:26.567 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d776eb7:::benchmark_data_smithi122_168655_object19028:head# 2024-03-20T09:26:26.568 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d77a4be:::benchmark_data_smithi122_168655_object87568:head# 2024-03-20T09:26:26.568 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d77e421:::benchmark_data_smithi122_168655_object66999:head# 2024-03-20T09:26:26.568 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d7941cb:::benchmark_data_smithi122_168655_object109267:head# 2024-03-20T09:26:26.569 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d79d647:::benchmark_data_smithi122_168655_object66099:head# 2024-03-20T09:26:26.569 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d7a2406:::benchmark_data_smithi122_168655_object22945:head# 2024-03-20T09:26:26.570 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d7a43eb:::benchmark_data_smithi122_168655_object112428:head# 2024-03-20T09:26:26.571 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d7b24a6:::benchmark_data_smithi122_168655_object61179:head# 2024-03-20T09:26:26.571 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d7b9aa8:::benchmark_data_smithi122_168655_object42895:head# 2024-03-20T09:26:26.571 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d7bec9e:::benchmark_data_smithi122_168655_object101650:head# 2024-03-20T09:26:26.572 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d7c2866:::benchmark_data_smithi122_168655_object15054:head# 2024-03-20T09:26:26.572 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d7cbe5f:::benchmark_data_smithi122_168655_object15770:head# 2024-03-20T09:26:26.573 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d7cf00b:::benchmark_data_smithi122_168655_object77082:head# 2024-03-20T09:26:26.573 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d7dedc3:::benchmark_data_smithi122_168655_object67923:head# 2024-03-20T09:26:26.573 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d7efb44:::benchmark_data_smithi122_168655_object24596:head# 2024-03-20T09:26:26.574 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d7f235b:::benchmark_data_smithi122_168655_object64472:head# 2024-03-20T09:26:26.574 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d8000e0:::benchmark_data_smithi122_168655_object32610:head# 2024-03-20T09:26:26.575 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d813934:::benchmark_data_smithi122_168655_object71602:head# 2024-03-20T09:26:26.575 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d826f7d:::benchmark_data_smithi122_168655_object100511:head# 2024-03-20T09:26:26.576 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d82942b:::benchmark_data_smithi122_168655_object76564:head# 2024-03-20T09:26:26.576 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d82985d:::benchmark_data_smithi122_168655_object39499:head# 2024-03-20T09:26:26.577 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d83b23f:::benchmark_data_smithi122_168655_object49683:head# 2024-03-20T09:26:26.577 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d844d09:::benchmark_data_smithi122_168655_object17566:head# 2024-03-20T09:26:26.577 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d852918:::benchmark_data_smithi122_168655_object75193:head# 2024-03-20T09:26:26.578 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d85647d:::benchmark_data_smithi122_168655_object101878:head# 2024-03-20T09:26:26.578 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d85e305:::benchmark_data_smithi122_168655_object109340:head# 2024-03-20T09:26:26.578 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d85eb16:::benchmark_data_smithi122_168655_object110765:head# 2024-03-20T09:26:26.579 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d86bf8f:::benchmark_data_smithi122_168655_object82463:head# 2024-03-20T09:26:26.579 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d86cba6:::benchmark_data_smithi122_168655_object54244:head# 2024-03-20T09:26:26.580 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d87162a:::benchmark_data_smithi122_168655_object55220:head# 2024-03-20T09:26:26.580 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d8781d0:::benchmark_data_smithi122_168655_object100344:head# 2024-03-20T09:26:26.583 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d87b650:::benchmark_data_smithi122_168655_object77005:head# 2024-03-20T09:26:26.584 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d88d4df:::benchmark_data_smithi122_168655_object40813:head# 2024-03-20T09:26:26.584 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d8a6a9c:::benchmark_data_smithi122_168655_object21853:head# 2024-03-20T09:26:26.584 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d8d559a:::benchmark_data_smithi122_168655_object68831:head# 2024-03-20T09:26:26.585 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d8e1003:::benchmark_data_smithi122_168655_object91252:head# 2024-03-20T09:26:26.585 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d8e9bf4:::benchmark_data_smithi122_168655_object113421:head# 2024-03-20T09:26:26.586 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d8ef2fd:::benchmark_data_smithi122_168655_object86638:head# 2024-03-20T09:26:26.586 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d8f3b7a:::benchmark_data_smithi122_168655_object88106:head# 2024-03-20T09:26:26.586 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d8f8375:::benchmark_data_smithi122_168655_object45585:head# 2024-03-20T09:26:26.587 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d902b22:::benchmark_data_smithi122_168655_object48146:head# 2024-03-20T09:26:26.587 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d904548:::benchmark_data_smithi122_168655_object108166:head# 2024-03-20T09:26:26.587 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d90634f:::benchmark_data_smithi122_168655_object109356:head# 2024-03-20T09:26:26.588 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d92c398:::benchmark_data_smithi122_168655_object114347:head# 2024-03-20T09:26:26.589 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d931553:::benchmark_data_smithi122_168655_object55802:head# 2024-03-20T09:26:26.589 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d93fc00:::benchmark_data_smithi122_168655_object24214:head# 2024-03-20T09:26:26.589 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d947f71:::benchmark_data_smithi122_168655_object69331:head# 2024-03-20T09:26:26.590 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d958128:::benchmark_data_smithi122_168655_object83971:head# 2024-03-20T09:26:26.590 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d9832e5:::benchmark_data_smithi122_168655_object15519:head# 2024-03-20T09:26:26.591 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d98453e:::benchmark_data_smithi122_168655_object116673:head# 2024-03-20T09:26:26.591 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d99e52f:::benchmark_data_smithi122_168655_object28066:head# 2024-03-20T09:26:26.591 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d9ab6b5:::benchmark_data_smithi122_168655_object88875:head# 2024-03-20T09:26:26.592 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d9b5fc7:::benchmark_data_smithi122_168655_object38154:head# 2024-03-20T09:26:26.592 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d9bc21a:::benchmark_data_smithi122_168655_object63348:head# 2024-03-20T09:26:26.593 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d9be452:::benchmark_data_smithi122_168655_object72941:head# 2024-03-20T09:26:26.593 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d9c0797:::benchmark_data_smithi122_168655_object67392:head# 2024-03-20T09:26:26.593 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d9c42b8:::benchmark_data_smithi122_168655_object68298:head# 2024-03-20T09:26:26.594 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d9cb872:::benchmark_data_smithi122_168655_object33356:head# 2024-03-20T09:26:26.594 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d9d4569:::benchmark_data_smithi122_168655_object64101:head# 2024-03-20T09:26:26.595 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d9e2b96:::benchmark_data_smithi122_168655_object93862:head# 2024-03-20T09:26:26.595 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d9e89bb:::benchmark_data_smithi122_168655_object33777:head# 2024-03-20T09:26:26.596 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d9eb03e:::benchmark_data_smithi122_168655_object44447:head# 2024-03-20T09:26:26.596 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d9ecd4b:::benchmark_data_smithi122_168655_object13257:head# 2024-03-20T09:26:26.597 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d9ee518:::benchmark_data_smithi122_168655_object61041:head# 2024-03-20T09:26:26.597 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8d9f1fd8:::benchmark_data_smithi122_168655_object16894:head# 2024-03-20T09:26:26.597 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8da096fd:::benchmark_data_smithi122_168655_object57061:head# 2024-03-20T09:26:26.598 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8da0c5c6:::benchmark_data_smithi122_168655_object61024:head# 2024-03-20T09:26:26.598 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8da0fb3b:::benchmark_data_smithi122_168655_object101479:head# 2024-03-20T09:26:26.599 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8da19c39:::benchmark_data_smithi122_168655_object48190:head# 2024-03-20T09:26:26.599 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8da1d7e3:::benchmark_data_smithi122_168655_object21893:head# 2024-03-20T09:26:26.600 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8da2432d:::benchmark_data_smithi122_168655_object98499:head# 2024-03-20T09:26:26.600 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8da24bb3:::benchmark_data_smithi122_168655_object119380:head# 2024-03-20T09:26:26.600 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8da2f527:::benchmark_data_smithi122_168655_object73896:head# 2024-03-20T09:26:26.601 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8da3e028:::benchmark_data_smithi122_168655_object86305:head# 2024-03-20T09:26:26.601 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8da3e0c5:::benchmark_data_smithi122_168655_object43141:head# 2024-03-20T09:26:26.602 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8da4765f:::benchmark_data_smithi122_168655_object105455:head# 2024-03-20T09:26:26.602 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8da49cc8:::benchmark_data_smithi122_168655_object90811:head# 2024-03-20T09:26:26.603 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8da56a43:::benchmark_data_smithi122_168655_object88240:head# 2024-03-20T09:26:26.603 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8da632c9:::benchmark_data_smithi122_168655_object14632:head# 2024-03-20T09:26:26.603 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8da6812e:::benchmark_data_smithi122_168655_object23808:head# 2024-03-20T09:26:26.604 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8da73147:::benchmark_data_smithi122_168655_object15771:head# 2024-03-20T09:26:26.605 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8da873b3:::benchmark_data_smithi122_168655_object46494:head# 2024-03-20T09:26:26.606 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8daa1eac:::benchmark_data_smithi122_168655_object113307:head# 2024-03-20T09:26:26.606 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8daa6312:::benchmark_data_smithi122_168655_object41663:head# 2024-03-20T09:26:26.607 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8daa690a:::benchmark_data_smithi122_168655_object58024:head# 2024-03-20T09:26:26.607 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8daa9eb9:::benchmark_data_smithi122_168655_object80268:head# 2024-03-20T09:26:26.608 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8daad4d8:::benchmark_data_smithi122_168655_object111197:head# 2024-03-20T09:26:26.608 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dab5052:::benchmark_data_smithi122_168655_object113838:head# 2024-03-20T09:26:26.609 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dab51a1:::benchmark_data_smithi122_168655_object108263:head# 2024-03-20T09:26:26.609 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dac779d:::benchmark_data_smithi122_168655_object33511:head# 2024-03-20T09:26:26.609 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dae1141:::benchmark_data_smithi122_168655_object97244:head# 2024-03-20T09:26:26.610 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8daed01b:::benchmark_data_smithi122_168655_object23029:head# 2024-03-20T09:26:26.610 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8daee22a:::benchmark_data_smithi122_168655_object24923:head# 2024-03-20T09:26:26.611 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8daee451:::benchmark_data_smithi122_168655_object100420:head# 2024-03-20T09:26:26.611 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8daf3153:::benchmark_data_smithi122_168655_object44827:head# 2024-03-20T09:26:26.612 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8db0ac65:::benchmark_data_smithi122_168655_object13879:head# 2024-03-20T09:26:26.612 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8db0adb3:::benchmark_data_smithi122_168655_object31891:head# 2024-03-20T09:26:26.612 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8db1345b:::benchmark_data_smithi122_168655_object69195:head# 2024-03-20T09:26:26.613 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8db213b8:::benchmark_data_smithi122_168655_object106493:head# 2024-03-20T09:26:26.613 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8db25546:::benchmark_data_smithi122_168655_object84077:head# 2024-03-20T09:26:26.614 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8db282be:::benchmark_data_smithi122_168655_object77719:head# 2024-03-20T09:26:26.614 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8db39366:::benchmark_data_smithi122_168655_object106656:head# 2024-03-20T09:26:26.615 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8db3ade6:::benchmark_data_smithi122_168655_object51808:head# 2024-03-20T09:26:26.615 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8db454de:::benchmark_data_smithi122_168655_object34098:head# 2024-03-20T09:26:26.615 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8db4af73:::benchmark_data_smithi122_168655_object48901:head# 2024-03-20T09:26:26.616 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8db4c268:::benchmark_data_smithi122_168655_object37204:head# 2024-03-20T09:26:26.616 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8db59b0c:::benchmark_data_smithi122_168655_object46783:head# 2024-03-20T09:26:26.616 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8db5a32c:::benchmark_data_smithi122_168655_object43616:head# 2024-03-20T09:26:26.617 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8db5ae7a:::benchmark_data_smithi122_168655_object94942:head# 2024-03-20T09:26:26.617 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8db645fe:::benchmark_data_smithi122_168655_object104167:head# 2024-03-20T09:26:26.618 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8db66923:::benchmark_data_smithi122_168655_object80997:head# 2024-03-20T09:26:26.618 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8db6e419:::benchmark_data_smithi122_168655_object96957:head# 2024-03-20T09:26:26.619 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8db6f53f:::benchmark_data_smithi122_168655_object91491:head# 2024-03-20T09:26:26.621 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8db7235b:::benchmark_data_smithi122_168655_object97123:head# 2024-03-20T09:26:26.622 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8db7876d:::benchmark_data_smithi122_168655_object34457:head# 2024-03-20T09:26:26.622 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8db8709b:::benchmark_data_smithi122_168655_object56837:head# 2024-03-20T09:26:26.623 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8db8e5d2:::benchmark_data_smithi122_168655_object105655:head# 2024-03-20T09:26:26.623 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8db8f7df:::benchmark_data_smithi122_168655_object70019:head# 2024-03-20T09:26:26.624 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8db93cd7:::benchmark_data_smithi122_168655_object64695:head# 2024-03-20T09:26:26.624 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dba0926:::benchmark_data_smithi122_168655_object53818:head# 2024-03-20T09:26:26.625 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dba0e88:::benchmark_data_smithi122_168655_object21211:head# 2024-03-20T09:26:26.625 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dbae245:::benchmark_data_smithi122_168655_object113088:head# 2024-03-20T09:26:26.625 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dc1659a:::benchmark_data_smithi122_168655_object103763:head# 2024-03-20T09:26:26.626 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dc1e066:::benchmark_data_smithi122_168655_object24614:head# 2024-03-20T09:26:26.626 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dc2049f:::benchmark_data_smithi122_168655_object49025:head# 2024-03-20T09:26:26.627 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dc42a7e:::benchmark_data_smithi122_168655_object63604:head# 2024-03-20T09:26:26.627 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dc63350:::benchmark_data_smithi122_168655_object95841:head# 2024-03-20T09:26:26.628 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dc6bc09:::benchmark_data_smithi122_168655_object53847:head# 2024-03-20T09:26:26.628 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dc72dbd:::benchmark_data_smithi122_168655_object60831:head# 2024-03-20T09:26:26.629 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dc7da3c:::benchmark_data_smithi122_168655_object40661:head# 2024-03-20T09:26:26.629 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dc8059c:::benchmark_data_smithi122_168655_object43714:head# 2024-03-20T09:26:26.630 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dc815ad:::benchmark_data_smithi122_168655_object35102:head# 2024-03-20T09:26:26.630 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dc8465f:::benchmark_data_smithi122_168655_object36140:head# 2024-03-20T09:26:26.631 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dc8988d:::benchmark_data_smithi122_168655_object15843:head# 2024-03-20T09:26:26.631 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dc91534:::benchmark_data_smithi122_168655_object71408:head# 2024-03-20T09:26:26.632 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dc96942:::benchmark_data_smithi122_168655_object42988:head# 2024-03-20T09:26:26.632 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dca9f5a:::benchmark_data_smithi122_168655_object104256:head# 2024-03-20T09:26:26.633 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dcbddb2:::benchmark_data_smithi122_168655_object55517:head# 2024-03-20T09:26:26.633 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dcc0b69:::benchmark_data_smithi122_168655_object85021:head# 2024-03-20T09:26:26.634 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dcc8fdb:::benchmark_data_smithi122_168655_object56751:head# 2024-03-20T09:26:26.636 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dcd08d2:::benchmark_data_smithi122_168655_object76546:head# 2024-03-20T09:26:26.637 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dce4e91:::benchmark_data_smithi122_168655_object104885:head# 2024-03-20T09:26:26.637 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dce604f:::benchmark_data_smithi122_168655_object45226:head# 2024-03-20T09:26:26.637 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dce7ac3:::benchmark_data_smithi122_168655_object14783:head# 2024-03-20T09:26:26.638 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dcee617:::benchmark_data_smithi122_168655_object52941:head# 2024-03-20T09:26:26.638 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dcf8050:::benchmark_data_smithi122_168655_object27836:head# 2024-03-20T09:26:26.639 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dcfc499:::benchmark_data_smithi122_168655_object102586:head# 2024-03-20T09:26:26.639 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dd0620f:::benchmark_data_smithi122_168655_object77308:head# 2024-03-20T09:26:26.640 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dd2699e:::benchmark_data_smithi122_168655_object16149:head# 2024-03-20T09:26:26.640 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dd2a72f:::benchmark_data_smithi122_168655_object54747:head# 2024-03-20T09:26:26.641 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dd46f7e:::benchmark_data_smithi122_168655_object20661:head# 2024-03-20T09:26:26.641 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dd56af0:::benchmark_data_smithi122_168655_object119695:head# 2024-03-20T09:26:26.642 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dd58858:::benchmark_data_smithi122_168655_object115691:head# 2024-03-20T09:26:26.642 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dd77f95:::benchmark_data_smithi122_168655_object107563:head# 2024-03-20T09:26:26.642 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dd826c8:::benchmark_data_smithi122_168655_object10211:head# 2024-03-20T09:26:26.643 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dd82c82:::benchmark_data_smithi122_168655_object50770:head# 2024-03-20T09:26:26.643 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dda07c1:::benchmark_data_smithi122_168655_object50870:head# 2024-03-20T09:26:26.644 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ddac286:::benchmark_data_smithi122_168655_object73485:head# 2024-03-20T09:26:26.644 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ddad7ad:::benchmark_data_smithi122_168655_object73974:head# 2024-03-20T09:26:26.645 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ddb6614:::benchmark_data_smithi122_168655_object11068:head# 2024-03-20T09:26:26.645 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ddb75c6:::benchmark_data_smithi122_168655_object57385:head# 2024-03-20T09:26:26.646 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ddcb4fc:::benchmark_data_smithi122_168655_object17043:head# 2024-03-20T09:26:26.646 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ddd1be3:::benchmark_data_smithi122_168655_object36489:head# 2024-03-20T09:26:26.647 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dddbf60:::benchmark_data_smithi122_168655_object41729:head# 2024-03-20T09:26:26.647 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ddfa7d0:::benchmark_data_smithi122_168655_object19215:head# 2024-03-20T09:26:26.648 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8de0b824:::benchmark_data_smithi122_168655_object39772:head# 2024-03-20T09:26:26.648 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8de272e4:::benchmark_data_smithi122_168655_object40279:head# 2024-03-20T09:26:26.649 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8de2c61c:::benchmark_data_smithi122_168655_object20126:head# 2024-03-20T09:26:26.649 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8de2f1df:::benchmark_data_smithi122_168655_object109190:head# 2024-03-20T09:26:26.650 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8de3494f:::benchmark_data_smithi122_168655_object98296:head# 2024-03-20T09:26:26.650 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8de36310:::benchmark_data_smithi122_168655_object35433:head# 2024-03-20T09:26:26.650 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8de40302:::benchmark_data_smithi122_168655_object41739:head# 2024-03-20T09:26:26.651 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8de415a1:::benchmark_data_smithi122_168655_object110052:head# 2024-03-20T09:26:26.651 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8de457de:::benchmark_data_smithi122_168655_object58138:head# 2024-03-20T09:26:26.652 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8de4bb10:::benchmark_data_smithi122_168655_object20275:head# 2024-03-20T09:26:26.652 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8de6f7c9:::benchmark_data_smithi122_168655_object21134:head# 2024-03-20T09:26:26.653 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8de74fba:::benchmark_data_smithi122_168655_object94819:head# 2024-03-20T09:26:26.654 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8de8f07a:::benchmark_data_smithi122_168655_object54827:head# 2024-03-20T09:26:26.655 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8de924c4:::benchmark_data_smithi122_168655_object19825:head# 2024-03-20T09:26:26.655 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8de9fba8:::benchmark_data_smithi122_168655_object84550:head# 2024-03-20T09:26:26.656 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8deb1ea8:::benchmark_data_smithi122_168655_object12502:head# 2024-03-20T09:26:26.656 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8deb4866:::benchmark_data_smithi122_168655_object38447:head# 2024-03-20T09:26:26.656 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dec4f67:::benchmark_data_smithi122_168655_object116617:head# 2024-03-20T09:26:26.657 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dec73b6:::benchmark_data_smithi122_168655_object43665:head# 2024-03-20T09:26:26.657 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8decd452:::benchmark_data_smithi122_168655_object21132:head# 2024-03-20T09:26:26.657 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dedd954:::benchmark_data_smithi122_168655_object47160:head# 2024-03-20T09:26:26.658 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dedf2e4:::benchmark_data_smithi122_168655_object52750:head# 2024-03-20T09:26:26.658 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8def1d68:::benchmark_data_smithi122_168655_object60544:head# 2024-03-20T09:26:26.659 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8def6667:::benchmark_data_smithi122_168655_object72846:head# 2024-03-20T09:26:26.659 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8df011b8:::benchmark_data_smithi122_168655_object116081:head# 2024-03-20T09:26:26.660 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8df18d96:::benchmark_data_smithi122_168655_object58608:head# 2024-03-20T09:26:26.660 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8df31638:::benchmark_data_smithi122_168655_object72339:head# 2024-03-20T09:26:26.660 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8df43e7b:::benchmark_data_smithi122_168655_object41479:head# 2024-03-20T09:26:26.661 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8df4467b:::benchmark_data_smithi122_168655_object104764:head# 2024-03-20T09:26:26.661 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8df4a552:::benchmark_data_smithi122_168655_object91764:head# 2024-03-20T09:26:26.662 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8df4ef85:::benchmark_data_smithi122_168655_object91996:head# 2024-03-20T09:26:26.662 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8df5199e:::benchmark_data_smithi122_168655_object37949:head# 2024-03-20T09:26:26.663 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8df6c2d5:::benchmark_data_smithi122_168655_object12315:head# 2024-03-20T09:26:26.663 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8df6f9f8:::benchmark_data_smithi122_168655_object100526:head# 2024-03-20T09:26:26.664 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8df854be:::benchmark_data_smithi122_168655_object102986:head# 2024-03-20T09:26:26.664 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8df8bf60:::benchmark_data_smithi122_168655_object101126:head# 2024-03-20T09:26:26.665 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8df9e25d:::benchmark_data_smithi122_168655_object97412:head# 2024-03-20T09:26:26.665 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dfad901:::benchmark_data_smithi122_168655_object106700:head# 2024-03-20T09:26:26.665 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dfaebbd:::benchmark_data_smithi122_168655_object110942:head# 2024-03-20T09:26:26.666 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dfb0576:::benchmark_data_smithi122_168655_object43819:head# 2024-03-20T09:26:26.666 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dfb0d49:::benchmark_data_smithi122_168655_object99092:head# 2024-03-20T09:26:26.669 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dfb4703:::benchmark_data_smithi122_168655_object34175:head# 2024-03-20T09:26:26.670 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dfc9ec4:::benchmark_data_smithi122_168655_object64420:head# 2024-03-20T09:26:26.670 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dfcda93:::benchmark_data_smithi122_168655_object66270:head# 2024-03-20T09:26:26.670 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dfd02d1:::benchmark_data_smithi122_168655_object27675:head# 2024-03-20T09:26:26.671 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dfdd411:::benchmark_data_smithi122_168655_object39545:head# 2024-03-20T09:26:26.671 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dfe0bd8:::benchmark_data_smithi122_168655_object40624:head# 2024-03-20T09:26:26.672 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dfe2ad1:::benchmark_data_smithi122_168655_object98260:head# 2024-03-20T09:26:26.672 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8dff1aad:::benchmark_data_smithi122_168655_object88860:head# 2024-03-20T09:26:26.673 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e0013f4:::benchmark_data_smithi122_168655_object97393:head# 2024-03-20T09:26:26.673 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e008267:::benchmark_data_smithi122_168655_object17197:head# 2024-03-20T09:26:26.673 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e012782:::benchmark_data_smithi122_168655_object119991:head# 2024-03-20T09:26:26.674 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e019ab5:::benchmark_data_smithi122_168655_object98763:head# 2024-03-20T09:26:26.674 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e01d9bd:::benchmark_data_smithi122_168655_object44974:head# 2024-03-20T09:26:26.675 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e01eede:::benchmark_data_smithi122_168655_object102583:head# 2024-03-20T09:26:26.675 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e0207e1:::benchmark_data_smithi122_168655_object16174:head# 2024-03-20T09:26:26.675 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e044739:::benchmark_data_smithi122_168655_object113808:head# 2024-03-20T09:26:26.676 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e070c29:::benchmark_data_smithi122_168655_object104061:head# 2024-03-20T09:26:26.676 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e07ce2c:::benchmark_data_smithi122_168655_object22953:head# 2024-03-20T09:26:26.677 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e085861:::benchmark_data_smithi122_168655_object59579:head# 2024-03-20T09:26:26.677 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e0916b4:::benchmark_data_smithi122_168655_object73174:head# 2024-03-20T09:26:26.678 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e09e238:::benchmark_data_smithi122_168655_object90346:head# 2024-03-20T09:26:26.678 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e0a7d1b:::benchmark_data_smithi122_168655_object55667:head# 2024-03-20T09:26:26.678 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e0af106:::benchmark_data_smithi122_168655_object99159:head# 2024-03-20T09:26:26.679 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e0afb75:::benchmark_data_smithi122_168655_object89111:head# 2024-03-20T09:26:26.679 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e0b2faa:::benchmark_data_smithi122_168655_object52568:head# 2024-03-20T09:26:26.680 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e0b3c22:::benchmark_data_smithi122_168655_object50512:head# 2024-03-20T09:26:26.680 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e0b846e:::benchmark_data_smithi122_168655_object91648:head# 2024-03-20T09:26:26.681 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e0c16fc:::benchmark_data_smithi122_168655_object72330:head# 2024-03-20T09:26:26.681 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e0cf91f:::benchmark_data_smithi122_168655_object50837:head# 2024-03-20T09:26:26.682 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e0d68b4:::benchmark_data_smithi122_168655_object80222:head# 2024-03-20T09:26:26.682 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e0df772:::benchmark_data_smithi122_168655_object98561:head# 2024-03-20T09:26:26.683 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e0e20ff:::benchmark_data_smithi122_168655_object43585:head# 2024-03-20T09:26:26.683 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e0e2f88:::benchmark_data_smithi122_168655_object52304:head# 2024-03-20T09:26:26.683 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e0e335e:::benchmark_data_smithi122_168655_object14054:head# 2024-03-20T09:26:26.684 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e0f2733:::benchmark_data_smithi122_168655_object108726:head# 2024-03-20T09:26:26.684 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e0f2914:::benchmark_data_smithi122_168655_object14617:head# 2024-03-20T09:26:26.685 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e0f304a:::benchmark_data_smithi122_168655_object34831:head# 2024-03-20T09:26:26.685 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e12cd15:::benchmark_data_smithi122_168655_object95439:head# 2024-03-20T09:26:26.686 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e133470:::benchmark_data_smithi122_168655_object85723:head# 2024-03-20T09:26:26.686 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e135620:::benchmark_data_smithi122_168655_object20424:head# 2024-03-20T09:26:26.686 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e13836d:::benchmark_data_smithi122_168655_object114702:head# 2024-03-20T09:26:26.687 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e1384e5:::benchmark_data_smithi122_168655_object41727:head# 2024-03-20T09:26:26.688 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e13f896:::benchmark_data_smithi122_168655_object87886:head# 2024-03-20T09:26:26.688 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e141581:::benchmark_data_smithi122_168655_object78783:head# 2024-03-20T09:26:26.688 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e1436a0:::benchmark_data_smithi122_168655_object35781:head# 2024-03-20T09:26:26.689 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e14edc5:::benchmark_data_smithi122_168655_object14263:head# 2024-03-20T09:26:26.689 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e1532fa:::benchmark_data_smithi122_168655_object77159:head# 2024-03-20T09:26:26.689 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e15cc99:::benchmark_data_smithi122_168655_object36235:head# 2024-03-20T09:26:26.691 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e162aa3:::benchmark_data_smithi122_168655_object61290:head# 2024-03-20T09:26:26.691 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e174300:::benchmark_data_smithi122_168655_object90030:head# 2024-03-20T09:26:26.691 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e174691:::benchmark_data_smithi122_168655_object88024:head# 2024-03-20T09:26:26.691 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e17fddb:::benchmark_data_smithi122_168655_object42747:head# 2024-03-20T09:26:26.692 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e18d4b4:::benchmark_data_smithi122_168655_object116556:head# 2024-03-20T09:26:26.692 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e190a35:::benchmark_data_smithi122_168655_object61213:head# 2024-03-20T09:26:26.692 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e1930e7:::benchmark_data_smithi122_168655_object30122:head# 2024-03-20T09:26:26.693 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e1a8e67:::benchmark_data_smithi122_168655_object70783:head# 2024-03-20T09:26:26.693 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e1ac104:::benchmark_data_smithi122_168655_object111608:head# 2024-03-20T09:26:26.694 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e1d0ec0:::benchmark_data_smithi122_168655_object38548:head# 2024-03-20T09:26:26.694 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e1e04ca:::benchmark_data_smithi122_168655_object11965:head# 2024-03-20T09:26:26.695 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e1e7ac0:::benchmark_data_smithi122_168655_object116872:head# 2024-03-20T09:26:26.695 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e1f50b6:::benchmark_data_smithi122_168655_object112502:head# 2024-03-20T09:26:26.695 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e1f8297:::benchmark_data_smithi122_168655_object41449:head# 2024-03-20T09:26:26.696 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e2180d3:::benchmark_data_smithi122_168655_object51588:head# 2024-03-20T09:26:26.697 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e220062:::benchmark_data_smithi122_168655_object69311:head# 2024-03-20T09:26:26.697 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e2283fb:::benchmark_data_smithi122_168655_object88592:head# 2024-03-20T09:26:26.697 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e22be6b:::benchmark_data_smithi122_168655_object109906:head# 2024-03-20T09:26:26.698 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e2453b1:::benchmark_data_smithi122_168655_object45495:head# 2024-03-20T09:26:26.698 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e262969:::benchmark_data_smithi122_168655_object29670:head# 2024-03-20T09:26:26.699 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e263f2d:::benchmark_data_smithi122_168655_object102960:head# 2024-03-20T09:26:26.699 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e26ef4f:::benchmark_data_smithi122_168655_object24758:head# 2024-03-20T09:26:26.700 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e28452e:::benchmark_data_smithi122_168655_object103274:head# 2024-03-20T09:26:26.700 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e288f89:::benchmark_data_smithi122_168655_object86298:head# 2024-03-20T09:26:26.701 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e29941e:::benchmark_data_smithi122_168655_object106809:head# 2024-03-20T09:26:26.701 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e2a2ebf:::benchmark_data_smithi122_168655_object58829:head# 2024-03-20T09:26:26.701 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e2aa6b9:::benchmark_data_smithi122_168655_object51045:head# 2024-03-20T09:26:26.702 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e2aae0d:::benchmark_data_smithi122_168655_object44678:head# 2024-03-20T09:26:26.711 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e2babeb:::benchmark_data_smithi122_168655_object48562:head# 2024-03-20T09:26:26.711 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e2be442:::benchmark_data_smithi122_168655_object113569:head# 2024-03-20T09:26:26.711 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e2c7d97:::benchmark_data_smithi122_168655_object109056:head# 2024-03-20T09:26:26.712 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e2deeb6:::benchmark_data_smithi122_168655_object112072:head# 2024-03-20T09:26:26.712 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e2e1461:::benchmark_data_smithi122_168655_object112418:head# 2024-03-20T09:26:26.712 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e306eef:::benchmark_data_smithi122_168655_object41605:head# 2024-03-20T09:26:26.712 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e310e36:::benchmark_data_smithi122_168655_object109089:head# 2024-03-20T09:26:26.712 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e31559f:::benchmark_data_smithi122_168655_object81651:head# 2024-03-20T09:26:26.712 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e31ba47:::benchmark_data_smithi122_168655_object98396:head# 2024-03-20T09:26:26.712 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e32646a:::benchmark_data_smithi122_168655_object20981:head# 2024-03-20T09:26:26.712 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e3277c4:::benchmark_data_smithi122_168655_object98834:head# 2024-03-20T09:26:26.712 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e32d39c:::benchmark_data_smithi122_168655_object55897:head# 2024-03-20T09:26:26.712 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e355e26:::benchmark_data_smithi122_168655_object105608:head# 2024-03-20T09:26:26.713 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e3624e7:::benchmark_data_smithi122_168655_object78287:head# 2024-03-20T09:26:26.713 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e364afa:::benchmark_data_smithi122_168655_object117382:head# 2024-03-20T09:26:26.713 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e374a3e:::benchmark_data_smithi122_168655_object114689:head# 2024-03-20T09:26:26.713 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e375f83:::benchmark_data_smithi122_168655_object108521:head# 2024-03-20T09:26:26.713 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e378bc6:::benchmark_data_smithi122_168655_object28839:head# 2024-03-20T09:26:26.713 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e37a622:::benchmark_data_smithi122_168655_object78321:head# 2024-03-20T09:26:26.713 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e39bb03:::benchmark_data_smithi122_168655_object42055:head# 2024-03-20T09:26:26.714 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e3c130b:::benchmark_data_smithi122_168655_object48828:head# 2024-03-20T09:26:26.714 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e3d2705:::benchmark_data_smithi122_168655_object36339:head# 2024-03-20T09:26:26.714 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e3da74c:::benchmark_data_smithi122_168655_object112937:head# 2024-03-20T09:26:26.715 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e3e1419:::benchmark_data_smithi122_168655_object31673:head# 2024-03-20T09:26:26.716 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e3e156c:::benchmark_data_smithi122_168655_object91537:head# 2024-03-20T09:26:26.716 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e3eaa3d:::benchmark_data_smithi122_168655_object82046:head# 2024-03-20T09:26:26.717 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e4016b8:::benchmark_data_smithi122_168655_object107999:head# 2024-03-20T09:26:26.717 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e407575:::benchmark_data_smithi122_168655_object91341:head# 2024-03-20T09:26:26.717 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e40a37d:::benchmark_data_smithi122_168655_object52591:head# 2024-03-20T09:26:26.718 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e4120c8:::benchmark_data_smithi122_168655_object45152:head# 2024-03-20T09:26:26.718 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e41afc1:::benchmark_data_smithi122_168655_object16478:head# 2024-03-20T09:26:26.718 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e424021:::benchmark_data_smithi122_168655_object61844:head# 2024-03-20T09:26:26.719 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e42947b:::benchmark_data_smithi122_168655_object18393:head# 2024-03-20T09:26:26.719 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e434a4b:::benchmark_data_smithi122_168655_object45870:head# 2024-03-20T09:26:26.722 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e43e156:::benchmark_data_smithi122_168655_object102807:head# 2024-03-20T09:26:26.722 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e44c592:::benchmark_data_smithi122_168655_object107765:head# 2024-03-20T09:26:26.722 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e45a4a1:::benchmark_data_smithi122_168655_object32182:head# 2024-03-20T09:26:26.723 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e45afef:::benchmark_data_smithi122_168655_object38220:head# 2024-03-20T09:26:26.724 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e45fd62:::benchmark_data_smithi122_168655_object54082:head# 2024-03-20T09:26:26.724 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e46bacb:::benchmark_data_smithi122_168655_object96470:head# 2024-03-20T09:26:26.725 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e46c73b:::benchmark_data_smithi122_168655_object77301:head# 2024-03-20T09:26:26.725 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e4719be:::benchmark_data_smithi122_168655_object106215:head# 2024-03-20T09:26:26.725 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e47ff20:::benchmark_data_smithi122_168655_object51865:head# 2024-03-20T09:26:26.727 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e48bae1:::benchmark_data_smithi122_168655_object88010:head# 2024-03-20T09:26:26.728 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e4951bb:::benchmark_data_smithi122_168655_object88373:head# 2024-03-20T09:26:26.728 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e4a6b6f:::benchmark_data_smithi122_168655_object46322:head# 2024-03-20T09:26:26.729 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e4b1e5c:::benchmark_data_smithi122_168655_object76206:head# 2024-03-20T09:26:26.729 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e4ba4ed:::benchmark_data_smithi122_168655_object26361:head# 2024-03-20T09:26:26.730 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e4c843c:::benchmark_data_smithi122_168655_object69539:head# 2024-03-20T09:26:26.730 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e4d9772:::benchmark_data_smithi122_168655_object107281:head# 2024-03-20T09:26:26.730 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e4f8dbf:::benchmark_data_smithi122_168655_object27539:head# 2024-03-20T09:26:26.731 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e4fbd5d:::benchmark_data_smithi122_168655_object22238:head# 2024-03-20T09:26:26.731 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e4fcd06:::benchmark_data_smithi122_168655_object105673:head# 2024-03-20T09:26:26.732 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e50c2fb:::benchmark_data_smithi122_168655_object60254:head# 2024-03-20T09:26:26.732 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e5115a0:::benchmark_data_smithi122_168655_object23989:head# 2024-03-20T09:26:26.733 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e511ff5:::benchmark_data_smithi122_168655_object90441:head# 2024-03-20T09:26:26.733 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e515ef5:::benchmark_data_smithi122_168655_object45573:head# 2024-03-20T09:26:26.734 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e52121d:::benchmark_data_smithi122_168655_object90188:head# 2024-03-20T09:26:26.734 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e521fbc:::benchmark_data_smithi122_168655_object27657:head# 2024-03-20T09:26:26.735 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e527fb4:::benchmark_data_smithi122_168655_object113945:head# 2024-03-20T09:26:26.735 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e54a568:::benchmark_data_smithi122_168655_object67677:head# 2024-03-20T09:26:26.736 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e556641:::benchmark_data_smithi122_168655_object44431:head# 2024-03-20T09:26:26.736 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e55c67d:::benchmark_data_smithi122_168655_object27923:head# 2024-03-20T09:26:26.737 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e55ff8f:::benchmark_data_smithi122_168655_object109071:head# 2024-03-20T09:26:26.737 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e55ffe7:::benchmark_data_smithi122_168655_object22787:head# 2024-03-20T09:26:26.737 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e562e22:::benchmark_data_smithi122_168655_object58122:head# 2024-03-20T09:26:26.738 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e5691a6:::benchmark_data_smithi122_168655_object49412:head# 2024-03-20T09:26:26.738 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e5691c0:::benchmark_data_smithi122_168655_object31101:head# 2024-03-20T09:26:26.739 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e56aa66:::benchmark_data_smithi122_168655_object86362:head# 2024-03-20T09:26:26.740 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e574726:::benchmark_data_smithi122_168655_object29040:head# 2024-03-20T09:26:26.742 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e5922b6:::benchmark_data_smithi122_168655_object66506:head# 2024-03-20T09:26:26.742 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e59a634:::benchmark_data_smithi122_168655_object50980:head# 2024-03-20T09:26:26.743 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e5a26d5:::benchmark_data_smithi122_168655_object13379:head# 2024-03-20T09:26:26.743 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e5b5d19:::benchmark_data_smithi122_168655_object64818:head# 2024-03-20T09:26:26.743 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e5c3ac6:::benchmark_data_smithi122_168655_object107287:head# 2024-03-20T09:26:26.744 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e5e0d04:::benchmark_data_smithi122_168655_object118605:head# 2024-03-20T09:26:26.744 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e5f9171:::benchmark_data_smithi122_168655_object89043:head# 2024-03-20T09:26:26.744 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e600ca4:::benchmark_data_smithi122_168655_object102442:head# 2024-03-20T09:26:26.745 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e602c3a:::benchmark_data_smithi122_168655_object78682:head# 2024-03-20T09:26:26.745 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e60ca72:::benchmark_data_smithi122_168655_object102331:head# 2024-03-20T09:26:26.746 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e60f5a4:::benchmark_data_smithi122_168655_object109302:head# 2024-03-20T09:26:26.746 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e62ed15:::benchmark_data_smithi122_168655_object15792:head# 2024-03-20T09:26:26.747 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e632d7c:::benchmark_data_smithi122_168655_object81942:head# 2024-03-20T09:26:26.747 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e633509:::benchmark_data_smithi122_168655_object40649:head# 2024-03-20T09:26:26.747 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e63a709:::benchmark_data_smithi122_168655_object116086:head# 2024-03-20T09:26:26.748 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e64055e:::benchmark_data_smithi122_168655_object18822:head# 2024-03-20T09:26:26.748 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e642a95:::benchmark_data_smithi122_168655_object102348:head# 2024-03-20T09:26:26.749 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e645a28:::benchmark_data_smithi122_168655_object35539:head# 2024-03-20T09:26:26.749 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e646d74:::benchmark_data_smithi122_168655_object12966:head# 2024-03-20T09:26:26.750 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e650730:::benchmark_data_smithi122_168655_object116554:head# 2024-03-20T09:26:26.750 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e655a3a:::benchmark_data_smithi122_168655_object30083:head# 2024-03-20T09:26:26.750 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e660dc1:::benchmark_data_smithi122_168655_object14649:head# 2024-03-20T09:26:26.751 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e66947c:::benchmark_data_smithi122_168655_object116485:head# 2024-03-20T09:26:26.751 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e675509:::benchmark_data_smithi122_168655_object44060:head# 2024-03-20T09:26:26.752 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e6776a1:::benchmark_data_smithi122_168655_object37164:head# 2024-03-20T09:26:26.752 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e67cc04:::benchmark_data_smithi122_168655_object47772:head# 2024-03-20T09:26:26.753 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e680381:::benchmark_data_smithi122_168655_object66360:head# 2024-03-20T09:26:26.753 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e686409:::benchmark_data_smithi122_168655_object17129:head# 2024-03-20T09:26:26.754 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e69ef50:::benchmark_data_smithi122_168655_object98521:head# 2024-03-20T09:26:26.755 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e6ad715:::benchmark_data_smithi122_168655_object105791:head# 2024-03-20T09:26:26.755 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e6c9390:::benchmark_data_smithi122_168655_object90920:head# 2024-03-20T09:26:26.755 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e6cbd39:::benchmark_data_smithi122_168655_object33141:head# 2024-03-20T09:26:26.756 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e6d3a4e:::benchmark_data_smithi122_168655_object115856:head# 2024-03-20T09:26:26.757 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e6d686f:::benchmark_data_smithi122_168655_object51462:head# 2024-03-20T09:26:26.757 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e6dd5b8:::benchmark_data_smithi122_168655_object45241:head# 2024-03-20T09:26:26.757 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e6e11fa:::benchmark_data_smithi122_168655_object73922:head# 2024-03-20T09:26:26.758 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e6e9a06:::benchmark_data_smithi122_168655_object108793:head# 2024-03-20T09:26:26.758 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e713486:::benchmark_data_smithi122_168655_object55371:head# 2024-03-20T09:26:26.758 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e717820:::benchmark_data_smithi122_168655_object46803:head# 2024-03-20T09:26:26.759 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e728248:::benchmark_data_smithi122_168655_object87086:head# 2024-03-20T09:26:26.759 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e741134:::benchmark_data_smithi122_168655_object48619:head# 2024-03-20T09:26:26.760 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e741cc1:::benchmark_data_smithi122_168655_object26320:head# 2024-03-20T09:26:26.760 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e744343:::benchmark_data_smithi122_168655_object24551:head# 2024-03-20T09:26:26.761 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e74b654:::benchmark_data_smithi122_168655_object11531:head# 2024-03-20T09:26:26.761 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e74cdf0:::benchmark_data_smithi122_168655_object61626:head# 2024-03-20T09:26:26.761 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e7584d6:::benchmark_data_smithi122_168655_object87416:head# 2024-03-20T09:26:26.762 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e77564a:::benchmark_data_smithi122_168655_object74548:head# 2024-03-20T09:26:26.762 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e778fc0:::benchmark_data_smithi122_168655_object35536:head# 2024-03-20T09:26:26.763 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e78209a:::benchmark_data_smithi122_168655_object55567:head# 2024-03-20T09:26:26.763 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e783580:::benchmark_data_smithi122_168655_object75605:head# 2024-03-20T09:26:26.763 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e78e74d:::benchmark_data_smithi122_168655_object43594:head# 2024-03-20T09:26:26.764 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e7a1796:::benchmark_data_smithi122_168655_object17214:head# 2024-03-20T09:26:26.764 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e7c1ccc:::benchmark_data_smithi122_168655_object88354:head# 2024-03-20T09:26:26.765 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e7c4b39:::benchmark_data_smithi122_168655_object66021:head# 2024-03-20T09:26:26.765 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e7c800b:::benchmark_data_smithi122_168655_object95447:head# 2024-03-20T09:26:26.766 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e80ed29:::benchmark_data_smithi122_168655_object84788:head# 2024-03-20T09:26:26.766 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e80fc0e:::benchmark_data_smithi122_168655_object56095:head# 2024-03-20T09:26:26.771 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e812a6c:::benchmark_data_smithi122_168655_object96184:head# 2024-03-20T09:26:26.771 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e8187d0:::benchmark_data_smithi122_168655_object51703:head# 2024-03-20T09:26:26.771 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e8188a0:::benchmark_data_smithi122_168655_object47348:head# 2024-03-20T09:26:26.771 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e818b00:::benchmark_data_smithi122_168655_object33113:head# 2024-03-20T09:26:26.771 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e823b89:::benchmark_data_smithi122_168655_object34279:head# 2024-03-20T09:26:26.771 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e825480:::benchmark_data_smithi122_168655_object88493:head# 2024-03-20T09:26:26.771 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e829df0:::benchmark_data_smithi122_168655_object101046:head# 2024-03-20T09:26:26.771 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e82adca:::benchmark_data_smithi122_168655_object44901:head# 2024-03-20T09:26:26.772 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e82f25c:::benchmark_data_smithi122_168655_object32773:head# 2024-03-20T09:26:26.772 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e82f72a:::benchmark_data_smithi122_168655_object57734:head# 2024-03-20T09:26:26.772 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e832c1e:::benchmark_data_smithi122_168655_object19318:head# 2024-03-20T09:26:26.773 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e83754e:::benchmark_data_smithi122_168655_object107876:head# 2024-03-20T09:26:26.773 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e8453dc:::benchmark_data_smithi122_168655_object35070:head# 2024-03-20T09:26:26.774 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e84ccdf:::benchmark_data_smithi122_168655_object46604:head# 2024-03-20T09:26:26.774 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e84d4bc:::benchmark_data_smithi122_168655_object104364:head# 2024-03-20T09:26:26.774 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e860904:::benchmark_data_smithi122_168655_object87635:head# 2024-03-20T09:26:26.774 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e873a76:::benchmark_data_smithi122_168655_object65794:head# 2024-03-20T09:26:26.775 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e877506:::benchmark_data_smithi122_168655_object69345:head# 2024-03-20T09:26:26.775 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e8789ca:::benchmark_data_smithi122_168655_object34014:head# 2024-03-20T09:26:26.775 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e883ccd:::benchmark_data_smithi122_168655_object52339:head# 2024-03-20T09:26:26.775 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e8b0514:::benchmark_data_smithi122_168655_object100887:head# 2024-03-20T09:26:26.778 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e8b66ed:::benchmark_data_smithi122_168655_object117481:head# 2024-03-20T09:26:26.778 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e8bbae2:::benchmark_data_smithi122_168655_object100411:head# 2024-03-20T09:26:26.779 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e8c5a7e:::benchmark_data_smithi122_168655_object116411:head# 2024-03-20T09:26:26.780 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e8c6b65:::benchmark_data_smithi122_168655_object78199:head# 2024-03-20T09:26:26.780 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e8c7f28:::benchmark_data_smithi122_168655_object41189:head# 2024-03-20T09:26:26.780 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e8d37ff:::benchmark_data_smithi122_168655_object37896:head# 2024-03-20T09:26:26.781 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e8e36a1:::benchmark_data_smithi122_168655_object23818:head# 2024-03-20T09:26:26.781 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e8f1179:::benchmark_data_smithi122_168655_object66766:head# 2024-03-20T09:26:26.782 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e9091e2:::benchmark_data_smithi122_168655_object33111:head# 2024-03-20T09:26:26.782 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e909e55:::benchmark_data_smithi122_168655_object54794:head# 2024-03-20T09:26:26.783 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e91215b:::benchmark_data_smithi122_168655_object48069:head# 2024-03-20T09:26:26.783 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e923a5f:::benchmark_data_smithi122_168655_object107937:head# 2024-03-20T09:26:26.783 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e93a020:::benchmark_data_smithi122_168655_object15622:head# 2024-03-20T09:26:26.784 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e95816a:::benchmark_data_smithi122_168655_object95379:head# 2024-03-20T09:26:26.784 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e962638:::benchmark_data_smithi122_168655_object93363:head# 2024-03-20T09:26:26.785 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e963a3c:::benchmark_data_smithi122_168655_object36021:head# 2024-03-20T09:26:26.785 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e9745c0:::benchmark_data_smithi122_168655_object78646:head# 2024-03-20T09:26:26.786 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e97ed29:::benchmark_data_smithi122_168655_object53668:head# 2024-03-20T09:26:26.786 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e9816a5:::benchmark_data_smithi122_168655_object114939:head# 2024-03-20T09:26:26.787 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e99da5d:::benchmark_data_smithi122_168655_object47124:head# 2024-03-20T09:26:26.787 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e9aa5b8:::benchmark_data_smithi122_168655_object21294:head# 2024-03-20T09:26:26.788 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e9ac70c:::benchmark_data_smithi122_168655_object63801:head# 2024-03-20T09:26:26.788 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e9d289e:::benchmark_data_smithi122_168655_object102754:head# 2024-03-20T09:26:26.788 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e9dcbc4:::benchmark_data_smithi122_168655_object21409:head# 2024-03-20T09:26:26.789 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e9e4f35:::benchmark_data_smithi122_168655_object48171:head# 2024-03-20T09:26:26.789 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e9f4ea7:::benchmark_data_smithi122_168655_object104581:head# 2024-03-20T09:26:26.790 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e9f6d12:::benchmark_data_smithi122_168655_object28817:head# 2024-03-20T09:26:26.790 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8e9f71b7:::benchmark_data_smithi122_168655_object98831:head# 2024-03-20T09:26:26.791 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ea07f8b:::benchmark_data_smithi122_168655_object116039:head# 2024-03-20T09:26:26.791 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ea210a9:::benchmark_data_smithi122_168655_object54131:head# 2024-03-20T09:26:26.792 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ea27404:::benchmark_data_smithi122_168655_object91465:head# 2024-03-20T09:26:26.792 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ea2bfe6:::benchmark_data_smithi122_168655_object99460:head# 2024-03-20T09:26:26.792 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ea2d160:::benchmark_data_smithi122_168655_object114991:head# 2024-03-20T09:26:26.793 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ea37ff1:::benchmark_data_smithi122_168655_object55031:head# 2024-03-20T09:26:26.793 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ea43da5:::benchmark_data_smithi122_168655_object117034:head# 2024-03-20T09:26:26.794 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ea4da1a:::benchmark_data_smithi122_168655_object71893:head# 2024-03-20T09:26:26.794 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ea613c0:::benchmark_data_smithi122_168655_object96293:head# 2024-03-20T09:26:26.795 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ea6270c:::benchmark_data_smithi122_168655_object47803:head# 2024-03-20T09:26:26.795 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ea77141:::benchmark_data_smithi122_168655_object31719:head# 2024-03-20T09:26:26.795 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ea859fd:::benchmark_data_smithi122_168655_object92958:head# 2024-03-20T09:26:26.796 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ea89489:::benchmark_data_smithi122_168655_object37626:head# 2024-03-20T09:26:26.796 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ea8ba6d:::benchmark_data_smithi122_168655_object87009:head# 2024-03-20T09:26:26.797 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ea9aeac:::benchmark_data_smithi122_168655_object86179:head# 2024-03-20T09:26:26.798 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eab9bdb:::benchmark_data_smithi122_168655_object64898:head# 2024-03-20T09:26:26.798 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eac9224:::benchmark_data_smithi122_168655_object61362:head# 2024-03-20T09:26:26.798 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ead89ba:::benchmark_data_smithi122_168655_object19954:head# 2024-03-20T09:26:26.799 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eadca23:::benchmark_data_smithi122_168655_object44941:head# 2024-03-20T09:26:26.799 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eaecf48:::benchmark_data_smithi122_168655_object49725:head# 2024-03-20T09:26:26.799 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eaef181:::benchmark_data_smithi122_168655_object101709:head# 2024-03-20T09:26:26.800 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eaf06c0:::benchmark_data_smithi122_168655_object118809:head# 2024-03-20T09:26:26.801 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eafb14e:::benchmark_data_smithi122_168655_object115875:head# 2024-03-20T09:26:26.801 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eafd6ab:::benchmark_data_smithi122_168655_object21447:head# 2024-03-20T09:26:26.802 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eb003c9:::benchmark_data_smithi122_168655_object75178:head# 2024-03-20T09:26:26.802 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eb0246e:::benchmark_data_smithi122_168655_object42672:head# 2024-03-20T09:26:26.802 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eb08819:::benchmark_data_smithi122_168655_object53683:head# 2024-03-20T09:26:26.804 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eb09a56:::benchmark_data_smithi122_168655_object39929:head# 2024-03-20T09:26:26.804 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eb0c938:::benchmark_data_smithi122_168655_object52572:head# 2024-03-20T09:26:26.805 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eb16924:::benchmark_data_smithi122_168655_object104743:head# 2024-03-20T09:26:26.805 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eb16abb:::benchmark_data_smithi122_168655_object115446:head# 2024-03-20T09:26:26.805 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eb1d9f6:::benchmark_data_smithi122_168655_object61040:head# 2024-03-20T09:26:26.806 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eb1e2e6:::benchmark_data_smithi122_168655_object115186:head# 2024-03-20T09:26:26.806 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eb1ed73:::benchmark_data_smithi122_168655_object28344:head# 2024-03-20T09:26:26.806 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eb34391:::benchmark_data_smithi122_168655_object69378:head# 2024-03-20T09:26:26.807 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eb37d12:::benchmark_data_smithi122_168655_object58203:head# 2024-03-20T09:26:26.807 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eb3b183:::benchmark_data_smithi122_168655_object54408:head# 2024-03-20T09:26:26.808 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eb4b39b:::benchmark_data_smithi122_168655_object69091:head# 2024-03-20T09:26:26.808 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eb52f85:::benchmark_data_smithi122_168655_object44392:head# 2024-03-20T09:26:26.809 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eb54f74:::benchmark_data_smithi122_168655_object58376:head# 2024-03-20T09:26:26.810 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eb55e32:::benchmark_data_smithi122_168655_object33962:head# 2024-03-20T09:26:26.810 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eb588cc:::benchmark_data_smithi122_168655_object93933:head# 2024-03-20T09:26:26.810 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eb5b728:::benchmark_data_smithi122_168655_object37403:head# 2024-03-20T09:26:26.811 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eb613c3:::benchmark_data_smithi122_168655_object85350:head# 2024-03-20T09:26:26.811 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eb62ec8:::benchmark_data_smithi122_168655_object13038:head# 2024-03-20T09:26:26.811 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eb67f0e:::benchmark_data_smithi122_168655_object71906:head# 2024-03-20T09:26:26.812 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eb6a7fd:::benchmark_data_smithi122_168655_object17559:head# 2024-03-20T09:26:26.813 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eb7644f:::benchmark_data_smithi122_168655_object83007:head# 2024-03-20T09:26:26.813 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eb78e33:::benchmark_data_smithi122_168655_object82323:head# 2024-03-20T09:26:26.813 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eb834d2:::benchmark_data_smithi122_168655_object73350:head# 2024-03-20T09:26:26.814 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eb84d1b:::benchmark_data_smithi122_168655_object27649:head# 2024-03-20T09:26:26.814 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eb8670e:::benchmark_data_smithi122_168655_object107664:head# 2024-03-20T09:26:26.815 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eb916bb:::benchmark_data_smithi122_168655_object106013:head# 2024-03-20T09:26:26.815 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eb97bd5:::benchmark_data_smithi122_168655_object63180:head# 2024-03-20T09:26:26.817 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ebb47ee:::benchmark_data_smithi122_168655_object63017:head# 2024-03-20T09:26:26.818 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ebc575d:::benchmark_data_smithi122_168655_object20215:head# 2024-03-20T09:26:26.818 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ebdba66:::benchmark_data_smithi122_168655_object39874:head# 2024-03-20T09:26:26.819 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ebdd194:::benchmark_data_smithi122_168655_object29625:head# 2024-03-20T09:26:26.819 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ebf263f:::benchmark_data_smithi122_168655_object46634:head# 2024-03-20T09:26:26.820 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ebfc5b3:::benchmark_data_smithi122_168655_object103869:head# 2024-03-20T09:26:26.820 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ec1746b:::benchmark_data_smithi122_168655_object41963:head# 2024-03-20T09:26:26.820 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ec1b068:::benchmark_data_smithi122_168655_object107102:head# 2024-03-20T09:26:26.821 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ec1c5c4:::benchmark_data_smithi122_168655_object87867:head# 2024-03-20T09:26:26.822 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ec26745:::benchmark_data_smithi122_168655_object76401:head# 2024-03-20T09:26:26.822 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ec27664:::benchmark_data_smithi122_168655_object68073:head# 2024-03-20T09:26:26.823 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ec2c60e:::benchmark_data_smithi122_168655_object34387:head# 2024-03-20T09:26:26.823 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ec2eac8:::benchmark_data_smithi122_168655_object77507:head# 2024-03-20T09:26:26.823 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ec31bbe:::benchmark_data_smithi122_168655_object48408:head# 2024-03-20T09:26:26.824 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ec3c303:::benchmark_data_smithi122_168655_object59599:head# 2024-03-20T09:26:26.824 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ec3d4aa:::benchmark_data_smithi122_168655_object99871:head# 2024-03-20T09:26:26.824 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ec412c9:::benchmark_data_smithi122_168655_object93885:head# 2024-03-20T09:26:26.824 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ec41f61:::benchmark_data_smithi122_168655_object100098:head# 2024-03-20T09:26:26.825 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ec44cf3:::benchmark_data_smithi122_168655_object28584:head# 2024-03-20T09:26:26.825 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ec49c1f:::benchmark_data_smithi122_168655_object48372:head# 2024-03-20T09:26:26.826 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ec57abb:::benchmark_data_smithi122_168655_object27271:head# 2024-03-20T09:26:26.826 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ec60f46:::benchmark_data_smithi122_168655_object107553:head# 2024-03-20T09:26:26.827 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ec62eb5:::benchmark_data_smithi122_168655_object18792:head# 2024-03-20T09:26:26.827 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ec6b5d6:::benchmark_data_smithi122_168655_object117875:head# 2024-03-20T09:26:26.828 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ec73efc:::benchmark_data_smithi122_168655_object39701:head# 2024-03-20T09:26:26.828 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ec7f964:::benchmark_data_smithi122_168655_object75770:head# 2024-03-20T09:26:26.829 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ec885e7:::benchmark_data_smithi122_168655_object88892:head# 2024-03-20T09:26:26.829 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eca2340:::benchmark_data_smithi122_168655_object42035:head# 2024-03-20T09:26:26.830 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eca5fe3:::benchmark_data_smithi122_168655_object93546:head# 2024-03-20T09:26:26.830 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ecb353c:::benchmark_data_smithi122_168655_object59422:head# 2024-03-20T09:26:26.831 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ecc0b54:::benchmark_data_smithi122_168655_object61533:head# 2024-03-20T09:26:26.831 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ecceb19:::benchmark_data_smithi122_168655_object59040:head# 2024-03-20T09:26:26.832 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eccfb86:::benchmark_data_smithi122_168655_object47536:head# 2024-03-20T09:26:26.832 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ecda99e:::benchmark_data_smithi122_168655_object108619:head# 2024-03-20T09:26:26.832 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ece02a0:::benchmark_data_smithi122_168655_object22763:head# 2024-03-20T09:26:26.833 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ece2680:::benchmark_data_smithi122_168655_object58490:head# 2024-03-20T09:26:26.833 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ecf4eec:::benchmark_data_smithi122_168655_object100693:head# 2024-03-20T09:26:26.834 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ecf9cee:::benchmark_data_smithi122_168655_object91954:head# 2024-03-20T09:26:26.834 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ecfc009:::benchmark_data_smithi122_168655_object51680:head# 2024-03-20T09:26:26.835 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ed0a9bf:::benchmark_data_smithi122_168655_object65901:head# 2024-03-20T09:26:26.835 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ed1259c:::benchmark_data_smithi122_168655_object93357:head# 2024-03-20T09:26:26.835 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ed315ff:::benchmark_data_smithi122_168655_object103046:head# 2024-03-20T09:26:26.836 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ed317de:::benchmark_data_smithi122_168655_object94482:head# 2024-03-20T09:26:26.836 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ed3ac17:::benchmark_data_smithi122_168655_object53607:head# 2024-03-20T09:26:26.837 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ed3bcea:::benchmark_data_smithi122_168655_object56125:head# 2024-03-20T09:26:26.837 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ed41199:::benchmark_data_smithi122_168655_object20806:head# 2024-03-20T09:26:26.838 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ed437b0:::benchmark_data_smithi122_168655_object31922:head# 2024-03-20T09:26:26.838 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ed519f5:::benchmark_data_smithi122_168655_object81715:head# 2024-03-20T09:26:26.839 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ed54f62:::benchmark_data_smithi122_168655_object88733:head# 2024-03-20T09:26:26.839 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ed607d2:::benchmark_data_smithi122_168655_object43315:head# 2024-03-20T09:26:26.840 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ed82ba4:::benchmark_data_smithi122_168655_object58375:head# 2024-03-20T09:26:26.840 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ed96b36:::benchmark_data_smithi122_168655_object24013:head# 2024-03-20T09:26:26.840 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8edab094:::benchmark_data_smithi122_168655_object40897:head# 2024-03-20T09:26:26.841 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8edb6475:::benchmark_data_smithi122_168655_object111582:head# 2024-03-20T09:26:26.841 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8edbaa2b:::benchmark_data_smithi122_168655_object40305:head# 2024-03-20T09:26:26.841 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8edc13ef:::benchmark_data_smithi122_168655_object70669:head# 2024-03-20T09:26:26.842 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8edc512f:::benchmark_data_smithi122_168655_object30654:head# 2024-03-20T09:26:26.842 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8edcb4b4:::benchmark_data_smithi122_168655_object35017:head# 2024-03-20T09:26:26.843 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8edd1665:::benchmark_data_smithi122_168655_object47644:head# 2024-03-20T09:26:26.843 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8edd2fb7:::benchmark_data_smithi122_168655_object101534:head# 2024-03-20T09:26:26.845 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8edfe336:::benchmark_data_smithi122_168655_object91939:head# 2024-03-20T09:26:26.845 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8edfe97f:::benchmark_data_smithi122_168655_object104433:head# 2024-03-20T09:26:26.846 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ee0465b:::benchmark_data_smithi122_168655_object94420:head# 2024-03-20T09:26:26.846 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ee140ad:::benchmark_data_smithi122_168655_object113811:head# 2024-03-20T09:26:26.847 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ee22e1a:::benchmark_data_smithi122_168655_object118336:head# 2024-03-20T09:26:26.847 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ee30db9:::benchmark_data_smithi122_168655_object40190:head# 2024-03-20T09:26:26.847 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ee31da8:::benchmark_data_smithi122_168655_object10739:head# 2024-03-20T09:26:26.847 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ee407cc:::benchmark_data_smithi122_168655_object30092:head# 2024-03-20T09:26:26.848 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ee4594a:::benchmark_data_smithi122_168655_object11485:head# 2024-03-20T09:26:26.848 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ee53d53:::benchmark_data_smithi122_168655_object59256:head# 2024-03-20T09:26:26.849 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ee608c2:::benchmark_data_smithi122_168655_object33311:head# 2024-03-20T09:26:26.849 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ee63f8e:::benchmark_data_smithi122_168655_object20843:head# 2024-03-20T09:26:26.849 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ee6dbac:::benchmark_data_smithi122_168655_object99199:head# 2024-03-20T09:26:26.850 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ee6f082:::benchmark_data_smithi122_168655_object20922:head# 2024-03-20T09:26:26.850 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ee81789:::benchmark_data_smithi122_168655_object67572:head# 2024-03-20T09:26:26.851 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eea6576:::benchmark_data_smithi122_168655_object13724:head# 2024-03-20T09:26:26.851 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eeaefba:::benchmark_data_smithi122_168655_object110619:head# 2024-03-20T09:26:26.852 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eec4501:::benchmark_data_smithi122_168655_object85215:head# 2024-03-20T09:26:26.852 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eec45ad:::benchmark_data_smithi122_168655_object118669:head# 2024-03-20T09:26:26.853 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eec7d00:::benchmark_data_smithi122_168655_object79369:head# 2024-03-20T09:26:26.853 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eec8257:::benchmark_data_smithi122_168655_object82429:head# 2024-03-20T09:26:26.853 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eee06dc:::benchmark_data_smithi122_168655_object74575:head# 2024-03-20T09:26:26.854 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eeec582:::benchmark_data_smithi122_168655_object61133:head# 2024-03-20T09:26:26.854 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eef11e9:::benchmark_data_smithi122_168655_object108511:head# 2024-03-20T09:26:26.855 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eef8515:::benchmark_data_smithi122_168655_object68745:head# 2024-03-20T09:26:26.855 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eefd506:::benchmark_data_smithi122_168655_object83383:head# 2024-03-20T09:26:26.855 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ef03355:::benchmark_data_smithi122_168655_object52855:head# 2024-03-20T09:26:26.856 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ef082c3:::benchmark_data_smithi122_168655_object72879:head# 2024-03-20T09:26:26.856 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ef0afa5:::benchmark_data_smithi122_168655_object97425:head# 2024-03-20T09:26:26.857 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ef0b5fa:::benchmark_data_smithi122_168655_object87814:head# 2024-03-20T09:26:26.857 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ef17a6d:::benchmark_data_smithi122_168655_object53421:head# 2024-03-20T09:26:26.858 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ef3ba60:::benchmark_data_smithi122_168655_object109687:head# 2024-03-20T09:26:26.858 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ef3bc9b:::benchmark_data_smithi122_168655_object56019:head# 2024-03-20T09:26:26.858 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ef40982:::benchmark_data_smithi122_168655_object87093:head# 2024-03-20T09:26:26.859 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ef514da:::benchmark_data_smithi122_168655_object109734:head# 2024-03-20T09:26:26.859 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ef6e2f1:::benchmark_data_smithi122_168655_object41667:head# 2024-03-20T09:26:26.860 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ef716ed:::benchmark_data_smithi122_168655_object76103:head# 2024-03-20T09:26:26.860 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ef8c525:::benchmark_data_smithi122_168655_object35587:head# 2024-03-20T09:26:26.861 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ef92e39:::benchmark_data_smithi122_168655_object36887:head# 2024-03-20T09:26:26.861 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ef9e42c:::benchmark_data_smithi122_168655_object84714:head# 2024-03-20T09:26:26.861 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8efa9757:::benchmark_data_smithi122_168655_object60772:head# 2024-03-20T09:26:26.862 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8efadf45:::benchmark_data_smithi122_168655_object48597:head# 2024-03-20T09:26:26.862 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8efae560:::benchmark_data_smithi122_168655_object45605:head# 2024-03-20T09:26:26.863 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8efb7200:::benchmark_data_smithi122_168655_object115849:head# 2024-03-20T09:26:26.863 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8efc19c6:::benchmark_data_smithi122_168655_object33590:head# 2024-03-20T09:26:26.864 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8efeb281:::benchmark_data_smithi122_168655_object17177:head# 2024-03-20T09:26:26.864 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8efef360:::benchmark_data_smithi122_168655_object23888:head# 2024-03-20T09:26:26.864 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8efef72b:::benchmark_data_smithi122_168655_object41395:head# 2024-03-20T09:26:26.865 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8eff34ab:::benchmark_data_smithi122_168655_object50906:head# 2024-03-20T09:26:26.865 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8effcbee:::benchmark_data_smithi122_168655_object19871:head# 2024-03-20T09:26:26.866 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f01429a:::benchmark_data_smithi122_168655_object114022:head# 2024-03-20T09:26:26.866 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f024041:::benchmark_data_smithi122_168655_object50762:head# 2024-03-20T09:26:26.867 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f0243ad:::benchmark_data_smithi122_168655_object21982:head# 2024-03-20T09:26:26.867 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f02b20c:::benchmark_data_smithi122_168655_object119814:head# 2024-03-20T09:26:26.867 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f02c6a5:::benchmark_data_smithi122_168655_object104191:head# 2024-03-20T09:26:26.868 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f03284f:::benchmark_data_smithi122_168655_object77746:head# 2024-03-20T09:26:26.868 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f037128:::benchmark_data_smithi122_168655_object19751:head# 2024-03-20T09:26:26.869 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f056c7b:::benchmark_data_smithi122_168655_object112640:head# 2024-03-20T09:26:26.871 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f05de6c:::benchmark_data_smithi122_168655_object93649:head# 2024-03-20T09:26:26.872 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f0612e5:::benchmark_data_smithi122_168655_object94853:head# 2024-03-20T09:26:26.872 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f064763:::benchmark_data_smithi122_168655_object44640:head# 2024-03-20T09:26:26.873 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f06500e:::benchmark_data_smithi122_168655_object23965:head# 2024-03-20T09:26:26.873 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f07123c:::benchmark_data_smithi122_168655_object76200:head# 2024-03-20T09:26:26.873 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f0893ef:::benchmark_data_smithi122_168655_object40621:head# 2024-03-20T09:26:26.874 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f0b12dc:::benchmark_data_smithi122_168655_object119142:head# 2024-03-20T09:26:26.874 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f0db6f6:::benchmark_data_smithi122_168655_object118435:head# 2024-03-20T09:26:26.875 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f0f95bd:::benchmark_data_smithi122_168655_object112344:head# 2024-03-20T09:26:26.875 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f105d59:::benchmark_data_smithi122_168655_object105430:head# 2024-03-20T09:26:26.875 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f10db7b:::benchmark_data_smithi122_168655_object29218:head# 2024-03-20T09:26:26.876 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f10e84f:::benchmark_data_smithi122_168655_object16325:head# 2024-03-20T09:26:26.876 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f1104a7:::benchmark_data_smithi122_168655_object88388:head# 2024-03-20T09:26:26.877 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f112bb8:::benchmark_data_smithi122_168655_object27227:head# 2024-03-20T09:26:26.877 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f12c012:::benchmark_data_smithi122_168655_object108668:head# 2024-03-20T09:26:26.878 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f133e33:::benchmark_data_smithi122_168655_object80868:head# 2024-03-20T09:26:26.879 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f138c8b:::benchmark_data_smithi122_168655_object49342:head# 2024-03-20T09:26:26.879 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f14f2ab:::benchmark_data_smithi122_168655_object108860:head# 2024-03-20T09:26:26.879 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f1507d2:::benchmark_data_smithi122_168655_object88967:head# 2024-03-20T09:26:26.880 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f152264:::benchmark_data_smithi122_168655_object41651:head# 2024-03-20T09:26:26.880 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f152b35:::benchmark_data_smithi122_168655_object93044:head# 2024-03-20T09:26:26.881 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f15f914:::benchmark_data_smithi122_168655_object80411:head# 2024-03-20T09:26:26.881 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f1603c0:::benchmark_data_smithi122_168655_object61596:head# 2024-03-20T09:26:26.882 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f166042:::benchmark_data_smithi122_168655_object11181:head# 2024-03-20T09:26:26.882 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f174bd6:::benchmark_data_smithi122_168655_object118475:head# 2024-03-20T09:26:26.883 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f17a882:::benchmark_data_smithi122_168655_object115260:head# 2024-03-20T09:26:26.883 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f17ef75:::benchmark_data_smithi122_168655_object56911:head# 2024-03-20T09:26:26.883 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f190f5b:::benchmark_data_smithi122_168655_object31197:head# 2024-03-20T09:26:26.884 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f191361:::benchmark_data_smithi122_168655_object31288:head# 2024-03-20T09:26:26.884 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f194d94:::benchmark_data_smithi122_168655_object47322:head# 2024-03-20T09:26:26.885 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f1a5a97:::benchmark_data_smithi122_168655_object99976:head# 2024-03-20T09:26:26.885 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f1b3966:::benchmark_data_smithi122_168655_object42474:head# 2024-03-20T09:26:26.886 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f1b8452:::benchmark_data_smithi122_168655_object17507:head# 2024-03-20T09:26:26.886 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f1b88bb:::benchmark_data_smithi122_168655_object27130:head# 2024-03-20T09:26:26.887 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f1bd868:::benchmark_data_smithi122_168655_object35181:head# 2024-03-20T09:26:26.887 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f1c78ed:::benchmark_data_smithi122_168655_object88876:head# 2024-03-20T09:26:26.887 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f1cabf5:::benchmark_data_smithi122_168655_object39285:head# 2024-03-20T09:26:26.888 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f1d5733:::benchmark_data_smithi122_168655_object66141:head# 2024-03-20T09:26:26.888 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f1d9b25:::benchmark_data_smithi122_168655_object89457:head# 2024-03-20T09:26:26.889 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f1e7844:::benchmark_data_smithi122_168655_object52829:head# 2024-03-20T09:26:26.889 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f20b49a:::benchmark_data_smithi122_168655_object109842:head# 2024-03-20T09:26:26.889 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f21e835:::benchmark_data_smithi122_168655_object76133:head# 2024-03-20T09:26:26.890 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f220f6b:::benchmark_data_smithi122_168655_object13439:head# 2024-03-20T09:26:26.890 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f22dba9:::benchmark_data_smithi122_168655_object107581:head# 2024-03-20T09:26:26.891 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f23e9f5:::benchmark_data_smithi122_168655_object12318:head# 2024-03-20T09:26:26.891 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f23efa7:::benchmark_data_smithi122_168655_object95283:head# 2024-03-20T09:26:26.892 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f248dce:::benchmark_data_smithi122_168655_object60790:head# 2024-03-20T09:26:26.892 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f249aeb:::benchmark_data_smithi122_168655_object10597:head# 2024-03-20T09:26:26.893 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f254309:::benchmark_data_smithi122_168655_object67632:head# 2024-03-20T09:26:26.893 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f25d2a9:::benchmark_data_smithi122_168655_object63202:head# 2024-03-20T09:26:26.893 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f25f095:::benchmark_data_smithi122_168655_object101003:head# 2024-03-20T09:26:26.894 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f2630b1:::benchmark_data_smithi122_168655_object25262:head# 2024-03-20T09:26:26.895 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f26bd2d:::benchmark_data_smithi122_168655_object60522:head# 2024-03-20T09:26:26.895 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f26fc40:::benchmark_data_smithi122_168655_object88051:head# 2024-03-20T09:26:26.896 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f272005:::benchmark_data_smithi122_168655_object103189:head# 2024-03-20T09:26:26.896 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f272d4f:::benchmark_data_smithi122_168655_object90448:head# 2024-03-20T09:26:26.896 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f274167:::benchmark_data_smithi122_168655_object57225:head# 2024-03-20T09:26:26.897 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f279559:::benchmark_data_smithi122_168655_object26660:head# 2024-03-20T09:26:26.897 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f28093f:::benchmark_data_smithi122_168655_object33786:head# 2024-03-20T09:26:26.898 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f28c797:::benchmark_data_smithi122_168655_object19607:head# 2024-03-20T09:26:26.900 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f290300:::benchmark_data_smithi122_168655_object64947:head# 2024-03-20T09:26:26.900 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f2906da:::benchmark_data_smithi122_168655_object72562:head# 2024-03-20T09:26:26.900 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f29610b:::benchmark_data_smithi122_168655_object112780:head# 2024-03-20T09:26:26.900 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f296134:::benchmark_data_smithi122_168655_object16222:head# 2024-03-20T09:26:26.900 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f29bd6e:::benchmark_data_smithi122_168655_object35858:head# 2024-03-20T09:26:26.901 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f29ee48:::benchmark_data_smithi122_168655_object90957:head# 2024-03-20T09:26:26.901 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f2a069b:::benchmark_data_smithi122_168655_object67210:head# 2024-03-20T09:26:26.902 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f2b1561:::benchmark_data_smithi122_168655_object104568:head# 2024-03-20T09:26:26.902 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f2b1cd5:::benchmark_data_smithi122_168655_object84608:head# 2024-03-20T09:26:26.902 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f2b6c0f:::benchmark_data_smithi122_168655_object94522:head# 2024-03-20T09:26:26.903 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f2bb4bc:::benchmark_data_smithi122_168655_object50656:head# 2024-03-20T09:26:26.903 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f2c0fb4:::benchmark_data_smithi122_168655_object64753:head# 2024-03-20T09:26:26.904 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f2c9e79:::benchmark_data_smithi122_168655_object27279:head# 2024-03-20T09:26:26.904 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f2e3999:::benchmark_data_smithi122_168655_object50484:head# 2024-03-20T09:26:26.904 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f2ecfde:::benchmark_data_smithi122_168655_object22802:head# 2024-03-20T09:26:26.905 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f2fe6ed:::benchmark_data_smithi122_168655_object108749:head# 2024-03-20T09:26:26.905 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f30a862:::benchmark_data_smithi122_168655_object59884:head# 2024-03-20T09:26:26.906 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f317b7d:::benchmark_data_smithi122_168655_object117218:head# 2024-03-20T09:26:26.906 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f317db2:::benchmark_data_smithi122_168655_object54204:head# 2024-03-20T09:26:26.907 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f31afd7:::benchmark_data_smithi122_168655_object67527:head# 2024-03-20T09:26:26.907 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f31b96a:::benchmark_data_smithi122_168655_object29014:head# 2024-03-20T09:26:26.907 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f31d580:::benchmark_data_smithi122_168655_object11864:head# 2024-03-20T09:26:26.908 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f324554:::benchmark_data_smithi122_168655_object29070:head# 2024-03-20T09:26:26.908 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f333d39:::benchmark_data_smithi122_168655_object36194:head# 2024-03-20T09:26:26.908 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f339fa5:::benchmark_data_smithi122_168655_object11452:head# 2024-03-20T09:26:26.909 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f34009f:::benchmark_data_smithi122_168655_object118563:head# 2024-03-20T09:26:26.909 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f341619:::benchmark_data_smithi122_168655_object74124:head# 2024-03-20T09:26:26.910 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f34de5b:::benchmark_data_smithi122_168655_object32776:head# 2024-03-20T09:26:26.910 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f354dee:::benchmark_data_smithi122_168655_object86242:head# 2024-03-20T09:26:26.910 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f357685:::benchmark_data_smithi122_168655_object118359:head# 2024-03-20T09:26:26.911 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f35d77e:::benchmark_data_smithi122_168655_object107255:head# 2024-03-20T09:26:26.911 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f3660f9:::benchmark_data_smithi122_168655_object100105:head# 2024-03-20T09:26:26.912 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f36b357:::benchmark_data_smithi122_168655_object112808:head# 2024-03-20T09:26:26.912 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f36c03a:::benchmark_data_smithi122_168655_object23584:head# 2024-03-20T09:26:26.913 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f36ef60:::benchmark_data_smithi122_168655_object32633:head# 2024-03-20T09:26:26.913 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f36fe50:::benchmark_data_smithi122_168655_object95329:head# 2024-03-20T09:26:26.914 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f381967:::benchmark_data_smithi122_168655_object116291:head# 2024-03-20T09:26:26.914 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f38cc5a:::benchmark_data_smithi122_168655_object87470:head# 2024-03-20T09:26:26.914 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f39886b:::benchmark_data_smithi122_168655_object98202:head# 2024-03-20T09:26:26.915 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f398de7:::benchmark_data_smithi122_168655_object54036:head# 2024-03-20T09:26:26.915 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f39de13:::benchmark_data_smithi122_168655_object18538:head# 2024-03-20T09:26:26.916 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f3ab0f8:::benchmark_data_smithi122_168655_object86266:head# 2024-03-20T09:26:26.916 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f3c144e:::benchmark_data_smithi122_168655_object105606:head# 2024-03-20T09:26:26.916 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f3c5f72:::benchmark_data_smithi122_168655_object39573:head# 2024-03-20T09:26:26.917 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f3cc8a6:::benchmark_data_smithi122_168655_object69656:head# 2024-03-20T09:26:26.917 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f3d9119:::benchmark_data_smithi122_168655_object10687:head# 2024-03-20T09:26:26.918 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f3da224:::benchmark_data_smithi122_168655_object96962:head# 2024-03-20T09:26:26.918 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f3e54e3:::benchmark_data_smithi122_168655_object38989:head# 2024-03-20T09:26:26.919 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f3fc9b8:::benchmark_data_smithi122_168655_object40115:head# 2024-03-20T09:26:26.919 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f411633:::benchmark_data_smithi122_168655_object31051:head# 2024-03-20T09:26:26.919 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f4149fc:::benchmark_data_smithi122_168655_object84557:head# 2024-03-20T09:26:26.920 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f415963:::benchmark_data_smithi122_168655_object62454:head# 2024-03-20T09:26:26.920 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f416f7f:::benchmark_data_smithi122_168655_object30077:head# 2024-03-20T09:26:26.921 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f43087d:::benchmark_data_smithi122_168655_object34951:head# 2024-03-20T09:26:26.921 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f430b0a:::benchmark_data_smithi122_168655_object18552:head# 2024-03-20T09:26:26.922 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f43d8bc:::benchmark_data_smithi122_168655_object64766:head# 2024-03-20T09:26:26.922 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f44319c:::benchmark_data_smithi122_168655_object116236:head# 2024-03-20T09:26:26.923 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f4486fd:::benchmark_data_smithi122_168655_object64946:head# 2024-03-20T09:26:26.923 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f44ac77:::benchmark_data_smithi122_168655_object109102:head# 2024-03-20T09:26:26.923 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f451031:::benchmark_data_smithi122_168655_object80727:head# 2024-03-20T09:26:26.924 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f452ee5:::benchmark_data_smithi122_168655_object21889:head# 2024-03-20T09:26:26.924 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f47e09b:::benchmark_data_smithi122_168655_object74973:head# 2024-03-20T09:26:26.925 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f48a435:::benchmark_data_smithi122_168655_object40389:head# 2024-03-20T09:26:26.925 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f48a70b:::benchmark_data_smithi122_168655_object119688:head# 2024-03-20T09:26:26.926 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f48c9cc:::benchmark_data_smithi122_168655_object113616:head# 2024-03-20T09:26:26.926 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f48d426:::benchmark_data_smithi122_168655_object10705:head# 2024-03-20T09:26:26.926 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f48d5d3:::benchmark_data_smithi122_168655_object30970:head# 2024-03-20T09:26:26.927 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f4958e2:::benchmark_data_smithi122_168655_object112246:head# 2024-03-20T09:26:26.927 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f4966d9:::benchmark_data_smithi122_168655_object118119:head# 2024-03-20T09:26:26.927 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f499e72:::benchmark_data_smithi122_168655_object117293:head# 2024-03-20T09:26:26.928 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f49d559:::benchmark_data_smithi122_168655_object83026:head# 2024-03-20T09:26:26.928 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f49d9ba:::benchmark_data_smithi122_168655_object26507:head# 2024-03-20T09:26:26.929 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f4aa566:::benchmark_data_smithi122_168655_object57239:head# 2024-03-20T09:26:26.929 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f4bd140:::benchmark_data_smithi122_168655_object96307:head# 2024-03-20T09:26:26.930 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f4c2026:::benchmark_data_smithi122_168655_object91161:head# 2024-03-20T09:26:26.930 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f4cd7be:::benchmark_data_smithi122_168655_object116293:head# 2024-03-20T09:26:26.930 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f4cdb67:::benchmark_data_smithi122_168655_object14879:head# 2024-03-20T09:26:26.931 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f4d3234:::benchmark_data_smithi122_168655_object29527:head# 2024-03-20T09:26:26.931 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f4ded33:::benchmark_data_smithi122_168655_object62416:head# 2024-03-20T09:26:26.932 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f4e2b93:::benchmark_data_smithi122_168655_object66584:head# 2024-03-20T09:26:26.932 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f4ea64f:::benchmark_data_smithi122_168655_object27942:head# 2024-03-20T09:26:26.932 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f4ef865:::benchmark_data_smithi122_168655_object47769:head# 2024-03-20T09:26:26.933 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f4f0658:::benchmark_data_smithi122_168655_object38719:head# 2024-03-20T09:26:26.933 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f4f7d06:::benchmark_data_smithi122_168655_object68216:head# 2024-03-20T09:26:26.934 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f504200:::benchmark_data_smithi122_168655_object58928:head# 2024-03-20T09:26:26.934 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f506424:::benchmark_data_smithi122_168655_object76787:head# 2024-03-20T09:26:26.935 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f51327e:::benchmark_data_smithi122_168655_object33346:head# 2024-03-20T09:26:26.935 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f51b088:::benchmark_data_smithi122_168655_object118207:head# 2024-03-20T09:26:26.936 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f51b889:::benchmark_data_smithi122_168655_object50276:head# 2024-03-20T09:26:26.936 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f52e930:::benchmark_data_smithi122_168655_object32070:head# 2024-03-20T09:26:26.936 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f531d8d:::benchmark_data_smithi122_168655_object61194:head# 2024-03-20T09:26:26.937 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f5321ec:::benchmark_data_smithi122_168655_object113106:head# 2024-03-20T09:26:26.937 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f54dec1:::benchmark_data_smithi122_168655_object23948:head# 2024-03-20T09:26:26.938 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f553d51:::benchmark_data_smithi122_168655_object70092:head# 2024-03-20T09:26:26.938 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f553eb3:::benchmark_data_smithi122_168655_object61899:head# 2024-03-20T09:26:26.938 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f5586bf:::benchmark_data_smithi122_168655_object55852:head# 2024-03-20T09:26:26.939 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f55a833:::benchmark_data_smithi122_168655_object66394:head# 2024-03-20T09:26:26.939 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f55bbd9:::benchmark_data_smithi122_168655_object33125:head# 2024-03-20T09:26:26.940 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f55c866:::benchmark_data_smithi122_168655_object83181:head# 2024-03-20T09:26:26.940 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f56389c:::benchmark_data_smithi122_168655_object105064:head# 2024-03-20T09:26:26.940 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f56569b:::benchmark_data_smithi122_168655_object68019:head# 2024-03-20T09:26:26.941 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f56613b:::benchmark_data_smithi122_168655_object86741:head# 2024-03-20T09:26:26.942 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f57362a:::benchmark_data_smithi122_168655_object52155:head# 2024-03-20T09:26:26.942 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f57a99b:::benchmark_data_smithi122_168655_object104871:head# 2024-03-20T09:26:26.942 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f57eba7:::benchmark_data_smithi122_168655_object10960:head# 2024-03-20T09:26:26.942 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f597f44:::benchmark_data_smithi122_168655_object67643:head# 2024-03-20T09:26:26.943 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f5afd9b:::benchmark_data_smithi122_168655_object15621:head# 2024-03-20T09:26:26.943 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f5b5783:::benchmark_data_smithi122_168655_object16794:head# 2024-03-20T09:26:26.944 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f5c4c90:::benchmark_data_smithi122_168655_object34315:head# 2024-03-20T09:26:26.944 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f5c6066:::benchmark_data_smithi122_168655_object43196:head# 2024-03-20T09:26:26.945 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f5d6206:::benchmark_data_smithi122_168655_object24891:head# 2024-03-20T09:26:26.945 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f5e8e9e:::benchmark_data_smithi122_168655_object69580:head# 2024-03-20T09:26:26.945 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f5fa851:::benchmark_data_smithi122_168655_object92182:head# 2024-03-20T09:26:26.946 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f60cb3f:::benchmark_data_smithi122_168655_object34180:head# 2024-03-20T09:26:26.946 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f613d3e:::benchmark_data_smithi122_168655_object68050:head# 2024-03-20T09:26:26.947 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f626c80:::benchmark_data_smithi122_168655_object119338:head# 2024-03-20T09:26:26.947 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f62b39c:::benchmark_data_smithi122_168655_object76120:head# 2024-03-20T09:26:26.948 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f63009e:::benchmark_data_smithi122_168655_object15314:head# 2024-03-20T09:26:26.948 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f637b2c:::benchmark_data_smithi122_168655_object75182:head# 2024-03-20T09:26:26.948 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f640554:::benchmark_data_smithi122_168655_object113525:head# 2024-03-20T09:26:26.949 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f642a27:::benchmark_data_smithi122_168655_object113554:head# 2024-03-20T09:26:26.949 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f649044:::benchmark_data_smithi122_168655_object111071:head# 2024-03-20T09:26:26.950 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f65dfcf:::benchmark_data_smithi122_168655_object73063:head# 2024-03-20T09:26:26.950 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f66a595:::benchmark_data_smithi122_168655_object31166:head# 2024-03-20T09:26:26.951 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f66c729:::benchmark_data_smithi122_168655_object54885:head# 2024-03-20T09:26:26.951 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f683d4b:::benchmark_data_smithi122_168655_object11655:head# 2024-03-20T09:26:26.951 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f68ef91:::benchmark_data_smithi122_168655_object54287:head# 2024-03-20T09:26:26.952 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f69326e:::benchmark_data_smithi122_168655_object36881:head# 2024-03-20T09:26:26.952 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f694321:::benchmark_data_smithi122_168655_object103081:head# 2024-03-20T09:26:26.953 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f6b3330:::benchmark_data_smithi122_168655_object17988:head# 2024-03-20T09:26:26.953 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f6bc41b:::benchmark_data_smithi122_168655_object81389:head# 2024-03-20T09:26:26.953 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f6bec67:::benchmark_data_smithi122_168655_object94566:head# 2024-03-20T09:26:26.954 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f6c0181:::benchmark_data_smithi122_168655_object74621:head# 2024-03-20T09:26:26.954 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f6da32a:::benchmark_data_smithi122_168655_object108867:head# 2024-03-20T09:26:26.955 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f6dd826:::benchmark_data_smithi122_168655_object27319:head# 2024-03-20T09:26:26.955 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f6e0caa:::benchmark_data_smithi122_168655_object76653:head# 2024-03-20T09:26:26.956 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f6ec473:::benchmark_data_smithi122_168655_object56653:head# 2024-03-20T09:26:26.956 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f6ecfa0:::benchmark_data_smithi122_168655_object42065:head# 2024-03-20T09:26:26.957 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f6f5598:::benchmark_data_smithi122_168655_object77535:head# 2024-03-20T09:26:26.957 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f6fd305:::benchmark_data_smithi122_168655_object81847:head# 2024-03-20T09:26:26.957 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f701870:::benchmark_data_smithi122_168655_object104578:head# 2024-03-20T09:26:26.958 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f710a1d:::benchmark_data_smithi122_168655_object94927:head# 2024-03-20T09:26:26.958 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f7155e5:::benchmark_data_smithi122_168655_object16429:head# 2024-03-20T09:26:26.959 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f726b12:::benchmark_data_smithi122_168655_object25972:head# 2024-03-20T09:26:26.959 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f729ad7:::benchmark_data_smithi122_168655_object96285:head# 2024-03-20T09:26:26.959 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f73d7ea:::benchmark_data_smithi122_168655_object67796:head# 2024-03-20T09:26:26.960 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f7409a6:::benchmark_data_smithi122_168655_object23711:head# 2024-03-20T09:26:26.960 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f7423cc:::benchmark_data_smithi122_168655_object117557:head# 2024-03-20T09:26:26.961 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f7574d7:::benchmark_data_smithi122_168655_object81447:head# 2024-03-20T09:26:26.961 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f758997:::benchmark_data_smithi122_168655_object117885:head# 2024-03-20T09:26:26.961 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f7628a4:::benchmark_data_smithi122_168655_object100209:head# 2024-03-20T09:26:26.962 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f762933:::benchmark_data_smithi122_168655_object28856:head# 2024-03-20T09:26:26.962 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f772475:::benchmark_data_smithi122_168655_object48912:head# 2024-03-20T09:26:26.963 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f77c7db:::benchmark_data_smithi122_168655_object24271:head# 2024-03-20T09:26:26.963 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f7819dc:::benchmark_data_smithi122_168655_object80952:head# 2024-03-20T09:26:26.963 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f786041:::benchmark_data_smithi122_168655_object38792:head# 2024-03-20T09:26:26.964 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f79f9a4:::benchmark_data_smithi122_168655_object107922:head# 2024-03-20T09:26:26.965 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f7a2be5:::benchmark_data_smithi122_168655_object54801:head# 2024-03-20T09:26:26.965 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f7ad740:::benchmark_data_smithi122_168655_object94458:head# 2024-03-20T09:26:26.965 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f7b498e:::benchmark_data_smithi122_168655_object77334:head# 2024-03-20T09:26:26.966 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f7bd6e1:::benchmark_data_smithi122_168655_object60528:head# 2024-03-20T09:26:26.966 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f7ccd95:::benchmark_data_smithi122_168655_object118325:head# 2024-03-20T09:26:26.967 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f7cdc76:::benchmark_data_smithi122_168655_object16324:head# 2024-03-20T09:26:26.967 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f7ce402:::benchmark_data_smithi122_168655_object11034:head# 2024-03-20T09:26:26.967 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f7f331a:::benchmark_data_smithi122_168655_object31570:head# 2024-03-20T09:26:26.968 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f7fb213:::benchmark_data_smithi122_168655_object16902:head# 2024-03-20T09:26:26.968 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f7fe15a:::benchmark_data_smithi122_168655_object32493:head# 2024-03-20T09:26:26.969 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f800b89:::benchmark_data_smithi122_168655_object46851:head# 2024-03-20T09:26:26.969 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f825ea7:::benchmark_data_smithi122_168655_object63166:head# 2024-03-20T09:26:26.970 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f82bbb4:::benchmark_data_smithi122_168655_object111606:head# 2024-03-20T09:26:26.970 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f82d7f0:::benchmark_data_smithi122_168655_object19613:head# 2024-03-20T09:26:26.970 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f830e91:::benchmark_data_smithi122_168655_object118985:head# 2024-03-20T09:26:26.974 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f851933:::benchmark_data_smithi122_168655_object25884:head# 2024-03-20T09:26:26.975 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f856360:::benchmark_data_smithi122_168655_object91408:head# 2024-03-20T09:26:26.975 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f85b392:::benchmark_data_smithi122_168655_object55466:head# 2024-03-20T09:26:26.975 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f8674ed:::benchmark_data_smithi122_168655_object67040:head# 2024-03-20T09:26:26.975 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f86b8cc:::benchmark_data_smithi122_168655_object111210:head# 2024-03-20T09:26:26.975 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f873144:::benchmark_data_smithi122_168655_object100422:head# 2024-03-20T09:26:26.975 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f8764a4:::benchmark_data_smithi122_168655_object73848:head# 2024-03-20T09:26:26.975 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f88c6fe:::benchmark_data_smithi122_168655_object111572:head# 2024-03-20T09:26:26.975 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f895731:::benchmark_data_smithi122_168655_object109497:head# 2024-03-20T09:26:26.976 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f896c4a:::benchmark_data_smithi122_168655_object60939:head# 2024-03-20T09:26:26.976 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f8a7925:::benchmark_data_smithi122_168655_object23364:head# 2024-03-20T09:26:26.976 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f8bac1e:::benchmark_data_smithi122_168655_object77135:head# 2024-03-20T09:26:26.977 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f8cd73f:::benchmark_data_smithi122_168655_object45154:head# 2024-03-20T09:26:26.977 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f8d14bb:::benchmark_data_smithi122_168655_object60328:head# 2024-03-20T09:26:26.977 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f8d1780:::benchmark_data_smithi122_168655_object83540:head# 2024-03-20T09:26:26.978 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f8de908:::benchmark_data_smithi122_168655_object70924:head# 2024-03-20T09:26:26.978 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f905110:::benchmark_data_smithi122_168655_object56349:head# 2024-03-20T09:26:26.979 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f907397:::benchmark_data_smithi122_168655_object51950:head# 2024-03-20T09:26:26.979 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f9098d1:::benchmark_data_smithi122_168655_object52294:head# 2024-03-20T09:26:26.980 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f911fbf:::benchmark_data_smithi122_168655_object52050:head# 2024-03-20T09:26:26.980 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f912238:::benchmark_data_smithi122_168655_object53015:head# 2024-03-20T09:26:26.981 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f91b9ec:::benchmark_data_smithi122_168655_object40076:head# 2024-03-20T09:26:26.981 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f91f0e6:::benchmark_data_smithi122_168655_object30412:head# 2024-03-20T09:26:26.981 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f924142:::benchmark_data_smithi122_168655_object82336:head# 2024-03-20T09:26:26.982 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f92572d:::benchmark_data_smithi122_168655_object90337:head# 2024-03-20T09:26:26.982 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f92f2b6:::benchmark_data_smithi122_168655_object109751:head# 2024-03-20T09:26:26.983 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f958716:::benchmark_data_smithi122_168655_object44492:head# 2024-03-20T09:26:26.983 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f95b789:::benchmark_data_smithi122_168655_object12839:head# 2024-03-20T09:26:26.984 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f9640de:::benchmark_data_smithi122_168655_object92924:head# 2024-03-20T09:26:26.984 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f974cfb:::benchmark_data_smithi122_168655_object73206:head# 2024-03-20T09:26:26.985 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f9795b4:::benchmark_data_smithi122_168655_object24389:head# 2024-03-20T09:26:26.985 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f97a4e0:::benchmark_data_smithi122_168655_object113027:head# 2024-03-20T09:26:26.986 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f987471:::benchmark_data_smithi122_168655_object92038:head# 2024-03-20T09:26:26.986 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f98769b:::benchmark_data_smithi122_168655_object85810:head# 2024-03-20T09:26:26.986 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f98d698:::benchmark_data_smithi122_168655_object119746:head# 2024-03-20T09:26:26.987 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f9a572c:::benchmark_data_smithi122_168655_object106613:head# 2024-03-20T09:26:26.988 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f9a76ed:::benchmark_data_smithi122_168655_object52928:head# 2024-03-20T09:26:26.988 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f9adf5a:::benchmark_data_smithi122_168655_object119082:head# 2024-03-20T09:26:26.988 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f9bd12d:::benchmark_data_smithi122_168655_object59446:head# 2024-03-20T09:26:26.989 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f9bf524:::benchmark_data_smithi122_168655_object19083:head# 2024-03-20T09:26:26.989 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f9c670a:::benchmark_data_smithi122_168655_object52792:head# 2024-03-20T09:26:26.990 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f9cb961:::benchmark_data_smithi122_168655_object82200:head# 2024-03-20T09:26:26.990 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f9ce8ed:::benchmark_data_smithi122_168655_object40636:head# 2024-03-20T09:26:26.991 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f9e0b77:::benchmark_data_smithi122_168655_object108488:head# 2024-03-20T09:26:26.991 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f9e248b:::benchmark_data_smithi122_168655_object43355:head# 2024-03-20T09:26:26.991 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f9e9dfb:::benchmark_data_smithi122_168655_object47981:head# 2024-03-20T09:26:26.992 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f9f70c6:::benchmark_data_smithi122_168655_object23357:head# 2024-03-20T09:26:26.992 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f9f829a:::benchmark_data_smithi122_168655_object14556:head# 2024-03-20T09:26:26.993 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8f9fc941:::benchmark_data_smithi122_168655_object86486:head# 2024-03-20T09:26:26.993 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fa04347:::benchmark_data_smithi122_168655_object97834:head# 2024-03-20T09:26:26.994 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fa20c65:::benchmark_data_smithi122_168655_object78568:head# 2024-03-20T09:26:26.994 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fa26c49:::benchmark_data_smithi122_168655_object69156:head# 2024-03-20T09:26:26.994 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fa2710c:::benchmark_data_smithi122_168655_object103865:head# 2024-03-20T09:26:26.995 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fa312db:::benchmark_data_smithi122_168655_object42647:head# 2024-03-20T09:26:26.995 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fa36723:::benchmark_data_smithi122_168655_object46297:head# 2024-03-20T09:26:26.995 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fa4249b:::benchmark_data_smithi122_168655_object65362:head# 2024-03-20T09:26:26.996 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fa4d455:::benchmark_data_smithi122_168655_object116167:head# 2024-03-20T09:26:26.996 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fa4f5b4:::benchmark_data_smithi122_168655_object85310:head# 2024-03-20T09:26:26.997 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fa535d6:::benchmark_data_smithi122_168655_object117391:head# 2024-03-20T09:26:26.997 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fa5cbcf:::benchmark_data_smithi122_168655_object111370:head# 2024-03-20T09:26:26.997 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fa6298b:::benchmark_data_smithi122_168655_object51474:head# 2024-03-20T09:26:26.998 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fa84440:::benchmark_data_smithi122_168655_object15853:head# 2024-03-20T09:26:26.998 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fa94a7b:::benchmark_data_smithi122_168655_object44680:head# 2024-03-20T09:26:26.999 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8faa7520:::benchmark_data_smithi122_168655_object56175:head# 2024-03-20T09:26:26.999 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fab2f6d:::benchmark_data_smithi122_168655_object13774:head# 2024-03-20T09:26:27.000 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fab3f7e:::benchmark_data_smithi122_168655_object89322:head# 2024-03-20T09:26:27.000 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fabb85f:::benchmark_data_smithi122_168655_object109822:head# 2024-03-20T09:26:27.001 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fac1316:::benchmark_data_smithi122_168655_object67483:head# 2024-03-20T09:26:27.001 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8facaf3b:::benchmark_data_smithi122_168655_object37558:head# 2024-03-20T09:26:27.002 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8facbe04:::benchmark_data_smithi122_168655_object91498:head# 2024-03-20T09:26:27.002 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fad3cbd:::benchmark_data_smithi122_168655_object81961:head# 2024-03-20T09:26:27.003 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fad6c8b:::benchmark_data_smithi122_168655_object18004:head# 2024-03-20T09:26:27.003 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fadff07:::benchmark_data_smithi122_168655_object91367:head# 2024-03-20T09:26:27.004 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8faec760:::benchmark_data_smithi122_168655_object17274:head# 2024-03-20T09:26:27.004 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8faf39c1:::benchmark_data_smithi122_168655_object13671:head# 2024-03-20T09:26:27.005 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8faf5b23:::benchmark_data_smithi122_168655_object79534:head# 2024-03-20T09:26:27.005 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fb14711:::benchmark_data_smithi122_168655_object101291:head# 2024-03-20T09:26:27.005 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fb158e5:::benchmark_data_smithi122_168655_object89435:head# 2024-03-20T09:26:27.006 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fb16fee:::benchmark_data_smithi122_168655_object68460:head# 2024-03-20T09:26:27.006 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fb1e6b8:::benchmark_data_smithi122_168655_object92561:head# 2024-03-20T09:26:27.007 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fb2f393:::benchmark_data_smithi122_168655_object17403:head# 2024-03-20T09:26:27.007 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fb30311:::benchmark_data_smithi122_168655_object65575:head# 2024-03-20T09:26:27.007 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fb317cb:::benchmark_data_smithi122_168655_object91502:head# 2024-03-20T09:26:27.008 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fb48d1e:::benchmark_data_smithi122_168655_object45082:head# 2024-03-20T09:26:27.008 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fb4ef1b:::benchmark_data_smithi122_168655_object42336:head# 2024-03-20T09:26:27.009 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fb67cab:::benchmark_data_smithi122_168655_object112202:head# 2024-03-20T09:26:27.009 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fb725b7:::benchmark_data_smithi122_168655_object115263:head# 2024-03-20T09:26:27.009 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fb7cd32:::benchmark_data_smithi122_168655_object109692:head# 2024-03-20T09:26:27.010 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fb99317:::benchmark_data_smithi122_168655_object63872:head# 2024-03-20T09:26:27.010 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fbb12db:::benchmark_data_smithi122_168655_object101732:head# 2024-03-20T09:26:27.011 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fbb8c81:::benchmark_data_smithi122_168655_object55158:head# 2024-03-20T09:26:27.011 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fbbda9d:::benchmark_data_smithi122_168655_object67451:head# 2024-03-20T09:26:27.011 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fbc78ff:::benchmark_data_smithi122_168655_object29722:head# 2024-03-20T09:26:27.012 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fbd6eac:::benchmark_data_smithi122_168655_object40510:head# 2024-03-20T09:26:27.012 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fbd81b3:::benchmark_data_smithi122_168655_object25395:head# 2024-03-20T09:26:27.013 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fbdb05d:::benchmark_data_smithi122_168655_object52303:head# 2024-03-20T09:26:27.013 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fbe3cbb:::benchmark_data_smithi122_168655_object47717:head# 2024-03-20T09:26:27.013 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fbfc93c:::benchmark_data_smithi122_168655_object102996:head# 2024-03-20T09:26:27.014 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fc043cf:::benchmark_data_smithi122_168655_object92784:head# 2024-03-20T09:26:27.014 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fc0692e:::benchmark_data_smithi122_168655_object38104:head# 2024-03-20T09:26:27.015 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fc0b470:::benchmark_data_smithi122_168655_object65779:head# 2024-03-20T09:26:27.015 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fc10be6:::benchmark_data_smithi122_168655_object23338:head# 2024-03-20T09:26:27.015 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fc124b9:::benchmark_data_smithi122_168655_object47066:head# 2024-03-20T09:26:27.016 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fc1b5db:::benchmark_data_smithi122_168655_object81677:head# 2024-03-20T09:26:27.016 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fc25581:::benchmark_data_smithi122_168655_object71276:head# 2024-03-20T09:26:27.017 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fc38b3d:::benchmark_data_smithi122_168655_object101936:head# 2024-03-20T09:26:27.017 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fc3ab7e:::benchmark_data_smithi122_168655_object69771:head# 2024-03-20T09:26:27.018 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fc51e21:::benchmark_data_smithi122_168655_object32629:head# 2024-03-20T09:26:27.018 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fc56d23:::benchmark_data_smithi122_168655_object100027:head# 2024-03-20T09:26:27.019 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fc6ea32:::benchmark_data_smithi122_168655_object98433:head# 2024-03-20T09:26:27.019 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fc703ee:::benchmark_data_smithi122_168655_object101031:head# 2024-03-20T09:26:27.020 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fc7eec0:::benchmark_data_smithi122_168655_object21826:head# 2024-03-20T09:26:27.020 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fc80823:::benchmark_data_smithi122_168655_object45047:head# 2024-03-20T09:26:27.021 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fc8c731:::benchmark_data_smithi122_168655_object105381:head# 2024-03-20T09:26:27.021 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fc913db:::benchmark_data_smithi122_168655_object31862:head# 2024-03-20T09:26:27.022 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fca1cc9:::benchmark_data_smithi122_168655_object45049:head# 2024-03-20T09:26:27.022 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fca2472:::benchmark_data_smithi122_168655_object88423:head# 2024-03-20T09:26:27.022 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fca8863:::benchmark_data_smithi122_168655_object24059:head# 2024-03-20T09:26:27.023 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fcb8ee0:::benchmark_data_smithi122_168655_object97813:head# 2024-03-20T09:26:27.023 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fcbe957:::benchmark_data_smithi122_168655_object113830:head# 2024-03-20T09:26:27.024 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fcc38ed:::benchmark_data_smithi122_168655_object38174:head# 2024-03-20T09:26:27.024 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fcc3f25:::benchmark_data_smithi122_168655_object12255:head# 2024-03-20T09:26:27.025 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fcc63ff:::benchmark_data_smithi122_168655_object35990:head# 2024-03-20T09:26:27.025 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fccb855:::benchmark_data_smithi122_168655_object55745:head# 2024-03-20T09:26:27.025 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fcd00bf:::benchmark_data_smithi122_168655_object32281:head# 2024-03-20T09:26:27.026 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fcd8edb:::benchmark_data_smithi122_168655_object102109:head# 2024-03-20T09:26:27.026 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fcde81d:::benchmark_data_smithi122_168655_object47388:head# 2024-03-20T09:26:27.027 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fce0691:::benchmark_data_smithi122_168655_object111462:head# 2024-03-20T09:26:27.027 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fce9140:::benchmark_data_smithi122_168655_object32682:head# 2024-03-20T09:26:27.027 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fceea71:::benchmark_data_smithi122_168655_object81052:head# 2024-03-20T09:26:27.028 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fcf7be3:::benchmark_data_smithi122_168655_object86997:head# 2024-03-20T09:26:27.028 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fd05bab:::benchmark_data_smithi122_168655_object72381:head# 2024-03-20T09:26:27.028 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fd38271:::benchmark_data_smithi122_168655_object114610:head# 2024-03-20T09:26:27.029 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fd3b61e:::benchmark_data_smithi122_168655_object59071:head# 2024-03-20T09:26:27.029 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fd3f2f7:::benchmark_data_smithi122_168655_object20201:head# 2024-03-20T09:26:27.030 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fd49f9a:::benchmark_data_smithi122_168655_object65591:head# 2024-03-20T09:26:27.030 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fd4bddc:::benchmark_data_smithi122_168655_object91799:head# 2024-03-20T09:26:27.031 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fd53cd1:::benchmark_data_smithi122_168655_object97548:head# 2024-03-20T09:26:27.031 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fd56db6:::benchmark_data_smithi122_168655_object77807:head# 2024-03-20T09:26:27.032 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fd59657:::benchmark_data_smithi122_168655_object31294:head# 2024-03-20T09:26:27.032 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fd5c0cb:::benchmark_data_smithi122_168655_object66508:head# 2024-03-20T09:26:27.032 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fd6ec49:::benchmark_data_smithi122_168655_object64095:head# 2024-03-20T09:26:27.033 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fd6edeb:::benchmark_data_smithi122_168655_object88040:head# 2024-03-20T09:26:27.033 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fda219d:::benchmark_data_smithi122_168655_object13834:head# 2024-03-20T09:26:27.034 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fda8c26:::benchmark_data_smithi122_168655_object83418:head# 2024-03-20T09:26:27.034 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fdb6932:::benchmark_data_smithi122_168655_object16582:head# 2024-03-20T09:26:27.035 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fdba900:::benchmark_data_smithi122_168655_object62551:head# 2024-03-20T09:26:27.035 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fdca405:::benchmark_data_smithi122_168655_object76384:head# 2024-03-20T09:26:27.036 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fdca5c5:::benchmark_data_smithi122_168655_object23813:head# 2024-03-20T09:26:27.036 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fdcf600:::benchmark_data_smithi122_168655_object31618:head# 2024-03-20T09:26:27.037 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fdd6110:::benchmark_data_smithi122_168655_object62422:head# 2024-03-20T09:26:27.038 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fdfa09a:::benchmark_data_smithi122_168655_object117090:head# 2024-03-20T09:26:27.039 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fdfcf06:::benchmark_data_smithi122_168655_object65111:head# 2024-03-20T09:26:27.039 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fe01b46:::benchmark_data_smithi122_168655_object39268:head# 2024-03-20T09:26:27.039 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fe08e0a:::benchmark_data_smithi122_168655_object45110:head# 2024-03-20T09:26:27.039 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fe23fbb:::benchmark_data_smithi122_168655_object82613:head# 2024-03-20T09:26:27.040 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fe2c666:::benchmark_data_smithi122_168655_object56457:head# 2024-03-20T09:26:27.040 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fe3a9d7:::benchmark_data_smithi122_168655_object83955:head# 2024-03-20T09:26:27.041 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fe420c1:::benchmark_data_smithi122_168655_object62917:head# 2024-03-20T09:26:27.041 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fe43dd8:::benchmark_data_smithi122_168655_object34732:head# 2024-03-20T09:26:27.042 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fe4eed8:::benchmark_data_smithi122_168655_object36625:head# 2024-03-20T09:26:27.042 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fe531ee:::benchmark_data_smithi122_168655_object26481:head# 2024-03-20T09:26:27.043 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fe6421c:::benchmark_data_smithi122_168655_object12359:head# 2024-03-20T09:26:27.043 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fe64c32:::benchmark_data_smithi122_168655_object81827:head# 2024-03-20T09:26:27.043 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fe67ba7:::benchmark_data_smithi122_168655_object18718:head# 2024-03-20T09:26:27.043 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fe6932a:::benchmark_data_smithi122_168655_object90059:head# 2024-03-20T09:26:27.044 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fe74f1b:::benchmark_data_smithi122_168655_object113253:head# 2024-03-20T09:26:27.044 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fe811eb:::benchmark_data_smithi122_168655_object49573:head# 2024-03-20T09:26:27.045 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fe84ce2:::benchmark_data_smithi122_168655_object46327:head# 2024-03-20T09:26:27.046 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fe9648c:::benchmark_data_smithi122_168655_object89624:head# 2024-03-20T09:26:27.046 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8feb3a0d:::benchmark_data_smithi122_168655_object11498:head# 2024-03-20T09:26:27.046 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8feb5c22:::benchmark_data_smithi122_168655_object26596:head# 2024-03-20T09:26:27.047 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8febacc4:::benchmark_data_smithi122_168655_object73746:head# 2024-03-20T09:26:27.047 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8febcc9a:::benchmark_data_smithi122_168655_object97411:head# 2024-03-20T09:26:27.047 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8febf3aa:::benchmark_data_smithi122_168655_object78250:head# 2024-03-20T09:26:27.048 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fec9fca:::benchmark_data_smithi122_168655_object10369:head# 2024-03-20T09:26:27.048 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fedc030:::benchmark_data_smithi122_168655_object46724:head# 2024-03-20T09:26:27.049 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8feeb67a:::benchmark_data_smithi122_168655_object38313:head# 2024-03-20T09:26:27.050 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8feeb7c5:::benchmark_data_smithi122_168655_object42132:head# 2024-03-20T09:26:27.050 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8feede18:::benchmark_data_smithi122_168655_object96244:head# 2024-03-20T09:26:27.051 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8feee9e5:::benchmark_data_smithi122_168655_object71377:head# 2024-03-20T09:26:27.051 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ff01d6c:::benchmark_data_smithi122_168655_object11126:head# 2024-03-20T09:26:27.051 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ff26d34:::benchmark_data_smithi122_168655_object63000:head# 2024-03-20T09:26:27.051 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ff2f092:::benchmark_data_smithi122_168655_object45709:head# 2024-03-20T09:26:27.052 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ff4704d:::benchmark_data_smithi122_168655_object10451:head# 2024-03-20T09:26:27.052 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ff6c155:::benchmark_data_smithi122_168655_object56131:head# 2024-03-20T09:26:27.052 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ff79352:::benchmark_data_smithi122_168655_object111059:head# 2024-03-20T09:26:27.053 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ff7c8de:::benchmark_data_smithi122_168655_object56857:head# 2024-03-20T09:26:27.053 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ff85589:::benchmark_data_smithi122_168655_object78531:head# 2024-03-20T09:26:27.054 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ff8a582:::benchmark_data_smithi122_168655_object61203:head# 2024-03-20T09:26:27.054 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ffb0897:::benchmark_data_smithi122_168655_object15605:head# 2024-03-20T09:26:27.055 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ffbd2e4:::benchmark_data_smithi122_168655_object34517:head# 2024-03-20T09:26:27.055 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ffc19c6:::benchmark_data_smithi122_168655_object23061:head# 2024-03-20T09:26:27.055 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ffc2472:::benchmark_data_smithi122_168655_object100533:head# 2024-03-20T09:26:27.056 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ffd4d36:::benchmark_data_smithi122_168655_object109902:head# 2024-03-20T09:26:27.056 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ffd6663:::benchmark_data_smithi122_168655_object61873:head# 2024-03-20T09:26:27.057 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8ffe6ae5:::benchmark_data_smithi122_168655_object72613:head# 2024-03-20T09:26:27.057 INFO:teuthology.orchestra.run.smithi122.stderr:Read #6:8fff4ca6:::benchmark_data_smithi122_168655_object38853:head# 2024-03-20T09:26:27.059 INFO:teuthology.orchestra.run.smithi122.stderr:Export successful 2024-03-20T09:26:27.372 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:26:27.602 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:27 smithi122 ceph-mon[73495]: pgmap v1857: 57 pgs: 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 4 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 37 active+clean; 6.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 251 op/s; 36761/319299 objects degraded (11.513%); 37103/319299 objects misplaced (11.620%); 41 MiB/s, 698 objects/s recovering 2024-03-20T09:26:27.602 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:27 smithi122 ceph-mon[73495]: osdmap e763: 12 total, 11 up, 12 in 2024-03-20T09:26:27.603 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:27 smithi026 ceph-mon[68309]: pgmap v1857: 57 pgs: 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 4 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 37 active+clean; 6.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 251 op/s; 36761/319299 objects degraded (11.513%); 37103/319299 objects misplaced (11.620%); 41 MiB/s, 698 objects/s recovering 2024-03-20T09:26:27.603 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:27 smithi026 ceph-mon[68309]: osdmap e763: 12 total, 11 up, 12 in 2024-03-20T09:26:27.612 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:26:27.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:27.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:26:27.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:27 smithi115 ceph-mon[72476]: pgmap v1857: 57 pgs: 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 4 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 37 active+clean; 6.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 251 op/s; 36761/319299 objects degraded (11.513%); 37103/319299 objects misplaced (11.620%); 41 MiB/s, 698 objects/s recovering 2024-03-20T09:26:27.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:27 smithi115 ceph-mon[72476]: osdmap e763: 12 total, 11 up, 12 in 2024-03-20T09:26:27.723 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:26:27.811 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:27.855 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:26:27.859 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:27.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:26:27.960 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:26:28.012 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:28.096 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:26:28.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:28.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:26:28.206 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:26:28.266 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:28.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:28 smithi115 ceph-mon[72476]: osdmap e764: 12 total, 11 up, 12 in 2024-03-20T09:26:28.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:28 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 36761/319299 objects degraded (11.513%), 8 pgs degraded (PG_DEGRADED) 2024-03-20T09:26:28.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:28 smithi122 ceph-mon[73495]: osdmap e764: 12 total, 11 up, 12 in 2024-03-20T09:26:28.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:28 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 36761/319299 objects degraded (11.513%), 8 pgs degraded (PG_DEGRADED) 2024-03-20T09:26:28.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:28 smithi026 ceph-mon[68309]: osdmap e764: 12 total, 11 up, 12 in 2024-03-20T09:26:28.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:28 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 36761/319299 objects degraded (11.513%), 8 pgs degraded (PG_DEGRADED) 2024-03-20T09:26:29.033 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:26:29.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:29.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:26:29.103 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:26:29.143 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:26:29.176 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:29.356 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:29 smithi026 ceph-mon[68309]: pgmap v1860: 57 pgs: 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 4 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 37 active+clean; 6.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 193 op/s; 36446/319095 objects degraded (11.422%); 36541/319095 objects misplaced (11.451%); 27 MiB/s, 480 objects/s recovering 2024-03-20T09:26:29.442 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:29 smithi122 ceph-mon[73495]: pgmap v1860: 57 pgs: 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 4 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 37 active+clean; 6.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 193 op/s; 36446/319095 objects degraded (11.422%); 36541/319095 objects misplaced (11.451%); 27 MiB/s, 480 objects/s recovering 2024-03-20T09:26:29.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:29 smithi115 ceph-mon[72476]: pgmap v1860: 57 pgs: 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 4 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 37 active+clean; 6.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 193 op/s; 36446/319095 objects degraded (11.422%); 36541/319095 objects misplaced (11.451%); 27 MiB/s, 480 objects/s recovering 2024-03-20T09:26:30.330 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:26:30.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:30.334 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:26:30.365 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:30 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:26:30.435 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:26:30.461 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:30.485 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:26:30.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:30.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:26:30.589 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:26:30.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:30 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:26:30.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:30 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:26:30.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:30 smithi115 ceph-mon[72476]: pgmap v1861: 57 pgs: 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 4 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 37 active+clean; 6.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 195 op/s; 32791/318018 objects degraded (10.311%); 37720/318018 objects misplaced (11.861%); 29 MiB/s, 499 objects/s recovering 2024-03-20T09:26:30.678 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:30 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:26:30.678 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:30 smithi026 ceph-mon[68309]: pgmap v1861: 57 pgs: 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 4 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 37 active+clean; 6.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 195 op/s; 32791/318018 objects degraded (10.311%); 37720/318018 objects misplaced (11.861%); 29 MiB/s, 499 objects/s recovering 2024-03-20T09:26:30.682 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:30.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:30 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:26:30.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:30 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:26:30.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:30 smithi122 ceph-mon[73495]: pgmap v1861: 57 pgs: 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 4 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 37 active+clean; 6.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 195 op/s; 32791/318018 objects degraded (10.311%); 37720/318018 objects misplaced (11.861%); 29 MiB/s, 499 objects/s recovering 2024-03-20T09:26:31.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:31 smithi115 ceph-mon[72476]: osdmap e765: 12 total, 11 up, 12 in 2024-03-20T09:26:31.678 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:31 smithi122 ceph-mon[73495]: osdmap e765: 12 total, 11 up, 12 in 2024-03-20T09:26:31.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:31 smithi026 ceph-mon[68309]: osdmap e765: 12 total, 11 up, 12 in 2024-03-20T09:26:32.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:32 smithi115 ceph-mon[72476]: pgmap v1863: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 4 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 38 active+clean; 6.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 121 op/s; 32409/317115 objects degraded (10.220%); 36542/317115 objects misplaced (11.523%); 22 MiB/s, 359 objects/s recovering 2024-03-20T09:26:32.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:32 smithi122 ceph-mon[73495]: pgmap v1863: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 4 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 38 active+clean; 6.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 121 op/s; 32409/317115 objects degraded (10.220%); 36542/317115 objects misplaced (11.523%); 22 MiB/s, 359 objects/s recovering 2024-03-20T09:26:32.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:32 smithi026 ceph-mon[68309]: pgmap v1863: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 4 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 38 active+clean; 6.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 121 op/s; 32409/317115 objects degraded (10.220%); 36542/317115 objects misplaced (11.523%); 22 MiB/s, 359 objects/s recovering 2024-03-20T09:26:33.653 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:33 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 32409/317115 objects degraded (10.220%), 8 pgs degraded (PG_DEGRADED) 2024-03-20T09:26:33.653 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:33 smithi115 ceph-mon[72476]: osdmap e766: 12 total, 11 up, 12 in 2024-03-20T09:26:33.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:33 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 32409/317115 objects degraded (10.220%), 8 pgs degraded (PG_DEGRADED) 2024-03-20T09:26:33.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:33 smithi122 ceph-mon[73495]: osdmap e766: 12 total, 11 up, 12 in 2024-03-20T09:26:33.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:33 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 32409/317115 objects degraded (10.220%), 8 pgs degraded (PG_DEGRADED) 2024-03-20T09:26:33.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:33 smithi026 ceph-mon[68309]: osdmap e766: 12 total, 11 up, 12 in 2024-03-20T09:26:34.062 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:26:34.114 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:26:34.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:34.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:26:34.223 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:26:34.298 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:34.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:34 smithi122 ceph-mon[73495]: pgmap v1865: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 4 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 38 active+clean; 6.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 134 op/s; 32283/316308 objects degraded (10.206%); 36029/316308 objects misplaced (11.390%); 19 MiB/s, 311 objects/s recovering 2024-03-20T09:26:34.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:34 smithi026 ceph-mon[68309]: pgmap v1865: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 4 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 38 active+clean; 6.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 134 op/s; 32283/316308 objects degraded (10.206%); 36029/316308 objects misplaced (11.390%); 19 MiB/s, 311 objects/s recovering 2024-03-20T09:26:34.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:34 smithi115 ceph-mon[72476]: pgmap v1865: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 4 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 38 active+clean; 6.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 134 op/s; 32283/316308 objects degraded (10.206%); 36029/316308 objects misplaced (11.390%); 19 MiB/s, 311 objects/s recovering 2024-03-20T09:26:34.993 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -n osd.10 --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-10 --journal-path /var/lib/ceph/osd/ceph-10/journal --force --op remove --pgid 6.11 2024-03-20T09:26:35.227 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/osd.10/config 2024-03-20T09:26:35.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:35 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:26:35.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:35 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:26:35.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:35 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:26:35.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:35 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:26:35.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:35 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:26:35.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:35 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:26:36.584 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:36 smithi122 ceph-mon[73495]: osdmap e767: 12 total, 11 up, 12 in 2024-03-20T09:26:36.585 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:36 smithi122 ceph-mon[73495]: pgmap v1867: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 4 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 38 active+clean; 6.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 201 op/s; 32151/314391 objects degraded (10.226%); 34029/314391 objects misplaced (10.824%); 26 MiB/s, 422 objects/s recovering 2024-03-20T09:26:36.627 INFO:teuthology.orchestra.run.smithi122.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-20T09:26:36.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:36 smithi026 ceph-mon[68309]: osdmap e767: 12 total, 11 up, 12 in 2024-03-20T09:26:36.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:36 smithi026 ceph-mon[68309]: pgmap v1867: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 4 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 38 active+clean; 6.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 201 op/s; 32151/314391 objects degraded (10.226%); 34029/314391 objects misplaced (10.824%); 26 MiB/s, 422 objects/s recovering 2024-03-20T09:26:36.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:36 smithi115 ceph-mon[72476]: osdmap e767: 12 total, 11 up, 12 in 2024-03-20T09:26:36.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:36 smithi115 ceph-mon[72476]: pgmap v1867: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 4 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 38 active+clean; 6.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 201 op/s; 32151/314391 objects degraded (10.226%); 34029/314391 objects misplaced (10.824%); 26 MiB/s, 422 objects/s recovering 2024-03-20T09:26:37.455 INFO:teuthology.orchestra.run.smithi122.stdout: marking collection for removal 2024-03-20T09:26:37.456 INFO:teuthology.orchestra.run.smithi122.stdout:setting '_remove' omap key 2024-03-20T09:26:37.456 INFO:teuthology.orchestra.run.smithi122.stdout:finish_remove_pgs 6.11_head removing 6.11 2024-03-20T09:26:37.554 INFO:teuthology.orchestra.run.smithi122.stdout:Remove successful 2024-03-20T09:26:37.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:37 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 32151/314391 objects degraded (10.226%), 8 pgs degraded (PG_DEGRADED) 2024-03-20T09:26:37.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:37 smithi115 ceph-mon[72476]: osdmap e768: 12 total, 11 up, 12 in 2024-03-20T09:26:37.942 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:37 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 32151/314391 objects degraded (10.226%), 8 pgs degraded (PG_DEGRADED) 2024-03-20T09:26:37.942 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:37 smithi122 ceph-mon[73495]: osdmap e768: 12 total, 11 up, 12 in 2024-03-20T09:26:37.967 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:37 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 32151/314391 objects degraded (10.226%), 8 pgs degraded (PG_DEGRADED) 2024-03-20T09:26:37.968 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:37 smithi026 ceph-mon[68309]: osdmap e768: 12 total, 11 up, 12 in 2024-03-20T09:26:38.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:38 smithi115 ceph-mon[72476]: pgmap v1869: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 4 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 38 active+clean; 6.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 161 op/s; 31825/314199 objects degraded (10.129%); 33443/314199 objects misplaced (10.644%); 20 MiB/s, 326 objects/s recovering 2024-03-20T09:26:38.942 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:38 smithi122 ceph-mon[73495]: pgmap v1869: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 4 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 38 active+clean; 6.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 161 op/s; 31825/314199 objects degraded (10.129%); 33443/314199 objects misplaced (10.644%); 20 MiB/s, 326 objects/s recovering 2024-03-20T09:26:38.966 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:38 smithi026 ceph-mon[68309]: pgmap v1869: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 4 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 38 active+clean; 6.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 161 op/s; 31825/314199 objects degraded (10.129%); 33443/314199 objects misplaced (10.644%); 20 MiB/s, 326 objects/s recovering 2024-03-20T09:26:40.100 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell -n osd.10 --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-10 --journal-path /var/lib/ceph/osd/ceph-10/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.6.11.10 2024-03-20T09:26:40.345 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/osd.10/config 2024-03-20T09:26:41.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:41 smithi115 ceph-mon[72476]: pgmap v1870: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 4 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 38 active+clean; 6.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 199 op/s; 31704/312453 objects degraded (10.147%); 31474/312453 objects misplaced (10.073%); 25 MiB/s, 396 objects/s recovering 2024-03-20T09:26:41.654 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:41 smithi122 ceph-mon[73495]: pgmap v1870: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 4 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 38 active+clean; 6.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 199 op/s; 31704/312453 objects degraded (10.147%); 31474/312453 objects misplaced (10.073%); 25 MiB/s, 396 objects/s recovering 2024-03-20T09:26:41.712 INFO:teuthology.orchestra.run.smithi122.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-20T09:26:41.751 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:41 smithi026 ceph-mon[68309]: pgmap v1870: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 4 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 38 active+clean; 6.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 199 op/s; 31704/312453 objects degraded (10.147%); 31474/312453 objects misplaced (10.073%); 25 MiB/s, 396 objects/s recovering 2024-03-20T09:26:42.317 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:42 smithi122 ceph-mon[73495]: osdmap e769: 12 total, 11 up, 12 in 2024-03-20T09:26:42.603 INFO:teuthology.orchestra.run.smithi122.stdout:get_pg_num_history pg_num_history pg_num_history(e753 pg_nums {1={358=31},5={605=16},6={709=16,749=26}} deleted_pools 704,5) 2024-03-20T09:26:42.604 INFO:teuthology.orchestra.run.smithi122.stdout:Importing pgid 6.11 2024-03-20T09:26:42.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:42 smithi115 ceph-mon[72476]: osdmap e769: 12 total, 11 up, 12 in 2024-03-20T09:26:42.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:42 smithi026 ceph-mon[68309]: osdmap e769: 12 total, 11 up, 12 in 2024-03-20T09:26:43.183 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88001f58:::benchmark_data_smithi122_168655_object105647:head# 2024-03-20T09:26:43.184 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.184 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88003a0f:::benchmark_data_smithi122_168655_object102923:head# 2024-03-20T09:26:43.184 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.184 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8801c79a:::benchmark_data_smithi122_168655_object74084:head# 2024-03-20T09:26:43.184 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.184 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88029288:::benchmark_data_smithi122_168655_object25976:head# 2024-03-20T09:26:43.184 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.184 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88048665:::benchmark_data_smithi122_168655_object11931:head# 2024-03-20T09:26:43.184 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.186 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88060d12:::benchmark_data_smithi122_168655_object111021:head# 2024-03-20T09:26:43.186 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.186 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8806b372:::benchmark_data_smithi122_168655_object67061:head# 2024-03-20T09:26:43.186 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.186 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:880736d4:::benchmark_data_smithi122_168655_object100489:head# 2024-03-20T09:26:43.187 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.187 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88074123:::benchmark_data_smithi122_168655_object61859:head# 2024-03-20T09:26:43.187 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.187 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8808ce3e:::benchmark_data_smithi122_168655_object34287:head# 2024-03-20T09:26:43.187 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.187 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8808dd3e:::benchmark_data_smithi122_168655_object39639:head# 2024-03-20T09:26:43.187 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.187 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88091bbc:::benchmark_data_smithi122_168655_object64846:head# 2024-03-20T09:26:43.187 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.187 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:880a375e:::benchmark_data_smithi122_168655_object77901:head# 2024-03-20T09:26:43.187 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.187 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:880a6b2c:::benchmark_data_smithi122_168655_object85921:head# 2024-03-20T09:26:43.187 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.188 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:880c9488:::benchmark_data_smithi122_168655_object47484:head# 2024-03-20T09:26:43.188 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.188 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:880ce0b9:::benchmark_data_smithi122_168655_object12047:head# 2024-03-20T09:26:43.190 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.190 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:880d2781:::benchmark_data_smithi122_168655_object51513:head# 2024-03-20T09:26:43.190 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.191 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:880d86b3:::benchmark_data_smithi122_168655_object52945:head# 2024-03-20T09:26:43.191 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.191 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:880dc9f5:::benchmark_data_smithi122_168655_object77800:head# 2024-03-20T09:26:43.191 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.191 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:880fb963:::benchmark_data_smithi122_168655_object98428:head# 2024-03-20T09:26:43.192 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.192 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88107f9f:::benchmark_data_smithi122_168655_object64041:head# 2024-03-20T09:26:43.192 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.192 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:881154e2:::benchmark_data_smithi122_168655_object34882:head# 2024-03-20T09:26:43.192 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.195 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88116cb5:::benchmark_data_smithi122_168655_object104784:head# 2024-03-20T09:26:43.195 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.196 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88126768:::benchmark_data_smithi122_168655_object84758:head# 2024-03-20T09:26:43.196 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.196 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88132006:::benchmark_data_smithi122_168655_object10336:head# 2024-03-20T09:26:43.196 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.196 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88136845:::benchmark_data_smithi122_168655_object20999:head# 2024-03-20T09:26:43.196 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.196 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88143b6d:::benchmark_data_smithi122_168655_object39556:head# 2024-03-20T09:26:43.196 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.196 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:881539c5:::benchmark_data_smithi122_168655_object15565:head# 2024-03-20T09:26:43.196 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.196 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8816b79a:::benchmark_data_smithi122_168655_object10867:head# 2024-03-20T09:26:43.196 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.196 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8816dc11:::benchmark_data_smithi122_168655_object20087:head# 2024-03-20T09:26:43.197 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.197 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8817a2bc:::benchmark_data_smithi122_168655_object90553:head# 2024-03-20T09:26:43.197 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.197 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:881800a5:::benchmark_data_smithi122_168655_object114273:head# 2024-03-20T09:26:43.197 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.197 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88182ae7:::benchmark_data_smithi122_168655_object113046:head# 2024-03-20T09:26:43.197 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.197 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8818b705:::benchmark_data_smithi122_168655_object91184:head# 2024-03-20T09:26:43.198 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.198 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8819b100:::benchmark_data_smithi122_168655_object50092:head# 2024-03-20T09:26:43.198 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.198 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:881a0c87:::benchmark_data_smithi122_168655_object56147:head# 2024-03-20T09:26:43.198 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.198 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:881a1596:::benchmark_data_smithi122_168655_object106064:head# 2024-03-20T09:26:43.198 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.198 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:881a5bed:::benchmark_data_smithi122_168655_object61578:head# 2024-03-20T09:26:43.198 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.198 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:881b608e:::benchmark_data_smithi122_168655_object38450:head# 2024-03-20T09:26:43.198 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.198 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:881ceaf1:::benchmark_data_smithi122_168655_object53854:head# 2024-03-20T09:26:43.199 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.200 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:881d3f59:::benchmark_data_smithi122_168655_object40802:head# 2024-03-20T09:26:43.200 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.200 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:881d5f19:::benchmark_data_smithi122_168655_object105803:head# 2024-03-20T09:26:43.200 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.200 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:881f4741:::benchmark_data_smithi122_168655_object92803:head# 2024-03-20T09:26:43.200 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.200 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88200d6e:::benchmark_data_smithi122_168655_object62948:head# 2024-03-20T09:26:43.201 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.201 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:882067da:::benchmark_data_smithi122_168655_object96005:head# 2024-03-20T09:26:43.201 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.201 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88211499:::benchmark_data_smithi122_168655_object40617:head# 2024-03-20T09:26:43.201 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.201 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8821a1f7:::benchmark_data_smithi122_168655_object24910:head# 2024-03-20T09:26:43.201 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.201 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8821a4dc:::benchmark_data_smithi122_168655_object70680:head# 2024-03-20T09:26:43.254 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.254 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8821bdb6:::benchmark_data_smithi122_168655_object107498:head# 2024-03-20T09:26:43.254 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.254 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88225d3d:::benchmark_data_smithi122_168655_object47675:head# 2024-03-20T09:26:43.254 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.256 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8822feee:::benchmark_data_smithi122_168655_object91493:head# 2024-03-20T09:26:43.256 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.256 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88239937:::benchmark_data_smithi122_168655_object43818:head# 2024-03-20T09:26:43.256 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.256 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8823d844:::benchmark_data_smithi122_168655_object40477:head# 2024-03-20T09:26:43.256 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.256 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8824f22b:::benchmark_data_smithi122_168655_object77122:head# 2024-03-20T09:26:43.256 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.256 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88274cda:::benchmark_data_smithi122_168655_object111124:head# 2024-03-20T09:26:43.256 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.256 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8828cf1f:::benchmark_data_smithi122_168655_object61435:head# 2024-03-20T09:26:43.256 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.267 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8828e455:::benchmark_data_smithi122_168655_object91274:head# 2024-03-20T09:26:43.267 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.268 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88292533:::benchmark_data_smithi122_168655_object17167:head# 2024-03-20T09:26:43.268 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.268 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88294f34:::benchmark_data_smithi122_168655_object52424:head# 2024-03-20T09:26:43.268 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.268 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8829b8c8:::benchmark_data_smithi122_168655_object27428:head# 2024-03-20T09:26:43.268 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.268 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8829baa8:::benchmark_data_smithi122_168655_object74127:head# 2024-03-20T09:26:43.268 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.268 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8829cd76:::benchmark_data_smithi122_168655_object73607:head# 2024-03-20T09:26:43.268 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.268 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:882a888b:::benchmark_data_smithi122_168655_object36826:head# 2024-03-20T09:26:43.268 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.268 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:882ad8cb:::benchmark_data_smithi122_168655_object19171:head# 2024-03-20T09:26:43.269 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.269 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:882bc5f4:::benchmark_data_smithi122_168655_object110053:head# 2024-03-20T09:26:43.269 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.269 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:882cb397:::benchmark_data_smithi122_168655_object114283:head# 2024-03-20T09:26:43.269 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.269 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:882d96cb:::benchmark_data_smithi122_168655_object80386:head# 2024-03-20T09:26:43.269 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.269 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:882df894:::benchmark_data_smithi122_168655_object23123:head# 2024-03-20T09:26:43.269 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.269 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:882e659e:::benchmark_data_smithi122_168655_object85776:head# 2024-03-20T09:26:43.269 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.269 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:882e725d:::benchmark_data_smithi122_168655_object38275:head# 2024-03-20T09:26:43.269 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.269 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:882ef242:::benchmark_data_smithi122_168655_object119455:head# 2024-03-20T09:26:43.270 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.270 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:882f08ee:::benchmark_data_smithi122_168655_object78591:head# 2024-03-20T09:26:43.270 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.270 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:882f0de8:::benchmark_data_smithi122_168655_object67896:head# 2024-03-20T09:26:43.270 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.270 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:882fa11f:::benchmark_data_smithi122_168655_object95064:head# 2024-03-20T09:26:43.270 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.270 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:882fb2e8:::benchmark_data_smithi122_168655_object111743:head# 2024-03-20T09:26:43.270 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.270 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88311fa2:::benchmark_data_smithi122_168655_object117822:head# 2024-03-20T09:26:43.270 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.270 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8831d676:::benchmark_data_smithi122_168655_object59434:head# 2024-03-20T09:26:43.270 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.270 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8832b473:::benchmark_data_smithi122_168655_object103727:head# 2024-03-20T09:26:43.270 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.271 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8832f41c:::benchmark_data_smithi122_168655_object54881:head# 2024-03-20T09:26:43.271 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.271 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8833aa26:::benchmark_data_smithi122_168655_object58989:head# 2024-03-20T09:26:43.271 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.271 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88341e4f:::benchmark_data_smithi122_168655_object91682:head# 2024-03-20T09:26:43.271 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.271 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8834bbeb:::benchmark_data_smithi122_168655_object79908:head# 2024-03-20T09:26:43.271 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.271 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8834be0a:::benchmark_data_smithi122_168655_object80840:head# 2024-03-20T09:26:43.271 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.271 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:883534ec:::benchmark_data_smithi122_168655_object70603:head# 2024-03-20T09:26:43.271 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.271 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88354f5e:::benchmark_data_smithi122_168655_object69305:head# 2024-03-20T09:26:43.272 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.272 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8836028d:::benchmark_data_smithi122_168655_object76457:head# 2024-03-20T09:26:43.272 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.272 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88360aca:::benchmark_data_smithi122_168655_object57011:head# 2024-03-20T09:26:43.272 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.272 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8837b11b:::benchmark_data_smithi122_168655_object97659:head# 2024-03-20T09:26:43.272 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.272 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8837d9a7:::benchmark_data_smithi122_168655_object39050:head# 2024-03-20T09:26:43.272 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.272 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88380a1b:::benchmark_data_smithi122_168655_object41965:head# 2024-03-20T09:26:43.272 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.272 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88386f61:::benchmark_data_smithi122_168655_object11918:head# 2024-03-20T09:26:43.272 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.272 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8838be91:::benchmark_data_smithi122_168655_object65281:head# 2024-03-20T09:26:43.273 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.273 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8838d1e9:::benchmark_data_smithi122_168655_object94632:head# 2024-03-20T09:26:43.273 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.273 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:883a797a:::benchmark_data_smithi122_168655_object119310:head# 2024-03-20T09:26:43.273 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.273 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:883b16bb:::benchmark_data_smithi122_168655_object52397:head# 2024-03-20T09:26:43.273 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.273 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:883b70fc:::benchmark_data_smithi122_168655_object102065:head# 2024-03-20T09:26:43.273 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.273 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:883c2239:::benchmark_data_smithi122_168655_object20360:head# 2024-03-20T09:26:43.273 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.273 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:883d7e8b:::benchmark_data_smithi122_168655_object65604:head# 2024-03-20T09:26:43.273 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.274 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:883dea46:::benchmark_data_smithi122_168655_object73243:head# 2024-03-20T09:26:43.274 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.286 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:883e8fa7:::benchmark_data_smithi122_168655_object46834:head# 2024-03-20T09:26:43.287 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.287 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:883f1bd4:::benchmark_data_smithi122_168655_object14604:head# 2024-03-20T09:26:43.287 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.287 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:883fab19:::benchmark_data_smithi122_168655_object43292:head# 2024-03-20T09:26:43.287 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.287 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88406623:::benchmark_data_smithi122_168655_object108261:head# 2024-03-20T09:26:43.287 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.287 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8841475e:::benchmark_data_smithi122_168655_object61773:head# 2024-03-20T09:26:43.287 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.287 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88415062:::benchmark_data_smithi122_168655_object118381:head# 2024-03-20T09:26:43.287 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.287 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8841e0c7:::benchmark_data_smithi122_168655_object59547:head# 2024-03-20T09:26:43.287 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.288 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88433bc4:::benchmark_data_smithi122_168655_object14603:head# 2024-03-20T09:26:43.288 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.288 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:884391f9:::benchmark_data_smithi122_168655_object79179:head# 2024-03-20T09:26:43.288 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.288 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8843b65d:::benchmark_data_smithi122_168655_object77162:head# 2024-03-20T09:26:43.288 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.288 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88443f53:::benchmark_data_smithi122_168655_object45395:head# 2024-03-20T09:26:43.288 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.288 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8844f493:::benchmark_data_smithi122_168655_object81326:head# 2024-03-20T09:26:43.288 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.288 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88458ff0:::benchmark_data_smithi122_168655_object22706:head# 2024-03-20T09:26:43.288 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.288 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88461fe1:::benchmark_data_smithi122_168655_object66627:head# 2024-03-20T09:26:43.288 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.289 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88462022:::benchmark_data_smithi122_168655_object19045:head# 2024-03-20T09:26:43.289 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.289 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88465414:::benchmark_data_smithi122_168655_object119476:head# 2024-03-20T09:26:43.289 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.289 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:884663fb:::benchmark_data_smithi122_168655_object71326:head# 2024-03-20T09:26:43.289 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.289 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8846ea3c:::benchmark_data_smithi122_168655_object57898:head# 2024-03-20T09:26:43.289 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.289 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:884735a7:::benchmark_data_smithi122_168655_object108606:head# 2024-03-20T09:26:43.289 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.289 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88476198:::benchmark_data_smithi122_168655_object72105:head# 2024-03-20T09:26:43.289 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.289 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8847d914:::benchmark_data_smithi122_168655_object119988:head# 2024-03-20T09:26:43.290 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.290 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:884862b1:::benchmark_data_smithi122_168655_object24215:head# 2024-03-20T09:26:43.290 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.290 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8848878f:::benchmark_data_smithi122_168655_object97670:head# 2024-03-20T09:26:43.290 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.290 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:884a06d5:::benchmark_data_smithi122_168655_object41298:head# 2024-03-20T09:26:43.290 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.290 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:884a1224:::benchmark_data_smithi122_168655_object64005:head# 2024-03-20T09:26:43.290 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.290 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:884b9b72:::benchmark_data_smithi122_168655_object63613:head# 2024-03-20T09:26:43.290 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.290 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:884be9db:::benchmark_data_smithi122_168655_object110993:head# 2024-03-20T09:26:43.290 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.290 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:884c0283:::benchmark_data_smithi122_168655_object48572:head# 2024-03-20T09:26:43.291 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.291 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:884ccc63:::benchmark_data_smithi122_168655_object90288:head# 2024-03-20T09:26:43.291 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.291 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:884ccdc9:::benchmark_data_smithi122_168655_object53067:head# 2024-03-20T09:26:43.291 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.291 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:884d1c36:::benchmark_data_smithi122_168655_object115821:head# 2024-03-20T09:26:43.291 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.291 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:884d9495:::benchmark_data_smithi122_168655_object28061:head# 2024-03-20T09:26:43.291 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.291 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:884ea10b:::benchmark_data_smithi122_168655_object77879:head# 2024-03-20T09:26:43.291 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.291 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:884ec59d:::benchmark_data_smithi122_168655_object95895:head# 2024-03-20T09:26:43.291 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.291 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:884f9c2d:::benchmark_data_smithi122_168655_object15494:head# 2024-03-20T09:26:43.292 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.292 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88501f3c:::benchmark_data_smithi122_168655_object13872:head# 2024-03-20T09:26:43.292 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.292 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88502139:::benchmark_data_smithi122_168655_object110411:head# 2024-03-20T09:26:43.292 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.292 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8850515e:::benchmark_data_smithi122_168655_object10612:head# 2024-03-20T09:26:43.292 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.292 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88508257:::benchmark_data_smithi122_168655_object38630:head# 2024-03-20T09:26:43.292 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.292 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88510083:::benchmark_data_smithi122_168655_object32227:head# 2024-03-20T09:26:43.292 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.292 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8852cded:::benchmark_data_smithi122_168655_object11637:head# 2024-03-20T09:26:43.292 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.293 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:885416a4:::benchmark_data_smithi122_168655_object37679:head# 2024-03-20T09:26:43.293 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.293 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88543ae6:::benchmark_data_smithi122_168655_object42184:head# 2024-03-20T09:26:43.293 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.293 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8854b4e5:::benchmark_data_smithi122_168655_object13087:head# 2024-03-20T09:26:43.293 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.293 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:885537a1:::benchmark_data_smithi122_168655_object84179:head# 2024-03-20T09:26:43.293 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.293 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88553d38:::benchmark_data_smithi122_168655_object37231:head# 2024-03-20T09:26:43.293 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.293 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8856f567:::benchmark_data_smithi122_168655_object37649:head# 2024-03-20T09:26:43.293 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.293 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8856f85a:::benchmark_data_smithi122_168655_object95801:head# 2024-03-20T09:26:43.293 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.293 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88572300:::benchmark_data_smithi122_168655_object26444:head# 2024-03-20T09:26:43.293 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.294 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8857fa1d:::benchmark_data_smithi122_168655_object66981:head# 2024-03-20T09:26:43.294 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.294 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8858dc20:::benchmark_data_smithi122_168655_object19939:head# 2024-03-20T09:26:43.294 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.294 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8858f0be:::benchmark_data_smithi122_168655_object71418:head# 2024-03-20T09:26:43.294 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.294 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:885aa3d6:::benchmark_data_smithi122_168655_object101521:head# 2024-03-20T09:26:43.294 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.294 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:885b3835:::benchmark_data_smithi122_168655_object112928:head# 2024-03-20T09:26:43.294 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.294 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:885b3ea4:::benchmark_data_smithi122_168655_object84801:head# 2024-03-20T09:26:43.294 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.294 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:885b4dd8:::benchmark_data_smithi122_168655_object18809:head# 2024-03-20T09:26:43.295 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.295 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:885ce753:::benchmark_data_smithi122_168655_object24947:head# 2024-03-20T09:26:43.295 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.295 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:885d0d54:::benchmark_data_smithi122_168655_object53853:head# 2024-03-20T09:26:43.295 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.295 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:885e8ddd:::benchmark_data_smithi122_168655_object47264:head# 2024-03-20T09:26:43.295 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.295 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:885f608f:::benchmark_data_smithi122_168655_object86412:head# 2024-03-20T09:26:43.295 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.295 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8860a8aa:::benchmark_data_smithi122_168655_object60309:head# 2024-03-20T09:26:43.295 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.295 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8860c5f5:::benchmark_data_smithi122_168655_object54976:head# 2024-03-20T09:26:43.295 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.295 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8860c7c0:::benchmark_data_smithi122_168655_object64848:head# 2024-03-20T09:26:43.296 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.296 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8861e879:::benchmark_data_smithi122_168655_object86409:head# 2024-03-20T09:26:43.296 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.296 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88621fd5:::benchmark_data_smithi122_168655_object65287:head# 2024-03-20T09:26:43.296 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.296 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8862f3bf:::benchmark_data_smithi122_168655_object78201:head# 2024-03-20T09:26:43.296 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.296 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:886393e2:::benchmark_data_smithi122_168655_object99338:head# 2024-03-20T09:26:43.296 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.296 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:886395e8:::benchmark_data_smithi122_168655_object73431:head# 2024-03-20T09:26:43.296 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.296 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8863d963:::benchmark_data_smithi122_168655_object53336:head# 2024-03-20T09:26:43.296 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.297 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88643d8c:::benchmark_data_smithi122_168655_object39485:head# 2024-03-20T09:26:43.297 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.297 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8865c41e:::benchmark_data_smithi122_168655_object51888:head# 2024-03-20T09:26:43.297 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.297 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:886696ad:::benchmark_data_smithi122_168655_object46787:head# 2024-03-20T09:26:43.297 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.297 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:886832cc:::benchmark_data_smithi122_168655_object16742:head# 2024-03-20T09:26:43.297 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.297 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8869b5c3:::benchmark_data_smithi122_168655_object47891:head# 2024-03-20T09:26:43.297 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.297 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:886a3423:::benchmark_data_smithi122_168655_object65982:head# 2024-03-20T09:26:43.297 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.297 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:886a7ead:::benchmark_data_smithi122_168655_object61482:head# 2024-03-20T09:26:43.298 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.298 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:886acca1:::benchmark_data_smithi122_168655_object85597:head# 2024-03-20T09:26:43.298 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.298 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:886c9e0e:::benchmark_data_smithi122_168655_object100486:head# 2024-03-20T09:26:43.298 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.298 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:886cf3fd:::benchmark_data_smithi122_168655_object57190:head# 2024-03-20T09:26:43.298 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.320 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:886d46bb:::benchmark_data_smithi122_168655_object110317:head# 2024-03-20T09:26:43.320 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.320 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:886dae55:::benchmark_data_smithi122_168655_object109298:head# 2024-03-20T09:26:43.320 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.320 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:886dc807:::benchmark_data_smithi122_168655_object81195:head# 2024-03-20T09:26:43.320 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.320 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:886e1de8:::benchmark_data_smithi122_168655_object30623:head# 2024-03-20T09:26:43.320 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.320 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:886fb059:::benchmark_data_smithi122_168655_object17665:head# 2024-03-20T09:26:43.320 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.320 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:887307cc:::benchmark_data_smithi122_168655_object94718:head# 2024-03-20T09:26:43.320 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.321 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8874645c:::benchmark_data_smithi122_168655_object61483:head# 2024-03-20T09:26:43.321 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.321 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88747712:::benchmark_data_smithi122_168655_object47132:head# 2024-03-20T09:26:43.321 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.321 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88750b68:::benchmark_data_smithi122_168655_object40444:head# 2024-03-20T09:26:43.321 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.321 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8875170b:::benchmark_data_smithi122_168655_object46490:head# 2024-03-20T09:26:43.321 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.321 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8875907a:::benchmark_data_smithi122_168655_object16117:head# 2024-03-20T09:26:43.321 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.321 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8876e23d:::benchmark_data_smithi122_168655_object107909:head# 2024-03-20T09:26:43.321 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.322 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:887701e4:::benchmark_data_smithi122_168655_object78941:head# 2024-03-20T09:26:43.322 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.322 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8877df59:::benchmark_data_smithi122_168655_object69916:head# 2024-03-20T09:26:43.322 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.322 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:887927a4:::benchmark_data_smithi122_168655_object72558:head# 2024-03-20T09:26:43.322 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.322 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8879a6e8:::benchmark_data_smithi122_168655_object80596:head# 2024-03-20T09:26:43.322 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.322 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8879ca77:::benchmark_data_smithi122_168655_object72075:head# 2024-03-20T09:26:43.322 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.322 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:887ae17e:::benchmark_data_smithi122_168655_object11178:head# 2024-03-20T09:26:43.322 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.322 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:887c2734:::benchmark_data_smithi122_168655_object79001:head# 2024-03-20T09:26:43.323 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.323 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:887c7392:::benchmark_data_smithi122_168655_object94975:head# 2024-03-20T09:26:43.323 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.323 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:887cb092:::benchmark_data_smithi122_168655_object25974:head# 2024-03-20T09:26:43.323 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.323 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:887cb747:::benchmark_data_smithi122_168655_object50522:head# 2024-03-20T09:26:43.323 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.323 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:887d6dc2:::benchmark_data_smithi122_168655_object17904:head# 2024-03-20T09:26:43.323 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.323 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:887de3ec:::benchmark_data_smithi122_168655_object52249:head# 2024-03-20T09:26:43.323 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.323 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:887e6e9c:::benchmark_data_smithi122_168655_object88571:head# 2024-03-20T09:26:43.323 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.323 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:887e9633:::benchmark_data_smithi122_168655_object85966:head# 2024-03-20T09:26:43.324 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.324 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:887e9916:::benchmark_data_smithi122_168655_object76580:head# 2024-03-20T09:26:43.324 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.324 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:887f4d8d:::benchmark_data_smithi122_168655_object51794:head# 2024-03-20T09:26:43.324 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.324 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88800cfe:::benchmark_data_smithi122_168655_object109579:head# 2024-03-20T09:26:43.324 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.324 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88805713:::benchmark_data_smithi122_168655_object47942:head# 2024-03-20T09:26:43.324 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.324 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8880778d:::benchmark_data_smithi122_168655_object22042:head# 2024-03-20T09:26:43.324 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.324 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88807c66:::benchmark_data_smithi122_168655_object87481:head# 2024-03-20T09:26:43.324 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.324 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88811cd5:::benchmark_data_smithi122_168655_object11062:head# 2024-03-20T09:26:43.324 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.324 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88818e7d:::benchmark_data_smithi122_168655_object84677:head# 2024-03-20T09:26:43.325 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.325 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8881e017:::benchmark_data_smithi122_168655_object87136:head# 2024-03-20T09:26:43.325 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.325 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8881e4df:::benchmark_data_smithi122_168655_object72354:head# 2024-03-20T09:26:43.325 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.325 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88824c5f:::benchmark_data_smithi122_168655_object90500:head# 2024-03-20T09:26:43.325 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.325 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88824d70:::benchmark_data_smithi122_168655_object44504:head# 2024-03-20T09:26:43.325 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.325 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8883d215:::benchmark_data_smithi122_168655_object49440:head# 2024-03-20T09:26:43.325 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.325 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88849785:::benchmark_data_smithi122_168655_object11533:head# 2024-03-20T09:26:43.325 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.325 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8884a4b9:::benchmark_data_smithi122_168655_object108169:head# 2024-03-20T09:26:43.326 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.326 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8885d57c:::benchmark_data_smithi122_168655_object58457:head# 2024-03-20T09:26:43.326 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.326 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8886bdd2:::benchmark_data_smithi122_168655_object68823:head# 2024-03-20T09:26:43.326 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.326 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8886bed5:::benchmark_data_smithi122_168655_object100895:head# 2024-03-20T09:26:43.326 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.326 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8888745e:::benchmark_data_smithi122_168655_object27145:head# 2024-03-20T09:26:43.326 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.326 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8889f1da:::benchmark_data_smithi122_168655_object18114:head# 2024-03-20T09:26:43.326 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.326 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:888a0a1d:::benchmark_data_smithi122_168655_object80177:head# 2024-03-20T09:26:43.326 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.327 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:888aa02d:::benchmark_data_smithi122_168655_object89558:head# 2024-03-20T09:26:43.327 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.327 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:888aef90:::benchmark_data_smithi122_168655_object40868:head# 2024-03-20T09:26:43.327 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.327 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:888b1e89:::benchmark_data_smithi122_168655_object35743:head# 2024-03-20T09:26:43.327 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.327 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:888bd1b6:::benchmark_data_smithi122_168655_object95556:head# 2024-03-20T09:26:43.327 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.327 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:888c91f0:::benchmark_data_smithi122_168655_object106356:head# 2024-03-20T09:26:43.327 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.327 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:888d29e1:::benchmark_data_smithi122_168655_object49289:head# 2024-03-20T09:26:43.327 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.327 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:888d5a84:::benchmark_data_smithi122_168655_object111900:head# 2024-03-20T09:26:43.327 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.328 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:888db302:::benchmark_data_smithi122_168655_object12006:head# 2024-03-20T09:26:43.328 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.328 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:888ec125:::benchmark_data_smithi122_168655_object15357:head# 2024-03-20T09:26:43.328 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.328 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:888eebb9:::benchmark_data_smithi122_168655_object14647:head# 2024-03-20T09:26:43.328 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.328 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:888fcc93:::benchmark_data_smithi122_168655_object32470:head# 2024-03-20T09:26:43.328 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.328 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:888ff0a4:::benchmark_data_smithi122_168655_object79181:head# 2024-03-20T09:26:43.328 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.328 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8890b4a0:::benchmark_data_smithi122_168655_object38120:head# 2024-03-20T09:26:43.328 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.328 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88919c20:::benchmark_data_smithi122_168655_object22908:head# 2024-03-20T09:26:43.328 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.329 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8891d764:::benchmark_data_smithi122_168655_object98251:head# 2024-03-20T09:26:43.329 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.329 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8894088d:::benchmark_data_smithi122_168655_object95855:head# 2024-03-20T09:26:43.329 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.329 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88941c0a:::benchmark_data_smithi122_168655_object22089:head# 2024-03-20T09:26:43.329 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.329 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88942542:::benchmark_data_smithi122_168655_object95652:head# 2024-03-20T09:26:43.329 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.329 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8894cd21:::benchmark_data_smithi122_168655_object36693:head# 2024-03-20T09:26:43.329 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.329 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8895c1a5:::benchmark_data_smithi122_168655_object25339:head# 2024-03-20T09:26:43.329 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.329 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8895cf78:::benchmark_data_smithi122_168655_object62779:head# 2024-03-20T09:26:43.329 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.329 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8896b039:::benchmark_data_smithi122_168655_object58637:head# 2024-03-20T09:26:43.330 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.330 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8896b832:::benchmark_data_smithi122_168655_object57655:head# 2024-03-20T09:26:43.330 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.330 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8897608d:::benchmark_data_smithi122_168655_object78007:head# 2024-03-20T09:26:43.330 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.330 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8897d4af:::benchmark_data_smithi122_168655_object104876:head# 2024-03-20T09:26:43.330 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.330 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8898162e:::benchmark_data_smithi122_168655_object28196:head# 2024-03-20T09:26:43.330 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.330 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88982a13:::benchmark_data_smithi122_168655_object43829:head# 2024-03-20T09:26:43.330 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.330 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8898651b:::benchmark_data_smithi122_168655_object10207:head# 2024-03-20T09:26:43.330 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.330 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8898a69b:::benchmark_data_smithi122_168655_object69596:head# 2024-03-20T09:26:43.330 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.331 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88996a6d:::benchmark_data_smithi122_168655_object21014:head# 2024-03-20T09:26:43.331 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.331 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8899bdd6:::benchmark_data_smithi122_168655_object17574:head# 2024-03-20T09:26:43.331 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.331 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:889ad81d:::benchmark_data_smithi122_168655_object50282:head# 2024-03-20T09:26:43.331 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.331 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:889b01ad:::benchmark_data_smithi122_168655_object32880:head# 2024-03-20T09:26:43.331 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.331 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:889b2ef2:::benchmark_data_smithi122_168655_object30401:head# 2024-03-20T09:26:43.331 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.331 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:889b4b97:::benchmark_data_smithi122_168655_object109466:head# 2024-03-20T09:26:43.332 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.332 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:889e3371:::benchmark_data_smithi122_168655_object116270:head# 2024-03-20T09:26:43.332 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.332 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:889ee76a:::benchmark_data_smithi122_168655_object81104:head# 2024-03-20T09:26:43.332 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.332 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:889f96a4:::benchmark_data_smithi122_168655_object36712:head# 2024-03-20T09:26:43.332 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.332 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:889fccc5:::benchmark_data_smithi122_168655_object26762:head# 2024-03-20T09:26:43.332 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.332 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88a10059:::benchmark_data_smithi122_168655_object102118:head# 2024-03-20T09:26:43.332 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.333 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88a14c06:::benchmark_data_smithi122_168655_object104095:head# 2024-03-20T09:26:43.333 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.333 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88a2099e:::benchmark_data_smithi122_168655_object11248:head# 2024-03-20T09:26:43.333 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.333 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88a250fe:::benchmark_data_smithi122_168655_object85664:head# 2024-03-20T09:26:43.333 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.333 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88a4a4ce:::benchmark_data_smithi122_168655_object110583:head# 2024-03-20T09:26:43.333 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.333 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88a4c93e:::benchmark_data_smithi122_168655_object20723:head# 2024-03-20T09:26:43.333 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.333 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88a59fb4:::benchmark_data_smithi122_168655_object113893:head# 2024-03-20T09:26:43.333 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.333 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88a5f150:::benchmark_data_smithi122_168655_object111121:head# 2024-03-20T09:26:43.333 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.334 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88a608f3:::benchmark_data_smithi122_168655_object104284:head# 2024-03-20T09:26:43.334 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.334 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88a656b1:::benchmark_data_smithi122_168655_object72249:head# 2024-03-20T09:26:43.334 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.334 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88a6848d:::benchmark_data_smithi122_168655_object44088:head# 2024-03-20T09:26:43.334 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.334 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88a735c1:::benchmark_data_smithi122_168655_object43216:head# 2024-03-20T09:26:43.334 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.334 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88a74dc4:::benchmark_data_smithi122_168655_object114175:head# 2024-03-20T09:26:43.334 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.334 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88a7ff5a:::benchmark_data_smithi122_168655_object24872:head# 2024-03-20T09:26:43.334 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.334 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88a800d0:::benchmark_data_smithi122_168655_object62472:head# 2024-03-20T09:26:43.334 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.335 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88a81b2a:::benchmark_data_smithi122_168655_object11977:head# 2024-03-20T09:26:43.335 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.335 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88a91158:::benchmark_data_smithi122_168655_object103984:head# 2024-03-20T09:26:43.335 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.335 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88a99677:::benchmark_data_smithi122_168655_object104019:head# 2024-03-20T09:26:43.335 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.335 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88aa5b17:::benchmark_data_smithi122_168655_object109146:head# 2024-03-20T09:26:43.335 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.335 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88aa64f9:::benchmark_data_smithi122_168655_object43406:head# 2024-03-20T09:26:43.335 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.335 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88aa6a34:::benchmark_data_smithi122_168655_object80481:head# 2024-03-20T09:26:43.335 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.335 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88abe8f1:::benchmark_data_smithi122_168655_object80627:head# 2024-03-20T09:26:43.335 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.336 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88ac31a2:::benchmark_data_smithi122_168655_object91529:head# 2024-03-20T09:26:43.336 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.336 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88acac35:::benchmark_data_smithi122_168655_object99553:head# 2024-03-20T09:26:43.336 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.336 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88acaebe:::benchmark_data_smithi122_168655_object36657:head# 2024-03-20T09:26:43.336 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.336 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88ad1253:::benchmark_data_smithi122_168655_object66926:head# 2024-03-20T09:26:43.336 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.336 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88af2a61:::benchmark_data_smithi122_168655_object90584:head# 2024-03-20T09:26:43.336 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.336 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88af8f4b:::benchmark_data_smithi122_168655_object16392:head# 2024-03-20T09:26:43.336 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.336 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88afa6c7:::benchmark_data_smithi122_168655_object39928:head# 2024-03-20T09:26:43.336 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.336 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88b16084:::benchmark_data_smithi122_168655_object106782:head# 2024-03-20T09:26:43.337 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.337 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88b1a0c8:::benchmark_data_smithi122_168655_object26182:head# 2024-03-20T09:26:43.337 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.337 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88b1cd39:::benchmark_data_smithi122_168655_object36741:head# 2024-03-20T09:26:43.337 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.337 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88b4273c:::benchmark_data_smithi122_168655_object32232:head# 2024-03-20T09:26:43.337 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.337 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88b58c0e:::benchmark_data_smithi122_168655_object56298:head# 2024-03-20T09:26:43.337 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.337 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88b5f03c:::benchmark_data_smithi122_168655_object50242:head# 2024-03-20T09:26:43.337 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.337 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88b5f52d:::benchmark_data_smithi122_168655_object65511:head# 2024-03-20T09:26:43.337 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.337 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88b688d2:::benchmark_data_smithi122_168655_object18872:head# 2024-03-20T09:26:43.338 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.338 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88b6e339:::benchmark_data_smithi122_168655_object59774:head# 2024-03-20T09:26:43.338 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.338 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88b73c09:::benchmark_data_smithi122_168655_object32410:head# 2024-03-20T09:26:43.338 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.338 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88b78ee7:::benchmark_data_smithi122_168655_object83106:head# 2024-03-20T09:26:43.338 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.338 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88b7d817:::benchmark_data_smithi122_168655_object59282:head# 2024-03-20T09:26:43.338 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.338 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88b7ecf0:::benchmark_data_smithi122_168655_object114220:head# 2024-03-20T09:26:43.338 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.338 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88b8646a:::benchmark_data_smithi122_168655_object70491:head# 2024-03-20T09:26:43.338 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.339 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88b9b476:::benchmark_data_smithi122_168655_object72578:head# 2024-03-20T09:26:43.339 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.339 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88ba015e:::benchmark_data_smithi122_168655_object51312:head# 2024-03-20T09:26:43.339 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.339 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88ba60f1:::benchmark_data_smithi122_168655_object56294:head# 2024-03-20T09:26:43.339 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.339 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88babf45:::benchmark_data_smithi122_168655_object12108:head# 2024-03-20T09:26:43.339 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.339 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88bac103:::benchmark_data_smithi122_168655_object48673:head# 2024-03-20T09:26:43.339 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.339 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88bb6ec4:::benchmark_data_smithi122_168655_object64222:head# 2024-03-20T09:26:43.339 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.339 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88bbdeb6:::benchmark_data_smithi122_168655_object16758:head# 2024-03-20T09:26:43.339 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.364 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88bc3830:::benchmark_data_smithi122_168655_object24679:head# 2024-03-20T09:26:43.364 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.364 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88bc686c:::benchmark_data_smithi122_168655_object108454:head# 2024-03-20T09:26:43.364 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.364 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88bcd0b8:::benchmark_data_smithi122_168655_object54171:head# 2024-03-20T09:26:43.364 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.364 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88bd44dd:::benchmark_data_smithi122_168655_object19879:head# 2024-03-20T09:26:43.364 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.364 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88bdd1f2:::benchmark_data_smithi122_168655_object45319:head# 2024-03-20T09:26:43.365 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.365 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88be4fa9:::benchmark_data_smithi122_168655_object19697:head# 2024-03-20T09:26:43.365 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.365 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88be535d:::benchmark_data_smithi122_168655_object77792:head# 2024-03-20T09:26:43.365 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.365 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88bf32f6:::benchmark_data_smithi122_168655_object43891:head# 2024-03-20T09:26:43.365 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.365 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88bf405a:::benchmark_data_smithi122_168655_object32420:head# 2024-03-20T09:26:43.365 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.365 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88bf42ef:::benchmark_data_smithi122_168655_object14520:head# 2024-03-20T09:26:43.365 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.365 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88bfa053:::benchmark_data_smithi122_168655_object87270:head# 2024-03-20T09:26:43.365 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.365 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88bfbbd0:::benchmark_data_smithi122_168655_object65040:head# 2024-03-20T09:26:43.366 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.366 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88bfcc7b:::benchmark_data_smithi122_168655_object60927:head# 2024-03-20T09:26:43.366 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.366 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88bfee75:::benchmark_data_smithi122_168655_object46356:head# 2024-03-20T09:26:43.366 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.366 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88c03e92:::benchmark_data_smithi122_168655_object18579:head# 2024-03-20T09:26:43.366 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.366 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88c110e4:::benchmark_data_smithi122_168655_object56666:head# 2024-03-20T09:26:43.366 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.366 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88c14280:::benchmark_data_smithi122_168655_object78407:head# 2024-03-20T09:26:43.366 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.366 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88c19968:::benchmark_data_smithi122_168655_object117346:head# 2024-03-20T09:26:43.366 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.366 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88c2249f:::benchmark_data_smithi122_168655_object38764:head# 2024-03-20T09:26:43.367 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.367 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88c282ba:::benchmark_data_smithi122_168655_object83809:head# 2024-03-20T09:26:43.367 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.367 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88c29376:::benchmark_data_smithi122_168655_object83979:head# 2024-03-20T09:26:43.367 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.367 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88c3c109:::benchmark_data_smithi122_168655_object105496:head# 2024-03-20T09:26:43.367 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.367 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88c51405:::benchmark_data_smithi122_168655_object65832:head# 2024-03-20T09:26:43.367 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.367 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88c5d99b:::benchmark_data_smithi122_168655_object97810:head# 2024-03-20T09:26:43.367 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.367 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88c5e6b6:::benchmark_data_smithi122_168655_object93492:head# 2024-03-20T09:26:43.367 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.367 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88c62e95:::benchmark_data_smithi122_168655_object52885:head# 2024-03-20T09:26:43.368 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.368 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88c63bbe:::benchmark_data_smithi122_168655_object83140:head# 2024-03-20T09:26:43.368 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.368 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88c642e4:::benchmark_data_smithi122_168655_object60340:head# 2024-03-20T09:26:43.368 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.368 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88c78667:::benchmark_data_smithi122_168655_object25208:head# 2024-03-20T09:26:43.368 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.368 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88c7c652:::benchmark_data_smithi122_168655_object31189:head# 2024-03-20T09:26:43.368 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.368 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88c7da07:::benchmark_data_smithi122_168655_object26433:head# 2024-03-20T09:26:43.368 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.368 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88c90c28:::benchmark_data_smithi122_168655_object63839:head# 2024-03-20T09:26:43.368 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.368 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88c94a64:::benchmark_data_smithi122_168655_object99812:head# 2024-03-20T09:26:43.369 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.369 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88c95942:::benchmark_data_smithi122_168655_object114493:head# 2024-03-20T09:26:43.369 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.369 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88c9939c:::benchmark_data_smithi122_168655_object94967:head# 2024-03-20T09:26:43.369 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.369 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88c9c6b4:::benchmark_data_smithi122_168655_object83618:head# 2024-03-20T09:26:43.369 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.369 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88ca2606:::benchmark_data_smithi122_168655_object112848:head# 2024-03-20T09:26:43.369 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.369 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88ca272a:::benchmark_data_smithi122_168655_object91359:head# 2024-03-20T09:26:43.369 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.369 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88ca9c67:::benchmark_data_smithi122_168655_object17522:head# 2024-03-20T09:26:43.369 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.370 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88cb8e78:::benchmark_data_smithi122_168655_object41773:head# 2024-03-20T09:26:43.370 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.370 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88cbd4f5:::benchmark_data_smithi122_168655_object49388:head# 2024-03-20T09:26:43.370 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.370 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88cc44a6:::benchmark_data_smithi122_168655_object115796:head# 2024-03-20T09:26:43.370 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.370 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88ccd6c0:::benchmark_data_smithi122_168655_object58284:head# 2024-03-20T09:26:43.370 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.370 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88cda85b:::benchmark_data_smithi122_168655_object58544:head# 2024-03-20T09:26:43.370 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.370 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88cdeda5:::benchmark_data_smithi122_168655_object118147:head# 2024-03-20T09:26:43.370 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.370 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88cec7a3:::benchmark_data_smithi122_168655_object81384:head# 2024-03-20T09:26:43.371 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.371 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88cfcc59:::benchmark_data_smithi122_168655_object82095:head# 2024-03-20T09:26:43.371 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.371 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88d0177f:::benchmark_data_smithi122_168655_object86085:head# 2024-03-20T09:26:43.371 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.371 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88d028b3:::benchmark_data_smithi122_168655_object50956:head# 2024-03-20T09:26:43.371 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.371 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88d05851:::benchmark_data_smithi122_168655_object118920:head# 2024-03-20T09:26:43.371 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.371 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88d10284:::benchmark_data_smithi122_168655_object113009:head# 2024-03-20T09:26:43.371 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.371 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88d14930:::benchmark_data_smithi122_168655_object119418:head# 2024-03-20T09:26:43.371 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.371 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88d19bf4:::benchmark_data_smithi122_168655_object38351:head# 2024-03-20T09:26:43.372 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.372 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88d25f73:::benchmark_data_smithi122_168655_object52482:head# 2024-03-20T09:26:43.372 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.372 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88d2f276:::benchmark_data_smithi122_168655_object65577:head# 2024-03-20T09:26:43.372 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.372 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88d3320e:::benchmark_data_smithi122_168655_object34814:head# 2024-03-20T09:26:43.372 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.372 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88d3b5c5:::benchmark_data_smithi122_168655_object77126:head# 2024-03-20T09:26:43.372 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.372 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88d532c9:::benchmark_data_smithi122_168655_object92324:head# 2024-03-20T09:26:43.372 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.372 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88d63628:::benchmark_data_smithi122_168655_object47535:head# 2024-03-20T09:26:43.372 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.373 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88d68c37:::benchmark_data_smithi122_168655_object32396:head# 2024-03-20T09:26:43.373 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.373 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88d6d2ed:::benchmark_data_smithi122_168655_object30551:head# 2024-03-20T09:26:43.373 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.373 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88d72e6a:::benchmark_data_smithi122_168655_object18402:head# 2024-03-20T09:26:43.373 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.373 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88d76447:::benchmark_data_smithi122_168655_object83017:head# 2024-03-20T09:26:43.373 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.373 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88d91954:::benchmark_data_smithi122_168655_object47820:head# 2024-03-20T09:26:43.373 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.373 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88d97974:::benchmark_data_smithi122_168655_object34384:head# 2024-03-20T09:26:43.373 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.373 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88dce609:::benchmark_data_smithi122_168655_object67027:head# 2024-03-20T09:26:43.374 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.374 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88dd3891:::benchmark_data_smithi122_168655_object51280:head# 2024-03-20T09:26:43.374 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.374 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88de717b:::benchmark_data_smithi122_168655_object59052:head# 2024-03-20T09:26:43.374 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.374 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88df99a0:::benchmark_data_smithi122_168655_object77403:head# 2024-03-20T09:26:43.374 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.374 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88e24ee0:::benchmark_data_smithi122_168655_object17824:head# 2024-03-20T09:26:43.374 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.374 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88e2e3ba:::benchmark_data_smithi122_168655_object86149:head# 2024-03-20T09:26:43.374 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.374 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88e330da:::benchmark_data_smithi122_168655_object96372:head# 2024-03-20T09:26:43.374 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.374 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88e3e51f:::benchmark_data_smithi122_168655_object19200:head# 2024-03-20T09:26:43.375 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.375 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88e3ea6c:::benchmark_data_smithi122_168655_object92838:head# 2024-03-20T09:26:43.375 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.375 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88e497bc:::benchmark_data_smithi122_168655_object80642:head# 2024-03-20T09:26:43.375 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.375 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88e5aa78:::benchmark_data_smithi122_168655_object51084:head# 2024-03-20T09:26:43.375 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.375 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88e5f31b:::benchmark_data_smithi122_168655_object88601:head# 2024-03-20T09:26:43.375 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.375 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88e6179a:::benchmark_data_smithi122_168655_object114992:head# 2024-03-20T09:26:43.375 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.375 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88e6c27f:::benchmark_data_smithi122_168655_object101428:head# 2024-03-20T09:26:43.375 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.376 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88e8d155:::benchmark_data_smithi122_168655_object85278:head# 2024-03-20T09:26:43.376 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.376 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88ea28e6:::benchmark_data_smithi122_168655_object39237:head# 2024-03-20T09:26:43.376 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.376 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88ea3bb3:::benchmark_data_smithi122_168655_object103464:head# 2024-03-20T09:26:43.376 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.376 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88ea9424:::benchmark_data_smithi122_168655_object30513:head# 2024-03-20T09:26:43.376 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.376 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88ebec7b:::benchmark_data_smithi122_168655_object103506:head# 2024-03-20T09:26:43.376 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.376 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88ec8df7:::benchmark_data_smithi122_168655_object22764:head# 2024-03-20T09:26:43.376 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.376 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88ecba51:::benchmark_data_smithi122_168655_object56346:head# 2024-03-20T09:26:43.376 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.376 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88ecdf8c:::benchmark_data_smithi122_168655_object69131:head# 2024-03-20T09:26:43.377 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.377 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88ee4931:::benchmark_data_smithi122_168655_object41074:head# 2024-03-20T09:26:43.377 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.377 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88eec399:::benchmark_data_smithi122_168655_object116634:head# 2024-03-20T09:26:43.377 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.377 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88f03544:::benchmark_data_smithi122_168655_object50734:head# 2024-03-20T09:26:43.377 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.377 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88f0e5b9:::benchmark_data_smithi122_168655_object21181:head# 2024-03-20T09:26:43.377 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.377 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88f1544c:::benchmark_data_smithi122_168655_object96458:head# 2024-03-20T09:26:43.377 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.377 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88f22bbd:::benchmark_data_smithi122_168655_object32252:head# 2024-03-20T09:26:43.377 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.377 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88f25c6d:::benchmark_data_smithi122_168655_object42129:head# 2024-03-20T09:26:43.378 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.378 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88f2f35d:::benchmark_data_smithi122_168655_object32607:head# 2024-03-20T09:26:43.378 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.378 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88f369a6:::benchmark_data_smithi122_168655_object37019:head# 2024-03-20T09:26:43.378 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.378 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88f3faf3:::benchmark_data_smithi122_168655_object58320:head# 2024-03-20T09:26:43.378 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.378 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88f4123a:::benchmark_data_smithi122_168655_object61575:head# 2024-03-20T09:26:43.378 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.378 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88f59e3c:::benchmark_data_smithi122_168655_object52448:head# 2024-03-20T09:26:43.378 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.378 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88f5ad95:::benchmark_data_smithi122_168655_object93319:head# 2024-03-20T09:26:43.379 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.379 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88f67a36:::benchmark_data_smithi122_168655_object107818:head# 2024-03-20T09:26:43.379 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.379 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88f6d522:::benchmark_data_smithi122_168655_object88274:head# 2024-03-20T09:26:43.379 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.379 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88f787cc:::benchmark_data_smithi122_168655_object57182:head# 2024-03-20T09:26:43.379 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.379 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88f78a28:::benchmark_data_smithi122_168655_object65059:head# 2024-03-20T09:26:43.379 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.379 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88f7edd5:::benchmark_data_smithi122_168655_object43158:head# 2024-03-20T09:26:43.379 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.379 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88f82f39:::benchmark_data_smithi122_168655_object114359:head# 2024-03-20T09:26:43.379 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.379 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88f8e6ef:::benchmark_data_smithi122_168655_object90345:head# 2024-03-20T09:26:43.379 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.380 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88f8f4fc:::benchmark_data_smithi122_168655_object25591:head# 2024-03-20T09:26:43.380 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.380 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88f938e5:::benchmark_data_smithi122_168655_object48269:head# 2024-03-20T09:26:43.380 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.380 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88fa033d:::benchmark_data_smithi122_168655_object76122:head# 2024-03-20T09:26:43.380 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.380 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88fa0d21:::benchmark_data_smithi122_168655_object37237:head# 2024-03-20T09:26:43.380 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.380 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88fba34a:::benchmark_data_smithi122_168655_object18511:head# 2024-03-20T09:26:43.380 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.380 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88fc9694:::benchmark_data_smithi122_168655_object21499:head# 2024-03-20T09:26:43.380 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.380 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88fcc539:::benchmark_data_smithi122_168655_object99829:head# 2024-03-20T09:26:43.380 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.380 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88fd24d4:::benchmark_data_smithi122_168655_object100318:head# 2024-03-20T09:26:43.381 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.381 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88fd3a6b:::benchmark_data_smithi122_168655_object120031:head# 2024-03-20T09:26:43.381 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.381 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88fd9fe7:::benchmark_data_smithi122_168655_object76389:head# 2024-03-20T09:26:43.381 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.381 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88fdb55f:::benchmark_data_smithi122_168655_object110675:head# 2024-03-20T09:26:43.381 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.381 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88fe9e52:::benchmark_data_smithi122_168655_object37009:head# 2024-03-20T09:26:43.381 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.381 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88fec790:::benchmark_data_smithi122_168655_object90454:head# 2024-03-20T09:26:43.381 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.381 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88ffcbef:::benchmark_data_smithi122_168655_object115090:head# 2024-03-20T09:26:43.381 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.381 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:88ffd1f2:::benchmark_data_smithi122_168655_object53137:head# 2024-03-20T09:26:43.382 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.382 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8900ee84:::benchmark_data_smithi122_168655_object72292:head# 2024-03-20T09:26:43.382 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.382 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:890118c9:::benchmark_data_smithi122_168655_object102512:head# 2024-03-20T09:26:43.382 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.382 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8902bb0b:::benchmark_data_smithi122_168655_object51324:head# 2024-03-20T09:26:43.382 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.382 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8902e836:::benchmark_data_smithi122_168655_object11615:head# 2024-03-20T09:26:43.382 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.382 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8902efef:::benchmark_data_smithi122_168655_object74513:head# 2024-03-20T09:26:43.382 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.383 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8903d12d:::benchmark_data_smithi122_168655_object65022:head# 2024-03-20T09:26:43.383 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.383 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8903d983:::benchmark_data_smithi122_168655_object15239:head# 2024-03-20T09:26:43.383 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.383 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:890444ee:::benchmark_data_smithi122_168655_object66054:head# 2024-03-20T09:26:43.383 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.383 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8904b8ec:::benchmark_data_smithi122_168655_object60315:head# 2024-03-20T09:26:43.383 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.383 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89055d0e:::benchmark_data_smithi122_168655_object79386:head# 2024-03-20T09:26:43.383 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.383 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89059a59:::benchmark_data_smithi122_168655_object56077:head# 2024-03-20T09:26:43.383 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.383 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8905b131:::benchmark_data_smithi122_168655_object95560:head# 2024-03-20T09:26:43.384 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.384 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89066abe:::benchmark_data_smithi122_168655_object67454:head# 2024-03-20T09:26:43.384 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.384 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:890671c9:::benchmark_data_smithi122_168655_object92925:head# 2024-03-20T09:26:43.384 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.384 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8907afab:::benchmark_data_smithi122_168655_object93024:head# 2024-03-20T09:26:43.384 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.384 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:890897f4:::benchmark_data_smithi122_168655_object24800:head# 2024-03-20T09:26:43.384 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.384 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8908fca5:::benchmark_data_smithi122_168655_object22873:head# 2024-03-20T09:26:43.384 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.384 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:890999a6:::benchmark_data_smithi122_168655_object76590:head# 2024-03-20T09:26:43.384 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.384 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:890af352:::benchmark_data_smithi122_168655_object87956:head# 2024-03-20T09:26:43.384 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.385 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:890b84d1:::benchmark_data_smithi122_168655_object68813:head# 2024-03-20T09:26:43.385 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.385 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:890bae07:::benchmark_data_smithi122_168655_object82800:head# 2024-03-20T09:26:43.385 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.385 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:890bc44b:::benchmark_data_smithi122_168655_object57001:head# 2024-03-20T09:26:43.385 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.385 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:890cb922:::benchmark_data_smithi122_168655_object52678:head# 2024-03-20T09:26:43.385 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.385 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:890d3bd3:::benchmark_data_smithi122_168655_object63349:head# 2024-03-20T09:26:43.385 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.385 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:890d527e:::benchmark_data_smithi122_168655_object19413:head# 2024-03-20T09:26:43.385 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.385 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:890da661:::benchmark_data_smithi122_168655_object75002:head# 2024-03-20T09:26:43.386 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.386 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:890e7eba:::benchmark_data_smithi122_168655_object82956:head# 2024-03-20T09:26:43.386 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.386 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:890f53d8:::benchmark_data_smithi122_168655_object100573:head# 2024-03-20T09:26:43.386 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.386 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8910665a:::benchmark_data_smithi122_168655_object37035:head# 2024-03-20T09:26:43.386 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.386 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8910b122:::benchmark_data_smithi122_168655_object96846:head# 2024-03-20T09:26:43.386 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.386 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8910feac:::benchmark_data_smithi122_168655_object38837:head# 2024-03-20T09:26:43.386 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.386 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89126172:::benchmark_data_smithi122_168655_object93422:head# 2024-03-20T09:26:43.386 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.386 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8913036c:::benchmark_data_smithi122_168655_object58013:head# 2024-03-20T09:26:43.387 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.387 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89136ce6:::benchmark_data_smithi122_168655_object51075:head# 2024-03-20T09:26:43.387 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.387 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8913732a:::benchmark_data_smithi122_168655_object25312:head# 2024-03-20T09:26:43.387 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.387 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89155932:::benchmark_data_smithi122_168655_object101093:head# 2024-03-20T09:26:43.387 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.387 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:891588e9:::benchmark_data_smithi122_168655_object119063:head# 2024-03-20T09:26:43.387 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.387 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89160be6:::benchmark_data_smithi122_168655_object66417:head# 2024-03-20T09:26:43.387 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.387 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8916b2ca:::benchmark_data_smithi122_168655_object89946:head# 2024-03-20T09:26:43.387 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.387 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8916d8b3:::benchmark_data_smithi122_168655_object111864:head# 2024-03-20T09:26:43.387 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.388 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89174b0f:::benchmark_data_smithi122_168655_object30696:head# 2024-03-20T09:26:43.388 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.388 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8917605b:::benchmark_data_smithi122_168655_object55636:head# 2024-03-20T09:26:43.388 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.388 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:891999bc:::benchmark_data_smithi122_168655_object34491:head# 2024-03-20T09:26:43.388 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.388 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8919df99:::benchmark_data_smithi122_168655_object49356:head# 2024-03-20T09:26:43.388 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.388 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:891acf64:::benchmark_data_smithi122_168655_object42028:head# 2024-03-20T09:26:43.388 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.388 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:891b35c6:::benchmark_data_smithi122_168655_object26959:head# 2024-03-20T09:26:43.388 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.388 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:891b904e:::benchmark_data_smithi122_168655_object60542:head# 2024-03-20T09:26:43.388 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.389 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:891b91c9:::benchmark_data_smithi122_168655_object38850:head# 2024-03-20T09:26:43.389 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.389 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:891bfbdc:::benchmark_data_smithi122_168655_object20666:head# 2024-03-20T09:26:43.389 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.389 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:891c292f:::benchmark_data_smithi122_168655_object13031:head# 2024-03-20T09:26:43.389 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.389 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:891c4c15:::benchmark_data_smithi122_168655_object82373:head# 2024-03-20T09:26:43.389 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.389 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:891cd8fb:::benchmark_data_smithi122_168655_object54225:head# 2024-03-20T09:26:43.389 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.389 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:891ce610:::benchmark_data_smithi122_168655_object100133:head# 2024-03-20T09:26:43.389 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.389 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:891d41b9:::benchmark_data_smithi122_168655_object97229:head# 2024-03-20T09:26:43.389 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.428 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:891dd116:::benchmark_data_smithi122_168655_object104939:head# 2024-03-20T09:26:43.428 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.428 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:891dde73:::benchmark_data_smithi122_168655_object59736:head# 2024-03-20T09:26:43.428 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.428 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:891e0d4f:::benchmark_data_smithi122_168655_object27580:head# 2024-03-20T09:26:43.428 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.428 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:891eb862:::benchmark_data_smithi122_168655_object17444:head# 2024-03-20T09:26:43.428 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.429 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:891efac0:::benchmark_data_smithi122_168655_object48020:head# 2024-03-20T09:26:43.429 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.429 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89204ef5:::benchmark_data_smithi122_168655_object110966:head# 2024-03-20T09:26:43.429 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.429 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:892188c8:::benchmark_data_smithi122_168655_object37140:head# 2024-03-20T09:26:43.429 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.429 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8921bb06:::benchmark_data_smithi122_168655_object31946:head# 2024-03-20T09:26:43.429 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.429 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89230b21:::benchmark_data_smithi122_168655_object54233:head# 2024-03-20T09:26:43.429 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.429 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8923d0f5:::benchmark_data_smithi122_168655_object88484:head# 2024-03-20T09:26:43.429 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.429 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8924442e:::benchmark_data_smithi122_168655_object66687:head# 2024-03-20T09:26:43.429 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.430 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89244548:::benchmark_data_smithi122_168655_object109851:head# 2024-03-20T09:26:43.430 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.430 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89268792:::benchmark_data_smithi122_168655_object38201:head# 2024-03-20T09:26:43.430 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.430 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8926e4e8:::benchmark_data_smithi122_168655_object62380:head# 2024-03-20T09:26:43.430 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.430 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89271163:::benchmark_data_smithi122_168655_object88381:head# 2024-03-20T09:26:43.430 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.430 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89272344:::benchmark_data_smithi122_168655_object72452:head# 2024-03-20T09:26:43.430 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.430 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89272b04:::benchmark_data_smithi122_168655_object28644:head# 2024-03-20T09:26:43.430 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.430 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89272bc0:::benchmark_data_smithi122_168655_object107119:head# 2024-03-20T09:26:43.430 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.431 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:892771c4:::benchmark_data_smithi122_168655_object54988:head# 2024-03-20T09:26:43.431 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.431 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8927cdd0:::benchmark_data_smithi122_168655_object63397:head# 2024-03-20T09:26:43.431 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.431 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:892805ac:::benchmark_data_smithi122_168655_object49198:head# 2024-03-20T09:26:43.431 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.431 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:892877f6:::benchmark_data_smithi122_168655_object56685:head# 2024-03-20T09:26:43.431 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.431 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89299e27:::benchmark_data_smithi122_168655_object44409:head# 2024-03-20T09:26:43.431 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.431 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8929f5e0:::benchmark_data_smithi122_168655_object86269:head# 2024-03-20T09:26:43.431 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.431 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:892a1d8f:::benchmark_data_smithi122_168655_object49284:head# 2024-03-20T09:26:43.431 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.432 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:892a704d:::benchmark_data_smithi122_168655_object30094:head# 2024-03-20T09:26:43.432 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.432 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:892a77a6:::benchmark_data_smithi122_168655_object88420:head# 2024-03-20T09:26:43.432 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.432 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:892b3dd1:::benchmark_data_smithi122_168655_object21458:head# 2024-03-20T09:26:43.432 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.432 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:892b57cb:::benchmark_data_smithi122_168655_object91219:head# 2024-03-20T09:26:43.432 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.432 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:892bc847:::benchmark_data_smithi122_168655_object102488:head# 2024-03-20T09:26:43.432 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.432 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:892bdf7e:::benchmark_data_smithi122_168655_object78152:head# 2024-03-20T09:26:43.432 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.432 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:892c0f8e:::benchmark_data_smithi122_168655_object47615:head# 2024-03-20T09:26:43.432 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.433 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:892d5662:::benchmark_data_smithi122_168655_object41382:head# 2024-03-20T09:26:43.433 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.433 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:892f2971:::benchmark_data_smithi122_168655_object74186:head# 2024-03-20T09:26:43.433 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.433 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:892f7009:::benchmark_data_smithi122_168655_object38543:head# 2024-03-20T09:26:43.433 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.433 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:892f9de7:::benchmark_data_smithi122_168655_object11735:head# 2024-03-20T09:26:43.433 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.433 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:892fe1e6:::benchmark_data_smithi122_168655_object80246:head# 2024-03-20T09:26:43.433 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.433 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8930ce7e:::benchmark_data_smithi122_168655_object40218:head# 2024-03-20T09:26:43.433 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.433 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89314294:::benchmark_data_smithi122_168655_object24377:head# 2024-03-20T09:26:43.433 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.434 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89316ac7:::benchmark_data_smithi122_168655_object74515:head# 2024-03-20T09:26:43.434 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.434 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8931a8a6:::benchmark_data_smithi122_168655_object22187:head# 2024-03-20T09:26:43.434 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.434 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8931b287:::benchmark_data_smithi122_168655_object74708:head# 2024-03-20T09:26:43.434 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.434 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89323c0b:::benchmark_data_smithi122_168655_object88199:head# 2024-03-20T09:26:43.434 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.434 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89329a1f:::benchmark_data_smithi122_168655_object72909:head# 2024-03-20T09:26:43.434 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.434 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8932a07a:::benchmark_data_smithi122_168655_object106729:head# 2024-03-20T09:26:43.434 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.435 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8935012d:::benchmark_data_smithi122_168655_object50385:head# 2024-03-20T09:26:43.435 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.435 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89350509:::benchmark_data_smithi122_168655_object78266:head# 2024-03-20T09:26:43.435 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.435 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8936010e:::benchmark_data_smithi122_168655_object28333:head# 2024-03-20T09:26:43.435 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.435 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8936ba1d:::benchmark_data_smithi122_168655_object36510:head# 2024-03-20T09:26:43.435 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.435 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8936e6ee:::benchmark_data_smithi122_168655_object79343:head# 2024-03-20T09:26:43.435 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.435 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89395074:::benchmark_data_smithi122_168655_object106627:head# 2024-03-20T09:26:43.435 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.435 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8939e16d:::benchmark_data_smithi122_168655_object107530:head# 2024-03-20T09:26:43.436 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.436 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:893a4162:::benchmark_data_smithi122_168655_object114036:head# 2024-03-20T09:26:43.436 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.436 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:893a9077:::benchmark_data_smithi122_168655_object69618:head# 2024-03-20T09:26:43.436 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.436 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:893aa760:::benchmark_data_smithi122_168655_object31438:head# 2024-03-20T09:26:43.436 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.436 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:893ae793:::benchmark_data_smithi122_168655_object73478:head# 2024-03-20T09:26:43.436 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.436 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:893af018:::benchmark_data_smithi122_168655_object37351:head# 2024-03-20T09:26:43.436 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.436 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:893b0479:::benchmark_data_smithi122_168655_object52543:head# 2024-03-20T09:26:43.436 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.436 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:893faa27:::benchmark_data_smithi122_168655_object12179:head# 2024-03-20T09:26:43.437 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.437 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:894054a5:::benchmark_data_smithi122_168655_object101400:head# 2024-03-20T09:26:43.437 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.437 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89406669:::benchmark_data_smithi122_168655_object65334:head# 2024-03-20T09:26:43.437 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.437 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8940a29a:::benchmark_data_smithi122_168655_object52110:head# 2024-03-20T09:26:43.437 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.437 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8941675f:::benchmark_data_smithi122_168655_object112272:head# 2024-03-20T09:26:43.437 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.437 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8941a8c4:::benchmark_data_smithi122_168655_object104495:head# 2024-03-20T09:26:43.437 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.437 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:894238c1:::benchmark_data_smithi122_168655_object21412:head# 2024-03-20T09:26:43.438 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.438 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8942c42c:::benchmark_data_smithi122_168655_object114304:head# 2024-03-20T09:26:43.438 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.438 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8942feaf:::benchmark_data_smithi122_168655_object18685:head# 2024-03-20T09:26:43.438 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.438 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8943e8be:::benchmark_data_smithi122_168655_object111780:head# 2024-03-20T09:26:43.438 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.438 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89442336:::benchmark_data_smithi122_168655_object90665:head# 2024-03-20T09:26:43.438 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.438 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8944aa9e:::benchmark_data_smithi122_168655_object22735:head# 2024-03-20T09:26:43.438 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.438 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8944b687:::benchmark_data_smithi122_168655_object114850:head# 2024-03-20T09:26:43.439 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.439 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8944bde4:::benchmark_data_smithi122_168655_object108617:head# 2024-03-20T09:26:43.439 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.439 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8944ce40:::benchmark_data_smithi122_168655_object66272:head# 2024-03-20T09:26:43.439 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.439 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:894645ff:::benchmark_data_smithi122_168655_object21891:head# 2024-03-20T09:26:43.439 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.439 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89473e18:::benchmark_data_smithi122_168655_object105399:head# 2024-03-20T09:26:43.439 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.439 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8947a1f7:::benchmark_data_smithi122_168655_object103030:head# 2024-03-20T09:26:43.439 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.439 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89481f9b:::benchmark_data_smithi122_168655_object55020:head# 2024-03-20T09:26:43.439 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.440 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8948aa04:::benchmark_data_smithi122_168655_object104170:head# 2024-03-20T09:26:43.440 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.440 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:894972f2:::benchmark_data_smithi122_168655_object24636:head# 2024-03-20T09:26:43.440 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.440 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8949b983:::benchmark_data_smithi122_168655_object80502:head# 2024-03-20T09:26:43.440 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.440 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8949d1d3:::benchmark_data_smithi122_168655_object73255:head# 2024-03-20T09:26:43.440 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.440 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:894a544c:::benchmark_data_smithi122_168655_object67056:head# 2024-03-20T09:26:43.440 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.440 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:894b1042:::benchmark_data_smithi122_168655_object114196:head# 2024-03-20T09:26:43.440 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.441 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:894bbbcb:::benchmark_data_smithi122_168655_object119194:head# 2024-03-20T09:26:43.441 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.441 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:894dfdf4:::benchmark_data_smithi122_168655_object16081:head# 2024-03-20T09:26:43.441 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.441 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:894e0181:::benchmark_data_smithi122_168655_object88138:head# 2024-03-20T09:26:43.441 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.441 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:894e0394:::benchmark_data_smithi122_168655_object53811:head# 2024-03-20T09:26:43.441 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.441 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:894e2c78:::benchmark_data_smithi122_168655_object31399:head# 2024-03-20T09:26:43.441 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.441 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:894e631a:::benchmark_data_smithi122_168655_object74405:head# 2024-03-20T09:26:43.441 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.441 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:894fc83b:::benchmark_data_smithi122_168655_object99561:head# 2024-03-20T09:26:43.441 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.442 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:894fdac8:::benchmark_data_smithi122_168655_object104715:head# 2024-03-20T09:26:43.442 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.442 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8950dad5:::benchmark_data_smithi122_168655_object33592:head# 2024-03-20T09:26:43.442 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.442 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8951057b:::benchmark_data_smithi122_168655_object20328:head# 2024-03-20T09:26:43.442 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.442 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:895233fc:::benchmark_data_smithi122_168655_object64808:head# 2024-03-20T09:26:43.442 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.442 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89528050:::benchmark_data_smithi122_168655_object112836:head# 2024-03-20T09:26:43.442 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.442 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89528ca1:::benchmark_data_smithi122_168655_object108198:head# 2024-03-20T09:26:43.442 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.442 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8952a6da:::benchmark_data_smithi122_168655_object35803:head# 2024-03-20T09:26:43.443 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.443 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89541353:::benchmark_data_smithi122_168655_object72647:head# 2024-03-20T09:26:43.443 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.443 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8955724f:::benchmark_data_smithi122_168655_object78013:head# 2024-03-20T09:26:43.443 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.443 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:895678d5:::benchmark_data_smithi122_168655_object97848:head# 2024-03-20T09:26:43.443 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.443 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8956f430:::benchmark_data_smithi122_168655_object46434:head# 2024-03-20T09:26:43.443 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.443 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89574ae2:::benchmark_data_smithi122_168655_object14034:head# 2024-03-20T09:26:43.443 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.443 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89576d5d:::benchmark_data_smithi122_168655_object66076:head# 2024-03-20T09:26:43.443 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.443 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89578252:::benchmark_data_smithi122_168655_object37529:head# 2024-03-20T09:26:43.444 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.444 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89583f27:::benchmark_data_smithi122_168655_object69244:head# 2024-03-20T09:26:43.444 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.444 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89588683:::benchmark_data_smithi122_168655_object22556:head# 2024-03-20T09:26:43.444 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.444 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89592cea:::benchmark_data_smithi122_168655_object15467:head# 2024-03-20T09:26:43.444 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.444 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89597e2b:::benchmark_data_smithi122_168655_object13275:head# 2024-03-20T09:26:43.444 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.444 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:895acb0b:::benchmark_data_smithi122_168655_object44950:head# 2024-03-20T09:26:43.444 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.444 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:895ba273:::benchmark_data_smithi122_168655_object64159:head# 2024-03-20T09:26:43.445 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.445 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:895be990:::benchmark_data_smithi122_168655_object45219:head# 2024-03-20T09:26:43.445 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.445 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:895c169c:::benchmark_data_smithi122_168655_object98981:head# 2024-03-20T09:26:43.445 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.445 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:895cc6cd:::benchmark_data_smithi122_168655_object104601:head# 2024-03-20T09:26:43.445 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.445 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:895d5502:::benchmark_data_smithi122_168655_object36500:head# 2024-03-20T09:26:43.445 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.445 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:895ebbb0:::benchmark_data_smithi122_168655_object10232:head# 2024-03-20T09:26:43.445 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.445 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8960e5fd:::benchmark_data_smithi122_168655_object85161:head# 2024-03-20T09:26:43.445 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.445 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8960ed73:::benchmark_data_smithi122_168655_object50905:head# 2024-03-20T09:26:43.446 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.446 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89611063:::benchmark_data_smithi122_168655_object106600:head# 2024-03-20T09:26:43.446 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.446 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89614099:::benchmark_data_smithi122_168655_object109284:head# 2024-03-20T09:26:43.446 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.446 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8961b6b5:::benchmark_data_smithi122_168655_object105466:head# 2024-03-20T09:26:43.446 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.446 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8961d411:::benchmark_data_smithi122_168655_object28236:head# 2024-03-20T09:26:43.446 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.446 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:896263ef:::benchmark_data_smithi122_168655_object113925:head# 2024-03-20T09:26:43.446 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.446 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8962c1ad:::benchmark_data_smithi122_168655_object58738:head# 2024-03-20T09:26:43.446 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.446 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:896311ed:::benchmark_data_smithi122_168655_object12394:head# 2024-03-20T09:26:43.447 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.447 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89632fa2:::benchmark_data_smithi122_168655_object103950:head# 2024-03-20T09:26:43.447 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.447 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8963915d:::benchmark_data_smithi122_168655_object40769:head# 2024-03-20T09:26:43.447 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.447 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89643b2c:::benchmark_data_smithi122_168655_object91852:head# 2024-03-20T09:26:43.447 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.447 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8964570d:::benchmark_data_smithi122_168655_object47265:head# 2024-03-20T09:26:43.447 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.447 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8965148c:::benchmark_data_smithi122_168655_object90889:head# 2024-03-20T09:26:43.447 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.447 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89659f5b:::benchmark_data_smithi122_168655_object84468:head# 2024-03-20T09:26:43.447 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.447 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8966bd19:::benchmark_data_smithi122_168655_object80378:head# 2024-03-20T09:26:43.448 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.448 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8966fff9:::benchmark_data_smithi122_168655_object102182:head# 2024-03-20T09:26:43.448 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.448 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8968d578:::benchmark_data_smithi122_168655_object19345:head# 2024-03-20T09:26:43.448 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.448 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:896916b9:::benchmark_data_smithi122_168655_object44047:head# 2024-03-20T09:26:43.448 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.448 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8969d465:::benchmark_data_smithi122_168655_object35445:head# 2024-03-20T09:26:43.448 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.448 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8969e15c:::benchmark_data_smithi122_168655_object108050:head# 2024-03-20T09:26:43.448 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.448 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:896a38f2:::benchmark_data_smithi122_168655_object17445:head# 2024-03-20T09:26:43.448 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.448 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:896aba59:::benchmark_data_smithi122_168655_object80178:head# 2024-03-20T09:26:43.449 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.449 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:896aeb7d:::benchmark_data_smithi122_168655_object69517:head# 2024-03-20T09:26:43.449 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.449 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:896b7786:::benchmark_data_smithi122_168655_object69850:head# 2024-03-20T09:26:43.449 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.449 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:896cc4be:::benchmark_data_smithi122_168655_object71003:head# 2024-03-20T09:26:43.449 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.449 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:896d572a:::benchmark_data_smithi122_168655_object82760:head# 2024-03-20T09:26:43.449 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.449 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:896dc0f5:::benchmark_data_smithi122_168655_object26655:head# 2024-03-20T09:26:43.449 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.449 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:896e3c67:::benchmark_data_smithi122_168655_object37339:head# 2024-03-20T09:26:43.449 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.449 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:896ef21e:::benchmark_data_smithi122_168655_object109121:head# 2024-03-20T09:26:43.450 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.450 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:896fc77d:::benchmark_data_smithi122_168655_object113458:head# 2024-03-20T09:26:43.450 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.450 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89705e21:::benchmark_data_smithi122_168655_object42785:head# 2024-03-20T09:26:43.450 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.450 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8970712e:::benchmark_data_smithi122_168655_object92574:head# 2024-03-20T09:26:43.450 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.450 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89708d88:::benchmark_data_smithi122_168655_object36754:head# 2024-03-20T09:26:43.450 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.450 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8971bca5:::benchmark_data_smithi122_168655_object24917:head# 2024-03-20T09:26:43.450 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.450 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8971e1ae:::benchmark_data_smithi122_168655_object117015:head# 2024-03-20T09:26:43.450 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.451 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89733dfc:::benchmark_data_smithi122_168655_object11331:head# 2024-03-20T09:26:43.451 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.451 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89735650:::benchmark_data_smithi122_168655_object47289:head# 2024-03-20T09:26:43.451 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.451 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8973a53e:::benchmark_data_smithi122_168655_object108013:head# 2024-03-20T09:26:43.451 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.451 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89744ddd:::benchmark_data_smithi122_168655_object84982:head# 2024-03-20T09:26:43.451 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.451 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8974c126:::benchmark_data_smithi122_168655_object52891:head# 2024-03-20T09:26:43.451 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.451 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:897677c3:::benchmark_data_smithi122_168655_object104697:head# 2024-03-20T09:26:43.451 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.451 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:897740d2:::benchmark_data_smithi122_168655_object107229:head# 2024-03-20T09:26:43.451 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.451 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8977e654:::benchmark_data_smithi122_168655_object22967:head# 2024-03-20T09:26:43.452 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.452 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8977fc31:::benchmark_data_smithi122_168655_object90429:head# 2024-03-20T09:26:43.452 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.452 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8978098d:::benchmark_data_smithi122_168655_object66858:head# 2024-03-20T09:26:43.452 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.452 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89792994:::benchmark_data_smithi122_168655_object95861:head# 2024-03-20T09:26:43.452 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.452 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8979538d:::benchmark_data_smithi122_168655_object22317:head# 2024-03-20T09:26:43.452 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.452 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8979f352:::benchmark_data_smithi122_168655_object104219:head# 2024-03-20T09:26:43.452 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.452 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:897a68f7:::benchmark_data_smithi122_168655_object90130:head# 2024-03-20T09:26:43.452 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.452 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:897b502e:::benchmark_data_smithi122_168655_object69477:head# 2024-03-20T09:26:43.453 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.453 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:897bf96d:::benchmark_data_smithi122_168655_object26626:head# 2024-03-20T09:26:43.453 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.453 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:897c760b:::benchmark_data_smithi122_168655_object33336:head# 2024-03-20T09:26:43.453 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.453 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:897cece5:::benchmark_data_smithi122_168655_object83756:head# 2024-03-20T09:26:43.453 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.453 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:897d067a:::benchmark_data_smithi122_168655_object90589:head# 2024-03-20T09:26:43.453 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.453 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:897f581f:::benchmark_data_smithi122_168655_object27505:head# 2024-03-20T09:26:43.453 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.453 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:897f9b71:::benchmark_data_smithi122_168655_object101243:head# 2024-03-20T09:26:43.453 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.453 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:897fbf6f:::benchmark_data_smithi122_168655_object99162:head# 2024-03-20T09:26:43.454 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.454 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:897fce37:::benchmark_data_smithi122_168655_object83469:head# 2024-03-20T09:26:43.454 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.454 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:897fe4dc:::benchmark_data_smithi122_168655_object54386:head# 2024-03-20T09:26:43.454 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.454 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89804a8b:::benchmark_data_smithi122_168655_object61408:head# 2024-03-20T09:26:43.454 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.454 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89809b35:::benchmark_data_smithi122_168655_object78588:head# 2024-03-20T09:26:43.454 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.454 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8980e2d8:::benchmark_data_smithi122_168655_object52835:head# 2024-03-20T09:26:43.454 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.454 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:898168a4:::benchmark_data_smithi122_168655_object107892:head# 2024-03-20T09:26:43.454 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.454 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8981b468:::benchmark_data_smithi122_168655_object25670:head# 2024-03-20T09:26:43.455 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.455 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8981c1f1:::benchmark_data_smithi122_168655_object55882:head# 2024-03-20T09:26:43.455 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.455 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8982aa81:::benchmark_data_smithi122_168655_object113949:head# 2024-03-20T09:26:43.455 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.455 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:898375d0:::benchmark_data_smithi122_168655_object69356:head# 2024-03-20T09:26:43.455 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.455 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89841a3a:::benchmark_data_smithi122_168655_object70018:head# 2024-03-20T09:26:43.455 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.455 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89842888:::benchmark_data_smithi122_168655_object76325:head# 2024-03-20T09:26:43.455 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.455 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89849a2b:::benchmark_data_smithi122_168655_object54210:head# 2024-03-20T09:26:43.455 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.455 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8984ba29:::benchmark_data_smithi122_168655_object94833:head# 2024-03-20T09:26:43.455 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.456 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8984d298:::benchmark_data_smithi122_168655_object103192:head# 2024-03-20T09:26:43.456 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.456 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89851f94:::benchmark_data_smithi122_168655_object99015:head# 2024-03-20T09:26:43.456 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.456 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8986187c:::benchmark_data_smithi122_168655_object73249:head# 2024-03-20T09:26:43.456 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.456 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89862e52:::benchmark_data_smithi122_168655_object111786:head# 2024-03-20T09:26:43.456 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.456 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8986bd44:::benchmark_data_smithi122_168655_object23454:head# 2024-03-20T09:26:43.456 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.456 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8986f9c5:::benchmark_data_smithi122_168655_object81316:head# 2024-03-20T09:26:43.456 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.456 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:898992a3:::benchmark_data_smithi122_168655_object30050:head# 2024-03-20T09:26:43.456 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.457 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:898a4bf5:::benchmark_data_smithi122_168655_object104051:head# 2024-03-20T09:26:43.457 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.457 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:898a9a79:::benchmark_data_smithi122_168655_object59285:head# 2024-03-20T09:26:43.457 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.457 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:898ac151:::benchmark_data_smithi122_168655_object77972:head# 2024-03-20T09:26:43.457 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.457 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:898b15d9:::benchmark_data_smithi122_168655_object72970:head# 2024-03-20T09:26:43.457 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.457 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:898b9023:::benchmark_data_smithi122_168655_object82720:head# 2024-03-20T09:26:43.457 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.457 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:898c2cc9:::benchmark_data_smithi122_168655_object90225:head# 2024-03-20T09:26:43.457 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.457 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:898c458e:::benchmark_data_smithi122_168655_object12338:head# 2024-03-20T09:26:43.457 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.457 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:898cb27f:::benchmark_data_smithi122_168655_object68528:head# 2024-03-20T09:26:43.458 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.458 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:898cc31e:::benchmark_data_smithi122_168655_object13687:head# 2024-03-20T09:26:43.458 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.458 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89917d09:::benchmark_data_smithi122_168655_object48927:head# 2024-03-20T09:26:43.458 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.458 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8991af4c:::benchmark_data_smithi122_168655_object44017:head# 2024-03-20T09:26:43.458 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.458 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8991bc48:::benchmark_data_smithi122_168655_object49416:head# 2024-03-20T09:26:43.458 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.458 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8992568a:::benchmark_data_smithi122_168655_object73989:head# 2024-03-20T09:26:43.458 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.458 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8993188f:::benchmark_data_smithi122_168655_object53863:head# 2024-03-20T09:26:43.458 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.459 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:899377f4:::benchmark_data_smithi122_168655_object56589:head# 2024-03-20T09:26:43.459 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.459 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:899398a0:::benchmark_data_smithi122_168655_object16602:head# 2024-03-20T09:26:43.459 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.459 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8993d76c:::benchmark_data_smithi122_168655_object50530:head# 2024-03-20T09:26:43.459 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.459 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8994c2bc:::benchmark_data_smithi122_168655_object109799:head# 2024-03-20T09:26:43.459 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.459 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89963f18:::benchmark_data_smithi122_168655_object73531:head# 2024-03-20T09:26:43.459 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.459 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:899706f6:::benchmark_data_smithi122_168655_object42764:head# 2024-03-20T09:26:43.459 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.459 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:899707a6:::benchmark_data_smithi122_168655_object110189:head# 2024-03-20T09:26:43.459 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.459 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:899769f4:::benchmark_data_smithi122_168655_object88521:head# 2024-03-20T09:26:43.460 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.460 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8997b632:::benchmark_data_smithi122_168655_object33668:head# 2024-03-20T09:26:43.460 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.460 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:899901a4:::benchmark_data_smithi122_168655_object70270:head# 2024-03-20T09:26:43.460 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.460 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:899947e3:::benchmark_data_smithi122_168655_object94488:head# 2024-03-20T09:26:43.460 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.460 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8999804c:::benchmark_data_smithi122_168655_object29826:head# 2024-03-20T09:26:43.460 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.460 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8999d672:::benchmark_data_smithi122_168655_object59896:head# 2024-03-20T09:26:43.460 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.460 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:899b0483:::benchmark_data_smithi122_168655_object53308:head# 2024-03-20T09:26:43.460 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.460 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:899b20ec:::benchmark_data_smithi122_168655_object84822:head# 2024-03-20T09:26:43.461 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.461 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:899b3b90:::benchmark_data_smithi122_168655_object86394:head# 2024-03-20T09:26:43.461 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.461 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:899bbbb5:::benchmark_data_smithi122_168655_object47296:head# 2024-03-20T09:26:43.461 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.461 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:899c4357:::benchmark_data_smithi122_168655_object62708:head# 2024-03-20T09:26:43.461 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.461 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:899ce2b4:::benchmark_data_smithi122_168655_object89361:head# 2024-03-20T09:26:43.461 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.461 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:899dad31:::benchmark_data_smithi122_168655_object93946:head# 2024-03-20T09:26:43.461 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.461 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:899e013b:::benchmark_data_smithi122_168655_object113079:head# 2024-03-20T09:26:43.461 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.461 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:899e399a:::benchmark_data_smithi122_168655_object65538:head# 2024-03-20T09:26:43.462 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.462 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:899f4d0b:::benchmark_data_smithi122_168655_object57369:head# 2024-03-20T09:26:43.462 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.462 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89a06275:::benchmark_data_smithi122_168655_object57901:head# 2024-03-20T09:26:43.462 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.462 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89a0d521:::benchmark_data_smithi122_168655_object115079:head# 2024-03-20T09:26:43.462 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.462 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89a130d5:::benchmark_data_smithi122_168655_object94923:head# 2024-03-20T09:26:43.462 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.462 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89a1314b:::benchmark_data_smithi122_168655_object77371:head# 2024-03-20T09:26:43.462 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.462 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89a3b341:::benchmark_data_smithi122_168655_object91647:head# 2024-03-20T09:26:43.462 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.463 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89a3bf4b:::benchmark_data_smithi122_168655_object98869:head# 2024-03-20T09:26:43.463 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.463 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89a501c1:::benchmark_data_smithi122_168655_object16849:head# 2024-03-20T09:26:43.463 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.463 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89a67fba:::benchmark_data_smithi122_168655_object120064:head# 2024-03-20T09:26:43.463 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.463 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89a6bb45:::benchmark_data_smithi122_168655_object37444:head# 2024-03-20T09:26:43.463 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.463 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89a7b97e:::benchmark_data_smithi122_168655_object103039:head# 2024-03-20T09:26:43.463 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.463 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89a82fc9:::benchmark_data_smithi122_168655_object13155:head# 2024-03-20T09:26:43.463 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.463 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89a8489f:::benchmark_data_smithi122_168655_object51923:head# 2024-03-20T09:26:43.463 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.464 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89a853ac:::benchmark_data_smithi122_168655_object105414:head# 2024-03-20T09:26:43.464 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.464 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89a8b6b6:::benchmark_data_smithi122_168655_object93876:head# 2024-03-20T09:26:43.464 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.464 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89a8be3e:::benchmark_data_smithi122_168655_object32656:head# 2024-03-20T09:26:43.464 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.464 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89a94f6a:::benchmark_data_smithi122_168655_object79522:head# 2024-03-20T09:26:43.464 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.464 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89aa7e37:::benchmark_data_smithi122_168655_object10325:head# 2024-03-20T09:26:43.464 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.464 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89aae8ca:::benchmark_data_smithi122_168655_object68730:head# 2024-03-20T09:26:43.464 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.464 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89abc733:::benchmark_data_smithi122_168655_object13099:head# 2024-03-20T09:26:43.465 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.465 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89abe253:::benchmark_data_smithi122_168655_object61577:head# 2024-03-20T09:26:43.465 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.465 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89adff06:::benchmark_data_smithi122_168655_object34810:head# 2024-03-20T09:26:43.465 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.465 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89ae28ac:::benchmark_data_smithi122_168655_object47033:head# 2024-03-20T09:26:43.465 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.465 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89ae30e7:::benchmark_data_smithi122_168655_object48451:head# 2024-03-20T09:26:43.465 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.465 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89af0106:::benchmark_data_smithi122_168655_object99106:head# 2024-03-20T09:26:43.465 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.465 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89af4475:::benchmark_data_smithi122_168655_object73436:head# 2024-03-20T09:26:43.465 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.465 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89b08e33:::benchmark_data_smithi122_168655_object75577:head# 2024-03-20T09:26:43.465 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.465 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89b0e219:::benchmark_data_smithi122_168655_object105333:head# 2024-03-20T09:26:43.466 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.466 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89b10e92:::benchmark_data_smithi122_168655_object47135:head# 2024-03-20T09:26:43.466 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.466 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89b135f9:::benchmark_data_smithi122_168655_object90993:head# 2024-03-20T09:26:43.466 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.466 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89b15146:::benchmark_data_smithi122_168655_object64661:head# 2024-03-20T09:26:43.466 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.466 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89b18708:::benchmark_data_smithi122_168655_object13375:head# 2024-03-20T09:26:43.466 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.466 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89b43535:::benchmark_data_smithi122_168655_object11024:head# 2024-03-20T09:26:43.466 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.466 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89b47b4b:::benchmark_data_smithi122_168655_object66096:head# 2024-03-20T09:26:43.466 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.467 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89b4cb66:::benchmark_data_smithi122_168655_object61185:head# 2024-03-20T09:26:43.467 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.535 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89b57194:::benchmark_data_smithi122_168655_object84578:head# 2024-03-20T09:26:43.535 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.535 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89b62232:::benchmark_data_smithi122_168655_object38522:head# 2024-03-20T09:26:43.536 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.536 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89b62e32:::benchmark_data_smithi122_168655_object78868:head# 2024-03-20T09:26:43.536 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.536 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89b6601f:::benchmark_data_smithi122_168655_object45768:head# 2024-03-20T09:26:43.536 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.536 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89b70b1a:::benchmark_data_smithi122_168655_object105296:head# 2024-03-20T09:26:43.536 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.536 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89b7b346:::benchmark_data_smithi122_168655_object110884:head# 2024-03-20T09:26:43.536 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.536 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89b7c803:::benchmark_data_smithi122_168655_object61475:head# 2024-03-20T09:26:43.536 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.536 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89b862f2:::benchmark_data_smithi122_168655_object58755:head# 2024-03-20T09:26:43.537 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.537 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89b88b21:::benchmark_data_smithi122_168655_object113400:head# 2024-03-20T09:26:43.537 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.537 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89b8d33a:::benchmark_data_smithi122_168655_object88309:head# 2024-03-20T09:26:43.537 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.537 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89b8e852:::benchmark_data_smithi122_168655_object60030:head# 2024-03-20T09:26:43.537 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.537 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89b9310c:::benchmark_data_smithi122_168655_object82103:head# 2024-03-20T09:26:43.537 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.537 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89b93207:::benchmark_data_smithi122_168655_object65631:head# 2024-03-20T09:26:43.537 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.537 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89b9b385:::benchmark_data_smithi122_168655_object25367:head# 2024-03-20T09:26:43.538 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.538 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89bb639c:::benchmark_data_smithi122_168655_object62701:head# 2024-03-20T09:26:43.538 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.538 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89bb6ffb:::benchmark_data_smithi122_168655_object83974:head# 2024-03-20T09:26:43.538 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.538 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89bceff5:::benchmark_data_smithi122_168655_object88261:head# 2024-03-20T09:26:43.538 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.538 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89bd7790:::benchmark_data_smithi122_168655_object45650:head# 2024-03-20T09:26:43.538 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.538 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89bda29d:::benchmark_data_smithi122_168655_object82399:head# 2024-03-20T09:26:43.538 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.538 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89bdfbf0:::benchmark_data_smithi122_168655_object89584:head# 2024-03-20T09:26:43.539 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.539 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89be8815:::benchmark_data_smithi122_168655_object111719:head# 2024-03-20T09:26:43.539 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.539 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89be9629:::benchmark_data_smithi122_168655_object112470:head# 2024-03-20T09:26:43.539 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.539 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89bea5e8:::benchmark_data_smithi122_168655_object35075:head# 2024-03-20T09:26:43.539 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.539 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89bebbc9:::benchmark_data_smithi122_168655_object35792:head# 2024-03-20T09:26:43.539 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.539 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89c076aa:::benchmark_data_smithi122_168655_object66295:head# 2024-03-20T09:26:43.539 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.539 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89c17fd2:::benchmark_data_smithi122_168655_object43621:head# 2024-03-20T09:26:43.539 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.540 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89c19889:::benchmark_data_smithi122_168655_object51245:head# 2024-03-20T09:26:43.540 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.540 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89c28638:::benchmark_data_smithi122_168655_object15934:head# 2024-03-20T09:26:43.540 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.540 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89c2a0d5:::benchmark_data_smithi122_168655_object30171:head# 2024-03-20T09:26:43.540 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.540 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89c2ff6f:::benchmark_data_smithi122_168655_object19472:head# 2024-03-20T09:26:43.540 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.540 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89c50957:::benchmark_data_smithi122_168655_object41439:head# 2024-03-20T09:26:43.540 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.540 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89c54439:::benchmark_data_smithi122_168655_object14309:head# 2024-03-20T09:26:43.540 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.540 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89c5a1a3:::benchmark_data_smithi122_168655_object98469:head# 2024-03-20T09:26:43.541 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.541 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89c6ee31:::benchmark_data_smithi122_168655_object93590:head# 2024-03-20T09:26:43.541 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.541 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89c82771:::benchmark_data_smithi122_168655_object88000:head# 2024-03-20T09:26:43.541 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.541 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89c8e982:::benchmark_data_smithi122_168655_object104132:head# 2024-03-20T09:26:43.541 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.541 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89c95a74:::benchmark_data_smithi122_168655_object99476:head# 2024-03-20T09:26:43.541 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.541 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89c97012:::benchmark_data_smithi122_168655_object40150:head# 2024-03-20T09:26:43.541 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.541 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89c97cb7:::benchmark_data_smithi122_168655_object51700:head# 2024-03-20T09:26:43.541 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.542 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89c9d475:::benchmark_data_smithi122_168655_object63950:head# 2024-03-20T09:26:43.542 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.542 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89ca1136:::benchmark_data_smithi122_168655_object28451:head# 2024-03-20T09:26:43.542 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.542 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89ca480a:::benchmark_data_smithi122_168655_object96416:head# 2024-03-20T09:26:43.542 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.542 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89ca66ef:::benchmark_data_smithi122_168655_object70023:head# 2024-03-20T09:26:43.542 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.542 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89caf004:::benchmark_data_smithi122_168655_object115778:head# 2024-03-20T09:26:43.542 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.542 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89cb8d00:::benchmark_data_smithi122_168655_object50099:head# 2024-03-20T09:26:43.543 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.543 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89cb9135:::benchmark_data_smithi122_168655_object119965:head# 2024-03-20T09:26:43.543 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.543 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89cbd671:::benchmark_data_smithi122_168655_object78371:head# 2024-03-20T09:26:43.543 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.543 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89cc0843:::benchmark_data_smithi122_168655_object94318:head# 2024-03-20T09:26:43.543 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.543 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89cc3940:::benchmark_data_smithi122_168655_object55318:head# 2024-03-20T09:26:43.543 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.543 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89cc6871:::benchmark_data_smithi122_168655_object35093:head# 2024-03-20T09:26:43.543 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.543 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89ccb185:::benchmark_data_smithi122_168655_object40908:head# 2024-03-20T09:26:43.544 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.544 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89ccddce:::benchmark_data_smithi122_168655_object28930:head# 2024-03-20T09:26:43.544 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.544 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89cdcb31:::benchmark_data_smithi122_168655_object110308:head# 2024-03-20T09:26:43.544 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.544 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89cdefba:::benchmark_data_smithi122_168655_object108958:head# 2024-03-20T09:26:43.544 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.544 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89cea8f1:::benchmark_data_smithi122_168655_object35363:head# 2024-03-20T09:26:43.544 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.544 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89cec6f6:::benchmark_data_smithi122_168655_object46685:head# 2024-03-20T09:26:43.544 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.544 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89cf199f:::benchmark_data_smithi122_168655_object25488:head# 2024-03-20T09:26:43.545 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.545 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89cf19db:::benchmark_data_smithi122_168655_object24509:head# 2024-03-20T09:26:43.545 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.545 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89d01c59:::benchmark_data_smithi122_168655_object90877:head# 2024-03-20T09:26:43.545 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.545 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89d03c6c:::benchmark_data_smithi122_168655_object81857:head# 2024-03-20T09:26:43.545 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.545 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89d0542c:::benchmark_data_smithi122_168655_object43716:head# 2024-03-20T09:26:43.545 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.545 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89d089d7:::benchmark_data_smithi122_168655_object66017:head# 2024-03-20T09:26:43.545 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.545 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89d121eb:::benchmark_data_smithi122_168655_object108384:head# 2024-03-20T09:26:43.546 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.546 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89d158bd:::benchmark_data_smithi122_168655_object29361:head# 2024-03-20T09:26:43.546 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.546 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89d1684c:::benchmark_data_smithi122_168655_object112195:head# 2024-03-20T09:26:43.546 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.546 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89d2f271:::benchmark_data_smithi122_168655_object108400:head# 2024-03-20T09:26:43.546 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.546 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89d3f7d0:::benchmark_data_smithi122_168655_object75681:head# 2024-03-20T09:26:43.546 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.546 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89d4d942:::benchmark_data_smithi122_168655_object45563:head# 2024-03-20T09:26:43.546 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.546 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89d554c2:::benchmark_data_smithi122_168655_object98130:head# 2024-03-20T09:26:43.546 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.547 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89d72667:::benchmark_data_smithi122_168655_object116860:head# 2024-03-20T09:26:43.547 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.547 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89d816f5:::benchmark_data_smithi122_168655_object51939:head# 2024-03-20T09:26:43.547 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.547 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89db0855:::benchmark_data_smithi122_168655_object103969:head# 2024-03-20T09:26:43.547 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.547 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89db17fc:::benchmark_data_smithi122_168655_object34258:head# 2024-03-20T09:26:43.547 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.547 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89db50b3:::benchmark_data_smithi122_168655_object42649:head# 2024-03-20T09:26:43.547 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.547 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89dbc268:::benchmark_data_smithi122_168655_object71000:head# 2024-03-20T09:26:43.547 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.547 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89dd5759:::benchmark_data_smithi122_168655_object50691:head# 2024-03-20T09:26:43.548 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.548 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89dd7e8f:::benchmark_data_smithi122_168655_object57187:head# 2024-03-20T09:26:43.548 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.548 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89ddeac5:::benchmark_data_smithi122_168655_object19731:head# 2024-03-20T09:26:43.548 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.548 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89de1cf2:::benchmark_data_smithi122_168655_object108102:head# 2024-03-20T09:26:43.548 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.548 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89de6f64:::benchmark_data_smithi122_168655_object63894:head# 2024-03-20T09:26:43.548 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.548 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89df0060:::benchmark_data_smithi122_168655_object96651:head# 2024-03-20T09:26:43.548 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.548 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89e05545:::benchmark_data_smithi122_168655_object105345:head# 2024-03-20T09:26:43.548 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.549 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89e0c888:::benchmark_data_smithi122_168655_object44779:head# 2024-03-20T09:26:43.549 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.549 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89e10554:::benchmark_data_smithi122_168655_object114097:head# 2024-03-20T09:26:43.549 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.549 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89e10802:::benchmark_data_smithi122_168655_object75165:head# 2024-03-20T09:26:43.549 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.549 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89e18faf:::benchmark_data_smithi122_168655_object53599:head# 2024-03-20T09:26:43.549 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.549 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89e1b921:::benchmark_data_smithi122_168655_object51894:head# 2024-03-20T09:26:43.549 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.549 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89e2011d:::benchmark_data_smithi122_168655_object50585:head# 2024-03-20T09:26:43.549 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.549 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89e2cfb9:::benchmark_data_smithi122_168655_object99968:head# 2024-03-20T09:26:43.549 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.550 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89e2e6d7:::benchmark_data_smithi122_168655_object86522:head# 2024-03-20T09:26:43.550 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.550 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89e2fd25:::benchmark_data_smithi122_168655_object102400:head# 2024-03-20T09:26:43.550 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.550 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89e34ab7:::benchmark_data_smithi122_168655_object40817:head# 2024-03-20T09:26:43.550 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.550 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89e3509a:::benchmark_data_smithi122_168655_object28972:head# 2024-03-20T09:26:43.550 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.550 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89e35428:::benchmark_data_smithi122_168655_object31174:head# 2024-03-20T09:26:43.550 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.550 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89e36a18:::benchmark_data_smithi122_168655_object91967:head# 2024-03-20T09:26:43.550 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.551 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89e44d91:::benchmark_data_smithi122_168655_object70051:head# 2024-03-20T09:26:43.551 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.551 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89e490fa:::benchmark_data_smithi122_168655_object10660:head# 2024-03-20T09:26:43.551 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.551 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89e5b798:::benchmark_data_smithi122_168655_object12278:head# 2024-03-20T09:26:43.551 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.551 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89e70ad8:::benchmark_data_smithi122_168655_object21782:head# 2024-03-20T09:26:43.551 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.551 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89e7ae8f:::benchmark_data_smithi122_168655_object101361:head# 2024-03-20T09:26:43.551 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.551 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89e8ea4a:::benchmark_data_smithi122_168655_object71026:head# 2024-03-20T09:26:43.551 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.552 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89e92fb8:::benchmark_data_smithi122_168655_object81849:head# 2024-03-20T09:26:43.552 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.552 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89e9fa61:::benchmark_data_smithi122_168655_object26375:head# 2024-03-20T09:26:43.552 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.552 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89eb1c91:::benchmark_data_smithi122_168655_object37132:head# 2024-03-20T09:26:43.552 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.552 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89eb9bec:::benchmark_data_smithi122_168655_object78333:head# 2024-03-20T09:26:43.552 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.552 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89ec3faa:::benchmark_data_smithi122_168655_object100123:head# 2024-03-20T09:26:43.552 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.552 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89ec8fb9:::benchmark_data_smithi122_168655_object68970:head# 2024-03-20T09:26:43.552 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.552 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89ed086e:::benchmark_data_smithi122_168655_object43526:head# 2024-03-20T09:26:43.553 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.553 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89ed0beb:::benchmark_data_smithi122_168655_object25853:head# 2024-03-20T09:26:43.553 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.553 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89ed6b33:::benchmark_data_smithi122_168655_object39812:head# 2024-03-20T09:26:43.553 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.553 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89ed75eb:::benchmark_data_smithi122_168655_object56237:head# 2024-03-20T09:26:43.553 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.553 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89ee1961:::benchmark_data_smithi122_168655_object77145:head# 2024-03-20T09:26:43.553 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.553 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89ee3ba4:::benchmark_data_smithi122_168655_object27266:head# 2024-03-20T09:26:43.553 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.553 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89efa379:::benchmark_data_smithi122_168655_object23757:head# 2024-03-20T09:26:43.553 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.554 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89f0cf9a:::benchmark_data_smithi122_168655_object98381:head# 2024-03-20T09:26:43.554 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.554 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89f0fff3:::benchmark_data_smithi122_168655_object53701:head# 2024-03-20T09:26:43.554 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.554 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89f10b96:::benchmark_data_smithi122_168655_object111119:head# 2024-03-20T09:26:43.554 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.554 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89f156e6:::benchmark_data_smithi122_168655_object78843:head# 2024-03-20T09:26:43.554 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.554 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89f19164:::benchmark_data_smithi122_168655_object25025:head# 2024-03-20T09:26:43.554 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.554 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89f2254c:::benchmark_data_smithi122_168655_object67825:head# 2024-03-20T09:26:43.554 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.555 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89f2b891:::benchmark_data_smithi122_168655_object67248:head# 2024-03-20T09:26:43.555 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.555 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89f2e07d:::benchmark_data_smithi122_168655_object46947:head# 2024-03-20T09:26:43.555 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.555 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89f328d1:::benchmark_data_smithi122_168655_object73809:head# 2024-03-20T09:26:43.555 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.555 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89f3dca5:::benchmark_data_smithi122_168655_object71017:head# 2024-03-20T09:26:43.555 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.555 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89f3f1c6:::benchmark_data_smithi122_168655_object59842:head# 2024-03-20T09:26:43.555 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.555 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89f43af3:::benchmark_data_smithi122_168655_object105114:head# 2024-03-20T09:26:43.555 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.555 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89f45328:::benchmark_data_smithi122_168655_object109243:head# 2024-03-20T09:26:43.556 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.556 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89f488be:::benchmark_data_smithi122_168655_object100043:head# 2024-03-20T09:26:43.556 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.556 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89f51367:::benchmark_data_smithi122_168655_object23162:head# 2024-03-20T09:26:43.556 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.556 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89f69163:::benchmark_data_smithi122_168655_object58543:head# 2024-03-20T09:26:43.556 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.556 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89f95d1e:::benchmark_data_smithi122_168655_object30636:head# 2024-03-20T09:26:43.556 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.556 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89fa3eb5:::benchmark_data_smithi122_168655_object14597:head# 2024-03-20T09:26:43.556 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.556 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89fa9794:::benchmark_data_smithi122_168655_object17385:head# 2024-03-20T09:26:43.556 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.556 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89fb6277:::benchmark_data_smithi122_168655_object15279:head# 2024-03-20T09:26:43.557 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.557 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89fe29fa:::benchmark_data_smithi122_168655_object17798:head# 2024-03-20T09:26:43.557 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.557 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89fe8cf5:::benchmark_data_smithi122_168655_object19694:head# 2024-03-20T09:26:43.557 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.557 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89febfee:::benchmark_data_smithi122_168655_object87690:head# 2024-03-20T09:26:43.557 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.557 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89ff1478:::benchmark_data_smithi122_168655_object48258:head# 2024-03-20T09:26:43.557 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.557 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89ffcf04:::benchmark_data_smithi122_168655_object31437:head# 2024-03-20T09:26:43.557 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.557 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:89ffdcd8:::benchmark_data_smithi122_168655_object107372:head# 2024-03-20T09:26:43.557 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.557 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a01543f:::benchmark_data_smithi122_168655_object105182:head# 2024-03-20T09:26:43.557 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.558 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a017857:::benchmark_data_smithi122_168655_object118481:head# 2024-03-20T09:26:43.558 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.558 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a01ac64:::benchmark_data_smithi122_168655_object95607:head# 2024-03-20T09:26:43.558 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.558 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a01b2b9:::benchmark_data_smithi122_168655_object25293:head# 2024-03-20T09:26:43.558 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.558 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a01b6a9:::benchmark_data_smithi122_168655_object95936:head# 2024-03-20T09:26:43.558 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.558 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a027842:::benchmark_data_smithi122_168655_object66269:head# 2024-03-20T09:26:43.558 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.558 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a034003:::benchmark_data_smithi122_168655_object72466:head# 2024-03-20T09:26:43.558 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.558 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a03aac2:::benchmark_data_smithi122_168655_object95518:head# 2024-03-20T09:26:43.558 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.558 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a04df79:::benchmark_data_smithi122_168655_object32781:head# 2024-03-20T09:26:43.559 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.559 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a064855:::benchmark_data_smithi122_168655_object14866:head# 2024-03-20T09:26:43.559 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.559 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a06f249:::benchmark_data_smithi122_168655_object97669:head# 2024-03-20T09:26:43.559 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.559 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a073682:::benchmark_data_smithi122_168655_object35635:head# 2024-03-20T09:26:43.559 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.559 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a07679b:::benchmark_data_smithi122_168655_object115874:head# 2024-03-20T09:26:43.559 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.559 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a077773:::benchmark_data_smithi122_168655_object24676:head# 2024-03-20T09:26:43.559 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.559 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a087acf:::benchmark_data_smithi122_168655_object94646:head# 2024-03-20T09:26:43.559 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.559 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a08d1b1:::benchmark_data_smithi122_168655_object112601:head# 2024-03-20T09:26:43.560 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.560 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a0a153f:::benchmark_data_smithi122_168655_object12367:head# 2024-03-20T09:26:43.560 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.560 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a0a7ec1:::benchmark_data_smithi122_168655_object57118:head# 2024-03-20T09:26:43.560 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.560 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a0a97e3:::benchmark_data_smithi122_168655_object58508:head# 2024-03-20T09:26:43.560 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.560 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a0b6366:::benchmark_data_smithi122_168655_object19848:head# 2024-03-20T09:26:43.560 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.560 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a0c0e18:::benchmark_data_smithi122_168655_object94419:head# 2024-03-20T09:26:43.560 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.560 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a0c3331:::benchmark_data_smithi122_168655_object23378:head# 2024-03-20T09:26:43.560 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.560 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a0cd8fa:::benchmark_data_smithi122_168655_object47928:head# 2024-03-20T09:26:43.561 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.561 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a0ea762:::benchmark_data_smithi122_168655_object86994:head# 2024-03-20T09:26:43.561 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.561 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a0ede90:::benchmark_data_smithi122_168655_object10929:head# 2024-03-20T09:26:43.561 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.561 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a0f9030:::benchmark_data_smithi122_168655_object113521:head# 2024-03-20T09:26:43.561 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.561 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a0fda02:::benchmark_data_smithi122_168655_object43725:head# 2024-03-20T09:26:43.561 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.561 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a115a30:::benchmark_data_smithi122_168655_object16965:head# 2024-03-20T09:26:43.561 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.561 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a119678:::benchmark_data_smithi122_168655_object103591:head# 2024-03-20T09:26:43.561 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.561 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a1248a0:::benchmark_data_smithi122_168655_object37468:head# 2024-03-20T09:26:43.562 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.562 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a133f4f:::benchmark_data_smithi122_168655_object58967:head# 2024-03-20T09:26:43.562 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.562 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a13bbf0:::benchmark_data_smithi122_168655_object18125:head# 2024-03-20T09:26:43.562 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.562 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a145a07:::benchmark_data_smithi122_168655_object66996:head# 2024-03-20T09:26:43.562 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.562 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a14c55d:::benchmark_data_smithi122_168655_object112672:head# 2024-03-20T09:26:43.562 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.562 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a15010b:::benchmark_data_smithi122_168655_object84872:head# 2024-03-20T09:26:43.562 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.562 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a151286:::benchmark_data_smithi122_168655_object18087:head# 2024-03-20T09:26:43.562 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.562 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a161efb:::benchmark_data_smithi122_168655_object68109:head# 2024-03-20T09:26:43.563 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.563 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a183b7a:::benchmark_data_smithi122_168655_object11493:head# 2024-03-20T09:26:43.563 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.563 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a186f7e:::benchmark_data_smithi122_168655_object89114:head# 2024-03-20T09:26:43.563 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.563 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a18ccfd:::benchmark_data_smithi122_168655_object64453:head# 2024-03-20T09:26:43.563 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.563 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a195342:::benchmark_data_smithi122_168655_object99990:head# 2024-03-20T09:26:43.563 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.563 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a1b3dc3:::benchmark_data_smithi122_168655_object114080:head# 2024-03-20T09:26:43.563 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.563 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a1c2801:::benchmark_data_smithi122_168655_object55644:head# 2024-03-20T09:26:43.563 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.563 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a1c4a6b:::benchmark_data_smithi122_168655_object39482:head# 2024-03-20T09:26:43.564 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.564 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a1d5ccd:::benchmark_data_smithi122_168655_object18281:head# 2024-03-20T09:26:43.564 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.564 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a1dad76:::benchmark_data_smithi122_168655_object41220:head# 2024-03-20T09:26:43.564 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.564 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a1dc88f:::benchmark_data_smithi122_168655_object36914:head# 2024-03-20T09:26:43.564 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.564 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a1dd735:::benchmark_data_smithi122_168655_object32240:head# 2024-03-20T09:26:43.564 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.564 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a1de307:::benchmark_data_smithi122_168655_object51388:head# 2024-03-20T09:26:43.564 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.564 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a1de334:::benchmark_data_smithi122_168655_object69994:head# 2024-03-20T09:26:43.564 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.564 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a1ec5a4:::benchmark_data_smithi122_168655_object38898:head# 2024-03-20T09:26:43.565 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.565 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a1f4286:::benchmark_data_smithi122_168655_object79406:head# 2024-03-20T09:26:43.565 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.565 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a2045b6:::benchmark_data_smithi122_168655_object17398:head# 2024-03-20T09:26:43.565 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.565 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a2079fd:::benchmark_data_smithi122_168655_object92633:head# 2024-03-20T09:26:43.565 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.565 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a218011:::benchmark_data_smithi122_168655_object99028:head# 2024-03-20T09:26:43.565 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.565 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a22033d:::benchmark_data_smithi122_168655_object92000:head# 2024-03-20T09:26:43.565 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.565 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a22081c:::benchmark_data_smithi122_168655_object62344:head# 2024-03-20T09:26:43.565 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.565 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a230219:::benchmark_data_smithi122_168655_object59037:head# 2024-03-20T09:26:43.565 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.566 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a2353d0:::benchmark_data_smithi122_168655_object92444:head# 2024-03-20T09:26:43.566 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.566 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a23dc1f:::benchmark_data_smithi122_168655_object42967:head# 2024-03-20T09:26:43.566 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.566 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a246aa8:::benchmark_data_smithi122_168655_object104458:head# 2024-03-20T09:26:43.566 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.566 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a24a678:::benchmark_data_smithi122_168655_object40060:head# 2024-03-20T09:26:43.566 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.566 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a25ca05:::benchmark_data_smithi122_168655_object72400:head# 2024-03-20T09:26:43.566 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.566 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a267c21:::benchmark_data_smithi122_168655_object91483:head# 2024-03-20T09:26:43.566 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.566 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a268770:::benchmark_data_smithi122_168655_object56440:head# 2024-03-20T09:26:43.566 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.566 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a26cd6c:::benchmark_data_smithi122_168655_object43521:head# 2024-03-20T09:26:43.567 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.567 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a2711b1:::benchmark_data_smithi122_168655_object21303:head# 2024-03-20T09:26:43.567 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.567 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a278cd0:::benchmark_data_smithi122_168655_object21274:head# 2024-03-20T09:26:43.567 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.567 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a280193:::benchmark_data_smithi122_168655_object70409:head# 2024-03-20T09:26:43.567 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.567 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a295069:::benchmark_data_smithi122_168655_object96061:head# 2024-03-20T09:26:43.567 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.567 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a2a2b36:::benchmark_data_smithi122_168655_object110438:head# 2024-03-20T09:26:43.567 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.567 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a2a74d8:::benchmark_data_smithi122_168655_object79746:head# 2024-03-20T09:26:43.567 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.567 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a2a9956:::benchmark_data_smithi122_168655_object32363:head# 2024-03-20T09:26:43.568 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.568 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a2b46e5:::benchmark_data_smithi122_168655_object118862:head# 2024-03-20T09:26:43.568 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.568 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a2d02e0:::benchmark_data_smithi122_168655_object97282:head# 2024-03-20T09:26:43.568 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.568 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a2df037:::benchmark_data_smithi122_168655_object24529:head# 2024-03-20T09:26:43.568 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.568 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a308090:::benchmark_data_smithi122_168655_object33274:head# 2024-03-20T09:26:43.568 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.568 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a309199:::benchmark_data_smithi122_168655_object47332:head# 2024-03-20T09:26:43.568 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.568 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a32a8e8:::benchmark_data_smithi122_168655_object19620:head# 2024-03-20T09:26:43.568 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.568 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a32b4ba:::benchmark_data_smithi122_168655_object109625:head# 2024-03-20T09:26:43.569 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.569 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a332341:::benchmark_data_smithi122_168655_object51870:head# 2024-03-20T09:26:43.569 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.569 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a33eb7a:::benchmark_data_smithi122_168655_object103119:head# 2024-03-20T09:26:43.569 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.569 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a3417a1:::benchmark_data_smithi122_168655_object17304:head# 2024-03-20T09:26:43.569 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.569 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a341c79:::benchmark_data_smithi122_168655_object76560:head# 2024-03-20T09:26:43.569 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.569 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a344b7c:::benchmark_data_smithi122_168655_object65782:head# 2024-03-20T09:26:43.569 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.569 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a3453c7:::benchmark_data_smithi122_168655_object76113:head# 2024-03-20T09:26:43.569 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.569 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a347c04:::benchmark_data_smithi122_168655_object91405:head# 2024-03-20T09:26:43.569 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.570 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a34c2f7:::benchmark_data_smithi122_168655_object32915:head# 2024-03-20T09:26:43.570 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.570 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a356fa6:::benchmark_data_smithi122_168655_object16206:head# 2024-03-20T09:26:43.570 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.570 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a35ad41:::benchmark_data_smithi122_168655_object12545:head# 2024-03-20T09:26:43.570 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.570 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a35be32:::benchmark_data_smithi122_168655_object27899:head# 2024-03-20T09:26:43.570 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.570 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a367cba:::benchmark_data_smithi122_168655_object33768:head# 2024-03-20T09:26:43.570 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.570 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a36d783:::benchmark_data_smithi122_168655_object58631:head# 2024-03-20T09:26:43.570 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.570 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a376737:::benchmark_data_smithi122_168655_object85839:head# 2024-03-20T09:26:43.570 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.571 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a3806fa:::benchmark_data_smithi122_168655_object35651:head# 2024-03-20T09:26:43.571 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.571 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a385d6e:::benchmark_data_smithi122_168655_object16268:head# 2024-03-20T09:26:43.571 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.571 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a387cc1:::benchmark_data_smithi122_168655_object47038:head# 2024-03-20T09:26:43.571 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.571 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a395afb:::benchmark_data_smithi122_168655_object38149:head# 2024-03-20T09:26:43.571 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.571 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a3999a6:::benchmark_data_smithi122_168655_object33852:head# 2024-03-20T09:26:43.571 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.571 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a3add29:::benchmark_data_smithi122_168655_object24813:head# 2024-03-20T09:26:43.571 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.571 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a3f1dfe:::benchmark_data_smithi122_168655_object83977:head# 2024-03-20T09:26:43.571 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.571 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a3fce8d:::benchmark_data_smithi122_168655_object27235:head# 2024-03-20T09:26:43.572 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.572 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a4035bb:::benchmark_data_smithi122_168655_object100447:head# 2024-03-20T09:26:43.572 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.572 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a40cdaf:::benchmark_data_smithi122_168655_object105313:head# 2024-03-20T09:26:43.572 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.572 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a41da0c:::benchmark_data_smithi122_168655_object72243:head# 2024-03-20T09:26:43.572 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.572 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a4316d3:::benchmark_data_smithi122_168655_object119353:head# 2024-03-20T09:26:43.573 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.573 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a43bd5d:::benchmark_data_smithi122_168655_object104098:head# 2024-03-20T09:26:43.573 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.573 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a458e28:::benchmark_data_smithi122_168655_object60985:head# 2024-03-20T09:26:43.573 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.573 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a46365e:::benchmark_data_smithi122_168655_object46753:head# 2024-03-20T09:26:43.573 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.573 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a46f601:::benchmark_data_smithi122_168655_object88519:head# 2024-03-20T09:26:43.573 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.573 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a46fed6:::benchmark_data_smithi122_168655_object109509:head# 2024-03-20T09:26:43.573 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.573 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a474401:::benchmark_data_smithi122_168655_object49609:head# 2024-03-20T09:26:43.573 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.574 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a488e17:::benchmark_data_smithi122_168655_object113909:head# 2024-03-20T09:26:43.574 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.574 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a49079f:::benchmark_data_smithi122_168655_object15486:head# 2024-03-20T09:26:43.574 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.574 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a496b46:::benchmark_data_smithi122_168655_object29112:head# 2024-03-20T09:26:43.574 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.574 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a49fd00:::benchmark_data_smithi122_168655_object30880:head# 2024-03-20T09:26:43.574 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.574 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a4a03ff:::benchmark_data_smithi122_168655_object40839:head# 2024-03-20T09:26:43.574 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.574 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a4a5c71:::benchmark_data_smithi122_168655_object64382:head# 2024-03-20T09:26:43.574 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.574 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a4a97dd:::benchmark_data_smithi122_168655_object21855:head# 2024-03-20T09:26:43.574 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.575 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a4ac5b3:::benchmark_data_smithi122_168655_object27046:head# 2024-03-20T09:26:43.575 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.575 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a4af637:::benchmark_data_smithi122_168655_object88832:head# 2024-03-20T09:26:43.575 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.575 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a4b3398:::benchmark_data_smithi122_168655_object106647:head# 2024-03-20T09:26:43.575 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.575 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a4b94c1:::benchmark_data_smithi122_168655_object79240:head# 2024-03-20T09:26:43.575 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.575 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a4c1d1b:::benchmark_data_smithi122_168655_object77216:head# 2024-03-20T09:26:43.575 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.575 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a4cbcb4:::benchmark_data_smithi122_168655_object79487:head# 2024-03-20T09:26:43.575 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.575 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a4d5259:::benchmark_data_smithi122_168655_object32499:head# 2024-03-20T09:26:43.576 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.576 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a4e1454:::benchmark_data_smithi122_168655_object68196:head# 2024-03-20T09:26:43.576 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.576 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a4f8879:::benchmark_data_smithi122_168655_object80032:head# 2024-03-20T09:26:43.576 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.576 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a50f821:::benchmark_data_smithi122_168655_object45311:head# 2024-03-20T09:26:43.576 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.576 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a512dc2:::benchmark_data_smithi122_168655_object96359:head# 2024-03-20T09:26:43.576 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.576 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a51ce3e:::benchmark_data_smithi122_168655_object65155:head# 2024-03-20T09:26:43.576 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.576 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a528b89:::benchmark_data_smithi122_168655_object65109:head# 2024-03-20T09:26:43.576 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.576 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a52be47:::benchmark_data_smithi122_168655_object30314:head# 2024-03-20T09:26:43.577 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.577 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a52ed3b:::benchmark_data_smithi122_168655_object66646:head# 2024-03-20T09:26:43.577 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.577 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a53d650:::benchmark_data_smithi122_168655_object67241:head# 2024-03-20T09:26:43.577 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.577 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a5496e4:::benchmark_data_smithi122_168655_object45533:head# 2024-03-20T09:26:43.577 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.577 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a5515ab:::benchmark_data_smithi122_168655_object110570:head# 2024-03-20T09:26:43.577 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.577 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a557e12:::benchmark_data_smithi122_168655_object11343:head# 2024-03-20T09:26:43.577 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.577 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a576788:::benchmark_data_smithi122_168655_object32293:head# 2024-03-20T09:26:43.577 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.577 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a57ae52:::benchmark_data_smithi122_168655_object24665:head# 2024-03-20T09:26:43.578 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.578 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a5b3130:::benchmark_data_smithi122_168655_object115047:head# 2024-03-20T09:26:43.578 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.578 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a5bd901:::benchmark_data_smithi122_168655_object69484:head# 2024-03-20T09:26:43.578 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.578 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a5d1bab:::benchmark_data_smithi122_168655_object43056:head# 2024-03-20T09:26:43.578 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.578 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a5ee9a1:::benchmark_data_smithi122_168655_object92531:head# 2024-03-20T09:26:43.578 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.578 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a60ab00:::benchmark_data_smithi122_168655_object74959:head# 2024-03-20T09:26:43.578 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.578 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a60ce63:::benchmark_data_smithi122_168655_object105360:head# 2024-03-20T09:26:43.578 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.578 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a6170cd:::benchmark_data_smithi122_168655_object63324:head# 2024-03-20T09:26:43.579 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.579 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a6188fc:::benchmark_data_smithi122_168655_object103892:head# 2024-03-20T09:26:43.579 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.579 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a62e695:::benchmark_data_smithi122_168655_object97427:head# 2024-03-20T09:26:43.579 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.579 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a637c0b:::benchmark_data_smithi122_168655_object59869:head# 2024-03-20T09:26:43.579 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.579 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a63ef5e:::benchmark_data_smithi122_168655_object59993:head# 2024-03-20T09:26:43.579 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.579 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a645d4c:::benchmark_data_smithi122_168655_object60216:head# 2024-03-20T09:26:43.579 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.579 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a652d1c:::benchmark_data_smithi122_168655_object50981:head# 2024-03-20T09:26:43.579 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.579 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a67b8b6:::benchmark_data_smithi122_168655_object114811:head# 2024-03-20T09:26:43.579 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.580 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a67bde5:::benchmark_data_smithi122_168655_object93133:head# 2024-03-20T09:26:43.580 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.580 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a67d5c2:::benchmark_data_smithi122_168655_object89984:head# 2024-03-20T09:26:43.580 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.580 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a689dcc:::benchmark_data_smithi122_168655_object109281:head# 2024-03-20T09:26:43.580 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.580 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a68c83d:::benchmark_data_smithi122_168655_object49951:head# 2024-03-20T09:26:43.580 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.580 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a698802:::benchmark_data_smithi122_168655_object63186:head# 2024-03-20T09:26:43.580 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.580 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a69bf26:::benchmark_data_smithi122_168655_object28574:head# 2024-03-20T09:26:43.580 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.580 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a6a2807:::benchmark_data_smithi122_168655_object56924:head# 2024-03-20T09:26:43.580 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.581 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a6a4397:::benchmark_data_smithi122_168655_object55109:head# 2024-03-20T09:26:43.581 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.581 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a6ac4b3:::benchmark_data_smithi122_168655_object87898:head# 2024-03-20T09:26:43.581 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.581 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a6cdd7c:::benchmark_data_smithi122_168655_object24752:head# 2024-03-20T09:26:43.581 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.581 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a6d3fb3:::benchmark_data_smithi122_168655_object45736:head# 2024-03-20T09:26:43.581 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.581 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a6e5bd6:::benchmark_data_smithi122_168655_object72156:head# 2024-03-20T09:26:43.581 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.581 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a6ef1a8:::benchmark_data_smithi122_168655_object98334:head# 2024-03-20T09:26:43.581 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.581 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a6f385b:::benchmark_data_smithi122_168655_object61584:head# 2024-03-20T09:26:43.581 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.581 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a6f8203:::benchmark_data_smithi122_168655_object91904:head# 2024-03-20T09:26:43.582 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.582 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a6fd74a:::benchmark_data_smithi122_168655_object76609:head# 2024-03-20T09:26:43.582 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.582 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a6fd8e2:::benchmark_data_smithi122_168655_object105809:head# 2024-03-20T09:26:43.582 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.582 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a7000f4:::benchmark_data_smithi122_168655_object111809:head# 2024-03-20T09:26:43.582 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.582 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a7008f7:::benchmark_data_smithi122_168655_object79261:head# 2024-03-20T09:26:43.582 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.582 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a708b8c:::benchmark_data_smithi122_168655_object109244:head# 2024-03-20T09:26:43.582 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.582 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a736f95:::benchmark_data_smithi122_168655_object92016:head# 2024-03-20T09:26:43.583 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.583 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a746788:::benchmark_data_smithi122_168655_object68432:head# 2024-03-20T09:26:43.583 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.583 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a74f2e6:::benchmark_data_smithi122_168655_object107558:head# 2024-03-20T09:26:43.583 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.583 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a760ef2:::benchmark_data_smithi122_168655_object15733:head# 2024-03-20T09:26:43.583 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.583 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a76f4a0:::benchmark_data_smithi122_168655_object22793:head# 2024-03-20T09:26:43.583 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.583 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a781415:::benchmark_data_smithi122_168655_object114252:head# 2024-03-20T09:26:43.583 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.583 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a79e76a:::benchmark_data_smithi122_168655_object94586:head# 2024-03-20T09:26:43.584 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.584 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a7ab687:::benchmark_data_smithi122_168655_object23434:head# 2024-03-20T09:26:43.584 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.584 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a7b9b6f:::benchmark_data_smithi122_168655_object67600:head# 2024-03-20T09:26:43.584 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.584 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a7ca337:::benchmark_data_smithi122_168655_object77748:head# 2024-03-20T09:26:43.584 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.584 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a7cbcf7:::benchmark_data_smithi122_168655_object17188:head# 2024-03-20T09:26:43.584 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.584 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a7d4e7b:::benchmark_data_smithi122_168655_object66513:head# 2024-03-20T09:26:43.584 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.584 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a7d7bbb:::benchmark_data_smithi122_168655_object40222:head# 2024-03-20T09:26:43.584 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.585 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a7dd137:::benchmark_data_smithi122_168655_object24159:head# 2024-03-20T09:26:43.585 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.585 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a7df4e9:::benchmark_data_smithi122_168655_object24162:head# 2024-03-20T09:26:43.585 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.585 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a7ed153:::benchmark_data_smithi122_168655_object104033:head# 2024-03-20T09:26:43.585 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.585 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a7f087b:::benchmark_data_smithi122_168655_object33456:head# 2024-03-20T09:26:43.585 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.585 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a7f99b8:::benchmark_data_smithi122_168655_object21732:head# 2024-03-20T09:26:43.585 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.585 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a7fb170:::benchmark_data_smithi122_168655_object81724:head# 2024-03-20T09:26:43.586 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.586 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a805ab0:::benchmark_data_smithi122_168655_object55127:head# 2024-03-20T09:26:43.586 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.586 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a82bcda:::benchmark_data_smithi122_168655_object45012:head# 2024-03-20T09:26:43.586 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.586 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a832cc2:::benchmark_data_smithi122_168655_object31491:head# 2024-03-20T09:26:43.586 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.586 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a83da97:::benchmark_data_smithi122_168655_object31857:head# 2024-03-20T09:26:43.586 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.586 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a842c43:::benchmark_data_smithi122_168655_object43038:head# 2024-03-20T09:26:43.586 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.587 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a84c5f4:::benchmark_data_smithi122_168655_object24812:head# 2024-03-20T09:26:43.587 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.587 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a84ff2b:::benchmark_data_smithi122_168655_object28873:head# 2024-03-20T09:26:43.587 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.587 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a8509db:::benchmark_data_smithi122_168655_object44488:head# 2024-03-20T09:26:43.587 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.587 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a852ddf:::benchmark_data_smithi122_168655_object36540:head# 2024-03-20T09:26:43.587 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.587 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a85b04f:::benchmark_data_smithi122_168655_object85330:head# 2024-03-20T09:26:43.587 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.587 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a8650bd:::benchmark_data_smithi122_168655_object72912:head# 2024-03-20T09:26:43.587 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.587 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a876793:::benchmark_data_smithi122_168655_object63858:head# 2024-03-20T09:26:43.588 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.588 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a87e696:::benchmark_data_smithi122_168655_object14931:head# 2024-03-20T09:26:43.588 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.588 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a883b09:::benchmark_data_smithi122_168655_object51276:head# 2024-03-20T09:26:43.588 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.588 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a88563c:::benchmark_data_smithi122_168655_object86786:head# 2024-03-20T09:26:43.588 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.588 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a88ac4a:::benchmark_data_smithi122_168655_object41600:head# 2024-03-20T09:26:43.588 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.588 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a8a2f8d:::benchmark_data_smithi122_168655_object88280:head# 2024-03-20T09:26:43.589 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.589 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a8a436e:::benchmark_data_smithi122_168655_object78631:head# 2024-03-20T09:26:43.589 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.589 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a8a462d:::benchmark_data_smithi122_168655_object111559:head# 2024-03-20T09:26:43.589 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.589 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a8a5b2f:::benchmark_data_smithi122_168655_object37871:head# 2024-03-20T09:26:43.589 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.589 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a8a7bb5:::benchmark_data_smithi122_168655_object52760:head# 2024-03-20T09:26:43.589 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.589 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a8aaec0:::benchmark_data_smithi122_168655_object51439:head# 2024-03-20T09:26:43.589 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.589 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a8c38cc:::benchmark_data_smithi122_168655_object64341:head# 2024-03-20T09:26:43.590 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.590 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a8d54a8:::benchmark_data_smithi122_168655_object81913:head# 2024-03-20T09:26:43.590 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.590 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a8d90ca:::benchmark_data_smithi122_168655_object24500:head# 2024-03-20T09:26:43.590 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.590 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a8ec2ae:::benchmark_data_smithi122_168655_object24777:head# 2024-03-20T09:26:43.590 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.590 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a903f1d:::benchmark_data_smithi122_168655_object74320:head# 2024-03-20T09:26:43.590 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.590 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a90d884:::benchmark_data_smithi122_168655_object73196:head# 2024-03-20T09:26:43.590 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.590 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a91c812:::benchmark_data_smithi122_168655_object98170:head# 2024-03-20T09:26:43.591 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.591 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a91fb1c:::benchmark_data_smithi122_168655_object116800:head# 2024-03-20T09:26:43.591 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.591 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a93b974:::benchmark_data_smithi122_168655_object58029:head# 2024-03-20T09:26:43.591 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.591 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a94443a:::benchmark_data_smithi122_168655_object88648:head# 2024-03-20T09:26:43.591 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.591 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a9526ac:::benchmark_data_smithi122_168655_object52784:head# 2024-03-20T09:26:43.591 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.591 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a953edd:::benchmark_data_smithi122_168655_object62766:head# 2024-03-20T09:26:43.591 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.591 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a953fe3:::benchmark_data_smithi122_168655_object79742:head# 2024-03-20T09:26:43.591 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.592 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a95ed76:::benchmark_data_smithi122_168655_object59570:head# 2024-03-20T09:26:43.592 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.592 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a96d984:::benchmark_data_smithi122_168655_object50884:head# 2024-03-20T09:26:43.592 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.592 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a9780c5:::benchmark_data_smithi122_168655_object25082:head# 2024-03-20T09:26:43.592 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.592 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a97f90f:::benchmark_data_smithi122_168655_object57031:head# 2024-03-20T09:26:43.592 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.592 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a98a5c7:::benchmark_data_smithi122_168655_object45053:head# 2024-03-20T09:26:43.592 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.592 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a99bdf5:::benchmark_data_smithi122_168655_object56906:head# 2024-03-20T09:26:43.592 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.593 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a9bbe18:::benchmark_data_smithi122_168655_object35496:head# 2024-03-20T09:26:43.593 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.593 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a9c2712:::benchmark_data_smithi122_168655_object48089:head# 2024-03-20T09:26:43.593 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.593 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a9c8757:::benchmark_data_smithi122_168655_object37157:head# 2024-03-20T09:26:43.593 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.593 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a9c9880:::benchmark_data_smithi122_168655_object112785:head# 2024-03-20T09:26:43.593 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.593 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a9d3744:::benchmark_data_smithi122_168655_object17261:head# 2024-03-20T09:26:43.593 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.593 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a9ee42e:::benchmark_data_smithi122_168655_object61493:head# 2024-03-20T09:26:43.593 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.593 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a9fa33b:::benchmark_data_smithi122_168655_object98969:head# 2024-03-20T09:26:43.594 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.594 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8a9fd13d:::benchmark_data_smithi122_168655_object93076:head# 2024-03-20T09:26:43.594 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.594 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aa0109b:::benchmark_data_smithi122_168655_object118105:head# 2024-03-20T09:26:43.594 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.594 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aa0eff0:::benchmark_data_smithi122_168655_object84776:head# 2024-03-20T09:26:43.594 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.594 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aa1a41c:::benchmark_data_smithi122_168655_object23916:head# 2024-03-20T09:26:43.594 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.594 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aa1a98d:::benchmark_data_smithi122_168655_object64383:head# 2024-03-20T09:26:43.594 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.594 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aa20e13:::benchmark_data_smithi122_168655_object84879:head# 2024-03-20T09:26:43.595 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.595 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aa297e3:::benchmark_data_smithi122_168655_object71479:head# 2024-03-20T09:26:43.595 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.595 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aa29c2c:::benchmark_data_smithi122_168655_object33318:head# 2024-03-20T09:26:43.595 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.595 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aa30274:::benchmark_data_smithi122_168655_object111191:head# 2024-03-20T09:26:43.595 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.595 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aa32de7:::benchmark_data_smithi122_168655_object110769:head# 2024-03-20T09:26:43.595 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.595 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aa37cc9:::benchmark_data_smithi122_168655_object98501:head# 2024-03-20T09:26:43.595 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.596 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aa3b1ef:::benchmark_data_smithi122_168655_object65607:head# 2024-03-20T09:26:43.596 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.596 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aa3f85e:::benchmark_data_smithi122_168655_object28560:head# 2024-03-20T09:26:43.596 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.596 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aa42dde:::benchmark_data_smithi122_168655_object78235:head# 2024-03-20T09:26:43.596 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.596 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aa4d1aa:::benchmark_data_smithi122_168655_object111736:head# 2024-03-20T09:26:43.596 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.596 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aa63903:::benchmark_data_smithi122_168655_object97765:head# 2024-03-20T09:26:43.596 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.596 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aa74a9c:::benchmark_data_smithi122_168655_object39717:head# 2024-03-20T09:26:43.597 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.597 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aa7f578:::benchmark_data_smithi122_168655_object99600:head# 2024-03-20T09:26:43.597 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.597 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aa8c3c7:::benchmark_data_smithi122_168655_object99915:head# 2024-03-20T09:26:43.597 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.597 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aa98a6e:::benchmark_data_smithi122_168655_object46767:head# 2024-03-20T09:26:43.597 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.597 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aaa6d72:::benchmark_data_smithi122_168655_object71993:head# 2024-03-20T09:26:43.597 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.597 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aac2484:::benchmark_data_smithi122_168655_object99969:head# 2024-03-20T09:26:43.597 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.597 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aac72e3:::benchmark_data_smithi122_168655_object95071:head# 2024-03-20T09:26:43.598 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.598 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aac8963:::benchmark_data_smithi122_168655_object96824:head# 2024-03-20T09:26:43.598 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.598 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aacd278:::benchmark_data_smithi122_168655_object65957:head# 2024-03-20T09:26:43.598 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.598 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aad4dd6:::benchmark_data_smithi122_168655_object44070:head# 2024-03-20T09:26:43.598 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.598 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aadebd3:::benchmark_data_smithi122_168655_object21019:head# 2024-03-20T09:26:43.598 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.598 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aae8e6e:::benchmark_data_smithi122_168655_object63974:head# 2024-03-20T09:26:43.598 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.599 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aae934d:::benchmark_data_smithi122_168655_object39059:head# 2024-03-20T09:26:43.599 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.599 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aaf04b7:::benchmark_data_smithi122_168655_object37347:head# 2024-03-20T09:26:43.599 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.599 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aaf6a90:::benchmark_data_smithi122_168655_object113988:head# 2024-03-20T09:26:43.599 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.599 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aaf9008:::benchmark_data_smithi122_168655_object76317:head# 2024-03-20T09:26:43.599 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.599 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aafbb11:::benchmark_data_smithi122_168655_object87924:head# 2024-03-20T09:26:43.599 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.599 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ab01391:::benchmark_data_smithi122_168655_object14130:head# 2024-03-20T09:26:43.599 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.600 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ab032ab:::benchmark_data_smithi122_168655_object87050:head# 2024-03-20T09:26:43.600 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.600 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ab06af1:::benchmark_data_smithi122_168655_object70996:head# 2024-03-20T09:26:43.600 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.600 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ab0a2af:::benchmark_data_smithi122_168655_object15004:head# 2024-03-20T09:26:43.600 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.600 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ab21620:::benchmark_data_smithi122_168655_object72847:head# 2024-03-20T09:26:43.600 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.600 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ab30278:::benchmark_data_smithi122_168655_object60576:head# 2024-03-20T09:26:43.600 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.600 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ab40779:::benchmark_data_smithi122_168655_object73726:head# 2024-03-20T09:26:43.601 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.601 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ab4732e:::benchmark_data_smithi122_168655_object89981:head# 2024-03-20T09:26:43.601 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.601 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ab4f961:::benchmark_data_smithi122_168655_object96871:head# 2024-03-20T09:26:43.601 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.601 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ab65f64:::benchmark_data_smithi122_168655_object58492:head# 2024-03-20T09:26:43.601 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.601 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ab8154f:::benchmark_data_smithi122_168655_object111397:head# 2024-03-20T09:26:43.601 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.601 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ab8eddf:::benchmark_data_smithi122_168655_object51062:head# 2024-03-20T09:26:43.601 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.601 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ab9ea30:::benchmark_data_smithi122_168655_object65437:head# 2024-03-20T09:26:43.602 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.602 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8abbc74d:::benchmark_data_smithi122_168655_object104481:head# 2024-03-20T09:26:43.602 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.602 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8abc9ff7:::benchmark_data_smithi122_168655_object115132:head# 2024-03-20T09:26:43.602 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.602 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8abce136:::benchmark_data_smithi122_168655_object48633:head# 2024-03-20T09:26:43.602 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.602 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8abe2195:::benchmark_data_smithi122_168655_object30633:head# 2024-03-20T09:26:43.602 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.602 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8abfb813:::benchmark_data_smithi122_168655_object25462:head# 2024-03-20T09:26:43.603 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.603 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8abfe3bc:::benchmark_data_smithi122_168655_object54403:head# 2024-03-20T09:26:43.603 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.603 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8abfeb9d:::benchmark_data_smithi122_168655_object60892:head# 2024-03-20T09:26:43.603 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.603 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ac0730d:::benchmark_data_smithi122_168655_object43669:head# 2024-03-20T09:26:43.603 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.603 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ac1123b:::benchmark_data_smithi122_168655_object51365:head# 2024-03-20T09:26:43.603 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.603 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ac13aa6:::benchmark_data_smithi122_168655_object89751:head# 2024-03-20T09:26:43.603 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.603 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ac1ae49:::benchmark_data_smithi122_168655_object79026:head# 2024-03-20T09:26:43.603 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.603 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ac1bb27:::benchmark_data_smithi122_168655_object73659:head# 2024-03-20T09:26:43.604 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.604 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ac20f79:::benchmark_data_smithi122_168655_object81017:head# 2024-03-20T09:26:43.604 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.604 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ac2e56f:::benchmark_data_smithi122_168655_object37373:head# 2024-03-20T09:26:43.604 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.604 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ac37f1e:::benchmark_data_smithi122_168655_object113131:head# 2024-03-20T09:26:43.604 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.604 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ac3ca41:::benchmark_data_smithi122_168655_object99018:head# 2024-03-20T09:26:43.604 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.604 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ac3ecaf:::benchmark_data_smithi122_168655_object109002:head# 2024-03-20T09:26:43.604 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.604 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ac44dff:::benchmark_data_smithi122_168655_object48603:head# 2024-03-20T09:26:43.604 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.604 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ac454ff:::benchmark_data_smithi122_168655_object80986:head# 2024-03-20T09:26:43.605 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.605 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ac4aab0:::benchmark_data_smithi122_168655_object86296:head# 2024-03-20T09:26:43.605 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.605 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ac4b5a3:::benchmark_data_smithi122_168655_object75699:head# 2024-03-20T09:26:43.605 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.605 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ac7fa0b:::benchmark_data_smithi122_168655_object28281:head# 2024-03-20T09:26:43.605 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.605 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ac8610a:::benchmark_data_smithi122_168655_object33761:head# 2024-03-20T09:26:43.605 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.605 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ac93939:::benchmark_data_smithi122_168655_object116290:head# 2024-03-20T09:26:43.605 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.605 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aca5530:::benchmark_data_smithi122_168655_object104719:head# 2024-03-20T09:26:43.605 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.605 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aca58d1:::benchmark_data_smithi122_168655_object85951:head# 2024-03-20T09:26:43.605 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.606 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aca7d73:::benchmark_data_smithi122_168655_object60357:head# 2024-03-20T09:26:43.606 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.606 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8acab157:::benchmark_data_smithi122_168655_object74995:head# 2024-03-20T09:26:43.606 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.606 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8acc31ef:::benchmark_data_smithi122_168655_object35611:head# 2024-03-20T09:26:43.606 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.606 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8acc3264:::benchmark_data_smithi122_168655_object54732:head# 2024-03-20T09:26:43.606 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.606 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8acd5f27:::benchmark_data_smithi122_168655_object30284:head# 2024-03-20T09:26:43.606 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.606 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8acd7a10:::benchmark_data_smithi122_168655_object69673:head# 2024-03-20T09:26:43.606 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.606 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ace0f9d:::benchmark_data_smithi122_168655_object71180:head# 2024-03-20T09:26:43.606 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.658 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:43 smithi115 ceph-mon[72476]: 1.f scrub starts 2024-03-20T09:26:43.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:43 smithi115 ceph-mon[72476]: 1.f scrub ok 2024-03-20T09:26:43.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:43 smithi115 ceph-mon[72476]: pgmap v1872: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 38 active+clean; 6.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 211 op/s; 31239/311154 objects degraded (10.040%); 30468/311154 objects misplaced (9.792%); 25 MiB/s, 447 objects/s recovering 2024-03-20T09:26:43.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:43 smithi115 ceph-mon[72476]: osdmap e770: 12 total, 11 up, 12 in 2024-03-20T09:26:43.659 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:43 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 31239/311154 objects degraded (10.040%), 8 pgs degraded (PG_DEGRADED) 2024-03-20T09:26:43.719 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ace1450:::benchmark_data_smithi122_168655_object76226:head# 2024-03-20T09:26:43.719 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.719 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8acec1fd:::benchmark_data_smithi122_168655_object71187:head# 2024-03-20T09:26:43.719 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.719 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8acef77a:::benchmark_data_smithi122_168655_object34919:head# 2024-03-20T09:26:43.720 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.720 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8acfab8a:::benchmark_data_smithi122_168655_object62944:head# 2024-03-20T09:26:43.720 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.720 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ad0a171:::benchmark_data_smithi122_168655_object57193:head# 2024-03-20T09:26:43.720 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.720 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ad10efe:::benchmark_data_smithi122_168655_object44674:head# 2024-03-20T09:26:43.720 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.720 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ad24917:::benchmark_data_smithi122_168655_object79576:head# 2024-03-20T09:26:43.720 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.720 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ad249ab:::benchmark_data_smithi122_168655_object106213:head# 2024-03-20T09:26:43.720 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.720 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ad28ec5:::benchmark_data_smithi122_168655_object112030:head# 2024-03-20T09:26:43.720 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.720 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ad293b5:::benchmark_data_smithi122_168655_object68932:head# 2024-03-20T09:26:43.721 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.721 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ad4e286:::benchmark_data_smithi122_168655_object83055:head# 2024-03-20T09:26:43.721 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.721 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ad625f4:::benchmark_data_smithi122_168655_object35667:head# 2024-03-20T09:26:43.721 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.721 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ad66853:::benchmark_data_smithi122_168655_object66855:head# 2024-03-20T09:26:43.721 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.721 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ad7161d:::benchmark_data_smithi122_168655_object104076:head# 2024-03-20T09:26:43.721 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.721 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ad77720:::benchmark_data_smithi122_168655_object101511:head# 2024-03-20T09:26:43.721 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.721 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ad7c5d9:::benchmark_data_smithi122_168655_object76462:head# 2024-03-20T09:26:43.721 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.722 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ad7d689:::benchmark_data_smithi122_168655_object71759:head# 2024-03-20T09:26:43.722 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.722 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ad7ffce:::benchmark_data_smithi122_168655_object117865:head# 2024-03-20T09:26:43.722 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.722 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ad8297b:::benchmark_data_smithi122_168655_object117685:head# 2024-03-20T09:26:43.722 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.722 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ad873b7:::benchmark_data_smithi122_168655_object106132:head# 2024-03-20T09:26:43.722 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.722 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ad9bce3:::benchmark_data_smithi122_168655_object45140:head# 2024-03-20T09:26:43.722 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.722 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ada081e:::benchmark_data_smithi122_168655_object11713:head# 2024-03-20T09:26:43.722 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.722 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ada0c56:::benchmark_data_smithi122_168655_object64935:head# 2024-03-20T09:26:43.722 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.723 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ada58d7:::benchmark_data_smithi122_168655_object63991:head# 2024-03-20T09:26:43.723 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.723 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8adb29ec:::benchmark_data_smithi122_168655_object30648:head# 2024-03-20T09:26:43.723 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.723 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8adc3333:::benchmark_data_smithi122_168655_object116414:head# 2024-03-20T09:26:43.723 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.723 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8adc49e9:::benchmark_data_smithi122_168655_object96451:head# 2024-03-20T09:26:43.723 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.723 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8adc716d:::benchmark_data_smithi122_168655_object30522:head# 2024-03-20T09:26:43.723 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.723 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8adcf96d:::benchmark_data_smithi122_168655_object61160:head# 2024-03-20T09:26:43.723 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.723 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8add55d2:::benchmark_data_smithi122_168655_object117035:head# 2024-03-20T09:26:43.723 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.724 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8adda190:::benchmark_data_smithi122_168655_object89287:head# 2024-03-20T09:26:43.724 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.724 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8adda2f7:::benchmark_data_smithi122_168655_object103815:head# 2024-03-20T09:26:43.724 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.724 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8adee73b:::benchmark_data_smithi122_168655_object92512:head# 2024-03-20T09:26:43.724 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.724 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ae13940:::benchmark_data_smithi122_168655_object101028:head# 2024-03-20T09:26:43.724 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.724 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ae1c21e:::benchmark_data_smithi122_168655_object69600:head# 2024-03-20T09:26:43.724 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.724 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ae23023:::benchmark_data_smithi122_168655_object12613:head# 2024-03-20T09:26:43.724 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.724 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ae25511:::benchmark_data_smithi122_168655_object49327:head# 2024-03-20T09:26:43.724 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.724 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ae3f17e:::benchmark_data_smithi122_168655_object116744:head# 2024-03-20T09:26:43.725 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.725 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ae45adb:::benchmark_data_smithi122_168655_object32717:head# 2024-03-20T09:26:43.725 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.725 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ae493b7:::benchmark_data_smithi122_168655_object80808:head# 2024-03-20T09:26:43.725 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.725 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ae65de3:::benchmark_data_smithi122_168655_object45624:head# 2024-03-20T09:26:43.725 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.725 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ae695df:::benchmark_data_smithi122_168655_object17278:head# 2024-03-20T09:26:43.725 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.725 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ae7772e:::benchmark_data_smithi122_168655_object33790:head# 2024-03-20T09:26:43.725 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.725 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ae7a267:::benchmark_data_smithi122_168655_object56648:head# 2024-03-20T09:26:43.725 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.725 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ae7a866:::benchmark_data_smithi122_168655_object76210:head# 2024-03-20T09:26:43.726 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.726 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ae82470:::benchmark_data_smithi122_168655_object17684:head# 2024-03-20T09:26:43.726 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.726 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ae8b83a:::benchmark_data_smithi122_168655_object113721:head# 2024-03-20T09:26:43.726 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.726 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ae98e7c:::benchmark_data_smithi122_168655_object99518:head# 2024-03-20T09:26:43.726 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.726 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aea1abb:::benchmark_data_smithi122_168655_object72081:head# 2024-03-20T09:26:43.726 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.726 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aeaaf2a:::benchmark_data_smithi122_168655_object40200:head# 2024-03-20T09:26:43.726 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.726 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aeb0053:::benchmark_data_smithi122_168655_object75126:head# 2024-03-20T09:26:43.726 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.726 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aeb2aa0:::benchmark_data_smithi122_168655_object115200:head# 2024-03-20T09:26:43.727 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.727 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aebee31:::benchmark_data_smithi122_168655_object100648:head# 2024-03-20T09:26:43.727 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.727 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aec4c43:::benchmark_data_smithi122_168655_object52101:head# 2024-03-20T09:26:43.727 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.727 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aec8ee3:::benchmark_data_smithi122_168655_object54849:head# 2024-03-20T09:26:43.727 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.727 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aee1f2d:::benchmark_data_smithi122_168655_object79595:head# 2024-03-20T09:26:43.727 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.727 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8aee38bf:::benchmark_data_smithi122_168655_object118055:head# 2024-03-20T09:26:43.727 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.727 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8af07aad:::benchmark_data_smithi122_168655_object112971:head# 2024-03-20T09:26:43.727 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.727 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8af0aa92:::benchmark_data_smithi122_168655_object87272:head# 2024-03-20T09:26:43.728 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.728 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8af23494:::benchmark_data_smithi122_168655_object60210:head# 2024-03-20T09:26:43.728 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.728 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8af26e92:::benchmark_data_smithi122_168655_object98697:head# 2024-03-20T09:26:43.728 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.728 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8af281da:::benchmark_data_smithi122_168655_object71822:head# 2024-03-20T09:26:43.728 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.728 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8af2d0d1:::benchmark_data_smithi122_168655_object99289:head# 2024-03-20T09:26:43.728 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.728 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8af42c7e:::benchmark_data_smithi122_168655_object23015:head# 2024-03-20T09:26:43.728 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.728 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8af66966:::benchmark_data_smithi122_168655_object77002:head# 2024-03-20T09:26:43.728 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.729 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8af66eb8:::benchmark_data_smithi122_168655_object46901:head# 2024-03-20T09:26:43.729 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.729 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8afa160e:::benchmark_data_smithi122_168655_object12708:head# 2024-03-20T09:26:43.729 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.729 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8afa21c2:::benchmark_data_smithi122_168655_object40847:head# 2024-03-20T09:26:43.729 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.729 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8afa7612:::benchmark_data_smithi122_168655_object62257:head# 2024-03-20T09:26:43.729 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.729 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8afb4eae:::benchmark_data_smithi122_168655_object55434:head# 2024-03-20T09:26:43.729 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.729 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8afb74b1:::benchmark_data_smithi122_168655_object47147:head# 2024-03-20T09:26:43.729 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.729 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8afca9de:::benchmark_data_smithi122_168655_object20347:head# 2024-03-20T09:26:43.729 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.730 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8afcb108:::benchmark_data_smithi122_168655_object67368:head# 2024-03-20T09:26:43.730 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.730 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8afcdeb5:::benchmark_data_smithi122_168655_object43633:head# 2024-03-20T09:26:43.730 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.730 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8afd4d89:::benchmark_data_smithi122_168655_object88387:head# 2024-03-20T09:26:43.730 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.730 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8afe1965:::benchmark_data_smithi122_168655_object67904:head# 2024-03-20T09:26:43.730 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.730 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8afebfa6:::benchmark_data_smithi122_168655_object38084:head# 2024-03-20T09:26:43.730 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.730 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b009f6c:::benchmark_data_smithi122_168655_object81008:head# 2024-03-20T09:26:43.730 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.730 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b0120ef:::benchmark_data_smithi122_168655_object115256:head# 2024-03-20T09:26:43.731 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.731 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b01ab71:::benchmark_data_smithi122_168655_object66247:head# 2024-03-20T09:26:43.731 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.731 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b02b388:::benchmark_data_smithi122_168655_object88140:head# 2024-03-20T09:26:43.731 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.731 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b02cfa1:::benchmark_data_smithi122_168655_object101136:head# 2024-03-20T09:26:43.731 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.731 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b02d9b3:::benchmark_data_smithi122_168655_object57927:head# 2024-03-20T09:26:43.731 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.731 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b03bab3:::benchmark_data_smithi122_168655_object31984:head# 2024-03-20T09:26:43.731 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.731 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b0465bc:::benchmark_data_smithi122_168655_object37145:head# 2024-03-20T09:26:43.731 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.731 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b04ade3:::benchmark_data_smithi122_168655_object79200:head# 2024-03-20T09:26:43.732 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.732 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b04db01:::benchmark_data_smithi122_168655_object60671:head# 2024-03-20T09:26:43.732 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.732 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b0514eb:::benchmark_data_smithi122_168655_object24554:head# 2024-03-20T09:26:43.732 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.732 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b0614d4:::benchmark_data_smithi122_168655_object70575:head# 2024-03-20T09:26:43.732 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.732 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b0773f8:::benchmark_data_smithi122_168655_object115948:head# 2024-03-20T09:26:43.732 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.732 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b07af7a:::benchmark_data_smithi122_168655_object48832:head# 2024-03-20T09:26:43.732 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.732 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b0843ad:::benchmark_data_smithi122_168655_object119531:head# 2024-03-20T09:26:43.732 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.732 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b0866f5:::benchmark_data_smithi122_168655_object14289:head# 2024-03-20T09:26:43.733 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.733 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b08ead3:::benchmark_data_smithi122_168655_object105155:head# 2024-03-20T09:26:43.733 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.733 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b0977e9:::benchmark_data_smithi122_168655_object119736:head# 2024-03-20T09:26:43.733 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.733 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b0b53d6:::benchmark_data_smithi122_168655_object35648:head# 2024-03-20T09:26:43.733 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.733 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b0c351a:::benchmark_data_smithi122_168655_object99856:head# 2024-03-20T09:26:43.733 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.733 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b0cb76e:::benchmark_data_smithi122_168655_object113951:head# 2024-03-20T09:26:43.733 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.733 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b0ceacc:::benchmark_data_smithi122_168655_object99272:head# 2024-03-20T09:26:43.733 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.734 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b0cf419:::benchmark_data_smithi122_168655_object102489:head# 2024-03-20T09:26:43.734 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.734 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b0d22d9:::benchmark_data_smithi122_168655_object116359:head# 2024-03-20T09:26:43.734 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.734 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b0e690d:::benchmark_data_smithi122_168655_object54759:head# 2024-03-20T09:26:43.734 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.734 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b1134b9:::benchmark_data_smithi122_168655_object93577:head# 2024-03-20T09:26:43.734 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.734 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b1196ad:::benchmark_data_smithi122_168655_object87827:head# 2024-03-20T09:26:43.734 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.734 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b12645f:::benchmark_data_smithi122_168655_object92817:head# 2024-03-20T09:26:43.734 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.735 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b12cd78:::benchmark_data_smithi122_168655_object34602:head# 2024-03-20T09:26:43.735 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.735 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b134f65:::benchmark_data_smithi122_168655_object97512:head# 2024-03-20T09:26:43.735 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.735 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b13549f:::benchmark_data_smithi122_168655_object59935:head# 2024-03-20T09:26:43.735 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.735 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b151349:::benchmark_data_smithi122_168655_object84819:head# 2024-03-20T09:26:43.735 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.735 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b1566ef:::benchmark_data_smithi122_168655_object65529:head# 2024-03-20T09:26:43.735 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.735 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b159a75:::benchmark_data_smithi122_168655_object49413:head# 2024-03-20T09:26:43.735 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.735 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b16c0c8:::benchmark_data_smithi122_168655_object103920:head# 2024-03-20T09:26:43.736 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.736 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b16fb49:::benchmark_data_smithi122_168655_object95208:head# 2024-03-20T09:26:43.736 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.736 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b1762ec:::benchmark_data_smithi122_168655_object40320:head# 2024-03-20T09:26:43.736 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.736 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b17a74b:::benchmark_data_smithi122_168655_object47426:head# 2024-03-20T09:26:43.736 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.736 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b17d4bd:::benchmark_data_smithi122_168655_object66891:head# 2024-03-20T09:26:43.736 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.736 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b18214d:::benchmark_data_smithi122_168655_object81506:head# 2024-03-20T09:26:43.736 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.736 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b18703f:::benchmark_data_smithi122_168655_object118453:head# 2024-03-20T09:26:43.736 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.736 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b18b08a:::benchmark_data_smithi122_168655_object82592:head# 2024-03-20T09:26:43.737 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.737 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b19939a:::benchmark_data_smithi122_168655_object56607:head# 2024-03-20T09:26:43.737 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.737 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b1a23f7:::benchmark_data_smithi122_168655_object115509:head# 2024-03-20T09:26:43.737 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.737 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b1acbfe:::benchmark_data_smithi122_168655_object100987:head# 2024-03-20T09:26:43.737 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.737 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b1b081e:::benchmark_data_smithi122_168655_object37196:head# 2024-03-20T09:26:43.737 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.737 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b1cd86e:::benchmark_data_smithi122_168655_object73477:head# 2024-03-20T09:26:43.737 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.737 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b1d0303:::benchmark_data_smithi122_168655_object115882:head# 2024-03-20T09:26:43.737 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.737 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b1d5678:::benchmark_data_smithi122_168655_object63800:head# 2024-03-20T09:26:43.737 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.738 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b1de169:::benchmark_data_smithi122_168655_object13093:head# 2024-03-20T09:26:43.738 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.738 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b1df4e0:::benchmark_data_smithi122_168655_object19586:head# 2024-03-20T09:26:43.738 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.738 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b1e35a1:::benchmark_data_smithi122_168655_object96321:head# 2024-03-20T09:26:43.738 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.738 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b1ff329:::benchmark_data_smithi122_168655_object64943:head# 2024-03-20T09:26:43.738 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.738 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b203139:::benchmark_data_smithi122_168655_object117143:head# 2024-03-20T09:26:43.738 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.738 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b20c9d5:::benchmark_data_smithi122_168655_object90901:head# 2024-03-20T09:26:43.738 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.738 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b21e460:::benchmark_data_smithi122_168655_object73938:head# 2024-03-20T09:26:43.738 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.739 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b21e81d:::benchmark_data_smithi122_168655_object46443:head# 2024-03-20T09:26:43.739 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.739 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b223f2c:::benchmark_data_smithi122_168655_object90532:head# 2024-03-20T09:26:43.739 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.739 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b240e31:::benchmark_data_smithi122_168655_object111954:head# 2024-03-20T09:26:43.739 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.739 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b24127e:::benchmark_data_smithi122_168655_object10256:head# 2024-03-20T09:26:43.739 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.739 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b246f69:::benchmark_data_smithi122_168655_object103128:head# 2024-03-20T09:26:43.739 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.739 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b254ea2:::benchmark_data_smithi122_168655_object106825:head# 2024-03-20T09:26:43.739 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.739 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b259cb5:::benchmark_data_smithi122_168655_object16566:head# 2024-03-20T09:26:43.739 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.739 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b25b9b5:::benchmark_data_smithi122_168655_object98634:head# 2024-03-20T09:26:43.740 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.740 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b272b06:::benchmark_data_smithi122_168655_object16077:head# 2024-03-20T09:26:43.740 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.740 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b28bb34:::benchmark_data_smithi122_168655_object66499:head# 2024-03-20T09:26:43.740 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.740 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b28ee6e:::benchmark_data_smithi122_168655_object53269:head# 2024-03-20T09:26:43.740 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.740 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b2a2c59:::benchmark_data_smithi122_168655_object52198:head# 2024-03-20T09:26:43.740 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.740 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b2b769b:::benchmark_data_smithi122_168655_object67130:head# 2024-03-20T09:26:43.740 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.740 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b2d29b8:::benchmark_data_smithi122_168655_object62718:head# 2024-03-20T09:26:43.740 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.740 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b2ee83f:::benchmark_data_smithi122_168655_object28523:head# 2024-03-20T09:26:43.741 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.741 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b2fc286:::benchmark_data_smithi122_168655_object15623:head# 2024-03-20T09:26:43.741 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.741 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b30507e:::benchmark_data_smithi122_168655_object51634:head# 2024-03-20T09:26:43.741 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.741 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b306be2:::benchmark_data_smithi122_168655_object31395:head# 2024-03-20T09:26:43.741 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.741 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b31211a:::benchmark_data_smithi122_168655_object104690:head# 2024-03-20T09:26:43.741 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.741 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b322e97:::benchmark_data_smithi122_168655_object18252:head# 2024-03-20T09:26:43.741 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.741 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b32658e:::benchmark_data_smithi122_168655_object32888:head# 2024-03-20T09:26:43.741 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.741 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b32ac82:::benchmark_data_smithi122_168655_object99612:head# 2024-03-20T09:26:43.742 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.742 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b32c069:::benchmark_data_smithi122_168655_object64628:head# 2024-03-20T09:26:43.742 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.742 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b32d5be:::benchmark_data_smithi122_168655_object57815:head# 2024-03-20T09:26:43.742 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.742 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b3332fd:::benchmark_data_smithi122_168655_object59330:head# 2024-03-20T09:26:43.742 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.742 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b341f4a:::benchmark_data_smithi122_168655_object43670:head# 2024-03-20T09:26:43.742 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.742 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b34b7f4:::benchmark_data_smithi122_168655_object112141:head# 2024-03-20T09:26:43.742 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.742 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b34c117:::benchmark_data_smithi122_168655_object68018:head# 2024-03-20T09:26:43.742 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.743 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b3500d3:::benchmark_data_smithi122_168655_object50436:head# 2024-03-20T09:26:43.743 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.743 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b35ac89:::benchmark_data_smithi122_168655_object83454:head# 2024-03-20T09:26:43.743 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.743 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b366345:::benchmark_data_smithi122_168655_object95460:head# 2024-03-20T09:26:43.743 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.743 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b37e7cd:::benchmark_data_smithi122_168655_object45084:head# 2024-03-20T09:26:43.743 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.743 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b38f71a:::benchmark_data_smithi122_168655_object97311:head# 2024-03-20T09:26:43.743 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.743 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b391733:::benchmark_data_smithi122_168655_object70066:head# 2024-03-20T09:26:43.743 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.743 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b395783:::benchmark_data_smithi122_168655_object41582:head# 2024-03-20T09:26:43.743 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.744 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b39c830:::benchmark_data_smithi122_168655_object66798:head# 2024-03-20T09:26:43.744 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.744 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b3a5851:::benchmark_data_smithi122_168655_object85942:head# 2024-03-20T09:26:43.744 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.744 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b3a5dd0:::benchmark_data_smithi122_168655_object51982:head# 2024-03-20T09:26:43.744 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.744 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b3aa4d6:::benchmark_data_smithi122_168655_object84240:head# 2024-03-20T09:26:43.744 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.744 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b3b806b:::benchmark_data_smithi122_168655_object98854:head# 2024-03-20T09:26:43.744 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.744 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b3bb45a:::benchmark_data_smithi122_168655_object96711:head# 2024-03-20T09:26:43.744 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.744 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b3c31ab:::benchmark_data_smithi122_168655_object46098:head# 2024-03-20T09:26:43.744 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.744 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b3c73bf:::benchmark_data_smithi122_168655_object87887:head# 2024-03-20T09:26:43.745 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.745 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b3cbc87:::benchmark_data_smithi122_168655_object113822:head# 2024-03-20T09:26:43.745 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.745 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b3e2399:::benchmark_data_smithi122_168655_object46984:head# 2024-03-20T09:26:43.745 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.745 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b3e3ec4:::benchmark_data_smithi122_168655_object40014:head# 2024-03-20T09:26:43.745 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.745 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b3e9f96:::benchmark_data_smithi122_168655_object101788:head# 2024-03-20T09:26:43.745 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.745 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b3f8664:::benchmark_data_smithi122_168655_object19645:head# 2024-03-20T09:26:43.745 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.745 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b3f9bbb:::benchmark_data_smithi122_168655_object13542:head# 2024-03-20T09:26:43.745 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.745 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b403155:::benchmark_data_smithi122_168655_object59280:head# 2024-03-20T09:26:43.745 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.746 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b411567:::benchmark_data_smithi122_168655_object59169:head# 2024-03-20T09:26:43.746 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.746 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b41b715:::benchmark_data_smithi122_168655_object47817:head# 2024-03-20T09:26:43.746 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.746 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b41fcee:::benchmark_data_smithi122_168655_object71828:head# 2024-03-20T09:26:43.746 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.746 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b42116c:::benchmark_data_smithi122_168655_object57985:head# 2024-03-20T09:26:43.746 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.746 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b4273b2:::benchmark_data_smithi122_168655_object82840:head# 2024-03-20T09:26:43.746 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.746 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b4326a3:::benchmark_data_smithi122_168655_object101501:head# 2024-03-20T09:26:43.746 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.746 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b43605a:::benchmark_data_smithi122_168655_object96573:head# 2024-03-20T09:26:43.747 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.747 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b43809d:::benchmark_data_smithi122_168655_object115300:head# 2024-03-20T09:26:43.747 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.747 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b4430ca:::benchmark_data_smithi122_168655_object57585:head# 2024-03-20T09:26:43.747 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.747 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b4463ec:::benchmark_data_smithi122_168655_object109889:head# 2024-03-20T09:26:43.747 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.747 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b4473d6:::benchmark_data_smithi122_168655_object54740:head# 2024-03-20T09:26:43.747 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.747 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b44c23e:::benchmark_data_smithi122_168655_object57160:head# 2024-03-20T09:26:43.747 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.747 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b45629d:::benchmark_data_smithi122_168655_object78856:head# 2024-03-20T09:26:43.747 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.748 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b465054:::benchmark_data_smithi122_168655_object65655:head# 2024-03-20T09:26:43.748 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.748 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b46664b:::benchmark_data_smithi122_168655_object24638:head# 2024-03-20T09:26:43.748 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.748 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b47b004:::benchmark_data_smithi122_168655_object54224:head# 2024-03-20T09:26:43.748 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.748 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b497114:::benchmark_data_smithi122_168655_object101724:head# 2024-03-20T09:26:43.748 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.748 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b49c2e4:::benchmark_data_smithi122_168655_object74848:head# 2024-03-20T09:26:43.748 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.748 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b4a083e:::benchmark_data_smithi122_168655_object36647:head# 2024-03-20T09:26:43.748 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.748 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b4a74e6:::benchmark_data_smithi122_168655_object40602:head# 2024-03-20T09:26:43.748 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.748 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b4b6606:::benchmark_data_smithi122_168655_object49208:head# 2024-03-20T09:26:43.749 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.749 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b4bc68e:::benchmark_data_smithi122_168655_object97887:head# 2024-03-20T09:26:43.749 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.749 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b4d62f6:::benchmark_data_smithi122_168655_object48290:head# 2024-03-20T09:26:43.749 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.749 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b4f956d:::benchmark_data_smithi122_168655_object81964:head# 2024-03-20T09:26:43.749 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.749 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b5142b4:::benchmark_data_smithi122_168655_object98019:head# 2024-03-20T09:26:43.749 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.749 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b5142cc:::benchmark_data_smithi122_168655_object46120:head# 2024-03-20T09:26:43.749 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.749 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b516daf:::benchmark_data_smithi122_168655_object62029:head# 2024-03-20T09:26:43.749 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.749 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b51c9e0:::benchmark_data_smithi122_168655_object60238:head# 2024-03-20T09:26:43.750 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.750 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b5210f9:::benchmark_data_smithi122_168655_object64624:head# 2024-03-20T09:26:43.750 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.750 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b567194:::benchmark_data_smithi122_168655_object46316:head# 2024-03-20T09:26:43.750 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.750 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b567d20:::benchmark_data_smithi122_168655_object65406:head# 2024-03-20T09:26:43.750 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.750 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b569269:::benchmark_data_smithi122_168655_object27034:head# 2024-03-20T09:26:43.750 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.750 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b576b33:::benchmark_data_smithi122_168655_object72424:head# 2024-03-20T09:26:43.750 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.750 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b587a12:::benchmark_data_smithi122_168655_object74317:head# 2024-03-20T09:26:43.750 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.751 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b589176:::benchmark_data_smithi122_168655_object26891:head# 2024-03-20T09:26:43.751 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.751 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b591ee6:::benchmark_data_smithi122_168655_object51912:head# 2024-03-20T09:26:43.751 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.751 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b598143:::benchmark_data_smithi122_168655_object25558:head# 2024-03-20T09:26:43.751 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.751 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b59d23e:::benchmark_data_smithi122_168655_object37089:head# 2024-03-20T09:26:43.751 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.751 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b59dbb1:::benchmark_data_smithi122_168655_object13222:head# 2024-03-20T09:26:43.751 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.751 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b5a1153:::benchmark_data_smithi122_168655_object14696:head# 2024-03-20T09:26:43.751 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.751 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b5a7c62:::benchmark_data_smithi122_168655_object104770:head# 2024-03-20T09:26:43.751 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.752 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b5b414b:::benchmark_data_smithi122_168655_object115804:head# 2024-03-20T09:26:43.752 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.752 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b5b4be6:::benchmark_data_smithi122_168655_object14772:head# 2024-03-20T09:26:43.752 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.752 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b5b54d0:::benchmark_data_smithi122_168655_object43202:head# 2024-03-20T09:26:43.752 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.752 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b5b5d4c:::benchmark_data_smithi122_168655_object50036:head# 2024-03-20T09:26:43.752 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.752 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b5c258f:::benchmark_data_smithi122_168655_object65682:head# 2024-03-20T09:26:43.752 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.752 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b5cf573:::benchmark_data_smithi122_168655_object45004:head# 2024-03-20T09:26:43.752 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.752 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b5d051a:::benchmark_data_smithi122_168655_object111836:head# 2024-03-20T09:26:43.752 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.753 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b5d33e4:::benchmark_data_smithi122_168655_object73424:head# 2024-03-20T09:26:43.753 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.753 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b5da02c:::benchmark_data_smithi122_168655_object100460:head# 2024-03-20T09:26:43.753 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.753 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b5e9bfb:::benchmark_data_smithi122_168655_object50688:head# 2024-03-20T09:26:43.753 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.753 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b5edce5:::benchmark_data_smithi122_168655_object72829:head# 2024-03-20T09:26:43.753 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.753 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b5eea46:::benchmark_data_smithi122_168655_object111921:head# 2024-03-20T09:26:43.753 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.753 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b5f17c2:::benchmark_data_smithi122_168655_object41543:head# 2024-03-20T09:26:43.753 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.753 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b5f80dd:::benchmark_data_smithi122_168655_object107342:head# 2024-03-20T09:26:43.753 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.754 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b5faeba:::benchmark_data_smithi122_168655_object82711:head# 2024-03-20T09:26:43.754 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.754 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b60a133:::benchmark_data_smithi122_168655_object105953:head# 2024-03-20T09:26:43.754 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.754 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b61dc60:::benchmark_data_smithi122_168655_object42972:head# 2024-03-20T09:26:43.754 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.754 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b622aa8:::benchmark_data_smithi122_168655_object101834:head# 2024-03-20T09:26:43.754 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.754 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b62413d:::benchmark_data_smithi122_168655_object46282:head# 2024-03-20T09:26:43.754 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.754 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b627875:::benchmark_data_smithi122_168655_object26497:head# 2024-03-20T09:26:43.754 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.754 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b631d44:::benchmark_data_smithi122_168655_object117413:head# 2024-03-20T09:26:43.754 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.755 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b648f03:::benchmark_data_smithi122_168655_object113707:head# 2024-03-20T09:26:43.755 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.755 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b66c158:::benchmark_data_smithi122_168655_object19827:head# 2024-03-20T09:26:43.755 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.755 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b66cb32:::benchmark_data_smithi122_168655_object56533:head# 2024-03-20T09:26:43.755 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.755 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b671978:::benchmark_data_smithi122_168655_object10691:head# 2024-03-20T09:26:43.755 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.755 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b675926:::benchmark_data_smithi122_168655_object92217:head# 2024-03-20T09:26:43.755 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.755 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b67cada:::benchmark_data_smithi122_168655_object14610:head# 2024-03-20T09:26:43.755 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.755 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b686d01:::benchmark_data_smithi122_168655_object85533:head# 2024-03-20T09:26:43.755 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.756 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b68c7ac:::benchmark_data_smithi122_168655_object58451:head# 2024-03-20T09:26:43.756 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.756 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b693259:::benchmark_data_smithi122_168655_object78642:head# 2024-03-20T09:26:43.756 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.756 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b697d82:::benchmark_data_smithi122_168655_object49519:head# 2024-03-20T09:26:43.756 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.756 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b6a2fbc:::benchmark_data_smithi122_168655_object18661:head# 2024-03-20T09:26:43.756 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.756 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b6a5ede:::benchmark_data_smithi122_168655_object102239:head# 2024-03-20T09:26:43.756 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.756 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b6d301f:::benchmark_data_smithi122_168655_object92715:head# 2024-03-20T09:26:43.756 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.756 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b6d5ced:::benchmark_data_smithi122_168655_object85005:head# 2024-03-20T09:26:43.756 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.757 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b6d7c0d:::benchmark_data_smithi122_168655_object119202:head# 2024-03-20T09:26:43.757 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.757 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b6da525:::benchmark_data_smithi122_168655_object56331:head# 2024-03-20T09:26:43.757 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.757 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b6e179f:::benchmark_data_smithi122_168655_object27381:head# 2024-03-20T09:26:43.757 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.757 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b6e874d:::benchmark_data_smithi122_168655_object30997:head# 2024-03-20T09:26:43.757 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.757 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b6ea6c4:::benchmark_data_smithi122_168655_object109658:head# 2024-03-20T09:26:43.757 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.757 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b6efad3:::benchmark_data_smithi122_168655_object90205:head# 2024-03-20T09:26:43.757 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.757 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b6f59bd:::benchmark_data_smithi122_168655_object56603:head# 2024-03-20T09:26:43.758 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.758 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b6f5cf1:::benchmark_data_smithi122_168655_object19359:head# 2024-03-20T09:26:43.758 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.758 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b70162b:::benchmark_data_smithi122_168655_object14983:head# 2024-03-20T09:26:43.758 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.758 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b705d33:::benchmark_data_smithi122_168655_object22128:head# 2024-03-20T09:26:43.758 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.758 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b7296c5:::benchmark_data_smithi122_168655_object114825:head# 2024-03-20T09:26:43.758 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.758 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b729d68:::benchmark_data_smithi122_168655_object38633:head# 2024-03-20T09:26:43.758 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.758 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b72e0dd:::benchmark_data_smithi122_168655_object42254:head# 2024-03-20T09:26:43.758 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.758 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b72e3c6:::benchmark_data_smithi122_168655_object34733:head# 2024-03-20T09:26:43.759 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.759 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b730ace:::benchmark_data_smithi122_168655_object91637:head# 2024-03-20T09:26:43.759 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.759 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b73d0d7:::benchmark_data_smithi122_168655_object84624:head# 2024-03-20T09:26:43.759 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.759 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b747cf6:::benchmark_data_smithi122_168655_object118736:head# 2024-03-20T09:26:43.759 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.759 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b7481f0:::benchmark_data_smithi122_168655_object42805:head# 2024-03-20T09:26:43.759 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.759 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b75a513:::benchmark_data_smithi122_168655_object78869:head# 2024-03-20T09:26:43.759 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.759 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b75f86c:::benchmark_data_smithi122_168655_object81856:head# 2024-03-20T09:26:43.759 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.759 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b769b6f:::benchmark_data_smithi122_168655_object107791:head# 2024-03-20T09:26:43.760 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.760 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b7703e4:::benchmark_data_smithi122_168655_object53085:head# 2024-03-20T09:26:43.760 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.760 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b77b0d4:::benchmark_data_smithi122_168655_object18460:head# 2024-03-20T09:26:43.760 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.760 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b77dee5:::benchmark_data_smithi122_168655_object65348:head# 2024-03-20T09:26:43.760 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.760 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b783dec:::benchmark_data_smithi122_168655_object76121:head# 2024-03-20T09:26:43.760 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.760 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b7914ca:::benchmark_data_smithi122_168655_object82288:head# 2024-03-20T09:26:43.760 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.760 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b793d78:::benchmark_data_smithi122_168655_object111002:head# 2024-03-20T09:26:43.760 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.760 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b7a2cc6:::benchmark_data_smithi122_168655_object77173:head# 2024-03-20T09:26:43.761 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.761 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b7a400e:::benchmark_data_smithi122_168655_object61093:head# 2024-03-20T09:26:43.761 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.761 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b7b465a:::benchmark_data_smithi122_168655_object21863:head# 2024-03-20T09:26:43.761 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.761 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b7b72b5:::benchmark_data_smithi122_168655_object12976:head# 2024-03-20T09:26:43.761 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.761 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b7b7ba9:::benchmark_data_smithi122_168655_object98327:head# 2024-03-20T09:26:43.761 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.761 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b7b944a:::benchmark_data_smithi122_168655_object23477:head# 2024-03-20T09:26:43.761 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.761 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b7cddb5:::benchmark_data_smithi122_168655_object116594:head# 2024-03-20T09:26:43.761 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.761 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b7d3405:::benchmark_data_smithi122_168655_object48494:head# 2024-03-20T09:26:43.762 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.762 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b7de3a9:::benchmark_data_smithi122_168655_object25602:head# 2024-03-20T09:26:43.762 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.762 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b7dfa5c:::benchmark_data_smithi122_168655_object112925:head# 2024-03-20T09:26:43.762 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.762 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b7e208b:::benchmark_data_smithi122_168655_object119267:head# 2024-03-20T09:26:43.762 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.762 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b7e4489:::benchmark_data_smithi122_168655_object32769:head# 2024-03-20T09:26:43.762 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.762 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b7e5581:::benchmark_data_smithi122_168655_object101194:head# 2024-03-20T09:26:43.762 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.762 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b7f18d8:::benchmark_data_smithi122_168655_object39622:head# 2024-03-20T09:26:43.762 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.762 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b7ff37d:::benchmark_data_smithi122_168655_object91771:head# 2024-03-20T09:26:43.763 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.763 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b803c95:::benchmark_data_smithi122_168655_object35505:head# 2024-03-20T09:26:43.763 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.763 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b8086c2:::benchmark_data_smithi122_168655_object71483:head# 2024-03-20T09:26:43.763 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.763 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b809699:::benchmark_data_smithi122_168655_object96289:head# 2024-03-20T09:26:43.763 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.763 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b812613:::benchmark_data_smithi122_168655_object96941:head# 2024-03-20T09:26:43.763 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.763 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b813617:::benchmark_data_smithi122_168655_object37331:head# 2024-03-20T09:26:43.763 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.763 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b8200c5:::benchmark_data_smithi122_168655_object11105:head# 2024-03-20T09:26:43.763 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.763 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b82fe61:::benchmark_data_smithi122_168655_object94958:head# 2024-03-20T09:26:43.764 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.764 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b835238:::benchmark_data_smithi122_168655_object44319:head# 2024-03-20T09:26:43.764 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.764 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b8429e7:::benchmark_data_smithi122_168655_object35899:head# 2024-03-20T09:26:43.764 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.764 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b84c24e:::benchmark_data_smithi122_168655_object112044:head# 2024-03-20T09:26:43.764 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.764 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b857c1a:::benchmark_data_smithi122_168655_object101841:head# 2024-03-20T09:26:43.764 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.764 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b864ef5:::benchmark_data_smithi122_168655_object108390:head# 2024-03-20T09:26:43.764 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.764 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b871083:::benchmark_data_smithi122_168655_object77025:head# 2024-03-20T09:26:43.764 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.764 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b875561:::benchmark_data_smithi122_168655_object113333:head# 2024-03-20T09:26:43.765 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.765 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b87cb58:::benchmark_data_smithi122_168655_object83992:head# 2024-03-20T09:26:43.765 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.765 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b886717:::benchmark_data_smithi122_168655_object54627:head# 2024-03-20T09:26:43.765 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.765 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b895b98:::benchmark_data_smithi122_168655_object86002:head# 2024-03-20T09:26:43.765 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.765 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b896d6a:::benchmark_data_smithi122_168655_object33667:head# 2024-03-20T09:26:43.765 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.765 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b8a13d3:::benchmark_data_smithi122_168655_object89759:head# 2024-03-20T09:26:43.765 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.765 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b8a6833:::benchmark_data_smithi122_168655_object57277:head# 2024-03-20T09:26:43.765 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.765 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b8a7751:::benchmark_data_smithi122_168655_object45382:head# 2024-03-20T09:26:43.766 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.766 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b8ab429:::benchmark_data_smithi122_168655_object94880:head# 2024-03-20T09:26:43.766 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.766 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b8ae4cc:::benchmark_data_smithi122_168655_object37550:head# 2024-03-20T09:26:43.766 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.766 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b8b3a65:::benchmark_data_smithi122_168655_object31511:head# 2024-03-20T09:26:43.766 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.766 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b8cd04d:::benchmark_data_smithi122_168655_object90428:head# 2024-03-20T09:26:43.766 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.766 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b8de3be:::benchmark_data_smithi122_168655_object88885:head# 2024-03-20T09:26:43.766 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.766 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b8debf4:::benchmark_data_smithi122_168655_object71760:head# 2024-03-20T09:26:43.766 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.766 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b8dfbc8:::benchmark_data_smithi122_168655_object81574:head# 2024-03-20T09:26:43.767 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.767 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b8e6349:::benchmark_data_smithi122_168655_object92845:head# 2024-03-20T09:26:43.767 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.767 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b8fd667:::benchmark_data_smithi122_168655_object81963:head# 2024-03-20T09:26:43.767 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.767 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b9009f8:::benchmark_data_smithi122_168655_object24472:head# 2024-03-20T09:26:43.767 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.767 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b9014d6:::benchmark_data_smithi122_168655_object20479:head# 2024-03-20T09:26:43.767 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.767 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b90c647:::benchmark_data_smithi122_168655_object13577:head# 2024-03-20T09:26:43.767 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.767 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b90ff94:::benchmark_data_smithi122_168655_object101688:head# 2024-03-20T09:26:43.767 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.767 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b93e9fb:::benchmark_data_smithi122_168655_object60438:head# 2024-03-20T09:26:43.767 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.767 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b942276:::benchmark_data_smithi122_168655_object35819:head# 2024-03-20T09:26:43.768 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.768 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b9426ff:::benchmark_data_smithi122_168655_object28229:head# 2024-03-20T09:26:43.768 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.768 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b94c8b9:::benchmark_data_smithi122_168655_object64226:head# 2024-03-20T09:26:43.768 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.768 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b94e790:::benchmark_data_smithi122_168655_object64683:head# 2024-03-20T09:26:43.768 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.768 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b94ec25:::benchmark_data_smithi122_168655_object32577:head# 2024-03-20T09:26:43.768 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.768 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b9503d7:::benchmark_data_smithi122_168655_object10557:head# 2024-03-20T09:26:43.768 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.768 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b97fe28:::benchmark_data_smithi122_168655_object117529:head# 2024-03-20T09:26:43.768 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.768 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b980ed1:::benchmark_data_smithi122_168655_object79759:head# 2024-03-20T09:26:43.769 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.769 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b982ecc:::benchmark_data_smithi122_168655_object21382:head# 2024-03-20T09:26:43.769 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.769 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b985733:::benchmark_data_smithi122_168655_object118280:head# 2024-03-20T09:26:43.769 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.769 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b985b93:::benchmark_data_smithi122_168655_object76500:head# 2024-03-20T09:26:43.769 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.769 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b9a06ad:::benchmark_data_smithi122_168655_object99756:head# 2024-03-20T09:26:43.769 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.769 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b9b43ca:::benchmark_data_smithi122_168655_object89260:head# 2024-03-20T09:26:43.769 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.769 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b9c5d38:::benchmark_data_smithi122_168655_object115557:head# 2024-03-20T09:26:43.769 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.769 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b9c613f:::benchmark_data_smithi122_168655_object60341:head# 2024-03-20T09:26:43.770 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.770 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b9c69b4:::benchmark_data_smithi122_168655_object63931:head# 2024-03-20T09:26:43.770 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.770 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b9cb16e:::benchmark_data_smithi122_168655_object106401:head# 2024-03-20T09:26:43.770 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.770 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b9f3dcd:::benchmark_data_smithi122_168655_object54748:head# 2024-03-20T09:26:43.770 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.770 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b9f7fb9:::benchmark_data_smithi122_168655_object102804:head# 2024-03-20T09:26:43.770 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.770 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8b9fb8b1:::benchmark_data_smithi122_168655_object33717:head# 2024-03-20T09:26:43.770 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.770 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ba109b9:::benchmark_data_smithi122_168655_object66039:head# 2024-03-20T09:26:43.770 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.770 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ba218bb:::benchmark_data_smithi122_168655_object45596:head# 2024-03-20T09:26:43.771 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.771 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ba297bc:::benchmark_data_smithi122_168655_object11056:head# 2024-03-20T09:26:43.771 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.771 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ba36dd0:::benchmark_data_smithi122_168655_object79189:head# 2024-03-20T09:26:43.771 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.771 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ba3fc29:::benchmark_data_smithi122_168655_object45805:head# 2024-03-20T09:26:43.771 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.771 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ba41de1:::benchmark_data_smithi122_168655_object43106:head# 2024-03-20T09:26:43.771 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.771 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ba45883:::benchmark_data_smithi122_168655_object84843:head# 2024-03-20T09:26:43.771 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.771 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ba50a0f:::benchmark_data_smithi122_168655_object82141:head# 2024-03-20T09:26:43.771 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.771 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ba56311:::benchmark_data_smithi122_168655_object116219:head# 2024-03-20T09:26:43.772 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.772 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ba5a26d:::benchmark_data_smithi122_168655_object112172:head# 2024-03-20T09:26:43.772 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.772 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ba6e03e:::benchmark_data_smithi122_168655_object111010:head# 2024-03-20T09:26:43.772 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.772 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ba7839f:::benchmark_data_smithi122_168655_object16805:head# 2024-03-20T09:26:43.772 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.772 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ba844d7:::benchmark_data_smithi122_168655_object109045:head# 2024-03-20T09:26:43.772 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.772 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ba870ba:::benchmark_data_smithi122_168655_object52199:head# 2024-03-20T09:26:43.772 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.772 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ba8ce14:::benchmark_data_smithi122_168655_object87522:head# 2024-03-20T09:26:43.772 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.772 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8baa292e:::benchmark_data_smithi122_168655_object89360:head# 2024-03-20T09:26:43.772 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.773 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8baacebb:::benchmark_data_smithi122_168655_object54120:head# 2024-03-20T09:26:43.773 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.773 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8babc2a1:::benchmark_data_smithi122_168655_object10318:head# 2024-03-20T09:26:43.773 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.773 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8babc908:::benchmark_data_smithi122_168655_object119863:head# 2024-03-20T09:26:43.773 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.773 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8babcc99:::benchmark_data_smithi122_168655_object108199:head# 2024-03-20T09:26:43.773 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.773 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bac5c97:::benchmark_data_smithi122_168655_object107555:head# 2024-03-20T09:26:43.773 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.773 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bada75d:::benchmark_data_smithi122_168655_object115683:head# 2024-03-20T09:26:43.773 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.773 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bae0b80:::benchmark_data_smithi122_168655_object43803:head# 2024-03-20T09:26:43.773 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.774 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bafa2b4:::benchmark_data_smithi122_168655_object115946:head# 2024-03-20T09:26:43.774 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.774 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bafbb5a:::benchmark_data_smithi122_168655_object116723:head# 2024-03-20T09:26:43.774 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.774 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bb09e30:::benchmark_data_smithi122_168655_object54520:head# 2024-03-20T09:26:43.774 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.774 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bb1218b:::benchmark_data_smithi122_168655_object60659:head# 2024-03-20T09:26:43.774 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.774 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bb131ec:::benchmark_data_smithi122_168655_object86426:head# 2024-03-20T09:26:43.774 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.774 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bb19b26:::benchmark_data_smithi122_168655_object108220:head# 2024-03-20T09:26:43.774 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.774 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bb3149d:::benchmark_data_smithi122_168655_object105811:head# 2024-03-20T09:26:43.774 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.775 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bb3cc27:::benchmark_data_smithi122_168655_object48054:head# 2024-03-20T09:26:43.775 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.775 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bb4fb40:::benchmark_data_smithi122_168655_object23503:head# 2024-03-20T09:26:43.775 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.775 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bb5259e:::benchmark_data_smithi122_168655_object60140:head# 2024-03-20T09:26:43.775 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.775 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bb760cc:::benchmark_data_smithi122_168655_object77444:head# 2024-03-20T09:26:43.775 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.775 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bb80e35:::benchmark_data_smithi122_168655_object61170:head# 2024-03-20T09:26:43.775 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.775 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bb85c82:::benchmark_data_smithi122_168655_object32513:head# 2024-03-20T09:26:43.775 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.775 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bb875f7:::benchmark_data_smithi122_168655_object77609:head# 2024-03-20T09:26:43.775 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.776 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bb9545c:::benchmark_data_smithi122_168655_object83310:head# 2024-03-20T09:26:43.776 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.776 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bbac5b3:::benchmark_data_smithi122_168655_object79912:head# 2024-03-20T09:26:43.776 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.776 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bbacf4d:::benchmark_data_smithi122_168655_object98660:head# 2024-03-20T09:26:43.776 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.776 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bbb6fb5:::benchmark_data_smithi122_168655_object87644:head# 2024-03-20T09:26:43.776 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.776 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bbbdb48:::benchmark_data_smithi122_168655_object35400:head# 2024-03-20T09:26:43.776 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.776 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bbc18af:::benchmark_data_smithi122_168655_object42823:head# 2024-03-20T09:26:43.776 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.776 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bbc5bc9:::benchmark_data_smithi122_168655_object13445:head# 2024-03-20T09:26:43.777 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.777 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bbc5f4b:::benchmark_data_smithi122_168655_object47665:head# 2024-03-20T09:26:43.777 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.777 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bbc74e2:::benchmark_data_smithi122_168655_object84687:head# 2024-03-20T09:26:43.777 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.777 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bbd36d6:::benchmark_data_smithi122_168655_object90179:head# 2024-03-20T09:26:43.777 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.777 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bbd4738:::benchmark_data_smithi122_168655_object82074:head# 2024-03-20T09:26:43.777 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.777 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bbdaa23:::benchmark_data_smithi122_168655_object31858:head# 2024-03-20T09:26:43.777 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.777 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bbe0823:::benchmark_data_smithi122_168655_object11680:head# 2024-03-20T09:26:43.777 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.778 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bbe2877:::benchmark_data_smithi122_168655_object73238:head# 2024-03-20T09:26:43.778 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.778 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bbe68ba:::benchmark_data_smithi122_168655_object98611:head# 2024-03-20T09:26:43.778 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.778 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bbe7dc3:::benchmark_data_smithi122_168655_object74752:head# 2024-03-20T09:26:43.778 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.778 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bbe8013:::benchmark_data_smithi122_168655_object25913:head# 2024-03-20T09:26:43.778 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.778 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bbf85fe:::benchmark_data_smithi122_168655_object95084:head# 2024-03-20T09:26:43.778 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.778 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bc028ae:::benchmark_data_smithi122_168655_object72333:head# 2024-03-20T09:26:43.778 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.779 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bc03430:::benchmark_data_smithi122_168655_object116804:head# 2024-03-20T09:26:43.779 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.779 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bc06957:::benchmark_data_smithi122_168655_object26089:head# 2024-03-20T09:26:43.779 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.779 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bc2d068:::benchmark_data_smithi122_168655_object69828:head# 2024-03-20T09:26:43.779 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.779 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bc381ee:::benchmark_data_smithi122_168655_object96187:head# 2024-03-20T09:26:43.779 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.779 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bc3a1ce:::benchmark_data_smithi122_168655_object49477:head# 2024-03-20T09:26:43.779 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.779 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bc3aedc:::benchmark_data_smithi122_168655_object35733:head# 2024-03-20T09:26:43.779 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.779 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bc405ec:::benchmark_data_smithi122_168655_object14949:head# 2024-03-20T09:26:43.779 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.780 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bc4f732:::benchmark_data_smithi122_168655_object73530:head# 2024-03-20T09:26:43.780 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.780 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bc53a96:::benchmark_data_smithi122_168655_object47057:head# 2024-03-20T09:26:43.780 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.780 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bc5fffa:::benchmark_data_smithi122_168655_object107792:head# 2024-03-20T09:26:43.780 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.780 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bc605a1:::benchmark_data_smithi122_168655_object109377:head# 2024-03-20T09:26:43.780 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.780 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bc7ba2d:::benchmark_data_smithi122_168655_object98712:head# 2024-03-20T09:26:43.780 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.780 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bc898bf:::benchmark_data_smithi122_168655_object79452:head# 2024-03-20T09:26:43.780 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.780 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bc8f701:::benchmark_data_smithi122_168655_object99767:head# 2024-03-20T09:26:43.780 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.781 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bc9f7d2:::benchmark_data_smithi122_168655_object65191:head# 2024-03-20T09:26:43.781 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.781 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bcad6ee:::benchmark_data_smithi122_168655_object103474:head# 2024-03-20T09:26:43.781 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.781 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bcc26e9:::benchmark_data_smithi122_168655_object16178:head# 2024-03-20T09:26:43.781 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.781 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bcc5f8b:::benchmark_data_smithi122_168655_object28824:head# 2024-03-20T09:26:43.781 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.781 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bcc9995:::benchmark_data_smithi122_168655_object56971:head# 2024-03-20T09:26:43.781 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.781 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bce0a60:::benchmark_data_smithi122_168655_object99928:head# 2024-03-20T09:26:43.781 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.781 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bcfc52f:::benchmark_data_smithi122_168655_object116701:head# 2024-03-20T09:26:43.782 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.782 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bd096a2:::benchmark_data_smithi122_168655_object14324:head# 2024-03-20T09:26:43.782 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.782 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bd16bb8:::benchmark_data_smithi122_168655_object20394:head# 2024-03-20T09:26:43.782 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.782 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bd24f09:::benchmark_data_smithi122_168655_object43708:head# 2024-03-20T09:26:43.782 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.782 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bd2af63:::benchmark_data_smithi122_168655_object117631:head# 2024-03-20T09:26:43.782 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.782 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bd2f3c5:::benchmark_data_smithi122_168655_object107519:head# 2024-03-20T09:26:43.782 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.782 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bd349fa:::benchmark_data_smithi122_168655_object88504:head# 2024-03-20T09:26:43.782 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.782 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bd4195b:::benchmark_data_smithi122_168655_object101102:head# 2024-03-20T09:26:43.783 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.783 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bd4d256:::benchmark_data_smithi122_168655_object32667:head# 2024-03-20T09:26:43.783 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.783 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bd5a3cf:::benchmark_data_smithi122_168655_object108877:head# 2024-03-20T09:26:43.783 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.785 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bd675d8:::benchmark_data_smithi122_168655_object71604:head# 2024-03-20T09:26:43.786 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.786 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bd725d0:::benchmark_data_smithi122_168655_object92823:head# 2024-03-20T09:26:43.786 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.786 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bd7eb80:::benchmark_data_smithi122_168655_object21991:head# 2024-03-20T09:26:43.786 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.787 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bd847b4:::benchmark_data_smithi122_168655_object73408:head# 2024-03-20T09:26:43.787 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.787 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bd88c10:::benchmark_data_smithi122_168655_object21644:head# 2024-03-20T09:26:43.787 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.787 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bd895b8:::benchmark_data_smithi122_168655_object81709:head# 2024-03-20T09:26:43.787 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.787 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bd9026e:::benchmark_data_smithi122_168655_object30297:head# 2024-03-20T09:26:43.787 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.787 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bd92eb7:::benchmark_data_smithi122_168655_object100528:head# 2024-03-20T09:26:43.787 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.787 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bd9cbad:::benchmark_data_smithi122_168655_object91535:head# 2024-03-20T09:26:43.787 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.787 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bda7688:::benchmark_data_smithi122_168655_object91022:head# 2024-03-20T09:26:43.788 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.788 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bdb929a:::benchmark_data_smithi122_168655_object14968:head# 2024-03-20T09:26:43.788 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.788 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bdb9b7d:::benchmark_data_smithi122_168655_object22987:head# 2024-03-20T09:26:43.788 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.788 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bdc6048:::benchmark_data_smithi122_168655_object48216:head# 2024-03-20T09:26:43.788 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.788 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bdc83e9:::benchmark_data_smithi122_168655_object104282:head# 2024-03-20T09:26:43.788 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.788 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bdce4f1:::benchmark_data_smithi122_168655_object74867:head# 2024-03-20T09:26:43.788 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.788 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bdd8efb:::benchmark_data_smithi122_168655_object64403:head# 2024-03-20T09:26:43.788 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.789 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bde0854:::benchmark_data_smithi122_168655_object38636:head# 2024-03-20T09:26:43.789 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.789 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bdee487:::benchmark_data_smithi122_168655_object48508:head# 2024-03-20T09:26:43.789 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.789 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bdf84d1:::benchmark_data_smithi122_168655_object115001:head# 2024-03-20T09:26:43.789 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.789 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8be02296:::benchmark_data_smithi122_168655_object69870:head# 2024-03-20T09:26:43.789 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.789 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8be10bbf:::benchmark_data_smithi122_168655_object31700:head# 2024-03-20T09:26:43.789 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.789 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8be1c1dc:::benchmark_data_smithi122_168655_object93393:head# 2024-03-20T09:26:43.789 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.789 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8be200f1:::benchmark_data_smithi122_168655_object73862:head# 2024-03-20T09:26:43.789 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.790 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8be301df:::benchmark_data_smithi122_168655_object36044:head# 2024-03-20T09:26:43.790 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.790 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8be3a990:::benchmark_data_smithi122_168655_object116822:head# 2024-03-20T09:26:43.790 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.790 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8be42230:::benchmark_data_smithi122_168655_object110776:head# 2024-03-20T09:26:43.790 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.790 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8be4fd23:::benchmark_data_smithi122_168655_object115259:head# 2024-03-20T09:26:43.790 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.790 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8be578a5:::benchmark_data_smithi122_168655_object79802:head# 2024-03-20T09:26:43.790 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.790 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8be61712:::benchmark_data_smithi122_168655_object56506:head# 2024-03-20T09:26:43.790 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.790 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8be643b1:::benchmark_data_smithi122_168655_object17533:head# 2024-03-20T09:26:43.791 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.791 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8be64dc0:::benchmark_data_smithi122_168655_object100501:head# 2024-03-20T09:26:43.791 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.791 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8be816e8:::benchmark_data_smithi122_168655_object105104:head# 2024-03-20T09:26:43.791 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.791 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8be8d0e9:::benchmark_data_smithi122_168655_object42342:head# 2024-03-20T09:26:43.791 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.791 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8be9c306:::benchmark_data_smithi122_168655_object43164:head# 2024-03-20T09:26:43.791 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.791 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bea6c35:::benchmark_data_smithi122_168655_object34864:head# 2024-03-20T09:26:43.791 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.791 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8beaec41:::benchmark_data_smithi122_168655_object18641:head# 2024-03-20T09:26:43.791 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.791 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8beb2add:::benchmark_data_smithi122_168655_object22514:head# 2024-03-20T09:26:43.792 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.792 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bed0a18:::benchmark_data_smithi122_168655_object14667:head# 2024-03-20T09:26:43.792 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.792 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bed4a3f:::benchmark_data_smithi122_168655_object11385:head# 2024-03-20T09:26:43.792 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.792 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bedb7de:::benchmark_data_smithi122_168655_object37448:head# 2024-03-20T09:26:43.792 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.792 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bee8f09:::benchmark_data_smithi122_168655_object65337:head# 2024-03-20T09:26:43.792 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.792 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bef1b07:::benchmark_data_smithi122_168655_object39460:head# 2024-03-20T09:26:43.792 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.792 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bef27b9:::benchmark_data_smithi122_168655_object40344:head# 2024-03-20T09:26:43.792 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.793 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bf1034b:::benchmark_data_smithi122_168655_object27960:head# 2024-03-20T09:26:43.793 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.793 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bf146b2:::benchmark_data_smithi122_168655_object31471:head# 2024-03-20T09:26:43.793 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.793 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bf1488f:::benchmark_data_smithi122_168655_object97973:head# 2024-03-20T09:26:43.793 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.793 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bf15aea:::benchmark_data_smithi122_168655_object36090:head# 2024-03-20T09:26:43.793 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.793 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bf1e45b:::benchmark_data_smithi122_168655_object112896:head# 2024-03-20T09:26:43.793 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.793 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bf27f02:::benchmark_data_smithi122_168655_object19015:head# 2024-03-20T09:26:43.793 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.793 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bf2b085:::benchmark_data_smithi122_168655_object103009:head# 2024-03-20T09:26:43.793 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.793 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bf32ed1:::benchmark_data_smithi122_168655_object19265:head# 2024-03-20T09:26:43.793 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.794 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bf39ac0:::benchmark_data_smithi122_168655_object29019:head# 2024-03-20T09:26:43.794 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.794 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bf43726:::benchmark_data_smithi122_168655_object104806:head# 2024-03-20T09:26:43.794 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.794 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bf596aa:::benchmark_data_smithi122_168655_object30002:head# 2024-03-20T09:26:43.794 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.794 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bf5c378:::benchmark_data_smithi122_168655_object87451:head# 2024-03-20T09:26:43.794 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.794 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bf6b9a3:::benchmark_data_smithi122_168655_object68886:head# 2024-03-20T09:26:43.794 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.794 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bf80589:::benchmark_data_smithi122_168655_object62392:head# 2024-03-20T09:26:43.794 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.794 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bf92ea6:::benchmark_data_smithi122_168655_object114826:head# 2024-03-20T09:26:43.794 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.795 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bf9336a:::benchmark_data_smithi122_168655_object118287:head# 2024-03-20T09:26:43.795 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.795 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bf945d9:::benchmark_data_smithi122_168655_object51005:head# 2024-03-20T09:26:43.795 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.795 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bfa13c7:::benchmark_data_smithi122_168655_object12834:head# 2024-03-20T09:26:43.795 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.795 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bfa4ba1:::benchmark_data_smithi122_168655_object119538:head# 2024-03-20T09:26:43.795 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.795 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bfb35d9:::benchmark_data_smithi122_168655_object102576:head# 2024-03-20T09:26:43.795 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.795 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bfc781f:::benchmark_data_smithi122_168655_object28594:head# 2024-03-20T09:26:43.795 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.795 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bfcc4f0:::benchmark_data_smithi122_168655_object48029:head# 2024-03-20T09:26:43.795 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.795 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bfd6638:::benchmark_data_smithi122_168655_object92255:head# 2024-03-20T09:26:43.795 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.796 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bfe8d23:::benchmark_data_smithi122_168655_object64557:head# 2024-03-20T09:26:43.796 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.796 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bfec4ae:::benchmark_data_smithi122_168655_object82360:head# 2024-03-20T09:26:43.796 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.796 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8bff76cf:::benchmark_data_smithi122_168655_object33423:head# 2024-03-20T09:26:43.796 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.796 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c00da0b:::benchmark_data_smithi122_168655_object58348:head# 2024-03-20T09:26:43.796 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.796 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c00f804:::benchmark_data_smithi122_168655_object106364:head# 2024-03-20T09:26:43.796 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.796 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c018cb4:::benchmark_data_smithi122_168655_object92210:head# 2024-03-20T09:26:43.796 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.796 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c032046:::benchmark_data_smithi122_168655_object38385:head# 2024-03-20T09:26:43.796 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.797 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c035801:::benchmark_data_smithi122_168655_object90090:head# 2024-03-20T09:26:43.797 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.797 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c0477da:::benchmark_data_smithi122_168655_object95195:head# 2024-03-20T09:26:43.797 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.797 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c04f66c:::benchmark_data_smithi122_168655_object117745:head# 2024-03-20T09:26:43.797 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.797 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c0522cd:::benchmark_data_smithi122_168655_object114897:head# 2024-03-20T09:26:43.797 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.797 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c060a9d:::benchmark_data_smithi122_168655_object26202:head# 2024-03-20T09:26:43.797 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.797 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c074460:::benchmark_data_smithi122_168655_object95503:head# 2024-03-20T09:26:43.797 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.798 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c0777dd:::benchmark_data_smithi122_168655_object69069:head# 2024-03-20T09:26:43.798 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.798 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c079c07:::benchmark_data_smithi122_168655_object93398:head# 2024-03-20T09:26:43.798 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.798 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c09d1a8:::benchmark_data_smithi122_168655_object114529:head# 2024-03-20T09:26:43.798 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.798 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c0a5001:::benchmark_data_smithi122_168655_object116524:head# 2024-03-20T09:26:43.798 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.798 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c0b9db9:::benchmark_data_smithi122_168655_object92953:head# 2024-03-20T09:26:43.798 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.798 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c0bd41f:::benchmark_data_smithi122_168655_object117881:head# 2024-03-20T09:26:43.798 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.798 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c0be8b2:::benchmark_data_smithi122_168655_object116781:head# 2024-03-20T09:26:43.799 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.799 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c0d36b0:::benchmark_data_smithi122_168655_object61002:head# 2024-03-20T09:26:43.799 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.799 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c0d9702:::benchmark_data_smithi122_168655_object39940:head# 2024-03-20T09:26:43.799 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.799 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c0dd536:::benchmark_data_smithi122_168655_object46019:head# 2024-03-20T09:26:43.799 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.799 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c0df1f2:::benchmark_data_smithi122_168655_object32823:head# 2024-03-20T09:26:43.799 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.799 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c101af0:::benchmark_data_smithi122_168655_object51134:head# 2024-03-20T09:26:43.799 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.799 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c106a23:::benchmark_data_smithi122_168655_object112386:head# 2024-03-20T09:26:43.799 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.800 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c1088c8:::benchmark_data_smithi122_168655_object71203:head# 2024-03-20T09:26:43.800 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.800 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c108cc8:::benchmark_data_smithi122_168655_object15907:head# 2024-03-20T09:26:43.800 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.800 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c11d0fa:::benchmark_data_smithi122_168655_object92238:head# 2024-03-20T09:26:43.800 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.800 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c11d61c:::benchmark_data_smithi122_168655_object30482:head# 2024-03-20T09:26:43.800 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.800 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c125aca:::benchmark_data_smithi122_168655_object118292:head# 2024-03-20T09:26:43.800 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.800 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c128765:::benchmark_data_smithi122_168655_object15633:head# 2024-03-20T09:26:43.800 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.800 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c142e2f:::benchmark_data_smithi122_168655_object82967:head# 2024-03-20T09:26:43.801 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.801 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c168f3c:::benchmark_data_smithi122_168655_object116958:head# 2024-03-20T09:26:43.801 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.801 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c172a88:::benchmark_data_smithi122_168655_object56660:head# 2024-03-20T09:26:43.801 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.801 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c1761f5:::benchmark_data_smithi122_168655_object54053:head# 2024-03-20T09:26:43.801 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.801 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c17a827:::benchmark_data_smithi122_168655_object27329:head# 2024-03-20T09:26:43.801 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.801 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c17d03f:::benchmark_data_smithi122_168655_object66143:head# 2024-03-20T09:26:43.801 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.801 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c18858c:::benchmark_data_smithi122_168655_object46238:head# 2024-03-20T09:26:43.801 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.801 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c197507:::benchmark_data_smithi122_168655_object99645:head# 2024-03-20T09:26:43.802 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.802 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c1999a8:::benchmark_data_smithi122_168655_object23076:head# 2024-03-20T09:26:43.802 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.802 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c1a13e9:::benchmark_data_smithi122_168655_object25643:head# 2024-03-20T09:26:43.802 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.802 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c1a1d9a:::benchmark_data_smithi122_168655_object82670:head# 2024-03-20T09:26:43.802 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.802 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c1b0880:::benchmark_data_smithi122_168655_object15806:head# 2024-03-20T09:26:43.802 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.802 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c1b4f17:::benchmark_data_smithi122_168655_object109835:head# 2024-03-20T09:26:43.802 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.802 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c1bf649:::benchmark_data_smithi122_168655_object12541:head# 2024-03-20T09:26:43.802 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.803 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c1c0713:::benchmark_data_smithi122_168655_object13196:head# 2024-03-20T09:26:43.803 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.803 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c1df08b:::benchmark_data_smithi122_168655_object76745:head# 2024-03-20T09:26:43.803 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.803 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c1e402c:::benchmark_data_smithi122_168655_object62884:head# 2024-03-20T09:26:43.803 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.803 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c20238a:::benchmark_data_smithi122_168655_object22409:head# 2024-03-20T09:26:43.803 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.803 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c2242a8:::benchmark_data_smithi122_168655_object12417:head# 2024-03-20T09:26:43.803 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.803 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c22c9a0:::benchmark_data_smithi122_168655_object32157:head# 2024-03-20T09:26:43.803 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.803 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c230328:::benchmark_data_smithi122_168655_object24972:head# 2024-03-20T09:26:43.804 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.804 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c24236e:::benchmark_data_smithi122_168655_object20742:head# 2024-03-20T09:26:43.804 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.804 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c243081:::benchmark_data_smithi122_168655_object88123:head# 2024-03-20T09:26:43.804 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.804 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c2433b2:::benchmark_data_smithi122_168655_object99637:head# 2024-03-20T09:26:43.804 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.804 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c24d33e:::benchmark_data_smithi122_168655_object44040:head# 2024-03-20T09:26:43.804 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.804 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c252f6d:::benchmark_data_smithi122_168655_object11193:head# 2024-03-20T09:26:43.804 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.804 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c25ab2a:::benchmark_data_smithi122_168655_object79264:head# 2024-03-20T09:26:43.804 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.804 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c269bd1:::benchmark_data_smithi122_168655_object15769:head# 2024-03-20T09:26:43.804 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.805 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c2777c3:::benchmark_data_smithi122_168655_object34671:head# 2024-03-20T09:26:43.805 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.805 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c27b46d:::benchmark_data_smithi122_168655_object35486:head# 2024-03-20T09:26:43.805 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.805 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c294af3:::benchmark_data_smithi122_168655_object90772:head# 2024-03-20T09:26:43.805 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.805 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c2a3f54:::benchmark_data_smithi122_168655_object88531:head# 2024-03-20T09:26:43.805 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.805 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c2a9d01:::benchmark_data_smithi122_168655_object83406:head# 2024-03-20T09:26:43.805 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.805 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c2b670f:::benchmark_data_smithi122_168655_object100734:head# 2024-03-20T09:26:43.805 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.806 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c2c72de:::benchmark_data_smithi122_168655_object92447:head# 2024-03-20T09:26:43.806 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.806 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c2c8326:::benchmark_data_smithi122_168655_object96690:head# 2024-03-20T09:26:43.806 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.806 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c2e29f1:::benchmark_data_smithi122_168655_object48359:head# 2024-03-20T09:26:43.806 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.806 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c2f4804:::benchmark_data_smithi122_168655_object39214:head# 2024-03-20T09:26:43.806 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.806 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c304bd4:::benchmark_data_smithi122_168655_object71320:head# 2024-03-20T09:26:43.806 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.806 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c314de6:::benchmark_data_smithi122_168655_object65948:head# 2024-03-20T09:26:43.806 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.806 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c31e4d2:::benchmark_data_smithi122_168655_object89940:head# 2024-03-20T09:26:43.807 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.807 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c32642a:::benchmark_data_smithi122_168655_object115428:head# 2024-03-20T09:26:43.807 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.807 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c32931e:::benchmark_data_smithi122_168655_object118224:head# 2024-03-20T09:26:43.807 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.807 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c329b97:::benchmark_data_smithi122_168655_object79271:head# 2024-03-20T09:26:43.807 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.807 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c32a623:::benchmark_data_smithi122_168655_object18666:head# 2024-03-20T09:26:43.807 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.807 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c33870b:::benchmark_data_smithi122_168655_object30690:head# 2024-03-20T09:26:43.807 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.807 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c346f1e:::benchmark_data_smithi122_168655_object38520:head# 2024-03-20T09:26:43.807 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.807 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c349fed:::benchmark_data_smithi122_168655_object67829:head# 2024-03-20T09:26:43.808 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.808 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c355a95:::benchmark_data_smithi122_168655_object51086:head# 2024-03-20T09:26:43.808 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.808 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c35ef7f:::benchmark_data_smithi122_168655_object112657:head# 2024-03-20T09:26:43.808 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.808 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c3819d3:::benchmark_data_smithi122_168655_object98828:head# 2024-03-20T09:26:43.808 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.808 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c3dc17d:::benchmark_data_smithi122_168655_object87042:head# 2024-03-20T09:26:43.808 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.808 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c3e5c08:::benchmark_data_smithi122_168655_object62389:head# 2024-03-20T09:26:43.808 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.808 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c3e90b7:::benchmark_data_smithi122_168655_object73905:head# 2024-03-20T09:26:43.808 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.809 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c3eaa1d:::benchmark_data_smithi122_168655_object65526:head# 2024-03-20T09:26:43.809 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.809 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c3ec89e:::benchmark_data_smithi122_168655_object40181:head# 2024-03-20T09:26:43.809 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.809 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c3f7101:::benchmark_data_smithi122_168655_object119257:head# 2024-03-20T09:26:43.809 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.809 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c3f878a:::benchmark_data_smithi122_168655_object32230:head# 2024-03-20T09:26:43.809 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.809 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c417821:::benchmark_data_smithi122_168655_object29222:head# 2024-03-20T09:26:43.809 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.809 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c426cf1:::benchmark_data_smithi122_168655_object51905:head# 2024-03-20T09:26:43.809 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.809 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c430e2a:::benchmark_data_smithi122_168655_object119840:head# 2024-03-20T09:26:43.809 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.810 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c4596ce:::benchmark_data_smithi122_168655_object38267:head# 2024-03-20T09:26:43.810 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.810 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c46171e:::benchmark_data_smithi122_168655_object77252:head# 2024-03-20T09:26:43.810 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.810 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c4671a8:::benchmark_data_smithi122_168655_object87509:head# 2024-03-20T09:26:43.810 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.810 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c47d455:::benchmark_data_smithi122_168655_object65854:head# 2024-03-20T09:26:43.810 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.810 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c47de3c:::benchmark_data_smithi122_168655_object77187:head# 2024-03-20T09:26:43.810 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.810 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c47e108:::benchmark_data_smithi122_168655_object51606:head# 2024-03-20T09:26:43.810 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.810 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c485653:::benchmark_data_smithi122_168655_object119412:head# 2024-03-20T09:26:43.810 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.811 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c48a84d:::benchmark_data_smithi122_168655_object54196:head# 2024-03-20T09:26:43.811 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.811 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c48d342:::benchmark_data_smithi122_168655_object30191:head# 2024-03-20T09:26:43.811 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.811 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c4a2a41:::benchmark_data_smithi122_168655_object89659:head# 2024-03-20T09:26:43.811 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.811 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c4a57b1:::benchmark_data_smithi122_168655_object36479:head# 2024-03-20T09:26:43.811 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.811 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c4a928b:::benchmark_data_smithi122_168655_object115031:head# 2024-03-20T09:26:43.811 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.811 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c4b3a6c:::benchmark_data_smithi122_168655_object94493:head# 2024-03-20T09:26:43.811 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.811 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c4b4642:::benchmark_data_smithi122_168655_object11455:head# 2024-03-20T09:26:43.811 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.812 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c4b54e6:::benchmark_data_smithi122_168655_object41950:head# 2024-03-20T09:26:43.812 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.812 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c4c8328:::benchmark_data_smithi122_168655_object33401:head# 2024-03-20T09:26:43.812 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.812 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c4cd5c1:::benchmark_data_smithi122_168655_object82590:head# 2024-03-20T09:26:43.812 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.812 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c4d3d0e:::benchmark_data_smithi122_168655_object79421:head# 2024-03-20T09:26:43.812 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.812 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c4e069d:::benchmark_data_smithi122_168655_object40747:head# 2024-03-20T09:26:43.812 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.812 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c4e7e84:::benchmark_data_smithi122_168655_object106354:head# 2024-03-20T09:26:43.812 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.812 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c4eb805:::benchmark_data_smithi122_168655_object78614:head# 2024-03-20T09:26:43.813 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.813 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c4f3c40:::benchmark_data_smithi122_168655_object94554:head# 2024-03-20T09:26:43.813 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.813 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c502125:::benchmark_data_smithi122_168655_object81995:head# 2024-03-20T09:26:43.813 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.813 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c505f52:::benchmark_data_smithi122_168655_object106054:head# 2024-03-20T09:26:43.813 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.813 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c50ed0a:::benchmark_data_smithi122_168655_object117092:head# 2024-03-20T09:26:43.813 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.813 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c517ee2:::benchmark_data_smithi122_168655_object22287:head# 2024-03-20T09:26:43.813 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.814 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c51af99:::benchmark_data_smithi122_168655_object77250:head# 2024-03-20T09:26:43.814 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.814 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c54b1d3:::benchmark_data_smithi122_168655_object24782:head# 2024-03-20T09:26:43.814 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.814 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c55434c:::benchmark_data_smithi122_168655_object43878:head# 2024-03-20T09:26:43.814 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.814 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c5639e6:::benchmark_data_smithi122_168655_object101151:head# 2024-03-20T09:26:43.814 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.814 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c573bc1:::benchmark_data_smithi122_168655_object43352:head# 2024-03-20T09:26:43.814 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.814 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c57f03b:::benchmark_data_smithi122_168655_object77771:head# 2024-03-20T09:26:43.814 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.814 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c57f9e9:::benchmark_data_smithi122_168655_object107968:head# 2024-03-20T09:26:43.815 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.815 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c584ba1:::benchmark_data_smithi122_168655_object114213:head# 2024-03-20T09:26:43.815 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.815 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c589c99:::benchmark_data_smithi122_168655_object32478:head# 2024-03-20T09:26:43.815 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.815 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c590620:::benchmark_data_smithi122_168655_object118543:head# 2024-03-20T09:26:43.815 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.815 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c59d6f1:::benchmark_data_smithi122_168655_object27571:head# 2024-03-20T09:26:43.815 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.815 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c5a85b8:::benchmark_data_smithi122_168655_object12534:head# 2024-03-20T09:26:43.815 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.815 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c5b7d0b:::benchmark_data_smithi122_168655_object83787:head# 2024-03-20T09:26:43.815 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.815 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c5c60ac:::benchmark_data_smithi122_168655_object113865:head# 2024-03-20T09:26:43.816 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.816 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c5cac18:::benchmark_data_smithi122_168655_object10244:head# 2024-03-20T09:26:43.816 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.816 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c5d5101:::benchmark_data_smithi122_168655_object56543:head# 2024-03-20T09:26:43.816 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.816 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c5d5892:::benchmark_data_smithi122_168655_object119157:head# 2024-03-20T09:26:43.816 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.816 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c5dcae5:::benchmark_data_smithi122_168655_object71747:head# 2024-03-20T09:26:43.816 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.816 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c5e16a8:::benchmark_data_smithi122_168655_object69116:head# 2024-03-20T09:26:43.816 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.816 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c5ef8ed:::benchmark_data_smithi122_168655_object92371:head# 2024-03-20T09:26:43.816 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.816 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c5f87b0:::benchmark_data_smithi122_168655_object94592:head# 2024-03-20T09:26:43.817 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.817 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c6006cd:::benchmark_data_smithi122_168655_object106611:head# 2024-03-20T09:26:43.817 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.817 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c607208:::benchmark_data_smithi122_168655_object62296:head# 2024-03-20T09:26:43.817 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.817 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c60bf75:::benchmark_data_smithi122_168655_object33598:head# 2024-03-20T09:26:43.817 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.817 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c6144f2:::benchmark_data_smithi122_168655_object109423:head# 2024-03-20T09:26:43.817 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.817 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c61b72e:::benchmark_data_smithi122_168655_object26165:head# 2024-03-20T09:26:43.817 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.817 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c61da00:::benchmark_data_smithi122_168655_object74169:head# 2024-03-20T09:26:43.817 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.817 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c61fe6a:::benchmark_data_smithi122_168655_object61820:head# 2024-03-20T09:26:43.818 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.818 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c623924:::benchmark_data_smithi122_168655_object18367:head# 2024-03-20T09:26:43.818 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.818 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c62f19b:::benchmark_data_smithi122_168655_object29421:head# 2024-03-20T09:26:43.818 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.818 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c63211e:::benchmark_data_smithi122_168655_object21494:head# 2024-03-20T09:26:43.818 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.818 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c6513f6:::benchmark_data_smithi122_168655_object13019:head# 2024-03-20T09:26:43.818 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.818 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c65a7bc:::benchmark_data_smithi122_168655_object14126:head# 2024-03-20T09:26:43.818 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.818 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c673352:::benchmark_data_smithi122_168655_object89901:head# 2024-03-20T09:26:43.819 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.819 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c675893:::benchmark_data_smithi122_168655_object116533:head# 2024-03-20T09:26:43.819 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.819 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c67eb1a:::benchmark_data_smithi122_168655_object25183:head# 2024-03-20T09:26:43.819 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.819 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c68c29c:::benchmark_data_smithi122_168655_object109818:head# 2024-03-20T09:26:43.819 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.819 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c6a42dd:::benchmark_data_smithi122_168655_object116388:head# 2024-03-20T09:26:43.819 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.819 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c6ae187:::benchmark_data_smithi122_168655_object13597:head# 2024-03-20T09:26:43.819 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.819 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c6b93ca:::benchmark_data_smithi122_168655_object30518:head# 2024-03-20T09:26:43.819 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.820 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c6b9d76:::benchmark_data_smithi122_168655_object71885:head# 2024-03-20T09:26:43.820 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.820 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c6d3e5e:::benchmark_data_smithi122_168655_object81997:head# 2024-03-20T09:26:43.820 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.820 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c6eb18f:::benchmark_data_smithi122_168655_object50275:head# 2024-03-20T09:26:43.820 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.820 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c6ee293:::benchmark_data_smithi122_168655_object64460:head# 2024-03-20T09:26:43.820 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.820 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c700f1e:::benchmark_data_smithi122_168655_object73827:head# 2024-03-20T09:26:43.820 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.820 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c7014ab:::benchmark_data_smithi122_168655_object21270:head# 2024-03-20T09:26:43.820 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.820 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c703cf3:::benchmark_data_smithi122_168655_object35596:head# 2024-03-20T09:26:43.820 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.821 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c7126fb:::benchmark_data_smithi122_168655_object92678:head# 2024-03-20T09:26:43.821 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.821 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c7134e2:::benchmark_data_smithi122_168655_object115637:head# 2024-03-20T09:26:43.821 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.821 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c724aa1:::benchmark_data_smithi122_168655_object24423:head# 2024-03-20T09:26:43.821 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.821 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c72bb98:::benchmark_data_smithi122_168655_object108026:head# 2024-03-20T09:26:43.821 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.821 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c733970:::benchmark_data_smithi122_168655_object109802:head# 2024-03-20T09:26:43.821 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.821 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c73f296:::benchmark_data_smithi122_168655_object81338:head# 2024-03-20T09:26:43.821 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.821 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c741aeb:::benchmark_data_smithi122_168655_object22333:head# 2024-03-20T09:26:43.821 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.821 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c742568:::benchmark_data_smithi122_168655_object106202:head# 2024-03-20T09:26:43.822 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.822 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c744027:::benchmark_data_smithi122_168655_object30054:head# 2024-03-20T09:26:43.822 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.822 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c7450f8:::benchmark_data_smithi122_168655_object11904:head# 2024-03-20T09:26:43.822 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.822 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c747284:::benchmark_data_smithi122_168655_object36305:head# 2024-03-20T09:26:43.822 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.822 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c74cfbf:::benchmark_data_smithi122_168655_object60138:head# 2024-03-20T09:26:43.822 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.822 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c758d2c:::benchmark_data_smithi122_168655_object62038:head# 2024-03-20T09:26:43.822 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.822 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c76cda5:::benchmark_data_smithi122_168655_object76702:head# 2024-03-20T09:26:43.822 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:43.823 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:43 smithi122 ceph-mon[73495]: 1.f scrub starts 2024-03-20T09:26:43.823 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:43 smithi122 ceph-mon[73495]: 1.f scrub ok 2024-03-20T09:26:43.823 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:43 smithi122 ceph-mon[73495]: pgmap v1872: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 38 active+clean; 6.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 211 op/s; 31239/311154 objects degraded (10.040%); 30468/311154 objects misplaced (9.792%); 25 MiB/s, 447 objects/s recovering 2024-03-20T09:26:43.823 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:43 smithi122 ceph-mon[73495]: osdmap e770: 12 total, 11 up, 12 in 2024-03-20T09:26:43.823 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:43 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 31239/311154 objects degraded (10.040%), 8 pgs degraded (PG_DEGRADED) 2024-03-20T09:26:43.824 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:43 smithi026 ceph-mon[68309]: 1.f scrub starts 2024-03-20T09:26:43.824 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:43 smithi026 ceph-mon[68309]: 1.f scrub ok 2024-03-20T09:26:43.824 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:43 smithi026 ceph-mon[68309]: pgmap v1872: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 38 active+clean; 6.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 211 op/s; 31239/311154 objects degraded (10.040%); 30468/311154 objects misplaced (9.792%); 25 MiB/s, 447 objects/s recovering 2024-03-20T09:26:43.825 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:43 smithi026 ceph-mon[68309]: osdmap e770: 12 total, 11 up, 12 in 2024-03-20T09:26:43.825 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:43 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 31239/311154 objects degraded (10.040%), 8 pgs degraded (PG_DEGRADED) 2024-03-20T09:26:44.044 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c76f11e:::benchmark_data_smithi122_168655_object22938:head# 2024-03-20T09:26:44.045 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.045 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c77f3db:::benchmark_data_smithi122_168655_object82987:head# 2024-03-20T09:26:44.045 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.045 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c79a461:::benchmark_data_smithi122_168655_object11706:head# 2024-03-20T09:26:44.045 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.045 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c79ea48:::benchmark_data_smithi122_168655_object105622:head# 2024-03-20T09:26:44.045 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.045 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c7ad6c0:::benchmark_data_smithi122_168655_object85294:head# 2024-03-20T09:26:44.045 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.045 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c7b191b:::benchmark_data_smithi122_168655_object66388:head# 2024-03-20T09:26:44.045 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.045 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c7c33a0:::benchmark_data_smithi122_168655_object117420:head# 2024-03-20T09:26:44.045 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.045 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c7c4da7:::benchmark_data_smithi122_168655_object93167:head# 2024-03-20T09:26:44.046 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.046 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c7c5a0b:::benchmark_data_smithi122_168655_object31760:head# 2024-03-20T09:26:44.046 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.046 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c7cd173:::benchmark_data_smithi122_168655_object21436:head# 2024-03-20T09:26:44.046 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.046 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c7d7107:::benchmark_data_smithi122_168655_object38264:head# 2024-03-20T09:26:44.046 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.046 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c7d8216:::benchmark_data_smithi122_168655_object100810:head# 2024-03-20T09:26:44.046 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.046 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c7ebcc5:::benchmark_data_smithi122_168655_object17770:head# 2024-03-20T09:26:44.046 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.046 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c7ec004:::benchmark_data_smithi122_168655_object29608:head# 2024-03-20T09:26:44.046 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.047 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c802d43:::benchmark_data_smithi122_168655_object108016:head# 2024-03-20T09:26:44.047 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.047 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c8163cc:::benchmark_data_smithi122_168655_object74852:head# 2024-03-20T09:26:44.047 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.047 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c816bda:::benchmark_data_smithi122_168655_object101314:head# 2024-03-20T09:26:44.047 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.047 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c822b8a:::benchmark_data_smithi122_168655_object14567:head# 2024-03-20T09:26:44.047 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.047 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c832453:::benchmark_data_smithi122_168655_object58141:head# 2024-03-20T09:26:44.047 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.047 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c8337b0:::benchmark_data_smithi122_168655_object63693:head# 2024-03-20T09:26:44.047 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.047 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c83f1b1:::benchmark_data_smithi122_168655_object92105:head# 2024-03-20T09:26:44.048 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.048 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c844f9d:::benchmark_data_smithi122_168655_object43651:head# 2024-03-20T09:26:44.048 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.048 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c84c2d0:::benchmark_data_smithi122_168655_object34576:head# 2024-03-20T09:26:44.048 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.048 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c867174:::benchmark_data_smithi122_168655_object118473:head# 2024-03-20T09:26:44.048 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.048 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c86f7d1:::benchmark_data_smithi122_168655_object24842:head# 2024-03-20T09:26:44.048 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.048 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c89b88b:::benchmark_data_smithi122_168655_object119926:head# 2024-03-20T09:26:44.048 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.048 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c8a19c7:::benchmark_data_smithi122_168655_object87135:head# 2024-03-20T09:26:44.048 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.049 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c8aabff:::benchmark_data_smithi122_168655_object86993:head# 2024-03-20T09:26:44.049 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.049 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c8af9c6:::benchmark_data_smithi122_168655_object26666:head# 2024-03-20T09:26:44.049 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.049 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c8b3715:::benchmark_data_smithi122_168655_object102448:head# 2024-03-20T09:26:44.049 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.049 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c8c007b:::benchmark_data_smithi122_168655_object25050:head# 2024-03-20T09:26:44.049 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.049 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c8c3cac:::benchmark_data_smithi122_168655_object41358:head# 2024-03-20T09:26:44.049 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.049 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c8d98c4:::benchmark_data_smithi122_168655_object92197:head# 2024-03-20T09:26:44.049 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.049 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c8dcebb:::benchmark_data_smithi122_168655_object93681:head# 2024-03-20T09:26:44.050 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.050 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c8ec6b3:::benchmark_data_smithi122_168655_object75953:head# 2024-03-20T09:26:44.050 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.050 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c8ee51a:::benchmark_data_smithi122_168655_object90603:head# 2024-03-20T09:26:44.050 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.050 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c8f1736:::benchmark_data_smithi122_168655_object119879:head# 2024-03-20T09:26:44.050 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.050 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c9074fc:::benchmark_data_smithi122_168655_object34488:head# 2024-03-20T09:26:44.050 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.050 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c90b349:::benchmark_data_smithi122_168655_object47919:head# 2024-03-20T09:26:44.050 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.050 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c90c2fa:::benchmark_data_smithi122_168655_object12118:head# 2024-03-20T09:26:44.050 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.051 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c91abe6:::benchmark_data_smithi122_168655_object106846:head# 2024-03-20T09:26:44.051 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.051 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c922db5:::benchmark_data_smithi122_168655_object50456:head# 2024-03-20T09:26:44.051 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.051 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c928bce:::benchmark_data_smithi122_168655_object93390:head# 2024-03-20T09:26:44.051 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.051 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c9342ba:::benchmark_data_smithi122_168655_object111890:head# 2024-03-20T09:26:44.051 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.051 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c93ac23:::benchmark_data_smithi122_168655_object93385:head# 2024-03-20T09:26:44.051 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.051 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c93e2ee:::benchmark_data_smithi122_168655_object103777:head# 2024-03-20T09:26:44.051 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.051 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c94655b:::benchmark_data_smithi122_168655_object114227:head# 2024-03-20T09:26:44.051 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.052 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c949c00:::benchmark_data_smithi122_168655_object32217:head# 2024-03-20T09:26:44.052 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.052 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c963847:::benchmark_data_smithi122_168655_object46654:head# 2024-03-20T09:26:44.052 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.052 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c967272:::benchmark_data_smithi122_168655_object46979:head# 2024-03-20T09:26:44.052 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.052 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c96729e:::benchmark_data_smithi122_168655_object98597:head# 2024-03-20T09:26:44.052 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.052 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c970a45:::benchmark_data_smithi122_168655_object92237:head# 2024-03-20T09:26:44.052 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.052 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c9726af:::benchmark_data_smithi122_168655_object56841:head# 2024-03-20T09:26:44.052 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.052 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c983b3a:::benchmark_data_smithi122_168655_object82286:head# 2024-03-20T09:26:44.053 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.053 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c9959ae:::benchmark_data_smithi122_168655_object29209:head# 2024-03-20T09:26:44.053 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.053 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c99c251:::benchmark_data_smithi122_168655_object78295:head# 2024-03-20T09:26:44.053 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.053 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c9ae215:::benchmark_data_smithi122_168655_object74685:head# 2024-03-20T09:26:44.053 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.053 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c9b8efc:::benchmark_data_smithi122_168655_object17779:head# 2024-03-20T09:26:44.053 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.053 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c9c039f:::benchmark_data_smithi122_168655_object74773:head# 2024-03-20T09:26:44.053 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.053 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c9c7eaf:::benchmark_data_smithi122_168655_object39219:head# 2024-03-20T09:26:44.053 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.053 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c9d28fd:::benchmark_data_smithi122_168655_object93824:head# 2024-03-20T09:26:44.053 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.054 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c9d2c3a:::benchmark_data_smithi122_168655_object25932:head# 2024-03-20T09:26:44.054 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.054 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c9e0abe:::benchmark_data_smithi122_168655_object109374:head# 2024-03-20T09:26:44.054 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.054 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c9e964d:::benchmark_data_smithi122_168655_object116122:head# 2024-03-20T09:26:44.054 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.054 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c9f365a:::benchmark_data_smithi122_168655_object119018:head# 2024-03-20T09:26:44.054 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.054 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c9fd935:::benchmark_data_smithi122_168655_object54787:head# 2024-03-20T09:26:44.054 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.054 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8c9ffe5a:::benchmark_data_smithi122_168655_object23264:head# 2024-03-20T09:26:44.054 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.054 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ca133a4:::benchmark_data_smithi122_168655_object48771:head# 2024-03-20T09:26:44.055 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.055 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ca2c1ff:::benchmark_data_smithi122_168655_object12193:head# 2024-03-20T09:26:44.055 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.055 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ca2f613:::benchmark_data_smithi122_168655_object28521:head# 2024-03-20T09:26:44.055 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.055 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ca355f8:::benchmark_data_smithi122_168655_object108492:head# 2024-03-20T09:26:44.055 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.055 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ca3e590:::benchmark_data_smithi122_168655_object107698:head# 2024-03-20T09:26:44.055 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.055 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ca49537:::benchmark_data_smithi122_168655_object53140:head# 2024-03-20T09:26:44.055 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.055 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ca528fb:::benchmark_data_smithi122_168655_object16381:head# 2024-03-20T09:26:44.055 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.055 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ca5df76:::benchmark_data_smithi122_168655_object99396:head# 2024-03-20T09:26:44.055 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.056 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ca70ddf:::benchmark_data_smithi122_168655_object11943:head# 2024-03-20T09:26:44.056 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.056 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ca71a5d:::benchmark_data_smithi122_168655_object109505:head# 2024-03-20T09:26:44.056 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.056 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ca72d7d:::benchmark_data_smithi122_168655_object62650:head# 2024-03-20T09:26:44.056 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.056 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ca75e46:::benchmark_data_smithi122_168655_object85667:head# 2024-03-20T09:26:44.056 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.056 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ca89fde:::benchmark_data_smithi122_168655_object51788:head# 2024-03-20T09:26:44.056 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.056 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ca9b5c7:::benchmark_data_smithi122_168655_object88871:head# 2024-03-20T09:26:44.056 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.056 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8caa2fc9:::benchmark_data_smithi122_168655_object64717:head# 2024-03-20T09:26:44.056 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.057 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8caa450d:::benchmark_data_smithi122_168655_object119124:head# 2024-03-20T09:26:44.057 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.057 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8caa61c5:::benchmark_data_smithi122_168655_object75617:head# 2024-03-20T09:26:44.057 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.057 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cac19b1:::benchmark_data_smithi122_168655_object110984:head# 2024-03-20T09:26:44.057 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.057 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cac1b5d:::benchmark_data_smithi122_168655_object56109:head# 2024-03-20T09:26:44.057 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.057 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cac5346:::benchmark_data_smithi122_168655_object105316:head# 2024-03-20T09:26:44.057 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.057 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cac9366:::benchmark_data_smithi122_168655_object34268:head# 2024-03-20T09:26:44.057 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.057 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cacf04b:::benchmark_data_smithi122_168655_object117779:head# 2024-03-20T09:26:44.057 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.057 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cad2635:::benchmark_data_smithi122_168655_object89686:head# 2024-03-20T09:26:44.058 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.058 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cadae8b:::benchmark_data_smithi122_168655_object116376:head# 2024-03-20T09:26:44.058 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.058 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cae0b9b:::benchmark_data_smithi122_168655_object39445:head# 2024-03-20T09:26:44.058 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.058 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8caecc46:::benchmark_data_smithi122_168655_object57194:head# 2024-03-20T09:26:44.058 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.058 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8caf1e6a:::benchmark_data_smithi122_168655_object27593:head# 2024-03-20T09:26:44.058 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.058 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cafd825:::benchmark_data_smithi122_168655_object63997:head# 2024-03-20T09:26:44.058 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.059 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cb06df0:::benchmark_data_smithi122_168655_object94751:head# 2024-03-20T09:26:44.059 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.059 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cb089c6:::benchmark_data_smithi122_168655_object49797:head# 2024-03-20T09:26:44.059 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.059 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cb11d8d:::benchmark_data_smithi122_168655_object118609:head# 2024-03-20T09:26:44.059 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.059 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cb1e5e5:::benchmark_data_smithi122_168655_object90927:head# 2024-03-20T09:26:44.059 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.059 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cb20d43:::benchmark_data_smithi122_168655_object99825:head# 2024-03-20T09:26:44.059 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.059 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cb21853:::benchmark_data_smithi122_168655_object33270:head# 2024-03-20T09:26:44.059 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.059 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cb288b1:::benchmark_data_smithi122_168655_object24906:head# 2024-03-20T09:26:44.060 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.060 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cb2e951:::benchmark_data_smithi122_168655_object94184:head# 2024-03-20T09:26:44.060 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.060 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cb3d8ad:::benchmark_data_smithi122_168655_object43659:head# 2024-03-20T09:26:44.060 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.060 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cb40243:::benchmark_data_smithi122_168655_object88783:head# 2024-03-20T09:26:44.060 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.060 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cb421e9:::benchmark_data_smithi122_168655_object113743:head# 2024-03-20T09:26:44.060 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.060 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cb44e40:::benchmark_data_smithi122_168655_object40033:head# 2024-03-20T09:26:44.060 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.060 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cb6317e:::benchmark_data_smithi122_168655_object99845:head# 2024-03-20T09:26:44.060 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.061 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cb66926:::benchmark_data_smithi122_168655_object72428:head# 2024-03-20T09:26:44.061 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.061 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cb6f5b1:::benchmark_data_smithi122_168655_object64012:head# 2024-03-20T09:26:44.061 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.061 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cb72d66:::benchmark_data_smithi122_168655_object91398:head# 2024-03-20T09:26:44.061 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.061 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cb78691:::benchmark_data_smithi122_168655_object70777:head# 2024-03-20T09:26:44.061 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.061 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cb806c2:::benchmark_data_smithi122_168655_object44298:head# 2024-03-20T09:26:44.061 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.061 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cb862f7:::benchmark_data_smithi122_168655_object111049:head# 2024-03-20T09:26:44.061 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.061 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cb8e7f0:::benchmark_data_smithi122_168655_object51931:head# 2024-03-20T09:26:44.061 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.062 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cb913f5:::benchmark_data_smithi122_168655_object100111:head# 2024-03-20T09:26:44.062 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.062 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cb9398b:::benchmark_data_smithi122_168655_object62258:head# 2024-03-20T09:26:44.062 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.062 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cb961d0:::benchmark_data_smithi122_168655_object22755:head# 2024-03-20T09:26:44.062 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.062 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cb967ce:::benchmark_data_smithi122_168655_object34147:head# 2024-03-20T09:26:44.062 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.062 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cb9b1f6:::benchmark_data_smithi122_168655_object89270:head# 2024-03-20T09:26:44.062 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.062 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cba38a7:::benchmark_data_smithi122_168655_object79023:head# 2024-03-20T09:26:44.062 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.063 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cba565f:::benchmark_data_smithi122_168655_object61317:head# 2024-03-20T09:26:44.063 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.063 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cbafe85:::benchmark_data_smithi122_168655_object94266:head# 2024-03-20T09:26:44.063 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.063 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cbbeaef:::benchmark_data_smithi122_168655_object65199:head# 2024-03-20T09:26:44.063 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.063 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cbc2a7d:::benchmark_data_smithi122_168655_object38088:head# 2024-03-20T09:26:44.063 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.063 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cbd3189:::benchmark_data_smithi122_168655_object37622:head# 2024-03-20T09:26:44.063 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.063 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cbf28fd:::benchmark_data_smithi122_168655_object74785:head# 2024-03-20T09:26:44.063 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.063 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cbfbbc9:::benchmark_data_smithi122_168655_object19153:head# 2024-03-20T09:26:44.064 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.064 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cc08dae:::benchmark_data_smithi122_168655_object41357:head# 2024-03-20T09:26:44.064 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.064 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cc2567d:::benchmark_data_smithi122_168655_object12406:head# 2024-03-20T09:26:44.064 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.064 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cc2b575:::benchmark_data_smithi122_168655_object101011:head# 2024-03-20T09:26:44.064 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.064 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cc34e3f:::benchmark_data_smithi122_168655_object13343:head# 2024-03-20T09:26:44.064 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.064 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cc356c5:::benchmark_data_smithi122_168655_object42820:head# 2024-03-20T09:26:44.064 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.064 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cc450cf:::benchmark_data_smithi122_168655_object22710:head# 2024-03-20T09:26:44.064 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.065 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cc458ac:::benchmark_data_smithi122_168655_object92334:head# 2024-03-20T09:26:44.065 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.065 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cc4cb0e:::benchmark_data_smithi122_168655_object110182:head# 2024-03-20T09:26:44.065 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.065 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cc5701d:::benchmark_data_smithi122_168655_object70354:head# 2024-03-20T09:26:44.065 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.065 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cc5aea9:::benchmark_data_smithi122_168655_object13288:head# 2024-03-20T09:26:44.065 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.065 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cc67d00:::benchmark_data_smithi122_168655_object20577:head# 2024-03-20T09:26:44.065 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.065 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cc6cb37:::benchmark_data_smithi122_168655_object84281:head# 2024-03-20T09:26:44.065 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.065 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cc8c163:::benchmark_data_smithi122_168655_object76647:head# 2024-03-20T09:26:44.066 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.066 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cc936f3:::benchmark_data_smithi122_168655_object108759:head# 2024-03-20T09:26:44.066 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.066 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cc9eacf:::benchmark_data_smithi122_168655_object23408:head# 2024-03-20T09:26:44.066 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.066 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ccbd68e:::benchmark_data_smithi122_168655_object20097:head# 2024-03-20T09:26:44.066 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.066 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ccd0d1c:::benchmark_data_smithi122_168655_object99839:head# 2024-03-20T09:26:44.066 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.066 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ccdb98c:::benchmark_data_smithi122_168655_object92130:head# 2024-03-20T09:26:44.066 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.066 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ccdba71:::benchmark_data_smithi122_168655_object16341:head# 2024-03-20T09:26:44.066 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.067 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cced362:::benchmark_data_smithi122_168655_object91978:head# 2024-03-20T09:26:44.067 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.067 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ccfafd5:::benchmark_data_smithi122_168655_object47230:head# 2024-03-20T09:26:44.067 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.067 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ccfdf0a:::benchmark_data_smithi122_168655_object50217:head# 2024-03-20T09:26:44.067 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.067 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cd04d0f:::benchmark_data_smithi122_168655_object18865:head# 2024-03-20T09:26:44.067 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.067 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cd10891:::benchmark_data_smithi122_168655_object36620:head# 2024-03-20T09:26:44.067 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.067 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cd175ec:::benchmark_data_smithi122_168655_object115190:head# 2024-03-20T09:26:44.067 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.067 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cd17d27:::benchmark_data_smithi122_168655_object68549:head# 2024-03-20T09:26:44.068 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.068 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cd19909:::benchmark_data_smithi122_168655_object15941:head# 2024-03-20T09:26:44.068 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.068 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cd1c59c:::benchmark_data_smithi122_168655_object31840:head# 2024-03-20T09:26:44.068 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.068 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cd2633a:::benchmark_data_smithi122_168655_object13154:head# 2024-03-20T09:26:44.068 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.068 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cd2b274:::benchmark_data_smithi122_168655_object119865:head# 2024-03-20T09:26:44.068 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.068 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cd2bae3:::benchmark_data_smithi122_168655_object85437:head# 2024-03-20T09:26:44.068 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.068 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cd2dc6d:::benchmark_data_smithi122_168655_object21976:head# 2024-03-20T09:26:44.068 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.068 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cd32fc0:::benchmark_data_smithi122_168655_object66634:head# 2024-03-20T09:26:44.069 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.069 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cd3d0ea:::benchmark_data_smithi122_168655_object31385:head# 2024-03-20T09:26:44.069 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.069 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cd53125:::benchmark_data_smithi122_168655_object29242:head# 2024-03-20T09:26:44.069 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.069 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cd5ada6:::benchmark_data_smithi122_168655_object67231:head# 2024-03-20T09:26:44.069 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.069 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cd5f4c5:::benchmark_data_smithi122_168655_object23126:head# 2024-03-20T09:26:44.069 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.069 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cd68750:::benchmark_data_smithi122_168655_object81238:head# 2024-03-20T09:26:44.069 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.069 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cd6e07b:::benchmark_data_smithi122_168655_object65495:head# 2024-03-20T09:26:44.069 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.069 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cd75128:::benchmark_data_smithi122_168655_object44988:head# 2024-03-20T09:26:44.070 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.070 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cd8b6a9:::benchmark_data_smithi122_168655_object43497:head# 2024-03-20T09:26:44.070 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.070 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cd94604:::benchmark_data_smithi122_168655_object57268:head# 2024-03-20T09:26:44.070 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.070 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cd964bf:::benchmark_data_smithi122_168655_object10876:head# 2024-03-20T09:26:44.070 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.070 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cd9c9bf:::benchmark_data_smithi122_168655_object65045:head# 2024-03-20T09:26:44.070 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.070 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cda8110:::benchmark_data_smithi122_168655_object82905:head# 2024-03-20T09:26:44.070 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.070 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cdb4af9:::benchmark_data_smithi122_168655_object33027:head# 2024-03-20T09:26:44.070 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.070 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cdc6a4d:::benchmark_data_smithi122_168655_object19866:head# 2024-03-20T09:26:44.071 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.071 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cdd09d2:::benchmark_data_smithi122_168655_object102710:head# 2024-03-20T09:26:44.071 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.071 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cddcc29:::benchmark_data_smithi122_168655_object14908:head# 2024-03-20T09:26:44.071 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.071 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cdde164:::benchmark_data_smithi122_168655_object38082:head# 2024-03-20T09:26:44.071 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.071 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cde4164:::benchmark_data_smithi122_168655_object36369:head# 2024-03-20T09:26:44.071 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.071 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cdf7d79:::benchmark_data_smithi122_168655_object18566:head# 2024-03-20T09:26:44.071 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.071 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ce070e2:::benchmark_data_smithi122_168655_object116729:head# 2024-03-20T09:26:44.071 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.071 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ce2f3d6:::benchmark_data_smithi122_168655_object42038:head# 2024-03-20T09:26:44.072 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.072 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ce3f9a1:::benchmark_data_smithi122_168655_object99195:head# 2024-03-20T09:26:44.072 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.072 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ce4df3c:::benchmark_data_smithi122_168655_object28622:head# 2024-03-20T09:26:44.072 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.072 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ce53bae:::benchmark_data_smithi122_168655_object77456:head# 2024-03-20T09:26:44.072 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.072 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ce5e0b8:::benchmark_data_smithi122_168655_object39985:head# 2024-03-20T09:26:44.072 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.072 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ce68386:::benchmark_data_smithi122_168655_object39184:head# 2024-03-20T09:26:44.072 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.072 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ce6c692:::benchmark_data_smithi122_168655_object50085:head# 2024-03-20T09:26:44.072 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.072 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ce84e1f:::benchmark_data_smithi122_168655_object111110:head# 2024-03-20T09:26:44.073 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.073 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ce86092:::benchmark_data_smithi122_168655_object56163:head# 2024-03-20T09:26:44.073 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.073 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ce94264:::benchmark_data_smithi122_168655_object13363:head# 2024-03-20T09:26:44.073 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.073 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ce9cc23:::benchmark_data_smithi122_168655_object49908:head# 2024-03-20T09:26:44.073 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.073 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cea1cf7:::benchmark_data_smithi122_168655_object21952:head# 2024-03-20T09:26:44.073 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.073 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cea9f4d:::benchmark_data_smithi122_168655_object76826:head# 2024-03-20T09:26:44.073 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.073 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cead519:::benchmark_data_smithi122_168655_object115269:head# 2024-03-20T09:26:44.073 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.073 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cead676:::benchmark_data_smithi122_168655_object68990:head# 2024-03-20T09:26:44.074 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.074 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ceba882:::benchmark_data_smithi122_168655_object63873:head# 2024-03-20T09:26:44.074 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.074 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cec0a3d:::benchmark_data_smithi122_168655_object12642:head# 2024-03-20T09:26:44.074 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.074 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cec91ba:::benchmark_data_smithi122_168655_object83155:head# 2024-03-20T09:26:44.074 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.074 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cedb6ec:::benchmark_data_smithi122_168655_object55804:head# 2024-03-20T09:26:44.074 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.074 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cee8f01:::benchmark_data_smithi122_168655_object102352:head# 2024-03-20T09:26:44.074 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.074 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ceefb3f:::benchmark_data_smithi122_168655_object108318:head# 2024-03-20T09:26:44.074 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.075 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cef531a:::benchmark_data_smithi122_168655_object59937:head# 2024-03-20T09:26:44.075 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.075 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cf0081a:::benchmark_data_smithi122_168655_object47093:head# 2024-03-20T09:26:44.075 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.075 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cf091ca:::benchmark_data_smithi122_168655_object28945:head# 2024-03-20T09:26:44.075 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.075 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cf3a915:::benchmark_data_smithi122_168655_object44828:head# 2024-03-20T09:26:44.075 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.075 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cf3cb5e:::benchmark_data_smithi122_168655_object115057:head# 2024-03-20T09:26:44.075 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.075 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cf5145b:::benchmark_data_smithi122_168655_object83715:head# 2024-03-20T09:26:44.075 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.075 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cf59ab7:::benchmark_data_smithi122_168655_object65283:head# 2024-03-20T09:26:44.075 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.075 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cf59d76:::benchmark_data_smithi122_168655_object51557:head# 2024-03-20T09:26:44.076 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.076 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cf6711c:::benchmark_data_smithi122_168655_object27425:head# 2024-03-20T09:26:44.076 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.076 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cf67314:::benchmark_data_smithi122_168655_object35629:head# 2024-03-20T09:26:44.076 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.076 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cf69b42:::benchmark_data_smithi122_168655_object14220:head# 2024-03-20T09:26:44.076 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.076 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cf7159f:::benchmark_data_smithi122_168655_object25724:head# 2024-03-20T09:26:44.076 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.076 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cf7338a:::benchmark_data_smithi122_168655_object33548:head# 2024-03-20T09:26:44.076 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.076 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cf73cf7:::benchmark_data_smithi122_168655_object40513:head# 2024-03-20T09:26:44.077 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.077 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cf7433d:::benchmark_data_smithi122_168655_object119792:head# 2024-03-20T09:26:44.077 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.077 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cf75098:::benchmark_data_smithi122_168655_object40647:head# 2024-03-20T09:26:44.077 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.077 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cf83233:::benchmark_data_smithi122_168655_object58389:head# 2024-03-20T09:26:44.077 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.077 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cf86fff:::benchmark_data_smithi122_168655_object108433:head# 2024-03-20T09:26:44.077 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.077 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cf8fd65:::benchmark_data_smithi122_168655_object112961:head# 2024-03-20T09:26:44.077 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.077 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cfadeb2:::benchmark_data_smithi122_168655_object113471:head# 2024-03-20T09:26:44.077 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.077 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cfb502d:::benchmark_data_smithi122_168655_object82119:head# 2024-03-20T09:26:44.078 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.078 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cfba573:::benchmark_data_smithi122_168655_object52713:head# 2024-03-20T09:26:44.078 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.078 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cfba6f8:::benchmark_data_smithi122_168655_object38236:head# 2024-03-20T09:26:44.078 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.078 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cfbdaa6:::benchmark_data_smithi122_168655_object48593:head# 2024-03-20T09:26:44.078 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.078 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cfc1ca1:::benchmark_data_smithi122_168655_object17097:head# 2024-03-20T09:26:44.078 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.078 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cfd5597:::benchmark_data_smithi122_168655_object73034:head# 2024-03-20T09:26:44.078 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.078 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cfe8ca9:::benchmark_data_smithi122_168655_object60488:head# 2024-03-20T09:26:44.078 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.079 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cfeb728:::benchmark_data_smithi122_168655_object99624:head# 2024-03-20T09:26:44.079 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.079 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cff2d65:::benchmark_data_smithi122_168655_object69267:head# 2024-03-20T09:26:44.079 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.079 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cff4270:::benchmark_data_smithi122_168655_object91385:head# 2024-03-20T09:26:44.079 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.079 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8cffaa98:::benchmark_data_smithi122_168655_object57798:head# 2024-03-20T09:26:44.079 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.079 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d00e726:::benchmark_data_smithi122_168655_object101797:head# 2024-03-20T09:26:44.079 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.079 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d01a20b:::benchmark_data_smithi122_168655_object118644:head# 2024-03-20T09:26:44.079 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.079 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d04ada5:::benchmark_data_smithi122_168655_object30169:head# 2024-03-20T09:26:44.079 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.080 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d0562d1:::benchmark_data_smithi122_168655_object85901:head# 2024-03-20T09:26:44.080 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.080 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d057f5f:::benchmark_data_smithi122_168655_object17402:head# 2024-03-20T09:26:44.080 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.080 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d0597f4:::benchmark_data_smithi122_168655_object87129:head# 2024-03-20T09:26:44.080 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.080 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d077078:::benchmark_data_smithi122_168655_object18332:head# 2024-03-20T09:26:44.080 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.080 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d07a7be:::benchmark_data_smithi122_168655_object23535:head# 2024-03-20T09:26:44.080 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.080 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d0838a3:::benchmark_data_smithi122_168655_object63330:head# 2024-03-20T09:26:44.080 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.080 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d0921df:::benchmark_data_smithi122_168655_object109213:head# 2024-03-20T09:26:44.081 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.081 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d09d55d:::benchmark_data_smithi122_168655_object85418:head# 2024-03-20T09:26:44.081 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.081 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d09dbd4:::benchmark_data_smithi122_168655_object101669:head# 2024-03-20T09:26:44.081 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.081 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d0a7fd3:::benchmark_data_smithi122_168655_object73872:head# 2024-03-20T09:26:44.081 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.081 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d0acb39:::benchmark_data_smithi122_168655_object12499:head# 2024-03-20T09:26:44.081 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.081 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d0afe4b:::benchmark_data_smithi122_168655_object49744:head# 2024-03-20T09:26:44.081 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.081 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d0f1c0d:::benchmark_data_smithi122_168655_object81210:head# 2024-03-20T09:26:44.081 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.082 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d0f2ea1:::benchmark_data_smithi122_168655_object110879:head# 2024-03-20T09:26:44.082 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.082 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d0f393c:::benchmark_data_smithi122_168655_object111218:head# 2024-03-20T09:26:44.082 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.082 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d107193:::benchmark_data_smithi122_168655_object114658:head# 2024-03-20T09:26:44.082 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.082 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d10d51c:::benchmark_data_smithi122_168655_object82006:head# 2024-03-20T09:26:44.082 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.082 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d1375bd:::benchmark_data_smithi122_168655_object115751:head# 2024-03-20T09:26:44.082 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.082 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d13dbd4:::benchmark_data_smithi122_168655_object82049:head# 2024-03-20T09:26:44.082 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.082 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d1431bd:::benchmark_data_smithi122_168655_object99971:head# 2024-03-20T09:26:44.083 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.083 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d1437a8:::benchmark_data_smithi122_168655_object66128:head# 2024-03-20T09:26:44.083 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.083 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d145e5f:::benchmark_data_smithi122_168655_object53101:head# 2024-03-20T09:26:44.083 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.083 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d1471dc:::benchmark_data_smithi122_168655_object29558:head# 2024-03-20T09:26:44.083 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.083 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d14e2db:::benchmark_data_smithi122_168655_object48191:head# 2024-03-20T09:26:44.083 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.083 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d152845:::benchmark_data_smithi122_168655_object116862:head# 2024-03-20T09:26:44.083 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.083 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d1528ce:::benchmark_data_smithi122_168655_object21968:head# 2024-03-20T09:26:44.083 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.083 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d152944:::benchmark_data_smithi122_168655_object61683:head# 2024-03-20T09:26:44.083 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.084 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d15aa08:::benchmark_data_smithi122_168655_object16275:head# 2024-03-20T09:26:44.084 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.084 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d15aaf2:::benchmark_data_smithi122_168655_object10675:head# 2024-03-20T09:26:44.084 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.084 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d16b5f8:::benchmark_data_smithi122_168655_object25741:head# 2024-03-20T09:26:44.084 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.084 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d16db24:::benchmark_data_smithi122_168655_object42160:head# 2024-03-20T09:26:44.084 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.084 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d178b90:::benchmark_data_smithi122_168655_object65397:head# 2024-03-20T09:26:44.084 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.084 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d17fea3:::benchmark_data_smithi122_168655_object31981:head# 2024-03-20T09:26:44.084 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.084 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d1879a1:::benchmark_data_smithi122_168655_object31245:head# 2024-03-20T09:26:44.084 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.085 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d1892bb:::benchmark_data_smithi122_168655_object77883:head# 2024-03-20T09:26:44.085 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.085 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d18bf2b:::benchmark_data_smithi122_168655_object75458:head# 2024-03-20T09:26:44.085 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.085 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d192cc0:::benchmark_data_smithi122_168655_object85508:head# 2024-03-20T09:26:44.085 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.085 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d1a92bd:::benchmark_data_smithi122_168655_object61988:head# 2024-03-20T09:26:44.085 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.085 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d1b1066:::benchmark_data_smithi122_168655_object99639:head# 2024-03-20T09:26:44.085 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.085 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d1b811f:::benchmark_data_smithi122_168655_object42897:head# 2024-03-20T09:26:44.085 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.085 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d1cd03e:::benchmark_data_smithi122_168655_object71856:head# 2024-03-20T09:26:44.086 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.086 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d1d5939:::benchmark_data_smithi122_168655_object93190:head# 2024-03-20T09:26:44.086 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.086 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d1d6dd1:::benchmark_data_smithi122_168655_object32021:head# 2024-03-20T09:26:44.086 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.086 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d1d9d97:::benchmark_data_smithi122_168655_object32928:head# 2024-03-20T09:26:44.086 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.086 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d1de94e:::benchmark_data_smithi122_168655_object85091:head# 2024-03-20T09:26:44.086 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.086 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d1e62b0:::benchmark_data_smithi122_168655_object86963:head# 2024-03-20T09:26:44.086 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.086 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d1e8786:::benchmark_data_smithi122_168655_object107231:head# 2024-03-20T09:26:44.086 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.086 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d1e9e77:::benchmark_data_smithi122_168655_object14460:head# 2024-03-20T09:26:44.087 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.087 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d1f56a7:::benchmark_data_smithi122_168655_object16960:head# 2024-03-20T09:26:44.087 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.087 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d20b6bb:::benchmark_data_smithi122_168655_object45263:head# 2024-03-20T09:26:44.087 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.087 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d226fe5:::benchmark_data_smithi122_168655_object34132:head# 2024-03-20T09:26:44.087 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.087 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d236263:::benchmark_data_smithi122_168655_object73908:head# 2024-03-20T09:26:44.087 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.087 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d23e70e:::benchmark_data_smithi122_168655_object45510:head# 2024-03-20T09:26:44.087 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.087 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d23ff26:::benchmark_data_smithi122_168655_object73910:head# 2024-03-20T09:26:44.087 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.087 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d241790:::benchmark_data_smithi122_168655_object68727:head# 2024-03-20T09:26:44.088 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.088 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d24f1b3:::benchmark_data_smithi122_168655_object50277:head# 2024-03-20T09:26:44.088 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.088 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d252232:::benchmark_data_smithi122_168655_object62201:head# 2024-03-20T09:26:44.088 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.088 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d272cb6:::benchmark_data_smithi122_168655_object42830:head# 2024-03-20T09:26:44.088 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.088 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d276b39:::benchmark_data_smithi122_168655_object110695:head# 2024-03-20T09:26:44.088 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.088 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d27a7e8:::benchmark_data_smithi122_168655_object105164:head# 2024-03-20T09:26:44.088 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.088 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d283556:::benchmark_data_smithi122_168655_object55377:head# 2024-03-20T09:26:44.088 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.089 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d28402d:::benchmark_data_smithi122_168655_object99942:head# 2024-03-20T09:26:44.089 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.089 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d2a3758:::benchmark_data_smithi122_168655_object64110:head# 2024-03-20T09:26:44.089 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.089 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d2ad066:::benchmark_data_smithi122_168655_object51659:head# 2024-03-20T09:26:44.089 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.089 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d2b8a03:::benchmark_data_smithi122_168655_object108801:head# 2024-03-20T09:26:44.089 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.089 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d2bd4f2:::benchmark_data_smithi122_168655_object100090:head# 2024-03-20T09:26:44.089 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.089 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d2beee3:::benchmark_data_smithi122_168655_object113922:head# 2024-03-20T09:26:44.089 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.089 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d2d95ad:::benchmark_data_smithi122_168655_object105717:head# 2024-03-20T09:26:44.089 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.090 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d2e3000:::benchmark_data_smithi122_168655_object51850:head# 2024-03-20T09:26:44.090 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.090 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d2e68b6:::benchmark_data_smithi122_168655_object113098:head# 2024-03-20T09:26:44.090 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.090 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d2eafbc:::benchmark_data_smithi122_168655_object30172:head# 2024-03-20T09:26:44.090 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.090 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d2f0b86:::benchmark_data_smithi122_168655_object90805:head# 2024-03-20T09:26:44.090 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.090 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d2f65aa:::benchmark_data_smithi122_168655_object50407:head# 2024-03-20T09:26:44.090 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.090 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d310f65:::benchmark_data_smithi122_168655_object78262:head# 2024-03-20T09:26:44.090 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.090 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d316119:::benchmark_data_smithi122_168655_object108684:head# 2024-03-20T09:26:44.090 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.091 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d31c993:::benchmark_data_smithi122_168655_object110265:head# 2024-03-20T09:26:44.091 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.091 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d32037b:::benchmark_data_smithi122_168655_object12927:head# 2024-03-20T09:26:44.091 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.091 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d32e12f:::benchmark_data_smithi122_168655_object95100:head# 2024-03-20T09:26:44.091 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.091 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d33049c:::benchmark_data_smithi122_168655_object61156:head# 2024-03-20T09:26:44.091 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.091 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d35fc29:::benchmark_data_smithi122_168655_object92614:head# 2024-03-20T09:26:44.091 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.091 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d360086:::benchmark_data_smithi122_168655_object63538:head# 2024-03-20T09:26:44.091 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.091 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d375ef9:::benchmark_data_smithi122_168655_object57823:head# 2024-03-20T09:26:44.092 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.092 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d37ee9a:::benchmark_data_smithi122_168655_object82503:head# 2024-03-20T09:26:44.092 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.092 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d387aea:::benchmark_data_smithi122_168655_object32539:head# 2024-03-20T09:26:44.092 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.092 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d392f2d:::benchmark_data_smithi122_168655_object51432:head# 2024-03-20T09:26:44.092 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.092 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d395e95:::benchmark_data_smithi122_168655_object36244:head# 2024-03-20T09:26:44.092 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.092 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d39d55b:::benchmark_data_smithi122_168655_object58996:head# 2024-03-20T09:26:44.092 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.093 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d3ab0b1:::benchmark_data_smithi122_168655_object30632:head# 2024-03-20T09:26:44.093 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.093 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d3c3ffc:::benchmark_data_smithi122_168655_object72383:head# 2024-03-20T09:26:44.093 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.093 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d3c5d36:::benchmark_data_smithi122_168655_object117921:head# 2024-03-20T09:26:44.093 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.093 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d3d42fd:::benchmark_data_smithi122_168655_object13353:head# 2024-03-20T09:26:44.093 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.093 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d3d4672:::benchmark_data_smithi122_168655_object56998:head# 2024-03-20T09:26:44.093 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.093 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d3e7d89:::benchmark_data_smithi122_168655_object106283:head# 2024-03-20T09:26:44.093 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.094 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d3f11dd:::benchmark_data_smithi122_168655_object78725:head# 2024-03-20T09:26:44.094 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.094 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d3f69f0:::benchmark_data_smithi122_168655_object49723:head# 2024-03-20T09:26:44.094 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.094 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d402ad1:::benchmark_data_smithi122_168655_object64076:head# 2024-03-20T09:26:44.094 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.094 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d41b19b:::benchmark_data_smithi122_168655_object35134:head# 2024-03-20T09:26:44.094 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.094 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d41fa2f:::benchmark_data_smithi122_168655_object55452:head# 2024-03-20T09:26:44.094 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.094 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d4249f8:::benchmark_data_smithi122_168655_object98091:head# 2024-03-20T09:26:44.094 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.094 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d42675f:::benchmark_data_smithi122_168655_object116273:head# 2024-03-20T09:26:44.095 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.095 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d42dde8:::benchmark_data_smithi122_168655_object52955:head# 2024-03-20T09:26:44.095 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.095 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d432441:::benchmark_data_smithi122_168655_object110473:head# 2024-03-20T09:26:44.095 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.095 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d43b9a4:::benchmark_data_smithi122_168655_object24350:head# 2024-03-20T09:26:44.095 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.095 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d448a03:::benchmark_data_smithi122_168655_object10351:head# 2024-03-20T09:26:44.095 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.095 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d449f3e:::benchmark_data_smithi122_168655_object106289:head# 2024-03-20T09:26:44.095 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.095 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d44c278:::benchmark_data_smithi122_168655_object35353:head# 2024-03-20T09:26:44.095 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.095 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d44e9ce:::benchmark_data_smithi122_168655_object75691:head# 2024-03-20T09:26:44.096 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.096 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d46022d:::benchmark_data_smithi122_168655_object44677:head# 2024-03-20T09:26:44.096 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.096 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d46bc30:::benchmark_data_smithi122_168655_object39279:head# 2024-03-20T09:26:44.096 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.096 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d46c117:::benchmark_data_smithi122_168655_object99373:head# 2024-03-20T09:26:44.096 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.096 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d476e3c:::benchmark_data_smithi122_168655_object83151:head# 2024-03-20T09:26:44.096 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.096 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d48c1a4:::benchmark_data_smithi122_168655_object67814:head# 2024-03-20T09:26:44.096 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.096 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d4be015:::benchmark_data_smithi122_168655_object93596:head# 2024-03-20T09:26:44.096 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.096 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d4bf8c0:::benchmark_data_smithi122_168655_object72819:head# 2024-03-20T09:26:44.097 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.097 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d4ca743:::benchmark_data_smithi122_168655_object37644:head# 2024-03-20T09:26:44.097 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.097 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d4cda70:::benchmark_data_smithi122_168655_object52146:head# 2024-03-20T09:26:44.097 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.097 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d4d2e52:::benchmark_data_smithi122_168655_object14086:head# 2024-03-20T09:26:44.097 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.097 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d4d5086:::benchmark_data_smithi122_168655_object115962:head# 2024-03-20T09:26:44.097 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.097 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d4dffbf:::benchmark_data_smithi122_168655_object17934:head# 2024-03-20T09:26:44.097 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.097 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d4e46b8:::benchmark_data_smithi122_168655_object90229:head# 2024-03-20T09:26:44.097 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.097 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d506513:::benchmark_data_smithi122_168655_object109063:head# 2024-03-20T09:26:44.097 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.097 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d513bd8:::benchmark_data_smithi122_168655_object44131:head# 2024-03-20T09:26:44.098 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.098 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d52020b:::benchmark_data_smithi122_168655_object56137:head# 2024-03-20T09:26:44.098 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.098 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d52703b:::benchmark_data_smithi122_168655_object80477:head# 2024-03-20T09:26:44.098 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.098 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d52f543:::benchmark_data_smithi122_168655_object104320:head# 2024-03-20T09:26:44.098 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.098 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d533f65:::benchmark_data_smithi122_168655_object118745:head# 2024-03-20T09:26:44.098 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.098 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d53f5f8:::benchmark_data_smithi122_168655_object96922:head# 2024-03-20T09:26:44.098 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.098 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d54544c:::benchmark_data_smithi122_168655_object82491:head# 2024-03-20T09:26:44.098 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.098 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d55958b:::benchmark_data_smithi122_168655_object85724:head# 2024-03-20T09:26:44.099 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.099 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d564a97:::benchmark_data_smithi122_168655_object69542:head# 2024-03-20T09:26:44.099 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.099 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d5671fb:::benchmark_data_smithi122_168655_object43257:head# 2024-03-20T09:26:44.099 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.099 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d57bdf3:::benchmark_data_smithi122_168655_object103393:head# 2024-03-20T09:26:44.099 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.099 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d57ef9a:::benchmark_data_smithi122_168655_object114121:head# 2024-03-20T09:26:44.099 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.099 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d59941c:::benchmark_data_smithi122_168655_object16590:head# 2024-03-20T09:26:44.099 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.099 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d5c51a8:::benchmark_data_smithi122_168655_object14910:head# 2024-03-20T09:26:44.099 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.099 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d5c72cb:::benchmark_data_smithi122_168655_object70451:head# 2024-03-20T09:26:44.100 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.100 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d5c8d88:::benchmark_data_smithi122_168655_object75431:head# 2024-03-20T09:26:44.100 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.100 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d5c95b3:::benchmark_data_smithi122_168655_object107217:head# 2024-03-20T09:26:44.100 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.100 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d5ce0c9:::benchmark_data_smithi122_168655_object12924:head# 2024-03-20T09:26:44.100 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.100 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d5ce98c:::benchmark_data_smithi122_168655_object33688:head# 2024-03-20T09:26:44.100 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.100 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d5d3e4c:::benchmark_data_smithi122_168655_object52414:head# 2024-03-20T09:26:44.100 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.100 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d5d93fc:::benchmark_data_smithi122_168655_object64250:head# 2024-03-20T09:26:44.100 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.100 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d5fe941:::benchmark_data_smithi122_168655_object103758:head# 2024-03-20T09:26:44.101 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.101 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d60a06d:::benchmark_data_smithi122_168655_object52127:head# 2024-03-20T09:26:44.101 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.101 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d60a7a9:::benchmark_data_smithi122_168655_object18147:head# 2024-03-20T09:26:44.101 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.101 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d614985:::benchmark_data_smithi122_168655_object62979:head# 2024-03-20T09:26:44.101 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.101 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d616605:::benchmark_data_smithi122_168655_object65141:head# 2024-03-20T09:26:44.101 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.101 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d624e24:::benchmark_data_smithi122_168655_object65363:head# 2024-03-20T09:26:44.101 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.101 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d62f6a8:::benchmark_data_smithi122_168655_object63255:head# 2024-03-20T09:26:44.101 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.102 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d63abf0:::benchmark_data_smithi122_168655_object97687:head# 2024-03-20T09:26:44.102 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.102 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d63fa20:::benchmark_data_smithi122_168655_object79440:head# 2024-03-20T09:26:44.102 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.102 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d642389:::benchmark_data_smithi122_168655_object24288:head# 2024-03-20T09:26:44.102 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.102 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d643c58:::benchmark_data_smithi122_168655_object56717:head# 2024-03-20T09:26:44.102 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.102 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d647218:::benchmark_data_smithi122_168655_object94873:head# 2024-03-20T09:26:44.102 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.102 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d64f2ff:::benchmark_data_smithi122_168655_object117379:head# 2024-03-20T09:26:44.102 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.102 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d650da9:::benchmark_data_smithi122_168655_object17922:head# 2024-03-20T09:26:44.103 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.103 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d659275:::benchmark_data_smithi122_168655_object109078:head# 2024-03-20T09:26:44.103 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.103 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d6609e1:::benchmark_data_smithi122_168655_object77374:head# 2024-03-20T09:26:44.103 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.103 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d6626b5:::benchmark_data_smithi122_168655_object28225:head# 2024-03-20T09:26:44.103 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.103 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d66cd01:::benchmark_data_smithi122_168655_object51267:head# 2024-03-20T09:26:44.103 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.103 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d66fb6f:::benchmark_data_smithi122_168655_object65391:head# 2024-03-20T09:26:44.103 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.103 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d67c5e2:::benchmark_data_smithi122_168655_object54309:head# 2024-03-20T09:26:44.103 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.104 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d67ce47:::benchmark_data_smithi122_168655_object88608:head# 2024-03-20T09:26:44.104 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.104 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d67ce95:::benchmark_data_smithi122_168655_object11106:head# 2024-03-20T09:26:44.104 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.104 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d68d0ff:::benchmark_data_smithi122_168655_object76954:head# 2024-03-20T09:26:44.104 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.104 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d6910c8:::benchmark_data_smithi122_168655_object94365:head# 2024-03-20T09:26:44.104 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.104 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d6b1f5f:::benchmark_data_smithi122_168655_object23903:head# 2024-03-20T09:26:44.104 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.104 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d6bb96a:::benchmark_data_smithi122_168655_object87492:head# 2024-03-20T09:26:44.104 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.104 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d6d77db:::benchmark_data_smithi122_168655_object114725:head# 2024-03-20T09:26:44.104 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.105 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d6e02e8:::benchmark_data_smithi122_168655_object109807:head# 2024-03-20T09:26:44.105 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.105 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d6e68c3:::benchmark_data_smithi122_168655_object93225:head# 2024-03-20T09:26:44.105 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.105 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d6e7213:::benchmark_data_smithi122_168655_object48142:head# 2024-03-20T09:26:44.105 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.105 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d6f0aee:::benchmark_data_smithi122_168655_object67559:head# 2024-03-20T09:26:44.105 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.105 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d6fa619:::benchmark_data_smithi122_168655_object51405:head# 2024-03-20T09:26:44.105 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.105 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d6fc662:::benchmark_data_smithi122_168655_object23412:head# 2024-03-20T09:26:44.105 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.105 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d7104a4:::benchmark_data_smithi122_168655_object53231:head# 2024-03-20T09:26:44.105 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.106 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d72e549:::benchmark_data_smithi122_168655_object106207:head# 2024-03-20T09:26:44.106 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.106 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d75f039:::benchmark_data_smithi122_168655_object64265:head# 2024-03-20T09:26:44.106 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.106 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d772e40:::benchmark_data_smithi122_168655_object102664:head# 2024-03-20T09:26:44.106 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.106 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d776eb7:::benchmark_data_smithi122_168655_object19028:head# 2024-03-20T09:26:44.106 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.106 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d77a4be:::benchmark_data_smithi122_168655_object87568:head# 2024-03-20T09:26:44.106 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.106 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d77e421:::benchmark_data_smithi122_168655_object66999:head# 2024-03-20T09:26:44.106 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.106 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d7941cb:::benchmark_data_smithi122_168655_object109267:head# 2024-03-20T09:26:44.107 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.107 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d79d647:::benchmark_data_smithi122_168655_object66099:head# 2024-03-20T09:26:44.107 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.107 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d7a2406:::benchmark_data_smithi122_168655_object22945:head# 2024-03-20T09:26:44.107 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.107 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d7a43eb:::benchmark_data_smithi122_168655_object112428:head# 2024-03-20T09:26:44.107 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.107 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d7b24a6:::benchmark_data_smithi122_168655_object61179:head# 2024-03-20T09:26:44.107 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.107 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d7b9aa8:::benchmark_data_smithi122_168655_object42895:head# 2024-03-20T09:26:44.107 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.107 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d7bec9e:::benchmark_data_smithi122_168655_object101650:head# 2024-03-20T09:26:44.107 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.108 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d7c2866:::benchmark_data_smithi122_168655_object15054:head# 2024-03-20T09:26:44.108 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.108 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d7cbe5f:::benchmark_data_smithi122_168655_object15770:head# 2024-03-20T09:26:44.108 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.108 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d7cf00b:::benchmark_data_smithi122_168655_object77082:head# 2024-03-20T09:26:44.108 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.108 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d7dedc3:::benchmark_data_smithi122_168655_object67923:head# 2024-03-20T09:26:44.108 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.108 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d7efb44:::benchmark_data_smithi122_168655_object24596:head# 2024-03-20T09:26:44.108 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.108 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d7f235b:::benchmark_data_smithi122_168655_object64472:head# 2024-03-20T09:26:44.108 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.108 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d8000e0:::benchmark_data_smithi122_168655_object32610:head# 2024-03-20T09:26:44.108 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.109 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d813934:::benchmark_data_smithi122_168655_object71602:head# 2024-03-20T09:26:44.109 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.109 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d826f7d:::benchmark_data_smithi122_168655_object100511:head# 2024-03-20T09:26:44.109 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.109 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d82942b:::benchmark_data_smithi122_168655_object76564:head# 2024-03-20T09:26:44.109 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.109 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d82985d:::benchmark_data_smithi122_168655_object39499:head# 2024-03-20T09:26:44.109 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.109 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d83b23f:::benchmark_data_smithi122_168655_object49683:head# 2024-03-20T09:26:44.109 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.109 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d844d09:::benchmark_data_smithi122_168655_object17566:head# 2024-03-20T09:26:44.109 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.109 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d852918:::benchmark_data_smithi122_168655_object75193:head# 2024-03-20T09:26:44.109 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.109 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d85647d:::benchmark_data_smithi122_168655_object101878:head# 2024-03-20T09:26:44.110 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.110 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d85e305:::benchmark_data_smithi122_168655_object109340:head# 2024-03-20T09:26:44.110 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.110 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d85eb16:::benchmark_data_smithi122_168655_object110765:head# 2024-03-20T09:26:44.110 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.110 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d86bf8f:::benchmark_data_smithi122_168655_object82463:head# 2024-03-20T09:26:44.110 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.110 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d86cba6:::benchmark_data_smithi122_168655_object54244:head# 2024-03-20T09:26:44.110 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.110 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d87162a:::benchmark_data_smithi122_168655_object55220:head# 2024-03-20T09:26:44.110 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.110 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d8781d0:::benchmark_data_smithi122_168655_object100344:head# 2024-03-20T09:26:44.110 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.111 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d87b650:::benchmark_data_smithi122_168655_object77005:head# 2024-03-20T09:26:44.111 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.111 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d88d4df:::benchmark_data_smithi122_168655_object40813:head# 2024-03-20T09:26:44.111 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.111 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d8a6a9c:::benchmark_data_smithi122_168655_object21853:head# 2024-03-20T09:26:44.111 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.111 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d8d559a:::benchmark_data_smithi122_168655_object68831:head# 2024-03-20T09:26:44.111 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.111 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d8e1003:::benchmark_data_smithi122_168655_object91252:head# 2024-03-20T09:26:44.111 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.111 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d8e9bf4:::benchmark_data_smithi122_168655_object113421:head# 2024-03-20T09:26:44.111 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.111 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d8ef2fd:::benchmark_data_smithi122_168655_object86638:head# 2024-03-20T09:26:44.111 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.112 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d8f3b7a:::benchmark_data_smithi122_168655_object88106:head# 2024-03-20T09:26:44.112 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.112 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d8f8375:::benchmark_data_smithi122_168655_object45585:head# 2024-03-20T09:26:44.112 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.112 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d902b22:::benchmark_data_smithi122_168655_object48146:head# 2024-03-20T09:26:44.112 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.112 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d904548:::benchmark_data_smithi122_168655_object108166:head# 2024-03-20T09:26:44.112 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.112 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d90634f:::benchmark_data_smithi122_168655_object109356:head# 2024-03-20T09:26:44.112 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.112 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d92c398:::benchmark_data_smithi122_168655_object114347:head# 2024-03-20T09:26:44.112 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.112 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d931553:::benchmark_data_smithi122_168655_object55802:head# 2024-03-20T09:26:44.112 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.113 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d93fc00:::benchmark_data_smithi122_168655_object24214:head# 2024-03-20T09:26:44.113 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.113 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d947f71:::benchmark_data_smithi122_168655_object69331:head# 2024-03-20T09:26:44.113 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.113 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d958128:::benchmark_data_smithi122_168655_object83971:head# 2024-03-20T09:26:44.113 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.113 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d9832e5:::benchmark_data_smithi122_168655_object15519:head# 2024-03-20T09:26:44.113 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.113 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d98453e:::benchmark_data_smithi122_168655_object116673:head# 2024-03-20T09:26:44.113 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.113 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d99e52f:::benchmark_data_smithi122_168655_object28066:head# 2024-03-20T09:26:44.113 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.113 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d9ab6b5:::benchmark_data_smithi122_168655_object88875:head# 2024-03-20T09:26:44.113 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.114 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d9b5fc7:::benchmark_data_smithi122_168655_object38154:head# 2024-03-20T09:26:44.114 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.114 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d9bc21a:::benchmark_data_smithi122_168655_object63348:head# 2024-03-20T09:26:44.114 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.114 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d9be452:::benchmark_data_smithi122_168655_object72941:head# 2024-03-20T09:26:44.114 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.114 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d9c0797:::benchmark_data_smithi122_168655_object67392:head# 2024-03-20T09:26:44.114 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.114 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d9c42b8:::benchmark_data_smithi122_168655_object68298:head# 2024-03-20T09:26:44.114 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.114 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d9cb872:::benchmark_data_smithi122_168655_object33356:head# 2024-03-20T09:26:44.114 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.114 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d9d4569:::benchmark_data_smithi122_168655_object64101:head# 2024-03-20T09:26:44.115 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.115 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d9e2b96:::benchmark_data_smithi122_168655_object93862:head# 2024-03-20T09:26:44.115 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.115 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d9e89bb:::benchmark_data_smithi122_168655_object33777:head# 2024-03-20T09:26:44.115 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.115 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d9eb03e:::benchmark_data_smithi122_168655_object44447:head# 2024-03-20T09:26:44.115 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.115 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d9ecd4b:::benchmark_data_smithi122_168655_object13257:head# 2024-03-20T09:26:44.115 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.115 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d9ee518:::benchmark_data_smithi122_168655_object61041:head# 2024-03-20T09:26:44.116 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.116 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8d9f1fd8:::benchmark_data_smithi122_168655_object16894:head# 2024-03-20T09:26:44.116 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.116 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8da096fd:::benchmark_data_smithi122_168655_object57061:head# 2024-03-20T09:26:44.116 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.116 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8da0c5c6:::benchmark_data_smithi122_168655_object61024:head# 2024-03-20T09:26:44.116 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.116 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8da0fb3b:::benchmark_data_smithi122_168655_object101479:head# 2024-03-20T09:26:44.116 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.116 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8da19c39:::benchmark_data_smithi122_168655_object48190:head# 2024-03-20T09:26:44.116 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.116 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8da1d7e3:::benchmark_data_smithi122_168655_object21893:head# 2024-03-20T09:26:44.116 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.117 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8da2432d:::benchmark_data_smithi122_168655_object98499:head# 2024-03-20T09:26:44.117 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.117 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8da24bb3:::benchmark_data_smithi122_168655_object119380:head# 2024-03-20T09:26:44.117 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.117 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8da2f527:::benchmark_data_smithi122_168655_object73896:head# 2024-03-20T09:26:44.117 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.117 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8da3e028:::benchmark_data_smithi122_168655_object86305:head# 2024-03-20T09:26:44.117 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.117 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8da3e0c5:::benchmark_data_smithi122_168655_object43141:head# 2024-03-20T09:26:44.117 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.117 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8da4765f:::benchmark_data_smithi122_168655_object105455:head# 2024-03-20T09:26:44.117 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.117 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8da49cc8:::benchmark_data_smithi122_168655_object90811:head# 2024-03-20T09:26:44.117 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.118 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8da56a43:::benchmark_data_smithi122_168655_object88240:head# 2024-03-20T09:26:44.118 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.118 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8da632c9:::benchmark_data_smithi122_168655_object14632:head# 2024-03-20T09:26:44.118 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.118 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8da6812e:::benchmark_data_smithi122_168655_object23808:head# 2024-03-20T09:26:44.118 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.118 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8da73147:::benchmark_data_smithi122_168655_object15771:head# 2024-03-20T09:26:44.118 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.118 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8da873b3:::benchmark_data_smithi122_168655_object46494:head# 2024-03-20T09:26:44.118 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.118 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8daa1eac:::benchmark_data_smithi122_168655_object113307:head# 2024-03-20T09:26:44.118 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.118 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8daa6312:::benchmark_data_smithi122_168655_object41663:head# 2024-03-20T09:26:44.119 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.119 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8daa690a:::benchmark_data_smithi122_168655_object58024:head# 2024-03-20T09:26:44.119 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.119 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8daa9eb9:::benchmark_data_smithi122_168655_object80268:head# 2024-03-20T09:26:44.119 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.119 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8daad4d8:::benchmark_data_smithi122_168655_object111197:head# 2024-03-20T09:26:44.119 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.119 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dab5052:::benchmark_data_smithi122_168655_object113838:head# 2024-03-20T09:26:44.119 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.119 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dab51a1:::benchmark_data_smithi122_168655_object108263:head# 2024-03-20T09:26:44.119 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.119 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dac779d:::benchmark_data_smithi122_168655_object33511:head# 2024-03-20T09:26:44.119 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.119 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dae1141:::benchmark_data_smithi122_168655_object97244:head# 2024-03-20T09:26:44.120 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.120 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8daed01b:::benchmark_data_smithi122_168655_object23029:head# 2024-03-20T09:26:44.120 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.120 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8daee22a:::benchmark_data_smithi122_168655_object24923:head# 2024-03-20T09:26:44.120 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.120 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8daee451:::benchmark_data_smithi122_168655_object100420:head# 2024-03-20T09:26:44.120 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.120 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8daf3153:::benchmark_data_smithi122_168655_object44827:head# 2024-03-20T09:26:44.120 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.120 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8db0ac65:::benchmark_data_smithi122_168655_object13879:head# 2024-03-20T09:26:44.120 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.120 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8db0adb3:::benchmark_data_smithi122_168655_object31891:head# 2024-03-20T09:26:44.120 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.120 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8db1345b:::benchmark_data_smithi122_168655_object69195:head# 2024-03-20T09:26:44.120 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.121 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8db213b8:::benchmark_data_smithi122_168655_object106493:head# 2024-03-20T09:26:44.121 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.121 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8db25546:::benchmark_data_smithi122_168655_object84077:head# 2024-03-20T09:26:44.121 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.121 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8db282be:::benchmark_data_smithi122_168655_object77719:head# 2024-03-20T09:26:44.121 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.121 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8db39366:::benchmark_data_smithi122_168655_object106656:head# 2024-03-20T09:26:44.121 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.121 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8db3ade6:::benchmark_data_smithi122_168655_object51808:head# 2024-03-20T09:26:44.121 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.121 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8db454de:::benchmark_data_smithi122_168655_object34098:head# 2024-03-20T09:26:44.121 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.121 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8db4af73:::benchmark_data_smithi122_168655_object48901:head# 2024-03-20T09:26:44.121 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.122 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8db4c268:::benchmark_data_smithi122_168655_object37204:head# 2024-03-20T09:26:44.122 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.122 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8db59b0c:::benchmark_data_smithi122_168655_object46783:head# 2024-03-20T09:26:44.122 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.122 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8db5a32c:::benchmark_data_smithi122_168655_object43616:head# 2024-03-20T09:26:44.122 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.122 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8db5ae7a:::benchmark_data_smithi122_168655_object94942:head# 2024-03-20T09:26:44.122 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.122 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8db645fe:::benchmark_data_smithi122_168655_object104167:head# 2024-03-20T09:26:44.122 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.122 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8db66923:::benchmark_data_smithi122_168655_object80997:head# 2024-03-20T09:26:44.122 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.122 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8db6e419:::benchmark_data_smithi122_168655_object96957:head# 2024-03-20T09:26:44.123 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.123 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8db6f53f:::benchmark_data_smithi122_168655_object91491:head# 2024-03-20T09:26:44.123 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.123 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8db7235b:::benchmark_data_smithi122_168655_object97123:head# 2024-03-20T09:26:44.123 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.123 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8db7876d:::benchmark_data_smithi122_168655_object34457:head# 2024-03-20T09:26:44.123 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.123 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8db8709b:::benchmark_data_smithi122_168655_object56837:head# 2024-03-20T09:26:44.123 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.123 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8db8e5d2:::benchmark_data_smithi122_168655_object105655:head# 2024-03-20T09:26:44.123 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.123 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8db8f7df:::benchmark_data_smithi122_168655_object70019:head# 2024-03-20T09:26:44.123 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.123 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8db93cd7:::benchmark_data_smithi122_168655_object64695:head# 2024-03-20T09:26:44.124 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.124 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dba0926:::benchmark_data_smithi122_168655_object53818:head# 2024-03-20T09:26:44.124 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.124 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dba0e88:::benchmark_data_smithi122_168655_object21211:head# 2024-03-20T09:26:44.124 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.124 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dbae245:::benchmark_data_smithi122_168655_object113088:head# 2024-03-20T09:26:44.124 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.124 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dc1659a:::benchmark_data_smithi122_168655_object103763:head# 2024-03-20T09:26:44.124 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.124 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dc1e066:::benchmark_data_smithi122_168655_object24614:head# 2024-03-20T09:26:44.124 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.124 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dc2049f:::benchmark_data_smithi122_168655_object49025:head# 2024-03-20T09:26:44.124 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.124 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dc42a7e:::benchmark_data_smithi122_168655_object63604:head# 2024-03-20T09:26:44.124 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.125 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dc63350:::benchmark_data_smithi122_168655_object95841:head# 2024-03-20T09:26:44.125 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.125 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dc6bc09:::benchmark_data_smithi122_168655_object53847:head# 2024-03-20T09:26:44.125 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.125 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dc72dbd:::benchmark_data_smithi122_168655_object60831:head# 2024-03-20T09:26:44.125 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.125 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dc7da3c:::benchmark_data_smithi122_168655_object40661:head# 2024-03-20T09:26:44.125 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.125 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dc8059c:::benchmark_data_smithi122_168655_object43714:head# 2024-03-20T09:26:44.125 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.125 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dc815ad:::benchmark_data_smithi122_168655_object35102:head# 2024-03-20T09:26:44.125 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.125 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dc8465f:::benchmark_data_smithi122_168655_object36140:head# 2024-03-20T09:26:44.125 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.125 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dc8988d:::benchmark_data_smithi122_168655_object15843:head# 2024-03-20T09:26:44.126 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.126 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dc91534:::benchmark_data_smithi122_168655_object71408:head# 2024-03-20T09:26:44.126 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.126 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dc96942:::benchmark_data_smithi122_168655_object42988:head# 2024-03-20T09:26:44.126 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.126 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dca9f5a:::benchmark_data_smithi122_168655_object104256:head# 2024-03-20T09:26:44.126 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.126 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dcbddb2:::benchmark_data_smithi122_168655_object55517:head# 2024-03-20T09:26:44.126 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.126 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dcc0b69:::benchmark_data_smithi122_168655_object85021:head# 2024-03-20T09:26:44.126 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.126 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dcc8fdb:::benchmark_data_smithi122_168655_object56751:head# 2024-03-20T09:26:44.127 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.127 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dcd08d2:::benchmark_data_smithi122_168655_object76546:head# 2024-03-20T09:26:44.127 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.127 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dce4e91:::benchmark_data_smithi122_168655_object104885:head# 2024-03-20T09:26:44.127 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.127 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dce604f:::benchmark_data_smithi122_168655_object45226:head# 2024-03-20T09:26:44.127 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.127 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dce7ac3:::benchmark_data_smithi122_168655_object14783:head# 2024-03-20T09:26:44.127 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.127 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dcee617:::benchmark_data_smithi122_168655_object52941:head# 2024-03-20T09:26:44.127 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.127 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dcf8050:::benchmark_data_smithi122_168655_object27836:head# 2024-03-20T09:26:44.127 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.128 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dcfc499:::benchmark_data_smithi122_168655_object102586:head# 2024-03-20T09:26:44.128 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.128 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dd0620f:::benchmark_data_smithi122_168655_object77308:head# 2024-03-20T09:26:44.128 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.128 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dd2699e:::benchmark_data_smithi122_168655_object16149:head# 2024-03-20T09:26:44.128 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.128 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dd2a72f:::benchmark_data_smithi122_168655_object54747:head# 2024-03-20T09:26:44.128 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.128 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dd46f7e:::benchmark_data_smithi122_168655_object20661:head# 2024-03-20T09:26:44.128 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.128 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dd56af0:::benchmark_data_smithi122_168655_object119695:head# 2024-03-20T09:26:44.128 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.128 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dd58858:::benchmark_data_smithi122_168655_object115691:head# 2024-03-20T09:26:44.128 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.129 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dd77f95:::benchmark_data_smithi122_168655_object107563:head# 2024-03-20T09:26:44.129 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.129 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dd826c8:::benchmark_data_smithi122_168655_object10211:head# 2024-03-20T09:26:44.129 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.129 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dd82c82:::benchmark_data_smithi122_168655_object50770:head# 2024-03-20T09:26:44.129 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.129 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dda07c1:::benchmark_data_smithi122_168655_object50870:head# 2024-03-20T09:26:44.129 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.129 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ddac286:::benchmark_data_smithi122_168655_object73485:head# 2024-03-20T09:26:44.129 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.129 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ddad7ad:::benchmark_data_smithi122_168655_object73974:head# 2024-03-20T09:26:44.129 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.129 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ddb6614:::benchmark_data_smithi122_168655_object11068:head# 2024-03-20T09:26:44.129 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.130 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ddb75c6:::benchmark_data_smithi122_168655_object57385:head# 2024-03-20T09:26:44.130 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.130 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ddcb4fc:::benchmark_data_smithi122_168655_object17043:head# 2024-03-20T09:26:44.130 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.130 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ddd1be3:::benchmark_data_smithi122_168655_object36489:head# 2024-03-20T09:26:44.130 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.130 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dddbf60:::benchmark_data_smithi122_168655_object41729:head# 2024-03-20T09:26:44.130 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.130 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ddfa7d0:::benchmark_data_smithi122_168655_object19215:head# 2024-03-20T09:26:44.130 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.130 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8de0b824:::benchmark_data_smithi122_168655_object39772:head# 2024-03-20T09:26:44.130 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.131 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8de272e4:::benchmark_data_smithi122_168655_object40279:head# 2024-03-20T09:26:44.131 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.131 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8de2c61c:::benchmark_data_smithi122_168655_object20126:head# 2024-03-20T09:26:44.131 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.131 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8de2f1df:::benchmark_data_smithi122_168655_object109190:head# 2024-03-20T09:26:44.131 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.131 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8de3494f:::benchmark_data_smithi122_168655_object98296:head# 2024-03-20T09:26:44.131 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.131 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8de36310:::benchmark_data_smithi122_168655_object35433:head# 2024-03-20T09:26:44.131 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.131 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8de40302:::benchmark_data_smithi122_168655_object41739:head# 2024-03-20T09:26:44.131 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.131 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8de415a1:::benchmark_data_smithi122_168655_object110052:head# 2024-03-20T09:26:44.131 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.132 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8de457de:::benchmark_data_smithi122_168655_object58138:head# 2024-03-20T09:26:44.132 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.132 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8de4bb10:::benchmark_data_smithi122_168655_object20275:head# 2024-03-20T09:26:44.132 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.132 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8de6f7c9:::benchmark_data_smithi122_168655_object21134:head# 2024-03-20T09:26:44.132 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.132 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8de74fba:::benchmark_data_smithi122_168655_object94819:head# 2024-03-20T09:26:44.132 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.132 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8de8f07a:::benchmark_data_smithi122_168655_object54827:head# 2024-03-20T09:26:44.132 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.132 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8de924c4:::benchmark_data_smithi122_168655_object19825:head# 2024-03-20T09:26:44.132 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.132 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8de9fba8:::benchmark_data_smithi122_168655_object84550:head# 2024-03-20T09:26:44.132 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.133 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8deb1ea8:::benchmark_data_smithi122_168655_object12502:head# 2024-03-20T09:26:44.133 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.133 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8deb4866:::benchmark_data_smithi122_168655_object38447:head# 2024-03-20T09:26:44.133 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.133 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dec4f67:::benchmark_data_smithi122_168655_object116617:head# 2024-03-20T09:26:44.133 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.133 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dec73b6:::benchmark_data_smithi122_168655_object43665:head# 2024-03-20T09:26:44.133 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.133 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8decd452:::benchmark_data_smithi122_168655_object21132:head# 2024-03-20T09:26:44.133 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.133 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dedd954:::benchmark_data_smithi122_168655_object47160:head# 2024-03-20T09:26:44.133 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.133 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dedf2e4:::benchmark_data_smithi122_168655_object52750:head# 2024-03-20T09:26:44.133 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.134 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8def1d68:::benchmark_data_smithi122_168655_object60544:head# 2024-03-20T09:26:44.134 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.134 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8def6667:::benchmark_data_smithi122_168655_object72846:head# 2024-03-20T09:26:44.134 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.134 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8df011b8:::benchmark_data_smithi122_168655_object116081:head# 2024-03-20T09:26:44.134 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.134 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8df18d96:::benchmark_data_smithi122_168655_object58608:head# 2024-03-20T09:26:44.134 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.134 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8df31638:::benchmark_data_smithi122_168655_object72339:head# 2024-03-20T09:26:44.134 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.134 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8df43e7b:::benchmark_data_smithi122_168655_object41479:head# 2024-03-20T09:26:44.134 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.134 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8df4467b:::benchmark_data_smithi122_168655_object104764:head# 2024-03-20T09:26:44.134 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.135 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8df4a552:::benchmark_data_smithi122_168655_object91764:head# 2024-03-20T09:26:44.135 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.135 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8df4ef85:::benchmark_data_smithi122_168655_object91996:head# 2024-03-20T09:26:44.135 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.135 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8df5199e:::benchmark_data_smithi122_168655_object37949:head# 2024-03-20T09:26:44.135 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.135 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8df6c2d5:::benchmark_data_smithi122_168655_object12315:head# 2024-03-20T09:26:44.135 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.135 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8df6f9f8:::benchmark_data_smithi122_168655_object100526:head# 2024-03-20T09:26:44.135 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.135 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8df854be:::benchmark_data_smithi122_168655_object102986:head# 2024-03-20T09:26:44.135 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.135 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8df8bf60:::benchmark_data_smithi122_168655_object101126:head# 2024-03-20T09:26:44.135 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.136 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8df9e25d:::benchmark_data_smithi122_168655_object97412:head# 2024-03-20T09:26:44.136 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.136 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dfad901:::benchmark_data_smithi122_168655_object106700:head# 2024-03-20T09:26:44.136 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.136 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dfaebbd:::benchmark_data_smithi122_168655_object110942:head# 2024-03-20T09:26:44.136 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.136 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dfb0576:::benchmark_data_smithi122_168655_object43819:head# 2024-03-20T09:26:44.136 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.136 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dfb0d49:::benchmark_data_smithi122_168655_object99092:head# 2024-03-20T09:26:44.136 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.136 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dfb4703:::benchmark_data_smithi122_168655_object34175:head# 2024-03-20T09:26:44.136 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.136 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dfc9ec4:::benchmark_data_smithi122_168655_object64420:head# 2024-03-20T09:26:44.137 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.137 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dfcda93:::benchmark_data_smithi122_168655_object66270:head# 2024-03-20T09:26:44.137 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.137 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dfd02d1:::benchmark_data_smithi122_168655_object27675:head# 2024-03-20T09:26:44.137 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.137 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dfdd411:::benchmark_data_smithi122_168655_object39545:head# 2024-03-20T09:26:44.137 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.137 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dfe0bd8:::benchmark_data_smithi122_168655_object40624:head# 2024-03-20T09:26:44.137 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.137 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dfe2ad1:::benchmark_data_smithi122_168655_object98260:head# 2024-03-20T09:26:44.137 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.137 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8dff1aad:::benchmark_data_smithi122_168655_object88860:head# 2024-03-20T09:26:44.137 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.137 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e0013f4:::benchmark_data_smithi122_168655_object97393:head# 2024-03-20T09:26:44.138 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.138 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e008267:::benchmark_data_smithi122_168655_object17197:head# 2024-03-20T09:26:44.138 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.138 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e012782:::benchmark_data_smithi122_168655_object119991:head# 2024-03-20T09:26:44.138 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.138 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e019ab5:::benchmark_data_smithi122_168655_object98763:head# 2024-03-20T09:26:44.138 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.138 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e01d9bd:::benchmark_data_smithi122_168655_object44974:head# 2024-03-20T09:26:44.138 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.138 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e01eede:::benchmark_data_smithi122_168655_object102583:head# 2024-03-20T09:26:44.138 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.138 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e0207e1:::benchmark_data_smithi122_168655_object16174:head# 2024-03-20T09:26:44.138 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.139 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e044739:::benchmark_data_smithi122_168655_object113808:head# 2024-03-20T09:26:44.139 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.139 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e070c29:::benchmark_data_smithi122_168655_object104061:head# 2024-03-20T09:26:44.139 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.139 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e07ce2c:::benchmark_data_smithi122_168655_object22953:head# 2024-03-20T09:26:44.139 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.139 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e085861:::benchmark_data_smithi122_168655_object59579:head# 2024-03-20T09:26:44.139 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.139 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e0916b4:::benchmark_data_smithi122_168655_object73174:head# 2024-03-20T09:26:44.139 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.139 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e09e238:::benchmark_data_smithi122_168655_object90346:head# 2024-03-20T09:26:44.139 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.139 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e0a7d1b:::benchmark_data_smithi122_168655_object55667:head# 2024-03-20T09:26:44.140 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.140 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e0af106:::benchmark_data_smithi122_168655_object99159:head# 2024-03-20T09:26:44.140 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.140 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e0afb75:::benchmark_data_smithi122_168655_object89111:head# 2024-03-20T09:26:44.140 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.140 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e0b2faa:::benchmark_data_smithi122_168655_object52568:head# 2024-03-20T09:26:44.140 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.140 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e0b3c22:::benchmark_data_smithi122_168655_object50512:head# 2024-03-20T09:26:44.140 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.140 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e0b846e:::benchmark_data_smithi122_168655_object91648:head# 2024-03-20T09:26:44.140 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.140 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e0c16fc:::benchmark_data_smithi122_168655_object72330:head# 2024-03-20T09:26:44.140 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.140 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e0cf91f:::benchmark_data_smithi122_168655_object50837:head# 2024-03-20T09:26:44.140 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.141 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e0d68b4:::benchmark_data_smithi122_168655_object80222:head# 2024-03-20T09:26:44.141 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.141 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e0df772:::benchmark_data_smithi122_168655_object98561:head# 2024-03-20T09:26:44.141 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.141 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e0e20ff:::benchmark_data_smithi122_168655_object43585:head# 2024-03-20T09:26:44.141 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.141 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e0e2f88:::benchmark_data_smithi122_168655_object52304:head# 2024-03-20T09:26:44.141 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.141 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e0e335e:::benchmark_data_smithi122_168655_object14054:head# 2024-03-20T09:26:44.141 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.141 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e0f2733:::benchmark_data_smithi122_168655_object108726:head# 2024-03-20T09:26:44.141 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.141 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e0f2914:::benchmark_data_smithi122_168655_object14617:head# 2024-03-20T09:26:44.141 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.141 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e0f304a:::benchmark_data_smithi122_168655_object34831:head# 2024-03-20T09:26:44.142 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.142 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e12cd15:::benchmark_data_smithi122_168655_object95439:head# 2024-03-20T09:26:44.142 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.142 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e133470:::benchmark_data_smithi122_168655_object85723:head# 2024-03-20T09:26:44.142 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.142 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e135620:::benchmark_data_smithi122_168655_object20424:head# 2024-03-20T09:26:44.142 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.142 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e13836d:::benchmark_data_smithi122_168655_object114702:head# 2024-03-20T09:26:44.142 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.142 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e1384e5:::benchmark_data_smithi122_168655_object41727:head# 2024-03-20T09:26:44.142 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.142 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e13f896:::benchmark_data_smithi122_168655_object87886:head# 2024-03-20T09:26:44.142 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.142 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e141581:::benchmark_data_smithi122_168655_object78783:head# 2024-03-20T09:26:44.143 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.143 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e1436a0:::benchmark_data_smithi122_168655_object35781:head# 2024-03-20T09:26:44.143 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.143 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e14edc5:::benchmark_data_smithi122_168655_object14263:head# 2024-03-20T09:26:44.143 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.143 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e1532fa:::benchmark_data_smithi122_168655_object77159:head# 2024-03-20T09:26:44.143 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.143 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e15cc99:::benchmark_data_smithi122_168655_object36235:head# 2024-03-20T09:26:44.143 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.143 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e162aa3:::benchmark_data_smithi122_168655_object61290:head# 2024-03-20T09:26:44.143 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.143 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e174300:::benchmark_data_smithi122_168655_object90030:head# 2024-03-20T09:26:44.143 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.144 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e174691:::benchmark_data_smithi122_168655_object88024:head# 2024-03-20T09:26:44.144 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.144 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e17fddb:::benchmark_data_smithi122_168655_object42747:head# 2024-03-20T09:26:44.144 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.144 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e18d4b4:::benchmark_data_smithi122_168655_object116556:head# 2024-03-20T09:26:44.144 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.144 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e190a35:::benchmark_data_smithi122_168655_object61213:head# 2024-03-20T09:26:44.144 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.144 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e1930e7:::benchmark_data_smithi122_168655_object30122:head# 2024-03-20T09:26:44.144 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.144 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e1a8e67:::benchmark_data_smithi122_168655_object70783:head# 2024-03-20T09:26:44.144 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.144 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e1ac104:::benchmark_data_smithi122_168655_object111608:head# 2024-03-20T09:26:44.144 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.145 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e1d0ec0:::benchmark_data_smithi122_168655_object38548:head# 2024-03-20T09:26:44.145 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.145 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e1e04ca:::benchmark_data_smithi122_168655_object11965:head# 2024-03-20T09:26:44.145 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.145 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e1e7ac0:::benchmark_data_smithi122_168655_object116872:head# 2024-03-20T09:26:44.145 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.145 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e1f50b6:::benchmark_data_smithi122_168655_object112502:head# 2024-03-20T09:26:44.145 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.145 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e1f8297:::benchmark_data_smithi122_168655_object41449:head# 2024-03-20T09:26:44.145 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.145 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e2180d3:::benchmark_data_smithi122_168655_object51588:head# 2024-03-20T09:26:44.145 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.145 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e220062:::benchmark_data_smithi122_168655_object69311:head# 2024-03-20T09:26:44.145 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.146 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e2283fb:::benchmark_data_smithi122_168655_object88592:head# 2024-03-20T09:26:44.146 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.146 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e22be6b:::benchmark_data_smithi122_168655_object109906:head# 2024-03-20T09:26:44.146 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.146 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e2453b1:::benchmark_data_smithi122_168655_object45495:head# 2024-03-20T09:26:44.146 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.146 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e262969:::benchmark_data_smithi122_168655_object29670:head# 2024-03-20T09:26:44.146 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.146 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e263f2d:::benchmark_data_smithi122_168655_object102960:head# 2024-03-20T09:26:44.146 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.146 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e26ef4f:::benchmark_data_smithi122_168655_object24758:head# 2024-03-20T09:26:44.146 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.146 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e28452e:::benchmark_data_smithi122_168655_object103274:head# 2024-03-20T09:26:44.146 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.147 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e288f89:::benchmark_data_smithi122_168655_object86298:head# 2024-03-20T09:26:44.147 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.147 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e29941e:::benchmark_data_smithi122_168655_object106809:head# 2024-03-20T09:26:44.147 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.147 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e2a2ebf:::benchmark_data_smithi122_168655_object58829:head# 2024-03-20T09:26:44.147 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.147 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e2aa6b9:::benchmark_data_smithi122_168655_object51045:head# 2024-03-20T09:26:44.147 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.147 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e2aae0d:::benchmark_data_smithi122_168655_object44678:head# 2024-03-20T09:26:44.147 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.147 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e2babeb:::benchmark_data_smithi122_168655_object48562:head# 2024-03-20T09:26:44.147 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.147 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e2be442:::benchmark_data_smithi122_168655_object113569:head# 2024-03-20T09:26:44.147 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.148 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e2c7d97:::benchmark_data_smithi122_168655_object109056:head# 2024-03-20T09:26:44.148 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.148 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e2deeb6:::benchmark_data_smithi122_168655_object112072:head# 2024-03-20T09:26:44.148 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.148 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e2e1461:::benchmark_data_smithi122_168655_object112418:head# 2024-03-20T09:26:44.148 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.148 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e306eef:::benchmark_data_smithi122_168655_object41605:head# 2024-03-20T09:26:44.148 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.148 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e310e36:::benchmark_data_smithi122_168655_object109089:head# 2024-03-20T09:26:44.148 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.148 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e31559f:::benchmark_data_smithi122_168655_object81651:head# 2024-03-20T09:26:44.148 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.148 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e31ba47:::benchmark_data_smithi122_168655_object98396:head# 2024-03-20T09:26:44.149 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.149 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e32646a:::benchmark_data_smithi122_168655_object20981:head# 2024-03-20T09:26:44.149 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.149 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e3277c4:::benchmark_data_smithi122_168655_object98834:head# 2024-03-20T09:26:44.149 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.149 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e32d39c:::benchmark_data_smithi122_168655_object55897:head# 2024-03-20T09:26:44.149 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.149 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e355e26:::benchmark_data_smithi122_168655_object105608:head# 2024-03-20T09:26:44.149 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.149 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e3624e7:::benchmark_data_smithi122_168655_object78287:head# 2024-03-20T09:26:44.149 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.149 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e364afa:::benchmark_data_smithi122_168655_object117382:head# 2024-03-20T09:26:44.149 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.150 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e374a3e:::benchmark_data_smithi122_168655_object114689:head# 2024-03-20T09:26:44.150 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.150 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e375f83:::benchmark_data_smithi122_168655_object108521:head# 2024-03-20T09:26:44.150 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.150 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e378bc6:::benchmark_data_smithi122_168655_object28839:head# 2024-03-20T09:26:44.150 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.150 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e37a622:::benchmark_data_smithi122_168655_object78321:head# 2024-03-20T09:26:44.150 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.150 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e39bb03:::benchmark_data_smithi122_168655_object42055:head# 2024-03-20T09:26:44.150 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.150 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e3c130b:::benchmark_data_smithi122_168655_object48828:head# 2024-03-20T09:26:44.150 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.150 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e3d2705:::benchmark_data_smithi122_168655_object36339:head# 2024-03-20T09:26:44.151 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.151 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e3da74c:::benchmark_data_smithi122_168655_object112937:head# 2024-03-20T09:26:44.151 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.151 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e3e1419:::benchmark_data_smithi122_168655_object31673:head# 2024-03-20T09:26:44.151 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.151 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e3e156c:::benchmark_data_smithi122_168655_object91537:head# 2024-03-20T09:26:44.151 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.151 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e3eaa3d:::benchmark_data_smithi122_168655_object82046:head# 2024-03-20T09:26:44.151 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.151 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e4016b8:::benchmark_data_smithi122_168655_object107999:head# 2024-03-20T09:26:44.151 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.151 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e407575:::benchmark_data_smithi122_168655_object91341:head# 2024-03-20T09:26:44.151 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.151 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e40a37d:::benchmark_data_smithi122_168655_object52591:head# 2024-03-20T09:26:44.152 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.152 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e4120c8:::benchmark_data_smithi122_168655_object45152:head# 2024-03-20T09:26:44.152 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.152 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e41afc1:::benchmark_data_smithi122_168655_object16478:head# 2024-03-20T09:26:44.152 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.152 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e424021:::benchmark_data_smithi122_168655_object61844:head# 2024-03-20T09:26:44.152 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.152 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e42947b:::benchmark_data_smithi122_168655_object18393:head# 2024-03-20T09:26:44.152 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.152 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e434a4b:::benchmark_data_smithi122_168655_object45870:head# 2024-03-20T09:26:44.152 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.152 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e43e156:::benchmark_data_smithi122_168655_object102807:head# 2024-03-20T09:26:44.152 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.152 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e44c592:::benchmark_data_smithi122_168655_object107765:head# 2024-03-20T09:26:44.153 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.153 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e45a4a1:::benchmark_data_smithi122_168655_object32182:head# 2024-03-20T09:26:44.153 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.153 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e45afef:::benchmark_data_smithi122_168655_object38220:head# 2024-03-20T09:26:44.153 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.153 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e45fd62:::benchmark_data_smithi122_168655_object54082:head# 2024-03-20T09:26:44.153 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.153 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e46bacb:::benchmark_data_smithi122_168655_object96470:head# 2024-03-20T09:26:44.153 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.153 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e46c73b:::benchmark_data_smithi122_168655_object77301:head# 2024-03-20T09:26:44.153 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.153 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e4719be:::benchmark_data_smithi122_168655_object106215:head# 2024-03-20T09:26:44.153 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.154 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e47ff20:::benchmark_data_smithi122_168655_object51865:head# 2024-03-20T09:26:44.154 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.154 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e48bae1:::benchmark_data_smithi122_168655_object88010:head# 2024-03-20T09:26:44.154 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.154 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e4951bb:::benchmark_data_smithi122_168655_object88373:head# 2024-03-20T09:26:44.154 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.154 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e4a6b6f:::benchmark_data_smithi122_168655_object46322:head# 2024-03-20T09:26:44.154 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.154 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e4b1e5c:::benchmark_data_smithi122_168655_object76206:head# 2024-03-20T09:26:44.154 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.154 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e4ba4ed:::benchmark_data_smithi122_168655_object26361:head# 2024-03-20T09:26:44.154 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.154 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e4c843c:::benchmark_data_smithi122_168655_object69539:head# 2024-03-20T09:26:44.155 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.155 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e4d9772:::benchmark_data_smithi122_168655_object107281:head# 2024-03-20T09:26:44.155 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.155 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e4f8dbf:::benchmark_data_smithi122_168655_object27539:head# 2024-03-20T09:26:44.155 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.155 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e4fbd5d:::benchmark_data_smithi122_168655_object22238:head# 2024-03-20T09:26:44.155 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.155 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e4fcd06:::benchmark_data_smithi122_168655_object105673:head# 2024-03-20T09:26:44.155 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.155 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e50c2fb:::benchmark_data_smithi122_168655_object60254:head# 2024-03-20T09:26:44.155 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.155 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e5115a0:::benchmark_data_smithi122_168655_object23989:head# 2024-03-20T09:26:44.155 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.156 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e511ff5:::benchmark_data_smithi122_168655_object90441:head# 2024-03-20T09:26:44.156 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.156 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e515ef5:::benchmark_data_smithi122_168655_object45573:head# 2024-03-20T09:26:44.156 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.156 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e52121d:::benchmark_data_smithi122_168655_object90188:head# 2024-03-20T09:26:44.156 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.156 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e521fbc:::benchmark_data_smithi122_168655_object27657:head# 2024-03-20T09:26:44.156 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.156 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e527fb4:::benchmark_data_smithi122_168655_object113945:head# 2024-03-20T09:26:44.156 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.156 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e54a568:::benchmark_data_smithi122_168655_object67677:head# 2024-03-20T09:26:44.156 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.156 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e556641:::benchmark_data_smithi122_168655_object44431:head# 2024-03-20T09:26:44.156 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.157 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e55c67d:::benchmark_data_smithi122_168655_object27923:head# 2024-03-20T09:26:44.157 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.157 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e55ff8f:::benchmark_data_smithi122_168655_object109071:head# 2024-03-20T09:26:44.157 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.157 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e55ffe7:::benchmark_data_smithi122_168655_object22787:head# 2024-03-20T09:26:44.157 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.157 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e562e22:::benchmark_data_smithi122_168655_object58122:head# 2024-03-20T09:26:44.157 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.157 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e5691a6:::benchmark_data_smithi122_168655_object49412:head# 2024-03-20T09:26:44.157 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.157 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e5691c0:::benchmark_data_smithi122_168655_object31101:head# 2024-03-20T09:26:44.157 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.158 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e56aa66:::benchmark_data_smithi122_168655_object86362:head# 2024-03-20T09:26:44.158 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.158 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e574726:::benchmark_data_smithi122_168655_object29040:head# 2024-03-20T09:26:44.158 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.158 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e5922b6:::benchmark_data_smithi122_168655_object66506:head# 2024-03-20T09:26:44.158 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.158 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e59a634:::benchmark_data_smithi122_168655_object50980:head# 2024-03-20T09:26:44.158 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.158 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e5a26d5:::benchmark_data_smithi122_168655_object13379:head# 2024-03-20T09:26:44.158 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.158 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e5b5d19:::benchmark_data_smithi122_168655_object64818:head# 2024-03-20T09:26:44.158 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.158 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e5c3ac6:::benchmark_data_smithi122_168655_object107287:head# 2024-03-20T09:26:44.159 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.159 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e5e0d04:::benchmark_data_smithi122_168655_object118605:head# 2024-03-20T09:26:44.159 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.159 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e5f9171:::benchmark_data_smithi122_168655_object89043:head# 2024-03-20T09:26:44.159 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.159 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e600ca4:::benchmark_data_smithi122_168655_object102442:head# 2024-03-20T09:26:44.159 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.159 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e602c3a:::benchmark_data_smithi122_168655_object78682:head# 2024-03-20T09:26:44.159 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.159 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e60ca72:::benchmark_data_smithi122_168655_object102331:head# 2024-03-20T09:26:44.159 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.159 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e60f5a4:::benchmark_data_smithi122_168655_object109302:head# 2024-03-20T09:26:44.159 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.159 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e62ed15:::benchmark_data_smithi122_168655_object15792:head# 2024-03-20T09:26:44.160 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.160 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e632d7c:::benchmark_data_smithi122_168655_object81942:head# 2024-03-20T09:26:44.160 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.160 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e633509:::benchmark_data_smithi122_168655_object40649:head# 2024-03-20T09:26:44.160 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.160 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e63a709:::benchmark_data_smithi122_168655_object116086:head# 2024-03-20T09:26:44.160 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.160 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e64055e:::benchmark_data_smithi122_168655_object18822:head# 2024-03-20T09:26:44.160 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.160 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e642a95:::benchmark_data_smithi122_168655_object102348:head# 2024-03-20T09:26:44.160 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.160 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e645a28:::benchmark_data_smithi122_168655_object35539:head# 2024-03-20T09:26:44.160 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.161 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e646d74:::benchmark_data_smithi122_168655_object12966:head# 2024-03-20T09:26:44.161 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.161 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e650730:::benchmark_data_smithi122_168655_object116554:head# 2024-03-20T09:26:44.161 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.161 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e655a3a:::benchmark_data_smithi122_168655_object30083:head# 2024-03-20T09:26:44.161 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.161 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e660dc1:::benchmark_data_smithi122_168655_object14649:head# 2024-03-20T09:26:44.161 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.161 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e66947c:::benchmark_data_smithi122_168655_object116485:head# 2024-03-20T09:26:44.161 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.161 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e675509:::benchmark_data_smithi122_168655_object44060:head# 2024-03-20T09:26:44.161 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.161 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e6776a1:::benchmark_data_smithi122_168655_object37164:head# 2024-03-20T09:26:44.161 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.162 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e67cc04:::benchmark_data_smithi122_168655_object47772:head# 2024-03-20T09:26:44.162 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.162 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e680381:::benchmark_data_smithi122_168655_object66360:head# 2024-03-20T09:26:44.162 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.162 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e686409:::benchmark_data_smithi122_168655_object17129:head# 2024-03-20T09:26:44.162 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.162 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e69ef50:::benchmark_data_smithi122_168655_object98521:head# 2024-03-20T09:26:44.162 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.162 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e6ad715:::benchmark_data_smithi122_168655_object105791:head# 2024-03-20T09:26:44.162 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.162 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e6c9390:::benchmark_data_smithi122_168655_object90920:head# 2024-03-20T09:26:44.162 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.162 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e6cbd39:::benchmark_data_smithi122_168655_object33141:head# 2024-03-20T09:26:44.163 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.163 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e6d3a4e:::benchmark_data_smithi122_168655_object115856:head# 2024-03-20T09:26:44.163 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.163 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e6d686f:::benchmark_data_smithi122_168655_object51462:head# 2024-03-20T09:26:44.163 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.163 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e6dd5b8:::benchmark_data_smithi122_168655_object45241:head# 2024-03-20T09:26:44.163 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.163 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e6e11fa:::benchmark_data_smithi122_168655_object73922:head# 2024-03-20T09:26:44.163 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.163 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e6e9a06:::benchmark_data_smithi122_168655_object108793:head# 2024-03-20T09:26:44.163 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.163 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e713486:::benchmark_data_smithi122_168655_object55371:head# 2024-03-20T09:26:44.163 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.164 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e717820:::benchmark_data_smithi122_168655_object46803:head# 2024-03-20T09:26:44.164 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.164 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e728248:::benchmark_data_smithi122_168655_object87086:head# 2024-03-20T09:26:44.164 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.164 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e741134:::benchmark_data_smithi122_168655_object48619:head# 2024-03-20T09:26:44.164 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.164 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e741cc1:::benchmark_data_smithi122_168655_object26320:head# 2024-03-20T09:26:44.164 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.164 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e744343:::benchmark_data_smithi122_168655_object24551:head# 2024-03-20T09:26:44.164 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.164 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e74b654:::benchmark_data_smithi122_168655_object11531:head# 2024-03-20T09:26:44.164 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.164 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e74cdf0:::benchmark_data_smithi122_168655_object61626:head# 2024-03-20T09:26:44.164 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.165 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e7584d6:::benchmark_data_smithi122_168655_object87416:head# 2024-03-20T09:26:44.165 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.165 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e77564a:::benchmark_data_smithi122_168655_object74548:head# 2024-03-20T09:26:44.165 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.165 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e778fc0:::benchmark_data_smithi122_168655_object35536:head# 2024-03-20T09:26:44.165 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.165 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e78209a:::benchmark_data_smithi122_168655_object55567:head# 2024-03-20T09:26:44.165 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.165 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e783580:::benchmark_data_smithi122_168655_object75605:head# 2024-03-20T09:26:44.165 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.165 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e78e74d:::benchmark_data_smithi122_168655_object43594:head# 2024-03-20T09:26:44.165 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.165 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e7a1796:::benchmark_data_smithi122_168655_object17214:head# 2024-03-20T09:26:44.165 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.166 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e7c1ccc:::benchmark_data_smithi122_168655_object88354:head# 2024-03-20T09:26:44.166 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.166 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e7c4b39:::benchmark_data_smithi122_168655_object66021:head# 2024-03-20T09:26:44.166 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.166 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e7c800b:::benchmark_data_smithi122_168655_object95447:head# 2024-03-20T09:26:44.166 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.166 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e80ed29:::benchmark_data_smithi122_168655_object84788:head# 2024-03-20T09:26:44.166 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.166 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e80fc0e:::benchmark_data_smithi122_168655_object56095:head# 2024-03-20T09:26:44.166 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.166 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e812a6c:::benchmark_data_smithi122_168655_object96184:head# 2024-03-20T09:26:44.166 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.167 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e8187d0:::benchmark_data_smithi122_168655_object51703:head# 2024-03-20T09:26:44.167 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.167 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e8188a0:::benchmark_data_smithi122_168655_object47348:head# 2024-03-20T09:26:44.167 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.167 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e818b00:::benchmark_data_smithi122_168655_object33113:head# 2024-03-20T09:26:44.167 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.167 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e823b89:::benchmark_data_smithi122_168655_object34279:head# 2024-03-20T09:26:44.167 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.167 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e825480:::benchmark_data_smithi122_168655_object88493:head# 2024-03-20T09:26:44.167 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.167 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e829df0:::benchmark_data_smithi122_168655_object101046:head# 2024-03-20T09:26:44.167 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.167 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e82adca:::benchmark_data_smithi122_168655_object44901:head# 2024-03-20T09:26:44.167 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.168 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e82f25c:::benchmark_data_smithi122_168655_object32773:head# 2024-03-20T09:26:44.168 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.168 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e82f72a:::benchmark_data_smithi122_168655_object57734:head# 2024-03-20T09:26:44.168 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.168 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e832c1e:::benchmark_data_smithi122_168655_object19318:head# 2024-03-20T09:26:44.168 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.168 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e83754e:::benchmark_data_smithi122_168655_object107876:head# 2024-03-20T09:26:44.168 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.168 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e8453dc:::benchmark_data_smithi122_168655_object35070:head# 2024-03-20T09:26:44.168 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.168 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e84ccdf:::benchmark_data_smithi122_168655_object46604:head# 2024-03-20T09:26:44.168 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.168 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e84d4bc:::benchmark_data_smithi122_168655_object104364:head# 2024-03-20T09:26:44.168 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.169 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e860904:::benchmark_data_smithi122_168655_object87635:head# 2024-03-20T09:26:44.169 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.169 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e873a76:::benchmark_data_smithi122_168655_object65794:head# 2024-03-20T09:26:44.169 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.169 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e877506:::benchmark_data_smithi122_168655_object69345:head# 2024-03-20T09:26:44.169 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.169 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e8789ca:::benchmark_data_smithi122_168655_object34014:head# 2024-03-20T09:26:44.169 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.169 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e883ccd:::benchmark_data_smithi122_168655_object52339:head# 2024-03-20T09:26:44.169 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.169 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e8b0514:::benchmark_data_smithi122_168655_object100887:head# 2024-03-20T09:26:44.169 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.169 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e8b66ed:::benchmark_data_smithi122_168655_object117481:head# 2024-03-20T09:26:44.170 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.170 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e8bbae2:::benchmark_data_smithi122_168655_object100411:head# 2024-03-20T09:26:44.170 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.170 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e8c5a7e:::benchmark_data_smithi122_168655_object116411:head# 2024-03-20T09:26:44.170 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.170 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e8c6b65:::benchmark_data_smithi122_168655_object78199:head# 2024-03-20T09:26:44.170 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.170 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e8c7f28:::benchmark_data_smithi122_168655_object41189:head# 2024-03-20T09:26:44.170 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.170 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e8d37ff:::benchmark_data_smithi122_168655_object37896:head# 2024-03-20T09:26:44.170 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.170 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e8e36a1:::benchmark_data_smithi122_168655_object23818:head# 2024-03-20T09:26:44.170 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.170 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e8f1179:::benchmark_data_smithi122_168655_object66766:head# 2024-03-20T09:26:44.171 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.171 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e9091e2:::benchmark_data_smithi122_168655_object33111:head# 2024-03-20T09:26:44.171 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.171 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e909e55:::benchmark_data_smithi122_168655_object54794:head# 2024-03-20T09:26:44.171 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.171 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e91215b:::benchmark_data_smithi122_168655_object48069:head# 2024-03-20T09:26:44.171 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.171 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e923a5f:::benchmark_data_smithi122_168655_object107937:head# 2024-03-20T09:26:44.171 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.171 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e93a020:::benchmark_data_smithi122_168655_object15622:head# 2024-03-20T09:26:44.171 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.171 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e95816a:::benchmark_data_smithi122_168655_object95379:head# 2024-03-20T09:26:44.171 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.171 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e962638:::benchmark_data_smithi122_168655_object93363:head# 2024-03-20T09:26:44.172 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.172 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e963a3c:::benchmark_data_smithi122_168655_object36021:head# 2024-03-20T09:26:44.172 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.172 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e9745c0:::benchmark_data_smithi122_168655_object78646:head# 2024-03-20T09:26:44.172 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.172 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e97ed29:::benchmark_data_smithi122_168655_object53668:head# 2024-03-20T09:26:44.172 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.172 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e9816a5:::benchmark_data_smithi122_168655_object114939:head# 2024-03-20T09:26:44.172 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.172 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e99da5d:::benchmark_data_smithi122_168655_object47124:head# 2024-03-20T09:26:44.172 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.172 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e9aa5b8:::benchmark_data_smithi122_168655_object21294:head# 2024-03-20T09:26:44.172 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.172 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e9ac70c:::benchmark_data_smithi122_168655_object63801:head# 2024-03-20T09:26:44.173 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.173 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e9d289e:::benchmark_data_smithi122_168655_object102754:head# 2024-03-20T09:26:44.173 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.173 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e9dcbc4:::benchmark_data_smithi122_168655_object21409:head# 2024-03-20T09:26:44.173 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.173 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e9e4f35:::benchmark_data_smithi122_168655_object48171:head# 2024-03-20T09:26:44.173 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.173 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e9f4ea7:::benchmark_data_smithi122_168655_object104581:head# 2024-03-20T09:26:44.173 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.173 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e9f6d12:::benchmark_data_smithi122_168655_object28817:head# 2024-03-20T09:26:44.173 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.173 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8e9f71b7:::benchmark_data_smithi122_168655_object98831:head# 2024-03-20T09:26:44.173 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.174 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ea07f8b:::benchmark_data_smithi122_168655_object116039:head# 2024-03-20T09:26:44.174 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.174 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ea210a9:::benchmark_data_smithi122_168655_object54131:head# 2024-03-20T09:26:44.174 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.174 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ea27404:::benchmark_data_smithi122_168655_object91465:head# 2024-03-20T09:26:44.174 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.174 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ea2bfe6:::benchmark_data_smithi122_168655_object99460:head# 2024-03-20T09:26:44.174 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.174 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ea2d160:::benchmark_data_smithi122_168655_object114991:head# 2024-03-20T09:26:44.174 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.174 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ea37ff1:::benchmark_data_smithi122_168655_object55031:head# 2024-03-20T09:26:44.174 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.174 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ea43da5:::benchmark_data_smithi122_168655_object117034:head# 2024-03-20T09:26:44.175 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.175 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ea4da1a:::benchmark_data_smithi122_168655_object71893:head# 2024-03-20T09:26:44.175 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.175 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ea613c0:::benchmark_data_smithi122_168655_object96293:head# 2024-03-20T09:26:44.175 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.175 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ea6270c:::benchmark_data_smithi122_168655_object47803:head# 2024-03-20T09:26:44.175 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.175 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ea77141:::benchmark_data_smithi122_168655_object31719:head# 2024-03-20T09:26:44.175 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.175 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ea859fd:::benchmark_data_smithi122_168655_object92958:head# 2024-03-20T09:26:44.175 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.175 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ea89489:::benchmark_data_smithi122_168655_object37626:head# 2024-03-20T09:26:44.175 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.175 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ea8ba6d:::benchmark_data_smithi122_168655_object87009:head# 2024-03-20T09:26:44.176 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.176 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ea9aeac:::benchmark_data_smithi122_168655_object86179:head# 2024-03-20T09:26:44.176 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.176 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eab9bdb:::benchmark_data_smithi122_168655_object64898:head# 2024-03-20T09:26:44.176 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.176 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eac9224:::benchmark_data_smithi122_168655_object61362:head# 2024-03-20T09:26:44.176 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.176 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ead89ba:::benchmark_data_smithi122_168655_object19954:head# 2024-03-20T09:26:44.176 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.176 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eadca23:::benchmark_data_smithi122_168655_object44941:head# 2024-03-20T09:26:44.176 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.176 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eaecf48:::benchmark_data_smithi122_168655_object49725:head# 2024-03-20T09:26:44.176 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.176 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eaef181:::benchmark_data_smithi122_168655_object101709:head# 2024-03-20T09:26:44.177 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.177 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eaf06c0:::benchmark_data_smithi122_168655_object118809:head# 2024-03-20T09:26:44.177 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.177 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eafb14e:::benchmark_data_smithi122_168655_object115875:head# 2024-03-20T09:26:44.177 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.177 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eafd6ab:::benchmark_data_smithi122_168655_object21447:head# 2024-03-20T09:26:44.177 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.177 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eb003c9:::benchmark_data_smithi122_168655_object75178:head# 2024-03-20T09:26:44.177 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.177 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eb0246e:::benchmark_data_smithi122_168655_object42672:head# 2024-03-20T09:26:44.177 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.177 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eb08819:::benchmark_data_smithi122_168655_object53683:head# 2024-03-20T09:26:44.177 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.177 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eb09a56:::benchmark_data_smithi122_168655_object39929:head# 2024-03-20T09:26:44.177 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.178 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eb0c938:::benchmark_data_smithi122_168655_object52572:head# 2024-03-20T09:26:44.178 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.178 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eb16924:::benchmark_data_smithi122_168655_object104743:head# 2024-03-20T09:26:44.178 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.178 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eb16abb:::benchmark_data_smithi122_168655_object115446:head# 2024-03-20T09:26:44.178 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.178 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eb1d9f6:::benchmark_data_smithi122_168655_object61040:head# 2024-03-20T09:26:44.178 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.178 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eb1e2e6:::benchmark_data_smithi122_168655_object115186:head# 2024-03-20T09:26:44.178 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.178 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eb1ed73:::benchmark_data_smithi122_168655_object28344:head# 2024-03-20T09:26:44.178 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.179 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eb34391:::benchmark_data_smithi122_168655_object69378:head# 2024-03-20T09:26:44.179 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.179 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eb37d12:::benchmark_data_smithi122_168655_object58203:head# 2024-03-20T09:26:44.179 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.179 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eb3b183:::benchmark_data_smithi122_168655_object54408:head# 2024-03-20T09:26:44.179 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.179 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eb4b39b:::benchmark_data_smithi122_168655_object69091:head# 2024-03-20T09:26:44.179 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.179 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eb52f85:::benchmark_data_smithi122_168655_object44392:head# 2024-03-20T09:26:44.179 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.179 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eb54f74:::benchmark_data_smithi122_168655_object58376:head# 2024-03-20T09:26:44.179 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.179 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eb55e32:::benchmark_data_smithi122_168655_object33962:head# 2024-03-20T09:26:44.179 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.179 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eb588cc:::benchmark_data_smithi122_168655_object93933:head# 2024-03-20T09:26:44.180 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.180 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eb5b728:::benchmark_data_smithi122_168655_object37403:head# 2024-03-20T09:26:44.180 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.180 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eb613c3:::benchmark_data_smithi122_168655_object85350:head# 2024-03-20T09:26:44.180 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.180 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eb62ec8:::benchmark_data_smithi122_168655_object13038:head# 2024-03-20T09:26:44.180 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.180 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eb67f0e:::benchmark_data_smithi122_168655_object71906:head# 2024-03-20T09:26:44.180 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.180 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eb6a7fd:::benchmark_data_smithi122_168655_object17559:head# 2024-03-20T09:26:44.180 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.180 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eb7644f:::benchmark_data_smithi122_168655_object83007:head# 2024-03-20T09:26:44.180 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.181 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eb78e33:::benchmark_data_smithi122_168655_object82323:head# 2024-03-20T09:26:44.181 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.181 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eb834d2:::benchmark_data_smithi122_168655_object73350:head# 2024-03-20T09:26:44.181 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.181 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eb84d1b:::benchmark_data_smithi122_168655_object27649:head# 2024-03-20T09:26:44.181 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.181 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eb8670e:::benchmark_data_smithi122_168655_object107664:head# 2024-03-20T09:26:44.181 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.181 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eb916bb:::benchmark_data_smithi122_168655_object106013:head# 2024-03-20T09:26:44.181 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.181 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eb97bd5:::benchmark_data_smithi122_168655_object63180:head# 2024-03-20T09:26:44.181 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.181 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ebb47ee:::benchmark_data_smithi122_168655_object63017:head# 2024-03-20T09:26:44.181 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.181 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ebc575d:::benchmark_data_smithi122_168655_object20215:head# 2024-03-20T09:26:44.182 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.182 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ebdba66:::benchmark_data_smithi122_168655_object39874:head# 2024-03-20T09:26:44.182 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.182 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ebdd194:::benchmark_data_smithi122_168655_object29625:head# 2024-03-20T09:26:44.182 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.182 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ebf263f:::benchmark_data_smithi122_168655_object46634:head# 2024-03-20T09:26:44.182 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.182 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ebfc5b3:::benchmark_data_smithi122_168655_object103869:head# 2024-03-20T09:26:44.182 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.182 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ec1746b:::benchmark_data_smithi122_168655_object41963:head# 2024-03-20T09:26:44.182 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.182 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ec1b068:::benchmark_data_smithi122_168655_object107102:head# 2024-03-20T09:26:44.182 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.183 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ec1c5c4:::benchmark_data_smithi122_168655_object87867:head# 2024-03-20T09:26:44.183 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.183 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ec26745:::benchmark_data_smithi122_168655_object76401:head# 2024-03-20T09:26:44.183 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.183 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ec27664:::benchmark_data_smithi122_168655_object68073:head# 2024-03-20T09:26:44.183 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.183 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ec2c60e:::benchmark_data_smithi122_168655_object34387:head# 2024-03-20T09:26:44.183 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.183 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ec2eac8:::benchmark_data_smithi122_168655_object77507:head# 2024-03-20T09:26:44.183 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.183 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ec31bbe:::benchmark_data_smithi122_168655_object48408:head# 2024-03-20T09:26:44.183 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.183 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ec3c303:::benchmark_data_smithi122_168655_object59599:head# 2024-03-20T09:26:44.183 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.184 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ec3d4aa:::benchmark_data_smithi122_168655_object99871:head# 2024-03-20T09:26:44.184 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.184 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ec412c9:::benchmark_data_smithi122_168655_object93885:head# 2024-03-20T09:26:44.184 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.184 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ec41f61:::benchmark_data_smithi122_168655_object100098:head# 2024-03-20T09:26:44.184 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.184 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ec44cf3:::benchmark_data_smithi122_168655_object28584:head# 2024-03-20T09:26:44.184 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.184 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ec49c1f:::benchmark_data_smithi122_168655_object48372:head# 2024-03-20T09:26:44.184 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.184 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ec57abb:::benchmark_data_smithi122_168655_object27271:head# 2024-03-20T09:26:44.184 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.184 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ec60f46:::benchmark_data_smithi122_168655_object107553:head# 2024-03-20T09:26:44.184 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.185 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ec62eb5:::benchmark_data_smithi122_168655_object18792:head# 2024-03-20T09:26:44.185 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.185 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ec6b5d6:::benchmark_data_smithi122_168655_object117875:head# 2024-03-20T09:26:44.185 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.185 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ec73efc:::benchmark_data_smithi122_168655_object39701:head# 2024-03-20T09:26:44.185 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.185 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ec7f964:::benchmark_data_smithi122_168655_object75770:head# 2024-03-20T09:26:44.185 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.185 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ec885e7:::benchmark_data_smithi122_168655_object88892:head# 2024-03-20T09:26:44.185 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.185 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eca2340:::benchmark_data_smithi122_168655_object42035:head# 2024-03-20T09:26:44.185 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.185 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eca5fe3:::benchmark_data_smithi122_168655_object93546:head# 2024-03-20T09:26:44.186 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.186 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ecb353c:::benchmark_data_smithi122_168655_object59422:head# 2024-03-20T09:26:44.186 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.186 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ecc0b54:::benchmark_data_smithi122_168655_object61533:head# 2024-03-20T09:26:44.186 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.186 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ecceb19:::benchmark_data_smithi122_168655_object59040:head# 2024-03-20T09:26:44.186 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.186 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eccfb86:::benchmark_data_smithi122_168655_object47536:head# 2024-03-20T09:26:44.186 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.186 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ecda99e:::benchmark_data_smithi122_168655_object108619:head# 2024-03-20T09:26:44.186 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.186 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ece02a0:::benchmark_data_smithi122_168655_object22763:head# 2024-03-20T09:26:44.186 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.186 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ece2680:::benchmark_data_smithi122_168655_object58490:head# 2024-03-20T09:26:44.187 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.187 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ecf4eec:::benchmark_data_smithi122_168655_object100693:head# 2024-03-20T09:26:44.187 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.187 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ecf9cee:::benchmark_data_smithi122_168655_object91954:head# 2024-03-20T09:26:44.187 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.187 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ecfc009:::benchmark_data_smithi122_168655_object51680:head# 2024-03-20T09:26:44.187 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.187 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ed0a9bf:::benchmark_data_smithi122_168655_object65901:head# 2024-03-20T09:26:44.187 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.187 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ed1259c:::benchmark_data_smithi122_168655_object93357:head# 2024-03-20T09:26:44.187 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.187 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ed315ff:::benchmark_data_smithi122_168655_object103046:head# 2024-03-20T09:26:44.187 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.188 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ed317de:::benchmark_data_smithi122_168655_object94482:head# 2024-03-20T09:26:44.188 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.188 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ed3ac17:::benchmark_data_smithi122_168655_object53607:head# 2024-03-20T09:26:44.188 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.188 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ed3bcea:::benchmark_data_smithi122_168655_object56125:head# 2024-03-20T09:26:44.188 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.188 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ed41199:::benchmark_data_smithi122_168655_object20806:head# 2024-03-20T09:26:44.188 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.188 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ed437b0:::benchmark_data_smithi122_168655_object31922:head# 2024-03-20T09:26:44.188 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.188 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ed519f5:::benchmark_data_smithi122_168655_object81715:head# 2024-03-20T09:26:44.188 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.188 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ed54f62:::benchmark_data_smithi122_168655_object88733:head# 2024-03-20T09:26:44.188 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.189 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ed607d2:::benchmark_data_smithi122_168655_object43315:head# 2024-03-20T09:26:44.189 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.189 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ed82ba4:::benchmark_data_smithi122_168655_object58375:head# 2024-03-20T09:26:44.189 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.189 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ed96b36:::benchmark_data_smithi122_168655_object24013:head# 2024-03-20T09:26:44.189 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.189 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8edab094:::benchmark_data_smithi122_168655_object40897:head# 2024-03-20T09:26:44.189 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.189 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8edb6475:::benchmark_data_smithi122_168655_object111582:head# 2024-03-20T09:26:44.189 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.189 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8edbaa2b:::benchmark_data_smithi122_168655_object40305:head# 2024-03-20T09:26:44.189 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.189 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8edc13ef:::benchmark_data_smithi122_168655_object70669:head# 2024-03-20T09:26:44.190 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.190 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8edc512f:::benchmark_data_smithi122_168655_object30654:head# 2024-03-20T09:26:44.190 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.190 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8edcb4b4:::benchmark_data_smithi122_168655_object35017:head# 2024-03-20T09:26:44.190 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.190 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8edd1665:::benchmark_data_smithi122_168655_object47644:head# 2024-03-20T09:26:44.190 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.190 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8edd2fb7:::benchmark_data_smithi122_168655_object101534:head# 2024-03-20T09:26:44.190 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.190 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8edfe336:::benchmark_data_smithi122_168655_object91939:head# 2024-03-20T09:26:44.190 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.190 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8edfe97f:::benchmark_data_smithi122_168655_object104433:head# 2024-03-20T09:26:44.190 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.191 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ee0465b:::benchmark_data_smithi122_168655_object94420:head# 2024-03-20T09:26:44.191 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.191 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ee140ad:::benchmark_data_smithi122_168655_object113811:head# 2024-03-20T09:26:44.191 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.191 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ee22e1a:::benchmark_data_smithi122_168655_object118336:head# 2024-03-20T09:26:44.191 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.191 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ee30db9:::benchmark_data_smithi122_168655_object40190:head# 2024-03-20T09:26:44.191 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.191 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ee31da8:::benchmark_data_smithi122_168655_object10739:head# 2024-03-20T09:26:44.191 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.191 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ee407cc:::benchmark_data_smithi122_168655_object30092:head# 2024-03-20T09:26:44.191 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.191 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ee4594a:::benchmark_data_smithi122_168655_object11485:head# 2024-03-20T09:26:44.192 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.192 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ee53d53:::benchmark_data_smithi122_168655_object59256:head# 2024-03-20T09:26:44.192 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.192 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ee608c2:::benchmark_data_smithi122_168655_object33311:head# 2024-03-20T09:26:44.192 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.192 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ee63f8e:::benchmark_data_smithi122_168655_object20843:head# 2024-03-20T09:26:44.192 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.192 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ee6dbac:::benchmark_data_smithi122_168655_object99199:head# 2024-03-20T09:26:44.192 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.192 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ee6f082:::benchmark_data_smithi122_168655_object20922:head# 2024-03-20T09:26:44.192 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.192 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ee81789:::benchmark_data_smithi122_168655_object67572:head# 2024-03-20T09:26:44.192 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.192 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eea6576:::benchmark_data_smithi122_168655_object13724:head# 2024-03-20T09:26:44.193 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.193 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eeaefba:::benchmark_data_smithi122_168655_object110619:head# 2024-03-20T09:26:44.193 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.193 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eec4501:::benchmark_data_smithi122_168655_object85215:head# 2024-03-20T09:26:44.193 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.193 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eec45ad:::benchmark_data_smithi122_168655_object118669:head# 2024-03-20T09:26:44.193 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.193 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eec7d00:::benchmark_data_smithi122_168655_object79369:head# 2024-03-20T09:26:44.193 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.193 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eec8257:::benchmark_data_smithi122_168655_object82429:head# 2024-03-20T09:26:44.193 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.193 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eee06dc:::benchmark_data_smithi122_168655_object74575:head# 2024-03-20T09:26:44.193 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.194 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eeec582:::benchmark_data_smithi122_168655_object61133:head# 2024-03-20T09:26:44.194 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.194 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eef11e9:::benchmark_data_smithi122_168655_object108511:head# 2024-03-20T09:26:44.194 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.194 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eef8515:::benchmark_data_smithi122_168655_object68745:head# 2024-03-20T09:26:44.194 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.194 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eefd506:::benchmark_data_smithi122_168655_object83383:head# 2024-03-20T09:26:44.194 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.194 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ef03355:::benchmark_data_smithi122_168655_object52855:head# 2024-03-20T09:26:44.194 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.194 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ef082c3:::benchmark_data_smithi122_168655_object72879:head# 2024-03-20T09:26:44.194 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.194 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ef0afa5:::benchmark_data_smithi122_168655_object97425:head# 2024-03-20T09:26:44.194 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.195 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ef0b5fa:::benchmark_data_smithi122_168655_object87814:head# 2024-03-20T09:26:44.195 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.195 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ef17a6d:::benchmark_data_smithi122_168655_object53421:head# 2024-03-20T09:26:44.195 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.195 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ef3ba60:::benchmark_data_smithi122_168655_object109687:head# 2024-03-20T09:26:44.195 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.195 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ef3bc9b:::benchmark_data_smithi122_168655_object56019:head# 2024-03-20T09:26:44.195 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.195 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ef40982:::benchmark_data_smithi122_168655_object87093:head# 2024-03-20T09:26:44.195 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.195 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ef514da:::benchmark_data_smithi122_168655_object109734:head# 2024-03-20T09:26:44.195 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.195 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ef6e2f1:::benchmark_data_smithi122_168655_object41667:head# 2024-03-20T09:26:44.196 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.196 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ef716ed:::benchmark_data_smithi122_168655_object76103:head# 2024-03-20T09:26:44.196 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.196 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ef8c525:::benchmark_data_smithi122_168655_object35587:head# 2024-03-20T09:26:44.196 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.196 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ef92e39:::benchmark_data_smithi122_168655_object36887:head# 2024-03-20T09:26:44.196 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.196 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ef9e42c:::benchmark_data_smithi122_168655_object84714:head# 2024-03-20T09:26:44.196 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.196 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8efa9757:::benchmark_data_smithi122_168655_object60772:head# 2024-03-20T09:26:44.196 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.196 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8efadf45:::benchmark_data_smithi122_168655_object48597:head# 2024-03-20T09:26:44.196 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.196 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8efae560:::benchmark_data_smithi122_168655_object45605:head# 2024-03-20T09:26:44.197 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.197 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8efb7200:::benchmark_data_smithi122_168655_object115849:head# 2024-03-20T09:26:44.197 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.197 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8efc19c6:::benchmark_data_smithi122_168655_object33590:head# 2024-03-20T09:26:44.197 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.197 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8efeb281:::benchmark_data_smithi122_168655_object17177:head# 2024-03-20T09:26:44.197 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.197 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8efef360:::benchmark_data_smithi122_168655_object23888:head# 2024-03-20T09:26:44.197 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.197 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8efef72b:::benchmark_data_smithi122_168655_object41395:head# 2024-03-20T09:26:44.197 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.197 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8eff34ab:::benchmark_data_smithi122_168655_object50906:head# 2024-03-20T09:26:44.197 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.197 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8effcbee:::benchmark_data_smithi122_168655_object19871:head# 2024-03-20T09:26:44.198 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.198 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f01429a:::benchmark_data_smithi122_168655_object114022:head# 2024-03-20T09:26:44.198 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.198 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f024041:::benchmark_data_smithi122_168655_object50762:head# 2024-03-20T09:26:44.198 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.198 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f0243ad:::benchmark_data_smithi122_168655_object21982:head# 2024-03-20T09:26:44.198 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.198 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f02b20c:::benchmark_data_smithi122_168655_object119814:head# 2024-03-20T09:26:44.198 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.198 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f02c6a5:::benchmark_data_smithi122_168655_object104191:head# 2024-03-20T09:26:44.198 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.198 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f03284f:::benchmark_data_smithi122_168655_object77746:head# 2024-03-20T09:26:44.198 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.199 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f037128:::benchmark_data_smithi122_168655_object19751:head# 2024-03-20T09:26:44.199 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.199 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f056c7b:::benchmark_data_smithi122_168655_object112640:head# 2024-03-20T09:26:44.199 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.199 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f05de6c:::benchmark_data_smithi122_168655_object93649:head# 2024-03-20T09:26:44.199 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.199 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f0612e5:::benchmark_data_smithi122_168655_object94853:head# 2024-03-20T09:26:44.199 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.199 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f064763:::benchmark_data_smithi122_168655_object44640:head# 2024-03-20T09:26:44.199 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.199 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f06500e:::benchmark_data_smithi122_168655_object23965:head# 2024-03-20T09:26:44.199 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.199 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f07123c:::benchmark_data_smithi122_168655_object76200:head# 2024-03-20T09:26:44.200 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.200 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f0893ef:::benchmark_data_smithi122_168655_object40621:head# 2024-03-20T09:26:44.200 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.200 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f0b12dc:::benchmark_data_smithi122_168655_object119142:head# 2024-03-20T09:26:44.200 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.200 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f0db6f6:::benchmark_data_smithi122_168655_object118435:head# 2024-03-20T09:26:44.200 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.200 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f0f95bd:::benchmark_data_smithi122_168655_object112344:head# 2024-03-20T09:26:44.200 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.200 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f105d59:::benchmark_data_smithi122_168655_object105430:head# 2024-03-20T09:26:44.200 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.200 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f10db7b:::benchmark_data_smithi122_168655_object29218:head# 2024-03-20T09:26:44.200 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.200 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f10e84f:::benchmark_data_smithi122_168655_object16325:head# 2024-03-20T09:26:44.201 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.201 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f1104a7:::benchmark_data_smithi122_168655_object88388:head# 2024-03-20T09:26:44.201 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.201 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f112bb8:::benchmark_data_smithi122_168655_object27227:head# 2024-03-20T09:26:44.201 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.201 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f12c012:::benchmark_data_smithi122_168655_object108668:head# 2024-03-20T09:26:44.201 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.201 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f133e33:::benchmark_data_smithi122_168655_object80868:head# 2024-03-20T09:26:44.201 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.201 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f138c8b:::benchmark_data_smithi122_168655_object49342:head# 2024-03-20T09:26:44.201 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.201 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f14f2ab:::benchmark_data_smithi122_168655_object108860:head# 2024-03-20T09:26:44.201 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.201 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f1507d2:::benchmark_data_smithi122_168655_object88967:head# 2024-03-20T09:26:44.202 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.202 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f152264:::benchmark_data_smithi122_168655_object41651:head# 2024-03-20T09:26:44.202 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.202 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f152b35:::benchmark_data_smithi122_168655_object93044:head# 2024-03-20T09:26:44.202 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.202 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f15f914:::benchmark_data_smithi122_168655_object80411:head# 2024-03-20T09:26:44.202 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.202 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f1603c0:::benchmark_data_smithi122_168655_object61596:head# 2024-03-20T09:26:44.202 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.202 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f166042:::benchmark_data_smithi122_168655_object11181:head# 2024-03-20T09:26:44.202 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.202 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f174bd6:::benchmark_data_smithi122_168655_object118475:head# 2024-03-20T09:26:44.202 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.203 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f17a882:::benchmark_data_smithi122_168655_object115260:head# 2024-03-20T09:26:44.203 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.203 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f17ef75:::benchmark_data_smithi122_168655_object56911:head# 2024-03-20T09:26:44.203 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.203 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f190f5b:::benchmark_data_smithi122_168655_object31197:head# 2024-03-20T09:26:44.203 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.203 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f191361:::benchmark_data_smithi122_168655_object31288:head# 2024-03-20T09:26:44.203 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.203 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f194d94:::benchmark_data_smithi122_168655_object47322:head# 2024-03-20T09:26:44.203 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.203 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f1a5a97:::benchmark_data_smithi122_168655_object99976:head# 2024-03-20T09:26:44.203 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.203 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f1b3966:::benchmark_data_smithi122_168655_object42474:head# 2024-03-20T09:26:44.204 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.204 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f1b8452:::benchmark_data_smithi122_168655_object17507:head# 2024-03-20T09:26:44.204 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.204 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f1b88bb:::benchmark_data_smithi122_168655_object27130:head# 2024-03-20T09:26:44.204 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.204 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f1bd868:::benchmark_data_smithi122_168655_object35181:head# 2024-03-20T09:26:44.204 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.204 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f1c78ed:::benchmark_data_smithi122_168655_object88876:head# 2024-03-20T09:26:44.204 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.204 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f1cabf5:::benchmark_data_smithi122_168655_object39285:head# 2024-03-20T09:26:44.204 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.204 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f1d5733:::benchmark_data_smithi122_168655_object66141:head# 2024-03-20T09:26:44.204 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.204 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f1d9b25:::benchmark_data_smithi122_168655_object89457:head# 2024-03-20T09:26:44.205 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.205 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f1e7844:::benchmark_data_smithi122_168655_object52829:head# 2024-03-20T09:26:44.205 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.205 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f20b49a:::benchmark_data_smithi122_168655_object109842:head# 2024-03-20T09:26:44.205 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.205 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f21e835:::benchmark_data_smithi122_168655_object76133:head# 2024-03-20T09:26:44.205 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.205 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f220f6b:::benchmark_data_smithi122_168655_object13439:head# 2024-03-20T09:26:44.205 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.205 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f22dba9:::benchmark_data_smithi122_168655_object107581:head# 2024-03-20T09:26:44.205 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.205 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f23e9f5:::benchmark_data_smithi122_168655_object12318:head# 2024-03-20T09:26:44.205 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.205 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f23efa7:::benchmark_data_smithi122_168655_object95283:head# 2024-03-20T09:26:44.206 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.206 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f248dce:::benchmark_data_smithi122_168655_object60790:head# 2024-03-20T09:26:44.206 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.206 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f249aeb:::benchmark_data_smithi122_168655_object10597:head# 2024-03-20T09:26:44.206 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.206 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f254309:::benchmark_data_smithi122_168655_object67632:head# 2024-03-20T09:26:44.206 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.206 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f25d2a9:::benchmark_data_smithi122_168655_object63202:head# 2024-03-20T09:26:44.206 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.206 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f25f095:::benchmark_data_smithi122_168655_object101003:head# 2024-03-20T09:26:44.206 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.206 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f2630b1:::benchmark_data_smithi122_168655_object25262:head# 2024-03-20T09:26:44.206 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.206 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f26bd2d:::benchmark_data_smithi122_168655_object60522:head# 2024-03-20T09:26:44.207 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.207 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f26fc40:::benchmark_data_smithi122_168655_object88051:head# 2024-03-20T09:26:44.207 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.207 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f272005:::benchmark_data_smithi122_168655_object103189:head# 2024-03-20T09:26:44.207 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.207 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f272d4f:::benchmark_data_smithi122_168655_object90448:head# 2024-03-20T09:26:44.207 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.207 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f274167:::benchmark_data_smithi122_168655_object57225:head# 2024-03-20T09:26:44.207 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.207 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f279559:::benchmark_data_smithi122_168655_object26660:head# 2024-03-20T09:26:44.207 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.207 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f28093f:::benchmark_data_smithi122_168655_object33786:head# 2024-03-20T09:26:44.207 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.207 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f28c797:::benchmark_data_smithi122_168655_object19607:head# 2024-03-20T09:26:44.208 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.208 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f290300:::benchmark_data_smithi122_168655_object64947:head# 2024-03-20T09:26:44.208 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.208 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f2906da:::benchmark_data_smithi122_168655_object72562:head# 2024-03-20T09:26:44.208 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.208 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f29610b:::benchmark_data_smithi122_168655_object112780:head# 2024-03-20T09:26:44.208 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.208 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f296134:::benchmark_data_smithi122_168655_object16222:head# 2024-03-20T09:26:44.208 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.208 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f29bd6e:::benchmark_data_smithi122_168655_object35858:head# 2024-03-20T09:26:44.208 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.208 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f29ee48:::benchmark_data_smithi122_168655_object90957:head# 2024-03-20T09:26:44.208 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.209 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f2a069b:::benchmark_data_smithi122_168655_object67210:head# 2024-03-20T09:26:44.209 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.209 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f2b1561:::benchmark_data_smithi122_168655_object104568:head# 2024-03-20T09:26:44.209 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.209 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f2b1cd5:::benchmark_data_smithi122_168655_object84608:head# 2024-03-20T09:26:44.209 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.209 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f2b6c0f:::benchmark_data_smithi122_168655_object94522:head# 2024-03-20T09:26:44.209 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.209 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f2bb4bc:::benchmark_data_smithi122_168655_object50656:head# 2024-03-20T09:26:44.209 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.209 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f2c0fb4:::benchmark_data_smithi122_168655_object64753:head# 2024-03-20T09:26:44.209 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.209 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f2c9e79:::benchmark_data_smithi122_168655_object27279:head# 2024-03-20T09:26:44.210 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.210 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f2e3999:::benchmark_data_smithi122_168655_object50484:head# 2024-03-20T09:26:44.210 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.210 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f2ecfde:::benchmark_data_smithi122_168655_object22802:head# 2024-03-20T09:26:44.210 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.210 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f2fe6ed:::benchmark_data_smithi122_168655_object108749:head# 2024-03-20T09:26:44.210 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.210 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f30a862:::benchmark_data_smithi122_168655_object59884:head# 2024-03-20T09:26:44.210 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.210 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f317b7d:::benchmark_data_smithi122_168655_object117218:head# 2024-03-20T09:26:44.210 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.211 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f317db2:::benchmark_data_smithi122_168655_object54204:head# 2024-03-20T09:26:44.211 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.211 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f31afd7:::benchmark_data_smithi122_168655_object67527:head# 2024-03-20T09:26:44.211 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.211 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f31b96a:::benchmark_data_smithi122_168655_object29014:head# 2024-03-20T09:26:44.211 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.211 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f31d580:::benchmark_data_smithi122_168655_object11864:head# 2024-03-20T09:26:44.211 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.211 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f324554:::benchmark_data_smithi122_168655_object29070:head# 2024-03-20T09:26:44.211 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.211 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f333d39:::benchmark_data_smithi122_168655_object36194:head# 2024-03-20T09:26:44.211 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.211 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f339fa5:::benchmark_data_smithi122_168655_object11452:head# 2024-03-20T09:26:44.211 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.212 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f34009f:::benchmark_data_smithi122_168655_object118563:head# 2024-03-20T09:26:44.212 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.212 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f341619:::benchmark_data_smithi122_168655_object74124:head# 2024-03-20T09:26:44.212 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.212 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f34de5b:::benchmark_data_smithi122_168655_object32776:head# 2024-03-20T09:26:44.212 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.212 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f354dee:::benchmark_data_smithi122_168655_object86242:head# 2024-03-20T09:26:44.212 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.212 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f357685:::benchmark_data_smithi122_168655_object118359:head# 2024-03-20T09:26:44.212 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.212 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f35d77e:::benchmark_data_smithi122_168655_object107255:head# 2024-03-20T09:26:44.212 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.212 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f3660f9:::benchmark_data_smithi122_168655_object100105:head# 2024-03-20T09:26:44.212 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.212 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f36b357:::benchmark_data_smithi122_168655_object112808:head# 2024-03-20T09:26:44.213 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.213 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f36c03a:::benchmark_data_smithi122_168655_object23584:head# 2024-03-20T09:26:44.213 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.213 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f36ef60:::benchmark_data_smithi122_168655_object32633:head# 2024-03-20T09:26:44.213 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.213 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f36fe50:::benchmark_data_smithi122_168655_object95329:head# 2024-03-20T09:26:44.213 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.213 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f381967:::benchmark_data_smithi122_168655_object116291:head# 2024-03-20T09:26:44.213 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.213 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f38cc5a:::benchmark_data_smithi122_168655_object87470:head# 2024-03-20T09:26:44.213 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.213 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f39886b:::benchmark_data_smithi122_168655_object98202:head# 2024-03-20T09:26:44.213 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.213 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f398de7:::benchmark_data_smithi122_168655_object54036:head# 2024-03-20T09:26:44.213 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.214 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f39de13:::benchmark_data_smithi122_168655_object18538:head# 2024-03-20T09:26:44.214 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.214 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f3ab0f8:::benchmark_data_smithi122_168655_object86266:head# 2024-03-20T09:26:44.214 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.214 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f3c144e:::benchmark_data_smithi122_168655_object105606:head# 2024-03-20T09:26:44.214 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.214 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f3c5f72:::benchmark_data_smithi122_168655_object39573:head# 2024-03-20T09:26:44.214 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.214 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f3cc8a6:::benchmark_data_smithi122_168655_object69656:head# 2024-03-20T09:26:44.214 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.214 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f3d9119:::benchmark_data_smithi122_168655_object10687:head# 2024-03-20T09:26:44.214 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.214 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f3da224:::benchmark_data_smithi122_168655_object96962:head# 2024-03-20T09:26:44.215 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.215 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f3e54e3:::benchmark_data_smithi122_168655_object38989:head# 2024-03-20T09:26:44.215 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.215 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f3fc9b8:::benchmark_data_smithi122_168655_object40115:head# 2024-03-20T09:26:44.215 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.215 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f411633:::benchmark_data_smithi122_168655_object31051:head# 2024-03-20T09:26:44.215 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.215 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f4149fc:::benchmark_data_smithi122_168655_object84557:head# 2024-03-20T09:26:44.215 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.215 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f415963:::benchmark_data_smithi122_168655_object62454:head# 2024-03-20T09:26:44.215 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.215 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f416f7f:::benchmark_data_smithi122_168655_object30077:head# 2024-03-20T09:26:44.215 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.215 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f43087d:::benchmark_data_smithi122_168655_object34951:head# 2024-03-20T09:26:44.216 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.216 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f430b0a:::benchmark_data_smithi122_168655_object18552:head# 2024-03-20T09:26:44.216 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.216 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f43d8bc:::benchmark_data_smithi122_168655_object64766:head# 2024-03-20T09:26:44.216 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.216 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f44319c:::benchmark_data_smithi122_168655_object116236:head# 2024-03-20T09:26:44.216 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.216 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f4486fd:::benchmark_data_smithi122_168655_object64946:head# 2024-03-20T09:26:44.216 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.216 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f44ac77:::benchmark_data_smithi122_168655_object109102:head# 2024-03-20T09:26:44.216 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.216 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f451031:::benchmark_data_smithi122_168655_object80727:head# 2024-03-20T09:26:44.216 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.217 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f452ee5:::benchmark_data_smithi122_168655_object21889:head# 2024-03-20T09:26:44.217 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.217 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f47e09b:::benchmark_data_smithi122_168655_object74973:head# 2024-03-20T09:26:44.217 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.217 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f48a435:::benchmark_data_smithi122_168655_object40389:head# 2024-03-20T09:26:44.217 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.217 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f48a70b:::benchmark_data_smithi122_168655_object119688:head# 2024-03-20T09:26:44.217 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.217 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f48c9cc:::benchmark_data_smithi122_168655_object113616:head# 2024-03-20T09:26:44.217 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.217 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f48d426:::benchmark_data_smithi122_168655_object10705:head# 2024-03-20T09:26:44.217 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.218 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f48d5d3:::benchmark_data_smithi122_168655_object30970:head# 2024-03-20T09:26:44.218 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.218 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f4958e2:::benchmark_data_smithi122_168655_object112246:head# 2024-03-20T09:26:44.218 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.218 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f4966d9:::benchmark_data_smithi122_168655_object118119:head# 2024-03-20T09:26:44.218 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.218 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f499e72:::benchmark_data_smithi122_168655_object117293:head# 2024-03-20T09:26:44.218 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.218 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f49d559:::benchmark_data_smithi122_168655_object83026:head# 2024-03-20T09:26:44.218 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.218 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f49d9ba:::benchmark_data_smithi122_168655_object26507:head# 2024-03-20T09:26:44.218 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.218 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f4aa566:::benchmark_data_smithi122_168655_object57239:head# 2024-03-20T09:26:44.219 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.219 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f4bd140:::benchmark_data_smithi122_168655_object96307:head# 2024-03-20T09:26:44.219 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.219 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f4c2026:::benchmark_data_smithi122_168655_object91161:head# 2024-03-20T09:26:44.219 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.219 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f4cd7be:::benchmark_data_smithi122_168655_object116293:head# 2024-03-20T09:26:44.219 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.219 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f4cdb67:::benchmark_data_smithi122_168655_object14879:head# 2024-03-20T09:26:44.219 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.219 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f4d3234:::benchmark_data_smithi122_168655_object29527:head# 2024-03-20T09:26:44.219 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.219 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f4ded33:::benchmark_data_smithi122_168655_object62416:head# 2024-03-20T09:26:44.219 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.219 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f4e2b93:::benchmark_data_smithi122_168655_object66584:head# 2024-03-20T09:26:44.220 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.220 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f4ea64f:::benchmark_data_smithi122_168655_object27942:head# 2024-03-20T09:26:44.220 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.220 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f4ef865:::benchmark_data_smithi122_168655_object47769:head# 2024-03-20T09:26:44.220 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.220 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f4f0658:::benchmark_data_smithi122_168655_object38719:head# 2024-03-20T09:26:44.220 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.220 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f4f7d06:::benchmark_data_smithi122_168655_object68216:head# 2024-03-20T09:26:44.220 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.220 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f504200:::benchmark_data_smithi122_168655_object58928:head# 2024-03-20T09:26:44.220 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.220 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f506424:::benchmark_data_smithi122_168655_object76787:head# 2024-03-20T09:26:44.220 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.221 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f51327e:::benchmark_data_smithi122_168655_object33346:head# 2024-03-20T09:26:44.221 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.221 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f51b088:::benchmark_data_smithi122_168655_object118207:head# 2024-03-20T09:26:44.221 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.221 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f51b889:::benchmark_data_smithi122_168655_object50276:head# 2024-03-20T09:26:44.221 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.221 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f52e930:::benchmark_data_smithi122_168655_object32070:head# 2024-03-20T09:26:44.221 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.221 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f531d8d:::benchmark_data_smithi122_168655_object61194:head# 2024-03-20T09:26:44.221 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.221 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f5321ec:::benchmark_data_smithi122_168655_object113106:head# 2024-03-20T09:26:44.221 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.221 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f54dec1:::benchmark_data_smithi122_168655_object23948:head# 2024-03-20T09:26:44.221 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.222 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f553d51:::benchmark_data_smithi122_168655_object70092:head# 2024-03-20T09:26:44.222 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.222 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f553eb3:::benchmark_data_smithi122_168655_object61899:head# 2024-03-20T09:26:44.222 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.222 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f5586bf:::benchmark_data_smithi122_168655_object55852:head# 2024-03-20T09:26:44.222 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.222 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f55a833:::benchmark_data_smithi122_168655_object66394:head# 2024-03-20T09:26:44.222 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.222 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f55bbd9:::benchmark_data_smithi122_168655_object33125:head# 2024-03-20T09:26:44.222 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.222 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f55c866:::benchmark_data_smithi122_168655_object83181:head# 2024-03-20T09:26:44.222 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.223 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f56389c:::benchmark_data_smithi122_168655_object105064:head# 2024-03-20T09:26:44.223 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.223 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f56569b:::benchmark_data_smithi122_168655_object68019:head# 2024-03-20T09:26:44.223 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.223 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f56613b:::benchmark_data_smithi122_168655_object86741:head# 2024-03-20T09:26:44.223 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.223 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f57362a:::benchmark_data_smithi122_168655_object52155:head# 2024-03-20T09:26:44.223 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.223 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f57a99b:::benchmark_data_smithi122_168655_object104871:head# 2024-03-20T09:26:44.223 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.223 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f57eba7:::benchmark_data_smithi122_168655_object10960:head# 2024-03-20T09:26:44.223 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.223 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f597f44:::benchmark_data_smithi122_168655_object67643:head# 2024-03-20T09:26:44.224 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.224 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f5afd9b:::benchmark_data_smithi122_168655_object15621:head# 2024-03-20T09:26:44.224 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.224 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f5b5783:::benchmark_data_smithi122_168655_object16794:head# 2024-03-20T09:26:44.224 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.224 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f5c4c90:::benchmark_data_smithi122_168655_object34315:head# 2024-03-20T09:26:44.224 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.224 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f5c6066:::benchmark_data_smithi122_168655_object43196:head# 2024-03-20T09:26:44.224 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.224 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f5d6206:::benchmark_data_smithi122_168655_object24891:head# 2024-03-20T09:26:44.224 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.224 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f5e8e9e:::benchmark_data_smithi122_168655_object69580:head# 2024-03-20T09:26:44.224 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.224 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f5fa851:::benchmark_data_smithi122_168655_object92182:head# 2024-03-20T09:26:44.225 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.225 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f60cb3f:::benchmark_data_smithi122_168655_object34180:head# 2024-03-20T09:26:44.225 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.225 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f613d3e:::benchmark_data_smithi122_168655_object68050:head# 2024-03-20T09:26:44.225 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.225 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f626c80:::benchmark_data_smithi122_168655_object119338:head# 2024-03-20T09:26:44.225 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.225 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f62b39c:::benchmark_data_smithi122_168655_object76120:head# 2024-03-20T09:26:44.225 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.225 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f63009e:::benchmark_data_smithi122_168655_object15314:head# 2024-03-20T09:26:44.225 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.225 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f637b2c:::benchmark_data_smithi122_168655_object75182:head# 2024-03-20T09:26:44.225 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.226 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f640554:::benchmark_data_smithi122_168655_object113525:head# 2024-03-20T09:26:44.226 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.226 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f642a27:::benchmark_data_smithi122_168655_object113554:head# 2024-03-20T09:26:44.226 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.226 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f649044:::benchmark_data_smithi122_168655_object111071:head# 2024-03-20T09:26:44.226 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.226 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f65dfcf:::benchmark_data_smithi122_168655_object73063:head# 2024-03-20T09:26:44.226 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.226 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f66a595:::benchmark_data_smithi122_168655_object31166:head# 2024-03-20T09:26:44.226 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.226 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f66c729:::benchmark_data_smithi122_168655_object54885:head# 2024-03-20T09:26:44.226 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.226 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f683d4b:::benchmark_data_smithi122_168655_object11655:head# 2024-03-20T09:26:44.226 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.227 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f68ef91:::benchmark_data_smithi122_168655_object54287:head# 2024-03-20T09:26:44.227 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.227 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f69326e:::benchmark_data_smithi122_168655_object36881:head# 2024-03-20T09:26:44.227 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.227 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f694321:::benchmark_data_smithi122_168655_object103081:head# 2024-03-20T09:26:44.227 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.227 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f6b3330:::benchmark_data_smithi122_168655_object17988:head# 2024-03-20T09:26:44.227 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.227 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f6bc41b:::benchmark_data_smithi122_168655_object81389:head# 2024-03-20T09:26:44.227 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.227 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f6bec67:::benchmark_data_smithi122_168655_object94566:head# 2024-03-20T09:26:44.227 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.227 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f6c0181:::benchmark_data_smithi122_168655_object74621:head# 2024-03-20T09:26:44.228 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.228 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f6da32a:::benchmark_data_smithi122_168655_object108867:head# 2024-03-20T09:26:44.228 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.228 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f6dd826:::benchmark_data_smithi122_168655_object27319:head# 2024-03-20T09:26:44.228 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.228 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f6e0caa:::benchmark_data_smithi122_168655_object76653:head# 2024-03-20T09:26:44.228 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.228 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f6ec473:::benchmark_data_smithi122_168655_object56653:head# 2024-03-20T09:26:44.228 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.228 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f6ecfa0:::benchmark_data_smithi122_168655_object42065:head# 2024-03-20T09:26:44.228 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.228 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f6f5598:::benchmark_data_smithi122_168655_object77535:head# 2024-03-20T09:26:44.228 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.229 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f6fd305:::benchmark_data_smithi122_168655_object81847:head# 2024-03-20T09:26:44.229 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.229 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f701870:::benchmark_data_smithi122_168655_object104578:head# 2024-03-20T09:26:44.229 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.229 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f710a1d:::benchmark_data_smithi122_168655_object94927:head# 2024-03-20T09:26:44.229 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.229 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f7155e5:::benchmark_data_smithi122_168655_object16429:head# 2024-03-20T09:26:44.229 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.229 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f726b12:::benchmark_data_smithi122_168655_object25972:head# 2024-03-20T09:26:44.229 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.229 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f729ad7:::benchmark_data_smithi122_168655_object96285:head# 2024-03-20T09:26:44.229 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.229 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f73d7ea:::benchmark_data_smithi122_168655_object67796:head# 2024-03-20T09:26:44.229 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.230 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f7409a6:::benchmark_data_smithi122_168655_object23711:head# 2024-03-20T09:26:44.230 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.230 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f7423cc:::benchmark_data_smithi122_168655_object117557:head# 2024-03-20T09:26:44.230 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.230 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f7574d7:::benchmark_data_smithi122_168655_object81447:head# 2024-03-20T09:26:44.230 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.230 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f758997:::benchmark_data_smithi122_168655_object117885:head# 2024-03-20T09:26:44.230 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.230 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f7628a4:::benchmark_data_smithi122_168655_object100209:head# 2024-03-20T09:26:44.230 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.230 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f762933:::benchmark_data_smithi122_168655_object28856:head# 2024-03-20T09:26:44.230 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.231 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f772475:::benchmark_data_smithi122_168655_object48912:head# 2024-03-20T09:26:44.231 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.231 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f77c7db:::benchmark_data_smithi122_168655_object24271:head# 2024-03-20T09:26:44.231 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.231 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f7819dc:::benchmark_data_smithi122_168655_object80952:head# 2024-03-20T09:26:44.231 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.231 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f786041:::benchmark_data_smithi122_168655_object38792:head# 2024-03-20T09:26:44.231 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.231 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f79f9a4:::benchmark_data_smithi122_168655_object107922:head# 2024-03-20T09:26:44.231 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.231 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f7a2be5:::benchmark_data_smithi122_168655_object54801:head# 2024-03-20T09:26:44.231 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.231 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f7ad740:::benchmark_data_smithi122_168655_object94458:head# 2024-03-20T09:26:44.231 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.232 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f7b498e:::benchmark_data_smithi122_168655_object77334:head# 2024-03-20T09:26:44.232 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.232 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f7bd6e1:::benchmark_data_smithi122_168655_object60528:head# 2024-03-20T09:26:44.232 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.232 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f7ccd95:::benchmark_data_smithi122_168655_object118325:head# 2024-03-20T09:26:44.232 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.232 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f7cdc76:::benchmark_data_smithi122_168655_object16324:head# 2024-03-20T09:26:44.232 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.232 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f7ce402:::benchmark_data_smithi122_168655_object11034:head# 2024-03-20T09:26:44.232 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.232 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f7f331a:::benchmark_data_smithi122_168655_object31570:head# 2024-03-20T09:26:44.232 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.233 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f7fb213:::benchmark_data_smithi122_168655_object16902:head# 2024-03-20T09:26:44.233 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.233 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f7fe15a:::benchmark_data_smithi122_168655_object32493:head# 2024-03-20T09:26:44.233 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.233 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f800b89:::benchmark_data_smithi122_168655_object46851:head# 2024-03-20T09:26:44.233 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.233 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f825ea7:::benchmark_data_smithi122_168655_object63166:head# 2024-03-20T09:26:44.233 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.233 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f82bbb4:::benchmark_data_smithi122_168655_object111606:head# 2024-03-20T09:26:44.233 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.233 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f82d7f0:::benchmark_data_smithi122_168655_object19613:head# 2024-03-20T09:26:44.233 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.233 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f830e91:::benchmark_data_smithi122_168655_object118985:head# 2024-03-20T09:26:44.233 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.234 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f851933:::benchmark_data_smithi122_168655_object25884:head# 2024-03-20T09:26:44.234 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.234 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f856360:::benchmark_data_smithi122_168655_object91408:head# 2024-03-20T09:26:44.234 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.234 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f85b392:::benchmark_data_smithi122_168655_object55466:head# 2024-03-20T09:26:44.234 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.234 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f8674ed:::benchmark_data_smithi122_168655_object67040:head# 2024-03-20T09:26:44.234 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.234 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f86b8cc:::benchmark_data_smithi122_168655_object111210:head# 2024-03-20T09:26:44.234 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.234 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f873144:::benchmark_data_smithi122_168655_object100422:head# 2024-03-20T09:26:44.234 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.234 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f8764a4:::benchmark_data_smithi122_168655_object73848:head# 2024-03-20T09:26:44.235 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.235 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f88c6fe:::benchmark_data_smithi122_168655_object111572:head# 2024-03-20T09:26:44.235 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.235 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f895731:::benchmark_data_smithi122_168655_object109497:head# 2024-03-20T09:26:44.235 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.235 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f896c4a:::benchmark_data_smithi122_168655_object60939:head# 2024-03-20T09:26:44.235 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.235 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f8a7925:::benchmark_data_smithi122_168655_object23364:head# 2024-03-20T09:26:44.235 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.235 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f8bac1e:::benchmark_data_smithi122_168655_object77135:head# 2024-03-20T09:26:44.235 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.235 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f8cd73f:::benchmark_data_smithi122_168655_object45154:head# 2024-03-20T09:26:44.235 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.235 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f8d14bb:::benchmark_data_smithi122_168655_object60328:head# 2024-03-20T09:26:44.236 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.236 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f8d1780:::benchmark_data_smithi122_168655_object83540:head# 2024-03-20T09:26:44.236 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.236 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f8de908:::benchmark_data_smithi122_168655_object70924:head# 2024-03-20T09:26:44.236 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.236 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f905110:::benchmark_data_smithi122_168655_object56349:head# 2024-03-20T09:26:44.236 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.236 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f907397:::benchmark_data_smithi122_168655_object51950:head# 2024-03-20T09:26:44.236 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.236 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f9098d1:::benchmark_data_smithi122_168655_object52294:head# 2024-03-20T09:26:44.236 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.236 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f911fbf:::benchmark_data_smithi122_168655_object52050:head# 2024-03-20T09:26:44.236 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.236 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f912238:::benchmark_data_smithi122_168655_object53015:head# 2024-03-20T09:26:44.236 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.237 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f91b9ec:::benchmark_data_smithi122_168655_object40076:head# 2024-03-20T09:26:44.237 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.237 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f91f0e6:::benchmark_data_smithi122_168655_object30412:head# 2024-03-20T09:26:44.237 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.237 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f924142:::benchmark_data_smithi122_168655_object82336:head# 2024-03-20T09:26:44.237 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.237 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f92572d:::benchmark_data_smithi122_168655_object90337:head# 2024-03-20T09:26:44.237 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.237 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f92f2b6:::benchmark_data_smithi122_168655_object109751:head# 2024-03-20T09:26:44.237 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.237 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f958716:::benchmark_data_smithi122_168655_object44492:head# 2024-03-20T09:26:44.237 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.237 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f95b789:::benchmark_data_smithi122_168655_object12839:head# 2024-03-20T09:26:44.237 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.238 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f9640de:::benchmark_data_smithi122_168655_object92924:head# 2024-03-20T09:26:44.238 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.238 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f974cfb:::benchmark_data_smithi122_168655_object73206:head# 2024-03-20T09:26:44.238 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.238 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f9795b4:::benchmark_data_smithi122_168655_object24389:head# 2024-03-20T09:26:44.238 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.238 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f97a4e0:::benchmark_data_smithi122_168655_object113027:head# 2024-03-20T09:26:44.238 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.238 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f987471:::benchmark_data_smithi122_168655_object92038:head# 2024-03-20T09:26:44.238 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.238 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f98769b:::benchmark_data_smithi122_168655_object85810:head# 2024-03-20T09:26:44.238 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.238 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f98d698:::benchmark_data_smithi122_168655_object119746:head# 2024-03-20T09:26:44.239 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.239 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f9a572c:::benchmark_data_smithi122_168655_object106613:head# 2024-03-20T09:26:44.239 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.239 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f9a76ed:::benchmark_data_smithi122_168655_object52928:head# 2024-03-20T09:26:44.239 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.239 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f9adf5a:::benchmark_data_smithi122_168655_object119082:head# 2024-03-20T09:26:44.239 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.239 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f9bd12d:::benchmark_data_smithi122_168655_object59446:head# 2024-03-20T09:26:44.239 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.239 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f9bf524:::benchmark_data_smithi122_168655_object19083:head# 2024-03-20T09:26:44.239 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.239 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f9c670a:::benchmark_data_smithi122_168655_object52792:head# 2024-03-20T09:26:44.239 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.240 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f9cb961:::benchmark_data_smithi122_168655_object82200:head# 2024-03-20T09:26:44.240 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.240 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f9ce8ed:::benchmark_data_smithi122_168655_object40636:head# 2024-03-20T09:26:44.240 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.240 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f9e0b77:::benchmark_data_smithi122_168655_object108488:head# 2024-03-20T09:26:44.240 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.240 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f9e248b:::benchmark_data_smithi122_168655_object43355:head# 2024-03-20T09:26:44.240 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.240 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f9e9dfb:::benchmark_data_smithi122_168655_object47981:head# 2024-03-20T09:26:44.240 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.240 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f9f70c6:::benchmark_data_smithi122_168655_object23357:head# 2024-03-20T09:26:44.240 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.240 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f9f829a:::benchmark_data_smithi122_168655_object14556:head# 2024-03-20T09:26:44.240 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.241 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8f9fc941:::benchmark_data_smithi122_168655_object86486:head# 2024-03-20T09:26:44.241 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.241 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fa04347:::benchmark_data_smithi122_168655_object97834:head# 2024-03-20T09:26:44.241 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.241 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fa20c65:::benchmark_data_smithi122_168655_object78568:head# 2024-03-20T09:26:44.241 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.241 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fa26c49:::benchmark_data_smithi122_168655_object69156:head# 2024-03-20T09:26:44.241 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.241 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fa2710c:::benchmark_data_smithi122_168655_object103865:head# 2024-03-20T09:26:44.241 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.241 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fa312db:::benchmark_data_smithi122_168655_object42647:head# 2024-03-20T09:26:44.241 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.241 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fa36723:::benchmark_data_smithi122_168655_object46297:head# 2024-03-20T09:26:44.242 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.242 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fa4249b:::benchmark_data_smithi122_168655_object65362:head# 2024-03-20T09:26:44.242 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.242 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fa4d455:::benchmark_data_smithi122_168655_object116167:head# 2024-03-20T09:26:44.242 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.242 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fa4f5b4:::benchmark_data_smithi122_168655_object85310:head# 2024-03-20T09:26:44.242 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.242 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fa535d6:::benchmark_data_smithi122_168655_object117391:head# 2024-03-20T09:26:44.242 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.242 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fa5cbcf:::benchmark_data_smithi122_168655_object111370:head# 2024-03-20T09:26:44.242 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.242 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fa6298b:::benchmark_data_smithi122_168655_object51474:head# 2024-03-20T09:26:44.242 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.242 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fa84440:::benchmark_data_smithi122_168655_object15853:head# 2024-03-20T09:26:44.243 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.243 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fa94a7b:::benchmark_data_smithi122_168655_object44680:head# 2024-03-20T09:26:44.243 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.243 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8faa7520:::benchmark_data_smithi122_168655_object56175:head# 2024-03-20T09:26:44.243 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.243 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fab2f6d:::benchmark_data_smithi122_168655_object13774:head# 2024-03-20T09:26:44.243 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.243 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fab3f7e:::benchmark_data_smithi122_168655_object89322:head# 2024-03-20T09:26:44.243 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.243 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fabb85f:::benchmark_data_smithi122_168655_object109822:head# 2024-03-20T09:26:44.243 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.243 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fac1316:::benchmark_data_smithi122_168655_object67483:head# 2024-03-20T09:26:44.243 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.243 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8facaf3b:::benchmark_data_smithi122_168655_object37558:head# 2024-03-20T09:26:44.243 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.244 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8facbe04:::benchmark_data_smithi122_168655_object91498:head# 2024-03-20T09:26:44.244 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.244 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fad3cbd:::benchmark_data_smithi122_168655_object81961:head# 2024-03-20T09:26:44.244 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.244 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fad6c8b:::benchmark_data_smithi122_168655_object18004:head# 2024-03-20T09:26:44.244 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.244 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fadff07:::benchmark_data_smithi122_168655_object91367:head# 2024-03-20T09:26:44.244 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.244 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8faec760:::benchmark_data_smithi122_168655_object17274:head# 2024-03-20T09:26:44.244 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.244 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8faf39c1:::benchmark_data_smithi122_168655_object13671:head# 2024-03-20T09:26:44.244 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.244 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8faf5b23:::benchmark_data_smithi122_168655_object79534:head# 2024-03-20T09:26:44.244 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.245 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fb14711:::benchmark_data_smithi122_168655_object101291:head# 2024-03-20T09:26:44.245 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.245 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fb158e5:::benchmark_data_smithi122_168655_object89435:head# 2024-03-20T09:26:44.245 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.245 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fb16fee:::benchmark_data_smithi122_168655_object68460:head# 2024-03-20T09:26:44.245 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.245 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fb1e6b8:::benchmark_data_smithi122_168655_object92561:head# 2024-03-20T09:26:44.245 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.245 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fb2f393:::benchmark_data_smithi122_168655_object17403:head# 2024-03-20T09:26:44.245 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.245 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fb30311:::benchmark_data_smithi122_168655_object65575:head# 2024-03-20T09:26:44.245 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.245 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fb317cb:::benchmark_data_smithi122_168655_object91502:head# 2024-03-20T09:26:44.246 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.246 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fb48d1e:::benchmark_data_smithi122_168655_object45082:head# 2024-03-20T09:26:44.246 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.246 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fb4ef1b:::benchmark_data_smithi122_168655_object42336:head# 2024-03-20T09:26:44.246 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.246 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fb67cab:::benchmark_data_smithi122_168655_object112202:head# 2024-03-20T09:26:44.246 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.246 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fb725b7:::benchmark_data_smithi122_168655_object115263:head# 2024-03-20T09:26:44.246 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.246 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fb7cd32:::benchmark_data_smithi122_168655_object109692:head# 2024-03-20T09:26:44.246 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.246 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fb99317:::benchmark_data_smithi122_168655_object63872:head# 2024-03-20T09:26:44.246 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.246 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fbb12db:::benchmark_data_smithi122_168655_object101732:head# 2024-03-20T09:26:44.247 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.247 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fbb8c81:::benchmark_data_smithi122_168655_object55158:head# 2024-03-20T09:26:44.247 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.247 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fbbda9d:::benchmark_data_smithi122_168655_object67451:head# 2024-03-20T09:26:44.247 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.247 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fbc78ff:::benchmark_data_smithi122_168655_object29722:head# 2024-03-20T09:26:44.247 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.247 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fbd6eac:::benchmark_data_smithi122_168655_object40510:head# 2024-03-20T09:26:44.247 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.247 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fbd81b3:::benchmark_data_smithi122_168655_object25395:head# 2024-03-20T09:26:44.247 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.247 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fbdb05d:::benchmark_data_smithi122_168655_object52303:head# 2024-03-20T09:26:44.247 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.247 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fbe3cbb:::benchmark_data_smithi122_168655_object47717:head# 2024-03-20T09:26:44.248 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.248 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fbfc93c:::benchmark_data_smithi122_168655_object102996:head# 2024-03-20T09:26:44.248 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.248 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fc043cf:::benchmark_data_smithi122_168655_object92784:head# 2024-03-20T09:26:44.248 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.248 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fc0692e:::benchmark_data_smithi122_168655_object38104:head# 2024-03-20T09:26:44.248 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.248 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fc0b470:::benchmark_data_smithi122_168655_object65779:head# 2024-03-20T09:26:44.248 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.248 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fc10be6:::benchmark_data_smithi122_168655_object23338:head# 2024-03-20T09:26:44.248 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.248 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fc124b9:::benchmark_data_smithi122_168655_object47066:head# 2024-03-20T09:26:44.248 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.248 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fc1b5db:::benchmark_data_smithi122_168655_object81677:head# 2024-03-20T09:26:44.248 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.248 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fc25581:::benchmark_data_smithi122_168655_object71276:head# 2024-03-20T09:26:44.249 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.249 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fc38b3d:::benchmark_data_smithi122_168655_object101936:head# 2024-03-20T09:26:44.249 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.249 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fc3ab7e:::benchmark_data_smithi122_168655_object69771:head# 2024-03-20T09:26:44.249 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.249 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fc51e21:::benchmark_data_smithi122_168655_object32629:head# 2024-03-20T09:26:44.249 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.249 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fc56d23:::benchmark_data_smithi122_168655_object100027:head# 2024-03-20T09:26:44.249 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.249 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fc6ea32:::benchmark_data_smithi122_168655_object98433:head# 2024-03-20T09:26:44.249 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.249 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fc703ee:::benchmark_data_smithi122_168655_object101031:head# 2024-03-20T09:26:44.249 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.249 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fc7eec0:::benchmark_data_smithi122_168655_object21826:head# 2024-03-20T09:26:44.250 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.250 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fc80823:::benchmark_data_smithi122_168655_object45047:head# 2024-03-20T09:26:44.250 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.250 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fc8c731:::benchmark_data_smithi122_168655_object105381:head# 2024-03-20T09:26:44.250 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.250 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fc913db:::benchmark_data_smithi122_168655_object31862:head# 2024-03-20T09:26:44.250 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.250 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fca1cc9:::benchmark_data_smithi122_168655_object45049:head# 2024-03-20T09:26:44.250 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.250 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fca2472:::benchmark_data_smithi122_168655_object88423:head# 2024-03-20T09:26:44.250 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.250 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fca8863:::benchmark_data_smithi122_168655_object24059:head# 2024-03-20T09:26:44.250 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.251 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fcb8ee0:::benchmark_data_smithi122_168655_object97813:head# 2024-03-20T09:26:44.251 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.251 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fcbe957:::benchmark_data_smithi122_168655_object113830:head# 2024-03-20T09:26:44.251 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.251 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fcc38ed:::benchmark_data_smithi122_168655_object38174:head# 2024-03-20T09:26:44.251 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.251 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fcc3f25:::benchmark_data_smithi122_168655_object12255:head# 2024-03-20T09:26:44.251 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.251 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fcc63ff:::benchmark_data_smithi122_168655_object35990:head# 2024-03-20T09:26:44.251 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.251 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fccb855:::benchmark_data_smithi122_168655_object55745:head# 2024-03-20T09:26:44.251 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.251 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fcd00bf:::benchmark_data_smithi122_168655_object32281:head# 2024-03-20T09:26:44.252 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.252 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fcd8edb:::benchmark_data_smithi122_168655_object102109:head# 2024-03-20T09:26:44.252 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.252 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fcde81d:::benchmark_data_smithi122_168655_object47388:head# 2024-03-20T09:26:44.252 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.252 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fce0691:::benchmark_data_smithi122_168655_object111462:head# 2024-03-20T09:26:44.252 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.252 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fce9140:::benchmark_data_smithi122_168655_object32682:head# 2024-03-20T09:26:44.252 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.252 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fceea71:::benchmark_data_smithi122_168655_object81052:head# 2024-03-20T09:26:44.252 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.252 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fcf7be3:::benchmark_data_smithi122_168655_object86997:head# 2024-03-20T09:26:44.252 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.252 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fd05bab:::benchmark_data_smithi122_168655_object72381:head# 2024-03-20T09:26:44.252 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.253 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fd38271:::benchmark_data_smithi122_168655_object114610:head# 2024-03-20T09:26:44.253 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.253 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fd3b61e:::benchmark_data_smithi122_168655_object59071:head# 2024-03-20T09:26:44.253 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.253 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fd3f2f7:::benchmark_data_smithi122_168655_object20201:head# 2024-03-20T09:26:44.253 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.253 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fd49f9a:::benchmark_data_smithi122_168655_object65591:head# 2024-03-20T09:26:44.253 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.253 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fd4bddc:::benchmark_data_smithi122_168655_object91799:head# 2024-03-20T09:26:44.253 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.253 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fd53cd1:::benchmark_data_smithi122_168655_object97548:head# 2024-03-20T09:26:44.253 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.253 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fd56db6:::benchmark_data_smithi122_168655_object77807:head# 2024-03-20T09:26:44.254 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.254 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fd59657:::benchmark_data_smithi122_168655_object31294:head# 2024-03-20T09:26:44.254 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.254 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fd5c0cb:::benchmark_data_smithi122_168655_object66508:head# 2024-03-20T09:26:44.254 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.254 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fd6ec49:::benchmark_data_smithi122_168655_object64095:head# 2024-03-20T09:26:44.254 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.254 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fd6edeb:::benchmark_data_smithi122_168655_object88040:head# 2024-03-20T09:26:44.254 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.254 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fda219d:::benchmark_data_smithi122_168655_object13834:head# 2024-03-20T09:26:44.254 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.254 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fda8c26:::benchmark_data_smithi122_168655_object83418:head# 2024-03-20T09:26:44.254 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.254 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fdb6932:::benchmark_data_smithi122_168655_object16582:head# 2024-03-20T09:26:44.255 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.255 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fdba900:::benchmark_data_smithi122_168655_object62551:head# 2024-03-20T09:26:44.255 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.255 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fdca405:::benchmark_data_smithi122_168655_object76384:head# 2024-03-20T09:26:44.255 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.255 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fdca5c5:::benchmark_data_smithi122_168655_object23813:head# 2024-03-20T09:26:44.255 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.255 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fdcf600:::benchmark_data_smithi122_168655_object31618:head# 2024-03-20T09:26:44.255 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.255 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fdd6110:::benchmark_data_smithi122_168655_object62422:head# 2024-03-20T09:26:44.255 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.255 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fdfa09a:::benchmark_data_smithi122_168655_object117090:head# 2024-03-20T09:26:44.255 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.255 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fdfcf06:::benchmark_data_smithi122_168655_object65111:head# 2024-03-20T09:26:44.256 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.256 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fe01b46:::benchmark_data_smithi122_168655_object39268:head# 2024-03-20T09:26:44.256 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.256 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fe08e0a:::benchmark_data_smithi122_168655_object45110:head# 2024-03-20T09:26:44.256 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.256 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fe23fbb:::benchmark_data_smithi122_168655_object82613:head# 2024-03-20T09:26:44.256 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.256 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fe2c666:::benchmark_data_smithi122_168655_object56457:head# 2024-03-20T09:26:44.256 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.256 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fe3a9d7:::benchmark_data_smithi122_168655_object83955:head# 2024-03-20T09:26:44.256 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.256 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fe420c1:::benchmark_data_smithi122_168655_object62917:head# 2024-03-20T09:26:44.256 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.257 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fe43dd8:::benchmark_data_smithi122_168655_object34732:head# 2024-03-20T09:26:44.257 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.257 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fe4eed8:::benchmark_data_smithi122_168655_object36625:head# 2024-03-20T09:26:44.257 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.257 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fe531ee:::benchmark_data_smithi122_168655_object26481:head# 2024-03-20T09:26:44.257 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.257 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fe6421c:::benchmark_data_smithi122_168655_object12359:head# 2024-03-20T09:26:44.257 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.257 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fe64c32:::benchmark_data_smithi122_168655_object81827:head# 2024-03-20T09:26:44.257 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.257 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fe67ba7:::benchmark_data_smithi122_168655_object18718:head# 2024-03-20T09:26:44.257 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.257 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fe6932a:::benchmark_data_smithi122_168655_object90059:head# 2024-03-20T09:26:44.257 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.257 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fe74f1b:::benchmark_data_smithi122_168655_object113253:head# 2024-03-20T09:26:44.258 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.258 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fe811eb:::benchmark_data_smithi122_168655_object49573:head# 2024-03-20T09:26:44.258 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.258 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fe84ce2:::benchmark_data_smithi122_168655_object46327:head# 2024-03-20T09:26:44.258 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.258 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fe9648c:::benchmark_data_smithi122_168655_object89624:head# 2024-03-20T09:26:44.258 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.258 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8feb3a0d:::benchmark_data_smithi122_168655_object11498:head# 2024-03-20T09:26:44.258 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.258 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8feb5c22:::benchmark_data_smithi122_168655_object26596:head# 2024-03-20T09:26:44.258 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.258 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8febacc4:::benchmark_data_smithi122_168655_object73746:head# 2024-03-20T09:26:44.258 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.259 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8febcc9a:::benchmark_data_smithi122_168655_object97411:head# 2024-03-20T09:26:44.259 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.259 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8febf3aa:::benchmark_data_smithi122_168655_object78250:head# 2024-03-20T09:26:44.259 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.259 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fec9fca:::benchmark_data_smithi122_168655_object10369:head# 2024-03-20T09:26:44.259 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.259 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fedc030:::benchmark_data_smithi122_168655_object46724:head# 2024-03-20T09:26:44.259 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.259 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8feeb67a:::benchmark_data_smithi122_168655_object38313:head# 2024-03-20T09:26:44.259 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.259 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8feeb7c5:::benchmark_data_smithi122_168655_object42132:head# 2024-03-20T09:26:44.259 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.259 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8feede18:::benchmark_data_smithi122_168655_object96244:head# 2024-03-20T09:26:44.259 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.260 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8feee9e5:::benchmark_data_smithi122_168655_object71377:head# 2024-03-20T09:26:44.260 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.260 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ff01d6c:::benchmark_data_smithi122_168655_object11126:head# 2024-03-20T09:26:44.260 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.260 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ff26d34:::benchmark_data_smithi122_168655_object63000:head# 2024-03-20T09:26:44.260 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.260 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ff2f092:::benchmark_data_smithi122_168655_object45709:head# 2024-03-20T09:26:44.260 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.260 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ff4704d:::benchmark_data_smithi122_168655_object10451:head# 2024-03-20T09:26:44.260 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.260 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ff6c155:::benchmark_data_smithi122_168655_object56131:head# 2024-03-20T09:26:44.260 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.260 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ff79352:::benchmark_data_smithi122_168655_object111059:head# 2024-03-20T09:26:44.260 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.261 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ff7c8de:::benchmark_data_smithi122_168655_object56857:head# 2024-03-20T09:26:44.261 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.261 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ff85589:::benchmark_data_smithi122_168655_object78531:head# 2024-03-20T09:26:44.261 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.261 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ff8a582:::benchmark_data_smithi122_168655_object61203:head# 2024-03-20T09:26:44.261 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.261 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ffb0897:::benchmark_data_smithi122_168655_object15605:head# 2024-03-20T09:26:44.261 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.261 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ffbd2e4:::benchmark_data_smithi122_168655_object34517:head# 2024-03-20T09:26:44.261 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.261 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ffc19c6:::benchmark_data_smithi122_168655_object23061:head# 2024-03-20T09:26:44.261 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.261 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ffc2472:::benchmark_data_smithi122_168655_object100533:head# 2024-03-20T09:26:44.261 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.262 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ffd4d36:::benchmark_data_smithi122_168655_object109902:head# 2024-03-20T09:26:44.262 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.262 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ffd6663:::benchmark_data_smithi122_168655_object61873:head# 2024-03-20T09:26:44.262 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.262 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8ffe6ae5:::benchmark_data_smithi122_168655_object72613:head# 2024-03-20T09:26:44.262 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.262 INFO:teuthology.orchestra.run.smithi122.stdout:Write #6:8fff4ca6:::benchmark_data_smithi122_168655_object38853:head# 2024-03-20T09:26:44.262 INFO:teuthology.orchestra.run.smithi122.stdout:snapset 0=[]:{} 2024-03-20T09:26:44.263 INFO:teuthology.orchestra.run.smithi122.stdout:write_pg epoch 753 info 6.11( v 750'8141 (0'0,750'8141] local-lis/les=749/750 n=3415 ec=749/709 lis/c=749/749 les/c/f=750/750/0 sis=752) 2024-03-20T09:26:44.263 INFO:teuthology.orchestra.run.smithi122.stdout:Import successful 2024-03-20T09:26:44.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:44 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:26:44.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:44 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:26:44.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:44 smithi122 ceph-mon[73495]: osdmap e771: 12 total, 11 up, 12 in 2024-03-20T09:26:44.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:44 smithi122 ceph-mon[73495]: pgmap v1875: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 38 active+clean; 6.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 213 op/s; 31118/310344 objects degraded (10.027%); 29842/310344 objects misplaced (9.616%); 22 MiB/s, 408 objects/s recovering 2024-03-20T09:26:44.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:44 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:26:44.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:44 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:26:44.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:44 smithi026 ceph-mon[68309]: osdmap e771: 12 total, 11 up, 12 in 2024-03-20T09:26:44.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:44 smithi026 ceph-mon[68309]: pgmap v1875: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 38 active+clean; 6.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 213 op/s; 31118/310344 objects degraded (10.027%); 29842/310344 objects misplaced (9.616%); 22 MiB/s, 408 objects/s recovering 2024-03-20T09:26:44.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:44 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:26:44.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:44 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:26:44.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:44 smithi115 ceph-mon[72476]: osdmap e771: 12 total, 11 up, 12 in 2024-03-20T09:26:44.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:44 smithi115 ceph-mon[72476]: pgmap v1875: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 38 active+clean; 6.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 213 op/s; 31118/310344 objects degraded (10.027%); 29842/310344 objects misplaced (9.616%); 22 MiB/s, 408 objects/s recovering 2024-03-20T09:26:45.397 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/exp.6.11.10 2024-03-20T09:26:45.402 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:26:45.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:45.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:26:45.509 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:26:45.595 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:45.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:45 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:26:45.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:45 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:26:45.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:45 smithi122 ceph-mon[73495]: 6.3 scrub starts 2024-03-20T09:26:45.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:45 smithi122 ceph-mon[73495]: 6.3 scrub ok 2024-03-20T09:26:45.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:45 smithi122 ceph-mon[73495]: osdmap e772: 12 total, 11 up, 12 in 2024-03-20T09:26:45.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:45 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:26:45.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:45 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:26:45.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:45 smithi026 ceph-mon[68309]: 6.3 scrub starts 2024-03-20T09:26:45.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:45 smithi026 ceph-mon[68309]: 6.3 scrub ok 2024-03-20T09:26:45.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:45 smithi026 ceph-mon[68309]: osdmap e772: 12 total, 11 up, 12 in 2024-03-20T09:26:45.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:45 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:26:45.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:45 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:26:45.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:45 smithi115 ceph-mon[72476]: 6.3 scrub starts 2024-03-20T09:26:45.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:45 smithi115 ceph-mon[72476]: 6.3 scrub ok 2024-03-20T09:26:45.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:45 smithi115 ceph-mon[72476]: osdmap e772: 12 total, 11 up, 12 in 2024-03-20T09:26:46.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:46 smithi122 ceph-mon[73495]: 6.5 scrub starts 2024-03-20T09:26:46.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:46 smithi122 ceph-mon[73495]: 6.5 scrub ok 2024-03-20T09:26:46.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:46 smithi122 ceph-mon[73495]: pgmap v1877: 57 pgs: 1 active+clean+scrubbing, 3 active+undersized+degraded, 4 active+recovering+undersized+degraded+remapped, 7 active+undersized, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 38 active+clean; 6.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27414/308193 objects degraded (8.895%); 30867/308193 objects misplaced (10.015%); 22 MiB/s, 359 objects/s recovering 2024-03-20T09:26:46.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:46 smithi026 ceph-mon[68309]: 6.5 scrub starts 2024-03-20T09:26:46.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:46 smithi026 ceph-mon[68309]: 6.5 scrub ok 2024-03-20T09:26:46.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:46 smithi026 ceph-mon[68309]: pgmap v1877: 57 pgs: 1 active+clean+scrubbing, 3 active+undersized+degraded, 4 active+recovering+undersized+degraded+remapped, 7 active+undersized, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 38 active+clean; 6.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27414/308193 objects degraded (8.895%); 30867/308193 objects misplaced (10.015%); 22 MiB/s, 359 objects/s recovering 2024-03-20T09:26:46.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:46 smithi115 ceph-mon[72476]: 6.5 scrub starts 2024-03-20T09:26:46.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:46 smithi115 ceph-mon[72476]: 6.5 scrub ok 2024-03-20T09:26:46.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:46 smithi115 ceph-mon[72476]: pgmap v1877: 57 pgs: 1 active+clean+scrubbing, 3 active+undersized+degraded, 4 active+recovering+undersized+degraded+remapped, 7 active+undersized, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 38 active+clean; 6.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27414/308193 objects degraded (8.895%); 30867/308193 objects misplaced (10.015%); 22 MiB/s, 359 objects/s recovering 2024-03-20T09:26:47.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:47 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:26:47.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:47 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:26:47.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:47 smithi122 ceph-mon[73495]: 6.15 scrub starts 2024-03-20T09:26:47.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:47 smithi122 ceph-mon[73495]: 6.15 scrub ok 2024-03-20T09:26:47.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:47 smithi122 ceph-mon[73495]: osdmap e773: 12 total, 11 up, 12 in 2024-03-20T09:26:47.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:47 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:26:47.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:47 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:26:47.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:47 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:26:47.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:47 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:26:47.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:47 smithi026 ceph-mon[68309]: 6.15 scrub starts 2024-03-20T09:26:47.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:47 smithi026 ceph-mon[68309]: 6.15 scrub ok 2024-03-20T09:26:47.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:47 smithi026 ceph-mon[68309]: osdmap e773: 12 total, 11 up, 12 in 2024-03-20T09:26:47.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:47 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:26:47.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:47 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:26:47.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:47 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:26:47.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:47 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:26:47.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:47 smithi115 ceph-mon[72476]: 6.15 scrub starts 2024-03-20T09:26:47.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:47 smithi115 ceph-mon[72476]: 6.15 scrub ok 2024-03-20T09:26:47.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:47 smithi115 ceph-mon[72476]: osdmap e773: 12 total, 11 up, 12 in 2024-03-20T09:26:47.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:47 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:26:47.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:47 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:26:48.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:48 smithi026 ceph-mon[68309]: osdmap e774: 12 total, 11 up, 12 in 2024-03-20T09:26:48.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:48 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 27414/308193 objects degraded (8.895%), 8 pgs degraded (PG_DEGRADED) 2024-03-20T09:26:48.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:48 smithi026 ceph-mon[68309]: pgmap v1880: 57 pgs: 1 active+clean+scrubbing, 3 active+undersized+degraded, 4 active+recovering+undersized+degraded+remapped, 7 active+undersized, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 38 active+clean; 6.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27037/307953 objects degraded (8.780%); 30197/307953 objects misplaced (9.806%); 23 MiB/s, 371 objects/s recovering 2024-03-20T09:26:48.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:48 smithi115 ceph-mon[72476]: osdmap e774: 12 total, 11 up, 12 in 2024-03-20T09:26:48.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:48 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 27414/308193 objects degraded (8.895%), 8 pgs degraded (PG_DEGRADED) 2024-03-20T09:26:48.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:48 smithi115 ceph-mon[72476]: pgmap v1880: 57 pgs: 1 active+clean+scrubbing, 3 active+undersized+degraded, 4 active+recovering+undersized+degraded+remapped, 7 active+undersized, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 38 active+clean; 6.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27037/307953 objects degraded (8.780%); 30197/307953 objects misplaced (9.806%); 23 MiB/s, 371 objects/s recovering 2024-03-20T09:26:48.942 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:48 smithi122 ceph-mon[73495]: osdmap e774: 12 total, 11 up, 12 in 2024-03-20T09:26:48.942 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:48 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 27414/308193 objects degraded (8.895%), 8 pgs degraded (PG_DEGRADED) 2024-03-20T09:26:48.943 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:48 smithi122 ceph-mon[73495]: pgmap v1880: 57 pgs: 1 active+clean+scrubbing, 3 active+undersized+degraded, 4 active+recovering+undersized+degraded+remapped, 7 active+undersized, 3 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded+remapped, 38 active+clean; 6.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27037/307953 objects degraded (8.780%); 30197/307953 objects misplaced (9.806%); 23 MiB/s, 371 objects/s recovering 2024-03-20T09:26:49.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:49 smithi026 ceph-mon[68309]: osdmap e775: 12 total, 11 up, 12 in 2024-03-20T09:26:49.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:49 smithi115 ceph-mon[72476]: osdmap e775: 12 total, 11 up, 12 in 2024-03-20T09:26:49.942 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:49 smithi122 ceph-mon[73495]: osdmap e775: 12 total, 11 up, 12 in 2024-03-20T09:26:50.408 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:26:50.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:50.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:26:50.499 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 8, 10, 11, 3, 4, 9] out_osds: [] dead_osds: [10] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 11, 5] 2024-03-20T09:26:50.499 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:26:50.499 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-20T09:26:50.499 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-20T09:26:50.516 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:26:50.546 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:50.567 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:26:50.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:50.571 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:26:50.672 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:26:50.701 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:50.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:50 smithi026 ceph-mon[68309]: osdmap e776: 12 total, 11 up, 12 in 2024-03-20T09:26:50.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:50 smithi026 ceph-mon[68309]: pgmap v1883: 57 pgs: 1 creating+peering, 1 active+clean+scrubbing, 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 3 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 38 active+clean; 6.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 201 op/s; 23457/305775 objects degraded (7.671%); 31294/305775 objects misplaced (10.234%); 17 MiB/s, 351 objects/s recovering 2024-03-20T09:26:50.726 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:26:50.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:50.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:26:50.830 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:26:50.856 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:50 smithi115 ceph-mon[72476]: osdmap e776: 12 total, 11 up, 12 in 2024-03-20T09:26:50.856 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:50 smithi115 ceph-mon[72476]: pgmap v1883: 57 pgs: 1 creating+peering, 1 active+clean+scrubbing, 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 3 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 38 active+clean; 6.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 201 op/s; 23457/305775 objects degraded (7.671%); 31294/305775 objects misplaced (10.234%); 17 MiB/s, 351 objects/s recovering 2024-03-20T09:26:50.858 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:50.884 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:26:50.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:50.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:26:50.942 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:50 smithi122 ceph-mon[73495]: osdmap e776: 12 total, 11 up, 12 in 2024-03-20T09:26:50.942 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:50 smithi122 ceph-mon[73495]: pgmap v1883: 57 pgs: 1 creating+peering, 1 active+clean+scrubbing, 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 3 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 38 active+clean; 6.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 201 op/s; 23457/305775 objects degraded (7.671%); 31294/305775 objects misplaced (10.234%); 17 MiB/s, 351 objects/s recovering 2024-03-20T09:26:50.988 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:26:51.016 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:51.037 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:26:51.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:51.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:26:51.141 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:26:51.171 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:51.191 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:26:51.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:51.198 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:26:51.298 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:26:51.330 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:51.356 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:26:51.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:26:51.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:26:51.464 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:26:51.532 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:26:51.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:51 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:26:51.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:51 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:26:51.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:51 smithi115 ceph-mon[72476]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-20T09:26:51.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:51 smithi115 ceph-mon[72476]: osdmap e777: 12 total, 11 up, 12 in 2024-03-20T09:26:51.942 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:51 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:26:51.942 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:51 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:26:51.942 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:51 smithi122 ceph-mon[73495]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-20T09:26:51.942 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:51 smithi122 ceph-mon[73495]: osdmap e777: 12 total, 11 up, 12 in 2024-03-20T09:26:51.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:51 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:26:51.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:51 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:26:51.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:51 smithi026 ceph-mon[68309]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-20T09:26:51.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:51 smithi026 ceph-mon[68309]: osdmap e777: 12 total, 11 up, 12 in 2024-03-20T09:26:52.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:52 smithi115 ceph-mon[72476]: pgmap v1885: 57 pgs: 1 creating+peering, 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 3 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 39 active+clean; 6.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22917/303975 objects degraded (7.539%); 30118/303975 objects misplaced (9.908%) 2024-03-20T09:26:52.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:52 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:26:52.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:52 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 23457/305775 objects degraded (7.671%), 7 pgs degraded (PG_DEGRADED) 2024-03-20T09:26:52.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:52 smithi115 ceph-mon[72476]: osdmap e778: 12 total, 11 up, 12 in 2024-03-20T09:26:52.942 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:52 smithi122 ceph-mon[73495]: pgmap v1885: 57 pgs: 1 creating+peering, 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 3 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 39 active+clean; 6.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22917/303975 objects degraded (7.539%); 30118/303975 objects misplaced (9.908%) 2024-03-20T09:26:52.942 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:52 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:26:52.942 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:52 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 23457/305775 objects degraded (7.671%), 7 pgs degraded (PG_DEGRADED) 2024-03-20T09:26:52.942 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:52 smithi122 ceph-mon[73495]: osdmap e778: 12 total, 11 up, 12 in 2024-03-20T09:26:52.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:52 smithi026 ceph-mon[68309]: pgmap v1885: 57 pgs: 1 creating+peering, 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 3 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 39 active+clean; 6.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22917/303975 objects degraded (7.539%); 30118/303975 objects misplaced (9.908%) 2024-03-20T09:26:52.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:52 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:26:52.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:52 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 23457/305775 objects degraded (7.671%), 7 pgs degraded (PG_DEGRADED) 2024-03-20T09:26:52.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:52 smithi026 ceph-mon[68309]: osdmap e778: 12 total, 11 up, 12 in 2024-03-20T09:26:55.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:54 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:26:55.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:54 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:26:55.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:54 smithi122 ceph-mon[73495]: osdmap e779: 12 total, 11 up, 12 in 2024-03-20T09:26:55.193 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:54 smithi122 ceph-mon[73495]: 1.1 scrub starts 2024-03-20T09:26:55.193 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:54 smithi122 ceph-mon[73495]: 1.1 scrub ok 2024-03-20T09:26:55.193 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:54 smithi122 ceph-mon[73495]: pgmap v1888: 57 pgs: 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 3 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 40 active+clean; 6.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22797/302754 objects degraded (7.530%); 29195/302754 objects misplaced (9.643%) 2024-03-20T09:26:55.193 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:54 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:26:55.193 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:54 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:26:55.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:54 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:26:55.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:54 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:26:55.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:54 smithi026 ceph-mon[68309]: osdmap e779: 12 total, 11 up, 12 in 2024-03-20T09:26:55.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:54 smithi026 ceph-mon[68309]: 1.1 scrub starts 2024-03-20T09:26:55.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:54 smithi026 ceph-mon[68309]: 1.1 scrub ok 2024-03-20T09:26:55.221 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:54 smithi026 ceph-mon[68309]: pgmap v1888: 57 pgs: 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 3 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 40 active+clean; 6.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22797/302754 objects degraded (7.530%); 29195/302754 objects misplaced (9.643%) 2024-03-20T09:26:55.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:54 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:26:55.222 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:54 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:26:55.402 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:54 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:26:55.402 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:54 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:26:55.402 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:54 smithi115 ceph-mon[72476]: osdmap e779: 12 total, 11 up, 12 in 2024-03-20T09:26:55.402 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:54 smithi115 ceph-mon[72476]: 1.1 scrub starts 2024-03-20T09:26:55.402 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:54 smithi115 ceph-mon[72476]: 1.1 scrub ok 2024-03-20T09:26:55.402 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:54 smithi115 ceph-mon[72476]: pgmap v1888: 57 pgs: 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 3 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 40 active+clean; 6.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22797/302754 objects degraded (7.530%); 29195/302754 objects misplaced (9.643%) 2024-03-20T09:26:55.402 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:54 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:26:55.402 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:54 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:26:55.675 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:26:55.675 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:26:55.675 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-03-20T09:26:55.675 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-03-20T09:26:55.675 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:26:56.216 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:55 smithi026 ceph-mon[68309]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-20T09:26:56.402 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:55 smithi115 ceph-mon[72476]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-20T09:26:56.442 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:55 smithi122 ceph-mon[73495]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-20T09:26:57.181 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:56 smithi122 ceph-mon[73495]: pgmap v1889: 57 pgs: 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 3 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 40 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22669/300471 objects degraded (7.544%); 27753/300471 objects misplaced (9.236%) 2024-03-20T09:26:57.402 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:56 smithi115 ceph-mon[72476]: pgmap v1889: 57 pgs: 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 3 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 40 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22669/300471 objects degraded (7.544%); 27753/300471 objects misplaced (9.236%) 2024-03-20T09:26:57.466 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:56 smithi026 ceph-mon[68309]: pgmap v1889: 57 pgs: 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 3 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 40 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22669/300471 objects degraded (7.544%); 27753/300471 objects misplaced (9.236%) 2024-03-20T09:26:58.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:58 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:26:58.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:58 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:26:58.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:58 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 22669/300471 objects degraded (7.544%), 7 pgs degraded (PG_DEGRADED) 2024-03-20T09:26:58.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:58 smithi115 ceph-mon[72476]: osdmap e780: 12 total, 11 up, 12 in 2024-03-20T09:26:58.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:58 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:26:58.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:58 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:26:58.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:58 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:26:58.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:58 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:26:58.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:58 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:26:58.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:58 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 22669/300471 objects degraded (7.544%), 7 pgs degraded (PG_DEGRADED) 2024-03-20T09:26:58.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:58 smithi122 ceph-mon[73495]: osdmap e780: 12 total, 11 up, 12 in 2024-03-20T09:26:58.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:58 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:26:58.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:58 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:26:58.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:58 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:26:58.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:58 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:26:58.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:58 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:26:58.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:58 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 22669/300471 objects degraded (7.544%), 7 pgs degraded (PG_DEGRADED) 2024-03-20T09:26:58.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:58 smithi026 ceph-mon[68309]: osdmap e780: 12 total, 11 up, 12 in 2024-03-20T09:26:58.720 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:58 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:26:58.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:58 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:26:58.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:58 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:26:59.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:26:59 smithi115 ceph-mon[72476]: pgmap v1891: 57 pgs: 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 3 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 40 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 207 op/s; 22109/300237 objects degraded (7.364%); 26725/300237 objects misplaced (8.901%); 16 MiB/s, 261 objects/s recovering 2024-03-20T09:26:59.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:26:59 smithi122 ceph-mon[73495]: pgmap v1891: 57 pgs: 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 3 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 40 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 207 op/s; 22109/300237 objects degraded (7.364%); 26725/300237 objects misplaced (8.901%); 16 MiB/s, 261 objects/s recovering 2024-03-20T09:26:59.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:26:59 smithi026 ceph-mon[68309]: pgmap v1891: 57 pgs: 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 3 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 40 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 207 op/s; 22109/300237 objects degraded (7.364%); 26725/300237 objects misplaced (8.901%); 16 MiB/s, 261 objects/s recovering 2024-03-20T09:27:01.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:01 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:27:01.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:01 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:27:01.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:01 smithi115 ceph-mon[72476]: osdmap e781: 12 total, 11 up, 12 in 2024-03-20T09:27:01.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:01 smithi115 ceph-mon[72476]: pgmap v1893: 57 pgs: 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 3 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 40 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 282 op/s; 21993/298155 objects degraded (7.376%); 24092/298155 objects misplaced (8.080%); 23 MiB/s, 372 objects/s recovering 2024-03-20T09:27:01.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:01 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:27:01.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:01 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:27:01.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:01 smithi122 ceph-mon[73495]: osdmap e781: 12 total, 11 up, 12 in 2024-03-20T09:27:01.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:01 smithi122 ceph-mon[73495]: pgmap v1893: 57 pgs: 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 3 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 40 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 282 op/s; 21993/298155 objects degraded (7.376%); 24092/298155 objects misplaced (8.080%); 23 MiB/s, 372 objects/s recovering 2024-03-20T09:27:01.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:01 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:27:01.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:01 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:27:01.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:01 smithi026 ceph-mon[68309]: osdmap e781: 12 total, 11 up, 12 in 2024-03-20T09:27:01.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:01 smithi026 ceph-mon[68309]: pgmap v1893: 57 pgs: 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 3 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 40 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 282 op/s; 21993/298155 objects degraded (7.376%); 24092/298155 objects misplaced (8.080%); 23 MiB/s, 372 objects/s recovering 2024-03-20T09:27:02.479 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:27:02.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:02.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:27:02.588 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:27:02.616 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:02 smithi115 ceph-mon[72476]: 1.1c scrub starts 2024-03-20T09:27:02.616 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:02 smithi115 ceph-mon[72476]: 1.1c scrub ok 2024-03-20T09:27:02.616 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:02 smithi115 ceph-mon[72476]: osdmap e782: 12 total, 11 up, 12 in 2024-03-20T09:27:02.619 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:02.639 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:27:02.641 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:02 smithi026 ceph-mon[68309]: 1.1c scrub starts 2024-03-20T09:27:02.641 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:02 smithi026 ceph-mon[68309]: 1.1c scrub ok 2024-03-20T09:27:02.641 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:02 smithi026 ceph-mon[68309]: osdmap e782: 12 total, 11 up, 12 in 2024-03-20T09:27:02.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:02.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:27:02.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:02 smithi122 ceph-mon[73495]: 1.1c scrub starts 2024-03-20T09:27:02.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:02 smithi122 ceph-mon[73495]: 1.1c scrub ok 2024-03-20T09:27:02.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:02 smithi122 ceph-mon[73495]: osdmap e782: 12 total, 11 up, 12 in 2024-03-20T09:27:02.743 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:27:02.774 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:02.796 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:27:02.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:02.799 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:27:02.900 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:27:02.928 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:02.948 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:27:02.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:02.950 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:27:03.051 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:27:03.079 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:03.097 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:27:03.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:03.100 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:27:03.201 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:27:03.253 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:03.442 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:03 smithi122 ceph-mon[73495]: 1.1b scrub starts 2024-03-20T09:27:03.442 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:03 smithi122 ceph-mon[73495]: 1.1b scrub ok 2024-03-20T09:27:03.442 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:03 smithi122 ceph-mon[73495]: pgmap v1895: 57 pgs: 1 peering, 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 2 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 40 active+clean; 6.0 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 226 op/s; 21823/296385 objects degraded (7.363%); 23823/296385 objects misplaced (8.038%); 19 MiB/s, 355 objects/s recovering 2024-03-20T09:27:03.442 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:03 smithi122 ceph-mon[73495]: osdmap e783: 12 total, 11 up, 12 in 2024-03-20T09:27:03.442 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:03 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 21993/298155 objects degraded (7.376%), 7 pgs degraded (PG_DEGRADED) 2024-03-20T09:27:03.442 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:03 smithi122 ceph-mon[73495]: osdmap e784: 12 total, 11 up, 12 in 2024-03-20T09:27:03.455 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:03 smithi026 ceph-mon[68309]: 1.1b scrub starts 2024-03-20T09:27:03.455 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:03 smithi026 ceph-mon[68309]: 1.1b scrub ok 2024-03-20T09:27:03.455 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:03 smithi026 ceph-mon[68309]: pgmap v1895: 57 pgs: 1 peering, 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 2 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 40 active+clean; 6.0 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 226 op/s; 21823/296385 objects degraded (7.363%); 23823/296385 objects misplaced (8.038%); 19 MiB/s, 355 objects/s recovering 2024-03-20T09:27:03.456 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:03 smithi026 ceph-mon[68309]: osdmap e783: 12 total, 11 up, 12 in 2024-03-20T09:27:03.456 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:03 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 21993/298155 objects degraded (7.376%), 7 pgs degraded (PG_DEGRADED) 2024-03-20T09:27:03.456 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:03 smithi026 ceph-mon[68309]: osdmap e784: 12 total, 11 up, 12 in 2024-03-20T09:27:03.462 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:27:03.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:03.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:27:03.568 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:27:03.612 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:03.653 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:03 smithi115 ceph-mon[72476]: 1.1b scrub starts 2024-03-20T09:27:03.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:03 smithi115 ceph-mon[72476]: 1.1b scrub ok 2024-03-20T09:27:03.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:03 smithi115 ceph-mon[72476]: pgmap v1895: 57 pgs: 1 peering, 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 2 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 40 active+clean; 6.0 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 226 op/s; 21823/296385 objects degraded (7.363%); 23823/296385 objects misplaced (8.038%); 19 MiB/s, 355 objects/s recovering 2024-03-20T09:27:03.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:03 smithi115 ceph-mon[72476]: osdmap e783: 12 total, 11 up, 12 in 2024-03-20T09:27:03.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:03 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 21993/298155 objects degraded (7.376%), 7 pgs degraded (PG_DEGRADED) 2024-03-20T09:27:03.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:03 smithi115 ceph-mon[72476]: osdmap e784: 12 total, 11 up, 12 in 2024-03-20T09:27:03.683 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:27:03.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:03.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:27:03.796 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:27:03.830 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:03.854 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:27:03.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:03.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:27:03.964 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:27:03.994 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:04.016 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:27:04.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:04.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:27:04.120 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:27:04.167 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:05.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:05 smithi115 ceph-mon[72476]: pgmap v1898: 57 pgs: 1 peering, 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 2 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 40 active+clean; 6.0 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 210 op/s; 21692/295614 objects degraded (7.338%); 23686/295614 objects misplaced (8.012%); 3.2 MiB/s, 128 objects/s recovering 2024-03-20T09:27:05.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:05 smithi122 ceph-mon[73495]: pgmap v1898: 57 pgs: 1 peering, 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 2 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 40 active+clean; 6.0 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 210 op/s; 21692/295614 objects degraded (7.338%); 23686/295614 objects misplaced (8.012%); 3.2 MiB/s, 128 objects/s recovering 2024-03-20T09:27:05.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:05 smithi026 ceph-mon[68309]: pgmap v1898: 57 pgs: 1 peering, 3 active+undersized+degraded, 3 active+recovering+undersized+degraded+remapped, 7 active+undersized, 2 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 40 active+clean; 6.0 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 210 op/s; 21692/295614 objects degraded (7.338%); 23686/295614 objects misplaced (8.012%); 3.2 MiB/s, 128 objects/s recovering 2024-03-20T09:27:05.836 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:27:05.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:05.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:27:05.947 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:27:05.993 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:07.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:07 smithi115 ceph-mon[72476]: pgmap v1899: 57 pgs: 1 peering, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 2 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 41 active+clean; 6.0 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 298 op/s; 21467/292773 objects degraded (7.332%); 21362/292773 objects misplaced (7.296%); 12 MiB/s, 305 objects/s recovering 2024-03-20T09:27:07.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:07 smithi122 ceph-mon[73495]: pgmap v1899: 57 pgs: 1 peering, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 2 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 41 active+clean; 6.0 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 298 op/s; 21467/292773 objects degraded (7.332%); 21362/292773 objects misplaced (7.296%); 12 MiB/s, 305 objects/s recovering 2024-03-20T09:27:07.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:07 smithi026 ceph-mon[68309]: pgmap v1899: 57 pgs: 1 peering, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 2 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 41 active+clean; 6.0 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 298 op/s; 21467/292773 objects degraded (7.332%); 21362/292773 objects misplaced (7.296%); 12 MiB/s, 305 objects/s recovering 2024-03-20T09:27:07.855 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:27:07.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:07.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:27:07.963 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:27:07.991 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:08.010 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:27:08.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:08.013 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:27:08.114 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:27:08.144 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:08.164 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:27:08.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:08.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:27:08.271 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:27:08.330 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:08.401 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:08 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 21467/292773 objects degraded (7.332%), 6 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-03-20T09:27:08.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:08 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 21467/292773 objects degraded (7.332%), 6 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-03-20T09:27:08.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:08 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 21467/292773 objects degraded (7.332%), 6 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-03-20T09:27:09.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:09 smithi115 ceph-mon[72476]: pgmap v1900: 57 pgs: 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 2 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 42 active+clean; 5.9 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 221 op/s; 21421/292098 objects degraded (7.333%); 21362/292098 objects misplaced (7.313%); 9.8 MiB/s, 249 objects/s recovering 2024-03-20T09:27:09.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:09 smithi122 ceph-mon[73495]: pgmap v1900: 57 pgs: 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 2 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 42 active+clean; 5.9 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 221 op/s; 21421/292098 objects degraded (7.333%); 21362/292098 objects misplaced (7.313%); 9.8 MiB/s, 249 objects/s recovering 2024-03-20T09:27:09.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:09 smithi026 ceph-mon[68309]: pgmap v1900: 57 pgs: 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 2 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 42 active+clean; 5.9 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 221 op/s; 21421/292098 objects degraded (7.333%); 21362/292098 objects misplaced (7.313%); 9.8 MiB/s, 249 objects/s recovering 2024-03-20T09:27:10.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:10 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:27:10.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:10 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:27:10.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:10 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:27:10.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:10 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:27:10.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:10 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:27:10.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:10 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:27:11.214 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:27:11.215 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:27:11.215 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:27:11.215 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:27:11.215 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:27:11.215 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:27:11.215 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:27:11.215 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:27:11.215 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:27:11.215 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:27:11.220 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:27:11.221 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:27:11.221 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:27:11.221 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:27:11.221 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:27:11.221 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:27:11.221 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:27:11.221 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:27:11.221 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:27:11.221 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:27:11.221 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:27:11.221 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-03-20T09:27:11.222 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:27:11.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:11 smithi115 ceph-mon[72476]: pgmap v1901: 57 pgs: 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 2 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 42 active+clean; 5.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 271 op/s; 21283/289857 objects degraded (7.343%); 18919/289857 objects misplaced (6.527%); 16 MiB/s, 294 objects/s recovering 2024-03-20T09:27:11.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:11 smithi115 ceph-mon[72476]: osdmap e785: 12 total, 11 up, 12 in 2024-03-20T09:27:11.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:11 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2643036945' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:27:11.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:11 smithi122 ceph-mon[73495]: pgmap v1901: 57 pgs: 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 2 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 42 active+clean; 5.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 271 op/s; 21283/289857 objects degraded (7.343%); 18919/289857 objects misplaced (6.527%); 16 MiB/s, 294 objects/s recovering 2024-03-20T09:27:11.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:11 smithi122 ceph-mon[73495]: osdmap e785: 12 total, 11 up, 12 in 2024-03-20T09:27:11.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:11 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2643036945' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:27:11.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:11 smithi026 ceph-mon[68309]: pgmap v1901: 57 pgs: 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 2 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 42 active+clean; 5.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 271 op/s; 21283/289857 objects degraded (7.343%); 18919/289857 objects misplaced (6.527%); 16 MiB/s, 294 objects/s recovering 2024-03-20T09:27:11.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:11 smithi026 ceph-mon[68309]: osdmap e785: 12 total, 11 up, 12 in 2024-03-20T09:27:11.719 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:11 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2643036945' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:27:13.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:13 smithi115 ceph-mon[72476]: pgmap v1903: 57 pgs: 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 2 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 42 active+clean; 5.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 298 op/s; 21081/287625 objects degraded (7.329%); 18701/287625 objects misplaced (6.502%); 14 MiB/s, 252 objects/s recovering 2024-03-20T09:27:13.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:13 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 21081/287625 objects degraded (7.329%), 6 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-03-20T09:27:13.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:13 smithi115 ceph-mon[72476]: osdmap e786: 12 total, 11 up, 12 in 2024-03-20T09:27:13.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:13 smithi122 ceph-mon[73495]: pgmap v1903: 57 pgs: 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 2 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 42 active+clean; 5.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 298 op/s; 21081/287625 objects degraded (7.329%); 18701/287625 objects misplaced (6.502%); 14 MiB/s, 252 objects/s recovering 2024-03-20T09:27:13.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:13 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 21081/287625 objects degraded (7.329%), 6 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-03-20T09:27:13.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:13 smithi122 ceph-mon[73495]: osdmap e786: 12 total, 11 up, 12 in 2024-03-20T09:27:13.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:13 smithi026 ceph-mon[68309]: pgmap v1903: 57 pgs: 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 2 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 42 active+clean; 5.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 298 op/s; 21081/287625 objects degraded (7.329%); 18701/287625 objects misplaced (6.502%); 14 MiB/s, 252 objects/s recovering 2024-03-20T09:27:13.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:13 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 21081/287625 objects degraded (7.329%), 6 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-03-20T09:27:13.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:13 smithi026 ceph-mon[68309]: osdmap e786: 12 total, 11 up, 12 in 2024-03-20T09:27:14.024 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-20T09:27:14.842 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:27:14.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:14.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:27:14.947 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:27:14.974 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:15.084 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:27:15.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:15.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:27:15.196 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:27:15.224 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:15.243 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:27:15.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:15.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:27:15.350 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:27:15.389 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:15.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:15 smithi115 ceph-mon[72476]: pgmap v1905: 57 pgs: 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 2 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 42 active+clean; 5.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 254 op/s; 20937/286659 objects degraded (7.304%); 18573/286659 objects misplaced (6.479%); 8.9 MiB/s, 143 objects/s recovering 2024-03-20T09:27:15.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:15 smithi122 ceph-mon[73495]: pgmap v1905: 57 pgs: 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 2 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 42 active+clean; 5.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 254 op/s; 20937/286659 objects degraded (7.304%); 18573/286659 objects misplaced (6.479%); 8.9 MiB/s, 143 objects/s recovering 2024-03-20T09:27:15.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:15 smithi026 ceph-mon[68309]: pgmap v1905: 57 pgs: 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 2 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 42 active+clean; 5.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 254 op/s; 20937/286659 objects degraded (7.304%); 18573/286659 objects misplaced (6.479%); 8.9 MiB/s, 143 objects/s recovering 2024-03-20T09:27:15.783 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:27:16.348 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:27:16.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:16 smithi115 ceph-mon[72476]: osdmap e787: 12 total, 11 up, 12 in 2024-03-20T09:27:16.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:16 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1247141271' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:27:16.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:16 smithi122 ceph-mon[73495]: osdmap e787: 12 total, 11 up, 12 in 2024-03-20T09:27:16.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:16 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1247141271' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:27:16.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:16 smithi026 ceph-mon[68309]: osdmap e787: 12 total, 11 up, 12 in 2024-03-20T09:27:16.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:16 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1247141271' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:27:17.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:17 smithi115 ceph-mon[72476]: pgmap v1907: 57 pgs: 1 active+undersized+remapped, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 1 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 42 active+clean; 5.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 334 op/s; 20937/283839 objects degraded (7.376%); 16496/283839 objects misplaced (5.812%); 11 MiB/s, 214 objects/s recovering 2024-03-20T09:27:17.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:17 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:27:17.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:17 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1247141271' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:27:17.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:17 smithi115 ceph-mon[72476]: osdmap e788: 12 total, 11 up, 12 in 2024-03-20T09:27:17.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:17 smithi122 ceph-mon[73495]: pgmap v1907: 57 pgs: 1 active+undersized+remapped, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 1 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 42 active+clean; 5.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 334 op/s; 20937/283839 objects degraded (7.376%); 16496/283839 objects misplaced (5.812%); 11 MiB/s, 214 objects/s recovering 2024-03-20T09:27:17.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:17 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:27:17.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:17 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1247141271' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:27:17.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:17 smithi122 ceph-mon[73495]: osdmap e788: 12 total, 11 up, 12 in 2024-03-20T09:27:17.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:17 smithi026 ceph-mon[68309]: pgmap v1907: 57 pgs: 1 active+undersized+remapped, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 7 active+undersized, 1 active+recovering+undersized+remapped, 1 active+recovering+degraded+remapped, 42 active+clean; 5.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 334 op/s; 20937/283839 objects degraded (7.376%); 16496/283839 objects misplaced (5.812%); 11 MiB/s, 214 objects/s recovering 2024-03-20T09:27:17.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:17 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:27:17.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:17 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1247141271' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:27:17.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:17 smithi026 ceph-mon[68309]: osdmap e788: 12 total, 11 up, 12 in 2024-03-20T09:27:18.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:18 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 20937/283839 objects degraded (7.376%), 6 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-03-20T09:27:18.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:18 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 20937/283839 objects degraded (7.376%), 6 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-03-20T09:27:18.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:18 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 20937/283839 objects degraded (7.376%), 6 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-03-20T09:27:19.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:19 smithi115 ceph-mon[72476]: pgmap v1909: 57 pgs: 1 peering, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 42 active+clean; 5.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 299 op/s; 20719/282240 objects degraded (7.341%); 16166/282240 objects misplaced (5.728%); 11 MiB/s, 224 objects/s recovering 2024-03-20T09:27:19.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:19 smithi122 ceph-mon[73495]: pgmap v1909: 57 pgs: 1 peering, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 42 active+clean; 5.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 299 op/s; 20719/282240 objects degraded (7.341%); 16166/282240 objects misplaced (5.728%); 11 MiB/s, 224 objects/s recovering 2024-03-20T09:27:19.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:19 smithi026 ceph-mon[68309]: pgmap v1909: 57 pgs: 1 peering, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 42 active+clean; 5.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 299 op/s; 20719/282240 objects degraded (7.341%); 16166/282240 objects misplaced (5.728%); 11 MiB/s, 224 objects/s recovering 2024-03-20T09:27:21.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:21 smithi115 ceph-mon[72476]: pgmap v1910: 57 pgs: 1 peering, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 42 active+clean; 5.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 333 op/s; 20557/279846 objects degraded (7.346%); 14666/279846 objects misplaced (5.241%); 15 MiB/s, 278 objects/s recovering 2024-03-20T09:27:21.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:21 smithi122 ceph-mon[73495]: pgmap v1910: 57 pgs: 1 peering, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 42 active+clean; 5.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 333 op/s; 20557/279846 objects degraded (7.346%); 14666/279846 objects misplaced (5.241%); 15 MiB/s, 278 objects/s recovering 2024-03-20T09:27:21.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:21 smithi026 ceph-mon[68309]: pgmap v1910: 57 pgs: 1 peering, 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 42 active+clean; 5.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 333 op/s; 20557/279846 objects degraded (7.346%); 14666/279846 objects misplaced (5.241%); 15 MiB/s, 278 objects/s recovering 2024-03-20T09:27:22.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:22 smithi115 ceph-mon[72476]: osdmap e789: 12 total, 11 up, 12 in 2024-03-20T09:27:22.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:22 smithi122 ceph-mon[73495]: osdmap e789: 12 total, 11 up, 12 in 2024-03-20T09:27:22.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:22 smithi026 ceph-mon[68309]: osdmap e789: 12 total, 11 up, 12 in 2024-03-20T09:27:23.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:23 smithi115 ceph-mon[72476]: 1.15 scrub starts 2024-03-20T09:27:23.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:23 smithi115 ceph-mon[72476]: pgmap v1912: 57 pgs: 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 43 active+clean; 5.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 379 op/s; 20334/277056 objects degraded (7.339%); 14404/277056 objects misplaced (5.199%); 9.5 MiB/s, 160 objects/s recovering 2024-03-20T09:27:23.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:23 smithi115 ceph-mon[72476]: 1.15 scrub ok 2024-03-20T09:27:23.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:23 smithi115 ceph-mon[72476]: osdmap e790: 12 total, 11 up, 12 in 2024-03-20T09:27:23.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:23 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 20334/277056 objects degraded (7.339%), 6 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-03-20T09:27:23.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:23 smithi122 ceph-mon[73495]: 1.15 scrub starts 2024-03-20T09:27:23.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:23 smithi122 ceph-mon[73495]: pgmap v1912: 57 pgs: 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 43 active+clean; 5.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 379 op/s; 20334/277056 objects degraded (7.339%); 14404/277056 objects misplaced (5.199%); 9.5 MiB/s, 160 objects/s recovering 2024-03-20T09:27:23.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:23 smithi122 ceph-mon[73495]: 1.15 scrub ok 2024-03-20T09:27:23.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:23 smithi122 ceph-mon[73495]: osdmap e790: 12 total, 11 up, 12 in 2024-03-20T09:27:23.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:23 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 20334/277056 objects degraded (7.339%), 6 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-03-20T09:27:23.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:23 smithi026 ceph-mon[68309]: 1.15 scrub starts 2024-03-20T09:27:23.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:23 smithi026 ceph-mon[68309]: pgmap v1912: 57 pgs: 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 43 active+clean; 5.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 379 op/s; 20334/277056 objects degraded (7.339%); 14404/277056 objects misplaced (5.199%); 9.5 MiB/s, 160 objects/s recovering 2024-03-20T09:27:23.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:23 smithi026 ceph-mon[68309]: 1.15 scrub ok 2024-03-20T09:27:23.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:23 smithi026 ceph-mon[68309]: osdmap e790: 12 total, 11 up, 12 in 2024-03-20T09:27:23.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:23 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 20334/277056 objects degraded (7.339%), 6 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-03-20T09:27:24.245 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:27:24.246 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:27:24.680 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:27:24.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:24.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:27:24.785 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:27:24.820 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:25.224 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:27:25.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:25.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:27:25.262 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:27:25.330 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:27:25.356 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:25.379 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:27:25.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:25.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:27:25.483 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:27:25.510 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:25.530 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:27:25.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:25.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:27:25.634 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:27:25.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:25 smithi115 ceph-mon[72476]: pgmap v1914: 57 pgs: 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 43 active+clean; 5.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 225 op/s; 20334/277056 objects degraded (7.339%); 14404/277056 objects misplaced (5.199%); 7.0 MiB/s, 119 objects/s recovering 2024-03-20T09:27:25.661 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:25.683 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:27:25.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:25.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:27:25.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:25 smithi122 ceph-mon[73495]: pgmap v1914: 57 pgs: 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 43 active+clean; 5.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 225 op/s; 20334/277056 objects degraded (7.339%); 14404/277056 objects misplaced (5.199%); 7.0 MiB/s, 119 objects/s recovering 2024-03-20T09:27:25.706 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:25 smithi026 ceph-mon[68309]: pgmap v1914: 57 pgs: 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 43 active+clean; 5.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 225 op/s; 20334/277056 objects degraded (7.339%); 14404/277056 objects misplaced (5.199%); 7.0 MiB/s, 119 objects/s recovering 2024-03-20T09:27:25.788 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:27:25.814 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:25.832 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:27:25.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:25.835 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:27:25.936 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:27:25.965 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:25.986 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:27:25.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:25.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:27:26.090 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:27:26.137 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:26.180 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:27:26.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:26.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:27:26.247 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:27:26.292 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:27:26.319 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:26.338 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:27:26.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:26.341 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:27:26.441 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:27:26.482 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:26.514 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:27:26.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:26.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:27:26.622 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:27:26.648 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:26.666 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:27:26.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:26.669 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:27:26.770 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:27:26.805 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:27.271 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:27:27.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:27 smithi115 ceph-mon[72476]: pgmap v1915: 57 pgs: 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 43 active+clean; 5.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 365 op/s; 20130/273468 objects degraded (7.361%); 14277/273468 objects misplaced (5.221%); 7.4 MiB/s, 126 objects/s recovering 2024-03-20T09:27:27.659 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:27:27.659 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:27:27.659 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 790, 2024-03-20T09:27:27.659 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "852fbb68-e696-11ee-95c9-87774f69a715", 2024-03-20T09:27:27.660 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-03-20T08:48:35.281088+0000", 2024-03-20T09:27:27.660 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-03-20T09:27:22.353467+0000", 2024-03-20T09:27:27.660 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2024-03-20T09:26:03.473344+0000", 2024-03-20T09:27:27.660 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2024-03-20T09:20:22.151285+0000", 2024-03-20T09:27:27.660 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-20T09:27:27.660 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5799936, 2024-03-20T09:27:27.660 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2024-03-20T09:27:27.660 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2024-03-20T09:27:27.660 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2024-03-20T09:27:27.660 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2024-03-20T09:27:27.660 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2024-03-20T09:27:27.660 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:27:27.660 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 27, 2024-03-20T09:27:27.661 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2024-03-20T09:27:27.661 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-20T09:27:27.661 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-20T09:27:27.661 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2024-03-20T09:27:27.661 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 6, 2024-03-20T09:27:27.661 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 12, 2024-03-20T09:27:27.661 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "luminous", 2024-03-20T09:27:27.661 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "luminous", 2024-03-20T09:27:27.661 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "squid", 2024-03-20T09:27:27.661 INFO:teuthology.orchestra.run.smithi026.stdout: "allow_crimson": false, 2024-03-20T09:27:27.661 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2024-03-20T09:27:27.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.661 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2024-03-20T09:27:27.661 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": ".mgr", 2024-03-20T09:27:27.662 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-03-20T08:52:17.442324+0000", 2024-03-20T09:27:27.662 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-03-20T09:27:27.662 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-03-20T09:27:27.662 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-03-20T09:27:27.662 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 3, 2024-03-20T09:27:27.662 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 2, 2024-03-20T09:27:27.662 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-03-20T09:27:27.662 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-03-20T09:27:27.662 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-03-20T09:27:27.662 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-03-20T09:27:27.662 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-20T09:27:27.662 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-03-20T09:27:27.662 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-03-20T09:27:27.662 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 31, 2024-03-20T09:27:27.663 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 31, 2024-03-20T09:27:27.663 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 31, 2024-03-20T09:27:27.663 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 31, 2024-03-20T09:27:27.663 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 31, 2024-03-20T09:27:27.663 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-03-20T09:27:27.663 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-03-20T09:27:27.663 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-03-20T09:27:27.663 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-03-20T09:27:27.663 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-03-20T09:27:27.663 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-03-20T09:27:27.663 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-03-20T09:27:27.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.663 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "360", 2024-03-20T09:27:27.664 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-03-20T09:27:27.664 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-20T09:27:27.664 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "358", 2024-03-20T09:27:27.664 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-03-20T09:27:27.664 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-03-20T09:27:27.664 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-03-20T09:27:27.664 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-03-20T09:27:27.665 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-03-20T09:27:27.665 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-03-20T09:27:27.665 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-03-20T09:27:27.665 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-03-20T09:27:27.665 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-03-20T09:27:27.665 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-03-20T09:27:27.665 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-03-20T09:27:27.665 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-03-20T09:27:27.665 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-03-20T09:27:27.665 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-03-20T09:27:27.665 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-03-20T09:27:27.665 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-20T09:27:27.666 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-20T09:27:27.666 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-20T09:27:27.666 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-03-20T09:27:27.666 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-03-20T09:27:27.666 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-03-20T09:27:27.666 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-03-20T09:27:27.666 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-03-20T09:27:27.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.666 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-03-20T09:27:27.666 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-03-20T09:27:27.666 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-03-20T09:27:27.666 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-03-20T09:27:27.666 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-03-20T09:27:27.667 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-03-20T09:27:27.667 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-03-20T09:27:27.667 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-03-20T09:27:27.667 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-03-20T09:27:27.667 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-03-20T09:27:27.667 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-03-20T09:27:27.667 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2024-03-20T09:27:27.667 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_max": 32, 2024-03-20T09:27:27.667 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2024-03-20T09:27:27.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.667 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-03-20T09:27:27.667 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": {} 2024-03-20T09:27:27.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.667 INFO:teuthology.orchestra.run.smithi026.stdout: "read_balance": { 2024-03-20T09:27:27.668 INFO:teuthology.orchestra.run.smithi026.stdout: "score_type": "Fair distribution", 2024-03-20T09:27:27.668 INFO:teuthology.orchestra.run.smithi026.stdout: "score_acting": 1.5399999618530273, 2024-03-20T09:27:27.668 INFO:teuthology.orchestra.run.smithi026.stdout: "score_stable": 1.5399999618530273, 2024-03-20T09:27:27.668 INFO:teuthology.orchestra.run.smithi026.stdout: "optimal_score": 0.92000001668930054, 2024-03-20T09:27:27.668 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_acting": 1.4199999570846558, 2024-03-20T09:27:27.668 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_stable": 1.4199999570846558, 2024-03-20T09:27:27.668 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity_weighted": 0.92150616645812988, 2024-03-20T09:27:27.668 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity": 0.92000001668930054, 2024-03-20T09:27:27.668 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-03-20T09:27:27.668 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.668 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 6, 2024-03-20T09:27:27.668 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "unique_pool_4", 2024-03-20T09:27:27.669 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-03-20T09:24:02.337365+0000", 2024-03-20T09:27:27.669 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-03-20T09:27:27.669 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-03-20T09:27:27.669 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-03-20T09:27:27.669 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 3, 2024-03-20T09:27:27.669 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 2, 2024-03-20T09:27:27.669 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-03-20T09:27:27.669 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-03-20T09:27:27.669 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-03-20T09:27:27.669 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-03-20T09:27:27.669 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-20T09:27:27.669 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-03-20T09:27:27.669 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-03-20T09:27:27.670 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 26, 2024-03-20T09:27:27.670 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 26, 2024-03-20T09:27:27.670 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 26, 2024-03-20T09:27:27.670 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 26, 2024-03-20T09:27:27.670 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 26, 2024-03-20T09:27:27.670 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-03-20T09:27:27.670 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-03-20T09:27:27.670 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-03-20T09:27:27.670 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-03-20T09:27:27.670 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-03-20T09:27:27.670 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-03-20T09:27:27.670 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-03-20T09:27:27.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.670 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "751", 2024-03-20T09:27:27.671 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-03-20T09:27:27.671 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-20T09:27:27.671 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "749", 2024-03-20T09:27:27.671 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-03-20T09:27:27.671 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-03-20T09:27:27.671 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-03-20T09:27:27.671 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-03-20T09:27:27.671 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-03-20T09:27:27.671 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-03-20T09:27:27.671 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-03-20T09:27:27.671 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-03-20T09:27:27.671 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-03-20T09:27:27.671 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-03-20T09:27:27.671 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-03-20T09:27:27.672 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-03-20T09:27:27.672 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-03-20T09:27:27.672 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-03-20T09:27:27.672 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-03-20T09:27:27.672 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-20T09:27:27.672 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-20T09:27:27.672 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-20T09:27:27.672 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-03-20T09:27:27.672 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-03-20T09:27:27.672 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-03-20T09:27:27.672 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-03-20T09:27:27.672 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-03-20T09:27:27.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.672 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-03-20T09:27:27.673 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-03-20T09:27:27.673 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-03-20T09:27:27.673 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-03-20T09:27:27.673 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-03-20T09:27:27.673 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-03-20T09:27:27.673 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-03-20T09:27:27.673 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-03-20T09:27:27.673 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-03-20T09:27:27.673 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-03-20T09:27:27.673 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-03-20T09:27:27.673 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2024-03-20T09:27:27.673 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-03-20T09:27:27.673 INFO:teuthology.orchestra.run.smithi026.stdout: "rados": {} 2024-03-20T09:27:27.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.673 INFO:teuthology.orchestra.run.smithi026.stdout: "read_balance": { 2024-03-20T09:27:27.674 INFO:teuthology.orchestra.run.smithi026.stdout: "score_type": "Fair distribution", 2024-03-20T09:27:27.674 INFO:teuthology.orchestra.run.smithi026.stdout: "score_acting": 2.2999999523162842, 2024-03-20T09:27:27.674 INFO:teuthology.orchestra.run.smithi026.stdout: "score_stable": 2.2999999523162842, 2024-03-20T09:27:27.674 INFO:teuthology.orchestra.run.smithi026.stdout: "optimal_score": 0.92000001668930054, 2024-03-20T09:27:27.674 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_acting": 2.119999885559082, 2024-03-20T09:27:27.674 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_stable": 2.119999885559082, 2024-03-20T09:27:27.674 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity_weighted": 0.92150616645812988, 2024-03-20T09:27:27.675 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity": 0.92000001668930054, 2024-03-20T09:27:27.675 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-03-20T09:27:27.675 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.675 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.675 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:27:27.675 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:27:27.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.675 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-03-20T09:27:27.675 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "ad3fd9c0-316f-4fe5-a63b-d480087c1729", 2024-03-20T09:27:27.675 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:27:27.675 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:27:27.675 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:27:27.676 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:27:27.676 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:27:27.676 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:27:27.676 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 8, 2024-03-20T09:27:27.676 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 789, 2024-03-20T09:27:27.676 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:27:27.676 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:27:27.676 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:27:27.676 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.676 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.676 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2024-03-20T09:27:27.676 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:27:27.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.677 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.677 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2024-03-20T09:27:27.677 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:27:27.677 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.677 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.677 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:27:27.677 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.678 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.678 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2024-03-20T09:27:27.678 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:27:27.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.678 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.678 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2024-03-20T09:27:27.678 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:27:27.678 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.678 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.678 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:27:27.678 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.679 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.679 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2024-03-20T09:27:27.679 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:27:27.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.679 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.679 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2024-03-20T09:27:27.679 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:27:27.679 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.679 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.679 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:27:27.680 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.680 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.680 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2024-03-20T09:27:27.680 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:27:27.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.680 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.680 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2024-03-20T09:27:27.680 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:27:27.680 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.680 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.681 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/2139805293", 2024-03-20T09:27:27.681 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/2139805293", 2024-03-20T09:27:27.681 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/2139805293", 2024-03-20T09:27:27.681 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/2139805293", 2024-03-20T09:27:27.681 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:27:27.681 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:27:27.681 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:27:27.681 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.681 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-03-20T09:27:27.681 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e85b94f5-2954-47a1-9c20-096f2606eb04", 2024-03-20T09:27:27.681 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:27:27.682 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:27:27.682 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:27:27.682 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:27:27.682 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:27:27.682 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:27:27.682 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2024-03-20T09:27:27.682 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 786, 2024-03-20T09:27:27.682 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:27:27.682 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:27:27.682 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:27:27.682 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.682 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.683 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2024-03-20T09:27:27.683 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:27:27.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.683 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.683 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2024-03-20T09:27:27.683 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:27:27.683 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.683 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.683 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:27:27.683 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.683 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.684 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2024-03-20T09:27:27.684 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:27:27.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.684 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.684 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2024-03-20T09:27:27.684 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:27:27.684 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.684 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.684 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:27:27.684 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.685 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.685 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2024-03-20T09:27:27.685 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:27:27.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.686 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.686 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2024-03-20T09:27:27.686 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:27:27.686 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.686 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.686 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:27:27.686 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.686 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.686 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2024-03-20T09:27:27.686 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:27:27.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.687 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.687 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2024-03-20T09:27:27.687 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:27:27.687 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.687 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.687 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/3853355134", 2024-03-20T09:27:27.687 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/3853355134", 2024-03-20T09:27:27.687 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/3853355134", 2024-03-20T09:27:27.687 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/3853355134", 2024-03-20T09:27:27.687 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:27:27.687 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:27:27.687 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:27:27.688 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.688 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-03-20T09:27:27.688 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "dd2b10f0-741c-47a4-b224-ac6bfe2c14ef", 2024-03-20T09:27:27.688 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:27:27.688 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:27:27.688 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:27:27.688 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:27:27.688 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:27:27.688 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:27:27.688 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 18, 2024-03-20T09:27:27.688 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 787, 2024-03-20T09:27:27.688 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:27:27.689 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:27:27.689 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:27:27.689 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.689 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.689 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2024-03-20T09:27:27.689 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:27:27.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.689 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.689 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2024-03-20T09:27:27.689 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:27:27.689 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.689 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.690 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:27:27.690 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.690 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.690 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2024-03-20T09:27:27.690 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:27:27.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.690 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.690 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2024-03-20T09:27:27.690 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:27:27.690 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.690 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.691 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:27:27.691 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.691 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.691 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2024-03-20T09:27:27.691 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:27:27.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.691 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.691 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2024-03-20T09:27:27.691 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:27:27.691 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.691 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.692 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:27:27.692 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.692 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.692 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2024-03-20T09:27:27.692 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:27:27.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.692 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.692 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2024-03-20T09:27:27.692 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:27:27.692 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.692 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.692 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/3993357967", 2024-03-20T09:27:27.693 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/3993357967", 2024-03-20T09:27:27.693 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/3993357967", 2024-03-20T09:27:27.693 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/3993357967", 2024-03-20T09:27:27.693 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:27:27.693 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:27:27.693 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:27:27.693 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.693 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-03-20T09:27:27.693 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1", 2024-03-20T09:27:27.693 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:27:27.693 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:27:27.693 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:27:27.693 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:27:27.694 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:27:27.694 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:27:27.694 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 26, 2024-03-20T09:27:27.694 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 788, 2024-03-20T09:27:27.694 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:27:27.694 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:27:27.694 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:27:27.694 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.694 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.694 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2024-03-20T09:27:27.694 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:27:27.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.695 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.695 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2024-03-20T09:27:27.695 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:27:27.695 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.695 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.695 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:27:27.695 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.695 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.695 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2024-03-20T09:27:27.695 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:27:27.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.696 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.696 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2024-03-20T09:27:27.696 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:27:27.696 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.696 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.696 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:27:27.696 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.696 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.696 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2024-03-20T09:27:27.696 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:27:27.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.697 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.697 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2024-03-20T09:27:27.697 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:27:27.697 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.697 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.697 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:27:27.697 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.697 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.697 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2024-03-20T09:27:27.697 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:27:27.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.698 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.698 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2024-03-20T09:27:27.698 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:27:27.698 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.698 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.698 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/4110225509", 2024-03-20T09:27:27.698 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/4110225509", 2024-03-20T09:27:27.698 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/4110225509", 2024-03-20T09:27:27.698 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/4110225509", 2024-03-20T09:27:27.698 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:27:27.698 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:27:27.699 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:27:27.699 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.699 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-03-20T09:27:27.699 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "f14e34d5-9829-42a3-9b25-31a0976c5ef7", 2024-03-20T09:27:27.699 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:27:27.699 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:27:27.699 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:27:27.699 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:27:27.699 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:27:27.699 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:27:27.699 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 32, 2024-03-20T09:27:27.699 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 749, 2024-03-20T09:27:27.700 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:27:27.700 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:27:27.700 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:27:27.700 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.700 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.700 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6800", 2024-03-20T09:27:27.700 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:27:27.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.700 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.700 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6801", 2024-03-20T09:27:27.700 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:27:27.700 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.701 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.701 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:27:27.701 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.701 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.701 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6802", 2024-03-20T09:27:27.701 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:27:27.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.701 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.701 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6803", 2024-03-20T09:27:27.701 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:27:27.702 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.702 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.702 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:27:27.702 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.702 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.702 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6806", 2024-03-20T09:27:27.702 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:27:27.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.702 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.702 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6807", 2024-03-20T09:27:27.703 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:27:27.703 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.703 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.703 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:27:27.703 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.703 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.703 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6804", 2024-03-20T09:27:27.703 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:27:27.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.704 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.704 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6805", 2024-03-20T09:27:27.704 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:27:27.704 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.704 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.704 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6801/477923058", 2024-03-20T09:27:27.704 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6803/477923058", 2024-03-20T09:27:27.704 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6807/477923058", 2024-03-20T09:27:27.704 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6805/477923058", 2024-03-20T09:27:27.704 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:27:27.704 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:27:27.704 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:27:27.705 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.705 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-03-20T09:27:27.705 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "cb431b6a-540d-4318-ae35-d379e1df0185", 2024-03-20T09:27:27.705 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:27:27.705 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:27:27.705 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:27:27.706 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:27:27.706 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 41, 2024-03-20T09:27:27.706 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 273, 2024-03-20T09:27:27.706 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 443, 2024-03-20T09:27:27.706 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 755, 2024-03-20T09:27:27.706 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 274, 2024-03-20T09:27:27.706 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:27:27.706 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:27:27.706 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.706 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.706 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6808", 2024-03-20T09:27:27.706 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:27:27.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.707 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.707 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6809", 2024-03-20T09:27:27.707 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:27:27.707 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.707 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.707 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:27:27.707 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.707 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.707 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6810", 2024-03-20T09:27:27.708 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:27:27.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.708 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.708 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6811", 2024-03-20T09:27:27.708 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:27:27.708 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.708 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.708 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:27:27.708 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.708 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.709 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6814", 2024-03-20T09:27:27.709 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:27:27.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.709 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.709 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6815", 2024-03-20T09:27:27.709 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:27:27.709 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.709 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.709 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:27:27.709 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.710 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.710 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6812", 2024-03-20T09:27:27.710 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:27:27.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.710 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.710 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6813", 2024-03-20T09:27:27.710 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:27:27.710 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.710 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.710 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6809/650416648", 2024-03-20T09:27:27.710 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6811/650416648", 2024-03-20T09:27:27.710 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6815/650416648", 2024-03-20T09:27:27.711 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6813/650416648", 2024-03-20T09:27:27.711 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:27:27.711 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:27:27.711 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:27:27.711 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.711 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2024-03-20T09:27:27.711 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "753ed830-e764-4ff3-9d9b-87e7fa748b81", 2024-03-20T09:27:27.711 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:27:27.711 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:27:27.711 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:27:27.711 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:27:27.711 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:27:27.712 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:27:27.712 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 51, 2024-03-20T09:27:27.712 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 768, 2024-03-20T09:27:27.712 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:27:27.712 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:27:27.712 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:27:27.712 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.712 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.712 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6816", 2024-03-20T09:27:27.712 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:27:27.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.713 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.713 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6817", 2024-03-20T09:27:27.713 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:27:27.713 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.713 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.713 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:27:27.713 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.713 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.713 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6818", 2024-03-20T09:27:27.713 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:27:27.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.714 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.714 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6819", 2024-03-20T09:27:27.714 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:27:27.714 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.714 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.714 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:27:27.714 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.714 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.714 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6822", 2024-03-20T09:27:27.715 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:27:27.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.715 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.715 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6823", 2024-03-20T09:27:27.715 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:27:27.715 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.715 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.715 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:27:27.715 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.715 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.715 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6820", 2024-03-20T09:27:27.715 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:27:27.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.716 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.716 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6821", 2024-03-20T09:27:27.716 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:27:27.716 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.716 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.716 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6817/3278301843", 2024-03-20T09:27:27.716 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6819/3278301843", 2024-03-20T09:27:27.716 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6823/3278301843", 2024-03-20T09:27:27.716 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6821/3278301843", 2024-03-20T09:27:27.716 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:27:27.716 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:27:27.717 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:27:27.717 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.717 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2024-03-20T09:27:27.717 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "55bcbd37-7a6f-4ab1-b5a9-91a951444d86", 2024-03-20T09:27:27.717 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:27:27.717 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:27:27.717 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:27:27.717 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 0.058074951171875, 2024-03-20T09:27:27.717 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:27:27.717 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:27:27.717 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 60, 2024-03-20T09:27:27.718 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 517, 2024-03-20T09:27:27.718 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:27:27.718 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:27:27.718 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:27:27.718 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.718 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.718 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6824", 2024-03-20T09:27:27.718 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:27:27.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.718 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.718 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6825", 2024-03-20T09:27:27.719 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:27:27.719 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.719 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.719 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:27:27.719 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.719 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.719 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6826", 2024-03-20T09:27:27.719 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:27:27.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.719 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.719 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6827", 2024-03-20T09:27:27.720 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:27:27.720 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.720 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.720 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:27:27.720 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.720 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.720 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6830", 2024-03-20T09:27:27.720 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:27:27.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.720 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.720 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6831", 2024-03-20T09:27:27.721 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:27:27.721 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.721 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.721 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:27:27.721 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.721 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.721 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6828", 2024-03-20T09:27:27.721 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:27:27.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.721 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.721 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6829", 2024-03-20T09:27:27.722 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:27:27.722 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.722 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.722 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6825/3152424001", 2024-03-20T09:27:27.722 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6827/3152424001", 2024-03-20T09:27:27.722 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6831/3152424001", 2024-03-20T09:27:27.722 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6829/3152424001", 2024-03-20T09:27:27.722 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:27:27.722 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:27:27.722 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:27:27.722 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.723 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 8, 2024-03-20T09:27:27.723 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "bc86248e-6bd0-4369-b8a2-e593bc2a610d", 2024-03-20T09:27:27.723 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:27:27.723 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:27:27.723 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:27:27.723 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:27:27.723 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:27:27.723 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:27:27.723 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 70, 2024-03-20T09:27:27.723 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 780, 2024-03-20T09:27:27.723 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:27:27.723 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:27:27.723 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:27:27.724 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.724 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.724 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6800", 2024-03-20T09:27:27.724 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:27:27.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.724 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.724 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6801", 2024-03-20T09:27:27.724 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:27:27.724 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.724 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.724 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:27:27.725 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.725 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.725 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6802", 2024-03-20T09:27:27.725 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:27:27.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.725 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.725 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6803", 2024-03-20T09:27:27.725 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:27:27.725 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.725 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.725 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:27:27.726 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.726 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.726 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6806", 2024-03-20T09:27:27.726 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:27:27.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.726 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.726 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6807", 2024-03-20T09:27:27.726 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:27:27.726 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.726 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.727 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:27:27.727 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.727 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.727 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6804", 2024-03-20T09:27:27.727 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:27:27.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.727 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.727 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6805", 2024-03-20T09:27:27.727 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:27:27.727 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.727 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.728 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6801/1309426740", 2024-03-20T09:27:27.728 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6803/1309426740", 2024-03-20T09:27:27.728 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6807/1309426740", 2024-03-20T09:27:27.728 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6805/1309426740", 2024-03-20T09:27:27.728 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:27:27.728 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:27:27.728 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:27:27.728 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.728 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 9, 2024-03-20T09:27:27.728 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b20a4aa-032a-4c09-ac75-153c554424e7", 2024-03-20T09:27:27.728 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:27:27.729 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:27:27.729 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:27:27.729 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:27:27.729 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:27:27.729 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:27:27.729 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 80, 2024-03-20T09:27:27.729 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 754, 2024-03-20T09:27:27.729 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:27:27.729 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:27:27.729 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:27:27.729 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.729 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.729 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6808", 2024-03-20T09:27:27.729 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:27:27.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.730 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.730 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6809", 2024-03-20T09:27:27.730 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:27:27.730 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.730 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.730 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:27:27.730 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.730 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.730 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6810", 2024-03-20T09:27:27.730 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:27:27.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.731 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.731 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6811", 2024-03-20T09:27:27.731 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:27:27.731 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.731 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.731 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:27:27.731 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.731 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.731 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6814", 2024-03-20T09:27:27.731 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:27:27.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.732 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.732 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6815", 2024-03-20T09:27:27.732 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:27:27.732 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.732 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.732 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:27:27.732 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.732 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.732 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6812", 2024-03-20T09:27:27.732 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:27:27.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.733 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.733 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6813", 2024-03-20T09:27:27.733 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:27:27.733 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.733 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.733 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6809/2953210076", 2024-03-20T09:27:27.733 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6811/2953210076", 2024-03-20T09:27:27.733 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6815/2953210076", 2024-03-20T09:27:27.734 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6813/2953210076", 2024-03-20T09:27:27.734 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:27:27.734 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:27:27.734 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:27:27.734 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.734 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 10, 2024-03-20T09:27:27.734 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e62c1fb3-5dd1-4d0d-997c-d20633d19361", 2024-03-20T09:27:27.734 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 0, 2024-03-20T09:27:27.734 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:27:27.734 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:27:27.735 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:27:27.735 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:27:27.735 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:27:27.735 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 89, 2024-03-20T09:27:27.735 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 751, 2024-03-20T09:27:27.735 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 754, 2024-03-20T09:27:27.735 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:27:27.735 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:27:27.735 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.735 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.735 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6816", 2024-03-20T09:27:27.735 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:27:27.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.736 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.736 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6817", 2024-03-20T09:27:27.736 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:27:27.736 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.736 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.736 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:27:27.736 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.736 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.736 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6818", 2024-03-20T09:27:27.736 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:27:27.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.737 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.737 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6819", 2024-03-20T09:27:27.737 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:27:27.737 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.737 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.737 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:27:27.737 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.737 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.737 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6822", 2024-03-20T09:27:27.737 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:27:27.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.738 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.738 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6823", 2024-03-20T09:27:27.738 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:27:27.738 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.738 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.738 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:27:27.738 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.738 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.738 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6820", 2024-03-20T09:27:27.738 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:27:27.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.739 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.739 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6821", 2024-03-20T09:27:27.739 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 409394340 2024-03-20T09:27:27.739 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.739 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.739 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6817/409394340", 2024-03-20T09:27:27.739 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6819/409394340", 2024-03-20T09:27:27.739 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6823/409394340", 2024-03-20T09:27:27.739 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6821/409394340", 2024-03-20T09:27:27.739 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:27:27.739 INFO:teuthology.orchestra.run.smithi026.stdout: "exists" 2024-03-20T09:27:27.739 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.740 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 11, 2024-03-20T09:27:27.740 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b83994d5-e5ef-48b0-9758-c26843cec3b6", 2024-03-20T09:27:27.740 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:27:27.740 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:27:27.740 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:27:27.740 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:27:27.740 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:27:27.740 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:27:27.740 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 96, 2024-03-20T09:27:27.740 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 749, 2024-03-20T09:27:27.740 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:27:27.740 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:27:27.740 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:27:27.741 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.741 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.741 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6824", 2024-03-20T09:27:27.741 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:27:27.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.741 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.741 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6825", 2024-03-20T09:27:27.741 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:27:27.741 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.741 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.741 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:27:27.742 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.742 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.742 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6826", 2024-03-20T09:27:27.742 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:27:27.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.742 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.742 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6827", 2024-03-20T09:27:27.742 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:27:27.742 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.742 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.742 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:27:27.743 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.743 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.743 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6830", 2024-03-20T09:27:27.743 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:27:27.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.743 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.743 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6831", 2024-03-20T09:27:27.743 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:27:27.743 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.743 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.743 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:27:27.744 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:27:27.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.744 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:27:27.744 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6828", 2024-03-20T09:27:27.744 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:27:27.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.744 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:27:27.744 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6829", 2024-03-20T09:27:27.744 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:27:27.744 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.744 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.744 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6825/2704069354", 2024-03-20T09:27:27.745 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6827/2704069354", 2024-03-20T09:27:27.745 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6831/2704069354", 2024-03-20T09:27:27.745 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6829/2704069354", 2024-03-20T09:27:27.745 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:27:27.745 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:27:27.745 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:27:27.745 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.745 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.745 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:27:27.745 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2024-03-20T09:27:27.745 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.745 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-03-20T09:27:27.745 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:27:27.745 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:27:27.745 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:27:27.746 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:27:27.746 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:27:27.746 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:26:34.539891+0000", 2024-03-20T09:27:27.746 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:27:27.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.746 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-03-20T09:27:27.746 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:27:27.746 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:27:27.746 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:27:27.746 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:27:27.746 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:27:27.746 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:26:53.896921+0000", 2024-03-20T09:27:27.747 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:27:27.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.747 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-03-20T09:27:27.747 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:27:27.747 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:27:27.747 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:27:27.747 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:27:27.747 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:27:27.747 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:26:59.685265+0000", 2024-03-20T09:27:27.747 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:27:27.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.747 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-03-20T09:27:27.747 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:27:27.748 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:27:27.748 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:27:27.748 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:27:27.748 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:27:27.748 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:26:29.756266+0000", 2024-03-20T09:27:27.748 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:27:27.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.748 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-03-20T09:27:27.748 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:27:27.748 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:27:27.748 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:27:27.748 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:27:27.748 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:27:27.748 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:26:50.368636+0000", 2024-03-20T09:27:27.749 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:27:27.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.749 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-03-20T09:27:27.749 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:27:27.749 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:27:27.749 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:27:27.749 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:27:27.749 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:27:27.749 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:26:21.650602+0000", 2024-03-20T09:27:27.749 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 273 2024-03-20T09:27:27.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.750 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2024-03-20T09:27:27.750 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:27:27.750 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:27:27.750 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:27:27.750 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:27:27.751 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:27:27.751 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:26:42.847510+0000", 2024-03-20T09:27:27.751 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:27:27.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.751 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.751 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2024-03-20T09:27:27.751 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:27:27.751 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:27:27.751 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:27:27.751 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:27:27.751 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:27:27.751 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:27:09.310842+0000", 2024-03-20T09:27:27.751 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:27:27.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.752 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 8, 2024-03-20T09:27:27.752 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:27:27.752 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:27:27.752 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:27:27.752 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:27:27.752 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:27:27.752 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:26:46.717755+0000", 2024-03-20T09:27:27.752 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:27:27.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.752 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 9, 2024-03-20T09:27:27.752 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:27:27.753 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:27:27.753 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:27:27.753 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:27:27.753 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:27:27.753 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:26:45.830725+0000", 2024-03-20T09:27:27.753 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:27:27.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.753 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 10, 2024-03-20T09:27:27.753 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:27:27.753 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:27:27.753 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:27:27.753 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:27:27.753 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:27:27.754 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:25:18.432474+0000", 2024-03-20T09:27:27.754 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 753 2024-03-20T09:27:27.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.754 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 11, 2024-03-20T09:27:27.754 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:27:27.754 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:27:27.754 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:27:27.754 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:27:27.754 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:27:27.754 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:26:43.527199+0000", 2024-03-20T09:27:27.754 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:27:27.754 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.755 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:27:27.755 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [ 2024-03-20T09:27:27.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.755 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "1.1", 2024-03-20T09:27:27.755 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:27:27.755 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-03-20T09:27:27.755 INFO:teuthology.orchestra.run.smithi026.stdout: 11, 2024-03-20T09:27:27.755 INFO:teuthology.orchestra.run.smithi026.stdout: 5 2024-03-20T09:27:27.755 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.755 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.755 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:27:27.755 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [ 2024-03-20T09:27:27.755 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.755 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "1.1c", 2024-03-20T09:27:27.755 INFO:teuthology.orchestra.run.smithi026.stdout: "mappings": [ 2024-03-20T09:27:27.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.756 INFO:teuthology.orchestra.run.smithi026.stdout: "from": 6, 2024-03-20T09:27:27.756 INFO:teuthology.orchestra.run.smithi026.stdout: "to": 2 2024-03-20T09:27:27.756 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.756 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.756 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.756 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:27:27.756 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_primaries": [], 2024-03-20T09:27:27.756 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [ 2024-03-20T09:27:27.756 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.756 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "6.12", 2024-03-20T09:27:27.756 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:27:27.756 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-03-20T09:27:27.756 INFO:teuthology.orchestra.run.smithi026.stdout: 4 2024-03-20T09:27:27.757 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.757 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.757 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "6.17", 2024-03-20T09:27:27.757 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:27:27.757 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-03-20T09:27:27.757 INFO:teuthology.orchestra.run.smithi026.stdout: 3 2024-03-20T09:27:27.757 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.757 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.757 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:27:27.757 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "6.19", 2024-03-20T09:27:27.757 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:27:27.757 INFO:teuthology.orchestra.run.smithi026.stdout: 5, 2024-03-20T09:27:27.757 INFO:teuthology.orchestra.run.smithi026.stdout: 1, 2024-03-20T09:27:27.758 INFO:teuthology.orchestra.run.smithi026.stdout: 0 2024-03-20T09:27:27.758 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:27:27.758 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.758 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:27:27.758 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2024-03-20T09:27:27.758 INFO:teuthology.orchestra.run.smithi026.stdout: "blocklist": { 2024-03-20T09:27:27.758 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/4291404191": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:27:27.758 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/642012566": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:27:27.758 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6801/1181459999": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:27:27.758 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/3984513952": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:27:27.758 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6800/1654945498": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:27:27.758 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6800/1181459999": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:27:27.758 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/137770981": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:27:27.758 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/1451237762": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:27:27.759 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/3938386631": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:27:27.759 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6801/1654945498": "2024-03-21T08:49:04.596754+0000" 2024-03-20T09:27:27.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.759 INFO:teuthology.orchestra.run.smithi026.stdout: "range_blocklist": {}, 2024-03-20T09:27:27.759 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2024-03-20T09:27:27.759 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2024-03-20T09:27:27.759 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2024-03-20T09:27:27.759 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2024-03-20T09:27:27.759 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2024-03-20T09:27:27.759 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2024-03-20T09:27:27.759 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2024-03-20T09:27:27.759 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.759 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:27:27.759 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2024-03-20T09:27:27.760 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2024-03-20T09:27:27.760 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2024-03-20T09:27:27.760 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2024-03-20T09:27:27.760 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {}, 2024-03-20T09:27:27.760 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": { 2024-03-20T09:27:27.760 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_enabled": false, 2024-03-20T09:27:27.760 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_bucket_count": 0, 2024-03-20T09:27:27.760 INFO:teuthology.orchestra.run.smithi026.stdout: "degraded_stretch_mode": 0, 2024-03-20T09:27:27.760 INFO:teuthology.orchestra.run.smithi026.stdout: "recovering_stretch_mode": 0, 2024-03-20T09:27:27.760 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_bucket": 0 2024-03-20T09:27:27.760 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:27:27.760 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:27:27.761 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:27 smithi122 ceph-mon[73495]: pgmap v1915: 57 pgs: 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 43 active+clean; 5.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 365 op/s; 20130/273468 objects degraded (7.361%); 14277/273468 objects misplaced (5.221%); 7.4 MiB/s, 126 objects/s recovering 2024-03-20T09:27:27.761 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:27 smithi026 ceph-mon[68309]: pgmap v1915: 57 pgs: 3 active+undersized+degraded, 2 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 43 active+clean; 5.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 365 op/s; 20130/273468 objects degraded (7.361%); 14277/273468 objects misplaced (5.221%); 7.4 MiB/s, 126 objects/s recovering 2024-03-20T09:27:28.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:28 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 20130/273468 objects degraded (7.361%), 6 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-03-20T09:27:28.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:28 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3738363413' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:27:28.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:28 smithi115 ceph-mon[72476]: pgmap v1916: 57 pgs: 3 active+undersized+degraded, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 347 op/s; 19789/271512 objects degraded (7.288%); 13484/271512 objects misplaced (4.966%); 4.1 MiB/s, 111 objects/s recovering 2024-03-20T09:27:28.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:28 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 20130/273468 objects degraded (7.361%), 6 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-03-20T09:27:28.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:28 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3738363413' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:27:28.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:28 smithi122 ceph-mon[73495]: pgmap v1916: 57 pgs: 3 active+undersized+degraded, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 347 op/s; 19789/271512 objects degraded (7.288%); 13484/271512 objects misplaced (4.966%); 4.1 MiB/s, 111 objects/s recovering 2024-03-20T09:27:28.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:28 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 20130/273468 objects degraded (7.361%), 6 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-03-20T09:27:28.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:28 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3738363413' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:27:28.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:28 smithi026 ceph-mon[68309]: pgmap v1916: 57 pgs: 3 active+undersized+degraded, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 347 op/s; 19789/271512 objects degraded (7.288%); 13484/271512 objects misplaced (4.966%); 4.1 MiB/s, 111 objects/s recovering 2024-03-20T09:27:30.024 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:27:30.029 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:27:30.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:30.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:27:30.134 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:27:30.168 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:31.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:31 smithi115 ceph-mon[72476]: pgmap v1917: 57 pgs: 3 active+undersized+degraded, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 294 op/s; 19653/269361 objects degraded (7.296%); 13305/269361 objects misplaced (4.939%); 3.4 MiB/s, 95 objects/s recovering 2024-03-20T09:27:31.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:31 smithi122 ceph-mon[73495]: pgmap v1917: 57 pgs: 3 active+undersized+degraded, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 294 op/s; 19653/269361 objects degraded (7.296%); 13305/269361 objects misplaced (4.939%); 3.4 MiB/s, 95 objects/s recovering 2024-03-20T09:27:31.820 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:31 smithi026 ceph-mon[68309]: pgmap v1917: 57 pgs: 3 active+undersized+degraded, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 294 op/s; 19653/269361 objects degraded (7.296%); 13305/269361 objects misplaced (4.939%); 3.4 MiB/s, 95 objects/s recovering 2024-03-20T09:27:33.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:33 smithi115 ceph-mon[72476]: pgmap v1918: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 362 op/s; 19409/266166 objects degraded (7.292%); 13057/266166 objects misplaced (4.906%); 3.8 MiB/s, 97 objects/s recovering 2024-03-20T09:27:33.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:33 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 19409/266166 objects degraded (7.292%), 5 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:27:33.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:33 smithi122 ceph-mon[73495]: pgmap v1918: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 362 op/s; 19409/266166 objects degraded (7.292%); 13057/266166 objects misplaced (4.906%); 3.8 MiB/s, 97 objects/s recovering 2024-03-20T09:27:33.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:33 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 19409/266166 objects degraded (7.292%), 5 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:27:33.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:33 smithi026 ceph-mon[68309]: pgmap v1918: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 362 op/s; 19409/266166 objects degraded (7.292%); 13057/266166 objects misplaced (4.906%); 3.8 MiB/s, 97 objects/s recovering 2024-03-20T09:27:33.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:33 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 19409/266166 objects degraded (7.292%), 5 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:27:35.511 INFO:tasks.thrashosds.thrasher:j is {'epoch': 790, 'fsid': '852fbb68-e696-11ee-95c9-87774f69a715', 'created': '2024-03-20T08:48:35.281088+0000', 'modified': '2024-03-20T09:27:22.353467+0000', 'last_up_change': '2024-03-20T09:26:03.473344+0000', 'last_in_change': '2024-03-20T09:20:22.151285+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 6, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-20T08:52:17.442324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '360', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '358', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5399999618530273, 'score_stable': 1.5399999618530273, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 1.4199999570846558, 'raw_score_stable': 1.4199999570846558, 'primary_affinity_weighted': 0.9215061664581299, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 6, 'pool_name': 'unique_pool_4', 'create_time': '2024-03-20T09:24:02.337365+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '751', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '749', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.299999952316284, 'score_stable': 2.299999952316284, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 2.119999885559082, 'raw_score_stable': 2.119999885559082, 'primary_affinity_weighted': 0.9215061664581299, 'average_primary_affinity': 0.9200000166893005, 'average_primary_affinity_weighted': 0.9200000166893005}}], 'osds': [{'osd': 0, 'uuid': 'ad3fd9c0-316f-4fe5-a63b-d480087c1729', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 789, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6802', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6803', 'nonce': 2139805293}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6804', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6805', 'nonce': 2139805293}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6808', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6809', 'nonce': 2139805293}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6806', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6807', 'nonce': 2139805293}]}, 'public_addr': '172.21.15.26:6803/2139805293', 'cluster_addr': '172.21.15.26:6805/2139805293', 'heartbeat_back_addr': '172.21.15.26:6809/2139805293', 'heartbeat_front_addr': '172.21.15.26:6807/2139805293', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'e85b94f5-2954-47a1-9c20-096f2606eb04', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 786, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6810', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6811', 'nonce': 3853355134}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6812', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6813', 'nonce': 3853355134}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6816', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6817', 'nonce': 3853355134}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6814', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6815', 'nonce': 3853355134}]}, 'public_addr': '172.21.15.26:6811/3853355134', 'cluster_addr': '172.21.15.26:6813/3853355134', 'heartbeat_back_addr': '172.21.15.26:6817/3853355134', 'heartbeat_front_addr': '172.21.15.26:6815/3853355134', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'dd2b10f0-741c-47a4-b224-ac6bfe2c14ef', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 787, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6818', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6819', 'nonce': 3993357967}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6820', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6821', 'nonce': 3993357967}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6824', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6825', 'nonce': 3993357967}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6822', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6823', 'nonce': 3993357967}]}, 'public_addr': '172.21.15.26:6819/3993357967', 'cluster_addr': '172.21.15.26:6821/3993357967', 'heartbeat_back_addr': '172.21.15.26:6825/3993357967', 'heartbeat_front_addr': '172.21.15.26:6823/3993357967', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 788, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6826', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6827', 'nonce': 4110225509}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6828', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6829', 'nonce': 4110225509}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6832', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6833', 'nonce': 4110225509}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6830', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6831', 'nonce': 4110225509}]}, 'public_addr': '172.21.15.26:6827/4110225509', 'cluster_addr': '172.21.15.26:6829/4110225509', 'heartbeat_back_addr': '172.21.15.26:6833/4110225509', 'heartbeat_front_addr': '172.21.15.26:6831/4110225509', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'f14e34d5-9829-42a3-9b25-31a0976c5ef7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 749, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6800', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6801', 'nonce': 477923058}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6802', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6803', 'nonce': 477923058}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6806', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6807', 'nonce': 477923058}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6804', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6805', 'nonce': 477923058}]}, 'public_addr': '172.21.15.115:6801/477923058', 'cluster_addr': '172.21.15.115:6803/477923058', 'heartbeat_back_addr': '172.21.15.115:6807/477923058', 'heartbeat_front_addr': '172.21.15.115:6805/477923058', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'cb431b6a-540d-4318-ae35-d379e1df0185', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 41, 'last_clean_end': 273, 'up_from': 443, 'up_thru': 755, 'down_at': 274, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6808', 'nonce': 650416648}, {'type': 'v1', 'addr': '172.21.15.115:6809', 'nonce': 650416648}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6810', 'nonce': 650416648}, {'type': 'v1', 'addr': '172.21.15.115:6811', 'nonce': 650416648}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6814', 'nonce': 650416648}, {'type': 'v1', 'addr': '172.21.15.115:6815', 'nonce': 650416648}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6812', 'nonce': 650416648}, {'type': 'v1', 'addr': '172.21.15.115:6813', 'nonce': 650416648}]}, 'public_addr': '172.21.15.115:6809/650416648', 'cluster_addr': '172.21.15.115:6811/650416648', 'heartbeat_back_addr': '172.21.15.115:6815/650416648', 'heartbeat_front_addr': '172.21.15.115:6813/650416648', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '753ed830-e764-4ff3-9d9b-87e7fa748b81', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 768, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6816', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6817', 'nonce': 3278301843}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6818', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6819', 'nonce': 3278301843}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6822', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6823', 'nonce': 3278301843}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6820', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6821', 'nonce': 3278301843}]}, 'public_addr': '172.21.15.115:6817/3278301843', 'cluster_addr': '172.21.15.115:6819/3278301843', 'heartbeat_back_addr': '172.21.15.115:6823/3278301843', 'heartbeat_front_addr': '172.21.15.115:6821/3278301843', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '55bcbd37-7a6f-4ab1-b5a9-91a951444d86', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.058074951171875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 517, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6824', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6825', 'nonce': 3152424001}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6826', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6827', 'nonce': 3152424001}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6830', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6831', 'nonce': 3152424001}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6828', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6829', 'nonce': 3152424001}]}, 'public_addr': '172.21.15.115:6825/3152424001', 'cluster_addr': '172.21.15.115:6827/3152424001', 'heartbeat_back_addr': '172.21.15.115:6831/3152424001', 'heartbeat_front_addr': '172.21.15.115:6829/3152424001', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'bc86248e-6bd0-4369-b8a2-e593bc2a610d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 70, 'up_thru': 780, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6800', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6801', 'nonce': 1309426740}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6802', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6803', 'nonce': 1309426740}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6806', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6807', 'nonce': 1309426740}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6804', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6805', 'nonce': 1309426740}]}, 'public_addr': '172.21.15.122:6801/1309426740', 'cluster_addr': '172.21.15.122:6803/1309426740', 'heartbeat_back_addr': '172.21.15.122:6807/1309426740', 'heartbeat_front_addr': '172.21.15.122:6805/1309426740', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '9b20a4aa-032a-4c09-ac75-153c554424e7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 754, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6808', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6809', 'nonce': 2953210076}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6810', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6811', 'nonce': 2953210076}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6814', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6815', 'nonce': 2953210076}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6812', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6813', 'nonce': 2953210076}]}, 'public_addr': '172.21.15.122:6809/2953210076', 'cluster_addr': '172.21.15.122:6811/2953210076', 'heartbeat_back_addr': '172.21.15.122:6815/2953210076', 'heartbeat_front_addr': '172.21.15.122:6813/2953210076', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'e62c1fb3-5dd1-4d0d-997c-d20633d19361', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 89, 'up_thru': 751, 'down_at': 754, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6816', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6817', 'nonce': 409394340}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6818', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6819', 'nonce': 409394340}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6822', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6823', 'nonce': 409394340}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6820', 'nonce': 409394340}, {'type': 'v1', 'addr': '172.21.15.122:6821', 'nonce': 409394340}]}, 'public_addr': '172.21.15.122:6817/409394340', 'cluster_addr': '172.21.15.122:6819/409394340', 'heartbeat_back_addr': '172.21.15.122:6823/409394340', 'heartbeat_front_addr': '172.21.15.122:6821/409394340', 'state': ['exists']}, {'osd': 11, 'uuid': 'b83994d5-e5ef-48b0-9758-c26843cec3b6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 96, 'up_thru': 749, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6824', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6825', 'nonce': 2704069354}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6826', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6827', 'nonce': 2704069354}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6830', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6831', 'nonce': 2704069354}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6828', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6829', 'nonce': 2704069354}]}, 'public_addr': '172.21.15.122:6825/2704069354', 'cluster_addr': '172.21.15.122:6827/2704069354', 'heartbeat_back_addr': '172.21.15.122:6831/2704069354', 'heartbeat_front_addr': '172.21.15.122:6829/2704069354', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:26:34.539891+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:26:53.896921+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:26:59.685265+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:26:29.756266+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:26:50.368636+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:26:21.650602+0000', 'dead_epoch': 273}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:26:42.847510+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:27:09.310842+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:26:46.717755+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:26:45.830725+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:25:18.432474+0000', 'dead_epoch': 753}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:26:43.527199+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.1', 'osds': [6, 11, 5]}], 'pg_upmap_items': [{'pgid': '1.1c', 'mappings': [{'from': 6, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '6.12', 'osds': [2, 4]}, {'pgid': '6.17', 'osds': [2, 3]}, {'pgid': '6.19', 'osds': [5, 1, 0]}], 'primary_temp': [], 'blocklist': {'172.21.15.26:0/4291404191': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/642012566': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:6801/1181459999': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/3984513952': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6800/1654945498': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6800/1181459999': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/137770981': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:0/1451237762': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/3938386631': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6801/1654945498': '2024-03-21T08:49:04.596754+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-20T09:27:35.512 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:27:35.587 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:35 smithi026 ceph-mon[68309]: pgmap v1919: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 304 op/s; 19409/266166 objects degraded (7.292%); 13057/266166 objects misplaced (4.906%); 3.2 MiB/s, 81 objects/s recovering 2024-03-20T09:27:35.653 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:35 smithi115 ceph-mon[72476]: pgmap v1919: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 304 op/s; 19409/266166 objects degraded (7.292%); 13057/266166 objects misplaced (4.906%); 3.2 MiB/s, 81 objects/s recovering 2024-03-20T09:27:35.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:35 smithi122 ceph-mon[73495]: pgmap v1919: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 304 op/s; 19409/266166 objects degraded (7.292%); 13057/266166 objects misplaced (4.906%); 3.2 MiB/s, 81 objects/s recovering 2024-03-20T09:27:37.295 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:27:37.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:37.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:27:37.425 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:27:37.452 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:37.471 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:27:37.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:37.474 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:27:37.575 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:27:37.600 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:37 smithi122 ceph-mon[73495]: pgmap v1920: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 414 op/s; 19219/262122 objects degraded (7.332%); 12915/262122 objects misplaced (4.927%); 3.6 MiB/s, 86 objects/s recovering 2024-03-20T09:27:37.600 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:37 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:27:37.600 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:37 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:27:37.602 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:37.620 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:27:37.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:37.622 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:27:37.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:37 smithi115 ceph-mon[72476]: pgmap v1920: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 414 op/s; 19219/262122 objects degraded (7.332%); 12915/262122 objects misplaced (4.927%); 3.6 MiB/s, 86 objects/s recovering 2024-03-20T09:27:37.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:37 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:27:37.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:37 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:27:37.683 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:37 smithi026 ceph-mon[68309]: pgmap v1920: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 414 op/s; 19219/262122 objects degraded (7.332%); 12915/262122 objects misplaced (4.927%); 3.6 MiB/s, 86 objects/s recovering 2024-03-20T09:27:37.683 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:37 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:27:37.683 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:37 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:27:37.724 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:27:37.752 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:37.778 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:27:37.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:37.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:27:37.882 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:27:37.909 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:37.933 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:27:37.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:37.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:27:38.038 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:27:38.066 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:38.087 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:27:38.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:38.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:27:38.194 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:27:38.230 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:38.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:38 smithi115 ceph-mon[72476]: osdmap e791: 12 total, 11 up, 12 in 2024-03-20T09:27:38.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:38 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 19219/262122 objects degraded (7.332%), 5 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:27:38.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:38 smithi115 ceph-mon[72476]: osdmap e792: 12 total, 11 up, 12 in 2024-03-20T09:27:38.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:38 smithi122 ceph-mon[73495]: osdmap e791: 12 total, 11 up, 12 in 2024-03-20T09:27:38.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:38 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 19219/262122 objects degraded (7.332%), 5 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:27:38.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:38 smithi122 ceph-mon[73495]: osdmap e792: 12 total, 11 up, 12 in 2024-03-20T09:27:38.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:38 smithi026 ceph-mon[68309]: osdmap e791: 12 total, 11 up, 12 in 2024-03-20T09:27:38.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:38 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 19219/262122 objects degraded (7.332%), 5 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:27:38.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:38 smithi026 ceph-mon[68309]: osdmap e792: 12 total, 11 up, 12 in 2024-03-20T09:27:39.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:39 smithi115 ceph-mon[72476]: pgmap v1923: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 387 op/s; 18962/260067 objects degraded (7.291%); 12429/260067 objects misplaced (4.779%); 3.4 MiB/s, 54 objects/s recovering 2024-03-20T09:27:39.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:39 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:27:39.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:39 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:27:39.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:39 smithi122 ceph-mon[73495]: pgmap v1923: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 387 op/s; 18962/260067 objects degraded (7.291%); 12429/260067 objects misplaced (4.779%); 3.4 MiB/s, 54 objects/s recovering 2024-03-20T09:27:39.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:39 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:27:39.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:39 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:27:39.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:39 smithi026 ceph-mon[68309]: pgmap v1923: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 387 op/s; 18962/260067 objects degraded (7.291%); 12429/260067 objects misplaced (4.779%); 3.4 MiB/s, 54 objects/s recovering 2024-03-20T09:27:39.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:39 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:27:39.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:39 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:27:40.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:40 smithi026 ceph-mon[68309]: osdmap e793: 12 total, 11 up, 12 in 2024-03-20T09:27:40.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:40 smithi026 ceph-mon[68309]: pgmap v1925: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 481 op/s; 18766/257505 objects degraded (7.288%); 12281/257505 objects misplaced (4.769%); 4.1 MiB/s, 64 objects/s recovering 2024-03-20T09:27:40.786 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:27:40.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:40.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:27:40.891 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:27:40.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:40 smithi115 ceph-mon[72476]: osdmap e793: 12 total, 11 up, 12 in 2024-03-20T09:27:40.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:40 smithi115 ceph-mon[72476]: pgmap v1925: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 481 op/s; 18766/257505 objects degraded (7.288%); 12281/257505 objects misplaced (4.769%); 4.1 MiB/s, 64 objects/s recovering 2024-03-20T09:27:40.923 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:40.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:40 smithi122 ceph-mon[73495]: osdmap e793: 12 total, 11 up, 12 in 2024-03-20T09:27:40.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:40 smithi122 ceph-mon[73495]: pgmap v1925: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 481 op/s; 18766/257505 objects degraded (7.288%); 12281/257505 objects misplaced (4.769%); 4.1 MiB/s, 64 objects/s recovering 2024-03-20T09:27:43.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:43 smithi115 ceph-mon[72476]: pgmap v1926: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 409 op/s; 18580/254757 objects degraded (7.293%); 12054/254757 objects misplaced (4.732%); 4.5 MiB/s, 71 objects/s recovering 2024-03-20T09:27:43.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:43 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 18580/254757 objects degraded (7.293%), 5 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:27:43.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:43 smithi115 ceph-mon[72476]: osdmap e794: 12 total, 11 up, 12 in 2024-03-20T09:27:43.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:43 smithi122 ceph-mon[73495]: pgmap v1926: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 409 op/s; 18580/254757 objects degraded (7.293%); 12054/254757 objects misplaced (4.732%); 4.5 MiB/s, 71 objects/s recovering 2024-03-20T09:27:43.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:43 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 18580/254757 objects degraded (7.293%), 5 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:27:43.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:43 smithi122 ceph-mon[73495]: osdmap e794: 12 total, 11 up, 12 in 2024-03-20T09:27:43.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:43 smithi026 ceph-mon[68309]: pgmap v1926: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 409 op/s; 18580/254757 objects degraded (7.293%); 12054/254757 objects misplaced (4.732%); 4.5 MiB/s, 71 objects/s recovering 2024-03-20T09:27:43.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:43 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 18580/254757 objects degraded (7.293%), 5 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:27:43.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:43 smithi026 ceph-mon[68309]: osdmap e794: 12 total, 11 up, 12 in 2024-03-20T09:27:45.361 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:27:45.361 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:27:45.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:45 smithi115 ceph-mon[72476]: pgmap v1928: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 261 op/s; 18580/254757 objects degraded (7.293%); 12054/254757 objects misplaced (4.732%); 1.7 MiB/s, 27 objects/s recovering 2024-03-20T09:27:45.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:45 smithi122 ceph-mon[73495]: pgmap v1928: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 261 op/s; 18580/254757 objects degraded (7.293%); 12054/254757 objects misplaced (4.732%); 1.7 MiB/s, 27 objects/s recovering 2024-03-20T09:27:45.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:45 smithi026 ceph-mon[68309]: pgmap v1928: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 261 op/s; 18580/254757 objects degraded (7.293%); 12054/254757 objects misplaced (4.732%); 1.7 MiB/s, 27 objects/s recovering 2024-03-20T09:27:47.186 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:27:47.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:47.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:27:47.291 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:27:47.318 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:47.337 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:27:47.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:47.340 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:27:47.441 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:27:47.465 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:47 smithi026 ceph-mon[68309]: pgmap v1929: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 375 op/s; 18400/251046 objects degraded (7.329%); 11892/251046 objects misplaced (4.737%); 2.1 MiB/s, 33 objects/s recovering 2024-03-20T09:27:47.469 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:47.490 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:27:47.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:47.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:27:47.595 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:27:47.619 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:47 smithi122 ceph-mon[73495]: pgmap v1929: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 375 op/s; 18400/251046 objects degraded (7.329%); 11892/251046 objects misplaced (4.737%); 2.1 MiB/s, 33 objects/s recovering 2024-03-20T09:27:47.621 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:47.641 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:27:47.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:47.644 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:27:47.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:47 smithi115 ceph-mon[72476]: pgmap v1929: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 375 op/s; 18400/251046 objects degraded (7.329%); 11892/251046 objects misplaced (4.737%); 2.1 MiB/s, 33 objects/s recovering 2024-03-20T09:27:47.744 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:27:47.774 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:47.793 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:27:47.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:47.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:27:47.897 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:27:47.925 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:47.944 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:27:47.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:47.947 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:27:48.048 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:27:48.080 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:48.210 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:27:48.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:48.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:27:48.234 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-20T09:27:48.315 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:27:48.343 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:48 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:27:48.343 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:48 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:27:48.343 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:48 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 18400/251046 objects degraded (7.329%), 5 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:27:48.345 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:48.392 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:27:48.400 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:27:48.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:48.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:27:48.436 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:27:48.507 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:27:48.521 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:27:48.536 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:48 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:27:48.536 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:48 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:27:48.536 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:48 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 18400/251046 objects degraded (7.329%), 5 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:27:48.538 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:48.562 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:27:48.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:48.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:27:48.667 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:27:48.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:48 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:27:48.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:48 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:27:48.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:48 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 18400/251046 objects degraded (7.329%), 5 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:27:48.694 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:48.713 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:27:48.716 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:48.717 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:27:48.818 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:27:48.844 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:48.864 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:27:48.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:48.867 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:27:48.968 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:27:49.004 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:49.034 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:27:49.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:49.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:27:49.139 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:27:49.166 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:49.185 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:27:49.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:49.188 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:27:49.289 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:27:49.316 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:49.336 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:27:49.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:49.339 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:27:49.402 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:49 smithi115 ceph-mon[72476]: pgmap v1930: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 380 op/s; 18160/248937 objects degraded (7.295%); 11687/248937 objects misplaced (4.695%); 2.1 MiB/s, 33 objects/s recovering 2024-03-20T09:27:49.402 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:49 smithi115 ceph-mon[72476]: osdmap e795: 12 total, 11 up, 12 in 2024-03-20T09:27:49.439 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:27:49.442 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:49 smithi122 ceph-mon[73495]: pgmap v1930: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 380 op/s; 18160/248937 objects degraded (7.295%); 11687/248937 objects misplaced (4.695%); 2.1 MiB/s, 33 objects/s recovering 2024-03-20T09:27:49.442 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:49 smithi122 ceph-mon[73495]: osdmap e795: 12 total, 11 up, 12 in 2024-03-20T09:27:49.466 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:49 smithi026 ceph-mon[68309]: pgmap v1930: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 380 op/s; 18160/248937 objects degraded (7.295%); 11687/248937 objects misplaced (4.695%); 2.1 MiB/s, 33 objects/s recovering 2024-03-20T09:27:49.466 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:49 smithi026 ceph-mon[68309]: osdmap e795: 12 total, 11 up, 12 in 2024-03-20T09:27:49.466 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:49.486 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:27:49.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:49.489 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:27:49.590 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:27:49.630 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:49.664 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:27:49.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:49.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:27:49.772 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:27:49.809 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:50.243 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:27:50.274 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:27:50.298 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:27:50.321 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:27:50.345 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:27:50.370 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:27:50.399 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:27:50.434 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:27:50.462 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:27:50.484 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:27:50.505 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:27:50.505 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:27:50.506 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:27:50.506 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:27:50.506 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:27:50.506 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:27:50.506 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:27:50.506 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:27:50.506 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:27:50.506 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:27:50.506 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:27:50.506 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-03-20T09:27:50.506 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:27:50.610 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:50 smithi026 ceph-mon[68309]: 6.3 scrub starts 2024-03-20T09:27:50.610 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:50 smithi026 ceph-mon[68309]: 6.3 scrub ok 2024-03-20T09:27:50.610 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:50 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2808788619' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:27:50.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:50 smithi115 ceph-mon[72476]: 6.3 scrub starts 2024-03-20T09:27:50.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:50 smithi115 ceph-mon[72476]: 6.3 scrub ok 2024-03-20T09:27:50.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:50 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2808788619' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:27:50.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:50 smithi122 ceph-mon[73495]: 6.3 scrub starts 2024-03-20T09:27:50.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:50 smithi122 ceph-mon[73495]: 6.3 scrub ok 2024-03-20T09:27:50.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:50 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2808788619' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:27:50.845 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:27:50.846 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:27:50.846 INFO:teuthology.orchestra.run.smithi115.stdout: "num_ops": 0 2024-03-20T09:27:50.846 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:27:51.263 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:27:51.264 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:27:51.527 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:51 smithi115 ceph-mon[72476]: pgmap v1932: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 339 op/s; 17988/246612 objects degraded (7.294%); 11518/246612 objects misplaced (4.670%); 2.1 MiB/s, 33 objects/s recovering 2024-03-20T09:27:51.559 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-20T09:27:51.598 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:51 smithi026 ceph-mon[68309]: pgmap v1932: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 339 op/s; 17988/246612 objects degraded (7.294%); 11518/246612 objects misplaced (4.670%); 2.1 MiB/s, 33 objects/s recovering 2024-03-20T09:27:51.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:51 smithi122 ceph-mon[73495]: pgmap v1932: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 339 op/s; 17988/246612 objects degraded (7.294%); 11518/246612 objects misplaced (4.670%); 2.1 MiB/s, 33 objects/s recovering 2024-03-20T09:27:51.915 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:27:51.961 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:27:51.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:51.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:27:52.066 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:27:52.092 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:52.111 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:27:52.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:52.114 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:27:52.215 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:27:52.271 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:52.478 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:27:52.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:52.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:27:52.584 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:27:52.623 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:52.983 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:27:52.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:52.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:27:53.089 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:27:53.123 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:53.160 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:27:53.165 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:27:53.265 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:27:53.498 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:27:53.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:53 smithi115 ceph-mon[72476]: pgmap v1933: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 379 op/s; 17739/243621 objects degraded (7.281%); 11308/243621 objects misplaced (4.642%); 2.4 MiB/s, 38 objects/s recovering 2024-03-20T09:27:53.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:53 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 17739/243621 objects degraded (7.281%), 5 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:27:53.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:53 smithi115 ceph-mon[72476]: osdmap e796: 12 total, 11 up, 12 in 2024-03-20T09:27:53.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:53 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3009280415' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:27:53.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:53 smithi122 ceph-mon[73495]: pgmap v1933: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 379 op/s; 17739/243621 objects degraded (7.281%); 11308/243621 objects misplaced (4.642%); 2.4 MiB/s, 38 objects/s recovering 2024-03-20T09:27:53.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:53 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 17739/243621 objects degraded (7.281%), 5 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:27:53.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:53 smithi122 ceph-mon[73495]: osdmap e796: 12 total, 11 up, 12 in 2024-03-20T09:27:53.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:53 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3009280415' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:27:53.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:53 smithi026 ceph-mon[68309]: pgmap v1933: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 379 op/s; 17739/243621 objects degraded (7.281%); 11308/243621 objects misplaced (4.642%); 2.4 MiB/s, 38 objects/s recovering 2024-03-20T09:27:53.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:53 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 17739/243621 objects degraded (7.281%), 5 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:27:53.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:53 smithi026 ceph-mon[68309]: osdmap e796: 12 total, 11 up, 12 in 2024-03-20T09:27:53.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:53 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3009280415' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:27:53.988 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":1934,"stamp":"2024-03-20T09:27:52.494255+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5322383220,"num_objects":81207,"num_object_clones":0,"num_object_copies":243621,"num_objects_missing_on_primary":5447,"num_objects_missing":5447,"num_objects_degraded":17739,"num_objects_misplaced":11308,"num_objects_unfound":0,"num_objects_dirty":81207,"num_whiteouts":0,"num_read":10,"num_read_kb":8,"num_write":146123,"num_write_kb":6968265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26972,"num_bytes_recovered":1637430141,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":243752,"ondisk_log_size":243752,"up":159,"acting":159,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":189,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":24644528,"kb_used_data":17861872,"kb_used_omap":198,"kb_used_meta":6782329,"kb_avail":1100297296,"statfs":{"total":1151940427776,"available":1126704431104,"internally_reserved":0,"allocated":18290556928,"data_stored":18284853254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":203677,"internal_metadata":6945104995},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-243269632,"num_objects":-3712,"num_object_clones":0,"num_object_copies":-11136,"num_objects_missing_on_primary":-290,"num_objects_missing":-290,"num_objects_degraded":-841,"num_objects_misplaced":-746,"num_objects_unfound":0,"num_objects_dirty":-3712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3712,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":375,"num_bytes_recovered":24576000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.227580"},"pg_stats":[{"pgid":"6.18","version":"795'8857","reported_seq":20120,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:51.190001+0000","last_change":"2024-03-20T09:27:22.377636+0000","last_active":"2024-03-20T09:27:51.190001+0000","last_peered":"2024-03-20T09:27:51.190001+0000","last_clean":"2024-03-20T09:27:51.190001+0000","last_became_active":"2024-03-20T09:27:22.365007+0000","last_became_peered":"2024-03-20T09:27:22.365007+0000","last_unstale":"2024-03-20T09:27:51.190001+0000","last_undegraded":"2024-03-20T09:27:51.190001+0000","last_fullsized":"2024-03-20T09:27:51.190001+0000","mapping_epoch":789,"log_start":"0'0","ondisk_log_start":"0'0","created":749,"last_epoch_clean":790,"parent":"0.0","parent_split_bits":5,"last_scrub":"740'6323","last_scrub_stamp":"2024-03-20T09:25:29.608640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:29.608640+0000","objects_scrubbed":6269,"log_size":8857,"log_dups_size":0,"ondisk_log_size":8857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:27:55.810125+0000","scrub_duration":1480,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164265984,"num_objects":2507,"num_object_clones":0,"num_object_copies":7521,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4888,"num_write_kb":236608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3693,"num_bytes_recovered":213975040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"795'8868","reported_seq":12512,"reported_epoch":795,"state":"active+recovering+degraded+remapped","last_fresh":"2024-03-20T09:27:49.473246+0000","last_change":"2024-03-20T09:26:47.744090+0000","last_active":"2024-03-20T09:27:49.473246+0000","last_peered":"2024-03-20T09:27:49.473246+0000","last_clean":"2024-03-20T09:25:58.407898+0000","last_became_active":"2024-03-20T09:26:12.889206+0000","last_became_peered":"2024-03-20T09:26:12.889206+0000","last_unstale":"2024-03-20T09:27:49.473246+0000","last_undegraded":"2024-03-20T09:26:08.946888+0000","last_fullsized":"2024-03-20T09:27:49.473246+0000","mapping_epoch":755,"log_start":"0'0","ondisk_log_start":"0'0","created":749,"last_epoch_clean":750,"parent":"0.0","parent_split_bits":5,"last_scrub":"739'6312","last_scrub_stamp":"2024-03-20T09:25:27.604184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:27.604184+0000","objects_scrubbed":6258,"log_size":8868,"log_dups_size":0,"ondisk_log_size":8868,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1892,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166264832,"num_objects":2537,"num_object_clones":0,"num_object_copies":7611,"num_objects_missing_on_primary":2548,"num_objects_missing":2548,"num_objects_degraded":2537,"num_objects_misplaced":4468,"num_objects_unfound":0,"num_objects_dirty":2537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4868,"num_write_kb":236960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1110,"num_bytes_recovered":72744960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1,0],"avail_no_missing":["0","8","9"],"object_location_counts":[{"shards":"0,1,5","objects":303},{"shards":"0,8,9","objects":2234}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1e","version":"20'137","reported_seq":1012,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:48.438932+0000","last_change":"2024-03-20T09:25:33.135367+0000","last_active":"2024-03-20T09:27:48.438932+0000","last_peered":"2024-03-20T09:27:48.438932+0000","last_clean":"2024-03-20T09:27:48.438932+0000","last_became_active":"2024-03-20T09:13:20.071655+0000","last_became_peered":"2024-03-20T09:13:20.071655+0000","last_unstale":"2024-03-20T09:27:48.438932+0000","last_undegraded":"2024-03-20T09:27:48.438932+0000","last_fullsized":"2024-03-20T09:27:48.438932+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:33.135222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:25:33.135222+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'137","reported_seq":1185,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:48.422185+0000","last_change":"2024-03-20T09:25:18.228015+0000","last_active":"2024-03-20T09:27:48.422185+0000","last_peered":"2024-03-20T09:27:48.422185+0000","last_clean":"2024-03-20T09:27:48.422185+0000","last_became_active":"2024-03-20T09:07:42.013245+0000","last_became_peered":"2024-03-20T09:07:42.013245+0000","last_unstale":"2024-03-20T09:27:48.422185+0000","last_undegraded":"2024-03-20T09:27:48.422185+0000","last_fullsized":"2024-03-20T09:27:48.422185+0000","mapping_epoch":363,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:18.227887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:25:18.227887+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"20'137","reported_seq":1660,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:48.419588+0000","last_change":"2024-03-20T09:27:01.038559+0000","last_active":"2024-03-20T09:27:48.419588+0000","last_peered":"2024-03-20T09:27:48.419588+0000","last_clean":"2024-03-20T09:27:48.419588+0000","last_became_active":"2024-03-20T09:25:35.482959+0000","last_became_peered":"2024-03-20T09:25:35.482959+0000","last_unstale":"2024-03-20T09:27:48.419588+0000","last_undegraded":"2024-03-20T09:27:48.419588+0000","last_fullsized":"2024-03-20T09:27:48.419588+0000","mapping_epoch":744,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":745,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:27:01.038468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:27:01.038468+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:28:03.577522+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,1],"acting":[8,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.19","version":"20'137","reported_seq":847,"reported_epoch":795,"state":"active+undersized","last_fresh":"2024-03-20T09:27:48.415701+0000","last_change":"2024-03-20T09:26:04.497255+0000","last_active":"2024-03-20T09:27:48.415701+0000","last_peered":"2024-03-20T09:27:48.415701+0000","last_clean":"2024-03-20T09:04:29.198571+0000","last_became_active":"2024-03-20T09:26:04.497255+0000","last_became_peered":"2024-03-20T09:26:04.497255+0000","last_unstale":"2024-03-20T09:27:48.415701+0000","last_undegraded":"2024-03-20T09:27:48.415701+0000","last_fullsized":"2024-03-20T09:26:04.493898+0000","mapping_epoch":754,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":364,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:15.799703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:25:15.799703+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":8,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":120211,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.18","version":"20'137","reported_seq":1151,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:48.440557+0000","last_change":"2024-03-20T09:25:21.295358+0000","last_active":"2024-03-20T09:27:48.440557+0000","last_peered":"2024-03-20T09:27:48.440557+0000","last_clean":"2024-03-20T09:27:48.440557+0000","last_became_active":"2024-03-20T09:10:35.277481+0000","last_became_peered":"2024-03-20T09:10:35.277481+0000","last_unstale":"2024-03-20T09:27:48.440557+0000","last_undegraded":"2024-03-20T09:27:48.440557+0000","last_fullsized":"2024-03-20T09:27:48.440557+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:21.295237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:25:21.295237+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1b","version":"20'137","reported_seq":1497,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:48.419487+0000","last_change":"2024-03-20T09:27:02.118065+0000","last_active":"2024-03-20T09:27:48.419487+0000","last_peered":"2024-03-20T09:27:48.419487+0000","last_clean":"2024-03-20T09:27:48.419487+0000","last_became_active":"2024-03-20T09:24:16.445287+0000","last_became_peered":"2024-03-20T09:24:16.445287+0000","last_unstale":"2024-03-20T09:27:48.419487+0000","last_undegraded":"2024-03-20T09:27:48.419487+0000","last_fullsized":"2024-03-20T09:27:48.419487+0000","mapping_epoch":718,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:27:02.117941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:27:02.117941+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:28:15.856621+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"6.f","version":"795'10073","reported_seq":10332,"reported_epoch":795,"state":"active+undersized+degraded","last_fresh":"2024-03-20T09:27:51.375009+0000","last_change":"2024-03-20T09:26:04.498209+0000","last_active":"2024-03-20T09:27:51.375009+0000","last_peered":"2024-03-20T09:27:51.375009+0000","last_clean":"2024-03-20T09:26:03.485351+0000","last_became_active":"2024-03-20T09:26:04.498209+0000","last_became_peered":"2024-03-20T09:26:04.498209+0000","last_unstale":"2024-03-20T09:27:51.375009+0000","last_undegraded":"2024-03-20T09:26:04.487264+0000","last_fullsized":"2024-03-20T09:26:04.487178+0000","mapping_epoch":754,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"739'6380","last_scrub_stamp":"2024-03-20T09:25:27.830146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:27.830146+0000","objects_scrubbed":6341,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1684,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333643776,"num_objects":5091,"num_object_clones":0,"num_object_copies":15273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5091,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10073,"num_write_kb":485248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9],"acting":[2,9],"avail_no_missing":["2","9"],"object_location_counts":[{"shards":"2,9","objects":5091}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"20'137","reported_seq":1150,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:48.440558+0000","last_change":"2024-03-20T09:25:41.351717+0000","last_active":"2024-03-20T09:27:48.440558+0000","last_peered":"2024-03-20T09:27:48.440558+0000","last_clean":"2024-03-20T09:27:48.440558+0000","last_became_active":"2024-03-20T09:10:35.215772+0000","last_became_peered":"2024-03-20T09:10:35.215772+0000","last_unstale":"2024-03-20T09:27:48.440558+0000","last_undegraded":"2024-03-20T09:27:48.440558+0000","last_fullsized":"2024-03-20T09:27:48.440558+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:41.351598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:25:41.351598+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"795'9789","reported_seq":10041,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:50.274333+0000","last_change":"2024-03-20T09:25:20.220593+0000","last_active":"2024-03-20T09:27:50.274333+0000","last_peered":"2024-03-20T09:27:50.274333+0000","last_clean":"2024-03-20T09:27:50.274333+0000","last_became_active":"2024-03-20T09:24:03.382302+0000","last_became_peered":"2024-03-20T09:24:03.382302+0000","last_unstale":"2024-03-20T09:27:50.274333+0000","last_undegraded":"2024-03-20T09:27:50.274333+0000","last_fullsized":"2024-03-20T09:27:50.274333+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"736'5773","last_scrub_stamp":"2024-03-20T09:25:20.220414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:20.220414+0000","objects_scrubbed":5729,"log_size":9789,"log_dups_size":0,"ondisk_log_size":9789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1658,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328138752,"num_objects":5007,"num_object_clones":0,"num_object_copies":15021,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9789,"num_write_kb":473472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"20'137","reported_seq":1724,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-20T09:27:42.500515+0000","last_change":"2024-03-20T09:25:14.298374+0000","last_active":"2024-03-20T09:27:42.500515+0000","last_peered":"2024-03-20T09:27:42.500515+0000","last_clean":"2024-03-20T09:27:42.500515+0000","last_became_active":"2024-03-20T09:20:25.341750+0000","last_became_peered":"2024-03-20T09:20:25.341750+0000","last_unstale":"2024-03-20T09:27:42.500515+0000","last_undegraded":"2024-03-20T09:27:42.500515+0000","last_fullsized":"2024-03-20T09:27:42.500515+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:14.298275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:25:14.298275+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,2],"acting":[9,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.d","version":"795'9917","reported_seq":10171,"reported_epoch":795,"state":"active+undersized+degraded","last_fresh":"2024-03-20T09:27:49.431016+0000","last_change":"2024-03-20T09:26:04.507822+0000","last_active":"2024-03-20T09:27:49.431016+0000","last_peered":"2024-03-20T09:27:49.431016+0000","last_clean":"2024-03-20T09:26:03.469193+0000","last_became_active":"2024-03-20T09:26:04.507822+0000","last_became_peered":"2024-03-20T09:26:04.507822+0000","last_unstale":"2024-03-20T09:27:49.431016+0000","last_undegraded":"2024-03-20T09:26:04.488571+0000","last_fullsized":"2024-03-20T09:26:04.488461+0000","mapping_epoch":754,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"736'5792","last_scrub_stamp":"2024-03-20T09:25:19.681737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:19.681737+0000","objects_scrubbed":5741,"log_size":9917,"log_dups_size":0,"ondisk_log_size":9917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1457,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330694680,"num_objects":5047,"num_object_clones":0,"num_object_copies":15141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5047,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9917,"num_write_kb":478785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":5047}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"20'137","reported_seq":413,"reported_epoch":794,"state":"active+clean","last_fresh":"2024-03-20T09:27:42.502234+0000","last_change":"2024-03-20T09:24:17.164244+0000","last_active":"2024-03-20T09:27:42.502234+0000","last_peered":"2024-03-20T09:27:42.502234+0000","last_clean":"2024-03-20T09:27:42.502234+0000","last_became_active":"2024-03-20T09:24:16.444232+0000","last_became_peered":"2024-03-20T09:24:16.444232+0000","last_unstale":"2024-03-20T09:27:42.502234+0000","last_undegraded":"2024-03-20T09:27:42.502234+0000","last_fullsized":"2024-03-20T09:27:42.502234+0000","mapping_epoch":718,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:17.164131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:24:17.164131+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.3","version":"795'8960","reported_seq":9293,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:51.023452+0000","last_change":"2024-03-20T09:27:49.444124+0000","last_active":"2024-03-20T09:27:51.023452+0000","last_peered":"2024-03-20T09:27:51.023452+0000","last_clean":"2024-03-20T09:27:51.023452+0000","last_became_active":"2024-03-20T09:25:57.395950+0000","last_became_peered":"2024-03-20T09:25:57.395950+0000","last_unstale":"2024-03-20T09:27:51.023452+0000","last_undegraded":"2024-03-20T09:27:51.023452+0000","last_fullsized":"2024-03-20T09:27:51.023452+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":750,"parent":"0.0","parent_split_bits":0,"last_scrub":"795'8938","last_scrub_stamp":"2024-03-20T09:27:49.444018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:27:49.444018+0000","objects_scrubbed":2517,"log_size":8960,"log_dups_size":0,"ondisk_log_size":8960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:29:16.673593+0000","scrub_duration":341,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163446784,"num_objects":2494,"num_object_clones":0,"num_object_copies":7482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":661,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,11],"acting":[6,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"20'137","reported_seq":1680,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:48.438805+0000","last_change":"2024-03-20T09:25:34.098294+0000","last_active":"2024-03-20T09:27:48.438805+0000","last_peered":"2024-03-20T09:27:48.438805+0000","last_clean":"2024-03-20T09:27:48.438805+0000","last_became_active":"2024-03-20T09:12:14.306384+0000","last_became_peered":"2024-03-20T09:12:14.306384+0000","last_unstale":"2024-03-20T09:27:48.438805+0000","last_undegraded":"2024-03-20T09:27:48.438805+0000","last_fullsized":"2024-03-20T09:27:48.438805+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:34.098199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:25:34.098199+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"795'8968","reported_seq":9226,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:50.280703+0000","last_change":"2024-03-20T09:25:57.466498+0000","last_active":"2024-03-20T09:27:50.280703+0000","last_peered":"2024-03-20T09:27:50.280703+0000","last_clean":"2024-03-20T09:27:50.280703+0000","last_became_active":"2024-03-20T09:25:57.407017+0000","last_became_peered":"2024-03-20T09:25:57.407017+0000","last_unstale":"2024-03-20T09:27:50.280703+0000","last_undegraded":"2024-03-20T09:27:50.280703+0000","last_fullsized":"2024-03-20T09:27:50.280703+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":750,"parent":"0.0","parent_split_bits":0,"last_scrub":"733'5454","last_scrub_stamp":"2024-03-20T09:25:14.989721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:14.989721+0000","objects_scrubbed":5409,"log_size":8968,"log_dups_size":0,"ondisk_log_size":8968,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:27:55.188241+0000","scrub_duration":1387,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169508864,"num_objects":2586,"num_object_clones":0,"num_object_copies":7758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4942,"num_write_kb":240928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"20'137","reported_seq":318,"reported_epoch":795,"state":"active+undersized","last_fresh":"2024-03-20T09:27:48.421076+0000","last_change":"2024-03-20T09:26:04.507104+0000","last_active":"2024-03-20T09:27:48.421076+0000","last_peered":"2024-03-20T09:27:48.421076+0000","last_clean":"2024-03-20T08:54:29.262652+0000","last_became_active":"2024-03-20T09:26:04.507104+0000","last_became_peered":"2024-03-20T09:26:04.507104+0000","last_unstale":"2024-03-20T09:27:48.421076+0000","last_undegraded":"2024-03-20T09:27:48.421076+0000","last_fullsized":"2024-03-20T09:26:04.486458+0000","mapping_epoch":754,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:29.690849+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:19:03.805185+0000","last_clean_scrub_stamp":"2024-03-20T09:25:29.690849+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:27:50.442213+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13417,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":4,"num_write_kb":66,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":24042,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"795'9086","reported_seq":9352,"reported_epoch":795,"state":"active+undersized+degraded","last_fresh":"2024-03-20T09:27:51.389546+0000","last_change":"2024-03-20T09:26:04.510544+0000","last_active":"2024-03-20T09:27:51.389546+0000","last_peered":"2024-03-20T09:27:51.389546+0000","last_clean":"2024-03-20T09:26:03.445982+0000","last_became_active":"2024-03-20T09:26:04.510544+0000","last_became_peered":"2024-03-20T09:26:04.510544+0000","last_unstale":"2024-03-20T09:27:51.389546+0000","last_undegraded":"2024-03-20T09:26:04.494069+0000","last_fullsized":"2024-03-20T09:26:04.493993+0000","mapping_epoch":754,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":750,"parent":"0.0","parent_split_bits":0,"last_scrub":"738'6080","last_scrub_stamp":"2024-03-20T09:25:24.081378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:24.081378+0000","objects_scrubbed":6048,"log_size":9086,"log_dups_size":0,"ondisk_log_size":9086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:27:52.023553+0000","scrub_duration":958,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161841152,"num_objects":2469,"num_object_clones":0,"num_object_copies":7407,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2469,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5029,"num_write_kb":239968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":["8","9"],"object_location_counts":[{"shards":"8,9","objects":2469}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"20'137","reported_seq":1519,"reported_epoch":794,"state":"active+undersized","last_fresh":"2024-03-20T09:27:42.500574+0000","last_change":"2024-03-20T09:26:04.492627+0000","last_active":"2024-03-20T09:27:42.500574+0000","last_peered":"2024-03-20T09:27:42.500574+0000","last_clean":"2024-03-20T09:24:10.405771+0000","last_became_active":"2024-03-20T09:26:04.492627+0000","last_became_peered":"2024-03-20T09:26:04.492627+0000","last_unstale":"2024-03-20T09:27:42.500574+0000","last_undegraded":"2024-03-20T09:27:42.500574+0000","last_fullsized":"2024-03-20T09:26:04.490383+0000","mapping_epoch":754,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:31.549603+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:24:31.549603+0000","last_clean_scrub_stamp":"2024-03-20T09:24:31.549603+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8],"acting":[9,8],"avail_no_missing":["9","8"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.4","version":"795'9027","reported_seq":9286,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:49.576615+0000","last_change":"2024-03-20T09:25:57.438466+0000","last_active":"2024-03-20T09:27:49.576615+0000","last_peered":"2024-03-20T09:27:49.576615+0000","last_clean":"2024-03-20T09:27:49.576615+0000","last_became_active":"2024-03-20T09:25:57.431703+0000","last_became_peered":"2024-03-20T09:25:57.431703+0000","last_unstale":"2024-03-20T09:27:49.576615+0000","last_undegraded":"2024-03-20T09:27:49.576615+0000","last_fullsized":"2024-03-20T09:27:49.576615+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":750,"parent":"0.0","parent_split_bits":0,"last_scrub":"736'5943","last_scrub_stamp":"2024-03-20T09:25:21.190503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:21.190503+0000","objects_scrubbed":5872,"log_size":9027,"log_dups_size":0,"ondisk_log_size":9027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:27:51.356656+0000","scrub_duration":2104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171704320,"num_objects":2620,"num_object_clones":0,"num_object_copies":7860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4971,"num_write_kb":242912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"20'137","reported_seq":1593,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:48.421900+0000","last_change":"2024-03-20T09:25:17.292883+0000","last_active":"2024-03-20T09:27:48.421900+0000","last_peered":"2024-03-20T09:27:48.421900+0000","last_clean":"2024-03-20T09:27:48.421900+0000","last_became_active":"2024-03-20T09:19:54.499210+0000","last_became_peered":"2024-03-20T09:19:54.499210+0000","last_unstale":"2024-03-20T09:27:48.421900+0000","last_undegraded":"2024-03-20T09:27:48.421900+0000","last_fullsized":"2024-03-20T09:27:48.421900+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:17.292735+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:18.788771+0000","last_clean_scrub_stamp":"2024-03-20T09:25:17.292735+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"795'9190","reported_seq":9452,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:49.465981+0000","last_change":"2024-03-20T09:25:57.406721+0000","last_active":"2024-03-20T09:27:49.465981+0000","last_peered":"2024-03-20T09:27:49.465981+0000","last_clean":"2024-03-20T09:27:49.465981+0000","last_became_active":"2024-03-20T09:25:57.399625+0000","last_became_peered":"2024-03-20T09:25:57.399625+0000","last_unstale":"2024-03-20T09:27:49.465981+0000","last_undegraded":"2024-03-20T09:27:49.465981+0000","last_fullsized":"2024-03-20T09:27:49.465981+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":750,"parent":"0.0","parent_split_bits":0,"last_scrub":"738'6270","last_scrub_stamp":"2024-03-20T09:25:24.693262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:24.693262+0000","objects_scrubbed":6219,"log_size":9190,"log_dups_size":0,"ondisk_log_size":9190,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:27:53.151383+0000","scrub_duration":1532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171016192,"num_objects":2609,"num_object_clones":0,"num_object_copies":7827,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5061,"num_write_kb":245472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":1651,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:48.438688+0000","last_change":"2024-03-20T09:25:31.128237+0000","last_active":"2024-03-20T09:27:48.438688+0000","last_peered":"2024-03-20T09:27:48.438688+0000","last_clean":"2024-03-20T09:27:48.438688+0000","last_became_active":"2024-03-20T09:12:14.305989+0000","last_became_peered":"2024-03-20T09:12:14.305989+0000","last_unstale":"2024-03-20T09:27:48.438688+0000","last_undegraded":"2024-03-20T09:27:48.438688+0000","last_fullsized":"2024-03-20T09:27:48.438688+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:31.128133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:25:31.128133+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.5","version":"795'9005","reported_seq":9312,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:50.993669+0000","last_change":"2024-03-20T09:26:45.402941+0000","last_active":"2024-03-20T09:27:50.993669+0000","last_peered":"2024-03-20T09:27:50.993669+0000","last_clean":"2024-03-20T09:27:50.993669+0000","last_became_active":"2024-03-20T09:25:57.383538+0000","last_became_peered":"2024-03-20T09:25:57.383538+0000","last_unstale":"2024-03-20T09:27:50.993669+0000","last_undegraded":"2024-03-20T09:27:50.993669+0000","last_fullsized":"2024-03-20T09:27:50.993669+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":750,"parent":"0.0","parent_split_bits":0,"last_scrub":"772'8332","last_scrub_stamp":"2024-03-20T09:26:45.402765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:26:45.402765+0000","objects_scrubbed":3222,"log_size":9005,"log_dups_size":0,"ondisk_log_size":9005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:28:13.634837+0000","scrub_duration":383,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166985728,"num_objects":2548,"num_object_clones":0,"num_object_copies":7644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":674,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"20'137","reported_seq":1908,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:48.415959+0000","last_change":"2024-03-20T09:25:29.866424+0000","last_active":"2024-03-20T09:27:48.415959+0000","last_peered":"2024-03-20T09:27:48.415959+0000","last_clean":"2024-03-20T09:27:48.415959+0000","last_became_active":"2024-03-20T09:19:54.492879+0000","last_became_peered":"2024-03-20T09:19:54.492879+0000","last_unstale":"2024-03-20T09:27:48.415959+0000","last_undegraded":"2024-03-20T09:27:48.415959+0000","last_fullsized":"2024-03-20T09:27:48.415959+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:29.866281+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:19.513511+0000","last_clean_scrub_stamp":"2024-03-20T09:25:29.866281+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"795'9136","reported_seq":9400,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:49.976219+0000","last_change":"2024-03-20T09:25:57.413253+0000","last_active":"2024-03-20T09:27:49.976219+0000","last_peered":"2024-03-20T09:27:49.976219+0000","last_clean":"2024-03-20T09:27:49.976219+0000","last_became_active":"2024-03-20T09:25:57.396659+0000","last_became_peered":"2024-03-20T09:25:57.396659+0000","last_unstale":"2024-03-20T09:27:49.976219+0000","last_undegraded":"2024-03-20T09:27:49.976219+0000","last_fullsized":"2024-03-20T09:27:49.976219+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":750,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'6530","last_scrub_stamp":"2024-03-20T09:25:29.595052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:29.595052+0000","objects_scrubbed":6463,"log_size":9136,"log_dups_size":0,"ondisk_log_size":9136,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:27:52.642351+0000","scrub_duration":2087,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167084032,"num_objects":2549,"num_object_clones":0,"num_object_copies":7647,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5050,"num_write_kb":243200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"20'137","reported_seq":1076,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:48.446261+0000","last_change":"2024-03-20T09:26:54.028722+0000","last_active":"2024-03-20T09:27:48.446261+0000","last_peered":"2024-03-20T09:27:48.446261+0000","last_clean":"2024-03-20T09:27:48.446261+0000","last_became_active":"2024-03-20T09:15:34.225022+0000","last_became_peered":"2024-03-20T09:15:34.225022+0000","last_unstale":"2024-03-20T09:27:48.446261+0000","last_undegraded":"2024-03-20T09:27:48.446261+0000","last_fullsized":"2024-03-20T09:27:48.446261+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:26:54.028578+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:20:12.169658+0000","last_clean_scrub_stamp":"2024-03-20T09:26:54.028578+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:28:14.904652+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,5],"acting":[6,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"795'8922","reported_seq":9190,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:51.211178+0000","last_change":"2024-03-20T09:25:57.427967+0000","last_active":"2024-03-20T09:27:51.211178+0000","last_peered":"2024-03-20T09:27:51.211178+0000","last_clean":"2024-03-20T09:27:51.211178+0000","last_became_active":"2024-03-20T09:25:57.422371+0000","last_became_peered":"2024-03-20T09:25:57.422371+0000","last_unstale":"2024-03-20T09:27:51.211178+0000","last_undegraded":"2024-03-20T09:27:51.211178+0000","last_fullsized":"2024-03-20T09:27:51.211178+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":750,"parent":"0.0","parent_split_bits":0,"last_scrub":"745'6949","last_scrub_stamp":"2024-03-20T09:25:38.184935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:38.184935+0000","objects_scrubbed":6887,"log_size":8922,"log_dups_size":0,"ondisk_log_size":8922,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:27:53.827926+0000","scrub_duration":2351,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164626432,"num_objects":2512,"num_object_clones":0,"num_object_copies":7536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4925,"num_write_kb":237984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"20'137","reported_seq":1606,"reported_epoch":795,"state":"active+undersized","last_fresh":"2024-03-20T09:27:48.446566+0000","last_change":"2024-03-20T09:26:04.492201+0000","last_active":"2024-03-20T09:27:48.446566+0000","last_peered":"2024-03-20T09:27:48.446566+0000","last_clean":"2024-03-20T09:26:00.452752+0000","last_became_active":"2024-03-20T09:26:04.492201+0000","last_became_peered":"2024-03-20T09:26:04.492201+0000","last_unstale":"2024-03-20T09:27:48.446566+0000","last_undegraded":"2024-03-20T09:27:48.446566+0000","last_fullsized":"2024-03-20T09:26:04.488584+0000","mapping_epoch":754,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:16.978732+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:14:10.560890+0000","last_clean_scrub_stamp":"2024-03-20T09:25:16.978732+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:27:52.116663+0000","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"795'10267","reported_seq":10518,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:49.844619+0000","last_change":"2024-03-20T09:25:16.245567+0000","last_active":"2024-03-20T09:27:49.844619+0000","last_peered":"2024-03-20T09:27:49.844619+0000","last_clean":"2024-03-20T09:27:49.844619+0000","last_became_active":"2024-03-20T09:24:03.380630+0000","last_became_peered":"2024-03-20T09:24:03.380630+0000","last_unstale":"2024-03-20T09:27:49.844619+0000","last_undegraded":"2024-03-20T09:27:49.844619+0000","last_fullsized":"2024-03-20T09:27:49.844619+0000","mapping_epoch":709,"log_start":"719'200","ondisk_log_start":"719'200","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"733'5704","last_scrub_stamp":"2024-03-20T09:25:16.245344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:16.245344+0000","objects_scrubbed":5644,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1702,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341901312,"num_objects":5217,"num_object_clones":0,"num_object_copies":15651,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10267,"num_write_kb":495488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"20'137","reported_seq":1456,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:48.439167+0000","last_change":"2024-03-20T09:25:17.104512+0000","last_active":"2024-03-20T09:27:48.439167+0000","last_peered":"2024-03-20T09:27:48.439167+0000","last_clean":"2024-03-20T09:27:48.439167+0000","last_became_active":"2024-03-20T09:13:20.070596+0000","last_became_peered":"2024-03-20T09:13:20.070596+0000","last_unstale":"2024-03-20T09:27:48.439167+0000","last_undegraded":"2024-03-20T09:27:48.439167+0000","last_fullsized":"2024-03-20T09:27:48.439167+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:17.104330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:25:17.104330+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.9","version":"795'8901","reported_seq":9168,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:51.228544+0000","last_change":"2024-03-20T09:25:57.411434+0000","last_active":"2024-03-20T09:27:51.228544+0000","last_peered":"2024-03-20T09:27:51.228544+0000","last_clean":"2024-03-20T09:27:51.228544+0000","last_became_active":"2024-03-20T09:25:57.404010+0000","last_became_peered":"2024-03-20T09:25:57.404010+0000","last_unstale":"2024-03-20T09:27:51.228544+0000","last_undegraded":"2024-03-20T09:27:51.228544+0000","last_fullsized":"2024-03-20T09:27:51.228544+0000","mapping_epoch":715,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":750,"parent":"0.0","parent_split_bits":0,"last_scrub":"739'6312","last_scrub_stamp":"2024-03-20T09:25:27.604184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:27.604184+0000","objects_scrubbed":6258,"log_size":8901,"log_dups_size":0,"ondisk_log_size":8901,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:27:53.827897+0000","scrub_duration":1892,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164102144,"num_objects":2504,"num_object_clones":0,"num_object_copies":7512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4901,"num_write_kb":236960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"20'137","reported_seq":880,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:48.437101+0000","last_change":"2024-03-20T09:25:23.627331+0000","last_active":"2024-03-20T09:27:48.437101+0000","last_peered":"2024-03-20T09:27:48.437101+0000","last_clean":"2024-03-20T09:27:48.437101+0000","last_became_active":"2024-03-20T09:19:54.506134+0000","last_became_peered":"2024-03-20T09:19:54.506134+0000","last_unstale":"2024-03-20T09:27:48.437101+0000","last_undegraded":"2024-03-20T09:27:48.437101+0000","last_fullsized":"2024-03-20T09:27:48.437101+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:23.627211+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:18:52.573886+0000","last_clean_scrub_stamp":"2024-03-20T09:25:23.627211+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"795'8872","reported_seq":9135,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:50.256573+0000","last_change":"2024-03-20T09:25:57.450474+0000","last_active":"2024-03-20T09:27:50.256573+0000","last_peered":"2024-03-20T09:27:50.256573+0000","last_clean":"2024-03-20T09:27:50.256573+0000","last_became_active":"2024-03-20T09:25:57.446849+0000","last_became_peered":"2024-03-20T09:25:57.446849+0000","last_unstale":"2024-03-20T09:27:50.256573+0000","last_undegraded":"2024-03-20T09:27:50.256573+0000","last_fullsized":"2024-03-20T09:27:50.256573+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":750,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'6323","last_scrub_stamp":"2024-03-20T09:25:29.608640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:29.608640+0000","objects_scrubbed":6269,"log_size":8872,"log_dups_size":0,"ondisk_log_size":8872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:27:51.070109+0000","scrub_duration":1480,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163282944,"num_objects":2491,"num_object_clones":0,"num_object_copies":7473,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4902,"num_write_kb":236608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,3],"acting":[11,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.f","version":"20'137","reported_seq":1636,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:48.446360+0000","last_change":"2024-03-20T09:26:42.152183+0000","last_active":"2024-03-20T09:27:48.446360+0000","last_peered":"2024-03-20T09:27:48.446360+0000","last_clean":"2024-03-20T09:27:48.446360+0000","last_became_active":"2024-03-20T09:19:54.524397+0000","last_became_peered":"2024-03-20T09:19:54.524397+0000","last_unstale":"2024-03-20T09:27:48.446360+0000","last_undegraded":"2024-03-20T09:27:48.446360+0000","last_fullsized":"2024-03-20T09:27:48.446360+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:26:42.152081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:26:42.152081+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:28:09.361950+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"795'9935","reported_seq":10196,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:50.048632+0000","last_change":"2024-03-20T09:25:36.618305+0000","last_active":"2024-03-20T09:27:50.048632+0000","last_peered":"2024-03-20T09:27:50.048632+0000","last_clean":"2024-03-20T09:27:50.048632+0000","last_became_active":"2024-03-20T09:24:03.378961+0000","last_became_peered":"2024-03-20T09:24:03.378961+0000","last_unstale":"2024-03-20T09:27:50.048632+0000","last_undegraded":"2024-03-20T09:27:50.048632+0000","last_fullsized":"2024-03-20T09:27:50.048632+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"745'6858","last_scrub_stamp":"2024-03-20T09:25:36.617966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:36.617966+0000","objects_scrubbed":6792,"log_size":9935,"log_dups_size":0,"ondisk_log_size":9935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2094,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328925184,"num_objects":5019,"num_object_clones":0,"num_object_copies":15057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9935,"num_write_kb":478528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"570'138","reported_seq":1621,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:48.334454+0000","last_change":"2024-03-20T09:24:18.167726+0000","last_active":"2024-03-20T09:27:48.334454+0000","last_peered":"2024-03-20T09:27:48.334454+0000","last_clean":"2024-03-20T09:27:48.334454+0000","last_became_active":"2024-03-20T09:13:27.087686+0000","last_became_peered":"2024-03-20T09:13:27.087686+0000","last_unstale":"2024-03-20T09:27:48.334454+0000","last_undegraded":"2024-03-20T09:27:48.334454+0000","last_fullsized":"2024-03-20T09:27:48.334454+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"570'138","last_scrub_stamp":"2024-03-20T09:24:18.167603+0000","last_deep_scrub":"570'138","last_deep_scrub_stamp":"2024-03-20T09:16:13.725035+0000","last_clean_scrub_stamp":"2024-03-20T09:24:18.167603+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"795'9660","reported_seq":9923,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:49.435238+0000","last_change":"2024-03-20T09:25:39.076376+0000","last_active":"2024-03-20T09:27:49.435238+0000","last_peered":"2024-03-20T09:27:49.435238+0000","last_clean":"2024-03-20T09:27:49.435238+0000","last_became_active":"2024-03-20T09:24:03.381499+0000","last_became_peered":"2024-03-20T09:24:03.381499+0000","last_unstale":"2024-03-20T09:27:49.435238+0000","last_undegraded":"2024-03-20T09:27:49.435238+0000","last_fullsized":"2024-03-20T09:27:49.435238+0000","mapping_epoch":709,"log_start":"0'0","ondisk_log_start":"0'0","created":709,"last_epoch_clean":710,"parent":"0.0","parent_split_bits":0,"last_scrub":"745'6934","last_scrub_stamp":"2024-03-20T09:25:39.076136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:39.076136+0000","objects_scrubbed":6878,"log_size":9660,"log_dups_size":0,"ondisk_log_size":9660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1735,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328990720,"num_objects":5020,"num_object_clones":0,"num_object_copies":15060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9660,"num_write_kb":469760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"20'137","reported_seq":1186,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:48.437044+0000","last_change":"2024-03-20T09:25:34.697040+0000","last_active":"2024-03-20T09:27:48.437044+0000","last_peered":"2024-03-20T09:27:48.437044+0000","last_clean":"2024-03-20T09:27:48.437044+0000","last_became_active":"2024-03-20T09:19:54.502529+0000","last_became_peered":"2024-03-20T09:19:54.502529+0000","last_unstale":"2024-03-20T09:27:48.437044+0000","last_undegraded":"2024-03-20T09:27:48.437044+0000","last_fullsized":"2024-03-20T09:27:48.437044+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:34.696897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:25:34.696897+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.15","version":"795'8987","reported_seq":16701,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:50.892516+0000","last_change":"2024-03-20T09:26:46.357175+0000","last_active":"2024-03-20T09:27:50.892516+0000","last_peered":"2024-03-20T09:27:50.892516+0000","last_clean":"2024-03-20T09:27:50.892516+0000","last_became_active":"2024-03-20T09:26:27.328065+0000","last_became_peered":"2024-03-20T09:26:27.328065+0000","last_unstale":"2024-03-20T09:27:50.892516+0000","last_undegraded":"2024-03-20T09:27:50.892516+0000","last_fullsized":"2024-03-20T09:27:50.892516+0000","mapping_epoch":763,"log_start":"0'0","ondisk_log_start":"0'0","created":749,"last_epoch_clean":764,"parent":"0.0","parent_split_bits":5,"last_scrub":"772'8306","last_scrub_stamp":"2024-03-20T09:26:46.356961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:26:46.356961+0000","objects_scrubbed":3184,"log_size":8987,"log_dups_size":0,"ondisk_log_size":8987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:27:56.490198+0000","scrub_duration":365,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163840000,"num_objects":2500,"num_object_clones":0,"num_object_copies":7500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":684,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"20'137","reported_seq":1491,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:48.440721+0000","last_change":"2024-03-20T09:25:34.404637+0000","last_active":"2024-03-20T09:27:48.440721+0000","last_peered":"2024-03-20T09:27:48.440721+0000","last_clean":"2024-03-20T09:27:48.440721+0000","last_became_active":"2024-03-20T09:10:34.096758+0000","last_became_peered":"2024-03-20T09:10:34.096758+0000","last_unstale":"2024-03-20T09:27:48.440721+0000","last_undegraded":"2024-03-20T09:27:48.440721+0000","last_fullsized":"2024-03-20T09:27:48.440721+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:34.404553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:25:34.404553+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"795'9078","reported_seq":24426,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:51.396252+0000","last_change":"2024-03-20T09:27:16.362919+0000","last_active":"2024-03-20T09:27:51.396252+0000","last_peered":"2024-03-20T09:27:51.396252+0000","last_clean":"2024-03-20T09:27:51.396252+0000","last_became_active":"2024-03-20T09:27:16.347722+0000","last_became_peered":"2024-03-20T09:27:16.347722+0000","last_unstale":"2024-03-20T09:27:51.396252+0000","last_undegraded":"2024-03-20T09:27:51.396252+0000","last_fullsized":"2024-03-20T09:27:51.396252+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":749,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":5,"last_scrub":"736'5943","last_scrub_stamp":"2024-03-20T09:25:21.190503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:21.190503+0000","objects_scrubbed":5872,"log_size":9078,"log_dups_size":0,"ondisk_log_size":9078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:27:55.507665+0000","scrub_duration":2104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168361984,"num_objects":2569,"num_object_clones":0,"num_object_copies":7707,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5022,"num_write_kb":242912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3770,"num_bytes_recovered":228917248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"20'137","reported_seq":1574,"reported_epoch":795,"state":"active+undersized","last_fresh":"2024-03-20T09:27:48.421673+0000","last_change":"2024-03-20T09:26:04.509153+0000","last_active":"2024-03-20T09:27:48.421673+0000","last_peered":"2024-03-20T09:27:48.421673+0000","last_clean":"2024-03-20T09:26:00.451330+0000","last_became_active":"2024-03-20T09:26:04.509153+0000","last_became_peered":"2024-03-20T09:26:04.509153+0000","last_unstale":"2024-03-20T09:27:48.421673+0000","last_undegraded":"2024-03-20T09:27:48.421673+0000","last_fullsized":"2024-03-20T09:26:04.498014+0000","mapping_epoch":754,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:29.370054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:25:29.370054+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11],"acting":[1,11],"avail_no_missing":["1","11"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"795'9205","reported_seq":12234,"reported_epoch":795,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-20T09:27:51.494759+0000","last_change":"2024-03-20T09:26:41.118685+0000","last_active":"2024-03-20T09:27:51.494759+0000","last_peered":"2024-03-20T09:27:51.494759+0000","last_clean":"2024-03-20T09:25:58.390751+0000","last_became_active":"2024-03-20T09:26:15.201956+0000","last_became_peered":"2024-03-20T09:26:15.201956+0000","last_unstale":"2024-03-20T09:27:51.494759+0000","last_undegraded":"2024-03-20T09:26:09.915082+0000","last_fullsized":"2024-03-20T09:26:05.489705+0000","mapping_epoch":755,"log_start":"0'0","ondisk_log_start":"0'0","created":749,"last_epoch_clean":750,"parent":"0.0","parent_split_bits":5,"last_scrub":"738'6270","last_scrub_stamp":"2024-03-20T09:25:24.693262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:24.693262+0000","objects_scrubbed":6219,"log_size":9205,"log_dups_size":0,"ondisk_log_size":9205,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1532,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170033152,"num_objects":2595,"num_object_clones":0,"num_object_copies":7785,"num_objects_missing_on_primary":2899,"num_objects_missing":2899,"num_objects_degraded":2595,"num_objects_misplaced":5126,"num_objects_unfound":0,"num_objects_dirty":2595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5077,"num_write_kb":245472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":905,"num_bytes_recovered":59244544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11],"acting":[2,3],"avail_no_missing":["3","4","5"],"object_location_counts":[{"shards":"2,3","objects":29},{"shards":"2,3,4,5","objects":6},{"shards":"3,4,5","objects":2560}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"20'137","reported_seq":1028,"reported_epoch":795,"state":"active+undersized","last_fresh":"2024-03-20T09:27:48.440686+0000","last_change":"2024-03-20T09:26:04.491492+0000","last_active":"2024-03-20T09:27:48.440686+0000","last_peered":"2024-03-20T09:27:48.440686+0000","last_clean":"2024-03-20T09:26:00.524966+0000","last_became_active":"2024-03-20T09:26:04.491492+0000","last_became_peered":"2024-03-20T09:26:04.491492+0000","last_unstale":"2024-03-20T09:27:48.440686+0000","last_undegraded":"2024-03-20T09:27:48.440686+0000","last_fullsized":"2024-03-20T09:26:04.488080+0000","mapping_epoch":754,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:31.378069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:25:31.378069+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"795'9115","reported_seq":16880,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:51.192868+0000","last_change":"2024-03-20T09:26:49.538638+0000","last_active":"2024-03-20T09:27:51.192868+0000","last_peered":"2024-03-20T09:27:51.192868+0000","last_clean":"2024-03-20T09:27:51.192868+0000","last_became_active":"2024-03-20T09:26:49.528656+0000","last_became_peered":"2024-03-20T09:26:49.528656+0000","last_unstale":"2024-03-20T09:27:51.192868+0000","last_undegraded":"2024-03-20T09:27:51.192868+0000","last_fullsized":"2024-03-20T09:27:51.192868+0000","mapping_epoch":775,"log_start":"0'0","ondisk_log_start":"0'0","created":749,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":5,"last_scrub":"740'6530","last_scrub_stamp":"2024-03-20T09:25:29.595052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:29.595052+0000","objects_scrubbed":6463,"log_size":9115,"log_dups_size":0,"ondisk_log_size":9115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:27:55.810100+0000","scrub_duration":2087,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168460288,"num_objects":2571,"num_object_clones":0,"num_object_copies":7713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5030,"num_write_kb":243200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3746,"num_bytes_recovered":224919552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.11","version":"20'137","reported_seq":1435,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:48.416029+0000","last_change":"2024-03-20T09:25:15.809993+0000","last_active":"2024-03-20T09:27:48.416029+0000","last_peered":"2024-03-20T09:27:48.416029+0000","last_clean":"2024-03-20T09:27:48.416029+0000","last_became_active":"2024-03-20T09:20:25.342982+0000","last_became_peered":"2024-03-20T09:20:25.342982+0000","last_unstale":"2024-03-20T09:27:48.416029+0000","last_undegraded":"2024-03-20T09:27:48.416029+0000","last_fullsized":"2024-03-20T09:27:48.416029+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:15.809834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:25:15.809834+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"795'9001","reported_seq":24118,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:49.788225+0000","last_change":"2024-03-20T09:27:01.272806+0000","last_active":"2024-03-20T09:27:49.788225+0000","last_peered":"2024-03-20T09:27:49.788225+0000","last_clean":"2024-03-20T09:27:49.788225+0000","last_became_active":"2024-03-20T09:27:01.260246+0000","last_became_peered":"2024-03-20T09:27:01.260246+0000","last_unstale":"2024-03-20T09:27:49.788225+0000","last_undegraded":"2024-03-20T09:27:49.788225+0000","last_fullsized":"2024-03-20T09:27:49.788225+0000","mapping_epoch":781,"log_start":"0'0","ondisk_log_start":"0'0","created":749,"last_epoch_clean":782,"parent":"0.0","parent_split_bits":5,"last_scrub":"738'6080","last_scrub_stamp":"2024-03-20T09:25:24.081378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:24.081378+0000","objects_scrubbed":6048,"log_size":9001,"log_dups_size":0,"ondisk_log_size":9001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:27:51.637878+0000","scrub_duration":958,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167411712,"num_objects":2555,"num_object_clones":0,"num_object_copies":7665,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4945,"num_write_kb":239968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3712,"num_bytes_recovered":221380608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,11],"acting":[3,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.16","version":"20'137","reported_seq":1259,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:48.422455+0000","last_change":"2024-03-20T09:25:25.209569+0000","last_active":"2024-03-20T09:27:48.422455+0000","last_peered":"2024-03-20T09:27:48.422455+0000","last_clean":"2024-03-20T09:27:48.422455+0000","last_became_active":"2024-03-20T09:13:27.087760+0000","last_became_peered":"2024-03-20T09:13:27.087760+0000","last_unstale":"2024-03-20T09:27:48.422455+0000","last_undegraded":"2024-03-20T09:27:48.422455+0000","last_fullsized":"2024-03-20T09:27:48.422455+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:25.209421+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:16:15.448920+0000","last_clean_scrub_stamp":"2024-03-20T09:25:25.209421+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.10","version":"795'8906","reported_seq":20283,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:51.246562+0000","last_change":"2024-03-20T09:27:02.283060+0000","last_active":"2024-03-20T09:27:51.246562+0000","last_peered":"2024-03-20T09:27:51.246562+0000","last_clean":"2024-03-20T09:27:51.246562+0000","last_became_active":"2024-03-20T09:27:02.273745+0000","last_became_peered":"2024-03-20T09:27:02.273745+0000","last_unstale":"2024-03-20T09:27:51.246562+0000","last_undegraded":"2024-03-20T09:27:51.246562+0000","last_fullsized":"2024-03-20T09:27:51.246562+0000","mapping_epoch":782,"log_start":"0'0","ondisk_log_start":"0'0","created":749,"last_epoch_clean":783,"parent":"0.0","parent_split_bits":5,"last_scrub":"745'6949","last_scrub_stamp":"2024-03-20T09:25:38.184935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:38.184935+0000","objects_scrubbed":6887,"log_size":8906,"log_dups_size":0,"ondisk_log_size":8906,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:27:51.872052+0000","scrub_duration":2351,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165675008,"num_objects":2528,"num_object_clones":0,"num_object_copies":7584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4909,"num_write_kb":237984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3700,"num_bytes_recovered":222101504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1],"acting":[0,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.17","version":"20'137","reported_seq":1589,"reported_epoch":795,"state":"active+undersized","last_fresh":"2024-03-20T09:27:48.419572+0000","last_change":"2024-03-20T09:26:04.492624+0000","last_active":"2024-03-20T09:27:48.419572+0000","last_peered":"2024-03-20T09:27:48.419572+0000","last_clean":"2024-03-20T09:26:00.449109+0000","last_became_active":"2024-03-20T09:26:04.492624+0000","last_became_peered":"2024-03-20T09:26:04.492624+0000","last_unstale":"2024-03-20T09:27:48.419572+0000","last_undegraded":"2024-03-20T09:27:48.419572+0000","last_fullsized":"2024-03-20T09:26:04.489715+0000","mapping_epoch":754,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:21.211866+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:23.640533+0000","last_clean_scrub_stamp":"2024-03-20T09:25:21.211866+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:27:52.023564+0000","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5],"acting":[8,5],"avail_no_missing":["8","5"],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"6.13","version":"795'8939","reported_seq":16712,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:50.020595+0000","last_change":"2024-03-20T09:26:42.365001+0000","last_active":"2024-03-20T09:27:50.020595+0000","last_peered":"2024-03-20T09:27:50.020595+0000","last_clean":"2024-03-20T09:27:50.020595+0000","last_became_active":"2024-03-20T09:26:42.353570+0000","last_became_peered":"2024-03-20T09:26:42.353570+0000","last_unstale":"2024-03-20T09:27:50.020595+0000","last_undegraded":"2024-03-20T09:27:50.020595+0000","last_fullsized":"2024-03-20T09:27:50.020595+0000","mapping_epoch":769,"log_start":"0'0","ondisk_log_start":"0'0","created":749,"last_epoch_clean":770,"parent":"0.0","parent_split_bits":5,"last_scrub":"739'6169","last_scrub_stamp":"2024-03-20T09:25:25.349835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:25.349835+0000","objects_scrubbed":6123,"log_size":8939,"log_dups_size":0,"ondisk_log_size":8939,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:27:52.642368+0000","scrub_duration":1411,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166756352,"num_objects":2545,"num_object_clones":0,"num_object_copies":7635,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4916,"num_write_kb":238720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3751,"num_bytes_recovered":224788480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.14","version":"20'137","reported_seq":1663,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:48.420966+0000","last_change":"2024-03-20T09:24:31.104958+0000","last_active":"2024-03-20T09:27:48.420966+0000","last_peered":"2024-03-20T09:27:48.420966+0000","last_clean":"2024-03-20T09:27:48.420966+0000","last_became_active":"2024-03-20T09:20:25.349082+0000","last_became_peered":"2024-03-20T09:20:25.349082+0000","last_unstale":"2024-03-20T09:27:48.420966+0000","last_undegraded":"2024-03-20T09:27:48.420966+0000","last_fullsized":"2024-03-20T09:27:48.420966+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:24:31.104823+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:16:15.754362+0000","last_clean_scrub_stamp":"2024-03-20T09:24:31.104823+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.12","version":"795'9040","reported_seq":17035,"reported_epoch":795,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-20T09:27:51.501334+0000","last_change":"2024-03-20T09:26:00.829028+0000","last_active":"2024-03-20T09:27:51.501334+0000","last_peered":"2024-03-20T09:27:51.501334+0000","last_clean":"2024-03-20T09:25:58.426910+0000","last_became_active":"2024-03-20T09:26:00.706066+0000","last_became_peered":"2024-03-20T09:26:00.706066+0000","last_unstale":"2024-03-20T09:27:51.501334+0000","last_undegraded":"2024-03-20T09:27:51.501334+0000","last_fullsized":"2024-03-20T09:26:00.448237+0000","mapping_epoch":752,"log_start":"0'0","ondisk_log_start":"0'0","created":749,"last_epoch_clean":750,"parent":"0.0","parent_split_bits":5,"last_scrub":"733'5454","last_scrub_stamp":"2024-03-20T09:25:14.989721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:24:02.368833+0000","last_clean_scrub_stamp":"2024-03-20T09:25:14.989721+0000","objects_scrubbed":5409,"log_size":9040,"log_dups_size":0,"ondisk_log_size":9040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1387,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164790272,"num_objects":2515,"num_object_clones":0,"num_object_copies":7545,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1714,"num_objects_unfound":0,"num_objects_dirty":2515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5015,"num_write_kb":240928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2581,"num_bytes_recovered":168886272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,4],"acting":[2,4],"avail_no_missing":["2","3","4"],"object_location_counts":[{"shards":"2,3,4","objects":857},{"shards":"2,4","objects":1658}],"blocked_by":[],"up_primary":9,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"20'137","reported_seq":927,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:48.334219+0000","last_change":"2024-03-20T09:27:22.292596+0000","last_active":"2024-03-20T09:27:48.334219+0000","last_peered":"2024-03-20T09:27:48.334219+0000","last_clean":"2024-03-20T09:27:48.334219+0000","last_became_active":"2024-03-20T09:12:14.307581+0000","last_became_peered":"2024-03-20T09:12:14.307581+0000","last_unstale":"2024-03-20T09:27:48.334219+0000","last_undegraded":"2024-03-20T09:27:48.334219+0000","last_fullsized":"2024-03-20T09:27:48.334219+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:27:22.292458+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:23:41.251546+0000","last_clean_scrub_stamp":"2024-03-20T09:27:22.292458+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:28:35.689404+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"20'137","reported_seq":1186,"reported_epoch":795,"state":"active+clean","last_fresh":"2024-03-20T09:27:48.438101+0000","last_change":"2024-03-20T09:25:32.603971+0000","last_active":"2024-03-20T09:27:48.438101+0000","last_peered":"2024-03-20T09:27:48.438101+0000","last_clean":"2024-03-20T09:27:48.438101+0000","last_became_active":"2024-03-20T09:19:54.502391+0000","last_became_peered":"2024-03-20T09:19:54.502391+0000","last_unstale":"2024-03-20T09:27:48.438101+0000","last_undegraded":"2024-03-20T09:27:48.438101+0000","last_fullsized":"2024-03-20T09:27:48.438101+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:32.603840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:25:32.603840+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":26,"stat_sum":{"num_bytes":5321752600,"num_objects":81205,"num_object_clones":0,"num_object_copies":243615,"num_objects_missing_on_primary":5447,"num_objects_missing":5447,"num_objects_degraded":17739,"num_objects_misplaced":11308,"num_objects_unfound":0,"num_objects_dirty":81205,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146111,"num_write_kb":6968065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26970,"num_bytes_recovered":1636958208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17974767616,"data_stored":17974755400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":239504,"ondisk_log_size":239504,"up":73,"acting":73,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":630620,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":7,"num_write":12,"num_write_kb":200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":471933,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4248,"ondisk_log_size":4248,"up":86,"acting":86,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860798,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1458340,"kb_used_data":751068,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":92286812,"statfs":{"total":95995035648,"available":94501695488,"internally_reserved":0,"allocated":769093632,"data_stored":768605007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":724236740},"hb_peers":[0,1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":89,"seq":382252089710,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1829176,"kb_used_data":1320424,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":91915976,"statfs":{"total":95995035648,"available":94121959424,"internally_reserved":0,"allocated":1352114176,"data_stored":1351799832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":520944064},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":80,"seq":343597384075,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1437336,"kb_used_data":1145168,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":92307816,"statfs":{"total":95995035648,"available":94523203584,"internally_reserved":0,"allocated":1172652032,"data_stored":1172167752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":299170240},"hb_peers":[0,1,2,3,4,5,6,7,8,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":70,"seq":300647711119,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1208356,"kb_used_data":798172,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92536796,"statfs":{"total":95995035648,"available":94757679104,"internally_reserved":0,"allocated":817328128,"data_stored":816839503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":420018630},"hb_peers":[0,1,2,3,4,5,6,7,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":60,"seq":257698038164,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2334384,"kb_used_data":1621540,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":91410768,"statfs":{"total":95995035648,"available":93604626432,"internally_reserved":0,"allocated":1660456960,"data_stored":1659964783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":729938378},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332505,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1880612,"kb_used_data":1299676,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":91864540,"statfs":{"total":95995035648,"available":94069288960,"internally_reserved":0,"allocated":1330868224,"data_stored":1330379599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":594868678},"hb_peers":[0,1,2,3,4,5,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575283,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2298092,"kb_used_data":1608864,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":91447060,"statfs":{"total":95995035648,"available":93641789440,"internally_reserved":0,"allocated":1647476736,"data_stored":1646984039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":705755590},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738802,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2153892,"kb_used_data":1450780,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":91591260,"statfs":{"total":95995035648,"available":93789450240,"internally_reserved":0,"allocated":1485598720,"data_stored":1485110095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411756,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2094192,"kb_used_data":1621540,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":91650960,"statfs":{"total":95995035648,"available":93850583040,"internally_reserved":0,"allocated":1660456960,"data_stored":1659964783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150120,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3626728,"kb_used_data":2937500,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":90118424,"statfs":{"total":95995035648,"available":92281266176,"internally_reserved":0,"allocated":3008000000,"data_stored":3007511375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":705755591},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953894,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2061928,"kb_used_data":1530972,"kb_used_omap":1,"kb_used_meta":530942,"kb_avail":91683224,"statfs":{"total":95995035648,"available":93883621376,"internally_reserved":0,"allocated":1567715328,"data_stored":1567226703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":543685061},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":443,"seq":1902670512336,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2261492,"kb_used_data":1776168,"kb_used_omap":181,"kb_used_meta":485130,"kb_avail":91483660,"statfs":{"total":95995035648,"available":93679267840,"internally_reserved":0,"allocated":1818796032,"data_stored":1818299783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":186131,"internal_metadata":496773357},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1479868416,"data_stored":1479868416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1640173568,"data_stored":1640169496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1451229184,"data_stored":1451229184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3002138624,"data_stored":3002138624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1564540928,"data_stored":1564540928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1755910144,"data_stored":1755906072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1329070080,"data_stored":1329070080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1653538816,"data_stored":1653538816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":811925504,"data_stored":811925504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1170341888,"data_stored":1170341888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1349062656,"data_stored":1349058584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":766967808,"data_stored":766967808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:27:53.992 INFO:tasks.thrashosds.thrasher:Setting 6.8 to [8, 11, 3] 2024-03-20T09:27:53.992 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '6.8', '8', '11', '3'] 2024-03-20T09:27:53.992 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pg-upmap 6.8 8 11 3 2024-03-20T09:27:54.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:54 smithi115 ceph-mon[72476]: from='client.26336 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:27:54.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:54 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:27:54.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:54 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3009280415' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:27:54.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:54 smithi115 ceph-mon[72476]: osdmap e797: 12 total, 11 up, 12 in 2024-03-20T09:27:54.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:54 smithi122 ceph-mon[73495]: from='client.26336 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:27:54.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:54 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:27:54.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:54 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3009280415' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:27:54.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:54 smithi122 ceph-mon[73495]: osdmap e797: 12 total, 11 up, 12 in 2024-03-20T09:27:54.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:54 smithi026 ceph-mon[68309]: from='client.26336 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:27:54.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:54 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:27:54.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:54 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3009280415' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:27:54.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:54 smithi026 ceph-mon[68309]: osdmap e797: 12 total, 11 up, 12 in 2024-03-20T09:27:54.832 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:27:54.906 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:27:55.054 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:27:55.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:55.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:27:55.160 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:27:55.196 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:55.302 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:27:55.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:55.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:27:55.407 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:27:55.461 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:55.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:55 smithi115 ceph-mon[72476]: pgmap v1936: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 294 op/s; 17739/243621 objects degraded (7.281%); 11308/243621 objects misplaced (4.642%); 2.0 MiB/s, 32 objects/s recovering 2024-03-20T09:27:55.688 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:27:55.688 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:27:55.688 INFO:teuthology.orchestra.run.smithi115.stdout: "complaint_time": 30, 2024-03-20T09:27:55.688 INFO:teuthology.orchestra.run.smithi115.stdout: "num_blocked_ops": 0 2024-03-20T09:27:55.688 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:27:55.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:55 smithi122 ceph-mon[73495]: pgmap v1936: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 294 op/s; 17739/243621 objects degraded (7.281%); 11308/243621 objects misplaced (4.642%); 2.0 MiB/s, 32 objects/s recovering 2024-03-20T09:27:55.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:55 smithi026 ceph-mon[68309]: pgmap v1936: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 294 op/s; 17739/243621 objects degraded (7.281%); 11308/243621 objects misplaced (4.642%); 2.0 MiB/s, 32 objects/s recovering 2024-03-20T09:27:56.489 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-20T09:27:56.785 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:27:56.910 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:27:57.368 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:57 smithi115 ceph-mon[72476]: pgmap v1937: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 4.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 450 op/s; 17544/238194 objects degraded (7.365%); 11143/238194 objects misplaced (4.678%); 2.2 MiB/s, 34 objects/s recovering 2024-03-20T09:27:57.368 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:57 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:27:57.368 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:57 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:27:57.368 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:57 smithi115 ceph-mon[72476]: osdmap e798: 12 total, 11 up, 12 in 2024-03-20T09:27:57.617 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:57 smithi026 ceph-mon[68309]: pgmap v1937: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 4.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 450 op/s; 17544/238194 objects degraded (7.365%); 11143/238194 objects misplaced (4.678%); 2.2 MiB/s, 34 objects/s recovering 2024-03-20T09:27:57.617 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:57 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:27:57.617 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:57 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:27:57.617 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:57 smithi026 ceph-mon[68309]: osdmap e798: 12 total, 11 up, 12 in 2024-03-20T09:27:57.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:57 smithi122 ceph-mon[73495]: pgmap v1937: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 4.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 450 op/s; 17544/238194 objects degraded (7.365%); 11143/238194 objects misplaced (4.678%); 2.2 MiB/s, 34 objects/s recovering 2024-03-20T09:27:57.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:57 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:27:57.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:57 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:27:57.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:57 smithi122 ceph-mon[73495]: osdmap e798: 12 total, 11 up, 12 in 2024-03-20T09:27:57.880 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:27:57.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:57.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:27:57.985 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:27:58.019 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:58.465 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:58 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 17544/238194 objects degraded (7.365%), 5 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:27:58.465 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:58 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:27:58.577 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:27:58.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:58.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:27:58.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:58 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 17544/238194 objects degraded (7.365%), 5 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:27:58.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:58 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:27:58.685 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:27:58.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:58 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 17544/238194 objects degraded (7.365%), 5 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:27:58.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:58 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:27:58.741 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:59.353 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:27:59 smithi026 ceph-mon[68309]: pgmap v1939: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 4.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 381 op/s; 17224/236745 objects degraded (7.275%); 10490/236745 objects misplaced (4.431%); 4.3 MiB/s, 68 objects/s recovering 2024-03-20T09:27:59.527 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:27:59 smithi115 ceph-mon[72476]: pgmap v1939: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 4.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 381 op/s; 17224/236745 objects degraded (7.275%); 10490/236745 objects misplaced (4.431%); 4.3 MiB/s, 68 objects/s recovering 2024-03-20T09:27:59.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:59.528 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:27:59.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:27:59.529 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:27:59.529 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:27:59.529 INFO:teuthology.orchestra.run.smithi115.stdout: "size": 20, 2024-03-20T09:27:59.529 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 600, 2024-03-20T09:27:59.530 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [ 2024-03-20T09:27:59.530 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.530 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:133737 5.1 e640/637 5:882b2c30:::benchmark_data_smithi122_157894_object3750:head v 640'8271, mlcod=611'228)", 2024-03-20T09:27:59.530 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:19:59.599626+0000", 2024-03-20T09:27:59.530 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 479.84856075499999, 2024-03-20T09:27:59.530 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.087846301000000002, 2024-03-20T09:27:59.530 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:27:59.530 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:27:59.530 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:27:59.530 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.530 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:27:59.531 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:19:59.599626+0000", 2024-03-20T09:27:59.531 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.531 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.531 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.531 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:27:59.531 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:19:59.599626+0000", 2024-03-20T09:27:59.531 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.531 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.531 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.531 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:27:59.531 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:19:59.599627+0000", 2024-03-20T09:27:59.531 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.2900000000000002e-07 2024-03-20T09:27:59.531 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.532 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.532 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:27:59.532 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:19:59.599635+0000", 2024-03-20T09:27:59.532 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.2649999999999999e-06 2024-03-20T09:27:59.532 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.532 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.532 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:27:59.532 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:19:59.599647+0000", 2024-03-20T09:27:59.532 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.199e-05 2024-03-20T09:27:59.532 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.532 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.532 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:27:59.532 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:19:59.599653+0000", 2024-03-20T09:27:59.533 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1970000000000001e-06 2024-03-20T09:27:59.533 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.533 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.533 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:27:59.533 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:19:59.623436+0000", 2024-03-20T09:27:59.533 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.023783420999999999 2024-03-20T09:27:59.533 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.533 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.533 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:27:59.533 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:19:59.623463+0000", 2024-03-20T09:27:59.533 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6812e-05 2024-03-20T09:27:59.533 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.533 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.533 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:27:59.534 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:19:59.687422+0000", 2024-03-20T09:27:59.534 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.063959489999999994 2024-03-20T09:27:59.534 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.534 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.534 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:27:59.534 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:19:59.687473+0000", 2024-03-20T09:27:59.534 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0297000000000003e-05 2024-03-20T09:27:59.534 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.534 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:27:59.534 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.534 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.534 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.534 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:192903 5.5 e668/637 5:ad82fe62:::benchmark_data_smithi122_157894_object62916:head v 668'12075, mlcod=620'3813)", 2024-03-20T09:27:59.534 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:21:56.998657+0000", 2024-03-20T09:27:59.535 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 362.44952995, 2024-03-20T09:27:59.535 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.087689238000000003, 2024-03-20T09:27:59.535 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:27:59.535 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:27:59.535 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:27:59.535 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.535 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:27:59.535 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:21:56.998657+0000", 2024-03-20T09:27:59.535 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.535 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.535 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.535 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:27:59.535 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:21:56.998657+0000", 2024-03-20T09:27:59.535 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.535 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.536 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.536 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:27:59.536 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:21:56.998658+0000", 2024-03-20T09:27:59.536 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7099999999999997e-07 2024-03-20T09:27:59.536 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.536 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.536 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:27:59.536 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:21:56.998662+0000", 2024-03-20T09:27:59.536 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6129999999999997e-06 2024-03-20T09:27:59.536 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.536 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.536 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:27:59.536 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:21:56.998669+0000", 2024-03-20T09:27:59.536 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.0679999999999999e-06 2024-03-20T09:27:59.537 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.537 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.537 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:27:59.537 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:21:56.998674+0000", 2024-03-20T09:27:59.537 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5279999999999997e-06 2024-03-20T09:27:59.537 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.537 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.537 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:27:59.537 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:21:57.017425+0000", 2024-03-20T09:27:59.537 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.018751196000000001 2024-03-20T09:27:59.537 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.537 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.537 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:27:59.537 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:21:57.017450+0000", 2024-03-20T09:27:59.538 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.4752999999999998e-05 2024-03-20T09:27:59.538 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.538 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.538 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:27:59.538 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:21:57.086306+0000", 2024-03-20T09:27:59.538 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.068855792999999998 2024-03-20T09:27:59.538 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.538 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.538 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:27:59.538 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:21:57.086346+0000", 2024-03-20T09:27:59.538 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.0915999999999999e-05 2024-03-20T09:27:59.538 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.538 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:27:59.538 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.539 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.539 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.539 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:208841 5.5 e675/637 5:a3eee9ed:::benchmark_data_smithi122_157894_object78854:head v 675'13076, mlcod=623'4831)", 2024-03-20T09:27:59.539 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:23.178048+0000", 2024-03-20T09:27:59.539 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 336.27013924699997, 2024-03-20T09:27:59.539 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.089451347, 2024-03-20T09:27:59.539 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:27:59.539 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:27:59.539 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:27:59.539 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.539 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:27:59.539 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178048+0000", 2024-03-20T09:27:59.539 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.539 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.540 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.540 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:27:59.540 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178048+0000", 2024-03-20T09:27:59.540 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.540 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.540 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.540 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:27:59.540 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178049+0000", 2024-03-20T09:27:59.540 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.153e-06 2024-03-20T09:27:59.540 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.540 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.540 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:27:59.540 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178059+0000", 2024-03-20T09:27:59.540 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0169e-05 2024-03-20T09:27:59.541 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.541 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.541 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:27:59.541 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178076+0000", 2024-03-20T09:27:59.541 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7096e-05 2024-03-20T09:27:59.541 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.541 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.541 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:27:59.541 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178086+0000", 2024-03-20T09:27:59.541 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.6320000000000003e-06 2024-03-20T09:27:59.541 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.541 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.541 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:27:59.541 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178186+0000", 2024-03-20T09:27:59.542 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000100503 2024-03-20T09:27:59.542 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.542 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.542 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:27:59.542 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178215+0000", 2024-03-20T09:27:59.542 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8164e-05 2024-03-20T09:27:59.542 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.542 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.542 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:27:59.542 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.267448+0000", 2024-03-20T09:27:59.542 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.089233281999999997 2024-03-20T09:27:59.542 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.542 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.542 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:27:59.542 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.267499+0000", 2024-03-20T09:27:59.543 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1347999999999997e-05 2024-03-20T09:27:59.543 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.543 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:27:59.543 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.543 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.543 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.543 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:208845 5.5 e675/637 5:ad73ae47:::benchmark_data_smithi122_157894_object78858:head v 675'13077, mlcod=623'4831)", 2024-03-20T09:27:59.543 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:23.178111+0000", 2024-03-20T09:27:59.543 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 336.27007615600002, 2024-03-20T09:27:59.543 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.089416893999999997, 2024-03-20T09:27:59.543 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:27:59.543 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:27:59.543 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:27:59.543 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.544 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:27:59.544 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178111+0000", 2024-03-20T09:27:59.544 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.544 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.544 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.544 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:27:59.544 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178111+0000", 2024-03-20T09:27:59.544 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.544 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.544 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.544 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:27:59.544 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178111+0000", 2024-03-20T09:27:59.544 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8999999999999998e-07 2024-03-20T09:27:59.544 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.545 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.545 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:27:59.545 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178115+0000", 2024-03-20T09:27:59.545 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7799999999999998e-06 2024-03-20T09:27:59.545 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.545 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.545 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:27:59.545 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178121+0000", 2024-03-20T09:27:59.545 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4219999999999999e-06 2024-03-20T09:27:59.545 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.545 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.545 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:27:59.545 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178125+0000", 2024-03-20T09:27:59.545 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.0740000000000003e-06 2024-03-20T09:27:59.546 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.546 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.546 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:27:59.546 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.194801+0000", 2024-03-20T09:27:59.546 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.016675964000000001 2024-03-20T09:27:59.546 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.546 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.546 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:27:59.546 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.194818+0000", 2024-03-20T09:27:59.546 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7150999999999999e-05 2024-03-20T09:27:59.546 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.546 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.546 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:27:59.546 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.267503+0000", 2024-03-20T09:27:59.547 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.072685657000000001 2024-03-20T09:27:59.547 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.547 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.547 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:27:59.547 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.267528+0000", 2024-03-20T09:27:59.547 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.4556e-05 2024-03-20T09:27:59.547 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.547 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:27:59.547 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.547 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.547 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.547 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:223322 5.5 e682/637 5:a0cca45d:::benchmark_data_smithi122_157894_object93335:head v 682'14007, mlcod=627'5694)", 2024-03-20T09:27:59.547 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:46.235324+0000", 2024-03-20T09:27:59.548 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 313.21286322899999, 2024-03-20T09:27:59.548 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.088264128999999997, 2024-03-20T09:27:59.548 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:27:59.548 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:27:59.548 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:27:59.548 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.548 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:27:59.548 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235324+0000", 2024-03-20T09:27:59.548 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.548 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.548 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.548 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:27:59.548 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235324+0000", 2024-03-20T09:27:59.548 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.549 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.549 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.549 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:27:59.549 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235324+0000", 2024-03-20T09:27:59.549 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.9700000000000002e-07 2024-03-20T09:27:59.549 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.549 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.549 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:27:59.549 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235329+0000", 2024-03-20T09:27:59.549 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.899e-06 2024-03-20T09:27:59.549 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.549 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.549 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:27:59.549 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235336+0000", 2024-03-20T09:27:59.550 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.9739999999999996e-06 2024-03-20T09:27:59.550 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.550 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.550 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:27:59.550 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235341+0000", 2024-03-20T09:27:59.550 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6999999999999999e-06 2024-03-20T09:27:59.550 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.550 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.550 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:27:59.550 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235536+0000", 2024-03-20T09:27:59.550 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00019468400000000001 2024-03-20T09:27:59.550 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.550 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.551 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:27:59.551 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235559+0000", 2024-03-20T09:27:59.551 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3881000000000001e-05 2024-03-20T09:27:59.551 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.551 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.551 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:27:59.551 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.323553+0000", 2024-03-20T09:27:59.551 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.087993431999999996 2024-03-20T09:27:59.551 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.551 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.551 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:27:59.551 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.323588+0000", 2024-03-20T09:27:59.551 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.5162e-05 2024-03-20T09:27:59.551 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.552 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:27:59.552 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.552 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.552 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.552 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:223326 5.5 e682/637 5:a90b760b:::benchmark_data_smithi122_157894_object93339:head v 682'14008, mlcod=627'5694)", 2024-03-20T09:27:59.552 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:46.235363+0000", 2024-03-20T09:27:59.552 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 313.21282465500002, 2024-03-20T09:27:59.552 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.088261112000000003, 2024-03-20T09:27:59.552 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:27:59.552 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:27:59.552 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:27:59.552 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.552 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:27:59.552 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235363+0000", 2024-03-20T09:27:59.552 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.553 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.553 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.553 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:27:59.553 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235363+0000", 2024-03-20T09:27:59.553 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.553 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.553 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.553 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:27:59.553 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235363+0000", 2024-03-20T09:27:59.553 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8099999999999999e-07 2024-03-20T09:27:59.553 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.553 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.553 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:27:59.553 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235366+0000", 2024-03-20T09:27:59.554 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.061e-06 2024-03-20T09:27:59.554 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.554 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.554 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:27:59.554 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235371+0000", 2024-03-20T09:27:59.554 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2979999999999999e-06 2024-03-20T09:27:59.554 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.554 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.554 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:27:59.554 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235375+0000", 2024-03-20T09:27:59.554 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.8619999999999996e-06 2024-03-20T09:27:59.554 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.554 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.555 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:27:59.555 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.248068+0000", 2024-03-20T09:27:59.555 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.012692554 2024-03-20T09:27:59.555 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.555 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.555 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:27:59.555 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.248090+0000", 2024-03-20T09:27:59.555 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2747000000000001e-05 2024-03-20T09:27:59.555 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.555 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.555 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:27:59.555 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.323592+0000", 2024-03-20T09:27:59.555 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.075502001999999999 2024-03-20T09:27:59.555 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.556 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.556 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:27:59.556 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.323624+0000", 2024-03-20T09:27:59.556 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.1306999999999997e-05 2024-03-20T09:27:59.556 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.556 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:27:59.556 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.556 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.556 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.556 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:223337 5.5 e682/637 5:a415780d:::benchmark_data_smithi122_157894_object93350:head v 682'14009, mlcod=627'5694)", 2024-03-20T09:27:59.556 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:46.235380+0000", 2024-03-20T09:27:59.556 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 313.21280668200001, 2024-03-20T09:27:59.556 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.088268737999999999, 2024-03-20T09:27:59.556 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:27:59.557 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:27:59.557 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:27:59.557 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.557 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:27:59.557 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235380+0000", 2024-03-20T09:27:59.557 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.557 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.557 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.557 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:27:59.557 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235380+0000", 2024-03-20T09:27:59.557 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.557 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.558 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.558 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:27:59.558 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235381+0000", 2024-03-20T09:27:59.558 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6999999999999999e-07 2024-03-20T09:27:59.558 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.558 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.558 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:27:59.558 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235387+0000", 2024-03-20T09:27:59.558 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1650000000000003e-06 2024-03-20T09:27:59.558 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.558 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.558 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:27:59.558 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235392+0000", 2024-03-20T09:27:59.558 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8420000000000004e-06 2024-03-20T09:27:59.559 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.559 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.559 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:27:59.559 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235396+0000", 2024-03-20T09:27:59.559 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.8929999999999998e-06 2024-03-20T09:27:59.559 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.559 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.559 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:27:59.559 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.260711+0000", 2024-03-20T09:27:59.559 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.025315500000000001 2024-03-20T09:27:59.559 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.559 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.559 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:27:59.560 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.260729+0000", 2024-03-20T09:27:59.560 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7963000000000001e-05 2024-03-20T09:27:59.560 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.560 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.560 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:27:59.560 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.323627+0000", 2024-03-20T09:27:59.560 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.062897733999999997 2024-03-20T09:27:59.560 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.560 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.560 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:27:59.560 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.323649+0000", 2024-03-20T09:27:59.560 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2470999999999999e-05 2024-03-20T09:27:59.560 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.560 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:27:59.561 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.561 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.561 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.561 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:223309 5.5 e682/637 5:a4c63307:::benchmark_data_smithi122_157894_object93322:head v 682'14011, mlcod=627'5694)", 2024-03-20T09:27:59.561 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:46.248926+0000", 2024-03-20T09:27:59.561 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 313.19926098899998, 2024-03-20T09:27:59.561 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.094989770000000001, 2024-03-20T09:27:59.561 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:27:59.561 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:27:59.561 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:27:59.561 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.561 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:27:59.561 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.248926+0000", 2024-03-20T09:27:59.561 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.562 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.562 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.562 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:27:59.562 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.248926+0000", 2024-03-20T09:27:59.562 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.562 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.562 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.562 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:27:59.562 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.248927+0000", 2024-03-20T09:27:59.562 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0440000000000001e-06 2024-03-20T09:27:59.562 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.562 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.562 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:27:59.562 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.248938+0000", 2024-03-20T09:27:59.563 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0560000000000001e-05 2024-03-20T09:27:59.563 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.563 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.563 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:27:59.563 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.248952+0000", 2024-03-20T09:27:59.563 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4297000000000001e-05 2024-03-20T09:27:59.563 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.563 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.563 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:27:59.563 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.248964+0000", 2024-03-20T09:27:59.563 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1912e-05 2024-03-20T09:27:59.563 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.563 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.563 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:27:59.564 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.281776+0000", 2024-03-20T09:27:59.564 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.032812458000000003 2024-03-20T09:27:59.564 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.564 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.564 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:27:59.564 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.281805+0000", 2024-03-20T09:27:59.564 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8765e-05 2024-03-20T09:27:59.564 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.564 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.564 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:27:59.564 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.343870+0000", 2024-03-20T09:27:59.564 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.062064805000000001 2024-03-20T09:27:59.564 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.564 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.565 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:27:59.565 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.343916+0000", 2024-03-20T09:27:59.565 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5929e-05 2024-03-20T09:27:59.565 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.565 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:27:59.565 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.565 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.565 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.565 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:223346 5.5 e682/637 5:a014a865:::benchmark_data_smithi122_157894_object93359:head v 682'14012, mlcod=627'5694)", 2024-03-20T09:27:59.565 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:46.254399+0000", 2024-03-20T09:27:59.565 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 313.19378774299997, 2024-03-20T09:27:59.565 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.089556495, 2024-03-20T09:27:59.565 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:27:59.566 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:27:59.566 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:27:59.566 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.566 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:27:59.566 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.254399+0000", 2024-03-20T09:27:59.566 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.566 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.566 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.566 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:27:59.566 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.254399+0000", 2024-03-20T09:27:59.566 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.566 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.566 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.567 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:27:59.567 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.254400+0000", 2024-03-20T09:27:59.567 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9599999999999999e-07 2024-03-20T09:27:59.567 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.567 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.567 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:27:59.567 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.254405+0000", 2024-03-20T09:27:59.567 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.011e-06 2024-03-20T09:27:59.567 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.567 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.567 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:27:59.567 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.254415+0000", 2024-03-20T09:27:59.567 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.9790000000000005e-06 2024-03-20T09:27:59.568 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.568 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.568 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:27:59.568 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.254420+0000", 2024-03-20T09:27:59.568 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2340000000000002e-06 2024-03-20T09:27:59.568 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.568 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.568 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:27:59.568 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.292675+0000", 2024-03-20T09:27:59.568 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.038254945999999998 2024-03-20T09:27:59.568 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.568 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.568 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:27:59.568 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.292700+0000", 2024-03-20T09:27:59.569 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5173000000000001e-05 2024-03-20T09:27:59.569 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.569 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.569 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:27:59.569 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.343920+0000", 2024-03-20T09:27:59.569 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.051219555 2024-03-20T09:27:59.569 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.569 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.569 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:27:59.569 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.343956+0000", 2024-03-20T09:27:59.569 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6001000000000003e-05 2024-03-20T09:27:59.569 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.569 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:27:59.569 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.570 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.570 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.570 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "MOSDPGPush(5.f 685/637 [PushOp(5:f1254502:::benchmark_data_smithi122_157894_object94700:head, version: 628'5882, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:f1254502:::benchmark_data_smithi122_157894_object94700:head@628'5882, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-20T09:27:59.570 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:47.826130+0000", 2024-03-20T09:27:59.570 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 311.62205756399999, 2024-03-20T09:27:59.570 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.088800740000000003, 2024-03-20T09:27:59.570 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:27:59.570 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:27:59.570 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:27:59.570 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.570 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:27:59.570 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.826130+0000", 2024-03-20T09:27:59.570 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.571 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.571 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.571 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:27:59.571 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.826130+0000", 2024-03-20T09:27:59.571 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.571 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.571 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.571 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:27:59.571 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.826130+0000", 2024-03-20T09:27:59.571 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.0800000000000001e-07 2024-03-20T09:27:59.571 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.571 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.571 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:27:59.571 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.826161+0000", 2024-03-20T09:27:59.572 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.1158999999999998e-05 2024-03-20T09:27:59.572 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.572 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.572 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:27:59.572 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.826187+0000", 2024-03-20T09:27:59.572 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5978000000000001e-05 2024-03-20T09:27:59.572 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.572 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.572 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:27:59.572 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.826204+0000", 2024-03-20T09:27:59.572 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6501e-05 2024-03-20T09:27:59.572 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.572 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.572 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:27:59.572 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.910972+0000", 2024-03-20T09:27:59.573 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.084768204999999999 2024-03-20T09:27:59.573 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.573 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.573 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:27:59.573 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.910987+0000", 2024-03-20T09:27:59.573 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5277999999999998e-05 2024-03-20T09:27:59.573 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.573 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.573 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:27:59.573 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.914930+0000", 2024-03-20T09:27:59.573 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0039433109999999997 2024-03-20T09:27:59.573 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.573 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:27:59.574 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.574 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.574 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.574 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:228015 5.5 e687/637 5:ad8250e8:::benchmark_data_smithi122_157894_object98028:head v 687'14312, mlcod=628'6123)", 2024-03-20T09:27:59.574 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:54.357401+0000", 2024-03-20T09:27:59.574 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 305.09078572499999, 2024-03-20T09:27:59.574 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.087389438, 2024-03-20T09:27:59.574 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:27:59.574 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:27:59.574 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:27:59.574 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.574 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:27:59.574 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:54.357401+0000", 2024-03-20T09:27:59.575 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.575 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.575 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.575 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:27:59.575 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:54.357401+0000", 2024-03-20T09:27:59.575 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.575 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.575 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.575 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:27:59.575 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:54.357402+0000", 2024-03-20T09:27:59.575 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.27e-07 2024-03-20T09:27:59.575 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.575 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.575 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:27:59.575 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:54.357406+0000", 2024-03-20T09:27:59.576 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6489999999999999e-06 2024-03-20T09:27:59.576 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.576 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.576 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:27:59.576 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:54.357412+0000", 2024-03-20T09:27:59.576 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0059999999999998e-06 2024-03-20T09:27:59.576 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.576 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.576 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:27:59.576 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:54.357416+0000", 2024-03-20T09:27:59.576 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.07e-06 2024-03-20T09:27:59.576 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.577 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.577 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:27:59.577 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:54.367499+0000", 2024-03-20T09:27:59.577 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.010082259999999999 2024-03-20T09:27:59.577 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.577 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.577 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:27:59.577 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:54.367528+0000", 2024-03-20T09:27:59.577 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9501000000000001e-05 2024-03-20T09:27:59.577 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.577 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.577 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:27:59.577 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:54.444737+0000", 2024-03-20T09:27:59.577 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.077208767999999997 2024-03-20T09:27:59.577 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.578 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.578 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:27:59.578 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:54.444791+0000", 2024-03-20T09:27:59.578 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3857000000000003e-05 2024-03-20T09:27:59.578 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.578 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:27:59.578 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.578 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.578 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.578 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:257605 5.5 e699/637 5:ade6c67d:::benchmark_data_smithi122_157894_object127618:head v 699'16216, mlcod=635'8024)", 2024-03-20T09:27:59.578 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:23:37.959926+0000", 2024-03-20T09:27:59.578 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 261.48826139900001, 2024-03-20T09:27:59.578 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.099523950999999999, 2024-03-20T09:27:59.578 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:27:59.578 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:27:59.579 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:27:59.579 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.579 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:27:59.579 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:37.959926+0000", 2024-03-20T09:27:59.579 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.579 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.579 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.579 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:27:59.579 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:37.959926+0000", 2024-03-20T09:27:59.579 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.579 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.579 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.579 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:27:59.579 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:37.959926+0000", 2024-03-20T09:27:59.580 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2700000000000001e-07 2024-03-20T09:27:59.580 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.580 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.580 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:27:59.580 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:37.959929+0000", 2024-03-20T09:27:59.580 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9409999999999999e-06 2024-03-20T09:27:59.580 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.580 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.580 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:27:59.580 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:37.959932+0000", 2024-03-20T09:27:59.580 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.3589999999999999e-06 2024-03-20T09:27:59.580 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.580 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.580 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:27:59.580 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:37.959935+0000", 2024-03-20T09:27:59.581 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5390000000000002e-06 2024-03-20T09:27:59.581 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.581 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.581 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:27:59.581 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:37.979588+0000", 2024-03-20T09:27:59.581 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.019652738999999999 2024-03-20T09:27:59.581 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.581 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.581 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:27:59.581 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:37.979610+0000", 2024-03-20T09:27:59.581 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2467999999999999e-05 2024-03-20T09:27:59.581 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.581 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.581 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:27:59.581 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:38.059401+0000", 2024-03-20T09:27:59.582 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.079791387000000005 2024-03-20T09:27:59.582 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.582 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.582 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:27:59.582 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:38.059450+0000", 2024-03-20T09:27:59.582 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8291000000000002e-05 2024-03-20T09:27:59.582 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.582 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:27:59.582 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.582 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.582 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.582 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:257606 5.5 e699/637 5:a912954b:::benchmark_data_smithi122_157894_object127619:head v 699'16217, mlcod=635'8024)", 2024-03-20T09:27:59.582 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:23:37.967251+0000", 2024-03-20T09:27:59.583 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 261.48093578499999, 2024-03-20T09:27:59.583 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.092234901999999994, 2024-03-20T09:27:59.583 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:27:59.583 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:27:59.583 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:27:59.583 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.583 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:27:59.583 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:37.967251+0000", 2024-03-20T09:27:59.583 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.583 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.583 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.583 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:27:59.583 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:37.967251+0000", 2024-03-20T09:27:59.583 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.584 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.584 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.584 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:27:59.584 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:37.967252+0000", 2024-03-20T09:27:59.584 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.5499999999999996e-07 2024-03-20T09:27:59.584 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.584 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.584 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:27:59.584 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:37.967261+0000", 2024-03-20T09:27:59.584 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.6300000000000004e-06 2024-03-20T09:27:59.584 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.584 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.584 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:27:59.584 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:37.967277+0000", 2024-03-20T09:27:59.584 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5968999999999999e-05 2024-03-20T09:27:59.585 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.585 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.585 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:27:59.585 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:37.967285+0000", 2024-03-20T09:27:59.585 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.3650000000000004e-06 2024-03-20T09:27:59.585 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.585 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.585 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:27:59.585 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:37.989840+0000", 2024-03-20T09:27:59.585 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.022555009000000001 2024-03-20T09:27:59.585 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.585 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.585 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:27:59.585 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:37.989868+0000", 2024-03-20T09:27:59.586 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8106000000000002e-05 2024-03-20T09:27:59.586 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.586 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.586 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:27:59.586 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:38.059454+0000", 2024-03-20T09:27:59.586 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.069586061000000005 2024-03-20T09:27:59.586 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.586 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.586 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:27:59.586 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:38.059486+0000", 2024-03-20T09:27:59.586 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.1807000000000002e-05 2024-03-20T09:27:59.586 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.586 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:27:59.586 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.587 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.587 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.587 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:92151 6.e 6:72a8fe95:::benchmark_data_smithi122_168655_object92150:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e735)", 2024-03-20T09:27:59.587 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:19.137168+0000", 2024-03-20T09:27:59.587 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 160.311019114, 2024-03-20T09:27:59.587 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.092309425000000001, 2024-03-20T09:27:59.587 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:27:59.587 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:27:59.587 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:27:59.587 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:27:59.587 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:27:59.587 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 92151 2024-03-20T09:27:59.587 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.587 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:27:59.588 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.588 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:27:59.588 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.137168+0000", 2024-03-20T09:27:59.588 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.588 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.588 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.588 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:27:59.588 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.137168+0000", 2024-03-20T09:27:59.588 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.588 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.588 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.588 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:27:59.588 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.137169+0000", 2024-03-20T09:27:59.588 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.116e-06 2024-03-20T09:27:59.589 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.589 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.589 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:27:59.589 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.137244+0000", 2024-03-20T09:27:59.589 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.4370999999999997e-05 2024-03-20T09:27:59.589 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.589 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.589 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:27:59.589 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.137263+0000", 2024-03-20T09:27:59.589 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9357000000000001e-05 2024-03-20T09:27:59.589 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.589 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.589 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:27:59.590 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.137274+0000", 2024-03-20T09:27:59.590 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0954e-05 2024-03-20T09:27:59.590 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.590 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.590 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:27:59.590 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.137322+0000", 2024-03-20T09:27:59.590 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8547999999999997e-05 2024-03-20T09:27:59.590 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.591 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.591 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:27:59.591 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.137477+0000", 2024-03-20T09:27:59.591 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00015410000000000001 2024-03-20T09:27:59.591 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.591 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.591 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,2", 2024-03-20T09:27:59.591 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.137648+0000", 2024-03-20T09:27:59.591 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00017198699999999999 2024-03-20T09:27:59.591 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.591 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.591 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:27:59.591 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.144467+0000", 2024-03-20T09:27:59.591 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0068180769999999996 2024-03-20T09:27:59.592 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.592 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.592 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:27:59.592 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.149111+0000", 2024-03-20T09:27:59.592 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0046444149999999998 2024-03-20T09:27:59.592 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.592 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.592 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:27:59.592 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.229349+0000", 2024-03-20T09:27:59.592 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.080238212000000003 2024-03-20T09:27:59.592 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.592 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.592 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:27:59.592 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.229416+0000", 2024-03-20T09:27:59.593 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.7138000000000007e-05 2024-03-20T09:27:59.593 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.593 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.593 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:27:59.593 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.229477+0000", 2024-03-20T09:27:59.593 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1149999999999996e-05 2024-03-20T09:27:59.593 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.593 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:27:59.593 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.593 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.593 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.593 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:92152 6.e 6:75dfc8b0:::benchmark_data_smithi122_168655_object92151:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e735)", 2024-03-20T09:27:59.593 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:19.137291+0000", 2024-03-20T09:27:59.593 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 160.31089598700001, 2024-03-20T09:27:59.594 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.092306768999999997, 2024-03-20T09:27:59.594 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:27:59.594 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:27:59.594 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:27:59.594 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:27:59.594 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:27:59.594 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 92152 2024-03-20T09:27:59.594 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.594 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:27:59.594 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.594 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:27:59.594 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.137291+0000", 2024-03-20T09:27:59.595 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.595 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.595 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.595 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:27:59.595 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.137291+0000", 2024-03-20T09:27:59.595 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.595 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.595 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.595 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:27:59.595 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.137292+0000", 2024-03-20T09:27:59.595 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.9999999999999998e-07 2024-03-20T09:27:59.595 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.595 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.595 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:27:59.596 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.137314+0000", 2024-03-20T09:27:59.596 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2867999999999999e-05 2024-03-20T09:27:59.596 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.596 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.596 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:27:59.596 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.137323+0000", 2024-03-20T09:27:59.596 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.2430000000000005e-06 2024-03-20T09:27:59.596 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.596 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.596 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:27:59.596 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.137327+0000", 2024-03-20T09:27:59.597 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.941e-06 2024-03-20T09:27:59.597 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.597 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.597 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:27:59.597 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.139923+0000", 2024-03-20T09:27:59.597 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0025965839999999999 2024-03-20T09:27:59.597 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.597 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.597 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:27:59.597 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.140055+0000", 2024-03-20T09:27:59.597 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013172100000000001 2024-03-20T09:27:59.597 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.597 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.597 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,2", 2024-03-20T09:27:59.598 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.140172+0000", 2024-03-20T09:27:59.598 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000116864 2024-03-20T09:27:59.598 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.598 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.598 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:27:59.598 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.146208+0000", 2024-03-20T09:27:59.598 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0060361920000000001 2024-03-20T09:27:59.598 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.598 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.598 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:27:59.598 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.149160+0000", 2024-03-20T09:27:59.598 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.002952091 2024-03-20T09:27:59.598 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.599 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.599 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:27:59.599 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.229523+0000", 2024-03-20T09:27:59.599 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.080362798999999999 2024-03-20T09:27:59.599 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.599 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.599 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:27:59.599 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.229560+0000", 2024-03-20T09:27:59.599 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7527999999999997e-05 2024-03-20T09:27:59.599 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.599 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.599 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:27:59.599 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.229598+0000", 2024-03-20T09:27:59.599 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7537999999999999e-05 2024-03-20T09:27:59.600 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.600 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:27:59.600 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.600 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.600 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.600 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:129144 6.2 6:47957f3d:::benchmark_data_smithi122_168655_object9035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e748)", 2024-03-20T09:27:59.600 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:56.335243+0000", 2024-03-20T09:27:59.600 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 123.112943699, 2024-03-20T09:27:59.600 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.223256595, 2024-03-20T09:27:59.600 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:27:59.600 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:27:59.600 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:27:59.601 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:27:59.601 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:27:59.601 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 129144 2024-03-20T09:27:59.601 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.601 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:27:59.601 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.601 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:27:59.601 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335243+0000", 2024-03-20T09:27:59.602 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.602 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.602 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.602 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:27:59.602 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335243+0000", 2024-03-20T09:27:59.602 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.602 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.602 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.602 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:27:59.602 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335244+0000", 2024-03-20T09:27:59.602 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0299999999999999e-07 2024-03-20T09:27:59.602 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.602 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.603 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:27:59.603 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335246+0000", 2024-03-20T09:27:59.603 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.4959999999999999e-06 2024-03-20T09:27:59.603 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.603 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.603 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:27:59.603 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335253+0000", 2024-03-20T09:27:59.603 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.6699999999999997e-06 2024-03-20T09:27:59.603 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.603 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.603 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:27:59.603 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335258+0000", 2024-03-20T09:27:59.603 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.7500000000000003e-06 2024-03-20T09:27:59.603 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.603 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.604 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:27:59.604 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335321+0000", 2024-03-20T09:27:59.604 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2765e-05 2024-03-20T09:27:59.604 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.604 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.604 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:27:59.604 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335440+0000", 2024-03-20T09:27:59.604 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000119141 2024-03-20T09:27:59.604 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.604 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.604 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,3", 2024-03-20T09:27:59.604 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335511+0000", 2024-03-20T09:27:59.604 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.1104000000000002e-05 2024-03-20T09:27:59.604 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.604 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.605 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:27:59.605 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.342379+0000", 2024-03-20T09:27:59.605 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0068685320000000001 2024-03-20T09:27:59.605 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.605 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.605 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:27:59.605 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.557233+0000", 2024-03-20T09:27:59.605 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.21485372999999999 2024-03-20T09:27:59.605 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.605 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.605 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:27:59.605 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.558500+0000", 2024-03-20T09:27:59.605 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0012669039999999999 2024-03-20T09:27:59.605 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.605 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:27:59.606 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.606 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.606 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.606 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:129123 6.2 6:450098af:::benchmark_data_smithi122_168655_object9014:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e749)", 2024-03-20T09:27:59.606 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:56.335273+0000", 2024-03-20T09:27:59.606 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 123.112913875, 2024-03-20T09:27:59.606 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0772038850000001, 2024-03-20T09:27:59.606 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:27:59.606 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:27:59.606 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:27:59.606 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:27:59.606 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:27:59.606 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 129123 2024-03-20T09:27:59.606 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.607 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:27:59.607 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.607 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:27:59.607 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335273+0000", 2024-03-20T09:27:59.607 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.607 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.607 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.607 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:27:59.607 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335273+0000", 2024-03-20T09:27:59.607 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.607 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.607 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.607 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:27:59.607 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335274+0000", 2024-03-20T09:27:59.607 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6300000000000001e-07 2024-03-20T09:27:59.608 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.608 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.608 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:27:59.608 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335276+0000", 2024-03-20T09:27:59.608 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2089999999999999e-06 2024-03-20T09:27:59.608 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.608 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.608 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:27:59.608 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335282+0000", 2024-03-20T09:27:59.608 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0800000000000002e-06 2024-03-20T09:27:59.608 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.608 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.608 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:27:59.608 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335287+0000", 2024-03-20T09:27:59.609 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4700000000000001e-06 2024-03-20T09:27:59.609 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.609 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.609 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:27:59.609 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.339142+0000", 2024-03-20T09:27:59.609 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0038545229999999999 2024-03-20T09:27:59.609 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.609 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.609 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op must wait for map", 2024-03-20T09:27:59.609 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.339150+0000", 2024-03-20T09:27:59.609 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.6279999999999996e-06 2024-03-20T09:27:59.609 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.609 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.609 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:27:59.609 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.557266+0000", 2024-03-20T09:27:59.610 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.21811618799999999 2024-03-20T09:27:59.610 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.610 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.610 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:27:59.610 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.557275+0000", 2024-03-20T09:27:59.610 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.8810000000000002e-06 2024-03-20T09:27:59.610 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.610 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.610 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:27:59.610 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:57.412381+0000", 2024-03-20T09:27:59.610 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.85510642599999998 2024-03-20T09:27:59.610 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.610 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.610 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:27:59.611 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:57.412477+0000", 2024-03-20T09:27:59.611 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.6117000000000001e-05 2024-03-20T09:27:59.611 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.611 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:27:59.611 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.611 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.611 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.611 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:129125 6.12 6:4e4918e7:::benchmark_data_smithi122_168655_object9016:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e749)", 2024-03-20T09:27:59.611 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:56.335325+0000", 2024-03-20T09:27:59.611 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 123.112861764, 2024-03-20T09:27:59.611 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.13051883, 2024-03-20T09:27:59.611 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:27:59.611 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:27:59.611 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:27:59.611 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:27:59.612 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:27:59.612 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 129125 2024-03-20T09:27:59.612 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.612 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:27:59.612 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.612 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:27:59.612 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335325+0000", 2024-03-20T09:27:59.612 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.612 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.612 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.612 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:27:59.612 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335325+0000", 2024-03-20T09:27:59.612 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.612 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.613 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.613 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:27:59.613 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335326+0000", 2024-03-20T09:27:59.613 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.4300000000000002e-07 2024-03-20T09:27:59.613 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.613 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.613 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:27:59.613 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335331+0000", 2024-03-20T09:27:59.613 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.566e-06 2024-03-20T09:27:59.613 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.613 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.613 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:27:59.613 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335339+0000", 2024-03-20T09:27:59.613 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.4430000000000003e-06 2024-03-20T09:27:59.613 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.614 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.614 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:27:59.614 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335346+0000", 2024-03-20T09:27:59.614 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.2339999999999997e-06 2024-03-20T09:27:59.614 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.614 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.614 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:27:59.614 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.558708+0000", 2024-03-20T09:27:59.614 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.223361277 2024-03-20T09:27:59.614 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.614 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.614 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:27:59.615 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.558717+0000", 2024-03-20T09:27:59.615 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.7350000000000008e-06 2024-03-20T09:27:59.615 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.615 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.615 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:27:59.615 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:57.465759+0000", 2024-03-20T09:27:59.615 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.90704171499999997 2024-03-20T09:27:59.615 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.615 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.615 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:27:59.615 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:57.465844+0000", 2024-03-20T09:27:59.615 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5117000000000005e-05 2024-03-20T09:27:59.615 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.615 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:27:59.616 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.616 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.616 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.616 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:129129 6.2 6:471d5bbb:::benchmark_data_smithi122_168655_object9020:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e749)", 2024-03-20T09:27:59.616 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:56.335359+0000", 2024-03-20T09:27:59.616 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 123.112828631, 2024-03-20T09:27:59.616 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.077221416, 2024-03-20T09:27:59.616 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:27:59.616 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:27:59.616 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:27:59.616 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:27:59.616 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:27:59.616 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 129129 2024-03-20T09:27:59.616 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.617 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:27:59.617 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.617 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:27:59.617 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335359+0000", 2024-03-20T09:27:59.617 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.617 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.617 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.617 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:27:59.617 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335359+0000", 2024-03-20T09:27:59.617 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.617 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.617 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.617 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:27:59.617 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335359+0000", 2024-03-20T09:27:59.617 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.2500000000000001e-07 2024-03-20T09:27:59.618 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.618 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.618 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:27:59.618 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335361+0000", 2024-03-20T09:27:59.618 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3240000000000001e-06 2024-03-20T09:27:59.618 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.618 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.618 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:27:59.618 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335368+0000", 2024-03-20T09:27:59.618 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.3790000000000003e-06 2024-03-20T09:27:59.618 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.618 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.618 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:27:59.618 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335372+0000", 2024-03-20T09:27:59.619 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.724e-06 2024-03-20T09:27:59.619 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.619 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.619 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:27:59.619 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.339225+0000", 2024-03-20T09:27:59.619 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0038523030000000001 2024-03-20T09:27:59.619 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.619 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.619 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting_for_map not empty", 2024-03-20T09:27:59.619 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.339230+0000", 2024-03-20T09:27:59.619 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9540000000000003e-06 2024-03-20T09:27:59.619 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.619 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.619 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:27:59.619 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.557298+0000", 2024-03-20T09:27:59.620 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.21806794900000001 2024-03-20T09:27:59.620 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.620 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.620 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:27:59.620 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.557305+0000", 2024-03-20T09:27:59.620 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3520000000000001e-06 2024-03-20T09:27:59.620 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.620 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.620 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:27:59.620 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:57.412500+0000", 2024-03-20T09:27:59.620 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.85519480299999995 2024-03-20T09:27:59.620 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.620 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.620 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:27:59.621 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:57.412580+0000", 2024-03-20T09:27:59.621 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.0203000000000004e-05 2024-03-20T09:27:59.621 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.621 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:27:59.621 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.621 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.621 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.621 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:129144 6.2 6:47957f3d:::benchmark_data_smithi122_168655_object9035:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e749)", 2024-03-20T09:27:59.621 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:56.335379+0000", 2024-03-20T09:27:59.621 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 123.112808078, 2024-03-20T09:27:59.621 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1031262369999999, 2024-03-20T09:27:59.621 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:27:59.621 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:27:59.621 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:27:59.621 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:27:59.622 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:27:59.622 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 129144 2024-03-20T09:27:59.622 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.622 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:27:59.622 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.622 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:27:59.622 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335379+0000", 2024-03-20T09:27:59.622 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.622 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.622 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.622 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:27:59.622 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335379+0000", 2024-03-20T09:27:59.623 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:27:59.623 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.623 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.623 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:27:59.623 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335379+0000", 2024-03-20T09:27:59.623 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.5900000000000003e-07 2024-03-20T09:27:59.623 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.623 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.623 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:27:59.623 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335382+0000", 2024-03-20T09:27:59.623 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.181e-06 2024-03-20T09:27:59.623 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.623 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.623 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:27:59.624 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335387+0000", 2024-03-20T09:27:59.624 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9799999999999998e-06 2024-03-20T09:27:59.624 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.624 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.624 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:27:59.624 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335391+0000", 2024-03-20T09:27:59.624 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5519999999999998e-06 2024-03-20T09:27:59.624 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.624 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.624 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:27:59.624 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.339468+0000", 2024-03-20T09:27:59.624 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0040772990000000004 2024-03-20T09:27:59.624 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.625 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.625 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting_for_map not empty", 2024-03-20T09:27:59.625 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.339473+0000", 2024-03-20T09:27:59.625 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5800000000000002e-06 2024-03-20T09:27:59.625 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.625 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.625 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:27:59.625 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.557326+0000", 2024-03-20T09:27:59.625 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.21785318400000001 2024-03-20T09:27:59.625 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.625 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.625 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:27:59.625 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.557333+0000", 2024-03-20T09:27:59.625 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.7719999999999997e-06 2024-03-20T09:27:59.626 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.626 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.626 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:27:59.626 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:57.412602+0000", 2024-03-20T09:27:59.626 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.85526948700000005 2024-03-20T09:27:59.626 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.626 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.626 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:27:59.626 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:57.412726+0000", 2024-03-20T09:27:59.626 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012335300000000001 2024-03-20T09:27:59.626 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.626 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.626 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:27:59.626 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:57.438420+0000", 2024-03-20T09:27:59.627 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.025694552999999998 2024-03-20T09:27:59.627 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:27:59.627 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:27:59.627 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:27:59.627 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:57.438505+0000", 2024-03-20T09:27:59.627 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.4937000000000003e-05 2024-03-20T09:27:59.627 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.627 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:27:59.627 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.627 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:27:59.627 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:27:59.627 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:27:59.630 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:27:59.673 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:27:59.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:27:59 smithi122 ceph-mon[73495]: pgmap v1939: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 4.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 381 op/s; 17224/236745 objects degraded (7.275%); 10490/236745 objects misplaced (4.431%); 4.3 MiB/s, 68 objects/s recovering 2024-03-20T09:27:59.705 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:27:59.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:27:59.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:27:59.813 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:27:59.854 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:00.252 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-20T09:28:00.465 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:00 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:28:00.466 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:00 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:28:00.466 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:00 smithi026 ceph-mon[68309]: osdmap e799: 12 total, 11 up, 12 in 2024-03-20T09:28:00.466 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:00 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:28:00.466 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:00 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:28:00.466 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:00 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:28:00.466 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:00 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:28:00.466 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:00 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:28:00.466 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:00 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:28:00.466 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:00 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:28:00.514 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:28:00.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:00.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:28:00.569 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:28:00.618 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:28:00.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:00 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:28:00.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:00 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:28:00.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:00 smithi115 ceph-mon[72476]: osdmap e799: 12 total, 11 up, 12 in 2024-03-20T09:28:00.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:00 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:28:00.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:00 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:28:00.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:00 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:28:00.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:00 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:28:00.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:00 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:28:00.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:00 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:28:00.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:00 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:28:00.657 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:00.690 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:28:00.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:00 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:28:00.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:00 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:28:00.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:00 smithi122 ceph-mon[73495]: osdmap e799: 12 total, 11 up, 12 in 2024-03-20T09:28:00.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:00 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:28:00.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:00 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:28:00.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:00 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:28:00.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:00 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:28:00.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:00 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:28:00.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:00 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:28:00.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:00 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:28:00.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:00.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:28:00.794 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:28:00.847 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:01.306 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:28:01.306 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:28:01.306 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:28:01.306 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:28:01.306 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:28:01.306 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:28:01.306 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:28:01.306 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:28:01.306 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:28:01.307 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:28:01.307 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:28:01.307 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:28:01.307 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:28:01.307 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:28:01.307 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:28:01.307 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:28:01.307 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:28:01.307 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:28:01.307 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:28:01.307 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:28:01.307 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:28:01.308 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-03-20T09:28:01.308 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:28:01.318 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:28:01.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:01.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:28:01.423 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:28:01.454 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:01 smithi026 ceph-mon[68309]: pgmap v1941: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 502 op/s; 16996/233412 objects degraded (7.282%); 10361/233412 objects misplaced (4.439%); 4.4 MiB/s, 69 objects/s recovering 2024-03-20T09:28:01.454 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:01 smithi026 ceph-mon[68309]: osdmap e800: 12 total, 11 up, 12 in 2024-03-20T09:28:01.454 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:01 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1289303042' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:28:01.454 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:01 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/49705782' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.8", "id": [8, 11, 3]}]: dispatch 2024-03-20T09:28:01.454 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:01 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.8", "id": [8, 11, 3]}]: dispatch 2024-03-20T09:28:01.457 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:01.557 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:01 smithi115 ceph-mon[72476]: pgmap v1941: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 502 op/s; 16996/233412 objects degraded (7.282%); 10361/233412 objects misplaced (4.439%); 4.4 MiB/s, 69 objects/s recovering 2024-03-20T09:28:01.558 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:01 smithi115 ceph-mon[72476]: osdmap e800: 12 total, 11 up, 12 in 2024-03-20T09:28:01.558 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:01 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1289303042' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:28:01.558 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:01 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/49705782' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.8", "id": [8, 11, 3]}]: dispatch 2024-03-20T09:28:01.558 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:01 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.8", "id": [8, 11, 3]}]: dispatch 2024-03-20T09:28:01.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:01 smithi122 ceph-mon[73495]: pgmap v1941: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 502 op/s; 16996/233412 objects degraded (7.282%); 10361/233412 objects misplaced (4.439%); 4.4 MiB/s, 69 objects/s recovering 2024-03-20T09:28:01.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:01 smithi122 ceph-mon[73495]: osdmap e800: 12 total, 11 up, 12 in 2024-03-20T09:28:01.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:01 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1289303042' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:28:01.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:01 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/49705782' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.8", "id": [8, 11, 3]}]: dispatch 2024-03-20T09:28:01.692 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:01 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.8", "id": [8, 11, 3]}]: dispatch 2024-03-20T09:28:01.843 INFO:teuthology.orchestra.run.smithi026.stderr:set 6.8 pg_upmap mapping to [8,11,3] 2024-03-20T09:28:02.591 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:28:02.591 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:28:02.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:02 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.8", "id": [8, 11, 3]}]': finished 2024-03-20T09:28:02.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:02 smithi115 ceph-mon[72476]: osdmap e801: 12 total, 11 up, 12 in 2024-03-20T09:28:02.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:02 smithi115 ceph-mon[72476]: pgmap v1944: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 469 op/s; 16729/229854 objects degraded (7.278%); 9675/229854 objects misplaced (4.209%); 7.7 MiB/s, 123 objects/s recovering 2024-03-20T09:28:02.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:02 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 16996/233412 objects degraded (7.282%), 5 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:28:02.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:02 smithi115 ceph-mon[72476]: osdmap e802: 12 total, 11 up, 12 in 2024-03-20T09:28:03.188 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:02 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.8", "id": [8, 11, 3]}]': finished 2024-03-20T09:28:03.188 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:02 smithi026 ceph-mon[68309]: osdmap e801: 12 total, 11 up, 12 in 2024-03-20T09:28:03.189 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:02 smithi026 ceph-mon[68309]: pgmap v1944: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 469 op/s; 16729/229854 objects degraded (7.278%); 9675/229854 objects misplaced (4.209%); 7.7 MiB/s, 123 objects/s recovering 2024-03-20T09:28:03.189 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:02 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 16996/233412 objects degraded (7.282%), 5 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:28:03.189 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:02 smithi026 ceph-mon[68309]: osdmap e802: 12 total, 11 up, 12 in 2024-03-20T09:28:03.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:02 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.8", "id": [8, 11, 3]}]': finished 2024-03-20T09:28:03.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:02 smithi122 ceph-mon[73495]: osdmap e801: 12 total, 11 up, 12 in 2024-03-20T09:28:03.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:02 smithi122 ceph-mon[73495]: pgmap v1944: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 469 op/s; 16729/229854 objects degraded (7.278%); 9675/229854 objects misplaced (4.209%); 7.7 MiB/s, 123 objects/s recovering 2024-03-20T09:28:03.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:02 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 16996/233412 objects degraded (7.282%), 5 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:28:03.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:02 smithi122 ceph-mon[73495]: osdmap e802: 12 total, 11 up, 12 in 2024-03-20T09:28:03.316 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:28:03.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:03.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:28:03.423 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:28:03.425 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-20T09:28:03.454 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:03.477 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:28:03.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:03.481 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:28:03.582 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:28:03.609 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:03.630 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:28:03.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:03.634 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:28:03.726 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:28:03.735 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:28:03.764 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:03.785 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:28:03.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:03.790 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:28:03.891 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:28:03.922 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:03.967 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:28:03.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:03.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:28:04.075 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:28:04.129 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:03 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:28:04.129 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:03 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:28:04.130 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:03 smithi026 ceph-mon[68309]: osdmap e803: 12 total, 11 up, 12 in 2024-03-20T09:28:04.135 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:04.150 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:03 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:28:04.151 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:03 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:28:04.151 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:03 smithi115 ceph-mon[72476]: osdmap e803: 12 total, 11 up, 12 in 2024-03-20T09:28:04.188 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:28:04.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:03 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:28:04.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:03 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:28:04.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:03 smithi122 ceph-mon[73495]: osdmap e803: 12 total, 11 up, 12 in 2024-03-20T09:28:04.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:04.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:28:04.241 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:28:04.242 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:28:04.298 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:28:04.337 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:04.556 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:28:04.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:04.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:28:04.660 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:28:04.691 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:04.858 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:04 smithi026 ceph-mon[68309]: pgmap v1947: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 296 op/s; 16729/229854 objects degraded (7.278%); 9675/229854 objects misplaced (4.209%); 5.3 MiB/s, 85 objects/s recovering 2024-03-20T09:28:05.151 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:04 smithi115 ceph-mon[72476]: pgmap v1947: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 296 op/s; 16729/229854 objects degraded (7.278%); 9675/229854 objects misplaced (4.209%); 5.3 MiB/s, 85 objects/s recovering 2024-03-20T09:28:05.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:04 smithi122 ceph-mon[73495]: pgmap v1947: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 296 op/s; 16729/229854 objects degraded (7.278%); 9675/229854 objects misplaced (4.209%); 5.3 MiB/s, 85 objects/s recovering 2024-03-20T09:28:05.776 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:28:05.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:05.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:28:05.880 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:28:05.933 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:06.243 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:28:06.292 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:28:06.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:06.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:28:06.401 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:28:06.446 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:06.556 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:28:06.557 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:28:07.210 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:28:07.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:07.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:28:07.249 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:28:07.312 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-20T09:28:07.320 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:28:07.387 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:07.411 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:28:07.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:07.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:28:07.517 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:28:07.549 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:07 smithi026 ceph-mon[68309]: pgmap v1948: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 43 active+clean; 4.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 451 op/s; 16574/225390 objects degraded (7.353%); 11723/225390 objects misplaced (5.201%); 5.6 MiB/s, 90 objects/s recovering 2024-03-20T09:28:07.550 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:07 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:28:07.550 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:07 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:28:07.552 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:07.632 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:28:07.653 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:07 smithi115 ceph-mon[72476]: pgmap v1948: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 43 active+clean; 4.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 451 op/s; 16574/225390 objects degraded (7.353%); 11723/225390 objects misplaced (5.201%); 5.6 MiB/s, 90 objects/s recovering 2024-03-20T09:28:07.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:07 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:28:07.654 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:07 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:28:07.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:07 smithi122 ceph-mon[73495]: pgmap v1948: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 1 active+recovering+undersized+remapped, 7 active+undersized, 1 active+recovering+degraded+remapped, 43 active+clean; 4.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 451 op/s; 16574/225390 objects degraded (7.353%); 11723/225390 objects misplaced (5.201%); 5.6 MiB/s, 90 objects/s recovering 2024-03-20T09:28:07.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:07 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:28:07.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:07 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:28:08.214 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:28:08.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:08.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:28:08.319 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:28:08.350 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:08.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:08 smithi115 ceph-mon[72476]: osdmap e804: 12 total, 11 up, 12 in 2024-03-20T09:28:08.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:08 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 16574/225390 objects degraded (7.353%), 6 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:28:08.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:08 smithi122 ceph-mon[73495]: osdmap e804: 12 total, 11 up, 12 in 2024-03-20T09:28:08.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:08 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 16574/225390 objects degraded (7.353%), 6 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:28:08.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:08 smithi026 ceph-mon[68309]: osdmap e804: 12 total, 11 up, 12 in 2024-03-20T09:28:08.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:08 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 16574/225390 objects degraded (7.353%), 6 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:28:09.103 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:28:09.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:09.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:28:09.207 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:28:09.234 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:09.253 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:28:09.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:09.256 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:28:09.357 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:28:09.391 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:09.400 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:09 smithi115 ceph-mon[72476]: pgmap v1950: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 7 active+undersized, 1 active+recovering+degraded+remapped, 43 active+clean; 4.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 305 op/s; 16344/224349 objects degraded (7.285%); 10988/224349 objects misplaced (4.898%); 5.7 MiB/s, 131 objects/s recovering 2024-03-20T09:28:09.401 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:09 smithi115 ceph-mon[72476]: osdmap e805: 12 total, 11 up, 12 in 2024-03-20T09:28:09.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:09 smithi122 ceph-mon[73495]: pgmap v1950: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 7 active+undersized, 1 active+recovering+degraded+remapped, 43 active+clean; 4.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 305 op/s; 16344/224349 objects degraded (7.285%); 10988/224349 objects misplaced (4.898%); 5.7 MiB/s, 131 objects/s recovering 2024-03-20T09:28:09.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:09 smithi122 ceph-mon[73495]: osdmap e805: 12 total, 11 up, 12 in 2024-03-20T09:28:09.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:09 smithi026 ceph-mon[68309]: pgmap v1950: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 7 active+undersized, 1 active+recovering+degraded+remapped, 43 active+clean; 4.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 305 op/s; 16344/224349 objects degraded (7.285%); 10988/224349 objects misplaced (4.898%); 5.7 MiB/s, 131 objects/s recovering 2024-03-20T09:28:09.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:09 smithi026 ceph-mon[68309]: osdmap e805: 12 total, 11 up, 12 in 2024-03-20T09:28:10.062 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 8, 10, 11, 3, 4, 9] out_osds: [] dead_osds: [10] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 11, 5] 2024-03-20T09:28:10.062 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:28:10.063 INFO:tasks.thrashosds.thrasher:Removing osd 10, in_osds are: [0, 1, 2, 5, 6, 7, 8, 10, 11, 3, 4, 9] 2024-03-20T09:28:10.063 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd out 10 2024-03-20T09:28:10.252 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:28:10.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:10.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:28:10.360 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:28:10.393 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:10.419 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:28:10.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:10.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:28:10.489 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:28:10.489 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:28:10.510 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:28:10.527 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:28:10.558 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:10.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:10 smithi115 ceph-mon[72476]: osdmap e806: 12 total, 11 up, 12 in 2024-03-20T09:28:10.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:10 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3367444762' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:28:10.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:10 smithi115 ceph-mon[72476]: pgmap v1953: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 7 active+undersized, 1 active+recovering+degraded+remapped, 43 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 445 op/s; 16158/221832 objects degraded (7.284%); 10349/221832 objects misplaced (4.665%); 11 MiB/s, 223 objects/s recovering 2024-03-20T09:28:10.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:10 smithi026 ceph-mon[68309]: osdmap e806: 12 total, 11 up, 12 in 2024-03-20T09:28:10.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:10 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3367444762' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:28:10.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:10 smithi026 ceph-mon[68309]: pgmap v1953: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 7 active+undersized, 1 active+recovering+degraded+remapped, 43 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 445 op/s; 16158/221832 objects degraded (7.284%); 10349/221832 objects misplaced (4.665%); 11 MiB/s, 223 objects/s recovering 2024-03-20T09:28:10.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:10 smithi122 ceph-mon[73495]: osdmap e806: 12 total, 11 up, 12 in 2024-03-20T09:28:10.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:10 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3367444762' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:28:10.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:10 smithi122 ceph-mon[73495]: pgmap v1953: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 7 active+undersized, 1 active+recovering+degraded+remapped, 43 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 445 op/s; 16158/221832 objects degraded (7.284%); 10349/221832 objects misplaced (4.665%); 11 MiB/s, 223 objects/s recovering 2024-03-20T09:28:11.314 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:28:11.315 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-20T09:28:11.317 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:11.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:28:11.418 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:28:11.473 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:11.542 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:28:11.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:11.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:28:11.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:11 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:28:11.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:11 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3367444762' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:28:11.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:11 smithi115 ceph-mon[72476]: osdmap e807: 12 total, 11 up, 12 in 2024-03-20T09:28:11.653 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:28:11.684 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:28:11.697 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:11.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:11 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:28:11.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:11 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3367444762' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:28:11.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:11 smithi122 ceph-mon[73495]: osdmap e807: 12 total, 11 up, 12 in 2024-03-20T09:28:11.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:11 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:28:11.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:11 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3367444762' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:28:11.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:11 smithi026 ceph-mon[68309]: osdmap e807: 12 total, 11 up, 12 in 2024-03-20T09:28:12.027 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:28:12.055 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:28:12.062 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:12.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:28:12.164 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:28:12.194 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:12.217 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:28:12.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:12.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:28:12.322 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:28:12.348 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:12.558 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:28:12.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:12.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:28:12.634 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:12 smithi026 ceph-mon[68309]: pgmap v1955: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 7 active+undersized, 1 active+recovering+degraded+remapped, 43 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 352 op/s; 15928/219117 objects degraded (7.269%); 10228/219117 objects misplaced (4.668%); 7.8 MiB/s, 124 objects/s recovering 2024-03-20T09:28:12.635 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:12 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 16158/221832 objects degraded (7.284%), 7 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:28:12.635 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:12 smithi026 ceph-mon[68309]: osdmap e808: 12 total, 11 up, 12 in 2024-03-20T09:28:12.663 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:28:12.692 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:12.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:12 smithi115 ceph-mon[72476]: pgmap v1955: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 7 active+undersized, 1 active+recovering+degraded+remapped, 43 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 352 op/s; 15928/219117 objects degraded (7.269%); 10228/219117 objects misplaced (4.668%); 7.8 MiB/s, 124 objects/s recovering 2024-03-20T09:28:12.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:12 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 16158/221832 objects degraded (7.284%), 7 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:28:12.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:12 smithi115 ceph-mon[72476]: osdmap e808: 12 total, 11 up, 12 in 2024-03-20T09:28:12.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:12 smithi122 ceph-mon[73495]: pgmap v1955: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 7 active+undersized, 1 active+recovering+degraded+remapped, 43 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 352 op/s; 15928/219117 objects degraded (7.269%); 10228/219117 objects misplaced (4.668%); 7.8 MiB/s, 124 objects/s recovering 2024-03-20T09:28:12.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:12 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 16158/221832 objects degraded (7.284%), 7 pgs degraded, 12 pgs undersized (PG_DEGRADED) 2024-03-20T09:28:12.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:12 smithi122 ceph-mon[73495]: osdmap e808: 12 total, 11 up, 12 in 2024-03-20T09:28:13.146 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:28:13.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:13.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:28:13.250 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:28:13.279 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:13.432 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:28:13.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:13.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:28:13.538 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:28:13.568 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:13.593 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:28:13.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:13 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:28:13.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:13 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:28:13.932 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:28:13.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:13.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:28:13.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:13 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:28:13.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:13 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:28:13.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:13 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:28:13.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:13 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:28:14.036 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:28:14.069 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:14.200 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:28:14.200 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:28:14.530 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:14 smithi115 ceph-mon[72476]: osdmap e809: 12 total, 11 up, 12 in 2024-03-20T09:28:14.530 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:14 smithi115 ceph-mon[72476]: pgmap v1958: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 383 op/s; 15928/218853 objects degraded (7.278%); 10228/218853 objects misplaced (4.673%); 8.0 MiB/s, 134 objects/s recovering 2024-03-20T09:28:14.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:14 smithi122 ceph-mon[73495]: osdmap e809: 12 total, 11 up, 12 in 2024-03-20T09:28:14.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:14 smithi122 ceph-mon[73495]: pgmap v1958: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 383 op/s; 15928/218853 objects degraded (7.278%); 10228/218853 objects misplaced (4.673%); 8.0 MiB/s, 134 objects/s recovering 2024-03-20T09:28:14.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:14 smithi026 ceph-mon[68309]: osdmap e809: 12 total, 11 up, 12 in 2024-03-20T09:28:14.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:14 smithi026 ceph-mon[68309]: pgmap v1958: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 383 op/s; 15928/218853 objects degraded (7.278%); 10228/218853 objects misplaced (4.673%); 8.0 MiB/s, 134 objects/s recovering 2024-03-20T09:28:15.045 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-20T09:28:15.394 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:28:15.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:15 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3349984854' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:28:15.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:15 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3349984854' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:28:15.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:15 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3349984854' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:28:16.335 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:28:16.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:16.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:28:16.450 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:28:16.450 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:28:16.450 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:28:16.450 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:28:16.450 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:28:16.450 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:28:16.450 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:28:16.450 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:28:16.450 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:28:16.451 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:28:16.451 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:28:16.451 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:28:16.451 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:28:16.451 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:28:16.451 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:28:16.451 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:28:16.451 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:28:16.451 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:28:16.451 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:28:16.451 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:28:16.451 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:28:16.452 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-03-20T09:28:16.452 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:28:16.452 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:28:16.483 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:16.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:16 smithi026 ceph-mon[68309]: pgmap v1959: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 4.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 427 op/s; 15759/214128 objects degraded (7.360%); 9501/214128 objects misplaced (4.437%); 7.1 MiB/s, 119 objects/s recovering 2024-03-20T09:28:16.854 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:16 smithi115 ceph-mon[72476]: pgmap v1959: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 4.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 427 op/s; 15759/214128 objects degraded (7.360%); 9501/214128 objects misplaced (4.437%); 7.1 MiB/s, 119 objects/s recovering 2024-03-20T09:28:16.935 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:28:16.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:16.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:28:16.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:16 smithi122 ceph-mon[73495]: pgmap v1959: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 4.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 427 op/s; 15759/214128 objects degraded (7.360%); 9501/214128 objects misplaced (4.437%); 7.1 MiB/s, 119 objects/s recovering 2024-03-20T09:28:17.040 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:28:17.067 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:17.088 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:28:17.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:17.091 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:28:17.124 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:28:17.124 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:28:17.191 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:28:17.238 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:17.369 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:28:17.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:17.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:28:17.475 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:28:17.476 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:28:17.476 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:28:17.476 INFO:teuthology.orchestra.run.smithi115.stdout: "complaint_time": 30, 2024-03-20T09:28:17.476 INFO:teuthology.orchestra.run.smithi115.stdout: "num_blocked_ops": 0 2024-03-20T09:28:17.476 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:28:17.515 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:17.716 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:28:17.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:17 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 15759/214128 objects degraded (7.360%), 6 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-03-20T09:28:17.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:17 smithi026 ceph-mon[68309]: osdmap e810: 12 total, 11 up, 12 in 2024-03-20T09:28:17.721 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:28:17.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:17.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:28:17.825 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:28:17.846 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:17 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 15759/214128 objects degraded (7.360%), 6 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-03-20T09:28:17.846 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:17 smithi115 ceph-mon[72476]: osdmap e810: 12 total, 11 up, 12 in 2024-03-20T09:28:17.851 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:17 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 15759/214128 objects degraded (7.360%), 6 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-03-20T09:28:17.851 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:17 smithi122 ceph-mon[73495]: osdmap e810: 12 total, 11 up, 12 in 2024-03-20T09:28:17.853 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:17.872 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:28:17.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:17.875 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:28:17.976 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:28:18.013 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:18.215 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:28:18.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:18.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:28:18.233 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-20T09:28:18.322 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:28:18.348 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:18.368 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:28:18.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:18.371 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:28:18.465 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:18 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mon-a[68286]: 2024-03-20T09:28:18.179+0000 7f683364b700 -1 mon.a@0(leader).osd e810 definitely_dead 0 2024-03-20T09:28:18.471 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:28:18.498 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:18.517 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:28:18.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:18.520 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:28:18.551 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:28:18.569 INFO:teuthology.orchestra.run.smithi026.stderr:marked out osd.10. 2024-03-20T09:28:18.621 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:28:18.645 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:18 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3715913652' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-03-20T09:28:18.645 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:18 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-03-20T09:28:18.645 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:18 smithi122 ceph-mon[73495]: Client client.admin marked osd.10 out, after it was down for 134 seconds 2024-03-20T09:28:18.645 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:18 smithi122 ceph-mon[73495]: pgmap v1961: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 4.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 336 op/s; 15512/213051 objects degraded (7.281%); 9368/213051 objects misplaced (4.397%); 7.2 MiB/s, 119 objects/s recovering 2024-03-20T09:28:18.645 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:18 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:28:18.645 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:18 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:28:18.647 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:18.666 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:28:18.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:18.669 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:28:18.770 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:28:18.797 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:18.817 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:28:18.817 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:18 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3715913652' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-03-20T09:28:18.817 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:18 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-03-20T09:28:18.817 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:18 smithi115 ceph-mon[72476]: Client client.admin marked osd.10 out, after it was down for 134 seconds 2024-03-20T09:28:18.817 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:18 smithi115 ceph-mon[72476]: pgmap v1961: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 4.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 336 op/s; 15512/213051 objects degraded (7.281%); 9368/213051 objects misplaced (4.397%); 7.2 MiB/s, 119 objects/s recovering 2024-03-20T09:28:18.817 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:18 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:28:18.818 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:18 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:28:18.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:18.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:28:18.921 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:28:18.947 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:18.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:18 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3715913652' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-03-20T09:28:18.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:18 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["10"]}]: dispatch 2024-03-20T09:28:18.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:18 smithi026 ceph-mon[68309]: Client client.admin marked osd.10 out, after it was down for 134 seconds 2024-03-20T09:28:18.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:18 smithi026 ceph-mon[68309]: pgmap v1961: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 7 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 4.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 336 op/s; 15512/213051 objects degraded (7.281%); 9368/213051 objects misplaced (4.397%); 7.2 MiB/s, 119 objects/s recovering 2024-03-20T09:28:18.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:18 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:28:18.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:18 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:28:19.126 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:28:19.294 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:28:19.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:19.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:28:19.398 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:28:19.429 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:19.681 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:28:19.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:19.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:28:19.687 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:19 smithi026 ceph-mon[68309]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-20T09:28:19.687 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:19 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-03-20T09:28:19.687 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:19 smithi026 ceph-mon[68309]: osdmap e811: 12 total, 11 up, 11 in 2024-03-20T09:28:19.787 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:28:19.812 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:19 smithi122 ceph-mon[73495]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-20T09:28:19.812 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:19 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-03-20T09:28:19.812 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:19 smithi122 ceph-mon[73495]: osdmap e811: 12 total, 11 up, 11 in 2024-03-20T09:28:19.813 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:19.833 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:28:19.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:19.836 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:28:19.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:19 smithi115 ceph-mon[72476]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-20T09:28:19.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:19 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-03-20T09:28:19.851 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:19 smithi115 ceph-mon[72476]: osdmap e811: 12 total, 11 up, 11 in 2024-03-20T09:28:19.937 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:28:19.965 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:20.163 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:28:20.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:20.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:28:20.267 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:28:20.298 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:20.425 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:28:20.426 INFO:teuthology.orchestra.run.smithi115.stdout: "size": 20, 2024-03-20T09:28:20.426 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 600, 2024-03-20T09:28:20.426 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [ 2024-03-20T09:28:20.426 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.426 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "rep_scrubmap(1.15 e491 from shard 8)", 2024-03-20T09:28:20.426 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:18:46.587440+0000", 2024-03-20T09:28:20.426 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 573.84126859499997, 2024-03-20T09:28:20.426 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.050810584999999998, 2024-03-20T09:28:20.426 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:28:20.426 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:28:20.427 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:28:20.427 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.427 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:28:20.427 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:46.587440+0000", 2024-03-20T09:28:20.427 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.427 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.427 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.427 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:28:20.427 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:46.587440+0000", 2024-03-20T09:28:20.427 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.427 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.427 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.428 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:28:20.428 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:46.587441+0000", 2024-03-20T09:28:20.428 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.06e-07 2024-03-20T09:28:20.428 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.428 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.428 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:28:20.428 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:46.587446+0000", 2024-03-20T09:28:20.428 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6280000000000004e-06 2024-03-20T09:28:20.428 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.428 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.428 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:28:20.428 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:46.587454+0000", 2024-03-20T09:28:20.428 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5080000000000005e-06 2024-03-20T09:28:20.429 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.429 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.429 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:28:20.429 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:46.587460+0000", 2024-03-20T09:28:20.429 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7339999999999996e-06 2024-03-20T09:28:20.429 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.429 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.429 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:28:20.429 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:46.638224+0000", 2024-03-20T09:28:20.429 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.050764338999999999 2024-03-20T09:28:20.429 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.429 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.429 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:28:20.430 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:46.638236+0000", 2024-03-20T09:28:20.430 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1909000000000001e-05 2024-03-20T09:28:20.430 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.430 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.430 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:28:20.430 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:46.638251+0000", 2024-03-20T09:28:20.430 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4761e-05 2024-03-20T09:28:20.430 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.430 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:28:20.430 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.431 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.431 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.431 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "rep_scrubmap(1.15 e491 from shard 5)", 2024-03-20T09:28:20.431 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:18:46.592022+0000", 2024-03-20T09:28:20.431 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 573.836686902, 2024-03-20T09:28:20.431 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.047802478000000002, 2024-03-20T09:28:20.431 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:28:20.431 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:28:20.431 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:28:20.431 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.431 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:28:20.431 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:46.592022+0000", 2024-03-20T09:28:20.431 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.432 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.432 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.432 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:28:20.432 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:46.592022+0000", 2024-03-20T09:28:20.432 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.432 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.432 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.432 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:28:20.432 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:46.592023+0000", 2024-03-20T09:28:20.432 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.3699999999999999e-07 2024-03-20T09:28:20.432 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.432 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.433 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:28:20.433 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:46.592029+0000", 2024-03-20T09:28:20.433 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4899999999999997e-06 2024-03-20T09:28:20.433 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.433 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.433 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:28:20.433 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:46.592040+0000", 2024-03-20T09:28:20.433 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0597e-05 2024-03-20T09:28:20.433 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.433 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.433 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:28:20.434 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:46.592047+0000", 2024-03-20T09:28:20.434 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.9450000000000004e-06 2024-03-20T09:28:20.434 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.434 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.434 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:28:20.434 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:46.639794+0000", 2024-03-20T09:28:20.434 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.047747448999999997 2024-03-20T09:28:20.434 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.434 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.434 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:28:20.434 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:46.639804+0000", 2024-03-20T09:28:20.434 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.5999999999999996e-06 2024-03-20T09:28:20.434 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.434 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.435 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:28:20.435 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:46.639825+0000", 2024-03-20T09:28:20.435 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0460000000000001e-05 2024-03-20T09:28:20.435 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.435 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:28:20.435 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.435 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.435 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.435 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:57998 5.d e619/605 5:bb4a3c37:::benchmark_data_smithi122_157894_object57997:head v 619'3678, mlcod=619'3675)", 2024-03-20T09:28:20.435 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:18:46.592087+0000", 2024-03-20T09:28:20.435 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 573.83662241800005, 2024-03-20T09:28:20.435 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.054305742999999997, 2024-03-20T09:28:20.435 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:28:20.436 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:28:20.436 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:28:20.436 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.436 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:28:20.436 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:46.592087+0000", 2024-03-20T09:28:20.436 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.436 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.436 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.436 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:28:20.436 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:46.592087+0000", 2024-03-20T09:28:20.436 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.436 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.437 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.437 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:28:20.437 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:46.592087+0000", 2024-03-20T09:28:20.437 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8500000000000002e-07 2024-03-20T09:28:20.437 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.437 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.437 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:28:20.437 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:46.592169+0000", 2024-03-20T09:28:20.437 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.1817000000000006e-05 2024-03-20T09:28:20.437 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.437 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.437 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:28:20.438 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:46.592177+0000", 2024-03-20T09:28:20.438 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.9759999999999995e-06 2024-03-20T09:28:20.438 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.438 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.438 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:28:20.438 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:46.592187+0000", 2024-03-20T09:28:20.438 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0679999999999999e-05 2024-03-20T09:28:20.438 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.438 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.438 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:28:20.439 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:46.638276+0000", 2024-03-20T09:28:20.439 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.046088203000000001 2024-03-20T09:28:20.439 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.439 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.439 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:28:20.439 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:46.638295+0000", 2024-03-20T09:28:20.439 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9232000000000002e-05 2024-03-20T09:28:20.439 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.439 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.439 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:28:20.439 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:46.646341+0000", 2024-03-20T09:28:20.439 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.008045768 2024-03-20T09:28:20.439 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.439 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.440 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:28:20.440 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:18:46.646392+0000", 2024-03-20T09:28:20.440 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1582000000000002e-05 2024-03-20T09:28:20.440 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.440 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:28:20.440 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.440 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.440 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.440 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:229938 5.9 e688/605 5:94157346:::benchmark_data_smithi122_157894_object99951:head v 688'14428, mlcod=688'14426)", 2024-03-20T09:28:20.440 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:57.310743+0000", 2024-03-20T09:28:20.440 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 323.117965841, 2024-03-20T09:28:20.440 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.056839437, 2024-03-20T09:28:20.440 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:28:20.440 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:28:20.440 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:28:20.441 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.441 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:28:20.441 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.310743+0000", 2024-03-20T09:28:20.441 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.441 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.441 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.441 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:28:20.441 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.310743+0000", 2024-03-20T09:28:20.441 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.441 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.441 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.441 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:28:20.441 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.310744+0000", 2024-03-20T09:28:20.442 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0640000000000001e-06 2024-03-20T09:28:20.442 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.442 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.442 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:28:20.442 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.310752+0000", 2024-03-20T09:28:20.442 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.2179999999999999e-06 2024-03-20T09:28:20.442 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.442 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.442 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:28:20.443 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.310766+0000", 2024-03-20T09:28:20.443 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3492999999999999e-05 2024-03-20T09:28:20.443 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.443 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.443 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:28:20.443 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.310774+0000", 2024-03-20T09:28:20.443 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.1049999999999995e-06 2024-03-20T09:28:20.443 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.443 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.443 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:28:20.443 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.310845+0000", 2024-03-20T09:28:20.443 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.1038999999999995e-05 2024-03-20T09:28:20.443 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.443 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.444 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:28:20.444 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.310865+0000", 2024-03-20T09:28:20.444 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9740000000000001e-05 2024-03-20T09:28:20.444 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.444 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.444 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:28:20.444 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.367534+0000", 2024-03-20T09:28:20.444 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.056668942 2024-03-20T09:28:20.444 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.444 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.444 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:28:20.444 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.367583+0000", 2024-03-20T09:28:20.444 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8835999999999998e-05 2024-03-20T09:28:20.444 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.444 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:28:20.445 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.445 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.445 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.445 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:229930 5.1 e688/637 5:85858a80:::benchmark_data_smithi122_157894_object99943:head v 688'14235, mlcod=628'6357)", 2024-03-20T09:28:20.445 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:57.318675+0000", 2024-03-20T09:28:20.445 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 323.11003397000002, 2024-03-20T09:28:20.445 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.048971187999999999, 2024-03-20T09:28:20.445 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:28:20.445 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:28:20.445 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:28:20.445 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.445 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:28:20.445 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.318675+0000", 2024-03-20T09:28:20.445 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.446 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.446 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.446 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:28:20.446 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.318675+0000", 2024-03-20T09:28:20.446 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.446 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.446 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.446 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:28:20.446 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.318675+0000", 2024-03-20T09:28:20.446 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.0999999999999999e-07 2024-03-20T09:28:20.446 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.446 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.446 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:28:20.446 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.318682+0000", 2024-03-20T09:28:20.447 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.7270000000000001e-06 2024-03-20T09:28:20.447 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.447 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.447 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:28:20.447 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.318689+0000", 2024-03-20T09:28:20.447 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4910000000000002e-06 2024-03-20T09:28:20.447 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.447 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.447 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:28:20.447 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.318693+0000", 2024-03-20T09:28:20.447 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.882e-06 2024-03-20T09:28:20.447 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.447 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.447 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:28:20.447 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.318753+0000", 2024-03-20T09:28:20.448 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9120000000000003e-05 2024-03-20T09:28:20.448 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.448 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.448 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:28:20.448 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.318780+0000", 2024-03-20T09:28:20.448 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.7080000000000002e-05 2024-03-20T09:28:20.448 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.448 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.448 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:28:20.448 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.367587+0000", 2024-03-20T09:28:20.448 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.048807017000000001 2024-03-20T09:28:20.448 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.448 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.449 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:28:20.449 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.367646+0000", 2024-03-20T09:28:20.449 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9460999999999999e-05 2024-03-20T09:28:20.449 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.449 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:28:20.449 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.449 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.449 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.449 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:229939 5.1 e688/637 5:8f3e9e8a:::benchmark_data_smithi122_157894_object99952:head v 688'14236, mlcod=628'6357)", 2024-03-20T09:28:20.449 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:57.318710+0000", 2024-03-20T09:28:20.449 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 323.10999909200001, 2024-03-20T09:28:20.449 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.048965195000000003, 2024-03-20T09:28:20.449 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:28:20.449 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:28:20.450 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:28:20.450 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.450 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:28:20.450 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.318710+0000", 2024-03-20T09:28:20.450 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.450 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.450 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.450 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:28:20.450 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.318710+0000", 2024-03-20T09:28:20.450 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.450 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.450 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.450 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:28:20.450 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.318710+0000", 2024-03-20T09:28:20.451 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.48e-07 2024-03-20T09:28:20.451 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.451 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.451 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:28:20.451 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.318714+0000", 2024-03-20T09:28:20.451 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.4400000000000001e-06 2024-03-20T09:28:20.451 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.451 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.451 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:28:20.451 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.318719+0000", 2024-03-20T09:28:20.451 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1549999999999998e-06 2024-03-20T09:28:20.451 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.451 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.451 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:28:20.452 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.318724+0000", 2024-03-20T09:28:20.452 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3210000000000004e-06 2024-03-20T09:28:20.452 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.452 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.452 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:28:20.452 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.323677+0000", 2024-03-20T09:28:20.452 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0049531159999999996 2024-03-20T09:28:20.452 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.452 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.452 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:28:20.452 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.323696+0000", 2024-03-20T09:28:20.452 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9012e-05 2024-03-20T09:28:20.452 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.452 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.453 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:28:20.453 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.367648+0000", 2024-03-20T09:28:20.453 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.043952264999999997 2024-03-20T09:28:20.453 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.453 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.453 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:28:20.453 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.367675+0000", 2024-03-20T09:28:20.453 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6638000000000001e-05 2024-03-20T09:28:20.453 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.453 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:28:20.453 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.453 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.453 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.453 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:231357 5.1 e690/637 5:862a1852:::benchmark_data_smithi122_157894_object101370:head v 690'14319, mlcod=628'6446)", 2024-03-20T09:28:20.454 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:59.484435+0000", 2024-03-20T09:28:20.454 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 320.94427369900001, 2024-03-20T09:28:20.454 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.046223444000000002, 2024-03-20T09:28:20.454 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:28:20.454 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:28:20.454 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:28:20.454 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.454 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:28:20.454 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.484435+0000", 2024-03-20T09:28:20.454 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.454 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.454 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.454 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:28:20.454 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.484435+0000", 2024-03-20T09:28:20.455 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.455 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.455 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.455 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:28:20.455 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.484436+0000", 2024-03-20T09:28:20.455 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.7999999999999993e-07 2024-03-20T09:28:20.455 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.455 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.455 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:28:20.455 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.484446+0000", 2024-03-20T09:28:20.455 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.8269999999999993e-06 2024-03-20T09:28:20.455 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.455 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.455 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:28:20.456 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.484459+0000", 2024-03-20T09:28:20.456 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2744e-05 2024-03-20T09:28:20.456 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.456 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.456 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:28:20.456 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.484468+0000", 2024-03-20T09:28:20.456 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.8310000000000007e-06 2024-03-20T09:28:20.456 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.456 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.456 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:28:20.456 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.484615+0000", 2024-03-20T09:28:20.456 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00014736600000000001 2024-03-20T09:28:20.456 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.456 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.457 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:28:20.457 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.484642+0000", 2024-03-20T09:28:20.457 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6698000000000001e-05 2024-03-20T09:28:20.457 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.457 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.457 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:28:20.457 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.530360+0000", 2024-03-20T09:28:20.457 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.045717854000000002 2024-03-20T09:28:20.457 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.457 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.457 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:28:20.457 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.530659+0000", 2024-03-20T09:28:20.457 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00029914399999999999 2024-03-20T09:28:20.457 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.458 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:28:20.458 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.458 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.458 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.458 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:231351 5.9 e690/605 5:9562aab7:::benchmark_data_smithi122_157894_object101364:head v 690'14515, mlcod=690'14513)", 2024-03-20T09:28:20.458 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:59.485231+0000", 2024-03-20T09:28:20.458 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 320.94347771399998, 2024-03-20T09:28:20.458 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.049753224999999998, 2024-03-20T09:28:20.458 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:28:20.458 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:28:20.458 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:28:20.458 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.458 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:28:20.459 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.485231+0000", 2024-03-20T09:28:20.459 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.459 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.459 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.459 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:28:20.459 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.485231+0000", 2024-03-20T09:28:20.459 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.459 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.459 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.459 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:28:20.459 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.485232+0000", 2024-03-20T09:28:20.459 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7100000000000001e-07 2024-03-20T09:28:20.459 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.460 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.460 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:28:20.460 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.485240+0000", 2024-03-20T09:28:20.460 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7700000000000001e-06 2024-03-20T09:28:20.460 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.460 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.460 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:28:20.460 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.485251+0000", 2024-03-20T09:28:20.460 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1608e-05 2024-03-20T09:28:20.460 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.460 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.460 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:28:20.460 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.485259+0000", 2024-03-20T09:28:20.461 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.289e-06 2024-03-20T09:28:20.461 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.461 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.461 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:28:20.461 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.489654+0000", 2024-03-20T09:28:20.461 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0043954060000000001 2024-03-20T09:28:20.461 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.461 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.461 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:28:20.461 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.489678+0000", 2024-03-20T09:28:20.461 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3524999999999999e-05 2024-03-20T09:28:20.461 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.461 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.462 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:28:20.462 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.534951+0000", 2024-03-20T09:28:20.462 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.045273571999999998 2024-03-20T09:28:20.462 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.462 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.462 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:28:20.462 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.534984+0000", 2024-03-20T09:28:20.462 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.3284000000000002e-05 2024-03-20T09:28:20.462 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.462 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:28:20.462 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.462 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.463 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.463 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:232222 5.d e690/605 5:b74bcbe9:::benchmark_data_smithi122_157894_object102235:head v 690'14600, mlcod=690'14597)", 2024-03-20T09:28:20.463 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:23:00.802652+0000", 2024-03-20T09:28:20.463 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 319.62605712599998, 2024-03-20T09:28:20.463 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.049633355999999997, 2024-03-20T09:28:20.463 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:28:20.463 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:28:20.463 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:28:20.463 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.463 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:28:20.463 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:00.802652+0000", 2024-03-20T09:28:20.463 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.464 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.464 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.464 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:28:20.464 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:00.802652+0000", 2024-03-20T09:28:20.464 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.464 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.464 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.464 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:28:20.464 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:00.802653+0000", 2024-03-20T09:28:20.464 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.22e-06 2024-03-20T09:28:20.464 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.464 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.464 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:28:20.465 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:00.802661+0000", 2024-03-20T09:28:20.465 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7090000000000002e-06 2024-03-20T09:28:20.465 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.465 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.465 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:28:20.465 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:00.802674+0000", 2024-03-20T09:28:20.465 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2901999999999999e-05 2024-03-20T09:28:20.465 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.465 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.465 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:28:20.465 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:00.802680+0000", 2024-03-20T09:28:20.465 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.5200000000000003e-06 2024-03-20T09:28:20.466 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.466 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.466 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:28:20.466 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:00.804961+0000", 2024-03-20T09:28:20.466 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0022811049999999999 2024-03-20T09:28:20.466 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.466 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.466 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:28:20.466 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:00.804982+0000", 2024-03-20T09:28:20.466 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0432999999999999e-05 2024-03-20T09:28:20.466 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.466 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.466 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:28:20.467 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:00.852239+0000", 2024-03-20T09:28:20.467 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.047257413999999998 2024-03-20T09:28:20.467 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.467 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.467 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:28:20.467 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:00.852285+0000", 2024-03-20T09:28:20.467 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6053000000000001e-05 2024-03-20T09:28:20.467 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.467 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:28:20.467 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.467 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.467 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.467 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:250737 5.9 e697/605 5:912b5ada:::benchmark_data_smithi122_157894_object120750:head v 697'15732, mlcod=697'15729)", 2024-03-20T09:28:20.468 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:23:28.731645+0000", 2024-03-20T09:28:20.468 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 291.69706396100003, 2024-03-20T09:28:20.468 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.046361630000000001, 2024-03-20T09:28:20.468 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:28:20.468 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:28:20.468 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:28:20.468 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.468 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:28:20.468 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.731645+0000", 2024-03-20T09:28:20.468 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.468 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.468 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.468 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:28:20.469 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.731645+0000", 2024-03-20T09:28:20.469 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.469 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.469 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.469 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:28:20.469 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.731646+0000", 2024-03-20T09:28:20.469 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.7000000000000003e-07 2024-03-20T09:28:20.469 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.469 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.470 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:28:20.470 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.731652+0000", 2024-03-20T09:28:20.470 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0379999999999996e-06 2024-03-20T09:28:20.470 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.470 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.470 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:28:20.470 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.731662+0000", 2024-03-20T09:28:20.470 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.995e-06 2024-03-20T09:28:20.470 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.470 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.470 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:28:20.470 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.731669+0000", 2024-03-20T09:28:20.470 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.5009999999999998e-06 2024-03-20T09:28:20.471 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.471 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.471 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:28:20.471 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.752990+0000", 2024-03-20T09:28:20.471 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.021320260000000001 2024-03-20T09:28:20.471 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.471 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.471 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:28:20.471 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.753014+0000", 2024-03-20T09:28:20.471 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.4199999999999999e-05 2024-03-20T09:28:20.471 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.472 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.472 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:28:20.472 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.777963+0000", 2024-03-20T09:28:20.472 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.024948682 2024-03-20T09:28:20.472 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.472 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.472 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:28:20.472 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.778007+0000", 2024-03-20T09:28:20.472 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4084e-05 2024-03-20T09:28:20.472 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.472 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:28:20.472 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.473 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.473 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.473 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:250756 5.1 e697/637 5:8e3474d2:::benchmark_data_smithi122_157894_object120769:head v 697'15500, mlcod=635'7965)", 2024-03-20T09:28:20.473 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:23:28.752816+0000", 2024-03-20T09:28:20.473 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 291.67589315100003, 2024-03-20T09:28:20.473 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.046701550000000001, 2024-03-20T09:28:20.473 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:28:20.473 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:28:20.473 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:28:20.473 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.473 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:28:20.473 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.752816+0000", 2024-03-20T09:28:20.473 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.474 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.474 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.474 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:28:20.474 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.752816+0000", 2024-03-20T09:28:20.474 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.474 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.474 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.474 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:28:20.474 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.752817+0000", 2024-03-20T09:28:20.474 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.4e-07 2024-03-20T09:28:20.474 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.474 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.474 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:28:20.475 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.752825+0000", 2024-03-20T09:28:20.475 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.8730000000000007e-06 2024-03-20T09:28:20.475 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.475 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.475 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:28:20.475 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.752835+0000", 2024-03-20T09:28:20.475 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0565e-05 2024-03-20T09:28:20.475 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.475 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.475 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:28:20.475 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.752843+0000", 2024-03-20T09:28:20.475 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.5410000000000003e-06 2024-03-20T09:28:20.475 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.475 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.476 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:28:20.476 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.774230+0000", 2024-03-20T09:28:20.476 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.021387782000000001 2024-03-20T09:28:20.476 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.476 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.476 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:28:20.476 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.774249+0000", 2024-03-20T09:28:20.476 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8221000000000001e-05 2024-03-20T09:28:20.476 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.476 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.476 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:28:20.476 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.799488+0000", 2024-03-20T09:28:20.476 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.025239444999999999 2024-03-20T09:28:20.476 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.477 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.477 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:28:20.477 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.799517+0000", 2024-03-20T09:28:20.477 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9283000000000001e-05 2024-03-20T09:28:20.477 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.477 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:28:20.477 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.477 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.477 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.477 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "rep_scrubmap(1.15 e491 from shard 5)", 2024-03-20T09:28:20.477 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:14.417982+0000", 2024-03-20T09:28:20.477 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 186.010726753, 2024-03-20T09:28:20.477 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.059782971999999997, 2024-03-20T09:28:20.478 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:28:20.478 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:28:20.478 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:28:20.478 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.478 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:28:20.478 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.417982+0000", 2024-03-20T09:28:20.478 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.478 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.478 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.478 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:28:20.478 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.417982+0000", 2024-03-20T09:28:20.478 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.478 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.479 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.479 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:28:20.479 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.417983+0000", 2024-03-20T09:28:20.479 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3099999999999997e-07 2024-03-20T09:28:20.479 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.479 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.479 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:28:20.479 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.417988+0000", 2024-03-20T09:28:20.479 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4790000000000004e-06 2024-03-20T09:28:20.479 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.479 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.479 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:28:20.479 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.417998+0000", 2024-03-20T09:28:20.479 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.2399999999999996e-06 2024-03-20T09:28:20.480 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.480 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.480 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:28:20.480 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.418003+0000", 2024-03-20T09:28:20.480 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5160000000000002e-06 2024-03-20T09:28:20.480 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.480 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.480 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:28:20.480 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.477737+0000", 2024-03-20T09:28:20.480 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.059734260999999997 2024-03-20T09:28:20.480 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.480 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.480 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:28:20.480 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.477749+0000", 2024-03-20T09:28:20.481 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.149e-05 2024-03-20T09:28:20.481 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.481 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.481 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:28:20.481 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.477765+0000", 2024-03-20T09:28:20.481 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6254999999999999e-05 2024-03-20T09:28:20.481 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.481 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:28:20.481 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.481 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.481 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.481 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "rep_scrubmap(1.15 e491 from shard 8)", 2024-03-20T09:28:20.481 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:14.425166+0000", 2024-03-20T09:28:20.481 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 186.00354264699999, 2024-03-20T09:28:20.481 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.052696894000000001, 2024-03-20T09:28:20.482 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:28:20.482 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:28:20.482 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:28:20.482 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.482 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:28:20.482 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.425166+0000", 2024-03-20T09:28:20.482 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.482 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.482 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.482 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:28:20.482 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.425166+0000", 2024-03-20T09:28:20.482 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.482 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.483 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.483 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:28:20.483 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.425167+0000", 2024-03-20T09:28:20.483 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.9599999999999998e-07 2024-03-20T09:28:20.483 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.483 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.483 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:28:20.483 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.425173+0000", 2024-03-20T09:28:20.483 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9529999999999996e-06 2024-03-20T09:28:20.483 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.483 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.483 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:28:20.483 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.425194+0000", 2024-03-20T09:28:20.483 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1285999999999999e-05 2024-03-20T09:28:20.484 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.484 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.484 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:28:20.484 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.425202+0000", 2024-03-20T09:28:20.484 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7039999999999994e-06 2024-03-20T09:28:20.484 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.484 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.484 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:28:20.484 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.477824+0000", 2024-03-20T09:28:20.484 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.052622206999999997 2024-03-20T09:28:20.484 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.484 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.484 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:28:20.484 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.477835+0000", 2024-03-20T09:28:20.485 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0788000000000001e-05 2024-03-20T09:28:20.485 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.485 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.485 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:28:20.485 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.477863+0000", 2024-03-20T09:28:20.485 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8160000000000001e-05 2024-03-20T09:28:20.485 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.485 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:28:20.485 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.485 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.485 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.485 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26216.0:86454 6.5 e733/709 6:a2958d06:::benchmark_data_smithi122_168655_object86453:head v 733'5375, mlcod=732'5373)", 2024-03-20T09:28:20.485 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:14.436942+0000", 2024-03-20T09:28:20.485 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 185.99176740799999, 2024-03-20T09:28:20.486 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.051506415, 2024-03-20T09:28:20.486 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:28:20.486 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:28:20.486 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:28:20.486 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.486 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:28:20.486 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.436942+0000", 2024-03-20T09:28:20.486 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.486 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.486 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.486 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:28:20.486 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.436942+0000", 2024-03-20T09:28:20.486 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.487 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.487 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.487 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:28:20.487 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.436943+0000", 2024-03-20T09:28:20.487 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.403e-06 2024-03-20T09:28:20.487 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.487 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.487 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:28:20.487 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.436964+0000", 2024-03-20T09:28:20.487 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0661000000000001e-05 2024-03-20T09:28:20.487 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.487 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.487 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:28:20.487 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.436980+0000", 2024-03-20T09:28:20.488 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6328e-05 2024-03-20T09:28:20.488 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.488 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.488 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:28:20.488 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.436988+0000", 2024-03-20T09:28:20.488 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.0169999999999996e-06 2024-03-20T09:28:20.488 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.488 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.488 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:28:20.488 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.477790+0000", 2024-03-20T09:28:20.488 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.040802120999999997 2024-03-20T09:28:20.488 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.488 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.488 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:28:20.489 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.477807+0000", 2024-03-20T09:28:20.489 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7297000000000001e-05 2024-03-20T09:28:20.489 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.489 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.489 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:28:20.489 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.488398+0000", 2024-03-20T09:28:20.489 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.010590373 2024-03-20T09:28:20.489 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.489 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.489 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:28:20.489 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.488448+0000", 2024-03-20T09:28:20.489 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0214999999999999e-05 2024-03-20T09:28:20.489 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.490 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:28:20.490 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.490 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.490 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.490 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26216.0:129134 6.b e748/709 6:dfb654b3:::benchmark_data_smithi122_168655_object9025:head v 748'8030, mlcod=748'8027)", 2024-03-20T09:28:20.490 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:56.321820+0000", 2024-03-20T09:28:20.490 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 144.106889282, 2024-03-20T09:28:20.490 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.26522621299999999, 2024-03-20T09:28:20.490 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:28:20.490 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:28:20.490 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:28:20.490 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.490 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:28:20.490 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.321820+0000", 2024-03-20T09:28:20.491 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.491 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.491 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.491 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:28:20.491 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.321820+0000", 2024-03-20T09:28:20.491 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.491 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.491 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.491 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:28:20.491 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.321821+0000", 2024-03-20T09:28:20.491 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.437e-06 2024-03-20T09:28:20.491 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.491 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.492 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:28:20.492 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.321829+0000", 2024-03-20T09:28:20.492 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7009999999999996e-06 2024-03-20T09:28:20.492 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.492 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.492 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:28:20.492 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.321845+0000", 2024-03-20T09:28:20.492 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5852999999999999e-05 2024-03-20T09:28:20.492 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.492 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.492 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:28:20.492 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.321853+0000", 2024-03-20T09:28:20.492 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.2379999999999997e-06 2024-03-20T09:28:20.493 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.493 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.493 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:28:20.493 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.321913+0000", 2024-03-20T09:28:20.493 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0377999999999997e-05 2024-03-20T09:28:20.493 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.493 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.493 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:28:20.493 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.321938+0000", 2024-03-20T09:28:20.493 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.4329e-05 2024-03-20T09:28:20.493 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.493 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.493 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:28:20.493 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.586991+0000", 2024-03-20T09:28:20.494 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.26505305800000001 2024-03-20T09:28:20.494 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.494 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.494 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:28:20.494 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.587046+0000", 2024-03-20T09:28:20.494 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5219000000000001e-05 2024-03-20T09:28:20.494 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.494 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:28:20.494 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.494 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.494 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.494 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26216.0:129139 6.8 e748/709 6:1a6b87cd:::benchmark_data_smithi122_168655_object9030:head v 748'7939, mlcod=748'7937)", 2024-03-20T09:28:20.494 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:56.323431+0000", 2024-03-20T09:28:20.495 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 144.10527763100001, 2024-03-20T09:28:20.495 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.234771442, 2024-03-20T09:28:20.495 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:28:20.495 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:28:20.495 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:28:20.495 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.495 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:28:20.495 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.323431+0000", 2024-03-20T09:28:20.495 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.495 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.495 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.495 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:28:20.495 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.323431+0000", 2024-03-20T09:28:20.496 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.496 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.496 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.496 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:28:20.496 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.323432+0000", 2024-03-20T09:28:20.496 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.4e-07 2024-03-20T09:28:20.496 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.496 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.496 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:28:20.496 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.323439+0000", 2024-03-20T09:28:20.496 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4559999999999997e-06 2024-03-20T09:28:20.496 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.496 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.497 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:28:20.497 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.323452+0000", 2024-03-20T09:28:20.497 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3574000000000001e-05 2024-03-20T09:28:20.497 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.497 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.497 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:28:20.497 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.323460+0000", 2024-03-20T09:28:20.497 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.5669999999999997e-06 2024-03-20T09:28:20.497 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.497 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.497 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:28:20.497 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.323522+0000", 2024-03-20T09:28:20.497 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2389000000000002e-05 2024-03-20T09:28:20.498 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.498 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.498 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:28:20.498 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.323544+0000", 2024-03-20T09:28:20.498 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1702e-05 2024-03-20T09:28:20.498 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.498 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.498 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:28:20.498 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.558203+0000", 2024-03-20T09:28:20.498 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.23465881399999999 2024-03-20T09:28:20.498 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.498 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:28:20.499 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.499 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.499 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.499 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26216.0:129140 6.3 e748/709)", 2024-03-20T09:28:20.499 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:56.327278+0000", 2024-03-20T09:28:20.499 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 144.101430798, 2024-03-20T09:28:20.499 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.26500126899999998, 2024-03-20T09:28:20.499 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:28:20.499 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:28:20.499 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:28:20.499 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.499 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:28:20.499 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.327278+0000", 2024-03-20T09:28:20.500 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.500 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.500 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.500 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:28:20.500 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.327278+0000", 2024-03-20T09:28:20.500 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.500 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.500 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.500 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:28:20.500 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.327279+0000", 2024-03-20T09:28:20.500 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0669999999999999e-06 2024-03-20T09:28:20.500 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.500 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.500 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:28:20.500 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.327288+0000", 2024-03-20T09:28:20.501 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5920000000000001e-06 2024-03-20T09:28:20.501 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.501 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.501 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:28:20.501 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.327301+0000", 2024-03-20T09:28:20.501 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2768e-05 2024-03-20T09:28:20.501 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.501 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.501 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:28:20.501 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.327308+0000", 2024-03-20T09:28:20.501 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.2579999999999998e-06 2024-03-20T09:28:20.501 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.502 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.502 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:28:20.502 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.592260+0000", 2024-03-20T09:28:20.502 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.26495219599999997 2024-03-20T09:28:20.502 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.502 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.502 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:28:20.502 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.592279+0000", 2024-03-20T09:28:20.502 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9388000000000001e-05 2024-03-20T09:28:20.502 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.502 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:28:20.502 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.502 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.503 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.503 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26216.0:129141 6.b e748/709 6:ddae97ab:::benchmark_data_smithi122_168655_object9032:head v 748'8031, mlcod=748'8028)", 2024-03-20T09:28:20.503 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:56.329794+0000", 2024-03-20T09:28:20.503 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 144.09891537600001, 2024-03-20T09:28:20.503 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.29588104700000001, 2024-03-20T09:28:20.503 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:28:20.503 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:28:20.503 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:28:20.503 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.503 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:28:20.503 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.329794+0000", 2024-03-20T09:28:20.503 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.503 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.503 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.503 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:28:20.504 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.329794+0000", 2024-03-20T09:28:20.504 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.504 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.504 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.504 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:28:20.504 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.329795+0000", 2024-03-20T09:28:20.504 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.083e-06 2024-03-20T09:28:20.504 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.504 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.504 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:28:20.504 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.329803+0000", 2024-03-20T09:28:20.504 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.7450000000000001e-06 2024-03-20T09:28:20.504 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.504 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.504 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:28:20.505 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.329815+0000", 2024-03-20T09:28:20.505 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1790000000000001e-05 2024-03-20T09:28:20.505 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.505 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.505 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:28:20.505 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.329822+0000", 2024-03-20T09:28:20.505 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.7440000000000001e-06 2024-03-20T09:28:20.505 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.505 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.505 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:28:20.505 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.582786+0000", 2024-03-20T09:28:20.505 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.25296439500000001 2024-03-20T09:28:20.505 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.505 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.506 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:28:20.506 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.582813+0000", 2024-03-20T09:28:20.506 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6806000000000001e-05 2024-03-20T09:28:20.506 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.506 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.506 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:28:20.506 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.625623+0000", 2024-03-20T09:28:20.506 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.042809502999999999 2024-03-20T09:28:20.506 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.506 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.506 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:28:20.506 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.625675+0000", 2024-03-20T09:28:20.506 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1981000000000003e-05 2024-03-20T09:28:20.506 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.507 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:28:20.507 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.507 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.507 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.507 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26216.0:129146 6.b e749/709 6:d9973fcf:::benchmark_data_smithi122_168655_object9037:head v 749'8032, mlcod=748'8028)", 2024-03-20T09:28:20.507 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:56.362248+0000", 2024-03-20T09:28:20.507 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 144.06646086399999, 2024-03-20T09:28:20.507 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.26344784799999998, 2024-03-20T09:28:20.507 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:28:20.507 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:28:20.507 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:28:20.507 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.507 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:28:20.507 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.362248+0000", 2024-03-20T09:28:20.508 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.508 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.508 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.508 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:28:20.508 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.362248+0000", 2024-03-20T09:28:20.508 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.508 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.508 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.508 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:28:20.508 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.362248+0000", 2024-03-20T09:28:20.508 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6600000000000003e-07 2024-03-20T09:28:20.508 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.508 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.508 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:28:20.508 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.362255+0000", 2024-03-20T09:28:20.509 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.917e-06 2024-03-20T09:28:20.509 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.509 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.509 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:28:20.509 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.362282+0000", 2024-03-20T09:28:20.509 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6344999999999999e-05 2024-03-20T09:28:20.509 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.509 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.509 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:28:20.509 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.362298+0000", 2024-03-20T09:28:20.509 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6087999999999999e-05 2024-03-20T09:28:20.509 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.509 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.509 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:28:20.510 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.587169+0000", 2024-03-20T09:28:20.510 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.224871712 2024-03-20T09:28:20.510 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.510 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.510 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:28:20.510 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.587197+0000", 2024-03-20T09:28:20.510 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.7977e-05 2024-03-20T09:28:20.510 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.510 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.510 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:28:20.510 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.625680+0000", 2024-03-20T09:28:20.510 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.038482502000000002 2024-03-20T09:28:20.511 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.511 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.511 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:28:20.511 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.625696+0000", 2024-03-20T09:28:20.511 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6041000000000001e-05 2024-03-20T09:28:20.511 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.511 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:28:20.511 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.511 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.511 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.511 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26216.0:163049 6.b e798/709 6:df9d6e11:::benchmark_data_smithi122_168655_object42940:head v 798'10192, mlcod=798'10189)", 2024-03-20T09:28:20.511 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:27:59.230098+0000", 2024-03-20T09:28:20.511 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 21.198610491, 2024-03-20T09:28:20.512 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.012153363, 2024-03-20T09:28:20.512 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:28:20.512 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:28:20.512 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:28:20.512 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.512 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:28:20.512 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:27:59.230098+0000", 2024-03-20T09:28:20.512 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.512 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.512 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.512 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:28:20.512 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:27:59.230098+0000", 2024-03-20T09:28:20.513 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:28:20.513 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.513 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.513 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:28:20.513 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:27:59.230100+0000", 2024-03-20T09:28:20.513 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.088e-06 2024-03-20T09:28:20.513 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.513 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.513 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:28:20.513 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:27:59.230108+0000", 2024-03-20T09:28:20.513 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5520000000000005e-06 2024-03-20T09:28:20.514 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.514 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.514 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:28:20.514 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:27:59.230125+0000", 2024-03-20T09:28:20.514 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7397999999999998e-05 2024-03-20T09:28:20.514 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.514 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.514 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:28:20.514 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:27:59.230133+0000", 2024-03-20T09:28:20.514 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.6769999999999995e-06 2024-03-20T09:28:20.514 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.515 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.515 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:28:20.515 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:27:59.230189+0000", 2024-03-20T09:28:20.515 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5522999999999997e-05 2024-03-20T09:28:20.515 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.515 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.515 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:28:20.515 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:27:59.230212+0000", 2024-03-20T09:28:20.515 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3300999999999999e-05 2024-03-20T09:28:20.515 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.515 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.515 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:28:20.515 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:27:59.242201+0000", 2024-03-20T09:28:20.515 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.011989077000000001 2024-03-20T09:28:20.516 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:28:20.516 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:28:20.516 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:28:20.516 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:27:59.242252+0000", 2024-03-20T09:28:20.516 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0747e-05 2024-03-20T09:28:20.516 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.516 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:28:20.516 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.516 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:28:20.516 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:28:20.516 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:28:20.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:20.517 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:28:20.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:28:20.618 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:28:20.648 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:20.711 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:28:20.713 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:20 smithi115 ceph-mon[72476]: osdmap e812: 12 total, 11 up, 11 in 2024-03-20T09:28:20.713 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:20 smithi115 ceph-mon[72476]: pgmap v1964: 57 pgs: 3 remapped+peering, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 5 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 4.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 437 op/s; 11087/210984 objects degraded (5.255%); 8633/210984 objects misplaced (4.092%); 15 MiB/s, 237 objects/s recovering 2024-03-20T09:28:20.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:20.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:28:20.815 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:28:20.843 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:20 smithi026 ceph-mon[68309]: osdmap e812: 12 total, 11 up, 11 in 2024-03-20T09:28:20.843 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:20 smithi026 ceph-mon[68309]: pgmap v1964: 57 pgs: 3 remapped+peering, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 5 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 4.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 437 op/s; 11087/210984 objects degraded (5.255%); 8633/210984 objects misplaced (4.092%); 15 MiB/s, 237 objects/s recovering 2024-03-20T09:28:20.847 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:20.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:20 smithi122 ceph-mon[73495]: osdmap e812: 12 total, 11 up, 11 in 2024-03-20T09:28:20.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:20 smithi122 ceph-mon[73495]: pgmap v1964: 57 pgs: 3 remapped+peering, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+degraded, 5 active+undersized, 1 active+recovering+degraded+remapped, 44 active+clean; 4.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 437 op/s; 11087/210984 objects degraded (5.255%); 8633/210984 objects misplaced (4.092%); 15 MiB/s, 237 objects/s recovering 2024-03-20T09:28:21.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:21 smithi115 ceph-mon[72476]: osdmap e813: 12 total, 11 up, 11 in 2024-03-20T09:28:21.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:21 smithi115 ceph-mon[72476]: 6.d starting backfill to osd.0 from (0'0,0'0] MAX to 810'10654 2024-03-20T09:28:21.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:21 smithi122 ceph-mon[73495]: osdmap e813: 12 total, 11 up, 11 in 2024-03-20T09:28:21.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:21 smithi122 ceph-mon[73495]: 6.d starting backfill to osd.0 from (0'0,0'0] MAX to 810'10654 2024-03-20T09:28:21.966 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:21 smithi026 ceph-mon[68309]: osdmap e813: 12 total, 11 up, 11 in 2024-03-20T09:28:21.966 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:21 smithi026 ceph-mon[68309]: 6.d starting backfill to osd.0 from (0'0,0'0] MAX to 810'10654 2024-03-20T09:28:22.139 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_ops_in_flight 2024-03-20T09:28:22.420 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:28:22.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:22 smithi115 ceph-mon[72476]: osdmap e814: 12 total, 11 up, 11 in 2024-03-20T09:28:22.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:22 smithi115 ceph-mon[72476]: pgmap v1967: 57 pgs: 1 activating+undersized+degraded+remapped, 1 peering, 4 remapped+peering, 3 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+undersized, 44 active+clean; 4.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 306 op/s; 11040/209367 objects degraded (5.273%); 6048/209367 objects misplaced (2.889%); 2225/69789 objects unfound (3.188%); 15 MiB/s, 239 objects/s recovering 2024-03-20T09:28:22.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:22 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 11087/210984 objects degraded (5.255%), 5 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-03-20T09:28:22.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:22 smithi122 ceph-mon[73495]: osdmap e814: 12 total, 11 up, 11 in 2024-03-20T09:28:22.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:22 smithi122 ceph-mon[73495]: pgmap v1967: 57 pgs: 1 activating+undersized+degraded+remapped, 1 peering, 4 remapped+peering, 3 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+undersized, 44 active+clean; 4.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 306 op/s; 11040/209367 objects degraded (5.273%); 6048/209367 objects misplaced (2.889%); 2225/69789 objects unfound (3.188%); 15 MiB/s, 239 objects/s recovering 2024-03-20T09:28:22.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:22 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 11087/210984 objects degraded (5.255%), 5 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-03-20T09:28:22.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:22 smithi026 ceph-mon[68309]: osdmap e814: 12 total, 11 up, 11 in 2024-03-20T09:28:22.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:22 smithi026 ceph-mon[68309]: pgmap v1967: 57 pgs: 1 activating+undersized+degraded+remapped, 1 peering, 4 remapped+peering, 3 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+undersized, 44 active+clean; 4.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 306 op/s; 11040/209367 objects degraded (5.273%); 6048/209367 objects misplaced (2.889%); 2225/69789 objects unfound (3.188%); 15 MiB/s, 239 objects/s recovering 2024-03-20T09:28:22.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:22 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 11087/210984 objects degraded (5.255%), 5 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-03-20T09:28:23.842 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:23 smithi122 ceph-mon[73495]: 6.f starting backfill to osd.5 from (0'0,0'0] MAX to 810'10808 2024-03-20T09:28:23.842 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:23 smithi122 ceph-mon[73495]: Health check failed: 2225/69789 objects unfound (3.188%) (OBJECT_UNFOUND) 2024-03-20T09:28:23.843 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:23 smithi122 ceph-mon[73495]: osdmap e815: 12 total, 11 up, 11 in 2024-03-20T09:28:23.843 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:23 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:28:23.843 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:23 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:28:23.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:23 smithi115 ceph-mon[72476]: 6.f starting backfill to osd.5 from (0'0,0'0] MAX to 810'10808 2024-03-20T09:28:23.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:23 smithi115 ceph-mon[72476]: Health check failed: 2225/69789 objects unfound (3.188%) (OBJECT_UNFOUND) 2024-03-20T09:28:23.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:23 smithi115 ceph-mon[72476]: osdmap e815: 12 total, 11 up, 11 in 2024-03-20T09:28:23.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:23 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:28:23.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:23 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:28:23.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:23 smithi026 ceph-mon[68309]: 6.f starting backfill to osd.5 from (0'0,0'0] MAX to 810'10808 2024-03-20T09:28:23.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:23 smithi026 ceph-mon[68309]: Health check failed: 2225/69789 objects unfound (3.188%) (OBJECT_UNFOUND) 2024-03-20T09:28:23.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:23 smithi026 ceph-mon[68309]: osdmap e815: 12 total, 11 up, 11 in 2024-03-20T09:28:23.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:23 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:28:23.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:23 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:28:24.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:24 smithi115 ceph-mon[72476]: osdmap e816: 12 total, 11 up, 11 in 2024-03-20T09:28:24.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:24 smithi115 ceph-mon[72476]: pgmap v1970: 57 pgs: 1 activating+undersized+degraded+remapped, 1 peering, 4 remapped+peering, 3 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 45 active+clean; 4.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 136 op/s; 11040/209340 objects degraded (5.274%); 6048/209340 objects misplaced (2.889%); 2225/69780 objects unfound (3.189%); 3.5 MiB/s, 55 objects/s recovering 2024-03-20T09:28:24.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:24 smithi122 ceph-mon[73495]: osdmap e816: 12 total, 11 up, 11 in 2024-03-20T09:28:24.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:24 smithi122 ceph-mon[73495]: pgmap v1970: 57 pgs: 1 activating+undersized+degraded+remapped, 1 peering, 4 remapped+peering, 3 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 45 active+clean; 4.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 136 op/s; 11040/209340 objects degraded (5.274%); 6048/209340 objects misplaced (2.889%); 2225/69780 objects unfound (3.189%); 3.5 MiB/s, 55 objects/s recovering 2024-03-20T09:28:24.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:24 smithi026 ceph-mon[68309]: osdmap e816: 12 total, 11 up, 11 in 2024-03-20T09:28:24.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:24 smithi026 ceph-mon[68309]: pgmap v1970: 57 pgs: 1 activating+undersized+degraded+remapped, 1 peering, 4 remapped+peering, 3 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 45 active+clean; 4.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 136 op/s; 11040/209340 objects degraded (5.274%); 6048/209340 objects misplaced (2.889%); 2225/69780 objects unfound (3.189%); 3.5 MiB/s, 55 objects/s recovering 2024-03-20T09:28:25.524 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 8, 11, 3, 4, 9] out_osds: [10] dead_osds: [10] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 11, 5] 2024-03-20T09:28:25.525 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:28:25.525 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-20T09:28:25.525 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.0 'trim stale osdmaps' 2024-03-20T09:28:25.809 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:28:25.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:25.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:28:25.824 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:28:25.849 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:28:25.849 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:28:25.849 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-03-20T09:28:25.849 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:28:25.918 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:28:25.952 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:26.004 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:28:26.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:26.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:28:26.108 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:28:26.135 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:26.155 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:28:26.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:26.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:28:26.262 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:28:26.299 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:27.627 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:27 smithi026 ceph-mon[68309]: pgmap v1971: 57 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating+undersized+degraded+remapped, 1 peering, 1 remapped+peering, 2 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 48 active+clean; 4.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 83 op/s; 13427/209217 objects degraded (6.418%); 5533/209217 objects misplaced (2.645%); 2225/69739 objects unfound (3.190%); 29 MiB/s, 595 objects/s recovering 2024-03-20T09:28:27.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:27 smithi115 ceph-mon[72476]: pgmap v1971: 57 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating+undersized+degraded+remapped, 1 peering, 1 remapped+peering, 2 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 48 active+clean; 4.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 83 op/s; 13427/209217 objects degraded (6.418%); 5533/209217 objects misplaced (2.645%); 2225/69739 objects unfound (3.190%); 29 MiB/s, 595 objects/s recovering 2024-03-20T09:28:27.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:27 smithi122 ceph-mon[73495]: pgmap v1971: 57 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating+undersized+degraded+remapped, 1 peering, 1 remapped+peering, 2 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 48 active+clean; 4.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 83 op/s; 13427/209217 objects degraded (6.418%); 5533/209217 objects misplaced (2.645%); 2225/69739 objects unfound (3.190%); 29 MiB/s, 595 objects/s recovering 2024-03-20T09:28:27.763 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:28:27.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:27.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:28:27.869 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:28:27.933 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:28.097 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:28:28.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:28.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:28:28.202 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:28:28.236 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:28.437 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:28:28.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:28.440 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:28:28.541 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:28:28.572 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:28.595 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:28:28.596 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:28 smithi115 ceph-mon[72476]: osdmap e817: 12 total, 11 up, 11 in 2024-03-20T09:28:28.596 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:28 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 13427/209217 objects degraded (6.418%), 5 pgs degraded (PG_DEGRADED) 2024-03-20T09:28:28.597 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:28 smithi115 ceph-mon[72476]: osdmap e818: 12 total, 11 up, 11 in 2024-03-20T09:28:28.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:28.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:28:28.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:28 smithi122 ceph-mon[73495]: osdmap e817: 12 total, 11 up, 11 in 2024-03-20T09:28:28.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:28 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 13427/209217 objects degraded (6.418%), 5 pgs degraded (PG_DEGRADED) 2024-03-20T09:28:28.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:28 smithi122 ceph-mon[73495]: osdmap e818: 12 total, 11 up, 11 in 2024-03-20T09:28:28.698 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:28:28.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:28 smithi026 ceph-mon[68309]: osdmap e817: 12 total, 11 up, 11 in 2024-03-20T09:28:28.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:28 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 13427/209217 objects degraded (6.418%), 5 pgs degraded (PG_DEGRADED) 2024-03-20T09:28:28.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:28 smithi026 ceph-mon[68309]: osdmap e818: 12 total, 11 up, 11 in 2024-03-20T09:28:28.729 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:28.749 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:28:28.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:28.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:28:28.853 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:28:28.901 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:29.289 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_blocked_ops 2024-03-20T09:28:29.550 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:28:29.586 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:28:29.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:29 smithi115 ceph-mon[72476]: pgmap v1974: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+undersized+degraded+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 51 active+clean; 4.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 12 op/s; 8167/209148 objects degraded (3.905%); 11478/209148 objects misplaced (5.488%); 37 MiB/s, 718 objects/s recovering 2024-03-20T09:28:29.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:29 smithi115 ceph-mon[72476]: Health check cleared: OBJECT_UNFOUND (was: 2225/69739 objects unfound (3.190%)) 2024-03-20T09:28:29.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:29 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/937684559' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:28:29.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:29 smithi122 ceph-mon[73495]: pgmap v1974: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+undersized+degraded+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 51 active+clean; 4.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 12 op/s; 8167/209148 objects degraded (3.905%); 11478/209148 objects misplaced (5.488%); 37 MiB/s, 718 objects/s recovering 2024-03-20T09:28:29.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:29 smithi122 ceph-mon[73495]: Health check cleared: OBJECT_UNFOUND (was: 2225/69739 objects unfound (3.190%)) 2024-03-20T09:28:29.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:29 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/937684559' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:28:29.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:29 smithi026 ceph-mon[68309]: pgmap v1974: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+undersized+degraded+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 51 active+clean; 4.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 12 op/s; 8167/209148 objects degraded (3.905%); 11478/209148 objects misplaced (5.488%); 37 MiB/s, 718 objects/s recovering 2024-03-20T09:28:29.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:29 smithi026 ceph-mon[68309]: Health check cleared: OBJECT_UNFOUND (was: 2225/69739 objects unfound (3.190%)) 2024-03-20T09:28:29.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:29 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/937684559' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:28:30.606 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:30 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:28:30.607 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:30 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/937684559' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:28:30.607 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:30 smithi122 ceph-mon[73495]: osdmap e819: 12 total, 11 up, 11 in 2024-03-20T09:28:30.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:30 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:28:30.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:30 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/937684559' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:28:30.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:30 smithi115 ceph-mon[72476]: osdmap e819: 12 total, 11 up, 11 in 2024-03-20T09:28:30.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:30 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:28:30.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:30 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/937684559' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:28:30.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:30 smithi026 ceph-mon[68309]: osdmap e819: 12 total, 11 up, 11 in 2024-03-20T09:28:31.573 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:28:31.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:31.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:28:31.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:31 smithi115 ceph-mon[72476]: pgmap v1976: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+undersized+degraded+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 51 active+clean; 4.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 85 op/s; 6227/207801 objects degraded (2.997%); 11478/207801 objects misplaced (5.524%); 55 MiB/s, 1.00k objects/s recovering 2024-03-20T09:28:31.678 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:28:31.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:31 smithi122 ceph-mon[73495]: pgmap v1976: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+undersized+degraded+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 51 active+clean; 4.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 85 op/s; 6227/207801 objects degraded (2.997%); 11478/207801 objects misplaced (5.524%); 55 MiB/s, 1.00k objects/s recovering 2024-03-20T09:28:31.708 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:31.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:31 smithi026 ceph-mon[68309]: pgmap v1976: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+undersized+degraded+remapped+backfilling, 3 active+recovering+undersized+degraded+remapped, 51 active+clean; 4.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 85 op/s; 6227/207801 objects degraded (2.997%); 11478/207801 objects misplaced (5.524%); 55 MiB/s, 1.00k objects/s recovering 2024-03-20T09:28:31.762 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:28:31.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:31.766 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:28:31.772 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:28:31.867 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:28:31.897 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:31.916 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:28:31.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:31.960 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:28:32.061 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:28:32.087 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:32.108 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:28:32.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:32.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:28:32.216 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:28:32.258 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:32.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:32 smithi115 ceph-mon[72476]: osdmap e820: 12 total, 11 up, 11 in 2024-03-20T09:28:32.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:32 smithi115 ceph-mon[72476]: pgmap v1978: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+undersized+degraded+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 52 active+clean; 4.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 267 op/s; 5365/205227 objects degraded (2.614%); 10600/205227 objects misplaced (5.165%); 38 MiB/s, 736 objects/s recovering 2024-03-20T09:28:32.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:32 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 6227/207801 objects degraded (2.997%), 5 pgs degraded (PG_DEGRADED) 2024-03-20T09:28:32.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:32 smithi122 ceph-mon[73495]: osdmap e820: 12 total, 11 up, 11 in 2024-03-20T09:28:32.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:32 smithi122 ceph-mon[73495]: pgmap v1978: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+undersized+degraded+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 52 active+clean; 4.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 267 op/s; 5365/205227 objects degraded (2.614%); 10600/205227 objects misplaced (5.165%); 38 MiB/s, 736 objects/s recovering 2024-03-20T09:28:32.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:32 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 6227/207801 objects degraded (2.997%), 5 pgs degraded (PG_DEGRADED) 2024-03-20T09:28:32.967 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:32 smithi026 ceph-mon[68309]: osdmap e820: 12 total, 11 up, 11 in 2024-03-20T09:28:32.967 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:32 smithi026 ceph-mon[68309]: pgmap v1978: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+undersized+degraded+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 52 active+clean; 4.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 267 op/s; 5365/205227 objects degraded (2.614%); 10600/205227 objects misplaced (5.165%); 38 MiB/s, 736 objects/s recovering 2024-03-20T09:28:32.967 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:32 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 6227/207801 objects degraded (2.997%), 5 pgs degraded (PG_DEGRADED) 2024-03-20T09:28:33.695 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:33 smithi026 ceph-mon[68309]: 6.6 scrub starts 2024-03-20T09:28:33.695 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:33 smithi026 ceph-mon[68309]: osdmap e821: 12 total, 11 up, 11 in 2024-03-20T09:28:33.695 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:33 smithi026 ceph-mon[68309]: 6.6 scrub ok 2024-03-20T09:28:33.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:33 smithi115 ceph-mon[72476]: 6.6 scrub starts 2024-03-20T09:28:33.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:33 smithi115 ceph-mon[72476]: osdmap e821: 12 total, 11 up, 11 in 2024-03-20T09:28:33.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:33 smithi115 ceph-mon[72476]: 6.6 scrub ok 2024-03-20T09:28:33.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:33 smithi122 ceph-mon[73495]: 6.6 scrub starts 2024-03-20T09:28:33.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:33 smithi122 ceph-mon[73495]: osdmap e821: 12 total, 11 up, 11 in 2024-03-20T09:28:33.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:33 smithi122 ceph-mon[73495]: 6.6 scrub ok 2024-03-20T09:28:34.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:34 smithi115 ceph-mon[72476]: 1.d scrub starts 2024-03-20T09:28:34.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:34 smithi115 ceph-mon[72476]: 1.d scrub ok 2024-03-20T09:28:34.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:34 smithi115 ceph-mon[72476]: 6.13 scrub starts 2024-03-20T09:28:34.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:34 smithi115 ceph-mon[72476]: 6.13 scrub ok 2024-03-20T09:28:34.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:34 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:28:34.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:34 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:28:34.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:34 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:28:34.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:34 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:28:34.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:34 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:28:34.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:34 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:28:34.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:34 smithi115 ceph-mon[72476]: pgmap v1980: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+undersized+degraded+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 52 active+clean; 4.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 225 op/s; 5365/205095 objects degraded (2.616%); 10600/205095 objects misplaced (5.168%); 31 MiB/s, 607 objects/s recovering 2024-03-20T09:28:34.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:34 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:28:34.936 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:28:34.937 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:28:34.937 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-03-20T09:28:34.937 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-03-20T09:28:34.937 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:28:34.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:34 smithi122 ceph-mon[73495]: 1.d scrub starts 2024-03-20T09:28:34.942 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:34 smithi122 ceph-mon[73495]: 1.d scrub ok 2024-03-20T09:28:34.942 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:34 smithi122 ceph-mon[73495]: 6.13 scrub starts 2024-03-20T09:28:34.942 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:34 smithi122 ceph-mon[73495]: 6.13 scrub ok 2024-03-20T09:28:34.942 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:34 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:28:34.942 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:34 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:28:34.942 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:34 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:28:34.942 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:34 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:28:34.942 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:34 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:28:34.942 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:34 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:28:34.942 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:34 smithi122 ceph-mon[73495]: pgmap v1980: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+undersized+degraded+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 52 active+clean; 4.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 225 op/s; 5365/205095 objects degraded (2.616%); 10600/205095 objects misplaced (5.168%); 31 MiB/s, 607 objects/s recovering 2024-03-20T09:28:34.942 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:34 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:28:34.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:34 smithi026 ceph-mon[68309]: 1.d scrub starts 2024-03-20T09:28:34.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:34 smithi026 ceph-mon[68309]: 1.d scrub ok 2024-03-20T09:28:34.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:34 smithi026 ceph-mon[68309]: 6.13 scrub starts 2024-03-20T09:28:34.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:34 smithi026 ceph-mon[68309]: 6.13 scrub ok 2024-03-20T09:28:34.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:34 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:28:34.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:34 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:28:34.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:34 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:28:34.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:34 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:28:34.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:34 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:28:34.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:34 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:28:34.966 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:34 smithi026 ceph-mon[68309]: pgmap v1980: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+undersized+degraded+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 52 active+clean; 4.2 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 225 op/s; 5365/205095 objects degraded (2.616%); 10600/205095 objects misplaced (5.168%); 31 MiB/s, 607 objects/s recovering 2024-03-20T09:28:34.966 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:34 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:28:35.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:35 smithi122 ceph-mon[73495]: 6.2 scrub starts 2024-03-20T09:28:35.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:35 smithi122 ceph-mon[73495]: 6.11 scrub starts 2024-03-20T09:28:35.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:35 smithi122 ceph-mon[73495]: 6.2 scrub ok 2024-03-20T09:28:35.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:35 smithi122 ceph-mon[73495]: 6.11 scrub ok 2024-03-20T09:28:35.798 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:35 smithi026 ceph-mon[68309]: 6.2 scrub starts 2024-03-20T09:28:35.799 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:35 smithi026 ceph-mon[68309]: 6.11 scrub starts 2024-03-20T09:28:35.799 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:35 smithi026 ceph-mon[68309]: 6.2 scrub ok 2024-03-20T09:28:35.799 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:35 smithi026 ceph-mon[68309]: 6.11 scrub ok 2024-03-20T09:28:35.861 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_historic_ops 2024-03-20T09:28:35.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:35 smithi115 ceph-mon[72476]: 6.2 scrub starts 2024-03-20T09:28:35.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:35 smithi115 ceph-mon[72476]: 6.11 scrub starts 2024-03-20T09:28:35.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:35 smithi115 ceph-mon[72476]: 6.2 scrub ok 2024-03-20T09:28:35.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:35 smithi115 ceph-mon[72476]: 6.11 scrub ok 2024-03-20T09:28:36.156 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:28:36.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:36 smithi122 ceph-mon[73495]: pgmap v1981: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 53 active+clean; 4.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 4918/202422 objects degraded (2.430%); 10600/202422 objects misplaced (5.237%) 2024-03-20T09:28:36.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:36 smithi115 ceph-mon[72476]: pgmap v1981: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 53 active+clean; 4.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 4918/202422 objects degraded (2.430%); 10600/202422 objects misplaced (5.237%) 2024-03-20T09:28:36.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:36 smithi026 ceph-mon[68309]: pgmap v1981: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 2 active+recovering+undersized+degraded+remapped, 53 active+clean; 4.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 4918/202422 objects degraded (2.430%); 10600/202422 objects misplaced (5.237%) 2024-03-20T09:28:37.483 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:28:37.483 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:28:37.483 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:28:37.503 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:28:37.532 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:28:37.581 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:28:37.581 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:28:37.612 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:28:37.635 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:28:37.659 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:28:37.694 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:28:37.695 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:28:37.695 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:28:37.695 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:28:37.695 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:28:37.695 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:28:37.695 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:28:37.695 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:28:37.695 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:28:37.695 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:28:37.695 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:28:37.695 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-03-20T09:28:37.696 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:28:37.825 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:37 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3316933003' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:28:37.825 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:37 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 4918/202422 objects degraded (2.430%), 3 pgs degraded (PG_DEGRADED) 2024-03-20T09:28:37.826 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:37 smithi122 ceph-mon[73495]: osdmap e822: 12 total, 11 up, 11 in 2024-03-20T09:28:37.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:37 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3316933003' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:28:37.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:37 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 4918/202422 objects degraded (2.430%), 3 pgs degraded (PG_DEGRADED) 2024-03-20T09:28:37.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:37 smithi115 ceph-mon[72476]: osdmap e822: 12 total, 11 up, 11 in 2024-03-20T09:28:37.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:37 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3316933003' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:28:37.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:37 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 4918/202422 objects degraded (2.430%), 3 pgs degraded (PG_DEGRADED) 2024-03-20T09:28:37.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:37 smithi026 ceph-mon[68309]: osdmap e822: 12 total, 11 up, 11 in 2024-03-20T09:28:38.222 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:28:38.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:38.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:28:38.327 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:28:38.351 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:28:38.351 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:28:38.369 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:38.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:38 smithi122 ceph-mon[73495]: 1.c scrub starts 2024-03-20T09:28:38.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:38 smithi122 ceph-mon[73495]: 1.c scrub ok 2024-03-20T09:28:38.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:38 smithi122 ceph-mon[73495]: pgmap v1983: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 2 active+recovering+undersized+remapped, 53 active+clean; 4.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 4238/200493 objects degraded (2.114%); 9562/200493 objects misplaced (4.769%) 2024-03-20T09:28:38.813 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:38 smithi026 ceph-mon[68309]: 1.c scrub starts 2024-03-20T09:28:38.813 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:38 smithi026 ceph-mon[68309]: 1.c scrub ok 2024-03-20T09:28:38.813 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:38 smithi026 ceph-mon[68309]: pgmap v1983: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 2 active+recovering+undersized+remapped, 53 active+clean; 4.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 4238/200493 objects degraded (2.114%); 9562/200493 objects misplaced (4.769%) 2024-03-20T09:28:38.814 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:28:38.814 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:28:38.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:38 smithi115 ceph-mon[72476]: 1.c scrub starts 2024-03-20T09:28:38.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:38 smithi115 ceph-mon[72476]: 1.c scrub ok 2024-03-20T09:28:38.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:38 smithi115 ceph-mon[72476]: pgmap v1983: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 2 active+recovering+undersized+remapped, 53 active+clean; 4.1 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 4238/200493 objects degraded (2.114%); 9562/200493 objects misplaced (4.769%) 2024-03-20T09:28:39.162 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_ops_in_flight 2024-03-20T09:28:39.250 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:28:39.253 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:39.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:28:39.354 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:28:39.396 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:39.438 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:28:39.449 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:28:39.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:39.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:28:39.559 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:28:39.612 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:39.749 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:28:39.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:39.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:28:39.855 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:28:39.879 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:39 smithi115 ceph-mon[72476]: 1.15 scrub starts 2024-03-20T09:28:39.879 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:39 smithi115 ceph-mon[72476]: 1.15 scrub ok 2024-03-20T09:28:39.881 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:39.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:39 smithi122 ceph-mon[73495]: 1.15 scrub starts 2024-03-20T09:28:39.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:39 smithi122 ceph-mon[73495]: 1.15 scrub ok 2024-03-20T09:28:39.949 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:28:39.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:39.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:28:39.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:39 smithi026 ceph-mon[68309]: 1.15 scrub starts 2024-03-20T09:28:39.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:39 smithi026 ceph-mon[68309]: 1.15 scrub ok 2024-03-20T09:28:40.053 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:28:40.067 INFO:teuthology.orchestra.run.smithi026.stderr:no valid command found; 10 closest matches: 2024-03-20T09:28:40.067 INFO:teuthology.orchestra.run.smithi026.stderr:0 2024-03-20T09:28:40.067 INFO:teuthology.orchestra.run.smithi026.stderr:1 2024-03-20T09:28:40.067 INFO:teuthology.orchestra.run.smithi026.stderr:2 2024-03-20T09:28:40.067 INFO:teuthology.orchestra.run.smithi026.stderr:abort 2024-03-20T09:28:40.067 INFO:teuthology.orchestra.run.smithi026.stderr:assert 2024-03-20T09:28:40.067 INFO:teuthology.orchestra.run.smithi026.stderr:bench [] [] [] [] 2024-03-20T09:28:40.067 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs debug_inject_read_zeros 2024-03-20T09:28:40.067 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs files list 2024-03-20T09:28:40.067 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs stats 2024-03-20T09:28:40.068 INFO:teuthology.orchestra.run.smithi026.stderr:bluestore allocator dump block 2024-03-20T09:28:40.068 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: invalid command 2024-03-20T09:28:40.095 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:40.223 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:28:40.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:40 smithi122 ceph-mon[73495]: 1.17 scrub starts 2024-03-20T09:28:40.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:40 smithi122 ceph-mon[73495]: 6.c scrub starts 2024-03-20T09:28:40.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:40 smithi122 ceph-mon[73495]: 1.17 scrub ok 2024-03-20T09:28:40.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:40 smithi122 ceph-mon[73495]: osdmap e823: 12 total, 11 up, 11 in 2024-03-20T09:28:40.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:40 smithi122 ceph-mon[73495]: 6.c scrub ok 2024-03-20T09:28:40.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:40 smithi122 ceph-mon[73495]: 1.7 scrub starts 2024-03-20T09:28:40.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:40 smithi122 ceph-mon[73495]: pgmap v1985: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 2 active+recovering+undersized+remapped, 53 active+clean; 4.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 4238/198564 objects degraded (2.134%); 9562/198564 objects misplaced (4.816%) 2024-03-20T09:28:40.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:40 smithi122 ceph-mon[73495]: 1.7 scrub ok 2024-03-20T09:28:40.815 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:28:40.852 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:28:40.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:40.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:28:40.857 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:40 smithi026 ceph-mon[68309]: 1.17 scrub starts 2024-03-20T09:28:40.858 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:40 smithi026 ceph-mon[68309]: 6.c scrub starts 2024-03-20T09:28:40.858 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:40 smithi026 ceph-mon[68309]: 1.17 scrub ok 2024-03-20T09:28:40.858 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:40 smithi026 ceph-mon[68309]: osdmap e823: 12 total, 11 up, 11 in 2024-03-20T09:28:40.858 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:40 smithi026 ceph-mon[68309]: 6.c scrub ok 2024-03-20T09:28:40.858 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:40 smithi026 ceph-mon[68309]: 1.7 scrub starts 2024-03-20T09:28:40.858 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:40 smithi026 ceph-mon[68309]: pgmap v1985: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 2 active+recovering+undersized+remapped, 53 active+clean; 4.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 4238/198564 objects degraded (2.134%); 9562/198564 objects misplaced (4.816%) 2024-03-20T09:28:40.858 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:40 smithi026 ceph-mon[68309]: 1.7 scrub ok 2024-03-20T09:28:40.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:40 smithi115 ceph-mon[72476]: 1.17 scrub starts 2024-03-20T09:28:40.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:40 smithi115 ceph-mon[72476]: 6.c scrub starts 2024-03-20T09:28:40.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:40 smithi115 ceph-mon[72476]: 1.17 scrub ok 2024-03-20T09:28:40.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:40 smithi115 ceph-mon[72476]: osdmap e823: 12 total, 11 up, 11 in 2024-03-20T09:28:40.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:40 smithi115 ceph-mon[72476]: 6.c scrub ok 2024-03-20T09:28:40.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:40 smithi115 ceph-mon[72476]: 1.7 scrub starts 2024-03-20T09:28:40.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:40 smithi115 ceph-mon[72476]: pgmap v1985: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 2 active+recovering+undersized+remapped, 53 active+clean; 4.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 4238/198564 objects degraded (2.134%); 9562/198564 objects misplaced (4.816%) 2024-03-20T09:28:40.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:40 smithi115 ceph-mon[72476]: 1.7 scrub ok 2024-03-20T09:28:40.958 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:28:41.063 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:41.365 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:28:41.366 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:28:41.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:41 smithi122 ceph-mon[73495]: 6.8 scrub starts 2024-03-20T09:28:41.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:41 smithi122 ceph-mon[73495]: 1.e scrub starts 2024-03-20T09:28:41.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:41 smithi122 ceph-mon[73495]: 6.b scrub starts 2024-03-20T09:28:41.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:41 smithi122 ceph-mon[73495]: 1.e scrub ok 2024-03-20T09:28:41.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:41 smithi122 ceph-mon[73495]: 6.8 scrub ok 2024-03-20T09:28:41.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:41 smithi122 ceph-mon[73495]: osdmap e824: 12 total, 11 up, 11 in 2024-03-20T09:28:41.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:41 smithi122 ceph-mon[73495]: 6.15 scrub starts 2024-03-20T09:28:41.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:41 smithi122 ceph-mon[73495]: 6.15 scrub ok 2024-03-20T09:28:41.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:41 smithi115 ceph-mon[72476]: 6.8 scrub starts 2024-03-20T09:28:41.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:41 smithi115 ceph-mon[72476]: 1.e scrub starts 2024-03-20T09:28:41.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:41 smithi115 ceph-mon[72476]: 6.b scrub starts 2024-03-20T09:28:41.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:41 smithi115 ceph-mon[72476]: 1.e scrub ok 2024-03-20T09:28:41.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:41 smithi115 ceph-mon[72476]: 6.8 scrub ok 2024-03-20T09:28:41.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:41 smithi115 ceph-mon[72476]: osdmap e824: 12 total, 11 up, 11 in 2024-03-20T09:28:41.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:41 smithi115 ceph-mon[72476]: 6.15 scrub starts 2024-03-20T09:28:41.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:41 smithi115 ceph-mon[72476]: 6.15 scrub ok 2024-03-20T09:28:41.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:41 smithi026 ceph-mon[68309]: 6.8 scrub starts 2024-03-20T09:28:41.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:41 smithi026 ceph-mon[68309]: 1.e scrub starts 2024-03-20T09:28:41.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:41 smithi026 ceph-mon[68309]: 6.b scrub starts 2024-03-20T09:28:41.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:41 smithi026 ceph-mon[68309]: 1.e scrub ok 2024-03-20T09:28:41.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:41 smithi026 ceph-mon[68309]: 6.8 scrub ok 2024-03-20T09:28:41.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:41 smithi026 ceph-mon[68309]: osdmap e824: 12 total, 11 up, 11 in 2024-03-20T09:28:41.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:41 smithi026 ceph-mon[68309]: 6.15 scrub starts 2024-03-20T09:28:41.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:41 smithi026 ceph-mon[68309]: 6.15 scrub ok 2024-03-20T09:28:42.127 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_blocked_ops 2024-03-20T09:28:42.393 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:28:42.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:42 smithi115 ceph-mon[72476]: 6.1 scrub starts 2024-03-20T09:28:42.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:42 smithi115 ceph-mon[72476]: 6.e scrub starts 2024-03-20T09:28:42.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:42 smithi115 ceph-mon[72476]: 6.1 scrub ok 2024-03-20T09:28:42.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:42 smithi115 ceph-mon[72476]: 6.b scrub ok 2024-03-20T09:28:42.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:42 smithi115 ceph-mon[72476]: pgmap v1987: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfilling, 53 active+clean; 4.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 4194/196035 objects degraded (2.139%); 8458/196035 objects misplaced (4.315%) 2024-03-20T09:28:42.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:42 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 4238/198564 objects degraded (2.134%), 2 pgs degraded (PG_DEGRADED) 2024-03-20T09:28:42.907 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:42 smithi122 ceph-mon[73495]: 6.1 scrub starts 2024-03-20T09:28:42.907 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:42 smithi122 ceph-mon[73495]: 6.e scrub starts 2024-03-20T09:28:42.907 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:42 smithi122 ceph-mon[73495]: 6.1 scrub ok 2024-03-20T09:28:42.907 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:42 smithi122 ceph-mon[73495]: 6.b scrub ok 2024-03-20T09:28:42.907 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:42 smithi122 ceph-mon[73495]: pgmap v1987: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfilling, 53 active+clean; 4.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 4194/196035 objects degraded (2.139%); 8458/196035 objects misplaced (4.315%) 2024-03-20T09:28:42.907 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:42 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 4238/198564 objects degraded (2.134%), 2 pgs degraded (PG_DEGRADED) 2024-03-20T09:28:42.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:42 smithi026 ceph-mon[68309]: 6.1 scrub starts 2024-03-20T09:28:42.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:42 smithi026 ceph-mon[68309]: 6.e scrub starts 2024-03-20T09:28:42.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:42 smithi026 ceph-mon[68309]: 6.1 scrub ok 2024-03-20T09:28:42.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:42 smithi026 ceph-mon[68309]: 6.b scrub ok 2024-03-20T09:28:42.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:42 smithi026 ceph-mon[68309]: pgmap v1987: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfilling, 53 active+clean; 4.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 4194/196035 objects degraded (2.139%); 8458/196035 objects misplaced (4.315%) 2024-03-20T09:28:42.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:42 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 4238/198564 objects degraded (2.134%), 2 pgs degraded (PG_DEGRADED) 2024-03-20T09:28:43.456 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:28:43.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:43.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:28:43.484 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T09:28:43.484 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-20T09:28:43.485 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.1 'trim stale osdmaps' 2024-03-20T09:28:43.520 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:28:43.572 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:28:43.575 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:28:43.599 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:43.629 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:28:43.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:43.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:28:43.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:43 smithi026 ceph-mon[68309]: 6.e scrub ok 2024-03-20T09:28:43.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:43 smithi026 ceph-mon[68309]: 1.1c scrub starts 2024-03-20T09:28:43.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:43 smithi026 ceph-mon[68309]: 1.1c scrub ok 2024-03-20T09:28:43.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:43 smithi026 ceph-mon[68309]: 6.12 scrub starts 2024-03-20T09:28:43.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:43 smithi026 ceph-mon[68309]: 6.12 scrub ok 2024-03-20T09:28:43.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:43 smithi026 ceph-mon[68309]: 1.f scrub starts 2024-03-20T09:28:43.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:43 smithi026 ceph-mon[68309]: 1.f scrub ok 2024-03-20T09:28:43.732 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:28:43.758 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:43.778 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:28:43.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:43.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:28:43.882 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:28:43.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:43 smithi115 ceph-mon[72476]: 6.e scrub ok 2024-03-20T09:28:43.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:43 smithi115 ceph-mon[72476]: 1.1c scrub starts 2024-03-20T09:28:43.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:43 smithi115 ceph-mon[72476]: 1.1c scrub ok 2024-03-20T09:28:43.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:43 smithi115 ceph-mon[72476]: 6.12 scrub starts 2024-03-20T09:28:43.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:43 smithi115 ceph-mon[72476]: 6.12 scrub ok 2024-03-20T09:28:43.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:43 smithi115 ceph-mon[72476]: 1.f scrub starts 2024-03-20T09:28:43.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:43 smithi115 ceph-mon[72476]: 1.f scrub ok 2024-03-20T09:28:43.909 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:43 smithi122 ceph-mon[73495]: 6.e scrub ok 2024-03-20T09:28:43.909 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:43 smithi122 ceph-mon[73495]: 1.1c scrub starts 2024-03-20T09:28:43.909 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:43 smithi122 ceph-mon[73495]: 1.1c scrub ok 2024-03-20T09:28:43.909 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:43 smithi122 ceph-mon[73495]: 6.12 scrub starts 2024-03-20T09:28:43.909 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:43 smithi122 ceph-mon[73495]: 6.12 scrub ok 2024-03-20T09:28:43.909 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:43 smithi122 ceph-mon[73495]: 1.f scrub starts 2024-03-20T09:28:43.909 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:43 smithi122 ceph-mon[73495]: 1.f scrub ok 2024-03-20T09:28:43.912 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:44.060 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:28:44.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:44.063 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:28:44.164 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:28:44.195 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:44.220 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:28:44.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:44.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:28:44.326 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:28:44.355 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:44.380 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:28:44.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:44.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:28:44.487 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:28:44.498 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:28:44.498 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:28:44.538 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:44.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:44 smithi115 ceph-mon[72476]: 6.5 scrub starts 2024-03-20T09:28:44.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:44 smithi115 ceph-mon[72476]: pgmap v1988: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfilling, 53 active+clean; 4.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 4194/195789 objects degraded (2.142%); 8458/195789 objects misplaced (4.320%) 2024-03-20T09:28:44.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:44 smithi115 ceph-mon[72476]: 6.5 scrub ok 2024-03-20T09:28:44.901 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:44 smithi122 ceph-mon[73495]: 6.5 scrub starts 2024-03-20T09:28:44.901 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:44 smithi122 ceph-mon[73495]: pgmap v1988: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfilling, 53 active+clean; 4.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 4194/195789 objects degraded (2.142%); 8458/195789 objects misplaced (4.320%) 2024-03-20T09:28:44.901 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:44 smithi122 ceph-mon[73495]: 6.5 scrub ok 2024-03-20T09:28:44.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:44 smithi026 ceph-mon[68309]: 6.5 scrub starts 2024-03-20T09:28:44.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:44 smithi026 ceph-mon[68309]: pgmap v1988: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfilling, 53 active+clean; 4.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 4194/195789 objects degraded (2.142%); 8458/195789 objects misplaced (4.320%) 2024-03-20T09:28:44.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:44 smithi026 ceph-mon[68309]: 6.5 scrub ok 2024-03-20T09:28:45.329 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_historic_ops 2024-03-20T09:28:45.606 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:28:45.877 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:45 smithi122 ceph-mon[73495]: 1.1 scrub starts 2024-03-20T09:28:45.877 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:45 smithi122 ceph-mon[73495]: 1.1 scrub ok 2024-03-20T09:28:45.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:45 smithi115 ceph-mon[72476]: 1.1 scrub starts 2024-03-20T09:28:45.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:45 smithi115 ceph-mon[72476]: 1.1 scrub ok 2024-03-20T09:28:45.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:45 smithi026 ceph-mon[68309]: 1.1 scrub starts 2024-03-20T09:28:45.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:45 smithi026 ceph-mon[68309]: 1.1 scrub ok 2024-03-20T09:28:46.092 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:28:46.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:46.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:28:46.198 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:28:46.227 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:46.877 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:28:46.882 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:46 smithi122 ceph-mon[73495]: 1.1a scrub starts 2024-03-20T09:28:46.882 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:46 smithi122 ceph-mon[73495]: 1.1a scrub ok 2024-03-20T09:28:46.882 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:46 smithi122 ceph-mon[73495]: pgmap v1989: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 54 active+clean; 3.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 4194/192780 objects degraded (2.176%); 8458/192780 objects misplaced (4.387%) 2024-03-20T09:28:46.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:46.883 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:28:46.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:46 smithi115 ceph-mon[72476]: 1.1a scrub starts 2024-03-20T09:28:46.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:46 smithi115 ceph-mon[72476]: 1.1a scrub ok 2024-03-20T09:28:46.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:46 smithi115 ceph-mon[72476]: pgmap v1989: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 54 active+clean; 3.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 4194/192780 objects degraded (2.176%); 8458/192780 objects misplaced (4.387%) 2024-03-20T09:28:46.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:46 smithi026 ceph-mon[68309]: 1.1a scrub starts 2024-03-20T09:28:46.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:46 smithi026 ceph-mon[68309]: 1.1a scrub ok 2024-03-20T09:28:46.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:46 smithi026 ceph-mon[68309]: pgmap v1989: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 54 active+clean; 3.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 4194/192780 objects degraded (2.176%); 8458/192780 objects misplaced (4.387%) 2024-03-20T09:28:46.984 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:28:47.012 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:47.209 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:28:47.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:47.217 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:28:47.318 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:28:47.346 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:47.496 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:28:47.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:47.499 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:28:47.587 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:28:47.587 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:28:47.600 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:28:47.635 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:47.678 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:28:47.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:47.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:28:47.793 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:28:47.821 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:47 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 4194/192780 objects degraded (2.176%), 2 pgs degraded (PG_DEGRADED) 2024-03-20T09:28:47.823 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:47.851 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:28:47.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:47.854 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:28:47.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:47 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 4194/192780 objects degraded (2.176%), 2 pgs degraded (PG_DEGRADED) 2024-03-20T09:28:47.954 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:28:47.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:47 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 4194/192780 objects degraded (2.176%), 2 pgs degraded (PG_DEGRADED) 2024-03-20T09:28:47.983 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:48.004 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:28:48.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:48.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:28:48.035 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:28:48.112 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:28:48.141 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:48.163 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:28:48.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:48.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:28:48.338 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:28:48.365 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:48.386 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:28:48.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:48.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:28:48.479 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_ops_in_flight 2024-03-20T09:28:48.492 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:28:48.521 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:48.542 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:28:48.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:48.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:28:48.646 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:28:48.692 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:48.753 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:28:48.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:48 smithi115 ceph-mon[72476]: 1.1b scrub starts 2024-03-20T09:28:48.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:48 smithi115 ceph-mon[72476]: 1.1b scrub ok 2024-03-20T09:28:48.902 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:48 smithi115 ceph-mon[72476]: pgmap v1990: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 54 active+clean; 3.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 4169/190725 objects degraded (2.186%); 7135/190725 objects misplaced (3.741%) 2024-03-20T09:28:48.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:48 smithi122 ceph-mon[73495]: 1.1b scrub starts 2024-03-20T09:28:48.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:48 smithi122 ceph-mon[73495]: 1.1b scrub ok 2024-03-20T09:28:48.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:48 smithi122 ceph-mon[73495]: pgmap v1990: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 54 active+clean; 3.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 4169/190725 objects degraded (2.186%); 7135/190725 objects misplaced (3.741%) 2024-03-20T09:28:48.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:48 smithi026 ceph-mon[68309]: 1.1b scrub starts 2024-03-20T09:28:48.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:48 smithi026 ceph-mon[68309]: 1.1b scrub ok 2024-03-20T09:28:48.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:48 smithi026 ceph-mon[68309]: pgmap v1990: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 54 active+clean; 3.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 4169/190725 objects degraded (2.186%); 7135/190725 objects misplaced (3.741%) 2024-03-20T09:28:50.063 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:28:50.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:50.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:28:50.171 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:28:50.201 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:50.227 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:28:50.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:50.230 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:28:50.331 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:28:50.358 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:50.431 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:28:50.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:50.434 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:28:50.535 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:28:50.575 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:50.601 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:28:50.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:50.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:28:50.706 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:28:50.716 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:28:50.717 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:28:50.733 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:50.754 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:28:50.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:50.757 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:28:50.858 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:28:50.886 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:50.958 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:28:50.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:50.961 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:28:51.062 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:28:51.089 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:51.108 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:28:51.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:51.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:28:51.212 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:28:51.242 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:51.539 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_blocked_ops 2024-03-20T09:28:51.589 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:28:51.591 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:51 smithi122 ceph-mon[73495]: 1.a scrub starts 2024-03-20T09:28:51.591 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:51 smithi122 ceph-mon[73495]: 1.a scrub ok 2024-03-20T09:28:51.591 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:51 smithi122 ceph-mon[73495]: pgmap v1991: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 54 active+clean; 3.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 4169/188685 objects degraded (2.210%); 7135/188685 objects misplaced (3.781%) 2024-03-20T09:28:51.592 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:51 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:28:51.592 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:51 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:28:51.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:51.593 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:28:51.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:51 smithi115 ceph-mon[72476]: 1.a scrub starts 2024-03-20T09:28:51.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:51 smithi115 ceph-mon[72476]: 1.a scrub ok 2024-03-20T09:28:51.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:51 smithi115 ceph-mon[72476]: pgmap v1991: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 54 active+clean; 3.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 4169/188685 objects degraded (2.210%); 7135/188685 objects misplaced (3.781%) 2024-03-20T09:28:51.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:51 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:28:51.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:51 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:28:51.694 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:28:51.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:51 smithi026 ceph-mon[68309]: 1.a scrub starts 2024-03-20T09:28:51.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:51 smithi026 ceph-mon[68309]: 1.a scrub ok 2024-03-20T09:28:51.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:51 smithi026 ceph-mon[68309]: pgmap v1991: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 54 active+clean; 3.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 4169/188685 objects degraded (2.210%); 7135/188685 objects misplaced (3.781%) 2024-03-20T09:28:51.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:51 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:28:51.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:51 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:28:51.721 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:51.740 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:28:51.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:51.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:28:51.844 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:28:51.851 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:28:51.878 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:51.898 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:28:51.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:51.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:28:52.005 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:28:52.033 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:52.107 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:28:52.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:52.112 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:28:52.213 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:28:52.243 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:52.341 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:28:52.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:52.344 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:28:52.445 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:28:52.471 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:52.492 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:28:52.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:52.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:28:52.595 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:28:52.620 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:52 smithi115 ceph-mon[72476]: osdmap e825: 12 total, 11 up, 11 in 2024-03-20T09:28:52.621 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:28:52.623 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:52.649 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:28:52.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:52.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:28:52.657 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:28:52.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:52 smithi122 ceph-mon[73495]: osdmap e825: 12 total, 11 up, 11 in 2024-03-20T09:28:52.704 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:28:52.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:52 smithi026 ceph-mon[68309]: osdmap e825: 12 total, 11 up, 11 in 2024-03-20T09:28:52.734 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:28:52.757 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:28:52.761 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:28:52.785 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:52.792 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:28:52.806 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:28:52.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:52.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:28:52.827 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:28:52.853 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:28:52.877 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:28:52.909 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:28:52.910 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:28:52.935 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:28:52.936 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:28:52.936 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:28:52.936 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:28:52.936 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:28:52.936 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:28:52.936 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:28:52.936 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:28:52.936 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:28:52.936 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:28:52.936 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:28:52.936 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-03-20T09:28:52.936 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:28:52.938 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:53.321 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:28:53.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:53.324 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:28:53.425 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:28:53.453 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:53.473 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:28:53.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:53.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:28:53.521 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:28:53.575 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:53 smithi122 ceph-mon[73495]: pgmap v1993: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 3.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 4104/186156 objects degraded (2.205%); 6269/186156 objects misplaced (3.368%) 2024-03-20T09:28:53.575 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:53 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 4104/186156 objects degraded (2.205%), 2 pgs degraded (PG_DEGRADED) 2024-03-20T09:28:53.575 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:53 smithi122 ceph-mon[73495]: osdmap e826: 12 total, 11 up, 11 in 2024-03-20T09:28:53.575 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:53 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/805222690' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:28:53.575 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:53 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1923687456' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:28:53.575 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:53 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:28:53.577 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:28:53.605 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:53.623 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:28:53.625 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:53 smithi115 ceph-mon[72476]: pgmap v1993: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 3.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 4104/186156 objects degraded (2.205%); 6269/186156 objects misplaced (3.368%) 2024-03-20T09:28:53.625 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:53 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 4104/186156 objects degraded (2.205%), 2 pgs degraded (PG_DEGRADED) 2024-03-20T09:28:53.625 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:53 smithi115 ceph-mon[72476]: osdmap e826: 12 total, 11 up, 11 in 2024-03-20T09:28:53.625 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:53 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/805222690' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:28:53.625 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:53 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1923687456' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:28:53.625 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:53 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:28:53.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:53.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:28:53.687 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:53 smithi026 ceph-mon[68309]: pgmap v1993: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 3.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 4104/186156 objects degraded (2.205%); 6269/186156 objects misplaced (3.368%) 2024-03-20T09:28:53.687 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:53 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 4104/186156 objects degraded (2.205%), 2 pgs degraded (PG_DEGRADED) 2024-03-20T09:28:53.687 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:53 smithi026 ceph-mon[68309]: osdmap e826: 12 total, 11 up, 11 in 2024-03-20T09:28:53.687 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:53 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/805222690' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:28:53.687 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:53 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1923687456' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:28:53.687 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:53 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:28:53.727 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:28:53.758 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:53.781 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:28:53.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:53.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:28:53.838 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:28:53.838 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:28:53.838 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-03-20T09:28:53.838 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-03-20T09:28:53.838 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:28:53.885 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:28:53.913 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:53.937 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:28:53.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:53.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:28:54.041 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:28:54.046 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:28:54.047 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:28:54.069 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:54.088 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:28:54.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:54.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:28:54.192 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:28:54.244 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:54.365 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:54 smithi026 ceph-mon[68309]: 1.6 scrub starts 2024-03-20T09:28:54.365 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:54 smithi026 ceph-mon[68309]: 1.6 scrub ok 2024-03-20T09:28:54.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:54 smithi115 ceph-mon[72476]: 1.6 scrub starts 2024-03-20T09:28:54.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:54 smithi115 ceph-mon[72476]: 1.6 scrub ok 2024-03-20T09:28:54.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:54 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:28:54.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:54 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:28:54.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:54 smithi115 ceph-mon[72476]: osdmap e827: 12 total, 11 up, 11 in 2024-03-20T09:28:54.664 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:54 smithi122 ceph-mon[73495]: 1.6 scrub starts 2024-03-20T09:28:54.665 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:54 smithi122 ceph-mon[73495]: 1.6 scrub ok 2024-03-20T09:28:54.665 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:54 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:28:54.665 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:54 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:28:54.665 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:54 smithi122 ceph-mon[73495]: osdmap e827: 12 total, 11 up, 11 in 2024-03-20T09:28:54.691 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_historic_ops 2024-03-20T09:28:54.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:54 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:28:54.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:54 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:28:54.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:54 smithi026 ceph-mon[68309]: osdmap e827: 12 total, 11 up, 11 in 2024-03-20T09:28:54.971 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:28:55.488 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:55 smithi122 ceph-mon[73495]: 1.9 scrub starts 2024-03-20T09:28:55.488 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:55 smithi122 ceph-mon[73495]: 1.9 scrub ok 2024-03-20T09:28:55.488 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:55 smithi122 ceph-mon[73495]: pgmap v1996: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 3.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 261 op/s; 4104/186018 objects degraded (2.206%); 6269/186018 objects misplaced (3.370%); 3.1 MiB/s, 102 objects/s recovering 2024-03-20T09:28:55.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:55 smithi115 ceph-mon[72476]: 1.9 scrub starts 2024-03-20T09:28:55.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:55 smithi115 ceph-mon[72476]: 1.9 scrub ok 2024-03-20T09:28:55.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:55 smithi115 ceph-mon[72476]: pgmap v1996: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 3.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 261 op/s; 4104/186018 objects degraded (2.206%); 6269/186018 objects misplaced (3.370%); 3.1 MiB/s, 102 objects/s recovering 2024-03-20T09:28:55.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:55 smithi026 ceph-mon[68309]: 1.9 scrub starts 2024-03-20T09:28:55.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:55 smithi026 ceph-mon[68309]: 1.9 scrub ok 2024-03-20T09:28:55.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:55 smithi026 ceph-mon[68309]: pgmap v1996: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 3.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 261 op/s; 4104/186018 objects degraded (2.206%); 6269/186018 objects misplaced (3.370%); 3.1 MiB/s, 102 objects/s recovering 2024-03-20T09:28:56.048 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:28:56.099 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:28:56.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:56.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:28:56.213 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:28:56.247 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:56.271 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:28:56.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:56.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:28:56.394 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:28:56.411 INFO:teuthology.orchestra.run.smithi026.stderr:no valid command found; 10 closest matches: 2024-03-20T09:28:56.411 INFO:teuthology.orchestra.run.smithi026.stderr:0 2024-03-20T09:28:56.411 INFO:teuthology.orchestra.run.smithi026.stderr:1 2024-03-20T09:28:56.411 INFO:teuthology.orchestra.run.smithi026.stderr:2 2024-03-20T09:28:56.412 INFO:teuthology.orchestra.run.smithi026.stderr:abort 2024-03-20T09:28:56.412 INFO:teuthology.orchestra.run.smithi026.stderr:assert 2024-03-20T09:28:56.412 INFO:teuthology.orchestra.run.smithi026.stderr:bench [] [] [] [] 2024-03-20T09:28:56.412 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs debug_inject_read_zeros 2024-03-20T09:28:56.412 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs files list 2024-03-20T09:28:56.412 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs stats 2024-03-20T09:28:56.412 INFO:teuthology.orchestra.run.smithi026.stderr:bluestore allocator dump block 2024-03-20T09:28:56.412 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: invalid command 2024-03-20T09:28:56.420 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:56.439 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:28:56.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:56.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:28:56.542 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:28:56.584 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:56.976 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:28:56.976 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-03-20T09:28:56.976 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-03-20T09:28:56.976 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-03-20T09:28:56.976 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.976 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26216.0:121973 6.8 6:1fbfa6bc:::benchmark_data_smithi122_168655_object1864:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e746)", 2024-03-20T09:28:56.976 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:25:49.661187+0000", 2024-03-20T09:28:56.976 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 187.318515894, 2024-03-20T09:28:56.976 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.21986793499999999, 2024-03-20T09:28:56.976 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:28:56.977 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:28:56.977 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:28:56.977 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26216", 2024-03-20T09:28:56.977 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:28:56.977 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 121973 2024-03-20T09:28:56.977 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.977 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:28:56.977 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.977 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:28:56.977 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.661187+0000", 2024-03-20T09:28:56.977 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:56.977 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.977 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.977 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:28:56.978 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.661187+0000", 2024-03-20T09:28:56.978 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:56.978 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.978 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.978 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:28:56.978 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.661188+0000", 2024-03-20T09:28:56.978 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.288e-06 2024-03-20T09:28:56.978 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.978 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.978 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:28:56.978 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.661194+0000", 2024-03-20T09:28:56.978 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6710000000000004e-06 2024-03-20T09:28:56.978 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.979 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.979 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:28:56.979 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.661206+0000", 2024-03-20T09:28:56.979 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2335e-05 2024-03-20T09:28:56.979 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.979 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.979 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:28:56.979 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.661214+0000", 2024-03-20T09:28:56.979 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9449999999999993e-06 2024-03-20T09:28:56.979 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.979 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.979 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:28:56.979 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.661262+0000", 2024-03-20T09:28:56.979 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7913e-05 2024-03-20T09:28:56.980 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.980 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.980 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:28:56.980 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.661394+0000", 2024-03-20T09:28:56.980 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013232200000000001 2024-03-20T09:28:56.980 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.980 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.980 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,7", 2024-03-20T09:28:56.980 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.661479+0000", 2024-03-20T09:28:56.980 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4894000000000004e-05 2024-03-20T09:28:56.980 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.980 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.980 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:28:56.981 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.664779+0000", 2024-03-20T09:28:56.981 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0032996409999999999 2024-03-20T09:28:56.981 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.981 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.981 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:28:56.981 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.665554+0000", 2024-03-20T09:28:56.981 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00077524799999999997 2024-03-20T09:28:56.981 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.981 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.981 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:28:56.981 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.880940+0000", 2024-03-20T09:28:56.981 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.215386046 2024-03-20T09:28:56.981 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.981 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.982 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:28:56.982 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.881000+0000", 2024-03-20T09:28:56.982 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0132e-05 2024-03-20T09:28:56.982 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.982 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.982 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:28:56.982 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.881055+0000", 2024-03-20T09:28:56.982 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4500000000000003e-05 2024-03-20T09:28:56.982 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:56.982 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:28:56.982 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:56.982 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.982 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.983 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26216.0:121979 6.8 6:198aa212:::benchmark_data_smithi122_168655_object1870:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e746)", 2024-03-20T09:28:56.983 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:25:49.670397+0000", 2024-03-20T09:28:56.983 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 187.30930576, 2024-03-20T09:28:56.983 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.21078769, 2024-03-20T09:28:56.983 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:28:56.983 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:28:56.983 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:28:56.983 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26216", 2024-03-20T09:28:56.983 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:28:56.983 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 121979 2024-03-20T09:28:56.983 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.983 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:28:56.983 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.983 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:28:56.984 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.670397+0000", 2024-03-20T09:28:56.984 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:56.984 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.984 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.984 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:28:56.984 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.670397+0000", 2024-03-20T09:28:56.984 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:56.984 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.984 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.984 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:28:56.984 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.670399+0000", 2024-03-20T09:28:56.984 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0389999999999999e-06 2024-03-20T09:28:56.984 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.984 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.985 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:28:56.985 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.670410+0000", 2024-03-20T09:28:56.985 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0845999999999999e-05 2024-03-20T09:28:56.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.985 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.985 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:28:56.985 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.670431+0000", 2024-03-20T09:28:56.985 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0811000000000001e-05 2024-03-20T09:28:56.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.985 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.985 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:28:56.985 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.670443+0000", 2024-03-20T09:28:56.985 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2330000000000001e-05 2024-03-20T09:28:56.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.986 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.986 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:28:56.986 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.670519+0000", 2024-03-20T09:28:56.986 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6626999999999997e-05 2024-03-20T09:28:56.986 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.986 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.986 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:28:56.986 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.670662+0000", 2024-03-20T09:28:56.986 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000142292 2024-03-20T09:28:56.986 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.987 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.987 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,7", 2024-03-20T09:28:56.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.670792+0000", 2024-03-20T09:28:56.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013070200000000001 2024-03-20T09:28:56.987 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.987 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.987 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:28:56.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.677843+0000", 2024-03-20T09:28:56.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0070506279999999998 2024-03-20T09:28:56.987 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.987 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.987 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:28:56.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.678215+0000", 2024-03-20T09:28:56.988 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00037175899999999999 2024-03-20T09:28:56.988 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.988 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:28:56.988 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.881097+0000", 2024-03-20T09:28:56.988 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20288172700000001 2024-03-20T09:28:56.988 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.988 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:28:56.988 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.881150+0000", 2024-03-20T09:28:56.988 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3795999999999998e-05 2024-03-20T09:28:56.988 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.988 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:28:56.989 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.881185+0000", 2024-03-20T09:28:56.989 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4133e-05 2024-03-20T09:28:56.989 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:56.989 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:28:56.989 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:56.989 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.989 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.989 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26216.0:121983 6.8 6:11afe848:::benchmark_data_smithi122_168655_object1874:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e746)", 2024-03-20T09:28:56.989 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:25:49.672785+0000", 2024-03-20T09:28:56.989 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 187.306917083, 2024-03-20T09:28:56.989 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20855220799999999, 2024-03-20T09:28:56.989 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:28:56.989 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:28:56.990 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:28:56.990 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26216", 2024-03-20T09:28:56.990 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:28:56.990 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 121983 2024-03-20T09:28:56.990 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.990 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:28:56.990 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:28:56.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.672785+0000", 2024-03-20T09:28:56.990 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:56.990 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.990 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:28:56.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.672785+0000", 2024-03-20T09:28:56.991 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:56.991 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.991 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.991 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:28:56.991 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.672788+0000", 2024-03-20T09:28:56.991 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1440000000000001e-06 2024-03-20T09:28:56.991 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.991 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.991 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:28:56.991 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.672797+0000", 2024-03-20T09:28:56.991 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.901e-06 2024-03-20T09:28:56.991 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.991 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.991 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:28:56.992 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.672818+0000", 2024-03-20T09:28:56.992 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1903999999999999e-05 2024-03-20T09:28:56.992 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.992 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.992 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:28:56.992 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.672830+0000", 2024-03-20T09:28:56.992 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1195999999999999e-05 2024-03-20T09:28:56.992 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.992 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.992 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:28:56.992 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.673913+0000", 2024-03-20T09:28:56.992 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010829450000000001 2024-03-20T09:28:56.992 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.992 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.993 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:28:56.993 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.674056+0000", 2024-03-20T09:28:56.993 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014391600000000001 2024-03-20T09:28:56.993 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.993 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.993 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,7", 2024-03-20T09:28:56.993 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.674183+0000", 2024-03-20T09:28:56.993 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012637299999999999 2024-03-20T09:28:56.993 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.993 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.993 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:28:56.993 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.677992+0000", 2024-03-20T09:28:56.993 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0038093530000000001 2024-03-20T09:28:56.993 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.994 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.994 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:28:56.994 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.692487+0000", 2024-03-20T09:28:56.994 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014494913999999999 2024-03-20T09:28:56.994 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.994 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.994 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:28:56.994 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.881237+0000", 2024-03-20T09:28:56.994 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18874946500000001 2024-03-20T09:28:56.994 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.994 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.994 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:28:56.994 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.881298+0000", 2024-03-20T09:28:56.994 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1573000000000002e-05 2024-03-20T09:28:56.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.995 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.995 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:28:56.995 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.881338+0000", 2024-03-20T09:28:56.995 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9524000000000003e-05 2024-03-20T09:28:56.995 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:56.995 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:28:56.995 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:56.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.995 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.995 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26216.0:121993 6.8 6:12144f5a:::benchmark_data_smithi122_168655_object1884:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e746)", 2024-03-20T09:28:56.995 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:25:49.680006+0000", 2024-03-20T09:28:56.995 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 187.29969686600001, 2024-03-20T09:28:56.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20216557099999999, 2024-03-20T09:28:56.996 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:28:56.996 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:28:56.996 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:28:56.996 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26216", 2024-03-20T09:28:56.996 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:28:56.996 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 121993 2024-03-20T09:28:56.996 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.996 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:28:56.996 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:28:56.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.680006+0000", 2024-03-20T09:28:56.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:56.996 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.998 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.998 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:28:56.998 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.680006+0000", 2024-03-20T09:28:56.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:56.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.998 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.998 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:28:56.998 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.680007+0000", 2024-03-20T09:28:56.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4300000000000001e-06 2024-03-20T09:28:56.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.998 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.998 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:28:56.999 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.680013+0000", 2024-03-20T09:28:56.999 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5310000000000001e-06 2024-03-20T09:28:56.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.999 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:28:56.999 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.680026+0000", 2024-03-20T09:28:56.999 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2941e-05 2024-03-20T09:28:56.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:56.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:56.999 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:28:56.999 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.680032+0000", 2024-03-20T09:28:56.999 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4060000000000002e-06 2024-03-20T09:28:56.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.000 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:28:57.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.680085+0000", 2024-03-20T09:28:57.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3437999999999999e-05 2024-03-20T09:28:57.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.000 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:28:57.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.680224+0000", 2024-03-20T09:28:57.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013856200000000001 2024-03-20T09:28:57.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.000 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,7", 2024-03-20T09:28:57.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.680328+0000", 2024-03-20T09:28:57.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010411700000000001 2024-03-20T09:28:57.001 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.001 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.001 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:28:57.001 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.692505+0000", 2024-03-20T09:28:57.001 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012176424 2024-03-20T09:28:57.001 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.001 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.001 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:28:57.001 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.692529+0000", 2024-03-20T09:28:57.001 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4658999999999999e-05 2024-03-20T09:28:57.001 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.001 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.001 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:28:57.001 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.882108+0000", 2024-03-20T09:28:57.001 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.189578944 2024-03-20T09:28:57.002 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.002 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.002 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:28:57.002 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.882142+0000", 2024-03-20T09:28:57.002 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3368999999999998e-05 2024-03-20T09:28:57.002 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.002 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.002 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:28:57.002 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.882171+0000", 2024-03-20T09:28:57.002 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9750000000000001e-05 2024-03-20T09:28:57.002 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:57.002 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:28:57.002 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:57.002 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.003 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.003 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26216.0:121995 6.8 6:10f6e117:::benchmark_data_smithi122_168655_object1886:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e746)", 2024-03-20T09:28:57.003 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:25:49.682067+0000", 2024-03-20T09:28:57.003 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 187.29763529600001, 2024-03-20T09:28:57.003 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.202431376, 2024-03-20T09:28:57.003 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:28:57.003 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:28:57.003 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:28:57.003 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26216", 2024-03-20T09:28:57.003 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:28:57.003 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 121995 2024-03-20T09:28:57.003 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.003 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:28:57.003 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.004 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:28:57.004 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.682067+0000", 2024-03-20T09:28:57.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:57.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.004 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.004 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:28:57.004 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.682067+0000", 2024-03-20T09:28:57.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:57.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.004 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.004 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:28:57.004 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.682069+0000", 2024-03-20T09:28:57.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.294e-06 2024-03-20T09:28:57.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.005 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:28:57.005 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.682073+0000", 2024-03-20T09:28:57.005 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5469999999999998e-06 2024-03-20T09:28:57.005 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.005 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:28:57.005 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.682088+0000", 2024-03-20T09:28:57.005 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4975e-05 2024-03-20T09:28:57.005 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.005 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:28:57.005 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.682098+0000", 2024-03-20T09:28:57.005 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7429999999999997e-06 2024-03-20T09:28:57.006 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.006 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.006 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:28:57.006 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.683330+0000", 2024-03-20T09:28:57.006 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001232609 2024-03-20T09:28:57.006 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.006 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.006 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:28:57.006 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.683428+0000", 2024-03-20T09:28:57.006 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7517999999999997e-05 2024-03-20T09:28:57.006 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.006 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.006 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,7", 2024-03-20T09:28:57.007 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.683509+0000", 2024-03-20T09:28:57.007 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1178000000000001e-05 2024-03-20T09:28:57.007 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.007 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.007 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:28:57.007 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.692566+0000", 2024-03-20T09:28:57.007 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0090563740000000007 2024-03-20T09:28:57.007 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.007 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.007 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:28:57.007 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.692578+0000", 2024-03-20T09:28:57.007 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.256e-05 2024-03-20T09:28:57.007 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.007 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.008 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:28:57.008 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.884404+0000", 2024-03-20T09:28:57.008 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19182617800000001 2024-03-20T09:28:57.008 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.008 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.008 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:28:57.008 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.884459+0000", 2024-03-20T09:28:57.008 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4324999999999999e-05 2024-03-20T09:28:57.008 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.008 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.008 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:28:57.008 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.884499+0000", 2024-03-20T09:28:57.008 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0074999999999998e-05 2024-03-20T09:28:57.008 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:57.009 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:28:57.009 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:57.009 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.009 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.009 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26216.0:121996 6.8 6:103fc448:::benchmark_data_smithi122_168655_object1887:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e746)", 2024-03-20T09:28:57.009 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:25:49.685531+0000", 2024-03-20T09:28:57.009 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 187.29417171, 2024-03-20T09:28:57.009 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20287249600000001, 2024-03-20T09:28:57.009 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:28:57.009 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:28:57.009 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:28:57.009 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26216", 2024-03-20T09:28:57.009 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:28:57.009 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 121996 2024-03-20T09:28:57.010 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.010 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:28:57.010 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.010 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:28:57.010 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.685531+0000", 2024-03-20T09:28:57.010 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:57.010 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.010 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.010 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:28:57.010 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.685531+0000", 2024-03-20T09:28:57.010 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:57.010 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.010 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.010 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:28:57.011 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.685532+0000", 2024-03-20T09:28:57.011 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.409e-06 2024-03-20T09:28:57.011 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.011 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.011 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:28:57.011 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.685538+0000", 2024-03-20T09:28:57.011 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6799999999999998e-06 2024-03-20T09:28:57.011 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.011 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.011 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:28:57.011 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.685551+0000", 2024-03-20T09:28:57.011 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3271999999999999e-05 2024-03-20T09:28:57.011 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.012 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.012 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:28:57.012 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.685559+0000", 2024-03-20T09:28:57.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6020000000000002e-06 2024-03-20T09:28:57.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.012 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.012 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:28:57.012 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.686141+0000", 2024-03-20T09:28:57.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00058253000000000005 2024-03-20T09:28:57.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.012 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.012 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:28:57.012 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.686256+0000", 2024-03-20T09:28:57.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000114231 2024-03-20T09:28:57.013 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.013 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.013 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,7", 2024-03-20T09:28:57.013 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.686332+0000", 2024-03-20T09:28:57.013 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5982000000000005e-05 2024-03-20T09:28:57.013 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.013 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.013 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:28:57.013 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.692584+0000", 2024-03-20T09:28:57.013 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0062523800000000001 2024-03-20T09:28:57.013 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.013 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.013 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:28:57.013 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.693243+0000", 2024-03-20T09:28:57.014 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00065878599999999996 2024-03-20T09:28:57.014 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.014 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.014 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:28:57.014 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.888327+0000", 2024-03-20T09:28:57.014 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.195084219 2024-03-20T09:28:57.014 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.014 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.014 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:28:57.014 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.888361+0000", 2024-03-20T09:28:57.014 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4286999999999999e-05 2024-03-20T09:28:57.014 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.014 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.015 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:28:57.015 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.888403+0000", 2024-03-20T09:28:57.015 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2117999999999999e-05 2024-03-20T09:28:57.015 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:57.015 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:28:57.015 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:57.015 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.015 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.015 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26216.0:121999 6.8 6:1ea2bba8:::benchmark_data_smithi122_168655_object1890:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e746)", 2024-03-20T09:28:57.015 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:25:49.685703+0000", 2024-03-20T09:28:57.015 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 187.29399937599999, 2024-03-20T09:28:57.015 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.204141506, 2024-03-20T09:28:57.015 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:28:57.015 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:28:57.015 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:28:57.016 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26216", 2024-03-20T09:28:57.016 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:28:57.016 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 121999 2024-03-20T09:28:57.016 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.016 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:28:57.016 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.016 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:28:57.016 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.685703+0000", 2024-03-20T09:28:57.016 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:57.016 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.016 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.016 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:28:57.016 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.685703+0000", 2024-03-20T09:28:57.016 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:57.017 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.017 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.017 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:28:57.017 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.685704+0000", 2024-03-20T09:28:57.017 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8400000000000004e-07 2024-03-20T09:28:57.017 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.017 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.017 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:28:57.017 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.685708+0000", 2024-03-20T09:28:57.017 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2660000000000003e-06 2024-03-20T09:28:57.017 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.017 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.017 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:28:57.017 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.685719+0000", 2024-03-20T09:28:57.018 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1008e-05 2024-03-20T09:28:57.018 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.018 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.018 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:28:57.018 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.685726+0000", 2024-03-20T09:28:57.018 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6619999999999999e-06 2024-03-20T09:28:57.018 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.018 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.018 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:28:57.018 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.689276+0000", 2024-03-20T09:28:57.018 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0035501550000000002 2024-03-20T09:28:57.018 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.018 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.019 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:28:57.019 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.689433+0000", 2024-03-20T09:28:57.019 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015742100000000001 2024-03-20T09:28:57.019 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.019 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.019 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,7", 2024-03-20T09:28:57.019 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.689533+0000", 2024-03-20T09:28:57.019 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9533999999999999e-05 2024-03-20T09:28:57.019 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.019 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.019 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:28:57.019 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.692611+0000", 2024-03-20T09:28:57.019 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0030784620000000001 2024-03-20T09:28:57.019 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.020 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.020 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:28:57.020 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.694031+0000", 2024-03-20T09:28:57.020 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001419322 2024-03-20T09:28:57.020 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.020 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.020 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:28:57.020 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.889787+0000", 2024-03-20T09:28:57.020 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.195755874 2024-03-20T09:28:57.020 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.020 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.020 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:28:57.020 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.889819+0000", 2024-03-20T09:28:57.020 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2817000000000001e-05 2024-03-20T09:28:57.021 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.021 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.021 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:28:57.021 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.889845+0000", 2024-03-20T09:28:57.021 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5301000000000001e-05 2024-03-20T09:28:57.021 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:57.021 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:28:57.021 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:57.021 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.021 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.021 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26216.0:129139 6.18 6:1a6b87cd:::benchmark_data_smithi122_168655_object9030:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e749)", 2024-03-20T09:28:57.021 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:25:56.334405+0000", 2024-03-20T09:28:57.021 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 180.645297266, 2024-03-20T09:28:57.021 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1316102589999999, 2024-03-20T09:28:57.022 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:28:57.022 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:28:57.022 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:28:57.022 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26216", 2024-03-20T09:28:57.022 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:28:57.022 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 129139 2024-03-20T09:28:57.022 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.022 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:28:57.022 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.022 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:28:57.022 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:56.334405+0000", 2024-03-20T09:28:57.022 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:57.022 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.023 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.023 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:28:57.023 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:56.334405+0000", 2024-03-20T09:28:57.023 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:57.023 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.023 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.023 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:28:57.023 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:56.334407+0000", 2024-03-20T09:28:57.023 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4529999999999999e-06 2024-03-20T09:28:57.023 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.023 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.023 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:28:57.023 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:56.334414+0000", 2024-03-20T09:28:57.023 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2679999999999997e-06 2024-03-20T09:28:57.023 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.024 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.024 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:28:57.024 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:56.334428+0000", 2024-03-20T09:28:57.024 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4443999999999999e-05 2024-03-20T09:28:57.024 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.024 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.024 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:28:57.024 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:56.334436+0000", 2024-03-20T09:28:57.024 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8890000000000002e-06 2024-03-20T09:28:57.024 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.024 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.024 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:28:57.024 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:56.487565+0000", 2024-03-20T09:28:57.024 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.153128978 2024-03-20T09:28:57.025 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.025 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.025 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:28:57.025 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:56.487575+0000", 2024-03-20T09:28:57.025 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3160000000000003e-06 2024-03-20T09:28:57.025 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.025 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.025 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:28:57.025 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:57.465932+0000", 2024-03-20T09:28:57.025 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.97835746599999995 2024-03-20T09:28:57.025 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.025 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.025 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:28:57.025 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:57.466016+0000", 2024-03-20T09:28:57.026 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3444999999999998e-05 2024-03-20T09:28:57.026 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:57.026 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:28:57.026 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:57.026 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.026 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.026 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.26216.0:170285 6.17 e816/812 6:efb71509:::benchmark_data_smithi122_168655_object50176:head v 816'9536, mlcod=0'0)", 2024-03-20T09:28:57.026 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:28:26.662483+0000", 2024-03-20T09:28:57.026 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 30.317219081000001, 2024-03-20T09:28:57.026 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0043973570000000002, 2024-03-20T09:28:57.026 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:28:57.026 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:28:57.026 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:28:57.026 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.027 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:28:57.027 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:26.662483+0000", 2024-03-20T09:28:57.027 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:57.027 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.027 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.027 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:28:57.027 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:26.662483+0000", 2024-03-20T09:28:57.027 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:57.027 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.027 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.027 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:28:57.027 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:26.662484+0000", 2024-03-20T09:28:57.027 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3900000000000004e-07 2024-03-20T09:28:57.027 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.028 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.028 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:28:57.028 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:26.662492+0000", 2024-03-20T09:28:57.028 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5820000000000004e-06 2024-03-20T09:28:57.028 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.028 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.028 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:28:57.028 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:26.662500+0000", 2024-03-20T09:28:57.028 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9519999999999994e-06 2024-03-20T09:28:57.028 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.028 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.028 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:28:57.028 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:26.662504+0000", 2024-03-20T09:28:57.028 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6059999999999996e-06 2024-03-20T09:28:57.029 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.029 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.029 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:28:57.029 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:26.662561+0000", 2024-03-20T09:28:57.029 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6966999999999999e-05 2024-03-20T09:28:57.029 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.029 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.029 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:28:57.029 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:26.662578+0000", 2024-03-20T09:28:57.029 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6433999999999999e-05 2024-03-20T09:28:57.029 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.029 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.029 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:28:57.029 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:26.666838+0000", 2024-03-20T09:28:57.029 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0042599969999999997 2024-03-20T09:28:57.030 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.030 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.030 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:28:57.030 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:26.666881+0000", 2024-03-20T09:28:57.030 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3080000000000001e-05 2024-03-20T09:28:57.030 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:57.030 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:28:57.030 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:57.030 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.030 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.030 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.26216.0:170951 6.17 e818/812 6:ebf33a53:::benchmark_data_smithi122_168655_object50842:head v 818'9561, mlcod=725'3206)", 2024-03-20T09:28:57.030 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:28:29.339818+0000", 2024-03-20T09:28:57.030 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 27.639884355, 2024-03-20T09:28:57.031 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0050113900000000001, 2024-03-20T09:28:57.031 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:28:57.031 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:28:57.031 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:28:57.031 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.031 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:28:57.031 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:29.339818+0000", 2024-03-20T09:28:57.031 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:57.031 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.031 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.031 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:28:57.031 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:29.339818+0000", 2024-03-20T09:28:57.031 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:57.031 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.032 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.032 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:28:57.032 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:29.339819+0000", 2024-03-20T09:28:57.032 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5099999999999998e-07 2024-03-20T09:28:57.032 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.032 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.032 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:28:57.032 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:29.339827+0000", 2024-03-20T09:28:57.032 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0579999999999998e-06 2024-03-20T09:28:57.032 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.032 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.032 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:28:57.032 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:29.339842+0000", 2024-03-20T09:28:57.032 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4738e-05 2024-03-20T09:28:57.033 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.033 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.033 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:28:57.033 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:29.339851+0000", 2024-03-20T09:28:57.033 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1330000000000005e-06 2024-03-20T09:28:57.033 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.033 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.033 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:28:57.033 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:29.339932+0000", 2024-03-20T09:28:57.033 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1087999999999993e-05 2024-03-20T09:28:57.033 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.033 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.033 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:28:57.033 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:29.339969+0000", 2024-03-20T09:28:57.033 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6511999999999999e-05 2024-03-20T09:28:57.034 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.034 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.034 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:28:57.034 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:29.344763+0000", 2024-03-20T09:28:57.034 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0047940550000000002 2024-03-20T09:28:57.034 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.034 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.034 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:28:57.034 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:29.344830+0000", 2024-03-20T09:28:57.034 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6854999999999996e-05 2024-03-20T09:28:57.034 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:57.034 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:28:57.034 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:57.035 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.035 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.035 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.26216.0:170904 6.17 e818/812 6:ebada111:::benchmark_data_smithi122_168655_object50795:head v 818'9562, mlcod=725'3206)", 2024-03-20T09:28:57.035 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:28:29.354663+0000", 2024-03-20T09:28:57.035 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 27.625039606000001, 2024-03-20T09:28:57.035 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0048595649999999997, 2024-03-20T09:28:57.035 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:28:57.035 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:28:57.035 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:28:57.035 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.035 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:28:57.035 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:29.354663+0000", 2024-03-20T09:28:57.035 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:57.036 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.036 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.036 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:28:57.036 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:29.354663+0000", 2024-03-20T09:28:57.036 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:57.036 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.036 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.036 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:28:57.036 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:29.354664+0000", 2024-03-20T09:28:57.036 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1510000000000001e-06 2024-03-20T09:28:57.036 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.036 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.036 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:28:57.036 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:29.354674+0000", 2024-03-20T09:28:57.036 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7690000000000008e-06 2024-03-20T09:28:57.037 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.037 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.037 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:28:57.037 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:29.354686+0000", 2024-03-20T09:28:57.037 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1969e-05 2024-03-20T09:28:57.037 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.037 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.037 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:28:57.037 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:29.354693+0000", 2024-03-20T09:28:57.037 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1910000000000003e-06 2024-03-20T09:28:57.037 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.037 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.037 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:28:57.038 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:29.354766+0000", 2024-03-20T09:28:57.038 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3423000000000005e-05 2024-03-20T09:28:57.038 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.038 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.038 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:28:57.038 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:29.354803+0000", 2024-03-20T09:28:57.038 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6764999999999998e-05 2024-03-20T09:28:57.038 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.038 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.038 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:28:57.038 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:29.359473+0000", 2024-03-20T09:28:57.038 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0046693189999999999 2024-03-20T09:28:57.038 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.038 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.039 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:28:57.039 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:29.359523+0000", 2024-03-20T09:28:57.039 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9978000000000002e-05 2024-03-20T09:28:57.039 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:57.039 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:28:57.039 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:57.039 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.039 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.039 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.26216.0:171331 6.17 e819/812 6:eb8bd120:::benchmark_data_smithi122_168655_object51222:head v 819'9574, mlcod=725'3238)", 2024-03-20T09:28:57.039 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:28:30.525102+0000", 2024-03-20T09:28:57.039 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 26.454600075999998, 2024-03-20T09:28:57.039 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0044245730000000002, 2024-03-20T09:28:57.039 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:28:57.039 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:28:57.040 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:28:57.040 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.040 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:28:57.040 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:30.525102+0000", 2024-03-20T09:28:57.040 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:57.040 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.040 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.040 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:28:57.040 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:30.525102+0000", 2024-03-20T09:28:57.040 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:57.040 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.040 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.040 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:28:57.040 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:30.525104+0000", 2024-03-20T09:28:57.041 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.085e-06 2024-03-20T09:28:57.041 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.041 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.041 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:28:57.041 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:30.525111+0000", 2024-03-20T09:28:57.041 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3459999999999996e-06 2024-03-20T09:28:57.041 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.041 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.041 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:28:57.041 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:30.525123+0000", 2024-03-20T09:28:57.041 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2148e-05 2024-03-20T09:28:57.041 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.041 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.041 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:28:57.041 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:30.525130+0000", 2024-03-20T09:28:57.042 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1910000000000003e-06 2024-03-20T09:28:57.042 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.042 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.042 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:28:57.042 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:30.525189+0000", 2024-03-20T09:28:57.042 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8459999999999999e-05 2024-03-20T09:28:57.042 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.042 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.042 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:28:57.042 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:30.525223+0000", 2024-03-20T09:28:57.042 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.447e-05 2024-03-20T09:28:57.042 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.042 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.043 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:28:57.043 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:30.529480+0000", 2024-03-20T09:28:57.043 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.004257205 2024-03-20T09:28:57.043 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.043 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.043 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:28:57.043 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:30.529527+0000", 2024-03-20T09:28:57.043 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6668000000000001e-05 2024-03-20T09:28:57.043 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:57.043 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:28:57.043 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:57.043 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.043 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.043 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.26216.0:171808 6.8 e819/817 6:119924f3:::benchmark_data_smithi122_168655_object51699:head v 819'9281, mlcod=819'9279)", 2024-03-20T09:28:57.044 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:28:30.860519+0000", 2024-03-20T09:28:57.044 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 26.119183928999998, 2024-03-20T09:28:57.044 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0048443920000000003, 2024-03-20T09:28:57.044 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:28:57.044 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:28:57.044 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:28:57.044 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.044 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:28:57.044 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:30.860519+0000", 2024-03-20T09:28:57.044 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:57.044 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.044 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.044 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:28:57.045 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:30.860519+0000", 2024-03-20T09:28:57.045 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:57.045 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.045 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.045 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:28:57.045 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:30.860520+0000", 2024-03-20T09:28:57.045 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9000000000000005e-07 2024-03-20T09:28:57.045 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.045 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.045 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:28:57.045 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:30.860527+0000", 2024-03-20T09:28:57.045 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6979999999999998e-06 2024-03-20T09:28:57.045 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.046 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.046 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:28:57.046 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:30.860539+0000", 2024-03-20T09:28:57.046 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1827999999999999e-05 2024-03-20T09:28:57.046 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.046 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.046 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:28:57.046 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:30.860548+0000", 2024-03-20T09:28:57.046 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9490000000000002e-06 2024-03-20T09:28:57.046 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.046 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.046 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:28:57.046 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:30.860602+0000", 2024-03-20T09:28:57.046 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3902e-05 2024-03-20T09:28:57.047 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.047 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.047 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:28:57.047 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:30.860626+0000", 2024-03-20T09:28:57.047 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4017999999999999e-05 2024-03-20T09:28:57.047 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.047 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.047 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:28:57.047 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:30.865318+0000", 2024-03-20T09:28:57.047 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0046915129999999996 2024-03-20T09:28:57.047 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.047 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.047 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:28:57.047 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:30.865363+0000", 2024-03-20T09:28:57.048 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5494000000000002e-05 2024-03-20T09:28:57.048 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:57.048 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:28:57.048 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:57.048 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.048 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.048 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.26216.0:171981 6.3 e819/749 6:c2f87cb5:::benchmark_data_smithi122_168655_object51872:head v 819'9366, mlcod=819'9364)", 2024-03-20T09:28:57.048 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:28:31.217463+0000", 2024-03-20T09:28:57.048 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 25.762239338000001, 2024-03-20T09:28:57.049 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0052865259999999997, 2024-03-20T09:28:57.049 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:28:57.049 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:28:57.049 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:28:57.049 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.049 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:28:57.049 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:31.217463+0000", 2024-03-20T09:28:57.049 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:57.049 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.049 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.049 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:28:57.049 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:31.217463+0000", 2024-03-20T09:28:57.049 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:57.049 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.050 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.050 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:28:57.050 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:31.217464+0000", 2024-03-20T09:28:57.050 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3499999999999996e-07 2024-03-20T09:28:57.050 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.050 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.050 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:28:57.050 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:31.217469+0000", 2024-03-20T09:28:57.050 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.096e-06 2024-03-20T09:28:57.050 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.050 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.050 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:28:57.050 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:31.217479+0000", 2024-03-20T09:28:57.051 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7950000000000002e-06 2024-03-20T09:28:57.051 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.051 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.051 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:28:57.051 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:31.217484+0000", 2024-03-20T09:28:57.051 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4469999999999997e-06 2024-03-20T09:28:57.051 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.051 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.051 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:28:57.051 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:31.217529+0000", 2024-03-20T09:28:57.051 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4821000000000003e-05 2024-03-20T09:28:57.051 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.051 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.051 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:28:57.052 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:31.217545+0000", 2024-03-20T09:28:57.052 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5747e-05 2024-03-20T09:28:57.052 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.052 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.052 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:28:57.052 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:31.222704+0000", 2024-03-20T09:28:57.052 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0051588529999999997 2024-03-20T09:28:57.052 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.052 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.052 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:28:57.052 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:31.222750+0000", 2024-03-20T09:28:57.052 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6131999999999998e-05 2024-03-20T09:28:57.052 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:57.053 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:28:57.053 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:57.053 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.053 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.053 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.26216.0:171979 6.11 e819/781 6:8a332341:::benchmark_data_smithi122_168655_object51870:head v 819'9423, mlcod=819'9421)", 2024-03-20T09:28:57.053 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:28:31.217605+0000", 2024-03-20T09:28:57.053 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 25.76209777, 2024-03-20T09:28:57.053 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0051206560000000003, 2024-03-20T09:28:57.053 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:28:57.053 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:28:57.053 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:28:57.053 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.053 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:28:57.053 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:31.217605+0000", 2024-03-20T09:28:57.054 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:57.054 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.054 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.054 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:28:57.054 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:31.217605+0000", 2024-03-20T09:28:57.054 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:57.054 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.054 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.054 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:28:57.054 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:31.217605+0000", 2024-03-20T09:28:57.054 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3499999999999996e-07 2024-03-20T09:28:57.054 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.054 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.054 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:28:57.055 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:31.217610+0000", 2024-03-20T09:28:57.055 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4569999999999998e-06 2024-03-20T09:28:57.055 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.055 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.055 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:28:57.055 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:31.217617+0000", 2024-03-20T09:28:57.055 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0389999999999999e-06 2024-03-20T09:28:57.055 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.055 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.055 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:28:57.055 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:31.217622+0000", 2024-03-20T09:28:57.055 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7740000000000004e-06 2024-03-20T09:28:57.055 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.055 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.056 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:28:57.056 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:31.217669+0000", 2024-03-20T09:28:57.056 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7006999999999999e-05 2024-03-20T09:28:57.056 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.056 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.056 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:28:57.056 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:31.217688+0000", 2024-03-20T09:28:57.056 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9528999999999999e-05 2024-03-20T09:28:57.056 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.056 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.056 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:28:57.056 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:31.222694+0000", 2024-03-20T09:28:57.056 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00500586 2024-03-20T09:28:57.056 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.057 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.057 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:28:57.057 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:31.222725+0000", 2024-03-20T09:28:57.057 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1455000000000003e-05 2024-03-20T09:28:57.057 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:57.057 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:28:57.057 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:57.057 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.057 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.057 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.26216.0:171799 6.17 e820/812 6:eff0c828:::benchmark_data_smithi122_168655_object51690:head v 820'9587, mlcod=725'3258)", 2024-03-20T09:28:57.057 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:28:31.763142+0000", 2024-03-20T09:28:57.057 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 25.216560741999999, 2024-03-20T09:28:57.057 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0049910509999999998, 2024-03-20T09:28:57.057 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:28:57.058 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:28:57.058 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:28:57.058 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.058 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:28:57.058 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:31.763142+0000", 2024-03-20T09:28:57.058 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:57.058 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.058 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.058 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:28:57.058 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:31.763142+0000", 2024-03-20T09:28:57.058 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:57.058 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.058 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.059 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:28:57.059 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:31.763144+0000", 2024-03-20T09:28:57.059 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.697e-06 2024-03-20T09:28:57.059 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.059 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.059 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:28:57.059 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:31.763155+0000", 2024-03-20T09:28:57.059 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.111e-05 2024-03-20T09:28:57.059 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.059 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.059 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:28:57.059 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:31.763173+0000", 2024-03-20T09:28:57.059 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8029999999999998e-05 2024-03-20T09:28:57.059 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.060 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.060 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:28:57.060 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:31.763183+0000", 2024-03-20T09:28:57.060 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.025e-05 2024-03-20T09:28:57.060 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.060 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.060 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:28:57.060 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:31.763251+0000", 2024-03-20T09:28:57.060 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7812999999999996e-05 2024-03-20T09:28:57.060 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.060 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.060 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:28:57.060 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:31.763280+0000", 2024-03-20T09:28:57.060 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9319000000000002e-05 2024-03-20T09:28:57.061 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.061 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.061 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:28:57.061 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:31.768072+0000", 2024-03-20T09:28:57.061 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0047923610000000002 2024-03-20T09:28:57.061 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.061 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.061 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:28:57.061 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:31.768133+0000", 2024-03-20T09:28:57.061 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0470999999999998e-05 2024-03-20T09:28:57.061 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:57.061 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:28:57.061 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:57.061 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.062 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.062 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.26216.0:172935 6.17 e821/812 6:e9405dc9:::benchmark_data_smithi122_168655_object52826:head v 821'9630, mlcod=726'3350)", 2024-03-20T09:28:57.062 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:28:35.842979+0000", 2024-03-20T09:28:57.062 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 21.136723673999999, 2024-03-20T09:28:57.062 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0044480589999999999, 2024-03-20T09:28:57.062 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:28:57.062 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:28:57.062 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:28:57.062 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.062 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:28:57.062 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:35.842979+0000", 2024-03-20T09:28:57.062 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:57.062 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.063 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.063 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:28:57.063 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:35.842979+0000", 2024-03-20T09:28:57.063 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:57.063 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.063 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.063 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:28:57.063 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:35.842980+0000", 2024-03-20T09:28:57.063 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.17e-06 2024-03-20T09:28:57.063 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.063 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.063 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:28:57.063 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:35.842989+0000", 2024-03-20T09:28:57.063 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7979999999999995e-06 2024-03-20T09:28:57.064 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.064 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.064 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:28:57.064 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:35.843003+0000", 2024-03-20T09:28:57.064 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4440000000000001e-05 2024-03-20T09:28:57.064 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.064 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.064 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:28:57.064 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:35.843011+0000", 2024-03-20T09:28:57.064 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8879999999999997e-06 2024-03-20T09:28:57.064 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.064 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.064 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:28:57.064 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:35.843066+0000", 2024-03-20T09:28:57.065 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4610000000000001e-05 2024-03-20T09:28:57.065 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.065 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.065 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:28:57.065 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:35.843091+0000", 2024-03-20T09:28:57.065 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4978999999999999e-05 2024-03-20T09:28:57.065 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.065 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.065 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:28:57.065 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:35.847358+0000", 2024-03-20T09:28:57.065 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0042672209999999999 2024-03-20T09:28:57.065 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.065 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.065 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:28:57.066 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:35.847427+0000", 2024-03-20T09:28:57.066 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8953000000000002e-05 2024-03-20T09:28:57.066 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:57.066 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:28:57.066 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:57.066 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.066 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.066 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.26216.0:173474 6.8 e821/817 6:168c2872:::benchmark_data_smithi122_168655_object53365:head v 821'9332, mlcod=821'9330)", 2024-03-20T09:28:57.066 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:28:35.911880+0000", 2024-03-20T09:28:57.066 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 21.067822083999999, 2024-03-20T09:28:57.066 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0049220820000000004, 2024-03-20T09:28:57.066 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:28:57.066 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:28:57.067 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:28:57.067 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.067 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:28:57.067 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:35.911880+0000", 2024-03-20T09:28:57.067 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:57.067 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.067 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.067 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:28:57.067 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:35.911880+0000", 2024-03-20T09:28:57.067 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:57.067 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.067 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.067 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:28:57.067 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:35.911881+0000", 2024-03-20T09:28:57.068 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8700000000000004e-07 2024-03-20T09:28:57.068 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.068 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.068 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:28:57.068 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:35.911889+0000", 2024-03-20T09:28:57.068 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3089999999999998e-06 2024-03-20T09:28:57.068 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.068 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.068 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:28:57.068 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:35.911903+0000", 2024-03-20T09:28:57.068 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3903999999999999e-05 2024-03-20T09:28:57.068 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.068 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.068 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:28:57.069 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:35.911912+0000", 2024-03-20T09:28:57.069 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2080000000000006e-06 2024-03-20T09:28:57.069 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.069 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.069 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:28:57.069 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:35.911981+0000", 2024-03-20T09:28:57.069 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9560000000000005e-05 2024-03-20T09:28:57.069 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.069 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.069 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:28:57.069 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:35.912006+0000", 2024-03-20T09:28:57.069 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4595000000000001e-05 2024-03-20T09:28:57.069 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.069 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.070 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:28:57.070 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:35.916763+0000", 2024-03-20T09:28:57.070 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0047572680000000003 2024-03-20T09:28:57.070 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.070 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.070 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:28:57.070 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:35.916803+0000", 2024-03-20T09:28:57.070 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9351000000000003e-05 2024-03-20T09:28:57.070 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:57.070 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:28:57.070 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:57.070 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.070 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.070 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.26216.0:178851 6.17 e826/812 6:ee5f9b36:::benchmark_data_smithi122_168655_object58742:head v 826'9819, mlcod=727'3737)", 2024-03-20T09:28:57.071 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:28:53.308685+0000", 2024-03-20T09:28:57.071 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 3.6710179040000002, 2024-03-20T09:28:57.071 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0046489569999999996, 2024-03-20T09:28:57.071 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:28:57.071 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:28:57.071 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:28:57.071 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.071 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:28:57.071 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:53.308685+0000", 2024-03-20T09:28:57.071 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:57.071 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.071 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.071 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:28:57.071 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:53.308685+0000", 2024-03-20T09:28:57.072 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:57.072 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.072 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.072 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:28:57.072 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:53.308686+0000", 2024-03-20T09:28:57.072 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2500000000000004e-07 2024-03-20T09:28:57.072 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.072 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.072 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:28:57.072 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:53.308693+0000", 2024-03-20T09:28:57.072 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8450000000000003e-06 2024-03-20T09:28:57.072 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.072 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.072 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:28:57.072 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:53.308707+0000", 2024-03-20T09:28:57.073 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3560999999999999e-05 2024-03-20T09:28:57.073 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.073 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.073 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:28:57.073 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:53.308716+0000", 2024-03-20T09:28:57.073 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5180000000000004e-06 2024-03-20T09:28:57.073 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.073 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.073 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:28:57.073 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:53.308787+0000", 2024-03-20T09:28:57.073 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1803000000000004e-05 2024-03-20T09:28:57.073 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.073 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.073 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:28:57.074 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:53.308822+0000", 2024-03-20T09:28:57.074 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4634000000000001e-05 2024-03-20T09:28:57.074 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.074 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.074 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:28:57.074 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:53.313266+0000", 2024-03-20T09:28:57.074 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0044439930000000002 2024-03-20T09:28:57.074 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.074 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.074 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:28:57.074 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:53.313334+0000", 2024-03-20T09:28:57.074 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7677999999999998e-05 2024-03-20T09:28:57.074 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:57.074 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:28:57.075 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:57.075 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.075 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.075 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.26216.0:179366 6.17 e827/812 6:ea36754a:::benchmark_data_smithi122_168655_object59257:head v 827'9838, mlcod=727'3761)", 2024-03-20T09:28:57.075 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:28:54.919818+0000", 2024-03-20T09:28:57.075 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 2.0598850620000002, 2024-03-20T09:28:57.075 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0050609499999999998, 2024-03-20T09:28:57.075 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:28:57.075 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:28:57.075 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:28:57.075 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.075 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:28:57.075 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:54.919818+0000", 2024-03-20T09:28:57.075 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:57.076 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.076 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.076 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:28:57.076 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:54.919818+0000", 2024-03-20T09:28:57.076 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:28:57.076 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.076 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.076 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:28:57.076 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:54.919819+0000", 2024-03-20T09:28:57.076 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5579999999999999e-06 2024-03-20T09:28:57.076 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.076 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.076 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:28:57.076 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:54.919829+0000", 2024-03-20T09:28:57.077 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0200000000000001e-05 2024-03-20T09:28:57.077 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.077 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.077 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:28:57.077 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:54.919848+0000", 2024-03-20T09:28:57.077 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9224000000000001e-05 2024-03-20T09:28:57.077 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.077 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.077 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:28:57.077 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:54.919859+0000", 2024-03-20T09:28:57.077 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0217999999999999e-05 2024-03-20T09:28:57.077 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.077 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.078 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:28:57.078 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:54.919946+0000", 2024-03-20T09:28:57.078 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7434000000000003e-05 2024-03-20T09:28:57.078 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.078 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.078 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:28:57.078 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:54.919985+0000", 2024-03-20T09:28:57.078 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9252000000000003e-05 2024-03-20T09:28:57.078 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.078 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.078 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:28:57.078 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:54.924811+0000", 2024-03-20T09:28:57.078 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.004825602 2024-03-20T09:28:57.078 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:28:57.079 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:28:57.079 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:28:57.079 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:28:54.924878+0000", 2024-03-20T09:28:57.079 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7461999999999999e-05 2024-03-20T09:28:57.079 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:57.079 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:28:57.079 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:57.079 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:28:57.079 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:28:57.079 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:28:57.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:57 smithi115 ceph-mon[72476]: pgmap v1997: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 3.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 317 op/s; 4104/182967 objects degraded (2.243%); 6269/182967 objects misplaced (3.426%); 3.1 MiB/s, 102 objects/s recovering 2024-03-20T09:28:57.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:57 smithi122 ceph-mon[73495]: pgmap v1997: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 3.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 317 op/s; 4104/182967 objects degraded (2.243%); 6269/182967 objects misplaced (3.426%); 3.1 MiB/s, 102 objects/s recovering 2024-03-20T09:28:57.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:57 smithi026 ceph-mon[68309]: pgmap v1997: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 active+recovering+undersized+degraded+remapped, 54 active+clean; 3.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 317 op/s; 4104/182967 objects degraded (2.243%); 6269/182967 objects misplaced (3.426%); 3.1 MiB/s, 102 objects/s recovering 2024-03-20T09:28:58.367 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:28:58.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:58.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:28:58.394 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:28:58.402 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T09:28:58.402 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-20T09:28:58.402 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.2 'trim stale osdmaps' 2024-03-20T09:28:58.474 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:28:58.503 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:58.527 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:28:58.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:58.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:28:58.632 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:28:58.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:58 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 4104/182967 objects degraded (2.243%), 2 pgs degraded (PG_DEGRADED) 2024-03-20T09:28:58.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:58 smithi115 ceph-mon[72476]: osdmap e828: 12 total, 11 up, 11 in 2024-03-20T09:28:58.658 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:58.673 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:58 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 4104/182967 objects degraded (2.243%), 2 pgs degraded (PG_DEGRADED) 2024-03-20T09:28:58.674 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:58 smithi026 ceph-mon[68309]: osdmap e828: 12 total, 11 up, 11 in 2024-03-20T09:28:58.679 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:28:58.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:58.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:28:58.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:58 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 4104/182967 objects degraded (2.243%), 2 pgs degraded (PG_DEGRADED) 2024-03-20T09:28:58.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:58 smithi122 ceph-mon[73495]: osdmap e828: 12 total, 11 up, 11 in 2024-03-20T09:28:58.783 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:28:58.826 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:59.089 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:28:59.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:59.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:28:59.200 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:28:59.234 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:28:59.299 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:28:59.388 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:28:59 smithi026 ceph-mon[68309]: pgmap v1999: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 54 active+clean; 3.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 297 op/s; 3992/180798 objects degraded (2.208%); 5974/180798 objects misplaced (3.304%); 1.9 MiB/s, 169 objects/s recovering 2024-03-20T09:28:59.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:28:59 smithi115 ceph-mon[72476]: pgmap v1999: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 54 active+clean; 3.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 297 op/s; 3992/180798 objects degraded (2.208%); 5974/180798 objects misplaced (3.304%); 1.9 MiB/s, 169 objects/s recovering 2024-03-20T09:28:59.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:28:59 smithi122 ceph-mon[73495]: pgmap v1999: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 54 active+clean; 3.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 297 op/s; 3992/180798 objects degraded (2.208%); 5974/180798 objects misplaced (3.304%); 1.9 MiB/s, 169 objects/s recovering 2024-03-20T09:28:59.785 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:28:59.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:28:59.789 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:28:59.808 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-20T09:28:59.825 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:28:59.890 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:28:59.929 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:00.105 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:29:00.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:00 smithi115 ceph-mon[72476]: osdmap e829: 12 total, 11 up, 11 in 2024-03-20T09:29:00.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:00 smithi115 ceph-mon[72476]: pgmap v2001: 57 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 54 active+clean; 3.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 367 op/s; 3722/178542 objects degraded (2.085%); 5704/178542 objects misplaced (3.195%); 1.7 MiB/s, 245 objects/s recovering 2024-03-20T09:29:00.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:00 smithi026 ceph-mon[68309]: osdmap e829: 12 total, 11 up, 11 in 2024-03-20T09:29:00.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:00 smithi026 ceph-mon[68309]: pgmap v2001: 57 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 54 active+clean; 3.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 367 op/s; 3722/178542 objects degraded (2.085%); 5704/178542 objects misplaced (3.195%); 1.7 MiB/s, 245 objects/s recovering 2024-03-20T09:29:00.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:00 smithi122 ceph-mon[73495]: osdmap e829: 12 total, 11 up, 11 in 2024-03-20T09:29:00.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:00 smithi122 ceph-mon[73495]: pgmap v2001: 57 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 54 active+clean; 3.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 367 op/s; 3722/178542 objects degraded (2.085%); 5704/178542 objects misplaced (3.195%); 1.7 MiB/s, 245 objects/s recovering 2024-03-20T09:29:01.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:01 smithi115 ceph-mon[72476]: 1.16 scrub starts 2024-03-20T09:29:01.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:01 smithi115 ceph-mon[72476]: 1.16 scrub ok 2024-03-20T09:29:01.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:01 smithi115 ceph-mon[72476]: osdmap e830: 12 total, 11 up, 11 in 2024-03-20T09:29:01.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:01 smithi122 ceph-mon[73495]: 1.16 scrub starts 2024-03-20T09:29:01.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:01 smithi122 ceph-mon[73495]: 1.16 scrub ok 2024-03-20T09:29:01.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:01 smithi122 ceph-mon[73495]: osdmap e830: 12 total, 11 up, 11 in 2024-03-20T09:29:01.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:01 smithi026 ceph-mon[68309]: 1.16 scrub starts 2024-03-20T09:29:01.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:01 smithi026 ceph-mon[68309]: 1.16 scrub ok 2024-03-20T09:29:01.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:01 smithi026 ceph-mon[68309]: osdmap e830: 12 total, 11 up, 11 in 2024-03-20T09:29:01.988 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:29:01.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:01.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:29:02.096 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:29:02.126 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:02.276 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:29:02.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:02.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:29:02.380 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:29:02.409 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:02.435 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:29:02.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:02.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:29:02.509 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:29:02.510 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [ 2024-03-20T09:29:02.510 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.510 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:182480 6.d 6:b5632f5a:::benchmark_data_smithi122_168655_object62371:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e830)", 2024-03-20T09:29:02.510 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:29:02.485929+0000", 2024-03-20T09:29:02.510 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.028871661, 2024-03-20T09:29:02.510 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.028909626000000001, 2024-03-20T09:29:02.510 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:29:02.510 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:29:02.510 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:29:02.510 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:29:02.510 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:29:02.511 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 182480 2024-03-20T09:29:02.511 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.511 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:29:02.511 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.511 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:29:02.511 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.485929+0000", 2024-03-20T09:29:02.511 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:29:02.511 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.511 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.511 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:29:02.511 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.485929+0000", 2024-03-20T09:29:02.511 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:29:02.511 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.511 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.512 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:29:02.512 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.485930+0000", 2024-03-20T09:29:02.512 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.6599999999999994e-07 2024-03-20T09:29:02.512 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.512 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.512 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:29:02.512 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.485935+0000", 2024-03-20T09:29:02.512 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.843e-06 2024-03-20T09:29:02.512 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.512 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.512 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:29:02.512 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.485944+0000", 2024-03-20T09:29:02.512 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.3989999999999993e-06 2024-03-20T09:29:02.512 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.513 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.513 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:29:02.513 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.485952+0000", 2024-03-20T09:29:02.513 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7659999999999998e-06 2024-03-20T09:29:02.513 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.513 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.513 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:29:02.513 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.485997+0000", 2024-03-20T09:29:02.513 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5346000000000003e-05 2024-03-20T09:29:02.513 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.513 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.513 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:29:02.513 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.486087+0000", 2024-03-20T09:29:02.514 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.0026999999999997e-05 2024-03-20T09:29:02.514 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.514 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.514 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 0,1", 2024-03-20T09:29:02.514 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.486164+0000", 2024-03-20T09:29:02.514 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.6434999999999996e-05 2024-03-20T09:29:02.514 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.514 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.518 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:29:02.518 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.490006+0000", 2024-03-20T09:29:02.518 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.003842177 2024-03-20T09:29:02.518 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.518 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.518 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:29:02.518 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.490104+0000", 2024-03-20T09:29:02.518 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.8197999999999997e-05 2024-03-20T09:29:02.518 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:29:02.518 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:29:02.518 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:29:02.518 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.519 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.519 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:182492 6.7 6:e1c2ed94:::benchmark_data_smithi122_168655_object62383:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e830)", 2024-03-20T09:29:02.519 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:29:02.511845+0000", 2024-03-20T09:29:02.519 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.002955414, 2024-03-20T09:29:02.519 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0030903749999999998, 2024-03-20T09:29:02.519 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:29:02.519 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:29:02.519 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:29:02.519 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:29:02.519 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:29:02.519 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 182492 2024-03-20T09:29:02.519 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.519 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:29:02.519 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.520 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:29:02.520 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.511845+0000", 2024-03-20T09:29:02.520 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:29:02.520 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.520 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.520 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:29:02.520 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.511845+0000", 2024-03-20T09:29:02.520 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:29:02.520 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.520 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.520 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:29:02.520 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.511846+0000", 2024-03-20T09:29:02.520 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.164e-06 2024-03-20T09:29:02.520 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.521 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.521 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:29:02.521 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.511852+0000", 2024-03-20T09:29:02.521 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4530000000000001e-06 2024-03-20T09:29:02.521 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.521 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.521 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:29:02.521 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.511865+0000", 2024-03-20T09:29:02.521 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2731999999999999e-05 2024-03-20T09:29:02.521 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.521 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.521 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:29:02.521 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.511871+0000", 2024-03-20T09:29:02.521 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.6730000000000003e-06 2024-03-20T09:29:02.521 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.522 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.522 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:29:02.522 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.511913+0000", 2024-03-20T09:29:02.522 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.1474000000000003e-05 2024-03-20T09:29:02.522 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.522 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.522 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:29:02.522 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.511996+0000", 2024-03-20T09:29:02.522 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.3410000000000003e-05 2024-03-20T09:29:02.522 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.522 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.522 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 3,4", 2024-03-20T09:29:02.522 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.512061+0000", 2024-03-20T09:29:02.522 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4757000000000004e-05 2024-03-20T09:29:02.523 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:29:02.523 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:29:02.523 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:29:02.523 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.523 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.523 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26216.0:182490 6.c e830/709 6:30c87c43:::benchmark_data_smithi122_168655_object62381:head v 830'11762, mlcod=830'11760)", 2024-03-20T09:29:02.523 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:29:02.512070+0000", 2024-03-20T09:29:02.523 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.0027303499999999999, 2024-03-20T09:29:02.523 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0029395549999999999, 2024-03-20T09:29:02.523 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:29:02.523 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:29:02.523 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:29:02.523 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.523 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:29:02.523 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.512070+0000", 2024-03-20T09:29:02.524 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:29:02.524 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.524 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.524 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:29:02.524 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.512070+0000", 2024-03-20T09:29:02.524 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:29:02.524 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.524 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.524 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:29:02.524 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.512071+0000", 2024-03-20T09:29:02.524 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3099999999999998e-07 2024-03-20T09:29:02.524 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.524 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.524 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:29:02.524 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.512075+0000", 2024-03-20T09:29:02.525 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.3379999999999998e-06 2024-03-20T09:29:02.525 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.525 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.525 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:29:02.525 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.512080+0000", 2024-03-20T09:29:02.525 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1399999999999999e-06 2024-03-20T09:29:02.525 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.525 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.525 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:29:02.525 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.512084+0000", 2024-03-20T09:29:02.525 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.433e-06 2024-03-20T09:29:02.525 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.525 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.525 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:29:02.525 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.512120+0000", 2024-03-20T09:29:02.526 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6040000000000001e-05 2024-03-20T09:29:02.526 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.526 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.526 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:29:02.526 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.512136+0000", 2024-03-20T09:29:02.526 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5667000000000001e-05 2024-03-20T09:29:02.526 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:29:02.526 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:29:02.526 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:29:02.526 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.526 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.526 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26216.0:182491 6.4 e830/749 6:226f6b85:::benchmark_data_smithi122_168655_object62382:head v 830'9804, mlcod=830'9802)", 2024-03-20T09:29:02.526 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:29:02.512243+0000", 2024-03-20T09:29:02.527 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.0025581419999999998, 2024-03-20T09:29:02.527 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0028267040000000002, 2024-03-20T09:29:02.527 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:29:02.527 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:29:02.527 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:29:02.527 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.527 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:29:02.527 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.512243+0000", 2024-03-20T09:29:02.527 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:29:02.527 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.527 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.527 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:29:02.527 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.512243+0000", 2024-03-20T09:29:02.528 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:29:02.528 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.528 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.528 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:29:02.528 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.512243+0000", 2024-03-20T09:29:02.528 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0800000000000004e-07 2024-03-20T09:29:02.528 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.528 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.528 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:29:02.528 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.512247+0000", 2024-03-20T09:29:02.528 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.2549999999999999e-06 2024-03-20T09:29:02.528 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.528 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.528 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:29:02.529 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.512257+0000", 2024-03-20T09:29:02.529 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.3980000000000005e-06 2024-03-20T09:29:02.529 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.529 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.529 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:29:02.529 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.512263+0000", 2024-03-20T09:29:02.529 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.5830000000000003e-06 2024-03-20T09:29:02.529 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.529 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.529 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:29:02.529 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.512362+0000", 2024-03-20T09:29:02.529 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.8788000000000004e-05 2024-03-20T09:29:02.529 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.529 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.529 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:29:02.530 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.512380+0000", 2024-03-20T09:29:02.530 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7878000000000001e-05 2024-03-20T09:29:02.530 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:29:02.530 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:29:02.530 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:29:02.530 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.530 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.530 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:182463 6.d 6:b9593923:::benchmark_data_smithi122_168655_object62354:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e830)", 2024-03-20T09:29:02.530 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:29:02.450698+0000", 2024-03-20T09:29:02.530 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.064102969999999995, 2024-03-20T09:29:02.530 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.064441463000000004, 2024-03-20T09:29:02.530 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:29:02.530 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:29:02.530 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:29:02.531 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:29:02.531 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:29:02.531 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 182463 2024-03-20T09:29:02.531 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.531 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:29:02.531 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.531 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:29:02.531 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.450698+0000", 2024-03-20T09:29:02.531 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:29:02.531 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.531 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.531 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:29:02.531 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.450698+0000", 2024-03-20T09:29:02.531 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:29:02.531 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.532 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.532 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:29:02.532 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.450699+0000", 2024-03-20T09:29:02.532 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.068e-06 2024-03-20T09:29:02.532 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.532 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.532 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:29:02.532 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.450704+0000", 2024-03-20T09:29:02.532 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.7509999999999999e-06 2024-03-20T09:29:02.532 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.532 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.532 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:29:02.532 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.450715+0000", 2024-03-20T09:29:02.532 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1348e-05 2024-03-20T09:29:02.532 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.533 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.533 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:29:02.533 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.450721+0000", 2024-03-20T09:29:02.533 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0050000000000001e-06 2024-03-20T09:29:02.533 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.533 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.533 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:29:02.533 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.450774+0000", 2024-03-20T09:29:02.533 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3103000000000003e-05 2024-03-20T09:29:02.533 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.533 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.533 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:29:02.533 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.450909+0000", 2024-03-20T09:29:02.533 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000134466 2024-03-20T09:29:02.533 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.533 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.534 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 0,1", 2024-03-20T09:29:02.534 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.451036+0000", 2024-03-20T09:29:02.534 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000127713 2024-03-20T09:29:02.534 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.534 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.534 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:29:02.534 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.455791+0000", 2024-03-20T09:29:02.534 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0047548870000000002 2024-03-20T09:29:02.534 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:29:02.534 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:29:02.534 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:29:02.534 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:02.456891+0000", 2024-03-20T09:29:02.534 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0010997470000000001 2024-03-20T09:29:02.534 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:29:02.535 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:29:02.535 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:29:02.535 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:29:02.535 INFO:teuthology.orchestra.run.smithi115.stdout: ], 2024-03-20T09:29:02.535 INFO:teuthology.orchestra.run.smithi115.stdout: "num_ops": 5 2024-03-20T09:29:02.535 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:29:02.541 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:29:02.601 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:02.684 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:29:02.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:02 smithi026 ceph-mon[68309]: pgmap v2003: 57 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 54 active+clean; 3.6 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 365 op/s; 3722/176388 objects degraded (2.110%); 5704/176388 objects misplaced (3.234%); 1.9 MiB/s, 276 objects/s recovering 2024-03-20T09:29:02.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:02 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 3722/178542 objects degraded (2.085%), 1 pg degraded (PG_DEGRADED) 2024-03-20T09:29:02.717 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:29:02.755 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:29:02.785 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:29:02.812 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:29:02.837 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:29:02.838 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:29:02.865 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:29:02.872 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:29:02.873 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:02 smithi115 ceph-mon[72476]: pgmap v2003: 57 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 54 active+clean; 3.6 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 365 op/s; 3722/176388 objects degraded (2.110%); 5704/176388 objects misplaced (3.234%); 1.9 MiB/s, 276 objects/s recovering 2024-03-20T09:29:02.873 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:02 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 3722/178542 objects degraded (2.085%), 1 pg degraded (PG_DEGRADED) 2024-03-20T09:29:02.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:02.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:29:02.888 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:29:02.907 INFO:teuthology.orchestra.run.smithi026.stderr:no valid command found; 10 closest matches: 2024-03-20T09:29:02.907 INFO:teuthology.orchestra.run.smithi026.stderr:0 2024-03-20T09:29:02.907 INFO:teuthology.orchestra.run.smithi026.stderr:1 2024-03-20T09:29:02.907 INFO:teuthology.orchestra.run.smithi026.stderr:2 2024-03-20T09:29:02.907 INFO:teuthology.orchestra.run.smithi026.stderr:abort 2024-03-20T09:29:02.907 INFO:teuthology.orchestra.run.smithi026.stderr:assert 2024-03-20T09:29:02.908 INFO:teuthology.orchestra.run.smithi026.stderr:bench [] [] [] [] 2024-03-20T09:29:02.908 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs debug_inject_read_zeros 2024-03-20T09:29:02.908 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs files list 2024-03-20T09:29:02.908 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs stats 2024-03-20T09:29:02.908 INFO:teuthology.orchestra.run.smithi026.stderr:bluestore allocator dump block 2024-03-20T09:29:02.908 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: invalid command 2024-03-20T09:29:02.909 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:29:02.936 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:29:02.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:02 smithi122 ceph-mon[73495]: pgmap v2003: 57 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 54 active+clean; 3.6 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 365 op/s; 3722/176388 objects degraded (2.110%); 5704/176388 objects misplaced (3.234%); 1.9 MiB/s, 276 objects/s recovering 2024-03-20T09:29:02.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:02 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 3722/178542 objects degraded (2.085%), 1 pg degraded (PG_DEGRADED) 2024-03-20T09:29:02.962 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:29:02.962 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:29:02.963 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:29:02.963 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:29:02.963 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:29:02.963 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:29:02.963 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:29:02.963 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:29:02.963 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:29:02.963 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:29:02.963 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:29:02.963 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-03-20T09:29:02.963 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:29:02.978 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:29:03.010 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:03.220 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:29:03.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:03.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:29:03.231 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-20T09:29:03.327 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:29:03.356 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:03.376 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:29:03.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:03.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:29:03.480 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:29:03.514 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:03.542 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:29:03.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:03.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:29:03.589 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:29:03.646 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:29:03.674 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:03.693 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:29:03.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:03.696 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:29:03.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:03 smithi026 ceph-mon[68309]: 1.1d deep-scrub starts 2024-03-20T09:29:03.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:03 smithi026 ceph-mon[68309]: 1.1d deep-scrub ok 2024-03-20T09:29:03.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:03 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2761431491' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:29:03.796 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:29:03.822 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:03 smithi115 ceph-mon[72476]: 1.1d deep-scrub starts 2024-03-20T09:29:03.822 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:03 smithi115 ceph-mon[72476]: 1.1d deep-scrub ok 2024-03-20T09:29:03.822 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:03 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2761431491' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:29:03.824 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:03.863 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:29:03.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:03.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:29:03.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:03 smithi122 ceph-mon[73495]: 1.1d deep-scrub starts 2024-03-20T09:29:03.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:03 smithi122 ceph-mon[73495]: 1.1d deep-scrub ok 2024-03-20T09:29:03.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:03 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2761431491' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:29:03.968 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:29:03.996 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:04.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:04 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:29:04.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:04 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:29:04.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:04 smithi026 ceph-mon[68309]: 6.4 deep-scrub starts 2024-03-20T09:29:04.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:04 smithi026 ceph-mon[68309]: osdmap e831: 12 total, 11 up, 11 in 2024-03-20T09:29:04.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:04 smithi026 ceph-mon[68309]: pgmap v2005: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 peering, 54 active+clean; 3.6 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 296 op/s; 3588/175458 objects degraded (2.045%); 5521/175458 objects misplaced (3.147%); 714 KiB/s, 117 objects/s recovering 2024-03-20T09:29:04.891 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T09:29:04.891 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-20T09:29:04.891 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.5 'trim stale osdmaps' 2024-03-20T09:29:04.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:04 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:29:04.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:04 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:29:04.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:04 smithi115 ceph-mon[72476]: 6.4 deep-scrub starts 2024-03-20T09:29:04.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:04 smithi115 ceph-mon[72476]: osdmap e831: 12 total, 11 up, 11 in 2024-03-20T09:29:04.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:04 smithi115 ceph-mon[72476]: pgmap v2005: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 peering, 54 active+clean; 3.6 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 296 op/s; 3588/175458 objects degraded (2.045%); 5521/175458 objects misplaced (3.147%); 714 KiB/s, 117 objects/s recovering 2024-03-20T09:29:04.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:04 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:29:04.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:04 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:29:04.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:04 smithi122 ceph-mon[73495]: 6.4 deep-scrub starts 2024-03-20T09:29:04.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:04 smithi122 ceph-mon[73495]: osdmap e831: 12 total, 11 up, 11 in 2024-03-20T09:29:04.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:04 smithi122 ceph-mon[73495]: pgmap v2005: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 peering, 54 active+clean; 3.6 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 296 op/s; 3588/175458 objects degraded (2.045%); 5521/175458 objects misplaced (3.147%); 714 KiB/s, 117 objects/s recovering 2024-03-20T09:29:05.913 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:29:05.946 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:29:05.946 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:29:06.849 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:29:06.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:06.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:29:06.955 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:29:06.983 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:07.361 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:29:07.444 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:29:07.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:07.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:29:07.548 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:29:07.576 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:07.594 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:29:07.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:07.597 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:29:07.634 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:07 smithi026 ceph-mon[68309]: pgmap v2006: 57 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 54 active+clean; 3.5 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 457 op/s; 3588/171495 objects degraded (2.092%); 5521/171495 objects misplaced (3.219%); 633 KiB/s, 104 objects/s recovering 2024-03-20T09:29:07.634 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:07 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1602188289' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:29:07.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:07 smithi115 ceph-mon[72476]: pgmap v2006: 57 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 54 active+clean; 3.5 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 457 op/s; 3588/171495 objects degraded (2.092%); 5521/171495 objects misplaced (3.219%); 633 KiB/s, 104 objects/s recovering 2024-03-20T09:29:07.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:07 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1602188289' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:29:07.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:07 smithi122 ceph-mon[73495]: pgmap v2006: 57 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 54 active+clean; 3.5 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 457 op/s; 3588/171495 objects degraded (2.092%); 5521/171495 objects misplaced (3.219%); 633 KiB/s, 104 objects/s recovering 2024-03-20T09:29:07.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:07 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1602188289' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:29:07.698 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:29:07.762 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:07.947 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:29:08.112 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:29:08.113 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:29:08.400 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:08 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:29:08.400 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:08 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1602188289' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:29:08.400 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:08 smithi115 ceph-mon[72476]: osdmap e832: 12 total, 11 up, 11 in 2024-03-20T09:29:08.401 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:08 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 3588/171495 objects degraded (2.092%), 1 pg degraded (PG_DEGRADED) 2024-03-20T09:29:08.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:08 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:29:08.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:08 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1602188289' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:29:08.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:08 smithi122 ceph-mon[73495]: osdmap e832: 12 total, 11 up, 11 in 2024-03-20T09:29:08.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:08 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 3588/171495 objects degraded (2.092%), 1 pg degraded (PG_DEGRADED) 2024-03-20T09:29:08.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:08 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:29:08.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:08 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1602188289' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:29:08.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:08 smithi026 ceph-mon[68309]: osdmap e832: 12 total, 11 up, 11 in 2024-03-20T09:29:08.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:08 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 3588/171495 objects degraded (2.092%), 1 pg degraded (PG_DEGRADED) 2024-03-20T09:29:08.914 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-20T09:29:09.237 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:29:09.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:09 smithi115 ceph-mon[72476]: pgmap v2008: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+scrubbing+deep, 54 active+clean; 3.4 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 396 op/s; 3454/169299 objects degraded (2.040%); 5348/169299 objects misplaced (3.159%); 1.0 MiB/s, 16 objects/s recovering 2024-03-20T09:29:09.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:09 smithi122 ceph-mon[73495]: pgmap v2008: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+scrubbing+deep, 54 active+clean; 3.4 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 396 op/s; 3454/169299 objects degraded (2.040%); 5348/169299 objects misplaced (3.159%); 1.0 MiB/s, 16 objects/s recovering 2024-03-20T09:29:09.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:09 smithi026 ceph-mon[68309]: pgmap v2008: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+scrubbing+deep, 54 active+clean; 3.4 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 396 op/s; 3454/169299 objects degraded (2.040%); 5348/169299 objects misplaced (3.159%); 1.0 MiB/s, 16 objects/s recovering 2024-03-20T09:29:09.828 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:29:09.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:09.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:29:09.936 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:29:09.984 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:10.062 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:29:10.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:10.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:29:10.171 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:29:10.204 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:10.253 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:29:10.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:10.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:29:10.360 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:29:10.387 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:10.406 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:29:10.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:10.409 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:29:10.510 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:29:10.520 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:29:10.545 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:10.571 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:29:10.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:10.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:29:10.676 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:29:10.706 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:10.857 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:29:10.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:10.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:29:10.964 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:29:10.965 INFO:teuthology.orchestra.run.smithi026.stderr:no valid command found; 10 closest matches: 2024-03-20T09:29:10.965 INFO:teuthology.orchestra.run.smithi026.stderr:0 2024-03-20T09:29:10.965 INFO:teuthology.orchestra.run.smithi026.stderr:1 2024-03-20T09:29:10.965 INFO:teuthology.orchestra.run.smithi026.stderr:2 2024-03-20T09:29:10.965 INFO:teuthology.orchestra.run.smithi026.stderr:abort 2024-03-20T09:29:10.965 INFO:teuthology.orchestra.run.smithi026.stderr:assert 2024-03-20T09:29:10.965 INFO:teuthology.orchestra.run.smithi026.stderr:bench [] [] [] [] 2024-03-20T09:29:10.965 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs debug_inject_read_zeros 2024-03-20T09:29:10.965 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs files list 2024-03-20T09:29:10.965 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs stats 2024-03-20T09:29:10.965 INFO:teuthology.orchestra.run.smithi026.stderr:bluestore allocator dump block 2024-03-20T09:29:10.965 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: invalid command 2024-03-20T09:29:10.996 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:11.018 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:29:11.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:11.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:29:11.121 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:29:11.149 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:11.168 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:29:11.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:11.171 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:29:11.257 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:29:11.257 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:29:11.272 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:29:11.309 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:11.440 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:11 smithi122 ceph-mon[73495]: pgmap v2009: 57 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 54 active+clean; 3.4 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 377 op/s; 3454/167334 objects degraded (2.064%); 5348/167334 objects misplaced (3.196%); 1.0 MiB/s, 16 objects/s recovering 2024-03-20T09:29:11.440 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:11 smithi122 ceph-mon[73495]: 6.4 deep-scrub ok 2024-03-20T09:29:11.616 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:11 smithi115 ceph-mon[72476]: pgmap v2009: 57 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 54 active+clean; 3.4 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 377 op/s; 3454/167334 objects degraded (2.064%); 5348/167334 objects misplaced (3.196%); 1.0 MiB/s, 16 objects/s recovering 2024-03-20T09:29:11.616 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:11 smithi115 ceph-mon[72476]: 6.4 deep-scrub ok 2024-03-20T09:29:11.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:11 smithi026 ceph-mon[68309]: pgmap v2009: 57 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 54 active+clean; 3.4 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 377 op/s; 3454/167334 objects degraded (2.064%); 5348/167334 objects misplaced (3.196%); 1.0 MiB/s, 16 objects/s recovering 2024-03-20T09:29:11.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:11 smithi026 ceph-mon[68309]: 6.4 deep-scrub ok 2024-03-20T09:29:11.768 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:29:11.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:11.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:29:11.869 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:29:11.878 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:29:11.961 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:11.984 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-20T09:29:13.033 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:29:13.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:13.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:29:13.138 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:29:13.166 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:13.184 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:29:13.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:13.188 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:29:13.289 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:29:13.317 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:13.341 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:29:13.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:13.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:29:13.440 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:13 smithi122 ceph-mon[73495]: pgmap v2010: 57 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 54 active+clean; 3.4 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 377 op/s; 3454/165585 objects degraded (2.086%); 5348/165585 objects misplaced (3.230%); 462 KiB/s, 7 objects/s recovering 2024-03-20T09:29:13.440 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:13 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:29:13.440 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:13 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:29:13.440 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:13 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 3454/165585 objects degraded (2.086%), 1 pg degraded (PG_DEGRADED) 2024-03-20T09:29:13.446 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:29:13.474 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:13.495 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:29:13.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:13.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:29:13.595 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T09:29:13.596 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-20T09:29:13.596 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.6 'trim stale osdmaps' 2024-03-20T09:29:13.599 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:29:13.627 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:13 smithi026 ceph-mon[68309]: pgmap v2010: 57 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 54 active+clean; 3.4 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 377 op/s; 3454/165585 objects degraded (2.086%); 5348/165585 objects misplaced (3.230%); 462 KiB/s, 7 objects/s recovering 2024-03-20T09:29:13.627 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:13 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:29:13.627 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:13 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:29:13.627 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:13 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 3454/165585 objects degraded (2.086%), 1 pg degraded (PG_DEGRADED) 2024-03-20T09:29:13.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:13 smithi115 ceph-mon[72476]: pgmap v2010: 57 pgs: 1 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 54 active+clean; 3.4 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 377 op/s; 3454/165585 objects degraded (2.086%); 5348/165585 objects misplaced (3.230%); 462 KiB/s, 7 objects/s recovering 2024-03-20T09:29:13.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:13 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:29:13.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:13 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:29:13.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:13 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 3454/165585 objects degraded (2.086%), 1 pg degraded (PG_DEGRADED) 2024-03-20T09:29:13.662 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:13.799 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:29:13.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:13.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:29:13.904 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:29:13.931 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:13.950 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:29:13.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:13.954 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:29:14.055 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:29:14.093 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:14.141 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:29:14.142 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:29:14.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:14.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:29:14.209 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:29:14.247 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:29:14.281 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:14.283 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:29:14.310 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:29:14.348 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:29:14.389 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:14 smithi026 ceph-mon[68309]: osdmap e833: 12 total, 11 up, 11 in 2024-03-20T09:29:14.390 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:14 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3133554333' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:29:14.390 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:29:14.414 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:29:14.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:14.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:29:14.425 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:29:14.456 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:29:14.485 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:29:14.508 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:29:14.520 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:29:14.538 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:29:14.577 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:14.582 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:29:14.629 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:29:14.629 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:29:14.629 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:29:14.629 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:29:14.629 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:29:14.629 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:29:14.629 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:29:14.630 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:29:14.630 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:29:14.630 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:29:14.630 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:29:14.630 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-03-20T09:29:14.630 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:29:14.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:14 smithi115 ceph-mon[72476]: osdmap e833: 12 total, 11 up, 11 in 2024-03-20T09:29:14.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:14 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3133554333' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:29:14.672 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:29:14.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:14.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:29:14.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:14 smithi122 ceph-mon[73495]: osdmap e833: 12 total, 11 up, 11 in 2024-03-20T09:29:14.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:14 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3133554333' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:29:14.783 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:29:14.809 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:14.827 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:29:14.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:14.830 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:29:14.930 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:29:14.957 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:14.977 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:29:14.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:14.982 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:29:15.083 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:29:15.113 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:15.385 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:15 smithi026 ceph-mon[68309]: pgmap v2012: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+scrubbing+deep, 54 active+clean; 3.4 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 279 op/s; 3342/164778 objects degraded (2.028%); 5164/164778 objects misplaced (3.134%); 1.0 MiB/s, 16 objects/s recovering 2024-03-20T09:29:15.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:15 smithi115 ceph-mon[72476]: pgmap v2012: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+scrubbing+deep, 54 active+clean; 3.4 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 279 op/s; 3342/164778 objects degraded (2.028%); 5164/164778 objects misplaced (3.134%); 1.0 MiB/s, 16 objects/s recovering 2024-03-20T09:29:15.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:15 smithi122 ceph-mon[73495]: pgmap v2012: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+scrubbing+deep, 54 active+clean; 3.4 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 279 op/s; 3342/164778 objects degraded (2.028%); 5164/164778 objects misplaced (3.134%); 1.0 MiB/s, 16 objects/s recovering 2024-03-20T09:29:16.829 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:29:16.879 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:29:16.880 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:29:17.274 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:29:17.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:17.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:29:17.378 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:29:17.404 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:17.422 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:29:17.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:17.428 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:29:17.529 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:29:17.557 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:17.596 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:29:17.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:17.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:29:17.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:17 smithi115 ceph-mon[72476]: pgmap v2013: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 3.3 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 3342/160743 objects degraded (2.079%); 5164/160743 objects misplaced (3.213%); 472 KiB/s, 7 objects/s recovering 2024-03-20T09:29:17.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:17 smithi122 ceph-mon[73495]: pgmap v2013: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 3.3 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 3342/160743 objects degraded (2.079%); 5164/160743 objects misplaced (3.213%); 472 KiB/s, 7 objects/s recovering 2024-03-20T09:29:17.708 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:29:17.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:17 smithi026 ceph-mon[68309]: pgmap v2013: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 3.3 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 3342/160743 objects degraded (2.079%); 5164/160743 objects misplaced (3.213%); 472 KiB/s, 7 objects/s recovering 2024-03-20T09:29:17.734 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:17.753 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:29:17.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:17.756 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:29:17.857 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:29:17.892 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:18.353 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:29:18.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:18.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:29:18.409 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:18 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 3342/160743 objects degraded (2.079%), 1 pg degraded (PG_DEGRADED) 2024-03-20T09:29:18.410 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:18 smithi026 ceph-mon[68309]: osdmap e834: 12 total, 11 up, 11 in 2024-03-20T09:29:18.459 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:29:18.490 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:18.568 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:29:18.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:18.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:29:18.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:18 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 3342/160743 objects degraded (2.079%), 1 pg degraded (PG_DEGRADED) 2024-03-20T09:29:18.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:18 smithi115 ceph-mon[72476]: osdmap e834: 12 total, 11 up, 11 in 2024-03-20T09:29:18.672 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:29:18.800 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:18 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 3342/160743 objects degraded (2.079%), 1 pg degraded (PG_DEGRADED) 2024-03-20T09:29:18.800 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:18 smithi122 ceph-mon[73495]: osdmap e834: 12 total, 11 up, 11 in 2024-03-20T09:29:18.860 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:18.880 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:29:18.992 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:29:18.993 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:29:18.993 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:18.993 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(6.17 835/812 [PushOp(6:e9ed6660:::benchmark_data_smithi122_168655_object68521:head, version: 728'4384, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:e9ed6660:::benchmark_data_smithi122_168655_object68521:head@728'4384, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-20T09:29:18.993 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:29:18.984956+0000", 2024-03-20T09:29:18.993 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0026393950000000001, 2024-03-20T09:29:18.994 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026949399999999998, 2024-03-20T09:29:18.994 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:29:18.994 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:29:18.994 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:29:18.994 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:18.994 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:29:18.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:29:18.984956+0000", 2024-03-20T09:29:18.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:18.995 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:18.995 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:18.995 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:29:18.995 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:29:18.984956+0000", 2024-03-20T09:29:18.995 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:18.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:18.996 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:18.996 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:29:18.996 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:29:18.984956+0000", 2024-03-20T09:29:18.996 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8899999999999998e-07 2024-03-20T09:29:18.996 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:18.997 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:18.997 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:29:18.998 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:29:18.984989+0000", 2024-03-20T09:29:18.998 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2301999999999997e-05 2024-03-20T09:29:18.998 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:18.998 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:18.998 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:29:18.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:29:18.985013+0000", 2024-03-20T09:29:18.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3796000000000001e-05 2024-03-20T09:29:18.999 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:18.999 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:18.999 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:29:18.999 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:29:18.985031+0000", 2024-03-20T09:29:18.999 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8289999999999999e-05 2024-03-20T09:29:19.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:19.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:19.000 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:19.000 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:29:18.985138+0000", 2024-03-20T09:29:19.000 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107456 2024-03-20T09:29:19.000 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:19.000 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:19.001 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:29:19.001 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:29:18.985154+0000", 2024-03-20T09:29:19.001 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5574999999999999e-05 2024-03-20T09:29:19.001 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:19.001 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:29:19.001 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:19.001 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:19.002 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:19.002 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26216.0:189008 6.9 6:9373461f:::benchmark_data_smithi122_168655_object68899:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e835)", 2024-03-20T09:29:19.002 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:29:18.987292+0000", 2024-03-20T09:29:19.002 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.00030292600000000002, 2024-03-20T09:29:19.002 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00044636099999999999, 2024-03-20T09:29:19.002 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:29:19.003 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:29:19.003 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:29:19.003 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26216", 2024-03-20T09:29:19.003 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:29:19.003 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 189008 2024-03-20T09:29:19.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:19.004 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:29:19.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:19.004 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:29:19.004 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:29:18.987292+0000", 2024-03-20T09:29:19.004 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:19.004 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:19.004 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:29:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:29:18.987292+0000", 2024-03-20T09:29:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:29:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:29:18.987293+0000", 2024-03-20T09:29:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9999999999999995e-07 2024-03-20T09:29:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:29:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:29:18.987298+0000", 2024-03-20T09:29:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7389999999999999e-06 2024-03-20T09:29:19.005 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:19.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:19.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:29:19.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:29:18.987305+0000", 2024-03-20T09:29:19.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3690000000000001e-06 2024-03-20T09:29:19.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:19.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:19.006 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:29:19.006 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:29:18.987310+0000", 2024-03-20T09:29:19.006 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.22e-06 2024-03-20T09:29:19.006 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:19.006 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:19.007 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:19.007 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:29:18.987363+0000", 2024-03-20T09:29:19.007 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2266999999999999e-05 2024-03-20T09:29:19.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:19.008 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:19.008 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:29:19.008 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:29:18.987477+0000", 2024-03-20T09:29:19.008 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011437100000000001 2024-03-20T09:29:19.008 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:19.009 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:19.009 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 8,9", 2024-03-20T09:29:19.009 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:29:18.987580+0000", 2024-03-20T09:29:19.009 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000102845 2024-03-20T09:29:19.009 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:19.009 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:29:19.009 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:19.009 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:19.009 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:29:19.009 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 2 2024-03-20T09:29:19.009 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:29:19.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:19 smithi115 ceph-mon[72476]: pgmap v2015: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 3.2 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3212/158283 objects degraded (2.029%); 4961/158283 objects misplaced (3.134%); 1.1 MiB/s, 17 objects/s recovering 2024-03-20T09:29:19.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:19 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:29:19.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:19 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:29:19.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:19 smithi115 ceph-mon[72476]: osdmap e835: 12 total, 11 up, 11 in 2024-03-20T09:29:19.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:19 smithi122 ceph-mon[73495]: pgmap v2015: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 3.2 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3212/158283 objects degraded (2.029%); 4961/158283 objects misplaced (3.134%); 1.1 MiB/s, 17 objects/s recovering 2024-03-20T09:29:19.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:19 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:29:19.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:19 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:29:19.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:19 smithi122 ceph-mon[73495]: osdmap e835: 12 total, 11 up, 11 in 2024-03-20T09:29:19.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:19 smithi026 ceph-mon[68309]: pgmap v2015: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 3.2 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3212/158283 objects degraded (2.029%); 4961/158283 objects misplaced (3.134%); 1.1 MiB/s, 17 objects/s recovering 2024-03-20T09:29:19.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:19 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:29:19.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:19 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:29:19.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:19 smithi026 ceph-mon[68309]: osdmap e835: 12 total, 11 up, 11 in 2024-03-20T09:29:20.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:20 smithi115 ceph-mon[72476]: pgmap v2017: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 3.2 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3212/155262 objects degraded (2.069%); 4961/155262 objects misplaced (3.195%); 666 KiB/s, 10 objects/s recovering 2024-03-20T09:29:20.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:20 smithi122 ceph-mon[73495]: pgmap v2017: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 3.2 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3212/155262 objects degraded (2.069%); 4961/155262 objects misplaced (3.195%); 666 KiB/s, 10 objects/s recovering 2024-03-20T09:29:20.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:20 smithi026 ceph-mon[68309]: pgmap v2017: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 3.2 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 3212/155262 objects degraded (2.069%); 4961/155262 objects misplaced (3.195%); 666 KiB/s, 10 objects/s recovering 2024-03-20T09:29:21.408 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:29:21.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:21 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2873831179' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:29:21.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:21 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:29:21.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:21 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2873831179' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:29:21.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:21 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:29:21.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:21 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2873831179' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:29:21.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:21 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:29:21.830 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:29:21.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:21.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:29:21.939 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:29:21.992 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:22.430 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:22 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:29:22.430 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:22 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:29:22.430 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:22 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:29:22.430 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:22 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:29:22.430 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:22 smithi026 ceph-mon[68309]: osdmap e836: 12 total, 11 up, 11 in 2024-03-20T09:29:22.430 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:22 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:29:22.431 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:22 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:29:22.431 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:22 smithi026 ceph-mon[68309]: pgmap v2019: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 3.1 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 457 op/s; 3212/152505 objects degraded (2.106%); 4961/152505 objects misplaced (3.253%); 768 KiB/s, 11 objects/s recovering 2024-03-20T09:29:22.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:22 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:29:22.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:22 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:29:22.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:22 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:29:22.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:22 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:29:22.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:22 smithi122 ceph-mon[73495]: osdmap e836: 12 total, 11 up, 11 in 2024-03-20T09:29:22.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:22 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:29:22.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:22 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:29:22.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:22 smithi122 ceph-mon[73495]: pgmap v2019: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 3.1 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 457 op/s; 3212/152505 objects degraded (2.106%); 4961/152505 objects misplaced (3.253%); 768 KiB/s, 11 objects/s recovering 2024-03-20T09:29:22.749 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:29:22.755 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:29:22.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:22.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:29:22.788 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-20T09:29:22.860 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:29:22.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:22 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:29:22.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:22 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:29:22.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:22 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:29:22.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:22 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:29:22.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:22 smithi115 ceph-mon[72476]: osdmap e836: 12 total, 11 up, 11 in 2024-03-20T09:29:22.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:22 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:29:22.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:22 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:29:22.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:22 smithi115 ceph-mon[72476]: pgmap v2019: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 3.1 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 457 op/s; 3212/152505 objects degraded (2.106%); 4961/152505 objects misplaced (3.253%); 768 KiB/s, 11 objects/s recovering 2024-03-20T09:29:22.902 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:23.052 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:29:23.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:23.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:29:23.156 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:29:23.184 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:23.243 INFO:teuthology.orchestra.run.smithi026.stderr:no valid command found; 10 closest matches: 2024-03-20T09:29:23.243 INFO:teuthology.orchestra.run.smithi026.stderr:0 2024-03-20T09:29:23.243 INFO:teuthology.orchestra.run.smithi026.stderr:1 2024-03-20T09:29:23.243 INFO:teuthology.orchestra.run.smithi026.stderr:2 2024-03-20T09:29:23.243 INFO:teuthology.orchestra.run.smithi026.stderr:abort 2024-03-20T09:29:23.243 INFO:teuthology.orchestra.run.smithi026.stderr:assert 2024-03-20T09:29:23.243 INFO:teuthology.orchestra.run.smithi026.stderr:bench [] [] [] [] 2024-03-20T09:29:23.243 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs debug_inject_read_zeros 2024-03-20T09:29:23.243 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs files list 2024-03-20T09:29:23.243 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs stats 2024-03-20T09:29:23.243 INFO:teuthology.orchestra.run.smithi026.stderr:bluestore allocator dump block 2024-03-20T09:29:23.243 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: invalid command 2024-03-20T09:29:23.681 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:23 smithi026 ceph-mon[68309]: osdmap e837: 12 total, 11 up, 11 in 2024-03-20T09:29:23.681 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:23 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 3212/152505 objects degraded (2.106%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:29:23.681 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:23 smithi026 ceph-mon[68309]: osdmap e838: 12 total, 11 up, 11 in 2024-03-20T09:29:23.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:23 smithi122 ceph-mon[73495]: osdmap e837: 12 total, 11 up, 11 in 2024-03-20T09:29:23.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:23 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 3212/152505 objects degraded (2.106%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:29:23.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:23 smithi122 ceph-mon[73495]: osdmap e838: 12 total, 11 up, 11 in 2024-03-20T09:29:23.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:23 smithi115 ceph-mon[72476]: osdmap e837: 12 total, 11 up, 11 in 2024-03-20T09:29:23.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:23 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 3212/152505 objects degraded (2.106%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:29:23.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:23 smithi115 ceph-mon[72476]: osdmap e838: 12 total, 11 up, 11 in 2024-03-20T09:29:24.166 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:29:24.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:24.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:29:24.203 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:29:24.205 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T09:29:24.206 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-20T09:29:24.206 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.7 'trim stale osdmaps' 2024-03-20T09:29:24.273 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:29:24.336 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:24.379 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:29:24.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:24.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:29:24.437 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:24 smithi026 ceph-mon[68309]: pgmap v2022: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 3.1 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 389 op/s; 3081/151545 objects degraded (2.033%); 4764/151545 objects misplaced (3.144%); 765 KiB/s, 11 objects/s recovering 2024-03-20T09:29:24.437 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:29:24.481 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:29:24.508 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:24.527 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:29:24.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:24.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:29:24.631 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:29:24.678 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:24.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:24 smithi122 ceph-mon[73495]: pgmap v2022: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 3.1 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 389 op/s; 3081/151545 objects degraded (2.033%); 4764/151545 objects misplaced (3.144%); 765 KiB/s, 11 objects/s recovering 2024-03-20T09:29:24.699 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:29:24.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:24.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:29:24.810 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:29:24.843 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:24.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:24 smithi115 ceph-mon[72476]: pgmap v2022: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 3.1 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 389 op/s; 3081/151545 objects degraded (2.033%); 4764/151545 objects misplaced (3.144%); 765 KiB/s, 11 objects/s recovering 2024-03-20T09:29:25.500 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:29:25.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:25.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:29:25.605 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:29:25.632 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:25.662 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:29:25.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:25.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:29:25.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:25 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:29:25.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:25 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:29:25.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:25 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:29:25.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:25 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:29:25.771 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:29:25.796 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:25 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:29:25.796 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:25 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:29:25.798 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:25.823 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:29:25.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:25.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:29:25.928 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:29:25.954 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:25.974 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:29:25.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:25.977 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:29:26.078 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:29:26.107 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:26.127 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:29:26.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:26.130 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:29:26.231 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:29:26.266 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:26.298 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:29:26.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:26.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:29:26.402 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:29:26.431 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:26.459 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:29:26.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:26.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:29:26.565 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:29:26.591 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:26.610 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:29:26.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:26.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:29:26.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:26 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:29:26.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:26 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:29:26.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:26 smithi122 ceph-mon[73495]: osdmap e839: 12 total, 11 up, 11 in 2024-03-20T09:29:26.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:26 smithi122 ceph-mon[73495]: pgmap v2024: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 3.0 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 319 op/s; 3081/148290 objects degraded (2.078%); 4764/148290 objects misplaced (3.213%); 902 KiB/s, 14 objects/s recovering 2024-03-20T09:29:26.708 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:29:26.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:26 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:29:26.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:26 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:29:26.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:26 smithi026 ceph-mon[68309]: osdmap e839: 12 total, 11 up, 11 in 2024-03-20T09:29:26.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:26 smithi026 ceph-mon[68309]: pgmap v2024: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 3.0 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 319 op/s; 3081/148290 objects degraded (2.078%); 4764/148290 objects misplaced (3.213%); 902 KiB/s, 14 objects/s recovering 2024-03-20T09:29:26.715 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:29:26.740 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:29:26.741 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:26.761 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:29:26.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:26.765 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:29:26.774 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:29:26.800 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:29:26.823 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:29:26.853 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:29:26.866 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:29:26.886 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:29:26.891 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:26 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:29:26.891 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:26 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:29:26.891 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:26 smithi115 ceph-mon[72476]: osdmap e839: 12 total, 11 up, 11 in 2024-03-20T09:29:26.891 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:26 smithi115 ceph-mon[72476]: pgmap v2024: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 3.0 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 319 op/s; 3081/148290 objects degraded (2.078%); 4764/148290 objects misplaced (3.213%); 902 KiB/s, 14 objects/s recovering 2024-03-20T09:29:26.893 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:26.911 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:29:26.911 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:29:26.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:26.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:29:26.948 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:29:26.971 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:29:27.002 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:29:27.002 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:29:27.002 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:29:27.002 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:29:27.002 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:29:27.002 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:29:27.002 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:29:27.002 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:29:27.003 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:29:27.003 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:29:27.003 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:29:27.003 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-03-20T09:29:27.003 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:29:27.015 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:29:27.041 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:27.060 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:29:27.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:27.063 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:29:27.164 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:29:27.191 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:27.206 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:29:27.206 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:29:27.209 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:29:27.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:27.212 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:29:27.313 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:29:27.341 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:27.360 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:29:27.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:27.363 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:29:27.464 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:29:27.491 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:27.512 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:29:27.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:27.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:29:27.565 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:29:27.565 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:29:27.619 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:29:27.656 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:27 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2303280204' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:29:27.660 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:27.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:27 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2303280204' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:29:27.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:27 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2303280204' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:29:28.238 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:29:28.238 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:29:28.464 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:28 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 3081/148290 objects degraded (2.078%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:29:28.464 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:28 smithi026 ceph-mon[68309]: osdmap e840: 12 total, 11 up, 11 in 2024-03-20T09:29:28.464 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:28 smithi026 ceph-mon[68309]: pgmap v2026: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 3.0 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 356 op/s; 2971/146202 objects degraded (2.032%); 4553/146202 objects misplaced (3.114%); 1.5 MiB/s, 24 objects/s recovering 2024-03-20T09:29:28.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:28 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 3081/148290 objects degraded (2.078%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:29:28.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:28 smithi115 ceph-mon[72476]: osdmap e840: 12 total, 11 up, 11 in 2024-03-20T09:29:28.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:28 smithi115 ceph-mon[72476]: pgmap v2026: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 3.0 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 356 op/s; 2971/146202 objects degraded (2.032%); 4553/146202 objects misplaced (3.114%); 1.5 MiB/s, 24 objects/s recovering 2024-03-20T09:29:28.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:28 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 3081/148290 objects degraded (2.078%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:29:28.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:28 smithi122 ceph-mon[73495]: osdmap e840: 12 total, 11 up, 11 in 2024-03-20T09:29:28.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:28 smithi122 ceph-mon[73495]: pgmap v2026: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 3.0 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 356 op/s; 2971/146202 objects degraded (2.032%); 4553/146202 objects misplaced (3.114%); 1.5 MiB/s, 24 objects/s recovering 2024-03-20T09:29:29.223 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-20T09:29:29.561 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:29:29.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:29.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:29:29.665 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:29:29.709 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:29.788 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:29:29.789 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:29 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:29:29.789 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:29 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:29:29.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:29.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:29:29.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:29 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:29:29.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:29 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:29:29.900 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:29:29.928 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:29.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:29 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:29:29.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:29 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:29:29.950 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:29:29.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:29.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:29:30.056 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:29:30.105 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:30.239 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:29:30.324 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:29:30.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:30.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:29:30.430 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:29:30.468 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:30.603 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:29:30.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:30.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:29:30.707 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:29:30.727 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:30 smithi026 ceph-mon[68309]: osdmap e841: 12 total, 11 up, 11 in 2024-03-20T09:29:30.727 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:30 smithi026 ceph-mon[68309]: pgmap v2028: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.9 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 455 op/s; 2971/143346 objects degraded (2.073%); 4553/143346 objects misplaced (3.176%); 800 KiB/s, 12 objects/s recovering 2024-03-20T09:29:30.759 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:30.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:30 smithi115 ceph-mon[72476]: osdmap e841: 12 total, 11 up, 11 in 2024-03-20T09:29:30.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:30 smithi115 ceph-mon[72476]: pgmap v2028: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.9 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 455 op/s; 2971/143346 objects degraded (2.073%); 4553/143346 objects misplaced (3.176%); 800 KiB/s, 12 objects/s recovering 2024-03-20T09:29:30.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:30 smithi122 ceph-mon[73495]: osdmap e841: 12 total, 11 up, 11 in 2024-03-20T09:29:30.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:30 smithi122 ceph-mon[73495]: pgmap v2028: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.9 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 455 op/s; 2971/143346 objects degraded (2.073%); 4553/143346 objects misplaced (3.176%); 800 KiB/s, 12 objects/s recovering 2024-03-20T09:29:31.114 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:29:31.119 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:31.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:29:31.221 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:29:31.250 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:31.274 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:29:31.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:31.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:29:31.382 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:29:31.415 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:31.438 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:29:31.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:31.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:29:31.547 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:29:31.573 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:31.594 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:29:31.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:31.597 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:29:31.698 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:29:31.728 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:31.748 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:29:31.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:31.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:29:31.853 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:29:31.900 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:31.948 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:29:31.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:31.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:29:32.052 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:29:32.079 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:32.100 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:29:32.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:32.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:29:32.127 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:29:32.204 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:29:32.243 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:32.277 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:29:32.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:32.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:29:32.354 INFO:teuthology.orchestra.run.smithi026.stderr:no valid command found; 10 closest matches: 2024-03-20T09:29:32.354 INFO:teuthology.orchestra.run.smithi026.stderr:0 2024-03-20T09:29:32.354 INFO:teuthology.orchestra.run.smithi026.stderr:1 2024-03-20T09:29:32.354 INFO:teuthology.orchestra.run.smithi026.stderr:2 2024-03-20T09:29:32.354 INFO:teuthology.orchestra.run.smithi026.stderr:abort 2024-03-20T09:29:32.355 INFO:teuthology.orchestra.run.smithi026.stderr:assert 2024-03-20T09:29:32.355 INFO:teuthology.orchestra.run.smithi026.stderr:bench [] [] [] [] 2024-03-20T09:29:32.355 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs debug_inject_read_zeros 2024-03-20T09:29:32.355 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs files list 2024-03-20T09:29:32.355 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs stats 2024-03-20T09:29:32.355 INFO:teuthology.orchestra.run.smithi026.stderr:bluestore allocator dump block 2024-03-20T09:29:32.355 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: invalid command 2024-03-20T09:29:32.381 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:29:32.416 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:33.538 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:29:33.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:33 smithi115 ceph-mon[72476]: pgmap v2029: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.9 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 430 op/s; 2971/140409 objects degraded (2.116%); 4553/140409 objects misplaced (3.243%); 699 KiB/s, 10 objects/s recovering 2024-03-20T09:29:33.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:33 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 2971/140409 objects degraded (2.116%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:29:33.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:33 smithi115 ceph-mon[72476]: osdmap e842: 12 total, 11 up, 11 in 2024-03-20T09:29:33.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:33 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1409471289' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:29:33.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:33 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:29:33.651 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:33 smithi026 ceph-mon[68309]: pgmap v2029: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.9 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 430 op/s; 2971/140409 objects degraded (2.116%); 4553/140409 objects misplaced (3.243%); 699 KiB/s, 10 objects/s recovering 2024-03-20T09:29:33.651 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:33 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 2971/140409 objects degraded (2.116%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:29:33.651 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:33 smithi026 ceph-mon[68309]: osdmap e842: 12 total, 11 up, 11 in 2024-03-20T09:29:33.651 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:33 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1409471289' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:29:33.651 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:33 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:29:33.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:33 smithi122 ceph-mon[73495]: pgmap v2029: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.9 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 430 op/s; 2971/140409 objects degraded (2.116%); 4553/140409 objects misplaced (3.243%); 699 KiB/s, 10 objects/s recovering 2024-03-20T09:29:33.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:33 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 2971/140409 objects degraded (2.116%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:29:33.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:33 smithi122 ceph-mon[73495]: osdmap e842: 12 total, 11 up, 11 in 2024-03-20T09:29:33.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:33 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1409471289' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:29:33.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:33 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:29:34.078 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:29:34.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:34.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:29:34.184 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:29:34.210 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:34.230 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:29:34.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:34.235 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:29:34.336 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:29:34.361 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:34 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:29:34.361 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:34 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:29:34.362 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:34 smithi026 ceph-mon[68309]: osdmap e843: 12 total, 11 up, 11 in 2024-03-20T09:29:34.365 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:34.388 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:29:34.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:34.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:29:34.496 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:29:34.520 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:34 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:29:34.520 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:34 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:29:34.520 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:34 smithi122 ceph-mon[73495]: osdmap e843: 12 total, 11 up, 11 in 2024-03-20T09:29:34.522 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:34.542 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:29:34.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:34.545 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:29:34.647 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:29:34.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:34 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:29:34.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:34 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:29:34.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:34 smithi115 ceph-mon[72476]: osdmap e843: 12 total, 11 up, 11 in 2024-03-20T09:29:34.676 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:34.705 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:29:34.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:34.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:29:34.811 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:29:34.882 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:35.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:35 smithi115 ceph-mon[72476]: pgmap v2032: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.8 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 376 op/s; 2832/139416 objects degraded (2.031%); 4343/139416 objects misplaced (3.115%); 800 KiB/s, 12 objects/s recovering 2024-03-20T09:29:35.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:29:35.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:35 smithi122 ceph-mon[73495]: pgmap v2032: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.8 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 376 op/s; 2832/139416 objects degraded (2.031%); 4343/139416 objects misplaced (3.115%); 800 KiB/s, 12 objects/s recovering 2024-03-20T09:29:35.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:35 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:29:35.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:35 smithi026 ceph-mon[68309]: pgmap v2032: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.8 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 376 op/s; 2832/139416 objects degraded (2.031%); 4343/139416 objects misplaced (3.115%); 800 KiB/s, 12 objects/s recovering 2024-03-20T09:29:35.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:35 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:29:36.045 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:29:36.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:36.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:29:36.152 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:29:36.178 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:36.198 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:29:36.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:36.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:29:36.302 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:29:36.329 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:36.348 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:29:36.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:36.351 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:29:36.452 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:29:36.477 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:36.496 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:29:36.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:36.499 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:29:36.600 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:29:36.625 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:36 smithi122 ceph-mon[73495]: 6.2 scrub starts 2024-03-20T09:29:36.625 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:36 smithi122 ceph-mon[73495]: 6.2 scrub ok 2024-03-20T09:29:36.625 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:36 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:29:36.625 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:36 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:29:36.625 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:36 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:29:36.626 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:36.645 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:29:36.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:36.648 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:29:36.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:36 smithi115 ceph-mon[72476]: 6.2 scrub starts 2024-03-20T09:29:36.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:36 smithi115 ceph-mon[72476]: 6.2 scrub ok 2024-03-20T09:29:36.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:36 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:29:36.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:36 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:29:36.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:36 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:29:36.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:36 smithi026 ceph-mon[68309]: 6.2 scrub starts 2024-03-20T09:29:36.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:36 smithi026 ceph-mon[68309]: 6.2 scrub ok 2024-03-20T09:29:36.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:36 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:29:36.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:36 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:29:36.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:36 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:29:36.749 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:29:36.776 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:36.796 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:29:36.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:36.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:29:36.903 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:29:36.930 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:36.952 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:29:36.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:36.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:29:37.056 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:29:37.087 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:37.111 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:29:37.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:37.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:29:37.216 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:29:37.244 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:37.265 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:29:37.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:37.270 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:29:37.371 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:29:37.406 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T09:29:37.407 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-20T09:29:37.407 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.8 'trim stale osdmaps' 2024-03-20T09:29:37.412 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:37.557 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:29:37.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:37.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:29:37.601 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:29:37.634 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:37 smithi026 ceph-mon[68309]: 1.13 scrub starts 2024-03-20T09:29:37.634 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:37 smithi026 ceph-mon[68309]: pgmap v2033: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.7 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 563 op/s; 2832/134760 objects degraded (2.102%); 4343/134760 objects misplaced (3.223%); 710 KiB/s, 11 objects/s recovering 2024-03-20T09:29:37.634 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:37 smithi026 ceph-mon[68309]: 1.13 scrub ok 2024-03-20T09:29:37.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:37 smithi115 ceph-mon[72476]: 1.13 scrub starts 2024-03-20T09:29:37.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:37 smithi115 ceph-mon[72476]: pgmap v2033: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.7 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 563 op/s; 2832/134760 objects degraded (2.102%); 4343/134760 objects misplaced (3.223%); 710 KiB/s, 11 objects/s recovering 2024-03-20T09:29:37.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:37 smithi115 ceph-mon[72476]: 1.13 scrub ok 2024-03-20T09:29:37.667 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:29:37.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:37 smithi122 ceph-mon[73495]: 1.13 scrub starts 2024-03-20T09:29:37.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:37 smithi122 ceph-mon[73495]: pgmap v2033: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.7 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 563 op/s; 2832/134760 objects degraded (2.102%); 4343/134760 objects misplaced (3.223%); 710 KiB/s, 11 objects/s recovering 2024-03-20T09:29:37.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:37 smithi122 ceph-mon[73495]: 1.13 scrub ok 2024-03-20T09:29:37.693 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:37.712 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:29:37.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:37.715 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:29:37.816 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:29:37.842 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:37.863 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:29:37.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:37.868 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:29:37.969 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:29:37.998 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:38.020 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:29:38.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:38.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:29:38.124 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:29:38.165 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:38.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:38 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 2832/134760 objects degraded (2.102%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:29:38.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:38 smithi115 ceph-mon[72476]: osdmap e844: 12 total, 11 up, 11 in 2024-03-20T09:29:38.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:38 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 2832/134760 objects degraded (2.102%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:29:38.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:38 smithi122 ceph-mon[73495]: osdmap e844: 12 total, 11 up, 11 in 2024-03-20T09:29:38.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:38 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 2832/134760 objects degraded (2.102%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:29:38.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:38 smithi026 ceph-mon[68309]: osdmap e844: 12 total, 11 up, 11 in 2024-03-20T09:29:38.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-20T09:29:38.783+0000 7f8b01557700 0 --2- 172.21.15.26:0/2687717819 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7f8afc1093f0 0x7f8afc074620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-03-20T09:29:38.818 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:29:38.825 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:29:38.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:38.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:29:38.847 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:29:38.877 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:29:38.921 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:29:38.929 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:29:38.941 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:29:38.955 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:38.961 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:29:38.973 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:29:38.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:38.976 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:29:38.995 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:29:39.032 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:29:39.064 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:29:39.077 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:29:39.104 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:39.106 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:29:39.130 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:29:39.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:39.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:29:39.136 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:29:39.154 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:29:39.154 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:29:39.154 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:29:39.154 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:29:39.154 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:29:39.154 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:29:39.154 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:29:39.155 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:29:39.155 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:29:39.155 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:29:39.155 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-03-20T09:29:39.155 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:29:39.235 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:29:39.269 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:39.358 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:29:39.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:39.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:29:39.373 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:39 smithi115 ceph-mon[72476]: pgmap v2035: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.7 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 462 op/s; 2678/132075 objects degraded (2.028%); 4120/132075 objects misplaced (3.119%); 1.6 MiB/s, 25 objects/s recovering 2024-03-20T09:29:39.373 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:39 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2687717819' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:29:39.464 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:39 smithi026 ceph-mon[68309]: pgmap v2035: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.7 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 462 op/s; 2678/132075 objects degraded (2.028%); 4120/132075 objects misplaced (3.119%); 1.6 MiB/s, 25 objects/s recovering 2024-03-20T09:29:39.464 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:39 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2687717819' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:29:39.464 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:29:39.491 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:39.509 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:29:39.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:39.515 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:29:39.616 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:29:39.647 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:39.668 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:29:39.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:39.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:29:39.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:39 smithi122 ceph-mon[73495]: pgmap v2035: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.7 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 462 op/s; 2678/132075 objects degraded (2.028%); 4120/132075 objects misplaced (3.119%); 1.6 MiB/s, 25 objects/s recovering 2024-03-20T09:29:39.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:39 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2687717819' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:29:39.772 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:29:39.804 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:39.828 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:29:39.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:39.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:29:39.934 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:29:39.960 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:39.979 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:29:39.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:39.982 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:29:40.051 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:29:40.052 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:29:40.083 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:29:40.114 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:40.136 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:29:40.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:40.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:29:40.243 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:29:40.270 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:40.288 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:29:40.288 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:29:40.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:40.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:29:40.393 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:29:40.419 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:40.437 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:29:40.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:40.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:29:40.541 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:29:40.568 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:40.580 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:29:40.580 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-03-20T09:29:40.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-03-20T09:29:40.581 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:29:40.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.581 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:132717 5.5 5:a85622dd:::benchmark_data_smithi122_157894_object2730:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e640)", 2024-03-20T09:29:40.581 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:19:57.993260+0000", 2024-03-20T09:29:40.581 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 582.58974440300005, 2024-03-20T09:29:40.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3269052480000001, 2024-03-20T09:29:40.582 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:29:40.582 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:29:40.582 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:29:40.582 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:29:40.583 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:29:40.583 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 132717 2024-03-20T09:29:40.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.583 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:29:40.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:29:40.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:57.993260+0000", 2024-03-20T09:29:40.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:29:40.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:57.993260+0000", 2024-03-20T09:29:40.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.585 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:29:40.585 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:57.993260+0000", 2024-03-20T09:29:40.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8500000000000002e-07 2024-03-20T09:29:40.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:29:40.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:57.993266+0000", 2024-03-20T09:29:40.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.93e-06 2024-03-20T09:29:40.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:29:40.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:57.993273+0000", 2024-03-20T09:29:40.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2579999999999998e-06 2024-03-20T09:29:40.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:29:40.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:57.993278+0000", 2024-03-20T09:29:40.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2499999999999997e-06 2024-03-20T09:29:40.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:57.999577+0000", 2024-03-20T09:29:40.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0062984790000000001 2024-03-20T09:29:40.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:29:40.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:57.999634+0000", 2024-03-20T09:29:40.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7414999999999998e-05 2024-03-20T09:29:40.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:00.320150+0000", 2024-03-20T09:29:40.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3205160149999999 2024-03-20T09:29:40.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:29:40.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:00.320165+0000", 2024-03-20T09:29:40.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4516e-05 2024-03-20T09:29:40.591 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.591 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:29:40.591 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.591 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:132729 5.5 5:a5fcbedb:::benchmark_data_smithi122_157894_object2742:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e640)", 2024-03-20T09:29:40.591 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:19:57.993290+0000", 2024-03-20T09:29:40.591 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 582.58971424200001, 2024-03-20T09:29:40.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3448420080000001, 2024-03-20T09:29:40.591 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:29:40.591 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:29:40.591 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:29:40.592 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:29:40.592 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:29:40.592 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 132729 2024-03-20T09:29:40.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.592 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:29:40.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:29:40.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:57.993290+0000", 2024-03-20T09:29:40.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:29:40.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:57.993290+0000", 2024-03-20T09:29:40.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:29:40.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:57.993290+0000", 2024-03-20T09:29:40.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4200000000000001e-07 2024-03-20T09:29:40.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:29:40.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:57.993292+0000", 2024-03-20T09:29:40.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.193e-06 2024-03-20T09:29:40.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:29:40.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:57.993300+0000", 2024-03-20T09:29:40.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.165e-06 2024-03-20T09:29:40.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:29:40.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:57.993305+0000", 2024-03-20T09:29:40.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5849999999999999e-06 2024-03-20T09:29:40.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:57.999662+0000", 2024-03-20T09:29:40.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0063571749999999996 2024-03-20T09:29:40.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:29:40.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:19:57.999726+0000", 2024-03-20T09:29:40.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3644000000000003e-05 2024-03-20T09:29:40.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:00.338117+0000", 2024-03-20T09:29:40.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3383908249999998 2024-03-20T09:29:40.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:29:40.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:00.338132+0000", 2024-03-20T09:29:40.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4979e-05 2024-03-20T09:29:40.595 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.595 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:29:40.596 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.596 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:134917 5.5 5:af4c6f6b:::benchmark_data_smithi122_157894_object4930:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e640)", 2024-03-20T09:29:40.596 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:20:01.472901+0000", 2024-03-20T09:29:40.596 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 579.11010282899997, 2024-03-20T09:29:40.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.608398652, 2024-03-20T09:29:40.596 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:29:40.596 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:29:40.596 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:29:40.596 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:29:40.596 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:29:40.596 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 134917 2024-03-20T09:29:40.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.597 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:29:40.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:29:40.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:01.472901+0000", 2024-03-20T09:29:40.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:29:40.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:01.472901+0000", 2024-03-20T09:29:40.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:29:40.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:01.472902+0000", 2024-03-20T09:29:40.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4400000000000001e-07 2024-03-20T09:29:40.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:29:40.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:01.472903+0000", 2024-03-20T09:29:40.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6589999999999999e-06 2024-03-20T09:29:40.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:29:40.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:01.472909+0000", 2024-03-20T09:29:40.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9159999999999998e-06 2024-03-20T09:29:40.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:29:40.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:01.472914+0000", 2024-03-20T09:29:40.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3680000000000004e-06 2024-03-20T09:29:40.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:01.472970+0000", 2024-03-20T09:29:40.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6706999999999998e-05 2024-03-20T09:29:40.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:29:40.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:01.473027+0000", 2024-03-20T09:29:40.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6439e-05 2024-03-20T09:29:40.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:04.081293+0000", 2024-03-20T09:29:40.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6082658990000001 2024-03-20T09:29:40.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:29:40.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:04.081300+0000", 2024-03-20T09:29:40.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3200000000000002e-06 2024-03-20T09:29:40.600 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.600 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:29:40.600 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.601 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:136502 5.5 5:aca98ce8:::benchmark_data_smithi122_157894_object6515:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e640)", 2024-03-20T09:29:40.601 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:20:04.032052+0000", 2024-03-20T09:29:40.601 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 576.55095199499999, 2024-03-20T09:29:40.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7508287569999998, 2024-03-20T09:29:40.601 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:29:40.601 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:29:40.601 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:29:40.601 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:29:40.601 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:29:40.601 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 136502 2024-03-20T09:29:40.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.602 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:29:40.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:29:40.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:04.032052+0000", 2024-03-20T09:29:40.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:29:40.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:04.032052+0000", 2024-03-20T09:29:40.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:29:40.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:04.032052+0000", 2024-03-20T09:29:40.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2899999999999999e-07 2024-03-20T09:29:40.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:29:40.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:04.032054+0000", 2024-03-20T09:29:40.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.866e-06 2024-03-20T09:29:40.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:29:40.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:04.032060+0000", 2024-03-20T09:29:40.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2340000000000002e-06 2024-03-20T09:29:40.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:29:40.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:04.032064+0000", 2024-03-20T09:29:40.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6310000000000002e-06 2024-03-20T09:29:40.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:04.032200+0000", 2024-03-20T09:29:40.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000135432 2024-03-20T09:29:40.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:29:40.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:04.032271+0000", 2024-03-20T09:29:40.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1408999999999993e-05 2024-03-20T09:29:40.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.782870+0000", 2024-03-20T09:29:40.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7505991829999998 2024-03-20T09:29:40.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:29:40.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.782881+0000", 2024-03-20T09:29:40.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0573e-05 2024-03-20T09:29:40.605 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.605 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:29:40.605 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.606 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:138116 5.5 5:a2509137:::benchmark_data_smithi122_157894_object8129:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e640)", 2024-03-20T09:29:40.606 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:20:06.707844+0000", 2024-03-20T09:29:40.606 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 573.87516052299998, 2024-03-20T09:29:40.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8833028590000005, 2024-03-20T09:29:40.606 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:29:40.606 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:29:40.606 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:29:40.606 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:29:40.606 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:29:40.606 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 138116 2024-03-20T09:29:40.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.606 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:29:40.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:29:40.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.707844+0000", 2024-03-20T09:29:40.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:29:40.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.707844+0000", 2024-03-20T09:29:40.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:29:40.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.707844+0000", 2024-03-20T09:29:40.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9400000000000001e-07 2024-03-20T09:29:40.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:29:40.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.707846+0000", 2024-03-20T09:29:40.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2199999999999999e-06 2024-03-20T09:29:40.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:29:40.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.707854+0000", 2024-03-20T09:29:40.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8450000000000003e-06 2024-03-20T09:29:40.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:29:40.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.707859+0000", 2024-03-20T09:29:40.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3669999999999996e-06 2024-03-20T09:29:40.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.708051+0000", 2024-03-20T09:29:40.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000192098 2024-03-20T09:29:40.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:29:40.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.708133+0000", 2024-03-20T09:29:40.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1102999999999999e-05 2024-03-20T09:29:40.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:12.591128+0000", 2024-03-20T09:29:40.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8829951319999996 2024-03-20T09:29:40.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:29:40.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:12.591146+0000", 2024-03-20T09:29:40.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8700000000000001e-05 2024-03-20T09:29:40.610 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.610 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:29:40.610 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.610 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:138118 5.5 5:a21f0c93:::benchmark_data_smithi122_157894_object8131:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e640)", 2024-03-20T09:29:40.610 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:20:06.707869+0000", 2024-03-20T09:29:40.611 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 573.87513533799995, 2024-03-20T09:29:40.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8948337950000003, 2024-03-20T09:29:40.611 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:29:40.611 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:29:40.611 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:29:40.611 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:29:40.611 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:29:40.611 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 138118 2024-03-20T09:29:40.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.611 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:29:40.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:29:40.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.707869+0000", 2024-03-20T09:29:40.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:29:40.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.707869+0000", 2024-03-20T09:29:40.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:29:40.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.707869+0000", 2024-03-20T09:29:40.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5000000000000003e-07 2024-03-20T09:29:40.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:29:40.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.707871+0000", 2024-03-20T09:29:40.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1900000000000002e-06 2024-03-20T09:29:40.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:29:40.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.707886+0000", 2024-03-20T09:29:40.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4849e-05 2024-03-20T09:29:40.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:29:40.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.707894+0000", 2024-03-20T09:29:40.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8509999999999999e-06 2024-03-20T09:29:40.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.708173+0000", 2024-03-20T09:29:40.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000278566 2024-03-20T09:29:40.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:29:40.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.708224+0000", 2024-03-20T09:29:40.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1570999999999998e-05 2024-03-20T09:29:40.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:12.602687+0000", 2024-03-20T09:29:40.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8944628400000001 2024-03-20T09:29:40.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:29:40.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:12.602703+0000", 2024-03-20T09:29:40.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5378000000000001e-05 2024-03-20T09:29:40.615 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.615 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:29:40.615 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.616 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:138125 5.5 5:acdf3955:::benchmark_data_smithi122_157894_object8138:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e640)", 2024-03-20T09:29:40.616 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:20:06.707929+0000", 2024-03-20T09:29:40.616 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 573.87507489899997, 2024-03-20T09:29:40.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9079474379999999, 2024-03-20T09:29:40.616 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:29:40.616 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:29:40.616 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:29:40.616 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:29:40.616 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:29:40.616 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 138125 2024-03-20T09:29:40.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.616 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:29:40.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:29:40.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.707929+0000", 2024-03-20T09:29:40.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:29:40.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.707929+0000", 2024-03-20T09:29:40.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:29:40.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.707930+0000", 2024-03-20T09:29:40.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5400000000000002e-07 2024-03-20T09:29:40.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:29:40.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.707932+0000", 2024-03-20T09:29:40.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1859999999999999e-06 2024-03-20T09:29:40.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:29:40.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.707939+0000", 2024-03-20T09:29:40.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1119999999999998e-06 2024-03-20T09:29:40.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:29:40.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.707944+0000", 2024-03-20T09:29:40.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8609999999999997e-06 2024-03-20T09:29:40.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.708250+0000", 2024-03-20T09:29:40.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00030575399999999999 2024-03-20T09:29:40.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:29:40.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:06.708342+0000", 2024-03-20T09:29:40.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2390000000000001e-05 2024-03-20T09:29:40.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:12.615860+0000", 2024-03-20T09:29:40.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9075176769999995 2024-03-20T09:29:40.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:29:40.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:20:12.615877+0000", 2024-03-20T09:29:40.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7003999999999998e-05 2024-03-20T09:29:40.620 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.620 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:29:40.620 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.620 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:177371 5.5 5:a1b6b191:::benchmark_data_smithi122_157894_object47384:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e663)", 2024-03-20T09:29:40.620 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:21:28.979500+0000", 2024-03-20T09:29:40.620 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 491.60350358400001, 2024-03-20T09:29:40.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8943260209999999, 2024-03-20T09:29:40.620 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:29:40.621 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:29:40.621 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:29:40.621 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:29:40.621 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:29:40.621 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 177371 2024-03-20T09:29:40.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.621 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:29:40.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:29:40.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:28.979500+0000", 2024-03-20T09:29:40.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:29:40.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:28.979500+0000", 2024-03-20T09:29:40.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:29:40.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:28.979501+0000", 2024-03-20T09:29:40.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2000000000000001e-07 2024-03-20T09:29:40.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:29:40.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:28.979503+0000", 2024-03-20T09:29:40.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8950000000000001e-06 2024-03-20T09:29:40.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:29:40.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:28.979508+0000", 2024-03-20T09:29:40.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5729999999999998e-06 2024-03-20T09:29:40.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:29:40.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:28.979513+0000", 2024-03-20T09:29:40.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9509999999999997e-06 2024-03-20T09:29:40.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:28.995185+0000", 2024-03-20T09:29:40.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015672063999999999 2024-03-20T09:29:40.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:29:40.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:28.995248+0000", 2024-03-20T09:29:40.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3153999999999999e-05 2024-03-20T09:29:40.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:30.873811+0000", 2024-03-20T09:29:40.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8785628060000001 2024-03-20T09:29:40.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:29:40.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:30.873827+0000", 2024-03-20T09:29:40.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5258e-05 2024-03-20T09:29:40.624 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.624 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:29:40.624 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.625 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:191814 5.5 5:a9b30849:::benchmark_data_smithi122_157894_object61827:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e668)", 2024-03-20T09:29:40.625 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:21:54.975493+0000", 2024-03-20T09:29:40.625 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 465.607510596, 2024-03-20T09:29:40.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3338137570000002, 2024-03-20T09:29:40.625 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:29:40.625 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:29:40.625 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:29:40.625 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:29:40.625 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:29:40.625 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 191814 2024-03-20T09:29:40.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.625 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:29:40.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:29:40.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:54.975493+0000", 2024-03-20T09:29:40.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:29:40.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:54.975493+0000", 2024-03-20T09:29:40.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:29:40.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:54.975494+0000", 2024-03-20T09:29:40.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2899999999999999e-07 2024-03-20T09:29:40.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:29:40.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:54.975496+0000", 2024-03-20T09:29:40.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.12e-06 2024-03-20T09:29:40.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:29:40.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:54.975501+0000", 2024-03-20T09:29:40.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3859999999999998e-06 2024-03-20T09:29:40.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:29:40.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:54.975506+0000", 2024-03-20T09:29:40.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5290000000000002e-06 2024-03-20T09:29:40.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:54.979081+0000", 2024-03-20T09:29:40.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035750059999999999 2024-03-20T09:29:40.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:29:40.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:54.979159+0000", 2024-03-20T09:29:40.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8078999999999996e-05 2024-03-20T09:29:40.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.309291+0000", 2024-03-20T09:29:40.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.330132366 2024-03-20T09:29:40.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:29:40.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.309307+0000", 2024-03-20T09:29:40.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5942000000000001e-05 2024-03-20T09:29:40.629 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.629 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:29:40.629 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.629 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:192991 5.5 5:a2123005:::benchmark_data_smithi122_157894_object63004:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e668)", 2024-03-20T09:29:40.629 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:21:57.197478+0000", 2024-03-20T09:29:40.630 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 463.38552627799999, 2024-03-20T09:29:40.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1310714370000001, 2024-03-20T09:29:40.630 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:29:40.630 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:29:40.630 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:29:40.630 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:29:40.630 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:29:40.630 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 192991 2024-03-20T09:29:40.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.630 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:29:40.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:29:40.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197478+0000", 2024-03-20T09:29:40.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:29:40.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197478+0000", 2024-03-20T09:29:40.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:29:40.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197478+0000", 2024-03-20T09:29:40.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7300000000000002e-07 2024-03-20T09:29:40.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:29:40.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197480+0000", 2024-03-20T09:29:40.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.277e-06 2024-03-20T09:29:40.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:29:40.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197487+0000", 2024-03-20T09:29:40.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9160000000000004e-06 2024-03-20T09:29:40.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:29:40.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197492+0000", 2024-03-20T09:29:40.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6739999999999996e-06 2024-03-20T09:29:40.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.204290+0000", 2024-03-20T09:29:40.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0067975529999999996 2024-03-20T09:29:40.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:29:40.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.204348+0000", 2024-03-20T09:29:40.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8020999999999999e-05 2024-03-20T09:29:40.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:01.328540+0000", 2024-03-20T09:29:40.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1241922119999996 2024-03-20T09:29:40.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:29:40.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:01.328549+0000", 2024-03-20T09:29:40.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4110000000000002e-06 2024-03-20T09:29:40.634 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.634 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:29:40.634 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.634 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:192996 5.5 5:a7785dc6:::benchmark_data_smithi122_157894_object63009:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e668)", 2024-03-20T09:29:40.634 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:21:57.197499+0000", 2024-03-20T09:29:40.634 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 463.38550547900002, 2024-03-20T09:29:40.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1424198329999999, 2024-03-20T09:29:40.634 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:29:40.634 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:29:40.634 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:29:40.635 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:29:40.635 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:29:40.635 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 192996 2024-03-20T09:29:40.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.635 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:29:40.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:29:40.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197499+0000", 2024-03-20T09:29:40.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:29:40.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197499+0000", 2024-03-20T09:29:40.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:29:40.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197499+0000", 2024-03-20T09:29:40.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3000000000000001e-07 2024-03-20T09:29:40.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:29:40.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197501+0000", 2024-03-20T09:29:40.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2910000000000002e-06 2024-03-20T09:29:40.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:29:40.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197507+0000", 2024-03-20T09:29:40.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8300000000000001e-06 2024-03-20T09:29:40.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:29:40.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197512+0000", 2024-03-20T09:29:40.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8250000000000004e-06 2024-03-20T09:29:40.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.204365+0000", 2024-03-20T09:29:40.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0068533769999999999 2024-03-20T09:29:40.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:29:40.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.204408+0000", 2024-03-20T09:29:40.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3028999999999997e-05 2024-03-20T09:29:40.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:01.339901+0000", 2024-03-20T09:29:40.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1354924190000002 2024-03-20T09:29:40.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:29:40.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:01.339918+0000", 2024-03-20T09:29:40.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7632e-05 2024-03-20T09:29:40.638 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.638 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:29:40.639 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.639 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:193014 5.5 5:ac49b0df:::benchmark_data_smithi122_157894_object63027:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e668)", 2024-03-20T09:29:40.639 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:21:57.197518+0000", 2024-03-20T09:29:40.639 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 463.38548637299999, 2024-03-20T09:29:40.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1533976619999997, 2024-03-20T09:29:40.639 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:29:40.639 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:29:40.639 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:29:40.639 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:29:40.639 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:29:40.639 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 193014 2024-03-20T09:29:40.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.640 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:29:40.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:29:40.640 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197518+0000", 2024-03-20T09:29:40.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:29:40.640 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197518+0000", 2024-03-20T09:29:40.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:29:40.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197518+0000", 2024-03-20T09:29:40.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6399999999999998e-07 2024-03-20T09:29:40.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:29:40.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197520+0000", 2024-03-20T09:29:40.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0839999999999999e-06 2024-03-20T09:29:40.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:29:40.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197526+0000", 2024-03-20T09:29:40.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4129999999999997e-06 2024-03-20T09:29:40.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:29:40.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197530+0000", 2024-03-20T09:29:40.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5600000000000004e-06 2024-03-20T09:29:40.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.204429+0000", 2024-03-20T09:29:40.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006899105 2024-03-20T09:29:40.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:29:40.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.204486+0000", 2024-03-20T09:29:40.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7281000000000003e-05 2024-03-20T09:29:40.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:01.350900+0000", 2024-03-20T09:29:40.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1464130609999996 2024-03-20T09:29:40.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:29:40.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:01.350915+0000", 2024-03-20T09:29:40.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5794000000000002e-05 2024-03-20T09:29:40.643 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.643 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:29:40.644 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.644 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:193025 5.5 5:a75663e7:::benchmark_data_smithi122_157894_object63038:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e668)", 2024-03-20T09:29:40.644 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:21:57.197554+0000", 2024-03-20T09:29:40.644 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 463.38545045900003, 2024-03-20T09:29:40.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1646879749999997, 2024-03-20T09:29:40.644 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:29:40.644 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:29:40.644 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:29:40.644 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:29:40.644 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:29:40.644 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 193025 2024-03-20T09:29:40.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.645 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:29:40.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:29:40.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197554+0000", 2024-03-20T09:29:40.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:29:40.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197554+0000", 2024-03-20T09:29:40.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:29:40.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197554+0000", 2024-03-20T09:29:40.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1199999999999999e-07 2024-03-20T09:29:40.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:29:40.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197557+0000", 2024-03-20T09:29:40.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.898e-06 2024-03-20T09:29:40.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:29:40.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197562+0000", 2024-03-20T09:29:40.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0440000000000003e-06 2024-03-20T09:29:40.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:29:40.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197566+0000", 2024-03-20T09:29:40.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5369999999999999e-06 2024-03-20T09:29:40.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.204505+0000", 2024-03-20T09:29:40.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0069383500000000002 2024-03-20T09:29:40.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:29:40.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.204546+0000", 2024-03-20T09:29:40.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699000000000002e-05 2024-03-20T09:29:40.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:01.362230+0000", 2024-03-20T09:29:40.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1576838379999996 2024-03-20T09:29:40.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:29:40.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:01.362242+0000", 2024-03-20T09:29:40.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1297e-05 2024-03-20T09:29:40.648 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.648 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:29:40.648 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.649 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:193028 5.5 5:a0a14a7f:::benchmark_data_smithi122_157894_object63041:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e668)", 2024-03-20T09:29:40.649 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:21:57.197572+0000", 2024-03-20T09:29:40.649 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 463.38543193599997, 2024-03-20T09:29:40.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1764598690000003, 2024-03-20T09:29:40.649 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:29:40.649 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:29:40.649 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:29:40.649 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:29:40.649 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:29:40.649 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 193028 2024-03-20T09:29:40.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.649 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:29:40.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:29:40.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197572+0000", 2024-03-20T09:29:40.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:29:40.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197572+0000", 2024-03-20T09:29:40.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:29:40.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197573+0000", 2024-03-20T09:29:40.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9699999999999996e-07 2024-03-20T09:29:40.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:29:40.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197575+0000", 2024-03-20T09:29:40.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9089999999999998e-06 2024-03-20T09:29:40.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:29:40.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197581+0000", 2024-03-20T09:29:40.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2110000000000003e-06 2024-03-20T09:29:40.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.651 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:29:40.651 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.197586+0000", 2024-03-20T09:29:40.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1120000000000004e-06 2024-03-20T09:29:40.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.204566+0000", 2024-03-20T09:29:40.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0069799229999999999 2024-03-20T09:29:40.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:29:40.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:21:57.204609+0000", 2024-03-20T09:29:40.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2979000000000003e-05 2024-03-20T09:29:40.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:01.374020+0000", 2024-03-20T09:29:40.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1694113149999996 2024-03-20T09:29:40.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:29:40.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:01.374032+0000", 2024-03-20T09:29:40.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1922999999999999e-05 2024-03-20T09:29:40.653 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.653 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:29:40.653 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.653 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:195329 5.5 5:afea5365:::benchmark_data_smithi122_157894_object65342:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e669)", 2024-03-20T09:29:40.653 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:22:01.285838+0000", 2024-03-20T09:29:40.653 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 459.29716622900003, 2024-03-20T09:29:40.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0307289530000001, 2024-03-20T09:29:40.653 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:29:40.653 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:29:40.654 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:29:40.654 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:29:40.654 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:29:40.654 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 195329 2024-03-20T09:29:40.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.654 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:29:40.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:29:40.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:01.285838+0000", 2024-03-20T09:29:40.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:29:40.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:01.285838+0000", 2024-03-20T09:29:40.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:29:40.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:01.285838+0000", 2024-03-20T09:29:40.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8999999999999997e-07 2024-03-20T09:29:40.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:29:40.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:01.285841+0000", 2024-03-20T09:29:40.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5519999999999999e-06 2024-03-20T09:29:40.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:29:40.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:01.285849+0000", 2024-03-20T09:29:40.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0600000000000008e-06 2024-03-20T09:29:40.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:29:40.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:01.285855+0000", 2024-03-20T09:29:40.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0209999999999996e-06 2024-03-20T09:29:40.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:01.289987+0000", 2024-03-20T09:29:40.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0041315759999999997 2024-03-20T09:29:40.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:29:40.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:01.290045+0000", 2024-03-20T09:29:40.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8044000000000002e-05 2024-03-20T09:29:40.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:03.316558+0000", 2024-03-20T09:29:40.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0265130240000002 2024-03-20T09:29:40.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.658 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:29:40.658 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:03.316567+0000", 2024-03-20T09:29:40.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1859999999999998e-06 2024-03-20T09:29:40.658 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.658 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:29:40.658 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.658 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:197729 5.5 5:aa48ba13:::benchmark_data_smithi122_157894_object67742:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e672)", 2024-03-20T09:29:40.658 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:22:05.320420+0000", 2024-03-20T09:29:40.658 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 455.262583861, 2024-03-20T09:29:40.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8930668800000001, 2024-03-20T09:29:40.658 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:29:40.659 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:29:40.659 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:29:40.659 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:29:40.659 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:29:40.659 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 197729 2024-03-20T09:29:40.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.659 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:29:40.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:29:40.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:05.320420+0000", 2024-03-20T09:29:40.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:29:40.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:05.320420+0000", 2024-03-20T09:29:40.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:29:40.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:05.320421+0000", 2024-03-20T09:29:40.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4600000000000004e-07 2024-03-20T09:29:40.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:29:40.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:05.320425+0000", 2024-03-20T09:29:40.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9879999999999998e-06 2024-03-20T09:29:40.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:29:40.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:05.320433+0000", 2024-03-20T09:29:40.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7060000000000004e-06 2024-03-20T09:29:40.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:29:40.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:05.320439+0000", 2024-03-20T09:29:40.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8369999999999999e-06 2024-03-20T09:29:40.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:05.323263+0000", 2024-03-20T09:29:40.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0028235460000000001 2024-03-20T09:29:40.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:29:40.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:05.323349+0000", 2024-03-20T09:29:40.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6416999999999996e-05 2024-03-20T09:29:40.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.213468+0000", 2024-03-20T09:29:40.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8901190510000001 2024-03-20T09:29:40.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:29:40.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.213487+0000", 2024-03-20T09:29:40.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8589000000000001e-05 2024-03-20T09:29:40.663 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.663 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:29:40.663 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.663 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:197732 5.5 5:a103aab1:::benchmark_data_smithi122_157894_object67745:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e672)", 2024-03-20T09:29:40.663 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:22:05.320454+0000", 2024-03-20T09:29:40.663 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 455.26255027000002, 2024-03-20T09:29:40.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9028468630000002, 2024-03-20T09:29:40.664 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:29:40.664 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:29:40.664 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:29:40.664 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:29:40.664 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:29:40.664 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 197732 2024-03-20T09:29:40.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.664 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:29:40.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:29:40.664 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:05.320454+0000", 2024-03-20T09:29:40.664 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.664 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.664 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.664 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:29:40.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:05.320454+0000", 2024-03-20T09:29:40.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:29:40.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:05.320454+0000", 2024-03-20T09:29:40.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2899999999999999e-07 2024-03-20T09:29:40.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:29:40.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:05.320457+0000", 2024-03-20T09:29:40.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3209999999999999e-06 2024-03-20T09:29:40.665 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.665 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:29:40.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:05.320462+0000", 2024-03-20T09:29:40.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7320000000000003e-06 2024-03-20T09:29:40.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:29:40.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:05.320467+0000", 2024-03-20T09:29:40.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6670000000000004e-06 2024-03-20T09:29:40.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:05.323380+0000", 2024-03-20T09:29:40.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0029125919999999999 2024-03-20T09:29:40.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.667 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:29:40.667 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:05.323448+0000", 2024-03-20T09:29:40.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8388999999999999e-05 2024-03-20T09:29:40.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.223285+0000", 2024-03-20T09:29:40.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8998367869999999 2024-03-20T09:29:40.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:29:40.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.223301+0000", 2024-03-20T09:29:40.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5945999999999999e-05 2024-03-20T09:29:40.668 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.668 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:29:40.668 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.668 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:199405 5.5 5:a1324d3e:::benchmark_data_smithi122_157894_object69418:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e673)", 2024-03-20T09:29:40.669 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:22:08.168607+0000", 2024-03-20T09:29:40.669 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 452.41439660700001, 2024-03-20T09:29:40.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.352534253, 2024-03-20T09:29:40.669 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:29:40.669 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:29:40.669 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:29:40.669 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:29:40.669 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:29:40.669 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 199405 2024-03-20T09:29:40.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.669 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:29:40.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:29:40.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.168607+0000", 2024-03-20T09:29:40.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:29:40.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.168607+0000", 2024-03-20T09:29:40.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:29:40.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.168608+0000", 2024-03-20T09:29:40.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1299999999999999e-07 2024-03-20T09:29:40.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:29:40.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.168609+0000", 2024-03-20T09:29:40.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.483e-06 2024-03-20T09:29:40.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:29:40.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.168612+0000", 2024-03-20T09:29:40.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2940000000000001e-06 2024-03-20T09:29:40.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:29:40.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.168616+0000", 2024-03-20T09:29:40.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.269e-06 2024-03-20T09:29:40.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.178863+0000", 2024-03-20T09:29:40.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010247439000000001 2024-03-20T09:29:40.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:29:40.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.178937+0000", 2024-03-20T09:29:40.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3949999999999995e-05 2024-03-20T09:29:40.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:10.521129+0000", 2024-03-20T09:29:40.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3421915129999999 2024-03-20T09:29:40.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:29:40.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:10.521142+0000", 2024-03-20T09:29:40.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3091999999999999e-05 2024-03-20T09:29:40.673 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.673 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:29:40.673 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.673 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:199411 5.5 5:a791d230:::benchmark_data_smithi122_157894_object69424:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e673)", 2024-03-20T09:29:40.673 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:22:08.168619+0000", 2024-03-20T09:29:40.673 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 452.41438513200001, 2024-03-20T09:29:40.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3647598240000001, 2024-03-20T09:29:40.673 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:29:40.673 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:29:40.674 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:29:40.674 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:29:40.674 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:29:40.674 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 199411 2024-03-20T09:29:40.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.674 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:29:40.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:29:40.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.168619+0000", 2024-03-20T09:29:40.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:29:40.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.168619+0000", 2024-03-20T09:29:40.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:29:40.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.168619+0000", 2024-03-20T09:29:40.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.36e-07 2024-03-20T09:29:40.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:29:40.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.168623+0000", 2024-03-20T09:29:40.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9740000000000004e-06 2024-03-20T09:29:40.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:29:40.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.168627+0000", 2024-03-20T09:29:40.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4089999999999999e-06 2024-03-20T09:29:40.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:29:40.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.168629+0000", 2024-03-20T09:29:40.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7499999999999999e-06 2024-03-20T09:29:40.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.178966+0000", 2024-03-20T09:29:40.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010337162 2024-03-20T09:29:40.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:29:40.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:08.179037+0000", 2024-03-20T09:29:40.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0805999999999999e-05 2024-03-20T09:29:40.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:10.533363+0000", 2024-03-20T09:29:40.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3543253019999999 2024-03-20T09:29:40.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:29:40.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:10.533379+0000", 2024-03-20T09:29:40.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6184999999999998e-05 2024-03-20T09:29:40.678 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.678 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:29:40.678 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.678 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.25966.0:222768 5.5 5:acab01bd:::benchmark_data_smithi122_157894_object92781:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e682)", 2024-03-20T09:29:40.678 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:22:45.242872+0000", 2024-03-20T09:29:40.678 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 415.34013158200003, 2024-03-20T09:29:40.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0710168700000002, 2024-03-20T09:29:40.678 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:29:40.678 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:29:40.678 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:29:40.678 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.25966", 2024-03-20T09:29:40.678 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/681057980", 2024-03-20T09:29:40.679 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 222768 2024-03-20T09:29:40.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.679 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:29:40.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:29:40.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:45.242872+0000", 2024-03-20T09:29:40.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:29:40.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:45.242872+0000", 2024-03-20T09:29:40.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:29:40.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:29:40.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:45.242873+0000", 2024-03-20T09:29:40.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4000000000000002e-07 2024-03-20T09:29:40.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:29:40.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:45.242876+0000", 2024-03-20T09:29:40.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8650000000000001e-06 2024-03-20T09:29:40.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:29:40.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:45.242883+0000", 2024-03-20T09:29:40.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0099999999999998e-06 2024-03-20T09:29:40.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:29:40.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:45.242889+0000", 2024-03-20T09:29:40.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9429999999999997e-06 2024-03-20T09:29:40.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:45.249232+0000", 2024-03-20T09:29:40.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.006343347 2024-03-20T09:29:40.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.681 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:29:40.681 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:45.249329+0000", 2024-03-20T09:29:40.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7199000000000002e-05 2024-03-20T09:29:40.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:29:40.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:49.313872+0000", 2024-03-20T09:29:40.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0645422570000003 2024-03-20T09:29:40.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:29:40.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:29:40.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:29:40.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:22:49.313889+0000", 2024-03-20T09:29:40.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7808999999999998e-05 2024-03-20T09:29:40.682 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.682 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:29:40.682 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.682 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:29:40.683 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:29:40.683 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:29:40.683 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:40 smithi026 ceph-mon[68309]: 1.3 scrub starts 2024-03-20T09:29:40.684 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:40 smithi026 ceph-mon[68309]: 1.3 scrub ok 2024-03-20T09:29:40.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:40.684 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:29:40.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:29:40.685 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:40 smithi115 ceph-mon[72476]: 1.3 scrub starts 2024-03-20T09:29:40.685 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:40 smithi115 ceph-mon[72476]: 1.3 scrub ok 2024-03-20T09:29:40.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:40 smithi122 ceph-mon[73495]: 1.3 scrub starts 2024-03-20T09:29:40.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:40 smithi122 ceph-mon[73495]: 1.3 scrub ok 2024-03-20T09:29:40.785 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:29:40.823 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:41.479 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:29:41.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:41.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:29:41.584 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:29:41.625 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:41.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:41 smithi115 ceph-mon[72476]: pgmap v2036: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.6 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 491 op/s; 2678/128949 objects degraded (2.077%); 4120/128949 objects misplaced (3.195%); 1.2 MiB/s, 19 objects/s recovering 2024-03-20T09:29:41.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:41 smithi122 ceph-mon[73495]: pgmap v2036: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.6 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 491 op/s; 2678/128949 objects degraded (2.077%); 4120/128949 objects misplaced (3.195%); 1.2 MiB/s, 19 objects/s recovering 2024-03-20T09:29:41.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:41 smithi026 ceph-mon[68309]: pgmap v2036: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.6 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 491 op/s; 2678/128949 objects degraded (2.077%); 4120/128949 objects misplaced (3.195%); 1.2 MiB/s, 19 objects/s recovering 2024-03-20T09:29:41.768 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:29:41.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:41.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:29:41.875 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:29:41.903 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:41.927 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:29:41.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:41.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:29:42.036 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:29:42.052 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:29:42.089 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:42.224 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:29:42.236 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:29:42.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:42.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:29:42.274 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-20T09:29:42.347 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:29:42.402 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:42.518 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:29:42.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:42.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:29:42.625 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:29:42.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:42 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:29:42.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:42 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:29:42.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:42 smithi115 ceph-mon[72476]: 1.17 scrub starts 2024-03-20T09:29:42.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:42 smithi115 ceph-mon[72476]: 1.17 scrub ok 2024-03-20T09:29:42.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:42 smithi115 ceph-mon[72476]: osdmap e845: 12 total, 11 up, 11 in 2024-03-20T09:29:42.653 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:42.675 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:29:42.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:42.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:29:42.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:42 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:29:42.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:42 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:29:42.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:42 smithi122 ceph-mon[73495]: 1.17 scrub starts 2024-03-20T09:29:42.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:42 smithi122 ceph-mon[73495]: 1.17 scrub ok 2024-03-20T09:29:42.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:42 smithi122 ceph-mon[73495]: osdmap e845: 12 total, 11 up, 11 in 2024-03-20T09:29:42.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:42 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:29:42.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:42 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:29:42.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:42 smithi026 ceph-mon[68309]: 1.17 scrub starts 2024-03-20T09:29:42.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:42 smithi026 ceph-mon[68309]: 1.17 scrub ok 2024-03-20T09:29:42.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:42 smithi026 ceph-mon[68309]: osdmap e845: 12 total, 11 up, 11 in 2024-03-20T09:29:42.780 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:29:42.806 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:42.824 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:29:42.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:42.827 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:29:42.928 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:29:42.987 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:43.395 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:43 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:29:43.395 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:43 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:29:43.395 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:43 smithi026 ceph-mon[68309]: 6.3 scrub starts 2024-03-20T09:29:43.395 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:43 smithi026 ceph-mon[68309]: 6.3 scrub ok 2024-03-20T09:29:43.395 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:43 smithi026 ceph-mon[68309]: pgmap v2038: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.6 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 538 op/s; 2678/126477 objects degraded (2.117%); 4120/126477 objects misplaced (3.258%); 640 KiB/s, 9 objects/s recovering 2024-03-20T09:29:43.395 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:43 smithi026 ceph-mon[68309]: osdmap e846: 12 total, 11 up, 11 in 2024-03-20T09:29:43.395 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:43 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 2678/126477 objects degraded (2.117%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:29:43.520 INFO:teuthology.orchestra.run.smithi026.stderr:no valid command found; 10 closest matches: 2024-03-20T09:29:43.520 INFO:teuthology.orchestra.run.smithi026.stderr:0 2024-03-20T09:29:43.520 INFO:teuthology.orchestra.run.smithi026.stderr:1 2024-03-20T09:29:43.520 INFO:teuthology.orchestra.run.smithi026.stderr:2 2024-03-20T09:29:43.520 INFO:teuthology.orchestra.run.smithi026.stderr:abort 2024-03-20T09:29:43.520 INFO:teuthology.orchestra.run.smithi026.stderr:assert 2024-03-20T09:29:43.520 INFO:teuthology.orchestra.run.smithi026.stderr:bench [] [] [] [] 2024-03-20T09:29:43.520 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs debug_inject_read_zeros 2024-03-20T09:29:43.520 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs files list 2024-03-20T09:29:43.520 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs stats 2024-03-20T09:29:43.520 INFO:teuthology.orchestra.run.smithi026.stderr:bluestore allocator dump block 2024-03-20T09:29:43.521 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: invalid command 2024-03-20T09:29:43.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:43 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:29:43.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:43 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:29:43.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:43 smithi115 ceph-mon[72476]: 6.3 scrub starts 2024-03-20T09:29:43.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:43 smithi115 ceph-mon[72476]: 6.3 scrub ok 2024-03-20T09:29:43.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:43 smithi115 ceph-mon[72476]: pgmap v2038: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.6 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 538 op/s; 2678/126477 objects degraded (2.117%); 4120/126477 objects misplaced (3.258%); 640 KiB/s, 9 objects/s recovering 2024-03-20T09:29:43.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:43 smithi115 ceph-mon[72476]: osdmap e846: 12 total, 11 up, 11 in 2024-03-20T09:29:43.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:43 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 2678/126477 objects degraded (2.117%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:29:43.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:43 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:29:43.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:43 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:29:43.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:43 smithi122 ceph-mon[73495]: 6.3 scrub starts 2024-03-20T09:29:43.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:43 smithi122 ceph-mon[73495]: 6.3 scrub ok 2024-03-20T09:29:43.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:43 smithi122 ceph-mon[73495]: pgmap v2038: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.6 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 538 op/s; 2678/126477 objects degraded (2.117%); 4120/126477 objects misplaced (3.258%); 640 KiB/s, 9 objects/s recovering 2024-03-20T09:29:43.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:43 smithi122 ceph-mon[73495]: osdmap e846: 12 total, 11 up, 11 in 2024-03-20T09:29:43.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:43 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 2678/126477 objects degraded (2.117%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:29:43.758 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:29:43.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:43.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:29:43.862 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:29:43.904 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:44.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:44 smithi122 ceph-mon[73495]: 6.11 scrub starts 2024-03-20T09:29:44.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:44 smithi122 ceph-mon[73495]: 6.11 scrub ok 2024-03-20T09:29:44.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:44 smithi026 ceph-mon[68309]: 6.11 scrub starts 2024-03-20T09:29:44.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:44 smithi026 ceph-mon[68309]: 6.11 scrub ok 2024-03-20T09:29:44.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:44 smithi115 ceph-mon[72476]: 6.11 scrub starts 2024-03-20T09:29:44.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:44 smithi115 ceph-mon[72476]: 6.11 scrub ok 2024-03-20T09:29:45.440 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:29:45.466 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:29:45.471 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:45.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:29:45.572 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:29:45.600 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:45.636 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:29:45.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:45.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:29:45.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:45 smithi122 ceph-mon[73495]: pgmap v2040: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.6 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 383 op/s; 2536/125418 objects degraded (2.022%); 3900/125418 objects misplaced (3.110%); 1.5 MiB/s, 23 objects/s recovering 2024-03-20T09:29:45.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:45 smithi026 ceph-mon[68309]: pgmap v2040: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.6 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 383 op/s; 2536/125418 objects degraded (2.022%); 3900/125418 objects misplaced (3.110%); 1.5 MiB/s, 23 objects/s recovering 2024-03-20T09:29:45.744 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:29:45.770 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:45 smithi115 ceph-mon[72476]: pgmap v2040: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.6 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 383 op/s; 2536/125418 objects degraded (2.022%); 3900/125418 objects misplaced (3.110%); 1.5 MiB/s, 23 objects/s recovering 2024-03-20T09:29:45.772 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:45.794 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:29:45.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:45.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:29:45.901 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:29:45.942 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:46.107 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:29:46.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:46.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:29:46.144 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T09:29:46.144 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-20T09:29:46.145 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.11 'trim stale osdmaps' 2024-03-20T09:29:46.220 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:29:46.248 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:46.269 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:29:46.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:46.275 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:29:46.376 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:29:46.452 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:46.627 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:29:46.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:46.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:29:46.633 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:46 smithi026 ceph-mon[68309]: pgmap v2041: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.5 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 446 op/s; 2536/121350 objects degraded (2.090%); 3900/121350 objects misplaced (3.214%); 640 KiB/s, 9 objects/s recovering 2024-03-20T09:29:46.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:46 smithi122 ceph-mon[73495]: pgmap v2041: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.5 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 446 op/s; 2536/121350 objects degraded (2.090%); 3900/121350 objects misplaced (3.214%); 640 KiB/s, 9 objects/s recovering 2024-03-20T09:29:46.733 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:29:46.759 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:46 smithi115 ceph-mon[72476]: pgmap v2041: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.5 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 446 op/s; 2536/121350 objects degraded (2.090%); 3900/121350 objects misplaced (3.214%); 640 KiB/s, 9 objects/s recovering 2024-03-20T09:29:46.761 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:46.781 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:29:46.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:46.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:29:46.885 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:29:46.912 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:46.941 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:29:46.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:46.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:29:47.044 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:29:47.081 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:47.192 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:29:47.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:47.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:29:47.301 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:29:47.329 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:47.350 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:29:47.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:47.354 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:29:47.455 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:29:47.482 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:47.503 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:29:47.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:47.505 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:29:47.606 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:29:47.645 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:47.670 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:29:47.673 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:47 smithi026 ceph-mon[68309]: 1.d scrub starts 2024-03-20T09:29:47.673 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:47 smithi026 ceph-mon[68309]: 1.d scrub ok 2024-03-20T09:29:47.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:47.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:29:47.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:47 smithi122 ceph-mon[73495]: 1.d scrub starts 2024-03-20T09:29:47.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:47 smithi122 ceph-mon[73495]: 1.d scrub ok 2024-03-20T09:29:47.775 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:29:47.805 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:47 smithi115 ceph-mon[72476]: 1.d scrub starts 2024-03-20T09:29:47.805 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:47 smithi115 ceph-mon[72476]: 1.d scrub ok 2024-03-20T09:29:47.807 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:47.831 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:29:47.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:47.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:29:47.936 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:29:47.963 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:47.983 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:29:47.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:47.986 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:29:48.087 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:29:48.115 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:48.135 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:29:48.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:48.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:29:48.162 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:29:48.164 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:29:48.239 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:29:48.257 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:29:48.265 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:48.286 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:29:48.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:48.289 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:29:48.298 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:29:48.349 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:29:48.389 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:29:48.403 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:29:48.450 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:29:48.456 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:29:48.474 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:48.491 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:29:48.535 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:29:48.575 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:29:48.620 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:29:48.659 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:29:48.686 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:29:48.687 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:29:48.687 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:29:48.687 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:29:48.687 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:29:48.687 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:29:48.687 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:29:48.687 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:29:48.687 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:29:48.688 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:29:48.688 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:29:48.688 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-03-20T09:29:48.688 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:29:48.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:48 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 2536/121350 objects degraded (2.090%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:29:48.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:48 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3262196115' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:29:48.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:48 smithi122 ceph-mon[73495]: pgmap v2042: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.4 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 436 op/s; 2389/118470 objects degraded (2.017%); 3662/118470 objects misplaced (3.091%); 1.3 MiB/s, 20 objects/s recovering 2024-03-20T09:29:48.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:48 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3044662237' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:29:48.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:48 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:29:48.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:48 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 2536/121350 objects degraded (2.090%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:29:48.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:48 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3262196115' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:29:48.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:48 smithi026 ceph-mon[68309]: pgmap v2042: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.4 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 436 op/s; 2389/118470 objects degraded (2.017%); 3662/118470 objects misplaced (3.091%); 1.3 MiB/s, 20 objects/s recovering 2024-03-20T09:29:48.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:48 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3044662237' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:29:48.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:48 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:29:48.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:48 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 2536/121350 objects degraded (2.090%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:29:48.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:48 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3262196115' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:29:48.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:48 smithi115 ceph-mon[72476]: pgmap v2042: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.4 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 436 op/s; 2389/118470 objects degraded (2.017%); 3662/118470 objects misplaced (3.091%); 1.3 MiB/s, 20 objects/s recovering 2024-03-20T09:29:48.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:48 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3044662237' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:29:48.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:48 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:29:49.253 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:29:49.256 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:49.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:29:49.357 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:29:49.392 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:49.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:49 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:29:49.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:49 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:29:49.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:49 smithi122 ceph-mon[73495]: osdmap e847: 12 total, 11 up, 11 in 2024-03-20T09:29:49.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:49 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:29:49.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:49 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:29:49.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:49 smithi026 ceph-mon[68309]: osdmap e847: 12 total, 11 up, 11 in 2024-03-20T09:29:49.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:49 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:29:49.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:49 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:29:49.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:49 smithi115 ceph-mon[72476]: osdmap e847: 12 total, 11 up, 11 in 2024-03-20T09:29:50.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:50 smithi122 ceph-mon[73495]: 1.1c scrub starts 2024-03-20T09:29:50.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:50 smithi122 ceph-mon[73495]: 1.1c scrub ok 2024-03-20T09:29:50.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:50 smithi122 ceph-mon[73495]: pgmap v2044: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.3 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 464 op/s; 2389/115314 objects degraded (2.072%); 3662/115314 objects misplaced (3.176%); 1.3 MiB/s, 20 objects/s recovering 2024-03-20T09:29:50.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:50 smithi026 ceph-mon[68309]: 1.1c scrub starts 2024-03-20T09:29:50.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:50 smithi026 ceph-mon[68309]: 1.1c scrub ok 2024-03-20T09:29:50.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:50 smithi026 ceph-mon[68309]: pgmap v2044: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.3 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 464 op/s; 2389/115314 objects degraded (2.072%); 3662/115314 objects misplaced (3.176%); 1.3 MiB/s, 20 objects/s recovering 2024-03-20T09:29:50.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:50 smithi115 ceph-mon[72476]: 1.1c scrub starts 2024-03-20T09:29:50.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:50 smithi115 ceph-mon[72476]: 1.1c scrub ok 2024-03-20T09:29:50.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:50 smithi115 ceph-mon[72476]: pgmap v2044: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.3 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 464 op/s; 2389/115314 objects degraded (2.072%); 3662/115314 objects misplaced (3.176%); 1.3 MiB/s, 20 objects/s recovering 2024-03-20T09:29:51.056 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:29:51.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:51.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:29:51.162 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:29:51.190 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:51.211 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:29:51.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:51.214 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:29:51.316 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:29:51.346 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:51.374 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:29:51.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:51.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:29:51.481 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:29:51.507 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:51.528 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:29:51.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:51.533 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:29:51.634 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:29:51.661 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:51.681 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:29:51.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:51.684 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:29:51.785 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:29:51.819 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:53.468 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:53 smithi026 ceph-mon[68309]: 1.1a scrub starts 2024-03-20T09:29:53.468 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:53 smithi026 ceph-mon[68309]: 1.1a scrub ok 2024-03-20T09:29:53.468 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:53 smithi026 ceph-mon[68309]: pgmap v2045: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.3 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 477 op/s; 2389/112266 objects degraded (2.128%); 3662/112266 objects misplaced (3.262%); 1.0 MiB/s, 16 objects/s recovering 2024-03-20T09:29:53.469 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:53 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 2389/112266 objects degraded (2.128%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:29:53.469 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:53 smithi026 ceph-mon[68309]: osdmap e848: 12 total, 11 up, 11 in 2024-03-20T09:29:53.469 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:53 smithi026 ceph-mon[68309]: 6.6 scrub starts 2024-03-20T09:29:53.469 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:53 smithi026 ceph-mon[68309]: 6.6 scrub ok 2024-03-20T09:29:53.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:53 smithi115 ceph-mon[72476]: 1.1a scrub starts 2024-03-20T09:29:53.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:53 smithi115 ceph-mon[72476]: 1.1a scrub ok 2024-03-20T09:29:53.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:53 smithi115 ceph-mon[72476]: pgmap v2045: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.3 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 477 op/s; 2389/112266 objects degraded (2.128%); 3662/112266 objects misplaced (3.262%); 1.0 MiB/s, 16 objects/s recovering 2024-03-20T09:29:53.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:53 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 2389/112266 objects degraded (2.128%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:29:53.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:53 smithi115 ceph-mon[72476]: osdmap e848: 12 total, 11 up, 11 in 2024-03-20T09:29:53.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:53 smithi115 ceph-mon[72476]: 6.6 scrub starts 2024-03-20T09:29:53.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:53 smithi115 ceph-mon[72476]: 6.6 scrub ok 2024-03-20T09:29:53.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:53 smithi122 ceph-mon[73495]: 1.1a scrub starts 2024-03-20T09:29:53.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:53 smithi122 ceph-mon[73495]: 1.1a scrub ok 2024-03-20T09:29:53.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:53 smithi122 ceph-mon[73495]: pgmap v2045: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.3 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 477 op/s; 2389/112266 objects degraded (2.128%); 3662/112266 objects misplaced (3.262%); 1.0 MiB/s, 16 objects/s recovering 2024-03-20T09:29:53.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:53 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 2389/112266 objects degraded (2.128%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:29:53.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:53 smithi122 ceph-mon[73495]: osdmap e848: 12 total, 11 up, 11 in 2024-03-20T09:29:53.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:53 smithi122 ceph-mon[73495]: 6.6 scrub starts 2024-03-20T09:29:53.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:53 smithi122 ceph-mon[73495]: 6.6 scrub ok 2024-03-20T09:29:53.915 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:29:53.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:53.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:29:53.950 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:29:53.951 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:29:54.026 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:29:54.057 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:54.079 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:29:54.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:54.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:29:54.184 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:29:54.211 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:54.232 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:29:54.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:54.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:29:54.337 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:29:54.367 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:54.388 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:29:54.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:54.394 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:29:54.495 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:29:54.522 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:54.541 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:29:54.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:54.544 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:29:54.645 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:29:54.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:54 smithi115 ceph-mon[72476]: 1.e deep-scrub starts 2024-03-20T09:29:54.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:54 smithi115 ceph-mon[72476]: 1.e deep-scrub ok 2024-03-20T09:29:54.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:54 smithi115 ceph-mon[72476]: 6.12 scrub starts 2024-03-20T09:29:54.670 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:54 smithi122 ceph-mon[73495]: 1.e deep-scrub starts 2024-03-20T09:29:54.670 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:54 smithi122 ceph-mon[73495]: 1.e deep-scrub ok 2024-03-20T09:29:54.670 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:54 smithi122 ceph-mon[73495]: 6.12 scrub starts 2024-03-20T09:29:54.672 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:54.677 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:29:54.677 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:29:54.695 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:29:54.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:54.698 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:29:54.707 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:54 smithi026 ceph-mon[68309]: 1.e deep-scrub starts 2024-03-20T09:29:54.707 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:54 smithi026 ceph-mon[68309]: 1.e deep-scrub ok 2024-03-20T09:29:54.707 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:54 smithi026 ceph-mon[68309]: 6.12 scrub starts 2024-03-20T09:29:54.799 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:29:54.836 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:54.858 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:29:54.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:54.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:29:54.963 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:29:54.990 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:55.011 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:29:55.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:55.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:29:55.043 INFO:teuthology.orchestra.run.smithi026.stderr:no valid command found; 10 closest matches: 2024-03-20T09:29:55.043 INFO:teuthology.orchestra.run.smithi026.stderr:0 2024-03-20T09:29:55.043 INFO:teuthology.orchestra.run.smithi026.stderr:1 2024-03-20T09:29:55.043 INFO:teuthology.orchestra.run.smithi026.stderr:2 2024-03-20T09:29:55.043 INFO:teuthology.orchestra.run.smithi026.stderr:abort 2024-03-20T09:29:55.043 INFO:teuthology.orchestra.run.smithi026.stderr:assert 2024-03-20T09:29:55.043 INFO:teuthology.orchestra.run.smithi026.stderr:bench [] [] [] [] 2024-03-20T09:29:55.043 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs debug_inject_read_zeros 2024-03-20T09:29:55.043 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs files list 2024-03-20T09:29:55.043 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs stats 2024-03-20T09:29:55.043 INFO:teuthology.orchestra.run.smithi026.stderr:bluestore allocator dump block 2024-03-20T09:29:55.043 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: invalid command 2024-03-20T09:29:55.115 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:29:55.142 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:55.160 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:29:55.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:55.163 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:29:55.264 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:29:55.293 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:55.318 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:29:55.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:55.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:29:55.375 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:55 smithi026 ceph-mon[68309]: 6.12 scrub ok 2024-03-20T09:29:55.375 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:55 smithi026 ceph-mon[68309]: 6.8 scrub starts 2024-03-20T09:29:55.375 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:55 smithi026 ceph-mon[68309]: 6.8 scrub ok 2024-03-20T09:29:55.375 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:55 smithi026 ceph-mon[68309]: pgmap v2047: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.3 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 425 op/s; 2244/111123 objects degraded (2.019%); 3421/111123 objects misplaced (3.079%); 1.3 MiB/s, 21 objects/s recovering 2024-03-20T09:29:55.426 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:29:55.440 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:55 smithi122 ceph-mon[73495]: 6.12 scrub ok 2024-03-20T09:29:55.440 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:55 smithi122 ceph-mon[73495]: 6.8 scrub starts 2024-03-20T09:29:55.440 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:55 smithi122 ceph-mon[73495]: 6.8 scrub ok 2024-03-20T09:29:55.440 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:55 smithi122 ceph-mon[73495]: pgmap v2047: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.3 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 425 op/s; 2244/111123 objects degraded (2.019%); 3421/111123 objects misplaced (3.079%); 1.3 MiB/s, 21 objects/s recovering 2024-03-20T09:29:55.457 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:55.482 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:29:55.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:55.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:29:55.586 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:29:55.629 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:29:55.638 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:55.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:55 smithi115 ceph-mon[72476]: 6.12 scrub ok 2024-03-20T09:29:55.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:55 smithi115 ceph-mon[72476]: 6.8 scrub starts 2024-03-20T09:29:55.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:55 smithi115 ceph-mon[72476]: 6.8 scrub ok 2024-03-20T09:29:55.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:55 smithi115 ceph-mon[72476]: pgmap v2047: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.3 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 425 op/s; 2244/111123 objects degraded (2.019%); 3421/111123 objects misplaced (3.079%); 1.3 MiB/s, 21 objects/s recovering 2024-03-20T09:29:55.686 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:29:55.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:55.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:29:55.733 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T09:29:55.733 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-20T09:29:55.734 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.3 'trim stale osdmaps' 2024-03-20T09:29:55.794 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:29:55.820 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:55.839 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:29:55.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:55.843 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:29:55.944 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:29:55.951 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:29:56.006 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:56.232 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:29:56.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:56.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:29:56.255 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-20T09:29:56.303 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:29:56.344 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:29:56.362 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:29:56.371 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:56.391 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:29:56.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:56.394 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:29:56.495 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:29:56.557 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:56.639 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:29:56.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:56.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:29:56.746 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:29:56.780 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:56.802 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:29:56.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:56.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:29:56.906 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:29:56.955 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:56.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:56.985 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:29:56.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:29:57.086 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:29:57.128 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:57.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:57 smithi115 ceph-mon[72476]: 1.a scrub starts 2024-03-20T09:29:57.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:57 smithi115 ceph-mon[72476]: 1.a scrub ok 2024-03-20T09:29:57.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:57 smithi115 ceph-mon[72476]: 1.f scrub starts 2024-03-20T09:29:57.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:57 smithi115 ceph-mon[72476]: 1.f scrub ok 2024-03-20T09:29:57.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:57 smithi115 ceph-mon[72476]: 1.1b scrub starts 2024-03-20T09:29:57.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:57 smithi115 ceph-mon[72476]: 1.1b scrub ok 2024-03-20T09:29:57.651 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:57 smithi115 ceph-mon[72476]: pgmap v2048: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 503 op/s; 2244/106383 objects degraded (2.109%); 3421/106383 objects misplaced (3.216%); 696 KiB/s, 10 objects/s recovering 2024-03-20T09:29:57.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:57 smithi122 ceph-mon[73495]: 1.a scrub starts 2024-03-20T09:29:57.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:57 smithi122 ceph-mon[73495]: 1.a scrub ok 2024-03-20T09:29:57.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:57 smithi122 ceph-mon[73495]: 1.f scrub starts 2024-03-20T09:29:57.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:57 smithi122 ceph-mon[73495]: 1.f scrub ok 2024-03-20T09:29:57.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:57 smithi122 ceph-mon[73495]: 1.1b scrub starts 2024-03-20T09:29:57.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:57 smithi122 ceph-mon[73495]: 1.1b scrub ok 2024-03-20T09:29:57.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:57 smithi122 ceph-mon[73495]: pgmap v2048: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 503 op/s; 2244/106383 objects degraded (2.109%); 3421/106383 objects misplaced (3.216%); 696 KiB/s, 10 objects/s recovering 2024-03-20T09:29:57.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:57 smithi026 ceph-mon[68309]: 1.a scrub starts 2024-03-20T09:29:57.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:57 smithi026 ceph-mon[68309]: 1.a scrub ok 2024-03-20T09:29:57.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:57 smithi026 ceph-mon[68309]: 1.f scrub starts 2024-03-20T09:29:57.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:57 smithi026 ceph-mon[68309]: 1.f scrub ok 2024-03-20T09:29:57.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:57 smithi026 ceph-mon[68309]: 1.1b scrub starts 2024-03-20T09:29:57.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:57 smithi026 ceph-mon[68309]: 1.1b scrub ok 2024-03-20T09:29:57.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:57 smithi026 ceph-mon[68309]: pgmap v2048: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 503 op/s; 2244/106383 objects degraded (2.109%); 3421/106383 objects misplaced (3.216%); 696 KiB/s, 10 objects/s recovering 2024-03-20T09:29:58.291 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:29:58.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:58.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:29:58.398 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:29:58.434 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:58.463 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:29:58.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:58.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:29:58.567 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:58 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 2244/106383 objects degraded (2.109%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:29:58.571 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:29:58.603 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:58.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:58 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 2244/106383 objects degraded (2.109%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:29:58.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:58 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 2244/106383 objects degraded (2.109%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:29:58.819 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:29:58.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:58.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:29:58.834 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:29:58.928 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:29:58.957 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:58.980 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:29:58.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:58.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:29:59.084 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:29:59.110 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:59.131 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:29:59.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:59.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:29:59.235 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:29:59.262 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:59.282 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:29:59.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:59.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:29:59.386 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:29:59.401 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:29:59.413 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:59.437 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:29:59.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:59.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:29:59.541 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:29:59.566 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:29:59 smithi115 ceph-mon[72476]: pgmap v2049: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.1 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 506 op/s; 2058/103458 objects degraded (1.989%); 3173/103458 objects misplaced (3.067%); 1.1 MiB/s, 17 objects/s recovering 2024-03-20T09:29:59.568 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:59.586 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:29:59.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:59.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:29:59.657 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:29:59 smithi026 ceph-mon[68309]: pgmap v2049: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.1 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 506 op/s; 2058/103458 objects degraded (1.989%); 3173/103458 objects misplaced (3.067%); 1.1 MiB/s, 17 objects/s recovering 2024-03-20T09:29:59.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:29:59 smithi122 ceph-mon[73495]: pgmap v2049: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.1 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 506 op/s; 2058/103458 objects degraded (1.989%); 3173/103458 objects misplaced (3.067%); 1.1 MiB/s, 17 objects/s recovering 2024-03-20T09:29:59.690 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:29:59.715 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:59.734 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:29:59.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:59.739 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:29:59.841 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:29:59.867 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:29:59.886 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:29:59.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:29:59.888 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:29:59.989 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:30:00.020 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:00.042 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:30:00.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:00.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:30:00.147 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:30:00.175 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:00.195 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:30:00.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:00.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:30:00.298 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:30:00.327 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:00.352 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:30:00.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:00.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:30:00.457 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:30:00.510 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:00.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:00 smithi115 ceph-mon[72476]: 6.b scrub starts 2024-03-20T09:30:00.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:00 smithi115 ceph-mon[72476]: 6.b scrub ok 2024-03-20T09:30:00.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:00 smithi115 ceph-mon[72476]: Health detail: HEALTH_WARN Degraded data redundancy: 2058/103458 objects degraded (1.989%), 1 pg degraded, 2 pgs undersized 2024-03-20T09:30:00.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:00 smithi115 ceph-mon[72476]: [WRN] PG_DEGRADED: Degraded data redundancy: 2058/103458 objects degraded (1.989%), 1 pg degraded, 2 pgs undersized 2024-03-20T09:30:00.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:00 smithi115 ceph-mon[72476]: pg 6.f is stuck undersized for 96s, current state active+undersized+degraded+remapped+backfilling, last acting [2,9] 2024-03-20T09:30:00.650 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:00 smithi115 ceph-mon[72476]: pg 6.17 is stuck undersized for 97s, current state active+recovering+undersized+remapped, last acting [2,11] 2024-03-20T09:30:00.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:00 smithi122 ceph-mon[73495]: 6.b scrub starts 2024-03-20T09:30:00.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:00 smithi122 ceph-mon[73495]: 6.b scrub ok 2024-03-20T09:30:00.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:00 smithi122 ceph-mon[73495]: Health detail: HEALTH_WARN Degraded data redundancy: 2058/103458 objects degraded (1.989%), 1 pg degraded, 2 pgs undersized 2024-03-20T09:30:00.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:00 smithi122 ceph-mon[73495]: [WRN] PG_DEGRADED: Degraded data redundancy: 2058/103458 objects degraded (1.989%), 1 pg degraded, 2 pgs undersized 2024-03-20T09:30:00.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:00 smithi122 ceph-mon[73495]: pg 6.f is stuck undersized for 96s, current state active+undersized+degraded+remapped+backfilling, last acting [2,9] 2024-03-20T09:30:00.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:00 smithi122 ceph-mon[73495]: pg 6.17 is stuck undersized for 97s, current state active+recovering+undersized+remapped, last acting [2,11] 2024-03-20T09:30:00.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:00 smithi026 ceph-mon[68309]: 6.b scrub starts 2024-03-20T09:30:00.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:00 smithi026 ceph-mon[68309]: 6.b scrub ok 2024-03-20T09:30:00.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:00 smithi026 ceph-mon[68309]: Health detail: HEALTH_WARN Degraded data redundancy: 2058/103458 objects degraded (1.989%), 1 pg degraded, 2 pgs undersized 2024-03-20T09:30:00.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:00 smithi026 ceph-mon[68309]: [WRN] PG_DEGRADED: Degraded data redundancy: 2058/103458 objects degraded (1.989%), 1 pg degraded, 2 pgs undersized 2024-03-20T09:30:00.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:00 smithi026 ceph-mon[68309]: pg 6.f is stuck undersized for 96s, current state active+undersized+degraded+remapped+backfilling, last acting [2,9] 2024-03-20T09:30:00.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:00 smithi026 ceph-mon[68309]: pg 6.17 is stuck undersized for 97s, current state active+recovering+undersized+remapped, last acting [2,11] 2024-03-20T09:30:01.273 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:30:01.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:01.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:30:01.378 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:30:01.415 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:01.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:01 smithi122 ceph-mon[73495]: 1.7 scrub starts 2024-03-20T09:30:01.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:01 smithi122 ceph-mon[73495]: 1.7 scrub ok 2024-03-20T09:30:01.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:01 smithi122 ceph-mon[73495]: pgmap v2050: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.0 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 505 op/s; 2058/100149 objects degraded (2.055%); 3173/100149 objects misplaced (3.168%); 1.1 MiB/s, 17 objects/s recovering 2024-03-20T09:30:01.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:01 smithi026 ceph-mon[68309]: 1.7 scrub starts 2024-03-20T09:30:01.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:01 smithi026 ceph-mon[68309]: 1.7 scrub ok 2024-03-20T09:30:01.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:01 smithi026 ceph-mon[68309]: pgmap v2050: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.0 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 505 op/s; 2058/100149 objects degraded (2.055%); 3173/100149 objects misplaced (3.168%); 1.1 MiB/s, 17 objects/s recovering 2024-03-20T09:30:01.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:01 smithi115 ceph-mon[72476]: 1.7 scrub starts 2024-03-20T09:30:01.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:01 smithi115 ceph-mon[72476]: 1.7 scrub ok 2024-03-20T09:30:01.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:01 smithi115 ceph-mon[72476]: pgmap v2050: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.0 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 505 op/s; 2058/100149 objects degraded (2.055%); 3173/100149 objects misplaced (3.168%); 1.1 MiB/s, 17 objects/s recovering 2024-03-20T09:30:02.282 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:30:02.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:02.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:30:02.396 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:30:02.421 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:02 smithi026 ceph-mon[68309]: 6.e scrub starts 2024-03-20T09:30:02.421 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:02 smithi026 ceph-mon[68309]: 6.e scrub ok 2024-03-20T09:30:02.423 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:02.449 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:30:02.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:02.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:30:02.554 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:30:02.581 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:02.603 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:30:02.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:02.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:30:02.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:02 smithi122 ceph-mon[73495]: 6.e scrub starts 2024-03-20T09:30:02.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:02 smithi122 ceph-mon[73495]: 6.e scrub ok 2024-03-20T09:30:02.711 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:30:02.740 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:02.760 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:30:02.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:02.763 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:30:02.864 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:30:02.892 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:02.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:02 smithi115 ceph-mon[72476]: 6.e scrub starts 2024-03-20T09:30:02.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:02 smithi115 ceph-mon[72476]: 6.e scrub ok 2024-03-20T09:30:02.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:02.919 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:30:02.919 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:30:03.020 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:30:03.047 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:03.067 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:30:03.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:03.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:30:03.170 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:30:03.220 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:03.284 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:30:03.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:03.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:30:03.397 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:30:03.425 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:03.447 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:30:03.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:03.452 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:30:03.494 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:03 smithi026 ceph-mon[68309]: pgmap v2051: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.0 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 500 op/s; 2058/97251 objects degraded (2.116%); 3173/97251 objects misplaced (3.263%); 1.1 MiB/s, 17 objects/s recovering 2024-03-20T09:30:03.494 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:03 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 2058/97251 objects degraded (2.116%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:30:03.553 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:30:03.632 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:03.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:03 smithi122 ceph-mon[73495]: pgmap v2051: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.0 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 500 op/s; 2058/97251 objects degraded (2.116%); 3173/97251 objects misplaced (3.263%); 1.1 MiB/s, 17 objects/s recovering 2024-03-20T09:30:03.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:03 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 2058/97251 objects degraded (2.116%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:30:03.824 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:30:03.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:03.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:30:03.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:03 smithi115 ceph-mon[72476]: pgmap v2051: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.0 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 500 op/s; 2058/97251 objects degraded (2.116%); 3173/97251 objects misplaced (3.263%); 1.1 MiB/s, 17 objects/s recovering 2024-03-20T09:30:03.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:03 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 2058/97251 objects degraded (2.116%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:30:03.929 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:30:03.965 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:03.994 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:30:03.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:03.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:30:04.098 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:30:04.131 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:04.160 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:30:04.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:04.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:30:04.267 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:30:04.331 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:04.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:04 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1081781656' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:30:04.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:04 smithi122 ceph-mon[73495]: pgmap v2052: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.0 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 456 op/s; 1921/96129 objects degraded (1.998%); 2899/96129 objects misplaced (3.016%); 1.5 MiB/s, 23 objects/s recovering 2024-03-20T09:30:04.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:04 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2476064768' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:30:04.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:04 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1081781656' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:30:04.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:04 smithi026 ceph-mon[68309]: pgmap v2052: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.0 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 456 op/s; 1921/96129 objects degraded (1.998%); 2899/96129 objects misplaced (3.016%); 1.5 MiB/s, 23 objects/s recovering 2024-03-20T09:30:04.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:04 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2476064768' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:30:04.856 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:30:04.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:04 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1081781656' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:30:04.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:04 smithi115 ceph-mon[72476]: pgmap v2052: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 2.0 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 456 op/s; 1921/96129 objects degraded (1.998%); 2899/96129 objects misplaced (3.016%); 1.5 MiB/s, 23 objects/s recovering 2024-03-20T09:30:04.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:04 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2476064768' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:30:05.564 INFO:teuthology.orchestra.run.smithi026.stderr:no valid command found; 10 closest matches: 2024-03-20T09:30:05.564 INFO:teuthology.orchestra.run.smithi026.stderr:0 2024-03-20T09:30:05.564 INFO:teuthology.orchestra.run.smithi026.stderr:1 2024-03-20T09:30:05.564 INFO:teuthology.orchestra.run.smithi026.stderr:2 2024-03-20T09:30:05.564 INFO:teuthology.orchestra.run.smithi026.stderr:abort 2024-03-20T09:30:05.564 INFO:teuthology.orchestra.run.smithi026.stderr:assert 2024-03-20T09:30:05.564 INFO:teuthology.orchestra.run.smithi026.stderr:bench [] [] [] [] 2024-03-20T09:30:05.564 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs debug_inject_read_zeros 2024-03-20T09:30:05.565 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs files list 2024-03-20T09:30:05.565 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs stats 2024-03-20T09:30:05.565 INFO:teuthology.orchestra.run.smithi026.stderr:bluestore allocator dump block 2024-03-20T09:30:05.565 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: invalid command 2024-03-20T09:30:05.597 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:05 smithi026 ceph-mon[68309]: 1.15 scrub starts 2024-03-20T09:30:05.598 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:05 smithi026 ceph-mon[68309]: 1.15 scrub ok 2024-03-20T09:30:05.598 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:05 smithi026 ceph-mon[68309]: 6.13 scrub starts 2024-03-20T09:30:05.598 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:05 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:30:05.598 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:05 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1081781656' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:30:05.598 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:05 smithi026 ceph-mon[68309]: osdmap e849: 12 total, 11 up, 11 in 2024-03-20T09:30:05.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:05 smithi122 ceph-mon[73495]: 1.15 scrub starts 2024-03-20T09:30:05.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:05 smithi122 ceph-mon[73495]: 1.15 scrub ok 2024-03-20T09:30:05.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:05 smithi122 ceph-mon[73495]: 6.13 scrub starts 2024-03-20T09:30:05.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:05 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:30:05.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:05 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1081781656' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:30:05.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:05 smithi122 ceph-mon[73495]: osdmap e849: 12 total, 11 up, 11 in 2024-03-20T09:30:05.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:05 smithi115 ceph-mon[72476]: 1.15 scrub starts 2024-03-20T09:30:05.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:05 smithi115 ceph-mon[72476]: 1.15 scrub ok 2024-03-20T09:30:05.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:05 smithi115 ceph-mon[72476]: 6.13 scrub starts 2024-03-20T09:30:05.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:05 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:30:05.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:05 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1081781656' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:30:05.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:05 smithi115 ceph-mon[72476]: osdmap e849: 12 total, 11 up, 11 in 2024-03-20T09:30:06.321 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:30:06.321 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:30:06.321 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:30:06.321 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:30:06.321 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:30:06.321 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:30:06.322 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:30:06.322 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:30:06.322 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:30:06.322 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:30:06.322 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:30:06.322 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:30:06.322 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:30:06.322 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:30:06.322 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:30:06.323 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:30:06.323 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:30:06.323 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:30:06.323 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:30:06.323 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:30:06.323 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:30:06.323 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-03-20T09:30:06.323 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:30:06.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:06 smithi122 ceph-mon[73495]: pgmap v2054: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 1.8 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 521 op/s; 1921/90738 objects degraded (2.117%); 2899/90738 objects misplaced (3.195%); 1.2 MiB/s, 18 objects/s recovering 2024-03-20T09:30:06.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:06 smithi026 ceph-mon[68309]: pgmap v2054: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 1.8 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 521 op/s; 1921/90738 objects degraded (2.117%); 2899/90738 objects misplaced (3.195%); 1.2 MiB/s, 18 objects/s recovering 2024-03-20T09:30:06.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:06 smithi115 ceph-mon[72476]: pgmap v2054: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 1.8 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 521 op/s; 1921/90738 objects degraded (2.117%); 2899/90738 objects misplaced (3.195%); 1.2 MiB/s, 18 objects/s recovering 2024-03-20T09:30:07.013 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:30:07.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:07.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:30:07.055 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:30:07.056 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:30:07.121 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:30:07.149 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:07.170 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:30:07.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:07.173 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:30:07.273 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:30:07.300 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:07.319 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:30:07.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:07.323 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:30:07.424 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:30:07.455 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:07.981 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:30:07.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:07.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:30:08.086 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:30:08.137 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:08.470 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:30:08.471 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:30:08.471 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-03-20T09:30:08.471 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-03-20T09:30:08.471 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:30:08.700 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:30:08.706 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:08 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:30:08.706 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:08 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:30:08.707 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:08 smithi026 ceph-mon[68309]: osdmap e850: 12 total, 11 up, 11 in 2024-03-20T09:30:08.707 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:08 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 1921/90738 objects degraded (2.117%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:30:08.707 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:08 smithi026 ceph-mon[68309]: pgmap v2056: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 1.8 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 528 op/s; 1732/87459 objects degraded (1.980%); 2624/87459 objects misplaced (3.000%); 1.6 MiB/s, 24 objects/s recovering 2024-03-20T09:30:08.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:08.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:30:08.810 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:30:08.835 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:08 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:30:08.835 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:08 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:30:08.835 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:08 smithi122 ceph-mon[73495]: osdmap e850: 12 total, 11 up, 11 in 2024-03-20T09:30:08.835 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:08 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 1921/90738 objects degraded (2.117%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:30:08.836 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:08 smithi122 ceph-mon[73495]: pgmap v2056: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 1.8 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 528 op/s; 1732/87459 objects degraded (1.980%); 2624/87459 objects misplaced (3.000%); 1.6 MiB/s, 24 objects/s recovering 2024-03-20T09:30:08.837 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:08.857 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:30:08.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:08.859 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:30:08.860 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T09:30:08.860 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-20T09:30:08.860 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.4 'trim stale osdmaps' 2024-03-20T09:30:08.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:08 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:30:08.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:08 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:30:08.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:08 smithi115 ceph-mon[72476]: osdmap e850: 12 total, 11 up, 11 in 2024-03-20T09:30:08.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:08 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 1921/90738 objects degraded (2.117%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:30:08.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:08 smithi115 ceph-mon[72476]: pgmap v2056: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 1.8 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 528 op/s; 1732/87459 objects degraded (1.980%); 2624/87459 objects misplaced (3.000%); 1.6 MiB/s, 24 objects/s recovering 2024-03-20T09:30:08.960 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:30:08.993 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:09.059 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:30:09.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:09.360 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:30:09.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:30:09.419 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-20T09:30:09.461 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:30:09.471 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:30:09.519 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:30:09.528 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:09.720 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:30:09.765 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:30:09.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:09.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:30:09.877 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:30:09.912 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:09.936 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:30:09.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:09.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:30:10.040 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:30:10.110 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:10.413 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:30:10.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:10.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:30:10.523 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:30:10.554 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:10.584 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:30:10.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:10.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:30:10.689 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:30:10.717 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:10.746 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:30:10.749 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:10.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:30:10.851 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:30:10.907 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:11.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:11 smithi115 ceph-mon[72476]: pgmap v2057: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 1.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 552 op/s; 1732/83982 objects degraded (2.062%); 2624/83982 objects misplaced (3.124%); 1.6 MiB/s, 24 objects/s recovering 2024-03-20T09:30:11.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:11 smithi122 ceph-mon[73495]: pgmap v2057: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 1.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 552 op/s; 1732/83982 objects degraded (2.062%); 2624/83982 objects misplaced (3.124%); 1.6 MiB/s, 24 objects/s recovering 2024-03-20T09:30:11.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:11 smithi026 ceph-mon[68309]: pgmap v2057: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 1.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 552 op/s; 1732/83982 objects degraded (2.062%); 2624/83982 objects misplaced (3.124%); 1.6 MiB/s, 24 objects/s recovering 2024-03-20T09:30:12.466 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:30:12.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:12.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:30:12.515 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:30:12.576 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:30:12.604 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:12.624 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:30:12.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:12.629 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:30:12.730 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:30:12.765 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:12.791 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:30:12.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:12.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:30:12.896 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:30:12.950 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:12.986 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:30:12.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:12.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:30:13.016 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:30:13.093 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:30:13.120 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:13.141 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:30:13.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:13.144 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:30:13.245 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:30:13.274 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:13.296 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:30:13.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:13.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:30:13.402 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:30:13.430 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:13.452 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:30:13.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:13.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:30:13.556 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:30:13.584 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:13 smithi115 ceph-mon[72476]: pgmap v2058: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 1.6 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 650 op/s; 1732/80514 objects degraded (2.151%); 2624/80514 objects misplaced (3.259%); 824 KiB/s, 12 objects/s recovering 2024-03-20T09:30:13.584 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:13 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:30:13.584 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:13 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:30:13.584 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:13 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 1732/80514 objects degraded (2.151%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:30:13.589 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:13.612 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:30:13.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:13.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:30:13.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:13 smithi122 ceph-mon[73495]: pgmap v2058: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 1.6 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 650 op/s; 1732/80514 objects degraded (2.151%); 2624/80514 objects misplaced (3.259%); 824 KiB/s, 12 objects/s recovering 2024-03-20T09:30:13.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:13 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:30:13.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:13 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:30:13.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:13 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 1732/80514 objects degraded (2.151%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:30:13.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:13 smithi026 ceph-mon[68309]: pgmap v2058: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 1.6 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 650 op/s; 1732/80514 objects degraded (2.151%); 2624/80514 objects misplaced (3.259%); 824 KiB/s, 12 objects/s recovering 2024-03-20T09:30:13.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:13 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:30:13.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:13 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:30:13.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:13 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 1732/80514 objects degraded (2.151%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:30:13.719 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:30:13.745 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:13.764 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:30:13.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:13.767 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:30:13.868 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:30:13.894 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:13.912 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:30:13.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:13.915 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:30:14.016 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:30:14.036 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:30:14.065 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:30:14.078 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:14.111 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:30:14.142 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:30:14.185 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:30:14.229 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:30:14.263 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:30:14.300 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:30:14.319 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:30:14.350 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:30:14.398 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:30:14.399 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:30:14.399 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:30:14.399 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:30:14.399 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:30:14.399 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:30:14.399 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:30:14.399 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:30:14.399 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:30:14.399 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:30:14.399 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:30:14.399 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-03-20T09:30:14.399 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:30:14.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:14 smithi115 ceph-mon[72476]: osdmap e851: 12 total, 11 up, 11 in 2024-03-20T09:30:14.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:14 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/181736159' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:30:14.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:14 smithi122 ceph-mon[73495]: osdmap e851: 12 total, 11 up, 11 in 2024-03-20T09:30:14.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:14 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/181736159' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:30:14.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:14 smithi026 ceph-mon[68309]: osdmap e851: 12 total, 11 up, 11 in 2024-03-20T09:30:14.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:14 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/181736159' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:30:15.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:15 smithi122 ceph-mon[73495]: pgmap v2060: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 1.6 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 488 op/s; 1537/79017 objects degraded (1.945%); 2335/79017 objects misplaced (2.955%); 1.7 MiB/s, 26 objects/s recovering 2024-03-20T09:30:15.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:15 smithi026 ceph-mon[68309]: pgmap v2060: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 1.6 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 488 op/s; 1537/79017 objects degraded (1.945%); 2335/79017 objects misplaced (2.955%); 1.7 MiB/s, 26 objects/s recovering 2024-03-20T09:30:15.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:15 smithi115 ceph-mon[72476]: pgmap v2060: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 1.6 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 488 op/s; 1537/79017 objects degraded (1.945%); 2335/79017 objects misplaced (2.955%); 1.7 MiB/s, 26 objects/s recovering 2024-03-20T09:30:16.101 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:30:16.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:16.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:30:16.209 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:30:16.264 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:16.289 INFO:teuthology.orchestra.run.smithi026.stderr:no valid command found; 10 closest matches: 2024-03-20T09:30:16.289 INFO:teuthology.orchestra.run.smithi026.stderr:0 2024-03-20T09:30:16.290 INFO:teuthology.orchestra.run.smithi026.stderr:1 2024-03-20T09:30:16.290 INFO:teuthology.orchestra.run.smithi026.stderr:2 2024-03-20T09:30:16.290 INFO:teuthology.orchestra.run.smithi026.stderr:abort 2024-03-20T09:30:16.290 INFO:teuthology.orchestra.run.smithi026.stderr:assert 2024-03-20T09:30:16.290 INFO:teuthology.orchestra.run.smithi026.stderr:bench [] [] [] [] 2024-03-20T09:30:16.290 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs debug_inject_read_zeros 2024-03-20T09:30:16.290 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs files list 2024-03-20T09:30:16.290 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs stats 2024-03-20T09:30:16.290 INFO:teuthology.orchestra.run.smithi026.stderr:bluestore allocator dump block 2024-03-20T09:30:16.290 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: invalid command 2024-03-20T09:30:16.333 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:30:16.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:16.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:30:16.442 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:30:16.471 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:16.491 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:30:16.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:16.494 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:30:16.595 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:30:16.624 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:16.926 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:30:16.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:16.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:30:17.030 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:30:17.054 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:30:17.055 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:30:17.097 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:17.415 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:17 smithi026 ceph-mon[68309]: pgmap v2061: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 1.5 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 592 op/s; 1537/73071 objects degraded (2.103%); 2335/73071 objects misplaced (3.196%); 1.5 MiB/s, 23 objects/s recovering 2024-03-20T09:30:17.437 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:30:17.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:17.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:30:17.544 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:30:17.579 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:17.603 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:30:17.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:17.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:30:17.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:17 smithi122 ceph-mon[73495]: pgmap v2061: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 1.5 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 592 op/s; 1537/73071 objects degraded (2.103%); 2335/73071 objects misplaced (3.196%); 1.5 MiB/s, 23 objects/s recovering 2024-03-20T09:30:17.707 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:30:17.735 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:17 smithi115 ceph-mon[72476]: pgmap v2061: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 1.5 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 592 op/s; 1537/73071 objects degraded (2.103%); 2335/73071 objects misplaced (3.196%); 1.5 MiB/s, 23 objects/s recovering 2024-03-20T09:30:17.737 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:17.758 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:30:17.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:17.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:30:17.866 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:30:17.892 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T09:30:17.892 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-20T09:30:17.892 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.9 'trim stale osdmaps' 2024-03-20T09:30:17.899 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:17.922 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:30:17.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:17.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:30:18.028 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:30:18.106 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:18.346 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:30:18.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:18.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:30:18.462 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:30:18.489 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:18.509 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:30:18.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:18.512 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:30:18.613 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:30:18.640 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:18.660 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:30:18.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:18.663 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:30:18.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:18 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:30:18.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:18 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:30:18.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:18 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 1537/73071 objects degraded (2.103%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:30:18.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:18 smithi122 ceph-mon[73495]: osdmap e852: 12 total, 11 up, 11 in 2024-03-20T09:30:18.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:18 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:30:18.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:18 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:30:18.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:18 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 1537/73071 objects degraded (2.103%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:30:18.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:18 smithi026 ceph-mon[68309]: osdmap e852: 12 total, 11 up, 11 in 2024-03-20T09:30:18.764 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:30:18.828 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:18.858 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:30:18.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:18.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:30:18.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:18 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:30:18.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:18 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:30:18.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:18 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 1537/73071 objects degraded (2.103%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:30:18.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:18 smithi115 ceph-mon[72476]: osdmap e852: 12 total, 11 up, 11 in 2024-03-20T09:30:18.962 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:30:18.991 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:19.010 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:30:19.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:19.014 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:30:19.055 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:30:19.115 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:30:19.143 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:19.164 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:30:19.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:19.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:30:19.270 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:30:19.367 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:19.549 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:30:19.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:19.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:30:19.638 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:19 smithi026 ceph-mon[68309]: pgmap v2063: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 1.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 597 op/s; 1360/69630 objects degraded (1.953%); 2014/69630 objects misplaced (2.892%); 1.8 MiB/s, 28 objects/s recovering 2024-03-20T09:30:19.659 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:30:19.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:19 smithi122 ceph-mon[73495]: pgmap v2063: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 1.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 597 op/s; 1360/69630 objects degraded (1.953%); 2014/69630 objects misplaced (2.892%); 1.8 MiB/s, 28 objects/s recovering 2024-03-20T09:30:19.685 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:19.705 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:30:19.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:19.708 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:30:19.809 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:30:19.856 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:19.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:19 smithi115 ceph-mon[72476]: pgmap v2063: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 1.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 597 op/s; 1360/69630 objects degraded (1.953%); 2014/69630 objects misplaced (2.892%); 1.8 MiB/s, 28 objects/s recovering 2024-03-20T09:30:20.454 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:30:20.635 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:30:20.635 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:30:20.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:20 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/121642088' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:30:20.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:20 smithi122 ceph-mon[73495]: pgmap v2064: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 1.3 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 616 op/s; 1360/65709 objects degraded (2.070%); 2014/65709 objects misplaced (3.065%); 1.8 MiB/s, 28 objects/s recovering 2024-03-20T09:30:20.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:20 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/121642088' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:30:20.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:20 smithi026 ceph-mon[68309]: pgmap v2064: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 1.3 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 616 op/s; 1360/65709 objects degraded (2.070%); 2014/65709 objects misplaced (3.065%); 1.8 MiB/s, 28 objects/s recovering 2024-03-20T09:30:20.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:20 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/121642088' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:30:20.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:20 smithi115 ceph-mon[72476]: pgmap v2064: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 1.3 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 616 op/s; 1360/65709 objects degraded (2.070%); 2014/65709 objects misplaced (3.065%); 1.8 MiB/s, 28 objects/s recovering 2024-03-20T09:30:21.237 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:30:21.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:21.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:30:21.341 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:30:21.393 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:21.437 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:30:21.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:21.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:30:21.544 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:21 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:30:21.545 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:21 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/121642088' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:30:21.545 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:21 smithi026 ceph-mon[68309]: osdmap e853: 12 total, 11 up, 11 in 2024-03-20T09:30:21.546 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:30:21.582 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:21.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:21 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:30:21.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:21 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/121642088' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:30:21.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:21 smithi122 ceph-mon[73495]: osdmap e853: 12 total, 11 up, 11 in 2024-03-20T09:30:21.702 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:30:21.709 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:21 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:30:21.709 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:21 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/121642088' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:30:21.709 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:21 smithi115 ceph-mon[72476]: osdmap e853: 12 total, 11 up, 11 in 2024-03-20T09:30:21.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:21.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:30:21.811 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:30:21.838 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:21.859 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:30:21.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:21.862 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:30:21.962 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:30:21.989 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:22.008 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:30:22.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:22.011 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:30:22.112 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:30:22.141 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:22.161 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:30:22.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:22.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:30:22.265 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:30:22.300 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:22.514 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:30:22.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:22.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:30:22.534 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:30:22.604 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:30:22.610 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-20T09:30:22.621 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:30:22.648 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:22.668 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:30:22.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:22.671 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:30:22.700 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:22 smithi026 ceph-mon[68309]: pgmap v2066: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 1.3 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 707 op/s; 1360/62031 objects degraded (2.192%); 2014/62031 objects misplaced (3.247%); 919 KiB/s, 14 objects/s recovering 2024-03-20T09:30:22.772 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:30:22.806 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:22 smithi115 ceph-mon[72476]: pgmap v2066: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 1.3 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 707 op/s; 1360/62031 objects degraded (2.192%); 2014/62031 objects misplaced (3.247%); 919 KiB/s, 14 objects/s recovering 2024-03-20T09:30:22.809 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:22.838 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:30:22.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:22.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:30:22.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:22 smithi122 ceph-mon[73495]: pgmap v2066: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 1.3 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 707 op/s; 1360/62031 objects degraded (2.192%); 2014/62031 objects misplaced (3.247%); 919 KiB/s, 14 objects/s recovering 2024-03-20T09:30:22.942 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:30:22.970 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:22.991 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:30:22.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:22.994 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:30:23.095 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:30:23.122 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:23.140 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:30:23.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:23.145 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:30:23.246 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:30:23.287 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:23.450 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:23 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 1360/62031 objects degraded (2.192%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:30:23.450 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:23 smithi026 ceph-mon[68309]: osdmap e854: 12 total, 11 up, 11 in 2024-03-20T09:30:23.561 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:30:23.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:23.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:30:23.667 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:30:23.695 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:23.714 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:30:23.716 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:23 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 1360/62031 objects degraded (2.192%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:30:23.716 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:23 smithi122 ceph-mon[73495]: osdmap e854: 12 total, 11 up, 11 in 2024-03-20T09:30:23.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:23.718 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:30:23.819 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:30:23.871 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:23.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:23 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 1360/62031 objects degraded (2.192%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:30:23.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:23 smithi115 ceph-mon[72476]: osdmap e854: 12 total, 11 up, 11 in 2024-03-20T09:30:24.089 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:30:24.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:24.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:30:24.157 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:30:24.193 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:30:24.225 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:24.250 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:30:24.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:24.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:30:24.354 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:30:24.486 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:24.568 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:30:24.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:24.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:30:24.684 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:30:24.722 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:24.759 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:30:24.768 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:24 smithi115 ceph-mon[72476]: pgmap v2068: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 1.2 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 518 op/s; 1151/60468 objects degraded (1.903%); 1695/60468 objects misplaced (2.803%); 2.3 MiB/s, 36 objects/s recovering 2024-03-20T09:30:24.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:24.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:30:24.871 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:30:24.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:24 smithi122 ceph-mon[73495]: pgmap v2068: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 1.2 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 518 op/s; 1151/60468 objects degraded (1.903%); 1695/60468 objects misplaced (2.803%); 2.3 MiB/s, 36 objects/s recovering 2024-03-20T09:30:24.954 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:24 smithi026 ceph-mon[68309]: pgmap v2068: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 1.2 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 518 op/s; 1151/60468 objects degraded (1.903%); 1695/60468 objects misplaced (2.803%); 2.3 MiB/s, 36 objects/s recovering 2024-03-20T09:30:24.957 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:25.016 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:30:25.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:25.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:30:25.136 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:30:25.199 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:25.256 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:30:25.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:25.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:30:25.267 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:30:25.366 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:30:25.411 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:25.449 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:30:25.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:25.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:30:25.556 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:30:25.647 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:25.785 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:30:25.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:25.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:30:25.895 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:30:25.923 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:25.946 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:30:25.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:25.951 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:30:26.052 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:30:26.085 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:26.120 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:30:26.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:26.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:30:26.228 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:30:26.255 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:26.277 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:30:26.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:26.280 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:30:26.381 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:30:26.409 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:26.443 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:30:26.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:26.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:30:26.474 INFO:teuthology.orchestra.run.smithi026.stderr:no valid command found; 10 closest matches: 2024-03-20T09:30:26.474 INFO:teuthology.orchestra.run.smithi026.stderr:0 2024-03-20T09:30:26.474 INFO:teuthology.orchestra.run.smithi026.stderr:1 2024-03-20T09:30:26.474 INFO:teuthology.orchestra.run.smithi026.stderr:2 2024-03-20T09:30:26.474 INFO:teuthology.orchestra.run.smithi026.stderr:abort 2024-03-20T09:30:26.474 INFO:teuthology.orchestra.run.smithi026.stderr:assert 2024-03-20T09:30:26.474 INFO:teuthology.orchestra.run.smithi026.stderr:bench [] [] [] [] 2024-03-20T09:30:26.474 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs debug_inject_read_zeros 2024-03-20T09:30:26.474 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs files list 2024-03-20T09:30:26.474 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs stats 2024-03-20T09:30:26.475 INFO:teuthology.orchestra.run.smithi026.stderr:bluestore allocator dump block 2024-03-20T09:30:26.475 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: invalid command 2024-03-20T09:30:26.549 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:30:26.578 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:26.700 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:30:26.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:26.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:30:26.807 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:30:26.837 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:26.857 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:30:26.860 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:26.861 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:30:26.962 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:30:27.048 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:27.188 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:30:27.188 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:30:27.188 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:30:27.189 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:30:27.189 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:30:27.189 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:30:27.189 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:30:27.189 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:30:27.189 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:30:27.189 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:30:27.189 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:30:27.189 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:30:27.189 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:30:27.189 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:30:27.190 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:30:27.190 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:30:27.190 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:30:27.190 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:30:27.190 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:30:27.190 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:30:27.190 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:30:27.190 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-03-20T09:30:27.190 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:30:27.198 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:30:27.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:27.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:30:27.311 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:30:27.338 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:27.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:27 smithi115 ceph-mon[72476]: pgmap v2069: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 1.1 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 647 op/s; 1151/54087 objects degraded (2.128%); 1695/54087 objects misplaced (3.134%); 1.0 MiB/s, 16 objects/s recovering 2024-03-20T09:30:27.652 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:27 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3190715360' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:30:27.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:27 smithi122 ceph-mon[73495]: pgmap v2069: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 1.1 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 647 op/s; 1151/54087 objects degraded (2.128%); 1695/54087 objects misplaced (3.134%); 1.0 MiB/s, 16 objects/s recovering 2024-03-20T09:30:27.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:27 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3190715360' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:30:27.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:27 smithi026 ceph-mon[68309]: pgmap v2069: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 1.1 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 647 op/s; 1151/54087 objects degraded (2.128%); 1695/54087 objects misplaced (3.134%); 1.0 MiB/s, 16 objects/s recovering 2024-03-20T09:30:27.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:27 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3190715360' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:30:28.407 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:30:28.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:28.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:30:28.517 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:30:28.595 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:28.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:28 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 1151/54087 objects degraded (2.128%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:30:28.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:28 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 1151/54087 objects degraded (2.128%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:30:28.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:28 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 1151/54087 objects degraded (2.128%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:30:28.732 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:30:28.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:28.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:30:28.759 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:30:28.760 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:30:28.841 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:30:28.895 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:28.959 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:30:28.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:28.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:30:29.067 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:30:29.120 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:29.392 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:29 smithi026 ceph-mon[68309]: pgmap v2070: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 1.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 649 op/s; 793/50112 objects degraded (1.582%); 1397/50112 objects misplaced (2.788%); 3.0 MiB/s, 47 objects/s recovering 2024-03-20T09:30:29.531 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T09:30:29.532 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-20T09:30:29.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:29 smithi115 ceph-mon[72476]: pgmap v2070: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 1.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 649 op/s; 793/50112 objects degraded (1.582%); 1397/50112 objects misplaced (2.788%); 3.0 MiB/s, 47 objects/s recovering 2024-03-20T09:30:29.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:29 smithi122 ceph-mon[73495]: pgmap v2070: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 1.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 649 op/s; 793/50112 objects degraded (1.582%); 1397/50112 objects misplaced (2.788%); 3.0 MiB/s, 47 objects/s recovering 2024-03-20T09:30:29.845 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:30:29.874 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:30:29.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:29.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:30:29.979 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:30:30.010 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:30.036 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:30:30.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:30.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:30:30.142 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:30:30.190 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:30.761 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:30:31.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:31 smithi122 ceph-mon[73495]: pgmap v2071: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 940 MiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 694 op/s; 793/45117 objects degraded (1.758%); 1397/45117 objects misplaced (3.096%); 2.4 MiB/s, 38 objects/s recovering 2024-03-20T09:30:31.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:31 smithi026 ceph-mon[68309]: pgmap v2071: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 940 MiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 694 op/s; 793/45117 objects degraded (1.758%); 1397/45117 objects misplaced (3.096%); 2.4 MiB/s, 38 objects/s recovering 2024-03-20T09:30:31.782 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:30:31.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:31.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:30:31.893 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:30:31.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:31 smithi115 ceph-mon[72476]: pgmap v2071: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 940 MiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 694 op/s; 793/45117 objects degraded (1.758%); 1397/45117 objects misplaced (3.096%); 2.4 MiB/s, 38 objects/s recovering 2024-03-20T09:30:31.974 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:32.014 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:30:32.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:32.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:30:32.124 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:30:32.151 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:32.170 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:30:32.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:32.173 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:30:32.274 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:30:32.377 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:32.660 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:30:32.663 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:32 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:30:32.663 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:32 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:30:32.663 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:32 smithi026 ceph-mon[68309]: osdmap e855: 12 total, 11 up, 11 in 2024-03-20T09:30:32.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:32.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:30:32.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:32 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:30:32.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:32 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:30:32.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:32 smithi122 ceph-mon[73495]: osdmap e855: 12 total, 11 up, 11 in 2024-03-20T09:30:32.766 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:30:32.793 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:32 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:30:32.793 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:32 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:30:32.793 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:32 smithi115 ceph-mon[72476]: osdmap e855: 12 total, 11 up, 11 in 2024-03-20T09:30:32.795 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:32.814 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:30:32.815 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:30:32.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.815 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPushReply(6.17 856/812 [PushReplyOp(6:ea56b17d:::benchmark_data_smithi122_168655_object108352:head)])", 2024-03-20T09:30:32.815 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:30:32.693042+0000", 2024-03-20T09:30:32.815 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.042590572, 2024-03-20T09:30:32.815 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.042613126000000001, 2024-03-20T09:30:32.815 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:30:32.815 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "queued for pg", 2024-03-20T09:30:32.815 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:30:32.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.815 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:30:32.816 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.693042+0000", 2024-03-20T09:30:32.816 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:30:32.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.816 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:30:32.816 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.693042+0000", 2024-03-20T09:30:32.816 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:30:32.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.816 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:30:32.816 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.693043+0000", 2024-03-20T09:30:32.816 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2900000000000002e-07 2024-03-20T09:30:32.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:30:32.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.693049+0000", 2024-03-20T09:30:32.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3679999999999998e-06 2024-03-20T09:30:32.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:30:32.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.693062+0000", 2024-03-20T09:30:32.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2522e-05 2024-03-20T09:30:32.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:30:32.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.693071+0000", 2024-03-20T09:30:32.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8880000000000003e-06 2024-03-20T09:30:32.818 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:30:32.818 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:30:32.818 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:30:32.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.818 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26216.0:228567 6.17 6:ed016901:::benchmark_data_smithi122_168655_object108458:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e856)", 2024-03-20T09:30:32.818 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:30:32.693908+0000", 2024-03-20T09:30:32.818 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.041724985999999999, 2024-03-20T09:30:32.818 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.041784870000000002, 2024-03-20T09:30:32.818 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:30:32.818 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "waiting for missing object", 2024-03-20T09:30:32.818 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:30:32.819 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26216", 2024-03-20T09:30:32.819 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:30:32.819 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 228567 2024-03-20T09:30:32.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.819 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:30:32.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.819 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:30:32.819 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.693908+0000", 2024-03-20T09:30:32.819 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:30:32.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.820 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:30:32.820 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.693908+0000", 2024-03-20T09:30:32.820 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:30:32.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.820 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:30:32.820 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.693909+0000", 2024-03-20T09:30:32.820 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2559999999999999e-06 2024-03-20T09:30:32.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.821 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:30:32.821 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.693918+0000", 2024-03-20T09:30:32.821 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4180000000000003e-06 2024-03-20T09:30:32.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.821 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:30:32.821 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.693928+0000", 2024-03-20T09:30:32.821 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9010000000000006e-06 2024-03-20T09:30:32.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.821 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:30:32.821 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.693934+0000", 2024-03-20T09:30:32.821 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.48e-06 2024-03-20T09:30:32.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.822 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:30:32.822 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.725205+0000", 2024-03-20T09:30:32.822 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031271404000000003 2024-03-20T09:30:32.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.822 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:30:32.822 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.726304+0000", 2024-03-20T09:30:32.822 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010993419999999999 2024-03-20T09:30:32.822 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:30:32.822 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:30:32.822 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:30:32.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.823 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26216.0:228417 6.17 6:eb4bcf90:::benchmark_data_smithi122_168655_object108308:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e855)", 2024-03-20T09:30:32.823 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:30:32.485355+0000", 2024-03-20T09:30:32.823 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.25027723299999999, 2024-03-20T09:30:32.823 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.25038228299999998, 2024-03-20T09:30:32.823 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:30:32.823 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "waiting for degraded object", 2024-03-20T09:30:32.823 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:30:32.823 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26216", 2024-03-20T09:30:32.823 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:30:32.823 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 228417 2024-03-20T09:30:32.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.823 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:30:32.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.824 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:30:32.824 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.485355+0000", 2024-03-20T09:30:32.824 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:30:32.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.824 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:30:32.824 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.485355+0000", 2024-03-20T09:30:32.824 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:30:32.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.824 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:30:32.824 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.485357+0000", 2024-03-20T09:30:32.824 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.426e-06 2024-03-20T09:30:32.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.825 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:30:32.825 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.485362+0000", 2024-03-20T09:30:32.825 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.592e-06 2024-03-20T09:30:32.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.825 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:30:32.825 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.485387+0000", 2024-03-20T09:30:32.825 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4329e-05 2024-03-20T09:30:32.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.825 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:30:32.825 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.485394+0000", 2024-03-20T09:30:32.825 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0489999999999998e-06 2024-03-20T09:30:32.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.826 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:30:32.826 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.495394+0000", 2024-03-20T09:30:32.826 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010000500000000001 2024-03-20T09:30:32.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.826 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:30:32.826 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.495950+0000", 2024-03-20T09:30:32.826 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00055609099999999999 2024-03-20T09:30:32.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.826 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:30:32.826 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.575420+0000", 2024-03-20T09:30:32.827 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.079469434000000005 2024-03-20T09:30:32.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.827 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for degraded object", 2024-03-20T09:30:32.827 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.576118+0000", 2024-03-20T09:30:32.827 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00069852200000000003 2024-03-20T09:30:32.827 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:30:32.827 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:30:32.827 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:30:32.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.827 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26216.0:228525 6.17 6:ef1287ab:::benchmark_data_smithi122_168655_object108416:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e856)", 2024-03-20T09:30:32.827 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:30:32.627905+0000", 2024-03-20T09:30:32.827 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.10772748, 2024-03-20T09:30:32.828 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.10788194299999999, 2024-03-20T09:30:32.828 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:30:32.828 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "waiting for degraded object", 2024-03-20T09:30:32.828 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:30:32.828 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26216", 2024-03-20T09:30:32.828 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:30:32.828 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 228525 2024-03-20T09:30:32.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.828 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:30:32.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.828 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:30:32.828 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.627905+0000", 2024-03-20T09:30:32.828 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:30:32.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.829 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:30:32.829 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.627905+0000", 2024-03-20T09:30:32.829 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:30:32.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.829 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:30:32.829 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.627907+0000", 2024-03-20T09:30:32.829 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.916e-06 2024-03-20T09:30:32.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.829 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:30:32.829 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.627914+0000", 2024-03-20T09:30:32.829 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0129999999999996e-06 2024-03-20T09:30:32.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.830 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:30:32.830 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.627925+0000", 2024-03-20T09:30:32.830 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1075e-05 2024-03-20T09:30:32.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.830 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:30:32.830 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.627934+0000", 2024-03-20T09:30:32.830 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1840000000000005e-06 2024-03-20T09:30:32.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.830 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:30:32.831 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.641782+0000", 2024-03-20T09:30:32.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013847454 2024-03-20T09:30:32.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.831 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:30:32.831 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.642824+0000", 2024-03-20T09:30:32.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001042563 2024-03-20T09:30:32.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.831 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:30:32.831 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.653915+0000", 2024-03-20T09:30:32.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011090186 2024-03-20T09:30:32.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.831 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for degraded object", 2024-03-20T09:30:32.832 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.654618+0000", 2024-03-20T09:30:32.832 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00070331799999999998 2024-03-20T09:30:32.832 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:30:32.832 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:30:32.832 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:30:32.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.832 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26216.0:228461 6.17 6:ea56b17d:::benchmark_data_smithi122_168655_object108352:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e855)", 2024-03-20T09:30:32.832 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:30:32.543430+0000", 2024-03-20T09:30:32.832 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.19220247800000001, 2024-03-20T09:30:32.832 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.19243157899999999, 2024-03-20T09:30:32.832 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:30:32.832 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "waiting for degraded object", 2024-03-20T09:30:32.832 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:30:32.833 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26216", 2024-03-20T09:30:32.833 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:30:32.833 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 228461 2024-03-20T09:30:32.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.833 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:30:32.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.833 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:30:32.833 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.543430+0000", 2024-03-20T09:30:32.833 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:30:32.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.833 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:30:32.833 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.543430+0000", 2024-03-20T09:30:32.833 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:30:32.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.834 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:30:32.834 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.543432+0000", 2024-03-20T09:30:32.834 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.522e-06 2024-03-20T09:30:32.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.834 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:30:32.834 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.543437+0000", 2024-03-20T09:30:32.834 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4700000000000001e-06 2024-03-20T09:30:32.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.834 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:30:32.834 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.543451+0000", 2024-03-20T09:30:32.835 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3848e-05 2024-03-20T09:30:32.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.835 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:30:32.835 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.543460+0000", 2024-03-20T09:30:32.835 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9609999999999994e-06 2024-03-20T09:30:32.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.835 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:30:32.835 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.560107+0000", 2024-03-20T09:30:32.835 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016647049000000001 2024-03-20T09:30:32.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.835 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:30:32.836 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.560539+0000", 2024-03-20T09:30:32.836 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000432373 2024-03-20T09:30:32.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.836 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:30:32.836 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.567117+0000", 2024-03-20T09:30:32.836 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0065777750000000001 2024-03-20T09:30:32.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.836 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for degraded object", 2024-03-20T09:30:32.836 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.567817+0000", 2024-03-20T09:30:32.836 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00070018000000000001 2024-03-20T09:30:32.836 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:30:32.836 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:30:32.836 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:30:32.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.837 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPushReply(6.17 856/812 [PushReplyOp(6:eb4bcf90:::benchmark_data_smithi122_168655_object108308:head)])", 2024-03-20T09:30:32.837 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:30:32.701846+0000", 2024-03-20T09:30:32.837 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.033786868999999997, 2024-03-20T09:30:32.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.034061910000000001, 2024-03-20T09:30:32.837 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:30:32.837 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "queued for pg", 2024-03-20T09:30:32.837 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:30:32.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.837 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:30:32.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.701846+0000", 2024-03-20T09:30:32.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:30:32.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:30:32.838 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.701846+0000", 2024-03-20T09:30:32.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:30:32.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:30:32.838 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.701847+0000", 2024-03-20T09:30:32.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4900000000000005e-07 2024-03-20T09:30:32.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:30:32.838 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.701852+0000", 2024-03-20T09:30:32.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0189999999999997e-06 2024-03-20T09:30:32.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.839 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:30:32.839 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.701864+0000", 2024-03-20T09:30:32.839 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2665e-05 2024-03-20T09:30:32.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.839 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:30:32.839 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.701874+0000", 2024-03-20T09:30:32.839 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5999999999999996e-06 2024-03-20T09:30:32.839 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:30:32.839 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:30:32.839 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:30:32.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.840 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26216.0:228533 6.17 6:eb95ff47:::benchmark_data_smithi122_168655_object108424:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e856)", 2024-03-20T09:30:32.840 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:30:32.637828+0000", 2024-03-20T09:30:32.840 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.097804123000000007, 2024-03-20T09:30:32.840 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.098108583999999999, 2024-03-20T09:30:32.840 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:30:32.840 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "waiting for degraded object", 2024-03-20T09:30:32.840 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:30:32.840 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26216", 2024-03-20T09:30:32.840 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:30:32.840 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 228533 2024-03-20T09:30:32.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.840 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:30:32.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.841 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:30:32.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.637828+0000", 2024-03-20T09:30:32.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:30:32.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.841 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:30:32.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.637828+0000", 2024-03-20T09:30:32.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:30:32.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.841 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:30:32.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.637829+0000", 2024-03-20T09:30:32.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.6700000000000002e-07 2024-03-20T09:30:32.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.842 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:30:32.842 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.637838+0000", 2024-03-20T09:30:32.842 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1170000000000004e-06 2024-03-20T09:30:32.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.842 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:30:32.842 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.637852+0000", 2024-03-20T09:30:32.842 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4501000000000001e-05 2024-03-20T09:30:32.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.842 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:30:32.842 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.637861+0000", 2024-03-20T09:30:32.842 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4309999999999994e-06 2024-03-20T09:30:32.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:30:32.843 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.654757+0000", 2024-03-20T09:30:32.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016896926 2024-03-20T09:30:32.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:30:32.843 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.656074+0000", 2024-03-20T09:30:32.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00131642 2024-03-20T09:30:32.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:30:32.843 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.679760+0000", 2024-03-20T09:30:32.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023685713000000001 2024-03-20T09:30:32.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.844 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for degraded object", 2024-03-20T09:30:32.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.680435+0000", 2024-03-20T09:30:32.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00067554500000000005 2024-03-20T09:30:32.844 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:30:32.844 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:30:32.844 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:30:32.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.844 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPushReply(6.17 856/812 [PushReplyOp(6:eb4bcf90:::benchmark_data_smithi122_168655_object108308:head)])", 2024-03-20T09:30:32.844 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:30:32.712518+0000", 2024-03-20T09:30:32.844 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.023114732999999998, 2024-03-20T09:30:32.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023482696000000001, 2024-03-20T09:30:32.844 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:30:32.844 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "queued for pg", 2024-03-20T09:30:32.844 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:30:32.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.845 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:30:32.845 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.712518+0000", 2024-03-20T09:30:32.845 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:30:32.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.845 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:30:32.845 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.712518+0000", 2024-03-20T09:30:32.845 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:30:32.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.845 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:30:32.845 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.712518+0000", 2024-03-20T09:30:32.845 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.06e-07 2024-03-20T09:30:32.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:30:32.846 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.712525+0000", 2024-03-20T09:30:32.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2570000000000004e-06 2024-03-20T09:30:32.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:30:32.846 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.712535+0000", 2024-03-20T09:30:32.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0319999999999999e-05 2024-03-20T09:30:32.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:30:32.846 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.712542+0000", 2024-03-20T09:30:32.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5479999999999998e-06 2024-03-20T09:30:32.846 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:30:32.847 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:30:32.847 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:30:32.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.847 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26216.0:228583 6.14 6:2f9c1e20:::benchmark_data_smithi122_168655_object108474:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e856)", 2024-03-20T09:30:32.847 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:30:32.724715+0000", 2024-03-20T09:30:32.847 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.010917292, 2024-03-20T09:30:32.847 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011315255, 2024-03-20T09:30:32.847 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:30:32.847 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:30:32.847 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:30:32.847 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26216", 2024-03-20T09:30:32.847 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:30:32.847 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 228583 2024-03-20T09:30:32.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.848 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:30:32.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.848 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:30:32.848 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.724715+0000", 2024-03-20T09:30:32.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:30:32.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.848 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:30:32.848 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.724715+0000", 2024-03-20T09:30:32.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:30:32.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.848 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:30:32.849 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.724716+0000", 2024-03-20T09:30:32.849 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1620000000000001e-06 2024-03-20T09:30:32.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.849 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:30:32.849 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.724722+0000", 2024-03-20T09:30:32.849 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0629999999999996e-06 2024-03-20T09:30:32.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.849 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:30:32.849 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.724730+0000", 2024-03-20T09:30:32.849 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8230000000000001e-06 2024-03-20T09:30:32.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:30:32.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.724736+0000", 2024-03-20T09:30:32.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6729999999999997e-06 2024-03-20T09:30:32.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:30:32.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.724792+0000", 2024-03-20T09:30:32.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6397000000000002e-05 2024-03-20T09:30:32.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:30:32.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.724961+0000", 2024-03-20T09:30:32.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000168178 2024-03-20T09:30:32.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4,7", 2024-03-20T09:30:32.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.725491+0000", 2024-03-20T09:30:32.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000530731 2024-03-20T09:30:32.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:30:32.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.734386+0000", 2024-03-20T09:30:32.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0088946540000000001 2024-03-20T09:30:32.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:30:32.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.734424+0000", 2024-03-20T09:30:32.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7897e-05 2024-03-20T09:30:32.852 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:30:32.852 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:30:32.852 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:30:32.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.852 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26216.0:228491 6.17 6:ef0f0d71:::benchmark_data_smithi122_168655_object108382:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e856)", 2024-03-20T09:30:32.852 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:30:32.587160+0000", 2024-03-20T09:30:32.852 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.14847226999999999, 2024-03-20T09:30:32.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.148920517, 2024-03-20T09:30:32.852 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:30:32.852 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "waiting for degraded object", 2024-03-20T09:30:32.852 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:30:32.852 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26216", 2024-03-20T09:30:32.853 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:30:32.853 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 228491 2024-03-20T09:30:32.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.853 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:30:32.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.853 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:30:32.853 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.587160+0000", 2024-03-20T09:30:32.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:30:32.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.853 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:30:32.853 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.587160+0000", 2024-03-20T09:30:32.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:30:32.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.854 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:30:32.854 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.587162+0000", 2024-03-20T09:30:32.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4619999999999999e-06 2024-03-20T09:30:32.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.854 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:30:32.854 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.587169+0000", 2024-03-20T09:30:32.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9800000000000001e-06 2024-03-20T09:30:32.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.854 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:30:32.854 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.587180+0000", 2024-03-20T09:30:32.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1698999999999999e-05 2024-03-20T09:30:32.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:30:32.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.587190+0000", 2024-03-20T09:30:32.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1200000000000008e-06 2024-03-20T09:30:32.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:30:32.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.603096+0000", 2024-03-20T09:30:32.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015906525000000001 2024-03-20T09:30:32.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for missing object", 2024-03-20T09:30:32.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.604119+0000", 2024-03-20T09:30:32.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001022926 2024-03-20T09:30:32.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:30:32.856 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.625437+0000", 2024-03-20T09:30:32.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.021317465000000001 2024-03-20T09:30:32.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for degraded object", 2024-03-20T09:30:32.856 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.626156+0000", 2024-03-20T09:30:32.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00071919400000000004 2024-03-20T09:30:32.856 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:30:32.856 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:30:32.856 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:30:32.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.857 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26216.0:228585 6.19 e856/829 6:9a6fe685:::benchmark_data_smithi122_168655_object108476:head v 856'11041, mlcod=856'11039)", 2024-03-20T09:30:32.857 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:30:32.726105+0000", 2024-03-20T09:30:32.857 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0095280090000000005, 2024-03-20T09:30:32.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010024767, 2024-03-20T09:30:32.857 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:30:32.857 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:30:32.857 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:30:32.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.857 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:30:32.857 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.726105+0000", 2024-03-20T09:30:32.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:30:32.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.857 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:30:32.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.726105+0000", 2024-03-20T09:30:32.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:30:32.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.858 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:30:32.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.726106+0000", 2024-03-20T09:30:32.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.195e-06 2024-03-20T09:30:32.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.858 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:30:32.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.726115+0000", 2024-03-20T09:30:32.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7930000000000003e-06 2024-03-20T09:30:32.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:30:32.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.726127+0000", 2024-03-20T09:30:32.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.221e-05 2024-03-20T09:30:32.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:30:32.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.726135+0000", 2024-03-20T09:30:32.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9459999999999998e-06 2024-03-20T09:30:32.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:30:32.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.726216+0000", 2024-03-20T09:30:32.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1342999999999997e-05 2024-03-20T09:30:32.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.860 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:30:32.860 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.726240+0000", 2024-03-20T09:30:32.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4360000000000001e-05 2024-03-20T09:30:32.860 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:30:32.860 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:30:32.860 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:30:32.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.860 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(6.17 856/812 [PushOp(6:ed016901:::benchmark_data_smithi122_168655_object108458:head, version: 743'6943, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:ed016901:::benchmark_data_smithi122_168655_object108458:head@743'6943, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-20T09:30:32.860 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:30:32.726893+0000", 2024-03-20T09:30:32.860 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0087399399999999999, 2024-03-20T09:30:32.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0092896160000000005, 2024-03-20T09:30:32.860 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:30:32.861 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:30:32.861 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:30:32.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.861 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:30:32.861 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.726893+0000", 2024-03-20T09:30:32.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:30:32.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.861 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:30:32.861 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.726893+0000", 2024-03-20T09:30:32.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:30:32.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.861 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:30:32.861 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.726893+0000", 2024-03-20T09:30:32.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8599999999999998e-07 2024-03-20T09:30:32.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.862 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:30:32.862 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.726959+0000", 2024-03-20T09:30:32.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5807000000000002e-05 2024-03-20T09:30:32.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.862 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:30:32.862 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.726973+0000", 2024-03-20T09:30:32.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3804e-05 2024-03-20T09:30:32.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.862 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:30:32.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.726983+0000", 2024-03-20T09:30:32.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0037000000000001e-05 2024-03-20T09:30:32.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:30:32.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.733660+0000", 2024-03-20T09:30:32.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0066774449999999997 2024-03-20T09:30:32.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:30:32.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:30:32.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:30:32.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:32.733774+0000", 2024-03-20T09:30:32.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011349 2024-03-20T09:30:32.863 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:30:32.863 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:30:32.863 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:30:32.864 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:30:32.864 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:30:32.864 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 12 2024-03-20T09:30:32.864 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:30:32.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:32.865 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:30:32.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:30:32.966 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:30:32.998 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:33.017 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:30:33.021 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:30:33.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:33.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:30:33.128 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:30:33.156 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:33.178 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:30:33.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:33.183 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:30:33.284 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:30:33.322 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:33.350 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:30:33.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:33.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:30:33.429 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:33 smithi026 ceph-mon[68309]: pgmap v2073: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 848 MiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 729 op/s; 793/40665 objects degraded (1.950%); 1397/40665 objects misplaced (3.435%); 2.5 MiB/s, 39 objects/s recovering 2024-03-20T09:30:33.429 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:33 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 793/40665 objects degraded (1.950%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:30:33.429 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:33 smithi026 ceph-mon[68309]: osdmap e856: 12 total, 11 up, 11 in 2024-03-20T09:30:33.429 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:33 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1590865699' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:30:33.429 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:33 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:30:33.455 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:30:33.506 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:33.528 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:30:33.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:33.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:30:33.626 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:30:33.635 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:30:33.639 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-20T09:30:33.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:33 smithi115 ceph-mon[72476]: pgmap v2073: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 848 MiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 729 op/s; 793/40665 objects degraded (1.950%); 1397/40665 objects misplaced (3.435%); 2.5 MiB/s, 39 objects/s recovering 2024-03-20T09:30:33.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:33 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 793/40665 objects degraded (1.950%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:30:33.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:33 smithi115 ceph-mon[72476]: osdmap e856: 12 total, 11 up, 11 in 2024-03-20T09:30:33.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:33 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1590865699' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:30:33.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:33 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:30:33.662 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:33.683 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:30:33.685 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:33 smithi122 ceph-mon[73495]: pgmap v2073: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 848 MiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 729 op/s; 793/40665 objects degraded (1.950%); 1397/40665 objects misplaced (3.435%); 2.5 MiB/s, 39 objects/s recovering 2024-03-20T09:30:33.685 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:33 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 793/40665 objects degraded (1.950%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:30:33.685 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:33 smithi122 ceph-mon[73495]: osdmap e856: 12 total, 11 up, 11 in 2024-03-20T09:30:33.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:33 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1590865699' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:30:33.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:33 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:30:33.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:33.687 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:30:33.788 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:30:33.915 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:34.026 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:30:34.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:34.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:30:34.148 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:30:34.267 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:34.430 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:34 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:30:34.431 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:34 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:30:34.431 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:34 smithi026 ceph-mon[68309]: osdmap e857: 12 total, 11 up, 11 in 2024-03-20T09:30:34.431 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:34 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:30:34.431 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:34 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:30:34.431 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:34 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:30:34.431 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:34 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:30:34.431 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:34 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:30:34.431 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:34 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:30:34.532 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 8, 11, 3, 4, 9] out_osds: [10] dead_osds: [10] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 11, 5] 2024-03-20T09:30:34.533 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:30:34.533 INFO:tasks.thrashosds.thrasher:Reviving osd 10 2024-03-20T09:30:34.533 INFO:tasks.cephadm.osd.10:Restarting osd.10 (starting--it wasn't running)... 2024-03-20T09:30:34.533 DEBUG:teuthology.orchestra.run.smithi122:osd.10> sudo journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.10.service 2024-03-20T09:30:34.535 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl start ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.10 2024-03-20T09:30:34.566 INFO:journalctl@ceph.osd.10.smithi122.stdout:-- Logs begin at Wed 2024-03-20 08:37:02 UTC. -- 2024-03-20T09:30:34.689 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:30:34 smithi122 systemd[1]: Starting Ceph osd.10 for 852fbb68-e696-11ee-95c9-87774f69a715... 2024-03-20T09:30:34.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:34 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:30:34.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:34 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:30:34.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:34 smithi122 ceph-mon[73495]: osdmap e857: 12 total, 11 up, 11 in 2024-03-20T09:30:34.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:34 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:30:34.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:34 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:30:34.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:34 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:30:34.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:34 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:30:34.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:34 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:30:34.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:34 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:30:34.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:34 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:30:34.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:34 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:30:34.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:34 smithi115 ceph-mon[72476]: osdmap e857: 12 total, 11 up, 11 in 2024-03-20T09:30:34.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:34 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:30:34.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:34 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:30:34.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:34 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:30:34.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:34 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:30:34.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:34 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:30:34.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:34 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:30:35.681 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:30:35.686 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:30:35.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:35 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:30:35.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:35 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:30:35.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:35 smithi122 ceph-mon[73495]: 1.1 scrub starts 2024-03-20T09:30:35.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:35 smithi122 ceph-mon[73495]: 1.1 scrub ok 2024-03-20T09:30:35.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:35 smithi122 ceph-mon[73495]: pgmap v2076: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 811 MiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 621 op/s; 571/38916 objects degraded (1.467%); 1022/38916 objects misplaced (2.626%); 1.7 MiB/s, 27 objects/s recovering 2024-03-20T09:30:35.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:35 smithi122 ceph-mon[73495]: osdmap e858: 12 total, 11 up, 11 in 2024-03-20T09:30:35.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:35.703 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:35 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:30:35.703 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:35 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:30:35.703 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:35 smithi026 ceph-mon[68309]: 1.1 scrub starts 2024-03-20T09:30:35.703 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:35 smithi026 ceph-mon[68309]: 1.1 scrub ok 2024-03-20T09:30:35.704 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:35 smithi026 ceph-mon[68309]: pgmap v2076: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 811 MiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 621 op/s; 571/38916 objects degraded (1.467%); 1022/38916 objects misplaced (2.626%); 1.7 MiB/s, 27 objects/s recovering 2024-03-20T09:30:35.704 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:35 smithi026 ceph-mon[68309]: osdmap e858: 12 total, 11 up, 11 in 2024-03-20T09:30:35.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:30:35.805 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:30:35.875 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:35.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:35 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:30:35.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:35 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:30:35.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:35 smithi115 ceph-mon[72476]: 1.1 scrub starts 2024-03-20T09:30:35.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:35 smithi115 ceph-mon[72476]: 1.1 scrub ok 2024-03-20T09:30:35.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:35 smithi115 ceph-mon[72476]: pgmap v2076: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 811 MiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 621 op/s; 571/38916 objects degraded (1.467%); 1022/38916 objects misplaced (2.626%); 1.7 MiB/s, 27 objects/s recovering 2024-03-20T09:30:35.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:35 smithi115 ceph-mon[72476]: osdmap e858: 12 total, 11 up, 11 in 2024-03-20T09:30:36.070 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:30:36.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:36.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:30:36.181 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:30:36.215 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:36.245 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:30:36.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:36.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:30:36.352 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:30:36.385 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:36.418 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:30:36.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:36.422 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:30:36.523 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:30:36.596 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:36.840 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:36 smithi026 ceph-mon[68309]: 1.16 scrub starts 2024-03-20T09:30:36.840 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:36 smithi026 ceph-mon[68309]: 1.16 scrub ok 2024-03-20T09:30:36.840 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:36 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:30:36.840 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:36 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:30:36.840 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:36 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:30:36.840 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:36 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:30:36.840 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:36 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:30:36.841 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:36 smithi026 ceph-mon[68309]: 6.5 scrub starts 2024-03-20T09:30:36.841 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:36 smithi026 ceph-mon[68309]: 6.5 scrub ok 2024-03-20T09:30:36.841 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:36 smithi026 ceph-mon[68309]: pgmap v2078: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 665 MiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 640 op/s; 571/31914 objects degraded (1.789%); 1022/31914 objects misplaced (3.202%); 2.1 MiB/s, 33 objects/s recovering 2024-03-20T09:30:36.841 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:36 smithi026 ceph-mon[68309]: 1.9 scrub starts 2024-03-20T09:30:36.841 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:36 smithi026 ceph-mon[68309]: 1.9 scrub ok 2024-03-20T09:30:36.939 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:30:36 smithi122 bash[184260]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-10 2024-03-20T09:30:36.939 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:30:36 smithi122 bash[184260]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-10 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-20T09:30:36.939 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:30:36 smithi122 bash[184260]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-20T09:30:36.939 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:30:36 smithi122 bash[184260]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-20T09:30:36.940 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:30:36 smithi122 bash[184260]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-10/block 2024-03-20T09:30:36.940 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:30:36 smithi122 bash[184260]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-10 2024-03-20T09:30:36.940 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:30:36 smithi122 bash[184260]: --> ceph-volume raw activate successful for osd ID: 10 2024-03-20T09:30:36.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:36 smithi122 ceph-mon[73495]: 1.16 scrub starts 2024-03-20T09:30:36.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:36 smithi122 ceph-mon[73495]: 1.16 scrub ok 2024-03-20T09:30:36.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:36 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:30:36.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:36 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:30:36.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:36 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:30:36.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:36 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:30:36.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:36 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:30:36.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:36 smithi122 ceph-mon[73495]: 6.5 scrub starts 2024-03-20T09:30:36.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:36 smithi122 ceph-mon[73495]: 6.5 scrub ok 2024-03-20T09:30:36.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:36 smithi122 ceph-mon[73495]: pgmap v2078: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 665 MiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 640 op/s; 571/31914 objects degraded (1.789%); 1022/31914 objects misplaced (3.202%); 2.1 MiB/s, 33 objects/s recovering 2024-03-20T09:30:36.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:36 smithi122 ceph-mon[73495]: 1.9 scrub starts 2024-03-20T09:30:36.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:36 smithi122 ceph-mon[73495]: 1.9 scrub ok 2024-03-20T09:30:37.153 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:36 smithi115 ceph-mon[72476]: 1.16 scrub starts 2024-03-20T09:30:37.154 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:36 smithi115 ceph-mon[72476]: 1.16 scrub ok 2024-03-20T09:30:37.154 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:36 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:30:37.154 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:36 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:30:37.154 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:36 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:30:37.154 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:36 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:30:37.154 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:36 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:30:37.154 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:36 smithi115 ceph-mon[72476]: 6.5 scrub starts 2024-03-20T09:30:37.154 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:36 smithi115 ceph-mon[72476]: 6.5 scrub ok 2024-03-20T09:30:37.154 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:36 smithi115 ceph-mon[72476]: pgmap v2078: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 665 MiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 640 op/s; 571/31914 objects degraded (1.789%); 1022/31914 objects misplaced (3.202%); 2.1 MiB/s, 33 objects/s recovering 2024-03-20T09:30:37.154 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:36 smithi115 ceph-mon[72476]: 1.9 scrub starts 2024-03-20T09:30:37.154 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:36 smithi115 ceph-mon[72476]: 1.9 scrub ok 2024-03-20T09:30:37.198 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:30:37.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:37.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:30:37.307 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:30:37.338 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:37.370 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:30:37.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:37.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:30:37.395 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:30:37.475 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:30:37.563 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:37.627 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:30:37.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:37.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:30:37.736 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:30:37.766 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:37.799 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:30:37.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:37.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:30:37.906 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:30:37.937 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:38.056 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:37 smithi026 ceph-mon[68309]: 1.1d scrub starts 2024-03-20T09:30:38.056 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:37 smithi026 ceph-mon[68309]: 1.1d scrub ok 2024-03-20T09:30:38.056 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:37 smithi026 ceph-mon[68309]: 6.15 scrub starts 2024-03-20T09:30:38.056 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:37 smithi026 ceph-mon[68309]: 6.15 scrub ok 2024-03-20T09:30:38.056 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:37 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 571/31914 objects degraded (1.789%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:30:38.067 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:30:38.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:38.070 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:30:38.149 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:37 smithi115 ceph-mon[72476]: 1.1d scrub starts 2024-03-20T09:30:38.149 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:37 smithi115 ceph-mon[72476]: 1.1d scrub ok 2024-03-20T09:30:38.149 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:37 smithi115 ceph-mon[72476]: 6.15 scrub starts 2024-03-20T09:30:38.149 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:37 smithi115 ceph-mon[72476]: 6.15 scrub ok 2024-03-20T09:30:38.149 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:37 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 571/31914 objects degraded (1.789%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:30:38.171 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:30:38.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:37 smithi122 ceph-mon[73495]: 1.1d scrub starts 2024-03-20T09:30:38.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:37 smithi122 ceph-mon[73495]: 1.1d scrub ok 2024-03-20T09:30:38.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:37 smithi122 ceph-mon[73495]: 6.15 scrub starts 2024-03-20T09:30:38.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:37 smithi122 ceph-mon[73495]: 6.15 scrub ok 2024-03-20T09:30:38.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:37 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 571/31914 objects degraded (1.789%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:30:38.258 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:38.315 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_ops_in_flight 2024-03-20T09:30:38.594 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:30:38.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:38.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:30:38.607 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:30:38.689 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:30:38 smithi122 ceph-osd[184651]: -- 172.21.15.122:0/874888533 <== mon.2 v2:172.21.15.115:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d282265680 con 0x55d28222ec00 2024-03-20T09:30:38.689 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:30:38 smithi122 bash[184581]: 07a45c912bab570db98c9e446e165dafa6d6bf91c53adb0d1c5f9d975df9394c 2024-03-20T09:30:38.690 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:30:38 smithi122 systemd[1]: Started Ceph osd.10 for 852fbb68-e696-11ee-95c9-87774f69a715. 2024-03-20T09:30:38.704 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:30:38.733 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:38.939 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:38 smithi026 ceph-mon[68309]: pgmap v2079: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 576 MiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 723 op/s; 208/27624 objects degraded (0.753%); 671/27624 objects misplaced (2.429%); 4.9 MiB/s, 79 objects/s recovering 2024-03-20T09:30:39.042 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:30:39.056 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:30:39.117 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:30:39.117 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:30:39.117 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-03-20T09:30:39.117 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-03-20T09:30:39.117 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:30:39.118 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:30:39.171 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:30:39.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:38 smithi122 ceph-mon[73495]: pgmap v2079: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 576 MiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 723 op/s; 208/27624 objects degraded (0.753%); 671/27624 objects misplaced (2.429%); 4.9 MiB/s, 79 objects/s recovering 2024-03-20T09:30:39.189 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:30:38 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:30:38.763+0000 7fa938f337c0 -1 Falling back to public interface 2024-03-20T09:30:39.233 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:30:39.278 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:30:39.288 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:30:39.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:39.291 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:30:39.331 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:30:39.391 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:30:39.392 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:30:39.398 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:38 smithi115 ceph-mon[72476]: pgmap v2079: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 576 MiB data, 8.8 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 723 op/s; 208/27624 objects degraded (0.753%); 671/27624 objects misplaced (2.429%); 4.9 MiB/s, 79 objects/s recovering 2024-03-20T09:30:39.429 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:39.437 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:30:39.455 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:30:39.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:39.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:30:39.485 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:30:39.515 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:30:39.548 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:30:39.548 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:30:39.548 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:30:39.548 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:30:39.548 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:30:39.548 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:30:39.548 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:30:39.548 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:30:39.548 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:30:39.548 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:30:39.549 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:30:39.549 INFO:teuthology.orchestra.run.smithi026.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-03-20T09:30:39.549 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:30:39.559 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:30:39.591 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:39.627 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:30:39.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:39.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:30:39.732 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:30:39.815 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:39.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:39 smithi026 ceph-mon[68309]: 6.4 scrub starts 2024-03-20T09:30:39.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:39 smithi026 ceph-mon[68309]: 6.4 scrub ok 2024-03-20T09:30:39.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:39 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/542524203' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:30:39.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:39 smithi026 ceph-mon[68309]: 6.1 scrub starts 2024-03-20T09:30:39.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:39 smithi026 ceph-mon[68309]: 6.1 scrub ok 2024-03-20T09:30:39.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:39 smithi026 ceph-mon[68309]: 1.6 scrub starts 2024-03-20T09:30:39.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:39 smithi026 ceph-mon[68309]: 1.6 scrub ok 2024-03-20T09:30:40.291 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:39 smithi122 ceph-mon[73495]: 6.4 scrub starts 2024-03-20T09:30:40.291 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:39 smithi122 ceph-mon[73495]: 6.4 scrub ok 2024-03-20T09:30:40.291 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:39 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/542524203' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:30:40.291 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:39 smithi122 ceph-mon[73495]: 6.1 scrub starts 2024-03-20T09:30:40.291 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:39 smithi122 ceph-mon[73495]: 6.1 scrub ok 2024-03-20T09:30:40.291 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:39 smithi122 ceph-mon[73495]: 1.6 scrub starts 2024-03-20T09:30:40.291 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:39 smithi122 ceph-mon[73495]: 1.6 scrub ok 2024-03-20T09:30:40.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:39 smithi115 ceph-mon[72476]: 6.4 scrub starts 2024-03-20T09:30:40.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:39 smithi115 ceph-mon[72476]: 6.4 scrub ok 2024-03-20T09:30:40.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:39 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/542524203' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:30:40.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:39 smithi115 ceph-mon[72476]: 6.1 scrub starts 2024-03-20T09:30:40.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:39 smithi115 ceph-mon[72476]: 6.1 scrub ok 2024-03-20T09:30:40.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:39 smithi115 ceph-mon[72476]: 1.6 scrub starts 2024-03-20T09:30:40.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:39 smithi115 ceph-mon[72476]: 1.6 scrub ok 2024-03-20T09:30:40.991 INFO:teuthology.orchestra.run.smithi122.stderr:no valid command found; 10 closest matches: 2024-03-20T09:30:40.992 INFO:teuthology.orchestra.run.smithi122.stderr:0 2024-03-20T09:30:40.992 INFO:teuthology.orchestra.run.smithi122.stderr:1 2024-03-20T09:30:40.992 INFO:teuthology.orchestra.run.smithi122.stderr:2 2024-03-20T09:30:40.992 INFO:teuthology.orchestra.run.smithi122.stderr:abort 2024-03-20T09:30:40.992 INFO:teuthology.orchestra.run.smithi122.stderr:assert 2024-03-20T09:30:40.992 INFO:teuthology.orchestra.run.smithi122.stderr:config diff 2024-03-20T09:30:40.992 INFO:teuthology.orchestra.run.smithi122.stderr:config diff get 2024-03-20T09:30:40.992 INFO:teuthology.orchestra.run.smithi122.stderr:config get 2024-03-20T09:30:40.992 INFO:teuthology.orchestra.run.smithi122.stderr:config help [] 2024-03-20T09:30:40.992 INFO:teuthology.orchestra.run.smithi122.stderr:config set ... 2024-03-20T09:30:40.992 INFO:teuthology.orchestra.run.smithi122.stderr:admin_socket: invalid command 2024-03-20T09:30:41.368 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:40 smithi122 ceph-mon[73495]: 6.13 scrub starts 2024-03-20T09:30:41.368 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:40 smithi122 ceph-mon[73495]: 6.13 scrub ok 2024-03-20T09:30:41.368 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:40 smithi122 ceph-mon[73495]: pgmap v2080: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 468 MiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 781 op/s; 208/22446 objects degraded (0.927%); 671/22446 objects misplaced (2.989%); 3.8 MiB/s, 61 objects/s recovering 2024-03-20T09:30:41.398 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:40 smithi115 ceph-mon[72476]: 6.13 scrub starts 2024-03-20T09:30:41.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:40 smithi115 ceph-mon[72476]: 6.13 scrub ok 2024-03-20T09:30:41.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:40 smithi115 ceph-mon[72476]: pgmap v2080: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 468 MiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 781 op/s; 208/22446 objects degraded (0.927%); 671/22446 objects misplaced (2.989%); 3.8 MiB/s, 61 objects/s recovering 2024-03-20T09:30:41.465 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:40 smithi026 ceph-mon[68309]: 6.13 scrub starts 2024-03-20T09:30:41.465 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:40 smithi026 ceph-mon[68309]: 6.13 scrub ok 2024-03-20T09:30:41.465 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:40 smithi026 ceph-mon[68309]: pgmap v2080: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 468 MiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 781 op/s; 208/22446 objects degraded (0.927%); 671/22446 objects misplaced (2.989%); 3.8 MiB/s, 61 objects/s recovering 2024-03-20T09:30:42.107 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:30:42.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:42.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:30:42.131 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T09:30:42.132 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-10, ['dump_ops_in_flight'] 2024-03-20T09:30:42.213 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:30:42.247 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:42.269 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:30:42.272 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:42 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:30:42.272 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:42 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:30:42.272 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:30:42 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:30:42.168+0000 7fa938f337c0 -1 osd.10 753 log_to_monitors true 2024-03-20T09:30:42.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:42.273 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:30:42.374 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:30:42.398 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:42 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:30:42.398 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:42 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:30:42.422 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:42 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:30:42.423 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:42 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:30:42.425 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:43.323 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:30:43.337 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:30:43.337 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:30:43.337 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: osdmap e859: 12 total, 11 up, 11 in 2024-03-20T09:30:43.337 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: from='osd.10 [v2:172.21.15.122:6816/2853088860,v1:172.21.15.122:6817/2853088860]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-20T09:30:43.337 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-20T09:30:43.337 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: pgmap v2082: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 368 MiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 885 op/s; 208/17652 objects degraded (1.178%); 671/17652 objects misplaced (3.801%); 2.4 MiB/s, 38 objects/s recovering 2024-03-20T09:30:43.337 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 208/22446 objects degraded (0.927%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:30:43.337 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-20T09:30:43.337 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: from='osd.10 [v2:172.21.15.122:6816/2853088860,v1:172.21.15.122:6817/2853088860]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-20T09:30:43.337 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: osdmap e860: 12 total, 11 up, 11 in 2024-03-20T09:30:43.337 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-20T09:30:43.338 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-20T09:30:43.338 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=57,logm=3} crc {auth=414148163,config=3599132250,health=34561630,kv=3379067375,logm=4044773816}) 2024-03-20T09:30:43.338 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2202345666}) 2024-03-20T09:30:43.338 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2111144228}) 2024-03-20T09:30:43.338 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3339523933}) 2024-03-20T09:30:43.338 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3209217667}) 2024-03-20T09:30:43.338 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1882255966}) 2024-03-20T09:30:43.338 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3783532648}) 2024-03-20T09:30:43.338 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=333656019}) 2024-03-20T09:30:43.338 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4072959271}) 2024-03-20T09:30:43.338 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3960040342}) 2024-03-20T09:30:43.338 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1927387634}) 2024-03-20T09:30:43.338 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1485768252}) 2024-03-20T09:30:43.339 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1650980092}) 2024-03-20T09:30:43.339 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2860176741}) 2024-03-20T09:30:43.339 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3228993712}) 2024-03-20T09:30:43.339 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2600731019}) 2024-03-20T09:30:43.339 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=90781852}) 2024-03-20T09:30:43.339 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1236941211}) 2024-03-20T09:30:43.339 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1462225100}) 2024-03-20T09:30:43.339 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2046090448}) 2024-03-20T09:30:43.339 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3280135892}) 2024-03-20T09:30:43.339 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2845478018}) 2024-03-20T09:30:43.339 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1723469209}) 2024-03-20T09:30:43.339 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2098411770}) 2024-03-20T09:30:43.339 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1479321126}) 2024-03-20T09:30:43.340 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4118457232}) 2024-03-20T09:30:43.340 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3642669505}) 2024-03-20T09:30:43.340 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2292260130}) 2024-03-20T09:30:43.340 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2812129074}) 2024-03-20T09:30:43.340 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2759718205}) 2024-03-20T09:30:43.340 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2635213625}) 2024-03-20T09:30:43.340 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=995107027}) 2024-03-20T09:30:43.340 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1528728856}) 2024-03-20T09:30:43.340 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3716164204}) 2024-03-20T09:30:43.340 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=265245435}) 2024-03-20T09:30:43.340 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2824756537}) 2024-03-20T09:30:43.340 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=661496145}) 2024-03-20T09:30:43.340 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4001725031}) 2024-03-20T09:30:43.341 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4047347226}) 2024-03-20T09:30:43.341 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=336620220}) 2024-03-20T09:30:43.341 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3600941964}) 2024-03-20T09:30:43.341 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1688412983}) 2024-03-20T09:30:43.341 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=805061760}) 2024-03-20T09:30:43.341 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=456454241}) 2024-03-20T09:30:43.341 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1244054033}) 2024-03-20T09:30:43.341 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3140656603}) 2024-03-20T09:30:43.341 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2831682078}) 2024-03-20T09:30:43.341 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=651041489}) 2024-03-20T09:30:43.341 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3290203085}) 2024-03-20T09:30:43.341 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4233591458}) 2024-03-20T09:30:43.341 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2227820846}) 2024-03-20T09:30:43.341 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3922319328}) 2024-03-20T09:30:43.341 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2520275631}) 2024-03-20T09:30:43.342 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3630040097}) 2024-03-20T09:30:43.342 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2723266549}) 2024-03-20T09:30:43.342 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=695585522}) 2024-03-20T09:30:43.342 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2730357871}) 2024-03-20T09:30:43.342 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2741530447}) 2024-03-20T09:30:43.342 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2281510726}) 2024-03-20T09:30:43.342 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1331343500}) 2024-03-20T09:30:43.342 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3516503145}) 2024-03-20T09:30:43.342 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1802394395}) 2024-03-20T09:30:43.342 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=982934604}) 2024-03-20T09:30:43.342 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=628551956}) 2024-03-20T09:30:43.342 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2618867919}) 2024-03-20T09:30:43.342 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2334738014}) 2024-03-20T09:30:43.343 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=597140261}) 2024-03-20T09:30:43.343 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3146104049}) 2024-03-20T09:30:43.343 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1457794569}) 2024-03-20T09:30:43.343 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2181943042}) 2024-03-20T09:30:43.343 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2408446137}) 2024-03-20T09:30:43.343 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2835131343}) 2024-03-20T09:30:43.343 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=77,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=1} crc {logm=3777504671,mdsmap=3984425268,mgr=427508465,mgr_command_descs=3831600018,mgr_metadata=2112171679}) 2024-03-20T09:30:43.343 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {mgr_metadata=1,mgrstat=10,mon_config_key=89} crc {mgr_metadata=2912525676,mgrstat=2388164258,mon_config_key=3553608912}) 2024-03-20T09:30:43.343 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2957981141}) 2024-03-20T09:30:43.343 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=89,monmap=5,osd_metadata=6} crc {mon_config_key=3274690714,monmap=3481182261,osd_metadata=1296877480}) 2024-03-20T09:30:43.343 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=6,osd_pg_creating=1,osdmap=93} crc {osd_metadata=2273555789,osd_pg_creating=332160715,osdmap=3340407400}) 2024-03-20T09:30:43.343 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1944444918}) 2024-03-20T09:30:43.343 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=64} crc {osdmap=2429811463}) 2024-03-20T09:30:43.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:43.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:30:43.371 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-20T09:30:43.371 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-20T09:30:43.401 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:30:43.401 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:30:43.402 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: osdmap e859: 12 total, 11 up, 11 in 2024-03-20T09:30:43.402 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: from='osd.10 [v2:172.21.15.122:6816/2853088860,v1:172.21.15.122:6817/2853088860]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-20T09:30:43.402 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-20T09:30:43.402 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: pgmap v2082: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 368 MiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 885 op/s; 208/17652 objects degraded (1.178%); 671/17652 objects misplaced (3.801%); 2.4 MiB/s, 38 objects/s recovering 2024-03-20T09:30:43.402 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 208/22446 objects degraded (0.927%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:30:43.402 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-20T09:30:43.402 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: from='osd.10 [v2:172.21.15.122:6816/2853088860,v1:172.21.15.122:6817/2853088860]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-20T09:30:43.402 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: osdmap e860: 12 total, 11 up, 11 in 2024-03-20T09:30:43.402 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-20T09:30:43.402 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-20T09:30:43.402 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=57,logm=3} crc {auth=414148163,config=3599132250,health=34561630,kv=3379067375,logm=4044773816}) 2024-03-20T09:30:43.402 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2202345666}) 2024-03-20T09:30:43.402 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2111144228}) 2024-03-20T09:30:43.403 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3339523933}) 2024-03-20T09:30:43.403 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3209217667}) 2024-03-20T09:30:43.403 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1882255966}) 2024-03-20T09:30:43.403 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3783532648}) 2024-03-20T09:30:43.403 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=333656019}) 2024-03-20T09:30:43.403 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4072959271}) 2024-03-20T09:30:43.403 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3960040342}) 2024-03-20T09:30:43.403 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1927387634}) 2024-03-20T09:30:43.403 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1485768252}) 2024-03-20T09:30:43.403 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1650980092}) 2024-03-20T09:30:43.403 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2860176741}) 2024-03-20T09:30:43.403 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3228993712}) 2024-03-20T09:30:43.403 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2600731019}) 2024-03-20T09:30:43.403 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=90781852}) 2024-03-20T09:30:43.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1236941211}) 2024-03-20T09:30:43.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1462225100}) 2024-03-20T09:30:43.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2046090448}) 2024-03-20T09:30:43.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3280135892}) 2024-03-20T09:30:43.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2845478018}) 2024-03-20T09:30:43.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1723469209}) 2024-03-20T09:30:43.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2098411770}) 2024-03-20T09:30:43.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1479321126}) 2024-03-20T09:30:43.404 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4118457232}) 2024-03-20T09:30:43.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3642669505}) 2024-03-20T09:30:43.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2292260130}) 2024-03-20T09:30:43.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2812129074}) 2024-03-20T09:30:43.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2759718205}) 2024-03-20T09:30:43.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2635213625}) 2024-03-20T09:30:43.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=995107027}) 2024-03-20T09:30:43.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1528728856}) 2024-03-20T09:30:43.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3716164204}) 2024-03-20T09:30:43.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=265245435}) 2024-03-20T09:30:43.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2824756537}) 2024-03-20T09:30:43.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=661496145}) 2024-03-20T09:30:43.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4001725031}) 2024-03-20T09:30:43.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4047347226}) 2024-03-20T09:30:43.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=336620220}) 2024-03-20T09:30:43.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3600941964}) 2024-03-20T09:30:43.405 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1688412983}) 2024-03-20T09:30:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=805061760}) 2024-03-20T09:30:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=456454241}) 2024-03-20T09:30:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1244054033}) 2024-03-20T09:30:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3140656603}) 2024-03-20T09:30:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2831682078}) 2024-03-20T09:30:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=651041489}) 2024-03-20T09:30:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3290203085}) 2024-03-20T09:30:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4233591458}) 2024-03-20T09:30:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2227820846}) 2024-03-20T09:30:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3922319328}) 2024-03-20T09:30:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2520275631}) 2024-03-20T09:30:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3630040097}) 2024-03-20T09:30:43.406 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2723266549}) 2024-03-20T09:30:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=695585522}) 2024-03-20T09:30:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2730357871}) 2024-03-20T09:30:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2741530447}) 2024-03-20T09:30:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2281510726}) 2024-03-20T09:30:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1331343500}) 2024-03-20T09:30:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3516503145}) 2024-03-20T09:30:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1802394395}) 2024-03-20T09:30:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=982934604}) 2024-03-20T09:30:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=628551956}) 2024-03-20T09:30:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2618867919}) 2024-03-20T09:30:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2334738014}) 2024-03-20T09:30:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=597140261}) 2024-03-20T09:30:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3146104049}) 2024-03-20T09:30:43.407 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1457794569}) 2024-03-20T09:30:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2181943042}) 2024-03-20T09:30:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2408446137}) 2024-03-20T09:30:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2835131343}) 2024-03-20T09:30:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=77,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=1} crc {logm=3777504671,mdsmap=3984425268,mgr=427508465,mgr_command_descs=3831600018,mgr_metadata=2112171679}) 2024-03-20T09:30:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {mgr_metadata=1,mgrstat=10,mon_config_key=89} crc {mgr_metadata=2912525676,mgrstat=2388164258,mon_config_key=3553608912}) 2024-03-20T09:30:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2957981141}) 2024-03-20T09:30:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=89,monmap=5,osd_metadata=6} crc {mon_config_key=3274690714,monmap=3481182261,osd_metadata=1296877480}) 2024-03-20T09:30:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=6,osd_pg_creating=1,osdmap=93} crc {osd_metadata=2273555789,osd_pg_creating=332160715,osdmap=3340407400}) 2024-03-20T09:30:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1944444918}) 2024-03-20T09:30:43.408 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=64} crc {osdmap=2429811463}) 2024-03-20T09:30:43.439 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:30:43.087+0000 7fa926383700 -1 osd.10 860 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-20T09:30:43.440 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:30:43.440 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:30:43.440 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: osdmap e859: 12 total, 11 up, 11 in 2024-03-20T09:30:43.440 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: from='osd.10 [v2:172.21.15.122:6816/2853088860,v1:172.21.15.122:6817/2853088860]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-20T09:30:43.440 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-20T09:30:43.440 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: pgmap v2082: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfilling, 55 active+clean; 368 MiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 885 op/s; 208/17652 objects degraded (1.178%); 671/17652 objects misplaced (3.801%); 2.4 MiB/s, 38 objects/s recovering 2024-03-20T09:30:43.440 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 208/22446 objects degraded (0.927%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-20T09:30:43.440 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-20T09:30:43.440 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: from='osd.10 [v2:172.21.15.122:6816/2853088860,v1:172.21.15.122:6817/2853088860]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-20T09:30:43.440 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: osdmap e860: 12 total, 11 up, 11 in 2024-03-20T09:30:43.440 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-03-20T09:30:43.440 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-20T09:30:43.440 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=57,logm=3} crc {auth=414148163,config=3599132250,health=34561630,kv=3379067375,logm=4044773816}) 2024-03-20T09:30:43.441 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2202345666}) 2024-03-20T09:30:43.441 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2111144228}) 2024-03-20T09:30:43.441 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3339523933}) 2024-03-20T09:30:43.441 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3209217667}) 2024-03-20T09:30:43.441 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1882255966}) 2024-03-20T09:30:43.441 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3783532648}) 2024-03-20T09:30:43.441 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=333656019}) 2024-03-20T09:30:43.441 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4072959271}) 2024-03-20T09:30:43.441 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3960040342}) 2024-03-20T09:30:43.441 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1927387634}) 2024-03-20T09:30:43.441 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1485768252}) 2024-03-20T09:30:43.441 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1650980092}) 2024-03-20T09:30:43.442 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2860176741}) 2024-03-20T09:30:43.442 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3228993712}) 2024-03-20T09:30:43.442 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2600731019}) 2024-03-20T09:30:43.442 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=90781852}) 2024-03-20T09:30:43.442 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1236941211}) 2024-03-20T09:30:43.442 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1462225100}) 2024-03-20T09:30:43.442 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2046090448}) 2024-03-20T09:30:43.442 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3280135892}) 2024-03-20T09:30:43.442 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2845478018}) 2024-03-20T09:30:43.442 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1723469209}) 2024-03-20T09:30:43.442 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2098411770}) 2024-03-20T09:30:43.442 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1479321126}) 2024-03-20T09:30:43.442 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4118457232}) 2024-03-20T09:30:43.443 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3642669505}) 2024-03-20T09:30:43.443 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2292260130}) 2024-03-20T09:30:43.443 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2812129074}) 2024-03-20T09:30:43.443 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2759718205}) 2024-03-20T09:30:43.443 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2635213625}) 2024-03-20T09:30:43.443 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=995107027}) 2024-03-20T09:30:43.443 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1528728856}) 2024-03-20T09:30:43.443 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3716164204}) 2024-03-20T09:30:43.443 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=265245435}) 2024-03-20T09:30:43.443 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2824756537}) 2024-03-20T09:30:43.443 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=661496145}) 2024-03-20T09:30:43.443 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4001725031}) 2024-03-20T09:30:43.443 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4047347226}) 2024-03-20T09:30:43.443 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=336620220}) 2024-03-20T09:30:43.443 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3600941964}) 2024-03-20T09:30:43.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1688412983}) 2024-03-20T09:30:43.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=805061760}) 2024-03-20T09:30:43.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=456454241}) 2024-03-20T09:30:43.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1244054033}) 2024-03-20T09:30:43.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3140656603}) 2024-03-20T09:30:43.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2831682078}) 2024-03-20T09:30:43.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=651041489}) 2024-03-20T09:30:43.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3290203085}) 2024-03-20T09:30:43.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4233591458}) 2024-03-20T09:30:43.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2227820846}) 2024-03-20T09:30:43.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3922319328}) 2024-03-20T09:30:43.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2520275631}) 2024-03-20T09:30:43.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3630040097}) 2024-03-20T09:30:43.444 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2723266549}) 2024-03-20T09:30:43.445 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=695585522}) 2024-03-20T09:30:43.445 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2730357871}) 2024-03-20T09:30:43.445 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2741530447}) 2024-03-20T09:30:43.445 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2281510726}) 2024-03-20T09:30:43.445 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1331343500}) 2024-03-20T09:30:43.445 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3516503145}) 2024-03-20T09:30:43.445 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1802394395}) 2024-03-20T09:30:43.445 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=982934604}) 2024-03-20T09:30:43.445 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=628551956}) 2024-03-20T09:30:43.445 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2618867919}) 2024-03-20T09:30:43.445 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2334738014}) 2024-03-20T09:30:43.445 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=597140261}) 2024-03-20T09:30:43.445 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3146104049}) 2024-03-20T09:30:43.445 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1457794569}) 2024-03-20T09:30:43.445 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2181943042}) 2024-03-20T09:30:43.445 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2408446137}) 2024-03-20T09:30:43.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2835131343}) 2024-03-20T09:30:43.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=77,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=1} crc {logm=3777504671,mdsmap=3984425268,mgr=427508465,mgr_command_descs=3831600018,mgr_metadata=2112171679}) 2024-03-20T09:30:43.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {mgr_metadata=1,mgrstat=10,mon_config_key=89} crc {mgr_metadata=2912525676,mgrstat=2388164258,mon_config_key=3553608912}) 2024-03-20T09:30:43.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2957981141}) 2024-03-20T09:30:43.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=89,monmap=5,osd_metadata=6} crc {mon_config_key=3274690714,monmap=3481182261,osd_metadata=1296877480}) 2024-03-20T09:30:43.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=6,osd_pg_creating=1,osdmap=93} crc {osd_metadata=2273555789,osd_pg_creating=332160715,osdmap=3340407400}) 2024-03-20T09:30:43.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1944444918}) 2024-03-20T09:30:43.446 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=64} crc {osdmap=2429811463}) 2024-03-20T09:30:43.449 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:30:43.524 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:43.904 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:30:43.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:43.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:30:43.960 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-20T09:30:44.019 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:30:44.129 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:44.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:44 smithi026 ceph-mon[68309]: from='osd.10 ' entity='osd.10' 2024-03-20T09:30:44.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:44 smithi026 ceph-mon[68309]: 6.d scrub starts 2024-03-20T09:30:44.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:44 smithi026 ceph-mon[68309]: 6.d scrub ok 2024-03-20T09:30:44.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:44 smithi115 ceph-mon[72476]: from='osd.10 ' entity='osd.10' 2024-03-20T09:30:44.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:44 smithi115 ceph-mon[72476]: 6.d scrub starts 2024-03-20T09:30:44.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:44 smithi115 ceph-mon[72476]: 6.d scrub ok 2024-03-20T09:30:44.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:44 smithi122 ceph-mon[73495]: from='osd.10 ' entity='osd.10' 2024-03-20T09:30:44.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:44 smithi122 ceph-mon[73495]: 6.d scrub starts 2024-03-20T09:30:44.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:44 smithi122 ceph-mon[73495]: 6.d scrub ok 2024-03-20T09:30:44.539 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:30:44.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:44.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:30:44.616 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:30:44.644 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:30:44.678 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:44.705 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:30:44.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:44.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:30:44.813 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:30:44.842 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:44.863 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:30:44.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:44.867 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:30:44.968 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:30:45.032 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:45.115 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:45 smithi122 ceph-mon[73495]: 6.2 scrub starts 2024-03-20T09:30:45.115 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:45 smithi122 ceph-mon[73495]: 6.2 scrub ok 2024-03-20T09:30:45.115 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:45 smithi122 ceph-mon[73495]: 1.17 scrub starts 2024-03-20T09:30:45.115 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:45 smithi122 ceph-mon[73495]: osd.10 [v2:172.21.15.122:6816/2853088860,v1:172.21.15.122:6817/2853088860] boot 2024-03-20T09:30:45.115 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:45 smithi122 ceph-mon[73495]: osdmap e861: 12 total, 12 up, 11 in 2024-03-20T09:30:45.115 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:45 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-20T09:30:45.115 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:45 smithi122 ceph-mon[73495]: 1.17 scrub ok 2024-03-20T09:30:45.116 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:45 smithi122 ceph-mon[73495]: 6.7 scrub starts 2024-03-20T09:30:45.116 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:45 smithi122 ceph-mon[73495]: 6.7 scrub ok 2024-03-20T09:30:45.116 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:45 smithi122 ceph-mon[73495]: pgmap v2085: 57 pgs: 1 active+recovering+undersized+remapped, 56 active+clean; 332 MiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 651 op/s; 214/15897 objects misplaced (1.346%); 3.3 MiB/s, 52 objects/s recovering 2024-03-20T09:30:45.146 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:30:45.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:45.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:30:45.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:45 smithi026 ceph-mon[68309]: 6.2 scrub starts 2024-03-20T09:30:45.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:45 smithi026 ceph-mon[68309]: 6.2 scrub ok 2024-03-20T09:30:45.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:45 smithi026 ceph-mon[68309]: 1.17 scrub starts 2024-03-20T09:30:45.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:45 smithi026 ceph-mon[68309]: osd.10 [v2:172.21.15.122:6816/2853088860,v1:172.21.15.122:6817/2853088860] boot 2024-03-20T09:30:45.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:45 smithi026 ceph-mon[68309]: osdmap e861: 12 total, 12 up, 11 in 2024-03-20T09:30:45.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:45 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-20T09:30:45.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:45 smithi026 ceph-mon[68309]: 1.17 scrub ok 2024-03-20T09:30:45.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:45 smithi026 ceph-mon[68309]: 6.7 scrub starts 2024-03-20T09:30:45.215 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:45 smithi026 ceph-mon[68309]: 6.7 scrub ok 2024-03-20T09:30:45.215 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:45 smithi026 ceph-mon[68309]: pgmap v2085: 57 pgs: 1 active+recovering+undersized+remapped, 56 active+clean; 332 MiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 651 op/s; 214/15897 objects misplaced (1.346%); 3.3 MiB/s, 52 objects/s recovering 2024-03-20T09:30:45.252 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:30:45.283 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:45.303 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:30:45.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:45.307 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:30:45.372 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:30:45.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:45 smithi115 ceph-mon[72476]: 6.2 scrub starts 2024-03-20T09:30:45.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:45 smithi115 ceph-mon[72476]: 6.2 scrub ok 2024-03-20T09:30:45.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:45 smithi115 ceph-mon[72476]: 1.17 scrub starts 2024-03-20T09:30:45.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:45 smithi115 ceph-mon[72476]: osd.10 [v2:172.21.15.122:6816/2853088860,v1:172.21.15.122:6817/2853088860] boot 2024-03-20T09:30:45.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:45 smithi115 ceph-mon[72476]: osdmap e861: 12 total, 12 up, 11 in 2024-03-20T09:30:45.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:45 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-20T09:30:45.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:45 smithi115 ceph-mon[72476]: 1.17 scrub ok 2024-03-20T09:30:45.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:45 smithi115 ceph-mon[72476]: 6.7 scrub starts 2024-03-20T09:30:45.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:45 smithi115 ceph-mon[72476]: 6.7 scrub ok 2024-03-20T09:30:45.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:45 smithi115 ceph-mon[72476]: pgmap v2085: 57 pgs: 1 active+recovering+undersized+remapped, 56 active+clean; 332 MiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 651 op/s; 214/15897 objects misplaced (1.346%); 3.3 MiB/s, 52 objects/s recovering 2024-03-20T09:30:45.408 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:30:45.454 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:45.479 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:30:45.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:45.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:30:45.584 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:30:45.635 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:45.665 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:30:45.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:45.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:30:45.771 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:30:45.804 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:45.837 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:30:45.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:45.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:30:45.945 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:30:45.972 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:45.995 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:30:45.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:45.999 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:30:46.100 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:30:46.126 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:46 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:30:46.126 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:46 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:30:46.126 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:46 smithi115 ceph-mon[72476]: osdmap e862: 12 total, 12 up, 11 in 2024-03-20T09:30:46.126 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:46 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3282463401' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:30:46.126 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:46 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:30:46.141 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:46.166 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:30:46.166 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:30:46.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:46.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:30:46.199 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:46 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:30:46.199 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:46 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:30:46.199 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:46 smithi026 ceph-mon[68309]: osdmap e862: 12 total, 12 up, 11 in 2024-03-20T09:30:46.199 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:46 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3282463401' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:30:46.199 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:46 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:30:46.272 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:30:46.334 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:46.432 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:30:46.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:46.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:30:46.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:46 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:30:46.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:46 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:30:46.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:46 smithi122 ceph-mon[73495]: osdmap e862: 12 total, 12 up, 11 in 2024-03-20T09:30:46.440 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:46 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3282463401' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:30:46.440 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:46 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:30:46.539 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:30:46.586 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:46.609 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:30:46.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:46.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:30:46.714 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:30:46.764 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:46.793 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:30:46.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:46.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:30:46.897 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:30:46.948 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:46.978 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:30:46.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:46.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:30:47.084 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:30:47.130 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:47.133 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_ops_in_flight 2024-03-20T09:30:47.157 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:30:47.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:47.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:30:47.264 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:30:47.292 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:47.321 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:30:47.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:47.327 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:30:47.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:47 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:30:47.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:47 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:30:47.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:47 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:30:47.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:47 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:30:47.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:47 smithi115 ceph-mon[72476]: osdmap e863: 12 total, 12 up, 11 in 2024-03-20T09:30:47.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:47 smithi115 ceph-mon[72476]: 6.c scrub starts 2024-03-20T09:30:47.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:47 smithi115 ceph-mon[72476]: 6.19 scrub starts 2024-03-20T09:30:47.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:47 smithi115 ceph-mon[72476]: 6.19 scrub ok 2024-03-20T09:30:47.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:47 smithi115 ceph-mon[72476]: pgmap v2088: 57 pgs: 1 active+recovering+undersized+remapped, 56 active+clean; 175 MiB data, 7.7 GiB used, 976 GiB / 983 GiB avail; 214/8361 objects misplaced (2.560%); 4.9 MiB/s, 78 objects/s recovering 2024-03-20T09:30:47.416 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Removed 120106 objects 2024-03-20T09:30:47.416 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Clean up completed and total clean up time :299.55 2024-03-20T09:30:47.428 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:30:47.435 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:30:47.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:47 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:30:47.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:47 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:30:47.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:47 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:30:47.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:47 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:30:47.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:47 smithi122 ceph-mon[73495]: osdmap e863: 12 total, 12 up, 11 in 2024-03-20T09:30:47.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:47 smithi122 ceph-mon[73495]: 6.c scrub starts 2024-03-20T09:30:47.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:47 smithi122 ceph-mon[73495]: 6.19 scrub starts 2024-03-20T09:30:47.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:47 smithi122 ceph-mon[73495]: 6.19 scrub ok 2024-03-20T09:30:47.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:47 smithi122 ceph-mon[73495]: pgmap v2088: 57 pgs: 1 active+recovering+undersized+remapped, 56 active+clean; 175 MiB data, 7.7 GiB used, 976 GiB / 983 GiB avail; 214/8361 objects misplaced (2.560%); 4.9 MiB/s, 78 objects/s recovering 2024-03-20T09:30:47.453 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:47.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:47 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:30:47.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:47 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:30:47.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:47 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:30:47.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:47 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:30:47.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:47 smithi026 ceph-mon[68309]: osdmap e863: 12 total, 12 up, 11 in 2024-03-20T09:30:47.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:47 smithi026 ceph-mon[68309]: 6.c scrub starts 2024-03-20T09:30:47.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:47 smithi026 ceph-mon[68309]: 6.19 scrub starts 2024-03-20T09:30:47.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:47 smithi026 ceph-mon[68309]: 6.19 scrub ok 2024-03-20T09:30:47.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:47 smithi026 ceph-mon[68309]: pgmap v2088: 57 pgs: 1 active+recovering+undersized+remapped, 56 active+clean; 175 MiB data, 7.7 GiB used, 976 GiB / 983 GiB avail; 214/8361 objects misplaced (2.560%); 4.9 MiB/s, 78 objects/s recovering 2024-03-20T09:30:47.472 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:30:47.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:47.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:30:47.576 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:30:47.607 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:48.359 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:30:48.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:48.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:30:48.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:48 smithi115 ceph-mon[72476]: 6.c scrub ok 2024-03-20T09:30:48.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:48 smithi115 ceph-mon[72476]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-03-20T09:30:48.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:48 smithi115 ceph-mon[72476]: osdmap e864: 12 total, 12 up, 11 in 2024-03-20T09:30:48.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:48 smithi115 ceph-mon[72476]: 1.3 deep-scrub starts 2024-03-20T09:30:48.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:48 smithi115 ceph-mon[72476]: 1.3 deep-scrub ok 2024-03-20T09:30:48.408 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:48 smithi122 ceph-mon[73495]: 6.c scrub ok 2024-03-20T09:30:48.408 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:48 smithi122 ceph-mon[73495]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-03-20T09:30:48.408 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:48 smithi122 ceph-mon[73495]: osdmap e864: 12 total, 12 up, 11 in 2024-03-20T09:30:48.409 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:48 smithi122 ceph-mon[73495]: 1.3 deep-scrub starts 2024-03-20T09:30:48.409 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:48 smithi122 ceph-mon[73495]: 1.3 deep-scrub ok 2024-03-20T09:30:48.428 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:30:48.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:48 smithi026 ceph-mon[68309]: 6.c scrub ok 2024-03-20T09:30:48.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:48 smithi026 ceph-mon[68309]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-03-20T09:30:48.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:48 smithi026 ceph-mon[68309]: osdmap e864: 12 total, 12 up, 11 in 2024-03-20T09:30:48.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:48 smithi026 ceph-mon[68309]: 1.3 deep-scrub starts 2024-03-20T09:30:48.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:48 smithi026 ceph-mon[68309]: 1.3 deep-scrub ok 2024-03-20T09:30:48.467 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:30:48.496 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:49.239 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:49 smithi026 ceph-mon[68309]: 1.12 scrub starts 2024-03-20T09:30:49.239 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:49 smithi026 ceph-mon[68309]: 1.12 scrub ok 2024-03-20T09:30:49.240 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:49 smithi026 ceph-mon[68309]: pgmap v2090: 57 pgs: 1 active+recovering+undersized+remapped, 56 active+clean; 86 MiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 1.1 MiB/s, 47 objects/s recovering 2024-03-20T09:30:49.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:49 smithi115 ceph-mon[72476]: 1.12 scrub starts 2024-03-20T09:30:49.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:49 smithi115 ceph-mon[72476]: 1.12 scrub ok 2024-03-20T09:30:49.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:49 smithi115 ceph-mon[72476]: pgmap v2090: 57 pgs: 1 active+recovering+undersized+remapped, 56 active+clean; 86 MiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 1.1 MiB/s, 47 objects/s recovering 2024-03-20T09:30:49.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:49 smithi122 ceph-mon[73495]: 1.12 scrub starts 2024-03-20T09:30:49.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:49 smithi122 ceph-mon[73495]: 1.12 scrub ok 2024-03-20T09:30:49.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:49 smithi122 ceph-mon[73495]: pgmap v2090: 57 pgs: 1 active+recovering+undersized+remapped, 56 active+clean; 86 MiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 1.1 MiB/s, 47 objects/s recovering 2024-03-20T09:30:49.514 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:30:49.514 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:30:49.514 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-03-20T09:30:49.514 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:30:49.546 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:30:49.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:49.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:30:49.652 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:30:49.684 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:50.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:50 smithi122 ceph-mon[73495]: 1.18 scrub starts 2024-03-20T09:30:50.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:50 smithi122 ceph-mon[73495]: 1.18 scrub ok 2024-03-20T09:30:50.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:50 smithi115 ceph-mon[72476]: 1.18 scrub starts 2024-03-20T09:30:50.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:50 smithi115 ceph-mon[72476]: 1.18 scrub ok 2024-03-20T09:30:50.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:50 smithi026 ceph-mon[68309]: 1.18 scrub starts 2024-03-20T09:30:50.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:50 smithi026 ceph-mon[68309]: 1.18 scrub ok 2024-03-20T09:30:51.022 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:30:51.405 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:30:51.406 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:30:51.426 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:51 smithi026 ceph-mon[68309]: 1.c scrub starts 2024-03-20T09:30:51.426 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:51 smithi026 ceph-mon[68309]: 1.c scrub ok 2024-03-20T09:30:51.426 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:51 smithi026 ceph-mon[68309]: pgmap v2091: 57 pgs: 1 active+recovering+undersized+remapped, 56 active+clean; 31 MiB data, 7.3 GiB used, 976 GiB / 983 GiB avail 2024-03-20T09:30:51.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:51 smithi122 ceph-mon[73495]: 1.c scrub starts 2024-03-20T09:30:51.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:51 smithi122 ceph-mon[73495]: 1.c scrub ok 2024-03-20T09:30:51.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:51 smithi122 ceph-mon[73495]: pgmap v2091: 57 pgs: 1 active+recovering+undersized+remapped, 56 active+clean; 31 MiB data, 7.3 GiB used, 976 GiB / 983 GiB avail 2024-03-20T09:30:51.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:51 smithi115 ceph-mon[72476]: 1.c scrub starts 2024-03-20T09:30:51.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:51 smithi115 ceph-mon[72476]: 1.c scrub ok 2024-03-20T09:30:51.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:51 smithi115 ceph-mon[72476]: pgmap v2091: 57 pgs: 1 active+recovering+undersized+remapped, 56 active+clean; 31 MiB data, 7.3 GiB used, 976 GiB / 983 GiB avail 2024-03-20T09:30:51.937 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:30:51.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:51.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:30:52.041 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:30:52.075 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:52.096 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:30:52.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:52.098 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:30:52.200 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:30:52.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:52 smithi026 ceph-mon[68309]: 1.8 scrub starts 2024-03-20T09:30:52.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:52 smithi026 ceph-mon[68309]: 1.8 scrub ok 2024-03-20T09:30:52.228 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:52.247 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:30:52.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:52.250 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:30:52.351 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:30:52.381 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:52.395 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-03-20T09:30:52.395 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-03-20T09:30:52.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:52 smithi122 ceph-mon[73495]: 1.8 scrub starts 2024-03-20T09:30:52.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:52 smithi122 ceph-mon[73495]: 1.8 scrub ok 2024-03-20T09:30:52.648 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:52 smithi115 ceph-mon[72476]: 1.8 scrub starts 2024-03-20T09:30:52.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:52 smithi115 ceph-mon[72476]: 1.8 scrub ok 2024-03-20T09:30:53.294 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-20T09:30:53.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:53 smithi122 ceph-mon[73495]: 1.1c scrub starts 2024-03-20T09:30:53.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:53 smithi122 ceph-mon[73495]: 1.1c scrub ok 2024-03-20T09:30:53.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:53 smithi122 ceph-mon[73495]: 6.11 scrub starts 2024-03-20T09:30:53.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:53 smithi122 ceph-mon[73495]: 6.11 scrub ok 2024-03-20T09:30:53.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:53 smithi122 ceph-mon[73495]: pgmap v2092: 57 pgs: 1 active+recovering+undersized+remapped, 56 active+clean; 7.1 MiB data, 7.1 GiB used, 976 GiB / 983 GiB avail 2024-03-20T09:30:53.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:53 smithi026 ceph-mon[68309]: 1.1c scrub starts 2024-03-20T09:30:53.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:53 smithi026 ceph-mon[68309]: 1.1c scrub ok 2024-03-20T09:30:53.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:53 smithi026 ceph-mon[68309]: 6.11 scrub starts 2024-03-20T09:30:53.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:53 smithi026 ceph-mon[68309]: 6.11 scrub ok 2024-03-20T09:30:53.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:53 smithi026 ceph-mon[68309]: pgmap v2092: 57 pgs: 1 active+recovering+undersized+remapped, 56 active+clean; 7.1 MiB data, 7.1 GiB used, 976 GiB / 983 GiB avail 2024-03-20T09:30:53.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:53 smithi115 ceph-mon[72476]: 1.1c scrub starts 2024-03-20T09:30:53.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:53 smithi115 ceph-mon[72476]: 1.1c scrub ok 2024-03-20T09:30:53.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:53 smithi115 ceph-mon[72476]: 6.11 scrub starts 2024-03-20T09:30:53.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:53 smithi115 ceph-mon[72476]: 6.11 scrub ok 2024-03-20T09:30:53.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:53 smithi115 ceph-mon[72476]: pgmap v2092: 57 pgs: 1 active+recovering+undersized+remapped, 56 active+clean; 7.1 MiB data, 7.1 GiB used, 976 GiB / 983 GiB avail 2024-03-20T09:30:53.733 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:30:53.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:53.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:30:53.754 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:30:53.800 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:30:53.838 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:30:53.871 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:53.892 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:30:53.894 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:53.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:30:53.996 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:30:54.029 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:54.228 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:30:54.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:54.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:30:54.333 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:30:54.366 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:54.388 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:30:54.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:54.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:30:54.492 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:30:54.522 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:54.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:54 smithi115 ceph-mon[72476]: osdmap e865: 12 total, 12 up, 11 in 2024-03-20T09:30:54.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:54 smithi115 ceph-mon[72476]: pgmap v2094: 57 pgs: 1 active+undersized+remapped, 56 active+clean; 7.1 MiB data, 7.1 GiB used, 976 GiB / 983 GiB avail 2024-03-20T09:30:54.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:54 smithi115 ceph-mon[72476]: 1.b scrub starts 2024-03-20T09:30:54.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:54 smithi026 ceph-mon[68309]: osdmap e865: 12 total, 12 up, 11 in 2024-03-20T09:30:54.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:54 smithi026 ceph-mon[68309]: pgmap v2094: 57 pgs: 1 active+undersized+remapped, 56 active+clean; 7.1 MiB data, 7.1 GiB used, 976 GiB / 983 GiB avail 2024-03-20T09:30:54.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:54 smithi026 ceph-mon[68309]: 1.b scrub starts 2024-03-20T09:30:54.772 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:30:54.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:54.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:30:54.876 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:30:54.906 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:54 smithi122 ceph-mon[73495]: osdmap e865: 12 total, 12 up, 11 in 2024-03-20T09:30:54.906 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:54 smithi122 ceph-mon[73495]: pgmap v2094: 57 pgs: 1 active+undersized+remapped, 56 active+clean; 7.1 MiB data, 7.1 GiB used, 976 GiB / 983 GiB avail 2024-03-20T09:30:54.906 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:54 smithi122 ceph-mon[73495]: 1.b scrub starts 2024-03-20T09:30:54.908 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:54.928 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:30:54.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:54.931 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:30:55.032 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:30:55.059 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:55.078 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:30:55.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:55.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:30:55.182 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:30:55.209 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:55.227 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:30:55.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:55.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:30:55.240 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 8, 11, 3, 4, 9] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 11, 5, 10] 2024-03-20T09:30:55.240 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:30:55.240 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-20T09:30:55.240 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.000000 2024-03-20T09:30:55.241 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd primary-affinity 1 0 2024-03-20T09:30:55.255 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:30:55.331 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:30:55.361 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:55.380 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:30:55.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:55.383 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:30:55.484 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:30:55.514 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:55.606 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:30:55.625 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:30:55.644 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:30:55.664 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:30:55.684 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:30:55.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:55 smithi122 ceph-mon[73495]: 1.14 deep-scrub starts 2024-03-20T09:30:55.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:55 smithi122 ceph-mon[73495]: 1.14 deep-scrub ok 2024-03-20T09:30:55.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:55 smithi122 ceph-mon[73495]: 6.9 scrub starts 2024-03-20T09:30:55.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:55 smithi122 ceph-mon[73495]: 6.9 scrub ok 2024-03-20T09:30:55.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:55 smithi122 ceph-mon[73495]: 1.10 scrub starts 2024-03-20T09:30:55.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:55 smithi122 ceph-mon[73495]: 1.10 scrub ok 2024-03-20T09:30:55.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:55 smithi122 ceph-mon[73495]: osdmap e866: 12 total, 12 up, 11 in 2024-03-20T09:30:55.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:55 smithi122 ceph-mon[73495]: 1.b scrub ok 2024-03-20T09:30:55.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:55 smithi122 ceph-mon[73495]: 6.3 scrub starts 2024-03-20T09:30:55.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:55 smithi122 ceph-mon[73495]: 6.3 scrub ok 2024-03-20T09:30:55.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:55 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3633481610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:30:55.704 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:30:55.725 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:30:55.746 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:30:55.764 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:30:55.785 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:30:55.805 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:30:55.825 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:30:55.826 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:30:55.826 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:30:55.826 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:30:55.826 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:30:55.826 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:30:55.826 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:30:55.826 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:30:55.826 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:30:55.826 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:30:55.826 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:30:55.827 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:30:55.827 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:30:55.845 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:55 smithi026 ceph-mon[68309]: 1.14 deep-scrub starts 2024-03-20T09:30:55.846 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:55 smithi026 ceph-mon[68309]: 1.14 deep-scrub ok 2024-03-20T09:30:55.846 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:55 smithi026 ceph-mon[68309]: 6.9 scrub starts 2024-03-20T09:30:55.846 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:55 smithi026 ceph-mon[68309]: 6.9 scrub ok 2024-03-20T09:30:55.846 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:55 smithi026 ceph-mon[68309]: 1.10 scrub starts 2024-03-20T09:30:55.846 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:55 smithi026 ceph-mon[68309]: 1.10 scrub ok 2024-03-20T09:30:55.846 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:55 smithi026 ceph-mon[68309]: osdmap e866: 12 total, 12 up, 11 in 2024-03-20T09:30:55.846 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:55 smithi026 ceph-mon[68309]: 1.b scrub ok 2024-03-20T09:30:55.846 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:55 smithi026 ceph-mon[68309]: 6.3 scrub starts 2024-03-20T09:30:55.846 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:55 smithi026 ceph-mon[68309]: 6.3 scrub ok 2024-03-20T09:30:55.846 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:55 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3633481610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:30:55.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:55 smithi115 ceph-mon[72476]: 1.14 deep-scrub starts 2024-03-20T09:30:55.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:55 smithi115 ceph-mon[72476]: 1.14 deep-scrub ok 2024-03-20T09:30:55.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:55 smithi115 ceph-mon[72476]: 6.9 scrub starts 2024-03-20T09:30:55.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:55 smithi115 ceph-mon[72476]: 6.9 scrub ok 2024-03-20T09:30:55.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:55 smithi115 ceph-mon[72476]: 1.10 scrub starts 2024-03-20T09:30:55.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:55 smithi115 ceph-mon[72476]: 1.10 scrub ok 2024-03-20T09:30:55.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:55 smithi115 ceph-mon[72476]: osdmap e866: 12 total, 12 up, 11 in 2024-03-20T09:30:55.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:55 smithi115 ceph-mon[72476]: 1.b scrub ok 2024-03-20T09:30:55.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:55 smithi115 ceph-mon[72476]: 6.3 scrub starts 2024-03-20T09:30:55.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:55 smithi115 ceph-mon[72476]: 6.3 scrub ok 2024-03-20T09:30:55.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:55 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3633481610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:30:56.465 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:30:56.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:56.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:30:56.570 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:30:56.604 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:56.624 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:30:56.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:56.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:30:56.729 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:30:56.763 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:56.784 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:30:56.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:56.787 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:30:56.889 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:30:56.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:56 smithi115 ceph-mon[72476]: 6.14 scrub starts 2024-03-20T09:30:56.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:56 smithi115 ceph-mon[72476]: 6.14 scrub ok 2024-03-20T09:30:56.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:56 smithi115 ceph-mon[72476]: 6.0 scrub starts 2024-03-20T09:30:56.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:56 smithi115 ceph-mon[72476]: 6.0 scrub ok 2024-03-20T09:30:56.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:56 smithi115 ceph-mon[72476]: pgmap v2096: 57 pgs: 1 active+undersized+remapped, 56 active+clean; 7.1 MiB data, 6.9 GiB used, 977 GiB / 983 GiB avail 2024-03-20T09:30:56.921 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:56.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:56 smithi122 ceph-mon[73495]: 6.14 scrub starts 2024-03-20T09:30:56.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:56 smithi122 ceph-mon[73495]: 6.14 scrub ok 2024-03-20T09:30:56.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:56 smithi122 ceph-mon[73495]: 6.0 scrub starts 2024-03-20T09:30:56.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:56 smithi122 ceph-mon[73495]: 6.0 scrub ok 2024-03-20T09:30:56.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:56 smithi122 ceph-mon[73495]: pgmap v2096: 57 pgs: 1 active+undersized+remapped, 56 active+clean; 7.1 MiB data, 6.9 GiB used, 977 GiB / 983 GiB avail 2024-03-20T09:30:56.942 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:30:56.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:56.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:30:56.966 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:56 smithi026 ceph-mon[68309]: 6.14 scrub starts 2024-03-20T09:30:56.966 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:56 smithi026 ceph-mon[68309]: 6.14 scrub ok 2024-03-20T09:30:56.966 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:56 smithi026 ceph-mon[68309]: 6.0 scrub starts 2024-03-20T09:30:56.966 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:56 smithi026 ceph-mon[68309]: 6.0 scrub ok 2024-03-20T09:30:56.966 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:56 smithi026 ceph-mon[68309]: pgmap v2096: 57 pgs: 1 active+undersized+remapped, 56 active+clean; 7.1 MiB data, 6.9 GiB used, 977 GiB / 983 GiB avail 2024-03-20T09:30:57.046 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:30:57.080 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:57.101 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:30:57.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:57.104 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:30:57.205 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:30:57.234 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:57.253 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:30:57.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:57.256 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:30:57.357 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:30:57.384 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:57.404 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:30:57.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:57.407 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:30:57.508 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:30:57.541 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:57.562 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:30:57.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:57.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:30:57.666 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:30:57.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:57 smithi122 ceph-mon[73495]: 1.19 deep-scrub starts 2024-03-20T09:30:57.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:57 smithi122 ceph-mon[73495]: 1.5 scrub starts 2024-03-20T09:30:57.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:57 smithi122 ceph-mon[73495]: 1.19 deep-scrub ok 2024-03-20T09:30:57.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:57 smithi122 ceph-mon[73495]: 1.5 scrub ok 2024-03-20T09:30:57.696 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:57.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:57 smithi026 ceph-mon[68309]: 1.19 deep-scrub starts 2024-03-20T09:30:57.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:57 smithi026 ceph-mon[68309]: 1.5 scrub starts 2024-03-20T09:30:57.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:57 smithi026 ceph-mon[68309]: 1.19 deep-scrub ok 2024-03-20T09:30:57.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:57 smithi026 ceph-mon[68309]: 1.5 scrub ok 2024-03-20T09:30:57.716 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:30:57.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:57.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:30:57.820 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:30:57.845 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:57 smithi115 ceph-mon[72476]: 1.19 deep-scrub starts 2024-03-20T09:30:57.845 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:57 smithi115 ceph-mon[72476]: 1.5 scrub starts 2024-03-20T09:30:57.845 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:57 smithi115 ceph-mon[72476]: 1.19 deep-scrub ok 2024-03-20T09:30:57.845 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:57 smithi115 ceph-mon[72476]: 1.5 scrub ok 2024-03-20T09:30:57.847 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:57.865 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:30:57.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:57.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:30:57.969 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:30:58.002 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:58.078 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:30:58.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:58.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:30:58.196 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:30:58.229 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:58.250 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:30:58.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:58.253 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:30:58.354 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:30:58.386 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:58.407 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:30:58.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:58.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:30:58.511 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:30:58.545 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:58.566 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:30:58.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:58.570 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:30:58.606 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'unique_pool_4' removed 2024-03-20T09:30:58.648 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:58 smithi115 ceph-mon[72476]: 6.17 deep-scrub starts 2024-03-20T09:30:58.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:58 smithi115 ceph-mon[72476]: 6.17 deep-scrub ok 2024-03-20T09:30:58.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:58 smithi115 ceph-mon[72476]: 6.16 scrub starts 2024-03-20T09:30:58.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:58 smithi115 ceph-mon[72476]: 6.16 scrub ok 2024-03-20T09:30:58.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:58 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1530976165' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-20T09:30:58.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:58 smithi115 ceph-mon[72476]: pgmap v2097: 57 pgs: 57 active+clean; 4.6 MiB data, 6.9 GiB used, 977 GiB / 983 GiB avail 2024-03-20T09:30:58.670 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:30:58.698 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:58.716 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:30:58.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:58.719 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:30:58.820 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:30:58.844 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:58 smithi122 ceph-mon[73495]: 6.17 deep-scrub starts 2024-03-20T09:30:58.844 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:58 smithi122 ceph-mon[73495]: 6.17 deep-scrub ok 2024-03-20T09:30:58.844 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:58 smithi122 ceph-mon[73495]: 6.16 scrub starts 2024-03-20T09:30:58.844 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:58 smithi122 ceph-mon[73495]: 6.16 scrub ok 2024-03-20T09:30:58.844 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:58 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1530976165' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-20T09:30:58.845 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:58 smithi122 ceph-mon[73495]: pgmap v2097: 57 pgs: 57 active+clean; 4.6 MiB data, 6.9 GiB used, 977 GiB / 983 GiB avail 2024-03-20T09:30:58.846 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:58.864 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:30:58.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:58.867 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:30:58.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:58 smithi026 ceph-mon[68309]: 6.17 deep-scrub starts 2024-03-20T09:30:58.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:58 smithi026 ceph-mon[68309]: 6.17 deep-scrub ok 2024-03-20T09:30:58.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:58 smithi026 ceph-mon[68309]: 6.16 scrub starts 2024-03-20T09:30:58.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:58 smithi026 ceph-mon[68309]: 6.16 scrub ok 2024-03-20T09:30:58.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:58 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1530976165' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-20T09:30:58.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:58 smithi026 ceph-mon[68309]: pgmap v2097: 57 pgs: 57 active+clean; 4.6 MiB data, 6.9 GiB used, 977 GiB / 983 GiB avail 2024-03-20T09:30:58.968 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:30:58.994 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:59.014 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:30:59.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:59.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:30:59.119 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:30:59.146 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:59.165 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:30:59.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:59.168 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:30:59.269 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:30:59.295 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:59.313 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:30:59.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:59.317 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:30:59.417 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:30:59.443 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:59.462 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:30:59.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:59.465 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:30:59.476 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-20T09:30:59.477 INFO:tasks.radosbench:Beginning radosbench... 2024-03-20T09:30:59.477 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_5 2024-03-20T09:30:59.478 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pool create unique_pool_5 16 2024-03-20T09:30:59.566 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:30:59.595 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:59.616 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:30:59.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:59.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:30:59.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:59 smithi026 ceph-mon[68309]: 6.10 scrub starts 2024-03-20T09:30:59.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:59 smithi026 ceph-mon[68309]: 6.10 scrub ok 2024-03-20T09:30:59.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:59 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:30:59.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:59 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:30:59.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:59 smithi026 ceph-mon[68309]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-03-20T09:30:59.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:59 smithi026 ceph-mon[68309]: Cluster is now healthy 2024-03-20T09:30:59.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:59 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1530976165' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-20T09:30:59.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:59 smithi026 ceph-mon[68309]: osdmap e867: 12 total, 12 up, 11 in 2024-03-20T09:30:59.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:59 smithi026 ceph-mon[68309]: 1.f scrub starts 2024-03-20T09:30:59.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:59 smithi026 ceph-mon[68309]: 1.f scrub ok 2024-03-20T09:30:59.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:59 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:30:59.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:30:59 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:30:59.719 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:30:59.744 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:59 smithi122 ceph-mon[73495]: 6.10 scrub starts 2024-03-20T09:30:59.744 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:59 smithi122 ceph-mon[73495]: 6.10 scrub ok 2024-03-20T09:30:59.744 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:59 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:30:59.744 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:59 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:30:59.744 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:59 smithi122 ceph-mon[73495]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-03-20T09:30:59.745 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:59 smithi122 ceph-mon[73495]: Cluster is now healthy 2024-03-20T09:30:59.745 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:59 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1530976165' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-20T09:30:59.745 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:59 smithi122 ceph-mon[73495]: osdmap e867: 12 total, 12 up, 11 in 2024-03-20T09:30:59.745 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:59 smithi122 ceph-mon[73495]: 1.f scrub starts 2024-03-20T09:30:59.745 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:59 smithi122 ceph-mon[73495]: 1.f scrub ok 2024-03-20T09:30:59.745 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:59 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:30:59.745 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:30:59 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:30:59.746 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:59.764 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:30:59.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:59.767 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:30:59.868 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:30:59.894 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:30:59.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:59 smithi115 ceph-mon[72476]: 6.10 scrub starts 2024-03-20T09:30:59.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:59 smithi115 ceph-mon[72476]: 6.10 scrub ok 2024-03-20T09:30:59.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:59 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:30:59.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:59 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:30:59.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:59 smithi115 ceph-mon[72476]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-03-20T09:30:59.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:59 smithi115 ceph-mon[72476]: Cluster is now healthy 2024-03-20T09:30:59.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:59 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1530976165' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-20T09:30:59.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:59 smithi115 ceph-mon[72476]: osdmap e867: 12 total, 12 up, 11 in 2024-03-20T09:30:59.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:59 smithi115 ceph-mon[72476]: 1.f scrub starts 2024-03-20T09:30:59.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:59 smithi115 ceph-mon[72476]: 1.f scrub ok 2024-03-20T09:30:59.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:59 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:30:59.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:30:59 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:30:59.912 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:30:59.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:30:59.916 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:31:00.017 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:31:00.050 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:00.071 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:31:00.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:00.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:31:00.175 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:31:00.202 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:00.220 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:31:00.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:00.223 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:31:00.324 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:31:00.351 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:00.369 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:31:00.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:00.372 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:31:00.473 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:31:00.500 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:00.518 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:31:00.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:00.521 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:31:00.622 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:31:00.654 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:00.675 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:31:00.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:00.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:31:00.751 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:31:00.751 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:31:00.751 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-03-20T09:31:00.751 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:31:00.779 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:31:00.810 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:00.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:00 smithi115 ceph-mon[72476]: 1.11 scrub starts 2024-03-20T09:31:00.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:00 smithi115 ceph-mon[72476]: 1.11 scrub ok 2024-03-20T09:31:00.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:00 smithi115 ceph-mon[72476]: osdmap e868: 12 total, 12 up, 11 in 2024-03-20T09:31:00.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:00 smithi115 ceph-mon[72476]: pgmap v2100: 31 pgs: 31 active+clean; 577 KiB data, 6.9 GiB used, 977 GiB / 983 GiB avail 2024-03-20T09:31:00.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:00 smithi122 ceph-mon[73495]: 1.11 scrub starts 2024-03-20T09:31:00.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:00 smithi122 ceph-mon[73495]: 1.11 scrub ok 2024-03-20T09:31:00.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:00 smithi122 ceph-mon[73495]: osdmap e868: 12 total, 12 up, 11 in 2024-03-20T09:31:00.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:00 smithi122 ceph-mon[73495]: pgmap v2100: 31 pgs: 31 active+clean; 577 KiB data, 6.9 GiB used, 977 GiB / 983 GiB avail 2024-03-20T09:31:00.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:00 smithi026 ceph-mon[68309]: 1.11 scrub starts 2024-03-20T09:31:00.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:00 smithi026 ceph-mon[68309]: 1.11 scrub ok 2024-03-20T09:31:00.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:00 smithi026 ceph-mon[68309]: osdmap e868: 12 total, 12 up, 11 in 2024-03-20T09:31:00.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:00 smithi026 ceph-mon[68309]: pgmap v2100: 31 pgs: 31 active+clean; 577 KiB data, 6.9 GiB used, 977 GiB / 983 GiB avail 2024-03-20T09:31:01.056 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:31:01.056 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:31:01.405 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:31:01.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:01.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:31:01.427 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-20T09:31:01.510 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:31:01.539 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:01.613 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:31:01.617 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:31:01.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:01.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:31:01.667 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:01 smithi026 ceph-mon[68309]: 1.2 scrub starts 2024-03-20T09:31:01.667 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:01 smithi026 ceph-mon[68309]: 1.2 scrub ok 2024-03-20T09:31:01.667 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:01 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/391729851' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:31:01.720 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:31:01.753 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:01.774 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:31:01.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:01.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:31:01.878 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:31:01.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:01 smithi115 ceph-mon[72476]: 1.2 scrub starts 2024-03-20T09:31:01.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:01 smithi115 ceph-mon[72476]: 1.2 scrub ok 2024-03-20T09:31:01.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:01 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/391729851' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:31:01.909 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:01 smithi122 ceph-mon[73495]: 1.2 scrub starts 2024-03-20T09:31:01.909 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:01 smithi122 ceph-mon[73495]: 1.2 scrub ok 2024-03-20T09:31:01.909 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:01 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/391729851' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:31:01.912 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:01.933 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:31:01.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:01.937 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:31:02.038 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:31:02.070 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:02.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:02.095 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:31:02.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:31:02.196 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:31:02.226 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:02.246 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:31:02.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:02.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:31:02.350 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:31:02.383 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:02.386 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:31:02.405 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:31:02.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:02.409 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:31:02.510 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:31:02.539 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:02.560 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:31:02.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:02.564 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:31:02.665 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:31:02.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:02 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:31:02.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:02 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/391729851' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:31:02.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:02 smithi122 ceph-mon[73495]: osdmap e869: 12 total, 12 up, 11 in 2024-03-20T09:31:02.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:02 smithi122 ceph-mon[73495]: pgmap v2102: 31 pgs: 31 active+clean; 577 KiB data, 6.9 GiB used, 977 GiB / 983 GiB avail 2024-03-20T09:31:02.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:02 smithi122 ceph-mon[73495]: osdmap e870: 12 total, 12 up, 11 in 2024-03-20T09:31:02.693 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:02.723 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:31:02.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:02.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:31:02.829 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:31:02.858 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:02.878 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:31:02.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:02.881 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:31:02.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:02 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:31:02.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:02 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/391729851' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:31:02.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:02 smithi115 ceph-mon[72476]: osdmap e869: 12 total, 12 up, 11 in 2024-03-20T09:31:02.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:02 smithi115 ceph-mon[72476]: pgmap v2102: 31 pgs: 31 active+clean; 577 KiB data, 6.9 GiB used, 977 GiB / 983 GiB avail 2024-03-20T09:31:02.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:02 smithi115 ceph-mon[72476]: osdmap e870: 12 total, 12 up, 11 in 2024-03-20T09:31:02.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:02 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:31:02.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:02 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/391729851' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:31:02.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:02 smithi026 ceph-mon[68309]: osdmap e869: 12 total, 12 up, 11 in 2024-03-20T09:31:02.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:02 smithi026 ceph-mon[68309]: pgmap v2102: 31 pgs: 31 active+clean; 577 KiB data, 6.9 GiB used, 977 GiB / 983 GiB avail 2024-03-20T09:31:02.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:02 smithi026 ceph-mon[68309]: osdmap e870: 12 total, 12 up, 11 in 2024-03-20T09:31:02.982 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:31:03.014 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:03.035 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:31:03.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:03.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:31:03.140 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:31:03.170 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:03.190 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:31:03.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:03.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:31:03.295 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:31:03.326 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:03.352 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:31:03.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:03.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:31:03.461 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:31:03.488 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:03.530 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:31:03.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:03.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:31:03.637 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:31:03.670 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:03.694 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:31:03.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:03.697 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:31:03.798 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:31:03.827 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:03.901 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:31:03.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:03.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:31:04.006 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:31:04.034 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:04.327 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:31:04.327 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:31:05.385 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:31:05.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:05.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:31:05.489 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:31:05.523 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:05.545 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:31:05.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:05.548 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:31:05.649 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:31:05.676 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:05 smithi026 ceph-mon[68309]: pgmap v2104: 31 pgs: 31 active+clean; 577 KiB data, 6.9 GiB used, 977 GiB / 983 GiB avail 2024-03-20T09:31:05.679 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:05.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:05 smithi122 ceph-mon[73495]: pgmap v2104: 31 pgs: 31 active+clean; 577 KiB data, 6.9 GiB used, 977 GiB / 983 GiB avail 2024-03-20T09:31:05.734 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:31:05.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:05.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:31:05.840 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:31:05.873 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:05.894 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:31:05.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:05.897 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:31:05.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:05 smithi115 ceph-mon[72476]: pgmap v2104: 31 pgs: 31 active+clean; 577 KiB data, 6.9 GiB used, 977 GiB / 983 GiB avail 2024-03-20T09:31:05.998 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:31:06.031 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:06.052 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:31:06.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:06.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:31:06.156 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:31:06.189 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:06.212 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:31:06.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:06.216 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:31:06.317 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:31:06.350 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:06.370 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:31:06.372 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:31:06.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:06.376 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:31:06.413 INFO:teuthology.orchestra.run.smithi026.stderr:set osd.1 primary-affinity to 0 (802) 2024-03-20T09:31:06.415 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'unique_pool_5' created 2024-03-20T09:31:06.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:06 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/521785902' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.0}]: dispatch 2024-03-20T09:31:06.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:06 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4206060392' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-03-20T09:31:06.476 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:31:06.508 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:06.529 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:31:06.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:06.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:31:06.634 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:31:06.660 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:06 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/521785902' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.0}]: dispatch 2024-03-20T09:31:06.660 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:06 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4206060392' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-03-20T09:31:06.662 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:06.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:06 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/521785902' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.0}]: dispatch 2024-03-20T09:31:06.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:06 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4206060392' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]: dispatch 2024-03-20T09:31:06.912 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:31:06.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:06.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:31:07.017 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:31:07.046 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:07.680 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:07 smithi026 ceph-mon[68309]: pgmap v2105: 31 pgs: 31 active+clean; 577 KiB data, 6.9 GiB used, 977 GiB / 983 GiB avail 2024-03-20T09:31:07.680 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:07 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/521785902' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.0}]': finished 2024-03-20T09:31:07.680 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:07 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4206060392' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-03-20T09:31:07.681 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:07 smithi026 ceph-mon[68309]: osdmap e871: 12 total, 12 up, 11 in 2024-03-20T09:31:07.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:07 smithi122 ceph-mon[73495]: pgmap v2105: 31 pgs: 31 active+clean; 577 KiB data, 6.9 GiB used, 977 GiB / 983 GiB avail 2024-03-20T09:31:07.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:07 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/521785902' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.0}]': finished 2024-03-20T09:31:07.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:07 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4206060392' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-03-20T09:31:07.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:07 smithi122 ceph-mon[73495]: osdmap e871: 12 total, 12 up, 11 in 2024-03-20T09:31:07.695 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:31:07.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:07.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:31:07.800 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:31:07.829 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:07 smithi115 ceph-mon[72476]: pgmap v2105: 31 pgs: 31 active+clean; 577 KiB data, 6.9 GiB used, 977 GiB / 983 GiB avail 2024-03-20T09:31:07.830 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:07 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/521785902' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.0}]': finished 2024-03-20T09:31:07.830 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:07 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4206060392' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-03-20T09:31:07.830 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:07 smithi115 ceph-mon[72476]: osdmap e871: 12 total, 12 up, 11 in 2024-03-20T09:31:07.832 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:07.853 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:31:07.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:07.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:31:07.958 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:31:07.991 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:08.013 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:31:08.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:08.016 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:31:08.117 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:31:08.146 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:08.167 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:31:08.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:08.170 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:31:08.271 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:31:08.303 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:08.314 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:31:08.314 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:31:08.314 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-03-20T09:31:08.314 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-03-20T09:31:08.314 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:31:08.324 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:31:08.327 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:08.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:31:08.429 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:31:08.463 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:08.563 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:31:08.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:08.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:31:08.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:08 smithi115 ceph-mon[72476]: osdmap e872: 12 total, 12 up, 11 in 2024-03-20T09:31:08.653 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:08 smithi026 ceph-mon[68309]: osdmap e872: 12 total, 12 up, 11 in 2024-03-20T09:31:08.669 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:31:08.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:08 smithi122 ceph-mon[73495]: osdmap e872: 12 total, 12 up, 11 in 2024-03-20T09:31:08.699 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:08.719 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:31:08.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:08.722 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:31:08.823 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:31:08.856 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:08.877 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:31:08.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:08.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:31:08.981 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:31:08.993 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pool application enable unique_pool_5 rados --yes-i-really-mean-it || true 2024-03-20T09:31:09.010 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:09.029 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:31:09.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:09.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:31:09.071 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:31:09.071 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:31:09.072 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:31:09.072 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:31:09.072 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:31:09.072 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:31:09.072 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:31:09.072 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:31:09.072 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:31:09.072 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:31:09.072 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:31:09.072 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:31:09.072 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:31:09.072 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:31:09.072 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:31:09.073 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:31:09.073 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:31:09.073 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:31:09.073 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:31:09.073 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:31:09.073 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:31:09.073 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:31:09.073 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:31:09.073 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:31:09.133 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:31:09.166 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:09.186 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:31:09.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:09.190 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:31:09.291 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:31:09.319 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:09.639 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:09 smithi026 ceph-mon[68309]: pgmap v2108: 47 pgs: 1 peering, 4 creating+peering, 9 unknown, 33 active+clean; 577 KiB data, 6.9 GiB used, 977 GiB / 983 GiB avail 2024-03-20T09:31:09.639 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:09 smithi026 ceph-mon[68309]: osdmap e873: 12 total, 12 up, 11 in 2024-03-20T09:31:09.639 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:09 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3586082786' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:31:09.640 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:09 smithi026 ceph-mon[68309]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-20T09:31:09.640 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:09 smithi026 ceph-mon[68309]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-20T09:31:09.669 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:31:09.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:09.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:31:09.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:09 smithi122 ceph-mon[73495]: pgmap v2108: 47 pgs: 1 peering, 4 creating+peering, 9 unknown, 33 active+clean; 577 KiB data, 6.9 GiB used, 977 GiB / 983 GiB avail 2024-03-20T09:31:09.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:09 smithi122 ceph-mon[73495]: osdmap e873: 12 total, 12 up, 11 in 2024-03-20T09:31:09.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:09 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3586082786' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:31:09.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:09 smithi122 ceph-mon[73495]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-20T09:31:09.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:09 smithi122 ceph-mon[73495]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-20T09:31:09.773 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:31:09.805 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:09.826 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:31:09.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:09.829 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:31:09.874 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:31:09.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:09 smithi115 ceph-mon[72476]: pgmap v2108: 47 pgs: 1 peering, 4 creating+peering, 9 unknown, 33 active+clean; 577 KiB data, 6.9 GiB used, 977 GiB / 983 GiB avail 2024-03-20T09:31:09.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:09 smithi115 ceph-mon[72476]: osdmap e873: 12 total, 12 up, 11 in 2024-03-20T09:31:09.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:09 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3586082786' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:31:09.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:09 smithi115 ceph-mon[72476]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-20T09:31:09.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:09 smithi115 ceph-mon[72476]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-20T09:31:09.930 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:31:09.962 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:09.984 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:31:09.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:09.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:31:10.088 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:31:10.117 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:10.265 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:31:10.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:10.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:31:10.286 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-20T09:31:10.369 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:31:10.402 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:10.424 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:31:10.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:10.428 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:31:10.528 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:31:10.556 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:10.754 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:31:10.756 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:31:10.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:10.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:31:10.859 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:31:10.891 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:10.913 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:31:10.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:10.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:31:11.018 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:31:11.051 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:11.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:11 smithi122 ceph-mon[73495]: pgmap v2110: 47 pgs: 1 peering, 4 creating+peering, 5 unknown, 37 active+clean; 577 KiB data, 6.9 GiB used, 977 GiB / 983 GiB avail 2024-03-20T09:31:11.702 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:11 smithi026 ceph-mon[68309]: pgmap v2110: 47 pgs: 1 peering, 4 creating+peering, 5 unknown, 37 active+clean; 577 KiB data, 6.9 GiB used, 977 GiB / 983 GiB avail 2024-03-20T09:31:11.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:11 smithi115 ceph-mon[72476]: pgmap v2110: 47 pgs: 1 peering, 4 creating+peering, 5 unknown, 37 active+clean; 577 KiB data, 6.9 GiB used, 977 GiB / 983 GiB avail 2024-03-20T09:31:12.795 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:31:12.811 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:12 smithi026 ceph-mon[68309]: pgmap v2111: 47 pgs: 47 active+clean; 590 KiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 511 B/s rd, 11 KiB/s wr, 1 op/s; 3.9 KiB/s, 0 objects/s recovering 2024-03-20T09:31:12.812 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:12 smithi026 ceph-mon[68309]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-20T09:31:12.877 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:31:12.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:12 smithi115 ceph-mon[72476]: pgmap v2111: 47 pgs: 47 active+clean; 590 KiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 511 B/s rd, 11 KiB/s wr, 1 op/s; 3.9 KiB/s, 0 objects/s recovering 2024-03-20T09:31:12.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:12 smithi115 ceph-mon[72476]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-20T09:31:12.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:12 smithi122 ceph-mon[73495]: pgmap v2111: 47 pgs: 47 active+clean; 590 KiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 511 B/s rd, 11 KiB/s wr, 1 op/s; 3.9 KiB/s, 0 objects/s recovering 2024-03-20T09:31:12.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:12 smithi122 ceph-mon[73495]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-20T09:31:13.102 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:31:13.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:13.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:31:13.206 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:31:13.239 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:13.259 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:31:13.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:13.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:31:13.364 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:31:13.393 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:13.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:13 smithi026 ceph-mon[68309]: osdmap e874: 12 total, 12 up, 11 in 2024-03-20T09:31:13.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:13 smithi115 ceph-mon[72476]: osdmap e874: 12 total, 12 up, 11 in 2024-03-20T09:31:13.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:13 smithi122 ceph-mon[73495]: osdmap e874: 12 total, 12 up, 11 in 2024-03-20T09:31:14.577 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:31:14.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:14.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:31:14.600 INFO:teuthology.orchestra.run.smithi026.stderr:enabled application 'rados' on pool 'unique_pool_5' 2024-03-20T09:31:14.663 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 8, 11, 3, 4, 9] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 11, 5, 10] 2024-03-20T09:31:14.663 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:31:14.682 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:31:14.714 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:14.736 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:31:14.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:14.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:31:14.840 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:31:14.868 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:14 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/131689090' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-20T09:31:14.868 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:14 smithi115 ceph-mon[72476]: pgmap v2113: 47 pgs: 47 active+clean; 590 KiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 442 B/s rd, 9.7 KiB/s wr, 1 op/s; 3.4 KiB/s, 0 objects/s recovering 2024-03-20T09:31:14.870 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:14.890 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:31:14.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:14.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:31:14.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:14 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/131689090' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-20T09:31:14.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:14 smithi122 ceph-mon[73495]: pgmap v2113: 47 pgs: 47 active+clean; 590 KiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 442 B/s rd, 9.7 KiB/s wr, 1 op/s; 3.4 KiB/s, 0 objects/s recovering 2024-03-20T09:31:14.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:14 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/131689090' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-20T09:31:14.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:14 smithi026 ceph-mon[68309]: pgmap v2113: 47 pgs: 47 active+clean; 590 KiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 442 B/s rd, 9.7 KiB/s wr, 1 op/s; 3.4 KiB/s, 0 objects/s recovering 2024-03-20T09:31:14.994 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:31:15.028 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:15.050 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:31:15.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:15.053 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:31:15.154 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:31:15.187 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:15.208 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:31:15.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:15.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:31:15.224 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:31:15.314 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:31:15.346 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:15.367 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:31:15.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:15.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:31:15.440 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:31:15.440 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:31:15.471 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:31:15.499 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:15.601 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:31:15.653 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:31:15.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:15.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:31:15.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:15 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/131689090' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-20T09:31:15.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:15 smithi026 ceph-mon[68309]: osdmap e875: 12 total, 12 up, 11 in 2024-03-20T09:31:15.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:15 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3298264063' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:31:15.758 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:31:15.790 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:15.811 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:31:15.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:15.814 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:31:15.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:15 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/131689090' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-20T09:31:15.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:15 smithi115 ceph-mon[72476]: osdmap e875: 12 total, 12 up, 11 in 2024-03-20T09:31:15.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:15 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3298264063' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:31:15.915 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:31:15.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:15 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/131689090' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-20T09:31:15.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:15 smithi122 ceph-mon[73495]: osdmap e875: 12 total, 12 up, 11 in 2024-03-20T09:31:15.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:15 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3298264063' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:31:15.948 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:15.969 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:31:15.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:15.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:31:16.073 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:31:16.105 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:16.126 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:31:16.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:16.129 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:31:16.230 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:31:16.262 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:16.284 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:31:16.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:16.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:31:16.389 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:31:16.418 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:16.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:16 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:31:16.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:16 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3298264063' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:31:16.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:16 smithi026 ceph-mon[68309]: osdmap e876: 12 total, 12 up, 11 in 2024-03-20T09:31:16.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:16 smithi026 ceph-mon[68309]: pgmap v2116: 47 pgs: 47 active+clean; 590 KiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 511 B/s rd, 11 KiB/s wr, 1 op/s; 3.9 KiB/s, 0 objects/s recovering 2024-03-20T09:31:16.870 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:31:16.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:16.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:31:16.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:16 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:31:16.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:16 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3298264063' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:31:16.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:16 smithi115 ceph-mon[72476]: osdmap e876: 12 total, 12 up, 11 in 2024-03-20T09:31:16.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:16 smithi115 ceph-mon[72476]: pgmap v2116: 47 pgs: 47 active+clean; 590 KiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 511 B/s rd, 11 KiB/s wr, 1 op/s; 3.9 KiB/s, 0 objects/s recovering 2024-03-20T09:31:16.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:16 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:31:16.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:16 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3298264063' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:31:16.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:16 smithi122 ceph-mon[73495]: osdmap e876: 12 total, 12 up, 11 in 2024-03-20T09:31:16.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:16 smithi122 ceph-mon[73495]: pgmap v2116: 47 pgs: 47 active+clean; 590 KiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 511 B/s rd, 11 KiB/s wr, 1 op/s; 3.9 KiB/s, 0 objects/s recovering 2024-03-20T09:31:16.974 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:31:17.003 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:17.553 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:31:17.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:17.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:31:17.659 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:31:17.689 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:17.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:17 smithi115 ceph-mon[72476]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-20T09:31:17.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:17 smithi122 ceph-mon[73495]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-20T09:31:17.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:17 smithi026 ceph-mon[68309]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-20T09:31:18.087 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-20T09:31:18.340 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:31:18.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:18.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:31:18.438 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:31:18.445 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:31:18.474 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:18.578 DEBUG:teuthology.orchestra.run.smithi122:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_5 bench 90 write' 2024-03-20T09:31:18.579 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_5 2024-03-20T09:31:18.579 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:31:18.582 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-20T09:31:18.582 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-20T09:31:18.609 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:31:18.632 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:31:18.638 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:hints = 1 2024-03-20T09:31:18.638 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-20T09:31:18.638 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Object prefix: benchmark_data_smithi122_187077 2024-03-20T09:31:18.639 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T09:31:18.639 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 0 0 0 0 0 0 - 0 2024-03-20T09:31:18.667 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:31:18.704 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:31:18.722 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:31:18.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:18 smithi115 ceph-mon[72476]: pgmap v2117: 47 pgs: 47 active+clean; 590 KiB data, 6.8 GiB used, 977 GiB / 983 GiB avail 2024-03-20T09:31:18.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:18 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2285225369' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:31:18.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:18.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:31:18.729 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:31:18.750 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:31:18.771 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:31:18.791 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:31:18.811 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:31:18.827 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:31:18.834 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:31:18.865 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:18 smithi026 ceph-mon[68309]: pgmap v2117: 47 pgs: 47 active+clean; 590 KiB data, 6.8 GiB used, 977 GiB / 983 GiB avail 2024-03-20T09:31:18.865 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:18 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2285225369' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:31:18.867 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:31:18.868 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:18.904 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:31:18.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:18.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:31:18.916 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:31:18.925 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:31:18.925 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:31:18.925 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:31:18.925 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:31:18.925 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:31:18.925 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:31:18.925 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:31:18.925 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:31:18.925 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:31:18.925 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:31:18.926 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:31:18.926 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:31:18.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:18 smithi122 ceph-mon[73495]: pgmap v2117: 47 pgs: 47 active+clean; 590 KiB data, 6.8 GiB used, 977 GiB / 983 GiB avail 2024-03-20T09:31:18.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:18 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2285225369' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:31:18.944 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:31:19.011 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:31:19.039 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:19.112 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:31:19.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:19.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:31:19.216 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:31:19.255 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:19.364 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:31:19.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:19.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:31:19.469 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:31:19.508 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:19.638 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 1 16 2604 2588 161.741 161.75 0.00315981 0.00616418 2024-03-20T09:31:19.962 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:31:19.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:19.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:31:20.013 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:31:20.068 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:31:20.108 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:20.264 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:31:20.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:20.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:31:20.370 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:31:20.425 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:20.638 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 2 16 6367 6351 198.456 235.188 0.00473879 0.00502569 2024-03-20T09:31:20.685 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:31:20.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:20.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:31:20.790 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:31:20.791 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:31:20.791 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [ 2024-03-20T09:31:20.791 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:20.791 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:6809 7.1 7:857f381c:::benchmark_data_smithi122_187077_object6808:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e876)", 2024-03-20T09:31:20.791 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:31:20.794311+0000", 2024-03-20T09:31:20.791 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.0030173130000000002, 2024-03-20T09:31:20.791 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0030530890000000001, 2024-03-20T09:31:20.792 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:20.792 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:31:20.792 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:31:20.792 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:31:20.792 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:31:20.792 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 6809 2024-03-20T09:31:20.792 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:20.792 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:20.793 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:20.793 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:20.793 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:20.794311+0000", 2024-03-20T09:31:20.793 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:20.793 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:20.793 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:20.793 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:20.793 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:20.794311+0000", 2024-03-20T09:31:20.794 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:20.794 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:20.794 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:20.794 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:20.794 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:20.794311+0000", 2024-03-20T09:31:20.794 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.0299999999999998e-07 2024-03-20T09:31:20.794 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:20.795 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:20.795 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:20.795 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:20.794409+0000", 2024-03-20T09:31:20.795 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.7942000000000005e-05 2024-03-20T09:31:20.795 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:20.795 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:20.795 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:20.796 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:20.794418+0000", 2024-03-20T09:31:20.796 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.3000000000000002e-06 2024-03-20T09:31:20.796 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:20.796 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:20.796 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:20.797 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:20.794425+0000", 2024-03-20T09:31:20.797 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.8349999999999998e-06 2024-03-20T09:31:20.797 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:20.797 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:20.797 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:20.797 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:20.794487+0000", 2024-03-20T09:31:20.797 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2730000000000004e-05 2024-03-20T09:31:20.798 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:20.798 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:20.798 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:20.798 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:20.794631+0000", 2024-03-20T09:31:20.798 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00014360100000000001 2024-03-20T09:31:20.798 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:20.798 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:20.799 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 3,6", 2024-03-20T09:31:20.799 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:20.794792+0000", 2024-03-20T09:31:20.799 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000161538 2024-03-20T09:31:20.799 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:20.799 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:20.799 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:31:20.799 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:20.795428+0000", 2024-03-20T09:31:20.799 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00063586399999999998 2024-03-20T09:31:20.800 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:20.800 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:20.800 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:20.800 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:20.800 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:20.800 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:6820 7.1 7:823806f5:::benchmark_data_smithi122_187077_object6819:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e876)", 2024-03-20T09:31:20.800 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:31:20.795856+0000", 2024-03-20T09:31:20.801 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.0014720250000000001, 2024-03-20T09:31:20.801 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0016176439999999999, 2024-03-20T09:31:20.801 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:20.801 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:31:20.801 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:31:20.801 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:31:20.801 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:31:20.802 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 6820 2024-03-20T09:31:20.802 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:20.802 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:20.802 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:20.802 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:20.802 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:20.795856+0000", 2024-03-20T09:31:20.803 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:20.803 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:20.803 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:20.803 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:20.803 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:20.795856+0000", 2024-03-20T09:31:20.803 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:20.804 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:20.804 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:20.804 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:20.804 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:20.795857+0000", 2024-03-20T09:31:20.804 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.9599999999999998e-07 2024-03-20T09:31:20.804 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:20.804 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:20.804 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:20.804 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:20.795923+0000", 2024-03-20T09:31:20.804 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.6549000000000003e-05 2024-03-20T09:31:20.804 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:20.804 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:20.804 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:20.804 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:20.795937+0000", 2024-03-20T09:31:20.805 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3750999999999999e-05 2024-03-20T09:31:20.805 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:20.805 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:20.805 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:20.805 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:20.795946+0000", 2024-03-20T09:31:20.805 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.8189999999999998e-06 2024-03-20T09:31:20.805 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:20.805 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:20.805 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:20.805 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:20.795998+0000", 2024-03-20T09:31:20.805 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2002000000000001e-05 2024-03-20T09:31:20.805 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:20.805 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:20.805 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:20.806 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:20.796117+0000", 2024-03-20T09:31:20.806 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000119191 2024-03-20T09:31:20.806 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:20.806 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:20.806 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 3,6", 2024-03-20T09:31:20.806 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:20.796260+0000", 2024-03-20T09:31:20.806 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00014255299999999999 2024-03-20T09:31:20.806 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:20.806 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:20.806 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:31:20.806 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:20.796837+0000", 2024-03-20T09:31:20.806 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00057749799999999997 2024-03-20T09:31:20.806 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:20.807 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:20.807 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:20.807 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:20.807 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:20.807 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop_reply(client.26605.0:6820 7.1 e876/871)", 2024-03-20T09:31:20.807 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:31:20.797509+0000", 2024-03-20T09:31:20.807 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 4294967295.9998193, 2024-03-20T09:31:20.807 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.7550999999999999e-05, 2024-03-20T09:31:20.807 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:20.807 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "queued for pg", 2024-03-20T09:31:20.807 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:20.807 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:20.807 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:20.807 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:20.797509+0000", 2024-03-20T09:31:20.808 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:20.808 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:20.808 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:20.808 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:20.808 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:20.797509+0000", 2024-03-20T09:31:20.808 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:20.808 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:20.808 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:20.808 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:20.808 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:20.797509+0000", 2024-03-20T09:31:20.808 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6100000000000001e-07 2024-03-20T09:31:20.808 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:20.808 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:20.808 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:20.809 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:20.797514+0000", 2024-03-20T09:31:20.809 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.0640000000000004e-06 2024-03-20T09:31:20.809 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:20.809 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:20.809 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:20.809 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:20.797523+0000", 2024-03-20T09:31:20.809 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.3570000000000004e-06 2024-03-20T09:31:20.809 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:20.809 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:20.809 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:20.809 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:20.797529+0000", 2024-03-20T09:31:20.809 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9649999999999996e-06 2024-03-20T09:31:20.809 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:20.809 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:20.810 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:20.810 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:20.810 INFO:teuthology.orchestra.run.smithi115.stdout: ], 2024-03-20T09:31:20.810 INFO:teuthology.orchestra.run.smithi115.stdout: "num_ops": 3 2024-03-20T09:31:20.810 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:31:20.851 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:21.359 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:31:21.412 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:21 smithi026 ceph-mon[68309]: pgmap v2118: 47 pgs: 47 active+clean; 59 MiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 7.5 MiB/s wr, 120 op/s 2024-03-20T09:31:21.638 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 3 16 9579 9563 199.215 200.75 0.0036244 0.00500802 2024-03-20T09:31:21.688 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:21 smithi122 ceph-mon[73495]: pgmap v2118: 47 pgs: 47 active+clean; 59 MiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 7.5 MiB/s wr, 120 op/s 2024-03-20T09:31:21.770 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:31:21.772 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:31:21.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:21 smithi115 ceph-mon[72476]: pgmap v2118: 47 pgs: 47 active+clean; 59 MiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 7.5 MiB/s wr, 120 op/s 2024-03-20T09:31:22.035 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:31:22.510 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:31:22.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:22.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:31:22.614 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:31:22.638 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 4 16 12137 12121 189.377 159.875 0.00384858 0.00527253 2024-03-20T09:31:22.666 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:22.984 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:31:22.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:22.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:31:23.044 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":2118,"stamp":"2024-03-20T09:31:20.355302+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":61748874,"num_objects":935,"num_object_clones":0,"num_object_copies":2805,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":935,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":938,"num_write_kb":59779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":351723,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5181,"ondisk_log_size":5181,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":143,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":7330072,"kb_used_data":423060,"kb_used_omap":197,"kb_used_meta":6906682,"kb_avail":1023866600,"statfs":{"total":1055945392128,"available":1048439398400,"internally_reserved":0,"allocated":433213440,"data_stored":424986640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202077,"internal_metadata":7072443043},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":61145088,"num_objects":933,"num_object_clones":0,"num_object_copies":2799,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":933,"num_write_kb":59712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.757220"},"pg_stats":[{"pgid":"1.1e","version":"20'137","reported_seq":1173,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-03-20T09:31:15.012230+0000","last_change":"2024-03-20T09:25:33.135367+0000","last_active":"2024-03-20T09:31:15.012230+0000","last_peered":"2024-03-20T09:31:15.012230+0000","last_clean":"2024-03-20T09:31:15.012230+0000","last_became_active":"2024-03-20T09:13:20.071655+0000","last_became_peered":"2024-03-20T09:13:20.071655+0000","last_unstale":"2024-03-20T09:31:15.012230+0000","last_undegraded":"2024-03-20T09:31:15.012230+0000","last_fullsized":"2024-03-20T09:31:15.012230+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:33.135222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:25:33.135222+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'137","reported_seq":252,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-03-20T09:31:15.012470+0000","last_change":"2024-03-20T09:31:07.425985+0000","last_active":"2024-03-20T09:31:15.012470+0000","last_peered":"2024-03-20T09:31:15.012470+0000","last_clean":"2024-03-20T09:31:15.012470+0000","last_became_active":"2024-03-20T09:31:07.425574+0000","last_became_peered":"2024-03-20T09:31:07.425574+0000","last_unstale":"2024-03-20T09:31:15.012470+0000","last_undegraded":"2024-03-20T09:31:15.012470+0000","last_fullsized":"2024-03-20T09:31:15.012470+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:30:36.629879+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:29:02.585165+0000","last_clean_scrub_stamp":"2024-03-20T09:30:36.629879+0000","objects_scrubbed":2,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:31:17.548433+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":5,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":24043,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2],"acting":[11,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"20'137","reported_seq":1845,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-03-20T09:31:15.012008+0000","last_change":"2024-03-20T09:30:52.063171+0000","last_active":"2024-03-20T09:31:15.012008+0000","last_peered":"2024-03-20T09:31:15.012008+0000","last_clean":"2024-03-20T09:31:15.012008+0000","last_became_active":"2024-03-20T09:25:35.482959+0000","last_became_peered":"2024-03-20T09:25:35.482959+0000","last_unstale":"2024-03-20T09:31:15.012008+0000","last_undegraded":"2024-03-20T09:31:15.012008+0000","last_fullsized":"2024-03-20T09:31:15.012008+0000","mapping_epoch":744,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":745,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:30:52.063063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:30:52.063063+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:32:19.218094+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,1],"acting":[8,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.19","version":"20'137","reported_seq":1028,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-03-20T09:31:16.378336+0000","last_change":"2024-03-20T09:30:56.201664+0000","last_active":"2024-03-20T09:31:16.378336+0000","last_peered":"2024-03-20T09:31:16.378336+0000","last_clean":"2024-03-20T09:31:16.378336+0000","last_became_active":"2024-03-20T09:28:22.591294+0000","last_became_peered":"2024-03-20T09:28:22.591294+0000","last_unstale":"2024-03-20T09:31:16.378336+0000","last_undegraded":"2024-03-20T09:31:16.378336+0000","last_fullsized":"2024-03-20T09:31:16.378336+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:30:56.201554+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:30:56.201554+0000","last_clean_scrub_stamp":"2024-03-20T09:30:56.201554+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:32:19.146369+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.18","version":"20'137","reported_seq":1322,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-03-20T09:31:15.647695+0000","last_change":"2024-03-20T09:30:49.297024+0000","last_active":"2024-03-20T09:31:15.647695+0000","last_peered":"2024-03-20T09:31:15.647695+0000","last_clean":"2024-03-20T09:31:15.647695+0000","last_became_active":"2024-03-20T09:10:35.277481+0000","last_became_peered":"2024-03-20T09:10:35.277481+0000","last_unstale":"2024-03-20T09:31:15.647695+0000","last_undegraded":"2024-03-20T09:31:15.647695+0000","last_fullsized":"2024-03-20T09:31:15.647695+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:30:49.296822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:30:49.296822+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:32:01.217697+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1b","version":"20'137","reported_seq":1674,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-03-20T09:31:15.012313+0000","last_change":"2024-03-20T09:29:56.286401+0000","last_active":"2024-03-20T09:31:15.012313+0000","last_peered":"2024-03-20T09:31:15.012313+0000","last_clean":"2024-03-20T09:31:15.012313+0000","last_became_active":"2024-03-20T09:24:16.445287+0000","last_became_peered":"2024-03-20T09:24:16.445287+0000","last_unstale":"2024-03-20T09:31:15.012313+0000","last_undegraded":"2024-03-20T09:31:15.012313+0000","last_fullsized":"2024-03-20T09:31:15.012313+0000","mapping_epoch":718,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:29:56.286318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:29:56.286318+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:31:19.175037+0000","scrub_duration":126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":20,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-03-20T09:31:15.607725+0000","last_change":"2024-03-20T09:31:07.427385+0000","last_active":"2024-03-20T09:31:15.607725+0000","last_peered":"2024-03-20T09:31:15.607725+0000","last_clean":"2024-03-20T09:31:15.607725+0000","last_became_active":"2024-03-20T09:31:07.427099+0000","last_became_peered":"2024-03-20T09:31:07.427099+0000","last_unstale":"2024-03-20T09:31:15.607725+0000","last_undegraded":"2024-03-20T09:31:15.607725+0000","last_fullsized":"2024-03-20T09:31:15.607725+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:31:06.406431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:32:09.941334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,5],"acting":[2,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"20'137","reported_seq":1322,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-03-20T09:31:15.647914+0000","last_change":"2024-03-20T09:30:51.268772+0000","last_active":"2024-03-20T09:31:15.647914+0000","last_peered":"2024-03-20T09:31:15.647914+0000","last_clean":"2024-03-20T09:31:15.647914+0000","last_became_active":"2024-03-20T09:10:35.215772+0000","last_became_peered":"2024-03-20T09:10:35.215772+0000","last_unstale":"2024-03-20T09:31:15.647914+0000","last_undegraded":"2024-03-20T09:31:15.647914+0000","last_fullsized":"2024-03-20T09:31:15.647914+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:30:51.268613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:30:51.268613+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:32:17.021492+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":20,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-03-20T09:31:16.378515+0000","last_change":"2024-03-20T09:31:07.535032+0000","last_active":"2024-03-20T09:31:16.378515+0000","last_peered":"2024-03-20T09:31:16.378515+0000","last_clean":"2024-03-20T09:31:16.378515+0000","last_became_active":"2024-03-20T09:31:07.534696+0000","last_became_peered":"2024-03-20T09:31:07.534696+0000","last_unstale":"2024-03-20T09:31:16.378515+0000","last_undegraded":"2024-03-20T09:31:16.378515+0000","last_fullsized":"2024-03-20T09:31:16.378515+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:31:06.406431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:32:09.851960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"20'137","reported_seq":1906,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-03-20T09:31:16.750256+0000","last_change":"2024-03-20T09:30:36.462054+0000","last_active":"2024-03-20T09:31:16.750256+0000","last_peered":"2024-03-20T09:31:16.750256+0000","last_clean":"2024-03-20T09:31:16.750256+0000","last_became_active":"2024-03-20T09:20:25.341750+0000","last_became_peered":"2024-03-20T09:20:25.341750+0000","last_unstale":"2024-03-20T09:31:16.750256+0000","last_undegraded":"2024-03-20T09:31:16.750256+0000","last_fullsized":"2024-03-20T09:31:16.750256+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:30:36.461893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:30:36.461893+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:31:53.251226+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,2],"acting":[9,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":18,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-03-20T09:31:14.611024+0000","last_change":"2024-03-20T09:31:07.427370+0000","last_active":"2024-03-20T09:31:14.611024+0000","last_peered":"2024-03-20T09:31:14.611024+0000","last_clean":"2024-03-20T09:31:14.611024+0000","last_became_active":"2024-03-20T09:31:07.426984+0000","last_became_peered":"2024-03-20T09:31:07.426984+0000","last_unstale":"2024-03-20T09:31:14.611024+0000","last_undegraded":"2024-03-20T09:31:14.611024+0000","last_fullsized":"2024-03-20T09:31:14.611024+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:31:06.406431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:32:12.561820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,11],"acting":[6,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"20'137","reported_seq":595,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-03-20T09:31:16.749942+0000","last_change":"2024-03-20T09:29:55.502951+0000","last_active":"2024-03-20T09:31:16.749942+0000","last_peered":"2024-03-20T09:31:16.749942+0000","last_clean":"2024-03-20T09:31:16.749942+0000","last_became_active":"2024-03-20T09:24:16.444232+0000","last_became_peered":"2024-03-20T09:24:16.444232+0000","last_unstale":"2024-03-20T09:31:16.749942+0000","last_undegraded":"2024-03-20T09:31:16.749942+0000","last_fullsized":"2024-03-20T09:31:16.749942+0000","mapping_epoch":718,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:29:55.502833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:29:55.502833+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:31:17.880148+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.2","version":"876'323","reported_seq":343,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-03-20T09:31:20.186296+0000","last_change":"2024-03-20T09:31:07.422586+0000","last_active":"2024-03-20T09:31:20.186296+0000","last_peered":"2024-03-20T09:31:20.186296+0000","last_clean":"2024-03-20T09:31:20.186296+0000","last_became_active":"2024-03-20T09:31:07.422232+0000","last_became_peered":"2024-03-20T09:31:07.422232+0000","last_unstale":"2024-03-20T09:31:20.186296+0000","last_undegraded":"2024-03-20T09:31:20.186296+0000","last_fullsized":"2024-03-20T09:31:20.186296+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:31:06.406431+0000","objects_scrubbed":0,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:32:09.407541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21168128,"num_objects":323,"num_object_clones":0,"num_object_copies":969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":323,"num_write_kb":20672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"20'137","reported_seq":1842,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-03-20T09:31:15.012747+0000","last_change":"2024-03-20T09:25:34.098294+0000","last_active":"2024-03-20T09:31:15.012747+0000","last_peered":"2024-03-20T09:31:15.012747+0000","last_clean":"2024-03-20T09:31:15.012747+0000","last_became_active":"2024-03-20T09:12:14.306384+0000","last_became_peered":"2024-03-20T09:12:14.306384+0000","last_unstale":"2024-03-20T09:31:15.012747+0000","last_undegraded":"2024-03-20T09:31:15.012747+0000","last_fullsized":"2024-03-20T09:31:15.012747+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:34.098199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:25:34.098199+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.3","version":"876'280","reported_seq":300,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-03-20T09:31:20.192115+0000","last_change":"2024-03-20T09:31:07.422816+0000","last_active":"2024-03-20T09:31:20.192115+0000","last_peered":"2024-03-20T09:31:20.192115+0000","last_clean":"2024-03-20T09:31:20.192115+0000","last_became_active":"2024-03-20T09:31:07.422509+0000","last_became_peered":"2024-03-20T09:31:07.422509+0000","last_unstale":"2024-03-20T09:31:20.192115+0000","last_undegraded":"2024-03-20T09:31:20.192115+0000","last_fullsized":"2024-03-20T09:31:20.192115+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:31:06.406431+0000","objects_scrubbed":0,"log_size":280,"log_dups_size":0,"ondisk_log_size":280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:32:17.499856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18350080,"num_objects":280,"num_object_clones":0,"num_object_copies":840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":280,"num_write_kb":17920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"20'137","reported_seq":499,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-03-20T09:31:15.607854+0000","last_change":"2024-03-20T09:30:56.243461+0000","last_active":"2024-03-20T09:31:15.607854+0000","last_peered":"2024-03-20T09:31:15.607854+0000","last_clean":"2024-03-20T09:31:15.607854+0000","last_became_active":"2024-03-20T09:28:22.587095+0000","last_became_peered":"2024-03-20T09:28:22.587095+0000","last_unstale":"2024-03-20T09:31:15.607854+0000","last_undegraded":"2024-03-20T09:31:15.607854+0000","last_fullsized":"2024-03-20T09:31:15.607854+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:30:56.243352+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:19:03.805185+0000","last_clean_scrub_stamp":"2024-03-20T09:30:56.243352+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:32:25.882802+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,11],"acting":[2,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.0","version":"876'159","reported_seq":179,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-03-20T09:31:19.718393+0000","last_change":"2024-03-20T09:31:07.426655+0000","last_active":"2024-03-20T09:31:19.718393+0000","last_peered":"2024-03-20T09:31:19.718393+0000","last_clean":"2024-03-20T09:31:19.718393+0000","last_became_active":"2024-03-20T09:31:07.426307+0000","last_became_peered":"2024-03-20T09:31:07.426307+0000","last_unstale":"2024-03-20T09:31:19.718393+0000","last_undegraded":"2024-03-20T09:31:19.718393+0000","last_fullsized":"2024-03-20T09:31:19.718393+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:31:06.406431+0000","objects_scrubbed":0,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:32:23.483455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10420224,"num_objects":159,"num_object_clones":0,"num_object_copies":477,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":159,"num_write_kb":10176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"20'137","reported_seq":1710,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-03-20T09:31:16.749987+0000","last_change":"2024-03-20T09:30:39.314485+0000","last_active":"2024-03-20T09:31:16.749987+0000","last_peered":"2024-03-20T09:31:16.749987+0000","last_clean":"2024-03-20T09:31:16.749987+0000","last_became_active":"2024-03-20T09:28:22.586751+0000","last_became_peered":"2024-03-20T09:28:22.586751+0000","last_unstale":"2024-03-20T09:31:16.749987+0000","last_undegraded":"2024-03-20T09:31:16.749987+0000","last_fullsized":"2024-03-20T09:31:16.749987+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:30:39.314345+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:24:31.549603+0000","last_clean_scrub_stamp":"2024-03-20T09:30:39.314345+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:31:44.873985+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":18,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-03-20T09:31:15.012475+0000","last_change":"2024-03-20T09:31:07.427441+0000","last_active":"2024-03-20T09:31:15.012475+0000","last_peered":"2024-03-20T09:31:15.012475+0000","last_clean":"2024-03-20T09:31:15.012475+0000","last_became_active":"2024-03-20T09:31:07.427044+0000","last_became_peered":"2024-03-20T09:31:07.427044+0000","last_unstale":"2024-03-20T09:31:15.012475+0000","last_undegraded":"2024-03-20T09:31:15.012475+0000","last_fullsized":"2024-03-20T09:31:15.012475+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:31:06.406431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:32:32.366053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,0],"acting":[11,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.3","version":"20'137","reported_seq":908,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-03-20T09:31:14.610852+0000","last_change":"2024-03-20T09:31:07.424970+0000","last_active":"2024-03-20T09:31:14.610852+0000","last_peered":"2024-03-20T09:31:14.610852+0000","last_clean":"2024-03-20T09:31:14.610852+0000","last_became_active":"2024-03-20T09:31:07.424588+0000","last_became_peered":"2024-03-20T09:31:07.424588+0000","last_unstale":"2024-03-20T09:31:14.610852+0000","last_undegraded":"2024-03-20T09:31:14.610852+0000","last_fullsized":"2024-03-20T09:31:14.610852+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:30:47.812125+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:30:47.812125+0000","last_clean_scrub_stamp":"2024-03-20T09:30:47.812125+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:31:59.238892+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":20,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-03-20T09:31:16.378465+0000","last_change":"2024-03-20T09:31:07.535369+0000","last_active":"2024-03-20T09:31:16.378465+0000","last_peered":"2024-03-20T09:31:16.378465+0000","last_clean":"2024-03-20T09:31:16.378465+0000","last_became_active":"2024-03-20T09:31:07.535053+0000","last_became_peered":"2024-03-20T09:31:07.535053+0000","last_unstale":"2024-03-20T09:31:16.378465+0000","last_undegraded":"2024-03-20T09:31:16.378465+0000","last_fullsized":"2024-03-20T09:31:16.378465+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:31:06.406431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:32:10.152858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":1813,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-03-20T09:31:15.012398+0000","last_change":"2024-03-20T09:25:31.128237+0000","last_active":"2024-03-20T09:31:15.012398+0000","last_peered":"2024-03-20T09:31:15.012398+0000","last_clean":"2024-03-20T09:31:15.012398+0000","last_became_active":"2024-03-20T09:12:14.305989+0000","last_became_peered":"2024-03-20T09:12:14.305989+0000","last_unstale":"2024-03-20T09:31:15.012398+0000","last_undegraded":"2024-03-20T09:31:15.012398+0000","last_fullsized":"2024-03-20T09:31:15.012398+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:25:31.128133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:25:31.128133+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":20,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-03-20T09:31:16.378404+0000","last_change":"2024-03-20T09:31:07.425178+0000","last_active":"2024-03-20T09:31:16.378404+0000","last_peered":"2024-03-20T09:31:16.378404+0000","last_clean":"2024-03-20T09:31:16.378404+0000","last_became_active":"2024-03-20T09:31:07.424820+0000","last_became_peered":"2024-03-20T09:31:07.424820+0000","last_unstale":"2024-03-20T09:31:16.378404+0000","last_undegraded":"2024-03-20T09:31:16.378404+0000","last_fullsized":"2024-03-20T09:31:16.378404+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:31:06.406431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:32:15.379615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"20'137","reported_seq":2079,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-03-20T09:31:16.378364+0000","last_change":"2024-03-20T09:31:00.255741+0000","last_active":"2024-03-20T09:31:16.378364+0000","last_peered":"2024-03-20T09:31:16.378364+0000","last_clean":"2024-03-20T09:31:16.378364+0000","last_became_active":"2024-03-20T09:19:54.492879+0000","last_became_peered":"2024-03-20T09:19:54.492879+0000","last_unstale":"2024-03-20T09:31:16.378364+0000","last_undegraded":"2024-03-20T09:31:16.378364+0000","last_fullsized":"2024-03-20T09:31:16.378364+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:31:00.255615+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:19.513511+0000","last_clean_scrub_stamp":"2024-03-20T09:31:00.255615+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:32:10.745753+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":18,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-03-20T09:31:14.611054+0000","last_change":"2024-03-20T09:31:07.535035+0000","last_active":"2024-03-20T09:31:14.611054+0000","last_peered":"2024-03-20T09:31:14.611054+0000","last_clean":"2024-03-20T09:31:14.611054+0000","last_became_active":"2024-03-20T09:31:07.534591+0000","last_became_peered":"2024-03-20T09:31:07.534591+0000","last_unstale":"2024-03-20T09:31:14.611054+0000","last_undegraded":"2024-03-20T09:31:14.611054+0000","last_fullsized":"2024-03-20T09:31:14.611054+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:31:06.406431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:32:35.119483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,3],"acting":[6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"20'137","reported_seq":1254,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-03-20T09:31:14.610799+0000","last_change":"2024-03-20T09:30:34.095244+0000","last_active":"2024-03-20T09:31:14.610799+0000","last_peered":"2024-03-20T09:31:14.610799+0000","last_clean":"2024-03-20T09:31:14.610799+0000","last_became_active":"2024-03-20T09:15:34.225022+0000","last_became_peered":"2024-03-20T09:15:34.225022+0000","last_unstale":"2024-03-20T09:31:14.610799+0000","last_undegraded":"2024-03-20T09:31:14.610799+0000","last_fullsized":"2024-03-20T09:31:14.610799+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:30:34.095074+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:20:12.169658+0000","last_clean_scrub_stamp":"2024-03-20T09:30:34.095074+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:31:52.818402+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,5],"acting":[6,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":18,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-03-20T09:31:15.011253+0000","last_change":"2024-03-20T09:31:07.422259+0000","last_active":"2024-03-20T09:31:15.011253+0000","last_peered":"2024-03-20T09:31:15.011253+0000","last_clean":"2024-03-20T09:31:15.011253+0000","last_became_active":"2024-03-20T09:31:07.421909+0000","last_became_peered":"2024-03-20T09:31:07.421909+0000","last_unstale":"2024-03-20T09:31:15.011253+0000","last_undegraded":"2024-03-20T09:31:15.011253+0000","last_fullsized":"2024-03-20T09:31:15.011253+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:31:06.406431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:32:32.053899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"20'137","reported_seq":1793,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-03-20T09:31:14.611176+0000","last_change":"2024-03-20T09:30:00.027098+0000","last_active":"2024-03-20T09:31:14.611176+0000","last_peered":"2024-03-20T09:31:14.611176+0000","last_clean":"2024-03-20T09:31:14.611176+0000","last_became_active":"2024-03-20T09:28:22.587861+0000","last_became_peered":"2024-03-20T09:28:22.587861+0000","last_unstale":"2024-03-20T09:31:14.611176+0000","last_undegraded":"2024-03-20T09:31:14.611176+0000","last_fullsized":"2024-03-20T09:31:14.611176+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:30:00.026957+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:14:10.560890+0000","last_clean_scrub_stamp":"2024-03-20T09:30:00.026957+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:31:15.822609+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":18,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-03-20T09:31:14.611021+0000","last_change":"2024-03-20T09:31:07.425353+0000","last_active":"2024-03-20T09:31:14.611021+0000","last_peered":"2024-03-20T09:31:14.611021+0000","last_clean":"2024-03-20T09:31:14.611021+0000","last_became_active":"2024-03-20T09:31:07.425062+0000","last_became_peered":"2024-03-20T09:31:07.425062+0000","last_unstale":"2024-03-20T09:31:14.611021+0000","last_undegraded":"2024-03-20T09:31:14.611021+0000","last_fullsized":"2024-03-20T09:31:14.611021+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:31:06.406431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:32:14.854673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"20'137","reported_seq":1625,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-03-20T09:31:15.012859+0000","last_change":"2024-03-20T09:30:54.622489+0000","last_active":"2024-03-20T09:31:15.012859+0000","last_peered":"2024-03-20T09:31:15.012859+0000","last_clean":"2024-03-20T09:31:15.012859+0000","last_became_active":"2024-03-20T09:13:20.070596+0000","last_became_peered":"2024-03-20T09:13:20.070596+0000","last_unstale":"2024-03-20T09:31:15.012859+0000","last_undegraded":"2024-03-20T09:31:15.012859+0000","last_fullsized":"2024-03-20T09:31:15.012859+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:30:54.622386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:30:54.622386+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:32:16.203177+0000","scrub_duration":161,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":20,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-03-20T09:31:15.607559+0000","last_change":"2024-03-20T09:31:07.535259+0000","last_active":"2024-03-20T09:31:15.607559+0000","last_peered":"2024-03-20T09:31:15.607559+0000","last_clean":"2024-03-20T09:31:15.607559+0000","last_became_active":"2024-03-20T09:31:07.534911+0000","last_became_peered":"2024-03-20T09:31:07.534911+0000","last_unstale":"2024-03-20T09:31:15.607559+0000","last_undegraded":"2024-03-20T09:31:15.607559+0000","last_fullsized":"2024-03-20T09:31:15.607559+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:31:06.406431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:32:33.302597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"20'137","reported_seq":1058,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-03-20T09:31:15.011302+0000","last_change":"2024-03-20T09:29:53.362315+0000","last_active":"2024-03-20T09:31:15.011302+0000","last_peered":"2024-03-20T09:31:15.011302+0000","last_clean":"2024-03-20T09:31:15.011302+0000","last_became_active":"2024-03-20T09:19:54.506134+0000","last_became_peered":"2024-03-20T09:19:54.506134+0000","last_unstale":"2024-03-20T09:31:15.011302+0000","last_undegraded":"2024-03-20T09:31:15.011302+0000","last_fullsized":"2024-03-20T09:31:15.011302+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:29:53.362224+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:29:53.362224+0000","last_clean_scrub_stamp":"2024-03-20T09:29:53.362224+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:31:15.151356+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":20,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-03-20T09:31:16.378324+0000","last_change":"2024-03-20T09:31:07.535007+0000","last_active":"2024-03-20T09:31:16.378324+0000","last_peered":"2024-03-20T09:31:16.378324+0000","last_clean":"2024-03-20T09:31:16.378324+0000","last_became_active":"2024-03-20T09:31:07.534654+0000","last_became_peered":"2024-03-20T09:31:07.534654+0000","last_unstale":"2024-03-20T09:31:16.378324+0000","last_undegraded":"2024-03-20T09:31:16.378324+0000","last_fullsized":"2024-03-20T09:31:16.378324+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:31:06.406431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:32:10.161441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"20'137","reported_seq":1822,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-03-20T09:31:14.611134+0000","last_change":"2024-03-20T09:30:59.066144+0000","last_active":"2024-03-20T09:31:14.611134+0000","last_peered":"2024-03-20T09:31:14.611134+0000","last_clean":"2024-03-20T09:31:14.611134+0000","last_became_active":"2024-03-20T09:19:54.524397+0000","last_became_peered":"2024-03-20T09:19:54.524397+0000","last_unstale":"2024-03-20T09:31:14.611134+0000","last_undegraded":"2024-03-20T09:31:14.611134+0000","last_fullsized":"2024-03-20T09:31:14.611134+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:30:59.066038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:30:59.066038+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:32:01.686861+0000","scrub_duration":140,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":20,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-03-20T09:31:16.750077+0000","last_change":"2024-03-20T09:31:07.429355+0000","last_active":"2024-03-20T09:31:16.750077+0000","last_peered":"2024-03-20T09:31:16.750077+0000","last_clean":"2024-03-20T09:31:16.750077+0000","last_became_active":"2024-03-20T09:31:07.428978+0000","last_became_peered":"2024-03-20T09:31:07.428978+0000","last_unstale":"2024-03-20T09:31:16.750077+0000","last_undegraded":"2024-03-20T09:31:16.750077+0000","last_fullsized":"2024-03-20T09:31:16.750077+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:31:06.406431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:32:12.732890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.c","version":"570'138","reported_seq":1800,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-03-20T09:31:16.749060+0000","last_change":"2024-03-20T09:30:50.276360+0000","last_active":"2024-03-20T09:31:16.749060+0000","last_peered":"2024-03-20T09:31:16.749060+0000","last_clean":"2024-03-20T09:31:16.749060+0000","last_became_active":"2024-03-20T09:13:27.087686+0000","last_became_peered":"2024-03-20T09:13:27.087686+0000","last_unstale":"2024-03-20T09:31:16.749060+0000","last_undegraded":"2024-03-20T09:31:16.749060+0000","last_fullsized":"2024-03-20T09:31:16.749060+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"570'138","last_scrub_stamp":"2024-03-20T09:30:50.276218+0000","last_deep_scrub":"570'138","last_deep_scrub_stamp":"2024-03-20T09:16:13.725035+0000","last_clean_scrub_stamp":"2024-03-20T09:30:50.276218+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:32:14.746937+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"876'171","reported_seq":191,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-03-20T09:31:19.716398+0000","last_change":"2024-03-20T09:31:07.425045+0000","last_active":"2024-03-20T09:31:19.716398+0000","last_peered":"2024-03-20T09:31:19.716398+0000","last_clean":"2024-03-20T09:31:19.716398+0000","last_became_active":"2024-03-20T09:31:07.424767+0000","last_became_peered":"2024-03-20T09:31:07.424767+0000","last_unstale":"2024-03-20T09:31:19.716398+0000","last_undegraded":"2024-03-20T09:31:19.716398+0000","last_fullsized":"2024-03-20T09:31:19.716398+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:31:06.406431+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:32:08.558196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11206656,"num_objects":171,"num_object_clones":0,"num_object_copies":513,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":171,"num_write_kb":10944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,11],"acting":[5,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"20'137","reported_seq":1364,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-03-20T09:31:15.011280+0000","last_change":"2024-03-20T09:29:46.361304+0000","last_active":"2024-03-20T09:31:15.011280+0000","last_peered":"2024-03-20T09:31:15.011280+0000","last_clean":"2024-03-20T09:31:15.011280+0000","last_became_active":"2024-03-20T09:19:54.502529+0000","last_became_peered":"2024-03-20T09:19:54.502529+0000","last_unstale":"2024-03-20T09:31:15.011280+0000","last_undegraded":"2024-03-20T09:31:15.011280+0000","last_fullsized":"2024-03-20T09:31:15.011280+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:29:46.361164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:29:46.361164+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:31:19.093620+0000","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"20'137","reported_seq":1662,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-03-20T09:31:15.647775+0000","last_change":"2024-03-20T09:30:48.287414+0000","last_active":"2024-03-20T09:31:15.647775+0000","last_peered":"2024-03-20T09:31:15.647775+0000","last_clean":"2024-03-20T09:31:15.647775+0000","last_became_active":"2024-03-20T09:10:34.096758+0000","last_became_peered":"2024-03-20T09:10:34.096758+0000","last_unstale":"2024-03-20T09:31:15.647775+0000","last_undegraded":"2024-03-20T09:31:15.647775+0000","last_fullsized":"2024-03-20T09:31:15.647775+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:30:48.287283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:30:48.287283+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:32:16.703215+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"20'137","reported_seq":405,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-03-20T09:31:15.012836+0000","last_change":"2024-03-20T09:31:07.425887+0000","last_active":"2024-03-20T09:31:15.012836+0000","last_peered":"2024-03-20T09:31:15.012836+0000","last_clean":"2024-03-20T09:31:15.012836+0000","last_became_active":"2024-03-20T09:31:07.425519+0000","last_became_peered":"2024-03-20T09:31:07.425519+0000","last_unstale":"2024-03-20T09:31:15.012836+0000","last_undegraded":"2024-03-20T09:31:15.012836+0000","last_fullsized":"2024-03-20T09:31:15.012836+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:29:36.467602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:29:36.467602+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:31:17.548379+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.10","version":"20'137","reported_seq":1208,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-03-20T09:31:15.647856+0000","last_change":"2024-03-20T09:30:54.259634+0000","last_active":"2024-03-20T09:31:15.647856+0000","last_peered":"2024-03-20T09:31:15.647856+0000","last_clean":"2024-03-20T09:31:15.647856+0000","last_became_active":"2024-03-20T09:28:22.584176+0000","last_became_peered":"2024-03-20T09:28:22.584176+0000","last_unstale":"2024-03-20T09:31:15.647856+0000","last_undegraded":"2024-03-20T09:31:15.647856+0000","last_fullsized":"2024-03-20T09:31:15.647856+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:30:54.259533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:30:54.259533+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:32:03.415283+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"20'137","reported_seq":1606,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-03-20T09:31:16.378445+0000","last_change":"2024-03-20T09:30:59.256026+0000","last_active":"2024-03-20T09:31:16.378445+0000","last_peered":"2024-03-20T09:31:16.378445+0000","last_clean":"2024-03-20T09:31:16.378445+0000","last_became_active":"2024-03-20T09:20:25.342982+0000","last_became_peered":"2024-03-20T09:20:25.342982+0000","last_unstale":"2024-03-20T09:31:16.378445+0000","last_undegraded":"2024-03-20T09:31:16.378445+0000","last_fullsized":"2024-03-20T09:31:16.378445+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:30:59.255939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:30:59.255939+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:32:15.922050+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"20'137","reported_seq":324,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-03-20T09:31:16.378527+0000","last_change":"2024-03-20T09:31:07.425535+0000","last_active":"2024-03-20T09:31:16.378527+0000","last_peered":"2024-03-20T09:31:16.378527+0000","last_clean":"2024-03-20T09:31:16.378527+0000","last_became_active":"2024-03-20T09:31:07.424851+0000","last_became_peered":"2024-03-20T09:31:07.424851+0000","last_unstale":"2024-03-20T09:31:16.378527+0000","last_undegraded":"2024-03-20T09:31:16.378527+0000","last_fullsized":"2024-03-20T09:31:16.378527+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:30:35.536443+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:16:15.448920+0000","last_clean_scrub_stamp":"2024-03-20T09:30:35.536443+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:31:18.207594+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.17","version":"20'137","reported_seq":1783,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-03-20T09:31:15.012700+0000","last_change":"2024-03-20T09:30:44.185222+0000","last_active":"2024-03-20T09:31:15.012700+0000","last_peered":"2024-03-20T09:31:15.012700+0000","last_clean":"2024-03-20T09:31:15.012700+0000","last_became_active":"2024-03-20T09:28:22.588303+0000","last_became_peered":"2024-03-20T09:28:22.588303+0000","last_unstale":"2024-03-20T09:31:15.012700+0000","last_undegraded":"2024-03-20T09:31:15.012700+0000","last_fullsized":"2024-03-20T09:31:15.012700+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:30:44.185085+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:23.640533+0000","last_clean_scrub_stamp":"2024-03-20T09:30:44.185085+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:32:05.821662+0000","scrub_duration":140,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"20'137","reported_seq":1834,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-03-20T09:31:15.607684+0000","last_change":"2024-03-20T09:30:54.106940+0000","last_active":"2024-03-20T09:31:15.607684+0000","last_peered":"2024-03-20T09:31:15.607684+0000","last_clean":"2024-03-20T09:31:15.607684+0000","last_became_active":"2024-03-20T09:20:25.349082+0000","last_became_peered":"2024-03-20T09:20:25.349082+0000","last_unstale":"2024-03-20T09:31:15.607684+0000","last_undegraded":"2024-03-20T09:31:15.607684+0000","last_fullsized":"2024-03-20T09:31:15.607684+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:30:54.106826+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:30:54.106826+0000","last_clean_scrub_stamp":"2024-03-20T09:30:54.106826+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:32:16.465898+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"20'137","reported_seq":1105,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-03-20T09:31:14.605390+0000","last_change":"2024-03-20T09:30:04.068149+0000","last_active":"2024-03-20T09:31:14.605390+0000","last_peered":"2024-03-20T09:31:14.605390+0000","last_clean":"2024-03-20T09:31:14.605390+0000","last_became_active":"2024-03-20T09:12:14.307581+0000","last_became_peered":"2024-03-20T09:12:14.307581+0000","last_unstale":"2024-03-20T09:31:14.605390+0000","last_undegraded":"2024-03-20T09:31:14.605390+0000","last_fullsized":"2024-03-20T09:31:14.605390+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:30:04.068063+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:23:41.251546+0000","last_clean_scrub_stamp":"2024-03-20T09:30:04.068063+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:31:19.068026+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"20'137","reported_seq":1363,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-03-20T09:31:15.011294+0000","last_change":"2024-03-20T09:29:52.272120+0000","last_active":"2024-03-20T09:31:15.011294+0000","last_peered":"2024-03-20T09:31:15.011294+0000","last_clean":"2024-03-20T09:31:15.011294+0000","last_became_active":"2024-03-20T09:19:54.502391+0000","last_became_peered":"2024-03-20T09:19:54.502391+0000","last_unstale":"2024-03-20T09:31:15.011294+0000","last_undegraded":"2024-03-20T09:31:15.011294+0000","last_fullsized":"2024-03-20T09:31:15.011294+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:29:52.272005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:29:52.272005+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:31:15.151333+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":61145088,"num_objects":933,"num_object_clones":0,"num_object_copies":2799,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":933,"num_write_kb":59712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":219021312,"data_stored":219021312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":933,"ondisk_log_size":933,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":603786,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":5,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":351723,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4248,"ondisk_log_size":4248,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860839,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":745380,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":92999772,"statfs":{"total":95995035648,"available":95231766528,"internally_reserved":0,"allocated":2056192,"data_stored":1315078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":761199044},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":3697966841865,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":80,"seq":343597384117,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":334460,"kb_used_data":2032,"kb_used_omap":1,"kb_used_meta":332414,"kb_avail":93410692,"statfs":{"total":95995035648,"available":95652548608,"internally_reserved":0,"allocated":2080768,"data_stored":1331022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":340392384},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":70,"seq":300647711160,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":444144,"kb_used_data":2020,"kb_used_omap":1,"kb_used_meta":442110,"kb_avail":93301008,"statfs":{"total":95995035648,"available":95540232192,"internally_reserved":0,"allocated":2068480,"data_stored":1323220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":452721094},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":60,"seq":257698038205,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":777532,"kb_used_data":2220,"kb_used_omap":1,"kb_used_meta":775294,"kb_avail":92967620,"statfs":{"total":95995035648,"available":95198842880,"internally_reserved":0,"allocated":2273280,"data_stored":1524468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":793901514},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332546,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":601328,"kb_used_data":2340,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":93143824,"statfs":{"total":95995035648,"available":95379275776,"internally_reserved":0,"allocated":2396160,"data_stored":1650900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":613349830},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575325,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":821368,"kb_used_data":23856,"kb_used_omap":1,"kb_used_meta":797502,"kb_avail":92923784,"statfs":{"total":95995035648,"available":95153954816,"internally_reserved":0,"allocated":24428544,"data_stored":23678798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":816642502},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738843,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":804020,"kb_used_data":2340,"kb_used_omap":1,"kb_used_meta":801662,"kb_avail":92941132,"statfs":{"total":95995035648,"available":95171719168,"internally_reserved":0,"allocated":2396160,"data_stored":1650900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":820902341},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411797,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":574856,"kb_used_data":2232,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":93170296,"statfs":{"total":95995035648,"available":95406383104,"internally_reserved":0,"allocated":2285568,"data_stored":1532270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":586348997},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150162,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":934584,"kb_used_data":199536,"kb_used_omap":1,"kb_used_meta":735038,"kb_avail":92810568,"statfs":{"total":95995035648,"available":95038021632,"internally_reserved":0,"allocated":204324864,"data_stored":203575118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":752679367},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953936,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":552432,"kb_used_data":2020,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":93192720,"statfs":{"total":95995035648,"available":95429345280,"internally_reserved":0,"allocated":2068480,"data_stored":1323220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":563608005},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":443,"seq":1902670512378,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":739968,"kb_used_data":182456,"kb_used_omap":181,"kb_used_meta":557322,"kb_avail":93005184,"statfs":{"total":95995035648,"available":95237308416,"internally_reserved":0,"allocated":186834944,"data_stored":186081646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":186131,"internal_metadata":570697965},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":48431104,"data_stored":48431104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":135856128,"data_stored":135856128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":34734080,"data_stored":34734080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:31:23.047 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pool set unique_pool_5 pgp_num 16 2024-03-20T09:31:23.089 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:31:23.124 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:23.150 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:31:23.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:23.186 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:31:23.287 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:31:23.337 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:23.516 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:31:23.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:23.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:31:23.579 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:31:23.629 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:31:23.638 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 5 16 14944 14928 186.586 175.438 0.00572728 0.00534978 2024-03-20T09:31:23.658 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:23.663 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:31:23.683 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:31:23.684 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:23 smithi122 ceph-mon[73495]: from='client.16719 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:31:23.684 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:23 smithi122 ceph-mon[73495]: pgmap v2119: 47 pgs: 47 active+clean; 338 MiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 42 MiB/s wr, 675 op/s 2024-03-20T09:31:23.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:23.685 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:31:23.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:23 smithi026 ceph-mon[68309]: from='client.16719 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:31:23.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:23 smithi026 ceph-mon[68309]: pgmap v2119: 47 pgs: 47 active+clean; 338 MiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 42 MiB/s wr, 675 op/s 2024-03-20T09:31:23.787 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:31:23.835 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:23.866 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:31:23.872 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:23 smithi115 ceph-mon[72476]: from='client.16719 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:31:23.872 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:23 smithi115 ceph-mon[72476]: pgmap v2119: 47 pgs: 47 active+clean; 338 MiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 42 MiB/s wr, 675 op/s 2024-03-20T09:31:23.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:23.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:31:23.899 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-20T09:31:23.974 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:31:24.008 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:24.034 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:31:24.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:24.037 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:31:24.138 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:31:24.196 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:24.232 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:31:24.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:24.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:31:24.273 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:31:24.342 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:31:24.376 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:24.401 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:31:24.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:24.405 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:31:24.506 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:31:24.547 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:24.573 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:31:24.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:24.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:31:24.653 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 6 16 17712 17696 184.32 173 0.00588766 0.00542019 2024-03-20T09:31:24.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:24 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:31:24.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:24 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:31:24.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:24 smithi122 ceph-mon[73495]: osdmap e877: 12 total, 12 up, 11 in 2024-03-20T09:31:24.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:24 smithi122 ceph-mon[73495]: pgmap v2121: 47 pgs: 47 active+clean; 494 MiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 56 MiB/s wr, 902 op/s 2024-03-20T09:31:24.752 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:31:24.809 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:24.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:24 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:31:24.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:24 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:31:24.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:24 smithi115 ceph-mon[72476]: osdmap e877: 12 total, 12 up, 11 in 2024-03-20T09:31:24.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:24 smithi115 ceph-mon[72476]: pgmap v2121: 47 pgs: 47 active+clean; 494 MiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 56 MiB/s wr, 902 op/s 2024-03-20T09:31:24.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:24 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:31:24.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:24 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:31:24.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:24 smithi026 ceph-mon[68309]: osdmap e877: 12 total, 12 up, 11 in 2024-03-20T09:31:24.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:24 smithi026 ceph-mon[68309]: pgmap v2121: 47 pgs: 47 active+clean; 494 MiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 56 MiB/s wr, 902 op/s 2024-03-20T09:31:25.373 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:31:25.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:25.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:31:25.478 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:31:25.509 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:25.531 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:31:25.535 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:25.536 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:31:25.637 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:31:25.638 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 7 16 20563 20547 183.442 178.188 0.00325119 0.00544569 2024-03-20T09:31:25.670 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:25.699 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:31:25.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:25.703 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:31:25.803 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:31:25.842 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:25.864 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:31:25.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:25.869 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:31:25.970 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:31:26.000 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:26.027 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:31:26.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:26.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:31:26.130 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:31:26.175 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:26.388 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:31:26.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:26.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:31:26.497 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:31:26.530 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:26.557 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:31:26.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:26.561 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:31:26.638 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 8 15 22961 22946 179.252 149.938 0.00248121 0.00557408 2024-03-20T09:31:26.662 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:31:26.716 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:27.484 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:31:27.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:27.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:31:27.589 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:31:27.620 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:27.638 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 9 16 25318 25302 175.695 147.25 0.00415599 0.00568687 2024-03-20T09:31:27.688 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:27 smithi122 ceph-mon[73495]: pgmap v2122: 47 pgs: 47 active+clean; 997 MiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 100 MiB/s wr, 1.59k op/s 2024-03-20T09:31:27.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:27 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:31:27.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:27 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:31:27.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:27 smithi026 ceph-mon[68309]: pgmap v2122: 47 pgs: 47 active+clean; 997 MiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 100 MiB/s wr, 1.59k op/s 2024-03-20T09:31:27.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:27 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:31:27.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:27 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:31:27.756 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:31:27.758 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:27 smithi115 ceph-mon[72476]: pgmap v2122: 47 pgs: 47 active+clean; 997 MiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 100 MiB/s wr, 1.59k op/s 2024-03-20T09:31:27.758 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:27 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:31:27.758 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:27 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:31:27.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:27.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:31:27.860 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:31:27.903 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:27.932 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:31:27.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:27.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:31:28.037 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:31:28.072 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:28.100 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:31:28.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:28.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:31:28.156 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:31:28.156 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:31:28.156 INFO:teuthology.orchestra.run.smithi115.stdout: "complaint_time": 30, 2024-03-20T09:31:28.157 INFO:teuthology.orchestra.run.smithi115.stdout: "num_blocked_ops": 0 2024-03-20T09:31:28.157 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:31:28.205 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:31:28.242 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:28.272 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:31:28.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:28.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:31:28.378 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:31:28.419 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:28.524 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:31:28.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:28.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:31:28.631 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:31:28.638 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 10 16 27853 27837 173.968 158.438 0.00459157 0.00574361 2024-03-20T09:31:28.674 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:28.688 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:28 smithi122 ceph-mon[73495]: osdmap e878: 12 total, 12 up, 11 in 2024-03-20T09:31:28.701 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:31:28.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:28.706 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:31:28.707 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:28 smithi115 ceph-mon[72476]: osdmap e878: 12 total, 12 up, 11 in 2024-03-20T09:31:28.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:28 smithi026 ceph-mon[68309]: osdmap e878: 12 total, 12 up, 11 in 2024-03-20T09:31:28.807 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:31:28.870 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:29.066 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-20T09:31:29.443 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:31:29.640 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 11 16 30321 30305 172.159 154.25 0.00821146 0.00580397 2024-03-20T09:31:29.688 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:29 smithi122 ceph-mon[73495]: pgmap v2124: 47 pgs: 47 active+clean; 1.3 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 163 MiB/s wr, 2.61k op/s 2024-03-20T09:31:29.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:29 smithi026 ceph-mon[68309]: pgmap v2124: 47 pgs: 47 active+clean; 1.3 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 163 MiB/s wr, 2.61k op/s 2024-03-20T09:31:29.724 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:29 smithi115 ceph-mon[72476]: pgmap v2124: 47 pgs: 47 active+clean; 1.3 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 163 MiB/s wr, 2.61k op/s 2024-03-20T09:31:30.461 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:31:30.640 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 12 16 32478 32462 169.034 134.812 0.010408 0.00591023 2024-03-20T09:31:30.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:30 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1853155429' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:31:30.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:30 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:31:30.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:30 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1853155429' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:31:30.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:30 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:31:30.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:30 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1853155429' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:31:30.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:30 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:31:31.641 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 13 16 33751 33735 162.152 79.5625 0.0078325 0.00616244 2024-03-20T09:31:31.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:31 smithi026 ceph-mon[68309]: pgmap v2125: 47 pgs: 47 active+clean; 1.5 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 152 MiB/s wr, 2.44k op/s 2024-03-20T09:31:31.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:31 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:31:31.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:31 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:31:31.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:31 smithi026 ceph-mon[68309]: osdmap e879: 12 total, 12 up, 11 in 2024-03-20T09:31:31.745 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:31:31.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:31.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:31:31.854 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:31:31.884 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:31 smithi122 ceph-mon[73495]: pgmap v2125: 47 pgs: 47 active+clean; 1.5 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 152 MiB/s wr, 2.44k op/s 2024-03-20T09:31:31.884 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:31 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:31:31.884 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:31 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:31:31.884 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:31 smithi122 ceph-mon[73495]: osdmap e879: 12 total, 12 up, 11 in 2024-03-20T09:31:31.887 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:31.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:31 smithi115 ceph-mon[72476]: pgmap v2125: 47 pgs: 47 active+clean; 1.5 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 152 MiB/s wr, 2.44k op/s 2024-03-20T09:31:31.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:31 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:31:31.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:31 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:31:31.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:31 smithi115 ceph-mon[72476]: osdmap e879: 12 total, 12 up, 11 in 2024-03-20T09:31:31.913 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:31:31.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:31.916 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:31:32.017 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:31:32.069 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:32.641 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 14 16 35886 35870 160.101 133.438 0.00308134 0.00622757 2024-03-20T09:31:32.648 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:32 smithi115 ceph-mon[72476]: 1.16 scrub starts 2024-03-20T09:31:32.648 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:32 smithi115 ceph-mon[72476]: 1.16 scrub ok 2024-03-20T09:31:32.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:32 smithi026 ceph-mon[68309]: 1.16 scrub starts 2024-03-20T09:31:32.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:32 smithi026 ceph-mon[68309]: 1.16 scrub ok 2024-03-20T09:31:32.908 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:31:32.908 INFO:teuthology.orchestra.run.smithi115.stdout: "size": 20, 2024-03-20T09:31:32.909 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 600, 2024-03-20T09:31:32.909 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [ 2024-03-20T09:31:32.909 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.909 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:208841 5.5 e675/637 5:a3eee9ed:::benchmark_data_smithi122_157894_object78854:head v 675'13076, mlcod=623'4831)", 2024-03-20T09:31:32.909 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:23.178048+0000", 2024-03-20T09:31:32.909 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 549.73576468700003, 2024-03-20T09:31:32.909 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.089451347, 2024-03-20T09:31:32.909 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:32.909 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:32.909 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:32.909 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.909 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:32.909 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178048+0000", 2024-03-20T09:31:32.909 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:32.910 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.910 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.910 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:32.910 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178048+0000", 2024-03-20T09:31:32.910 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:32.910 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.910 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.910 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:32.910 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178049+0000", 2024-03-20T09:31:32.910 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.153e-06 2024-03-20T09:31:32.910 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.910 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.910 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:32.911 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178059+0000", 2024-03-20T09:31:32.911 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0169e-05 2024-03-20T09:31:32.911 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.911 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.911 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:32.911 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178076+0000", 2024-03-20T09:31:32.911 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7096e-05 2024-03-20T09:31:32.911 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.911 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.911 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:32.911 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178086+0000", 2024-03-20T09:31:32.911 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.6320000000000003e-06 2024-03-20T09:31:32.911 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.911 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.911 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:32.912 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178186+0000", 2024-03-20T09:31:32.912 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000100503 2024-03-20T09:31:32.912 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.912 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.912 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:32.912 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178215+0000", 2024-03-20T09:31:32.912 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8164e-05 2024-03-20T09:31:32.912 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.912 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.912 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:32.912 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.267448+0000", 2024-03-20T09:31:32.912 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.089233281999999997 2024-03-20T09:31:32.912 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.913 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.913 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:32.913 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.267499+0000", 2024-03-20T09:31:32.913 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1347999999999997e-05 2024-03-20T09:31:32.913 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:32.913 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:32.913 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:32.913 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.913 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.913 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:208845 5.5 e675/637 5:ad73ae47:::benchmark_data_smithi122_157894_object78858:head v 675'13077, mlcod=623'4831)", 2024-03-20T09:31:32.913 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:23.178111+0000", 2024-03-20T09:31:32.913 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 549.73570159600001, 2024-03-20T09:31:32.913 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.089416893999999997, 2024-03-20T09:31:32.913 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:32.913 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:32.914 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:32.914 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.914 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:32.914 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178111+0000", 2024-03-20T09:31:32.914 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:32.914 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.914 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.914 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:32.914 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178111+0000", 2024-03-20T09:31:32.914 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:32.914 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.914 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.914 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:32.915 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178111+0000", 2024-03-20T09:31:32.915 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8999999999999998e-07 2024-03-20T09:31:32.915 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.915 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.915 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:32.915 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178115+0000", 2024-03-20T09:31:32.915 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7799999999999998e-06 2024-03-20T09:31:32.915 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.915 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.915 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:32.915 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178121+0000", 2024-03-20T09:31:32.915 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4219999999999999e-06 2024-03-20T09:31:32.915 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.915 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.916 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:32.916 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.178125+0000", 2024-03-20T09:31:32.916 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.0740000000000003e-06 2024-03-20T09:31:32.916 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.916 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.916 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:32.916 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.194801+0000", 2024-03-20T09:31:32.916 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.016675964000000001 2024-03-20T09:31:32.916 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.916 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.916 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:32.916 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.194818+0000", 2024-03-20T09:31:32.916 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7150999999999999e-05 2024-03-20T09:31:32.916 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.917 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.917 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:32.917 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.267503+0000", 2024-03-20T09:31:32.917 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.072685657000000001 2024-03-20T09:31:32.917 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.917 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.917 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:32.917 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:23.267528+0000", 2024-03-20T09:31:32.917 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.4556e-05 2024-03-20T09:31:32.917 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:32.917 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:32.917 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:32.917 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.917 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.917 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:223322 5.5 e682/637 5:a0cca45d:::benchmark_data_smithi122_157894_object93335:head v 682'14007, mlcod=627'5694)", 2024-03-20T09:31:32.918 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:46.235324+0000", 2024-03-20T09:31:32.918 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 526.67848866899999, 2024-03-20T09:31:32.918 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.088264128999999997, 2024-03-20T09:31:32.918 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:32.918 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:32.918 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:32.918 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.918 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:32.918 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235324+0000", 2024-03-20T09:31:32.918 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:32.918 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.918 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.918 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:32.919 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235324+0000", 2024-03-20T09:31:32.919 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:32.919 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.919 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.919 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:32.919 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235324+0000", 2024-03-20T09:31:32.919 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.9700000000000002e-07 2024-03-20T09:31:32.919 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.919 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.919 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:32.919 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235329+0000", 2024-03-20T09:31:32.919 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.899e-06 2024-03-20T09:31:32.919 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.919 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.919 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:32.920 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235336+0000", 2024-03-20T09:31:32.920 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.9739999999999996e-06 2024-03-20T09:31:32.920 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.920 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.920 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:32.920 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235341+0000", 2024-03-20T09:31:32.920 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6999999999999999e-06 2024-03-20T09:31:32.920 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.920 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.920 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:32.920 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235536+0000", 2024-03-20T09:31:32.920 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00019468400000000001 2024-03-20T09:31:32.920 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.920 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.920 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:32.921 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235559+0000", 2024-03-20T09:31:32.921 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3881000000000001e-05 2024-03-20T09:31:32.921 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.921 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.921 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:32.921 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.323553+0000", 2024-03-20T09:31:32.921 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.087993431999999996 2024-03-20T09:31:32.921 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.921 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.921 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:32.921 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.323588+0000", 2024-03-20T09:31:32.921 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.5162e-05 2024-03-20T09:31:32.921 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:32.921 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:32.922 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:32.922 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.922 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.922 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:223326 5.5 e682/637 5:a90b760b:::benchmark_data_smithi122_157894_object93339:head v 682'14008, mlcod=627'5694)", 2024-03-20T09:31:32.922 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:46.235363+0000", 2024-03-20T09:31:32.922 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 526.67845009500002, 2024-03-20T09:31:32.922 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.088261112000000003, 2024-03-20T09:31:32.922 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:32.922 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:32.922 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:32.922 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.922 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:32.922 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235363+0000", 2024-03-20T09:31:32.923 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:32.923 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.923 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.923 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:32.923 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235363+0000", 2024-03-20T09:31:32.923 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:32.923 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.923 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.923 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:32.923 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235363+0000", 2024-03-20T09:31:32.923 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8099999999999999e-07 2024-03-20T09:31:32.923 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.923 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.923 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:32.924 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235366+0000", 2024-03-20T09:31:32.924 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.061e-06 2024-03-20T09:31:32.924 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.924 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.924 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:32.924 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235371+0000", 2024-03-20T09:31:32.924 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2979999999999999e-06 2024-03-20T09:31:32.924 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.924 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.924 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:32.924 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235375+0000", 2024-03-20T09:31:32.924 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.8619999999999996e-06 2024-03-20T09:31:32.924 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.924 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.925 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:32.925 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.248068+0000", 2024-03-20T09:31:32.925 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.012692554 2024-03-20T09:31:32.925 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.925 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.925 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:32.925 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.248090+0000", 2024-03-20T09:31:32.925 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2747000000000001e-05 2024-03-20T09:31:32.925 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.925 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.925 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:32.925 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.323592+0000", 2024-03-20T09:31:32.925 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.075502001999999999 2024-03-20T09:31:32.925 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.925 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.926 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:32.926 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.323624+0000", 2024-03-20T09:31:32.926 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.1306999999999997e-05 2024-03-20T09:31:32.926 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:32.926 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:32.926 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:32.926 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.926 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.926 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:223337 5.5 e682/637 5:a415780d:::benchmark_data_smithi122_157894_object93350:head v 682'14009, mlcod=627'5694)", 2024-03-20T09:31:32.926 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:46.235380+0000", 2024-03-20T09:31:32.926 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 526.678432122, 2024-03-20T09:31:32.926 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.088268737999999999, 2024-03-20T09:31:32.926 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:32.926 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:32.927 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:32.927 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.927 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:32.927 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235380+0000", 2024-03-20T09:31:32.927 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:32.927 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.927 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.927 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:32.927 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235380+0000", 2024-03-20T09:31:32.927 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:32.927 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.927 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.927 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:32.927 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235381+0000", 2024-03-20T09:31:32.928 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6999999999999999e-07 2024-03-20T09:31:32.928 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.928 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.928 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:32.928 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235387+0000", 2024-03-20T09:31:32.928 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1650000000000003e-06 2024-03-20T09:31:32.928 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.928 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.928 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:32.928 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235392+0000", 2024-03-20T09:31:32.928 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8420000000000004e-06 2024-03-20T09:31:32.928 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.928 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.928 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:32.928 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.235396+0000", 2024-03-20T09:31:32.929 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.8929999999999998e-06 2024-03-20T09:31:32.929 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.929 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.929 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:32.929 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.260711+0000", 2024-03-20T09:31:32.929 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.025315500000000001 2024-03-20T09:31:32.929 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.929 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.929 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:32.929 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.260729+0000", 2024-03-20T09:31:32.929 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7963000000000001e-05 2024-03-20T09:31:32.929 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.929 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.929 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:32.930 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.323627+0000", 2024-03-20T09:31:32.930 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.062897733999999997 2024-03-20T09:31:32.930 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.930 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.930 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:32.930 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.323649+0000", 2024-03-20T09:31:32.930 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2470999999999999e-05 2024-03-20T09:31:32.930 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:32.930 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:32.930 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:32.930 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.930 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.930 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:223309 5.5 e682/637 5:a4c63307:::benchmark_data_smithi122_157894_object93322:head v 682'14011, mlcod=627'5694)", 2024-03-20T09:31:32.930 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:46.248926+0000", 2024-03-20T09:31:32.931 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 526.66488642900003, 2024-03-20T09:31:32.931 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.094989770000000001, 2024-03-20T09:31:32.931 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:32.931 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:32.931 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:32.931 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.931 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:32.931 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.248926+0000", 2024-03-20T09:31:32.931 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:32.931 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.931 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.931 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:32.931 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.248926+0000", 2024-03-20T09:31:32.931 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:32.931 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.932 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.932 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:32.932 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.248927+0000", 2024-03-20T09:31:32.932 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0440000000000001e-06 2024-03-20T09:31:32.932 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.932 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.932 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:32.932 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.248938+0000", 2024-03-20T09:31:32.932 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0560000000000001e-05 2024-03-20T09:31:32.932 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.932 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.932 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:32.932 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.248952+0000", 2024-03-20T09:31:32.933 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4297000000000001e-05 2024-03-20T09:31:32.933 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.933 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.933 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:32.933 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.248964+0000", 2024-03-20T09:31:32.933 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1912e-05 2024-03-20T09:31:32.933 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.933 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.933 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:32.933 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.281776+0000", 2024-03-20T09:31:32.933 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.032812458000000003 2024-03-20T09:31:32.933 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.933 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.933 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:32.933 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.281805+0000", 2024-03-20T09:31:32.934 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8765e-05 2024-03-20T09:31:32.934 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.934 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.934 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:32.934 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.343870+0000", 2024-03-20T09:31:32.934 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.062064805000000001 2024-03-20T09:31:32.934 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.934 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.934 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:32.934 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.343916+0000", 2024-03-20T09:31:32.934 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5929e-05 2024-03-20T09:31:32.934 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:32.934 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:32.934 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:32.935 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.935 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.935 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:223346 5.5 e682/637 5:a014a865:::benchmark_data_smithi122_157894_object93359:head v 682'14012, mlcod=627'5694)", 2024-03-20T09:31:32.935 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:46.254399+0000", 2024-03-20T09:31:32.935 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 526.65941318299997, 2024-03-20T09:31:32.935 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.089556495, 2024-03-20T09:31:32.935 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:32.935 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:32.935 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:32.935 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.935 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:32.935 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.254399+0000", 2024-03-20T09:31:32.935 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:32.935 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.936 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.936 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:32.936 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.254399+0000", 2024-03-20T09:31:32.936 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:32.936 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.936 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.936 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:32.936 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.254400+0000", 2024-03-20T09:31:32.936 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9599999999999999e-07 2024-03-20T09:31:32.936 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.936 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.936 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:32.936 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.254405+0000", 2024-03-20T09:31:32.936 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.011e-06 2024-03-20T09:31:32.937 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.937 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.937 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:32.937 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.254415+0000", 2024-03-20T09:31:32.937 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.9790000000000005e-06 2024-03-20T09:31:32.937 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.937 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.937 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:32.937 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.254420+0000", 2024-03-20T09:31:32.937 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2340000000000002e-06 2024-03-20T09:31:32.937 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.937 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.937 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:32.937 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.292675+0000", 2024-03-20T09:31:32.937 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.038254945999999998 2024-03-20T09:31:32.938 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.938 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.938 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:32.938 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.292700+0000", 2024-03-20T09:31:32.938 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5173000000000001e-05 2024-03-20T09:31:32.938 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.938 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.938 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:32.938 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.343920+0000", 2024-03-20T09:31:32.938 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.051219555 2024-03-20T09:31:32.938 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.938 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.938 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:32.938 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:46.343956+0000", 2024-03-20T09:31:32.939 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6001000000000003e-05 2024-03-20T09:31:32.939 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:32.939 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:32.939 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:32.939 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.939 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.939 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "MOSDPGPush(5.f 685/637 [PushOp(5:f1254502:::benchmark_data_smithi122_157894_object94700:head, version: 628'5882, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:f1254502:::benchmark_data_smithi122_157894_object94700:head@628'5882, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-20T09:31:32.939 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:47.826130+0000", 2024-03-20T09:31:32.939 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 525.08768300400004, 2024-03-20T09:31:32.939 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.088800740000000003, 2024-03-20T09:31:32.939 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:32.939 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:31:32.939 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:32.939 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.940 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:32.940 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.826130+0000", 2024-03-20T09:31:32.940 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:32.940 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.940 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.940 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:32.940 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.826130+0000", 2024-03-20T09:31:32.940 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:32.940 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.940 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.940 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:32.940 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.826130+0000", 2024-03-20T09:31:32.940 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.0800000000000001e-07 2024-03-20T09:31:32.940 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.941 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.941 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:32.941 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.826161+0000", 2024-03-20T09:31:32.941 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.1158999999999998e-05 2024-03-20T09:31:32.941 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.941 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.941 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:32.941 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.826187+0000", 2024-03-20T09:31:32.941 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5978000000000001e-05 2024-03-20T09:31:32.941 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.941 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.941 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:32.941 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.826204+0000", 2024-03-20T09:31:32.941 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6501e-05 2024-03-20T09:31:32.941 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.942 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.942 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:32.942 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.910972+0000", 2024-03-20T09:31:32.942 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.084768204999999999 2024-03-20T09:31:32.942 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.942 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.942 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:32.942 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.910987+0000", 2024-03-20T09:31:32.942 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5277999999999998e-05 2024-03-20T09:31:32.942 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.942 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.942 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:32.942 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:47.914930+0000", 2024-03-20T09:31:32.943 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0039433109999999997 2024-03-20T09:31:32.943 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:32.943 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:32.943 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:32.943 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.943 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.943 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:257605 5.5 e699/637 5:ade6c67d:::benchmark_data_smithi122_157894_object127618:head v 699'16216, mlcod=635'8024)", 2024-03-20T09:31:32.943 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:23:37.959926+0000", 2024-03-20T09:31:32.943 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 474.95388683900001, 2024-03-20T09:31:32.943 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.099523950999999999, 2024-03-20T09:31:32.943 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:32.943 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:32.943 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:32.943 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.943 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:32.944 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:37.959926+0000", 2024-03-20T09:31:32.944 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:32.944 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.944 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.944 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:32.944 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:37.959926+0000", 2024-03-20T09:31:32.944 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:32.944 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.944 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.944 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:32.944 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:37.959926+0000", 2024-03-20T09:31:32.944 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2700000000000001e-07 2024-03-20T09:31:32.944 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.944 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.946 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:32.946 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:37.959929+0000", 2024-03-20T09:31:32.946 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9409999999999999e-06 2024-03-20T09:31:32.946 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.946 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.946 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:32.946 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:37.959932+0000", 2024-03-20T09:31:32.946 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.3589999999999999e-06 2024-03-20T09:31:32.946 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.946 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.946 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:32.947 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:37.959935+0000", 2024-03-20T09:31:32.947 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5390000000000002e-06 2024-03-20T09:31:32.947 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.947 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.947 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:32.947 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:37.979588+0000", 2024-03-20T09:31:32.947 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.019652738999999999 2024-03-20T09:31:32.947 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.947 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.947 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:32.947 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:37.979610+0000", 2024-03-20T09:31:32.947 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2467999999999999e-05 2024-03-20T09:31:32.947 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.947 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.948 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:32.948 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:38.059401+0000", 2024-03-20T09:31:32.948 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.079791387000000005 2024-03-20T09:31:32.948 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.948 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.948 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:32.948 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:38.059450+0000", 2024-03-20T09:31:32.948 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8291000000000002e-05 2024-03-20T09:31:32.948 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:32.948 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:32.948 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:32.948 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.948 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.949 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:257606 5.5 e699/637 5:a912954b:::benchmark_data_smithi122_157894_object127619:head v 699'16217, mlcod=635'8024)", 2024-03-20T09:31:32.949 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:23:37.967251+0000", 2024-03-20T09:31:32.949 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 474.94656122499998, 2024-03-20T09:31:32.949 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.092234901999999994, 2024-03-20T09:31:32.949 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:32.949 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:32.949 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:32.949 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.949 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:32.949 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:37.967251+0000", 2024-03-20T09:31:32.949 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:32.949 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.949 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.949 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:32.950 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:37.967251+0000", 2024-03-20T09:31:32.950 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:32.950 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.950 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.950 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:32.950 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:37.967252+0000", 2024-03-20T09:31:32.950 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.5499999999999996e-07 2024-03-20T09:31:32.950 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.950 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.950 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:32.950 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:37.967261+0000", 2024-03-20T09:31:32.950 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.6300000000000004e-06 2024-03-20T09:31:32.950 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.951 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.951 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:32.951 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:37.967277+0000", 2024-03-20T09:31:32.951 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5968999999999999e-05 2024-03-20T09:31:32.951 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.951 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.951 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:32.951 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:37.967285+0000", 2024-03-20T09:31:32.951 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.3650000000000004e-06 2024-03-20T09:31:32.951 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.951 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.951 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:32.951 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:37.989840+0000", 2024-03-20T09:31:32.951 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.022555009000000001 2024-03-20T09:31:32.952 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.952 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.952 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:32.952 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:37.989868+0000", 2024-03-20T09:31:32.952 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8106000000000002e-05 2024-03-20T09:31:32.952 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.952 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.952 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:32.952 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:38.059454+0000", 2024-03-20T09:31:32.952 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.069586061000000005 2024-03-20T09:31:32.952 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.952 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.952 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:32.952 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:38.059486+0000", 2024-03-20T09:31:32.953 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.1807000000000002e-05 2024-03-20T09:31:32.953 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:32.953 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:32.953 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:32.953 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.953 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.953 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:92151 6.e 6:72a8fe95:::benchmark_data_smithi122_168655_object92150:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e735)", 2024-03-20T09:31:32.953 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:19.137168+0000", 2024-03-20T09:31:32.953 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 373.77664455399997, 2024-03-20T09:31:32.953 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.092309425000000001, 2024-03-20T09:31:32.953 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:32.953 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:32.953 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:31:32.954 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:31:32.954 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:31:32.954 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 92151 2024-03-20T09:31:32.954 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.954 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:32.954 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.954 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:32.954 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.137168+0000", 2024-03-20T09:31:32.954 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:32.954 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.954 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.954 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:32.954 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.137168+0000", 2024-03-20T09:31:32.955 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:32.955 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.955 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.955 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:32.955 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.137169+0000", 2024-03-20T09:31:32.955 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.116e-06 2024-03-20T09:31:32.955 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.955 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.955 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:32.955 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.137244+0000", 2024-03-20T09:31:32.955 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.4370999999999997e-05 2024-03-20T09:31:32.955 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.955 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.955 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:32.956 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.137263+0000", 2024-03-20T09:31:32.956 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9357000000000001e-05 2024-03-20T09:31:32.956 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.956 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.956 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:32.956 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.137274+0000", 2024-03-20T09:31:32.956 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0954e-05 2024-03-20T09:31:32.956 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.956 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.956 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:32.956 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.137322+0000", 2024-03-20T09:31:32.956 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8547999999999997e-05 2024-03-20T09:31:32.956 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.956 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.957 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:32.957 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.137477+0000", 2024-03-20T09:31:32.957 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00015410000000000001 2024-03-20T09:31:32.957 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.957 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.957 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,2", 2024-03-20T09:31:32.957 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.137648+0000", 2024-03-20T09:31:32.957 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00017198699999999999 2024-03-20T09:31:32.957 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.957 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.957 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:31:32.957 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.144467+0000", 2024-03-20T09:31:32.957 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0068180769999999996 2024-03-20T09:31:32.957 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.958 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.958 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:31:32.958 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.149111+0000", 2024-03-20T09:31:32.958 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0046444149999999998 2024-03-20T09:31:32.958 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.958 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.958 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:31:32.958 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.229349+0000", 2024-03-20T09:31:32.958 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.080238212000000003 2024-03-20T09:31:32.958 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.958 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.958 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:32.958 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.229416+0000", 2024-03-20T09:31:32.958 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.7138000000000007e-05 2024-03-20T09:31:32.959 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.959 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.959 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:32.959 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.229477+0000", 2024-03-20T09:31:32.959 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1149999999999996e-05 2024-03-20T09:31:32.959 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:32.959 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:32.959 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:32.959 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.959 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.959 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:92152 6.e 6:75dfc8b0:::benchmark_data_smithi122_168655_object92151:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e735)", 2024-03-20T09:31:32.959 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:19.137291+0000", 2024-03-20T09:31:32.959 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 373.77652142699998, 2024-03-20T09:31:32.959 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.092306768999999997, 2024-03-20T09:31:32.960 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:32.960 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:32.960 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:31:32.960 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:31:32.960 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:31:32.960 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 92152 2024-03-20T09:31:32.960 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.960 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:32.960 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.960 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:32.960 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.137291+0000", 2024-03-20T09:31:32.960 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:32.960 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.960 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.961 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:32.961 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.137291+0000", 2024-03-20T09:31:32.961 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:32.961 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.961 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.961 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:32.961 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.137292+0000", 2024-03-20T09:31:32.961 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.9999999999999998e-07 2024-03-20T09:31:32.961 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.961 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.961 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:32.961 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.137314+0000", 2024-03-20T09:31:32.961 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2867999999999999e-05 2024-03-20T09:31:32.961 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.961 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.962 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:32.962 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.137323+0000", 2024-03-20T09:31:32.962 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.2430000000000005e-06 2024-03-20T09:31:32.962 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.962 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.962 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:32.962 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.137327+0000", 2024-03-20T09:31:32.962 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.941e-06 2024-03-20T09:31:32.962 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.962 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.962 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:32.962 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.139923+0000", 2024-03-20T09:31:32.962 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0025965839999999999 2024-03-20T09:31:32.963 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.963 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.963 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:32.963 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.140055+0000", 2024-03-20T09:31:32.963 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013172100000000001 2024-03-20T09:31:32.963 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.963 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.963 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,2", 2024-03-20T09:31:32.963 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.140172+0000", 2024-03-20T09:31:32.963 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000116864 2024-03-20T09:31:32.963 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.963 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.963 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:31:32.964 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.146208+0000", 2024-03-20T09:31:32.964 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0060361920000000001 2024-03-20T09:31:32.964 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.964 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.964 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:31:32.964 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.149160+0000", 2024-03-20T09:31:32.964 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.002952091 2024-03-20T09:31:32.964 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.964 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.964 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:31:32.964 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.229523+0000", 2024-03-20T09:31:32.964 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.080362798999999999 2024-03-20T09:31:32.964 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.964 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.964 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:32.965 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.229560+0000", 2024-03-20T09:31:32.965 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7527999999999997e-05 2024-03-20T09:31:32.965 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.965 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.965 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:32.965 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:19.229598+0000", 2024-03-20T09:31:32.965 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7537999999999999e-05 2024-03-20T09:31:32.965 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:32.965 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:32.965 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:32.965 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.965 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.965 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:129144 6.2 6:47957f3d:::benchmark_data_smithi122_168655_object9035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e748)", 2024-03-20T09:31:32.965 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:56.335243+0000", 2024-03-20T09:31:32.966 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 336.57856913900002, 2024-03-20T09:31:32.966 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.223256595, 2024-03-20T09:31:32.966 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:32.966 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:31:32.966 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:31:32.966 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:31:32.966 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:31:32.966 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 129144 2024-03-20T09:31:32.966 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.966 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:32.966 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.966 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:32.966 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335243+0000", 2024-03-20T09:31:32.967 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:32.967 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.967 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.967 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:32.967 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335243+0000", 2024-03-20T09:31:32.967 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:32.967 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.967 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.967 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:32.967 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335244+0000", 2024-03-20T09:31:32.967 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0299999999999999e-07 2024-03-20T09:31:32.967 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.967 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.967 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:32.968 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335246+0000", 2024-03-20T09:31:32.968 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.4959999999999999e-06 2024-03-20T09:31:32.968 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.968 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.968 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:32.968 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335253+0000", 2024-03-20T09:31:32.968 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.6699999999999997e-06 2024-03-20T09:31:32.968 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.968 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.968 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:32.968 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335258+0000", 2024-03-20T09:31:32.968 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.7500000000000003e-06 2024-03-20T09:31:32.968 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.968 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.969 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:32.969 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335321+0000", 2024-03-20T09:31:32.969 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2765e-05 2024-03-20T09:31:32.969 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.969 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.969 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:32.969 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335440+0000", 2024-03-20T09:31:32.969 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000119141 2024-03-20T09:31:32.969 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.969 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.969 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,3", 2024-03-20T09:31:32.969 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335511+0000", 2024-03-20T09:31:32.969 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.1104000000000002e-05 2024-03-20T09:31:32.970 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.970 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.970 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:31:32.970 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.342379+0000", 2024-03-20T09:31:32.970 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0068685320000000001 2024-03-20T09:31:32.970 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.970 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.970 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:32.970 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.557233+0000", 2024-03-20T09:31:32.970 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.21485372999999999 2024-03-20T09:31:32.970 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.970 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.971 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:32.971 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.558500+0000", 2024-03-20T09:31:32.971 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0012669039999999999 2024-03-20T09:31:32.971 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:32.971 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:32.971 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:32.971 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.971 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.971 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:129123 6.2 6:450098af:::benchmark_data_smithi122_168655_object9014:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e749)", 2024-03-20T09:31:32.971 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:56.335273+0000", 2024-03-20T09:31:32.971 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 336.578539315, 2024-03-20T09:31:32.971 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0772038850000001, 2024-03-20T09:31:32.971 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:32.971 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:31:32.972 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:31:32.972 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:31:32.972 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:31:32.972 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 129123 2024-03-20T09:31:32.972 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.972 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:32.972 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.972 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:32.972 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335273+0000", 2024-03-20T09:31:32.972 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:32.972 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.972 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.972 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:32.972 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335273+0000", 2024-03-20T09:31:32.973 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:32.973 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.973 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.973 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:32.973 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335274+0000", 2024-03-20T09:31:32.973 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6300000000000001e-07 2024-03-20T09:31:32.973 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.973 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.973 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:32.973 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335276+0000", 2024-03-20T09:31:32.973 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2089999999999999e-06 2024-03-20T09:31:32.973 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.973 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.973 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:32.974 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335282+0000", 2024-03-20T09:31:32.974 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0800000000000002e-06 2024-03-20T09:31:32.974 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.974 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.974 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:32.974 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335287+0000", 2024-03-20T09:31:32.974 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4700000000000001e-06 2024-03-20T09:31:32.974 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.974 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.974 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:32.974 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.339142+0000", 2024-03-20T09:31:32.974 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0038545229999999999 2024-03-20T09:31:32.974 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.974 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.975 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op must wait for map", 2024-03-20T09:31:32.975 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.339150+0000", 2024-03-20T09:31:32.975 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.6279999999999996e-06 2024-03-20T09:31:32.975 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.975 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.975 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:32.975 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.557266+0000", 2024-03-20T09:31:32.975 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.21811618799999999 2024-03-20T09:31:32.975 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.975 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.975 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:31:32.975 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.557275+0000", 2024-03-20T09:31:32.975 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.8810000000000002e-06 2024-03-20T09:31:32.976 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.976 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.976 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:32.976 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:57.412381+0000", 2024-03-20T09:31:32.976 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.85510642599999998 2024-03-20T09:31:32.976 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.976 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.976 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:32.976 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:57.412477+0000", 2024-03-20T09:31:32.976 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.6117000000000001e-05 2024-03-20T09:31:32.976 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:32.976 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:32.976 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:32.976 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.976 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.977 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:129125 6.12 6:4e4918e7:::benchmark_data_smithi122_168655_object9016:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e749)", 2024-03-20T09:31:32.977 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:56.335325+0000", 2024-03-20T09:31:32.977 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 336.578487204, 2024-03-20T09:31:32.977 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.13051883, 2024-03-20T09:31:32.977 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:32.977 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:31:32.977 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:31:32.977 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:31:32.977 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:31:32.977 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 129125 2024-03-20T09:31:32.977 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.977 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:32.977 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.977 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:32.978 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335325+0000", 2024-03-20T09:31:32.978 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:32.978 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.978 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.978 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:32.978 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335325+0000", 2024-03-20T09:31:32.978 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:32.978 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.978 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.978 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:32.978 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335326+0000", 2024-03-20T09:31:32.978 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.4300000000000002e-07 2024-03-20T09:31:32.978 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.979 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.979 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:32.979 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335331+0000", 2024-03-20T09:31:32.979 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.566e-06 2024-03-20T09:31:32.979 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.979 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.979 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:32.979 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335339+0000", 2024-03-20T09:31:32.979 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.4430000000000003e-06 2024-03-20T09:31:32.979 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.979 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.979 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:32.979 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335346+0000", 2024-03-20T09:31:32.979 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.2339999999999997e-06 2024-03-20T09:31:32.980 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.980 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.980 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:32.980 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.558708+0000", 2024-03-20T09:31:32.980 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.223361277 2024-03-20T09:31:32.980 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.980 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.980 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:31:32.980 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.558717+0000", 2024-03-20T09:31:32.980 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.7350000000000008e-06 2024-03-20T09:31:32.980 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.980 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.980 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:32.980 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:57.465759+0000", 2024-03-20T09:31:32.980 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.90704171499999997 2024-03-20T09:31:32.981 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.981 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.981 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:32.981 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:57.465844+0000", 2024-03-20T09:31:32.981 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5117000000000005e-05 2024-03-20T09:31:32.981 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:32.981 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:32.981 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:32.981 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.981 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.981 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:129129 6.2 6:471d5bbb:::benchmark_data_smithi122_168655_object9020:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e749)", 2024-03-20T09:31:32.981 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:56.335359+0000", 2024-03-20T09:31:32.981 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 336.57845407100001, 2024-03-20T09:31:32.981 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.077221416, 2024-03-20T09:31:32.982 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:32.982 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:31:32.982 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:31:32.982 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:31:32.982 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:31:32.982 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 129129 2024-03-20T09:31:32.982 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.982 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:32.982 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.982 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:32.982 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335359+0000", 2024-03-20T09:31:32.982 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:32.982 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.983 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.983 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:32.983 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335359+0000", 2024-03-20T09:31:32.983 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:32.983 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.983 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.983 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:32.983 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335359+0000", 2024-03-20T09:31:32.983 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.2500000000000001e-07 2024-03-20T09:31:32.983 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.983 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.983 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:32.983 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335361+0000", 2024-03-20T09:31:32.983 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3240000000000001e-06 2024-03-20T09:31:32.984 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.984 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.984 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:32.984 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335368+0000", 2024-03-20T09:31:32.984 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.3790000000000003e-06 2024-03-20T09:31:32.984 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.984 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.984 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:32.984 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335372+0000", 2024-03-20T09:31:32.984 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.724e-06 2024-03-20T09:31:32.984 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.984 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.984 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:32.985 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.339225+0000", 2024-03-20T09:31:32.985 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0038523030000000001 2024-03-20T09:31:32.985 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.985 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.985 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting_for_map not empty", 2024-03-20T09:31:32.985 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.339230+0000", 2024-03-20T09:31:32.985 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9540000000000003e-06 2024-03-20T09:31:32.985 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.985 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.985 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:32.985 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.557298+0000", 2024-03-20T09:31:32.985 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.21806794900000001 2024-03-20T09:31:32.985 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.985 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.986 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:31:32.986 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.557305+0000", 2024-03-20T09:31:32.986 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3520000000000001e-06 2024-03-20T09:31:32.986 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.986 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.986 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:32.986 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:57.412500+0000", 2024-03-20T09:31:32.986 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.85519480299999995 2024-03-20T09:31:32.986 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.986 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.986 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:32.986 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:57.412580+0000", 2024-03-20T09:31:32.986 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.0203000000000004e-05 2024-03-20T09:31:32.986 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:32.987 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:32.987 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:32.987 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.987 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.987 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:129144 6.2 6:47957f3d:::benchmark_data_smithi122_168655_object9035:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e749)", 2024-03-20T09:31:32.987 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:56.335379+0000", 2024-03-20T09:31:32.987 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 336.578433518, 2024-03-20T09:31:32.987 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1031262369999999, 2024-03-20T09:31:32.987 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:32.987 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:31:32.987 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:31:32.987 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:31:32.987 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:31:32.988 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 129144 2024-03-20T09:31:32.988 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.988 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:32.988 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.988 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:32.988 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335379+0000", 2024-03-20T09:31:32.988 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:32.988 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.988 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.988 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:32.988 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335379+0000", 2024-03-20T09:31:32.988 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:32.988 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.988 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.989 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:32.989 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335379+0000", 2024-03-20T09:31:32.989 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.5900000000000003e-07 2024-03-20T09:31:32.989 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.989 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.989 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:32.989 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335382+0000", 2024-03-20T09:31:32.989 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.181e-06 2024-03-20T09:31:32.989 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.989 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.989 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:32.989 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335387+0000", 2024-03-20T09:31:32.989 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9799999999999998e-06 2024-03-20T09:31:32.989 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.990 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.990 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:32.990 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335391+0000", 2024-03-20T09:31:32.990 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5519999999999998e-06 2024-03-20T09:31:32.990 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.990 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.990 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:32.990 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.339468+0000", 2024-03-20T09:31:32.990 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0040772990000000004 2024-03-20T09:31:32.990 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.990 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.990 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting_for_map not empty", 2024-03-20T09:31:32.990 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.339473+0000", 2024-03-20T09:31:32.990 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5800000000000002e-06 2024-03-20T09:31:32.991 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.991 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.991 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:32.991 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.557326+0000", 2024-03-20T09:31:32.991 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.21785318400000001 2024-03-20T09:31:32.991 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.991 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.991 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:31:32.991 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.557333+0000", 2024-03-20T09:31:32.991 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.7719999999999997e-06 2024-03-20T09:31:32.991 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.991 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.991 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:32.992 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:57.412602+0000", 2024-03-20T09:31:32.992 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.85526948700000005 2024-03-20T09:31:32.992 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.992 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.992 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:31:32.992 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:57.412726+0000", 2024-03-20T09:31:32.992 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012335300000000001 2024-03-20T09:31:32.992 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.992 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.992 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:32.992 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:57.438420+0000", 2024-03-20T09:31:32.992 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.025694552999999998 2024-03-20T09:31:32.992 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.992 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.993 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:32.993 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:57.438505+0000", 2024-03-20T09:31:32.993 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.4937000000000003e-05 2024-03-20T09:31:32.993 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:32.993 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:32.993 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:32.993 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.993 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.993 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:224651 6.2 6:423cf26c:::benchmark_data_smithi122_168655_object104542:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e854)", 2024-03-20T09:31:32.993 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:30:27.284691+0000", 2024-03-20T09:31:32.993 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 65.629121440999995, 2024-03-20T09:31:32.993 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.091321880999999994, 2024-03-20T09:31:32.993 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:32.993 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:32.994 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:31:32.994 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:31:32.994 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:31:32.994 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 224651 2024-03-20T09:31:32.994 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.994 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:32.994 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.994 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:32.994 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:27.284691+0000", 2024-03-20T09:31:32.994 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:32.994 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.994 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.994 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:32.995 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:27.284691+0000", 2024-03-20T09:31:32.995 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:32.995 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.995 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.995 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:32.995 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:27.284692+0000", 2024-03-20T09:31:32.995 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2899999999999999e-06 2024-03-20T09:31:32.995 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.995 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.995 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:32.995 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:27.284698+0000", 2024-03-20T09:31:32.995 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4569999999999996e-06 2024-03-20T09:31:32.995 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.995 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.996 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:32.996 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:27.284708+0000", 2024-03-20T09:31:32.996 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0263e-05 2024-03-20T09:31:32.996 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.996 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.996 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:32.996 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:27.284717+0000", 2024-03-20T09:31:32.996 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.4509999999999992e-06 2024-03-20T09:31:32.996 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.996 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.996 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:32.996 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:27.284835+0000", 2024-03-20T09:31:32.996 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00011815 2024-03-20T09:31:32.997 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.997 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.997 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:32.997 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:27.284925+0000", 2024-03-20T09:31:32.997 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.0235000000000006e-05 2024-03-20T09:31:32.997 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.997 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.997 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,3", 2024-03-20T09:31:32.997 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:27.285453+0000", 2024-03-20T09:31:32.997 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00052750099999999999 2024-03-20T09:31:32.997 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.997 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.997 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:31:32.997 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:27.288536+0000", 2024-03-20T09:31:32.998 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0030835770000000001 2024-03-20T09:31:32.998 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.998 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.998 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:31:32.998 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:27.293827+0000", 2024-03-20T09:31:32.998 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0052905440000000003 2024-03-20T09:31:32.998 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.998 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.998 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:31:32.998 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:27.375944+0000", 2024-03-20T09:31:32.998 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.082117281 2024-03-20T09:31:32.998 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.998 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.999 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:32.999 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:27.375981+0000", 2024-03-20T09:31:32.999 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6952000000000001e-05 2024-03-20T09:31:32.999 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.999 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.999 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:32.999 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:27.376013+0000", 2024-03-20T09:31:32.999 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.218e-05 2024-03-20T09:31:32.999 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:32.999 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:32.999 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:32.999 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:32.999 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:32.999 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:225247 6.e 6:7d028362:::benchmark_data_smithi122_168655_object105138:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e854)", 2024-03-20T09:31:33.000 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:30:27.997378+0000", 2024-03-20T09:31:33.000 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 64.916434425999995, 2024-03-20T09:31:33.000 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.092459318999999998, 2024-03-20T09:31:33.000 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:33.000 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:33.000 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:31:33.000 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:31:33.000 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:31:33.000 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 225247 2024-03-20T09:31:33.000 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:33.000 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:33.000 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:33.000 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:33.000 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:27.997378+0000", 2024-03-20T09:31:33.001 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:33.001 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:33.001 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:33.001 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:33.001 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:27.997378+0000", 2024-03-20T09:31:33.001 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:33.001 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:33.001 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:33.001 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:33.001 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:27.997379+0000", 2024-03-20T09:31:33.001 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0580000000000001e-06 2024-03-20T09:31:33.001 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:33.001 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:33.001 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:33.002 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:27.997383+0000", 2024-03-20T09:31:33.002 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.0230000000000003e-06 2024-03-20T09:31:33.002 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:33.002 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:33.002 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:33.002 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:27.997393+0000", 2024-03-20T09:31:33.002 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.5270000000000004e-06 2024-03-20T09:31:33.002 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:33.002 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:33.002 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:33.002 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:27.997400+0000", 2024-03-20T09:31:33.002 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.4719999999999998e-06 2024-03-20T09:31:33.002 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:33.002 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:33.003 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:33.003 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:27.997458+0000", 2024-03-20T09:31:33.003 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8167000000000001e-05 2024-03-20T09:31:33.003 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:33.003 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:33.003 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:33.003 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:27.997595+0000", 2024-03-20T09:31:33.003 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000136229 2024-03-20T09:31:33.003 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:33.003 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:33.003 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,2", 2024-03-20T09:31:33.003 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:27.997734+0000", 2024-03-20T09:31:33.003 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013887900000000001 2024-03-20T09:31:33.003 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:33.004 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:33.004 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:31:33.004 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:28.002597+0000", 2024-03-20T09:31:33.004 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0048634749999999999 2024-03-20T09:31:33.004 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:33.004 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:33.004 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:31:33.004 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:28.014372+0000", 2024-03-20T09:31:33.004 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.011774880999999999 2024-03-20T09:31:33.004 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:33.004 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:33.004 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:31:33.004 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:28.089732+0000", 2024-03-20T09:31:33.004 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.075360187999999995 2024-03-20T09:31:33.005 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:33.005 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:33.005 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:33.005 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:28.089788+0000", 2024-03-20T09:31:33.005 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5507999999999998e-05 2024-03-20T09:31:33.005 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:33.005 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:33.005 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:33.005 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:28.089838+0000", 2024-03-20T09:31:33.005 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9911999999999999e-05 2024-03-20T09:31:33.005 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:33.005 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:33.005 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:33.005 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:33.006 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:33.006 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:32470 7.1 7:83406e34:::benchmark_data_smithi122_187077_object32469:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e879)", 2024-03-20T09:31:33.006 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:31:30.643697+0000", 2024-03-20T09:31:33.006 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 2.2701158719999999, 2024-03-20T09:31:33.006 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.142067165, 2024-03-20T09:31:33.006 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:33.006 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:33.006 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:31:33.006 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:31:33.006 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:31:33.006 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 32470 2024-03-20T09:31:33.006 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:33.006 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:33.007 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:33.007 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:33.007 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:30.643697+0000", 2024-03-20T09:31:33.007 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:33.007 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:33.007 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:33.007 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:33.007 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:30.643697+0000", 2024-03-20T09:31:33.007 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:33.007 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:33.007 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:33.007 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:33.007 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:30.643698+0000", 2024-03-20T09:31:33.007 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.249e-06 2024-03-20T09:31:33.008 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:33.008 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:33.008 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:33.008 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:30.643771+0000", 2024-03-20T09:31:33.008 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3231000000000004e-05 2024-03-20T09:31:33.008 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:33.008 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:33.008 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:33.008 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:30.643785+0000", 2024-03-20T09:31:33.008 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3594999999999999e-05 2024-03-20T09:31:33.008 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:33.008 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:33.008 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:33.008 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:30.643793+0000", 2024-03-20T09:31:33.008 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5339999999999999e-06 2024-03-20T09:31:33.009 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:33.009 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:33.009 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:33.009 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:30.643856+0000", 2024-03-20T09:31:33.009 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2430999999999999e-05 2024-03-20T09:31:33.009 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:33.009 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:33.009 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:33.009 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:30.643988+0000", 2024-03-20T09:31:33.009 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013209800000000001 2024-03-20T09:31:33.009 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:33.009 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:33.009 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 3,6", 2024-03-20T09:31:33.009 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:30.644135+0000", 2024-03-20T09:31:33.010 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00014720600000000001 2024-03-20T09:31:33.010 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:33.010 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:33.010 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:31:33.010 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:30.645281+0000", 2024-03-20T09:31:33.010 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001145648 2024-03-20T09:31:33.010 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:33.010 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:33.010 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:31:33.010 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:30.645563+0000", 2024-03-20T09:31:33.010 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000282133 2024-03-20T09:31:33.010 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:33.010 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:33.011 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:31:33.011 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:30.785699+0000", 2024-03-20T09:31:33.011 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.14013568000000001 2024-03-20T09:31:33.011 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:33.011 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:33.011 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:33.011 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:30.785737+0000", 2024-03-20T09:31:33.011 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.8809000000000001e-05 2024-03-20T09:31:33.011 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:33.011 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:33.011 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:33.011 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:30.785764+0000", 2024-03-20T09:31:33.011 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6551e-05 2024-03-20T09:31:33.011 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:33.011 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:33.012 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:33.012 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:33.012 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:33.012 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:31:33.012 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:32 smithi122 ceph-mon[73495]: 1.16 scrub starts 2024-03-20T09:31:33.013 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:32 smithi122 ceph-mon[73495]: 1.16 scrub ok 2024-03-20T09:31:33.641 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 15 16 37673 37657 156.873 111.688 0.00881993 0.00636922 2024-03-20T09:31:33.648 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:33 smithi026 ceph-mon[68309]: 1.15 scrub starts 2024-03-20T09:31:33.649 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:33 smithi026 ceph-mon[68309]: pgmap v2127: 47 pgs: 47 active+clean; 1.9 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 182 MiB/s wr, 2.91k op/s 2024-03-20T09:31:33.649 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:33 smithi026 ceph-mon[68309]: 1.15 scrub ok 2024-03-20T09:31:33.649 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:33 smithi026 ceph-mon[68309]: osdmap e880: 12 total, 12 up, 11 in 2024-03-20T09:31:33.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:33 smithi115 ceph-mon[72476]: 1.15 scrub starts 2024-03-20T09:31:33.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:33 smithi115 ceph-mon[72476]: pgmap v2127: 47 pgs: 47 active+clean; 1.9 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 182 MiB/s wr, 2.91k op/s 2024-03-20T09:31:33.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:33 smithi115 ceph-mon[72476]: 1.15 scrub ok 2024-03-20T09:31:33.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:33 smithi115 ceph-mon[72476]: osdmap e880: 12 total, 12 up, 11 in 2024-03-20T09:31:33.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:33 smithi122 ceph-mon[73495]: 1.15 scrub starts 2024-03-20T09:31:33.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:33 smithi122 ceph-mon[73495]: pgmap v2127: 47 pgs: 47 active+clean; 1.9 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 182 MiB/s wr, 2.91k op/s 2024-03-20T09:31:33.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:33 smithi122 ceph-mon[73495]: 1.15 scrub ok 2024-03-20T09:31:33.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:33 smithi122 ceph-mon[73495]: osdmap e880: 12 total, 12 up, 11 in 2024-03-20T09:31:33.988 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:31:33.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:33.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:31:34.093 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:31:34.126 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:34.302 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-20T09:31:34.575 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:34 smithi026 ceph-mon[68309]: 1.13 scrub starts 2024-03-20T09:31:34.575 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:34 smithi026 ceph-mon[68309]: 1.1a scrub starts 2024-03-20T09:31:34.575 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:34 smithi026 ceph-mon[68309]: 1.1a scrub ok 2024-03-20T09:31:34.641 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 16 15 39379 39364 153.736 106.688 0.00455903 0.00649235 2024-03-20T09:31:34.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:34 smithi115 ceph-mon[72476]: 1.13 scrub starts 2024-03-20T09:31:34.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:34 smithi115 ceph-mon[72476]: 1.1a scrub starts 2024-03-20T09:31:34.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:34 smithi115 ceph-mon[72476]: 1.1a scrub ok 2024-03-20T09:31:34.668 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:31:34.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:34 smithi122 ceph-mon[73495]: 1.13 scrub starts 2024-03-20T09:31:34.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:34 smithi122 ceph-mon[73495]: 1.1a scrub starts 2024-03-20T09:31:34.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:34 smithi122 ceph-mon[73495]: 1.1a scrub ok 2024-03-20T09:31:35.128 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:31:35.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:35.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:31:35.234 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:31:35.273 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:35.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:35 smithi115 ceph-mon[72476]: 1.13 scrub ok 2024-03-20T09:31:35.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:35 smithi115 ceph-mon[72476]: 1.7 scrub starts 2024-03-20T09:31:35.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:35 smithi115 ceph-mon[72476]: 1.7 scrub ok 2024-03-20T09:31:35.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:35 smithi115 ceph-mon[72476]: 1.1b scrub starts 2024-03-20T09:31:35.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:35 smithi115 ceph-mon[72476]: 1.1b scrub ok 2024-03-20T09:31:35.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:35 smithi115 ceph-mon[72476]: pgmap v2129: 47 pgs: 47 active+clean; 2.0 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 123 MiB/s wr, 1.97k op/s 2024-03-20T09:31:35.600 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:35 smithi115 ceph-mon[72476]: 1.1d scrub starts 2024-03-20T09:31:35.641 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 17 16 41192 41176 151.355 113.25 0.00565978 0.00660174 2024-03-20T09:31:35.746 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:31:35.793 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:31:35.853 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:31:35.884 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:31:35.927 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:31:35.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:35.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:31:35.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:35 smithi122 ceph-mon[73495]: 1.13 scrub ok 2024-03-20T09:31:35.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:35 smithi122 ceph-mon[73495]: 1.7 scrub starts 2024-03-20T09:31:35.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:35 smithi122 ceph-mon[73495]: 1.7 scrub ok 2024-03-20T09:31:35.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:35 smithi122 ceph-mon[73495]: 1.1b scrub starts 2024-03-20T09:31:35.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:35 smithi122 ceph-mon[73495]: 1.1b scrub ok 2024-03-20T09:31:35.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:35 smithi122 ceph-mon[73495]: pgmap v2129: 47 pgs: 47 active+clean; 2.0 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 123 MiB/s wr, 1.97k op/s 2024-03-20T09:31:35.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:35 smithi122 ceph-mon[73495]: 1.1d scrub starts 2024-03-20T09:31:35.942 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:31:35.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:35 smithi026 ceph-mon[68309]: 1.13 scrub ok 2024-03-20T09:31:35.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:35 smithi026 ceph-mon[68309]: 1.7 scrub starts 2024-03-20T09:31:35.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:35 smithi026 ceph-mon[68309]: 1.7 scrub ok 2024-03-20T09:31:35.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:35 smithi026 ceph-mon[68309]: 1.1b scrub starts 2024-03-20T09:31:35.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:35 smithi026 ceph-mon[68309]: 1.1b scrub ok 2024-03-20T09:31:35.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:35 smithi026 ceph-mon[68309]: pgmap v2129: 47 pgs: 47 active+clean; 2.0 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 123 MiB/s wr, 1.97k op/s 2024-03-20T09:31:35.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:35 smithi026 ceph-mon[68309]: 1.1d scrub starts 2024-03-20T09:31:35.986 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:31:36.032 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:31:36.042 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:31:36.088 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:31:36.108 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:36.123 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:31:36.141 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:31:36.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:36.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:31:36.160 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:31:36.195 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:31:36.232 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:31:36.247 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:31:36.253 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:31:36.253 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:31:36.253 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:31:36.253 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:31:36.254 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:31:36.254 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:31:36.254 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:31:36.254 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:31:36.254 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:31:36.254 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:31:36.254 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:31:36.254 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:31:36.288 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:36.323 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:31:36.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:36.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:31:36.427 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:31:36.463 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:36.491 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:31:36.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:36.496 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:31:36.498 INFO:teuthology.orchestra.run.smithi026.stderr:set pool 7 pgp_num to 16 2024-03-20T09:31:36.597 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:31:36.623 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:31:36.633 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:36.642 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 18 16 42869 42853 148.756 104.812 0.0128993 0.00671726 2024-03-20T09:31:36.648 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:36 smithi115 ceph-mon[72476]: 1.1d scrub ok 2024-03-20T09:31:36.648 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:36 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3967545746' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:31:36.648 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:36 smithi115 ceph-mon[72476]: 1.e scrub starts 2024-03-20T09:31:36.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:36 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2820318660' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-20T09:31:36.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:36 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-20T09:31:36.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:36 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:31:36.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:36 smithi115 ceph-mon[72476]: pgmap v2130: 47 pgs: 47 active+clean; 2.3 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-03-20T09:31:36.661 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:31:36.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:36.666 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:31:36.725 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:31:36.725 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:31:36.768 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:31:36.794 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:36 smithi122 ceph-mon[73495]: 1.1d scrub ok 2024-03-20T09:31:36.794 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:36 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3967545746' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:31:36.794 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:36 smithi122 ceph-mon[73495]: 1.e scrub starts 2024-03-20T09:31:36.794 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:36 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2820318660' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-20T09:31:36.794 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:36 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-20T09:31:36.794 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:36 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:31:36.794 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:36 smithi122 ceph-mon[73495]: pgmap v2130: 47 pgs: 47 active+clean; 2.3 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-03-20T09:31:36.796 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:36.821 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:31:36.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:36.824 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:31:36.925 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:31:36.956 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:36.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:36 smithi026 ceph-mon[68309]: 1.1d scrub ok 2024-03-20T09:31:36.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:36 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3967545746' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:31:36.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:36 smithi026 ceph-mon[68309]: 1.e scrub starts 2024-03-20T09:31:36.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:36 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2820318660' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-20T09:31:36.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:36 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-20T09:31:36.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:36 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:31:36.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:36 smithi026 ceph-mon[68309]: pgmap v2130: 47 pgs: 47 active+clean; 2.3 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 127 MiB/s wr, 2.03k op/s 2024-03-20T09:31:36.979 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:31:36.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:36.983 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:31:37.085 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:31:37.127 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:37.381 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:31:37.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:37.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:31:37.428 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-20T09:31:37.495 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:31:37.567 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:37.642 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 19 16 44259 44243 145.5 86.875 0.00732112 0.0068677 2024-03-20T09:31:37.648 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:37 smithi115 ceph-mon[72476]: 1.e scrub ok 2024-03-20T09:31:37.648 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:37 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]': finished 2024-03-20T09:31:37.648 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:37 smithi115 ceph-mon[72476]: osdmap e881: 12 total, 12 up, 11 in 2024-03-20T09:31:37.817 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:31:37.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:37 smithi122 ceph-mon[73495]: 1.e scrub ok 2024-03-20T09:31:37.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:37 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]': finished 2024-03-20T09:31:37.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:37 smithi122 ceph-mon[73495]: osdmap e881: 12 total, 12 up, 11 in 2024-03-20T09:31:37.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:37 smithi026 ceph-mon[68309]: 1.e scrub ok 2024-03-20T09:31:37.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:37 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num", "val": "16"}]': finished 2024-03-20T09:31:37.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:37 smithi026 ceph-mon[68309]: osdmap e881: 12 total, 12 up, 11 in 2024-03-20T09:31:38.040 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:31:38.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:38.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:31:38.146 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:31:38.202 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:38.495 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:31:38.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:38.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:31:38.585 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:31:38.614 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:31:38.644 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:2024-03-20T09:31:38.651274+0000 min lat: 0.00116204 max lat: 0.239973 avg lat: 0.00693677 2024-03-20T09:31:38.644 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T09:31:38.644 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 20 16 46128 46112 144.065 116.812 0.0103179 0.00693677 2024-03-20T09:31:38.647 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:38.648 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:38 smithi115 ceph-mon[72476]: 1.a scrub starts 2024-03-20T09:31:38.648 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:38 smithi115 ceph-mon[72476]: 1.a scrub ok 2024-03-20T09:31:38.648 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:38 smithi115 ceph-mon[72476]: osdmap e882: 12 total, 12 up, 11 in 2024-03-20T09:31:38.648 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:38 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:31:38.648 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:38 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:31:38.648 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:38 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:31:38.648 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:38 smithi115 ceph-mon[72476]: pgmap v2133: 47 pgs: 47 active+clean; 2.6 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-03-20T09:31:38.675 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:31:38.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:38.678 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:31:38.779 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:31:38.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:38 smithi122 ceph-mon[73495]: 1.a scrub starts 2024-03-20T09:31:38.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:38 smithi122 ceph-mon[73495]: 1.a scrub ok 2024-03-20T09:31:38.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:38 smithi122 ceph-mon[73495]: osdmap e882: 12 total, 12 up, 11 in 2024-03-20T09:31:38.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:38 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:31:38.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:38 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:31:38.810 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:38 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:31:38.811 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:38 smithi122 ceph-mon[73495]: pgmap v2133: 47 pgs: 47 active+clean; 2.6 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-03-20T09:31:38.812 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:38.835 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:31:38.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:38.840 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:31:38.895 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:38 smithi026 ceph-mon[68309]: 1.a scrub starts 2024-03-20T09:31:38.896 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:38 smithi026 ceph-mon[68309]: 1.a scrub ok 2024-03-20T09:31:38.896 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:38 smithi026 ceph-mon[68309]: osdmap e882: 12 total, 12 up, 11 in 2024-03-20T09:31:38.896 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:38 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:31:38.896 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:38 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:31:38.896 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:38 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:31:38.896 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:38 smithi026 ceph-mon[68309]: pgmap v2133: 47 pgs: 47 active+clean; 2.6 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-03-20T09:31:38.940 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:31:38.989 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:39.014 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:31:39.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:39.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:31:39.119 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:31:39.146 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:39.503 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:31:39.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:39.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:31:39.610 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:31:39.643 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 21 15 47756 47741 142.053 101.812 0.00669717 0.00703522 2024-03-20T09:31:39.645 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:39.669 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:31:39.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:39.671 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:31:39.772 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:31:39.807 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:39.835 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:31:39.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:39.839 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:31:39.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:39 smithi122 ceph-mon[73495]: 1.d scrub starts 2024-03-20T09:31:39.971 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:31:39.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:39 smithi026 ceph-mon[68309]: 1.d scrub starts 2024-03-20T09:31:39.980 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:31:39.980 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:31:40.017 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:40.043 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:31:40.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:40.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:31:40.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:39 smithi115 ceph-mon[72476]: 1.d scrub starts 2024-03-20T09:31:40.148 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:31:40.220 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:40.308 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:31:40.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:40.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:31:40.416 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:31:40.478 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:40.509 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:31:40.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:40.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:31:40.558 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:31:40.619 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:31:40.643 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 22 16 49546 49530 140.678 111.812 0.00889965 0.00710335 2024-03-20T09:31:40.659 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:40.827 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:40 smithi026 ceph-mon[68309]: 1.d scrub ok 2024-03-20T09:31:40.827 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:40 smithi026 ceph-mon[68309]: pgmap v2134: 47 pgs: 47 active+clean; 2.7 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-03-20T09:31:40.827 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:40 smithi026 ceph-mon[68309]: 1.0 scrub starts 2024-03-20T09:31:40.827 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:40 smithi026 ceph-mon[68309]: 1.0 scrub ok 2024-03-20T09:31:41.060 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:31:41.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:40 smithi115 ceph-mon[72476]: 1.d scrub ok 2024-03-20T09:31:41.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:40 smithi115 ceph-mon[72476]: pgmap v2134: 47 pgs: 47 active+clean; 2.7 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-03-20T09:31:41.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:40 smithi115 ceph-mon[72476]: 1.0 scrub starts 2024-03-20T09:31:41.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:40 smithi115 ceph-mon[72476]: 1.0 scrub ok 2024-03-20T09:31:41.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:40 smithi122 ceph-mon[73495]: 1.d scrub ok 2024-03-20T09:31:41.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:40 smithi122 ceph-mon[73495]: pgmap v2134: 47 pgs: 47 active+clean; 2.7 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-03-20T09:31:41.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:40 smithi122 ceph-mon[73495]: 1.0 scrub starts 2024-03-20T09:31:41.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:40 smithi122 ceph-mon[73495]: 1.0 scrub ok 2024-03-20T09:31:41.643 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 23 16 51218 51202 139.105 104.5 0.0101656 0.00718443 2024-03-20T09:31:41.853 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:31:42.113 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:31:42.116 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:41 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:31:42.116 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:41 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:31:42.117 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:41 smithi115 ceph-mon[72476]: osdmap e883: 12 total, 12 up, 11 in 2024-03-20T09:31:42.117 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:41 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/283160827' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:31:42.117 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:41 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:31:42.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:42.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:31:42.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:41 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:31:42.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:41 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:31:42.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:41 smithi122 ceph-mon[73495]: osdmap e883: 12 total, 12 up, 11 in 2024-03-20T09:31:42.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:41 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/283160827' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:31:42.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:41 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:31:42.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:41 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:31:42.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:41 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:31:42.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:41 smithi026 ceph-mon[68309]: osdmap e883: 12 total, 12 up, 11 in 2024-03-20T09:31:42.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:41 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/283160827' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:31:42.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:41 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:31:42.219 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:31:42.253 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:42.629 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:31:42.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:42.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:31:42.643 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 24 16 52834 52818 137.517 101 0.0112475 0.00726778 2024-03-20T09:31:42.672 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-20T09:31:42.742 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:31:42.789 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:42.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:42 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:31:42.900 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:42 smithi115 ceph-mon[72476]: Cluster is now healthy 2024-03-20T09:31:42.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:42 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:31:42.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:42 smithi115 ceph-mon[72476]: osdmap e884: 12 total, 12 up, 11 in 2024-03-20T09:31:42.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:42 smithi115 ceph-mon[72476]: pgmap v2137: 47 pgs: 47 active+clean; 3.0 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-03-20T09:31:42.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:42 smithi115 ceph-mon[72476]: 1.4 scrub starts 2024-03-20T09:31:42.901 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:42 smithi115 ceph-mon[72476]: 1.4 scrub ok 2024-03-20T09:31:42.932 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:31:42.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:42.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:31:43.044 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:31:43.057 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 8, 11, 3, 4, 9] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 11, 5, 10] 2024-03-20T09:31:43.057 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 0 1 2024-03-20T09:31:43.074 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:31:43.087 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:43.105 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:42 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:31:43.105 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:42 smithi026 ceph-mon[68309]: Cluster is now healthy 2024-03-20T09:31:43.105 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:42 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:31:43.105 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:42 smithi026 ceph-mon[68309]: osdmap e884: 12 total, 12 up, 11 in 2024-03-20T09:31:43.106 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:42 smithi026 ceph-mon[68309]: pgmap v2137: 47 pgs: 47 active+clean; 3.0 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-03-20T09:31:43.106 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:42 smithi026 ceph-mon[68309]: 1.4 scrub starts 2024-03-20T09:31:43.106 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:42 smithi026 ceph-mon[68309]: 1.4 scrub ok 2024-03-20T09:31:43.121 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:31:43.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:43.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:31:43.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:42 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:31:43.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:42 smithi122 ceph-mon[73495]: Cluster is now healthy 2024-03-20T09:31:43.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:42 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:31:43.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:42 smithi122 ceph-mon[73495]: osdmap e884: 12 total, 12 up, 11 in 2024-03-20T09:31:43.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:42 smithi122 ceph-mon[73495]: pgmap v2137: 47 pgs: 47 active+clean; 3.0 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-03-20T09:31:43.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:42 smithi122 ceph-mon[73495]: 1.4 scrub starts 2024-03-20T09:31:43.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:42 smithi122 ceph-mon[73495]: 1.4 scrub ok 2024-03-20T09:31:43.243 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:31:43.316 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:43.643 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 25 16 54391 54375 135.909 97.3125 0.010457 0.00735404 2024-03-20T09:31:44.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:43 smithi115 ceph-mon[72476]: 1.1e scrub starts 2024-03-20T09:31:44.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:43 smithi115 ceph-mon[72476]: 1.1e scrub ok 2024-03-20T09:31:44.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:43 smithi122 ceph-mon[73495]: 1.1e scrub starts 2024-03-20T09:31:44.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:43 smithi122 ceph-mon[73495]: 1.1e scrub ok 2024-03-20T09:31:44.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:43 smithi026 ceph-mon[68309]: 1.1e scrub starts 2024-03-20T09:31:44.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:43 smithi026 ceph-mon[68309]: 1.1e scrub ok 2024-03-20T09:31:44.226 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:31:44.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:44.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:31:44.333 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:31:44.364 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:44.387 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:31:44.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:44.392 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:31:44.493 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:31:44.540 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:44.643 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 26 16 56113 56097 134.821 107.625 0.00623601 0.00741365 2024-03-20T09:31:44.687 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:31:44.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:44.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:31:44.795 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:31:44.824 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:45.105 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:44 smithi026 ceph-mon[68309]: pgmap v2138: 47 pgs: 47 active+clean; 3.1 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 92 MiB/s wr, 1.48k op/s 2024-03-20T09:31:45.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:44 smithi115 ceph-mon[72476]: pgmap v2138: 47 pgs: 47 active+clean; 3.1 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 92 MiB/s wr, 1.48k op/s 2024-03-20T09:31:45.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:44 smithi122 ceph-mon[73495]: pgmap v2138: 47 pgs: 47 active+clean; 3.1 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 92 MiB/s wr, 1.48k op/s 2024-03-20T09:31:45.344 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:31:45.374 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:31:45.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:45.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:31:45.480 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:31:45.522 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:45.643 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 27 16 57576 57560 133.214 91.4375 0.0060242 0.00750263 2024-03-20T09:31:45.678 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:31:45.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:45.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:31:45.782 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:31:45.849 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:45.906 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:31:45.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:45.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:31:46.003 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:31:46.017 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:31:46.041 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:31:46.075 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:45 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:31:46.075 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:45 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:31:46.075 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:45 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:31:46.075 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:45 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:31:46.081 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:46.086 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:31:46.129 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:31:46.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:45 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:31:46.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:45 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:31:46.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:45 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:31:46.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:45 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:31:46.158 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:31:46.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:45 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:31:46.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:45 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:31:46.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:45 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:31:46.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:45 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:31:46.193 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:31:46.241 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:31:46.254 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:31:46.254 INFO:teuthology.orchestra.run.smithi115.stdout: "size": 20, 2024-03-20T09:31:46.254 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 600, 2024-03-20T09:31:46.254 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [ 2024-03-20T09:31:46.254 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.254 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:27764 6.3 6:c157248a:::benchmark_data_smithi122_168655_object27763:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e722)", 2024-03-20T09:31:46.254 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:24:26.247070+0000", 2024-03-20T09:31:46.254 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 440.01132475899999, 2024-03-20T09:31:46.254 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.700266586, 2024-03-20T09:31:46.254 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:46.254 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:46.255 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:31:46.255 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:31:46.255 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:31:46.255 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 27764 2024-03-20T09:31:46.255 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.255 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:46.255 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.255 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:46.255 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.247070+0000", 2024-03-20T09:31:46.255 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.255 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.255 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.255 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:46.255 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.247070+0000", 2024-03-20T09:31:46.255 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.256 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.256 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.256 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:46.256 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.247072+0000", 2024-03-20T09:31:46.256 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.471e-06 2024-03-20T09:31:46.256 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.256 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.256 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:46.256 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.247963+0000", 2024-03-20T09:31:46.256 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00089083200000000004 2024-03-20T09:31:46.256 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.256 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.256 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:46.256 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.247976+0000", 2024-03-20T09:31:46.257 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3020000000000001e-05 2024-03-20T09:31:46.257 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.257 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.257 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:46.257 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.247984+0000", 2024-03-20T09:31:46.257 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.0129999999999993e-06 2024-03-20T09:31:46.257 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.257 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.257 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:46.257 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.248049+0000", 2024-03-20T09:31:46.257 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.5067000000000006e-05 2024-03-20T09:31:46.257 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.257 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.257 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:46.258 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.248189+0000", 2024-03-20T09:31:46.258 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00014039400000000001 2024-03-20T09:31:46.258 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.258 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.258 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,11", 2024-03-20T09:31:46.258 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.248337+0000", 2024-03-20T09:31:46.258 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000147801 2024-03-20T09:31:46.258 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.258 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.258 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:31:46.258 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.249671+0000", 2024-03-20T09:31:46.258 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0013343859999999999 2024-03-20T09:31:46.258 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.259 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.259 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:31:46.259 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.249968+0000", 2024-03-20T09:31:46.259 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00029656700000000002 2024-03-20T09:31:46.259 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.259 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.259 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:31:46.259 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.947208+0000", 2024-03-20T09:31:46.259 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.697240152999999 2024-03-20T09:31:46.259 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.259 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.259 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:46.259 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.947270+0000", 2024-03-20T09:31:46.259 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2199000000000005e-05 2024-03-20T09:31:46.259 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.260 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.260 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:46.260 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.947337+0000", 2024-03-20T09:31:46.260 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.6682999999999998e-05 2024-03-20T09:31:46.260 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.260 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:46.260 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.260 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.260 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.260 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:27780 6.3 6:cea521fc:::benchmark_data_smithi122_168655_object27779:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e722)", 2024-03-20T09:31:46.260 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:24:26.257510+0000", 2024-03-20T09:31:46.260 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 440.000885348, 2024-03-20T09:31:46.260 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.691143661, 2024-03-20T09:31:46.260 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:46.261 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:46.261 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:31:46.261 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:31:46.261 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:31:46.261 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 27780 2024-03-20T09:31:46.261 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.261 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:46.261 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.261 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:46.261 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.257510+0000", 2024-03-20T09:31:46.261 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.261 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.261 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.261 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:46.262 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.257510+0000", 2024-03-20T09:31:46.262 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.262 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.262 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.262 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:46.262 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.257511+0000", 2024-03-20T09:31:46.262 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3179999999999999e-06 2024-03-20T09:31:46.262 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.262 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.262 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:46.262 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.257581+0000", 2024-03-20T09:31:46.262 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.0049000000000006e-05 2024-03-20T09:31:46.262 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.263 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.263 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:46.263 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.257594+0000", 2024-03-20T09:31:46.263 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2582000000000001e-05 2024-03-20T09:31:46.263 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.263 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.263 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:46.263 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.257601+0000", 2024-03-20T09:31:46.263 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.4580000000000004e-06 2024-03-20T09:31:46.263 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.263 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.263 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:46.263 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.257681+0000", 2024-03-20T09:31:46.263 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.9741999999999996e-05 2024-03-20T09:31:46.264 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.264 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.264 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:46.264 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.257855+0000", 2024-03-20T09:31:46.264 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00017359299999999999 2024-03-20T09:31:46.264 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.264 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.264 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,11", 2024-03-20T09:31:46.264 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.258012+0000", 2024-03-20T09:31:46.264 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00015787900000000001 2024-03-20T09:31:46.264 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.264 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.264 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:31:46.264 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.259114+0000", 2024-03-20T09:31:46.264 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001101788 2024-03-20T09:31:46.265 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.265 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.265 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:31:46.265 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.259458+0000", 2024-03-20T09:31:46.265 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00034384900000000002 2024-03-20T09:31:46.265 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.265 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.265 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:31:46.265 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.948548+0000", 2024-03-20T09:31:46.265 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.689090175 2024-03-20T09:31:46.265 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.265 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.265 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:46.265 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.948601+0000", 2024-03-20T09:31:46.266 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2358999999999998e-05 2024-03-20T09:31:46.266 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.266 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.266 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:46.266 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.948654+0000", 2024-03-20T09:31:46.266 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2868999999999998e-05 2024-03-20T09:31:46.266 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.266 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:46.266 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.266 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.266 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.266 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:27784 6.5 6:a86bc021:::benchmark_data_smithi122_168655_object27783:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e722)", 2024-03-20T09:31:46.266 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:24:26.260259+0000", 2024-03-20T09:31:46.266 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 439.99813581299998, 2024-03-20T09:31:46.267 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.687536504000001, 2024-03-20T09:31:46.267 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:46.267 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:46.267 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:31:46.267 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:31:46.267 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:31:46.267 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 27784 2024-03-20T09:31:46.267 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.267 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:46.267 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.267 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:46.267 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.260259+0000", 2024-03-20T09:31:46.267 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.267 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.268 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.268 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:46.268 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.260259+0000", 2024-03-20T09:31:46.268 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.268 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.268 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.268 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:46.268 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.260261+0000", 2024-03-20T09:31:46.268 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.141e-06 2024-03-20T09:31:46.268 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.268 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.268 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:46.269 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.260337+0000", 2024-03-20T09:31:46.269 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.6427999999999994e-05 2024-03-20T09:31:46.269 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.269 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.269 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:46.269 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.260348+0000", 2024-03-20T09:31:46.269 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0754000000000001e-05 2024-03-20T09:31:46.269 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.269 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.269 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:46.269 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.260353+0000", 2024-03-20T09:31:46.269 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3800000000000002e-06 2024-03-20T09:31:46.269 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.269 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.269 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:46.269 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.260398+0000", 2024-03-20T09:31:46.270 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4920999999999999e-05 2024-03-20T09:31:46.270 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.270 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.270 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:46.270 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.260489+0000", 2024-03-20T09:31:46.270 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.1057999999999995e-05 2024-03-20T09:31:46.270 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.270 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.270 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:31:46.270 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.260577+0000", 2024-03-20T09:31:46.270 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.7798000000000002e-05 2024-03-20T09:31:46.270 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.270 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.271 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:31:46.271 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.261724+0000", 2024-03-20T09:31:46.271 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0011468889999999999 2024-03-20T09:31:46.271 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.271 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.271 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:31:46.271 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.262004+0000", 2024-03-20T09:31:46.271 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00028036899999999998 2024-03-20T09:31:46.271 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.271 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.271 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:31:46.271 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.947719+0000", 2024-03-20T09:31:46.271 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.685714427000001 2024-03-20T09:31:46.271 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.271 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.272 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:46.272 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.947761+0000", 2024-03-20T09:31:46.272 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.2048000000000001e-05 2024-03-20T09:31:46.272 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.272 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.272 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:46.272 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.947796+0000", 2024-03-20T09:31:46.272 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.5290999999999998e-05 2024-03-20T09:31:46.272 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.272 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:46.272 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.272 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.272 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.273 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:27789 6.3 6:c6ec49ce:::benchmark_data_smithi122_168655_object27788:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e722)", 2024-03-20T09:31:46.273 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:24:26.261806+0000", 2024-03-20T09:31:46.273 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 439.99658933500001, 2024-03-20T09:31:46.273 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.686972707000001, 2024-03-20T09:31:46.273 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:46.273 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:46.273 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:31:46.273 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:31:46.273 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:31:46.273 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 27789 2024-03-20T09:31:46.273 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.273 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:46.273 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.273 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:46.274 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.261806+0000", 2024-03-20T09:31:46.274 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.274 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.274 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.274 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:46.274 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.261806+0000", 2024-03-20T09:31:46.274 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.274 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.274 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.274 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:46.274 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.261807+0000", 2024-03-20T09:31:46.274 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.001e-06 2024-03-20T09:31:46.274 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.274 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.275 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:46.275 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.261884+0000", 2024-03-20T09:31:46.275 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7506e-05 2024-03-20T09:31:46.275 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.275 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.275 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:46.275 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.261896+0000", 2024-03-20T09:31:46.275 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1725999999999999e-05 2024-03-20T09:31:46.275 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.275 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.275 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:46.275 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.261903+0000", 2024-03-20T09:31:46.275 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.148e-06 2024-03-20T09:31:46.275 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.276 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.276 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:46.276 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.261965+0000", 2024-03-20T09:31:46.276 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1789000000000001e-05 2024-03-20T09:31:46.276 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.276 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.276 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:46.276 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.262086+0000", 2024-03-20T09:31:46.276 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000120986 2024-03-20T09:31:46.276 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.276 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.276 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,11", 2024-03-20T09:31:46.276 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.262233+0000", 2024-03-20T09:31:46.276 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00014692700000000001 2024-03-20T09:31:46.276 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.277 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.277 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:31:46.277 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.263109+0000", 2024-03-20T09:31:46.277 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00087649799999999997 2024-03-20T09:31:46.277 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.277 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.277 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:31:46.277 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.264119+0000", 2024-03-20T09:31:46.277 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00100994 2024-03-20T09:31:46.277 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.277 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.277 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:31:46.277 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.948685+0000", 2024-03-20T09:31:46.277 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.684565557999999 2024-03-20T09:31:46.278 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.278 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.278 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:46.278 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.948719+0000", 2024-03-20T09:31:46.278 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.4150000000000003e-05 2024-03-20T09:31:46.278 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.278 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.278 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:46.278 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.948779+0000", 2024-03-20T09:31:46.278 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9478000000000002e-05 2024-03-20T09:31:46.278 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.278 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:46.278 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.278 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.279 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.279 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:27794 6.5 6:aca21792:::benchmark_data_smithi122_168655_object27793:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e722)", 2024-03-20T09:31:46.279 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:24:26.267124+0000", 2024-03-20T09:31:46.279 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 439.99127161799998, 2024-03-20T09:31:46.279 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.685529216000001, 2024-03-20T09:31:46.279 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:46.279 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:46.279 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:31:46.279 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:31:46.279 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:31:46.279 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 27794 2024-03-20T09:31:46.279 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.279 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:46.279 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.279 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:46.280 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.267124+0000", 2024-03-20T09:31:46.280 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.280 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.280 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.280 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:46.280 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.267124+0000", 2024-03-20T09:31:46.280 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.280 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.280 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.280 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:46.280 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.267125+0000", 2024-03-20T09:31:46.280 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.032e-06 2024-03-20T09:31:46.280 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.280 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.281 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:46.281 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.267167+0000", 2024-03-20T09:31:46.281 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.2706000000000001e-05 2024-03-20T09:31:46.281 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.281 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.281 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:46.281 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.267186+0000", 2024-03-20T09:31:46.281 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8581e-05 2024-03-20T09:31:46.281 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.281 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.281 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:46.281 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.267193+0000", 2024-03-20T09:31:46.281 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.9700000000000002e-06 2024-03-20T09:31:46.281 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.282 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.282 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:46.282 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.267239+0000", 2024-03-20T09:31:46.282 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6238e-05 2024-03-20T09:31:46.282 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.282 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.282 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:46.282 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.267337+0000", 2024-03-20T09:31:46.282 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.7416000000000004e-05 2024-03-20T09:31:46.282 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.282 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.282 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:31:46.282 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.267425+0000", 2024-03-20T09:31:46.283 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.8558000000000002e-05 2024-03-20T09:31:46.283 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.283 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.283 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:31:46.283 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.268432+0000", 2024-03-20T09:31:46.283 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001006744 2024-03-20T09:31:46.283 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.283 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.283 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:31:46.283 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.268810+0000", 2024-03-20T09:31:46.283 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00037827299999999999 2024-03-20T09:31:46.283 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.283 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.283 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:31:46.283 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.952575+0000", 2024-03-20T09:31:46.284 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.683765093 2024-03-20T09:31:46.284 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.284 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.284 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:46.284 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.952611+0000", 2024-03-20T09:31:46.284 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.5329e-05 2024-03-20T09:31:46.284 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.284 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.284 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:46.284 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.952653+0000", 2024-03-20T09:31:46.284 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.2276e-05 2024-03-20T09:31:46.284 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.284 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:46.284 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.285 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.285 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.285 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:27796 6.3 6:c91031ab:::benchmark_data_smithi122_168655_object27795:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e722)", 2024-03-20T09:31:46.285 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:24:26.267834+0000", 2024-03-20T09:31:46.285 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 439.99056123399998, 2024-03-20T09:31:46.285 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.681402842000001, 2024-03-20T09:31:46.285 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:46.285 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:46.285 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:31:46.285 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:31:46.285 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:31:46.285 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 27796 2024-03-20T09:31:46.285 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.285 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:46.286 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.286 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:46.286 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.267834+0000", 2024-03-20T09:31:46.286 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.286 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.286 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.286 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:46.286 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.267834+0000", 2024-03-20T09:31:46.286 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.286 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.286 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.286 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:46.286 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.267835+0000", 2024-03-20T09:31:46.286 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2200000000000004e-07 2024-03-20T09:31:46.287 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.287 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.287 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:46.287 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.267892+0000", 2024-03-20T09:31:46.287 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7289e-05 2024-03-20T09:31:46.287 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.287 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.287 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:46.287 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.267902+0000", 2024-03-20T09:31:46.287 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0212e-05 2024-03-20T09:31:46.287 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.287 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.287 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:46.287 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.267907+0000", 2024-03-20T09:31:46.288 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6689999999999997e-06 2024-03-20T09:31:46.288 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.288 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.288 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:46.288 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.267950+0000", 2024-03-20T09:31:46.288 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.3646000000000002e-05 2024-03-20T09:31:46.288 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.288 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.288 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:46.288 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.268071+0000", 2024-03-20T09:31:46.288 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000121023 2024-03-20T09:31:46.288 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.288 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.288 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,11", 2024-03-20T09:31:46.289 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.268177+0000", 2024-03-20T09:31:46.289 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000105575 2024-03-20T09:31:46.289 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.289 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.289 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:31:46.289 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.269148+0000", 2024-03-20T09:31:46.289 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00097128000000000004 2024-03-20T09:31:46.289 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.289 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.289 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:31:46.289 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.269914+0000", 2024-03-20T09:31:46.289 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00076567599999999997 2024-03-20T09:31:46.289 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.289 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.290 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:31:46.290 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.949132+0000", 2024-03-20T09:31:46.290 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.679217613 2024-03-20T09:31:46.290 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.290 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.290 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:46.290 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.949185+0000", 2024-03-20T09:31:46.290 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3606000000000001e-05 2024-03-20T09:31:46.290 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.290 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.290 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:46.290 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.949237+0000", 2024-03-20T09:31:46.290 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1631000000000001e-05 2024-03-20T09:31:46.290 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.291 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:46.291 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.291 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.291 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.291 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:27809 6.5 6:af1ebbd3:::benchmark_data_smithi122_168655_object27808:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e722)", 2024-03-20T09:31:46.291 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:24:26.272616+0000", 2024-03-20T09:31:46.291 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 439.985779127, 2024-03-20T09:31:46.291 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.680115118, 2024-03-20T09:31:46.291 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:46.291 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:46.291 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:31:46.291 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:31:46.291 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:31:46.291 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 27809 2024-03-20T09:31:46.292 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.292 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:46.292 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.292 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:46.292 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.272616+0000", 2024-03-20T09:31:46.292 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.292 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.292 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.292 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:46.292 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.272616+0000", 2024-03-20T09:31:46.292 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.292 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.292 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.292 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:46.293 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.272618+0000", 2024-03-20T09:31:46.293 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6980000000000001e-06 2024-03-20T09:31:46.293 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.293 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.293 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:46.293 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.272665+0000", 2024-03-20T09:31:46.293 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.702e-05 2024-03-20T09:31:46.293 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.293 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.293 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:46.293 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.272679+0000", 2024-03-20T09:31:46.293 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4620000000000001e-05 2024-03-20T09:31:46.293 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.293 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.294 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:46.294 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.272688+0000", 2024-03-20T09:31:46.294 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.303e-06 2024-03-20T09:31:46.294 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.294 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.294 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:46.294 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.272748+0000", 2024-03-20T09:31:46.294 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0643000000000002e-05 2024-03-20T09:31:46.294 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.294 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.294 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:46.294 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.272860+0000", 2024-03-20T09:31:46.295 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000111921 2024-03-20T09:31:46.295 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.295 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.295 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:31:46.295 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.272969+0000", 2024-03-20T09:31:46.295 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00010882099999999999 2024-03-20T09:31:46.295 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.295 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.295 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:31:46.295 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.275264+0000", 2024-03-20T09:31:46.295 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.002294618 2024-03-20T09:31:46.295 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.295 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.295 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:31:46.296 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.275282+0000", 2024-03-20T09:31:46.296 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8516999999999999e-05 2024-03-20T09:31:46.296 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.296 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.296 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:31:46.296 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.952675+0000", 2024-03-20T09:31:46.296 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.677393199000001 2024-03-20T09:31:46.296 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.296 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.296 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:46.296 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.952702+0000", 2024-03-20T09:31:46.296 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6358000000000001e-05 2024-03-20T09:31:46.296 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.296 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.296 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:46.297 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.952731+0000", 2024-03-20T09:31:46.297 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.94e-05 2024-03-20T09:31:46.297 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.297 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:46.297 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.297 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.297 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.297 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:27810 6.5 6:a94a99a0:::benchmark_data_smithi122_168655_object27809:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e722)", 2024-03-20T09:31:46.297 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:24:26.273015+0000", 2024-03-20T09:31:46.297 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 439.985380265, 2024-03-20T09:31:46.297 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.679854547, 2024-03-20T09:31:46.297 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:46.297 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:46.298 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:31:46.298 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:31:46.298 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:31:46.298 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 27810 2024-03-20T09:31:46.298 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.298 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:46.298 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.298 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:46.298 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.273015+0000", 2024-03-20T09:31:46.298 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.298 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.298 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.299 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:46.299 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.273015+0000", 2024-03-20T09:31:46.299 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.299 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.299 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.299 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:46.299 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.273016+0000", 2024-03-20T09:31:46.299 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.0100000000000004e-07 2024-03-20T09:31:46.299 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.299 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.299 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:46.299 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.273089+0000", 2024-03-20T09:31:46.299 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3157000000000004e-05 2024-03-20T09:31:46.299 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.300 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.300 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:46.300 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.273106+0000", 2024-03-20T09:31:46.300 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6871000000000002e-05 2024-03-20T09:31:46.300 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.300 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.300 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:46.300 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.273117+0000", 2024-03-20T09:31:46.300 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1401e-05 2024-03-20T09:31:46.300 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.300 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.300 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:46.300 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.274101+0000", 2024-03-20T09:31:46.300 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00098364900000000007 2024-03-20T09:31:46.301 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.301 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.301 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:46.301 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.274290+0000", 2024-03-20T09:31:46.301 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00018891700000000001 2024-03-20T09:31:46.301 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.301 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.301 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:31:46.301 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.274428+0000", 2024-03-20T09:31:46.301 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000138508 2024-03-20T09:31:46.301 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.301 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.301 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:31:46.301 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.276075+0000", 2024-03-20T09:31:46.301 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0016466409999999999 2024-03-20T09:31:46.302 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.302 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.302 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:31:46.302 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.276837+0000", 2024-03-20T09:31:46.302 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00076209300000000004 2024-03-20T09:31:46.302 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.302 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.302 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:31:46.302 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.952809+0000", 2024-03-20T09:31:46.305 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.675971977 2024-03-20T09:31:46.306 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.306 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.306 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:46.306 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.952838+0000", 2024-03-20T09:31:46.306 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9349999999999999e-05 2024-03-20T09:31:46.306 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.306 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.306 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:46.306 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.952870+0000", 2024-03-20T09:31:46.306 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.1182000000000001e-05 2024-03-20T09:31:46.306 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.306 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:46.307 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.307 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.307 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.307 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:27815 6.3 6:cdefc7cc:::benchmark_data_smithi122_168655_object27814:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e722)", 2024-03-20T09:31:46.307 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:24:26.274340+0000", 2024-03-20T09:31:46.307 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 439.9840552, 2024-03-20T09:31:46.307 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.675027760000001, 2024-03-20T09:31:46.307 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:46.307 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:46.307 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:31:46.307 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:31:46.307 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:31:46.307 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 27815 2024-03-20T09:31:46.308 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.308 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:46.308 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.308 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:46.308 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.274340+0000", 2024-03-20T09:31:46.308 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.308 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.308 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.308 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:46.308 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.274340+0000", 2024-03-20T09:31:46.308 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.308 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.308 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.308 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:46.308 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.274341+0000", 2024-03-20T09:31:46.309 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.4200000000000005e-07 2024-03-20T09:31:46.309 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.309 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.309 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:46.309 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.274372+0000", 2024-03-20T09:31:46.309 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.1476999999999997e-05 2024-03-20T09:31:46.309 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.309 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.309 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:46.309 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.274381+0000", 2024-03-20T09:31:46.309 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.4139999999999994e-06 2024-03-20T09:31:46.309 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.309 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.309 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:46.310 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.274387+0000", 2024-03-20T09:31:46.310 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.6880000000000002e-06 2024-03-20T09:31:46.310 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.310 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.310 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:46.310 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.274442+0000", 2024-03-20T09:31:46.310 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4400000000000001e-05 2024-03-20T09:31:46.310 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.310 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.310 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:46.310 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.274559+0000", 2024-03-20T09:31:46.310 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000117246 2024-03-20T09:31:46.310 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.310 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.311 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,11", 2024-03-20T09:31:46.311 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.274720+0000", 2024-03-20T09:31:46.311 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00016123400000000001 2024-03-20T09:31:46.311 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.311 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.311 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:31:46.311 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.276663+0000", 2024-03-20T09:31:46.311 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0019423660000000001 2024-03-20T09:31:46.311 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.311 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.311 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:31:46.311 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.276683+0000", 2024-03-20T09:31:46.311 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0664000000000001e-05 2024-03-20T09:31:46.311 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.312 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.312 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:31:46.312 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.949272+0000", 2024-03-20T09:31:46.312 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.672588147000001 2024-03-20T09:31:46.312 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.312 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.312 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:46.312 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.949319+0000", 2024-03-20T09:31:46.312 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.7284999999999999e-05 2024-03-20T09:31:46.312 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.312 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.312 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:46.312 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.949368+0000", 2024-03-20T09:31:46.313 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8996999999999999e-05 2024-03-20T09:31:46.313 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.313 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:46.313 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.313 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.313 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.313 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:27764 6.3 6:c157248a:::benchmark_data_smithi122_168655_object27763:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e722)", 2024-03-20T09:31:46.313 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:24:26.284281+0000", 2024-03-20T09:31:46.313 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 439.97411461899998, 2024-03-20T09:31:46.313 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.663006097, 2024-03-20T09:31:46.313 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:46.313 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for ondisk", 2024-03-20T09:31:46.313 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:31:46.314 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:31:46.314 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:31:46.314 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 27764 2024-03-20T09:31:46.314 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.314 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:46.314 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.314 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:46.314 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284281+0000", 2024-03-20T09:31:46.314 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.314 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.314 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.314 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:46.314 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284281+0000", 2024-03-20T09:31:46.315 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.315 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.315 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.315 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:46.315 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284282+0000", 2024-03-20T09:31:46.315 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.034e-06 2024-03-20T09:31:46.315 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.315 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.315 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:46.315 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284446+0000", 2024-03-20T09:31:46.315 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00016390799999999999 2024-03-20T09:31:46.316 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.316 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.316 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:46.316 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284456+0000", 2024-03-20T09:31:46.316 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0513000000000001e-05 2024-03-20T09:31:46.316 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.316 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.316 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:46.316 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284464+0000", 2024-03-20T09:31:46.316 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.0760000000000003e-06 2024-03-20T09:31:46.316 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.316 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.316 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:46.316 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284551+0000", 2024-03-20T09:31:46.317 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.6704000000000002e-05 2024-03-20T09:31:46.317 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.317 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.317 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for ondisk", 2024-03-20T09:31:46.317 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284646+0000", 2024-03-20T09:31:46.317 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.5379999999999998e-05 2024-03-20T09:31:46.317 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.317 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.317 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:46.317 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.947287+0000", 2024-03-20T09:31:46.317 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.662640482 2024-03-20T09:31:46.317 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.317 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:46.317 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.318 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.318 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.318 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:27780 6.3 6:cea521fc:::benchmark_data_smithi122_168655_object27779:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e722)", 2024-03-20T09:31:46.318 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:24:26.284482+0000", 2024-03-20T09:31:46.318 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 439.97391316400001, 2024-03-20T09:31:46.318 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.664129623000001, 2024-03-20T09:31:46.318 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:46.318 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for ondisk", 2024-03-20T09:31:46.318 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:31:46.318 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:31:46.318 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:31:46.318 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 27780 2024-03-20T09:31:46.318 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.319 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:46.319 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.319 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:46.319 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284482+0000", 2024-03-20T09:31:46.319 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.319 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.319 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.319 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:46.319 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284482+0000", 2024-03-20T09:31:46.319 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.319 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.319 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.319 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:46.320 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284483+0000", 2024-03-20T09:31:46.320 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.8299999999999996e-07 2024-03-20T09:31:46.320 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.320 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.320 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:46.320 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284516+0000", 2024-03-20T09:31:46.320 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.3677000000000003e-05 2024-03-20T09:31:46.320 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.320 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.320 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:46.320 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284528+0000", 2024-03-20T09:31:46.320 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1223000000000001e-05 2024-03-20T09:31:46.320 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.321 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.321 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:46.321 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284537+0000", 2024-03-20T09:31:46.321 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.1719999999999996e-06 2024-03-20T09:31:46.321 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.321 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.321 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:46.321 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284702+0000", 2024-03-20T09:31:46.321 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000164711 2024-03-20T09:31:46.321 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.321 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.321 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for ondisk", 2024-03-20T09:31:46.321 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284754+0000", 2024-03-20T09:31:46.321 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2601e-05 2024-03-20T09:31:46.321 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.322 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.322 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:46.322 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.948612+0000", 2024-03-20T09:31:46.322 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.663857556 2024-03-20T09:31:46.322 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.322 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:46.322 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.322 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.322 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.322 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:27784 6.5 6:a86bc021:::benchmark_data_smithi122_168655_object27783:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e722)", 2024-03-20T09:31:46.322 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:24:26.284546+0000", 2024-03-20T09:31:46.322 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 439.97384897500001, 2024-03-20T09:31:46.322 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.663221893999999, 2024-03-20T09:31:46.323 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:46.323 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for ondisk", 2024-03-20T09:31:46.323 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:31:46.323 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:31:46.323 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:31:46.323 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 27784 2024-03-20T09:31:46.323 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.323 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:46.323 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.323 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:46.323 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284546+0000", 2024-03-20T09:31:46.323 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.323 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.323 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.323 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:46.323 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284546+0000", 2024-03-20T09:31:46.324 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.324 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.324 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.324 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:46.324 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284547+0000", 2024-03-20T09:31:46.324 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.5499999999999999e-07 2024-03-20T09:31:46.324 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.324 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.324 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:46.324 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284578+0000", 2024-03-20T09:31:46.324 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.1059000000000002e-05 2024-03-20T09:31:46.324 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.324 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.324 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:46.324 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284586+0000", 2024-03-20T09:31:46.325 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5189999999999992e-06 2024-03-20T09:31:46.325 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.325 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.325 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:46.325 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284593+0000", 2024-03-20T09:31:46.325 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.2459999999999997e-06 2024-03-20T09:31:46.325 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.325 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.325 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:46.325 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284644+0000", 2024-03-20T09:31:46.325 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0599999999999997e-05 2024-03-20T09:31:46.325 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.325 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.325 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for ondisk", 2024-03-20T09:31:46.326 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284708+0000", 2024-03-20T09:31:46.326 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4455000000000005e-05 2024-03-20T09:31:46.326 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.326 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.326 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:46.326 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.947768+0000", 2024-03-20T09:31:46.326 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.66305966 2024-03-20T09:31:46.326 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.326 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:46.326 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.326 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.326 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.326 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:27789 6.3 6:c6ec49ce:::benchmark_data_smithi122_168655_object27788:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e722)", 2024-03-20T09:31:46.326 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:24:26.284608+0000", 2024-03-20T09:31:46.327 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 439.97378710499999, 2024-03-20T09:31:46.327 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.664123590999999, 2024-03-20T09:31:46.327 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:46.327 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for ondisk", 2024-03-20T09:31:46.327 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:31:46.327 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:31:46.327 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:31:46.327 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 27789 2024-03-20T09:31:46.327 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.327 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:46.327 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.327 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:46.327 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284608+0000", 2024-03-20T09:31:46.327 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.328 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.328 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.328 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:46.328 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284608+0000", 2024-03-20T09:31:46.328 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.328 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.328 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.328 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:46.328 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284609+0000", 2024-03-20T09:31:46.328 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1200000000000003e-07 2024-03-20T09:31:46.328 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.328 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.328 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:46.328 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284639+0000", 2024-03-20T09:31:46.329 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.0003000000000001e-05 2024-03-20T09:31:46.329 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.329 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.329 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:46.329 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284645+0000", 2024-03-20T09:31:46.329 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4330000000000001e-06 2024-03-20T09:31:46.329 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.329 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.329 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:46.329 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284649+0000", 2024-03-20T09:31:46.329 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.5439999999999999e-06 2024-03-20T09:31:46.329 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.329 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.329 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:46.330 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284791+0000", 2024-03-20T09:31:46.330 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00014212900000000001 2024-03-20T09:31:46.330 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.330 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.330 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for ondisk", 2024-03-20T09:31:46.330 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284851+0000", 2024-03-20T09:31:46.330 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0123e-05 2024-03-20T09:31:46.330 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.330 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.330 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:46.330 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.948732+0000", 2024-03-20T09:31:46.330 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.663880847 2024-03-20T09:31:46.330 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.331 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:46.331 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.331 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.331 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.331 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:27794 6.5 6:aca21792:::benchmark_data_smithi122_168655_object27793:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e722)", 2024-03-20T09:31:46.331 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:24:26.284655+0000", 2024-03-20T09:31:46.331 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 439.973740077, 2024-03-20T09:31:46.331 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.667965731000001, 2024-03-20T09:31:46.331 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:46.331 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for ondisk", 2024-03-20T09:31:46.331 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:31:46.331 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:31:46.331 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:31:46.331 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 27794 2024-03-20T09:31:46.331 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.332 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:46.332 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.332 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:46.332 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284655+0000", 2024-03-20T09:31:46.332 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.332 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.332 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.332 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:46.332 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284655+0000", 2024-03-20T09:31:46.332 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.332 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.332 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.332 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:46.332 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284655+0000", 2024-03-20T09:31:46.332 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.0499999999999999e-07 2024-03-20T09:31:46.333 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.333 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.333 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:46.333 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284782+0000", 2024-03-20T09:31:46.333 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012621600000000001 2024-03-20T09:31:46.333 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.333 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.333 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:46.333 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284790+0000", 2024-03-20T09:31:46.333 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.9920000000000007e-06 2024-03-20T09:31:46.333 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.333 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.333 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:46.333 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284794+0000", 2024-03-20T09:31:46.334 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4769999999999997e-06 2024-03-20T09:31:46.334 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.334 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.334 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:46.334 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284842+0000", 2024-03-20T09:31:46.334 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8182e-05 2024-03-20T09:31:46.334 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.334 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.334 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for ondisk", 2024-03-20T09:31:46.334 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284893+0000", 2024-03-20T09:31:46.334 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.083e-05 2024-03-20T09:31:46.334 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.334 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.335 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:46.335 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.952621+0000", 2024-03-20T09:31:46.335 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.667727728999999 2024-03-20T09:31:46.335 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.335 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:46.335 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.335 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.335 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.335 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:27796 6.3 6:c91031ab:::benchmark_data_smithi122_168655_object27795:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e722)", 2024-03-20T09:31:46.335 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:24:26.284809+0000", 2024-03-20T09:31:46.335 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 439.97358614000001, 2024-03-20T09:31:46.335 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.664389506999999, 2024-03-20T09:31:46.335 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:46.335 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for ondisk", 2024-03-20T09:31:46.336 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:31:46.336 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:31:46.336 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:31:46.336 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 27796 2024-03-20T09:31:46.336 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.336 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:46.336 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.336 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:46.336 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284809+0000", 2024-03-20T09:31:46.336 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.336 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.336 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.336 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:46.336 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284809+0000", 2024-03-20T09:31:46.337 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.337 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.337 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.337 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:46.337 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284810+0000", 2024-03-20T09:31:46.337 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.9100000000000003e-07 2024-03-20T09:31:46.337 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.337 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.337 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:46.337 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284837+0000", 2024-03-20T09:31:46.337 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.7245000000000001e-05 2024-03-20T09:31:46.337 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.337 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.337 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:46.337 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284842+0000", 2024-03-20T09:31:46.338 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3850000000000001e-06 2024-03-20T09:31:46.338 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.338 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.338 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:46.338 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284847+0000", 2024-03-20T09:31:46.338 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5549999999999996e-06 2024-03-20T09:31:46.338 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.338 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.338 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:46.338 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284895+0000", 2024-03-20T09:31:46.338 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8056000000000002e-05 2024-03-20T09:31:46.338 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.338 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.339 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for ondisk", 2024-03-20T09:31:46.339 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284958+0000", 2024-03-20T09:31:46.339 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.3266e-05 2024-03-20T09:31:46.339 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.339 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.339 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:46.339 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.949199+0000", 2024-03-20T09:31:46.339 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.664240309 2024-03-20T09:31:46.339 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.339 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:46.339 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.339 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.339 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.339 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:27809 6.5 6:af1ebbd3:::benchmark_data_smithi122_168655_object27808:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e722)", 2024-03-20T09:31:46.340 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:24:26.284855+0000", 2024-03-20T09:31:46.340 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 439.973540464, 2024-03-20T09:31:46.340 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.667852657999999, 2024-03-20T09:31:46.340 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:46.340 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for ondisk", 2024-03-20T09:31:46.340 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:31:46.340 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:31:46.340 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:31:46.340 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 27809 2024-03-20T09:31:46.340 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.340 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:46.340 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.340 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:46.341 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284855+0000", 2024-03-20T09:31:46.341 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.341 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.341 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.341 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:46.341 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284855+0000", 2024-03-20T09:31:46.341 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.341 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.341 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.341 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:46.341 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284855+0000", 2024-03-20T09:31:46.341 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7800000000000001e-07 2024-03-20T09:31:46.341 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.341 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.341 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:46.342 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284883+0000", 2024-03-20T09:31:46.342 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.7455999999999999e-05 2024-03-20T09:31:46.342 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.342 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.342 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:46.342 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284889+0000", 2024-03-20T09:31:46.342 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2890000000000003e-06 2024-03-20T09:31:46.342 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.342 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.342 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:46.342 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284894+0000", 2024-03-20T09:31:46.342 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8559999999999998e-06 2024-03-20T09:31:46.342 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.343 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.343 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:46.343 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284937+0000", 2024-03-20T09:31:46.343 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.2867000000000001e-05 2024-03-20T09:31:46.343 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.343 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.343 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for ondisk", 2024-03-20T09:31:46.343 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284995+0000", 2024-03-20T09:31:46.343 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8335999999999998e-05 2024-03-20T09:31:46.343 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.343 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.343 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:46.343 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.952707+0000", 2024-03-20T09:31:46.343 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.667712276 2024-03-20T09:31:46.344 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.344 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:46.344 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.344 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.344 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.344 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:27810 6.5 6:a94a99a0:::benchmark_data_smithi122_168655_object27809:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e722)", 2024-03-20T09:31:46.344 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:24:26.284907+0000", 2024-03-20T09:31:46.344 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 439.97348823800002, 2024-03-20T09:31:46.344 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.66793777, 2024-03-20T09:31:46.344 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:46.344 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for ondisk", 2024-03-20T09:31:46.344 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:31:46.344 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:31:46.344 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:31:46.345 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 27810 2024-03-20T09:31:46.345 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.345 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:46.345 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.345 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:46.345 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284907+0000", 2024-03-20T09:31:46.345 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.345 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.345 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.345 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:46.345 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284907+0000", 2024-03-20T09:31:46.345 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.345 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.345 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.346 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:46.346 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284907+0000", 2024-03-20T09:31:46.346 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.58e-07 2024-03-20T09:31:46.346 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.346 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.346 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:46.346 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284945+0000", 2024-03-20T09:31:46.346 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7991999999999998e-05 2024-03-20T09:31:46.346 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.346 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.346 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:46.346 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284953+0000", 2024-03-20T09:31:46.346 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3520000000000001e-06 2024-03-20T09:31:46.347 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.347 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.347 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:46.347 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284959+0000", 2024-03-20T09:31:46.347 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.84e-06 2024-03-20T09:31:46.347 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.347 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.347 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:46.347 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.285020+0000", 2024-03-20T09:31:46.347 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1742999999999995e-05 2024-03-20T09:31:46.347 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.347 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.347 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for ondisk", 2024-03-20T09:31:46.347 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.285079+0000", 2024-03-20T09:31:46.348 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8390000000000002e-05 2024-03-20T09:31:46.348 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.348 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.348 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:46.348 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.952845+0000", 2024-03-20T09:31:46.348 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.667765995 2024-03-20T09:31:46.348 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.348 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:46.348 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.348 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.348 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.348 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:27815 6.3 6:cdefc7cc:::benchmark_data_smithi122_168655_object27814:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e722)", 2024-03-20T09:31:46.348 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:24:26.284967+0000", 2024-03-20T09:31:46.348 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 439.97342787899998, 2024-03-20T09:31:46.348 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.664357655, 2024-03-20T09:31:46.349 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:46.349 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for ondisk", 2024-03-20T09:31:46.349 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:31:46.349 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:31:46.349 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:31:46.349 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 27815 2024-03-20T09:31:46.349 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.349 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:46.349 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.349 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:46.349 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284967+0000", 2024-03-20T09:31:46.349 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.349 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.349 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.350 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:46.350 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284967+0000", 2024-03-20T09:31:46.350 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.350 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.350 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.350 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:46.350 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284968+0000", 2024-03-20T09:31:46.350 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2200000000000004e-07 2024-03-20T09:31:46.350 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.350 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.350 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:46.350 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.284995+0000", 2024-03-20T09:31:46.350 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.747e-05 2024-03-20T09:31:46.351 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.351 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.351 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:46.351 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.285000+0000", 2024-03-20T09:31:46.351 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9799999999999998e-06 2024-03-20T09:31:46.351 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.351 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.351 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:46.351 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.285006+0000", 2024-03-20T09:31:46.351 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7599999999999999e-06 2024-03-20T09:31:46.351 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.351 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.351 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:46.351 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.285051+0000", 2024-03-20T09:31:46.351 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4824000000000003e-05 2024-03-20T09:31:46.352 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.352 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.352 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for ondisk", 2024-03-20T09:31:46.352 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.285115+0000", 2024-03-20T09:31:46.352 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4344000000000006e-05 2024-03-20T09:31:46.352 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.352 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.352 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:46.352 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.949325+0000", 2024-03-20T09:31:46.352 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.664209655000001 2024-03-20T09:31:46.352 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.352 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:46.352 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.352 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.353 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.353 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:27832 6.3 6:c4f6abb6:::benchmark_data_smithi122_168655_object27831:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e722)", 2024-03-20T09:31:46.353 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:24:26.285017+0000", 2024-03-20T09:31:46.353 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 439.97337855000001, 2024-03-20T09:31:46.353 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.664482944, 2024-03-20T09:31:46.353 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:46.353 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:46.353 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:31:46.353 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:31:46.353 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:31:46.353 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 27832 2024-03-20T09:31:46.353 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.353 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:46.353 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.354 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:46.354 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.285017+0000", 2024-03-20T09:31:46.354 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.354 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.354 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.354 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:46.354 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.285017+0000", 2024-03-20T09:31:46.354 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.354 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.354 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.354 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:46.354 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.285017+0000", 2024-03-20T09:31:46.354 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9800000000000004e-07 2024-03-20T09:31:46.354 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.355 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.355 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:46.355 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.285043+0000", 2024-03-20T09:31:46.355 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5973e-05 2024-03-20T09:31:46.355 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.355 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.355 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:46.355 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.285050+0000", 2024-03-20T09:31:46.355 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.691e-06 2024-03-20T09:31:46.355 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.355 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.355 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:46.355 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.285055+0000", 2024-03-20T09:31:46.355 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3480000000000003e-06 2024-03-20T09:31:46.356 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.356 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.356 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:46.356 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.285145+0000", 2024-03-20T09:31:46.356 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.0056999999999995e-05 2024-03-20T09:31:46.356 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.356 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.356 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:46.356 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.285296+0000", 2024-03-20T09:31:46.356 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00015097400000000001 2024-03-20T09:31:46.356 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.356 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.356 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,11", 2024-03-20T09:31:46.356 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.285441+0000", 2024-03-20T09:31:46.357 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000144365 2024-03-20T09:31:46.357 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.357 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.357 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:31:46.357 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.286512+0000", 2024-03-20T09:31:46.357 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0010709280000000001 2024-03-20T09:31:46.357 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.357 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.357 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:31:46.357 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.287123+0000", 2024-03-20T09:31:46.357 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000611248 2024-03-20T09:31:46.357 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.357 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.357 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:31:46.358 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.949405+0000", 2024-03-20T09:31:46.358 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.662282682000001 2024-03-20T09:31:46.358 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.358 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.358 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:46.358 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.949460+0000", 2024-03-20T09:31:46.358 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4246999999999997e-05 2024-03-20T09:31:46.358 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.358 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.358 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:46.358 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.949500+0000", 2024-03-20T09:31:46.358 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.9932999999999999e-05 2024-03-20T09:31:46.358 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.359 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:46.359 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.359 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.359 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.359 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:27841 6.5 6:a63c32c6:::benchmark_data_smithi122_168655_object27840:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e722)", 2024-03-20T09:31:46.359 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:24:26.287822+0000", 2024-03-20T09:31:46.359 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 439.97057322799998, 2024-03-20T09:31:46.359 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.665164598, 2024-03-20T09:31:46.359 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:46.359 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:46.359 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:31:46.359 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:31:46.359 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:31:46.359 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 27841 2024-03-20T09:31:46.360 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.360 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:46.360 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.360 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:46.360 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.287822+0000", 2024-03-20T09:31:46.360 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.360 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.360 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.360 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:46.360 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.287822+0000", 2024-03-20T09:31:46.360 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:46.360 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.360 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.360 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:46.360 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.287823+0000", 2024-03-20T09:31:46.361 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1570000000000001e-06 2024-03-20T09:31:46.361 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.361 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.361 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:46.361 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.287888+0000", 2024-03-20T09:31:46.361 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4547000000000004e-05 2024-03-20T09:31:46.361 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.361 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.361 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:46.361 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.287902+0000", 2024-03-20T09:31:46.361 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.437e-05 2024-03-20T09:31:46.361 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.361 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.362 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:46.362 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.287910+0000", 2024-03-20T09:31:46.362 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.2090000000000005e-06 2024-03-20T09:31:46.362 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.362 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.362 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:46.362 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.287969+0000", 2024-03-20T09:31:46.362 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8647999999999998e-05 2024-03-20T09:31:46.362 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.362 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.362 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:46.362 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.288102+0000", 2024-03-20T09:31:46.362 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013281799999999999 2024-03-20T09:31:46.362 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.363 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.363 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:31:46.363 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.288243+0000", 2024-03-20T09:31:46.363 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00014170799999999999 2024-03-20T09:31:46.363 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.363 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.363 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:31:46.363 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.289363+0000", 2024-03-20T09:31:46.363 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0011194620000000001 2024-03-20T09:31:46.363 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.363 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.363 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:31:46.363 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:26.289536+0000", 2024-03-20T09:31:46.364 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00017353599999999999 2024-03-20T09:31:46.364 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.364 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.364 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:31:46.364 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.952906+0000", 2024-03-20T09:31:46.364 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 11.663370039 2024-03-20T09:31:46.364 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.364 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.364 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:46.364 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.952953+0000", 2024-03-20T09:31:46.364 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6694999999999999e-05 2024-03-20T09:31:46.364 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:46.364 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:46.364 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:46.365 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:24:37.952987+0000", 2024-03-20T09:31:46.365 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.3408999999999998e-05 2024-03-20T09:31:46.365 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.365 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:46.365 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.365 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:46.365 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:46.365 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:31:46.365 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:31:46.366 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:31:46.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:46.367 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:31:46.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:31:46.368 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:31:46.368 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:31:46.405 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:31:46.449 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:31:46.468 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:31:46.469 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:31:46.469 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:31:46.469 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:31:46.469 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:31:46.469 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:31:46.469 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:31:46.469 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:31:46.469 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:31:46.469 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:31:46.469 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:31:46.470 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:31:46.470 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:31:46.585 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:46.643 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 28 16 59025 59009 131.691 90.5625 0.0157732 0.00758981 2024-03-20T09:31:46.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:46 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:31:46.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:46 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:31:46.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:46 smithi115 ceph-mon[72476]: osdmap e885: 12 total, 12 up, 11 in 2024-03-20T09:31:46.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:46 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2456417918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:31:46.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:46 smithi115 ceph-mon[72476]: pgmap v2140: 47 pgs: 47 active+clean; 3.4 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-03-20T09:31:46.993 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-20T09:31:47.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:46 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:31:47.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:46 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:31:47.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:46 smithi122 ceph-mon[73495]: osdmap e885: 12 total, 12 up, 11 in 2024-03-20T09:31:47.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:46 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2456417918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:31:47.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:46 smithi122 ceph-mon[73495]: pgmap v2140: 47 pgs: 47 active+clean; 3.4 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-03-20T09:31:47.217 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:46 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:31:47.217 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:46 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:31:47.217 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:46 smithi026 ceph-mon[68309]: osdmap e885: 12 total, 12 up, 11 in 2024-03-20T09:31:47.217 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:46 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2456417918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:31:47.217 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:46 smithi026 ceph-mon[68309]: pgmap v2140: 47 pgs: 47 active+clean; 3.4 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-03-20T09:31:47.381 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:31:47.470 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:31:47.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:47.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:31:47.579 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:31:47.612 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:47.638 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:31:47.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:47.643 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:31:47.643 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 29 16 60634 60618 130.617 100.562 0.0199473 0.00765203 2024-03-20T09:31:47.744 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:31:47.782 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:47.804 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:31:47.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:47.807 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:31:47.907 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:31:47.938 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:47 smithi115 ceph-mon[72476]: 1.6 scrub starts 2024-03-20T09:31:47.939 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:47 smithi115 ceph-mon[72476]: 1.6 scrub ok 2024-03-20T09:31:47.939 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:47 smithi115 ceph-mon[72476]: osdmap e886: 12 total, 12 up, 11 in 2024-03-20T09:31:47.942 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:48.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:47 smithi122 ceph-mon[73495]: 1.6 scrub starts 2024-03-20T09:31:48.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:47 smithi122 ceph-mon[73495]: 1.6 scrub ok 2024-03-20T09:31:48.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:47 smithi122 ceph-mon[73495]: osdmap e886: 12 total, 12 up, 11 in 2024-03-20T09:31:48.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:47 smithi026 ceph-mon[68309]: 1.6 scrub starts 2024-03-20T09:31:48.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:47 smithi026 ceph-mon[68309]: 1.6 scrub ok 2024-03-20T09:31:48.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:47 smithi026 ceph-mon[68309]: osdmap e886: 12 total, 12 up, 11 in 2024-03-20T09:31:48.501 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:31:48.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:48.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:31:48.607 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:31:48.629 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:31:48.643 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 30 16 62216 62200 129.559 98.875 0.023429 0.0077135 2024-03-20T09:31:48.693 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:48.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:48 smithi026 ceph-mon[68309]: pgmap v2142: 47 pgs: 47 active+clean; 3.6 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 92 MiB/s wr, 1.46k op/s 2024-03-20T09:31:49.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:48 smithi115 ceph-mon[72476]: pgmap v2142: 47 pgs: 47 active+clean; 3.6 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 92 MiB/s wr, 1.46k op/s 2024-03-20T09:31:49.177 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:31:49.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:49.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:31:49.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:48 smithi122 ceph-mon[73495]: pgmap v2142: 47 pgs: 47 active+clean; 3.6 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 92 MiB/s wr, 1.46k op/s 2024-03-20T09:31:49.281 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:31:49.316 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:49.346 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:31:49.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:49.353 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:31:49.454 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:31:49.490 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:49.517 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:31:49.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:49.520 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:31:49.621 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:31:49.643 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 31 16 63686 63670 128.343 91.875 0.00401139 0.00778765 2024-03-20T09:31:49.687 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:49.762 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:31:49.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:49.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:31:49.868 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:31:49.925 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.0 to 1 (10000) 2024-03-20T09:31:49.926 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:49.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:49 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1630171900' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-20T09:31:50.057 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:31:50.119 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:49 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1630171900' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-20T09:31:50.146 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:31:50.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:50.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:31:50.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:49 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1630171900' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-20T09:31:50.260 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:31:50.292 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:50.318 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:31:50.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:50.323 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:31:50.425 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:31:50.469 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:50.493 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:31:50.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:50.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:31:50.598 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:31:50.613 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:31:50.613 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:31:50.613 INFO:teuthology.orchestra.run.smithi115.stdout: "num_ops": 0 2024-03-20T09:31:50.613 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:31:50.643 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 32 16 65154 65138 127.199 91.75 0.00895131 0.00785782 2024-03-20T09:31:50.654 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:50.909 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:50 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:31:50.909 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:50 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:31:50.909 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:50 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1630171900' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-20T09:31:50.909 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:50 smithi026 ceph-mon[68309]: osdmap e887: 12 total, 12 up, 11 in 2024-03-20T09:31:50.909 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:50 smithi026 ceph-mon[68309]: pgmap v2144: 47 pgs: 47 active+clean; 3.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-03-20T09:31:51.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:50 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:31:51.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:50 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:31:51.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:50 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1630171900' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-20T09:31:51.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:50 smithi122 ceph-mon[73495]: osdmap e887: 12 total, 12 up, 11 in 2024-03-20T09:31:51.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:50 smithi122 ceph-mon[73495]: pgmap v2144: 47 pgs: 47 active+clean; 3.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-03-20T09:31:51.320 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:50 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:31:51.320 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:50 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:31:51.321 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:50 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1630171900' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-20T09:31:51.321 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:50 smithi115 ceph-mon[72476]: osdmap e887: 12 total, 12 up, 11 in 2024-03-20T09:31:51.321 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:50 smithi115 ceph-mon[72476]: pgmap v2144: 47 pgs: 47 active+clean; 3.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-03-20T09:31:51.366 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-20T09:31:51.643 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 33 16 66730 66714 126.33 98.5 0.00727066 0.0079121 2024-03-20T09:31:51.736 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:31:52.644 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 34 16 68293 68277 125.487 97.6875 0.0181911 0.00796414 2024-03-20T09:31:53.292 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:31:53.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:53.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:31:53.332 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 1 1 2024-03-20T09:31:53.399 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:31:53.490 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:31:53.527 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:53 smithi026 ceph-mon[68309]: pgmap v2145: 47 pgs: 47 active+clean; 3.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 107 MiB/s wr, 1.71k op/s 2024-03-20T09:31:53.528 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:53 smithi026 ceph-mon[68309]: osdmap e888: 12 total, 12 up, 11 in 2024-03-20T09:31:53.528 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:53 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:31:53.528 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:53 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:31:53.534 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:53.644 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 35 16 69662 69646 124.345 85.5625 0.0119926 0.00803712 2024-03-20T09:31:53.783 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:31:53.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:53.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:31:53.889 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:31:53.889 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:31:53.889 INFO:teuthology.orchestra.run.smithi115.stdout: "complaint_time": 30, 2024-03-20T09:31:53.889 INFO:teuthology.orchestra.run.smithi115.stdout: "num_blocked_ops": 0 2024-03-20T09:31:53.890 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:31:53.892 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:31:53.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:53 smithi115 ceph-mon[72476]: pgmap v2145: 47 pgs: 47 active+clean; 3.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 107 MiB/s wr, 1.71k op/s 2024-03-20T09:31:53.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:53 smithi115 ceph-mon[72476]: osdmap e888: 12 total, 12 up, 11 in 2024-03-20T09:31:53.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:53 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:31:53.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:53 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:31:53.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:53 smithi122 ceph-mon[73495]: pgmap v2145: 47 pgs: 47 active+clean; 3.9 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 107 MiB/s wr, 1.71k op/s 2024-03-20T09:31:53.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:53 smithi122 ceph-mon[73495]: osdmap e888: 12 total, 12 up, 11 in 2024-03-20T09:31:53.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:53 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:31:53.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:53 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:31:53.944 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:54.568 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:54 smithi115 ceph-mon[72476]: 1.9 scrub starts 2024-03-20T09:31:54.568 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:54 smithi115 ceph-mon[72476]: 1.9 scrub ok 2024-03-20T09:31:54.568 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:54 smithi115 ceph-mon[72476]: osdmap e889: 12 total, 12 up, 11 in 2024-03-20T09:31:54.594 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-20T09:31:54.644 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 36 16 70678 70662 122.655 63.5 0.00299537 0.00808754 2024-03-20T09:31:54.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:54 smithi026 ceph-mon[68309]: 1.9 scrub starts 2024-03-20T09:31:54.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:54 smithi026 ceph-mon[68309]: 1.9 scrub ok 2024-03-20T09:31:54.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:54 smithi026 ceph-mon[68309]: osdmap e889: 12 total, 12 up, 11 in 2024-03-20T09:31:54.853 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:31:54.905 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:31:54.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:54.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:31:54.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:54 smithi122 ceph-mon[73495]: 1.9 scrub starts 2024-03-20T09:31:54.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:54 smithi122 ceph-mon[73495]: 1.9 scrub ok 2024-03-20T09:31:54.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:54 smithi122 ceph-mon[73495]: osdmap e889: 12 total, 12 up, 11 in 2024-03-20T09:31:55.009 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:31:55.053 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:55.097 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:31:55.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:55.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:31:55.204 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:31:55.237 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:55.262 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:31:55.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:55.264 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:31:55.366 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:31:55.401 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:55.644 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 37 16 71675 71659 121.024 62.3125 0.00794401 0.00825874 2024-03-20T09:31:55.649 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:55 smithi115 ceph-mon[72476]: pgmap v2148: 47 pgs: 47 active+clean; 4.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-03-20T09:31:55.688 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:55 smithi122 ceph-mon[73495]: pgmap v2148: 47 pgs: 47 active+clean; 4.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-03-20T09:31:55.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:55 smithi026 ceph-mon[68309]: pgmap v2148: 47 pgs: 47 active+clean; 4.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-03-20T09:31:56.475 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:31:56.495 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:31:56.644 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 38 16 73107 73091 120.195 89.5 0.0142246 0.00831571 2024-03-20T09:31:56.648 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:56 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3944824153' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:31:56.648 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:56 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:31:56.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:56 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3944824153' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:31:56.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:56 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:31:56.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:56 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3944824153' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:31:56.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:56 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:31:57.458 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:31:57.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:57.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:31:57.564 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:31:57.612 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:57.644 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 39 16 74503 74487 119.35 87.25 0.00589959 0.00836379 2024-03-20T09:31:57.889 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:57 smithi115 ceph-mon[72476]: 1.1 scrub starts 2024-03-20T09:31:57.889 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:57 smithi115 ceph-mon[72476]: 1.1 scrub ok 2024-03-20T09:31:57.889 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:57 smithi115 ceph-mon[72476]: pgmap v2149: 47 pgs: 47 active+clean; 4.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-03-20T09:31:57.889 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:57 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:31:57.889 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:57 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:31:57.889 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:57 smithi115 ceph-mon[72476]: osdmap e890: 12 total, 12 up, 11 in 2024-03-20T09:31:57.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:57 smithi122 ceph-mon[73495]: 1.1 scrub starts 2024-03-20T09:31:57.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:57 smithi122 ceph-mon[73495]: 1.1 scrub ok 2024-03-20T09:31:57.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:57 smithi122 ceph-mon[73495]: pgmap v2149: 47 pgs: 47 active+clean; 4.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-03-20T09:31:57.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:57 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:31:57.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:57 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:31:57.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:57 smithi122 ceph-mon[73495]: osdmap e890: 12 total, 12 up, 11 in 2024-03-20T09:31:57.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:57 smithi026 ceph-mon[68309]: 1.1 scrub starts 2024-03-20T09:31:57.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:57 smithi026 ceph-mon[68309]: 1.1 scrub ok 2024-03-20T09:31:57.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:57 smithi026 ceph-mon[68309]: pgmap v2149: 47 pgs: 47 active+clean; 4.3 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-03-20T09:31:57.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:57 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:31:57.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:57 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:31:57.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:57 smithi026 ceph-mon[68309]: osdmap e890: 12 total, 12 up, 11 in 2024-03-20T09:31:58.644 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:2024-03-20T09:31:58.653176+0000 min lat: 0.00116204 max lat: 0.615754 avg lat: 0.00845274 2024-03-20T09:31:58.644 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T09:31:58.645 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 40 16 75706 75690 118.246 75.1875 0.0111712 0.00845274 2024-03-20T09:31:58.802 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:31:58.802 INFO:teuthology.orchestra.run.smithi115.stdout: "size": 20, 2024-03-20T09:31:58.802 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 600, 2024-03-20T09:31:58.802 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [ 2024-03-20T09:31:58.802 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.802 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:229938 5.9 e688/605 5:94157346:::benchmark_data_smithi122_157894_object99951:head v 688'14428, mlcod=688'14426)", 2024-03-20T09:31:58.802 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:57.310743+0000", 2024-03-20T09:31:58.802 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 541.49700979900001, 2024-03-20T09:31:58.803 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.056839437, 2024-03-20T09:31:58.803 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:58.803 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:58.803 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:58.803 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.803 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:58.803 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.310743+0000", 2024-03-20T09:31:58.803 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.803 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.803 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.803 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:58.803 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.310743+0000", 2024-03-20T09:31:58.803 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.803 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.803 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.804 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:58.804 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.310744+0000", 2024-03-20T09:31:58.804 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0640000000000001e-06 2024-03-20T09:31:58.804 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.804 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.804 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:58.804 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.310752+0000", 2024-03-20T09:31:58.804 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.2179999999999999e-06 2024-03-20T09:31:58.804 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.804 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.804 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:58.804 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.310766+0000", 2024-03-20T09:31:58.804 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3492999999999999e-05 2024-03-20T09:31:58.805 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.805 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.805 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:58.805 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.310774+0000", 2024-03-20T09:31:58.805 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.1049999999999995e-06 2024-03-20T09:31:58.805 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.805 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.805 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:58.805 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.310845+0000", 2024-03-20T09:31:58.805 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.1038999999999995e-05 2024-03-20T09:31:58.805 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.805 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.805 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:58.805 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.310865+0000", 2024-03-20T09:31:58.805 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9740000000000001e-05 2024-03-20T09:31:58.806 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.806 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.806 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:58.806 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.367534+0000", 2024-03-20T09:31:58.806 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.056668942 2024-03-20T09:31:58.806 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.806 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.806 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:58.806 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.367583+0000", 2024-03-20T09:31:58.806 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8835999999999998e-05 2024-03-20T09:31:58.806 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.806 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:58.806 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.806 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.807 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.807 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:229930 5.1 e688/637 5:85858a80:::benchmark_data_smithi122_157894_object99943:head v 688'14235, mlcod=628'6357)", 2024-03-20T09:31:58.807 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:57.318675+0000", 2024-03-20T09:31:58.807 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 541.48907792800003, 2024-03-20T09:31:58.807 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.048971187999999999, 2024-03-20T09:31:58.807 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:58.807 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:58.807 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:58.807 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.807 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:58.807 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.318675+0000", 2024-03-20T09:31:58.807 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.807 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.808 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.808 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:58.808 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.318675+0000", 2024-03-20T09:31:58.808 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.808 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.808 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.808 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:58.808 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.318675+0000", 2024-03-20T09:31:58.808 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.0999999999999999e-07 2024-03-20T09:31:58.808 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.808 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.808 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:58.808 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.318682+0000", 2024-03-20T09:31:58.809 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.7270000000000001e-06 2024-03-20T09:31:58.809 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.809 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.809 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:58.809 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.318689+0000", 2024-03-20T09:31:58.809 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4910000000000002e-06 2024-03-20T09:31:58.809 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.809 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.809 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:58.809 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.318693+0000", 2024-03-20T09:31:58.809 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.882e-06 2024-03-20T09:31:58.809 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.809 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.809 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:58.809 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.318753+0000", 2024-03-20T09:31:58.810 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9120000000000003e-05 2024-03-20T09:31:58.810 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.810 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.810 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:58.810 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.318780+0000", 2024-03-20T09:31:58.810 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.7080000000000002e-05 2024-03-20T09:31:58.810 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.810 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.810 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:58.810 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.367587+0000", 2024-03-20T09:31:58.810 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.048807017000000001 2024-03-20T09:31:58.810 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.810 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.810 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:58.811 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.367646+0000", 2024-03-20T09:31:58.811 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9460999999999999e-05 2024-03-20T09:31:58.811 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.811 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:58.811 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.811 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.811 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.811 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:229939 5.1 e688/637 5:8f3e9e8a:::benchmark_data_smithi122_157894_object99952:head v 688'14236, mlcod=628'6357)", 2024-03-20T09:31:58.811 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:57.318710+0000", 2024-03-20T09:31:58.811 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 541.48904304999996, 2024-03-20T09:31:58.811 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.048965195000000003, 2024-03-20T09:31:58.811 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:58.811 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:58.811 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:58.812 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.812 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:58.812 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.318710+0000", 2024-03-20T09:31:58.812 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.812 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.812 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.812 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:58.812 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.318710+0000", 2024-03-20T09:31:58.812 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.812 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.812 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.812 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:58.812 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.318710+0000", 2024-03-20T09:31:58.812 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.48e-07 2024-03-20T09:31:58.812 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.813 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.813 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:58.813 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.318714+0000", 2024-03-20T09:31:58.813 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.4400000000000001e-06 2024-03-20T09:31:58.813 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.813 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.813 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:58.813 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.318719+0000", 2024-03-20T09:31:58.813 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1549999999999998e-06 2024-03-20T09:31:58.813 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.813 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.813 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:58.813 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.318724+0000", 2024-03-20T09:31:58.813 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3210000000000004e-06 2024-03-20T09:31:58.813 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.814 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.814 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:58.814 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.323677+0000", 2024-03-20T09:31:58.814 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0049531159999999996 2024-03-20T09:31:58.814 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.814 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.814 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:58.814 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.323696+0000", 2024-03-20T09:31:58.814 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9012e-05 2024-03-20T09:31:58.814 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.814 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.814 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:58.815 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.367648+0000", 2024-03-20T09:31:58.815 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.043952264999999997 2024-03-20T09:31:58.815 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.815 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.815 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:58.815 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:57.367675+0000", 2024-03-20T09:31:58.815 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6638000000000001e-05 2024-03-20T09:31:58.815 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.815 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:58.815 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.815 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.815 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.815 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:231357 5.1 e690/637 5:862a1852:::benchmark_data_smithi122_157894_object101370:head v 690'14319, mlcod=628'6446)", 2024-03-20T09:31:58.815 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:59.484435+0000", 2024-03-20T09:31:58.816 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 539.32331765699996, 2024-03-20T09:31:58.816 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.046223444000000002, 2024-03-20T09:31:58.816 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:58.816 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:58.816 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:58.816 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.816 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:58.816 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.484435+0000", 2024-03-20T09:31:58.816 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.816 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.816 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.816 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:58.816 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.484435+0000", 2024-03-20T09:31:58.816 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.817 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.817 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.817 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:58.817 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.484436+0000", 2024-03-20T09:31:58.817 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.7999999999999993e-07 2024-03-20T09:31:58.817 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.817 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.817 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:58.817 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.484446+0000", 2024-03-20T09:31:58.817 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.8269999999999993e-06 2024-03-20T09:31:58.817 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.817 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.817 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:58.817 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.484459+0000", 2024-03-20T09:31:58.817 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2744e-05 2024-03-20T09:31:58.818 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.818 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.818 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:58.818 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.484468+0000", 2024-03-20T09:31:58.818 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.8310000000000007e-06 2024-03-20T09:31:58.818 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.818 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.818 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:58.818 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.484615+0000", 2024-03-20T09:31:58.818 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00014736600000000001 2024-03-20T09:31:58.818 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.818 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.818 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:58.818 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.484642+0000", 2024-03-20T09:31:58.819 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6698000000000001e-05 2024-03-20T09:31:58.819 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.819 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.819 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:58.819 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.530360+0000", 2024-03-20T09:31:58.819 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.045717854000000002 2024-03-20T09:31:58.819 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.819 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.819 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:58.819 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.530659+0000", 2024-03-20T09:31:58.819 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00029914399999999999 2024-03-20T09:31:58.819 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.819 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:58.819 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.819 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.820 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.820 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:231351 5.9 e690/605 5:9562aab7:::benchmark_data_smithi122_157894_object101364:head v 690'14515, mlcod=690'14513)", 2024-03-20T09:31:58.820 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:22:59.485231+0000", 2024-03-20T09:31:58.820 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 539.32252167199999, 2024-03-20T09:31:58.820 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.049753224999999998, 2024-03-20T09:31:58.820 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:58.820 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:58.820 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:58.820 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.820 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:58.820 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.485231+0000", 2024-03-20T09:31:58.820 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.820 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.820 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.820 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:58.820 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.485231+0000", 2024-03-20T09:31:58.821 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.821 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.821 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.821 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:58.821 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.485232+0000", 2024-03-20T09:31:58.821 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7100000000000001e-07 2024-03-20T09:31:58.821 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.821 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.821 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:58.821 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.485240+0000", 2024-03-20T09:31:58.821 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7700000000000001e-06 2024-03-20T09:31:58.821 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.821 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.821 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:58.821 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.485251+0000", 2024-03-20T09:31:58.822 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1608e-05 2024-03-20T09:31:58.822 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.822 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.822 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:58.822 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.485259+0000", 2024-03-20T09:31:58.822 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.289e-06 2024-03-20T09:31:58.822 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.822 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.822 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:58.822 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.489654+0000", 2024-03-20T09:31:58.822 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0043954060000000001 2024-03-20T09:31:58.822 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.822 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.822 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:58.823 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.489678+0000", 2024-03-20T09:31:58.823 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3524999999999999e-05 2024-03-20T09:31:58.823 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.823 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.823 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:58.823 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.534951+0000", 2024-03-20T09:31:58.823 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.045273571999999998 2024-03-20T09:31:58.823 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.823 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.823 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:58.823 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:22:59.534984+0000", 2024-03-20T09:31:58.823 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.3284000000000002e-05 2024-03-20T09:31:58.823 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.823 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:58.824 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.824 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.824 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.824 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:232222 5.d e690/605 5:b74bcbe9:::benchmark_data_smithi122_157894_object102235:head v 690'14600, mlcod=690'14597)", 2024-03-20T09:31:58.824 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:23:00.802652+0000", 2024-03-20T09:31:58.824 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 538.00510108399999, 2024-03-20T09:31:58.824 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.049633355999999997, 2024-03-20T09:31:58.824 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:58.824 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:58.824 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:58.824 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.824 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:58.824 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:00.802652+0000", 2024-03-20T09:31:58.824 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.824 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.825 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.825 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:58.825 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:00.802652+0000", 2024-03-20T09:31:58.825 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.825 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.825 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.825 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:58.825 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:00.802653+0000", 2024-03-20T09:31:58.825 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.22e-06 2024-03-20T09:31:58.825 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.825 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.825 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:58.825 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:00.802661+0000", 2024-03-20T09:31:58.826 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7090000000000002e-06 2024-03-20T09:31:58.826 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.826 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.826 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:58.826 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:00.802674+0000", 2024-03-20T09:31:58.826 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2901999999999999e-05 2024-03-20T09:31:58.826 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.826 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.826 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:58.826 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:00.802680+0000", 2024-03-20T09:31:58.826 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.5200000000000003e-06 2024-03-20T09:31:58.826 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.826 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.826 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:58.826 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:00.804961+0000", 2024-03-20T09:31:58.827 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0022811049999999999 2024-03-20T09:31:58.827 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.827 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.827 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:58.827 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:00.804982+0000", 2024-03-20T09:31:58.827 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0432999999999999e-05 2024-03-20T09:31:58.827 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.827 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.827 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:58.827 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:00.852239+0000", 2024-03-20T09:31:58.827 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.047257413999999998 2024-03-20T09:31:58.827 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.827 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.827 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:58.828 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:00.852285+0000", 2024-03-20T09:31:58.828 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6053000000000001e-05 2024-03-20T09:31:58.828 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.828 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:58.828 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.828 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.828 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.828 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:250737 5.9 e697/605 5:912b5ada:::benchmark_data_smithi122_157894_object120750:head v 697'15732, mlcod=697'15729)", 2024-03-20T09:31:58.828 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:23:28.731645+0000", 2024-03-20T09:31:58.828 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 510.07610791899998, 2024-03-20T09:31:58.828 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.046361630000000001, 2024-03-20T09:31:58.828 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:58.828 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:58.828 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:58.828 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.829 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:58.829 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.731645+0000", 2024-03-20T09:31:58.829 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.829 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.829 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.829 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:58.829 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.731645+0000", 2024-03-20T09:31:58.829 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.829 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.829 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.829 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:58.829 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.731646+0000", 2024-03-20T09:31:58.829 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.7000000000000003e-07 2024-03-20T09:31:58.829 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.829 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.829 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:58.830 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.731652+0000", 2024-03-20T09:31:58.830 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0379999999999996e-06 2024-03-20T09:31:58.830 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.830 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.830 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:58.830 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.731662+0000", 2024-03-20T09:31:58.830 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.995e-06 2024-03-20T09:31:58.830 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.830 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.830 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:58.830 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.731669+0000", 2024-03-20T09:31:58.830 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.5009999999999998e-06 2024-03-20T09:31:58.830 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.830 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.831 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:58.831 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.752990+0000", 2024-03-20T09:31:58.831 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.021320260000000001 2024-03-20T09:31:58.831 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.831 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.831 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:58.831 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.753014+0000", 2024-03-20T09:31:58.831 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.4199999999999999e-05 2024-03-20T09:31:58.831 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.831 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.831 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:58.831 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.777963+0000", 2024-03-20T09:31:58.831 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.024948682 2024-03-20T09:31:58.831 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.831 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.831 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:58.832 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.778007+0000", 2024-03-20T09:31:58.832 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4084e-05 2024-03-20T09:31:58.832 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.832 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:58.832 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.832 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.832 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.832 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.25966.0:250756 5.1 e697/637 5:8e3474d2:::benchmark_data_smithi122_157894_object120769:head v 697'15500, mlcod=635'7965)", 2024-03-20T09:31:58.832 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:23:28.752816+0000", 2024-03-20T09:31:58.832 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 510.05493710899998, 2024-03-20T09:31:58.832 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.046701550000000001, 2024-03-20T09:31:58.832 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:58.832 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:58.832 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:58.832 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.833 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:58.833 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.752816+0000", 2024-03-20T09:31:58.833 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.833 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.833 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.833 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:58.833 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.752816+0000", 2024-03-20T09:31:58.833 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.833 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.833 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.833 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:58.833 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.752817+0000", 2024-03-20T09:31:58.833 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.4e-07 2024-03-20T09:31:58.833 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.833 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.833 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:58.834 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.752825+0000", 2024-03-20T09:31:58.834 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.8730000000000007e-06 2024-03-20T09:31:58.834 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.834 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.834 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:58.834 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.752835+0000", 2024-03-20T09:31:58.834 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0565e-05 2024-03-20T09:31:58.834 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.834 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.834 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:58.834 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.752843+0000", 2024-03-20T09:31:58.834 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.5410000000000003e-06 2024-03-20T09:31:58.834 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.834 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.834 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:58.835 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.774230+0000", 2024-03-20T09:31:58.835 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.021387782000000001 2024-03-20T09:31:58.835 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.835 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.835 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:58.835 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.774249+0000", 2024-03-20T09:31:58.835 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8221000000000001e-05 2024-03-20T09:31:58.835 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.835 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.835 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:58.835 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.799488+0000", 2024-03-20T09:31:58.835 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.025239444999999999 2024-03-20T09:31:58.835 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.835 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.835 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:58.836 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:23:28.799517+0000", 2024-03-20T09:31:58.836 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9283000000000001e-05 2024-03-20T09:31:58.836 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.836 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:58.836 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.836 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.836 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.836 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "rep_scrubmap(1.15 e491 from shard 5)", 2024-03-20T09:31:58.836 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:14.417982+0000", 2024-03-20T09:31:58.836 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 404.38977071099998, 2024-03-20T09:31:58.836 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.059782971999999997, 2024-03-20T09:31:58.836 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:58.836 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:31:58.836 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:58.836 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.837 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:58.837 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.417982+0000", 2024-03-20T09:31:58.837 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.837 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.837 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.837 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:58.837 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.417982+0000", 2024-03-20T09:31:58.837 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.837 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.837 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.837 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:58.837 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.417983+0000", 2024-03-20T09:31:58.837 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3099999999999997e-07 2024-03-20T09:31:58.837 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.837 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.838 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:58.838 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.417988+0000", 2024-03-20T09:31:58.838 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4790000000000004e-06 2024-03-20T09:31:58.838 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.838 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.838 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:58.838 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.417998+0000", 2024-03-20T09:31:58.838 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.2399999999999996e-06 2024-03-20T09:31:58.838 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.838 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.838 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:58.838 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.418003+0000", 2024-03-20T09:31:58.838 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5160000000000002e-06 2024-03-20T09:31:58.838 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.839 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.839 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:58.839 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.477737+0000", 2024-03-20T09:31:58.839 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.059734260999999997 2024-03-20T09:31:58.839 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.839 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.839 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:58.839 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.477749+0000", 2024-03-20T09:31:58.839 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.149e-05 2024-03-20T09:31:58.839 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.839 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.839 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:58.839 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.477765+0000", 2024-03-20T09:31:58.839 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6254999999999999e-05 2024-03-20T09:31:58.839 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.839 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:58.840 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.840 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.840 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.840 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "rep_scrubmap(1.15 e491 from shard 8)", 2024-03-20T09:31:58.840 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:14.425166+0000", 2024-03-20T09:31:58.840 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 404.38258660499997, 2024-03-20T09:31:58.840 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.052696894000000001, 2024-03-20T09:31:58.840 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:58.840 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:31:58.840 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:58.840 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.840 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:58.840 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.425166+0000", 2024-03-20T09:31:58.840 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.840 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.841 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.841 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:58.841 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.425166+0000", 2024-03-20T09:31:58.841 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.841 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.841 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.841 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:58.841 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.425167+0000", 2024-03-20T09:31:58.841 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.9599999999999998e-07 2024-03-20T09:31:58.841 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.841 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.841 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:58.841 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.425173+0000", 2024-03-20T09:31:58.841 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9529999999999996e-06 2024-03-20T09:31:58.841 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.841 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.842 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:58.842 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.425194+0000", 2024-03-20T09:31:58.842 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1285999999999999e-05 2024-03-20T09:31:58.842 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.842 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.842 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:58.842 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.425202+0000", 2024-03-20T09:31:58.842 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7039999999999994e-06 2024-03-20T09:31:58.842 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.842 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.842 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:58.842 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.477824+0000", 2024-03-20T09:31:58.842 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.052622206999999997 2024-03-20T09:31:58.843 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.843 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.843 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:58.843 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.477835+0000", 2024-03-20T09:31:58.843 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0788000000000001e-05 2024-03-20T09:31:58.843 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.843 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.843 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:58.843 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.477863+0000", 2024-03-20T09:31:58.843 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8160000000000001e-05 2024-03-20T09:31:58.843 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.843 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:58.843 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.843 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.843 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.843 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26216.0:86454 6.5 e733/709 6:a2958d06:::benchmark_data_smithi122_168655_object86453:head v 733'5375, mlcod=732'5373)", 2024-03-20T09:31:58.844 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:14.436942+0000", 2024-03-20T09:31:58.844 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 404.370811366, 2024-03-20T09:31:58.844 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.051506415, 2024-03-20T09:31:58.844 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:58.844 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:58.844 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:58.844 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.844 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:58.844 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.436942+0000", 2024-03-20T09:31:58.844 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.844 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.844 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.844 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:58.844 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.436942+0000", 2024-03-20T09:31:58.844 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.845 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.845 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.845 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:58.845 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.436943+0000", 2024-03-20T09:31:58.845 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.403e-06 2024-03-20T09:31:58.845 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.845 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.845 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:58.845 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.436964+0000", 2024-03-20T09:31:58.845 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0661000000000001e-05 2024-03-20T09:31:58.845 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.845 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.845 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:58.845 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.436980+0000", 2024-03-20T09:31:58.845 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6328e-05 2024-03-20T09:31:58.846 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.846 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.846 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:58.846 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.436988+0000", 2024-03-20T09:31:58.846 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.0169999999999996e-06 2024-03-20T09:31:58.846 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.846 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.846 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:58.846 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.477790+0000", 2024-03-20T09:31:58.846 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.040802120999999997 2024-03-20T09:31:58.846 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.846 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.846 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:58.846 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.477807+0000", 2024-03-20T09:31:58.847 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7297000000000001e-05 2024-03-20T09:31:58.847 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.847 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.847 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:58.847 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.488398+0000", 2024-03-20T09:31:58.847 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.010590373 2024-03-20T09:31:58.847 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.847 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.847 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:58.847 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:14.488448+0000", 2024-03-20T09:31:58.847 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0214999999999999e-05 2024-03-20T09:31:58.847 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.847 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:58.848 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.848 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.848 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.848 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26216.0:129134 6.b e748/709 6:dfb654b3:::benchmark_data_smithi122_168655_object9025:head v 748'8030, mlcod=748'8027)", 2024-03-20T09:31:58.848 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:56.321820+0000", 2024-03-20T09:31:58.848 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 362.48593324000001, 2024-03-20T09:31:58.848 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.26522621299999999, 2024-03-20T09:31:58.848 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:58.848 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:58.848 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:58.848 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.848 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:58.848 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.321820+0000", 2024-03-20T09:31:58.848 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.848 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.849 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.849 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:58.849 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.321820+0000", 2024-03-20T09:31:58.849 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.849 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.849 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.849 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:58.849 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.321821+0000", 2024-03-20T09:31:58.849 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.437e-06 2024-03-20T09:31:58.849 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.849 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.849 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:58.849 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.321829+0000", 2024-03-20T09:31:58.849 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7009999999999996e-06 2024-03-20T09:31:58.849 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.850 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.850 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:58.850 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.321845+0000", 2024-03-20T09:31:58.850 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5852999999999999e-05 2024-03-20T09:31:58.850 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.850 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.850 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:58.850 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.321853+0000", 2024-03-20T09:31:58.850 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.2379999999999997e-06 2024-03-20T09:31:58.850 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.850 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.850 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:58.850 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.321913+0000", 2024-03-20T09:31:58.850 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0377999999999997e-05 2024-03-20T09:31:58.851 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.851 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.851 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:58.851 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.321938+0000", 2024-03-20T09:31:58.851 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.4329e-05 2024-03-20T09:31:58.851 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.851 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.851 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:58.851 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.586991+0000", 2024-03-20T09:31:58.851 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.26505305800000001 2024-03-20T09:31:58.851 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.851 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.851 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:58.851 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.587046+0000", 2024-03-20T09:31:58.851 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5219000000000001e-05 2024-03-20T09:31:58.852 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.852 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:58.852 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.852 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.852 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.852 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26216.0:129139 6.8 e748/709 6:1a6b87cd:::benchmark_data_smithi122_168655_object9030:head v 748'7939, mlcod=748'7937)", 2024-03-20T09:31:58.852 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:56.323431+0000", 2024-03-20T09:31:58.852 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 362.48432158899999, 2024-03-20T09:31:58.852 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.234771442, 2024-03-20T09:31:58.852 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:58.852 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:31:58.852 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:58.852 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.852 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:58.853 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.323431+0000", 2024-03-20T09:31:58.853 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.853 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.853 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.853 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:58.853 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.323431+0000", 2024-03-20T09:31:58.853 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.853 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.853 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.853 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:58.853 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.323432+0000", 2024-03-20T09:31:58.853 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.4e-07 2024-03-20T09:31:58.853 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.853 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.854 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:58.854 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.323439+0000", 2024-03-20T09:31:58.854 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4559999999999997e-06 2024-03-20T09:31:58.854 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.854 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.854 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:58.854 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.323452+0000", 2024-03-20T09:31:58.854 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3574000000000001e-05 2024-03-20T09:31:58.854 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.854 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.854 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:58.854 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.323460+0000", 2024-03-20T09:31:58.854 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.5669999999999997e-06 2024-03-20T09:31:58.854 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.855 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.855 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:58.855 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.323522+0000", 2024-03-20T09:31:58.855 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2389000000000002e-05 2024-03-20T09:31:58.855 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.855 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.855 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:58.855 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.323544+0000", 2024-03-20T09:31:58.855 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1702e-05 2024-03-20T09:31:58.855 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.855 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.855 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:58.855 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.558203+0000", 2024-03-20T09:31:58.855 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.23465881399999999 2024-03-20T09:31:58.855 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.856 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:58.856 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.856 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.856 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.856 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26216.0:129140 6.3 e748/709)", 2024-03-20T09:31:58.856 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:56.327278+0000", 2024-03-20T09:31:58.856 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 362.48047475599998, 2024-03-20T09:31:58.856 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.26500126899999998, 2024-03-20T09:31:58.856 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:58.856 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:31:58.856 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:58.856 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.856 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:58.856 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.327278+0000", 2024-03-20T09:31:58.856 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.857 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.857 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.857 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:58.857 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.327278+0000", 2024-03-20T09:31:58.857 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.857 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.857 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.857 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:58.857 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.327279+0000", 2024-03-20T09:31:58.857 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0669999999999999e-06 2024-03-20T09:31:58.857 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.857 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.857 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:58.857 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.327288+0000", 2024-03-20T09:31:58.857 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5920000000000001e-06 2024-03-20T09:31:58.858 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.858 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.858 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:58.858 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.327301+0000", 2024-03-20T09:31:58.858 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2768e-05 2024-03-20T09:31:58.858 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.858 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.858 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:58.858 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.327308+0000", 2024-03-20T09:31:58.858 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.2579999999999998e-06 2024-03-20T09:31:58.858 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.858 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.858 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:58.858 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.592260+0000", 2024-03-20T09:31:58.859 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.26495219599999997 2024-03-20T09:31:58.859 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.859 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.859 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:58.859 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.592279+0000", 2024-03-20T09:31:58.859 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9388000000000001e-05 2024-03-20T09:31:58.859 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.859 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:58.859 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.859 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.859 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.859 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26216.0:129141 6.b e748/709 6:ddae97ab:::benchmark_data_smithi122_168655_object9032:head v 748'8031, mlcod=748'8028)", 2024-03-20T09:31:58.859 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:56.329794+0000", 2024-03-20T09:31:58.859 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 362.47795933399999, 2024-03-20T09:31:58.859 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.29588104700000001, 2024-03-20T09:31:58.859 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:58.860 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:58.860 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:58.860 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.860 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:58.860 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.329794+0000", 2024-03-20T09:31:58.860 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.860 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.860 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.860 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:58.860 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.329794+0000", 2024-03-20T09:31:58.860 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.860 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.860 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.860 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:58.860 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.329795+0000", 2024-03-20T09:31:58.860 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.083e-06 2024-03-20T09:31:58.861 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.861 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.861 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:58.861 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.329803+0000", 2024-03-20T09:31:58.861 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.7450000000000001e-06 2024-03-20T09:31:58.861 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.861 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.861 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:58.861 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.329815+0000", 2024-03-20T09:31:58.861 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1790000000000001e-05 2024-03-20T09:31:58.861 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.861 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.861 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:58.861 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.329822+0000", 2024-03-20T09:31:58.861 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.7440000000000001e-06 2024-03-20T09:31:58.862 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.862 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.862 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:58.862 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.582786+0000", 2024-03-20T09:31:58.862 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.25296439500000001 2024-03-20T09:31:58.862 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.862 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.862 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:58.862 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.582813+0000", 2024-03-20T09:31:58.862 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6806000000000001e-05 2024-03-20T09:31:58.862 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.862 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.862 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:58.862 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.625623+0000", 2024-03-20T09:31:58.863 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.042809502999999999 2024-03-20T09:31:58.863 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.863 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.863 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:58.863 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.625675+0000", 2024-03-20T09:31:58.863 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1981000000000003e-05 2024-03-20T09:31:58.863 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.863 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:58.863 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.863 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.863 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.863 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26216.0:129146 6.b e749/709 6:d9973fcf:::benchmark_data_smithi122_168655_object9037:head v 749'8032, mlcod=748'8028)", 2024-03-20T09:31:58.863 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:56.362248+0000", 2024-03-20T09:31:58.863 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 362.44550482199998, 2024-03-20T09:31:58.864 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.26344784799999998, 2024-03-20T09:31:58.864 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:58.864 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:58.864 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:58.864 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.864 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:58.864 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.362248+0000", 2024-03-20T09:31:58.864 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.864 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.864 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.864 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:58.864 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.362248+0000", 2024-03-20T09:31:58.864 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.864 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.864 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.864 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:58.865 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.362248+0000", 2024-03-20T09:31:58.865 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6600000000000003e-07 2024-03-20T09:31:58.865 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.865 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.865 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:58.865 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.362255+0000", 2024-03-20T09:31:58.865 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.917e-06 2024-03-20T09:31:58.865 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.865 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.865 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:58.865 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.362282+0000", 2024-03-20T09:31:58.865 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6344999999999999e-05 2024-03-20T09:31:58.865 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.865 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.865 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:58.866 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.362298+0000", 2024-03-20T09:31:58.866 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6087999999999999e-05 2024-03-20T09:31:58.866 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.866 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.866 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:58.866 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.587169+0000", 2024-03-20T09:31:58.866 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.224871712 2024-03-20T09:31:58.866 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.866 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.866 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:58.866 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.587197+0000", 2024-03-20T09:31:58.866 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.7977e-05 2024-03-20T09:31:58.866 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.866 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.867 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:58.867 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.625680+0000", 2024-03-20T09:31:58.867 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.038482502000000002 2024-03-20T09:31:58.867 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.867 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.867 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:58.867 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.625696+0000", 2024-03-20T09:31:58.867 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6041000000000001e-05 2024-03-20T09:31:58.867 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.867 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:58.867 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.867 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.867 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.867 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26216.0:231442 6.1 e858/823 6:86646aa3:::benchmark_data_smithi122_168655_object111333:head v 858'11315, mlcod=858'11313)", 2024-03-20T09:31:58.868 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:30:36.418993+0000", 2024-03-20T09:31:58.868 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 82.388759592, 2024-03-20T09:31:58.868 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.042099804999999997, 2024-03-20T09:31:58.868 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:58.868 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:58.868 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:58.868 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.868 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:58.868 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:36.418993+0000", 2024-03-20T09:31:58.868 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.868 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.868 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.868 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:58.868 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:36.418993+0000", 2024-03-20T09:31:58.869 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.869 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.869 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.869 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:58.869 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:36.418995+0000", 2024-03-20T09:31:58.869 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.249e-06 2024-03-20T09:31:58.869 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.869 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.869 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:58.869 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:36.419005+0000", 2024-03-20T09:31:58.869 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0699e-05 2024-03-20T09:31:58.869 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.869 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.869 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:58.869 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:36.419022+0000", 2024-03-20T09:31:58.870 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6252e-05 2024-03-20T09:31:58.870 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.870 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.870 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:58.870 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:36.419031+0000", 2024-03-20T09:31:58.870 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.7909999999999999e-06 2024-03-20T09:31:58.870 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.870 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.870 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:58.870 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:36.419103+0000", 2024-03-20T09:31:58.870 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.1476000000000005e-05 2024-03-20T09:31:58.870 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.870 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.870 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:58.871 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:36.419128+0000", 2024-03-20T09:31:58.871 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5377000000000001e-05 2024-03-20T09:31:58.871 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.871 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.871 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:58.871 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:36.461052+0000", 2024-03-20T09:31:58.871 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.041924318000000002 2024-03-20T09:31:58.871 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.871 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.871 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:58.871 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:36.461093+0000", 2024-03-20T09:31:58.871 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.0642999999999997e-05 2024-03-20T09:31:58.871 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.871 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:58.871 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.871 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.872 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.872 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26216.0:231454 6.1 e858/823 6:8031778a:::benchmark_data_smithi122_168655_object111345:head v 858'11316, mlcod=858'11313)", 2024-03-20T09:31:58.872 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:30:36.426994+0000", 2024-03-20T09:31:58.872 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 82.380759040000001, 2024-03-20T09:31:58.872 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.034126780000000002, 2024-03-20T09:31:58.872 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:58.872 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:31:58.872 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:58.872 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.872 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:58.872 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:36.426994+0000", 2024-03-20T09:31:58.872 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.872 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.872 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.873 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:58.873 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:36.426994+0000", 2024-03-20T09:31:58.873 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.873 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.873 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.873 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:58.873 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:36.426995+0000", 2024-03-20T09:31:58.873 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.091e-06 2024-03-20T09:31:58.873 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.873 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.873 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:58.873 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:36.427004+0000", 2024-03-20T09:31:58.873 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.8589999999999994e-06 2024-03-20T09:31:58.873 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.873 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.874 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:58.874 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:36.427017+0000", 2024-03-20T09:31:58.874 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3454e-05 2024-03-20T09:31:58.874 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.874 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.874 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:58.874 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:36.427023+0000", 2024-03-20T09:31:58.874 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1580000000000002e-06 2024-03-20T09:31:58.874 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.874 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.874 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:58.874 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:36.427078+0000", 2024-03-20T09:31:58.874 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4338000000000001e-05 2024-03-20T09:31:58.874 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.875 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.875 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:58.875 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:36.427103+0000", 2024-03-20T09:31:58.875 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5199999999999999e-05 2024-03-20T09:31:58.875 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.875 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.875 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:31:58.875 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:36.461096+0000", 2024-03-20T09:31:58.875 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.033993203 2024-03-20T09:31:58.875 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.875 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.875 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:58.875 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:36.461121+0000", 2024-03-20T09:31:58.875 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.4477e-05 2024-03-20T09:31:58.875 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.876 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:58.876 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.876 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.876 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.876 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "rep_scrubmap(1.15 e491 from shard 8)", 2024-03-20T09:31:58.876 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:31:32.342334+0000", 2024-03-20T09:31:58.876 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 26.465418449000001, 2024-03-20T09:31:58.876 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.068097784999999994, 2024-03-20T09:31:58.876 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:58.876 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:31:58.876 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:58.876 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.876 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:58.876 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.342334+0000", 2024-03-20T09:31:58.876 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.877 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.877 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.877 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:58.877 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.342334+0000", 2024-03-20T09:31:58.877 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.877 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.877 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.877 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:58.877 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.342335+0000", 2024-03-20T09:31:58.877 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.9699999999999995e-07 2024-03-20T09:31:58.877 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.877 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.877 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:58.877 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.342341+0000", 2024-03-20T09:31:58.878 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5620000000000003e-06 2024-03-20T09:31:58.878 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.878 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.878 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:58.878 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.342351+0000", 2024-03-20T09:31:58.878 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0621999999999999e-05 2024-03-20T09:31:58.878 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.878 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.878 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:58.878 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.342357+0000", 2024-03-20T09:31:58.878 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7440000000000004e-06 2024-03-20T09:31:58.878 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.878 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.878 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:58.878 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.410402+0000", 2024-03-20T09:31:58.879 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.068045091000000002 2024-03-20T09:31:58.879 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.879 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.879 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:58.879 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.410414+0000", 2024-03-20T09:31:58.879 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1867e-05 2024-03-20T09:31:58.879 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.879 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.879 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:58.879 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.410432+0000", 2024-03-20T09:31:58.879 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8102e-05 2024-03-20T09:31:58.879 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.879 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:58.879 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.879 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.880 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.880 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "rep_scrubmap(1.15 e491 from shard 5)", 2024-03-20T09:31:58.880 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:31:32.346280+0000", 2024-03-20T09:31:58.880 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 26.461472580999999, 2024-03-20T09:31:58.880 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.064250347999999999, 2024-03-20T09:31:58.880 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:31:58.880 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:31:58.880 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:31:58.880 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.880 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:31:58.880 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.346280+0000", 2024-03-20T09:31:58.880 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.880 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.880 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.881 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:31:58.881 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.346280+0000", 2024-03-20T09:31:58.881 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:31:58.881 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.881 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.881 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:31:58.881 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.346281+0000", 2024-03-20T09:31:58.881 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.5999999999999991e-07 2024-03-20T09:31:58.881 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.881 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.881 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:31:58.881 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.346287+0000", 2024-03-20T09:31:58.881 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4299999999999997e-06 2024-03-20T09:31:58.881 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.882 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.882 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:31:58.882 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.346301+0000", 2024-03-20T09:31:58.882 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4088e-05 2024-03-20T09:31:58.882 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.882 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.882 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:31:58.882 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.346308+0000", 2024-03-20T09:31:58.882 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.1319999999999996e-06 2024-03-20T09:31:58.882 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.882 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.882 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:31:58.882 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.410482+0000", 2024-03-20T09:31:58.883 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.064174155999999996 2024-03-20T09:31:58.883 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.883 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.883 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:31:58.883 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.410494+0000", 2024-03-20T09:31:58.883 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1439e-05 2024-03-20T09:31:58.883 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:31:58.883 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:31:58.883 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:31:58.883 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.410531+0000", 2024-03-20T09:31:58.883 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7143e-05 2024-03-20T09:31:58.883 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.883 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:58.883 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.884 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:31:58.884 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:31:58.884 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:31:58.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:31:58 smithi115 ceph-mon[72476]: pgmap v2151: 47 pgs: 47 active+clean; 4.4 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-03-20T09:31:58.911 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:31:58 smithi026 ceph-mon[68309]: pgmap v2151: 47 pgs: 47 active+clean; 4.4 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-03-20T09:31:58.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:31:58 smithi122 ceph-mon[73495]: pgmap v2151: 47 pgs: 47 active+clean; 4.4 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-03-20T09:31:59.184 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:31:59.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:31:59.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:31:59.291 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:31:59.343 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:31:59.521 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_ops_in_flight 2024-03-20T09:31:59.644 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 41 16 77126 77110 117.526 88.75 0.00855779 0.00850482 2024-03-20T09:31:59.673 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:31:59.860 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:32:00.426 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:32:00.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:00.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:32:00.533 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:32:00.573 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:00.598 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:32:00.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:00.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:32:00.644 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 42 16 78235 78219 116.378 69.3125 0.00762691 0.00858822 2024-03-20T09:32:00.703 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:32:00.737 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:00.762 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:32:00.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:00.766 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:32:00.867 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:32:00.939 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:00.962 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:32:01.017 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:32:01.056 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:32:01.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:01.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:32:01.068 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:32:01.130 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:32:01.166 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:32:01.184 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:32:01.197 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:01.220 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:32:01.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:01.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:32:01.234 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:32:01.292 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:32:01.325 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:32:01.348 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:32:01.397 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:01.431 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:32:01.446 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:32:01.491 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:32:01.517 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:32:01.525 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:32:01.525 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:32:01.525 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:32:01.525 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:32:01.525 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:32:01.525 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:32:01.526 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:32:01.526 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:32:01.526 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:32:01.526 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:32:01.526 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:32:01.526 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:32:01.645 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 43 16 79386 79370 115.345 71.9375 0.015366 0.00866573 2024-03-20T09:32:01.647 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:01 smithi115 ceph-mon[72476]: pgmap v2152: 47 pgs: 47 active+clean; 4.5 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-03-20T09:32:01.648 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:01 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2237937887' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:32:01.688 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:01 smithi122 ceph-mon[73495]: pgmap v2152: 47 pgs: 47 active+clean; 4.5 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-03-20T09:32:01.688 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:01 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2237937887' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:32:01.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:01 smithi026 ceph-mon[68309]: pgmap v2152: 47 pgs: 47 active+clean; 4.5 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-03-20T09:32:01.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:01 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2237937887' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:32:01.870 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:32:01.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:01.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:32:01.979 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:32:02.020 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:02.379 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:32:02.448 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.1 to 1 (10000) 2024-03-20T09:32:02.645 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 44 16 80703 80687 114.594 82.3125 0.00716968 0.00872206 2024-03-20T09:32:02.688 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:02 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4135250292' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-20T09:32:02.688 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:02 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-20T09:32:02.689 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:02 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4135250292' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-20T09:32:02.689 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:02 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-20T09:32:02.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:02 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4135250292' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-20T09:32:02.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:02 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-20T09:32:03.502 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 2 1 2024-03-20T09:32:03.645 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 45 16 82026 82010 113.885 82.6875 0.00674076 0.00877685 2024-03-20T09:32:03.688 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:03 smithi122 ceph-mon[73495]: pgmap v2153: 47 pgs: 47 active+clean; 4.7 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-03-20T09:32:03.688 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:03 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-20T09:32:03.688 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:03 smithi122 ceph-mon[73495]: osdmap e891: 12 total, 12 up, 11 in 2024-03-20T09:32:03.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:03 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:32:03.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:03 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:32:03.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:03 smithi122 ceph-mon[73495]: osdmap e892: 12 total, 12 up, 11 in 2024-03-20T09:32:03.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:03 smithi026 ceph-mon[68309]: pgmap v2153: 47 pgs: 47 active+clean; 4.7 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-03-20T09:32:03.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:03 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-20T09:32:03.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:03 smithi026 ceph-mon[68309]: osdmap e891: 12 total, 12 up, 11 in 2024-03-20T09:32:03.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:03 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:32:03.717 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:03 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:32:03.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:03 smithi026 ceph-mon[68309]: osdmap e892: 12 total, 12 up, 11 in 2024-03-20T09:32:03.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:03 smithi115 ceph-mon[72476]: pgmap v2153: 47 pgs: 47 active+clean; 4.7 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-03-20T09:32:03.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:03 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-20T09:32:03.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:03 smithi115 ceph-mon[72476]: osdmap e891: 12 total, 12 up, 11 in 2024-03-20T09:32:03.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:03 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:32:03.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:03 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:32:03.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:03 smithi115 ceph-mon[72476]: osdmap e892: 12 total, 12 up, 11 in 2024-03-20T09:32:04.348 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:32:04.367 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:32:04.399 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:32:04.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:04.404 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:32:04.505 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:32:04.538 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:04.574 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:32:04.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:04.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:32:04.645 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 46 16 83174 83158 112.969 71.75 0.0088169 0.00884804 2024-03-20T09:32:04.679 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:32:04.716 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:04.744 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:32:04.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:04.751 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:32:04.852 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:32:04.891 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:04.919 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:32:04.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:04.925 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:32:05.026 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:32:05.090 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:05.354 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:32:05.355 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:32:05.645 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 47 16 84382 84366 112.171 75.5 0.00303386 0.00889783 2024-03-20T09:32:05.688 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:05 smithi122 ceph-mon[73495]: pgmap v2156: 47 pgs: 47 active+clean; 4.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 60 MiB/s wr, 957 op/s 2024-03-20T09:32:05.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:05 smithi026 ceph-mon[68309]: pgmap v2156: 47 pgs: 47 active+clean; 4.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 60 MiB/s wr, 957 op/s 2024-03-20T09:32:05.867 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:32:05.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:05.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:32:05.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:05 smithi115 ceph-mon[72476]: pgmap v2156: 47 pgs: 47 active+clean; 4.8 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 60 MiB/s wr, 957 op/s 2024-03-20T09:32:05.974 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:32:06.005 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:06.108 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:32:06.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:06.113 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:32:06.132 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_blocked_ops 2024-03-20T09:32:06.214 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:32:06.267 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:06.475 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:32:06.519 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:32:06.645 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 48 16 85452 85436 111.228 66.875 0.0098672 0.00898645 2024-03-20T09:32:06.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:06 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3475366062' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:32:06.791 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:06 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3475366062' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:32:06.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:06 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3475366062' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:32:07.645 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 49 16 86789 86773 110.663 83.5625 0.0135149 0.0090324 2024-03-20T09:32:07.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:07 smithi026 ceph-mon[68309]: pgmap v2157: 47 pgs: 47 active+clean; 5.0 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-03-20T09:32:07.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:07 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:32:07.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:07 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3475366062' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:32:07.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:07 smithi026 ceph-mon[68309]: osdmap e893: 12 total, 12 up, 11 in 2024-03-20T09:32:07.829 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:07 smithi122 ceph-mon[73495]: pgmap v2157: 47 pgs: 47 active+clean; 5.0 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-03-20T09:32:07.829 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:07 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:32:07.829 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:07 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3475366062' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:32:07.829 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:07 smithi122 ceph-mon[73495]: osdmap e893: 12 total, 12 up, 11 in 2024-03-20T09:32:07.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:07 smithi115 ceph-mon[72476]: pgmap v2157: 47 pgs: 47 active+clean; 5.0 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-03-20T09:32:07.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:07 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:32:07.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:07 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3475366062' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:32:07.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:07 smithi115 ceph-mon[72476]: osdmap e893: 12 total, 12 up, 11 in 2024-03-20T09:32:08.361 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:32:08.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:08.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:32:08.469 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:32:08.475 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:32:08.476 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:32:08.503 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:08.531 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:32:08.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:08.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:32:08.634 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:08 smithi026 ceph-mon[68309]: osdmap e894: 12 total, 12 up, 11 in 2024-03-20T09:32:08.635 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:08 smithi026 ceph-mon[68309]: pgmap v2160: 47 pgs: 47 active+clean; 5.2 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-03-20T09:32:08.638 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:32:08.645 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 50 16 88028 88012 109.998 77.4375 0.0147342 0.00908679 2024-03-20T09:32:08.669 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:08.692 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:32:08.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:08.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:32:08.799 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:32:08.827 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:08 smithi115 ceph-mon[72476]: osdmap e894: 12 total, 12 up, 11 in 2024-03-20T09:32:08.827 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:08 smithi115 ceph-mon[72476]: pgmap v2160: 47 pgs: 47 active+clean; 5.2 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-03-20T09:32:08.830 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:08.831 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:08 smithi122 ceph-mon[73495]: osdmap e894: 12 total, 12 up, 11 in 2024-03-20T09:32:08.831 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:08 smithi122 ceph-mon[73495]: pgmap v2160: 47 pgs: 47 active+clean; 5.2 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-03-20T09:32:08.867 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:32:08.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:08.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:32:08.973 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:32:09.002 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:09.027 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:32:09.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:09.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:32:09.135 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:32:09.161 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:32:09.212 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:09.216 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_historic_ops 2024-03-20T09:32:09.433 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:32:09.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:09.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:32:09.540 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:32:09.570 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:32:09.597 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:09.645 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 51 16 89202 89186 109.28 73.375 0.0122205 0.00914663 2024-03-20T09:32:09.670 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.2 to 1 (10000) 2024-03-20T09:32:09.696 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:32:09.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:09.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:32:09.701 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:09 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3073200300' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:32:09.701 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:09 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:32:09.801 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:32:09.862 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:09.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:09 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3073200300' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:32:09.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:09 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:32:10.038 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:32:10.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:10.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:32:10.144 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:32:10.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:09 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3073200300' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:32:10.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:09 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:32:10.186 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:10.645 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 52 16 90287 90271 108.483 67.8125 0.00828734 0.00921337 2024-03-20T09:32:10.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:10 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-20T09:32:10.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:10 smithi122 ceph-mon[73495]: osdmap e895: 12 total, 12 up, 11 in 2024-03-20T09:32:10.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:10 smithi122 ceph-mon[73495]: pgmap v2162: 47 pgs: 47 active+clean; 5.3 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-03-20T09:32:10.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:10 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-20T09:32:10.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:10 smithi026 ceph-mon[68309]: osdmap e895: 12 total, 12 up, 11 in 2024-03-20T09:32:10.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:10 smithi026 ceph-mon[68309]: pgmap v2162: 47 pgs: 47 active+clean; 5.3 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-03-20T09:32:11.026 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:32:11.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:10 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-20T09:32:11.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:10 smithi115 ceph-mon[72476]: osdmap e895: 12 total, 12 up, 11 in 2024-03-20T09:32:11.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:10 smithi115 ceph-mon[72476]: pgmap v2162: 47 pgs: 47 active+clean; 5.3 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-03-20T09:32:11.231 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 5 1 2024-03-20T09:32:11.344 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:32:11.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:11.347 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:32:11.448 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:32:11.518 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:11.645 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 53 16 91341 91325 107.679 65.875 0.00413496 0.00928303 2024-03-20T09:32:11.899 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:32:11.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:11.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:32:12.005 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:32:12.047 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:12.600 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:32:12.600 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:32:12.645 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 54 15 92469 92454 106.991 70.5625 0.00431826 0.00934191 2024-03-20T09:32:12.814 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:32:12.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:12.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:32:12.905 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:32:12.909 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:32:12.920 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:32:12.949 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:32:12.954 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:12.990 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:32:13.044 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:32:13.057 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:32:13.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:13.060 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:32:13.080 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:32:13.162 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:32:13.164 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:32:13.164 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:32:13.164 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:32:13.182 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:32:13.193 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:13.214 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:32:13.220 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:32:13.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:13.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:32:13.234 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:32:13.254 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:32:13.255 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:32:13.255 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:32:13.255 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:32:13.255 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:32:13.255 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:32:13.256 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:32:13.256 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:32:13.256 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:32:13.256 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:32:13.256 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:32:13.256 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:32:13.256 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:32:13.324 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:32:13.358 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:13.645 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 55 15 93393 93378 106.096 57.75 0.020582 0.00942147 2024-03-20T09:32:13.647 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:13 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:32:13.648 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:13 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:32:13.648 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:13 smithi115 ceph-mon[72476]: osdmap e896: 12 total, 12 up, 11 in 2024-03-20T09:32:13.648 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:13 smithi115 ceph-mon[72476]: pgmap v2164: 47 pgs: 47 active+clean; 5.5 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-03-20T09:32:13.648 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:13 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3742979757' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:32:13.672 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:13 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:32:13.672 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:13 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:32:13.672 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:13 smithi026 ceph-mon[68309]: osdmap e896: 12 total, 12 up, 11 in 2024-03-20T09:32:13.672 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:13 smithi026 ceph-mon[68309]: pgmap v2164: 47 pgs: 47 active+clean; 5.5 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-03-20T09:32:13.673 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:13 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3742979757' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:32:13.688 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:13 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:32:13.688 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:13 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:32:13.688 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:13 smithi122 ceph-mon[73495]: osdmap e896: 12 total, 12 up, 11 in 2024-03-20T09:32:13.688 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:13 smithi122 ceph-mon[73495]: pgmap v2164: 47 pgs: 47 active+clean; 5.5 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-03-20T09:32:13.688 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:13 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3742979757' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:32:13.763 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_ops_in_flight 2024-03-20T09:32:13.915 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:32:14.117 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:32:14.645 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 56 16 94602 94586 105.55 75.5 0.00752553 0.00947063 2024-03-20T09:32:14.949 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:32:14.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:14.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:32:15.058 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:32:15.106 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:15.187 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:32:15.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:15.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:32:15.300 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:32:15.334 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:15.366 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:32:15.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:15.372 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:32:15.462 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:15 smithi026 ceph-mon[68309]: pgmap v2165: 47 pgs: 47 active+clean; 5.6 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-03-20T09:32:15.472 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:32:15.503 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:15.529 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:32:15.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:15.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:32:15.633 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:32:15.646 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 57 16 95796 95780 105.007 74.625 0.0166716 0.00951979 2024-03-20T09:32:15.673 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:15.688 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:15 smithi122 ceph-mon[73495]: pgmap v2165: 47 pgs: 47 active+clean; 5.6 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-03-20T09:32:15.701 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:32:15.705 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:15 smithi115 ceph-mon[72476]: pgmap v2165: 47 pgs: 47 active+clean; 5.6 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-03-20T09:32:15.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:15.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:32:15.808 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:32:15.841 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:15.866 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:32:15.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:15.869 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:32:15.970 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:32:16.022 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:16.051 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:32:16.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:16.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:32:16.156 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:32:16.194 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:16.327 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:32:16.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:16.330 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:32:16.431 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:32:16.466 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:16.487 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:32:16.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:16.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:32:16.593 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:32:16.630 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:16.646 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 58 16 96961 96945 104.452 72.8125 0.00879934 0.00956951 2024-03-20T09:32:16.652 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:32:16.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:16.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:32:16.718 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:32:16.718 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-03-20T09:32:16.718 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:16.718 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26605.0:97023 7.a 7:5b8c9252:::benchmark_data_smithi122_187077_object97022:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e896)", 2024-03-20T09:32:16.718 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:32:16.709947+0000", 2024-03-20T09:32:16.718 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.015852337000000001, 2024-03-20T09:32:16.719 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.015886418999999999, 2024-03-20T09:32:16.719 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:32:16.719 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:32:16.719 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:32:16.719 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26605", 2024-03-20T09:32:16.719 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:32:16.719 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 97023 2024-03-20T09:32:16.719 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:16.719 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:32:16.719 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:16.719 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:32:16.719 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:16.709947+0000", 2024-03-20T09:32:16.720 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:16.720 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:16.720 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:16.720 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:32:16.720 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:16.709947+0000", 2024-03-20T09:32:16.720 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:16.720 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:16.720 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:16.720 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:32:16.720 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:16.709948+0000", 2024-03-20T09:32:16.720 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.438e-06 2024-03-20T09:32:16.720 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:16.720 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:16.720 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:32:16.720 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:16.709971+0000", 2024-03-20T09:32:16.720 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2852000000000001e-05 2024-03-20T09:32:16.721 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:16.721 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:16.721 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:32:16.721 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:16.709985+0000", 2024-03-20T09:32:16.721 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4008e-05 2024-03-20T09:32:16.721 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:16.721 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:16.721 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:32:16.721 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:16.709993+0000", 2024-03-20T09:32:16.721 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8189999999999992e-06 2024-03-20T09:32:16.721 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:16.721 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:16.721 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:32:16.721 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:16.710059+0000", 2024-03-20T09:32:16.721 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6750999999999998e-05 2024-03-20T09:32:16.721 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:16.722 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:16.722 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:32:16.722 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:16.710207+0000", 2024-03-20T09:32:16.722 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000147443 2024-03-20T09:32:16.722 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:16.722 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:16.722 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2,11", 2024-03-20T09:32:16.722 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:16.710393+0000", 2024-03-20T09:32:16.722 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00018587000000000001 2024-03-20T09:32:16.722 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:16.722 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:16.722 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:32:16.722 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:16.714509+0000", 2024-03-20T09:32:16.723 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.004116247 2024-03-20T09:32:16.723 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:16.723 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:16.723 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:32:16.723 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:16.716383+0000", 2024-03-20T09:32:16.723 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0018742889999999999 2024-03-20T09:32:16.723 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:16.723 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:16.723 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:32:16.723 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:16.725836+0000", 2024-03-20T09:32:16.723 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0094530710000000004 2024-03-20T09:32:16.723 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:16.723 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:32:16.723 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:16.724 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:16.724 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:16.724 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.26605.0:97042 7.e e896/871 7:7a584061:::benchmark_data_smithi122_187077_object97041:head v 896'6105, mlcod=896'6103)", 2024-03-20T09:32:16.724 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:32:16.725311+0000", 2024-03-20T09:32:16.724 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.00048800699999999999, 2024-03-20T09:32:16.724 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00063612300000000005, 2024-03-20T09:32:16.724 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:32:16.724 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-03-20T09:32:16.724 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:32:16.724 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:16.724 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:32:16.724 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:16.725311+0000", 2024-03-20T09:32:16.724 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:16.725 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:16.725 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:16.725 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:32:16.725 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:16.725311+0000", 2024-03-20T09:32:16.725 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:16.725 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:16.725 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:16.725 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:32:16.725 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:16.725312+0000", 2024-03-20T09:32:16.725 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.569999999999999e-07 2024-03-20T09:32:16.725 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:16.725 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:16.725 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:32:16.725 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:16.725368+0000", 2024-03-20T09:32:16.726 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5841000000000003e-05 2024-03-20T09:32:16.726 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:16.726 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:16.726 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:32:16.726 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:16.725381+0000", 2024-03-20T09:32:16.726 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2992e-05 2024-03-20T09:32:16.726 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:16.726 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:16.726 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:32:16.726 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:16.725388+0000", 2024-03-20T09:32:16.727 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0679999999999999e-06 2024-03-20T09:32:16.727 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:16.727 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:16.727 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:32:16.727 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:16.725455+0000", 2024-03-20T09:32:16.727 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7317000000000006e-05 2024-03-20T09:32:16.727 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:16.727 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:16.727 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:32:16.727 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:16.725479+0000", 2024-03-20T09:32:16.727 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3453000000000001e-05 2024-03-20T09:32:16.727 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:16.727 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:32:16.727 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:16.728 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:16.728 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:16.728 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.26605.0:97023 7.a e896/871 ondisk, result = 0)", 2024-03-20T09:32:16.728 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:32:16.725725+0000", 2024-03-20T09:32:16.728 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 7.3806000000000005e-05, 2024-03-20T09:32:16.728 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00027784400000000002, 2024-03-20T09:32:16.728 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:32:16.728 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-03-20T09:32:16.728 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:32:16.728 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:16.728 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:32:16.728 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:16.725725+0000", 2024-03-20T09:32:16.728 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:16.728 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:16.729 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:16.729 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:32:16.729 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:16.725725+0000", 2024-03-20T09:32:16.729 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:16.729 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:16.729 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:16.729 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:32:16.729 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:16.725726+0000", 2024-03-20T09:32:16.729 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9299999999999998e-07 2024-03-20T09:32:16.729 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:16.729 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:16.729 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:32:16.729 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:16.725730+0000", 2024-03-20T09:32:16.729 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.228e-06 2024-03-20T09:32:16.729 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:16.730 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:16.730 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:32:16.730 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:16.725739+0000", 2024-03-20T09:32:16.730 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2180000000000005e-06 2024-03-20T09:32:16.730 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:16.730 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:16.730 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:32:16.730 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:16.725745+0000", 2024-03-20T09:32:16.730 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1310000000000003e-06 2024-03-20T09:32:16.730 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:16.730 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:16.730 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:32:16.730 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:16.725823+0000", 2024-03-20T09:32:16.730 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7509000000000006e-05 2024-03-20T09:32:16.731 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:16.731 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:16.731 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:32:16.731 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:16.725830+0000", 2024-03-20T09:32:16.731 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6669999999999996e-06 2024-03-20T09:32:16.731 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:16.731 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:16.731 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:32:16.731 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:16.726003+0000", 2024-03-20T09:32:16.731 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00017259799999999999 2024-03-20T09:32:16.731 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:16.731 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:32:16.731 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:16.731 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:16.731 INFO:teuthology.orchestra.run.smithi122.stdout: ], 2024-03-20T09:32:16.732 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 3 2024-03-20T09:32:16.732 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:32:16.756 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:32:16.791 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:16.965 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:32:17.444 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:32:17.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:17.447 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:32:17.490 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:32:17.548 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:32:17.613 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:17.645 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 59 16 97982 97966 103.763 63.8125 0.0158948 0.00963325 2024-03-20T09:32:17.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:17 smithi026 ceph-mon[68309]: pgmap v2166: 47 pgs: 47 active+clean; 5.7 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-03-20T09:32:17.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:17 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/575665349' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:32:17.718 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:17 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:32:17.864 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:17 smithi122 ceph-mon[73495]: pgmap v2166: 47 pgs: 47 active+clean; 5.7 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-03-20T09:32:17.864 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:17 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/575665349' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:32:17.864 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:17 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:32:17.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:17 smithi115 ceph-mon[72476]: pgmap v2166: 47 pgs: 47 active+clean; 5.7 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-03-20T09:32:17.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:17 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/575665349' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:32:17.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:17 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:32:17.901 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_blocked_ops 2024-03-20T09:32:18.225 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:32:18.646 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:2024-03-20T09:32:18.654790+0000 min lat: 0.00116204 max lat: 0.615754 avg lat: 0.00969035 2024-03-20T09:32:18.646 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T09:32:18.646 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 60 15 99052 99037 103.149 66.9375 0.0234477 0.00969035 2024-03-20T09:32:18.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:18 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:32:18.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:18 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:32:18.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:18 smithi026 ceph-mon[68309]: osdmap e897: 12 total, 12 up, 11 in 2024-03-20T09:32:18.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:18 smithi026 ceph-mon[68309]: osdmap e898: 12 total, 12 up, 11 in 2024-03-20T09:32:18.814 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:32:18.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:18.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:32:18.836 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:18 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:32:18.836 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:18 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:32:18.837 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:18 smithi122 ceph-mon[73495]: osdmap e897: 12 total, 12 up, 11 in 2024-03-20T09:32:18.837 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:18 smithi122 ceph-mon[73495]: osdmap e898: 12 total, 12 up, 11 in 2024-03-20T09:32:18.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:18 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:32:18.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:18 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:32:18.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:18 smithi115 ceph-mon[72476]: osdmap e897: 12 total, 12 up, 11 in 2024-03-20T09:32:18.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:18 smithi115 ceph-mon[72476]: osdmap e898: 12 total, 12 up, 11 in 2024-03-20T09:32:18.922 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:32:18.986 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:19.017 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:32:19.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:19.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:32:19.122 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:32:19.184 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:19.471 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:19 smithi026 ceph-mon[68309]: pgmap v2169: 47 pgs: 47 active+clean; 5.9 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-03-20T09:32:19.602 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:19 smithi122 ceph-mon[73495]: pgmap v2169: 47 pgs: 47 active+clean; 5.9 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-03-20T09:32:19.646 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 61 16 100186 100170 102.619 70.8125 0.0043928 0.00973341 2024-03-20T09:32:19.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:19 smithi115 ceph-mon[72476]: pgmap v2169: 47 pgs: 47 active+clean; 5.9 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-03-20T09:32:20.483 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:32:20.484 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:32:20.484 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-03-20T09:32:20.484 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-03-20T09:32:20.484 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:32:20.646 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 62 16 101143 101127 101.929 59.8125 0.0191782 0.00980655 2024-03-20T09:32:20.669 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:32:20.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:20.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:32:20.744 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:32:20.781 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:32:20.810 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:20 smithi115 ceph-mon[72476]: 1.18 scrub starts 2024-03-20T09:32:20.810 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:20 smithi115 ceph-mon[72476]: 1.18 scrub ok 2024-03-20T09:32:20.811 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:20.834 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:32:20.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:20.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:32:20.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:20 smithi122 ceph-mon[73495]: 1.18 scrub starts 2024-03-20T09:32:20.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:20 smithi122 ceph-mon[73495]: 1.18 scrub ok 2024-03-20T09:32:20.940 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:32:20.967 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:20 smithi026 ceph-mon[68309]: 1.18 scrub starts 2024-03-20T09:32:20.967 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:20 smithi026 ceph-mon[68309]: 1.18 scrub ok 2024-03-20T09:32:20.973 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:20.996 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:32:21.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:21.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:32:21.103 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:32:21.137 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:21.159 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:32:21.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:21.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:32:21.268 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:32:21.300 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:21.326 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:32:21.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:21.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:32:21.432 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:32:21.511 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.5 to 1 (10000) 2024-03-20T09:32:21.523 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:21.573 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:21 smithi122 ceph-mon[73495]: 7.2 scrub starts 2024-03-20T09:32:21.573 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:21 smithi122 ceph-mon[73495]: 1.17 scrub starts 2024-03-20T09:32:21.573 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:21 smithi122 ceph-mon[73495]: pgmap v2170: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.0 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 62 MiB/s wr, 985 op/s 2024-03-20T09:32:21.573 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:21 smithi122 ceph-mon[73495]: 1.17 scrub ok 2024-03-20T09:32:21.573 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:21 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/850143723' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:32:21.573 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:21 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:32:21.605 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_historic_ops 2024-03-20T09:32:21.646 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 63 16 102079 102063 101.239 58.5 0.00994787 0.00987276 2024-03-20T09:32:21.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:21 smithi026 ceph-mon[68309]: 7.2 scrub starts 2024-03-20T09:32:21.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:21 smithi026 ceph-mon[68309]: 1.17 scrub starts 2024-03-20T09:32:21.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:21 smithi026 ceph-mon[68309]: pgmap v2170: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.0 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 62 MiB/s wr, 985 op/s 2024-03-20T09:32:21.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:21 smithi026 ceph-mon[68309]: 1.17 scrub ok 2024-03-20T09:32:21.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:21 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/850143723' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:32:21.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:21 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:32:21.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:21 smithi115 ceph-mon[72476]: 7.2 scrub starts 2024-03-20T09:32:21.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:21 smithi115 ceph-mon[72476]: 1.17 scrub starts 2024-03-20T09:32:21.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:21 smithi115 ceph-mon[72476]: pgmap v2170: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.0 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 62 MiB/s wr, 985 op/s 2024-03-20T09:32:21.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:21 smithi115 ceph-mon[72476]: 1.17 scrub ok 2024-03-20T09:32:21.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:21 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/850143723' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:32:21.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:21 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:32:21.961 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:32:22.306 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 6 1 2024-03-20T09:32:22.315 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:32:22.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:22.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:32:22.421 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:32:22.459 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:22.482 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:32:22.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:22.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:32:22.510 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:22 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-20T09:32:22.510 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:22 smithi026 ceph-mon[68309]: osdmap e899: 12 total, 12 up, 11 in 2024-03-20T09:32:22.589 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:32:22.646 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 64 16 103125 103109 100.679 65.375 0.0158712 0.00992803 2024-03-20T09:32:22.660 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:22.727 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:32:22.758 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:32:22.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:22.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:32:22.871 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:32:22.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:22 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-20T09:32:22.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:22 smithi115 ceph-mon[72476]: osdmap e899: 12 total, 12 up, 11 in 2024-03-20T09:32:22.898 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:22.916 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:32:22.918 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:32:22.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:22.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:32:22.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:22 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-20T09:32:22.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:22 smithi122 ceph-mon[73495]: osdmap e899: 12 total, 12 up, 11 in 2024-03-20T09:32:23.023 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:32:23.102 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:23.166 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:32:23.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:23.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:32:23.275 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:32:23.305 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:23.329 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:32:23.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:23.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:32:23.435 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:32:23.472 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:23.646 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 65 16 103979 103963 99.9509 53.375 0.00838441 0.0100003 2024-03-20T09:32:23.648 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:23 smithi115 ceph-mon[72476]: 1.3 scrub starts 2024-03-20T09:32:23.648 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:23 smithi115 ceph-mon[72476]: 7.2 scrub ok 2024-03-20T09:32:23.648 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:23 smithi115 ceph-mon[72476]: 1.3 scrub ok 2024-03-20T09:32:23.648 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:23 smithi115 ceph-mon[72476]: 1.10 scrub starts 2024-03-20T09:32:23.648 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:23 smithi115 ceph-mon[72476]: 1.10 scrub ok 2024-03-20T09:32:23.648 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:23 smithi115 ceph-mon[72476]: 7.5 scrub starts 2024-03-20T09:32:23.648 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:23 smithi115 ceph-mon[72476]: pgmap v2172: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.1 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-03-20T09:32:23.648 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:23 smithi115 ceph-mon[72476]: osdmap e900: 12 total, 12 up, 11 in 2024-03-20T09:32:23.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:23 smithi026 ceph-mon[68309]: 1.3 scrub starts 2024-03-20T09:32:23.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:23 smithi026 ceph-mon[68309]: 7.2 scrub ok 2024-03-20T09:32:23.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:23 smithi026 ceph-mon[68309]: 1.3 scrub ok 2024-03-20T09:32:23.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:23 smithi026 ceph-mon[68309]: 1.10 scrub starts 2024-03-20T09:32:23.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:23 smithi026 ceph-mon[68309]: 1.10 scrub ok 2024-03-20T09:32:23.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:23 smithi026 ceph-mon[68309]: 7.5 scrub starts 2024-03-20T09:32:23.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:23 smithi026 ceph-mon[68309]: pgmap v2172: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.1 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-03-20T09:32:23.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:23 smithi026 ceph-mon[68309]: osdmap e900: 12 total, 12 up, 11 in 2024-03-20T09:32:23.727 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:32:23.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:23 smithi122 ceph-mon[73495]: 1.3 scrub starts 2024-03-20T09:32:23.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:23 smithi122 ceph-mon[73495]: 7.2 scrub ok 2024-03-20T09:32:23.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:23 smithi122 ceph-mon[73495]: 1.3 scrub ok 2024-03-20T09:32:23.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:23 smithi122 ceph-mon[73495]: 1.10 scrub starts 2024-03-20T09:32:23.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:23 smithi122 ceph-mon[73495]: 1.10 scrub ok 2024-03-20T09:32:23.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:23 smithi122 ceph-mon[73495]: 7.5 scrub starts 2024-03-20T09:32:23.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:23 smithi122 ceph-mon[73495]: pgmap v2172: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.1 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-03-20T09:32:23.729 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:23 smithi122 ceph-mon[73495]: osdmap e900: 12 total, 12 up, 11 in 2024-03-20T09:32:23.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:23.730 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:32:23.832 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:32:23.864 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:23.907 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:32:23.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:23.914 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:32:24.015 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:32:24.099 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:24.502 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:32:24.553 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:32:24.582 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:32:24.614 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:32:24.642 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:32:24.646 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 66 16 104883 104867 99.2925 56.5 0.00410942 0.0100507 2024-03-20T09:32:24.669 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:32:24.694 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:32:24.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:24 smithi026 ceph-mon[68309]: 7.3 scrub starts 2024-03-20T09:32:24.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:24 smithi026 ceph-mon[68309]: 1.b scrub starts 2024-03-20T09:32:24.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:24 smithi026 ceph-mon[68309]: 1.1c scrub starts 2024-03-20T09:32:24.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:24 smithi026 ceph-mon[68309]: 1.1c scrub ok 2024-03-20T09:32:24.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:24 smithi026 ceph-mon[68309]: 1.b scrub ok 2024-03-20T09:32:24.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:24 smithi026 ceph-mon[68309]: 1.2 scrub starts 2024-03-20T09:32:24.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:24 smithi026 ceph-mon[68309]: 1.2 scrub ok 2024-03-20T09:32:24.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:24 smithi026 ceph-mon[68309]: 7.5 scrub ok 2024-03-20T09:32:24.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:24 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/29250994' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:32:24.715 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:32:24.719 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:32:24.749 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:32:24.770 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:32:24.790 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:32:24.810 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:32:24.811 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:32:24.811 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:32:24.811 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:32:24.811 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:32:24.812 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:32:24.812 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:32:24.812 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:32:24.812 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:32:24.812 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:32:24.812 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:32:24.812 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:32:24.812 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:32:24.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:24 smithi115 ceph-mon[72476]: 7.3 scrub starts 2024-03-20T09:32:24.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:24 smithi115 ceph-mon[72476]: 1.b scrub starts 2024-03-20T09:32:24.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:24 smithi115 ceph-mon[72476]: 1.1c scrub starts 2024-03-20T09:32:24.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:24 smithi115 ceph-mon[72476]: 1.1c scrub ok 2024-03-20T09:32:24.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:24 smithi115 ceph-mon[72476]: 1.b scrub ok 2024-03-20T09:32:24.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:24 smithi115 ceph-mon[72476]: 1.2 scrub starts 2024-03-20T09:32:24.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:24 smithi115 ceph-mon[72476]: 1.2 scrub ok 2024-03-20T09:32:24.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:24 smithi115 ceph-mon[72476]: 7.5 scrub ok 2024-03-20T09:32:24.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:24 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/29250994' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:32:24.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:24 smithi122 ceph-mon[73495]: 7.3 scrub starts 2024-03-20T09:32:24.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:24 smithi122 ceph-mon[73495]: 1.b scrub starts 2024-03-20T09:32:24.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:24 smithi122 ceph-mon[73495]: 1.1c scrub starts 2024-03-20T09:32:24.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:24 smithi122 ceph-mon[73495]: 1.1c scrub ok 2024-03-20T09:32:24.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:24 smithi122 ceph-mon[73495]: 1.b scrub ok 2024-03-20T09:32:24.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:24 smithi122 ceph-mon[73495]: 1.2 scrub starts 2024-03-20T09:32:24.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:24 smithi122 ceph-mon[73495]: 1.2 scrub ok 2024-03-20T09:32:24.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:24 smithi122 ceph-mon[73495]: 7.5 scrub ok 2024-03-20T09:32:24.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:24 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/29250994' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:32:25.161 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:32:25.161 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:32:25.646 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 67 16 105787 105771 98.6538 56.5 0.0212699 0.0101323 2024-03-20T09:32:25.862 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:25 smithi122 ceph-mon[73495]: 1.f scrub starts 2024-03-20T09:32:25.862 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:25 smithi122 ceph-mon[73495]: 1.f scrub ok 2024-03-20T09:32:25.863 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:25 smithi122 ceph-mon[73495]: pgmap v2174: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.2 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 58 MiB/s wr, 935 op/s 2024-03-20T09:32:25.863 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:25 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:32:25.863 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:25 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:32:25.863 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:25 smithi122 ceph-mon[73495]: 7.0 scrub starts 2024-03-20T09:32:25.863 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:25 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:32:25.863 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:25 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:32:25.886 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:25 smithi026 ceph-mon[68309]: 1.f scrub starts 2024-03-20T09:32:25.886 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:25 smithi026 ceph-mon[68309]: 1.f scrub ok 2024-03-20T09:32:25.886 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:25 smithi026 ceph-mon[68309]: pgmap v2174: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.2 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 58 MiB/s wr, 935 op/s 2024-03-20T09:32:25.886 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:25 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:32:25.886 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:25 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:32:25.886 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:25 smithi026 ceph-mon[68309]: 7.0 scrub starts 2024-03-20T09:32:25.886 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:25 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:32:25.886 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:25 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:32:25.897 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_ops_in_flight 2024-03-20T09:32:25.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:25 smithi115 ceph-mon[72476]: 1.f scrub starts 2024-03-20T09:32:25.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:25 smithi115 ceph-mon[72476]: 1.f scrub ok 2024-03-20T09:32:25.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:25 smithi115 ceph-mon[72476]: pgmap v2174: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.2 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 58 MiB/s wr, 935 op/s 2024-03-20T09:32:25.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:25 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:32:25.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:25 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:32:25.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:25 smithi115 ceph-mon[72476]: 7.0 scrub starts 2024-03-20T09:32:25.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:25 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:32:25.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:25 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:32:26.244 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:32:26.646 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 68 16 106775 106759 98.111 61.75 0.0179188 0.010188 2024-03-20T09:32:26.815 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:26 smithi026 ceph-mon[68309]: 7.3 scrub ok 2024-03-20T09:32:26.816 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:26 smithi026 ceph-mon[68309]: 1.11 scrub starts 2024-03-20T09:32:26.816 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:26 smithi026 ceph-mon[68309]: 1.11 scrub ok 2024-03-20T09:32:26.816 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:26 smithi026 ceph-mon[68309]: osdmap e901: 12 total, 12 up, 11 in 2024-03-20T09:32:26.816 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:26 smithi026 ceph-mon[68309]: 7.d scrub starts 2024-03-20T09:32:26.816 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:26 smithi026 ceph-mon[68309]: 1.c scrub starts 2024-03-20T09:32:26.816 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:26 smithi026 ceph-mon[68309]: 7.b scrub starts 2024-03-20T09:32:26.816 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:26 smithi026 ceph-mon[68309]: 1.c scrub ok 2024-03-20T09:32:26.816 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:26 smithi026 ceph-mon[68309]: pgmap v2176: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.4 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-03-20T09:32:26.841 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:26 smithi122 ceph-mon[73495]: 7.3 scrub ok 2024-03-20T09:32:26.841 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:26 smithi122 ceph-mon[73495]: 1.11 scrub starts 2024-03-20T09:32:26.841 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:26 smithi122 ceph-mon[73495]: 1.11 scrub ok 2024-03-20T09:32:26.841 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:26 smithi122 ceph-mon[73495]: osdmap e901: 12 total, 12 up, 11 in 2024-03-20T09:32:26.841 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:26 smithi122 ceph-mon[73495]: 7.d scrub starts 2024-03-20T09:32:26.841 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:26 smithi122 ceph-mon[73495]: 1.c scrub starts 2024-03-20T09:32:26.841 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:26 smithi122 ceph-mon[73495]: 7.b scrub starts 2024-03-20T09:32:26.841 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:26 smithi122 ceph-mon[73495]: 1.c scrub ok 2024-03-20T09:32:26.841 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:26 smithi122 ceph-mon[73495]: pgmap v2176: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.4 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-03-20T09:32:26.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:26 smithi115 ceph-mon[72476]: 7.3 scrub ok 2024-03-20T09:32:26.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:26 smithi115 ceph-mon[72476]: 1.11 scrub starts 2024-03-20T09:32:26.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:26 smithi115 ceph-mon[72476]: 1.11 scrub ok 2024-03-20T09:32:26.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:26 smithi115 ceph-mon[72476]: osdmap e901: 12 total, 12 up, 11 in 2024-03-20T09:32:26.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:26 smithi115 ceph-mon[72476]: 7.d scrub starts 2024-03-20T09:32:26.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:26 smithi115 ceph-mon[72476]: 1.c scrub starts 2024-03-20T09:32:26.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:26 smithi115 ceph-mon[72476]: 7.b scrub starts 2024-03-20T09:32:26.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:26 smithi115 ceph-mon[72476]: 1.c scrub ok 2024-03-20T09:32:26.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:26 smithi115 ceph-mon[72476]: pgmap v2176: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.4 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-03-20T09:32:27.529 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:32:27.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:27.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:32:27.640 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:32:27.646 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 69 16 107690 107674 97.5179 57.1875 0.0197628 0.0102504 2024-03-20T09:32:27.671 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:27.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:27 smithi115 ceph-mon[72476]: 7.8 scrub starts 2024-03-20T09:32:27.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:27 smithi115 ceph-mon[72476]: 7.0 scrub ok 2024-03-20T09:32:27.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:27 smithi115 ceph-mon[72476]: 7.7 scrub starts 2024-03-20T09:32:27.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:27 smithi115 ceph-mon[72476]: 1.12 scrub starts 2024-03-20T09:32:27.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:27 smithi115 ceph-mon[72476]: 7.d scrub ok 2024-03-20T09:32:27.929 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:32:27.930 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:27 smithi122 ceph-mon[73495]: 7.8 scrub starts 2024-03-20T09:32:27.930 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:27 smithi122 ceph-mon[73495]: 7.0 scrub ok 2024-03-20T09:32:27.930 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:27 smithi122 ceph-mon[73495]: 7.7 scrub starts 2024-03-20T09:32:27.930 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:27 smithi122 ceph-mon[73495]: 1.12 scrub starts 2024-03-20T09:32:27.931 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:27 smithi122 ceph-mon[73495]: 7.d scrub ok 2024-03-20T09:32:27.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:27.932 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:32:27.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:27 smithi026 ceph-mon[68309]: 7.8 scrub starts 2024-03-20T09:32:27.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:27 smithi026 ceph-mon[68309]: 7.0 scrub ok 2024-03-20T09:32:27.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:27 smithi026 ceph-mon[68309]: 7.7 scrub starts 2024-03-20T09:32:27.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:27 smithi026 ceph-mon[68309]: 1.12 scrub starts 2024-03-20T09:32:27.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:27 smithi026 ceph-mon[68309]: 7.d scrub ok 2024-03-20T09:32:28.032 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:32:28.067 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:28.097 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:32:28.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:28.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:32:28.202 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:32:28.238 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:28.402 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:32:28.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:28.407 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:32:28.508 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:32:28.537 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:28.646 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 70 16 108521 108505 96.8667 51.9375 0.00390396 0.0102966 2024-03-20T09:32:28.656 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.6 to 1 (10000) 2024-03-20T09:32:28.737 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:32:28.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:28.740 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:32:28.818 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:32:28.841 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:32:28.868 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:28.886 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:32:28.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:28.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:32:28.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:28 smithi122 ceph-mon[73495]: 1.12 scrub ok 2024-03-20T09:32:28.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:28 smithi122 ceph-mon[73495]: 7.b scrub ok 2024-03-20T09:32:28.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:28 smithi122 ceph-mon[73495]: osdmap e902: 12 total, 12 up, 11 in 2024-03-20T09:32:28.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:28 smithi122 ceph-mon[73495]: 1.8 scrub starts 2024-03-20T09:32:28.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:28 smithi122 ceph-mon[73495]: 7.7 scrub ok 2024-03-20T09:32:28.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:28 smithi122 ceph-mon[73495]: pgmap v2178: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 6.5 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 59 MiB/s wr, 939 op/s 2024-03-20T09:32:28.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:28 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/319157135' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-20T09:32:28.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:28 smithi026 ceph-mon[68309]: 1.12 scrub ok 2024-03-20T09:32:28.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:28 smithi026 ceph-mon[68309]: 7.b scrub ok 2024-03-20T09:32:28.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:28 smithi026 ceph-mon[68309]: osdmap e902: 12 total, 12 up, 11 in 2024-03-20T09:32:28.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:28 smithi026 ceph-mon[68309]: 1.8 scrub starts 2024-03-20T09:32:28.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:28 smithi026 ceph-mon[68309]: 7.7 scrub ok 2024-03-20T09:32:28.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:28 smithi026 ceph-mon[68309]: pgmap v2178: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 6.5 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 59 MiB/s wr, 939 op/s 2024-03-20T09:32:28.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:28 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/319157135' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-20T09:32:28.989 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:32:29.014 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:28 smithi115 ceph-mon[72476]: 1.12 scrub ok 2024-03-20T09:32:29.014 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:28 smithi115 ceph-mon[72476]: 7.b scrub ok 2024-03-20T09:32:29.014 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:28 smithi115 ceph-mon[72476]: osdmap e902: 12 total, 12 up, 11 in 2024-03-20T09:32:29.014 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:28 smithi115 ceph-mon[72476]: 1.8 scrub starts 2024-03-20T09:32:29.014 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:28 smithi115 ceph-mon[72476]: 7.7 scrub ok 2024-03-20T09:32:29.014 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:28 smithi115 ceph-mon[72476]: pgmap v2178: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 6.5 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 59 MiB/s wr, 939 op/s 2024-03-20T09:32:29.014 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:28 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/319157135' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-20T09:32:29.016 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:29.035 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:32:29.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:29.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:32:29.141 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:32:29.194 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:29.209 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:32:29.209 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:32:29.227 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:32:29.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:29.233 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:32:29.334 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:32:29.360 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 7 1 2024-03-20T09:32:29.365 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:29.492 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:32:29.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:29.498 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:32:29.519 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:32:29.599 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:32:29.646 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 71 16 109125 109109 96.034 37.75 0.0181799 0.0104093 2024-03-20T09:32:29.672 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:29.688 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:32:29.922 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:29 smithi122 ceph-mon[73495]: 1.8 scrub ok 2024-03-20T09:32:29.922 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:29 smithi122 ceph-mon[73495]: 7.8 scrub ok 2024-03-20T09:32:29.922 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:29 smithi122 ceph-mon[73495]: 1.19 scrub starts 2024-03-20T09:32:29.923 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:29 smithi122 ceph-mon[73495]: 1.19 scrub ok 2024-03-20T09:32:29.923 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:29 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/319157135' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-20T09:32:29.923 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:29 smithi122 ceph-mon[73495]: osdmap e903: 12 total, 12 up, 11 in 2024-03-20T09:32:29.923 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:29 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/259564885' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:32:29.929 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:29 smithi026 ceph-mon[68309]: 1.8 scrub ok 2024-03-20T09:32:29.929 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:29 smithi026 ceph-mon[68309]: 7.8 scrub ok 2024-03-20T09:32:29.929 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:29 smithi026 ceph-mon[68309]: 1.19 scrub starts 2024-03-20T09:32:29.929 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:29 smithi026 ceph-mon[68309]: 1.19 scrub ok 2024-03-20T09:32:29.930 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:29 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/319157135' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-20T09:32:29.930 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:29 smithi026 ceph-mon[68309]: osdmap e903: 12 total, 12 up, 11 in 2024-03-20T09:32:29.930 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:29 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/259564885' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:32:29.955 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_blocked_ops 2024-03-20T09:32:30.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:29 smithi115 ceph-mon[72476]: 1.8 scrub ok 2024-03-20T09:32:30.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:29 smithi115 ceph-mon[72476]: 7.8 scrub ok 2024-03-20T09:32:30.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:29 smithi115 ceph-mon[72476]: 1.19 scrub starts 2024-03-20T09:32:30.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:29 smithi115 ceph-mon[72476]: 1.19 scrub ok 2024-03-20T09:32:30.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:29 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/319157135' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-20T09:32:30.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:29 smithi115 ceph-mon[72476]: osdmap e903: 12 total, 12 up, 11 in 2024-03-20T09:32:30.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:29 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/259564885' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:32:30.310 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:32:30.647 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 72 16 110192 110176 95.6263 66.6875 0.00888687 0.0104528 2024-03-20T09:32:30.676 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:30 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:32:30.676 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:30 smithi026 ceph-mon[68309]: Cluster is now healthy 2024-03-20T09:32:30.676 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:30 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/259564885' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:32:30.676 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:30 smithi026 ceph-mon[68309]: osdmap e904: 12 total, 12 up, 11 in 2024-03-20T09:32:30.676 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:30 smithi026 ceph-mon[68309]: pgmap v2181: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 6.6 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 55 MiB/s wr, 886 op/s 2024-03-20T09:32:30.858 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:30 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:32:30.859 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:30 smithi122 ceph-mon[73495]: Cluster is now healthy 2024-03-20T09:32:30.859 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:30 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/259564885' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:32:30.859 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:30 smithi122 ceph-mon[73495]: osdmap e904: 12 total, 12 up, 11 in 2024-03-20T09:32:30.859 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:30 smithi122 ceph-mon[73495]: pgmap v2181: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 6.6 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 55 MiB/s wr, 886 op/s 2024-03-20T09:32:30.987 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:32:30.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:30.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:32:31.096 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:32:31.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:30 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:32:31.164 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:30 smithi115 ceph-mon[72476]: Cluster is now healthy 2024-03-20T09:32:31.164 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:30 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/259564885' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:32:31.164 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:30 smithi115 ceph-mon[72476]: osdmap e904: 12 total, 12 up, 11 in 2024-03-20T09:32:31.164 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:30 smithi115 ceph-mon[72476]: pgmap v2181: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 6.6 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 55 MiB/s wr, 886 op/s 2024-03-20T09:32:31.173 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:31.542 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:32:31.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:31.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:32:31.627 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:32:31.647 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 73 16 111204 111188 95.1828 63.25 0.0370935 0.0105019 2024-03-20T09:32:31.648 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:32:31.685 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:31.907 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:31 smithi026 ceph-mon[68309]: 7.a scrub starts 2024-03-20T09:32:31.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:31 smithi122 ceph-mon[73495]: 7.a scrub starts 2024-03-20T09:32:32.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:31 smithi115 ceph-mon[72476]: 7.a scrub starts 2024-03-20T09:32:32.343 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:32:32.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:32.350 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:32:32.451 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:32:32.567 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:32.647 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 74 16 112058 112042 94.6178 53.375 0.024018 0.0105642 2024-03-20T09:32:32.666 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:32:32.708 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:32:32.759 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:32:32.800 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:32:32.837 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:32:32.893 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:32:32.937 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:32:32.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:32 smithi026 ceph-mon[68309]: 7.9 scrub starts 2024-03-20T09:32:32.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:32 smithi026 ceph-mon[68309]: 7.c scrub starts 2024-03-20T09:32:32.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:32 smithi026 ceph-mon[68309]: pgmap v2182: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 6.7 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 59 MiB/s wr, 939 op/s 2024-03-20T09:32:32.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:32 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3946006360' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:32:32.984 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:32:33.001 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:32:33.029 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:32:33.079 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:32:33.146 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:32:33.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:32 smithi115 ceph-mon[72476]: 7.9 scrub starts 2024-03-20T09:32:33.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:32 smithi115 ceph-mon[72476]: 7.c scrub starts 2024-03-20T09:32:33.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:32 smithi115 ceph-mon[72476]: pgmap v2182: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 6.7 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 59 MiB/s wr, 939 op/s 2024-03-20T09:32:33.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:32 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3946006360' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:32:33.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:32 smithi122 ceph-mon[73495]: 7.9 scrub starts 2024-03-20T09:32:33.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:32 smithi122 ceph-mon[73495]: 7.c scrub starts 2024-03-20T09:32:33.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:32 smithi122 ceph-mon[73495]: pgmap v2182: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 6.7 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 59 MiB/s wr, 939 op/s 2024-03-20T09:32:33.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:32 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3946006360' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:32:33.199 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:32:33.199 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:32:33.199 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-03-20T09:32:33.199 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-03-20T09:32:33.199 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:32:33.200 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:32:33.229 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:32:33.229 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:32:33.229 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:32:33.229 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:32:33.230 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:32:33.230 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:32:33.230 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:32:33.230 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:32:33.230 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:32:33.230 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:32:33.230 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:32:33.230 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:32:33.647 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 75 16 112972 112956 94.1178 57.125 0.0156442 0.0106208 2024-03-20T09:32:34.101 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:33 smithi122 ceph-mon[73495]: 1.14 scrub starts 2024-03-20T09:32:34.101 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:33 smithi122 ceph-mon[73495]: 1.14 scrub ok 2024-03-20T09:32:34.101 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:33 smithi122 ceph-mon[73495]: 7.a scrub ok 2024-03-20T09:32:34.101 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:33 smithi122 ceph-mon[73495]: 7.c scrub ok 2024-03-20T09:32:34.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:33 smithi115 ceph-mon[72476]: 1.14 scrub starts 2024-03-20T09:32:34.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:33 smithi115 ceph-mon[72476]: 1.14 scrub ok 2024-03-20T09:32:34.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:33 smithi115 ceph-mon[72476]: 7.a scrub ok 2024-03-20T09:32:34.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:33 smithi115 ceph-mon[72476]: 7.c scrub ok 2024-03-20T09:32:34.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:33 smithi026 ceph-mon[68309]: 1.14 scrub starts 2024-03-20T09:32:34.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:33 smithi026 ceph-mon[68309]: 1.14 scrub ok 2024-03-20T09:32:34.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:33 smithi026 ceph-mon[68309]: 7.a scrub ok 2024-03-20T09:32:34.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:33 smithi026 ceph-mon[68309]: 7.c scrub ok 2024-03-20T09:32:34.500 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_historic_ops 2024-03-20T09:32:34.647 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 76 16 114010 113994 93.733 64.875 0.00377084 0.0106614 2024-03-20T09:32:34.862 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:32:35.129 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:34 smithi122 ceph-mon[73495]: 7.9 scrub ok 2024-03-20T09:32:35.130 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:34 smithi122 ceph-mon[73495]: pgmap v2183: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.8 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 49 MiB/s wr, 782 op/s 2024-03-20T09:32:35.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:34 smithi115 ceph-mon[72476]: 7.9 scrub ok 2024-03-20T09:32:35.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:34 smithi115 ceph-mon[72476]: pgmap v2183: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.8 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 49 MiB/s wr, 782 op/s 2024-03-20T09:32:35.215 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:34 smithi026 ceph-mon[68309]: 7.9 scrub ok 2024-03-20T09:32:35.215 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:34 smithi026 ceph-mon[68309]: pgmap v2183: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.8 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 49 MiB/s wr, 782 op/s 2024-03-20T09:32:35.272 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:32:35.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:35.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:32:35.379 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:32:35.415 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:35.438 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:32:35.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:35.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:32:35.545 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:32:35.597 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:35.647 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 77 16 115006 114990 93.3241 62.25 0.0132156 0.0107116 2024-03-20T09:32:35.761 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:35 smithi122 ceph-mon[73495]: 7.e scrub starts 2024-03-20T09:32:35.762 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:35 smithi115 ceph-mon[72476]: 7.e scrub starts 2024-03-20T09:32:35.966 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:35 smithi026 ceph-mon[68309]: 7.e scrub starts 2024-03-20T09:32:36.647 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 78 16 115896 115880 92.8407 55.625 0.0241336 0.0107674 2024-03-20T09:32:37.051 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:36 smithi122 ceph-mon[73495]: 7.1 scrub starts 2024-03-20T09:32:37.051 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:36 smithi122 ceph-mon[73495]: pgmap v2184: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.9 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 55 MiB/s wr, 885 op/s 2024-03-20T09:32:37.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:36 smithi115 ceph-mon[72476]: 7.1 scrub starts 2024-03-20T09:32:37.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:36 smithi115 ceph-mon[72476]: pgmap v2184: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.9 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 55 MiB/s wr, 885 op/s 2024-03-20T09:32:37.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:36 smithi026 ceph-mon[68309]: 7.1 scrub starts 2024-03-20T09:32:37.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:36 smithi026 ceph-mon[68309]: pgmap v2184: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.9 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 55 MiB/s wr, 885 op/s 2024-03-20T09:32:37.394 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:32:37.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:37.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:32:37.506 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:32:37.551 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:37.585 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:32:37.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:37.588 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:32:37.601 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:32:37.601 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-03-20T09:32:37.601 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-03-20T09:32:37.601 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-03-20T09:32:37.601 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.601 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26216.0:121973 6.8 6:1fbfa6bc:::benchmark_data_smithi122_168655_object1864:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e746)", 2024-03-20T09:32:37.601 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:25:49.661187+0000", 2024-03-20T09:32:37.601 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 407.94592049599999, 2024-03-20T09:32:37.601 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.21986793499999999, 2024-03-20T09:32:37.601 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:32:37.601 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:32:37.601 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:32:37.601 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26216", 2024-03-20T09:32:37.601 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:32:37.602 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 121973 2024-03-20T09:32:37.602 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.602 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:32:37.602 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.602 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:32:37.602 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.661187+0000", 2024-03-20T09:32:37.602 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.602 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.602 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.602 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:32:37.602 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.661187+0000", 2024-03-20T09:32:37.602 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.602 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.602 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.603 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:32:37.603 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.661188+0000", 2024-03-20T09:32:37.603 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.288e-06 2024-03-20T09:32:37.603 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.603 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.603 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:32:37.603 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.661194+0000", 2024-03-20T09:32:37.603 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6710000000000004e-06 2024-03-20T09:32:37.603 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.603 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.603 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:32:37.603 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.661206+0000", 2024-03-20T09:32:37.603 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2335e-05 2024-03-20T09:32:37.603 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.604 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.604 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:32:37.604 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.661214+0000", 2024-03-20T09:32:37.604 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9449999999999993e-06 2024-03-20T09:32:37.604 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.604 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.604 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:32:37.604 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.661262+0000", 2024-03-20T09:32:37.604 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7913e-05 2024-03-20T09:32:37.604 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.604 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.604 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:32:37.604 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.661394+0000", 2024-03-20T09:32:37.604 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013232200000000001 2024-03-20T09:32:37.605 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.605 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.605 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,7", 2024-03-20T09:32:37.605 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.661479+0000", 2024-03-20T09:32:37.605 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4894000000000004e-05 2024-03-20T09:32:37.605 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.605 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.605 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:32:37.605 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.664779+0000", 2024-03-20T09:32:37.605 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0032996409999999999 2024-03-20T09:32:37.605 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.605 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.605 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:32:37.605 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.665554+0000", 2024-03-20T09:32:37.605 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00077524799999999997 2024-03-20T09:32:37.606 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.606 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.606 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:32:37.606 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.880940+0000", 2024-03-20T09:32:37.606 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.215386046 2024-03-20T09:32:37.606 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.606 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.606 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:32:37.606 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.881000+0000", 2024-03-20T09:32:37.606 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0132e-05 2024-03-20T09:32:37.606 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.606 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.606 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:32:37.606 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.881055+0000", 2024-03-20T09:32:37.607 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4500000000000003e-05 2024-03-20T09:32:37.607 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.607 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:32:37.607 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.607 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.607 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.607 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26216.0:121979 6.8 6:198aa212:::benchmark_data_smithi122_168655_object1870:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e746)", 2024-03-20T09:32:37.607 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:25:49.670397+0000", 2024-03-20T09:32:37.607 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 407.93671036199999, 2024-03-20T09:32:37.607 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.21078769, 2024-03-20T09:32:37.608 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:32:37.608 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:32:37.608 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:32:37.608 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26216", 2024-03-20T09:32:37.608 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:32:37.608 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 121979 2024-03-20T09:32:37.608 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.608 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:32:37.608 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.608 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:32:37.608 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.670397+0000", 2024-03-20T09:32:37.608 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.608 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.608 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.609 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:32:37.609 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.670397+0000", 2024-03-20T09:32:37.609 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.609 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.609 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.609 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:32:37.609 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.670399+0000", 2024-03-20T09:32:37.609 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0389999999999999e-06 2024-03-20T09:32:37.609 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.609 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.609 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:32:37.609 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.670410+0000", 2024-03-20T09:32:37.609 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0845999999999999e-05 2024-03-20T09:32:37.609 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.609 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.610 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:32:37.610 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.670431+0000", 2024-03-20T09:32:37.610 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0811000000000001e-05 2024-03-20T09:32:37.610 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.610 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.610 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:32:37.610 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.670443+0000", 2024-03-20T09:32:37.610 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2330000000000001e-05 2024-03-20T09:32:37.610 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.610 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.610 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:32:37.610 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.670519+0000", 2024-03-20T09:32:37.610 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6626999999999997e-05 2024-03-20T09:32:37.610 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.611 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.611 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:32:37.611 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.670662+0000", 2024-03-20T09:32:37.611 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000142292 2024-03-20T09:32:37.611 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.611 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.611 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,7", 2024-03-20T09:32:37.611 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.670792+0000", 2024-03-20T09:32:37.611 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013070200000000001 2024-03-20T09:32:37.611 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.611 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.611 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:32:37.611 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.677843+0000", 2024-03-20T09:32:37.611 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0070506279999999998 2024-03-20T09:32:37.612 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.612 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.612 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:32:37.612 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.678215+0000", 2024-03-20T09:32:37.612 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00037175899999999999 2024-03-20T09:32:37.612 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.612 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.612 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:32:37.612 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.881097+0000", 2024-03-20T09:32:37.612 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20288172700000001 2024-03-20T09:32:37.612 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.612 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.612 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:32:37.612 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.881150+0000", 2024-03-20T09:32:37.613 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3795999999999998e-05 2024-03-20T09:32:37.613 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.613 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.613 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:32:37.613 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.881185+0000", 2024-03-20T09:32:37.613 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4133e-05 2024-03-20T09:32:37.613 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.613 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:32:37.613 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.613 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.613 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.613 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26216.0:121983 6.8 6:11afe848:::benchmark_data_smithi122_168655_object1874:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e746)", 2024-03-20T09:32:37.613 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:25:49.672785+0000", 2024-03-20T09:32:37.613 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 407.93432168499999, 2024-03-20T09:32:37.613 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20855220799999999, 2024-03-20T09:32:37.614 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:32:37.614 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:32:37.614 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:32:37.614 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26216", 2024-03-20T09:32:37.614 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:32:37.614 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 121983 2024-03-20T09:32:37.614 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.614 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:32:37.614 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.614 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:32:37.614 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.672785+0000", 2024-03-20T09:32:37.614 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.614 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.614 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.615 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:32:37.615 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.672785+0000", 2024-03-20T09:32:37.615 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.615 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.615 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.615 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:32:37.615 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.672788+0000", 2024-03-20T09:32:37.615 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1440000000000001e-06 2024-03-20T09:32:37.615 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.615 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.615 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:32:37.615 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.672797+0000", 2024-03-20T09:32:37.615 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.901e-06 2024-03-20T09:32:37.615 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.615 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.616 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:32:37.616 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.672818+0000", 2024-03-20T09:32:37.616 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1903999999999999e-05 2024-03-20T09:32:37.616 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.616 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.616 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:32:37.616 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.672830+0000", 2024-03-20T09:32:37.616 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1195999999999999e-05 2024-03-20T09:32:37.616 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.616 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.617 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:32:37.617 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.673913+0000", 2024-03-20T09:32:37.617 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010829450000000001 2024-03-20T09:32:37.617 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.617 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.617 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:32:37.617 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.674056+0000", 2024-03-20T09:32:37.617 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014391600000000001 2024-03-20T09:32:37.617 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.617 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.618 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,7", 2024-03-20T09:32:37.618 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.674183+0000", 2024-03-20T09:32:37.618 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012637299999999999 2024-03-20T09:32:37.618 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.618 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.618 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:32:37.618 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.677992+0000", 2024-03-20T09:32:37.618 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0038093530000000001 2024-03-20T09:32:37.618 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.619 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.619 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:32:37.619 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.692487+0000", 2024-03-20T09:32:37.619 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014494913999999999 2024-03-20T09:32:37.619 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.619 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.619 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:32:37.619 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.881237+0000", 2024-03-20T09:32:37.619 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18874946500000001 2024-03-20T09:32:37.619 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.619 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.619 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:32:37.619 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.881298+0000", 2024-03-20T09:32:37.619 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1573000000000002e-05 2024-03-20T09:32:37.619 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.620 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.620 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:32:37.620 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.881338+0000", 2024-03-20T09:32:37.620 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9524000000000003e-05 2024-03-20T09:32:37.620 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.620 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:32:37.620 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.620 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.620 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.620 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26216.0:121993 6.8 6:12144f5a:::benchmark_data_smithi122_168655_object1884:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e746)", 2024-03-20T09:32:37.620 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:25:49.680006+0000", 2024-03-20T09:32:37.620 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 407.92710146799999, 2024-03-20T09:32:37.620 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20216557099999999, 2024-03-20T09:32:37.620 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:32:37.620 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:32:37.621 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:32:37.621 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26216", 2024-03-20T09:32:37.621 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:32:37.621 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 121993 2024-03-20T09:32:37.621 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.621 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:32:37.621 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.621 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:32:37.621 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.680006+0000", 2024-03-20T09:32:37.621 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.621 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.621 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.621 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:32:37.621 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.680006+0000", 2024-03-20T09:32:37.622 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.622 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.622 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.622 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:32:37.622 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.680007+0000", 2024-03-20T09:32:37.622 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4300000000000001e-06 2024-03-20T09:32:37.622 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.622 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.622 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:32:37.622 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.680013+0000", 2024-03-20T09:32:37.622 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5310000000000001e-06 2024-03-20T09:32:37.622 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.622 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.622 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:32:37.623 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.680026+0000", 2024-03-20T09:32:37.623 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2941e-05 2024-03-20T09:32:37.623 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.623 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.623 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:32:37.623 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.680032+0000", 2024-03-20T09:32:37.623 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4060000000000002e-06 2024-03-20T09:32:37.623 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.623 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.623 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:32:37.623 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.680085+0000", 2024-03-20T09:32:37.623 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3437999999999999e-05 2024-03-20T09:32:37.623 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.624 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.624 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:32:37.624 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.680224+0000", 2024-03-20T09:32:37.624 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013856200000000001 2024-03-20T09:32:37.624 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.624 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.624 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,7", 2024-03-20T09:32:37.624 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.680328+0000", 2024-03-20T09:32:37.624 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010411700000000001 2024-03-20T09:32:37.625 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.625 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.625 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:32:37.625 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.692505+0000", 2024-03-20T09:32:37.625 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012176424 2024-03-20T09:32:37.625 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.625 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.625 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:32:37.625 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.692529+0000", 2024-03-20T09:32:37.626 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4658999999999999e-05 2024-03-20T09:32:37.626 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.626 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.626 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:32:37.626 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.882108+0000", 2024-03-20T09:32:37.626 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.189578944 2024-03-20T09:32:37.626 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.626 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.626 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:32:37.627 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.882142+0000", 2024-03-20T09:32:37.627 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3368999999999998e-05 2024-03-20T09:32:37.627 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.627 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.627 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:32:37.627 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.882171+0000", 2024-03-20T09:32:37.627 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9750000000000001e-05 2024-03-20T09:32:37.628 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.628 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:32:37.628 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.628 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.628 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.628 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26216.0:121995 6.8 6:10f6e117:::benchmark_data_smithi122_168655_object1886:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e746)", 2024-03-20T09:32:37.629 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:25:49.682067+0000", 2024-03-20T09:32:37.629 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 407.92503989800002, 2024-03-20T09:32:37.629 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.202431376, 2024-03-20T09:32:37.629 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:32:37.629 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:32:37.629 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:32:37.630 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26216", 2024-03-20T09:32:37.630 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:32:37.630 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 121995 2024-03-20T09:32:37.630 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.630 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:32:37.630 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.630 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:32:37.631 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.682067+0000", 2024-03-20T09:32:37.631 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.631 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.631 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.631 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:32:37.631 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.682067+0000", 2024-03-20T09:32:37.631 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.631 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.631 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.632 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:32:37.632 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.682069+0000", 2024-03-20T09:32:37.632 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.294e-06 2024-03-20T09:32:37.632 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.632 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.632 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:32:37.632 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.682073+0000", 2024-03-20T09:32:37.633 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5469999999999998e-06 2024-03-20T09:32:37.633 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.633 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.633 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:32:37.633 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.682088+0000", 2024-03-20T09:32:37.633 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4975e-05 2024-03-20T09:32:37.634 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.634 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.634 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:32:37.634 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.682098+0000", 2024-03-20T09:32:37.634 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7429999999999997e-06 2024-03-20T09:32:37.634 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.634 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.634 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:32:37.635 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.683330+0000", 2024-03-20T09:32:37.635 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001232609 2024-03-20T09:32:37.635 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.635 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.635 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:32:37.635 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.683428+0000", 2024-03-20T09:32:37.635 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7517999999999997e-05 2024-03-20T09:32:37.635 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.635 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.635 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,7", 2024-03-20T09:32:37.635 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.683509+0000", 2024-03-20T09:32:37.635 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1178000000000001e-05 2024-03-20T09:32:37.635 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.636 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.636 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:32:37.636 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.692566+0000", 2024-03-20T09:32:37.636 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0090563740000000007 2024-03-20T09:32:37.636 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.636 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.636 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:32:37.636 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.692578+0000", 2024-03-20T09:32:37.636 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.256e-05 2024-03-20T09:32:37.636 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.636 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.636 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:32:37.636 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.884404+0000", 2024-03-20T09:32:37.637 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19182617800000001 2024-03-20T09:32:37.637 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.637 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.637 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:32:37.637 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.884459+0000", 2024-03-20T09:32:37.637 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4324999999999999e-05 2024-03-20T09:32:37.637 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.637 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.637 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:32:37.637 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.884499+0000", 2024-03-20T09:32:37.637 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0074999999999998e-05 2024-03-20T09:32:37.637 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.637 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:32:37.638 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.638 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.638 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.638 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26216.0:121996 6.8 6:103fc448:::benchmark_data_smithi122_168655_object1887:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e746)", 2024-03-20T09:32:37.638 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:25:49.685531+0000", 2024-03-20T09:32:37.638 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 407.92157631200001, 2024-03-20T09:32:37.638 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20287249600000001, 2024-03-20T09:32:37.638 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:32:37.638 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:32:37.638 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:32:37.638 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26216", 2024-03-20T09:32:37.638 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:32:37.638 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 121996 2024-03-20T09:32:37.639 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.639 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:32:37.639 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.639 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:32:37.639 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.685531+0000", 2024-03-20T09:32:37.639 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.639 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.639 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.639 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:32:37.639 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.685531+0000", 2024-03-20T09:32:37.639 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.639 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.639 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.639 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:32:37.640 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.685532+0000", 2024-03-20T09:32:37.640 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.409e-06 2024-03-20T09:32:37.640 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.640 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.640 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:32:37.640 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.685538+0000", 2024-03-20T09:32:37.640 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6799999999999998e-06 2024-03-20T09:32:37.640 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.640 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.640 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:32:37.640 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.685551+0000", 2024-03-20T09:32:37.640 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3271999999999999e-05 2024-03-20T09:32:37.640 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.640 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.640 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:32:37.641 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.685559+0000", 2024-03-20T09:32:37.641 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6020000000000002e-06 2024-03-20T09:32:37.641 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.641 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.641 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:32:37.641 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.686141+0000", 2024-03-20T09:32:37.641 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00058253000000000005 2024-03-20T09:32:37.641 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.641 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.641 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:32:37.641 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.686256+0000", 2024-03-20T09:32:37.641 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000114231 2024-03-20T09:32:37.641 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.642 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.642 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,7", 2024-03-20T09:32:37.642 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.686332+0000", 2024-03-20T09:32:37.642 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5982000000000005e-05 2024-03-20T09:32:37.642 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.642 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.642 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:32:37.642 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.692584+0000", 2024-03-20T09:32:37.642 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0062523800000000001 2024-03-20T09:32:37.642 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.642 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.643 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:32:37.643 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.693243+0000", 2024-03-20T09:32:37.643 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00065878599999999996 2024-03-20T09:32:37.643 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.643 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.643 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:32:37.643 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.888327+0000", 2024-03-20T09:32:37.643 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.195084219 2024-03-20T09:32:37.643 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.643 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.643 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:32:37.643 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.888361+0000", 2024-03-20T09:32:37.643 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4286999999999999e-05 2024-03-20T09:32:37.643 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.643 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.644 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:32:37.644 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.888403+0000", 2024-03-20T09:32:37.644 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2117999999999999e-05 2024-03-20T09:32:37.644 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.644 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:32:37.644 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.644 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.644 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.644 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26216.0:121999 6.8 6:1ea2bba8:::benchmark_data_smithi122_168655_object1890:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e746)", 2024-03-20T09:32:37.644 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:25:49.685703+0000", 2024-03-20T09:32:37.644 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 407.921403978, 2024-03-20T09:32:37.644 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.204141506, 2024-03-20T09:32:37.644 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:32:37.644 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:32:37.644 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:32:37.645 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26216", 2024-03-20T09:32:37.645 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:32:37.645 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 121999 2024-03-20T09:32:37.645 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.645 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:32:37.645 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.645 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:32:37.645 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.685703+0000", 2024-03-20T09:32:37.645 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.645 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.645 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.645 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:32:37.645 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.685703+0000", 2024-03-20T09:32:37.645 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.645 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.646 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.646 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:32:37.646 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.685704+0000", 2024-03-20T09:32:37.646 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8400000000000004e-07 2024-03-20T09:32:37.646 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.646 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.646 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:32:37.646 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.685708+0000", 2024-03-20T09:32:37.646 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2660000000000003e-06 2024-03-20T09:32:37.646 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.646 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.646 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:32:37.647 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.685719+0000", 2024-03-20T09:32:37.647 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1008e-05 2024-03-20T09:32:37.647 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.647 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.648 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:32:37.648 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.685726+0000", 2024-03-20T09:32:37.648 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6619999999999999e-06 2024-03-20T09:32:37.648 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.648 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.648 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:32:37.648 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.689276+0000", 2024-03-20T09:32:37.648 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0035501550000000002 2024-03-20T09:32:37.648 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.648 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.648 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:32:37.648 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.689433+0000", 2024-03-20T09:32:37.648 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015742100000000001 2024-03-20T09:32:37.648 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.649 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.649 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,7", 2024-03-20T09:32:37.649 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.689533+0000", 2024-03-20T09:32:37.649 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9533999999999999e-05 2024-03-20T09:32:37.649 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.649 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.649 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:32:37.649 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.692611+0000", 2024-03-20T09:32:37.649 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0030784620000000001 2024-03-20T09:32:37.649 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.649 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.649 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:32:37.649 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.694031+0000", 2024-03-20T09:32:37.649 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001419322 2024-03-20T09:32:37.650 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.650 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.650 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:32:37.650 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.889787+0000", 2024-03-20T09:32:37.650 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.195755874 2024-03-20T09:32:37.650 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.650 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.650 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:32:37.650 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.889819+0000", 2024-03-20T09:32:37.650 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2817000000000001e-05 2024-03-20T09:32:37.650 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.650 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.650 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:32:37.651 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:49.889845+0000", 2024-03-20T09:32:37.651 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5301000000000001e-05 2024-03-20T09:32:37.651 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.651 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:32:37.651 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.651 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.651 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.651 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26216.0:129139 6.18 6:1a6b87cd:::benchmark_data_smithi122_168655_object9030:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e749)", 2024-03-20T09:32:37.651 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:25:56.334405+0000", 2024-03-20T09:32:37.651 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 401.27270186800001, 2024-03-20T09:32:37.651 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1316102589999999, 2024-03-20T09:32:37.651 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:32:37.651 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:32:37.651 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:32:37.651 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26216", 2024-03-20T09:32:37.651 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:32:37.652 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 129139 2024-03-20T09:32:37.652 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.652 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:32:37.652 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.652 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:32:37.652 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:56.334405+0000", 2024-03-20T09:32:37.652 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.652 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.652 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.652 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:32:37.652 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:56.334405+0000", 2024-03-20T09:32:37.652 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.652 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.652 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.652 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:32:37.653 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:56.334407+0000", 2024-03-20T09:32:37.653 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4529999999999999e-06 2024-03-20T09:32:37.653 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.653 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.653 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:32:37.653 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:56.334414+0000", 2024-03-20T09:32:37.653 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2679999999999997e-06 2024-03-20T09:32:37.653 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.653 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.653 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:32:37.653 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:56.334428+0000", 2024-03-20T09:32:37.654 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4443999999999999e-05 2024-03-20T09:32:37.654 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.654 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.654 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:32:37.654 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:56.334436+0000", 2024-03-20T09:32:37.654 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8890000000000002e-06 2024-03-20T09:32:37.654 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.654 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.654 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:32:37.654 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:56.487565+0000", 2024-03-20T09:32:37.654 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.153128978 2024-03-20T09:32:37.654 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.654 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.655 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:32:37.655 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:56.487575+0000", 2024-03-20T09:32:37.655 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3160000000000003e-06 2024-03-20T09:32:37.655 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.655 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.655 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:32:37.655 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:57.465932+0000", 2024-03-20T09:32:37.655 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.97835746599999995 2024-03-20T09:32:37.655 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.655 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.655 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:32:37.655 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:25:57.466016+0000", 2024-03-20T09:32:37.655 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3444999999999998e-05 2024-03-20T09:32:37.655 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.656 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:32:37.656 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.656 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.656 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.656 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.26605.0:37643 7.b e880/871 7:dd3319c2:::benchmark_data_smithi122_187077_object37642:head v 880'2366, mlcod=880'2364)", 2024-03-20T09:32:37.656 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:31:33.634943+0000", 2024-03-20T09:32:37.656 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 63.972164114000002, 2024-03-20T09:32:37.656 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.088034921000000002, 2024-03-20T09:32:37.656 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:32:37.656 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:32:37.656 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:32:37.656 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.656 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:32:37.656 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.634943+0000", 2024-03-20T09:32:37.657 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.657 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.657 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.657 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:32:37.657 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.634943+0000", 2024-03-20T09:32:37.657 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.657 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.657 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.657 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:32:37.657 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.634944+0000", 2024-03-20T09:32:37.657 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9600000000000008e-07 2024-03-20T09:32:37.657 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.657 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.658 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:32:37.658 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.635021+0000", 2024-03-20T09:32:37.658 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7070999999999994e-05 2024-03-20T09:32:37.658 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.658 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.658 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:32:37.658 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.635033+0000", 2024-03-20T09:32:37.658 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2122e-05 2024-03-20T09:32:37.658 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.658 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.658 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:32:37.658 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.635040+0000", 2024-03-20T09:32:37.658 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6379999999999998e-06 2024-03-20T09:32:37.658 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.659 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.659 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:32:37.659 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.635091+0000", 2024-03-20T09:32:37.659 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1110999999999999e-05 2024-03-20T09:32:37.659 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.659 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.659 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:32:37.659 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.635110+0000", 2024-03-20T09:32:37.659 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9485999999999999e-05 2024-03-20T09:32:37.659 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.659 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.659 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:32:37.659 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.722933+0000", 2024-03-20T09:32:37.659 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.087822607999999996 2024-03-20T09:32:37.659 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.660 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.660 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:32:37.660 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.722978+0000", 2024-03-20T09:32:37.660 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4888999999999997e-05 2024-03-20T09:32:37.660 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.660 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:32:37.660 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.660 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.660 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.660 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.26605.0:37659 7.b e880/871 7:d0351353:::benchmark_data_smithi122_187077_object37658:head v 880'2367, mlcod=880'2364)", 2024-03-20T09:32:37.660 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:31:33.641074+0000", 2024-03-20T09:32:37.660 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 63.966033482999997, 2024-03-20T09:32:37.660 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.081934724, 2024-03-20T09:32:37.660 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:32:37.661 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:32:37.661 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:32:37.661 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.661 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:32:37.661 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.641074+0000", 2024-03-20T09:32:37.661 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.661 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.661 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.661 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:32:37.661 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.641074+0000", 2024-03-20T09:32:37.662 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.662 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.662 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.662 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:32:37.662 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.641074+0000", 2024-03-20T09:32:37.662 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7300000000000005e-07 2024-03-20T09:32:37.662 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.662 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.662 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:32:37.662 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.641174+0000", 2024-03-20T09:32:37.663 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9967e-05 2024-03-20T09:32:37.663 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.663 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.663 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:32:37.663 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.641185+0000", 2024-03-20T09:32:37.663 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0773e-05 2024-03-20T09:32:37.663 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.663 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.663 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:32:37.663 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.641192+0000", 2024-03-20T09:32:37.663 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.455e-06 2024-03-20T09:32:37.663 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.663 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.664 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:32:37.664 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.641242+0000", 2024-03-20T09:32:37.664 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0611e-05 2024-03-20T09:32:37.664 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.664 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.664 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:32:37.664 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.641263+0000", 2024-03-20T09:32:37.664 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0698000000000001e-05 2024-03-20T09:32:37.664 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.664 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.664 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:32:37.664 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.722982+0000", 2024-03-20T09:32:37.664 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.081719037999999994 2024-03-20T09:32:37.664 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.665 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.665 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:32:37.665 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.723008+0000", 2024-03-20T09:32:37.665 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6409000000000001e-05 2024-03-20T09:32:37.665 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.665 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:32:37.665 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.665 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.665 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.665 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.26605.0:37661 7.b e880/871 7:d5517298:::benchmark_data_smithi122_187077_object37660:head v 880'2368, mlcod=880'2364)", 2024-03-20T09:32:37.666 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:31:33.642545+0000", 2024-03-20T09:32:37.666 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 63.964562409000003, 2024-03-20T09:32:37.666 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.080487900000000001, 2024-03-20T09:32:37.666 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:32:37.666 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:32:37.666 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:32:37.666 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.666 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:32:37.666 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.642545+0000", 2024-03-20T09:32:37.667 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.667 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.667 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.667 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:32:37.667 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.642545+0000", 2024-03-20T09:32:37.667 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.667 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.667 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.667 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:32:37.667 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.642545+0000", 2024-03-20T09:32:37.668 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8699999999999995e-07 2024-03-20T09:32:37.668 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.668 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.668 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:32:37.668 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.642600+0000", 2024-03-20T09:32:37.668 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4725999999999997e-05 2024-03-20T09:32:37.668 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.668 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.668 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:32:37.668 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.642610+0000", 2024-03-20T09:32:37.668 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0114e-05 2024-03-20T09:32:37.669 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.669 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.669 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:32:37.669 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.642618+0000", 2024-03-20T09:32:37.669 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1270000000000003e-06 2024-03-20T09:32:37.669 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.669 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.669 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:32:37.669 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.642660+0000", 2024-03-20T09:32:37.669 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1192000000000001e-05 2024-03-20T09:32:37.669 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.669 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.669 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:32:37.669 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.642676+0000", 2024-03-20T09:32:37.670 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6543999999999999e-05 2024-03-20T09:32:37.670 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.670 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.670 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:32:37.670 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.723012+0000", 2024-03-20T09:32:37.670 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.080335838000000007 2024-03-20T09:32:37.670 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.670 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.670 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:32:37.670 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.723033+0000", 2024-03-20T09:32:37.670 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0772e-05 2024-03-20T09:32:37.670 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.671 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:32:37.671 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.671 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.671 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.671 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.26605.0:37664 7.b e880/871 7:d0c2b077:::benchmark_data_smithi122_187077_object37663:head v 880'2369, mlcod=880'2364)", 2024-03-20T09:32:37.671 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:31:33.644112+0000", 2024-03-20T09:32:37.671 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 63.962994729000002, 2024-03-20T09:32:37.671 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.078948946000000006, 2024-03-20T09:32:37.671 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:32:37.671 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:32:37.671 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:32:37.671 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.671 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:32:37.672 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.644112+0000", 2024-03-20T09:32:37.672 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.672 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.672 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.672 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:32:37.672 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.644112+0000", 2024-03-20T09:32:37.672 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.672 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.672 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.672 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:32:37.672 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.644113+0000", 2024-03-20T09:32:37.672 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3e-07 2024-03-20T09:32:37.672 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.672 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.672 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:32:37.673 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.644176+0000", 2024-03-20T09:32:37.673 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2793999999999995e-05 2024-03-20T09:32:37.673 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.673 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.673 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:32:37.673 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.644184+0000", 2024-03-20T09:32:37.673 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6149999999999997e-06 2024-03-20T09:32:37.673 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.673 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.673 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:32:37.673 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.644190+0000", 2024-03-20T09:32:37.673 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0629999999999996e-06 2024-03-20T09:32:37.673 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.673 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.673 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:32:37.674 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.644229+0000", 2024-03-20T09:32:37.674 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9755999999999997e-05 2024-03-20T09:32:37.674 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.674 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.674 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:32:37.674 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.644248+0000", 2024-03-20T09:32:37.674 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8224e-05 2024-03-20T09:32:37.674 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.674 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.674 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:32:37.674 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.723036+0000", 2024-03-20T09:32:37.674 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.078788475999999996 2024-03-20T09:32:37.674 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.675 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.675 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:32:37.675 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:33.723061+0000", 2024-03-20T09:32:37.675 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5388000000000002e-05 2024-03-20T09:32:37.675 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.675 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:32:37.675 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.675 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.675 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.675 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.26605.0:103517 7.b e900/871 7:dd9badec:::benchmark_data_smithi122_187077_object103516:head v 900'6603, mlcod=900'6600)", 2024-03-20T09:32:37.675 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:32:23.143262+0000", 2024-03-20T09:32:37.675 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 14.463845296000001, 2024-03-20T09:32:37.675 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.064699219000000002, 2024-03-20T09:32:37.675 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:32:37.676 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:32:37.676 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:32:37.676 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.676 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:32:37.676 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.143262+0000", 2024-03-20T09:32:37.676 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.676 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.676 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.676 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:32:37.676 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.143262+0000", 2024-03-20T09:32:37.676 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.676 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.676 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.676 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:32:37.676 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.143269+0000", 2024-03-20T09:32:37.677 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6479999999999997e-06 2024-03-20T09:32:37.677 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.677 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.677 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:32:37.677 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.143318+0000", 2024-03-20T09:32:37.677 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8999000000000003e-05 2024-03-20T09:32:37.677 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.677 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.677 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:32:37.677 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.143329+0000", 2024-03-20T09:32:37.677 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1144999999999999e-05 2024-03-20T09:32:37.677 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.677 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.677 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:32:37.678 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.143335+0000", 2024-03-20T09:32:37.678 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6619999999999999e-06 2024-03-20T09:32:37.678 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.678 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.678 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:32:37.678 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.195323+0000", 2024-03-20T09:32:37.678 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.051987428000000002 2024-03-20T09:32:37.678 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.678 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.678 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:32:37.678 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.195407+0000", 2024-03-20T09:32:37.678 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4104999999999995e-05 2024-03-20T09:32:37.679 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.679 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.679 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:32:37.679 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.207915+0000", 2024-03-20T09:32:37.679 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012507878 2024-03-20T09:32:37.679 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.679 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.679 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:32:37.679 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.207961+0000", 2024-03-20T09:32:37.680 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6353999999999997e-05 2024-03-20T09:32:37.680 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.680 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:32:37.680 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.680 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.680 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.680 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.26605.0:103528 7.b e900/871 7:d1a22673:::benchmark_data_smithi122_187077_object103527:head v 900'6605, mlcod=900'6600)", 2024-03-20T09:32:37.680 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:32:23.152991+0000", 2024-03-20T09:32:37.680 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 14.454116313, 2024-03-20T09:32:37.680 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.118829804, 2024-03-20T09:32:37.680 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:32:37.680 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:32:37.680 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:32:37.680 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.681 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:32:37.681 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.152991+0000", 2024-03-20T09:32:37.681 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.681 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.681 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.681 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:32:37.681 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.152991+0000", 2024-03-20T09:32:37.681 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.681 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.681 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.681 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:32:37.681 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.152992+0000", 2024-03-20T09:32:37.682 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4099999999999998e-07 2024-03-20T09:32:37.682 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.682 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.682 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:32:37.682 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.153062+0000", 2024-03-20T09:32:37.682 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0405000000000001e-05 2024-03-20T09:32:37.682 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.682 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.682 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:32:37.682 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.153075+0000", 2024-03-20T09:32:37.682 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2663000000000001e-05 2024-03-20T09:32:37.682 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.682 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.683 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:32:37.683 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.153081+0000", 2024-03-20T09:32:37.683 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5379999999999999e-06 2024-03-20T09:32:37.683 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.683 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.683 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:32:37.683 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.201366+0000", 2024-03-20T09:32:37.683 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.048284979999999998 2024-03-20T09:32:37.683 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.683 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.683 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:32:37.683 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.201379+0000", 2024-03-20T09:32:37.683 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3209e-05 2024-03-20T09:32:37.683 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.683 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.684 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:32:37.684 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.271774+0000", 2024-03-20T09:32:37.684 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.070394396999999997 2024-03-20T09:32:37.684 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.684 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.684 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:32:37.684 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.271821+0000", 2024-03-20T09:32:37.684 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6870999999999999e-05 2024-03-20T09:32:37.684 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.684 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:32:37.684 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.684 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.684 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.684 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.26605.0:103547 7.b e900/871 7:db1afe55:::benchmark_data_smithi122_187077_object103546:head v 900'6606, mlcod=900'6600)", 2024-03-20T09:32:37.685 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:32:23.175670+0000", 2024-03-20T09:32:37.685 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 14.431437429000001, 2024-03-20T09:32:37.685 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.096181174999999994, 2024-03-20T09:32:37.685 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:32:37.685 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:32:37.685 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:32:37.685 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.685 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:32:37.685 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.175670+0000", 2024-03-20T09:32:37.685 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.685 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.685 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.685 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:32:37.685 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.175670+0000", 2024-03-20T09:32:37.685 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.686 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.686 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.686 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:32:37.686 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.175671+0000", 2024-03-20T09:32:37.686 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3699999999999999e-07 2024-03-20T09:32:37.686 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.686 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.686 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:32:37.686 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.175738+0000", 2024-03-20T09:32:37.686 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6969000000000002e-05 2024-03-20T09:32:37.686 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.686 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.686 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:32:37.687 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.175747+0000", 2024-03-20T09:32:37.687 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.8670000000000006e-06 2024-03-20T09:32:37.687 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.687 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.687 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:32:37.687 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.175753+0000", 2024-03-20T09:32:37.687 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5090000000000001e-06 2024-03-20T09:32:37.687 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.687 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.687 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:32:37.687 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.203914+0000", 2024-03-20T09:32:37.687 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.028161309999999998 2024-03-20T09:32:37.687 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.687 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.688 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:32:37.688 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.203927+0000", 2024-03-20T09:32:37.688 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3010000000000001e-05 2024-03-20T09:32:37.688 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.688 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.688 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:32:37.688 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.271828+0000", 2024-03-20T09:32:37.688 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.067900710000000003 2024-03-20T09:32:37.688 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.688 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.688 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:32:37.689 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.271851+0000", 2024-03-20T09:32:37.689 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2963e-05 2024-03-20T09:32:37.689 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.689 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:32:37.689 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.689 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.690 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.690 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.26605.0:103549 7.b e900/871 7:ddbb94fa:::benchmark_data_smithi122_187077_object103548:head v 900'6607, mlcod=900'6600)", 2024-03-20T09:32:37.690 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:32:23.179297+0000", 2024-03-20T09:32:37.690 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 14.427809714, 2024-03-20T09:32:37.690 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.095781409999999997, 2024-03-20T09:32:37.690 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:32:37.690 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:32:37.690 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:32:37.690 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.690 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:32:37.690 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.179297+0000", 2024-03-20T09:32:37.691 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.691 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.691 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.691 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:32:37.691 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.179297+0000", 2024-03-20T09:32:37.691 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.691 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.691 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.691 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:32:37.691 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.179298+0000", 2024-03-20T09:32:37.691 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6300000000000005e-07 2024-03-20T09:32:37.691 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.691 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.692 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:32:37.692 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.179350+0000", 2024-03-20T09:32:37.692 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2164999999999999e-05 2024-03-20T09:32:37.692 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.692 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.692 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:32:37.692 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.179360+0000", 2024-03-20T09:32:37.692 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.8919999999999995e-06 2024-03-20T09:32:37.692 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.692 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.692 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:32:37.692 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.179365+0000", 2024-03-20T09:32:37.692 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2009999999999998e-06 2024-03-20T09:32:37.692 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.693 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.693 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:32:37.693 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.206114+0000", 2024-03-20T09:32:37.693 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.026748843000000001 2024-03-20T09:32:37.693 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.693 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.693 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:32:37.693 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.206133+0000", 2024-03-20T09:32:37.693 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9326000000000001e-05 2024-03-20T09:32:37.693 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.693 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.693 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:32:37.693 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.275010+0000", 2024-03-20T09:32:37.693 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.068876574999999995 2024-03-20T09:32:37.694 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.694 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.694 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:32:37.694 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.275079+0000", 2024-03-20T09:32:37.694 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8844999999999996e-05 2024-03-20T09:32:37.694 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.694 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:32:37.694 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.694 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.694 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.694 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.26605.0:103561 7.b e900/871 7:d0bff5ee:::benchmark_data_smithi122_187077_object103560:head v 900'6608, mlcod=900'6600)", 2024-03-20T09:32:37.694 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:32:23.197760+0000", 2024-03-20T09:32:37.694 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 14.409347508, 2024-03-20T09:32:37.694 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.077346312, 2024-03-20T09:32:37.695 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:32:37.695 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:32:37.695 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:32:37.695 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.695 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:32:37.695 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.197760+0000", 2024-03-20T09:32:37.695 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.695 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.695 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.695 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:32:37.695 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.197760+0000", 2024-03-20T09:32:37.695 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.695 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.695 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.696 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:32:37.696 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.197761+0000", 2024-03-20T09:32:37.696 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4100000000000001e-06 2024-03-20T09:32:37.696 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.696 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.696 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:32:37.696 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.197824+0000", 2024-03-20T09:32:37.696 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3388999999999999e-05 2024-03-20T09:32:37.696 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.696 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.696 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:32:37.696 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.197835+0000", 2024-03-20T09:32:37.696 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0689999999999999e-05 2024-03-20T09:32:37.696 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.697 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.697 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:32:37.697 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.197843+0000", 2024-03-20T09:32:37.697 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0460000000000006e-06 2024-03-20T09:32:37.697 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.697 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.697 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:32:37.697 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.208013+0000", 2024-03-20T09:32:37.697 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.01017025 2024-03-20T09:32:37.697 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.698 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.698 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:32:37.698 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.208026+0000", 2024-03-20T09:32:37.698 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2588e-05 2024-03-20T09:32:37.698 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.698 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.698 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:32:37.698 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.275083+0000", 2024-03-20T09:32:37.698 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.06705701 2024-03-20T09:32:37.698 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.699 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.699 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:32:37.699 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.275106+0000", 2024-03-20T09:32:37.699 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2929e-05 2024-03-20T09:32:37.699 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.699 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:32:37.700 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.700 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.700 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.700 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "rep_scrubmap(1.b e518 from shard 0)", 2024-03-20T09:32:37.700 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:32:23.206396+0000", 2024-03-20T09:32:37.700 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 14.400711089, 2024-03-20T09:32:37.700 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.065457229000000006, 2024-03-20T09:32:37.700 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:32:37.700 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-03-20T09:32:37.700 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:32:37.700 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.700 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:32:37.701 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.206396+0000", 2024-03-20T09:32:37.701 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.701 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.701 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.701 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:32:37.701 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.206396+0000", 2024-03-20T09:32:37.701 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.701 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.701 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.701 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:32:37.701 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.206397+0000", 2024-03-20T09:32:37.701 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2799999999999995e-07 2024-03-20T09:32:37.701 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.701 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.702 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:32:37.702 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.206402+0000", 2024-03-20T09:32:37.702 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2800000000000003e-06 2024-03-20T09:32:37.702 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.702 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.702 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:32:37.702 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.206412+0000", 2024-03-20T09:32:37.702 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0193999999999999e-05 2024-03-20T09:32:37.702 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.702 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.702 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:32:37.702 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.206418+0000", 2024-03-20T09:32:37.703 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1360000000000002e-06 2024-03-20T09:32:37.703 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.703 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.703 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:32:37.703 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.208003+0000", 2024-03-20T09:32:37.703 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0015848660000000001 2024-03-20T09:32:37.703 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.703 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.703 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:32:37.703 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.208010+0000", 2024-03-20T09:32:37.703 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5520000000000001e-06 2024-03-20T09:32:37.703 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.703 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.703 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:32:37.704 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.271853+0000", 2024-03-20T09:32:37.704 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.063843472999999998 2024-03-20T09:32:37.704 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.704 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:32:37.704 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.704 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.704 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.704 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.26605.0:103570 7.a e900/871 7:599749be:::benchmark_data_smithi122_187077_object103569:head v 900'6483, mlcod=900'6480)", 2024-03-20T09:32:37.704 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:32:23.208780+0000", 2024-03-20T09:32:37.704 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 14.398327321, 2024-03-20T09:32:37.704 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.069344954, 2024-03-20T09:32:37.704 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:32:37.704 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:32:37.704 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:32:37.705 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.705 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:32:37.705 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.208780+0000", 2024-03-20T09:32:37.705 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.705 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.705 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.705 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:32:37.705 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.208780+0000", 2024-03-20T09:32:37.705 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.705 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.705 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.705 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:32:37.705 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.208781+0000", 2024-03-20T09:32:37.705 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.99e-07 2024-03-20T09:32:37.706 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.706 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.706 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:32:37.706 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.208800+0000", 2024-03-20T09:32:37.706 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9429e-05 2024-03-20T09:32:37.706 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.706 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.706 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:32:37.706 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.208808+0000", 2024-03-20T09:32:37.706 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1580000000000005e-06 2024-03-20T09:32:37.706 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.706 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.706 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:32:37.706 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.208813+0000", 2024-03-20T09:32:37.707 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8210000000000001e-06 2024-03-20T09:32:37.707 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.707 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.707 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:32:37.707 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.271368+0000", 2024-03-20T09:32:37.707 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.062555364000000002 2024-03-20T09:32:37.707 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.707 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.707 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:32:37.707 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.271395+0000", 2024-03-20T09:32:37.707 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6466e-05 2024-03-20T09:32:37.707 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.707 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.707 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:32:37.708 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.278068+0000", 2024-03-20T09:32:37.708 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0066734799999999999 2024-03-20T09:32:37.708 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.708 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.708 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:32:37.708 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.278125+0000", 2024-03-20T09:32:37.708 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6536999999999998e-05 2024-03-20T09:32:37.708 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.708 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:32:37.708 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.708 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.708 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.708 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.26605.0:103568 7.9 e900/871 7:985e6ed5:::benchmark_data_smithi122_187077_object103567:head v 900'6574, mlcod=900'6572)", 2024-03-20T09:32:37.708 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:32:23.210502+0000", 2024-03-20T09:32:37.708 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 14.396605382000001, 2024-03-20T09:32:37.709 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.064690480999999994, 2024-03-20T09:32:37.709 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:32:37.709 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:32:37.709 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:32:37.709 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.709 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:32:37.709 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.210502+0000", 2024-03-20T09:32:37.709 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.709 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.709 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.709 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:32:37.709 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.210502+0000", 2024-03-20T09:32:37.709 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:32:37.710 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.710 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.710 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:32:37.710 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.210503+0000", 2024-03-20T09:32:37.710 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.16e-07 2024-03-20T09:32:37.710 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.710 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.710 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:32:37.710 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.210544+0000", 2024-03-20T09:32:37.710 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1473000000000001e-05 2024-03-20T09:32:37.710 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.711 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.711 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:32:37.711 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.271665+0000", 2024-03-20T09:32:37.711 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.061121188999999999 2024-03-20T09:32:37.711 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.711 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.711 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:32:37.711 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.271675+0000", 2024-03-20T09:32:37.711 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6069999999999997e-06 2024-03-20T09:32:37.711 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.711 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.711 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:32:37.711 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.271735+0000", 2024-03-20T09:32:37.711 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0529999999999998e-05 2024-03-20T09:32:37.712 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.712 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.712 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:32:37.712 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.271756+0000", 2024-03-20T09:32:37.712 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0525000000000001e-05 2024-03-20T09:32:37.712 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.712 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.712 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:32:37.712 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.275157+0000", 2024-03-20T09:32:37.712 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0034009880000000002 2024-03-20T09:32:37.712 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:32:37.712 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:32:37.712 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:32:37.712 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:32:23.275192+0000", 2024-03-20T09:32:37.713 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5352999999999998e-05 2024-03-20T09:32:37.713 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.713 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:32:37.713 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.713 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:32:37.713 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:32:37.713 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:32:37.713 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:32:37.715 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 79 16 116787 116771 92.3704 55.6875 0.0401999 0.0108214 2024-03-20T09:32:37.749 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:37.775 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:32:37.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:37.779 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:32:37.880 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:32:37.911 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:37.939 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:32:37.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:37 smithi122 ceph-mon[73495]: 7.f scrub starts 2024-03-20T09:32:37.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:37 smithi122 ceph-mon[73495]: 7.e scrub ok 2024-03-20T09:32:37.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:37 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:32:37.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:37 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:32:37.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:37 smithi122 ceph-mon[73495]: osdmap e905: 12 total, 12 up, 11 in 2024-03-20T09:32:37.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:37 smithi122 ceph-mon[73495]: 1.5 scrub starts 2024-03-20T09:32:37.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:37 smithi122 ceph-mon[73495]: 1.5 scrub ok 2024-03-20T09:32:37.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:37.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:32:38.043 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:32:38.082 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:38.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:37 smithi115 ceph-mon[72476]: 7.f scrub starts 2024-03-20T09:32:38.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:37 smithi115 ceph-mon[72476]: 7.e scrub ok 2024-03-20T09:32:38.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:37 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:32:38.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:37 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:32:38.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:37 smithi115 ceph-mon[72476]: osdmap e905: 12 total, 12 up, 11 in 2024-03-20T09:32:38.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:37 smithi115 ceph-mon[72476]: 1.5 scrub starts 2024-03-20T09:32:38.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:37 smithi115 ceph-mon[72476]: 1.5 scrub ok 2024-03-20T09:32:38.216 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:37 smithi026 ceph-mon[68309]: 7.f scrub starts 2024-03-20T09:32:38.216 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:37 smithi026 ceph-mon[68309]: 7.e scrub ok 2024-03-20T09:32:38.216 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:37 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:32:38.216 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:37 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:32:38.216 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:37 smithi026 ceph-mon[68309]: osdmap e905: 12 total, 12 up, 11 in 2024-03-20T09:32:38.216 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:37 smithi026 ceph-mon[68309]: 1.5 scrub starts 2024-03-20T09:32:38.216 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:37 smithi026 ceph-mon[68309]: 1.5 scrub ok 2024-03-20T09:32:38.288 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:32:38.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:38.291 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:32:38.393 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:32:38.435 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:38.459 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:32:38.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:38.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:32:38.564 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:32:38.601 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:38.632 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:32:38.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:38.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:32:38.647 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:2024-03-20T09:32:38.656525+0000 min lat: 0.00116204 max lat: 0.626907 avg lat: 0.0108764 2024-03-20T09:32:38.647 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T09:32:38.647 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 80 16 117681 117665 91.9142 55.875 0.0134672 0.0108764 2024-03-20T09:32:38.736 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:32:38.770 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:38.780 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-20T09:32:38.803 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:32:38.803 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:38 smithi115 ceph-mon[72476]: 7.1 scrub ok 2024-03-20T09:32:38.803 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:38 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:32:38.803 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:38 smithi115 ceph-mon[72476]: pgmap v2186: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.1 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-03-20T09:32:38.804 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:38 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:32:38.804 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:38 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:32:38.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:38.808 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:32:38.908 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:32:38.956 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:38.988 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:32:38.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:38.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:32:39.093 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:32:39.141 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:39.170 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:32:39.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:39.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:32:39.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:38 smithi122 ceph-mon[73495]: 7.1 scrub ok 2024-03-20T09:32:39.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:38 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:32:39.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:38 smithi122 ceph-mon[73495]: pgmap v2186: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.1 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-03-20T09:32:39.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:38 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:32:39.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:38 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:32:39.194 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:32:39.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:38 smithi026 ceph-mon[68309]: 7.1 scrub ok 2024-03-20T09:32:39.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:38 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:32:39.215 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:38 smithi026 ceph-mon[68309]: pgmap v2186: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.1 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-03-20T09:32:39.215 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:38 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:32:39.215 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:38 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:32:39.274 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:32:39.320 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:39.645 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:32:39.647 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 81 16 118570 118554 91.4653 55.5625 0.01251 0.0109291 2024-03-20T09:32:39.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:39.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:32:39.751 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:32:39.831 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:39.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:39 smithi026 ceph-mon[68309]: 7.f scrub ok 2024-03-20T09:32:39.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:39 smithi026 ceph-mon[68309]: 1.1a scrub starts 2024-03-20T09:32:39.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:39 smithi026 ceph-mon[68309]: 1.1a scrub ok 2024-03-20T09:32:39.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:39 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:32:39.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:39 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:32:40.092 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:32:40.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:40.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:32:40.141 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:32:40.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:39 smithi115 ceph-mon[72476]: 7.f scrub ok 2024-03-20T09:32:40.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:39 smithi115 ceph-mon[72476]: 1.1a scrub starts 2024-03-20T09:32:40.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:39 smithi115 ceph-mon[72476]: 1.1a scrub ok 2024-03-20T09:32:40.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:39 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:32:40.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:39 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:32:40.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:39 smithi122 ceph-mon[73495]: 7.f scrub ok 2024-03-20T09:32:40.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:39 smithi122 ceph-mon[73495]: 1.1a scrub starts 2024-03-20T09:32:40.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:39 smithi122 ceph-mon[73495]: 1.1a scrub ok 2024-03-20T09:32:40.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:39 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:32:40.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:39 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:32:40.200 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:32:40.230 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:40.256 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:32:40.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:40.260 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:32:40.361 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:32:40.411 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:40.495 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:32:40.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:40.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:32:40.598 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:32:40.647 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 82 16 119445 119429 91.0168 54.6875 0.00783651 0.0109827 2024-03-20T09:32:40.685 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:40.936 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:40 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:32:40.937 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:40 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:32:40.937 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:40 smithi026 ceph-mon[68309]: osdmap e906: 12 total, 12 up, 11 in 2024-03-20T09:32:40.937 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:40 smithi026 ceph-mon[68309]: 7.6 scrub starts 2024-03-20T09:32:40.937 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:40 smithi026 ceph-mon[68309]: pgmap v2188: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.1 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 52 MiB/s wr, 832 op/s 2024-03-20T09:32:40.937 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:40 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:32:40.937 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:40 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:32:40.937 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:40 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:32:41.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:40 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:32:41.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:40 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:32:41.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:40 smithi115 ceph-mon[72476]: osdmap e906: 12 total, 12 up, 11 in 2024-03-20T09:32:41.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:40 smithi115 ceph-mon[72476]: 7.6 scrub starts 2024-03-20T09:32:41.149 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:40 smithi115 ceph-mon[72476]: pgmap v2188: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.1 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 52 MiB/s wr, 832 op/s 2024-03-20T09:32:41.149 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:40 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:32:41.149 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:40 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:32:41.149 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:40 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:32:41.171 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:32:41.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:41.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:32:41.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:40 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:32:41.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:40 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:32:41.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:40 smithi122 ceph-mon[73495]: osdmap e906: 12 total, 12 up, 11 in 2024-03-20T09:32:41.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:40 smithi122 ceph-mon[73495]: 7.6 scrub starts 2024-03-20T09:32:41.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:40 smithi122 ceph-mon[73495]: pgmap v2188: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.1 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 52 MiB/s wr, 832 op/s 2024-03-20T09:32:41.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:40 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:32:41.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:40 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:32:41.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:40 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:32:41.278 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:32:41.297 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:32:41.298 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [ 2024-03-20T09:32:41.298 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:32:41.298 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:120009 7.0 7:06bae3df:::benchmark_data_smithi122_187077_object120008:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e906)", 2024-03-20T09:32:41.298 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:32:41.296476+0000", 2024-03-20T09:32:41.298 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.0095215179999999996, 2024-03-20T09:32:41.298 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0095474059999999996, 2024-03-20T09:32:41.298 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:32:41.298 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:32:41.298 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:32:41.298 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:32:41.298 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:32:41.298 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 120009 2024-03-20T09:32:41.298 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:32:41.299 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:32:41.299 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:32:41.299 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:32:41.299 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:41.296476+0000", 2024-03-20T09:32:41.299 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:32:41.299 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:32:41.299 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:32:41.299 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:32:41.299 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:41.296476+0000", 2024-03-20T09:32:41.299 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:32:41.299 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:32:41.299 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:32:41.299 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:32:41.299 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:41.296478+0000", 2024-03-20T09:32:41.299 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.505e-06 2024-03-20T09:32:41.300 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:32:41.300 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:32:41.300 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:32:41.300 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:41.296538+0000", 2024-03-20T09:32:41.300 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0238000000000002e-05 2024-03-20T09:32:41.300 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:32:41.300 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:32:41.300 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:32:41.300 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:41.296557+0000", 2024-03-20T09:32:41.300 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9259e-05 2024-03-20T09:32:41.300 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:32:41.300 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:32:41.300 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:32:41.300 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:41.296567+0000", 2024-03-20T09:32:41.301 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0115999999999999e-05 2024-03-20T09:32:41.301 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:32:41.301 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:32:41.301 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:32:41.301 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:41.296632+0000", 2024-03-20T09:32:41.301 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4778999999999998e-05 2024-03-20T09:32:41.301 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:32:41.301 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:32:41.301 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:32:41.301 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:41.296795+0000", 2024-03-20T09:32:41.301 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00016253799999999999 2024-03-20T09:32:41.301 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:32:41.301 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:32:41.301 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 3,9", 2024-03-20T09:32:41.301 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:41.296946+0000", 2024-03-20T09:32:41.302 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00015148900000000001 2024-03-20T09:32:41.302 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:32:41.302 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:32:41.302 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:32:41.302 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:41.301357+0000", 2024-03-20T09:32:41.302 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0044106010000000001 2024-03-20T09:32:41.302 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:32:41.302 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:32:41.302 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:32:41.302 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:41.303601+0000", 2024-03-20T09:32:41.302 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0022440149999999998 2024-03-20T09:32:41.302 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:32:41.302 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:32:41.302 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:32:41.303 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:32:41.303 INFO:teuthology.orchestra.run.smithi115.stdout: ], 2024-03-20T09:32:41.303 INFO:teuthology.orchestra.run.smithi115.stdout: "num_ops": 1 2024-03-20T09:32:41.303 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:32:41.361 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:41.647 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 83 16 120294 120278 90.5595 53.0625 0.00934058 0.0110356 2024-03-20T09:32:41.703 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:32:41.986 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.7 to 1 (10000) 2024-03-20T09:32:42.014 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:41 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2532058247' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:32:42.014 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:41 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:32:42.033 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:32:42.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:42.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:32:42.043 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-20T09:32:42.139 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:32:42.207 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:42.400 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:41 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2532058247' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:32:42.400 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:41 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:32:42.431 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:32:42.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:41 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2532058247' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:32:42.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:41 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:32:42.647 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 84 16 121048 121032 90.0424 47.125 0.0139266 0.0111015 2024-03-20T09:32:43.346 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:42 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-20T09:32:43.346 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:42 smithi026 ceph-mon[68309]: osdmap e907: 12 total, 12 up, 11 in 2024-03-20T09:32:43.346 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:42 smithi026 ceph-mon[68309]: 1.16 scrub starts 2024-03-20T09:32:43.346 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:42 smithi026 ceph-mon[68309]: 1.16 scrub ok 2024-03-20T09:32:43.346 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:42 smithi026 ceph-mon[68309]: pgmap v2190: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.3 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 60 MiB/s wr, 953 op/s 2024-03-20T09:32:43.351 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:32:43.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:43.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:32:43.385 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:42 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-20T09:32:43.386 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:42 smithi115 ceph-mon[72476]: osdmap e907: 12 total, 12 up, 11 in 2024-03-20T09:32:43.386 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:42 smithi115 ceph-mon[72476]: 1.16 scrub starts 2024-03-20T09:32:43.386 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:42 smithi115 ceph-mon[72476]: 1.16 scrub ok 2024-03-20T09:32:43.386 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:42 smithi115 ceph-mon[72476]: pgmap v2190: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.3 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 60 MiB/s wr, 953 op/s 2024-03-20T09:32:43.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:42 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-20T09:32:43.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:42 smithi122 ceph-mon[73495]: osdmap e907: 12 total, 12 up, 11 in 2024-03-20T09:32:43.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:42 smithi122 ceph-mon[73495]: 1.16 scrub starts 2024-03-20T09:32:43.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:42 smithi122 ceph-mon[73495]: 1.16 scrub ok 2024-03-20T09:32:43.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:42 smithi122 ceph-mon[73495]: pgmap v2190: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.3 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 60 MiB/s wr, 953 op/s 2024-03-20T09:32:43.461 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:32:43.492 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:43.513 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:32:43.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:43.516 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:32:43.617 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:32:43.645 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:43.647 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 85 16 121945 121929 89.6426 56.0625 0.0128555 0.0111511 2024-03-20T09:32:43.666 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:32:43.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:43.670 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:32:43.771 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:32:43.842 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:43.922 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:32:43.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:43.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:32:44.036 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:32:44.069 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:44.093 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:32:44.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:44.096 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:32:44.197 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:32:44.227 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:44.251 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:32:44.253 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:43 smithi122 ceph-mon[73495]: 7.6 scrub ok 2024-03-20T09:32:44.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:44.254 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:32:44.355 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:32:44.401 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:43 smithi115 ceph-mon[72476]: 7.6 scrub ok 2024-03-20T09:32:44.407 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:43 smithi026 ceph-mon[68309]: 7.6 scrub ok 2024-03-20T09:32:44.415 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:44.480 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:32:44.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:44.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:32:44.548 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:32:44.548 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:32:44.548 INFO:teuthology.orchestra.run.smithi115.stdout: "complaint_time": 30, 2024-03-20T09:32:44.548 INFO:teuthology.orchestra.run.smithi115.stdout: "num_blocked_ops": 0 2024-03-20T09:32:44.548 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:32:44.593 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:32:44.631 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:44.647 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 86 16 122912 122896 89.303 60.4375 0.0120563 0.0111941 2024-03-20T09:32:44.657 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:32:44.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:44.660 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:32:44.711 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:32:44.761 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:32:44.804 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 8 1 2024-03-20T09:32:44.839 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:44.912 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:32:44.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:44.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:32:45.012 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:45 smithi026 ceph-mon[68309]: pgmap v2191: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.3 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 44 MiB/s wr, 699 op/s 2024-03-20T09:32:45.017 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:32:45.043 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:45.065 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:32:45.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:45.071 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:32:45.172 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:32:45.208 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:45.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:45 smithi115 ceph-mon[72476]: pgmap v2191: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.3 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 44 MiB/s wr, 699 op/s 2024-03-20T09:32:45.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:45 smithi122 ceph-mon[73495]: pgmap v2191: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.3 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 44 MiB/s wr, 699 op/s 2024-03-20T09:32:45.648 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 87 16 123731 123715 88.8648 51.1875 0.0205134 0.0112484 2024-03-20T09:32:46.014 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:46 smithi026 ceph-mon[68309]: 1.1d scrub starts 2024-03-20T09:32:46.014 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:46 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3181706347' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:32:46.014 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:46 smithi026 ceph-mon[68309]: 1.1d scrub ok 2024-03-20T09:32:46.042 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:32:46.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:46 smithi115 ceph-mon[72476]: 1.1d scrub starts 2024-03-20T09:32:46.398 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:46 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3181706347' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:32:46.398 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:46 smithi115 ceph-mon[72476]: 1.1d scrub ok 2024-03-20T09:32:46.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:46 smithi122 ceph-mon[73495]: 1.1d scrub starts 2024-03-20T09:32:46.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:46 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3181706347' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:32:46.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:46 smithi122 ceph-mon[73495]: 1.1d scrub ok 2024-03-20T09:32:46.648 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 88 16 124754 124738 88.5815 63.9375 0.0204645 0.011285 2024-03-20T09:32:47.120 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:47 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:32:47.120 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:47 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3181706347' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:32:47.120 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:47 smithi026 ceph-mon[68309]: osdmap e908: 12 total, 12 up, 11 in 2024-03-20T09:32:47.120 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:47 smithi026 ceph-mon[68309]: pgmap v2193: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.5 GiB data, 30 GiB used, 954 GiB / 983 GiB avail; 61 MiB/s wr, 968 op/s 2024-03-20T09:32:47.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:47 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:32:47.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:47 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3181706347' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:32:47.400 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:47 smithi115 ceph-mon[72476]: osdmap e908: 12 total, 12 up, 11 in 2024-03-20T09:32:47.400 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:47 smithi115 ceph-mon[72476]: pgmap v2193: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.5 GiB data, 30 GiB used, 954 GiB / 983 GiB avail; 61 MiB/s wr, 968 op/s 2024-03-20T09:32:47.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:47 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:32:47.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:47 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3181706347' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:32:47.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:47 smithi122 ceph-mon[73495]: osdmap e908: 12 total, 12 up, 11 in 2024-03-20T09:32:47.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:47 smithi122 ceph-mon[73495]: pgmap v2193: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.5 GiB data, 30 GiB used, 954 GiB / 983 GiB avail; 61 MiB/s wr, 968 op/s 2024-03-20T09:32:47.519 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:32:47.648 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:32:47.648 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 89 16 125714 125698 88.2603 60 0.00935565 0.0113264 2024-03-20T09:32:47.691 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:32:47.693 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:32:47.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:47.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:32:47.717 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-20T09:32:47.750 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:32:47.794 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:32:47.796 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:32:47.825 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:47.857 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:32:47.862 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:32:47.926 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:32:47.970 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:32:48.003 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:32:48.035 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:32:48.072 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:32:48.089 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:32:48.094 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:48 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2937766867' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:32:48.103 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:32:48.127 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:32:48.134 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:32:48.134 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:32:48.134 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:32:48.134 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:32:48.134 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:32:48.134 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:32:48.134 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:32:48.134 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:32:48.134 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:32:48.134 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:32:48.134 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:32:48.134 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:32:48.149 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:48 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2937766867' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:32:48.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:48.151 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:32:48.252 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:32:48.285 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:48 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2937766867' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:32:48.287 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:48.366 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:32:48.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:48.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:32:48.472 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:32:48.503 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:48.527 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:32:48.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:48.531 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:32:48.632 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:32:48.648 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 90 3 126608 126605 87.9095 56.6875 0.0126707 0.0113719 2024-03-20T09:32:48.671 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:48.875 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:32:49.120 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:49 smithi115 ceph-mon[72476]: pgmap v2194: 47 pgs: 47 active+clean; 7.6 GiB data, 30 GiB used, 953 GiB / 983 GiB avail; 60 MiB/s wr, 960 op/s 2024-03-20T09:32:49.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:49 smithi122 ceph-mon[73495]: pgmap v2194: 47 pgs: 47 active+clean; 7.6 GiB data, 30 GiB used, 953 GiB / 983 GiB avail; 60 MiB/s wr, 960 op/s 2024-03-20T09:32:49.462 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:49 smithi026 ceph-mon[68309]: pgmap v2194: 47 pgs: 47 active+clean; 7.6 GiB data, 30 GiB used, 953 GiB / 983 GiB avail; 60 MiB/s wr, 960 op/s 2024-03-20T09:32:49.648 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Total time run: 90.0118 2024-03-20T09:32:49.648 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Total writes made: 126608 2024-03-20T09:32:49.648 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Write size: 65536 2024-03-20T09:32:49.649 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Object size: 65536 2024-03-20T09:32:49.649 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Bandwidth (MB/sec): 87.9107 2024-03-20T09:32:49.649 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Stddev Bandwidth: 38.2314 2024-03-20T09:32:49.649 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Max bandwidth (MB/sec): 235.188 2024-03-20T09:32:49.649 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Min bandwidth (MB/sec): 37.75 2024-03-20T09:32:49.649 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Average IOPS: 1406 2024-03-20T09:32:49.649 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Stddev IOPS: 611.703 2024-03-20T09:32:49.649 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Max IOPS: 3763 2024-03-20T09:32:49.649 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Min IOPS: 604 2024-03-20T09:32:49.649 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Average Latency(s): 0.0113719 2024-03-20T09:32:49.649 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Stddev Latency(s): 0.0154693 2024-03-20T09:32:49.649 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Max latency(s): 0.626907 2024-03-20T09:32:49.649 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Min latency(s): 0.00116204 2024-03-20T09:32:49.655 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Cleaning up (deleting benchmark objects) 2024-03-20T09:32:50.355 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:32:50.355 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:32:50.638 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:32:50.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:50.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:32:50.752 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:32:50.801 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:51.137 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_ops_in_flight 2024-03-20T09:32:51.158 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:32:51.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:51.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:32:51.229 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:32:51.263 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:32:51.343 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:51.397 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:32:51.405 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:51 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:32:51.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:51.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:32:51.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:51 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:32:51.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:51 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:32:51.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:51 smithi122 ceph-mon[73495]: osdmap e909: 12 total, 12 up, 11 in 2024-03-20T09:32:51.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:51 smithi122 ceph-mon[73495]: pgmap v2196: 47 pgs: 47 active+clean; 7.6 GiB data, 30 GiB used, 953 GiB / 983 GiB avail; 47 MiB/s wr, 764 op/s 2024-03-20T09:32:51.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:51 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:32:51.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:51 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:32:51.466 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:32:51.508 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:32:51.541 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:51.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:51 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:32:51.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:51 smithi026 ceph-mon[68309]: osdmap e909: 12 total, 12 up, 11 in 2024-03-20T09:32:51.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:51 smithi026 ceph-mon[68309]: pgmap v2196: 47 pgs: 47 active+clean; 7.6 GiB data, 30 GiB used, 953 GiB / 983 GiB avail; 47 MiB/s wr, 764 op/s 2024-03-20T09:32:51.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:51 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:32:51.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:51 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:32:51.746 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:32:51.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:51.749 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:32:51.850 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:32:51.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:51 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:32:51.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:51 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:32:51.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:51 smithi115 ceph-mon[72476]: osdmap e909: 12 total, 12 up, 11 in 2024-03-20T09:32:51.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:51 smithi115 ceph-mon[72476]: pgmap v2196: 47 pgs: 47 active+clean; 7.6 GiB data, 30 GiB used, 953 GiB / 983 GiB avail; 47 MiB/s wr, 764 op/s 2024-03-20T09:32:51.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:51 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:32:51.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:51 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:32:51.918 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:52.646 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:32:52.688 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:52 smithi122 ceph-mon[73495]: osdmap e910: 12 total, 12 up, 11 in 2024-03-20T09:32:52.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:52 smithi026 ceph-mon[68309]: osdmap e910: 12 total, 12 up, 11 in 2024-03-20T09:32:52.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:52 smithi115 ceph-mon[72476]: osdmap e910: 12 total, 12 up, 11 in 2024-03-20T09:32:53.426 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:32:53.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:53.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:32:53.464 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.8 to 1 (10000) 2024-03-20T09:32:53.536 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:32:53.622 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:53.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:53 smithi122 ceph-mon[73495]: pgmap v2198: 47 pgs: 47 active+clean; 7.7 GiB data, 30 GiB used, 953 GiB / 983 GiB avail; 160 B/s rd, 49 MiB/s wr, 940 op/s 2024-03-20T09:32:53.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:53 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1667660095' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:32:53.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:53 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:32:53.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:53 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:32:53.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:53 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:32:53.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:53 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:32:53.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:53 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:32:53.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:53 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:32:53.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:53 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:32:53.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:53 smithi026 ceph-mon[68309]: pgmap v2198: 47 pgs: 47 active+clean; 7.7 GiB data, 30 GiB used, 953 GiB / 983 GiB avail; 160 B/s rd, 49 MiB/s wr, 940 op/s 2024-03-20T09:32:53.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:53 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1667660095' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:32:53.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:53 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:32:53.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:53 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:32:53.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:53 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:32:53.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:53 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:32:53.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:53 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:32:53.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:53 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:32:53.716 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:53 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:32:53.816 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:32:53.816 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:32:53.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:53 smithi115 ceph-mon[72476]: pgmap v2198: 47 pgs: 47 active+clean; 7.7 GiB data, 30 GiB used, 953 GiB / 983 GiB avail; 160 B/s rd, 49 MiB/s wr, 940 op/s 2024-03-20T09:32:53.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:53 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1667660095' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:32:53.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:53 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:32:53.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:53 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:32:53.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:53 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:32:53.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:53 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:32:53.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:53 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:32:53.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:53 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:32:53.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:53 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:32:54.430 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:54 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1667660095' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-03-20T09:32:54.431 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:54 smithi026 ceph-mon[68309]: osdmap e911: 12 total, 12 up, 11 in 2024-03-20T09:32:54.688 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:54 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1667660095' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-03-20T09:32:54.688 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:54 smithi122 ceph-mon[73495]: osdmap e911: 12 total, 12 up, 11 in 2024-03-20T09:32:54.823 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:32:54.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:54.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:32:54.850 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_blocked_ops 2024-03-20T09:32:54.872 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 11 1 2024-03-20T09:32:54.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:54 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1667660095' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-03-20T09:32:54.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:54 smithi115 ceph-mon[72476]: osdmap e911: 12 total, 12 up, 11 in 2024-03-20T09:32:54.919 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:32:54.931 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:32:54.965 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:54.991 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:32:54.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:54.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:32:55.096 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:32:55.131 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:55.157 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:32:55.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:55.163 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:32:55.184 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:32:55.264 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:32:55.308 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:55.356 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:32:55.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:55.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:32:55.461 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:32:55.465 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:55 smithi026 ceph-mon[68309]: pgmap v2200: 47 pgs: 47 active+clean; 7.6 GiB data, 30 GiB used, 953 GiB / 983 GiB avail; 170 B/s rd, 24 MiB/s wr, 655 op/s 2024-03-20T09:32:55.550 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:55.745 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:55 smithi122 ceph-mon[73495]: pgmap v2200: 47 pgs: 47 active+clean; 7.6 GiB data, 30 GiB used, 953 GiB / 983 GiB avail; 170 B/s rd, 24 MiB/s wr, 655 op/s 2024-03-20T09:32:55.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:55 smithi115 ceph-mon[72476]: pgmap v2200: 47 pgs: 47 active+clean; 7.6 GiB data, 30 GiB used, 953 GiB / 983 GiB avail; 170 B/s rd, 24 MiB/s wr, 655 op/s 2024-03-20T09:32:56.031 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:32:56.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:56.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:32:56.140 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:32:56.181 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:56.531 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:32:56.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:56.535 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:32:56.636 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:32:56.718 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:57.362 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:32:57.362 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:32:57.400 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:32:57.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:57.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:32:57.510 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:32:57.519 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:32:57.557 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:57.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:57 smithi122 ceph-mon[73495]: pgmap v2201: 47 pgs: 47 active+clean; 7.5 GiB data, 30 GiB used, 954 GiB / 983 GiB avail; 170 B/s rd, 15 MiB/s wr, 935 op/s 2024-03-20T09:32:57.688 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:57 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1654131762' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:32:57.688 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:57 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:32:57.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:57 smithi026 ceph-mon[68309]: pgmap v2201: 47 pgs: 47 active+clean; 7.5 GiB data, 30 GiB used, 954 GiB / 983 GiB avail; 170 B/s rd, 15 MiB/s wr, 935 op/s 2024-03-20T09:32:57.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:57 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1654131762' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:32:57.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:57 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:32:57.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:57 smithi115 ceph-mon[72476]: pgmap v2201: 47 pgs: 47 active+clean; 7.5 GiB data, 30 GiB used, 954 GiB / 983 GiB avail; 170 B/s rd, 15 MiB/s wr, 935 op/s 2024-03-20T09:32:57.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:57 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1654131762' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:32:57.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:57 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:32:58.131 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_historic_ops 2024-03-20T09:32:58.229 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:32:58.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:58.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:32:58.336 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:32:58.382 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:58.411 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:32:58.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:58.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:32:58.474 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:32:58.495 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:58 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:32:58.495 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:58 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:32:58.495 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:58 smithi026 ceph-mon[68309]: osdmap e912: 12 total, 12 up, 11 in 2024-03-20T09:32:58.516 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:32:58.582 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:58.647 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:58 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:32:58.647 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:58 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:32:58.647 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:58 smithi115 ceph-mon[72476]: osdmap e912: 12 total, 12 up, 11 in 2024-03-20T09:32:58.761 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:58 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:32:58.761 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:58 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:32:58.761 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:58 smithi122 ceph-mon[73495]: osdmap e912: 12 total, 12 up, 11 in 2024-03-20T09:32:58.826 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:32:58.827 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:32:58.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:32:58.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:32:58.953 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:32:58.992 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:32:59.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:32:59 smithi115 ceph-mon[72476]: pgmap v2203: 47 pgs: 47 active+clean; 7.3 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 5.2 MiB/s wr, 923 op/s 2024-03-20T09:32:59.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:32:59 smithi122 ceph-mon[73495]: pgmap v2203: 47 pgs: 47 active+clean; 7.3 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 5.2 MiB/s wr, 923 op/s 2024-03-20T09:32:59.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:32:59 smithi026 ceph-mon[68309]: pgmap v2203: 47 pgs: 47 active+clean; 7.3 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 5.2 MiB/s wr, 923 op/s 2024-03-20T09:33:00.046 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:33:00.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:00.048 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:33:00.094 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:33:00.149 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:33:00.149 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:33:00.178 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:00.191 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:33:00.206 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:33:00.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:00.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:33:00.235 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:33:00.284 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:33:00.311 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:33:00.329 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:33:00.342 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:00.364 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:33:00.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:00.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:33:00.376 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:33:00.418 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:33:00.460 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:33:00.472 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:33:00.492 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:33:00.532 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:33:00.535 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:00 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4211646627' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:33:00.562 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:00.578 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:33:00.623 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:33:00.623 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:33:00.623 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:33:00.623 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:33:00.623 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:33:00.623 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:33:00.623 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:33:00.623 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:33:00.623 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:33:00.623 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:33:00.623 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:33:00.623 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:33:00.646 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:00 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4211646627' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:33:00.879 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:33:00.924 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:00 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4211646627' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:33:01.377 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:33:01.377 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-03-20T09:33:01.377 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-03-20T09:33:01.377 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [] 2024-03-20T09:33:01.377 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:33:01.539 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:01 smithi026 ceph-mon[68309]: pgmap v2204: 47 pgs: 47 active+clean; 7.3 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 704 KiB/s wr, 837 op/s 2024-03-20T09:33:01.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:01 smithi122 ceph-mon[73495]: pgmap v2204: 47 pgs: 47 active+clean; 7.3 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 704 KiB/s wr, 837 op/s 2024-03-20T09:33:01.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:01 smithi115 ceph-mon[72476]: pgmap v2204: 47 pgs: 47 active+clean; 7.3 GiB data, 29 GiB used, 954 GiB / 983 GiB avail; 704 KiB/s wr, 837 op/s 2024-03-20T09:33:02.568 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-20T09:33:03.517 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:33:03.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:03.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:33:03.609 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:33:03.624 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:33:03.658 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:03.679 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:33:03.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:03.683 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:33:03.735 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.11 to 1 (10000) 2024-03-20T09:33:03.782 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:03 smithi026 ceph-mon[68309]: pgmap v2205: 47 pgs: 47 active+clean; 7.1 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 959 op/s 2024-03-20T09:33:03.782 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:03 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/953168365' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-03-20T09:33:03.782 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:03 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-03-20T09:33:03.784 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:33:03.812 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:03 smithi115 ceph-mon[72476]: pgmap v2205: 47 pgs: 47 active+clean; 7.1 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 959 op/s 2024-03-20T09:33:03.812 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:03 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/953168365' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-03-20T09:33:03.812 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:03 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-03-20T09:33:03.815 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:03.843 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:33:03.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:03.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:33:03.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:03 smithi122 ceph-mon[73495]: pgmap v2205: 47 pgs: 47 active+clean; 7.1 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 959 op/s 2024-03-20T09:33:03.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:03 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/953168365' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-03-20T09:33:03.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:03 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-03-20T09:33:03.949 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:33:03.978 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:04.009 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:33:04.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:04.016 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:33:04.117 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:33:04.233 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:04.602 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:33:04.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:04.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:33:04.707 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:33:04.757 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:04.780 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:33:04.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:04.786 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:33:04.887 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:33:04.940 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:04 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:33:04.941 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:04 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:33:04.941 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:04 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-03-20T09:33:04.941 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:04 smithi026 ceph-mon[68309]: osdmap e913: 12 total, 12 up, 11 in 2024-03-20T09:33:04.941 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:04 smithi026 ceph-mon[68309]: pgmap v2207: 47 pgs: 47 active+clean; 7.0 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 856 op/s 2024-03-20T09:33:04.946 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:05.097 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:33:05.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:05.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:33:05.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:04 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:33:05.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:04 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:33:05.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:04 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-03-20T09:33:05.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:04 smithi115 ceph-mon[72476]: osdmap e913: 12 total, 12 up, 11 in 2024-03-20T09:33:05.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:04 smithi115 ceph-mon[72476]: pgmap v2207: 47 pgs: 47 active+clean; 7.0 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 856 op/s 2024-03-20T09:33:05.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:04 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:33:05.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:04 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:33:05.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:04 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-03-20T09:33:05.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:04 smithi122 ceph-mon[73495]: osdmap e913: 12 total, 12 up, 11 in 2024-03-20T09:33:05.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:04 smithi122 ceph-mon[73495]: pgmap v2207: 47 pgs: 47 active+clean; 7.0 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 856 op/s 2024-03-20T09:33:05.204 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:33:05.235 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:05.258 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:33:05.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:05.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:33:05.364 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:33:05.402 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:05.430 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:33:05.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:05.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:33:05.536 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:33:05.567 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:05.585 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:33:05.588 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:33:05.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:05.591 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:33:05.611 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:33:05.656 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 3 1 2024-03-20T09:33:05.691 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:33:05.719 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:05.742 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:33:05.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:05.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:33:05.849 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:33:05.929 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:06.010 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:33:06.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:06.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:33:06.123 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:33:06.150 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:06.169 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:33:06.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:06.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:33:06.272 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:33:06.305 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:06.326 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:33:06.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:06.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:33:06.433 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:33:06.498 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:06.543 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:33:06.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:06.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:33:06.653 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:33:06.685 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:06.713 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:33:06.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:06.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:33:06.822 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:33:06.856 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:06.886 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:33:06.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:06.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:33:06.993 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:33:07.046 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:06 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:33:07.046 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:06 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:33:07.047 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:06 smithi026 ceph-mon[68309]: osdmap e914: 12 total, 12 up, 11 in 2024-03-20T09:33:07.047 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:06 smithi026 ceph-mon[68309]: pgmap v2209: 47 pgs: 47 active+clean; 6.9 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 911 op/s 2024-03-20T09:33:07.077 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:07.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:06 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:33:07.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:06 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:33:07.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:06 smithi115 ceph-mon[72476]: osdmap e914: 12 total, 12 up, 11 in 2024-03-20T09:33:07.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:06 smithi115 ceph-mon[72476]: pgmap v2209: 47 pgs: 47 active+clean; 6.9 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 911 op/s 2024-03-20T09:33:07.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:06 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:33:07.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:06 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:33:07.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:06 smithi122 ceph-mon[73495]: osdmap e914: 12 total, 12 up, 11 in 2024-03-20T09:33:07.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:06 smithi122 ceph-mon[73495]: pgmap v2209: 47 pgs: 47 active+clean; 6.9 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 911 op/s 2024-03-20T09:33:07.816 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:33:07.966 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:07 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:33:07.966 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:07 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:33:07.966 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:07 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/755429637' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:33:07.966 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:07 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:33:08.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:07 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:33:08.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:07 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:33:08.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:07 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/755429637' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:33:08.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:07 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:33:08.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:07 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:33:08.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:07 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:33:08.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:07 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/755429637' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:33:08.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:07 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:33:09.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:08 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:33:09.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:08 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:33:09.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:08 smithi115 ceph-mon[72476]: osdmap e915: 12 total, 12 up, 11 in 2024-03-20T09:33:09.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:08 smithi115 ceph-mon[72476]: pgmap v2211: 47 pgs: 47 active+clean; 6.8 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 899 op/s 2024-03-20T09:33:09.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:08 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:33:09.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:08 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:33:09.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:08 smithi122 ceph-mon[73495]: osdmap e915: 12 total, 12 up, 11 in 2024-03-20T09:33:09.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:08 smithi122 ceph-mon[73495]: pgmap v2211: 47 pgs: 47 active+clean; 6.8 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 899 op/s 2024-03-20T09:33:09.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:08 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:33:09.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:08 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:33:09.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:08 smithi026 ceph-mon[68309]: osdmap e915: 12 total, 12 up, 11 in 2024-03-20T09:33:09.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:08 smithi026 ceph-mon[68309]: pgmap v2211: 47 pgs: 47 active+clean; 6.8 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 899 op/s 2024-03-20T09:33:09.474 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:33:09.501 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:33:09.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:09.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:33:09.608 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:33:09.665 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:09.745 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:33:09.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:09.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:33:09.855 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:33:09.884 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:09 smithi122 ceph-mon[73495]: 1.15 scrub starts 2024-03-20T09:33:09.884 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:09 smithi122 ceph-mon[73495]: 1.15 scrub ok 2024-03-20T09:33:09.885 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:09.910 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:33:09.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:09.913 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:33:10.014 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:33:10.123 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:09 smithi026 ceph-mon[68309]: 1.15 scrub starts 2024-03-20T09:33:10.123 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:09 smithi026 ceph-mon[68309]: 1.15 scrub ok 2024-03-20T09:33:10.126 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:10.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:09 smithi115 ceph-mon[72476]: 1.15 scrub starts 2024-03-20T09:33:10.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:09 smithi115 ceph-mon[72476]: 1.15 scrub ok 2024-03-20T09:33:10.174 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:33:10.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:10.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:33:10.283 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:33:10.320 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:10.343 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:33:10.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:10.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:33:10.449 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:33:10.458 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:33:10.458 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:33:10.458 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:10.458 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:145897 7.6 7:6ae8a13e:::benchmark_data_smithi122_187077_object19286:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e915)", 2024-03-20T09:33:10.458 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:33:10.452523+0000", 2024-03-20T09:33:10.458 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.010557504000000001, 2024-03-20T09:33:10.458 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010599000000000001, 2024-03-20T09:33:10.459 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:33:10.459 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:33:10.459 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:33:10.459 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:33:10.459 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:33:10.459 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 145897 2024-03-20T09:33:10.459 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:10.459 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:33:10.459 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:10.459 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:33:10.459 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:10.452523+0000", 2024-03-20T09:33:10.459 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:10.459 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:10.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:10.460 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:33:10.460 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:10.452523+0000", 2024-03-20T09:33:10.460 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:10.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:10.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:10.460 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:33:10.460 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:10.452524+0000", 2024-03-20T09:33:10.460 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1289999999999999e-06 2024-03-20T09:33:10.460 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:10.460 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:10.460 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:33:10.460 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:10.452529+0000", 2024-03-20T09:33:10.461 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7940000000000002e-06 2024-03-20T09:33:10.461 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:10.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:10.461 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:33:10.461 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:10.452540+0000", 2024-03-20T09:33:10.461 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1158999999999999e-05 2024-03-20T09:33:10.461 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:10.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:10.461 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:33:10.461 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:10.452547+0000", 2024-03-20T09:33:10.461 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5660000000000001e-06 2024-03-20T09:33:10.461 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:10.461 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:10.462 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:33:10.462 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:10.452626+0000", 2024-03-20T09:33:10.462 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8487999999999998e-05 2024-03-20T09:33:10.462 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:10.462 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:10.462 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:33:10.462 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:10.452744+0000", 2024-03-20T09:33:10.462 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000118171 2024-03-20T09:33:10.462 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:10.462 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:10.462 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,7", 2024-03-20T09:33:10.462 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:10.452862+0000", 2024-03-20T09:33:10.462 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000118393 2024-03-20T09:33:10.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:10.463 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:10.463 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:33:10.463 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:10.461536+0000", 2024-03-20T09:33:10.463 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0086736729999999998 2024-03-20T09:33:10.463 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:10.463 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:10.463 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:33:10.463 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:10.461632+0000", 2024-03-20T09:33:10.463 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5488999999999993e-05 2024-03-20T09:33:10.463 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:10.463 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:33:10.463 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:10.464 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:10.464 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:10.464 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:145907 7.4 7:2ae5ee0b:::benchmark_data_smithi122_187077_object19296:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e915)", 2024-03-20T09:33:10.464 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:33:10.462473+0000", 2024-03-20T09:33:10.464 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.000607736, 2024-03-20T09:33:10.465 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00074598700000000004, 2024-03-20T09:33:10.465 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:33:10.465 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:33:10.466 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:33:10.466 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:33:10.466 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:33:10.466 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 145907 2024-03-20T09:33:10.466 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:10.466 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:33:10.466 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:10.466 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:33:10.466 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:10.462473+0000", 2024-03-20T09:33:10.467 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:10.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:10.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:10.467 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:33:10.467 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:10.462473+0000", 2024-03-20T09:33:10.467 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:10.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:10.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:10.467 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:33:10.467 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:10.462474+0000", 2024-03-20T09:33:10.467 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3769999999999999e-06 2024-03-20T09:33:10.467 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:10.467 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:10.467 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:33:10.468 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:10.462479+0000", 2024-03-20T09:33:10.468 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0200000000000002e-06 2024-03-20T09:33:10.468 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:10.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:10.468 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:33:10.468 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:10.462491+0000", 2024-03-20T09:33:10.468 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2272999999999999e-05 2024-03-20T09:33:10.468 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:10.468 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:10.468 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:33:10.468 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:10.462499+0000", 2024-03-20T09:33:10.468 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4440000000000002e-06 2024-03-20T09:33:10.468 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:10.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:10.469 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:33:10.469 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:10.462549+0000", 2024-03-20T09:33:10.469 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.075e-05 2024-03-20T09:33:10.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:10.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:10.469 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:33:10.469 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:10.462689+0000", 2024-03-20T09:33:10.469 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013967 2024-03-20T09:33:10.469 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:10.469 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:10.469 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 1,4", 2024-03-20T09:33:10.469 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:10.462843+0000", 2024-03-20T09:33:10.470 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015413699999999999 2024-03-20T09:33:10.470 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:10.470 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:33:10.470 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:10.470 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:10.470 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:10.470 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:145909 7.f 7:fe325552:::benchmark_data_smithi122_187077_object19298:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e915)", 2024-03-20T09:33:10.470 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:33:10.462523+0000", 2024-03-20T09:33:10.470 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.00055753599999999997, 2024-03-20T09:33:10.470 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00077041399999999997, 2024-03-20T09:33:10.470 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:33:10.470 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:33:10.470 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:33:10.470 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:33:10.471 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:33:10.471 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 145909 2024-03-20T09:33:10.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:10.471 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:33:10.471 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:10.471 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:33:10.471 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:10.462523+0000", 2024-03-20T09:33:10.471 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:10.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:10.471 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:10.471 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:33:10.471 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:10.462523+0000", 2024-03-20T09:33:10.471 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:10.471 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:10.471 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:10.472 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:33:10.472 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:10.462523+0000", 2024-03-20T09:33:10.472 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.82e-07 2024-03-20T09:33:10.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:10.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:10.472 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:33:10.472 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:10.462526+0000", 2024-03-20T09:33:10.472 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6639999999999998e-06 2024-03-20T09:33:10.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:10.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:10.472 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:33:10.472 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:10.462533+0000", 2024-03-20T09:33:10.472 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.199e-06 2024-03-20T09:33:10.472 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:10.472 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:10.473 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:33:10.473 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:10.462539+0000", 2024-03-20T09:33:10.473 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6269999999999997e-06 2024-03-20T09:33:10.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:10.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:10.473 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:33:10.473 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:10.462708+0000", 2024-03-20T09:33:10.473 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00016948899999999999 2024-03-20T09:33:10.473 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:10.473 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:10.473 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:33:10.474 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:10.462857+0000", 2024-03-20T09:33:10.474 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014863800000000001 2024-03-20T09:33:10.474 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:10.474 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:10.474 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 8,9", 2024-03-20T09:33:10.474 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:10.462957+0000", 2024-03-20T09:33:10.474 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9735000000000006e-05 2024-03-20T09:33:10.474 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:10.474 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:33:10.474 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:10.474 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:10.474 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:33:10.474 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 3 2024-03-20T09:33:10.474 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:33:10.491 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:10.513 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:33:10.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:10.518 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:33:10.619 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:33:10.701 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:10.738 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:33:10.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:10.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:33:10.816 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:10 smithi122 ceph-mon[73495]: 1.e scrub starts 2024-03-20T09:33:10.816 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:10 smithi122 ceph-mon[73495]: 1.e scrub ok 2024-03-20T09:33:10.816 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:10 smithi122 ceph-mon[73495]: 1.4 scrub starts 2024-03-20T09:33:10.816 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:10 smithi122 ceph-mon[73495]: 1.4 scrub ok 2024-03-20T09:33:10.816 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:10 smithi122 ceph-mon[73495]: pgmap v2212: 47 pgs: 47 active+clean; 6.7 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 834 op/s 2024-03-20T09:33:10.816 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:10 smithi122 ceph-mon[73495]: 1.a scrub starts 2024-03-20T09:33:10.816 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:10 smithi122 ceph-mon[73495]: 1.a scrub ok 2024-03-20T09:33:10.844 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:33:10.881 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:10.902 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:33:10.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:10.907 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:33:11.008 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:33:11.039 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:11.067 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:33:11.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:11.073 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:33:11.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:10 smithi115 ceph-mon[72476]: 1.e scrub starts 2024-03-20T09:33:11.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:10 smithi115 ceph-mon[72476]: 1.e scrub ok 2024-03-20T09:33:11.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:10 smithi115 ceph-mon[72476]: 1.4 scrub starts 2024-03-20T09:33:11.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:10 smithi115 ceph-mon[72476]: 1.4 scrub ok 2024-03-20T09:33:11.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:10 smithi115 ceph-mon[72476]: pgmap v2212: 47 pgs: 47 active+clean; 6.7 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 834 op/s 2024-03-20T09:33:11.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:10 smithi115 ceph-mon[72476]: 1.a scrub starts 2024-03-20T09:33:11.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:10 smithi115 ceph-mon[72476]: 1.a scrub ok 2024-03-20T09:33:11.174 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:33:11.178 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:10 smithi026 ceph-mon[68309]: 1.e scrub starts 2024-03-20T09:33:11.178 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:10 smithi026 ceph-mon[68309]: 1.e scrub ok 2024-03-20T09:33:11.178 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:10 smithi026 ceph-mon[68309]: 1.4 scrub starts 2024-03-20T09:33:11.178 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:10 smithi026 ceph-mon[68309]: 1.4 scrub ok 2024-03-20T09:33:11.178 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:10 smithi026 ceph-mon[68309]: pgmap v2212: 47 pgs: 47 active+clean; 6.7 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 834 op/s 2024-03-20T09:33:11.178 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:10 smithi026 ceph-mon[68309]: 1.a scrub starts 2024-03-20T09:33:11.178 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:10 smithi026 ceph-mon[68309]: 1.a scrub ok 2024-03-20T09:33:11.216 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:11.235 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:33:11.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:11.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:33:11.341 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:33:11.372 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:11.400 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:33:11.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:11.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:33:11.504 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:33:11.515 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:33:11.553 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:11.606 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:33:11.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:11.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:33:11.719 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:33:11.765 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:11.833 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:11 smithi026 ceph-mon[68309]: 1.13 scrub starts 2024-03-20T09:33:11.834 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:11 smithi026 ceph-mon[68309]: 1.13 scrub ok 2024-03-20T09:33:12.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:11 smithi115 ceph-mon[72476]: 1.13 scrub starts 2024-03-20T09:33:12.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:11 smithi115 ceph-mon[72476]: 1.13 scrub ok 2024-03-20T09:33:12.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:11 smithi122 ceph-mon[73495]: 1.13 scrub starts 2024-03-20T09:33:12.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:11 smithi122 ceph-mon[73495]: 1.13 scrub ok 2024-03-20T09:33:12.867 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:33:12.924 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-20T09:33:12.933 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:33:12.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:12.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:33:13.044 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:33:13.074 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:13.097 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:33:13.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:13.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:33:13.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:12 smithi115 ceph-mon[72476]: 1.d scrub starts 2024-03-20T09:33:13.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:12 smithi115 ceph-mon[72476]: 1.d scrub ok 2024-03-20T09:33:13.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:12 smithi115 ceph-mon[72476]: 1.1e scrub starts 2024-03-20T09:33:13.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:12 smithi115 ceph-mon[72476]: 1.1e scrub ok 2024-03-20T09:33:13.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:12 smithi115 ceph-mon[72476]: pgmap v2213: 47 pgs: 47 active+clean; 6.6 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s 2024-03-20T09:33:13.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:12 smithi122 ceph-mon[73495]: 1.d scrub starts 2024-03-20T09:33:13.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:12 smithi122 ceph-mon[73495]: 1.d scrub ok 2024-03-20T09:33:13.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:12 smithi122 ceph-mon[73495]: 1.1e scrub starts 2024-03-20T09:33:13.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:12 smithi122 ceph-mon[73495]: 1.1e scrub ok 2024-03-20T09:33:13.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:12 smithi122 ceph-mon[73495]: pgmap v2213: 47 pgs: 47 active+clean; 6.6 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s 2024-03-20T09:33:13.203 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:33:13.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:12 smithi026 ceph-mon[68309]: 1.d scrub starts 2024-03-20T09:33:13.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:12 smithi026 ceph-mon[68309]: 1.d scrub ok 2024-03-20T09:33:13.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:12 smithi026 ceph-mon[68309]: 1.1e scrub starts 2024-03-20T09:33:13.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:12 smithi026 ceph-mon[68309]: 1.1e scrub ok 2024-03-20T09:33:13.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:12 smithi026 ceph-mon[68309]: pgmap v2213: 47 pgs: 47 active+clean; 6.6 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s 2024-03-20T09:33:13.232 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:13.256 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:33:13.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:13.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:33:13.363 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:33:13.395 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:13.420 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:33:13.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:13.428 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:33:13.529 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:33:13.619 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:13.744 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:33:13.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:13.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:33:13.852 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:33:13.890 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:13.892 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:33:13.912 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:33:13.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:13.915 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:33:14.016 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:33:14.086 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:13 smithi026 ceph-mon[68309]: 1.6 scrub starts 2024-03-20T09:33:14.086 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:13 smithi026 ceph-mon[68309]: 1.6 scrub ok 2024-03-20T09:33:14.086 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:13 smithi026 ceph-mon[68309]: 1.7 scrub starts 2024-03-20T09:33:14.089 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:14.139 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:33:14.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:13 smithi115 ceph-mon[72476]: 1.6 scrub starts 2024-03-20T09:33:14.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:13 smithi115 ceph-mon[72476]: 1.6 scrub ok 2024-03-20T09:33:14.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:13 smithi115 ceph-mon[72476]: 1.7 scrub starts 2024-03-20T09:33:14.175 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:33:14.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:13 smithi122 ceph-mon[73495]: 1.6 scrub starts 2024-03-20T09:33:14.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:13 smithi122 ceph-mon[73495]: 1.6 scrub ok 2024-03-20T09:33:14.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:13 smithi122 ceph-mon[73495]: 1.7 scrub starts 2024-03-20T09:33:14.244 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:33:14.292 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:33:14.369 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:33:14.381 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:33:14.416 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:33:14.456 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:33:14.496 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:33:14.542 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:33:14.595 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:33:14.636 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:33:14.667 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:33:14.668 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:33:14.668 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:33:14.668 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:33:14.668 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:33:14.668 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:33:14.668 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:33:14.668 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:33:14.668 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:33:14.668 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:33:14.668 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:33:14.668 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:33:14.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:14 smithi026 ceph-mon[68309]: 1.1b scrub starts 2024-03-20T09:33:14.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:14 smithi026 ceph-mon[68309]: 1.7 scrub ok 2024-03-20T09:33:14.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:14 smithi026 ceph-mon[68309]: 1.1b scrub ok 2024-03-20T09:33:14.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:14 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2897450919' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:33:14.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:14 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:33:14.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:14 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:33:14.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:14 smithi026 ceph-mon[68309]: 1.9 scrub starts 2024-03-20T09:33:14.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:14 smithi026 ceph-mon[68309]: 1.9 scrub ok 2024-03-20T09:33:14.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:14 smithi026 ceph-mon[68309]: pgmap v2214: 47 pgs: 47 active+clean; 6.5 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 823 op/s 2024-03-20T09:33:14.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:14 smithi026 ceph-mon[68309]: 1.1 scrub starts 2024-03-20T09:33:14.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:14 smithi026 ceph-mon[68309]: 1.1 scrub ok 2024-03-20T09:33:14.971 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:33:14.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:14.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:33:15.082 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:33:15.124 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:15.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:14 smithi115 ceph-mon[72476]: 1.1b scrub starts 2024-03-20T09:33:15.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:14 smithi115 ceph-mon[72476]: 1.7 scrub ok 2024-03-20T09:33:15.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:14 smithi115 ceph-mon[72476]: 1.1b scrub ok 2024-03-20T09:33:15.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:14 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2897450919' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:33:15.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:14 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:33:15.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:14 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:33:15.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:14 smithi115 ceph-mon[72476]: 1.9 scrub starts 2024-03-20T09:33:15.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:14 smithi115 ceph-mon[72476]: 1.9 scrub ok 2024-03-20T09:33:15.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:14 smithi115 ceph-mon[72476]: pgmap v2214: 47 pgs: 47 active+clean; 6.5 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 823 op/s 2024-03-20T09:33:15.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:14 smithi115 ceph-mon[72476]: 1.1 scrub starts 2024-03-20T09:33:15.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:14 smithi115 ceph-mon[72476]: 1.1 scrub ok 2024-03-20T09:33:15.152 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:33:15.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:15.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:33:15.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:14 smithi122 ceph-mon[73495]: 1.1b scrub starts 2024-03-20T09:33:15.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:14 smithi122 ceph-mon[73495]: 1.7 scrub ok 2024-03-20T09:33:15.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:14 smithi122 ceph-mon[73495]: 1.1b scrub ok 2024-03-20T09:33:15.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:14 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2897450919' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:33:15.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:14 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:33:15.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:14 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:33:15.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:14 smithi122 ceph-mon[73495]: 1.9 scrub starts 2024-03-20T09:33:15.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:14 smithi122 ceph-mon[73495]: 1.9 scrub ok 2024-03-20T09:33:15.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:14 smithi122 ceph-mon[73495]: pgmap v2214: 47 pgs: 47 active+clean; 6.5 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 823 op/s 2024-03-20T09:33:15.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:14 smithi122 ceph-mon[73495]: 1.1 scrub starts 2024-03-20T09:33:15.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:14 smithi122 ceph-mon[73495]: 1.1 scrub ok 2024-03-20T09:33:15.258 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:33:15.343 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:15.503 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:33:15.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:15.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:33:15.622 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:33:15.650 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:15.674 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:33:15.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:15.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:33:15.782 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:33:15.816 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:15.841 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:33:15.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:15.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:33:15.948 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:33:15.980 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:16.008 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:33:16.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:16.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:33:16.115 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:33:16.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:15 smithi122 ceph-mon[73495]: 7.4 scrub starts 2024-03-20T09:33:16.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:15 smithi122 ceph-mon[73495]: osdmap e916: 12 total, 12 up, 11 in 2024-03-20T09:33:16.189 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:15 smithi026 ceph-mon[68309]: 7.4 scrub starts 2024-03-20T09:33:16.190 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:15 smithi026 ceph-mon[68309]: osdmap e916: 12 total, 12 up, 11 in 2024-03-20T09:33:16.192 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:16.316 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:33:16.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:16.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:33:16.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:15 smithi115 ceph-mon[72476]: 7.4 scrub starts 2024-03-20T09:33:16.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:15 smithi115 ceph-mon[72476]: osdmap e916: 12 total, 12 up, 11 in 2024-03-20T09:33:16.428 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:33:16.503 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:16.682 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:33:16.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:16.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:33:16.791 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:33:16.885 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:17.218 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:16 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:33:17.219 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:16 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:33:17.219 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:16 smithi026 ceph-mon[68309]: osdmap e917: 12 total, 12 up, 11 in 2024-03-20T09:33:17.219 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:16 smithi026 ceph-mon[68309]: 7.4 scrub ok 2024-03-20T09:33:17.219 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:16 smithi026 ceph-mon[68309]: pgmap v2217: 47 pgs: 47 active+clean; 6.4 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 836 op/s 2024-03-20T09:33:17.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:16 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:33:17.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:16 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:33:17.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:16 smithi115 ceph-mon[72476]: osdmap e917: 12 total, 12 up, 11 in 2024-03-20T09:33:17.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:16 smithi115 ceph-mon[72476]: 7.4 scrub ok 2024-03-20T09:33:17.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:16 smithi115 ceph-mon[72476]: pgmap v2217: 47 pgs: 47 active+clean; 6.4 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 836 op/s 2024-03-20T09:33:17.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:16 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:33:17.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:16 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:33:17.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:16 smithi122 ceph-mon[73495]: osdmap e917: 12 total, 12 up, 11 in 2024-03-20T09:33:17.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:16 smithi122 ceph-mon[73495]: 7.4 scrub ok 2024-03-20T09:33:17.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:16 smithi122 ceph-mon[73495]: pgmap v2217: 47 pgs: 47 active+clean; 6.4 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 836 op/s 2024-03-20T09:33:18.058 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:33:18.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:17 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:33:18.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:17 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:33:18.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:17 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:33:18.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:17 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:33:18.468 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:17 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:33:18.469 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:17 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:33:18.771 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:33:19.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:18 smithi026 ceph-mon[68309]: osdmap e918: 12 total, 12 up, 11 in 2024-03-20T09:33:19.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:18 smithi026 ceph-mon[68309]: pgmap v2219: 47 pgs: 47 active+clean; 6.3 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 825 op/s 2024-03-20T09:33:19.331 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:33:19.398 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:18 smithi115 ceph-mon[72476]: osdmap e918: 12 total, 12 up, 11 in 2024-03-20T09:33:19.398 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:18 smithi115 ceph-mon[72476]: pgmap v2219: 47 pgs: 47 active+clean; 6.3 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 825 op/s 2024-03-20T09:33:19.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:18 smithi122 ceph-mon[73495]: osdmap e918: 12 total, 12 up, 11 in 2024-03-20T09:33:19.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:18 smithi122 ceph-mon[73495]: pgmap v2219: 47 pgs: 47 active+clean; 6.3 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 825 op/s 2024-03-20T09:33:19.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:19.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:33:19.763 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:33:19.793 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:19.818 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:33:19.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:19.821 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:33:19.922 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:33:19.954 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:19.978 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:33:19.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:19.984 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:33:20.031 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.3 to 1 (10000) 2024-03-20T09:33:20.084 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:33:20.114 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:20.146 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:33:20.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:20.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:33:20.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:20 smithi122 ceph-mon[73495]: 1.0 scrub starts 2024-03-20T09:33:20.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:20 smithi122 ceph-mon[73495]: 1.0 scrub ok 2024-03-20T09:33:20.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:20 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2963099869' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-20T09:33:20.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:20 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-20T09:33:20.254 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:33:20.349 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:20 smithi026 ceph-mon[68309]: 1.0 scrub starts 2024-03-20T09:33:20.350 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:20 smithi026 ceph-mon[68309]: 1.0 scrub ok 2024-03-20T09:33:20.350 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:20 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2963099869' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-20T09:33:20.350 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:20 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-20T09:33:20.358 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:20.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:20 smithi115 ceph-mon[72476]: 1.0 scrub starts 2024-03-20T09:33:20.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:20 smithi115 ceph-mon[72476]: 1.0 scrub ok 2024-03-20T09:33:20.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:20 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2963099869' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-20T09:33:20.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:20 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-20T09:33:20.596 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:33:20.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:20.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:33:20.704 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:33:20.744 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:20.776 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:33:20.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:20.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:33:20.882 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:33:20.923 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:20.949 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:33:20.952 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:20.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:33:21.001 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:21 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-20T09:33:21.001 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:21 smithi115 ceph-mon[72476]: osdmap e919: 12 total, 12 up, 11 in 2024-03-20T09:33:21.001 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:21 smithi115 ceph-mon[72476]: pgmap v2221: 47 pgs: 47 active+clean; 6.2 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 869 op/s 2024-03-20T09:33:21.053 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:33:21.085 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:21.106 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:33:21.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:21.110 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:33:21.139 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 4 1 2024-03-20T09:33:21.210 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:33:21.271 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:21 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-20T09:33:21.272 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:21 smithi026 ceph-mon[68309]: osdmap e919: 12 total, 12 up, 11 in 2024-03-20T09:33:21.272 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:21 smithi026 ceph-mon[68309]: pgmap v2221: 47 pgs: 47 active+clean; 6.2 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 869 op/s 2024-03-20T09:33:21.282 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:21.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:21 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-20T09:33:21.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:21 smithi122 ceph-mon[73495]: osdmap e919: 12 total, 12 up, 11 in 2024-03-20T09:33:21.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:21 smithi122 ceph-mon[73495]: pgmap v2221: 47 pgs: 47 active+clean; 6.2 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 869 op/s 2024-03-20T09:33:22.033 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:33:22.059 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:33:22.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:22.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:33:22.106 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:33:22.169 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:33:22.202 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:22.224 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:33:22.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:22.229 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:33:22.330 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:33:22.358 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:22.383 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:33:22.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:22.387 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:33:22.488 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:33:22.522 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:22.553 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:33:22.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:22.556 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:33:22.657 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:33:22.688 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:22.712 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:33:22.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:22.716 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:33:22.816 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:33:22.890 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:23.553 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:33:23.807 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:23 smithi026 ceph-mon[68309]: pgmap v2222: 47 pgs: 47 active+clean; 6.0 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 939 op/s 2024-03-20T09:33:23.807 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:23 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/522142400' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:33:23.807 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:23 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:33:23.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:23 smithi115 ceph-mon[72476]: pgmap v2222: 47 pgs: 47 active+clean; 6.0 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 939 op/s 2024-03-20T09:33:23.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:23 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/522142400' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:33:23.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:23 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:33:23.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:23 smithi122 ceph-mon[73495]: pgmap v2222: 47 pgs: 47 active+clean; 6.0 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 939 op/s 2024-03-20T09:33:23.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:23 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/522142400' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:33:23.941 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:23 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:33:24.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:24 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:33:24.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:24 smithi026 ceph-mon[68309]: Cluster is now healthy 2024-03-20T09:33:24.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:24 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:33:24.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:24 smithi026 ceph-mon[68309]: osdmap e920: 12 total, 12 up, 11 in 2024-03-20T09:33:24.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:24 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:33:24.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:24 smithi115 ceph-mon[72476]: Cluster is now healthy 2024-03-20T09:33:24.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:24 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:33:24.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:24 smithi115 ceph-mon[72476]: osdmap e920: 12 total, 12 up, 11 in 2024-03-20T09:33:24.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:24 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:33:24.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:24 smithi122 ceph-mon[73495]: Cluster is now healthy 2024-03-20T09:33:24.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:24 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:33:24.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:24 smithi122 ceph-mon[73495]: osdmap e920: 12 total, 12 up, 11 in 2024-03-20T09:33:25.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:25 smithi115 ceph-mon[72476]: pgmap v2224: 47 pgs: 47 active+clean; 6.0 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 788 op/s 2024-03-20T09:33:25.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:25 smithi122 ceph-mon[73495]: pgmap v2224: 47 pgs: 47 active+clean; 6.0 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 788 op/s 2024-03-20T09:33:25.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:25 smithi026 ceph-mon[68309]: pgmap v2224: 47 pgs: 47 active+clean; 6.0 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 788 op/s 2024-03-20T09:33:25.987 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:33:25.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:25.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:33:26.100 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:33:26.127 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:26.146 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:33:26.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:26.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:33:26.250 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:33:26.293 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:26.317 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:33:26.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:26.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:33:26.425 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:33:26.454 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:26.479 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:33:26.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:26.486 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:33:26.587 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:33:26.614 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:26.636 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:33:26.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:26.642 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:33:26.743 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:33:26.814 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:26.880 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:33:26.880 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:33:27.105 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:33:27.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:27.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:33:27.214 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:33:27.278 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:27.565 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-20T09:33:27.768 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:33:27.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:27.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:33:27.822 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:27 smithi026 ceph-mon[68309]: pgmap v2225: 47 pgs: 47 active+clean; 5.8 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 845 op/s 2024-03-20T09:33:27.822 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:27 smithi026 ceph-mon[68309]: 1.3 scrub starts 2024-03-20T09:33:27.822 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:27 smithi026 ceph-mon[68309]: 1.3 scrub ok 2024-03-20T09:33:27.880 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:33:27.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:27 smithi115 ceph-mon[72476]: pgmap v2225: 47 pgs: 47 active+clean; 5.8 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 845 op/s 2024-03-20T09:33:27.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:27 smithi115 ceph-mon[72476]: 1.3 scrub starts 2024-03-20T09:33:27.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:27 smithi115 ceph-mon[72476]: 1.3 scrub ok 2024-03-20T09:33:27.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:27 smithi122 ceph-mon[73495]: pgmap v2225: 47 pgs: 47 active+clean; 5.8 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 845 op/s 2024-03-20T09:33:27.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:27 smithi122 ceph-mon[73495]: 1.3 scrub starts 2024-03-20T09:33:27.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:27 smithi122 ceph-mon[73495]: 1.3 scrub ok 2024-03-20T09:33:27.969 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:28.132 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:33:28.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:28.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:33:28.238 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:33:28.271 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:28.306 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:33:28.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:28.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:33:28.415 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:33:28.537 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:28.694 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:33:28.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:28.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:33:28.717 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:33:28.800 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:33:28.827 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:28.847 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:33:28.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:28.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:33:28.953 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:33:28.981 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:29.001 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:33:29.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:29.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:33:29.106 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:33:29.216 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:29.256 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:33:29.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:29.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:33:29.378 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:33:29.407 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:29.430 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:33:29.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:29.436 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:33:29.537 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:33:29.564 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:29.585 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:33:29.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:29.588 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:33:29.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:29 smithi122 ceph-mon[73495]: 7.3 scrub starts 2024-03-20T09:33:29.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:29 smithi122 ceph-mon[73495]: pgmap v2226: 47 pgs: 47 active+clean; 5.7 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 952 op/s 2024-03-20T09:33:29.689 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:33:29.717 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:29.738 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:33:29.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:29.743 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:33:29.822 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:33:29.844 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:33:29.875 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:29 smithi115 ceph-mon[72476]: 7.3 scrub starts 2024-03-20T09:33:29.875 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:29 smithi115 ceph-mon[72476]: pgmap v2226: 47 pgs: 47 active+clean; 5.7 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 952 op/s 2024-03-20T09:33:29.877 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:29.901 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:33:29.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:29.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:33:29.913 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:29 smithi026 ceph-mon[68309]: 7.3 scrub starts 2024-03-20T09:33:29.913 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:29 smithi026 ceph-mon[68309]: pgmap v2226: 47 pgs: 47 active+clean; 5.7 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 952 op/s 2024-03-20T09:33:30.005 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:33:30.035 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:30.057 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:33:30.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:30.061 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:33:30.162 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:33:30.236 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:30.582 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:30 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/433727432' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:33:30.582 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:30 smithi026 ceph-mon[68309]: 1.b scrub starts 2024-03-20T09:33:30.582 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:30 smithi026 ceph-mon[68309]: 1.b scrub ok 2024-03-20T09:33:30.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:30 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/433727432' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:33:30.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:30 smithi115 ceph-mon[72476]: 1.b scrub starts 2024-03-20T09:33:30.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:30 smithi115 ceph-mon[72476]: 1.b scrub ok 2024-03-20T09:33:30.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:30 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/433727432' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:33:30.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:30 smithi122 ceph-mon[73495]: 1.b scrub starts 2024-03-20T09:33:30.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:30 smithi122 ceph-mon[73495]: 1.b scrub ok 2024-03-20T09:33:31.540 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:33:31.540 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:33:31.541 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:33:31.541 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:33:31.541 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:33:31.541 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:33:31.541 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:33:31.541 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:33:31.541 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:33:31.541 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:33:31.541 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:33:31.541 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:33:31.541 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:33:31.542 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:33:31.542 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:33:31.542 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:33:31.542 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:33:31.542 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:33:31.542 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:33:31.542 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:33:31.542 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:33:31.542 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:33:31.542 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:33:31.542 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:33:31.850 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:33:31.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:31.857 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:31 smithi026 ceph-mon[68309]: 7.3 scrub ok 2024-03-20T09:33:31.857 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:31 smithi026 ceph-mon[68309]: pgmap v2227: 47 pgs: 47 active+clean; 5.7 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 846 op/s 2024-03-20T09:33:31.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:33:31.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:31 smithi115 ceph-mon[72476]: 7.3 scrub ok 2024-03-20T09:33:31.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:31 smithi115 ceph-mon[72476]: pgmap v2227: 47 pgs: 47 active+clean; 5.7 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 846 op/s 2024-03-20T09:33:31.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:31 smithi122 ceph-mon[73495]: 7.3 scrub ok 2024-03-20T09:33:31.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:31 smithi122 ceph-mon[73495]: pgmap v2227: 47 pgs: 47 active+clean; 5.7 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 846 op/s 2024-03-20T09:33:31.958 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:33:32.059 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:32.632 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.4 to 1 (10000) 2024-03-20T09:33:32.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:32 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2416166475' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-20T09:33:32.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:32 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2416166475' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-20T09:33:32.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:32 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2416166475' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-20T09:33:33.202 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:33:33.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:33.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:33:33.314 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:33:33.376 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:33.722 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 9 1 2024-03-20T09:33:33.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:33 smithi115 ceph-mon[72476]: pgmap v2228: 47 pgs: 47 active+clean; 5.5 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 829 op/s 2024-03-20T09:33:33.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:33 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2416166475' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-20T09:33:33.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:33 smithi115 ceph-mon[72476]: osdmap e921: 12 total, 12 up, 11 in 2024-03-20T09:33:33.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:33 smithi115 ceph-mon[72476]: 1.f scrub starts 2024-03-20T09:33:33.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:33 smithi122 ceph-mon[73495]: pgmap v2228: 47 pgs: 47 active+clean; 5.5 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 829 op/s 2024-03-20T09:33:33.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:33 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2416166475' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-20T09:33:33.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:33 smithi122 ceph-mon[73495]: osdmap e921: 12 total, 12 up, 11 in 2024-03-20T09:33:33.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:33 smithi122 ceph-mon[73495]: 1.f scrub starts 2024-03-20T09:33:33.967 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:33 smithi026 ceph-mon[68309]: pgmap v2228: 47 pgs: 47 active+clean; 5.5 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 829 op/s 2024-03-20T09:33:33.967 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:33 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2416166475' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-20T09:33:33.967 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:33 smithi026 ceph-mon[68309]: osdmap e921: 12 total, 12 up, 11 in 2024-03-20T09:33:33.968 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:33 smithi026 ceph-mon[68309]: 1.f scrub starts 2024-03-20T09:33:34.760 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:33:34.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:34.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:33:34.866 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:33:34.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:34 smithi115 ceph-mon[72476]: 1.c scrub starts 2024-03-20T09:33:34.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:34 smithi115 ceph-mon[72476]: 1.c scrub ok 2024-03-20T09:33:34.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:34 smithi115 ceph-mon[72476]: 1.2 scrub starts 2024-03-20T09:33:34.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:34 smithi115 ceph-mon[72476]: 1.2 scrub ok 2024-03-20T09:33:34.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:34 smithi115 ceph-mon[72476]: 1.f scrub ok 2024-03-20T09:33:34.902 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:34.910 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:33:34.925 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:33:34.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:34.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:33:34.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:34 smithi122 ceph-mon[73495]: 1.c scrub starts 2024-03-20T09:33:34.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:34 smithi122 ceph-mon[73495]: 1.c scrub ok 2024-03-20T09:33:34.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:34 smithi122 ceph-mon[73495]: 1.2 scrub starts 2024-03-20T09:33:34.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:34 smithi122 ceph-mon[73495]: 1.2 scrub ok 2024-03-20T09:33:34.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:34 smithi122 ceph-mon[73495]: 1.f scrub ok 2024-03-20T09:33:34.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:34 smithi026 ceph-mon[68309]: 1.c scrub starts 2024-03-20T09:33:34.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:34 smithi026 ceph-mon[68309]: 1.c scrub ok 2024-03-20T09:33:34.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:34 smithi026 ceph-mon[68309]: 1.2 scrub starts 2024-03-20T09:33:34.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:34 smithi026 ceph-mon[68309]: 1.2 scrub ok 2024-03-20T09:33:34.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:34 smithi026 ceph-mon[68309]: 1.f scrub ok 2024-03-20T09:33:34.965 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:33:35.029 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:33:35.065 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:35.094 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:33:35.100 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:33:35.100 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-03-20T09:33:35.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-03-20T09:33:35.101 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:33:35.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.101 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:32463 7.6 7:69f982c3:::benchmark_data_smithi122_187077_object32462:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e879)", 2024-03-20T09:33:35.101 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:31:30.644282+0000", 2024-03-20T09:33:35.101 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 124.46079197500001, 2024-03-20T09:33:35.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.14261175800000001, 2024-03-20T09:33:35.101 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:33:35.101 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:33:35.101 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:33:35.101 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:33:35.101 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:33:35.101 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32463 2024-03-20T09:33:35.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.102 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:33:35.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:33:35.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.644282+0000", 2024-03-20T09:33:35.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:33:35.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.644282+0000", 2024-03-20T09:33:35.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:33:35.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.644283+0000", 2024-03-20T09:33:35.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.204e-06 2024-03-20T09:33:35.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:33:35.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.644329+0000", 2024-03-20T09:33:35.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5501999999999999e-05 2024-03-20T09:33:35.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:33:35.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.644339+0000", 2024-03-20T09:33:35.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7869999999999996e-06 2024-03-20T09:33:35.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.103 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:33:35.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.644348+0000", 2024-03-20T09:33:35.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4439999999999997e-06 2024-03-20T09:33:35.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:33:35.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.644404+0000", 2024-03-20T09:33:35.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6406999999999997e-05 2024-03-20T09:33:35.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:33:35.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.644538+0000", 2024-03-20T09:33:35.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013340199999999999 2024-03-20T09:33:35.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,7", 2024-03-20T09:33:35.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.644667+0000", 2024-03-20T09:33:35.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000128882 2024-03-20T09:33:35.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:33:35.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.646171+0000", 2024-03-20T09:33:35.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015041830000000001 2024-03-20T09:33:35.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:33:35.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.647207+0000", 2024-03-20T09:33:35.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010365769999999999 2024-03-20T09:33:35.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:33:35.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.786756+0000", 2024-03-20T09:33:35.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.139548635 2024-03-20T09:33:35.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:33:35.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.786833+0000", 2024-03-20T09:33:35.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7367000000000007e-05 2024-03-20T09:33:35.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:33:35.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.786894+0000", 2024-03-20T09:33:35.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0368000000000002e-05 2024-03-20T09:33:35.106 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.107 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:33:35.107 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.107 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:32479 7.6 7:6215124e:::benchmark_data_smithi122_187077_object32478:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e879)", 2024-03-20T09:33:35.107 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:31:30.649128+0000", 2024-03-20T09:33:35.107 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 124.455945602, 2024-03-20T09:33:35.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.13804851300000001, 2024-03-20T09:33:35.107 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:33:35.107 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:33:35.107 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:33:35.107 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:33:35.107 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:33:35.108 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32479 2024-03-20T09:33:35.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.108 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:33:35.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:33:35.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.649128+0000", 2024-03-20T09:33:35.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:33:35.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.649128+0000", 2024-03-20T09:33:35.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:33:35.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.649129+0000", 2024-03-20T09:33:35.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9599999999999998e-07 2024-03-20T09:33:35.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:33:35.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.649175+0000", 2024-03-20T09:33:35.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5909999999999999e-05 2024-03-20T09:33:35.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:33:35.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.649185+0000", 2024-03-20T09:33:35.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4660000000000005e-06 2024-03-20T09:33:35.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:33:35.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.649192+0000", 2024-03-20T09:33:35.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8469999999999996e-06 2024-03-20T09:33:35.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:33:35.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.649248+0000", 2024-03-20T09:33:35.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5044999999999999e-05 2024-03-20T09:33:35.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:33:35.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.649410+0000", 2024-03-20T09:33:35.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000162743 2024-03-20T09:33:35.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,7", 2024-03-20T09:33:35.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.649571+0000", 2024-03-20T09:33:35.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00016048399999999999 2024-03-20T09:33:35.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:33:35.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.650915+0000", 2024-03-20T09:33:35.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0013441239999999999 2024-03-20T09:33:35.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:33:35.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.650939+0000", 2024-03-20T09:33:35.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4000999999999999e-05 2024-03-20T09:33:35.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:33:35.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.787078+0000", 2024-03-20T09:33:35.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.13613929799999999 2024-03-20T09:33:35.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:33:35.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.787136+0000", 2024-03-20T09:33:35.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7880999999999997e-05 2024-03-20T09:33:35.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:33:35.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.787177+0000", 2024-03-20T09:33:35.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0917999999999997e-05 2024-03-20T09:33:35.113 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.113 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:33:35.113 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.113 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:32489 7.6 7:6336278f:::benchmark_data_smithi122_187077_object32488:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e879)", 2024-03-20T09:33:35.113 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:31:30.653150+0000", 2024-03-20T09:33:35.113 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 124.451923573, 2024-03-20T09:33:35.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.13692343600000001, 2024-03-20T09:33:35.113 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:33:35.113 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:33:35.113 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:33:35.113 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:33:35.113 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:33:35.114 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 32489 2024-03-20T09:33:35.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.114 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:33:35.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:33:35.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.653150+0000", 2024-03-20T09:33:35.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:33:35.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.653150+0000", 2024-03-20T09:33:35.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:33:35.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.653151+0000", 2024-03-20T09:33:35.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6899999999999996e-07 2024-03-20T09:33:35.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:33:35.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.653200+0000", 2024-03-20T09:33:35.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8693000000000003e-05 2024-03-20T09:33:35.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:33:35.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.653212+0000", 2024-03-20T09:33:35.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1994000000000001e-05 2024-03-20T09:33:35.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:33:35.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.653220+0000", 2024-03-20T09:33:35.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1219999999999995e-06 2024-03-20T09:33:35.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:33:35.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.653279+0000", 2024-03-20T09:33:35.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8594000000000002e-05 2024-03-20T09:33:35.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:33:35.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.653402+0000", 2024-03-20T09:33:35.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012360800000000001 2024-03-20T09:33:35.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,7", 2024-03-20T09:33:35.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.653504+0000", 2024-03-20T09:33:35.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101415 2024-03-20T09:33:35.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:33:35.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.654652+0000", 2024-03-20T09:33:35.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001148268 2024-03-20T09:33:35.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:33:35.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.654921+0000", 2024-03-20T09:33:35.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00026873799999999997 2024-03-20T09:33:35.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:33:35.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.789967+0000", 2024-03-20T09:33:35.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.135046269 2024-03-20T09:33:35.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:33:35.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.790028+0000", 2024-03-20T09:33:35.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0747e-05 2024-03-20T09:33:35.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:33:35.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:30.790074+0000", 2024-03-20T09:33:35.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6218999999999999e-05 2024-03-20T09:33:35.119 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.119 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:33:35.119 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.119 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:92989 7.9 7:9acea8b7:::benchmark_data_smithi122_187077_object92988:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e896)", 2024-03-20T09:33:35.119 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:32:13.188982+0000", 2024-03-20T09:33:35.119 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 81.916091940000001, 2024-03-20T09:33:35.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.16973955399999999, 2024-03-20T09:33:35.119 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:33:35.119 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:33:35.119 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:33:35.120 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:33:35.120 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:33:35.120 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92989 2024-03-20T09:33:35.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.120 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:33:35.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:33:35.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.188982+0000", 2024-03-20T09:33:35.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:33:35.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.188982+0000", 2024-03-20T09:33:35.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:33:35.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.188984+0000", 2024-03-20T09:33:35.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6959999999999999e-06 2024-03-20T09:33:35.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:33:35.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.189016+0000", 2024-03-20T09:33:35.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1942e-05 2024-03-20T09:33:35.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:33:35.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.189029+0000", 2024-03-20T09:33:35.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2972e-05 2024-03-20T09:33:35.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:33:35.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.189037+0000", 2024-03-20T09:33:35.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4649999999999994e-06 2024-03-20T09:33:35.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:33:35.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.192323+0000", 2024-03-20T09:33:35.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0032854220000000001 2024-03-20T09:33:35.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:33:35.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.192478+0000", 2024-03-20T09:33:35.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015525200000000001 2024-03-20T09:33:35.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 8,11", 2024-03-20T09:33:35.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.192614+0000", 2024-03-20T09:33:35.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013578200000000001 2024-03-20T09:33:35.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:33:35.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.198831+0000", 2024-03-20T09:33:35.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0062176169999999999 2024-03-20T09:33:35.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:33:35.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.198871+0000", 2024-03-20T09:33:35.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9858999999999999e-05 2024-03-20T09:33:35.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:33:35.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.202280+0000", 2024-03-20T09:33:35.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003408496 2024-03-20T09:33:35.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:33:35.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.358665+0000", 2024-03-20T09:33:35.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.15638491700000001 2024-03-20T09:33:35.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:33:35.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.358722+0000", 2024-03-20T09:33:35.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7133999999999999e-05 2024-03-20T09:33:35.125 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.125 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:33:35.125 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.125 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:92996 7.9 7:97522f60:::benchmark_data_smithi122_187077_object92995:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e896)", 2024-03-20T09:33:35.125 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:32:13.193939+0000", 2024-03-20T09:33:35.125 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 81.911135466000005, 2024-03-20T09:33:35.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.16490886399999999, 2024-03-20T09:33:35.125 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:33:35.125 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:33:35.125 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:33:35.126 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:33:35.126 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:33:35.126 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92996 2024-03-20T09:33:35.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.126 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:33:35.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:33:35.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.193939+0000", 2024-03-20T09:33:35.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:33:35.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.193939+0000", 2024-03-20T09:33:35.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:33:35.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.193939+0000", 2024-03-20T09:33:35.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9999999999999996e-07 2024-03-20T09:33:35.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:33:35.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.193969+0000", 2024-03-20T09:33:35.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9620000000000001e-05 2024-03-20T09:33:35.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:33:35.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.193981+0000", 2024-03-20T09:33:35.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1527e-05 2024-03-20T09:33:35.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:33:35.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.193988+0000", 2024-03-20T09:33:35.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5329999999999997e-06 2024-03-20T09:33:35.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:33:35.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.195940+0000", 2024-03-20T09:33:35.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001951501 2024-03-20T09:33:35.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:33:35.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.196087+0000", 2024-03-20T09:33:35.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014773799999999999 2024-03-20T09:33:35.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 8,11", 2024-03-20T09:33:35.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.196221+0000", 2024-03-20T09:33:35.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133115 2024-03-20T09:33:35.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:33:35.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.199749+0000", 2024-03-20T09:33:35.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035285680000000002 2024-03-20T09:33:35.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:33:35.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.199989+0000", 2024-03-20T09:33:35.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00023962699999999999 2024-03-20T09:33:35.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:33:35.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.358731+0000", 2024-03-20T09:33:35.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.158742085 2024-03-20T09:33:35.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:33:35.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.358773+0000", 2024-03-20T09:33:35.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2366000000000001e-05 2024-03-20T09:33:35.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:33:35.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.358847+0000", 2024-03-20T09:33:35.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4283999999999996e-05 2024-03-20T09:33:35.131 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.131 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:33:35.131 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.131 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:92998 7.4 7:2d9b9ef8:::benchmark_data_smithi122_187077_object92997:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e896)", 2024-03-20T09:33:35.131 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:32:13.196209+0000", 2024-03-20T09:33:35.131 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 81.908865258000006, 2024-03-20T09:33:35.132 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.162681508, 2024-03-20T09:33:35.132 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:33:35.132 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:33:35.132 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:33:35.132 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:33:35.132 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:33:35.132 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92998 2024-03-20T09:33:35.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.132 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:33:35.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.132 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:33:35.132 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.196209+0000", 2024-03-20T09:33:35.132 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.133 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:33:35.133 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.196209+0000", 2024-03-20T09:33:35.133 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.133 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:33:35.133 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.196210+0000", 2024-03-20T09:33:35.133 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-03-20T09:33:35.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.133 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:33:35.133 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.196346+0000", 2024-03-20T09:33:35.133 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013629700000000001 2024-03-20T09:33:35.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.134 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:33:35.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.196357+0000", 2024-03-20T09:33:35.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0516999999999999e-05 2024-03-20T09:33:35.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.134 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:33:35.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.196364+0000", 2024-03-20T09:33:35.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3640000000000001e-06 2024-03-20T09:33:35.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:33:35.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.196419+0000", 2024-03-20T09:33:35.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5500000000000001e-05 2024-03-20T09:33:35.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:33:35.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.196569+0000", 2024-03-20T09:33:35.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000149901 2024-03-20T09:33:35.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 1,4", 2024-03-20T09:33:35.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.196703+0000", 2024-03-20T09:33:35.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013317299999999999 2024-03-20T09:33:35.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:33:35.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.358636+0000", 2024-03-20T09:33:35.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.16193320899999999 2024-03-20T09:33:35.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:33:35.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.358757+0000", 2024-03-20T09:33:35.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012163399999999999 2024-03-20T09:33:35.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.137 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:33:35.137 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.358808+0000", 2024-03-20T09:33:35.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0284999999999997e-05 2024-03-20T09:33:35.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.137 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:33:35.137 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.358861+0000", 2024-03-20T09:33:35.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3244e-05 2024-03-20T09:33:35.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.137 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:33:35.137 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.358890+0000", 2024-03-20T09:33:35.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9386999999999998e-05 2024-03-20T09:33:35.138 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.138 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:33:35.138 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.138 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop_reply(client.26605.0:92998 7.4 e896/871 ondisk, result = 0)", 2024-03-20T09:33:35.138 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:32:13.203114+0000", 2024-03-20T09:33:35.138 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 81.901959914000003, 2024-03-20T09:33:35.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.15578231100000001, 2024-03-20T09:33:35.138 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:33:35.138 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:33:35.138 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:33:35.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:33:35.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.203114+0000", 2024-03-20T09:33:35.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:33:35.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.203114+0000", 2024-03-20T09:33:35.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:33:35.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.203115+0000", 2024-03-20T09:33:35.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4900000000000005e-07 2024-03-20T09:33:35.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:33:35.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.203123+0000", 2024-03-20T09:33:35.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.746e-06 2024-03-20T09:33:35.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:33:35.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.358624+0000", 2024-03-20T09:33:35.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.15550082200000001 2024-03-20T09:33:35.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:33:35.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.358634+0000", 2024-03-20T09:33:35.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0006999999999999e-05 2024-03-20T09:33:35.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:33:35.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.358788+0000", 2024-03-20T09:33:35.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015420000000000001 2024-03-20T09:33:35.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:33:35.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.358802+0000", 2024-03-20T09:33:35.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4684e-05 2024-03-20T09:33:35.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:33:35.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.358896+0000", 2024-03-20T09:33:35.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4003000000000001e-05 2024-03-20T09:33:35.142 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.142 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:33:35.142 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.142 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop_reply(client.26605.0:92998 7.4 e896/871 ondisk, result = 0)", 2024-03-20T09:33:35.142 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:32:13.205041+0000", 2024-03-20T09:33:35.142 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 81.900033221000001, 2024-03-20T09:33:35.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.15372565099999999, 2024-03-20T09:33:35.142 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:33:35.142 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:33:35.142 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:33:35.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:33:35.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.205041+0000", 2024-03-20T09:33:35.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:33:35.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.205041+0000", 2024-03-20T09:33:35.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:33:35.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.205042+0000", 2024-03-20T09:33:35.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6499999999999998e-07 2024-03-20T09:33:35.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.144 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:33:35.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.205048+0000", 2024-03-20T09:33:35.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.066e-06 2024-03-20T09:33:35.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.144 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:33:35.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.358598+0000", 2024-03-20T09:33:35.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.15355015999999999 2024-03-20T09:33:35.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.144 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:33:35.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.358620+0000", 2024-03-20T09:33:35.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2042999999999999e-05 2024-03-20T09:33:35.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.145 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:33:35.145 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.358741+0000", 2024-03-20T09:33:35.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0001209 2024-03-20T09:33:35.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.145 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:33:35.145 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.358750+0000", 2024-03-20T09:33:35.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.217e-06 2024-03-20T09:33:35.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.145 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:33:35.145 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.358766+0000", 2024-03-20T09:33:35.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6399999999999999e-05 2024-03-20T09:33:35.145 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.146 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:33:35.146 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.146 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:93010 7.f 7:fd136a01:::benchmark_data_smithi122_187077_object93009:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e896)", 2024-03-20T09:33:35.146 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:32:13.205498+0000", 2024-03-20T09:33:35.146 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 81.899575863999999, 2024-03-20T09:33:35.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.17318462600000001, 2024-03-20T09:33:35.146 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:33:35.146 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:33:35.146 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:33:35.146 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:33:35.146 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:33:35.146 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 93010 2024-03-20T09:33:35.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.147 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:33:35.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:33:35.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.205498+0000", 2024-03-20T09:33:35.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:33:35.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.205498+0000", 2024-03-20T09:33:35.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:33:35.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.205500+0000", 2024-03-20T09:33:35.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.313e-06 2024-03-20T09:33:35.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:33:35.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.205564+0000", 2024-03-20T09:33:35.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4016999999999994e-05 2024-03-20T09:33:35.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:33:35.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.358604+0000", 2024-03-20T09:33:35.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.15304008599999999 2024-03-20T09:33:35.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:33:35.149 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.358620+0000", 2024-03-20T09:33:35.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6804999999999999e-05 2024-03-20T09:33:35.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:33:35.149 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.358684+0000", 2024-03-20T09:33:35.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3546000000000004e-05 2024-03-20T09:33:35.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:33:35.149 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.358877+0000", 2024-03-20T09:33:35.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019259499999999999 2024-03-20T09:33:35.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 8,9", 2024-03-20T09:33:35.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.358995+0000", 2024-03-20T09:33:35.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011817800000000001 2024-03-20T09:33:35.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:33:35.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.368423+0000", 2024-03-20T09:33:35.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0094282889999999994 2024-03-20T09:33:35.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:33:35.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.368473+0000", 2024-03-20T09:33:35.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0108999999999997e-05 2024-03-20T09:33:35.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.151 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:33:35.151 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.378605+0000", 2024-03-20T09:33:35.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010131611 2024-03-20T09:33:35.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.151 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:33:35.151 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.378637+0000", 2024-03-20T09:33:35.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1971000000000003e-05 2024-03-20T09:33:35.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:33:35.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:13.378683+0000", 2024-03-20T09:33:35.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6106000000000002e-05 2024-03-20T09:33:35.152 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.152 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:33:35.152 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.152 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26605.0:121134 7.5 e907/871 7:a5a43b5f:::benchmark_data_smithi122_187077_object121133:head v 907'7538, mlcod=907'7536)", 2024-03-20T09:33:35.152 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:32:42.748620+0000", 2024-03-20T09:33:35.152 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 52.356454352999997, 2024-03-20T09:33:35.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.12053833, 2024-03-20T09:33:35.152 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:33:35.153 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:33:35.153 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:33:35.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:33:35.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.748620+0000", 2024-03-20T09:33:35.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:33:35.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.748620+0000", 2024-03-20T09:33:35.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:33:35.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.748620+0000", 2024-03-20T09:33:35.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5400000000000001e-07 2024-03-20T09:33:35.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:33:35.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.748670+0000", 2024-03-20T09:33:35.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9614999999999998e-05 2024-03-20T09:33:35.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:33:35.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.748679+0000", 2024-03-20T09:33:35.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9509999999999995e-06 2024-03-20T09:33:35.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:33:35.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.748685+0000", 2024-03-20T09:33:35.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9030000000000001e-06 2024-03-20T09:33:35.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:33:35.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.749179+0000", 2024-03-20T09:33:35.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00049463000000000003 2024-03-20T09:33:35.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:33:35.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.749200+0000", 2024-03-20T09:33:35.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0968999999999999e-05 2024-03-20T09:33:35.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.156 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:33:35.156 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.762813+0000", 2024-03-20T09:33:35.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013613017999999999 2024-03-20T09:33:35.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.156 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:33:35.156 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.869158+0000", 2024-03-20T09:33:35.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.10634469000000001 2024-03-20T09:33:35.156 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.156 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:33:35.156 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.157 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26605.0:121135 7.5 e907/871 7:ab4b6f7b:::benchmark_data_smithi122_187077_object121134:head v 907'7539, mlcod=907'7536)", 2024-03-20T09:33:35.157 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:32:42.752195+0000", 2024-03-20T09:33:35.157 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 52.352878689000001, 2024-03-20T09:33:35.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.11700550999999999, 2024-03-20T09:33:35.157 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:33:35.157 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:33:35.157 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:33:35.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.157 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:33:35.157 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.752195+0000", 2024-03-20T09:33:35.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:33:35.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.752195+0000", 2024-03-20T09:33:35.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:33:35.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.752196+0000", 2024-03-20T09:33:35.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2200000000000003e-07 2024-03-20T09:33:35.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:33:35.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.752266+0000", 2024-03-20T09:33:35.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9416000000000001e-05 2024-03-20T09:33:35.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.159 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:33:35.159 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.752276+0000", 2024-03-20T09:33:35.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.022e-05 2024-03-20T09:33:35.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.159 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:33:35.159 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.752282+0000", 2024-03-20T09:33:35.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3239999999999999e-06 2024-03-20T09:33:35.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.159 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:33:35.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.752419+0000", 2024-03-20T09:33:35.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013675699999999999 2024-03-20T09:33:35.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:33:35.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.752438+0000", 2024-03-20T09:33:35.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9341e-05 2024-03-20T09:33:35.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:33:35.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.869169+0000", 2024-03-20T09:33:35.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.116731133 2024-03-20T09:33:35.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:33:35.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.869201+0000", 2024-03-20T09:33:35.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1597000000000003e-05 2024-03-20T09:33:35.161 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.161 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:33:35.161 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.161 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26605.0:121137 7.d e907/871 7:b127da31:::benchmark_data_smithi122_187077_object121136:head v 907'7554, mlcod=907'7552)", 2024-03-20T09:33:35.161 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:32:42.752860+0000", 2024-03-20T09:33:35.161 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 52.352213890999998, 2024-03-20T09:33:35.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.116381081, 2024-03-20T09:33:35.161 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:33:35.161 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:33:35.162 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:33:35.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.162 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:33:35.162 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.752860+0000", 2024-03-20T09:33:35.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.162 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:33:35.162 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.752860+0000", 2024-03-20T09:33:35.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.162 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:33:35.162 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.752861+0000", 2024-03-20T09:33:35.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8400000000000004e-07 2024-03-20T09:33:35.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:33:35.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.752896+0000", 2024-03-20T09:33:35.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5706e-05 2024-03-20T09:33:35.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:33:35.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.752906+0000", 2024-03-20T09:33:35.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3430000000000002e-06 2024-03-20T09:33:35.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:33:35.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.752912+0000", 2024-03-20T09:33:35.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2480000000000001e-06 2024-03-20T09:33:35.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:33:35.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.756012+0000", 2024-03-20T09:33:35.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031000519999999998 2024-03-20T09:33:35.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:33:35.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.756025+0000", 2024-03-20T09:33:35.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2841e-05 2024-03-20T09:33:35.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:33:35.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.869206+0000", 2024-03-20T09:33:35.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.1131809 2024-03-20T09:33:35.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:33:35.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.869241+0000", 2024-03-20T09:33:35.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5407000000000002e-05 2024-03-20T09:33:35.165 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.165 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:33:35.165 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.165 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:121141 7.4 7:2070463e:::benchmark_data_smithi122_187077_object121140:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e907)", 2024-03-20T09:33:35.165 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:32:42.757121+0000", 2024-03-20T09:33:35.166 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 52.347953392000001, 2024-03-20T09:33:35.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.11222684300000001, 2024-03-20T09:33:35.166 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:33:35.166 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:33:35.166 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:33:35.166 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:33:35.166 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:33:35.166 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 121141 2024-03-20T09:33:35.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.166 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:33:35.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:33:35.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.757121+0000", 2024-03-20T09:33:35.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:33:35.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.757121+0000", 2024-03-20T09:33:35.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:33:35.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.757122+0000", 2024-03-20T09:33:35.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1790000000000001e-06 2024-03-20T09:33:35.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:33:35.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.757166+0000", 2024-03-20T09:33:35.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4580999999999999e-05 2024-03-20T09:33:35.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:33:35.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.757180+0000", 2024-03-20T09:33:35.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3808000000000001e-05 2024-03-20T09:33:35.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:33:35.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.757187+0000", 2024-03-20T09:33:35.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5899999999999996e-06 2024-03-20T09:33:35.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:33:35.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.757236+0000", 2024-03-20T09:33:35.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9347e-05 2024-03-20T09:33:35.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:33:35.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.757416+0000", 2024-03-20T09:33:35.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018000199999999999 2024-03-20T09:33:35.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 1,4", 2024-03-20T09:33:35.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.757577+0000", 2024-03-20T09:33:35.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00016121700000000001 2024-03-20T09:33:35.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:33:35.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.869199+0000", 2024-03-20T09:33:35.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.11162151200000001 2024-03-20T09:33:35.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.172 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:33:35.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.869241+0000", 2024-03-20T09:33:35.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1705999999999997e-05 2024-03-20T09:33:35.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.172 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:33:35.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.869249+0000", 2024-03-20T09:33:35.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5890000000000003e-06 2024-03-20T09:33:35.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.172 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:33:35.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.869289+0000", 2024-03-20T09:33:35.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9852999999999999e-05 2024-03-20T09:33:35.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:33:35.173 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.869348+0000", 2024-03-20T09:33:35.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8458999999999997e-05 2024-03-20T09:33:35.173 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.173 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:33:35.173 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.173 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:121143 7.9 7:96a21d73:::benchmark_data_smithi122_187077_object121142:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e907)", 2024-03-20T09:33:35.173 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:32:42.757215+0000", 2024-03-20T09:33:35.173 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 52.347858561999999, 2024-03-20T09:33:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.116333226, 2024-03-20T09:33:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:33:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:33:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:33:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:33:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:33:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 121143 2024-03-20T09:33:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:33:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:33:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.757215+0000", 2024-03-20T09:33:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:33:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.757215+0000", 2024-03-20T09:33:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:33:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.757216+0000", 2024-03-20T09:33:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4899999999999995e-07 2024-03-20T09:33:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:33:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.757335+0000", 2024-03-20T09:33:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011885800000000001 2024-03-20T09:33:35.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:33:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.757344+0000", 2024-03-20T09:33:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9279999999999999e-06 2024-03-20T09:33:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:33:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.757351+0000", 2024-03-20T09:33:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9129999999999997e-06 2024-03-20T09:33:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:33:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.757406+0000", 2024-03-20T09:33:35.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4959e-05 2024-03-20T09:33:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:33:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.757527+0000", 2024-03-20T09:33:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000121533 2024-03-20T09:33:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 8,11", 2024-03-20T09:33:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.757717+0000", 2024-03-20T09:33:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018968199999999999 2024-03-20T09:33:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:33:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.869120+0000", 2024-03-20T09:33:35.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.111403215 2024-03-20T09:33:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:33:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.869171+0000", 2024-03-20T09:33:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1177000000000001e-05 2024-03-20T09:33:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:33:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.873416+0000", 2024-03-20T09:33:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042450760000000004 2024-03-20T09:33:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:33:35.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.873496+0000", 2024-03-20T09:33:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9087999999999999e-05 2024-03-20T09:33:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:33:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.873549+0000", 2024-03-20T09:33:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3148e-05 2024-03-20T09:33:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:33:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:121151 7.6 7:64a8dd54:::benchmark_data_smithi122_187077_object121150:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e907)", 2024-03-20T09:33:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:32:42.760863+0000", 2024-03-20T09:33:35.179 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 52.344210857999997, 2024-03-20T09:33:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.14362404000000001, 2024-03-20T09:33:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:33:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:33:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:33:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:33:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:33:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 121151 2024-03-20T09:33:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:33:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:33:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.760863+0000", 2024-03-20T09:33:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:33:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.760863+0000", 2024-03-20T09:33:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:33:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.760865+0000", 2024-03-20T09:33:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.403e-06 2024-03-20T09:33:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:33:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.760906+0000", 2024-03-20T09:33:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1709999999999999e-05 2024-03-20T09:33:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:33:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.760917+0000", 2024-03-20T09:33:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1112e-05 2024-03-20T09:33:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:33:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.760926+0000", 2024-03-20T09:33:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9649999999999997e-06 2024-03-20T09:33:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:33:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.761227+0000", 2024-03-20T09:33:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00030022099999999998 2024-03-20T09:33:35.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:33:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.761320+0000", 2024-03-20T09:33:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3045000000000001e-05 2024-03-20T09:33:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,7", 2024-03-20T09:33:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.869231+0000", 2024-03-20T09:33:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.10791105099999999 2024-03-20T09:33:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:33:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.877558+0000", 2024-03-20T09:33:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0083276050000000001 2024-03-20T09:33:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:33:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.886668+0000", 2024-03-20T09:33:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0091094430000000001 2024-03-20T09:33:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:33:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.904366+0000", 2024-03-20T09:33:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017698728 2024-03-20T09:33:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:33:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.904425+0000", 2024-03-20T09:33:35.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8792999999999998e-05 2024-03-20T09:33:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:33:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.904487+0000", 2024-03-20T09:33:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1964000000000005e-05 2024-03-20T09:33:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:33:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26605.0:121146 7.5 e907/871 7:a5bc028d:::benchmark_data_smithi122_187077_object121145:head v 907'7540, mlcod=907'7536)", 2024-03-20T09:33:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:32:42.760959+0000", 2024-03-20T09:33:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 52.344115258999999, 2024-03-20T09:33:35.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.119812267, 2024-03-20T09:33:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:33:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:33:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:33:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:33:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.760959+0000", 2024-03-20T09:33:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:33:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.760959+0000", 2024-03-20T09:33:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:33:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.760959+0000", 2024-03-20T09:33:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7100000000000002e-07 2024-03-20T09:33:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:33:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.760999+0000", 2024-03-20T09:33:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9242000000000001e-05 2024-03-20T09:33:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:33:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.761006+0000", 2024-03-20T09:33:35.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7069999999999992e-06 2024-03-20T09:33:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:33:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.761014+0000", 2024-03-20T09:33:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.481e-06 2024-03-20T09:33:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:33:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.869231+0000", 2024-03-20T09:33:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.108216802 2024-03-20T09:33:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:33:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.869254+0000", 2024-03-20T09:33:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3685000000000001e-05 2024-03-20T09:33:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:33:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.880724+0000", 2024-03-20T09:33:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011470009999999999 2024-03-20T09:33:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:33:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:42.880771+0000", 2024-03-20T09:33:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6768999999999999e-05 2024-03-20T09:33:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.189 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:33:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:164068 7.4 7:2c036318:::benchmark_data_smithi122_187077_object37457:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e920)", 2024-03-20T09:33:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:33:32.217836+0000", 2024-03-20T09:33:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 2.8872380680000003, 2024-03-20T09:33:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.122065252, 2024-03-20T09:33:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:33:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:33:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:33:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:33:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:33:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 164068 2024-03-20T09:33:35.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:33:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:33:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.217836+0000", 2024-03-20T09:33:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:33:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.217836+0000", 2024-03-20T09:33:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:33:35.191 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.217837+0000", 2024-03-20T09:33:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3999999999999999e-06 2024-03-20T09:33:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:33:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.217841+0000", 2024-03-20T09:33:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4999999999999999e-06 2024-03-20T09:33:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:33:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.217851+0000", 2024-03-20T09:33:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0349999999999999e-05 2024-03-20T09:33:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.192 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:33:35.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.217858+0000", 2024-03-20T09:33:35.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2010000000000002e-06 2024-03-20T09:33:35.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:33:35.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.217928+0000", 2024-03-20T09:33:35.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9964999999999998e-05 2024-03-20T09:33:35.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:33:35.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.218039+0000", 2024-03-20T09:33:35.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000110834 2024-03-20T09:33:35.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 1,4", 2024-03-20T09:33:35.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.218148+0000", 2024-03-20T09:33:35.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000108849 2024-03-20T09:33:35.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:33:35.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.229671+0000", 2024-03-20T09:33:35.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011522681999999999 2024-03-20T09:33:35.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:33:35.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.234629+0000", 2024-03-20T09:33:35.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049586830000000002 2024-03-20T09:33:35.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.195 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:33:35.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.339785+0000", 2024-03-20T09:33:35.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.10515546200000001 2024-03-20T09:33:35.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.195 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:33:35.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.339848+0000", 2024-03-20T09:33:35.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3003000000000006e-05 2024-03-20T09:33:35.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.195 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:33:35.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.339901+0000", 2024-03-20T09:33:35.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3322999999999998e-05 2024-03-20T09:33:35.196 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.196 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:33:35.196 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.196 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:164069 7.4 7:2ef55e6b:::benchmark_data_smithi122_187077_object37458:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e920)", 2024-03-20T09:33:35.196 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:33:32.224317+0000", 2024-03-20T09:33:35.196 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 2.880756758, 2024-03-20T09:33:35.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.115770631, 2024-03-20T09:33:35.196 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:33:35.196 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:33:35.196 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:33:35.196 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:33:35.196 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:33:35.196 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 164069 2024-03-20T09:33:35.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.197 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:33:35.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:33:35.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.224317+0000", 2024-03-20T09:33:35.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:33:35.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.224317+0000", 2024-03-20T09:33:35.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:33:35.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.224319+0000", 2024-03-20T09:33:35.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8390000000000001e-06 2024-03-20T09:33:35.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:33:35.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.224330+0000", 2024-03-20T09:33:35.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0844e-05 2024-03-20T09:33:35.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:33:35.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.224343+0000", 2024-03-20T09:33:35.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3359e-05 2024-03-20T09:33:35.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:33:35.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.224352+0000", 2024-03-20T09:33:35.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8370000000000003e-06 2024-03-20T09:33:35.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:33:35.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.224637+0000", 2024-03-20T09:33:35.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00028526000000000002 2024-03-20T09:33:35.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:33:35.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.224746+0000", 2024-03-20T09:33:35.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000108355 2024-03-20T09:33:35.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 1,4", 2024-03-20T09:33:35.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.224863+0000", 2024-03-20T09:33:35.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000117401 2024-03-20T09:33:35.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:33:35.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.234636+0000", 2024-03-20T09:33:35.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097724769999999999 2024-03-20T09:33:35.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:33:35.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.339945+0000", 2024-03-20T09:33:35.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.105309131 2024-03-20T09:33:35.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:33:35.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.339988+0000", 2024-03-20T09:33:35.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2758e-05 2024-03-20T09:33:35.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:33:35.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.340035+0000", 2024-03-20T09:33:35.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7649999999999999e-05 2024-03-20T09:33:35.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:33:35.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.340088+0000", 2024-03-20T09:33:35.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2719999999999997e-05 2024-03-20T09:33:35.202 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.202 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:33:35.202 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.202 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:164071 7.4 7:27c6646f:::benchmark_data_smithi122_187077_object37460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e920)", 2024-03-20T09:33:35.202 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:33:32.224374+0000", 2024-03-20T09:33:35.202 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 2.8806999060000003, 2024-03-20T09:33:35.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.12627749099999999, 2024-03-20T09:33:35.202 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:33:35.202 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:33:35.202 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:33:35.202 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:33:35.203 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:33:35.203 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 164071 2024-03-20T09:33:35.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.203 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:33:35.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:33:35.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.224374+0000", 2024-03-20T09:33:35.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:33:35.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.224374+0000", 2024-03-20T09:33:35.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:33:35.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.224375+0000", 2024-03-20T09:33:35.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.63e-07 2024-03-20T09:33:35.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:33:35.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.224378+0000", 2024-03-20T09:33:35.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8930000000000001e-06 2024-03-20T09:33:35.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:33:35.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.224384+0000", 2024-03-20T09:33:35.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3459999999999999e-06 2024-03-20T09:33:35.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:33:35.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.224390+0000", 2024-03-20T09:33:35.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7259999999999999e-06 2024-03-20T09:33:35.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:33:35.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.229704+0000", 2024-03-20T09:33:35.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0053144430000000003 2024-03-20T09:33:35.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:33:35.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.229822+0000", 2024-03-20T09:33:35.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011838800000000001 2024-03-20T09:33:35.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 1,4", 2024-03-20T09:33:35.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.229935+0000", 2024-03-20T09:33:35.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000112213 2024-03-20T09:33:35.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:33:35.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.340249+0000", 2024-03-20T09:33:35.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.110314105 2024-03-20T09:33:35.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:33:35.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.340265+0000", 2024-03-20T09:33:35.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6446000000000001e-05 2024-03-20T09:33:35.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:33:35.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.350548+0000", 2024-03-20T09:33:35.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010282741 2024-03-20T09:33:35.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:33:35.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.350606+0000", 2024-03-20T09:33:35.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8284e-05 2024-03-20T09:33:35.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:33:35.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.350652+0000", 2024-03-20T09:33:35.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5442999999999998e-05 2024-03-20T09:33:35.208 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.208 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:33:35.208 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.208 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:164079 7.4 7:2d0cff84:::benchmark_data_smithi122_187077_object37468:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e920)", 2024-03-20T09:33:35.208 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:33:32.234858+0000", 2024-03-20T09:33:35.208 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 2.870216229, 2024-03-20T09:33:35.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.116322148, 2024-03-20T09:33:35.208 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:33:35.209 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:33:35.209 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:33:35.209 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:33:35.209 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:33:35.209 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 164079 2024-03-20T09:33:35.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.209 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:33:35.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:33:35.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.234858+0000", 2024-03-20T09:33:35.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:33:35.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.234858+0000", 2024-03-20T09:33:35.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:33:35.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:33:35.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.234858+0000", 2024-03-20T09:33:35.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.34e-07 2024-03-20T09:33:35.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:33:35.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.234860+0000", 2024-03-20T09:33:35.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9980000000000002e-06 2024-03-20T09:33:35.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:33:35.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.234864+0000", 2024-03-20T09:33:35.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1880000000000004e-06 2024-03-20T09:33:35.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:33:35.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.234868+0000", 2024-03-20T09:33:35.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3529999999999999e-06 2024-03-20T09:33:35.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:33:35.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.334665+0000", 2024-03-20T09:33:35.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.099797371999999995 2024-03-20T09:33:35.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:33:35.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.334804+0000", 2024-03-20T09:33:35.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013928700000000001 2024-03-20T09:33:35.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 1,4", 2024-03-20T09:33:35.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.334933+0000", 2024-03-20T09:33:35.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012868800000000001 2024-03-20T09:33:35.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:33:35.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.350339+0000", 2024-03-20T09:33:35.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015406095 2024-03-20T09:33:35.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:33:35.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.350384+0000", 2024-03-20T09:33:35.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4477999999999997e-05 2024-03-20T09:33:35.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:33:35.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.351089+0000", 2024-03-20T09:33:35.213 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00070541799999999997 2024-03-20T09:33:35.213 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.213 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.213 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:33:35.213 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.351137+0000", 2024-03-20T09:33:35.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7802000000000001e-05 2024-03-20T09:33:35.214 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:33:35.214 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:33:35.214 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:33:35.214 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:32.351180+0000", 2024-03-20T09:33:35.214 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3135e-05 2024-03-20T09:33:35.214 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.214 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:33:35.214 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.214 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:33:35.214 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:33:35.214 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:33:35.215 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:35.216 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:33:35.317 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:33:35.390 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:35.474 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:33:35.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:35.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:33:35.590 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:33:35.670 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:35 smithi026 ceph-mon[68309]: pgmap v2230: 47 pgs: 47 active+clean; 5.5 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 821 op/s 2024-03-20T09:33:35.672 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:35.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:35 smithi115 ceph-mon[72476]: pgmap v2230: 47 pgs: 47 active+clean; 5.5 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 821 op/s 2024-03-20T09:33:35.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:35 smithi122 ceph-mon[73495]: pgmap v2230: 47 pgs: 47 active+clean; 5.5 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 821 op/s 2024-03-20T09:33:36.577 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:33:37.645 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:37 smithi026 ceph-mon[68309]: pgmap v2231: 47 pgs: 47 active+clean; 5.3 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 800 op/s 2024-03-20T09:33:37.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:37 smithi115 ceph-mon[72476]: pgmap v2231: 47 pgs: 47 active+clean; 5.3 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 800 op/s 2024-03-20T09:33:37.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:37 smithi122 ceph-mon[73495]: pgmap v2231: 47 pgs: 47 active+clean; 5.3 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 800 op/s 2024-03-20T09:33:38.726 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:33:38.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:38.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:33:38.760 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-20T09:33:38.836 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:33:38.872 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:38.896 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:33:38.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:38.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:33:38.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:38 smithi026 ceph-mon[68309]: pgmap v2232: 47 pgs: 47 active+clean; 5.2 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 802 op/s 2024-03-20T09:33:39.002 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:33:39.029 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:38 smithi122 ceph-mon[73495]: pgmap v2232: 47 pgs: 47 active+clean; 5.2 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 802 op/s 2024-03-20T09:33:39.030 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:39.050 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:33:39.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:39.053 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:33:39.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:38 smithi115 ceph-mon[72476]: pgmap v2232: 47 pgs: 47 active+clean; 5.2 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 802 op/s 2024-03-20T09:33:39.153 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:33:39.183 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:39.209 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:33:39.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:39.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:33:39.313 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:33:39.340 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:39.363 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:33:39.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:39.366 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:33:39.467 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:33:39.501 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:39.529 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:33:39.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:39.533 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:33:39.634 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:33:39.663 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:39.684 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:33:39.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:39.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:33:39.792 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:33:39.878 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:39 smithi026 ceph-mon[68309]: 1.10 scrub starts 2024-03-20T09:33:39.879 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:39 smithi026 ceph-mon[68309]: 1.10 scrub ok 2024-03-20T09:33:39.879 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:39 smithi026 ceph-mon[68309]: 7.a scrub starts 2024-03-20T09:33:39.882 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:40.122 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:33:40.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:40.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:33:40.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:39 smithi115 ceph-mon[72476]: 1.10 scrub starts 2024-03-20T09:33:40.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:39 smithi115 ceph-mon[72476]: 1.10 scrub ok 2024-03-20T09:33:40.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:39 smithi115 ceph-mon[72476]: 7.a scrub starts 2024-03-20T09:33:40.162 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:33:40.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:39 smithi122 ceph-mon[73495]: 1.10 scrub starts 2024-03-20T09:33:40.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:39 smithi122 ceph-mon[73495]: 1.10 scrub ok 2024-03-20T09:33:40.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:39 smithi122 ceph-mon[73495]: 7.a scrub starts 2024-03-20T09:33:40.202 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:33:40.229 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:33:40.257 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:40.279 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:33:40.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:40.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:33:40.384 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:33:40.472 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:40.792 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:40 smithi115 ceph-mon[72476]: 1.17 scrub starts 2024-03-20T09:33:40.792 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:40 smithi115 ceph-mon[72476]: 1.17 scrub ok 2024-03-20T09:33:40.793 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:40 smithi115 ceph-mon[72476]: pgmap v2233: 47 pgs: 47 active+clean; 5.2 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 804 op/s 2024-03-20T09:33:40.793 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:40 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3110412396' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-03-20T09:33:40.793 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:40 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-03-20T09:33:40.793 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:40 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1378139833' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:33:40.793 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:40 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:33:40.811 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:33:40.813 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.9 to 1 (10000) 2024-03-20T09:33:41.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:40 smithi122 ceph-mon[73495]: 1.17 scrub starts 2024-03-20T09:33:41.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:40 smithi122 ceph-mon[73495]: 1.17 scrub ok 2024-03-20T09:33:41.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:40 smithi122 ceph-mon[73495]: pgmap v2233: 47 pgs: 47 active+clean; 5.2 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 804 op/s 2024-03-20T09:33:41.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:40 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3110412396' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-03-20T09:33:41.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:40 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-03-20T09:33:41.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:40 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1378139833' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:33:41.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:40 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:33:41.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:40 smithi026 ceph-mon[68309]: 1.17 scrub starts 2024-03-20T09:33:41.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:40 smithi026 ceph-mon[68309]: 1.17 scrub ok 2024-03-20T09:33:41.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:40 smithi026 ceph-mon[68309]: pgmap v2233: 47 pgs: 47 active+clean; 5.2 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 804 op/s 2024-03-20T09:33:41.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:40 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3110412396' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-03-20T09:33:41.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:40 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-03-20T09:33:41.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:40 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1378139833' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:33:41.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:40 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:33:42.064 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:41 smithi026 ceph-mon[68309]: 1.18 scrub starts 2024-03-20T09:33:42.064 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:41 smithi026 ceph-mon[68309]: 1.18 scrub ok 2024-03-20T09:33:42.064 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:41 smithi026 ceph-mon[68309]: 7.a scrub ok 2024-03-20T09:33:42.064 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:41 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:33:42.064 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:41 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-03-20T09:33:42.064 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:41 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:33:42.064 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:41 smithi026 ceph-mon[68309]: osdmap e922: 12 total, 12 up, 11 in 2024-03-20T09:33:42.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:41 smithi115 ceph-mon[72476]: 1.18 scrub starts 2024-03-20T09:33:42.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:41 smithi115 ceph-mon[72476]: 1.18 scrub ok 2024-03-20T09:33:42.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:41 smithi115 ceph-mon[72476]: 7.a scrub ok 2024-03-20T09:33:42.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:41 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:33:42.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:41 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-03-20T09:33:42.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:41 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:33:42.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:41 smithi115 ceph-mon[72476]: osdmap e922: 12 total, 12 up, 11 in 2024-03-20T09:33:42.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:41 smithi122 ceph-mon[73495]: 1.18 scrub starts 2024-03-20T09:33:42.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:41 smithi122 ceph-mon[73495]: 1.18 scrub ok 2024-03-20T09:33:42.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:41 smithi122 ceph-mon[73495]: 7.a scrub ok 2024-03-20T09:33:42.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:41 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:33:42.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:41 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-03-20T09:33:42.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:41 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:33:42.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:41 smithi122 ceph-mon[73495]: osdmap e922: 12 total, 12 up, 11 in 2024-03-20T09:33:42.611 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:33:42.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:42.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:33:42.720 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:33:42.762 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:42.792 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:33:42.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:42.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:33:42.899 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:33:42.932 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:42.958 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:33:42.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:42.961 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:33:42.966 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:42 smithi026 ceph-mon[68309]: pgmap v2235: 47 pgs: 47 active+clean; 5.0 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 820 op/s 2024-03-20T09:33:43.062 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:33:43.125 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:43.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:42 smithi115 ceph-mon[72476]: pgmap v2235: 47 pgs: 47 active+clean; 5.0 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 820 op/s 2024-03-20T09:33:43.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:42 smithi122 ceph-mon[73495]: pgmap v2235: 47 pgs: 47 active+clean; 5.0 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 820 op/s 2024-03-20T09:33:43.355 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:33:43.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:43.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:33:43.461 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:33:43.510 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:43.664 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:33:43.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:43.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:33:43.785 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:33:43.819 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:43.846 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:33:43.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:43.849 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:33:43.950 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:33:44.035 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:44.317 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:33:44.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:44.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:33:44.367 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-20T09:33:44.367 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:33:44.433 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:33:44.473 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:44.500 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:33:44.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:44.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:33:44.604 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:33:44.634 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:44.658 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:33:44.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:44.661 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:33:44.763 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:33:44.841 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:45.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:45 smithi115 ceph-mon[72476]: pgmap v2236: 47 pgs: 47 active+clean; 5.0 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 814 op/s 2024-03-20T09:33:45.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:45 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:33:45.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:45 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:33:45.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:45 smithi122 ceph-mon[73495]: pgmap v2236: 47 pgs: 47 active+clean; 5.0 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 814 op/s 2024-03-20T09:33:45.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:45 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:33:45.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:45 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:33:45.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:45 smithi026 ceph-mon[68309]: pgmap v2236: 47 pgs: 47 active+clean; 5.0 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 814 op/s 2024-03-20T09:33:45.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:45 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:33:45.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:45 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:33:46.018 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:33:46.064 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:33:46.089 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:33:46.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:46.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:33:46.120 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:33:46.171 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:33:46.205 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:33:46.214 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:33:46.234 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:46.255 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:33:46.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:46.258 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:33:46.266 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:33:46.298 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:33:46.332 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:33:46.359 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:33:46.382 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:33:46.416 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:33:46.451 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:33:46.466 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:46.501 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:33:46.517 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:33:46.517 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:33:46.517 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:33:46.517 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:33:46.518 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:33:46.518 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:33:46.518 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:33:46.518 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:33:46.518 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:33:46.518 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:33:46.518 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:33:46.518 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:33:46.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:46 smithi122 ceph-mon[73495]: osdmap e923: 12 total, 12 up, 11 in 2024-03-20T09:33:46.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:46 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:33:46.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:46 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:33:46.558 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:46 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1750122393' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:33:46.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:46 smithi115 ceph-mon[72476]: osdmap e923: 12 total, 12 up, 11 in 2024-03-20T09:33:46.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:46 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:33:46.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:46 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:33:46.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:46 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1750122393' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:33:46.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:46 smithi026 ceph-mon[68309]: osdmap e923: 12 total, 12 up, 11 in 2024-03-20T09:33:46.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:46 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:33:46.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:46 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:33:46.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:46 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1750122393' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:33:47.086 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:33:47.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:47.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:33:47.190 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:33:47.256 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:47.824 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:47 smithi026 ceph-mon[68309]: pgmap v2238: 47 pgs: 47 active+clean; 4.8 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 795 op/s 2024-03-20T09:33:47.824 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:47 smithi026 ceph-mon[68309]: osdmap e924: 12 total, 12 up, 11 in 2024-03-20T09:33:47.878 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:33:47.878 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:33:47.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:47 smithi115 ceph-mon[72476]: pgmap v2238: 47 pgs: 47 active+clean; 4.8 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 795 op/s 2024-03-20T09:33:47.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:47 smithi115 ceph-mon[72476]: osdmap e924: 12 total, 12 up, 11 in 2024-03-20T09:33:47.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:47 smithi122 ceph-mon[73495]: pgmap v2238: 47 pgs: 47 active+clean; 4.8 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 795 op/s 2024-03-20T09:33:47.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:47 smithi122 ceph-mon[73495]: osdmap e924: 12 total, 12 up, 11 in 2024-03-20T09:33:48.127 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:33:48.542 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:33:48.549 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:33:48.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:48.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:33:48.588 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-20T09:33:48.664 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:33:48.693 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:48.725 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:33:48.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:48.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:33:48.832 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:33:48.862 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:48.889 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:33:48.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:48.894 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:33:48.905 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:48 smithi026 ceph-mon[68309]: pgmap v2240: 47 pgs: 47 active+clean; 4.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 867 op/s 2024-03-20T09:33:48.994 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:33:49.049 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:49.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:48 smithi115 ceph-mon[72476]: pgmap v2240: 47 pgs: 47 active+clean; 4.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 867 op/s 2024-03-20T09:33:49.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:48 smithi122 ceph-mon[73495]: pgmap v2240: 47 pgs: 47 active+clean; 4.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 867 op/s 2024-03-20T09:33:49.500 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:33:49.521 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:33:49.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:49.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:33:49.633 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:33:49.692 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:49.826 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:33:50.016 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:33:50.109 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:33:50.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:50.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:33:50.233 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:33:50.261 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:50.289 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:33:50.292 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:50.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:33:50.394 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:33:50.426 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:50.449 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:33:50.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:50.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:33:50.553 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:33:50.653 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:51.883 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:33:51.885 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:51 smithi026 ceph-mon[68309]: pgmap v2241: 47 pgs: 47 active+clean; 4.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 731 op/s 2024-03-20T09:33:51.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:51.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:33:51.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:51 smithi115 ceph-mon[72476]: pgmap v2241: 47 pgs: 47 active+clean; 4.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 731 op/s 2024-03-20T09:33:51.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:51 smithi122 ceph-mon[73495]: pgmap v2241: 47 pgs: 47 active+clean; 4.7 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 731 op/s 2024-03-20T09:33:51.989 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:33:52.017 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:52.052 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:33:52.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:52.055 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:33:52.156 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:33:52.184 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:52.218 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:33:52.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:52.222 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:33:52.323 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:33:52.349 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:52.352 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:33:52.353 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:33:52.368 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:33:52.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:52.371 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:33:52.472 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:33:52.552 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:53.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:53 smithi115 ceph-mon[72476]: from='client.26720 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:33:53.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:53 smithi115 ceph-mon[72476]: pgmap v2242: 47 pgs: 47 active+clean; 4.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 913 op/s 2024-03-20T09:33:53.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:53 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:33:53.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:53 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:33:53.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:53 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:33:53.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:53 smithi122 ceph-mon[73495]: from='client.26720 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:33:53.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:53 smithi122 ceph-mon[73495]: pgmap v2242: 47 pgs: 47 active+clean; 4.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 913 op/s 2024-03-20T09:33:53.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:53 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:33:53.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:53 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:33:53.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:53 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:33:53.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:53 smithi026 ceph-mon[68309]: from='client.26720 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:33:53.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:53 smithi026 ceph-mon[68309]: pgmap v2242: 47 pgs: 47 active+clean; 4.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 913 op/s 2024-03-20T09:33:53.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:53 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:33:53.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:53 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:33:53.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:53 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:33:54.337 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:33:54.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:54.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:33:54.446 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:33:54.478 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:54.511 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:33:54.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:54.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:33:54.615 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:33:54.645 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:54.669 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:33:54.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:54.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:33:54.775 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:33:54.808 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:54.841 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:33:54.849 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:54 smithi122 ceph-mon[73495]: osdmap e925: 12 total, 12 up, 11 in 2024-03-20T09:33:54.849 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:54 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:33:54.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:54.850 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:33:54.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:54 smithi115 ceph-mon[72476]: osdmap e925: 12 total, 12 up, 11 in 2024-03-20T09:33:54.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:54 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:33:54.951 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:33:54.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:54 smithi026 ceph-mon[68309]: osdmap e925: 12 total, 12 up, 11 in 2024-03-20T09:33:54.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:54 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:33:54.980 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:55.000 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:33:55.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:55.011 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:33:55.112 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:33:55.207 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:55.603 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:55 smithi026 ceph-mon[68309]: pgmap v2244: 47 pgs: 47 active+clean; 4.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 715 op/s 2024-03-20T09:33:55.604 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:55 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:33:55.604 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:55 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:33:55.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:55 smithi115 ceph-mon[72476]: pgmap v2244: 47 pgs: 47 active+clean; 4.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 715 op/s 2024-03-20T09:33:55.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:55 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:33:55.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:55 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:33:55.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:55 smithi122 ceph-mon[73495]: pgmap v2244: 47 pgs: 47 active+clean; 4.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 715 op/s 2024-03-20T09:33:55.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:55 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:33:55.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:55 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:33:56.088 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":2241,"stamp":"2024-03-20T09:33:50.463569+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5025759800,"num_objects":76681,"num_object_clones":0,"num_object_copies":230043,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76681,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":176539,"num_write_kb":8102913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":165087,"ondisk_log_size":165087,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":22636184,"kb_used_data":14798892,"kb_used_omap":197,"kb_used_meta":7836858,"kb_avail":1008560488,"statfs":{"total":1055945392128,"available":1032765939712,"internally_reserved":0,"allocated":15154065408,"data_stored":15150796064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202077,"internal_metadata":8024943267},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,4,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-383844352,"num_objects":-5857,"num_object_clones":0,"num_object_copies":-17571,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5857,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.007677"},"pg_stats":[{"pgid":"1.1e","version":"20'137","reported_seq":1285,"reported_epoch":922,"state":"active+clean","last_fresh":"2024-03-20T09:33:40.798410+0000","last_change":"2024-03-20T09:33:12.168422+0000","last_active":"2024-03-20T09:33:40.798410+0000","last_peered":"2024-03-20T09:33:40.798410+0000","last_clean":"2024-03-20T09:33:40.798410+0000","last_became_active":"2024-03-20T09:13:20.071655+0000","last_became_peered":"2024-03-20T09:13:20.071655+0000","last_unstale":"2024-03-20T09:33:40.798410+0000","last_undegraded":"2024-03-20T09:33:40.798410+0000","last_fullsized":"2024-03-20T09:33:40.798410+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:12.168339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:33:12.168339+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:33.973327+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'137","reported_seq":363,"reported_epoch":922,"state":"active+clean","last_fresh":"2024-03-20T09:33:40.798467+0000","last_change":"2024-03-20T09:32:45.273878+0000","last_active":"2024-03-20T09:33:40.798467+0000","last_peered":"2024-03-20T09:33:40.798467+0000","last_clean":"2024-03-20T09:33:40.798467+0000","last_became_active":"2024-03-20T09:31:07.425574+0000","last_became_peered":"2024-03-20T09:31:07.425574+0000","last_unstale":"2024-03-20T09:33:40.798467+0000","last_undegraded":"2024-03-20T09:33:40.798467+0000","last_fullsized":"2024-03-20T09:33:40.798467+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:32:45.273763+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:29:02.585165+0000","last_clean_scrub_stamp":"2024-03-20T09:32:45.273763+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:01.647867+0000","scrub_duration":162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2],"acting":[11,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"20'137","reported_seq":1952,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-20T09:33:46.579200+0000","last_change":"2024-03-20T09:32:23.261615+0000","last_active":"2024-03-20T09:33:46.579200+0000","last_peered":"2024-03-20T09:33:46.579200+0000","last_clean":"2024-03-20T09:33:46.579200+0000","last_became_active":"2024-03-20T09:25:35.482959+0000","last_became_peered":"2024-03-20T09:25:35.482959+0000","last_unstale":"2024-03-20T09:33:46.579200+0000","last_undegraded":"2024-03-20T09:33:46.579200+0000","last_fullsized":"2024-03-20T09:33:46.579200+0000","mapping_epoch":744,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":745,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:32:23.261521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:32:23.261521+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:33:46.616430+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,1],"acting":[8,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.19","version":"20'137","reported_seq":1133,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-20T09:33:46.578125+0000","last_change":"2024-03-20T09:32:28.348352+0000","last_active":"2024-03-20T09:33:46.578125+0000","last_peered":"2024-03-20T09:33:46.578125+0000","last_clean":"2024-03-20T09:33:46.578125+0000","last_became_active":"2024-03-20T09:28:22.591294+0000","last_became_peered":"2024-03-20T09:28:22.591294+0000","last_unstale":"2024-03-20T09:33:46.578125+0000","last_undegraded":"2024-03-20T09:33:46.578125+0000","last_fullsized":"2024-03-20T09:33:46.578125+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:32:28.348215+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:30:56.201554+0000","last_clean_scrub_stamp":"2024-03-20T09:32:28.348215+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.18","version":"20'137","reported_seq":1435,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-20T09:33:46.579252+0000","last_change":"2024-03-20T09:33:40.415955+0000","last_active":"2024-03-20T09:33:46.579252+0000","last_peered":"2024-03-20T09:33:46.579252+0000","last_clean":"2024-03-20T09:33:46.579252+0000","last_became_active":"2024-03-20T09:10:35.277481+0000","last_became_peered":"2024-03-20T09:10:35.277481+0000","last_unstale":"2024-03-20T09:33:46.579252+0000","last_undegraded":"2024-03-20T09:33:46.579252+0000","last_fullsized":"2024-03-20T09:33:46.579252+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:40.415846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:33:40.415846+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:52.941527+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1b","version":"20'137","reported_seq":1789,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-20T09:33:46.579156+0000","last_change":"2024-03-20T09:33:13.685851+0000","last_active":"2024-03-20T09:33:46.579156+0000","last_peered":"2024-03-20T09:33:46.579156+0000","last_clean":"2024-03-20T09:33:46.579156+0000","last_became_active":"2024-03-20T09:24:16.445287+0000","last_became_peered":"2024-03-20T09:24:16.445287+0000","last_unstale":"2024-03-20T09:33:46.579156+0000","last_undegraded":"2024-03-20T09:33:46.579156+0000","last_fullsized":"2024-03-20T09:33:46.579156+0000","mapping_epoch":718,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:13.685707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:33:13.685707+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:14.072191+0000","scrub_duration":188,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.e","version":"924'11100","reported_seq":11298,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-20T09:33:48.031879+0000","last_change":"2024-03-20T09:32:36.556340+0000","last_active":"2024-03-20T09:33:48.031879+0000","last_peered":"2024-03-20T09:33:48.031879+0000","last_clean":"2024-03-20T09:33:48.031879+0000","last_became_active":"2024-03-20T09:31:07.427099+0000","last_became_peered":"2024-03-20T09:31:07.427099+0000","last_unstale":"2024-03-20T09:33:48.031879+0000","last_undegraded":"2024-03-20T09:33:48.031879+0000","last_fullsized":"2024-03-20T09:33:48.031879+0000","mapping_epoch":871,"log_start":"877'1000","ondisk_log_start":"877'1000","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"904'7287","last_scrub_stamp":"2024-03-20T09:32:36.556054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:36.556054+0000","objects_scrubbed":7222,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:00.317576+0000","scrub_duration":2205,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310247424,"num_objects":4734,"num_object_clones":0,"num_object_copies":14202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11100,"num_write_kb":506688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,5],"acting":[2,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"20'137","reported_seq":1427,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-20T09:33:46.579358+0000","last_change":"2024-03-20T09:32:28.004445+0000","last_active":"2024-03-20T09:33:46.579358+0000","last_peered":"2024-03-20T09:33:46.579358+0000","last_clean":"2024-03-20T09:33:46.579358+0000","last_became_active":"2024-03-20T09:10:35.215772+0000","last_became_peered":"2024-03-20T09:10:35.215772+0000","last_unstale":"2024-03-20T09:33:46.579358+0000","last_undegraded":"2024-03-20T09:33:46.579358+0000","last_fullsized":"2024-03-20T09:33:46.579358+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:32:28.004327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:32:28.004327+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"924'10969","reported_seq":11166,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-20T09:33:46.710682+0000","last_change":"2024-03-20T09:32:38.283331+0000","last_active":"2024-03-20T09:33:46.710682+0000","last_peered":"2024-03-20T09:33:46.710682+0000","last_clean":"2024-03-20T09:33:46.710682+0000","last_became_active":"2024-03-20T09:31:07.534696+0000","last_became_peered":"2024-03-20T09:31:07.534696+0000","last_unstale":"2024-03-20T09:33:46.710682+0000","last_undegraded":"2024-03-20T09:33:46.710682+0000","last_fullsized":"2024-03-20T09:33:46.710682+0000","mapping_epoch":871,"log_start":"876'900","ondisk_log_start":"876'900","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"905'7295","last_scrub_stamp":"2024-03-20T09:32:38.283112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:38.283112+0000","objects_scrubbed":7235,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:06.368167+0000","scrub_duration":2079,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318046208,"num_objects":4853,"num_object_clones":0,"num_object_copies":14559,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10969,"num_write_kb":506304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"20'137","reported_seq":2019,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-20T09:33:46.568502+0000","last_change":"2024-03-20T09:33:14.407075+0000","last_active":"2024-03-20T09:33:46.568502+0000","last_peered":"2024-03-20T09:33:46.568502+0000","last_clean":"2024-03-20T09:33:46.568502+0000","last_became_active":"2024-03-20T09:20:25.341750+0000","last_became_peered":"2024-03-20T09:20:25.341750+0000","last_unstale":"2024-03-20T09:33:46.568502+0000","last_undegraded":"2024-03-20T09:33:46.568502+0000","last_fullsized":"2024-03-20T09:33:46.568502+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:14.406974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:33:14.406974+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:29.321937+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,2],"acting":[9,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.c","version":"923'10947","reported_seq":11142,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-03-20T09:33:46.276301+0000","last_change":"2024-03-20T09:32:33.031806+0000","last_active":"2024-03-20T09:33:46.276301+0000","last_peered":"2024-03-20T09:33:46.276301+0000","last_clean":"2024-03-20T09:33:46.276301+0000","last_became_active":"2024-03-20T09:31:07.426984+0000","last_became_peered":"2024-03-20T09:31:07.426984+0000","last_unstale":"2024-03-20T09:33:46.276301+0000","last_undegraded":"2024-03-20T09:33:46.276301+0000","last_fullsized":"2024-03-20T09:33:46.276301+0000","mapping_epoch":871,"log_start":"876'900","ondisk_log_start":"876'900","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"904'7023","last_scrub_stamp":"2024-03-20T09:32:33.031701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:33.031701+0000","objects_scrubbed":6989,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1492,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315949056,"num_objects":4821,"num_object_clones":0,"num_object_copies":14463,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10947,"num_write_kb":504576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,11],"acting":[6,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"20'137","reported_seq":708,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-20T09:33:46.568873+0000","last_change":"2024-03-20T09:33:10.442796+0000","last_active":"2024-03-20T09:33:46.568873+0000","last_peered":"2024-03-20T09:33:46.568873+0000","last_clean":"2024-03-20T09:33:46.568873+0000","last_became_active":"2024-03-20T09:24:16.444232+0000","last_became_peered":"2024-03-20T09:24:16.444232+0000","last_unstale":"2024-03-20T09:33:46.568873+0000","last_undegraded":"2024-03-20T09:33:46.568873+0000","last_fullsized":"2024-03-20T09:33:46.568873+0000","mapping_epoch":718,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:10.442719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:33:10.442719+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:26.643584+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.2","version":"924'11205","reported_seq":11395,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-20T09:33:50.256379+0000","last_change":"2024-03-20T09:32:21.786210+0000","last_active":"2024-03-20T09:33:50.256379+0000","last_peered":"2024-03-20T09:33:50.256379+0000","last_clean":"2024-03-20T09:33:50.256379+0000","last_became_active":"2024-03-20T09:31:07.422232+0000","last_became_peered":"2024-03-20T09:31:07.422232+0000","last_unstale":"2024-03-20T09:33:50.256379+0000","last_undegraded":"2024-03-20T09:33:50.256379+0000","last_fullsized":"2024-03-20T09:33:50.256379+0000","mapping_epoch":871,"log_start":"877'1200","ondisk_log_start":"877'1200","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'6455","last_scrub_stamp":"2024-03-20T09:32:21.785983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:21.785983+0000","objects_scrubbed":6398,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:33:54.924584+0000","scrub_duration":1934,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":309919744,"num_objects":4729,"num_object_clones":0,"num_object_copies":14187,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11205,"num_write_kb":509888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"20'137","reported_seq":1953,"reported_epoch":922,"state":"active+clean","last_fresh":"2024-03-20T09:33:40.801686+0000","last_change":"2024-03-20T09:33:10.120697+0000","last_active":"2024-03-20T09:33:40.801686+0000","last_peered":"2024-03-20T09:33:40.801686+0000","last_clean":"2024-03-20T09:33:40.801686+0000","last_became_active":"2024-03-20T09:12:14.306384+0000","last_became_peered":"2024-03-20T09:12:14.306384+0000","last_unstale":"2024-03-20T09:33:40.801686+0000","last_undegraded":"2024-03-20T09:33:40.801686+0000","last_fullsized":"2024-03-20T09:33:40.801686+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:10.120609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:33:10.120609+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:37.759793+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.3","version":"924'11175","reported_seq":11431,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-20T09:33:50.256371+0000","last_change":"2024-03-20T09:33:29.905371+0000","last_active":"2024-03-20T09:33:50.256371+0000","last_peered":"2024-03-20T09:33:50.256371+0000","last_clean":"2024-03-20T09:33:50.256371+0000","last_became_active":"2024-03-20T09:31:07.422509+0000","last_became_peered":"2024-03-20T09:31:07.422509+0000","last_unstale":"2024-03-20T09:33:50.256371+0000","last_undegraded":"2024-03-20T09:33:50.256371+0000","last_fullsized":"2024-03-20T09:33:50.256371+0000","mapping_epoch":871,"log_start":"877'1100","ondisk_log_start":"877'1100","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"920'10157","last_scrub_stamp":"2024-03-20T09:33:29.905190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:33:29.905190+0000","objects_scrubbed":5720,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:49.933850+0000","scrub_duration":1973,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304807936,"num_objects":4651,"num_object_clones":0,"num_object_copies":13953,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11175,"num_write_kb":506432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"20'137","reported_seq":604,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-20T09:33:46.585341+0000","last_change":"2024-03-20T09:32:37.603204+0000","last_active":"2024-03-20T09:33:46.585341+0000","last_peered":"2024-03-20T09:33:46.585341+0000","last_clean":"2024-03-20T09:33:46.585341+0000","last_became_active":"2024-03-20T09:28:22.587095+0000","last_became_peered":"2024-03-20T09:28:22.587095+0000","last_unstale":"2024-03-20T09:33:46.585341+0000","last_undegraded":"2024-03-20T09:33:46.585341+0000","last_fullsized":"2024-03-20T09:33:46.585341+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:32:37.603082+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:19:03.805185+0000","last_clean_scrub_stamp":"2024-03-20T09:32:37.603082+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:05.647425+0000","scrub_duration":191,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,11],"acting":[2,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.0","version":"924'11018","reported_seq":11210,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-20T09:33:49.788059+0000","last_change":"2024-03-20T09:32:26.456393+0000","last_active":"2024-03-20T09:33:49.788059+0000","last_peered":"2024-03-20T09:33:49.788059+0000","last_clean":"2024-03-20T09:33:49.788059+0000","last_became_active":"2024-03-20T09:31:07.426307+0000","last_became_peered":"2024-03-20T09:31:07.426307+0000","last_unstale":"2024-03-20T09:33:49.788059+0000","last_undegraded":"2024-03-20T09:33:49.788059+0000","last_fullsized":"2024-03-20T09:33:49.788059+0000","mapping_epoch":871,"log_start":"877'1000","ondisk_log_start":"877'1000","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"901'6561","last_scrub_stamp":"2024-03-20T09:32:26.456235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:26.456235+0000","objects_scrubbed":6499,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:33:52.167550+0000","scrub_duration":1520,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304480256,"num_objects":4646,"num_object_clones":0,"num_object_copies":13938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11018,"num_write_kb":501248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"20'137","reported_seq":1823,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-20T09:33:46.572476+0000","last_change":"2024-03-20T09:33:13.369995+0000","last_active":"2024-03-20T09:33:46.572476+0000","last_peered":"2024-03-20T09:33:46.572476+0000","last_clean":"2024-03-20T09:33:46.572476+0000","last_became_active":"2024-03-20T09:28:22.586751+0000","last_became_peered":"2024-03-20T09:28:22.586751+0000","last_unstale":"2024-03-20T09:33:46.572476+0000","last_undegraded":"2024-03-20T09:33:46.572476+0000","last_fullsized":"2024-03-20T09:33:46.572476+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:13.369863+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:24:31.549603+0000","last_clean_scrub_stamp":"2024-03-20T09:33:13.369863+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:39.364154+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.5","version":"922'10852","reported_seq":11037,"reported_epoch":922,"state":"active+clean","last_fresh":"2024-03-20T09:33:45.480572+0000","last_change":"2024-03-20T09:32:23.538659+0000","last_active":"2024-03-20T09:33:45.480572+0000","last_peered":"2024-03-20T09:33:45.480572+0000","last_clean":"2024-03-20T09:33:45.480572+0000","last_became_active":"2024-03-20T09:31:07.427044+0000","last_became_peered":"2024-03-20T09:31:07.427044+0000","last_unstale":"2024-03-20T09:33:45.480572+0000","last_undegraded":"2024-03-20T09:33:45.480572+0000","last_fullsized":"2024-03-20T09:33:45.480572+0000","mapping_epoch":871,"log_start":"876'800","ondisk_log_start":"876'800","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"900'6405","last_scrub_stamp":"2024-03-20T09:32:23.538400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:23.538400+0000","objects_scrubbed":6353,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:33:48.306666+0000","scrub_duration":1492,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324534272,"num_objects":4952,"num_object_clones":0,"num_object_copies":14856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10852,"num_write_kb":505728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,0],"acting":[11,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.3","version":"20'137","reported_seq":1021,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-03-20T09:33:45.551852+0000","last_change":"2024-03-20T09:33:26.640631+0000","last_active":"2024-03-20T09:33:45.551852+0000","last_peered":"2024-03-20T09:33:45.551852+0000","last_clean":"2024-03-20T09:33:45.551852+0000","last_became_active":"2024-03-20T09:31:07.424588+0000","last_became_peered":"2024-03-20T09:31:07.424588+0000","last_unstale":"2024-03-20T09:33:45.551852+0000","last_undegraded":"2024-03-20T09:33:45.551852+0000","last_fullsized":"2024-03-20T09:33:45.551852+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:26.640505+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:30:47.812125+0000","last_clean_scrub_stamp":"2024-03-20T09:33:26.640505+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:56.230688+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"924'11037","reported_seq":11239,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-20T09:33:46.700011+0000","last_change":"2024-03-20T09:32:43.005957+0000","last_active":"2024-03-20T09:33:46.700011+0000","last_peered":"2024-03-20T09:33:46.700011+0000","last_clean":"2024-03-20T09:33:46.700011+0000","last_became_active":"2024-03-20T09:31:07.535053+0000","last_became_peered":"2024-03-20T09:31:07.535053+0000","last_unstale":"2024-03-20T09:33:46.700011+0000","last_undegraded":"2024-03-20T09:33:46.700011+0000","last_fullsized":"2024-03-20T09:33:46.700011+0000","mapping_epoch":871,"log_start":"877'1000","ondisk_log_start":"877'1000","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"907'7583","last_scrub_stamp":"2024-03-20T09:32:43.005812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:43.005812+0000","objects_scrubbed":7513,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:07.560498+0000","scrub_duration":2762,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314507264,"num_objects":4799,"num_object_clones":0,"num_object_copies":14397,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11037,"num_write_kb":506752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":1924,"reported_epoch":922,"state":"active+clean","last_fresh":"2024-03-20T09:33:40.798229+0000","last_change":"2024-03-20T09:33:19.153580+0000","last_active":"2024-03-20T09:33:40.798229+0000","last_peered":"2024-03-20T09:33:40.798229+0000","last_clean":"2024-03-20T09:33:40.798229+0000","last_became_active":"2024-03-20T09:12:14.305989+0000","last_became_peered":"2024-03-20T09:12:14.305989+0000","last_unstale":"2024-03-20T09:33:40.798229+0000","last_undegraded":"2024-03-20T09:33:40.798229+0000","last_fullsized":"2024-03-20T09:33:40.798229+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:19.153464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:33:19.153464+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:32.868996+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.4","version":"924'11058","reported_seq":11250,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-20T09:33:46.689291+0000","last_change":"2024-03-20T09:33:16.169687+0000","last_active":"2024-03-20T09:33:46.689291+0000","last_peered":"2024-03-20T09:33:46.689291+0000","last_clean":"2024-03-20T09:33:46.689291+0000","last_became_active":"2024-03-20T09:31:07.424820+0000","last_became_peered":"2024-03-20T09:31:07.424820+0000","last_unstale":"2024-03-20T09:33:46.689291+0000","last_undegraded":"2024-03-20T09:33:46.689291+0000","last_fullsized":"2024-03-20T09:33:46.689291+0000","mapping_epoch":871,"log_start":"877'1000","ondisk_log_start":"877'1000","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"917'9492","last_scrub_stamp":"2024-03-20T09:33:16.169522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:33:16.169522+0000","objects_scrubbed":6563,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:40.296707+0000","scrub_duration":1975,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324796416,"num_objects":4956,"num_object_clones":0,"num_object_copies":14868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11058,"num_write_kb":512448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"20'137","reported_seq":2193,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-20T09:33:46.576089+0000","last_change":"2024-03-20T09:33:33.249232+0000","last_active":"2024-03-20T09:33:46.576089+0000","last_peered":"2024-03-20T09:33:46.576089+0000","last_clean":"2024-03-20T09:33:46.576089+0000","last_became_active":"2024-03-20T09:19:54.492879+0000","last_became_peered":"2024-03-20T09:19:54.492879+0000","last_unstale":"2024-03-20T09:33:46.576089+0000","last_undegraded":"2024-03-20T09:33:46.576089+0000","last_fullsized":"2024-03-20T09:33:46.576089+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:33.249098+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:19.513511+0000","last_clean_scrub_stamp":"2024-03-20T09:33:33.249098+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:54.587251+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"923'10713","reported_seq":10903,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-03-20T09:33:46.240328+0000","last_change":"2024-03-20T09:32:28.146865+0000","last_active":"2024-03-20T09:33:46.240328+0000","last_peered":"2024-03-20T09:33:46.240328+0000","last_clean":"2024-03-20T09:33:46.240328+0000","last_became_active":"2024-03-20T09:31:07.534591+0000","last_became_peered":"2024-03-20T09:31:07.534591+0000","last_unstale":"2024-03-20T09:33:46.240328+0000","last_undegraded":"2024-03-20T09:33:46.240328+0000","last_fullsized":"2024-03-20T09:33:46.240328+0000","mapping_epoch":871,"log_start":"876'700","ondisk_log_start":"876'700","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"902'6591","last_scrub_stamp":"2024-03-20T09:32:28.146586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:28.146586+0000","objects_scrubbed":6548,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:33:51.170927+0000","scrub_duration":1556,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306905088,"num_objects":4683,"num_object_clones":0,"num_object_copies":14049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10713,"num_write_kb":492672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,3],"acting":[6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"20'137","reported_seq":1367,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-03-20T09:33:45.551664+0000","last_change":"2024-03-20T09:33:14.457333+0000","last_active":"2024-03-20T09:33:45.551664+0000","last_peered":"2024-03-20T09:33:45.551664+0000","last_clean":"2024-03-20T09:33:45.551664+0000","last_became_active":"2024-03-20T09:15:34.225022+0000","last_became_peered":"2024-03-20T09:15:34.225022+0000","last_unstale":"2024-03-20T09:33:45.551664+0000","last_undegraded":"2024-03-20T09:33:45.551664+0000","last_fullsized":"2024-03-20T09:33:45.551664+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:14.457198+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:20:12.169658+0000","last_clean_scrub_stamp":"2024-03-20T09:33:14.457198+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:16.366898+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,5],"acting":[6,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"923'10962","reported_seq":11159,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-03-20T09:33:45.616491+0000","last_change":"2024-03-20T09:32:37.563227+0000","last_active":"2024-03-20T09:33:45.616491+0000","last_peered":"2024-03-20T09:33:45.616491+0000","last_clean":"2024-03-20T09:33:45.616491+0000","last_became_active":"2024-03-20T09:31:07.421909+0000","last_became_peered":"2024-03-20T09:31:07.421909+0000","last_unstale":"2024-03-20T09:33:45.616491+0000","last_undegraded":"2024-03-20T09:33:45.616491+0000","last_fullsized":"2024-03-20T09:33:45.616491+0000","mapping_epoch":871,"log_start":"876'900","ondisk_log_start":"876'900","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"905'7297","last_scrub_stamp":"2024-03-20T09:32:37.563105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:37.563105+0000","objects_scrubbed":7240,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:04.740357+0000","scrub_duration":1719,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318636032,"num_objects":4862,"num_object_clones":0,"num_object_copies":14586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10962,"num_write_kb":506368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"20'137","reported_seq":1906,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-03-20T09:33:45.552101+0000","last_change":"2024-03-20T09:33:13.684716+0000","last_active":"2024-03-20T09:33:45.552101+0000","last_peered":"2024-03-20T09:33:45.552101+0000","last_clean":"2024-03-20T09:33:45.552101+0000","last_became_active":"2024-03-20T09:28:22.587861+0000","last_became_peered":"2024-03-20T09:28:22.587861+0000","last_unstale":"2024-03-20T09:33:45.552101+0000","last_undegraded":"2024-03-20T09:33:45.552101+0000","last_fullsized":"2024-03-20T09:33:45.552101+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:13.684628+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:14:10.560890+0000","last_clean_scrub_stamp":"2024-03-20T09:33:13.684628+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:35.420402+0000","scrub_duration":213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.d","version":"923'10984","reported_seq":11176,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-03-20T09:33:46.243915+0000","last_change":"2024-03-20T09:32:27.343018+0000","last_active":"2024-03-20T09:33:46.243915+0000","last_peered":"2024-03-20T09:33:46.243915+0000","last_clean":"2024-03-20T09:33:46.243915+0000","last_became_active":"2024-03-20T09:31:07.425062+0000","last_became_peered":"2024-03-20T09:31:07.425062+0000","last_unstale":"2024-03-20T09:33:46.243915+0000","last_undegraded":"2024-03-20T09:33:46.243915+0000","last_fullsized":"2024-03-20T09:33:46.243915+0000","mapping_epoch":871,"log_start":"876'900","ondisk_log_start":"876'900","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"901'6691","last_scrub_stamp":"2024-03-20T09:32:27.342772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:27.342772+0000","objects_scrubbed":6641,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1744,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319356952,"num_objects":4874,"num_object_clones":0,"num_object_copies":14622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4874,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10984,"num_write_kb":507393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"20'137","reported_seq":1737,"reported_epoch":922,"state":"active+clean","last_fresh":"2024-03-20T09:33:40.798354+0000","last_change":"2024-03-20T09:33:30.436327+0000","last_active":"2024-03-20T09:33:40.798354+0000","last_peered":"2024-03-20T09:33:40.798354+0000","last_clean":"2024-03-20T09:33:40.798354+0000","last_became_active":"2024-03-20T09:13:20.070596+0000","last_became_peered":"2024-03-20T09:13:20.070596+0000","last_unstale":"2024-03-20T09:33:40.798354+0000","last_undegraded":"2024-03-20T09:33:40.798354+0000","last_fullsized":"2024-03-20T09:33:40.798354+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:30.436196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:33:30.436196+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:34.613856+0000","scrub_duration":208,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.8","version":"924'10960","reported_seq":11154,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-20T09:33:48.020570+0000","last_change":"2024-03-20T09:32:28.110802+0000","last_active":"2024-03-20T09:33:48.020570+0000","last_peered":"2024-03-20T09:33:48.020570+0000","last_clean":"2024-03-20T09:33:48.020570+0000","last_became_active":"2024-03-20T09:31:07.534911+0000","last_became_peered":"2024-03-20T09:31:07.534911+0000","last_unstale":"2024-03-20T09:33:48.020570+0000","last_undegraded":"2024-03-20T09:33:48.020570+0000","last_fullsized":"2024-03-20T09:33:48.020570+0000","mapping_epoch":871,"log_start":"876'900","ondisk_log_start":"876'900","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"902'6716","last_scrub_stamp":"2024-03-20T09:32:28.110647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:28.110647+0000","objects_scrubbed":6665,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:33:49.978451+0000","scrub_duration":1785,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311296000,"num_objects":4750,"num_object_clones":0,"num_object_copies":14250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10960,"num_write_kb":502720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"20'137","reported_seq":1171,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-03-20T09:33:45.552114+0000","last_change":"2024-03-20T09:33:09.882859+0000","last_active":"2024-03-20T09:33:45.552114+0000","last_peered":"2024-03-20T09:33:45.552114+0000","last_clean":"2024-03-20T09:33:45.552114+0000","last_became_active":"2024-03-20T09:19:54.506134+0000","last_became_peered":"2024-03-20T09:19:54.506134+0000","last_unstale":"2024-03-20T09:33:45.552114+0000","last_undegraded":"2024-03-20T09:33:45.552114+0000","last_fullsized":"2024-03-20T09:33:45.552114+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:09.882745+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:29:53.362224+0000","last_clean_scrub_stamp":"2024-03-20T09:33:09.882745+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:29.991713+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"924'11150","reported_seq":11347,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-20T09:33:46.676129+0000","last_change":"2024-03-20T09:32:33.285447+0000","last_active":"2024-03-20T09:33:46.676129+0000","last_peered":"2024-03-20T09:33:46.676129+0000","last_clean":"2024-03-20T09:33:46.676129+0000","last_became_active":"2024-03-20T09:31:07.534654+0000","last_became_peered":"2024-03-20T09:31:07.534654+0000","last_unstale":"2024-03-20T09:33:46.676129+0000","last_undegraded":"2024-03-20T09:33:46.676129+0000","last_fullsized":"2024-03-20T09:33:46.676129+0000","mapping_epoch":871,"log_start":"877'1100","ondisk_log_start":"877'1100","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"904'7122","last_scrub_stamp":"2024-03-20T09:32:33.285246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:33.285246+0000","objects_scrubbed":7070,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:33:53.018572+0000","scrub_duration":2081,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315228160,"num_objects":4810,"num_object_clones":0,"num_object_copies":14430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11150,"num_write_kb":510720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"20'137","reported_seq":1935,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-03-20T09:33:45.552007+0000","last_change":"2024-03-20T09:33:33.560515+0000","last_active":"2024-03-20T09:33:45.552007+0000","last_peered":"2024-03-20T09:33:45.552007+0000","last_clean":"2024-03-20T09:33:45.552007+0000","last_became_active":"2024-03-20T09:19:54.524397+0000","last_became_peered":"2024-03-20T09:19:54.524397+0000","last_unstale":"2024-03-20T09:33:45.552007+0000","last_undegraded":"2024-03-20T09:33:45.552007+0000","last_fullsized":"2024-03-20T09:33:45.552007+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:33.560343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:33:33.560343+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:56.357887+0000","scrub_duration":184,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"924'11034","reported_seq":11291,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-20T09:33:48.065501+0000","last_change":"2024-03-20T09:33:40.727393+0000","last_active":"2024-03-20T09:33:48.065501+0000","last_peered":"2024-03-20T09:33:48.065501+0000","last_clean":"2024-03-20T09:33:48.065501+0000","last_became_active":"2024-03-20T09:31:07.428978+0000","last_became_peered":"2024-03-20T09:31:07.428978+0000","last_unstale":"2024-03-20T09:33:48.065501+0000","last_undegraded":"2024-03-20T09:33:48.065501+0000","last_fullsized":"2024-03-20T09:33:48.065501+0000","mapping_epoch":871,"log_start":"877'1000","ondisk_log_start":"877'1000","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"921'10653","last_scrub_stamp":"2024-03-20T09:33:40.727237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:33:40.727237+0000","objects_scrubbed":5157,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:35:05.691874+0000","scrub_duration":1252,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311296000,"num_objects":4750,"num_object_clones":0,"num_object_copies":14250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11034,"num_write_kb":505088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.c","version":"570'138","reported_seq":1914,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-20T09:33:46.587174+0000","last_change":"2024-03-20T09:33:33.082457+0000","last_active":"2024-03-20T09:33:46.587174+0000","last_peered":"2024-03-20T09:33:46.587174+0000","last_clean":"2024-03-20T09:33:46.587174+0000","last_became_active":"2024-03-20T09:13:27.087686+0000","last_became_peered":"2024-03-20T09:13:27.087686+0000","last_unstale":"2024-03-20T09:33:46.587174+0000","last_undegraded":"2024-03-20T09:33:46.587174+0000","last_fullsized":"2024-03-20T09:33:46.587174+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"570'138","last_scrub_stamp":"2024-03-20T09:33:33.082331+0000","last_deep_scrub":"570'138","last_deep_scrub_stamp":"2024-03-20T09:16:13.725035+0000","last_clean_scrub_stamp":"2024-03-20T09:33:33.082331+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:40.049713+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"924'11375","reported_seq":11568,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-20T09:33:49.760419+0000","last_change":"2024-03-20T09:32:27.484409+0000","last_active":"2024-03-20T09:33:49.760419+0000","last_peered":"2024-03-20T09:33:49.760419+0000","last_clean":"2024-03-20T09:33:49.760419+0000","last_became_active":"2024-03-20T09:31:07.424767+0000","last_became_peered":"2024-03-20T09:31:07.424767+0000","last_unstale":"2024-03-20T09:33:49.760419+0000","last_undegraded":"2024-03-20T09:33:49.760419+0000","last_fullsized":"2024-03-20T09:33:49.760419+0000","mapping_epoch":871,"log_start":"877'1300","ondisk_log_start":"877'1300","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"901'6848","last_scrub_stamp":"2024-03-20T09:32:27.484264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:27.484264+0000","objects_scrubbed":6805,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:33:56.967211+0000","scrub_duration":1598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315162624,"num_objects":4809,"num_object_clones":0,"num_object_copies":14427,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11375,"num_write_kb":517888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,11],"acting":[5,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"20'137","reported_seq":1477,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-03-20T09:33:45.551998+0000","last_change":"2024-03-20T09:33:11.988607+0000","last_active":"2024-03-20T09:33:45.551998+0000","last_peered":"2024-03-20T09:33:45.551998+0000","last_clean":"2024-03-20T09:33:45.551998+0000","last_became_active":"2024-03-20T09:19:54.502529+0000","last_became_peered":"2024-03-20T09:19:54.502529+0000","last_unstale":"2024-03-20T09:33:45.551998+0000","last_undegraded":"2024-03-20T09:33:45.551998+0000","last_fullsized":"2024-03-20T09:33:45.551998+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:11.988521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:33:11.988521+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:25.929908+0000","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"20'137","reported_seq":1767,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-20T09:33:46.579304+0000","last_change":"2024-03-20T09:32:27.000506+0000","last_active":"2024-03-20T09:33:46.579304+0000","last_peered":"2024-03-20T09:33:46.579304+0000","last_clean":"2024-03-20T09:33:46.579304+0000","last_became_active":"2024-03-20T09:10:34.096758+0000","last_became_peered":"2024-03-20T09:10:34.096758+0000","last_unstale":"2024-03-20T09:33:46.579304+0000","last_undegraded":"2024-03-20T09:33:46.579304+0000","last_fullsized":"2024-03-20T09:33:46.579304+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:32:27.000415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:32:27.000415+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"20'137","reported_seq":516,"reported_epoch":922,"state":"active+clean","last_fresh":"2024-03-20T09:33:40.798452+0000","last_change":"2024-03-20T09:33:11.373371+0000","last_active":"2024-03-20T09:33:40.798452+0000","last_peered":"2024-03-20T09:33:40.798452+0000","last_clean":"2024-03-20T09:33:40.798452+0000","last_became_active":"2024-03-20T09:31:07.425519+0000","last_became_peered":"2024-03-20T09:31:07.425519+0000","last_unstale":"2024-03-20T09:33:40.798452+0000","last_undegraded":"2024-03-20T09:33:40.798452+0000","last_fullsized":"2024-03-20T09:33:40.798452+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:11.373245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:33:11.373245+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:28.744893+0000","scrub_duration":229,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.10","version":"20'137","reported_seq":1322,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-20T09:33:46.579502+0000","last_change":"2024-03-20T09:33:38.307496+0000","last_active":"2024-03-20T09:33:46.579502+0000","last_peered":"2024-03-20T09:33:46.579502+0000","last_clean":"2024-03-20T09:33:46.579502+0000","last_became_active":"2024-03-20T09:28:22.584176+0000","last_became_peered":"2024-03-20T09:28:22.584176+0000","last_unstale":"2024-03-20T09:33:46.579502+0000","last_undegraded":"2024-03-20T09:33:46.579502+0000","last_fullsized":"2024-03-20T09:33:46.579502+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:38.307394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:33:38.307394+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:45.805742+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"20'137","reported_seq":1711,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-20T09:33:46.578184+0000","last_change":"2024-03-20T09:32:25.305425+0000","last_active":"2024-03-20T09:33:46.578184+0000","last_peered":"2024-03-20T09:33:46.578184+0000","last_clean":"2024-03-20T09:33:46.578184+0000","last_became_active":"2024-03-20T09:20:25.342982+0000","last_became_peered":"2024-03-20T09:20:25.342982+0000","last_unstale":"2024-03-20T09:33:46.578184+0000","last_undegraded":"2024-03-20T09:33:46.578184+0000","last_fullsized":"2024-03-20T09:33:46.578184+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:32:25.305287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:32:25.305287+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"20'137","reported_seq":437,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-20T09:33:46.581648+0000","last_change":"2024-03-20T09:32:42.323789+0000","last_active":"2024-03-20T09:33:46.581648+0000","last_peered":"2024-03-20T09:33:46.581648+0000","last_clean":"2024-03-20T09:33:46.581648+0000","last_became_active":"2024-03-20T09:31:07.424851+0000","last_became_peered":"2024-03-20T09:31:07.424851+0000","last_unstale":"2024-03-20T09:33:46.581648+0000","last_undegraded":"2024-03-20T09:33:46.581648+0000","last_fullsized":"2024-03-20T09:33:46.581648+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:32:42.323661+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:16:15.448920+0000","last_clean_scrub_stamp":"2024-03-20T09:32:42.323661+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:33:49.020839+0000","scrub_duration":144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.17","version":"20'137","reported_seq":1898,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-20T09:33:46.582438+0000","last_change":"2024-03-20T09:33:39.823553+0000","last_active":"2024-03-20T09:33:46.582438+0000","last_peered":"2024-03-20T09:33:46.582438+0000","last_clean":"2024-03-20T09:33:46.582438+0000","last_became_active":"2024-03-20T09:28:22.588303+0000","last_became_peered":"2024-03-20T09:28:22.588303+0000","last_unstale":"2024-03-20T09:33:46.582438+0000","last_undegraded":"2024-03-20T09:33:46.582438+0000","last_fullsized":"2024-03-20T09:33:46.582438+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:39.823418+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:23.640533+0000","last_clean_scrub_stamp":"2024-03-20T09:33:39.823418+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:58.632784+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"20'137","reported_seq":1939,"reported_epoch":924,"state":"active+clean","last_fresh":"2024-03-20T09:33:46.585282+0000","last_change":"2024-03-20T09:32:32.410744+0000","last_active":"2024-03-20T09:33:46.585282+0000","last_peered":"2024-03-20T09:33:46.585282+0000","last_clean":"2024-03-20T09:33:46.585282+0000","last_became_active":"2024-03-20T09:20:25.349082+0000","last_became_peered":"2024-03-20T09:20:25.349082+0000","last_unstale":"2024-03-20T09:33:46.585282+0000","last_undegraded":"2024-03-20T09:33:46.585282+0000","last_fullsized":"2024-03-20T09:33:46.585282+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:32:32.410595+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:30:54.106826+0000","last_clean_scrub_stamp":"2024-03-20T09:32:32.410595+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"20'137","reported_seq":1218,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-03-20T09:33:45.567004+0000","last_change":"2024-03-20T09:33:09.047464+0000","last_active":"2024-03-20T09:33:45.567004+0000","last_peered":"2024-03-20T09:33:45.567004+0000","last_clean":"2024-03-20T09:33:45.567004+0000","last_became_active":"2024-03-20T09:12:14.307581+0000","last_became_peered":"2024-03-20T09:12:14.307581+0000","last_unstale":"2024-03-20T09:33:45.567004+0000","last_undegraded":"2024-03-20T09:33:45.567004+0000","last_fullsized":"2024-03-20T09:33:45.567004+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:09.047372+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:23:41.251546+0000","last_clean_scrub_stamp":"2024-03-20T09:33:09.047372+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:37.240972+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"20'137","reported_seq":1476,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-03-20T09:33:45.551885+0000","last_change":"2024-03-20T09:32:38.833000+0000","last_active":"2024-03-20T09:33:45.551885+0000","last_peered":"2024-03-20T09:33:45.551885+0000","last_clean":"2024-03-20T09:33:45.551885+0000","last_became_active":"2024-03-20T09:19:54.502391+0000","last_became_peered":"2024-03-20T09:19:54.502391+0000","last_unstale":"2024-03-20T09:33:45.551885+0000","last_undegraded":"2024-03-20T09:33:45.551885+0000","last_fullsized":"2024-03-20T09:33:45.551885+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:32:38.832878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:32:38.832878+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:02.148297+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":5025169432,"num_objects":76679,"num_object_clones":0,"num_object_copies":230037,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76679,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":176539,"num_write_kb":8102913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15061692416,"data_stored":15061680200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160839,"ondisk_log_size":160839,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4248,"ondisk_log_size":4248,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860869,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2426664,"kb_used_data":1575008,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":91318488,"statfs":{"total":95995035648,"available":93510131712,"internally_reserved":0,"allocated":1612808192,"data_stored":1612518704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":872085956},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":3697966841895,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":80,"seq":343597384147,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1921800,"kb_used_data":1533816,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":91823352,"statfs":{"total":95995035648,"available":94027112448,"internally_reserved":0,"allocated":1570627584,"data_stored":1570329224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":397277632},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":70,"seq":300647711191,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1409096,"kb_used_data":918392,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":92336056,"statfs":{"total":95995035648,"available":94552121344,"internally_reserved":0,"allocated":940433408,"data_stored":940135048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":502462918},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":60,"seq":257698038235,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1419712,"kb_used_data":622196,"kb_used_omap":1,"kb_used_meta":797502,"kb_avail":92325440,"statfs":{"total":95995035648,"available":94541250560,"internally_reserved":0,"allocated":637128704,"data_stored":636831228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":816642506},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332576,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2219052,"kb_used_data":1556196,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":91526100,"statfs":{"total":95995035648,"available":93722726400,"internally_reserved":0,"allocated":1593544704,"data_stored":1593251144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":678754758},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575355,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1762692,"kb_used_data":923452,"kb_used_omap":1,"kb_used_meta":839166,"kb_avail":91982460,"statfs":{"total":95995035648,"available":94190039040,"internally_reserved":0,"allocated":945614848,"data_stored":945312416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":859306438},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738873,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2804536,"kb_used_data":1880688,"kb_used_omap":1,"kb_used_meta":923838,"kb_avail":90940616,"statfs":{"total":95995035648,"available":93123190784,"internally_reserved":0,"allocated":1925824512,"data_stored":1925526516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":946010565},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411827,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2189896,"kb_used_data":1536768,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":91555256,"statfs":{"total":95995035648,"available":93752582144,"internally_reserved":0,"allocated":1573650432,"data_stored":1573348520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150192,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2978880,"kb_used_data":2077176,"kb_used_omap":1,"kb_used_meta":901630,"kb_avail":90766272,"statfs":{"total":95995035648,"available":92944662528,"internally_reserved":0,"allocated":2127028224,"data_stored":2126729864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":923269575},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953966,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1947496,"kb_used_data":1266592,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":91797656,"statfs":{"total":95995035648,"available":94000799744,"internally_reserved":0,"allocated":1296990208,"data_stored":1296700720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":443,"seq":1902670512408,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1556360,"kb_used_data":908608,"kb_used_omap":181,"kb_used_meta":647562,"kb_avail":92188792,"statfs":{"total":95995035648,"available":94401323008,"internally_reserved":0,"allocated":930414592,"data_stored":930112680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":186131,"internal_metadata":663103725},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1911951360,"data_stored":1911947288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":935006208,"data_stored":935002136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1561460736,"data_stored":1561460736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2118516736,"data_stored":2118516736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1285881856,"data_stored":1285881856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":923729920,"data_stored":923729920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1579880448,"data_stored":1579876376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":635764736,"data_stored":635764736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":939196416,"data_stored":939196416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1561657344,"data_stored":1561657344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1608646656,"data_stored":1608646656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:33:56.091 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:33:56.092 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:33:56.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:56.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:33:56.200 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:33:56.230 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:56.259 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:33:56.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:56.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:33:56.365 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:33:56.401 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:56.427 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:33:56.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:56.430 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:33:56.533 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:33:56.548 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:33:56.560 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:56.582 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:33:56.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:56.587 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:33:56.659 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:33:56.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:56 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3735652933' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:33:56.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:56 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:33:56.687 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:33:56.723 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:56.855 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:33:56.858 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:56 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3735652933' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:33:56.858 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:56 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:33:56.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:56.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:33:56.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:56 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3735652933' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:33:56.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:56 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:33:56.964 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:33:57.000 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:57.021 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:33:57.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:57.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:33:57.125 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:33:57.157 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:57.181 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:33:57.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:57.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:33:57.287 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:33:57.362 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:57.636 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:57 smithi026 ceph-mon[68309]: pgmap v2245: 47 pgs: 47 active+clean; 4.4 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 808 op/s 2024-03-20T09:33:57.636 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:57 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:33:57.636 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:57 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:33:57.636 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:57 smithi026 ceph-mon[68309]: osdmap e926: 12 total, 12 up, 11 in 2024-03-20T09:33:57.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:57 smithi115 ceph-mon[72476]: pgmap v2245: 47 pgs: 47 active+clean; 4.4 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 808 op/s 2024-03-20T09:33:57.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:57 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:33:57.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:57 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:33:57.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:57 smithi115 ceph-mon[72476]: osdmap e926: 12 total, 12 up, 11 in 2024-03-20T09:33:57.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:57 smithi122 ceph-mon[73495]: pgmap v2245: 47 pgs: 47 active+clean; 4.4 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 808 op/s 2024-03-20T09:33:57.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:57 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:33:57.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:57 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:33:57.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:57 smithi122 ceph-mon[73495]: osdmap e926: 12 total, 12 up, 11 in 2024-03-20T09:33:58.862 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:33:58.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:58.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:33:58.971 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:33:59.037 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:33:58 smithi026 ceph-mon[68309]: pgmap v2247: 47 pgs: 47 active+clean; 4.3 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 875 op/s 2024-03-20T09:33:59.044 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:33:59.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:33:58 smithi115 ceph-mon[72476]: pgmap v2247: 47 pgs: 47 active+clean; 4.3 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 875 op/s 2024-03-20T09:33:59.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:33:58 smithi122 ceph-mon[73495]: pgmap v2247: 47 pgs: 47 active+clean; 4.3 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 875 op/s 2024-03-20T09:33:59.928 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:33:59.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:33:59.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:34:00.033 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:34:00.069 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:00.095 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:34:00.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:00.099 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:34:00.200 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:34:00.240 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:00.267 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:34:00.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:00.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:34:00.371 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:34:00.401 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:00.426 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:34:00.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:00.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:34:00.520 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:34:00.520 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:34:00.531 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:34:00.561 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:00.593 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:34:00.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:00.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:34:00.697 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:34:00.785 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:00.983 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:34:00.999 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:34:01.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:01.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:34:01.105 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:34:01.176 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:01.487 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:34:01.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:01.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:34:01.592 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:34:01.620 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:01.645 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:34:01.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:01.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:34:01.750 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:34:01.777 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:01.802 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:34:01.804 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:01 smithi122 ceph-mon[73495]: pgmap v2248: 47 pgs: 47 active+clean; 4.2 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 722 op/s 2024-03-20T09:34:01.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:01.805 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:34:01.838 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:34:01.845 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:34:01.886 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:34:01.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:01 smithi115 ceph-mon[72476]: pgmap v2248: 47 pgs: 47 active+clean; 4.2 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 722 op/s 2024-03-20T09:34:01.906 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:34:01.921 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:01 smithi026 ceph-mon[68309]: pgmap v2248: 47 pgs: 47 active+clean; 4.2 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 722 op/s 2024-03-20T09:34:01.932 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:34:01.933 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:01.953 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:34:01.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:01.956 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:34:01.976 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:34:02.032 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:34:02.057 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:34:02.085 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:02.087 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:34:02.107 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:34:02.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:02.112 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:34:02.127 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:34:02.167 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:34:02.213 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:34:02.236 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:34:02.302 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:02.302 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:34:02.336 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:34:02.389 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:34:02.398 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:34:02.398 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:34:02.398 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:34:02.399 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:34:02.399 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:34:02.399 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:34:02.399 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:34:02.399 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:34:02.399 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:34:02.399 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:34:02.399 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:34:02.399 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:34:02.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:02 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1229081679' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:34:02.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:02 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1229081679' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:34:02.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:02 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1229081679' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:34:03.482 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:34:03.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:03.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:34:03.601 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:34:03.634 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:03.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:03 smithi115 ceph-mon[72476]: pgmap v2249: 47 pgs: 47 active+clean; 4.1 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 805 op/s 2024-03-20T09:34:03.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:03 smithi122 ceph-mon[73495]: pgmap v2249: 47 pgs: 47 active+clean; 4.1 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 805 op/s 2024-03-20T09:34:03.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:03 smithi026 ceph-mon[68309]: pgmap v2249: 47 pgs: 47 active+clean; 4.1 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 805 op/s 2024-03-20T09:34:04.378 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-20T09:34:04.611 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:34:04.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:04.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:34:04.720 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:34:04.766 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:04.787 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:34:04.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:04.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:34:04.891 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:34:04.965 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:05.121 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:34:05.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:05.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:34:05.240 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:34:05.278 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:05.313 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:34:05.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:05.317 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:34:05.418 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:34:05.447 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:05.476 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:34:05.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:05.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:34:05.482 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:34:05.537 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:34:05.538 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:34:05.542 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:34:05.580 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:34:05.607 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:05.628 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:34:05.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:05.631 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:34:05.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:05 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:34:05.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:05 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:34:05.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:05 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:34:05.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:05 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:34:05.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:05 smithi026 ceph-mon[68309]: pgmap v2250: 47 pgs: 47 active+clean; 4.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 790 op/s 2024-03-20T09:34:05.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:05 smithi026 ceph-mon[68309]: osdmap e927: 12 total, 12 up, 11 in 2024-03-20T09:34:05.732 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:34:05.761 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:05.785 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:34:05.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:05.793 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:34:05.895 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:34:05.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:05 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:34:05.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:05 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:34:05.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:05 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:34:05.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:05 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:34:05.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:05 smithi115 ceph-mon[72476]: pgmap v2250: 47 pgs: 47 active+clean; 4.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 790 op/s 2024-03-20T09:34:05.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:05 smithi115 ceph-mon[72476]: osdmap e927: 12 total, 12 up, 11 in 2024-03-20T09:34:05.921 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:05 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:34:05.921 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:05 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:34:05.921 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:05 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:34:05.921 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:05 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:34:05.921 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:05 smithi122 ceph-mon[73495]: pgmap v2250: 47 pgs: 47 active+clean; 4.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 790 op/s 2024-03-20T09:34:05.921 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:05 smithi122 ceph-mon[73495]: osdmap e927: 12 total, 12 up, 11 in 2024-03-20T09:34:05.923 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:05.944 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:34:05.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:05.948 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:34:06.049 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:34:06.078 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:06.100 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:34:06.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:06.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:34:06.209 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:34:06.248 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:06.279 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:34:06.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:06.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:34:06.384 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:34:06.415 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:06.439 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:34:06.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:06.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:34:06.543 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:34:06.637 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:06.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:06 smithi115 ceph-mon[72476]: from='client.26764 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:34:06.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:06 smithi122 ceph-mon[73495]: from='client.26764 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:34:06.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:06 smithi026 ceph-mon[68309]: from='client.26764 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:34:07.520 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:34:07.852 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:34:07.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:07.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:34:07.857 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:07 smithi026 ceph-mon[68309]: pgmap v2252: 47 pgs: 47 active+clean; 3.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 805 op/s 2024-03-20T09:34:07.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:07 smithi115 ceph-mon[72476]: pgmap v2252: 47 pgs: 47 active+clean; 3.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 805 op/s 2024-03-20T09:34:07.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:07 smithi122 ceph-mon[73495]: pgmap v2252: 47 pgs: 47 active+clean; 3.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 805 op/s 2024-03-20T09:34:07.955 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:34:07.992 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:08.017 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:34:08.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:08.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:34:08.124 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:34:08.161 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:08.189 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:34:08.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:08.195 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:34:08.296 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:34:08.368 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:08.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:08 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:34:08.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:08 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:34:08.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:08 smithi122 ceph-mon[73495]: osdmap e928: 12 total, 12 up, 11 in 2024-03-20T09:34:08.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:08 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:34:08.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:08 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:34:08.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:08 smithi026 ceph-mon[68309]: osdmap e928: 12 total, 12 up, 11 in 2024-03-20T09:34:09.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:08 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:34:09.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:08 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:34:09.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:08 smithi115 ceph-mon[72476]: osdmap e928: 12 total, 12 up, 11 in 2024-03-20T09:34:09.351 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":2251,"stamp":"2024-03-20T09:34:04.625253+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4311417400,"num_objects":65781,"num_object_clones":0,"num_object_copies":197343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65781,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187439,"num_write_kb":8102913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":165087,"ondisk_log_size":165087,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":20555848,"kb_used_data":12726828,"kb_used_omap":197,"kb_used_meta":7828538,"kb_avail":1010640824,"statfs":{"total":1055945392128,"available":1034896203776,"internally_reserved":0,"allocated":13032271872,"data_stored":13029367044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202077,"internal_metadata":8016423587},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-368967680,"num_objects":-5630,"num_object_clones":0,"num_object_copies":-16890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5630,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.157683"},"pg_stats":[{"pgid":"1.1e","version":"20'137","reported_seq":1293,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:33:56.645808+0000","last_change":"2024-03-20T09:33:12.168422+0000","last_active":"2024-03-20T09:33:56.645808+0000","last_peered":"2024-03-20T09:33:56.645808+0000","last_clean":"2024-03-20T09:33:56.645808+0000","last_became_active":"2024-03-20T09:13:20.071655+0000","last_became_peered":"2024-03-20T09:13:20.071655+0000","last_unstale":"2024-03-20T09:33:56.645808+0000","last_undegraded":"2024-03-20T09:33:56.645808+0000","last_fullsized":"2024-03-20T09:33:56.645808+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:12.168339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:33:12.168339+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:33.973327+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'137","reported_seq":371,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:33:56.646018+0000","last_change":"2024-03-20T09:32:45.273878+0000","last_active":"2024-03-20T09:33:56.646018+0000","last_peered":"2024-03-20T09:33:56.646018+0000","last_clean":"2024-03-20T09:33:56.646018+0000","last_became_active":"2024-03-20T09:31:07.425574+0000","last_became_peered":"2024-03-20T09:31:07.425574+0000","last_unstale":"2024-03-20T09:33:56.646018+0000","last_undegraded":"2024-03-20T09:33:56.646018+0000","last_fullsized":"2024-03-20T09:33:56.646018+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:32:45.273763+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:29:02.585165+0000","last_clean_scrub_stamp":"2024-03-20T09:32:45.273763+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:01.647867+0000","scrub_duration":162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2],"acting":[11,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"20'137","reported_seq":1956,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:33:56.648193+0000","last_change":"2024-03-20T09:32:23.261615+0000","last_active":"2024-03-20T09:33:56.648193+0000","last_peered":"2024-03-20T09:33:56.648193+0000","last_clean":"2024-03-20T09:33:56.648193+0000","last_became_active":"2024-03-20T09:25:35.482959+0000","last_became_peered":"2024-03-20T09:25:35.482959+0000","last_unstale":"2024-03-20T09:33:56.648193+0000","last_undegraded":"2024-03-20T09:33:56.648193+0000","last_fullsized":"2024-03-20T09:33:56.648193+0000","mapping_epoch":744,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":745,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:32:23.261521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:32:23.261521+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:01.600329+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,1],"acting":[8,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.19","version":"20'137","reported_seq":1137,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:33:56.637783+0000","last_change":"2024-03-20T09:32:28.348352+0000","last_active":"2024-03-20T09:33:56.637783+0000","last_peered":"2024-03-20T09:33:56.637783+0000","last_clean":"2024-03-20T09:33:56.637783+0000","last_became_active":"2024-03-20T09:28:22.591294+0000","last_became_peered":"2024-03-20T09:28:22.591294+0000","last_unstale":"2024-03-20T09:33:56.637783+0000","last_undegraded":"2024-03-20T09:33:56.637783+0000","last_fullsized":"2024-03-20T09:33:56.637783+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:32:28.348215+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:30:56.201554+0000","last_clean_scrub_stamp":"2024-03-20T09:32:28.348215+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.18","version":"20'137","reported_seq":1439,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:33:56.643876+0000","last_change":"2024-03-20T09:33:40.415955+0000","last_active":"2024-03-20T09:33:56.643876+0000","last_peered":"2024-03-20T09:33:56.643876+0000","last_clean":"2024-03-20T09:33:56.643876+0000","last_became_active":"2024-03-20T09:10:35.277481+0000","last_became_peered":"2024-03-20T09:10:35.277481+0000","last_unstale":"2024-03-20T09:33:56.643876+0000","last_undegraded":"2024-03-20T09:33:56.643876+0000","last_fullsized":"2024-03-20T09:33:56.643876+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:40.415846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:33:40.415846+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:52.941527+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1b","version":"20'137","reported_seq":1793,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:33:56.648145+0000","last_change":"2024-03-20T09:33:13.685851+0000","last_active":"2024-03-20T09:33:56.648145+0000","last_peered":"2024-03-20T09:33:56.648145+0000","last_clean":"2024-03-20T09:33:56.648145+0000","last_became_active":"2024-03-20T09:24:16.445287+0000","last_became_peered":"2024-03-20T09:24:16.445287+0000","last_unstale":"2024-03-20T09:33:56.648145+0000","last_undegraded":"2024-03-20T09:33:56.648145+0000","last_fullsized":"2024-03-20T09:33:56.648145+0000","mapping_epoch":718,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:13.685707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:33:13.685707+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:14.072191+0000","scrub_duration":188,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.e","version":"926'11844","reported_seq":12046,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:34:03.060337+0000","last_change":"2024-03-20T09:32:36.556340+0000","last_active":"2024-03-20T09:34:03.060337+0000","last_peered":"2024-03-20T09:34:03.060337+0000","last_clean":"2024-03-20T09:34:03.060337+0000","last_became_active":"2024-03-20T09:31:07.427099+0000","last_became_peered":"2024-03-20T09:31:07.427099+0000","last_unstale":"2024-03-20T09:34:03.060337+0000","last_undegraded":"2024-03-20T09:34:03.060337+0000","last_fullsized":"2024-03-20T09:34:03.060337+0000","mapping_epoch":871,"log_start":"878'1800","ondisk_log_start":"878'1800","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"904'7287","last_scrub_stamp":"2024-03-20T09:32:36.556054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:36.556054+0000","objects_scrubbed":7222,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:05.335073+0000","scrub_duration":2205,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261488640,"num_objects":3990,"num_object_clones":0,"num_object_copies":11970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11844,"num_write_kb":506688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,5],"acting":[2,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"20'137","reported_seq":1431,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:33:56.643982+0000","last_change":"2024-03-20T09:32:28.004445+0000","last_active":"2024-03-20T09:33:56.643982+0000","last_peered":"2024-03-20T09:33:56.643982+0000","last_clean":"2024-03-20T09:33:56.643982+0000","last_became_active":"2024-03-20T09:10:35.215772+0000","last_became_peered":"2024-03-20T09:10:35.215772+0000","last_unstale":"2024-03-20T09:33:56.643982+0000","last_undegraded":"2024-03-20T09:33:56.643982+0000","last_fullsized":"2024-03-20T09:33:56.643982+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:32:28.004327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:32:28.004327+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:33:57.313212+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"926'11712","reported_seq":11914,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:34:01.675121+0000","last_change":"2024-03-20T09:32:38.283331+0000","last_active":"2024-03-20T09:34:01.675121+0000","last_peered":"2024-03-20T09:34:01.675121+0000","last_clean":"2024-03-20T09:34:01.675121+0000","last_became_active":"2024-03-20T09:31:07.534696+0000","last_became_peered":"2024-03-20T09:31:07.534696+0000","last_unstale":"2024-03-20T09:34:01.675121+0000","last_undegraded":"2024-03-20T09:34:01.675121+0000","last_fullsized":"2024-03-20T09:34:01.675121+0000","mapping_epoch":871,"log_start":"878'1700","ondisk_log_start":"878'1700","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"905'7295","last_scrub_stamp":"2024-03-20T09:32:38.283112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:38.283112+0000","objects_scrubbed":7235,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:06.368167+0000","scrub_duration":2079,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269352960,"num_objects":4110,"num_object_clones":0,"num_object_copies":12330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11712,"num_write_kb":506304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"20'137","reported_seq":2023,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:33:56.646154+0000","last_change":"2024-03-20T09:33:14.407075+0000","last_active":"2024-03-20T09:33:56.646154+0000","last_peered":"2024-03-20T09:33:56.646154+0000","last_clean":"2024-03-20T09:33:56.646154+0000","last_became_active":"2024-03-20T09:20:25.341750+0000","last_became_peered":"2024-03-20T09:20:25.341750+0000","last_unstale":"2024-03-20T09:33:56.646154+0000","last_undegraded":"2024-03-20T09:33:56.646154+0000","last_fullsized":"2024-03-20T09:33:56.646154+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:14.406974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:33:14.406974+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:29.321937+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,2],"acting":[9,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.c","version":"926'11672","reported_seq":11873,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:34:01.275410+0000","last_change":"2024-03-20T09:32:33.031806+0000","last_active":"2024-03-20T09:34:01.275410+0000","last_peered":"2024-03-20T09:34:01.275410+0000","last_clean":"2024-03-20T09:34:01.275410+0000","last_became_active":"2024-03-20T09:31:07.426984+0000","last_became_peered":"2024-03-20T09:31:07.426984+0000","last_unstale":"2024-03-20T09:34:01.275410+0000","last_undegraded":"2024-03-20T09:34:01.275410+0000","last_fullsized":"2024-03-20T09:34:01.275410+0000","mapping_epoch":871,"log_start":"878'1600","ondisk_log_start":"878'1600","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"904'7023","last_scrub_stamp":"2024-03-20T09:32:33.031701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:33.031701+0000","objects_scrubbed":6989,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:03.439628+0000","scrub_duration":1492,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268435456,"num_objects":4096,"num_object_clones":0,"num_object_copies":12288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11672,"num_write_kb":504576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,11],"acting":[6,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"20'137","reported_seq":712,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:33:56.646254+0000","last_change":"2024-03-20T09:33:10.442796+0000","last_active":"2024-03-20T09:33:56.646254+0000","last_peered":"2024-03-20T09:33:56.646254+0000","last_clean":"2024-03-20T09:33:56.646254+0000","last_became_active":"2024-03-20T09:24:16.444232+0000","last_became_peered":"2024-03-20T09:24:16.444232+0000","last_unstale":"2024-03-20T09:33:56.646254+0000","last_undegraded":"2024-03-20T09:33:56.646254+0000","last_fullsized":"2024-03-20T09:33:56.646254+0000","mapping_epoch":718,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:10.442719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:33:10.442719+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:26.643584+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.2","version":"926'11713","reported_seq":11907,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:34:00.299326+0000","last_change":"2024-03-20T09:32:21.786210+0000","last_active":"2024-03-20T09:34:00.299326+0000","last_peered":"2024-03-20T09:34:00.299326+0000","last_clean":"2024-03-20T09:34:00.299326+0000","last_became_active":"2024-03-20T09:31:07.422232+0000","last_became_peered":"2024-03-20T09:31:07.422232+0000","last_unstale":"2024-03-20T09:34:00.299326+0000","last_undegraded":"2024-03-20T09:34:00.299326+0000","last_fullsized":"2024-03-20T09:34:00.299326+0000","mapping_epoch":871,"log_start":"878'1700","ondisk_log_start":"878'1700","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'6455","last_scrub_stamp":"2024-03-20T09:32:21.785983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:21.785983+0000","objects_scrubbed":6398,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1934,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276627456,"num_objects":4221,"num_object_clones":0,"num_object_copies":12663,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11713,"num_write_kb":509888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"20'137","reported_seq":1961,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:33:56.645851+0000","last_change":"2024-03-20T09:33:10.120697+0000","last_active":"2024-03-20T09:33:56.645851+0000","last_peered":"2024-03-20T09:33:56.645851+0000","last_clean":"2024-03-20T09:33:56.645851+0000","last_became_active":"2024-03-20T09:12:14.306384+0000","last_became_peered":"2024-03-20T09:12:14.306384+0000","last_unstale":"2024-03-20T09:33:56.645851+0000","last_undegraded":"2024-03-20T09:33:56.645851+0000","last_fullsized":"2024-03-20T09:33:56.645851+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:10.120609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:33:10.120609+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:37.759793+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.3","version":"926'11680","reported_seq":11940,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:34:00.291519+0000","last_change":"2024-03-20T09:33:29.905371+0000","last_active":"2024-03-20T09:34:00.291519+0000","last_peered":"2024-03-20T09:34:00.291519+0000","last_clean":"2024-03-20T09:34:00.291519+0000","last_became_active":"2024-03-20T09:31:07.422509+0000","last_became_peered":"2024-03-20T09:31:07.422509+0000","last_unstale":"2024-03-20T09:34:00.291519+0000","last_undegraded":"2024-03-20T09:34:00.291519+0000","last_fullsized":"2024-03-20T09:34:00.291519+0000","mapping_epoch":871,"log_start":"878'1600","ondisk_log_start":"878'1600","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"920'10157","last_scrub_stamp":"2024-03-20T09:33:29.905190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:33:29.905190+0000","objects_scrubbed":5720,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:49.933850+0000","scrub_duration":1973,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271712256,"num_objects":4146,"num_object_clones":0,"num_object_copies":12438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11680,"num_write_kb":506432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"20'137","reported_seq":608,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:33:56.640962+0000","last_change":"2024-03-20T09:32:37.603204+0000","last_active":"2024-03-20T09:33:56.640962+0000","last_peered":"2024-03-20T09:33:56.640962+0000","last_clean":"2024-03-20T09:33:56.640962+0000","last_became_active":"2024-03-20T09:28:22.587095+0000","last_became_peered":"2024-03-20T09:28:22.587095+0000","last_unstale":"2024-03-20T09:33:56.640962+0000","last_undegraded":"2024-03-20T09:33:56.640962+0000","last_fullsized":"2024-03-20T09:33:56.640962+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:32:37.603082+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:19:03.805185+0000","last_clean_scrub_stamp":"2024-03-20T09:32:37.603082+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:05.647425+0000","scrub_duration":191,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,11],"acting":[2,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.0","version":"926'11502","reported_seq":11696,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:33:59.775721+0000","last_change":"2024-03-20T09:32:26.456393+0000","last_active":"2024-03-20T09:33:59.775721+0000","last_peered":"2024-03-20T09:33:59.775721+0000","last_clean":"2024-03-20T09:33:59.775721+0000","last_became_active":"2024-03-20T09:31:07.426307+0000","last_became_peered":"2024-03-20T09:31:07.426307+0000","last_unstale":"2024-03-20T09:33:59.775721+0000","last_undegraded":"2024-03-20T09:33:59.775721+0000","last_fullsized":"2024-03-20T09:33:59.775721+0000","mapping_epoch":871,"log_start":"877'1500","ondisk_log_start":"877'1500","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"901'6561","last_scrub_stamp":"2024-03-20T09:32:26.456235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:26.456235+0000","objects_scrubbed":6499,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:03.324982+0000","scrub_duration":1520,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272760832,"num_objects":4162,"num_object_clones":0,"num_object_copies":12486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11502,"num_write_kb":501248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"20'137","reported_seq":1827,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:33:56.646300+0000","last_change":"2024-03-20T09:33:13.369995+0000","last_active":"2024-03-20T09:33:56.646300+0000","last_peered":"2024-03-20T09:33:56.646300+0000","last_clean":"2024-03-20T09:33:56.646300+0000","last_became_active":"2024-03-20T09:28:22.586751+0000","last_became_peered":"2024-03-20T09:28:22.586751+0000","last_unstale":"2024-03-20T09:33:56.646300+0000","last_undegraded":"2024-03-20T09:33:56.646300+0000","last_fullsized":"2024-03-20T09:33:56.646300+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:13.369863+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:24:31.549603+0000","last_clean_scrub_stamp":"2024-03-20T09:33:13.369863+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:39.364154+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.5","version":"926'11558","reported_seq":11752,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:34:00.495813+0000","last_change":"2024-03-20T09:32:23.538659+0000","last_active":"2024-03-20T09:34:00.495813+0000","last_peered":"2024-03-20T09:34:00.495813+0000","last_clean":"2024-03-20T09:34:00.495813+0000","last_became_active":"2024-03-20T09:31:07.427044+0000","last_became_peered":"2024-03-20T09:31:07.427044+0000","last_unstale":"2024-03-20T09:34:00.495813+0000","last_undegraded":"2024-03-20T09:34:00.495813+0000","last_fullsized":"2024-03-20T09:34:00.495813+0000","mapping_epoch":871,"log_start":"877'1500","ondisk_log_start":"877'1500","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"900'6405","last_scrub_stamp":"2024-03-20T09:32:23.538400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:23.538400+0000","objects_scrubbed":6353,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1492,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278265856,"num_objects":4246,"num_object_clones":0,"num_object_copies":12738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11558,"num_write_kb":505728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,0],"acting":[11,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.3","version":"20'137","reported_seq":1027,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:33:56.646478+0000","last_change":"2024-03-20T09:33:26.640631+0000","last_active":"2024-03-20T09:33:56.646478+0000","last_peered":"2024-03-20T09:33:56.646478+0000","last_clean":"2024-03-20T09:33:56.646478+0000","last_became_active":"2024-03-20T09:31:07.424588+0000","last_became_peered":"2024-03-20T09:31:07.424588+0000","last_unstale":"2024-03-20T09:33:56.646478+0000","last_undegraded":"2024-03-20T09:33:56.646478+0000","last_fullsized":"2024-03-20T09:33:56.646478+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:26.640505+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:30:47.812125+0000","last_clean_scrub_stamp":"2024-03-20T09:33:26.640505+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:56.230688+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"926'11822","reported_seq":12028,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:34:01.717111+0000","last_change":"2024-03-20T09:32:43.005957+0000","last_active":"2024-03-20T09:34:01.717111+0000","last_peered":"2024-03-20T09:34:01.717111+0000","last_clean":"2024-03-20T09:34:01.717111+0000","last_became_active":"2024-03-20T09:31:07.535053+0000","last_became_peered":"2024-03-20T09:31:07.535053+0000","last_unstale":"2024-03-20T09:34:01.717111+0000","last_undegraded":"2024-03-20T09:34:01.717111+0000","last_fullsized":"2024-03-20T09:34:01.717111+0000","mapping_epoch":871,"log_start":"878'1800","ondisk_log_start":"878'1800","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"907'7583","last_scrub_stamp":"2024-03-20T09:32:43.005812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:43.005812+0000","objects_scrubbed":7513,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:07.560498+0000","scrub_duration":2762,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263061504,"num_objects":4014,"num_object_clones":0,"num_object_copies":12042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11822,"num_write_kb":506752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":1932,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:33:56.645721+0000","last_change":"2024-03-20T09:33:19.153580+0000","last_active":"2024-03-20T09:33:56.645721+0000","last_peered":"2024-03-20T09:33:56.645721+0000","last_clean":"2024-03-20T09:33:56.645721+0000","last_became_active":"2024-03-20T09:12:14.305989+0000","last_became_peered":"2024-03-20T09:12:14.305989+0000","last_unstale":"2024-03-20T09:33:56.645721+0000","last_undegraded":"2024-03-20T09:33:56.645721+0000","last_fullsized":"2024-03-20T09:33:56.645721+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:19.153464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:33:19.153464+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:32.868996+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.4","version":"926'11800","reported_seq":11996,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:34:01.708059+0000","last_change":"2024-03-20T09:33:16.169687+0000","last_active":"2024-03-20T09:34:01.708059+0000","last_peered":"2024-03-20T09:34:01.708059+0000","last_clean":"2024-03-20T09:34:01.708059+0000","last_became_active":"2024-03-20T09:31:07.424820+0000","last_became_peered":"2024-03-20T09:31:07.424820+0000","last_unstale":"2024-03-20T09:34:01.708059+0000","last_undegraded":"2024-03-20T09:34:01.708059+0000","last_fullsized":"2024-03-20T09:34:01.708059+0000","mapping_epoch":871,"log_start":"878'1700","ondisk_log_start":"878'1700","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"917'9492","last_scrub_stamp":"2024-03-20T09:33:16.169522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:33:16.169522+0000","objects_scrubbed":6563,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:40.296707+0000","scrub_duration":1975,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276168704,"num_objects":4214,"num_object_clones":0,"num_object_copies":12642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11800,"num_write_kb":512448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"20'137","reported_seq":2197,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:33:56.637683+0000","last_change":"2024-03-20T09:33:33.249232+0000","last_active":"2024-03-20T09:33:56.637683+0000","last_peered":"2024-03-20T09:33:56.637683+0000","last_clean":"2024-03-20T09:33:56.637683+0000","last_became_active":"2024-03-20T09:19:54.492879+0000","last_became_peered":"2024-03-20T09:19:54.492879+0000","last_unstale":"2024-03-20T09:33:56.637683+0000","last_undegraded":"2024-03-20T09:33:56.637683+0000","last_fullsized":"2024-03-20T09:33:56.637683+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:33.249098+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:19.513511+0000","last_clean_scrub_stamp":"2024-03-20T09:33:33.249098+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:54.587251+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"926'11478","reported_seq":11674,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:34:01.265216+0000","last_change":"2024-03-20T09:32:28.146865+0000","last_active":"2024-03-20T09:34:01.265216+0000","last_peered":"2024-03-20T09:34:01.265216+0000","last_clean":"2024-03-20T09:34:01.265216+0000","last_became_active":"2024-03-20T09:31:07.534591+0000","last_became_peered":"2024-03-20T09:31:07.534591+0000","last_unstale":"2024-03-20T09:34:01.265216+0000","last_undegraded":"2024-03-20T09:34:01.265216+0000","last_fullsized":"2024-03-20T09:34:01.265216+0000","mapping_epoch":871,"log_start":"877'1400","ondisk_log_start":"877'1400","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"902'6591","last_scrub_stamp":"2024-03-20T09:32:28.146586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:28.146586+0000","objects_scrubbed":6548,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:03.439818+0000","scrub_duration":1556,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256770048,"num_objects":3918,"num_object_clones":0,"num_object_copies":11754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11478,"num_write_kb":492672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,3],"acting":[6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"20'137","reported_seq":1373,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:33:56.646378+0000","last_change":"2024-03-20T09:33:14.457333+0000","last_active":"2024-03-20T09:33:56.646378+0000","last_peered":"2024-03-20T09:33:56.646378+0000","last_clean":"2024-03-20T09:33:56.646378+0000","last_became_active":"2024-03-20T09:15:34.225022+0000","last_became_peered":"2024-03-20T09:15:34.225022+0000","last_unstale":"2024-03-20T09:33:56.646378+0000","last_undegraded":"2024-03-20T09:33:56.646378+0000","last_fullsized":"2024-03-20T09:33:56.646378+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:14.457198+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:20:12.169658+0000","last_clean_scrub_stamp":"2024-03-20T09:33:14.457198+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:16.366898+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,5],"acting":[6,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"926'11690","reported_seq":11892,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:34:00.591138+0000","last_change":"2024-03-20T09:32:37.563227+0000","last_active":"2024-03-20T09:34:00.591138+0000","last_peered":"2024-03-20T09:34:00.591138+0000","last_clean":"2024-03-20T09:34:00.591138+0000","last_became_active":"2024-03-20T09:31:07.421909+0000","last_became_peered":"2024-03-20T09:31:07.421909+0000","last_unstale":"2024-03-20T09:34:00.591138+0000","last_undegraded":"2024-03-20T09:34:00.591138+0000","last_fullsized":"2024-03-20T09:34:00.591138+0000","mapping_epoch":871,"log_start":"878'1600","ondisk_log_start":"878'1600","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"905'7297","last_scrub_stamp":"2024-03-20T09:32:37.563105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:37.563105+0000","objects_scrubbed":7240,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:04.740357+0000","scrub_duration":1719,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270925824,"num_objects":4134,"num_object_clones":0,"num_object_copies":12402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11690,"num_write_kb":506368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"20'137","reported_seq":1912,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:33:56.646751+0000","last_change":"2024-03-20T09:33:13.684716+0000","last_active":"2024-03-20T09:33:56.646751+0000","last_peered":"2024-03-20T09:33:56.646751+0000","last_clean":"2024-03-20T09:33:56.646751+0000","last_became_active":"2024-03-20T09:28:22.587861+0000","last_became_peered":"2024-03-20T09:28:22.587861+0000","last_unstale":"2024-03-20T09:33:56.646751+0000","last_undegraded":"2024-03-20T09:33:56.646751+0000","last_fullsized":"2024-03-20T09:33:56.646751+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:13.684628+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:14:10.560890+0000","last_clean_scrub_stamp":"2024-03-20T09:33:13.684628+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:35.420402+0000","scrub_duration":213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.d","version":"926'11717","reported_seq":11915,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:34:01.248559+0000","last_change":"2024-03-20T09:32:27.343018+0000","last_active":"2024-03-20T09:34:01.248559+0000","last_peered":"2024-03-20T09:34:01.248559+0000","last_clean":"2024-03-20T09:34:01.248559+0000","last_became_active":"2024-03-20T09:31:07.425062+0000","last_became_peered":"2024-03-20T09:31:07.425062+0000","last_unstale":"2024-03-20T09:34:01.248559+0000","last_undegraded":"2024-03-20T09:34:01.248559+0000","last_fullsized":"2024-03-20T09:34:01.248559+0000","mapping_epoch":871,"log_start":"878'1700","ondisk_log_start":"878'1700","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"901'6691","last_scrub_stamp":"2024-03-20T09:32:27.342772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:27.342772+0000","objects_scrubbed":6641,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:03.439674+0000","scrub_duration":1744,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271319064,"num_objects":4141,"num_object_clones":0,"num_object_copies":12423,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4141,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11717,"num_write_kb":507393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"20'137","reported_seq":1745,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:33:56.649046+0000","last_change":"2024-03-20T09:33:30.436327+0000","last_active":"2024-03-20T09:33:56.649046+0000","last_peered":"2024-03-20T09:33:56.649046+0000","last_clean":"2024-03-20T09:33:56.649046+0000","last_became_active":"2024-03-20T09:13:20.070596+0000","last_became_peered":"2024-03-20T09:13:20.070596+0000","last_unstale":"2024-03-20T09:33:56.649046+0000","last_undegraded":"2024-03-20T09:33:56.649046+0000","last_fullsized":"2024-03-20T09:33:56.649046+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:30.436196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:33:30.436196+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:34.613856+0000","scrub_duration":208,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.8","version":"926'11663","reported_seq":11861,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:34:03.024162+0000","last_change":"2024-03-20T09:32:28.110802+0000","last_active":"2024-03-20T09:34:03.024162+0000","last_peered":"2024-03-20T09:34:03.024162+0000","last_clean":"2024-03-20T09:34:03.024162+0000","last_became_active":"2024-03-20T09:31:07.534911+0000","last_became_peered":"2024-03-20T09:31:07.534911+0000","last_unstale":"2024-03-20T09:34:03.024162+0000","last_undegraded":"2024-03-20T09:34:03.024162+0000","last_fullsized":"2024-03-20T09:34:03.024162+0000","mapping_epoch":871,"log_start":"878'1600","ondisk_log_start":"878'1600","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"902'6716","last_scrub_stamp":"2024-03-20T09:32:28.110647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:28.110647+0000","objects_scrubbed":6665,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:04.354322+0000","scrub_duration":1785,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265224192,"num_objects":4047,"num_object_clones":0,"num_object_copies":12141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11663,"num_write_kb":502720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"20'137","reported_seq":1177,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:33:56.644400+0000","last_change":"2024-03-20T09:33:09.882859+0000","last_active":"2024-03-20T09:33:56.644400+0000","last_peered":"2024-03-20T09:33:56.644400+0000","last_clean":"2024-03-20T09:33:56.644400+0000","last_became_active":"2024-03-20T09:19:54.506134+0000","last_became_peered":"2024-03-20T09:19:54.506134+0000","last_unstale":"2024-03-20T09:33:56.644400+0000","last_undegraded":"2024-03-20T09:33:56.644400+0000","last_fullsized":"2024-03-20T09:33:56.644400+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:09.882745+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:29:53.362224+0000","last_clean_scrub_stamp":"2024-03-20T09:33:09.882745+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:29.991713+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"926'11933","reported_seq":12134,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:34:01.694218+0000","last_change":"2024-03-20T09:32:33.285447+0000","last_active":"2024-03-20T09:34:01.694218+0000","last_peered":"2024-03-20T09:34:01.694218+0000","last_clean":"2024-03-20T09:34:01.694218+0000","last_became_active":"2024-03-20T09:31:07.534654+0000","last_became_peered":"2024-03-20T09:31:07.534654+0000","last_unstale":"2024-03-20T09:34:01.694218+0000","last_undegraded":"2024-03-20T09:34:01.694218+0000","last_fullsized":"2024-03-20T09:34:01.694218+0000","mapping_epoch":871,"log_start":"878'1900","ondisk_log_start":"878'1900","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"904'7122","last_scrub_stamp":"2024-03-20T09:32:33.285246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:33.285246+0000","objects_scrubbed":7070,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2081,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263913472,"num_objects":4027,"num_object_clones":0,"num_object_copies":12081,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11933,"num_write_kb":510720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"20'137","reported_seq":1941,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:33:56.646654+0000","last_change":"2024-03-20T09:33:33.560515+0000","last_active":"2024-03-20T09:33:56.646654+0000","last_peered":"2024-03-20T09:33:56.646654+0000","last_clean":"2024-03-20T09:33:56.646654+0000","last_became_active":"2024-03-20T09:19:54.524397+0000","last_became_peered":"2024-03-20T09:19:54.524397+0000","last_unstale":"2024-03-20T09:33:56.646654+0000","last_undegraded":"2024-03-20T09:33:56.646654+0000","last_fullsized":"2024-03-20T09:33:56.646654+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:33.560343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:33:33.560343+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:56.357887+0000","scrub_duration":184,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"926'11769","reported_seq":12030,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:34:03.044198+0000","last_change":"2024-03-20T09:33:40.727393+0000","last_active":"2024-03-20T09:34:03.044198+0000","last_peered":"2024-03-20T09:34:03.044198+0000","last_clean":"2024-03-20T09:34:03.044198+0000","last_became_active":"2024-03-20T09:31:07.428978+0000","last_became_peered":"2024-03-20T09:31:07.428978+0000","last_unstale":"2024-03-20T09:34:03.044198+0000","last_undegraded":"2024-03-20T09:34:03.044198+0000","last_fullsized":"2024-03-20T09:34:03.044198+0000","mapping_epoch":871,"log_start":"878'1700","ondisk_log_start":"878'1700","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"921'10653","last_scrub_stamp":"2024-03-20T09:33:40.727237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:33:40.727237+0000","objects_scrubbed":5157,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:35:05.691874+0000","scrub_duration":1252,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263127040,"num_objects":4015,"num_object_clones":0,"num_object_copies":12045,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11769,"num_write_kb":505088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.c","version":"570'138","reported_seq":1918,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:33:56.647127+0000","last_change":"2024-03-20T09:33:33.082457+0000","last_active":"2024-03-20T09:33:56.647127+0000","last_peered":"2024-03-20T09:33:56.647127+0000","last_clean":"2024-03-20T09:33:56.647127+0000","last_became_active":"2024-03-20T09:13:27.087686+0000","last_became_peered":"2024-03-20T09:13:27.087686+0000","last_unstale":"2024-03-20T09:33:56.647127+0000","last_undegraded":"2024-03-20T09:33:56.647127+0000","last_fullsized":"2024-03-20T09:33:56.647127+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"570'138","last_scrub_stamp":"2024-03-20T09:33:33.082331+0000","last_deep_scrub":"570'138","last_deep_scrub_stamp":"2024-03-20T09:16:13.725035+0000","last_clean_scrub_stamp":"2024-03-20T09:33:33.082331+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:40.049713+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"926'11886","reported_seq":12083,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:33:59.787882+0000","last_change":"2024-03-20T09:32:27.484409+0000","last_active":"2024-03-20T09:33:59.787882+0000","last_peered":"2024-03-20T09:33:59.787882+0000","last_clean":"2024-03-20T09:33:59.787882+0000","last_became_active":"2024-03-20T09:31:07.424767+0000","last_became_peered":"2024-03-20T09:31:07.424767+0000","last_unstale":"2024-03-20T09:33:59.787882+0000","last_undegraded":"2024-03-20T09:33:59.787882+0000","last_fullsized":"2024-03-20T09:33:59.787882+0000","mapping_epoch":871,"log_start":"878'1800","ondisk_log_start":"878'1800","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"901'6848","last_scrub_stamp":"2024-03-20T09:32:27.484264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:27.484264+0000","objects_scrubbed":6805,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:03.325004+0000","scrub_duration":1598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281673728,"num_objects":4298,"num_object_clones":0,"num_object_copies":12894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11886,"num_write_kb":517888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,11],"acting":[5,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"20'137","reported_seq":1483,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:33:56.644373+0000","last_change":"2024-03-20T09:33:11.988607+0000","last_active":"2024-03-20T09:33:56.644373+0000","last_peered":"2024-03-20T09:33:56.644373+0000","last_clean":"2024-03-20T09:33:56.644373+0000","last_became_active":"2024-03-20T09:19:54.502529+0000","last_became_peered":"2024-03-20T09:19:54.502529+0000","last_unstale":"2024-03-20T09:33:56.644373+0000","last_undegraded":"2024-03-20T09:33:56.644373+0000","last_fullsized":"2024-03-20T09:33:56.644373+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:11.988521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:33:11.988521+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:25.929908+0000","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"20'137","reported_seq":1772,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:33:56.643991+0000","last_change":"2024-03-20T09:32:27.000506+0000","last_active":"2024-03-20T09:33:56.643991+0000","last_peered":"2024-03-20T09:33:56.643991+0000","last_clean":"2024-03-20T09:33:56.643991+0000","last_became_active":"2024-03-20T09:10:34.096758+0000","last_became_peered":"2024-03-20T09:10:34.096758+0000","last_unstale":"2024-03-20T09:33:56.643991+0000","last_undegraded":"2024-03-20T09:33:56.643991+0000","last_fullsized":"2024-03-20T09:33:56.643991+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:32:27.000415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:32:27.000415+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:33:57.313174+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"20'137","reported_seq":524,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:33:56.649113+0000","last_change":"2024-03-20T09:33:11.373371+0000","last_active":"2024-03-20T09:33:56.649113+0000","last_peered":"2024-03-20T09:33:56.649113+0000","last_clean":"2024-03-20T09:33:56.649113+0000","last_became_active":"2024-03-20T09:31:07.425519+0000","last_became_peered":"2024-03-20T09:31:07.425519+0000","last_unstale":"2024-03-20T09:33:56.649113+0000","last_undegraded":"2024-03-20T09:33:56.649113+0000","last_fullsized":"2024-03-20T09:33:56.649113+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:11.373245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:33:11.373245+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:28.744893+0000","scrub_duration":229,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.10","version":"20'137","reported_seq":1326,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:33:56.644075+0000","last_change":"2024-03-20T09:33:38.307496+0000","last_active":"2024-03-20T09:33:56.644075+0000","last_peered":"2024-03-20T09:33:56.644075+0000","last_clean":"2024-03-20T09:33:56.644075+0000","last_became_active":"2024-03-20T09:28:22.584176+0000","last_became_peered":"2024-03-20T09:28:22.584176+0000","last_unstale":"2024-03-20T09:33:56.644075+0000","last_undegraded":"2024-03-20T09:33:56.644075+0000","last_fullsized":"2024-03-20T09:33:56.644075+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:38.307394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:33:38.307394+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:45.805742+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"20'137","reported_seq":1715,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:33:56.637909+0000","last_change":"2024-03-20T09:32:25.305425+0000","last_active":"2024-03-20T09:33:56.637909+0000","last_peered":"2024-03-20T09:33:56.637909+0000","last_clean":"2024-03-20T09:33:56.637909+0000","last_became_active":"2024-03-20T09:20:25.342982+0000","last_became_peered":"2024-03-20T09:20:25.342982+0000","last_unstale":"2024-03-20T09:33:56.637909+0000","last_undegraded":"2024-03-20T09:33:56.637909+0000","last_fullsized":"2024-03-20T09:33:56.637909+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:32:25.305287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:32:25.305287+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"20'137","reported_seq":441,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:33:56.638144+0000","last_change":"2024-03-20T09:32:42.323789+0000","last_active":"2024-03-20T09:33:56.638144+0000","last_peered":"2024-03-20T09:33:56.638144+0000","last_clean":"2024-03-20T09:33:56.638144+0000","last_became_active":"2024-03-20T09:31:07.424851+0000","last_became_peered":"2024-03-20T09:31:07.424851+0000","last_unstale":"2024-03-20T09:33:56.638144+0000","last_undegraded":"2024-03-20T09:33:56.638144+0000","last_fullsized":"2024-03-20T09:33:56.638144+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:32:42.323661+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:16:15.448920+0000","last_clean_scrub_stamp":"2024-03-20T09:32:42.323661+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.17","version":"20'137","reported_seq":1902,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:33:56.651589+0000","last_change":"2024-03-20T09:33:39.823553+0000","last_active":"2024-03-20T09:33:56.651589+0000","last_peered":"2024-03-20T09:33:56.651589+0000","last_clean":"2024-03-20T09:33:56.651589+0000","last_became_active":"2024-03-20T09:28:22.588303+0000","last_became_peered":"2024-03-20T09:28:22.588303+0000","last_unstale":"2024-03-20T09:33:56.651589+0000","last_undegraded":"2024-03-20T09:33:56.651589+0000","last_fullsized":"2024-03-20T09:33:56.651589+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:39.823418+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:23.640533+0000","last_clean_scrub_stamp":"2024-03-20T09:33:39.823418+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:58.632784+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"20'137","reported_seq":1943,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:33:56.640091+0000","last_change":"2024-03-20T09:32:32.410744+0000","last_active":"2024-03-20T09:33:56.640091+0000","last_peered":"2024-03-20T09:33:56.640091+0000","last_clean":"2024-03-20T09:33:56.640091+0000","last_became_active":"2024-03-20T09:20:25.349082+0000","last_became_peered":"2024-03-20T09:20:25.349082+0000","last_unstale":"2024-03-20T09:33:56.640091+0000","last_undegraded":"2024-03-20T09:33:56.640091+0000","last_fullsized":"2024-03-20T09:33:56.640091+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:32:32.410595+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:30:54.106826+0000","last_clean_scrub_stamp":"2024-03-20T09:32:32.410595+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:33:58.396036+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"20'137","reported_seq":1224,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:33:56.643888+0000","last_change":"2024-03-20T09:33:09.047464+0000","last_active":"2024-03-20T09:33:56.643888+0000","last_peered":"2024-03-20T09:33:56.643888+0000","last_clean":"2024-03-20T09:33:56.643888+0000","last_became_active":"2024-03-20T09:12:14.307581+0000","last_became_peered":"2024-03-20T09:12:14.307581+0000","last_unstale":"2024-03-20T09:33:56.643888+0000","last_undegraded":"2024-03-20T09:33:56.643888+0000","last_fullsized":"2024-03-20T09:33:56.643888+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:09.047372+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:23:41.251546+0000","last_clean_scrub_stamp":"2024-03-20T09:33:09.047372+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:37.240972+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"20'137","reported_seq":1482,"reported_epoch":926,"state":"active+clean","last_fresh":"2024-03-20T09:33:56.644212+0000","last_change":"2024-03-20T09:32:38.833000+0000","last_active":"2024-03-20T09:33:56.644212+0000","last_peered":"2024-03-20T09:33:56.644212+0000","last_clean":"2024-03-20T09:33:56.644212+0000","last_became_active":"2024-03-20T09:19:54.502391+0000","last_became_peered":"2024-03-20T09:19:54.502391+0000","last_unstale":"2024-03-20T09:33:56.644212+0000","last_undegraded":"2024-03-20T09:33:56.644212+0000","last_fullsized":"2024-03-20T09:33:56.644212+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:32:38.832878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:32:38.832878+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:02.148297+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":4310827032,"num_objects":65779,"num_object_clones":0,"num_object_copies":197337,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65779,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187439,"num_write_kb":8102913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12934983680,"data_stored":12934971464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160839,"ondisk_log_size":160839,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4248,"ondisk_log_size":4248,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860872,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2188512,"kb_used_data":1336856,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":91556640,"statfs":{"total":95995035648,"available":93753999360,"internally_reserved":0,"allocated":1368940544,"data_stored":1368678548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":872085956},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":3697966841898,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":80,"seq":343597384150,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1703456,"kb_used_data":1303000,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":92041696,"statfs":{"total":95995035648,"available":94250696704,"internally_reserved":0,"allocated":1334272000,"data_stored":1334010004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":410057152},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":70,"seq":300647711194,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1255456,"kb_used_data":773080,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":92489696,"statfs":{"total":95995035648,"available":94709448704,"internally_reserved":0,"allocated":791633920,"data_stored":791371924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":493943238},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":60,"seq":257698038238,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1315688,"kb_used_data":526496,"kb_used_omap":1,"kb_used_meta":789182,"kb_avail":92429464,"statfs":{"total":95995035648,"available":94647771136,"internally_reserved":0,"allocated":539131904,"data_stored":538866356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":808122826},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332579,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1984740,"kb_used_data":1321884,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":91760412,"statfs":{"total":95995035648,"available":93962661888,"internally_reserved":0,"allocated":1353609216,"data_stored":1353343148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":678754758},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575357,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1668904,"kb_used_data":825436,"kb_used_omap":1,"kb_used_meta":843326,"kb_avail":92076248,"statfs":{"total":95995035648,"available":94286077952,"internally_reserved":0,"allocated":845246464,"data_stored":844980396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":863566278},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738876,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2512100,"kb_used_data":1592412,"kb_used_omap":1,"kb_used_meta":919678,"kb_avail":91233052,"statfs":{"total":95995035648,"available":93422645248,"internally_reserved":0,"allocated":1630629888,"data_stored":1630363820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":941750725},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411830,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1949992,"kb_used_data":1300960,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":91795160,"statfs":{"total":95995035648,"available":93998243840,"internally_reserved":0,"allocated":1332183040,"data_stored":1331917492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150194,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2753056,"kb_used_data":1855576,"kb_used_omap":1,"kb_used_meta":897470,"kb_avail":90992096,"statfs":{"total":95995035648,"available":93175906304,"internally_reserved":0,"allocated":1900109824,"data_stored":1899847828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":919009735},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953969,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1771040,"kb_used_data":1077656,"kb_used_omap":1,"kb_used_meta":693374,"kb_avail":91974112,"statfs":{"total":95995035648,"available":94181490688,"internally_reserved":0,"allocated":1103519744,"data_stored":1103257748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":710015429},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":443,"seq":1902670512410,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1452904,"kb_used_data":813472,"kb_used_omap":181,"kb_used_meta":639242,"kb_avail":92292248,"statfs":{"total":95995035648,"available":94507261952,"internally_reserved":0,"allocated":832995328,"data_stored":832729780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":186131,"internal_metadata":654584045},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1616711680,"data_stored":1616707608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":836833280,"data_stored":836829208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1316225024,"data_stored":1316225024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1891368960,"data_stored":1891368960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1092878336,"data_stored":1092878336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":826605568,"data_stored":826605568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1337528320,"data_stored":1337524248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":537198592,"data_stored":537198592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":789774336,"data_stored":789774336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1321730048,"data_stored":1321730048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1368129536,"data_stored":1368129536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:34:09.353 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-20T09:34:09.353 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:34:09.354 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_5 2024-03-20T09:34:09.354 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:34:09.763 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:34:09.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:09.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:34:09.873 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:34:09.902 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:09.924 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:34:09.929 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:09 smithi122 ceph-mon[73495]: pgmap v2254: 47 pgs: 47 active+clean; 3.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 860 op/s 2024-03-20T09:34:09.929 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:09.930 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:34:09.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:09 smithi026 ceph-mon[68309]: pgmap v2254: 47 pgs: 47 active+clean; 3.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 860 op/s 2024-03-20T09:34:10.031 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:34:10.059 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:09 smithi115 ceph-mon[72476]: pgmap v2254: 47 pgs: 47 active+clean; 3.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 860 op/s 2024-03-20T09:34:10.062 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:10.106 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:34:10.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:10.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:34:10.212 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:34:10.243 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:10.261 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:34:10.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:10.266 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:34:10.367 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:34:10.452 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:10.630 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:34:10.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:10.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:34:10.748 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:34:10.781 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:10.810 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:34:10.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:10.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:34:10.917 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:34:10.947 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:10.971 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:34:10.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:10.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:34:11.076 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:34:11.108 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:11.139 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:34:11.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:11.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:34:11.157 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:34:11.163 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:34:11.243 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:34:11.282 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:34:11.282 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-03-20T09:34:11.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-03-20T09:34:11.282 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:34:11.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.283 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26216.0:92127 6.4 6:218a10af:::benchmark_data_smithi122_168655_object92126:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e735)", 2024-03-20T09:34:11.283 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:25:19.118194+0000", 2024-03-20T09:34:11.283 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 532.05100458899994, 2024-03-20T09:34:11.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.104795362, 2024-03-20T09:34:11.283 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:34:11.283 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:34:11.283 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:34:11.283 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26216", 2024-03-20T09:34:11.283 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:34:11.283 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92127 2024-03-20T09:34:11.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.283 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:34:11.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:34:11.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.118194+0000", 2024-03-20T09:34:11.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:34:11.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.118194+0000", 2024-03-20T09:34:11.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:34:11.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.118195+0000", 2024-03-20T09:34:11.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.573e-06 2024-03-20T09:34:11.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:34:11.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.118228+0000", 2024-03-20T09:34:11.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2341000000000001e-05 2024-03-20T09:34:11.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:34:11.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.118240+0000", 2024-03-20T09:34:11.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2469e-05 2024-03-20T09:34:11.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:34:11.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.118248+0000", 2024-03-20T09:34:11.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5859999999999999e-06 2024-03-20T09:34:11.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:34:11.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.118583+0000", 2024-03-20T09:34:11.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00033515900000000002 2024-03-20T09:34:11.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:34:11.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.118677+0000", 2024-03-20T09:34:11.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3877999999999998e-05 2024-03-20T09:34:11.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,5", 2024-03-20T09:34:11.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.118788+0000", 2024-03-20T09:34:11.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011106499999999999 2024-03-20T09:34:11.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:34:11.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.123253+0000", 2024-03-20T09:34:11.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0044652260000000001 2024-03-20T09:34:11.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:34:11.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.123301+0000", 2024-03-20T09:34:11.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7629000000000001e-05 2024-03-20T09:34:11.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:34:11.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.222856+0000", 2024-03-20T09:34:11.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.099555313000000006 2024-03-20T09:34:11.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:34:11.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.222921+0000", 2024-03-20T09:34:11.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4956999999999995e-05 2024-03-20T09:34:11.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:34:11.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.222989+0000", 2024-03-20T09:34:11.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8165999999999998e-05 2024-03-20T09:34:11.288 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.289 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:11.289 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.289 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26216.0:92142 6.4 6:244d4b3c:::benchmark_data_smithi122_168655_object92141:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e735)", 2024-03-20T09:34:11.289 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:25:19.135911+0000", 2024-03-20T09:34:11.289 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 532.03328754999995, 2024-03-20T09:34:11.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.093497942000000001, 2024-03-20T09:34:11.289 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:34:11.289 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:34:11.289 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:34:11.289 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26216", 2024-03-20T09:34:11.289 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:34:11.289 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 92142 2024-03-20T09:34:11.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.290 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:34:11.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:34:11.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.135911+0000", 2024-03-20T09:34:11.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:34:11.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.135911+0000", 2024-03-20T09:34:11.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:34:11.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.135911+0000", 2024-03-20T09:34:11.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8999999999999996e-07 2024-03-20T09:34:11.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:34:11.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.135954+0000", 2024-03-20T09:34:11.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2265000000000003e-05 2024-03-20T09:34:11.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:34:11.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.135963+0000", 2024-03-20T09:34:11.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0129999999999999e-06 2024-03-20T09:34:11.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:34:11.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.135970+0000", 2024-03-20T09:34:11.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8369999999999999e-06 2024-03-20T09:34:11.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:34:11.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.223074+0000", 2024-03-20T09:34:11.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.087104138999999997 2024-03-20T09:34:11.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:34:11.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.223241+0000", 2024-03-20T09:34:11.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000166835 2024-03-20T09:34:11.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,5", 2024-03-20T09:34:11.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.223393+0000", 2024-03-20T09:34:11.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000152423 2024-03-20T09:34:11.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:34:11.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.229110+0000", 2024-03-20T09:34:11.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057167980000000004 2024-03-20T09:34:11.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:34:11.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.229249+0000", 2024-03-20T09:34:11.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013882400000000001 2024-03-20T09:34:11.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:34:11.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.229294+0000", 2024-03-20T09:34:11.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5352000000000002e-05 2024-03-20T09:34:11.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:34:11.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.229349+0000", 2024-03-20T09:34:11.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4969000000000002e-05 2024-03-20T09:34:11.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:34:11.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:19.229409+0000", 2024-03-20T09:34:11.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9896999999999999e-05 2024-03-20T09:34:11.294 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.295 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:11.295 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.295 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26216.0:137161 6.4 6:20bdbb8f:::benchmark_data_smithi122_168655_object17052:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e770)", 2024-03-20T09:34:11.295 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:26:43.100543+0000", 2024-03-20T09:34:11.295 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 448.06865558499999, 2024-03-20T09:34:11.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.21258897900000001, 2024-03-20T09:34:11.295 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:34:11.295 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:34:11.295 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:34:11.295 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26216", 2024-03-20T09:34:11.295 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:34:11.295 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 137161 2024-03-20T09:34:11.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.296 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:34:11.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:34:11.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.100543+0000", 2024-03-20T09:34:11.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:34:11.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.100543+0000", 2024-03-20T09:34:11.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:34:11.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.100544+0000", 2024-03-20T09:34:11.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.643e-06 2024-03-20T09:34:11.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:34:11.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.100552+0000", 2024-03-20T09:34:11.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3769999999999998e-06 2024-03-20T09:34:11.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:34:11.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.100565+0000", 2024-03-20T09:34:11.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3604000000000001e-05 2024-03-20T09:34:11.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:34:11.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.100574+0000", 2024-03-20T09:34:11.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8529999999999998e-06 2024-03-20T09:34:11.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:34:11.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.100630+0000", 2024-03-20T09:34:11.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6147999999999998e-05 2024-03-20T09:34:11.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:34:11.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.100762+0000", 2024-03-20T09:34:11.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000131951 2024-03-20T09:34:11.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,5", 2024-03-20T09:34:11.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.100892+0000", 2024-03-20T09:34:11.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012919299999999999 2024-03-20T09:34:11.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:34:11.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.105582+0000", 2024-03-20T09:34:11.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0046907700000000004 2024-03-20T09:34:11.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:34:11.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.114181+0000", 2024-03-20T09:34:11.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0085988100000000001 2024-03-20T09:34:11.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:34:11.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.313024+0000", 2024-03-20T09:34:11.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.198842989 2024-03-20T09:34:11.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:34:11.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.313083+0000", 2024-03-20T09:34:11.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9265000000000003e-05 2024-03-20T09:34:11.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:34:11.301 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.313132+0000", 2024-03-20T09:34:11.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8375999999999999e-05 2024-03-20T09:34:11.301 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.301 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:11.301 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.301 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26216.0:137247 6.14 6:2d9ee346:::benchmark_data_smithi122_168655_object17138:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e771)", 2024-03-20T09:34:11.301 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:26:43.639609+0000", 2024-03-20T09:34:11.301 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 447.52958932500002, 2024-03-20T09:34:11.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.20117981099999999, 2024-03-20T09:34:11.301 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:34:11.301 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:34:11.301 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:34:11.302 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26216", 2024-03-20T09:34:11.302 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:34:11.302 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 137247 2024-03-20T09:34:11.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.302 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:34:11.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:34:11.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.639609+0000", 2024-03-20T09:34:11.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:34:11.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.639609+0000", 2024-03-20T09:34:11.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:34:11.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.639611+0000", 2024-03-20T09:34:11.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.722e-06 2024-03-20T09:34:11.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:34:11.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.639618+0000", 2024-03-20T09:34:11.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5669999999999997e-06 2024-03-20T09:34:11.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:34:11.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.639766+0000", 2024-03-20T09:34:11.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014748100000000001 2024-03-20T09:34:11.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:34:11.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.639780+0000", 2024-03-20T09:34:11.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4141000000000001e-05 2024-03-20T09:34:11.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:34:11.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.644074+0000", 2024-03-20T09:34:11.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042937339999999996 2024-03-20T09:34:11.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:34:11.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.644226+0000", 2024-03-20T09:34:11.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000152774 2024-03-20T09:34:11.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2,4,7", 2024-03-20T09:34:11.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.644328+0000", 2024-03-20T09:34:11.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101246 2024-03-20T09:34:11.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:34:11.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.646928+0000", 2024-03-20T09:34:11.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0025998359999999999 2024-03-20T09:34:11.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:34:11.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.650501+0000", 2024-03-20T09:34:11.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035736919999999998 2024-03-20T09:34:11.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:34:11.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.655566+0000", 2024-03-20T09:34:11.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.005065089 2024-03-20T09:34:11.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:34:11.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.838998+0000", 2024-03-20T09:34:11.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.18343200600000001 2024-03-20T09:34:11.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:34:11.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.839082+0000", 2024-03-20T09:34:11.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4073999999999995e-05 2024-03-20T09:34:11.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:34:11.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.840789+0000", 2024-03-20T09:34:11.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0017064490000000001 2024-03-20T09:34:11.307 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.307 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:11.307 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.308 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26216.0:137256 6.14 6:2eed8d05:::benchmark_data_smithi122_168655_object17147:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e771)", 2024-03-20T09:34:11.308 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:26:43.659230+0000", 2024-03-20T09:34:11.308 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 447.509968055, 2024-03-20T09:34:11.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.18461831000000001, 2024-03-20T09:34:11.308 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:34:11.308 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:34:11.308 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:34:11.308 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26216", 2024-03-20T09:34:11.308 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:34:11.308 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 137256 2024-03-20T09:34:11.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.308 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:34:11.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:34:11.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.659230+0000", 2024-03-20T09:34:11.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.309 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:34:11.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.659230+0000", 2024-03-20T09:34:11.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.309 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:34:11.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.659232+0000", 2024-03-20T09:34:11.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2440000000000001e-06 2024-03-20T09:34:11.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:34:11.310 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.659237+0000", 2024-03-20T09:34:11.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.045e-06 2024-03-20T09:34:11.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:34:11.310 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.659247+0000", 2024-03-20T09:34:11.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0710000000000001e-05 2024-03-20T09:34:11.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:34:11.310 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.659254+0000", 2024-03-20T09:34:11.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4489999999999996e-06 2024-03-20T09:34:11.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:34:11.311 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.659308+0000", 2024-03-20T09:34:11.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4033999999999998e-05 2024-03-20T09:34:11.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:34:11.311 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.659453+0000", 2024-03-20T09:34:11.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000145148 2024-03-20T09:34:11.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2,4,7", 2024-03-20T09:34:11.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.659545+0000", 2024-03-20T09:34:11.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2514000000000002e-05 2024-03-20T09:34:11.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:34:11.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.664460+0000", 2024-03-20T09:34:11.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049144749999999997 2024-03-20T09:34:11.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:34:11.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.664801+0000", 2024-03-20T09:34:11.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00034128799999999998 2024-03-20T09:34:11.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:34:11.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.665000+0000", 2024-03-20T09:34:11.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019872299999999999 2024-03-20T09:34:11.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:34:11.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.841883+0000", 2024-03-20T09:34:11.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.176882662 2024-03-20T09:34:11.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:34:11.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.841955+0000", 2024-03-20T09:34:11.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2274000000000006e-05 2024-03-20T09:34:11.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:34:11.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.843849+0000", 2024-03-20T09:34:11.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0018937439999999999 2024-03-20T09:34:11.314 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.314 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:11.314 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.314 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26216.0:137264 6.14 6:2ba7dd29:::benchmark_data_smithi122_168655_object17155:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e771)", 2024-03-20T09:34:11.314 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:26:43.699033+0000", 2024-03-20T09:34:11.314 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 447.470165818, 2024-03-20T09:34:11.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.14657466699999999, 2024-03-20T09:34:11.314 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:34:11.314 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:34:11.315 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:34:11.315 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26216", 2024-03-20T09:34:11.315 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:34:11.315 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 137264 2024-03-20T09:34:11.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.315 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:34:11.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:34:11.315 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.699033+0000", 2024-03-20T09:34:11.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:34:11.315 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.699033+0000", 2024-03-20T09:34:11.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.316 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:34:11.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.699034+0000", 2024-03-20T09:34:11.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.658e-06 2024-03-20T09:34:11.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.316 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:34:11.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.699042+0000", 2024-03-20T09:34:11.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2910000000000002e-06 2024-03-20T09:34:11.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.316 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:34:11.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.699057+0000", 2024-03-20T09:34:11.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5777000000000002e-05 2024-03-20T09:34:11.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:34:11.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.699067+0000", 2024-03-20T09:34:11.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3139999999999993e-06 2024-03-20T09:34:11.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:34:11.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.699132+0000", 2024-03-20T09:34:11.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5524999999999994e-05 2024-03-20T09:34:11.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:34:11.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.699290+0000", 2024-03-20T09:34:11.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000158293 2024-03-20T09:34:11.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 2,4,7", 2024-03-20T09:34:11.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.699437+0000", 2024-03-20T09:34:11.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014632800000000001 2024-03-20T09:34:11.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:34:11.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.703564+0000", 2024-03-20T09:34:11.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0041275799999999996 2024-03-20T09:34:11.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:34:11.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.703822+0000", 2024-03-20T09:34:11.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025728699999999998 2024-03-20T09:34:11.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:34:11.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.704903+0000", 2024-03-20T09:34:11.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001081023 2024-03-20T09:34:11.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:34:11.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.843903+0000", 2024-03-20T09:34:11.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.139000548 2024-03-20T09:34:11.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:34:11.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.843972+0000", 2024-03-20T09:34:11.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8827999999999999e-05 2024-03-20T09:34:11.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:34:11.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:43.845607+0000", 2024-03-20T09:34:11.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001635215 2024-03-20T09:34:11.320 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.320 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:11.320 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.320 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPushReply(6.14 779/752 [PushReplyOp(6:2ef921eb:::benchmark_data_smithi122_168655_object75880:head)])", 2024-03-20T09:34:11.320 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:26:57.043709+0000", 2024-03-20T09:34:11.320 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 434.125489281, 2024-03-20T09:34:11.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.100162324, 2024-03-20T09:34:11.321 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:34:11.321 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:34:11.321 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:34:11.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.321 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:34:11.321 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:57.043709+0000", 2024-03-20T09:34:11.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.321 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:34:11.321 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:57.043709+0000", 2024-03-20T09:34:11.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:34:11.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:57.043710+0000", 2024-03-20T09:34:11.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7800000000000001e-07 2024-03-20T09:34:11.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:34:11.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:57.043714+0000", 2024-03-20T09:34:11.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9929999999999997e-06 2024-03-20T09:34:11.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:34:11.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:57.043723+0000", 2024-03-20T09:34:11.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7999999999999993e-06 2024-03-20T09:34:11.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:34:11.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:57.043731+0000", 2024-03-20T09:34:11.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8480000000000001e-06 2024-03-20T09:34:11.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:34:11.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:57.044817+0000", 2024-03-20T09:34:11.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001085983 2024-03-20T09:34:11.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:34:11.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:57.143871+0000", 2024-03-20T09:34:11.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.099054121999999994 2024-03-20T09:34:11.323 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.324 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:11.324 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.324 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDPGPush(6.19 779/755 [PushOp(6:9aaf9f74:::benchmark_data_smithi122_168655_object22062:head, version: 721'1326, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:9aaf9f74:::benchmark_data_smithi122_168655_object22062:head@721'1326, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-20T09:34:11.324 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:26:57.049839+0000", 2024-03-20T09:34:11.324 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 434.11935934500002, 2024-03-20T09:34:11.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.094772406000000003, 2024-03-20T09:34:11.324 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:34:11.324 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:34:11.324 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:34:11.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:34:11.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:57.049839+0000", 2024-03-20T09:34:11.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:34:11.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:57.049839+0000", 2024-03-20T09:34:11.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:34:11.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:57.049840+0000", 2024-03-20T09:34:11.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4099999999999997e-07 2024-03-20T09:34:11.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:34:11.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:57.049888+0000", 2024-03-20T09:34:11.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8473999999999997e-05 2024-03-20T09:34:11.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:34:11.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:57.142347+0000", 2024-03-20T09:34:11.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.092458234 2024-03-20T09:34:11.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:34:11.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:57.142381+0000", 2024-03-20T09:34:11.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4687999999999997e-05 2024-03-20T09:34:11.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:34:11.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:57.142470+0000", 2024-03-20T09:34:11.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8302999999999998e-05 2024-03-20T09:34:11.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:34:11.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:57.142482+0000", 2024-03-20T09:34:11.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2136999999999999e-05 2024-03-20T09:34:11.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:34:11.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:57.144611+0000", 2024-03-20T09:34:11.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021296290000000001 2024-03-20T09:34:11.327 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.327 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:11.328 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.328 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26216.0:182463 6.d e830/820 6:b9593923:::benchmark_data_smithi122_168655_object62354:head v 830'11372, mlcod=830'11370)", 2024-03-20T09:34:11.328 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:29:02.451312+0000", 2024-03-20T09:34:11.328 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 308.71788687700001, 2024-03-20T09:34:11.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.128268877, 2024-03-20T09:34:11.328 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:34:11.328 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:34:11.328 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:34:11.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.328 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:34:11.328 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:29:02.451312+0000", 2024-03-20T09:34:11.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:34:11.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:29:02.451312+0000", 2024-03-20T09:34:11.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:34:11.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:29:02.451312+0000", 2024-03-20T09:34:11.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9400000000000005e-07 2024-03-20T09:34:11.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:34:11.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:29:02.451318+0000", 2024-03-20T09:34:11.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8329999999999999e-06 2024-03-20T09:34:11.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:34:11.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:29:02.451330+0000", 2024-03-20T09:34:11.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1652e-05 2024-03-20T09:34:11.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:34:11.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:29:02.451335+0000", 2024-03-20T09:34:11.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2689999999999999e-06 2024-03-20T09:34:11.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:34:11.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:29:02.451381+0000", 2024-03-20T09:34:11.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5741000000000001e-05 2024-03-20T09:34:11.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:34:11.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:29:02.451405+0000", 2024-03-20T09:34:11.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4142e-05 2024-03-20T09:34:11.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:34:11.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:29:02.579526+0000", 2024-03-20T09:34:11.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.12812094199999999 2024-03-20T09:34:11.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:34:11.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:29:02.579580+0000", 2024-03-20T09:34:11.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4703999999999997e-05 2024-03-20T09:34:11.332 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.332 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:11.332 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.332 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26216.0:231491 6.d e858/820 6:b3643d5d:::benchmark_data_smithi122_168655_object111382:head v 858'14420, mlcod=858'14418)", 2024-03-20T09:34:11.332 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:30:36.467662+0000", 2024-03-20T09:34:11.332 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 214.701536175, 2024-03-20T09:34:11.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.148936125, 2024-03-20T09:34:11.332 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:34:11.332 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:34:11.332 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:34:11.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:34:11.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:36.467662+0000", 2024-03-20T09:34:11.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:34:11.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:36.467662+0000", 2024-03-20T09:34:11.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:34:11.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:36.467663+0000", 2024-03-20T09:34:11.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1629999999999999e-06 2024-03-20T09:34:11.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:34:11.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:36.467693+0000", 2024-03-20T09:34:11.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9964e-05 2024-03-20T09:34:11.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:34:11.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:36.467700+0000", 2024-03-20T09:34:11.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8719999999999996e-06 2024-03-20T09:34:11.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:34:11.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:36.467708+0000", 2024-03-20T09:34:11.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4340000000000003e-06 2024-03-20T09:34:11.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:34:11.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:36.470146+0000", 2024-03-20T09:34:11.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0024384710000000002 2024-03-20T09:34:11.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:34:11.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:36.470166+0000", 2024-03-20T09:34:11.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.942e-05 2024-03-20T09:34:11.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:34:11.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:36.616554+0000", 2024-03-20T09:34:11.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.146388776 2024-03-20T09:34:11.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:34:11.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:30:36.616598+0000", 2024-03-20T09:34:11.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4024999999999999e-05 2024-03-20T09:34:11.336 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.336 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:11.336 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.336 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26605.0:39258 7.d e880/871 7:b42b52fd:::benchmark_data_smithi122_187077_object39257:head v 880'2480, mlcod=880'2477)", 2024-03-20T09:34:11.336 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:31:34.561861+0000", 2024-03-20T09:34:11.336 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 156.607337771, 2024-03-20T09:34:11.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.094761710999999998, 2024-03-20T09:34:11.336 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:34:11.337 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:34:11.337 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:34:11.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.337 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:34:11.337 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:34.561861+0000", 2024-03-20T09:34:11.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.337 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:34:11.337 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:34.561861+0000", 2024-03-20T09:34:11.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.337 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:34:11.337 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:34.561861+0000", 2024-03-20T09:34:11.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6899999999999998e-07 2024-03-20T09:34:11.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:34:11.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:34.561899+0000", 2024-03-20T09:34:11.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7777000000000001e-05 2024-03-20T09:34:11.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:34:11.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:34.561908+0000", 2024-03-20T09:34:11.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4280000000000002e-06 2024-03-20T09:34:11.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:34:11.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:34.561914+0000", 2024-03-20T09:34:11.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9449999999999998e-06 2024-03-20T09:34:11.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:34:11.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:34.563226+0000", 2024-03-20T09:34:11.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001311406 2024-03-20T09:34:11.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:34:11.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:34.563246+0000", 2024-03-20T09:34:11.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0800000000000001e-05 2024-03-20T09:34:11.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:34:11.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:34.656595+0000", 2024-03-20T09:34:11.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.093349007999999997 2024-03-20T09:34:11.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:34:11.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:31:34.656622+0000", 2024-03-20T09:34:11.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6978000000000002e-05 2024-03-20T09:34:11.340 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.340 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:11.340 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.340 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26605.0:102111 7.b e899/871 7:d38335cc:::benchmark_data_smithi122_187077_object102110:head v 899'6508, mlcod=899'6506)", 2024-03-20T09:34:11.340 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:32:21.688515+0000", 2024-03-20T09:34:11.341 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 109.480683826, 2024-03-20T09:34:11.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.12018569799999999, 2024-03-20T09:34:11.341 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:34:11.341 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:34:11.341 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:34:11.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:34:11.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.688515+0000", 2024-03-20T09:34:11.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:34:11.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.688515+0000", 2024-03-20T09:34:11.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:34:11.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.688516+0000", 2024-03-20T09:34:11.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0950000000000001e-06 2024-03-20T09:34:11.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:34:11.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.689061+0000", 2024-03-20T09:34:11.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00054485999999999998 2024-03-20T09:34:11.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:34:11.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.689076+0000", 2024-03-20T09:34:11.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5780000000000001e-05 2024-03-20T09:34:11.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:34:11.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.689084+0000", 2024-03-20T09:34:11.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8369999999999997e-06 2024-03-20T09:34:11.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:34:11.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.689176+0000", 2024-03-20T09:34:11.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1724999999999993e-05 2024-03-20T09:34:11.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:34:11.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.689196+0000", 2024-03-20T09:34:11.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9956e-05 2024-03-20T09:34:11.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:34:11.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.808666+0000", 2024-03-20T09:34:11.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.11946981600000001 2024-03-20T09:34:11.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:34:11.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.808700+0000", 2024-03-20T09:34:11.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4629000000000003e-05 2024-03-20T09:34:11.344 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.344 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:11.344 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.345 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26605.0:102120 7.b e899/871 7:d9b83c24:::benchmark_data_smithi122_187077_object102119:head v 899'6509, mlcod=899'6506)", 2024-03-20T09:34:11.345 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:32:21.697837+0000", 2024-03-20T09:34:11.345 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 109.47136102, 2024-03-20T09:34:11.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.11089062399999999, 2024-03-20T09:34:11.345 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:34:11.345 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:34:11.345 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:34:11.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:34:11.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.697837+0000", 2024-03-20T09:34:11.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:34:11.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.697837+0000", 2024-03-20T09:34:11.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:34:11.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.697839+0000", 2024-03-20T09:34:11.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4050000000000001e-06 2024-03-20T09:34:11.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:34:11.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.697878+0000", 2024-03-20T09:34:11.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9359e-05 2024-03-20T09:34:11.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:34:11.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.697892+0000", 2024-03-20T09:34:11.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4246999999999999e-05 2024-03-20T09:34:11.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:34:11.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.697901+0000", 2024-03-20T09:34:11.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5629999999999992e-06 2024-03-20T09:34:11.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:34:11.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.697974+0000", 2024-03-20T09:34:11.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3530999999999998e-05 2024-03-20T09:34:11.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.348 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:34:11.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.697995+0000", 2024-03-20T09:34:11.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0234e-05 2024-03-20T09:34:11.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.348 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:34:11.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.808704+0000", 2024-03-20T09:34:11.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.11070951900000001 2024-03-20T09:34:11.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.348 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:34:11.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.808728+0000", 2024-03-20T09:34:11.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3765999999999999e-05 2024-03-20T09:34:11.349 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.349 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:11.349 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.349 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26605.0:102123 7.b e899/871 7:da80447b:::benchmark_data_smithi122_187077_object102122:head v 899'6510, mlcod=899'6506)", 2024-03-20T09:34:11.349 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:32:21.702780+0000", 2024-03-20T09:34:11.349 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 109.466418678, 2024-03-20T09:34:11.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.105972337, 2024-03-20T09:34:11.349 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:34:11.349 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:34:11.349 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:34:11.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:34:11.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.702780+0000", 2024-03-20T09:34:11.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:34:11.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.702780+0000", 2024-03-20T09:34:11.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:34:11.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.702781+0000", 2024-03-20T09:34:11.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.173e-06 2024-03-20T09:34:11.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:34:11.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.702843+0000", 2024-03-20T09:34:11.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2426000000000002e-05 2024-03-20T09:34:11.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:34:11.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.702854+0000", 2024-03-20T09:34:11.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0667e-05 2024-03-20T09:34:11.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:34:11.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.702862+0000", 2024-03-20T09:34:11.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5320000000000001e-06 2024-03-20T09:34:11.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:34:11.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.702935+0000", 2024-03-20T09:34:11.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3360000000000002e-05 2024-03-20T09:34:11.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:34:11.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.702956+0000", 2024-03-20T09:34:11.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0988999999999998e-05 2024-03-20T09:34:11.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:34:11.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.808730+0000", 2024-03-20T09:34:11.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.105774294 2024-03-20T09:34:11.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:34:11.353 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.808752+0000", 2024-03-20T09:34:11.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1895999999999998e-05 2024-03-20T09:34:11.353 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.353 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:11.353 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.353 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26605.0:102129 7.b e899/871 7:d06d3109:::benchmark_data_smithi122_187077_object102128:head v 899'6511, mlcod=899'6506)", 2024-03-20T09:34:11.353 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:32:21.706346+0000", 2024-03-20T09:34:11.353 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 109.462852623, 2024-03-20T09:34:11.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.102430682, 2024-03-20T09:34:11.353 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:34:11.353 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:34:11.354 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:34:11.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.354 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:34:11.354 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.706346+0000", 2024-03-20T09:34:11.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.354 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:34:11.354 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.706346+0000", 2024-03-20T09:34:11.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.354 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:34:11.354 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.706347+0000", 2024-03-20T09:34:11.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6000000000000002e-07 2024-03-20T09:34:11.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:34:11.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.706409+0000", 2024-03-20T09:34:11.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1915999999999995e-05 2024-03-20T09:34:11.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:34:11.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.706420+0000", 2024-03-20T09:34:11.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1467e-05 2024-03-20T09:34:11.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:34:11.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.706428+0000", 2024-03-20T09:34:11.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4919999999999996e-06 2024-03-20T09:34:11.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:34:11.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.706489+0000", 2024-03-20T09:34:11.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1057999999999997e-05 2024-03-20T09:34:11.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:34:11.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.706507+0000", 2024-03-20T09:34:11.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8073000000000001e-05 2024-03-20T09:34:11.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:34:11.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.808754+0000", 2024-03-20T09:34:11.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.10224725799999999 2024-03-20T09:34:11.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.357 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:34:11.357 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.808776+0000", 2024-03-20T09:34:11.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2558e-05 2024-03-20T09:34:11.357 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.357 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:11.357 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.357 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26605.0:123288 7.d e907/871 7:b14c167e:::benchmark_data_smithi122_187077_object123287:head v 907'7696, mlcod=907'7694)", 2024-03-20T09:34:11.357 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:32:45.159595+0000", 2024-03-20T09:34:11.357 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 86.009603863999999, 2024-03-20T09:34:11.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.111427225, 2024-03-20T09:34:11.358 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:34:11.358 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:34:11.358 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:34:11.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.358 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:34:11.358 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.159595+0000", 2024-03-20T09:34:11.358 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.358 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:34:11.358 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.159595+0000", 2024-03-20T09:34:11.358 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:34:11.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.159596+0000", 2024-03-20T09:34:11.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-03-20T09:34:11.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:34:11.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.159907+0000", 2024-03-20T09:34:11.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00031138500000000001 2024-03-20T09:34:11.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:34:11.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.159929+0000", 2024-03-20T09:34:11.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1746000000000002e-05 2024-03-20T09:34:11.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.360 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:34:11.360 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.159939+0000", 2024-03-20T09:34:11.360 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9469999999999998e-06 2024-03-20T09:34:11.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.360 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:34:11.360 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.160053+0000", 2024-03-20T09:34:11.360 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000113445 2024-03-20T09:34:11.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.360 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:34:11.360 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.160075+0000", 2024-03-20T09:34:11.360 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2898e-05 2024-03-20T09:34:11.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.361 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:34:11.361 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.270998+0000", 2024-03-20T09:34:11.361 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.110922303 2024-03-20T09:34:11.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.361 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:34:11.361 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.271022+0000", 2024-03-20T09:34:11.361 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4017000000000001e-05 2024-03-20T09:34:11.361 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.361 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:11.361 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.361 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26605.0:123289 7.d e907/871 7:b88215d3:::benchmark_data_smithi122_187077_object123288:head v 907'7697, mlcod=907'7694)", 2024-03-20T09:34:11.362 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:32:45.163937+0000", 2024-03-20T09:34:11.362 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 86.005261782000005, 2024-03-20T09:34:11.362 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.107102487, 2024-03-20T09:34:11.362 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:34:11.362 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:34:11.362 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:34:11.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.362 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:34:11.362 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.163937+0000", 2024-03-20T09:34:11.362 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.362 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:34:11.363 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.163937+0000", 2024-03-20T09:34:11.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.363 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:34:11.363 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.163937+0000", 2024-03-20T09:34:11.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4200000000000005e-07 2024-03-20T09:34:11.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.363 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:34:11.363 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.163973+0000", 2024-03-20T09:34:11.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5136999999999999e-05 2024-03-20T09:34:11.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.363 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:34:11.364 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.163983+0000", 2024-03-20T09:34:11.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0553999999999999e-05 2024-03-20T09:34:11.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.364 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:34:11.364 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.163989+0000", 2024-03-20T09:34:11.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0460000000000002e-06 2024-03-20T09:34:11.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.364 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:34:11.364 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.164057+0000", 2024-03-20T09:34:11.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7362999999999999e-05 2024-03-20T09:34:11.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.365 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:34:11.365 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.164071+0000", 2024-03-20T09:34:11.365 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4460999999999999e-05 2024-03-20T09:34:11.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.365 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:34:11.365 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.271025+0000", 2024-03-20T09:34:11.365 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.106953985 2024-03-20T09:34:11.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.365 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:34:11.365 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.271039+0000", 2024-03-20T09:34:11.365 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4099e-05 2024-03-20T09:34:11.365 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.365 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:11.366 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.366 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26605.0:123293 7.d e907/871 7:b75d82f3:::benchmark_data_smithi122_187077_object123292:head v 907'7698, mlcod=907'7694)", 2024-03-20T09:34:11.366 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:32:45.167587+0000", 2024-03-20T09:34:11.366 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 86.001611577999995, 2024-03-20T09:34:11.366 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.10346756999999999, 2024-03-20T09:34:11.366 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:34:11.366 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:34:11.366 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:34:11.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.366 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:34:11.366 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.167587+0000", 2024-03-20T09:34:11.367 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.367 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:34:11.367 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.167587+0000", 2024-03-20T09:34:11.367 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.367 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:34:11.367 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.167588+0000", 2024-03-20T09:34:11.367 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.003e-06 2024-03-20T09:34:11.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.367 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:34:11.367 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.167617+0000", 2024-03-20T09:34:11.368 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9200000000000002e-05 2024-03-20T09:34:11.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:34:11.368 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.167630+0000", 2024-03-20T09:34:11.368 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2644e-05 2024-03-20T09:34:11.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:34:11.368 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.167636+0000", 2024-03-20T09:34:11.368 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1650000000000003e-06 2024-03-20T09:34:11.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:34:11.368 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.167689+0000", 2024-03-20T09:34:11.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2743e-05 2024-03-20T09:34:11.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.369 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:34:11.369 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.167714+0000", 2024-03-20T09:34:11.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5369999999999999e-05 2024-03-20T09:34:11.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.369 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:34:11.369 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.271042+0000", 2024-03-20T09:34:11.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.103327617 2024-03-20T09:34:11.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.369 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:34:11.369 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.271054+0000", 2024-03-20T09:34:11.370 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2828e-05 2024-03-20T09:34:11.370 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.370 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:11.370 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.370 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26605.0:146616 7.b e915/871 7:d8c7451a:::benchmark_data_smithi122_187077_object20005:head v 915'9296, mlcod=915'9289)", 2024-03-20T09:34:11.370 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:33:11.246953+0000", 2024-03-20T09:34:11.370 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 59.922245594000003, 2024-03-20T09:34:11.370 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.12578051700000001, 2024-03-20T09:34:11.370 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:34:11.370 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:34:11.370 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:34:11.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.371 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:34:11.371 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.246953+0000", 2024-03-20T09:34:11.371 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.371 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:34:11.371 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.246953+0000", 2024-03-20T09:34:11.371 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.371 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:34:11.371 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.246954+0000", 2024-03-20T09:34:11.371 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7800000000000001e-07 2024-03-20T09:34:11.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.372 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:34:11.372 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.246960+0000", 2024-03-20T09:34:11.372 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8669999999999996e-06 2024-03-20T09:34:11.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.372 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:34:11.372 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.246971+0000", 2024-03-20T09:34:11.372 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0763e-05 2024-03-20T09:34:11.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.373 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:34:11.373 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.246978+0000", 2024-03-20T09:34:11.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0369999999999997e-06 2024-03-20T09:34:11.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.373 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:34:11.373 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.247047+0000", 2024-03-20T09:34:11.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8619000000000001e-05 2024-03-20T09:34:11.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.373 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:34:11.373 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.247071+0000", 2024-03-20T09:34:11.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4000000000000001e-05 2024-03-20T09:34:11.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.374 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:34:11.374 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.372697+0000", 2024-03-20T09:34:11.374 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.125626549 2024-03-20T09:34:11.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.374 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:34:11.374 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.372733+0000", 2024-03-20T09:34:11.374 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5904e-05 2024-03-20T09:34:11.374 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.374 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:11.374 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.375 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26605.0:146623 7.b e915/871 7:d7b7e0c1:::benchmark_data_smithi122_187077_object20012:head v 915'9297, mlcod=915'9289)", 2024-03-20T09:34:11.375 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:33:11.257535+0000", 2024-03-20T09:34:11.375 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 59.911662962000001, 2024-03-20T09:34:11.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.11522500099999999, 2024-03-20T09:34:11.375 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:34:11.375 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:34:11.375 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:34:11.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.375 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:34:11.375 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.257535+0000", 2024-03-20T09:34:11.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.375 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:34:11.375 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.257535+0000", 2024-03-20T09:34:11.376 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:34:11.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.376 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:34:11.376 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.257536+0000", 2024-03-20T09:34:11.376 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5499999999999997e-07 2024-03-20T09:34:11.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.376 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:34:11.376 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.257543+0000", 2024-03-20T09:34:11.376 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7800000000000003e-06 2024-03-20T09:34:11.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.377 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:34:11.377 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.257559+0000", 2024-03-20T09:34:11.377 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5531000000000001e-05 2024-03-20T09:34:11.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.377 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:34:11.377 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.257566+0000", 2024-03-20T09:34:11.377 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1600000000000001e-06 2024-03-20T09:34:11.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.377 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:34:11.377 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.257613+0000", 2024-03-20T09:34:11.377 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7508e-05 2024-03-20T09:34:11.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.378 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:34:11.378 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.257632+0000", 2024-03-20T09:34:11.378 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.84e-05 2024-03-20T09:34:11.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.378 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:34:11.378 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.372736+0000", 2024-03-20T09:34:11.378 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.11510437599999999 2024-03-20T09:34:11.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:11.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:11.378 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:34:11.378 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.372760+0000", 2024-03-20T09:34:11.378 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4391000000000001e-05 2024-03-20T09:34:11.378 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.379 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:11.379 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.379 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:11.379 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:11.379 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:34:11.379 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:11.754 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:34:11.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:11 smithi026 ceph-mon[68309]: pgmap v2255: 47 pgs: 47 active+clean; 3.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 724 op/s 2024-03-20T09:34:11.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:11 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3470985557' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:34:11.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:11 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:34:12.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:11 smithi115 ceph-mon[72476]: pgmap v2255: 47 pgs: 47 active+clean; 3.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 724 op/s 2024-03-20T09:34:12.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:11 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3470985557' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:34:12.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:11 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:34:12.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:11 smithi122 ceph-mon[73495]: pgmap v2255: 47 pgs: 47 active+clean; 3.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 724 op/s 2024-03-20T09:34:12.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:11 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3470985557' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:34:12.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:11 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:34:12.270 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:34:12.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:12.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:34:12.379 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:34:12.407 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:12.429 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:34:12.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:12.432 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:34:12.533 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:34:12.564 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:12.588 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:34:12.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:12.592 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:34:12.693 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:34:12.795 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:12.988 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:12 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:34:12.988 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:12 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:34:12.988 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:12 smithi026 ceph-mon[68309]: osdmap e929: 12 total, 12 up, 11 in 2024-03-20T09:34:12.988 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:12 smithi026 ceph-mon[68309]: pgmap v2257: 47 pgs: 47 active+clean; 3.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 942 op/s 2024-03-20T09:34:13.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:12 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:34:13.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:12 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:34:13.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:12 smithi115 ceph-mon[72476]: osdmap e929: 12 total, 12 up, 11 in 2024-03-20T09:34:13.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:12 smithi115 ceph-mon[72476]: pgmap v2257: 47 pgs: 47 active+clean; 3.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 942 op/s 2024-03-20T09:34:13.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:12 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:34:13.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:12 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:34:13.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:12 smithi122 ceph-mon[73495]: osdmap e929: 12 total, 12 up, 11 in 2024-03-20T09:34:13.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:12 smithi122 ceph-mon[73495]: pgmap v2257: 47 pgs: 47 active+clean; 3.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 942 op/s 2024-03-20T09:34:13.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:13 smithi026 ceph-mon[68309]: 1.11 scrub starts 2024-03-20T09:34:13.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:13 smithi026 ceph-mon[68309]: 1.11 scrub ok 2024-03-20T09:34:13.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:13 smithi026 ceph-mon[68309]: 1.14 scrub starts 2024-03-20T09:34:13.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:13 smithi026 ceph-mon[68309]: 1.14 scrub ok 2024-03-20T09:34:13.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:13 smithi026 ceph-mon[68309]: 1.19 scrub starts 2024-03-20T09:34:14.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:13 smithi115 ceph-mon[72476]: 1.11 scrub starts 2024-03-20T09:34:14.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:13 smithi115 ceph-mon[72476]: 1.11 scrub ok 2024-03-20T09:34:14.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:13 smithi115 ceph-mon[72476]: 1.14 scrub starts 2024-03-20T09:34:14.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:13 smithi115 ceph-mon[72476]: 1.14 scrub ok 2024-03-20T09:34:14.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:13 smithi115 ceph-mon[72476]: 1.19 scrub starts 2024-03-20T09:34:14.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:13 smithi122 ceph-mon[73495]: 1.11 scrub starts 2024-03-20T09:34:14.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:13 smithi122 ceph-mon[73495]: 1.11 scrub ok 2024-03-20T09:34:14.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:13 smithi122 ceph-mon[73495]: 1.14 scrub starts 2024-03-20T09:34:14.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:13 smithi122 ceph-mon[73495]: 1.14 scrub ok 2024-03-20T09:34:14.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:13 smithi122 ceph-mon[73495]: 1.19 scrub starts 2024-03-20T09:34:14.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:14 smithi026 ceph-mon[68309]: 1.19 scrub ok 2024-03-20T09:34:14.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:14 smithi026 ceph-mon[68309]: 7.c scrub starts 2024-03-20T09:34:14.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:14 smithi026 ceph-mon[68309]: pgmap v2258: 47 pgs: 47 active+clean; 3.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 727 op/s 2024-03-20T09:34:15.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:14 smithi115 ceph-mon[72476]: 1.19 scrub ok 2024-03-20T09:34:15.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:14 smithi115 ceph-mon[72476]: 7.c scrub starts 2024-03-20T09:34:15.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:14 smithi115 ceph-mon[72476]: pgmap v2258: 47 pgs: 47 active+clean; 3.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 727 op/s 2024-03-20T09:34:15.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:14 smithi122 ceph-mon[73495]: 1.19 scrub ok 2024-03-20T09:34:15.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:14 smithi122 ceph-mon[73495]: 7.c scrub starts 2024-03-20T09:34:15.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:14 smithi122 ceph-mon[73495]: pgmap v2258: 47 pgs: 47 active+clean; 3.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 727 op/s 2024-03-20T09:34:15.616 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:34:15.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:15.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:34:15.725 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:34:15.794 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:15.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:15 smithi026 ceph-mon[68309]: 7.c scrub ok 2024-03-20T09:34:16.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:15 smithi115 ceph-mon[72476]: 7.c scrub ok 2024-03-20T09:34:16.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:15 smithi122 ceph-mon[73495]: 7.c scrub ok 2024-03-20T09:34:16.261 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:34:16.427 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-20T09:34:16.501 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:34:16.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:16.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:34:16.610 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:34:16.641 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:16.669 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:34:16.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:16.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:34:16.774 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:34:16.775 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:16 smithi026 ceph-mon[68309]: 7.5 scrub starts 2024-03-20T09:34:16.775 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:16 smithi026 ceph-mon[68309]: 7.2 scrub starts 2024-03-20T09:34:16.775 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:16 smithi026 ceph-mon[68309]: 7.5 scrub ok 2024-03-20T09:34:16.775 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:16 smithi026 ceph-mon[68309]: pgmap v2259: 47 pgs: 47 active+clean; 3.4 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 814 op/s 2024-03-20T09:34:16.805 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:16.829 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:34:16.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:16.832 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:34:16.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:16 smithi115 ceph-mon[72476]: 7.5 scrub starts 2024-03-20T09:34:16.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:16 smithi115 ceph-mon[72476]: 7.2 scrub starts 2024-03-20T09:34:16.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:16 smithi115 ceph-mon[72476]: 7.5 scrub ok 2024-03-20T09:34:16.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:16 smithi115 ceph-mon[72476]: pgmap v2259: 47 pgs: 47 active+clean; 3.4 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 814 op/s 2024-03-20T09:34:16.933 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:34:16.964 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:16.985 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:34:16.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:16.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:34:17.090 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:34:17.113 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:16 smithi122 ceph-mon[73495]: 7.5 scrub starts 2024-03-20T09:34:17.113 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:16 smithi122 ceph-mon[73495]: 7.2 scrub starts 2024-03-20T09:34:17.114 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:16 smithi122 ceph-mon[73495]: 7.5 scrub ok 2024-03-20T09:34:17.114 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:16 smithi122 ceph-mon[73495]: pgmap v2259: 47 pgs: 47 active+clean; 3.4 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 814 op/s 2024-03-20T09:34:17.115 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:17.137 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:34:17.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:17.144 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:34:17.245 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:34:17.280 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:17.305 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:34:17.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:17.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:34:17.409 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:34:17.438 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:17.460 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:34:17.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:17.463 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:34:17.563 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:34:17.598 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:17.620 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:34:17.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:17.628 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:34:17.729 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:34:17.829 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:18.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:17 smithi115 ceph-mon[72476]: 1.16 scrub starts 2024-03-20T09:34:18.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:17 smithi115 ceph-mon[72476]: 1.16 scrub ok 2024-03-20T09:34:18.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:17 smithi115 ceph-mon[72476]: 7.2 scrub ok 2024-03-20T09:34:18.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:17 smithi122 ceph-mon[73495]: 1.16 scrub starts 2024-03-20T09:34:18.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:17 smithi122 ceph-mon[73495]: 1.16 scrub ok 2024-03-20T09:34:18.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:17 smithi122 ceph-mon[73495]: 7.2 scrub ok 2024-03-20T09:34:18.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:17 smithi026 ceph-mon[68309]: 1.16 scrub starts 2024-03-20T09:34:18.215 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:17 smithi026 ceph-mon[68309]: 1.16 scrub ok 2024-03-20T09:34:18.215 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:17 smithi026 ceph-mon[68309]: 7.2 scrub ok 2024-03-20T09:34:18.623 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:34:18.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:18.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:34:18.741 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:34:18.767 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:18.776 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:34:18.777 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:34:18.791 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:34:18.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:18.798 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:34:18.899 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:34:18.929 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:18.957 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:34:18.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:18.960 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:34:19.061 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:34:19.114 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:18 smithi026 ceph-mon[68309]: 1.1c scrub starts 2024-03-20T09:34:19.114 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:18 smithi026 ceph-mon[68309]: 1.1c scrub ok 2024-03-20T09:34:19.114 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:18 smithi026 ceph-mon[68309]: 1.1d scrub starts 2024-03-20T09:34:19.114 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:18 smithi026 ceph-mon[68309]: 1.1d scrub ok 2024-03-20T09:34:19.114 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:18 smithi026 ceph-mon[68309]: from='client.16875 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:34:19.114 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:18 smithi026 ceph-mon[68309]: pgmap v2260: 47 pgs: 47 active+clean; 3.3 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 825 op/s 2024-03-20T09:34:19.128 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:19.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:18 smithi115 ceph-mon[72476]: 1.1c scrub starts 2024-03-20T09:34:19.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:18 smithi115 ceph-mon[72476]: 1.1c scrub ok 2024-03-20T09:34:19.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:18 smithi115 ceph-mon[72476]: 1.1d scrub starts 2024-03-20T09:34:19.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:18 smithi115 ceph-mon[72476]: 1.1d scrub ok 2024-03-20T09:34:19.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:18 smithi115 ceph-mon[72476]: from='client.16875 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:34:19.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:18 smithi115 ceph-mon[72476]: pgmap v2260: 47 pgs: 47 active+clean; 3.3 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 825 op/s 2024-03-20T09:34:19.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:18 smithi122 ceph-mon[73495]: 1.1c scrub starts 2024-03-20T09:34:19.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:18 smithi122 ceph-mon[73495]: 1.1c scrub ok 2024-03-20T09:34:19.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:18 smithi122 ceph-mon[73495]: 1.1d scrub starts 2024-03-20T09:34:19.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:18 smithi122 ceph-mon[73495]: 1.1d scrub ok 2024-03-20T09:34:19.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:18 smithi122 ceph-mon[73495]: from='client.16875 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:34:19.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:18 smithi122 ceph-mon[73495]: pgmap v2260: 47 pgs: 47 active+clean; 3.3 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 825 op/s 2024-03-20T09:34:20.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:19 smithi115 ceph-mon[72476]: 7.8 scrub starts 2024-03-20T09:34:20.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:19 smithi115 ceph-mon[72476]: 1.1b scrub starts 2024-03-20T09:34:20.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:19 smithi115 ceph-mon[72476]: 1.1b scrub ok 2024-03-20T09:34:20.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:19 smithi122 ceph-mon[73495]: 7.8 scrub starts 2024-03-20T09:34:20.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:19 smithi122 ceph-mon[73495]: 1.1b scrub starts 2024-03-20T09:34:20.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:19 smithi122 ceph-mon[73495]: 1.1b scrub ok 2024-03-20T09:34:20.215 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:19 smithi026 ceph-mon[68309]: 7.8 scrub starts 2024-03-20T09:34:20.215 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:19 smithi026 ceph-mon[68309]: 1.1b scrub starts 2024-03-20T09:34:20.215 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:19 smithi026 ceph-mon[68309]: 1.1b scrub ok 2024-03-20T09:34:20.387 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:34:20.425 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:34:20.470 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:34:20.532 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:34:20.769 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:34:20.812 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:34:20.851 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:34:20.891 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:34:20.941 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:34:20.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:20 smithi026 ceph-mon[68309]: 7.8 scrub ok 2024-03-20T09:34:20.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:20 smithi026 ceph-mon[68309]: 1.1a scrub starts 2024-03-20T09:34:20.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:20 smithi026 ceph-mon[68309]: 1.1a scrub ok 2024-03-20T09:34:20.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:20 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:34:20.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:20 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:34:20.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:20 smithi026 ceph-mon[68309]: 7.d scrub starts 2024-03-20T09:34:20.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:20 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3718110007' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:34:20.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:20 smithi026 ceph-mon[68309]: pgmap v2261: 47 pgs: 47 active+clean; 3.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 819 op/s 2024-03-20T09:34:21.007 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:34:21.042 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:34:21.076 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:34:21.097 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:34:21.097 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:34:21.097 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:34:21.097 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:34:21.097 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:34:21.097 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:34:21.097 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:34:21.098 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:34:21.098 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:34:21.098 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:34:21.098 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:34:21.098 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:34:21.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:20 smithi115 ceph-mon[72476]: 7.8 scrub ok 2024-03-20T09:34:21.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:20 smithi115 ceph-mon[72476]: 1.1a scrub starts 2024-03-20T09:34:21.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:20 smithi115 ceph-mon[72476]: 1.1a scrub ok 2024-03-20T09:34:21.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:20 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:34:21.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:20 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:34:21.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:20 smithi115 ceph-mon[72476]: 7.d scrub starts 2024-03-20T09:34:21.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:20 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3718110007' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:34:21.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:20 smithi115 ceph-mon[72476]: pgmap v2261: 47 pgs: 47 active+clean; 3.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 819 op/s 2024-03-20T09:34:21.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:20 smithi122 ceph-mon[73495]: 7.8 scrub ok 2024-03-20T09:34:21.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:20 smithi122 ceph-mon[73495]: 1.1a scrub starts 2024-03-20T09:34:21.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:20 smithi122 ceph-mon[73495]: 1.1a scrub ok 2024-03-20T09:34:21.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:20 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:34:21.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:20 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:34:21.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:20 smithi122 ceph-mon[73495]: 7.d scrub starts 2024-03-20T09:34:21.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:20 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3718110007' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:34:21.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:20 smithi122 ceph-mon[73495]: pgmap v2261: 47 pgs: 47 active+clean; 3.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 819 op/s 2024-03-20T09:34:21.617 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":2259,"stamp":"2024-03-20T09:34:16.474560+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3625648696,"num_objects":55317,"num_object_clones":0,"num_object_copies":165951,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55317,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":197903,"num_write_kb":8102913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":164851,"ondisk_log_size":164851,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":18555712,"kb_used_data":10708384,"kb_used_omap":197,"kb_used_meta":7846586,"kb_avail":1012640960,"statfs":{"total":1055945392128,"available":1036944343040,"internally_reserved":0,"allocated":10965385216,"data_stored":10962342972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":202077,"internal_metadata":8034904739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,7,3,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-471269376,"num_objects":-7191,"num_object_clones":0,"num_object_copies":-21573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7191,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.827811"},"pg_stats":[{"pgid":"1.1e","version":"20'137","reported_seq":1299,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:11.714419+0000","last_change":"2024-03-20T09:33:12.168422+0000","last_active":"2024-03-20T09:34:11.714419+0000","last_peered":"2024-03-20T09:34:11.714419+0000","last_clean":"2024-03-20T09:34:11.714419+0000","last_became_active":"2024-03-20T09:13:20.071655+0000","last_became_peered":"2024-03-20T09:13:20.071655+0000","last_unstale":"2024-03-20T09:34:11.714419+0000","last_undegraded":"2024-03-20T09:34:11.714419+0000","last_fullsized":"2024-03-20T09:34:11.714419+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:12.168339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:33:12.168339+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:33.973327+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'137","reported_seq":377,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:11.714465+0000","last_change":"2024-03-20T09:32:45.273878+0000","last_active":"2024-03-20T09:34:11.714465+0000","last_peered":"2024-03-20T09:34:11.714465+0000","last_clean":"2024-03-20T09:34:11.714465+0000","last_became_active":"2024-03-20T09:31:07.425574+0000","last_became_peered":"2024-03-20T09:31:07.425574+0000","last_unstale":"2024-03-20T09:34:11.714465+0000","last_undegraded":"2024-03-20T09:34:11.714465+0000","last_fullsized":"2024-03-20T09:34:11.714465+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:32:45.273763+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:29:02.585165+0000","last_clean_scrub_stamp":"2024-03-20T09:32:45.273763+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:15.549479+0000","scrub_duration":162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2],"acting":[11,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"20'137","reported_seq":1960,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-03-20T09:34:07.655400+0000","last_change":"2024-03-20T09:32:23.261615+0000","last_active":"2024-03-20T09:34:07.655400+0000","last_peered":"2024-03-20T09:34:07.655400+0000","last_clean":"2024-03-20T09:34:07.655400+0000","last_became_active":"2024-03-20T09:25:35.482959+0000","last_became_peered":"2024-03-20T09:25:35.482959+0000","last_unstale":"2024-03-20T09:34:07.655400+0000","last_undegraded":"2024-03-20T09:34:07.655400+0000","last_fullsized":"2024-03-20T09:34:07.655400+0000","mapping_epoch":744,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":745,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:32:23.261521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:32:23.261521+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,1],"acting":[8,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.19","version":"20'137","reported_seq":1143,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:11.721309+0000","last_change":"2024-03-20T09:32:28.348352+0000","last_active":"2024-03-20T09:34:11.721309+0000","last_peered":"2024-03-20T09:34:11.721309+0000","last_clean":"2024-03-20T09:34:11.721309+0000","last_became_active":"2024-03-20T09:28:22.591294+0000","last_became_peered":"2024-03-20T09:28:22.591294+0000","last_unstale":"2024-03-20T09:34:11.721309+0000","last_undegraded":"2024-03-20T09:34:11.721309+0000","last_fullsized":"2024-03-20T09:34:11.721309+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:32:28.348215+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:30:56.201554+0000","last_clean_scrub_stamp":"2024-03-20T09:32:28.348215+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.18","version":"20'137","reported_seq":1445,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:11.709117+0000","last_change":"2024-03-20T09:33:40.415955+0000","last_active":"2024-03-20T09:34:11.709117+0000","last_peered":"2024-03-20T09:34:11.709117+0000","last_clean":"2024-03-20T09:34:11.709117+0000","last_became_active":"2024-03-20T09:10:35.277481+0000","last_became_peered":"2024-03-20T09:10:35.277481+0000","last_unstale":"2024-03-20T09:34:11.709117+0000","last_undegraded":"2024-03-20T09:34:11.709117+0000","last_fullsized":"2024-03-20T09:34:11.709117+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:40.415846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:33:40.415846+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:52.941527+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1b","version":"20'137","reported_seq":1797,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-03-20T09:34:07.655330+0000","last_change":"2024-03-20T09:33:13.685851+0000","last_active":"2024-03-20T09:34:07.655330+0000","last_peered":"2024-03-20T09:34:07.655330+0000","last_clean":"2024-03-20T09:34:07.655330+0000","last_became_active":"2024-03-20T09:24:16.445287+0000","last_became_peered":"2024-03-20T09:24:16.445287+0000","last_unstale":"2024-03-20T09:34:07.655330+0000","last_undegraded":"2024-03-20T09:34:07.655330+0000","last_fullsized":"2024-03-20T09:34:07.655330+0000","mapping_epoch":718,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:13.685707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:33:13.685707+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:14.072191+0000","scrub_duration":188,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"7.e","version":"929'12318","reported_seq":12526,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:13.132680+0000","last_change":"2024-03-20T09:32:36.556340+0000","last_active":"2024-03-20T09:34:13.132680+0000","last_peered":"2024-03-20T09:34:13.132680+0000","last_clean":"2024-03-20T09:34:13.132680+0000","last_became_active":"2024-03-20T09:31:07.427099+0000","last_became_peered":"2024-03-20T09:31:07.427099+0000","last_unstale":"2024-03-20T09:34:13.132680+0000","last_undegraded":"2024-03-20T09:34:13.132680+0000","last_fullsized":"2024-03-20T09:34:13.132680+0000","mapping_epoch":871,"log_start":"880'2300","ondisk_log_start":"880'2300","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"904'7287","last_scrub_stamp":"2024-03-20T09:32:36.556054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:36.556054+0000","objects_scrubbed":7222,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2205,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230424576,"num_objects":3516,"num_object_clones":0,"num_object_copies":10548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12318,"num_write_kb":506688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,5],"acting":[2,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"20'137","reported_seq":1437,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:11.709194+0000","last_change":"2024-03-20T09:32:28.004445+0000","last_active":"2024-03-20T09:34:11.709194+0000","last_peered":"2024-03-20T09:34:11.709194+0000","last_clean":"2024-03-20T09:34:11.709194+0000","last_became_active":"2024-03-20T09:10:35.215772+0000","last_became_peered":"2024-03-20T09:10:35.215772+0000","last_unstale":"2024-03-20T09:34:11.709194+0000","last_undegraded":"2024-03-20T09:34:11.709194+0000","last_fullsized":"2024-03-20T09:34:11.709194+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:32:28.004327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:32:28.004327+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:14.302825+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"929'12201","reported_seq":12409,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:11.747273+0000","last_change":"2024-03-20T09:32:38.283331+0000","last_active":"2024-03-20T09:34:11.747273+0000","last_peered":"2024-03-20T09:34:11.747273+0000","last_clean":"2024-03-20T09:34:11.747273+0000","last_became_active":"2024-03-20T09:31:07.534696+0000","last_became_peered":"2024-03-20T09:31:07.534696+0000","last_unstale":"2024-03-20T09:34:11.747273+0000","last_undegraded":"2024-03-20T09:34:11.747273+0000","last_fullsized":"2024-03-20T09:34:11.747273+0000","mapping_epoch":871,"log_start":"879'2200","ondisk_log_start":"879'2200","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"905'7295","last_scrub_stamp":"2024-03-20T09:32:38.283112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:38.283112+0000","objects_scrubbed":7235,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:13.967965+0000","scrub_duration":2079,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237305856,"num_objects":3621,"num_object_clones":0,"num_object_copies":10863,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12201,"num_write_kb":506304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"20'137","reported_seq":2029,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:11.722393+0000","last_change":"2024-03-20T09:33:14.407075+0000","last_active":"2024-03-20T09:34:11.722393+0000","last_peered":"2024-03-20T09:34:11.722393+0000","last_clean":"2024-03-20T09:34:11.722393+0000","last_became_active":"2024-03-20T09:20:25.341750+0000","last_became_peered":"2024-03-20T09:20:25.341750+0000","last_unstale":"2024-03-20T09:34:11.722393+0000","last_undegraded":"2024-03-20T09:34:11.722393+0000","last_fullsized":"2024-03-20T09:34:11.722393+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:14.406974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:33:14.406974+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:29.321937+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,2],"acting":[9,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.c","version":"929'12431","reported_seq":12680,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:16.257476+0000","last_change":"2024-03-20T09:34:15.101306+0000","last_active":"2024-03-20T09:34:16.257476+0000","last_peered":"2024-03-20T09:34:16.257476+0000","last_clean":"2024-03-20T09:34:16.257476+0000","last_became_active":"2024-03-20T09:31:07.426984+0000","last_became_peered":"2024-03-20T09:31:07.426984+0000","last_unstale":"2024-03-20T09:34:16.257476+0000","last_undegraded":"2024-03-20T09:34:16.257476+0000","last_fullsized":"2024-03-20T09:34:16.257476+0000","mapping_epoch":871,"log_start":"880'2400","ondisk_log_start":"880'2400","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"929'12361","last_scrub_stamp":"2024-03-20T09:34:15.101159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:34:15.101159+0000","objects_scrubbed":3425,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:35:30.796184+0000","scrub_duration":751,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218693632,"num_objects":3337,"num_object_clones":0,"num_object_copies":10011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12431,"num_write_kb":504576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,11],"acting":[6,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"20'137","reported_seq":718,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:11.722406+0000","last_change":"2024-03-20T09:33:10.442796+0000","last_active":"2024-03-20T09:34:11.722406+0000","last_peered":"2024-03-20T09:34:11.722406+0000","last_clean":"2024-03-20T09:34:11.722406+0000","last_became_active":"2024-03-20T09:24:16.444232+0000","last_became_peered":"2024-03-20T09:24:16.444232+0000","last_unstale":"2024-03-20T09:34:11.722406+0000","last_undegraded":"2024-03-20T09:34:11.722406+0000","last_fullsized":"2024-03-20T09:34:11.722406+0000","mapping_epoch":718,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:10.442719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:33:10.442719+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:26.643584+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.2","version":"929'12527","reported_seq":12727,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:15.291154+0000","last_change":"2024-03-20T09:32:21.786210+0000","last_active":"2024-03-20T09:34:15.291154+0000","last_peered":"2024-03-20T09:34:15.291154+0000","last_clean":"2024-03-20T09:34:15.291154+0000","last_became_active":"2024-03-20T09:31:07.422232+0000","last_became_peered":"2024-03-20T09:31:07.422232+0000","last_unstale":"2024-03-20T09:34:15.291154+0000","last_undegraded":"2024-03-20T09:34:15.291154+0000","last_fullsized":"2024-03-20T09:34:15.291154+0000","mapping_epoch":871,"log_start":"880'2500","ondisk_log_start":"880'2500","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"899'6455","last_scrub_stamp":"2024-03-20T09:32:21.785983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:21.785983+0000","objects_scrubbed":6398,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1934,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223281152,"num_objects":3407,"num_object_clones":0,"num_object_copies":10221,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12527,"num_write_kb":509888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"20'137","reported_seq":1967,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:11.714435+0000","last_change":"2024-03-20T09:33:10.120697+0000","last_active":"2024-03-20T09:34:11.714435+0000","last_peered":"2024-03-20T09:34:11.714435+0000","last_clean":"2024-03-20T09:34:11.714435+0000","last_became_active":"2024-03-20T09:12:14.306384+0000","last_became_peered":"2024-03-20T09:12:14.306384+0000","last_unstale":"2024-03-20T09:34:11.714435+0000","last_undegraded":"2024-03-20T09:34:11.714435+0000","last_fullsized":"2024-03-20T09:34:11.714435+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:10.120609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:33:10.120609+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:37.759793+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.3","version":"929'12467","reported_seq":12732,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:15.350754+0000","last_change":"2024-03-20T09:33:29.905371+0000","last_active":"2024-03-20T09:34:15.350754+0000","last_peered":"2024-03-20T09:34:15.350754+0000","last_clean":"2024-03-20T09:34:15.350754+0000","last_became_active":"2024-03-20T09:31:07.422509+0000","last_became_peered":"2024-03-20T09:31:07.422509+0000","last_unstale":"2024-03-20T09:34:15.350754+0000","last_undegraded":"2024-03-20T09:34:15.350754+0000","last_fullsized":"2024-03-20T09:34:15.350754+0000","mapping_epoch":871,"log_start":"880'2400","ondisk_log_start":"880'2400","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"920'10157","last_scrub_stamp":"2024-03-20T09:33:29.905190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:33:29.905190+0000","objects_scrubbed":5720,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:49.933850+0000","scrub_duration":1973,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220135424,"num_objects":3359,"num_object_clones":0,"num_object_copies":10077,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12467,"num_write_kb":506432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"20'137","reported_seq":614,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:11.722885+0000","last_change":"2024-03-20T09:32:37.603204+0000","last_active":"2024-03-20T09:34:11.722885+0000","last_peered":"2024-03-20T09:34:11.722885+0000","last_clean":"2024-03-20T09:34:11.722885+0000","last_became_active":"2024-03-20T09:28:22.587095+0000","last_became_peered":"2024-03-20T09:28:22.587095+0000","last_unstale":"2024-03-20T09:34:11.722885+0000","last_undegraded":"2024-03-20T09:34:11.722885+0000","last_fullsized":"2024-03-20T09:34:11.722885+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:32:37.603082+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:19:03.805185+0000","last_clean_scrub_stamp":"2024-03-20T09:32:37.603082+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:12.240772+0000","scrub_duration":191,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,11],"acting":[2,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.0","version":"929'12264","reported_seq":12466,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:14.795346+0000","last_change":"2024-03-20T09:32:26.456393+0000","last_active":"2024-03-20T09:34:14.795346+0000","last_peered":"2024-03-20T09:34:14.795346+0000","last_clean":"2024-03-20T09:34:14.795346+0000","last_became_active":"2024-03-20T09:31:07.426307+0000","last_became_peered":"2024-03-20T09:31:07.426307+0000","last_unstale":"2024-03-20T09:34:14.795346+0000","last_undegraded":"2024-03-20T09:34:14.795346+0000","last_fullsized":"2024-03-20T09:34:14.795346+0000","mapping_epoch":871,"log_start":"879'2200","ondisk_log_start":"879'2200","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"901'6561","last_scrub_stamp":"2024-03-20T09:32:26.456235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:26.456235+0000","objects_scrubbed":6499,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1520,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222822400,"num_objects":3400,"num_object_clones":0,"num_object_copies":10200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12264,"num_write_kb":501248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"20'137","reported_seq":1833,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:11.722575+0000","last_change":"2024-03-20T09:33:13.369995+0000","last_active":"2024-03-20T09:34:11.722575+0000","last_peered":"2024-03-20T09:34:11.722575+0000","last_clean":"2024-03-20T09:34:11.722575+0000","last_became_active":"2024-03-20T09:28:22.586751+0000","last_became_peered":"2024-03-20T09:28:22.586751+0000","last_unstale":"2024-03-20T09:34:11.722575+0000","last_undegraded":"2024-03-20T09:34:11.722575+0000","last_fullsized":"2024-03-20T09:34:11.722575+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:13.369863+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:24:31.549603+0000","last_clean_scrub_stamp":"2024-03-20T09:33:13.369863+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:39.364154+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.5","version":"929'12314","reported_seq":12514,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:15.504980+0000","last_change":"2024-03-20T09:32:23.538659+0000","last_active":"2024-03-20T09:34:15.504980+0000","last_peered":"2024-03-20T09:34:15.504980+0000","last_clean":"2024-03-20T09:34:15.504980+0000","last_became_active":"2024-03-20T09:31:07.427044+0000","last_became_peered":"2024-03-20T09:31:07.427044+0000","last_unstale":"2024-03-20T09:34:15.504980+0000","last_undegraded":"2024-03-20T09:34:15.504980+0000","last_fullsized":"2024-03-20T09:34:15.504980+0000","mapping_epoch":871,"log_start":"880'2300","ondisk_log_start":"880'2300","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"900'6405","last_scrub_stamp":"2024-03-20T09:32:23.538400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:23.538400+0000","objects_scrubbed":6353,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:15.549459+0000","scrub_duration":1492,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228720640,"num_objects":3490,"num_object_clones":0,"num_object_copies":10470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12314,"num_write_kb":505728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,0],"acting":[11,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.3","version":"20'137","reported_seq":1033,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:11.713689+0000","last_change":"2024-03-20T09:33:26.640631+0000","last_active":"2024-03-20T09:34:11.713689+0000","last_peered":"2024-03-20T09:34:11.713689+0000","last_clean":"2024-03-20T09:34:11.713689+0000","last_became_active":"2024-03-20T09:31:07.424588+0000","last_became_peered":"2024-03-20T09:31:07.424588+0000","last_unstale":"2024-03-20T09:34:11.713689+0000","last_undegraded":"2024-03-20T09:34:11.713689+0000","last_fullsized":"2024-03-20T09:34:11.713689+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:26.640505+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:30:47.812125+0000","last_clean_scrub_stamp":"2024-03-20T09:33:26.640505+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:56.230688+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"929'12328","reported_seq":12540,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:11.782203+0000","last_change":"2024-03-20T09:32:43.005957+0000","last_active":"2024-03-20T09:34:11.782203+0000","last_peered":"2024-03-20T09:34:11.782203+0000","last_clean":"2024-03-20T09:34:11.782203+0000","last_became_active":"2024-03-20T09:31:07.535053+0000","last_became_peered":"2024-03-20T09:31:07.535053+0000","last_unstale":"2024-03-20T09:34:11.782203+0000","last_undegraded":"2024-03-20T09:34:11.782203+0000","last_fullsized":"2024-03-20T09:34:11.782203+0000","mapping_epoch":871,"log_start":"880'2300","ondisk_log_start":"880'2300","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"907'7583","last_scrub_stamp":"2024-03-20T09:32:43.005812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:43.005812+0000","objects_scrubbed":7513,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:13.967985+0000","scrub_duration":2762,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229900288,"num_objects":3508,"num_object_clones":0,"num_object_copies":10524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12328,"num_write_kb":506752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":1939,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:11.714334+0000","last_change":"2024-03-20T09:33:19.153580+0000","last_active":"2024-03-20T09:34:11.714334+0000","last_peered":"2024-03-20T09:34:11.714334+0000","last_clean":"2024-03-20T09:34:11.714334+0000","last_became_active":"2024-03-20T09:12:14.305989+0000","last_became_peered":"2024-03-20T09:12:14.305989+0000","last_unstale":"2024-03-20T09:34:11.714334+0000","last_undegraded":"2024-03-20T09:34:11.714334+0000","last_fullsized":"2024-03-20T09:34:11.714334+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:19.153464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:33:19.153464+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:32.868996+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.4","version":"929'12326","reported_seq":12528,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:11.721770+0000","last_change":"2024-03-20T09:33:16.169687+0000","last_active":"2024-03-20T09:34:11.721770+0000","last_peered":"2024-03-20T09:34:11.721770+0000","last_clean":"2024-03-20T09:34:11.721770+0000","last_became_active":"2024-03-20T09:31:07.424820+0000","last_became_peered":"2024-03-20T09:31:07.424820+0000","last_unstale":"2024-03-20T09:34:11.721770+0000","last_undegraded":"2024-03-20T09:34:11.721770+0000","last_fullsized":"2024-03-20T09:34:11.721770+0000","mapping_epoch":871,"log_start":"880'2300","ondisk_log_start":"880'2300","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"917'9492","last_scrub_stamp":"2024-03-20T09:33:16.169522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:33:16.169522+0000","objects_scrubbed":6563,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:40.296707+0000","scrub_duration":1975,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241696768,"num_objects":3688,"num_object_clones":0,"num_object_copies":11064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12326,"num_write_kb":512448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"20'137","reported_seq":2203,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:11.721418+0000","last_change":"2024-03-20T09:33:33.249232+0000","last_active":"2024-03-20T09:34:11.721418+0000","last_peered":"2024-03-20T09:34:11.721418+0000","last_clean":"2024-03-20T09:34:11.721418+0000","last_became_active":"2024-03-20T09:19:54.492879+0000","last_became_peered":"2024-03-20T09:19:54.492879+0000","last_unstale":"2024-03-20T09:34:11.721418+0000","last_undegraded":"2024-03-20T09:34:11.721418+0000","last_fullsized":"2024-03-20T09:34:11.721418+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:33.249098+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:19.513511+0000","last_clean_scrub_stamp":"2024-03-20T09:33:33.249098+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:54.587251+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"929'12220","reported_seq":12422,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:16.300351+0000","last_change":"2024-03-20T09:32:28.146865+0000","last_active":"2024-03-20T09:34:16.300351+0000","last_peered":"2024-03-20T09:34:16.300351+0000","last_clean":"2024-03-20T09:34:16.300351+0000","last_became_active":"2024-03-20T09:31:07.534591+0000","last_became_peered":"2024-03-20T09:31:07.534591+0000","last_unstale":"2024-03-20T09:34:16.300351+0000","last_undegraded":"2024-03-20T09:34:16.300351+0000","last_fullsized":"2024-03-20T09:34:16.300351+0000","mapping_epoch":871,"log_start":"879'2200","ondisk_log_start":"879'2200","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"902'6591","last_scrub_stamp":"2024-03-20T09:32:28.146586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:28.146586+0000","objects_scrubbed":6548,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1556,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208142336,"num_objects":3176,"num_object_clones":0,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12220,"num_write_kb":492672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,3],"acting":[6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"20'137","reported_seq":1380,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:11.713645+0000","last_change":"2024-03-20T09:33:14.457333+0000","last_active":"2024-03-20T09:34:11.713645+0000","last_peered":"2024-03-20T09:34:11.713645+0000","last_clean":"2024-03-20T09:34:11.713645+0000","last_became_active":"2024-03-20T09:15:34.225022+0000","last_became_peered":"2024-03-20T09:15:34.225022+0000","last_unstale":"2024-03-20T09:34:11.713645+0000","last_undegraded":"2024-03-20T09:34:11.713645+0000","last_fullsized":"2024-03-20T09:34:11.713645+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:14.457198+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:20:12.169658+0000","last_clean_scrub_stamp":"2024-03-20T09:33:14.457198+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:16.366898+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,5],"acting":[6,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"929'12515","reported_seq":12724,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:15.605133+0000","last_change":"2024-03-20T09:32:37.563227+0000","last_active":"2024-03-20T09:34:15.605133+0000","last_peered":"2024-03-20T09:34:15.605133+0000","last_clean":"2024-03-20T09:34:15.605133+0000","last_became_active":"2024-03-20T09:31:07.421909+0000","last_became_peered":"2024-03-20T09:31:07.421909+0000","last_unstale":"2024-03-20T09:34:15.605133+0000","last_undegraded":"2024-03-20T09:34:15.605133+0000","last_fullsized":"2024-03-20T09:34:15.605133+0000","mapping_epoch":871,"log_start":"880'2500","ondisk_log_start":"880'2500","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"905'7297","last_scrub_stamp":"2024-03-20T09:32:37.563105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:37.563105+0000","objects_scrubbed":7240,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:15.772759+0000","scrub_duration":1719,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216858624,"num_objects":3309,"num_object_clones":0,"num_object_copies":9927,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12515,"num_write_kb":506368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"20'137","reported_seq":1918,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:11.713784+0000","last_change":"2024-03-20T09:33:13.684716+0000","last_active":"2024-03-20T09:34:11.713784+0000","last_peered":"2024-03-20T09:34:11.713784+0000","last_clean":"2024-03-20T09:34:11.713784+0000","last_became_active":"2024-03-20T09:28:22.587861+0000","last_became_peered":"2024-03-20T09:28:22.587861+0000","last_unstale":"2024-03-20T09:34:11.713784+0000","last_undegraded":"2024-03-20T09:34:11.713784+0000","last_fullsized":"2024-03-20T09:34:11.713784+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:13.684628+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:14:10.560890+0000","last_clean_scrub_stamp":"2024-03-20T09:33:13.684628+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:35.420402+0000","scrub_duration":213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.d","version":"929'12455","reported_seq":12659,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:16.246415+0000","last_change":"2024-03-20T09:32:27.343018+0000","last_active":"2024-03-20T09:34:16.246415+0000","last_peered":"2024-03-20T09:34:16.246415+0000","last_clean":"2024-03-20T09:34:16.246415+0000","last_became_active":"2024-03-20T09:31:07.425062+0000","last_became_peered":"2024-03-20T09:31:07.425062+0000","last_unstale":"2024-03-20T09:34:16.246415+0000","last_undegraded":"2024-03-20T09:34:16.246415+0000","last_fullsized":"2024-03-20T09:34:16.246415+0000","mapping_epoch":871,"log_start":"880'2400","ondisk_log_start":"880'2400","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"901'6691","last_scrub_stamp":"2024-03-20T09:32:27.342772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:27.342772+0000","objects_scrubbed":6641,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1744,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222953496,"num_objects":3403,"num_object_clones":0,"num_object_copies":10209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3403,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12455,"num_write_kb":507393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"20'137","reported_seq":1751,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:11.717730+0000","last_change":"2024-03-20T09:33:30.436327+0000","last_active":"2024-03-20T09:34:11.717730+0000","last_peered":"2024-03-20T09:34:11.717730+0000","last_clean":"2024-03-20T09:34:11.717730+0000","last_became_active":"2024-03-20T09:13:20.070596+0000","last_became_peered":"2024-03-20T09:13:20.070596+0000","last_unstale":"2024-03-20T09:34:11.717730+0000","last_undegraded":"2024-03-20T09:34:11.717730+0000","last_fullsized":"2024-03-20T09:34:11.717730+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:30.436196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:33:30.436196+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:34.613856+0000","scrub_duration":208,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.8","version":"929'12176","reported_seq":12379,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:13.116286+0000","last_change":"2024-03-20T09:32:28.110802+0000","last_active":"2024-03-20T09:34:13.116286+0000","last_peered":"2024-03-20T09:34:13.116286+0000","last_clean":"2024-03-20T09:34:13.116286+0000","last_became_active":"2024-03-20T09:31:07.534911+0000","last_became_peered":"2024-03-20T09:31:07.534911+0000","last_unstale":"2024-03-20T09:34:13.116286+0000","last_undegraded":"2024-03-20T09:34:13.116286+0000","last_fullsized":"2024-03-20T09:34:13.116286+0000","mapping_epoch":871,"log_start":"879'2100","ondisk_log_start":"879'2100","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"902'6716","last_scrub_stamp":"2024-03-20T09:32:28.110647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:28.110647+0000","objects_scrubbed":6665,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1785,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231604224,"num_objects":3534,"num_object_clones":0,"num_object_copies":10602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12176,"num_write_kb":502720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"20'137","reported_seq":1183,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:11.719365+0000","last_change":"2024-03-20T09:33:09.882859+0000","last_active":"2024-03-20T09:34:11.719365+0000","last_peered":"2024-03-20T09:34:11.719365+0000","last_clean":"2024-03-20T09:34:11.719365+0000","last_became_active":"2024-03-20T09:19:54.506134+0000","last_became_peered":"2024-03-20T09:19:54.506134+0000","last_unstale":"2024-03-20T09:34:11.719365+0000","last_undegraded":"2024-03-20T09:34:11.719365+0000","last_fullsized":"2024-03-20T09:34:11.719365+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:09.882745+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:29:53.362224+0000","last_clean_scrub_stamp":"2024-03-20T09:33:09.882745+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:29.991713+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"929'12436","reported_seq":12643,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:11.758060+0000","last_change":"2024-03-20T09:32:33.285447+0000","last_active":"2024-03-20T09:34:11.758060+0000","last_peered":"2024-03-20T09:34:11.758060+0000","last_clean":"2024-03-20T09:34:11.758060+0000","last_became_active":"2024-03-20T09:31:07.534654+0000","last_became_peered":"2024-03-20T09:31:07.534654+0000","last_unstale":"2024-03-20T09:34:11.758060+0000","last_undegraded":"2024-03-20T09:34:11.758060+0000","last_fullsized":"2024-03-20T09:34:11.758060+0000","mapping_epoch":871,"log_start":"880'2400","ondisk_log_start":"880'2400","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"904'7122","last_scrub_stamp":"2024-03-20T09:32:33.285246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:33.285246+0000","objects_scrubbed":7070,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2081,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230948864,"num_objects":3524,"num_object_clones":0,"num_object_copies":10572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12436,"num_write_kb":510720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"20'137","reported_seq":1947,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:11.713710+0000","last_change":"2024-03-20T09:33:33.560515+0000","last_active":"2024-03-20T09:34:11.713710+0000","last_peered":"2024-03-20T09:34:11.713710+0000","last_clean":"2024-03-20T09:34:11.713710+0000","last_became_active":"2024-03-20T09:19:54.524397+0000","last_became_peered":"2024-03-20T09:19:54.524397+0000","last_unstale":"2024-03-20T09:34:11.713710+0000","last_undegraded":"2024-03-20T09:34:11.713710+0000","last_fullsized":"2024-03-20T09:34:11.713710+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:33.560343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:33:33.560343+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:56.357887+0000","scrub_duration":184,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"929'12257","reported_seq":12524,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:13.035925+0000","last_change":"2024-03-20T09:33:40.727393+0000","last_active":"2024-03-20T09:34:13.035925+0000","last_peered":"2024-03-20T09:34:13.035925+0000","last_clean":"2024-03-20T09:34:13.035925+0000","last_became_active":"2024-03-20T09:31:07.428978+0000","last_became_peered":"2024-03-20T09:31:07.428978+0000","last_unstale":"2024-03-20T09:34:13.035925+0000","last_undegraded":"2024-03-20T09:34:13.035925+0000","last_fullsized":"2024-03-20T09:34:13.035925+0000","mapping_epoch":871,"log_start":"879'2200","ondisk_log_start":"879'2200","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"921'10653","last_scrub_stamp":"2024-03-20T09:33:40.727237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:33:40.727237+0000","objects_scrubbed":5157,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:35:05.691874+0000","scrub_duration":1252,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231145472,"num_objects":3527,"num_object_clones":0,"num_object_copies":10581,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12257,"num_write_kb":505088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.c","version":"570'138","reported_seq":1924,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:11.713217+0000","last_change":"2024-03-20T09:33:33.082457+0000","last_active":"2024-03-20T09:34:11.713217+0000","last_peered":"2024-03-20T09:34:11.713217+0000","last_clean":"2024-03-20T09:34:11.713217+0000","last_became_active":"2024-03-20T09:13:27.087686+0000","last_became_peered":"2024-03-20T09:13:27.087686+0000","last_unstale":"2024-03-20T09:34:11.713217+0000","last_undegraded":"2024-03-20T09:34:11.713217+0000","last_fullsized":"2024-03-20T09:34:11.713217+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"570'138","last_scrub_stamp":"2024-03-20T09:33:33.082331+0000","last_deep_scrub":"570'138","last_deep_scrub_stamp":"2024-03-20T09:16:13.725035+0000","last_clean_scrub_stamp":"2024-03-20T09:33:33.082331+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:40.049713+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"929'12668","reported_seq":12872,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:14.771047+0000","last_change":"2024-03-20T09:32:27.484409+0000","last_active":"2024-03-20T09:34:14.771047+0000","last_peered":"2024-03-20T09:34:14.771047+0000","last_clean":"2024-03-20T09:34:14.771047+0000","last_became_active":"2024-03-20T09:31:07.424767+0000","last_became_peered":"2024-03-20T09:31:07.424767+0000","last_unstale":"2024-03-20T09:34:14.771047+0000","last_undegraded":"2024-03-20T09:34:14.771047+0000","last_fullsized":"2024-03-20T09:34:14.771047+0000","mapping_epoch":871,"log_start":"880'2600","ondisk_log_start":"880'2600","created":871,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":0,"last_scrub":"901'6848","last_scrub_stamp":"2024-03-20T09:32:27.484264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:31:06.406431+0000","last_clean_scrub_stamp":"2024-03-20T09:32:27.484264+0000","objects_scrubbed":6805,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230424576,"num_objects":3516,"num_object_clones":0,"num_object_copies":10548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12668,"num_write_kb":517888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,11],"acting":[5,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"20'137","reported_seq":1489,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:11.719242+0000","last_change":"2024-03-20T09:33:11.988607+0000","last_active":"2024-03-20T09:34:11.719242+0000","last_peered":"2024-03-20T09:34:11.719242+0000","last_clean":"2024-03-20T09:34:11.719242+0000","last_became_active":"2024-03-20T09:19:54.502529+0000","last_became_peered":"2024-03-20T09:19:54.502529+0000","last_unstale":"2024-03-20T09:34:11.719242+0000","last_undegraded":"2024-03-20T09:34:11.719242+0000","last_fullsized":"2024-03-20T09:34:11.719242+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:11.988521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:33:11.988521+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:25.929908+0000","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"20'137","reported_seq":1778,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:11.709160+0000","last_change":"2024-03-20T09:32:27.000506+0000","last_active":"2024-03-20T09:34:11.709160+0000","last_peered":"2024-03-20T09:34:11.709160+0000","last_clean":"2024-03-20T09:34:11.709160+0000","last_became_active":"2024-03-20T09:10:34.096758+0000","last_became_peered":"2024-03-20T09:10:34.096758+0000","last_unstale":"2024-03-20T09:34:11.709160+0000","last_undegraded":"2024-03-20T09:34:11.709160+0000","last_fullsized":"2024-03-20T09:34:11.709160+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:32:27.000415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:32:27.000415+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:14.302789+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"20'137","reported_seq":530,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:11.717774+0000","last_change":"2024-03-20T09:33:11.373371+0000","last_active":"2024-03-20T09:34:11.717774+0000","last_peered":"2024-03-20T09:34:11.717774+0000","last_clean":"2024-03-20T09:34:11.717774+0000","last_became_active":"2024-03-20T09:31:07.425519+0000","last_became_peered":"2024-03-20T09:31:07.425519+0000","last_unstale":"2024-03-20T09:34:11.717774+0000","last_undegraded":"2024-03-20T09:34:11.717774+0000","last_fullsized":"2024-03-20T09:34:11.717774+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:11.373245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:33:11.373245+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:28.744893+0000","scrub_duration":229,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.10","version":"20'137","reported_seq":1332,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:11.709234+0000","last_change":"2024-03-20T09:33:38.307496+0000","last_active":"2024-03-20T09:34:11.709234+0000","last_peered":"2024-03-20T09:34:11.709234+0000","last_clean":"2024-03-20T09:34:11.709234+0000","last_became_active":"2024-03-20T09:28:22.584176+0000","last_became_peered":"2024-03-20T09:28:22.584176+0000","last_unstale":"2024-03-20T09:34:11.709234+0000","last_undegraded":"2024-03-20T09:34:11.709234+0000","last_fullsized":"2024-03-20T09:34:11.709234+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:38.307394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:33:38.307394+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:45.805742+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"20'137","reported_seq":1722,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:11.721388+0000","last_change":"2024-03-20T09:32:25.305425+0000","last_active":"2024-03-20T09:34:11.721388+0000","last_peered":"2024-03-20T09:34:11.721388+0000","last_clean":"2024-03-20T09:34:11.721388+0000","last_became_active":"2024-03-20T09:20:25.342982+0000","last_became_peered":"2024-03-20T09:20:25.342982+0000","last_unstale":"2024-03-20T09:34:11.721388+0000","last_undegraded":"2024-03-20T09:34:11.721388+0000","last_fullsized":"2024-03-20T09:34:11.721388+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:32:25.305287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:32:25.305287+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"20'137","reported_seq":447,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:11.722029+0000","last_change":"2024-03-20T09:32:42.323789+0000","last_active":"2024-03-20T09:34:11.722029+0000","last_peered":"2024-03-20T09:34:11.722029+0000","last_clean":"2024-03-20T09:34:11.722029+0000","last_became_active":"2024-03-20T09:31:07.424851+0000","last_became_peered":"2024-03-20T09:31:07.424851+0000","last_unstale":"2024-03-20T09:34:11.722029+0000","last_undegraded":"2024-03-20T09:34:11.722029+0000","last_fullsized":"2024-03-20T09:34:11.722029+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:32:42.323661+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:16:15.448920+0000","last_clean_scrub_stamp":"2024-03-20T09:32:42.323661+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.17","version":"20'137","reported_seq":1906,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-03-20T09:34:07.658695+0000","last_change":"2024-03-20T09:33:39.823553+0000","last_active":"2024-03-20T09:34:07.658695+0000","last_peered":"2024-03-20T09:34:07.658695+0000","last_clean":"2024-03-20T09:34:07.658695+0000","last_became_active":"2024-03-20T09:28:22.588303+0000","last_became_peered":"2024-03-20T09:28:22.588303+0000","last_unstale":"2024-03-20T09:34:07.658695+0000","last_undegraded":"2024-03-20T09:34:07.658695+0000","last_fullsized":"2024-03-20T09:34:07.658695+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:39.823418+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:23.640533+0000","last_clean_scrub_stamp":"2024-03-20T09:33:39.823418+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:58.632784+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"20'137","reported_seq":1957,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:12.331741+0000","last_change":"2024-03-20T09:34:12.331782+0000","last_active":"2024-03-20T09:34:12.331741+0000","last_peered":"2024-03-20T09:34:12.331741+0000","last_clean":"2024-03-20T09:34:12.331741+0000","last_became_active":"2024-03-20T09:20:25.349082+0000","last_became_peered":"2024-03-20T09:20:25.349082+0000","last_unstale":"2024-03-20T09:34:12.331741+0000","last_undegraded":"2024-03-20T09:34:12.331741+0000","last_fullsized":"2024-03-20T09:34:12.331741+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:34:12.331653+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:30:54.106826+0000","last_clean_scrub_stamp":"2024-03-20T09:34:12.331653+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:35:35.383257+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"20'137","reported_seq":1230,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:11.728660+0000","last_change":"2024-03-20T09:33:09.047464+0000","last_active":"2024-03-20T09:34:11.728660+0000","last_peered":"2024-03-20T09:34:11.728660+0000","last_clean":"2024-03-20T09:34:11.728660+0000","last_became_active":"2024-03-20T09:12:14.307581+0000","last_became_peered":"2024-03-20T09:12:14.307581+0000","last_unstale":"2024-03-20T09:34:11.728660+0000","last_undegraded":"2024-03-20T09:34:11.728660+0000","last_fullsized":"2024-03-20T09:34:11.728660+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:33:09.047372+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:23:41.251546+0000","last_clean_scrub_stamp":"2024-03-20T09:33:09.047372+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:37.240972+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"20'137","reported_seq":1488,"reported_epoch":929,"state":"active+clean","last_fresh":"2024-03-20T09:34:11.719121+0000","last_change":"2024-03-20T09:32:38.833000+0000","last_active":"2024-03-20T09:34:11.719121+0000","last_peered":"2024-03-20T09:34:11.719121+0000","last_clean":"2024-03-20T09:34:11.719121+0000","last_became_active":"2024-03-20T09:19:54.502391+0000","last_became_peered":"2024-03-20T09:19:54.502391+0000","last_unstale":"2024-03-20T09:34:11.719121+0000","last_undegraded":"2024-03-20T09:34:11.719121+0000","last_fullsized":"2024-03-20T09:34:11.719121+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:32:38.832878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:32:38.832878+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:34:15.764551+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":3625058328,"num_objects":55315,"num_object_clones":0,"num_object_copies":165945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":197903,"num_write_kb":8102913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10848645120,"data_stored":10848632904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160603,"ondisk_log_size":160603,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4248,"ondisk_log_size":4248,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860875,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1955908,"kb_used_data":1108412,"kb_used_omap":1,"kb_used_meta":847486,"kb_avail":91789244,"statfs":{"total":95995035648,"available":93992185856,"internally_reserved":0,"allocated":1135013888,"data_stored":1134738584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":867826116},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":3697966841900,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":80,"seq":343597384152,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1537864,"kb_used_data":1141564,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":92207288,"statfs":{"total":95995035648,"available":94420262912,"internally_reserved":0,"allocated":1168961536,"data_stored":1168686232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":405797312},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":70,"seq":300647711196,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1168760,"kb_used_data":678064,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":92576392,"statfs":{"total":95995035648,"available":94798225408,"internally_reserved":0,"allocated":694337536,"data_stored":694066718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":502462918},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":60,"seq":257698038241,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1222284,"kb_used_data":428868,"kb_used_omap":1,"kb_used_meta":793342,"kb_avail":92522868,"statfs":{"total":95995035648,"available":94743416832,"internally_reserved":0,"allocated":439160832,"data_stored":438881976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":812382666},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332582,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1739208,"kb_used_data":1076352,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":92005944,"statfs":{"total":95995035648,"available":94214086656,"internally_reserved":0,"allocated":1102184448,"data_stored":1101905072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":678754758},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575360,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1532296,"kb_used_data":680576,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":92212856,"statfs":{"total":95995035648,"available":94425964544,"internally_reserved":0,"allocated":696909824,"data_stored":696630448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":872085958},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738878,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2330364,"kb_used_data":1400884,"kb_used_omap":1,"kb_used_meta":929406,"kb_avail":91414788,"statfs":{"total":95995035648,"available":93608742912,"internally_reserved":0,"allocated":1434505216,"data_stored":1434230326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":951712197},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411832,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1781840,"kb_used_data":1140932,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":91963312,"statfs":{"total":95995035648,"available":94170431488,"internally_reserved":0,"allocated":1168314368,"data_stored":1168035512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150197,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2403588,"kb_used_data":1506108,"kb_used_omap":1,"kb_used_meta":897470,"kb_avail":91341564,"statfs":{"total":95995035648,"available":93533761536,"internally_reserved":0,"allocated":1542254592,"data_stored":1541979288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":919009735},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953972,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1568132,"kb_used_data":878908,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":92177020,"statfs":{"total":95995035648,"available":94389268480,"internally_reserved":0,"allocated":900001792,"data_stored":899726488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":443,"seq":1902670512413,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1315468,"kb_used_data":667716,"kb_used_omap":181,"kb_used_meta":647562,"kb_avail":92429684,"statfs":{"total":95995035648,"available":94647996416,"internally_reserved":0,"allocated":683741184,"data_stored":683462328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":186131,"internal_metadata":663103725},"hb_peers":[0,1,2,3,4,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1418858496,"data_stored":1418854424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":685707264,"data_stored":685703192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1152188416,"data_stored":1152188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1534459904,"data_stored":1534459904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":885915648,"data_stored":885915648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":676986880,"data_stored":676986880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1084428288,"data_stored":1084424216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":437846016,"data_stored":437846016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":692715520,"data_stored":692715520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1160314880,"data_stored":1160314880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1119223808,"data_stored":1119223808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:34:21.620 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-20T09:34:21.620 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pool set unique_pool_5 pg_num 26 2024-03-20T09:34:21.637 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:34:21.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:21.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:34:21.746 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:34:21.777 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:21.800 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:34:21.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:21.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:34:21.904 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:34:21.951 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:21.983 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:34:21.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:21.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:34:22.087 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:34:22.121 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:21 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:34:22.122 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:21 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:34:22.122 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:21 smithi115 ceph-mon[72476]: 7.9 scrub starts 2024-03-20T09:34:22.122 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:21 smithi115 ceph-mon[72476]: osdmap e930: 12 total, 12 up, 11 in 2024-03-20T09:34:22.122 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:21 smithi115 ceph-mon[72476]: 7.7 scrub starts 2024-03-20T09:34:22.122 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:21 smithi115 ceph-mon[72476]: 7.d scrub ok 2024-03-20T09:34:22.124 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:22.150 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:34:22.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:22.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:34:22.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:21 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:34:22.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:21 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:34:22.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:21 smithi122 ceph-mon[73495]: 7.9 scrub starts 2024-03-20T09:34:22.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:21 smithi122 ceph-mon[73495]: osdmap e930: 12 total, 12 up, 11 in 2024-03-20T09:34:22.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:21 smithi122 ceph-mon[73495]: 7.7 scrub starts 2024-03-20T09:34:22.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:21 smithi122 ceph-mon[73495]: 7.d scrub ok 2024-03-20T09:34:22.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:21 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:34:22.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:21 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:34:22.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:21 smithi026 ceph-mon[68309]: 7.9 scrub starts 2024-03-20T09:34:22.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:21 smithi026 ceph-mon[68309]: osdmap e930: 12 total, 12 up, 11 in 2024-03-20T09:34:22.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:21 smithi026 ceph-mon[68309]: 7.7 scrub starts 2024-03-20T09:34:22.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:21 smithi026 ceph-mon[68309]: 7.d scrub ok 2024-03-20T09:34:22.258 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:34:22.273 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:34:22.287 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:22.310 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:34:22.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:22.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:34:22.351 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:34:22.415 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:34:22.488 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:22.623 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:34:22.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:22.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:34:22.738 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:34:22.774 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:22.798 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:34:22.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:22.804 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:34:22.905 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:34:22.919 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:22 smithi026 ceph-mon[68309]: 7.9 scrub ok 2024-03-20T09:34:22.919 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:22 smithi026 ceph-mon[68309]: 7.7 scrub ok 2024-03-20T09:34:22.919 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:22 smithi026 ceph-mon[68309]: pgmap v2263: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 3.1 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 798 op/s 2024-03-20T09:34:22.989 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:23.076 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:34:23.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:23.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:34:23.186 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:34:23.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:22 smithi122 ceph-mon[73495]: 7.9 scrub ok 2024-03-20T09:34:23.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:22 smithi122 ceph-mon[73495]: 7.7 scrub ok 2024-03-20T09:34:23.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:22 smithi122 ceph-mon[73495]: pgmap v2263: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 3.1 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 798 op/s 2024-03-20T09:34:23.215 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:22 smithi115 ceph-mon[72476]: 7.9 scrub ok 2024-03-20T09:34:23.215 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:22 smithi115 ceph-mon[72476]: 7.7 scrub ok 2024-03-20T09:34:23.215 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:22 smithi115 ceph-mon[72476]: pgmap v2263: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 3.1 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 798 op/s 2024-03-20T09:34:23.217 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:23.240 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:34:23.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:23.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:34:23.349 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:34:23.376 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:23.398 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:34:23.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:23.401 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:34:23.439 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:34:23.501 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:34:23.533 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:23.554 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:34:23.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:23.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:34:23.661 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:34:23.695 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:23.722 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:34:23.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:23.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:34:23.826 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:34:23.906 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:24.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:23 smithi122 ceph-mon[73495]: 1.1 scrub starts 2024-03-20T09:34:24.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:23 smithi122 ceph-mon[73495]: 1.1 scrub ok 2024-03-20T09:34:24.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:23 smithi026 ceph-mon[68309]: 1.1 scrub starts 2024-03-20T09:34:24.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:23 smithi026 ceph-mon[68309]: 1.1 scrub ok 2024-03-20T09:34:24.325 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:34:24.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:23 smithi115 ceph-mon[72476]: 1.1 scrub starts 2024-03-20T09:34:24.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:23 smithi115 ceph-mon[72476]: 1.1 scrub ok 2024-03-20T09:34:25.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:24 smithi122 ceph-mon[73495]: pgmap v2264: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 3.0 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 801 op/s 2024-03-20T09:34:25.215 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:24 smithi026 ceph-mon[68309]: pgmap v2264: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 3.0 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 801 op/s 2024-03-20T09:34:25.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:24 smithi115 ceph-mon[72476]: pgmap v2264: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 3.0 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 801 op/s 2024-03-20T09:34:25.707 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:34:25.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:25.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:34:25.812 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:34:25.844 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:25.867 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:34:25.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:25.873 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:34:25.948 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:25 smithi026 ceph-mon[68309]: 7.1 scrub starts 2024-03-20T09:34:25.948 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:25 smithi026 ceph-mon[68309]: 7.f scrub starts 2024-03-20T09:34:25.974 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:34:26.061 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:26.170 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:34:26.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:26.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:34:26.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:25 smithi122 ceph-mon[73495]: 7.1 scrub starts 2024-03-20T09:34:26.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:25 smithi122 ceph-mon[73495]: 7.f scrub starts 2024-03-20T09:34:26.279 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:34:26.362 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:26.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:25 smithi115 ceph-mon[72476]: 7.1 scrub starts 2024-03-20T09:34:26.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:25 smithi115 ceph-mon[72476]: 7.f scrub starts 2024-03-20T09:34:27.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:26 smithi026 ceph-mon[68309]: 7.e scrub starts 2024-03-20T09:34:27.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:26 smithi026 ceph-mon[68309]: 7.1 scrub ok 2024-03-20T09:34:27.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:26 smithi026 ceph-mon[68309]: 7.e scrub ok 2024-03-20T09:34:27.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:26 smithi026 ceph-mon[68309]: 7.6 scrub starts 2024-03-20T09:34:27.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:26 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:34:27.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:26 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:34:27.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:26 smithi026 ceph-mon[68309]: osdmap e931: 12 total, 12 up, 11 in 2024-03-20T09:34:27.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:26 smithi026 ceph-mon[68309]: 1.5 scrub starts 2024-03-20T09:34:27.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:26 smithi026 ceph-mon[68309]: 1.5 scrub ok 2024-03-20T09:34:27.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:26 smithi026 ceph-mon[68309]: pgmap v2266: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 2.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 755 op/s 2024-03-20T09:34:27.398 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:26 smithi115 ceph-mon[72476]: 7.e scrub starts 2024-03-20T09:34:27.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:26 smithi115 ceph-mon[72476]: 7.1 scrub ok 2024-03-20T09:34:27.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:26 smithi115 ceph-mon[72476]: 7.e scrub ok 2024-03-20T09:34:27.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:26 smithi115 ceph-mon[72476]: 7.6 scrub starts 2024-03-20T09:34:27.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:26 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:34:27.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:26 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:34:27.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:26 smithi115 ceph-mon[72476]: osdmap e931: 12 total, 12 up, 11 in 2024-03-20T09:34:27.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:26 smithi115 ceph-mon[72476]: 1.5 scrub starts 2024-03-20T09:34:27.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:26 smithi115 ceph-mon[72476]: 1.5 scrub ok 2024-03-20T09:34:27.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:26 smithi115 ceph-mon[72476]: pgmap v2266: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 2.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 755 op/s 2024-03-20T09:34:27.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:26 smithi122 ceph-mon[73495]: 7.e scrub starts 2024-03-20T09:34:27.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:26 smithi122 ceph-mon[73495]: 7.1 scrub ok 2024-03-20T09:34:27.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:26 smithi122 ceph-mon[73495]: 7.e scrub ok 2024-03-20T09:34:27.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:26 smithi122 ceph-mon[73495]: 7.6 scrub starts 2024-03-20T09:34:27.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:26 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:34:27.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:26 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:34:27.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:26 smithi122 ceph-mon[73495]: osdmap e931: 12 total, 12 up, 11 in 2024-03-20T09:34:27.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:26 smithi122 ceph-mon[73495]: 1.5 scrub starts 2024-03-20T09:34:27.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:26 smithi122 ceph-mon[73495]: 1.5 scrub ok 2024-03-20T09:34:27.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:26 smithi122 ceph-mon[73495]: pgmap v2266: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 2.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 755 op/s 2024-03-20T09:34:27.489 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:34:27.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:27.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:34:27.600 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:34:27.679 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:28.369 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:34:28.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:28.385 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:27 smithi026 ceph-mon[68309]: 7.f scrub ok 2024-03-20T09:34:28.385 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:27 smithi026 ceph-mon[68309]: 1.12 scrub starts 2024-03-20T09:34:28.385 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:27 smithi026 ceph-mon[68309]: 1.12 scrub ok 2024-03-20T09:34:28.385 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:27 smithi026 ceph-mon[68309]: 1.d scrub starts 2024-03-20T09:34:28.385 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:27 smithi026 ceph-mon[68309]: 7.6 scrub ok 2024-03-20T09:34:28.386 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:27 smithi026 ceph-mon[68309]: 1.d scrub ok 2024-03-20T09:34:28.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:34:28.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:27 smithi115 ceph-mon[72476]: 7.f scrub ok 2024-03-20T09:34:28.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:27 smithi115 ceph-mon[72476]: 1.12 scrub starts 2024-03-20T09:34:28.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:27 smithi115 ceph-mon[72476]: 1.12 scrub ok 2024-03-20T09:34:28.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:27 smithi115 ceph-mon[72476]: 1.d scrub starts 2024-03-20T09:34:28.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:27 smithi115 ceph-mon[72476]: 7.6 scrub ok 2024-03-20T09:34:28.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:27 smithi115 ceph-mon[72476]: 1.d scrub ok 2024-03-20T09:34:28.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:27 smithi122 ceph-mon[73495]: 7.f scrub ok 2024-03-20T09:34:28.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:27 smithi122 ceph-mon[73495]: 1.12 scrub starts 2024-03-20T09:34:28.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:27 smithi122 ceph-mon[73495]: 1.12 scrub ok 2024-03-20T09:34:28.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:27 smithi122 ceph-mon[73495]: 1.d scrub starts 2024-03-20T09:34:28.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:27 smithi122 ceph-mon[73495]: 7.6 scrub ok 2024-03-20T09:34:28.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:27 smithi122 ceph-mon[73495]: 1.d scrub ok 2024-03-20T09:34:28.487 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:34:28.519 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:28.543 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:34:28.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:28.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:34:28.647 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:34:28.710 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:29.217 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:29 smithi026 ceph-mon[68309]: 1.8 scrub starts 2024-03-20T09:34:29.217 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:29 smithi026 ceph-mon[68309]: 1.8 scrub ok 2024-03-20T09:34:29.217 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:29 smithi026 ceph-mon[68309]: pgmap v2267: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 2.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 843 op/s 2024-03-20T09:34:29.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:29 smithi115 ceph-mon[72476]: 1.8 scrub starts 2024-03-20T09:34:29.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:29 smithi115 ceph-mon[72476]: 1.8 scrub ok 2024-03-20T09:34:29.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:29 smithi115 ceph-mon[72476]: pgmap v2267: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 2.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 843 op/s 2024-03-20T09:34:29.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:29 smithi122 ceph-mon[73495]: 1.8 scrub starts 2024-03-20T09:34:29.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:29 smithi122 ceph-mon[73495]: 1.8 scrub ok 2024-03-20T09:34:29.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:29 smithi122 ceph-mon[73495]: pgmap v2267: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 2.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 843 op/s 2024-03-20T09:34:30.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:30 smithi115 ceph-mon[72476]: 1.13 scrub starts 2024-03-20T09:34:30.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:30 smithi115 ceph-mon[72476]: 1.13 scrub ok 2024-03-20T09:34:30.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:30 smithi115 ceph-mon[72476]: 7.0 scrub starts 2024-03-20T09:34:30.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:30 smithi115 ceph-mon[72476]: 1.a scrub starts 2024-03-20T09:34:30.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:30 smithi115 ceph-mon[72476]: 1.a scrub ok 2024-03-20T09:34:30.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:30 smithi115 ceph-mon[72476]: 7.0 scrub ok 2024-03-20T09:34:30.420 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:34:30.420 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:34:30.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:30 smithi122 ceph-mon[73495]: 1.13 scrub starts 2024-03-20T09:34:30.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:30 smithi122 ceph-mon[73495]: 1.13 scrub ok 2024-03-20T09:34:30.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:30 smithi122 ceph-mon[73495]: 7.0 scrub starts 2024-03-20T09:34:30.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:30 smithi122 ceph-mon[73495]: 1.a scrub starts 2024-03-20T09:34:30.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:30 smithi122 ceph-mon[73495]: 1.a scrub ok 2024-03-20T09:34:30.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:30 smithi122 ceph-mon[73495]: 7.0 scrub ok 2024-03-20T09:34:30.445 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:30 smithi026 ceph-mon[68309]: 1.13 scrub starts 2024-03-20T09:34:30.445 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:30 smithi026 ceph-mon[68309]: 1.13 scrub ok 2024-03-20T09:34:30.445 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:30 smithi026 ceph-mon[68309]: 7.0 scrub starts 2024-03-20T09:34:30.445 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:30 smithi026 ceph-mon[68309]: 1.a scrub starts 2024-03-20T09:34:30.445 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:30 smithi026 ceph-mon[68309]: 1.a scrub ok 2024-03-20T09:34:30.446 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:30 smithi026 ceph-mon[68309]: 7.0 scrub ok 2024-03-20T09:34:30.924 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:34:31.305 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:31 smithi026 ceph-mon[68309]: pgmap v2268: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 2.7 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 759 op/s 2024-03-20T09:34:31.305 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:31 smithi026 ceph-mon[68309]: 1.9 scrub starts 2024-03-20T09:34:31.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:31 smithi115 ceph-mon[72476]: pgmap v2268: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 2.7 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 759 op/s 2024-03-20T09:34:31.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:31 smithi115 ceph-mon[72476]: 1.9 scrub starts 2024-03-20T09:34:31.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:31 smithi122 ceph-mon[73495]: pgmap v2268: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 2.7 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 759 op/s 2024-03-20T09:34:31.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:31 smithi122 ceph-mon[73495]: 1.9 scrub starts 2024-03-20T09:34:31.817 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-20T09:34:32.063 INFO:teuthology.orchestra.run.smithi026.stderr:set pool 7 pg_num to 26 2024-03-20T09:34:32.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:32 smithi115 ceph-mon[72476]: 1.9 scrub ok 2024-03-20T09:34:32.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:32 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3276390776' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num", "val": "26"}]: dispatch 2024-03-20T09:34:32.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:32 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num", "val": "26"}]: dispatch 2024-03-20T09:34:32.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:32 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:34:32.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:32 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:34:32.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:32 smithi115 ceph-mon[72476]: 7.b scrub starts 2024-03-20T09:34:32.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:32 smithi115 ceph-mon[72476]: 7.b scrub ok 2024-03-20T09:34:32.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:32 smithi122 ceph-mon[73495]: 1.9 scrub ok 2024-03-20T09:34:32.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:32 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3276390776' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num", "val": "26"}]: dispatch 2024-03-20T09:34:32.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:32 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num", "val": "26"}]: dispatch 2024-03-20T09:34:32.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:32 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:34:32.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:32 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:34:32.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:32 smithi122 ceph-mon[73495]: 7.b scrub starts 2024-03-20T09:34:32.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:32 smithi122 ceph-mon[73495]: 7.b scrub ok 2024-03-20T09:34:32.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:32 smithi026 ceph-mon[68309]: 1.9 scrub ok 2024-03-20T09:34:32.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:32 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3276390776' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num", "val": "26"}]: dispatch 2024-03-20T09:34:32.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:32 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num", "val": "26"}]: dispatch 2024-03-20T09:34:32.462 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:32 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:34:32.462 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:32 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:34:32.462 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:32 smithi026 ceph-mon[68309]: 7.b scrub starts 2024-03-20T09:34:32.462 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:32 smithi026 ceph-mon[68309]: 7.b scrub ok 2024-03-20T09:34:32.627 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:34:32.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:32.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:34:32.742 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:34:32.797 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:33.173 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:34:33.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:33 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num", "val": "26"}]': finished 2024-03-20T09:34:33.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:33 smithi115 ceph-mon[72476]: osdmap e932: 12 total, 12 up, 11 in 2024-03-20T09:34:33.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:33 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/132181302' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:34:33.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:33 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:34:33.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:33 smithi115 ceph-mon[72476]: pgmap v2270: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 2.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 897 op/s 2024-03-20T09:34:33.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:33 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-20T09:34:33.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:33 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-20T09:34:33.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:33 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num", "val": "26"}]': finished 2024-03-20T09:34:33.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:33 smithi122 ceph-mon[73495]: osdmap e932: 12 total, 12 up, 11 in 2024-03-20T09:34:33.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:33 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/132181302' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:34:33.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:33 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:34:33.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:33 smithi122 ceph-mon[73495]: pgmap v2270: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 2.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 897 op/s 2024-03-20T09:34:33.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:33 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-20T09:34:33.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:33 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-20T09:34:33.462 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:33 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num", "val": "26"}]': finished 2024-03-20T09:34:33.462 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:33 smithi026 ceph-mon[68309]: osdmap e932: 12 total, 12 up, 11 in 2024-03-20T09:34:33.462 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:33 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/132181302' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:34:33.462 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:33 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:34:33.462 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:33 smithi026 ceph-mon[68309]: pgmap v2270: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 2.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 897 op/s 2024-03-20T09:34:33.462 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:33 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-20T09:34:33.462 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:33 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-20T09:34:34.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:34 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:34:34.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:34 smithi026 ceph-mon[68309]: Cluster is now healthy 2024-03-20T09:34:34.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:34 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:34:34.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:34 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-20T09:34:34.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:34 smithi026 ceph-mon[68309]: osdmap e933: 12 total, 12 up, 11 in 2024-03-20T09:34:34.305 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:34:34.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:34.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:34:34.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:34 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:34:34.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:34 smithi115 ceph-mon[72476]: Cluster is now healthy 2024-03-20T09:34:34.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:34 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:34:34.400 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:34 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-20T09:34:34.400 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:34 smithi115 ceph-mon[72476]: osdmap e933: 12 total, 12 up, 11 in 2024-03-20T09:34:34.416 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:34:34.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:34 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:34:34.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:34 smithi122 ceph-mon[73495]: Cluster is now healthy 2024-03-20T09:34:34.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:34 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:34:34.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:34 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-20T09:34:34.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:34 smithi122 ceph-mon[73495]: osdmap e933: 12 total, 12 up, 11 in 2024-03-20T09:34:34.479 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:35.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:35 smithi026 ceph-mon[68309]: osdmap e934: 12 total, 12 up, 11 in 2024-03-20T09:34:35.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:35 smithi026 ceph-mon[68309]: 7.2 scrub starts 2024-03-20T09:34:35.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:35 smithi026 ceph-mon[68309]: 7.0 scrub starts 2024-03-20T09:34:35.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:35 smithi026 ceph-mon[68309]: 7.7 scrub starts 2024-03-20T09:34:35.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:35 smithi026 ceph-mon[68309]: pgmap v2273: 57 pgs: 1 peering, 10 unknown, 1 active+clean+scrubbing, 45 active+clean; 2.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail 2024-03-20T09:34:35.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:35 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-20T09:34:35.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:35 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-20T09:34:35.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:35 smithi026 ceph-mon[68309]: 7.0 scrub ok 2024-03-20T09:34:35.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:35 smithi026 ceph-mon[68309]: 7.7 scrub ok 2024-03-20T09:34:35.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:35 smithi026 ceph-mon[68309]: 7.2 scrub ok 2024-03-20T09:34:35.350 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:34:35.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:35.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:34:35.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:35 smithi115 ceph-mon[72476]: osdmap e934: 12 total, 12 up, 11 in 2024-03-20T09:34:35.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:35 smithi115 ceph-mon[72476]: 7.2 scrub starts 2024-03-20T09:34:35.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:35 smithi115 ceph-mon[72476]: 7.0 scrub starts 2024-03-20T09:34:35.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:35 smithi115 ceph-mon[72476]: 7.7 scrub starts 2024-03-20T09:34:35.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:35 smithi115 ceph-mon[72476]: pgmap v2273: 57 pgs: 1 peering, 10 unknown, 1 active+clean+scrubbing, 45 active+clean; 2.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail 2024-03-20T09:34:35.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:35 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-20T09:34:35.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:35 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-20T09:34:35.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:35 smithi115 ceph-mon[72476]: 7.0 scrub ok 2024-03-20T09:34:35.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:35 smithi115 ceph-mon[72476]: 7.7 scrub ok 2024-03-20T09:34:35.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:35 smithi115 ceph-mon[72476]: 7.2 scrub ok 2024-03-20T09:34:35.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:35 smithi122 ceph-mon[73495]: osdmap e934: 12 total, 12 up, 11 in 2024-03-20T09:34:35.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:35 smithi122 ceph-mon[73495]: 7.2 scrub starts 2024-03-20T09:34:35.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:35 smithi122 ceph-mon[73495]: 7.0 scrub starts 2024-03-20T09:34:35.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:35 smithi122 ceph-mon[73495]: 7.7 scrub starts 2024-03-20T09:34:35.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:35 smithi122 ceph-mon[73495]: pgmap v2273: 57 pgs: 1 peering, 10 unknown, 1 active+clean+scrubbing, 45 active+clean; 2.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail 2024-03-20T09:34:35.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:35 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-20T09:34:35.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:35 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-20T09:34:35.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:35 smithi122 ceph-mon[73495]: 7.0 scrub ok 2024-03-20T09:34:35.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:35 smithi122 ceph-mon[73495]: 7.7 scrub ok 2024-03-20T09:34:35.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:35 smithi122 ceph-mon[73495]: 7.2 scrub ok 2024-03-20T09:34:35.456 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:34:35.484 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:35.505 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:34:35.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:35.510 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:34:35.610 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:34:35.643 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:35.745 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:34:35.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:35.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:34:35.850 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:34:35.880 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:35.899 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:34:35.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:35.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:34:35.915 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:34:35.939 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:34:35.965 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:34:35.998 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:34:36.003 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:34:36.020 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:34:36.030 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:36.049 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:34:36.050 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:34:36.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:36.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:34:36.070 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:34:36.093 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:34:36.113 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:34:36.140 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:34:36.144 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:34:36.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:36 smithi115 ceph-mon[72476]: 7.5 deep-scrub starts 2024-03-20T09:34:36.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:36 smithi115 ceph-mon[72476]: 7.1 scrub starts 2024-03-20T09:34:36.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:36 smithi115 ceph-mon[72476]: 7.9 scrub starts 2024-03-20T09:34:36.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:36 smithi115 ceph-mon[72476]: 7.1 scrub ok 2024-03-20T09:34:36.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:36 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-20T09:34:36.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:36 smithi115 ceph-mon[72476]: osdmap e935: 12 total, 12 up, 11 in 2024-03-20T09:34:36.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:36 smithi115 ceph-mon[72476]: 7.8 scrub starts 2024-03-20T09:34:36.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:36 smithi115 ceph-mon[72476]: 7.3 scrub starts 2024-03-20T09:34:36.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:36 smithi115 ceph-mon[72476]: 7.8 scrub ok 2024-03-20T09:34:36.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:36 smithi115 ceph-mon[72476]: 7.3 scrub ok 2024-03-20T09:34:36.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:36 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1481857922' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:34:36.153 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:34:36.168 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:34:36.178 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:36.191 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:34:36.194 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:34:36.194 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:34:36.194 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:34:36.194 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:34:36.194 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:34:36.194 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:34:36.194 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:34:36.194 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:34:36.194 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:34:36.194 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:34:36.195 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:34:36.195 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:34:36.197 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:34:36.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:36.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:34:36.302 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:34:36.337 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:36.404 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:36 smithi026 ceph-mon[68309]: 7.5 deep-scrub starts 2024-03-20T09:34:36.404 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:36 smithi026 ceph-mon[68309]: 7.1 scrub starts 2024-03-20T09:34:36.404 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:36 smithi026 ceph-mon[68309]: 7.9 scrub starts 2024-03-20T09:34:36.404 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:36 smithi026 ceph-mon[68309]: 7.1 scrub ok 2024-03-20T09:34:36.404 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:36 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-20T09:34:36.404 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:36 smithi026 ceph-mon[68309]: osdmap e935: 12 total, 12 up, 11 in 2024-03-20T09:34:36.404 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:36 smithi026 ceph-mon[68309]: 7.8 scrub starts 2024-03-20T09:34:36.404 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:36 smithi026 ceph-mon[68309]: 7.3 scrub starts 2024-03-20T09:34:36.405 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:36 smithi026 ceph-mon[68309]: 7.8 scrub ok 2024-03-20T09:34:36.405 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:36 smithi026 ceph-mon[68309]: 7.3 scrub ok 2024-03-20T09:34:36.405 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:36 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1481857922' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:34:36.412 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:34:36.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:36.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:34:36.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:36 smithi122 ceph-mon[73495]: 7.5 deep-scrub starts 2024-03-20T09:34:36.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:36 smithi122 ceph-mon[73495]: 7.1 scrub starts 2024-03-20T09:34:36.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:36 smithi122 ceph-mon[73495]: 7.9 scrub starts 2024-03-20T09:34:36.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:36 smithi122 ceph-mon[73495]: 7.1 scrub ok 2024-03-20T09:34:36.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:36 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-20T09:34:36.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:36 smithi122 ceph-mon[73495]: osdmap e935: 12 total, 12 up, 11 in 2024-03-20T09:34:36.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:36 smithi122 ceph-mon[73495]: 7.8 scrub starts 2024-03-20T09:34:36.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:36 smithi122 ceph-mon[73495]: 7.3 scrub starts 2024-03-20T09:34:36.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:36 smithi122 ceph-mon[73495]: 7.8 scrub ok 2024-03-20T09:34:36.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:36 smithi122 ceph-mon[73495]: 7.3 scrub ok 2024-03-20T09:34:36.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:36 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1481857922' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:34:36.517 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:34:36.545 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:36.568 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:34:36.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:36.571 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:34:36.672 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:34:36.700 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:36.720 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:34:36.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:36.723 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:34:36.824 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:34:36.854 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:37.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:37 smithi115 ceph-mon[72476]: 7.9 scrub ok 2024-03-20T09:34:37.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:37 smithi115 ceph-mon[72476]: 1.e scrub starts 2024-03-20T09:34:37.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:37 smithi115 ceph-mon[72476]: 7.6 scrub starts 2024-03-20T09:34:37.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:37 smithi115 ceph-mon[72476]: 1.e scrub ok 2024-03-20T09:34:37.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:37 smithi115 ceph-mon[72476]: 7.6 scrub ok 2024-03-20T09:34:37.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:37 smithi115 ceph-mon[72476]: osdmap e936: 12 total, 12 up, 11 in 2024-03-20T09:34:37.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:37 smithi115 ceph-mon[72476]: 1.7 scrub starts 2024-03-20T09:34:37.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:37 smithi115 ceph-mon[72476]: 1.7 scrub ok 2024-03-20T09:34:37.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:37 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:34:37.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:37 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:34:37.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:37 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:34:37.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:37 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:34:37.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:37 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:34:37.398 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:37 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:34:37.398 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:37 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:34:37.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:37 smithi122 ceph-mon[73495]: 7.9 scrub ok 2024-03-20T09:34:37.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:37 smithi122 ceph-mon[73495]: 1.e scrub starts 2024-03-20T09:34:37.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:37 smithi122 ceph-mon[73495]: 7.6 scrub starts 2024-03-20T09:34:37.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:37 smithi122 ceph-mon[73495]: 1.e scrub ok 2024-03-20T09:34:37.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:37 smithi122 ceph-mon[73495]: 7.6 scrub ok 2024-03-20T09:34:37.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:37 smithi122 ceph-mon[73495]: osdmap e936: 12 total, 12 up, 11 in 2024-03-20T09:34:37.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:37 smithi122 ceph-mon[73495]: 1.7 scrub starts 2024-03-20T09:34:37.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:37 smithi122 ceph-mon[73495]: 1.7 scrub ok 2024-03-20T09:34:37.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:37 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:34:37.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:37 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:34:37.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:37 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:34:37.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:37 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:34:37.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:37 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:34:37.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:37 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:34:37.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:37 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:34:37.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:37 smithi026 ceph-mon[68309]: 7.9 scrub ok 2024-03-20T09:34:37.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:37 smithi026 ceph-mon[68309]: 1.e scrub starts 2024-03-20T09:34:37.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:37 smithi026 ceph-mon[68309]: 7.6 scrub starts 2024-03-20T09:34:37.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:37 smithi026 ceph-mon[68309]: 1.e scrub ok 2024-03-20T09:34:37.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:37 smithi026 ceph-mon[68309]: 7.6 scrub ok 2024-03-20T09:34:37.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:37 smithi026 ceph-mon[68309]: osdmap e936: 12 total, 12 up, 11 in 2024-03-20T09:34:37.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:37 smithi026 ceph-mon[68309]: 1.7 scrub starts 2024-03-20T09:34:37.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:37 smithi026 ceph-mon[68309]: 1.7 scrub ok 2024-03-20T09:34:37.464 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:37 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:34:37.464 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:37 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:34:37.464 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:37 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:34:37.464 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:37 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:34:37.464 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:37 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:34:37.464 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:37 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:34:37.464 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:37 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:34:38.012 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:34:38.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:38.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:34:38.120 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:34:38.154 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:38.182 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:34:38.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:38.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:34:38.188 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:34:38.279 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:38 smithi026 ceph-mon[68309]: 7.4 scrub starts 2024-03-20T09:34:38.279 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:38 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:34:38.279 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:38 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:34:38.279 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:38 smithi026 ceph-mon[68309]: 7.4 scrub ok 2024-03-20T09:34:38.279 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:38 smithi026 ceph-mon[68309]: 7.18 starting backfill to osd.0 from (0'0,0'0] MAX to 934'13216 2024-03-20T09:34:38.280 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:38 smithi026 ceph-mon[68309]: 7.13 starting backfill to osd.0 from (0'0,0'0] MAX to 934'13374 2024-03-20T09:34:38.280 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:38 smithi026 ceph-mon[68309]: 7.12 starting backfill to osd.0 from (0'0,0'0] MAX to 934'13446 2024-03-20T09:34:38.280 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:38 smithi026 ceph-mon[68309]: 7.19 starting backfill to osd.2 from (0'0,0'0] MAX to 934'13505 2024-03-20T09:34:38.280 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:38 smithi026 ceph-mon[68309]: 7.11 starting backfill to osd.7 from (0'0,0'0] MAX to 934'13406 2024-03-20T09:34:38.280 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:38 smithi026 ceph-mon[68309]: 7.15 starting backfill to osd.1 from (0'0,0'0] MAX to 934'13228 2024-03-20T09:34:38.280 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:38 smithi026 ceph-mon[68309]: osdmap e937: 12 total, 12 up, 11 in 2024-03-20T09:34:38.280 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:38 smithi026 ceph-mon[68309]: 7.16 starting backfill to osd.8 from (0'0,0'0] MAX to 934'13393 2024-03-20T09:34:38.280 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:38 smithi026 ceph-mon[68309]: 7.14 starting backfill to osd.5 from (0'0,0'0] MAX to 934'13432 2024-03-20T09:34:38.280 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:38 smithi026 ceph-mon[68309]: 7.10 starting backfill to osd.7 from (0'0,0'0] MAX to 934'13182 2024-03-20T09:34:38.280 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:38 smithi026 ceph-mon[68309]: 7.11 starting backfill to osd.9 from (0'0,0'0] MAX to 934'13406 2024-03-20T09:34:38.280 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:38 smithi026 ceph-mon[68309]: 7.18 starting backfill to osd.8 from (0'0,0'0] MAX to 934'13216 2024-03-20T09:34:38.280 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:38 smithi026 ceph-mon[68309]: 7.13 starting backfill to osd.7 from (0'0,0'0] MAX to 934'13374 2024-03-20T09:34:38.280 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:38 smithi026 ceph-mon[68309]: 7.17 starting backfill to osd.2 from (0'0,0'0] MAX to 934'12994 2024-03-20T09:34:38.280 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:38 smithi026 ceph-mon[68309]: 7.13 starting backfill to osd.11 from (0'0,0'0] MAX to 934'13374 2024-03-20T09:34:38.281 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:38 smithi026 ceph-mon[68309]: 7.17 starting backfill to osd.5 from (0'0,0'0] MAX to 934'12994 2024-03-20T09:34:38.281 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:38 smithi026 ceph-mon[68309]: 7.17 starting backfill to osd.7 from (0'0,0'0] MAX to 934'12994 2024-03-20T09:34:38.288 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:34:38.318 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:38.343 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:34:38.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:38.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:34:38.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:38 smithi115 ceph-mon[72476]: 7.4 scrub starts 2024-03-20T09:34:38.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:38 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:34:38.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:38 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:34:38.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:38 smithi115 ceph-mon[72476]: 7.4 scrub ok 2024-03-20T09:34:38.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:38 smithi115 ceph-mon[72476]: 7.18 starting backfill to osd.0 from (0'0,0'0] MAX to 934'13216 2024-03-20T09:34:38.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:38 smithi115 ceph-mon[72476]: 7.13 starting backfill to osd.0 from (0'0,0'0] MAX to 934'13374 2024-03-20T09:34:38.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:38 smithi115 ceph-mon[72476]: 7.12 starting backfill to osd.0 from (0'0,0'0] MAX to 934'13446 2024-03-20T09:34:38.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:38 smithi115 ceph-mon[72476]: 7.19 starting backfill to osd.2 from (0'0,0'0] MAX to 934'13505 2024-03-20T09:34:38.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:38 smithi115 ceph-mon[72476]: 7.11 starting backfill to osd.7 from (0'0,0'0] MAX to 934'13406 2024-03-20T09:34:38.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:38 smithi115 ceph-mon[72476]: 7.15 starting backfill to osd.1 from (0'0,0'0] MAX to 934'13228 2024-03-20T09:34:38.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:38 smithi115 ceph-mon[72476]: osdmap e937: 12 total, 12 up, 11 in 2024-03-20T09:34:38.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:38 smithi115 ceph-mon[72476]: 7.16 starting backfill to osd.8 from (0'0,0'0] MAX to 934'13393 2024-03-20T09:34:38.398 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:38 smithi115 ceph-mon[72476]: 7.14 starting backfill to osd.5 from (0'0,0'0] MAX to 934'13432 2024-03-20T09:34:38.398 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:38 smithi115 ceph-mon[72476]: 7.10 starting backfill to osd.7 from (0'0,0'0] MAX to 934'13182 2024-03-20T09:34:38.398 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:38 smithi115 ceph-mon[72476]: 7.11 starting backfill to osd.9 from (0'0,0'0] MAX to 934'13406 2024-03-20T09:34:38.398 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:38 smithi115 ceph-mon[72476]: 7.18 starting backfill to osd.8 from (0'0,0'0] MAX to 934'13216 2024-03-20T09:34:38.398 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:38 smithi115 ceph-mon[72476]: 7.13 starting backfill to osd.7 from (0'0,0'0] MAX to 934'13374 2024-03-20T09:34:38.398 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:38 smithi115 ceph-mon[72476]: 7.17 starting backfill to osd.2 from (0'0,0'0] MAX to 934'12994 2024-03-20T09:34:38.398 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:38 smithi115 ceph-mon[72476]: 7.13 starting backfill to osd.11 from (0'0,0'0] MAX to 934'13374 2024-03-20T09:34:38.398 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:38 smithi115 ceph-mon[72476]: 7.17 starting backfill to osd.5 from (0'0,0'0] MAX to 934'12994 2024-03-20T09:34:38.398 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:38 smithi115 ceph-mon[72476]: 7.17 starting backfill to osd.7 from (0'0,0'0] MAX to 934'12994 2024-03-20T09:34:38.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:38 smithi122 ceph-mon[73495]: 7.4 scrub starts 2024-03-20T09:34:38.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:38 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:34:38.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:38 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:34:38.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:38 smithi122 ceph-mon[73495]: 7.4 scrub ok 2024-03-20T09:34:38.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:38 smithi122 ceph-mon[73495]: 7.18 starting backfill to osd.0 from (0'0,0'0] MAX to 934'13216 2024-03-20T09:34:38.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:38 smithi122 ceph-mon[73495]: 7.13 starting backfill to osd.0 from (0'0,0'0] MAX to 934'13374 2024-03-20T09:34:38.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:38 smithi122 ceph-mon[73495]: 7.12 starting backfill to osd.0 from (0'0,0'0] MAX to 934'13446 2024-03-20T09:34:38.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:38 smithi122 ceph-mon[73495]: 7.19 starting backfill to osd.2 from (0'0,0'0] MAX to 934'13505 2024-03-20T09:34:38.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:38 smithi122 ceph-mon[73495]: 7.11 starting backfill to osd.7 from (0'0,0'0] MAX to 934'13406 2024-03-20T09:34:38.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:38 smithi122 ceph-mon[73495]: 7.15 starting backfill to osd.1 from (0'0,0'0] MAX to 934'13228 2024-03-20T09:34:38.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:38 smithi122 ceph-mon[73495]: osdmap e937: 12 total, 12 up, 11 in 2024-03-20T09:34:38.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:38 smithi122 ceph-mon[73495]: 7.16 starting backfill to osd.8 from (0'0,0'0] MAX to 934'13393 2024-03-20T09:34:38.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:38 smithi122 ceph-mon[73495]: 7.14 starting backfill to osd.5 from (0'0,0'0] MAX to 934'13432 2024-03-20T09:34:38.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:38 smithi122 ceph-mon[73495]: 7.10 starting backfill to osd.7 from (0'0,0'0] MAX to 934'13182 2024-03-20T09:34:38.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:38 smithi122 ceph-mon[73495]: 7.11 starting backfill to osd.9 from (0'0,0'0] MAX to 934'13406 2024-03-20T09:34:38.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:38 smithi122 ceph-mon[73495]: 7.18 starting backfill to osd.8 from (0'0,0'0] MAX to 934'13216 2024-03-20T09:34:38.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:38 smithi122 ceph-mon[73495]: 7.13 starting backfill to osd.7 from (0'0,0'0] MAX to 934'13374 2024-03-20T09:34:38.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:38 smithi122 ceph-mon[73495]: 7.17 starting backfill to osd.2 from (0'0,0'0] MAX to 934'12994 2024-03-20T09:34:38.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:38 smithi122 ceph-mon[73495]: 7.13 starting backfill to osd.11 from (0'0,0'0] MAX to 934'13374 2024-03-20T09:34:38.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:38 smithi122 ceph-mon[73495]: 7.17 starting backfill to osd.5 from (0'0,0'0] MAX to 934'12994 2024-03-20T09:34:38.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:38 smithi122 ceph-mon[73495]: 7.17 starting backfill to osd.7 from (0'0,0'0] MAX to 934'12994 2024-03-20T09:34:38.447 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:34:38.476 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:38.498 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:34:38.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:38.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:34:38.604 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:34:38.636 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:38.658 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:34:38.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:38.661 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:34:38.763 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:34:38.846 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:34:38.846 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:34:38.846 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-03-20T09:34:38.846 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-03-20T09:34:38.846 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:34:38.861 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:34:38.870 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:38.873 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 8, 11, 3, 4, 9] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 11, 5, 10] 2024-03-20T09:34:38.874 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:34:38.874 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-20T09:34:38.874 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-20T09:34:39.109 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:34:39.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:39.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:34:39.222 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:34:39.256 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:39.281 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:34:39.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:39.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:34:39.294 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:34:39.371 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:39 smithi026 ceph-mon[68309]: 7.16 starting backfill to osd.11 from (0'0,0'0] MAX to 934'13393 2024-03-20T09:34:39.372 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:39 smithi026 ceph-mon[68309]: 7.19 starting backfill to osd.6 from (0'0,0'0] MAX to 934'13505 2024-03-20T09:34:39.372 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:39 smithi026 ceph-mon[68309]: 7.15 starting backfill to osd.7 from (0'0,0'0] MAX to 934'13228 2024-03-20T09:34:39.372 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:39 smithi026 ceph-mon[68309]: 7.14 starting backfill to osd.11 from (0'0,0'0] MAX to 934'13432 2024-03-20T09:34:39.372 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:39 smithi026 ceph-mon[68309]: 7.19 starting backfill to osd.7 from (0'0,0'0] MAX to 934'13505 2024-03-20T09:34:39.372 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:39 smithi026 ceph-mon[68309]: 7.15 starting backfill to osd.8 from (0'0,0'0] MAX to 934'13228 2024-03-20T09:34:39.372 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:39 smithi026 ceph-mon[68309]: pgmap v2278: 57 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 5 remapped+peering, 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 3 unknown, 44 active+clean; 2.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 5821/110460 objects misplaced (5.270%); 990 KiB/s, 15 objects/s recovering 2024-03-20T09:34:39.372 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:39 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:34:39.372 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:39 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:34:39.372 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:39 smithi026 ceph-mon[68309]: 7.5 deep-scrub ok 2024-03-20T09:34:39.385 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:34:39.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:39 smithi122 ceph-mon[73495]: 7.16 starting backfill to osd.11 from (0'0,0'0] MAX to 934'13393 2024-03-20T09:34:39.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:39 smithi122 ceph-mon[73495]: 7.19 starting backfill to osd.6 from (0'0,0'0] MAX to 934'13505 2024-03-20T09:34:39.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:39 smithi122 ceph-mon[73495]: 7.15 starting backfill to osd.7 from (0'0,0'0] MAX to 934'13228 2024-03-20T09:34:39.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:39 smithi122 ceph-mon[73495]: 7.14 starting backfill to osd.11 from (0'0,0'0] MAX to 934'13432 2024-03-20T09:34:39.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:39 smithi122 ceph-mon[73495]: 7.19 starting backfill to osd.7 from (0'0,0'0] MAX to 934'13505 2024-03-20T09:34:39.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:39 smithi122 ceph-mon[73495]: 7.15 starting backfill to osd.8 from (0'0,0'0] MAX to 934'13228 2024-03-20T09:34:39.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:39 smithi122 ceph-mon[73495]: pgmap v2278: 57 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 5 remapped+peering, 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 3 unknown, 44 active+clean; 2.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 5821/110460 objects misplaced (5.270%); 990 KiB/s, 15 objects/s recovering 2024-03-20T09:34:39.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:39 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:34:39.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:39 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:34:39.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:39 smithi122 ceph-mon[73495]: 7.5 deep-scrub ok 2024-03-20T09:34:39.470 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:39.647 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:39 smithi115 ceph-mon[72476]: 7.16 starting backfill to osd.11 from (0'0,0'0] MAX to 934'13393 2024-03-20T09:34:39.647 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:39 smithi115 ceph-mon[72476]: 7.19 starting backfill to osd.6 from (0'0,0'0] MAX to 934'13505 2024-03-20T09:34:39.647 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:39 smithi115 ceph-mon[72476]: 7.15 starting backfill to osd.7 from (0'0,0'0] MAX to 934'13228 2024-03-20T09:34:39.647 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:39 smithi115 ceph-mon[72476]: 7.14 starting backfill to osd.11 from (0'0,0'0] MAX to 934'13432 2024-03-20T09:34:39.647 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:39 smithi115 ceph-mon[72476]: 7.19 starting backfill to osd.7 from (0'0,0'0] MAX to 934'13505 2024-03-20T09:34:39.647 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:39 smithi115 ceph-mon[72476]: 7.15 starting backfill to osd.8 from (0'0,0'0] MAX to 934'13228 2024-03-20T09:34:39.647 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:39 smithi115 ceph-mon[72476]: pgmap v2278: 57 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 5 remapped+peering, 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 3 unknown, 44 active+clean; 2.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 5821/110460 objects misplaced (5.270%); 990 KiB/s, 15 objects/s recovering 2024-03-20T09:34:39.647 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:39 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:34:39.647 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:39 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:34:39.647 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:39 smithi115 ceph-mon[72476]: 7.5 deep-scrub ok 2024-03-20T09:34:39.825 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-20T09:34:39.914 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:34:39.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:39.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:34:40.029 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:34:40.060 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:40.087 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:34:40.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:40.092 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:34:40.193 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:34:40.222 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:40.247 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:34:40.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:40.252 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:34:40.353 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:34:40.431 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:40.433 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:40 smithi026 ceph-mon[68309]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-20T09:34:40.433 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:40 smithi026 ceph-mon[68309]: osdmap e938: 12 total, 12 up, 11 in 2024-03-20T09:34:40.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:40 smithi122 ceph-mon[73495]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-20T09:34:40.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:40 smithi122 ceph-mon[73495]: osdmap e938: 12 total, 12 up, 11 in 2024-03-20T09:34:40.646 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:40 smithi115 ceph-mon[72476]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-20T09:34:40.647 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:40 smithi115 ceph-mon[72476]: osdmap e938: 12 total, 12 up, 11 in 2024-03-20T09:34:40.690 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:34:40.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:40.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:34:40.802 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:34:40.884 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:41.466 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:41 smithi026 ceph-mon[68309]: pgmap v2280: 57 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 5 remapped+peering, 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 45 active+clean; 2.4 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12103/115797 objects misplaced (10.452%); 6.1 MiB/s, 98 objects/s recovering 2024-03-20T09:34:41.646 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:41 smithi115 ceph-mon[72476]: pgmap v2280: 57 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 5 remapped+peering, 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 45 active+clean; 2.4 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12103/115797 objects misplaced (10.452%); 6.1 MiB/s, 98 objects/s recovering 2024-03-20T09:34:41.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:41 smithi122 ceph-mon[73495]: pgmap v2280: 57 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 5 remapped+peering, 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 45 active+clean; 2.4 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 12103/115797 objects misplaced (10.452%); 6.1 MiB/s, 98 objects/s recovering 2024-03-20T09:34:42.868 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:34:42.870 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:42 smithi026 ceph-mon[68309]: 1.6 scrub starts 2024-03-20T09:34:42.870 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:42 smithi026 ceph-mon[68309]: 1.6 scrub ok 2024-03-20T09:34:42.874 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:34:42.874 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:34:42.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:42.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:34:42.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:42 smithi115 ceph-mon[72476]: 1.6 scrub starts 2024-03-20T09:34:42.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:42 smithi115 ceph-mon[72476]: 1.6 scrub ok 2024-03-20T09:34:42.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:42 smithi122 ceph-mon[73495]: 1.6 scrub starts 2024-03-20T09:34:42.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:42 smithi122 ceph-mon[73495]: 1.6 scrub ok 2024-03-20T09:34:42.951 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:34:42.977 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:34:43.008 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:43.032 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:34:43.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:43.035 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:34:43.136 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:34:43.207 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:43.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:43 smithi115 ceph-mon[72476]: pgmap v2281: 57 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 47 active+clean; 2.2 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 22577/110181 objects misplaced (20.491%); 17 MiB/s, 273 objects/s recovering 2024-03-20T09:34:43.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:43 smithi122 ceph-mon[73495]: pgmap v2281: 57 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 47 active+clean; 2.2 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 22577/110181 objects misplaced (20.491%); 17 MiB/s, 273 objects/s recovering 2024-03-20T09:34:43.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:43 smithi026 ceph-mon[68309]: pgmap v2281: 57 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 47 active+clean; 2.2 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 22577/110181 objects misplaced (20.491%); 17 MiB/s, 273 objects/s recovering 2024-03-20T09:34:45.115 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:34:45.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:45.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:34:45.236 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:34:45.265 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:45.288 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:34:45.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:45.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:34:45.392 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:34:45.418 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:45.437 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:34:45.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:45.441 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:34:45.542 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:34:45.591 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:45.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:45 smithi122 ceph-mon[73495]: pgmap v2282: 57 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 47 active+clean; 2.2 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 21292/107256 objects misplaced (19.852%); 16 MiB/s, 257 objects/s recovering 2024-03-20T09:34:45.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:45 smithi122 ceph-mon[73495]: osdmap e939: 12 total, 12 up, 11 in 2024-03-20T09:34:45.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:45 smithi026 ceph-mon[68309]: pgmap v2282: 57 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 47 active+clean; 2.2 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 21292/107256 objects misplaced (19.852%); 16 MiB/s, 257 objects/s recovering 2024-03-20T09:34:45.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:45 smithi026 ceph-mon[68309]: osdmap e939: 12 total, 12 up, 11 in 2024-03-20T09:34:45.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:45 smithi115 ceph-mon[72476]: pgmap v2282: 57 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 47 active+clean; 2.2 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 21292/107256 objects misplaced (19.852%); 16 MiB/s, 257 objects/s recovering 2024-03-20T09:34:45.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:45 smithi115 ceph-mon[72476]: osdmap e939: 12 total, 12 up, 11 in 2024-03-20T09:34:46.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:46 smithi115 ceph-mon[72476]: osdmap e940: 12 total, 12 up, 11 in 2024-03-20T09:34:46.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:46 smithi115 ceph-mon[72476]: 7.12 scrub starts 2024-03-20T09:34:46.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:46 smithi115 ceph-mon[72476]: 7.12 scrub ok 2024-03-20T09:34:46.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:46 smithi122 ceph-mon[73495]: osdmap e940: 12 total, 12 up, 11 in 2024-03-20T09:34:46.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:46 smithi122 ceph-mon[73495]: 7.12 scrub starts 2024-03-20T09:34:46.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:46 smithi122 ceph-mon[73495]: 7.12 scrub ok 2024-03-20T09:34:46.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:46 smithi026 ceph-mon[68309]: osdmap e940: 12 total, 12 up, 11 in 2024-03-20T09:34:46.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:46 smithi026 ceph-mon[68309]: 7.12 scrub starts 2024-03-20T09:34:46.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:46 smithi026 ceph-mon[68309]: 7.12 scrub ok 2024-03-20T09:34:47.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:47 smithi122 ceph-mon[73495]: 7.10 scrub starts 2024-03-20T09:34:47.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:47 smithi122 ceph-mon[73495]: 7.10 scrub ok 2024-03-20T09:34:47.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:47 smithi122 ceph-mon[73495]: pgmap v2285: 57 pgs: 1 active+clean+remapped, 1 peering, 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 47 active+clean; 2.1 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 19166/101094 objects misplaced (18.959%); 20 MiB/s, 312 objects/s recovering 2024-03-20T09:34:47.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:47 smithi122 ceph-mon[73495]: osdmap e941: 12 total, 12 up, 11 in 2024-03-20T09:34:47.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:47 smithi026 ceph-mon[68309]: 7.10 scrub starts 2024-03-20T09:34:47.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:47 smithi026 ceph-mon[68309]: 7.10 scrub ok 2024-03-20T09:34:47.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:47 smithi026 ceph-mon[68309]: pgmap v2285: 57 pgs: 1 active+clean+remapped, 1 peering, 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 47 active+clean; 2.1 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 19166/101094 objects misplaced (18.959%); 20 MiB/s, 312 objects/s recovering 2024-03-20T09:34:47.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:47 smithi026 ceph-mon[68309]: osdmap e941: 12 total, 12 up, 11 in 2024-03-20T09:34:48.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:47 smithi115 ceph-mon[72476]: 7.10 scrub starts 2024-03-20T09:34:48.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:47 smithi115 ceph-mon[72476]: 7.10 scrub ok 2024-03-20T09:34:48.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:47 smithi115 ceph-mon[72476]: pgmap v2285: 57 pgs: 1 active+clean+remapped, 1 peering, 3 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 47 active+clean; 2.1 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 19166/101094 objects misplaced (18.959%); 20 MiB/s, 312 objects/s recovering 2024-03-20T09:34:48.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:47 smithi115 ceph-mon[72476]: osdmap e941: 12 total, 12 up, 11 in 2024-03-20T09:34:48.474 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:34:48.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:48.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:34:48.586 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:34:48.623 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:48.651 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:34:48.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:48.654 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:34:48.755 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:34:48.817 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:48 smithi026 ceph-mon[68309]: 1.c scrub starts 2024-03-20T09:34:48.817 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:48 smithi026 ceph-mon[68309]: 1.c scrub ok 2024-03-20T09:34:48.817 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:48 smithi026 ceph-mon[68309]: osdmap e942: 12 total, 12 up, 11 in 2024-03-20T09:34:48.817 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:48 smithi026 ceph-mon[68309]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-20T09:34:48.818 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:48 smithi026 ceph-mon[68309]: osdmap e943: 12 total, 12 up, 11 in 2024-03-20T09:34:48.818 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:48 smithi026 ceph-mon[68309]: 1.15 deep-scrub starts 2024-03-20T09:34:48.818 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:48 smithi026 ceph-mon[68309]: 7.18 scrub starts 2024-03-20T09:34:48.818 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:48 smithi026 ceph-mon[68309]: 1.15 deep-scrub ok 2024-03-20T09:34:48.821 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:48.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:48 smithi122 ceph-mon[73495]: 1.c scrub starts 2024-03-20T09:34:48.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:48 smithi122 ceph-mon[73495]: 1.c scrub ok 2024-03-20T09:34:48.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:48 smithi122 ceph-mon[73495]: osdmap e942: 12 total, 12 up, 11 in 2024-03-20T09:34:48.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:48 smithi122 ceph-mon[73495]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-20T09:34:48.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:48 smithi122 ceph-mon[73495]: osdmap e943: 12 total, 12 up, 11 in 2024-03-20T09:34:48.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:48 smithi122 ceph-mon[73495]: 1.15 deep-scrub starts 2024-03-20T09:34:48.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:48 smithi122 ceph-mon[73495]: 7.18 scrub starts 2024-03-20T09:34:48.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:48 smithi122 ceph-mon[73495]: 1.15 deep-scrub ok 2024-03-20T09:34:49.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:48 smithi115 ceph-mon[72476]: 1.c scrub starts 2024-03-20T09:34:49.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:48 smithi115 ceph-mon[72476]: 1.c scrub ok 2024-03-20T09:34:49.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:48 smithi115 ceph-mon[72476]: osdmap e942: 12 total, 12 up, 11 in 2024-03-20T09:34:49.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:48 smithi115 ceph-mon[72476]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-20T09:34:49.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:48 smithi115 ceph-mon[72476]: osdmap e943: 12 total, 12 up, 11 in 2024-03-20T09:34:49.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:48 smithi115 ceph-mon[72476]: 1.15 deep-scrub starts 2024-03-20T09:34:49.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:48 smithi115 ceph-mon[72476]: 7.18 scrub starts 2024-03-20T09:34:49.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:48 smithi115 ceph-mon[72476]: 1.15 deep-scrub ok 2024-03-20T09:34:49.688 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:49 smithi026 ceph-mon[68309]: 7.18 scrub ok 2024-03-20T09:34:49.688 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:49 smithi026 ceph-mon[68309]: 7.16 scrub starts 2024-03-20T09:34:49.688 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:49 smithi026 ceph-mon[68309]: 7.16 scrub ok 2024-03-20T09:34:49.688 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:49 smithi026 ceph-mon[68309]: pgmap v2289: 57 pgs: 2 active+clean+remapped, 1 peering, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 47 active+clean; 1.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 15247/95631 objects misplaced (15.944%); 26 MiB/s, 421 objects/s recovering 2024-03-20T09:34:49.688 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:49 smithi026 ceph-mon[68309]: 7.14 scrub starts 2024-03-20T09:34:49.689 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:49 smithi026 ceph-mon[68309]: 7.14 scrub ok 2024-03-20T09:34:49.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:49 smithi122 ceph-mon[73495]: 7.18 scrub ok 2024-03-20T09:34:49.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:49 smithi122 ceph-mon[73495]: 7.16 scrub starts 2024-03-20T09:34:49.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:49 smithi122 ceph-mon[73495]: 7.16 scrub ok 2024-03-20T09:34:49.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:49 smithi122 ceph-mon[73495]: pgmap v2289: 57 pgs: 2 active+clean+remapped, 1 peering, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 47 active+clean; 1.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 15247/95631 objects misplaced (15.944%); 26 MiB/s, 421 objects/s recovering 2024-03-20T09:34:49.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:49 smithi122 ceph-mon[73495]: 7.14 scrub starts 2024-03-20T09:34:49.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:49 smithi122 ceph-mon[73495]: 7.14 scrub ok 2024-03-20T09:34:50.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:49 smithi115 ceph-mon[72476]: 7.18 scrub ok 2024-03-20T09:34:50.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:49 smithi115 ceph-mon[72476]: 7.16 scrub starts 2024-03-20T09:34:50.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:49 smithi115 ceph-mon[72476]: 7.16 scrub ok 2024-03-20T09:34:50.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:49 smithi115 ceph-mon[72476]: pgmap v2289: 57 pgs: 2 active+clean+remapped, 1 peering, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 47 active+clean; 1.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 15247/95631 objects misplaced (15.944%); 26 MiB/s, 421 objects/s recovering 2024-03-20T09:34:50.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:49 smithi115 ceph-mon[72476]: 7.14 scrub starts 2024-03-20T09:34:50.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:49 smithi115 ceph-mon[72476]: 7.14 scrub ok 2024-03-20T09:34:50.822 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:34:50.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:50.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:34:50.931 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:34:50.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:50 smithi122 ceph-mon[73495]: 1.10 scrub starts 2024-03-20T09:34:50.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:50 smithi122 ceph-mon[73495]: 1.10 scrub ok 2024-03-20T09:34:50.939 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:50 smithi026 ceph-mon[68309]: 1.10 scrub starts 2024-03-20T09:34:50.939 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:50 smithi026 ceph-mon[68309]: 1.10 scrub ok 2024-03-20T09:34:50.966 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:50 smithi115 ceph-mon[72476]: 1.10 scrub starts 2024-03-20T09:34:50.966 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:50 smithi115 ceph-mon[72476]: 1.10 scrub ok 2024-03-20T09:34:50.968 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:50.996 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:34:50.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:51.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:34:51.100 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:34:51.143 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:51.174 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:34:51.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:51.177 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:34:51.278 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:34:51.313 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:51.339 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:34:51.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:51.345 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:34:51.450 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:34:51.490 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:51.511 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:34:51.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:51.517 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:34:51.618 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:34:51.654 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:51.677 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:34:51.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:51.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:34:51.782 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:34:51.819 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:51 smithi122 ceph-mon[73495]: pgmap v2290: 57 pgs: 2 active+clean+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 48 active+clean; 1.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14685/92658 objects misplaced (15.849%) 2024-03-20T09:34:51.821 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:51.844 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:34:51.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:51.847 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:34:51.948 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:34:51.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:51 smithi026 ceph-mon[68309]: pgmap v2290: 57 pgs: 2 active+clean+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 48 active+clean; 1.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14685/92658 objects misplaced (15.849%) 2024-03-20T09:34:51.978 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:51 smithi115 ceph-mon[72476]: pgmap v2290: 57 pgs: 2 active+clean+remapped, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 48 active+clean; 1.9 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 14685/92658 objects misplaced (15.849%) 2024-03-20T09:34:51.980 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:52.005 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:34:52.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:52.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:34:52.038 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:34:52.039 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:34:52.039 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 943, 2024-03-20T09:34:52.039 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "852fbb68-e696-11ee-95c9-87774f69a715", 2024-03-20T09:34:52.039 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-03-20T08:48:35.281088+0000", 2024-03-20T09:34:52.039 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-03-20T09:34:47.775453+0000", 2024-03-20T09:34:52.039 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2024-03-20T09:30:44.068387+0000", 2024-03-20T09:34:52.039 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2024-03-20T09:28:18.556434+0000", 2024-03-20T09:34:52.039 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-20T09:34:52.039 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5799936, 2024-03-20T09:34:52.039 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2024-03-20T09:34:52.039 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2024-03-20T09:34:52.039 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2024-03-20T09:34:52.040 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2024-03-20T09:34:52.040 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2024-03-20T09:34:52.040 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:34:52.040 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 28, 2024-03-20T09:34:52.040 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2024-03-20T09:34:52.040 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-20T09:34:52.040 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-20T09:34:52.040 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2024-03-20T09:34:52.040 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 7, 2024-03-20T09:34:52.040 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 12, 2024-03-20T09:34:52.040 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "luminous", 2024-03-20T09:34:52.040 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "luminous", 2024-03-20T09:34:52.040 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "squid", 2024-03-20T09:34:52.041 INFO:teuthology.orchestra.run.smithi026.stdout: "allow_crimson": false, 2024-03-20T09:34:52.041 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2024-03-20T09:34:52.041 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.041 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2024-03-20T09:34:52.041 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": ".mgr", 2024-03-20T09:34:52.041 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-03-20T08:52:17.442324+0000", 2024-03-20T09:34:52.041 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-03-20T09:34:52.041 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-03-20T09:34:52.041 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-03-20T09:34:52.041 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 3, 2024-03-20T09:34:52.041 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 2, 2024-03-20T09:34:52.041 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-03-20T09:34:52.041 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-03-20T09:34:52.041 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-03-20T09:34:52.041 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-03-20T09:34:52.042 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-20T09:34:52.042 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-03-20T09:34:52.042 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-03-20T09:34:52.042 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 31, 2024-03-20T09:34:52.042 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 31, 2024-03-20T09:34:52.042 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 31, 2024-03-20T09:34:52.042 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 31, 2024-03-20T09:34:52.042 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 31, 2024-03-20T09:34:52.042 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-03-20T09:34:52.042 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-03-20T09:34:52.042 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-03-20T09:34:52.042 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-03-20T09:34:52.042 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-03-20T09:34:52.043 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-03-20T09:34:52.043 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-03-20T09:34:52.043 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.043 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "360", 2024-03-20T09:34:52.043 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-03-20T09:34:52.043 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-20T09:34:52.043 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "358", 2024-03-20T09:34:52.043 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-03-20T09:34:52.043 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-03-20T09:34:52.043 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-03-20T09:34:52.043 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-03-20T09:34:52.044 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-03-20T09:34:52.044 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-03-20T09:34:52.044 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-03-20T09:34:52.044 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-03-20T09:34:52.044 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-03-20T09:34:52.044 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-03-20T09:34:52.044 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-03-20T09:34:52.044 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-03-20T09:34:52.044 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-03-20T09:34:52.044 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-03-20T09:34:52.044 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-03-20T09:34:52.044 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-20T09:34:52.044 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-20T09:34:52.045 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-20T09:34:52.045 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-03-20T09:34:52.045 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-03-20T09:34:52.045 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-03-20T09:34:52.045 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-03-20T09:34:52.045 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-03-20T09:34:52.045 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.045 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-03-20T09:34:52.045 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-03-20T09:34:52.045 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-03-20T09:34:52.045 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-03-20T09:34:52.045 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-03-20T09:34:52.045 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-03-20T09:34:52.046 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-03-20T09:34:52.046 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-03-20T09:34:52.046 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-03-20T09:34:52.046 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-03-20T09:34:52.046 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-03-20T09:34:52.046 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2024-03-20T09:34:52.046 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_max": 32, 2024-03-20T09:34:52.046 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2024-03-20T09:34:52.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.046 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-03-20T09:34:52.046 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": {} 2024-03-20T09:34:52.046 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.047 INFO:teuthology.orchestra.run.smithi026.stdout: "read_balance": { 2024-03-20T09:34:52.047 INFO:teuthology.orchestra.run.smithi026.stdout: "score_type": "Fair distribution", 2024-03-20T09:34:52.047 INFO:teuthology.orchestra.run.smithi026.stdout: "score_acting": 2.130000114440918, 2024-03-20T09:34:52.047 INFO:teuthology.orchestra.run.smithi026.stdout: "score_stable": 2.130000114440918, 2024-03-20T09:34:52.047 INFO:teuthology.orchestra.run.smithi026.stdout: "optimal_score": 1, 2024-03-20T09:34:52.047 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_acting": 2.130000114440918, 2024-03-20T09:34:52.047 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_stable": 2.130000114440918, 2024-03-20T09:34:52.047 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity_weighted": 0.83817285299301147, 2024-03-20T09:34:52.047 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity": 0.9100000262260437, 2024-03-20T09:34:52.047 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-03-20T09:34:52.047 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.048 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.048 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.048 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 7, 2024-03-20T09:34:52.048 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "unique_pool_5", 2024-03-20T09:34:52.048 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-03-20T09:31:06.265628+0000", 2024-03-20T09:34:52.048 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-03-20T09:34:52.048 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-03-20T09:34:52.048 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-03-20T09:34:52.048 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 3, 2024-03-20T09:34:52.048 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 2, 2024-03-20T09:34:52.048 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-03-20T09:34:52.048 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-03-20T09:34:52.049 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-03-20T09:34:52.049 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-03-20T09:34:52.049 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-20T09:34:52.049 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-03-20T09:34:52.049 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-03-20T09:34:52.049 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 26, 2024-03-20T09:34:52.049 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 26, 2024-03-20T09:34:52.049 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 26, 2024-03-20T09:34:52.049 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 26, 2024-03-20T09:34:52.049 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 26, 2024-03-20T09:34:52.049 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-03-20T09:34:52.049 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-03-20T09:34:52.050 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-03-20T09:34:52.050 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-03-20T09:34:52.050 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-03-20T09:34:52.050 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-03-20T09:34:52.050 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-03-20T09:34:52.050 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.050 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "935", 2024-03-20T09:34:52.050 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-03-20T09:34:52.050 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-20T09:34:52.050 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "933", 2024-03-20T09:34:52.050 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-03-20T09:34:52.050 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-03-20T09:34:52.050 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-03-20T09:34:52.050 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-03-20T09:34:52.051 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-03-20T09:34:52.051 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-03-20T09:34:52.051 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-03-20T09:34:52.051 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-03-20T09:34:52.051 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-03-20T09:34:52.051 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-03-20T09:34:52.051 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-03-20T09:34:52.051 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-03-20T09:34:52.051 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-03-20T09:34:52.051 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-03-20T09:34:52.051 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-03-20T09:34:52.051 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-20T09:34:52.051 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-20T09:34:52.051 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-20T09:34:52.051 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-03-20T09:34:52.052 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-03-20T09:34:52.052 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-03-20T09:34:52.052 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-03-20T09:34:52.052 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-03-20T09:34:52.052 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.052 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-03-20T09:34:52.052 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-03-20T09:34:52.052 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-03-20T09:34:52.052 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-03-20T09:34:52.052 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-03-20T09:34:52.052 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-03-20T09:34:52.052 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-03-20T09:34:52.052 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-03-20T09:34:52.052 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-03-20T09:34:52.053 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-03-20T09:34:52.053 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-03-20T09:34:52.053 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2024-03-20T09:34:52.053 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-03-20T09:34:52.053 INFO:teuthology.orchestra.run.smithi026.stdout: "rados": {} 2024-03-20T09:34:52.053 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.053 INFO:teuthology.orchestra.run.smithi026.stdout: "read_balance": { 2024-03-20T09:34:52.053 INFO:teuthology.orchestra.run.smithi026.stdout: "score_type": "Fair distribution", 2024-03-20T09:34:52.053 INFO:teuthology.orchestra.run.smithi026.stdout: "score_acting": 2.9600000381469727, 2024-03-20T09:34:52.053 INFO:teuthology.orchestra.run.smithi026.stdout: "score_stable": 2.5399999618530273, 2024-03-20T09:34:52.053 INFO:teuthology.orchestra.run.smithi026.stdout: "optimal_score": 1, 2024-03-20T09:34:52.053 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_acting": 2.9600000381469727, 2024-03-20T09:34:52.053 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_stable": 2.5399999618530273, 2024-03-20T09:34:52.053 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity_weighted": 0.83817285299301147, 2024-03-20T09:34:52.053 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity": 0.9100000262260437, 2024-03-20T09:34:52.054 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity_weighted": 0.8399999737739563 2024-03-20T09:34:52.054 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.054 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.054 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:34:52.054 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:34:52.054 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.054 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-03-20T09:34:52.054 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "ad3fd9c0-316f-4fe5-a63b-d480087c1729", 2024-03-20T09:34:52.054 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:34:52.054 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:34:52.054 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:34:52.054 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:34:52.054 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:34:52.054 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:34:52.055 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 8, 2024-03-20T09:34:52.055 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 942, 2024-03-20T09:34:52.055 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:34:52.055 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:34:52.055 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:34:52.055 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.055 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.055 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2024-03-20T09:34:52.055 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:34:52.055 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.055 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.055 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.055 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2024-03-20T09:34:52.055 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:34:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:34:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2024-03-20T09:34:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:34:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2024-03-20T09:34:52.056 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:34:52.057 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.057 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.057 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:34:52.057 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.057 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.057 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2024-03-20T09:34:52.057 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:34:52.057 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.057 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.057 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.057 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2024-03-20T09:34:52.057 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:34:52.058 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.058 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.058 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.058 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:34:52.058 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.058 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.058 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.058 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2024-03-20T09:34:52.061 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:34:52.061 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.061 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.061 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2024-03-20T09:34:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:34:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/2139805293", 2024-03-20T09:34:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/2139805293", 2024-03-20T09:34:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/2139805293", 2024-03-20T09:34:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/2139805293", 2024-03-20T09:34:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:34:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:34:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:34:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-03-20T09:34:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e85b94f5-2954-47a1-9c20-096f2606eb04", 2024-03-20T09:34:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:34:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:34:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:34:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 0, 2024-03-20T09:34:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:34:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:34:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2024-03-20T09:34:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 814, 2024-03-20T09:34:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:34:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:34:52.063 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:34:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2024-03-20T09:34:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:34:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2024-03-20T09:34:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:34:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:34:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2024-03-20T09:34:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:34:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2024-03-20T09:34:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:34:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:34:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2024-03-20T09:34:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:34:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2024-03-20T09:34:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:34:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.066 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.067 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:34:52.067 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.067 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.067 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2024-03-20T09:34:52.067 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:34:52.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.067 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.067 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2024-03-20T09:34:52.067 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:34:52.067 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.067 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.068 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/3853355134", 2024-03-20T09:34:52.068 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/3853355134", 2024-03-20T09:34:52.068 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/3853355134", 2024-03-20T09:34:52.068 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/3853355134", 2024-03-20T09:34:52.068 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:34:52.068 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:34:52.068 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:34:52.068 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.068 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-03-20T09:34:52.068 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "dd2b10f0-741c-47a4-b224-ac6bfe2c14ef", 2024-03-20T09:34:52.069 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:34:52.069 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:34:52.069 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:34:52.069 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:34:52.069 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:34:52.069 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:34:52.069 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 18, 2024-03-20T09:34:52.069 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 941, 2024-03-20T09:34:52.069 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:34:52.069 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:34:52.069 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:34:52.069 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.069 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.070 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2024-03-20T09:34:52.070 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:34:52.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.070 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.070 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2024-03-20T09:34:52.070 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:34:52.070 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.070 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.070 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:34:52.070 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.071 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.071 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2024-03-20T09:34:52.071 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:34:52.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.071 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.071 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2024-03-20T09:34:52.071 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:34:52.071 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.071 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.071 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:34:52.071 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.072 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.072 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2024-03-20T09:34:52.072 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:34:52.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.072 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.072 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2024-03-20T09:34:52.072 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:34:52.072 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.072 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.072 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:34:52.072 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.073 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.073 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2024-03-20T09:34:52.073 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:34:52.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.073 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.073 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2024-03-20T09:34:52.073 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:34:52.073 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.073 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.073 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/3993357967", 2024-03-20T09:34:52.073 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/3993357967", 2024-03-20T09:34:52.074 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/3993357967", 2024-03-20T09:34:52.074 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/3993357967", 2024-03-20T09:34:52.074 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:34:52.074 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:34:52.074 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:34:52.074 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.074 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-03-20T09:34:52.074 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1", 2024-03-20T09:34:52.074 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:34:52.074 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:34:52.074 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:34:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:34:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:34:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:34:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 26, 2024-03-20T09:34:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 939, 2024-03-20T09:34:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:34:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:34:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:34:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2024-03-20T09:34:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:34:52.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2024-03-20T09:34:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:34:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:34:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2024-03-20T09:34:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:34:52.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2024-03-20T09:34:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:34:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:34:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2024-03-20T09:34:52.077 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:34:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2024-03-20T09:34:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:34:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:34:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.078 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2024-03-20T09:34:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:34:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2024-03-20T09:34:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:34:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/4110225509", 2024-03-20T09:34:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/4110225509", 2024-03-20T09:34:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/4110225509", 2024-03-20T09:34:52.079 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/4110225509", 2024-03-20T09:34:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:34:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:34:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:34:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-03-20T09:34:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "f14e34d5-9829-42a3-9b25-31a0976c5ef7", 2024-03-20T09:34:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:34:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:34:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:34:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:34:52.080 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:34:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:34:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 32, 2024-03-20T09:34:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 933, 2024-03-20T09:34:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:34:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:34:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:34:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6800", 2024-03-20T09:34:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:34:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.081 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6801", 2024-03-20T09:34:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:34:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:34:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6802", 2024-03-20T09:34:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:34:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.082 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6803", 2024-03-20T09:34:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:34:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:34:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6806", 2024-03-20T09:34:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:34:52.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6807", 2024-03-20T09:34:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:34:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:34:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.084 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6804", 2024-03-20T09:34:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:34:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6805", 2024-03-20T09:34:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:34:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6801/477923058", 2024-03-20T09:34:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6803/477923058", 2024-03-20T09:34:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6807/477923058", 2024-03-20T09:34:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6805/477923058", 2024-03-20T09:34:52.085 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:34:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:34:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:34:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-03-20T09:34:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "cb431b6a-540d-4318-ae35-d379e1df0185", 2024-03-20T09:34:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:34:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:34:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:34:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:34:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 41, 2024-03-20T09:34:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 273, 2024-03-20T09:34:52.086 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 443, 2024-03-20T09:34:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 935, 2024-03-20T09:34:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 274, 2024-03-20T09:34:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:34:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:34:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6808", 2024-03-20T09:34:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:34:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6809", 2024-03-20T09:34:52.087 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:34:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:34:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6810", 2024-03-20T09:34:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:34:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6811", 2024-03-20T09:34:52.088 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:34:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:34:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6814", 2024-03-20T09:34:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:34:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6815", 2024-03-20T09:34:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:34:52.089 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:34:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6812", 2024-03-20T09:34:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:34:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6813", 2024-03-20T09:34:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:34:52.090 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6809/650416648", 2024-03-20T09:34:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6811/650416648", 2024-03-20T09:34:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6815/650416648", 2024-03-20T09:34:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6813/650416648", 2024-03-20T09:34:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:34:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:34:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:34:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2024-03-20T09:34:52.091 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "753ed830-e764-4ff3-9d9b-87e7fa748b81", 2024-03-20T09:34:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:34:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:34:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:34:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:34:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:34:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:34:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 51, 2024-03-20T09:34:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 941, 2024-03-20T09:34:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:34:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:34:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:34:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.092 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6816", 2024-03-20T09:34:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:34:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6817", 2024-03-20T09:34:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:34:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:34:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.093 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6818", 2024-03-20T09:34:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:34:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6819", 2024-03-20T09:34:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:34:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:34:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.094 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.095 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6822", 2024-03-20T09:34:52.095 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:34:52.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.095 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.095 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6823", 2024-03-20T09:34:52.095 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:34:52.095 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.095 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.095 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:34:52.095 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.096 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.096 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6820", 2024-03-20T09:34:52.096 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:34:52.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.096 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.096 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6821", 2024-03-20T09:34:52.096 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:34:52.096 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.096 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.096 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6817/3278301843", 2024-03-20T09:34:52.096 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6819/3278301843", 2024-03-20T09:34:52.097 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6823/3278301843", 2024-03-20T09:34:52.097 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6821/3278301843", 2024-03-20T09:34:52.097 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:34:52.097 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:34:52.097 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:34:52.097 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.097 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2024-03-20T09:34:52.097 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "55bcbd37-7a6f-4ab1-b5a9-91a951444d86", 2024-03-20T09:34:52.097 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:34:52.097 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:34:52.097 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:34:52.097 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 0.058074951171875, 2024-03-20T09:34:52.098 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:34:52.098 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:34:52.098 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 60, 2024-03-20T09:34:52.098 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 517, 2024-03-20T09:34:52.098 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:34:52.098 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:34:52.098 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:34:52.098 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.098 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.098 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6824", 2024-03-20T09:34:52.098 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:34:52.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.099 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.099 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6825", 2024-03-20T09:34:52.099 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:34:52.099 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.099 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.099 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:34:52.099 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.099 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.099 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6826", 2024-03-20T09:34:52.099 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:34:52.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.100 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.100 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6827", 2024-03-20T09:34:52.100 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:34:52.100 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.100 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.100 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:34:52.100 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.100 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.100 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6830", 2024-03-20T09:34:52.100 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:34:52.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.101 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.101 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6831", 2024-03-20T09:34:52.101 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:34:52.101 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.101 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.101 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:34:52.101 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.101 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.101 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6828", 2024-03-20T09:34:52.101 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:34:52.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.102 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.102 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6829", 2024-03-20T09:34:52.102 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:34:52.102 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.102 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.102 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6825/3152424001", 2024-03-20T09:34:52.102 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6827/3152424001", 2024-03-20T09:34:52.102 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6831/3152424001", 2024-03-20T09:34:52.102 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6829/3152424001", 2024-03-20T09:34:52.102 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:34:52.103 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:34:52.103 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:34:52.103 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.103 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 8, 2024-03-20T09:34:52.103 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "bc86248e-6bd0-4369-b8a2-e593bc2a610d", 2024-03-20T09:34:52.103 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:34:52.103 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:34:52.103 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:34:52.103 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:34:52.103 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:34:52.103 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:34:52.103 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 70, 2024-03-20T09:34:52.103 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 942, 2024-03-20T09:34:52.104 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:34:52.104 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:34:52.104 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:34:52.104 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.104 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.104 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6800", 2024-03-20T09:34:52.104 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:34:52.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.104 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.104 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6801", 2024-03-20T09:34:52.104 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:34:52.104 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.105 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.105 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:34:52.105 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.105 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.105 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6802", 2024-03-20T09:34:52.105 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:34:52.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.105 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.105 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6803", 2024-03-20T09:34:52.105 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:34:52.105 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.105 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.106 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:34:52.106 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.106 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.106 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6806", 2024-03-20T09:34:52.106 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:34:52.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.106 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.106 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6807", 2024-03-20T09:34:52.106 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:34:52.106 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.107 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.107 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:34:52.107 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.107 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.107 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6804", 2024-03-20T09:34:52.107 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:34:52.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.107 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.107 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6805", 2024-03-20T09:34:52.107 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:34:52.107 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.108 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.108 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6801/1309426740", 2024-03-20T09:34:52.108 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6803/1309426740", 2024-03-20T09:34:52.108 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6807/1309426740", 2024-03-20T09:34:52.108 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6805/1309426740", 2024-03-20T09:34:52.108 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:34:52.108 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:34:52.108 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:34:52.108 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.108 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 9, 2024-03-20T09:34:52.109 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b20a4aa-032a-4c09-ac75-153c554424e7", 2024-03-20T09:34:52.109 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:34:52.109 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:34:52.109 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:34:52.109 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:34:52.109 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:34:52.109 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:34:52.109 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 80, 2024-03-20T09:34:52.109 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 935, 2024-03-20T09:34:52.109 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:34:52.109 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:34:52.109 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:34:52.109 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.109 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.110 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6808", 2024-03-20T09:34:52.110 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:34:52.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.110 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.110 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6809", 2024-03-20T09:34:52.110 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:34:52.110 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.110 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.110 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:34:52.110 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.110 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.111 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6810", 2024-03-20T09:34:52.111 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:34:52.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.111 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.111 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6811", 2024-03-20T09:34:52.111 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:34:52.111 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.111 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.111 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:34:52.111 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.112 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.112 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6814", 2024-03-20T09:34:52.112 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:34:52.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.112 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.112 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6815", 2024-03-20T09:34:52.112 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:34:52.112 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.112 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.112 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:34:52.112 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.113 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.113 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6812", 2024-03-20T09:34:52.113 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:34:52.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.113 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.113 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6813", 2024-03-20T09:34:52.113 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:34:52.113 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.113 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.113 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6809/2953210076", 2024-03-20T09:34:52.113 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6811/2953210076", 2024-03-20T09:34:52.113 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6815/2953210076", 2024-03-20T09:34:52.113 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6813/2953210076", 2024-03-20T09:34:52.114 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:34:52.114 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:34:52.114 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:34:52.114 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.114 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 10, 2024-03-20T09:34:52.114 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e62c1fb3-5dd1-4d0d-997c-d20633d19361", 2024-03-20T09:34:52.114 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:34:52.114 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 0, 2024-03-20T09:34:52.114 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-03-20T09:34:52.114 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:34:52.114 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 89, 2024-03-20T09:34:52.115 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 753, 2024-03-20T09:34:52.115 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 861, 2024-03-20T09:34:52.115 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 751, 2024-03-20T09:34:52.115 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 754, 2024-03-20T09:34:52.115 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:34:52.115 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:34:52.115 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.115 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.115 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6816", 2024-03-20T09:34:52.115 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2853088860 2024-03-20T09:34:52.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.115 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.116 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6817", 2024-03-20T09:34:52.116 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2853088860 2024-03-20T09:34:52.116 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.116 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.116 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:34:52.116 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.116 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.116 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6818", 2024-03-20T09:34:52.116 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2853088860 2024-03-20T09:34:52.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.116 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.117 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6819", 2024-03-20T09:34:52.117 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2853088860 2024-03-20T09:34:52.117 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.117 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.117 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:34:52.117 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.117 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.117 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6822", 2024-03-20T09:34:52.117 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2853088860 2024-03-20T09:34:52.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.117 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.118 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6823", 2024-03-20T09:34:52.118 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2853088860 2024-03-20T09:34:52.118 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.118 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.118 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:34:52.118 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.118 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.118 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6820", 2024-03-20T09:34:52.118 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2853088860 2024-03-20T09:34:52.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.119 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.119 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6821", 2024-03-20T09:34:52.119 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2853088860 2024-03-20T09:34:52.119 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.119 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.119 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6817/2853088860", 2024-03-20T09:34:52.119 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6819/2853088860", 2024-03-20T09:34:52.119 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6823/2853088860", 2024-03-20T09:34:52.119 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6821/2853088860", 2024-03-20T09:34:52.119 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:34:52.119 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:34:52.119 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:34:52.119 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.120 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 11, 2024-03-20T09:34:52.120 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b83994d5-e5ef-48b0-9758-c26843cec3b6", 2024-03-20T09:34:52.120 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:34:52.120 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:34:52.120 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:34:52.120 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:34:52.120 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:34:52.120 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:34:52.120 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 96, 2024-03-20T09:34:52.120 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 942, 2024-03-20T09:34:52.120 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:34:52.120 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:34:52.121 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:34:52.121 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.121 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.121 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6824", 2024-03-20T09:34:52.121 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:34:52.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.121 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.121 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6825", 2024-03-20T09:34:52.121 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:34:52.121 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.121 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.122 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:34:52.122 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.122 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.122 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6826", 2024-03-20T09:34:52.122 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:34:52.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.122 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.122 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6827", 2024-03-20T09:34:52.122 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:34:52.122 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.122 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.123 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:34:52.123 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.123 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.123 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6830", 2024-03-20T09:34:52.123 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:34:52.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.123 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.123 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6831", 2024-03-20T09:34:52.123 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:34:52.123 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.123 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.124 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:34:52.124 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:34:52.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.124 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:34:52.124 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6828", 2024-03-20T09:34:52.124 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:34:52.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.124 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:34:52.124 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6829", 2024-03-20T09:34:52.124 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:34:52.124 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.125 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.125 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6825/2704069354", 2024-03-20T09:34:52.125 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6827/2704069354", 2024-03-20T09:34:52.125 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6831/2704069354", 2024-03-20T09:34:52.125 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6829/2704069354", 2024-03-20T09:34:52.125 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:34:52.125 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:34:52.125 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:34:52.125 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.125 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.125 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:34:52.125 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2024-03-20T09:34:52.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.126 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-03-20T09:34:52.126 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:34:52.126 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:34:52.126 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:34:52.126 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:34:52.126 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:34:52.126 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:33:52.803286+0000", 2024-03-20T09:34:52.126 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:34:52.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.126 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-03-20T09:34:52.126 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:34:52.127 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:34:52.127 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:34:52.127 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:34:52.127 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:34:52.127 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:34:36.933749+0000", 2024-03-20T09:34:52.127 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:34:52.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.127 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-03-20T09:34:52.127 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:34:52.127 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:34:52.127 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:34:52.127 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:34:52.127 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:34:52.128 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:34:04.075039+0000", 2024-03-20T09:34:52.128 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:34:52.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.128 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-03-20T09:34:52.128 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:34:52.128 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:34:52.128 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:34:52.128 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:34:52.128 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:34:52.128 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:34:06.840195+0000", 2024-03-20T09:34:52.128 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:34:52.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.129 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-03-20T09:34:52.129 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:34:52.129 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:34:52.129 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:34:52.129 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:34:52.129 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:34:52.129 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:34:25.908052+0000", 2024-03-20T09:34:52.129 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:34:52.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.129 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-03-20T09:34:52.129 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:34:52.129 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:34:52.130 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:34:52.130 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:34:52.130 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:34:52.130 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:34:04.442555+0000", 2024-03-20T09:34:52.130 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 273 2024-03-20T09:34:52.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.130 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2024-03-20T09:34:52.130 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:34:52.130 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:34:52.130 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:34:52.130 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:34:52.130 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:34:52.130 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:33:45.910738+0000", 2024-03-20T09:34:52.131 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:34:52.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.131 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2024-03-20T09:34:52.131 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:34:52.131 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:34:52.131 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:34:52.131 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:34:52.131 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:34:52.131 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:33:44.962155+0000", 2024-03-20T09:34:52.131 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:34:52.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.131 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 8, 2024-03-20T09:34:52.132 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:34:52.132 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:34:52.132 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:34:52.132 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:34:52.132 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:34:52.132 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:34:38.522562+0000", 2024-03-20T09:34:52.132 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:34:52.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.132 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 9, 2024-03-20T09:34:52.132 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:34:52.132 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:34:52.132 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:34:52.133 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:34:52.133 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:34:52.133 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:34:20.225535+0000", 2024-03-20T09:34:52.133 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:34:52.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.133 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 10, 2024-03-20T09:34:52.133 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:34:52.133 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:34:52.133 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:34:52.133 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:34:52.133 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 65536, 2024-03-20T09:34:52.133 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:34:20.366163+0000", 2024-03-20T09:34:52.134 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 753 2024-03-20T09:34:52.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.134 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 11, 2024-03-20T09:34:52.134 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:34:52.134 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:34:52.134 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:34:52.134 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:34:52.134 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:34:52.134 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:34:31.159224+0000", 2024-03-20T09:34:52.134 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:34:52.134 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.134 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:34:52.134 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [ 2024-03-20T09:34:52.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.135 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "1.1", 2024-03-20T09:34:52.135 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:34:52.135 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-03-20T09:34:52.135 INFO:teuthology.orchestra.run.smithi026.stdout: 11, 2024-03-20T09:34:52.135 INFO:teuthology.orchestra.run.smithi026.stdout: 5 2024-03-20T09:34:52.135 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.135 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.135 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:34:52.135 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [ 2024-03-20T09:34:52.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.135 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "1.1c", 2024-03-20T09:34:52.135 INFO:teuthology.orchestra.run.smithi026.stdout: "mappings": [ 2024-03-20T09:34:52.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.136 INFO:teuthology.orchestra.run.smithi026.stdout: "from": 6, 2024-03-20T09:34:52.136 INFO:teuthology.orchestra.run.smithi026.stdout: "to": 2 2024-03-20T09:34:52.136 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.136 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.136 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.136 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:34:52.136 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_primaries": [], 2024-03-20T09:34:52.136 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [ 2024-03-20T09:34:52.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.136 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "7.11", 2024-03-20T09:34:52.136 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:34:52.136 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-03-20T09:34:52.136 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-03-20T09:34:52.136 INFO:teuthology.orchestra.run.smithi026.stdout: 4 2024-03-20T09:34:52.136 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.137 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "7.13", 2024-03-20T09:34:52.137 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:34:52.137 INFO:teuthology.orchestra.run.smithi026.stdout: 2, 2024-03-20T09:34:52.137 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-03-20T09:34:52.137 INFO:teuthology.orchestra.run.smithi026.stdout: 4 2024-03-20T09:34:52.137 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.137 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "7.15", 2024-03-20T09:34:52.137 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:34:52.137 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-03-20T09:34:52.137 INFO:teuthology.orchestra.run.smithi026.stdout: 9, 2024-03-20T09:34:52.138 INFO:teuthology.orchestra.run.smithi026.stdout: 11 2024-03-20T09:34:52.138 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.138 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "7.17", 2024-03-20T09:34:52.138 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:34:52.138 INFO:teuthology.orchestra.run.smithi026.stdout: 3, 2024-03-20T09:34:52.138 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-03-20T09:34:52.138 INFO:teuthology.orchestra.run.smithi026.stdout: 8 2024-03-20T09:34:52.138 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:34:52.138 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "7.19", 2024-03-20T09:34:52.138 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:34:52.139 INFO:teuthology.orchestra.run.smithi026.stdout: 0, 2024-03-20T09:34:52.139 INFO:teuthology.orchestra.run.smithi026.stdout: 8, 2024-03-20T09:34:52.139 INFO:teuthology.orchestra.run.smithi026.stdout: 11 2024-03-20T09:34:52.139 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:34:52.139 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.139 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:34:52.139 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2024-03-20T09:34:52.139 INFO:teuthology.orchestra.run.smithi026.stdout: "blocklist": { 2024-03-20T09:34:52.139 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/4291404191": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:34:52.139 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/642012566": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:34:52.139 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6801/1181459999": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:34:52.139 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/3984513952": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:34:52.139 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6800/1654945498": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:34:52.139 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6800/1181459999": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:34:52.140 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/137770981": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:34:52.140 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/1451237762": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:34:52.140 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/3938386631": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:34:52.140 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6801/1654945498": "2024-03-21T08:49:04.596754+0000" 2024-03-20T09:34:52.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.140 INFO:teuthology.orchestra.run.smithi026.stdout: "range_blocklist": {}, 2024-03-20T09:34:52.140 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2024-03-20T09:34:52.140 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2024-03-20T09:34:52.140 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2024-03-20T09:34:52.140 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2024-03-20T09:34:52.140 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2024-03-20T09:34:52.140 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2024-03-20T09:34:52.140 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2024-03-20T09:34:52.140 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:34:52.141 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2024-03-20T09:34:52.141 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2024-03-20T09:34:52.141 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2024-03-20T09:34:52.141 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2024-03-20T09:34:52.141 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {}, 2024-03-20T09:34:52.141 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": { 2024-03-20T09:34:52.141 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_enabled": false, 2024-03-20T09:34:52.141 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_bucket_count": 0, 2024-03-20T09:34:52.141 INFO:teuthology.orchestra.run.smithi026.stdout: "degraded_stretch_mode": 0, 2024-03-20T09:34:52.141 INFO:teuthology.orchestra.run.smithi026.stdout: "recovering_stretch_mode": 0, 2024-03-20T09:34:52.141 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_bucket": 0 2024-03-20T09:34:52.141 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:34:52.141 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:34:52.142 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:34:52.185 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:52.208 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:34:52.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:52.214 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:34:52.315 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:34:52.350 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:52.376 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:34:52.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:52.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:34:52.482 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:34:52.552 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:52.627 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:34:52.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:52.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:34:52.732 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:34:52.753 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:34:52.764 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:52 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2268437073' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:34:52.764 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:52 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3602005619' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:34:52.764 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:52 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:34:52.841 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:53.118 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-20T09:34:53.122+0000 7fcb4d655700 0 --2- 172.21.15.26:0/2736060006 >> [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] conn(0x7fcb481014b0 0x7fcb48112ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-03-20T09:34:53.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:52 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2268437073' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:34:53.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:52 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3602005619' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:34:53.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:52 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:34:53.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:52 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2268437073' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:34:53.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:52 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3602005619' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:34:53.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:52 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:34:53.193 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:34:53.232 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:34:53.291 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:34:53.349 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:34:53.389 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:34:53.435 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:34:53.473 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:34:53.516 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:34:53.567 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:34:53.606 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:34:53.657 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:34:53.697 INFO:tasks.thrashosds.thrasher:j is {'epoch': 943, 'fsid': '852fbb68-e696-11ee-95c9-87774f69a715', 'created': '2024-03-20T08:48:35.281088+0000', 'modified': '2024-03-20T09:34:47.775453+0000', 'last_up_change': '2024-03-20T09:30:44.068387+0000', 'last_in_change': '2024-03-20T09:28:18.556434+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 28, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-20T08:52:17.442324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '360', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '358', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.130000114440918, 'score_stable': 2.130000114440918, 'optimal_score': 1, 'raw_score_acting': 2.130000114440918, 'raw_score_stable': 2.130000114440918, 'primary_affinity_weighted': 0.8381728529930115, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.8399999737739563}}, {'pool': 7, 'pool_name': 'unique_pool_5', 'create_time': '2024-03-20T09:31:06.265628+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '935', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '933', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.9600000381469727, 'score_stable': 2.5399999618530273, 'optimal_score': 1, 'raw_score_acting': 2.9600000381469727, 'raw_score_stable': 2.5399999618530273, 'primary_affinity_weighted': 0.8381728529930115, 'average_primary_affinity': 0.9100000262260437, 'average_primary_affinity_weighted': 0.8399999737739563}}], 'osds': [{'osd': 0, 'uuid': 'ad3fd9c0-316f-4fe5-a63b-d480087c1729', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 942, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6802', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6803', 'nonce': 2139805293}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6804', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6805', 'nonce': 2139805293}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6808', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6809', 'nonce': 2139805293}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6806', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6807', 'nonce': 2139805293}]}, 'public_addr': '172.21.15.26:6803/2139805293', 'cluster_addr': '172.21.15.26:6805/2139805293', 'heartbeat_back_addr': '172.21.15.26:6809/2139805293', 'heartbeat_front_addr': '172.21.15.26:6807/2139805293', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'e85b94f5-2954-47a1-9c20-096f2606eb04', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 814, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6810', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6811', 'nonce': 3853355134}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6812', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6813', 'nonce': 3853355134}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6816', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6817', 'nonce': 3853355134}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6814', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6815', 'nonce': 3853355134}]}, 'public_addr': '172.21.15.26:6811/3853355134', 'cluster_addr': '172.21.15.26:6813/3853355134', 'heartbeat_back_addr': '172.21.15.26:6817/3853355134', 'heartbeat_front_addr': '172.21.15.26:6815/3853355134', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'dd2b10f0-741c-47a4-b224-ac6bfe2c14ef', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 941, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6818', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6819', 'nonce': 3993357967}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6820', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6821', 'nonce': 3993357967}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6824', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6825', 'nonce': 3993357967}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6822', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6823', 'nonce': 3993357967}]}, 'public_addr': '172.21.15.26:6819/3993357967', 'cluster_addr': '172.21.15.26:6821/3993357967', 'heartbeat_back_addr': '172.21.15.26:6825/3993357967', 'heartbeat_front_addr': '172.21.15.26:6823/3993357967', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 939, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6826', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6827', 'nonce': 4110225509}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6828', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6829', 'nonce': 4110225509}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6832', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6833', 'nonce': 4110225509}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6830', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6831', 'nonce': 4110225509}]}, 'public_addr': '172.21.15.26:6827/4110225509', 'cluster_addr': '172.21.15.26:6829/4110225509', 'heartbeat_back_addr': '172.21.15.26:6833/4110225509', 'heartbeat_front_addr': '172.21.15.26:6831/4110225509', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'f14e34d5-9829-42a3-9b25-31a0976c5ef7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 933, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6800', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6801', 'nonce': 477923058}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6802', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6803', 'nonce': 477923058}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6806', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6807', 'nonce': 477923058}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6804', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6805', 'nonce': 477923058}]}, 'public_addr': '172.21.15.115:6801/477923058', 'cluster_addr': '172.21.15.115:6803/477923058', 'heartbeat_back_addr': '172.21.15.115:6807/477923058', 'heartbeat_front_addr': '172.21.15.115:6805/477923058', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'cb431b6a-540d-4318-ae35-d379e1df0185', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 41, 'last_clean_end': 273, 'up_from': 443, 'up_thru': 935, 'down_at': 274, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6808', 'nonce': 650416648}, {'type': 'v1', 'addr': '172.21.15.115:6809', 'nonce': 650416648}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6810', 'nonce': 650416648}, {'type': 'v1', 'addr': '172.21.15.115:6811', 'nonce': 650416648}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6814', 'nonce': 650416648}, {'type': 'v1', 'addr': '172.21.15.115:6815', 'nonce': 650416648}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6812', 'nonce': 650416648}, {'type': 'v1', 'addr': '172.21.15.115:6813', 'nonce': 650416648}]}, 'public_addr': '172.21.15.115:6809/650416648', 'cluster_addr': '172.21.15.115:6811/650416648', 'heartbeat_back_addr': '172.21.15.115:6815/650416648', 'heartbeat_front_addr': '172.21.15.115:6813/650416648', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '753ed830-e764-4ff3-9d9b-87e7fa748b81', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 941, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6816', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6817', 'nonce': 3278301843}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6818', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6819', 'nonce': 3278301843}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6822', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6823', 'nonce': 3278301843}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6820', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6821', 'nonce': 3278301843}]}, 'public_addr': '172.21.15.115:6817/3278301843', 'cluster_addr': '172.21.15.115:6819/3278301843', 'heartbeat_back_addr': '172.21.15.115:6823/3278301843', 'heartbeat_front_addr': '172.21.15.115:6821/3278301843', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '55bcbd37-7a6f-4ab1-b5a9-91a951444d86', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.058074951171875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 517, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6824', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6825', 'nonce': 3152424001}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6826', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6827', 'nonce': 3152424001}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6830', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6831', 'nonce': 3152424001}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6828', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6829', 'nonce': 3152424001}]}, 'public_addr': '172.21.15.115:6825/3152424001', 'cluster_addr': '172.21.15.115:6827/3152424001', 'heartbeat_back_addr': '172.21.15.115:6831/3152424001', 'heartbeat_front_addr': '172.21.15.115:6829/3152424001', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'bc86248e-6bd0-4369-b8a2-e593bc2a610d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 70, 'up_thru': 942, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6800', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6801', 'nonce': 1309426740}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6802', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6803', 'nonce': 1309426740}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6806', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6807', 'nonce': 1309426740}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6804', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6805', 'nonce': 1309426740}]}, 'public_addr': '172.21.15.122:6801/1309426740', 'cluster_addr': '172.21.15.122:6803/1309426740', 'heartbeat_back_addr': '172.21.15.122:6807/1309426740', 'heartbeat_front_addr': '172.21.15.122:6805/1309426740', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '9b20a4aa-032a-4c09-ac75-153c554424e7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 935, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6808', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6809', 'nonce': 2953210076}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6810', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6811', 'nonce': 2953210076}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6814', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6815', 'nonce': 2953210076}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6812', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6813', 'nonce': 2953210076}]}, 'public_addr': '172.21.15.122:6809/2953210076', 'cluster_addr': '172.21.15.122:6811/2953210076', 'heartbeat_back_addr': '172.21.15.122:6815/2953210076', 'heartbeat_front_addr': '172.21.15.122:6813/2953210076', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'e62c1fb3-5dd1-4d0d-997c-d20633d19361', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 89, 'last_clean_end': 753, 'up_from': 861, 'up_thru': 751, 'down_at': 754, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6816', 'nonce': 2853088860}, {'type': 'v1', 'addr': '172.21.15.122:6817', 'nonce': 2853088860}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6818', 'nonce': 2853088860}, {'type': 'v1', 'addr': '172.21.15.122:6819', 'nonce': 2853088860}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6822', 'nonce': 2853088860}, {'type': 'v1', 'addr': '172.21.15.122:6823', 'nonce': 2853088860}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6820', 'nonce': 2853088860}, {'type': 'v1', 'addr': '172.21.15.122:6821', 'nonce': 2853088860}]}, 'public_addr': '172.21.15.122:6817/2853088860', 'cluster_addr': '172.21.15.122:6819/2853088860', 'heartbeat_back_addr': '172.21.15.122:6823/2853088860', 'heartbeat_front_addr': '172.21.15.122:6821/2853088860', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'b83994d5-e5ef-48b0-9758-c26843cec3b6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 96, 'up_thru': 942, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6824', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6825', 'nonce': 2704069354}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6826', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6827', 'nonce': 2704069354}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6830', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6831', 'nonce': 2704069354}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6828', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6829', 'nonce': 2704069354}]}, 'public_addr': '172.21.15.122:6825/2704069354', 'cluster_addr': '172.21.15.122:6827/2704069354', 'heartbeat_back_addr': '172.21.15.122:6831/2704069354', 'heartbeat_front_addr': '172.21.15.122:6829/2704069354', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:33:52.803286+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:34:36.933749+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:34:04.075039+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:34:06.840195+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:34:25.908052+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:34:04.442555+0000', 'dead_epoch': 273}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:33:45.910738+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:33:44.962155+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:34:38.522562+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:34:20.225535+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-20T09:34:20.366163+0000', 'dead_epoch': 753}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:34:31.159224+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.1', 'osds': [6, 11, 5]}], 'pg_upmap_items': [{'pgid': '1.1c', 'mappings': [{'from': 6, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '7.11', 'osds': [3, 6, 4]}, {'pgid': '7.13', 'osds': [2, 3, 4]}, {'pgid': '7.15', 'osds': [0, 9, 11]}, {'pgid': '7.17', 'osds': [3, 6, 8]}, {'pgid': '7.19', 'osds': [0, 8, 11]}], 'primary_temp': [], 'blocklist': {'172.21.15.26:0/4291404191': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/642012566': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:6801/1181459999': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/3984513952': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6800/1654945498': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6800/1181459999': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/137770981': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:0/1451237762': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/3938386631': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6801/1654945498': '2024-03-21T08:49:04.596754+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-20T09:34:53.698 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 1.1c 2024-03-20T09:34:53.698 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd rm-pg-upmap-items 1.1c 2024-03-20T09:34:53.714 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:34:53.731 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:34:53.731 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:34:53.731 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:34:53.731 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:34:53.731 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:34:53.731 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:34:53.731 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:34:53.731 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:34:53.731 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:34:53.731 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:34:53.731 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:34:53.732 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:34:53.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:53 smithi026 ceph-mon[68309]: pgmap v2291: 57 pgs: 5 active+remapped+backfill_toofull, 52 active+clean; 1.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 12370/85224 objects misplaced (14.515%) 2024-03-20T09:34:53.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:53 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:34:53.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:53 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:34:53.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:53 smithi026 ceph-mon[68309]: osdmap e944: 12 total, 12 up, 11 in 2024-03-20T09:34:53.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:53 smithi026 ceph-mon[68309]: osdmap e945: 12 total, 12 up, 11 in 2024-03-20T09:34:53.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:53 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2719288419' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:34:54.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:53 smithi115 ceph-mon[72476]: pgmap v2291: 57 pgs: 5 active+remapped+backfill_toofull, 52 active+clean; 1.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 12370/85224 objects misplaced (14.515%) 2024-03-20T09:34:54.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:53 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:34:54.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:53 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:34:54.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:53 smithi115 ceph-mon[72476]: osdmap e944: 12 total, 12 up, 11 in 2024-03-20T09:34:54.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:53 smithi115 ceph-mon[72476]: osdmap e945: 12 total, 12 up, 11 in 2024-03-20T09:34:54.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:53 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2719288419' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:34:54.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:53 smithi122 ceph-mon[73495]: pgmap v2291: 57 pgs: 5 active+remapped+backfill_toofull, 52 active+clean; 1.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 12370/85224 objects misplaced (14.515%) 2024-03-20T09:34:54.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:53 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:34:54.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:53 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:34:54.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:53 smithi122 ceph-mon[73495]: osdmap e944: 12 total, 12 up, 11 in 2024-03-20T09:34:54.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:53 smithi122 ceph-mon[73495]: osdmap e945: 12 total, 12 up, 11 in 2024-03-20T09:34:54.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:53 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2719288419' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:34:54.643 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:34:54.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:54.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:34:54.748 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:34:54.789 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:54.809 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:34:54.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:54.813 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:34:54.915 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:34:54.975 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:55.144 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:34:55.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:55.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:34:55.253 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:34:55.303 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:55.330 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:34:55.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:55.337 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:34:55.395 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:34:55.438 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:34:55.512 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:55.694 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:34:55.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:55.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:34:55.799 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:34:55.831 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:55.857 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:34:55.862 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:55.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:34:55.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:55 smithi026 ceph-mon[68309]: pgmap v2294: 57 pgs: 5 active+remapped+backfill_toofull, 52 active+clean; 1.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 11914/82371 objects misplaced (14.464%) 2024-03-20T09:34:55.964 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:34:55.992 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:55 smithi115 ceph-mon[72476]: pgmap v2294: 57 pgs: 5 active+remapped+backfill_toofull, 52 active+clean; 1.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 11914/82371 objects misplaced (14.464%) 2024-03-20T09:34:55.994 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:56.005 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:34:56.005 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:34:56.020 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:34:56.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:56.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:34:56.044 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:34:56.126 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:34:56.156 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:55 smithi122 ceph-mon[73495]: pgmap v2294: 57 pgs: 5 active+remapped+backfill_toofull, 52 active+clean; 1.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 11914/82371 objects misplaced (14.464%) 2024-03-20T09:34:56.158 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:56.182 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:34:56.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:56.186 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:34:56.287 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:34:56.318 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:56.341 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:34:56.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:56.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:34:56.448 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:34:56.487 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:56.514 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:34:56.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:56.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:34:56.620 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:34:56.651 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:56.677 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:34:56.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:56.681 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:34:56.782 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:34:56.828 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:57.061 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:34:57.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:57.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:34:57.168 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:34:57.178 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:34:57.207 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:57.232 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:34:57.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:57.234 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:34:57.335 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:34:57.366 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:57.393 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:34:57.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:57.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:34:57.499 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:34:57.539 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:57.565 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:34:57.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:57.570 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:34:57.643 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-20T09:34:57.671 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:34:57.766 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:57.834 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:34:57.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:57.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:34:57.860 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:57 smithi026 ceph-mon[68309]: pgmap v2295: 57 pgs: 5 active+remapped+backfill_toofull, 52 active+clean; 1.5 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 11155/74553 objects misplaced (14.963%) 2024-03-20T09:34:57.936 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:34:57.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:57 smithi122 ceph-mon[73495]: pgmap v2295: 57 pgs: 5 active+remapped+backfill_toofull, 52 active+clean; 1.5 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 11155/74553 objects misplaced (14.963%) 2024-03-20T09:34:57.952 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:34:57.953 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:34:58.058 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:58.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:57 smithi115 ceph-mon[72476]: pgmap v2295: 57 pgs: 5 active+remapped+backfill_toofull, 52 active+clean; 1.5 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 11155/74553 objects misplaced (14.963%) 2024-03-20T09:34:58.150 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:34:58.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:58.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:34:58.258 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:34:58.324 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:59.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:58 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:34:59.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:58 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:34:59.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:58 smithi115 ceph-mon[72476]: pgmap v2296: 57 pgs: 5 active+remapped+backfill_toofull, 52 active+clean; 1.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 9652/67353 objects misplaced (14.330%) 2024-03-20T09:34:59.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:58 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:34:59.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:58 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:34:59.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:58 smithi122 ceph-mon[73495]: pgmap v2296: 57 pgs: 5 active+remapped+backfill_toofull, 52 active+clean; 1.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 9652/67353 objects misplaced (14.330%) 2024-03-20T09:34:59.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:58 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:34:59.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:58 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:34:59.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:58 smithi026 ceph-mon[68309]: pgmap v2296: 57 pgs: 5 active+remapped+backfill_toofull, 52 active+clean; 1.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 9652/67353 objects misplaced (14.330%) 2024-03-20T09:34:59.338 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:34:59.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:59.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:34:59.444 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:34:59.486 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:59.513 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:34:59.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:59.516 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:34:59.617 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:34:59.732 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:34:59.770 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:34:59.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:34:59.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:34:59.846 INFO:teuthology.orchestra.run.smithi026.stderr:clear 1.1c pg_upmap_items mapping 2024-03-20T09:34:59.876 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:34:59.987 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:00.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:59 smithi115 ceph-mon[72476]: osdmap e946: 12 total, 12 up, 11 in 2024-03-20T09:35:00.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:59 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2594633554' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "1.1c"}]: dispatch 2024-03-20T09:35:00.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:34:59 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "1.1c"}]: dispatch 2024-03-20T09:35:00.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:59 smithi122 ceph-mon[73495]: osdmap e946: 12 total, 12 up, 11 in 2024-03-20T09:35:00.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:59 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2594633554' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "1.1c"}]: dispatch 2024-03-20T09:35:00.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:34:59 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "1.1c"}]: dispatch 2024-03-20T09:35:00.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:59 smithi026 ceph-mon[68309]: osdmap e946: 12 total, 12 up, 11 in 2024-03-20T09:35:00.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:59 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2594633554' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "1.1c"}]: dispatch 2024-03-20T09:35:00.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:34:59 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "1.1c"}]: dispatch 2024-03-20T09:35:00.782 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:35:00.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:00.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:35:00.887 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:35:00.937 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:00.960 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:35:00.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:00.965 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:35:01.066 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:35:01.094 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:00 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:35:01.094 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:00 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:35:01.094 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:00 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "1.1c"}]': finished 2024-03-20T09:35:01.094 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:00 smithi115 ceph-mon[72476]: osdmap e947: 12 total, 12 up, 11 in 2024-03-20T09:35:01.094 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:00 smithi115 ceph-mon[72476]: pgmap v2299: 57 pgs: 5 active+remapped+backfill_toofull, 52 active+clean; 1.3 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 930 op/s; 8964/63654 objects misplaced (14.082%) 2024-03-20T09:35:01.097 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:01.123 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:35:01.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:01.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:35:01.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:00 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:35:01.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:00 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:35:01.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:00 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "1.1c"}]': finished 2024-03-20T09:35:01.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:00 smithi122 ceph-mon[73495]: osdmap e947: 12 total, 12 up, 11 in 2024-03-20T09:35:01.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:00 smithi122 ceph-mon[73495]: pgmap v2299: 57 pgs: 5 active+remapped+backfill_toofull, 52 active+clean; 1.3 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 930 op/s; 8964/63654 objects misplaced (14.082%) 2024-03-20T09:35:01.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:00 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:35:01.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:00 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:35:01.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:00 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "1.1c"}]': finished 2024-03-20T09:35:01.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:00 smithi026 ceph-mon[68309]: osdmap e947: 12 total, 12 up, 11 in 2024-03-20T09:35:01.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:00 smithi026 ceph-mon[68309]: pgmap v2299: 57 pgs: 5 active+remapped+backfill_toofull, 52 active+clean; 1.3 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 930 op/s; 8964/63654 objects misplaced (14.082%) 2024-03-20T09:35:01.228 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:35:01.261 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:01.295 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:35:01.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:01.299 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:35:01.400 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:35:01.430 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:01.453 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:35:01.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:01.457 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:35:01.558 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:35:01.591 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:01.619 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:35:01.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:01.627 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:35:01.728 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:35:01.827 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:01.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:01 smithi122 ceph-mon[73495]: osdmap e948: 12 total, 12 up, 11 in 2024-03-20T09:35:01.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:01 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/358591311' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:35:02.074 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:01 smithi026 ceph-mon[68309]: osdmap e948: 12 total, 12 up, 11 in 2024-03-20T09:35:02.074 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:01 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/358591311' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:35:02.093 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:35:02.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:02.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:35:02.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:01 smithi115 ceph-mon[72476]: osdmap e948: 12 total, 12 up, 11 in 2024-03-20T09:35:02.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:01 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/358591311' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:35:02.200 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:35:02.234 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:35:02.234 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:35:02.234 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:35:02.234 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:35:02.234 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:35:02.234 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:35:02.234 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:35:02.235 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:35:02.235 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:35:02.235 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:35:02.235 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:35:02.235 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:35:02.235 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:35:02.235 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:35:02.235 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:35:02.235 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:35:02.235 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:35:02.235 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:35:02.236 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:35:02.236 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:35:02.236 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:35:02.236 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:35:02.236 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:35:02.236 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:35:02.244 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:02.275 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:35:02.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:02.280 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:35:02.381 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:35:02.449 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:02.935 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:35:02.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:02.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:35:03.041 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:35:03.071 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:03.102 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:35:03.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:03.109 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:35:03.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:02 smithi115 ceph-mon[72476]: osdmap e949: 12 total, 12 up, 11 in 2024-03-20T09:35:03.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:02 smithi115 ceph-mon[72476]: pgmap v2302: 57 pgs: 1 remapped+peering, 5 active+remapped+backfill_toofull, 51 active+clean; 1.1 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s; 7923/55104 objects misplaced (14.378%) 2024-03-20T09:35:03.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:02 smithi115 ceph-mon[72476]: osdmap e950: 12 total, 12 up, 11 in 2024-03-20T09:35:03.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:02 smithi122 ceph-mon[73495]: osdmap e949: 12 total, 12 up, 11 in 2024-03-20T09:35:03.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:02 smithi122 ceph-mon[73495]: pgmap v2302: 57 pgs: 1 remapped+peering, 5 active+remapped+backfill_toofull, 51 active+clean; 1.1 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s; 7923/55104 objects misplaced (14.378%) 2024-03-20T09:35:03.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:02 smithi122 ceph-mon[73495]: osdmap e950: 12 total, 12 up, 11 in 2024-03-20T09:35:03.193 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:02 smithi026 ceph-mon[68309]: osdmap e949: 12 total, 12 up, 11 in 2024-03-20T09:35:03.194 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:02 smithi026 ceph-mon[68309]: pgmap v2302: 57 pgs: 1 remapped+peering, 5 active+remapped+backfill_toofull, 51 active+clean; 1.1 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s; 7923/55104 objects misplaced (14.378%) 2024-03-20T09:35:03.194 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:02 smithi026 ceph-mon[68309]: osdmap e950: 12 total, 12 up, 11 in 2024-03-20T09:35:03.210 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:35:03.271 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:03.481 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:35:03.499 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:35:03.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:03.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:35:03.622 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:35:03.651 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:03.681 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:35:03.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:03.686 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:35:03.787 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:35:03.819 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:03.842 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:35:03.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:03.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:35:03.948 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:35:03.978 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:04.005 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:35:04.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:04.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:35:04.110 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:35:04.154 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:04.179 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:35:04.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:04.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:35:04.284 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:35:04.320 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:04.346 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:35:04.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:04.349 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:35:04.450 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:35:04.486 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:04.517 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:35:04.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:04.522 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:35:04.623 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:35:04.668 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:04.695 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:35:04.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:04.701 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:35:04.802 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:35:04.807 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:04 smithi026 ceph-mon[68309]: osdmap e951: 12 total, 12 up, 11 in 2024-03-20T09:35:04.808 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:04 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3836843829' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:35:04.808 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:04 smithi026 ceph-mon[68309]: pgmap v2305: 57 pgs: 1 remapped+peering, 5 active+remapped+backfill_toofull, 51 active+clean; 1.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 943 op/s; 7434/52332 objects misplaced (14.205%) 2024-03-20T09:35:04.818 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:35:04.838 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:04.860 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:35:04.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:04.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:35:04.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:04 smithi122 ceph-mon[73495]: osdmap e951: 12 total, 12 up, 11 in 2024-03-20T09:35:04.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:04 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3836843829' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:35:04.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:04 smithi122 ceph-mon[73495]: pgmap v2305: 57 pgs: 1 remapped+peering, 5 active+remapped+backfill_toofull, 51 active+clean; 1.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 943 op/s; 7434/52332 objects misplaced (14.205%) 2024-03-20T09:35:04.961 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:35:04.964 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:35:05.003 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:05.036 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:35:05.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:05.043 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:35:05.143 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:35:05.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:04 smithi115 ceph-mon[72476]: osdmap e951: 12 total, 12 up, 11 in 2024-03-20T09:35:05.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:04 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3836843829' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:35:05.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:04 smithi115 ceph-mon[72476]: pgmap v2305: 57 pgs: 1 remapped+peering, 5 active+remapped+backfill_toofull, 51 active+clean; 1.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 943 op/s; 7434/52332 objects misplaced (14.205%) 2024-03-20T09:35:05.173 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:05.205 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:35:05.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:05.207 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:35:05.308 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:35:05.340 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:05.368 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:35:05.372 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:05.373 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:35:05.474 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:35:05.557 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:05.612 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:35:05.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:05.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:35:05.733 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:35:05.767 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:05.796 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:35:05.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:05.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:35:05.894 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:05 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:35:05.895 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:05 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3836843829' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:35:05.895 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:05 smithi026 ceph-mon[68309]: osdmap e952: 12 total, 12 up, 11 in 2024-03-20T09:35:05.903 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:35:05.971 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:06.098 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:35:06.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:06.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:35:06.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:05 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:35:06.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:05 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3836843829' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:35:06.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:05 smithi115 ceph-mon[72476]: osdmap e952: 12 total, 12 up, 11 in 2024-03-20T09:35:06.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:05 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:35:06.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:05 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3836843829' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:35:06.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:05 smithi122 ceph-mon[73495]: osdmap e952: 12 total, 12 up, 11 in 2024-03-20T09:35:06.206 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:35:06.274 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:35:06.296 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:35:06.296 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:35:06.297 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-03-20T09:35:06.297 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:35:06.314 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:06.594 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 8, 11, 3, 4, 9] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 11, 5, 10] 2024-03-20T09:35:06.594 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:35:06.594 INFO:tasks.thrashosds.thrasher:Adding osd 10 2024-03-20T09:35:06.594 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd in 10 2024-03-20T09:35:06.701 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:35:06.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:06.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:35:06.816 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:35:06.849 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:06.875 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:35:06.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:06.879 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:35:06.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:06 smithi026 ceph-mon[68309]: pgmap v2307: 57 pgs: 1 remapped+peering, 5 active+remapped+backfill_toofull, 51 active+clean; 922 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s; 6714/44217 objects misplaced (15.184%) 2024-03-20T09:35:06.979 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:35:07.009 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:07.038 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:35:07.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:07.043 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:35:07.144 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:35:07.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:06 smithi115 ceph-mon[72476]: pgmap v2307: 57 pgs: 1 remapped+peering, 5 active+remapped+backfill_toofull, 51 active+clean; 922 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s; 6714/44217 objects misplaced (15.184%) 2024-03-20T09:35:07.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:06 smithi122 ceph-mon[73495]: pgmap v2307: 57 pgs: 1 remapped+peering, 5 active+remapped+backfill_toofull, 51 active+clean; 922 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s; 6714/44217 objects misplaced (15.184%) 2024-03-20T09:35:07.193 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:07.221 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:35:07.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:07.225 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:35:07.326 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:35:07.359 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:07.385 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:35:07.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:07.389 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:35:07.490 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:35:07.503 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-20T09:35:07.520 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:07.550 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:35:07.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:07.554 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:35:07.655 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:35:07.666 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:35:07.690 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:07.714 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:35:07.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:07.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:35:07.821 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:35:07.850 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:07.877 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:35:07.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:07.881 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:35:07.981 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:35:08.100 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:08.698 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:35:08.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:08.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:35:08.718 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:35:08.785 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:35:08.793 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:35:08.803 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:35:08.887 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:08.960 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:35:08.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:08 smithi026 ceph-mon[68309]: osdmap e953: 12 total, 12 up, 11 in 2024-03-20T09:35:08.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:08 smithi026 ceph-mon[68309]: pgmap v2309: 57 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 52 active+clean; 774 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s; 5187/37143 objects misplaced (13.965%); 594 KiB/s, 9 objects/s recovering 2024-03-20T09:35:08.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:08.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:35:09.074 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:35:09.103 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:08 smithi115 ceph-mon[72476]: osdmap e953: 12 total, 12 up, 11 in 2024-03-20T09:35:09.103 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:08 smithi115 ceph-mon[72476]: pgmap v2309: 57 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 52 active+clean; 774 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s; 5187/37143 objects misplaced (13.965%); 594 KiB/s, 9 objects/s recovering 2024-03-20T09:35:09.105 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:09.132 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:35:09.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:09.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:35:09.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:08 smithi122 ceph-mon[73495]: osdmap e953: 12 total, 12 up, 11 in 2024-03-20T09:35:09.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:08 smithi122 ceph-mon[73495]: pgmap v2309: 57 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 52 active+clean; 774 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s; 5187/37143 objects misplaced (13.965%); 594 KiB/s, 9 objects/s recovering 2024-03-20T09:35:09.240 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:35:09.282 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:09.308 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:35:09.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:09.317 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:35:09.418 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:35:09.446 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:09.473 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:35:09.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:09.477 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:35:09.579 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:35:09.618 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:09.647 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:35:09.650 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:09.651 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:35:09.752 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:35:09.823 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:09.927 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:35:09.928 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:35:09.928 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:35:09.928 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:35:09.928 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:35:09.928 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:35:09.928 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:35:09.928 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:35:09.928 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:35:09.928 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:35:09.928 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:35:09.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:09 smithi122 ceph-mon[73495]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-20T09:35:09.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:09 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3969159512' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:35:09.939 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:35:09.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:09.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:35:09.960 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:35:09.984 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:35:09.984 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:35:09.984 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:35:09.984 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:35:09.984 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:35:09.984 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:35:09.984 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:35:09.984 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:35:09.984 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:35:09.984 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:35:09.984 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:35:09.985 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:35:10.046 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:35:10.078 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:10.104 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:35:10.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:10.110 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:35:10.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:09 smithi115 ceph-mon[72476]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-20T09:35:10.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:09 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3969159512' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:35:10.210 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:35:10.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:09 smithi026 ceph-mon[68309]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-20T09:35:10.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:09 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3969159512' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:35:10.283 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:10.329 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:35:10.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:10.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:35:10.440 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:35:10.484 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:10.507 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:35:10.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:10.512 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:35:10.613 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:35:10.643 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:10.666 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:35:10.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:10.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:35:10.774 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:35:10.807 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:10.827 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:10 smithi122 ceph-mon[73495]: pgmap v2310: 57 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 52 active+clean; 704 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 921 op/s; 3861/33771 objects misplaced (11.433%); 3.5 MiB/s, 55 objects/s recovering 2024-03-20T09:35:10.838 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:35:10.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:10.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:35:10.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:10 smithi115 ceph-mon[72476]: pgmap v2310: 57 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 52 active+clean; 704 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 921 op/s; 3861/33771 objects misplaced (11.433%); 3.5 MiB/s, 55 objects/s recovering 2024-03-20T09:35:10.942 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:35:10.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:10 smithi026 ceph-mon[68309]: pgmap v2310: 57 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 52 active+clean; 704 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 921 op/s; 3861/33771 objects misplaced (11.433%); 3.5 MiB/s, 55 objects/s recovering 2024-03-20T09:35:10.971 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:10.994 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:35:10.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:10.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:35:11.098 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:35:11.150 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:11.612 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:35:11.615 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:11.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:35:11.716 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:35:11.744 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:11.765 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:35:11.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:11.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:35:11.880 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:35:11.914 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:11 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:35:11.914 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:11 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:35:11.914 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:11 smithi026 ceph-mon[68309]: osdmap e954: 12 total, 12 up, 11 in 2024-03-20T09:35:11.920 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:12.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:11 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:35:12.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:11 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:35:12.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:11 smithi115 ceph-mon[72476]: osdmap e954: 12 total, 12 up, 11 in 2024-03-20T09:35:12.176 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:35:12.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:12.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:35:12.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:11 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:35:12.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:11 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:35:12.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:11 smithi122 ceph-mon[73495]: osdmap e954: 12 total, 12 up, 11 in 2024-03-20T09:35:12.280 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:35:12.308 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:12.456 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:35:12.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:12.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:35:12.560 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:35:12.589 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:12.609 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:35:12.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:12.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:35:12.713 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:35:12.740 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:12.759 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:35:12.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:12.762 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:35:12.863 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:35:12.893 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:12 smithi115 ceph-mon[72476]: osdmap e955: 12 total, 12 up, 11 in 2024-03-20T09:35:12.893 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:12 smithi115 ceph-mon[72476]: pgmap v2313: 57 pgs: 1 peering, 1 active+clean+remapped, 3 active+remapped+backfill_toofull, 52 active+clean; 558 MiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 969 op/s; 2694/26757 objects misplaced (10.068%); 6.4 MiB/s, 102 objects/s recovering 2024-03-20T09:35:12.897 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:12.935 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:35:12.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:12.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:35:13.040 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:35:13.073 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:12 smithi122 ceph-mon[73495]: osdmap e955: 12 total, 12 up, 11 in 2024-03-20T09:35:13.073 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:12 smithi122 ceph-mon[73495]: pgmap v2313: 57 pgs: 1 peering, 1 active+clean+remapped, 3 active+remapped+backfill_toofull, 52 active+clean; 558 MiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 969 op/s; 2694/26757 objects misplaced (10.068%); 6.4 MiB/s, 102 objects/s recovering 2024-03-20T09:35:13.076 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:13.099 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:35:13.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:13.103 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:35:13.204 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:35:13.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:12 smithi026 ceph-mon[68309]: osdmap e955: 12 total, 12 up, 11 in 2024-03-20T09:35:13.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:12 smithi026 ceph-mon[68309]: pgmap v2313: 57 pgs: 1 peering, 1 active+clean+remapped, 3 active+remapped+backfill_toofull, 52 active+clean; 558 MiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 969 op/s; 2694/26757 objects misplaced (10.068%); 6.4 MiB/s, 102 objects/s recovering 2024-03-20T09:35:13.253 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:13.715 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:35:13.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:13.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:35:13.828 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:35:13.876 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:13.909 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:35:13.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:13.915 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:35:13.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:13 smithi026 ceph-mon[68309]: osdmap e956: 12 total, 12 up, 11 in 2024-03-20T09:35:14.016 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:35:14.052 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:14.076 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:35:14.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:14.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:35:14.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:13 smithi115 ceph-mon[72476]: osdmap e956: 12 total, 12 up, 11 in 2024-03-20T09:35:14.180 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:35:14.186 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:35:14.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:13 smithi122 ceph-mon[73495]: osdmap e956: 12 total, 12 up, 11 in 2024-03-20T09:35:14.219 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:14.245 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:35:14.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:14.249 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:35:14.350 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:35:14.381 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:14.411 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:35:14.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:14.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:35:14.515 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:35:14.545 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:14.576 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:35:14.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:14.580 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:35:14.681 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:35:14.708 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:14.734 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:35:14.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:14.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:35:14.838 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:35:14.876 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:14.900 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:35:14.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:14.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:35:14.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:14 smithi026 ceph-mon[68309]: pgmap v2315: 57 pgs: 1 peering, 3 active+remapped+backfill_toofull, 53 active+clean; 522 MiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 672 op/s; 2694/25035 objects misplaced (10.761%); 5.9 MiB/s, 94 objects/s recovering 2024-03-20T09:35:15.004 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:35:15.049 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:15.071 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:35:15.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:15.077 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:35:15.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:14 smithi115 ceph-mon[72476]: pgmap v2315: 57 pgs: 1 peering, 3 active+remapped+backfill_toofull, 53 active+clean; 522 MiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 672 op/s; 2694/25035 objects misplaced (10.761%); 5.9 MiB/s, 94 objects/s recovering 2024-03-20T09:35:15.177 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:35:15.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:14 smithi122 ceph-mon[73495]: pgmap v2315: 57 pgs: 1 peering, 3 active+remapped+backfill_toofull, 53 active+clean; 522 MiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 672 op/s; 2694/25035 objects misplaced (10.761%); 5.9 MiB/s, 94 objects/s recovering 2024-03-20T09:35:15.211 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:15.240 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:35:15.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:15.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:35:15.345 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:35:15.416 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:15.433 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:15 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mon-a[68286]: 2024-03-20T09:35:15.170+0000 7f683364b700 -1 mon.a@0(leader).osd e956 definitely_dead 0 2024-03-20T09:35:15.533 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:35:15.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:15.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:35:15.644 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:35:15.673 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:15.696 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:35:15.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:15.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:35:15.748 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:35:15.748 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:35:15.800 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:35:15.832 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:15.870 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:35:15.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:15.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:35:15.901 INFO:teuthology.orchestra.run.smithi026.stderr:marked in osd.10. 2024-03-20T09:35:15.977 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:35:16.002 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:15 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3332616146' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["10"]}]: dispatch 2024-03-20T09:35:16.002 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:15 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["10"]}]: dispatch 2024-03-20T09:35:16.003 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:16.024 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:35:16.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:16.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:35:16.128 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:35:16.152 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:15 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3332616146' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["10"]}]: dispatch 2024-03-20T09:35:16.152 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:15 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["10"]}]: dispatch 2024-03-20T09:35:16.153 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:16.172 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:35:16.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:16.175 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:35:16.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:15 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3332616146' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["10"]}]: dispatch 2024-03-20T09:35:16.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:15 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["10"]}]: dispatch 2024-03-20T09:35:16.277 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:35:16.304 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:16.322 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:35:16.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:16.325 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:35:16.427 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:35:16.458 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:16.781 INFO:tasks.thrashosds.thrasher:Added osd 10 2024-03-20T09:35:16.873 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:35:17.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:16 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-03-20T09:35:17.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:16 smithi115 ceph-mon[72476]: osdmap e957: 12 total, 12 up, 12 in 2024-03-20T09:35:17.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:16 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/634637333' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:35:17.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:16 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:35:17.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:16 smithi115 ceph-mon[72476]: pgmap v2317: 57 pgs: 2 peering, 3 active+remapped+backfill_toofull, 52 active+clean; 412 MiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 742 op/s; 2292/19767 objects misplaced (11.595%); 2.7 MiB/s, 43 objects/s recovering 2024-03-20T09:35:17.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:16 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:35:17.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:16 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:35:17.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:16 smithi115 ceph-mon[72476]: osdmap e958: 12 total, 12 up, 12 in 2024-03-20T09:35:17.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:16 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-03-20T09:35:17.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:16 smithi122 ceph-mon[73495]: osdmap e957: 12 total, 12 up, 12 in 2024-03-20T09:35:17.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:16 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/634637333' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:35:17.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:16 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:35:17.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:16 smithi122 ceph-mon[73495]: pgmap v2317: 57 pgs: 2 peering, 3 active+remapped+backfill_toofull, 52 active+clean; 412 MiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 742 op/s; 2292/19767 objects misplaced (11.595%); 2.7 MiB/s, 43 objects/s recovering 2024-03-20T09:35:17.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:16 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:35:17.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:16 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:35:17.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:16 smithi122 ceph-mon[73495]: osdmap e958: 12 total, 12 up, 12 in 2024-03-20T09:35:17.188 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:16 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-03-20T09:35:17.188 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:16 smithi026 ceph-mon[68309]: osdmap e957: 12 total, 12 up, 12 in 2024-03-20T09:35:17.188 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:16 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/634637333' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:35:17.188 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:16 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:35:17.189 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:16 smithi026 ceph-mon[68309]: pgmap v2317: 57 pgs: 2 peering, 3 active+remapped+backfill_toofull, 52 active+clean; 412 MiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 742 op/s; 2292/19767 objects misplaced (11.595%); 2.7 MiB/s, 43 objects/s recovering 2024-03-20T09:35:17.189 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:16 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:35:17.189 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:16 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:35:17.189 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:16 smithi026 ceph-mon[68309]: osdmap e958: 12 total, 12 up, 12 in 2024-03-20T09:35:17.208 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:35:17.212 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-20T09:35:17.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:17.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:35:17.314 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:35:17.342 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:17.463 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:35:17.465 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:35:17.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:17.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:35:17.569 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:35:17.602 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:17.623 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:35:17.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:17.626 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:35:17.727 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:35:17.755 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:17.774 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:35:17.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:17.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:35:17.878 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:35:17.893 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:17 smithi122 ceph-mon[73495]: 1.3 scrub starts 2024-03-20T09:35:17.893 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:17 smithi122 ceph-mon[73495]: 1.3 scrub ok 2024-03-20T09:35:17.893 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:17 smithi122 ceph-mon[73495]: 7.11 scrub starts 2024-03-20T09:35:17.893 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:17 smithi122 ceph-mon[73495]: 7.11 scrub ok 2024-03-20T09:35:17.894 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:17 smithi122 ceph-mon[73495]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-20T09:35:17.894 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:17 smithi122 ceph-mon[73495]: 7.10 starting backfill to osd.10 from (0'0,0'0] MAX to 956'14257 2024-03-20T09:35:17.894 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:17 smithi122 ceph-mon[73495]: osdmap e959: 12 total, 12 up, 12 in 2024-03-20T09:35:17.909 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:18.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:17 smithi115 ceph-mon[72476]: 1.3 scrub starts 2024-03-20T09:35:18.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:17 smithi115 ceph-mon[72476]: 1.3 scrub ok 2024-03-20T09:35:18.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:17 smithi115 ceph-mon[72476]: 7.11 scrub starts 2024-03-20T09:35:18.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:17 smithi115 ceph-mon[72476]: 7.11 scrub ok 2024-03-20T09:35:18.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:17 smithi115 ceph-mon[72476]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-20T09:35:18.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:17 smithi115 ceph-mon[72476]: 7.10 starting backfill to osd.10 from (0'0,0'0] MAX to 956'14257 2024-03-20T09:35:18.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:17 smithi115 ceph-mon[72476]: osdmap e959: 12 total, 12 up, 12 in 2024-03-20T09:35:18.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:17 smithi026 ceph-mon[68309]: 1.3 scrub starts 2024-03-20T09:35:18.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:17 smithi026 ceph-mon[68309]: 1.3 scrub ok 2024-03-20T09:35:18.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:17 smithi026 ceph-mon[68309]: 7.11 scrub starts 2024-03-20T09:35:18.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:17 smithi026 ceph-mon[68309]: 7.11 scrub ok 2024-03-20T09:35:18.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:17 smithi026 ceph-mon[68309]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-20T09:35:18.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:17 smithi026 ceph-mon[68309]: 7.10 starting backfill to osd.10 from (0'0,0'0] MAX to 956'14257 2024-03-20T09:35:18.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:17 smithi026 ceph-mon[68309]: osdmap e959: 12 total, 12 up, 12 in 2024-03-20T09:35:18.820 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:35:18.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:18.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:35:18.869 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:35:18.926 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:35:18.966 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:19.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:18 smithi115 ceph-mon[72476]: 1.0 scrub starts 2024-03-20T09:35:19.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:18 smithi115 ceph-mon[72476]: 1.0 scrub ok 2024-03-20T09:35:19.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:18 smithi115 ceph-mon[72476]: 7.1 starting backfill to osd.10 from (0'0,0'0] MAX to 956'14494 2024-03-20T09:35:19.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:18 smithi115 ceph-mon[72476]: 7.e starting backfill to osd.10 from (0'0,0'0] MAX to 956'15545 2024-03-20T09:35:19.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:18 smithi115 ceph-mon[72476]: 7.14 starting backfill to osd.10 from (0'0,0'0] MAX to 956'14579 2024-03-20T09:35:19.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:18 smithi115 ceph-mon[72476]: 7.2 starting backfill to osd.10 from (0'0,0'0] MAX to 956'14495 2024-03-20T09:35:19.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:18 smithi115 ceph-mon[72476]: 1.f scrub starts 2024-03-20T09:35:19.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:18 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:35:19.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:18 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:35:19.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:18 smithi115 ceph-mon[72476]: 1.f scrub ok 2024-03-20T09:35:19.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:18 smithi115 ceph-mon[72476]: 7.a scrub starts 2024-03-20T09:35:19.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:18 smithi115 ceph-mon[72476]: 7.a scrub ok 2024-03-20T09:35:19.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:18 smithi115 ceph-mon[72476]: pgmap v2320: 57 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+remapped, 1 remapped+peering, 2 peering, 2 active+remapped+backfill_toofull, 50 active+clean; 305 MiB data, 8.9 GiB used, 974 GiB / 983 GiB avail; 1121/14643 objects misplaced (7.656%) 2024-03-20T09:35:19.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:18 smithi115 ceph-mon[72476]: osdmap e960: 12 total, 12 up, 12 in 2024-03-20T09:35:19.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:18 smithi122 ceph-mon[73495]: 1.0 scrub starts 2024-03-20T09:35:19.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:18 smithi122 ceph-mon[73495]: 1.0 scrub ok 2024-03-20T09:35:19.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:18 smithi122 ceph-mon[73495]: 7.1 starting backfill to osd.10 from (0'0,0'0] MAX to 956'14494 2024-03-20T09:35:19.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:18 smithi122 ceph-mon[73495]: 7.e starting backfill to osd.10 from (0'0,0'0] MAX to 956'15545 2024-03-20T09:35:19.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:18 smithi122 ceph-mon[73495]: 7.14 starting backfill to osd.10 from (0'0,0'0] MAX to 956'14579 2024-03-20T09:35:19.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:18 smithi122 ceph-mon[73495]: 7.2 starting backfill to osd.10 from (0'0,0'0] MAX to 956'14495 2024-03-20T09:35:19.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:18 smithi122 ceph-mon[73495]: 1.f scrub starts 2024-03-20T09:35:19.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:18 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:35:19.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:18 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:35:19.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:18 smithi122 ceph-mon[73495]: 1.f scrub ok 2024-03-20T09:35:19.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:18 smithi122 ceph-mon[73495]: 7.a scrub starts 2024-03-20T09:35:19.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:18 smithi122 ceph-mon[73495]: 7.a scrub ok 2024-03-20T09:35:19.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:18 smithi122 ceph-mon[73495]: pgmap v2320: 57 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+remapped, 1 remapped+peering, 2 peering, 2 active+remapped+backfill_toofull, 50 active+clean; 305 MiB data, 8.9 GiB used, 974 GiB / 983 GiB avail; 1121/14643 objects misplaced (7.656%) 2024-03-20T09:35:19.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:18 smithi122 ceph-mon[73495]: osdmap e960: 12 total, 12 up, 12 in 2024-03-20T09:35:19.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:18 smithi026 ceph-mon[68309]: 1.0 scrub starts 2024-03-20T09:35:19.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:18 smithi026 ceph-mon[68309]: 1.0 scrub ok 2024-03-20T09:35:19.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:18 smithi026 ceph-mon[68309]: 7.1 starting backfill to osd.10 from (0'0,0'0] MAX to 956'14494 2024-03-20T09:35:19.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:18 smithi026 ceph-mon[68309]: 7.e starting backfill to osd.10 from (0'0,0'0] MAX to 956'15545 2024-03-20T09:35:19.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:18 smithi026 ceph-mon[68309]: 7.14 starting backfill to osd.10 from (0'0,0'0] MAX to 956'14579 2024-03-20T09:35:19.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:18 smithi026 ceph-mon[68309]: 7.2 starting backfill to osd.10 from (0'0,0'0] MAX to 956'14495 2024-03-20T09:35:19.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:18 smithi026 ceph-mon[68309]: 1.f scrub starts 2024-03-20T09:35:19.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:18 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:35:19.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:18 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:35:19.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:18 smithi026 ceph-mon[68309]: 1.f scrub ok 2024-03-20T09:35:19.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:18 smithi026 ceph-mon[68309]: 7.a scrub starts 2024-03-20T09:35:19.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:18 smithi026 ceph-mon[68309]: 7.a scrub ok 2024-03-20T09:35:19.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:18 smithi026 ceph-mon[68309]: pgmap v2320: 57 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+remapped, 1 remapped+peering, 2 peering, 2 active+remapped+backfill_toofull, 50 active+clean; 305 MiB data, 8.9 GiB used, 974 GiB / 983 GiB avail; 1121/14643 objects misplaced (7.656%) 2024-03-20T09:35:19.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:18 smithi026 ceph-mon[68309]: osdmap e960: 12 total, 12 up, 12 in 2024-03-20T09:35:20.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:19 smithi122 ceph-mon[73495]: 7.13 scrub starts 2024-03-20T09:35:20.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:19 smithi122 ceph-mon[73495]: 7.13 scrub ok 2024-03-20T09:35:20.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:19 smithi122 ceph-mon[73495]: osdmap e961: 12 total, 12 up, 12 in 2024-03-20T09:35:20.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:19 smithi026 ceph-mon[68309]: 7.13 scrub starts 2024-03-20T09:35:20.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:19 smithi026 ceph-mon[68309]: 7.13 scrub ok 2024-03-20T09:35:20.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:19 smithi026 ceph-mon[68309]: osdmap e961: 12 total, 12 up, 12 in 2024-03-20T09:35:20.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:19 smithi115 ceph-mon[72476]: 7.13 scrub starts 2024-03-20T09:35:20.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:19 smithi115 ceph-mon[72476]: 7.13 scrub ok 2024-03-20T09:35:20.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:19 smithi115 ceph-mon[72476]: osdmap e961: 12 total, 12 up, 12 in 2024-03-20T09:35:20.408 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:35:21.002 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:35:21.002 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:35:21.002 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:35:21.002 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:35:21.002 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:35:21.002 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:35:21.002 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:35:21.002 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:35:21.002 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:35:21.002 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:35:21.003 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:35:21.003 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:35:21.003 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:35:21.003 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:35:21.003 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:35:21.003 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:35:21.003 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:35:21.003 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:35:21.003 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:35:21.003 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:35:21.003 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:35:21.004 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:35:21.004 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:35:21.004 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:35:21.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:20 smithi122 ceph-mon[73495]: 7.15 scrub starts 2024-03-20T09:35:21.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:20 smithi122 ceph-mon[73495]: 7.15 scrub ok 2024-03-20T09:35:21.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:20 smithi122 ceph-mon[73495]: 1.1e scrub starts 2024-03-20T09:35:21.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:20 smithi122 ceph-mon[73495]: 1.1e scrub ok 2024-03-20T09:35:21.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:20 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/161331191' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:35:21.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:20 smithi122 ceph-mon[73495]: pgmap v2323: 57 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+remapped, 1 remapped+peering, 2 peering, 3 active+remapped+backfill_toofull, 49 active+clean; 281 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1099/13452 objects misplaced (8.170%); 1.8 MiB/s, 28 objects/s recovering 2024-03-20T09:35:21.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:20 smithi122 ceph-mon[73495]: osdmap e962: 12 total, 12 up, 12 in 2024-03-20T09:35:21.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:20 smithi026 ceph-mon[68309]: 7.15 scrub starts 2024-03-20T09:35:21.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:20 smithi026 ceph-mon[68309]: 7.15 scrub ok 2024-03-20T09:35:21.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:20 smithi026 ceph-mon[68309]: 1.1e scrub starts 2024-03-20T09:35:21.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:20 smithi026 ceph-mon[68309]: 1.1e scrub ok 2024-03-20T09:35:21.215 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:20 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/161331191' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:35:21.215 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:20 smithi026 ceph-mon[68309]: pgmap v2323: 57 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+remapped, 1 remapped+peering, 2 peering, 3 active+remapped+backfill_toofull, 49 active+clean; 281 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1099/13452 objects misplaced (8.170%); 1.8 MiB/s, 28 objects/s recovering 2024-03-20T09:35:21.215 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:20 smithi026 ceph-mon[68309]: osdmap e962: 12 total, 12 up, 12 in 2024-03-20T09:35:21.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:20 smithi115 ceph-mon[72476]: 7.15 scrub starts 2024-03-20T09:35:21.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:20 smithi115 ceph-mon[72476]: 7.15 scrub ok 2024-03-20T09:35:21.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:20 smithi115 ceph-mon[72476]: 1.1e scrub starts 2024-03-20T09:35:21.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:20 smithi115 ceph-mon[72476]: 1.1e scrub ok 2024-03-20T09:35:21.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:20 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/161331191' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:35:21.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:20 smithi115 ceph-mon[72476]: pgmap v2323: 57 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+remapped, 1 remapped+peering, 2 peering, 3 active+remapped+backfill_toofull, 49 active+clean; 281 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1099/13452 objects misplaced (8.170%); 1.8 MiB/s, 28 objects/s recovering 2024-03-20T09:35:21.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:20 smithi115 ceph-mon[72476]: osdmap e962: 12 total, 12 up, 12 in 2024-03-20T09:35:21.781 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 8, 11, 3, 4, 9, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 11, 5, 10] 2024-03-20T09:35:21.782 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:35:21.782 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-20T09:35:21.782 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.928009 2024-03-20T09:35:21.782 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd primary-affinity 5 0.9280087288595185 2024-03-20T09:35:21.909 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:35:22.177 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:21 smithi026 ceph-mon[68309]: 1.17 scrub starts 2024-03-20T09:35:22.177 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:21 smithi026 ceph-mon[68309]: 1.17 scrub ok 2024-03-20T09:35:22.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:21 smithi122 ceph-mon[73495]: 1.17 scrub starts 2024-03-20T09:35:22.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:21 smithi122 ceph-mon[73495]: 1.17 scrub ok 2024-03-20T09:35:22.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:21 smithi115 ceph-mon[72476]: 1.17 scrub starts 2024-03-20T09:35:22.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:21 smithi115 ceph-mon[72476]: 1.17 scrub ok 2024-03-20T09:35:22.615 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:35:22.615 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-03-20T09:35:22.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-03-20T09:35:22.615 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:35:22.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.615 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26216.0:120094 6.6 6:64bfe71c:::benchmark_data_smithi122_168655_object120093:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e746)", 2024-03-20T09:35:22.615 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:25:45.845981+0000", 2024-03-20T09:35:22.615 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 576.76670398299996, 2024-03-20T09:35:22.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5547022909999999, 2024-03-20T09:35:22.616 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:35:22.616 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:22.616 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:35:22.616 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26216", 2024-03-20T09:35:22.616 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:35:22.616 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 120094 2024-03-20T09:35:22.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.616 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:35:22.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:35:22.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:45.845981+0000", 2024-03-20T09:35:22.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:35:22.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:45.845981+0000", 2024-03-20T09:35:22.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:35:22.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:45.845982+0000", 2024-03-20T09:35:22.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.879999999999999e-07 2024-03-20T09:35:22.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:35:22.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:45.846070+0000", 2024-03-20T09:35:22.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7961999999999996e-05 2024-03-20T09:35:22.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:35:22.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:45.846082+0000", 2024-03-20T09:35:22.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1717e-05 2024-03-20T09:35:22.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:35:22.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:45.846091+0000", 2024-03-20T09:35:22.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2860000000000005e-06 2024-03-20T09:35:22.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:45.846150+0000", 2024-03-20T09:35:22.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9305000000000002e-05 2024-03-20T09:35:22.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:35:22.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:45.846291+0000", 2024-03-20T09:35:22.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014065799999999999 2024-03-20T09:35:22.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4,7", 2024-03-20T09:35:22.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:45.846448+0000", 2024-03-20T09:35:22.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015677399999999999 2024-03-20T09:35:22.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:35:22.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:45.849968+0000", 2024-03-20T09:35:22.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035201199999999998 2024-03-20T09:35:22.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:45.856942+0000", 2024-03-20T09:35:22.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0069744189999999999 2024-03-20T09:35:22.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:47.400589+0000", 2024-03-20T09:35:22.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5436472110000001 2024-03-20T09:35:22.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:35:22.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:47.400641+0000", 2024-03-20T09:35:22.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2009000000000003e-05 2024-03-20T09:35:22.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:35:22.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:47.400683+0000", 2024-03-20T09:35:22.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1841999999999997e-05 2024-03-20T09:35:22.621 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.621 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:35:22.621 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.622 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26216.0:129158 6.6 6:66af82a9:::benchmark_data_smithi122_168655_object9049:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e749)", 2024-03-20T09:35:22.622 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:25:56.648871+0000", 2024-03-20T09:35:22.622 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 565.963813942, 2024-03-20T09:35:22.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.79442431199999997, 2024-03-20T09:35:22.622 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:35:22.622 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:22.622 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:35:22.622 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26216", 2024-03-20T09:35:22.622 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:35:22.622 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 129158 2024-03-20T09:35:22.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.622 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:35:22.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:35:22.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:56.648871+0000", 2024-03-20T09:35:22.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:35:22.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:56.648871+0000", 2024-03-20T09:35:22.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:35:22.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:56.648871+0000", 2024-03-20T09:35:22.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3799999999999998e-07 2024-03-20T09:35:22.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:35:22.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:56.648874+0000", 2024-03-20T09:35:22.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.526e-06 2024-03-20T09:35:22.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:35:22.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:56.648879+0000", 2024-03-20T09:35:22.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7509999999999999e-06 2024-03-20T09:35:22.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:35:22.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:56.648883+0000", 2024-03-20T09:35:22.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4599999999999996e-06 2024-03-20T09:35:22.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:56.648922+0000", 2024-03-20T09:35:22.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9005999999999999e-05 2024-03-20T09:35:22.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:35:22.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:56.648930+0000", 2024-03-20T09:35:22.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6890000000000004e-06 2024-03-20T09:35:22.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:57.413434+0000", 2024-03-20T09:35:22.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.76450424800000005 2024-03-20T09:35:22.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:35:22.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:57.413551+0000", 2024-03-20T09:35:22.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011723 2024-03-20T09:35:22.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4,7", 2024-03-20T09:35:22.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:57.421734+0000", 2024-03-20T09:35:22.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0081829480000000007 2024-03-20T09:35:22.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:35:22.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:57.438845+0000", 2024-03-20T09:35:22.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017111033000000001 2024-03-20T09:35:22.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:57.442046+0000", 2024-03-20T09:35:22.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0032006449999999998 2024-03-20T09:35:22.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:57.443190+0000", 2024-03-20T09:35:22.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001143724 2024-03-20T09:35:22.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:35:22.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:57.443253+0000", 2024-03-20T09:35:22.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3384000000000002e-05 2024-03-20T09:35:22.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:35:22.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:25:57.443295+0000", 2024-03-20T09:35:22.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2330000000000003e-05 2024-03-20T09:35:22.628 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.629 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:35:22.629 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.629 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26216.0:136781 6.13 6:c933b1cc:::benchmark_data_smithi122_168655_object16672:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e769)", 2024-03-20T09:35:22.629 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:26:41.503673+0000", 2024-03-20T09:35:22.629 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 521.10901224600002, 2024-03-20T09:35:22.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.87030609000000003, 2024-03-20T09:35:22.629 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:35:22.629 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:22.629 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:35:22.630 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26216", 2024-03-20T09:35:22.630 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:35:22.630 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 136781 2024-03-20T09:35:22.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.630 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:35:22.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:35:22.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:41.503673+0000", 2024-03-20T09:35:22.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:35:22.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:41.503673+0000", 2024-03-20T09:35:22.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:35:22.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:41.503674+0000", 2024-03-20T09:35:22.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1999999999999999e-06 2024-03-20T09:35:22.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.631 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:35:22.631 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:41.503679+0000", 2024-03-20T09:35:22.631 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4160000000000003e-06 2024-03-20T09:35:22.631 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.631 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:35:22.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:41.503691+0000", 2024-03-20T09:35:22.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2085e-05 2024-03-20T09:35:22.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:35:22.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:41.503699+0000", 2024-03-20T09:35:22.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5549999999999997e-06 2024-03-20T09:35:22.632 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.632 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.632 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.632 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:41.503771+0000", 2024-03-20T09:35:22.632 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2434000000000005e-05 2024-03-20T09:35:22.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:35:22.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:41.503785+0000", 2024-03-20T09:35:22.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3719e-05 2024-03-20T09:35:22.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.633 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:42.362613+0000", 2024-03-20T09:35:22.633 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.85882753099999998 2024-03-20T09:35:22.633 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.633 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.633 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:35:22.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:42.362744+0000", 2024-03-20T09:35:22.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000131302 2024-03-20T09:35:22.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 6,11", 2024-03-20T09:35:22.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:42.362871+0000", 2024-03-20T09:35:22.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012693700000000001 2024-03-20T09:35:22.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.634 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.634 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:35:22.634 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:42.373671+0000", 2024-03-20T09:35:22.634 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010800561 2024-03-20T09:35:22.634 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:42.373818+0000", 2024-03-20T09:35:22.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000146272 2024-03-20T09:35:22.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:42.373871+0000", 2024-03-20T09:35:22.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3653999999999998e-05 2024-03-20T09:35:22.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.635 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:35:22.635 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:42.373936+0000", 2024-03-20T09:35:22.635 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4234999999999997e-05 2024-03-20T09:35:22.635 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.635 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.636 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:35:22.636 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:42.373979+0000", 2024-03-20T09:35:22.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3189000000000003e-05 2024-03-20T09:35:22.636 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.636 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:35:22.636 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.636 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.636 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.636 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26216.0:136783 6.13 6:cbde5e76:::benchmark_data_smithi122_168655_object16674:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e769)", 2024-03-20T09:35:22.636 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:26:41.505168+0000", 2024-03-20T09:35:22.636 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 521.10751731100004, 2024-03-20T09:35:22.636 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.87320019800000004, 2024-03-20T09:35:22.636 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:35:22.636 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:22.636 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:35:22.637 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26216", 2024-03-20T09:35:22.637 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:35:22.637 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 136783 2024-03-20T09:35:22.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.637 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:35:22.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:35:22.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:41.505168+0000", 2024-03-20T09:35:22.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.637 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.637 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:35:22.637 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:41.505168+0000", 2024-03-20T09:35:22.637 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.637 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:35:22.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:41.505169+0000", 2024-03-20T09:35:22.638 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5230000000000001e-06 2024-03-20T09:35:22.638 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.638 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.638 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:35:22.638 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:41.505176+0000", 2024-03-20T09:35:22.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8299999999999998e-06 2024-03-20T09:35:22.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:35:22.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:41.505192+0000", 2024-03-20T09:35:22.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5761999999999999e-05 2024-03-20T09:35:22.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.639 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.639 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:35:22.639 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:41.505200+0000", 2024-03-20T09:35:22.639 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7560000000000005e-06 2024-03-20T09:35:22.639 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.640 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:41.505273+0000", 2024-03-20T09:35:22.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.258e-05 2024-03-20T09:35:22.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:35:22.640 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:41.505284+0000", 2024-03-20T09:35:22.640 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1318000000000001e-05 2024-03-20T09:35:22.640 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.640 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.640 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.640 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:42.365177+0000", 2024-03-20T09:35:22.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.85989257500000005 2024-03-20T09:35:22.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:35:22.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:42.365309+0000", 2024-03-20T09:35:22.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013191 2024-03-20T09:35:22.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 6,11", 2024-03-20T09:35:22.641 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:42.365422+0000", 2024-03-20T09:35:22.641 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000112946 2024-03-20T09:35:22.641 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.641 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.641 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:35:22.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:42.378205+0000", 2024-03-20T09:35:22.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012783470999999999 2024-03-20T09:35:22.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:42.378243+0000", 2024-03-20T09:35:22.642 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7775999999999999e-05 2024-03-20T09:35:22.642 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.642 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.642 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.642 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:42.378272+0000", 2024-03-20T09:35:22.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8796999999999999e-05 2024-03-20T09:35:22.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:35:22.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:42.378321+0000", 2024-03-20T09:35:22.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9054999999999997e-05 2024-03-20T09:35:22.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.643 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.643 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:35:22.643 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:26:42.378368+0000", 2024-03-20T09:35:22.643 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6898999999999999e-05 2024-03-20T09:35:22.643 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.643 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:35:22.643 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.643 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.644 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.644 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26216.0:141491 6.11 6:8b982ecc:::benchmark_data_smithi122_168655_object21382:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e781)", 2024-03-20T09:35:22.644 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:27:00.410198+0000", 2024-03-20T09:35:22.644 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 502.20248675200003, 2024-03-20T09:35:22.644 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.87562146299999999, 2024-03-20T09:35:22.644 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:35:22.644 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:22.644 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:35:22.644 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26216", 2024-03-20T09:35:22.644 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:35:22.644 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 141491 2024-03-20T09:35:22.644 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.644 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:35:22.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:35:22.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:00.410198+0000", 2024-03-20T09:35:22.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:35:22.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:00.410198+0000", 2024-03-20T09:35:22.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.645 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.645 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.645 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:35:22.645 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:00.410199+0000", 2024-03-20T09:35:22.645 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.249e-06 2024-03-20T09:35:22.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:35:22.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:00.410205+0000", 2024-03-20T09:35:22.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2220000000000001e-06 2024-03-20T09:35:22.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:35:22.646 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:00.410219+0000", 2024-03-20T09:35:22.646 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4103e-05 2024-03-20T09:35:22.646 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.646 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.646 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:35:22.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:00.410227+0000", 2024-03-20T09:35:22.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3040000000000005e-06 2024-03-20T09:35:22.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:00.410283+0000", 2024-03-20T09:35:22.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6368e-05 2024-03-20T09:35:22.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.647 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.647 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:35:22.647 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:00.410291+0000", 2024-03-20T09:35:22.647 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0450000000000001e-06 2024-03-20T09:35:22.647 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:01.270090+0000", 2024-03-20T09:35:22.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.85979871100000005 2024-03-20T09:35:22.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:35:22.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:01.270241+0000", 2024-03-20T09:35:22.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000150679 2024-03-20T09:35:22.648 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.648 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.648 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:35:22.648 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:01.270382+0000", 2024-03-20T09:35:22.648 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000141557 2024-03-20T09:35:22.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:35:22.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:01.281507+0000", 2024-03-20T09:35:22.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011124271 2024-03-20T09:35:22.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:01.281663+0000", 2024-03-20T09:35:22.649 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015648300000000001 2024-03-20T09:35:22.649 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.649 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.649 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.649 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:01.285710+0000", 2024-03-20T09:35:22.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040467469999999998 2024-03-20T09:35:22.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:35:22.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:01.285763+0000", 2024-03-20T09:35:22.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2590000000000003e-05 2024-03-20T09:35:22.650 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.650 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.650 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:35:22.650 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:01.285820+0000", 2024-03-20T09:35:22.650 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7133999999999999e-05 2024-03-20T09:35:22.650 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.650 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:35:22.651 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.651 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.651 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26216.0:141518 6.11 6:8e9dcbc4:::benchmark_data_smithi122_168655_object21409:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e781)", 2024-03-20T09:35:22.651 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:27:00.543878+0000", 2024-03-20T09:35:22.651 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 502.06880665800003, 2024-03-20T09:35:22.651 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.74208995200000005, 2024-03-20T09:35:22.651 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:35:22.651 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:22.651 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:35:22.651 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26216", 2024-03-20T09:35:22.651 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:35:22.651 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 141518 2024-03-20T09:35:22.651 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.652 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:35:22.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:35:22.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:00.543878+0000", 2024-03-20T09:35:22.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:35:22.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:00.543878+0000", 2024-03-20T09:35:22.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.652 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.652 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.652 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:35:22.652 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:00.543880+0000", 2024-03-20T09:35:22.652 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1009999999999999e-06 2024-03-20T09:35:22.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:35:22.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:00.543890+0000", 2024-03-20T09:35:22.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4040000000000001e-06 2024-03-20T09:35:22.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:35:22.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:00.543911+0000", 2024-03-20T09:35:22.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0903999999999998e-05 2024-03-20T09:35:22.653 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.653 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.653 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:35:22.653 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:00.543921+0000", 2024-03-20T09:35:22.653 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0373999999999999e-05 2024-03-20T09:35:22.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:00.543995+0000", 2024-03-20T09:35:22.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3477000000000001e-05 2024-03-20T09:35:22.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:35:22.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:00.544005+0000", 2024-03-20T09:35:22.654 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0643e-05 2024-03-20T09:35:22.654 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.654 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.654 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.654 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:01.272967+0000", 2024-03-20T09:35:22.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.72896214699999995 2024-03-20T09:35:22.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:35:22.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:01.273087+0000", 2024-03-20T09:35:22.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000119496 2024-03-20T09:35:22.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:35:22.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:01.273202+0000", 2024-03-20T09:35:22.655 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000115326 2024-03-20T09:35:22.655 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.655 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.655 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:35:22.655 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:01.281675+0000", 2024-03-20T09:35:22.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0084731049999999999 2024-03-20T09:35:22.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:01.281730+0000", 2024-03-20T09:35:22.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4311000000000002e-05 2024-03-20T09:35:22.656 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.656 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.656 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.656 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:01.285877+0000", 2024-03-20T09:35:22.656 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004147085 2024-03-20T09:35:22.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:35:22.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:01.285923+0000", 2024-03-20T09:35:22.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.655e-05 2024-03-20T09:35:22.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.657 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:35:22.657 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:01.285968+0000", 2024-03-20T09:35:22.657 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5028999999999999e-05 2024-03-20T09:35:22.657 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.657 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:35:22.657 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.657 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.657 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.658 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26216.0:141521 6.11 6:894238c1:::benchmark_data_smithi122_168655_object21412:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e781)", 2024-03-20T09:35:22.658 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:27:00.550308+0000", 2024-03-20T09:35:22.658 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 502.06237714299999, 2024-03-20T09:35:22.658 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.73581360699999998, 2024-03-20T09:35:22.658 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:35:22.658 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:22.658 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:35:22.658 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26216", 2024-03-20T09:35:22.658 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:35:22.658 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 141521 2024-03-20T09:35:22.658 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.658 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:35:22.658 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.658 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:35:22.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:00.550308+0000", 2024-03-20T09:35:22.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:35:22.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:00.550308+0000", 2024-03-20T09:35:22.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.659 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:35:22.659 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:00.550308+0000", 2024-03-20T09:35:22.659 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1699999999999998e-07 2024-03-20T09:35:22.659 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.659 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:35:22.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:00.550311+0000", 2024-03-20T09:35:22.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.112e-06 2024-03-20T09:35:22.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:35:22.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:00.550319+0000", 2024-03-20T09:35:22.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7989999999999994e-06 2024-03-20T09:35:22.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.660 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:35:22.660 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:00.550326+0000", 2024-03-20T09:35:22.660 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6239999999999996e-06 2024-03-20T09:35:22.660 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.660 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:00.550387+0000", 2024-03-20T09:35:22.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1465999999999997e-05 2024-03-20T09:35:22.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:35:22.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:00.550399+0000", 2024-03-20T09:35:22.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1406999999999999e-05 2024-03-20T09:35:22.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.661 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.661 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:01.275263+0000", 2024-03-20T09:35:22.661 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.72486446000000004 2024-03-20T09:35:22.661 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.661 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:35:22.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:01.275367+0000", 2024-03-20T09:35:22.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010376499999999999 2024-03-20T09:35:22.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:35:22.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:01.275455+0000", 2024-03-20T09:35:22.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7997000000000005e-05 2024-03-20T09:35:22.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.662 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.662 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:35:22.662 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:01.281681+0000", 2024-03-20T09:35:22.662 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0062264379999999999 2024-03-20T09:35:22.662 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.663 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.663 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:01.281777+0000", 2024-03-20T09:35:22.663 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5515999999999998e-05 2024-03-20T09:35:22.663 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.663 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.665 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.665 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:01.286031+0000", 2024-03-20T09:35:22.665 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042543260000000001 2024-03-20T09:35:22.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:35:22.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:01.286081+0000", 2024-03-20T09:35:22.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9889000000000003e-05 2024-03-20T09:35:22.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.666 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.666 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:35:22.666 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:01.286121+0000", 2024-03-20T09:35:22.666 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0290999999999998e-05 2024-03-20T09:35:22.666 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.666 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:35:22.666 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.666 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.667 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26216.0:141545 6.11 6:8c7cd173:::benchmark_data_smithi122_168655_object21436:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e781)", 2024-03-20T09:35:22.667 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:27:00.693995+0000", 2024-03-20T09:35:22.667 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 501.91869017599998, 2024-03-20T09:35:22.667 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.59230662700000003, 2024-03-20T09:35:22.667 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:35:22.667 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:22.667 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:35:22.667 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26216", 2024-03-20T09:35:22.667 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:35:22.667 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 141545 2024-03-20T09:35:22.667 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.667 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:35:22.667 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:35:22.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:00.693995+0000", 2024-03-20T09:35:22.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:35:22.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:00.693995+0000", 2024-03-20T09:35:22.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.668 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:35:22.668 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:00.693996+0000", 2024-03-20T09:35:22.668 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6220000000000001e-06 2024-03-20T09:35:22.668 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.668 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:35:22.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:00.694004+0000", 2024-03-20T09:35:22.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.199e-06 2024-03-20T09:35:22.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:35:22.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:00.694017+0000", 2024-03-20T09:35:22.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3753e-05 2024-03-20T09:35:22.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.669 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:35:22.669 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:00.694031+0000", 2024-03-20T09:35:22.669 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3193999999999999e-05 2024-03-20T09:35:22.669 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.669 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:00.694094+0000", 2024-03-20T09:35:22.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3041999999999997e-05 2024-03-20T09:35:22.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:35:22.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:00.694103+0000", 2024-03-20T09:35:22.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9509999999999995e-06 2024-03-20T09:35:22.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.670 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.670 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.670 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:01.277156+0000", 2024-03-20T09:35:22.670 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.58305325900000005 2024-03-20T09:35:22.670 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:35:22.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:01.277288+0000", 2024-03-20T09:35:22.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013226000000000001 2024-03-20T09:35:22.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,11", 2024-03-20T09:35:22.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:01.277423+0000", 2024-03-20T09:35:22.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013462500000000001 2024-03-20T09:35:22.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.671 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.671 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:35:22.671 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:01.281687+0000", 2024-03-20T09:35:22.671 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042647650000000002 2024-03-20T09:35:22.671 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:01.286175+0000", 2024-03-20T09:35:22.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0044874119999999997 2024-03-20T09:35:22.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:01.286217+0000", 2024-03-20T09:35:22.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2484999999999997e-05 2024-03-20T09:35:22.672 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.672 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.672 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:35:22.672 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:01.286263+0000", 2024-03-20T09:35:22.672 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5454000000000002e-05 2024-03-20T09:35:22.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.673 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:35:22.673 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:27:01.286301+0000", 2024-03-20T09:35:22.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8606000000000003e-05 2024-03-20T09:35:22.673 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.673 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:35:22.673 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.673 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.673 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.673 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:213819 7.11 7:8d9623d5:::benchmark_data_smithi122_187077_object87208:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e936)", 2024-03-20T09:35:22.673 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:36.133360+0000", 2024-03-20T09:35:22.673 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 46.479324615000003, 2024-03-20T09:35:22.673 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.225427131, 2024-03-20T09:35:22.674 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:35:22.674 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:22.674 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:35:22.674 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:35:22.674 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:35:22.674 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 213819 2024-03-20T09:35:22.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.674 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:35:22.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.674 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:35:22.674 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.133360+0000", 2024-03-20T09:35:22.674 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.674 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.674 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:35:22.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.133360+0000", 2024-03-20T09:35:22.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:35:22.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.133361+0000", 2024-03-20T09:35:22.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0070000000000001e-06 2024-03-20T09:35:22.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.675 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:35:22.675 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.133365+0000", 2024-03-20T09:35:22.675 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6049999999999998e-06 2024-03-20T09:35:22.675 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.675 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:35:22.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.133376+0000", 2024-03-20T09:35:22.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1289e-05 2024-03-20T09:35:22.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:35:22.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.133385+0000", 2024-03-20T09:35:22.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1800000000000002e-06 2024-03-20T09:35:22.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.676 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.676 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.676 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.135917+0000", 2024-03-20T09:35:22.676 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002531789 2024-03-20T09:35:22.676 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:35:22.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.135928+0000", 2024-03-20T09:35:22.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0859000000000001e-05 2024-03-20T09:35:22.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.329225+0000", 2024-03-20T09:35:22.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.193297351 2024-03-20T09:35:22.677 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.677 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.677 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:35:22.677 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.329367+0000", 2024-03-20T09:35:22.677 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014130300000000001 2024-03-20T09:35:22.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4,6,7,9", 2024-03-20T09:35:22.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.329686+0000", 2024-03-20T09:35:22.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000319024 2024-03-20T09:35:22.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:35:22.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.331282+0000", 2024-03-20T09:35:22.678 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015964869999999999 2024-03-20T09:35:22.678 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.678 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.678 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.678 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.335355+0000", 2024-03-20T09:35:22.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040729449999999997 2024-03-20T09:35:22.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.335396+0000", 2024-03-20T09:35:22.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0850999999999999e-05 2024-03-20T09:35:22.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.679 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.339952+0000", 2024-03-20T09:35:22.679 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0045558939999999996 2024-03-20T09:35:22.679 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.679 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.679 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.358609+0000", 2024-03-20T09:35:22.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018657183000000001 2024-03-20T09:35:22.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:35:22.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.358672+0000", 2024-03-20T09:35:22.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2490000000000006e-05 2024-03-20T09:35:22.680 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.680 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.680 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:35:22.680 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.358787+0000", 2024-03-20T09:35:22.680 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000115874 2024-03-20T09:35:22.680 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.680 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:35:22.681 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.681 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.681 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:220094 7.10 7:0de3b1d8:::benchmark_data_smithi122_187077_object93483:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e939)", 2024-03-20T09:35:22.681 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:44.621484+0000", 2024-03-20T09:35:22.681 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 37.991201314999998, 2024-03-20T09:35:22.681 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.044103123, 2024-03-20T09:35:22.681 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:35:22.681 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:22.681 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:35:22.681 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:35:22.681 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:35:22.681 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 220094 2024-03-20T09:35:22.681 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.682 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:35:22.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:35:22.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.621484+0000", 2024-03-20T09:35:22.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:35:22.682 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.621484+0000", 2024-03-20T09:35:22.682 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.682 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.682 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.682 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:35:22.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.621484+0000", 2024-03-20T09:35:22.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8400000000000003e-07 2024-03-20T09:35:22.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:35:22.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.621488+0000", 2024-03-20T09:35:22.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3409999999999999e-06 2024-03-20T09:35:22.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:35:22.683 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.621495+0000", 2024-03-20T09:35:22.683 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4769999999999997e-06 2024-03-20T09:35:22.683 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.683 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.683 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:35:22.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.621500+0000", 2024-03-20T09:35:22.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6430000000000003e-06 2024-03-20T09:35:22.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.624950+0000", 2024-03-20T09:35:22.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034496700000000002 2024-03-20T09:35:22.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.684 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:35:22.684 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.624959+0000", 2024-03-20T09:35:22.684 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3999999999999998e-06 2024-03-20T09:35:22.684 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.684 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.660516+0000", 2024-03-20T09:35:22.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0355567640000001 2024-03-20T09:35:22.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:35:22.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.660602+0000", 2024-03-20T09:35:22.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6598e-05 2024-03-20T09:35:22.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.685 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.685 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:35:22.685 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.660906+0000", 2024-03-20T09:35:22.685 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00030355699999999998 2024-03-20T09:35:22.685 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.665420+0000", 2024-03-20T09:35:22.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0045136559999999996 2024-03-20T09:35:22.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:35:22.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.665427+0000", 2024-03-20T09:35:22.686 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9440000000000005e-06 2024-03-20T09:35:22.686 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.686 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.686 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.686 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.665506+0000", 2024-03-20T09:35:22.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8060999999999997e-05 2024-03-20T09:35:22.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:35:22.687 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.665541+0000", 2024-03-20T09:35:22.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5843999999999998e-05 2024-03-20T09:35:22.687 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.687 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.687 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:35:22.687 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.665587+0000", 2024-03-20T09:35:22.687 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5383999999999998e-05 2024-03-20T09:35:22.687 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.687 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:35:22.687 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.688 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.688 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:220162 7.10 7:0fd788c0:::benchmark_data_smithi122_187077_object93551:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e939)", 2024-03-20T09:35:22.688 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:44.694214+0000", 2024-03-20T09:35:22.688 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 37.918470646999999, 2024-03-20T09:35:22.688 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.972725795, 2024-03-20T09:35:22.688 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:35:22.688 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:22.688 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:35:22.688 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:35:22.688 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:35:22.688 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 220162 2024-03-20T09:35:22.688 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.688 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:35:22.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:35:22.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.694214+0000", 2024-03-20T09:35:22.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:35:22.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.694214+0000", 2024-03-20T09:35:22.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.689 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.689 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:35:22.689 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.694216+0000", 2024-03-20T09:35:22.689 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-03-20T09:35:22.689 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:35:22.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.694221+0000", 2024-03-20T09:35:22.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.994e-06 2024-03-20T09:35:22.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:35:22.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.694232+0000", 2024-03-20T09:35:22.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1725e-05 2024-03-20T09:35:22.690 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.690 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.690 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:35:22.690 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.694240+0000", 2024-03-20T09:35:22.690 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3030000000000002e-06 2024-03-20T09:35:22.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.696827+0000", 2024-03-20T09:35:22.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0025872740000000001 2024-03-20T09:35:22.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:35:22.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.696837+0000", 2024-03-20T09:35:22.691 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0171e-05 2024-03-20T09:35:22.691 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.691 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.691 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.691 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.662591+0000", 2024-03-20T09:35:22.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.96575427800000002 2024-03-20T09:35:22.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:35:22.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.662669+0000", 2024-03-20T09:35:22.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.784e-05 2024-03-20T09:35:22.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:35:22.692 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.662838+0000", 2024-03-20T09:35:22.692 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000169092 2024-03-20T09:35:22.692 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.692 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.692 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:35:22.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.665482+0000", 2024-03-20T09:35:22.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026433369999999999 2024-03-20T09:35:22.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.665627+0000", 2024-03-20T09:35:22.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000145088 2024-03-20T09:35:22.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.693 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.666839+0000", 2024-03-20T09:35:22.693 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012126769999999999 2024-03-20T09:35:22.693 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.693 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.693 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:35:22.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.666895+0000", 2024-03-20T09:35:22.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5424999999999999e-05 2024-03-20T09:35:22.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.694 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:35:22.694 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.666940+0000", 2024-03-20T09:35:22.694 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5351e-05 2024-03-20T09:35:22.694 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.694 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:35:22.694 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.694 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.694 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.694 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:220174 7.10 7:0823f244:::benchmark_data_smithi122_187077_object93563:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e939)", 2024-03-20T09:35:22.695 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:44.705145+0000", 2024-03-20T09:35:22.695 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 37.907539929000002, 2024-03-20T09:35:22.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.96493656299999997, 2024-03-20T09:35:22.695 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:35:22.695 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:22.695 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:35:22.695 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:35:22.695 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:35:22.695 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 220174 2024-03-20T09:35:22.695 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.695 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:35:22.695 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.695 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:35:22.695 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.705145+0000", 2024-03-20T09:35:22.695 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:35:22.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.705145+0000", 2024-03-20T09:35:22.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:35:22.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.705146+0000", 2024-03-20T09:35:22.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.226e-06 2024-03-20T09:35:22.696 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.696 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.696 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:35:22.696 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.705154+0000", 2024-03-20T09:35:22.696 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0409999999999998e-06 2024-03-20T09:35:22.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:35:22.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.705165+0000", 2024-03-20T09:35:22.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1042e-05 2024-03-20T09:35:22.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:35:22.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.705172+0000", 2024-03-20T09:35:22.697 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4450000000000001e-06 2024-03-20T09:35:22.697 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.697 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.697 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.697 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.705224+0000", 2024-03-20T09:35:22.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2682000000000001e-05 2024-03-20T09:35:22.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:35:22.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.705234+0000", 2024-03-20T09:35:22.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7999999999999993e-06 2024-03-20T09:35:22.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.698 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.663989+0000", 2024-03-20T09:35:22.698 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.95875515899999997 2024-03-20T09:35:22.698 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.698 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.698 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:35:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.664080+0000", 2024-03-20T09:35:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0752000000000001e-05 2024-03-20T09:35:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:35:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.664253+0000", 2024-03-20T09:35:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00017302800000000001 2024-03-20T09:35:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.665670+0000", 2024-03-20T09:35:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014167610000000001 2024-03-20T09:35:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.699 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.666991+0000", 2024-03-20T09:35:22.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001321196 2024-03-20T09:35:22.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:35:22.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.670018+0000", 2024-03-20T09:35:22.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0030268510000000001 2024-03-20T09:35:22.700 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.700 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.700 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:35:22.700 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.670052+0000", 2024-03-20T09:35:22.700 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3794999999999997e-05 2024-03-20T09:35:22.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.701 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:35:22.701 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.670082+0000", 2024-03-20T09:35:22.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9785e-05 2024-03-20T09:35:22.701 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.701 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:35:22.701 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.701 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.701 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.701 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:220176 7.10 7:0fc144d7:::benchmark_data_smithi122_187077_object93565:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e939)", 2024-03-20T09:35:22.701 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:44.705190+0000", 2024-03-20T09:35:22.701 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 37.907494655999997, 2024-03-20T09:35:22.701 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.97056713500000003, 2024-03-20T09:35:22.702 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:35:22.702 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:22.702 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:35:22.702 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:35:22.702 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:35:22.702 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 220176 2024-03-20T09:35:22.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.702 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:35:22.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.702 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:35:22.702 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.705190+0000", 2024-03-20T09:35:22.702 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.702 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:35:22.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.705190+0000", 2024-03-20T09:35:22.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:35:22.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.705191+0000", 2024-03-20T09:35:22.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1399999999999997e-07 2024-03-20T09:35:22.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:35:22.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.705194+0000", 2024-03-20T09:35:22.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.016e-06 2024-03-20T09:35:22.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:35:22.704 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.705201+0000", 2024-03-20T09:35:22.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9099999999999999e-06 2024-03-20T09:35:22.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:35:22.704 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.705207+0000", 2024-03-20T09:35:22.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0449999999999997e-06 2024-03-20T09:35:22.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.704 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.705259+0000", 2024-03-20T09:35:22.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2466000000000002e-05 2024-03-20T09:35:22.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.705 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:35:22.705 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.705265+0000", 2024-03-20T09:35:22.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-06 2024-03-20T09:35:22.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.705 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.705 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.665695+0000", 2024-03-20T09:35:22.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.96042964600000003 2024-03-20T09:35:22.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.705 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:35:22.705 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.665763+0000", 2024-03-20T09:35:22.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8093e-05 2024-03-20T09:35:22.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:35:22.706 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.665928+0000", 2024-03-20T09:35:22.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000164809 2024-03-20T09:35:22.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:35:22.706 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.670090+0000", 2024-03-20T09:35:22.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0041622650000000001 2024-03-20T09:35:22.706 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.706 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.670154+0000", 2024-03-20T09:35:22.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3595000000000004e-05 2024-03-20T09:35:22.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.707 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.675690+0000", 2024-03-20T09:35:22.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0055369069999999998 2024-03-20T09:35:22.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.707 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:35:22.707 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.675727+0000", 2024-03-20T09:35:22.707 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6189000000000002e-05 2024-03-20T09:35:22.707 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.707 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.708 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:35:22.708 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.675757+0000", 2024-03-20T09:35:22.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0800000000000003e-05 2024-03-20T09:35:22.708 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.708 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:35:22.708 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.708 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.708 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.708 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:220274 7.10 7:0e598cf8:::benchmark_data_smithi122_187077_object93663:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e939)", 2024-03-20T09:35:22.708 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:44.817402+0000", 2024-03-20T09:35:22.708 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 37.795282532000002, 2024-03-20T09:35:22.708 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.858534405, 2024-03-20T09:35:22.708 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:35:22.708 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:22.709 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:35:22.709 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:35:22.709 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:35:22.709 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 220274 2024-03-20T09:35:22.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.709 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:35:22.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:35:22.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.817402+0000", 2024-03-20T09:35:22.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.709 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.709 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.709 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:35:22.709 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.817402+0000", 2024-03-20T09:35:22.709 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:35:22.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.817404+0000", 2024-03-20T09:35:22.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2529999999999999e-06 2024-03-20T09:35:22.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:35:22.710 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.817409+0000", 2024-03-20T09:35:22.710 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3380000000000004e-06 2024-03-20T09:35:22.710 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.710 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.710 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:35:22.711 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.817422+0000", 2024-03-20T09:35:22.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.309e-05 2024-03-20T09:35:22.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:35:22.711 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.817430+0000", 2024-03-20T09:35:22.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7489999999999998e-06 2024-03-20T09:35:22.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.711 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.817493+0000", 2024-03-20T09:35:22.711 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3461999999999996e-05 2024-03-20T09:35:22.711 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.711 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.711 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:35:22.712 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.817503+0000", 2024-03-20T09:35:22.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9580000000000002e-06 2024-03-20T09:35:22.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.712 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.712 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.667032+0000", 2024-03-20T09:35:22.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.849528426 2024-03-20T09:35:22.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.712 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:35:22.712 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.667153+0000", 2024-03-20T09:35:22.712 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000121532 2024-03-20T09:35:22.712 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.712 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.712 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:35:22.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.667368+0000", 2024-03-20T09:35:22.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000214305 2024-03-20T09:35:22.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.713 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.675643+0000", 2024-03-20T09:35:22.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0082752980000000004 2024-03-20T09:35:22.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.713 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.713 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.675819+0000", 2024-03-20T09:35:22.713 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000175921 2024-03-20T09:35:22.713 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.713 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:35:22.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.675835+0000", 2024-03-20T09:35:22.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6268000000000001e-05 2024-03-20T09:35:22.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:35:22.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.675890+0000", 2024-03-20T09:35:22.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4960999999999998e-05 2024-03-20T09:35:22.714 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.714 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.714 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:35:22.714 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.675937+0000", 2024-03-20T09:35:22.714 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6844000000000001e-05 2024-03-20T09:35:22.714 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.715 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:35:22.715 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.715 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.715 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:220312 7.10 7:0dcc3d73:::benchmark_data_smithi122_187077_object93701:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e939)", 2024-03-20T09:35:22.715 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:44.859829+0000", 2024-03-20T09:35:22.715 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 37.752855791999998, 2024-03-20T09:35:22.715 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.82423321599999999, 2024-03-20T09:35:22.715 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:35:22.715 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:22.715 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:35:22.715 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:35:22.715 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:35:22.715 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 220312 2024-03-20T09:35:22.715 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.716 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:35:22.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:35:22.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.859829+0000", 2024-03-20T09:35:22.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:35:22.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.859829+0000", 2024-03-20T09:35:22.716 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.716 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.716 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.716 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:35:22.716 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.859830+0000", 2024-03-20T09:35:22.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4099999999999999e-07 2024-03-20T09:35:22.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.717 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:35:22.717 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.859833+0000", 2024-03-20T09:35:22.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2559999999999998e-06 2024-03-20T09:35:22.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.717 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:35:22.717 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.859839+0000", 2024-03-20T09:35:22.717 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4829999999999996e-06 2024-03-20T09:35:22.717 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.717 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:35:22.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.859843+0000", 2024-03-20T09:35:22.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5499999999999999e-06 2024-03-20T09:35:22.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.859940+0000", 2024-03-20T09:35:22.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.6996000000000004e-05 2024-03-20T09:35:22.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.718 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.718 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:35:22.718 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.859947+0000", 2024-03-20T09:35:22.718 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6760000000000001e-06 2024-03-20T09:35:22.718 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.719 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.719 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.668600+0000", 2024-03-20T09:35:22.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.80865322500000003 2024-03-20T09:35:22.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.719 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:35:22.719 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.668701+0000", 2024-03-20T09:35:22.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101279 2024-03-20T09:35:22.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.719 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.719 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:35:22.719 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.668889+0000", 2024-03-20T09:35:22.719 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000188137 2024-03-20T09:35:22.719 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.720 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.720 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.683765+0000", 2024-03-20T09:35:22.720 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01487551 2024-03-20T09:35:22.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.720 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.720 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.683788+0000", 2024-03-20T09:35:22.720 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3748e-05 2024-03-20T09:35:22.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.720 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.720 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:35:22.720 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.683956+0000", 2024-03-20T09:35:22.720 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00016788799999999999 2024-03-20T09:35:22.720 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:35:22.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684013+0000", 2024-03-20T09:35:22.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6891000000000001e-05 2024-03-20T09:35:22.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.721 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:35:22.721 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684062+0000", 2024-03-20T09:35:22.721 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9135999999999998e-05 2024-03-20T09:35:22.721 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.721 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:35:22.721 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.721 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.721 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.722 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:220381 7.10 7:0b0ef641:::benchmark_data_smithi122_187077_object93770:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e939)", 2024-03-20T09:35:22.722 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:44.942756+0000", 2024-03-20T09:35:22.722 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 37.669929312000001, 2024-03-20T09:35:22.722 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.74139817399999997, 2024-03-20T09:35:22.722 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:35:22.722 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:22.722 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:35:22.722 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:35:22.722 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:35:22.722 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 220381 2024-03-20T09:35:22.722 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.722 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:35:22.722 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.722 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:35:22.723 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.942756+0000", 2024-03-20T09:35:22.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.723 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:35:22.723 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.942756+0000", 2024-03-20T09:35:22.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.723 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:35:22.723 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.942757+0000", 2024-03-20T09:35:22.723 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.672e-06 2024-03-20T09:35:22.723 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.723 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.723 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:35:22.724 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.942764+0000", 2024-03-20T09:35:22.724 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3099999999999997e-06 2024-03-20T09:35:22.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.724 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:35:22.724 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.942780+0000", 2024-03-20T09:35:22.724 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5928000000000001e-05 2024-03-20T09:35:22.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.724 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:35:22.724 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.942787+0000", 2024-03-20T09:35:22.724 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7319999999999998e-06 2024-03-20T09:35:22.724 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.724 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.724 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.942851+0000", 2024-03-20T09:35:22.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3829000000000002e-05 2024-03-20T09:35:22.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:35:22.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.942860+0000", 2024-03-20T09:35:22.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5660000000000007e-06 2024-03-20T09:35:22.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.725 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.725 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.670195+0000", 2024-03-20T09:35:22.725 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.72733487699999999 2024-03-20T09:35:22.725 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.725 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.726 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:35:22.726 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.670305+0000", 2024-03-20T09:35:22.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011026099999999999 2024-03-20T09:35:22.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.726 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:35:22.726 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.670561+0000", 2024-03-20T09:35:22.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025577500000000002 2024-03-20T09:35:22.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.726 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.726 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.683818+0000", 2024-03-20T09:35:22.726 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013257193 2024-03-20T09:35:22.726 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.726 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.727 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.683838+0000", 2024-03-20T09:35:22.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0089e-05 2024-03-20T09:35:22.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.727 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:35:22.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684072+0000", 2024-03-20T09:35:22.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00023379999999999999 2024-03-20T09:35:22.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.727 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.727 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:35:22.727 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684116+0000", 2024-03-20T09:35:22.727 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4186e-05 2024-03-20T09:35:22.727 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.728 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:35:22.728 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684154+0000", 2024-03-20T09:35:22.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7956e-05 2024-03-20T09:35:22.728 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.728 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:35:22.728 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.728 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.728 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.728 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:220445 7.10 7:0f889c96:::benchmark_data_smithi122_187077_object93834:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e939)", 2024-03-20T09:35:22.728 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:45.016832+0000", 2024-03-20T09:35:22.728 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 37.595852440999998, 2024-03-20T09:35:22.728 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.66740921499999994, 2024-03-20T09:35:22.728 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:35:22.729 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:22.729 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:35:22.729 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:35:22.729 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:35:22.729 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 220445 2024-03-20T09:35:22.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.729 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:35:22.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.729 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:35:22.729 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.016832+0000", 2024-03-20T09:35:22.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.729 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.729 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.729 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:35:22.729 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.016832+0000", 2024-03-20T09:35:22.729 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:35:22.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.016833+0000", 2024-03-20T09:35:22.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7600000000000006e-07 2024-03-20T09:35:22.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:35:22.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.016837+0000", 2024-03-20T09:35:22.730 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1939999999999998e-06 2024-03-20T09:35:22.730 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.730 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.730 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:35:22.730 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.016848+0000", 2024-03-20T09:35:22.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1442e-05 2024-03-20T09:35:22.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:35:22.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.016854+0000", 2024-03-20T09:35:22.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8000000000000004e-06 2024-03-20T09:35:22.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.731 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.016912+0000", 2024-03-20T09:35:22.731 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7976999999999997e-05 2024-03-20T09:35:22.731 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.731 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.731 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:35:22.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.016917+0000", 2024-03-20T09:35:22.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3820000000000003e-06 2024-03-20T09:35:22.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.671579+0000", 2024-03-20T09:35:22.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.65466185200000004 2024-03-20T09:35:22.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:35:22.732 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.671701+0000", 2024-03-20T09:35:22.732 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000121835 2024-03-20T09:35:22.732 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.732 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.732 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:35:22.733 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.671919+0000", 2024-03-20T09:35:22.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000217653 2024-03-20T09:35:22.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.733 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.733 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.683860+0000", 2024-03-20T09:35:22.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011941230000000001 2024-03-20T09:35:22.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.733 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.733 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.683885+0000", 2024-03-20T09:35:22.733 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4890999999999999e-05 2024-03-20T09:35:22.733 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.733 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.733 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:35:22.734 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684162+0000", 2024-03-20T09:35:22.734 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00027684200000000002 2024-03-20T09:35:22.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.734 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:35:22.734 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684204+0000", 2024-03-20T09:35:22.734 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2537999999999998e-05 2024-03-20T09:35:22.734 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.734 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.734 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:35:22.734 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684242+0000", 2024-03-20T09:35:22.734 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7602999999999999e-05 2024-03-20T09:35:22.734 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.734 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:35:22.735 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.735 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.735 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:220450 7.10 7:083cb24d:::benchmark_data_smithi122_187077_object93839:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e939)", 2024-03-20T09:35:22.735 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:45.022699+0000", 2024-03-20T09:35:22.735 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 37.589986273999997, 2024-03-20T09:35:22.735 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.66164454900000003, 2024-03-20T09:35:22.735 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:35:22.735 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:22.735 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:35:22.735 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:35:22.735 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:35:22.735 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 220450 2024-03-20T09:35:22.735 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.735 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:35:22.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:35:22.736 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.022699+0000", 2024-03-20T09:35:22.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:35:22.736 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.022699+0000", 2024-03-20T09:35:22.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.736 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.736 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:35:22.736 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.022700+0000", 2024-03-20T09:35:22.736 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1650000000000001e-06 2024-03-20T09:35:22.736 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:35:22.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.022708+0000", 2024-03-20T09:35:22.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3659999999999992e-06 2024-03-20T09:35:22.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:35:22.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.022719+0000", 2024-03-20T09:35:22.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1113e-05 2024-03-20T09:35:22.737 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.737 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.737 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:35:22.737 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.022725+0000", 2024-03-20T09:35:22.737 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.015e-06 2024-03-20T09:35:22.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.738 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.738 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.022771+0000", 2024-03-20T09:35:22.738 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5750000000000001e-05 2024-03-20T09:35:22.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.738 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:35:22.738 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.022779+0000", 2024-03-20T09:35:22.738 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4889999999999998e-06 2024-03-20T09:35:22.738 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.738 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.738 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.738 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.672870+0000", 2024-03-20T09:35:22.739 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.65009129499999996 2024-03-20T09:35:22.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.739 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:35:22.739 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.672971+0000", 2024-03-20T09:35:22.739 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010152700000000001 2024-03-20T09:35:22.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.739 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:35:22.739 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.673129+0000", 2024-03-20T09:35:22.739 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015808 2024-03-20T09:35:22.739 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.739 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.739 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.739 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.683940+0000", 2024-03-20T09:35:22.740 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010810336 2024-03-20T09:35:22.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.740 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:35:22.740 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684250+0000", 2024-03-20T09:35:22.740 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00031030499999999998 2024-03-20T09:35:22.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.740 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.740 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684286+0000", 2024-03-20T09:35:22.740 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5509000000000002e-05 2024-03-20T09:35:22.740 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.740 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.740 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:35:22.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684315+0000", 2024-03-20T09:35:22.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9624999999999998e-05 2024-03-20T09:35:22.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.741 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:35:22.741 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684343+0000", 2024-03-20T09:35:22.741 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7974e-05 2024-03-20T09:35:22.741 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.741 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:35:22.741 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.741 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.741 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.741 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:220478 7.10 7:0f3ce08f:::benchmark_data_smithi122_187077_object93867:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e939)", 2024-03-20T09:35:22.741 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:45.066044+0000", 2024-03-20T09:35:22.742 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 37.546641344000001, 2024-03-20T09:35:22.742 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.61846596399999998, 2024-03-20T09:35:22.742 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:35:22.742 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:22.742 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:35:22.742 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:35:22.742 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:35:22.742 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 220478 2024-03-20T09:35:22.742 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.742 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:35:22.742 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.742 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:35:22.742 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.066044+0000", 2024-03-20T09:35:22.742 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:35:22.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.066044+0000", 2024-03-20T09:35:22.743 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:35:22.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.066045+0000", 2024-03-20T09:35:22.743 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.424e-06 2024-03-20T09:35:22.743 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.743 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.743 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:35:22.743 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.066051+0000", 2024-03-20T09:35:22.744 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7119999999999997e-06 2024-03-20T09:35:22.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.744 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:35:22.744 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.066065+0000", 2024-03-20T09:35:22.744 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.38e-05 2024-03-20T09:35:22.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.744 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:35:22.744 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.066073+0000", 2024-03-20T09:35:22.744 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2139999999999996e-06 2024-03-20T09:35:22.744 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.744 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.744 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.744 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.066137+0000", 2024-03-20T09:35:22.745 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3876999999999998e-05 2024-03-20T09:35:22.745 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.745 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.745 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:35:22.745 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.066148+0000", 2024-03-20T09:35:22.745 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1484999999999999e-05 2024-03-20T09:35:22.745 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.745 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.745 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.745 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.674079+0000", 2024-03-20T09:35:22.745 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.60793130500000003 2024-03-20T09:35:22.745 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.745 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.746 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:35:22.746 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.674195+0000", 2024-03-20T09:35:22.746 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000115457 2024-03-20T09:35:22.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.746 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:35:22.746 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.674398+0000", 2024-03-20T09:35:22.746 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00020298700000000001 2024-03-20T09:35:22.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.746 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.746 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:35:22.746 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684257+0000", 2024-03-20T09:35:22.746 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0098594200000000007 2024-03-20T09:35:22.746 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684371+0000", 2024-03-20T09:35:22.747 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000113529 2024-03-20T09:35:22.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684426+0000", 2024-03-20T09:35:22.747 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5336e-05 2024-03-20T09:35:22.747 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.747 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.747 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:35:22.747 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684480+0000", 2024-03-20T09:35:22.747 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4166000000000003e-05 2024-03-20T09:35:22.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.748 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:35:22.748 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684510+0000", 2024-03-20T09:35:22.748 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9252e-05 2024-03-20T09:35:22.748 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.748 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:35:22.748 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.748 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.748 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.748 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:220492 7.10 7:0816362b:::benchmark_data_smithi122_187077_object93881:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e939)", 2024-03-20T09:35:22.748 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:45.097130+0000", 2024-03-20T09:35:22.748 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 37.515554903999998, 2024-03-20T09:35:22.748 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.58752722199999996, 2024-03-20T09:35:22.749 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:35:22.749 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:22.749 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:35:22.749 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:35:22.749 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:35:22.749 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 220492 2024-03-20T09:35:22.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.749 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:35:22.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.749 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:35:22.749 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.097130+0000", 2024-03-20T09:35:22.749 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.749 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.749 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.749 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:35:22.750 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.097130+0000", 2024-03-20T09:35:22.750 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:35:22.750 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.750 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:35:22.750 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.097132+0000", 2024-03-20T09:35:22.750 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5269999999999999e-06 2024-03-20T09:35:22.750 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.750 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:35:22.750 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.097139+0000", 2024-03-20T09:35:22.750 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0910000000000004e-06 2024-03-20T09:35:22.750 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.750 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.751 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:35:22.751 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.097155+0000", 2024-03-20T09:35:22.751 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6062e-05 2024-03-20T09:35:22.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.751 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.751 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:35:22.751 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.097164+0000", 2024-03-20T09:35:22.751 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.38e-06 2024-03-20T09:35:22.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.751 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.751 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.751 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.097226+0000", 2024-03-20T09:35:22.751 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2038999999999993e-05 2024-03-20T09:35:22.751 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.752 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:35:22.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.097236+0000", 2024-03-20T09:35:22.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3999999999999998e-06 2024-03-20T09:35:22.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.752 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:35:22.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.676020+0000", 2024-03-20T09:35:22.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.57878407700000001 2024-03-20T09:35:22.752 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.752 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.752 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:35:22.752 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.676095+0000", 2024-03-20T09:35:22.752 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5822999999999996e-05 2024-03-20T09:35:22.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.753 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:35:22.753 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.676258+0000", 2024-03-20T09:35:22.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00016275400000000001 2024-03-20T09:35:22.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.753 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:35:22.753 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684523+0000", 2024-03-20T09:35:22.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0082645029999999994 2024-03-20T09:35:22.753 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.753 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.753 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.753 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684573+0000", 2024-03-20T09:35:22.753 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0234e-05 2024-03-20T09:35:22.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:22.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684601+0000", 2024-03-20T09:35:22.754 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7659999999999999e-05 2024-03-20T09:35:22.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:35:22.754 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684629+0000", 2024-03-20T09:35:22.754 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8184e-05 2024-03-20T09:35:22.754 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:35:22.754 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:35:22.754 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:35:22.755 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684657+0000", 2024-03-20T09:35:22.755 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8487999999999999e-05 2024-03-20T09:35:22.755 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.755 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:35:22.755 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.755 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:35:22.755 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:35:22.755 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:35:22.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:22.756 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:35:22.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:35:22.858 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:35:22.889 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:22.912 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:35:22.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:22.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:35:22.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:22 smithi026 ceph-mon[68309]: 1.2 scrub starts 2024-03-20T09:35:22.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:22 smithi026 ceph-mon[68309]: pgmap v2325: 57 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+remapped, 1 remapped+peering, 3 active+remapped+backfill_toofull, 51 active+clean; 213 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 928/10185 objects misplaced (9.111%) 2024-03-20T09:35:22.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:22 smithi026 ceph-mon[68309]: osdmap e963: 12 total, 12 up, 12 in 2024-03-20T09:35:23.018 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:35:23.048 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:23.075 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:35:23.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:23.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:35:23.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:22 smithi115 ceph-mon[72476]: 1.2 scrub starts 2024-03-20T09:35:23.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:22 smithi115 ceph-mon[72476]: pgmap v2325: 57 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+remapped, 1 remapped+peering, 3 active+remapped+backfill_toofull, 51 active+clean; 213 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 928/10185 objects misplaced (9.111%) 2024-03-20T09:35:23.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:22 smithi115 ceph-mon[72476]: osdmap e963: 12 total, 12 up, 12 in 2024-03-20T09:35:23.181 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:35:23.263 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:23.311 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:35:23.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:23.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:35:23.421 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:35:23.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:22 smithi122 ceph-mon[73495]: 1.2 scrub starts 2024-03-20T09:35:23.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:22 smithi122 ceph-mon[73495]: pgmap v2325: 57 pgs: 1 active+remapped+backfill_wait, 1 active+undersized+remapped, 1 remapped+peering, 3 active+remapped+backfill_toofull, 51 active+clean; 213 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 928/10185 objects misplaced (9.111%) 2024-03-20T09:35:23.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:22 smithi122 ceph-mon[73495]: osdmap e963: 12 total, 12 up, 12 in 2024-03-20T09:35:23.453 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:23.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:23.482 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:35:23.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:35:23.584 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:35:23.668 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:23.868 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:35:23.898 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:35:23.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:23.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:35:23.959 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:35:24.009 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:35:24.141 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:24.166 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:35:24.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:24.173 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:35:24.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:23 smithi026 ceph-mon[68309]: 1.2 scrub ok 2024-03-20T09:35:24.215 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:23 smithi026 ceph-mon[68309]: 1.18 scrub starts 2024-03-20T09:35:24.215 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:23 smithi026 ceph-mon[68309]: 1.18 scrub ok 2024-03-20T09:35:24.215 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:23 smithi026 ceph-mon[68309]: 1.b scrub starts 2024-03-20T09:35:24.215 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:23 smithi026 ceph-mon[68309]: 1.b scrub ok 2024-03-20T09:35:24.273 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:35:24.302 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:23 smithi115 ceph-mon[72476]: 1.2 scrub ok 2024-03-20T09:35:24.302 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:23 smithi115 ceph-mon[72476]: 1.18 scrub starts 2024-03-20T09:35:24.302 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:23 smithi115 ceph-mon[72476]: 1.18 scrub ok 2024-03-20T09:35:24.302 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:23 smithi115 ceph-mon[72476]: 1.b scrub starts 2024-03-20T09:35:24.302 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:23 smithi115 ceph-mon[72476]: 1.b scrub ok 2024-03-20T09:35:24.308 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:24.329 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:35:24.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:24.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:35:24.434 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:35:24.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:23 smithi122 ceph-mon[73495]: 1.2 scrub ok 2024-03-20T09:35:24.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:23 smithi122 ceph-mon[73495]: 1.18 scrub starts 2024-03-20T09:35:24.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:23 smithi122 ceph-mon[73495]: 1.18 scrub ok 2024-03-20T09:35:24.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:23 smithi122 ceph-mon[73495]: 1.b scrub starts 2024-03-20T09:35:24.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:23 smithi122 ceph-mon[73495]: 1.b scrub ok 2024-03-20T09:35:24.461 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:24.473 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-20T09:35:24.480 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:35:24.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:24.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:35:24.524 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Removed 126608 objects 2024-03-20T09:35:24.524 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Clean up completed and total clean up time :154.87 2024-03-20T09:35:24.584 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:35:24.611 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:24.630 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:35:24.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:24.633 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:35:24.735 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:35:24.751 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:35:24.761 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:24.780 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:35:24.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:24.783 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:35:24.884 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:35:24.910 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:24.929 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:35:24.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:24.932 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:35:25.033 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:35:25.064 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:25.116 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:35:25.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:25.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:35:25.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:24 smithi122 ceph-mon[73495]: 1.19 scrub starts 2024-03-20T09:35:25.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:24 smithi122 ceph-mon[73495]: 1.19 scrub ok 2024-03-20T09:35:25.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:24 smithi122 ceph-mon[73495]: 1.4 scrub starts 2024-03-20T09:35:25.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:24 smithi122 ceph-mon[73495]: 1.4 scrub ok 2024-03-20T09:35:25.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:24 smithi122 ceph-mon[73495]: pgmap v2327: 57 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 53 active+clean; 174 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 674/8337 objects misplaced (8.084%); 572 KiB/s, 8 objects/s recovering 2024-03-20T09:35:25.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:24 smithi026 ceph-mon[68309]: 1.19 scrub starts 2024-03-20T09:35:25.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:24 smithi026 ceph-mon[68309]: 1.19 scrub ok 2024-03-20T09:35:25.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:24 smithi026 ceph-mon[68309]: 1.4 scrub starts 2024-03-20T09:35:25.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:24 smithi026 ceph-mon[68309]: 1.4 scrub ok 2024-03-20T09:35:25.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:24 smithi026 ceph-mon[68309]: pgmap v2327: 57 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 53 active+clean; 174 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 674/8337 objects misplaced (8.084%); 572 KiB/s, 8 objects/s recovering 2024-03-20T09:35:25.223 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:35:25.254 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:25.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:24 smithi115 ceph-mon[72476]: 1.19 scrub starts 2024-03-20T09:35:25.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:24 smithi115 ceph-mon[72476]: 1.19 scrub ok 2024-03-20T09:35:25.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:24 smithi115 ceph-mon[72476]: 1.4 scrub starts 2024-03-20T09:35:25.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:24 smithi115 ceph-mon[72476]: 1.4 scrub ok 2024-03-20T09:35:25.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:24 smithi115 ceph-mon[72476]: pgmap v2327: 57 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 53 active+clean; 174 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 674/8337 objects misplaced (8.084%); 572 KiB/s, 8 objects/s recovering 2024-03-20T09:35:25.971 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:35:25.983 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:35:25.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:25.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:35:26.000 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:35:26.088 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:35:26.118 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:26.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:25 smithi115 ceph-mon[72476]: 1.11 scrub starts 2024-03-20T09:35:26.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:25 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/235590976' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:35:26.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:25 smithi026 ceph-mon[68309]: 1.11 scrub starts 2024-03-20T09:35:26.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:25 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/235590976' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:35:26.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:25 smithi122 ceph-mon[73495]: 1.11 scrub starts 2024-03-20T09:35:26.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:25 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/235590976' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:35:26.669 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:35:26.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:26.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:35:26.772 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:35:26.810 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:26.982 INFO:teuthology.orchestra.run.smithi026.stderr:set osd.5 primary-affinity to 0.928009 (8608172) 2024-03-20T09:35:27.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:26 smithi115 ceph-mon[72476]: 1.11 scrub ok 2024-03-20T09:35:27.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:26 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:35:27.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:26 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/235590976' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:35:27.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:26 smithi115 ceph-mon[72476]: osdmap e964: 12 total, 12 up, 12 in 2024-03-20T09:35:27.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:26 smithi115 ceph-mon[72476]: pgmap v2329: 57 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 53 active+clean; 62 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 237/2961 objects misplaced (8.004%) 2024-03-20T09:35:27.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:26 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/774502252' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.9280087288595185}]: dispatch 2024-03-20T09:35:27.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:26 smithi122 ceph-mon[73495]: 1.11 scrub ok 2024-03-20T09:35:27.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:26 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:35:27.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:26 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/235590976' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:35:27.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:26 smithi122 ceph-mon[73495]: osdmap e964: 12 total, 12 up, 12 in 2024-03-20T09:35:27.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:26 smithi122 ceph-mon[73495]: pgmap v2329: 57 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 53 active+clean; 62 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 237/2961 objects misplaced (8.004%) 2024-03-20T09:35:27.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:26 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/774502252' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.9280087288595185}]: dispatch 2024-03-20T09:35:27.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:26 smithi026 ceph-mon[68309]: 1.11 scrub ok 2024-03-20T09:35:27.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:26 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:35:27.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:26 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/235590976' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:35:27.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:26 smithi026 ceph-mon[68309]: osdmap e964: 12 total, 12 up, 12 in 2024-03-20T09:35:27.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:26 smithi026 ceph-mon[68309]: pgmap v2329: 57 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 53 active+clean; 62 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 237/2961 objects misplaced (8.004%) 2024-03-20T09:35:27.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:26 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/774502252' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.9280087288595185}]: dispatch 2024-03-20T09:35:28.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:27 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/774502252' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.9280087288595185}]': finished 2024-03-20T09:35:28.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:27 smithi115 ceph-mon[72476]: osdmap e965: 12 total, 12 up, 12 in 2024-03-20T09:35:28.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:27 smithi115 ceph-mon[72476]: 1.16 deep-scrub starts 2024-03-20T09:35:28.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:27 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/774502252' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.9280087288595185}]': finished 2024-03-20T09:35:28.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:27 smithi122 ceph-mon[73495]: osdmap e965: 12 total, 12 up, 12 in 2024-03-20T09:35:28.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:27 smithi122 ceph-mon[73495]: 1.16 deep-scrub starts 2024-03-20T09:35:28.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:27 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/774502252' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.9280087288595185}]': finished 2024-03-20T09:35:28.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:27 smithi026 ceph-mon[68309]: osdmap e965: 12 total, 12 up, 12 in 2024-03-20T09:35:28.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:27 smithi026 ceph-mon[68309]: 1.16 deep-scrub starts 2024-03-20T09:35:28.857 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_5 2024-03-20T09:35:28.858 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pool rm unique_pool_5 unique_pool_5 --yes-i-really-really-mean-it 2024-03-20T09:35:29.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:28 smithi026 ceph-mon[68309]: 1.16 deep-scrub ok 2024-03-20T09:35:29.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:28 smithi026 ceph-mon[68309]: osdmap e966: 12 total, 12 up, 12 in 2024-03-20T09:35:29.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:28 smithi026 ceph-mon[68309]: pgmap v2332: 57 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 53 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 593 op/s; 2.4 MiB/s, 39 objects/s recovering 2024-03-20T09:35:29.236 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:35:29.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:29.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:35:29.341 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:35:29.370 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:29.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:29 smithi115 ceph-mon[72476]: 1.16 deep-scrub ok 2024-03-20T09:35:29.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:29 smithi115 ceph-mon[72476]: osdmap e966: 12 total, 12 up, 12 in 2024-03-20T09:35:29.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:29 smithi115 ceph-mon[72476]: pgmap v2332: 57 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 53 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 593 op/s; 2.4 MiB/s, 39 objects/s recovering 2024-03-20T09:35:29.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:28 smithi122 ceph-mon[73495]: 1.16 deep-scrub ok 2024-03-20T09:35:29.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:28 smithi122 ceph-mon[73495]: osdmap e966: 12 total, 12 up, 12 in 2024-03-20T09:35:29.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:28 smithi122 ceph-mon[73495]: pgmap v2332: 57 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfill_toofull, 53 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 593 op/s; 2.4 MiB/s, 39 objects/s recovering 2024-03-20T09:35:30.037 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:35:30.037 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:35:30.037 INFO:teuthology.orchestra.run.smithi115.stdout: "num_ops": 0 2024-03-20T09:35:30.038 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:35:30.743 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-20T09:35:31.008 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:35:31.849 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:31 smithi115 ceph-mon[72476]: pgmap v2333: 57 pgs: 3 active+remapped+backfill_toofull, 54 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 463 op/s 2024-03-20T09:35:31.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:31 smithi122 ceph-mon[73495]: pgmap v2333: 57 pgs: 3 active+remapped+backfill_toofull, 54 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 463 op/s 2024-03-20T09:35:31.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:31 smithi026 ceph-mon[68309]: pgmap v2333: 57 pgs: 3 active+remapped+backfill_toofull, 54 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 463 op/s 2024-03-20T09:35:32.646 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:32 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:35:32.646 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:32 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:35:32.646 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:32 smithi115 ceph-mon[72476]: osdmap e967: 12 total, 12 up, 12 in 2024-03-20T09:35:32.711 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:35:32.712 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:35:32.712 INFO:teuthology.orchestra.run.smithi115.stdout: "complaint_time": 30, 2024-03-20T09:35:32.712 INFO:teuthology.orchestra.run.smithi115.stdout: "num_blocked_ops": 0 2024-03-20T09:35:32.712 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:35:32.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:32 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:35:32.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:32 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:35:32.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:32 smithi122 ceph-mon[73495]: osdmap e967: 12 total, 12 up, 12 in 2024-03-20T09:35:32.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:32 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:35:32.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:32 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:35:32.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:32 smithi026 ceph-mon[68309]: osdmap e967: 12 total, 12 up, 12 in 2024-03-20T09:35:33.346 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-20T09:35:33.613 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:35:33.846 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:33 smithi115 ceph-mon[72476]: pgmap v2335: 57 pgs: 3 active+remapped+backfill_toofull, 54 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 164 op/s 2024-03-20T09:35:33.847 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:33 smithi115 ceph-mon[72476]: 7.f scrub starts 2024-03-20T09:35:33.847 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:33 smithi115 ceph-mon[72476]: 7.f scrub ok 2024-03-20T09:35:33.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:33 smithi122 ceph-mon[73495]: pgmap v2335: 57 pgs: 3 active+remapped+backfill_toofull, 54 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 164 op/s 2024-03-20T09:35:33.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:33 smithi122 ceph-mon[73495]: 7.f scrub starts 2024-03-20T09:35:33.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:33 smithi122 ceph-mon[73495]: 7.f scrub ok 2024-03-20T09:35:33.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:33 smithi026 ceph-mon[68309]: pgmap v2335: 57 pgs: 3 active+remapped+backfill_toofull, 54 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 164 op/s 2024-03-20T09:35:33.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:33 smithi026 ceph-mon[68309]: 7.f scrub starts 2024-03-20T09:35:33.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:33 smithi026 ceph-mon[68309]: 7.f scrub ok 2024-03-20T09:35:34.728 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:34 smithi115 ceph-mon[72476]: 1.1c scrub starts 2024-03-20T09:35:34.728 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:34 smithi115 ceph-mon[72476]: 1.1c scrub ok 2024-03-20T09:35:34.728 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:34 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:35:34.728 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:34 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:35:34.728 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:34 smithi115 ceph-mon[72476]: 1.12 scrub starts 2024-03-20T09:35:34.728 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:34 smithi115 ceph-mon[72476]: 1.12 scrub ok 2024-03-20T09:35:34.728 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:34 smithi115 ceph-mon[72476]: osdmap e968: 12 total, 12 up, 12 in 2024-03-20T09:35:34.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:34 smithi122 ceph-mon[73495]: 1.1c scrub starts 2024-03-20T09:35:34.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:34 smithi122 ceph-mon[73495]: 1.1c scrub ok 2024-03-20T09:35:34.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:34 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:35:34.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:34 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:35:34.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:34 smithi122 ceph-mon[73495]: 1.12 scrub starts 2024-03-20T09:35:34.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:34 smithi122 ceph-mon[73495]: 1.12 scrub ok 2024-03-20T09:35:34.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:34 smithi122 ceph-mon[73495]: osdmap e968: 12 total, 12 up, 12 in 2024-03-20T09:35:34.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:34 smithi026 ceph-mon[68309]: 1.1c scrub starts 2024-03-20T09:35:34.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:34 smithi026 ceph-mon[68309]: 1.1c scrub ok 2024-03-20T09:35:34.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:34 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:35:34.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:34 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:35:34.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:34 smithi026 ceph-mon[68309]: 1.12 scrub starts 2024-03-20T09:35:34.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:34 smithi026 ceph-mon[68309]: 1.12 scrub ok 2024-03-20T09:35:34.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:34 smithi026 ceph-mon[68309]: osdmap e968: 12 total, 12 up, 12 in 2024-03-20T09:35:35.172 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:35:35.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:35.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:35:35.260 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:35:35.260 INFO:teuthology.orchestra.run.smithi115.stdout: "size": 20, 2024-03-20T09:35:35.260 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 600, 2024-03-20T09:35:35.260 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [ 2024-03-20T09:35:35.260 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.260 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:129144 6.2 6:47957f3d:::benchmark_data_smithi122_168655_object9035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e748)", 2024-03-20T09:35:35.260 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:56.335243+0000", 2024-03-20T09:35:35.260 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 578.93173694200004, 2024-03-20T09:35:35.260 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.223256595, 2024-03-20T09:35:35.260 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:35.261 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:35:35.261 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:35:35.261 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:35:35.261 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:35:35.261 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 129144 2024-03-20T09:35:35.261 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.261 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:35.261 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.261 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:35.261 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335243+0000", 2024-03-20T09:35:35.261 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.261 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.261 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.261 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:35.262 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335243+0000", 2024-03-20T09:35:35.262 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.262 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.262 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.262 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:35.262 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335244+0000", 2024-03-20T09:35:35.262 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0299999999999999e-07 2024-03-20T09:35:35.262 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.262 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.262 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:35.262 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335246+0000", 2024-03-20T09:35:35.262 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.4959999999999999e-06 2024-03-20T09:35:35.262 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.263 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.263 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:35.263 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335253+0000", 2024-03-20T09:35:35.263 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.6699999999999997e-06 2024-03-20T09:35:35.263 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.263 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.263 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:35.263 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335258+0000", 2024-03-20T09:35:35.263 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.7500000000000003e-06 2024-03-20T09:35:35.263 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.263 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.263 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:35.263 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335321+0000", 2024-03-20T09:35:35.263 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2765e-05 2024-03-20T09:35:35.264 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.264 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.264 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:35:35.264 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335440+0000", 2024-03-20T09:35:35.264 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000119141 2024-03-20T09:35:35.264 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.264 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.264 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 2,3", 2024-03-20T09:35:35.264 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335511+0000", 2024-03-20T09:35:35.264 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.1104000000000002e-05 2024-03-20T09:35:35.264 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.264 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.264 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:35.264 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.342379+0000", 2024-03-20T09:35:35.265 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0068685320000000001 2024-03-20T09:35:35.265 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.265 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.265 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:35.265 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.557233+0000", 2024-03-20T09:35:35.265 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.21485372999999999 2024-03-20T09:35:35.265 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.265 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.265 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:35.265 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.558500+0000", 2024-03-20T09:35:35.265 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0012669039999999999 2024-03-20T09:35:35.265 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.265 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:35.265 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.266 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.266 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.266 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:129123 6.2 6:450098af:::benchmark_data_smithi122_168655_object9014:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e749)", 2024-03-20T09:35:35.266 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:56.335273+0000", 2024-03-20T09:35:35.266 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 578.93170711799996, 2024-03-20T09:35:35.266 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0772038850000001, 2024-03-20T09:35:35.266 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:35.266 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:35:35.266 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:35:35.266 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:35:35.266 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:35:35.266 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 129123 2024-03-20T09:35:35.266 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.266 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:35.267 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.267 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:35.267 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335273+0000", 2024-03-20T09:35:35.267 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.267 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.267 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.267 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:35.267 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335273+0000", 2024-03-20T09:35:35.267 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.267 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.267 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.267 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:35.267 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335274+0000", 2024-03-20T09:35:35.267 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6300000000000001e-07 2024-03-20T09:35:35.268 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.268 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.268 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:35.268 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335276+0000", 2024-03-20T09:35:35.268 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2089999999999999e-06 2024-03-20T09:35:35.268 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.268 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.268 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:35.268 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335282+0000", 2024-03-20T09:35:35.268 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0800000000000002e-06 2024-03-20T09:35:35.268 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.268 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.268 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:35.268 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335287+0000", 2024-03-20T09:35:35.269 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4700000000000001e-06 2024-03-20T09:35:35.269 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.269 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.269 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:35.269 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.339142+0000", 2024-03-20T09:35:35.269 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0038545229999999999 2024-03-20T09:35:35.269 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.269 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.269 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op must wait for map", 2024-03-20T09:35:35.269 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.339150+0000", 2024-03-20T09:35:35.269 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.6279999999999996e-06 2024-03-20T09:35:35.269 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.269 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.269 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:35.270 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.557266+0000", 2024-03-20T09:35:35.270 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.21811618799999999 2024-03-20T09:35:35.270 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.270 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.270 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:35:35.270 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.557275+0000", 2024-03-20T09:35:35.270 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.8810000000000002e-06 2024-03-20T09:35:35.270 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.270 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.270 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:35.270 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:57.412381+0000", 2024-03-20T09:35:35.270 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.85510642599999998 2024-03-20T09:35:35.270 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.270 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.271 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:35.271 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:57.412477+0000", 2024-03-20T09:35:35.271 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.6117000000000001e-05 2024-03-20T09:35:35.271 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.271 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:35.271 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.271 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.271 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.271 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:129125 6.12 6:4e4918e7:::benchmark_data_smithi122_168655_object9016:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e749)", 2024-03-20T09:35:35.271 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:56.335325+0000", 2024-03-20T09:35:35.271 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 578.93165500700002, 2024-03-20T09:35:35.271 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.13051883, 2024-03-20T09:35:35.271 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:35.271 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:35:35.271 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:35:35.272 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:35:35.272 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:35:35.272 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 129125 2024-03-20T09:35:35.272 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.272 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:35.272 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.272 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:35.272 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335325+0000", 2024-03-20T09:35:35.272 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.272 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.272 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.272 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:35.272 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335325+0000", 2024-03-20T09:35:35.272 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.273 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.273 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.273 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:35.273 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335326+0000", 2024-03-20T09:35:35.273 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.4300000000000002e-07 2024-03-20T09:35:35.273 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.273 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.273 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:35.273 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335331+0000", 2024-03-20T09:35:35.273 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.566e-06 2024-03-20T09:35:35.273 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.273 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.273 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:35.273 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335339+0000", 2024-03-20T09:35:35.274 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.4430000000000003e-06 2024-03-20T09:35:35.274 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.274 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.274 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:35.274 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335346+0000", 2024-03-20T09:35:35.274 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.2339999999999997e-06 2024-03-20T09:35:35.274 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.274 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.274 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:35.274 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.558708+0000", 2024-03-20T09:35:35.274 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.223361277 2024-03-20T09:35:35.274 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.274 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.275 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:35:35.275 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.558717+0000", 2024-03-20T09:35:35.275 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.7350000000000008e-06 2024-03-20T09:35:35.275 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.275 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.275 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:35.275 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:57.465759+0000", 2024-03-20T09:35:35.275 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.90704171499999997 2024-03-20T09:35:35.275 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.275 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.275 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:35.275 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:57.465844+0000", 2024-03-20T09:35:35.275 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5117000000000005e-05 2024-03-20T09:35:35.275 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.276 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:35.276 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.276 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.276 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.276 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:129129 6.2 6:471d5bbb:::benchmark_data_smithi122_168655_object9020:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e749)", 2024-03-20T09:35:35.276 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:56.335359+0000", 2024-03-20T09:35:35.276 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 578.93162187400003, 2024-03-20T09:35:35.276 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.077221416, 2024-03-20T09:35:35.276 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:35.276 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:35:35.276 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:35:35.276 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:35:35.276 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:35:35.276 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 129129 2024-03-20T09:35:35.277 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.277 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:35.277 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.277 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:35.277 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335359+0000", 2024-03-20T09:35:35.277 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.277 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.277 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.277 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:35.277 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335359+0000", 2024-03-20T09:35:35.278 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.278 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.278 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.278 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:35.278 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335359+0000", 2024-03-20T09:35:35.278 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.2500000000000001e-07 2024-03-20T09:35:35.278 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.278 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.279 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:35.279 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335361+0000", 2024-03-20T09:35:35.279 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3240000000000001e-06 2024-03-20T09:35:35.279 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.279 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.279 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:35.279 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335368+0000", 2024-03-20T09:35:35.279 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.3790000000000003e-06 2024-03-20T09:35:35.280 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.280 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.280 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:35.280 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335372+0000", 2024-03-20T09:35:35.280 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.724e-06 2024-03-20T09:35:35.280 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.281 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.281 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:35.281 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.339225+0000", 2024-03-20T09:35:35.281 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0038523030000000001 2024-03-20T09:35:35.281 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.281 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.281 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting_for_map not empty", 2024-03-20T09:35:35.282 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.339230+0000", 2024-03-20T09:35:35.282 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9540000000000003e-06 2024-03-20T09:35:35.282 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.283 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.283 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:35.283 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.557298+0000", 2024-03-20T09:35:35.283 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.21806794900000001 2024-03-20T09:35:35.284 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.284 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.284 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:35:35.284 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.557305+0000", 2024-03-20T09:35:35.284 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3520000000000001e-06 2024-03-20T09:35:35.285 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.285 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.285 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:35.285 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:57.412500+0000", 2024-03-20T09:35:35.285 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.85519480299999995 2024-03-20T09:35:35.285 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.286 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.286 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:35.286 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:57.412580+0000", 2024-03-20T09:35:35.286 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.0203000000000004e-05 2024-03-20T09:35:35.286 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.286 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:35.287 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.287 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.287 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.287 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:129144 6.2 6:47957f3d:::benchmark_data_smithi122_168655_object9035:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e749)", 2024-03-20T09:35:35.287 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:56.335379+0000", 2024-03-20T09:35:35.287 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 578.93160132100002, 2024-03-20T09:35:35.287 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1031262369999999, 2024-03-20T09:35:35.288 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:35.288 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:35:35.288 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:35:35.288 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:35:35.288 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:35:35.288 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 129144 2024-03-20T09:35:35.288 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.288 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:35.288 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.288 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:35.289 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335379+0000", 2024-03-20T09:35:35.289 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.289 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.289 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.289 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:35.289 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335379+0000", 2024-03-20T09:35:35.289 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.289 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.289 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.289 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:35.289 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335379+0000", 2024-03-20T09:35:35.289 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.5900000000000003e-07 2024-03-20T09:35:35.289 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.289 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.290 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:35.290 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335382+0000", 2024-03-20T09:35:35.290 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.181e-06 2024-03-20T09:35:35.290 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.290 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.290 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:35.290 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335387+0000", 2024-03-20T09:35:35.290 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9799999999999998e-06 2024-03-20T09:35:35.290 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.290 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.290 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:35.290 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.335391+0000", 2024-03-20T09:35:35.290 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5519999999999998e-06 2024-03-20T09:35:35.291 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.291 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.291 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:35.291 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.339468+0000", 2024-03-20T09:35:35.291 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0040772990000000004 2024-03-20T09:35:35.291 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.291 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.291 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting_for_map not empty", 2024-03-20T09:35:35.291 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.339473+0000", 2024-03-20T09:35:35.291 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5800000000000002e-06 2024-03-20T09:35:35.291 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.291 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.291 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:35.291 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.557326+0000", 2024-03-20T09:35:35.291 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.21785318400000001 2024-03-20T09:35:35.292 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.292 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.292 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:35:35.292 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.557333+0000", 2024-03-20T09:35:35.292 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.7719999999999997e-06 2024-03-20T09:35:35.292 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.292 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.292 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:35.292 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:57.412602+0000", 2024-03-20T09:35:35.292 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.85526948700000005 2024-03-20T09:35:35.292 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.292 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.292 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for missing object", 2024-03-20T09:35:35.292 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:57.412726+0000", 2024-03-20T09:35:35.293 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012335300000000001 2024-03-20T09:35:35.293 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.293 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.293 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:35.293 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:57.438420+0000", 2024-03-20T09:35:35.293 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.025694552999999998 2024-03-20T09:35:35.293 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.293 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.293 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:35.293 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:57.438505+0000", 2024-03-20T09:35:35.293 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.4937000000000003e-05 2024-03-20T09:35:35.293 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.293 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:35.294 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.294 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.294 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.294 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:32470 7.1 7:83406e34:::benchmark_data_smithi122_187077_object32469:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e879)", 2024-03-20T09:35:35.294 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:31:30.643697+0000", 2024-03-20T09:35:35.294 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 244.62328367500001, 2024-03-20T09:35:35.295 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.142067165, 2024-03-20T09:35:35.295 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:35.295 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:35.295 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:35:35.295 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:35:35.295 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:35:35.295 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 32470 2024-03-20T09:35:35.295 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.295 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:35.296 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.296 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:35.296 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:30.643697+0000", 2024-03-20T09:35:35.296 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.296 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.296 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.296 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:35.296 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:30.643697+0000", 2024-03-20T09:35:35.296 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.297 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.297 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.297 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:35.297 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:30.643698+0000", 2024-03-20T09:35:35.297 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.249e-06 2024-03-20T09:35:35.297 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.297 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.297 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:35.297 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:30.643771+0000", 2024-03-20T09:35:35.297 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3231000000000004e-05 2024-03-20T09:35:35.297 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.297 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.297 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:35.297 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:30.643785+0000", 2024-03-20T09:35:35.298 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3594999999999999e-05 2024-03-20T09:35:35.298 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.298 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.298 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:35.298 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:30.643793+0000", 2024-03-20T09:35:35.298 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5339999999999999e-06 2024-03-20T09:35:35.298 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.298 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.298 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:35.298 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:30.643856+0000", 2024-03-20T09:35:35.298 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2430999999999999e-05 2024-03-20T09:35:35.298 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.298 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.299 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:35:35.299 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:30.643988+0000", 2024-03-20T09:35:35.299 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013209800000000001 2024-03-20T09:35:35.299 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.299 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.299 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 3,6", 2024-03-20T09:35:35.299 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:30.644135+0000", 2024-03-20T09:35:35.299 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00014720600000000001 2024-03-20T09:35:35.299 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.299 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.299 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:35:35.299 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:30.645281+0000", 2024-03-20T09:35:35.299 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001145648 2024-03-20T09:35:35.299 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.299 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.300 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:35.300 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:30.645563+0000", 2024-03-20T09:35:35.300 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000282133 2024-03-20T09:35:35.300 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.300 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.300 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:35.300 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:30.785699+0000", 2024-03-20T09:35:35.300 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.14013568000000001 2024-03-20T09:35:35.300 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.300 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.300 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:35:35.300 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:30.785737+0000", 2024-03-20T09:35:35.301 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.8809000000000001e-05 2024-03-20T09:35:35.301 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.301 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.301 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:35.301 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:30.785764+0000", 2024-03-20T09:35:35.301 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6551e-05 2024-03-20T09:35:35.301 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.301 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:35.301 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.301 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.301 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.301 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:37216 7.1 7:8f9a83fa:::benchmark_data_smithi122_187077_object37215:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e880)", 2024-03-20T09:35:35.301 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:31:33.355390+0000", 2024-03-20T09:35:35.301 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 241.91159072799999, 2024-03-20T09:35:35.302 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.114137998, 2024-03-20T09:35:35.302 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:35.302 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:35.302 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:35:35.302 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:35:35.302 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:35:35.302 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 37216 2024-03-20T09:35:35.302 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.302 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:35.302 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.303 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:35.303 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:33.355390+0000", 2024-03-20T09:35:35.303 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.303 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.303 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.303 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:35.303 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:33.355390+0000", 2024-03-20T09:35:35.303 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.303 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.303 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.303 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:35.303 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:33.355391+0000", 2024-03-20T09:35:35.303 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.212e-06 2024-03-20T09:35:35.303 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.303 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.304 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:35.304 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:33.355457+0000", 2024-03-20T09:35:35.304 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.6502999999999997e-05 2024-03-20T09:35:35.304 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.304 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.304 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:35.304 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:33.355470+0000", 2024-03-20T09:35:35.304 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2176e-05 2024-03-20T09:35:35.304 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.304 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.305 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:35.305 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:33.355477+0000", 2024-03-20T09:35:35.305 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.5290000000000003e-06 2024-03-20T09:35:35.305 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.305 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.305 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:35.305 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:33.355536+0000", 2024-03-20T09:35:35.305 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8854000000000003e-05 2024-03-20T09:35:35.305 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.305 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.305 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:35:35.305 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:33.355676+0000", 2024-03-20T09:35:35.305 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000140536 2024-03-20T09:35:35.306 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.306 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.306 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 3,6", 2024-03-20T09:35:35.306 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:33.355832+0000", 2024-03-20T09:35:35.306 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00015580300000000001 2024-03-20T09:35:35.306 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.306 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.306 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:35:35.306 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:33.357016+0000", 2024-03-20T09:35:35.306 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0011835750000000001 2024-03-20T09:35:35.307 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.307 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.307 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:35.307 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:33.359527+0000", 2024-03-20T09:35:35.307 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0025109260000000001 2024-03-20T09:35:35.307 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.307 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.307 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:35.307 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:33.469410+0000", 2024-03-20T09:35:35.307 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.109883323 2024-03-20T09:35:35.307 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.307 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.307 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:35:35.308 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:33.469475+0000", 2024-03-20T09:35:35.308 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.444e-05 2024-03-20T09:35:35.308 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.308 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.308 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:35.308 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:33.469528+0000", 2024-03-20T09:35:35.308 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3121000000000002e-05 2024-03-20T09:35:35.308 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.308 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:35.308 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.308 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.308 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.309 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:75477 7.1 7:89955dc0:::benchmark_data_smithi122_187077_object75476:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e890)", 2024-03-20T09:35:35.309 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:31:58.394953+0000", 2024-03-20T09:35:35.309 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 216.872027148, 2024-03-20T09:35:35.309 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.11389574500000001, 2024-03-20T09:35:35.309 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:35.309 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:35.309 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:35:35.310 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:35:35.310 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:35:35.310 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 75477 2024-03-20T09:35:35.310 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.310 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:35.310 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.311 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:35.311 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:58.394953+0000", 2024-03-20T09:35:35.311 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.311 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.311 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.311 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:35.311 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:58.394953+0000", 2024-03-20T09:35:35.312 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.312 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.312 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.312 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:35.312 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:58.394955+0000", 2024-03-20T09:35:35.312 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4330000000000001e-06 2024-03-20T09:35:35.313 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.313 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.313 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:35.313 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:58.394996+0000", 2024-03-20T09:35:35.313 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.1108e-05 2024-03-20T09:35:35.313 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.313 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.314 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:35.314 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:58.395011+0000", 2024-03-20T09:35:35.314 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5648e-05 2024-03-20T09:35:35.314 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.314 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.314 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:35.315 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:58.395020+0000", 2024-03-20T09:35:35.315 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5399999999999996e-06 2024-03-20T09:35:35.315 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.315 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.315 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:35.315 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:58.396718+0000", 2024-03-20T09:35:35.316 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0016976070000000001 2024-03-20T09:35:35.316 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.316 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.316 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:35:35.316 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:58.396870+0000", 2024-03-20T09:35:35.316 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00015249299999999999 2024-03-20T09:35:35.316 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.317 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.317 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 3,6", 2024-03-20T09:35:35.317 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:58.397020+0000", 2024-03-20T09:35:35.317 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00014951799999999999 2024-03-20T09:35:35.317 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.317 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.318 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:35:35.318 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:58.402291+0000", 2024-03-20T09:35:35.318 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0052717440000000001 2024-03-20T09:35:35.318 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.318 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.319 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:35.319 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:58.404450+0000", 2024-03-20T09:35:35.319 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.002158798 2024-03-20T09:35:35.319 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.319 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.320 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:35.321 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:58.508735+0000", 2024-03-20T09:35:35.321 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.10428483099999999 2024-03-20T09:35:35.321 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.321 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.321 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:35:35.321 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:58.508798+0000", 2024-03-20T09:35:35.321 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.3361000000000005e-05 2024-03-20T09:35:35.321 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.321 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.321 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:35.321 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:58.508849+0000", 2024-03-20T09:35:35.321 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0664000000000001e-05 2024-03-20T09:35:35.321 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.321 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:35.322 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.322 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.322 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.322 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:189201 7.1 7:8830b297:::benchmark_data_smithi122_187077_object62590:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e926)", 2024-03-20T09:35:35.322 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:03.436110+0000", 2024-03-20T09:35:35.322 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 91.830870184000005, 2024-03-20T09:35:35.322 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.213765964, 2024-03-20T09:35:35.322 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:35.322 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:35.322 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:35:35.322 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:35:35.322 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:35:35.322 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 189201 2024-03-20T09:35:35.322 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.323 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:35.323 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.323 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:35.323 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.436110+0000", 2024-03-20T09:35:35.323 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.323 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.323 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.323 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:35.323 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.436110+0000", 2024-03-20T09:35:35.323 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.323 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.323 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.323 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:35.323 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.436112+0000", 2024-03-20T09:35:35.324 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.359e-06 2024-03-20T09:35:35.324 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.324 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.324 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:35.324 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.436118+0000", 2024-03-20T09:35:35.324 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.6479999999999997e-06 2024-03-20T09:35:35.324 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.324 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.324 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:35.324 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.436130+0000", 2024-03-20T09:35:35.324 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1761e-05 2024-03-20T09:35:35.324 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.324 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.324 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:35.325 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.436140+0000", 2024-03-20T09:35:35.325 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.6350000000000001e-06 2024-03-20T09:35:35.325 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.325 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.325 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:35.325 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.437375+0000", 2024-03-20T09:35:35.325 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0012352820000000001 2024-03-20T09:35:35.325 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.325 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.325 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:35:35.325 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.437516+0000", 2024-03-20T09:35:35.325 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00014065799999999999 2024-03-20T09:35:35.325 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.325 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.325 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 3,6", 2024-03-20T09:35:35.326 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.437644+0000", 2024-03-20T09:35:35.326 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012795999999999999 2024-03-20T09:35:35.326 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.326 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.326 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:35:35.326 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.441847+0000", 2024-03-20T09:35:35.326 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0042038010000000001 2024-03-20T09:35:35.326 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.326 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.326 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:35.326 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.448723+0000", 2024-03-20T09:35:35.326 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0068752739999999998 2024-03-20T09:35:35.326 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.326 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.327 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:35.327 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.649776+0000", 2024-03-20T09:35:35.327 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.20105339999999999 2024-03-20T09:35:35.327 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.327 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.327 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:35:35.327 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.649823+0000", 2024-03-20T09:35:35.327 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6761000000000002e-05 2024-03-20T09:35:35.327 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.327 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.327 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:35.327 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.649876+0000", 2024-03-20T09:35:35.327 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3424999999999998e-05 2024-03-20T09:35:35.327 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.327 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:35.328 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.328 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.328 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.328 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:189209 7.1 7:886d864e:::benchmark_data_smithi122_187077_object62598:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e926)", 2024-03-20T09:35:35.328 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:03.443419+0000", 2024-03-20T09:35:35.328 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 91.823561741999995, 2024-03-20T09:35:35.328 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.24267246100000001, 2024-03-20T09:35:35.328 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:35.328 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:35.328 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:35:35.328 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:35:35.328 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:35:35.328 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 189209 2024-03-20T09:35:35.328 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.329 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:35.329 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.329 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:35.329 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.443419+0000", 2024-03-20T09:35:35.329 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.329 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.329 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.329 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:35.329 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.443419+0000", 2024-03-20T09:35:35.329 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.329 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.329 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.329 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:35.329 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.443420+0000", 2024-03-20T09:35:35.329 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4640000000000001e-06 2024-03-20T09:35:35.330 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.330 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.330 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:35.330 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.443429+0000", 2024-03-20T09:35:35.330 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.473e-06 2024-03-20T09:35:35.330 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.330 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.330 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:35.330 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.443444+0000", 2024-03-20T09:35:35.330 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5101e-05 2024-03-20T09:35:35.330 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.330 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.330 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:35.331 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.443452+0000", 2024-03-20T09:35:35.331 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.6479999999999992e-06 2024-03-20T09:35:35.331 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.331 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.331 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:35.331 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.443507+0000", 2024-03-20T09:35:35.331 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4317999999999997e-05 2024-03-20T09:35:35.331 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.331 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.331 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:35:35.331 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.443632+0000", 2024-03-20T09:35:35.331 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000124881 2024-03-20T09:35:35.331 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.331 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.332 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 3,6", 2024-03-20T09:35:35.332 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.443753+0000", 2024-03-20T09:35:35.332 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000121249 2024-03-20T09:35:35.332 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.332 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.332 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:35.332 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.448757+0000", 2024-03-20T09:35:35.332 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0050043969999999998 2024-03-20T09:35:35.332 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.332 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.332 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:35:35.332 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.453242+0000", 2024-03-20T09:35:35.332 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0044844819999999997 2024-03-20T09:35:35.332 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.332 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.333 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:35.333 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.686007+0000", 2024-03-20T09:35:35.333 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.23276515 2024-03-20T09:35:35.333 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.333 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.333 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:35:35.333 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.686045+0000", 2024-03-20T09:35:35.333 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.8432000000000001e-05 2024-03-20T09:35:35.333 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.333 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.333 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:35.333 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.686091+0000", 2024-03-20T09:35:35.333 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5865999999999998e-05 2024-03-20T09:35:35.333 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.334 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:35.334 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.334 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.334 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.334 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:189210 7.1 7:89f0a701:::benchmark_data_smithi122_187077_object62599:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e926)", 2024-03-20T09:35:35.334 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:03.444159+0000", 2024-03-20T09:35:35.334 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 91.822820993999997, 2024-03-20T09:35:35.334 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.25194981300000002, 2024-03-20T09:35:35.334 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:35.334 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:35.334 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:35:35.334 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:35:35.334 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:35:35.334 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 189210 2024-03-20T09:35:35.335 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.335 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:35.335 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.335 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:35.335 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.444159+0000", 2024-03-20T09:35:35.335 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.335 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.335 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.335 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:35.335 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.444159+0000", 2024-03-20T09:35:35.335 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.335 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.335 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.335 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:35.335 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.444160+0000", 2024-03-20T09:35:35.336 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.9299999999999997e-07 2024-03-20T09:35:35.336 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.336 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.336 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:35.336 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.444164+0000", 2024-03-20T09:35:35.336 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.9249999999999997e-06 2024-03-20T09:35:35.336 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.336 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.336 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:35.336 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.444176+0000", 2024-03-20T09:35:35.336 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1746000000000001e-05 2024-03-20T09:35:35.336 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.336 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.336 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:35.337 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.444195+0000", 2024-03-20T09:35:35.337 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9476000000000001e-05 2024-03-20T09:35:35.337 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.337 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.337 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:35.337 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.448785+0000", 2024-03-20T09:35:35.337 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0045896330000000001 2024-03-20T09:35:35.337 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.337 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.337 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:35:35.338 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.448915+0000", 2024-03-20T09:35:35.338 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013024299999999999 2024-03-20T09:35:35.338 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.338 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.338 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 3,6", 2024-03-20T09:35:35.338 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.449037+0000", 2024-03-20T09:35:35.338 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000122033 2024-03-20T09:35:35.338 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.338 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.338 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:35:35.338 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.453388+0000", 2024-03-20T09:35:35.338 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.004350919 2024-03-20T09:35:35.338 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.338 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.339 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:35.339 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.457738+0000", 2024-03-20T09:35:35.339 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0043497340000000001 2024-03-20T09:35:35.339 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.339 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.339 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:35.339 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.696004+0000", 2024-03-20T09:35:35.339 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.23826560399999999 2024-03-20T09:35:35.339 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.339 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.339 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:35:35.339 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.696062+0000", 2024-03-20T09:35:35.339 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8498999999999997e-05 2024-03-20T09:35:35.339 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.339 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.340 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:35.340 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.696109+0000", 2024-03-20T09:35:35.340 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.7207999999999999e-05 2024-03-20T09:35:35.340 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.340 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:35.340 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.340 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.340 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.340 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:189227 7.1 7:85bcbe46:::benchmark_data_smithi122_187077_object62616:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e926)", 2024-03-20T09:35:35.340 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:03.464934+0000", 2024-03-20T09:35:35.340 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 91.802046270999995, 2024-03-20T09:35:35.340 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.23131220599999999, 2024-03-20T09:35:35.340 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:35.340 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:35.341 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:35:35.341 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:35:35.341 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:35:35.341 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 189227 2024-03-20T09:35:35.341 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.341 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:35.341 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.341 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:35.341 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.464934+0000", 2024-03-20T09:35:35.341 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.341 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.341 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.341 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:35.341 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.464934+0000", 2024-03-20T09:35:35.341 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.342 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.342 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.342 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:35.342 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.464936+0000", 2024-03-20T09:35:35.342 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.691e-06 2024-03-20T09:35:35.342 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.342 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.342 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:35.342 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.464942+0000", 2024-03-20T09:35:35.342 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8000000000000004e-06 2024-03-20T09:35:35.342 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.342 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.342 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:35.342 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.464959+0000", 2024-03-20T09:35:35.343 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7223000000000001e-05 2024-03-20T09:35:35.343 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.343 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.343 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:35.343 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.464968+0000", 2024-03-20T09:35:35.343 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.2040000000000003e-06 2024-03-20T09:35:35.343 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.343 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.343 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:35.343 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.465029+0000", 2024-03-20T09:35:35.343 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0912999999999998e-05 2024-03-20T09:35:35.343 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.343 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.343 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:35:35.344 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.465164+0000", 2024-03-20T09:35:35.344 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013462500000000001 2024-03-20T09:35:35.344 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.344 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.344 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 3,6", 2024-03-20T09:35:35.344 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.465295+0000", 2024-03-20T09:35:35.344 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000131001 2024-03-20T09:35:35.344 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.344 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.344 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:35.344 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.470164+0000", 2024-03-20T09:35:35.344 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0048693800000000004 2024-03-20T09:35:35.344 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.344 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.344 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:35:35.345 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.470244+0000", 2024-03-20T09:35:35.345 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.0130999999999996e-05 2024-03-20T09:35:35.345 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.345 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.345 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:35.345 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.696160+0000", 2024-03-20T09:35:35.345 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.225915796 2024-03-20T09:35:35.345 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.345 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.345 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:35:35.346 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.696201+0000", 2024-03-20T09:35:35.346 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.1075000000000002e-05 2024-03-20T09:35:35.346 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.346 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.346 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:35.346 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.696246+0000", 2024-03-20T09:35:35.346 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5367000000000001e-05 2024-03-20T09:35:35.346 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.346 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:35.346 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.346 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.346 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.346 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:189246 7.1 7:87cbc540:::benchmark_data_smithi122_187077_object62635:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e926)", 2024-03-20T09:35:35.347 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:03.486420+0000", 2024-03-20T09:35:35.347 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 91.780560692999998, 2024-03-20T09:35:35.347 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.20995987299999999, 2024-03-20T09:35:35.347 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:35.347 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:35.347 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:35:35.347 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:35:35.347 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:35:35.347 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 189246 2024-03-20T09:35:35.347 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.347 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:35.347 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.347 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:35.347 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.486420+0000", 2024-03-20T09:35:35.348 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.348 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.348 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.348 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:35.348 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.486420+0000", 2024-03-20T09:35:35.348 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.348 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.348 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.348 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:35.348 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.486422+0000", 2024-03-20T09:35:35.348 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9080000000000002e-06 2024-03-20T09:35:35.348 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.348 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.348 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:35.348 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.486430+0000", 2024-03-20T09:35:35.349 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.9489999999999996e-06 2024-03-20T09:35:35.349 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.349 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.349 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:35.349 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.486451+0000", 2024-03-20T09:35:35.349 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1212999999999998e-05 2024-03-20T09:35:35.349 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.349 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.349 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:35.349 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.486461+0000", 2024-03-20T09:35:35.349 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.7969999999999995e-06 2024-03-20T09:35:35.349 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.349 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.349 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:35.350 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.486524+0000", 2024-03-20T09:35:35.350 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.3845999999999998e-05 2024-03-20T09:35:35.350 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.350 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.350 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:35:35.350 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.486631+0000", 2024-03-20T09:35:35.350 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000107042 2024-03-20T09:35:35.350 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.350 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.350 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 3,6", 2024-03-20T09:35:35.350 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.486756+0000", 2024-03-20T09:35:35.351 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012472699999999999 2024-03-20T09:35:35.351 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.351 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.351 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:35.351 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.490840+0000", 2024-03-20T09:35:35.351 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0040834729999999998 2024-03-20T09:35:35.351 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.351 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.351 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:35:35.351 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.490903+0000", 2024-03-20T09:35:35.351 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.3607000000000003e-05 2024-03-20T09:35:35.351 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.351 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.351 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:35.351 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.696294+0000", 2024-03-20T09:35:35.352 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.20539083699999999 2024-03-20T09:35:35.352 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.352 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.352 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:35:35.352 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.696345+0000", 2024-03-20T09:35:35.352 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0844999999999999e-05 2024-03-20T09:35:35.352 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.352 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.352 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:35.352 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.696380+0000", 2024-03-20T09:35:35.352 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.4629000000000003e-05 2024-03-20T09:35:35.352 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.352 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:35.352 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.353 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.353 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.353 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:189262 7.1 7:83e62354:::benchmark_data_smithi122_187077_object62651:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e926)", 2024-03-20T09:35:35.353 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:03.514200+0000", 2024-03-20T09:35:35.353 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 91.752780817000001, 2024-03-20T09:35:35.353 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.18234990300000001, 2024-03-20T09:35:35.353 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:35.353 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:35.353 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:35:35.353 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:35:35.353 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:35:35.353 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 189262 2024-03-20T09:35:35.353 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.353 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:35.354 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.354 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:35.354 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.514200+0000", 2024-03-20T09:35:35.354 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.354 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.354 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.354 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:35.354 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.514200+0000", 2024-03-20T09:35:35.354 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.354 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.354 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.354 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:35.354 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.514201+0000", 2024-03-20T09:35:35.354 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.384e-06 2024-03-20T09:35:35.354 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.355 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.355 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:35.355 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.514207+0000", 2024-03-20T09:35:35.355 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.071e-06 2024-03-20T09:35:35.355 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.355 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.355 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:35.355 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.514223+0000", 2024-03-20T09:35:35.355 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5724999999999999e-05 2024-03-20T09:35:35.355 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.355 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.355 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:35.355 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.514233+0000", 2024-03-20T09:35:35.355 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.8789999999999998e-06 2024-03-20T09:35:35.356 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.356 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.356 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:35.356 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.514313+0000", 2024-03-20T09:35:35.356 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.0693999999999997e-05 2024-03-20T09:35:35.356 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.356 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.356 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:35:35.356 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.514451+0000", 2024-03-20T09:35:35.356 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013721100000000001 2024-03-20T09:35:35.356 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.356 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.356 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 3,6", 2024-03-20T09:35:35.356 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.514580+0000", 2024-03-20T09:35:35.356 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012985500000000001 2024-03-20T09:35:35.357 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.357 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.357 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:35.357 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.519209+0000", 2024-03-20T09:35:35.357 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0046285290000000002 2024-03-20T09:35:35.357 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.357 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.357 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:35:35.357 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.521364+0000", 2024-03-20T09:35:35.357 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.002155246 2024-03-20T09:35:35.357 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.357 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.357 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:35.357 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.696450+0000", 2024-03-20T09:35:35.358 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.175085565 2024-03-20T09:35:35.358 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.358 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.358 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:35:35.358 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.696507+0000", 2024-03-20T09:35:35.358 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7649000000000003e-05 2024-03-20T09:35:35.358 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.358 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.358 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:35.358 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.696550+0000", 2024-03-20T09:35:35.358 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.2095000000000003e-05 2024-03-20T09:35:35.358 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.358 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:35.358 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.359 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.359 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.359 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:189279 7.1 7:87e1d3c8:::benchmark_data_smithi122_187077_object62668:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e926)", 2024-03-20T09:35:35.359 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:03.534432+0000", 2024-03-20T09:35:35.359 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 91.732548790999999, 2024-03-20T09:35:35.359 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.16222515700000001, 2024-03-20T09:35:35.359 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:35.359 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:35.359 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:35:35.359 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:35:35.359 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:35:35.359 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 189279 2024-03-20T09:35:35.359 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.360 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:35.360 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.360 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:35.360 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.534432+0000", 2024-03-20T09:35:35.360 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.360 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.360 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.360 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:35.360 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.534432+0000", 2024-03-20T09:35:35.360 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.360 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.360 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.360 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:35.360 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.534433+0000", 2024-03-20T09:35:35.360 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5969999999999999e-06 2024-03-20T09:35:35.361 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.361 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.361 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:35.361 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.534440+0000", 2024-03-20T09:35:35.361 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.2429999999999999e-06 2024-03-20T09:35:35.361 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.361 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.361 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:35.361 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.534458+0000", 2024-03-20T09:35:35.361 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7977000000000001e-05 2024-03-20T09:35:35.361 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.361 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.361 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:35.361 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.534469+0000", 2024-03-20T09:35:35.362 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0162e-05 2024-03-20T09:35:35.362 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.362 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.362 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:35.362 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.534535+0000", 2024-03-20T09:35:35.362 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.6614999999999998e-05 2024-03-20T09:35:35.362 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.362 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.362 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:35:35.362 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.534658+0000", 2024-03-20T09:35:35.362 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012228400000000001 2024-03-20T09:35:35.362 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.362 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.362 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 3,6", 2024-03-20T09:35:35.363 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.534782+0000", 2024-03-20T09:35:35.363 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012413300000000001 2024-03-20T09:35:35.363 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.363 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.363 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:35:35.363 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.542741+0000", 2024-03-20T09:35:35.363 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0079590330000000008 2024-03-20T09:35:35.363 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.363 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.363 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:35.363 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.546849+0000", 2024-03-20T09:35:35.363 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.004108347 2024-03-20T09:35:35.363 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.363 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.363 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:35.364 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.696591+0000", 2024-03-20T09:35:35.364 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.14974216000000001 2024-03-20T09:35:35.364 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.364 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.364 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:35:35.364 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.696622+0000", 2024-03-20T09:35:35.364 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.0454999999999999e-05 2024-03-20T09:35:35.364 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.364 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.364 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:35.364 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.696657+0000", 2024-03-20T09:35:35.364 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.5151000000000003e-05 2024-03-20T09:35:35.364 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.364 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:35.365 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.365 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.365 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.365 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:189281 7.1 7:8c7cb7f4:::benchmark_data_smithi122_187077_object62670:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e926)", 2024-03-20T09:35:35.365 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:03.534799+0000", 2024-03-20T09:35:35.365 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 91.732181658000002, 2024-03-20T09:35:35.365 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.16196291500000001, 2024-03-20T09:35:35.365 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:35.365 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:35.365 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:35:35.365 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:35:35.365 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:35:35.365 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 189281 2024-03-20T09:35:35.365 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.366 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:35.366 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.366 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:35.366 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.534799+0000", 2024-03-20T09:35:35.366 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.366 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.366 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.366 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:35.366 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.534799+0000", 2024-03-20T09:35:35.366 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.366 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.366 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.366 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:35.366 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.534800+0000", 2024-03-20T09:35:35.367 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.8299999999999996e-07 2024-03-20T09:35:35.367 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.367 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.367 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:35.367 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.534806+0000", 2024-03-20T09:35:35.367 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4180000000000002e-06 2024-03-20T09:35:35.367 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.367 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.367 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:35.367 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.534814+0000", 2024-03-20T09:35:35.367 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.4700000000000002e-06 2024-03-20T09:35:35.367 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.367 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.367 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:35.367 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.534819+0000", 2024-03-20T09:35:35.368 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.7759999999999997e-06 2024-03-20T09:35:35.368 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.368 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.368 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:35.368 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.538737+0000", 2024-03-20T09:35:35.368 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0039175249999999998 2024-03-20T09:35:35.368 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.368 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.368 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:35:35.368 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.538874+0000", 2024-03-20T09:35:35.368 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000137735 2024-03-20T09:35:35.368 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.368 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.368 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 3,6", 2024-03-20T09:35:35.369 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.539008+0000", 2024-03-20T09:35:35.369 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013369200000000001 2024-03-20T09:35:35.369 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.369 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.369 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:35:35.369 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.546868+0000", 2024-03-20T09:35:35.369 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0078597960000000005 2024-03-20T09:35:35.369 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.369 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.369 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:35.369 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.550993+0000", 2024-03-20T09:35:35.369 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0041246490000000002 2024-03-20T09:35:35.369 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.369 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.369 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:35.370 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.696699+0000", 2024-03-20T09:35:35.370 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.14570614800000001 2024-03-20T09:35:35.370 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.370 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.370 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:35:35.370 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.696739+0000", 2024-03-20T09:35:35.370 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.0465e-05 2024-03-20T09:35:35.370 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.370 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.370 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:35.370 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.696762+0000", 2024-03-20T09:35:35.370 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2458000000000001e-05 2024-03-20T09:35:35.370 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.370 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:35.371 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.371 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.371 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.371 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:189285 7.1 7:8c518b7c:::benchmark_data_smithi122_187077_object62674:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e926)", 2024-03-20T09:35:35.371 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:03.539884+0000", 2024-03-20T09:35:35.371 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 91.727096630000005, 2024-03-20T09:35:35.371 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.16507955899999999, 2024-03-20T09:35:35.371 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:35.371 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:35.371 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:35:35.371 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:35:35.371 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:35:35.371 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 189285 2024-03-20T09:35:35.371 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.372 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:35.372 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.372 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:35.372 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.539884+0000", 2024-03-20T09:35:35.372 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.372 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.372 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.372 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:35.372 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.539884+0000", 2024-03-20T09:35:35.372 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.372 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.372 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.372 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:35.372 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.539885+0000", 2024-03-20T09:35:35.372 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3340000000000001e-06 2024-03-20T09:35:35.373 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.373 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.373 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:35.373 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.539891+0000", 2024-03-20T09:35:35.373 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9209999999999997e-06 2024-03-20T09:35:35.373 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.373 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.373 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:35.373 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.539904+0000", 2024-03-20T09:35:35.373 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.309e-05 2024-03-20T09:35:35.373 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.373 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.373 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:35.373 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.539913+0000", 2024-03-20T09:35:35.374 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5469999999999997e-06 2024-03-20T09:35:35.374 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.374 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.374 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:35.374 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.542777+0000", 2024-03-20T09:35:35.374 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0028648100000000002 2024-03-20T09:35:35.374 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.374 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.374 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:35:35.374 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.542910+0000", 2024-03-20T09:35:35.374 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000132667 2024-03-20T09:35:35.374 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.374 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.374 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 3,6", 2024-03-20T09:35:35.375 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.543036+0000", 2024-03-20T09:35:35.375 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012627 2024-03-20T09:35:35.375 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.375 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.375 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:35:35.375 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.551012+0000", 2024-03-20T09:35:35.375 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0079757830000000002 2024-03-20T09:35:35.375 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.375 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.375 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:35.375 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.551066+0000", 2024-03-20T09:35:35.375 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3535000000000002e-05 2024-03-20T09:35:35.375 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.375 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.376 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:35.376 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.704866+0000", 2024-03-20T09:35:35.376 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.153799777 2024-03-20T09:35:35.376 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.376 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.376 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:35:35.376 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.704921+0000", 2024-03-20T09:35:35.376 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5059999999999998e-05 2024-03-20T09:35:35.376 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.376 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.376 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:35.376 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.704963+0000", 2024-03-20T09:35:35.376 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.2765000000000001e-05 2024-03-20T09:35:35.376 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.376 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:35.377 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.377 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.377 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.377 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:189289 7.1 7:8551da0a:::benchmark_data_smithi122_187077_object62678:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e926)", 2024-03-20T09:35:35.377 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:03.543959+0000", 2024-03-20T09:35:35.377 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 91.723021566, 2024-03-20T09:35:35.377 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.16113079299999999, 2024-03-20T09:35:35.377 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:35.377 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:35.377 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:35:35.377 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:35:35.377 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:35:35.377 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 189289 2024-03-20T09:35:35.377 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.377 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:35.378 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.378 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:35.378 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.543959+0000", 2024-03-20T09:35:35.378 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.378 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.378 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.378 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:35.378 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.543959+0000", 2024-03-20T09:35:35.378 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.378 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.378 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.378 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:35.378 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.543960+0000", 2024-03-20T09:35:35.378 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.201e-06 2024-03-20T09:35:35.379 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.379 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.379 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:35.379 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.543965+0000", 2024-03-20T09:35:35.379 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2680000000000002e-06 2024-03-20T09:35:35.379 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.379 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.379 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:35.379 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.543978+0000", 2024-03-20T09:35:35.379 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2712e-05 2024-03-20T09:35:35.379 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.379 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.379 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:35.379 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.543985+0000", 2024-03-20T09:35:35.380 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.2570000000000001e-06 2024-03-20T09:35:35.380 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.380 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.380 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:35.380 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.546886+0000", 2024-03-20T09:35:35.380 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00290033 2024-03-20T09:35:35.380 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.380 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.380 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:35:35.380 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.546994+0000", 2024-03-20T09:35:35.380 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00010830799999999999 2024-03-20T09:35:35.380 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.380 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.380 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 3,6", 2024-03-20T09:35:35.380 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.547072+0000", 2024-03-20T09:35:35.381 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.8065999999999994e-05 2024-03-20T09:35:35.381 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.381 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.381 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:35.381 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.551112+0000", 2024-03-20T09:35:35.381 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0040395320000000002 2024-03-20T09:35:35.381 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.381 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.381 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:35:35.381 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.551125+0000", 2024-03-20T09:35:35.381 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3127e-05 2024-03-20T09:35:35.381 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.381 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.381 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:35.381 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.705002+0000", 2024-03-20T09:35:35.382 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.15387756499999999 2024-03-20T09:35:35.382 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.382 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.382 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:35:35.382 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.705051+0000", 2024-03-20T09:35:35.382 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9084e-05 2024-03-20T09:35:35.382 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.382 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.382 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:35.382 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.705090+0000", 2024-03-20T09:35:35.382 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.8343000000000002e-05 2024-03-20T09:35:35.382 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.382 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:35.382 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.383 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.383 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.383 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:189305 7.1 7:80689ebf:::benchmark_data_smithi122_187077_object62694:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e926)", 2024-03-20T09:35:35.383 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:03.574118+0000", 2024-03-20T09:35:35.383 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 91.692862536999996, 2024-03-20T09:35:35.383 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.133746842, 2024-03-20T09:35:35.383 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:35.383 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:35.383 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:35:35.383 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:35:35.383 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:35:35.383 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 189305 2024-03-20T09:35:35.383 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.383 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:35.383 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.384 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:35.384 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.574118+0000", 2024-03-20T09:35:35.384 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.384 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.384 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.384 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:35.384 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.574118+0000", 2024-03-20T09:35:35.384 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.384 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.384 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.384 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:35.384 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.574119+0000", 2024-03-20T09:35:35.384 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.079e-06 2024-03-20T09:35:35.384 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.385 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.385 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:35.385 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.574125+0000", 2024-03-20T09:35:35.385 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5470000000000004e-06 2024-03-20T09:35:35.385 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.385 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.385 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:35.385 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.574136+0000", 2024-03-20T09:35:35.385 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1233000000000001e-05 2024-03-20T09:35:35.385 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.385 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.385 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:35.385 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.574143+0000", 2024-03-20T09:35:35.385 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.0219999999999999e-06 2024-03-20T09:35:35.386 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.386 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.386 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:35.386 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.574203+0000", 2024-03-20T09:35:35.386 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9861999999999997e-05 2024-03-20T09:35:35.386 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.386 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.386 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:35:35.386 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.574330+0000", 2024-03-20T09:35:35.386 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012700600000000001 2024-03-20T09:35:35.386 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.386 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.386 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 3,6", 2024-03-20T09:35:35.386 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.574462+0000", 2024-03-20T09:35:35.387 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013200899999999999 2024-03-20T09:35:35.387 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.387 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.387 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:35:35.387 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.578382+0000", 2024-03-20T09:35:35.387 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0039203329999999998 2024-03-20T09:35:35.387 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.387 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.387 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:35.387 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.579292+0000", 2024-03-20T09:35:35.387 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00090975099999999998 2024-03-20T09:35:35.387 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.387 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.387 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:35.387 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.707764+0000", 2024-03-20T09:35:35.388 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.128472685 2024-03-20T09:35:35.388 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.388 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.388 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:35:35.388 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.707819+0000", 2024-03-20T09:35:35.388 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4593999999999999e-05 2024-03-20T09:35:35.388 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.388 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.388 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:35.388 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.707865+0000", 2024-03-20T09:35:35.388 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5720999999999998e-05 2024-03-20T09:35:35.388 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.388 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:35.388 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.389 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.389 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.389 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:189310 7.1 7:822f447c:::benchmark_data_smithi122_187077_object62699:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e926)", 2024-03-20T09:35:35.389 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:03.581036+0000", 2024-03-20T09:35:35.389 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 91.685944774000006, 2024-03-20T09:35:35.389 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.12694814300000001, 2024-03-20T09:35:35.389 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:35.389 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:35.389 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:35:35.389 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:35:35.389 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:35:35.389 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 189310 2024-03-20T09:35:35.389 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.389 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:35.389 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.390 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:35.390 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.581036+0000", 2024-03-20T09:35:35.390 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.390 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.390 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.390 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:35.390 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.581036+0000", 2024-03-20T09:35:35.390 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:35.390 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.390 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.390 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:35.390 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.581037+0000", 2024-03-20T09:35:35.390 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.702e-06 2024-03-20T09:35:35.390 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.391 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.391 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:35.391 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.581047+0000", 2024-03-20T09:35:35.391 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.9289999999999993e-06 2024-03-20T09:35:35.391 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.391 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.391 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:35.391 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.581060+0000", 2024-03-20T09:35:35.391 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2347000000000001e-05 2024-03-20T09:35:35.391 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.391 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.391 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:35.391 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.581068+0000", 2024-03-20T09:35:35.391 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.4600000000000003e-06 2024-03-20T09:35:35.392 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.392 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.392 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:35.392 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.581125+0000", 2024-03-20T09:35:35.392 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7197999999999997e-05 2024-03-20T09:35:35.392 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.392 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.392 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:35:35.392 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.581218+0000", 2024-03-20T09:35:35.392 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.3029999999999995e-05 2024-03-20T09:35:35.392 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.392 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.392 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 3,6", 2024-03-20T09:35:35.392 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.581297+0000", 2024-03-20T09:35:35.392 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.8955000000000006e-05 2024-03-20T09:35:35.393 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.393 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.393 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:35:35.393 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.586239+0000", 2024-03-20T09:35:35.393 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0049416750000000004 2024-03-20T09:35:35.393 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.393 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.393 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:35.393 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.586635+0000", 2024-03-20T09:35:35.393 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00039621699999999998 2024-03-20T09:35:35.393 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.393 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.393 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:35:35.393 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.707903+0000", 2024-03-20T09:35:35.394 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.121268321 2024-03-20T09:35:35.394 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.394 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.394 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:35:35.394 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.707948+0000", 2024-03-20T09:35:35.394 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4323000000000002e-05 2024-03-20T09:35:35.394 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:35.394 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:35.394 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:35.394 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:03.707984+0000", 2024-03-20T09:35:35.394 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.5985999999999997e-05 2024-03-20T09:35:35.394 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.394 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:35.394 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.395 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:35.395 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:35.395 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:35:35.395 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:35:35.428 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:35.449 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:35:35.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:35.453 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:35:35.554 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:35:35.585 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:35.607 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:35:35.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:35.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:35:35.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:35 smithi122 ceph-mon[73495]: pgmap v2337: 57 pgs: 3 active+remapped+backfill_toofull, 54 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 28 op/s 2024-03-20T09:35:35.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:35 smithi122 ceph-mon[73495]: 1.5 scrub starts 2024-03-20T09:35:35.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:35 smithi122 ceph-mon[73495]: 1.5 scrub ok 2024-03-20T09:35:35.719 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:35:35.752 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:35 smithi026 ceph-mon[68309]: pgmap v2337: 57 pgs: 3 active+remapped+backfill_toofull, 54 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 28 op/s 2024-03-20T09:35:35.752 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:35 smithi026 ceph-mon[68309]: 1.5 scrub starts 2024-03-20T09:35:35.753 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:35 smithi026 ceph-mon[68309]: 1.5 scrub ok 2024-03-20T09:35:35.754 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:35.882 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:35:35.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:35.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:35:35.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:35 smithi115 ceph-mon[72476]: pgmap v2337: 57 pgs: 3 active+remapped+backfill_toofull, 54 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 28 op/s 2024-03-20T09:35:35.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:35 smithi115 ceph-mon[72476]: 1.5 scrub starts 2024-03-20T09:35:35.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:35 smithi115 ceph-mon[72476]: 1.5 scrub ok 2024-03-20T09:35:35.926 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-20T09:35:35.987 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:35:36.019 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:36.041 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:35:36.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:36.044 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:35:36.146 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:35:36.176 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:36.186 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:35:36.200 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:35:36.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:36.203 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:35:36.304 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:35:36.331 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:36.404 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:35:36.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:36.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:35:36.492 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:35:36.507 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:35:36.534 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:36.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:36 smithi026 ceph-mon[68309]: 7.d scrub starts 2024-03-20T09:35:36.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:36 smithi026 ceph-mon[68309]: 7.d scrub ok 2024-03-20T09:35:36.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:36 smithi115 ceph-mon[72476]: 7.d scrub starts 2024-03-20T09:35:36.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:36 smithi115 ceph-mon[72476]: 7.d scrub ok 2024-03-20T09:35:36.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:36 smithi122 ceph-mon[73495]: 7.d scrub starts 2024-03-20T09:35:36.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:36 smithi122 ceph-mon[73495]: 7.d scrub ok 2024-03-20T09:35:37.325 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:35:37.345 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:35:37.366 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:35:37.386 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:35:37.407 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:35:37.428 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:35:37.448 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:35:37.468 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:35:37.489 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:35:37.508 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:35:37.528 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:35:37.548 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:35:37.550 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:35:37.550 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:35:37.550 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:35:37.550 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:35:37.550 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:35:37.550 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:35:37.551 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:35:37.551 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:35:37.551 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:35:37.551 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:35:37.551 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:35:37.551 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:35:37.762 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:35:37.766 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:37.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:35:37.868 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:35:37.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:37 smithi115 ceph-mon[72476]: 1.1b scrub starts 2024-03-20T09:35:37.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:37 smithi115 ceph-mon[72476]: 1.1b scrub ok 2024-03-20T09:35:37.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:37 smithi115 ceph-mon[72476]: pgmap v2338: 57 pgs: 3 active+remapped+backfill_toofull, 54 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:35:37.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:37 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:35:37.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:37 smithi115 ceph-mon[72476]: 1.14 deep-scrub starts 2024-03-20T09:35:37.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:37 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2776586666' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:35:37.898 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:37.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:37 smithi122 ceph-mon[73495]: 1.1b scrub starts 2024-03-20T09:35:37.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:37 smithi122 ceph-mon[73495]: 1.1b scrub ok 2024-03-20T09:35:37.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:37 smithi122 ceph-mon[73495]: pgmap v2338: 57 pgs: 3 active+remapped+backfill_toofull, 54 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:35:37.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:37 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:35:37.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:37 smithi122 ceph-mon[73495]: 1.14 deep-scrub starts 2024-03-20T09:35:37.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:37 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2776586666' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:35:37.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:37 smithi026 ceph-mon[68309]: 1.1b scrub starts 2024-03-20T09:35:37.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:37 smithi026 ceph-mon[68309]: 1.1b scrub ok 2024-03-20T09:35:37.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:37 smithi026 ceph-mon[68309]: pgmap v2338: 57 pgs: 3 active+remapped+backfill_toofull, 54 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:35:37.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:37 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:35:37.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:37 smithi026 ceph-mon[68309]: 1.14 deep-scrub starts 2024-03-20T09:35:37.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:37 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2776586666' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:35:37.998 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:35:38.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:38.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:35:38.102 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:35:38.133 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:38.152 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:35:38.155 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:38.155 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:35:38.256 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:35:38.285 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:38.304 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:35:38.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:38.308 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:35:38.410 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:35:38.439 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:38.457 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:35:38.457 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:35:38.459 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:35:38.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:38.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:35:38.491 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:35:38.564 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:35:38.593 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:38.845 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:35:38.847 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:38 smithi026 ceph-mon[68309]: 1.14 deep-scrub ok 2024-03-20T09:35:38.848 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:38 smithi026 ceph-mon[68309]: 1.d scrub starts 2024-03-20T09:35:38.848 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:38 smithi026 ceph-mon[68309]: 1.d scrub ok 2024-03-20T09:35:38.848 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:38 smithi026 ceph-mon[68309]: osdmap e969: 12 total, 12 up, 12 in 2024-03-20T09:35:38.848 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:38 smithi026 ceph-mon[68309]: pgmap v2340: 57 pgs: 3 active+remapped+backfill_toofull, 54 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:35:38.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:38.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:35:38.892 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:35:38.950 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:35:38.983 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:39.005 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:35:39.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:39.008 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:35:39.059 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:38 smithi115 ceph-mon[72476]: 1.14 deep-scrub ok 2024-03-20T09:35:39.059 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:38 smithi115 ceph-mon[72476]: 1.d scrub starts 2024-03-20T09:35:39.059 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:38 smithi115 ceph-mon[72476]: 1.d scrub ok 2024-03-20T09:35:39.059 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:38 smithi115 ceph-mon[72476]: osdmap e969: 12 total, 12 up, 12 in 2024-03-20T09:35:39.060 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:38 smithi115 ceph-mon[72476]: pgmap v2340: 57 pgs: 3 active+remapped+backfill_toofull, 54 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:35:39.084 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-20T09:35:39.109 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:35:39.139 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:38 smithi122 ceph-mon[73495]: 1.14 deep-scrub ok 2024-03-20T09:35:39.140 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:38 smithi122 ceph-mon[73495]: 1.d scrub starts 2024-03-20T09:35:39.140 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:38 smithi122 ceph-mon[73495]: 1.d scrub ok 2024-03-20T09:35:39.140 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:38 smithi122 ceph-mon[73495]: osdmap e969: 12 total, 12 up, 12 in 2024-03-20T09:35:39.140 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:38 smithi122 ceph-mon[73495]: pgmap v2340: 57 pgs: 3 active+remapped+backfill_toofull, 54 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:35:39.142 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:39.163 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:35:39.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:39.166 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:35:39.267 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:35:39.299 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:39.317 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:35:39.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:39.320 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:35:39.346 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:35:39.420 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:35:39.447 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:39.570 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:35:39.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:39.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:35:39.676 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:35:39.709 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:39.730 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:35:39.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:39.733 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:35:39.834 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:35:39.864 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:39.884 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:35:39.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:39.889 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:35:39.990 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:35:40.017 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:39 smithi122 ceph-mon[73495]: 1.1a scrub starts 2024-03-20T09:35:40.017 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:39 smithi122 ceph-mon[73495]: 1.1a scrub ok 2024-03-20T09:35:40.017 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:39 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:35:40.017 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:39 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:35:40.018 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:40.038 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:35:40.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:40.041 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:35:40.142 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:35:40.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:39 smithi115 ceph-mon[72476]: 1.1a scrub starts 2024-03-20T09:35:40.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:39 smithi115 ceph-mon[72476]: 1.1a scrub ok 2024-03-20T09:35:40.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:39 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:35:40.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:39 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:35:40.171 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:39 smithi026 ceph-mon[68309]: 1.1a scrub starts 2024-03-20T09:35:40.171 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:39 smithi026 ceph-mon[68309]: 1.1a scrub ok 2024-03-20T09:35:40.172 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:39 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:35:40.172 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:39 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:35:40.174 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:40.199 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 8, 11, 3, 4, 9, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 11, 5, 10] 2024-03-20T09:35:40.200 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:35:40.200 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:35:40.304 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:35:40.307 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:40.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:35:40.348 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:35:40.408 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:35:40.439 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:40.461 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:35:40.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:40.464 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:35:40.565 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:35:40.593 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:40.793 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:35:40.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:40.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:35:40.815 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'unique_pool_5' removed 2024-03-20T09:35:40.897 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:35:40.927 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:41.025 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:35:41.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:41.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:35:41.066 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:40 smithi026 ceph-mon[68309]: 1.8 scrub starts 2024-03-20T09:35:41.067 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:40 smithi026 ceph-mon[68309]: 1.8 scrub ok 2024-03-20T09:35:41.067 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:40 smithi026 ceph-mon[68309]: osdmap e970: 12 total, 12 up, 12 in 2024-03-20T09:35:41.067 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:40 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2437843011' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-20T09:35:41.067 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:40 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:35:41.067 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:40 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:35:41.067 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:40 smithi026 ceph-mon[68309]: pgmap v2342: 57 pgs: 3 active+remapped+backfill_toofull, 54 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:35:41.077 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:35:41.077 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:35:41.099 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:40 smithi115 ceph-mon[72476]: 1.8 scrub starts 2024-03-20T09:35:41.099 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:40 smithi115 ceph-mon[72476]: 1.8 scrub ok 2024-03-20T09:35:41.099 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:40 smithi115 ceph-mon[72476]: osdmap e970: 12 total, 12 up, 12 in 2024-03-20T09:35:41.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:40 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2437843011' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-20T09:35:41.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:40 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:35:41.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:40 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:35:41.100 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:40 smithi115 ceph-mon[72476]: pgmap v2342: 57 pgs: 3 active+remapped+backfill_toofull, 54 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:35:41.130 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:35:41.158 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:40 smithi122 ceph-mon[73495]: 1.8 scrub starts 2024-03-20T09:35:41.158 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:40 smithi122 ceph-mon[73495]: 1.8 scrub ok 2024-03-20T09:35:41.158 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:40 smithi122 ceph-mon[73495]: osdmap e970: 12 total, 12 up, 12 in 2024-03-20T09:35:41.158 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:40 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2437843011' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-20T09:35:41.159 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:40 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:35:41.159 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:40 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:35:41.159 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:40 smithi122 ceph-mon[73495]: pgmap v2342: 57 pgs: 3 active+remapped+backfill_toofull, 54 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:35:41.160 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:41.179 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:35:41.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:41.182 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:35:41.284 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:35:41.314 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:41.333 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:35:41.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:41.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:35:41.436 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:35:41.463 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:41.864 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-20T09:35:41.912 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:35:41.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:41.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:35:42.016 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:35:42.041 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:42.060 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:35:42.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:42.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:35:42.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:41 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2437843011' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-03-20T09:35:42.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:41 smithi115 ceph-mon[72476]: osdmap e971: 12 total, 12 up, 12 in 2024-03-20T09:35:42.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:41 smithi115 ceph-mon[72476]: 1.7 scrub starts 2024-03-20T09:35:42.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:41 smithi115 ceph-mon[72476]: 1.7 scrub ok 2024-03-20T09:35:42.164 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:35:42.168 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:35:42.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:41 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2437843011' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-03-20T09:35:42.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:41 smithi122 ceph-mon[73495]: osdmap e971: 12 total, 12 up, 12 in 2024-03-20T09:35:42.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:41 smithi122 ceph-mon[73495]: 1.7 scrub starts 2024-03-20T09:35:42.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:41 smithi122 ceph-mon[73495]: 1.7 scrub ok 2024-03-20T09:35:42.194 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:41 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2437843011' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-03-20T09:35:42.194 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:41 smithi026 ceph-mon[68309]: osdmap e971: 12 total, 12 up, 12 in 2024-03-20T09:35:42.194 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:41 smithi026 ceph-mon[68309]: 1.7 scrub starts 2024-03-20T09:35:42.194 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:41 smithi026 ceph-mon[68309]: 1.7 scrub ok 2024-03-20T09:35:42.196 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:42.385 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-20T09:35:42.386 INFO:tasks.radosbench:Beginning radosbench... 2024-03-20T09:35:42.386 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_6 2024-03-20T09:35:42.387 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pool create unique_pool_6 16 2024-03-20T09:35:42.397 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:35:42.400 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:42.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:35:42.472 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:35:42.473 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:35:42.502 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:35:42.530 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:42.629 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:35:42.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:42.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:35:42.733 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:35:42.765 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:42.788 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:35:42.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:42.791 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:35:42.821 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:35:42.893 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:35:42.921 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:42.941 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:35:42.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:42.944 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:35:43.022 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1411957205' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:35:43.023 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: pgmap v2344: 31 pgs: 31 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:35:43.023 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {auth=27,config=2,health=12,kv=56,logm=3} crc {auth=3655719975,config=2679774077,health=867466929,kv=2164098447,logm=4026747588}) 2024-03-20T09:35:43.023 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=972085614}) 2024-03-20T09:35:43.023 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2479610167}) 2024-03-20T09:35:43.023 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3570635113}) 2024-03-20T09:35:43.023 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3562604419}) 2024-03-20T09:35:43.023 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2342207383}) 2024-03-20T09:35:43.023 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2224917327}) 2024-03-20T09:35:43.023 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2913992118}) 2024-03-20T09:35:43.023 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=803174728}) 2024-03-20T09:35:43.023 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3209611664}) 2024-03-20T09:35:43.023 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3978782549}) 2024-03-20T09:35:43.024 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=449492164}) 2024-03-20T09:35:43.024 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3566244103}) 2024-03-20T09:35:43.024 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3945872326}) 2024-03-20T09:35:43.024 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2536795037}) 2024-03-20T09:35:43.024 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=605856714}) 2024-03-20T09:35:43.024 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=508526024}) 2024-03-20T09:35:43.024 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2458693099}) 2024-03-20T09:35:43.024 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2572042000}) 2024-03-20T09:35:43.024 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3730062582}) 2024-03-20T09:35:43.024 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3119076368}) 2024-03-20T09:35:43.024 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3254490068}) 2024-03-20T09:35:43.024 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1851724166}) 2024-03-20T09:35:43.024 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2461752356}) 2024-03-20T09:35:43.024 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2006342876}) 2024-03-20T09:35:43.025 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3299044232}) 2024-03-20T09:35:43.025 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=354660859}) 2024-03-20T09:35:43.025 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3724603789}) 2024-03-20T09:35:43.025 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2089420935}) 2024-03-20T09:35:43.025 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=536309508}) 2024-03-20T09:35:43.025 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=608923790}) 2024-03-20T09:35:43.025 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1340658296}) 2024-03-20T09:35:43.025 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1620317400}) 2024-03-20T09:35:43.025 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4215085782}) 2024-03-20T09:35:43.025 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=891735571}) 2024-03-20T09:35:43.025 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3382134428}) 2024-03-20T09:35:43.025 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=532716548}) 2024-03-20T09:35:43.025 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2676581289}) 2024-03-20T09:35:43.025 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3735539872}) 2024-03-20T09:35:43.026 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1941609701}) 2024-03-20T09:35:43.026 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1894776810}) 2024-03-20T09:35:43.026 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1861650426}) 2024-03-20T09:35:43.026 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2867951368}) 2024-03-20T09:35:43.026 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1351397202}) 2024-03-20T09:35:43.026 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1072803563}) 2024-03-20T09:35:43.026 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1146185805}) 2024-03-20T09:35:43.026 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3821673284}) 2024-03-20T09:35:43.026 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077556405}) 2024-03-20T09:35:43.026 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3388018269}) 2024-03-20T09:35:43.026 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1227798906}) 2024-03-20T09:35:43.026 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2264504858}) 2024-03-20T09:35:43.026 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3567298663}) 2024-03-20T09:35:43.026 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1791030274}) 2024-03-20T09:35:43.027 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2626724951}) 2024-03-20T09:35:43.027 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2882505303}) 2024-03-20T09:35:43.027 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=663775761}) 2024-03-20T09:35:43.027 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=210858903}) 2024-03-20T09:35:43.027 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2696256894}) 2024-03-20T09:35:43.027 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=131554386}) 2024-03-20T09:35:43.027 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2837931978}) 2024-03-20T09:35:43.027 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4256544408}) 2024-03-20T09:35:43.027 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1789075588}) 2024-03-20T09:35:43.027 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4239143097}) 2024-03-20T09:35:43.027 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=165657729}) 2024-03-20T09:35:43.027 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3897887933}) 2024-03-20T09:35:43.027 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=585292734}) 2024-03-20T09:35:43.027 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2721361516}) 2024-03-20T09:35:43.027 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2216028291}) 2024-03-20T09:35:43.028 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1493566249}) 2024-03-20T09:35:43.028 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=662273485}) 2024-03-20T09:35:43.028 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3318290998}) 2024-03-20T09:35:43.028 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4069209546}) 2024-03-20T09:35:43.028 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2631195685}) 2024-03-20T09:35:43.028 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=747294676}) 2024-03-20T09:35:43.028 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=491048198}) 2024-03-20T09:35:43.028 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1930336072}) 2024-03-20T09:35:43.028 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1758942520}) 2024-03-20T09:35:43.028 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4172856492}) 2024-03-20T09:35:43.028 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1648339271}) 2024-03-20T09:35:43.028 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=424544355}) 2024-03-20T09:35:43.028 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3311646959}) 2024-03-20T09:35:43.028 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=808867416}) 2024-03-20T09:35:43.029 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1271279339}) 2024-03-20T09:35:43.029 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=18,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=48} crc {logm=1750455586,mdsmap=3984425268,mgr=427508465,mgr_command_descs=3831600018,mgr_metadata=2702834764,mgrstat=1782472068,mon_config_key=1690536913}) 2024-03-20T09:35:43.029 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4198601519}) 2024-03-20T09:35:43.029 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1336854569}) 2024-03-20T09:35:43.029 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=33,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=49} crc {mon_config_key=3986612187,monmap=3481182261,osd_metadata=222046520,osd_pg_creating=96506557,osdmap=806984109}) 2024-03-20T09:35:43.029 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4099577489}) 2024-03-20T09:35:43.029 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:42 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=28} crc {osdmap=3359675718}) 2024-03-20T09:35:43.042 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1411957205' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:35:43.042 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: pgmap v2344: 31 pgs: 31 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:35:43.042 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {auth=27,config=2,health=12,kv=56,logm=3} crc {auth=3655719975,config=2679774077,health=867466929,kv=2164098447,logm=4026747588}) 2024-03-20T09:35:43.042 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=972085614}) 2024-03-20T09:35:43.042 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2479610167}) 2024-03-20T09:35:43.042 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3570635113}) 2024-03-20T09:35:43.042 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3562604419}) 2024-03-20T09:35:43.042 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2342207383}) 2024-03-20T09:35:43.042 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2224917327}) 2024-03-20T09:35:43.042 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2913992118}) 2024-03-20T09:35:43.042 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=803174728}) 2024-03-20T09:35:43.042 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3209611664}) 2024-03-20T09:35:43.043 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3978782549}) 2024-03-20T09:35:43.043 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=449492164}) 2024-03-20T09:35:43.043 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3566244103}) 2024-03-20T09:35:43.043 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3945872326}) 2024-03-20T09:35:43.043 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2536795037}) 2024-03-20T09:35:43.043 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=605856714}) 2024-03-20T09:35:43.043 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=508526024}) 2024-03-20T09:35:43.043 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2458693099}) 2024-03-20T09:35:43.043 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2572042000}) 2024-03-20T09:35:43.043 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3730062582}) 2024-03-20T09:35:43.043 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3119076368}) 2024-03-20T09:35:43.043 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3254490068}) 2024-03-20T09:35:43.043 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1851724166}) 2024-03-20T09:35:43.043 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2461752356}) 2024-03-20T09:35:43.044 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2006342876}) 2024-03-20T09:35:43.044 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3299044232}) 2024-03-20T09:35:43.044 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=354660859}) 2024-03-20T09:35:43.044 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3724603789}) 2024-03-20T09:35:43.044 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2089420935}) 2024-03-20T09:35:43.044 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=536309508}) 2024-03-20T09:35:43.044 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=608923790}) 2024-03-20T09:35:43.044 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1340658296}) 2024-03-20T09:35:43.044 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1620317400}) 2024-03-20T09:35:43.044 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4215085782}) 2024-03-20T09:35:43.044 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=891735571}) 2024-03-20T09:35:43.044 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3382134428}) 2024-03-20T09:35:43.044 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=532716548}) 2024-03-20T09:35:43.044 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2676581289}) 2024-03-20T09:35:43.044 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3735539872}) 2024-03-20T09:35:43.045 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1941609701}) 2024-03-20T09:35:43.045 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1894776810}) 2024-03-20T09:35:43.045 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1861650426}) 2024-03-20T09:35:43.045 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2867951368}) 2024-03-20T09:35:43.045 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1351397202}) 2024-03-20T09:35:43.045 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1072803563}) 2024-03-20T09:35:43.045 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1146185805}) 2024-03-20T09:35:43.045 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3821673284}) 2024-03-20T09:35:43.045 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077556405}) 2024-03-20T09:35:43.045 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3388018269}) 2024-03-20T09:35:43.045 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1227798906}) 2024-03-20T09:35:43.045 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2264504858}) 2024-03-20T09:35:43.045 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3567298663}) 2024-03-20T09:35:43.045 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1791030274}) 2024-03-20T09:35:43.046 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2626724951}) 2024-03-20T09:35:43.046 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2882505303}) 2024-03-20T09:35:43.046 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=663775761}) 2024-03-20T09:35:43.046 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=210858903}) 2024-03-20T09:35:43.046 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2696256894}) 2024-03-20T09:35:43.046 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=131554386}) 2024-03-20T09:35:43.046 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2837931978}) 2024-03-20T09:35:43.046 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4256544408}) 2024-03-20T09:35:43.046 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1789075588}) 2024-03-20T09:35:43.046 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4239143097}) 2024-03-20T09:35:43.046 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=165657729}) 2024-03-20T09:35:43.046 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3897887933}) 2024-03-20T09:35:43.046 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=585292734}) 2024-03-20T09:35:43.046 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2721361516}) 2024-03-20T09:35:43.046 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2216028291}) 2024-03-20T09:35:43.047 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1493566249}) 2024-03-20T09:35:43.047 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=662273485}) 2024-03-20T09:35:43.047 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3318290998}) 2024-03-20T09:35:43.047 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4069209546}) 2024-03-20T09:35:43.047 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2631195685}) 2024-03-20T09:35:43.047 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=747294676}) 2024-03-20T09:35:43.047 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=491048198}) 2024-03-20T09:35:43.047 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1930336072}) 2024-03-20T09:35:43.047 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1758942520}) 2024-03-20T09:35:43.047 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4172856492}) 2024-03-20T09:35:43.047 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1648339271}) 2024-03-20T09:35:43.047 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=424544355}) 2024-03-20T09:35:43.047 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3311646959}) 2024-03-20T09:35:43.047 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=808867416}) 2024-03-20T09:35:43.048 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1271279339}) 2024-03-20T09:35:43.048 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=18,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=48} crc {logm=1750455586,mdsmap=3984425268,mgr=427508465,mgr_command_descs=3831600018,mgr_metadata=2702834764,mgrstat=1782472068,mon_config_key=1690536913}) 2024-03-20T09:35:43.048 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4198601519}) 2024-03-20T09:35:43.048 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1336854569}) 2024-03-20T09:35:43.048 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=33,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=49} crc {mon_config_key=3986612187,monmap=3481182261,osd_metadata=222046520,osd_pg_creating=96506557,osdmap=806984109}) 2024-03-20T09:35:43.048 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4099577489}) 2024-03-20T09:35:43.048 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:42 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=28} crc {osdmap=3359675718}) 2024-03-20T09:35:43.048 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:35:43.075 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:43.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1411957205' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:35:43.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: pgmap v2344: 31 pgs: 31 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:35:43.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {auth=27,config=2,health=12,kv=56,logm=3} crc {auth=3655719975,config=2679774077,health=867466929,kv=2164098447,logm=4026747588}) 2024-03-20T09:35:43.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=972085614}) 2024-03-20T09:35:43.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2479610167}) 2024-03-20T09:35:43.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3570635113}) 2024-03-20T09:35:43.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3562604419}) 2024-03-20T09:35:43.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2342207383}) 2024-03-20T09:35:43.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2224917327}) 2024-03-20T09:35:43.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2913992118}) 2024-03-20T09:35:43.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=803174728}) 2024-03-20T09:35:43.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3209611664}) 2024-03-20T09:35:43.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3978782549}) 2024-03-20T09:35:43.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=449492164}) 2024-03-20T09:35:43.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3566244103}) 2024-03-20T09:35:43.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3945872326}) 2024-03-20T09:35:43.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2536795037}) 2024-03-20T09:35:43.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=605856714}) 2024-03-20T09:35:43.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=508526024}) 2024-03-20T09:35:43.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2458693099}) 2024-03-20T09:35:43.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2572042000}) 2024-03-20T09:35:43.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3730062582}) 2024-03-20T09:35:43.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3119076368}) 2024-03-20T09:35:43.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3254490068}) 2024-03-20T09:35:43.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1851724166}) 2024-03-20T09:35:43.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2461752356}) 2024-03-20T09:35:43.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2006342876}) 2024-03-20T09:35:43.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3299044232}) 2024-03-20T09:35:43.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=354660859}) 2024-03-20T09:35:43.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3724603789}) 2024-03-20T09:35:43.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2089420935}) 2024-03-20T09:35:43.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=536309508}) 2024-03-20T09:35:43.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=608923790}) 2024-03-20T09:35:43.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1340658296}) 2024-03-20T09:35:43.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1620317400}) 2024-03-20T09:35:43.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4215085782}) 2024-03-20T09:35:43.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=891735571}) 2024-03-20T09:35:43.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3382134428}) 2024-03-20T09:35:43.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=532716548}) 2024-03-20T09:35:43.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2676581289}) 2024-03-20T09:35:43.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3735539872}) 2024-03-20T09:35:43.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1941609701}) 2024-03-20T09:35:43.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1894776810}) 2024-03-20T09:35:43.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1861650426}) 2024-03-20T09:35:43.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2867951368}) 2024-03-20T09:35:43.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1351397202}) 2024-03-20T09:35:43.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1072803563}) 2024-03-20T09:35:43.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1146185805}) 2024-03-20T09:35:43.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3821673284}) 2024-03-20T09:35:43.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1077556405}) 2024-03-20T09:35:43.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3388018269}) 2024-03-20T09:35:43.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1227798906}) 2024-03-20T09:35:43.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2264504858}) 2024-03-20T09:35:43.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3567298663}) 2024-03-20T09:35:43.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1791030274}) 2024-03-20T09:35:43.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2626724951}) 2024-03-20T09:35:43.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2882505303}) 2024-03-20T09:35:43.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=663775761}) 2024-03-20T09:35:43.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=210858903}) 2024-03-20T09:35:43.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2696256894}) 2024-03-20T09:35:43.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=131554386}) 2024-03-20T09:35:43.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2837931978}) 2024-03-20T09:35:43.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4256544408}) 2024-03-20T09:35:43.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1789075588}) 2024-03-20T09:35:43.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4239143097}) 2024-03-20T09:35:43.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=165657729}) 2024-03-20T09:35:43.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3897887933}) 2024-03-20T09:35:43.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=585292734}) 2024-03-20T09:35:43.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2721361516}) 2024-03-20T09:35:43.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2216028291}) 2024-03-20T09:35:43.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1493566249}) 2024-03-20T09:35:43.193 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=662273485}) 2024-03-20T09:35:43.193 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3318290998}) 2024-03-20T09:35:43.193 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4069209546}) 2024-03-20T09:35:43.193 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2631195685}) 2024-03-20T09:35:43.193 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=747294676}) 2024-03-20T09:35:43.193 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=491048198}) 2024-03-20T09:35:43.193 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1930336072}) 2024-03-20T09:35:43.193 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1758942520}) 2024-03-20T09:35:43.194 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4172856492}) 2024-03-20T09:35:43.194 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1648339271}) 2024-03-20T09:35:43.194 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=424544355}) 2024-03-20T09:35:43.194 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3311646959}) 2024-03-20T09:35:43.194 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=808867416}) 2024-03-20T09:35:43.194 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1271279339}) 2024-03-20T09:35:43.194 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=18,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=48} crc {logm=1750455586,mdsmap=3984425268,mgr=427508465,mgr_command_descs=3831600018,mgr_metadata=2702834764,mgrstat=1782472068,mon_config_key=1690536913}) 2024-03-20T09:35:43.195 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4198601519}) 2024-03-20T09:35:43.195 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1336854569}) 2024-03-20T09:35:43.195 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=33,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=49} crc {mon_config_key=3986612187,monmap=3481182261,osd_metadata=222046520,osd_pg_creating=96506557,osdmap=806984109}) 2024-03-20T09:35:43.195 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4099577489}) 2024-03-20T09:35:43.195 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:42 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=28} crc {osdmap=3359675718}) 2024-03-20T09:35:43.424 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:35:43.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:43.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:35:43.528 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:35:43.555 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:43.654 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:35:43.656 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:43.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:35:43.758 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:35:43.786 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:43.805 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:35:43.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:43.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:35:43.908 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:35:43.933 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:43 smithi115 ceph-mon[72476]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull) 2024-03-20T09:35:43.933 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:43 smithi115 ceph-mon[72476]: Cluster is now healthy 2024-03-20T09:35:43.933 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:43 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:35:43.933 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:43 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1411957205' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:35:43.933 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:43 smithi115 ceph-mon[72476]: osdmap e972: 12 total, 12 up, 12 in 2024-03-20T09:35:43.933 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:43 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:35:43.934 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:43 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:35:43.935 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:43.955 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:35:43.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:43.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:35:44.042 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:35:44.042 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:35:44.059 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:35:44.085 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:44.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:43 smithi122 ceph-mon[73495]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull) 2024-03-20T09:35:44.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:43 smithi122 ceph-mon[73495]: Cluster is now healthy 2024-03-20T09:35:44.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:43 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:35:44.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:43 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1411957205' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:35:44.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:43 smithi122 ceph-mon[73495]: osdmap e972: 12 total, 12 up, 12 in 2024-03-20T09:35:44.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:43 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:35:44.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:43 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:35:44.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:43 smithi026 ceph-mon[68309]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull) 2024-03-20T09:35:44.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:43 smithi026 ceph-mon[68309]: Cluster is now healthy 2024-03-20T09:35:44.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:43 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:35:44.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:43 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1411957205' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:35:44.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:43 smithi026 ceph-mon[68309]: osdmap e972: 12 total, 12 up, 12 in 2024-03-20T09:35:44.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:43 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:35:44.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:43 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:35:44.334 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:35:44.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:44.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:35:44.438 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:35:44.473 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:44.789 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-20T09:35:45.051 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:35:45.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:44 smithi115 ceph-mon[72476]: pgmap v2346: 31 pgs: 31 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:35:45.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:44 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:35:45.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:44 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:35:45.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:44 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:35:45.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:44 smithi122 ceph-mon[73495]: pgmap v2346: 31 pgs: 31 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:35:45.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:44 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:35:45.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:44 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:35:45.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:44 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:35:45.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:44 smithi026 ceph-mon[68309]: pgmap v2346: 31 pgs: 31 active+clean; 577 KiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:35:45.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:44 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:35:45.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:44 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:35:45.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:44 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:35:45.226 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:35:45.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:45.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:35:45.329 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:35:45.362 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:45.384 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:35:45.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:45.387 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:35:45.488 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:35:45.517 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:46.267 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:35:46.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:46.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:35:46.316 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:35:46.371 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:35:46.403 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:46.424 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:35:46.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:46.428 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:35:46.529 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:35:46.558 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:46.578 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:35:46.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:46.581 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:35:46.648 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:35:46.650 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:35:46.682 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:35:46.711 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:46.730 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:35:46.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:46.733 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:35:46.791 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:35:46.791 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:35:46.834 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:35:46.861 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:46.881 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:35:46.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:46.883 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:35:46.984 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:35:47.011 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:47.029 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:35:47.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:47.032 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:35:47.134 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:35:47.165 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:47.281 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:47 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:35:47.281 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:47 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:35:47.281 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:47 smithi026 ceph-mon[68309]: osdmap e973: 12 total, 12 up, 12 in 2024-03-20T09:35:47.281 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:47 smithi026 ceph-mon[68309]: pgmap v2348: 31 pgs: 31 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:35:47.281 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:47 smithi026 ceph-mon[68309]: from='client.16968 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:35:47.378 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:47 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:35:47.378 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:47 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:35:47.378 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:47 smithi115 ceph-mon[72476]: osdmap e973: 12 total, 12 up, 12 in 2024-03-20T09:35:47.378 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:47 smithi115 ceph-mon[72476]: pgmap v2348: 31 pgs: 31 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:35:47.378 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:47 smithi115 ceph-mon[72476]: from='client.16968 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:35:47.403 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-20T09:35:47.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:47 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:35:47.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:47 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:35:47.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:47 smithi122 ceph-mon[73495]: osdmap e973: 12 total, 12 up, 12 in 2024-03-20T09:35:47.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:47 smithi122 ceph-mon[73495]: pgmap v2348: 31 pgs: 31 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:35:47.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:47 smithi122 ceph-mon[73495]: from='client.16968 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:35:47.668 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:35:47.668 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:35:47.668 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:35:47.668 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:35:47.668 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:35:47.668 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:35:47.668 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:35:47.668 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:35:47.668 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:35:47.668 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:35:47.668 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:35:47.669 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:35:47.669 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:35:47.669 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:35:47.669 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:35:47.669 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:35:47.669 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:35:47.669 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:35:47.669 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:35:47.669 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:35:47.669 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:35:47.669 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:35:47.669 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:35:47.670 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:35:47.670 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:35:47.745 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:35:48.395 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:48 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2247472259' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:35:48.628 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:48 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2247472259' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:35:48.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:48 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2247472259' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:35:48.788 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":2348,"stamp":"2024-03-20T09:35:46.523606+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4248,"ondisk_log_size":4248,"up":93,"acting":93,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":128,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":8467416,"kb_used_data":14500,"kb_used_omap":301,"kb_used_meta":8450834,"kb_avail":1116474408,"statfs":{"total":1151940427776,"available":1143269793792,"internally_reserved":0,"allocated":14848000,"data_stored":9909515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":308246,"internal_metadata":8653655018},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":128,"apply_latency_ms":128,"commit_latency_ns":128000000,"apply_latency_ns":128000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-128471,"num_write_kb":-3549633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-647,"num_bytes_recovered":-42336256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001205"},"pg_stats":[{"pgid":"1.1e","version":"20'137","reported_seq":1393,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-03-20T09:35:43.465887+0000","last_change":"2024-03-20T09:35:19.735309+0000","last_active":"2024-03-20T09:35:43.465887+0000","last_peered":"2024-03-20T09:35:43.465887+0000","last_clean":"2024-03-20T09:35:43.465887+0000","last_became_active":"2024-03-20T09:13:20.071655+0000","last_became_peered":"2024-03-20T09:13:20.071655+0000","last_unstale":"2024-03-20T09:35:43.465887+0000","last_undegraded":"2024-03-20T09:35:43.465887+0000","last_fullsized":"2024-03-20T09:35:43.465887+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:35:19.735193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:35:19.735193+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:36:33.446899+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'137","reported_seq":470,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-03-20T09:35:43.466102+0000","last_change":"2024-03-20T09:34:17.803689+0000","last_active":"2024-03-20T09:35:43.466102+0000","last_peered":"2024-03-20T09:35:43.466102+0000","last_clean":"2024-03-20T09:35:43.466102+0000","last_became_active":"2024-03-20T09:31:07.425574+0000","last_became_peered":"2024-03-20T09:31:07.425574+0000","last_unstale":"2024-03-20T09:35:43.466102+0000","last_undegraded":"2024-03-20T09:35:43.466102+0000","last_fullsized":"2024-03-20T09:35:43.466102+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:34:17.803535+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:29:02.585165+0000","last_clean_scrub_stamp":"2024-03-20T09:34:17.803535+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:35:43.722650+0000","scrub_duration":162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2],"acting":[11,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"20'137","reported_seq":2072,"reported_epoch":971,"state":"active+clean","last_fresh":"2024-03-20T09:35:41.250540+0000","last_change":"2024-03-20T09:35:33.160097+0000","last_active":"2024-03-20T09:35:41.250540+0000","last_peered":"2024-03-20T09:35:41.250540+0000","last_clean":"2024-03-20T09:35:41.250540+0000","last_became_active":"2024-03-20T09:35:03.815532+0000","last_became_peered":"2024-03-20T09:35:03.815532+0000","last_unstale":"2024-03-20T09:35:41.250540+0000","last_undegraded":"2024-03-20T09:35:41.250540+0000","last_fullsized":"2024-03-20T09:35:41.250540+0000","mapping_epoch":950,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":951,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:35:33.159984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:35:33.159984+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:36:40.778517+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.19","version":"20'137","reported_seq":1249,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-03-20T09:35:43.465774+0000","last_change":"2024-03-20T09:35:23.700007+0000","last_active":"2024-03-20T09:35:43.465774+0000","last_peered":"2024-03-20T09:35:43.465774+0000","last_clean":"2024-03-20T09:35:43.465774+0000","last_became_active":"2024-03-20T09:35:19.819786+0000","last_became_peered":"2024-03-20T09:35:19.819786+0000","last_unstale":"2024-03-20T09:35:43.465774+0000","last_undegraded":"2024-03-20T09:35:43.465774+0000","last_fullsized":"2024-03-20T09:35:43.465774+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:35:23.699885+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:30:56.201554+0000","last_clean_scrub_stamp":"2024-03-20T09:35:23.699885+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:36:46.746974+0000","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"20'137","reported_seq":1539,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-03-20T09:35:43.464571+0000","last_change":"2024-03-20T09:35:22.479168+0000","last_active":"2024-03-20T09:35:43.464571+0000","last_peered":"2024-03-20T09:35:43.464571+0000","last_clean":"2024-03-20T09:35:43.464571+0000","last_became_active":"2024-03-20T09:10:35.277481+0000","last_became_peered":"2024-03-20T09:10:35.277481+0000","last_unstale":"2024-03-20T09:35:43.464571+0000","last_undegraded":"2024-03-20T09:35:43.464571+0000","last_fullsized":"2024-03-20T09:35:43.464571+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:35:22.479083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:35:22.479083+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:36:30.399248+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1b","version":"20'137","reported_seq":1900,"reported_epoch":971,"state":"active+clean","last_fresh":"2024-03-20T09:35:41.251276+0000","last_change":"2024-03-20T09:35:36.311672+0000","last_active":"2024-03-20T09:35:41.251276+0000","last_peered":"2024-03-20T09:35:41.251276+0000","last_clean":"2024-03-20T09:35:41.251276+0000","last_became_active":"2024-03-20T09:24:16.445287+0000","last_became_peered":"2024-03-20T09:24:16.445287+0000","last_unstale":"2024-03-20T09:35:41.251276+0000","last_undegraded":"2024-03-20T09:35:41.251276+0000","last_fullsized":"2024-03-20T09:35:41.251276+0000","mapping_epoch":718,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:35:36.311539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:35:36.311539+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:36:49.806892+0000","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.8","version":"20'137","reported_seq":674,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-03-20T09:35:42.829461+0000","last_change":"2024-03-20T09:35:39.043902+0000","last_active":"2024-03-20T09:35:42.829461+0000","last_peered":"2024-03-20T09:35:42.829461+0000","last_clean":"2024-03-20T09:35:42.829461+0000","last_became_active":"2024-03-20T09:35:28.000367+0000","last_became_peered":"2024-03-20T09:35:28.000367+0000","last_unstale":"2024-03-20T09:35:42.829461+0000","last_undegraded":"2024-03-20T09:35:42.829461+0000","last_fullsized":"2024-03-20T09:35:42.829461+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":966,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:35:39.043786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:35:39.043786+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:36:49.100882+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"20'137","reported_seq":2122,"reported_epoch":971,"state":"active+clean","last_fresh":"2024-03-20T09:35:41.252959+0000","last_change":"2024-03-20T09:34:30.596038+0000","last_active":"2024-03-20T09:35:41.252959+0000","last_peered":"2024-03-20T09:35:41.252959+0000","last_clean":"2024-03-20T09:35:41.252959+0000","last_became_active":"2024-03-20T09:20:25.341750+0000","last_became_peered":"2024-03-20T09:20:25.341750+0000","last_unstale":"2024-03-20T09:35:41.252959+0000","last_undegraded":"2024-03-20T09:35:41.252959+0000","last_fullsized":"2024-03-20T09:35:41.252959+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:34:30.595898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:34:30.595898+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:35:43.171408+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,2],"acting":[9,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"20'137","reported_seq":811,"reported_epoch":971,"state":"active+clean","last_fresh":"2024-03-20T09:35:41.252604+0000","last_change":"2024-03-20T09:34:29.492837+0000","last_active":"2024-03-20T09:35:41.252604+0000","last_peered":"2024-03-20T09:35:41.252604+0000","last_clean":"2024-03-20T09:35:41.252604+0000","last_became_active":"2024-03-20T09:24:16.444232+0000","last_became_peered":"2024-03-20T09:24:16.444232+0000","last_unstale":"2024-03-20T09:35:41.252604+0000","last_undegraded":"2024-03-20T09:35:41.252604+0000","last_fullsized":"2024-03-20T09:35:41.252604+0000","mapping_epoch":718,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:34:29.492723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:34:29.492723+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:35:53.971246+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"20'137","reported_seq":2061,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-03-20T09:35:43.466021+0000","last_change":"2024-03-20T09:35:23.763445+0000","last_active":"2024-03-20T09:35:43.466021+0000","last_peered":"2024-03-20T09:35:43.466021+0000","last_clean":"2024-03-20T09:35:43.466021+0000","last_became_active":"2024-03-20T09:12:14.306384+0000","last_became_peered":"2024-03-20T09:12:14.306384+0000","last_unstale":"2024-03-20T09:35:43.466021+0000","last_undegraded":"2024-03-20T09:35:43.466021+0000","last_fullsized":"2024-03-20T09:35:43.466021+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:35:23.760323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:35:23.760323+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:36:24.718073+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"20'137","reported_seq":721,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-03-20T09:35:43.465965+0000","last_change":"2024-03-20T09:35:34.675149+0000","last_active":"2024-03-20T09:35:43.465965+0000","last_peered":"2024-03-20T09:35:43.465965+0000","last_clean":"2024-03-20T09:35:43.465965+0000","last_became_active":"2024-03-20T09:35:19.820895+0000","last_became_peered":"2024-03-20T09:35:19.820895+0000","last_unstale":"2024-03-20T09:35:43.465965+0000","last_undegraded":"2024-03-20T09:35:43.465965+0000","last_fullsized":"2024-03-20T09:35:43.465965+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:35:34.674986+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:19:03.805185+0000","last_clean_scrub_stamp":"2024-03-20T09:35:34.674986+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:36:39.546649+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"20'137","reported_seq":1591,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-03-20T09:35:43.465932+0000","last_change":"2024-03-20T09:35:19.822675+0000","last_active":"2024-03-20T09:35:43.465932+0000","last_peered":"2024-03-20T09:35:43.465932+0000","last_clean":"2024-03-20T09:35:43.465932+0000","last_became_active":"2024-03-20T09:35:19.822321+0000","last_became_peered":"2024-03-20T09:35:19.822321+0000","last_unstale":"2024-03-20T09:35:43.465932+0000","last_undegraded":"2024-03-20T09:35:43.465932+0000","last_fullsized":"2024-03-20T09:35:43.465932+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:34:42.486808+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:24:31.549603+0000","last_clean_scrub_stamp":"2024-03-20T09:34:42.486808+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:35:43.809477+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.3","version":"20'137","reported_seq":1127,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-03-20T09:35:42.829208+0000","last_change":"2024-03-20T09:35:17.017805+0000","last_active":"2024-03-20T09:35:42.829208+0000","last_peered":"2024-03-20T09:35:42.829208+0000","last_clean":"2024-03-20T09:35:42.829208+0000","last_became_active":"2024-03-20T09:31:07.424588+0000","last_became_peered":"2024-03-20T09:31:07.424588+0000","last_unstale":"2024-03-20T09:35:42.829208+0000","last_undegraded":"2024-03-20T09:35:42.829208+0000","last_fullsized":"2024-03-20T09:35:42.829208+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:35:17.017672+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:30:47.812125+0000","last_clean_scrub_stamp":"2024-03-20T09:35:17.017672+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:36:39.598582+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":2032,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-03-20T09:35:43.466135+0000","last_change":"2024-03-20T09:35:17.719384+0000","last_active":"2024-03-20T09:35:43.466135+0000","last_peered":"2024-03-20T09:35:43.466135+0000","last_clean":"2024-03-20T09:35:43.466135+0000","last_became_active":"2024-03-20T09:12:14.305989+0000","last_became_peered":"2024-03-20T09:12:14.305989+0000","last_unstale":"2024-03-20T09:35:43.466135+0000","last_undegraded":"2024-03-20T09:35:43.466135+0000","last_fullsized":"2024-03-20T09:35:43.466135+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:35:17.719287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:35:17.719287+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:36:24.402787+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.2","version":"20'137","reported_seq":2295,"reported_epoch":971,"state":"active+clean","last_fresh":"2024-03-20T09:35:41.252373+0000","last_change":"2024-03-20T09:35:22.278586+0000","last_active":"2024-03-20T09:35:41.252373+0000","last_peered":"2024-03-20T09:35:41.252373+0000","last_clean":"2024-03-20T09:35:41.252373+0000","last_became_active":"2024-03-20T09:19:54.492879+0000","last_became_peered":"2024-03-20T09:19:54.492879+0000","last_unstale":"2024-03-20T09:35:41.252373+0000","last_undegraded":"2024-03-20T09:35:41.252373+0000","last_fullsized":"2024-03-20T09:35:41.252373+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:35:22.278456+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:19.513511+0000","last_clean_scrub_stamp":"2024-03-20T09:35:22.278456+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:36:30.512953+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"20'137","reported_seq":1473,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-03-20T09:35:42.829093+0000","last_change":"2024-03-20T09:34:23.341298+0000","last_active":"2024-03-20T09:35:42.829093+0000","last_peered":"2024-03-20T09:35:42.829093+0000","last_clean":"2024-03-20T09:35:42.829093+0000","last_became_active":"2024-03-20T09:15:34.225022+0000","last_became_peered":"2024-03-20T09:15:34.225022+0000","last_unstale":"2024-03-20T09:35:42.829093+0000","last_undegraded":"2024-03-20T09:35:42.829093+0000","last_fullsized":"2024-03-20T09:35:42.829093+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:34:23.341209+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:20:12.169658+0000","last_clean_scrub_stamp":"2024-03-20T09:34:23.341209+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:35:43.771913+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,5],"acting":[6,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"20'137","reported_seq":2029,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-03-20T09:35:42.829302+0000","last_change":"2024-03-20T09:35:41.199079+0000","last_active":"2024-03-20T09:35:42.829302+0000","last_peered":"2024-03-20T09:35:42.829302+0000","last_clean":"2024-03-20T09:35:42.829302+0000","last_became_active":"2024-03-20T09:35:19.820396+0000","last_became_peered":"2024-03-20T09:35:19.820396+0000","last_unstale":"2024-03-20T09:35:42.829302+0000","last_undegraded":"2024-03-20T09:35:42.829302+0000","last_fullsized":"2024-03-20T09:35:42.829302+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:35:41.198972+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:14:10.560890+0000","last_clean_scrub_stamp":"2024-03-20T09:35:41.198972+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:37:07.910112+0000","scrub_duration":213,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"20'137","reported_seq":1844,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-03-20T09:35:43.466220+0000","last_change":"2024-03-20T09:35:22.938187+0000","last_active":"2024-03-20T09:35:43.466220+0000","last_peered":"2024-03-20T09:35:43.466220+0000","last_clean":"2024-03-20T09:35:43.466220+0000","last_became_active":"2024-03-20T09:13:20.070596+0000","last_became_peered":"2024-03-20T09:13:20.070596+0000","last_unstale":"2024-03-20T09:35:43.466220+0000","last_undegraded":"2024-03-20T09:35:43.466220+0000","last_fullsized":"2024-03-20T09:35:43.466220+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:35:22.938073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:35:22.938073+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:36:25.222701+0000","scrub_duration":241,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.e","version":"20'137","reported_seq":1277,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-03-20T09:35:43.291419+0000","last_change":"2024-03-20T09:34:35.985747+0000","last_active":"2024-03-20T09:35:43.291419+0000","last_peered":"2024-03-20T09:35:43.291419+0000","last_clean":"2024-03-20T09:35:43.291419+0000","last_became_active":"2024-03-20T09:19:54.506134+0000","last_became_peered":"2024-03-20T09:19:54.506134+0000","last_unstale":"2024-03-20T09:35:43.291419+0000","last_undegraded":"2024-03-20T09:35:43.291419+0000","last_fullsized":"2024-03-20T09:35:43.291419+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:34:35.985574+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:29:53.362224+0000","last_clean_scrub_stamp":"2024-03-20T09:34:35.985574+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:36:03.864098+0000","scrub_duration":118,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"20'137","reported_seq":2041,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-03-20T09:35:42.829194+0000","last_change":"2024-03-20T09:35:18.096669+0000","last_active":"2024-03-20T09:35:42.829194+0000","last_peered":"2024-03-20T09:35:42.829194+0000","last_clean":"2024-03-20T09:35:42.829194+0000","last_became_active":"2024-03-20T09:19:54.524397+0000","last_became_peered":"2024-03-20T09:19:54.524397+0000","last_unstale":"2024-03-20T09:35:42.829194+0000","last_undegraded":"2024-03-20T09:35:42.829194+0000","last_fullsized":"2024-03-20T09:35:42.829194+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:35:18.096555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:35:18.096555+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:36:39.029666+0000","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"570'138","reported_seq":2017,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-03-20T09:35:42.851011+0000","last_change":"2024-03-20T09:34:47.149614+0000","last_active":"2024-03-20T09:35:42.851011+0000","last_peered":"2024-03-20T09:35:42.851011+0000","last_clean":"2024-03-20T09:35:42.851011+0000","last_became_active":"2024-03-20T09:13:27.087686+0000","last_became_peered":"2024-03-20T09:13:27.087686+0000","last_unstale":"2024-03-20T09:35:42.851011+0000","last_undegraded":"2024-03-20T09:35:42.851011+0000","last_fullsized":"2024-03-20T09:35:42.851011+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"570'138","last_scrub_stamp":"2024-03-20T09:34:47.149527+0000","last_deep_scrub":"570'138","last_deep_scrub_stamp":"2024-03-20T09:16:13.725035+0000","last_clean_scrub_stamp":"2024-03-20T09:34:47.149527+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:36:01.735944+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"20'137","reported_seq":1591,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-03-20T09:35:43.291330+0000","last_change":"2024-03-20T09:35:37.346605+0000","last_active":"2024-03-20T09:35:43.291330+0000","last_peered":"2024-03-20T09:35:43.291330+0000","last_clean":"2024-03-20T09:35:43.291330+0000","last_became_active":"2024-03-20T09:19:54.502529+0000","last_became_peered":"2024-03-20T09:19:54.502529+0000","last_unstale":"2024-03-20T09:35:43.291330+0000","last_undegraded":"2024-03-20T09:35:43.291330+0000","last_fullsized":"2024-03-20T09:35:43.291330+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:35:37.346501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:35:37.346501+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:36:49.478815+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"20'137","reported_seq":1879,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-03-20T09:35:43.464700+0000","last_change":"2024-03-20T09:35:33.482736+0000","last_active":"2024-03-20T09:35:43.464700+0000","last_peered":"2024-03-20T09:35:43.464700+0000","last_clean":"2024-03-20T09:35:43.464700+0000","last_became_active":"2024-03-20T09:10:34.096758+0000","last_became_peered":"2024-03-20T09:10:34.096758+0000","last_unstale":"2024-03-20T09:35:43.464700+0000","last_undegraded":"2024-03-20T09:35:43.464700+0000","last_fullsized":"2024-03-20T09:35:43.464700+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:35:33.482629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:35:33.482629+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:36:58.891104+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"20'137","reported_seq":632,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-03-20T09:35:43.466253+0000","last_change":"2024-03-20T09:35:19.822721+0000","last_active":"2024-03-20T09:35:43.466253+0000","last_peered":"2024-03-20T09:35:43.466253+0000","last_clean":"2024-03-20T09:35:43.466253+0000","last_became_active":"2024-03-20T09:35:19.822242+0000","last_became_peered":"2024-03-20T09:35:19.822242+0000","last_unstale":"2024-03-20T09:35:43.466253+0000","last_undegraded":"2024-03-20T09:35:43.466253+0000","last_fullsized":"2024-03-20T09:35:43.466253+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:34:29.005597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:34:29.005597+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:35:49.326598+0000","scrub_duration":243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.10","version":"20'137","reported_seq":1434,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-03-20T09:35:43.464698+0000","last_change":"2024-03-20T09:35:19.820972+0000","last_active":"2024-03-20T09:35:43.464698+0000","last_peered":"2024-03-20T09:35:43.464698+0000","last_clean":"2024-03-20T09:35:43.464698+0000","last_became_active":"2024-03-20T09:35:19.820595+0000","last_became_peered":"2024-03-20T09:35:19.820595+0000","last_unstale":"2024-03-20T09:35:43.464698+0000","last_undegraded":"2024-03-20T09:35:43.464698+0000","last_fullsized":"2024-03-20T09:35:43.464698+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:34:49.348046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:34:49.348046+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:36:07.740616+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"20'137","reported_seq":1821,"reported_epoch":971,"state":"active+clean","last_fresh":"2024-03-20T09:35:41.252238+0000","last_change":"2024-03-20T09:35:25.255520+0000","last_active":"2024-03-20T09:35:41.252238+0000","last_peered":"2024-03-20T09:35:41.252238+0000","last_clean":"2024-03-20T09:35:41.252238+0000","last_became_active":"2024-03-20T09:20:25.342982+0000","last_became_peered":"2024-03-20T09:20:25.342982+0000","last_unstale":"2024-03-20T09:35:41.252238+0000","last_undegraded":"2024-03-20T09:35:41.252238+0000","last_fullsized":"2024-03-20T09:35:41.252238+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:35:25.255346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:35:25.255346+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:36:40.542213+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"20'137","reported_seq":546,"reported_epoch":971,"state":"active+clean","last_fresh":"2024-03-20T09:35:41.253654+0000","last_change":"2024-03-20T09:35:27.264305+0000","last_active":"2024-03-20T09:35:41.253654+0000","last_peered":"2024-03-20T09:35:41.253654+0000","last_clean":"2024-03-20T09:35:41.253654+0000","last_became_active":"2024-03-20T09:31:07.424851+0000","last_became_peered":"2024-03-20T09:31:07.424851+0000","last_unstale":"2024-03-20T09:35:41.253654+0000","last_undegraded":"2024-03-20T09:35:41.253654+0000","last_fullsized":"2024-03-20T09:35:41.253654+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:35:27.264203+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:35:27.264203+0000","last_clean_scrub_stamp":"2024-03-20T09:35:27.264203+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:36:46.835135+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.17","version":"20'137","reported_seq":2010,"reported_epoch":971,"state":"active+clean","last_fresh":"2024-03-20T09:35:41.251877+0000","last_change":"2024-03-20T09:35:20.274710+0000","last_active":"2024-03-20T09:35:41.251877+0000","last_peered":"2024-03-20T09:35:41.251877+0000","last_clean":"2024-03-20T09:35:41.251877+0000","last_became_active":"2024-03-20T09:35:19.822429+0000","last_became_peered":"2024-03-20T09:35:19.822429+0000","last_unstale":"2024-03-20T09:35:41.251877+0000","last_undegraded":"2024-03-20T09:35:41.251877+0000","last_fullsized":"2024-03-20T09:35:41.251877+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:35:20.274612+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:23.640533+0000","last_clean_scrub_stamp":"2024-03-20T09:35:20.274612+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:36:25.802969+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"20'137","reported_seq":2052,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-03-20T09:35:42.829518+0000","last_change":"2024-03-20T09:35:37.033618+0000","last_active":"2024-03-20T09:35:42.829518+0000","last_peered":"2024-03-20T09:35:42.829518+0000","last_clean":"2024-03-20T09:35:42.829518+0000","last_became_active":"2024-03-20T09:20:25.349082+0000","last_became_peered":"2024-03-20T09:20:25.349082+0000","last_unstale":"2024-03-20T09:35:42.829518+0000","last_undegraded":"2024-03-20T09:35:42.829518+0000","last_fullsized":"2024-03-20T09:35:42.829518+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:35:37.033504+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:35:37.033504+0000","last_clean_scrub_stamp":"2024-03-20T09:35:37.033504+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:36:43.338399+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"20'137","reported_seq":1325,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-03-20T09:35:43.464907+0000","last_change":"2024-03-20T09:34:48.079348+0000","last_active":"2024-03-20T09:35:43.464907+0000","last_peered":"2024-03-20T09:35:43.464907+0000","last_clean":"2024-03-20T09:35:43.464907+0000","last_became_active":"2024-03-20T09:12:14.307581+0000","last_became_peered":"2024-03-20T09:12:14.307581+0000","last_unstale":"2024-03-20T09:35:43.464907+0000","last_undegraded":"2024-03-20T09:35:43.464907+0000","last_fullsized":"2024-03-20T09:35:43.464907+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:34:48.079272+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:34:48.079272+0000","last_clean_scrub_stamp":"2024-03-20T09:34:48.079272+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:35:48.949706+0000","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"20'137","reported_seq":1590,"reported_epoch":972,"state":"active+clean","last_fresh":"2024-03-20T09:35:43.291352+0000","last_change":"2024-03-20T09:35:38.290596+0000","last_active":"2024-03-20T09:35:43.291352+0000","last_peered":"2024-03-20T09:35:43.291352+0000","last_clean":"2024-03-20T09:35:43.291352+0000","last_became_active":"2024-03-20T09:19:54.502391+0000","last_became_peered":"2024-03-20T09:19:54.502391+0000","last_unstale":"2024-03-20T09:35:43.291352+0000","last_undegraded":"2024-03-20T09:35:43.291352+0000","last_fullsized":"2024-03-20T09:35:43.291352+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:35:38.290504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:35:38.290504+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:36:49.545806+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4248,"ondisk_log_size":4248,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860893,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":907024,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":905790,"kb_avail":92838128,"statfs":{"total":95995035648,"available":95066243072,"internally_reserved":0,"allocated":1064960,"data_stored":667875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":927529412},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[]},{"osd":10,"up_from":861,"seq":3697966841918,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":109968,"kb_used_data":1040,"kb_used_omap":103,"kb_used_meta":108632,"kb_avail":93635184,"statfs":{"total":95995035648,"available":95882428416,"internally_reserved":0,"allocated":1064960,"data_stored":667875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":106169,"internal_metadata":111239495},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":9,"up_from":80,"seq":343597384170,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":451656,"kb_used_data":1208,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":93293496,"statfs":{"total":95995035648,"available":95532539904,"internally_reserved":0,"allocated":1236992,"data_stored":778522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":461240768},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[]},{"osd":8,"up_from":70,"seq":300647711214,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":537716,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":93207436,"statfs":{"total":95995035648,"available":95444414464,"internally_reserved":0,"allocated":1220608,"data_stored":770628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":549386694},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[]},{"osd":7,"up_from":60,"seq":257698038259,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":835032,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":833598,"kb_avail":92910120,"statfs":{"total":95995035648,"available":95139962880,"internally_reserved":0,"allocated":1269760,"data_stored":869123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":853604810},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332600,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":704656,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93040496,"statfs":{"total":95995035648,"available":95273467904,"internally_reserved":0,"allocated":1392640,"data_stored":995555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":719976902},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575378,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":880660,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":879422,"kb_avail":92864492,"statfs":{"total":95995035648,"available":95093239808,"internally_reserved":0,"allocated":1064960,"data_stored":667875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":900528582},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738896,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":993384,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":991870,"kb_avail":92751768,"statfs":{"total":95995035648,"available":94977810432,"internally_reserved":0,"allocated":1536000,"data_stored":1090386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1015675333},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411850,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":712860,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":711422,"kb_avail":93032292,"statfs":{"total":95995035648,"available":95265067008,"internally_reserved":0,"allocated":1269760,"data_stored":869123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":728496581},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150215,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":953172,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":951614,"kb_avail":92791980,"statfs":{"total":95995035648,"available":95018987520,"internally_reserved":0,"allocated":1392640,"data_stored":995555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":974453191},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953990,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":704336,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93040816,"statfs":{"total":95995035648,"available":95273795584,"internally_reserved":0,"allocated":1064960,"data_stored":667875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":5,"up_from":443,"seq":1902670512431,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":676952,"kb_used_data":1240,"kb_used_omap":181,"kb_used_meta":675338,"kb_avail":93068200,"statfs":{"total":95995035648,"available":95301836800,"internally_reserved":0,"allocated":1269760,"data_stored":869123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":186131,"internal_metadata":691546349},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:35:49.440 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:35:49.440 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:35:49.440 INFO:teuthology.orchestra.run.smithi115.stdout: "complaint_time": 30, 2024-03-20T09:35:49.441 INFO:teuthology.orchestra.run.smithi115.stdout: "num_blocked_ops": 0 2024-03-20T09:35:49.441 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:35:49.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:49 smithi026 ceph-mon[68309]: pgmap v2349: 31 pgs: 31 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:35:49.464 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:49 smithi115 ceph-mon[72476]: pgmap v2349: 31 pgs: 31 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:35:49.553 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:35:49.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:49.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:35:49.657 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:35:49.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:49 smithi122 ceph-mon[73495]: pgmap v2349: 31 pgs: 31 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:35:49.687 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:35:49.688 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:49.712 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:35:49.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:49.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:35:49.816 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:35:49.847 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:50.046 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:35:50.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:50.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:35:50.067 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-20T09:35:50.150 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:35:50.179 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:50.257 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'unique_pool_6' created 2024-03-20T09:35:50.371 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:35:50.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:50 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3313238374' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-03-20T09:35:50.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:50 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3313238374' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-03-20T09:35:50.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:50 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3313238374' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]: dispatch 2024-03-20T09:35:51.129 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:35:51.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:51.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:35:51.233 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:35:51.270 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:51.291 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:35:51.293 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:51.294 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:35:51.395 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:35:51.424 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:51.444 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:35:51.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:51.447 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:35:51.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:51 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3313238374' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-03-20T09:35:51.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:51 smithi026 ceph-mon[68309]: osdmap e974: 12 total, 12 up, 12 in 2024-03-20T09:35:51.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:51 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:35:51.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:51 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:35:51.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:51 smithi026 ceph-mon[68309]: pgmap v2351: 47 pgs: 1 creating+peering, 15 unknown, 31 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:35:51.548 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:35:51.554 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:35:51.581 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:51.581 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:51 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3313238374' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-03-20T09:35:51.582 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:51 smithi115 ceph-mon[72476]: osdmap e974: 12 total, 12 up, 12 in 2024-03-20T09:35:51.582 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:51 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:35:51.582 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:51 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:35:51.582 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:51 smithi115 ceph-mon[72476]: pgmap v2351: 47 pgs: 1 creating+peering, 15 unknown, 31 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:35:51.681 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:35:51.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:51.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:35:51.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:51 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3313238374' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-03-20T09:35:51.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:51 smithi122 ceph-mon[73495]: osdmap e974: 12 total, 12 up, 12 in 2024-03-20T09:35:51.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:51 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:35:51.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:51 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:35:51.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:51 smithi122 ceph-mon[73495]: pgmap v2351: 47 pgs: 1 creating+peering, 15 unknown, 31 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:35:51.698 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pool application enable unique_pool_6 rados --yes-i-really-mean-it || true 2024-03-20T09:35:51.786 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:35:51.817 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:51.838 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:35:51.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:51.841 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:35:51.942 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:35:51.978 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:52.159 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:35:52.159 INFO:teuthology.orchestra.run.smithi115.stdout: "size": 20, 2024-03-20T09:35:52.159 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 600, 2024-03-20T09:35:52.159 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [ 2024-03-20T09:35:52.159 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.159 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26216.0:129134 6.b e748/709 6:dfb654b3:::benchmark_data_smithi122_168655_object9025:head v 748'8030, mlcod=748'8027)", 2024-03-20T09:35:52.160 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:56.321820+0000", 2024-03-20T09:35:52.160 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 595.84547696499999, 2024-03-20T09:35:52.160 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.26522621299999999, 2024-03-20T09:35:52.160 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:52.160 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:52.160 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:52.160 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.160 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:52.160 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.321820+0000", 2024-03-20T09:35:52.160 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.160 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.160 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.160 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:52.160 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.321820+0000", 2024-03-20T09:35:52.161 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.161 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.161 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.161 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:52.161 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.321821+0000", 2024-03-20T09:35:52.161 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.437e-06 2024-03-20T09:35:52.161 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.161 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.161 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:52.161 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.321829+0000", 2024-03-20T09:35:52.161 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7009999999999996e-06 2024-03-20T09:35:52.161 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.161 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.161 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:52.162 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.321845+0000", 2024-03-20T09:35:52.162 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5852999999999999e-05 2024-03-20T09:35:52.162 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.162 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.162 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:52.162 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.321853+0000", 2024-03-20T09:35:52.162 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.2379999999999997e-06 2024-03-20T09:35:52.162 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.162 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.162 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:52.162 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.321913+0000", 2024-03-20T09:35:52.162 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0377999999999997e-05 2024-03-20T09:35:52.163 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.163 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.163 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:35:52.163 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.321938+0000", 2024-03-20T09:35:52.163 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.4329e-05 2024-03-20T09:35:52.163 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.163 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.163 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:35:52.163 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.586991+0000", 2024-03-20T09:35:52.163 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.26505305800000001 2024-03-20T09:35:52.163 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.164 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.164 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:52.164 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.587046+0000", 2024-03-20T09:35:52.164 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5219000000000001e-05 2024-03-20T09:35:52.164 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.164 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:52.164 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.164 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.164 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.164 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26216.0:129139 6.8 e748/709 6:1a6b87cd:::benchmark_data_smithi122_168655_object9030:head v 748'7939, mlcod=748'7937)", 2024-03-20T09:35:52.164 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:56.323431+0000", 2024-03-20T09:35:52.164 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 595.84386531400003, 2024-03-20T09:35:52.164 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.234771442, 2024-03-20T09:35:52.164 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:52.165 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:35:52.165 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:52.165 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.165 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:52.165 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.323431+0000", 2024-03-20T09:35:52.165 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.165 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.165 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.165 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:52.165 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.323431+0000", 2024-03-20T09:35:52.165 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.165 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.165 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.165 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:52.166 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.323432+0000", 2024-03-20T09:35:52.166 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.4e-07 2024-03-20T09:35:52.166 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.167 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.167 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:52.167 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.323439+0000", 2024-03-20T09:35:52.167 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4559999999999997e-06 2024-03-20T09:35:52.167 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.168 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.168 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:52.168 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.323452+0000", 2024-03-20T09:35:52.168 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3574000000000001e-05 2024-03-20T09:35:52.168 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.168 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.168 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:52.169 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.323460+0000", 2024-03-20T09:35:52.169 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.5669999999999997e-06 2024-03-20T09:35:52.169 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.169 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.169 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:52.169 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.323522+0000", 2024-03-20T09:35:52.170 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2389000000000002e-05 2024-03-20T09:35:52.170 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.170 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.170 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:35:52.170 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.323544+0000", 2024-03-20T09:35:52.170 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1702e-05 2024-03-20T09:35:52.171 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.171 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.171 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:52.171 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.558203+0000", 2024-03-20T09:35:52.171 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.23465881399999999 2024-03-20T09:35:52.171 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.172 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:52.172 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.172 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.172 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.172 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26216.0:129140 6.3 e748/709)", 2024-03-20T09:35:52.172 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:56.327278+0000", 2024-03-20T09:35:52.173 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 595.84001848100002, 2024-03-20T09:35:52.173 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.26500126899999998, 2024-03-20T09:35:52.173 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:52.173 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:35:52.173 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:52.173 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.174 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:52.174 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.327278+0000", 2024-03-20T09:35:52.174 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.174 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.174 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.174 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:52.175 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.327278+0000", 2024-03-20T09:35:52.175 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.175 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.175 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.175 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:52.175 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.327279+0000", 2024-03-20T09:35:52.176 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0669999999999999e-06 2024-03-20T09:35:52.176 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.176 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.176 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:52.176 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.327288+0000", 2024-03-20T09:35:52.176 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5920000000000001e-06 2024-03-20T09:35:52.177 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.177 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.177 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:52.177 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.327301+0000", 2024-03-20T09:35:52.177 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2768e-05 2024-03-20T09:35:52.178 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.178 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.178 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:52.178 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.327308+0000", 2024-03-20T09:35:52.178 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.2579999999999998e-06 2024-03-20T09:35:52.178 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.179 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.179 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:52.179 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.592260+0000", 2024-03-20T09:35:52.179 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.26495219599999997 2024-03-20T09:35:52.179 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.179 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.180 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:52.180 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.592279+0000", 2024-03-20T09:35:52.180 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9388000000000001e-05 2024-03-20T09:35:52.180 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.180 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:52.181 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.181 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.181 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.181 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26216.0:129141 6.b e748/709 6:ddae97ab:::benchmark_data_smithi122_168655_object9032:head v 748'8031, mlcod=748'8028)", 2024-03-20T09:35:52.181 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:56.329794+0000", 2024-03-20T09:35:52.181 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 595.83750305900003, 2024-03-20T09:35:52.182 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.29588104700000001, 2024-03-20T09:35:52.182 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:52.182 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:52.182 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:52.182 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.183 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:52.183 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.329794+0000", 2024-03-20T09:35:52.183 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.183 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.183 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.183 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:52.183 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.329794+0000", 2024-03-20T09:35:52.184 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.184 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.184 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.184 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:52.185 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.329795+0000", 2024-03-20T09:35:52.185 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.083e-06 2024-03-20T09:35:52.185 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.185 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.185 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:52.185 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.329803+0000", 2024-03-20T09:35:52.186 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.7450000000000001e-06 2024-03-20T09:35:52.186 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.186 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.186 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:52.186 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.329815+0000", 2024-03-20T09:35:52.186 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1790000000000001e-05 2024-03-20T09:35:52.187 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.187 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.187 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:52.187 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.329822+0000", 2024-03-20T09:35:52.188 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.7440000000000001e-06 2024-03-20T09:35:52.188 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.188 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.188 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:52.188 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.582786+0000", 2024-03-20T09:35:52.188 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.25296439500000001 2024-03-20T09:35:52.188 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.189 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.189 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:35:52.189 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.582813+0000", 2024-03-20T09:35:52.189 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6806000000000001e-05 2024-03-20T09:35:52.189 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.189 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.189 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:35:52.190 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.625623+0000", 2024-03-20T09:35:52.190 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.042809502999999999 2024-03-20T09:35:52.190 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.190 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.190 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:52.190 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.625675+0000", 2024-03-20T09:35:52.191 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1981000000000003e-05 2024-03-20T09:35:52.191 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.191 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:52.191 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.191 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.191 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.191 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26216.0:129146 6.b e749/709 6:d9973fcf:::benchmark_data_smithi122_168655_object9037:head v 749'8032, mlcod=748'8028)", 2024-03-20T09:35:52.192 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:25:56.362248+0000", 2024-03-20T09:35:52.192 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 595.80504854699996, 2024-03-20T09:35:52.192 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.26344784799999998, 2024-03-20T09:35:52.192 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:52.192 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:52.192 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:52.193 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.193 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:52.193 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.362248+0000", 2024-03-20T09:35:52.193 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.193 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.193 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.193 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:52.194 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.362248+0000", 2024-03-20T09:35:52.194 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.194 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.194 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.194 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:52.194 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.362248+0000", 2024-03-20T09:35:52.194 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6600000000000003e-07 2024-03-20T09:35:52.195 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.195 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.195 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:52.195 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.362255+0000", 2024-03-20T09:35:52.195 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.917e-06 2024-03-20T09:35:52.195 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.195 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.196 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:52.196 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.362282+0000", 2024-03-20T09:35:52.196 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6344999999999999e-05 2024-03-20T09:35:52.196 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.196 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.196 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:52.197 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.362298+0000", 2024-03-20T09:35:52.197 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6087999999999999e-05 2024-03-20T09:35:52.197 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.197 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.197 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:52.197 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.587169+0000", 2024-03-20T09:35:52.197 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.224871712 2024-03-20T09:35:52.197 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.198 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.198 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:35:52.198 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.587197+0000", 2024-03-20T09:35:52.198 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.7977e-05 2024-03-20T09:35:52.198 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.198 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.198 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:35:52.199 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.625680+0000", 2024-03-20T09:35:52.199 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.038482502000000002 2024-03-20T09:35:52.199 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.199 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.199 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:52.199 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:25:56.625696+0000", 2024-03-20T09:35:52.199 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6041000000000001e-05 2024-03-20T09:35:52.200 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.200 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:52.200 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.200 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.200 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.200 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26216.0:231442 6.1 e858/823 6:86646aa3:::benchmark_data_smithi122_168655_object111333:head v 858'11315, mlcod=858'11313)", 2024-03-20T09:35:52.200 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:30:36.418993+0000", 2024-03-20T09:35:52.201 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 315.74830331700002, 2024-03-20T09:35:52.201 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.042099804999999997, 2024-03-20T09:35:52.201 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:52.201 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:52.201 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:52.201 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.201 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:52.202 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:36.418993+0000", 2024-03-20T09:35:52.202 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.202 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.202 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.202 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:52.202 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:36.418993+0000", 2024-03-20T09:35:52.203 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.203 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.203 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.203 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:52.203 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:36.418995+0000", 2024-03-20T09:35:52.203 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.249e-06 2024-03-20T09:35:52.203 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.203 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.204 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:52.204 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:36.419005+0000", 2024-03-20T09:35:52.204 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0699e-05 2024-03-20T09:35:52.204 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.204 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.204 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:52.204 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:36.419022+0000", 2024-03-20T09:35:52.204 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6252e-05 2024-03-20T09:35:52.204 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.205 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.205 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:52.205 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:36.419031+0000", 2024-03-20T09:35:52.205 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.7909999999999999e-06 2024-03-20T09:35:52.205 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.205 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.205 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:52.205 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:36.419103+0000", 2024-03-20T09:35:52.205 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.1476000000000005e-05 2024-03-20T09:35:52.206 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.206 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.206 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:35:52.206 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:36.419128+0000", 2024-03-20T09:35:52.206 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5377000000000001e-05 2024-03-20T09:35:52.206 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.206 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.206 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:35:52.207 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:36.461052+0000", 2024-03-20T09:35:52.207 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.041924318000000002 2024-03-20T09:35:52.207 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.207 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.207 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:52.207 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:30:36.461093+0000", 2024-03-20T09:35:52.207 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.0642999999999997e-05 2024-03-20T09:35:52.207 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.208 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:52.208 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.208 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.208 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.208 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "rep_scrubmap(1.15 e491 from shard 8)", 2024-03-20T09:35:52.208 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:31:32.342334+0000", 2024-03-20T09:35:52.208 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 259.82496217400001, 2024-03-20T09:35:52.209 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.068097784999999994, 2024-03-20T09:35:52.209 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:52.209 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:35:52.209 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:52.209 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.209 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:52.210 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.342334+0000", 2024-03-20T09:35:52.210 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.210 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.210 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.210 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:52.210 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.342334+0000", 2024-03-20T09:35:52.211 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.211 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.211 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.211 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:52.211 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.342335+0000", 2024-03-20T09:35:52.211 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.9699999999999995e-07 2024-03-20T09:35:52.211 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.212 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.212 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:52.212 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.342341+0000", 2024-03-20T09:35:52.212 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5620000000000003e-06 2024-03-20T09:35:52.212 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.212 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.213 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:52.213 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.342351+0000", 2024-03-20T09:35:52.213 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0621999999999999e-05 2024-03-20T09:35:52.213 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.213 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.213 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:52.214 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.342357+0000", 2024-03-20T09:35:52.214 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7440000000000004e-06 2024-03-20T09:35:52.214 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.214 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.214 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:52.214 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.410402+0000", 2024-03-20T09:35:52.215 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.068045091000000002 2024-03-20T09:35:52.215 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.215 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.215 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:35:52.215 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.410414+0000", 2024-03-20T09:35:52.215 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1867e-05 2024-03-20T09:35:52.215 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.215 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.216 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:52.216 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.410432+0000", 2024-03-20T09:35:52.216 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8102e-05 2024-03-20T09:35:52.216 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.216 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:52.216 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.217 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.217 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.217 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "rep_scrubmap(1.15 e491 from shard 5)", 2024-03-20T09:35:52.217 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:31:32.346280+0000", 2024-03-20T09:35:52.217 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 259.82101630599999, 2024-03-20T09:35:52.217 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.064250347999999999, 2024-03-20T09:35:52.218 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:52.218 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:35:52.218 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:52.218 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.218 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:52.218 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.346280+0000", 2024-03-20T09:35:52.219 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.219 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.219 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.219 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:52.219 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.346280+0000", 2024-03-20T09:35:52.219 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.220 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.220 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.220 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:52.220 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.346281+0000", 2024-03-20T09:35:52.220 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.5999999999999991e-07 2024-03-20T09:35:52.220 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.220 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.221 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:52.221 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.346287+0000", 2024-03-20T09:35:52.221 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4299999999999997e-06 2024-03-20T09:35:52.221 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.221 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.221 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:52.222 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.346301+0000", 2024-03-20T09:35:52.222 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4088e-05 2024-03-20T09:35:52.222 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.222 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.222 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:52.222 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.346308+0000", 2024-03-20T09:35:52.222 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.1319999999999996e-06 2024-03-20T09:35:52.222 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.222 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.223 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:52.223 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.410482+0000", 2024-03-20T09:35:52.223 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.064174155999999996 2024-03-20T09:35:52.223 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.223 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.223 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:35:52.223 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.410494+0000", 2024-03-20T09:35:52.223 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1439e-05 2024-03-20T09:35:52.223 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.223 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.223 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:52.223 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.410531+0000", 2024-03-20T09:35:52.223 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7143e-05 2024-03-20T09:35:52.223 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.224 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:52.224 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.224 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.224 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.224 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "rep_scrubmap(1.15 e491 from shard 5)", 2024-03-20T09:35:52.224 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:33:08.996571+0000", 2024-03-20T09:35:52.224 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 163.170725809, 2024-03-20T09:35:52.224 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.050566558999999997, 2024-03-20T09:35:52.225 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:52.225 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:35:52.225 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:52.225 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.225 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:52.225 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:33:08.996571+0000", 2024-03-20T09:35:52.225 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.226 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.226 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.226 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:52.226 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:33:08.996571+0000", 2024-03-20T09:35:52.227 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.227 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.227 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.227 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:52.227 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:33:08.996572+0000", 2024-03-20T09:35:52.227 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4839999999999999e-06 2024-03-20T09:35:52.228 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.228 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.228 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:52.228 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:33:08.996581+0000", 2024-03-20T09:35:52.228 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.6729999999999998e-06 2024-03-20T09:35:52.228 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.228 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.229 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:52.229 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:33:08.996597+0000", 2024-03-20T09:35:52.229 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6483000000000001e-05 2024-03-20T09:35:52.229 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.229 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.229 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:52.230 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:33:08.996607+0000", 2024-03-20T09:35:52.230 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.7270000000000002e-06 2024-03-20T09:35:52.230 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.230 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.230 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:52.230 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:33:09.047117+0000", 2024-03-20T09:35:52.231 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.050509999 2024-03-20T09:35:52.231 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.231 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.231 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:35:52.231 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:33:09.047125+0000", 2024-03-20T09:35:52.231 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.977e-06 2024-03-20T09:35:52.231 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.232 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.232 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:52.232 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:33:09.047137+0000", 2024-03-20T09:35:52.232 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2216e-05 2024-03-20T09:35:52.232 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.232 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:52.233 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.233 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.233 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.233 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "rep_scrubmap(1.15 e491 from shard 8)", 2024-03-20T09:35:52.233 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:33:09.002863+0000", 2024-03-20T09:35:52.233 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 163.16443347000001, 2024-03-20T09:35:52.234 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.044336167000000003, 2024-03-20T09:35:52.234 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:52.234 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:35:52.234 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:52.234 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.234 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:52.235 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:33:09.002863+0000", 2024-03-20T09:35:52.235 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.235 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.235 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.235 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:52.235 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:33:09.002863+0000", 2024-03-20T09:35:52.235 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.236 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.236 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.236 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:52.236 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:33:09.002864+0000", 2024-03-20T09:35:52.236 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.8700000000000004e-07 2024-03-20T09:35:52.236 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.236 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.237 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:52.237 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:33:09.002871+0000", 2024-03-20T09:35:52.237 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.5599999999999999e-06 2024-03-20T09:35:52.237 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.237 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.237 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:52.238 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:33:09.002883+0000", 2024-03-20T09:35:52.238 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1980999999999999e-05 2024-03-20T09:35:52.238 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.238 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.238 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:52.238 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:33:09.002890+0000", 2024-03-20T09:35:52.239 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.1409999999999999e-06 2024-03-20T09:35:52.239 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.239 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.239 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:52.239 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:33:09.047161+0000", 2024-03-20T09:35:52.239 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.044271208999999999 2024-03-20T09:35:52.240 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.240 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.240 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:35:52.240 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:33:09.047169+0000", 2024-03-20T09:35:52.240 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.0749999999999998e-06 2024-03-20T09:35:52.240 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.241 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.241 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:52.241 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:33:09.047199+0000", 2024-03-20T09:35:52.241 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.0213999999999999e-05 2024-03-20T09:35:52.241 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.241 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:52.242 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.242 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.242 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.242 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26605.0:147322 7.6 e915/871 7:66bfb195:::benchmark_data_smithi122_187077_object20711:head v 915'9235, mlcod=915'9233)", 2024-03-20T09:35:52.242 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:33:12.123263+0000", 2024-03-20T09:35:52.242 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 160.04403342200001, 2024-03-20T09:35:52.243 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.048797140000000003, 2024-03-20T09:35:52.243 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:52.243 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:52.243 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:52.243 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.244 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:52.244 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:33:12.123263+0000", 2024-03-20T09:35:52.244 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.244 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.244 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.244 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:52.245 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:33:12.123263+0000", 2024-03-20T09:35:52.245 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.245 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.245 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.246 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:52.246 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:33:12.123265+0000", 2024-03-20T09:35:52.246 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.438e-06 2024-03-20T09:35:52.246 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.246 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.246 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:52.247 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:33:12.123274+0000", 2024-03-20T09:35:52.247 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.4450000000000002e-06 2024-03-20T09:35:52.247 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.247 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.247 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:52.247 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:33:12.123291+0000", 2024-03-20T09:35:52.247 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7397999999999998e-05 2024-03-20T09:35:52.247 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.247 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.247 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:52.247 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:33:12.123301+0000", 2024-03-20T09:35:52.247 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.5370000000000003e-06 2024-03-20T09:35:52.248 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.248 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.248 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:52.248 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:33:12.167635+0000", 2024-03-20T09:35:52.248 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.044333758000000001 2024-03-20T09:35:52.248 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.248 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.248 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:35:52.248 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:33:12.167659+0000", 2024-03-20T09:35:52.248 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.4442000000000001e-05 2024-03-20T09:35:52.248 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.249 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.249 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:35:52.249 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:33:12.172013+0000", 2024-03-20T09:35:52.249 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0043535739999999998 2024-03-20T09:35:52.249 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.249 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.249 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:52.249 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:33:12.172060+0000", 2024-03-20T09:35:52.249 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.7547999999999999e-05 2024-03-20T09:35:52.249 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.249 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:52.249 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.249 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.250 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.250 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26605.0:222035 7.15 e943/936 7:ad9477d6:::benchmark_data_smithi122_187077_object95424:head v 943'13498, mlcod=942'13496)", 2024-03-20T09:35:52.250 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:47.897707+0000", 2024-03-20T09:35:52.250 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 64.269589244000002, 2024-03-20T09:35:52.250 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.086363878000000005, 2024-03-20T09:35:52.250 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:52.250 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:52.250 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:52.250 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.250 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:52.250 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.897707+0000", 2024-03-20T09:35:52.250 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.250 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.250 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.251 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:52.251 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.897707+0000", 2024-03-20T09:35:52.251 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.251 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.251 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.251 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:52.251 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.897708+0000", 2024-03-20T09:35:52.251 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.85e-07 2024-03-20T09:35:52.251 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.251 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.251 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:52.251 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.897718+0000", 2024-03-20T09:35:52.251 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0064999999999999e-05 2024-03-20T09:35:52.252 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.252 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.252 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:52.252 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.897732+0000", 2024-03-20T09:35:52.252 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3771000000000001e-05 2024-03-20T09:35:52.252 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.252 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.252 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:52.252 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.897740+0000", 2024-03-20T09:35:52.252 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.9680000000000006e-06 2024-03-20T09:35:52.252 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.252 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.252 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:52.252 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.903105+0000", 2024-03-20T09:35:52.253 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0053646609999999997 2024-03-20T09:35:52.253 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.253 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.253 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:35:52.253 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.903126+0000", 2024-03-20T09:35:52.253 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1685e-05 2024-03-20T09:35:52.253 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.253 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.253 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:35:52.253 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.984025+0000", 2024-03-20T09:35:52.253 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.080898484000000007 2024-03-20T09:35:52.253 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.253 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.253 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:52.253 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.984071+0000", 2024-03-20T09:35:52.254 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6359000000000001e-05 2024-03-20T09:35:52.254 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.254 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:52.254 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.254 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.254 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.254 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26605.0:222074 7.15 e943/936 7:adebcb1b:::benchmark_data_smithi122_187077_object95463:head v 943'13499, mlcod=942'13496)", 2024-03-20T09:35:52.254 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:47.936093+0000", 2024-03-20T09:35:52.254 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 64.231203338, 2024-03-20T09:35:52.254 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.048009769000000001, 2024-03-20T09:35:52.254 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:52.254 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:52.254 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:52.255 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.255 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:52.255 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.936093+0000", 2024-03-20T09:35:52.255 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.255 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.255 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.255 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:52.255 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.936093+0000", 2024-03-20T09:35:52.255 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.255 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.255 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.256 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:52.256 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.936094+0000", 2024-03-20T09:35:52.256 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.0400000000000002e-07 2024-03-20T09:35:52.256 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.256 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.256 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:52.256 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.936098+0000", 2024-03-20T09:35:52.256 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.1840000000000001e-06 2024-03-20T09:35:52.256 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.256 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.256 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:52.256 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.936103+0000", 2024-03-20T09:35:52.256 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0760000000000002e-06 2024-03-20T09:35:52.256 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.256 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.257 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:52.257 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.936108+0000", 2024-03-20T09:35:52.257 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8130000000000003e-06 2024-03-20T09:35:52.257 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.258 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.258 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:52.258 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.936782+0000", 2024-03-20T09:35:52.258 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00067471700000000003 2024-03-20T09:35:52.258 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.258 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.258 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:35:52.258 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.936796+0000", 2024-03-20T09:35:52.258 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3531e-05 2024-03-20T09:35:52.258 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.258 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.258 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:35:52.258 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.984075+0000", 2024-03-20T09:35:52.259 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.047279132000000001 2024-03-20T09:35:52.259 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.259 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.259 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:52.259 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.984103+0000", 2024-03-20T09:35:52.259 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.7912e-05 2024-03-20T09:35:52.259 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.259 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:52.259 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.259 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.259 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.259 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26605.0:222091 7.15 e943/936 7:aed88c30:::benchmark_data_smithi122_187077_object95480:head v 943'13500, mlcod=942'13496)", 2024-03-20T09:35:52.259 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:47.941768+0000", 2024-03-20T09:35:52.259 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 64.225528797999999, 2024-03-20T09:35:52.260 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.042362952000000002, 2024-03-20T09:35:52.260 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:52.260 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:52.260 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:52.260 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.260 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:52.260 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.941768+0000", 2024-03-20T09:35:52.260 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.260 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.260 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.260 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:52.260 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.941768+0000", 2024-03-20T09:35:52.260 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.261 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.261 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.261 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:52.261 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.941769+0000", 2024-03-20T09:35:52.261 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.0900000000000001e-07 2024-03-20T09:35:52.261 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.261 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.261 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:52.261 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.941774+0000", 2024-03-20T09:35:52.261 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8119999999999996e-06 2024-03-20T09:35:52.261 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.261 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.261 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:52.262 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.941787+0000", 2024-03-20T09:35:52.262 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2393e-05 2024-03-20T09:35:52.262 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.262 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.262 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:52.262 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.941795+0000", 2024-03-20T09:35:52.262 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.8220000000000007e-06 2024-03-20T09:35:52.262 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.262 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.262 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:52.262 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.941851+0000", 2024-03-20T09:35:52.262 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6373999999999999e-05 2024-03-20T09:35:52.262 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.262 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.263 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:35:52.263 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.941877+0000", 2024-03-20T09:35:52.263 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6503e-05 2024-03-20T09:35:52.263 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.263 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.263 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:35:52.263 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.984105+0000", 2024-03-20T09:35:52.263 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.042227998000000003 2024-03-20T09:35:52.263 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.263 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.264 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:52.264 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.984131+0000", 2024-03-20T09:35:52.264 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5341e-05 2024-03-20T09:35:52.264 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.264 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:52.264 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.264 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.264 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.264 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "rep_scrubmap(1.15 e491 from shard 5)", 2024-03-20T09:35:52.264 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:48.016412+0000", 2024-03-20T09:35:52.264 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 64.150884963999999, 2024-03-20T09:35:52.265 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.060988498000000002, 2024-03-20T09:35:52.265 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:52.265 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:35:52.265 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:52.265 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.265 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:52.265 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.016412+0000", 2024-03-20T09:35:52.265 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.265 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.265 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.265 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:52.265 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.016412+0000", 2024-03-20T09:35:52.265 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.265 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.265 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.266 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:52.266 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.016413+0000", 2024-03-20T09:35:52.266 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.4099999999999997e-07 2024-03-20T09:35:52.266 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.266 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.266 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:52.266 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.016418+0000", 2024-03-20T09:35:52.266 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0939999999999998e-06 2024-03-20T09:35:52.266 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.266 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.266 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:52.266 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.016430+0000", 2024-03-20T09:35:52.266 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2795e-05 2024-03-20T09:35:52.267 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.267 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.267 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:52.267 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.016438+0000", 2024-03-20T09:35:52.267 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.1500000000000002e-06 2024-03-20T09:35:52.267 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.267 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.267 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:52.267 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.077379+0000", 2024-03-20T09:35:52.267 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.060941654999999997 2024-03-20T09:35:52.267 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.267 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.267 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:35:52.267 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.077387+0000", 2024-03-20T09:35:52.267 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.1019999999999997e-06 2024-03-20T09:35:52.268 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.268 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.268 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:52.268 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.077400+0000", 2024-03-20T09:35:52.268 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2761e-05 2024-03-20T09:35:52.268 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.268 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:52.268 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.268 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.268 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.268 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "rep_scrubmap(1.15 e491 from shard 8)", 2024-03-20T09:35:52.268 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:48.017154+0000", 2024-03-20T09:35:52.268 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 64.150142205999998, 2024-03-20T09:35:52.268 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.060297227000000002, 2024-03-20T09:35:52.268 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:52.269 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:35:52.269 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:52.269 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.269 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:52.269 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.017154+0000", 2024-03-20T09:35:52.269 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.269 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.269 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.269 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:52.269 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.017154+0000", 2024-03-20T09:35:52.269 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.269 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.269 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.269 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:52.269 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.017155+0000", 2024-03-20T09:35:52.270 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4199999999999995e-07 2024-03-20T09:35:52.270 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.270 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.270 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:52.270 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.017159+0000", 2024-03-20T09:35:52.270 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7100000000000001e-06 2024-03-20T09:35:52.270 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.270 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.270 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:52.270 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.017167+0000", 2024-03-20T09:35:52.270 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.4659999999999999e-06 2024-03-20T09:35:52.270 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.270 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.270 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:52.271 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.017172+0000", 2024-03-20T09:35:52.271 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0409999999999997e-06 2024-03-20T09:35:52.271 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.271 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.271 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:52.271 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.077425+0000", 2024-03-20T09:35:52.271 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.06025316 2024-03-20T09:35:52.271 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.271 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.271 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:35:52.271 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.077432+0000", 2024-03-20T09:35:52.271 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.7270000000000001e-06 2024-03-20T09:35:52.271 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.271 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.272 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:52.272 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.077452+0000", 2024-03-20T09:35:52.272 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9480999999999999e-05 2024-03-20T09:35:52.272 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.272 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:52.272 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.272 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.272 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.272 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26605.0:222193 7.15 e943/936 7:abacefbc:::benchmark_data_smithi122_187077_object95582:head v 943'13503, mlcod=943'13501)", 2024-03-20T09:35:52.272 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:48.046124+0000", 2024-03-20T09:35:52.272 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 64.121172521999995, 2024-03-20T09:35:52.272 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.034419373000000003, 2024-03-20T09:35:52.272 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:52.272 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:52.272 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:52.273 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.273 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:52.273 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.046124+0000", 2024-03-20T09:35:52.273 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.273 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.273 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.273 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:52.273 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.046124+0000", 2024-03-20T09:35:52.273 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.273 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.273 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.273 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:52.273 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.046125+0000", 2024-03-20T09:35:52.273 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.003e-06 2024-03-20T09:35:52.273 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.274 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.274 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:52.274 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.046133+0000", 2024-03-20T09:35:52.274 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.9780000000000005e-06 2024-03-20T09:35:52.274 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.274 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.274 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:52.274 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.046147+0000", 2024-03-20T09:35:52.274 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4082e-05 2024-03-20T09:35:52.274 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.274 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.274 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:52.274 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.046155+0000", 2024-03-20T09:35:52.274 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.1810000000000001e-06 2024-03-20T09:35:52.275 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.275 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.275 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:52.275 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.077421+0000", 2024-03-20T09:35:52.275 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.031265293999999999 2024-03-20T09:35:52.275 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.275 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.275 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:35:52.275 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.077434+0000", 2024-03-20T09:35:52.275 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3246e-05 2024-03-20T09:35:52.275 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.275 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.275 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:35:52.275 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.080510+0000", 2024-03-20T09:35:52.275 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0030756389999999998 2024-03-20T09:35:52.276 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.276 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.276 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:52.276 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.080543+0000", 2024-03-20T09:35:52.276 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.3949999999999999e-05 2024-03-20T09:35:52.276 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.276 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:52.276 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.276 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.276 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.276 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26605.0:251375 7.8 e962/933 7:16ab4fa9:::benchmark_data_smithi122_187077_object124764:head v 962'14378, mlcod=962'14376)", 2024-03-20T09:35:52.276 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:35:22.597803+0000", 2024-03-20T09:35:52.276 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 29.569494004999999, 2024-03-20T09:35:52.277 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0082276220000000004, 2024-03-20T09:35:52.277 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:52.277 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:52.277 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:52.277 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.277 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:52.277 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:35:22.597803+0000", 2024-03-20T09:35:52.277 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.277 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.277 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.277 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:52.277 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:35:22.597803+0000", 2024-03-20T09:35:52.277 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.278 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.278 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.278 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:52.278 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:35:22.597804+0000", 2024-03-20T09:35:52.278 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.7400000000000002e-07 2024-03-20T09:35:52.278 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.278 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.278 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:52.278 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:35:22.597811+0000", 2024-03-20T09:35:52.278 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3760000000000002e-06 2024-03-20T09:35:52.278 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.278 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.278 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:52.278 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:35:22.597823+0000", 2024-03-20T09:35:52.279 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2459999999999999e-05 2024-03-20T09:35:52.279 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.279 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.279 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:52.279 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:35:22.597831+0000", 2024-03-20T09:35:52.279 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.6799999999999993e-06 2024-03-20T09:35:52.279 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.279 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.279 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:52.279 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:35:22.598282+0000", 2024-03-20T09:35:52.279 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00045069899999999999 2024-03-20T09:35:52.279 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.279 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.280 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:35:52.280 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:35:22.598303+0000", 2024-03-20T09:35:52.280 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.139e-05 2024-03-20T09:35:52.280 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.280 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.280 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:35:52.280 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:35:22.605990+0000", 2024-03-20T09:35:52.280 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0076872520000000003 2024-03-20T09:35:52.280 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.280 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.280 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:52.280 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:35:22.606030+0000", 2024-03-20T09:35:52.280 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.9891000000000001e-05 2024-03-20T09:35:52.280 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.281 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:52.281 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.281 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.281 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.281 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26605.0:251379 7.8 e962/933 7:1012d6a8:::benchmark_data_smithi122_187077_object124768:head v 962'14379, mlcod=962'14376)", 2024-03-20T09:35:52.281 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:35:22.598605+0000", 2024-03-20T09:35:52.281 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 29.568691184999999, 2024-03-20T09:35:52.281 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0088252480000000008, 2024-03-20T09:35:52.281 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:52.281 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:52.281 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:52.281 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.281 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:52.281 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:35:22.598605+0000", 2024-03-20T09:35:52.282 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.282 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.282 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.282 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:52.282 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:35:22.598605+0000", 2024-03-20T09:35:52.282 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.282 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.282 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.282 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:52.282 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:35:22.598606+0000", 2024-03-20T09:35:52.282 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0999999999999998e-07 2024-03-20T09:35:52.282 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.282 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.282 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:52.283 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:35:22.598611+0000", 2024-03-20T09:35:52.283 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3489999999999999e-06 2024-03-20T09:35:52.283 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.283 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.283 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:52.283 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:35:22.598624+0000", 2024-03-20T09:35:52.283 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2588999999999999e-05 2024-03-20T09:35:52.283 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.283 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.283 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:52.283 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:35:22.598630+0000", 2024-03-20T09:35:52.283 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2689999999999996e-06 2024-03-20T09:35:52.283 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.284 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.284 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:52.284 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:35:22.601394+0000", 2024-03-20T09:35:52.284 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00276399 2024-03-20T09:35:52.284 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.284 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.284 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:35:52.284 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:35:22.601423+0000", 2024-03-20T09:35:52.284 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8793999999999999e-05 2024-03-20T09:35:52.284 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.284 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.284 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:35:52.284 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:35:22.607412+0000", 2024-03-20T09:35:52.284 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0059888880000000004 2024-03-20T09:35:52.284 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.285 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.285 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:52.285 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:35:22.607431+0000", 2024-03-20T09:35:52.285 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8759000000000001e-05 2024-03-20T09:35:52.285 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.285 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:52.285 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.285 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.285 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.285 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26605.0:251573 7.10 e963/958 7:0a5f0cbe:::benchmark_data_smithi122_187077_object124962:head v 963'14375, mlcod=962'14373)", 2024-03-20T09:35:52.285 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:35:22.817161+0000", 2024-03-20T09:35:52.285 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 29.350135400999999, 2024-03-20T09:35:52.285 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0082699169999999999, 2024-03-20T09:35:52.285 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:35:52.286 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:35:52.286 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:35:52.286 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.286 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:35:52.286 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:35:22.817161+0000", 2024-03-20T09:35:52.286 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.286 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.286 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.286 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:35:52.286 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:35:22.817161+0000", 2024-03-20T09:35:52.286 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:35:52.286 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.286 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.287 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:35:52.287 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:35:22.817161+0000", 2024-03-20T09:35:52.287 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.4400000000000001e-07 2024-03-20T09:35:52.287 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.287 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.287 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:35:52.287 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:35:22.817169+0000", 2024-03-20T09:35:52.287 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.0249999999999997e-06 2024-03-20T09:35:52.287 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.287 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.287 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:35:52.287 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:35:22.817189+0000", 2024-03-20T09:35:52.287 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0687e-05 2024-03-20T09:35:52.287 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.287 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.288 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:35:52.288 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:35:22.817199+0000", 2024-03-20T09:35:52.288 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.8339999999999994e-06 2024-03-20T09:35:52.288 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.288 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.288 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:35:52.288 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:35:22.817255+0000", 2024-03-20T09:35:52.288 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6382999999999999e-05 2024-03-20T09:35:52.288 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.288 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.288 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:35:52.288 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:35:22.817291+0000", 2024-03-20T09:35:52.288 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.5200999999999997e-05 2024-03-20T09:35:52.288 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.289 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.289 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:35:52.289 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:35:22.825391+0000", 2024-03-20T09:35:52.289 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0081000010000000008 2024-03-20T09:35:52.289 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:35:52.289 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:35:52.289 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:35:52.289 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:35:22.825431+0000", 2024-03-20T09:35:52.289 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.0541999999999999e-05 2024-03-20T09:35:52.289 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.289 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:52.289 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.289 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:35:52.289 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:35:52.289 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:35:52.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:52.291 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:35:52.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:35:52.291 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:35:52.292 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:35:52.392 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:35:52.423 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:52.500 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:52 smithi026 ceph-mon[68309]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-20T09:35:52.500 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:52 smithi026 ceph-mon[68309]: osdmap e975: 12 total, 12 up, 12 in 2024-03-20T09:35:52.500 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:52 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:35:52.500 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:52 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:35:52.508 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:52 smithi115 ceph-mon[72476]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-20T09:35:52.508 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:52 smithi115 ceph-mon[72476]: osdmap e975: 12 total, 12 up, 12 in 2024-03-20T09:35:52.508 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:52 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:35:52.508 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:52 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:35:52.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:52 smithi122 ceph-mon[73495]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-20T09:35:52.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:52 smithi122 ceph-mon[73495]: osdmap e975: 12 total, 12 up, 12 in 2024-03-20T09:35:52.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:52 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:35:52.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:52 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:35:52.857 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_ops_in_flight 2024-03-20T09:35:52.873 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:35:52.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:52.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:35:52.977 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:35:53.005 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:53.025 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:35:53.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:53.028 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:35:53.129 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:35:53.133 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:35:53.157 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:53.273 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:35:53.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:53 smithi122 ceph-mon[73495]: osdmap e976: 12 total, 12 up, 12 in 2024-03-20T09:35:53.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:53 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/199720624' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:35:53.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:53 smithi122 ceph-mon[73495]: pgmap v2354: 47 pgs: 3 creating+activating, 6 creating+peering, 7 unknown, 31 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:35:53.439 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:35:53.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:53.442 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:35:53.543 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:35:53.571 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:53 smithi026 ceph-mon[68309]: osdmap e976: 12 total, 12 up, 12 in 2024-03-20T09:35:53.571 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:53 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/199720624' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:35:53.571 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:53 smithi026 ceph-mon[68309]: pgmap v2354: 47 pgs: 3 creating+activating, 6 creating+peering, 7 unknown, 31 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:35:53.573 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:53.611 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:35:53.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:53.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:35:53.646 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:53 smithi115 ceph-mon[72476]: osdmap e976: 12 total, 12 up, 12 in 2024-03-20T09:35:53.646 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:53 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/199720624' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:35:53.646 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:53 smithi115 ceph-mon[72476]: pgmap v2354: 47 pgs: 3 creating+activating, 6 creating+peering, 7 unknown, 31 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:35:53.716 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:35:53.745 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:53.766 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:35:53.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:53.769 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:35:53.791 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 8, 11, 3, 4, 9, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 11, 5, 10] 2024-03-20T09:35:53.791 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:35:53.792 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-20T09:35:53.792 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.030717 2024-03-20T09:35:53.792 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd primary-affinity 1 0.030716924881176055 2024-03-20T09:35:53.869 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:35:53.898 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:53.936 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:35:53.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:53.939 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:35:54.040 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:35:54.070 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:54.227 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:35:54.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:54.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:35:54.332 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:35:54.365 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:54.386 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:35:54.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:54.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:35:54.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:54 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:35:54.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:54 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/199720624' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:35:54.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:54 smithi122 ceph-mon[73495]: osdmap e977: 12 total, 12 up, 12 in 2024-03-20T09:35:54.491 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:35:54.522 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:54.539 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:54 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:35:54.539 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:54 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/199720624' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:35:54.539 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:54 smithi026 ceph-mon[68309]: osdmap e977: 12 total, 12 up, 12 in 2024-03-20T09:35:54.542 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:35:54.544 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:54 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:35:54.544 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:54 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/199720624' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:35:54.544 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:54 smithi115 ceph-mon[72476]: osdmap e977: 12 total, 12 up, 12 in 2024-03-20T09:35:54.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:54.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:35:54.646 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:35:54.675 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:55.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:55 smithi122 ceph-mon[73495]: pgmap v2356: 47 pgs: 3 creating+activating, 6 creating+peering, 2 unknown, 36 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:35:55.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:55 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2647656692' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:35:55.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:55 smithi026 ceph-mon[68309]: pgmap v2356: 47 pgs: 3 creating+activating, 6 creating+peering, 2 unknown, 36 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:35:55.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:55 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2647656692' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:35:55.519 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:35:55.519 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:35:55.646 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:55 smithi115 ceph-mon[72476]: pgmap v2356: 47 pgs: 3 creating+activating, 6 creating+peering, 2 unknown, 36 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:35:55.646 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:55 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2647656692' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:35:55.666 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:35:55.666 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:35:55.666 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:35:55.666 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:35:55.666 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:35:55.666 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:35:55.666 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:35:55.666 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:35:55.666 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:35:55.667 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:35:55.667 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:35:55.667 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:35:55.667 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:35:55.667 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:35:55.667 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:35:55.667 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:35:55.667 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:35:55.667 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:35:55.667 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:35:55.667 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:35:55.668 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:35:55.668 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:35:55.668 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:35:55.668 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:35:56.151 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_blocked_ops 2024-03-20T09:35:56.331 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:35:56.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:56.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:35:56.411 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:35:56.436 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:35:56.461 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:35:56.464 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:56.712 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:35:56.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:56.715 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:35:56.818 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:35:56.845 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:56.883 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:35:56.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:56.887 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:35:56.988 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:35:57.014 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:57.578 INFO:teuthology.orchestra.run.smithi026.stderr:enabled application 'rados' on pool 'unique_pool_6' 2024-03-20T09:35:57.580 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:35:57.642 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:35:57.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:57.645 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:35:57.749 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:35:57.785 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:57.807 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:35:57.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:57.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:35:57.821 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:57 smithi122 ceph-mon[73495]: pgmap v2357: 47 pgs: 3 creating+activating, 44 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:35:57.821 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:57 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/729078007' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-20T09:35:57.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:57 smithi115 ceph-mon[72476]: pgmap v2357: 47 pgs: 3 creating+activating, 44 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:35:57.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:57 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/729078007' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-20T09:35:57.912 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:35:57.939 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:57 smithi026 ceph-mon[68309]: pgmap v2357: 47 pgs: 3 creating+activating, 44 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:35:57.939 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:57 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/729078007' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-20T09:35:57.941 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:58.076 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:35:58.076 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:35:58.258 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:35:58.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:58 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/729078007' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-20T09:35:58.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:58 smithi122 ceph-mon[73495]: osdmap e978: 12 total, 12 up, 12 in 2024-03-20T09:35:58.691 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:35:58.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:58.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:35:58.758 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_historic_ops 2024-03-20T09:35:58.796 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:35:58.827 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:58 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/729078007' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-20T09:35:58.828 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:58 smithi115 ceph-mon[72476]: osdmap e978: 12 total, 12 up, 12 in 2024-03-20T09:35:58.830 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:58.852 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:35:58.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:58.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:35:58.948 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:35:58.956 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:35:58.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:58 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/729078007' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-20T09:35:58.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:58 smithi026 ceph-mon[68309]: osdmap e978: 12 total, 12 up, 12 in 2024-03-20T09:35:58.987 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:59.008 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:35:59.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:59.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:35:59.019 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:35:59.112 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:35:59.140 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:59.160 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:35:59.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:59.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:35:59.219 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:35:59.264 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:35:59.292 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:59.311 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:35:59.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:59.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:35:59.415 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:35:59.447 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:59.467 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:35:59.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:59.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:35:59.572 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:35:59.601 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:35:59.704 DEBUG:teuthology.orchestra.run.smithi122:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_6 bench 90 write' 2024-03-20T09:35:59.707 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-20T09:35:59.707 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-20T09:35:59.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:59 smithi026 ceph-mon[68309]: pgmap v2359: 47 pgs: 47 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:35:59.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:59 smithi026 ceph-mon[68309]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-20T09:35:59.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:35:59 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1749777130' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.030716924881176055}]: dispatch 2024-03-20T09:35:59.754 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:hints = 1 2024-03-20T09:35:59.754 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-20T09:35:59.754 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Object prefix: benchmark_data_smithi122_197775 2024-03-20T09:35:59.754 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T09:35:59.754 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 0 0 0 0 0 0 - 0 2024-03-20T09:35:59.807 INFO:teuthology.orchestra.run.smithi026.stderr:set osd.1 primary-affinity to 0.0307169 (820132) 2024-03-20T09:35:59.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:59 smithi115 ceph-mon[72476]: pgmap v2359: 47 pgs: 47 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:35:59.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:59 smithi115 ceph-mon[72476]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-20T09:35:59.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:35:59 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1749777130' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.030716924881176055}]: dispatch 2024-03-20T09:35:59.900 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:59 smithi122 ceph-mon[73495]: pgmap v2359: 47 pgs: 47 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:35:59.900 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:59 smithi122 ceph-mon[73495]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-20T09:35:59.900 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:35:59 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1749777130' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.030716924881176055}]: dispatch 2024-03-20T09:35:59.905 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:35:59.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:35:59.913 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:36:00.014 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:36:00.065 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:00.225 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:36:00.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:00.230 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:36:00.331 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:36:00.371 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:00.754 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 1 15 2751 2736 170.992 171 0.00652935 0.00581944 2024-03-20T09:36:00.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:00 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:36:00.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:00 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:36:00.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:00 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1749777130' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.030716924881176055}]': finished 2024-03-20T09:36:00.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:00 smithi122 ceph-mon[73495]: osdmap e979: 12 total, 12 up, 12 in 2024-03-20T09:36:00.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:00 smithi122 ceph-mon[73495]: pgmap v2361: 47 pgs: 47 active+clean; 7.2 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s wr, 14 op/s 2024-03-20T09:36:00.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:00 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:36:00.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:00 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:36:00.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:00 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1749777130' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.030716924881176055}]': finished 2024-03-20T09:36:00.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:00 smithi026 ceph-mon[68309]: osdmap e979: 12 total, 12 up, 12 in 2024-03-20T09:36:00.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:00 smithi026 ceph-mon[68309]: pgmap v2361: 47 pgs: 47 active+clean; 7.2 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s wr, 14 op/s 2024-03-20T09:36:01.056 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:36:01.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:01.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:36:01.139 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:36:01.140 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:36:01.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:00 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:36:01.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:00 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:36:01.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:00 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1749777130' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.030716924881176055}]': finished 2024-03-20T09:36:01.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:00 smithi115 ceph-mon[72476]: osdmap e979: 12 total, 12 up, 12 in 2024-03-20T09:36:01.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:00 smithi115 ceph-mon[72476]: pgmap v2361: 47 pgs: 47 active+clean; 7.2 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s wr, 14 op/s 2024-03-20T09:36:01.161 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:36:01.198 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:01.755 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 2 16 5635 5619 175.486 180.188 0.00284902 0.00568631 2024-03-20T09:36:01.889 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_ops_in_flight 2024-03-20T09:36:02.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:02 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3763965389' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:36:02.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:02 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:36:02.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:02 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3763965389' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:36:02.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:02 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:36:02.275 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:36:02.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:02 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3763965389' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:36:02.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:02 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:36:02.455 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:36:02.755 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 3 15 7709 7694 160.224 129.688 0.00623852 0.00623234 2024-03-20T09:36:03.062 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:36:03.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:03.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:36:03.169 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:36:03.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:03 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:36:03.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:03 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:36:03.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:03 smithi122 ceph-mon[73495]: osdmap e980: 12 total, 12 up, 12 in 2024-03-20T09:36:03.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:03 smithi122 ceph-mon[73495]: pgmap v2363: 47 pgs: 47 active+clean; 110 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 290 op/s 2024-03-20T09:36:03.201 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:03.224 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:36:03.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:03.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:36:03.328 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:36:03.356 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:03 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:36:03.356 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:03 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:36:03.356 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:03 smithi115 ceph-mon[72476]: osdmap e980: 12 total, 12 up, 12 in 2024-03-20T09:36:03.356 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:03 smithi115 ceph-mon[72476]: pgmap v2363: 47 pgs: 47 active+clean; 110 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 290 op/s 2024-03-20T09:36:03.358 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:03.380 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:36:03.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:03.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:36:03.394 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:03 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:36:03.394 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:03 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:36:03.394 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:03 smithi026 ceph-mon[68309]: osdmap e980: 12 total, 12 up, 12 in 2024-03-20T09:36:03.394 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:03 smithi026 ceph-mon[68309]: pgmap v2363: 47 pgs: 47 active+clean; 110 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s wr, 290 op/s 2024-03-20T09:36:03.485 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:36:03.531 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:03.665 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:36:03.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:03.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:36:03.743 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:36:03.757 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 4 16 10770 10754 167.952 191.25 0.00283318 0.00594524 2024-03-20T09:36:03.770 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:36:03.775 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:36:03.809 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:36:03.812 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:03.847 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:36:03.849 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:36:03.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:03.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:36:03.873 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:36:03.897 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:36:03.917 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:36:03.936 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:36:03.952 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:36:03.956 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:36:03.976 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:36:03.978 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:04.001 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:36:04.003 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:04.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:36:04.037 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:36:04.072 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:36:04.074 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:36:04.075 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:36:04.075 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:36:04.075 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:36:04.075 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:36:04.075 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:36:04.075 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:36:04.075 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:36:04.075 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:36:04.075 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:36:04.075 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:36:04.075 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:36:04.105 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:36:04.137 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:04.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:04 smithi115 ceph-mon[72476]: 1.9 scrub starts 2024-03-20T09:36:04.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:04 smithi115 ceph-mon[72476]: 1.9 scrub ok 2024-03-20T09:36:04.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:04 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1810580022' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:36:04.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:04 smithi122 ceph-mon[73495]: 1.9 scrub starts 2024-03-20T09:36:04.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:04 smithi122 ceph-mon[73495]: 1.9 scrub ok 2024-03-20T09:36:04.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:04 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1810580022' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:36:04.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:04 smithi026 ceph-mon[68309]: 1.9 scrub starts 2024-03-20T09:36:04.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:04 smithi026 ceph-mon[68309]: 1.9 scrub ok 2024-03-20T09:36:04.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:04 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1810580022' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:36:04.756 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 5 16 12943 12927 161.524 135.812 0.00498657 0.00618234 2024-03-20T09:36:05.110 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:36:05.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:05 smithi115 ceph-mon[72476]: 1.c scrub starts 2024-03-20T09:36:05.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:05 smithi115 ceph-mon[72476]: 1.c scrub ok 2024-03-20T09:36:05.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:05 smithi115 ceph-mon[72476]: 1.a scrub starts 2024-03-20T09:36:05.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:05 smithi115 ceph-mon[72476]: 1.a scrub ok 2024-03-20T09:36:05.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:05 smithi115 ceph-mon[72476]: pgmap v2364: 47 pgs: 47 active+clean; 306 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 702 op/s 2024-03-20T09:36:05.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:05 smithi122 ceph-mon[73495]: 1.c scrub starts 2024-03-20T09:36:05.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:05 smithi122 ceph-mon[73495]: 1.c scrub ok 2024-03-20T09:36:05.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:05 smithi122 ceph-mon[73495]: 1.a scrub starts 2024-03-20T09:36:05.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:05 smithi122 ceph-mon[73495]: 1.a scrub ok 2024-03-20T09:36:05.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:05 smithi122 ceph-mon[73495]: pgmap v2364: 47 pgs: 47 active+clean; 306 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 702 op/s 2024-03-20T09:36:05.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:05 smithi026 ceph-mon[68309]: 1.c scrub starts 2024-03-20T09:36:05.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:05 smithi026 ceph-mon[68309]: 1.c scrub ok 2024-03-20T09:36:05.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:05 smithi026 ceph-mon[68309]: 1.a scrub starts 2024-03-20T09:36:05.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:05 smithi026 ceph-mon[68309]: 1.a scrub ok 2024-03-20T09:36:05.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:05 smithi026 ceph-mon[68309]: pgmap v2364: 47 pgs: 47 active+clean; 306 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 702 op/s 2024-03-20T09:36:05.529 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:36:05.630 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 8, 11, 3, 4, 9, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 11, 5, 10] 2024-03-20T09:36:05.630 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 0 1 2024-03-20T09:36:05.756 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 6 16 15709 15693 163.413 172.875 0.00764437 0.00611366 2024-03-20T09:36:05.899 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:36:05.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:05.905 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:36:06.006 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:36:06.052 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:06.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:06 smithi122 ceph-mon[73495]: 1.e scrub starts 2024-03-20T09:36:06.218 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:06 smithi122 ceph-mon[73495]: 1.15 scrub starts 2024-03-20T09:36:06.218 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:06 smithi122 ceph-mon[73495]: 1.15 scrub ok 2024-03-20T09:36:06.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:06 smithi115 ceph-mon[72476]: 1.e scrub starts 2024-03-20T09:36:06.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:06 smithi115 ceph-mon[72476]: 1.15 scrub starts 2024-03-20T09:36:06.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:06 smithi115 ceph-mon[72476]: 1.15 scrub ok 2024-03-20T09:36:06.453 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:36:06.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:06 smithi026 ceph-mon[68309]: 1.e scrub starts 2024-03-20T09:36:06.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:06 smithi026 ceph-mon[68309]: 1.15 scrub starts 2024-03-20T09:36:06.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:06 smithi026 ceph-mon[68309]: 1.15 scrub ok 2024-03-20T09:36:06.756 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 7 16 18108 18092 161.486 149.938 0.00156803 0.0061457 2024-03-20T09:36:07.182 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:36:07.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:07 smithi026 ceph-mon[68309]: 1.e scrub ok 2024-03-20T09:36:07.220 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:07 smithi026 ceph-mon[68309]: pgmap v2365: 47 pgs: 47 active+clean; 714 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s 2024-03-20T09:36:07.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:07 smithi115 ceph-mon[72476]: 1.e scrub ok 2024-03-20T09:36:07.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:07 smithi115 ceph-mon[72476]: pgmap v2365: 47 pgs: 47 active+clean; 714 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s 2024-03-20T09:36:07.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:07 smithi122 ceph-mon[73495]: 1.e scrub ok 2024-03-20T09:36:07.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:07 smithi122 ceph-mon[73495]: pgmap v2365: 47 pgs: 47 active+clean; 714 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.43k op/s 2024-03-20T09:36:07.756 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 8 15 20284 20269 158.307 136.062 0.00685038 0.00631216 2024-03-20T09:36:08.167 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:36:08.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:08.172 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:36:08.272 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:36:08.315 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:08 smithi026 ceph-mon[68309]: 1.1d scrub starts 2024-03-20T09:36:08.318 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:08.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:08 smithi115 ceph-mon[72476]: 1.1d scrub starts 2024-03-20T09:36:08.439 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:08 smithi122 ceph-mon[73495]: 1.1d scrub starts 2024-03-20T09:36:08.585 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:36:08.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:08.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:36:08.694 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:36:08.724 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:08.756 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 9 16 22775 22759 158.008 155.625 0.00543338 0.00632323 2024-03-20T09:36:08.827 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:36:09.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:09.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:36:09.150 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:36:09.200 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:09.227 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:36:09.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:09.229 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:36:09.330 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:36:09.356 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:09 smithi115 ceph-mon[72476]: 1.1d scrub ok 2024-03-20T09:36:09.357 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:09 smithi115 ceph-mon[72476]: pgmap v2366: 47 pgs: 47 active+clean; 961 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 110 MiB/s wr, 1.76k op/s 2024-03-20T09:36:09.359 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:09.382 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:36:09.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:09.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:36:09.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:09 smithi122 ceph-mon[73495]: 1.1d scrub ok 2024-03-20T09:36:09.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:09 smithi122 ceph-mon[73495]: pgmap v2366: 47 pgs: 47 active+clean; 961 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 110 MiB/s wr, 1.76k op/s 2024-03-20T09:36:09.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:09 smithi026 ceph-mon[68309]: 1.1d scrub ok 2024-03-20T09:36:09.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:09 smithi026 ceph-mon[68309]: pgmap v2366: 47 pgs: 47 active+clean; 961 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 110 MiB/s wr, 1.76k op/s 2024-03-20T09:36:09.487 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:36:09.520 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:09.543 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:36:09.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:09.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:36:09.647 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:36:09.678 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:09.701 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:36:09.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:09.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:36:09.756 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 10 16 24792 24776 154.812 126.062 0.00307301 0.00645397 2024-03-20T09:36:09.805 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:36:09.836 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:09.861 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:36:09.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:09.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:36:09.946 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:36:09.964 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:36:10.007 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:10.066 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:36:10.330 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:36:10.331 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-03-20T09:36:10.331 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:10.331 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.26917.0:26087 8.8 e981/974 8:1fa179d7:::benchmark_data_smithi122_197775_object26086:head v 981'1663, mlcod=981'1661)", 2024-03-20T09:36:10.331 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:36:10.334684+0000", 2024-03-20T09:36:10.331 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.001777619, 2024-03-20T09:36:10.331 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001806261, 2024-03-20T09:36:10.331 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:36:10.331 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-03-20T09:36:10.331 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:36:10.331 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:10.332 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:36:10.332 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:10.334684+0000", 2024-03-20T09:36:10.332 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:36:10.332 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:10.332 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:10.332 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:36:10.332 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:10.334684+0000", 2024-03-20T09:36:10.332 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:36:10.332 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:10.332 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:10.332 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:36:10.332 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:10.334685+0000", 2024-03-20T09:36:10.332 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6099999999999999e-07 2024-03-20T09:36:10.333 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:10.333 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:10.333 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:36:10.333 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:10.334788+0000", 2024-03-20T09:36:10.333 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000103304 2024-03-20T09:36:10.333 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:10.333 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:10.333 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:36:10.333 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:10.334801+0000", 2024-03-20T09:36:10.333 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2463000000000001e-05 2024-03-20T09:36:10.333 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:10.333 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:10.333 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:36:10.333 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:10.334809+0000", 2024-03-20T09:36:10.334 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9160000000000001e-06 2024-03-20T09:36:10.334 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:10.334 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:10.334 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:36:10.334 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:10.334876+0000", 2024-03-20T09:36:10.334 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7069000000000004e-05 2024-03-20T09:36:10.334 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:10.334 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:10.334 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:36:10.334 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:10.334897+0000", 2024-03-20T09:36:10.334 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1549e-05 2024-03-20T09:36:10.335 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:36:10.335 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:36:10.335 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:36:10.335 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:10.335 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:10.335 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.26917.0:26088 8.0 e981/974 8:0dfd7b29:::benchmark_data_smithi122_197775_object26087:head v 981'1569, mlcod=981'1567)", 2024-03-20T09:36:10.335 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:36:10.335665+0000", 2024-03-20T09:36:10.335 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.000796534, 2024-03-20T09:36:10.335 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00089145499999999998, 2024-03-20T09:36:10.335 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:36:10.335 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-03-20T09:36:10.335 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:36:10.335 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:10.335 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:36:10.336 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:10.335665+0000", 2024-03-20T09:36:10.336 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:36:10.336 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:10.336 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:10.336 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:36:10.336 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:10.335665+0000", 2024-03-20T09:36:10.336 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:36:10.336 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:10.336 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:10.336 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:36:10.336 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:10.335666+0000", 2024-03-20T09:36:10.336 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2200000000000003e-07 2024-03-20T09:36:10.336 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:10.337 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:10.337 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:36:10.337 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:10.335702+0000", 2024-03-20T09:36:10.337 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6350999999999998e-05 2024-03-20T09:36:10.337 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:10.337 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:10.337 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:36:10.337 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:10.335714+0000", 2024-03-20T09:36:10.337 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1391e-05 2024-03-20T09:36:10.337 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:10.337 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:10.338 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:36:10.338 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:10.335721+0000", 2024-03-20T09:36:10.338 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.844e-06 2024-03-20T09:36:10.338 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:10.338 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:10.338 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:36:10.338 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:10.335769+0000", 2024-03-20T09:36:10.338 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8764000000000003e-05 2024-03-20T09:36:10.338 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:10.338 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:10.338 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:36:10.338 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:10.335790+0000", 2024-03-20T09:36:10.338 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0458999999999998e-05 2024-03-20T09:36:10.339 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:36:10.339 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:36:10.339 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:36:10.339 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:10.339 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:10.339 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:26091 8.1 8.cb4401c1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e981)", 2024-03-20T09:36:10.339 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:36:10.336521+0000", 2024-03-20T09:36:10.339 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 4294967295.9999413, 2024-03-20T09:36:10.339 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.8939999999999998e-05, 2024-03-20T09:36:10.340 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:36:10.340 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "queued for pg", 2024-03-20T09:36:10.340 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:36:10.340 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:36:10.340 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:36:10.340 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 26091 2024-03-20T09:36:10.340 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:10.340 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:36:10.341 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:10.341 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:36:10.341 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:10.336521+0000", 2024-03-20T09:36:10.341 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:36:10.341 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:10.341 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:10.341 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:36:10.341 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:10.336521+0000", 2024-03-20T09:36:10.341 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:36:10.341 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:10.341 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:10.341 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:36:10.341 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:10.336522+0000", 2024-03-20T09:36:10.341 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3039999999999999e-06 2024-03-20T09:36:10.342 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:10.342 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:10.342 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:36:10.342 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:10.336547+0000", 2024-03-20T09:36:10.342 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4658999999999999e-05 2024-03-20T09:36:10.342 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:10.342 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:10.342 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:36:10.342 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:10.336561+0000", 2024-03-20T09:36:10.342 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4087000000000001e-05 2024-03-20T09:36:10.342 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:10.342 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:10.343 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:36:10.343 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:10.336568+0000", 2024-03-20T09:36:10.343 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7449999999999995e-06 2024-03-20T09:36:10.343 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:36:10.343 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:36:10.343 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:36:10.343 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:36:10.343 INFO:teuthology.orchestra.run.smithi122.stdout: ], 2024-03-20T09:36:10.343 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 3 2024-03-20T09:36:10.343 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:36:10.358 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:10 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3980864280' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:36:10.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:10 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3980864280' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:36:10.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:10 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3980864280' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:36:10.757 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 11 16 27080 27064 153.736 143 0.00655671 0.00649868 2024-03-20T09:36:11.379 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:11 smithi122 ceph-mon[73495]: 1.6 scrub starts 2024-03-20T09:36:11.380 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:11 smithi122 ceph-mon[73495]: 1.6 scrub ok 2024-03-20T09:36:11.380 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:11 smithi122 ceph-mon[73495]: 1.1 scrub starts 2024-03-20T09:36:11.380 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:11 smithi122 ceph-mon[73495]: 1.1 scrub ok 2024-03-20T09:36:11.380 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:11 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:36:11.380 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:11 smithi122 ceph-mon[73495]: Cluster is now healthy 2024-03-20T09:36:11.380 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:11 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3980864280' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:36:11.380 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:11 smithi122 ceph-mon[73495]: osdmap e981: 12 total, 12 up, 12 in 2024-03-20T09:36:11.380 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:11 smithi122 ceph-mon[73495]: pgmap v2368: 47 pgs: 47 active+clean; 1.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 132 MiB/s wr, 2.11k op/s 2024-03-20T09:36:11.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:11 smithi115 ceph-mon[72476]: 1.6 scrub starts 2024-03-20T09:36:11.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:11 smithi115 ceph-mon[72476]: 1.6 scrub ok 2024-03-20T09:36:11.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:11 smithi115 ceph-mon[72476]: 1.1 scrub starts 2024-03-20T09:36:11.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:11 smithi115 ceph-mon[72476]: 1.1 scrub ok 2024-03-20T09:36:11.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:11 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:36:11.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:11 smithi115 ceph-mon[72476]: Cluster is now healthy 2024-03-20T09:36:11.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:11 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3980864280' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:36:11.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:11 smithi115 ceph-mon[72476]: osdmap e981: 12 total, 12 up, 12 in 2024-03-20T09:36:11.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:11 smithi115 ceph-mon[72476]: pgmap v2368: 47 pgs: 47 active+clean; 1.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 132 MiB/s wr, 2.11k op/s 2024-03-20T09:36:11.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:11 smithi026 ceph-mon[68309]: 1.6 scrub starts 2024-03-20T09:36:11.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:11 smithi026 ceph-mon[68309]: 1.6 scrub ok 2024-03-20T09:36:11.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:11 smithi026 ceph-mon[68309]: 1.1 scrub starts 2024-03-20T09:36:11.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:11 smithi026 ceph-mon[68309]: 1.1 scrub ok 2024-03-20T09:36:11.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:11 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:36:11.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:11 smithi026 ceph-mon[68309]: Cluster is now healthy 2024-03-20T09:36:11.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:11 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3980864280' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:36:11.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:11 smithi026 ceph-mon[68309]: osdmap e981: 12 total, 12 up, 12 in 2024-03-20T09:36:11.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:11 smithi026 ceph-mon[68309]: pgmap v2368: 47 pgs: 47 active+clean; 1.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 132 MiB/s wr, 2.11k op/s 2024-03-20T09:36:11.757 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 12 16 29068 29052 151.278 124.25 0.00290506 0.00660585 2024-03-20T09:36:11.786 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_blocked_ops 2024-03-20T09:36:12.078 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:12 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3604750647' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-20T09:36:12.086 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.0 to 1 (10000) 2024-03-20T09:36:12.243 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:36:12.390 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:12 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3604750647' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-20T09:36:12.390 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:12 smithi026 ceph-mon[68309]: 1.10 scrub starts 2024-03-20T09:36:12.390 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:12 smithi026 ceph-mon[68309]: 1.10 scrub ok 2024-03-20T09:36:12.390 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:12 smithi026 ceph-mon[68309]: 1.13 deep-scrub starts 2024-03-20T09:36:12.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:12 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3604750647' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-20T09:36:12.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:12 smithi115 ceph-mon[72476]: 1.10 scrub starts 2024-03-20T09:36:12.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:12 smithi115 ceph-mon[72476]: 1.10 scrub ok 2024-03-20T09:36:12.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:12 smithi115 ceph-mon[72476]: 1.13 deep-scrub starts 2024-03-20T09:36:12.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:12 smithi122 ceph-mon[73495]: 1.10 scrub starts 2024-03-20T09:36:12.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:12 smithi122 ceph-mon[73495]: 1.10 scrub ok 2024-03-20T09:36:12.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:12 smithi122 ceph-mon[73495]: 1.13 deep-scrub starts 2024-03-20T09:36:12.757 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 13 16 30737 30721 147.665 104.312 0.199846 0.0067202 2024-03-20T09:36:13.271 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:36:13.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:13.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:36:13.275 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:13 smithi026 ceph-mon[68309]: 1.13 deep-scrub ok 2024-03-20T09:36:13.275 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:13 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3604750647' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-20T09:36:13.276 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:13 smithi026 ceph-mon[68309]: osdmap e982: 12 total, 12 up, 12 in 2024-03-20T09:36:13.276 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:13 smithi026 ceph-mon[68309]: pgmap v2370: 47 pgs: 47 active+clean; 1.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 161 MiB/s wr, 2.57k op/s 2024-03-20T09:36:13.376 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:36:13.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:13 smithi115 ceph-mon[72476]: 1.13 deep-scrub ok 2024-03-20T09:36:13.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:13 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3604750647' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-20T09:36:13.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:13 smithi115 ceph-mon[72476]: osdmap e982: 12 total, 12 up, 12 in 2024-03-20T09:36:13.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:13 smithi115 ceph-mon[72476]: pgmap v2370: 47 pgs: 47 active+clean; 1.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 161 MiB/s wr, 2.57k op/s 2024-03-20T09:36:13.427 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:13.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:13 smithi122 ceph-mon[73495]: 1.13 deep-scrub ok 2024-03-20T09:36:13.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:13 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3604750647' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-20T09:36:13.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:13 smithi122 ceph-mon[73495]: osdmap e982: 12 total, 12 up, 12 in 2024-03-20T09:36:13.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:13 smithi122 ceph-mon[73495]: pgmap v2370: 47 pgs: 47 active+clean; 1.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 161 MiB/s wr, 2.57k op/s 2024-03-20T09:36:13.758 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 14 16 32867 32851 146.626 133.125 0.00202408 0.00680614 2024-03-20T09:36:14.757 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 15 16 34418 34402 143.314 96.9375 0.011566 0.0069723 2024-03-20T09:36:15.591 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:36:15.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:15.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:36:15.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:15 smithi122 ceph-mon[73495]: pgmap v2371: 47 pgs: 47 active+clean; 1.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 135 MiB/s wr, 2.15k op/s 2024-03-20T09:36:15.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:15 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:36:15.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:15 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:36:15.696 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:36:15.753 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:15.758 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 1 1 2024-03-20T09:36:15.759 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 16 16 36364 36348 141.941 121.625 0.00622506 0.00703985 2024-03-20T09:36:15.790 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:36:15.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:15.796 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:36:15.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:15 smithi115 ceph-mon[72476]: pgmap v2371: 47 pgs: 47 active+clean; 1.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 135 MiB/s wr, 2.15k op/s 2024-03-20T09:36:15.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:15 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:36:15.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:15 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:36:15.897 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:36:15.915 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:36:15.915 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:36:15.915 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-03-20T09:36:15.915 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-03-20T09:36:15.915 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:36:15.927 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:15.949 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:36:15.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:15.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:36:15.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:15 smithi026 ceph-mon[68309]: pgmap v2371: 47 pgs: 47 active+clean; 1.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 135 MiB/s wr, 2.15k op/s 2024-03-20T09:36:15.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:15 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:36:15.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:15 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:36:16.054 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:36:16.084 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:16.106 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:36:16.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:16.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:36:16.178 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:36:16.210 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:36:16.262 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:16.597 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:16 smithi122 ceph-mon[73495]: osdmap e983: 12 total, 12 up, 12 in 2024-03-20T09:36:16.692 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:16 smithi026 ceph-mon[68309]: osdmap e983: 12 total, 12 up, 12 in 2024-03-20T09:36:16.759 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 17 16 38315 38299 140.764 121.938 0.00462604 0.00709942 2024-03-20T09:36:16.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:16 smithi115 ceph-mon[72476]: osdmap e983: 12 total, 12 up, 12 in 2024-03-20T09:36:17.427 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:36:17.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:17.432 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:36:17.445 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:36:17.446 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:36:17.446 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:36:17.533 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:36:17.574 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:17.606 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:36:17.682 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:36:17.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:17 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:36:17.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:17 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:36:17.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:17 smithi122 ceph-mon[73495]: pgmap v2373: 47 pgs: 47 active+clean; 2.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 138 MiB/s wr, 2.21k op/s 2024-03-20T09:36:17.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:17 smithi122 ceph-mon[73495]: osdmap e984: 12 total, 12 up, 12 in 2024-03-20T09:36:17.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:17 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:36:17.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:17 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:36:17.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:17 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3224455486' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:36:17.714 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:36:17.749 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:36:17.759 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 18 16 39894 39878 138.426 98.6875 0.0130251 0.00721911 2024-03-20T09:36:17.781 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:36:17.814 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:36:17.836 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:36:17.881 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:36:17.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:17 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:36:17.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:17 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:36:17.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:17 smithi115 ceph-mon[72476]: pgmap v2373: 47 pgs: 47 active+clean; 2.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 138 MiB/s wr, 2.21k op/s 2024-03-20T09:36:17.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:17 smithi115 ceph-mon[72476]: osdmap e984: 12 total, 12 up, 12 in 2024-03-20T09:36:17.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:17 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:36:17.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:17 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:36:17.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:17 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3224455486' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:36:17.916 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:36:17.948 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:36:17.949 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:36:17.950 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:36:17.950 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:36:17.950 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:36:17.950 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:36:17.950 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:36:17.950 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:36:17.950 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:36:17.950 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:36:17.950 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:36:17.950 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:36:17.950 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:36:17.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:17 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:36:17.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:17 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:36:17.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:17 smithi026 ceph-mon[68309]: pgmap v2373: 47 pgs: 47 active+clean; 2.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 138 MiB/s wr, 2.21k op/s 2024-03-20T09:36:17.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:17 smithi026 ceph-mon[68309]: osdmap e984: 12 total, 12 up, 12 in 2024-03-20T09:36:17.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:17 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:36:17.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:17 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:36:17.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:17 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3224455486' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:36:18.417 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_historic_ops 2024-03-20T09:36:18.688 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:18 smithi122 ceph-mon[73495]: osdmap e985: 12 total, 12 up, 12 in 2024-03-20T09:36:18.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:18 smithi026 ceph-mon[68309]: osdmap e985: 12 total, 12 up, 12 in 2024-03-20T09:36:18.759 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 19 16 41718 41702 137.14 114 0.0035063 0.0072868 2024-03-20T09:36:18.848 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:36:18.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:18 smithi115 ceph-mon[72476]: osdmap e985: 12 total, 12 up, 12 in 2024-03-20T09:36:18.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:18.933 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:36:18.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:36:19.035 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:36:19.083 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:19.147 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:36:19.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:19.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:36:19.254 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:36:19.283 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:19.305 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:36:19.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:19.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:36:19.409 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:36:19.439 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:19.463 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:36:19.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:19.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:36:19.568 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:36:19.607 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:19.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:19 smithi026 ceph-mon[68309]: pgmap v2376: 47 pgs: 47 active+clean; 2.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 111 MiB/s wr, 1.77k op/s 2024-03-20T09:36:19.759 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:2024-03-20T09:36:19.770335+0000 min lat: 0.00110904 max lat: 0.219489 avg lat: 0.00734564 2024-03-20T09:36:19.759 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T09:36:19.759 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 20 16 43563 43547 136.048 115.312 0.00329908 0.00734564 2024-03-20T09:36:19.799 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:36:19.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:19.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:36:19.898 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:19 smithi115 ceph-mon[72476]: pgmap v2376: 47 pgs: 47 active+clean; 2.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 111 MiB/s wr, 1.77k op/s 2024-03-20T09:36:19.913 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:36:19.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:19 smithi122 ceph-mon[73495]: pgmap v2376: 47 pgs: 47 active+clean; 2.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 111 MiB/s wr, 1.77k op/s 2024-03-20T09:36:19.948 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:20.188 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:36:20.646 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.1 to 1 (10000) 2024-03-20T09:36:20.759 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 21 16 45281 45265 134.681 107.375 0.0049396 0.00742066 2024-03-20T09:36:20.804 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:20 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3621865781' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-20T09:36:20.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:20 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3621865781' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-20T09:36:20.922 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:20 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3621865781' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-20T09:36:21.644 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:21 smithi026 ceph-mon[68309]: pgmap v2377: 47 pgs: 47 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 113 MiB/s wr, 1.80k op/s 2024-03-20T09:36:21.644 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:21 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3621865781' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-20T09:36:21.644 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:21 smithi026 ceph-mon[68309]: osdmap e986: 12 total, 12 up, 12 in 2024-03-20T09:36:21.703 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:36:21.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:21.715 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:36:21.760 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 22 16 47099 47083 133.724 113.625 0.00614933 0.00747296 2024-03-20T09:36:21.816 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:36:21.840 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:36:21.845 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:21.866 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:36:21.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:21.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:36:21.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:21 smithi115 ceph-mon[72476]: pgmap v2377: 47 pgs: 47 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 113 MiB/s wr, 1.80k op/s 2024-03-20T09:36:21.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:21 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3621865781' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-20T09:36:21.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:21 smithi115 ceph-mon[72476]: osdmap e986: 12 total, 12 up, 12 in 2024-03-20T09:36:21.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:21 smithi122 ceph-mon[73495]: pgmap v2377: 47 pgs: 47 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 113 MiB/s wr, 1.80k op/s 2024-03-20T09:36:21.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:21 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3621865781' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-20T09:36:21.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:21 smithi122 ceph-mon[73495]: osdmap e986: 12 total, 12 up, 12 in 2024-03-20T09:36:21.970 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:36:22.030 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:22.066 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:36:22.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:22.071 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:36:22.172 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:36:22.222 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:22.442 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:36:22.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:22.446 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:36:22.547 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:36:22.576 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:22.601 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:36:22.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:22.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:36:22.693 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:36:22.693 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:36:22.705 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:36:22.732 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:22.750 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:36:22.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:22.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:36:22.760 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 23 16 48684 48668 132.217 99.0625 0.00296287 0.0075358 2024-03-20T09:36:22.853 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:36:22.894 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:23.141 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:36:23.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:23.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:36:23.351 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:36:23.399 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:23.760 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 24 16 49929 49913 129.95 77.8125 0.00799177 0.00769091 2024-03-20T09:36:23.863 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:36:23.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:23.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:36:23.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:23 smithi115 ceph-mon[72476]: pgmap v2379: 47 pgs: 47 active+clean; 2.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 111 MiB/s wr, 1.77k op/s 2024-03-20T09:36:23.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:23 smithi115 ceph-mon[72476]: osdmap e987: 12 total, 12 up, 12 in 2024-03-20T09:36:23.898 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:23 smithi026 ceph-mon[68309]: pgmap v2379: 47 pgs: 47 active+clean; 2.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 111 MiB/s wr, 1.77k op/s 2024-03-20T09:36:23.898 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:23 smithi026 ceph-mon[68309]: osdmap e987: 12 total, 12 up, 12 in 2024-03-20T09:36:23.899 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 2 1 2024-03-20T09:36:23.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:23 smithi122 ceph-mon[73495]: pgmap v2379: 47 pgs: 47 active+clean; 2.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 111 MiB/s wr, 1.77k op/s 2024-03-20T09:36:23.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:23 smithi122 ceph-mon[73495]: osdmap e987: 12 total, 12 up, 12 in 2024-03-20T09:36:23.972 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:36:24.004 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:24.022 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:36:24.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:24.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:36:24.127 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:36:24.172 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:24.531 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:36:24.589 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:36:24.669 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:36:24.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:24.675 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:36:24.689 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_ops_in_flight 2024-03-20T09:36:24.760 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 25 16 51524 51508 128.739 99.6875 0.00536053 0.00776165 2024-03-20T09:36:24.776 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:36:24.820 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:24.863 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:36:24.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:24.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:36:24.973 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:36:25.002 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:25.024 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:36:25.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:25.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:36:25.128 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:36:25.147 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:36:25.168 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:25.377 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:36:25.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:25.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:36:25.485 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:36:25.515 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:25.536 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:36:25.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:25.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:36:25.640 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:36:25.664 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:36:25.668 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:25.673 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:25 smithi026 ceph-mon[68309]: pgmap v2381: 47 pgs: 47 active+clean; 2.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s wr, 1.68k op/s 2024-03-20T09:36:25.673 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:25 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1105656407' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:36:25.673 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:25 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:36:25.694 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:36:25.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:25.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:36:25.760 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 26 15 52902 52887 127.103 86.1875 0.00321932 0.00786352 2024-03-20T09:36:25.798 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:36:25.837 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:25 smithi122 ceph-mon[73495]: pgmap v2381: 47 pgs: 47 active+clean; 2.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s wr, 1.68k op/s 2024-03-20T09:36:25.837 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:25 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1105656407' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:36:25.837 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:25 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:36:25.846 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:25.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:25 smithi115 ceph-mon[72476]: pgmap v2381: 47 pgs: 47 active+clean; 2.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 105 MiB/s wr, 1.68k op/s 2024-03-20T09:36:25.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:25 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1105656407' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:36:25.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:25 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:36:26.412 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:36:26.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:26.417 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:36:26.518 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:36:26.550 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:26.572 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:36:26.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:26.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:36:26.676 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:36:26.719 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:26 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:36:26.719 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:26 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:36:26.719 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:26 smithi122 ceph-mon[73495]: osdmap e988: 12 total, 12 up, 12 in 2024-03-20T09:36:26.748 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:26.760 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 27 15 54575 54560 126.268 104.562 0.00760656 0.00791552 2024-03-20T09:36:26.773 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:36:26.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:26.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:36:26.878 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:36:26.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:26 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:36:26.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:26 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:36:26.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:26 smithi115 ceph-mon[72476]: osdmap e988: 12 total, 12 up, 12 in 2024-03-20T09:36:26.908 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:26.931 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:36:26.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:26.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:36:26.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:26 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:36:26.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:26 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:36:26.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:26 smithi026 ceph-mon[68309]: osdmap e988: 12 total, 12 up, 12 in 2024-03-20T09:36:27.036 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:36:27.064 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:27.087 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:36:27.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:27.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:36:27.193 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:36:27.246 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:27.280 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:36:27.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:27.285 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:36:27.386 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:36:27.416 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:27.438 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:36:27.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:27.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:36:27.542 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:36:27.588 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:27.625 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:36:27.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:27.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:36:27.732 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:36:27.760 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 28 16 56154 56138 125.28 98.625 0.00415071 0.00797723 2024-03-20T09:36:27.763 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:27.787 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:36:27.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:27.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:36:27.893 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:36:27.920 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:27 smithi026 ceph-mon[68309]: pgmap v2383: 47 pgs: 47 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 116 MiB/s wr, 1.85k op/s 2024-03-20T09:36:27.920 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:27 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:36:27.920 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:27 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:36:27.923 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:27.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:27 smithi122 ceph-mon[73495]: pgmap v2383: 47 pgs: 47 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 116 MiB/s wr, 1.85k op/s 2024-03-20T09:36:27.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:27 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:36:27.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:27 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:36:28.124 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:36:28.124 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:36:28.133 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:36:28.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:28.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:36:28.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:27 smithi115 ceph-mon[72476]: pgmap v2383: 47 pgs: 47 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 116 MiB/s wr, 1.85k op/s 2024-03-20T09:36:28.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:27 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:36:28.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:27 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:36:28.238 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:36:28.267 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:28.289 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:36:28.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:28.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:36:28.395 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:36:28.442 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:28.687 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:28 smithi026 ceph-mon[68309]: osdmap e989: 12 total, 12 up, 12 in 2024-03-20T09:36:28.687 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:28 smithi026 ceph-mon[68309]: osdmap e990: 12 total, 12 up, 12 in 2024-03-20T09:36:28.762 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 29 16 57360 57344 123.555 75.375 0.0083244 0.00808865 2024-03-20T09:36:28.823 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:36:28.849 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:36:28.877 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:36:28.914 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:36:28.945 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:36:28.985 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:36:29.022 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:36:29.061 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:36:29.094 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:36:29.128 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:36:29.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:28 smithi115 ceph-mon[72476]: osdmap e989: 12 total, 12 up, 12 in 2024-03-20T09:36:29.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:28 smithi115 ceph-mon[72476]: osdmap e990: 12 total, 12 up, 12 in 2024-03-20T09:36:29.164 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:36:29.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:28 smithi122 ceph-mon[73495]: osdmap e989: 12 total, 12 up, 12 in 2024-03-20T09:36:29.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:28 smithi122 ceph-mon[73495]: osdmap e990: 12 total, 12 up, 12 in 2024-03-20T09:36:29.192 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:36:29.193 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:36:29.193 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:36:29.193 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:36:29.193 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:36:29.193 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:36:29.193 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:36:29.193 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:36:29.193 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:36:29.193 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:36:29.193 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:36:29.193 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:36:29.194 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:36:29.688 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:29 smithi026 ceph-mon[68309]: pgmap v2386: 47 pgs: 47 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.56k op/s 2024-03-20T09:36:29.688 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:29 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2565032535' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:36:29.748 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:36:29.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:29.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:36:29.761 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 30 16 58678 58662 122.183 82.375 0.0124019 0.00817959 2024-03-20T09:36:29.852 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:36:29.891 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:30.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:29 smithi115 ceph-mon[72476]: pgmap v2386: 47 pgs: 47 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.56k op/s 2024-03-20T09:36:30.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:29 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2565032535' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:36:30.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:29 smithi122 ceph-mon[73495]: pgmap v2386: 47 pgs: 47 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.56k op/s 2024-03-20T09:36:30.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:29 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2565032535' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:36:30.315 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_blocked_ops 2024-03-20T09:36:30.739 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:36:30.764 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 31 16 60168 60152 121.245 93.125 0.014456 0.00824267 2024-03-20T09:36:30.847 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:36:30.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:30.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:36:30.953 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:36:30.982 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:31.003 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:36:31.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:31.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:36:31.107 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:36:31.154 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:31.184 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:36:31.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:31.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:36:31.289 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:36:31.318 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:31.340 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:36:31.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:31.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:36:31.444 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:36:31.493 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:31.529 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:36:31.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:31.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:36:31.634 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:36:31.664 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:31.688 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:36:31.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:31.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:36:31.702 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:31 smithi122 ceph-mon[73495]: pgmap v2387: 47 pgs: 47 active+clean; 3.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-03-20T09:36:31.702 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:31 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/166427878' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:36:31.702 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:31 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:36:31.706 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.2 to 1 (10000) 2024-03-20T09:36:31.761 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 32 16 61559 61543 120.173 86.9375 0.00707138 0.00831689 2024-03-20T09:36:31.762 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:36:31.793 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:36:31.803 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:31 smithi026 ceph-mon[68309]: pgmap v2387: 47 pgs: 47 active+clean; 3.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-03-20T09:36:31.803 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:31 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/166427878' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:36:31.803 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:31 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:36:31.823 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:31.847 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:36:31.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:31.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:36:31.954 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:36:31.998 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:32.079 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:36:32.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:32.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:36:32.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:31 smithi115 ceph-mon[72476]: pgmap v2387: 47 pgs: 47 active+clean; 3.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-03-20T09:36:32.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:31 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/166427878' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:36:32.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:31 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:36:32.190 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:36:32.218 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:32.240 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:36:32.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:32.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:36:32.344 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:36:32.390 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:32.503 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:36:32.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:32.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:36:32.534 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 5 1 2024-03-20T09:36:32.609 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:36:32.658 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:36:32.676 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:32.694 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:36:32.717 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:36:32.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:32.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:36:32.761 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 33 16 62895 62879 119.061 83.5 0.0156454 0.00839475 2024-03-20T09:36:32.824 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:36:32.923 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:32.942 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:32 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-20T09:36:32.942 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:32 smithi122 ceph-mon[73495]: osdmap e991: 12 total, 12 up, 12 in 2024-03-20T09:36:32.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:32 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-20T09:36:32.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:32 smithi026 ceph-mon[68309]: osdmap e991: 12 total, 12 up, 12 in 2024-03-20T09:36:32.967 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:36:32.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:32.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:36:33.076 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:36:33.102 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:32 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-20T09:36:33.103 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:32 smithi115 ceph-mon[72476]: osdmap e991: 12 total, 12 up, 12 in 2024-03-20T09:36:33.106 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:33.128 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:36:33.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:33.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:36:33.234 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:36:33.315 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:33.357 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:36:33.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:33.363 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:36:33.402 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:36:33.402 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:36:33.403 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-03-20T09:36:33.403 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-03-20T09:36:33.403 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:36:33.464 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:36:33.512 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:33.539 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:36:33.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:33.544 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:36:33.646 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:36:33.650 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:36:33.674 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:33.696 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:36:33.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:33.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:36:33.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:33 smithi026 ceph-mon[68309]: pgmap v2389: 47 pgs: 47 active+clean; 3.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s 2024-03-20T09:36:33.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:33 smithi026 ceph-mon[68309]: osdmap e992: 12 total, 12 up, 12 in 2024-03-20T09:36:33.762 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 34 16 64270 64254 118.087 85.9375 0.00784215 0.0084641 2024-03-20T09:36:33.799 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:36:33.837 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:34.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:33 smithi115 ceph-mon[72476]: pgmap v2389: 47 pgs: 47 active+clean; 3.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s 2024-03-20T09:36:34.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:33 smithi115 ceph-mon[72476]: osdmap e992: 12 total, 12 up, 12 in 2024-03-20T09:36:34.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:33 smithi122 ceph-mon[73495]: pgmap v2389: 47 pgs: 47 active+clean; 3.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s 2024-03-20T09:36:34.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:33 smithi122 ceph-mon[73495]: osdmap e992: 12 total, 12 up, 12 in 2024-03-20T09:36:34.764 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 35 16 65680 65664 117.223 88.125 0.00845155 0.00852671 2024-03-20T09:36:34.991 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:36:35.003 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:36:35.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:35.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:36:35.111 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:36:35.140 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:35.167 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:36:35.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:35.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:36:35.270 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:36:35.301 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:35.322 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:36:35.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:35.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:36:35.428 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:36:35.457 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:35.481 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:36:35.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:35.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:36:35.587 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:36:35.639 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:35.764 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 36 16 67104 67088 116.439 89 0.00495072 0.00857867 2024-03-20T09:36:35.831 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:36:35.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:35.835 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:36:35.840 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_historic_ops 2024-03-20T09:36:35.936 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:36:35.967 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:35 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:36:35.968 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:35 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:36:35.968 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:35 smithi115 ceph-mon[72476]: pgmap v2391: 47 pgs: 47 active+clean; 3.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-03-20T09:36:35.968 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:35 smithi115 ceph-mon[72476]: osdmap e993: 12 total, 12 up, 12 in 2024-03-20T09:36:35.968 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:35.992 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:35 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:36:35.992 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:35 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:36:35.992 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:35 smithi026 ceph-mon[68309]: pgmap v2391: 47 pgs: 47 active+clean; 3.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-03-20T09:36:35.992 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:35 smithi026 ceph-mon[68309]: osdmap e993: 12 total, 12 up, 12 in 2024-03-20T09:36:35.992 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:36:35.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:35.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:36:36.097 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:36:36.146 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:36.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:35 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:36:36.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:35 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:36:36.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:35 smithi122 ceph-mon[73495]: pgmap v2391: 47 pgs: 47 active+clean; 3.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-03-20T09:36:36.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:35 smithi122 ceph-mon[73495]: osdmap e993: 12 total, 12 up, 12 in 2024-03-20T09:36:36.338 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:36:36.766 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 37 16 68392 68376 115.463 80.5 0.00785917 0.00865647 2024-03-20T09:36:36.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:36 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:36:36.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:36 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:36:36.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:36 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:36:36.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:36 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:36:37.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:36 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:36:37.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:36 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:36:37.705 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:36:37.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:37.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:36:37.766 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 38 16 69852 69836 114.826 91.25 0.0112016 0.00870496 2024-03-20T09:36:37.811 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:36:37.847 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:37.873 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:36:37.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:37.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:36:37.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:37 smithi026 ceph-mon[68309]: pgmap v2393: 47 pgs: 47 active+clean; 4.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.54k op/s 2024-03-20T09:36:37.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:37 smithi026 ceph-mon[68309]: osdmap e994: 12 total, 12 up, 12 in 2024-03-20T09:36:37.978 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:36:38.016 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:38.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:37 smithi115 ceph-mon[72476]: pgmap v2393: 47 pgs: 47 active+clean; 4.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.54k op/s 2024-03-20T09:36:38.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:37 smithi115 ceph-mon[72476]: osdmap e994: 12 total, 12 up, 12 in 2024-03-20T09:36:38.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:37 smithi122 ceph-mon[73495]: pgmap v2393: 47 pgs: 47 active+clean; 4.0 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.54k op/s 2024-03-20T09:36:38.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:37 smithi122 ceph-mon[73495]: osdmap e994: 12 total, 12 up, 12 in 2024-03-20T09:36:38.766 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 39 16 71083 71067 113.854 76.9375 0.00287626 0.00875637 2024-03-20T09:36:38.844 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:36:38.873 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:36:38.881 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:38.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:36:38.983 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:36:39.038 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:39.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:38 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3543353784' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:36:39.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:38 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:36:39.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:38 smithi115 ceph-mon[72476]: pgmap v2395: 47 pgs: 47 active+clean; 4.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-03-20T09:36:39.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:38 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3543353784' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:36:39.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:38 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:36:39.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:38 smithi122 ceph-mon[73495]: pgmap v2395: 47 pgs: 47 active+clean; 4.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-03-20T09:36:39.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:38 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3543353784' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:36:39.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:38 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:36:39.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:38 smithi026 ceph-mon[68309]: pgmap v2395: 47 pgs: 47 active+clean; 4.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-03-20T09:36:39.495 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:36:39.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:39.499 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:36:39.600 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:36:39.630 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:39.653 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:36:39.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:39.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:36:39.759 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:36:39.766 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:2024-03-20T09:36:39.777099+0000 min lat: 0.00110904 max lat: 0.219489 avg lat: 0.00886196 2024-03-20T09:36:39.766 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T09:36:39.766 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 40 16 72221 72205 112.786 71.125 0.0158016 0.00886196 2024-03-20T09:36:39.805 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:39.839 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:36:39.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:39.845 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:36:39.847 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:36:39.878 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:36:39.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:39 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:36:39.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:39 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:36:39.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:39 smithi115 ceph-mon[72476]: osdmap e995: 12 total, 12 up, 12 in 2024-03-20T09:36:39.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:39 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2054332635' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:36:39.907 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:36:39.941 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:36:39.946 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:36:39.974 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:36:39.977 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:40.004 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:36:40.005 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:36:40.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:40.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:36:40.037 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:36:40.070 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:36:40.099 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:36:40.110 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:36:40.137 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:36:40.150 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:39 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:36:40.150 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:39 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:36:40.151 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:39 smithi026 ceph-mon[68309]: osdmap e995: 12 total, 12 up, 12 in 2024-03-20T09:36:40.151 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:39 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2054332635' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:36:40.155 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:40.177 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:36:40.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:39 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:36:40.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:39 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:36:40.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:39 smithi122 ceph-mon[73495]: osdmap e995: 12 total, 12 up, 12 in 2024-03-20T09:36:40.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:39 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2054332635' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:36:40.214 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:36:40.220 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:36:40.220 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:36:40.220 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:36:40.221 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:36:40.221 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:36:40.221 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:36:40.221 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:36:40.221 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:36:40.221 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:36:40.222 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:36:40.222 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:36:40.222 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:36:40.425 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:36:40.425 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:36:40.766 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 41 16 73535 73519 112.038 82.125 0.00759139 0.00892057 2024-03-20T09:36:40.854 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.5 to 1 (10000) 2024-03-20T09:36:41.121 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:40 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1249786680' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:36:41.121 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:40 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:36:41.121 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:40 smithi026 ceph-mon[68309]: pgmap v2397: 47 pgs: 47 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-03-20T09:36:41.122 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:40 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1249786680' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:36:41.122 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:40 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:36:41.122 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:40 smithi122 ceph-mon[73495]: pgmap v2397: 47 pgs: 47 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-03-20T09:36:41.140 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:36:41.145 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:41.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:36:41.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:40 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1249786680' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:36:41.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:40 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:36:41.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:40 smithi115 ceph-mon[72476]: pgmap v2397: 47 pgs: 47 active+clean; 4.2 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s 2024-03-20T09:36:41.166 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-20T09:36:41.247 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:36:41.296 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:41.325 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:36:41.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:41.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:36:41.435 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:36:41.469 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:41.488 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:36:41.495 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:36:41.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:41.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:36:41.601 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:36:41.639 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:41.755 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:36:41.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:41.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:36:41.766 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 42 16 74947 74931 111.472 88.25 0.00580678 0.00896583 2024-03-20T09:36:41.803 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 6 1 2024-03-20T09:36:41.865 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:36:41.923 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:41.953 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:36:41.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:41.957 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:36:42.057 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:36:42.104 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:41 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-20T09:36:42.105 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:41 smithi122 ceph-mon[73495]: osdmap e996: 12 total, 12 up, 12 in 2024-03-20T09:36:42.112 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:42.127 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:41 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-20T09:36:42.127 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:41 smithi115 ceph-mon[72476]: osdmap e996: 12 total, 12 up, 12 in 2024-03-20T09:36:42.149 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:36:42.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:42.157 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:36:42.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:41 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-20T09:36:42.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:41 smithi026 ceph-mon[68309]: osdmap e996: 12 total, 12 up, 12 in 2024-03-20T09:36:42.230 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:36:42.258 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:36:42.330 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:42.357 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:36:42.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:42.361 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:36:42.462 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:36:42.496 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:42.518 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:36:42.769 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 43 15 76301 76286 110.841 84.6875 0.013062 0.00901727 2024-03-20T09:36:42.984 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:36:42.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:42.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:36:43.090 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:36:43.142 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:42 smithi122 ceph-mon[73495]: pgmap v2399: 47 pgs: 47 active+clean; 4.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-03-20T09:36:43.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:42 smithi115 ceph-mon[72476]: pgmap v2399: 47 pgs: 47 active+clean; 4.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-03-20T09:36:43.149 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:43.163 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:36:43.182 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:36:43.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:43.187 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:36:43.196 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:42 smithi026 ceph-mon[68309]: pgmap v2399: 47 pgs: 47 active+clean; 4.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-03-20T09:36:43.288 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:36:43.343 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:43.619 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:36:43.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:43.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:36:43.702 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:36:43.726 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:36:43.769 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 44 16 77638 77622 110.22 83.5 0.0134897 0.00906863 2024-03-20T09:36:43.773 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:43.809 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:36:43.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:43.817 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:36:43.888 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:36:43.888 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:36:43.918 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:36:43.965 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:43.994 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:36:44.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:44.001 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:36:44.101 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:36:44.136 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:44.160 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:36:44.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:44.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:36:44.266 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:36:44.328 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:44.360 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:36:44.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:44.367 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:36:44.468 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:36:44.500 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:44.747 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-20T09:36:44.751 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:36:44.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:44.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:36:44.769 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 45 16 78987 78971 109.644 84.3125 0.00573991 0.00911585 2024-03-20T09:36:44.857 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:36:44.890 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:44.916 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:36:44.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:44.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:36:45.020 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:36:45.065 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:45.094 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:36:45.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:45.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:36:45.109 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:36:45.202 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:36:45.251 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:45.258 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:36:45.274 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:36:45.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:45.277 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:36:45.378 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:36:45.440 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:45.463 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:36:45.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:45.466 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:36:45.567 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:36:45.595 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.6 to 1 (10000) 2024-03-20T09:36:45.595 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:45.616 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:36:45.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:45.620 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:36:45.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:45 smithi115 ceph-mon[72476]: pgmap v2400: 47 pgs: 47 active+clean; 4.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s 2024-03-20T09:36:45.646 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:45 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:36:45.646 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:45 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3957151243' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-20T09:36:45.646 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:45 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:36:45.646 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:45 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:36:45.721 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:36:45.768 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:45.768 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:45 smithi026 ceph-mon[68309]: pgmap v2400: 47 pgs: 47 active+clean; 4.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s 2024-03-20T09:36:45.768 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:45 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:36:45.768 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:45 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3957151243' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-20T09:36:45.768 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:45 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:36:45.768 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:45 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:36:45.769 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 46 16 79960 79944 108.583 60.8125 0.0121324 0.00920509 2024-03-20T09:36:45.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:45 smithi122 ceph-mon[73495]: pgmap v2400: 47 pgs: 47 active+clean; 4.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s 2024-03-20T09:36:45.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:45 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:36:45.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:45 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3957151243' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-20T09:36:45.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:45 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:36:45.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:45 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:36:46.341 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:36:46.348 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:46.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:36:46.449 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:36:46.492 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:46.524 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:36:46.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:46.529 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:36:46.630 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:36:46.682 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:46.770 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 47 16 81297 81281 108.05 83.5625 0.00727518 0.0092509 2024-03-20T09:36:46.802 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 7 1 2024-03-20T09:36:46.835 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:36:46.868 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:36:46.868 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:46 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3957151243' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-20T09:36:46.868 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:46 smithi026 ceph-mon[68309]: osdmap e997: 12 total, 12 up, 12 in 2024-03-20T09:36:46.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:36:46.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:36:46.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:36:46.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:36:46.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:36:46.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:36:46.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:36:46.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:46 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1380462955' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:36:46.869 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:46 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:36:46.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:36:46.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:46 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3957151243' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-20T09:36:46.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:46 smithi115 ceph-mon[72476]: osdmap e997: 12 total, 12 up, 12 in 2024-03-20T09:36:46.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:36:46.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:36:46.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:36:46.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:36:46.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:36:46.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:36:46.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:36:46.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:46 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1380462955' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:36:46.897 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:46 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:36:46.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:36:46.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:46 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3957151243' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-20T09:36:46.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:46 smithi122 ceph-mon[73495]: osdmap e997: 12 total, 12 up, 12 in 2024-03-20T09:36:46.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:36:46.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:36:46.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:36:46.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:36:46.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:36:46.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:36:46.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:36:46.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:46 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1380462955' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:36:46.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:46 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:36:47.211 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:36:47.211 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:36:47.770 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 48 16 82398 82382 107.232 68.8125 0.0117669 0.00932147 2024-03-20T09:36:47.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:47 smithi115 ceph-mon[72476]: pgmap v2402: 47 pgs: 47 active+clean; 4.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.27k op/s 2024-03-20T09:36:47.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:47 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:36:47.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:47 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:36:47.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:47 smithi115 ceph-mon[72476]: osdmap e998: 12 total, 12 up, 12 in 2024-03-20T09:36:47.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:47 smithi115 ceph-mon[72476]: 1.5 scrub starts 2024-03-20T09:36:47.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:47 smithi122 ceph-mon[73495]: pgmap v2402: 47 pgs: 47 active+clean; 4.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.27k op/s 2024-03-20T09:36:47.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:47 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:36:47.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:47 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:36:47.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:47 smithi122 ceph-mon[73495]: osdmap e998: 12 total, 12 up, 12 in 2024-03-20T09:36:47.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:47 smithi122 ceph-mon[73495]: 1.5 scrub starts 2024-03-20T09:36:47.960 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-20T09:36:47.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:47 smithi026 ceph-mon[68309]: pgmap v2402: 47 pgs: 47 active+clean; 4.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.27k op/s 2024-03-20T09:36:47.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:47 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:36:47.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:47 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:36:47.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:47 smithi026 ceph-mon[68309]: osdmap e998: 12 total, 12 up, 12 in 2024-03-20T09:36:47.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:47 smithi026 ceph-mon[68309]: 1.5 scrub starts 2024-03-20T09:36:48.329 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:36:48.387 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:36:48.421 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:36:48.451 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:36:48.482 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:36:48.512 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:36:48.542 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:36:48.579 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:36:48.610 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:36:48.644 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:36:48.660 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:36:48.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:48.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:36:48.681 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:36:48.718 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:36:48.729 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:36:48.761 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:36:48.765 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:36:48.770 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 49 16 83677 83661 106.675 79.9375 0.0123228 0.0093702 2024-03-20T09:36:48.772 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:36:48.772 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:36:48.772 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:36:48.772 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:36:48.772 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:36:48.772 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:36:48.772 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:36:48.772 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:36:48.772 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:36:48.772 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:36:48.772 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:36:48.772 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:36:48.819 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:48.875 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:48 smithi115 ceph-mon[72476]: 1.5 scrub ok 2024-03-20T09:36:48.875 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:48 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4078392938' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:36:48.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:48 smithi122 ceph-mon[73495]: 1.5 scrub ok 2024-03-20T09:36:48.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:48 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4078392938' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:36:48.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:48 smithi026 ceph-mon[68309]: 1.5 scrub ok 2024-03-20T09:36:48.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:48 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4078392938' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:36:49.174 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:36:49.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:49.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:36:49.279 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:36:49.324 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:49.354 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:36:49.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:49.359 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:36:49.461 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:36:49.517 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:49.553 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:36:49.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:49.556 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:36:49.612 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:49 smithi026 ceph-mon[68309]: 1.17 scrub starts 2024-03-20T09:36:49.612 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:49 smithi026 ceph-mon[68309]: 1.17 scrub ok 2024-03-20T09:36:49.612 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:49 smithi026 ceph-mon[68309]: 1.19 scrub starts 2024-03-20T09:36:49.612 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:49 smithi026 ceph-mon[68309]: 1.19 scrub ok 2024-03-20T09:36:49.612 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:49 smithi026 ceph-mon[68309]: pgmap v2404: 47 pgs: 47 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-03-20T09:36:49.613 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:49 smithi026 ceph-mon[68309]: 1.f scrub starts 2024-03-20T09:36:49.613 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:49 smithi026 ceph-mon[68309]: 1.f scrub ok 2024-03-20T09:36:49.657 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:36:49.706 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:49.737 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:36:49.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:49.741 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:36:49.770 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 50 16 84974 84958 106.163 81.0625 0.0065243 0.00941476 2024-03-20T09:36:49.842 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:36:49.871 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:49 smithi115 ceph-mon[72476]: 1.17 scrub starts 2024-03-20T09:36:49.871 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:49 smithi115 ceph-mon[72476]: 1.17 scrub ok 2024-03-20T09:36:49.871 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:49 smithi115 ceph-mon[72476]: 1.19 scrub starts 2024-03-20T09:36:49.871 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:49 smithi115 ceph-mon[72476]: 1.19 scrub ok 2024-03-20T09:36:49.871 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:49 smithi115 ceph-mon[72476]: pgmap v2404: 47 pgs: 47 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-03-20T09:36:49.871 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:49 smithi115 ceph-mon[72476]: 1.f scrub starts 2024-03-20T09:36:49.871 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:49 smithi115 ceph-mon[72476]: 1.f scrub ok 2024-03-20T09:36:49.876 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:49.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:49 smithi122 ceph-mon[73495]: 1.17 scrub starts 2024-03-20T09:36:49.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:49 smithi122 ceph-mon[73495]: 1.17 scrub ok 2024-03-20T09:36:49.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:49 smithi122 ceph-mon[73495]: 1.19 scrub starts 2024-03-20T09:36:49.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:49 smithi122 ceph-mon[73495]: 1.19 scrub ok 2024-03-20T09:36:49.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:49 smithi122 ceph-mon[73495]: pgmap v2404: 47 pgs: 47 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-03-20T09:36:49.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:49 smithi122 ceph-mon[73495]: 1.f scrub starts 2024-03-20T09:36:49.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:49 smithi122 ceph-mon[73495]: 1.f scrub ok 2024-03-20T09:36:50.006 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:36:50.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:50.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:36:50.115 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:36:50.180 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:50.216 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:36:50.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:50.220 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:36:50.321 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:36:50.335 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:36:50.335 INFO:teuthology.orchestra.run.smithi115.stdout: "size": 20, 2024-03-20T09:36:50.335 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 600, 2024-03-20T09:36:50.335 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [ 2024-03-20T09:36:50.335 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.335 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:170377 6.d 6:b77beaad:::benchmark_data_smithi122_168655_object50268:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-03-20T09:36:50.335 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:28:18.585048+0000", 2024-03-20T09:36:50.335 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 511.756957984, 2024-03-20T09:36:50.336 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.202490477, 2024-03-20T09:36:50.336 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:36:50.336 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:36:50.336 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:36:50.336 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:36:50.336 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:36:50.336 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 170377 2024-03-20T09:36:50.336 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.336 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:36:50.336 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.336 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:36:50.336 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:18.585048+0000", 2024-03-20T09:36:50.336 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.336 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.337 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.337 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:36:50.337 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:18.585048+0000", 2024-03-20T09:36:50.337 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.337 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.337 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.337 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:36:50.337 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:18.585050+0000", 2024-03-20T09:36:50.337 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.432e-06 2024-03-20T09:36:50.337 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.337 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.337 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:36:50.337 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:18.585055+0000", 2024-03-20T09:36:50.337 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3839999999999996e-06 2024-03-20T09:36:50.337 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.338 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.338 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:36:50.338 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:18.585067+0000", 2024-03-20T09:36:50.338 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2388999999999999e-05 2024-03-20T09:36:50.338 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.338 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.338 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:36:50.338 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:18.585074+0000", 2024-03-20T09:36:50.338 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.5740000000000001e-06 2024-03-20T09:36:50.338 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.339 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.339 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.339 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:18.585133+0000", 2024-03-20T09:36:50.339 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8971000000000002e-05 2024-03-20T09:36:50.339 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.339 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.339 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:36:50.339 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:18.585143+0000", 2024-03-20T09:36:50.339 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.9909999999999997e-06 2024-03-20T09:36:50.339 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.339 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.339 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.340 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:19.568167+0000", 2024-03-20T09:36:50.340 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.98302418800000002 2024-03-20T09:36:50.340 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.340 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.340 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:36:50.340 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:19.568176+0000", 2024-03-20T09:36:50.340 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.2560000000000008e-06 2024-03-20T09:36:50.340 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.341 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.341 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.341 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:20.774026+0000", 2024-03-20T09:36:50.341 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2058492350000001 2024-03-20T09:36:50.341 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.341 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.341 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:36:50.342 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:20.774117+0000", 2024-03-20T09:36:50.342 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.1719999999999996e-05 2024-03-20T09:36:50.342 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.342 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.342 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 0,1", 2024-03-20T09:36:50.342 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:20.774211+0000", 2024-03-20T09:36:50.342 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.3734999999999996e-05 2024-03-20T09:36:50.342 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.342 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.342 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:36:50.342 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:20.781151+0000", 2024-03-20T09:36:50.342 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.006939771 2024-03-20T09:36:50.342 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.343 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.343 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:36:50.343 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:20.781191+0000", 2024-03-20T09:36:50.343 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.9780000000000002e-05 2024-03-20T09:36:50.343 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.343 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.343 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:36:50.343 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:20.787422+0000", 2024-03-20T09:36:50.343 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0062316189999999999 2024-03-20T09:36:50.343 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.343 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.343 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:36:50.343 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:20.787469+0000", 2024-03-20T09:36:50.343 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6295000000000003e-05 2024-03-20T09:36:50.344 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.344 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.344 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:36:50.344 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:20.787539+0000", 2024-03-20T09:36:50.344 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.0136999999999996e-05 2024-03-20T09:36:50.344 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.344 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:36:50.344 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.344 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.344 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.344 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:170381 6.19 6.b3770839 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-03-20T09:36:50.344 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:28:18.598640+0000", 2024-03-20T09:36:50.344 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 511.74336575500001, 2024-03-20T09:36:50.344 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.96918979900000002, 2024-03-20T09:36:50.345 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:36:50.345 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:36:50.345 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:36:50.345 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:36:50.345 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:36:50.345 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 170381 2024-03-20T09:36:50.345 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.345 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:36:50.345 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.345 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:36:50.345 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:18.598640+0000", 2024-03-20T09:36:50.345 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.345 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.345 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.346 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:36:50.346 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:18.598640+0000", 2024-03-20T09:36:50.346 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.346 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.346 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.346 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:36:50.346 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:18.598641+0000", 2024-03-20T09:36:50.346 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.6899999999999996e-07 2024-03-20T09:36:50.346 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.346 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.346 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:36:50.346 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:18.598646+0000", 2024-03-20T09:36:50.346 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0880000000000002e-06 2024-03-20T09:36:50.347 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.347 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.347 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:36:50.347 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:18.598655+0000", 2024-03-20T09:36:50.347 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.9099999999999994e-06 2024-03-20T09:36:50.347 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.347 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.347 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:36:50.347 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:18.598660+0000", 2024-03-20T09:36:50.347 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2739999999999998e-06 2024-03-20T09:36:50.347 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.347 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.347 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.347 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:18.598711+0000", 2024-03-20T09:36:50.347 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0426000000000001e-05 2024-03-20T09:36:50.348 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.348 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.348 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:36:50.348 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:18.598721+0000", 2024-03-20T09:36:50.348 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.9869999999999994e-06 2024-03-20T09:36:50.348 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.348 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.348 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.348 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:19.567819+0000", 2024-03-20T09:36:50.348 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.96909767800000002 2024-03-20T09:36:50.348 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.348 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.348 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:36:50.348 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:19.567830+0000", 2024-03-20T09:36:50.349 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1667e-05 2024-03-20T09:36:50.349 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.349 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:36:50.349 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.349 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.349 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.349 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:170388 6.d 6:bd0ca3a5:::benchmark_data_smithi122_168655_object50279:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e811)", 2024-03-20T09:36:50.349 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:28:18.629824+0000", 2024-03-20T09:36:50.349 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 511.71218241399998, 2024-03-20T09:36:50.349 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1579281269999999, 2024-03-20T09:36:50.349 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:36:50.349 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:36:50.349 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:36:50.349 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:36:50.349 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:36:50.350 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 170388 2024-03-20T09:36:50.350 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.350 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:36:50.350 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.350 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:36:50.350 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:18.629824+0000", 2024-03-20T09:36:50.350 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.350 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.350 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.350 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:36:50.350 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:18.629824+0000", 2024-03-20T09:36:50.350 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.350 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.351 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.351 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:36:50.351 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:18.629826+0000", 2024-03-20T09:36:50.351 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.4260000000000002e-06 2024-03-20T09:36:50.351 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.351 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.351 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:36:50.351 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:18.629838+0000", 2024-03-20T09:36:50.351 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1953000000000001e-05 2024-03-20T09:36:50.351 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.351 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.351 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:36:50.351 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:18.629868+0000", 2024-03-20T09:36:50.351 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9980000000000001e-05 2024-03-20T09:36:50.352 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.352 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.352 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:36:50.352 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:18.629885+0000", 2024-03-20T09:36:50.352 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7070000000000001e-05 2024-03-20T09:36:50.352 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.352 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.352 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.352 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:18.629970+0000", 2024-03-20T09:36:50.352 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.4376999999999995e-05 2024-03-20T09:36:50.352 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.352 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.352 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:36:50.352 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:18.629981+0000", 2024-03-20T09:36:50.352 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1821000000000001e-05 2024-03-20T09:36:50.353 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.353 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.353 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.353 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:19.568206+0000", 2024-03-20T09:36:50.353 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.93822490400000003 2024-03-20T09:36:50.353 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.353 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.353 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:36:50.353 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:19.568213+0000", 2024-03-20T09:36:50.353 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4590000000000003e-06 2024-03-20T09:36:50.353 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.353 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.353 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.353 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:20.778052+0000", 2024-03-20T09:36:50.354 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2098395 2024-03-20T09:36:50.354 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.354 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.354 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:36:50.354 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:20.778133+0000", 2024-03-20T09:36:50.354 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.1269999999999999e-05 2024-03-20T09:36:50.354 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.354 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.354 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 0,1", 2024-03-20T09:36:50.354 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:20.778223+0000", 2024-03-20T09:36:50.354 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.9049999999999996e-05 2024-03-20T09:36:50.354 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.354 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.354 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:36:50.354 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:20.781324+0000", 2024-03-20T09:36:50.355 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.003101468 2024-03-20T09:36:50.355 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.355 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.355 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:36:50.355 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:20.782759+0000", 2024-03-20T09:36:50.355 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0014353429999999999 2024-03-20T09:36:50.355 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.355 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.355 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:36:50.355 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:20.787582+0000", 2024-03-20T09:36:50.355 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00482242 2024-03-20T09:36:50.355 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.355 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.355 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:36:50.356 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:20.787643+0000", 2024-03-20T09:36:50.356 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1224999999999997e-05 2024-03-20T09:36:50.356 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.356 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.356 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:36:50.356 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:20.787752+0000", 2024-03-20T09:36:50.356 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000108861 2024-03-20T09:36:50.356 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.356 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:36:50.356 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.356 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.356 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.356 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:172118 6.d 6:bd233136:::benchmark_data_smithi122_168655_object52009:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e820)", 2024-03-20T09:36:50.356 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:28:31.653411+0000", 2024-03-20T09:36:50.357 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 498.688595561, 2024-03-20T09:36:50.357 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.92630386200000003, 2024-03-20T09:36:50.357 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:36:50.357 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:36:50.357 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:36:50.357 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:36:50.357 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:36:50.357 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 172118 2024-03-20T09:36:50.357 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.357 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:36:50.357 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.357 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:36:50.357 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.653411+0000", 2024-03-20T09:36:50.357 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.357 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.358 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.358 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:36:50.358 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.653411+0000", 2024-03-20T09:36:50.358 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.358 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.358 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.358 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:36:50.358 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.653411+0000", 2024-03-20T09:36:50.358 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9399999999999995e-07 2024-03-20T09:36:50.358 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.358 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.358 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:36:50.358 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.653414+0000", 2024-03-20T09:36:50.358 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.819e-06 2024-03-20T09:36:50.359 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.359 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.359 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:36:50.359 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.653421+0000", 2024-03-20T09:36:50.359 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.686e-06 2024-03-20T09:36:50.359 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.359 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.359 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:36:50.359 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.653425+0000", 2024-03-20T09:36:50.359 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.2100000000000003e-06 2024-03-20T09:36:50.359 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.359 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.359 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.359 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.653470+0000", 2024-03-20T09:36:50.360 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4838e-05 2024-03-20T09:36:50.360 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.360 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.360 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:36:50.360 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.653479+0000", 2024-03-20T09:36:50.360 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.5380000000000008e-06 2024-03-20T09:36:50.360 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.360 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.360 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.360 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.560440+0000", 2024-03-20T09:36:50.360 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.90696046799999996 2024-03-20T09:36:50.360 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.360 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.360 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:36:50.361 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.560540+0000", 2024-03-20T09:36:50.361 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000100738 2024-03-20T09:36:50.361 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.361 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.361 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 0,1", 2024-03-20T09:36:50.361 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.560631+0000", 2024-03-20T09:36:50.361 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.0290999999999993e-05 2024-03-20T09:36:50.361 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.361 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.361 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:36:50.361 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.572063+0000", 2024-03-20T09:36:50.361 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.011432526 2024-03-20T09:36:50.361 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.361 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.361 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:36:50.362 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.579584+0000", 2024-03-20T09:36:50.362 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0075208590000000004 2024-03-20T09:36:50.362 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.362 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.362 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:36:50.362 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.579631+0000", 2024-03-20T09:36:50.362 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.7187000000000001e-05 2024-03-20T09:36:50.362 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.362 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.362 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:36:50.362 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.579669+0000", 2024-03-20T09:36:50.362 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.8040000000000002e-05 2024-03-20T09:36:50.362 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.362 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.363 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:36:50.363 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.579714+0000", 2024-03-20T09:36:50.363 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5167999999999999e-05 2024-03-20T09:36:50.363 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.363 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:36:50.363 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.363 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.363 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.363 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:172122 6.d 6:bdc1585d:::benchmark_data_smithi122_168655_object52013:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e820)", 2024-03-20T09:36:50.363 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:28:31.670996+0000", 2024-03-20T09:36:50.363 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 498.67101051399999, 2024-03-20T09:36:50.363 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.90886192399999999, 2024-03-20T09:36:50.363 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:36:50.364 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:36:50.364 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:36:50.364 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:36:50.364 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:36:50.364 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 172122 2024-03-20T09:36:50.364 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.364 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:36:50.364 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.364 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:36:50.364 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.670996+0000", 2024-03-20T09:36:50.364 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.364 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.364 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.364 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:36:50.364 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.670996+0000", 2024-03-20T09:36:50.365 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.365 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.365 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.365 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:36:50.365 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.670998+0000", 2024-03-20T09:36:50.365 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.007e-06 2024-03-20T09:36:50.365 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.365 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.365 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:36:50.365 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.671009+0000", 2024-03-20T09:36:50.365 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1093e-05 2024-03-20T09:36:50.365 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.365 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.365 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:36:50.366 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.671032+0000", 2024-03-20T09:36:50.366 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3183000000000001e-05 2024-03-20T09:36:50.366 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.366 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.366 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:36:50.366 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.671053+0000", 2024-03-20T09:36:50.366 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1529e-05 2024-03-20T09:36:50.366 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.366 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.366 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.366 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.671162+0000", 2024-03-20T09:36:50.366 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000108677 2024-03-20T09:36:50.366 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.367 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.367 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:36:50.367 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.671171+0000", 2024-03-20T09:36:50.367 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.3090000000000001e-06 2024-03-20T09:36:50.367 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.368 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.368 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.368 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.564711+0000", 2024-03-20T09:36:50.368 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.89353968699999997 2024-03-20T09:36:50.368 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.368 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.368 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:36:50.368 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.564829+0000", 2024-03-20T09:36:50.368 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000117593 2024-03-20T09:36:50.368 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.368 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.368 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 0,1", 2024-03-20T09:36:50.369 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.564920+0000", 2024-03-20T09:36:50.369 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.0921000000000006e-05 2024-03-20T09:36:50.369 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.369 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.369 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:36:50.369 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.579741+0000", 2024-03-20T09:36:50.369 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.014821196 2024-03-20T09:36:50.369 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.369 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.369 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:36:50.369 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.579775+0000", 2024-03-20T09:36:50.369 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.3679000000000001e-05 2024-03-20T09:36:50.369 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.369 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.370 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:36:50.370 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.579808+0000", 2024-03-20T09:36:50.370 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.3019999999999999e-05 2024-03-20T09:36:50.370 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.370 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.370 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:36:50.370 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.579837+0000", 2024-03-20T09:36:50.370 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9291000000000001e-05 2024-03-20T09:36:50.370 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.370 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.370 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:36:50.370 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.579858+0000", 2024-03-20T09:36:50.370 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0738999999999999e-05 2024-03-20T09:36:50.370 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.371 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:36:50.371 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.371 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.371 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.371 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:172137 6.d 6:b5fe4317:::benchmark_data_smithi122_168655_object52028:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e820)", 2024-03-20T09:36:50.371 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:28:31.743217+0000", 2024-03-20T09:36:50.371 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 498.59878878199999, 2024-03-20T09:36:50.371 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.85305413399999996, 2024-03-20T09:36:50.371 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:36:50.371 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:36:50.371 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:36:50.371 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:36:50.371 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:36:50.371 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 172137 2024-03-20T09:36:50.372 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.372 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:36:50.372 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.372 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:36:50.372 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.743217+0000", 2024-03-20T09:36:50.372 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.372 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.372 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.372 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:36:50.372 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.743217+0000", 2024-03-20T09:36:50.372 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.372 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.372 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.372 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:36:50.373 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.743218+0000", 2024-03-20T09:36:50.373 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.128e-06 2024-03-20T09:36:50.373 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.373 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.373 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:36:50.373 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.743223+0000", 2024-03-20T09:36:50.373 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9420000000000003e-06 2024-03-20T09:36:50.373 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.373 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.373 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:36:50.373 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.743236+0000", 2024-03-20T09:36:50.373 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2483000000000001e-05 2024-03-20T09:36:50.373 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.373 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.374 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:36:50.374 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.743244+0000", 2024-03-20T09:36:50.374 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.8620000000000003e-06 2024-03-20T09:36:50.374 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.374 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.374 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.374 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.743292+0000", 2024-03-20T09:36:50.374 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8133999999999997e-05 2024-03-20T09:36:50.374 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.374 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.374 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:36:50.374 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.743298+0000", 2024-03-20T09:36:50.374 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5829999999999997e-06 2024-03-20T09:36:50.374 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.375 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.375 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.375 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.568292+0000", 2024-03-20T09:36:50.375 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.82499437900000006 2024-03-20T09:36:50.375 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.375 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.375 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:36:50.375 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.568377+0000", 2024-03-20T09:36:50.375 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.4839999999999994e-05 2024-03-20T09:36:50.375 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.375 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.375 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 0,1", 2024-03-20T09:36:50.375 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.568461+0000", 2024-03-20T09:36:50.376 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.4568999999999996e-05 2024-03-20T09:36:50.376 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.376 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.376 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:36:50.376 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.579865+0000", 2024-03-20T09:36:50.376 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.011403234999999999 2024-03-20T09:36:50.376 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.376 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.376 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:36:50.376 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.579902+0000", 2024-03-20T09:36:50.376 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7805000000000001e-05 2024-03-20T09:36:50.376 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.376 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.376 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:36:50.376 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.596165+0000", 2024-03-20T09:36:50.377 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.016263158 2024-03-20T09:36:50.377 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.377 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.377 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:36:50.377 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.596222+0000", 2024-03-20T09:36:50.377 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6953999999999997e-05 2024-03-20T09:36:50.377 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.377 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.377 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:36:50.377 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.596272+0000", 2024-03-20T09:36:50.377 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9061999999999999e-05 2024-03-20T09:36:50.377 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.377 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:36:50.377 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.377 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.378 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.378 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:172147 6.d 6:b739c53d:::benchmark_data_smithi122_168655_object52038:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e820)", 2024-03-20T09:36:50.378 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:28:31.836308+0000", 2024-03-20T09:36:50.378 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 498.50569826200001, 2024-03-20T09:36:50.378 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.76364185299999998, 2024-03-20T09:36:50.378 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:36:50.378 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:36:50.378 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:36:50.378 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:36:50.378 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:36:50.378 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 172147 2024-03-20T09:36:50.378 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.378 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:36:50.379 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.379 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:36:50.379 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.836308+0000", 2024-03-20T09:36:50.379 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.379 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.379 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.379 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:36:50.379 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.836308+0000", 2024-03-20T09:36:50.379 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.379 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.379 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.379 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:36:50.379 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.836309+0000", 2024-03-20T09:36:50.379 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.102e-06 2024-03-20T09:36:50.380 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.380 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.380 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:36:50.380 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.836316+0000", 2024-03-20T09:36:50.380 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.5250000000000002e-06 2024-03-20T09:36:50.380 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.380 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.380 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:36:50.380 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.836327+0000", 2024-03-20T09:36:50.380 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1922999999999999e-05 2024-03-20T09:36:50.380 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.380 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.380 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:36:50.380 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.836333+0000", 2024-03-20T09:36:50.380 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5430000000000001e-06 2024-03-20T09:36:50.381 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.381 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.381 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.381 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.836384+0000", 2024-03-20T09:36:50.381 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1295000000000003e-05 2024-03-20T09:36:50.381 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.381 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.381 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:36:50.381 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.836394+0000", 2024-03-20T09:36:50.381 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.7529999999999996e-06 2024-03-20T09:36:50.381 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.381 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.381 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.381 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.572095+0000", 2024-03-20T09:36:50.381 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.73570098799999994 2024-03-20T09:36:50.382 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.382 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.382 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:36:50.382 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.572173+0000", 2024-03-20T09:36:50.382 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7714999999999997e-05 2024-03-20T09:36:50.382 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.382 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.382 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 0,1", 2024-03-20T09:36:50.382 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.572295+0000", 2024-03-20T09:36:50.382 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012248599999999999 2024-03-20T09:36:50.382 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.382 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.382 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:36:50.382 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.579868+0000", 2024-03-20T09:36:50.383 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0075730509999999999 2024-03-20T09:36:50.383 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.383 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.383 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:36:50.383 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.596323+0000", 2024-03-20T09:36:50.383 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.01645487 2024-03-20T09:36:50.383 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.383 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.383 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:36:50.383 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.599881+0000", 2024-03-20T09:36:50.383 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0035574780000000002 2024-03-20T09:36:50.383 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.383 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.383 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:36:50.383 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.599914+0000", 2024-03-20T09:36:50.384 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.3711000000000003e-05 2024-03-20T09:36:50.384 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.384 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.384 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:36:50.384 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.599950+0000", 2024-03-20T09:36:50.384 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.5413000000000001e-05 2024-03-20T09:36:50.384 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.384 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:36:50.384 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.384 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.384 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.384 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:172150 6.d 6:b8d67e85:::benchmark_data_smithi122_168655_object52041:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e820)", 2024-03-20T09:36:50.384 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:28:31.846107+0000", 2024-03-20T09:36:50.384 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 498.49589873500003, 2024-03-20T09:36:50.385 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.75398030900000002, 2024-03-20T09:36:50.385 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:36:50.385 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:36:50.385 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:36:50.385 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:36:50.385 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:36:50.385 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 172150 2024-03-20T09:36:50.385 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.385 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:36:50.385 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.385 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:36:50.385 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.846107+0000", 2024-03-20T09:36:50.385 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.385 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.386 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.386 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:36:50.386 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.846107+0000", 2024-03-20T09:36:50.386 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.386 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.386 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.386 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:36:50.386 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.846109+0000", 2024-03-20T09:36:50.386 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.314e-06 2024-03-20T09:36:50.386 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.386 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.386 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:36:50.386 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.846117+0000", 2024-03-20T09:36:50.386 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.994e-06 2024-03-20T09:36:50.387 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.387 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.387 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:36:50.387 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.846129+0000", 2024-03-20T09:36:50.387 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2578999999999999e-05 2024-03-20T09:36:50.387 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.387 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.387 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:36:50.387 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.846150+0000", 2024-03-20T09:36:50.387 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0795999999999999e-05 2024-03-20T09:36:50.387 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.387 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.387 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.387 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.846204+0000", 2024-03-20T09:36:50.387 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3962999999999998e-05 2024-03-20T09:36:50.388 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.388 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.388 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:36:50.388 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.846212+0000", 2024-03-20T09:36:50.388 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.4619999999999999e-06 2024-03-20T09:36:50.388 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.388 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.388 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.388 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.576030+0000", 2024-03-20T09:36:50.388 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.72981820399999997 2024-03-20T09:36:50.388 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.388 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.388 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:36:50.388 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.576135+0000", 2024-03-20T09:36:50.388 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000104907 2024-03-20T09:36:50.389 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.389 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.389 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 0,1", 2024-03-20T09:36:50.389 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.576221+0000", 2024-03-20T09:36:50.389 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.6723000000000003e-05 2024-03-20T09:36:50.389 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.389 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.389 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:36:50.389 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.579872+0000", 2024-03-20T09:36:50.389 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0036503439999999998 2024-03-20T09:36:50.389 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.389 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.389 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:36:50.389 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.596366+0000", 2024-03-20T09:36:50.389 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.016493872 2024-03-20T09:36:50.390 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.390 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.390 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:36:50.390 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.599993+0000", 2024-03-20T09:36:50.390 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0036273170000000001 2024-03-20T09:36:50.390 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.390 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.390 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:36:50.390 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.600047+0000", 2024-03-20T09:36:50.390 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3643999999999997e-05 2024-03-20T09:36:50.390 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.390 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.390 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:36:50.390 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.600088+0000", 2024-03-20T09:36:50.391 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.1189999999999997e-05 2024-03-20T09:36:50.391 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.391 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:36:50.391 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.391 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.391 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.391 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:172193 6.d 6:b46d8338:::benchmark_data_smithi122_168655_object52084:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e820)", 2024-03-20T09:36:50.391 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:28:31.999950+0000", 2024-03-20T09:36:50.391 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 498.342055802, 2024-03-20T09:36:50.391 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.60022316799999997, 2024-03-20T09:36:50.391 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:36:50.391 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:36:50.391 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:36:50.391 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:36:50.392 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:36:50.392 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 172193 2024-03-20T09:36:50.392 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.392 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:36:50.392 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.392 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:36:50.392 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.999950+0000", 2024-03-20T09:36:50.392 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.392 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.392 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.392 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:36:50.392 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.999950+0000", 2024-03-20T09:36:50.392 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.392 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.392 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.393 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:36:50.393 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.999951+0000", 2024-03-20T09:36:50.393 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5700000000000001e-07 2024-03-20T09:36:50.393 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.393 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.393 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:36:50.393 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.999955+0000", 2024-03-20T09:36:50.393 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.788e-06 2024-03-20T09:36:50.393 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.393 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.393 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:36:50.393 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.999965+0000", 2024-03-20T09:36:50.393 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.5750000000000007e-06 2024-03-20T09:36:50.393 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.394 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.394 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:36:50.394 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:31.999970+0000", 2024-03-20T09:36:50.394 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1449999999999999e-06 2024-03-20T09:36:50.394 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.394 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.394 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.394 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.000023+0000", 2024-03-20T09:36:50.394 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3517999999999998e-05 2024-03-20T09:36:50.394 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.394 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.394 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:36:50.394 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.000035+0000", 2024-03-20T09:36:50.394 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1610000000000001e-05 2024-03-20T09:36:50.395 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.395 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.395 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.395 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.579927+0000", 2024-03-20T09:36:50.395 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.57989245700000003 2024-03-20T09:36:50.395 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.395 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.395 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:36:50.395 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.580000+0000", 2024-03-20T09:36:50.395 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3064000000000004e-05 2024-03-20T09:36:50.395 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.395 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.395 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 0,1", 2024-03-20T09:36:50.395 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.580074+0000", 2024-03-20T09:36:50.395 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3589000000000003e-05 2024-03-20T09:36:50.396 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.396 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.396 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:36:50.396 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.592700+0000", 2024-03-20T09:36:50.396 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.012625958 2024-03-20T09:36:50.396 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.396 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.396 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:36:50.396 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.596406+0000", 2024-03-20T09:36:50.396 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0037057320000000002 2024-03-20T09:36:50.396 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.396 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.396 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:36:50.396 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.600123+0000", 2024-03-20T09:36:50.397 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0037176739999999998 2024-03-20T09:36:50.397 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.397 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.397 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:36:50.397 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.600150+0000", 2024-03-20T09:36:50.397 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6727999999999999e-05 2024-03-20T09:36:50.397 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.397 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.397 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:36:50.397 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.600174+0000", 2024-03-20T09:36:50.397 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3473e-05 2024-03-20T09:36:50.397 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.397 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:36:50.397 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.397 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.398 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.398 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:172202 6.d 6:b604b677:::benchmark_data_smithi122_168655_object52093:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e820)", 2024-03-20T09:36:50.398 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:28:32.030271+0000", 2024-03-20T09:36:50.398 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 498.31173558699999, 2024-03-20T09:36:50.398 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.58191322599999995, 2024-03-20T09:36:50.398 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:36:50.398 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:36:50.398 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:36:50.398 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:36:50.398 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:36:50.398 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 172202 2024-03-20T09:36:50.398 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.398 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:36:50.398 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.399 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:36:50.399 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.030271+0000", 2024-03-20T09:36:50.399 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.399 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.399 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.399 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:36:50.399 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.030271+0000", 2024-03-20T09:36:50.399 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.399 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.399 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.399 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:36:50.399 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.030272+0000", 2024-03-20T09:36:50.399 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.7199999999999997e-07 2024-03-20T09:36:50.399 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.400 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.400 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:36:50.400 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.030277+0000", 2024-03-20T09:36:50.400 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.198e-06 2024-03-20T09:36:50.400 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.400 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.400 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:36:50.400 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.030289+0000", 2024-03-20T09:36:50.400 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.198e-05 2024-03-20T09:36:50.400 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.400 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.400 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:36:50.400 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.030294+0000", 2024-03-20T09:36:50.400 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7409999999999998e-06 2024-03-20T09:36:50.400 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.401 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.401 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.401 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.030354+0000", 2024-03-20T09:36:50.401 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9212999999999997e-05 2024-03-20T09:36:50.401 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.401 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.401 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:36:50.401 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.030364+0000", 2024-03-20T09:36:50.401 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0587999999999999e-05 2024-03-20T09:36:50.401 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.401 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.401 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.401 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.583206+0000", 2024-03-20T09:36:50.401 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.55284200999999999 2024-03-20T09:36:50.401 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.402 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.402 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:36:50.402 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.583270+0000", 2024-03-20T09:36:50.402 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4109000000000006e-05 2024-03-20T09:36:50.402 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.402 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.402 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 0,1", 2024-03-20T09:36:50.402 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.583336+0000", 2024-03-20T09:36:50.402 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.5628000000000003e-05 2024-03-20T09:36:50.402 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.402 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.402 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:36:50.402 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.596332+0000", 2024-03-20T09:36:50.402 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.012995626999999999 2024-03-20T09:36:50.403 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.403 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.403 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:36:50.403 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.596449+0000", 2024-03-20T09:36:50.403 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000117287 2024-03-20T09:36:50.403 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.403 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.403 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:36:50.403 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.612069+0000", 2024-03-20T09:36:50.403 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.015619773 2024-03-20T09:36:50.403 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.403 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.403 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:36:50.403 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.612128+0000", 2024-03-20T09:36:50.403 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9335e-05 2024-03-20T09:36:50.404 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.404 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.404 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:36:50.404 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:32.612184+0000", 2024-03-20T09:36:50.404 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5764999999999999e-05 2024-03-20T09:36:50.404 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.404 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:36:50.404 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.404 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.404 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.404 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:181254 6.19 6:9bc787a5:::benchmark_data_smithi122_168655_object61145:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e829)", 2024-03-20T09:36:50.404 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:28:59.528423+0000", 2024-03-20T09:36:50.404 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 470.81358306099997, 2024-03-20T09:36:50.405 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0209822319999999, 2024-03-20T09:36:50.405 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:36:50.405 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:36:50.405 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:36:50.405 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:36:50.405 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:36:50.405 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 181254 2024-03-20T09:36:50.405 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.405 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:36:50.405 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.405 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:36:50.405 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.528423+0000", 2024-03-20T09:36:50.405 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.405 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.406 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.406 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:36:50.406 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.528423+0000", 2024-03-20T09:36:50.406 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.406 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.406 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.406 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:36:50.406 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.528424+0000", 2024-03-20T09:36:50.406 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.218e-06 2024-03-20T09:36:50.406 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.406 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.406 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:36:50.406 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.528432+0000", 2024-03-20T09:36:50.406 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7800000000000001e-06 2024-03-20T09:36:50.407 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.407 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.407 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:36:50.407 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.528448+0000", 2024-03-20T09:36:50.407 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5631e-05 2024-03-20T09:36:50.407 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.407 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.407 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:36:50.407 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.528455+0000", 2024-03-20T09:36:50.407 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.4030000000000001e-06 2024-03-20T09:36:50.407 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.407 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.407 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.407 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.528519+0000", 2024-03-20T09:36:50.407 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4294999999999993e-05 2024-03-20T09:36:50.408 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.408 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.408 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op must wait for map", 2024-03-20T09:36:50.408 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.528527+0000", 2024-03-20T09:36:50.408 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.4830000000000002e-06 2024-03-20T09:36:50.408 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.408 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.408 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.408 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.531668+0000", 2024-03-20T09:36:50.408 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0031411550000000001 2024-03-20T09:36:50.408 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.408 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.408 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:36:50.408 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.531676+0000", 2024-03-20T09:36:50.408 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.712e-06 2024-03-20T09:36:50.409 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.409 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.409 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.409 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.549334+0000", 2024-03-20T09:36:50.409 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.017658647 2024-03-20T09:36:50.409 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.409 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.409 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:36:50.409 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.549405+0000", 2024-03-20T09:36:50.409 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.0908000000000006e-05 2024-03-20T09:36:50.409 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.409 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:36:50.409 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.409 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.410 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.410 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:181292 6.19 6:9c0233c7:::benchmark_data_smithi122_168655_object61183:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e829)", 2024-03-20T09:36:50.410 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:28:59.590688+0000", 2024-03-20T09:36:50.410 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 470.75131831499999, 2024-03-20T09:36:50.410 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.96936758700000003, 2024-03-20T09:36:50.410 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:36:50.410 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:36:50.410 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:36:50.410 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:36:50.410 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:36:50.410 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 181292 2024-03-20T09:36:50.410 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.410 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:36:50.410 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.410 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:36:50.411 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.590688+0000", 2024-03-20T09:36:50.411 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.411 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.411 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.411 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:36:50.411 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.590688+0000", 2024-03-20T09:36:50.411 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.411 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.411 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.411 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:36:50.411 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.590689+0000", 2024-03-20T09:36:50.411 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.254e-06 2024-03-20T09:36:50.411 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.411 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.412 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:36:50.412 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.590695+0000", 2024-03-20T09:36:50.412 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9610000000000002e-06 2024-03-20T09:36:50.412 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.412 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.412 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:36:50.412 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.590709+0000", 2024-03-20T09:36:50.412 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3553999999999999e-05 2024-03-20T09:36:50.412 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.412 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.412 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:36:50.412 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.590715+0000", 2024-03-20T09:36:50.412 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.8279999999999997e-06 2024-03-20T09:36:50.412 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.412 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.413 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.413 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.590761+0000", 2024-03-20T09:36:50.413 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6050000000000001e-05 2024-03-20T09:36:50.413 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.413 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.413 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:36:50.413 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.590769+0000", 2024-03-20T09:36:50.413 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.1849999999999998e-06 2024-03-20T09:36:50.413 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.413 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.413 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.413 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.549727+0000", 2024-03-20T09:36:50.413 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.95895840799999998 2024-03-20T09:36:50.413 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.413 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.414 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:36:50.414 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.549801+0000", 2024-03-20T09:36:50.414 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3460999999999993e-05 2024-03-20T09:36:50.414 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.414 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.414 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,2", 2024-03-20T09:36:50.414 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.549888+0000", 2024-03-20T09:36:50.414 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.7310000000000003e-05 2024-03-20T09:36:50.414 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.414 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.414 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:36:50.414 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.555251+0000", 2024-03-20T09:36:50.414 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0053627960000000004 2024-03-20T09:36:50.414 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.415 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.415 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:36:50.415 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.559905+0000", 2024-03-20T09:36:50.415 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0046541630000000002 2024-03-20T09:36:50.415 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.415 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.415 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:36:50.415 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.559958+0000", 2024-03-20T09:36:50.415 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3063999999999999e-05 2024-03-20T09:36:50.415 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.415 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.415 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:36:50.415 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.560011+0000", 2024-03-20T09:36:50.415 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3062999999999997e-05 2024-03-20T09:36:50.416 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.416 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.416 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:36:50.416 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.560055+0000", 2024-03-20T09:36:50.416 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4490000000000003e-05 2024-03-20T09:36:50.416 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.416 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:36:50.416 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.416 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.416 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.416 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:181337 6.19 6:9fac6875:::benchmark_data_smithi122_168655_object61228:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e829)", 2024-03-20T09:36:50.416 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:28:59.654079+0000", 2024-03-20T09:36:50.416 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 470.68792693699999, 2024-03-20T09:36:50.416 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.912636636, 2024-03-20T09:36:50.416 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:36:50.417 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:36:50.417 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:36:50.417 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:36:50.417 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:36:50.417 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 181337 2024-03-20T09:36:50.417 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.417 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:36:50.417 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.417 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:36:50.417 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.654079+0000", 2024-03-20T09:36:50.417 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.417 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.417 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.418 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:36:50.418 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.654079+0000", 2024-03-20T09:36:50.418 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.418 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.418 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.418 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:36:50.418 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.654080+0000", 2024-03-20T09:36:50.418 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.158e-06 2024-03-20T09:36:50.418 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.418 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.418 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:36:50.418 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.654086+0000", 2024-03-20T09:36:50.418 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6350000000000003e-06 2024-03-20T09:36:50.418 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.419 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.419 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:36:50.419 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.654096+0000", 2024-03-20T09:36:50.419 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.6940000000000007e-06 2024-03-20T09:36:50.419 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.419 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.419 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:36:50.419 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.654102+0000", 2024-03-20T09:36:50.419 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8930000000000002e-06 2024-03-20T09:36:50.419 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.419 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.419 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.419 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.654148+0000", 2024-03-20T09:36:50.419 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6711000000000001e-05 2024-03-20T09:36:50.420 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.420 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.420 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:36:50.420 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.654155+0000", 2024-03-20T09:36:50.420 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.6499999999999999e-06 2024-03-20T09:36:50.420 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.420 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.420 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.420 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.551575+0000", 2024-03-20T09:36:50.420 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.89742018400000001 2024-03-20T09:36:50.420 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.420 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.420 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:36:50.420 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.551636+0000", 2024-03-20T09:36:50.421 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0914e-05 2024-03-20T09:36:50.421 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.421 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.421 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,2", 2024-03-20T09:36:50.421 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.551701+0000", 2024-03-20T09:36:50.421 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.5252000000000005e-05 2024-03-20T09:36:50.421 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.421 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.421 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:36:50.421 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.555255+0000", 2024-03-20T09:36:50.421 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0035532509999999999 2024-03-20T09:36:50.421 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.421 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.421 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:36:50.422 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.566603+0000", 2024-03-20T09:36:50.422 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.011348722 2024-03-20T09:36:50.422 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.422 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.422 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:36:50.422 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.566643+0000", 2024-03-20T09:36:50.422 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.9291e-05 2024-03-20T09:36:50.422 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.422 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.422 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:36:50.422 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.566674+0000", 2024-03-20T09:36:50.422 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.1591999999999998e-05 2024-03-20T09:36:50.422 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.422 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.423 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:36:50.423 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.566716+0000", 2024-03-20T09:36:50.423 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.1689e-05 2024-03-20T09:36:50.423 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.423 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:36:50.423 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.423 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.423 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.423 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:181364 6.19 6:9a0663c5:::benchmark_data_smithi122_168655_object61255:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e829)", 2024-03-20T09:36:50.423 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:28:59.694152+0000", 2024-03-20T09:36:50.423 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 470.64785395000001, 2024-03-20T09:36:50.423 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.88174901800000005, 2024-03-20T09:36:50.423 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:36:50.423 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:36:50.423 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:36:50.424 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:36:50.424 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:36:50.424 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 181364 2024-03-20T09:36:50.424 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.424 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:36:50.424 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.424 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:36:50.424 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.694152+0000", 2024-03-20T09:36:50.424 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.424 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.424 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.424 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:36:50.424 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.694152+0000", 2024-03-20T09:36:50.424 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.425 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.425 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.425 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:36:50.425 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.694153+0000", 2024-03-20T09:36:50.425 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1060000000000001e-06 2024-03-20T09:36:50.425 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.425 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.425 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:36:50.425 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.694159+0000", 2024-03-20T09:36:50.425 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2170000000000002e-06 2024-03-20T09:36:50.425 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.425 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.425 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:36:50.425 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.694171+0000", 2024-03-20T09:36:50.425 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.216e-05 2024-03-20T09:36:50.426 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.426 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.426 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:36:50.426 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.694176+0000", 2024-03-20T09:36:50.426 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6359999999999999e-06 2024-03-20T09:36:50.426 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.426 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.426 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.426 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.694231+0000", 2024-03-20T09:36:50.426 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5059999999999998e-05 2024-03-20T09:36:50.426 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.426 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.426 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:36:50.426 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.694238+0000", 2024-03-20T09:36:50.427 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.0119999999999999e-06 2024-03-20T09:36:50.427 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.427 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.427 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.427 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.552698+0000", 2024-03-20T09:36:50.427 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.85845930400000003 2024-03-20T09:36:50.427 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.427 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.427 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:36:50.427 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.552765+0000", 2024-03-20T09:36:50.427 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.7522999999999997e-05 2024-03-20T09:36:50.427 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.427 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.427 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,2", 2024-03-20T09:36:50.427 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.552836+0000", 2024-03-20T09:36:50.428 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.1223000000000006e-05 2024-03-20T09:36:50.428 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.428 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.428 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:36:50.428 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.559854+0000", 2024-03-20T09:36:50.428 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0070175339999999998 2024-03-20T09:36:50.428 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.428 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.428 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:36:50.428 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.570166+0000", 2024-03-20T09:36:50.428 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.010312231999999999 2024-03-20T09:36:50.428 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.428 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.428 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:36:50.428 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.575791+0000", 2024-03-20T09:36:50.429 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0056249339999999998 2024-03-20T09:36:50.429 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.429 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.429 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:36:50.429 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.575851+0000", 2024-03-20T09:36:50.429 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9979000000000003e-05 2024-03-20T09:36:50.429 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.429 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.429 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:36:50.429 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.575901+0000", 2024-03-20T09:36:50.429 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0098e-05 2024-03-20T09:36:50.429 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.429 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:36:50.429 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.430 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.430 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.430 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:181390 6.19 6:988ce146:::benchmark_data_smithi122_168655_object61281:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e829)", 2024-03-20T09:36:50.430 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:28:59.737246+0000", 2024-03-20T09:36:50.430 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 470.60475986099999, 2024-03-20T09:36:50.430 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.83879204200000002, 2024-03-20T09:36:50.430 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:36:50.430 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:36:50.430 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:36:50.430 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:36:50.430 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:36:50.430 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 181390 2024-03-20T09:36:50.430 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.430 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:36:50.431 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.431 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:36:50.431 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.737246+0000", 2024-03-20T09:36:50.431 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.431 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.431 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.431 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:36:50.431 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.737246+0000", 2024-03-20T09:36:50.431 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.431 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.431 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.431 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:36:50.431 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.737247+0000", 2024-03-20T09:36:50.431 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.9100000000000003e-07 2024-03-20T09:36:50.431 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.432 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.432 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:36:50.432 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.737250+0000", 2024-03-20T09:36:50.432 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.0230000000000001e-06 2024-03-20T09:36:50.432 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.432 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.432 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:36:50.432 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.737258+0000", 2024-03-20T09:36:50.432 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.3380000000000005e-06 2024-03-20T09:36:50.432 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.432 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.432 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:36:50.432 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.737265+0000", 2024-03-20T09:36:50.432 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.759e-06 2024-03-20T09:36:50.433 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.433 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.433 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.433 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.737302+0000", 2024-03-20T09:36:50.433 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6501999999999997e-05 2024-03-20T09:36:50.433 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.433 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.433 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:36:50.433 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.737306+0000", 2024-03-20T09:36:50.433 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.7199999999999997e-06 2024-03-20T09:36:50.433 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.433 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.433 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.433 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.554037+0000", 2024-03-20T09:36:50.433 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.81673037500000001 2024-03-20T09:36:50.434 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.434 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.434 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:36:50.434 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.554101+0000", 2024-03-20T09:36:50.434 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4214000000000006e-05 2024-03-20T09:36:50.434 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.434 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.434 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,2", 2024-03-20T09:36:50.434 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.554167+0000", 2024-03-20T09:36:50.434 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.6373999999999998e-05 2024-03-20T09:36:50.434 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.434 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.434 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:36:50.434 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.560066+0000", 2024-03-20T09:36:50.435 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0058986990000000003 2024-03-20T09:36:50.435 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.435 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.435 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:36:50.435 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.570209+0000", 2024-03-20T09:36:50.435 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.010142791999999999 2024-03-20T09:36:50.435 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.435 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.435 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:36:50.435 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.575955+0000", 2024-03-20T09:36:50.435 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0057464179999999997 2024-03-20T09:36:50.435 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.435 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.435 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:36:50.435 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.576002+0000", 2024-03-20T09:36:50.436 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6563999999999997e-05 2024-03-20T09:36:50.436 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.436 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.436 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:36:50.436 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.576038+0000", 2024-03-20T09:36:50.436 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6572999999999997e-05 2024-03-20T09:36:50.436 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.436 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:36:50.436 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.436 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.436 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.436 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:181428 6.19 6:9eb55ba5:::benchmark_data_smithi122_168655_object61319:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e829)", 2024-03-20T09:36:50.436 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:28:59.805617+0000", 2024-03-20T09:36:50.436 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 470.53638951200003, 2024-03-20T09:36:50.437 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.77596964700000004, 2024-03-20T09:36:50.437 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:36:50.437 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:36:50.437 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:36:50.437 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:36:50.437 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:36:50.437 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 181428 2024-03-20T09:36:50.437 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.437 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:36:50.437 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.437 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:36:50.437 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.805617+0000", 2024-03-20T09:36:50.437 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.437 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.437 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.438 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:36:50.438 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.805617+0000", 2024-03-20T09:36:50.438 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.438 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.438 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.438 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:36:50.438 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.805617+0000", 2024-03-20T09:36:50.438 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.4200000000000005e-07 2024-03-20T09:36:50.438 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.438 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.438 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:36:50.438 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.805621+0000", 2024-03-20T09:36:50.438 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.715e-06 2024-03-20T09:36:50.438 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.439 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.439 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:36:50.439 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.805630+0000", 2024-03-20T09:36:50.439 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5159999999999994e-06 2024-03-20T09:36:50.439 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.439 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.439 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:36:50.439 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.805635+0000", 2024-03-20T09:36:50.439 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8500000000000002e-06 2024-03-20T09:36:50.439 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.439 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.439 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.439 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.805688+0000", 2024-03-20T09:36:50.439 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3671999999999997e-05 2024-03-20T09:36:50.439 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.440 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.440 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:36:50.440 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:28:59.805697+0000", 2024-03-20T09:36:50.440 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.7539999999999995e-06 2024-03-20T09:36:50.440 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.440 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.440 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.440 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.555288+0000", 2024-03-20T09:36:50.440 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.74959142899999998 2024-03-20T09:36:50.440 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.440 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.440 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:36:50.440 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.555374+0000", 2024-03-20T09:36:50.440 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5894000000000001e-05 2024-03-20T09:36:50.440 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.441 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.441 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,2", 2024-03-20T09:36:50.441 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.555436+0000", 2024-03-20T09:36:50.441 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1285999999999996e-05 2024-03-20T09:36:50.441 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.441 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.441 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:36:50.441 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.560072+0000", 2024-03-20T09:36:50.441 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0046361830000000003 2024-03-20T09:36:50.441 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.441 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.441 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:36:50.441 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.576088+0000", 2024-03-20T09:36:50.441 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.016015830000000002 2024-03-20T09:36:50.441 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.442 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.442 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:36:50.442 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.581521+0000", 2024-03-20T09:36:50.442 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0054332989999999999 2024-03-20T09:36:50.442 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.442 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.442 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:36:50.442 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.581554+0000", 2024-03-20T09:36:50.442 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.2678999999999997e-05 2024-03-20T09:36:50.442 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.442 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.442 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:36:50.442 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.581586+0000", 2024-03-20T09:36:50.442 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.2697999999999998e-05 2024-03-20T09:36:50.443 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.443 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:36:50.443 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.443 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.443 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.443 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26216.0:181488 6.19 6:98b1b68c:::benchmark_data_smithi122_168655_object61379:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e829)", 2024-03-20T09:36:50.443 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:29:00.037703+0000", 2024-03-20T09:36:50.443 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 470.304303258, 2024-03-20T09:36:50.443 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54402109099999996, 2024-03-20T09:36:50.443 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:36:50.443 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:36:50.443 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:36:50.443 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26216", 2024-03-20T09:36:50.443 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/3902145104", 2024-03-20T09:36:50.443 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 181488 2024-03-20T09:36:50.444 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.444 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:36:50.444 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.444 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:36:50.444 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.037703+0000", 2024-03-20T09:36:50.444 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.444 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.444 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.444 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:36:50.444 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.037703+0000", 2024-03-20T09:36:50.444 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.444 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.444 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.444 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:36:50.445 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.037704+0000", 2024-03-20T09:36:50.445 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0589999999999999e-06 2024-03-20T09:36:50.445 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.445 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.445 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:36:50.445 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.037710+0000", 2024-03-20T09:36:50.445 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5999999999999997e-06 2024-03-20T09:36:50.445 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.445 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.445 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:36:50.445 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.037722+0000", 2024-03-20T09:36:50.445 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2112e-05 2024-03-20T09:36:50.445 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.445 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.446 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:36:50.446 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.037728+0000", 2024-03-20T09:36:50.446 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.3749999999999999e-06 2024-03-20T09:36:50.446 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.446 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.446 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.446 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.037773+0000", 2024-03-20T09:36:50.446 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5333000000000001e-05 2024-03-20T09:36:50.446 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.446 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.446 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:36:50.446 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.037782+0000", 2024-03-20T09:36:50.446 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.1869999999999997e-06 2024-03-20T09:36:50.447 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.447 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.447 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.447 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.556447+0000", 2024-03-20T09:36:50.447 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.51866584500000001 2024-03-20T09:36:50.447 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.447 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.447 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:36:50.447 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.556510+0000", 2024-03-20T09:36:50.447 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2515999999999996e-05 2024-03-20T09:36:50.447 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.447 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.447 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,2", 2024-03-20T09:36:50.447 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.556566+0000", 2024-03-20T09:36:50.448 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6193999999999998e-05 2024-03-20T09:36:50.448 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.448 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.448 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:36:50.448 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.560078+0000", 2024-03-20T09:36:50.448 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0035114690000000001 2024-03-20T09:36:50.448 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.448 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.448 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:36:50.448 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.576137+0000", 2024-03-20T09:36:50.448 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.016059455 2024-03-20T09:36:50.448 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.448 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.448 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:36:50.448 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.581630+0000", 2024-03-20T09:36:50.449 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0054933569999999999 2024-03-20T09:36:50.449 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.449 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.449 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:36:50.449 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.581687+0000", 2024-03-20T09:36:50.449 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6106000000000001e-05 2024-03-20T09:36:50.449 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.449 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.449 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:36:50.449 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:29:00.581724+0000", 2024-03-20T09:36:50.449 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7483e-05 2024-03-20T09:36:50.449 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.449 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:36:50.449 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.449 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.450 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.450 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:100031 7.0 7:0fe7022e:::benchmark_data_smithi122_187077_object100030:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e898)", 2024-03-20T09:36:50.450 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:32:19.515521+0000", 2024-03-20T09:36:50.450 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 270.826485053, 2024-03-20T09:36:50.450 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.220244419, 2024-03-20T09:36:50.450 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:36:50.450 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:36:50.450 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:36:50.450 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:36:50.450 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:36:50.450 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 100031 2024-03-20T09:36:50.450 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.450 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:36:50.450 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.451 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:36:50.451 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:19.515521+0000", 2024-03-20T09:36:50.451 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.451 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.451 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.451 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:36:50.451 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:19.515521+0000", 2024-03-20T09:36:50.451 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.451 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.451 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.451 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:36:50.451 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:19.515522+0000", 2024-03-20T09:36:50.451 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.243e-06 2024-03-20T09:36:50.451 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.452 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.453 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:36:50.453 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:19.515585+0000", 2024-03-20T09:36:50.453 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2323000000000006e-05 2024-03-20T09:36:50.453 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.453 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.453 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:36:50.453 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:19.515596+0000", 2024-03-20T09:36:50.453 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1015e-05 2024-03-20T09:36:50.454 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.454 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.454 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:36:50.454 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:19.515603+0000", 2024-03-20T09:36:50.454 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.5009999999999998e-06 2024-03-20T09:36:50.454 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.454 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.454 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.454 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:19.515648+0000", 2024-03-20T09:36:50.454 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5161999999999999e-05 2024-03-20T09:36:50.454 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.454 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.454 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:36:50.454 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:19.515762+0000", 2024-03-20T09:36:50.455 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00011344399999999999 2024-03-20T09:36:50.455 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.455 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.455 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 3,9", 2024-03-20T09:36:50.455 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:19.515863+0000", 2024-03-20T09:36:50.455 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000101523 2024-03-20T09:36:50.455 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.455 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.455 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:36:50.455 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:19.518652+0000", 2024-03-20T09:36:50.455 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.002788249 2024-03-20T09:36:50.455 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.455 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.455 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:36:50.456 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:19.518750+0000", 2024-03-20T09:36:50.456 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.7994999999999999e-05 2024-03-20T09:36:50.456 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.456 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.456 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:36:50.456 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:19.735661+0000", 2024-03-20T09:36:50.456 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.216911351 2024-03-20T09:36:50.456 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.456 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.456 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:36:50.456 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:19.735717+0000", 2024-03-20T09:36:50.456 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6385000000000003e-05 2024-03-20T09:36:50.456 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.456 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.456 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:36:50.457 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:19.735766+0000", 2024-03-20T09:36:50.457 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8228e-05 2024-03-20T09:36:50.457 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.457 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:36:50.457 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.457 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.457 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.457 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26917.0:71673 8.4 8:201f9a55:::benchmark_data_smithi122_197775_object71672:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e995)", 2024-03-20T09:36:50.457 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:39.218332+0000", 2024-03-20T09:36:50.457 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 11.123674338000001, 2024-03-20T09:36:50.457 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.18304461599999999, 2024-03-20T09:36:50.457 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:36:50.457 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:36:50.457 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:36:50.457 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26917", 2024-03-20T09:36:50.458 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:36:50.458 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 71673 2024-03-20T09:36:50.458 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.458 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:36:50.458 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.458 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:36:50.458 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.218332+0000", 2024-03-20T09:36:50.458 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.458 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.458 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.458 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:36:50.458 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.218332+0000", 2024-03-20T09:36:50.458 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.458 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.459 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.459 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:36:50.459 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.218333+0000", 2024-03-20T09:36:50.459 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.254e-06 2024-03-20T09:36:50.459 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.459 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.459 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:36:50.459 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.218391+0000", 2024-03-20T09:36:50.459 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7714999999999999e-05 2024-03-20T09:36:50.459 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.459 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.459 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:36:50.459 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.218400+0000", 2024-03-20T09:36:50.459 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.217e-06 2024-03-20T09:36:50.460 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.460 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.460 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:36:50.460 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.218407+0000", 2024-03-20T09:36:50.460 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.0879999999999997e-06 2024-03-20T09:36:50.460 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.460 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.460 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.460 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.218456+0000", 2024-03-20T09:36:50.460 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8973e-05 2024-03-20T09:36:50.460 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.460 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.460 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:36:50.460 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.218620+0000", 2024-03-20T09:36:50.460 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00016423300000000001 2024-03-20T09:36:50.461 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.461 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.461 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 9,10", 2024-03-20T09:36:50.461 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.218778+0000", 2024-03-20T09:36:50.461 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00015741099999999999 2024-03-20T09:36:50.461 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.461 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.461 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:36:50.461 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.220708+0000", 2024-03-20T09:36:50.461 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001930385 2024-03-20T09:36:50.461 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.461 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.461 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:36:50.461 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.221968+0000", 2024-03-20T09:36:50.461 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001259549 2024-03-20T09:36:50.462 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.462 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.462 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:36:50.462 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.401299+0000", 2024-03-20T09:36:50.462 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.17933179399999999 2024-03-20T09:36:50.462 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.462 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.462 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:36:50.462 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.401341+0000", 2024-03-20T09:36:50.462 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.1513e-05 2024-03-20T09:36:50.462 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.462 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.462 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:36:50.462 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.401376+0000", 2024-03-20T09:36:50.463 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.5484000000000001e-05 2024-03-20T09:36:50.463 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.463 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:36:50.463 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.463 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.463 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.463 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26917.0:71685 8.4 8:2bfcec03:::benchmark_data_smithi122_197775_object71684:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e995)", 2024-03-20T09:36:50.463 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:39.228848+0000", 2024-03-20T09:36:50.463 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 11.113157679, 2024-03-20T09:36:50.463 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.17282698899999999, 2024-03-20T09:36:50.463 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:36:50.463 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:36:50.463 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:36:50.463 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26917", 2024-03-20T09:36:50.463 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:36:50.464 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 71685 2024-03-20T09:36:50.464 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.464 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:36:50.464 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.464 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:36:50.464 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.228848+0000", 2024-03-20T09:36:50.464 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.464 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.464 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.464 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:36:50.464 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.228848+0000", 2024-03-20T09:36:50.464 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:36:50.464 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.464 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.465 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:36:50.465 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.228849+0000", 2024-03-20T09:36:50.465 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5899999999999996e-07 2024-03-20T09:36:50.465 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.465 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.465 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:36:50.465 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.228875+0000", 2024-03-20T09:36:50.465 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6030999999999999e-05 2024-03-20T09:36:50.465 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.465 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.465 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:36:50.465 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.228883+0000", 2024-03-20T09:36:50.465 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.1349999999999992e-06 2024-03-20T09:36:50.465 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.466 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.466 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:36:50.466 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.228889+0000", 2024-03-20T09:36:50.466 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2140000000000001e-06 2024-03-20T09:36:50.466 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.466 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.466 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:36:50.466 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.231223+0000", 2024-03-20T09:36:50.466 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0023334179999999999 2024-03-20T09:36:50.466 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.466 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.466 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:36:50.466 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.231338+0000", 2024-03-20T09:36:50.466 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000115443 2024-03-20T09:36:50.466 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.467 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.467 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 9,10", 2024-03-20T09:36:50.467 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.231433+0000", 2024-03-20T09:36:50.467 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.4357999999999993e-05 2024-03-20T09:36:50.467 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.467 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.467 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:36:50.467 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.232837+0000", 2024-03-20T09:36:50.467 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001404375 2024-03-20T09:36:50.467 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.467 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.467 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:36:50.467 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.233580+0000", 2024-03-20T09:36:50.467 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00074284199999999996 2024-03-20T09:36:50.467 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.468 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.468 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:36:50.468 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.401607+0000", 2024-03-20T09:36:50.468 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.16802669200000001 2024-03-20T09:36:50.468 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.468 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.468 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:36:50.468 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.401647+0000", 2024-03-20T09:36:50.468 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.0287000000000002e-05 2024-03-20T09:36:50.468 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:36:50.468 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:36:50.468 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:36:50.468 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.401675+0000", 2024-03-20T09:36:50.468 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8634999999999999e-05 2024-03-20T09:36:50.469 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.469 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:36:50.469 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.469 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:36:50.469 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:36:50.469 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:36:50.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:50.470 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:50.470 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:36:50.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:36:50.571 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:36:50.625 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:50.651 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:36:50.655 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:36:50.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:50.665 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:36:50.766 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:36:50.770 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 51 15 86210 86195 105.597 77.3125 0.0156752 0.00946564 2024-03-20T09:36:50.812 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:50.836 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:36:50.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:50.839 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:36:50.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:50 smithi115 ceph-mon[72476]: 1.d scrub starts 2024-03-20T09:36:50.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:50 smithi115 ceph-mon[72476]: 1.d scrub ok 2024-03-20T09:36:50.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:50 smithi122 ceph-mon[73495]: 1.d scrub starts 2024-03-20T09:36:50.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:50 smithi122 ceph-mon[73495]: 1.d scrub ok 2024-03-20T09:36:50.940 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:36:50.947 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:50 smithi026 ceph-mon[68309]: 1.d scrub starts 2024-03-20T09:36:50.947 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:50 smithi026 ceph-mon[68309]: 1.d scrub ok 2024-03-20T09:36:50.993 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:51.148 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:36:51.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:51.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:36:51.190 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_ops_in_flight 2024-03-20T09:36:51.252 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:36:51.293 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:51.317 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:36:51.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:51.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:36:51.340 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:36:51.421 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:36:51.465 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:51.629 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:36:51.669 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:36:51.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:51.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:36:51.770 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 52 16 87362 87346 104.95 71.9375 0.018299 0.00952413 2024-03-20T09:36:51.774 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:36:51.810 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:51.861 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:36:51.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:51.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:36:51.880 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:51 smithi026 ceph-mon[68309]: pgmap v2405: 47 pgs: 47 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-03-20T09:36:51.880 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:51 smithi026 ceph-mon[68309]: 1.2 scrub starts 2024-03-20T09:36:51.880 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:51 smithi026 ceph-mon[68309]: 1.2 scrub ok 2024-03-20T09:36:51.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:51 smithi115 ceph-mon[72476]: pgmap v2405: 47 pgs: 47 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-03-20T09:36:51.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:51 smithi115 ceph-mon[72476]: 1.2 scrub starts 2024-03-20T09:36:51.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:51 smithi115 ceph-mon[72476]: 1.2 scrub ok 2024-03-20T09:36:51.916 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:51 smithi122 ceph-mon[73495]: pgmap v2405: 47 pgs: 47 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-03-20T09:36:51.916 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:51 smithi122 ceph-mon[73495]: 1.2 scrub starts 2024-03-20T09:36:51.916 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:51 smithi122 ceph-mon[73495]: 1.2 scrub ok 2024-03-20T09:36:51.967 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:36:52.009 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:52.036 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:36:52.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:52.042 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:36:52.143 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:36:52.179 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:52.205 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:36:52.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:52.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:36:52.311 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:36:52.354 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:52.635 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:52 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2023373793' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:36:52.636 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:52 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:36:52.636 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:52 smithi122 ceph-mon[73495]: 1.11 scrub starts 2024-03-20T09:36:52.636 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:52 smithi122 ceph-mon[73495]: 1.11 scrub ok 2024-03-20T09:36:52.636 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:52 smithi122 ceph-mon[73495]: 1.1a scrub starts 2024-03-20T09:36:52.636 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:52 smithi122 ceph-mon[73495]: 1.1a scrub ok 2024-03-20T09:36:52.656 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.7 to 1 (10000) 2024-03-20T09:36:52.707 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:36:52.770 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 53 16 88554 88538 104.375 74.5 0.0199623 0.00957664 2024-03-20T09:36:52.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:52 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2023373793' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:36:52.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:52 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:36:52.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:52 smithi115 ceph-mon[72476]: 1.11 scrub starts 2024-03-20T09:36:52.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:52 smithi115 ceph-mon[72476]: 1.11 scrub ok 2024-03-20T09:36:52.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:52 smithi115 ceph-mon[72476]: 1.1a scrub starts 2024-03-20T09:36:52.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:52 smithi115 ceph-mon[72476]: 1.1a scrub ok 2024-03-20T09:36:52.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:52 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2023373793' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:36:52.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:52 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:36:52.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:52 smithi026 ceph-mon[68309]: 1.11 scrub starts 2024-03-20T09:36:52.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:52 smithi026 ceph-mon[68309]: 1.11 scrub ok 2024-03-20T09:36:52.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:52 smithi026 ceph-mon[68309]: 1.1a scrub starts 2024-03-20T09:36:52.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:52 smithi026 ceph-mon[68309]: 1.1a scrub ok 2024-03-20T09:36:53.622 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:36:53.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:53.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:36:53.729 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:36:53.772 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 54 16 89732 89716 103.802 73.625 0.0127175 0.00962952 2024-03-20T09:36:53.786 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:53.836 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:36:53.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:53.848 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:36:53.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:53 smithi122 ceph-mon[73495]: pgmap v2406: 47 pgs: 47 active+clean; 5.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-03-20T09:36:53.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:53 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-20T09:36:53.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:53 smithi122 ceph-mon[73495]: osdmap e999: 12 total, 12 up, 12 in 2024-03-20T09:36:53.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:53 smithi122 ceph-mon[73495]: osdmap e1000: 12 total, 12 up, 12 in 2024-03-20T09:36:53.950 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:36:53.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:53 smithi026 ceph-mon[68309]: pgmap v2406: 47 pgs: 47 active+clean; 5.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-03-20T09:36:53.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:53 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-20T09:36:53.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:53 smithi026 ceph-mon[68309]: osdmap e999: 12 total, 12 up, 12 in 2024-03-20T09:36:53.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:53 smithi026 ceph-mon[68309]: osdmap e1000: 12 total, 12 up, 12 in 2024-03-20T09:36:53.979 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:53 smithi115 ceph-mon[72476]: pgmap v2406: 47 pgs: 47 active+clean; 5.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-03-20T09:36:53.979 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:53 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-20T09:36:53.979 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:53 smithi115 ceph-mon[72476]: osdmap e999: 12 total, 12 up, 12 in 2024-03-20T09:36:53.979 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:53 smithi115 ceph-mon[72476]: osdmap e1000: 12 total, 12 up, 12 in 2024-03-20T09:36:53.982 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:54.007 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:36:54.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:54.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:36:54.113 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:36:54.141 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:54.162 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:36:54.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:54.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:36:54.267 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:36:54.318 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:54.347 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:36:54.348 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-03-20T09:36:54.348 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.348 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:90353 8.2 8:4f9c9781:::benchmark_data_smithi122_197775_object90352:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1001)", 2024-03-20T09:36:54.348 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:36:54.342108+0000", 2024-03-20T09:36:54.348 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.010261694, 2024-03-20T09:36:54.348 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010287011, 2024-03-20T09:36:54.348 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:36:54.348 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:36:54.348 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:36:54.348 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:36:54.348 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:36:54.349 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 90353 2024-03-20T09:36:54.349 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.349 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:36:54.349 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.349 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:36:54.349 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.342108+0000", 2024-03-20T09:36:54.349 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:36:54.349 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.349 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.349 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:36:54.349 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.342108+0000", 2024-03-20T09:36:54.349 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:36:54.349 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.349 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.350 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:36:54.350 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.342109+0000", 2024-03-20T09:36:54.350 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5999999999999999e-07 2024-03-20T09:36:54.350 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.350 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.350 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:36:54.350 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.342128+0000", 2024-03-20T09:36:54.350 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9474e-05 2024-03-20T09:36:54.350 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.350 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.350 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:36:54.350 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.342135+0000", 2024-03-20T09:36:54.350 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.584e-06 2024-03-20T09:36:54.351 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.351 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.351 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:36:54.351 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.342140+0000", 2024-03-20T09:36:54.351 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3109999999999996e-06 2024-03-20T09:36:54.351 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.351 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.351 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:36:54.351 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.347789+0000", 2024-03-20T09:36:54.351 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0056493410000000004 2024-03-20T09:36:54.351 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.351 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.351 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:36:54.351 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.347915+0000", 2024-03-20T09:36:54.352 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012575100000000001 2024-03-20T09:36:54.352 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.352 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.352 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,4", 2024-03-20T09:36:54.352 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.348079+0000", 2024-03-20T09:36:54.352 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016329699999999999 2024-03-20T09:36:54.352 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:36:54.352 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:36:54.352 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:36:54.352 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.352 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.352 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:90356 8.2 8:40f7858d:::benchmark_data_smithi122_197775_object90355:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1001)", 2024-03-20T09:36:54.352 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:36:54.347194+0000", 2024-03-20T09:36:54.352 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.0051757590000000003, 2024-03-20T09:36:54.353 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0052662120000000001, 2024-03-20T09:36:54.353 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:36:54.353 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:36:54.353 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:36:54.353 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:36:54.353 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:36:54.353 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 90356 2024-03-20T09:36:54.353 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.353 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:36:54.353 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.353 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:36:54.353 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.347194+0000", 2024-03-20T09:36:54.353 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:36:54.354 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.354 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.354 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:36:54.354 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.347194+0000", 2024-03-20T09:36:54.354 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:36:54.354 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.354 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.354 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:36:54.354 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.347195+0000", 2024-03-20T09:36:54.354 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.11e-06 2024-03-20T09:36:54.354 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.354 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.354 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:36:54.355 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.347215+0000", 2024-03-20T09:36:54.355 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9160999999999998e-05 2024-03-20T09:36:54.355 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.355 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.355 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:36:54.355 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.347225+0000", 2024-03-20T09:36:54.355 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0567000000000001e-05 2024-03-20T09:36:54.355 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.355 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.355 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:36:54.355 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.347231+0000", 2024-03-20T09:36:54.355 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0530000000000003e-06 2024-03-20T09:36:54.355 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.355 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.356 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:36:54.356 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.351143+0000", 2024-03-20T09:36:54.356 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0039114500000000003 2024-03-20T09:36:54.356 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.356 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.356 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:36:54.356 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.351322+0000", 2024-03-20T09:36:54.356 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000179631 2024-03-20T09:36:54.356 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.356 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.356 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,4", 2024-03-20T09:36:54.356 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.351491+0000", 2024-03-20T09:36:54.356 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016861600000000001 2024-03-20T09:36:54.357 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:36:54.357 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:36:54.357 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:36:54.357 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.357 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.357 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop_reply(client.26917.0:90350 8.2 e1001/974)", 2024-03-20T09:36:54.357 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:36:54.350995+0000", 2024-03-20T09:36:54.357 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.0013751690000000001, 2024-03-20T09:36:54.357 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0015101310000000001, 2024-03-20T09:36:54.357 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:36:54.357 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "queued for pg", 2024-03-20T09:36:54.357 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:36:54.357 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.357 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:36:54.358 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.350995+0000", 2024-03-20T09:36:54.358 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:36:54.358 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.358 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.358 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:36:54.358 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.350995+0000", 2024-03-20T09:36:54.358 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:36:54.358 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.358 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.358 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:36:54.358 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.350996+0000", 2024-03-20T09:36:54.358 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5700000000000002e-07 2024-03-20T09:36:54.358 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.359 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.359 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:36:54.359 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.351002+0000", 2024-03-20T09:36:54.359 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0599999999999996e-06 2024-03-20T09:36:54.359 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.359 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.359 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:36:54.359 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.351016+0000", 2024-03-20T09:36:54.359 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4353e-05 2024-03-20T09:36:54.359 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.359 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.360 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:36:54.360 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.351023+0000", 2024-03-20T09:36:54.360 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.17e-06 2024-03-20T09:36:54.360 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:36:54.360 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:36:54.360 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:36:54.360 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.360 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.360 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:90347 8.2 8:43cc9a41:::benchmark_data_smithi122_197775_object90346:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1001)", 2024-03-20T09:36:54.360 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:36:54.341364+0000", 2024-03-20T09:36:54.360 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.011005812, 2024-03-20T09:36:54.360 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.011184632999999999, 2024-03-20T09:36:54.360 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:36:54.361 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:36:54.361 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:36:54.361 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:36:54.361 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:36:54.361 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 90347 2024-03-20T09:36:54.361 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.361 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:36:54.361 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.361 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:36:54.361 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.341364+0000", 2024-03-20T09:36:54.361 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:36:54.361 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.362 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.362 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:36:54.362 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.341364+0000", 2024-03-20T09:36:54.362 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:36:54.362 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.362 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.362 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:36:54.362 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.341366+0000", 2024-03-20T09:36:54.362 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5e-06 2024-03-20T09:36:54.362 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.362 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.362 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:36:54.362 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.341390+0000", 2024-03-20T09:36:54.363 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4066999999999998e-05 2024-03-20T09:36:54.363 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.363 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.363 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:36:54.363 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.341402+0000", 2024-03-20T09:36:54.363 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1843e-05 2024-03-20T09:36:54.363 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.363 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.363 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:36:54.363 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.341409+0000", 2024-03-20T09:36:54.363 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8800000000000008e-06 2024-03-20T09:36:54.363 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.363 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.363 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:36:54.364 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.341463+0000", 2024-03-20T09:36:54.364 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3967e-05 2024-03-20T09:36:54.364 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.364 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.364 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:36:54.364 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.341605+0000", 2024-03-20T09:36:54.364 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014108600000000001 2024-03-20T09:36:54.364 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.364 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.364 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,4", 2024-03-20T09:36:54.364 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.341745+0000", 2024-03-20T09:36:54.364 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000140772 2024-03-20T09:36:54.365 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.365 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.365 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:36:54.365 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.347745+0000", 2024-03-20T09:36:54.365 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0059998050000000004 2024-03-20T09:36:54.365 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.365 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.365 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:36:54.365 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.350964+0000", 2024-03-20T09:36:54.365 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0032192100000000001 2024-03-20T09:36:54.365 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:36:54.365 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:36:54.365 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:36:54.365 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.365 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.366 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:90350 8.2 8:464f2ae5:::benchmark_data_smithi122_197775_object90349:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1001)", 2024-03-20T09:36:54.366 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:36:54.342052+0000", 2024-03-20T09:36:54.366 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.010318447999999999, 2024-03-20T09:36:54.366 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010594279, 2024-03-20T09:36:54.366 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:36:54.366 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:36:54.366 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:36:54.366 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:36:54.366 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:36:54.366 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 90350 2024-03-20T09:36:54.367 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.367 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:36:54.367 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.367 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:36:54.367 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.342052+0000", 2024-03-20T09:36:54.367 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:36:54.367 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.367 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.367 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:36:54.367 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.342052+0000", 2024-03-20T09:36:54.367 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:36:54.367 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.367 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.368 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:36:54.368 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.342053+0000", 2024-03-20T09:36:54.368 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9999999999999995e-07 2024-03-20T09:36:54.368 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.368 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.368 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:36:54.368 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.342074+0000", 2024-03-20T09:36:54.368 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1202000000000001e-05 2024-03-20T09:36:54.368 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.368 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.368 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:36:54.368 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.342085+0000", 2024-03-20T09:36:54.368 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1223000000000001e-05 2024-03-20T09:36:54.368 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.368 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.369 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:36:54.369 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.342092+0000", 2024-03-20T09:36:54.369 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5039999999999996e-06 2024-03-20T09:36:54.369 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.369 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.369 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:36:54.369 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.344768+0000", 2024-03-20T09:36:54.369 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002675204 2024-03-20T09:36:54.369 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.369 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.369 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:36:54.369 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.344909+0000", 2024-03-20T09:36:54.369 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014133599999999999 2024-03-20T09:36:54.369 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.370 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.370 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 0,4", 2024-03-20T09:36:54.370 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.345057+0000", 2024-03-20T09:36:54.370 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000147816 2024-03-20T09:36:54.370 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:36:54.370 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:36:54.370 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:36:54.370 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:54.350970+0000", 2024-03-20T09:36:54.370 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0059134510000000001 2024-03-20T09:36:54.370 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:36:54.370 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:36:54.370 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:36:54.370 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:36:54.371 INFO:teuthology.orchestra.run.smithi122.stdout: ], 2024-03-20T09:36:54.371 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 5 2024-03-20T09:36:54.371 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:36:54.372 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 8 1 2024-03-20T09:36:54.402 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:36:54.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:54.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:36:54.511 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:36:54.540 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:54.561 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:36:54.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:54.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:36:54.627 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:36:54.666 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:36:54.772 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 55 16 90799 90783 103.128 66.6875 0.0146437 0.00969096 2024-03-20T09:36:54.777 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:55.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:54 smithi115 ceph-mon[72476]: 1.1c scrub starts 2024-03-20T09:36:55.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:54 smithi115 ceph-mon[72476]: 1.1c scrub ok 2024-03-20T09:36:55.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:54 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:36:55.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:54 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:36:55.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:54 smithi115 ceph-mon[72476]: 1.18 scrub starts 2024-03-20T09:36:55.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:54 smithi115 ceph-mon[72476]: 1.18 scrub ok 2024-03-20T09:36:55.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:54 smithi115 ceph-mon[72476]: osdmap e1001: 12 total, 12 up, 12 in 2024-03-20T09:36:55.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:54 smithi115 ceph-mon[72476]: 8.5 scrub starts 2024-03-20T09:36:55.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:54 smithi115 ceph-mon[72476]: pgmap v2410: 47 pgs: 47 active+clean; 5.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-03-20T09:36:55.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:54 smithi122 ceph-mon[73495]: 1.1c scrub starts 2024-03-20T09:36:55.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:54 smithi122 ceph-mon[73495]: 1.1c scrub ok 2024-03-20T09:36:55.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:54 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:36:55.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:54 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:36:55.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:54 smithi122 ceph-mon[73495]: 1.18 scrub starts 2024-03-20T09:36:55.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:54 smithi122 ceph-mon[73495]: 1.18 scrub ok 2024-03-20T09:36:55.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:54 smithi122 ceph-mon[73495]: osdmap e1001: 12 total, 12 up, 12 in 2024-03-20T09:36:55.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:54 smithi122 ceph-mon[73495]: 8.5 scrub starts 2024-03-20T09:36:55.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:54 smithi122 ceph-mon[73495]: pgmap v2410: 47 pgs: 47 active+clean; 5.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-03-20T09:36:55.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:54 smithi026 ceph-mon[68309]: 1.1c scrub starts 2024-03-20T09:36:55.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:54 smithi026 ceph-mon[68309]: 1.1c scrub ok 2024-03-20T09:36:55.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:54 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:36:55.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:54 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:36:55.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:54 smithi026 ceph-mon[68309]: 1.18 scrub starts 2024-03-20T09:36:55.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:54 smithi026 ceph-mon[68309]: 1.18 scrub ok 2024-03-20T09:36:55.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:54 smithi026 ceph-mon[68309]: osdmap e1001: 12 total, 12 up, 12 in 2024-03-20T09:36:55.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:54 smithi026 ceph-mon[68309]: 8.5 scrub starts 2024-03-20T09:36:55.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:54 smithi026 ceph-mon[68309]: pgmap v2410: 47 pgs: 47 active+clean; 5.3 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s 2024-03-20T09:36:55.358 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:36:55.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:55.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:36:55.464 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:36:55.506 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:55.550 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:36:55.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:55.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:36:55.659 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:36:55.718 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:55.772 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 56 16 91852 91836 102.461 65.8125 0.0196143 0.00975509 2024-03-20T09:36:55.849 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:36:55.851 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:55 smithi026 ceph-mon[68309]: 1.1b scrub starts 2024-03-20T09:36:55.851 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:55 smithi026 ceph-mon[68309]: 1.1b scrub ok 2024-03-20T09:36:55.851 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:55 smithi026 ceph-mon[68309]: 1.3 scrub starts 2024-03-20T09:36:55.852 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:55 smithi026 ceph-mon[68309]: 1.16 scrub starts 2024-03-20T09:36:55.852 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:55 smithi026 ceph-mon[68309]: 1.16 scrub ok 2024-03-20T09:36:55.852 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:55 smithi026 ceph-mon[68309]: 1.3 scrub ok 2024-03-20T09:36:55.852 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:55 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1493720509' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:36:56.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:55 smithi115 ceph-mon[72476]: 1.1b scrub starts 2024-03-20T09:36:56.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:55 smithi115 ceph-mon[72476]: 1.1b scrub ok 2024-03-20T09:36:56.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:55 smithi115 ceph-mon[72476]: 1.3 scrub starts 2024-03-20T09:36:56.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:55 smithi115 ceph-mon[72476]: 1.16 scrub starts 2024-03-20T09:36:56.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:55 smithi115 ceph-mon[72476]: 1.16 scrub ok 2024-03-20T09:36:56.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:55 smithi115 ceph-mon[72476]: 1.3 scrub ok 2024-03-20T09:36:56.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:55 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1493720509' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:36:56.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:55 smithi122 ceph-mon[73495]: 1.1b scrub starts 2024-03-20T09:36:56.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:55 smithi122 ceph-mon[73495]: 1.1b scrub ok 2024-03-20T09:36:56.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:55 smithi122 ceph-mon[73495]: 1.3 scrub starts 2024-03-20T09:36:56.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:55 smithi122 ceph-mon[73495]: 1.16 scrub starts 2024-03-20T09:36:56.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:55 smithi122 ceph-mon[73495]: 1.16 scrub ok 2024-03-20T09:36:56.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:55 smithi122 ceph-mon[73495]: 1.3 scrub ok 2024-03-20T09:36:56.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:55 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1493720509' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:36:56.774 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 57 16 92892 92876 101.804 65 0.0122625 0.00981909 2024-03-20T09:36:56.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:56 smithi026 ceph-mon[68309]: 8.5 scrub ok 2024-03-20T09:36:56.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:56 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:36:56.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:56 smithi026 ceph-mon[68309]: Cluster is now healthy 2024-03-20T09:36:56.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:56 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1493720509' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:36:56.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:56 smithi026 ceph-mon[68309]: osdmap e1002: 12 total, 12 up, 12 in 2024-03-20T09:36:56.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:56 smithi026 ceph-mon[68309]: pgmap v2412: 47 pgs: 47 active+clean; 5.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.19k op/s 2024-03-20T09:36:56.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:56 smithi026 ceph-mon[68309]: 8.1 deep-scrub starts 2024-03-20T09:36:57.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:56 smithi115 ceph-mon[72476]: 8.5 scrub ok 2024-03-20T09:36:57.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:56 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:36:57.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:56 smithi115 ceph-mon[72476]: Cluster is now healthy 2024-03-20T09:36:57.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:56 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1493720509' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:36:57.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:56 smithi115 ceph-mon[72476]: osdmap e1002: 12 total, 12 up, 12 in 2024-03-20T09:36:57.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:56 smithi115 ceph-mon[72476]: pgmap v2412: 47 pgs: 47 active+clean; 5.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.19k op/s 2024-03-20T09:36:57.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:56 smithi115 ceph-mon[72476]: 8.1 deep-scrub starts 2024-03-20T09:36:57.150 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:36:57.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:56 smithi122 ceph-mon[73495]: 8.5 scrub ok 2024-03-20T09:36:57.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:56 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:36:57.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:56 smithi122 ceph-mon[73495]: Cluster is now healthy 2024-03-20T09:36:57.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:56 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1493720509' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:36:57.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:56 smithi122 ceph-mon[73495]: osdmap e1002: 12 total, 12 up, 12 in 2024-03-20T09:36:57.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:56 smithi122 ceph-mon[73495]: pgmap v2412: 47 pgs: 47 active+clean; 5.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.19k op/s 2024-03-20T09:36:57.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:56 smithi122 ceph-mon[73495]: 8.1 deep-scrub starts 2024-03-20T09:36:57.773 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 58 16 93869 93853 101.102 61.0625 0.00800084 0.00988646 2024-03-20T09:36:58.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:57 smithi115 ceph-mon[72476]: 1.14 deep-scrub starts 2024-03-20T09:36:58.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:57 smithi115 ceph-mon[72476]: 1.14 deep-scrub ok 2024-03-20T09:36:58.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:57 smithi122 ceph-mon[73495]: 1.14 deep-scrub starts 2024-03-20T09:36:58.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:57 smithi122 ceph-mon[73495]: 1.14 deep-scrub ok 2024-03-20T09:36:58.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:57 smithi026 ceph-mon[68309]: 1.14 deep-scrub starts 2024-03-20T09:36:58.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:57 smithi026 ceph-mon[68309]: 1.14 deep-scrub ok 2024-03-20T09:36:58.490 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:36:58.530 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:36:58.562 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:36:58.595 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:36:58.625 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:36:58.660 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:36:58.685 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:36:58.723 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:36:58.736 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:36:58.766 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:36:58.772 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 59 16 94871 94855 100.45 62.625 0.00704254 0.0099509 2024-03-20T09:36:58.797 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:36:58.841 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:36:58.875 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:36:58.880 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:58 smithi122 ceph-mon[73495]: 1.0 scrub starts 2024-03-20T09:36:58.880 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:58 smithi122 ceph-mon[73495]: 1.0 scrub ok 2024-03-20T09:36:58.880 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:58 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/599348092' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:36:58.880 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:58 smithi122 ceph-mon[73495]: pgmap v2413: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s 2024-03-20T09:36:58.880 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:58 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:36:58.880 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:58 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:36:58.887 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:36:58.887 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:36:58.887 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:36:58.887 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:36:58.887 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:36:58.887 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:36:58.887 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:36:58.887 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:36:58.887 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:36:58.888 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:36:58.888 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:36:58.888 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:36:58.907 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:36:58.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:58.916 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:36:58.944 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_blocked_ops 2024-03-20T09:36:59.017 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:36:59.048 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:59.071 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:36:59.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:59.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:36:59.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:58 smithi115 ceph-mon[72476]: 1.0 scrub starts 2024-03-20T09:36:59.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:58 smithi115 ceph-mon[72476]: 1.0 scrub ok 2024-03-20T09:36:59.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:58 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/599348092' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:36:59.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:58 smithi115 ceph-mon[72476]: pgmap v2413: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s 2024-03-20T09:36:59.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:58 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:36:59.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:58 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:36:59.176 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:36:59.203 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:59.218 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:58 smithi026 ceph-mon[68309]: 1.0 scrub starts 2024-03-20T09:36:59.219 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:58 smithi026 ceph-mon[68309]: 1.0 scrub ok 2024-03-20T09:36:59.219 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:58 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/599348092' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:36:59.219 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:58 smithi026 ceph-mon[68309]: pgmap v2413: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.6 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 69 MiB/s wr, 1.10k op/s 2024-03-20T09:36:59.219 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:58 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:36:59.219 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:58 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:36:59.222 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:36:59.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:59.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:36:59.326 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:36:59.388 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:59.417 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:36:59.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:59.429 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:36:59.449 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:36:59.530 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:36:59.559 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:59.583 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:36:59.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:59.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:36:59.690 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:36:59.735 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:59.776 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:2024-03-20T09:36:59.787327+0000 min lat: 0.00110904 max lat: 0.220633 avg lat: 0.0100226 2024-03-20T09:36:59.776 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T09:36:59.776 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 60 16 95787 95771 99.7244 57.25 0.0179374 0.0100226 2024-03-20T09:36:59.783 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:36:59.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:59.788 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:36:59.889 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:36:59.918 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:36:59.939 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:36:59.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:36:59.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:37:00.044 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:37:00.077 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:00.100 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:37:00.103 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:36:59 smithi122 ceph-mon[73495]: osdmap e1003: 12 total, 12 up, 12 in 2024-03-20T09:37:00.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:00.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:37:00.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:36:59 smithi115 ceph-mon[72476]: osdmap e1003: 12 total, 12 up, 12 in 2024-03-20T09:37:00.206 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:37:00.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:36:59 smithi026 ceph-mon[68309]: osdmap e1003: 12 total, 12 up, 12 in 2024-03-20T09:37:00.234 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:00.255 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:37:00.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:00.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:37:00.358 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:37:00.434 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:00.776 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 61 16 96737 96721 99.063 59.375 0.0114735 0.0100905 2024-03-20T09:37:01.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:00 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:37:01.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:00 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:37:01.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:00 smithi115 ceph-mon[72476]: pgmap v2415: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 952 op/s 2024-03-20T09:37:01.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:00 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:37:01.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:00 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:37:01.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:00 smithi122 ceph-mon[73495]: pgmap v2415: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 952 op/s 2024-03-20T09:37:01.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:00 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:37:01.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:00 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:37:01.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:00 smithi026 ceph-mon[68309]: pgmap v2415: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 952 op/s 2024-03-20T09:37:01.776 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 62 16 97711 97695 98.4472 60.875 0.00820843 0.0101527 2024-03-20T09:37:02.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:01 smithi115 ceph-mon[72476]: 1.4 scrub starts 2024-03-20T09:37:02.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:01 smithi115 ceph-mon[72476]: 1.4 scrub ok 2024-03-20T09:37:02.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:01 smithi115 ceph-mon[72476]: osdmap e1004: 12 total, 12 up, 12 in 2024-03-20T09:37:02.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:01 smithi122 ceph-mon[73495]: 1.4 scrub starts 2024-03-20T09:37:02.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:01 smithi122 ceph-mon[73495]: 1.4 scrub ok 2024-03-20T09:37:02.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:01 smithi122 ceph-mon[73495]: osdmap e1004: 12 total, 12 up, 12 in 2024-03-20T09:37:02.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:01 smithi026 ceph-mon[68309]: 1.4 scrub starts 2024-03-20T09:37:02.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:01 smithi026 ceph-mon[68309]: 1.4 scrub ok 2024-03-20T09:37:02.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:01 smithi026 ceph-mon[68309]: osdmap e1004: 12 total, 12 up, 12 in 2024-03-20T09:37:02.449 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:37:02.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:02.453 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:37:02.502 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:37:02.553 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:37:02.597 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:02.776 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 63 15 98490 98475 97.6585 48.75 0.00993523 0.0102348 2024-03-20T09:37:02.810 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:37:02.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:02.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:37:02.916 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:37:02.929 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.8 to 1 (10000) 2024-03-20T09:37:02.954 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:03.059 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:37:03.063 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:02 smithi122 ceph-mon[73495]: 1.8 deep-scrub starts 2024-03-20T09:37:03.063 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:02 smithi122 ceph-mon[73495]: 1.8 deep-scrub ok 2024-03-20T09:37:03.064 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:02 smithi122 ceph-mon[73495]: 8.9 scrub starts 2024-03-20T09:37:03.064 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:02 smithi122 ceph-mon[73495]: pgmap v2417: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 898 op/s 2024-03-20T09:37:03.064 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:02 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3187275973' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:37:03.064 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:02 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:37:03.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:03.065 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:37:03.166 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:37:03.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:02 smithi026 ceph-mon[68309]: 1.8 deep-scrub starts 2024-03-20T09:37:03.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:02 smithi026 ceph-mon[68309]: 1.8 deep-scrub ok 2024-03-20T09:37:03.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:02 smithi026 ceph-mon[68309]: 8.9 scrub starts 2024-03-20T09:37:03.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:02 smithi026 ceph-mon[68309]: pgmap v2417: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 898 op/s 2024-03-20T09:37:03.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:02 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3187275973' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:37:03.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:02 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:37:03.217 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:03.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:02 smithi115 ceph-mon[72476]: 1.8 deep-scrub starts 2024-03-20T09:37:03.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:02 smithi115 ceph-mon[72476]: 1.8 deep-scrub ok 2024-03-20T09:37:03.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:02 smithi115 ceph-mon[72476]: 8.9 scrub starts 2024-03-20T09:37:03.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:02 smithi115 ceph-mon[72476]: pgmap v2417: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 5.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 898 op/s 2024-03-20T09:37:03.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:02 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3187275973' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:37:03.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:02 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:37:03.776 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 64 16 99273 99257 96.8964 48.875 0.00931686 0.010315 2024-03-20T09:37:04.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:03 smithi122 ceph-mon[73495]: 8.e scrub starts 2024-03-20T09:37:04.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:03 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:37:04.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:03 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:37:04.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:03 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-03-20T09:37:04.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:03 smithi122 ceph-mon[73495]: osdmap e1005: 12 total, 12 up, 12 in 2024-03-20T09:37:04.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:03 smithi026 ceph-mon[68309]: 8.e scrub starts 2024-03-20T09:37:04.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:03 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:37:04.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:03 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:37:04.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:03 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-03-20T09:37:04.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:03 smithi026 ceph-mon[68309]: osdmap e1005: 12 total, 12 up, 12 in 2024-03-20T09:37:04.326 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:37:04.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:03 smithi115 ceph-mon[72476]: 8.e scrub starts 2024-03-20T09:37:04.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:03 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:37:04.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:03 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:37:04.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:03 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-03-20T09:37:04.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:03 smithi115 ceph-mon[72476]: osdmap e1005: 12 total, 12 up, 12 in 2024-03-20T09:37:04.548 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:37:04.548 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:37:04.777 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 65 16 100169 100153 96.2673 56 0.0112705 0.010382 2024-03-20T09:37:05.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:04 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:37:05.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:04 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:37:05.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:04 smithi122 ceph-mon[73495]: 8.9 scrub ok 2024-03-20T09:37:05.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:04 smithi122 ceph-mon[73495]: 8.4 scrub starts 2024-03-20T09:37:05.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:04 smithi122 ceph-mon[73495]: osdmap e1006: 12 total, 12 up, 12 in 2024-03-20T09:37:05.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:04 smithi122 ceph-mon[73495]: 8.e scrub ok 2024-03-20T09:37:05.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:04 smithi122 ceph-mon[73495]: pgmap v2420: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.0 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 850 op/s 2024-03-20T09:37:05.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:04 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:37:05.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:04 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:37:05.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:04 smithi026 ceph-mon[68309]: 8.9 scrub ok 2024-03-20T09:37:05.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:04 smithi026 ceph-mon[68309]: 8.4 scrub starts 2024-03-20T09:37:05.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:04 smithi026 ceph-mon[68309]: osdmap e1006: 12 total, 12 up, 12 in 2024-03-20T09:37:05.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:04 smithi026 ceph-mon[68309]: 8.e scrub ok 2024-03-20T09:37:05.211 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:04 smithi026 ceph-mon[68309]: pgmap v2420: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.0 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 850 op/s 2024-03-20T09:37:05.365 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:37:05.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:05.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:37:05.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:04 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:37:05.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:04 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:37:05.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:04 smithi115 ceph-mon[72476]: 8.9 scrub ok 2024-03-20T09:37:05.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:04 smithi115 ceph-mon[72476]: 8.4 scrub starts 2024-03-20T09:37:05.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:04 smithi115 ceph-mon[72476]: osdmap e1006: 12 total, 12 up, 12 in 2024-03-20T09:37:05.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:04 smithi115 ceph-mon[72476]: 8.e scrub ok 2024-03-20T09:37:05.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:04 smithi115 ceph-mon[72476]: pgmap v2420: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.0 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 850 op/s 2024-03-20T09:37:05.397 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 11 1 2024-03-20T09:37:05.448 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_historic_ops 2024-03-20T09:37:05.476 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:37:05.545 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:05.675 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:37:05.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:05.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:37:05.780 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 66 16 101105 101089 95.6931 58.5 0.0149487 0.0104433 2024-03-20T09:37:05.783 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:37:05.844 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:05.877 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:37:05.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:05.884 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:37:05.930 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:37:05.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:05 smithi026 ceph-mon[68309]: 8.4 scrub ok 2024-03-20T09:37:05.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:05 smithi026 ceph-mon[68309]: 1.b scrub starts 2024-03-20T09:37:05.985 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:37:06.013 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:06.035 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:37:06.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:06.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:37:06.139 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:37:06.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:05 smithi122 ceph-mon[73495]: 8.4 scrub ok 2024-03-20T09:37:06.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:05 smithi122 ceph-mon[73495]: 1.b scrub starts 2024-03-20T09:37:06.226 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:37:06.231 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:06.257 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:37:06.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:05 smithi115 ceph-mon[72476]: 8.4 scrub ok 2024-03-20T09:37:06.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:05 smithi115 ceph-mon[72476]: 1.b scrub starts 2024-03-20T09:37:06.538 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:37:06.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:06.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:37:06.648 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:37:06.677 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:06.699 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:37:06.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:06.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:37:06.778 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 67 16 101919 101903 95.0243 50.875 0.00464263 0.0105173 2024-03-20T09:37:06.803 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:37:06.830 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:06.852 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:37:06.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:06.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:37:06.957 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:37:06.985 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:07.006 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:37:07.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:07.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:37:07.110 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:37:07.136 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:07.160 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:37:07.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:07.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:37:07.186 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:06 smithi026 ceph-mon[68309]: 1.b scrub ok 2024-03-20T09:37:07.186 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:06 smithi026 ceph-mon[68309]: pgmap v2421: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.00k op/s 2024-03-20T09:37:07.264 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:37:07.318 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:07.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:06 smithi115 ceph-mon[72476]: 1.b scrub ok 2024-03-20T09:37:07.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:06 smithi115 ceph-mon[72476]: pgmap v2421: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.00k op/s 2024-03-20T09:37:07.403 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:06 smithi122 ceph-mon[73495]: 1.b scrub ok 2024-03-20T09:37:07.403 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:06 smithi122 ceph-mon[73495]: pgmap v2421: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.00k op/s 2024-03-20T09:37:07.778 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 68 16 102820 102804 94.455 56.3125 0.00758093 0.0105809 2024-03-20T09:37:07.976 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:37:08.099 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:37:08.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:08.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:37:08.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:07 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1083124562' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:37:08.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:07 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:37:08.205 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:37:08.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:07 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1083124562' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:37:08.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:07 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:37:08.232 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:07 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1083124562' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:37:08.232 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:07 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:37:08.234 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:08.256 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:37:08.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:08.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:37:08.362 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:37:08.390 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:08.411 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:37:08.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:08.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:37:08.516 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:37:08.564 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:08.779 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 69 16 103770 103754 93.9448 59.375 0.012793 0.0106389 2024-03-20T09:37:08.954 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:08 smithi026 ceph-mon[68309]: 1.9 scrub starts 2024-03-20T09:37:08.954 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:08 smithi026 ceph-mon[68309]: 1.12 scrub starts 2024-03-20T09:37:08.954 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:08 smithi026 ceph-mon[68309]: 1.12 scrub ok 2024-03-20T09:37:08.955 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:08 smithi026 ceph-mon[68309]: 1.9 scrub ok 2024-03-20T09:37:08.955 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:08 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:37:08.955 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:08 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:37:08.955 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:08 smithi026 ceph-mon[68309]: osdmap e1007: 12 total, 12 up, 12 in 2024-03-20T09:37:08.955 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:08 smithi026 ceph-mon[68309]: pgmap v2423: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 826 op/s 2024-03-20T09:37:09.142 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:37:09.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:09.147 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:37:09.248 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:37:09.292 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:08 smithi122 ceph-mon[73495]: 1.9 scrub starts 2024-03-20T09:37:09.292 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:08 smithi122 ceph-mon[73495]: 1.12 scrub starts 2024-03-20T09:37:09.292 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:08 smithi122 ceph-mon[73495]: 1.12 scrub ok 2024-03-20T09:37:09.292 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:08 smithi122 ceph-mon[73495]: 1.9 scrub ok 2024-03-20T09:37:09.292 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:08 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:37:09.292 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:08 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:37:09.292 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:08 smithi122 ceph-mon[73495]: osdmap e1007: 12 total, 12 up, 12 in 2024-03-20T09:37:09.292 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:08 smithi122 ceph-mon[73495]: pgmap v2423: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 826 op/s 2024-03-20T09:37:09.299 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:09.334 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:37:09.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:09.343 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:37:09.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:08 smithi115 ceph-mon[72476]: 1.9 scrub starts 2024-03-20T09:37:09.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:08 smithi115 ceph-mon[72476]: 1.12 scrub starts 2024-03-20T09:37:09.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:08 smithi115 ceph-mon[72476]: 1.12 scrub ok 2024-03-20T09:37:09.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:08 smithi115 ceph-mon[72476]: 1.9 scrub ok 2024-03-20T09:37:09.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:08 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:37:09.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:08 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:37:09.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:08 smithi115 ceph-mon[72476]: osdmap e1007: 12 total, 12 up, 12 in 2024-03-20T09:37:09.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:08 smithi115 ceph-mon[72476]: pgmap v2423: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 826 op/s 2024-03-20T09:37:09.444 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:37:09.493 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:09.524 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:37:09.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:09.532 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:37:09.633 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:37:09.662 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:09.683 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:37:09.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:09.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:37:09.779 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 70 16 104668 104652 93.4046 56.125 0.00918943 0.0107004 2024-03-20T09:37:09.788 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:37:09.849 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:09.888 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:37:09.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:09.893 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:37:09.994 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:37:10.023 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:10.044 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:37:10.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:10.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:37:10.149 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:37:10.157 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:37:10.157 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:37:10.179 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:10.200 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:37:10.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:10.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:37:10.304 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:37:10.358 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:10.780 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 71 16 105584 105568 92.8954 57.25 0.0151526 0.0107593 2024-03-20T09:37:11.054 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:37:11.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:11.059 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:37:11.074 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:37:11.160 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:37:11.206 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:11.678 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:37:11.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:11.683 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:37:11.707 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:37:11.709 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-20T09:37:11.750 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:37:11.780 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 72 16 106259 106243 92.1913 42.1875 0.239306 0.0108279 2024-03-20T09:37:11.783 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:37:11.806 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:37:11.823 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:11.850 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:37:11.853 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:11 smithi122 ceph-mon[73495]: pgmap v2424: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 870 op/s 2024-03-20T09:37:11.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:11.854 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:37:11.863 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:37:11.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:11 smithi115 ceph-mon[72476]: pgmap v2424: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 870 op/s 2024-03-20T09:37:11.943 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:37:11.955 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:37:11.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:11 smithi026 ceph-mon[68309]: pgmap v2424: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 870 op/s 2024-03-20T09:37:11.978 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:37:11.982 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:12.002 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:37:12.005 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:37:12.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:12.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:37:12.054 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:37:12.093 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:37:12.107 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:37:12.120 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:37:12.162 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:37:12.165 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:12.200 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:37:12.202 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:37:12.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:12.205 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:37:12.234 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:37:12.245 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:37:12.245 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:37:12.245 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:37:12.245 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:37:12.245 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:37:12.245 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:37:12.245 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:37:12.245 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:37:12.245 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:37:12.245 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:37:12.245 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:37:12.245 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:37:12.306 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:37:12.339 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:12.361 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:37:12.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:12.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:37:12.466 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:37:12.513 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:12.543 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:37:12.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:12.552 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:37:12.638 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.11 to 1 (10000) 2024-03-20T09:37:12.653 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:37:12.727 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:12.783 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 73 16 107099 107083 91.6438 52.5 0.0125924 0.010907 2024-03-20T09:37:12.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:12 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1961022380' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:37:12.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:12 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3644929674' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-03-20T09:37:12.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:12 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-03-20T09:37:12.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:12 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1961022380' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:37:12.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:12 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3644929674' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-03-20T09:37:12.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:12 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-03-20T09:37:12.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:12 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1961022380' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:37:12.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:12 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3644929674' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-03-20T09:37:12.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:12 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-03-20T09:37:13.697 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:37:13.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:13.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:37:13.783 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 74 16 108060 108044 91.2171 60.0625 0.015112 0.0109582 2024-03-20T09:37:13.805 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:37:13.858 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:13 smithi026 ceph-mon[68309]: pgmap v2425: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 862 op/s 2024-03-20T09:37:13.858 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:13 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-03-20T09:37:13.858 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:13 smithi026 ceph-mon[68309]: osdmap e1008: 12 total, 12 up, 12 in 2024-03-20T09:37:13.865 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:13.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:13 smithi115 ceph-mon[72476]: pgmap v2425: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 862 op/s 2024-03-20T09:37:13.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:13 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-03-20T09:37:13.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:13 smithi115 ceph-mon[72476]: osdmap e1008: 12 total, 12 up, 12 in 2024-03-20T09:37:13.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:13 smithi122 ceph-mon[73495]: pgmap v2425: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 862 op/s 2024-03-20T09:37:13.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:13 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-03-20T09:37:13.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:13 smithi122 ceph-mon[73495]: osdmap e1008: 12 total, 12 up, 12 in 2024-03-20T09:37:14.079 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:37:14.086 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:14.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:37:14.188 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:37:14.228 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:14.783 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 75 16 108982 108966 90.7693 57.625 0.0239344 0.0110124 2024-03-20T09:37:15.099 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:37:15.099 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:37:15.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:15.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:37:15.139 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 3 1 2024-03-20T09:37:15.209 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:37:15.319 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:15.363 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:37:15.374 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:15.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:37:15.476 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:37:15.518 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:15.542 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:37:15.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:15.550 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:37:15.621 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:37:15.622 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:15 smithi026 ceph-mon[68309]: pgmap v2427: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.5 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 827 op/s 2024-03-20T09:37:15.651 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:37:15.693 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:15.735 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:37:15.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:15.743 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:37:15.783 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 76 16 109813 109797 90.2584 51.9375 0.0143235 0.0110734 2024-03-20T09:37:15.844 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:37:15.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:15 smithi115 ceph-mon[72476]: pgmap v2427: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.5 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 827 op/s 2024-03-20T09:37:15.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:15 smithi122 ceph-mon[73495]: pgmap v2427: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.5 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 827 op/s 2024-03-20T09:37:15.946 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:16.785 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 77 16 110714 110698 89.815 56.3125 0.0113522 0.0111296 2024-03-20T09:37:16.824 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:37:16.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:16.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:37:16.929 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:37:16.958 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:16.979 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:37:16.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:16.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:37:17.084 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:37:17.111 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:17.132 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:37:17.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:17.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:37:17.235 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:37:17.303 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:17.632 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:17 smithi026 ceph-mon[68309]: pgmap v2428: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 923 op/s 2024-03-20T09:37:17.632 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:17 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2872240092' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:37:17.632 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:17 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:37:17.642 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:37:17.787 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 78 16 111649 111633 89.4115 58.4375 0.0134921 0.0111799 2024-03-20T09:37:17.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:17 smithi115 ceph-mon[72476]: pgmap v2428: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 923 op/s 2024-03-20T09:37:17.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:17 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2872240092' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:37:17.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:17 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:37:17.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:17 smithi122 ceph-mon[73495]: pgmap v2428: 47 pgs: 1 active+clean+scrubbing+deep, 46 active+clean; 6.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 923 op/s 2024-03-20T09:37:17.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:17 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2872240092' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:37:17.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:17 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:37:17.991 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:37:17.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:17.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:37:18.095 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:37:18.132 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:37:18.134 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:37:18.174 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:18.406 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:37:18.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:18.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:37:18.516 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:37:18.561 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:18.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:18 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:37:18.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:18 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:37:18.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:18 smithi026 ceph-mon[68309]: osdmap e1009: 12 total, 12 up, 12 in 2024-03-20T09:37:18.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:18 smithi026 ceph-mon[68309]: osdmap e1010: 12 total, 12 up, 12 in 2024-03-20T09:37:18.789 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 79 16 112699 112683 89.1082 65.625 0.00894544 0.0112176 2024-03-20T09:37:18.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:18 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:37:18.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:18 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:37:18.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:18 smithi115 ceph-mon[72476]: osdmap e1009: 12 total, 12 up, 12 in 2024-03-20T09:37:18.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:18 smithi115 ceph-mon[72476]: osdmap e1010: 12 total, 12 up, 12 in 2024-03-20T09:37:18.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:18 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:37:18.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:18 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:37:18.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:18 smithi122 ceph-mon[73495]: osdmap e1009: 12 total, 12 up, 12 in 2024-03-20T09:37:18.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:18 smithi122 ceph-mon[73495]: osdmap e1010: 12 total, 12 up, 12 in 2024-03-20T09:37:19.630 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:37:19.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:19.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:37:19.741 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:37:19.785 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:19.789 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:2024-03-20T09:37:19.800385+0000 min lat: 0.00110904 max lat: 0.263321 avg lat: 0.0112554 2024-03-20T09:37:19.789 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T09:37:19.789 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 80 16 113746 113730 88.8123 65.4375 0.0151978 0.0112554 2024-03-20T09:37:19.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:19 smithi122 ceph-mon[73495]: pgmap v2431: 47 pgs: 47 active+clean; 6.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 808 op/s 2024-03-20T09:37:19.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:19 smithi026 ceph-mon[68309]: pgmap v2431: 47 pgs: 47 active+clean; 6.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 808 op/s 2024-03-20T09:37:20.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:19 smithi115 ceph-mon[72476]: pgmap v2431: 47 pgs: 47 active+clean; 6.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 808 op/s 2024-03-20T09:37:20.345 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:37:20.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:20.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:37:20.452 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:37:20.503 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:20.564 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:37:20.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:20.569 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:37:20.670 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:37:20.710 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:20.789 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 81 16 114778 114762 88.5122 64.5 0.0258505 0.0112936 2024-03-20T09:37:20.852 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:37:20.852 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:37:20.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:20.852 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26917.0:114832 8.9 e1011/974 8:914463a9:::benchmark_data_smithi122_197775_object114831:head v 1011'7197, mlcod=1011'7195)", 2024-03-20T09:37:20.852 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:37:20.852347+0000", 2024-03-20T09:37:20.852 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.003994254, 2024-03-20T09:37:20.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004018204, 2024-03-20T09:37:20.852 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:37:20.853 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:37:20.853 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:37:20.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:20.853 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:37:20.853 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:20.852347+0000", 2024-03-20T09:37:20.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:20.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:20.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:20.853 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:37:20.853 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:20.852347+0000", 2024-03-20T09:37:20.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:20.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:20.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:20.853 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:37:20.854 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:20.852348+0000", 2024-03-20T09:37:20.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4200000000000004e-07 2024-03-20T09:37:20.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:20.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:20.854 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:37:20.854 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:20.852385+0000", 2024-03-20T09:37:20.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6740999999999999e-05 2024-03-20T09:37:20.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:20.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:20.854 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:37:20.854 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:20.852402+0000", 2024-03-20T09:37:20.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7734999999999999e-05 2024-03-20T09:37:20.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:20.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:20.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:37:20.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:20.852409+0000", 2024-03-20T09:37:20.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5930000000000002e-06 2024-03-20T09:37:20.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:20.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:20.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:37:20.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:20.852498+0000", 2024-03-20T09:37:20.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8969999999999997e-05 2024-03-20T09:37:20.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:20.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:20.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:37:20.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:20.852532+0000", 2024-03-20T09:37:20.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4548999999999997e-05 2024-03-20T09:37:20.855 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:20.855 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:37:20.856 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:20.856 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:20.856 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:37:20.856 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 1 2024-03-20T09:37:20.856 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:37:20.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:20 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:37:20.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:20 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:37:20.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:20 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:37:20.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:20 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:37:20.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:20 smithi026 ceph-mon[68309]: osdmap e1011: 12 total, 12 up, 12 in 2024-03-20T09:37:20.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:20 smithi026 ceph-mon[68309]: pgmap v2433: 47 pgs: 47 active+clean; 6.8 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 888 op/s 2024-03-20T09:37:21.007 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:37:21.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:20 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:37:21.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:20 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:37:21.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:20 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:37:21.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:20 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:37:21.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:20 smithi115 ceph-mon[72476]: osdmap e1011: 12 total, 12 up, 12 in 2024-03-20T09:37:21.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:20 smithi115 ceph-mon[72476]: pgmap v2433: 47 pgs: 47 active+clean; 6.8 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 888 op/s 2024-03-20T09:37:21.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:20 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:37:21.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:20 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:37:21.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:20 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:37:21.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:20 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:37:21.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:20 smithi122 ceph-mon[73495]: osdmap e1011: 12 total, 12 up, 12 in 2024-03-20T09:37:21.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:20 smithi122 ceph-mon[73495]: pgmap v2433: 47 pgs: 47 active+clean; 6.8 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 888 op/s 2024-03-20T09:37:21.789 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 82 16 115840 115824 88.2423 66.375 0.0103938 0.0113289 2024-03-20T09:37:21.979 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:37:21.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:21.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:37:22.095 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:37:22.145 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:22.171 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:37:22.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:22.177 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:37:22.278 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:37:22.313 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:22.334 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:37:22.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:22.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:37:22.445 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:37:22.496 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:22.526 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:37:22.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:22.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:37:22.634 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:37:22.683 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:22.789 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 83 16 116872 116856 87.9562 64.5 0.0155352 0.0113649 2024-03-20T09:37:23.455 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:37:23.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:23.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:37:23.571 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:37:23.608 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:23.789 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 84 16 117889 117873 87.6659 63.5625 0.0149145 0.0114032 2024-03-20T09:37:23.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:23 smithi115 ceph-mon[72476]: pgmap v2434: 47 pgs: 47 active+clean; 7.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 952 op/s 2024-03-20T09:37:23.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:23 smithi115 ceph-mon[72476]: osdmap e1012: 12 total, 12 up, 12 in 2024-03-20T09:37:23.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:23 smithi122 ceph-mon[73495]: pgmap v2434: 47 pgs: 47 active+clean; 7.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 952 op/s 2024-03-20T09:37:23.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:23 smithi122 ceph-mon[73495]: osdmap e1012: 12 total, 12 up, 12 in 2024-03-20T09:37:23.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:23 smithi026 ceph-mon[68309]: pgmap v2434: 47 pgs: 47 active+clean; 7.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 952 op/s 2024-03-20T09:37:23.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:23 smithi026 ceph-mon[68309]: osdmap e1012: 12 total, 12 up, 12 in 2024-03-20T09:37:24.000 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:37:24.037 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:37:24.082 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:37:24.082 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:37:24.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:24.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:37:24.136 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:37:24.181 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:37:24.208 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:37:24.234 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:37:24.254 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:24.278 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:37:24.285 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:37:24.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:24.292 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:37:24.327 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:37:24.364 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:37:24.392 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:37:24.393 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:37:24.423 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:24.434 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:37:24.443 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:37:24.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:24.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:37:24.467 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:37:24.473 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:37:24.474 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:37:24.474 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:37:24.474 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:37:24.474 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:37:24.474 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:37:24.474 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:37:24.474 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:37:24.474 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:37:24.474 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:37:24.474 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:37:24.474 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:37:24.547 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:37:24.592 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:24.603 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-20T09:37:24.620 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:37:24.624 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:24.624 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:37:24.639 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.3 to 1 (10000) 2024-03-20T09:37:24.725 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:37:24.753 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:24 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3438155364' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:37:24.753 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:24 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3934545073' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-20T09:37:24.755 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:24.779 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:37:24.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:24.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:37:24.789 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 85 16 118856 118840 87.3456 60.4375 0.00814752 0.0114447 2024-03-20T09:37:24.888 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:37:24.917 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:24.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:24 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3438155364' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:37:24.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:24 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3934545073' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-20T09:37:24.939 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:37:24.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:24.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:37:24.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:24 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3438155364' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:37:24.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:24 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3934545073' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-20T09:37:25.044 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:37:25.073 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:25.097 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:37:25.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:25.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:37:25.201 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:37:25.242 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:25.566 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:37:25.577 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 4 1 2024-03-20T09:37:25.789 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 86 16 119869 119853 87.0662 63.3125 0.0172483 0.0114811 2024-03-20T09:37:25.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:25 smithi115 ceph-mon[72476]: pgmap v2436: 47 pgs: 47 active+clean; 7.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 959 op/s 2024-03-20T09:37:25.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:25 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3934545073' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-20T09:37:25.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:25 smithi115 ceph-mon[72476]: osdmap e1013: 12 total, 12 up, 12 in 2024-03-20T09:37:25.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:25 smithi122 ceph-mon[73495]: pgmap v2436: 47 pgs: 47 active+clean; 7.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 959 op/s 2024-03-20T09:37:25.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:25 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3934545073' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-20T09:37:25.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:25 smithi122 ceph-mon[73495]: osdmap e1013: 12 total, 12 up, 12 in 2024-03-20T09:37:25.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:25 smithi026 ceph-mon[68309]: pgmap v2436: 47 pgs: 47 active+clean; 7.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 959 op/s 2024-03-20T09:37:25.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:25 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3934545073' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-20T09:37:25.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:25 smithi026 ceph-mon[68309]: osdmap e1013: 12 total, 12 up, 12 in 2024-03-20T09:37:26.012 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:37:26.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:26.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:37:26.080 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:37:26.125 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:37:26.151 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:26.170 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:37:26.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:26.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:37:26.274 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:37:26.313 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:26.789 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 87 16 120560 120544 86.5619 43.1875 0.00950004 0.0115479 2024-03-20T09:37:27.789 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 88 16 121577 121561 86.3006 63.5625 0.0047814 0.0115822 2024-03-20T09:37:27.933 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:27 smithi026 ceph-mon[68309]: pgmap v2438: 47 pgs: 47 active+clean; 7.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-03-20T09:37:27.935 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:27 smithi122 ceph-mon[73495]: pgmap v2438: 47 pgs: 47 active+clean; 7.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-03-20T09:37:27.973 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:37:27.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:27.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:37:28.046 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:37:28.078 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:37:28.134 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:28.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:27 smithi115 ceph-mon[72476]: pgmap v2438: 47 pgs: 47 active+clean; 7.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-03-20T09:37:28.166 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:37:28.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:28.171 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:37:28.272 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:37:28.313 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:28.402 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:37:28.404 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:37:28.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:28.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:37:28.514 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:37:28.584 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:28.669 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:37:28.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:28.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:37:28.777 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:37:28.789 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 89 16 122374 122358 85.8907 49.8125 0.0188917 0.0116384 2024-03-20T09:37:28.808 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:28.831 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:37:28.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:28.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:37:28.846 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:28 smithi026 ceph-mon[68309]: osdmap e1014: 12 total, 12 up, 12 in 2024-03-20T09:37:28.846 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:28 smithi026 ceph-mon[68309]: pgmap v2440: 47 pgs: 47 active+clean; 7.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 921 op/s 2024-03-20T09:37:28.937 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:37:28.966 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:28.986 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:37:28.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:28.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:37:29.090 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:37:29.144 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:28 smithi122 ceph-mon[73495]: osdmap e1014: 12 total, 12 up, 12 in 2024-03-20T09:37:29.144 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:28 smithi122 ceph-mon[73495]: pgmap v2440: 47 pgs: 47 active+clean; 7.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 921 op/s 2024-03-20T09:37:29.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:28 smithi115 ceph-mon[72476]: osdmap e1014: 12 total, 12 up, 12 in 2024-03-20T09:37:29.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:28 smithi115 ceph-mon[72476]: pgmap v2440: 47 pgs: 47 active+clean; 7.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 921 op/s 2024-03-20T09:37:29.146 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:29.174 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:37:29.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:29.177 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:37:29.276 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:37:29.276 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:37:29.278 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:37:29.320 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:29.347 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:37:29.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:29.353 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:37:29.454 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:37:29.515 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:29.547 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:37:29.553 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:29.554 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:37:29.655 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:37:29.683 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:29.705 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:37:29.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:29.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:37:29.790 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Total time run: 90.0089 2024-03-20T09:37:29.791 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Total writes made: 123301 2024-03-20T09:37:29.791 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Write size: 65536 2024-03-20T09:37:29.791 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Object size: 65536 2024-03-20T09:37:29.791 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Bandwidth (MB/sec): 85.6172 2024-03-20T09:37:29.791 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Stddev Bandwidth: 33.1168 2024-03-20T09:37:29.791 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Max bandwidth (MB/sec): 191.25 2024-03-20T09:37:29.791 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Min bandwidth (MB/sec): 42.1875 2024-03-20T09:37:29.791 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Average IOPS: 1369 2024-03-20T09:37:29.791 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Stddev IOPS: 529.87 2024-03-20T09:37:29.792 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Max IOPS: 3060 2024-03-20T09:37:29.792 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Min IOPS: 675 2024-03-20T09:37:29.792 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Average Latency(s): 0.0116763 2024-03-20T09:37:29.792 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Stddev Latency(s): 0.0143439 2024-03-20T09:37:29.792 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Max latency(s): 0.263321 2024-03-20T09:37:29.792 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Min latency(s): 0.00110904 2024-03-20T09:37:29.797 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Cleaning up (deleting benchmark objects) 2024-03-20T09:37:29.811 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:37:29.873 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:29.913 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:37:29.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:29.917 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:37:30.018 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:37:30.046 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:30.065 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:37:30.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:30.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:37:30.170 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:37:30.198 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:30.221 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:37:30.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:30.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:37:30.326 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:37:30.356 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:30.378 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:37:30.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:30.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:37:30.482 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:37:30.513 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:30.533 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:37:30.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:30.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:37:30.638 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:37:30.667 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:30.689 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:37:30.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:30.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:37:30.796 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:37:30.828 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:30.855 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:37:30.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:30.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:37:30.959 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:37:31.018 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:31.312 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:37:31.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:31.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:37:31.421 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:37:31.483 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:31.508 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:37:31.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:31.514 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:37:31.529 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:37:31.615 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:37:31.683 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:31.747 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:37:31.758 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:31.759 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:31 smithi122 ceph-mon[73495]: pgmap v2441: 47 pgs: 47 active+clean; 7.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 941 op/s 2024-03-20T09:37:31.759 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:37:31.861 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:37:31.888 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:31 smithi115 ceph-mon[72476]: pgmap v2441: 47 pgs: 47 active+clean; 7.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 941 op/s 2024-03-20T09:37:31.890 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:31.906 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:31 smithi026 ceph-mon[68309]: pgmap v2441: 47 pgs: 47 active+clean; 7.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 58 MiB/s wr, 941 op/s 2024-03-20T09:37:31.914 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:37:31.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:31.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:37:32.018 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:37:32.064 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:32.106 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:37:32.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:32.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:37:32.214 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:37:32.254 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:32.290 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:37:32.298 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:32.298 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:37:32.399 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:37:32.443 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:33.753 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:37:33.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:33.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:37:33.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:33 smithi026 ceph-mon[68309]: pgmap v2442: 47 pgs: 47 active+clean; 7.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 57 MiB/s wr, 987 op/s 2024-03-20T09:37:33.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:33 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2483016458' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:37:33.758 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:33 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3903016594' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-20T09:37:33.856 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.4 to 1 (10000) 2024-03-20T09:37:33.858 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:37:33.868 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:37:33.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:33 smithi115 ceph-mon[72476]: pgmap v2442: 47 pgs: 47 active+clean; 7.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 57 MiB/s wr, 987 op/s 2024-03-20T09:37:33.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:33 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2483016458' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:37:33.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:33 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3903016594' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-20T09:37:33.935 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:33 smithi122 ceph-mon[73495]: pgmap v2442: 47 pgs: 47 active+clean; 7.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 57 MiB/s wr, 987 op/s 2024-03-20T09:37:33.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:33 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2483016458' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:37:33.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:33 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3903016594' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-20T09:37:33.938 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:34.011 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:37:34.022 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:34.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:37:34.124 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:37:34.165 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:34.392 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:37:34.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:34.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:37:34.500 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:37:34.537 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:34.557 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:37:34.560 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:34.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:37:34.661 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:37:34.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:34 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:37:34.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:34 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2483016458' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:37:34.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:34 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3903016594' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-20T09:37:34.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:34 smithi026 ceph-mon[68309]: osdmap e1015: 12 total, 12 up, 12 in 2024-03-20T09:37:34.726 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:34.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:34 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:37:34.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:34 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2483016458' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:37:34.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:34 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3903016594' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-20T09:37:34.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:34 smithi115 ceph-mon[72476]: osdmap e1015: 12 total, 12 up, 12 in 2024-03-20T09:37:34.935 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:34 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:37:34.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:34 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2483016458' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:37:34.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:34 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3903016594' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-20T09:37:34.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:34 smithi122 ceph-mon[73495]: osdmap e1015: 12 total, 12 up, 12 in 2024-03-20T09:37:34.948 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:37:34.955 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:34.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:37:34.988 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 9 1 2024-03-20T09:37:35.057 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:37:35.164 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:35.375 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:37:35.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:35.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:37:35.423 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-20T09:37:35.483 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:37:35.566 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:35.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:35 smithi026 ceph-mon[68309]: pgmap v2444: 47 pgs: 47 active+clean; 7.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 40 MiB/s wr, 857 op/s 2024-03-20T09:37:35.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:35 smithi115 ceph-mon[72476]: pgmap v2444: 47 pgs: 47 active+clean; 7.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 40 MiB/s wr, 857 op/s 2024-03-20T09:37:35.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:35 smithi122 ceph-mon[73495]: pgmap v2444: 47 pgs: 47 active+clean; 7.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 40 MiB/s wr, 857 op/s 2024-03-20T09:37:35.944 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:37:35.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:35.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:37:36.051 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:37:36.105 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:36.133 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:37:36.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:36.142 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:37:36.244 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:37:36.337 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:37:36.364 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:36.369 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:37:36.397 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:37:36.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:36.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:37:36.511 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:37:36.565 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:36.635 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:37:36.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:36.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:37:36.743 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:37:36.776 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:36.800 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:37:36.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:36.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:37:36.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:36 smithi115 ceph-mon[72476]: 1.15 scrub starts 2024-03-20T09:37:36.905 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:37:36.935 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:36 smithi122 ceph-mon[73495]: 1.15 scrub starts 2024-03-20T09:37:36.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:36 smithi026 ceph-mon[68309]: 1.15 scrub starts 2024-03-20T09:37:36.965 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:37.003 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:37:37.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:37.012 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:37:37.113 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:37:37.145 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:37:37.170 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:37.200 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:37:37.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:37.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:37:37.219 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:37:37.245 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:37:37.280 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:37:37.310 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:37:37.315 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:37:37.350 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:37.361 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:37:37.376 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:37:37.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:37.384 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:37:37.388 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:37:37.419 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:37:37.476 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:37:37.484 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:37:37.514 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:37.519 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:37:37.536 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:37:37.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:37.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:37:37.553 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:37:37.589 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:37:37.591 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:37:37.592 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:37:37.592 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:37:37.592 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:37:37.592 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:37:37.592 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:37:37.592 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:37:37.593 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:37:37.593 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:37:37.593 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:37:37.593 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:37:37.593 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:37:37.640 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:37:37.685 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:37 smithi026 ceph-mon[68309]: 8.b scrub starts 2024-03-20T09:37:37.686 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:37 smithi026 ceph-mon[68309]: 1.15 scrub ok 2024-03-20T09:37:37.686 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:37 smithi026 ceph-mon[68309]: pgmap v2445: 47 pgs: 47 active+clean; 7.3 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 117 B/s rd, 30 MiB/s wr, 974 op/s 2024-03-20T09:37:37.686 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:37 smithi026 ceph-mon[68309]: 8.8 scrub starts 2024-03-20T09:37:37.686 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:37 smithi026 ceph-mon[68309]: 8.1 scrub starts 2024-03-20T09:37:37.686 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:37 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1803595600' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:37:37.686 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:37.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:37 smithi115 ceph-mon[72476]: 8.b scrub starts 2024-03-20T09:37:37.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:37 smithi115 ceph-mon[72476]: 1.15 scrub ok 2024-03-20T09:37:37.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:37 smithi115 ceph-mon[72476]: pgmap v2445: 47 pgs: 47 active+clean; 7.3 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 117 B/s rd, 30 MiB/s wr, 974 op/s 2024-03-20T09:37:37.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:37 smithi115 ceph-mon[72476]: 8.8 scrub starts 2024-03-20T09:37:37.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:37 smithi115 ceph-mon[72476]: 8.1 scrub starts 2024-03-20T09:37:37.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:37 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1803595600' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:37:37.924 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:37:37.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:37 smithi122 ceph-mon[73495]: 8.b scrub starts 2024-03-20T09:37:37.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:37 smithi122 ceph-mon[73495]: 1.15 scrub ok 2024-03-20T09:37:37.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:37 smithi122 ceph-mon[73495]: pgmap v2445: 47 pgs: 47 active+clean; 7.3 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 117 B/s rd, 30 MiB/s wr, 974 op/s 2024-03-20T09:37:37.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:37 smithi122 ceph-mon[73495]: 8.8 scrub starts 2024-03-20T09:37:37.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:37 smithi122 ceph-mon[73495]: 8.1 scrub starts 2024-03-20T09:37:37.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:37 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1803595600' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:37:38.341 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:37:38.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:38.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:37:38.446 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:37:38.478 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:38.502 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:37:38.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:38.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:37:38.608 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:37:38.651 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:38.681 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:37:38.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:38.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:37:38.790 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:37:38.826 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:38.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:38 smithi115 ceph-mon[72476]: 1.1e scrub starts 2024-03-20T09:37:38.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:38 smithi115 ceph-mon[72476]: 1.1e scrub ok 2024-03-20T09:37:38.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:38 smithi115 ceph-mon[72476]: 8.2 scrub starts 2024-03-20T09:37:38.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:38 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:37:38.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:38 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:37:38.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:38 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:37:38.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:38 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:37:38.896 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:38 smithi115 ceph-mon[72476]: 8.c scrub starts 2024-03-20T09:37:38.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:38 smithi122 ceph-mon[73495]: 1.1e scrub starts 2024-03-20T09:37:38.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:38 smithi122 ceph-mon[73495]: 1.1e scrub ok 2024-03-20T09:37:38.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:38 smithi122 ceph-mon[73495]: 8.2 scrub starts 2024-03-20T09:37:38.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:38 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:37:38.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:38 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:37:38.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:38 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:37:38.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:38 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:37:38.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:38 smithi122 ceph-mon[73495]: 8.c scrub starts 2024-03-20T09:37:38.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:38 smithi026 ceph-mon[68309]: 1.1e scrub starts 2024-03-20T09:37:38.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:38 smithi026 ceph-mon[68309]: 1.1e scrub ok 2024-03-20T09:37:38.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:38 smithi026 ceph-mon[68309]: 8.2 scrub starts 2024-03-20T09:37:38.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:38 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:37:38.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:38 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:37:38.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:38 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:37:38.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:38 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:37:38.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:38 smithi026 ceph-mon[68309]: 8.c scrub starts 2024-03-20T09:37:39.035 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:37:39.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:39.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:37:39.144 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:37:39.180 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:39.214 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:37:39.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:39.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:37:39.318 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:37:39.384 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:39.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:39 smithi122 ceph-mon[73495]: 8.b scrub ok 2024-03-20T09:37:39.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:39 smithi122 ceph-mon[73495]: 8.8 scrub ok 2024-03-20T09:37:39.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:39 smithi122 ceph-mon[73495]: pgmap v2446: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 7.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 23 MiB/s wr, 952 op/s 2024-03-20T09:37:39.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:39 smithi122 ceph-mon[73495]: 1.10 scrub starts 2024-03-20T09:37:39.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:39 smithi122 ceph-mon[73495]: 1.10 scrub ok 2024-03-20T09:37:39.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:39 smithi122 ceph-mon[73495]: osdmap e1016: 12 total, 12 up, 12 in 2024-03-20T09:37:39.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:39 smithi122 ceph-mon[73495]: 1.a scrub starts 2024-03-20T09:37:39.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:39 smithi122 ceph-mon[73495]: 1.a scrub ok 2024-03-20T09:37:39.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:39 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:37:39.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:39 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:37:39.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:39 smithi122 ceph-mon[73495]: 8.1 scrub ok 2024-03-20T09:37:39.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:39 smithi122 ceph-mon[73495]: 8.2 scrub ok 2024-03-20T09:37:39.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:39 smithi026 ceph-mon[68309]: 8.b scrub ok 2024-03-20T09:37:39.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:39 smithi026 ceph-mon[68309]: 8.8 scrub ok 2024-03-20T09:37:39.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:39 smithi026 ceph-mon[68309]: pgmap v2446: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 7.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 23 MiB/s wr, 952 op/s 2024-03-20T09:37:39.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:39 smithi026 ceph-mon[68309]: 1.10 scrub starts 2024-03-20T09:37:39.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:39 smithi026 ceph-mon[68309]: 1.10 scrub ok 2024-03-20T09:37:39.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:39 smithi026 ceph-mon[68309]: osdmap e1016: 12 total, 12 up, 12 in 2024-03-20T09:37:39.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:39 smithi026 ceph-mon[68309]: 1.a scrub starts 2024-03-20T09:37:39.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:39 smithi026 ceph-mon[68309]: 1.a scrub ok 2024-03-20T09:37:39.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:39 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:37:39.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:39 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:37:39.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:39 smithi026 ceph-mon[68309]: 8.1 scrub ok 2024-03-20T09:37:39.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:39 smithi026 ceph-mon[68309]: 8.2 scrub ok 2024-03-20T09:37:40.061 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:37:40.061 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-03-20T09:37:40.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-03-20T09:37:40.061 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:37:40.061 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.061 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:213755 7.15 7:a872df88:::benchmark_data_smithi122_187077_object87144:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e936)", 2024-03-20T09:37:40.061 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:36.132651+0000", 2024-03-20T09:37:40.061 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 183.936215256, 2024-03-20T09:37:40.061 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2435835669999999, 2024-03-20T09:37:40.062 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:37:40.062 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:37:40.062 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:37:40.062 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:37:40.062 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:37:40.062 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 213755 2024-03-20T09:37:40.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.062 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:37:40.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.062 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:37:40.062 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132651+0000", 2024-03-20T09:37:40.062 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.062 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.062 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:37:40.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132651+0000", 2024-03-20T09:37:40.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:37:40.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132652+0000", 2024-03-20T09:37:40.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3120000000000001e-06 2024-03-20T09:37:40.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.063 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.063 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:37:40.063 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132659+0000", 2024-03-20T09:37:40.063 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3799999999999999e-06 2024-03-20T09:37:40.063 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:37:40.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132675+0000", 2024-03-20T09:37:40.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6294e-05 2024-03-20T09:37:40.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:37:40.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132683+0000", 2024-03-20T09:37:40.064 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2800000000000003e-06 2024-03-20T09:37:40.064 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.064 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.064 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:37:40.064 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132746+0000", 2024-03-20T09:37:40.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2664999999999997e-05 2024-03-20T09:37:40.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.065 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.065 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op must wait for map", 2024-03-20T09:37:40.065 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132753+0000", 2024-03-20T09:37:40.065 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1720000000000001e-06 2024-03-20T09:37:40.065 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:37:40.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.136252+0000", 2024-03-20T09:37:40.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003499173 2024-03-20T09:37:40.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:37:40.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.136261+0000", 2024-03-20T09:37:40.066 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8799999999999997e-06 2024-03-20T09:37:40.066 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.066 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.066 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:37:40.066 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.376116+0000", 2024-03-20T09:37:40.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2398546189999999 2024-03-20T09:37:40.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.067 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:37:40.067 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.376235+0000", 2024-03-20T09:37:40.067 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000118792 2024-03-20T09:37:40.067 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.067 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:37:40.067 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.067 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.067 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.067 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:213794 7.16 7:69d71a3c:::benchmark_data_smithi122_187077_object87183:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e936)", 2024-03-20T09:37:40.067 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:36.132714+0000", 2024-03-20T09:37:40.067 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 183.93615233599999, 2024-03-20T09:37:40.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.226967184, 2024-03-20T09:37:40.068 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:37:40.068 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:37:40.068 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:37:40.068 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:37:40.068 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:37:40.068 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 213794 2024-03-20T09:37:40.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.068 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:37:40.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.068 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:37:40.068 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132714+0000", 2024-03-20T09:37:40.068 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.068 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.068 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:37:40.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132714+0000", 2024-03-20T09:37:40.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:37:40.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132715+0000", 2024-03-20T09:37:40.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6499999999999999e-07 2024-03-20T09:37:40.069 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.069 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.069 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:37:40.069 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132718+0000", 2024-03-20T09:37:40.069 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3139999999999999e-06 2024-03-20T09:37:40.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:37:40.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132726+0000", 2024-03-20T09:37:40.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2109999999999998e-06 2024-03-20T09:37:40.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:37:40.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132733+0000", 2024-03-20T09:37:40.070 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9120000000000001e-06 2024-03-20T09:37:40.070 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.070 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.070 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:37:40.070 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132931+0000", 2024-03-20T09:37:40.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019818599999999999 2024-03-20T09:37:40.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op must wait for map", 2024-03-20T09:37:40.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132938+0000", 2024-03-20T09:37:40.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8630000000000002e-06 2024-03-20T09:37:40.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:37:40.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.134074+0000", 2024-03-20T09:37:40.071 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001136319 2024-03-20T09:37:40.071 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.071 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.071 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:37:40.071 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.134083+0000", 2024-03-20T09:37:40.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9740000000000008e-06 2024-03-20T09:37:40.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:37:40.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.332852+0000", 2024-03-20T09:37:40.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1987690790000001 2024-03-20T09:37:40.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:37:40.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.332997+0000", 2024-03-20T09:37:40.072 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000144191 2024-03-20T09:37:40.072 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.072 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.072 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,7,8,11", 2024-03-20T09:37:40.072 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.333138+0000", 2024-03-20T09:37:40.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014141499999999999 2024-03-20T09:37:40.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:37:40.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.337922+0000", 2024-03-20T09:37:40.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0047840570000000004 2024-03-20T09:37:40.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:37:40.073 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.338331+0000", 2024-03-20T09:37:40.073 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00040852899999999998 2024-03-20T09:37:40.073 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.073 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.073 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:37:40.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.347271+0000", 2024-03-20T09:37:40.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0089403139999999996 2024-03-20T09:37:40.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:37:40.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.347592+0000", 2024-03-20T09:37:40.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00032139799999999997 2024-03-20T09:37:40.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.074 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.074 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:37:40.074 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.359473+0000", 2024-03-20T09:37:40.074 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011880658000000001 2024-03-20T09:37:40.074 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:37:40.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.359539+0000", 2024-03-20T09:37:40.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5635000000000004e-05 2024-03-20T09:37:40.075 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.075 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.075 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:37:40.075 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.359681+0000", 2024-03-20T09:37:40.075 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000142564 2024-03-20T09:37:40.075 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.075 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:37:40.075 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.076 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.076 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.076 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:213795 7.14 7:29024fd8:::benchmark_data_smithi122_187077_object87184:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e936)", 2024-03-20T09:37:40.076 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:36.132749+0000", 2024-03-20T09:37:40.076 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 183.93611756300001, 2024-03-20T09:37:40.076 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.217728784, 2024-03-20T09:37:40.076 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:37:40.076 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:37:40.076 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:37:40.076 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:37:40.076 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:37:40.077 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 213795 2024-03-20T09:37:40.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.077 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:37:40.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:37:40.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132749+0000", 2024-03-20T09:37:40.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.077 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:37:40.077 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132749+0000", 2024-03-20T09:37:40.077 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.077 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.077 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:37:40.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132749+0000", 2024-03-20T09:37:40.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8800000000000003e-07 2024-03-20T09:37:40.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:37:40.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132752+0000", 2024-03-20T09:37:40.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5009999999999999e-06 2024-03-20T09:37:40.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.078 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.078 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:37:40.078 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132758+0000", 2024-03-20T09:37:40.078 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4999999999999996e-06 2024-03-20T09:37:40.078 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:37:40.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132765+0000", 2024-03-20T09:37:40.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4999999999999996e-06 2024-03-20T09:37:40.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:37:40.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.133009+0000", 2024-03-20T09:37:40.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00024404 2024-03-20T09:37:40.079 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.079 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.079 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op must wait for map", 2024-03-20T09:37:40.079 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.133016+0000", 2024-03-20T09:37:40.079 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5270000000000001e-06 2024-03-20T09:37:40.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:37:40.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.135692+0000", 2024-03-20T09:37:40.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026753390000000001 2024-03-20T09:37:40.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:37:40.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.135701+0000", 2024-03-20T09:37:40.080 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8519999999999993e-06 2024-03-20T09:37:40.080 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.080 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.080 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:37:40.080 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.342134+0000", 2024-03-20T09:37:40.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.206433724 2024-03-20T09:37:40.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:37:40.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.342264+0000", 2024-03-20T09:37:40.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012935999999999999 2024-03-20T09:37:40.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 1,4,5,11", 2024-03-20T09:37:40.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.342638+0000", 2024-03-20T09:37:40.081 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00037430800000000001 2024-03-20T09:37:40.081 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.081 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.081 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:37:40.081 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.349288+0000", 2024-03-20T09:37:40.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0066500750000000001 2024-03-20T09:37:40.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:37:40.082 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.349303+0000", 2024-03-20T09:37:40.082 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5293000000000001e-05 2024-03-20T09:37:40.082 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.082 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.082 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:37:40.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.349339+0000", 2024-03-20T09:37:40.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5503999999999998e-05 2024-03-20T09:37:40.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:37:40.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.350180+0000", 2024-03-20T09:37:40.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084116400000000004 2024-03-20T09:37:40.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.083 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:37:40.083 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.350306+0000", 2024-03-20T09:37:40.083 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012647 2024-03-20T09:37:40.083 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.083 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:37:40.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.350373+0000", 2024-03-20T09:37:40.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6805999999999997e-05 2024-03-20T09:37:40.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.084 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:37:40.084 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.350478+0000", 2024-03-20T09:37:40.084 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010433300000000001 2024-03-20T09:37:40.084 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.084 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:37:40.084 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.084 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.084 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.084 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:213809 7.14 7:2fbe0354:::benchmark_data_smithi122_187077_object87198:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e936)", 2024-03-20T09:37:40.085 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:36.132780+0000", 2024-03-20T09:37:40.085 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 183.936085918, 2024-03-20T09:37:40.085 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2180650239999999, 2024-03-20T09:37:40.085 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:37:40.085 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:37:40.085 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:37:40.085 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:37:40.085 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:37:40.085 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 213809 2024-03-20T09:37:40.085 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.085 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:37:40.085 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.085 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:37:40.085 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132780+0000", 2024-03-20T09:37:40.086 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.086 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.086 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.086 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:37:40.086 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132780+0000", 2024-03-20T09:37:40.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.087 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:37:40.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132781+0000", 2024-03-20T09:37:40.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6199999999999998e-07 2024-03-20T09:37:40.087 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.087 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.087 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:37:40.087 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132784+0000", 2024-03-20T09:37:40.087 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6529999999999998e-06 2024-03-20T09:37:40.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:37:40.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132791+0000", 2024-03-20T09:37:40.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2420000000000003e-06 2024-03-20T09:37:40.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:37:40.088 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132802+0000", 2024-03-20T09:37:40.088 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0694e-05 2024-03-20T09:37:40.088 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.088 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.088 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:37:40.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.133053+0000", 2024-03-20T09:37:40.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025135799999999997 2024-03-20T09:37:40.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting_for_map not empty", 2024-03-20T09:37:40.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.133058+0000", 2024-03-20T09:37:40.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4129999999999997e-06 2024-03-20T09:37:40.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.089 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:37:40.089 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.135784+0000", 2024-03-20T09:37:40.089 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027260119999999999 2024-03-20T09:37:40.089 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.089 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:37:40.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.135792+0000", 2024-03-20T09:37:40.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6720000000000004e-06 2024-03-20T09:37:40.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:37:40.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.345302+0000", 2024-03-20T09:37:40.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.209510152 2024-03-20T09:37:40.090 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.090 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.090 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:37:40.090 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.345448+0000", 2024-03-20T09:37:40.090 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014538800000000001 2024-03-20T09:37:40.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 1,4,5,11", 2024-03-20T09:37:40.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.345850+0000", 2024-03-20T09:37:40.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00040290999999999998 2024-03-20T09:37:40.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:37:40.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.349380+0000", 2024-03-20T09:37:40.091 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035300259999999999 2024-03-20T09:37:40.091 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.091 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.091 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:37:40.091 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.349418+0000", 2024-03-20T09:37:40.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7716999999999998e-05 2024-03-20T09:37:40.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:37:40.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.349513+0000", 2024-03-20T09:37:40.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4426999999999996e-05 2024-03-20T09:37:40.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:37:40.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.350598+0000", 2024-03-20T09:37:40.092 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010858199999999999 2024-03-20T09:37:40.092 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.092 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.092 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:37:40.092 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.350643+0000", 2024-03-20T09:37:40.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4882000000000002e-05 2024-03-20T09:37:40.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:37:40.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.350705+0000", 2024-03-20T09:37:40.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2027999999999996e-05 2024-03-20T09:37:40.093 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.093 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.093 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:37:40.093 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.350845+0000", 2024-03-20T09:37:40.093 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000140168 2024-03-20T09:37:40.093 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.093 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:37:40.093 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.094 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:213816 7.15 7:ab5b4b63:::benchmark_data_smithi122_187077_object87205:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e936)", 2024-03-20T09:37:40.094 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:36.132816+0000", 2024-03-20T09:37:40.094 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 183.93605047299999, 2024-03-20T09:37:40.094 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.255353135, 2024-03-20T09:37:40.094 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:37:40.094 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:37:40.094 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:37:40.094 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:37:40.094 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:37:40.094 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 213816 2024-03-20T09:37:40.094 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.094 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:37:40.094 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:37:40.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132816+0000", 2024-03-20T09:37:40.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:37:40.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132816+0000", 2024-03-20T09:37:40.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.095 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:37:40.095 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132817+0000", 2024-03-20T09:37:40.095 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0800000000000004e-07 2024-03-20T09:37:40.095 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.095 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:37:40.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132820+0000", 2024-03-20T09:37:40.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1070000000000001e-06 2024-03-20T09:37:40.096 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.096 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.096 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:37:40.096 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132828+0000", 2024-03-20T09:37:40.096 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9340000000000006e-06 2024-03-20T09:37:40.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:37:40.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132834+0000", 2024-03-20T09:37:40.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3500000000000002e-06 2024-03-20T09:37:40.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:37:40.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132987+0000", 2024-03-20T09:37:40.097 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015329499999999999 2024-03-20T09:37:40.097 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.097 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.097 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting_for_map not empty", 2024-03-20T09:37:40.097 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132990+0000", 2024-03-20T09:37:40.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0249999999999998e-06 2024-03-20T09:37:40.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:37:40.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.141152+0000", 2024-03-20T09:37:40.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0081620890000000008 2024-03-20T09:37:40.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:37:40.098 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.141162+0000", 2024-03-20T09:37:40.098 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4900000000000006e-06 2024-03-20T09:37:40.098 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.098 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.098 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:37:40.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.376864+0000", 2024-03-20T09:37:40.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2357016459999999 2024-03-20T09:37:40.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.099 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:37:40.099 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.376973+0000", 2024-03-20T09:37:40.099 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000109552 2024-03-20T09:37:40.099 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.099 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 1,7,8,9,11", 2024-03-20T09:37:40.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.377366+0000", 2024-03-20T09:37:40.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00039325299999999998 2024-03-20T09:37:40.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:37:40.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.382051+0000", 2024-03-20T09:37:40.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004684517 2024-03-20T09:37:40.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.100 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:37:40.100 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.382100+0000", 2024-03-20T09:37:40.100 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9307000000000001e-05 2024-03-20T09:37:40.100 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.100 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:37:40.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.385223+0000", 2024-03-20T09:37:40.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031232389999999999 2024-03-20T09:37:40.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:37:40.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.385240+0000", 2024-03-20T09:37:40.101 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6174000000000001e-05 2024-03-20T09:37:40.101 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.101 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.101 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:37:40.101 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.388008+0000", 2024-03-20T09:37:40.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027688019999999999 2024-03-20T09:37:40.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:37:40.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.388056+0000", 2024-03-20T09:37:40.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7735000000000003e-05 2024-03-20T09:37:40.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:37:40.102 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.388094+0000", 2024-03-20T09:37:40.102 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8309000000000002e-05 2024-03-20T09:37:40.102 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.102 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.102 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:37:40.103 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.388169+0000", 2024-03-20T09:37:40.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4603000000000004e-05 2024-03-20T09:37:40.103 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.103 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:37:40.103 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.103 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.103 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.103 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:213826 7.16 7:6e7a6238:::benchmark_data_smithi122_187077_object87215:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e936)", 2024-03-20T09:37:40.103 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:36.132847+0000", 2024-03-20T09:37:40.103 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 183.936019778, 2024-03-20T09:37:40.103 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2273559549999999, 2024-03-20T09:37:40.103 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:37:40.103 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:37:40.103 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:37:40.103 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:37:40.104 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:37:40.104 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 213826 2024-03-20T09:37:40.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.104 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:37:40.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:37:40.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132847+0000", 2024-03-20T09:37:40.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.104 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.104 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:37:40.104 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132847+0000", 2024-03-20T09:37:40.104 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.104 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:37:40.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132847+0000", 2024-03-20T09:37:40.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5999999999999999e-07 2024-03-20T09:37:40.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:37:40.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132849+0000", 2024-03-20T09:37:40.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.424e-06 2024-03-20T09:37:40.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.105 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.105 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:37:40.105 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132857+0000", 2024-03-20T09:37:40.105 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4590000000000001e-06 2024-03-20T09:37:40.105 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:37:40.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132862+0000", 2024-03-20T09:37:40.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4129999999999997e-06 2024-03-20T09:37:40.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:37:40.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132965+0000", 2024-03-20T09:37:40.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000102144 2024-03-20T09:37:40.106 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.106 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.106 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting_for_map not empty", 2024-03-20T09:37:40.106 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132970+0000", 2024-03-20T09:37:40.106 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9039999999999997e-06 2024-03-20T09:37:40.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:37:40.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.134113+0000", 2024-03-20T09:37:40.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001142421 2024-03-20T09:37:40.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:37:40.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.134121+0000", 2024-03-20T09:37:40.107 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9410000000000007e-06 2024-03-20T09:37:40.107 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.107 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.107 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:37:40.107 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.335484+0000", 2024-03-20T09:37:40.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2013633829999999 2024-03-20T09:37:40.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:37:40.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.335622+0000", 2024-03-20T09:37:40.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013765199999999999 2024-03-20T09:37:40.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,7,8,11", 2024-03-20T09:37:40.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.335874+0000", 2024-03-20T09:37:40.108 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025246399999999999 2024-03-20T09:37:40.108 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.108 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.108 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:37:40.108 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.338823+0000", 2024-03-20T09:37:40.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0029486400000000002 2024-03-20T09:37:40.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:37:40.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.344311+0000", 2024-03-20T09:37:40.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054885280000000003 2024-03-20T09:37:40.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:37:40.109 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.347296+0000", 2024-03-20T09:37:40.109 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0029843869999999998 2024-03-20T09:37:40.109 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.109 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.109 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:37:40.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.347633+0000", 2024-03-20T09:37:40.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00033691899999999998 2024-03-20T09:37:40.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:37:40.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.360012+0000", 2024-03-20T09:37:40.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012379028 2024-03-20T09:37:40.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.110 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:37:40.110 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.360077+0000", 2024-03-20T09:37:40.110 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4912999999999993e-05 2024-03-20T09:37:40.110 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.110 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.111 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:37:40.111 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.360203+0000", 2024-03-20T09:37:40.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012587500000000001 2024-03-20T09:37:40.111 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.111 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:37:40.111 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.111 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.111 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.111 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:213827 7.15 7:af646400:::benchmark_data_smithi122_187077_object87216:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e936)", 2024-03-20T09:37:40.111 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:36.132869+0000", 2024-03-20T09:37:40.111 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 183.93599736900001, 2024-03-20T09:37:40.111 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2728044220000001, 2024-03-20T09:37:40.111 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:37:40.111 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:37:40.111 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:37:40.112 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:37:40.112 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:37:40.112 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 213827 2024-03-20T09:37:40.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.112 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:37:40.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:37:40.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132869+0000", 2024-03-20T09:37:40.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.112 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:37:40.112 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132869+0000", 2024-03-20T09:37:40.112 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.112 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.112 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:37:40.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132869+0000", 2024-03-20T09:37:40.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9099999999999999e-07 2024-03-20T09:37:40.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:37:40.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132872+0000", 2024-03-20T09:37:40.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3420000000000002e-06 2024-03-20T09:37:40.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.113 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:37:40.113 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132878+0000", 2024-03-20T09:37:40.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4309999999999999e-06 2024-03-20T09:37:40.113 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:37:40.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132884+0000", 2024-03-20T09:37:40.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5450000000000003e-06 2024-03-20T09:37:40.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:37:40.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.133012+0000", 2024-03-20T09:37:40.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012817500000000001 2024-03-20T09:37:40.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting_for_map not empty", 2024-03-20T09:37:40.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.133017+0000", 2024-03-20T09:37:40.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1590000000000001e-06 2024-03-20T09:37:40.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:37:40.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.141188+0000", 2024-03-20T09:37:40.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0081712980000000004 2024-03-20T09:37:40.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:37:40.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.141195+0000", 2024-03-20T09:37:40.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-03-20T09:37:40.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:37:40.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.379506+0000", 2024-03-20T09:37:40.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.238310228 2024-03-20T09:37:40.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:37:40.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.379646+0000", 2024-03-20T09:37:40.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000139877 2024-03-20T09:37:40.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 1,7,8,9,11", 2024-03-20T09:37:40.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.379998+0000", 2024-03-20T09:37:40.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00035252500000000002 2024-03-20T09:37:40.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:37:40.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.387968+0000", 2024-03-20T09:37:40.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0079701100000000007 2024-03-20T09:37:40.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:37:40.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.388034+0000", 2024-03-20T09:37:40.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5349000000000001e-05 2024-03-20T09:37:40.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:37:40.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.388194+0000", 2024-03-20T09:37:40.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00016032899999999999 2024-03-20T09:37:40.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:37:40.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.388216+0000", 2024-03-20T09:37:40.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1957e-05 2024-03-20T09:37:40.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:37:40.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.388237+0000", 2024-03-20T09:37:40.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0829999999999999e-05 2024-03-20T09:37:40.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:37:40.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.405472+0000", 2024-03-20T09:37:40.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017235527 2024-03-20T09:37:40.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:37:40.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.405532+0000", 2024-03-20T09:37:40.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9740000000000001e-05 2024-03-20T09:37:40.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.119 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:37:40.119 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.405673+0000", 2024-03-20T09:37:40.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000141509 2024-03-20T09:37:40.119 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.119 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:37:40.119 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.119 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:194297 7.5 e936/933 7:a6b384b5:::benchmark_data_smithi122_187077_object91199:head v 936'14280, mlcod=936'14266)", 2024-03-20T09:37:40.119 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:37.177108+0000", 2024-03-20T09:37:40.119 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 182.89175799200001, 2024-03-20T09:37:40.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.22801196200000001, 2024-03-20T09:37:40.120 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:37:40.120 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:37:40.120 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:37:40.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:37:40.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177108+0000", 2024-03-20T09:37:40.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:37:40.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177108+0000", 2024-03-20T09:37:40.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:37:40.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177109+0000", 2024-03-20T09:37:40.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7599999999999998e-07 2024-03-20T09:37:40.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:37:40.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177113+0000", 2024-03-20T09:37:40.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1629999999999998e-06 2024-03-20T09:37:40.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:37:40.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177118+0000", 2024-03-20T09:37:40.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8910000000000003e-06 2024-03-20T09:37:40.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:37:40.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177122+0000", 2024-03-20T09:37:40.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1819999999999999e-06 2024-03-20T09:37:40.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:37:40.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.376413+0000", 2024-03-20T09:37:40.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.199290627 2024-03-20T09:37:40.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:37:40.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.376434+0000", 2024-03-20T09:37:40.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1922000000000001e-05 2024-03-20T09:37:40.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:37:40.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.405072+0000", 2024-03-20T09:37:40.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.028637244999999999 2024-03-20T09:37:40.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:37:40.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.405120+0000", 2024-03-20T09:37:40.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8656000000000003e-05 2024-03-20T09:37:40.123 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.123 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:37:40.123 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.124 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:194298 7.5 e936/933 7:a6b44921:::benchmark_data_smithi122_187077_object98893:head v 936'14281, mlcod=936'14266)", 2024-03-20T09:37:40.124 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:37.177128+0000", 2024-03-20T09:37:40.124 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 182.89173849299999, 2024-03-20T09:37:40.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.24610715, 2024-03-20T09:37:40.124 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:37:40.124 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:37:40.124 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:37:40.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:37:40.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177128+0000", 2024-03-20T09:37:40.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:37:40.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177128+0000", 2024-03-20T09:37:40.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:37:40.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177128+0000", 2024-03-20T09:37:40.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9299999999999999e-07 2024-03-20T09:37:40.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:37:40.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177137+0000", 2024-03-20T09:37:40.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.183e-06 2024-03-20T09:37:40.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:37:40.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177143+0000", 2024-03-20T09:37:40.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9889999999999997e-06 2024-03-20T09:37:40.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:37:40.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177147+0000", 2024-03-20T09:37:40.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.985e-06 2024-03-20T09:37:40.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:37:40.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.382133+0000", 2024-03-20T09:37:40.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.20498566300000001 2024-03-20T09:37:40.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:37:40.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.382152+0000", 2024-03-20T09:37:40.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9562999999999999e-05 2024-03-20T09:37:40.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:37:40.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.423195+0000", 2024-03-20T09:37:40.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.041042661000000001 2024-03-20T09:37:40.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:37:40.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.423235+0000", 2024-03-20T09:37:40.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9913000000000002e-05 2024-03-20T09:37:40.127 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.127 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:37:40.128 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.128 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:194299 7.5 e936/933 7:a6b6325f:::benchmark_data_smithi122_187077_object103984:head v 936'14282, mlcod=936'14266)", 2024-03-20T09:37:40.128 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:37.177154+0000", 2024-03-20T09:37:40.128 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 182.89171235399999, 2024-03-20T09:37:40.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.24611186600000001, 2024-03-20T09:37:40.128 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:37:40.128 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:37:40.128 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:37:40.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:37:40.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177154+0000", 2024-03-20T09:37:40.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:37:40.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177154+0000", 2024-03-20T09:37:40.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:37:40.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177154+0000", 2024-03-20T09:37:40.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1500000000000001e-07 2024-03-20T09:37:40.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:37:40.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177158+0000", 2024-03-20T09:37:40.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.596e-06 2024-03-20T09:37:40.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:37:40.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177163+0000", 2024-03-20T09:37:40.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0699999999999997e-06 2024-03-20T09:37:40.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:37:40.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177167+0000", 2024-03-20T09:37:40.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1740000000000002e-06 2024-03-20T09:37:40.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:37:40.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.385210+0000", 2024-03-20T09:37:40.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.20804297899999999 2024-03-20T09:37:40.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:37:40.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.385224+0000", 2024-03-20T09:37:40.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3679999999999999e-05 2024-03-20T09:37:40.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:37:40.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.423240+0000", 2024-03-20T09:37:40.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.038015938999999999 2024-03-20T09:37:40.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.132 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:37:40.132 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.423266+0000", 2024-03-20T09:37:40.132 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6213000000000001e-05 2024-03-20T09:37:40.132 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.132 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:37:40.132 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.132 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:194300 7.5 e936/933 7:a6b663f8:::benchmark_data_smithi122_187077_object119994:head v 936'14283, mlcod=936'14266)", 2024-03-20T09:37:40.132 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:37.177173+0000", 2024-03-20T09:37:40.132 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 182.89169333800001, 2024-03-20T09:37:40.132 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.24612334699999999, 2024-03-20T09:37:40.132 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:37:40.132 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:37:40.133 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:37:40.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.133 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:37:40.133 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177173+0000", 2024-03-20T09:37:40.133 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.133 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:37:40.133 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177173+0000", 2024-03-20T09:37:40.133 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.133 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:37:40.133 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177173+0000", 2024-03-20T09:37:40.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0100000000000001e-07 2024-03-20T09:37:40.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.134 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:37:40.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177180+0000", 2024-03-20T09:37:40.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8889999999999996e-06 2024-03-20T09:37:40.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.134 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:37:40.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177188+0000", 2024-03-20T09:37:40.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6159999999999996e-06 2024-03-20T09:37:40.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.134 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:37:40.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177192+0000", 2024-03-20T09:37:40.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3080000000000001e-06 2024-03-20T09:37:40.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:37:40.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.388253+0000", 2024-03-20T09:37:40.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.21106075399999999 2024-03-20T09:37:40.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:37:40.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.388264+0000", 2024-03-20T09:37:40.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0688e-05 2024-03-20T09:37:40.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:37:40.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.423270+0000", 2024-03-20T09:37:40.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.035006154999999997 2024-03-20T09:37:40.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:37:40.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.423296+0000", 2024-03-20T09:37:40.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6735999999999999e-05 2024-03-20T09:37:40.136 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.136 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:37:40.136 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.136 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:194301 7.5 e936/933 7:a6b82fee:::benchmark_data_smithi122_187077_object103950:head v 936'14284, mlcod=936'14266)", 2024-03-20T09:37:40.137 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:37.185194+0000", 2024-03-20T09:37:40.137 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 182.883672722, 2024-03-20T09:37:40.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.238133769, 2024-03-20T09:37:40.137 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:37:40.137 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:37:40.137 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:37:40.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.137 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:37:40.137 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.185194+0000", 2024-03-20T09:37:40.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.138 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:37:40.138 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.185194+0000", 2024-03-20T09:37:40.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.138 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:37:40.138 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.185195+0000", 2024-03-20T09:37:40.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.423e-06 2024-03-20T09:37:40.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.138 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:37:40.138 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.185210+0000", 2024-03-20T09:37:40.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.505e-05 2024-03-20T09:37:40.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:37:40.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199550+0000", 2024-03-20T09:37:40.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01433941 2024-03-20T09:37:40.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:37:40.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199570+0000", 2024-03-20T09:37:40.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0043000000000001e-05 2024-03-20T09:37:40.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:37:40.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.391901+0000", 2024-03-20T09:37:40.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.19233090899999999 2024-03-20T09:37:40.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:37:40.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.391919+0000", 2024-03-20T09:37:40.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8754999999999999e-05 2024-03-20T09:37:40.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:37:40.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.423300+0000", 2024-03-20T09:37:40.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031380659999999998 2024-03-20T09:37:40.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:37:40.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.423327+0000", 2024-03-20T09:37:40.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7518999999999999e-05 2024-03-20T09:37:40.141 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.141 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:37:40.141 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.141 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:194302 7.5 e936/933 7:a6b84dcf:::benchmark_data_smithi122_187077_object125423:head v 936'14285, mlcod=936'14266)", 2024-03-20T09:37:40.141 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:37.199600+0000", 2024-03-20T09:37:40.141 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 182.86926626100001, 2024-03-20T09:37:40.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.23229795, 2024-03-20T09:37:40.141 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:37:40.141 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:37:40.141 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:37:40.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:37:40.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199600+0000", 2024-03-20T09:37:40.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.142 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:37:40.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199600+0000", 2024-03-20T09:37:40.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.142 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:37:40.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199601+0000", 2024-03-20T09:37:40.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.161e-06 2024-03-20T09:37:40.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.142 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:37:40.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199613+0000", 2024-03-20T09:37:40.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1545999999999999e-05 2024-03-20T09:37:40.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:37:40.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199625+0000", 2024-03-20T09:37:40.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-03-20T09:37:40.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:37:40.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199632+0000", 2024-03-20T09:37:40.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2450000000000001e-06 2024-03-20T09:37:40.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:37:40.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.399899+0000", 2024-03-20T09:37:40.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.20026687100000001 2024-03-20T09:37:40.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.144 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:37:40.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.399918+0000", 2024-03-20T09:37:40.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9215000000000002e-05 2024-03-20T09:37:40.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.144 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:37:40.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.431855+0000", 2024-03-20T09:37:40.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031937385999999998 2024-03-20T09:37:40.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.145 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:37:40.145 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.431898+0000", 2024-03-20T09:37:40.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2759000000000002e-05 2024-03-20T09:37:40.145 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.145 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:37:40.145 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.145 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:194303 7.5 e936/933 7:a6c193fd:::benchmark_data_smithi122_187077_object102722:head v 936'14286, mlcod=936'14266)", 2024-03-20T09:37:40.145 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:37.199641+0000", 2024-03-20T09:37:40.145 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 182.86922566300001, 2024-03-20T09:37:40.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.23519206300000001, 2024-03-20T09:37:40.145 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:37:40.145 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:37:40.146 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:37:40.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:37:40.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199641+0000", 2024-03-20T09:37:40.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:37:40.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199641+0000", 2024-03-20T09:37:40.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:37:40.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199641+0000", 2024-03-20T09:37:40.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8999999999999998e-07 2024-03-20T09:37:40.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:37:40.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199668+0000", 2024-03-20T09:37:40.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6720000000000002e-05 2024-03-20T09:37:40.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:37:40.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199675+0000", 2024-03-20T09:37:40.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2169999999999997e-06 2024-03-20T09:37:40.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:37:40.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199680+0000", 2024-03-20T09:37:40.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4790000000000004e-06 2024-03-20T09:37:40.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:37:40.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.405442+0000", 2024-03-20T09:37:40.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.20576148499999999 2024-03-20T09:37:40.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:37:40.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.405953+0000", 2024-03-20T09:37:40.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00051126299999999997 2024-03-20T09:37:40.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:37:40.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.434800+0000", 2024-03-20T09:37:40.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.028847212000000001 2024-03-20T09:37:40.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:37:40.149 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.434833+0000", 2024-03-20T09:37:40.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2397000000000002e-05 2024-03-20T09:37:40.149 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.149 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:37:40.149 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.149 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:194304 7.5 e936/933 7:a6c45218:::benchmark_data_smithi122_187077_object124432:head v 936'14287, mlcod=936'14266)", 2024-03-20T09:37:40.149 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:37.199688+0000", 2024-03-20T09:37:40.149 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 182.86917881100001, 2024-03-20T09:37:40.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.23516978199999999, 2024-03-20T09:37:40.150 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:37:40.150 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:37:40.150 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:37:40.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:37:40.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199688+0000", 2024-03-20T09:37:40.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:37:40.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199688+0000", 2024-03-20T09:37:40.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.151 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:37:40.151 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199688+0000", 2024-03-20T09:37:40.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.96e-07 2024-03-20T09:37:40.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.151 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:37:40.151 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199692+0000", 2024-03-20T09:37:40.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6399999999999999e-06 2024-03-20T09:37:40.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.151 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:37:40.151 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199696+0000", 2024-03-20T09:37:40.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7110000000000003e-06 2024-03-20T09:37:40.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:37:40.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199701+0000", 2024-03-20T09:37:40.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9570000000000001e-06 2024-03-20T09:37:40.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:37:40.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.408849+0000", 2024-03-20T09:37:40.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.209147626 2024-03-20T09:37:40.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:37:40.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.408873+0000", 2024-03-20T09:37:40.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4329e-05 2024-03-20T09:37:40.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:37:40.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.434835+0000", 2024-03-20T09:37:40.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.025961992999999999 2024-03-20T09:37:40.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:37:40.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.434857+0000", 2024-03-20T09:37:40.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2229999999999999e-05 2024-03-20T09:37:40.153 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.153 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:37:40.153 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.154 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:194305 7.5 e936/933 7:a6c84c47:::benchmark_data_smithi122_187077_object119692:head v 936'14288, mlcod=936'14266)", 2024-03-20T09:37:40.154 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:37.199706+0000", 2024-03-20T09:37:40.154 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 182.86915998800001, 2024-03-20T09:37:40.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.23517816, 2024-03-20T09:37:40.154 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:37:40.154 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:37:40.154 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:37:40.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:37:40.154 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199706+0000", 2024-03-20T09:37:40.154 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.154 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:37:40.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199706+0000", 2024-03-20T09:37:40.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:37:40.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199707+0000", 2024-03-20T09:37:40.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5100000000000001e-07 2024-03-20T09:37:40.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:37:40.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199715+0000", 2024-03-20T09:37:40.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3520000000000007e-06 2024-03-20T09:37:40.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:37:40.156 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199720+0000", 2024-03-20T09:37:40.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.887e-06 2024-03-20T09:37:40.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.156 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:37:40.156 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199724+0000", 2024-03-20T09:37:40.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3630000000000004e-06 2024-03-20T09:37:40.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.156 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:37:40.156 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.411804+0000", 2024-03-20T09:37:40.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.212080137 2024-03-20T09:37:40.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.156 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:37:40.157 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.411824+0000", 2024-03-20T09:37:40.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9315999999999999e-05 2024-03-20T09:37:40.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.157 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:37:40.157 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.434860+0000", 2024-03-20T09:37:40.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.02303585 2024-03-20T09:37:40.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.157 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:37:40.157 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.434885+0000", 2024-03-20T09:37:40.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5004e-05 2024-03-20T09:37:40.157 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.157 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:37:40.157 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.158 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:194306 7.5 e936/933 7:a6ca5a25:::benchmark_data_smithi122_187077_object115441:head v 936'14289, mlcod=936'14266)", 2024-03-20T09:37:40.158 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:37.199730+0000", 2024-03-20T09:37:40.158 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 182.86913606900001, 2024-03-20T09:37:40.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.23517706799999999, 2024-03-20T09:37:40.158 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:37:40.158 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:37:40.158 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:37:40.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:37:40.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199730+0000", 2024-03-20T09:37:40.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.159 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:37:40.159 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199730+0000", 2024-03-20T09:37:40.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.159 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:37:40.159 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199731+0000", 2024-03-20T09:37:40.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2000000000000001e-07 2024-03-20T09:37:40.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.159 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:37:40.159 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199735+0000", 2024-03-20T09:37:40.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1749999999999998e-06 2024-03-20T09:37:40.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:37:40.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199740+0000", 2024-03-20T09:37:40.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5380000000000002e-06 2024-03-20T09:37:40.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:37:40.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199746+0000", 2024-03-20T09:37:40.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.071e-06 2024-03-20T09:37:40.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:37:40.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.414623+0000", 2024-03-20T09:37:40.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.21487640499999999 2024-03-20T09:37:40.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:37:40.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.414643+0000", 2024-03-20T09:37:40.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0581000000000002e-05 2024-03-20T09:37:40.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:37:40.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.434887+0000", 2024-03-20T09:37:40.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020243351 2024-03-20T09:37:40.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.162 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:37:40.162 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.434907+0000", 2024-03-20T09:37:40.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0727e-05 2024-03-20T09:37:40.162 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.162 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:37:40.162 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.162 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:194307 7.5 e936/933 7:a6ca90f5:::benchmark_data_smithi122_187077_object124172:head v 936'14290, mlcod=936'14266)", 2024-03-20T09:37:40.162 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:37.199751+0000", 2024-03-20T09:37:40.162 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 182.86911499799999, 2024-03-20T09:37:40.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.23628909000000001, 2024-03-20T09:37:40.162 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:37:40.162 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:37:40.163 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:37:40.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:37:40.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199751+0000", 2024-03-20T09:37:40.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:37:40.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199751+0000", 2024-03-20T09:37:40.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:37:40.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199752+0000", 2024-03-20T09:37:40.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.85e-07 2024-03-20T09:37:40.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:37:40.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199760+0000", 2024-03-20T09:37:40.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2039999999999997e-06 2024-03-20T09:37:40.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:37:40.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199766+0000", 2024-03-20T09:37:40.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1229999999999997e-06 2024-03-20T09:37:40.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:37:40.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.209565+0000", 2024-03-20T09:37:40.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097986749999999997 2024-03-20T09:37:40.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:37:40.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.420852+0000", 2024-03-20T09:37:40.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.21128715300000001 2024-03-20T09:37:40.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:37:40.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.420878+0000", 2024-03-20T09:37:40.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6217999999999999e-05 2024-03-20T09:37:40.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:37:40.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.436012+0000", 2024-03-20T09:37:40.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01513452 2024-03-20T09:37:40.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:37:40.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.436040+0000", 2024-03-20T09:37:40.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8011999999999999e-05 2024-03-20T09:37:40.166 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.166 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:37:40.166 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.166 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:194308 7.5 e936/933 7:a6cbfedc:::benchmark_data_smithi122_187077_object123819:head v 936'14291, mlcod=936'14266)", 2024-03-20T09:37:40.166 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:37.209605+0000", 2024-03-20T09:37:40.166 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 182.859261379, 2024-03-20T09:37:40.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.22646349499999999, 2024-03-20T09:37:40.167 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:37:40.167 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:37:40.167 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:37:40.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:37:40.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.209605+0000", 2024-03-20T09:37:40.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:37:40.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.209605+0000", 2024-03-20T09:37:40.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.168 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:37:40.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.209607+0000", 2024-03-20T09:37:40.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5239999999999999e-06 2024-03-20T09:37:40.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.168 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:37:40.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.209620+0000", 2024-03-20T09:37:40.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3729e-05 2024-03-20T09:37:40.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.168 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:37:40.168 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.209633+0000", 2024-03-20T09:37:40.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2918999999999999e-05 2024-03-20T09:37:40.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:37:40.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.209645+0000", 2024-03-20T09:37:40.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1367e-05 2024-03-20T09:37:40.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:37:40.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.423350+0000", 2024-03-20T09:37:40.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.21370534699999999 2024-03-20T09:37:40.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:37:40.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.423363+0000", 2024-03-20T09:37:40.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3475000000000001e-05 2024-03-20T09:37:40.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:37:40.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.436043+0000", 2024-03-20T09:37:40.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012679693000000001 2024-03-20T09:37:40.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:37:40.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.436069+0000", 2024-03-20T09:37:40.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5440999999999999e-05 2024-03-20T09:37:40.170 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.170 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:37:40.170 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.171 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:194309 7.5 e936/933 7:a6cdc98c:::benchmark_data_smithi122_187077_object97831:head v 936'14292, mlcod=936'14266)", 2024-03-20T09:37:40.171 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:37.209653+0000", 2024-03-20T09:37:40.171 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 182.85921358300001, 2024-03-20T09:37:40.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.22644837500000001, 2024-03-20T09:37:40.171 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:37:40.171 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:37:40.171 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:37:40.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.172 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:39 smithi115 ceph-mon[72476]: 8.b scrub ok 2024-03-20T09:37:40.172 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:39 smithi115 ceph-mon[72476]: 8.8 scrub ok 2024-03-20T09:37:40.172 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:39 smithi115 ceph-mon[72476]: pgmap v2446: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 7.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 23 MiB/s wr, 952 op/s 2024-03-20T09:37:40.172 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:39 smithi115 ceph-mon[72476]: 1.10 scrub starts 2024-03-20T09:37:40.172 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:39 smithi115 ceph-mon[72476]: 1.10 scrub ok 2024-03-20T09:37:40.172 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:39 smithi115 ceph-mon[72476]: osdmap e1016: 12 total, 12 up, 12 in 2024-03-20T09:37:40.172 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:39 smithi115 ceph-mon[72476]: 1.a scrub starts 2024-03-20T09:37:40.173 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:39 smithi115 ceph-mon[72476]: 1.a scrub ok 2024-03-20T09:37:40.173 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:39 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:37:40.173 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:39 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:37:40.173 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:39 smithi115 ceph-mon[72476]: 8.1 scrub ok 2024-03-20T09:37:40.173 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:39 smithi115 ceph-mon[72476]: 8.2 scrub ok 2024-03-20T09:37:40.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:37:40.173 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.209653+0000", 2024-03-20T09:37:40.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.173 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:37:40.173 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.209653+0000", 2024-03-20T09:37:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:37:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.209653+0000", 2024-03-20T09:37:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6399999999999998e-07 2024-03-20T09:37:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:37:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.209673+0000", 2024-03-20T09:37:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9653e-05 2024-03-20T09:37:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:37:40.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.209679+0000", 2024-03-20T09:37:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3929999999999996e-06 2024-03-20T09:37:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:37:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.209685+0000", 2024-03-20T09:37:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6799999999999998e-06 2024-03-20T09:37:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:37:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.425864+0000", 2024-03-20T09:37:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.21617926100000001 2024-03-20T09:37:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:37:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.425884+0000", 2024-03-20T09:37:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9993e-05 2024-03-20T09:37:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:37:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.436073+0000", 2024-03-20T09:37:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010189119 2024-03-20T09:37:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:37:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.436101+0000", 2024-03-20T09:37:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8011999999999999e-05 2024-03-20T09:37:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:37:40.176 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.177 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:40.177 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:37:40.177 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:37:40.645 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:37:40.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:40.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:37:40.753 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:37:40.783 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:40.805 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:37:40.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:40.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:37:40.910 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:37:40.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:40 smithi122 ceph-mon[73495]: osdmap e1017: 12 total, 12 up, 12 in 2024-03-20T09:37:40.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:40 smithi122 ceph-mon[73495]: 8.0 scrub starts 2024-03-20T09:37:40.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:40 smithi122 ceph-mon[73495]: 8.c scrub ok 2024-03-20T09:37:40.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:40 smithi026 ceph-mon[68309]: osdmap e1017: 12 total, 12 up, 12 in 2024-03-20T09:37:40.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:40 smithi026 ceph-mon[68309]: 8.0 scrub starts 2024-03-20T09:37:40.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:40 smithi026 ceph-mon[68309]: 8.c scrub ok 2024-03-20T09:37:40.965 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:41.050 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:37:41.115 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:37:41.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:41.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:37:41.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:40 smithi115 ceph-mon[72476]: osdmap e1017: 12 total, 12 up, 12 in 2024-03-20T09:37:41.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:40 smithi115 ceph-mon[72476]: 8.0 scrub starts 2024-03-20T09:37:41.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:40 smithi115 ceph-mon[72476]: 8.c scrub ok 2024-03-20T09:37:41.154 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:37:41.156 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-20T09:37:41.226 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:37:41.302 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:41.346 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:37:41.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:41.350 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:37:41.451 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:37:41.536 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:41.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:41 smithi122 ceph-mon[73495]: pgmap v2449: 47 pgs: 5 active+clean+scrubbing, 42 active+clean; 7.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 840 op/s 2024-03-20T09:37:41.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:41 smithi122 ceph-mon[73495]: 1.c scrub starts 2024-03-20T09:37:41.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:41 smithi026 ceph-mon[68309]: pgmap v2449: 47 pgs: 5 active+clean+scrubbing, 42 active+clean; 7.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 840 op/s 2024-03-20T09:37:41.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:41 smithi026 ceph-mon[68309]: 1.c scrub starts 2024-03-20T09:37:42.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:41 smithi115 ceph-mon[72476]: pgmap v2449: 47 pgs: 5 active+clean+scrubbing, 42 active+clean; 7.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 840 op/s 2024-03-20T09:37:42.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:41 smithi115 ceph-mon[72476]: 1.c scrub starts 2024-03-20T09:37:42.410 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:37:42.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:42.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:37:42.516 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:37:42.604 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:42.652 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:37:42.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:42.661 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:37:42.762 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:37:42.793 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:42.814 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:37:42.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:42.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:37:42.918 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:37:42.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:42 smithi122 ceph-mon[73495]: 1.c scrub ok 2024-03-20T09:37:42.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:42 smithi122 ceph-mon[73495]: 8.d scrub starts 2024-03-20T09:37:42.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:42 smithi122 ceph-mon[73495]: 8.a scrub starts 2024-03-20T09:37:42.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:42 smithi122 ceph-mon[73495]: 8.0 scrub ok 2024-03-20T09:37:42.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:42 smithi122 ceph-mon[73495]: 1.6 scrub starts 2024-03-20T09:37:42.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:42 smithi026 ceph-mon[68309]: 1.c scrub ok 2024-03-20T09:37:42.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:42 smithi026 ceph-mon[68309]: 8.d scrub starts 2024-03-20T09:37:42.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:42 smithi026 ceph-mon[68309]: 8.a scrub starts 2024-03-20T09:37:42.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:42 smithi026 ceph-mon[68309]: 8.0 scrub ok 2024-03-20T09:37:42.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:42 smithi026 ceph-mon[68309]: 1.6 scrub starts 2024-03-20T09:37:42.958 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:43.075 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:37:43.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:43.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:37:43.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:42 smithi115 ceph-mon[72476]: 1.c scrub ok 2024-03-20T09:37:43.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:42 smithi115 ceph-mon[72476]: 8.d scrub starts 2024-03-20T09:37:43.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:42 smithi115 ceph-mon[72476]: 8.a scrub starts 2024-03-20T09:37:43.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:42 smithi115 ceph-mon[72476]: 8.0 scrub ok 2024-03-20T09:37:43.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:42 smithi115 ceph-mon[72476]: 1.6 scrub starts 2024-03-20T09:37:43.183 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:37:43.230 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:43.453 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:37:43.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:43.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:37:43.559 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:37:43.606 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:43.724 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.9 to 1 (10000) 2024-03-20T09:37:43.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:43 smithi026 ceph-mon[68309]: 1.6 scrub ok 2024-03-20T09:37:43.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:43 smithi026 ceph-mon[68309]: pgmap v2450: 47 pgs: 5 active+clean+scrubbing, 42 active+clean; 6.9 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 958 op/s 2024-03-20T09:37:43.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:43 smithi026 ceph-mon[68309]: 1.e scrub starts 2024-03-20T09:37:43.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:43 smithi026 ceph-mon[68309]: 1.e scrub ok 2024-03-20T09:37:43.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:43 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2590597554' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-03-20T09:37:43.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:43 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-03-20T09:37:44.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:43 smithi115 ceph-mon[72476]: 1.6 scrub ok 2024-03-20T09:37:44.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:43 smithi115 ceph-mon[72476]: pgmap v2450: 47 pgs: 5 active+clean+scrubbing, 42 active+clean; 6.9 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 958 op/s 2024-03-20T09:37:44.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:43 smithi115 ceph-mon[72476]: 1.e scrub starts 2024-03-20T09:37:44.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:43 smithi115 ceph-mon[72476]: 1.e scrub ok 2024-03-20T09:37:44.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:43 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2590597554' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-03-20T09:37:44.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:43 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-03-20T09:37:44.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:43 smithi122 ceph-mon[73495]: 1.6 scrub ok 2024-03-20T09:37:44.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:43 smithi122 ceph-mon[73495]: pgmap v2450: 47 pgs: 5 active+clean+scrubbing, 42 active+clean; 6.9 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 958 op/s 2024-03-20T09:37:44.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:43 smithi122 ceph-mon[73495]: 1.e scrub starts 2024-03-20T09:37:44.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:43 smithi122 ceph-mon[73495]: 1.e scrub ok 2024-03-20T09:37:44.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:43 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2590597554' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-03-20T09:37:44.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:43 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-03-20T09:37:45.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:44 smithi115 ceph-mon[72476]: 8.d scrub ok 2024-03-20T09:37:45.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:44 smithi115 ceph-mon[72476]: 8.6 scrub starts 2024-03-20T09:37:45.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:44 smithi115 ceph-mon[72476]: 8.a scrub ok 2024-03-20T09:37:45.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:44 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-03-20T09:37:45.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:44 smithi115 ceph-mon[72476]: osdmap e1018: 12 total, 12 up, 12 in 2024-03-20T09:37:45.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:44 smithi122 ceph-mon[73495]: 8.d scrub ok 2024-03-20T09:37:45.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:44 smithi122 ceph-mon[73495]: 8.6 scrub starts 2024-03-20T09:37:45.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:44 smithi122 ceph-mon[73495]: 8.a scrub ok 2024-03-20T09:37:45.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:44 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-03-20T09:37:45.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:44 smithi122 ceph-mon[73495]: osdmap e1018: 12 total, 12 up, 12 in 2024-03-20T09:37:45.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:44 smithi026 ceph-mon[68309]: 8.d scrub ok 2024-03-20T09:37:45.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:44 smithi026 ceph-mon[68309]: 8.6 scrub starts 2024-03-20T09:37:45.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:44 smithi026 ceph-mon[68309]: 8.a scrub ok 2024-03-20T09:37:45.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:44 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-03-20T09:37:45.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:44 smithi026 ceph-mon[68309]: osdmap e1018: 12 total, 12 up, 12 in 2024-03-20T09:37:45.665 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:37:45.672 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:37:45.672 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:37:45.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:45.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:37:45.774 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:37:45.829 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:45.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:45 smithi026 ceph-mon[68309]: pgmap v2452: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 6.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 820 op/s 2024-03-20T09:37:45.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:45 smithi026 ceph-mon[68309]: 1.7 scrub starts 2024-03-20T09:37:45.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:45 smithi026 ceph-mon[68309]: 1.7 scrub ok 2024-03-20T09:37:45.962 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:37:45.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:45.969 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:37:46.070 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:37:46.096 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:45 smithi115 ceph-mon[72476]: pgmap v2452: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 6.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 820 op/s 2024-03-20T09:37:46.096 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:45 smithi115 ceph-mon[72476]: 1.7 scrub starts 2024-03-20T09:37:46.096 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:45 smithi115 ceph-mon[72476]: 1.7 scrub ok 2024-03-20T09:37:46.098 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:46.120 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:37:46.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:46.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:37:46.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:45 smithi122 ceph-mon[73495]: pgmap v2452: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 6.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 820 op/s 2024-03-20T09:37:46.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:45 smithi122 ceph-mon[73495]: 1.7 scrub starts 2024-03-20T09:37:46.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:45 smithi122 ceph-mon[73495]: 1.7 scrub ok 2024-03-20T09:37:46.226 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:37:46.256 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:46.283 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:37:46.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:46.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:37:46.389 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:37:46.391 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 10 1 2024-03-20T09:37:46.503 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:46.812 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:37:46.825 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:46.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:37:46.926 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:37:46.972 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:47.097 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:46 smithi026 ceph-mon[68309]: 8.6 scrub ok 2024-03-20T09:37:47.098 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:46 smithi026 ceph-mon[68309]: 1.1 scrub starts 2024-03-20T09:37:47.098 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:46 smithi026 ceph-mon[68309]: 1.1 scrub ok 2024-03-20T09:37:47.098 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:37:47.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:46 smithi115 ceph-mon[72476]: 8.6 scrub ok 2024-03-20T09:37:47.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:46 smithi115 ceph-mon[72476]: 1.1 scrub starts 2024-03-20T09:37:47.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:46 smithi115 ceph-mon[72476]: 1.1 scrub ok 2024-03-20T09:37:47.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:37:47.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:46 smithi122 ceph-mon[73495]: 8.6 scrub ok 2024-03-20T09:37:47.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:46 smithi122 ceph-mon[73495]: 1.1 scrub starts 2024-03-20T09:37:47.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:46 smithi122 ceph-mon[73495]: 1.1 scrub ok 2024-03-20T09:37:47.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:37:47.633 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:37:47.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:47.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:37:47.737 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:37:47.771 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:47.799 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:37:47.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:47.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:37:47.903 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:37:47.961 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:47.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:47 smithi026 ceph-mon[68309]: pgmap v2453: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 888 op/s 2024-03-20T09:37:47.991 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:37:47.997 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:47 smithi122 ceph-mon[73495]: pgmap v2453: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 888 op/s 2024-03-20T09:37:47.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:47.998 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:37:48.099 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:37:48.126 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:47 smithi115 ceph-mon[72476]: pgmap v2453: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 888 op/s 2024-03-20T09:37:48.127 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:48.150 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:37:48.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:48.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:37:48.254 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:37:48.297 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:48.465 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:37:48.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:48.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:37:48.570 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:37:48.618 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:48.739 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:37:48.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:48.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:37:48.751 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:37:48.844 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:37:48.872 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:48.893 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:37:48.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:48.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:37:48.955 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:48 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:37:48.956 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:48 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:37:48.956 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:48 smithi026 ceph-mon[68309]: 8.f scrub starts 2024-03-20T09:37:48.956 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:48 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3515408105' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:37:48.956 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:48 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:37:48.956 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:48 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:37:48.956 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:48 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:37:48.956 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:48 smithi026 ceph-mon[68309]: pgmap v2454: 47 pgs: 47 active+clean; 6.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 846 op/s 2024-03-20T09:37:48.998 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:37:49.006 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:37:49.057 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:49.115 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:37:49.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:49.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:37:49.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:48 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:37:49.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:48 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:37:49.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:48 smithi115 ceph-mon[72476]: 8.f scrub starts 2024-03-20T09:37:49.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:48 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3515408105' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:37:49.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:48 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:37:49.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:48 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:37:49.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:48 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:37:49.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:48 smithi115 ceph-mon[72476]: pgmap v2454: 47 pgs: 47 active+clean; 6.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 846 op/s 2024-03-20T09:37:49.220 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:37:49.266 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:48 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:37:49.266 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:48 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:37:49.266 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:48 smithi122 ceph-mon[73495]: 8.f scrub starts 2024-03-20T09:37:49.266 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:48 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3515408105' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:37:49.266 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:48 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:37:49.266 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:48 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:37:49.266 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:48 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:37:49.267 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:48 smithi122 ceph-mon[73495]: pgmap v2454: 47 pgs: 47 active+clean; 6.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 846 op/s 2024-03-20T09:37:49.276 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:49.316 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:37:49.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:49.319 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:37:49.420 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:37:49.475 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:49.520 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:37:49.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:49.524 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:37:49.625 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:37:49.654 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:49.675 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:37:49.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:49.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:37:49.702 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:37:49.702 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:37:49.702 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:49.702 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26917.0:140953 8.f e1019/974 8:f82ea186:::benchmark_data_smithi122_197775_object17649:head v 1019'8664, mlcod=1019'8662)", 2024-03-20T09:37:49.702 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:37:49.708393+0000", 2024-03-20T09:37:49.702 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.004446775, 2024-03-20T09:37:49.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0044699850000000001, 2024-03-20T09:37:49.703 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:37:49.703 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:37:49.703 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:37:49.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:49.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:37:49.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:49.708393+0000", 2024-03-20T09:37:49.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:49.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:49.703 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:49.703 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:37:49.703 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:49.708393+0000", 2024-03-20T09:37:49.703 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:37:49.703 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:49.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:49.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:37:49.704 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:49.708394+0000", 2024-03-20T09:37:49.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5600000000000004e-07 2024-03-20T09:37:49.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:49.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:49.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:37:49.704 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:49.708402+0000", 2024-03-20T09:37:49.704 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5379999999999997e-06 2024-03-20T09:37:49.704 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:49.704 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:49.704 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:37:49.704 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:49.708419+0000", 2024-03-20T09:37:49.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7501999999999999e-05 2024-03-20T09:37:49.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:49.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:49.705 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:37:49.705 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:49.708428+0000", 2024-03-20T09:37:49.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1900000000000001e-06 2024-03-20T09:37:49.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:49.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:49.705 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:37:49.705 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:49.708509+0000", 2024-03-20T09:37:49.705 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0310999999999997e-05 2024-03-20T09:37:49.705 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:37:49.705 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:37:49.706 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:37:49.706 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:49.708531+0000", 2024-03-20T09:37:49.706 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2495999999999999e-05 2024-03-20T09:37:49.706 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:49.706 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:37:49.706 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:49.706 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:37:49.706 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:37:49.707 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 1 2024-03-20T09:37:49.707 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:37:49.780 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:37:49.836 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:49.867 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:37:49.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:49.871 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:37:49.972 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:37:49.999 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:50.022 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:37:50.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:50.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:37:50.125 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:49 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:37:50.125 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:49 smithi026 ceph-mon[68309]: Cluster is now healthy 2024-03-20T09:37:50.126 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:49 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3515408105' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:37:50.126 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:49 smithi026 ceph-mon[68309]: osdmap e1019: 12 total, 12 up, 12 in 2024-03-20T09:37:50.126 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:49 smithi026 ceph-mon[68309]: 8.3 scrub starts 2024-03-20T09:37:50.126 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:37:50.155 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:50.176 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:37:50.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:50.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:37:50.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:49 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:37:50.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:49 smithi122 ceph-mon[73495]: Cluster is now healthy 2024-03-20T09:37:50.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:49 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3515408105' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:37:50.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:49 smithi122 ceph-mon[73495]: osdmap e1019: 12 total, 12 up, 12 in 2024-03-20T09:37:50.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:49 smithi122 ceph-mon[73495]: 8.3 scrub starts 2024-03-20T09:37:50.284 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:37:50.356 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:50.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:49 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:37:50.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:49 smithi115 ceph-mon[72476]: Cluster is now healthy 2024-03-20T09:37:50.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:49 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3515408105' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:37:50.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:49 smithi115 ceph-mon[72476]: osdmap e1019: 12 total, 12 up, 12 in 2024-03-20T09:37:50.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:49 smithi115 ceph-mon[72476]: 8.3 scrub starts 2024-03-20T09:37:50.421 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:37:50.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:50.425 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:37:50.526 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:37:50.577 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:51.204 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-20T09:37:51.245 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:50 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:37:51.245 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:50 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:37:51.245 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:50 smithi026 ceph-mon[68309]: 8.7 scrub starts 2024-03-20T09:37:51.245 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:50 smithi026 ceph-mon[68309]: 8.f scrub ok 2024-03-20T09:37:51.245 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:50 smithi026 ceph-mon[68309]: pgmap v2456: 47 pgs: 47 active+clean; 6.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 792 op/s 2024-03-20T09:37:51.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:50 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:37:51.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:50 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:37:51.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:50 smithi115 ceph-mon[72476]: 8.7 scrub starts 2024-03-20T09:37:51.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:50 smithi115 ceph-mon[72476]: 8.f scrub ok 2024-03-20T09:37:51.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:50 smithi115 ceph-mon[72476]: pgmap v2456: 47 pgs: 47 active+clean; 6.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 792 op/s 2024-03-20T09:37:51.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:50 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:37:51.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:50 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:37:51.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:50 smithi122 ceph-mon[73495]: 8.7 scrub starts 2024-03-20T09:37:51.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:50 smithi122 ceph-mon[73495]: 8.f scrub ok 2024-03-20T09:37:51.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:50 smithi122 ceph-mon[73495]: pgmap v2456: 47 pgs: 47 active+clean; 6.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 792 op/s 2024-03-20T09:37:51.441 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:37:51.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:51.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:37:51.556 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:37:51.609 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:51.652 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:37:51.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:51.658 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:37:51.759 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:37:51.787 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:51.810 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:37:51.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:51.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:37:51.917 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:37:51.967 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:51.998 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:37:52.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:52.005 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:37:52.105 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:37:52.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:51 smithi115 ceph-mon[72476]: 8.3 scrub ok 2024-03-20T09:37:52.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:51 smithi115 ceph-mon[72476]: osdmap e1020: 12 total, 12 up, 12 in 2024-03-20T09:37:52.151 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:52.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:51 smithi122 ceph-mon[73495]: 8.3 scrub ok 2024-03-20T09:37:52.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:51 smithi122 ceph-mon[73495]: osdmap e1020: 12 total, 12 up, 12 in 2024-03-20T09:37:52.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:51 smithi026 ceph-mon[68309]: 8.3 scrub ok 2024-03-20T09:37:52.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:51 smithi026 ceph-mon[68309]: osdmap e1020: 12 total, 12 up, 12 in 2024-03-20T09:37:53.029 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:37:53.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:53.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:37:53.140 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:37:53.168 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:53.193 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:37:53.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:53.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:37:53.298 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:37:53.347 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:53 smithi122 ceph-mon[73495]: 8.7 scrub ok 2024-03-20T09:37:53.348 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:53 smithi122 ceph-mon[73495]: 1.13 deep-scrub starts 2024-03-20T09:37:53.348 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:53 smithi122 ceph-mon[73495]: 1.13 deep-scrub ok 2024-03-20T09:37:53.348 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:53 smithi122 ceph-mon[73495]: pgmap v2458: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 6.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 898 op/s 2024-03-20T09:37:53.352 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:53.388 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:37:53.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:53.395 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:37:53.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:53 smithi115 ceph-mon[72476]: 8.7 scrub ok 2024-03-20T09:37:53.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:53 smithi115 ceph-mon[72476]: 1.13 deep-scrub starts 2024-03-20T09:37:53.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:53 smithi115 ceph-mon[72476]: 1.13 deep-scrub ok 2024-03-20T09:37:53.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:53 smithi115 ceph-mon[72476]: pgmap v2458: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 6.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 898 op/s 2024-03-20T09:37:53.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:53 smithi026 ceph-mon[68309]: 8.7 scrub ok 2024-03-20T09:37:53.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:53 smithi026 ceph-mon[68309]: 1.13 deep-scrub starts 2024-03-20T09:37:53.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:53 smithi026 ceph-mon[68309]: 1.13 deep-scrub ok 2024-03-20T09:37:53.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:53 smithi026 ceph-mon[68309]: pgmap v2458: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 6.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 898 op/s 2024-03-20T09:37:53.496 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:37:53.526 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:53.548 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:37:53.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:53.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:37:53.653 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:37:53.709 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:53.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:53.753 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:37:53.753 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:37:53.854 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:37:53.907 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:53.944 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:37:53.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:53.948 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:37:54.049 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:37:54.078 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:54.099 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:37:54.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:54.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:37:54.127 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:37:54.204 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:37:54.230 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:37:54.233 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:54.252 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:37:54.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:54.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:37:54.273 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:37:54.315 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:37:54.352 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:37:54.357 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:37:54.390 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:37:54.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:54 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/701409761' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:37:54.401 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:54.421 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:37:54.427 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:37:54.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:54.434 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:37:54.454 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:37:54.484 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:37:54.524 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:37:54.534 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:37:54.567 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:37:54.575 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:54 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/701409761' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:37:54.583 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:54.613 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:54 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/701409761' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:37:54.616 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:37:54.617 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:37:54.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:54.622 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:37:54.629 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:37:54.672 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:37:54.687 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:37:54.687 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:37:54.687 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:37:54.687 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:37:54.687 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:37:54.687 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:37:54.687 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:37:54.688 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:37:54.688 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:37:54.688 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:37:54.688 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:37:54.688 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:37:54.723 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:37:54.842 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:54.942 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:37:54.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:54.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:37:55.051 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:37:55.113 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:55.280 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.10 to 1 (10000) 2024-03-20T09:37:55.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:55 smithi115 ceph-mon[72476]: 1.1d scrub starts 2024-03-20T09:37:55.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:55 smithi115 ceph-mon[72476]: 1.1d scrub ok 2024-03-20T09:37:55.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:55 smithi115 ceph-mon[72476]: pgmap v2459: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 6.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 797 op/s 2024-03-20T09:37:55.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:55 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1399047504' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-03-20T09:37:55.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:55 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-03-20T09:37:55.685 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:55 smithi122 ceph-mon[73495]: 1.1d scrub starts 2024-03-20T09:37:55.685 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:55 smithi122 ceph-mon[73495]: 1.1d scrub ok 2024-03-20T09:37:55.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:55 smithi122 ceph-mon[73495]: pgmap v2459: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 6.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 797 op/s 2024-03-20T09:37:55.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:55 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1399047504' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-03-20T09:37:55.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:55 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-03-20T09:37:55.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:55 smithi026 ceph-mon[68309]: 1.1d scrub starts 2024-03-20T09:37:55.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:55 smithi026 ceph-mon[68309]: 1.1d scrub ok 2024-03-20T09:37:55.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:55 smithi026 ceph-mon[68309]: pgmap v2459: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 6.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 797 op/s 2024-03-20T09:37:55.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:55 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1399047504' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-03-20T09:37:55.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:55 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-03-20T09:37:56.302 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:37:56.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:56.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:37:56.410 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:37:56.457 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:56.488 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:37:56.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:56.491 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:37:56.592 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:37:56.640 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:56 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-03-20T09:37:56.640 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:56 smithi026 ceph-mon[68309]: osdmap e1021: 12 total, 12 up, 12 in 2024-03-20T09:37:56.643 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:56.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:56 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-03-20T09:37:56.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:56 smithi115 ceph-mon[72476]: osdmap e1021: 12 total, 12 up, 12 in 2024-03-20T09:37:56.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:56 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-03-20T09:37:56.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:56 smithi122 ceph-mon[73495]: osdmap e1021: 12 total, 12 up, 12 in 2024-03-20T09:37:56.693 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:37:56.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:56.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:37:56.801 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:37:56.838 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:57.002 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:37:57.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:57.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:37:57.107 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:37:57.149 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:57.406 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:37:57.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:57.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:37:57.440 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-20T09:37:57.441 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:37:57.516 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:37:57.554 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:57 smithi122 ceph-mon[73495]: 1.1c scrub starts 2024-03-20T09:37:57.554 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:57 smithi122 ceph-mon[73495]: 1.1c scrub ok 2024-03-20T09:37:57.554 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:57 smithi122 ceph-mon[73495]: pgmap v2461: 47 pgs: 47 active+clean; 6.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 842 op/s 2024-03-20T09:37:57.556 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:57.594 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:37:57.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:57.602 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:37:57.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:57 smithi115 ceph-mon[72476]: 1.1c scrub starts 2024-03-20T09:37:57.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:57 smithi115 ceph-mon[72476]: 1.1c scrub ok 2024-03-20T09:37:57.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:57 smithi115 ceph-mon[72476]: pgmap v2461: 47 pgs: 47 active+clean; 6.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 842 op/s 2024-03-20T09:37:57.703 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:37:57.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:57 smithi026 ceph-mon[68309]: 1.1c scrub starts 2024-03-20T09:37:57.714 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:57 smithi026 ceph-mon[68309]: 1.1c scrub ok 2024-03-20T09:37:57.715 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:57 smithi026 ceph-mon[68309]: pgmap v2461: 47 pgs: 47 active+clean; 6.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 842 op/s 2024-03-20T09:37:57.732 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:57.756 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:37:57.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:57.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:37:57.862 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:37:57.892 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:57.914 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:37:57.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:57.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:37:58.020 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:37:58.078 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:58.270 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:37:58.270 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:37:58.750 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:37:58.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:58.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:37:58.854 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:37:58.898 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:59.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:58 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:37:59.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:58 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:37:59.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:58 smithi115 ceph-mon[72476]: osdmap e1022: 12 total, 12 up, 12 in 2024-03-20T09:37:59.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:37:58 smithi115 ceph-mon[72476]: pgmap v2463: 47 pgs: 47 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 745 op/s 2024-03-20T09:37:59.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:58 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:37:59.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:58 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:37:59.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:58 smithi122 ceph-mon[73495]: osdmap e1022: 12 total, 12 up, 12 in 2024-03-20T09:37:59.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:37:58 smithi122 ceph-mon[73495]: pgmap v2463: 47 pgs: 47 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 745 op/s 2024-03-20T09:37:59.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:58 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:37:59.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:58 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:37:59.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:58 smithi026 ceph-mon[68309]: osdmap e1022: 12 total, 12 up, 12 in 2024-03-20T09:37:59.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:37:58 smithi026 ceph-mon[68309]: pgmap v2463: 47 pgs: 47 active+clean; 6.1 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 745 op/s 2024-03-20T09:37:59.245 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-20T09:37:59.255 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:37:59.259 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:59.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:37:59.310 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:37:59.325 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:37:59.360 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:37:59.407 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:59.430 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:37:59.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:59.433 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:37:59.534 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:37:59.562 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:59.584 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:37:59.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:59.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:37:59.689 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:37:59.739 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:37:59.847 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:37:59.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:37:59.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:37:59.953 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:38:00.007 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:00.043 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:38:00.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:00.055 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:38:00.157 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:38:00.185 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:00.204 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:38:00.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:00.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:38:00.309 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:38:00.361 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:00.385 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:38:00.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:00.393 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:38:00.494 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:38:00.539 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:00.558 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:38:00.624 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:00 smithi026 ceph-mon[68309]: 1.19 scrub starts 2024-03-20T09:38:00.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:00 smithi115 ceph-mon[72476]: 1.19 scrub starts 2024-03-20T09:38:00.685 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:00 smithi122 ceph-mon[73495]: 1.19 scrub starts 2024-03-20T09:38:00.813 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:38:00.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:00.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:38:00.922 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:38:00.960 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:01.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:01 smithi115 ceph-mon[72476]: 1.19 scrub ok 2024-03-20T09:38:01.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:01 smithi115 ceph-mon[72476]: pgmap v2464: 47 pgs: 47 active+clean; 6.0 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 813 op/s 2024-03-20T09:38:01.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:01 smithi115 ceph-mon[72476]: 1.d scrub starts 2024-03-20T09:38:01.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:01 smithi115 ceph-mon[72476]: 1.d scrub ok 2024-03-20T09:38:01.685 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:01 smithi122 ceph-mon[73495]: 1.19 scrub ok 2024-03-20T09:38:01.685 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:01 smithi122 ceph-mon[73495]: pgmap v2464: 47 pgs: 47 active+clean; 6.0 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 813 op/s 2024-03-20T09:38:01.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:01 smithi122 ceph-mon[73495]: 1.d scrub starts 2024-03-20T09:38:01.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:01 smithi122 ceph-mon[73495]: 1.d scrub ok 2024-03-20T09:38:01.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:01 smithi026 ceph-mon[68309]: 1.19 scrub ok 2024-03-20T09:38:01.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:01 smithi026 ceph-mon[68309]: pgmap v2464: 47 pgs: 47 active+clean; 6.0 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 813 op/s 2024-03-20T09:38:01.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:01 smithi026 ceph-mon[68309]: 1.d scrub starts 2024-03-20T09:38:01.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:01 smithi026 ceph-mon[68309]: 1.d scrub ok 2024-03-20T09:38:02.324 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:02 smithi026 ceph-mon[68309]: 1.f scrub starts 2024-03-20T09:38:02.429 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:38:02.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:02.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:38:02.454 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:38:02.534 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:38:02.563 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:02.586 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:38:02.589 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:02 smithi115 ceph-mon[72476]: 1.f scrub starts 2024-03-20T09:38:02.589 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:02 smithi115 ceph-mon[72476]: 1.f scrub ok 2024-03-20T09:38:02.589 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:02 smithi115 ceph-mon[72476]: 1.17 deep-scrub starts 2024-03-20T09:38:02.589 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:02 smithi115 ceph-mon[72476]: 1.17 deep-scrub ok 2024-03-20T09:38:02.590 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:02 smithi115 ceph-mon[72476]: 1.1a scrub starts 2024-03-20T09:38:02.590 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:02 smithi115 ceph-mon[72476]: 1.1a scrub ok 2024-03-20T09:38:02.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:02.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:38:02.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:02 smithi122 ceph-mon[73495]: 1.f scrub starts 2024-03-20T09:38:02.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:02 smithi122 ceph-mon[73495]: 1.f scrub ok 2024-03-20T09:38:02.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:02 smithi122 ceph-mon[73495]: 1.17 deep-scrub starts 2024-03-20T09:38:02.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:02 smithi122 ceph-mon[73495]: 1.17 deep-scrub ok 2024-03-20T09:38:02.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:02 smithi122 ceph-mon[73495]: 1.1a scrub starts 2024-03-20T09:38:02.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:02 smithi122 ceph-mon[73495]: 1.1a scrub ok 2024-03-20T09:38:02.691 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:38:02.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:02 smithi026 ceph-mon[68309]: 1.f scrub ok 2024-03-20T09:38:02.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:02 smithi026 ceph-mon[68309]: 1.17 deep-scrub starts 2024-03-20T09:38:02.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:02 smithi026 ceph-mon[68309]: 1.17 deep-scrub ok 2024-03-20T09:38:02.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:02 smithi026 ceph-mon[68309]: 1.1a scrub starts 2024-03-20T09:38:02.713 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:02 smithi026 ceph-mon[68309]: 1.1a scrub ok 2024-03-20T09:38:02.722 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:02.744 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:38:02.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:02.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:38:02.849 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:38:02.898 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:02.928 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:38:02.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:02.934 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:38:03.035 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:38:03.063 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:03.087 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:38:03.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:03.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:38:03.192 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:38:03.219 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:03.242 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:38:03.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:03.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:38:03.350 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:38:03.418 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:03.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:03 smithi115 ceph-mon[72476]: pgmap v2465: 47 pgs: 47 active+clean; 5.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 930 op/s 2024-03-20T09:38:03.685 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:03 smithi122 ceph-mon[73495]: pgmap v2465: 47 pgs: 47 active+clean; 5.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 930 op/s 2024-03-20T09:38:03.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:03 smithi026 ceph-mon[68309]: pgmap v2465: 47 pgs: 47 active+clean; 5.9 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 930 op/s 2024-03-20T09:38:04.310 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:38:04.314 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:38:04.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:04 smithi115 ceph-mon[72476]: 1.0 scrub starts 2024-03-20T09:38:04.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:04 smithi115 ceph-mon[72476]: 1.0 scrub ok 2024-03-20T09:38:04.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:04 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:38:04.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:04 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:38:04.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:04 smithi122 ceph-mon[73495]: 1.0 scrub starts 2024-03-20T09:38:04.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:04 smithi122 ceph-mon[73495]: 1.0 scrub ok 2024-03-20T09:38:04.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:04 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:38:04.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:04 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:38:04.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:04 smithi026 ceph-mon[68309]: 1.0 scrub starts 2024-03-20T09:38:04.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:04 smithi026 ceph-mon[68309]: 1.0 scrub ok 2024-03-20T09:38:04.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:04 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:38:04.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:04 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:38:05.507 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:38:05.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:05.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:38:05.579 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:05 smithi026 ceph-mon[68309]: osdmap e1023: 12 total, 12 up, 12 in 2024-03-20T09:38:05.579 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:05 smithi026 ceph-mon[68309]: pgmap v2467: 47 pgs: 47 active+clean; 5.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 827 op/s 2024-03-20T09:38:05.579 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:05 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:38:05.579 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:05 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:38:05.579 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:05 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:38:05.579 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:05 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:38:05.613 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:38:05.639 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:05 smithi115 ceph-mon[72476]: osdmap e1023: 12 total, 12 up, 12 in 2024-03-20T09:38:05.639 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:05 smithi115 ceph-mon[72476]: pgmap v2467: 47 pgs: 47 active+clean; 5.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 827 op/s 2024-03-20T09:38:05.639 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:05 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:38:05.639 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:05 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:38:05.639 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:05 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:38:05.639 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:05 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:38:05.641 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:05.662 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:38:05.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:05.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:38:05.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:05 smithi122 ceph-mon[73495]: osdmap e1023: 12 total, 12 up, 12 in 2024-03-20T09:38:05.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:05 smithi122 ceph-mon[73495]: pgmap v2467: 47 pgs: 47 active+clean; 5.8 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 827 op/s 2024-03-20T09:38:05.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:05 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:38:05.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:05 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:38:05.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:05 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:38:05.690 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:05 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:38:05.766 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:38:05.795 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:05.819 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:38:05.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:05.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:38:05.924 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:38:05.972 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:06.016 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:38:06.025 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:06.026 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:38:06.075 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:38:06.075 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:38:06.127 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:38:06.183 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:06.395 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:06 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:38:06.395 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:06 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:38:06.396 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:06 smithi026 ceph-mon[68309]: osdmap e1024: 12 total, 12 up, 12 in 2024-03-20T09:38:06.396 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:06 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1803972026' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:38:06.396 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:06 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:38:06.431 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:38:06.557 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:38:06.560 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:06 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:38:06.560 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:06 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:38:06.561 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:06 smithi122 ceph-mon[73495]: osdmap e1024: 12 total, 12 up, 12 in 2024-03-20T09:38:06.561 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:06 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1803972026' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:38:06.561 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:06 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:38:06.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:06.562 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:38:06.592 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":2465,"stamp":"2024-03-20T09:38:02.578435+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6333006392,"num_objects":96628,"num_object_clones":0,"num_object_copies":289884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96628,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149978,"num_write_kb":7891265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155018,"ondisk_log_size":155018,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":27774068,"kb_used_data":18689468,"kb_used_omap":301,"kb_used_meta":9083922,"kb_avail":1097167756,"statfs":{"total":1151940427776,"available":1123499782144,"internally_reserved":0,"allocated":19138015232,"data_stored":19134470566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":308246,"internal_metadata":9301937130},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,4,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-488243200,"num_objects":-7450,"num_object_clones":0,"num_object_copies":-22350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7450,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003788"},"pg_stats":[{"pgid":"1.1e","version":"20'137","reported_seq":1502,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:37:58.321705+0000","last_change":"2024-03-20T09:37:37.183464+0000","last_active":"2024-03-20T09:37:58.321705+0000","last_peered":"2024-03-20T09:37:58.321705+0000","last_clean":"2024-03-20T09:37:58.321705+0000","last_became_active":"2024-03-20T09:13:20.071655+0000","last_became_peered":"2024-03-20T09:13:20.071655+0000","last_unstale":"2024-03-20T09:37:58.321705+0000","last_undegraded":"2024-03-20T09:37:58.321705+0000","last_fullsized":"2024-03-20T09:37:58.321705+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:37.183343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:37:37.183343+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:48.745062+0000","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'137","reported_seq":587,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:37:58.320850+0000","last_change":"2024-03-20T09:37:54.221807+0000","last_active":"2024-03-20T09:37:58.320850+0000","last_peered":"2024-03-20T09:37:58.320850+0000","last_clean":"2024-03-20T09:37:58.320850+0000","last_became_active":"2024-03-20T09:31:07.425574+0000","last_became_peered":"2024-03-20T09:31:07.425574+0000","last_unstale":"2024-03-20T09:37:58.320850+0000","last_undegraded":"2024-03-20T09:37:58.320850+0000","last_fullsized":"2024-03-20T09:37:58.320850+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:54.221681+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:29:02.585165+0000","last_clean_scrub_stamp":"2024-03-20T09:37:54.221681+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:13.870776+0000","scrub_duration":216,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2],"acting":[11,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"20'137","reported_seq":2191,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:37:58.323738+0000","last_change":"2024-03-20T09:37:56.375223+0000","last_active":"2024-03-20T09:37:58.323738+0000","last_peered":"2024-03-20T09:37:58.323738+0000","last_clean":"2024-03-20T09:37:58.323738+0000","last_became_active":"2024-03-20T09:35:03.815532+0000","last_became_peered":"2024-03-20T09:35:03.815532+0000","last_unstale":"2024-03-20T09:37:58.323738+0000","last_undegraded":"2024-03-20T09:37:58.323738+0000","last_fullsized":"2024-03-20T09:37:58.323738+0000","mapping_epoch":950,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":951,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:56.375139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:37:56.375139+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:15.044311+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.19","version":"20'137","reported_seq":1358,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:37:58.323622+0000","last_change":"2024-03-20T09:36:48.501403+0000","last_active":"2024-03-20T09:37:58.323622+0000","last_peered":"2024-03-20T09:37:58.323622+0000","last_clean":"2024-03-20T09:37:58.323622+0000","last_became_active":"2024-03-20T09:35:19.819786+0000","last_became_peered":"2024-03-20T09:35:19.819786+0000","last_unstale":"2024-03-20T09:37:58.323622+0000","last_undegraded":"2024-03-20T09:37:58.323622+0000","last_fullsized":"2024-03-20T09:37:58.323622+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:36:48.501276+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:30:56.201554+0000","last_clean_scrub_stamp":"2024-03-20T09:36:48.501276+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"20'137","reported_seq":1648,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:37:58.321564+0000","last_change":"2024-03-20T09:36:53.714268+0000","last_active":"2024-03-20T09:37:58.321564+0000","last_peered":"2024-03-20T09:37:58.321564+0000","last_clean":"2024-03-20T09:37:58.321564+0000","last_became_active":"2024-03-20T09:10:35.277481+0000","last_became_peered":"2024-03-20T09:10:35.277481+0000","last_unstale":"2024-03-20T09:37:58.321564+0000","last_undegraded":"2024-03-20T09:37:58.321564+0000","last_fullsized":"2024-03-20T09:37:58.321564+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:36:53.714149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:36:53.714149+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:08.648756+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1b","version":"20'137","reported_seq":2011,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:37:58.323848+0000","last_change":"2024-03-20T09:36:54.315917+0000","last_active":"2024-03-20T09:37:58.323848+0000","last_peered":"2024-03-20T09:37:58.323848+0000","last_clean":"2024-03-20T09:37:58.323848+0000","last_became_active":"2024-03-20T09:24:16.445287+0000","last_became_peered":"2024-03-20T09:24:16.445287+0000","last_unstale":"2024-03-20T09:37:58.323848+0000","last_undegraded":"2024-03-20T09:37:58.323848+0000","last_fullsized":"2024-03-20T09:37:58.323848+0000","mapping_epoch":718,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:36:54.315794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:36:54.315794+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:20.780820+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.1","version":"1022'12173","reported_seq":12537,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:37:58.394344+0000","last_change":"2024-03-20T09:37:38.850438+0000","last_active":"2024-03-20T09:37:58.394344+0000","last_peered":"2024-03-20T09:37:58.394344+0000","last_clean":"2024-03-20T09:37:58.394344+0000","last_became_active":"2024-03-20T09:35:51.943955+0000","last_became_peered":"2024-03-20T09:35:51.943955+0000","last_unstale":"2024-03-20T09:37:58.394344+0000","last_undegraded":"2024-03-20T09:37:58.394344+0000","last_fullsized":"2024-03-20T09:37:58.394344+0000","mapping_epoch":974,"log_start":"982'2100","ondisk_log_start":"982'2100","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1016'11148","last_scrub_stamp":"2024-03-20T09:37:38.850275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:38.850275+0000","objects_scrubbed":7129,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:47.172912+0000","scrub_duration":2090,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":396820480,"num_objects":6055,"num_object_clones":0,"num_object_copies":18165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9281,"num_write_kb":490752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,2],"acting":[9,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.8","version":"20'137","reported_seq":783,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:37:58.321448+0000","last_change":"2024-03-20T09:37:01.184325+0000","last_active":"2024-03-20T09:37:58.321448+0000","last_peered":"2024-03-20T09:37:58.321448+0000","last_clean":"2024-03-20T09:37:58.321448+0000","last_became_active":"2024-03-20T09:35:28.000367+0000","last_became_peered":"2024-03-20T09:35:28.000367+0000","last_unstale":"2024-03-20T09:37:58.321448+0000","last_undegraded":"2024-03-20T09:37:58.321448+0000","last_fullsized":"2024-03-20T09:37:58.321448+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":966,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:01.184206+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:37:01.184206+0000","last_clean_scrub_stamp":"2024-03-20T09:37:01.184206+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:17.102319+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"1022'9278","reported_seq":9464,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:37:59.834232+0000","last_change":"2024-03-20T09:37:41.579524+0000","last_active":"2024-03-20T09:37:59.834232+0000","last_peered":"2024-03-20T09:37:59.834232+0000","last_clean":"2024-03-20T09:37:59.834232+0000","last_became_active":"2024-03-20T09:35:51.276462+0000","last_became_peered":"2024-03-20T09:35:51.276462+0000","last_unstale":"2024-03-20T09:37:59.834232+0000","last_undegraded":"2024-03-20T09:37:59.834232+0000","last_fullsized":"2024-03-20T09:37:59.834232+0000","mapping_epoch":974,"log_start":"0'0","ondisk_log_start":"0'0","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1017'8324","last_scrub_stamp":"2024-03-20T09:37:41.579322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:41.579322+0000","objects_scrubbed":7101,"log_size":9278,"log_dups_size":0,"ondisk_log_size":9278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:55.312205+0000","scrub_duration":1793,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400293888,"num_objects":6108,"num_object_clones":0,"num_object_copies":18324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9278,"num_write_kb":492352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"20'137","reported_seq":2241,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:37:58.324153+0000","last_change":"2024-03-20T09:37:07.903188+0000","last_active":"2024-03-20T09:37:58.324153+0000","last_peered":"2024-03-20T09:37:58.324153+0000","last_clean":"2024-03-20T09:37:58.324153+0000","last_became_active":"2024-03-20T09:20:25.341750+0000","last_became_peered":"2024-03-20T09:20:25.341750+0000","last_unstale":"2024-03-20T09:37:58.324153+0000","last_undegraded":"2024-03-20T09:37:58.324153+0000","last_fullsized":"2024-03-20T09:37:58.324153+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:07.903066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:37:07.903066+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:33.906783+0000","scrub_duration":253,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,2],"acting":[9,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.3","version":"1022'9489","reported_seq":9672,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:38:00.621895+0000","last_change":"2024-03-20T09:37:50.941074+0000","last_active":"2024-03-20T09:38:00.621895+0000","last_peered":"2024-03-20T09:38:00.621895+0000","last_clean":"2024-03-20T09:38:00.621895+0000","last_became_active":"2024-03-20T09:35:51.674444+0000","last_became_peered":"2024-03-20T09:35:51.674444+0000","last_unstale":"2024-03-20T09:38:00.621895+0000","last_undegraded":"2024-03-20T09:38:00.621895+0000","last_fullsized":"2024-03-20T09:38:00.621895+0000","mapping_epoch":974,"log_start":"0'0","ondisk_log_start":"0'0","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1019'8929","last_scrub_stamp":"2024-03-20T09:37:50.940775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:50.940775+0000","objects_scrubbed":6673,"log_size":9489,"log_dups_size":0,"ondisk_log_size":9489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:18.254318+0000","scrub_duration":1914,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397737984,"num_objects":6069,"num_object_clones":0,"num_object_copies":18207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9489,"num_write_kb":497856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"20'137","reported_seq":930,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:37:58.323644+0000","last_change":"2024-03-20T09:37:38.749161+0000","last_active":"2024-03-20T09:37:58.323644+0000","last_peered":"2024-03-20T09:37:58.323644+0000","last_clean":"2024-03-20T09:37:58.323644+0000","last_became_active":"2024-03-20T09:24:16.444232+0000","last_became_peered":"2024-03-20T09:24:16.444232+0000","last_unstale":"2024-03-20T09:37:58.323644+0000","last_undegraded":"2024-03-20T09:37:58.323644+0000","last_fullsized":"2024-03-20T09:37:58.323644+0000","mapping_epoch":718,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:38.749086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:37:38.749086+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:01.052060+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.d","version":"1022'9411","reported_seq":9599,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:38:00.585271+0000","last_change":"2024-03-20T09:37:42.977733+0000","last_active":"2024-03-20T09:38:00.585271+0000","last_peered":"2024-03-20T09:38:00.585271+0000","last_clean":"2024-03-20T09:38:00.585271+0000","last_became_active":"2024-03-20T09:35:51.943827+0000","last_became_peered":"2024-03-20T09:35:51.943827+0000","last_unstale":"2024-03-20T09:38:00.585271+0000","last_undegraded":"2024-03-20T09:38:00.585271+0000","last_fullsized":"2024-03-20T09:38:00.585271+0000","mapping_epoch":974,"log_start":"0'0","ondisk_log_start":"0'0","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1017'8430","last_scrub_stamp":"2024-03-20T09:37:42.977486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:42.977486+0000","objects_scrubbed":7043,"log_size":9411,"log_dups_size":0,"ondisk_log_size":9411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:01.233557+0000","scrub_duration":1930,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394395672,"num_objects":6019,"num_object_clones":0,"num_object_copies":18057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6019,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9411,"num_write_kb":493697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,2],"acting":[11,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"20'137","reported_seq":2170,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:37:58.320597+0000","last_change":"2024-03-20T09:37:00.843889+0000","last_active":"2024-03-20T09:37:58.320597+0000","last_peered":"2024-03-20T09:37:58.320597+0000","last_clean":"2024-03-20T09:37:58.320597+0000","last_became_active":"2024-03-20T09:12:14.306384+0000","last_became_peered":"2024-03-20T09:12:14.306384+0000","last_unstale":"2024-03-20T09:37:58.320597+0000","last_undegraded":"2024-03-20T09:37:58.320597+0000","last_fullsized":"2024-03-20T09:37:58.320597+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:00.843763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:37:00.843763+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:26.792435+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.c","version":"1022'9360","reported_seq":9547,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:37:58.371606+0000","last_change":"2024-03-20T09:37:39.960010+0000","last_active":"2024-03-20T09:37:58.371606+0000","last_peered":"2024-03-20T09:37:58.371606+0000","last_clean":"2024-03-20T09:37:58.371606+0000","last_became_active":"2024-03-20T09:35:51.278227+0000","last_became_peered":"2024-03-20T09:35:51.278227+0000","last_unstale":"2024-03-20T09:37:58.371606+0000","last_undegraded":"2024-03-20T09:37:58.371606+0000","last_fullsized":"2024-03-20T09:37:58.371606+0000","mapping_epoch":974,"log_start":"0'0","ondisk_log_start":"0'0","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1017'8325","last_scrub_stamp":"2024-03-20T09:37:39.959895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:39.959895+0000","objects_scrubbed":7210,"log_size":9360,"log_dups_size":0,"ondisk_log_size":9360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:50.065860+0000","scrub_duration":2195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400424960,"num_objects":6110,"num_object_clones":0,"num_object_copies":18330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9360,"num_write_kb":495040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.5","version":"20'137","reported_seq":830,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:37:58.323354+0000","last_change":"2024-03-20T09:36:47.476093+0000","last_active":"2024-03-20T09:37:58.323354+0000","last_peered":"2024-03-20T09:37:58.323354+0000","last_clean":"2024-03-20T09:37:58.323354+0000","last_became_active":"2024-03-20T09:35:19.820895+0000","last_became_peered":"2024-03-20T09:35:19.820895+0000","last_unstale":"2024-03-20T09:37:58.323354+0000","last_undegraded":"2024-03-20T09:37:58.323354+0000","last_fullsized":"2024-03-20T09:37:58.323354+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:36:47.475965+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:19:03.805185+0000","last_clean_scrub_stamp":"2024-03-20T09:36:47.475965+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:13.637180+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.f","version":"1022'9237","reported_seq":9418,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:38:00.632466+0000","last_change":"2024-03-20T09:37:50.253230+0000","last_active":"2024-03-20T09:38:00.632466+0000","last_peered":"2024-03-20T09:38:00.632466+0000","last_clean":"2024-03-20T09:38:00.632466+0000","last_became_active":"2024-03-20T09:35:51.943804+0000","last_became_peered":"2024-03-20T09:35:51.943804+0000","last_unstale":"2024-03-20T09:38:00.632466+0000","last_undegraded":"2024-03-20T09:38:00.632466+0000","last_fullsized":"2024-03-20T09:38:00.632466+0000","mapping_epoch":974,"log_start":"0'0","ondisk_log_start":"0'0","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1019'8690","last_scrub_stamp":"2024-03-20T09:37:50.253006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:50.253006+0000","objects_scrubbed":6576,"log_size":9237,"log_dups_size":0,"ondisk_log_size":9237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:15.232988+0000","scrub_duration":2230,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391053312,"num_objects":5967,"num_object_clones":0,"num_object_copies":17901,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9237,"num_write_kb":486528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"20'137","reported_seq":1708,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:37:58.324953+0000","last_change":"2024-03-20T09:37:42.468637+0000","last_active":"2024-03-20T09:37:58.324953+0000","last_peered":"2024-03-20T09:37:58.324953+0000","last_clean":"2024-03-20T09:37:58.324953+0000","last_became_active":"2024-03-20T09:35:19.822321+0000","last_became_peered":"2024-03-20T09:35:19.822321+0000","last_unstale":"2024-03-20T09:37:58.324953+0000","last_undegraded":"2024-03-20T09:37:58.324953+0000","last_fullsized":"2024-03-20T09:37:58.324953+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:42.468502+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:24:31.549603+0000","last_clean_scrub_stamp":"2024-03-20T09:37:42.468502+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:02.303736+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.a","version":"1022'9487","reported_seq":9674,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:38:01.710376+0000","last_change":"2024-03-20T09:37:43.187842+0000","last_active":"2024-03-20T09:38:01.710376+0000","last_peered":"2024-03-20T09:38:01.710376+0000","last_clean":"2024-03-20T09:38:01.710376+0000","last_became_active":"2024-03-20T09:35:51.944099+0000","last_became_peered":"2024-03-20T09:35:51.944099+0000","last_unstale":"2024-03-20T09:38:01.710376+0000","last_undegraded":"2024-03-20T09:38:01.710376+0000","last_fullsized":"2024-03-20T09:38:01.710376+0000","mapping_epoch":974,"log_start":"0'0","ondisk_log_start":"0'0","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1017'8466","last_scrub_stamp":"2024-03-20T09:37:43.183079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:43.183079+0000","objects_scrubbed":7069,"log_size":9487,"log_dups_size":0,"ondisk_log_size":9487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:57.208328+0000","scrub_duration":1937,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393936896,"num_objects":6011,"num_object_clones":0,"num_object_copies":18033,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9487,"num_write_kb":495936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"20'137","reported_seq":1235,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:37:58.690650+0000","last_change":"2024-03-20T09:36:54.845047+0000","last_active":"2024-03-20T09:37:58.690650+0000","last_peered":"2024-03-20T09:37:58.690650+0000","last_clean":"2024-03-20T09:37:58.690650+0000","last_became_active":"2024-03-20T09:31:07.424588+0000","last_became_peered":"2024-03-20T09:31:07.424588+0000","last_unstale":"2024-03-20T09:37:58.690650+0000","last_undegraded":"2024-03-20T09:37:58.690650+0000","last_fullsized":"2024-03-20T09:37:58.690650+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:36:54.844902+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:30:47.812125+0000","last_clean_scrub_stamp":"2024-03-20T09:36:54.844902+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:15.580612+0000","scrub_duration":211,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.9","version":"1022'9365","reported_seq":9540,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:37:58.695162+0000","last_change":"2024-03-20T09:37:03.599757+0000","last_active":"2024-03-20T09:37:58.695162+0000","last_peered":"2024-03-20T09:37:58.695162+0000","last_clean":"2024-03-20T09:37:58.695162+0000","last_became_active":"2024-03-20T09:35:51.274816+0000","last_became_peered":"2024-03-20T09:35:51.274816+0000","last_unstale":"2024-03-20T09:37:58.695162+0000","last_undegraded":"2024-03-20T09:37:58.695162+0000","last_fullsized":"2024-03-20T09:37:58.695162+0000","mapping_epoch":974,"log_start":"0'0","ondisk_log_start":"0'0","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1005'6201","last_scrub_stamp":"2024-03-20T09:37:03.599526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:03.599526+0000","objects_scrubbed":6170,"log_size":9365,"log_dups_size":0,"ondisk_log_size":9365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:29.249509+0000","scrub_duration":1194,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403636224,"num_objects":6159,"num_object_clones":0,"num_object_copies":18477,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9365,"num_write_kb":496768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":2141,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:37:58.320536+0000","last_change":"2024-03-20T09:36:57.835852+0000","last_active":"2024-03-20T09:37:58.320536+0000","last_peered":"2024-03-20T09:37:58.320536+0000","last_clean":"2024-03-20T09:37:58.320536+0000","last_became_active":"2024-03-20T09:12:14.305989+0000","last_became_peered":"2024-03-20T09:12:14.305989+0000","last_unstale":"2024-03-20T09:37:58.320536+0000","last_undegraded":"2024-03-20T09:37:58.320536+0000","last_fullsized":"2024-03-20T09:37:58.320536+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:36:57.835682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:36:57.835682+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:02.503806+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.b","version":"1022'9531","reported_seq":9718,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:38:01.708088+0000","last_change":"2024-03-20T09:37:38.084925+0000","last_active":"2024-03-20T09:38:01.708088+0000","last_peered":"2024-03-20T09:38:01.708088+0000","last_clean":"2024-03-20T09:38:01.708088+0000","last_became_active":"2024-03-20T09:35:51.942846+0000","last_became_peered":"2024-03-20T09:35:51.942846+0000","last_unstale":"2024-03-20T09:38:01.708088+0000","last_undegraded":"2024-03-20T09:38:01.708088+0000","last_fullsized":"2024-03-20T09:38:01.708088+0000","mapping_epoch":974,"log_start":"0'0","ondisk_log_start":"0'0","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1015'8221","last_scrub_stamp":"2024-03-20T09:37:38.084645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:38.084645+0000","objects_scrubbed":7285,"log_size":9531,"log_dups_size":0,"ondisk_log_size":9531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:52.604722+0000","scrub_duration":1768,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388825088,"num_objects":5933,"num_object_clones":0,"num_object_copies":17799,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9531,"num_write_kb":494848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"20'137","reported_seq":2406,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:37:58.327752+0000","last_change":"2024-03-20T09:36:50.854490+0000","last_active":"2024-03-20T09:37:58.327752+0000","last_peered":"2024-03-20T09:37:58.327752+0000","last_clean":"2024-03-20T09:37:58.327752+0000","last_became_active":"2024-03-20T09:19:54.492879+0000","last_became_peered":"2024-03-20T09:19:54.492879+0000","last_unstale":"2024-03-20T09:37:58.327752+0000","last_undegraded":"2024-03-20T09:37:58.327752+0000","last_fullsized":"2024-03-20T09:37:58.327752+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:36:50.854413+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:19.513511+0000","last_clean_scrub_stamp":"2024-03-20T09:36:50.854413+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:14.992458+0000","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"1022'9458","reported_seq":9645,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:37:59.942321+0000","last_change":"2024-03-20T09:37:38.178742+0000","last_active":"2024-03-20T09:37:59.942321+0000","last_peered":"2024-03-20T09:37:59.942321+0000","last_clean":"2024-03-20T09:37:59.942321+0000","last_became_active":"2024-03-20T09:35:51.276320+0000","last_became_peered":"2024-03-20T09:35:51.276320+0000","last_unstale":"2024-03-20T09:37:59.942321+0000","last_undegraded":"2024-03-20T09:37:59.942321+0000","last_fullsized":"2024-03-20T09:37:59.942321+0000","mapping_epoch":974,"log_start":"0'0","ondisk_log_start":"0'0","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1015'8253","last_scrub_stamp":"2024-03-20T09:37:38.178617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:38.178617+0000","objects_scrubbed":7305,"log_size":9458,"log_dups_size":0,"ondisk_log_size":9458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:59.551600+0000","scrub_duration":1541,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397934592,"num_objects":6072,"num_object_clones":0,"num_object_copies":18216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9458,"num_write_kb":496960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"20'137","reported_seq":1589,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:37:58.690603+0000","last_change":"2024-03-20T09:37:45.841316+0000","last_active":"2024-03-20T09:37:58.690603+0000","last_peered":"2024-03-20T09:37:58.690603+0000","last_clean":"2024-03-20T09:37:58.690603+0000","last_became_active":"2024-03-20T09:15:34.225022+0000","last_became_peered":"2024-03-20T09:15:34.225022+0000","last_unstale":"2024-03-20T09:37:58.690603+0000","last_undegraded":"2024-03-20T09:37:58.690603+0000","last_fullsized":"2024-03-20T09:37:58.690603+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:45.841176+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:20:12.169658+0000","last_clean_scrub_stamp":"2024-03-20T09:37:45.841176+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:11.852244+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,5],"acting":[6,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"1022'9426","reported_seq":9602,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:38:01.927953+0000","last_change":"2024-03-20T09:37:04.086352+0000","last_active":"2024-03-20T09:38:01.927953+0000","last_peered":"2024-03-20T09:38:01.927953+0000","last_clean":"2024-03-20T09:38:01.927953+0000","last_became_active":"2024-03-20T09:35:51.943717+0000","last_became_peered":"2024-03-20T09:35:51.943717+0000","last_unstale":"2024-03-20T09:38:01.927953+0000","last_undegraded":"2024-03-20T09:38:01.927953+0000","last_fullsized":"2024-03-20T09:38:01.927953+0000","mapping_epoch":974,"log_start":"0'0","ondisk_log_start":"0'0","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1006'6187","last_scrub_stamp":"2024-03-20T09:37:04.086220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:04.086220+0000","objects_scrubbed":6142,"log_size":9426,"log_dups_size":0,"ondisk_log_size":9426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:04.544114+0000","scrub_duration":1249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385220608,"num_objects":5878,"num_object_clones":0,"num_object_copies":17634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9426,"num_write_kb":489728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"20'137","reported_seq":2137,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:37:58.690820+0000","last_change":"2024-03-20T09:37:45.012425+0000","last_active":"2024-03-20T09:37:58.690820+0000","last_peered":"2024-03-20T09:37:58.690820+0000","last_clean":"2024-03-20T09:37:58.690820+0000","last_became_active":"2024-03-20T09:35:19.820396+0000","last_became_peered":"2024-03-20T09:35:19.820396+0000","last_unstale":"2024-03-20T09:37:58.690820+0000","last_undegraded":"2024-03-20T09:37:58.690820+0000","last_fullsized":"2024-03-20T09:37:58.690820+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:45.012294+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:14:10.560890+0000","last_clean_scrub_stamp":"2024-03-20T09:37:45.012294+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:57.274595+0000","scrub_duration":222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2","version":"1022'9326","reported_seq":9513,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:37:58.751854+0000","last_change":"2024-03-20T09:37:39.223390+0000","last_active":"2024-03-20T09:37:58.751854+0000","last_peered":"2024-03-20T09:37:58.751854+0000","last_clean":"2024-03-20T09:37:58.751854+0000","last_became_active":"2024-03-20T09:35:51.942847+0000","last_became_peered":"2024-03-20T09:35:51.942847+0000","last_unstale":"2024-03-20T09:37:58.751854+0000","last_undegraded":"2024-03-20T09:37:58.751854+0000","last_fullsized":"2024-03-20T09:37:58.751854+0000","mapping_epoch":974,"log_start":"0'0","ondisk_log_start":"0'0","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1016'8306","last_scrub_stamp":"2024-03-20T09:37:39.223182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:39.223182+0000","objects_scrubbed":7252,"log_size":9326,"log_dups_size":0,"ondisk_log_size":9326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:51.271166+0000","scrub_duration":1777,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405929984,"num_objects":6194,"num_object_clones":0,"num_object_copies":18582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9326,"num_write_kb":496640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,4],"acting":[10,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.b","version":"20'137","reported_seq":1953,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:37:58.321949+0000","last_change":"2024-03-20T09:37:06.065195+0000","last_active":"2024-03-20T09:37:58.321949+0000","last_peered":"2024-03-20T09:37:58.321949+0000","last_clean":"2024-03-20T09:37:58.321949+0000","last_became_active":"2024-03-20T09:13:20.070596+0000","last_became_peered":"2024-03-20T09:13:20.070596+0000","last_unstale":"2024-03-20T09:37:58.321949+0000","last_undegraded":"2024-03-20T09:37:58.321949+0000","last_fullsized":"2024-03-20T09:37:58.321949+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:06.065105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:37:06.065105+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:35.637879+0000","scrub_duration":316,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.7","version":"1022'9386","reported_seq":9566,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:38:00.630179+0000","last_change":"2024-03-20T09:37:51.929007+0000","last_active":"2024-03-20T09:38:00.630179+0000","last_peered":"2024-03-20T09:38:00.630179+0000","last_clean":"2024-03-20T09:38:00.630179+0000","last_became_active":"2024-03-20T09:35:51.673870+0000","last_became_peered":"2024-03-20T09:35:51.673870+0000","last_unstale":"2024-03-20T09:38:00.630179+0000","last_undegraded":"2024-03-20T09:38:00.630179+0000","last_fullsized":"2024-03-20T09:38:00.630179+0000","mapping_epoch":974,"log_start":"0'0","ondisk_log_start":"0'0","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1020'8885","last_scrub_stamp":"2024-03-20T09:37:51.928857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:51.928857+0000","objects_scrubbed":6515,"log_size":9386,"log_dups_size":0,"ondisk_log_size":9386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:10.833584+0000","scrub_duration":1835,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391249920,"num_objects":5970,"num_object_clones":0,"num_object_copies":17910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9386,"num_write_kb":491392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.e","version":"20'137","reported_seq":1394,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:37:58.325689+0000","last_change":"2024-03-20T09:37:42.890342+0000","last_active":"2024-03-20T09:37:58.325689+0000","last_peered":"2024-03-20T09:37:58.325689+0000","last_clean":"2024-03-20T09:37:58.325689+0000","last_became_active":"2024-03-20T09:19:54.506134+0000","last_became_peered":"2024-03-20T09:19:54.506134+0000","last_unstale":"2024-03-20T09:37:58.325689+0000","last_undegraded":"2024-03-20T09:37:58.325689+0000","last_fullsized":"2024-03-20T09:37:58.325689+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:42.890257+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:29:53.362224+0000","last_clean_scrub_stamp":"2024-03-20T09:37:42.890257+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:47.227463+0000","scrub_duration":157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"1022'9488","reported_seq":9672,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:38:00.650061+0000","last_change":"2024-03-20T09:37:44.856776+0000","last_active":"2024-03-20T09:38:00.650061+0000","last_peered":"2024-03-20T09:38:00.650061+0000","last_clean":"2024-03-20T09:38:00.650061+0000","last_became_active":"2024-03-20T09:35:51.312260+0000","last_became_peered":"2024-03-20T09:35:51.312260+0000","last_unstale":"2024-03-20T09:38:00.650061+0000","last_undegraded":"2024-03-20T09:38:00.650061+0000","last_fullsized":"2024-03-20T09:38:00.650061+0000","mapping_epoch":974,"log_start":"0'0","ondisk_log_start":"0'0","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1018'8599","last_scrub_stamp":"2024-03-20T09:37:44.856610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:44.856610+0000","objects_scrubbed":6942,"log_size":9488,"log_dups_size":0,"ondisk_log_size":9488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:52.896041+0000","scrub_duration":1869,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393084928,"num_objects":5998,"num_object_clones":0,"num_object_copies":17994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9488,"num_write_kb":495552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,7],"acting":[11,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.f","version":"20'137","reported_seq":2157,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:38:01.026382+0000","last_change":"2024-03-20T09:38:01.026424+0000","last_active":"2024-03-20T09:38:01.026382+0000","last_peered":"2024-03-20T09:38:01.026382+0000","last_clean":"2024-03-20T09:38:01.026382+0000","last_became_active":"2024-03-20T09:19:54.524397+0000","last_became_peered":"2024-03-20T09:19:54.524397+0000","last_unstale":"2024-03-20T09:38:01.026382+0000","last_undegraded":"2024-03-20T09:38:01.026382+0000","last_fullsized":"2024-03-20T09:38:01.026382+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:38:01.026288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:38:01.026288+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:09.847998+0000","scrub_duration":214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.5","version":"1022'9233","reported_seq":9406,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:37:58.717356+0000","last_change":"2024-03-20T09:36:55.745524+0000","last_active":"2024-03-20T09:37:58.717356+0000","last_peered":"2024-03-20T09:37:58.717356+0000","last_clean":"2024-03-20T09:37:58.717356+0000","last_became_active":"2024-03-20T09:35:51.942869+0000","last_became_peered":"2024-03-20T09:35:51.942869+0000","last_unstale":"2024-03-20T09:37:58.717356+0000","last_undegraded":"2024-03-20T09:37:58.717356+0000","last_fullsized":"2024-03-20T09:37:58.717356+0000","mapping_epoch":974,"log_start":"0'0","ondisk_log_start":"0'0","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'5720","last_scrub_stamp":"2024-03-20T09:36:55.745369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:36:55.745369+0000","objects_scrubbed":5679,"log_size":9233,"log_dups_size":0,"ondisk_log_size":9233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:12.875325+0000","scrub_duration":1369,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404553728,"num_objects":6173,"num_object_clones":0,"num_object_copies":18519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9233,"num_write_kb":492992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,1],"acting":[10,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.c","version":"570'138","reported_seq":2134,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:37:58.322894+0000","last_change":"2024-03-20T09:37:40.922108+0000","last_active":"2024-03-20T09:37:58.322894+0000","last_peered":"2024-03-20T09:37:58.322894+0000","last_clean":"2024-03-20T09:37:58.322894+0000","last_became_active":"2024-03-20T09:13:27.087686+0000","last_became_peered":"2024-03-20T09:13:27.087686+0000","last_unstale":"2024-03-20T09:37:58.322894+0000","last_undegraded":"2024-03-20T09:37:58.322894+0000","last_fullsized":"2024-03-20T09:37:58.322894+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"570'138","last_scrub_stamp":"2024-03-20T09:37:40.921974+0000","last_deep_scrub":"570'138","last_deep_scrub_stamp":"2024-03-20T09:16:13.725035+0000","last_clean_scrub_stamp":"2024-03-20T09:37:40.921974+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:43.602278+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"1022'9222","reported_seq":9399,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:37:59.962237+0000","last_change":"2024-03-20T09:37:04.918258+0000","last_active":"2024-03-20T09:37:59.962237+0000","last_peered":"2024-03-20T09:37:59.962237+0000","last_clean":"2024-03-20T09:37:59.962237+0000","last_became_active":"2024-03-20T09:35:51.277801+0000","last_became_peered":"2024-03-20T09:35:51.277801+0000","last_unstale":"2024-03-20T09:37:59.962237+0000","last_undegraded":"2024-03-20T09:37:59.962237+0000","last_fullsized":"2024-03-20T09:37:59.962237+0000","mapping_epoch":974,"log_start":"0'0","ondisk_log_start":"0'0","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1006'6159","last_scrub_stamp":"2024-03-20T09:37:04.918103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:04.918103+0000","objects_scrubbed":6123,"log_size":9222,"log_dups_size":0,"ondisk_log_size":9222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:14.259459+0000","scrub_duration":1242,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":387317760,"num_objects":5910,"num_object_clones":0,"num_object_copies":17730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9222,"num_write_kb":484224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,10],"acting":[5,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"20'137","reported_seq":1700,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:37:58.328877+0000","last_change":"2024-03-20T09:36:50.123146+0000","last_active":"2024-03-20T09:37:58.328877+0000","last_peered":"2024-03-20T09:37:58.328877+0000","last_clean":"2024-03-20T09:37:58.328877+0000","last_became_active":"2024-03-20T09:19:54.502529+0000","last_became_peered":"2024-03-20T09:19:54.502529+0000","last_unstale":"2024-03-20T09:37:58.328877+0000","last_undegraded":"2024-03-20T09:37:58.328877+0000","last_fullsized":"2024-03-20T09:37:58.328877+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:36:50.123032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:36:50.123032+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"20'137","reported_seq":1988,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:37:58.321596+0000","last_change":"2024-03-20T09:37:07.729241+0000","last_active":"2024-03-20T09:37:58.321596+0000","last_peered":"2024-03-20T09:37:58.321596+0000","last_clean":"2024-03-20T09:37:58.321596+0000","last_became_active":"2024-03-20T09:10:34.096758+0000","last_became_peered":"2024-03-20T09:10:34.096758+0000","last_unstale":"2024-03-20T09:37:58.321596+0000","last_undegraded":"2024-03-20T09:37:58.321596+0000","last_fullsized":"2024-03-20T09:37:58.321596+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:07.729109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:37:07.729109+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:14.194912+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"20'137","reported_seq":749,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:37:58.322135+0000","last_change":"2024-03-20T09:37:52.360050+0000","last_active":"2024-03-20T09:37:58.322135+0000","last_peered":"2024-03-20T09:37:58.322135+0000","last_clean":"2024-03-20T09:37:58.322135+0000","last_became_active":"2024-03-20T09:35:19.822242+0000","last_became_peered":"2024-03-20T09:35:19.822242+0000","last_unstale":"2024-03-20T09:37:58.322135+0000","last_undegraded":"2024-03-20T09:37:58.322135+0000","last_fullsized":"2024-03-20T09:37:58.322135+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:52.359932+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:37:52.359932+0000","last_clean_scrub_stamp":"2024-03-20T09:37:52.359932+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:00.922956+0000","scrub_duration":315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.10","version":"20'137","reported_seq":1551,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:37:58.321685+0000","last_change":"2024-03-20T09:37:38.621260+0000","last_active":"2024-03-20T09:37:58.321685+0000","last_peered":"2024-03-20T09:37:58.321685+0000","last_clean":"2024-03-20T09:37:58.321685+0000","last_became_active":"2024-03-20T09:35:19.820595+0000","last_became_peered":"2024-03-20T09:35:19.820595+0000","last_unstale":"2024-03-20T09:37:58.321685+0000","last_undegraded":"2024-03-20T09:37:58.321685+0000","last_fullsized":"2024-03-20T09:37:58.321685+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:38.621168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:37:38.621168+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:43.836018+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"20'137","reported_seq":1932,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:37:58.327697+0000","last_change":"2024-03-20T09:36:51.839369+0000","last_active":"2024-03-20T09:37:58.327697+0000","last_peered":"2024-03-20T09:37:58.327697+0000","last_clean":"2024-03-20T09:37:58.327697+0000","last_became_active":"2024-03-20T09:20:25.342982+0000","last_became_peered":"2024-03-20T09:20:25.342982+0000","last_unstale":"2024-03-20T09:37:58.327697+0000","last_undegraded":"2024-03-20T09:37:58.327697+0000","last_fullsized":"2024-03-20T09:37:58.327697+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:36:51.839234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:36:51.839234+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:19.300010+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"20'137","reported_seq":657,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:37:58.327873+0000","last_change":"2024-03-20T09:36:54.828126+0000","last_active":"2024-03-20T09:37:58.327873+0000","last_peered":"2024-03-20T09:37:58.327873+0000","last_clean":"2024-03-20T09:37:58.327873+0000","last_became_active":"2024-03-20T09:31:07.424851+0000","last_became_peered":"2024-03-20T09:31:07.424851+0000","last_unstale":"2024-03-20T09:37:58.327873+0000","last_undegraded":"2024-03-20T09:37:58.327873+0000","last_fullsized":"2024-03-20T09:37:58.327873+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:36:54.827943+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:35:27.264203+0000","last_clean_scrub_stamp":"2024-03-20T09:36:54.827943+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:19.315177+0000","scrub_duration":174,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.17","version":"20'137","reported_seq":2129,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:38:01.419976+0000","last_change":"2024-03-20T09:38:01.420016+0000","last_active":"2024-03-20T09:38:01.419976+0000","last_peered":"2024-03-20T09:38:01.419976+0000","last_clean":"2024-03-20T09:38:01.419976+0000","last_became_active":"2024-03-20T09:35:19.822429+0000","last_became_peered":"2024-03-20T09:35:19.822429+0000","last_unstale":"2024-03-20T09:38:01.419976+0000","last_undegraded":"2024-03-20T09:38:01.419976+0000","last_fullsized":"2024-03-20T09:38:01.419976+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:38:01.419884+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:38:01.419884+0000","last_clean_scrub_stamp":"2024-03-20T09:38:01.419884+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:22.126398+0000","scrub_duration":169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"20'137","reported_seq":2161,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:37:58.321524+0000","last_change":"2024-03-20T09:36:56.204036+0000","last_active":"2024-03-20T09:37:58.321524+0000","last_peered":"2024-03-20T09:37:58.321524+0000","last_clean":"2024-03-20T09:37:58.321524+0000","last_became_active":"2024-03-20T09:20:25.349082+0000","last_became_peered":"2024-03-20T09:20:25.349082+0000","last_unstale":"2024-03-20T09:37:58.321524+0000","last_undegraded":"2024-03-20T09:37:58.321524+0000","last_fullsized":"2024-03-20T09:37:58.321524+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:36:56.203907+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:36:56.203907+0000","last_clean_scrub_stamp":"2024-03-20T09:36:56.203907+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:11.989863+0000","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"20'137","reported_seq":1440,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:37:58.324560+0000","last_change":"2024-03-20T09:37:36.549435+0000","last_active":"2024-03-20T09:37:58.324560+0000","last_peered":"2024-03-20T09:37:58.324560+0000","last_clean":"2024-03-20T09:37:58.324560+0000","last_became_active":"2024-03-20T09:12:14.307581+0000","last_became_peered":"2024-03-20T09:12:14.307581+0000","last_unstale":"2024-03-20T09:37:58.324560+0000","last_undegraded":"2024-03-20T09:37:58.324560+0000","last_fullsized":"2024-03-20T09:37:58.324560+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:36.549355+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:34:48.079272+0000","last_clean_scrub_stamp":"2024-03-20T09:37:36.549355+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:04.698728+0000","scrub_duration":206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"20'137","reported_seq":1699,"reported_epoch":1022,"state":"active+clean","last_fresh":"2024-03-20T09:37:58.329199+0000","last_change":"2024-03-20T09:36:51.982934+0000","last_active":"2024-03-20T09:37:58.329199+0000","last_peered":"2024-03-20T09:37:58.329199+0000","last_clean":"2024-03-20T09:37:58.329199+0000","last_became_active":"2024-03-20T09:19:54.502391+0000","last_became_peered":"2024-03-20T09:19:54.502391+0000","last_unstale":"2024-03-20T09:37:58.329199+0000","last_undegraded":"2024-03-20T09:37:58.329199+0000","last_fullsized":"2024-03-20T09:37:58.329199+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:36:51.982846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:36:51.982846+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:37:59.657927+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":6332416024,"num_objects":96626,"num_object_clones":0,"num_object_copies":289878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96626,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149978,"num_write_kb":7891265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19044184064,"data_stored":19044171848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150770,"ondisk_log_size":150770,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4248,"ondisk_log_size":4248,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860920,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2882928,"kb_used_data":1935464,"kb_used_omap":1,"kb_used_meta":947454,"kb_avail":90862224,"statfs":{"total":95995035648,"available":93042917376,"internally_reserved":0,"allocated":1981915136,"data_stored":1981616837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":970193348},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":861,"seq":3697966841945,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2514800,"kb_used_data":2355876,"kb_used_omap":103,"kb_used_meta":158616,"kb_avail":91230352,"statfs":{"total":95995035648,"available":93419880448,"internally_reserved":0,"allocated":2412417024,"data_stored":2412122797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":106169,"internal_metadata":162423111},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":80,"seq":343597384197,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2919712,"kb_used_data":2356824,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":90825440,"statfs":{"total":95995035648,"available":93005250560,"internally_reserved":0,"allocated":2413387776,"data_stored":2413098014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":576387520},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":70,"seq":300647711242,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1725548,"kb_used_data":1148708,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":92019604,"statfs":{"total":95995035648,"available":94228074496,"internally_reserved":0,"allocated":1176276992,"data_stored":1175982765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":590608838},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":60,"seq":257698038286,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1668668,"kb_used_data":771184,"kb_used_omap":1,"kb_used_meta":897470,"kb_avail":92076484,"statfs":{"total":95995035648,"available":94286319616,"internally_reserved":0,"allocated":789692416,"data_stored":789390565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":919009738},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332627,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":704176,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93040976,"statfs":{"total":95995035648,"available":95273959424,"internally_reserved":0,"allocated":1085440,"data_stored":791213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":719976902},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575405,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3293232,"kb_used_data":2345764,"kb_used_omap":1,"kb_used_meta":947454,"kb_avail":90451920,"statfs":{"total":95995035648,"available":92622766080,"internally_reserved":0,"allocated":2402062336,"data_stored":2401768109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":970193350},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738924,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2171372,"kb_used_data":1147556,"kb_used_omap":1,"kb_used_meta":1023806,"kb_avail":91573780,"statfs":{"total":95995035648,"available":93771550720,"internally_reserved":0,"allocated":1175097344,"data_stored":1174803117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1048377797},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411877,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2759596,"kb_used_data":1962084,"kb_used_omap":1,"kb_used_meta":797502,"kb_avail":90985556,"statfs":{"total":95995035648,"available":93169209344,"internally_reserved":0,"allocated":2009174016,"data_stored":2008876630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":816642501},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150242,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2145644,"kb_used_data":1167652,"kb_used_omap":1,"kb_used_meta":977982,"kb_avail":91599508,"statfs":{"total":95995035648,"available":93797896192,"internally_reserved":0,"allocated":1195675648,"data_stored":1195381421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1001454023},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438954017,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2311856,"kb_used_data":1554596,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":91433296,"statfs":{"total":95995035648,"available":93627695104,"internally_reserved":0,"allocated":1591906304,"data_stored":1591612077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":775420357},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":443,"seq":1902670512458,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2676536,"kb_used_data":1942700,"kb_used_omap":181,"kb_used_meta":733642,"kb_avail":91068616,"statfs":{"total":95995035648,"available":93254262784,"internally_reserved":0,"allocated":1989324800,"data_stored":1989027021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":186131,"internal_metadata":751249645},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1172766720,"data_stored":1172766720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2381709312,"data_stored":2381709312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2001735680,"data_stored":2001731608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1193017344,"data_stored":1193017344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1579024384,"data_stored":1579024384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1984102400,"data_stored":1984102400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":785715200,"data_stored":785711128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1169883136,"data_stored":1169883136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2405302272,"data_stored":2405302272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2403663872,"data_stored":2403663872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1967263744,"data_stored":1967259672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:38:06.595 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:38:06.663 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:38:06.722 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:06.762 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:38:06.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:06.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:38:06.874 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:38:06.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:06 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:38:06.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:06 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:38:06.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:06 smithi115 ceph-mon[72476]: osdmap e1024: 12 total, 12 up, 12 in 2024-03-20T09:38:06.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:06 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1803972026' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:38:06.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:06 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:38:06.907 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:06.931 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:38:06.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:06.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:38:06.959 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:38:07.035 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:38:07.063 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:07.086 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:38:07.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:07.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:38:07.191 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:38:07.220 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:07.242 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:38:07.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:07.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:38:07.346 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:38:07.399 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:07.429 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:38:07.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:07.441 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:38:07.541 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:38:07.596 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:07.629 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:38:07.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:07.635 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:38:07.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:07 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:38:07.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:07 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:38:07.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:07 smithi115 ceph-mon[72476]: osdmap e1025: 12 total, 12 up, 12 in 2024-03-20T09:38:07.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:07 smithi115 ceph-mon[72476]: pgmap v2470: 47 pgs: 47 active+clean; 5.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 948 op/s 2024-03-20T09:38:07.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:07 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:38:07.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:07 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:38:07.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:07 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:38:07.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:07 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:38:07.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:07 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:38:07.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:07 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:38:07.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:07 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:38:07.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:07 smithi122 ceph-mon[73495]: osdmap e1025: 12 total, 12 up, 12 in 2024-03-20T09:38:07.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:07 smithi122 ceph-mon[73495]: pgmap v2470: 47 pgs: 47 active+clean; 5.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 948 op/s 2024-03-20T09:38:07.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:07 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:38:07.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:07 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:38:07.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:07 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:38:07.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:07 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:38:07.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:07 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:38:07.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:07 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:38:07.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:07 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:38:07.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:07 smithi026 ceph-mon[68309]: osdmap e1025: 12 total, 12 up, 12 in 2024-03-20T09:38:07.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:07 smithi026 ceph-mon[68309]: pgmap v2470: 47 pgs: 47 active+clean; 5.7 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 948 op/s 2024-03-20T09:38:07.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:07 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:38:07.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:07 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:38:07.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:07 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:38:07.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:07 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:38:07.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:07 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:38:07.736 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:38:07.769 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:07.789 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:38:07.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:07.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:38:07.893 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:38:07.928 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:07.946 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:38:07.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:07.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:38:08.051 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:38:08.105 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:08.211 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:38:08.219 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:08.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:38:08.239 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-20T09:38:08.321 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:38:08.384 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:08.710 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:38:09.140 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:08 smithi026 ceph-mon[68309]: pgmap v2471: 47 pgs: 47 active+clean; 5.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 793 op/s 2024-03-20T09:38:09.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:08 smithi115 ceph-mon[72476]: pgmap v2471: 47 pgs: 47 active+clean; 5.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 793 op/s 2024-03-20T09:38:09.159 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:38:09.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:09.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:38:09.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:08 smithi122 ceph-mon[73495]: pgmap v2471: 47 pgs: 47 active+clean; 5.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 793 op/s 2024-03-20T09:38:09.267 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:38:09.329 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:09.361 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:38:09.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:09.367 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:38:09.468 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:38:09.497 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:09.520 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:38:09.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:09.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:38:09.528 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:38:09.627 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:38:09.684 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:09.718 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:38:09.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:09.723 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:38:09.824 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:38:09.860 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:09.989 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:38:09.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:09.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:38:10.095 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:38:10.130 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:10.150 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:38:10.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:10.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:38:10.254 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:38:10.288 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:10.869 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:38:10.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:10.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:38:10.906 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:38:10.939 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:38:10.969 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:38:10.974 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:38:10.996 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:38:11.025 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:38:11.035 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:11.050 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:38:11.076 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:38:11.077 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:38:11.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:11.081 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:38:11.101 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:38:11.137 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:38:11.174 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:38:11.182 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:38:11.205 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:38:11.212 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:11.234 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:38:11.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:11.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:38:11.241 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:38:11.248 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:38:11.248 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:38:11.248 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:38:11.248 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:38:11.248 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:38:11.248 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:38:11.248 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:38:11.249 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:38:11.249 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:38:11.249 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:38:11.249 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:38:11.249 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:38:11.339 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:38:11.369 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:11.392 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:38:11.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:11.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:38:11.496 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:38:11.549 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:11.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:11 smithi115 ceph-mon[72476]: pgmap v2472: 47 pgs: 47 active+clean; 5.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 805 op/s 2024-03-20T09:38:11.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:11 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3868230863' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:38:11.895 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:11 smithi026 ceph-mon[68309]: pgmap v2472: 47 pgs: 47 active+clean; 5.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 805 op/s 2024-03-20T09:38:11.896 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:11 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3868230863' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:38:11.935 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:11 smithi122 ceph-mon[73495]: pgmap v2472: 47 pgs: 47 active+clean; 5.5 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 805 op/s 2024-03-20T09:38:11.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:11 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3868230863' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:38:12.764 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:38:13.655 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:13 smithi026 ceph-mon[68309]: pgmap v2473: 47 pgs: 47 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 878 op/s 2024-03-20T09:38:13.840 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:38:13.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:13.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:38:13.845 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:38:13.935 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:13 smithi122 ceph-mon[73495]: pgmap v2473: 47 pgs: 47 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 878 op/s 2024-03-20T09:38:13.945 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:38:13.970 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:13 smithi115 ceph-mon[72476]: pgmap v2473: 47 pgs: 47 active+clean; 5.4 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 878 op/s 2024-03-20T09:38:13.972 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:13.996 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:38:14.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:14.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:38:14.073 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:38:14.082 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:38:14.102 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:38:14.171 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:14.206 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:38:14.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:14.211 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:38:14.312 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:38:14.349 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:14.444 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:38:15.662 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:15 smithi026 ceph-mon[68309]: from='client.17145 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:38:15.662 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:15 smithi026 ceph-mon[68309]: pgmap v2474: 47 pgs: 47 active+clean; 5.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 826 op/s 2024-03-20T09:38:15.810 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:38:15.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:15.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:38:15.917 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:38:15.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:15 smithi122 ceph-mon[73495]: from='client.17145 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:38:15.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:15 smithi122 ceph-mon[73495]: pgmap v2474: 47 pgs: 47 active+clean; 5.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 826 op/s 2024-03-20T09:38:15.943 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:15 smithi115 ceph-mon[72476]: from='client.17145 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:38:15.943 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:15 smithi115 ceph-mon[72476]: pgmap v2474: 47 pgs: 47 active+clean; 5.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 826 op/s 2024-03-20T09:38:15.945 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:15.969 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:38:15.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:15.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:38:16.074 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:38:16.125 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:16.172 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:38:16.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:16.181 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:38:16.282 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:38:16.320 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:16.348 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:38:16.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:16.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:38:16.457 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:38:16.485 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:16.506 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:38:16.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:16.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:38:16.613 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:38:16.662 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:16.694 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:38:16.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:16.701 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:38:16.802 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:38:16.849 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:16.886 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:38:16.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:16.889 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:38:16.935 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":2473,"stamp":"2024-03-20T09:38:12.587708+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5782766136,"num_objects":88232,"num_object_clones":0,"num_object_copies":264696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88232,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158374,"num_write_kb":7891265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162914,"ondisk_log_size":162914,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":26149756,"kb_used_data":17069060,"kb_used_omap":301,"kb_used_meta":9079762,"kb_avail":1098792068,"statfs":{"total":1151940427776,"available":1125163077632,"internally_reserved":0,"allocated":17478717440,"data_stored":17475002504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":308246,"internal_metadata":9297677290},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-461307904,"num_objects":-7039,"num_object_clones":0,"num_object_copies":-21117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7039,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.008723"},"pg_stats":[{"pgid":"1.1e","version":"20'137","reported_seq":1508,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:06.424145+0000","last_change":"2024-03-20T09:37:37.183464+0000","last_active":"2024-03-20T09:38:06.424145+0000","last_peered":"2024-03-20T09:38:06.424145+0000","last_clean":"2024-03-20T09:38:06.424145+0000","last_became_active":"2024-03-20T09:13:20.071655+0000","last_became_peered":"2024-03-20T09:13:20.071655+0000","last_unstale":"2024-03-20T09:38:06.424145+0000","last_undegraded":"2024-03-20T09:38:06.424145+0000","last_fullsized":"2024-03-20T09:38:06.424145+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:37.183343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:37:37.183343+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:48.745062+0000","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'137","reported_seq":593,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:06.424054+0000","last_change":"2024-03-20T09:37:54.221807+0000","last_active":"2024-03-20T09:38:06.424054+0000","last_peered":"2024-03-20T09:38:06.424054+0000","last_clean":"2024-03-20T09:38:06.424054+0000","last_became_active":"2024-03-20T09:31:07.425574+0000","last_became_peered":"2024-03-20T09:31:07.425574+0000","last_unstale":"2024-03-20T09:38:06.424054+0000","last_undegraded":"2024-03-20T09:38:06.424054+0000","last_fullsized":"2024-03-20T09:38:06.424054+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:54.221681+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:29:02.585165+0000","last_clean_scrub_stamp":"2024-03-20T09:37:54.221681+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:13.870776+0000","scrub_duration":216,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2],"acting":[11,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"20'137","reported_seq":2197,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:06.418311+0000","last_change":"2024-03-20T09:37:56.375223+0000","last_active":"2024-03-20T09:38:06.418311+0000","last_peered":"2024-03-20T09:38:06.418311+0000","last_clean":"2024-03-20T09:38:06.418311+0000","last_became_active":"2024-03-20T09:35:03.815532+0000","last_became_peered":"2024-03-20T09:35:03.815532+0000","last_unstale":"2024-03-20T09:38:06.418311+0000","last_undegraded":"2024-03-20T09:38:06.418311+0000","last_fullsized":"2024-03-20T09:38:06.418311+0000","mapping_epoch":950,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":951,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:56.375139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:37:56.375139+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:15.044311+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.19","version":"20'137","reported_seq":1372,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:06.428118+0000","last_change":"2024-03-20T09:37:59.476380+0000","last_active":"2024-03-20T09:38:06.428118+0000","last_peered":"2024-03-20T09:38:06.428118+0000","last_clean":"2024-03-20T09:38:06.428118+0000","last_became_active":"2024-03-20T09:35:19.819786+0000","last_became_peered":"2024-03-20T09:35:19.819786+0000","last_unstale":"2024-03-20T09:38:06.428118+0000","last_undegraded":"2024-03-20T09:38:06.428118+0000","last_fullsized":"2024-03-20T09:38:06.428118+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:59.476249+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:30:56.201554+0000","last_clean_scrub_stamp":"2024-03-20T09:37:59.476249+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:23.603620+0000","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"20'137","reported_seq":1654,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:06.427354+0000","last_change":"2024-03-20T09:36:53.714268+0000","last_active":"2024-03-20T09:38:06.427354+0000","last_peered":"2024-03-20T09:38:06.427354+0000","last_clean":"2024-03-20T09:38:06.427354+0000","last_became_active":"2024-03-20T09:10:35.277481+0000","last_became_peered":"2024-03-20T09:10:35.277481+0000","last_unstale":"2024-03-20T09:38:06.427354+0000","last_undegraded":"2024-03-20T09:38:06.427354+0000","last_fullsized":"2024-03-20T09:38:06.427354+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:36:53.714149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:36:53.714149+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:08.648756+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1b","version":"20'137","reported_seq":2017,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:06.418562+0000","last_change":"2024-03-20T09:36:54.315917+0000","last_active":"2024-03-20T09:38:06.418562+0000","last_peered":"2024-03-20T09:38:06.418562+0000","last_clean":"2024-03-20T09:38:06.418562+0000","last_became_active":"2024-03-20T09:24:16.445287+0000","last_became_peered":"2024-03-20T09:24:16.445287+0000","last_unstale":"2024-03-20T09:38:06.418562+0000","last_undegraded":"2024-03-20T09:38:06.418562+0000","last_fullsized":"2024-03-20T09:38:06.418562+0000","mapping_epoch":718,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:36:54.315794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:36:54.315794+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:20.780820+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.1","version":"1025'12685","reported_seq":13055,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:08.384821+0000","last_change":"2024-03-20T09:37:38.850438+0000","last_active":"2024-03-20T09:38:08.384821+0000","last_peered":"2024-03-20T09:38:08.384821+0000","last_clean":"2024-03-20T09:38:08.384821+0000","last_became_active":"2024-03-20T09:35:51.943955+0000","last_became_peered":"2024-03-20T09:35:51.943955+0000","last_unstale":"2024-03-20T09:38:08.384821+0000","last_undegraded":"2024-03-20T09:38:08.384821+0000","last_fullsized":"2024-03-20T09:38:08.384821+0000","mapping_epoch":974,"log_start":"985'2600","ondisk_log_start":"985'2600","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1016'11148","last_scrub_stamp":"2024-03-20T09:37:38.850275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:38.850275+0000","objects_scrubbed":7129,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:47.172912+0000","scrub_duration":2090,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363266048,"num_objects":5543,"num_object_clones":0,"num_object_copies":16629,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9793,"num_write_kb":490752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,2],"acting":[9,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.8","version":"20'137","reported_seq":789,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:06.433504+0000","last_change":"2024-03-20T09:37:01.184325+0000","last_active":"2024-03-20T09:38:06.433504+0000","last_peered":"2024-03-20T09:38:06.433504+0000","last_clean":"2024-03-20T09:38:06.433504+0000","last_became_active":"2024-03-20T09:35:28.000367+0000","last_became_peered":"2024-03-20T09:35:28.000367+0000","last_unstale":"2024-03-20T09:38:06.433504+0000","last_undegraded":"2024-03-20T09:38:06.433504+0000","last_fullsized":"2024-03-20T09:38:06.433504+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":966,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:01.184206+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:37:01.184206+0000","last_clean_scrub_stamp":"2024-03-20T09:37:01.184206+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:17.102319+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"1025'9809","reported_seq":10001,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:09.842359+0000","last_change":"2024-03-20T09:37:41.579524+0000","last_active":"2024-03-20T09:38:09.842359+0000","last_peered":"2024-03-20T09:38:09.842359+0000","last_clean":"2024-03-20T09:38:09.842359+0000","last_became_active":"2024-03-20T09:35:51.276462+0000","last_became_peered":"2024-03-20T09:35:51.276462+0000","last_unstale":"2024-03-20T09:38:09.842359+0000","last_undegraded":"2024-03-20T09:38:09.842359+0000","last_fullsized":"2024-03-20T09:38:09.842359+0000","mapping_epoch":974,"log_start":"0'0","ondisk_log_start":"0'0","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1017'8324","last_scrub_stamp":"2024-03-20T09:37:41.579322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:41.579322+0000","objects_scrubbed":7101,"log_size":9809,"log_dups_size":0,"ondisk_log_size":9809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:55.312205+0000","scrub_duration":1793,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365494272,"num_objects":5577,"num_object_clones":0,"num_object_copies":16731,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9809,"num_write_kb":492352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"20'137","reported_seq":2247,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:06.421605+0000","last_change":"2024-03-20T09:37:07.903188+0000","last_active":"2024-03-20T09:38:06.421605+0000","last_peered":"2024-03-20T09:38:06.421605+0000","last_clean":"2024-03-20T09:38:06.421605+0000","last_became_active":"2024-03-20T09:20:25.341750+0000","last_became_peered":"2024-03-20T09:20:25.341750+0000","last_unstale":"2024-03-20T09:38:06.421605+0000","last_undegraded":"2024-03-20T09:38:06.421605+0000","last_fullsized":"2024-03-20T09:38:06.421605+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:07.903066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:37:07.903066+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:33.906783+0000","scrub_duration":253,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,2],"acting":[9,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.3","version":"1025'9978","reported_seq":10167,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:10.646882+0000","last_change":"2024-03-20T09:37:50.941074+0000","last_active":"2024-03-20T09:38:10.646882+0000","last_peered":"2024-03-20T09:38:10.646882+0000","last_clean":"2024-03-20T09:38:10.646882+0000","last_became_active":"2024-03-20T09:35:51.674444+0000","last_became_peered":"2024-03-20T09:35:51.674444+0000","last_unstale":"2024-03-20T09:38:10.646882+0000","last_undegraded":"2024-03-20T09:38:10.646882+0000","last_fullsized":"2024-03-20T09:38:10.646882+0000","mapping_epoch":974,"log_start":"0'0","ondisk_log_start":"0'0","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1019'8929","last_scrub_stamp":"2024-03-20T09:37:50.940775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:50.940775+0000","objects_scrubbed":6673,"log_size":9978,"log_dups_size":0,"ondisk_log_size":9978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:18.254318+0000","scrub_duration":1914,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365690880,"num_objects":5580,"num_object_clones":0,"num_object_copies":16740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9978,"num_write_kb":497856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"20'137","reported_seq":936,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:06.422040+0000","last_change":"2024-03-20T09:37:38.749161+0000","last_active":"2024-03-20T09:38:06.422040+0000","last_peered":"2024-03-20T09:38:06.422040+0000","last_clean":"2024-03-20T09:38:06.422040+0000","last_became_active":"2024-03-20T09:24:16.444232+0000","last_became_peered":"2024-03-20T09:24:16.444232+0000","last_unstale":"2024-03-20T09:38:06.422040+0000","last_undegraded":"2024-03-20T09:38:06.422040+0000","last_fullsized":"2024-03-20T09:38:06.422040+0000","mapping_epoch":718,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:38.749086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:37:38.749086+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:01.052060+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.d","version":"1025'9923","reported_seq":10117,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:10.656223+0000","last_change":"2024-03-20T09:37:42.977733+0000","last_active":"2024-03-20T09:38:10.656223+0000","last_peered":"2024-03-20T09:38:10.656223+0000","last_clean":"2024-03-20T09:38:10.656223+0000","last_became_active":"2024-03-20T09:35:51.943827+0000","last_became_peered":"2024-03-20T09:35:51.943827+0000","last_unstale":"2024-03-20T09:38:10.656223+0000","last_undegraded":"2024-03-20T09:38:10.656223+0000","last_fullsized":"2024-03-20T09:38:10.656223+0000","mapping_epoch":974,"log_start":"0'0","ondisk_log_start":"0'0","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1017'8430","last_scrub_stamp":"2024-03-20T09:37:42.977486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:42.977486+0000","objects_scrubbed":7043,"log_size":9923,"log_dups_size":0,"ondisk_log_size":9923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:01.233557+0000","scrub_duration":1930,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360841240,"num_objects":5507,"num_object_clones":0,"num_object_copies":16521,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5507,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9923,"num_write_kb":493697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,2],"acting":[11,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"20'137","reported_seq":2176,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:06.424032+0000","last_change":"2024-03-20T09:37:00.843889+0000","last_active":"2024-03-20T09:38:06.424032+0000","last_peered":"2024-03-20T09:38:06.424032+0000","last_clean":"2024-03-20T09:38:06.424032+0000","last_became_active":"2024-03-20T09:12:14.306384+0000","last_became_peered":"2024-03-20T09:12:14.306384+0000","last_unstale":"2024-03-20T09:38:06.424032+0000","last_undegraded":"2024-03-20T09:38:06.424032+0000","last_fullsized":"2024-03-20T09:38:06.424032+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:00.843763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:37:00.843763+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:26.792435+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.c","version":"1025'9910","reported_seq":10103,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:08.354395+0000","last_change":"2024-03-20T09:37:39.960010+0000","last_active":"2024-03-20T09:38:08.354395+0000","last_peered":"2024-03-20T09:38:08.354395+0000","last_clean":"2024-03-20T09:38:08.354395+0000","last_became_active":"2024-03-20T09:35:51.278227+0000","last_became_peered":"2024-03-20T09:35:51.278227+0000","last_unstale":"2024-03-20T09:38:08.354395+0000","last_undegraded":"2024-03-20T09:38:08.354395+0000","last_fullsized":"2024-03-20T09:38:08.354395+0000","mapping_epoch":974,"log_start":"0'0","ondisk_log_start":"0'0","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1017'8325","last_scrub_stamp":"2024-03-20T09:37:39.959895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:39.959895+0000","objects_scrubbed":7210,"log_size":9910,"log_dups_size":0,"ondisk_log_size":9910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:50.065860+0000","scrub_duration":2195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364380160,"num_objects":5560,"num_object_clones":0,"num_object_copies":16680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9910,"num_write_kb":495040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.5","version":"20'137","reported_seq":836,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:06.432486+0000","last_change":"2024-03-20T09:36:47.476093+0000","last_active":"2024-03-20T09:38:06.432486+0000","last_peered":"2024-03-20T09:38:06.432486+0000","last_clean":"2024-03-20T09:38:06.432486+0000","last_became_active":"2024-03-20T09:35:19.820895+0000","last_became_peered":"2024-03-20T09:35:19.820895+0000","last_unstale":"2024-03-20T09:38:06.432486+0000","last_undegraded":"2024-03-20T09:38:06.432486+0000","last_fullsized":"2024-03-20T09:38:06.432486+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:36:47.475965+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:19:03.805185+0000","last_clean_scrub_stamp":"2024-03-20T09:36:47.475965+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:13.637180+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.f","version":"1025'9791","reported_seq":9979,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:10.574974+0000","last_change":"2024-03-20T09:37:50.253230+0000","last_active":"2024-03-20T09:38:10.574974+0000","last_peered":"2024-03-20T09:38:10.574974+0000","last_clean":"2024-03-20T09:38:10.574974+0000","last_became_active":"2024-03-20T09:35:51.943804+0000","last_became_peered":"2024-03-20T09:35:51.943804+0000","last_unstale":"2024-03-20T09:38:10.574974+0000","last_undegraded":"2024-03-20T09:38:10.574974+0000","last_fullsized":"2024-03-20T09:38:10.574974+0000","mapping_epoch":974,"log_start":"0'0","ondisk_log_start":"0'0","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1019'8690","last_scrub_stamp":"2024-03-20T09:37:50.253006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:50.253006+0000","objects_scrubbed":6576,"log_size":9791,"log_dups_size":0,"ondisk_log_size":9791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:15.232988+0000","scrub_duration":2230,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":354746368,"num_objects":5413,"num_object_clones":0,"num_object_copies":16239,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9791,"num_write_kb":486528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"20'137","reported_seq":1714,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:06.432853+0000","last_change":"2024-03-20T09:37:42.468637+0000","last_active":"2024-03-20T09:38:06.432853+0000","last_peered":"2024-03-20T09:38:06.432853+0000","last_clean":"2024-03-20T09:38:06.432853+0000","last_became_active":"2024-03-20T09:35:19.822321+0000","last_became_peered":"2024-03-20T09:35:19.822321+0000","last_unstale":"2024-03-20T09:38:06.432853+0000","last_undegraded":"2024-03-20T09:38:06.432853+0000","last_fullsized":"2024-03-20T09:38:06.432853+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:42.468502+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:24:31.549603+0000","last_clean_scrub_stamp":"2024-03-20T09:37:42.468502+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:02.303736+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.a","version":"1025'10000","reported_seq":10193,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:11.707710+0000","last_change":"2024-03-20T09:37:43.187842+0000","last_active":"2024-03-20T09:38:11.707710+0000","last_peered":"2024-03-20T09:38:11.707710+0000","last_clean":"2024-03-20T09:38:11.707710+0000","last_became_active":"2024-03-20T09:35:51.944099+0000","last_became_peered":"2024-03-20T09:35:51.944099+0000","last_unstale":"2024-03-20T09:38:11.707710+0000","last_undegraded":"2024-03-20T09:38:11.707710+0000","last_fullsized":"2024-03-20T09:38:11.707710+0000","mapping_epoch":974,"log_start":"0'0","ondisk_log_start":"0'0","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1017'8466","last_scrub_stamp":"2024-03-20T09:37:43.183079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:43.183079+0000","objects_scrubbed":7069,"log_size":10000,"log_dups_size":0,"ondisk_log_size":10000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:57.208328+0000","scrub_duration":1937,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360316928,"num_objects":5498,"num_object_clones":0,"num_object_copies":16494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10000,"num_write_kb":495936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"20'137","reported_seq":1241,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:06.415856+0000","last_change":"2024-03-20T09:36:54.845047+0000","last_active":"2024-03-20T09:38:06.415856+0000","last_peered":"2024-03-20T09:38:06.415856+0000","last_clean":"2024-03-20T09:38:06.415856+0000","last_became_active":"2024-03-20T09:31:07.424588+0000","last_became_peered":"2024-03-20T09:31:07.424588+0000","last_unstale":"2024-03-20T09:38:06.415856+0000","last_undegraded":"2024-03-20T09:38:06.415856+0000","last_fullsized":"2024-03-20T09:38:06.415856+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:36:54.844902+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:30:47.812125+0000","last_clean_scrub_stamp":"2024-03-20T09:36:54.844902+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:15.580612+0000","scrub_duration":211,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.9","version":"1025'9929","reported_seq":10111,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:08.723230+0000","last_change":"2024-03-20T09:37:03.599757+0000","last_active":"2024-03-20T09:38:08.723230+0000","last_peered":"2024-03-20T09:38:08.723230+0000","last_clean":"2024-03-20T09:38:08.723230+0000","last_became_active":"2024-03-20T09:35:51.274816+0000","last_became_peered":"2024-03-20T09:35:51.274816+0000","last_unstale":"2024-03-20T09:38:08.723230+0000","last_undegraded":"2024-03-20T09:38:08.723230+0000","last_fullsized":"2024-03-20T09:38:08.723230+0000","mapping_epoch":974,"log_start":"0'0","ondisk_log_start":"0'0","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1005'6201","last_scrub_stamp":"2024-03-20T09:37:03.599526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:03.599526+0000","objects_scrubbed":6170,"log_size":9929,"log_dups_size":0,"ondisk_log_size":9929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:29.249509+0000","scrub_duration":1194,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366673920,"num_objects":5595,"num_object_clones":0,"num_object_copies":16785,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9929,"num_write_kb":496768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":2155,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:06.423848+0000","last_change":"2024-03-20T09:38:02.890362+0000","last_active":"2024-03-20T09:38:06.423848+0000","last_peered":"2024-03-20T09:38:06.423848+0000","last_clean":"2024-03-20T09:38:06.423848+0000","last_became_active":"2024-03-20T09:12:14.305989+0000","last_became_peered":"2024-03-20T09:12:14.305989+0000","last_unstale":"2024-03-20T09:38:06.423848+0000","last_undegraded":"2024-03-20T09:38:06.423848+0000","last_fullsized":"2024-03-20T09:38:06.423848+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:38:02.890224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:38:02.890224+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:27.163539+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.b","version":"1025'10063","reported_seq":10256,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:11.712176+0000","last_change":"2024-03-20T09:37:38.084925+0000","last_active":"2024-03-20T09:38:11.712176+0000","last_peered":"2024-03-20T09:38:11.712176+0000","last_clean":"2024-03-20T09:38:11.712176+0000","last_became_active":"2024-03-20T09:35:51.942846+0000","last_became_peered":"2024-03-20T09:35:51.942846+0000","last_unstale":"2024-03-20T09:38:11.712176+0000","last_undegraded":"2024-03-20T09:38:11.712176+0000","last_fullsized":"2024-03-20T09:38:11.712176+0000","mapping_epoch":974,"log_start":"0'0","ondisk_log_start":"0'0","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1015'8221","last_scrub_stamp":"2024-03-20T09:37:38.084645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:38.084645+0000","objects_scrubbed":7285,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:52.604722+0000","scrub_duration":1768,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353959936,"num_objects":5401,"num_object_clones":0,"num_object_copies":16203,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10063,"num_write_kb":494848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"20'137","reported_seq":2412,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:06.426169+0000","last_change":"2024-03-20T09:36:50.854490+0000","last_active":"2024-03-20T09:38:06.426169+0000","last_peered":"2024-03-20T09:38:06.426169+0000","last_clean":"2024-03-20T09:38:06.426169+0000","last_became_active":"2024-03-20T09:19:54.492879+0000","last_became_peered":"2024-03-20T09:19:54.492879+0000","last_unstale":"2024-03-20T09:38:06.426169+0000","last_undegraded":"2024-03-20T09:38:06.426169+0000","last_fullsized":"2024-03-20T09:38:06.426169+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:36:50.854413+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:19.513511+0000","last_clean_scrub_stamp":"2024-03-20T09:36:50.854413+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:14.992458+0000","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"1025'9967","reported_seq":10160,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:09.978736+0000","last_change":"2024-03-20T09:37:38.178742+0000","last_active":"2024-03-20T09:38:09.978736+0000","last_peered":"2024-03-20T09:38:09.978736+0000","last_clean":"2024-03-20T09:38:09.978736+0000","last_became_active":"2024-03-20T09:35:51.276320+0000","last_became_peered":"2024-03-20T09:35:51.276320+0000","last_unstale":"2024-03-20T09:38:09.978736+0000","last_undegraded":"2024-03-20T09:38:09.978736+0000","last_fullsized":"2024-03-20T09:38:09.978736+0000","mapping_epoch":974,"log_start":"0'0","ondisk_log_start":"0'0","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1015'8253","last_scrub_stamp":"2024-03-20T09:37:38.178617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:38.178617+0000","objects_scrubbed":7305,"log_size":9967,"log_dups_size":0,"ondisk_log_size":9967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:59.551600+0000","scrub_duration":1541,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364576768,"num_objects":5563,"num_object_clones":0,"num_object_copies":16689,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9967,"num_write_kb":496960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"20'137","reported_seq":1595,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:06.415850+0000","last_change":"2024-03-20T09:37:45.841316+0000","last_active":"2024-03-20T09:38:06.415850+0000","last_peered":"2024-03-20T09:38:06.415850+0000","last_clean":"2024-03-20T09:38:06.415850+0000","last_became_active":"2024-03-20T09:15:34.225022+0000","last_became_peered":"2024-03-20T09:15:34.225022+0000","last_unstale":"2024-03-20T09:38:06.415850+0000","last_undegraded":"2024-03-20T09:38:06.415850+0000","last_fullsized":"2024-03-20T09:38:06.415850+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:45.841176+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:20:12.169658+0000","last_clean_scrub_stamp":"2024-03-20T09:37:45.841176+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:11.852244+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,5],"acting":[6,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"1025'9930","reported_seq":10113,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:11.926664+0000","last_change":"2024-03-20T09:37:04.086352+0000","last_active":"2024-03-20T09:38:11.926664+0000","last_peered":"2024-03-20T09:38:11.926664+0000","last_clean":"2024-03-20T09:38:11.926664+0000","last_became_active":"2024-03-20T09:35:51.943717+0000","last_became_peered":"2024-03-20T09:35:51.943717+0000","last_unstale":"2024-03-20T09:38:11.926664+0000","last_undegraded":"2024-03-20T09:38:11.926664+0000","last_fullsized":"2024-03-20T09:38:11.926664+0000","mapping_epoch":974,"log_start":"0'0","ondisk_log_start":"0'0","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1006'6187","last_scrub_stamp":"2024-03-20T09:37:04.086220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:04.086220+0000","objects_scrubbed":6142,"log_size":9930,"log_dups_size":0,"ondisk_log_size":9930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:13.782435+0000","scrub_duration":1249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352190464,"num_objects":5374,"num_object_clones":0,"num_object_copies":16122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9930,"num_write_kb":489728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"20'137","reported_seq":2143,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:06.416000+0000","last_change":"2024-03-20T09:37:45.012425+0000","last_active":"2024-03-20T09:38:06.416000+0000","last_peered":"2024-03-20T09:38:06.416000+0000","last_clean":"2024-03-20T09:38:06.416000+0000","last_became_active":"2024-03-20T09:35:19.820396+0000","last_became_peered":"2024-03-20T09:35:19.820396+0000","last_unstale":"2024-03-20T09:38:06.416000+0000","last_undegraded":"2024-03-20T09:38:06.416000+0000","last_fullsized":"2024-03-20T09:38:06.416000+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:45.012294+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:14:10.560890+0000","last_clean_scrub_stamp":"2024-03-20T09:37:45.012294+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:57.274595+0000","scrub_duration":222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2","version":"1025'9855","reported_seq":10048,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:08.734768+0000","last_change":"2024-03-20T09:37:39.223390+0000","last_active":"2024-03-20T09:38:08.734768+0000","last_peered":"2024-03-20T09:38:08.734768+0000","last_clean":"2024-03-20T09:38:08.734768+0000","last_became_active":"2024-03-20T09:35:51.942847+0000","last_became_peered":"2024-03-20T09:35:51.942847+0000","last_unstale":"2024-03-20T09:38:08.734768+0000","last_undegraded":"2024-03-20T09:38:08.734768+0000","last_fullsized":"2024-03-20T09:38:08.734768+0000","mapping_epoch":974,"log_start":"0'0","ondisk_log_start":"0'0","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1016'8306","last_scrub_stamp":"2024-03-20T09:37:39.223182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:39.223182+0000","objects_scrubbed":7252,"log_size":9855,"log_dups_size":0,"ondisk_log_size":9855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:51.271166+0000","scrub_duration":1777,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371261440,"num_objects":5665,"num_object_clones":0,"num_object_copies":16995,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9855,"num_write_kb":496640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,4],"acting":[10,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.b","version":"20'137","reported_seq":1960,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:06.423858+0000","last_change":"2024-03-20T09:37:06.065195+0000","last_active":"2024-03-20T09:38:06.423858+0000","last_peered":"2024-03-20T09:38:06.423858+0000","last_clean":"2024-03-20T09:38:06.423858+0000","last_became_active":"2024-03-20T09:13:20.070596+0000","last_became_peered":"2024-03-20T09:13:20.070596+0000","last_unstale":"2024-03-20T09:38:06.423858+0000","last_undegraded":"2024-03-20T09:38:06.423858+0000","last_fullsized":"2024-03-20T09:38:06.423858+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:06.065105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:37:06.065105+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:35.637879+0000","scrub_duration":316,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.7","version":"1025'9924","reported_seq":10111,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:10.646951+0000","last_change":"2024-03-20T09:37:51.929007+0000","last_active":"2024-03-20T09:38:10.646951+0000","last_peered":"2024-03-20T09:38:10.646951+0000","last_clean":"2024-03-20T09:38:10.646951+0000","last_became_active":"2024-03-20T09:35:51.673870+0000","last_became_peered":"2024-03-20T09:35:51.673870+0000","last_unstale":"2024-03-20T09:38:10.646951+0000","last_undegraded":"2024-03-20T09:38:10.646951+0000","last_fullsized":"2024-03-20T09:38:10.646951+0000","mapping_epoch":974,"log_start":"0'0","ondisk_log_start":"0'0","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1020'8885","last_scrub_stamp":"2024-03-20T09:37:51.928857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:51.928857+0000","objects_scrubbed":6515,"log_size":9924,"log_dups_size":0,"ondisk_log_size":9924,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:10.833584+0000","scrub_duration":1835,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":355991552,"num_objects":5432,"num_object_clones":0,"num_object_copies":16296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9924,"num_write_kb":491392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.e","version":"20'137","reported_seq":1400,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:06.423444+0000","last_change":"2024-03-20T09:37:42.890342+0000","last_active":"2024-03-20T09:38:06.423444+0000","last_peered":"2024-03-20T09:38:06.423444+0000","last_clean":"2024-03-20T09:38:06.423444+0000","last_became_active":"2024-03-20T09:19:54.506134+0000","last_became_peered":"2024-03-20T09:19:54.506134+0000","last_unstale":"2024-03-20T09:38:06.423444+0000","last_undegraded":"2024-03-20T09:38:06.423444+0000","last_fullsized":"2024-03-20T09:38:06.423444+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:42.890257+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:29:53.362224+0000","last_clean_scrub_stamp":"2024-03-20T09:37:42.890257+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:47.227463+0000","scrub_duration":157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"1025'9999","reported_seq":10190,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:10.624387+0000","last_change":"2024-03-20T09:37:44.856776+0000","last_active":"2024-03-20T09:38:10.624387+0000","last_peered":"2024-03-20T09:38:10.624387+0000","last_clean":"2024-03-20T09:38:10.624387+0000","last_became_active":"2024-03-20T09:35:51.312260+0000","last_became_peered":"2024-03-20T09:35:51.312260+0000","last_unstale":"2024-03-20T09:38:10.624387+0000","last_undegraded":"2024-03-20T09:38:10.624387+0000","last_fullsized":"2024-03-20T09:38:10.624387+0000","mapping_epoch":974,"log_start":"0'0","ondisk_log_start":"0'0","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1018'8599","last_scrub_stamp":"2024-03-20T09:37:44.856610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:44.856610+0000","objects_scrubbed":6942,"log_size":9999,"log_dups_size":0,"ondisk_log_size":9999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:52.896041+0000","scrub_duration":1869,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":359596032,"num_objects":5487,"num_object_clones":0,"num_object_copies":16461,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9999,"num_write_kb":495552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,7],"acting":[11,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.f","version":"20'137","reported_seq":2163,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:06.415997+0000","last_change":"2024-03-20T09:38:01.026424+0000","last_active":"2024-03-20T09:38:06.415997+0000","last_peered":"2024-03-20T09:38:06.415997+0000","last_clean":"2024-03-20T09:38:06.415997+0000","last_became_active":"2024-03-20T09:19:54.524397+0000","last_became_peered":"2024-03-20T09:19:54.524397+0000","last_unstale":"2024-03-20T09:38:06.415997+0000","last_undegraded":"2024-03-20T09:38:06.415997+0000","last_fullsized":"2024-03-20T09:38:06.415997+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:38:01.026288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:38:01.026288+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:09.847998+0000","scrub_duration":214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.5","version":"1025'9759","reported_seq":9938,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:08.708940+0000","last_change":"2024-03-20T09:36:55.745524+0000","last_active":"2024-03-20T09:38:08.708940+0000","last_peered":"2024-03-20T09:38:08.708940+0000","last_clean":"2024-03-20T09:38:08.708940+0000","last_became_active":"2024-03-20T09:35:51.942869+0000","last_became_peered":"2024-03-20T09:35:51.942869+0000","last_unstale":"2024-03-20T09:38:08.708940+0000","last_undegraded":"2024-03-20T09:38:08.708940+0000","last_fullsized":"2024-03-20T09:38:08.708940+0000","mapping_epoch":974,"log_start":"0'0","ondisk_log_start":"0'0","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1001'5720","last_scrub_stamp":"2024-03-20T09:36:55.745369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:36:55.745369+0000","objects_scrubbed":5679,"log_size":9759,"log_dups_size":0,"ondisk_log_size":9759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:12.875325+0000","scrub_duration":1369,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370081792,"num_objects":5647,"num_object_clones":0,"num_object_copies":16941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9759,"num_write_kb":492992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,1],"acting":[10,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.c","version":"570'138","reported_seq":2140,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:06.431239+0000","last_change":"2024-03-20T09:37:40.922108+0000","last_active":"2024-03-20T09:38:06.431239+0000","last_peered":"2024-03-20T09:38:06.431239+0000","last_clean":"2024-03-20T09:38:06.431239+0000","last_became_active":"2024-03-20T09:13:27.087686+0000","last_became_peered":"2024-03-20T09:13:27.087686+0000","last_unstale":"2024-03-20T09:38:06.431239+0000","last_undegraded":"2024-03-20T09:38:06.431239+0000","last_fullsized":"2024-03-20T09:38:06.431239+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"570'138","last_scrub_stamp":"2024-03-20T09:37:40.921974+0000","last_deep_scrub":"570'138","last_deep_scrub_stamp":"2024-03-20T09:16:13.725035+0000","last_clean_scrub_stamp":"2024-03-20T09:37:40.921974+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:43.602278+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"1025'9744","reported_seq":9927,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:09.995979+0000","last_change":"2024-03-20T09:37:04.918258+0000","last_active":"2024-03-20T09:38:09.995979+0000","last_peered":"2024-03-20T09:38:09.995979+0000","last_clean":"2024-03-20T09:38:09.995979+0000","last_became_active":"2024-03-20T09:35:51.277801+0000","last_became_peered":"2024-03-20T09:35:51.277801+0000","last_unstale":"2024-03-20T09:38:09.995979+0000","last_undegraded":"2024-03-20T09:38:09.995979+0000","last_fullsized":"2024-03-20T09:38:09.995979+0000","mapping_epoch":974,"log_start":"0'0","ondisk_log_start":"0'0","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1006'6159","last_scrub_stamp":"2024-03-20T09:37:04.918103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:04.918103+0000","objects_scrubbed":6123,"log_size":9744,"log_dups_size":0,"ondisk_log_size":9744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:14.259459+0000","scrub_duration":1242,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353107968,"num_objects":5388,"num_object_clones":0,"num_object_copies":16164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9744,"num_write_kb":484224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,10],"acting":[5,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"20'137","reported_seq":1714,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:06.423401+0000","last_change":"2024-03-20T09:38:01.008767+0000","last_active":"2024-03-20T09:38:06.423401+0000","last_peered":"2024-03-20T09:38:06.423401+0000","last_clean":"2024-03-20T09:38:06.423401+0000","last_became_active":"2024-03-20T09:19:54.502529+0000","last_became_peered":"2024-03-20T09:19:54.502529+0000","last_unstale":"2024-03-20T09:38:06.423401+0000","last_undegraded":"2024-03-20T09:38:06.423401+0000","last_fullsized":"2024-03-20T09:38:06.423401+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:38:01.008634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:38:01.008634+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:09.649906+0000","scrub_duration":168,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"20'137","reported_seq":1994,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:06.427436+0000","last_change":"2024-03-20T09:37:07.729241+0000","last_active":"2024-03-20T09:38:06.427436+0000","last_peered":"2024-03-20T09:38:06.427436+0000","last_clean":"2024-03-20T09:38:06.427436+0000","last_became_active":"2024-03-20T09:10:34.096758+0000","last_became_peered":"2024-03-20T09:10:34.096758+0000","last_unstale":"2024-03-20T09:38:06.427436+0000","last_undegraded":"2024-03-20T09:38:06.427436+0000","last_fullsized":"2024-03-20T09:38:06.427436+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:07.729109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:37:07.729109+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:14.194912+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"20'137","reported_seq":756,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:06.424009+0000","last_change":"2024-03-20T09:37:52.360050+0000","last_active":"2024-03-20T09:38:06.424009+0000","last_peered":"2024-03-20T09:38:06.424009+0000","last_clean":"2024-03-20T09:38:06.424009+0000","last_became_active":"2024-03-20T09:35:19.822242+0000","last_became_peered":"2024-03-20T09:35:19.822242+0000","last_unstale":"2024-03-20T09:38:06.424009+0000","last_undegraded":"2024-03-20T09:38:06.424009+0000","last_fullsized":"2024-03-20T09:38:06.424009+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:52.359932+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:37:52.359932+0000","last_clean_scrub_stamp":"2024-03-20T09:37:52.359932+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:00.922956+0000","scrub_duration":315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.10","version":"20'137","reported_seq":1557,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:06.427454+0000","last_change":"2024-03-20T09:37:38.621260+0000","last_active":"2024-03-20T09:38:06.427454+0000","last_peered":"2024-03-20T09:38:06.427454+0000","last_clean":"2024-03-20T09:38:06.427454+0000","last_became_active":"2024-03-20T09:35:19.820595+0000","last_became_peered":"2024-03-20T09:35:19.820595+0000","last_unstale":"2024-03-20T09:38:06.427454+0000","last_undegraded":"2024-03-20T09:38:06.427454+0000","last_fullsized":"2024-03-20T09:38:06.427454+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:38.621168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:37:38.621168+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:43.836018+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"20'137","reported_seq":1938,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:06.426183+0000","last_change":"2024-03-20T09:36:51.839369+0000","last_active":"2024-03-20T09:38:06.426183+0000","last_peered":"2024-03-20T09:38:06.426183+0000","last_clean":"2024-03-20T09:38:06.426183+0000","last_became_active":"2024-03-20T09:20:25.342982+0000","last_became_peered":"2024-03-20T09:20:25.342982+0000","last_unstale":"2024-03-20T09:38:06.426183+0000","last_undegraded":"2024-03-20T09:38:06.426183+0000","last_fullsized":"2024-03-20T09:38:06.426183+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:36:51.839234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:36:51.839234+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:19.300010+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"20'137","reported_seq":663,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:06.426310+0000","last_change":"2024-03-20T09:36:54.828126+0000","last_active":"2024-03-20T09:38:06.426310+0000","last_peered":"2024-03-20T09:38:06.426310+0000","last_clean":"2024-03-20T09:38:06.426310+0000","last_became_active":"2024-03-20T09:31:07.424851+0000","last_became_peered":"2024-03-20T09:31:07.424851+0000","last_unstale":"2024-03-20T09:38:06.426310+0000","last_undegraded":"2024-03-20T09:38:06.426310+0000","last_fullsized":"2024-03-20T09:38:06.426310+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:36:54.827943+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:35:27.264203+0000","last_clean_scrub_stamp":"2024-03-20T09:36:54.827943+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:19.315177+0000","scrub_duration":174,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.17","version":"20'137","reported_seq":2135,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:06.418423+0000","last_change":"2024-03-20T09:38:01.420016+0000","last_active":"2024-03-20T09:38:06.418423+0000","last_peered":"2024-03-20T09:38:06.418423+0000","last_clean":"2024-03-20T09:38:06.418423+0000","last_became_active":"2024-03-20T09:35:19.822429+0000","last_became_peered":"2024-03-20T09:35:19.822429+0000","last_unstale":"2024-03-20T09:38:06.418423+0000","last_undegraded":"2024-03-20T09:38:06.418423+0000","last_fullsized":"2024-03-20T09:38:06.418423+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:38:01.419884+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:38:01.419884+0000","last_clean_scrub_stamp":"2024-03-20T09:38:01.419884+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:22.126398+0000","scrub_duration":169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"20'137","reported_seq":2167,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:06.435912+0000","last_change":"2024-03-20T09:36:56.204036+0000","last_active":"2024-03-20T09:38:06.435912+0000","last_peered":"2024-03-20T09:38:06.435912+0000","last_clean":"2024-03-20T09:38:06.435912+0000","last_became_active":"2024-03-20T09:20:25.349082+0000","last_became_peered":"2024-03-20T09:20:25.349082+0000","last_unstale":"2024-03-20T09:38:06.435912+0000","last_undegraded":"2024-03-20T09:38:06.435912+0000","last_fullsized":"2024-03-20T09:38:06.435912+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:36:56.203907+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:36:56.203907+0000","last_clean_scrub_stamp":"2024-03-20T09:36:56.203907+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:11.989863+0000","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"20'137","reported_seq":1447,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:06.424945+0000","last_change":"2024-03-20T09:37:36.549435+0000","last_active":"2024-03-20T09:38:06.424945+0000","last_peered":"2024-03-20T09:38:06.424945+0000","last_clean":"2024-03-20T09:38:06.424945+0000","last_became_active":"2024-03-20T09:12:14.307581+0000","last_became_peered":"2024-03-20T09:12:14.307581+0000","last_unstale":"2024-03-20T09:38:06.424945+0000","last_undegraded":"2024-03-20T09:38:06.424945+0000","last_fullsized":"2024-03-20T09:38:06.424945+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:36.549355+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:34:48.079272+0000","last_clean_scrub_stamp":"2024-03-20T09:37:36.549355+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:04.698728+0000","scrub_duration":206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"20'137","reported_seq":1713,"reported_epoch":1025,"state":"active+clean","last_fresh":"2024-03-20T09:38:06.423418+0000","last_change":"2024-03-20T09:38:01.843191+0000","last_active":"2024-03-20T09:38:06.423418+0000","last_peered":"2024-03-20T09:38:06.423418+0000","last_clean":"2024-03-20T09:38:06.423418+0000","last_became_active":"2024-03-20T09:19:54.502391+0000","last_became_peered":"2024-03-20T09:19:54.502391+0000","last_unstale":"2024-03-20T09:38:06.423418+0000","last_undegraded":"2024-03-20T09:38:06.423418+0000","last_fullsized":"2024-03-20T09:38:06.423418+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:38:01.843093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:38:01.843093+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:31.633431+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":5782175768,"num_objects":88230,"num_object_clones":0,"num_object_copies":264690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88230,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158374,"num_write_kb":7891265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17386975232,"data_stored":17386963016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158666,"ondisk_log_size":158666,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4248,"ondisk_log_size":4248,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860922,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2723668,"kb_used_data":1767884,"kb_used_omap":1,"kb_used_meta":955774,"kb_avail":91021484,"statfs":{"total":95995035648,"available":93205999616,"internally_reserved":0,"allocated":1810313216,"data_stored":1810001472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":978713028},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":861,"seq":3697966841947,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2303376,"kb_used_data":2148744,"kb_used_omap":103,"kb_used_meta":154456,"kb_avail":91441776,"statfs":{"total":95995035648,"available":93636378624,"internally_reserved":0,"allocated":2200313856,"data_stored":2200006184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":106169,"internal_metadata":158163271},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":80,"seq":343597384199,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2717780,"kb_used_data":2158920,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":91027372,"statfs":{"total":95995035648,"available":93212028928,"internally_reserved":0,"allocated":2210734080,"data_stored":2210426408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":572127680},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":70,"seq":300647711244,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1629648,"kb_used_data":1048648,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":92115504,"statfs":{"total":95995035648,"available":94326276096,"internally_reserved":0,"allocated":1073815552,"data_stored":1073507880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":594868678},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":60,"seq":257698038288,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1607004,"kb_used_data":705364,"kb_used_omap":1,"kb_used_meta":901630,"kb_avail":92138148,"statfs":{"total":95995035648,"available":94349463552,"internally_reserved":0,"allocated":722292736,"data_stored":721977440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":923269578},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332629,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":704220,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93040932,"statfs":{"total":95995035648,"available":95273914368,"internally_reserved":0,"allocated":1122304,"data_stored":814632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":719976902},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575407,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3101396,"kb_used_data":2141384,"kb_used_omap":1,"kb_used_meta":959934,"kb_avail":90643756,"statfs":{"total":95995035648,"available":92819206144,"internally_reserved":0,"allocated":2192777216,"data_stored":2192469544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":982972870},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738926,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2074768,"kb_used_data":1046792,"kb_used_omap":1,"kb_used_meta":1027966,"kb_avail":91670384,"statfs":{"total":95995035648,"available":93870473216,"internally_reserved":0,"allocated":1071915008,"data_stored":1071607336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1052637637},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411879,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2583008,"kb_used_data":1793812,"kb_used_omap":1,"kb_used_meta":789182,"kb_avail":91162144,"statfs":{"total":95995035648,"available":93350035456,"internally_reserved":0,"allocated":1836863488,"data_stored":1836548192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":808122821},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150244,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2035344,"kb_used_data":1065672,"kb_used_omap":1,"kb_used_meta":969662,"kb_avail":91709808,"statfs":{"total":95995035648,"available":93910843392,"internally_reserved":0,"allocated":1091248128,"data_stored":1090940456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":992934343},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438954019,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2166288,"kb_used_data":1417160,"kb_used_omap":1,"kb_used_meta":748926,"kb_avail":91578864,"statfs":{"total":95995035648,"available":93776756736,"internally_reserved":0,"allocated":1451171840,"data_stored":1450864168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":766900677},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":443,"seq":1902670512460,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2503256,"kb_used_data":1773584,"kb_used_omap":181,"kb_used_meta":729482,"kb_avail":91241896,"statfs":{"total":95995035648,"available":93431701504,"internally_reserved":0,"allocated":1816150016,"data_stored":1815838792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":186131,"internal_metadata":746989805},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1067253760,"data_stored":1067253760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2172780544,"data_stored":2172780544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1827868672,"data_stored":1827864600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1090060288,"data_stored":1090060288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1441792000,"data_stored":1441792000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1812791296,"data_stored":1812791296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":718934016,"data_stored":718929944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1066795008,"data_stored":1066795008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2198798336,"data_stored":2198798336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2193227776,"data_stored":2193227776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1796673536,"data_stored":1796669464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:38:16.937 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-20T09:38:16.937 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:38:16.938 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-03-20T09:38:16.938 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-20T09:38:16.938 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-20T09:38:16.938 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-03-20T09:38:16.989 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:38:16.989 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:38:16.989 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:38:16.989 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26917.0:163611 8.d e1025/974 8:b151ab35:::benchmark_data_smithi122_197775_object40307:head v 1025'10199, mlcod=1025'10196)", 2024-03-20T09:38:16.989 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:38:16.367941+0000", 2024-03-20T09:38:16.989 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.00080051799999999995, 2024-03-20T09:38:16.989 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000829848, 2024-03-20T09:38:16.989 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:38:16.989 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:38:16.990 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:38:16.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:38:16.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:38:16.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:16.367941+0000", 2024-03-20T09:38:16.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:38:16.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:38:16.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:38:16.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:38:16.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:16.367941+0000", 2024-03-20T09:38:16.990 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:38:16.990 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:38:16.990 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:38:16.990 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:38:16.990 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:16.367942+0000", 2024-03-20T09:38:16.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5700000000000001e-07 2024-03-20T09:38:16.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:38:16.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:38:16.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:38:16.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:16.367951+0000", 2024-03-20T09:38:16.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9260000000000006e-06 2024-03-20T09:38:16.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:38:16.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:38:16.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:38:16.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:16.367963+0000", 2024-03-20T09:38:16.991 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2605999999999999e-05 2024-03-20T09:38:16.991 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:38:16.991 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:38:16.991 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:38:16.991 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:16.367970+0000", 2024-03-20T09:38:16.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9959999999999996e-06 2024-03-20T09:38:16.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:38:16.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:38:16.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:38:16.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:16.368025+0000", 2024-03-20T09:38:16.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4336999999999998e-05 2024-03-20T09:38:16.992 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:38:16.992 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:38:16.992 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:38:16.992 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:16.368049+0000", 2024-03-20T09:38:16.992 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4882999999999999e-05 2024-03-20T09:38:16.992 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:38:16.992 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:38:16.992 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:38:16.992 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:38:16.993 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:38:16.993 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 1 2024-03-20T09:38:16.993 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:38:16.993 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:38:17.032 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:17.059 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:38:17.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:17.067 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:38:17.168 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:38:17.198 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:38:17.201 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:17.223 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:38:17.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:17.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:38:17.329 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:38:17.360 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:17.388 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:38:17.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:17.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:38:17.492 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:38:17.555 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:17.599 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:38:17.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:17.605 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:38:17.706 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:38:17.735 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:17.758 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:38:17.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:17.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:38:17.862 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:38:17.892 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:17.913 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:38:17.917 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:17 smithi115 ceph-mon[72476]: pgmap v2475: 47 pgs: 47 active+clean; 5.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 814 op/s 2024-03-20T09:38:17.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:17.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:38:17.940 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:17 smithi122 ceph-mon[73495]: pgmap v2475: 47 pgs: 47 active+clean; 5.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 814 op/s 2024-03-20T09:38:17.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:17 smithi026 ceph-mon[68309]: pgmap v2475: 47 pgs: 47 active+clean; 5.2 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 814 op/s 2024-03-20T09:38:18.019 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:38:18.080 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:18.208 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:38:18.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:18.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:38:18.245 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-20T09:38:18.322 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:38:18.352 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:18.374 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:38:18.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:18.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:38:18.482 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:38:18.592 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:18.632 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:38:18.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:18.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:38:18.736 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:38:18.765 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:18.792 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:38:18.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:18.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:38:18.864 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:18 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2795791358' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:38:18.865 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:18 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:38:18.865 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:18 smithi026 ceph-mon[68309]: pgmap v2476: 47 pgs: 47 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 803 op/s 2024-03-20T09:38:18.886 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:38:18.898 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:38:18.955 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:18.987 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:38:18.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:18.995 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:38:19.095 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:38:19.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:18 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2795791358' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:38:19.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:18 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:38:19.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:18 smithi115 ceph-mon[72476]: pgmap v2476: 47 pgs: 47 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 803 op/s 2024-03-20T09:38:19.149 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:19.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:18 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2795791358' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:38:19.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:18 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:38:19.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:18 smithi122 ceph-mon[73495]: pgmap v2476: 47 pgs: 47 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 803 op/s 2024-03-20T09:38:19.380 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:38:19.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:19.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:38:19.488 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:38:19.529 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:20.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:19 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:38:20.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:19 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:38:20.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:19 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:38:20.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:19 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:38:20.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:19 smithi115 ceph-mon[72476]: osdmap e1026: 12 total, 12 up, 12 in 2024-03-20T09:38:20.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:19 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:38:20.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:19 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:38:20.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:19 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:38:20.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:19 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:38:20.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:19 smithi122 ceph-mon[73495]: osdmap e1026: 12 total, 12 up, 12 in 2024-03-20T09:38:20.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:19 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:38:20.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:19 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:38:20.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:19 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:38:20.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:19 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:38:20.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:19 smithi026 ceph-mon[68309]: osdmap e1026: 12 total, 12 up, 12 in 2024-03-20T09:38:20.401 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:38:20.404 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:38:20.617 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:38:20.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:20.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:38:20.736 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:38:20.808 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:20.848 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:38:20.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:20.851 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:38:20.952 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:38:20.999 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:21.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:20 smithi115 ceph-mon[72476]: pgmap v2478: 47 pgs: 47 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 832 op/s 2024-03-20T09:38:21.169 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:20 smithi026 ceph-mon[68309]: pgmap v2478: 47 pgs: 47 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 832 op/s 2024-03-20T09:38:21.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:20 smithi122 ceph-mon[73495]: pgmap v2478: 47 pgs: 47 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 832 op/s 2024-03-20T09:38:21.846 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:38:21.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:21.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:38:21.955 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:38:21.985 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:22.006 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:38:22.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:22.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:38:22.114 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:38:22.173 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:22.208 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:38:22.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:22.213 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:38:22.314 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:38:22.318 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:38:22.345 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:22.346 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:38:22.367 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:38:22.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:22.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:38:22.385 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:38:22.403 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:38:22.420 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:38:22.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:22 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2794208775' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:38:22.458 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:38:22.471 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:38:22.495 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:38:22.515 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:22.538 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:38:22.539 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:38:22.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:22.546 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:38:22.580 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:38:22.625 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:38:22.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:22 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2794208775' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:38:22.647 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:38:22.654 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:22 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2794208775' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:38:22.666 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:38:22.712 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:38:22.743 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:22.771 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:38:22.789 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:38:22.790 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:38:22.790 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:38:22.790 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:38:22.790 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:38:22.790 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:38:22.790 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:38:22.790 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:38:22.790 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:38:22.790 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:38:22.790 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:38:22.790 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:38:23.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:23 smithi115 ceph-mon[72476]: pgmap v2479: 47 pgs: 47 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 838 op/s 2024-03-20T09:38:23.685 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:23 smithi122 ceph-mon[73495]: pgmap v2479: 47 pgs: 47 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 838 op/s 2024-03-20T09:38:23.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:23 smithi026 ceph-mon[68309]: pgmap v2479: 47 pgs: 47 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 838 op/s 2024-03-20T09:38:25.641 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:25 smithi026 ceph-mon[68309]: pgmap v2480: 47 pgs: 47 active+clean; 4.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 837 op/s 2024-03-20T09:38:25.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:25 smithi115 ceph-mon[72476]: pgmap v2480: 47 pgs: 47 active+clean; 4.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 837 op/s 2024-03-20T09:38:25.935 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:25 smithi122 ceph-mon[73495]: pgmap v2480: 47 pgs: 47 active+clean; 4.8 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 837 op/s 2024-03-20T09:38:26.490 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:38:26.490 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:38:27.137 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:38:27.138 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "" 2024-03-20T09:38:27.138 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:38:27.566 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:38:27.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:27.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:38:27.673 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:38:27.735 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:27.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:27 smithi122 ceph-mon[73495]: pgmap v2481: 47 pgs: 47 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 849 op/s 2024-03-20T09:38:27.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:27 smithi026 ceph-mon[68309]: pgmap v2481: 47 pgs: 47 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 849 op/s 2024-03-20T09:38:28.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:27 smithi115 ceph-mon[72476]: pgmap v2481: 47 pgs: 47 active+clean; 4.7 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 849 op/s 2024-03-20T09:38:28.393 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:38:28.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:28.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:38:28.433 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-20T09:38:28.498 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:38:28.560 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:28.618 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:38:28.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:28.625 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:38:28.673 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:28 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:38:28.673 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:28 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:38:28.673 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:28 smithi026 ceph-mon[68309]: osdmap e1027: 12 total, 12 up, 12 in 2024-03-20T09:38:28.673 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:28 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:38:28.674 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:28 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:38:28.726 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:38:28.765 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:28.798 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:38:28.799 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:38:28.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:28.805 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:38:28.906 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:38:28.931 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:28 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:38:28.931 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:28 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:38:28.931 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:28 smithi115 ceph-mon[72476]: osdmap e1027: 12 total, 12 up, 12 in 2024-03-20T09:38:28.931 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:28 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:38:28.931 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:28 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:38:28.933 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:28.935 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:28 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:38:28.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:28 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:38:28.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:28 smithi122 ceph-mon[73495]: osdmap e1027: 12 total, 12 up, 12 in 2024-03-20T09:38:28.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:28 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:38:28.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:28 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:38:28.960 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:38:28.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:28.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:38:29.066 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:38:29.130 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:29.162 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:38:29.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:29.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:38:29.274 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:38:29.325 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:29.355 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:38:29.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:29.363 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:38:29.464 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:38:29.493 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:29.520 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:38:29.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:29.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:38:29.624 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:38:29.670 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:29.700 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:38:29.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:29.707 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:38:29.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:29 smithi026 ceph-mon[68309]: pgmap v2483: 47 pgs: 47 active+clean; 4.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 785 op/s 2024-03-20T09:38:29.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:29 smithi026 ceph-mon[68309]: osdmap e1028: 12 total, 12 up, 12 in 2024-03-20T09:38:29.719 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:38:29.808 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:38:29.836 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:29 smithi115 ceph-mon[72476]: pgmap v2483: 47 pgs: 47 active+clean; 4.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 785 op/s 2024-03-20T09:38:29.836 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:29 smithi115 ceph-mon[72476]: osdmap e1028: 12 total, 12 up, 12 in 2024-03-20T09:38:29.839 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:29.861 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:38:29.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:29.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:38:29.938 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:29 smithi122 ceph-mon[73495]: pgmap v2483: 47 pgs: 47 active+clean; 4.6 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 785 op/s 2024-03-20T09:38:29.939 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:29 smithi122 ceph-mon[73495]: osdmap e1028: 12 total, 12 up, 12 in 2024-03-20T09:38:29.965 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:38:30.002 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:30.033 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:38:30.036 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:30.036 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:38:30.137 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:38:30.195 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:30.581 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:38:30.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:30.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:38:30.694 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:38:30.723 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:30.746 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:38:30.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:30.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:38:30.780 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:38:30.852 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:38:30.897 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:30.936 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:38:30.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:30.948 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:38:31.049 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:38:31.130 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:31.177 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:38:31.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:31.184 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:38:31.285 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:38:31.331 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:31.385 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:38:31.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:31.391 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:38:31.492 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:38:31.519 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:31.543 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:38:31.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:31.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:38:31.651 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:38:31.682 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:31.701 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:38:31.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:31.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:38:31.806 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:38:31.868 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:31.894 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:31 smithi115 ceph-mon[72476]: pgmap v2485: 47 pgs: 47 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 797 op/s 2024-03-20T09:38:31.903 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:38:31.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:31.915 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:38:31.934 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:31 smithi026 ceph-mon[68309]: pgmap v2485: 47 pgs: 47 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 797 op/s 2024-03-20T09:38:31.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:31 smithi122 ceph-mon[73495]: pgmap v2485: 47 pgs: 47 active+clean; 4.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 797 op/s 2024-03-20T09:38:31.992 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:38:32.016 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:38:32.065 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:32.097 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:38:32.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:32.100 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:38:32.201 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:38:32.246 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:32.284 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:38:32.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:32.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:38:32.389 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:38:32.420 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:32.445 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:38:32.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:32.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:38:32.550 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:38:32.606 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:32.654 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:38:32.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:32.660 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:38:32.714 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:38:32.760 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:38:32.813 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:32.856 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:38:32.869 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:32 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/400482434' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:38:32.869 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:32 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:38:32.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:32.870 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:38:32.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:32 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/400482434' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:38:32.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:32 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:38:32.971 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:38:32.977 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:38:32.977 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:38:33.032 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:33.065 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:38:33.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:33.072 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:38:33.126 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 8, 11, 3, 4, 9, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 11, 5, 10] 2024-03-20T09:38:33.126 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:38:33.126 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_6 2024-03-20T09:38:33.127 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:38:33.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:32 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/400482434' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:38:33.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:32 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:38:33.173 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:38:33.258 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:33.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:33 smithi122 ceph-mon[73495]: pgmap v2486: 47 pgs: 47 active+clean; 4.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 902 op/s 2024-03-20T09:38:33.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:33 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:38:33.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:33 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:38:33.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:33 smithi122 ceph-mon[73495]: osdmap e1029: 12 total, 12 up, 12 in 2024-03-20T09:38:33.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:33 smithi122 ceph-mon[73495]: 1.14 scrub starts 2024-03-20T09:38:33.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:33 smithi122 ceph-mon[73495]: 1.14 scrub ok 2024-03-20T09:38:33.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:33 smithi122 ceph-mon[73495]: 8.9 scrub starts 2024-03-20T09:38:33.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:33 smithi026 ceph-mon[68309]: pgmap v2486: 47 pgs: 47 active+clean; 4.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 902 op/s 2024-03-20T09:38:33.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:33 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:38:33.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:33 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:38:33.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:33 smithi026 ceph-mon[68309]: osdmap e1029: 12 total, 12 up, 12 in 2024-03-20T09:38:33.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:33 smithi026 ceph-mon[68309]: 1.14 scrub starts 2024-03-20T09:38:33.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:33 smithi026 ceph-mon[68309]: 1.14 scrub ok 2024-03-20T09:38:33.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:33 smithi026 ceph-mon[68309]: 8.9 scrub starts 2024-03-20T09:38:34.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:33 smithi115 ceph-mon[72476]: pgmap v2486: 47 pgs: 47 active+clean; 4.4 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 902 op/s 2024-03-20T09:38:34.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:33 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:38:34.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:33 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:38:34.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:33 smithi115 ceph-mon[72476]: osdmap e1029: 12 total, 12 up, 12 in 2024-03-20T09:38:34.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:33 smithi115 ceph-mon[72476]: 1.14 scrub starts 2024-03-20T09:38:34.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:33 smithi115 ceph-mon[72476]: 1.14 scrub ok 2024-03-20T09:38:34.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:33 smithi115 ceph-mon[72476]: 8.9 scrub starts 2024-03-20T09:38:34.341 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:38:34.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:34.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:38:34.449 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:38:34.490 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:34.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:34 smithi026 ceph-mon[68309]: 1.18 scrub starts 2024-03-20T09:38:34.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:34 smithi026 ceph-mon[68309]: 1.18 scrub ok 2024-03-20T09:38:34.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:34 smithi026 ceph-mon[68309]: 8.9 scrub ok 2024-03-20T09:38:35.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:34 smithi115 ceph-mon[72476]: 1.18 scrub starts 2024-03-20T09:38:35.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:34 smithi115 ceph-mon[72476]: 1.18 scrub ok 2024-03-20T09:38:35.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:34 smithi115 ceph-mon[72476]: 8.9 scrub ok 2024-03-20T09:38:35.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:34 smithi122 ceph-mon[73495]: 1.18 scrub starts 2024-03-20T09:38:35.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:34 smithi122 ceph-mon[73495]: 1.18 scrub ok 2024-03-20T09:38:35.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:34 smithi122 ceph-mon[73495]: 8.9 scrub ok 2024-03-20T09:38:35.597 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:38:35.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:35.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:38:35.708 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:38:35.751 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:35.803 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:38:35.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:35.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:38:35.912 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:38:35.940 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:35.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:35 smithi026 ceph-mon[68309]: pgmap v2488: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 803 op/s 2024-03-20T09:38:35.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:35 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:38:35.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:35 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:38:35.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:35 smithi026 ceph-mon[68309]: 1.12 scrub starts 2024-03-20T09:38:35.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:35 smithi026 ceph-mon[68309]: 1.12 scrub ok 2024-03-20T09:38:35.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:35 smithi026 ceph-mon[68309]: 8.5 scrub starts 2024-03-20T09:38:35.966 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:38:35.969 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:35 smithi115 ceph-mon[72476]: pgmap v2488: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 803 op/s 2024-03-20T09:38:35.969 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:35 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:38:35.969 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:35 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:38:35.969 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:35 smithi115 ceph-mon[72476]: 1.12 scrub starts 2024-03-20T09:38:35.969 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:35 smithi115 ceph-mon[72476]: 1.12 scrub ok 2024-03-20T09:38:35.969 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:35 smithi115 ceph-mon[72476]: 8.5 scrub starts 2024-03-20T09:38:35.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:35.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:38:36.072 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:38:36.109 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:35 smithi122 ceph-mon[73495]: pgmap v2488: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.3 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 803 op/s 2024-03-20T09:38:36.109 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:35 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:38:36.109 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:35 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:38:36.109 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:35 smithi122 ceph-mon[73495]: 1.12 scrub starts 2024-03-20T09:38:36.109 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:35 smithi122 ceph-mon[73495]: 1.12 scrub ok 2024-03-20T09:38:36.109 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:35 smithi122 ceph-mon[73495]: 8.5 scrub starts 2024-03-20T09:38:36.110 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:36.145 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:38:36.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:36.153 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:38:36.163 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-20T09:38:36.174 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:38:36.253 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:38:36.334 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:36.405 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:38:36.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:36.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:38:36.518 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:38:36.547 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:36.570 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:38:36.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:36.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:38:36.676 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:38:36.730 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:36.774 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:38:36.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:36.779 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:38:36.880 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:38:36.894 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:36 smithi115 ceph-mon[72476]: osdmap e1030: 12 total, 12 up, 12 in 2024-03-20T09:38:36.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:36 smithi115 ceph-mon[72476]: 1.8 scrub starts 2024-03-20T09:38:36.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:36 smithi115 ceph-mon[72476]: 1.8 scrub ok 2024-03-20T09:38:36.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:36 smithi115 ceph-mon[72476]: 8.5 scrub ok 2024-03-20T09:38:36.923 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:36.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:36 smithi026 ceph-mon[68309]: osdmap e1030: 12 total, 12 up, 12 in 2024-03-20T09:38:36.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:36 smithi026 ceph-mon[68309]: 1.8 scrub starts 2024-03-20T09:38:36.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:36 smithi026 ceph-mon[68309]: 1.8 scrub ok 2024-03-20T09:38:36.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:36 smithi026 ceph-mon[68309]: 8.5 scrub ok 2024-03-20T09:38:36.982 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:38:36.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:36.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:38:37.087 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:38:37.143 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:36 smithi122 ceph-mon[73495]: osdmap e1030: 12 total, 12 up, 12 in 2024-03-20T09:38:37.143 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:36 smithi122 ceph-mon[73495]: 1.8 scrub starts 2024-03-20T09:38:37.144 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:36 smithi122 ceph-mon[73495]: 1.8 scrub ok 2024-03-20T09:38:37.144 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:36 smithi122 ceph-mon[73495]: 8.5 scrub ok 2024-03-20T09:38:37.147 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:38:37.148 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:37.195 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:38:37.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:37.203 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:38:37.304 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:38:37.332 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:37.358 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:38:37.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:37.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:38:37.462 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:38:37.490 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:37.515 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:38:37.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:37.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:38:37.621 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:38:37.694 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:37.728 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:38:37.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:37.735 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:38:37.836 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:38:37.869 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:37.897 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:38:37.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:37.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:38:37.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:37 smithi026 ceph-mon[68309]: pgmap v2490: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 902 op/s 2024-03-20T09:38:37.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:37 smithi026 ceph-mon[68309]: 1.5 scrub starts 2024-03-20T09:38:37.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:37 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2875123902' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:38:38.001 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:38:38.052 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:37 smithi122 ceph-mon[73495]: pgmap v2490: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 902 op/s 2024-03-20T09:38:38.052 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:37 smithi122 ceph-mon[73495]: 1.5 scrub starts 2024-03-20T09:38:38.052 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:37 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2875123902' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:38:38.056 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:38.105 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:38:38.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:38.109 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:38:38.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:37 smithi115 ceph-mon[72476]: pgmap v2490: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 902 op/s 2024-03-20T09:38:38.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:37 smithi115 ceph-mon[72476]: 1.5 scrub starts 2024-03-20T09:38:38.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:37 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2875123902' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:38:38.210 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:38:38.252 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:38.366 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:38:38.367 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:38:38.367 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:38:38.367 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:38:38.367 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:38:38.367 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:38:38.367 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:38:38.367 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:38:38.367 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:38:38.367 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:38:38.367 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:38:38.367 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:38:38.367 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:38:38.368 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:38:38.368 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:38:38.368 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:38:38.368 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:38:38.368 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:38:38.368 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:38:38.368 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:38:38.368 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:38:38.368 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:38:38.368 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:38:38.368 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:38:38.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:38 smithi026 ceph-mon[68309]: 1.1b scrub starts 2024-03-20T09:38:38.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:38 smithi026 ceph-mon[68309]: 1.1b scrub ok 2024-03-20T09:38:38.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:38 smithi026 ceph-mon[68309]: 1.5 scrub ok 2024-03-20T09:38:38.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:38 smithi026 ceph-mon[68309]: 8.4 scrub starts 2024-03-20T09:38:38.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:38 smithi026 ceph-mon[68309]: 1.9 scrub starts 2024-03-20T09:38:38.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:38 smithi026 ceph-mon[68309]: 1.9 scrub ok 2024-03-20T09:38:39.011 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:38:39.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:39.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:38:39.119 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:38:39.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:38 smithi115 ceph-mon[72476]: 1.1b scrub starts 2024-03-20T09:38:39.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:38 smithi115 ceph-mon[72476]: 1.1b scrub ok 2024-03-20T09:38:39.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:38 smithi115 ceph-mon[72476]: 1.5 scrub ok 2024-03-20T09:38:39.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:38 smithi115 ceph-mon[72476]: 8.4 scrub starts 2024-03-20T09:38:39.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:38 smithi115 ceph-mon[72476]: 1.9 scrub starts 2024-03-20T09:38:39.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:38 smithi115 ceph-mon[72476]: 1.9 scrub ok 2024-03-20T09:38:39.146 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:39.169 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:38:39.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:39.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:38:39.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:38 smithi122 ceph-mon[73495]: 1.1b scrub starts 2024-03-20T09:38:39.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:38 smithi122 ceph-mon[73495]: 1.1b scrub ok 2024-03-20T09:38:39.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:38 smithi122 ceph-mon[73495]: 1.5 scrub ok 2024-03-20T09:38:39.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:38 smithi122 ceph-mon[73495]: 8.4 scrub starts 2024-03-20T09:38:39.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:38 smithi122 ceph-mon[73495]: 1.9 scrub starts 2024-03-20T09:38:39.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:38 smithi122 ceph-mon[73495]: 1.9 scrub ok 2024-03-20T09:38:39.273 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:38:39.301 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:39.322 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:38:39.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:39.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:38:39.426 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:38:39.447 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:38:39.449 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:38:39.470 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:39.490 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:38:39.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:39.493 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:38:39.594 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:38:39.666 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:39.698 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:38:39.704 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:39.704 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:38:39.804 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:38:39.853 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:38:39.854 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:38:39.903 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:39.941 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:38:39.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:39.944 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:38:40.045 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:38:40.083 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:39 smithi122 ceph-mon[73495]: pgmap v2491: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 845 op/s 2024-03-20T09:38:40.084 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:39 smithi122 ceph-mon[73495]: 8.4 scrub ok 2024-03-20T09:38:40.084 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:39 smithi122 ceph-mon[73495]: 1.3 scrub starts 2024-03-20T09:38:40.084 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:39 smithi122 ceph-mon[73495]: 1.3 scrub ok 2024-03-20T09:38:40.084 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:40.125 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:38:40.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:40.131 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:38:40.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:39 smithi115 ceph-mon[72476]: pgmap v2491: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 845 op/s 2024-03-20T09:38:40.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:39 smithi115 ceph-mon[72476]: 8.4 scrub ok 2024-03-20T09:38:40.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:39 smithi115 ceph-mon[72476]: 1.3 scrub starts 2024-03-20T09:38:40.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:39 smithi115 ceph-mon[72476]: 1.3 scrub ok 2024-03-20T09:38:40.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:39 smithi026 ceph-mon[68309]: pgmap v2491: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.1 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 845 op/s 2024-03-20T09:38:40.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:39 smithi026 ceph-mon[68309]: 8.4 scrub ok 2024-03-20T09:38:40.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:39 smithi026 ceph-mon[68309]: 1.3 scrub starts 2024-03-20T09:38:40.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:39 smithi026 ceph-mon[68309]: 1.3 scrub ok 2024-03-20T09:38:40.233 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:38:40.297 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:40.330 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:38:40.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:40.335 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:38:40.436 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:38:40.506 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:40.538 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:38:40.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:40.546 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:38:40.648 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:38:40.676 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:40.698 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:38:40.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:40.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:38:40.804 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:38:40.863 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:40.892 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:38:40.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:40.899 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:38:41.000 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:38:41.024 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:38:41.051 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:40 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:38:41.051 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:40 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:38:41.051 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:40 smithi122 ceph-mon[73495]: osdmap e1031: 12 total, 12 up, 12 in 2024-03-20T09:38:41.052 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:40 smithi122 ceph-mon[73495]: 8.e scrub starts 2024-03-20T09:38:41.052 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:40 smithi122 ceph-mon[73495]: from='client.27052 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:38:41.057 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:41.071 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:40 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:38:41.071 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:40 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:38:41.071 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:40 smithi026 ceph-mon[68309]: osdmap e1031: 12 total, 12 up, 12 in 2024-03-20T09:38:41.071 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:40 smithi026 ceph-mon[68309]: 8.e scrub starts 2024-03-20T09:38:41.071 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:40 smithi026 ceph-mon[68309]: from='client.27052 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:38:41.099 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:38:41.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:41.102 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:38:41.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:40 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:38:41.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:40 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:38:41.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:40 smithi115 ceph-mon[72476]: osdmap e1031: 12 total, 12 up, 12 in 2024-03-20T09:38:41.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:40 smithi115 ceph-mon[72476]: 8.e scrub starts 2024-03-20T09:38:41.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:40 smithi115 ceph-mon[72476]: from='client.27052 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:38:41.203 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:38:41.255 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:41.280 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:38:41.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:41.287 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:38:41.388 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:38:41.448 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:42.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:41 smithi115 ceph-mon[72476]: pgmap v2493: 47 pgs: 47 active+clean; 4.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 768 op/s 2024-03-20T09:38:42.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:41 smithi115 ceph-mon[72476]: 8.e scrub ok 2024-03-20T09:38:42.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:41 smithi122 ceph-mon[73495]: pgmap v2493: 47 pgs: 47 active+clean; 4.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 768 op/s 2024-03-20T09:38:42.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:41 smithi122 ceph-mon[73495]: 8.e scrub ok 2024-03-20T09:38:42.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:41 smithi026 ceph-mon[68309]: pgmap v2493: 47 pgs: 47 active+clean; 4.0 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 768 op/s 2024-03-20T09:38:42.214 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:41 smithi026 ceph-mon[68309]: 8.e scrub ok 2024-03-20T09:38:42.619 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:38:42.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:42.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:38:42.647 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":2492,"stamp":"2024-03-20T09:38:39.755992+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4320854584,"num_objects":65925,"num_object_clones":0,"num_object_copies":197775,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65925,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180681,"num_write_kb":7891265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":165021,"ondisk_log_size":165021,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":21829332,"kb_used_data":12666652,"kb_used_omap":301,"kb_used_meta":9161682,"kb_avail":1103112492,"statfs":{"total":1151940427776,"available":1129587191808,"internally_reserved":0,"allocated":12970651648,"data_stored":12967287310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":308246,"internal_metadata":9381563370},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-351666176,"num_objects":-5366,"num_object_clones":0,"num_object_copies":-16098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5366,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.157308"},"pg_stats":[{"pgid":"1.1e","version":"20'137","reported_seq":1518,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:35.718721+0000","last_change":"2024-03-20T09:37:37.183464+0000","last_active":"2024-03-20T09:38:35.718721+0000","last_peered":"2024-03-20T09:38:35.718721+0000","last_clean":"2024-03-20T09:38:35.718721+0000","last_became_active":"2024-03-20T09:13:20.071655+0000","last_became_peered":"2024-03-20T09:13:20.071655+0000","last_unstale":"2024-03-20T09:38:35.718721+0000","last_undegraded":"2024-03-20T09:38:35.718721+0000","last_fullsized":"2024-03-20T09:38:35.718721+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:37.183343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:37:37.183343+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:48.745062+0000","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'137","reported_seq":603,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:35.717488+0000","last_change":"2024-03-20T09:37:54.221807+0000","last_active":"2024-03-20T09:38:35.717488+0000","last_peered":"2024-03-20T09:38:35.717488+0000","last_clean":"2024-03-20T09:38:35.717488+0000","last_became_active":"2024-03-20T09:31:07.425574+0000","last_became_peered":"2024-03-20T09:31:07.425574+0000","last_unstale":"2024-03-20T09:38:35.717488+0000","last_undegraded":"2024-03-20T09:38:35.717488+0000","last_fullsized":"2024-03-20T09:38:35.717488+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:54.221681+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:29:02.585165+0000","last_clean_scrub_stamp":"2024-03-20T09:37:54.221681+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:13.870776+0000","scrub_duration":216,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2],"acting":[11,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"20'137","reported_seq":2207,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:35.707805+0000","last_change":"2024-03-20T09:37:56.375223+0000","last_active":"2024-03-20T09:38:35.707805+0000","last_peered":"2024-03-20T09:38:35.707805+0000","last_clean":"2024-03-20T09:38:35.707805+0000","last_became_active":"2024-03-20T09:35:03.815532+0000","last_became_peered":"2024-03-20T09:35:03.815532+0000","last_unstale":"2024-03-20T09:38:35.707805+0000","last_undegraded":"2024-03-20T09:38:35.707805+0000","last_fullsized":"2024-03-20T09:38:35.707805+0000","mapping_epoch":950,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":951,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:56.375139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:37:56.375139+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:15.044311+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.19","version":"20'137","reported_seq":1382,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:35.708761+0000","last_change":"2024-03-20T09:37:59.476380+0000","last_active":"2024-03-20T09:38:35.708761+0000","last_peered":"2024-03-20T09:38:35.708761+0000","last_clean":"2024-03-20T09:38:35.708761+0000","last_became_active":"2024-03-20T09:35:19.819786+0000","last_became_peered":"2024-03-20T09:35:19.819786+0000","last_unstale":"2024-03-20T09:38:35.708761+0000","last_undegraded":"2024-03-20T09:38:35.708761+0000","last_fullsized":"2024-03-20T09:38:35.708761+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:59.476249+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:30:56.201554+0000","last_clean_scrub_stamp":"2024-03-20T09:37:59.476249+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:23.603620+0000","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"20'137","reported_seq":1670,"reported_epoch":1029,"state":"active+clean","last_fresh":"2024-03-20T09:38:33.938995+0000","last_change":"2024-03-20T09:38:33.939048+0000","last_active":"2024-03-20T09:38:33.938995+0000","last_peered":"2024-03-20T09:38:33.938995+0000","last_clean":"2024-03-20T09:38:33.938995+0000","last_became_active":"2024-03-20T09:10:35.277481+0000","last_became_peered":"2024-03-20T09:10:35.277481+0000","last_unstale":"2024-03-20T09:38:33.938995+0000","last_undegraded":"2024-03-20T09:38:33.938995+0000","last_fullsized":"2024-03-20T09:38:33.938995+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:38:33.938915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:38:33.938915+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:37.571239+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1b","version":"20'137","reported_seq":2027,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:35.707867+0000","last_change":"2024-03-20T09:36:54.315917+0000","last_active":"2024-03-20T09:38:35.707867+0000","last_peered":"2024-03-20T09:38:35.707867+0000","last_clean":"2024-03-20T09:38:35.707867+0000","last_became_active":"2024-03-20T09:24:16.445287+0000","last_became_peered":"2024-03-20T09:24:16.445287+0000","last_unstale":"2024-03-20T09:38:35.707867+0000","last_undegraded":"2024-03-20T09:38:35.707867+0000","last_fullsized":"2024-03-20T09:38:35.707867+0000","mapping_epoch":718,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:36:54.315794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:36:54.315794+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:37.205960+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"8.1","version":"1030'14225","reported_seq":14605,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:38.413966+0000","last_change":"2024-03-20T09:37:38.850438+0000","last_active":"2024-03-20T09:38:38.413966+0000","last_peered":"2024-03-20T09:38:38.413966+0000","last_clean":"2024-03-20T09:38:38.413966+0000","last_became_active":"2024-03-20T09:35:51.943955+0000","last_became_peered":"2024-03-20T09:35:51.943955+0000","last_unstale":"2024-03-20T09:38:38.413966+0000","last_undegraded":"2024-03-20T09:38:38.413966+0000","last_fullsized":"2024-03-20T09:38:38.413966+0000","mapping_epoch":974,"log_start":"993'4200","ondisk_log_start":"993'4200","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1016'11148","last_scrub_stamp":"2024-03-20T09:37:38.850275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:38.850275+0000","objects_scrubbed":7129,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:47.172912+0000","scrub_duration":2090,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262340608,"num_objects":4003,"num_object_clones":0,"num_object_copies":12009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11333,"num_write_kb":490752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,2],"acting":[9,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.8","version":"20'137","reported_seq":807,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:35.949146+0000","last_change":"2024-03-20T09:38:35.949170+0000","last_active":"2024-03-20T09:38:35.949146+0000","last_peered":"2024-03-20T09:38:35.949146+0000","last_clean":"2024-03-20T09:38:35.949146+0000","last_became_active":"2024-03-20T09:35:28.000367+0000","last_became_peered":"2024-03-20T09:35:28.000367+0000","last_unstale":"2024-03-20T09:38:35.949146+0000","last_undegraded":"2024-03-20T09:38:35.949146+0000","last_fullsized":"2024-03-20T09:38:35.949146+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":966,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:38:35.949076+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:37:01.184206+0000","last_clean_scrub_stamp":"2024-03-20T09:38:35.949076+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:38.649879+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"1029'11138","reported_seq":11338,"reported_epoch":1029,"state":"active+clean","last_fresh":"2024-03-20T09:38:34.859368+0000","last_change":"2024-03-20T09:37:41.579524+0000","last_active":"2024-03-20T09:38:34.859368+0000","last_peered":"2024-03-20T09:38:34.859368+0000","last_clean":"2024-03-20T09:38:34.859368+0000","last_became_active":"2024-03-20T09:35:51.276462+0000","last_became_peered":"2024-03-20T09:35:51.276462+0000","last_unstale":"2024-03-20T09:38:34.859368+0000","last_undegraded":"2024-03-20T09:38:34.859368+0000","last_fullsized":"2024-03-20T09:38:34.859368+0000","mapping_epoch":974,"log_start":"980'1100","ondisk_log_start":"980'1100","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1017'8324","last_scrub_stamp":"2024-03-20T09:37:41.579322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:41.579322+0000","objects_scrubbed":7101,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:55.312205+0000","scrub_duration":1793,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278396928,"num_objects":4248,"num_object_clones":0,"num_object_copies":12744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11138,"num_write_kb":492352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"20'137","reported_seq":2265,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:38.161181+0000","last_change":"2024-03-20T09:38:38.161217+0000","last_active":"2024-03-20T09:38:38.161181+0000","last_peered":"2024-03-20T09:38:38.161181+0000","last_clean":"2024-03-20T09:38:38.161181+0000","last_became_active":"2024-03-20T09:20:25.341750+0000","last_became_peered":"2024-03-20T09:20:25.341750+0000","last_unstale":"2024-03-20T09:38:38.161181+0000","last_undegraded":"2024-03-20T09:38:38.161181+0000","last_fullsized":"2024-03-20T09:38:38.161181+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:38:38.161101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:38:38.161101+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:40:05.787956+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,2],"acting":[9,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.3","version":"1030'11327","reported_seq":11525,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:35.716349+0000","last_change":"2024-03-20T09:37:50.941074+0000","last_active":"2024-03-20T09:38:35.716349+0000","last_peered":"2024-03-20T09:38:35.716349+0000","last_clean":"2024-03-20T09:38:35.716349+0000","last_became_active":"2024-03-20T09:35:51.674444+0000","last_became_peered":"2024-03-20T09:35:51.674444+0000","last_unstale":"2024-03-20T09:38:35.716349+0000","last_undegraded":"2024-03-20T09:38:35.716349+0000","last_fullsized":"2024-03-20T09:38:35.716349+0000","mapping_epoch":974,"log_start":"980'1300","ondisk_log_start":"980'1300","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1019'8929","last_scrub_stamp":"2024-03-20T09:37:50.940775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:50.940775+0000","objects_scrubbed":6673,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:18.254318+0000","scrub_duration":1914,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":277282816,"num_objects":4231,"num_object_clones":0,"num_object_copies":12693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11327,"num_write_kb":497856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"20'137","reported_seq":946,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:35.715932+0000","last_change":"2024-03-20T09:37:38.749161+0000","last_active":"2024-03-20T09:38:35.715932+0000","last_peered":"2024-03-20T09:38:35.715932+0000","last_clean":"2024-03-20T09:38:35.715932+0000","last_became_active":"2024-03-20T09:24:16.444232+0000","last_became_peered":"2024-03-20T09:24:16.444232+0000","last_unstale":"2024-03-20T09:38:35.715932+0000","last_undegraded":"2024-03-20T09:38:35.715932+0000","last_fullsized":"2024-03-20T09:38:35.715932+0000","mapping_epoch":718,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:38.749086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:37:38.749086+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:01.052060+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.d","version":"1030'11260","reported_seq":11464,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:35.717409+0000","last_change":"2024-03-20T09:37:42.977733+0000","last_active":"2024-03-20T09:38:35.717409+0000","last_peered":"2024-03-20T09:38:35.717409+0000","last_clean":"2024-03-20T09:38:35.717409+0000","last_became_active":"2024-03-20T09:35:51.943827+0000","last_became_peered":"2024-03-20T09:35:51.943827+0000","last_unstale":"2024-03-20T09:38:35.717409+0000","last_undegraded":"2024-03-20T09:38:35.717409+0000","last_fullsized":"2024-03-20T09:38:35.717409+0000","mapping_epoch":974,"log_start":"980'1200","ondisk_log_start":"980'1200","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1017'8430","last_scrub_stamp":"2024-03-20T09:37:42.977486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:42.977486+0000","objects_scrubbed":7043,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:01.233557+0000","scrub_duration":1930,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273219608,"num_objects":4170,"num_object_clones":0,"num_object_copies":12510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4170,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11260,"num_write_kb":493697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,2],"acting":[11,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"20'137","reported_seq":2186,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:35.716305+0000","last_change":"2024-03-20T09:37:00.843889+0000","last_active":"2024-03-20T09:38:35.716305+0000","last_peered":"2024-03-20T09:38:35.716305+0000","last_clean":"2024-03-20T09:38:35.716305+0000","last_became_active":"2024-03-20T09:12:14.306384+0000","last_became_peered":"2024-03-20T09:12:14.306384+0000","last_unstale":"2024-03-20T09:38:35.716305+0000","last_undegraded":"2024-03-20T09:38:35.716305+0000","last_fullsized":"2024-03-20T09:38:35.716305+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:00.843763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:37:00.843763+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:37.146428+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.c","version":"1030'11484","reported_seq":11686,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:38.441341+0000","last_change":"2024-03-20T09:37:39.960010+0000","last_active":"2024-03-20T09:38:38.441341+0000","last_peered":"2024-03-20T09:38:38.441341+0000","last_clean":"2024-03-20T09:38:38.441341+0000","last_became_active":"2024-03-20T09:35:51.278227+0000","last_became_peered":"2024-03-20T09:35:51.278227+0000","last_unstale":"2024-03-20T09:38:38.441341+0000","last_undegraded":"2024-03-20T09:38:38.441341+0000","last_fullsized":"2024-03-20T09:38:38.441341+0000","mapping_epoch":974,"log_start":"980'1400","ondisk_log_start":"980'1400","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1017'8325","last_scrub_stamp":"2024-03-20T09:37:39.959895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:39.959895+0000","objects_scrubbed":7210,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:50.065860+0000","scrub_duration":2195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261226496,"num_objects":3986,"num_object_clones":0,"num_object_copies":11958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11484,"num_write_kb":495040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.5","version":"20'137","reported_seq":854,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:37.543167+0000","last_change":"2024-03-20T09:38:37.543192+0000","last_active":"2024-03-20T09:38:37.543167+0000","last_peered":"2024-03-20T09:38:37.543167+0000","last_clean":"2024-03-20T09:38:37.543167+0000","last_became_active":"2024-03-20T09:35:19.820895+0000","last_became_peered":"2024-03-20T09:35:19.820895+0000","last_unstale":"2024-03-20T09:38:37.543167+0000","last_undegraded":"2024-03-20T09:38:37.543167+0000","last_fullsized":"2024-03-20T09:38:37.543167+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:38:37.543102+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:19:03.805185+0000","last_clean_scrub_stamp":"2024-03-20T09:38:37.543102+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:55.130257+0000","scrub_duration":148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.f","version":"1029'11161","reported_seq":11359,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:35.716313+0000","last_change":"2024-03-20T09:37:50.253230+0000","last_active":"2024-03-20T09:38:35.716313+0000","last_peered":"2024-03-20T09:38:35.716313+0000","last_clean":"2024-03-20T09:38:35.716313+0000","last_became_active":"2024-03-20T09:35:51.943804+0000","last_became_peered":"2024-03-20T09:35:51.943804+0000","last_unstale":"2024-03-20T09:38:35.716313+0000","last_undegraded":"2024-03-20T09:38:35.716313+0000","last_fullsized":"2024-03-20T09:38:35.716313+0000","mapping_epoch":974,"log_start":"980'1100","ondisk_log_start":"980'1100","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1019'8690","last_scrub_stamp":"2024-03-20T09:37:50.253006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:50.253006+0000","objects_scrubbed":6576,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:15.232988+0000","scrub_duration":2230,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264962048,"num_objects":4043,"num_object_clones":0,"num_object_copies":12129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11161,"num_write_kb":486528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"20'137","reported_seq":1724,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:35.709415+0000","last_change":"2024-03-20T09:37:42.468637+0000","last_active":"2024-03-20T09:38:35.709415+0000","last_peered":"2024-03-20T09:38:35.709415+0000","last_clean":"2024-03-20T09:38:35.709415+0000","last_became_active":"2024-03-20T09:35:19.822321+0000","last_became_peered":"2024-03-20T09:35:19.822321+0000","last_unstale":"2024-03-20T09:38:35.709415+0000","last_undegraded":"2024-03-20T09:38:35.709415+0000","last_fullsized":"2024-03-20T09:38:35.709415+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:42.468502+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:24:31.549603+0000","last_clean_scrub_stamp":"2024-03-20T09:37:42.468502+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:02.303736+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.a","version":"1030'11290","reported_seq":11493,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:36.731791+0000","last_change":"2024-03-20T09:37:43.187842+0000","last_active":"2024-03-20T09:38:36.731791+0000","last_peered":"2024-03-20T09:38:36.731791+0000","last_clean":"2024-03-20T09:38:36.731791+0000","last_became_active":"2024-03-20T09:35:51.944099+0000","last_became_peered":"2024-03-20T09:35:51.944099+0000","last_unstale":"2024-03-20T09:38:36.731791+0000","last_undegraded":"2024-03-20T09:38:36.731791+0000","last_fullsized":"2024-03-20T09:38:36.731791+0000","mapping_epoch":974,"log_start":"980'1200","ondisk_log_start":"980'1200","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1017'8466","last_scrub_stamp":"2024-03-20T09:37:43.183079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:43.183079+0000","objects_scrubbed":7069,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:57.208328+0000","scrub_duration":1937,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275775488,"num_objects":4208,"num_object_clones":0,"num_object_copies":12624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11290,"num_write_kb":495936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"20'137","reported_seq":1251,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:35.892059+0000","last_change":"2024-03-20T09:36:54.845047+0000","last_active":"2024-03-20T09:38:35.892059+0000","last_peered":"2024-03-20T09:38:35.892059+0000","last_clean":"2024-03-20T09:38:35.892059+0000","last_became_active":"2024-03-20T09:31:07.424588+0000","last_became_peered":"2024-03-20T09:31:07.424588+0000","last_unstale":"2024-03-20T09:38:35.892059+0000","last_undegraded":"2024-03-20T09:38:35.892059+0000","last_fullsized":"2024-03-20T09:38:35.892059+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:36:54.844902+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:30:47.812125+0000","last_clean_scrub_stamp":"2024-03-20T09:36:54.844902+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":211,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.9","version":"1030'11471","reported_seq":11714,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:38.824127+0000","last_change":"2024-03-20T09:38:34.308897+0000","last_active":"2024-03-20T09:38:38.824127+0000","last_peered":"2024-03-20T09:38:38.824127+0000","last_clean":"2024-03-20T09:38:38.824127+0000","last_became_active":"2024-03-20T09:35:51.274816+0000","last_became_peered":"2024-03-20T09:35:51.274816+0000","last_unstale":"2024-03-20T09:38:38.824127+0000","last_undegraded":"2024-03-20T09:38:38.824127+0000","last_fullsized":"2024-03-20T09:38:38.824127+0000","mapping_epoch":974,"log_start":"980'1400","ondisk_log_start":"980'1400","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1029'11258","last_scrub_stamp":"2024-03-20T09:38:34.308652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:38:34.308652+0000","objects_scrubbed":4290,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:40.063962+0000","scrub_duration":1000,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265617408,"num_objects":4053,"num_object_clones":0,"num_object_copies":12159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11471,"num_write_kb":496768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":2166,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:35.716195+0000","last_change":"2024-03-20T09:38:02.890362+0000","last_active":"2024-03-20T09:38:35.716195+0000","last_peered":"2024-03-20T09:38:35.716195+0000","last_clean":"2024-03-20T09:38:35.716195+0000","last_became_active":"2024-03-20T09:12:14.305989+0000","last_became_peered":"2024-03-20T09:12:14.305989+0000","last_unstale":"2024-03-20T09:38:35.716195+0000","last_undegraded":"2024-03-20T09:38:35.716195+0000","last_fullsized":"2024-03-20T09:38:35.716195+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:38:02.890224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:38:02.890224+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:27.163539+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.b","version":"1030'11333","reported_seq":11536,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:36.753834+0000","last_change":"2024-03-20T09:37:38.084925+0000","last_active":"2024-03-20T09:38:36.753834+0000","last_peered":"2024-03-20T09:38:36.753834+0000","last_clean":"2024-03-20T09:38:36.753834+0000","last_became_active":"2024-03-20T09:35:51.942846+0000","last_became_peered":"2024-03-20T09:35:51.942846+0000","last_unstale":"2024-03-20T09:38:36.753834+0000","last_undegraded":"2024-03-20T09:38:36.753834+0000","last_fullsized":"2024-03-20T09:38:36.753834+0000","mapping_epoch":974,"log_start":"980'1300","ondisk_log_start":"980'1300","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1015'8221","last_scrub_stamp":"2024-03-20T09:37:38.084645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:38.084645+0000","objects_scrubbed":7285,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:52.604722+0000","scrub_duration":1768,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270729216,"num_objects":4131,"num_object_clones":0,"num_object_copies":12393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11333,"num_write_kb":494848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"20'137","reported_seq":2422,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:35.708554+0000","last_change":"2024-03-20T09:36:50.854490+0000","last_active":"2024-03-20T09:38:35.708554+0000","last_peered":"2024-03-20T09:38:35.708554+0000","last_clean":"2024-03-20T09:38:35.708554+0000","last_became_active":"2024-03-20T09:19:54.492879+0000","last_became_peered":"2024-03-20T09:19:54.492879+0000","last_unstale":"2024-03-20T09:38:35.708554+0000","last_undegraded":"2024-03-20T09:38:35.708554+0000","last_fullsized":"2024-03-20T09:38:35.708554+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:36:50.854413+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:19.513511+0000","last_clean_scrub_stamp":"2024-03-20T09:36:50.854413+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"1029'11249","reported_seq":11451,"reported_epoch":1029,"state":"active+clean","last_fresh":"2024-03-20T09:38:34.942796+0000","last_change":"2024-03-20T09:37:38.178742+0000","last_active":"2024-03-20T09:38:34.942796+0000","last_peered":"2024-03-20T09:38:34.942796+0000","last_clean":"2024-03-20T09:38:34.942796+0000","last_became_active":"2024-03-20T09:35:51.276320+0000","last_became_peered":"2024-03-20T09:35:51.276320+0000","last_unstale":"2024-03-20T09:38:34.942796+0000","last_undegraded":"2024-03-20T09:38:34.942796+0000","last_fullsized":"2024-03-20T09:38:34.942796+0000","mapping_epoch":974,"log_start":"980'1200","ondisk_log_start":"980'1200","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1015'8253","last_scrub_stamp":"2024-03-20T09:37:38.178617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:38.178617+0000","objects_scrubbed":7305,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:59.551600+0000","scrub_duration":1541,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280559616,"num_objects":4281,"num_object_clones":0,"num_object_copies":12843,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11249,"num_write_kb":496960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"20'137","reported_seq":1606,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:35.892059+0000","last_change":"2024-03-20T09:37:45.841316+0000","last_active":"2024-03-20T09:38:35.892059+0000","last_peered":"2024-03-20T09:38:35.892059+0000","last_clean":"2024-03-20T09:38:35.892059+0000","last_became_active":"2024-03-20T09:15:34.225022+0000","last_became_peered":"2024-03-20T09:15:34.225022+0000","last_unstale":"2024-03-20T09:38:35.892059+0000","last_undegraded":"2024-03-20T09:38:35.892059+0000","last_fullsized":"2024-03-20T09:38:35.892059+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:45.841176+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:20:12.169658+0000","last_clean_scrub_stamp":"2024-03-20T09:37:45.841176+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:11.852244+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,5],"acting":[6,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"1030'11283","reported_seq":11476,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:36.984668+0000","last_change":"2024-03-20T09:37:04.086352+0000","last_active":"2024-03-20T09:38:36.984668+0000","last_peered":"2024-03-20T09:38:36.984668+0000","last_clean":"2024-03-20T09:38:36.984668+0000","last_became_active":"2024-03-20T09:35:51.943717+0000","last_became_peered":"2024-03-20T09:35:51.943717+0000","last_unstale":"2024-03-20T09:38:36.984668+0000","last_undegraded":"2024-03-20T09:38:36.984668+0000","last_fullsized":"2024-03-20T09:38:36.984668+0000","mapping_epoch":974,"log_start":"980'1200","ondisk_log_start":"980'1200","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1006'6187","last_scrub_stamp":"2024-03-20T09:37:04.086220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:04.086220+0000","objects_scrubbed":6142,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263520256,"num_objects":4021,"num_object_clones":0,"num_object_copies":12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11283,"num_write_kb":489728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"20'137","reported_seq":2153,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:35.892233+0000","last_change":"2024-03-20T09:37:45.012425+0000","last_active":"2024-03-20T09:38:35.892233+0000","last_peered":"2024-03-20T09:38:35.892233+0000","last_clean":"2024-03-20T09:38:35.892233+0000","last_became_active":"2024-03-20T09:35:19.820396+0000","last_became_peered":"2024-03-20T09:35:19.820396+0000","last_unstale":"2024-03-20T09:38:35.892233+0000","last_undegraded":"2024-03-20T09:38:35.892233+0000","last_fullsized":"2024-03-20T09:38:35.892233+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:45.012294+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:14:10.560890+0000","last_clean_scrub_stamp":"2024-03-20T09:37:45.012294+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:57.274595+0000","scrub_duration":222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2","version":"1030'11448","reported_seq":11651,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:38.776022+0000","last_change":"2024-03-20T09:37:39.223390+0000","last_active":"2024-03-20T09:38:38.776022+0000","last_peered":"2024-03-20T09:38:38.776022+0000","last_clean":"2024-03-20T09:38:38.776022+0000","last_became_active":"2024-03-20T09:35:51.942847+0000","last_became_peered":"2024-03-20T09:35:51.942847+0000","last_unstale":"2024-03-20T09:38:38.776022+0000","last_undegraded":"2024-03-20T09:38:38.776022+0000","last_fullsized":"2024-03-20T09:38:38.776022+0000","mapping_epoch":974,"log_start":"980'1400","ondisk_log_start":"980'1400","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1016'8306","last_scrub_stamp":"2024-03-20T09:37:39.223182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:39.223182+0000","objects_scrubbed":7252,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:51.271166+0000","scrub_duration":1777,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266862592,"num_objects":4072,"num_object_clones":0,"num_object_copies":12216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11448,"num_write_kb":496640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,4],"acting":[10,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.b","version":"20'137","reported_seq":1970,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:35.716167+0000","last_change":"2024-03-20T09:37:06.065195+0000","last_active":"2024-03-20T09:38:35.716167+0000","last_peered":"2024-03-20T09:38:35.716167+0000","last_clean":"2024-03-20T09:38:35.716167+0000","last_became_active":"2024-03-20T09:13:20.070596+0000","last_became_peered":"2024-03-20T09:13:20.070596+0000","last_unstale":"2024-03-20T09:38:35.716167+0000","last_undegraded":"2024-03-20T09:38:35.716167+0000","last_fullsized":"2024-03-20T09:38:35.716167+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:06.065105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:37:06.065105+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":316,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.7","version":"1029'11216","reported_seq":11413,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:35.716424+0000","last_change":"2024-03-20T09:37:51.929007+0000","last_active":"2024-03-20T09:38:35.716424+0000","last_peered":"2024-03-20T09:38:35.716424+0000","last_clean":"2024-03-20T09:38:35.716424+0000","last_became_active":"2024-03-20T09:35:51.673870+0000","last_became_peered":"2024-03-20T09:35:51.673870+0000","last_unstale":"2024-03-20T09:38:35.716424+0000","last_undegraded":"2024-03-20T09:38:35.716424+0000","last_fullsized":"2024-03-20T09:38:35.716424+0000","mapping_epoch":974,"log_start":"980'1200","ondisk_log_start":"980'1200","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1020'8885","last_scrub_stamp":"2024-03-20T09:37:51.928857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:51.928857+0000","objects_scrubbed":6515,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:10.833584+0000","scrub_duration":1835,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271319040,"num_objects":4140,"num_object_clones":0,"num_object_copies":12420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11216,"num_write_kb":491392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.e","version":"20'137","reported_seq":1410,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:35.704498+0000","last_change":"2024-03-20T09:37:42.890342+0000","last_active":"2024-03-20T09:38:35.704498+0000","last_peered":"2024-03-20T09:38:35.704498+0000","last_clean":"2024-03-20T09:38:35.704498+0000","last_became_active":"2024-03-20T09:19:54.506134+0000","last_became_peered":"2024-03-20T09:19:54.506134+0000","last_unstale":"2024-03-20T09:38:35.704498+0000","last_undegraded":"2024-03-20T09:38:35.704498+0000","last_fullsized":"2024-03-20T09:38:35.704498+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:42.890257+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:29:53.362224+0000","last_clean_scrub_stamp":"2024-03-20T09:37:42.890257+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:47.227463+0000","scrub_duration":157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"1030'11324","reported_seq":11525,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:35.718660+0000","last_change":"2024-03-20T09:37:44.856776+0000","last_active":"2024-03-20T09:38:35.718660+0000","last_peered":"2024-03-20T09:38:35.718660+0000","last_clean":"2024-03-20T09:38:35.718660+0000","last_became_active":"2024-03-20T09:35:51.312260+0000","last_became_peered":"2024-03-20T09:35:51.312260+0000","last_unstale":"2024-03-20T09:38:35.718660+0000","last_undegraded":"2024-03-20T09:38:35.718660+0000","last_fullsized":"2024-03-20T09:38:35.718660+0000","mapping_epoch":974,"log_start":"980'1300","ondisk_log_start":"980'1300","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1018'8599","last_scrub_stamp":"2024-03-20T09:37:44.856610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:44.856610+0000","objects_scrubbed":6942,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:52.896041+0000","scrub_duration":1869,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272760832,"num_objects":4162,"num_object_clones":0,"num_object_copies":12486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11324,"num_write_kb":495552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,7],"acting":[11,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.f","version":"20'137","reported_seq":2173,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:35.892140+0000","last_change":"2024-03-20T09:38:01.026424+0000","last_active":"2024-03-20T09:38:35.892140+0000","last_peered":"2024-03-20T09:38:35.892140+0000","last_clean":"2024-03-20T09:38:35.892140+0000","last_became_active":"2024-03-20T09:19:54.524397+0000","last_became_peered":"2024-03-20T09:19:54.524397+0000","last_unstale":"2024-03-20T09:38:35.892140+0000","last_undegraded":"2024-03-20T09:38:35.892140+0000","last_fullsized":"2024-03-20T09:38:35.892140+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:38:01.026288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:38:01.026288+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:09.847998+0000","scrub_duration":214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.5","version":"1030'11349","reported_seq":11588,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:38.821539+0000","last_change":"2024-03-20T09:38:36.409170+0000","last_active":"2024-03-20T09:38:38.821539+0000","last_peered":"2024-03-20T09:38:38.821539+0000","last_clean":"2024-03-20T09:38:38.821539+0000","last_became_active":"2024-03-20T09:35:51.942869+0000","last_became_peered":"2024-03-20T09:35:51.942869+0000","last_unstale":"2024-03-20T09:38:38.821539+0000","last_undegraded":"2024-03-20T09:38:38.821539+0000","last_fullsized":"2024-03-20T09:38:38.821539+0000","mapping_epoch":974,"log_start":"980'1300","ondisk_log_start":"980'1300","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1030'11194","last_scrub_stamp":"2024-03-20T09:38:36.408947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:38:36.408947+0000","objects_scrubbed":4230,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:57.457124+0000","scrub_duration":1034,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265879552,"num_objects":4057,"num_object_clones":0,"num_object_copies":12171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11349,"num_write_kb":492992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,1],"acting":[10,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.c","version":"570'138","reported_seq":2149,"reported_epoch":1029,"state":"active+clean","last_fresh":"2024-03-20T09:38:32.704271+0000","last_change":"2024-03-20T09:37:40.922108+0000","last_active":"2024-03-20T09:38:32.704271+0000","last_peered":"2024-03-20T09:38:32.704271+0000","last_clean":"2024-03-20T09:38:32.704271+0000","last_became_active":"2024-03-20T09:13:27.087686+0000","last_became_peered":"2024-03-20T09:13:27.087686+0000","last_unstale":"2024-03-20T09:38:32.704271+0000","last_undegraded":"2024-03-20T09:38:32.704271+0000","last_fullsized":"2024-03-20T09:38:32.704271+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":4,"last_scrub":"570'138","last_scrub_stamp":"2024-03-20T09:37:40.921974+0000","last_deep_scrub":"570'138","last_deep_scrub_stamp":"2024-03-20T09:16:13.725035+0000","last_clean_scrub_stamp":"2024-03-20T09:37:40.921974+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:43.602278+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"1029'11015","reported_seq":11205,"reported_epoch":1029,"state":"active+clean","last_fresh":"2024-03-20T09:38:35.021341+0000","last_change":"2024-03-20T09:37:04.918258+0000","last_active":"2024-03-20T09:38:35.021341+0000","last_peered":"2024-03-20T09:38:35.021341+0000","last_clean":"2024-03-20T09:38:35.021341+0000","last_became_active":"2024-03-20T09:35:51.277801+0000","last_became_peered":"2024-03-20T09:35:51.277801+0000","last_unstale":"2024-03-20T09:38:35.021341+0000","last_undegraded":"2024-03-20T09:38:35.021341+0000","last_fullsized":"2024-03-20T09:38:35.021341+0000","mapping_epoch":974,"log_start":"980'1000","ondisk_log_start":"980'1000","created":974,"last_epoch_clean":975,"parent":"0.0","parent_split_bits":0,"last_scrub":"1006'6159","last_scrub_stamp":"2024-03-20T09:37:04.918103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:35:50.257864+0000","last_clean_scrub_stamp":"2024-03-20T09:37:04.918103+0000","objects_scrubbed":6123,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1242,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269811712,"num_objects":4117,"num_object_clones":0,"num_object_copies":12351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11015,"num_write_kb":484224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,10],"acting":[5,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"20'137","reported_seq":1724,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:35.706811+0000","last_change":"2024-03-20T09:38:01.008767+0000","last_active":"2024-03-20T09:38:35.706811+0000","last_peered":"2024-03-20T09:38:35.706811+0000","last_clean":"2024-03-20T09:38:35.706811+0000","last_became_active":"2024-03-20T09:19:54.502529+0000","last_became_peered":"2024-03-20T09:19:54.502529+0000","last_unstale":"2024-03-20T09:38:35.706811+0000","last_undegraded":"2024-03-20T09:38:35.706811+0000","last_fullsized":"2024-03-20T09:38:35.706811+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:38:01.008634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:38:01.008634+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:09.649906+0000","scrub_duration":168,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"20'137","reported_seq":2011,"reported_epoch":1029,"state":"active+clean","last_fresh":"2024-03-20T09:38:34.930329+0000","last_change":"2024-03-20T09:38:34.930329+0000","last_active":"2024-03-20T09:38:34.930329+0000","last_peered":"2024-03-20T09:38:34.930329+0000","last_clean":"2024-03-20T09:38:34.930329+0000","last_became_active":"2024-03-20T09:10:34.096758+0000","last_became_peered":"2024-03-20T09:10:34.096758+0000","last_unstale":"2024-03-20T09:38:34.930329+0000","last_undegraded":"2024-03-20T09:38:34.930329+0000","last_fullsized":"2024-03-20T09:38:34.930329+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:38:34.930215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:38:34.930215+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:50.113701+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"20'137","reported_seq":766,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:35.716419+0000","last_change":"2024-03-20T09:37:52.360050+0000","last_active":"2024-03-20T09:38:35.716419+0000","last_peered":"2024-03-20T09:38:35.716419+0000","last_clean":"2024-03-20T09:38:35.716419+0000","last_became_active":"2024-03-20T09:35:19.822242+0000","last_became_peered":"2024-03-20T09:35:19.822242+0000","last_unstale":"2024-03-20T09:38:35.716419+0000","last_undegraded":"2024-03-20T09:38:35.716419+0000","last_fullsized":"2024-03-20T09:38:35.716419+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:52.359932+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:37:52.359932+0000","last_clean_scrub_stamp":"2024-03-20T09:37:52.359932+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:00.922956+0000","scrub_duration":315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.10","version":"20'137","reported_seq":1566,"reported_epoch":1029,"state":"active+clean","last_fresh":"2024-03-20T09:38:32.697095+0000","last_change":"2024-03-20T09:37:38.621260+0000","last_active":"2024-03-20T09:38:32.697095+0000","last_peered":"2024-03-20T09:38:32.697095+0000","last_clean":"2024-03-20T09:38:32.697095+0000","last_became_active":"2024-03-20T09:35:19.820595+0000","last_became_peered":"2024-03-20T09:35:19.820595+0000","last_unstale":"2024-03-20T09:38:32.697095+0000","last_undegraded":"2024-03-20T09:38:32.697095+0000","last_fullsized":"2024-03-20T09:38:32.697095+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:38.621168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:37:38.621168+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:38:43.836018+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"20'137","reported_seq":1948,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:35.708597+0000","last_change":"2024-03-20T09:36:51.839369+0000","last_active":"2024-03-20T09:38:35.708597+0000","last_peered":"2024-03-20T09:38:35.708597+0000","last_clean":"2024-03-20T09:38:35.708597+0000","last_became_active":"2024-03-20T09:20:25.342982+0000","last_became_peered":"2024-03-20T09:20:25.342982+0000","last_unstale":"2024-03-20T09:38:35.708597+0000","last_undegraded":"2024-03-20T09:38:35.708597+0000","last_fullsized":"2024-03-20T09:38:35.708597+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:36:51.839234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:36:51.839234+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"20'137","reported_seq":674,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:35.708963+0000","last_change":"2024-03-20T09:36:54.828126+0000","last_active":"2024-03-20T09:38:35.708963+0000","last_peered":"2024-03-20T09:38:35.708963+0000","last_clean":"2024-03-20T09:38:35.708963+0000","last_became_active":"2024-03-20T09:31:07.424851+0000","last_became_peered":"2024-03-20T09:31:07.424851+0000","last_unstale":"2024-03-20T09:38:35.708963+0000","last_undegraded":"2024-03-20T09:38:35.708963+0000","last_fullsized":"2024-03-20T09:38:35.708963+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:36:54.827943+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:35:27.264203+0000","last_clean_scrub_stamp":"2024-03-20T09:36:54.827943+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":174,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.17","version":"20'137","reported_seq":2145,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:35.707933+0000","last_change":"2024-03-20T09:38:01.420016+0000","last_active":"2024-03-20T09:38:35.707933+0000","last_peered":"2024-03-20T09:38:35.707933+0000","last_clean":"2024-03-20T09:38:35.707933+0000","last_became_active":"2024-03-20T09:35:19.822429+0000","last_became_peered":"2024-03-20T09:35:19.822429+0000","last_unstale":"2024-03-20T09:38:35.707933+0000","last_undegraded":"2024-03-20T09:38:35.707933+0000","last_fullsized":"2024-03-20T09:38:35.707933+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:38:01.419884+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:38:01.419884+0000","last_clean_scrub_stamp":"2024-03-20T09:38:01.419884+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:22.126398+0000","scrub_duration":169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"20'137","reported_seq":2185,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:35.714967+0000","last_change":"2024-03-20T09:38:33.049757+0000","last_active":"2024-03-20T09:38:35.714967+0000","last_peered":"2024-03-20T09:38:35.714967+0000","last_clean":"2024-03-20T09:38:35.714967+0000","last_became_active":"2024-03-20T09:20:25.349082+0000","last_became_peered":"2024-03-20T09:20:25.349082+0000","last_unstale":"2024-03-20T09:38:35.714967+0000","last_undegraded":"2024-03-20T09:38:35.714967+0000","last_fullsized":"2024-03-20T09:38:35.714967+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:38:33.049631+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:36:56.203907+0000","last_clean_scrub_stamp":"2024-03-20T09:38:33.049631+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:40:02.169945+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"20'137","reported_seq":1457,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:35.722945+0000","last_change":"2024-03-20T09:37:36.549435+0000","last_active":"2024-03-20T09:38:35.722945+0000","last_peered":"2024-03-20T09:38:35.722945+0000","last_clean":"2024-03-20T09:38:35.722945+0000","last_became_active":"2024-03-20T09:12:14.307581+0000","last_became_peered":"2024-03-20T09:12:14.307581+0000","last_unstale":"2024-03-20T09:38:35.722945+0000","last_undegraded":"2024-03-20T09:38:35.722945+0000","last_fullsized":"2024-03-20T09:38:35.722945+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:37:36.549355+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:34:48.079272+0000","last_clean_scrub_stamp":"2024-03-20T09:37:36.549355+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:04.698728+0000","scrub_duration":206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"20'137","reported_seq":1723,"reported_epoch":1030,"state":"active+clean","last_fresh":"2024-03-20T09:38:35.704327+0000","last_change":"2024-03-20T09:38:01.843191+0000","last_active":"2024-03-20T09:38:35.704327+0000","last_peered":"2024-03-20T09:38:35.704327+0000","last_clean":"2024-03-20T09:38:35.704327+0000","last_became_active":"2024-03-20T09:19:54.502391+0000","last_became_peered":"2024-03-20T09:19:54.502391+0000","last_unstale":"2024-03-20T09:38:35.704327+0000","last_undegraded":"2024-03-20T09:38:35.704327+0000","last_fullsized":"2024-03-20T09:38:35.704327+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:38:01.843093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:38:01.843093+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:39:31.633431+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":4320264216,"num_objects":65923,"num_object_clones":0,"num_object_copies":197769,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65923,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180681,"num_write_kb":7891265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12880982016,"data_stored":12880969800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160773,"ondisk_log_size":160773,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4248,"ondisk_log_size":4248,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860928,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2287172,"kb_used_data":1339384,"kb_used_omap":1,"kb_used_meta":947454,"kb_avail":91457980,"statfs":{"total":95995035648,"available":93652971520,"internally_reserved":0,"allocated":1371529216,"data_stored":1371248921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":970193348},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":861,"seq":3697966841953,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1726092,"kb_used_data":1553472,"kb_used_omap":103,"kb_used_meta":172504,"kb_avail":92019060,"statfs":{"total":95995035648,"available":94227517440,"internally_reserved":0,"allocated":1590755328,"data_stored":1590474640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":106169,"internal_metadata":176644423},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":80,"seq":343597384205,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2138824,"kb_used_data":1561984,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":91606328,"statfs":{"total":95995035648,"available":93804879872,"internally_reserved":0,"allocated":1599471616,"data_stored":1599190928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":590608832},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":70,"seq":300647711249,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1381772,"kb_used_data":800768,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":92363380,"statfs":{"total":95995035648,"available":94580101120,"internally_reserved":0,"allocated":819986432,"data_stored":819705744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":594868678},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":60,"seq":257698038293,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1434120,"kb_used_data":536640,"kb_used_omap":1,"kb_used_meta":897470,"kb_avail":92311032,"statfs":{"total":95995035648,"available":94526496768,"internally_reserved":0,"allocated":549519360,"data_stored":549235513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":919009738},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332634,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":704128,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93041024,"statfs":{"total":95995035648,"available":95274008576,"internally_reserved":0,"allocated":1036288,"data_stored":760065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":719976902},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575413,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2498248,"kb_used_data":1538304,"kb_used_omap":1,"kb_used_meta":959934,"kb_avail":91246904,"statfs":{"total":95995035648,"available":93436829696,"internally_reserved":0,"allocated":1575223296,"data_stored":1574942608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":982972870},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738931,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1819788,"kb_used_data":791808,"kb_used_omap":1,"kb_used_meta":1027966,"kb_avail":91925364,"statfs":{"total":95995035648,"available":94131572736,"internally_reserved":0,"allocated":810811392,"data_stored":810530704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1052637637},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411885,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2114004,"kb_used_data":1292812,"kb_used_omap":1,"kb_used_meta":821118,"kb_avail":91631148,"statfs":{"total":95995035648,"available":93830295552,"internally_reserved":0,"allocated":1323839488,"data_stored":1323551176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":840825285},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150249,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1798208,"kb_used_data":814580,"kb_used_omap":1,"kb_used_meta":983550,"kb_avail":91946944,"statfs":{"total":95995035648,"available":94153670656,"internally_reserved":0,"allocated":834129920,"data_stored":833853697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1007155655},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438954024,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1831932,"kb_used_data":1078836,"kb_used_omap":1,"kb_used_meta":753086,"kb_avail":91913220,"statfs":{"total":95995035648,"available":94119137280,"internally_reserved":0,"allocated":1104728064,"data_stored":1104451841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":771160517},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":443,"seq":1902670512465,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2095044,"kb_used_data":1357052,"kb_used_omap":181,"kb_used_meta":737802,"kb_avail":91650108,"statfs":{"total":95995035648,"available":93849710592,"internally_reserved":0,"allocated":1389621248,"data_stored":1389341473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":186131,"internal_metadata":755509485},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":807600128,"data_stored":807600128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1557069824,"data_stored":1557069824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1314721792,"data_stored":1314717720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":833028096,"data_stored":833028096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1095761920,"data_stored":1095761920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1385627648,"data_stored":1385627648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":544215040,"data_stored":544210968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":813367296,"data_stored":813367296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1590099968,"data_stored":1590099968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1580269568,"data_stored":1580269568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1359220736,"data_stored":1359216664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:38:42.649 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pool set unique_pool_6 pgp_num 16 2024-03-20T09:38:42.730 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:38:42.811 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:42.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:42 smithi026 ceph-mon[68309]: 1.2 scrub starts 2024-03-20T09:38:42.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:42 smithi026 ceph-mon[68309]: 1.2 scrub ok 2024-03-20T09:38:43.110 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:38:43.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:43.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:38:43.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:42 smithi115 ceph-mon[72476]: 1.2 scrub starts 2024-03-20T09:38:43.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:42 smithi115 ceph-mon[72476]: 1.2 scrub ok 2024-03-20T09:38:43.152 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:38:43.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:42 smithi122 ceph-mon[73495]: 1.2 scrub starts 2024-03-20T09:38:43.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:42 smithi122 ceph-mon[73495]: 1.2 scrub ok 2024-03-20T09:38:43.216 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:38:43.244 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:43.796 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:43 smithi026 ceph-mon[68309]: pgmap v2494: 47 pgs: 47 active+clean; 3.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 852 op/s 2024-03-20T09:38:43.797 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:43 smithi026 ceph-mon[68309]: 1.11 scrub starts 2024-03-20T09:38:43.797 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:43 smithi026 ceph-mon[68309]: 1.11 scrub ok 2024-03-20T09:38:44.035 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:38:44.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:44.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:38:44.046 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:38:44.143 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:38:44.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:43 smithi115 ceph-mon[72476]: pgmap v2494: 47 pgs: 47 active+clean; 3.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 852 op/s 2024-03-20T09:38:44.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:43 smithi115 ceph-mon[72476]: 1.11 scrub starts 2024-03-20T09:38:44.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:43 smithi115 ceph-mon[72476]: 1.11 scrub ok 2024-03-20T09:38:44.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:43 smithi122 ceph-mon[73495]: pgmap v2494: 47 pgs: 47 active+clean; 3.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 852 op/s 2024-03-20T09:38:44.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:43 smithi122 ceph-mon[73495]: 1.11 scrub starts 2024-03-20T09:38:44.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:43 smithi122 ceph-mon[73495]: 1.11 scrub ok 2024-03-20T09:38:44.200 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:44.246 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:38:44.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:44.252 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:38:44.353 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:38:44.384 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:44.839 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:38:44.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:44.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:38:44.950 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:38:44.978 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:45.003 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:38:45.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:45.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:38:45.109 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:38:45.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:44 smithi115 ceph-mon[72476]: 1.4 scrub starts 2024-03-20T09:38:45.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:44 smithi115 ceph-mon[72476]: 1.4 scrub ok 2024-03-20T09:38:45.164 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:44 smithi026 ceph-mon[68309]: 1.4 scrub starts 2024-03-20T09:38:45.164 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:44 smithi026 ceph-mon[68309]: 1.4 scrub ok 2024-03-20T09:38:45.168 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:45.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:44 smithi122 ceph-mon[73495]: 1.4 scrub starts 2024-03-20T09:38:45.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:44 smithi122 ceph-mon[73495]: 1.4 scrub ok 2024-03-20T09:38:45.204 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:38:45.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:45.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:38:45.309 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:38:45.349 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:45.385 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:38:45.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:45.390 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:38:45.491 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:38:45.535 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:45.804 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:38:46.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:45 smithi115 ceph-mon[72476]: 1.b scrub starts 2024-03-20T09:38:46.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:45 smithi115 ceph-mon[72476]: 1.b scrub ok 2024-03-20T09:38:46.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:45 smithi115 ceph-mon[72476]: pgmap v2495: 47 pgs: 47 active+clean; 3.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 821 op/s 2024-03-20T09:38:46.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:45 smithi115 ceph-mon[72476]: 1.16 scrub starts 2024-03-20T09:38:46.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:45 smithi115 ceph-mon[72476]: 1.10 scrub starts 2024-03-20T09:38:46.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:45 smithi115 ceph-mon[72476]: 1.10 scrub ok 2024-03-20T09:38:46.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:45 smithi115 ceph-mon[72476]: 1.16 scrub ok 2024-03-20T09:38:46.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:45 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3658377978' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:38:46.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:45 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:38:46.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:45 smithi122 ceph-mon[73495]: 1.b scrub starts 2024-03-20T09:38:46.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:45 smithi122 ceph-mon[73495]: 1.b scrub ok 2024-03-20T09:38:46.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:45 smithi122 ceph-mon[73495]: pgmap v2495: 47 pgs: 47 active+clean; 3.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 821 op/s 2024-03-20T09:38:46.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:45 smithi122 ceph-mon[73495]: 1.16 scrub starts 2024-03-20T09:38:46.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:45 smithi122 ceph-mon[73495]: 1.10 scrub starts 2024-03-20T09:38:46.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:45 smithi122 ceph-mon[73495]: 1.10 scrub ok 2024-03-20T09:38:46.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:45 smithi122 ceph-mon[73495]: 1.16 scrub ok 2024-03-20T09:38:46.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:45 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3658377978' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:38:46.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:45 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:38:46.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:45 smithi026 ceph-mon[68309]: 1.b scrub starts 2024-03-20T09:38:46.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:45 smithi026 ceph-mon[68309]: 1.b scrub ok 2024-03-20T09:38:46.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:45 smithi026 ceph-mon[68309]: pgmap v2495: 47 pgs: 47 active+clean; 3.8 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 821 op/s 2024-03-20T09:38:46.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:45 smithi026 ceph-mon[68309]: 1.16 scrub starts 2024-03-20T09:38:46.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:45 smithi026 ceph-mon[68309]: 1.10 scrub starts 2024-03-20T09:38:46.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:45 smithi026 ceph-mon[68309]: 1.10 scrub ok 2024-03-20T09:38:46.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:45 smithi026 ceph-mon[68309]: 1.16 scrub ok 2024-03-20T09:38:46.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:45 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3658377978' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:38:46.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:45 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:38:46.309 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:38:46.310 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:38:46.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:38:46.310 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26917.0:186815 8.c e1031/974 8:343b2f07:::benchmark_data_smithi122_197775_object63511:head v 1031'11792, mlcod=1031'11790)", 2024-03-20T09:38:46.310 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:38:44.535132+0000", 2024-03-20T09:38:46.310 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0020434440000000002, 2024-03-20T09:38:46.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0020675419999999999, 2024-03-20T09:38:46.310 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:38:46.310 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:38:46.310 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:38:46.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:38:46.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:38:46.311 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:44.535132+0000", 2024-03-20T09:38:46.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:38:46.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:38:46.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:38:46.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:38:46.311 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:44.535132+0000", 2024-03-20T09:38:46.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:38:46.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:38:46.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:38:46.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:38:46.311 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:44.535133+0000", 2024-03-20T09:38:46.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5099999999999998e-07 2024-03-20T09:38:46.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:38:46.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:38:46.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:38:46.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:44.535139+0000", 2024-03-20T09:38:46.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.049e-06 2024-03-20T09:38:46.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:38:46.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:38:46.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:38:46.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:44.535152+0000", 2024-03-20T09:38:46.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2631e-05 2024-03-20T09:38:46.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:38:46.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:38:46.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:38:46.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:44.535159+0000", 2024-03-20T09:38:46.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1389999999999998e-06 2024-03-20T09:38:46.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:38:46.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:38:46.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:38:46.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:44.535213+0000", 2024-03-20T09:38:46.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3851000000000003e-05 2024-03-20T09:38:46.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:38:46.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:38:46.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:38:46.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:44.535235+0000", 2024-03-20T09:38:46.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2188000000000002e-05 2024-03-20T09:38:46.313 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:38:46.313 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:38:46.313 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:38:46.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:38:46.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:38:46.314 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26917.0:186818 8.8 e1031/974)", 2024-03-20T09:38:46.314 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:38:44.536289+0000", 2024-03-20T09:38:46.314 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.00088706099999999997, 2024-03-20T09:38:46.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00095009000000000001, 2024-03-20T09:38:46.314 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:38:46.314 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "queued for pg", 2024-03-20T09:38:46.314 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:38:46.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:38:46.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:38:46.315 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:44.536289+0000", 2024-03-20T09:38:46.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:38:46.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:38:46.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:38:46.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:38:46.315 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:44.536289+0000", 2024-03-20T09:38:46.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:38:46.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:38:46.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:38:46.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:38:46.315 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:44.536289+0000", 2024-03-20T09:38:46.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9599999999999998e-07 2024-03-20T09:38:46.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:38:46.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:38:46.316 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:38:46.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:44.536296+0000", 2024-03-20T09:38:46.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3579999999999999e-06 2024-03-20T09:38:46.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:38:46.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:38:46.316 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:38:46.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:44.536309+0000", 2024-03-20T09:38:46.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2811e-05 2024-03-20T09:38:46.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:38:46.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:38:46.316 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:38:46.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:44.536316+0000", 2024-03-20T09:38:46.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0190000000000001e-06 2024-03-20T09:38:46.316 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:38:46.317 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:38:46.317 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:38:46.317 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:38:46.317 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:38:46.317 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 2 2024-03-20T09:38:46.317 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:38:46.505 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:38:46.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:46.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:38:46.614 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:38:46.677 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:46.708 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:38:46.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:46.715 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:38:46.815 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:46 smithi026 ceph-mon[68309]: 1.c scrub starts 2024-03-20T09:38:46.816 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:46 smithi026 ceph-mon[68309]: 1.c scrub ok 2024-03-20T09:38:46.816 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:46 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:38:46.816 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:46 smithi026 ceph-mon[68309]: Cluster is now healthy 2024-03-20T09:38:46.816 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:46 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:38:46.816 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:46 smithi026 ceph-mon[68309]: osdmap e1032: 12 total, 12 up, 12 in 2024-03-20T09:38:46.816 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:38:46.860 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:46.935 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:46 smithi122 ceph-mon[73495]: 1.c scrub starts 2024-03-20T09:38:46.935 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:46 smithi122 ceph-mon[73495]: 1.c scrub ok 2024-03-20T09:38:46.935 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:46 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:38:46.935 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:46 smithi122 ceph-mon[73495]: Cluster is now healthy 2024-03-20T09:38:46.935 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:46 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:38:46.935 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:46 smithi122 ceph-mon[73495]: osdmap e1032: 12 total, 12 up, 12 in 2024-03-20T09:38:47.012 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:38:47.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:47.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:38:47.055 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-20T09:38:47.121 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:38:47.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:46 smithi115 ceph-mon[72476]: 1.c scrub starts 2024-03-20T09:38:47.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:46 smithi115 ceph-mon[72476]: 1.c scrub ok 2024-03-20T09:38:47.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:46 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:38:47.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:46 smithi115 ceph-mon[72476]: Cluster is now healthy 2024-03-20T09:38:47.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:46 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:38:47.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:46 smithi115 ceph-mon[72476]: osdmap e1032: 12 total, 12 up, 12 in 2024-03-20T09:38:47.158 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:47.194 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:38:47.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:47.205 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:38:47.306 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:38:47.350 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:47.381 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:38:47.391 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:47.392 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:38:47.493 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:38:47.552 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:47.593 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:38:47.600 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:47.600 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:38:47.701 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:38:47.730 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:47.753 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:38:47.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:47.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:38:47.833 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:47 smithi026 ceph-mon[68309]: pgmap v2497: 47 pgs: 47 active+clean; 3.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 842 op/s 2024-03-20T09:38:47.842 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:47 smithi122 ceph-mon[73495]: pgmap v2497: 47 pgs: 47 active+clean; 3.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 842 op/s 2024-03-20T09:38:47.858 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:38:47.899 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:47.925 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:38:47.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:47.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:38:48.030 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:38:48.057 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:48.076 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:38:48.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:48.079 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:38:48.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:47 smithi115 ceph-mon[72476]: pgmap v2497: 47 pgs: 47 active+clean; 3.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 842 op/s 2024-03-20T09:38:48.180 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:38:48.209 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:48.240 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:38:48.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:48.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:38:48.346 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:38:48.405 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:48.435 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:38:48.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:48.441 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:38:48.542 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:38:48.571 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:48.597 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:38:48.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:48.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:38:48.703 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:38:48.731 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:48.751 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:38:48.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:48.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:38:48.856 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:38:48.917 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:48.945 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:38:48.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:48.955 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:38:49.056 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:38:49.085 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:49.107 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:38:49.113 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:48 smithi115 ceph-mon[72476]: 8.1 scrub starts 2024-03-20T09:38:49.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:49.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:38:49.128 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:48 smithi026 ceph-mon[68309]: 8.1 scrub starts 2024-03-20T09:38:49.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:48 smithi122 ceph-mon[73495]: 8.1 scrub starts 2024-03-20T09:38:49.215 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:38:49.224 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:38:49.258 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:49.292 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:38:49.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:49.310 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:38:49.411 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:38:49.447 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:49.469 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:38:49.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:49.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:38:49.574 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:38:49.602 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:49.624 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:38:49.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:49.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:38:49.730 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:38:49.793 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:49.834 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:38:49.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:49.838 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:38:49.939 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:38:49.950 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:38:49.969 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:49.990 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:38:49.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:49.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:38:50.050 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:49 smithi026 ceph-mon[68309]: pgmap v2498: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 815 op/s 2024-03-20T09:38:50.050 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:49 smithi026 ceph-mon[68309]: 8.1 scrub ok 2024-03-20T09:38:50.094 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:38:50.122 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:49 smithi115 ceph-mon[72476]: pgmap v2498: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 815 op/s 2024-03-20T09:38:50.122 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:49 smithi115 ceph-mon[72476]: 8.1 scrub ok 2024-03-20T09:38:50.124 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:50.146 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:38:50.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:50.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:38:50.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:49 smithi122 ceph-mon[73495]: pgmap v2498: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 815 op/s 2024-03-20T09:38:50.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:49 smithi122 ceph-mon[73495]: 8.1 scrub ok 2024-03-20T09:38:50.250 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:38:50.310 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:50.332 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:38:50.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:50.336 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:38:50.437 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:38:50.469 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:50.494 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:38:50.496 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:50.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:38:50.597 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:38:50.639 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:50.671 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:38:50.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:50.675 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:38:50.776 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:38:50.830 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:50.869 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:38:50.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:50.873 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:38:50.885 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:50 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3604880779' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:38:50.908 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:38:50.908 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:38:50.909 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:38:50.909 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:38:50.909 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:38:50.909 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:38:50.909 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:38:50.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:50 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3604880779' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:38:50.937 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:38:50.974 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:38:50.996 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:38:51.001 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:51.011 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:38:51.022 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:38:51.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:51.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:38:51.073 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:38:51.101 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:38:51.128 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:38:51.140 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:38:51.147 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:38:51.147 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:38:51.147 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:38:51.147 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:38:51.147 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:38:51.147 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:38:51.147 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:38:51.147 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:38:51.147 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:38:51.147 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:38:51.148 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:38:51.148 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:38:51.155 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:51.179 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:38:51.181 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:50 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3604880779' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:38:51.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:51.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:38:51.285 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:38:51.314 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:51.335 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:38:51.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:51.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:38:51.438 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:38:51.491 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:51.514 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:38:51.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:51.519 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:38:51.620 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:38:51.653 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:51.692 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:38:51.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:51.696 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:38:51.797 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:38:51.858 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:51.896 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:51 smithi122 ceph-mon[73495]: 1.1e scrub starts 2024-03-20T09:38:51.896 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:51 smithi122 ceph-mon[73495]: 1.1e scrub ok 2024-03-20T09:38:51.896 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:51 smithi122 ceph-mon[73495]: pgmap v2499: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 798 op/s 2024-03-20T09:38:51.896 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:51 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3477848061' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-20T09:38:51.896 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:51 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-20T09:38:51.896 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:51 smithi122 ceph-mon[73495]: 8.2 scrub starts 2024-03-20T09:38:51.896 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:51 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:38:51.896 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:51 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:38:51.948 INFO:teuthology.orchestra.run.smithi026.stderr:set pool 8 pgp_num to 16 2024-03-20T09:38:52.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:51 smithi115 ceph-mon[72476]: 1.1e scrub starts 2024-03-20T09:38:52.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:51 smithi115 ceph-mon[72476]: 1.1e scrub ok 2024-03-20T09:38:52.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:51 smithi115 ceph-mon[72476]: pgmap v2499: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 798 op/s 2024-03-20T09:38:52.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:51 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3477848061' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-20T09:38:52.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:51 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-20T09:38:52.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:51 smithi115 ceph-mon[72476]: 8.2 scrub starts 2024-03-20T09:38:52.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:51 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:38:52.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:51 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:38:52.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:51 smithi026 ceph-mon[68309]: 1.1e scrub starts 2024-03-20T09:38:52.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:51 smithi026 ceph-mon[68309]: 1.1e scrub ok 2024-03-20T09:38:52.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:51 smithi026 ceph-mon[68309]: pgmap v2499: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 798 op/s 2024-03-20T09:38:52.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:51 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3477848061' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-20T09:38:52.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:51 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-20T09:38:52.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:51 smithi026 ceph-mon[68309]: 8.2 scrub starts 2024-03-20T09:38:52.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:51 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:38:52.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:51 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:38:53.171 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:52 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num", "val": "16"}]': finished 2024-03-20T09:38:53.171 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:52 smithi026 ceph-mon[68309]: osdmap e1033: 12 total, 12 up, 12 in 2024-03-20T09:38:53.172 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:52 smithi026 ceph-mon[68309]: 8.2 scrub ok 2024-03-20T09:38:53.172 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:52 smithi026 ceph-mon[68309]: pgmap v2501: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 838 op/s 2024-03-20T09:38:53.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:52 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num", "val": "16"}]': finished 2024-03-20T09:38:53.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:52 smithi122 ceph-mon[73495]: osdmap e1033: 12 total, 12 up, 12 in 2024-03-20T09:38:53.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:52 smithi122 ceph-mon[73495]: 8.2 scrub ok 2024-03-20T09:38:53.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:52 smithi122 ceph-mon[73495]: pgmap v2501: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 838 op/s 2024-03-20T09:38:53.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:52 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_6", "var": "pgp_num", "val": "16"}]': finished 2024-03-20T09:38:53.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:52 smithi115 ceph-mon[72476]: osdmap e1033: 12 total, 12 up, 12 in 2024-03-20T09:38:53.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:52 smithi115 ceph-mon[72476]: 8.2 scrub ok 2024-03-20T09:38:53.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:52 smithi115 ceph-mon[72476]: pgmap v2501: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 838 op/s 2024-03-20T09:38:53.526 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:38:53.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:53.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:38:53.631 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:38:53.722 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:53.757 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:38:53.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:53.766 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:38:53.867 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:38:53.894 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:53.915 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:38:53.918 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:53.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:38:53.925 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:53 smithi026 ceph-mon[68309]: 8.8 scrub starts 2024-03-20T09:38:53.925 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:53 smithi026 ceph-mon[68309]: 1.e scrub starts 2024-03-20T09:38:53.925 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:53 smithi026 ceph-mon[68309]: 8.6 scrub starts 2024-03-20T09:38:53.925 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:53 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:38:53.925 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:53 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:38:54.019 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:38:54.078 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:53 smithi122 ceph-mon[73495]: 8.8 scrub starts 2024-03-20T09:38:54.078 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:53 smithi122 ceph-mon[73495]: 1.e scrub starts 2024-03-20T09:38:54.078 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:53 smithi122 ceph-mon[73495]: 8.6 scrub starts 2024-03-20T09:38:54.078 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:53 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:38:54.078 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:53 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:38:54.081 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:54.131 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:38:54.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:54.142 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:38:54.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:53 smithi115 ceph-mon[72476]: 8.8 scrub starts 2024-03-20T09:38:54.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:53 smithi115 ceph-mon[72476]: 1.e scrub starts 2024-03-20T09:38:54.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:53 smithi115 ceph-mon[72476]: 8.6 scrub starts 2024-03-20T09:38:54.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:53 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:38:54.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:53 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:38:54.243 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:38:54.300 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:54.327 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:38:54.330 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:54.330 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:38:54.431 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:38:54.460 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:54.486 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:38:54.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:54.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:38:54.593 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:38:54.622 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:54.649 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:38:54.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:54.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:38:54.753 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:38:54.794 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:54.880 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:38:54.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:54.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:38:54.989 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:38:55.019 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:55.040 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:38:55.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:55.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:38:55.145 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:38:55.198 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:54 smithi026 ceph-mon[68309]: 1.e scrub ok 2024-03-20T09:38:55.198 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:54 smithi026 ceph-mon[68309]: 8.8 scrub ok 2024-03-20T09:38:55.198 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:54 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:38:55.198 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:54 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:38:55.198 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:54 smithi026 ceph-mon[68309]: osdmap e1034: 12 total, 12 up, 12 in 2024-03-20T09:38:55.198 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:54 smithi026 ceph-mon[68309]: 8.6 scrub ok 2024-03-20T09:38:55.198 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:54 smithi026 ceph-mon[68309]: pgmap v2503: 47 pgs: 47 active+clean; 3.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 742 op/s 2024-03-20T09:38:55.201 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:55.231 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:38:55.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:55.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:38:55.341 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:38:55.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:54 smithi115 ceph-mon[72476]: 1.e scrub ok 2024-03-20T09:38:55.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:54 smithi115 ceph-mon[72476]: 8.8 scrub ok 2024-03-20T09:38:55.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:54 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:38:55.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:54 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:38:55.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:54 smithi115 ceph-mon[72476]: osdmap e1034: 12 total, 12 up, 12 in 2024-03-20T09:38:55.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:54 smithi115 ceph-mon[72476]: 8.6 scrub ok 2024-03-20T09:38:55.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:54 smithi115 ceph-mon[72476]: pgmap v2503: 47 pgs: 47 active+clean; 3.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 742 op/s 2024-03-20T09:38:55.408 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:54 smithi122 ceph-mon[73495]: 1.e scrub ok 2024-03-20T09:38:55.408 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:54 smithi122 ceph-mon[73495]: 8.8 scrub ok 2024-03-20T09:38:55.408 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:54 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:38:55.408 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:54 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:38:55.408 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:54 smithi122 ceph-mon[73495]: osdmap e1034: 12 total, 12 up, 12 in 2024-03-20T09:38:55.408 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:54 smithi122 ceph-mon[73495]: 8.6 scrub ok 2024-03-20T09:38:55.408 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:54 smithi122 ceph-mon[73495]: pgmap v2503: 47 pgs: 47 active+clean; 3.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 742 op/s 2024-03-20T09:38:55.414 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:55.444 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:38:55.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:55.449 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:38:55.550 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:38:55.596 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:55.634 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:38:55.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:55.641 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:38:55.742 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:38:55.789 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:55.842 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:38:55.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:55.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:38:55.957 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:38:55.986 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:56.011 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:38:56.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:56.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:38:56.017 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:38:56.017 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:38:56.077 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:38:56.115 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:38:56.188 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:56.328 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:38:56.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:56.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:38:56.337 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:38:56.432 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:38:56.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:56 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2636188976' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:38:56.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:56 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:38:56.480 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:56.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:56 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2636188976' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:38:56.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:56 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:38:56.685 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:56 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2636188976' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:38:56.685 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:56 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:38:56.833 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-20T09:38:57.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:57 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:38:57.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:57 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:38:57.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:57 smithi026 ceph-mon[68309]: osdmap e1035: 12 total, 12 up, 12 in 2024-03-20T09:38:57.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:57 smithi026 ceph-mon[68309]: pgmap v2505: 47 pgs: 47 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 870 op/s 2024-03-20T09:38:57.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:57 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:38:57.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:57 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:38:57.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:57 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:38:57.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:57 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:38:57.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:57 smithi115 ceph-mon[72476]: osdmap e1035: 12 total, 12 up, 12 in 2024-03-20T09:38:57.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:57 smithi115 ceph-mon[72476]: pgmap v2505: 47 pgs: 47 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 870 op/s 2024-03-20T09:38:57.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:57 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:38:57.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:57 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:38:57.646 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:38:57.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:57.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:38:57.685 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:57 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:38:57.685 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:57 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:38:57.685 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:57 smithi122 ceph-mon[73495]: osdmap e1035: 12 total, 12 up, 12 in 2024-03-20T09:38:57.685 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:57 smithi122 ceph-mon[73495]: pgmap v2505: 47 pgs: 47 active+clean; 3.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 870 op/s 2024-03-20T09:38:57.685 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:57 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:38:57.685 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:57 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:38:57.753 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:38:57.755 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:38:57.791 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:38:57.795 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:57.828 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:38:57.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:57.837 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:38:57.938 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:38:57.983 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:58.015 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:38:58.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:58.025 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:38:58.126 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:38:58.155 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:58.175 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:38:58.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:58.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:38:58.217 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 8, 11, 3, 4, 9, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 11, 5, 10] 2024-03-20T09:38:58.217 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:38:58.217 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-03-20T09:38:58.217 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-20T09:38:58.218 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-20T09:38:58.218 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-03-20T09:38:58.283 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:38:58.347 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:58.378 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:38:58.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:58.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:38:58.492 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:38:58.540 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:38:58.556 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:58.609 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:38:58.620 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:58 smithi122 ceph-mon[73495]: osdmap e1036: 12 total, 12 up, 12 in 2024-03-20T09:38:58.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:58.620 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:38:58.644 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:58 smithi115 ceph-mon[72476]: osdmap e1036: 12 total, 12 up, 12 in 2024-03-20T09:38:58.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:58 smithi026 ceph-mon[68309]: osdmap e1036: 12 total, 12 up, 12 in 2024-03-20T09:38:58.721 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:38:58.748 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:58.869 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:38:58.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:58.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:38:58.976 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:38:59.013 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:59.122 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:38:59.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:59.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:38:59.229 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:38:59.286 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:38:59.568 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:38:59.570 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:38:59 smithi026 ceph-mon[68309]: pgmap v2507: 47 pgs: 47 active+clean; 3.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 748 op/s 2024-03-20T09:38:59.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:38:59.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:38:59.622 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:38:59.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:38:59 smithi115 ceph-mon[72476]: pgmap v2507: 47 pgs: 47 active+clean; 3.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 748 op/s 2024-03-20T09:38:59.672 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:38:59.685 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:38:59 smithi122 ceph-mon[73495]: pgmap v2507: 47 pgs: 47 active+clean; 3.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 748 op/s 2024-03-20T09:38:59.746 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:00.921 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:39:00.921 INFO:teuthology.orchestra.run.smithi115.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-20T09:39:00.921 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:39:01.006 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:39:01.006 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:39:01.016 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:39:01.044 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:39:01.079 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:39:01.116 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:39:01.144 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:39:01.175 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:39:01.209 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:39:01.256 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:39:01.300 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:39:01.335 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:39:01.341 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:39:01.341 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:39:01.341 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:39:01.341 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:39:01.341 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:39:01.341 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:39:01.341 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:39:01.341 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:39:01.341 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:39:01.341 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:39:01.341 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:39:01.342 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:39:01.669 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:01 smithi026 ceph-mon[68309]: pgmap v2508: 47 pgs: 47 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 729 op/s 2024-03-20T09:39:01.669 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:01 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3491528932' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:39:01.672 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:39:01.673 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:39:01.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:01 smithi115 ceph-mon[72476]: pgmap v2508: 47 pgs: 47 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 729 op/s 2024-03-20T09:39:01.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:01 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3491528932' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:39:01.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:01 smithi122 ceph-mon[73495]: pgmap v2508: 47 pgs: 47 active+clean; 3.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 729 op/s 2024-03-20T09:39:01.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:01 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3491528932' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:39:03.250 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:39:03.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:03.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:39:03.287 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-20T09:39:03.358 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:39:03.428 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:03.620 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:39:03.892 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:39:03.907 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:39:03.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:03.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:39:03.927 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:03 smithi115 ceph-mon[72476]: pgmap v2509: 47 pgs: 47 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 860 op/s 2024-03-20T09:39:03.935 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:03 smithi122 ceph-mon[73495]: pgmap v2509: 47 pgs: 47 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 860 op/s 2024-03-20T09:39:03.936 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:03 smithi026 ceph-mon[68309]: pgmap v2509: 47 pgs: 47 active+clean; 2.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 860 op/s 2024-03-20T09:39:04.016 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:39:04.050 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:04.167 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:39:04.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:04.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:39:04.280 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:39:04.311 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:04.412 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:39:04.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:04.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:39:04.516 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:39:04.572 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:05.346 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:39:05.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:05.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:39:05.452 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:39:05.475 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:39:05.475 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:39:05.476 INFO:teuthology.orchestra.run.smithi115.stdout: "num_ops": 0 2024-03-20T09:39:05.476 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:39:05.488 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:05.513 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:39:05.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:05.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:39:05.617 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:39:05.678 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:05.679 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:05 smithi026 ceph-mon[68309]: pgmap v2510: 47 pgs: 47 active+clean; 2.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 743 op/s 2024-03-20T09:39:05.718 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:39:05.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:05.723 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:39:05.824 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:39:05.850 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:05 smithi115 ceph-mon[72476]: pgmap v2510: 47 pgs: 47 active+clean; 2.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 743 op/s 2024-03-20T09:39:05.854 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:05.936 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:39:05.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:05 smithi122 ceph-mon[73495]: pgmap v2510: 47 pgs: 47 active+clean; 2.8 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 743 op/s 2024-03-20T09:39:06.206 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:39:06.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:06.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:39:06.232 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-20T09:39:06.311 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:39:06.344 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:06.371 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:39:06.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:06.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:39:06.408 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:39:06.460 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-20T09:39:06.471+0000 7fb7b1def700 0 --2- 172.21.15.26:0/257401956 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fb7ac1108f0 0x7fb7ac109190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-03-20T09:39:06.476 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:39:06.510 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:06.535 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:39:06.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:06.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:39:06.612 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:39:06.641 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:39:06.671 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 8, 11, 3, 4, 9, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 11, 5, 10] 2024-03-20T09:39:06.671 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:39:06.671 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 1, 2, 5, 6, 7, 8, 11, 3, 4, 9, 10] 2024-03-20T09:39:06.671 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd out 0 2024-03-20T09:39:06.707 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:06.764 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:39:06.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:06.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:39:06.875 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:39:06.918 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:06.940 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:39:06.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:06.943 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:39:07.044 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:39:07.116 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:07.392 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:39:07.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:07.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:39:07.499 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:39:07.534 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:07.688 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:07 smithi026 ceph-mon[68309]: pgmap v2511: 47 pgs: 47 active+clean; 2.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 812 op/s 2024-03-20T09:39:07.688 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:07 smithi026 ceph-mon[68309]: from='client.27070 172.21.15.26:0/684053556' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:39:07.834 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:39:07.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:07 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:39:08.004 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:39:08.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:08.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:39:08.110 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:39:08.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:07 smithi115 ceph-mon[72476]: pgmap v2511: 47 pgs: 47 active+clean; 2.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 812 op/s 2024-03-20T09:39:08.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:07 smithi115 ceph-mon[72476]: from='client.27070 172.21.15.26:0/684053556' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:39:08.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:07 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:39:08.151 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:08.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:07 smithi122 ceph-mon[73495]: pgmap v2511: 47 pgs: 47 active+clean; 2.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 812 op/s 2024-03-20T09:39:08.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:07 smithi122 ceph-mon[73495]: from='client.27070 172.21.15.26:0/684053556' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:39:08.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:07 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:39:08.203 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:39:08.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:08.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:39:08.317 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:39:08.377 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:08.690 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:08 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:39:08.690 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:08 smithi026 ceph-mon[68309]: from='client.27070 172.21.15.26:0/684053556' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:39:08.690 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:08 smithi026 ceph-mon[68309]: osdmap e1037: 12 total, 12 up, 12 in 2024-03-20T09:39:09.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:08 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:39:09.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:08 smithi115 ceph-mon[72476]: from='client.27070 172.21.15.26:0/684053556' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:39:09.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:08 smithi115 ceph-mon[72476]: osdmap e1037: 12 total, 12 up, 12 in 2024-03-20T09:39:09.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:08 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:39:09.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:08 smithi122 ceph-mon[73495]: from='client.27070 172.21.15.26:0/684053556' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:39:09.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:08 smithi122 ceph-mon[73495]: osdmap e1037: 12 total, 12 up, 12 in 2024-03-20T09:39:09.558 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:39:09.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:09.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:39:09.663 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:39:09.726 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:09.765 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:39:09.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:09.769 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:39:09.870 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:39:09.876 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:39:09.889 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:09 smithi115 ceph-mon[72476]: pgmap v2513: 47 pgs: 47 active+clean; 2.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 824 op/s 2024-03-20T09:39:09.889 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:09 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:39:09.889 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:09 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:39:09.889 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:09 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:39:09.910 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:09.937 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:39:09.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:09.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:39:09.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:09 smithi026 ceph-mon[68309]: pgmap v2513: 47 pgs: 47 active+clean; 2.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 824 op/s 2024-03-20T09:39:09.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:09 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:39:09.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:09 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:39:09.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:09 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:39:10.045 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:39:10.108 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:10.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:09 smithi122 ceph-mon[73495]: pgmap v2513: 47 pgs: 47 active+clean; 2.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 824 op/s 2024-03-20T09:39:10.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:09 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:39:10.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:09 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:39:10.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:09 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:39:10.193 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:39:10.199 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:39:10.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:10.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:39:10.237 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:39:10.273 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:39:10.309 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:39:10.312 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:39:10.346 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:39:10.374 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:10.399 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:39:10.427 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:39:10.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:10.466 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:39:10.467 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:39:10.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:39:10.499 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:39:10.523 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:39:10.523 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:39:10.542 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:39:10.567 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:39:10.587 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:39:10.613 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:10.626 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:39:10.628 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:39:10.628 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:39:10.628 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:39:10.628 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:39:10.628 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:39:10.628 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:39:10.628 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:39:10.628 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:39:10.629 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:39:10.629 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:39:10.629 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:39:10.629 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:39:10.647 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:39:10.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:10.652 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:39:10.753 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:39:10.794 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:10.894 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:10 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3889197388' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:39:10.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:10 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3889197388' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:39:10.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:10 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3889197388' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:39:11.281 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:39:11.604 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-20T09:39:11.647 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:39:11.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:11.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:39:11.756 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:39:11.814 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:11 smithi026 ceph-mon[68309]: pgmap v2514: 47 pgs: 47 active+clean; 2.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 818 op/s 2024-03-20T09:39:11.818 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:11.899 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:11 smithi115 ceph-mon[72476]: pgmap v2514: 47 pgs: 47 active+clean; 2.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 818 op/s 2024-03-20T09:39:11.959 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:39:12.042 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:39:12.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:12.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:39:12.126 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:39:12.155 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:39:12.183 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:12.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:11 smithi122 ceph-mon[73495]: pgmap v2514: 47 pgs: 47 active+clean; 2.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 818 op/s 2024-03-20T09:39:12.224 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:39:12.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:12.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:39:12.330 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:39:12.407 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:12.435 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:39:12.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:12.452 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:39:12.553 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:39:12.602 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:12.637 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:39:12.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:12.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:39:12.745 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:39:12.793 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:12.814 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:39:12.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:12.817 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:39:12.918 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:39:12.966 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:13.332 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:39:13.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:13.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:39:13.438 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:39:13.469 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:13.556 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:39:13.591 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:13 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mon-a[68286]: 2024-03-20T09:39:13.259+0000 7f683364b700 -1 mon.a@0(leader).osd e1037 definitely_dead 0 2024-03-20T09:39:13.626 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:39:13.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:13.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:39:13.731 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:39:13.788 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:13.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:13 smithi115 ceph-mon[72476]: pgmap v2515: 47 pgs: 47 active+clean; 2.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 820 op/s 2024-03-20T09:39:13.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:13 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/268588052' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-03-20T09:39:13.895 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:13 smithi115 ceph-mon[72476]: Client client.admin marked osd.0 out, while it was still marked up 2024-03-20T09:39:13.895 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:13 smithi026 ceph-mon[68309]: pgmap v2515: 47 pgs: 47 active+clean; 2.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 820 op/s 2024-03-20T09:39:13.895 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:13 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/268588052' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-03-20T09:39:13.895 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:13 smithi026 ceph-mon[68309]: Client client.admin marked osd.0 out, while it was still marked up 2024-03-20T09:39:13.907 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:39:13.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:13.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:39:13.914 INFO:teuthology.orchestra.run.smithi026.stderr:marked out osd.0. 2024-03-20T09:39:13.941 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:39:13.941 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:39:14.012 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:39:14.040 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:14.163 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:39:14.164 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:39:14.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:14.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:39:14.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:13 smithi122 ceph-mon[73495]: pgmap v2515: 47 pgs: 47 active+clean; 2.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 820 op/s 2024-03-20T09:39:14.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:13 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/268588052' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-03-20T09:39:14.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:13 smithi122 ceph-mon[73495]: Client client.admin marked osd.0 out, while it was still marked up 2024-03-20T09:39:14.267 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:39:14.296 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:15.061 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:39:15.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:15.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:39:15.088 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-20T09:39:15.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:14 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/268588052' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-03-20T09:39:15.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:14 smithi115 ceph-mon[72476]: osdmap e1038: 12 total, 12 up, 11 in 2024-03-20T09:39:15.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:14 smithi115 ceph-mon[72476]: pgmap v2517: 47 pgs: 47 active+clean; 2.3 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 782 op/s 2024-03-20T09:39:15.166 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:39:15.177 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:14 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/268588052' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-03-20T09:39:15.177 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:14 smithi026 ceph-mon[68309]: osdmap e1038: 12 total, 12 up, 11 in 2024-03-20T09:39:15.177 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:14 smithi026 ceph-mon[68309]: pgmap v2517: 47 pgs: 47 active+clean; 2.3 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 782 op/s 2024-03-20T09:39:15.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:14 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/268588052' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-03-20T09:39:15.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:14 smithi122 ceph-mon[73495]: osdmap e1038: 12 total, 12 up, 11 in 2024-03-20T09:39:15.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:14 smithi122 ceph-mon[73495]: pgmap v2517: 47 pgs: 47 active+clean; 2.3 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 782 op/s 2024-03-20T09:39:15.203 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:15.388 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:39:16.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:15 smithi122 ceph-mon[73495]: osdmap e1039: 12 total, 12 up, 11 in 2024-03-20T09:39:16.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:15 smithi026 ceph-mon[68309]: osdmap e1039: 12 total, 12 up, 11 in 2024-03-20T09:39:16.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:15 smithi115 ceph-mon[72476]: osdmap e1039: 12 total, 12 up, 11 in 2024-03-20T09:39:17.085 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:16 smithi115 ceph-mon[72476]: osdmap e1040: 12 total, 12 up, 11 in 2024-03-20T09:39:17.085 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:16 smithi115 ceph-mon[72476]: 8.e starting backfill to osd.7 from (0'0,0'0] MAX to 1037'13100 2024-03-20T09:39:17.085 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:16 smithi115 ceph-mon[72476]: 8.2 starting backfill to osd.9 from (0'0,0'0] MAX to 1037'13236 2024-03-20T09:39:17.085 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:16 smithi115 ceph-mon[72476]: 8.9 starting backfill to osd.3 from (0'0,0'0] MAX to 1037'13210 2024-03-20T09:39:17.085 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:16 smithi115 ceph-mon[72476]: pgmap v2520: 47 pgs: 4 remapped+peering, 43 active+clean; 2.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 796 op/s 2024-03-20T09:39:17.085 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:16 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/580914253' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:39:17.086 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:16 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:39:17.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:16 smithi122 ceph-mon[73495]: osdmap e1040: 12 total, 12 up, 11 in 2024-03-20T09:39:17.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:16 smithi122 ceph-mon[73495]: 8.e starting backfill to osd.7 from (0'0,0'0] MAX to 1037'13100 2024-03-20T09:39:17.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:16 smithi122 ceph-mon[73495]: 8.2 starting backfill to osd.9 from (0'0,0'0] MAX to 1037'13236 2024-03-20T09:39:17.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:16 smithi122 ceph-mon[73495]: 8.9 starting backfill to osd.3 from (0'0,0'0] MAX to 1037'13210 2024-03-20T09:39:17.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:16 smithi122 ceph-mon[73495]: pgmap v2520: 47 pgs: 4 remapped+peering, 43 active+clean; 2.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 796 op/s 2024-03-20T09:39:17.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:16 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/580914253' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:39:17.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:16 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:39:17.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:16 smithi026 ceph-mon[68309]: osdmap e1040: 12 total, 12 up, 11 in 2024-03-20T09:39:17.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:16 smithi026 ceph-mon[68309]: 8.e starting backfill to osd.7 from (0'0,0'0] MAX to 1037'13100 2024-03-20T09:39:17.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:16 smithi026 ceph-mon[68309]: 8.2 starting backfill to osd.9 from (0'0,0'0] MAX to 1037'13236 2024-03-20T09:39:17.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:16 smithi026 ceph-mon[68309]: 8.9 starting backfill to osd.3 from (0'0,0'0] MAX to 1037'13210 2024-03-20T09:39:17.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:16 smithi026 ceph-mon[68309]: pgmap v2520: 47 pgs: 4 remapped+peering, 43 active+clean; 2.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 796 op/s 2024-03-20T09:39:17.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:16 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/580914253' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:39:17.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:16 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:39:17.417 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:39:17.732 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:39:17.732 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:39:18.097 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:39:18.104 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:17 smithi115 ceph-mon[72476]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-20T09:39:18.104 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:17 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:39:18.105 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:17 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:39:18.105 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:17 smithi115 ceph-mon[72476]: osdmap e1041: 12 total, 12 up, 11 in 2024-03-20T09:39:18.105 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:17 smithi115 ceph-mon[72476]: 8.3 scrub starts 2024-03-20T09:39:18.105 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:17 smithi115 ceph-mon[72476]: 8.3 scrub ok 2024-03-20T09:39:18.108 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:17 smithi026 ceph-mon[68309]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-20T09:39:18.108 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:17 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:39:18.108 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:17 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:39:18.108 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:17 smithi026 ceph-mon[68309]: osdmap e1041: 12 total, 12 up, 11 in 2024-03-20T09:39:18.109 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:17 smithi026 ceph-mon[68309]: 8.3 scrub starts 2024-03-20T09:39:18.109 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:17 smithi026 ceph-mon[68309]: 8.3 scrub ok 2024-03-20T09:39:18.110 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:18.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:39:18.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:17 smithi122 ceph-mon[73495]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-20T09:39:18.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:17 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:39:18.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:17 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:39:18.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:17 smithi122 ceph-mon[73495]: osdmap e1041: 12 total, 12 up, 11 in 2024-03-20T09:39:18.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:17 smithi122 ceph-mon[73495]: 8.3 scrub starts 2024-03-20T09:39:18.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:17 smithi122 ceph-mon[73495]: 8.3 scrub ok 2024-03-20T09:39:18.211 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:39:18.243 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:18.440 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:39:18.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:18.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:39:18.451 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-20T09:39:18.544 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:39:18.576 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:18.636 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:39:18.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:18.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:39:18.744 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:39:18.775 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:18.790 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:39:18.798 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:39:18.801 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:18.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:39:18.903 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:39:18.961 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:19.002 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:39:19.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:19.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:39:19.061 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:18 smithi115 ceph-mon[72476]: osdmap e1042: 12 total, 12 up, 11 in 2024-03-20T09:39:19.061 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:18 smithi115 ceph-mon[72476]: pgmap v2523: 47 pgs: 1 active+remapped+backfill_toofull, 2 active+undersized+remapped, 4 remapped+peering, 40 active+clean; 2.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 552 op/s; 2100/106203 objects misplaced (1.977%) 2024-03-20T09:39:19.106 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:39:19.150 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:19.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:18 smithi122 ceph-mon[73495]: osdmap e1042: 12 total, 12 up, 11 in 2024-03-20T09:39:19.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:18 smithi122 ceph-mon[73495]: pgmap v2523: 47 pgs: 1 active+remapped+backfill_toofull, 2 active+undersized+remapped, 4 remapped+peering, 40 active+clean; 2.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 552 op/s; 2100/106203 objects misplaced (1.977%) 2024-03-20T09:39:19.187 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:39:19.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:19.191 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:39:19.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:18 smithi026 ceph-mon[68309]: osdmap e1042: 12 total, 12 up, 11 in 2024-03-20T09:39:19.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:18 smithi026 ceph-mon[68309]: pgmap v2523: 47 pgs: 1 active+remapped+backfill_toofull, 2 active+undersized+remapped, 4 remapped+peering, 40 active+clean; 2.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 552 op/s; 2100/106203 objects misplaced (1.977%) 2024-03-20T09:39:19.217 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-20T09:39:19.227+0000 7f77c3fff700 0 --2- 172.21.15.26:0/1351165271 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f77c410e330 0x7f77c4115f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-03-20T09:39:19.252 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:39:19.276 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:39:19.292 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:39:19.315 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:39:19.351 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:19.355 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:39:19.386 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:39:19.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:19.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:39:19.398 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:39:19.432 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:39:19.464 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:39:19.488 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:39:19.494 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:39:19.518 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:39:19.525 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:19.556 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:39:19.593 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:39:19.637 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:39:19.639 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:39:19.639 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:39:19.639 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:39:19.639 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:39:19.639 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:39:19.639 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:39:19.639 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:39:19.640 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:39:19.640 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:39:19.640 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:39:19.640 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:39:19.663 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 5, 6, 7, 8, 11, 3, 4, 9, 10] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 11, 5, 10] 2024-03-20T09:39:19.663 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:39:19.663 INFO:tasks.thrashosds.thrasher:inject_pause on osd.10 2024-03-20T09:39:19.664 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-20T09:39:19.664 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-20T09:39:19.664 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 config set filestore_inject_stall 3 2024-03-20T09:39:19.678 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:39:19.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:19.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:39:19.782 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:39:19.853 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:19.940 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:19 smithi026 ceph-mon[68309]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-20T09:39:19.940 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:19 smithi026 ceph-mon[68309]: 1.d scrub starts 2024-03-20T09:39:19.940 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:19 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/905129447' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:39:19.948 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:39:19.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:19.954 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:39:20.055 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:39:20.083 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:20.129 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:39:20.139 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:39:20.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:20.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:39:20.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:19 smithi122 ceph-mon[73495]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-20T09:39:20.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:19 smithi122 ceph-mon[73495]: 1.d scrub starts 2024-03-20T09:39:20.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:19 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/905129447' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:39:20.243 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:39:20.245 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:39:20.305 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:20.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:19 smithi115 ceph-mon[72476]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-20T09:39:20.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:19 smithi115 ceph-mon[72476]: 1.d scrub starts 2024-03-20T09:39:20.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:19 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/905129447' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:39:20.453 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:39:20.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:20.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:39:20.558 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:39:20.589 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:20.624 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:39:20.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:20.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:39:20.665 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:39:20.728 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:39:20.772 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:39:20.819 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:39:20.819 INFO:teuthology.orchestra.run.smithi115.stdout: "complaint_time": 30, 2024-03-20T09:39:20.819 INFO:teuthology.orchestra.run.smithi115.stdout: "num_blocked_ops": 0 2024-03-20T09:39:20.819 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:39:20.820 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:21.128 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:20 smithi115 ceph-mon[72476]: 8.b scrub starts 2024-03-20T09:39:21.128 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:20 smithi115 ceph-mon[72476]: 1.d scrub ok 2024-03-20T09:39:21.128 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:20 smithi115 ceph-mon[72476]: 8.b scrub ok 2024-03-20T09:39:21.128 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:20 smithi115 ceph-mon[72476]: pgmap v2524: 47 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 2 active+undersized+remapped, 2 remapped+peering, 40 active+clean; 2.1 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 537 op/s; 5467/103617 objects misplaced (5.276%); 11 MiB/s, 174 objects/s recovering 2024-03-20T09:39:21.182 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:39:21.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:20 smithi122 ceph-mon[73495]: 8.b scrub starts 2024-03-20T09:39:21.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:20 smithi122 ceph-mon[73495]: 1.d scrub ok 2024-03-20T09:39:21.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:20 smithi122 ceph-mon[73495]: 8.b scrub ok 2024-03-20T09:39:21.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:20 smithi122 ceph-mon[73495]: pgmap v2524: 47 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 2 active+undersized+remapped, 2 remapped+peering, 40 active+clean; 2.1 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 537 op/s; 5467/103617 objects misplaced (5.276%); 11 MiB/s, 174 objects/s recovering 2024-03-20T09:39:21.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:21.193 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:39:21.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:20 smithi026 ceph-mon[68309]: 8.b scrub starts 2024-03-20T09:39:21.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:20 smithi026 ceph-mon[68309]: 1.d scrub ok 2024-03-20T09:39:21.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:20 smithi026 ceph-mon[68309]: 8.b scrub ok 2024-03-20T09:39:21.213 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:20 smithi026 ceph-mon[68309]: pgmap v2524: 47 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 2 active+undersized+remapped, 2 remapped+peering, 40 active+clean; 2.1 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 537 op/s; 5467/103617 objects misplaced (5.276%); 11 MiB/s, 174 objects/s recovering 2024-03-20T09:39:21.293 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:39:21.326 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:21.451 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:39:21.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:21.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:39:21.483 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-20T09:39:21.556 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:39:21.610 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:21.643 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:39:21.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:21.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:39:21.750 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:39:21.808 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:21.830 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:39:21.955 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:21 smithi026 ceph-mon[68309]: 8.a scrub starts 2024-03-20T09:39:21.955 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:21 smithi026 ceph-mon[68309]: 8.a scrub ok 2024-03-20T09:39:22.092 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:39:22.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:22.101 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:39:22.101 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:21 smithi122 ceph-mon[73495]: 8.a scrub starts 2024-03-20T09:39:22.101 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:21 smithi122 ceph-mon[73495]: 8.a scrub ok 2024-03-20T09:39:22.201 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:39:22.235 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:21 smithi115 ceph-mon[72476]: 8.a scrub starts 2024-03-20T09:39:22.236 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:21 smithi115 ceph-mon[72476]: 8.a scrub ok 2024-03-20T09:39:22.238 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:22.500 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:39:22.698 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:39:22.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:22.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:39:22.803 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:39:22.834 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:22.907 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:39:22.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:22.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:39:22.985 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:39:23.011 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:39:23.084 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:23.173 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:39:23.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:23.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:39:23.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:22 smithi122 ceph-mon[73495]: 1.1c scrub starts 2024-03-20T09:39:23.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:22 smithi122 ceph-mon[73495]: 1.1c scrub ok 2024-03-20T09:39:23.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:22 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:39:23.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:22 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:39:23.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:22 smithi122 ceph-mon[73495]: osdmap e1043: 12 total, 12 up, 11 in 2024-03-20T09:39:23.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:22 smithi122 ceph-mon[73495]: 1.7 scrub starts 2024-03-20T09:39:23.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:22 smithi122 ceph-mon[73495]: 1.7 scrub ok 2024-03-20T09:39:23.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:22 smithi122 ceph-mon[73495]: pgmap v2526: 47 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 44 active+clean; 2.0 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 489 op/s; 5467/99690 objects misplaced (5.484%); 10 MiB/s, 166 objects/s recovering 2024-03-20T09:39:23.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:22 smithi122 ceph-mon[73495]: Health check update: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-20T09:39:23.211 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:39:23.212 INFO:teuthology.orchestra.run.smithi122.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-20T09:39:23.212 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:39:23.284 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:39:23.300 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:22 smithi026 ceph-mon[68309]: 1.1c scrub starts 2024-03-20T09:39:23.300 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:22 smithi026 ceph-mon[68309]: 1.1c scrub ok 2024-03-20T09:39:23.300 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:22 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:39:23.300 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:22 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:39:23.300 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:22 smithi026 ceph-mon[68309]: osdmap e1043: 12 total, 12 up, 11 in 2024-03-20T09:39:23.300 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:22 smithi026 ceph-mon[68309]: 1.7 scrub starts 2024-03-20T09:39:23.300 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:22 smithi026 ceph-mon[68309]: 1.7 scrub ok 2024-03-20T09:39:23.300 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:22 smithi026 ceph-mon[68309]: pgmap v2526: 47 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 44 active+clean; 2.0 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 489 op/s; 5467/99690 objects misplaced (5.484%); 10 MiB/s, 166 objects/s recovering 2024-03-20T09:39:23.300 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:22 smithi026 ceph-mon[68309]: Health check update: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-20T09:39:23.347 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:23.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:22 smithi115 ceph-mon[72476]: 1.1c scrub starts 2024-03-20T09:39:23.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:22 smithi115 ceph-mon[72476]: 1.1c scrub ok 2024-03-20T09:39:23.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:22 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:39:23.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:22 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:39:23.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:22 smithi115 ceph-mon[72476]: osdmap e1043: 12 total, 12 up, 11 in 2024-03-20T09:39:23.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:22 smithi115 ceph-mon[72476]: 1.7 scrub starts 2024-03-20T09:39:23.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:22 smithi115 ceph-mon[72476]: 1.7 scrub ok 2024-03-20T09:39:23.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:22 smithi115 ceph-mon[72476]: pgmap v2526: 47 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 44 active+clean; 2.0 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 489 op/s; 5467/99690 objects misplaced (5.484%); 10 MiB/s, 166 objects/s recovering 2024-03-20T09:39:23.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:22 smithi115 ceph-mon[72476]: Health check update: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-20T09:39:23.845 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:39:23.846 INFO:teuthology.orchestra.run.smithi115.stdout: "size": 20, 2024-03-20T09:39:23.846 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 600, 2024-03-20T09:39:23.846 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [ 2024-03-20T09:39:23.846 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.846 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221363 7.12 7:4d24fa2c:::benchmark_data_smithi122_187077_object94752:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e940)", 2024-03-20T09:39:23.846 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:46.500402+0000", 2024-03-20T09:39:23.846 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 277.35359289899998, 2024-03-20T09:39:23.846 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1879329379999999, 2024-03-20T09:39:23.846 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:23.846 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:39:23.846 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:39:23.846 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:39:23.847 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:39:23.847 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221363 2024-03-20T09:39:23.847 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.847 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:23.847 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.847 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:23.847 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.500402+0000", 2024-03-20T09:39:23.847 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.847 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.847 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.847 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:23.847 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.500402+0000", 2024-03-20T09:39:23.847 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.848 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.848 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.848 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:23.848 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.500403+0000", 2024-03-20T09:39:23.848 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.7100000000000011e-07 2024-03-20T09:39:23.848 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.848 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.848 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:23.848 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.500408+0000", 2024-03-20T09:39:23.848 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2129999999999999e-06 2024-03-20T09:39:23.848 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.848 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.848 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:23.848 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.500420+0000", 2024-03-20T09:39:23.849 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2519e-05 2024-03-20T09:39:23.849 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.849 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.849 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:23.849 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.500428+0000", 2024-03-20T09:39:23.849 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.6329999999999996e-06 2024-03-20T09:39:23.849 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.849 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.849 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.849 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.500484+0000", 2024-03-20T09:39:23.849 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6329000000000002e-05 2024-03-20T09:39:23.849 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.849 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.850 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:39:23.850 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.500606+0000", 2024-03-20T09:39:23.850 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000121401 2024-03-20T09:39:23.850 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.850 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.850 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 0,2,3", 2024-03-20T09:39:23.850 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.500957+0000", 2024-03-20T09:39:23.850 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000351274 2024-03-20T09:39:23.850 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.850 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.850 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:39:23.850 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.504667+0000", 2024-03-20T09:39:23.850 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0037103090000000002 2024-03-20T09:39:23.851 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.851 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.851 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:39:23.851 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.505148+0000", 2024-03-20T09:39:23.851 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00048108100000000001 2024-03-20T09:39:23.851 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.851 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.851 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:39:23.851 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.510137+0000", 2024-03-20T09:39:23.851 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0049886030000000003 2024-03-20T09:39:23.851 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.851 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.851 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.851 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.666198+0000", 2024-03-20T09:39:23.852 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.15606091599999999 2024-03-20T09:39:23.852 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.852 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.853 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:39:23.853 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.666208+0000", 2024-03-20T09:39:23.853 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0298e-05 2024-03-20T09:39:23.853 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.853 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.853 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.853 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.688325+0000", 2024-03-20T09:39:23.853 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.022116459 2024-03-20T09:39:23.853 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.854 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.854 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:23.854 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.688335+0000", 2024-03-20T09:39:23.854 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.9320000000000008e-06 2024-03-20T09:39:23.854 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.854 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:23.854 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.854 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.854 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.854 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221365 7.12 7:496d3fcf:::benchmark_data_smithi122_187077_object94754:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e940)", 2024-03-20T09:39:23.854 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:46.500453+0000", 2024-03-20T09:39:23.854 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 277.353541753, 2024-03-20T09:39:23.854 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1882682339999999, 2024-03-20T09:39:23.855 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:23.855 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:39:23.855 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:39:23.855 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:39:23.855 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:39:23.855 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221365 2024-03-20T09:39:23.855 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.855 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:23.855 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.855 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:23.855 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.500453+0000", 2024-03-20T09:39:23.855 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.855 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.855 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.856 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:23.856 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.500453+0000", 2024-03-20T09:39:23.856 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.856 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.856 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.856 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:23.856 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.500453+0000", 2024-03-20T09:39:23.856 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5400000000000002e-07 2024-03-20T09:39:23.856 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.856 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.856 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:23.856 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.500456+0000", 2024-03-20T09:39:23.856 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3360000000000002e-06 2024-03-20T09:39:23.856 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.857 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.857 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:23.857 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.500466+0000", 2024-03-20T09:39:23.857 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0223e-05 2024-03-20T09:39:23.857 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.857 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.857 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:23.857 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.500471+0000", 2024-03-20T09:39:23.857 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6130000000000003e-06 2024-03-20T09:39:23.857 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.857 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.857 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.857 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.502725+0000", 2024-03-20T09:39:23.857 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0022532020000000002 2024-03-20T09:39:23.858 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.858 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.858 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:39:23.858 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.502849+0000", 2024-03-20T09:39:23.858 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000124595 2024-03-20T09:39:23.858 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.858 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.858 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 0,2,3", 2024-03-20T09:39:23.858 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.503098+0000", 2024-03-20T09:39:23.858 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00024829499999999999 2024-03-20T09:39:23.858 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.858 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.859 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:39:23.859 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.504782+0000", 2024-03-20T09:39:23.859 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001684533 2024-03-20T09:39:23.859 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.859 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.860 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:39:23.860 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.507368+0000", 2024-03-20T09:39:23.860 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0025858510000000001 2024-03-20T09:39:23.860 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.860 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.860 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:39:23.860 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.510175+0000", 2024-03-20T09:39:23.860 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0028068479999999998 2024-03-20T09:39:23.860 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.860 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.860 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.860 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.666236+0000", 2024-03-20T09:39:23.861 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.15606133699999999 2024-03-20T09:39:23.861 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.861 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.861 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:39:23.861 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.666248+0000", 2024-03-20T09:39:23.861 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2027e-05 2024-03-20T09:39:23.861 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.861 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.861 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.861 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.688712+0000", 2024-03-20T09:39:23.861 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0224640629999999 2024-03-20T09:39:23.862 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.862 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.862 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:23.862 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.688721+0000", 2024-03-20T09:39:23.862 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.8570000000000001e-06 2024-03-20T09:39:23.862 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.862 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:23.862 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.862 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.862 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.862 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221389 7.12 7.24f0b3b2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e941)", 2024-03-20T09:39:23.862 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:46.694341+0000", 2024-03-20T09:39:23.863 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 277.15965348999998, 2024-03-20T09:39:23.863 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.99442238800000005, 2024-03-20T09:39:23.863 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:23.863 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:39:23.863 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:39:23.863 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:39:23.863 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:39:23.863 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221389 2024-03-20T09:39:23.863 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.865 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:23.865 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.865 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:23.865 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.694341+0000", 2024-03-20T09:39:23.865 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.865 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.865 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.866 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:23.866 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.694341+0000", 2024-03-20T09:39:23.866 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.866 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.866 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.866 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:23.866 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.694342+0000", 2024-03-20T09:39:23.866 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5899999999999996e-07 2024-03-20T09:39:23.866 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.866 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.866 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:23.866 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.694345+0000", 2024-03-20T09:39:23.866 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.4300000000000002e-06 2024-03-20T09:39:23.867 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.867 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.867 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:23.867 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.694356+0000", 2024-03-20T09:39:23.867 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0692e-05 2024-03-20T09:39:23.867 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.867 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.867 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:23.867 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.694362+0000", 2024-03-20T09:39:23.867 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8119999999999996e-06 2024-03-20T09:39:23.867 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.867 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.868 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.868 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.694400+0000", 2024-03-20T09:39:23.868 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.8597000000000003e-05 2024-03-20T09:39:23.868 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.868 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.868 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:39:23.868 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.694406+0000", 2024-03-20T09:39:23.868 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6400000000000002e-06 2024-03-20T09:39:23.868 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.868 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.868 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.868 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.688751+0000", 2024-03-20T09:39:23.868 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.99434531000000004 2024-03-20T09:39:23.869 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.869 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.869 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:23.869 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.688763+0000", 2024-03-20T09:39:23.869 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2347999999999999e-05 2024-03-20T09:39:23.869 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.869 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:23.869 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.869 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.869 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.869 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221410 7.12 7.4e88c252 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e941)", 2024-03-20T09:39:23.869 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:46.729117+0000", 2024-03-20T09:39:23.869 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 277.12487751499998, 2024-03-20T09:39:23.869 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.95966132800000004, 2024-03-20T09:39:23.870 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:23.870 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:39:23.870 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:39:23.870 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:39:23.870 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:39:23.870 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221410 2024-03-20T09:39:23.870 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.870 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:23.870 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.870 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:23.870 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.729117+0000", 2024-03-20T09:39:23.870 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.870 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.871 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.871 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:23.871 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.729117+0000", 2024-03-20T09:39:23.871 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.871 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.871 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.871 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:23.871 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.729118+0000", 2024-03-20T09:39:23.871 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9999999999999998e-07 2024-03-20T09:39:23.871 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.871 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.871 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:23.871 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.729120+0000", 2024-03-20T09:39:23.871 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9189999999999999e-06 2024-03-20T09:39:23.872 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.872 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.872 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:23.872 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.729133+0000", 2024-03-20T09:39:23.872 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2908000000000001e-05 2024-03-20T09:39:23.872 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.872 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.872 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:23.872 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.729139+0000", 2024-03-20T09:39:23.872 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2979999999999999e-06 2024-03-20T09:39:23.872 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.872 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.872 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.872 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.729203+0000", 2024-03-20T09:39:23.873 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4102000000000004e-05 2024-03-20T09:39:23.873 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.873 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.873 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:39:23.873 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.729209+0000", 2024-03-20T09:39:23.873 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9619999999999998e-06 2024-03-20T09:39:23.873 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.873 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.873 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.873 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.688774+0000", 2024-03-20T09:39:23.873 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.95956517799999996 2024-03-20T09:39:23.873 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.873 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.873 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:23.874 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.688778+0000", 2024-03-20T09:39:23.874 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4610000000000001e-06 2024-03-20T09:39:23.874 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.874 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:23.874 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.874 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.874 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.874 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221471 7.12 7.c01da832 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e941)", 2024-03-20T09:39:23.874 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:46.800088+0000", 2024-03-20T09:39:23.874 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 277.05390650200002, 2024-03-20T09:39:23.874 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.88872703399999997, 2024-03-20T09:39:23.874 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:23.874 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:39:23.874 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:39:23.875 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:39:23.875 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:39:23.875 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221471 2024-03-20T09:39:23.875 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.875 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:23.875 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.875 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:23.875 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.800088+0000", 2024-03-20T09:39:23.875 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.875 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.875 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.875 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:23.875 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.800088+0000", 2024-03-20T09:39:23.875 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.875 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.876 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.876 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:23.876 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.800089+0000", 2024-03-20T09:39:23.876 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.341e-06 2024-03-20T09:39:23.876 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.876 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.876 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:23.876 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.800098+0000", 2024-03-20T09:39:23.876 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.3639999999999999e-06 2024-03-20T09:39:23.876 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.876 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.876 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:23.876 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.800113+0000", 2024-03-20T09:39:23.877 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4805e-05 2024-03-20T09:39:23.877 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.877 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.877 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:23.877 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.800120+0000", 2024-03-20T09:39:23.877 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.5700000000000004e-06 2024-03-20T09:39:23.877 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.877 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.877 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.877 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.800196+0000", 2024-03-20T09:39:23.877 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.6099000000000004e-05 2024-03-20T09:39:23.877 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.877 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.877 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:39:23.878 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.800203+0000", 2024-03-20T09:39:23.878 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.5950000000000004e-06 2024-03-20T09:39:23.878 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.878 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.878 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.878 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.688806+0000", 2024-03-20T09:39:23.878 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.88860339099999996 2024-03-20T09:39:23.878 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.878 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.878 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:23.878 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.688815+0000", 2024-03-20T09:39:23.878 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.8689999999999993e-06 2024-03-20T09:39:23.878 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.879 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:23.879 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.879 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.879 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.879 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221511 7.12 7.31478952 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e941)", 2024-03-20T09:39:23.879 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:46.846911+0000", 2024-03-20T09:39:23.879 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 277.00708333, 2024-03-20T09:39:23.879 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.841917727, 2024-03-20T09:39:23.879 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:23.879 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:39:23.879 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:39:23.879 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:39:23.879 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:39:23.879 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221511 2024-03-20T09:39:23.880 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.880 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:23.880 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.880 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:23.880 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.846911+0000", 2024-03-20T09:39:23.880 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.880 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.880 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.880 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:23.880 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.846911+0000", 2024-03-20T09:39:23.880 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.880 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.880 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.880 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:23.881 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.846912+0000", 2024-03-20T09:39:23.881 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1319999999999999e-06 2024-03-20T09:39:23.881 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.881 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.881 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:23.881 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.846918+0000", 2024-03-20T09:39:23.881 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1989999999999997e-06 2024-03-20T09:39:23.881 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.881 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.881 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:23.881 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.846930+0000", 2024-03-20T09:39:23.881 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2761e-05 2024-03-20T09:39:23.881 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.881 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.882 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:23.882 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.846938+0000", 2024-03-20T09:39:23.882 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.123e-06 2024-03-20T09:39:23.882 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.882 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.882 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.882 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.846981+0000", 2024-03-20T09:39:23.882 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.2945999999999998e-05 2024-03-20T09:39:23.882 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.882 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.882 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:39:23.882 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.846989+0000", 2024-03-20T09:39:23.882 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7570000000000004e-06 2024-03-20T09:39:23.882 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.883 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.883 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.883 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.688825+0000", 2024-03-20T09:39:23.883 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.84183545000000004 2024-03-20T09:39:23.883 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.883 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.883 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:23.883 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.688829+0000", 2024-03-20T09:39:23.883 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.3590000000000001e-06 2024-03-20T09:39:23.883 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.883 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:23.883 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.883 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.883 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.883 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221573 7.12 7.a7b14f12 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e941)", 2024-03-20T09:39:23.884 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:46.914545+0000", 2024-03-20T09:39:23.884 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 276.93944914500003, 2024-03-20T09:39:23.884 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.77439364200000005, 2024-03-20T09:39:23.884 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:23.884 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:39:23.884 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:39:23.884 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:39:23.884 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:39:23.884 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221573 2024-03-20T09:39:23.884 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.884 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:23.884 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.884 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:23.885 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.914545+0000", 2024-03-20T09:39:23.885 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.885 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.885 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.885 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:23.885 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.914545+0000", 2024-03-20T09:39:23.885 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.885 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.885 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.885 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:23.885 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.914547+0000", 2024-03-20T09:39:23.885 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.186e-06 2024-03-20T09:39:23.885 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.885 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.886 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:23.886 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.914552+0000", 2024-03-20T09:39:23.886 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4750000000000001e-06 2024-03-20T09:39:23.886 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.886 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.886 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:23.886 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.914567+0000", 2024-03-20T09:39:23.886 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.519e-05 2024-03-20T09:39:23.886 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.886 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.886 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:23.886 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.914576+0000", 2024-03-20T09:39:23.886 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.7630000000000006e-06 2024-03-20T09:39:23.886 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.887 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.887 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.887 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.914639+0000", 2024-03-20T09:39:23.887 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.347e-05 2024-03-20T09:39:23.887 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.887 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.887 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:39:23.887 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.914650+0000", 2024-03-20T09:39:23.887 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0397999999999999e-05 2024-03-20T09:39:23.887 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.887 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.887 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.887 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.688843+0000", 2024-03-20T09:39:23.887 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.77419343399999996 2024-03-20T09:39:23.888 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.888 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.888 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:23.888 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.688939+0000", 2024-03-20T09:39:23.888 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.5725999999999998e-05 2024-03-20T09:39:23.888 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.888 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:23.888 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.888 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.888 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.888 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221579 7.12 7.e0ab76d2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e941)", 2024-03-20T09:39:23.888 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:46.920156+0000", 2024-03-20T09:39:23.888 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 276.93383815599998, 2024-03-20T09:39:23.888 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.76870872099999998, 2024-03-20T09:39:23.889 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:23.889 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:39:23.889 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:39:23.889 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:39:23.889 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:39:23.889 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221579 2024-03-20T09:39:23.889 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.889 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:23.889 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.889 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:23.889 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.920156+0000", 2024-03-20T09:39:23.889 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.889 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.889 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.890 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:23.890 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.920156+0000", 2024-03-20T09:39:23.890 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.890 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.890 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.890 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:23.890 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.920157+0000", 2024-03-20T09:39:23.890 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0869999999999999e-06 2024-03-20T09:39:23.890 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.890 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.890 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:23.890 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.920162+0000", 2024-03-20T09:39:23.890 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9300000000000002e-06 2024-03-20T09:39:23.890 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.890 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.891 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:23.891 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.920177+0000", 2024-03-20T09:39:23.891 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4321999999999999e-05 2024-03-20T09:39:23.891 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.891 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.891 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:23.891 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.920190+0000", 2024-03-20T09:39:23.891 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.329e-05 2024-03-20T09:39:23.891 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.891 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.891 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.891 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.920259+0000", 2024-03-20T09:39:23.891 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.9474999999999995e-05 2024-03-20T09:39:23.891 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.892 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.892 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:39:23.892 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.920272+0000", 2024-03-20T09:39:23.892 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2228e-05 2024-03-20T09:39:23.892 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.892 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.892 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.892 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.688861+0000", 2024-03-20T09:39:23.892 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.76858884900000002 2024-03-20T09:39:23.892 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.892 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.892 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:23.893 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.688865+0000", 2024-03-20T09:39:23.893 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5399999999999997e-06 2024-03-20T09:39:23.893 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.893 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:23.893 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.893 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.893 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.893 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221590 7.12 7.6fbad972 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e941)", 2024-03-20T09:39:23.893 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:46.931680+0000", 2024-03-20T09:39:23.893 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 276.92231442999997, 2024-03-20T09:39:23.893 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.75729486499999998, 2024-03-20T09:39:23.893 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:23.893 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:39:23.893 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:39:23.894 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:39:23.894 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:39:23.894 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221590 2024-03-20T09:39:23.894 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.894 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:23.894 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.894 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:23.894 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.931680+0000", 2024-03-20T09:39:23.894 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.894 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.894 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.894 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:23.894 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.931680+0000", 2024-03-20T09:39:23.895 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.895 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.895 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.895 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:23.895 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.931682+0000", 2024-03-20T09:39:23.895 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6700000000000001e-06 2024-03-20T09:39:23.895 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.895 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.895 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:23.895 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.931687+0000", 2024-03-20T09:39:23.895 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.626e-06 2024-03-20T09:39:23.895 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.895 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.895 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:23.896 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.931701+0000", 2024-03-20T09:39:23.896 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3550000000000001e-05 2024-03-20T09:39:23.896 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.896 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.896 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:23.896 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.931709+0000", 2024-03-20T09:39:23.896 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.644e-06 2024-03-20T09:39:23.896 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.896 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.896 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.896 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.931772+0000", 2024-03-20T09:39:23.896 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2957e-05 2024-03-20T09:39:23.896 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.897 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.897 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:39:23.897 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.931783+0000", 2024-03-20T09:39:23.897 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1347e-05 2024-03-20T09:39:23.897 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.897 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.897 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.897 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.688968+0000", 2024-03-20T09:39:23.897 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.75718535300000001 2024-03-20T09:39:23.897 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.897 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.897 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:23.897 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.688975+0000", 2024-03-20T09:39:23.897 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.7179999999999999e-06 2024-03-20T09:39:23.898 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.898 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:23.898 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.898 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.898 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.898 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221593 7.12 7.1ad4eaf2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e941)", 2024-03-20T09:39:23.898 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:46.933926+0000", 2024-03-20T09:39:23.898 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 276.92006818099998, 2024-03-20T09:39:23.898 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.75508512000000005, 2024-03-20T09:39:23.898 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:23.898 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:39:23.898 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:39:23.898 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:39:23.898 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:39:23.898 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221593 2024-03-20T09:39:23.899 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.899 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:23.899 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.899 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:23.899 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.933926+0000", 2024-03-20T09:39:23.899 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.899 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.899 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.899 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:23.899 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.933926+0000", 2024-03-20T09:39:23.899 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.899 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.899 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.899 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:23.900 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.933927+0000", 2024-03-20T09:39:23.900 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.7999999999999995e-07 2024-03-20T09:39:23.900 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.900 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.900 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:23.900 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.933930+0000", 2024-03-20T09:39:23.900 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.1599999999999998e-06 2024-03-20T09:39:23.900 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.900 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.900 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:23.900 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.933938+0000", 2024-03-20T09:39:23.900 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7300000000000005e-06 2024-03-20T09:39:23.900 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.900 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.900 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:23.901 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.933942+0000", 2024-03-20T09:39:23.901 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5340000000000001e-06 2024-03-20T09:39:23.901 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.901 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.901 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.901 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.933998+0000", 2024-03-20T09:39:23.901 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5813000000000002e-05 2024-03-20T09:39:23.901 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.901 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.901 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:39:23.901 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.934006+0000", 2024-03-20T09:39:23.901 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.8420000000000005e-06 2024-03-20T09:39:23.901 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.901 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.901 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.902 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689002+0000", 2024-03-20T09:39:23.902 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.75499634800000004 2024-03-20T09:39:23.902 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.902 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.902 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:23.902 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689011+0000", 2024-03-20T09:39:23.902 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.0129999999999999e-06 2024-03-20T09:39:23.902 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.902 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:23.902 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.902 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.902 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.902 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221706 7.12 7.6687b8b2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e941)", 2024-03-20T09:39:23.902 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:47.085552+0000", 2024-03-20T09:39:23.903 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 276.76844220800001, 2024-03-20T09:39:23.903 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.60347295400000001, 2024-03-20T09:39:23.903 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:23.903 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:39:23.903 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:39:23.903 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:39:23.903 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:39:23.903 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221706 2024-03-20T09:39:23.903 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.903 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:23.903 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.903 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:23.903 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.085552+0000", 2024-03-20T09:39:23.903 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.904 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.904 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.904 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:23.904 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.085552+0000", 2024-03-20T09:39:23.904 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.904 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.904 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.904 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:23.904 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.085554+0000", 2024-03-20T09:39:23.904 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.359e-06 2024-03-20T09:39:23.904 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.904 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.904 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:23.904 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.085560+0000", 2024-03-20T09:39:23.904 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0039999999999996e-06 2024-03-20T09:39:23.905 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.905 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.905 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:23.905 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.085574+0000", 2024-03-20T09:39:23.905 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4746000000000001e-05 2024-03-20T09:39:23.905 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.905 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.905 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:23.905 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.085583+0000", 2024-03-20T09:39:23.905 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.816e-06 2024-03-20T09:39:23.905 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.905 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.905 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.905 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.085645+0000", 2024-03-20T09:39:23.906 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2094999999999994e-05 2024-03-20T09:39:23.906 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.906 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.906 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:39:23.906 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.085655+0000", 2024-03-20T09:39:23.906 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.6309999999999998e-06 2024-03-20T09:39:23.906 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.906 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.906 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.906 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689021+0000", 2024-03-20T09:39:23.906 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.60336599300000004 2024-03-20T09:39:23.906 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.906 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.907 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:23.907 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689025+0000", 2024-03-20T09:39:23.907 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.3100000000000002e-06 2024-03-20T09:39:23.907 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.907 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:23.907 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.907 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.907 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.907 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221363 7.12 7.345f24b2 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e941)", 2024-03-20T09:39:23.908 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:47.146989+0000", 2024-03-20T09:39:23.908 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 276.70700521600003, 2024-03-20T09:39:23.908 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54205875999999997, 2024-03-20T09:39:23.908 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:23.908 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:39:23.908 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:39:23.908 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:39:23.908 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:39:23.908 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221363 2024-03-20T09:39:23.908 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.908 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:23.908 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.908 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:23.908 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.146989+0000", 2024-03-20T09:39:23.909 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.909 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.909 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.909 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:23.909 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.146989+0000", 2024-03-20T09:39:23.909 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.909 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.909 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.909 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:23.909 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.146990+0000", 2024-03-20T09:39:23.909 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.6000000000000003e-07 2024-03-20T09:39:23.909 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.909 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.909 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:23.910 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.146993+0000", 2024-03-20T09:39:23.910 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.326e-06 2024-03-20T09:39:23.910 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.910 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.910 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:23.910 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147000+0000", 2024-03-20T09:39:23.910 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.9519999999999997e-06 2024-03-20T09:39:23.910 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.910 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.910 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:23.910 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147006+0000", 2024-03-20T09:39:23.910 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1930000000000001e-06 2024-03-20T09:39:23.910 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.910 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.911 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.911 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147054+0000", 2024-03-20T09:39:23.911 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.7976999999999998e-05 2024-03-20T09:39:23.911 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.911 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.911 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:39:23.911 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147060+0000", 2024-03-20T09:39:23.911 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1630000000000001e-06 2024-03-20T09:39:23.911 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.911 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.911 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.911 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689039+0000", 2024-03-20T09:39:23.911 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54197916000000002 2024-03-20T09:39:23.912 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.912 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.912 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:23.912 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689048+0000", 2024-03-20T09:39:23.912 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.2289999999999992e-06 2024-03-20T09:39:23.912 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.912 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:23.912 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.912 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.912 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.912 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221365 7.12 7.f3fcb692 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e941)", 2024-03-20T09:39:23.912 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:47.147019+0000", 2024-03-20T09:39:23.912 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 276.70697589999997, 2024-03-20T09:39:23.912 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54204330700000003, 2024-03-20T09:39:23.913 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:23.913 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:39:23.913 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:39:23.913 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:39:23.913 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:39:23.913 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221365 2024-03-20T09:39:23.913 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.913 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:23.913 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.913 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:23.913 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147019+0000", 2024-03-20T09:39:23.913 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.913 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.913 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.914 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:23.914 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147019+0000", 2024-03-20T09:39:23.914 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.914 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.914 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.914 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:23.914 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147019+0000", 2024-03-20T09:39:23.914 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.7300000000000002e-07 2024-03-20T09:39:23.914 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.914 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.914 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:23.914 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147021+0000", 2024-03-20T09:39:23.914 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7230000000000001e-06 2024-03-20T09:39:23.914 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.915 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.915 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:23.915 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147025+0000", 2024-03-20T09:39:23.915 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5770000000000004e-06 2024-03-20T09:39:23.915 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.915 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.915 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:23.915 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147028+0000", 2024-03-20T09:39:23.915 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.0989999999999999e-06 2024-03-20T09:39:23.915 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.915 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.915 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.915 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147089+0000", 2024-03-20T09:39:23.915 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0770999999999998e-05 2024-03-20T09:39:23.916 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.916 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.916 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:39:23.916 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147095+0000", 2024-03-20T09:39:23.916 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1770000000000003e-06 2024-03-20T09:39:23.916 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.916 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.916 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.916 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689057+0000", 2024-03-20T09:39:23.916 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54196217199999996 2024-03-20T09:39:23.916 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.916 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.916 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:23.916 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689062+0000", 2024-03-20T09:39:23.916 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.515e-06 2024-03-20T09:39:23.917 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.917 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:23.917 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.917 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.917 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.917 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221471 7.12 7.c01da832 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e941)", 2024-03-20T09:39:23.917 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:47.147080+0000", 2024-03-20T09:39:23.917 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 276.70691447199999, 2024-03-20T09:39:23.917 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54204185800000004, 2024-03-20T09:39:23.917 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:23.917 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:39:23.917 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:39:23.917 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:39:23.918 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:39:23.918 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221471 2024-03-20T09:39:23.918 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.918 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:23.918 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.918 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:23.918 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147080+0000", 2024-03-20T09:39:23.918 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.918 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.918 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.918 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:23.918 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147080+0000", 2024-03-20T09:39:23.918 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.919 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.919 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.919 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:23.919 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147080+0000", 2024-03-20T09:39:23.919 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0100000000000001e-07 2024-03-20T09:39:23.919 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.919 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.919 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:23.919 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147084+0000", 2024-03-20T09:39:23.919 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.1729999999999997e-06 2024-03-20T09:39:23.919 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.919 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.919 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:23.920 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147089+0000", 2024-03-20T09:39:23.920 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.1030000000000004e-06 2024-03-20T09:39:23.920 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.920 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.920 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:23.920 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147092+0000", 2024-03-20T09:39:23.920 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.038e-06 2024-03-20T09:39:23.920 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.920 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.920 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.920 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147172+0000", 2024-03-20T09:39:23.920 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.0086000000000005e-05 2024-03-20T09:39:23.921 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.921 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.921 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:39:23.921 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147175+0000", 2024-03-20T09:39:23.921 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.4929999999999998e-06 2024-03-20T09:39:23.921 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.921 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.921 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.921 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689114+0000", 2024-03-20T09:39:23.921 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54193840100000001 2024-03-20T09:39:23.921 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.921 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.921 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:23.922 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689122+0000", 2024-03-20T09:39:23.922 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.3629999999999994e-06 2024-03-20T09:39:23.922 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.922 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:23.922 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.922 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.922 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.922 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221511 7.12 7.31478952 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e941)", 2024-03-20T09:39:23.922 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:47.147097+0000", 2024-03-20T09:39:23.922 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 276.70689799799999, 2024-03-20T09:39:23.922 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54203903899999994, 2024-03-20T09:39:23.922 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:23.923 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:39:23.923 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:39:23.923 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:39:23.923 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:39:23.923 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221511 2024-03-20T09:39:23.923 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.923 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:23.923 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.923 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:23.923 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147097+0000", 2024-03-20T09:39:23.923 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.923 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.923 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.923 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:23.924 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147097+0000", 2024-03-20T09:39:23.924 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.924 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.924 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.924 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:23.924 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147097+0000", 2024-03-20T09:39:23.924 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1799999999999999e-07 2024-03-20T09:39:23.924 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.924 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.924 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:23.924 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147098+0000", 2024-03-20T09:39:23.924 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2279999999999999e-06 2024-03-20T09:39:23.924 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.924 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.925 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:23.925 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147101+0000", 2024-03-20T09:39:23.925 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.0850000000000001e-06 2024-03-20T09:39:23.925 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.925 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.925 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:23.925 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147104+0000", 2024-03-20T09:39:23.925 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6570000000000001e-06 2024-03-20T09:39:23.925 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.925 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.925 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.925 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147206+0000", 2024-03-20T09:39:23.925 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000102191 2024-03-20T09:39:23.925 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.925 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.926 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:39:23.926 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147210+0000", 2024-03-20T09:39:23.926 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6320000000000001e-06 2024-03-20T09:39:23.926 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.926 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.926 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.926 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689131+0000", 2024-03-20T09:39:23.926 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54192172900000002 2024-03-20T09:39:23.926 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.926 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.926 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:23.926 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689136+0000", 2024-03-20T09:39:23.926 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.2989999999999998e-06 2024-03-20T09:39:23.926 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.927 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:23.927 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.927 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.927 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.927 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221573 7.12 7.a7b14f12 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e941)", 2024-03-20T09:39:23.927 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:47.147114+0000", 2024-03-20T09:39:23.927 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 276.706880286, 2024-03-20T09:39:23.927 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54204345600000003, 2024-03-20T09:39:23.927 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:23.927 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:39:23.927 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:39:23.927 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:39:23.927 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:39:23.927 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221573 2024-03-20T09:39:23.927 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.928 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:23.928 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.928 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:23.928 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147114+0000", 2024-03-20T09:39:23.928 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.928 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.928 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.928 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:23.928 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147114+0000", 2024-03-20T09:39:23.928 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.928 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.928 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.928 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:23.928 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147114+0000", 2024-03-20T09:39:23.929 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.23e-07 2024-03-20T09:39:23.929 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.929 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.929 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:23.929 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147116+0000", 2024-03-20T09:39:23.929 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.333e-06 2024-03-20T09:39:23.929 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.929 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.929 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:23.929 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147119+0000", 2024-03-20T09:39:23.929 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6509999999999999e-06 2024-03-20T09:39:23.929 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.929 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.929 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:23.929 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147122+0000", 2024-03-20T09:39:23.930 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8279999999999999e-06 2024-03-20T09:39:23.930 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.930 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.930 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.930 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147238+0000", 2024-03-20T09:39:23.930 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000115467 2024-03-20T09:39:23.930 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.930 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.930 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:39:23.930 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147244+0000", 2024-03-20T09:39:23.930 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.3400000000000003e-06 2024-03-20T09:39:23.930 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.930 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.931 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.931 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689149+0000", 2024-03-20T09:39:23.931 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54190539699999996 2024-03-20T09:39:23.931 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.931 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.931 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:23.931 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689158+0000", 2024-03-20T09:39:23.931 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.2169999999999994e-06 2024-03-20T09:39:23.931 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.931 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:23.931 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.931 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.931 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.931 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221579 7.12 7.e0ab76d2 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e941)", 2024-03-20T09:39:23.932 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:47.147132+0000", 2024-03-20T09:39:23.932 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 276.70686235800002, 2024-03-20T09:39:23.932 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54204047399999999, 2024-03-20T09:39:23.932 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:23.932 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:39:23.932 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:39:23.932 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:39:23.932 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:39:23.932 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221579 2024-03-20T09:39:23.932 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.932 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:23.932 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.932 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:23.932 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147132+0000", 2024-03-20T09:39:23.932 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.933 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.933 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.933 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:23.933 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147132+0000", 2024-03-20T09:39:23.933 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.933 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.933 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.933 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:23.933 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147132+0000", 2024-03-20T09:39:23.933 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.16e-07 2024-03-20T09:39:23.933 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.933 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.933 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:23.934 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147134+0000", 2024-03-20T09:39:23.934 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.291e-06 2024-03-20T09:39:23.934 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.934 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.934 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:23.934 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147138+0000", 2024-03-20T09:39:23.934 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.1189999999999999e-06 2024-03-20T09:39:23.934 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.934 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.934 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:23.934 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147141+0000", 2024-03-20T09:39:23.934 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.0319999999999999e-06 2024-03-20T09:39:23.934 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.934 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.935 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.935 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147275+0000", 2024-03-20T09:39:23.935 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013443600000000001 2024-03-20T09:39:23.935 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.935 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.935 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:39:23.935 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147282+0000", 2024-03-20T09:39:23.935 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4749999999999998e-06 2024-03-20T09:39:23.935 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.935 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.935 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.935 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689167+0000", 2024-03-20T09:39:23.935 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54188511299999997 2024-03-20T09:39:23.935 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.935 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.936 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:23.936 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689173+0000", 2024-03-20T09:39:23.936 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7919999999999998e-06 2024-03-20T09:39:23.936 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.936 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:23.936 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.936 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.936 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.936 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221590 7.12 7.6fbad972 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e941)", 2024-03-20T09:39:23.936 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:47.147144+0000", 2024-03-20T09:39:23.936 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 276.70685003800003, 2024-03-20T09:39:23.936 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54206811799999999, 2024-03-20T09:39:23.936 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:23.937 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:39:23.937 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:39:23.937 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:39:23.937 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:39:23.937 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221590 2024-03-20T09:39:23.937 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.937 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:23.937 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.937 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:23.937 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147144+0000", 2024-03-20T09:39:23.937 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.937 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.937 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.937 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:23.937 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147144+0000", 2024-03-20T09:39:23.938 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.938 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.938 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.938 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:23.938 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147145+0000", 2024-03-20T09:39:23.938 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.99e-07 2024-03-20T09:39:23.938 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.938 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.938 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:23.938 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147146+0000", 2024-03-20T09:39:23.938 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.629e-06 2024-03-20T09:39:23.938 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.938 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.939 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:23.939 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147149+0000", 2024-03-20T09:39:23.939 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.1020000000000001e-06 2024-03-20T09:39:23.939 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.939 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.939 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:23.939 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147152+0000", 2024-03-20T09:39:23.939 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.0060000000000001e-06 2024-03-20T09:39:23.939 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.939 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.939 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.939 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147315+0000", 2024-03-20T09:39:23.939 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000162547 2024-03-20T09:39:23.939 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.939 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.940 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:39:23.940 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147319+0000", 2024-03-20T09:39:23.940 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.3209999999999998e-06 2024-03-20T09:39:23.940 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.940 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.940 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.940 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689204+0000", 2024-03-20T09:39:23.940 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54188447900000003 2024-03-20T09:39:23.940 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.940 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.940 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:23.940 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689213+0000", 2024-03-20T09:39:23.940 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.8349999999999993e-06 2024-03-20T09:39:23.940 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.941 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:23.941 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.941 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.941 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.941 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221593 7.12 7.1ad4eaf2 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e941)", 2024-03-20T09:39:23.941 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:47.147166+0000", 2024-03-20T09:39:23.941 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 276.70682840900002, 2024-03-20T09:39:23.941 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54207133299999999, 2024-03-20T09:39:23.941 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:23.941 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:39:23.941 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:39:23.941 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:39:23.941 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:39:23.941 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221593 2024-03-20T09:39:23.942 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.942 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:23.942 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.942 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:23.942 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147166+0000", 2024-03-20T09:39:23.942 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.942 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.942 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.942 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:23.942 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147166+0000", 2024-03-20T09:39:23.942 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.942 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.942 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.942 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:23.943 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147166+0000", 2024-03-20T09:39:23.943 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0900000000000001e-07 2024-03-20T09:39:23.943 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.943 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.943 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:23.943 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147168+0000", 2024-03-20T09:39:23.943 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.308e-06 2024-03-20T09:39:23.943 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.943 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.943 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:23.943 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147173+0000", 2024-03-20T09:39:23.943 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2619999999999998e-06 2024-03-20T09:39:23.944 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.944 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.944 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:23.944 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147176+0000", 2024-03-20T09:39:23.944 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.7209999999999999e-06 2024-03-20T09:39:23.944 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.944 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.944 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.944 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147348+0000", 2024-03-20T09:39:23.944 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00017213800000000001 2024-03-20T09:39:23.944 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.944 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.944 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:39:23.945 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147351+0000", 2024-03-20T09:39:23.945 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6940000000000001e-06 2024-03-20T09:39:23.945 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.945 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.945 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.945 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689233+0000", 2024-03-20T09:39:23.945 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54188119899999998 2024-03-20T09:39:23.945 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.945 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.945 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:23.945 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689237+0000", 2024-03-20T09:39:23.945 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8019999999999999e-06 2024-03-20T09:39:23.945 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.945 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:23.946 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.946 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.946 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.946 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221706 7.12 7.6687b8b2 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e941)", 2024-03-20T09:39:23.946 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:47.147193+0000", 2024-03-20T09:39:23.946 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 276.70680111799999, 2024-03-20T09:39:23.946 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54207810999999995, 2024-03-20T09:39:23.946 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:23.946 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:39:23.946 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:39:23.946 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:39:23.946 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:39:23.946 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221706 2024-03-20T09:39:23.947 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.947 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:23.947 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.947 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:23.947 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147193+0000", 2024-03-20T09:39:23.947 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.947 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.947 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.947 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:23.947 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147193+0000", 2024-03-20T09:39:23.947 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:23.947 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.947 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.947 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:23.947 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147194+0000", 2024-03-20T09:39:23.948 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.34e-07 2024-03-20T09:39:23.948 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.948 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.948 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:23.948 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147195+0000", 2024-03-20T09:39:23.948 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.643e-06 2024-03-20T09:39:23.948 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.948 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.948 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:23.948 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147199+0000", 2024-03-20T09:39:23.948 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7309999999999999e-06 2024-03-20T09:39:23.948 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.948 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.948 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:23.949 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147202+0000", 2024-03-20T09:39:23.949 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.96e-06 2024-03-20T09:39:23.949 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.949 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.949 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.949 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147364+0000", 2024-03-20T09:39:23.949 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000162306 2024-03-20T09:39:23.949 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.949 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.949 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:39:23.949 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147368+0000", 2024-03-20T09:39:23.949 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.3900000000000002e-06 2024-03-20T09:39:23.949 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.949 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.950 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:23.950 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689263+0000", 2024-03-20T09:39:23.950 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54189511499999998 2024-03-20T09:39:23.950 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:23.950 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:23.950 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:23.950 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689272+0000", 2024-03-20T09:39:23.950 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.7309999999999999e-06 2024-03-20T09:39:23.950 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.950 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:23.950 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.950 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:23.950 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:23.951 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:39:24.014 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:39:24.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:24 smithi115 ceph-mon[72476]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-20T09:39:24.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:24 smithi115 ceph-mon[72476]: 1.f scrub starts 2024-03-20T09:39:24.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:24 smithi115 ceph-mon[72476]: 8.f scrub starts 2024-03-20T09:39:24.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:24 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4274660467' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:39:24.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:24 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:39:24.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:24 smithi115 ceph-mon[72476]: 1.f scrub ok 2024-03-20T09:39:24.429 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:24 smithi026 ceph-mon[68309]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-20T09:39:24.429 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:24 smithi026 ceph-mon[68309]: 1.f scrub starts 2024-03-20T09:39:24.429 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:24 smithi026 ceph-mon[68309]: 8.f scrub starts 2024-03-20T09:39:24.429 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:24 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4274660467' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:39:24.429 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:24 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:39:24.429 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:24 smithi026 ceph-mon[68309]: 1.f scrub ok 2024-03-20T09:39:24.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:24 smithi122 ceph-mon[73495]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-20T09:39:24.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:24 smithi122 ceph-mon[73495]: 1.f scrub starts 2024-03-20T09:39:24.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:24 smithi122 ceph-mon[73495]: 8.f scrub starts 2024-03-20T09:39:24.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:24 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4274660467' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:39:24.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:24 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:39:24.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:24 smithi122 ceph-mon[73495]: 1.f scrub ok 2024-03-20T09:39:24.550 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-20T09:39:24.883 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:39:25.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:25 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:39:25.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:25 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:39:25.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:25 smithi026 ceph-mon[68309]: osdmap e1044: 12 total, 12 up, 11 in 2024-03-20T09:39:25.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:25 smithi026 ceph-mon[68309]: 8.f scrub ok 2024-03-20T09:39:25.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:25 smithi026 ceph-mon[68309]: 8.d scrub starts 2024-03-20T09:39:25.210 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:25 smithi026 ceph-mon[68309]: pgmap v2528: 47 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 44 active+clean; 2.0 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 586 op/s; 3901/95850 objects misplaced (4.070%); 20 MiB/s, 312 objects/s recovering 2024-03-20T09:39:25.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:25 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:39:25.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:25 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:39:25.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:25 smithi115 ceph-mon[72476]: osdmap e1044: 12 total, 12 up, 11 in 2024-03-20T09:39:25.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:25 smithi115 ceph-mon[72476]: 8.f scrub ok 2024-03-20T09:39:25.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:25 smithi115 ceph-mon[72476]: 8.d scrub starts 2024-03-20T09:39:25.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:25 smithi115 ceph-mon[72476]: pgmap v2528: 47 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 44 active+clean; 2.0 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 586 op/s; 3901/95850 objects misplaced (4.070%); 20 MiB/s, 312 objects/s recovering 2024-03-20T09:39:25.425 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:39:25.429 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:25.429 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:39:25.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:25 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:39:25.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:25 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:39:25.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:25 smithi122 ceph-mon[73495]: osdmap e1044: 12 total, 12 up, 11 in 2024-03-20T09:39:25.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:25 smithi122 ceph-mon[73495]: 8.f scrub ok 2024-03-20T09:39:25.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:25 smithi122 ceph-mon[73495]: 8.d scrub starts 2024-03-20T09:39:25.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:25 smithi122 ceph-mon[73495]: pgmap v2528: 47 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 44 active+clean; 2.0 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 586 op/s; 3901/95850 objects misplaced (4.070%); 20 MiB/s, 312 objects/s recovering 2024-03-20T09:39:25.531 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:39:25.590 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:25.598 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:39:25.598 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:39:25.598 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:39:25.598 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:39:25.598 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:39:25.598 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:39:25.598 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:39:25.598 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:39:25.635 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:39:25.686 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:39:25.719 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:39:25.759 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:39:25.762 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:39:25.762 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:39:25.762 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:39:25.762 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:39:25.762 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:39:25.762 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:39:25.762 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:39:25.762 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:39:25.762 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:39:25.763 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:39:25.763 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:39:25.763 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:39:26.117 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:26 smithi115 ceph-mon[72476]: 1.17 scrub starts 2024-03-20T09:39:26.118 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:26 smithi115 ceph-mon[72476]: 1.17 scrub ok 2024-03-20T09:39:26.118 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:26 smithi115 ceph-mon[72476]: 8.d scrub ok 2024-03-20T09:39:26.118 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:26 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2751792013' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:39:26.118 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:26 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:39:26.118 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:26 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:39:26.128 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:26 smithi026 ceph-mon[68309]: 1.17 scrub starts 2024-03-20T09:39:26.128 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:26 smithi026 ceph-mon[68309]: 1.17 scrub ok 2024-03-20T09:39:26.128 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:26 smithi026 ceph-mon[68309]: 8.d scrub ok 2024-03-20T09:39:26.128 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:26 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2751792013' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:39:26.128 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:26 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:39:26.128 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:26 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:39:26.374 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:26 smithi122 ceph-mon[73495]: 1.17 scrub starts 2024-03-20T09:39:26.374 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:26 smithi122 ceph-mon[73495]: 1.17 scrub ok 2024-03-20T09:39:26.374 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:26 smithi122 ceph-mon[73495]: 8.d scrub ok 2024-03-20T09:39:26.374 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:26 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2751792013' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:39:26.374 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:26 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:39:26.374 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:26 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:39:26.757 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:39:26.758 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:39:27.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:27 smithi115 ceph-mon[72476]: osdmap e1045: 12 total, 12 up, 11 in 2024-03-20T09:39:27.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:27 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:39:27.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:27 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:39:27.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:27 smithi115 ceph-mon[72476]: 1.13 scrub starts 2024-03-20T09:39:27.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:27 smithi115 ceph-mon[72476]: 1.13 scrub ok 2024-03-20T09:39:27.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:27 smithi115 ceph-mon[72476]: pgmap v2530: 47 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 44 active+clean; 1.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 745 op/s; 3901/90198 objects misplaced (4.325%); 11 MiB/s, 182 objects/s recovering 2024-03-20T09:39:27.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:27 smithi122 ceph-mon[73495]: osdmap e1045: 12 total, 12 up, 11 in 2024-03-20T09:39:27.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:27 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:39:27.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:27 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:39:27.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:27 smithi122 ceph-mon[73495]: 1.13 scrub starts 2024-03-20T09:39:27.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:27 smithi122 ceph-mon[73495]: 1.13 scrub ok 2024-03-20T09:39:27.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:27 smithi122 ceph-mon[73495]: pgmap v2530: 47 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 44 active+clean; 1.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 745 op/s; 3901/90198 objects misplaced (4.325%); 11 MiB/s, 182 objects/s recovering 2024-03-20T09:39:27.448 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-20T09:39:27.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:27 smithi026 ceph-mon[68309]: osdmap e1045: 12 total, 12 up, 11 in 2024-03-20T09:39:27.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:27 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:39:27.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:27 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:39:27.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:27 smithi026 ceph-mon[68309]: 1.13 scrub starts 2024-03-20T09:39:27.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:27 smithi026 ceph-mon[68309]: 1.13 scrub ok 2024-03-20T09:39:27.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:27 smithi026 ceph-mon[68309]: pgmap v2530: 47 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 44 active+clean; 1.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 745 op/s; 3901/90198 objects misplaced (4.325%); 11 MiB/s, 182 objects/s recovering 2024-03-20T09:39:27.530 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:39:27.772 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:39:27.949 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:39:28.307 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:28 smithi115 ceph-mon[72476]: osdmap e1046: 12 total, 12 up, 11 in 2024-03-20T09:39:28.307 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:28 smithi115 ceph-mon[72476]: 1.1d scrub starts 2024-03-20T09:39:28.307 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:28 smithi115 ceph-mon[72476]: 1.1d scrub ok 2024-03-20T09:39:28.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:28 smithi122 ceph-mon[73495]: osdmap e1046: 12 total, 12 up, 11 in 2024-03-20T09:39:28.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:28 smithi122 ceph-mon[73495]: 1.1d scrub starts 2024-03-20T09:39:28.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:28 smithi122 ceph-mon[73495]: 1.1d scrub ok 2024-03-20T09:39:28.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:28 smithi026 ceph-mon[68309]: osdmap e1046: 12 total, 12 up, 11 in 2024-03-20T09:39:28.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:28 smithi026 ceph-mon[68309]: 1.1d scrub starts 2024-03-20T09:39:28.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:28 smithi026 ceph-mon[68309]: 1.1d scrub ok 2024-03-20T09:39:28.479 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:39:29.134 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:29 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:39:29.134 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:29 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:39:29.134 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:29 smithi115 ceph-mon[72476]: pgmap v2532: 47 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 44 active+clean; 1.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 710 op/s; 3679/86907 objects misplaced (4.233%); 11 MiB/s, 182 objects/s recovering 2024-03-20T09:39:29.381 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:29 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:39:29.381 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:29 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:39:29.381 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:29 smithi026 ceph-mon[68309]: pgmap v2532: 47 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 44 active+clean; 1.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 710 op/s; 3679/86907 objects misplaced (4.233%); 11 MiB/s, 182 objects/s recovering 2024-03-20T09:39:29.425 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:39:29.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:29 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:39:29.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:29 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:39:29.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:29 smithi122 ceph-mon[73495]: pgmap v2532: 47 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 44 active+clean; 1.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 710 op/s; 3679/86907 objects misplaced (4.233%); 11 MiB/s, 182 objects/s recovering 2024-03-20T09:39:29.722 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:39:29.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:29.727 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:39:29.790 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:39:29.791 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:39:29.828 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:39:29.873 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:29.901 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:39:29.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:29.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:39:30.007 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:39:30.035 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:30.057 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:39:30.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:30.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:39:30.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:30 smithi115 ceph-mon[72476]: osdmap e1047: 12 total, 12 up, 11 in 2024-03-20T09:39:30.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:30 smithi115 ceph-mon[72476]: 1.1 scrub starts 2024-03-20T09:39:30.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:30 smithi115 ceph-mon[72476]: 1.1 scrub ok 2024-03-20T09:39:30.161 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:39:30.214 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:30.335 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:39:30.339 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:30 smithi026 ceph-mon[68309]: osdmap e1047: 12 total, 12 up, 11 in 2024-03-20T09:39:30.339 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:30 smithi026 ceph-mon[68309]: 1.1 scrub starts 2024-03-20T09:39:30.339 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:30 smithi026 ceph-mon[68309]: 1.1 scrub ok 2024-03-20T09:39:30.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:30.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:39:30.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:30 smithi122 ceph-mon[73495]: osdmap e1047: 12 total, 12 up, 11 in 2024-03-20T09:39:30.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:30 smithi122 ceph-mon[73495]: 1.1 scrub starts 2024-03-20T09:39:30.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:30 smithi122 ceph-mon[73495]: 1.1 scrub ok 2024-03-20T09:39:30.441 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:39:30.498 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:31.082 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:39:31.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:31 smithi115 ceph-mon[72476]: 1.15 scrub starts 2024-03-20T09:39:31.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:31 smithi115 ceph-mon[72476]: 1.15 scrub ok 2024-03-20T09:39:31.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:31 smithi115 ceph-mon[72476]: 8.7 scrub starts 2024-03-20T09:39:31.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:31 smithi115 ceph-mon[72476]: pgmap v2534: 47 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 44 active+clean; 1.7 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 726 op/s; 2516/82776 objects misplaced (3.040%); 10 MiB/s, 166 objects/s recovering 2024-03-20T09:39:31.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:31 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3415171546' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:39:31.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:31 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:39:31.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:31 smithi122 ceph-mon[73495]: 1.15 scrub starts 2024-03-20T09:39:31.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:31 smithi122 ceph-mon[73495]: 1.15 scrub ok 2024-03-20T09:39:31.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:31 smithi122 ceph-mon[73495]: 8.7 scrub starts 2024-03-20T09:39:31.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:31 smithi122 ceph-mon[73495]: pgmap v2534: 47 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 44 active+clean; 1.7 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 726 op/s; 2516/82776 objects misplaced (3.040%); 10 MiB/s, 166 objects/s recovering 2024-03-20T09:39:31.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:31 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3415171546' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:39:31.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:31 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:39:31.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:31 smithi026 ceph-mon[68309]: 1.15 scrub starts 2024-03-20T09:39:31.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:31 smithi026 ceph-mon[68309]: 1.15 scrub ok 2024-03-20T09:39:31.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:31 smithi026 ceph-mon[68309]: 8.7 scrub starts 2024-03-20T09:39:31.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:31 smithi026 ceph-mon[68309]: pgmap v2534: 47 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 44 active+clean; 1.7 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 726 op/s; 2516/82776 objects misplaced (3.040%); 10 MiB/s, 166 objects/s recovering 2024-03-20T09:39:31.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:31 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3415171546' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:39:31.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:31 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:39:31.470 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-20T09:39:31.793 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:39:32.368 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:32 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:39:32.368 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:32 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:39:32.368 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:32 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:39:32.368 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:32 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:39:32.368 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:32 smithi115 ceph-mon[72476]: 8.7 scrub ok 2024-03-20T09:39:32.368 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:32 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:39:32.369 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:32 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:39:32.369 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:32 smithi115 ceph-mon[72476]: osdmap e1048: 12 total, 12 up, 11 in 2024-03-20T09:39:32.376 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:39:32.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:32.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:39:32.380 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:32 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:39:32.380 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:32 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:39:32.380 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:32 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:39:32.380 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:32 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:39:32.380 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:32 smithi026 ceph-mon[68309]: 8.7 scrub ok 2024-03-20T09:39:32.380 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:32 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:39:32.380 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:32 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:39:32.380 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:32 smithi026 ceph-mon[68309]: osdmap e1048: 12 total, 12 up, 11 in 2024-03-20T09:39:32.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:32 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:39:32.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:32 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:39:32.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:32 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:39:32.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:32 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:39:32.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:32 smithi122 ceph-mon[73495]: 8.7 scrub ok 2024-03-20T09:39:32.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:32 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:39:32.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:32 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:39:32.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:32 smithi122 ceph-mon[73495]: osdmap e1048: 12 total, 12 up, 11 in 2024-03-20T09:39:32.480 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:39:32.511 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:32.534 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:39:32.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:32.537 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:39:32.613 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:39:32.632 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:39:32.638 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:39:32.652 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:39:32.669 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:32.672 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:39:32.692 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:39:32.712 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:39:32.732 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:39:32.752 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:39:32.773 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:39:32.793 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:39:32.813 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:39:32.833 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:39:32.835 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:39:32.835 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:39:32.835 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:39:32.835 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:39:32.835 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:39:32.835 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:39:32.835 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:39:32.835 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:39:32.835 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:39:32.835 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:39:32.835 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:39:32.835 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:39:33.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:33 smithi026 ceph-mon[68309]: osdmap e1049: 12 total, 12 up, 11 in 2024-03-20T09:39:33.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:33 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/702265323' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:39:33.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:33 smithi026 ceph-mon[68309]: pgmap v2537: 47 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 43 active+clean; 1.6 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 645 op/s; 2516/78114 objects misplaced (3.221%); 11 MiB/s, 179 objects/s recovering 2024-03-20T09:39:33.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:33 smithi026 ceph-mon[68309]: osdmap e1050: 12 total, 12 up, 11 in 2024-03-20T09:39:33.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:33 smithi115 ceph-mon[72476]: osdmap e1049: 12 total, 12 up, 11 in 2024-03-20T09:39:33.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:33 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/702265323' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:39:33.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:33 smithi115 ceph-mon[72476]: pgmap v2537: 47 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 43 active+clean; 1.6 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 645 op/s; 2516/78114 objects misplaced (3.221%); 11 MiB/s, 179 objects/s recovering 2024-03-20T09:39:33.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:33 smithi115 ceph-mon[72476]: osdmap e1050: 12 total, 12 up, 11 in 2024-03-20T09:39:33.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:33 smithi122 ceph-mon[73495]: osdmap e1049: 12 total, 12 up, 11 in 2024-03-20T09:39:33.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:33 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/702265323' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:39:33.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:33 smithi122 ceph-mon[73495]: pgmap v2537: 47 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 43 active+clean; 1.6 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 645 op/s; 2516/78114 objects misplaced (3.221%); 11 MiB/s, 179 objects/s recovering 2024-03-20T09:39:33.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:33 smithi122 ceph-mon[73495]: osdmap e1050: 12 total, 12 up, 11 in 2024-03-20T09:39:33.516 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:39:33.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:33.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:39:33.620 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:39:33.652 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:33.672 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:39:33.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:33.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:39:33.778 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:39:33.805 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:33.825 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:39:33.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:33.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:39:33.857 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:39:33.929 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:39:33.958 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:33.985 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:39:33.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:33.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:39:34.091 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:39:34.150 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:34.164 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:39:34.164 INFO:teuthology.orchestra.run.smithi115.stdout: "size": 20, 2024-03-20T09:39:34.164 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 600, 2024-03-20T09:39:34.164 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [ 2024-03-20T09:39:34.164 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.164 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "rep_scrubmap(1.15 e491 from shard 8)", 2024-03-20T09:39:34.164 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:31:32.342334+0000", 2024-03-20T09:39:34.164 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 481.83098539700001, 2024-03-20T09:39:34.164 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.068097784999999994, 2024-03-20T09:39:34.164 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:34.164 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:39:34.164 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:34.165 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.165 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:34.165 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.342334+0000", 2024-03-20T09:39:34.165 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.165 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.165 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.165 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:34.165 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.342334+0000", 2024-03-20T09:39:34.165 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.165 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.165 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.165 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:34.165 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.342335+0000", 2024-03-20T09:39:34.165 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.9699999999999995e-07 2024-03-20T09:39:34.166 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.166 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.166 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:34.166 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.342341+0000", 2024-03-20T09:39:34.166 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5620000000000003e-06 2024-03-20T09:39:34.166 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.166 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.166 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:34.166 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.342351+0000", 2024-03-20T09:39:34.166 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0621999999999999e-05 2024-03-20T09:39:34.166 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.166 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.166 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:34.166 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.342357+0000", 2024-03-20T09:39:34.167 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7440000000000004e-06 2024-03-20T09:39:34.167 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.167 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.167 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:34.167 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.410402+0000", 2024-03-20T09:39:34.167 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.068045091000000002 2024-03-20T09:39:34.167 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.167 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.167 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:39:34.167 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.410414+0000", 2024-03-20T09:39:34.167 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1867e-05 2024-03-20T09:39:34.167 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.167 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.167 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:34.168 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.410432+0000", 2024-03-20T09:39:34.168 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8102e-05 2024-03-20T09:39:34.168 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.168 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:34.168 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.168 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.168 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.168 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "rep_scrubmap(1.15 e491 from shard 5)", 2024-03-20T09:39:34.168 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:31:32.346280+0000", 2024-03-20T09:39:34.168 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 481.82703952899999, 2024-03-20T09:39:34.168 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.064250347999999999, 2024-03-20T09:39:34.168 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:34.168 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:39:34.168 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:34.169 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.169 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:34.169 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.346280+0000", 2024-03-20T09:39:34.169 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.169 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.169 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.169 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:34.169 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.346280+0000", 2024-03-20T09:39:34.169 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.169 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.169 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.169 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:34.169 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.346281+0000", 2024-03-20T09:39:34.169 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.5999999999999991e-07 2024-03-20T09:39:34.170 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.170 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.170 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:34.170 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.346287+0000", 2024-03-20T09:39:34.170 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4299999999999997e-06 2024-03-20T09:39:34.170 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.170 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.170 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:34.170 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.346301+0000", 2024-03-20T09:39:34.170 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4088e-05 2024-03-20T09:39:34.170 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.170 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.170 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:34.171 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.346308+0000", 2024-03-20T09:39:34.171 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.1319999999999996e-06 2024-03-20T09:39:34.171 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.171 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.171 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:34.171 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.410482+0000", 2024-03-20T09:39:34.171 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.064174155999999996 2024-03-20T09:39:34.171 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.171 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.171 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:39:34.171 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.410494+0000", 2024-03-20T09:39:34.171 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1439e-05 2024-03-20T09:39:34.171 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.171 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.171 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:34.172 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:31:32.410531+0000", 2024-03-20T09:39:34.172 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7143e-05 2024-03-20T09:39:34.172 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.172 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:34.172 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.172 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.172 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.172 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "rep_scrubmap(1.15 e491 from shard 5)", 2024-03-20T09:39:34.172 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:33:08.996571+0000", 2024-03-20T09:39:34.172 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 385.17674903199998, 2024-03-20T09:39:34.172 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.050566558999999997, 2024-03-20T09:39:34.172 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:34.172 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:39:34.173 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:34.173 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.173 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:34.173 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:33:08.996571+0000", 2024-03-20T09:39:34.173 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.173 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.173 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.173 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:34.173 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:33:08.996571+0000", 2024-03-20T09:39:34.173 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.173 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.173 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.173 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:34.173 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:33:08.996572+0000", 2024-03-20T09:39:34.174 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4839999999999999e-06 2024-03-20T09:39:34.174 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.174 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.174 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:34.174 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:33:08.996581+0000", 2024-03-20T09:39:34.174 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.6729999999999998e-06 2024-03-20T09:39:34.174 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.174 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.174 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:34.174 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:33:08.996597+0000", 2024-03-20T09:39:34.174 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6483000000000001e-05 2024-03-20T09:39:34.174 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.174 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.174 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:34.175 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:33:08.996607+0000", 2024-03-20T09:39:34.175 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.7270000000000002e-06 2024-03-20T09:39:34.175 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.175 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.175 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:34.175 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:33:09.047117+0000", 2024-03-20T09:39:34.175 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.050509999 2024-03-20T09:39:34.175 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.175 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.175 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:39:34.175 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:33:09.047125+0000", 2024-03-20T09:39:34.175 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.977e-06 2024-03-20T09:39:34.175 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.176 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.176 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:34.176 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:33:09.047137+0000", 2024-03-20T09:39:34.176 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2216e-05 2024-03-20T09:39:34.176 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.176 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:34.176 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.176 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.176 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.176 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26605.0:222035 7.15 e943/936 7:ad9477d6:::benchmark_data_smithi122_187077_object95424:head v 943'13498, mlcod=942'13496)", 2024-03-20T09:39:34.176 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:47.897707+0000", 2024-03-20T09:39:34.176 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 286.27561246699997, 2024-03-20T09:39:34.176 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.086363878000000005, 2024-03-20T09:39:34.176 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:34.177 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:39:34.177 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:34.177 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.177 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:34.177 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.897707+0000", 2024-03-20T09:39:34.177 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.177 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.177 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.177 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:34.177 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.897707+0000", 2024-03-20T09:39:34.177 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.177 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.177 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.177 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:34.177 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.897708+0000", 2024-03-20T09:39:34.178 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.85e-07 2024-03-20T09:39:34.178 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.178 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.178 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:34.178 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.897718+0000", 2024-03-20T09:39:34.178 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0064999999999999e-05 2024-03-20T09:39:34.178 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.178 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.178 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:34.178 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.897732+0000", 2024-03-20T09:39:34.178 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3771000000000001e-05 2024-03-20T09:39:34.178 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.178 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.179 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:34.179 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.897740+0000", 2024-03-20T09:39:34.179 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.9680000000000006e-06 2024-03-20T09:39:34.179 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.179 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.179 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:34.179 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.903105+0000", 2024-03-20T09:39:34.179 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0053646609999999997 2024-03-20T09:39:34.179 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.179 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.179 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:39:34.179 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.903126+0000", 2024-03-20T09:39:34.179 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1685e-05 2024-03-20T09:39:34.179 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.180 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.180 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:39:34.180 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.984025+0000", 2024-03-20T09:39:34.180 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.080898484000000007 2024-03-20T09:39:34.180 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.180 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.180 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:34.180 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.984071+0000", 2024-03-20T09:39:34.180 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6359000000000001e-05 2024-03-20T09:39:34.180 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.180 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:34.180 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.180 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.180 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.180 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "rep_scrubmap(1.15 e491 from shard 5)", 2024-03-20T09:39:34.181 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:48.016412+0000", 2024-03-20T09:39:34.181 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 286.156908187, 2024-03-20T09:39:34.181 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.060988498000000002, 2024-03-20T09:39:34.181 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:34.181 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:39:34.181 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:34.181 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.181 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:34.181 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.016412+0000", 2024-03-20T09:39:34.181 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.181 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.181 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.181 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:34.181 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.016412+0000", 2024-03-20T09:39:34.182 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.182 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.182 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.182 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:34.182 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.016413+0000", 2024-03-20T09:39:34.182 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.4099999999999997e-07 2024-03-20T09:39:34.182 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.182 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.182 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:34.182 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.016418+0000", 2024-03-20T09:39:34.182 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0939999999999998e-06 2024-03-20T09:39:34.182 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.182 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.182 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:34.183 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.016430+0000", 2024-03-20T09:39:34.183 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2795e-05 2024-03-20T09:39:34.183 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.183 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.183 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:34.183 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.016438+0000", 2024-03-20T09:39:34.183 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.1500000000000002e-06 2024-03-20T09:39:34.183 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.183 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.183 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:34.183 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.077379+0000", 2024-03-20T09:39:34.183 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.060941654999999997 2024-03-20T09:39:34.183 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.183 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.184 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:39:34.184 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.077387+0000", 2024-03-20T09:39:34.184 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.1019999999999997e-06 2024-03-20T09:39:34.184 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.184 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.184 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:34.184 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.077400+0000", 2024-03-20T09:39:34.184 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2761e-05 2024-03-20T09:39:34.184 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.184 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:34.184 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.184 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.184 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.185 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "rep_scrubmap(1.15 e491 from shard 8)", 2024-03-20T09:39:34.185 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:48.017154+0000", 2024-03-20T09:39:34.185 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 286.156165429, 2024-03-20T09:39:34.185 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.060297227000000002, 2024-03-20T09:39:34.185 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:34.185 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:39:34.185 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:34.185 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.185 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:34.185 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.017154+0000", 2024-03-20T09:39:34.185 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.185 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.185 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.185 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:34.186 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.017154+0000", 2024-03-20T09:39:34.186 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.186 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.186 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.186 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:34.186 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.017155+0000", 2024-03-20T09:39:34.186 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4199999999999995e-07 2024-03-20T09:39:34.186 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.186 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.186 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:34.186 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.017159+0000", 2024-03-20T09:39:34.186 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7100000000000001e-06 2024-03-20T09:39:34.186 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.186 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.187 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:34.187 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.017167+0000", 2024-03-20T09:39:34.187 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.4659999999999999e-06 2024-03-20T09:39:34.187 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.187 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.187 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:34.187 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.017172+0000", 2024-03-20T09:39:34.187 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0409999999999997e-06 2024-03-20T09:39:34.187 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.187 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.187 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:34.187 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.077425+0000", 2024-03-20T09:39:34.187 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.06025316 2024-03-20T09:39:34.188 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.188 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.188 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:39:34.188 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.077432+0000", 2024-03-20T09:39:34.188 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.7270000000000001e-06 2024-03-20T09:39:34.188 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.188 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.188 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:34.188 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.077452+0000", 2024-03-20T09:39:34.188 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9480999999999999e-05 2024-03-20T09:39:34.188 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.188 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:34.188 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.188 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.188 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.189 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:14523 8.d e980/974 8:b1c3d5d8:::benchmark_data_smithi122_197775_object14522:head v 980'870, mlcod=980'868)", 2024-03-20T09:39:34.189 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:05.332829+0000", 2024-03-20T09:39:34.189 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 208.840491351, 2024-03-20T09:39:34.189 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.080553347999999997, 2024-03-20T09:39:34.189 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:34.189 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:39:34.189 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:34.189 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.189 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:34.189 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.332829+0000", 2024-03-20T09:39:34.189 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.189 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.189 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.189 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:34.190 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.332829+0000", 2024-03-20T09:39:34.190 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.190 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.190 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.190 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:34.190 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.332829+0000", 2024-03-20T09:39:34.190 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5000000000000001e-07 2024-03-20T09:39:34.190 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.190 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.190 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:34.190 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.332921+0000", 2024-03-20T09:39:34.190 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.1848999999999994e-05 2024-03-20T09:39:34.190 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.190 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.191 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:34.191 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.332933+0000", 2024-03-20T09:39:34.191 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2071e-05 2024-03-20T09:39:34.191 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.191 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.191 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:34.191 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.332941+0000", 2024-03-20T09:39:34.191 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.8520000000000004e-06 2024-03-20T09:39:34.191 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.191 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.191 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:34.191 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.332994+0000", 2024-03-20T09:39:34.191 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2398000000000002e-05 2024-03-20T09:39:34.191 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.191 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.192 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:39:34.192 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.333015+0000", 2024-03-20T09:39:34.192 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1807999999999999e-05 2024-03-20T09:39:34.192 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.192 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.192 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:39:34.192 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.413352+0000", 2024-03-20T09:39:34.192 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.080337069999999997 2024-03-20T09:39:34.192 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.192 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.192 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:34.192 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.413382+0000", 2024-03-20T09:39:34.192 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9450000000000001e-05 2024-03-20T09:39:34.192 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.193 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:34.193 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.193 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.193 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.193 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:14525 8.d e980/974 8:bd7f84eb:::benchmark_data_smithi122_197775_object14524:head v 980'871, mlcod=980'868)", 2024-03-20T09:39:34.193 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:05.335283+0000", 2024-03-20T09:39:34.193 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 208.838036985, 2024-03-20T09:39:34.193 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.078128146999999995, 2024-03-20T09:39:34.193 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:34.193 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:39:34.193 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:34.193 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.193 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:34.193 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.335283+0000", 2024-03-20T09:39:34.194 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.194 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.194 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.194 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:34.194 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.335283+0000", 2024-03-20T09:39:34.194 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.194 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.194 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.194 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:34.194 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.335283+0000", 2024-03-20T09:39:34.194 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4700000000000001e-07 2024-03-20T09:39:34.194 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.194 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.194 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:34.195 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.335453+0000", 2024-03-20T09:39:34.195 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00016961600000000001 2024-03-20T09:39:34.195 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.195 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.195 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:34.195 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.335462+0000", 2024-03-20T09:39:34.195 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.2499999999999995e-06 2024-03-20T09:39:34.195 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.195 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.195 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:34.195 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.335468+0000", 2024-03-20T09:39:34.195 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8189999999999997e-06 2024-03-20T09:39:34.195 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.195 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.196 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:34.196 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.335511+0000", 2024-03-20T09:39:34.196 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.2478999999999998e-05 2024-03-20T09:39:34.196 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.196 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.196 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:39:34.196 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.335531+0000", 2024-03-20T09:39:34.196 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0040000000000001e-05 2024-03-20T09:39:34.196 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.196 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.196 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:39:34.196 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.413385+0000", 2024-03-20T09:39:34.196 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.077854520999999996 2024-03-20T09:39:34.196 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.197 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.197 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:34.197 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.413411+0000", 2024-03-20T09:39:34.197 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5874999999999999e-05 2024-03-20T09:39:34.197 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.197 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:34.197 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.197 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.197 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.197 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:14534 8.d e980/974 8:bd505d36:::benchmark_data_smithi122_197775_object14533:head v 980'872, mlcod=980'868)", 2024-03-20T09:39:34.197 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:05.337841+0000", 2024-03-20T09:39:34.197 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 208.835479109, 2024-03-20T09:39:34.197 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.075595009000000005, 2024-03-20T09:39:34.198 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:34.198 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:39:34.198 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:34.198 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.198 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:34.198 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.337841+0000", 2024-03-20T09:39:34.198 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.198 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.198 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.198 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:34.198 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.337841+0000", 2024-03-20T09:39:34.198 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.198 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.198 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.199 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:34.199 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.337841+0000", 2024-03-20T09:39:34.199 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.5199999999999998e-07 2024-03-20T09:39:34.199 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.199 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.199 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:34.199 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.337928+0000", 2024-03-20T09:39:34.199 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.6570000000000006e-05 2024-03-20T09:39:34.199 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.199 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.199 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:34.199 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.337935+0000", 2024-03-20T09:39:34.199 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.4059999999999999e-06 2024-03-20T09:39:34.199 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.200 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.200 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:34.200 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.337941+0000", 2024-03-20T09:39:34.200 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8699999999999997e-06 2024-03-20T09:39:34.200 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.200 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.200 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:34.200 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.337985+0000", 2024-03-20T09:39:34.200 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4030999999999999e-05 2024-03-20T09:39:34.200 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.200 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.200 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:39:34.200 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.338003+0000", 2024-03-20T09:39:34.200 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7759000000000001e-05 2024-03-20T09:39:34.201 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.201 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.201 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:39:34.201 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.413414+0000", 2024-03-20T09:39:34.201 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.075411117 2024-03-20T09:39:34.201 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.201 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.201 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:34.201 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.413436+0000", 2024-03-20T09:39:34.201 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1903999999999999e-05 2024-03-20T09:39:34.201 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.201 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:34.201 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.201 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.201 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.202 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:14545 8.d e980/974 8:b7f9247f:::benchmark_data_smithi122_197775_object14544:head v 980'873, mlcod=980'868)", 2024-03-20T09:39:34.202 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:05.340825+0000", 2024-03-20T09:39:34.202 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 208.83249506300001, 2024-03-20T09:39:34.202 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.072634562999999999, 2024-03-20T09:39:34.202 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:34.202 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:39:34.202 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:34.202 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.202 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:34.202 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.340825+0000", 2024-03-20T09:39:34.202 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.202 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.202 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.203 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:34.203 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.340825+0000", 2024-03-20T09:39:34.203 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.203 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.203 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.203 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:34.203 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.340825+0000", 2024-03-20T09:39:34.203 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.75e-07 2024-03-20T09:39:34.203 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.203 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.203 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:34.203 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.340902+0000", 2024-03-20T09:39:34.203 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7058999999999995e-05 2024-03-20T09:39:34.203 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.203 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.204 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:34.204 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.340911+0000", 2024-03-20T09:39:34.204 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.7399999999999993e-06 2024-03-20T09:39:34.204 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.204 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.204 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:34.204 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.340918+0000", 2024-03-20T09:39:34.204 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.5910000000000001e-06 2024-03-20T09:39:34.204 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.204 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.204 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:34.204 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.340957+0000", 2024-03-20T09:39:34.204 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.9146000000000001e-05 2024-03-20T09:39:34.204 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.205 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.205 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:39:34.205 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.340972+0000", 2024-03-20T09:39:34.205 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4972e-05 2024-03-20T09:39:34.205 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.205 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.205 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:39:34.205 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.413438+0000", 2024-03-20T09:39:34.205 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.072465871000000001 2024-03-20T09:39:34.205 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.205 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.205 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:34.205 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.413459+0000", 2024-03-20T09:39:34.205 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1608999999999999e-05 2024-03-20T09:39:34.205 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.206 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:34.206 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.206 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.206 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.206 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:14559 8.d e980/974 8:bfa51422:::benchmark_data_smithi122_197775_object14558:head v 980'874, mlcod=980'868)", 2024-03-20T09:39:34.206 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:05.348897+0000", 2024-03-20T09:39:34.206 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 208.824422825, 2024-03-20T09:39:34.206 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.064589677999999998, 2024-03-20T09:39:34.206 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:34.206 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:39:34.206 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:34.206 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.206 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:34.206 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.348897+0000", 2024-03-20T09:39:34.207 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.207 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.207 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.207 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:34.207 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.348897+0000", 2024-03-20T09:39:34.207 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.207 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.207 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.207 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:34.207 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.348898+0000", 2024-03-20T09:39:34.207 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.023e-06 2024-03-20T09:39:34.207 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.207 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.207 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:34.207 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.349026+0000", 2024-03-20T09:39:34.208 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012763799999999999 2024-03-20T09:39:34.208 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.208 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.208 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:34.208 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.349042+0000", 2024-03-20T09:39:34.208 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6256000000000002e-05 2024-03-20T09:39:34.208 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.208 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.208 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:34.208 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.349048+0000", 2024-03-20T09:39:34.208 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5709999999999997e-06 2024-03-20T09:39:34.208 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.208 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.208 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:34.209 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.349584+0000", 2024-03-20T09:39:34.209 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000536288 2024-03-20T09:39:34.209 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.209 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.209 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:39:34.209 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.349604+0000", 2024-03-20T09:39:34.209 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9874e-05 2024-03-20T09:39:34.209 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.209 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.209 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:39:34.209 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.413462+0000", 2024-03-20T09:39:34.209 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.063857842999999997 2024-03-20T09:39:34.209 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.209 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.210 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:34.210 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.413487+0000", 2024-03-20T09:39:34.210 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5185e-05 2024-03-20T09:39:34.210 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.210 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:34.210 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.210 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.210 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.210 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:14601 8.d e980/974 8:be1d5a33:::benchmark_data_smithi122_197775_object14600:head v 980'875, mlcod=980'868)", 2024-03-20T09:39:34.210 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:05.361367+0000", 2024-03-20T09:39:34.210 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 208.811952754, 2024-03-20T09:39:34.210 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.052143722000000003, 2024-03-20T09:39:34.210 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:34.210 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:39:34.211 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:34.211 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.211 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:34.211 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.361367+0000", 2024-03-20T09:39:34.211 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.211 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.211 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.211 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:34.211 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.361367+0000", 2024-03-20T09:39:34.211 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.211 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.211 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.211 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:34.211 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.361368+0000", 2024-03-20T09:39:34.211 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.048e-06 2024-03-20T09:39:34.212 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.212 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.212 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:34.212 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.361486+0000", 2024-03-20T09:39:34.212 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00011807399999999999 2024-03-20T09:39:34.212 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.212 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.212 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:34.212 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.361499+0000", 2024-03-20T09:39:34.212 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2991e-05 2024-03-20T09:39:34.212 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.212 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.212 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:34.212 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.361508+0000", 2024-03-20T09:39:34.213 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.6230000000000003e-06 2024-03-20T09:39:34.213 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.213 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.213 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:34.213 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.361565+0000", 2024-03-20T09:39:34.213 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7589e-05 2024-03-20T09:39:34.213 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.213 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.213 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:39:34.213 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.361590+0000", 2024-03-20T09:39:34.213 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.4677000000000001e-05 2024-03-20T09:39:34.213 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.213 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.213 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:39:34.214 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.413489+0000", 2024-03-20T09:39:34.214 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.051898442000000003 2024-03-20T09:39:34.214 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.214 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.214 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:34.214 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.413511+0000", 2024-03-20T09:39:34.214 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2277999999999999e-05 2024-03-20T09:39:34.214 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.214 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:34.214 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.214 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.214 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.214 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "rep_scrubmap(1.15 e491 from shard 5)", 2024-03-20T09:39:34.215 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:05.445441+0000", 2024-03-20T09:39:34.215 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 208.72787884100001, 2024-03-20T09:39:34.215 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.056021108, 2024-03-20T09:39:34.215 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:34.215 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:39:34.215 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:34.215 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.215 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:34.215 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.445441+0000", 2024-03-20T09:39:34.215 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.215 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.215 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.215 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:34.215 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.445441+0000", 2024-03-20T09:39:34.216 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.216 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.216 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.216 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:34.216 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.445442+0000", 2024-03-20T09:39:34.216 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0890000000000001e-06 2024-03-20T09:39:34.216 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.216 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.216 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:34.216 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.445449+0000", 2024-03-20T09:39:34.216 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.6839999999999999e-06 2024-03-20T09:39:34.216 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.216 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.216 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:34.217 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.445461+0000", 2024-03-20T09:39:34.217 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1946999999999999e-05 2024-03-20T09:39:34.217 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.217 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.217 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:34.217 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.445467+0000", 2024-03-20T09:39:34.217 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1539999999999999e-06 2024-03-20T09:39:34.217 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.217 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.217 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:34.217 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.501431+0000", 2024-03-20T09:39:34.217 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.055964106999999999 2024-03-20T09:39:34.217 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.217 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.217 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:39:34.218 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.501445+0000", 2024-03-20T09:39:34.218 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3645000000000001e-05 2024-03-20T09:39:34.218 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.218 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.218 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:34.218 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.501462+0000", 2024-03-20T09:39:34.218 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7482e-05 2024-03-20T09:39:34.218 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.218 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:34.218 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.218 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.218 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.218 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:85294 8.d e998/974 8:bf9f6105:::benchmark_data_smithi122_197775_object85293:head v 998'5362, mlcod=998'5360)", 2024-03-20T09:39:34.218 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:50.030225+0000", 2024-03-20T09:39:34.219 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 164.14309456399999, 2024-03-20T09:39:34.219 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.094034662000000005, 2024-03-20T09:39:34.219 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:34.219 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:39:34.219 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:34.219 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.219 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:34.219 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.030225+0000", 2024-03-20T09:39:34.219 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.219 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.219 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.219 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:34.219 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.030225+0000", 2024-03-20T09:39:34.219 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.220 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.220 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.220 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:34.220 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.030227+0000", 2024-03-20T09:39:34.220 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.33e-06 2024-03-20T09:39:34.220 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.220 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.220 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:34.220 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.030284+0000", 2024-03-20T09:39:34.220 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6991999999999999e-05 2024-03-20T09:39:34.220 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.220 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.220 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:34.220 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.030305+0000", 2024-03-20T09:39:34.220 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1276000000000001e-05 2024-03-20T09:39:34.221 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.221 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.221 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:34.221 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.030322+0000", 2024-03-20T09:39:34.221 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6739e-05 2024-03-20T09:39:34.221 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.221 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.221 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:34.221 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.030394+0000", 2024-03-20T09:39:34.221 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.2361999999999996e-05 2024-03-20T09:39:34.221 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.221 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.221 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:39:34.221 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.030416+0000", 2024-03-20T09:39:34.222 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1920999999999999e-05 2024-03-20T09:39:34.222 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.222 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.222 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:39:34.222 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124230+0000", 2024-03-20T09:39:34.222 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.093813721000000003 2024-03-20T09:39:34.222 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.222 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.222 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:34.222 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124260+0000", 2024-03-20T09:39:34.222 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.0321e-05 2024-03-20T09:39:34.222 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.222 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:34.222 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.223 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.223 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.223 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:85304 8.d e998/974 8:b9b9ee18:::benchmark_data_smithi122_197775_object85303:head v 998'5363, mlcod=998'5360)", 2024-03-20T09:39:34.223 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:50.035062+0000", 2024-03-20T09:39:34.223 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 164.13825778699999, 2024-03-20T09:39:34.223 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.089231913999999996, 2024-03-20T09:39:34.223 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:34.223 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:39:34.223 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:34.223 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.223 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:34.223 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.035062+0000", 2024-03-20T09:39:34.223 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.224 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.224 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.224 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:34.224 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.035062+0000", 2024-03-20T09:39:34.224 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.224 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.224 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.224 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:34.224 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.035063+0000", 2024-03-20T09:39:34.224 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.9999999999999996e-07 2024-03-20T09:39:34.224 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.224 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.224 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:34.224 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.035117+0000", 2024-03-20T09:39:34.225 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.363e-05 2024-03-20T09:39:34.225 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.225 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.225 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:34.225 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.035128+0000", 2024-03-20T09:39:34.225 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1561e-05 2024-03-20T09:39:34.225 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.225 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.225 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:34.225 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.035136+0000", 2024-03-20T09:39:34.225 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.4749999999999996e-06 2024-03-20T09:39:34.225 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.225 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.225 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:34.225 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.035232+0000", 2024-03-20T09:39:34.226 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.6311000000000007e-05 2024-03-20T09:39:34.226 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.226 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.226 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:39:34.226 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.035267+0000", 2024-03-20T09:39:34.226 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.4971000000000001e-05 2024-03-20T09:39:34.226 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.226 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.226 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:39:34.226 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124264+0000", 2024-03-20T09:39:34.226 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.088997007000000003 2024-03-20T09:39:34.226 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.226 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.227 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:34.227 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124294+0000", 2024-03-20T09:39:34.227 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.0059000000000001e-05 2024-03-20T09:39:34.227 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.227 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:34.227 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.227 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.227 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.227 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:85309 8.d e998/974 8:b696b706:::benchmark_data_smithi122_197775_object85308:head v 998'5364, mlcod=998'5360)", 2024-03-20T09:39:34.227 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:50.039124+0000", 2024-03-20T09:39:34.227 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 164.134196288, 2024-03-20T09:39:34.227 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.085202424999999998, 2024-03-20T09:39:34.227 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:34.227 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:39:34.228 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:34.228 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.228 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:34.228 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.039124+0000", 2024-03-20T09:39:34.228 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.228 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.228 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.228 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:34.228 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.039124+0000", 2024-03-20T09:39:34.228 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.228 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.228 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.228 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:34.228 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.039124+0000", 2024-03-20T09:39:34.229 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.6400000000000001e-07 2024-03-20T09:39:34.229 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.229 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.229 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:34.229 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.039228+0000", 2024-03-20T09:39:34.229 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000103757 2024-03-20T09:39:34.229 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.229 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.229 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:34.229 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.039240+0000", 2024-03-20T09:39:34.229 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1652e-05 2024-03-20T09:39:34.229 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.229 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.229 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:34.230 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.039247+0000", 2024-03-20T09:39:34.230 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3359999999999997e-06 2024-03-20T09:39:34.230 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.230 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.230 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:34.230 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.039311+0000", 2024-03-20T09:39:34.230 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.3662000000000001e-05 2024-03-20T09:39:34.230 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.230 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.230 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:39:34.230 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.039335+0000", 2024-03-20T09:39:34.230 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3725000000000001e-05 2024-03-20T09:39:34.230 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.230 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.231 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:39:34.231 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124296+0000", 2024-03-20T09:39:34.231 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.084961693000000005 2024-03-20T09:39:34.231 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.231 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.231 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:34.231 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124326+0000", 2024-03-20T09:39:34.231 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9736000000000001e-05 2024-03-20T09:39:34.231 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.231 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:34.231 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.231 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.231 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.231 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:85325 8.d e998/974 8:bc21c74a:::benchmark_data_smithi122_197775_object85324:head v 998'5365, mlcod=998'5360)", 2024-03-20T09:39:34.232 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:50.052544+0000", 2024-03-20T09:39:34.232 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 164.12077593399999, 2024-03-20T09:39:34.232 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.071814343000000003, 2024-03-20T09:39:34.232 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:34.232 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:39:34.232 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:34.232 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.232 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:34.232 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.052544+0000", 2024-03-20T09:39:34.232 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.232 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.232 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.232 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:34.232 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.052544+0000", 2024-03-20T09:39:34.233 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.233 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.233 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.233 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:34.233 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.052545+0000", 2024-03-20T09:39:34.233 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.415e-06 2024-03-20T09:39:34.233 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.233 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.233 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:34.233 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.052580+0000", 2024-03-20T09:39:34.233 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.4459999999999999e-05 2024-03-20T09:39:34.233 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.233 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.233 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:34.233 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.052600+0000", 2024-03-20T09:39:34.234 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9853000000000001e-05 2024-03-20T09:39:34.234 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.234 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.234 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:34.234 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.052610+0000", 2024-03-20T09:39:34.234 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0655999999999999e-05 2024-03-20T09:39:34.234 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.234 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.234 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:34.234 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.052671+0000", 2024-03-20T09:39:34.234 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0565e-05 2024-03-20T09:39:34.234 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.234 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.235 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:39:34.235 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.052693+0000", 2024-03-20T09:39:34.235 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2308000000000001e-05 2024-03-20T09:39:34.235 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.235 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.235 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:39:34.235 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124329+0000", 2024-03-20T09:39:34.235 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.071635628000000007 2024-03-20T09:39:34.235 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.235 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.235 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:34.235 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124358+0000", 2024-03-20T09:39:34.235 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9458000000000002e-05 2024-03-20T09:39:34.235 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.236 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:34.236 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.236 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.236 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.236 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:85327 8.d e998/974 8:b1150ede:::benchmark_data_smithi122_197775_object85326:head v 998'5366, mlcod=998'5360)", 2024-03-20T09:39:34.236 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:50.055053+0000", 2024-03-20T09:39:34.236 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 164.118266852, 2024-03-20T09:39:34.236 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.069332682000000007, 2024-03-20T09:39:34.236 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:34.236 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:39:34.236 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:34.236 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.236 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:34.236 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.055053+0000", 2024-03-20T09:39:34.236 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.237 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.237 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.237 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:34.237 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.055053+0000", 2024-03-20T09:39:34.237 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.237 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.237 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.237 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:34.237 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.055054+0000", 2024-03-20T09:39:34.237 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2470000000000001e-06 2024-03-20T09:39:34.237 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.237 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.237 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:34.237 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.055097+0000", 2024-03-20T09:39:34.238 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.2617999999999998e-05 2024-03-20T09:39:34.238 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.238 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.238 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:34.238 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.055110+0000", 2024-03-20T09:39:34.238 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3117e-05 2024-03-20T09:39:34.238 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.238 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.238 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:34.238 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.055116+0000", 2024-03-20T09:39:34.238 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4570000000000002e-06 2024-03-20T09:39:34.238 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.238 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.238 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:34.239 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.055321+0000", 2024-03-20T09:39:34.239 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000204231 2024-03-20T09:39:34.239 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.239 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.239 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:39:34.239 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.055340+0000", 2024-03-20T09:39:34.239 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8856999999999999e-05 2024-03-20T09:39:34.239 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.239 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.239 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:39:34.239 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124360+0000", 2024-03-20T09:39:34.239 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.069020916000000002 2024-03-20T09:39:34.239 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.239 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.240 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:34.240 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124386+0000", 2024-03-20T09:39:34.240 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5239e-05 2024-03-20T09:39:34.240 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.240 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:34.240 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.240 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.240 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.240 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:85328 8.d e998/974 8:b7d49900:::benchmark_data_smithi122_197775_object85327:head v 998'5367, mlcod=998'5360)", 2024-03-20T09:39:34.240 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:50.058952+0000", 2024-03-20T09:39:34.240 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 164.114368353, 2024-03-20T09:39:34.240 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.065465018, 2024-03-20T09:39:34.240 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:34.240 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:39:34.241 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:34.241 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.241 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:34.241 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.058952+0000", 2024-03-20T09:39:34.241 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.241 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.241 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.241 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:34.241 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.058952+0000", 2024-03-20T09:39:34.241 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.241 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.241 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.241 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:34.241 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.058953+0000", 2024-03-20T09:39:34.242 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0690000000000001e-06 2024-03-20T09:39:34.242 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.242 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.242 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:34.242 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.058989+0000", 2024-03-20T09:39:34.242 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6922000000000003e-05 2024-03-20T09:39:34.242 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.242 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.242 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:34.242 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.059003+0000", 2024-03-20T09:39:34.242 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3787e-05 2024-03-20T09:39:34.242 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.242 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.242 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:34.243 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.059011+0000", 2024-03-20T09:39:34.243 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.9330000000000001e-06 2024-03-20T09:39:34.243 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.243 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.243 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:34.243 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.059083+0000", 2024-03-20T09:39:34.243 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.2071999999999997e-05 2024-03-20T09:39:34.243 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.243 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.243 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:39:34.243 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.059105+0000", 2024-03-20T09:39:34.243 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1699999999999999e-05 2024-03-20T09:39:34.243 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.243 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.244 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:39:34.244 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124388+0000", 2024-03-20T09:39:34.244 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.065282824000000003 2024-03-20T09:39:34.244 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.244 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.244 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:34.244 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124417+0000", 2024-03-20T09:39:34.244 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8711e-05 2024-03-20T09:39:34.244 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.244 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:34.244 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.244 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.244 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.244 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:85334 8.d e998/974 8:b8549932:::benchmark_data_smithi122_197775_object85333:head v 998'5368, mlcod=998'5360)", 2024-03-20T09:39:34.245 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:50.062866+0000", 2024-03-20T09:39:34.245 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 164.110453608, 2024-03-20T09:39:34.245 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.061575585000000002, 2024-03-20T09:39:34.245 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:39:34.245 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:39:34.245 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:39:34.245 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.245 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:39:34.245 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.062866+0000", 2024-03-20T09:39:34.245 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.245 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.245 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.245 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:39:34.245 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.062866+0000", 2024-03-20T09:39:34.245 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:39:34.246 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.246 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.246 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:39:34.246 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.062867+0000", 2024-03-20T09:39:34.246 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0890000000000001e-06 2024-03-20T09:39:34.246 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.246 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.246 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:39:34.246 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.062972+0000", 2024-03-20T09:39:34.246 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000104344 2024-03-20T09:39:34.246 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.246 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.246 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:39:34.247 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.062987+0000", 2024-03-20T09:39:34.247 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5738000000000001e-05 2024-03-20T09:39:34.247 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.247 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.247 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:39:34.247 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.062997+0000", 2024-03-20T09:39:34.247 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.6129999999999993e-06 2024-03-20T09:39:34.247 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.247 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.247 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:39:34.247 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.063058+0000", 2024-03-20T09:39:34.247 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1370000000000004e-05 2024-03-20T09:39:34.247 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.247 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.247 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:39:34.248 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.063085+0000", 2024-03-20T09:39:34.248 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6676999999999999e-05 2024-03-20T09:39:34.248 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.248 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.248 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:39:34.248 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124419+0000", 2024-03-20T09:39:34.248 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.061333490999999997 2024-03-20T09:39:34.248 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:39:34.248 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:39:34.248 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:39:34.248 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124442+0000", 2024-03-20T09:39:34.248 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3263000000000001e-05 2024-03-20T09:39:34.248 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.248 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:34.249 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.249 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:39:34.249 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:39:34.249 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:39:34.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:34.250 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:39:34.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:39:34.251 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:39:34.352 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:39:34.386 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:34.410 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:39:34.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:34.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:39:34.515 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:39:34.549 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:34.733 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 5, 6, 7, 8, 11, 3, 4, 9, 10] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 11, 5, 10] 2024-03-20T09:39:34.733 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:39:34.733 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-20T09:39:34.733 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.1 'trim stale osdmaps' 2024-03-20T09:39:34.826 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:39:34.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:34.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:39:34.854 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_ops_in_flight 2024-03-20T09:39:34.933 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:39:34.965 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:34.989 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:39:34.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:34.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:39:35.094 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:39:35.120 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:34 smithi115 ceph-mon[72476]: osdmap e1051: 12 total, 12 up, 11 in 2024-03-20T09:39:35.123 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:35.145 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:39:35.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:35.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:39:35.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:34 smithi122 ceph-mon[73495]: osdmap e1051: 12 total, 12 up, 11 in 2024-03-20T09:39:35.197 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:34 smithi026 ceph-mon[68309]: osdmap e1051: 12 total, 12 up, 11 in 2024-03-20T09:39:35.250 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:39:35.306 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:39:35.316 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:35.342 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:39:35.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:35.345 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:39:35.446 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:39:35.481 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:35.614 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:39:35.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:35.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:39:35.649 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:39:35.698 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:39:35.720 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:39:35.796 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:35.903 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:39:35.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:35.914 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:39:36.015 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:39:36.053 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:36.114 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:39:36.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:36.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:39:36.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:35 smithi122 ceph-mon[73495]: pgmap v2540: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 45 active+clean; 1.5 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 683 op/s; 1475/74574 objects misplaced (1.978%); 12 MiB/s, 196 objects/s recovering 2024-03-20T09:39:36.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:35 smithi026 ceph-mon[68309]: pgmap v2540: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 45 active+clean; 1.5 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 683 op/s; 1475/74574 objects misplaced (1.978%); 12 MiB/s, 196 objects/s recovering 2024-03-20T09:39:36.221 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:39:36.249 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:35 smithi115 ceph-mon[72476]: pgmap v2540: 47 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 45 active+clean; 1.5 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 683 op/s; 1475/74574 objects misplaced (1.978%); 12 MiB/s, 196 objects/s recovering 2024-03-20T09:39:36.250 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:36.299 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:39:36.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:36.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:39:36.405 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:39:36.450 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:36.722 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:39:36.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:36.727 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:39:36.828 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:39:36.871 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:36.967 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:39:37.096 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:36 smithi122 ceph-mon[73495]: pgmap v2541: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 549 op/s; 1475/70248 objects misplaced (2.100%); 8.9 MiB/s, 142 objects/s recovering 2024-03-20T09:39:37.096 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:36 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/131618999' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:39:37.096 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:36 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:39:37.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:36 smithi026 ceph-mon[68309]: pgmap v2541: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 549 op/s; 1475/70248 objects misplaced (2.100%); 8.9 MiB/s, 142 objects/s recovering 2024-03-20T09:39:37.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:36 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/131618999' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:39:37.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:36 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:39:37.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:36 smithi115 ceph-mon[72476]: pgmap v2541: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 549 op/s; 1475/70248 objects misplaced (2.100%); 8.9 MiB/s, 142 objects/s recovering 2024-03-20T09:39:37.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:36 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/131618999' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:39:37.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:36 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:39:37.526 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:39:37.526 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-03-20T09:39:37.526 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:39:37.527 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:225499 8.b 8:ddf8d12f:::benchmark_data_smithi122_197775_object102195:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1052)", 2024-03-20T09:39:37.527 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:39:37.532059+0000", 2024-03-20T09:39:37.527 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.0056915739999999996, 2024-03-20T09:39:37.527 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.005719493, 2024-03-20T09:39:37.527 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:39:37.527 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:39:37.527 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:39:37.527 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:39:37.527 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:39:37.527 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 225499 2024-03-20T09:39:37.527 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:39:37.527 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:39:37.527 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:39:37.527 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:39:37.527 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:37.532059+0000", 2024-03-20T09:39:37.528 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:39:37.528 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:39:37.528 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:39:37.528 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:39:37.528 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:37.532059+0000", 2024-03-20T09:39:37.528 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:39:37.528 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:39:37.528 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:39:37.528 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:39:37.528 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:37.532060+0000", 2024-03-20T09:39:37.528 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.358e-06 2024-03-20T09:39:37.528 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:39:37.528 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:39:37.528 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:39:37.528 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:37.532065+0000", 2024-03-20T09:39:37.529 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5510000000000001e-06 2024-03-20T09:39:37.529 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:39:37.529 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:39:37.529 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:39:37.529 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:37.532076+0000", 2024-03-20T09:39:37.529 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1415e-05 2024-03-20T09:39:37.529 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:39:37.529 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:39:37.529 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:39:37.529 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:37.532084+0000", 2024-03-20T09:39:37.529 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5689999999999999e-06 2024-03-20T09:39:37.529 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:39:37.529 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:39:37.529 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:39:37.530 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:37.532189+0000", 2024-03-20T09:39:37.530 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104559 2024-03-20T09:39:37.530 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:39:37.530 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:39:37.530 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:39:37.530 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:37.532272+0000", 2024-03-20T09:39:37.530 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3071000000000005e-05 2024-03-20T09:39:37.530 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:39:37.530 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:39:37.530 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,4", 2024-03-20T09:39:37.530 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:37.532380+0000", 2024-03-20T09:39:37.530 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010837599999999999 2024-03-20T09:39:37.530 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:39:37.530 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:39:37.531 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:39:37.531 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:39:37.531 INFO:teuthology.orchestra.run.smithi122.stdout: ], 2024-03-20T09:39:37.531 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 1 2024-03-20T09:39:37.531 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:39:37.751 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:39:37.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:37.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:39:37.859 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:39:37.896 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:37.919 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:37 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:39:37.919 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:37 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:39:37.919 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:37 smithi122 ceph-mon[73495]: osdmap e1052: 12 total, 12 up, 11 in 2024-03-20T09:39:37.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:37 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:39:37.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:37 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:39:37.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:37 smithi026 ceph-mon[68309]: osdmap e1052: 12 total, 12 up, 11 in 2024-03-20T09:39:38.111 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:39:38.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:38.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:39:38.176 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:39:38.217 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:39:38.266 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:38.311 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_blocked_ops 2024-03-20T09:39:38.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:37 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:39:38.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:37 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:39:38.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:37 smithi115 ceph-mon[72476]: osdmap e1052: 12 total, 12 up, 11 in 2024-03-20T09:39:38.403 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:39:38.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:38.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:39:38.513 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:39:38.542 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:38.564 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:39:38.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:38.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:39:38.668 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:39:38.695 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:38.716 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:39:38.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:38.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:39:38.769 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:39:38.825 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:39:38.883 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:38.948 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:39:38.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:38.959 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:39:39.060 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:39:39.119 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:39.148 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:39:39.150 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:38 smithi122 ceph-mon[73495]: pgmap v2543: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 615 op/s; 1226/67026 objects misplaced (1.829%); 8.2 MiB/s, 131 objects/s recovering 2024-03-20T09:39:39.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:39.151 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:39:39.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:38 smithi026 ceph-mon[68309]: pgmap v2543: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 615 op/s; 1226/67026 objects misplaced (1.829%); 8.2 MiB/s, 131 objects/s recovering 2024-03-20T09:39:39.253 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:39:39.295 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:39.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:38 smithi115 ceph-mon[72476]: pgmap v2543: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 615 op/s; 1226/67026 objects misplaced (1.829%); 8.2 MiB/s, 131 objects/s recovering 2024-03-20T09:39:39.833 INFO:teuthology.orchestra.run.smithi026.stderr:no valid command found; 10 closest matches: 2024-03-20T09:39:39.833 INFO:teuthology.orchestra.run.smithi026.stderr:0 2024-03-20T09:39:39.833 INFO:teuthology.orchestra.run.smithi026.stderr:1 2024-03-20T09:39:39.833 INFO:teuthology.orchestra.run.smithi026.stderr:2 2024-03-20T09:39:39.833 INFO:teuthology.orchestra.run.smithi026.stderr:abort 2024-03-20T09:39:39.833 INFO:teuthology.orchestra.run.smithi026.stderr:assert 2024-03-20T09:39:39.833 INFO:teuthology.orchestra.run.smithi026.stderr:bench [] [] [] [] 2024-03-20T09:39:39.833 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs debug_inject_read_zeros 2024-03-20T09:39:39.833 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs files list 2024-03-20T09:39:39.833 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs stats 2024-03-20T09:39:39.833 INFO:teuthology.orchestra.run.smithi026.stderr:bluestore allocator dump block 2024-03-20T09:39:39.834 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: invalid command 2024-03-20T09:39:40.112 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:39:40.542 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T09:39:40.543 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-20T09:39:40.543 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.2 'trim stale osdmaps' 2024-03-20T09:39:40.572 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:39:40.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:40.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:39:40.684 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:39:40.743 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:40.822 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:39:40.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:40.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:39:40.933 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:39:40.983 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:41.017 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:39:41.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:41.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:39:41.122 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:39:41.150 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:39:41.197 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:41.214 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:39:41.236 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:39:41.261 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:39:41.289 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:39:41.323 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:39:41.349 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:39:41.370 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:39:41.390 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:39:41.418 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:39:41.457 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:39:41.512 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:39:41.553 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:39:41.580 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:39:41.581 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:39:41.582 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:39:41.582 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:39:41.582 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:39:41.582 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:39:41.582 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:39:41.582 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:39:41.582 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:39:41.582 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:39:41.582 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:39:41.582 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:39:41.582 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:39:41.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:41 smithi026 ceph-mon[68309]: pgmap v2544: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.3 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 672 op/s; 1226/62517 objects misplaced (1.961%); 6.4 MiB/s, 101 objects/s recovering 2024-03-20T09:39:41.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:41 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3164080956' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:39:41.935 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:41 smithi122 ceph-mon[73495]: pgmap v2544: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.3 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 672 op/s; 1226/62517 objects misplaced (1.961%); 6.4 MiB/s, 101 objects/s recovering 2024-03-20T09:39:41.935 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:41 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3164080956' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:39:42.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:41 smithi115 ceph-mon[72476]: pgmap v2544: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.3 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 672 op/s; 1226/62517 objects misplaced (1.961%); 6.4 MiB/s, 101 objects/s recovering 2024-03-20T09:39:42.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:41 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3164080956' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:39:42.242 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:39:42.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:42.246 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:39:42.347 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:39:42.374 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:42.397 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:39:42.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:42.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:39:42.504 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:39:42.535 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:42.560 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:39:42.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:42.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:39:42.664 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:39:42.724 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:42.792 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:39:42.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:42.796 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:39:42.897 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:39:42.970 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:43.018 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:39:43.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:43.019 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:39:43.035 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:39:43.036 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:39:43.120 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:39:43.147 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:43.167 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:39:43.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:43.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:39:43.271 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:39:43.304 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:43.323 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:39:43.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:43.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:39:43.428 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:39:43.480 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:43.530 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:39:43.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:43.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:39:43.635 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:39:43.685 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:43.765 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:43 smithi026 ceph-mon[68309]: pgmap v2545: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.2 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 720 op/s; 1226/57318 objects misplaced (2.139%); 5.6 MiB/s, 90 objects/s recovering 2024-03-20T09:39:43.785 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_historic_ops 2024-03-20T09:39:44.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:43 smithi115 ceph-mon[72476]: pgmap v2545: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.2 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 720 op/s; 1226/57318 objects misplaced (2.139%); 5.6 MiB/s, 90 objects/s recovering 2024-03-20T09:39:44.196 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:43 smithi122 ceph-mon[73495]: pgmap v2545: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.2 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 720 op/s; 1226/57318 objects misplaced (2.139%); 5.6 MiB/s, 90 objects/s recovering 2024-03-20T09:39:44.230 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:39:44.654 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:39:44.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:44.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:39:44.760 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:39:44.788 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:45.519 INFO:teuthology.orchestra.run.smithi026.stderr:no valid command found; 10 closest matches: 2024-03-20T09:39:45.519 INFO:teuthology.orchestra.run.smithi026.stderr:0 2024-03-20T09:39:45.519 INFO:teuthology.orchestra.run.smithi026.stderr:1 2024-03-20T09:39:45.519 INFO:teuthology.orchestra.run.smithi026.stderr:2 2024-03-20T09:39:45.519 INFO:teuthology.orchestra.run.smithi026.stderr:abort 2024-03-20T09:39:45.519 INFO:teuthology.orchestra.run.smithi026.stderr:assert 2024-03-20T09:39:45.520 INFO:teuthology.orchestra.run.smithi026.stderr:bench [] [] [] [] 2024-03-20T09:39:45.520 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs debug_inject_read_zeros 2024-03-20T09:39:45.520 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs files list 2024-03-20T09:39:45.520 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs stats 2024-03-20T09:39:45.520 INFO:teuthology.orchestra.run.smithi026.stderr:bluestore allocator dump block 2024-03-20T09:39:45.520 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: invalid command 2024-03-20T09:39:45.705 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:39:45.739 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:39:45.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:45.748 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:39:45.849 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:39:45.880 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:45.901 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:39:45.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:45.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:39:45.935 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:45 smithi122 ceph-mon[73495]: pgmap v2546: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 722 op/s; 983/52887 objects misplaced (1.859%) 2024-03-20T09:39:45.935 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:45 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3269933685' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:39:45.935 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:45 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:39:45.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:45 smithi026 ceph-mon[68309]: pgmap v2546: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 722 op/s; 983/52887 objects misplaced (1.859%) 2024-03-20T09:39:45.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:45 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3269933685' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:39:45.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:45 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:39:46.004 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:39:46.029 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:45 smithi115 ceph-mon[72476]: pgmap v2546: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 1.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 722 op/s; 983/52887 objects misplaced (1.859%) 2024-03-20T09:39:46.029 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:45 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3269933685' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:39:46.029 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:45 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:39:46.031 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:46.053 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:39:46.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:46.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:39:46.157 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:39:46.188 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:46.214 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:39:46.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:46.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:39:46.318 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:39:46.370 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:46.451 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:39:46.451 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:39:46.466 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:39:46.496 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:39:46.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:46.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:39:46.608 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:39:46.677 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:46.827 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:46 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:39:46.827 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:46 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:39:46.827 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:46 smithi122 ceph-mon[73495]: osdmap e1053: 12 total, 12 up, 11 in 2024-03-20T09:39:46.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:46 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:39:46.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:46 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:39:46.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:46 smithi026 ceph-mon[68309]: osdmap e1053: 12 total, 12 up, 11 in 2024-03-20T09:39:47.012 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T09:39:47.012 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-20T09:39:47.012 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.5 'trim stale osdmaps' 2024-03-20T09:39:47.055 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:39:47.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:47.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:39:47.106 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:39:47.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:46 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:39:47.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:46 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:39:47.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:46 smithi115 ceph-mon[72476]: osdmap e1053: 12 total, 12 up, 11 in 2024-03-20T09:39:47.161 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:39:47.194 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_ops_in_flight 2024-03-20T09:39:47.217 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:47.436 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:39:47.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:47.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:39:47.541 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:39:47.583 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:47.606 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:39:47.616 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:39:47.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:47.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:39:47.721 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:39:47.759 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:47.781 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:39:47.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:47.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:39:47.886 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:39:47.914 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:47.938 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:39:47.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:47.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:39:47.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:47 smithi026 ceph-mon[68309]: 1.0 scrub starts 2024-03-20T09:39:47.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:47 smithi026 ceph-mon[68309]: 1.0 scrub ok 2024-03-20T09:39:47.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:47 smithi026 ceph-mon[68309]: pgmap v2548: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 984 MiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 753 op/s; 983/47226 objects misplaced (2.081%) 2024-03-20T09:39:47.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:47 smithi026 ceph-mon[68309]: 1.1b scrub starts 2024-03-20T09:39:47.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:47 smithi026 ceph-mon[68309]: 1.6 scrub starts 2024-03-20T09:39:47.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:47 smithi026 ceph-mon[68309]: 1.6 scrub ok 2024-03-20T09:39:48.043 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:39:48.085 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:48.097 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:47 smithi122 ceph-mon[73495]: 1.0 scrub starts 2024-03-20T09:39:48.097 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:47 smithi122 ceph-mon[73495]: 1.0 scrub ok 2024-03-20T09:39:48.097 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:47 smithi122 ceph-mon[73495]: pgmap v2548: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 984 MiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 753 op/s; 983/47226 objects misplaced (2.081%) 2024-03-20T09:39:48.097 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:47 smithi122 ceph-mon[73495]: 1.1b scrub starts 2024-03-20T09:39:48.097 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:47 smithi122 ceph-mon[73495]: 1.6 scrub starts 2024-03-20T09:39:48.097 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:47 smithi122 ceph-mon[73495]: 1.6 scrub ok 2024-03-20T09:39:48.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:47 smithi115 ceph-mon[72476]: 1.0 scrub starts 2024-03-20T09:39:48.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:47 smithi115 ceph-mon[72476]: 1.0 scrub ok 2024-03-20T09:39:48.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:47 smithi115 ceph-mon[72476]: pgmap v2548: 47 pgs: 1 active+remapped+backfill_toofull, 46 active+clean; 984 MiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 753 op/s; 983/47226 objects misplaced (2.081%) 2024-03-20T09:39:48.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:47 smithi115 ceph-mon[72476]: 1.1b scrub starts 2024-03-20T09:39:48.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:47 smithi115 ceph-mon[72476]: 1.6 scrub starts 2024-03-20T09:39:48.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:47 smithi115 ceph-mon[72476]: 1.6 scrub ok 2024-03-20T09:39:48.195 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:39:48.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:48.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:39:48.245 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:39:48.306 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:39:48.366 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:48.549 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:39:48.897 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:48 smithi026 ceph-mon[68309]: 1.1b scrub ok 2024-03-20T09:39:48.897 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:48 smithi026 ceph-mon[68309]: 1.b scrub starts 2024-03-20T09:39:48.897 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:48 smithi026 ceph-mon[68309]: 1.b scrub ok 2024-03-20T09:39:48.897 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:48 smithi026 ceph-mon[68309]: 1.1a scrub starts 2024-03-20T09:39:48.897 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:48 smithi026 ceph-mon[68309]: 1.1a scrub ok 2024-03-20T09:39:48.897 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:48 smithi026 ceph-mon[68309]: pgmap v2549: 47 pgs: 1 active+clean+remapped, 46 active+clean; 910 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 777 op/s; 5.0 MiB/s, 79 objects/s recovering 2024-03-20T09:39:48.901 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:48 smithi122 ceph-mon[73495]: 1.1b scrub ok 2024-03-20T09:39:48.902 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:48 smithi122 ceph-mon[73495]: 1.b scrub starts 2024-03-20T09:39:48.902 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:48 smithi122 ceph-mon[73495]: 1.b scrub ok 2024-03-20T09:39:48.902 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:48 smithi122 ceph-mon[73495]: 1.1a scrub starts 2024-03-20T09:39:48.902 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:48 smithi122 ceph-mon[73495]: 1.1a scrub ok 2024-03-20T09:39:48.902 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:48 smithi122 ceph-mon[73495]: pgmap v2549: 47 pgs: 1 active+clean+remapped, 46 active+clean; 910 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 777 op/s; 5.0 MiB/s, 79 objects/s recovering 2024-03-20T09:39:49.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:48 smithi115 ceph-mon[72476]: 1.1b scrub ok 2024-03-20T09:39:49.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:48 smithi115 ceph-mon[72476]: 1.b scrub starts 2024-03-20T09:39:49.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:48 smithi115 ceph-mon[72476]: 1.b scrub ok 2024-03-20T09:39:49.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:48 smithi115 ceph-mon[72476]: 1.1a scrub starts 2024-03-20T09:39:49.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:48 smithi115 ceph-mon[72476]: 1.1a scrub ok 2024-03-20T09:39:49.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:48 smithi115 ceph-mon[72476]: pgmap v2549: 47 pgs: 1 active+clean+remapped, 46 active+clean; 910 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 777 op/s; 5.0 MiB/s, 79 objects/s recovering 2024-03-20T09:39:50.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:49 smithi122 ceph-mon[73495]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-20T09:39:50.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:49 smithi122 ceph-mon[73495]: osdmap e1054: 12 total, 12 up, 11 in 2024-03-20T09:39:50.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:49 smithi122 ceph-mon[73495]: 1.2 scrub starts 2024-03-20T09:39:50.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:49 smithi122 ceph-mon[73495]: 1.2 scrub ok 2024-03-20T09:39:50.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:49 smithi026 ceph-mon[68309]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-20T09:39:50.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:49 smithi026 ceph-mon[68309]: osdmap e1054: 12 total, 12 up, 11 in 2024-03-20T09:39:50.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:49 smithi026 ceph-mon[68309]: 1.2 scrub starts 2024-03-20T09:39:50.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:49 smithi026 ceph-mon[68309]: 1.2 scrub ok 2024-03-20T09:39:50.332 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:39:50.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:50.340 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:39:50.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:49 smithi115 ceph-mon[72476]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-20T09:39:50.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:49 smithi115 ceph-mon[72476]: osdmap e1054: 12 total, 12 up, 11 in 2024-03-20T09:39:50.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:49 smithi115 ceph-mon[72476]: 1.2 scrub starts 2024-03-20T09:39:50.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:49 smithi115 ceph-mon[72476]: 1.2 scrub ok 2024-03-20T09:39:50.440 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:39:50.482 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:50.915 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:50 smithi026 ceph-mon[68309]: 1.8 scrub starts 2024-03-20T09:39:50.916 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:50 smithi026 ceph-mon[68309]: 1.8 scrub ok 2024-03-20T09:39:50.916 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:50 smithi026 ceph-mon[68309]: 1.18 scrub starts 2024-03-20T09:39:50.916 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:50 smithi026 ceph-mon[68309]: 1.18 scrub ok 2024-03-20T09:39:50.916 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:50 smithi026 ceph-mon[68309]: osdmap e1055: 12 total, 12 up, 11 in 2024-03-20T09:39:50.916 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:50 smithi026 ceph-mon[68309]: pgmap v2552: 47 pgs: 1 active+clean+remapped, 46 active+clean; 824 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 740 op/s; 8.3 MiB/s, 133 objects/s recovering 2024-03-20T09:39:51.134 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:50 smithi122 ceph-mon[73495]: 1.8 scrub starts 2024-03-20T09:39:51.134 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:50 smithi122 ceph-mon[73495]: 1.8 scrub ok 2024-03-20T09:39:51.134 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:50 smithi122 ceph-mon[73495]: 1.18 scrub starts 2024-03-20T09:39:51.134 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:50 smithi122 ceph-mon[73495]: 1.18 scrub ok 2024-03-20T09:39:51.134 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:50 smithi122 ceph-mon[73495]: osdmap e1055: 12 total, 12 up, 11 in 2024-03-20T09:39:51.135 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:50 smithi122 ceph-mon[73495]: pgmap v2552: 47 pgs: 1 active+clean+remapped, 46 active+clean; 824 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 740 op/s; 8.3 MiB/s, 133 objects/s recovering 2024-03-20T09:39:51.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:50 smithi115 ceph-mon[72476]: 1.8 scrub starts 2024-03-20T09:39:51.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:50 smithi115 ceph-mon[72476]: 1.8 scrub ok 2024-03-20T09:39:51.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:50 smithi115 ceph-mon[72476]: 1.18 scrub starts 2024-03-20T09:39:51.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:50 smithi115 ceph-mon[72476]: 1.18 scrub ok 2024-03-20T09:39:51.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:50 smithi115 ceph-mon[72476]: osdmap e1055: 12 total, 12 up, 11 in 2024-03-20T09:39:51.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:50 smithi115 ceph-mon[72476]: pgmap v2552: 47 pgs: 1 active+clean+remapped, 46 active+clean; 824 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 740 op/s; 8.3 MiB/s, 133 objects/s recovering 2024-03-20T09:39:51.533 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:39:51.533 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:39:51.918 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:51 smithi122 ceph-mon[73495]: 1.19 scrub starts 2024-03-20T09:39:51.919 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:51 smithi122 ceph-mon[73495]: 1.19 scrub ok 2024-03-20T09:39:52.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:51 smithi026 ceph-mon[68309]: 1.19 scrub starts 2024-03-20T09:39:52.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:51 smithi026 ceph-mon[68309]: 1.19 scrub ok 2024-03-20T09:39:52.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:51 smithi115 ceph-mon[72476]: 1.19 scrub starts 2024-03-20T09:39:52.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:51 smithi115 ceph-mon[72476]: 1.19 scrub ok 2024-03-20T09:39:52.879 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_blocked_ops 2024-03-20T09:39:53.147 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:39:53.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:53.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:39:53.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:52 smithi122 ceph-mon[73495]: pgmap v2553: 47 pgs: 1 active+clean+remapped, 46 active+clean; 738 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 739 op/s; 7.2 MiB/s, 115 objects/s recovering 2024-03-20T09:39:53.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:52 smithi026 ceph-mon[68309]: pgmap v2553: 47 pgs: 1 active+clean+remapped, 46 active+clean; 738 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 739 op/s; 7.2 MiB/s, 115 objects/s recovering 2024-03-20T09:39:53.252 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:39:53.293 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:53.313 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:39:53.334 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:39:53.366 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:39:53.393 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:39:53.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:52 smithi115 ceph-mon[72476]: pgmap v2553: 47 pgs: 1 active+clean+remapped, 46 active+clean; 738 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 739 op/s; 7.2 MiB/s, 115 objects/s recovering 2024-03-20T09:39:53.420 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:39:53.449 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:39:53.478 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:39:53.508 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:39:53.509 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:39:53.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:53.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:39:53.541 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:39:53.569 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:39:53.604 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:39:53.614 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:39:53.635 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:39:53.673 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:39:53.679 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:39:53.679 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:39:53.679 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:39:53.679 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:39:53.679 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:39:53.679 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:39:53.679 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:39:53.679 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:39:53.679 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:39:53.679 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:39:53.679 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:39:53.679 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:39:53.681 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:53.741 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:39:53.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:53.748 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:39:53.849 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:39:53.876 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:53.896 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:39:53.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:53.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:39:53.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:53 smithi122 ceph-mon[73495]: 8.e scrub starts 2024-03-20T09:39:53.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:53 smithi122 ceph-mon[73495]: 8.e scrub ok 2024-03-20T09:39:53.936 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:53 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1720561950' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:39:53.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:53 smithi026 ceph-mon[68309]: 8.e scrub starts 2024-03-20T09:39:53.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:53 smithi026 ceph-mon[68309]: 8.e scrub ok 2024-03-20T09:39:53.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:53 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1720561950' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:39:54.002 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:39:54.038 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:54.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:53 smithi115 ceph-mon[72476]: 8.e scrub starts 2024-03-20T09:39:54.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:53 smithi115 ceph-mon[72476]: 8.e scrub ok 2024-03-20T09:39:54.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:53 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1720561950' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:39:54.954 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:54 smithi026 ceph-mon[68309]: pgmap v2554: 47 pgs: 47 active+clean; 661 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 645 op/s; 6.2 MiB/s, 99 objects/s recovering 2024-03-20T09:39:55.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:54 smithi122 ceph-mon[73495]: pgmap v2554: 47 pgs: 47 active+clean; 661 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 645 op/s; 6.2 MiB/s, 99 objects/s recovering 2024-03-20T09:39:55.324 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:39:55.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:55.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:39:55.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:54 smithi115 ceph-mon[72476]: pgmap v2554: 47 pgs: 47 active+clean; 661 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 645 op/s; 6.2 MiB/s, 99 objects/s recovering 2024-03-20T09:39:55.433 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:39:55.461 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:55.482 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:39:55.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:55.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:39:55.586 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:39:55.602 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:39:55.643 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:55.709 INFO:teuthology.orchestra.run.smithi026.stderr:no valid command found; 10 closest matches: 2024-03-20T09:39:55.709 INFO:teuthology.orchestra.run.smithi026.stderr:0 2024-03-20T09:39:55.709 INFO:teuthology.orchestra.run.smithi026.stderr:1 2024-03-20T09:39:55.709 INFO:teuthology.orchestra.run.smithi026.stderr:2 2024-03-20T09:39:55.709 INFO:teuthology.orchestra.run.smithi026.stderr:abort 2024-03-20T09:39:55.709 INFO:teuthology.orchestra.run.smithi026.stderr:assert 2024-03-20T09:39:55.709 INFO:teuthology.orchestra.run.smithi026.stderr:bench [] [] [] [] 2024-03-20T09:39:55.709 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs debug_inject_read_zeros 2024-03-20T09:39:55.709 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs files list 2024-03-20T09:39:55.709 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs stats 2024-03-20T09:39:55.710 INFO:teuthology.orchestra.run.smithi026.stderr:bluestore allocator dump block 2024-03-20T09:39:55.710 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: invalid command 2024-03-20T09:39:56.003 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:39:56.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:56.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:39:56.095 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:39:56.095 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:39:56.095 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-03-20T09:39:56.095 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-03-20T09:39:56.096 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:39:56.109 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:39:56.173 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:56.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:55 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:39:56.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:55 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:39:56.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:55 smithi026 ceph-mon[68309]: 8.9 scrub starts 2024-03-20T09:39:56.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:55 smithi026 ceph-mon[68309]: osdmap e1056: 12 total, 12 up, 11 in 2024-03-20T09:39:56.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:55 smithi026 ceph-mon[68309]: 8.9 scrub ok 2024-03-20T09:39:56.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:55 smithi026 ceph-mon[68309]: 1.1e scrub starts 2024-03-20T09:39:56.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:55 smithi026 ceph-mon[68309]: 1.1e scrub ok 2024-03-20T09:39:56.224 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:39:56.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:56.233 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:39:56.237 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:55 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:39:56.237 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:55 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:39:56.237 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:55 smithi122 ceph-mon[73495]: 8.9 scrub starts 2024-03-20T09:39:56.237 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:55 smithi122 ceph-mon[73495]: osdmap e1056: 12 total, 12 up, 11 in 2024-03-20T09:39:56.237 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:55 smithi122 ceph-mon[73495]: 8.9 scrub ok 2024-03-20T09:39:56.237 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:55 smithi122 ceph-mon[73495]: 1.1e scrub starts 2024-03-20T09:39:56.237 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:55 smithi122 ceph-mon[73495]: 1.1e scrub ok 2024-03-20T09:39:56.334 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:39:56.372 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:56.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:55 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:39:56.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:55 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:39:56.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:55 smithi115 ceph-mon[72476]: 8.9 scrub starts 2024-03-20T09:39:56.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:55 smithi115 ceph-mon[72476]: osdmap e1056: 12 total, 12 up, 11 in 2024-03-20T09:39:56.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:55 smithi115 ceph-mon[72476]: 8.9 scrub ok 2024-03-20T09:39:56.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:55 smithi115 ceph-mon[72476]: 1.1e scrub starts 2024-03-20T09:39:56.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:55 smithi115 ceph-mon[72476]: 1.1e scrub ok 2024-03-20T09:39:56.645 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T09:39:56.646 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-20T09:39:56.646 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.6 'trim stale osdmaps' 2024-03-20T09:39:57.032 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:39:57.244 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:56 smithi026 ceph-mon[68309]: 1.12 scrub starts 2024-03-20T09:39:57.244 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:56 smithi026 ceph-mon[68309]: 1.12 scrub ok 2024-03-20T09:39:57.244 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:56 smithi026 ceph-mon[68309]: pgmap v2556: 47 pgs: 47 active+clean; 544 MiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 757 op/s 2024-03-20T09:39:57.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:56 smithi115 ceph-mon[72476]: 1.12 scrub starts 2024-03-20T09:39:57.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:56 smithi115 ceph-mon[72476]: 1.12 scrub ok 2024-03-20T09:39:57.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:56 smithi115 ceph-mon[72476]: pgmap v2556: 47 pgs: 47 active+clean; 544 MiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 757 op/s 2024-03-20T09:39:57.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:56 smithi122 ceph-mon[73495]: 1.12 scrub starts 2024-03-20T09:39:57.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:56 smithi122 ceph-mon[73495]: 1.12 scrub ok 2024-03-20T09:39:57.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:56 smithi122 ceph-mon[73495]: pgmap v2556: 47 pgs: 47 active+clean; 544 MiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 757 op/s 2024-03-20T09:39:57.519 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:39:57.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:57.528 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:39:57.554 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_historic_ops 2024-03-20T09:39:57.629 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:39:57.647 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:39:57.687 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:57.691 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:39:57.718 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:39:57.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:57.723 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:39:57.823 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:39:57.872 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:57.930 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:39:57.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:57.945 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:39:58.046 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:39:58.050 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:39:58.088 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:58.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:57 smithi122 ceph-mon[73495]: 1.10 scrub starts 2024-03-20T09:39:58.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:57 smithi122 ceph-mon[73495]: 1.10 scrub ok 2024-03-20T09:39:58.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:57 smithi122 ceph-mon[73495]: 1.11 scrub starts 2024-03-20T09:39:58.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:57 smithi122 ceph-mon[73495]: 1.11 scrub ok 2024-03-20T09:39:58.197 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:39:58.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:58.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:39:58.241 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:57 smithi026 ceph-mon[68309]: 1.10 scrub starts 2024-03-20T09:39:58.242 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:57 smithi026 ceph-mon[68309]: 1.10 scrub ok 2024-03-20T09:39:58.242 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:57 smithi026 ceph-mon[68309]: 1.11 scrub starts 2024-03-20T09:39:58.242 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:57 smithi026 ceph-mon[68309]: 1.11 scrub ok 2024-03-20T09:39:58.302 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:39:58.331 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:57 smithi115 ceph-mon[72476]: 1.10 scrub starts 2024-03-20T09:39:58.331 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:57 smithi115 ceph-mon[72476]: 1.10 scrub ok 2024-03-20T09:39:58.331 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:57 smithi115 ceph-mon[72476]: 1.11 scrub starts 2024-03-20T09:39:58.331 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:57 smithi115 ceph-mon[72476]: 1.11 scrub ok 2024-03-20T09:39:58.333 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:58.363 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:39:58.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:58.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:39:58.469 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:39:58.523 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:58.570 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:39:58.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:58.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:39:58.675 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:39:58.706 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:58.729 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:39:58.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:58.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:39:58.835 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:39:58.864 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:58.886 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:39:58.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:58.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:39:58.975 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:39:58.989 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:39:59.026 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:39:59.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:58 smithi122 ceph-mon[73495]: 1.5 scrub starts 2024-03-20T09:39:59.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:58 smithi122 ceph-mon[73495]: 1.5 scrub ok 2024-03-20T09:39:59.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:58 smithi122 ceph-mon[73495]: 8.0 scrub starts 2024-03-20T09:39:59.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:58 smithi122 ceph-mon[73495]: 1.a scrub starts 2024-03-20T09:39:59.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:58 smithi122 ceph-mon[73495]: 8.0 scrub ok 2024-03-20T09:39:59.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:58 smithi122 ceph-mon[73495]: 1.a scrub ok 2024-03-20T09:39:59.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:58 smithi122 ceph-mon[73495]: pgmap v2557: 47 pgs: 47 active+clean; 470 MiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 724 op/s 2024-03-20T09:39:59.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:58 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3952918873' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:39:59.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:58 smithi115 ceph-mon[72476]: 1.5 scrub starts 2024-03-20T09:39:59.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:58 smithi115 ceph-mon[72476]: 1.5 scrub ok 2024-03-20T09:39:59.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:58 smithi115 ceph-mon[72476]: 8.0 scrub starts 2024-03-20T09:39:59.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:58 smithi115 ceph-mon[72476]: 1.a scrub starts 2024-03-20T09:39:59.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:58 smithi115 ceph-mon[72476]: 8.0 scrub ok 2024-03-20T09:39:59.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:58 smithi115 ceph-mon[72476]: 1.a scrub ok 2024-03-20T09:39:59.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:58 smithi115 ceph-mon[72476]: pgmap v2557: 47 pgs: 47 active+clean; 470 MiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 724 op/s 2024-03-20T09:39:59.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:58 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3952918873' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:39:59.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:58 smithi026 ceph-mon[68309]: 1.5 scrub starts 2024-03-20T09:39:59.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:58 smithi026 ceph-mon[68309]: 1.5 scrub ok 2024-03-20T09:39:59.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:58 smithi026 ceph-mon[68309]: 8.0 scrub starts 2024-03-20T09:39:59.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:58 smithi026 ceph-mon[68309]: 1.a scrub starts 2024-03-20T09:39:59.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:58 smithi026 ceph-mon[68309]: 8.0 scrub ok 2024-03-20T09:39:59.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:58 smithi026 ceph-mon[68309]: 1.a scrub ok 2024-03-20T09:39:59.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:58 smithi026 ceph-mon[68309]: pgmap v2557: 47 pgs: 47 active+clean; 470 MiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 724 op/s 2024-03-20T09:39:59.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:58 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3952918873' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:39:59.789 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:39:59.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:39:59.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:39:59.894 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:39:59.940 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:00.092 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:59 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:40:00.092 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:59 smithi026 ceph-mon[68309]: Cluster is now healthy 2024-03-20T09:40:00.092 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:59 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3952918873' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:40:00.092 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:39:59 smithi026 ceph-mon[68309]: osdmap e1057: 12 total, 12 up, 11 in 2024-03-20T09:40:00.106 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:40:00.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:00.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:40:00.214 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:40:00.238 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:59 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:40:00.238 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:59 smithi115 ceph-mon[72476]: Cluster is now healthy 2024-03-20T09:40:00.238 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:59 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3952918873' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:40:00.238 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:39:59 smithi115 ceph-mon[72476]: osdmap e1057: 12 total, 12 up, 11 in 2024-03-20T09:40:00.241 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:00.265 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:40:00.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:00.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:40:00.370 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:40:00.398 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:00.414 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:59 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:40:00.414 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:59 smithi122 ceph-mon[73495]: Cluster is now healthy 2024-03-20T09:40:00.414 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:59 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3952918873' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:40:00.414 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:39:59 smithi122 ceph-mon[73495]: osdmap e1057: 12 total, 12 up, 11 in 2024-03-20T09:40:00.424 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:40:00.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:00.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:40:00.528 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:40:00.557 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:00.580 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:40:00.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:00.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:40:00.688 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:40:00.746 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:00.786 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:40:00.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:00.795 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:40:00.896 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:40:00.947 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:01.119 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:40:01.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:01.129 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:40:01.206 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:40:01.206 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-03-20T09:40:01.206 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-03-20T09:40:01.206 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-03-20T09:40:01.207 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.207 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26605.0:70582 7.a 7:5a44d4a8:::benchmark_data_smithi122_187077_object70581:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e889)", 2024-03-20T09:40:01.207 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:31:54.334821+0000", 2024-03-20T09:40:01.207 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 486.87775587499999, 2024-03-20T09:40:01.207 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.615454053, 2024-03-20T09:40:01.207 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:40:01.207 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:40:01.207 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:40:01.207 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26605", 2024-03-20T09:40:01.207 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:40:01.207 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 70582 2024-03-20T09:40:01.207 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.208 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:40:01.208 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.208 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:40:01.208 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.334821+0000", 2024-03-20T09:40:01.208 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.208 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.208 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.208 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:40:01.208 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.334821+0000", 2024-03-20T09:40:01.208 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.208 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.208 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.208 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:40:01.209 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.334823+0000", 2024-03-20T09:40:01.209 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2759999999999999e-06 2024-03-20T09:40:01.209 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.209 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.209 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:40:01.209 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.334847+0000", 2024-03-20T09:40:01.209 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4369e-05 2024-03-20T09:40:01.209 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.209 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.209 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:40:01.209 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.334859+0000", 2024-03-20T09:40:01.209 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2241e-05 2024-03-20T09:40:01.209 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.210 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.210 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:40:01.210 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.334867+0000", 2024-03-20T09:40:01.210 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1540000000000002e-06 2024-03-20T09:40:01.210 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.210 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.210 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:40:01.210 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.334920+0000", 2024-03-20T09:40:01.210 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3072999999999998e-05 2024-03-20T09:40:01.210 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.210 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.211 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:40:01.211 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.335079+0000", 2024-03-20T09:40:01.211 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000158245 2024-03-20T09:40:01.211 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.211 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.211 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2,11", 2024-03-20T09:40:01.211 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.335233+0000", 2024-03-20T09:40:01.211 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015473799999999999 2024-03-20T09:40:01.211 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.211 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.211 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:40:01.211 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.337088+0000", 2024-03-20T09:40:01.211 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0018542420000000001 2024-03-20T09:40:01.211 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.212 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.212 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:01.212 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.337517+0000", 2024-03-20T09:40:01.212 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00042966800000000002 2024-03-20T09:40:01.212 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.212 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.212 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:01.212 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.950176+0000", 2024-03-20T09:40:01.212 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.61265878699999998 2024-03-20T09:40:01.212 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.212 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.212 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:40:01.212 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.950238+0000", 2024-03-20T09:40:01.213 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1743999999999997e-05 2024-03-20T09:40:01.213 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.213 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.213 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:40:01.213 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.950275+0000", 2024-03-20T09:40:01.213 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7515999999999998e-05 2024-03-20T09:40:01.213 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.213 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:40:01.213 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.213 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.213 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.213 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26605.0:70594 7.a 7:515d4916:::benchmark_data_smithi122_187077_object70593:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e889)", 2024-03-20T09:40:01.213 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:31:54.343556+0000", 2024-03-20T09:40:01.214 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 486.86902133900003, 2024-03-20T09:40:01.214 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.61242518099999999, 2024-03-20T09:40:01.214 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:40:01.214 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:40:01.214 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:40:01.214 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26605", 2024-03-20T09:40:01.214 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:40:01.214 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 70594 2024-03-20T09:40:01.214 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.214 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:40:01.214 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.214 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:40:01.214 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.343556+0000", 2024-03-20T09:40:01.215 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.215 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.215 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.215 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:40:01.215 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.343556+0000", 2024-03-20T09:40:01.215 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.215 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.215 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.215 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:40:01.215 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.343557+0000", 2024-03-20T09:40:01.215 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3820000000000001e-06 2024-03-20T09:40:01.215 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.215 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.215 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:40:01.216 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.343576+0000", 2024-03-20T09:40:01.216 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8674000000000001e-05 2024-03-20T09:40:01.216 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.216 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.216 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:40:01.216 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.343587+0000", 2024-03-20T09:40:01.216 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1365e-05 2024-03-20T09:40:01.216 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.216 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.216 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:40:01.216 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.343593+0000", 2024-03-20T09:40:01.216 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7899999999999996e-06 2024-03-20T09:40:01.216 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.216 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.216 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:40:01.217 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.343639+0000", 2024-03-20T09:40:01.217 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6409000000000002e-05 2024-03-20T09:40:01.217 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.217 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.217 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:40:01.217 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.343742+0000", 2024-03-20T09:40:01.217 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000102854 2024-03-20T09:40:01.217 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.217 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.217 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2,11", 2024-03-20T09:40:01.217 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.343845+0000", 2024-03-20T09:40:01.217 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000102686 2024-03-20T09:40:01.217 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.217 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.217 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:40:01.217 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.345876+0000", 2024-03-20T09:40:01.218 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0020312889999999999 2024-03-20T09:40:01.218 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.218 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.218 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:01.218 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.346540+0000", 2024-03-20T09:40:01.218 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00066372200000000005 2024-03-20T09:40:01.218 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.218 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.218 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:01.218 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.955911+0000", 2024-03-20T09:40:01.218 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.60937121299999997 2024-03-20T09:40:01.218 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.218 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.218 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:40:01.219 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.955950+0000", 2024-03-20T09:40:01.219 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8522000000000001e-05 2024-03-20T09:40:01.219 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.219 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.219 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:40:01.219 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.955981+0000", 2024-03-20T09:40:01.219 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1275000000000001e-05 2024-03-20T09:40:01.219 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.219 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:40:01.219 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.219 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.219 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.219 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26605.0:70609 7.a 7:53409cd0:::benchmark_data_smithi122_187077_object70608:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e889)", 2024-03-20T09:40:01.219 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:31:54.352442+0000", 2024-03-20T09:40:01.220 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 486.86013519599999, 2024-03-20T09:40:01.220 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.60363811599999995, 2024-03-20T09:40:01.220 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:40:01.220 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:40:01.220 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:40:01.220 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26605", 2024-03-20T09:40:01.220 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:40:01.220 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 70609 2024-03-20T09:40:01.220 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.220 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:40:01.220 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.220 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:40:01.220 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.352442+0000", 2024-03-20T09:40:01.221 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.221 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.221 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.221 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:40:01.221 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.352442+0000", 2024-03-20T09:40:01.221 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.221 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.221 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.221 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:40:01.221 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.352443+0000", 2024-03-20T09:40:01.221 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2300000000000001e-06 2024-03-20T09:40:01.221 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.221 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.221 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:40:01.222 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.352462+0000", 2024-03-20T09:40:01.222 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8989999999999999e-05 2024-03-20T09:40:01.222 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.222 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.222 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:40:01.222 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.352475+0000", 2024-03-20T09:40:01.222 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2425999999999999e-05 2024-03-20T09:40:01.222 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.222 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.222 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:40:01.222 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.352481+0000", 2024-03-20T09:40:01.222 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7549999999999997e-06 2024-03-20T09:40:01.222 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.223 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.223 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:40:01.223 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.352527+0000", 2024-03-20T09:40:01.223 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5989000000000003e-05 2024-03-20T09:40:01.223 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.223 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.223 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:40:01.223 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.352625+0000", 2024-03-20T09:40:01.223 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7887000000000007e-05 2024-03-20T09:40:01.223 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.223 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.223 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2,11", 2024-03-20T09:40:01.223 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.352724+0000", 2024-03-20T09:40:01.224 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.8548999999999994e-05 2024-03-20T09:40:01.224 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.224 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.224 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:40:01.224 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.354608+0000", 2024-03-20T09:40:01.224 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0018842959999999999 2024-03-20T09:40:01.224 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.224 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.224 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:01.224 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.354970+0000", 2024-03-20T09:40:01.224 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00036139899999999999 2024-03-20T09:40:01.224 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.224 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.227 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:01.227 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.956013+0000", 2024-03-20T09:40:01.227 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.601043255 2024-03-20T09:40:01.227 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.227 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.228 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:40:01.228 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.956050+0000", 2024-03-20T09:40:01.228 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7682000000000003e-05 2024-03-20T09:40:01.228 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.228 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.228 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:40:01.228 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.956080+0000", 2024-03-20T09:40:01.228 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9658e-05 2024-03-20T09:40:01.228 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.228 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:40:01.228 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.228 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.228 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.228 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26605.0:70640 7.a 7:52f51c38:::benchmark_data_smithi122_187077_object70639:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e889)", 2024-03-20T09:40:01.229 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:31:54.374960+0000", 2024-03-20T09:40:01.229 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 486.83761694499998, 2024-03-20T09:40:01.229 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.58312417999999999, 2024-03-20T09:40:01.229 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:40:01.229 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:40:01.229 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:40:01.229 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26605", 2024-03-20T09:40:01.229 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:40:01.229 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 70640 2024-03-20T09:40:01.229 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.229 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:40:01.229 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.229 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:40:01.229 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.374960+0000", 2024-03-20T09:40:01.229 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.229 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.230 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.230 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:40:01.230 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.374960+0000", 2024-03-20T09:40:01.230 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.230 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.230 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.230 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:40:01.230 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.374962+0000", 2024-03-20T09:40:01.230 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1299999999999999e-06 2024-03-20T09:40:01.230 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.230 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.230 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:40:01.230 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.375004+0000", 2024-03-20T09:40:01.230 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1672999999999999e-05 2024-03-20T09:40:01.231 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.231 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.231 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:40:01.231 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.375041+0000", 2024-03-20T09:40:01.231 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6752000000000003e-05 2024-03-20T09:40:01.231 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.231 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.231 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:40:01.231 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.375049+0000", 2024-03-20T09:40:01.231 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7950000000000008e-06 2024-03-20T09:40:01.231 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.231 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.231 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:40:01.231 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.375099+0000", 2024-03-20T09:40:01.231 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0249e-05 2024-03-20T09:40:01.232 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.232 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.232 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:40:01.232 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.375214+0000", 2024-03-20T09:40:01.232 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000114936 2024-03-20T09:40:01.232 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.232 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.232 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2,11", 2024-03-20T09:40:01.232 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.375371+0000", 2024-03-20T09:40:01.232 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015677700000000001 2024-03-20T09:40:01.232 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.232 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.232 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:40:01.232 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.379839+0000", 2024-03-20T09:40:01.233 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0044681449999999998 2024-03-20T09:40:01.233 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.233 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.233 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:01.233 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.379957+0000", 2024-03-20T09:40:01.233 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000117959 2024-03-20T09:40:01.233 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.233 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.233 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:01.233 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.957974+0000", 2024-03-20T09:40:01.233 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.57801762999999995 2024-03-20T09:40:01.233 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.233 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.233 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:40:01.233 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.958048+0000", 2024-03-20T09:40:01.233 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3269999999999995e-05 2024-03-20T09:40:01.234 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.234 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.234 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:40:01.234 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.958084+0000", 2024-03-20T09:40:01.234 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6863999999999998e-05 2024-03-20T09:40:01.234 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.234 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:40:01.234 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.234 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.234 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.234 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26605.0:70644 7.a 7:508ee616:::benchmark_data_smithi122_187077_object70643:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e889)", 2024-03-20T09:40:01.234 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:31:54.375731+0000", 2024-03-20T09:40:01.234 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 486.83684612600001, 2024-03-20T09:40:01.234 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.58246920400000002, 2024-03-20T09:40:01.235 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:40:01.235 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:40:01.235 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:40:01.235 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26605", 2024-03-20T09:40:01.235 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:40:01.235 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 70644 2024-03-20T09:40:01.235 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.235 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:40:01.235 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.235 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:40:01.235 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.375731+0000", 2024-03-20T09:40:01.235 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.235 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.235 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.235 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:40:01.236 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.375731+0000", 2024-03-20T09:40:01.236 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.236 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.236 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.236 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:40:01.236 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.375732+0000", 2024-03-20T09:40:01.236 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1999999999999999e-07 2024-03-20T09:40:01.236 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.236 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.236 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:40:01.236 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.375749+0000", 2024-03-20T09:40:01.236 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7119999999999999e-05 2024-03-20T09:40:01.236 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.236 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.237 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:40:01.237 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.375759+0000", 2024-03-20T09:40:01.237 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0016e-05 2024-03-20T09:40:01.237 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.237 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.237 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:40:01.237 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.375766+0000", 2024-03-20T09:40:01.237 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.148e-06 2024-03-20T09:40:01.237 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.237 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.237 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:40:01.237 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.377312+0000", 2024-03-20T09:40:01.237 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001546157 2024-03-20T09:40:01.237 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.237 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.238 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:40:01.238 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.377450+0000", 2024-03-20T09:40:01.238 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000137288 2024-03-20T09:40:01.238 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.238 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.238 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2,11", 2024-03-20T09:40:01.238 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.377594+0000", 2024-03-20T09:40:01.238 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000144325 2024-03-20T09:40:01.238 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.238 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.238 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:40:01.238 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.379910+0000", 2024-03-20T09:40:01.238 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0023163099999999998 2024-03-20T09:40:01.238 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.239 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.239 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:01.239 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.382436+0000", 2024-03-20T09:40:01.239 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0025263519999999999 2024-03-20T09:40:01.239 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.239 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.239 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:01.239 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.958127+0000", 2024-03-20T09:40:01.239 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.57569085399999997 2024-03-20T09:40:01.239 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.239 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.239 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:40:01.239 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.958167+0000", 2024-03-20T09:40:01.239 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9263e-05 2024-03-20T09:40:01.239 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.240 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.240 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:40:01.240 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.958200+0000", 2024-03-20T09:40:01.240 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3651e-05 2024-03-20T09:40:01.240 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.240 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:40:01.240 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.240 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.240 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.240 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26605.0:70666 7.a 7:574d09fb:::benchmark_data_smithi122_187077_object70665:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e889)", 2024-03-20T09:40:01.240 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:31:54.395626+0000", 2024-03-20T09:40:01.240 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 486.81695094700001, 2024-03-20T09:40:01.240 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.56274611399999996, 2024-03-20T09:40:01.240 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:40:01.241 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:40:01.241 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:40:01.241 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26605", 2024-03-20T09:40:01.241 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:40:01.241 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 70666 2024-03-20T09:40:01.241 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.241 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:40:01.241 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.241 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:40:01.241 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.395626+0000", 2024-03-20T09:40:01.241 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.241 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.241 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.241 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:40:01.241 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.395626+0000", 2024-03-20T09:40:01.242 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.242 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.242 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.242 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:40:01.242 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.395627+0000", 2024-03-20T09:40:01.242 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.111e-06 2024-03-20T09:40:01.242 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.242 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.242 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:40:01.242 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.395646+0000", 2024-03-20T09:40:01.242 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9125000000000001e-05 2024-03-20T09:40:01.242 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.242 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.243 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:40:01.243 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.395658+0000", 2024-03-20T09:40:01.243 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1532000000000001e-05 2024-03-20T09:40:01.243 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.243 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.243 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:40:01.243 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.395664+0000", 2024-03-20T09:40:01.243 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7010000000000001e-06 2024-03-20T09:40:01.243 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.243 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.243 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:40:01.243 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.395721+0000", 2024-03-20T09:40:01.243 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7779999999999999e-05 2024-03-20T09:40:01.243 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.244 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.244 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:40:01.244 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.395877+0000", 2024-03-20T09:40:01.244 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015524500000000001 2024-03-20T09:40:01.244 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.244 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.244 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2,11", 2024-03-20T09:40:01.244 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.395975+0000", 2024-03-20T09:40:01.244 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.8369000000000006e-05 2024-03-20T09:40:01.244 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.244 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.244 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:40:01.244 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.397872+0000", 2024-03-20T09:40:01.244 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0018965690000000001 2024-03-20T09:40:01.244 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.245 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.245 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:01.245 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.398086+0000", 2024-03-20T09:40:01.245 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000214222 2024-03-20T09:40:01.245 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.245 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.245 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:01.245 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.958253+0000", 2024-03-20T09:40:01.245 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.56016706400000005 2024-03-20T09:40:01.245 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.245 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.245 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:40:01.245 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.958322+0000", 2024-03-20T09:40:01.245 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9209999999999996e-05 2024-03-20T09:40:01.245 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.245 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.246 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:40:01.246 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.958372+0000", 2024-03-20T09:40:01.246 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0185999999999997e-05 2024-03-20T09:40:01.246 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.246 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:40:01.246 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.246 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.246 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.246 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26605.0:70678 7.a 7:5892cbde:::benchmark_data_smithi122_187077_object70677:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e889)", 2024-03-20T09:40:01.246 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:31:54.469987+0000", 2024-03-20T09:40:01.246 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 486.74258993400002, 2024-03-20T09:40:01.246 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.488575174, 2024-03-20T09:40:01.246 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:40:01.246 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:40:01.246 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:40:01.247 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26605", 2024-03-20T09:40:01.247 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:40:01.247 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 70678 2024-03-20T09:40:01.247 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.247 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:40:01.247 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.247 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:40:01.247 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.469987+0000", 2024-03-20T09:40:01.247 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.247 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.247 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.247 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:40:01.247 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.469987+0000", 2024-03-20T09:40:01.247 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.247 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.247 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.248 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:40:01.248 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.469990+0000", 2024-03-20T09:40:01.248 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5969999999999999e-06 2024-03-20T09:40:01.248 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.248 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.248 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:40:01.248 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.470037+0000", 2024-03-20T09:40:01.248 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7178000000000001e-05 2024-03-20T09:40:01.248 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.248 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.248 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:40:01.248 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.470049+0000", 2024-03-20T09:40:01.248 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1576999999999999e-05 2024-03-20T09:40:01.248 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.248 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.249 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:40:01.249 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.470057+0000", 2024-03-20T09:40:01.249 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1340000000000004e-06 2024-03-20T09:40:01.249 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.249 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.249 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:40:01.249 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.470109+0000", 2024-03-20T09:40:01.249 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1854000000000002e-05 2024-03-20T09:40:01.249 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.249 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.249 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:40:01.249 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.470215+0000", 2024-03-20T09:40:01.249 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000106408 2024-03-20T09:40:01.250 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.250 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.250 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2,11", 2024-03-20T09:40:01.250 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.470324+0000", 2024-03-20T09:40:01.250 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000109221 2024-03-20T09:40:01.250 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.250 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.250 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:40:01.250 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.472160+0000", 2024-03-20T09:40:01.250 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001835602 2024-03-20T09:40:01.250 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.250 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.250 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:01.251 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.472471+0000", 2024-03-20T09:40:01.251 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00031078300000000002 2024-03-20T09:40:01.251 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.251 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.251 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:01.251 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.958461+0000", 2024-03-20T09:40:01.251 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.48599062900000001 2024-03-20T09:40:01.251 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.251 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.251 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:40:01.251 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.958526+0000", 2024-03-20T09:40:01.251 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4471e-05 2024-03-20T09:40:01.251 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.251 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.252 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:40:01.252 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:31:54.958562+0000", 2024-03-20T09:40:01.252 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6720000000000001e-05 2024-03-20T09:40:01.252 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.252 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:40:01.252 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.252 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.252 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.252 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:11286 8.1 8:80a48722:::benchmark_data_smithi122_197775_object11285:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e980)", 2024-03-20T09:40:01.252 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:36:04.077724+0000", 2024-03-20T09:40:01.252 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 237.13485298000001, 2024-03-20T09:40:01.252 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20547268299999999, 2024-03-20T09:40:01.252 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:40:01.252 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:40:01.252 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:40:01.253 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:40:01.253 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:40:01.253 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 11286 2024-03-20T09:40:01.253 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.253 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:40:01.253 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.253 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:40:01.253 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.077724+0000", 2024-03-20T09:40:01.253 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.253 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.253 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.253 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:40:01.253 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.077724+0000", 2024-03-20T09:40:01.253 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.253 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.254 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.254 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:40:01.254 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.077726+0000", 2024-03-20T09:40:01.254 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4720000000000001e-06 2024-03-20T09:40:01.254 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.254 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.254 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:40:01.254 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.077753+0000", 2024-03-20T09:40:01.254 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7667000000000001e-05 2024-03-20T09:40:01.254 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.254 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.254 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:40:01.254 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.077769+0000", 2024-03-20T09:40:01.254 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5857000000000001e-05 2024-03-20T09:40:01.255 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.255 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.255 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:40:01.255 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.077778+0000", 2024-03-20T09:40:01.255 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9639999999999992e-06 2024-03-20T09:40:01.255 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.255 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.255 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:40:01.255 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.077839+0000", 2024-03-20T09:40:01.255 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1326000000000002e-05 2024-03-20T09:40:01.255 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.255 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.255 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:40:01.255 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.077952+0000", 2024-03-20T09:40:01.256 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000112459 2024-03-20T09:40:01.256 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.256 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.256 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,2", 2024-03-20T09:40:01.256 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.078075+0000", 2024-03-20T09:40:01.256 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012269200000000001 2024-03-20T09:40:01.256 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.256 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.256 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:40:01.256 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.079021+0000", 2024-03-20T09:40:01.256 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00094592199999999997 2024-03-20T09:40:01.256 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.256 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.256 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:01.257 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.080268+0000", 2024-03-20T09:40:01.257 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0012472799999999999 2024-03-20T09:40:01.257 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.257 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.257 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:01.257 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.283126+0000", 2024-03-20T09:40:01.257 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20285774600000001 2024-03-20T09:40:01.257 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.257 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.257 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:40:01.257 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.283159+0000", 2024-03-20T09:40:01.257 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3890999999999998e-05 2024-03-20T09:40:01.257 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.257 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.258 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:40:01.258 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.283197+0000", 2024-03-20T09:40:01.258 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7407000000000003e-05 2024-03-20T09:40:01.258 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.258 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:40:01.258 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.258 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.258 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.258 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:11290 8.1 8:80c13a40:::benchmark_data_smithi122_197775_object11289:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e980)", 2024-03-20T09:40:01.258 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:36:04.079238+0000", 2024-03-20T09:40:01.258 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 237.133338869, 2024-03-20T09:40:01.258 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20571076599999999, 2024-03-20T09:40:01.258 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:40:01.259 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:40:01.259 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:40:01.259 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:40:01.259 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:40:01.259 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 11290 2024-03-20T09:40:01.259 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.259 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:40:01.259 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.259 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:40:01.259 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.079238+0000", 2024-03-20T09:40:01.259 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.259 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.259 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.259 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:40:01.259 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.079238+0000", 2024-03-20T09:40:01.260 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.260 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.260 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.260 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:40:01.260 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.079240+0000", 2024-03-20T09:40:01.260 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.291e-06 2024-03-20T09:40:01.260 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.260 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.260 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:40:01.260 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.079262+0000", 2024-03-20T09:40:01.260 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2673000000000001e-05 2024-03-20T09:40:01.260 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.260 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.260 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:40:01.260 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.079275+0000", 2024-03-20T09:40:01.261 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3011e-05 2024-03-20T09:40:01.261 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.261 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.261 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:40:01.261 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.079294+0000", 2024-03-20T09:40:01.261 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8641e-05 2024-03-20T09:40:01.261 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.261 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.261 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:40:01.261 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.079356+0000", 2024-03-20T09:40:01.261 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2083999999999997e-05 2024-03-20T09:40:01.261 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.261 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.262 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:40:01.262 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.079516+0000", 2024-03-20T09:40:01.262 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016011000000000001 2024-03-20T09:40:01.262 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.262 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.262 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,2", 2024-03-20T09:40:01.262 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.079662+0000", 2024-03-20T09:40:01.262 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014622499999999999 2024-03-20T09:40:01.262 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.262 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.262 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:40:01.263 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.080579+0000", 2024-03-20T09:40:01.263 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00091626100000000003 2024-03-20T09:40:01.263 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.263 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.263 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:01.263 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.081707+0000", 2024-03-20T09:40:01.263 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0011281010000000001 2024-03-20T09:40:01.263 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.263 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.263 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:01.263 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.284880+0000", 2024-03-20T09:40:01.263 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.203173522 2024-03-20T09:40:01.263 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.263 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.264 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:40:01.264 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.284912+0000", 2024-03-20T09:40:01.264 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1931999999999999e-05 2024-03-20T09:40:01.264 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.264 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.264 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:40:01.264 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.284949+0000", 2024-03-20T09:40:01.264 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6915000000000001e-05 2024-03-20T09:40:01.264 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.264 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:40:01.264 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.264 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.264 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.264 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:11293 8.1 8:8391fa78:::benchmark_data_smithi122_197775_object11292:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e980)", 2024-03-20T09:40:01.265 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:36:04.080539+0000", 2024-03-20T09:40:01.265 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 237.13203827000001, 2024-03-20T09:40:01.265 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20506249200000001, 2024-03-20T09:40:01.265 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:40:01.265 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:40:01.265 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:40:01.265 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:40:01.265 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:40:01.265 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 11293 2024-03-20T09:40:01.265 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.265 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:40:01.265 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.265 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:40:01.265 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.080539+0000", 2024-03-20T09:40:01.266 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.266 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.266 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.266 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:40:01.266 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.080539+0000", 2024-03-20T09:40:01.266 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.266 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.266 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.266 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:40:01.266 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.080540+0000", 2024-03-20T09:40:01.266 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2359999999999999e-06 2024-03-20T09:40:01.266 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.266 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.266 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:40:01.267 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.080570+0000", 2024-03-20T09:40:01.267 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9910999999999999e-05 2024-03-20T09:40:01.267 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.267 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.267 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:40:01.267 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.080581+0000", 2024-03-20T09:40:01.267 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1286e-05 2024-03-20T09:40:01.267 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.267 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.267 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:40:01.267 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.080589+0000", 2024-03-20T09:40:01.267 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6240000000000002e-06 2024-03-20T09:40:01.267 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.267 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.267 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:40:01.268 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.080660+0000", 2024-03-20T09:40:01.268 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0764000000000002e-05 2024-03-20T09:40:01.268 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.268 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.268 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:40:01.268 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.080767+0000", 2024-03-20T09:40:01.268 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010772099999999999 2024-03-20T09:40:01.268 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.268 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.268 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,2", 2024-03-20T09:40:01.268 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.080882+0000", 2024-03-20T09:40:01.268 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000114647 2024-03-20T09:40:01.268 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.268 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.268 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:40:01.269 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.081465+0000", 2024-03-20T09:40:01.269 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00058243499999999996 2024-03-20T09:40:01.269 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.269 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.269 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:01.269 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.081836+0000", 2024-03-20T09:40:01.269 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00037194099999999998 2024-03-20T09:40:01.269 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.269 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.269 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:01.269 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.285534+0000", 2024-03-20T09:40:01.269 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.203697238 2024-03-20T09:40:01.269 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.269 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.269 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:40:01.270 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.285564+0000", 2024-03-20T09:40:01.270 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0026e-05 2024-03-20T09:40:01.270 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.270 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.270 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:40:01.270 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.285601+0000", 2024-03-20T09:40:01.270 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7663000000000002e-05 2024-03-20T09:40:01.270 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.270 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:40:01.270 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.270 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.270 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.270 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:11295 8.1 8:8757dbea:::benchmark_data_smithi122_197775_object11294:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e980)", 2024-03-20T09:40:01.270 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:36:04.081987+0000", 2024-03-20T09:40:01.271 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 237.130589778, 2024-03-20T09:40:01.271 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20374166699999999, 2024-03-20T09:40:01.271 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:40:01.271 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:40:01.271 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:40:01.271 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:40:01.271 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:40:01.271 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 11295 2024-03-20T09:40:01.271 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.271 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:40:01.271 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.271 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:40:01.271 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.081987+0000", 2024-03-20T09:40:01.271 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.271 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.272 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.272 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:40:01.272 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.081987+0000", 2024-03-20T09:40:01.272 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.272 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.272 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.272 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:40:01.272 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.081989+0000", 2024-03-20T09:40:01.272 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2309999999999999e-06 2024-03-20T09:40:01.272 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.272 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.272 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:40:01.272 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.082026+0000", 2024-03-20T09:40:01.272 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7067999999999998e-05 2024-03-20T09:40:01.273 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.273 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.273 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:40:01.273 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.082039+0000", 2024-03-20T09:40:01.273 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3472000000000001e-05 2024-03-20T09:40:01.273 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.273 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.273 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:40:01.273 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.082047+0000", 2024-03-20T09:40:01.273 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4329999999999998e-06 2024-03-20T09:40:01.273 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.273 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.273 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:40:01.273 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.082108+0000", 2024-03-20T09:40:01.274 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1233000000000001e-05 2024-03-20T09:40:01.274 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.274 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.274 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:40:01.274 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.082233+0000", 2024-03-20T09:40:01.274 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012532999999999999 2024-03-20T09:40:01.274 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.274 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.274 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,2", 2024-03-20T09:40:01.274 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.082392+0000", 2024-03-20T09:40:01.274 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015852300000000001 2024-03-20T09:40:01.274 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.274 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.274 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:40:01.274 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.084160+0000", 2024-03-20T09:40:01.274 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001768751 2024-03-20T09:40:01.275 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.275 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.275 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:01.275 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.084241+0000", 2024-03-20T09:40:01.275 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0992e-05 2024-03-20T09:40:01.275 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.275 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.275 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:01.275 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.285630+0000", 2024-03-20T09:40:01.275 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20138837700000001 2024-03-20T09:40:01.275 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.275 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.275 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:40:01.276 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.285679+0000", 2024-03-20T09:40:01.276 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9206000000000003e-05 2024-03-20T09:40:01.276 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.276 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.276 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:40:01.276 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.285729+0000", 2024-03-20T09:40:01.276 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0050999999999999e-05 2024-03-20T09:40:01.276 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.276 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:40:01.276 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.276 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.276 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.276 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:11300 8.1 8:8fc41842:::benchmark_data_smithi122_197775_object11299:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e980)", 2024-03-20T09:40:01.276 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:36:04.082159+0000", 2024-03-20T09:40:01.277 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 237.13041780699999, 2024-03-20T09:40:01.277 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20485677899999999, 2024-03-20T09:40:01.277 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:40:01.277 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:40:01.277 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:40:01.277 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:40:01.277 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:40:01.277 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 11300 2024-03-20T09:40:01.277 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.277 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:40:01.277 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.277 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:40:01.277 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.082159+0000", 2024-03-20T09:40:01.277 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.278 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.278 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.278 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:40:01.278 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.082159+0000", 2024-03-20T09:40:01.278 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.278 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.278 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.278 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:40:01.278 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.082160+0000", 2024-03-20T09:40:01.278 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5899999999999996e-07 2024-03-20T09:40:01.278 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.278 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.278 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:40:01.278 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.082182+0000", 2024-03-20T09:40:01.278 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1566e-05 2024-03-20T09:40:01.279 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.279 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.279 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:40:01.279 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.082196+0000", 2024-03-20T09:40:01.279 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4599e-05 2024-03-20T09:40:01.279 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.279 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.279 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:40:01.279 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.082203+0000", 2024-03-20T09:40:01.279 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1110000000000002e-06 2024-03-20T09:40:01.279 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.279 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.279 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:40:01.279 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.083032+0000", 2024-03-20T09:40:01.280 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00082847699999999999 2024-03-20T09:40:01.280 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.280 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.280 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:40:01.280 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.083148+0000", 2024-03-20T09:40:01.280 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000116221 2024-03-20T09:40:01.280 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.280 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.280 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,2", 2024-03-20T09:40:01.280 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.083293+0000", 2024-03-20T09:40:01.280 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000144578 2024-03-20T09:40:01.280 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.280 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.280 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:40:01.281 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.084208+0000", 2024-03-20T09:40:01.281 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00091588100000000001 2024-03-20T09:40:01.281 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.281 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.281 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:01.281 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.084838+0000", 2024-03-20T09:40:01.281 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00062971700000000002 2024-03-20T09:40:01.281 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.281 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.281 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:01.281 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.286934+0000", 2024-03-20T09:40:01.281 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.202095357 2024-03-20T09:40:01.281 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.281 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.281 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:40:01.282 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.286972+0000", 2024-03-20T09:40:01.282 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8900999999999999e-05 2024-03-20T09:40:01.282 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.282 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.282 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:40:01.282 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.287016+0000", 2024-03-20T09:40:01.282 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3711999999999998e-05 2024-03-20T09:40:01.282 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.282 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:40:01.282 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.282 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.282 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.282 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:11321 8.1 8:886b3c97:::benchmark_data_smithi122_197775_object11320:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e980)", 2024-03-20T09:40:01.282 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:36:04.090452+0000", 2024-03-20T09:40:01.283 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 237.12212530299999, 2024-03-20T09:40:01.283 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19670465000000001, 2024-03-20T09:40:01.283 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:40:01.283 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:40:01.283 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:40:01.283 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:40:01.283 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:40:01.283 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 11321 2024-03-20T09:40:01.283 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.283 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:40:01.283 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.283 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:40:01.283 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.090452+0000", 2024-03-20T09:40:01.283 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.284 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.284 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.284 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:40:01.284 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.090452+0000", 2024-03-20T09:40:01.284 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.284 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.284 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.284 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:40:01.284 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.090453+0000", 2024-03-20T09:40:01.284 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0800000000000003e-07 2024-03-20T09:40:01.284 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.284 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.284 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:40:01.284 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.090477+0000", 2024-03-20T09:40:01.284 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4362000000000002e-05 2024-03-20T09:40:01.285 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.285 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.285 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:40:01.285 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.090490+0000", 2024-03-20T09:40:01.285 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2909e-05 2024-03-20T09:40:01.285 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.285 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.285 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:40:01.285 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.090498+0000", 2024-03-20T09:40:01.285 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9019999999999999e-06 2024-03-20T09:40:01.285 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.285 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.285 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:40:01.285 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.090557+0000", 2024-03-20T09:40:01.285 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8949000000000001e-05 2024-03-20T09:40:01.285 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.286 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.286 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:40:01.286 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.090689+0000", 2024-03-20T09:40:01.286 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000132256 2024-03-20T09:40:01.286 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.286 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.286 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,2", 2024-03-20T09:40:01.286 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.090809+0000", 2024-03-20T09:40:01.286 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000120148 2024-03-20T09:40:01.286 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.286 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.286 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:01.286 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.092230+0000", 2024-03-20T09:40:01.286 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001420665 2024-03-20T09:40:01.286 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.287 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.287 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:40:01.287 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.092243+0000", 2024-03-20T09:40:01.287 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3030999999999999e-05 2024-03-20T09:40:01.287 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.287 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.287 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:01.287 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.287061+0000", 2024-03-20T09:40:01.287 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.194817502 2024-03-20T09:40:01.287 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.287 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.287 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:40:01.287 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.287112+0000", 2024-03-20T09:40:01.288 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1928999999999997e-05 2024-03-20T09:40:01.288 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.288 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.288 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:40:01.288 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.287157+0000", 2024-03-20T09:40:01.288 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4088999999999997e-05 2024-03-20T09:40:01.288 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.288 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:40:01.288 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.288 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.288 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.288 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:11295 8.1 8:8757dbea:::benchmark_data_smithi122_197775_object11294:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e980)", 2024-03-20T09:40:01.288 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:36:04.099012+0000", 2024-03-20T09:40:01.288 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 237.11356553799999, 2024-03-20T09:40:01.289 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18667910300000001, 2024-03-20T09:40:01.289 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:40:01.289 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "waiting for ondisk", 2024-03-20T09:40:01.289 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:40:01.289 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:40:01.289 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:40:01.289 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 11295 2024-03-20T09:40:01.289 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.289 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:40:01.289 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.289 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:40:01.289 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.099012+0000", 2024-03-20T09:40:01.289 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.289 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.289 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.289 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:40:01.290 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.099012+0000", 2024-03-20T09:40:01.290 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.290 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.290 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.290 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:40:01.290 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.099012+0000", 2024-03-20T09:40:01.290 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1100000000000001e-07 2024-03-20T09:40:01.290 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.290 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.290 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:40:01.290 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.099045+0000", 2024-03-20T09:40:01.290 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2895000000000003e-05 2024-03-20T09:40:01.290 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.290 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.291 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:40:01.291 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.099052+0000", 2024-03-20T09:40:01.291 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3490000000000003e-06 2024-03-20T09:40:01.291 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.291 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.291 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:40:01.291 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.099058+0000", 2024-03-20T09:40:01.291 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5180000000000004e-06 2024-03-20T09:40:01.291 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.291 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.291 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:40:01.291 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.099259+0000", 2024-03-20T09:40:01.291 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00020135699999999999 2024-03-20T09:40:01.291 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.291 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.292 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for ondisk", 2024-03-20T09:40:01.292 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.099314+0000", 2024-03-20T09:40:01.292 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4598999999999997e-05 2024-03-20T09:40:01.292 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.292 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.292 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:40:01.292 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.285691+0000", 2024-03-20T09:40:01.292 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.186376974 2024-03-20T09:40:01.292 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.292 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:40:01.292 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.292 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.292 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.292 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:11300 8.1 8:8fc41842:::benchmark_data_smithi122_197775_object11299:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e980)", 2024-03-20T09:40:01.292 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:36:04.099123+0000", 2024-03-20T09:40:01.293 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 237.11345431000001, 2024-03-20T09:40:01.293 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.187860109, 2024-03-20T09:40:01.293 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:40:01.293 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "waiting for ondisk", 2024-03-20T09:40:01.293 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:40:01.293 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:40:01.293 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:40:01.293 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 11300 2024-03-20T09:40:01.293 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.293 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:40:01.293 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.293 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:40:01.293 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.099123+0000", 2024-03-20T09:40:01.293 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.294 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.294 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.294 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:40:01.294 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.099123+0000", 2024-03-20T09:40:01.294 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.294 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.294 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.294 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:40:01.294 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.099124+0000", 2024-03-20T09:40:01.294 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4300000000000002e-07 2024-03-20T09:40:01.294 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.294 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.294 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:40:01.295 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.099154+0000", 2024-03-20T09:40:01.295 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9916e-05 2024-03-20T09:40:01.295 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.295 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.295 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:40:01.295 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.099162+0000", 2024-03-20T09:40:01.295 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7660000000000004e-06 2024-03-20T09:40:01.295 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.295 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.295 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:40:01.295 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.099169+0000", 2024-03-20T09:40:01.295 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0280000000000003e-06 2024-03-20T09:40:01.295 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.295 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.295 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:40:01.296 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.099338+0000", 2024-03-20T09:40:01.296 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016870700000000001 2024-03-20T09:40:01.296 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.296 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.296 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for ondisk", 2024-03-20T09:40:01.296 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.099374+0000", 2024-03-20T09:40:01.296 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6111e-05 2024-03-20T09:40:01.296 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.296 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.296 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:40:01.296 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.286983+0000", 2024-03-20T09:40:01.296 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.187608838 2024-03-20T09:40:01.296 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.296 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:40:01.296 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.297 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.297 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.297 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:11321 8.1 8:886b3c97:::benchmark_data_smithi122_197775_object11320:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e980)", 2024-03-20T09:40:01.297 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:36:04.099356+0000", 2024-03-20T09:40:01.297 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 237.113221227, 2024-03-20T09:40:01.297 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18776670600000001, 2024-03-20T09:40:01.297 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:40:01.297 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "waiting for ondisk", 2024-03-20T09:40:01.297 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:40:01.297 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:40:01.297 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:40:01.297 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 11321 2024-03-20T09:40:01.297 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.298 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:40:01.298 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.298 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:40:01.298 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.099356+0000", 2024-03-20T09:40:01.298 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.298 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.298 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.298 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:40:01.298 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.099356+0000", 2024-03-20T09:40:01.298 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.298 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.298 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.298 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:40:01.299 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.099356+0000", 2024-03-20T09:40:01.299 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3599999999999999e-07 2024-03-20T09:40:01.299 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.299 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.299 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:40:01.299 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.099371+0000", 2024-03-20T09:40:01.299 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4853e-05 2024-03-20T09:40:01.299 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.299 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.299 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:40:01.299 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.099375+0000", 2024-03-20T09:40:01.299 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.963e-06 2024-03-20T09:40:01.299 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.299 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.299 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:40:01.299 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.099378+0000", 2024-03-20T09:40:01.300 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0769999999999999e-06 2024-03-20T09:40:01.300 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.300 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.300 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:40:01.300 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.099411+0000", 2024-03-20T09:40:01.300 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2548000000000001e-05 2024-03-20T09:40:01.300 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.300 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.300 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for ondisk", 2024-03-20T09:40:01.300 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.099441+0000", 2024-03-20T09:40:01.300 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0642000000000002e-05 2024-03-20T09:40:01.300 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.300 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.301 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:40:01.301 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.287123+0000", 2024-03-20T09:40:01.301 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.187681187 2024-03-20T09:40:01.301 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.301 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:40:01.301 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.301 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.301 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.301 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:11334 8.1 8:85645c3e:::benchmark_data_smithi122_197775_object11333:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e980)", 2024-03-20T09:40:01.301 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:36:04.099412+0000", 2024-03-20T09:40:01.301 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 237.11316473900001, 2024-03-20T09:40:01.301 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18791055300000001, 2024-03-20T09:40:01.301 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:40:01.302 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:40:01.302 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:40:01.302 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:40:01.302 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:40:01.302 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 11334 2024-03-20T09:40:01.302 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.302 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:40:01.302 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.302 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:40:01.302 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.099412+0000", 2024-03-20T09:40:01.302 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.302 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.302 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.302 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:40:01.303 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.099412+0000", 2024-03-20T09:40:01.303 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.303 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.303 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.303 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:40:01.303 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.099413+0000", 2024-03-20T09:40:01.303 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3900000000000001e-07 2024-03-20T09:40:01.303 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.303 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.303 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:40:01.303 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.099431+0000", 2024-03-20T09:40:01.303 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8411e-05 2024-03-20T09:40:01.303 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.304 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.304 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:40:01.304 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.099437+0000", 2024-03-20T09:40:01.304 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3679999999999998e-06 2024-03-20T09:40:01.304 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.304 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.304 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:40:01.304 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.099443+0000", 2024-03-20T09:40:01.304 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.823e-06 2024-03-20T09:40:01.304 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.304 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.304 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:40:01.304 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.099475+0000", 2024-03-20T09:40:01.304 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1741e-05 2024-03-20T09:40:01.304 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.304 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.305 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:40:01.305 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.099556+0000", 2024-03-20T09:40:01.305 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1111000000000003e-05 2024-03-20T09:40:01.305 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.305 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.305 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,2", 2024-03-20T09:40:01.305 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.099644+0000", 2024-03-20T09:40:01.305 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7913999999999999e-05 2024-03-20T09:40:01.305 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.305 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.305 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:40:01.305 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.100305+0000", 2024-03-20T09:40:01.305 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00066120899999999995 2024-03-20T09:40:01.305 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.305 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.306 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:01.306 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.100731+0000", 2024-03-20T09:40:01.306 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00042609100000000002 2024-03-20T09:40:01.306 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.306 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.306 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:01.306 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.287215+0000", 2024-03-20T09:40:01.306 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18648332500000001 2024-03-20T09:40:01.306 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.306 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.306 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:40:01.306 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.287275+0000", 2024-03-20T09:40:01.306 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0618000000000001e-05 2024-03-20T09:40:01.306 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.307 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.307 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:40:01.307 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.287323+0000", 2024-03-20T09:40:01.307 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7703000000000001e-05 2024-03-20T09:40:01.307 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.307 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:40:01.307 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.307 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.307 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.307 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:238411 8.1 8:86d760dd:::benchmark_data_smithi122_197775_object115107:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1056)", 2024-03-20T09:40:01.307 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:39:55.048854+0000", 2024-03-20T09:40:01.307 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 6.1637228259999999, 2024-03-20T09:40:01.307 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.216121966, 2024-03-20T09:40:01.308 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:40:01.308 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:40:01.308 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:40:01.308 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:40:01.308 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:40:01.308 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 238411 2024-03-20T09:40:01.308 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.308 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:40:01.308 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.308 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:40:01.308 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.048854+0000", 2024-03-20T09:40:01.308 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.308 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.308 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.308 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:40:01.308 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.048854+0000", 2024-03-20T09:40:01.309 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.309 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.309 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.309 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:40:01.309 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.048855+0000", 2024-03-20T09:40:01.309 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7500000000000001e-07 2024-03-20T09:40:01.309 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.309 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.309 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:40:01.309 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.048858+0000", 2024-03-20T09:40:01.309 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2200000000000001e-06 2024-03-20T09:40:01.309 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.309 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.309 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:40:01.310 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.048864+0000", 2024-03-20T09:40:01.310 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.421e-06 2024-03-20T09:40:01.310 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.310 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.310 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:40:01.310 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.048870+0000", 2024-03-20T09:40:01.310 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4e-06 2024-03-20T09:40:01.310 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.310 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.310 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:40:01.310 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.053672+0000", 2024-03-20T09:40:01.310 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0048026240000000001 2024-03-20T09:40:01.310 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.310 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.311 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:40:01.311 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.053784+0000", 2024-03-20T09:40:01.311 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000111839 2024-03-20T09:40:01.311 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.311 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.311 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,2", 2024-03-20T09:40:01.311 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.053854+0000", 2024-03-20T09:40:01.311 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9915999999999999e-05 2024-03-20T09:40:01.311 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.311 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.311 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:40:01.311 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.057950+0000", 2024-03-20T09:40:01.311 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040960759999999997 2024-03-20T09:40:01.311 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.312 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.312 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:01.312 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.061752+0000", 2024-03-20T09:40:01.312 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0038016019999999998 2024-03-20T09:40:01.312 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.312 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.312 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:01.312 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.264887+0000", 2024-03-20T09:40:01.312 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.203135396 2024-03-20T09:40:01.312 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.312 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.312 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:40:01.312 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.264938+0000", 2024-03-20T09:40:01.312 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0494000000000001e-05 2024-03-20T09:40:01.312 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.312 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.313 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:40:01.313 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.264976+0000", 2024-03-20T09:40:01.313 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8603000000000003e-05 2024-03-20T09:40:01.313 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.313 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:40:01.313 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.313 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.313 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.313 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:238427 8.1 8:8f1c7409:::benchmark_data_smithi122_197775_object115123:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1056)", 2024-03-20T09:40:01.313 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:39:55.066135+0000", 2024-03-20T09:40:01.313 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 6.146442392, 2024-03-20T09:40:01.313 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.21237999900000001, 2024-03-20T09:40:01.313 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:40:01.313 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:40:01.314 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:40:01.314 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:40:01.314 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:40:01.314 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 238427 2024-03-20T09:40:01.314 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.314 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:40:01.314 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.314 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:40:01.314 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.066135+0000", 2024-03-20T09:40:01.314 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.314 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.314 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.314 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:40:01.314 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.066135+0000", 2024-03-20T09:40:01.315 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.315 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.315 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.315 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:40:01.315 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.066136+0000", 2024-03-20T09:40:01.315 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.519e-06 2024-03-20T09:40:01.315 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.315 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.315 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:40:01.315 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.066142+0000", 2024-03-20T09:40:01.315 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1140000000000002e-06 2024-03-20T09:40:01.315 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.315 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.315 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:40:01.315 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.066154+0000", 2024-03-20T09:40:01.316 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1418e-05 2024-03-20T09:40:01.316 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.316 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.316 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:40:01.316 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.066162+0000", 2024-03-20T09:40:01.316 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2570000000000007e-06 2024-03-20T09:40:01.316 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.316 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.316 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:40:01.316 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.066228+0000", 2024-03-20T09:40:01.316 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5634000000000002e-05 2024-03-20T09:40:01.316 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.316 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.316 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:40:01.316 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.066360+0000", 2024-03-20T09:40:01.316 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000132352 2024-03-20T09:40:01.317 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.317 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.317 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,2", 2024-03-20T09:40:01.317 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.066467+0000", 2024-03-20T09:40:01.317 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000106449 2024-03-20T09:40:01.317 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.317 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.317 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:40:01.317 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.071716+0000", 2024-03-20T09:40:01.317 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0052489870000000001 2024-03-20T09:40:01.317 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.317 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.317 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:01.317 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.080421+0000", 2024-03-20T09:40:01.317 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0087058599999999993 2024-03-20T09:40:01.317 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.318 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.318 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:01.318 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.278418+0000", 2024-03-20T09:40:01.318 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.197996124 2024-03-20T09:40:01.318 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.318 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.318 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:40:01.318 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.278471+0000", 2024-03-20T09:40:01.318 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3458000000000002e-05 2024-03-20T09:40:01.318 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.318 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.318 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:40:01.318 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.278515+0000", 2024-03-20T09:40:01.318 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3826999999999999e-05 2024-03-20T09:40:01.318 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.318 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:40:01.319 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.319 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.319 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.319 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:238445 8.1 8:848892f9:::benchmark_data_smithi122_197775_object115141:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1056)", 2024-03-20T09:40:01.319 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:39:55.088192+0000", 2024-03-20T09:40:01.319 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 6.1243852069999996, 2024-03-20T09:40:01.319 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.190447121, 2024-03-20T09:40:01.319 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:40:01.319 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:40:01.319 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:40:01.319 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:40:01.319 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:40:01.319 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 238445 2024-03-20T09:40:01.319 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.319 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:40:01.319 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.320 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:40:01.320 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.088192+0000", 2024-03-20T09:40:01.320 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.320 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.320 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.320 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:40:01.320 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.088192+0000", 2024-03-20T09:40:01.320 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:40:01.320 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.320 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.320 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:40:01.320 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.088194+0000", 2024-03-20T09:40:01.320 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.584e-06 2024-03-20T09:40:01.320 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.320 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.321 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:40:01.321 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.088201+0000", 2024-03-20T09:40:01.321 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0559999999999999e-06 2024-03-20T09:40:01.321 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.321 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.321 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:40:01.321 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.088216+0000", 2024-03-20T09:40:01.321 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.504e-05 2024-03-20T09:40:01.321 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.321 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.321 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:40:01.321 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.088224+0000", 2024-03-20T09:40:01.321 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3340000000000002e-06 2024-03-20T09:40:01.321 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.321 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.322 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:40:01.322 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.088291+0000", 2024-03-20T09:40:01.322 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7389000000000001e-05 2024-03-20T09:40:01.322 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.322 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.322 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:40:01.322 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.088420+0000", 2024-03-20T09:40:01.322 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012860200000000001 2024-03-20T09:40:01.322 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.322 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.322 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,2", 2024-03-20T09:40:01.322 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.088538+0000", 2024-03-20T09:40:01.322 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011762 2024-03-20T09:40:01.322 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.322 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.323 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:01.323 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.093432+0000", 2024-03-20T09:40:01.323 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0048940099999999999 2024-03-20T09:40:01.323 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.323 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.323 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:40:01.323 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.093996+0000", 2024-03-20T09:40:01.323 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00056413000000000004 2024-03-20T09:40:01.323 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.323 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.323 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:01.323 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.278552+0000", 2024-03-20T09:40:01.323 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18455661600000001 2024-03-20T09:40:01.323 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.324 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.324 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:40:01.324 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.278602+0000", 2024-03-20T09:40:01.324 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0120000000000001e-05 2024-03-20T09:40:01.324 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:40:01.324 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:40:01.324 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:40:01.324 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.278639+0000", 2024-03-20T09:40:01.324 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6619999999999998e-05 2024-03-20T09:40:01.324 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.324 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:40:01.324 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.324 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:40:01.324 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:40:01.324 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:40:01.325 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:40:01.326 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:01 smithi026 ceph-mon[68309]: overall HEALTH_OK 2024-03-20T09:40:01.326 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:01 smithi026 ceph-mon[68309]: 1.3 scrub starts 2024-03-20T09:40:01.326 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:01 smithi026 ceph-mon[68309]: pgmap v2559: 47 pgs: 47 active+clean; 376 MiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 723 op/s 2024-03-20T09:40:01.326 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:01 smithi026 ceph-mon[68309]: 1.16 scrub starts 2024-03-20T09:40:01.371 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:01.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:01 smithi115 ceph-mon[72476]: overall HEALTH_OK 2024-03-20T09:40:01.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:01 smithi115 ceph-mon[72476]: 1.3 scrub starts 2024-03-20T09:40:01.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:01 smithi115 ceph-mon[72476]: pgmap v2559: 47 pgs: 47 active+clean; 376 MiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 723 op/s 2024-03-20T09:40:01.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:01 smithi115 ceph-mon[72476]: 1.16 scrub starts 2024-03-20T09:40:01.435 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:40:01.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:01 smithi122 ceph-mon[73495]: overall HEALTH_OK 2024-03-20T09:40:01.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:01 smithi122 ceph-mon[73495]: 1.3 scrub starts 2024-03-20T09:40:01.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:01 smithi122 ceph-mon[73495]: pgmap v2559: 47 pgs: 47 active+clean; 376 MiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 723 op/s 2024-03-20T09:40:01.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:01 smithi122 ceph-mon[73495]: 1.16 scrub starts 2024-03-20T09:40:01.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:01.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:40:01.540 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:40:01.575 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:01.984 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_ops_in_flight 2024-03-20T09:40:02.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:02 smithi026 ceph-mon[68309]: 1.3 scrub ok 2024-03-20T09:40:02.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:02 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:40:02.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:02 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:40:02.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:02 smithi026 ceph-mon[68309]: osdmap e1058: 12 total, 12 up, 11 in 2024-03-20T09:40:02.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:02 smithi026 ceph-mon[68309]: 1.16 scrub ok 2024-03-20T09:40:02.236 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:40:02.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:02.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:40:02.344 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:40:02.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:02 smithi115 ceph-mon[72476]: 1.3 scrub ok 2024-03-20T09:40:02.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:02 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:40:02.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:02 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:40:02.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:02 smithi115 ceph-mon[72476]: osdmap e1058: 12 total, 12 up, 11 in 2024-03-20T09:40:02.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:02 smithi115 ceph-mon[72476]: 1.16 scrub ok 2024-03-20T09:40:02.401 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:02 smithi122 ceph-mon[73495]: 1.3 scrub ok 2024-03-20T09:40:02.401 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:02 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:40:02.401 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:02 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:40:02.401 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:02 smithi122 ceph-mon[73495]: osdmap e1058: 12 total, 12 up, 11 in 2024-03-20T09:40:02.401 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:02 smithi122 ceph-mon[73495]: 1.16 scrub ok 2024-03-20T09:40:02.401 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:02.414 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:40:02.447 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:40:02.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:02.457 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:40:02.558 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:40:02.585 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:02.608 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:40:02.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:02.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:40:02.711 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:40:02.789 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:02.992 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:40:03.128 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:40:03.146 INFO:teuthology.orchestra.run.smithi026.stderr:no valid command found; 10 closest matches: 2024-03-20T09:40:03.147 INFO:teuthology.orchestra.run.smithi026.stderr:0 2024-03-20T09:40:03.147 INFO:teuthology.orchestra.run.smithi026.stderr:1 2024-03-20T09:40:03.147 INFO:teuthology.orchestra.run.smithi026.stderr:2 2024-03-20T09:40:03.147 INFO:teuthology.orchestra.run.smithi026.stderr:abort 2024-03-20T09:40:03.147 INFO:teuthology.orchestra.run.smithi026.stderr:assert 2024-03-20T09:40:03.147 INFO:teuthology.orchestra.run.smithi026.stderr:bench [] [] [] [] 2024-03-20T09:40:03.147 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs debug_inject_read_zeros 2024-03-20T09:40:03.147 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs files list 2024-03-20T09:40:03.147 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs stats 2024-03-20T09:40:03.147 INFO:teuthology.orchestra.run.smithi026.stderr:bluestore allocator dump block 2024-03-20T09:40:03.147 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: invalid command 2024-03-20T09:40:03.166 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:40:03.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:03 smithi122 ceph-mon[73495]: 8.c scrub starts 2024-03-20T09:40:03.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:03 smithi122 ceph-mon[73495]: 8.c scrub ok 2024-03-20T09:40:03.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:03 smithi122 ceph-mon[73495]: pgmap v2561: 47 pgs: 47 active+clean; 268 MiB data, 8.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 819 op/s 2024-03-20T09:40:03.212 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:40:03.261 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:40:03.306 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:40:03.339 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:40:03.365 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:40:03.392 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:40:03.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:03 smithi115 ceph-mon[72476]: 8.c scrub starts 2024-03-20T09:40:03.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:03 smithi115 ceph-mon[72476]: 8.c scrub ok 2024-03-20T09:40:03.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:03 smithi115 ceph-mon[72476]: pgmap v2561: 47 pgs: 47 active+clean; 268 MiB data, 8.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 819 op/s 2024-03-20T09:40:03.441 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:40:03.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:03 smithi026 ceph-mon[68309]: 8.c scrub starts 2024-03-20T09:40:03.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:03 smithi026 ceph-mon[68309]: 8.c scrub ok 2024-03-20T09:40:03.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:03 smithi026 ceph-mon[68309]: pgmap v2561: 47 pgs: 47 active+clean; 268 MiB data, 8.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 819 op/s 2024-03-20T09:40:03.486 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:40:03.516 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:40:03.562 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:40:03.566 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:40:03.567 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:40:03.567 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:40:03.567 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:40:03.567 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:40:03.567 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:40:03.567 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:40:03.567 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:40:03.567 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:40:03.567 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:40:03.567 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:40:03.567 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:40:03.661 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:40:03.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:03.670 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:40:03.771 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:40:03.802 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:04.014 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:40:04.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:04.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:40:04.036 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T09:40:04.036 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-20T09:40:04.037 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.7 'trim stale osdmaps' 2024-03-20T09:40:04.122 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:40:04.152 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:04.176 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:40:04.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:04.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:40:04.282 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:40:04.312 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:04 smithi115 ceph-mon[72476]: 8.5 deep-scrub starts 2024-03-20T09:40:04.312 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:04 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3427195025' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:40:04.312 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:04 smithi115 ceph-mon[72476]: 8.1 scrub starts 2024-03-20T09:40:04.312 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:04 smithi115 ceph-mon[72476]: 8.1 scrub ok 2024-03-20T09:40:04.312 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:04 smithi115 ceph-mon[72476]: 8.4 deep-scrub starts 2024-03-20T09:40:04.314 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:04.336 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:40:04.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:04.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:40:04.405 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:04 smithi026 ceph-mon[68309]: 8.5 deep-scrub starts 2024-03-20T09:40:04.406 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:04 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3427195025' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:40:04.406 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:04 smithi026 ceph-mon[68309]: 8.1 scrub starts 2024-03-20T09:40:04.406 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:04 smithi026 ceph-mon[68309]: 8.1 scrub ok 2024-03-20T09:40:04.406 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:04 smithi026 ceph-mon[68309]: 8.4 deep-scrub starts 2024-03-20T09:40:04.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:04 smithi122 ceph-mon[73495]: 8.5 deep-scrub starts 2024-03-20T09:40:04.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:04 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3427195025' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:40:04.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:04 smithi122 ceph-mon[73495]: 8.1 scrub starts 2024-03-20T09:40:04.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:04 smithi122 ceph-mon[73495]: 8.1 scrub ok 2024-03-20T09:40:04.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:04 smithi122 ceph-mon[73495]: 8.4 deep-scrub starts 2024-03-20T09:40:04.440 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:40:04.503 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:04.799 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:40:05.076 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:05 smithi026 ceph-mon[68309]: 8.5 deep-scrub ok 2024-03-20T09:40:05.076 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:05 smithi026 ceph-mon[68309]: 8.4 deep-scrub ok 2024-03-20T09:40:05.076 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:05 smithi026 ceph-mon[68309]: pgmap v2562: 47 pgs: 47 active+clean; 183 MiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 722 op/s 2024-03-20T09:40:05.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:05 smithi115 ceph-mon[72476]: 8.5 deep-scrub ok 2024-03-20T09:40:05.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:05 smithi115 ceph-mon[72476]: 8.4 deep-scrub ok 2024-03-20T09:40:05.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:05 smithi115 ceph-mon[72476]: pgmap v2562: 47 pgs: 47 active+clean; 183 MiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 722 op/s 2024-03-20T09:40:05.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:05 smithi122 ceph-mon[73495]: 8.5 deep-scrub ok 2024-03-20T09:40:05.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:05 smithi122 ceph-mon[73495]: 8.4 deep-scrub ok 2024-03-20T09:40:05.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:05 smithi122 ceph-mon[73495]: pgmap v2562: 47 pgs: 47 active+clean; 183 MiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 722 op/s 2024-03-20T09:40:05.439 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:40:05.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:05.444 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:40:05.545 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:40:05.583 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:05.618 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:40:05.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:05.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:40:05.723 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:40:05.767 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:05.997 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Removed 123301 objects 2024-03-20T09:40:05.997 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Clean up completed and total clean up time :156.201 2024-03-20T09:40:06.152 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:40:06.172 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:40:06.172 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:40:06.625 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:40:06.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:06.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:40:06.729 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:40:06.759 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:06.824 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_blocked_ops 2024-03-20T09:40:06.840 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:40:06.884 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:40:06.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:06.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:40:06.989 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:40:07.021 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:07.085 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:40:07.691 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:40:07.780 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:40:07.783 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:07.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:40:07.884 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:40:07.915 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:07.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:07 smithi026 ceph-mon[68309]: pgmap v2563: 47 pgs: 47 active+clean; 70 MiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 799 op/s 2024-03-20T09:40:07.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:07 smithi026 ceph-mon[68309]: 1.e scrub starts 2024-03-20T09:40:07.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:07 smithi026 ceph-mon[68309]: 8.6 scrub starts 2024-03-20T09:40:07.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:07 smithi026 ceph-mon[68309]: 8.6 scrub ok 2024-03-20T09:40:07.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:07 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1005041648' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:40:07.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:07 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:40:07.968 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:40:07.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:07.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:40:08.072 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:40:08.102 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:07 smithi115 ceph-mon[72476]: pgmap v2563: 47 pgs: 47 active+clean; 70 MiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 799 op/s 2024-03-20T09:40:08.102 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:07 smithi115 ceph-mon[72476]: 1.e scrub starts 2024-03-20T09:40:08.103 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:07 smithi115 ceph-mon[72476]: 8.6 scrub starts 2024-03-20T09:40:08.103 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:07 smithi115 ceph-mon[72476]: 8.6 scrub ok 2024-03-20T09:40:08.103 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:07 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1005041648' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:40:08.103 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:07 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:40:08.104 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:08.125 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:40:08.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:08.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:40:08.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:07 smithi122 ceph-mon[73495]: pgmap v2563: 47 pgs: 47 active+clean; 70 MiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 799 op/s 2024-03-20T09:40:08.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:07 smithi122 ceph-mon[73495]: 1.e scrub starts 2024-03-20T09:40:08.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:07 smithi122 ceph-mon[73495]: 8.6 scrub starts 2024-03-20T09:40:08.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:07 smithi122 ceph-mon[73495]: 8.6 scrub ok 2024-03-20T09:40:08.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:07 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1005041648' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:40:08.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:07 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:40:08.229 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:40:08.257 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:08.381 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:40:08.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:08.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:40:08.401 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:40:08.487 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:40:08.514 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:08.663 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:40:08.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:08.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:40:08.767 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:40:08.778 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:40:08.778 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:40:08.795 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:08 smithi122 ceph-mon[73495]: 1.e scrub ok 2024-03-20T09:40:08.795 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:08 smithi122 ceph-mon[73495]: 1.9 scrub starts 2024-03-20T09:40:08.795 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:08 smithi122 ceph-mon[73495]: 1.9 scrub ok 2024-03-20T09:40:08.795 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:08 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:40:08.795 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:08 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:40:08.795 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:08 smithi122 ceph-mon[73495]: osdmap e1059: 12 total, 12 up, 11 in 2024-03-20T09:40:08.797 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:08.816 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:40:08.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:08.820 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:40:08.870 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:08 smithi026 ceph-mon[68309]: 1.e scrub ok 2024-03-20T09:40:08.870 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:08 smithi026 ceph-mon[68309]: 1.9 scrub starts 2024-03-20T09:40:08.870 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:08 smithi026 ceph-mon[68309]: 1.9 scrub ok 2024-03-20T09:40:08.870 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:08 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:40:08.871 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:08 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:40:08.871 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:08 smithi026 ceph-mon[68309]: osdmap e1059: 12 total, 12 up, 11 in 2024-03-20T09:40:08.921 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:40:08.951 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:08 smithi115 ceph-mon[72476]: 1.e scrub ok 2024-03-20T09:40:08.951 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:08 smithi115 ceph-mon[72476]: 1.9 scrub starts 2024-03-20T09:40:08.951 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:08 smithi115 ceph-mon[72476]: 1.9 scrub ok 2024-03-20T09:40:08.952 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:08 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:40:08.952 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:08 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:40:08.952 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:08 smithi115 ceph-mon[72476]: osdmap e1059: 12 total, 12 up, 11 in 2024-03-20T09:40:08.954 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:08.975 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:40:08.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:08.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:40:09.079 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:40:09.107 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:09.431 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_historic_ops 2024-03-20T09:40:09.455 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:40:09.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:09.458 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:40:09.505 INFO:teuthology.orchestra.run.smithi026.stderr:no valid command found; 10 closest matches: 2024-03-20T09:40:09.505 INFO:teuthology.orchestra.run.smithi026.stderr:0 2024-03-20T09:40:09.505 INFO:teuthology.orchestra.run.smithi026.stderr:1 2024-03-20T09:40:09.505 INFO:teuthology.orchestra.run.smithi026.stderr:2 2024-03-20T09:40:09.505 INFO:teuthology.orchestra.run.smithi026.stderr:abort 2024-03-20T09:40:09.505 INFO:teuthology.orchestra.run.smithi026.stderr:assert 2024-03-20T09:40:09.505 INFO:teuthology.orchestra.run.smithi026.stderr:bench [] [] [] [] 2024-03-20T09:40:09.505 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs debug_inject_read_zeros 2024-03-20T09:40:09.506 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs files list 2024-03-20T09:40:09.506 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs stats 2024-03-20T09:40:09.506 INFO:teuthology.orchestra.run.smithi026.stderr:bluestore allocator dump block 2024-03-20T09:40:09.506 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: invalid command 2024-03-20T09:40:09.558 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:40:09.585 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:09.603 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:40:09.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:09.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:40:09.707 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:40:09.720 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:40:09.734 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:09.753 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:40:09.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:09.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:40:09.857 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:40:09.867 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_6 2024-03-20T09:40:09.868 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pool rm unique_pool_6 unique_pool_6 --yes-i-really-really-mean-it 2024-03-20T09:40:09.880 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:09 smithi115 ceph-mon[72476]: pgmap v2565: 47 pgs: 47 active+clean; 28 MiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 695 op/s 2024-03-20T09:40:09.880 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:09 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:40:09.882 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:09.894 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:09 smithi026 ceph-mon[68309]: pgmap v2565: 47 pgs: 47 active+clean; 28 MiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 695 op/s 2024-03-20T09:40:09.894 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:09 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:40:09.901 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:40:09.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:09.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:40:09.975 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:09 smithi122 ceph-mon[73495]: pgmap v2565: 47 pgs: 47 active+clean; 28 MiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 695 op/s 2024-03-20T09:40:09.975 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:09 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:40:10.005 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:40:10.031 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:10.049 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:40:10.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:10.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:40:10.153 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:40:10.180 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:10.403 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:40:10.652 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T09:40:10.652 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-20T09:40:10.653 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.8 'trim stale osdmaps' 2024-03-20T09:40:10.693 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:40:10.828 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:40:10.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:10.831 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:40:10.926 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:40:10.932 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:40:10.935 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:10 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:40:10.935 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:10 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:40:10.935 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:10 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:40:10.935 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:10 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:40:10.935 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:10 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:40:10.950 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:10 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:40:10.950 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:10 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:40:10.950 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:10 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:40:10.950 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:10 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:40:10.950 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:10 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:40:10.963 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:10 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:40:10.963 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:10 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:40:10.963 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:10 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:40:10.963 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:10 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:40:10.963 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:10 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:40:10.965 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:10.987 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:40:10.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:10.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:40:11.092 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:40:11.121 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:11.175 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:40:11.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:11.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:40:11.278 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:40:11.311 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:11.333 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:40:11.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:11.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:40:11.394 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:40:11.395 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:40:11.437 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:40:11.467 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:11.487 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:40:11.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:11.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:40:11.591 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:40:11.619 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:11.638 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:40:11.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:11.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:40:11.742 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:40:11.771 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:11.894 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:11 smithi115 ceph-mon[72476]: pgmap v2566: 47 pgs: 47 active+clean; 3.6 MiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 488 op/s 2024-03-20T09:40:11.894 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:11 smithi115 ceph-mon[72476]: osdmap e1060: 12 total, 12 up, 11 in 2024-03-20T09:40:11.896 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:40:11.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:11.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:40:12.000 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:40:12.025 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:11 smithi122 ceph-mon[73495]: pgmap v2566: 47 pgs: 47 active+clean; 3.6 MiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 488 op/s 2024-03-20T09:40:12.025 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:11 smithi122 ceph-mon[73495]: osdmap e1060: 12 total, 12 up, 11 in 2024-03-20T09:40:12.034 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:12.054 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:40:12.058 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-20T09:40:12.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:12.059 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:40:12.160 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:40:12.187 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:11 smithi026 ceph-mon[68309]: pgmap v2566: 47 pgs: 47 active+clean; 3.6 MiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 488 op/s 2024-03-20T09:40:12.187 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:11 smithi026 ceph-mon[68309]: osdmap e1060: 12 total, 12 up, 11 in 2024-03-20T09:40:12.190 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:12.291 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:40:12.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:12.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:40:12.321 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:40:12.401 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:40:12.429 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:12.759 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:40:12.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:12.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:40:12.865 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:40:12.894 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:12 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:40:12.894 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:12 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:40:12.896 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:12.920 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:40:12.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:12.923 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:40:13.024 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:40:13.054 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:13.128 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:12 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:40:13.128 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:12 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:40:13.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:12 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:40:13.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:12 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:40:13.202 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:40:13.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:13.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:40:13.306 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:40:13.333 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:13.627 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:40:13.627 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:40:13.627 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:40:13.627 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:40:13.627 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:40:13.627 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:40:13.627 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:40:13.627 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:40:13.627 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:40:13.627 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:40:13.628 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:40:13.628 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:40:13.628 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:40:13.628 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:40:13.628 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:40:13.628 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:40:13.628 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:40:13.628 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:40:13.628 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:40:13.628 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:40:13.628 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:40:13.628 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:40:13.629 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:40:13.629 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:40:13.681 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:40:13.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:13.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:40:13.785 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:40:13.817 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:13.839 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:40:13.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:13.843 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:40:13.885 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:13 smithi115 ceph-mon[72476]: pgmap v2568: 47 pgs: 47 active+clean; 577 KiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 365 op/s 2024-03-20T09:40:13.886 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:13 smithi115 ceph-mon[72476]: osdmap e1061: 12 total, 12 up, 11 in 2024-03-20T09:40:13.886 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:13 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/297276639' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:40:13.943 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:40:13.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:13 smithi026 ceph-mon[68309]: pgmap v2568: 47 pgs: 47 active+clean; 577 KiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 365 op/s 2024-03-20T09:40:13.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:13 smithi026 ceph-mon[68309]: osdmap e1061: 12 total, 12 up, 11 in 2024-03-20T09:40:13.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:13 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/297276639' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:40:13.974 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:13.995 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:40:13.997 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:13 smithi122 ceph-mon[73495]: pgmap v2568: 47 pgs: 47 active+clean; 577 KiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 365 op/s 2024-03-20T09:40:13.997 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:13 smithi122 ceph-mon[73495]: osdmap e1061: 12 total, 12 up, 11 in 2024-03-20T09:40:13.997 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:13 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/297276639' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:40:13.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:13.998 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:40:14.099 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:40:14.128 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:14.148 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:40:14.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:14.152 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:40:14.171 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:40:14.172 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:40:14.172 INFO:teuthology.orchestra.run.smithi115.stdout: "num_ops": 0 2024-03-20T09:40:14.172 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:40:14.252 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:40:14.280 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:14.299 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:40:14.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:14.303 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:40:14.403 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:40:14.435 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:14.533 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:40:14.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:14.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:40:14.638 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:40:14.667 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:14.686 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:40:14.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:14.689 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:40:14.737 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'unique_pool_6' removed 2024-03-20T09:40:14.767 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:40:14.789 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:40:14.815 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:14 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/979380878' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-20T09:40:14.818 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:14.838 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:40:14.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:14.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:40:14.854 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-20T09:40:14.934 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:14 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/979380878' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-20T09:40:14.942 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:40:14.967 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:14.986 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:40:14.988 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:14.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:40:15.090 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:40:15.116 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:15.134 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:40:15.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:15.137 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:40:15.156 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:40:15.197 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:14 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/979380878' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-20T09:40:15.238 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:40:15.267 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:15.745 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:40:16.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:15 smithi115 ceph-mon[72476]: pgmap v2570: 47 pgs: 47 active+clean; 577 KiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 105 op/s 2024-03-20T09:40:16.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:15 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/979380878' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-03-20T09:40:16.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:15 smithi115 ceph-mon[72476]: osdmap e1062: 12 total, 12 up, 11 in 2024-03-20T09:40:16.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:15 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2789993609' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:40:16.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:15 smithi122 ceph-mon[73495]: pgmap v2570: 47 pgs: 47 active+clean; 577 KiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 105 op/s 2024-03-20T09:40:16.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:15 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/979380878' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-03-20T09:40:16.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:15 smithi122 ceph-mon[73495]: osdmap e1062: 12 total, 12 up, 11 in 2024-03-20T09:40:16.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:15 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2789993609' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:40:16.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:15 smithi026 ceph-mon[68309]: pgmap v2570: 47 pgs: 47 active+clean; 577 KiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 105 op/s 2024-03-20T09:40:16.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:15 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/979380878' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-03-20T09:40:16.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:15 smithi026 ceph-mon[68309]: osdmap e1062: 12 total, 12 up, 11 in 2024-03-20T09:40:16.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:15 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2789993609' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:40:16.426 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:40:16.891 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:40:16.891 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:40:16.891 INFO:teuthology.orchestra.run.smithi115.stdout: "complaint_time": 30, 2024-03-20T09:40:16.891 INFO:teuthology.orchestra.run.smithi115.stdout: "num_blocked_ops": 0 2024-03-20T09:40:16.891 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:40:16.894 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:16 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:40:16.894 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:16 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2789993609' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:40:16.894 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:16 smithi115 ceph-mon[72476]: osdmap e1063: 12 total, 12 up, 11 in 2024-03-20T09:40:16.894 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:16 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:40:16.894 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:16 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:40:17.170 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:16 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:40:17.170 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:16 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2789993609' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:40:17.170 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:16 smithi026 ceph-mon[68309]: osdmap e1063: 12 total, 12 up, 11 in 2024-03-20T09:40:17.170 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:16 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:40:17.170 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:16 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:40:17.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:16 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:40:17.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:16 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2789993609' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:40:17.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:16 smithi122 ceph-mon[73495]: osdmap e1063: 12 total, 12 up, 11 in 2024-03-20T09:40:17.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:16 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:40:17.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:16 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:40:17.613 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-20T09:40:17.614 INFO:tasks.radosbench:Beginning radosbench... 2024-03-20T09:40:17.614 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_7 2024-03-20T09:40:17.614 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pool create unique_pool_7 16 2024-03-20T09:40:17.820 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:40:17.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:17.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:40:17.851 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:17 smithi026 ceph-mon[68309]: pgmap v2573: 31 pgs: 31 active+clean; 577 KiB data, 8.0 GiB used, 975 GiB / 983 GiB avail 2024-03-20T09:40:17.851 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:17 smithi026 ceph-mon[68309]: osdmap e1064: 12 total, 12 up, 11 in 2024-03-20T09:40:17.925 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:40:17.958 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:17.979 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:40:17.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:17.982 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:40:18.083 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:40:18.111 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:17 smithi122 ceph-mon[73495]: pgmap v2573: 31 pgs: 31 active+clean; 577 KiB data, 8.0 GiB used, 975 GiB / 983 GiB avail 2024-03-20T09:40:18.111 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:17 smithi122 ceph-mon[73495]: osdmap e1064: 12 total, 12 up, 11 in 2024-03-20T09:40:18.113 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:18.134 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:40:18.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:18.137 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:40:18.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:17 smithi115 ceph-mon[72476]: pgmap v2573: 31 pgs: 31 active+clean; 577 KiB data, 8.0 GiB used, 975 GiB / 983 GiB avail 2024-03-20T09:40:18.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:17 smithi115 ceph-mon[72476]: osdmap e1064: 12 total, 12 up, 11 in 2024-03-20T09:40:18.238 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:40:18.268 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:18.292 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:40:18.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:18.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:40:18.399 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:40:18.429 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:18.450 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:40:18.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:18.453 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:40:18.554 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:40:18.584 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:18.733 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-20T09:40:18.734 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:40:18.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:18.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:40:18.738 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:40:18.838 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:40:18.867 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:19.001 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:40:19.123 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:40:19.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:19.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:40:19.195 INFO:teuthology.orchestra.run.smithi026.stderr:no valid command found; 10 closest matches: 2024-03-20T09:40:19.195 INFO:teuthology.orchestra.run.smithi026.stderr:0 2024-03-20T09:40:19.195 INFO:teuthology.orchestra.run.smithi026.stderr:1 2024-03-20T09:40:19.196 INFO:teuthology.orchestra.run.smithi026.stderr:2 2024-03-20T09:40:19.196 INFO:teuthology.orchestra.run.smithi026.stderr:abort 2024-03-20T09:40:19.196 INFO:teuthology.orchestra.run.smithi026.stderr:assert 2024-03-20T09:40:19.196 INFO:teuthology.orchestra.run.smithi026.stderr:bench [] [] [] [] 2024-03-20T09:40:19.196 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs debug_inject_read_zeros 2024-03-20T09:40:19.196 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs files list 2024-03-20T09:40:19.196 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs stats 2024-03-20T09:40:19.196 INFO:teuthology.orchestra.run.smithi026.stderr:bluestore allocator dump block 2024-03-20T09:40:19.196 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: invalid command 2024-03-20T09:40:19.228 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:40:19.259 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:19.358 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:40:19.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:19.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:40:19.462 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:40:19.490 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:20.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:19 smithi115 ceph-mon[72476]: pgmap v2575: 31 pgs: 31 active+clean; 577 KiB data, 8.1 GiB used, 975 GiB / 983 GiB avail 2024-03-20T09:40:20.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:19 smithi122 ceph-mon[73495]: pgmap v2575: 31 pgs: 31 active+clean; 577 KiB data, 8.1 GiB used, 975 GiB / 983 GiB avail 2024-03-20T09:40:20.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:19 smithi026 ceph-mon[68309]: pgmap v2575: 31 pgs: 31 active+clean; 577 KiB data, 8.1 GiB used, 975 GiB / 983 GiB avail 2024-03-20T09:40:20.238 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:40:20.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:20.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:40:20.326 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:40:20.345 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:40:20.372 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:20.374 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T09:40:20.375 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-20T09:40:20.375 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.11 'trim stale osdmaps' 2024-03-20T09:40:20.409 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:40:20.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:20.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:40:20.433 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:40:20.513 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:40:20.546 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:20.567 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:40:20.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:20.571 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:40:20.672 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:40:20.693 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:40:20.693 INFO:teuthology.orchestra.run.smithi115.stdout: "size": 20, 2024-03-20T09:40:20.693 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 600, 2024-03-20T09:40:20.693 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [ 2024-03-20T09:40:20.693 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.693 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:100031 7.0 7:0fe7022e:::benchmark_data_smithi122_187077_object100030:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e898)", 2024-03-20T09:40:20.693 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:32:19.515521+0000", 2024-03-20T09:40:20.693 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 481.18646631500002, 2024-03-20T09:40:20.693 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.220244419, 2024-03-20T09:40:20.694 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:40:20.694 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:40:20.694 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:40:20.694 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:40:20.694 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:40:20.694 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 100031 2024-03-20T09:40:20.694 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.694 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:40:20.694 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.694 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:40:20.694 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:19.515521+0000", 2024-03-20T09:40:20.694 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.694 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.695 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.695 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:40:20.695 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:19.515521+0000", 2024-03-20T09:40:20.695 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.695 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.695 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.695 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:40:20.695 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:19.515522+0000", 2024-03-20T09:40:20.695 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.243e-06 2024-03-20T09:40:20.695 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.695 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.695 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:40:20.695 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:19.515585+0000", 2024-03-20T09:40:20.695 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2323000000000006e-05 2024-03-20T09:40:20.696 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.696 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.696 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:40:20.696 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:19.515596+0000", 2024-03-20T09:40:20.696 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1015e-05 2024-03-20T09:40:20.696 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.696 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.696 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:40:20.696 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:19.515603+0000", 2024-03-20T09:40:20.696 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.5009999999999998e-06 2024-03-20T09:40:20.696 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.696 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.696 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:40:20.696 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:19.515648+0000", 2024-03-20T09:40:20.697 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5161999999999999e-05 2024-03-20T09:40:20.697 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.697 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.697 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:40:20.697 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:19.515762+0000", 2024-03-20T09:40:20.697 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00011344399999999999 2024-03-20T09:40:20.697 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.697 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.697 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 3,9", 2024-03-20T09:40:20.697 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:19.515863+0000", 2024-03-20T09:40:20.697 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000101523 2024-03-20T09:40:20.697 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.697 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.698 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:40:20.698 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:19.518652+0000", 2024-03-20T09:40:20.698 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.002788249 2024-03-20T09:40:20.698 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.698 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.698 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.698 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:19.518750+0000", 2024-03-20T09:40:20.698 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.7994999999999999e-05 2024-03-20T09:40:20.698 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.698 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.698 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.698 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:19.735661+0000", 2024-03-20T09:40:20.698 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.216911351 2024-03-20T09:40:20.699 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.699 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.699 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:40:20.699 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:19.735717+0000", 2024-03-20T09:40:20.699 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6385000000000003e-05 2024-03-20T09:40:20.699 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.699 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.699 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:40:20.699 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:32:19.735766+0000", 2024-03-20T09:40:20.699 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8228e-05 2024-03-20T09:40:20.699 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.699 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:40:20.699 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.700 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.700 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.700 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26917.0:71673 8.4 8:201f9a55:::benchmark_data_smithi122_197775_object71672:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e995)", 2024-03-20T09:40:20.700 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:39.218332+0000", 2024-03-20T09:40:20.700 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 221.48365559999999, 2024-03-20T09:40:20.700 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.18304461599999999, 2024-03-20T09:40:20.700 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:40:20.700 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:40:20.700 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:40:20.700 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26917", 2024-03-20T09:40:20.700 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:40:20.700 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 71673 2024-03-20T09:40:20.700 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.701 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:40:20.701 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.701 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:40:20.701 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.218332+0000", 2024-03-20T09:40:20.701 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.701 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.701 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.701 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:40:20.701 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.218332+0000", 2024-03-20T09:40:20.701 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.701 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.701 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.701 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:40:20.701 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.218333+0000", 2024-03-20T09:40:20.702 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.254e-06 2024-03-20T09:40:20.702 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.702 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.702 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:40:20.702 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.218391+0000", 2024-03-20T09:40:20.702 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7714999999999999e-05 2024-03-20T09:40:20.702 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.702 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.702 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:40:20.702 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.218400+0000", 2024-03-20T09:40:20.702 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.217e-06 2024-03-20T09:40:20.702 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.702 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.703 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:40:20.703 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.218407+0000", 2024-03-20T09:40:20.703 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.0879999999999997e-06 2024-03-20T09:40:20.703 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.703 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.703 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:40:20.703 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.218456+0000", 2024-03-20T09:40:20.703 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8973e-05 2024-03-20T09:40:20.703 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.703 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.703 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:40:20.703 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.218620+0000", 2024-03-20T09:40:20.703 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00016423300000000001 2024-03-20T09:40:20.703 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.704 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.704 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 9,10", 2024-03-20T09:40:20.704 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.218778+0000", 2024-03-20T09:40:20.704 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00015741099999999999 2024-03-20T09:40:20.704 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.704 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.704 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:40:20.704 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.220708+0000", 2024-03-20T09:40:20.704 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001930385 2024-03-20T09:40:20.704 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.704 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.704 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.704 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.221968+0000", 2024-03-20T09:40:20.704 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001259549 2024-03-20T09:40:20.704 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.705 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.705 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.705 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.401299+0000", 2024-03-20T09:40:20.705 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.17933179399999999 2024-03-20T09:40:20.705 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.705 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.705 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:40:20.705 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.401341+0000", 2024-03-20T09:40:20.705 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.1513e-05 2024-03-20T09:40:20.705 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.705 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.705 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:40:20.705 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.401376+0000", 2024-03-20T09:40:20.705 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.5484000000000001e-05 2024-03-20T09:40:20.705 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.706 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:40:20.706 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.706 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.706 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.706 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26917.0:71685 8.4 8:2bfcec03:::benchmark_data_smithi122_197775_object71684:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e995)", 2024-03-20T09:40:20.706 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:39.228848+0000", 2024-03-20T09:40:20.706 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 221.473138941, 2024-03-20T09:40:20.706 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.17282698899999999, 2024-03-20T09:40:20.706 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:40:20.706 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:40:20.706 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:40:20.706 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26917", 2024-03-20T09:40:20.706 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:40:20.706 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 71685 2024-03-20T09:40:20.706 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.707 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:40:20.707 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.707 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:40:20.707 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.228848+0000", 2024-03-20T09:40:20.707 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.707 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.707 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.707 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:40:20.707 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.228848+0000", 2024-03-20T09:40:20.707 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.707 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.707 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.707 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:40:20.707 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.228849+0000", 2024-03-20T09:40:20.708 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5899999999999996e-07 2024-03-20T09:40:20.708 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.708 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.708 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:40:20.708 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.228875+0000", 2024-03-20T09:40:20.708 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6030999999999999e-05 2024-03-20T09:40:20.708 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.708 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.708 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:40:20.708 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.228883+0000", 2024-03-20T09:40:20.708 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.1349999999999992e-06 2024-03-20T09:40:20.708 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.708 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.708 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:40:20.708 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.228889+0000", 2024-03-20T09:40:20.709 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2140000000000001e-06 2024-03-20T09:40:20.709 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.709 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.709 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:40:20.709 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.231223+0000", 2024-03-20T09:40:20.709 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0023334179999999999 2024-03-20T09:40:20.709 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.709 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.709 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:40:20.709 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.231338+0000", 2024-03-20T09:40:20.709 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000115443 2024-03-20T09:40:20.709 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.709 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.709 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 9,10", 2024-03-20T09:40:20.709 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.231433+0000", 2024-03-20T09:40:20.709 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.4357999999999993e-05 2024-03-20T09:40:20.710 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.710 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.710 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:40:20.710 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.232837+0000", 2024-03-20T09:40:20.710 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001404375 2024-03-20T09:40:20.710 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.710 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.710 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.710 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.233580+0000", 2024-03-20T09:40:20.710 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00074284199999999996 2024-03-20T09:40:20.710 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.710 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.710 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.711 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.401607+0000", 2024-03-20T09:40:20.711 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.16802669200000001 2024-03-20T09:40:20.711 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.711 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.711 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:40:20.711 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.401647+0000", 2024-03-20T09:40:20.711 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.0287000000000002e-05 2024-03-20T09:40:20.711 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.711 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.711 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:40:20.711 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:39.401675+0000", 2024-03-20T09:40:20.711 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8634999999999999e-05 2024-03-20T09:40:20.711 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.711 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:40:20.712 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.712 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.712 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.712 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26917.0:181753 8.8 8:161463d9:::benchmark_data_smithi122_197775_object58449:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1030)", 2024-03-20T09:40:20.712 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:38:38.101013+0000", 2024-03-20T09:40:20.712 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 102.60097460199999, 2024-03-20T09:40:20.712 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.045072041, 2024-03-20T09:40:20.712 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:40:20.712 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:40:20.712 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:40:20.712 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26917", 2024-03-20T09:40:20.712 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:40:20.712 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 181753 2024-03-20T09:40:20.712 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.713 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:40:20.713 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.713 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:40:20.713 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:38.101013+0000", 2024-03-20T09:40:20.713 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.713 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.713 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.713 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:40:20.713 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:38.101013+0000", 2024-03-20T09:40:20.713 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.713 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.713 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.713 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:40:20.713 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:38.101014+0000", 2024-03-20T09:40:20.713 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5000000000000001e-07 2024-03-20T09:40:20.714 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.714 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.714 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:40:20.714 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:38.101018+0000", 2024-03-20T09:40:20.714 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5499999999999996e-06 2024-03-20T09:40:20.714 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.714 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.714 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:40:20.714 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:38.101028+0000", 2024-03-20T09:40:20.714 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.8290000000000003e-06 2024-03-20T09:40:20.714 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.714 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.714 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:40:20.714 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:38.101034+0000", 2024-03-20T09:40:20.715 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.3709999999999996e-06 2024-03-20T09:40:20.715 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.715 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.715 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:40:20.715 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:38.102900+0000", 2024-03-20T09:40:20.715 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0018656289999999999 2024-03-20T09:40:20.715 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.715 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.715 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:40:20.715 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:38.103017+0000", 2024-03-20T09:40:20.715 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000116871 2024-03-20T09:40:20.715 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.715 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.715 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 3,9", 2024-03-20T09:40:20.716 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:38.103146+0000", 2024-03-20T09:40:20.716 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012906199999999999 2024-03-20T09:40:20.716 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.716 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.716 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:40:20.716 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:38.112135+0000", 2024-03-20T09:40:20.716 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0089889289999999997 2024-03-20T09:40:20.716 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.716 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.716 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.716 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:38.120413+0000", 2024-03-20T09:40:20.716 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0082777249999999997 2024-03-20T09:40:20.716 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.716 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.716 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.716 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:38.145975+0000", 2024-03-20T09:40:20.717 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.02556247 2024-03-20T09:40:20.717 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.717 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.717 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:40:20.717 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:38.146039+0000", 2024-03-20T09:40:20.717 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.3479999999999995e-05 2024-03-20T09:40:20.717 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.717 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.717 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:40:20.717 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:38.146085+0000", 2024-03-20T09:40:20.717 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6275e-05 2024-03-20T09:40:20.717 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.717 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:40:20.717 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.718 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.718 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.718 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26917.0:184572 8.4 8:29b8748a:::benchmark_data_smithi122_197775_object61268:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1031)", 2024-03-20T09:40:20.718 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:38:41.579577+0000", 2024-03-20T09:40:20.718 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 99.122410579000004, 2024-03-20T09:40:20.718 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.045825662000000003, 2024-03-20T09:40:20.718 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:40:20.718 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:40:20.718 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:40:20.718 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26917", 2024-03-20T09:40:20.718 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:40:20.718 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 184572 2024-03-20T09:40:20.718 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.718 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:40:20.719 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.719 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:40:20.719 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:41.579577+0000", 2024-03-20T09:40:20.719 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.719 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.719 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.719 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:40:20.719 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:41.579577+0000", 2024-03-20T09:40:20.719 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.719 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.719 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.719 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:40:20.719 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:41.579577+0000", 2024-03-20T09:40:20.719 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6899999999999998e-07 2024-03-20T09:40:20.720 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.720 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.720 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:40:20.720 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:41.579579+0000", 2024-03-20T09:40:20.720 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9520000000000001e-06 2024-03-20T09:40:20.720 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.720 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.720 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:40:20.720 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:41.579585+0000", 2024-03-20T09:40:20.720 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3499999999999996e-06 2024-03-20T09:40:20.720 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.720 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.720 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:40:20.721 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:41.579589+0000", 2024-03-20T09:40:20.721 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5360000000000003e-06 2024-03-20T09:40:20.721 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.721 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.721 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:40:20.721 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:41.587663+0000", 2024-03-20T09:40:20.721 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0080743369999999991 2024-03-20T09:40:20.721 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.721 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.721 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:40:20.721 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:41.587747+0000", 2024-03-20T09:40:20.721 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.3545000000000001e-05 2024-03-20T09:40:20.722 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.722 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.722 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 9,10", 2024-03-20T09:40:20.722 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:41.587854+0000", 2024-03-20T09:40:20.722 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00010666300000000001 2024-03-20T09:40:20.722 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.722 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.722 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:40:20.722 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:41.597158+0000", 2024-03-20T09:40:20.722 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0093047100000000008 2024-03-20T09:40:20.722 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.722 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.722 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.722 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:41.601927+0000", 2024-03-20T09:40:20.722 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0047691189999999996 2024-03-20T09:40:20.722 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.723 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.723 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.723 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:41.625297+0000", 2024-03-20T09:40:20.723 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.023369862000000002 2024-03-20T09:40:20.723 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.723 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.723 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:40:20.723 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:41.625356+0000", 2024-03-20T09:40:20.723 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8504000000000001e-05 2024-03-20T09:40:20.723 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.723 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.723 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:40:20.723 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:41.625402+0000", 2024-03-20T09:40:20.723 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6715000000000003e-05 2024-03-20T09:40:20.723 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.724 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:40:20.724 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.724 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.724 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.724 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:186729 8.3 e1031/974 8:c8df7d72:::benchmark_data_smithi122_197775_object63425:head v 1031'11789, mlcod=1031'11787)", 2024-03-20T09:40:20.724 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:38:44.285035+0000", 2024-03-20T09:40:20.724 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 96.416952601000006, 2024-03-20T09:40:20.724 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.059650697000000003, 2024-03-20T09:40:20.724 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:40:20.724 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:40:20.724 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:40:20.724 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.724 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:40:20.724 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:44.285035+0000", 2024-03-20T09:40:20.725 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.725 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.725 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.725 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:40:20.725 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:44.285035+0000", 2024-03-20T09:40:20.725 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.725 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.725 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.725 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:40:20.725 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:44.285036+0000", 2024-03-20T09:40:20.725 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.9899999999999999e-07 2024-03-20T09:40:20.725 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.725 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.725 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:40:20.725 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:44.285043+0000", 2024-03-20T09:40:20.726 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.2960000000000001e-06 2024-03-20T09:40:20.726 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.726 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.726 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:40:20.726 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:44.285055+0000", 2024-03-20T09:40:20.726 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1928e-05 2024-03-20T09:40:20.726 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.726 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.726 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:40:20.726 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:44.285061+0000", 2024-03-20T09:40:20.726 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2249999999999997e-06 2024-03-20T09:40:20.726 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.726 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.726 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:40:20.727 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:44.285113+0000", 2024-03-20T09:40:20.727 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2173999999999999e-05 2024-03-20T09:40:20.727 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.727 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.727 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:40:20.727 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:44.285134+0000", 2024-03-20T09:40:20.727 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0662e-05 2024-03-20T09:40:20.727 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.727 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.727 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:40:20.727 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:44.344646+0000", 2024-03-20T09:40:20.727 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.059512190999999999 2024-03-20T09:40:20.727 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.727 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.728 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:40:20.728 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:44.344686+0000", 2024-03-20T09:40:20.728 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.9422000000000003e-05 2024-03-20T09:40:20.728 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.728 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:40:20.728 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.728 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.728 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.728 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26917.0:189723 8.8 8:124f2a9c:::benchmark_data_smithi122_197775_object66419:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1032)", 2024-03-20T09:40:20.728 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:38:48.135656+0000", 2024-03-20T09:40:20.728 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 92.566330934000007, 2024-03-20T09:40:20.728 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.046885088999999998, 2024-03-20T09:40:20.728 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:40:20.728 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:40:20.729 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:40:20.729 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26917", 2024-03-20T09:40:20.729 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:40:20.729 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 189723 2024-03-20T09:40:20.729 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.729 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:40:20.729 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.729 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:40:20.729 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:48.135656+0000", 2024-03-20T09:40:20.729 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.729 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.729 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.729 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:40:20.730 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:48.135656+0000", 2024-03-20T09:40:20.730 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.730 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.730 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.730 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:40:20.730 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:48.135657+0000", 2024-03-20T09:40:20.730 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.7999999999999995e-07 2024-03-20T09:40:20.730 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.730 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.730 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:40:20.730 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:48.135660+0000", 2024-03-20T09:40:20.731 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.224e-06 2024-03-20T09:40:20.731 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.731 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.731 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:40:20.731 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:48.135671+0000", 2024-03-20T09:40:20.731 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0744000000000001e-05 2024-03-20T09:40:20.731 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.731 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.731 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:40:20.731 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:48.135679+0000", 2024-03-20T09:40:20.731 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.4000000000000003e-06 2024-03-20T09:40:20.731 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.732 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.732 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:40:20.732 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:48.135725+0000", 2024-03-20T09:40:20.732 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6437999999999998e-05 2024-03-20T09:40:20.732 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.732 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.732 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:40:20.732 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:48.135829+0000", 2024-03-20T09:40:20.732 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000103937 2024-03-20T09:40:20.732 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.732 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.732 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 3,9", 2024-03-20T09:40:20.732 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:48.135933+0000", 2024-03-20T09:40:20.733 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000104436 2024-03-20T09:40:20.733 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.733 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.733 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:40:20.733 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:48.140675+0000", 2024-03-20T09:40:20.733 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.004741215 2024-03-20T09:40:20.733 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.733 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.733 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.733 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:48.140958+0000", 2024-03-20T09:40:20.733 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000283372 2024-03-20T09:40:20.733 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.733 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.734 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.734 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:48.182427+0000", 2024-03-20T09:40:20.734 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.041469310000000002 2024-03-20T09:40:20.734 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.734 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.734 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:40:20.734 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:48.182487+0000", 2024-03-20T09:40:20.734 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9425000000000001e-05 2024-03-20T09:40:20.734 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.734 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.734 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:40:20.734 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:38:48.182542+0000", 2024-03-20T09:40:20.734 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4907999999999997e-05 2024-03-20T09:40:20.734 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.735 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:40:20.735 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.735 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.735 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.735 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26917.0:213303 8.4 8:21aab939:::benchmark_data_smithi122_197775_object89999:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1042)", 2024-03-20T09:40:20.735 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:39:19.867018+0000", 2024-03-20T09:40:20.735 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 60.834969299999997, 2024-03-20T09:40:20.735 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.046207452000000003, 2024-03-20T09:40:20.735 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:40:20.735 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:40:20.735 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:40:20.735 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26917", 2024-03-20T09:40:20.735 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:40:20.735 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 213303 2024-03-20T09:40:20.735 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.736 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:40:20.736 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.736 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:40:20.736 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.867018+0000", 2024-03-20T09:40:20.736 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.736 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.736 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.736 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:40:20.736 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.867018+0000", 2024-03-20T09:40:20.736 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.736 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.736 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.736 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:40:20.736 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.867019+0000", 2024-03-20T09:40:20.736 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0470000000000001e-06 2024-03-20T09:40:20.736 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.737 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.737 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:40:20.737 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.867025+0000", 2024-03-20T09:40:20.737 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3700000000000003e-06 2024-03-20T09:40:20.737 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.737 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.737 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:40:20.737 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.867039+0000", 2024-03-20T09:40:20.737 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4185000000000001e-05 2024-03-20T09:40:20.737 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.737 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.737 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:40:20.737 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.867060+0000", 2024-03-20T09:40:20.737 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1233000000000002e-05 2024-03-20T09:40:20.737 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.738 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.738 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:40:20.738 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.867125+0000", 2024-03-20T09:40:20.738 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4890999999999999e-05 2024-03-20T09:40:20.738 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.738 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.738 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:40:20.738 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.867277+0000", 2024-03-20T09:40:20.738 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00015222900000000001 2024-03-20T09:40:20.738 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.738 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.738 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 9,10", 2024-03-20T09:40:20.738 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.867416+0000", 2024-03-20T09:40:20.738 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013936299999999999 2024-03-20T09:40:20.739 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.739 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.739 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:40:20.739 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.874654+0000", 2024-03-20T09:40:20.739 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.007237322 2024-03-20T09:40:20.739 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.739 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.739 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.739 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.890569+0000", 2024-03-20T09:40:20.739 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.015915480999999999 2024-03-20T09:40:20.739 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.739 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.739 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.739 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.913115+0000", 2024-03-20T09:40:20.739 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.022545910999999998 2024-03-20T09:40:20.739 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.740 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.740 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:40:20.740 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.913169+0000", 2024-03-20T09:40:20.740 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3495000000000002e-05 2024-03-20T09:40:20.740 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.740 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.740 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:40:20.740 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.913226+0000", 2024-03-20T09:40:20.740 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6925000000000001e-05 2024-03-20T09:40:20.740 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.740 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:40:20.740 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.740 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.740 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.740 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26917.0:213306 8.4 8:2aa693d3:::benchmark_data_smithi122_197775_object90002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1042)", 2024-03-20T09:40:20.741 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:39:19.871198+0000", 2024-03-20T09:40:20.741 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 60.830788943999998, 2024-03-20T09:40:20.741 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.048464710000000001, 2024-03-20T09:40:20.741 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:40:20.741 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:40:20.741 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:40:20.741 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26917", 2024-03-20T09:40:20.741 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:40:20.741 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 213306 2024-03-20T09:40:20.741 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.741 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:40:20.741 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.741 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:40:20.742 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.871198+0000", 2024-03-20T09:40:20.742 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.742 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.742 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.742 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:40:20.742 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.871198+0000", 2024-03-20T09:40:20.742 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.742 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.742 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.742 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:40:20.742 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.871200+0000", 2024-03-20T09:40:20.742 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5319999999999999e-06 2024-03-20T09:40:20.742 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.742 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.743 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:40:20.743 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.871207+0000", 2024-03-20T09:40:20.743 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.533e-06 2024-03-20T09:40:20.743 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.743 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.743 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:40:20.743 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.871221+0000", 2024-03-20T09:40:20.743 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4673e-05 2024-03-20T09:40:20.743 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.743 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.743 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:40:20.743 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.871233+0000", 2024-03-20T09:40:20.743 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1793e-05 2024-03-20T09:40:20.743 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.744 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.744 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:40:20.744 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.871301+0000", 2024-03-20T09:40:20.744 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.7528999999999997e-05 2024-03-20T09:40:20.744 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.744 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.744 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:40:20.744 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.871405+0000", 2024-03-20T09:40:20.744 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000104898 2024-03-20T09:40:20.744 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.744 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.744 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 9,10", 2024-03-20T09:40:20.744 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.871534+0000", 2024-03-20T09:40:20.744 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012845000000000001 2024-03-20T09:40:20.745 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.745 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.745 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:40:20.745 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.874771+0000", 2024-03-20T09:40:20.745 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0032368240000000001 2024-03-20T09:40:20.745 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.745 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.745 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.745 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.890604+0000", 2024-03-20T09:40:20.745 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0158335 2024-03-20T09:40:20.745 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.745 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.745 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.745 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.919591+0000", 2024-03-20T09:40:20.745 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.028986639000000002 2024-03-20T09:40:20.746 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.746 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.746 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:40:20.746 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.919629+0000", 2024-03-20T09:40:20.746 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.8046999999999997e-05 2024-03-20T09:40:20.746 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.746 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.746 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:40:20.746 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.919663+0000", 2024-03-20T09:40:20.746 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.4292000000000003e-05 2024-03-20T09:40:20.746 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.746 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:40:20.746 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.746 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.747 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.747 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26917.0:213309 8.4 8:25f8e2d1:::benchmark_data_smithi122_197775_object90005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1042)", 2024-03-20T09:40:20.747 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:39:19.879149+0000", 2024-03-20T09:40:20.747 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 60.822838208, 2024-03-20T09:40:20.747 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.051138688000000002, 2024-03-20T09:40:20.747 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:40:20.747 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:40:20.747 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:40:20.747 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26917", 2024-03-20T09:40:20.747 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:40:20.747 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 213309 2024-03-20T09:40:20.747 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.747 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:40:20.747 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.748 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:40:20.748 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.879149+0000", 2024-03-20T09:40:20.748 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.748 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.748 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.748 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:40:20.748 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.879149+0000", 2024-03-20T09:40:20.748 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.748 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.748 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.748 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:40:20.748 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.879150+0000", 2024-03-20T09:40:20.748 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1379999999999999e-06 2024-03-20T09:40:20.748 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.749 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.749 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:40:20.749 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.879156+0000", 2024-03-20T09:40:20.749 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6890000000000001e-06 2024-03-20T09:40:20.749 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.749 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.749 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:40:20.749 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.879168+0000", 2024-03-20T09:40:20.749 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1934e-05 2024-03-20T09:40:20.749 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.749 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.749 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:40:20.749 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.879175+0000", 2024-03-20T09:40:20.749 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.6479999999999997e-06 2024-03-20T09:40:20.749 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.749 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.750 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:40:20.750 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.879240+0000", 2024-03-20T09:40:20.750 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.5148999999999996e-05 2024-03-20T09:40:20.750 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.750 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.750 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:40:20.750 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.879376+0000", 2024-03-20T09:40:20.750 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000136296 2024-03-20T09:40:20.750 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.750 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.750 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 9,10", 2024-03-20T09:40:20.750 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.880015+0000", 2024-03-20T09:40:20.750 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00063888499999999995 2024-03-20T09:40:20.750 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.751 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.751 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:40:20.751 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.884012+0000", 2024-03-20T09:40:20.751 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0039972089999999998 2024-03-20T09:40:20.751 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.751 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.751 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.751 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.890643+0000", 2024-03-20T09:40:20.751 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0066304659999999998 2024-03-20T09:40:20.751 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.751 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.752 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.752 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.930206+0000", 2024-03-20T09:40:20.752 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.039563447000000002 2024-03-20T09:40:20.752 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.752 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.752 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:40:20.752 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.930246+0000", 2024-03-20T09:40:20.752 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.9580999999999999e-05 2024-03-20T09:40:20.752 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.752 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.752 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:40:20.752 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:19.930288+0000", 2024-03-20T09:40:20.752 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.2246000000000002e-05 2024-03-20T09:40:20.753 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.753 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:40:20.753 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.753 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.753 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.753 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26917.0:214683 8.8 8:1ed9dc25:::benchmark_data_smithi122_197775_object91379:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1042)", 2024-03-20T09:40:20.753 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:39:21.842962+0000", 2024-03-20T09:40:20.753 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 58.859025404, 2024-03-20T09:40:20.753 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.051274574000000003, 2024-03-20T09:40:20.753 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:40:20.753 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:40:20.753 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:40:20.753 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26917", 2024-03-20T09:40:20.753 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:40:20.754 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 214683 2024-03-20T09:40:20.754 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.754 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:40:20.754 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.754 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:40:20.754 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:21.842962+0000", 2024-03-20T09:40:20.754 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.754 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.754 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.754 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:40:20.754 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:21.842962+0000", 2024-03-20T09:40:20.754 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.754 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.755 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.755 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:40:20.755 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:21.842964+0000", 2024-03-20T09:40:20.755 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.838e-06 2024-03-20T09:40:20.755 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.755 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.755 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:40:20.755 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:21.842972+0000", 2024-03-20T09:40:20.755 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.0839999999999992e-06 2024-03-20T09:40:20.755 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.755 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.755 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:40:20.755 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:21.842989+0000", 2024-03-20T09:40:20.755 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7351e-05 2024-03-20T09:40:20.755 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.755 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.756 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:40:20.756 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:21.842998+0000", 2024-03-20T09:40:20.756 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.0019999999999995e-06 2024-03-20T09:40:20.756 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.756 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.756 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:40:20.756 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:21.843076+0000", 2024-03-20T09:40:20.756 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.8083000000000004e-05 2024-03-20T09:40:20.756 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.756 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.756 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:40:20.756 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:21.843175+0000", 2024-03-20T09:40:20.756 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.8684000000000006e-05 2024-03-20T09:40:20.756 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.756 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.757 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 3,9", 2024-03-20T09:40:20.757 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:21.843263+0000", 2024-03-20T09:40:20.757 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.8239999999999995e-05 2024-03-20T09:40:20.757 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.757 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.757 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:40:20.757 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:21.852626+0000", 2024-03-20T09:40:20.757 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0093622589999999995 2024-03-20T09:40:20.757 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.757 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.757 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.757 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:21.852683+0000", 2024-03-20T09:40:20.757 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7624000000000003e-05 2024-03-20T09:40:20.758 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.758 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.758 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.758 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:21.894142+0000", 2024-03-20T09:40:20.758 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.041458768 2024-03-20T09:40:20.758 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.758 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.758 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:40:20.758 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:21.894206+0000", 2024-03-20T09:40:20.758 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4406999999999995e-05 2024-03-20T09:40:20.758 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.758 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.758 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:40:20.759 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:21.894237+0000", 2024-03-20T09:40:20.759 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.0233999999999999e-05 2024-03-20T09:40:20.759 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.759 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:40:20.759 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.759 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.759 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.759 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26917.0:214685 8.8 8:1457f225:::benchmark_data_smithi122_197775_object91381:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1042)", 2024-03-20T09:40:20.759 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:39:21.843018+0000", 2024-03-20T09:40:20.759 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 58.858969119999998, 2024-03-20T09:40:20.759 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.055561851000000002, 2024-03-20T09:40:20.759 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:40:20.759 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:40:20.759 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:40:20.760 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26917", 2024-03-20T09:40:20.760 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:40:20.760 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 214685 2024-03-20T09:40:20.760 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.760 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:40:20.760 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.760 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:40:20.760 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:21.843018+0000", 2024-03-20T09:40:20.760 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.760 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.760 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.760 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:40:20.760 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:21.843018+0000", 2024-03-20T09:40:20.760 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.760 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.761 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.761 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:40:20.761 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:21.843019+0000", 2024-03-20T09:40:20.761 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9200000000000002e-07 2024-03-20T09:40:20.761 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.761 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.761 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:40:20.761 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:21.843036+0000", 2024-03-20T09:40:20.761 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7652999999999999e-05 2024-03-20T09:40:20.761 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.761 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.761 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:40:20.762 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:21.843041+0000", 2024-03-20T09:40:20.762 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6650000000000002e-06 2024-03-20T09:40:20.762 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.762 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.762 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:40:20.762 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:21.843045+0000", 2024-03-20T09:40:20.762 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.8e-06 2024-03-20T09:40:20.762 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.762 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.762 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:40:20.762 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:21.847338+0000", 2024-03-20T09:40:20.762 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0042930039999999996 2024-03-20T09:40:20.762 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.763 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.763 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:40:20.763 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:21.847494+0000", 2024-03-20T09:40:20.763 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00015592000000000001 2024-03-20T09:40:20.763 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.763 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.763 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 3,9", 2024-03-20T09:40:20.763 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:21.847639+0000", 2024-03-20T09:40:20.763 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000145363 2024-03-20T09:40:20.763 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.763 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.763 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.763 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:21.852731+0000", 2024-03-20T09:40:20.763 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.005092147 2024-03-20T09:40:20.764 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.764 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.764 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:40:20.764 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:21.852800+0000", 2024-03-20T09:40:20.764 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.8930999999999994e-05 2024-03-20T09:40:20.764 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.764 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.764 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.764 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:21.898480+0000", 2024-03-20T09:40:20.764 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.045680305999999997 2024-03-20T09:40:20.764 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.764 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.764 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:40:20.764 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:21.898540+0000", 2024-03-20T09:40:20.765 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9191999999999998e-05 2024-03-20T09:40:20.765 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.765 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.765 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:40:20.765 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:21.898580+0000", 2024-03-20T09:40:20.765 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.0577999999999997e-05 2024-03-20T09:40:20.765 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.765 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:40:20.765 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.765 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.765 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.765 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26917.0:215917 8.4 8:2558530f:::benchmark_data_smithi122_197775_object92613:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1043)", 2024-03-20T09:40:20.765 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:39:23.545659+0000", 2024-03-20T09:40:20.766 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 57.156328268999999, 2024-03-20T09:40:20.766 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.056135733, 2024-03-20T09:40:20.766 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:40:20.766 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:40:20.766 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:40:20.766 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26917", 2024-03-20T09:40:20.766 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:40:20.766 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 215917 2024-03-20T09:40:20.766 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.766 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:40:20.766 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.766 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:40:20.766 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:23.545659+0000", 2024-03-20T09:40:20.766 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.767 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.767 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.767 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:40:20.767 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:23.545659+0000", 2024-03-20T09:40:20.767 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.767 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.767 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.767 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:40:20.767 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:23.545660+0000", 2024-03-20T09:40:20.767 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1170000000000001e-06 2024-03-20T09:40:20.767 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.767 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.767 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:40:20.767 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:23.545665+0000", 2024-03-20T09:40:20.768 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0649999999999998e-06 2024-03-20T09:40:20.768 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.768 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.768 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:40:20.768 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:23.545677+0000", 2024-03-20T09:40:20.768 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1474999999999999e-05 2024-03-20T09:40:20.768 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.768 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.768 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:40:20.768 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:23.545683+0000", 2024-03-20T09:40:20.768 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0090000000000004e-06 2024-03-20T09:40:20.768 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.768 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.768 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:40:20.769 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:23.549913+0000", 2024-03-20T09:40:20.769 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.004230223 2024-03-20T09:40:20.769 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.769 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.769 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:40:20.769 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:23.550049+0000", 2024-03-20T09:40:20.769 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013632800000000001 2024-03-20T09:40:20.769 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.769 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.769 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 9,10", 2024-03-20T09:40:20.769 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:23.550173+0000", 2024-03-20T09:40:20.769 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012387699999999999 2024-03-20T09:40:20.769 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.770 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.770 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:40:20.770 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:23.554457+0000", 2024-03-20T09:40:20.770 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0042839360000000003 2024-03-20T09:40:20.770 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.770 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.770 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.770 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:23.569143+0000", 2024-03-20T09:40:20.770 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.014686227 2024-03-20T09:40:20.770 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.770 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.770 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.770 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:23.601701+0000", 2024-03-20T09:40:20.770 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.032557547999999999 2024-03-20T09:40:20.771 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.771 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.771 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:40:20.771 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:23.601745+0000", 2024-03-20T09:40:20.771 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.3696999999999999e-05 2024-03-20T09:40:20.771 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.771 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.771 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:40:20.771 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:23.601795+0000", 2024-03-20T09:40:20.771 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0231000000000001e-05 2024-03-20T09:40:20.771 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.771 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:40:20.771 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.771 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.772 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.772 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26917.0:215938 8.4 8:29015642:::benchmark_data_smithi122_197775_object92634:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1043)", 2024-03-20T09:40:20.772 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:39:23.575570+0000", 2024-03-20T09:40:20.772 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 57.126417222000001, 2024-03-20T09:40:20.772 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.054719365999999998, 2024-03-20T09:40:20.772 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:40:20.772 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:40:20.772 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:40:20.772 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26917", 2024-03-20T09:40:20.772 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:40:20.772 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 215938 2024-03-20T09:40:20.772 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.772 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:40:20.772 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.773 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:40:20.773 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:23.575570+0000", 2024-03-20T09:40:20.773 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.773 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.773 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.773 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:40:20.773 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:23.575570+0000", 2024-03-20T09:40:20.773 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.773 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.773 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.773 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:40:20.773 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:23.575571+0000", 2024-03-20T09:40:20.773 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.1100000000000001e-07 2024-03-20T09:40:20.773 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.774 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.774 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:40:20.774 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:23.575574+0000", 2024-03-20T09:40:20.774 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9519999999999999e-06 2024-03-20T09:40:20.774 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.774 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.774 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:40:20.774 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:23.575580+0000", 2024-03-20T09:40:20.774 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.8000000000000001e-06 2024-03-20T09:40:20.774 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.774 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.774 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:40:20.774 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:23.575587+0000", 2024-03-20T09:40:20.774 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.9879999999999998e-06 2024-03-20T09:40:20.775 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.775 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.775 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:40:20.775 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:23.579529+0000", 2024-03-20T09:40:20.775 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0039415750000000001 2024-03-20T09:40:20.775 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.775 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.775 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:40:20.775 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:23.579622+0000", 2024-03-20T09:40:20.775 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.2899999999999995e-05 2024-03-20T09:40:20.775 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.775 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.776 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 9,10", 2024-03-20T09:40:20.776 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:23.579728+0000", 2024-03-20T09:40:20.776 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000106473 2024-03-20T09:40:20.776 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.776 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.776 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:40:20.776 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:23.584040+0000", 2024-03-20T09:40:20.776 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0043121799999999997 2024-03-20T09:40:20.776 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.776 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.776 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.776 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:23.591153+0000", 2024-03-20T09:40:20.776 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0071125959999999997 2024-03-20T09:40:20.776 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.777 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.777 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.777 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:23.630211+0000", 2024-03-20T09:40:20.777 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.039057728999999999 2024-03-20T09:40:20.777 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.777 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.777 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:40:20.777 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:23.630250+0000", 2024-03-20T09:40:20.777 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.8949999999999998e-05 2024-03-20T09:40:20.777 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.777 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.777 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:40:20.778 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:23.630290+0000", 2024-03-20T09:40:20.778 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.9811999999999998e-05 2024-03-20T09:40:20.778 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.778 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:40:20.778 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.778 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.778 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.778 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26917.0:216541 8.4 8:2e48fc82:::benchmark_data_smithi122_197775_object93237:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1044)", 2024-03-20T09:40:20.778 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:39:24.397427+0000", 2024-03-20T09:40:20.778 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 56.304560637999998, 2024-03-20T09:40:20.778 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.063495650000000001, 2024-03-20T09:40:20.778 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:40:20.778 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:40:20.779 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:40:20.779 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26917", 2024-03-20T09:40:20.779 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:40:20.779 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 216541 2024-03-20T09:40:20.779 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.779 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:40:20.779 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.779 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:40:20.779 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:24.397427+0000", 2024-03-20T09:40:20.779 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.779 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.779 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.779 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:40:20.779 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:24.397427+0000", 2024-03-20T09:40:20.779 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.780 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.780 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.780 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:40:20.780 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:24.397428+0000", 2024-03-20T09:40:20.780 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.418e-06 2024-03-20T09:40:20.780 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.780 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.780 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:40:20.780 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:24.397435+0000", 2024-03-20T09:40:20.780 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.8630000000000002e-06 2024-03-20T09:40:20.780 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.780 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.780 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:40:20.780 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:24.397450+0000", 2024-03-20T09:40:20.780 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4722000000000001e-05 2024-03-20T09:40:20.780 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.781 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.781 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:40:20.781 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:24.397475+0000", 2024-03-20T09:40:20.781 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5227000000000001e-05 2024-03-20T09:40:20.781 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.781 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.781 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:40:20.781 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:24.397524+0000", 2024-03-20T09:40:20.781 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8659999999999998e-05 2024-03-20T09:40:20.781 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.781 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.781 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:40:20.781 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:24.397620+0000", 2024-03-20T09:40:20.781 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.6199999999999994e-05 2024-03-20T09:40:20.781 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.781 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.782 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 9,10", 2024-03-20T09:40:20.782 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:24.397711+0000", 2024-03-20T09:40:20.782 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.1274999999999996e-05 2024-03-20T09:40:20.782 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.782 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.782 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:40:20.782 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:24.401530+0000", 2024-03-20T09:40:20.782 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0038187540000000002 2024-03-20T09:40:20.782 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.782 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.782 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.782 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:24.401846+0000", 2024-03-20T09:40:20.782 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00031627100000000002 2024-03-20T09:40:20.783 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.783 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.783 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.783 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:24.460656+0000", 2024-03-20T09:40:20.783 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.058810308999999998 2024-03-20T09:40:20.783 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.783 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.783 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:40:20.783 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:24.460820+0000", 2024-03-20T09:40:20.783 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00016355800000000001 2024-03-20T09:40:20.783 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.783 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.783 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:40:20.784 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:24.460922+0000", 2024-03-20T09:40:20.784 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00010239299999999999 2024-03-20T09:40:20.784 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.784 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:40:20.784 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.784 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.784 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.784 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26917.0:223175 8.4 8:2ecdf805:::benchmark_data_smithi122_197775_object99871:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1051)", 2024-03-20T09:40:20.784 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:39:34.531897+0000", 2024-03-20T09:40:20.784 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 46.170090285999997, 2024-03-20T09:40:20.784 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.047543548999999997, 2024-03-20T09:40:20.784 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:40:20.784 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:40:20.784 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:40:20.785 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26917", 2024-03-20T09:40:20.785 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:40:20.785 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 223175 2024-03-20T09:40:20.785 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.785 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:40:20.785 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.785 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:40:20.785 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:34.531897+0000", 2024-03-20T09:40:20.785 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.785 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.785 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.785 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:40:20.785 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:34.531897+0000", 2024-03-20T09:40:20.785 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.785 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.786 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.786 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:40:20.786 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:34.531899+0000", 2024-03-20T09:40:20.786 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3960000000000001e-06 2024-03-20T09:40:20.786 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.786 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.786 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:40:20.786 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:34.531906+0000", 2024-03-20T09:40:20.786 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.4309999999999997e-06 2024-03-20T09:40:20.786 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.786 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.786 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:40:20.786 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:34.531921+0000", 2024-03-20T09:40:20.786 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5192e-05 2024-03-20T09:40:20.786 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.787 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.787 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:40:20.787 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:34.531930+0000", 2024-03-20T09:40:20.787 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.6969999999999999e-06 2024-03-20T09:40:20.787 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.787 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.787 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:40:20.787 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:34.531994+0000", 2024-03-20T09:40:20.787 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4559000000000003e-05 2024-03-20T09:40:20.787 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.787 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.787 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:40:20.787 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:34.532158+0000", 2024-03-20T09:40:20.788 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000163998 2024-03-20T09:40:20.788 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.788 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.788 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 9,10", 2024-03-20T09:40:20.788 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:34.532307+0000", 2024-03-20T09:40:20.788 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000148529 2024-03-20T09:40:20.788 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.788 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.788 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:40:20.788 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:34.536653+0000", 2024-03-20T09:40:20.788 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0043457560000000001 2024-03-20T09:40:20.788 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.788 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.788 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.789 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:34.542189+0000", 2024-03-20T09:40:20.789 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0055358760000000003 2024-03-20T09:40:20.789 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.789 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.789 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.789 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:34.579363+0000", 2024-03-20T09:40:20.789 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.037174064 2024-03-20T09:40:20.789 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.789 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.789 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:40:20.789 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:34.579401+0000", 2024-03-20T09:40:20.789 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.8541000000000003e-05 2024-03-20T09:40:20.789 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.789 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.789 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:40:20.790 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:34.579441+0000", 2024-03-20T09:40:20.790 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.9509999999999999e-05 2024-03-20T09:40:20.790 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.790 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:40:20.790 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.790 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.790 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.790 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26917.0:239421 8.4 8:288e42bb:::benchmark_data_smithi122_197775_object116117:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1056)", 2024-03-20T09:40:20.790 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:39:56.353140+0000", 2024-03-20T09:40:20.790 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 24.348847541000001, 2024-03-20T09:40:20.790 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.045449219999999999, 2024-03-20T09:40:20.790 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:40:20.790 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:40:20.790 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:40:20.791 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26917", 2024-03-20T09:40:20.791 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:40:20.791 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 239421 2024-03-20T09:40:20.791 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.791 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:40:20.791 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.791 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:40:20.791 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:56.353140+0000", 2024-03-20T09:40:20.791 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.791 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.791 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.791 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:40:20.791 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:56.353140+0000", 2024-03-20T09:40:20.791 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.791 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.791 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.792 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:40:20.792 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:56.353141+0000", 2024-03-20T09:40:20.792 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0499999999999999e-06 2024-03-20T09:40:20.792 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.792 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.792 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:40:20.792 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:56.353145+0000", 2024-03-20T09:40:20.792 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4129999999999999e-06 2024-03-20T09:40:20.792 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.792 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.792 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:40:20.792 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:56.353157+0000", 2024-03-20T09:40:20.792 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1282e-05 2024-03-20T09:40:20.792 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.792 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.793 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:40:20.793 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:56.353163+0000", 2024-03-20T09:40:20.793 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.6290000000000004e-06 2024-03-20T09:40:20.793 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.793 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.793 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:40:20.793 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:56.353232+0000", 2024-03-20T09:40:20.793 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.9088e-05 2024-03-20T09:40:20.793 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.793 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.793 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:40:20.793 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:56.353328+0000", 2024-03-20T09:40:20.793 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.5956000000000001e-05 2024-03-20T09:40:20.793 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.794 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.794 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 9,10", 2024-03-20T09:40:20.794 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:56.353406+0000", 2024-03-20T09:40:20.794 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7668000000000002e-05 2024-03-20T09:40:20.794 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.794 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.794 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:40:20.794 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:56.358548+0000", 2024-03-20T09:40:20.794 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0051423160000000001 2024-03-20T09:40:20.794 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.794 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.794 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.794 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:56.366766+0000", 2024-03-20T09:40:20.794 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.008217762 2024-03-20T09:40:20.795 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.795 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.795 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.795 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:56.398482+0000", 2024-03-20T09:40:20.795 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.031715575000000003 2024-03-20T09:40:20.795 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.795 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.795 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:40:20.795 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:56.398540+0000", 2024-03-20T09:40:20.795 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8402000000000001e-05 2024-03-20T09:40:20.795 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.795 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.795 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:40:20.795 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:56.398589+0000", 2024-03-20T09:40:20.796 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9079000000000002e-05 2024-03-20T09:40:20.796 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.796 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:40:20.796 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.796 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.796 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.796 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26917.0:239431 8.4 8:28d87bf3:::benchmark_data_smithi122_197775_object116127:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1056)", 2024-03-20T09:40:20.796 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:39:56.361882+0000", 2024-03-20T09:40:20.796 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 24.340105366, 2024-03-20T09:40:20.796 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.045872636000000001, 2024-03-20T09:40:20.796 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:40:20.796 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:40:20.796 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:40:20.796 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26917", 2024-03-20T09:40:20.796 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:40:20.796 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 239431 2024-03-20T09:40:20.797 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.797 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:40:20.797 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.797 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:40:20.797 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:56.361882+0000", 2024-03-20T09:40:20.797 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.797 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.797 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.797 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:40:20.797 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:56.361882+0000", 2024-03-20T09:40:20.797 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.797 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.797 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.797 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:40:20.797 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:56.361883+0000", 2024-03-20T09:40:20.798 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.7599999999999996e-07 2024-03-20T09:40:20.798 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.798 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.798 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:40:20.798 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:56.361888+0000", 2024-03-20T09:40:20.798 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8629999999999999e-06 2024-03-20T09:40:20.798 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.798 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.798 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:40:20.798 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:56.361901+0000", 2024-03-20T09:40:20.798 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2933e-05 2024-03-20T09:40:20.798 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.798 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.798 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:40:20.799 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:56.361908+0000", 2024-03-20T09:40:20.799 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.6170000000000001e-06 2024-03-20T09:40:20.799 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.799 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.799 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:40:20.799 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:56.363102+0000", 2024-03-20T09:40:20.799 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001193663 2024-03-20T09:40:20.799 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.799 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.799 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:40:20.799 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:56.363176+0000", 2024-03-20T09:40:20.799 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3866000000000001e-05 2024-03-20T09:40:20.799 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.799 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.799 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 9,10", 2024-03-20T09:40:20.800 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:56.363269+0000", 2024-03-20T09:40:20.800 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.3107000000000002e-05 2024-03-20T09:40:20.800 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.800 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.800 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:40:20.800 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:56.366969+0000", 2024-03-20T09:40:20.800 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0037000240000000001 2024-03-20T09:40:20.800 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.800 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.800 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.800 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:56.382237+0000", 2024-03-20T09:40:20.800 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.01526831 2024-03-20T09:40:20.800 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.800 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.800 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.800 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:56.407680+0000", 2024-03-20T09:40:20.801 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.025442412000000001 2024-03-20T09:40:20.801 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.801 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.801 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:40:20.801 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:56.407720+0000", 2024-03-20T09:40:20.801 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.0420999999999998e-05 2024-03-20T09:40:20.801 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.801 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.801 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:40:20.801 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:39:56.407755+0000", 2024-03-20T09:40:20.801 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.4544e-05 2024-03-20T09:40:20.801 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.801 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:40:20.801 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.801 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.802 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.802 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26917.0:244245 8.4 8:2047d8d1:::benchmark_data_smithi122_197775_object120941:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1058)", 2024-03-20T09:40:20.802 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:40:02.543433+0000", 2024-03-20T09:40:20.802 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 18.158554899999999, 2024-03-20T09:40:20.802 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.058907818000000001, 2024-03-20T09:40:20.802 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:40:20.802 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:40:20.802 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:40:20.802 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26917", 2024-03-20T09:40:20.802 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:40:20.802 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 244245 2024-03-20T09:40:20.802 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.802 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:40:20.802 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.803 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:40:20.803 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:02.543433+0000", 2024-03-20T09:40:20.803 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.803 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.803 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.803 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:40:20.803 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:02.543433+0000", 2024-03-20T09:40:20.803 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.803 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.803 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.803 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:40:20.803 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:02.543433+0000", 2024-03-20T09:40:20.803 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9799999999999999e-07 2024-03-20T09:40:20.803 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.803 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.804 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:40:20.804 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:02.543435+0000", 2024-03-20T09:40:20.804 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8300000000000001e-06 2024-03-20T09:40:20.804 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.804 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.804 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:40:20.804 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:02.543439+0000", 2024-03-20T09:40:20.804 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.7190000000000001e-06 2024-03-20T09:40:20.804 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.804 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.804 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:40:20.804 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:02.543443+0000", 2024-03-20T09:40:20.804 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.1690000000000001e-06 2024-03-20T09:40:20.805 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.805 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.805 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:40:20.805 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:02.551093+0000", 2024-03-20T09:40:20.805 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0076500689999999998 2024-03-20T09:40:20.805 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.805 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.805 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:40:20.805 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:02.551223+0000", 2024-03-20T09:40:20.805 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013011899999999999 2024-03-20T09:40:20.805 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.805 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.805 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 9,10", 2024-03-20T09:40:20.806 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:02.551352+0000", 2024-03-20T09:40:20.806 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000129102 2024-03-20T09:40:20.806 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.806 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.806 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:40:20.806 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:02.559325+0000", 2024-03-20T09:40:20.806 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0079734490000000005 2024-03-20T09:40:20.806 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.806 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.806 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.806 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:02.562429+0000", 2024-03-20T09:40:20.806 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0031038929999999999 2024-03-20T09:40:20.806 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.807 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.807 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.807 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:02.602257+0000", 2024-03-20T09:40:20.807 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.039828229 2024-03-20T09:40:20.807 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.807 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.807 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:40:20.807 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:02.602303+0000", 2024-03-20T09:40:20.807 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5139000000000003e-05 2024-03-20T09:40:20.807 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.807 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.807 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:40:20.807 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:02.602340+0000", 2024-03-20T09:40:20.808 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7802000000000002e-05 2024-03-20T09:40:20.808 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.808 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:40:20.808 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.808 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.808 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.808 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26917.0:244246 8.4 8:2df85ab6:::benchmark_data_smithi122_197775_object120942:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1058)", 2024-03-20T09:40:20.808 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:40:02.543453+0000", 2024-03-20T09:40:20.808 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 18.158534634999999, 2024-03-20T09:40:20.808 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.059018978, 2024-03-20T09:40:20.808 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:40:20.808 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:40:20.808 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:40:20.809 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26917", 2024-03-20T09:40:20.809 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:40:20.809 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 244246 2024-03-20T09:40:20.809 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.809 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:40:20.809 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.809 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:40:20.809 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:02.543453+0000", 2024-03-20T09:40:20.809 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.809 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.809 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.809 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:40:20.809 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:02.543453+0000", 2024-03-20T09:40:20.810 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:40:20.810 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.810 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.810 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:40:20.810 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:02.543453+0000", 2024-03-20T09:40:20.810 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.7599999999999998e-07 2024-03-20T09:40:20.810 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.810 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.810 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:40:20.810 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:02.543455+0000", 2024-03-20T09:40:20.810 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.745e-06 2024-03-20T09:40:20.810 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.811 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.811 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:40:20.811 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:02.543459+0000", 2024-03-20T09:40:20.811 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.2980000000000002e-06 2024-03-20T09:40:20.811 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.811 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.811 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:40:20.811 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:02.543462+0000", 2024-03-20T09:40:20.811 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.2440000000000002e-06 2024-03-20T09:40:20.811 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.811 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.811 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:40:20.811 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:02.555064+0000", 2024-03-20T09:40:20.812 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.011602114 2024-03-20T09:40:20.812 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.812 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.812 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:40:20.812 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:02.555174+0000", 2024-03-20T09:40:20.812 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000109825 2024-03-20T09:40:20.812 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.812 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.812 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 9,10", 2024-03-20T09:40:20.812 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:02.555262+0000", 2024-03-20T09:40:20.812 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.7808999999999999e-05 2024-03-20T09:40:20.812 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.813 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.813 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:40:20.813 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:02.559442+0000", 2024-03-20T09:40:20.813 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0041797620000000001 2024-03-20T09:40:20.813 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.813 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.813 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.813 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:02.562521+0000", 2024-03-20T09:40:20.813 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.003078683 2024-03-20T09:40:20.813 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.813 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.813 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:40:20.813 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:02.602381+0000", 2024-03-20T09:40:20.813 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.039859994000000003 2024-03-20T09:40:20.813 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.814 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.814 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:40:20.814 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:02.602429+0000", 2024-03-20T09:40:20.814 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8946999999999997e-05 2024-03-20T09:40:20.814 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:40:20.814 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:40:20.814 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:40:20.814 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:02.602472+0000", 2024-03-20T09:40:20.814 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.2280999999999997e-05 2024-03-20T09:40:20.814 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.814 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:40:20.814 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.814 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:40:20.814 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:40:20.815 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:40:20.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:20.816 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:20.816 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:40:20.816 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:40:20.917 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:40:20.947 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:20.967 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:40:20.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:20.970 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:40:21.071 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:40:21.100 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:21.427 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_ops_in_flight 2024-03-20T09:40:21.449 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:40:21.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:21.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:40:21.552 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:40:21.582 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:21.702 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:40:21.707 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:40:21.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:21.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:40:21.811 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:40:21.840 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:21.861 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:40:21.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:21.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:40:21.981 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:40:22.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:21 smithi026 ceph-mon[68309]: pgmap v2576: 31 pgs: 31 active+clean; 577 KiB data, 8.1 GiB used, 975 GiB / 983 GiB avail 2024-03-20T09:40:22.008 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:22.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:21 smithi115 ceph-mon[72476]: pgmap v2576: 31 pgs: 31 active+clean; 577 KiB data, 8.1 GiB used, 975 GiB / 983 GiB avail 2024-03-20T09:40:22.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:21 smithi122 ceph-mon[73495]: pgmap v2576: 31 pgs: 31 active+clean; 577 KiB data, 8.1 GiB used, 975 GiB / 983 GiB avail 2024-03-20T09:40:22.258 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:40:22.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:22.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:40:22.363 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:40:22.397 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:22.418 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:40:22.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:22.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:40:22.522 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:40:22.541 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:40:22.550 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:22.560 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:40:22.569 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:40:22.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:22.572 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:40:22.580 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:40:22.599 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:40:22.620 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:40:22.643 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:40:22.663 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:40:22.674 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:40:22.683 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:40:22.701 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:22.704 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:40:22.726 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:40:22.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:22.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:40:22.732 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:40:22.758 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:40:22.780 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:40:22.781 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:40:22.781 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:40:22.781 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:40:22.781 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:40:22.781 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:40:22.781 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:40:22.782 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:40:22.782 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:40:22.782 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:40:22.782 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:40:22.782 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:40:22.782 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:40:22.831 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:40:22.856 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:22 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:40:22.856 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:22 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:40:22.856 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:22 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/452720315' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:40:22.858 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:22.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:22 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:40:22.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:22 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:40:22.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:22 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/452720315' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:40:22.981 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:40:22.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:22.985 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:40:23.086 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:40:23.116 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:22 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:40:23.117 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:22 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:40:23.117 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:22 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/452720315' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:40:23.119 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:23.140 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:40:23.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:23.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:40:23.244 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:40:23.276 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:23.298 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:40:23.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:23.301 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:40:23.402 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:40:23.434 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:23.454 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:40:23.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:23.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:40:23.522 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:40:23.523 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:40:23.558 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:40:23.586 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:23.605 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:40:23.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:23.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:40:23.709 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:40:23.719 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:40:23.719 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:40:23.736 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:23.755 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:40:23.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:23.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:40:23.769 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'unique_pool_7' created 2024-03-20T09:40:23.798 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:23 smithi026 ceph-mon[68309]: pgmap v2577: 31 pgs: 31 active+clean; 577 KiB data, 8.1 GiB used, 975 GiB / 983 GiB avail 2024-03-20T09:40:23.798 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:23 smithi026 ceph-mon[68309]: osdmap e1065: 12 total, 12 up, 11 in 2024-03-20T09:40:23.798 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:23 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2114595354' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-03-20T09:40:23.859 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:40:23.889 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:23.900 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:23 smithi122 ceph-mon[73495]: pgmap v2577: 31 pgs: 31 active+clean; 577 KiB data, 8.1 GiB used, 975 GiB / 983 GiB avail 2024-03-20T09:40:23.900 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:23 smithi122 ceph-mon[73495]: osdmap e1065: 12 total, 12 up, 11 in 2024-03-20T09:40:23.900 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:23 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2114595354' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-03-20T09:40:24.138 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:40:24.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:24.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:40:24.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:23 smithi115 ceph-mon[72476]: pgmap v2577: 31 pgs: 31 active+clean; 577 KiB data, 8.1 GiB used, 975 GiB / 983 GiB avail 2024-03-20T09:40:24.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:23 smithi115 ceph-mon[72476]: osdmap e1065: 12 total, 12 up, 11 in 2024-03-20T09:40:24.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:23 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2114595354' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]: dispatch 2024-03-20T09:40:24.214 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_blocked_ops 2024-03-20T09:40:24.243 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:40:24.269 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:24.289 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:40:24.291 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:24.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:40:24.392 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:40:24.420 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:24.439 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:40:24.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:24.443 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:40:24.476 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:40:24.543 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:40:24.570 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:24.588 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:40:24.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:24.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:40:24.692 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:40:24.723 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:24.894 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:24 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2114595354' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-03-20T09:40:24.894 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:24 smithi115 ceph-mon[72476]: osdmap e1066: 12 total, 12 up, 11 in 2024-03-20T09:40:24.920 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:24 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2114595354' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-03-20T09:40:24.920 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:24 smithi026 ceph-mon[68309]: osdmap e1066: 12 total, 12 up, 11 in 2024-03-20T09:40:24.924 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:40:24.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:24.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:40:25.028 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:40:25.052 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:24 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2114595354' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-03-20T09:40:25.052 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:24 smithi122 ceph-mon[73495]: osdmap e1066: 12 total, 12 up, 11 in 2024-03-20T09:40:25.055 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:25.435 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pool application enable unique_pool_7 rados --yes-i-really-mean-it || true 2024-03-20T09:40:25.569 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:40:25.894 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:25 smithi122 ceph-mon[73495]: pgmap v2580: 47 pgs: 3 creating+peering, 13 unknown, 31 active+clean; 577 KiB data, 8.1 GiB used, 975 GiB / 983 GiB avail 2024-03-20T09:40:25.895 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:25 smithi122 ceph-mon[73495]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-20T09:40:25.895 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:25 smithi122 ceph-mon[73495]: osdmap e1067: 12 total, 12 up, 11 in 2024-03-20T09:40:25.903 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:40:25.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:25.906 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:40:25.913 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:25 smithi026 ceph-mon[68309]: pgmap v2580: 47 pgs: 3 creating+peering, 13 unknown, 31 active+clean; 577 KiB data, 8.1 GiB used, 975 GiB / 983 GiB avail 2024-03-20T09:40:25.913 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:25 smithi026 ceph-mon[68309]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-20T09:40:25.913 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:25 smithi026 ceph-mon[68309]: osdmap e1067: 12 total, 12 up, 11 in 2024-03-20T09:40:26.006 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:40:26.036 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:26.058 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:40:26.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:26.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:40:26.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:25 smithi115 ceph-mon[72476]: pgmap v2580: 47 pgs: 3 creating+peering, 13 unknown, 31 active+clean; 577 KiB data, 8.1 GiB used, 975 GiB / 983 GiB avail 2024-03-20T09:40:26.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:25 smithi115 ceph-mon[72476]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-20T09:40:26.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:25 smithi115 ceph-mon[72476]: osdmap e1067: 12 total, 12 up, 11 in 2024-03-20T09:40:26.163 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:40:26.173 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:40:26.173 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:40:26.195 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:26.215 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:40:26.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:26.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:40:26.319 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:40:26.348 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:26.699 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:40:26.708 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:26.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:40:26.711 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:40:26.713 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:40:26.786 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:40:26.810 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:40:26.844 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:26.851 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_historic_ops 2024-03-20T09:40:26.864 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:40:26.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:26.867 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:40:26.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:26 smithi026 ceph-mon[68309]: osdmap e1068: 12 total, 12 up, 11 in 2024-03-20T09:40:26.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:26 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1866331183' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:40:26.967 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:40:26.999 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:27.021 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:40:27.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:27.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:40:27.118 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:40:27.125 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:40:27.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:26 smithi115 ceph-mon[72476]: osdmap e1068: 12 total, 12 up, 11 in 2024-03-20T09:40:27.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:26 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1866331183' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:40:27.150 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:26 smithi122 ceph-mon[73495]: osdmap e1068: 12 total, 12 up, 11 in 2024-03-20T09:40:27.150 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:26 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1866331183' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:40:27.153 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:27.401 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:40:27.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:27.404 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:40:27.505 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:40:27.537 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:27.687 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:40:27.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:27.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:40:27.791 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:40:27.823 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:27.844 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:40:27.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:27.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:40:27.948 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:40:27.975 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:28.090 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:27 smithi026 ceph-mon[68309]: pgmap v2583: 47 pgs: 3 creating+peering, 3 unknown, 41 active+clean; 577 KiB data, 8.1 GiB used, 975 GiB / 983 GiB avail 2024-03-20T09:40:28.090 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:27 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:40:28.090 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:27 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1866331183' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:40:28.090 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:27 smithi026 ceph-mon[68309]: osdmap e1069: 12 total, 12 up, 11 in 2024-03-20T09:40:28.101 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:40:28.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:28.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:40:28.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:27 smithi115 ceph-mon[72476]: pgmap v2583: 47 pgs: 3 creating+peering, 3 unknown, 41 active+clean; 577 KiB data, 8.1 GiB used, 975 GiB / 983 GiB avail 2024-03-20T09:40:28.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:27 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:40:28.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:27 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1866331183' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:40:28.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:27 smithi115 ceph-mon[72476]: osdmap e1069: 12 total, 12 up, 11 in 2024-03-20T09:40:28.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:27 smithi122 ceph-mon[73495]: pgmap v2583: 47 pgs: 3 creating+peering, 3 unknown, 41 active+clean; 577 KiB data, 8.1 GiB used, 975 GiB / 983 GiB avail 2024-03-20T09:40:28.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:27 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:40:28.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:27 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1866331183' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:40:28.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:27 smithi122 ceph-mon[73495]: osdmap e1069: 12 total, 12 up, 11 in 2024-03-20T09:40:28.205 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:40:28.234 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:28.937 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:40:28.937 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:40:29.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:28 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:40:29.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:28 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:40:29.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:28 smithi122 ceph-mon[73495]: pgmap v2585: 47 pgs: 3 creating+peering, 44 active+clean; 577 KiB data, 8.1 GiB used, 975 GiB / 983 GiB avail 2024-03-20T09:40:29.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:28 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:40:29.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:28 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:40:29.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:28 smithi026 ceph-mon[68309]: pgmap v2585: 47 pgs: 3 creating+peering, 44 active+clean; 577 KiB data, 8.1 GiB used, 975 GiB / 983 GiB avail 2024-03-20T09:40:29.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:28 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:40:29.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:28 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:40:29.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:28 smithi115 ceph-mon[72476]: pgmap v2585: 47 pgs: 3 creating+peering, 44 active+clean; 577 KiB data, 8.1 GiB used, 975 GiB / 983 GiB avail 2024-03-20T09:40:29.615 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-20T09:40:30.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:29 smithi122 ceph-mon[73495]: 1.c scrub starts 2024-03-20T09:40:30.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:29 smithi122 ceph-mon[73495]: osdmap e1070: 12 total, 12 up, 11 in 2024-03-20T09:40:30.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:29 smithi026 ceph-mon[68309]: 1.c scrub starts 2024-03-20T09:40:30.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:29 smithi026 ceph-mon[68309]: osdmap e1070: 12 total, 12 up, 11 in 2024-03-20T09:40:30.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:29 smithi115 ceph-mon[72476]: 1.c scrub starts 2024-03-20T09:40:30.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:29 smithi115 ceph-mon[72476]: osdmap e1070: 12 total, 12 up, 11 in 2024-03-20T09:40:30.807 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:40:30.815 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:40:30.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:30.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:40:30.829 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:40:30.849 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:40:30.868 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:40:30.890 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:40:30.911 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:40:30.919 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:40:30.934 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:40:30.944 INFO:teuthology.orchestra.run.smithi026.stderr:no valid command found; 10 closest matches: 2024-03-20T09:40:30.944 INFO:teuthology.orchestra.run.smithi026.stderr:0 2024-03-20T09:40:30.944 INFO:teuthology.orchestra.run.smithi026.stderr:1 2024-03-20T09:40:30.944 INFO:teuthology.orchestra.run.smithi026.stderr:2 2024-03-20T09:40:30.944 INFO:teuthology.orchestra.run.smithi026.stderr:abort 2024-03-20T09:40:30.944 INFO:teuthology.orchestra.run.smithi026.stderr:assert 2024-03-20T09:40:30.944 INFO:teuthology.orchestra.run.smithi026.stderr:bench [] [] [] [] 2024-03-20T09:40:30.944 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs debug_inject_read_zeros 2024-03-20T09:40:30.944 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs files list 2024-03-20T09:40:30.945 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs stats 2024-03-20T09:40:30.945 INFO:teuthology.orchestra.run.smithi026.stderr:bluestore allocator dump block 2024-03-20T09:40:30.945 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: invalid command 2024-03-20T09:40:30.950 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:30.955 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:40:30.972 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:40:30.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:30.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:40:30.977 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:40:30.997 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:40:31.017 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:40:31.037 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:40:31.039 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:40:31.039 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:40:31.039 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:40:31.039 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:40:31.039 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:40:31.039 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:40:31.039 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:40:31.039 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:40:31.039 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:40:31.039 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:40:31.039 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:40:31.039 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:40:31.076 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:40:31.108 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:31.129 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:40:31.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:31.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:40:31.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:30 smithi122 ceph-mon[73495]: 1.c scrub ok 2024-03-20T09:40:31.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:30 smithi122 ceph-mon[73495]: pgmap v2587: 47 pgs: 47 active+clean; 577 KiB data, 8.1 GiB used, 975 GiB / 983 GiB avail 2024-03-20T09:40:31.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:30 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/582182481' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:40:31.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:30 smithi026 ceph-mon[68309]: 1.c scrub ok 2024-03-20T09:40:31.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:30 smithi026 ceph-mon[68309]: pgmap v2587: 47 pgs: 47 active+clean; 577 KiB data, 8.1 GiB used, 975 GiB / 983 GiB avail 2024-03-20T09:40:31.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:30 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/582182481' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:40:31.233 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:40:31.263 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:31.316 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:40:31.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:31.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:40:31.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:30 smithi115 ceph-mon[72476]: 1.c scrub ok 2024-03-20T09:40:31.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:30 smithi115 ceph-mon[72476]: pgmap v2587: 47 pgs: 47 active+clean; 577 KiB data, 8.1 GiB used, 975 GiB / 983 GiB avail 2024-03-20T09:40:31.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:30 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/582182481' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:40:31.421 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:40:31.448 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:31.698 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:40:31.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:31.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:40:31.709 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:40:31.803 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:40:31.836 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:31.955 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:40:31.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:31.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:40:32.059 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:40:32.088 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:32.213 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:40:32.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:32.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:40:32.314 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:40:32.317 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:40:32.355 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:32.378 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:40:32.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:32.382 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:40:32.483 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:40:32.512 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:32.964 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:40:32.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:32.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:40:32.992 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T09:40:32.992 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-20T09:40:32.993 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.3 'trim stale osdmaps' 2024-03-20T09:40:33.069 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:40:33.104 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:33.125 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:40:33.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:33.128 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:40:33.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:32 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:40:33.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:32 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:40:33.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:32 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:40:33.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:32 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:40:33.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:32 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:40:33.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:32 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:40:33.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:32 smithi122 ceph-mon[73495]: osdmap e1071: 12 total, 12 up, 11 in 2024-03-20T09:40:33.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:32 smithi122 ceph-mon[73495]: 1.17 scrub starts 2024-03-20T09:40:33.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:32 smithi122 ceph-mon[73495]: pgmap v2589: 47 pgs: 47 active+clean; 577 KiB data, 8.1 GiB used, 975 GiB / 983 GiB avail 2024-03-20T09:40:33.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:32 smithi122 ceph-mon[73495]: osdmap e1072: 12 total, 12 up, 11 in 2024-03-20T09:40:33.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:32 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:40:33.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:32 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:40:33.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:32 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:40:33.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:32 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:40:33.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:32 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:40:33.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:32 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:40:33.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:32 smithi026 ceph-mon[68309]: osdmap e1071: 12 total, 12 up, 11 in 2024-03-20T09:40:33.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:32 smithi026 ceph-mon[68309]: 1.17 scrub starts 2024-03-20T09:40:33.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:32 smithi026 ceph-mon[68309]: pgmap v2589: 47 pgs: 47 active+clean; 577 KiB data, 8.1 GiB used, 975 GiB / 983 GiB avail 2024-03-20T09:40:33.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:32 smithi026 ceph-mon[68309]: osdmap e1072: 12 total, 12 up, 11 in 2024-03-20T09:40:33.230 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:40:33.261 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:32 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:40:33.261 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:32 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:40:33.261 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:32 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:40:33.261 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:32 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:40:33.261 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:32 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:40:33.261 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:32 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:40:33.261 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:32 smithi115 ceph-mon[72476]: osdmap e1071: 12 total, 12 up, 11 in 2024-03-20T09:40:33.261 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:32 smithi115 ceph-mon[72476]: 1.17 scrub starts 2024-03-20T09:40:33.261 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:32 smithi115 ceph-mon[72476]: pgmap v2589: 47 pgs: 47 active+clean; 577 KiB data, 8.1 GiB used, 975 GiB / 983 GiB avail 2024-03-20T09:40:33.261 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:32 smithi115 ceph-mon[72476]: osdmap e1072: 12 total, 12 up, 11 in 2024-03-20T09:40:33.263 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:33.284 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:40:33.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:33.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:40:33.388 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:40:33.416 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:33.667 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:40:33.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:33.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:40:33.694 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:40:33.696 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:40:33.736 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:40:33.772 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:40:33.807 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:33.906 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:40:33.910 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:33.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:40:33.934 INFO:teuthology.orchestra.run.smithi026.stderr:enabled application 'rados' on pool 'unique_pool_7' 2024-03-20T09:40:34.012 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:40:34.044 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:34.066 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:40:34.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:34.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:40:34.170 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:40:34.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:33 smithi122 ceph-mon[73495]: 1.17 scrub ok 2024-03-20T09:40:34.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:33 smithi122 ceph-mon[73495]: 1.4 scrub starts 2024-03-20T09:40:34.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:33 smithi122 ceph-mon[73495]: 1.4 scrub ok 2024-03-20T09:40:34.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:33 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/895719157' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-20T09:40:34.204 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:34.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:33 smithi026 ceph-mon[68309]: 1.17 scrub ok 2024-03-20T09:40:34.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:33 smithi026 ceph-mon[68309]: 1.4 scrub starts 2024-03-20T09:40:34.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:33 smithi026 ceph-mon[68309]: 1.4 scrub ok 2024-03-20T09:40:34.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:33 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/895719157' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-20T09:40:34.225 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:40:34.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:34.228 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:40:34.329 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:40:34.358 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:33 smithi115 ceph-mon[72476]: 1.17 scrub ok 2024-03-20T09:40:34.359 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:33 smithi115 ceph-mon[72476]: 1.4 scrub starts 2024-03-20T09:40:34.359 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:33 smithi115 ceph-mon[72476]: 1.4 scrub ok 2024-03-20T09:40:34.359 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:33 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/895719157' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-20T09:40:34.361 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:34.382 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:40:34.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:34.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:40:34.487 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:40:34.514 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:34.532 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:40:34.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:34.535 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:40:34.629 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:40:34.629 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:40:34.629 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-03-20T09:40:34.629 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:40:34.636 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:40:34.661 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:34.680 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:40:34.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:34.683 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:40:34.783 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:40:34.810 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:34.828 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:40:34.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:34.831 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:40:34.931 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:40:34.962 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:35.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:34 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/895719157' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-20T09:40:35.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:34 smithi122 ceph-mon[73495]: osdmap e1073: 12 total, 12 up, 11 in 2024-03-20T09:40:35.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:34 smithi122 ceph-mon[73495]: pgmap v2592: 47 pgs: 47 active+clean; 577 KiB data, 8.1 GiB used, 975 GiB / 983 GiB avail 2024-03-20T09:40:35.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:34 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/895719157' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-20T09:40:35.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:34 smithi026 ceph-mon[68309]: osdmap e1073: 12 total, 12 up, 11 in 2024-03-20T09:40:35.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:34 smithi026 ceph-mon[68309]: pgmap v2592: 47 pgs: 47 active+clean; 577 KiB data, 8.1 GiB used, 975 GiB / 983 GiB avail 2024-03-20T09:40:35.345 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:40:35.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:35.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:40:35.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:34 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/895719157' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-20T09:40:35.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:34 smithi115 ceph-mon[72476]: osdmap e1073: 12 total, 12 up, 11 in 2024-03-20T09:40:35.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:34 smithi115 ceph-mon[72476]: pgmap v2592: 47 pgs: 47 active+clean; 577 KiB data, 8.1 GiB used, 975 GiB / 983 GiB avail 2024-03-20T09:40:35.452 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:40:35.479 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:35.498 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:40:35.501 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:35.501 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:40:35.602 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:40:35.634 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:35.655 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:40:35.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:35.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:40:35.759 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:40:35.789 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:35.843 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:40:35.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:35.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:40:35.935 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:35 smithi026 ceph-mon[68309]: 1.13 scrub starts 2024-03-20T09:40:35.954 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:40:35.983 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:36.183 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:40:36.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:35 smithi122 ceph-mon[73495]: 1.13 scrub starts 2024-03-20T09:40:36.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:35 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:40:36.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:35 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:40:36.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:35 smithi122 ceph-mon[73495]: 1.13 scrub ok 2024-03-20T09:40:36.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:35 smithi122 ceph-mon[73495]: 1.14 scrub starts 2024-03-20T09:40:36.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:35 smithi122 ceph-mon[73495]: 1.14 scrub ok 2024-03-20T09:40:36.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:35 smithi122 ceph-mon[73495]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-20T09:40:36.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:35 smithi122 ceph-mon[73495]: osdmap e1074: 12 total, 12 up, 11 in 2024-03-20T09:40:36.186 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:35 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:40:36.186 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:35 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:40:36.186 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:35 smithi026 ceph-mon[68309]: 1.13 scrub ok 2024-03-20T09:40:36.186 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:35 smithi026 ceph-mon[68309]: 1.14 scrub starts 2024-03-20T09:40:36.186 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:35 smithi026 ceph-mon[68309]: 1.14 scrub ok 2024-03-20T09:40:36.186 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:35 smithi026 ceph-mon[68309]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-20T09:40:36.186 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:35 smithi026 ceph-mon[68309]: osdmap e1074: 12 total, 12 up, 11 in 2024-03-20T09:40:36.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:36.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:40:36.196 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-20T09:40:36.288 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:40:36.321 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:36.342 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:40:36.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:36.346 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:40:36.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:35 smithi115 ceph-mon[72476]: 1.13 scrub starts 2024-03-20T09:40:36.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:35 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:40:36.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:35 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:40:36.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:35 smithi115 ceph-mon[72476]: 1.13 scrub ok 2024-03-20T09:40:36.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:35 smithi115 ceph-mon[72476]: 1.14 scrub starts 2024-03-20T09:40:36.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:35 smithi115 ceph-mon[72476]: 1.14 scrub ok 2024-03-20T09:40:36.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:35 smithi115 ceph-mon[72476]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-20T09:40:36.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:35 smithi115 ceph-mon[72476]: osdmap e1074: 12 total, 12 up, 11 in 2024-03-20T09:40:36.446 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:40:36.478 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:36.499 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:40:36.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:36.502 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:40:36.603 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:40:36.632 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:36.652 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:40:36.654 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:36.655 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:40:36.756 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:40:36.784 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:36.897 INFO:teuthology.orchestra.run.smithi026.stderr:no valid command found; 10 closest matches: 2024-03-20T09:40:36.897 INFO:teuthology.orchestra.run.smithi026.stderr:0 2024-03-20T09:40:36.897 INFO:teuthology.orchestra.run.smithi026.stderr:1 2024-03-20T09:40:36.897 INFO:teuthology.orchestra.run.smithi026.stderr:2 2024-03-20T09:40:36.897 INFO:teuthology.orchestra.run.smithi026.stderr:abort 2024-03-20T09:40:36.897 INFO:teuthology.orchestra.run.smithi026.stderr:assert 2024-03-20T09:40:36.898 INFO:teuthology.orchestra.run.smithi026.stderr:bench [] [] [] [] 2024-03-20T09:40:36.898 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs debug_inject_read_zeros 2024-03-20T09:40:36.898 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs files list 2024-03-20T09:40:36.898 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs stats 2024-03-20T09:40:36.898 INFO:teuthology.orchestra.run.smithi026.stderr:bluestore allocator dump block 2024-03-20T09:40:36.898 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: invalid command 2024-03-20T09:40:37.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:36 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:40:37.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:36 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:40:37.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:36 smithi026 ceph-mon[68309]: pgmap v2594: 47 pgs: 47 active+clean; 577 KiB data, 8.1 GiB used, 975 GiB / 983 GiB avail 2024-03-20T09:40:37.232 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:40:37.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:37.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:40:37.337 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:40:37.367 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:37.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:36 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:40:37.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:36 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:40:37.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:36 smithi115 ceph-mon[72476]: pgmap v2594: 47 pgs: 47 active+clean; 577 KiB data, 8.1 GiB used, 975 GiB / 983 GiB avail 2024-03-20T09:40:37.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:36 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:40:37.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:36 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:40:37.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:36 smithi122 ceph-mon[73495]: pgmap v2594: 47 pgs: 47 active+clean; 577 KiB data, 8.1 GiB used, 975 GiB / 983 GiB avail 2024-03-20T09:40:37.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:37 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:40:37.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:37 smithi026 ceph-mon[68309]: osdmap e1075: 12 total, 12 up, 11 in 2024-03-20T09:40:37.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:37 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:40:37.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:37 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:40:37.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:37 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:40:37.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:37 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:40:37.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:37 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:40:37.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:37 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:40:37.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:37 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:40:37.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:37 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:40:37.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:37 smithi026 ceph-mon[68309]: osdmap e1076: 12 total, 12 up, 11 in 2024-03-20T09:40:38.017 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:40:38.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:38.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:40:38.124 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:40:38.157 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:38.178 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:40:38.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:38.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:40:38.283 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:40:38.313 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:38.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:37 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:40:38.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:37 smithi115 ceph-mon[72476]: osdmap e1075: 12 total, 12 up, 11 in 2024-03-20T09:40:38.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:37 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:40:38.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:37 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:40:38.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:37 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:40:38.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:37 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:40:38.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:37 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:40:38.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:37 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:40:38.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:37 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:40:38.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:37 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:40:38.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:37 smithi115 ceph-mon[72476]: osdmap e1076: 12 total, 12 up, 11 in 2024-03-20T09:40:38.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:37 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:40:38.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:37 smithi122 ceph-mon[73495]: osdmap e1075: 12 total, 12 up, 11 in 2024-03-20T09:40:38.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:37 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:40:38.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:37 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:40:38.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:37 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:40:38.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:37 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:40:38.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:37 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:40:38.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:37 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:40:38.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:37 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:40:38.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:37 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:40:38.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:37 smithi122 ceph-mon[73495]: osdmap e1076: 12 total, 12 up, 11 in 2024-03-20T09:40:38.949 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:40:39.163 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:40:39.165 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:38 smithi026 ceph-mon[68309]: 1.1d scrub starts 2024-03-20T09:40:39.165 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:38 smithi026 ceph-mon[68309]: 1.1d scrub ok 2024-03-20T09:40:39.165 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:38 smithi026 ceph-mon[68309]: pgmap v2597: 47 pgs: 47 active+clean; 577 KiB data, 8.1 GiB used, 975 GiB / 983 GiB avail 2024-03-20T09:40:39.165 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:38 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1569296522' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:40:39.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:39.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:40:39.191 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T09:40:39.191 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-20T09:40:39.192 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.4 'trim stale osdmaps' 2024-03-20T09:40:39.267 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:40:39.297 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:38 smithi115 ceph-mon[72476]: 1.1d scrub starts 2024-03-20T09:40:39.297 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:38 smithi115 ceph-mon[72476]: 1.1d scrub ok 2024-03-20T09:40:39.297 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:38 smithi115 ceph-mon[72476]: pgmap v2597: 47 pgs: 47 active+clean; 577 KiB data, 8.1 GiB used, 975 GiB / 983 GiB avail 2024-03-20T09:40:39.297 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:38 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1569296522' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:40:39.299 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:39.321 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:40:39.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:39.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:40:39.427 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:40:39.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:38 smithi122 ceph-mon[73495]: 1.1d scrub starts 2024-03-20T09:40:39.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:38 smithi122 ceph-mon[73495]: 1.1d scrub ok 2024-03-20T09:40:39.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:38 smithi122 ceph-mon[73495]: pgmap v2597: 47 pgs: 47 active+clean; 577 KiB data, 8.1 GiB used, 975 GiB / 983 GiB avail 2024-03-20T09:40:39.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:38 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1569296522' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:40:39.458 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:39.748 DEBUG:teuthology.orchestra.run.smithi122:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_7 bench 90 write' 2024-03-20T09:40:39.751 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-20T09:40:39.751 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-20T09:40:39.806 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:hints = 1 2024-03-20T09:40:39.807 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-20T09:40:39.807 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Object prefix: benchmark_data_smithi122_210164 2024-03-20T09:40:39.807 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T09:40:39.807 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 0 0 0 0 0 0 - 0 2024-03-20T09:40:39.903 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:40:39.903 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:40:39.909 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:40:39.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:39.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:40:40.014 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:40:40.055 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:40.166 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:39 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:40:40.167 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:39 smithi026 ceph-mon[68309]: Cluster is now healthy 2024-03-20T09:40:40.167 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:39 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1569296522' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:40:40.167 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:39 smithi026 ceph-mon[68309]: osdmap e1077: 12 total, 12 up, 11 in 2024-03-20T09:40:40.167 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:40:40.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:40.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:40:40.271 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:40:40.301 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:40.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:39 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:40:40.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:39 smithi115 ceph-mon[72476]: Cluster is now healthy 2024-03-20T09:40:40.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:39 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1569296522' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:40:40.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:39 smithi115 ceph-mon[72476]: osdmap e1077: 12 total, 12 up, 11 in 2024-03-20T09:40:40.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:39 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:40:40.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:39 smithi122 ceph-mon[73495]: Cluster is now healthy 2024-03-20T09:40:40.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:39 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1569296522' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:40:40.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:39 smithi122 ceph-mon[73495]: osdmap e1077: 12 total, 12 up, 11 in 2024-03-20T09:40:40.807 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 1 16 2936 2920 182.482 182.5 0.00279409 0.00546155 2024-03-20T09:40:41.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:40 smithi026 ceph-mon[68309]: pgmap v2599: 47 pgs: 47 active+clean; 11 MiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 1.8 MiB/s wr, 28 op/s 2024-03-20T09:40:41.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:40 smithi115 ceph-mon[72476]: pgmap v2599: 47 pgs: 47 active+clean; 11 MiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 1.8 MiB/s wr, 28 op/s 2024-03-20T09:40:41.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:40 smithi122 ceph-mon[73495]: pgmap v2599: 47 pgs: 47 active+clean; 11 MiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 1.8 MiB/s wr, 28 op/s 2024-03-20T09:40:41.761 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:40:41.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:41.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:40:41.807 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 2 16 7215 7199 224.948 267.438 0.00386341 0.00443479 2024-03-20T09:40:41.863 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:40:41.865 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:40:41.908 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:41.939 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:40:41.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:41.943 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:40:42.044 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:40:42.082 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:42.223 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:40:42.224 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:40:42.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:42.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:40:42.332 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:40:42.372 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:42.410 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:40:42.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:42.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:40:42.516 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:40:42.568 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:42.807 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 3 16 10727 10711 223.124 219.5 0.00156396 0.00435661 2024-03-20T09:40:43.725 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:40:43.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:43.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:40:43.808 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 4 16 13379 13363 208.751 165.75 0.00371488 0.00478411 2024-03-20T09:40:43.833 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:40:43.873 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:43.901 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:40:43.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:43.906 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:40:43.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: pgmap v2600: 47 pgs: 47 active+clean; 82 MiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 14 MiB/s wr, 216 op/s 2024-03-20T09:40:43.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=1459252283,config=2679774077,health=3897446751,kv=3109999694}) 2024-03-20T09:40:43.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {kv=4,logm=96} crc {kv=2611619683,logm=4061965953}) 2024-03-20T09:40:43.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3816000716}) 2024-03-20T09:40:43.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=552736292}) 2024-03-20T09:40:43.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4255880675}) 2024-03-20T09:40:43.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1403808191}) 2024-03-20T09:40:43.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=727904562}) 2024-03-20T09:40:43.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3721651530}) 2024-03-20T09:40:43.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2987650992}) 2024-03-20T09:40:43.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=899141402}) 2024-03-20T09:40:43.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=216708196}) 2024-03-20T09:40:43.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=430334118}) 2024-03-20T09:40:43.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=695745003}) 2024-03-20T09:40:43.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2756282447}) 2024-03-20T09:40:43.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2044370894}) 2024-03-20T09:40:43.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=976301773}) 2024-03-20T09:40:43.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=966901692}) 2024-03-20T09:40:43.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=551803912}) 2024-03-20T09:40:43.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2771065172}) 2024-03-20T09:40:43.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4168458402}) 2024-03-20T09:40:43.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4164972498}) 2024-03-20T09:40:43.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3744944030}) 2024-03-20T09:40:43.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3410586158}) 2024-03-20T09:40:43.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1112099785}) 2024-03-20T09:40:43.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2211309470}) 2024-03-20T09:40:43.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1807998637}) 2024-03-20T09:40:43.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2741208903}) 2024-03-20T09:40:43.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=755180952}) 2024-03-20T09:40:43.961 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3139720030}) 2024-03-20T09:40:43.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2851780669}) 2024-03-20T09:40:43.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2891984286}) 2024-03-20T09:40:43.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=535266206}) 2024-03-20T09:40:43.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1611260731}) 2024-03-20T09:40:43.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=579491267}) 2024-03-20T09:40:43.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=949713885}) 2024-03-20T09:40:43.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3854731370}) 2024-03-20T09:40:43.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2577368877}) 2024-03-20T09:40:43.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3623817511}) 2024-03-20T09:40:43.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3478384908}) 2024-03-20T09:40:43.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1672955173}) 2024-03-20T09:40:43.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3755971900}) 2024-03-20T09:40:43.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3811466872}) 2024-03-20T09:40:43.962 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=468574562}) 2024-03-20T09:40:43.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=182998958}) 2024-03-20T09:40:43.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3924658658}) 2024-03-20T09:40:43.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2130137810}) 2024-03-20T09:40:43.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3052355971}) 2024-03-20T09:40:43.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3578113571}) 2024-03-20T09:40:43.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=745395023}) 2024-03-20T09:40:43.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=705591931}) 2024-03-20T09:40:43.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2850406136}) 2024-03-20T09:40:43.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1298037928}) 2024-03-20T09:40:43.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1639468522}) 2024-03-20T09:40:43.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=969636141}) 2024-03-20T09:40:43.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=788498667}) 2024-03-20T09:40:43.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=677282633}) 2024-03-20T09:40:43.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=67620056}) 2024-03-20T09:40:43.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4196680172}) 2024-03-20T09:40:43.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2517833265}) 2024-03-20T09:40:43.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4114589883}) 2024-03-20T09:40:43.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2186541504}) 2024-03-20T09:40:43.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=7441233}) 2024-03-20T09:40:43.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2874290297}) 2024-03-20T09:40:43.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3454734623}) 2024-03-20T09:40:43.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4538534}) 2024-03-20T09:40:43.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1206625138}) 2024-03-20T09:40:43.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=743650594}) 2024-03-20T09:40:43.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=370081508}) 2024-03-20T09:40:43.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2572542877}) 2024-03-20T09:40:43.964 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3481819269}) 2024-03-20T09:40:43.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=286270145}) 2024-03-20T09:40:43.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1514788773}) 2024-03-20T09:40:43.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1456470230}) 2024-03-20T09:40:43.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1091863190}) 2024-03-20T09:40:43.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3791426839}) 2024-03-20T09:40:43.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3116810662}) 2024-03-20T09:40:43.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3383896874}) 2024-03-20T09:40:43.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1011751225}) 2024-03-20T09:40:43.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3447688358}) 2024-03-20T09:40:43.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1373609323}) 2024-03-20T09:40:43.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=65312836}) 2024-03-20T09:40:43.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1976617378}) 2024-03-20T09:40:43.965 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2307554122}) 2024-03-20T09:40:43.966 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2872794461}) 2024-03-20T09:40:43.966 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4265771982}) 2024-03-20T09:40:43.966 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4063742080}) 2024-03-20T09:40:43.966 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3408041018}) 2024-03-20T09:40:43.966 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3328835579}) 2024-03-20T09:40:43.966 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=921687109}) 2024-03-20T09:40:43.966 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1693229564}) 2024-03-20T09:40:43.966 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1986656128}) 2024-03-20T09:40:43.966 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2479383653}) 2024-03-20T09:40:43.966 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2621766837}) 2024-03-20T09:40:43.966 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=4,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=16,mon_config_key=56} crc {logm=4664024,mdsmap=3984425268,mgr=427508465,mgr_command_descs=3831600018,mgr_metadata=2702834764,mgrstat=3680531134,mon_config_key=2001311954}) 2024-03-20T09:40:43.966 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3010701535}) 2024-03-20T09:40:43.966 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2749986453}) 2024-03-20T09:40:43.966 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=25,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=57} crc {mon_config_key=1789472682,monmap=3481182261,osd_metadata=222046520,osd_pg_creating=3485773448,osdmap=2812041400}) 2024-03-20T09:40:43.967 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=81} crc {osdmap=2692972602}) 2024-03-20T09:40:44.007 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:40:44.049 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:44.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: pgmap v2600: 47 pgs: 47 active+clean; 82 MiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 14 MiB/s wr, 216 op/s 2024-03-20T09:40:44.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=1459252283,config=2679774077,health=3897446751,kv=3109999694}) 2024-03-20T09:40:44.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {kv=4,logm=96} crc {kv=2611619683,logm=4061965953}) 2024-03-20T09:40:44.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3816000716}) 2024-03-20T09:40:44.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=552736292}) 2024-03-20T09:40:44.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4255880675}) 2024-03-20T09:40:44.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1403808191}) 2024-03-20T09:40:44.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=727904562}) 2024-03-20T09:40:44.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3721651530}) 2024-03-20T09:40:44.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2987650992}) 2024-03-20T09:40:44.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=899141402}) 2024-03-20T09:40:44.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=216708196}) 2024-03-20T09:40:44.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=430334118}) 2024-03-20T09:40:44.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=695745003}) 2024-03-20T09:40:44.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2756282447}) 2024-03-20T09:40:44.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2044370894}) 2024-03-20T09:40:44.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=976301773}) 2024-03-20T09:40:44.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=966901692}) 2024-03-20T09:40:44.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=551803912}) 2024-03-20T09:40:44.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2771065172}) 2024-03-20T09:40:44.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4168458402}) 2024-03-20T09:40:44.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4164972498}) 2024-03-20T09:40:44.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3744944030}) 2024-03-20T09:40:44.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3410586158}) 2024-03-20T09:40:44.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1112099785}) 2024-03-20T09:40:44.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2211309470}) 2024-03-20T09:40:44.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1807998637}) 2024-03-20T09:40:44.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2741208903}) 2024-03-20T09:40:44.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=755180952}) 2024-03-20T09:40:44.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3139720030}) 2024-03-20T09:40:44.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2851780669}) 2024-03-20T09:40:44.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2891984286}) 2024-03-20T09:40:44.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=535266206}) 2024-03-20T09:40:44.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1611260731}) 2024-03-20T09:40:44.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=579491267}) 2024-03-20T09:40:44.149 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=949713885}) 2024-03-20T09:40:44.149 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3854731370}) 2024-03-20T09:40:44.149 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2577368877}) 2024-03-20T09:40:44.149 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3623817511}) 2024-03-20T09:40:44.149 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3478384908}) 2024-03-20T09:40:44.149 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1672955173}) 2024-03-20T09:40:44.149 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3755971900}) 2024-03-20T09:40:44.149 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3811466872}) 2024-03-20T09:40:44.149 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=468574562}) 2024-03-20T09:40:44.149 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=182998958}) 2024-03-20T09:40:44.149 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3924658658}) 2024-03-20T09:40:44.149 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2130137810}) 2024-03-20T09:40:44.149 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3052355971}) 2024-03-20T09:40:44.150 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3578113571}) 2024-03-20T09:40:44.150 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=745395023}) 2024-03-20T09:40:44.150 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=705591931}) 2024-03-20T09:40:44.150 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2850406136}) 2024-03-20T09:40:44.150 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1298037928}) 2024-03-20T09:40:44.150 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1639468522}) 2024-03-20T09:40:44.150 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=969636141}) 2024-03-20T09:40:44.150 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=788498667}) 2024-03-20T09:40:44.150 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=677282633}) 2024-03-20T09:40:44.150 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=67620056}) 2024-03-20T09:40:44.150 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4196680172}) 2024-03-20T09:40:44.150 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2517833265}) 2024-03-20T09:40:44.150 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4114589883}) 2024-03-20T09:40:44.150 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2186541504}) 2024-03-20T09:40:44.151 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=7441233}) 2024-03-20T09:40:44.151 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2874290297}) 2024-03-20T09:40:44.151 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3454734623}) 2024-03-20T09:40:44.151 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4538534}) 2024-03-20T09:40:44.151 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1206625138}) 2024-03-20T09:40:44.151 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=743650594}) 2024-03-20T09:40:44.151 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=370081508}) 2024-03-20T09:40:44.151 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2572542877}) 2024-03-20T09:40:44.151 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3481819269}) 2024-03-20T09:40:44.151 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=286270145}) 2024-03-20T09:40:44.151 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1514788773}) 2024-03-20T09:40:44.151 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1456470230}) 2024-03-20T09:40:44.151 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1091863190}) 2024-03-20T09:40:44.151 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3791426839}) 2024-03-20T09:40:44.151 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3116810662}) 2024-03-20T09:40:44.152 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3383896874}) 2024-03-20T09:40:44.152 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1011751225}) 2024-03-20T09:40:44.152 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3447688358}) 2024-03-20T09:40:44.152 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1373609323}) 2024-03-20T09:40:44.152 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=65312836}) 2024-03-20T09:40:44.152 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1976617378}) 2024-03-20T09:40:44.152 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2307554122}) 2024-03-20T09:40:44.152 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2872794461}) 2024-03-20T09:40:44.152 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4265771982}) 2024-03-20T09:40:44.152 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4063742080}) 2024-03-20T09:40:44.152 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3408041018}) 2024-03-20T09:40:44.152 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3328835579}) 2024-03-20T09:40:44.152 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=921687109}) 2024-03-20T09:40:44.152 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1693229564}) 2024-03-20T09:40:44.152 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1986656128}) 2024-03-20T09:40:44.153 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2479383653}) 2024-03-20T09:40:44.153 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2621766837}) 2024-03-20T09:40:44.153 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=4,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=16,mon_config_key=56} crc {logm=4664024,mdsmap=3984425268,mgr=427508465,mgr_command_descs=3831600018,mgr_metadata=2702834764,mgrstat=3680531134,mon_config_key=2001311954}) 2024-03-20T09:40:44.153 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3010701535}) 2024-03-20T09:40:44.153 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2749986453}) 2024-03-20T09:40:44.153 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=25,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=57} crc {mon_config_key=1789472682,monmap=3481182261,osd_metadata=222046520,osd_pg_creating=3485773448,osdmap=2812041400}) 2024-03-20T09:40:44.153 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=81} crc {osdmap=2692972602}) 2024-03-20T09:40:44.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: pgmap v2600: 47 pgs: 47 active+clean; 82 MiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 14 MiB/s wr, 216 op/s 2024-03-20T09:40:44.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=1459252283,config=2679774077,health=3897446751,kv=3109999694}) 2024-03-20T09:40:44.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {kv=4,logm=96} crc {kv=2611619683,logm=4061965953}) 2024-03-20T09:40:44.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3816000716}) 2024-03-20T09:40:44.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=552736292}) 2024-03-20T09:40:44.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4255880675}) 2024-03-20T09:40:44.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1403808191}) 2024-03-20T09:40:44.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=727904562}) 2024-03-20T09:40:44.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3721651530}) 2024-03-20T09:40:44.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2987650992}) 2024-03-20T09:40:44.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=899141402}) 2024-03-20T09:40:44.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=216708196}) 2024-03-20T09:40:44.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=430334118}) 2024-03-20T09:40:44.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=695745003}) 2024-03-20T09:40:44.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2756282447}) 2024-03-20T09:40:44.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2044370894}) 2024-03-20T09:40:44.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=976301773}) 2024-03-20T09:40:44.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=966901692}) 2024-03-20T09:40:44.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=551803912}) 2024-03-20T09:40:44.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2771065172}) 2024-03-20T09:40:44.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4168458402}) 2024-03-20T09:40:44.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4164972498}) 2024-03-20T09:40:44.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3744944030}) 2024-03-20T09:40:44.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3410586158}) 2024-03-20T09:40:44.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1112099785}) 2024-03-20T09:40:44.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2211309470}) 2024-03-20T09:40:44.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1807998637}) 2024-03-20T09:40:44.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2741208903}) 2024-03-20T09:40:44.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=755180952}) 2024-03-20T09:40:44.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3139720030}) 2024-03-20T09:40:44.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2851780669}) 2024-03-20T09:40:44.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2891984286}) 2024-03-20T09:40:44.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=535266206}) 2024-03-20T09:40:44.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1611260731}) 2024-03-20T09:40:44.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=579491267}) 2024-03-20T09:40:44.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=949713885}) 2024-03-20T09:40:44.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3854731370}) 2024-03-20T09:40:44.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2577368877}) 2024-03-20T09:40:44.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3623817511}) 2024-03-20T09:40:44.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3478384908}) 2024-03-20T09:40:44.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1672955173}) 2024-03-20T09:40:44.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3755971900}) 2024-03-20T09:40:44.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3811466872}) 2024-03-20T09:40:44.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=468574562}) 2024-03-20T09:40:44.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=182998958}) 2024-03-20T09:40:44.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3924658658}) 2024-03-20T09:40:44.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2130137810}) 2024-03-20T09:40:44.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3052355971}) 2024-03-20T09:40:44.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3578113571}) 2024-03-20T09:40:44.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=745395023}) 2024-03-20T09:40:44.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=705591931}) 2024-03-20T09:40:44.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2850406136}) 2024-03-20T09:40:44.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1298037928}) 2024-03-20T09:40:44.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1639468522}) 2024-03-20T09:40:44.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=969636141}) 2024-03-20T09:40:44.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=788498667}) 2024-03-20T09:40:44.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=677282633}) 2024-03-20T09:40:44.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=67620056}) 2024-03-20T09:40:44.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4196680172}) 2024-03-20T09:40:44.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2517833265}) 2024-03-20T09:40:44.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4114589883}) 2024-03-20T09:40:44.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2186541504}) 2024-03-20T09:40:44.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=7441233}) 2024-03-20T09:40:44.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2874290297}) 2024-03-20T09:40:44.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3454734623}) 2024-03-20T09:40:44.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4538534}) 2024-03-20T09:40:44.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1206625138}) 2024-03-20T09:40:44.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=743650594}) 2024-03-20T09:40:44.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=370081508}) 2024-03-20T09:40:44.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2572542877}) 2024-03-20T09:40:44.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3481819269}) 2024-03-20T09:40:44.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=286270145}) 2024-03-20T09:40:44.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1514788773}) 2024-03-20T09:40:44.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1456470230}) 2024-03-20T09:40:44.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1091863190}) 2024-03-20T09:40:44.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3791426839}) 2024-03-20T09:40:44.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3116810662}) 2024-03-20T09:40:44.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3383896874}) 2024-03-20T09:40:44.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1011751225}) 2024-03-20T09:40:44.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3447688358}) 2024-03-20T09:40:44.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1373609323}) 2024-03-20T09:40:44.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=65312836}) 2024-03-20T09:40:44.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1976617378}) 2024-03-20T09:40:44.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2307554122}) 2024-03-20T09:40:44.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2872794461}) 2024-03-20T09:40:44.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4265771982}) 2024-03-20T09:40:44.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4063742080}) 2024-03-20T09:40:44.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3408041018}) 2024-03-20T09:40:44.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3328835579}) 2024-03-20T09:40:44.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=921687109}) 2024-03-20T09:40:44.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1693229564}) 2024-03-20T09:40:44.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1986656128}) 2024-03-20T09:40:44.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2479383653}) 2024-03-20T09:40:44.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2621766837}) 2024-03-20T09:40:44.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=4,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=16,mon_config_key=56} crc {logm=4664024,mdsmap=3984425268,mgr=427508465,mgr_command_descs=3831600018,mgr_metadata=2702834764,mgrstat=3680531134,mon_config_key=2001311954}) 2024-03-20T09:40:44.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3010701535}) 2024-03-20T09:40:44.193 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2749986453}) 2024-03-20T09:40:44.193 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=25,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=57} crc {mon_config_key=1789472682,monmap=3481182261,osd_metadata=222046520,osd_pg_creating=3485773448,osdmap=2812041400}) 2024-03-20T09:40:44.193 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=81} crc {osdmap=2692972602}) 2024-03-20T09:40:44.808 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 5 16 16851 16835 210.392 217 0.00282805 0.00474708 2024-03-20T09:40:45.219 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:40:45.248 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:40:45.280 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:40:45.312 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:40:45.343 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:40:45.376 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:40:45.405 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:40:45.425 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:40:45.445 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:40:45.465 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:40:45.484 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:40:45.505 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:40:45.506 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:40:45.506 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:40:45.506 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:40:45.506 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:40:45.507 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:40:45.507 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:40:45.507 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:40:45.507 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:40:45.507 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:40:45.507 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:40:45.507 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:40:45.507 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:40:45.808 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 6 15 19545 19530 203.398 168.438 0.00915208 0.00491051 2024-03-20T09:40:45.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:45 smithi026 ceph-mon[68309]: pgmap v2601: 47 pgs: 47 active+clean; 345 MiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 45 MiB/s wr, 716 op/s 2024-03-20T09:40:45.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:45 smithi026 ceph-mon[68309]: 1.1c scrub starts 2024-03-20T09:40:45.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:45 smithi026 ceph-mon[68309]: 1.1c scrub ok 2024-03-20T09:40:45.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:45 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2154050539' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:40:46.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:45 smithi115 ceph-mon[72476]: pgmap v2601: 47 pgs: 47 active+clean; 345 MiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 45 MiB/s wr, 716 op/s 2024-03-20T09:40:46.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:45 smithi115 ceph-mon[72476]: 1.1c scrub starts 2024-03-20T09:40:46.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:45 smithi115 ceph-mon[72476]: 1.1c scrub ok 2024-03-20T09:40:46.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:45 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2154050539' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:40:46.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:45 smithi122 ceph-mon[73495]: pgmap v2601: 47 pgs: 47 active+clean; 345 MiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 45 MiB/s wr, 716 op/s 2024-03-20T09:40:46.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:45 smithi122 ceph-mon[73495]: 1.1c scrub starts 2024-03-20T09:40:46.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:45 smithi122 ceph-mon[73495]: 1.1c scrub ok 2024-03-20T09:40:46.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:45 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2154050539' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:40:46.411 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:40:46.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:46.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:40:46.524 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:40:46.567 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:46.603 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:40:46.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:46.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:40:46.716 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:40:46.757 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:46.785 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:40:46.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:46.789 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:40:46.808 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 7 16 22076 22060 196.929 158.125 0.00282784 0.00507199 2024-03-20T09:40:46.890 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:40:46.933 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:46.963 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:40:46.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:46.967 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:40:47.068 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:40:47.094 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:47.113 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:40:47.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:47.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:40:47.217 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:40:47.268 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:47.367 INFO:teuthology.orchestra.run.smithi026.stderr:no valid command found; 10 closest matches: 2024-03-20T09:40:47.368 INFO:teuthology.orchestra.run.smithi026.stderr:0 2024-03-20T09:40:47.368 INFO:teuthology.orchestra.run.smithi026.stderr:1 2024-03-20T09:40:47.368 INFO:teuthology.orchestra.run.smithi026.stderr:2 2024-03-20T09:40:47.368 INFO:teuthology.orchestra.run.smithi026.stderr:abort 2024-03-20T09:40:47.368 INFO:teuthology.orchestra.run.smithi026.stderr:assert 2024-03-20T09:40:47.368 INFO:teuthology.orchestra.run.smithi026.stderr:bench [] [] [] [] 2024-03-20T09:40:47.368 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs debug_inject_read_zeros 2024-03-20T09:40:47.368 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs files list 2024-03-20T09:40:47.368 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs stats 2024-03-20T09:40:47.368 INFO:teuthology.orchestra.run.smithi026.stderr:bluestore allocator dump block 2024-03-20T09:40:47.368 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: invalid command 2024-03-20T09:40:47.808 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 8 16 24428 24412 190.687 147 0.00304692 0.00523959 2024-03-20T09:40:47.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:47 smithi026 ceph-mon[68309]: 1.15 scrub starts 2024-03-20T09:40:47.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:47 smithi026 ceph-mon[68309]: pgmap v2602: 47 pgs: 47 active+clean; 1005 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-03-20T09:40:47.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:47 smithi026 ceph-mon[68309]: 1.15 scrub ok 2024-03-20T09:40:47.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:47 smithi026 ceph-mon[68309]: 1.7 scrub starts 2024-03-20T09:40:48.108 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:40:48.124 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T09:40:48.125 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-20T09:40:48.125 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.9 'trim stale osdmaps' 2024-03-20T09:40:48.127 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:40:48.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:48.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:40:48.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:47 smithi115 ceph-mon[72476]: 1.15 scrub starts 2024-03-20T09:40:48.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:47 smithi115 ceph-mon[72476]: pgmap v2602: 47 pgs: 47 active+clean; 1005 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-03-20T09:40:48.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:47 smithi115 ceph-mon[72476]: 1.15 scrub ok 2024-03-20T09:40:48.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:47 smithi115 ceph-mon[72476]: 1.7 scrub starts 2024-03-20T09:40:48.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:47 smithi122 ceph-mon[73495]: 1.15 scrub starts 2024-03-20T09:40:48.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:47 smithi122 ceph-mon[73495]: pgmap v2602: 47 pgs: 47 active+clean; 1005 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-03-20T09:40:48.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:47 smithi122 ceph-mon[73495]: 1.15 scrub ok 2024-03-20T09:40:48.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:47 smithi122 ceph-mon[73495]: 1.7 scrub starts 2024-03-20T09:40:48.232 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:40:48.277 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:48.302 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:40:48.305 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:48.305 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:40:48.406 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:40:48.451 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:40:48.451 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:40:48.471 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:48.609 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:40:48.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:48.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:40:48.715 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:40:48.755 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:48.808 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 9 15 26709 26694 185.346 142.625 0.00476187 0.00539122 2024-03-20T09:40:49.013 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:40:49.020 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:48 smithi026 ceph-mon[68309]: 1.7 scrub ok 2024-03-20T09:40:49.020 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:48 smithi026 ceph-mon[68309]: pgmap v2603: 47 pgs: 47 active+clean; 1.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-03-20T09:40:49.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:48 smithi122 ceph-mon[73495]: 1.7 scrub ok 2024-03-20T09:40:49.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:48 smithi122 ceph-mon[73495]: pgmap v2603: 47 pgs: 47 active+clean; 1.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-03-20T09:40:49.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:48 smithi115 ceph-mon[72476]: 1.7 scrub ok 2024-03-20T09:40:49.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:48 smithi115 ceph-mon[72476]: pgmap v2603: 47 pgs: 47 active+clean; 1.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-03-20T09:40:49.808 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 10 16 29321 29305 183.13 163.188 0.00511524 0.00545599 2024-03-20T09:40:50.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:49 smithi122 ceph-mon[73495]: 1.d scrub starts 2024-03-20T09:40:50.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:49 smithi026 ceph-mon[68309]: 1.d scrub starts 2024-03-20T09:40:50.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:49 smithi115 ceph-mon[72476]: 1.d scrub starts 2024-03-20T09:40:50.809 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 11 16 31350 31334 177.998 126.812 0.00595559 0.00561363 2024-03-20T09:40:51.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:50 smithi122 ceph-mon[73495]: 1.d scrub ok 2024-03-20T09:40:51.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:50 smithi122 ceph-mon[73495]: pgmap v2604: 47 pgs: 47 active+clean; 1.6 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 142 MiB/s wr, 2.26k op/s 2024-03-20T09:40:51.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:50 smithi026 ceph-mon[68309]: 1.d scrub ok 2024-03-20T09:40:51.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:50 smithi026 ceph-mon[68309]: pgmap v2604: 47 pgs: 47 active+clean; 1.6 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 142 MiB/s wr, 2.26k op/s 2024-03-20T09:40:51.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:50 smithi115 ceph-mon[72476]: 1.d scrub ok 2024-03-20T09:40:51.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:50 smithi115 ceph-mon[72476]: pgmap v2604: 47 pgs: 47 active+clean; 1.6 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 142 MiB/s wr, 2.26k op/s 2024-03-20T09:40:51.809 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 12 16 33992 33976 176.924 165.125 0.00460141 0.00564751 2024-03-20T09:40:51.934 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:40:51.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:51.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:40:51.990 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-20T09:40:52.039 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:40:52.080 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:52.108 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:40:52.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:52.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:40:52.213 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:40:52.253 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:52.281 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:40:52.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:52.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:40:52.387 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:40:52.424 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:52.469 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:40:52.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:52.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:40:52.479 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:40:52.543 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:40:52.575 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:40:52.614 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:52.661 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:40:52.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:52.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:40:52.770 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:40:52.809 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 13 16 36213 36197 173.992 138.812 0.00674589 0.00573597 2024-03-20T09:40:52.846 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:52.873 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:40:52.876 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:52.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:40:52.977 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:40:53.035 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:53.063 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:40:53.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:53.067 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:40:53.168 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:40:53.211 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:53.241 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:40:53.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:53.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:40:53.346 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:40:53.386 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:53.809 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 14 16 38509 38493 171.814 143.5 0.0067083 0.00581587 2024-03-20T09:40:53.853 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:40:53.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:53.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:40:53.890 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:40:53.958 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:40:53.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:53 smithi026 ceph-mon[68309]: 1.f scrub starts 2024-03-20T09:40:53.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:53 smithi026 ceph-mon[68309]: pgmap v2605: 47 pgs: 47 active+clean; 1.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 153 MiB/s wr, 2.44k op/s 2024-03-20T09:40:53.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:53 smithi026 ceph-mon[68309]: 1.f scrub ok 2024-03-20T09:40:53.997 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:54.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:53 smithi115 ceph-mon[72476]: 1.f scrub starts 2024-03-20T09:40:54.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:53 smithi115 ceph-mon[72476]: pgmap v2605: 47 pgs: 47 active+clean; 1.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 153 MiB/s wr, 2.44k op/s 2024-03-20T09:40:54.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:53 smithi115 ceph-mon[72476]: 1.f scrub ok 2024-03-20T09:40:54.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:53 smithi122 ceph-mon[73495]: 1.f scrub starts 2024-03-20T09:40:54.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:53 smithi122 ceph-mon[73495]: pgmap v2605: 47 pgs: 47 active+clean; 1.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 153 MiB/s wr, 2.44k op/s 2024-03-20T09:40:54.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:53 smithi122 ceph-mon[73495]: 1.f scrub ok 2024-03-20T09:40:54.809 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 15 16 40872 40856 170.205 147.688 0.00493237 0.00587016 2024-03-20T09:40:55.160 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:40:55.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:55.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:40:55.267 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:40:55.313 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:55.346 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:40:55.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:55.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:40:55.453 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:40:55.503 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:55.538 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:40:55.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:55.543 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:40:55.644 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:40:55.688 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:55.812 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 16 16 43103 43087 168.271 139.438 0.00342298 0.00593904 2024-03-20T09:40:55.934 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:55 smithi122 ceph-mon[73495]: pgmap v2606: 47 pgs: 47 active+clean; 2.0 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 166 MiB/s wr, 2.65k op/s 2024-03-20T09:40:55.935 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:55 smithi122 ceph-mon[73495]: 1.18 scrub starts 2024-03-20T09:40:55.935 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:55 smithi122 ceph-mon[73495]: 1.18 scrub ok 2024-03-20T09:40:55.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:55 smithi026 ceph-mon[68309]: pgmap v2606: 47 pgs: 47 active+clean; 2.0 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 166 MiB/s wr, 2.65k op/s 2024-03-20T09:40:55.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:55 smithi026 ceph-mon[68309]: 1.18 scrub starts 2024-03-20T09:40:55.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:55 smithi026 ceph-mon[68309]: 1.18 scrub ok 2024-03-20T09:40:56.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:55 smithi115 ceph-mon[72476]: pgmap v2606: 47 pgs: 47 active+clean; 2.0 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 166 MiB/s wr, 2.65k op/s 2024-03-20T09:40:56.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:55 smithi115 ceph-mon[72476]: 1.18 scrub starts 2024-03-20T09:40:56.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:55 smithi115 ceph-mon[72476]: 1.18 scrub ok 2024-03-20T09:40:56.446 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:40:56.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:56.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:40:56.555 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:40:56.591 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:40:56.592 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:40:56.592 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:40:56.592 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:40:56.592 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:40:56.592 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:40:56.592 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:40:56.592 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:40:56.592 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:40:56.592 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:40:56.592 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:40:56.610 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:56.610 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:40:56.614 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:40:56.614 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:40:56.615 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:40:56.615 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:40:56.615 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:40:56.615 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:40:56.615 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:40:56.615 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:40:56.615 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:40:56.615 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:40:56.615 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:40:56.615 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:40:56.636 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:40:56.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:40:56.642 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:40:56.709 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:56 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1461881101' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:40:56.743 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:40:56.796 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:40:56.810 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 17 16 45287 45271 166.401 136.5 0.00727739 0.0060051 2024-03-20T09:40:56.935 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:56 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1461881101' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:40:57.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:56 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1461881101' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:40:57.811 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 18 16 47356 47340 164.341 129.312 0.0061527 0.00608104 2024-03-20T09:40:58.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:57 smithi115 ceph-mon[72476]: pgmap v2607: 47 pgs: 47 active+clean; 2.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 182 MiB/s wr, 2.91k op/s 2024-03-20T09:40:58.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:57 smithi115 ceph-mon[72476]: 1.b scrub starts 2024-03-20T09:40:58.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:57 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/101707796' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:40:58.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:57 smithi122 ceph-mon[73495]: pgmap v2607: 47 pgs: 47 active+clean; 2.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 182 MiB/s wr, 2.91k op/s 2024-03-20T09:40:58.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:57 smithi122 ceph-mon[73495]: 1.b scrub starts 2024-03-20T09:40:58.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:57 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/101707796' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:40:58.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:57 smithi026 ceph-mon[68309]: pgmap v2607: 47 pgs: 47 active+clean; 2.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 182 MiB/s wr, 2.91k op/s 2024-03-20T09:40:58.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:57 smithi026 ceph-mon[68309]: 1.b scrub starts 2024-03-20T09:40:58.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:57 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/101707796' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:40:58.402 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:40:58.811 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 19 16 49282 49266 162.026 120.375 0.00231673 0.00616445 2024-03-20T09:40:59.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:58 smithi115 ceph-mon[72476]: 1.b scrub ok 2024-03-20T09:40:59.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:58 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:40:59.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:58 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/101707796' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:40:59.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:58 smithi115 ceph-mon[72476]: osdmap e1078: 12 total, 12 up, 11 in 2024-03-20T09:40:59.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:58 smithi115 ceph-mon[72476]: osdmap e1079: 12 total, 12 up, 11 in 2024-03-20T09:40:59.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:58 smithi122 ceph-mon[73495]: 1.b scrub ok 2024-03-20T09:40:59.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:58 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:40:59.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:58 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/101707796' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:40:59.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:58 smithi122 ceph-mon[73495]: osdmap e1078: 12 total, 12 up, 11 in 2024-03-20T09:40:59.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:58 smithi122 ceph-mon[73495]: osdmap e1079: 12 total, 12 up, 11 in 2024-03-20T09:40:59.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:58 smithi026 ceph-mon[68309]: 1.b scrub ok 2024-03-20T09:40:59.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:58 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:40:59.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:58 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/101707796' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:40:59.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:58 smithi026 ceph-mon[68309]: osdmap e1078: 12 total, 12 up, 11 in 2024-03-20T09:40:59.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:58 smithi026 ceph-mon[68309]: osdmap e1079: 12 total, 12 up, 11 in 2024-03-20T09:40:59.811 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:2024-03-20T09:40:59.823772+0000 min lat: 0.00113956 max lat: 0.21195 avg lat: 0.0062719 2024-03-20T09:40:59.811 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T09:40:59.811 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 20 15 51016 51001 159.347 108.438 0.011114 0.0062719 2024-03-20T09:40:59.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:40:59 smithi026 ceph-mon[68309]: pgmap v2610: 47 pgs: 47 active+clean; 2.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-03-20T09:40:59.989 INFO:teuthology.orchestra.run.smithi026.stderr:no valid command found; 10 closest matches: 2024-03-20T09:40:59.990 INFO:teuthology.orchestra.run.smithi026.stderr:0 2024-03-20T09:40:59.990 INFO:teuthology.orchestra.run.smithi026.stderr:1 2024-03-20T09:40:59.990 INFO:teuthology.orchestra.run.smithi026.stderr:2 2024-03-20T09:40:59.990 INFO:teuthology.orchestra.run.smithi026.stderr:abort 2024-03-20T09:40:59.990 INFO:teuthology.orchestra.run.smithi026.stderr:assert 2024-03-20T09:40:59.990 INFO:teuthology.orchestra.run.smithi026.stderr:bench [] [] [] [] 2024-03-20T09:40:59.990 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs debug_inject_read_zeros 2024-03-20T09:40:59.990 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs files list 2024-03-20T09:40:59.990 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs stats 2024-03-20T09:40:59.990 INFO:teuthology.orchestra.run.smithi026.stderr:bluestore allocator dump block 2024-03-20T09:40:59.990 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: invalid command 2024-03-20T09:41:00.006 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:41:00.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:00.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:41:00.110 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:41:00.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:40:59 smithi115 ceph-mon[72476]: pgmap v2610: 47 pgs: 47 active+clean; 2.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-03-20T09:41:00.151 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:00.183 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:41:00.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:40:59 smithi122 ceph-mon[73495]: pgmap v2610: 47 pgs: 47 active+clean; 2.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 125 MiB/s wr, 2.00k op/s 2024-03-20T09:41:00.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:00.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:41:00.290 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:41:00.316 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:00.336 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:41:00.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:00.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:41:00.440 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:41:00.478 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:00.517 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:41:00.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:00.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:41:00.623 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:41:00.665 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:00.695 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:41:00.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:00.700 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:41:00.801 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:41:00.811 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 21 16 52831 52815 157.157 113.375 0.00468704 0.0063593 2024-03-20T09:41:00.848 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:00.872 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:41:00.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:00.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:41:00.978 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:41:01.020 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:01.812 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 22 16 54829 54813 155.68 124.875 0.0129755 0.00641952 2024-03-20T09:41:02.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:01 smithi115 ceph-mon[72476]: pgmap v2611: 47 pgs: 47 active+clean; 3.0 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 152 MiB/s wr, 2.42k op/s 2024-03-20T09:41:02.166 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:01 smithi026 ceph-mon[68309]: pgmap v2611: 47 pgs: 47 active+clean; 3.0 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 152 MiB/s wr, 2.42k op/s 2024-03-20T09:41:02.178 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:41:02.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:02.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:41:02.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:01 smithi122 ceph-mon[73495]: pgmap v2611: 47 pgs: 47 active+clean; 3.0 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 152 MiB/s wr, 2.42k op/s 2024-03-20T09:41:02.207 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T09:41:02.207 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-20T09:41:02.207 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.10 'trim stale osdmaps' 2024-03-20T09:41:02.283 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:41:02.333 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:02.360 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:41:02.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:02.364 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:41:02.465 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:41:02.514 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:02.546 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:41:02.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:02.552 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:41:02.557 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:41:02.653 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:41:02.716 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:02.751 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:41:02.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:02.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:41:02.813 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 23 16 56682 56666 153.946 115.812 0.00433603 0.00649186 2024-03-20T09:41:02.830 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:41:02.830 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-03-20T09:41:02.830 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-03-20T09:41:02.830 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:41:02.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.830 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:213755 7.15 7:a872df88:::benchmark_data_smithi122_187077_object87144:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e936)", 2024-03-20T09:41:02.830 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:36.132651+0000", 2024-03-20T09:41:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 386.70612054999998, 2024-03-20T09:41:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2435835669999999, 2024-03-20T09:41:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:41:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:41:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:41:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:41:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 213755 2024-03-20T09:41:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:02.831 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132651+0000", 2024-03-20T09:41:02.832 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.832 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:02.832 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132651+0000", 2024-03-20T09:41:02.832 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.832 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:02.832 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132652+0000", 2024-03-20T09:41:02.832 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3120000000000001e-06 2024-03-20T09:41:02.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132659+0000", 2024-03-20T09:41:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3799999999999999e-06 2024-03-20T09:41:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132675+0000", 2024-03-20T09:41:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6294e-05 2024-03-20T09:41:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132683+0000", 2024-03-20T09:41:02.833 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2800000000000003e-06 2024-03-20T09:41:02.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.834 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:02.834 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132746+0000", 2024-03-20T09:41:02.834 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2664999999999997e-05 2024-03-20T09:41:02.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.834 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op must wait for map", 2024-03-20T09:41:02.834 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132753+0000", 2024-03-20T09:41:02.834 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1720000000000001e-06 2024-03-20T09:41:02.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.834 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:02.835 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.136252+0000", 2024-03-20T09:41:02.835 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.003499173 2024-03-20T09:41:02.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.835 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:41:02.835 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.136261+0000", 2024-03-20T09:41:02.835 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8799999999999997e-06 2024-03-20T09:41:02.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.835 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:02.835 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.376116+0000", 2024-03-20T09:41:02.835 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2398546189999999 2024-03-20T09:41:02.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.836 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:02.836 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.376235+0000", 2024-03-20T09:41:02.836 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000118792 2024-03-20T09:41:02.836 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.836 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:02.836 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.836 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:213794 7.16 7:69d71a3c:::benchmark_data_smithi122_187077_object87183:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e936)", 2024-03-20T09:41:02.836 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:36.132714+0000", 2024-03-20T09:41:02.836 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 386.70605762999998, 2024-03-20T09:41:02.836 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.226967184, 2024-03-20T09:41:02.836 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:02.836 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:02.836 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:41:02.837 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:41:02.837 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:41:02.837 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 213794 2024-03-20T09:41:02.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.837 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:02.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.837 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:02.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132714+0000", 2024-03-20T09:41:02.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.837 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:02.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132714+0000", 2024-03-20T09:41:02.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:02.838 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132715+0000", 2024-03-20T09:41:02.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6499999999999999e-07 2024-03-20T09:41:02.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:02.838 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132718+0000", 2024-03-20T09:41:02.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3139999999999999e-06 2024-03-20T09:41:02.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.839 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:02.839 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132726+0000", 2024-03-20T09:41:02.839 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2109999999999998e-06 2024-03-20T09:41:02.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.839 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:02.839 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132733+0000", 2024-03-20T09:41:02.839 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9120000000000001e-06 2024-03-20T09:41:02.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.839 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:02.839 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132931+0000", 2024-03-20T09:41:02.839 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019818599999999999 2024-03-20T09:41:02.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.840 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op must wait for map", 2024-03-20T09:41:02.840 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132938+0000", 2024-03-20T09:41:02.840 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8630000000000002e-06 2024-03-20T09:41:02.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.840 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:02.840 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.134074+0000", 2024-03-20T09:41:02.840 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001136319 2024-03-20T09:41:02.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.840 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:41:02.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.134083+0000", 2024-03-20T09:41:02.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9740000000000008e-06 2024-03-20T09:41:02.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.841 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:02.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.332852+0000", 2024-03-20T09:41:02.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1987690790000001 2024-03-20T09:41:02.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.841 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:02.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.332997+0000", 2024-03-20T09:41:02.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000144191 2024-03-20T09:41:02.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.841 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,7,8,11", 2024-03-20T09:41:02.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.333138+0000", 2024-03-20T09:41:02.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014141499999999999 2024-03-20T09:41:02.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.842 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:41:02.842 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.337922+0000", 2024-03-20T09:41:02.842 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0047840570000000004 2024-03-20T09:41:02.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.842 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:41:02.842 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.338331+0000", 2024-03-20T09:41:02.842 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00040852899999999998 2024-03-20T09:41:02.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.842 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:41:02.842 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.347271+0000", 2024-03-20T09:41:02.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0089403139999999996 2024-03-20T09:41:02.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:41:02.843 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.347592+0000", 2024-03-20T09:41:02.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00032139799999999997 2024-03-20T09:41:02.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:41:02.843 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.359473+0000", 2024-03-20T09:41:02.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011880658000000001 2024-03-20T09:41:02.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:02.843 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.359539+0000", 2024-03-20T09:41:02.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5635000000000004e-05 2024-03-20T09:41:02.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.844 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:02.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.359681+0000", 2024-03-20T09:41:02.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000142564 2024-03-20T09:41:02.844 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.844 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:02.844 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.844 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:213795 7.14 7:29024fd8:::benchmark_data_smithi122_187077_object87184:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e936)", 2024-03-20T09:41:02.844 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:36.132749+0000", 2024-03-20T09:41:02.844 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 386.70602285699999, 2024-03-20T09:41:02.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.217728784, 2024-03-20T09:41:02.844 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:02.844 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:02.845 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:41:02.845 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:41:02.845 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:41:02.845 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 213795 2024-03-20T09:41:02.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.845 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:02.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.845 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:02.845 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132749+0000", 2024-03-20T09:41:02.845 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.845 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.845 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.845 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:02.845 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132749+0000", 2024-03-20T09:41:02.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:02.846 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132749+0000", 2024-03-20T09:41:02.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8800000000000003e-07 2024-03-20T09:41:02.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:02.847 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132752+0000", 2024-03-20T09:41:02.847 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5009999999999999e-06 2024-03-20T09:41:02.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.847 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:02.847 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132758+0000", 2024-03-20T09:41:02.847 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4999999999999996e-06 2024-03-20T09:41:02.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.847 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:02.847 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132765+0000", 2024-03-20T09:41:02.847 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4999999999999996e-06 2024-03-20T09:41:02.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.848 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:02.848 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.133009+0000", 2024-03-20T09:41:02.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00024404 2024-03-20T09:41:02.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.848 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op must wait for map", 2024-03-20T09:41:02.848 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.133016+0000", 2024-03-20T09:41:02.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5270000000000001e-06 2024-03-20T09:41:02.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.848 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:02.848 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.135692+0000", 2024-03-20T09:41:02.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026753390000000001 2024-03-20T09:41:02.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.849 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:41:02.849 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.135701+0000", 2024-03-20T09:41:02.849 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8519999999999993e-06 2024-03-20T09:41:02.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.849 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:02.849 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.342134+0000", 2024-03-20T09:41:02.849 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.206433724 2024-03-20T09:41:02.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.849 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:02.849 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.342264+0000", 2024-03-20T09:41:02.849 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012935999999999999 2024-03-20T09:41:02.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 1,4,5,11", 2024-03-20T09:41:02.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.342638+0000", 2024-03-20T09:41:02.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00037430800000000001 2024-03-20T09:41:02.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:41:02.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.349288+0000", 2024-03-20T09:41:02.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0066500750000000001 2024-03-20T09:41:02.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:41:02.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.349303+0000", 2024-03-20T09:41:02.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5293000000000001e-05 2024-03-20T09:41:02.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:41:02.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.349339+0000", 2024-03-20T09:41:02.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5503999999999998e-05 2024-03-20T09:41:02.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:41:02.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.350180+0000", 2024-03-20T09:41:02.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00084116400000000004 2024-03-20T09:41:02.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:41:02.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.350306+0000", 2024-03-20T09:41:02.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012647 2024-03-20T09:41:02.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:02.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.350373+0000", 2024-03-20T09:41:02.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6805999999999997e-05 2024-03-20T09:41:02.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:02.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.350478+0000", 2024-03-20T09:41:02.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010433300000000001 2024-03-20T09:41:02.852 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.852 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:02.852 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.853 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.853 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:213809 7.14 7:2fbe0354:::benchmark_data_smithi122_187077_object87198:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e936)", 2024-03-20T09:41:02.853 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:36.132780+0000", 2024-03-20T09:41:02.853 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 386.70599121200001, 2024-03-20T09:41:02.853 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2180650239999999, 2024-03-20T09:41:02.853 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:02.853 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:02.853 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:41:02.853 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:41:02.853 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:41:02.853 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 213809 2024-03-20T09:41:02.853 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.853 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:02.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.854 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:02.854 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132780+0000", 2024-03-20T09:41:02.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.854 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:02.854 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132780+0000", 2024-03-20T09:41:02.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.854 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.854 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:02.854 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132781+0000", 2024-03-20T09:41:02.854 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6199999999999998e-07 2024-03-20T09:41:02.854 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:02.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132784+0000", 2024-03-20T09:41:02.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6529999999999998e-06 2024-03-20T09:41:02.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:02.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132791+0000", 2024-03-20T09:41:02.855 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2420000000000003e-06 2024-03-20T09:41:02.855 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.855 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.855 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:02.855 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132802+0000", 2024-03-20T09:41:02.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0694e-05 2024-03-20T09:41:02.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:02.856 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.133053+0000", 2024-03-20T09:41:02.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025135799999999997 2024-03-20T09:41:02.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.856 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.856 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting_for_map not empty", 2024-03-20T09:41:02.856 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.133058+0000", 2024-03-20T09:41:02.856 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4129999999999997e-06 2024-03-20T09:41:02.856 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.857 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:02.857 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.135784+0000", 2024-03-20T09:41:02.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027260119999999999 2024-03-20T09:41:02.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.857 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:41:02.857 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.135792+0000", 2024-03-20T09:41:02.857 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6720000000000004e-06 2024-03-20T09:41:02.857 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.857 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.857 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:02.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.345302+0000", 2024-03-20T09:41:02.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.209510152 2024-03-20T09:41:02.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.858 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:02.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.345448+0000", 2024-03-20T09:41:02.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014538800000000001 2024-03-20T09:41:02.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.858 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 1,4,5,11", 2024-03-20T09:41:02.858 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.345850+0000", 2024-03-20T09:41:02.858 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00040290999999999998 2024-03-20T09:41:02.858 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.858 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:41:02.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.349380+0000", 2024-03-20T09:41:02.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0035300259999999999 2024-03-20T09:41:02.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:41:02.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.349418+0000", 2024-03-20T09:41:02.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7716999999999998e-05 2024-03-20T09:41:02.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.859 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:41:02.859 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.349513+0000", 2024-03-20T09:41:02.859 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4426999999999996e-05 2024-03-20T09:41:02.859 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.859 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.860 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:41:02.860 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.350598+0000", 2024-03-20T09:41:02.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0010858199999999999 2024-03-20T09:41:02.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.860 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:41:02.860 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.350643+0000", 2024-03-20T09:41:02.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4882000000000002e-05 2024-03-20T09:41:02.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.860 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:02.860 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.350705+0000", 2024-03-20T09:41:02.860 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2027999999999996e-05 2024-03-20T09:41:02.860 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.860 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.860 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:02.861 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.350845+0000", 2024-03-20T09:41:02.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000140168 2024-03-20T09:41:02.861 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.861 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:02.861 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.861 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.861 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.861 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:213816 7.15 7:ab5b4b63:::benchmark_data_smithi122_187077_object87205:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e936)", 2024-03-20T09:41:02.861 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:36.132816+0000", 2024-03-20T09:41:02.861 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 386.70595576699998, 2024-03-20T09:41:02.861 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.255353135, 2024-03-20T09:41:02.861 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:02.861 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:02.861 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:41:02.861 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:41:02.861 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:41:02.862 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 213816 2024-03-20T09:41:02.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.862 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:02.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.862 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:02.862 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132816+0000", 2024-03-20T09:41:02.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.862 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:02.862 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132816+0000", 2024-03-20T09:41:02.862 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.862 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.862 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:02.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132817+0000", 2024-03-20T09:41:02.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0800000000000004e-07 2024-03-20T09:41:02.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:02.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132820+0000", 2024-03-20T09:41:02.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1070000000000001e-06 2024-03-20T09:41:02.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:02.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132828+0000", 2024-03-20T09:41:02.863 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9340000000000006e-06 2024-03-20T09:41:02.863 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.863 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.863 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:02.863 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132834+0000", 2024-03-20T09:41:02.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3500000000000002e-06 2024-03-20T09:41:02.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:02.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132987+0000", 2024-03-20T09:41:02.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015329499999999999 2024-03-20T09:41:02.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting_for_map not empty", 2024-03-20T09:41:02.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132990+0000", 2024-03-20T09:41:02.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0249999999999998e-06 2024-03-20T09:41:02.864 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.864 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.864 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:02.864 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.141152+0000", 2024-03-20T09:41:02.864 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0081620890000000008 2024-03-20T09:41:02.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.865 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:41:02.865 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.141162+0000", 2024-03-20T09:41:02.865 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4900000000000006e-06 2024-03-20T09:41:02.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.865 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:02.865 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.376864+0000", 2024-03-20T09:41:02.865 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2357016459999999 2024-03-20T09:41:02.865 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.865 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.865 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:02.865 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.376973+0000", 2024-03-20T09:41:02.865 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000109552 2024-03-20T09:41:02.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.866 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 1,7,8,9,11", 2024-03-20T09:41:02.866 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.377366+0000", 2024-03-20T09:41:02.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00039325299999999998 2024-03-20T09:41:02.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.866 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:41:02.866 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.382051+0000", 2024-03-20T09:41:02.866 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004684517 2024-03-20T09:41:02.866 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.866 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.866 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:41:02.866 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.382100+0000", 2024-03-20T09:41:02.867 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9307000000000001e-05 2024-03-20T09:41:02.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.867 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:41:02.867 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.385223+0000", 2024-03-20T09:41:02.867 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0031232389999999999 2024-03-20T09:41:02.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.867 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:41:02.867 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.385240+0000", 2024-03-20T09:41:02.867 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6174000000000001e-05 2024-03-20T09:41:02.867 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.867 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.867 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:41:02.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.388008+0000", 2024-03-20T09:41:02.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027688019999999999 2024-03-20T09:41:02.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:41:02.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.388056+0000", 2024-03-20T09:41:02.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7735000000000003e-05 2024-03-20T09:41:02.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:02.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.388094+0000", 2024-03-20T09:41:02.868 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8309000000000002e-05 2024-03-20T09:41:02.868 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.868 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.868 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:02.868 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.388169+0000", 2024-03-20T09:41:02.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4603000000000004e-05 2024-03-20T09:41:02.869 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.869 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:02.869 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.869 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.869 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.869 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:213826 7.16 7:6e7a6238:::benchmark_data_smithi122_187077_object87215:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e936)", 2024-03-20T09:41:02.869 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:36.132847+0000", 2024-03-20T09:41:02.869 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 386.70592507200001, 2024-03-20T09:41:02.869 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2273559549999999, 2024-03-20T09:41:02.869 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:02.869 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:02.869 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:41:02.869 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:41:02.869 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:41:02.870 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 213826 2024-03-20T09:41:02.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.870 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:02.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.870 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:02.870 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132847+0000", 2024-03-20T09:41:02.870 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.870 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:02.870 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132847+0000", 2024-03-20T09:41:02.870 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.870 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.870 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.871 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:02.871 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132847+0000", 2024-03-20T09:41:02.871 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5999999999999999e-07 2024-03-20T09:41:02.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.871 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:02.871 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132849+0000", 2024-03-20T09:41:02.871 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.424e-06 2024-03-20T09:41:02.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.871 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:02.871 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132857+0000", 2024-03-20T09:41:02.871 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4590000000000001e-06 2024-03-20T09:41:02.871 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.871 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.872 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:02.872 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132862+0000", 2024-03-20T09:41:02.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4129999999999997e-06 2024-03-20T09:41:02.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.872 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:02.872 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132965+0000", 2024-03-20T09:41:02.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000102144 2024-03-20T09:41:02.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.872 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting_for_map not empty", 2024-03-20T09:41:02.872 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132970+0000", 2024-03-20T09:41:02.872 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9039999999999997e-06 2024-03-20T09:41:02.872 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.872 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:02.873 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.134113+0000", 2024-03-20T09:41:02.873 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001142421 2024-03-20T09:41:02.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:41:02.873 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.134121+0000", 2024-03-20T09:41:02.873 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9410000000000007e-06 2024-03-20T09:41:02.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.873 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.873 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:02.873 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.335484+0000", 2024-03-20T09:41:02.873 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2013633829999999 2024-03-20T09:41:02.873 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.874 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:02.874 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.335622+0000", 2024-03-20T09:41:02.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013765199999999999 2024-03-20T09:41:02.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.874 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 3,7,8,11", 2024-03-20T09:41:02.874 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.335874+0000", 2024-03-20T09:41:02.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025246399999999999 2024-03-20T09:41:02.874 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.874 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.874 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:41:02.874 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.338823+0000", 2024-03-20T09:41:02.874 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0029486400000000002 2024-03-20T09:41:02.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:41:02.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.344311+0000", 2024-03-20T09:41:02.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054885280000000003 2024-03-20T09:41:02.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:41:02.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.347296+0000", 2024-03-20T09:41:02.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0029843869999999998 2024-03-20T09:41:02.875 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.875 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.875 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:41:02.875 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.347633+0000", 2024-03-20T09:41:02.875 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00033691899999999998 2024-03-20T09:41:02.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:41:02.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.360012+0000", 2024-03-20T09:41:02.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012379028 2024-03-20T09:41:02.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:02.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.360077+0000", 2024-03-20T09:41:02.876 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4912999999999993e-05 2024-03-20T09:41:02.876 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.876 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.876 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:02.876 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.360203+0000", 2024-03-20T09:41:02.877 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012587500000000001 2024-03-20T09:41:02.877 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.877 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:02.877 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.877 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.877 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.877 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:213827 7.15 7:af646400:::benchmark_data_smithi122_187077_object87216:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e936)", 2024-03-20T09:41:02.877 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:36.132869+0000", 2024-03-20T09:41:02.877 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 386.70590266300002, 2024-03-20T09:41:02.877 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2728044220000001, 2024-03-20T09:41:02.877 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:02.877 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:02.877 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:41:02.877 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:41:02.878 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:41:02.878 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 213827 2024-03-20T09:41:02.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.878 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:02.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.878 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:02.878 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132869+0000", 2024-03-20T09:41:02.878 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.878 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:02.878 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132869+0000", 2024-03-20T09:41:02.878 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.878 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.879 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:02.879 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132869+0000", 2024-03-20T09:41:02.879 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9099999999999999e-07 2024-03-20T09:41:02.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.879 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:02.879 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132872+0000", 2024-03-20T09:41:02.879 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3420000000000002e-06 2024-03-20T09:41:02.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.879 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.879 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:02.879 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132878+0000", 2024-03-20T09:41:02.879 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4309999999999999e-06 2024-03-20T09:41:02.879 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:02.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.132884+0000", 2024-03-20T09:41:02.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5450000000000003e-06 2024-03-20T09:41:02.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:02.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.133012+0000", 2024-03-20T09:41:02.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012817500000000001 2024-03-20T09:41:02.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting_for_map not empty", 2024-03-20T09:41:02.880 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.133017+0000", 2024-03-20T09:41:02.880 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1590000000000001e-06 2024-03-20T09:41:02.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.880 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.880 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:02.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.141188+0000", 2024-03-20T09:41:02.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0081712980000000004 2024-03-20T09:41:02.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:41:02.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.141195+0000", 2024-03-20T09:41:02.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0999999999999998e-06 2024-03-20T09:41:02.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.881 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:02.881 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.379506+0000", 2024-03-20T09:41:02.881 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.238310228 2024-03-20T09:41:02.881 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.881 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.882 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:02.882 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.379646+0000", 2024-03-20T09:41:02.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000139877 2024-03-20T09:41:02.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.882 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 1,7,8,9,11", 2024-03-20T09:41:02.882 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.379998+0000", 2024-03-20T09:41:02.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00035252500000000002 2024-03-20T09:41:02.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.882 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.882 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:41:02.882 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.387968+0000", 2024-03-20T09:41:02.882 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0079701100000000007 2024-03-20T09:41:02.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.883 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:41:02.883 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.388034+0000", 2024-03-20T09:41:02.883 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5349000000000001e-05 2024-03-20T09:41:02.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.883 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:41:02.883 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.388194+0000", 2024-03-20T09:41:02.883 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00016032899999999999 2024-03-20T09:41:02.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.883 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.883 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:41:02.883 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.388216+0000", 2024-03-20T09:41:02.884 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1957e-05 2024-03-20T09:41:02.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.884 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:41:02.884 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.388237+0000", 2024-03-20T09:41:02.884 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0829999999999999e-05 2024-03-20T09:41:02.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.884 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:41:02.884 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.405472+0000", 2024-03-20T09:41:02.884 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.017235527 2024-03-20T09:41:02.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.884 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:02.885 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.405532+0000", 2024-03-20T09:41:02.885 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9740000000000001e-05 2024-03-20T09:41:02.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.885 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:02.885 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.405673+0000", 2024-03-20T09:41:02.885 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000141509 2024-03-20T09:41:02.885 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.885 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:02.885 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.885 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.885 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.885 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:194297 7.5 e936/933 7:a6b384b5:::benchmark_data_smithi122_187077_object91199:head v 936'14280, mlcod=936'14266)", 2024-03-20T09:41:02.886 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:37.177108+0000", 2024-03-20T09:41:02.886 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 385.66166328600002, 2024-03-20T09:41:02.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.22801196200000001, 2024-03-20T09:41:02.886 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:02.886 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:02.886 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:02.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:02.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177108+0000", 2024-03-20T09:41:02.886 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.886 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.886 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:02.886 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177108+0000", 2024-03-20T09:41:02.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.887 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:02.887 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177109+0000", 2024-03-20T09:41:02.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7599999999999998e-07 2024-03-20T09:41:02.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.887 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:02.887 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177113+0000", 2024-03-20T09:41:02.887 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1629999999999998e-06 2024-03-20T09:41:02.887 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.887 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.888 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:02.888 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177118+0000", 2024-03-20T09:41:02.888 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8910000000000003e-06 2024-03-20T09:41:02.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.888 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:02.888 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177122+0000", 2024-03-20T09:41:02.888 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1819999999999999e-06 2024-03-20T09:41:02.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.888 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.888 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:02.888 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.376413+0000", 2024-03-20T09:41:02.888 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.199290627 2024-03-20T09:41:02.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.889 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:02.889 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.376434+0000", 2024-03-20T09:41:02.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1922000000000001e-05 2024-03-20T09:41:02.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.889 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:02.889 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.405072+0000", 2024-03-20T09:41:02.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.028637244999999999 2024-03-20T09:41:02.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.889 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.889 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:02.889 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.405120+0000", 2024-03-20T09:41:02.889 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8656000000000003e-05 2024-03-20T09:41:02.890 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.890 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:02.890 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.890 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.890 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:194298 7.5 e936/933 7:a6b44921:::benchmark_data_smithi122_187077_object98893:head v 936'14281, mlcod=936'14266)", 2024-03-20T09:41:02.890 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:37.177128+0000", 2024-03-20T09:41:02.890 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 385.661643787, 2024-03-20T09:41:02.890 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.24610715, 2024-03-20T09:41:02.890 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:02.890 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:02.890 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:02.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.890 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:02.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177128+0000", 2024-03-20T09:41:02.891 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.891 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:02.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177128+0000", 2024-03-20T09:41:02.891 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.891 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:02.891 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177128+0000", 2024-03-20T09:41:02.891 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9299999999999999e-07 2024-03-20T09:41:02.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.892 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:02.892 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177137+0000", 2024-03-20T09:41:02.892 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.183e-06 2024-03-20T09:41:02.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.892 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:02.892 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177143+0000", 2024-03-20T09:41:02.892 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9889999999999997e-06 2024-03-20T09:41:02.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.892 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:02.892 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177147+0000", 2024-03-20T09:41:02.892 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.985e-06 2024-03-20T09:41:02.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.893 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:02.893 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.382133+0000", 2024-03-20T09:41:02.893 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.20498566300000001 2024-03-20T09:41:02.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.893 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:02.893 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.382152+0000", 2024-03-20T09:41:02.893 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9562999999999999e-05 2024-03-20T09:41:02.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.893 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:02.893 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.423195+0000", 2024-03-20T09:41:02.893 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.041042661000000001 2024-03-20T09:41:02.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.893 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:02.894 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.423235+0000", 2024-03-20T09:41:02.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9913000000000002e-05 2024-03-20T09:41:02.894 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.894 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:02.894 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.894 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:194299 7.5 e936/933 7:a6b6325f:::benchmark_data_smithi122_187077_object103984:head v 936'14282, mlcod=936'14266)", 2024-03-20T09:41:02.894 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:37.177154+0000", 2024-03-20T09:41:02.894 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 385.661617648, 2024-03-20T09:41:02.894 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.24611186600000001, 2024-03-20T09:41:02.894 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:02.894 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:02.894 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:02.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:02.895 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177154+0000", 2024-03-20T09:41:02.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:02.895 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177154+0000", 2024-03-20T09:41:02.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:02.895 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177154+0000", 2024-03-20T09:41:02.895 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1500000000000001e-07 2024-03-20T09:41:02.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.895 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:02.896 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177158+0000", 2024-03-20T09:41:02.896 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.596e-06 2024-03-20T09:41:02.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.896 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:02.896 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177163+0000", 2024-03-20T09:41:02.896 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0699999999999997e-06 2024-03-20T09:41:02.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.896 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:02.896 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177167+0000", 2024-03-20T09:41:02.896 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1740000000000002e-06 2024-03-20T09:41:02.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.896 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:02.897 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.385210+0000", 2024-03-20T09:41:02.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.20804297899999999 2024-03-20T09:41:02.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.897 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:02.897 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.385224+0000", 2024-03-20T09:41:02.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3679999999999999e-05 2024-03-20T09:41:02.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.897 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:02.897 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.423240+0000", 2024-03-20T09:41:02.897 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.038015938999999999 2024-03-20T09:41:02.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.898 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:02.898 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.423266+0000", 2024-03-20T09:41:02.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6213000000000001e-05 2024-03-20T09:41:02.898 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.898 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:02.898 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.898 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:194300 7.5 e936/933 7:a6b663f8:::benchmark_data_smithi122_187077_object119994:head v 936'14283, mlcod=936'14266)", 2024-03-20T09:41:02.898 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:37.177173+0000", 2024-03-20T09:41:02.898 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 385.66159863199999, 2024-03-20T09:41:02.898 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.24612334699999999, 2024-03-20T09:41:02.898 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:02.899 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:02.899 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:02.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:02.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177173+0000", 2024-03-20T09:41:02.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:02.899 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177173+0000", 2024-03-20T09:41:02.899 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.899 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:02.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177173+0000", 2024-03-20T09:41:02.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0100000000000001e-07 2024-03-20T09:41:02.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:02.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177180+0000", 2024-03-20T09:41:02.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8889999999999996e-06 2024-03-20T09:41:02.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.900 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:02.900 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177188+0000", 2024-03-20T09:41:02.900 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6159999999999996e-06 2024-03-20T09:41:02.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.900 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.901 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:02.901 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.177192+0000", 2024-03-20T09:41:02.901 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3080000000000001e-06 2024-03-20T09:41:02.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.901 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:02.901 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.388253+0000", 2024-03-20T09:41:02.901 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.21106075399999999 2024-03-20T09:41:02.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.901 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:02.901 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.388264+0000", 2024-03-20T09:41:02.901 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0688e-05 2024-03-20T09:41:02.901 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.902 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:02.902 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.423270+0000", 2024-03-20T09:41:02.902 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.035006154999999997 2024-03-20T09:41:02.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.902 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:02.902 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.423296+0000", 2024-03-20T09:41:02.902 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6735999999999999e-05 2024-03-20T09:41:02.902 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.902 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:02.902 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.903 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:194301 7.5 e936/933 7:a6b82fee:::benchmark_data_smithi122_187077_object103950:head v 936'14284, mlcod=936'14266)", 2024-03-20T09:41:02.903 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:37.185194+0000", 2024-03-20T09:41:02.903 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 385.65357801599998, 2024-03-20T09:41:02.903 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.238133769, 2024-03-20T09:41:02.903 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:02.903 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:02.903 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:02.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.903 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:02.903 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.185194+0000", 2024-03-20T09:41:02.903 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.903 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:02.903 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.185194+0000", 2024-03-20T09:41:02.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:02.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.185195+0000", 2024-03-20T09:41:02.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.423e-06 2024-03-20T09:41:02.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:02.904 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.185210+0000", 2024-03-20T09:41:02.904 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.505e-05 2024-03-20T09:41:02.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.904 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:02.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199550+0000", 2024-03-20T09:41:02.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01433941 2024-03-20T09:41:02.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.905 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:02.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199570+0000", 2024-03-20T09:41:02.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0043000000000001e-05 2024-03-20T09:41:02.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.905 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:02.905 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.391901+0000", 2024-03-20T09:41:02.905 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.19233090899999999 2024-03-20T09:41:02.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.905 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:02.906 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.391919+0000", 2024-03-20T09:41:02.906 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8754999999999999e-05 2024-03-20T09:41:02.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:02.906 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.423300+0000", 2024-03-20T09:41:02.906 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031380659999999998 2024-03-20T09:41:02.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.906 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:02.906 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.423327+0000", 2024-03-20T09:41:02.906 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7518999999999999e-05 2024-03-20T09:41:02.906 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.906 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:02.907 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.907 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:194302 7.5 e936/933 7:a6b84dcf:::benchmark_data_smithi122_187077_object125423:head v 936'14285, mlcod=936'14266)", 2024-03-20T09:41:02.907 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:37.199600+0000", 2024-03-20T09:41:02.907 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 385.63917155500002, 2024-03-20T09:41:02.907 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.23229795, 2024-03-20T09:41:02.907 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:02.907 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:02.907 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:02.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.907 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:02.907 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199600+0000", 2024-03-20T09:41:02.907 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.908 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:02.908 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199600+0000", 2024-03-20T09:41:02.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.908 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:02.908 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199601+0000", 2024-03-20T09:41:02.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.161e-06 2024-03-20T09:41:02.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.908 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:02.908 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199613+0000", 2024-03-20T09:41:02.908 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1545999999999999e-05 2024-03-20T09:41:02.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.909 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:02.909 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199625+0000", 2024-03-20T09:41:02.909 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1766999999999999e-05 2024-03-20T09:41:02.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.909 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:02.909 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199632+0000", 2024-03-20T09:41:02.909 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2450000000000001e-06 2024-03-20T09:41:02.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.909 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:02.909 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.399899+0000", 2024-03-20T09:41:02.909 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.20026687100000001 2024-03-20T09:41:02.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:02.910 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.399918+0000", 2024-03-20T09:41:02.910 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9215000000000002e-05 2024-03-20T09:41:02.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:02.910 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.431855+0000", 2024-03-20T09:41:02.910 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031937385999999998 2024-03-20T09:41:02.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.910 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:02.911 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.431898+0000", 2024-03-20T09:41:02.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2759000000000002e-05 2024-03-20T09:41:02.911 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.911 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:02.911 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.911 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:194303 7.5 e936/933 7:a6c193fd:::benchmark_data_smithi122_187077_object102722:head v 936'14286, mlcod=936'14266)", 2024-03-20T09:41:02.911 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:37.199641+0000", 2024-03-20T09:41:02.911 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 385.63913095700002, 2024-03-20T09:41:02.911 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.23519206300000001, 2024-03-20T09:41:02.911 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:02.911 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:02.911 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:02.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.912 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:02.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199641+0000", 2024-03-20T09:41:02.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.912 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:02.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199641+0000", 2024-03-20T09:41:02.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.912 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:02.912 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199641+0000", 2024-03-20T09:41:02.912 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8999999999999998e-07 2024-03-20T09:41:02.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.913 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:02.913 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199668+0000", 2024-03-20T09:41:02.913 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6720000000000002e-05 2024-03-20T09:41:02.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.913 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:02.913 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199675+0000", 2024-03-20T09:41:02.913 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2169999999999997e-06 2024-03-20T09:41:02.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.913 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:02.913 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199680+0000", 2024-03-20T09:41:02.914 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4790000000000004e-06 2024-03-20T09:41:02.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.914 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:02.914 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.405442+0000", 2024-03-20T09:41:02.914 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.20576148499999999 2024-03-20T09:41:02.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.914 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:02.914 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.405953+0000", 2024-03-20T09:41:02.914 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00051126299999999997 2024-03-20T09:41:02.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.914 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:02.914 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.434800+0000", 2024-03-20T09:41:02.915 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.028847212000000001 2024-03-20T09:41:02.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.915 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:02.915 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.434833+0000", 2024-03-20T09:41:02.915 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2397000000000002e-05 2024-03-20T09:41:02.915 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.915 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:02.915 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.915 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:194304 7.5 e936/933 7:a6c45218:::benchmark_data_smithi122_187077_object124432:head v 936'14287, mlcod=936'14266)", 2024-03-20T09:41:02.915 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:37.199688+0000", 2024-03-20T09:41:02.915 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 385.63908410499999, 2024-03-20T09:41:02.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.23516978199999999, 2024-03-20T09:41:02.916 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:02.916 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:02.916 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:02.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.916 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:02.916 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199688+0000", 2024-03-20T09:41:02.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.916 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:02.916 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199688+0000", 2024-03-20T09:41:02.916 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:02.917 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199688+0000", 2024-03-20T09:41:02.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.96e-07 2024-03-20T09:41:02.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:02.917 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199692+0000", 2024-03-20T09:41:02.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6399999999999999e-06 2024-03-20T09:41:02.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.917 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:02.917 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199696+0000", 2024-03-20T09:41:02.917 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7110000000000003e-06 2024-03-20T09:41:02.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.918 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:02.918 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199701+0000", 2024-03-20T09:41:02.918 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9570000000000001e-06 2024-03-20T09:41:02.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.918 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:02.918 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.408849+0000", 2024-03-20T09:41:02.918 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.209147626 2024-03-20T09:41:02.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.918 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:02.918 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.408873+0000", 2024-03-20T09:41:02.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4329e-05 2024-03-20T09:41:02.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.919 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:02.919 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.434835+0000", 2024-03-20T09:41:02.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.025961992999999999 2024-03-20T09:41:02.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.919 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:02.919 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.434857+0000", 2024-03-20T09:41:02.919 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2229999999999999e-05 2024-03-20T09:41:02.919 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.919 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:02.919 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.920 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:194305 7.5 e936/933 7:a6c84c47:::benchmark_data_smithi122_187077_object119692:head v 936'14288, mlcod=936'14266)", 2024-03-20T09:41:02.920 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:37.199706+0000", 2024-03-20T09:41:02.920 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 385.63906528199999, 2024-03-20T09:41:02.920 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.23517816, 2024-03-20T09:41:02.920 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:02.920 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:02.920 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:02.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.920 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:02.920 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199706+0000", 2024-03-20T09:41:02.920 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.921 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:02.921 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199706+0000", 2024-03-20T09:41:02.921 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.921 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:02.921 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199707+0000", 2024-03-20T09:41:02.921 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5100000000000001e-07 2024-03-20T09:41:02.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.921 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:02.921 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199715+0000", 2024-03-20T09:41:02.921 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3520000000000007e-06 2024-03-20T09:41:02.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.922 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:02.922 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199720+0000", 2024-03-20T09:41:02.922 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.887e-06 2024-03-20T09:41:02.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.922 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:02.922 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199724+0000", 2024-03-20T09:41:02.922 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3630000000000004e-06 2024-03-20T09:41:02.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.922 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:02.922 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.411804+0000", 2024-03-20T09:41:02.922 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.212080137 2024-03-20T09:41:02.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:02.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.411824+0000", 2024-03-20T09:41:02.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9315999999999999e-05 2024-03-20T09:41:02.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:02.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.434860+0000", 2024-03-20T09:41:02.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.02303585 2024-03-20T09:41:02.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.923 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:02.923 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.434885+0000", 2024-03-20T09:41:02.923 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5004e-05 2024-03-20T09:41:02.924 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.924 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:02.924 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.924 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:194306 7.5 e936/933 7:a6ca5a25:::benchmark_data_smithi122_187077_object115441:head v 936'14289, mlcod=936'14266)", 2024-03-20T09:41:02.924 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:37.199730+0000", 2024-03-20T09:41:02.924 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 385.63904136299999, 2024-03-20T09:41:02.924 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.23517706799999999, 2024-03-20T09:41:02.924 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:02.924 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:02.924 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:02.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.924 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:02.925 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199730+0000", 2024-03-20T09:41:02.925 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.925 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:02.925 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199730+0000", 2024-03-20T09:41:02.925 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.925 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:02.925 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199731+0000", 2024-03-20T09:41:02.925 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2000000000000001e-07 2024-03-20T09:41:02.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.925 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:02.926 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199735+0000", 2024-03-20T09:41:02.926 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1749999999999998e-06 2024-03-20T09:41:02.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.926 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:02.926 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199740+0000", 2024-03-20T09:41:02.926 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5380000000000002e-06 2024-03-20T09:41:02.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.926 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:02.926 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199746+0000", 2024-03-20T09:41:02.926 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.071e-06 2024-03-20T09:41:02.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.927 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:02.927 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.414623+0000", 2024-03-20T09:41:02.927 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.21487640499999999 2024-03-20T09:41:02.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.927 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:02.927 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.414643+0000", 2024-03-20T09:41:02.927 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0581000000000002e-05 2024-03-20T09:41:02.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.927 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:02.927 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.434887+0000", 2024-03-20T09:41:02.927 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020243351 2024-03-20T09:41:02.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.928 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:02.928 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.434907+0000", 2024-03-20T09:41:02.928 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0727e-05 2024-03-20T09:41:02.928 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.928 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:02.928 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.928 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:194307 7.5 e936/933 7:a6ca90f5:::benchmark_data_smithi122_187077_object124172:head v 936'14290, mlcod=936'14266)", 2024-03-20T09:41:02.928 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:37.199751+0000", 2024-03-20T09:41:02.928 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 385.639020292, 2024-03-20T09:41:02.928 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.23628909000000001, 2024-03-20T09:41:02.928 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:02.928 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:02.929 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:02.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.929 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:02.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199751+0000", 2024-03-20T09:41:02.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.929 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:02.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199751+0000", 2024-03-20T09:41:02.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.929 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:02.929 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199752+0000", 2024-03-20T09:41:02.929 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.85e-07 2024-03-20T09:41:02.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:02.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199760+0000", 2024-03-20T09:41:02.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2039999999999997e-06 2024-03-20T09:41:02.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:02.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.199766+0000", 2024-03-20T09:41:02.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1229999999999997e-06 2024-03-20T09:41:02.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.930 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:02.930 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.209565+0000", 2024-03-20T09:41:02.930 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097986749999999997 2024-03-20T09:41:02.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:02.931 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.420852+0000", 2024-03-20T09:41:02.931 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.21128715300000001 2024-03-20T09:41:02.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:02.931 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.420878+0000", 2024-03-20T09:41:02.931 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6217999999999999e-05 2024-03-20T09:41:02.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.931 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:02.931 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.436012+0000", 2024-03-20T09:41:02.931 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01513452 2024-03-20T09:41:02.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.932 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:02.932 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.436040+0000", 2024-03-20T09:41:02.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8011999999999999e-05 2024-03-20T09:41:02.932 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.932 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:02.932 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.932 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:194308 7.5 e936/933 7:a6cbfedc:::benchmark_data_smithi122_187077_object123819:head v 936'14291, mlcod=936'14266)", 2024-03-20T09:41:02.932 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:37.209605+0000", 2024-03-20T09:41:02.932 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 385.62916667299999, 2024-03-20T09:41:02.932 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.22646349499999999, 2024-03-20T09:41:02.932 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:02.933 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:02.933 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:02.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.933 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:02.933 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.209605+0000", 2024-03-20T09:41:02.933 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.933 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:02.933 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.209605+0000", 2024-03-20T09:41:02.933 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.933 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:02.934 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.209607+0000", 2024-03-20T09:41:02.934 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5239999999999999e-06 2024-03-20T09:41:02.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.934 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:02.934 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.209620+0000", 2024-03-20T09:41:02.934 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3729e-05 2024-03-20T09:41:02.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.934 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:02.934 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.209633+0000", 2024-03-20T09:41:02.934 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2918999999999999e-05 2024-03-20T09:41:02.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.935 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:02.935 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.209645+0000", 2024-03-20T09:41:02.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1367e-05 2024-03-20T09:41:02.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.935 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:02.935 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.423350+0000", 2024-03-20T09:41:02.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.21370534699999999 2024-03-20T09:41:02.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.935 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:02.935 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.423363+0000", 2024-03-20T09:41:02.935 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3475000000000001e-05 2024-03-20T09:41:02.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.935 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:02.935 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.436043+0000", 2024-03-20T09:41:02.936 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012679693000000001 2024-03-20T09:41:02.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.936 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:02.936 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.436069+0000", 2024-03-20T09:41:02.936 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5440999999999999e-05 2024-03-20T09:41:02.936 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.936 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:02.936 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.936 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.11.0:194309 7.5 e936/933 7:a6cdc98c:::benchmark_data_smithi122_187077_object97831:head v 936'14292, mlcod=936'14266)", 2024-03-20T09:41:02.936 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:37.209653+0000", 2024-03-20T09:41:02.936 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 385.629118877, 2024-03-20T09:41:02.936 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.22644837500000001, 2024-03-20T09:41:02.937 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:02.937 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:02.937 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:02.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.937 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:02.937 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.209653+0000", 2024-03-20T09:41:02.937 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.937 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:02.937 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.209653+0000", 2024-03-20T09:41:02.937 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:02.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.937 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:02.938 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.209653+0000", 2024-03-20T09:41:02.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6399999999999998e-07 2024-03-20T09:41:02.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.938 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:02.938 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.209673+0000", 2024-03-20T09:41:02.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9653e-05 2024-03-20T09:41:02.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.938 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:02.938 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.209679+0000", 2024-03-20T09:41:02.938 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3929999999999996e-06 2024-03-20T09:41:02.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.939 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:02.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.209685+0000", 2024-03-20T09:41:02.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6799999999999998e-06 2024-03-20T09:41:02.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.939 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:02.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.425864+0000", 2024-03-20T09:41:02.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.21617926100000001 2024-03-20T09:41:02.939 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.939 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:02.939 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.425884+0000", 2024-03-20T09:41:02.939 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9993e-05 2024-03-20T09:41:02.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.940 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:02.940 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.436073+0000", 2024-03-20T09:41:02.940 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010189119 2024-03-20T09:41:02.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:02.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:02.940 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:02.940 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.436101+0000", 2024-03-20T09:41:02.940 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8011999999999999e-05 2024-03-20T09:41:02.940 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.940 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:02.940 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.940 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:02.940 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:02.941 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:41:02.941 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:41:02.942 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:41:02.983 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:03.010 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:41:03.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:03.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:41:03.114 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:41:03.155 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:03.193 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:41:03.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:03.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:41:03.298 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:41:03.347 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:03.754 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:41:03.813 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 24 16 58586 58570 152.49 119 0.00844784 0.00655333 2024-03-20T09:41:04.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:03 smithi115 ceph-mon[72476]: pgmap v2612: 47 pgs: 47 active+clean; 3.1 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 141 MiB/s wr, 2.26k op/s 2024-03-20T09:41:04.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:03 smithi122 ceph-mon[73495]: pgmap v2612: 47 pgs: 47 active+clean; 3.1 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 141 MiB/s wr, 2.26k op/s 2024-03-20T09:41:04.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:03 smithi026 ceph-mon[68309]: pgmap v2612: 47 pgs: 47 active+clean; 3.1 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 141 MiB/s wr, 2.26k op/s 2024-03-20T09:41:04.326 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-20T09:41:04.813 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 25 16 60442 60426 151.03 116 0.00373117 0.00661734 2024-03-20T09:41:05.813 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 26 16 62298 62282 149.683 116 0.00430181 0.0066766 2024-03-20T09:41:05.954 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:41:06.016 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:41:06.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:06.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:41:06.025 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:05 smithi026 ceph-mon[68309]: pgmap v2613: 47 pgs: 47 active+clean; 3.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 108 MiB/s wr, 1.72k op/s 2024-03-20T09:41:06.032 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:41:06.032 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:41:06.121 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:41:06.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:05 smithi115 ceph-mon[72476]: pgmap v2613: 47 pgs: 47 active+clean; 3.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 108 MiB/s wr, 1.72k op/s 2024-03-20T09:41:06.178 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:06.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:05 smithi122 ceph-mon[73495]: pgmap v2613: 47 pgs: 47 active+clean; 3.3 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 108 MiB/s wr, 1.72k op/s 2024-03-20T09:41:06.238 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:41:06.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:06.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:41:06.344 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:41:06.393 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:06.813 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 27 16 64024 64008 148.134 107.875 0.0129784 0.00674662 2024-03-20T09:41:07.813 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 28 16 65611 65595 146.386 99.1875 0.00743591 0.00682732 2024-03-20T09:41:07.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:07 smithi026 ceph-mon[68309]: pgmap v2614: 47 pgs: 47 active+clean; 3.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 133 MiB/s wr, 2.13k op/s 2024-03-20T09:41:07.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:07 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:41:07.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:07 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:41:08.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:07 smithi115 ceph-mon[72476]: pgmap v2614: 47 pgs: 47 active+clean; 3.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 133 MiB/s wr, 2.13k op/s 2024-03-20T09:41:08.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:07 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:41:08.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:07 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:41:08.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:07 smithi122 ceph-mon[73495]: pgmap v2614: 47 pgs: 47 active+clean; 3.7 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 133 MiB/s wr, 2.13k op/s 2024-03-20T09:41:08.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:07 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:41:08.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:07 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:41:08.813 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 29 16 67224 67208 144.814 100.812 0.014968 0.0069017 2024-03-20T09:41:09.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:08 smithi115 ceph-mon[72476]: osdmap e1080: 12 total, 12 up, 11 in 2024-03-20T09:41:09.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:08 smithi115 ceph-mon[72476]: osdmap e1081: 12 total, 12 up, 11 in 2024-03-20T09:41:09.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:08 smithi122 ceph-mon[73495]: osdmap e1080: 12 total, 12 up, 11 in 2024-03-20T09:41:09.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:08 smithi122 ceph-mon[73495]: osdmap e1081: 12 total, 12 up, 11 in 2024-03-20T09:41:09.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:08 smithi026 ceph-mon[68309]: osdmap e1080: 12 total, 12 up, 11 in 2024-03-20T09:41:09.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:08 smithi026 ceph-mon[68309]: osdmap e1081: 12 total, 12 up, 11 in 2024-03-20T09:41:09.814 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 30 16 68880 68864 143.437 103.5 0.00833647 0.00696763 2024-03-20T09:41:10.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:09 smithi115 ceph-mon[72476]: pgmap v2617: 47 pgs: 47 active+clean; 3.8 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-03-20T09:41:10.155 INFO:teuthology.orchestra.run.smithi026.stderr:no valid command found; 10 closest matches: 2024-03-20T09:41:10.155 INFO:teuthology.orchestra.run.smithi026.stderr:0 2024-03-20T09:41:10.155 INFO:teuthology.orchestra.run.smithi026.stderr:1 2024-03-20T09:41:10.155 INFO:teuthology.orchestra.run.smithi026.stderr:2 2024-03-20T09:41:10.155 INFO:teuthology.orchestra.run.smithi026.stderr:abort 2024-03-20T09:41:10.156 INFO:teuthology.orchestra.run.smithi026.stderr:assert 2024-03-20T09:41:10.156 INFO:teuthology.orchestra.run.smithi026.stderr:bench [] [] [] [] 2024-03-20T09:41:10.156 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs debug_inject_read_zeros 2024-03-20T09:41:10.156 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs files list 2024-03-20T09:41:10.156 INFO:teuthology.orchestra.run.smithi026.stderr:bluefs stats 2024-03-20T09:41:10.156 INFO:teuthology.orchestra.run.smithi026.stderr:bluestore allocator dump block 2024-03-20T09:41:10.156 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: invalid command 2024-03-20T09:41:10.171 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:09 smithi026 ceph-mon[68309]: pgmap v2617: 47 pgs: 47 active+clean; 3.8 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-03-20T09:41:10.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:09 smithi122 ceph-mon[73495]: pgmap v2617: 47 pgs: 47 active+clean; 3.8 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-03-20T09:41:10.257 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:41:10.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:10.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:41:10.362 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:41:10.403 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:10.428 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:41:10.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:10.435 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:41:10.536 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:41:10.581 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:10.813 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 31 16 70241 70225 141.554 85.0625 0.00347059 0.00706093 2024-03-20T09:41:11.142 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:41:11.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:11.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:41:11.250 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:41:11.289 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:11.548 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:41:11.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:11.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:41:11.578 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T09:41:11.578 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-20T09:41:11.666 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:41:11.708 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:11.737 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:41:11.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:11.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:41:11.814 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:11 smithi026 ceph-mon[68309]: pgmap v2618: 47 pgs: 47 active+clean; 4.1 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 124 MiB/s wr, 1.98k op/s 2024-03-20T09:41:11.816 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 32 16 71915 71899 140.386 104.625 0.0104574 0.00711913 2024-03-20T09:41:11.841 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:41:11.886 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:12.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:11 smithi115 ceph-mon[72476]: pgmap v2618: 47 pgs: 47 active+clean; 4.1 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 124 MiB/s wr, 1.98k op/s 2024-03-20T09:41:12.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:11 smithi122 ceph-mon[73495]: pgmap v2618: 47 pgs: 47 active+clean; 4.1 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 124 MiB/s wr, 1.98k op/s 2024-03-20T09:41:12.250 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:41:12.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:12.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:41:12.354 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:41:12.396 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:12.427 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:41:12.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:12.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:41:12.533 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:41:12.572 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:12.604 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:41:12.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:12.608 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:41:12.709 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:41:12.756 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:12.785 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:41:12.817 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 33 16 73502 73486 139.134 99.1875 0.00747164 0.00718331 2024-03-20T09:41:12.934 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:12 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/951893960' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:41:12.935 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:12 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:41:12.948 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:41:12.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:12 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/951893960' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:41:12.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:12 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:41:12.975 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:41:13.008 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:41:13.032 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:41:13.071 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:41:13.101 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:41:13.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:12 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/951893960' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:41:13.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:12 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:41:13.332 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:41:13.371 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:41:13.404 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:41:13.437 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:41:13.469 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:41:13.503 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:41:13.505 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:41:13.505 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:41:13.505 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:41:13.505 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:41:13.505 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:41:13.505 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:41:13.505 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:41:13.505 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:41:13.506 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:41:13.506 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:41:13.506 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:41:13.506 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:41:13.817 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 34 15 75113 75098 138.005 100.75 0.00984989 0.00724214 2024-03-20T09:41:13.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:13 smithi026 ceph-mon[68309]: pgmap v2619: 47 pgs: 47 active+clean; 4.2 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 115 MiB/s wr, 1.85k op/s 2024-03-20T09:41:13.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:13 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:41:13.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:13 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:41:13.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:13 smithi026 ceph-mon[68309]: osdmap e1082: 12 total, 12 up, 11 in 2024-03-20T09:41:13.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:13 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:41:13.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:13 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:41:13.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:13 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1750580329' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:41:13.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:13 smithi026 ceph-mon[68309]: osdmap e1083: 12 total, 12 up, 11 in 2024-03-20T09:41:14.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:13 smithi115 ceph-mon[72476]: pgmap v2619: 47 pgs: 47 active+clean; 4.2 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 115 MiB/s wr, 1.85k op/s 2024-03-20T09:41:14.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:13 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:41:14.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:13 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:41:14.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:13 smithi115 ceph-mon[72476]: osdmap e1082: 12 total, 12 up, 11 in 2024-03-20T09:41:14.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:13 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:41:14.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:13 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:41:14.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:13 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1750580329' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:41:14.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:13 smithi115 ceph-mon[72476]: osdmap e1083: 12 total, 12 up, 11 in 2024-03-20T09:41:14.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:13 smithi122 ceph-mon[73495]: pgmap v2619: 47 pgs: 47 active+clean; 4.2 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 115 MiB/s wr, 1.85k op/s 2024-03-20T09:41:14.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:13 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:41:14.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:13 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:41:14.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:13 smithi122 ceph-mon[73495]: osdmap e1082: 12 total, 12 up, 11 in 2024-03-20T09:41:14.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:13 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:41:14.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:13 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:41:14.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:13 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1750580329' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:41:14.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:13 smithi122 ceph-mon[73495]: osdmap e1083: 12 total, 12 up, 11 in 2024-03-20T09:41:14.818 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 35 16 76505 76489 136.546 86.9375 0.00615463 0.00731898 2024-03-20T09:41:15.065 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:41:15.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:15.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:41:15.170 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:41:15.223 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:15.261 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:41:15.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:15.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:41:15.367 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:41:15.404 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:15.491 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:41:15.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:15.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:41:15.601 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:41:15.636 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:15.661 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:41:15.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:15.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:41:15.767 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:41:15.796 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:15.818 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 36 16 77958 77942 135.276 90.8125 0.00940878 0.00738817 2024-03-20T09:41:15.893 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:15 smithi115 ceph-mon[72476]: pgmap v2622: 47 pgs: 47 active+clean; 4.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-03-20T09:41:15.929 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:41:15.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:15.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:41:16.036 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:41:16.083 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:16.112 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:41:16.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:16.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:41:16.140 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:15 smithi026 ceph-mon[68309]: pgmap v2622: 47 pgs: 47 active+clean; 4.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-03-20T09:41:16.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:15 smithi122 ceph-mon[73495]: pgmap v2622: 47 pgs: 47 active+clean; 4.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-03-20T09:41:16.218 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:41:16.244 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:16.263 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:41:16.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:16.266 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:41:16.366 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:41:16.403 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:16.433 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:41:16.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:16.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:41:16.539 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:41:16.576 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:16.579 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 5, 6, 7, 8, 11, 3, 4, 9, 10] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 11, 5, 10] 2024-03-20T09:41:16.580 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:41:16.580 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-20T09:41:16.580 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-20T09:41:16.602 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:41:16.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:16.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:41:16.633 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:41:16.633 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:41:16.708 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:41:16.755 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:16.784 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:41:16.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:16.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:41:16.818 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 37 16 79209 79193 133.733 78.1875 0.00614166 0.00747353 2024-03-20T09:41:16.890 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:41:16.940 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:17.105 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:41:17.496 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:41:17.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:17.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:41:17.505 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-20T09:41:17.530 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:41:17.601 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:41:17.653 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:17.797 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:41:17.808 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:17.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:41:17.818 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 38 16 80727 80711 132.71 94.875 0.0102336 0.00753171 2024-03-20T09:41:17.839 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:41:17.909 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:41:17.968 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:17.968 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:41:17.997 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:41:18.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:18.001 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:41:18.102 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:41:18.133 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:17 smithi026 ceph-mon[68309]: pgmap v2623: 47 pgs: 47 active+clean; 4.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 113 MiB/s wr, 1.80k op/s 2024-03-20T09:41:18.135 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:18.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:17 smithi115 ceph-mon[72476]: pgmap v2623: 47 pgs: 47 active+clean; 4.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 113 MiB/s wr, 1.80k op/s 2024-03-20T09:41:18.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:17 smithi122 ceph-mon[73495]: pgmap v2623: 47 pgs: 47 active+clean; 4.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 113 MiB/s wr, 1.80k op/s 2024-03-20T09:41:18.273 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:41:18.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:18.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:41:18.378 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:41:18.419 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:18.449 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:41:18.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:18.453 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:41:18.554 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:41:18.594 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:18.628 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:41:18.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:18.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:41:18.734 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:41:18.779 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:18.806 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:41:18.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:18.810 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:41:18.818 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 39 16 82109 82093 131.522 86.375 0.00508003 0.0075988 2024-03-20T09:41:18.912 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:41:18.953 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:18.987 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:41:18.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:18.992 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:41:19.093 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:41:19.141 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:19.171 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:41:19.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:19.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:41:19.278 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:41:19.328 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:19.363 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:41:19.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:19.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:41:19.471 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:41:19.531 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:19.558 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:41:19.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:19.565 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:41:19.666 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:41:19.713 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:41:19.741 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:19.818 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:2024-03-20T09:41:19.830903+0000 min lat: 0.00113956 max lat: 0.214394 avg lat: 0.00765678 2024-03-20T09:41:19.818 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T09:41:19.818 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 40 16 83582 83566 130.536 92.0625 0.00670529 0.00765678 2024-03-20T09:41:19.828 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:41:19.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:19.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:41:19.935 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:41:19.985 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:20.010 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:41:20.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:20.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:41:20.095 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:19 smithi026 ceph-mon[68309]: pgmap v2624: 47 pgs: 47 active+clean; 4.7 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-03-20T09:41:20.116 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:41:20.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:19 smithi115 ceph-mon[72476]: pgmap v2624: 47 pgs: 47 active+clean; 4.7 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-03-20T09:41:20.163 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:20.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:19 smithi122 ceph-mon[73495]: pgmap v2624: 47 pgs: 47 active+clean; 4.7 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-03-20T09:41:20.820 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 41 16 85074 85058 129.619 93.25 0.00970035 0.00771128 2024-03-20T09:41:21.820 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 42 16 86501 86485 128.656 89.1875 0.00700725 0.00776874 2024-03-20T09:41:21.876 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:41:21.876 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:41:21.876 INFO:teuthology.orchestra.run.smithi026.stdout: "epoch": 1083, 2024-03-20T09:41:21.876 INFO:teuthology.orchestra.run.smithi026.stdout: "fsid": "852fbb68-e696-11ee-95c9-87774f69a715", 2024-03-20T09:41:21.876 INFO:teuthology.orchestra.run.smithi026.stdout: "created": "2024-03-20T08:48:35.281088+0000", 2024-03-20T09:41:21.876 INFO:teuthology.orchestra.run.smithi026.stdout: "modified": "2024-03-20T09:41:12.925149+0000", 2024-03-20T09:41:21.876 INFO:teuthology.orchestra.run.smithi026.stdout: "last_up_change": "2024-03-20T09:30:44.068387+0000", 2024-03-20T09:41:21.876 INFO:teuthology.orchestra.run.smithi026.stdout: "last_in_change": "2024-03-20T09:39:13.893852+0000", 2024-03-20T09:41:21.876 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-20T09:41:21.877 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_num": 5806080, 2024-03-20T09:41:21.877 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_set": [ 2024-03-20T09:41:21.877 INFO:teuthology.orchestra.run.smithi026.stdout: "nodeep-scrub", 2024-03-20T09:41:21.877 INFO:teuthology.orchestra.run.smithi026.stdout: "noscrub", 2024-03-20T09:41:21.877 INFO:teuthology.orchestra.run.smithi026.stdout: "pglog_hardlimit", 2024-03-20T09:41:21.877 INFO:teuthology.orchestra.run.smithi026.stdout: "purged_snapdirs", 2024-03-20T09:41:21.877 INFO:teuthology.orchestra.run.smithi026.stdout: "recovery_deletes", 2024-03-20T09:41:21.877 INFO:teuthology.orchestra.run.smithi026.stdout: "sortbitwise" 2024-03-20T09:41:21.877 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:41:21.877 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_version": 28, 2024-03-20T09:41:21.877 INFO:teuthology.orchestra.run.smithi026.stdout: "full_ratio": 0.94999998807907104, 2024-03-20T09:41:21.877 INFO:teuthology.orchestra.run.smithi026.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-20T09:41:21.877 INFO:teuthology.orchestra.run.smithi026.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-20T09:41:21.877 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_snapshot": "", 2024-03-20T09:41:21.877 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_max": 9, 2024-03-20T09:41:21.878 INFO:teuthology.orchestra.run.smithi026.stdout: "max_osd": 12, 2024-03-20T09:41:21.878 INFO:teuthology.orchestra.run.smithi026.stdout: "require_min_compat_client": "luminous", 2024-03-20T09:41:21.878 INFO:teuthology.orchestra.run.smithi026.stdout: "min_compat_client": "luminous", 2024-03-20T09:41:21.878 INFO:teuthology.orchestra.run.smithi026.stdout: "require_osd_release": "squid", 2024-03-20T09:41:21.878 INFO:teuthology.orchestra.run.smithi026.stdout: "allow_crimson": false, 2024-03-20T09:41:21.878 INFO:teuthology.orchestra.run.smithi026.stdout: "pools": [ 2024-03-20T09:41:21.878 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.878 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 1, 2024-03-20T09:41:21.878 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": ".mgr", 2024-03-20T09:41:21.878 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-03-20T08:52:17.442324+0000", 2024-03-20T09:41:21.878 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-03-20T09:41:21.878 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-03-20T09:41:21.878 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-03-20T09:41:21.879 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 3, 2024-03-20T09:41:21.879 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 2, 2024-03-20T09:41:21.879 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-03-20T09:41:21.879 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-03-20T09:41:21.879 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-03-20T09:41:21.879 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-03-20T09:41:21.879 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-20T09:41:21.879 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-03-20T09:41:21.879 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-03-20T09:41:21.879 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 31, 2024-03-20T09:41:21.879 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 31, 2024-03-20T09:41:21.879 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 31, 2024-03-20T09:41:21.879 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 31, 2024-03-20T09:41:21.879 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 31, 2024-03-20T09:41:21.880 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-03-20T09:41:21.880 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-03-20T09:41:21.880 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-03-20T09:41:21.880 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-03-20T09:41:21.880 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-03-20T09:41:21.880 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-03-20T09:41:21.880 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-03-20T09:41:21.880 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.880 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "360", 2024-03-20T09:41:21.880 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-03-20T09:41:21.880 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-20T09:41:21.880 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "358", 2024-03-20T09:41:21.880 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-03-20T09:41:21.880 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-03-20T09:41:21.881 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-03-20T09:41:21.881 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-03-20T09:41:21.881 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-03-20T09:41:21.881 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-03-20T09:41:21.881 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-03-20T09:41:21.881 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-03-20T09:41:21.881 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-03-20T09:41:21.881 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-03-20T09:41:21.881 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-03-20T09:41:21.881 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-03-20T09:41:21.881 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-03-20T09:41:21.881 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-03-20T09:41:21.881 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-03-20T09:41:21.881 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-20T09:41:21.881 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-20T09:41:21.882 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-20T09:41:21.882 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-03-20T09:41:21.882 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-03-20T09:41:21.882 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-03-20T09:41:21.882 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-03-20T09:41:21.882 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-03-20T09:41:21.882 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.882 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-03-20T09:41:21.882 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-03-20T09:41:21.882 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-03-20T09:41:21.882 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-03-20T09:41:21.882 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-03-20T09:41:21.882 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-03-20T09:41:21.882 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-03-20T09:41:21.883 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-03-20T09:41:21.883 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-03-20T09:41:21.883 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-03-20T09:41:21.883 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-03-20T09:41:21.883 INFO:teuthology.orchestra.run.smithi026.stdout: "options": { 2024-03-20T09:41:21.883 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_max": 32, 2024-03-20T09:41:21.883 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_min": 1 2024-03-20T09:41:21.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.883 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-03-20T09:41:21.883 INFO:teuthology.orchestra.run.smithi026.stdout: "mgr": {} 2024-03-20T09:41:21.883 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.883 INFO:teuthology.orchestra.run.smithi026.stdout: "read_balance": { 2024-03-20T09:41:21.883 INFO:teuthology.orchestra.run.smithi026.stdout: "score_type": "Fair distribution", 2024-03-20T09:41:21.883 INFO:teuthology.orchestra.run.smithi026.stdout: "score_acting": 2.3199999332427979, 2024-03-20T09:41:21.883 INFO:teuthology.orchestra.run.smithi026.stdout: "score_stable": 2.3199999332427979, 2024-03-20T09:41:21.884 INFO:teuthology.orchestra.run.smithi026.stdout: "optimal_score": 0.92000001668930054, 2024-03-20T09:41:21.884 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_acting": 2.130000114440918, 2024-03-20T09:41:21.884 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_stable": 2.130000114440918, 2024-03-20T09:41:21.884 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity_weighted": 0.83473199605941772, 2024-03-20T09:41:21.884 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity": 0.82999998331069946, 2024-03-20T09:41:21.884 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-03-20T09:41:21.884 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.884 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.884 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.884 INFO:teuthology.orchestra.run.smithi026.stdout: "pool": 9, 2024-03-20T09:41:21.884 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_name": "unique_pool_7", 2024-03-20T09:41:21.884 INFO:teuthology.orchestra.run.smithi026.stdout: "create_time": "2024-03-20T09:40:23.232804+0000", 2024-03-20T09:41:21.884 INFO:teuthology.orchestra.run.smithi026.stdout: "flags": 1, 2024-03-20T09:41:21.884 INFO:teuthology.orchestra.run.smithi026.stdout: "flags_names": "hashpspool", 2024-03-20T09:41:21.885 INFO:teuthology.orchestra.run.smithi026.stdout: "type": 1, 2024-03-20T09:41:21.885 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 3, 2024-03-20T09:41:21.885 INFO:teuthology.orchestra.run.smithi026.stdout: "min_size": 2, 2024-03-20T09:41:21.885 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_rule": 0, 2024-03-20T09:41:21.885 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_count": 0, 2024-03-20T09:41:21.885 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_target": 0, 2024-03-20T09:41:21.885 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_barrier": 0, 2024-03-20T09:41:21.885 INFO:teuthology.orchestra.run.smithi026.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-20T09:41:21.885 INFO:teuthology.orchestra.run.smithi026.stdout: "object_hash": 2, 2024-03-20T09:41:21.885 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_autoscale_mode": "off", 2024-03-20T09:41:21.885 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num": 16, 2024-03-20T09:41:21.885 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num": 16, 2024-03-20T09:41:21.885 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_placement_num_target": 16, 2024-03-20T09:41:21.885 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_target": 16, 2024-03-20T09:41:21.885 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_num_pending": 16, 2024-03-20T09:41:21.886 INFO:teuthology.orchestra.run.smithi026.stdout: "last_pg_merge_meta": { 2024-03-20T09:41:21.886 INFO:teuthology.orchestra.run.smithi026.stdout: "source_pgid": "0.0", 2024-03-20T09:41:21.886 INFO:teuthology.orchestra.run.smithi026.stdout: "ready_epoch": 0, 2024-03-20T09:41:21.886 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_started": 0, 2024-03-20T09:41:21.886 INFO:teuthology.orchestra.run.smithi026.stdout: "last_epoch_clean": 0, 2024-03-20T09:41:21.886 INFO:teuthology.orchestra.run.smithi026.stdout: "source_version": "0'0", 2024-03-20T09:41:21.886 INFO:teuthology.orchestra.run.smithi026.stdout: "target_version": "0'0" 2024-03-20T09:41:21.886 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.886 INFO:teuthology.orchestra.run.smithi026.stdout: "last_change": "1073", 2024-03-20T09:41:21.886 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend": "0", 2024-03-20T09:41:21.886 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-20T09:41:21.886 INFO:teuthology.orchestra.run.smithi026.stdout: "last_force_op_resend_preluminous": "0", 2024-03-20T09:41:21.886 INFO:teuthology.orchestra.run.smithi026.stdout: "auid": 0, 2024-03-20T09:41:21.886 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_mode": "selfmanaged", 2024-03-20T09:41:21.886 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_seq": 0, 2024-03-20T09:41:21.887 INFO:teuthology.orchestra.run.smithi026.stdout: "snap_epoch": 0, 2024-03-20T09:41:21.887 INFO:teuthology.orchestra.run.smithi026.stdout: "pool_snaps": [], 2024-03-20T09:41:21.887 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps": "[]", 2024-03-20T09:41:21.887 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_bytes": 0, 2024-03-20T09:41:21.887 INFO:teuthology.orchestra.run.smithi026.stdout: "quota_max_objects": 0, 2024-03-20T09:41:21.887 INFO:teuthology.orchestra.run.smithi026.stdout: "tiers": [], 2024-03-20T09:41:21.887 INFO:teuthology.orchestra.run.smithi026.stdout: "tier_of": -1, 2024-03-20T09:41:21.887 INFO:teuthology.orchestra.run.smithi026.stdout: "read_tier": -1, 2024-03-20T09:41:21.887 INFO:teuthology.orchestra.run.smithi026.stdout: "write_tier": -1, 2024-03-20T09:41:21.887 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_mode": "none", 2024-03-20T09:41:21.887 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_bytes": 0, 2024-03-20T09:41:21.887 INFO:teuthology.orchestra.run.smithi026.stdout: "target_max_objects": 0, 2024-03-20T09:41:21.887 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-20T09:41:21.887 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-20T09:41:21.887 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-20T09:41:21.887 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_flush_age": 0, 2024-03-20T09:41:21.888 INFO:teuthology.orchestra.run.smithi026.stdout: "cache_min_evict_age": 0, 2024-03-20T09:41:21.888 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profile": "", 2024-03-20T09:41:21.888 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_params": { 2024-03-20T09:41:21.888 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "none" 2024-03-20T09:41:21.888 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.888 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_period": 0, 2024-03-20T09:41:21.888 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_count": 0, 2024-03-20T09:41:21.888 INFO:teuthology.orchestra.run.smithi026.stdout: "use_gmt_hitset": true, 2024-03-20T09:41:21.888 INFO:teuthology.orchestra.run.smithi026.stdout: "min_read_recency_for_promote": 0, 2024-03-20T09:41:21.888 INFO:teuthology.orchestra.run.smithi026.stdout: "min_write_recency_for_promote": 0, 2024-03-20T09:41:21.888 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_grade_decay_rate": 0, 2024-03-20T09:41:21.888 INFO:teuthology.orchestra.run.smithi026.stdout: "hit_set_search_last_n": 0, 2024-03-20T09:41:21.888 INFO:teuthology.orchestra.run.smithi026.stdout: "grade_table": [], 2024-03-20T09:41:21.888 INFO:teuthology.orchestra.run.smithi026.stdout: "stripe_width": 0, 2024-03-20T09:41:21.889 INFO:teuthology.orchestra.run.smithi026.stdout: "expected_num_objects": 0, 2024-03-20T09:41:21.889 INFO:teuthology.orchestra.run.smithi026.stdout: "fast_read": false, 2024-03-20T09:41:21.889 INFO:teuthology.orchestra.run.smithi026.stdout: "options": {}, 2024-03-20T09:41:21.889 INFO:teuthology.orchestra.run.smithi026.stdout: "application_metadata": { 2024-03-20T09:41:21.889 INFO:teuthology.orchestra.run.smithi026.stdout: "rados": {} 2024-03-20T09:41:21.889 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.889 INFO:teuthology.orchestra.run.smithi026.stdout: "read_balance": { 2024-03-20T09:41:21.889 INFO:teuthology.orchestra.run.smithi026.stdout: "score_type": "Fair distribution", 2024-03-20T09:41:21.889 INFO:teuthology.orchestra.run.smithi026.stdout: "score_acting": 3.0099999904632568, 2024-03-20T09:41:21.889 INFO:teuthology.orchestra.run.smithi026.stdout: "score_stable": 3.0099999904632568, 2024-03-20T09:41:21.889 INFO:teuthology.orchestra.run.smithi026.stdout: "optimal_score": 0.82999998331069946, 2024-03-20T09:41:21.889 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_acting": 2.5, 2024-03-20T09:41:21.889 INFO:teuthology.orchestra.run.smithi026.stdout: "raw_score_stable": 2.5, 2024-03-20T09:41:21.889 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity_weighted": 0.83473199605941772, 2024-03-20T09:41:21.889 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity": 0.82999998331069946, 2024-03-20T09:41:21.889 INFO:teuthology.orchestra.run.smithi026.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-03-20T09:41:21.890 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.890 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.890 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:41:21.890 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:41:21.890 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.890 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-03-20T09:41:21.890 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "ad3fd9c0-316f-4fe5-a63b-d480087c1729", 2024-03-20T09:41:21.890 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:41:21.890 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 0, 2024-03-20T09:41:21.890 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 0, 2024-03-20T09:41:21.890 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:41:21.890 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:41:21.890 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:41:21.890 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 8, 2024-03-20T09:41:21.891 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 974, 2024-03-20T09:41:21.891 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:41:21.891 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:41:21.891 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:41:21.891 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.891 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.891 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6802", 2024-03-20T09:41:21.891 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:41:21.891 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.891 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.891 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.891 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6803", 2024-03-20T09:41:21.891 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:41:21.891 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.891 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.892 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:41:21.892 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.892 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.892 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6804", 2024-03-20T09:41:21.892 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:41:21.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.892 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.892 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.892 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6805", 2024-03-20T09:41:21.892 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:41:21.892 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.892 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.892 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.892 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:41:21.893 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.893 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.893 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6808", 2024-03-20T09:41:21.893 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:41:21.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.893 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.893 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.893 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6809", 2024-03-20T09:41:21.893 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:41:21.893 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.893 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.893 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.894 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:41:21.894 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.894 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.894 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6806", 2024-03-20T09:41:21.894 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:41:21.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.894 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.894 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.894 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6807", 2024-03-20T09:41:21.894 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2139805293 2024-03-20T09:41:21.894 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.894 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.894 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.895 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6803/2139805293", 2024-03-20T09:41:21.895 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6805/2139805293", 2024-03-20T09:41:21.895 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6809/2139805293", 2024-03-20T09:41:21.895 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6807/2139805293", 2024-03-20T09:41:21.895 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:41:21.895 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:41:21.895 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:41:21.895 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.895 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.895 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.895 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-03-20T09:41:21.895 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e85b94f5-2954-47a1-9c20-096f2606eb04", 2024-03-20T09:41:21.895 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:41:21.895 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:41:21.895 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:41:21.895 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 0.0307159423828125, 2024-03-20T09:41:21.896 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:41:21.896 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:41:21.896 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 13, 2024-03-20T09:41:21.896 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 814, 2024-03-20T09:41:21.896 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:41:21.896 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:41:21.896 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:41:21.896 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.896 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.896 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6810", 2024-03-20T09:41:21.896 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:41:21.896 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.896 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.896 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.897 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6811", 2024-03-20T09:41:21.897 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:41:21.897 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.897 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.897 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:41:21.897 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.897 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.897 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6812", 2024-03-20T09:41:21.897 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:41:21.897 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.897 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.897 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.898 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6813", 2024-03-20T09:41:21.898 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:41:21.898 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.898 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.898 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:41:21.898 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.898 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.898 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6816", 2024-03-20T09:41:21.898 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:41:21.898 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.898 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.898 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.899 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6817", 2024-03-20T09:41:21.899 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:41:21.899 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.899 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.899 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:41:21.899 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.899 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.899 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6814", 2024-03-20T09:41:21.899 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:41:21.899 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.899 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.899 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.899 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6815", 2024-03-20T09:41:21.900 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3853355134 2024-03-20T09:41:21.900 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.900 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.900 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6811/3853355134", 2024-03-20T09:41:21.900 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6813/3853355134", 2024-03-20T09:41:21.900 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6817/3853355134", 2024-03-20T09:41:21.900 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6815/3853355134", 2024-03-20T09:41:21.900 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:41:21.900 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:41:21.900 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:41:21.900 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.900 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.901 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.901 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-03-20T09:41:21.901 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "dd2b10f0-741c-47a4-b224-ac6bfe2c14ef", 2024-03-20T09:41:21.901 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:41:21.901 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:41:21.901 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:41:21.901 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:41:21.901 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:41:21.901 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:41:21.901 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 18, 2024-03-20T09:41:21.901 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 1054, 2024-03-20T09:41:21.901 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:41:21.901 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:41:21.901 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:41:21.901 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.902 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.902 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6818", 2024-03-20T09:41:21.902 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:41:21.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.902 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.902 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.902 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6819", 2024-03-20T09:41:21.902 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:41:21.902 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.902 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.902 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.902 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:41:21.902 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.903 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.903 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6820", 2024-03-20T09:41:21.903 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:41:21.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.903 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.903 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6821", 2024-03-20T09:41:21.903 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:41:21.903 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.903 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.903 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.903 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:41:21.903 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.903 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.903 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.904 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6824", 2024-03-20T09:41:21.904 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:41:21.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.904 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.904 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6825", 2024-03-20T09:41:21.904 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:41:21.904 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.904 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.904 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.904 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:41:21.904 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.904 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.904 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.904 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6822", 2024-03-20T09:41:21.905 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:41:21.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.905 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.905 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.905 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6823", 2024-03-20T09:41:21.905 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3993357967 2024-03-20T09:41:21.905 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.905 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.905 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.905 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6819/3993357967", 2024-03-20T09:41:21.905 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6821/3993357967", 2024-03-20T09:41:21.905 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6825/3993357967", 2024-03-20T09:41:21.905 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6823/3993357967", 2024-03-20T09:41:21.905 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:41:21.905 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:41:21.905 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:41:21.906 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.906 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.906 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.906 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-03-20T09:41:21.906 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1", 2024-03-20T09:41:21.906 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:41:21.906 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:41:21.906 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:41:21.906 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:41:21.906 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:41:21.906 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:41:21.906 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 26, 2024-03-20T09:41:21.906 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 1066, 2024-03-20T09:41:21.906 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:41:21.907 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:41:21.907 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:41:21.907 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.907 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.907 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6826", 2024-03-20T09:41:21.907 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:41:21.907 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.907 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.907 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.907 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6827", 2024-03-20T09:41:21.907 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:41:21.907 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.907 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.908 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:41:21.908 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.908 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.908 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6828", 2024-03-20T09:41:21.908 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:41:21.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.908 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.908 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.908 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6829", 2024-03-20T09:41:21.908 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:41:21.908 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.908 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.908 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.908 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:41:21.909 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.909 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.909 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6832", 2024-03-20T09:41:21.909 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:41:21.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.909 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.909 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.909 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6833", 2024-03-20T09:41:21.909 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:41:21.909 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.909 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.909 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.909 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:41:21.909 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.910 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.910 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6830", 2024-03-20T09:41:21.910 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:41:21.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.910 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.910 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.910 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.26:6831", 2024-03-20T09:41:21.910 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 4110225509 2024-03-20T09:41:21.910 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.910 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.910 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.910 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.26:6827/4110225509", 2024-03-20T09:41:21.910 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.26:6829/4110225509", 2024-03-20T09:41:21.911 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.26:6833/4110225509", 2024-03-20T09:41:21.911 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.26:6831/4110225509", 2024-03-20T09:41:21.911 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:41:21.911 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:41:21.911 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:41:21.911 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.911 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.911 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.911 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-03-20T09:41:21.911 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "f14e34d5-9829-42a3-9b25-31a0976c5ef7", 2024-03-20T09:41:21.911 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:41:21.911 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:41:21.911 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:41:21.911 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:41:21.911 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:41:21.912 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:41:21.912 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 32, 2024-03-20T09:41:21.912 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 1066, 2024-03-20T09:41:21.912 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:41:21.912 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:41:21.912 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:41:21.912 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.912 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.912 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6800", 2024-03-20T09:41:21.912 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:41:21.912 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.912 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.913 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.913 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6801", 2024-03-20T09:41:21.913 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:41:21.913 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.913 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.913 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:41:21.913 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.913 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.913 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6802", 2024-03-20T09:41:21.913 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:41:21.913 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.913 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.914 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.914 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6803", 2024-03-20T09:41:21.914 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:41:21.914 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.914 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.914 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:41:21.914 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.914 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.914 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.914 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6806", 2024-03-20T09:41:21.914 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:41:21.914 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.915 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.915 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6807", 2024-03-20T09:41:21.915 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:41:21.915 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.915 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.915 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:41:21.915 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.915 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.915 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.915 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6804", 2024-03-20T09:41:21.915 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:41:21.915 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.916 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.916 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.916 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6805", 2024-03-20T09:41:21.916 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 477923058 2024-03-20T09:41:21.916 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.916 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.916 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.916 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6801/477923058", 2024-03-20T09:41:21.916 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6803/477923058", 2024-03-20T09:41:21.916 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6807/477923058", 2024-03-20T09:41:21.916 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6805/477923058", 2024-03-20T09:41:21.916 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:41:21.916 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:41:21.916 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:41:21.917 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.917 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.917 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.917 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-03-20T09:41:21.917 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "cb431b6a-540d-4318-ae35-d379e1df0185", 2024-03-20T09:41:21.917 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:41:21.917 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:41:21.917 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:41:21.917 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 0.9279937744140625, 2024-03-20T09:41:21.917 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 41, 2024-03-20T09:41:21.917 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 273, 2024-03-20T09:41:21.917 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 443, 2024-03-20T09:41:21.917 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 1066, 2024-03-20T09:41:21.917 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 274, 2024-03-20T09:41:21.917 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:41:21.918 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:41:21.918 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.918 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.918 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6808", 2024-03-20T09:41:21.918 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:41:21.918 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.918 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.918 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.918 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6809", 2024-03-20T09:41:21.918 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:41:21.918 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.918 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.919 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:41:21.919 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.919 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.919 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6810", 2024-03-20T09:41:21.919 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:41:21.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.919 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.919 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.919 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6811", 2024-03-20T09:41:21.919 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:41:21.919 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.919 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.919 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.920 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:41:21.920 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.920 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.920 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6814", 2024-03-20T09:41:21.920 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:41:21.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.920 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.920 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.920 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6815", 2024-03-20T09:41:21.920 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:41:21.920 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.920 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.920 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.921 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:41:21.921 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.921 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.921 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6812", 2024-03-20T09:41:21.921 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:41:21.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.921 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.921 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.921 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6813", 2024-03-20T09:41:21.921 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 650416648 2024-03-20T09:41:21.921 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.921 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.921 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.921 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6809/650416648", 2024-03-20T09:41:21.921 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6811/650416648", 2024-03-20T09:41:21.922 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6815/650416648", 2024-03-20T09:41:21.922 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6813/650416648", 2024-03-20T09:41:21.922 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:41:21.922 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:41:21.922 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:41:21.922 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.922 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.922 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.922 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2024-03-20T09:41:21.922 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "753ed830-e764-4ff3-9d9b-87e7fa748b81", 2024-03-20T09:41:21.922 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:41:21.922 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:41:21.922 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:41:21.923 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:41:21.923 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:41:21.923 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:41:21.923 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 51, 2024-03-20T09:41:21.923 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 1066, 2024-03-20T09:41:21.923 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:41:21.923 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:41:21.923 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:41:21.923 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.923 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.923 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.923 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6816", 2024-03-20T09:41:21.923 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:41:21.923 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.924 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.924 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6817", 2024-03-20T09:41:21.924 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:41:21.924 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.924 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.924 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:41:21.924 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.924 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.924 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.924 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6818", 2024-03-20T09:41:21.924 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:41:21.924 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.925 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.925 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6819", 2024-03-20T09:41:21.925 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:41:21.925 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.925 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.925 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:41:21.925 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.925 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.925 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6822", 2024-03-20T09:41:21.925 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:41:21.925 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.925 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.926 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.926 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6823", 2024-03-20T09:41:21.926 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:41:21.926 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.926 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.926 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:41:21.926 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.926 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.926 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.926 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6820", 2024-03-20T09:41:21.926 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:41:21.926 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.927 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.927 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.927 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6821", 2024-03-20T09:41:21.927 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3278301843 2024-03-20T09:41:21.927 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.927 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.927 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.927 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6817/3278301843", 2024-03-20T09:41:21.927 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6819/3278301843", 2024-03-20T09:41:21.927 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6823/3278301843", 2024-03-20T09:41:21.927 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6821/3278301843", 2024-03-20T09:41:21.927 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:41:21.927 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:41:21.927 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:41:21.927 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.928 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.928 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.928 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2024-03-20T09:41:21.928 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "55bcbd37-7a6f-4ab1-b5a9-91a951444d86", 2024-03-20T09:41:21.928 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:41:21.928 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:41:21.928 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:41:21.928 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 0.058074951171875, 2024-03-20T09:41:21.928 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:41:21.928 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:41:21.928 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 60, 2024-03-20T09:41:21.928 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 517, 2024-03-20T09:41:21.928 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:41:21.928 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:41:21.929 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:41:21.929 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.929 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.929 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6824", 2024-03-20T09:41:21.929 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:41:21.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.929 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.929 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.929 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6825", 2024-03-20T09:41:21.929 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:41:21.929 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.929 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.929 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.930 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:41:21.930 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.930 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.930 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6826", 2024-03-20T09:41:21.930 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:41:21.930 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.930 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.930 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.930 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6827", 2024-03-20T09:41:21.930 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:41:21.930 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.930 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.931 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:41:21.931 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.931 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.931 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6830", 2024-03-20T09:41:21.931 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:41:21.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.931 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.931 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.931 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6831", 2024-03-20T09:41:21.931 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:41:21.931 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.931 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.931 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.931 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:41:21.932 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.932 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.932 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6828", 2024-03-20T09:41:21.932 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:41:21.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.932 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.932 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.932 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.115:6829", 2024-03-20T09:41:21.932 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 3152424001 2024-03-20T09:41:21.932 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.932 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.932 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.932 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.115:6825/3152424001", 2024-03-20T09:41:21.932 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.115:6827/3152424001", 2024-03-20T09:41:21.932 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.115:6831/3152424001", 2024-03-20T09:41:21.933 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.115:6829/3152424001", 2024-03-20T09:41:21.933 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:41:21.933 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:41:21.933 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:41:21.933 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.933 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.933 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.933 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 8, 2024-03-20T09:41:21.933 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "bc86248e-6bd0-4369-b8a2-e593bc2a610d", 2024-03-20T09:41:21.933 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:41:21.933 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:41:21.933 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:41:21.933 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:41:21.933 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:41:21.933 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:41:21.934 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 70, 2024-03-20T09:41:21.934 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 1066, 2024-03-20T09:41:21.934 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:41:21.934 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:41:21.934 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:41:21.934 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.934 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.934 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6800", 2024-03-20T09:41:21.934 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:41:21.934 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.934 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.934 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.935 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6801", 2024-03-20T09:41:21.935 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:41:21.935 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.935 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.935 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:41:21.935 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.935 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.935 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6802", 2024-03-20T09:41:21.935 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:41:21.935 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.935 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.935 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.935 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6803", 2024-03-20T09:41:21.936 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:41:21.936 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.936 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.936 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:41:21.936 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.936 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.936 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6806", 2024-03-20T09:41:21.936 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:41:21.936 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.936 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.936 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.936 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6807", 2024-03-20T09:41:21.936 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:41:21.937 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.937 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.937 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:41:21.937 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.937 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.937 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6804", 2024-03-20T09:41:21.937 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:41:21.937 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.937 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.937 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.937 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6805", 2024-03-20T09:41:21.938 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 1309426740 2024-03-20T09:41:21.938 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.938 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.938 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6801/1309426740", 2024-03-20T09:41:21.938 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6803/1309426740", 2024-03-20T09:41:21.938 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6807/1309426740", 2024-03-20T09:41:21.938 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6805/1309426740", 2024-03-20T09:41:21.938 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:41:21.938 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:41:21.938 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:41:21.938 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.938 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.938 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.938 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 9, 2024-03-20T09:41:21.939 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "9b20a4aa-032a-4c09-ac75-153c554424e7", 2024-03-20T09:41:21.939 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:41:21.939 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:41:21.939 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:41:21.939 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:41:21.939 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:41:21.939 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:41:21.939 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 80, 2024-03-20T09:41:21.939 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 1066, 2024-03-20T09:41:21.939 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:41:21.939 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:41:21.939 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:41:21.939 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.939 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.939 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.940 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6808", 2024-03-20T09:41:21.940 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:41:21.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.940 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.940 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6809", 2024-03-20T09:41:21.940 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:41:21.940 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.940 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.940 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.940 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:41:21.940 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.940 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.940 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.940 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6810", 2024-03-20T09:41:21.941 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:41:21.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.941 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.941 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6811", 2024-03-20T09:41:21.941 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:41:21.941 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.941 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.941 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.941 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:41:21.941 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.941 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.941 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.941 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6814", 2024-03-20T09:41:21.942 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:41:21.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.942 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.942 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6815", 2024-03-20T09:41:21.942 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:41:21.942 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.942 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.942 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.942 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:41:21.942 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.942 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.942 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.942 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6812", 2024-03-20T09:41:21.943 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:41:21.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.943 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.943 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.943 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6813", 2024-03-20T09:41:21.943 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2953210076 2024-03-20T09:41:21.943 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.943 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.943 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.943 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6809/2953210076", 2024-03-20T09:41:21.943 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6811/2953210076", 2024-03-20T09:41:21.943 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6815/2953210076", 2024-03-20T09:41:21.943 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6813/2953210076", 2024-03-20T09:41:21.943 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:41:21.944 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:41:21.944 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:41:21.944 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.944 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.944 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.944 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 10, 2024-03-20T09:41:21.944 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "e62c1fb3-5dd1-4d0d-997c-d20633d19361", 2024-03-20T09:41:21.944 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:41:21.944 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:41:21.944 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:41:21.944 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:41:21.944 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 89, 2024-03-20T09:41:21.944 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 753, 2024-03-20T09:41:21.944 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 861, 2024-03-20T09:41:21.945 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 1066, 2024-03-20T09:41:21.945 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 754, 2024-03-20T09:41:21.945 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:41:21.945 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:41:21.945 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.945 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.945 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6816", 2024-03-20T09:41:21.945 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2853088860 2024-03-20T09:41:21.945 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.945 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.945 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.945 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6817", 2024-03-20T09:41:21.945 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2853088860 2024-03-20T09:41:21.946 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.946 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.946 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:41:21.946 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.946 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.946 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.946 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6818", 2024-03-20T09:41:21.946 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2853088860 2024-03-20T09:41:21.946 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.946 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.946 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.946 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6819", 2024-03-20T09:41:21.946 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2853088860 2024-03-20T09:41:21.947 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.947 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.947 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:41:21.947 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.947 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.947 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6822", 2024-03-20T09:41:21.947 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2853088860 2024-03-20T09:41:21.947 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.947 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.947 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.947 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6823", 2024-03-20T09:41:21.947 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2853088860 2024-03-20T09:41:21.948 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.948 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.948 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:41:21.948 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.948 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.948 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6820", 2024-03-20T09:41:21.948 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2853088860 2024-03-20T09:41:21.948 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.948 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.948 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.948 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6821", 2024-03-20T09:41:21.948 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2853088860 2024-03-20T09:41:21.949 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.949 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.949 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6817/2853088860", 2024-03-20T09:41:21.949 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6819/2853088860", 2024-03-20T09:41:21.949 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6823/2853088860", 2024-03-20T09:41:21.949 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6821/2853088860", 2024-03-20T09:41:21.949 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:41:21.949 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:41:21.949 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:41:21.949 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.949 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.949 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.949 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 11, 2024-03-20T09:41:21.949 INFO:teuthology.orchestra.run.smithi026.stdout: "uuid": "b83994d5-e5ef-48b0-9758-c26843cec3b6", 2024-03-20T09:41:21.949 INFO:teuthology.orchestra.run.smithi026.stdout: "up": 1, 2024-03-20T09:41:21.950 INFO:teuthology.orchestra.run.smithi026.stdout: "in": 1, 2024-03-20T09:41:21.950 INFO:teuthology.orchestra.run.smithi026.stdout: "weight": 1, 2024-03-20T09:41:21.950 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_affinity": 1, 2024-03-20T09:41:21.950 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_begin": 0, 2024-03-20T09:41:21.950 INFO:teuthology.orchestra.run.smithi026.stdout: "last_clean_end": 0, 2024-03-20T09:41:21.950 INFO:teuthology.orchestra.run.smithi026.stdout: "up_from": 96, 2024-03-20T09:41:21.950 INFO:teuthology.orchestra.run.smithi026.stdout: "up_thru": 1066, 2024-03-20T09:41:21.950 INFO:teuthology.orchestra.run.smithi026.stdout: "down_at": 0, 2024-03-20T09:41:21.950 INFO:teuthology.orchestra.run.smithi026.stdout: "lost_at": 0, 2024-03-20T09:41:21.950 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addrs": { 2024-03-20T09:41:21.950 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.950 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.950 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.951 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6824", 2024-03-20T09:41:21.951 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:41:21.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.951 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.951 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6825", 2024-03-20T09:41:21.951 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:41:21.951 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.951 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.951 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.951 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addrs": { 2024-03-20T09:41:21.951 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.951 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.951 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.951 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6826", 2024-03-20T09:41:21.951 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:41:21.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.952 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.952 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6827", 2024-03-20T09:41:21.952 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:41:21.952 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.952 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.952 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addrs": { 2024-03-20T09:41:21.952 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.952 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.952 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6830", 2024-03-20T09:41:21.952 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:41:21.952 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.952 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.952 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.953 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6831", 2024-03-20T09:41:21.953 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:41:21.953 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.953 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.953 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addrs": { 2024-03-20T09:41:21.953 INFO:teuthology.orchestra.run.smithi026.stdout: "addrvec": [ 2024-03-20T09:41:21.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.953 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v2", 2024-03-20T09:41:21.953 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6828", 2024-03-20T09:41:21.953 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:41:21.953 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.953 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.953 INFO:teuthology.orchestra.run.smithi026.stdout: "type": "v1", 2024-03-20T09:41:21.953 INFO:teuthology.orchestra.run.smithi026.stdout: "addr": "172.21.15.122:6829", 2024-03-20T09:41:21.954 INFO:teuthology.orchestra.run.smithi026.stdout: "nonce": 2704069354 2024-03-20T09:41:21.954 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.954 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.954 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.954 INFO:teuthology.orchestra.run.smithi026.stdout: "public_addr": "172.21.15.122:6825/2704069354", 2024-03-20T09:41:21.954 INFO:teuthology.orchestra.run.smithi026.stdout: "cluster_addr": "172.21.15.122:6827/2704069354", 2024-03-20T09:41:21.954 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_back_addr": "172.21.15.122:6831/2704069354", 2024-03-20T09:41:21.954 INFO:teuthology.orchestra.run.smithi026.stdout: "heartbeat_front_addr": "172.21.15.122:6829/2704069354", 2024-03-20T09:41:21.954 INFO:teuthology.orchestra.run.smithi026.stdout: "state": [ 2024-03-20T09:41:21.954 INFO:teuthology.orchestra.run.smithi026.stdout: "exists", 2024-03-20T09:41:21.954 INFO:teuthology.orchestra.run.smithi026.stdout: "up" 2024-03-20T09:41:21.954 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.954 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.954 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:41:21.954 INFO:teuthology.orchestra.run.smithi026.stdout: "osd_xinfo": [ 2024-03-20T09:41:21.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.955 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 0, 2024-03-20T09:41:21.955 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:41:21.955 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:41:21.955 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:41:21.955 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:41:21.955 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 65536, 2024-03-20T09:41:21.955 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:40:12.115649+0000", 2024-03-20T09:41:21.955 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:41:21.955 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.955 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.955 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 1, 2024-03-20T09:41:21.955 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:41:21.955 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:41:21.955 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:41:21.956 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:41:21.956 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:41:21.956 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:40:36.398914+0000", 2024-03-20T09:41:21.956 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:41:21.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.956 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.956 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 2, 2024-03-20T09:41:21.956 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:41:21.956 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:41:21.956 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:41:21.956 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:41:21.956 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:41:21.956 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:41:12.918727+0000", 2024-03-20T09:41:21.956 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:41:21.956 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.957 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 3, 2024-03-20T09:41:21.957 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:41:21.957 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:41:21.957 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:41:21.957 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:41:21.957 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:41:21.957 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:40:09.835254+0000", 2024-03-20T09:41:21.957 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:41:21.957 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.957 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.957 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 4, 2024-03-20T09:41:21.957 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:41:21.957 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:41:21.957 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:41:21.958 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:41:21.958 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:41:21.958 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:40:31.856113+0000", 2024-03-20T09:41:21.958 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:41:21.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.958 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.958 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 5, 2024-03-20T09:41:21.958 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:41:21.958 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:41:21.958 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:41:21.958 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:41:21.958 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:41:21.958 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:40:31.703122+0000", 2024-03-20T09:41:21.958 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 273 2024-03-20T09:41:21.958 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.959 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 6, 2024-03-20T09:41:21.959 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:41:21.959 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:41:21.959 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:41:21.959 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:41:21.959 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:41:21.959 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:40:22.303344+0000", 2024-03-20T09:41:21.959 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:41:21.959 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.959 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.959 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 7, 2024-03-20T09:41:21.959 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:41:21.959 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:41:21.959 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:41:21.959 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:41:21.960 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:41:21.960 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:40:16.091733+0000", 2024-03-20T09:41:21.960 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:41:21.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.960 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 8, 2024-03-20T09:41:21.960 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:41:21.960 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:41:21.960 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:41:21.960 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:41:21.960 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:41:21.960 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:41:06.866413+0000", 2024-03-20T09:41:21.960 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:41:21.960 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.960 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.961 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 9, 2024-03-20T09:41:21.961 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:41:21.961 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:41:21.961 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:41:21.961 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:41:21.961 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:41:21.961 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:40:31.833383+0000", 2024-03-20T09:41:21.961 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:41:21.961 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.961 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.961 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 10, 2024-03-20T09:41:21.961 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:41:21.961 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:41:21.961 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:41:21.961 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:41:21.961 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:41:21.962 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:40:28.071187+0000", 2024-03-20T09:41:21.962 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 753 2024-03-20T09:41:21.962 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.962 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.962 INFO:teuthology.orchestra.run.smithi026.stdout: "osd": 11, 2024-03-20T09:41:21.962 INFO:teuthology.orchestra.run.smithi026.stdout: "down_stamp": "0.000000", 2024-03-20T09:41:21.962 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_probability": 0, 2024-03-20T09:41:21.962 INFO:teuthology.orchestra.run.smithi026.stdout: "laggy_interval": 0, 2024-03-20T09:41:21.962 INFO:teuthology.orchestra.run.smithi026.stdout: "features": 4540701547738038271, 2024-03-20T09:41:21.962 INFO:teuthology.orchestra.run.smithi026.stdout: "old_weight": 0, 2024-03-20T09:41:21.962 INFO:teuthology.orchestra.run.smithi026.stdout: "last_purged_snaps_scrub": "2024-03-20T09:40:34.310869+0000", 2024-03-20T09:41:21.962 INFO:teuthology.orchestra.run.smithi026.stdout: "dead_epoch": 0 2024-03-20T09:41:21.963 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.963 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:41:21.963 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap": [ 2024-03-20T09:41:21.963 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:21.963 INFO:teuthology.orchestra.run.smithi026.stdout: "pgid": "1.1", 2024-03-20T09:41:21.963 INFO:teuthology.orchestra.run.smithi026.stdout: "osds": [ 2024-03-20T09:41:21.963 INFO:teuthology.orchestra.run.smithi026.stdout: 6, 2024-03-20T09:41:21.963 INFO:teuthology.orchestra.run.smithi026.stdout: 11, 2024-03-20T09:41:21.963 INFO:teuthology.orchestra.run.smithi026.stdout: 5 2024-03-20T09:41:21.963 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:21.963 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.963 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:41:21.963 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_items": [], 2024-03-20T09:41:21.963 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_upmap_primaries": [], 2024-03-20T09:41:21.964 INFO:teuthology.orchestra.run.smithi026.stdout: "pg_temp": [], 2024-03-20T09:41:21.964 INFO:teuthology.orchestra.run.smithi026.stdout: "primary_temp": [], 2024-03-20T09:41:21.964 INFO:teuthology.orchestra.run.smithi026.stdout: "blocklist": { 2024-03-20T09:41:21.964 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/4291404191": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:41:21.964 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/642012566": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:41:21.964 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6801/1181459999": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:41:21.964 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/3984513952": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:41:21.964 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6800/1654945498": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:41:21.964 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6800/1181459999": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:41:21.964 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/137770981": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:41:21.964 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/1451237762": "2024-03-21T08:49:35.336921+0000", 2024-03-20T09:41:21.964 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:0/3938386631": "2024-03-21T08:49:04.596754+0000", 2024-03-20T09:41:21.964 INFO:teuthology.orchestra.run.smithi026.stdout: "172.21.15.26:6801/1654945498": "2024-03-21T08:49:04.596754+0000" 2024-03-20T09:41:21.964 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.965 INFO:teuthology.orchestra.run.smithi026.stdout: "range_blocklist": {}, 2024-03-20T09:41:21.965 INFO:teuthology.orchestra.run.smithi026.stdout: "erasure_code_profiles": { 2024-03-20T09:41:21.965 INFO:teuthology.orchestra.run.smithi026.stdout: "default": { 2024-03-20T09:41:21.965 INFO:teuthology.orchestra.run.smithi026.stdout: "crush-failure-domain": "osd", 2024-03-20T09:41:21.965 INFO:teuthology.orchestra.run.smithi026.stdout: "k": "2", 2024-03-20T09:41:21.965 INFO:teuthology.orchestra.run.smithi026.stdout: "m": "1", 2024-03-20T09:41:21.965 INFO:teuthology.orchestra.run.smithi026.stdout: "plugin": "jerasure", 2024-03-20T09:41:21.965 INFO:teuthology.orchestra.run.smithi026.stdout: "technique": "reed_sol_van" 2024-03-20T09:41:21.965 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.965 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:21.965 INFO:teuthology.orchestra.run.smithi026.stdout: "removed_snaps_queue": [], 2024-03-20T09:41:21.965 INFO:teuthology.orchestra.run.smithi026.stdout: "new_removed_snaps": [], 2024-03-20T09:41:21.965 INFO:teuthology.orchestra.run.smithi026.stdout: "new_purged_snaps": [], 2024-03-20T09:41:21.965 INFO:teuthology.orchestra.run.smithi026.stdout: "crush_node_flags": {}, 2024-03-20T09:41:21.965 INFO:teuthology.orchestra.run.smithi026.stdout: "device_class_flags": {}, 2024-03-20T09:41:21.966 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode": { 2024-03-20T09:41:21.966 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_enabled": false, 2024-03-20T09:41:21.966 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_bucket_count": 0, 2024-03-20T09:41:21.966 INFO:teuthology.orchestra.run.smithi026.stdout: "degraded_stretch_mode": 0, 2024-03-20T09:41:21.966 INFO:teuthology.orchestra.run.smithi026.stdout: "recovering_stretch_mode": 0, 2024-03-20T09:41:21.966 INFO:teuthology.orchestra.run.smithi026.stdout: "stretch_mode_bucket": 0 2024-03-20T09:41:21.966 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:21.966 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:41:21.967 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:21 smithi026 ceph-mon[68309]: pgmap v2625: 47 pgs: 47 active+clean; 5.0 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-03-20T09:41:21.967 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:21 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4176686525' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:41:22.072 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:41:22.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:21 smithi115 ceph-mon[72476]: pgmap v2625: 47 pgs: 47 active+clean; 5.0 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-03-20T09:41:22.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:21 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4176686525' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:41:22.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:21 smithi122 ceph-mon[73495]: pgmap v2625: 47 pgs: 47 active+clean; 5.0 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-03-20T09:41:22.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:21 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4176686525' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-20T09:41:22.820 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 43 16 87958 87942 127.782 91.0625 0.00718616 0.00782213 2024-03-20T09:41:23.327 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:41:23.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:23.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:41:23.358 INFO:tasks.thrashosds.thrasher:j is {'epoch': 1083, 'fsid': '852fbb68-e696-11ee-95c9-87774f69a715', 'created': '2024-03-20T08:48:35.281088+0000', 'modified': '2024-03-20T09:41:12.925149+0000', 'last_up_change': '2024-03-20T09:30:44.068387+0000', 'last_in_change': '2024-03-20T09:39:13.893852+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 28, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-20T08:52:17.442324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '360', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '358', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.319999933242798, 'score_stable': 2.319999933242798, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 2.130000114440918, 'raw_score_stable': 2.130000114440918, 'primary_affinity_weighted': 0.8347319960594177, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 9, 'pool_name': 'unique_pool_7', 'create_time': '2024-03-20T09:40:23.232804+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '1073', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.009999990463257, 'score_stable': 3.009999990463257, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.8347319960594177, 'average_primary_affinity': 0.8299999833106995, 'average_primary_affinity_weighted': 0.8299999833106995}}], 'osds': [{'osd': 0, 'uuid': 'ad3fd9c0-316f-4fe5-a63b-d480087c1729', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 974, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6802', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6803', 'nonce': 2139805293}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6804', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6805', 'nonce': 2139805293}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6808', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6809', 'nonce': 2139805293}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6806', 'nonce': 2139805293}, {'type': 'v1', 'addr': '172.21.15.26:6807', 'nonce': 2139805293}]}, 'public_addr': '172.21.15.26:6803/2139805293', 'cluster_addr': '172.21.15.26:6805/2139805293', 'heartbeat_back_addr': '172.21.15.26:6809/2139805293', 'heartbeat_front_addr': '172.21.15.26:6807/2139805293', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'e85b94f5-2954-47a1-9c20-096f2606eb04', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.0307159423828125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 814, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6810', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6811', 'nonce': 3853355134}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6812', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6813', 'nonce': 3853355134}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6816', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6817', 'nonce': 3853355134}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6814', 'nonce': 3853355134}, {'type': 'v1', 'addr': '172.21.15.26:6815', 'nonce': 3853355134}]}, 'public_addr': '172.21.15.26:6811/3853355134', 'cluster_addr': '172.21.15.26:6813/3853355134', 'heartbeat_back_addr': '172.21.15.26:6817/3853355134', 'heartbeat_front_addr': '172.21.15.26:6815/3853355134', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'dd2b10f0-741c-47a4-b224-ac6bfe2c14ef', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 1054, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6818', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6819', 'nonce': 3993357967}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6820', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6821', 'nonce': 3993357967}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6824', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6825', 'nonce': 3993357967}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6822', 'nonce': 3993357967}, {'type': 'v1', 'addr': '172.21.15.26:6823', 'nonce': 3993357967}]}, 'public_addr': '172.21.15.26:6819/3993357967', 'cluster_addr': '172.21.15.26:6821/3993357967', 'heartbeat_back_addr': '172.21.15.26:6825/3993357967', 'heartbeat_front_addr': '172.21.15.26:6823/3993357967', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 1066, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6826', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6827', 'nonce': 4110225509}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6828', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6829', 'nonce': 4110225509}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6832', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6833', 'nonce': 4110225509}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.26:6830', 'nonce': 4110225509}, {'type': 'v1', 'addr': '172.21.15.26:6831', 'nonce': 4110225509}]}, 'public_addr': '172.21.15.26:6827/4110225509', 'cluster_addr': '172.21.15.26:6829/4110225509', 'heartbeat_back_addr': '172.21.15.26:6833/4110225509', 'heartbeat_front_addr': '172.21.15.26:6831/4110225509', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'f14e34d5-9829-42a3-9b25-31a0976c5ef7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 1066, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6800', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6801', 'nonce': 477923058}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6802', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6803', 'nonce': 477923058}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6806', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6807', 'nonce': 477923058}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6804', 'nonce': 477923058}, {'type': 'v1', 'addr': '172.21.15.115:6805', 'nonce': 477923058}]}, 'public_addr': '172.21.15.115:6801/477923058', 'cluster_addr': '172.21.15.115:6803/477923058', 'heartbeat_back_addr': '172.21.15.115:6807/477923058', 'heartbeat_front_addr': '172.21.15.115:6805/477923058', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'cb431b6a-540d-4318-ae35-d379e1df0185', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.9279937744140625, 'last_clean_begin': 41, 'last_clean_end': 273, 'up_from': 443, 'up_thru': 1066, 'down_at': 274, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6808', 'nonce': 650416648}, {'type': 'v1', 'addr': '172.21.15.115:6809', 'nonce': 650416648}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6810', 'nonce': 650416648}, {'type': 'v1', 'addr': '172.21.15.115:6811', 'nonce': 650416648}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6814', 'nonce': 650416648}, {'type': 'v1', 'addr': '172.21.15.115:6815', 'nonce': 650416648}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6812', 'nonce': 650416648}, {'type': 'v1', 'addr': '172.21.15.115:6813', 'nonce': 650416648}]}, 'public_addr': '172.21.15.115:6809/650416648', 'cluster_addr': '172.21.15.115:6811/650416648', 'heartbeat_back_addr': '172.21.15.115:6815/650416648', 'heartbeat_front_addr': '172.21.15.115:6813/650416648', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '753ed830-e764-4ff3-9d9b-87e7fa748b81', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 1066, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6816', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6817', 'nonce': 3278301843}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6818', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6819', 'nonce': 3278301843}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6822', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6823', 'nonce': 3278301843}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6820', 'nonce': 3278301843}, {'type': 'v1', 'addr': '172.21.15.115:6821', 'nonce': 3278301843}]}, 'public_addr': '172.21.15.115:6817/3278301843', 'cluster_addr': '172.21.15.115:6819/3278301843', 'heartbeat_back_addr': '172.21.15.115:6823/3278301843', 'heartbeat_front_addr': '172.21.15.115:6821/3278301843', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '55bcbd37-7a6f-4ab1-b5a9-91a951444d86', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.058074951171875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 517, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6824', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6825', 'nonce': 3152424001}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6826', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6827', 'nonce': 3152424001}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6830', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6831', 'nonce': 3152424001}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.115:6828', 'nonce': 3152424001}, {'type': 'v1', 'addr': '172.21.15.115:6829', 'nonce': 3152424001}]}, 'public_addr': '172.21.15.115:6825/3152424001', 'cluster_addr': '172.21.15.115:6827/3152424001', 'heartbeat_back_addr': '172.21.15.115:6831/3152424001', 'heartbeat_front_addr': '172.21.15.115:6829/3152424001', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'bc86248e-6bd0-4369-b8a2-e593bc2a610d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 70, 'up_thru': 1066, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6800', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6801', 'nonce': 1309426740}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6802', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6803', 'nonce': 1309426740}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6806', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6807', 'nonce': 1309426740}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6804', 'nonce': 1309426740}, {'type': 'v1', 'addr': '172.21.15.122:6805', 'nonce': 1309426740}]}, 'public_addr': '172.21.15.122:6801/1309426740', 'cluster_addr': '172.21.15.122:6803/1309426740', 'heartbeat_back_addr': '172.21.15.122:6807/1309426740', 'heartbeat_front_addr': '172.21.15.122:6805/1309426740', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '9b20a4aa-032a-4c09-ac75-153c554424e7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 1066, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6808', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6809', 'nonce': 2953210076}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6810', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6811', 'nonce': 2953210076}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6814', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6815', 'nonce': 2953210076}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6812', 'nonce': 2953210076}, {'type': 'v1', 'addr': '172.21.15.122:6813', 'nonce': 2953210076}]}, 'public_addr': '172.21.15.122:6809/2953210076', 'cluster_addr': '172.21.15.122:6811/2953210076', 'heartbeat_back_addr': '172.21.15.122:6815/2953210076', 'heartbeat_front_addr': '172.21.15.122:6813/2953210076', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'e62c1fb3-5dd1-4d0d-997c-d20633d19361', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 89, 'last_clean_end': 753, 'up_from': 861, 'up_thru': 1066, 'down_at': 754, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6816', 'nonce': 2853088860}, {'type': 'v1', 'addr': '172.21.15.122:6817', 'nonce': 2853088860}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6818', 'nonce': 2853088860}, {'type': 'v1', 'addr': '172.21.15.122:6819', 'nonce': 2853088860}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6822', 'nonce': 2853088860}, {'type': 'v1', 'addr': '172.21.15.122:6823', 'nonce': 2853088860}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6820', 'nonce': 2853088860}, {'type': 'v1', 'addr': '172.21.15.122:6821', 'nonce': 2853088860}]}, 'public_addr': '172.21.15.122:6817/2853088860', 'cluster_addr': '172.21.15.122:6819/2853088860', 'heartbeat_back_addr': '172.21.15.122:6823/2853088860', 'heartbeat_front_addr': '172.21.15.122:6821/2853088860', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'b83994d5-e5ef-48b0-9758-c26843cec3b6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 96, 'up_thru': 1066, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6824', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6825', 'nonce': 2704069354}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6826', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6827', 'nonce': 2704069354}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6830', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6831', 'nonce': 2704069354}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6828', 'nonce': 2704069354}, {'type': 'v1', 'addr': '172.21.15.122:6829', 'nonce': 2704069354}]}, 'public_addr': '172.21.15.122:6825/2704069354', 'cluster_addr': '172.21.15.122:6827/2704069354', 'heartbeat_back_addr': '172.21.15.122:6831/2704069354', 'heartbeat_front_addr': '172.21.15.122:6829/2704069354', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-20T09:40:12.115649+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:40:36.398914+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:41:12.918727+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:40:09.835254+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:40:31.856113+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:40:31.703122+0000', 'dead_epoch': 273}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:40:22.303344+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:40:16.091733+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:41:06.866413+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:40:31.833383+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:40:28.071187+0000', 'dead_epoch': 753}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-20T09:40:34.310869+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.1', 'osds': [6, 11, 5]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.26:0/4291404191': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/642012566': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:6801/1181459999': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/3984513952': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6800/1654945498': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6800/1181459999': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/137770981': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:0/1451237762': '2024-03-21T08:49:35.336921+0000', '172.21.15.26:0/3938386631': '2024-03-21T08:49:04.596754+0000', '172.21.15.26:6801/1654945498': '2024-03-21T08:49:04.596754+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-20T09:41:23.359 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:41:23.434 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:41:23.482 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:23.516 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:41:23.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:23.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:41:23.623 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:41:23.674 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:23.700 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:41:23.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:23.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:41:23.808 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:41:23.822 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 44 16 89379 89363 126.891 88.8125 0.00965223 0.00787661 2024-03-20T09:41:23.846 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:24.051 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:41:24.057 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:23 smithi026 ceph-mon[68309]: pgmap v2626: 47 pgs: 47 active+clean; 5.1 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-03-20T09:41:24.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:24.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:41:24.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:23 smithi115 ceph-mon[72476]: pgmap v2626: 47 pgs: 47 active+clean; 5.1 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-03-20T09:41:24.161 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:41:24.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:23 smithi122 ceph-mon[73495]: pgmap v2626: 47 pgs: 47 active+clean; 5.1 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-03-20T09:41:24.203 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:24.231 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:41:24.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:24.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:41:24.335 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:41:24.389 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:24.824 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 45 16 90778 90762 126.014 87.4375 0.00759945 0.0079318 2024-03-20T09:41:25.134 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:41:25.134 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:41:25.456 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:41:25.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:25.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:41:25.562 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:41:25.610 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:25.642 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:41:25.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:25.646 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:41:25.701 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:41:25.747 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:41:25.787 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:25.823 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 46 16 92156 92140 125.147 86.125 0.00689946 0.00798651 2024-03-20T09:41:25.823 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:41:25.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:25.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:41:25.933 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:41:25.934 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:25 smithi122 ceph-mon[73495]: pgmap v2627: 47 pgs: 47 active+clean; 5.3 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-03-20T09:41:25.991 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:26.016 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:41:26.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:26.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:41:26.122 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:41:26.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:25 smithi115 ceph-mon[72476]: pgmap v2627: 47 pgs: 47 active+clean; 5.3 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-03-20T09:41:26.164 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:26.186 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:25 smithi026 ceph-mon[68309]: pgmap v2627: 47 pgs: 47 active+clean; 5.3 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-03-20T09:41:26.193 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:41:26.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:26.199 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:41:26.300 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:41:26.344 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:26.374 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:41:26.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:26.378 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:41:26.479 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:41:26.523 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:26.553 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:41:26.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:26.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:41:26.659 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:41:26.702 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:26.724 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-20T09:41:26.823 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 47 16 93508 93492 124.282 84.5 0.00978027 0.0080423 2024-03-20T09:41:26.906 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:41:26.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:26.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:41:27.011 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:41:27.042 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:27.077 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:41:27.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:27.080 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:41:27.181 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:41:27.238 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:27.266 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:41:27.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:27.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:41:27.311 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:41:27.337 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:41:27.369 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:41:27.372 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:41:27.412 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:41:27.424 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:27.451 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:41:27.485 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:41:27.499 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:41:27.511 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:27.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:41:27.526 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:41:27.560 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:41:27.598 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:41:27.613 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:41:27.629 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:41:27.661 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:41:27.665 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:27.703 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:41:27.709 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:41:27.709 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:41:27.709 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:41:27.709 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:41:27.709 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:41:27.709 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:41:27.709 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:41:27.709 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:41:27.710 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:41:27.710 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:41:27.710 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:41:27.710 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:41:27.823 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 48 16 94791 94775 123.364 80.1875 0.00797432 0.00810192 2024-03-20T09:41:27.935 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:41:28.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:27 smithi115 ceph-mon[72476]: pgmap v2628: 47 pgs: 47 active+clean; 5.5 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-03-20T09:41:28.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:27 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3690173362' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:41:28.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:27 smithi122 ceph-mon[73495]: pgmap v2628: 47 pgs: 47 active+clean; 5.5 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-03-20T09:41:28.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:27 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3690173362' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:41:28.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:27 smithi026 ceph-mon[68309]: pgmap v2628: 47 pgs: 47 active+clean; 5.5 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-03-20T09:41:28.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:27 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3690173362' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:41:28.823 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 49 16 96122 96106 122.544 83.1875 0.0232588 0.00815625 2024-03-20T09:41:29.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:28 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:41:29.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:28 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:41:29.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:28 smithi115 ceph-mon[72476]: osdmap e1084: 12 total, 12 up, 11 in 2024-03-20T09:41:29.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:28 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3273251656' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:41:29.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:28 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:41:29.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:28 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:41:29.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:28 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:41:29.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:28 smithi115 ceph-mon[72476]: osdmap e1085: 12 total, 12 up, 11 in 2024-03-20T09:41:29.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:28 smithi115 ceph-mon[72476]: 1.19 scrub starts 2024-03-20T09:41:29.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:28 smithi115 ceph-mon[72476]: 1.19 scrub ok 2024-03-20T09:41:29.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:28 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:41:29.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:28 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:41:29.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:28 smithi122 ceph-mon[73495]: osdmap e1084: 12 total, 12 up, 11 in 2024-03-20T09:41:29.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:28 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3273251656' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:41:29.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:28 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:41:29.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:28 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:41:29.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:28 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:41:29.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:28 smithi122 ceph-mon[73495]: osdmap e1085: 12 total, 12 up, 11 in 2024-03-20T09:41:29.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:28 smithi122 ceph-mon[73495]: 1.19 scrub starts 2024-03-20T09:41:29.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:28 smithi122 ceph-mon[73495]: 1.19 scrub ok 2024-03-20T09:41:29.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:28 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:41:29.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:28 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:41:29.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:28 smithi026 ceph-mon[68309]: osdmap e1084: 12 total, 12 up, 11 in 2024-03-20T09:41:29.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:28 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3273251656' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:41:29.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:28 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:41:29.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:28 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:41:29.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:28 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:41:29.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:28 smithi026 ceph-mon[68309]: osdmap e1085: 12 total, 12 up, 11 in 2024-03-20T09:41:29.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:28 smithi026 ceph-mon[68309]: 1.19 scrub starts 2024-03-20T09:41:29.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:28 smithi026 ceph-mon[68309]: 1.19 scrub ok 2024-03-20T09:41:29.336 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:41:29.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:29.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:41:29.441 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:41:29.492 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:29.527 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:41:29.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:29.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:41:29.632 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:41:29.684 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:29.730 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:41:29.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:29.736 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:41:29.823 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 50 16 97381 97365 121.667 78.6875 0.00982255 0.00821513 2024-03-20T09:41:29.837 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:41:29.887 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:29.919 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:41:29.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:29.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:41:29.934 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:29 smithi122 ceph-mon[73495]: 1.1a scrub starts 2024-03-20T09:41:29.934 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:29 smithi122 ceph-mon[73495]: 1.1a scrub ok 2024-03-20T09:41:29.934 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:29 smithi122 ceph-mon[73495]: pgmap v2631: 47 pgs: 47 active+clean; 5.6 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-03-20T09:41:29.934 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:29 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:41:29.935 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:29 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:41:29.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:29 smithi026 ceph-mon[68309]: 1.1a scrub starts 2024-03-20T09:41:29.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:29 smithi026 ceph-mon[68309]: 1.1a scrub ok 2024-03-20T09:41:29.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:29 smithi026 ceph-mon[68309]: pgmap v2631: 47 pgs: 47 active+clean; 5.6 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-03-20T09:41:29.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:29 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:41:29.959 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:29 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:41:30.024 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:41:30.076 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:30.098 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:41:30.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:30.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:41:30.102 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:29 smithi115 ceph-mon[72476]: 1.1a scrub starts 2024-03-20T09:41:30.102 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:29 smithi115 ceph-mon[72476]: 1.1a scrub ok 2024-03-20T09:41:30.102 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:29 smithi115 ceph-mon[72476]: pgmap v2631: 47 pgs: 47 active+clean; 5.6 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-03-20T09:41:30.102 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:29 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:41:30.102 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:29 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:41:30.202 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:41:30.254 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:30.286 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:41:30.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:30.291 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:41:30.393 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:41:30.453 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:30.483 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:41:30.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:30.486 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:41:30.587 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:41:30.635 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:30.666 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:41:30.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:30.671 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:41:30.772 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:41:30.821 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:30.823 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 51 15 98648 98633 120.834 79.25 0.0110148 0.00827192 2024-03-20T09:41:30.852 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:41:30.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:30.853 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:41:30.954 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:41:30.996 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:31.042 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:41:31.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:31.045 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:41:31.146 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:41:31.177 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:31.383 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:30 smithi026 ceph-mon[68309]: 1.6 scrub starts 2024-03-20T09:41:31.383 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:30 smithi026 ceph-mon[68309]: osdmap e1086: 12 total, 12 up, 11 in 2024-03-20T09:41:31.383 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:30 smithi026 ceph-mon[68309]: 1.6 scrub ok 2024-03-20T09:41:31.383 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:30 smithi026 ceph-mon[68309]: pgmap v2633: 47 pgs: 47 active+clean; 5.9 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 103 MiB/s wr, 1.64k op/s 2024-03-20T09:41:31.392 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:41:31.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:30 smithi115 ceph-mon[72476]: 1.6 scrub starts 2024-03-20T09:41:31.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:30 smithi115 ceph-mon[72476]: osdmap e1086: 12 total, 12 up, 11 in 2024-03-20T09:41:31.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:30 smithi115 ceph-mon[72476]: 1.6 scrub ok 2024-03-20T09:41:31.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:30 smithi115 ceph-mon[72476]: pgmap v2633: 47 pgs: 47 active+clean; 5.9 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 103 MiB/s wr, 1.64k op/s 2024-03-20T09:41:31.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:31.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:41:31.431 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:41:31.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:30 smithi122 ceph-mon[73495]: 1.6 scrub starts 2024-03-20T09:41:31.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:30 smithi122 ceph-mon[73495]: osdmap e1086: 12 total, 12 up, 11 in 2024-03-20T09:41:31.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:30 smithi122 ceph-mon[73495]: 1.6 scrub ok 2024-03-20T09:41:31.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:30 smithi122 ceph-mon[73495]: pgmap v2633: 47 pgs: 47 active+clean; 5.9 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 103 MiB/s wr, 1.64k op/s 2024-03-20T09:41:31.498 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:41:31.532 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:31.569 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:41:31.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:31.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:41:31.679 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:41:31.730 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:31.824 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 52 16 99909 99893 120.025 78.75 0.0127653 0.00832735 2024-03-20T09:41:32.248 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:31 smithi026 ceph-mon[68309]: 1.1e scrub starts 2024-03-20T09:41:32.248 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:31 smithi026 ceph-mon[68309]: 1.1e scrub ok 2024-03-20T09:41:32.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:31 smithi115 ceph-mon[72476]: 1.1e scrub starts 2024-03-20T09:41:32.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:31 smithi115 ceph-mon[72476]: 1.1e scrub ok 2024-03-20T09:41:32.405 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:41:32.419 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:41:32.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:32.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:41:32.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:31 smithi122 ceph-mon[73495]: 1.1e scrub starts 2024-03-20T09:41:32.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:31 smithi122 ceph-mon[73495]: 1.1e scrub ok 2024-03-20T09:41:32.525 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:41:32.582 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:32.609 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:41:32.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:32.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:41:32.715 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:41:32.749 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:41:32.754 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:41:32.756 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:32.786 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:41:32.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:32.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:41:32.823 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 53 16 101144 101128 119.217 77.1875 0.0127127 0.00838421 2024-03-20T09:41:32.894 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:41:32.960 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:32.997 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:41:33.001 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:41:33.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:33.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:41:33.035 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:32 smithi026 ceph-mon[68309]: 1.1b scrub starts 2024-03-20T09:41:33.035 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:32 smithi026 ceph-mon[68309]: 1.16 scrub starts 2024-03-20T09:41:33.035 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:32 smithi026 ceph-mon[68309]: 1.0 scrub starts 2024-03-20T09:41:33.035 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:32 smithi026 ceph-mon[68309]: 1.0 scrub ok 2024-03-20T09:41:33.035 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:32 smithi026 ceph-mon[68309]: 1.2 scrub starts 2024-03-20T09:41:33.035 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:32 smithi026 ceph-mon[68309]: 1.16 scrub ok 2024-03-20T09:41:33.035 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:32 smithi026 ceph-mon[68309]: pgmap v2634: 47 pgs: 47 active+clean; 6.0 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-03-20T09:41:33.035 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:32 smithi026 ceph-mon[68309]: from='client.17409 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:41:33.035 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:32 smithi026 ceph-mon[68309]: osdmap e1087: 12 total, 12 up, 11 in 2024-03-20T09:41:33.103 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:41:33.153 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:33.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:32 smithi115 ceph-mon[72476]: 1.1b scrub starts 2024-03-20T09:41:33.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:32 smithi115 ceph-mon[72476]: 1.16 scrub starts 2024-03-20T09:41:33.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:32 smithi115 ceph-mon[72476]: 1.0 scrub starts 2024-03-20T09:41:33.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:32 smithi115 ceph-mon[72476]: 1.0 scrub ok 2024-03-20T09:41:33.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:32 smithi115 ceph-mon[72476]: 1.2 scrub starts 2024-03-20T09:41:33.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:32 smithi115 ceph-mon[72476]: 1.16 scrub ok 2024-03-20T09:41:33.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:32 smithi115 ceph-mon[72476]: pgmap v2634: 47 pgs: 47 active+clean; 6.0 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-03-20T09:41:33.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:32 smithi115 ceph-mon[72476]: from='client.17409 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:41:33.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:32 smithi115 ceph-mon[72476]: osdmap e1087: 12 total, 12 up, 11 in 2024-03-20T09:41:33.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:32 smithi122 ceph-mon[73495]: 1.1b scrub starts 2024-03-20T09:41:33.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:32 smithi122 ceph-mon[73495]: 1.16 scrub starts 2024-03-20T09:41:33.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:32 smithi122 ceph-mon[73495]: 1.0 scrub starts 2024-03-20T09:41:33.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:32 smithi122 ceph-mon[73495]: 1.0 scrub ok 2024-03-20T09:41:33.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:32 smithi122 ceph-mon[73495]: 1.2 scrub starts 2024-03-20T09:41:33.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:32 smithi122 ceph-mon[73495]: 1.16 scrub ok 2024-03-20T09:41:33.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:32 smithi122 ceph-mon[73495]: pgmap v2634: 47 pgs: 47 active+clean; 6.0 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-03-20T09:41:33.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:32 smithi122 ceph-mon[73495]: from='client.17409 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:41:33.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:32 smithi122 ceph-mon[73495]: osdmap e1087: 12 total, 12 up, 11 in 2024-03-20T09:41:33.824 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 54 16 102346 102330 118.4 75.125 0.00572023 0.00844179 2024-03-20T09:41:34.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:33 smithi115 ceph-mon[72476]: 1.1b scrub ok 2024-03-20T09:41:34.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:33 smithi115 ceph-mon[72476]: 1.2 scrub ok 2024-03-20T09:41:34.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:33 smithi115 ceph-mon[72476]: 1.11 scrub starts 2024-03-20T09:41:34.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:33 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:41:34.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:33 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:41:34.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:33 smithi122 ceph-mon[73495]: 1.1b scrub ok 2024-03-20T09:41:34.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:33 smithi122 ceph-mon[73495]: 1.2 scrub ok 2024-03-20T09:41:34.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:33 smithi122 ceph-mon[73495]: 1.11 scrub starts 2024-03-20T09:41:34.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:33 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:41:34.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:33 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:41:34.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:33 smithi026 ceph-mon[68309]: 1.1b scrub ok 2024-03-20T09:41:34.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:33 smithi026 ceph-mon[68309]: 1.2 scrub ok 2024-03-20T09:41:34.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:33 smithi026 ceph-mon[68309]: 1.11 scrub starts 2024-03-20T09:41:34.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:33 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:41:34.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:33 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:41:34.824 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 55 16 103569 103553 117.638 76.4375 0.0150175 0.00849661 2024-03-20T09:41:34.849 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":2634,"stamp":"2024-03-20T09:41:32.660805+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6395200032,"num_objects":97576,"num_object_clones":0,"num_object_copies":292728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97574,"num_write_kb":6244736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101822,"ondisk_log_size":101822,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":27630980,"kb_used_data":18638920,"kb_used_omap":299,"kb_used_meta":8991636,"kb_avail":1003565692,"statfs":{"total":1055945392128,"available":1027651268608,"internally_reserved":0,"allocated":19086254080,"data_stored":19081935822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":306651,"internal_metadata":9207435813},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,6,6,7],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":463339520,"num_objects":7070,"num_object_clones":0,"num_object_copies":21210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7070,"num_write_kb":452480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001234"},"pg_stats":[{"pgid":"1.1e","version":"20'137","reported_seq":1646,"reported_epoch":1086,"state":"active+clean","last_fresh":"2024-03-20T09:41:29.957826+0000","last_change":"2024-03-20T09:39:55.326984+0000","last_active":"2024-03-20T09:41:29.957826+0000","last_peered":"2024-03-20T09:41:29.957826+0000","last_clean":"2024-03-20T09:41:29.957826+0000","last_became_active":"2024-03-20T09:13:20.071655+0000","last_became_peered":"2024-03-20T09:13:20.071655+0000","last_unstale":"2024-03-20T09:41:29.957826+0000","last_undegraded":"2024-03-20T09:41:29.957826+0000","last_fullsized":"2024-03-20T09:41:29.957826+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:39:55.326860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:39:55.326860+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'137","reported_seq":731,"reported_epoch":1086,"state":"active+clean","last_fresh":"2024-03-20T09:41:29.959459+0000","last_change":"2024-03-20T09:40:37.470797+0000","last_active":"2024-03-20T09:41:29.959459+0000","last_peered":"2024-03-20T09:41:29.959459+0000","last_clean":"2024-03-20T09:41:29.959459+0000","last_became_active":"2024-03-20T09:31:07.425574+0000","last_became_peered":"2024-03-20T09:31:07.425574+0000","last_unstale":"2024-03-20T09:41:29.959459+0000","last_undegraded":"2024-03-20T09:41:29.959459+0000","last_fullsized":"2024-03-20T09:41:29.959459+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:40:37.470664+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:29:02.585165+0000","last_clean_scrub_stamp":"2024-03-20T09:40:37.470664+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:42:01.066261+0000","scrub_duration":150,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2],"acting":[11,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"20'137","reported_seq":2335,"reported_epoch":1086,"state":"active+clean","last_fresh":"2024-03-20T09:41:29.960943+0000","last_change":"2024-03-20T09:40:45.056415+0000","last_active":"2024-03-20T09:41:29.960943+0000","last_peered":"2024-03-20T09:41:29.960943+0000","last_clean":"2024-03-20T09:41:29.960943+0000","last_became_active":"2024-03-20T09:35:03.815532+0000","last_became_peered":"2024-03-20T09:35:03.815532+0000","last_unstale":"2024-03-20T09:41:29.960943+0000","last_undegraded":"2024-03-20T09:41:29.960943+0000","last_fullsized":"2024-03-20T09:41:29.960943+0000","mapping_epoch":950,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":951,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:40:45.056295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:40:45.056295+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:41:47.362074+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.19","version":"20'137","reported_seq":1517,"reported_epoch":1085,"state":"active+clean","last_fresh":"2024-03-20T09:41:28.064807+0000","last_change":"2024-03-20T09:41:28.064846+0000","last_active":"2024-03-20T09:41:28.064807+0000","last_peered":"2024-03-20T09:41:28.064807+0000","last_clean":"2024-03-20T09:41:28.064807+0000","last_became_active":"2024-03-20T09:39:17.953184+0000","last_became_peered":"2024-03-20T09:39:17.953184+0000","last_unstale":"2024-03-20T09:41:28.064807+0000","last_undegraded":"2024-03-20T09:41:28.064807+0000","last_fullsized":"2024-03-20T09:41:28.064807+0000","mapping_epoch":1041,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":1042,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:41:28.064734+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:30:56.201554+0000","last_clean_scrub_stamp":"2024-03-20T09:41:28.064734+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:42:29.244475+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,7],"acting":[10,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"20'137","reported_seq":1800,"reported_epoch":1086,"state":"active+clean","last_fresh":"2024-03-20T09:41:29.958789+0000","last_change":"2024-03-20T09:40:54.937655+0000","last_active":"2024-03-20T09:41:29.958789+0000","last_peered":"2024-03-20T09:41:29.958789+0000","last_clean":"2024-03-20T09:41:29.958789+0000","last_became_active":"2024-03-20T09:10:35.277481+0000","last_became_peered":"2024-03-20T09:10:35.277481+0000","last_unstale":"2024-03-20T09:41:29.958789+0000","last_undegraded":"2024-03-20T09:41:29.958789+0000","last_fullsized":"2024-03-20T09:41:29.958789+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:40:54.937537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:40:54.937537+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:42:15.250272+0000","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1b","version":"20'137","reported_seq":2164,"reported_epoch":1086,"state":"active+clean","last_fresh":"2024-03-20T09:41:29.960335+0000","last_change":"2024-03-20T09:39:47.095766+0000","last_active":"2024-03-20T09:41:29.960335+0000","last_peered":"2024-03-20T09:41:29.960335+0000","last_clean":"2024-03-20T09:41:29.960335+0000","last_became_active":"2024-03-20T09:39:17.951408+0000","last_became_peered":"2024-03-20T09:39:17.951408+0000","last_unstale":"2024-03-20T09:41:29.960335+0000","last_undegraded":"2024-03-20T09:41:29.960335+0000","last_fullsized":"2024-03-20T09:41:29.960335+0000","mapping_epoch":1041,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":1042,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:39:47.095686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:39:47.095686+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":191,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,2],"acting":[8,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.0","version":"1086'6391","reported_seq":6441,"reported_epoch":1086,"state":"active+clean","last_fresh":"2024-03-20T09:41:31.392922+0000","last_change":"2024-03-20T09:40:24.790300+0000","last_active":"2024-03-20T09:41:31.392922+0000","last_peered":"2024-03-20T09:41:31.392922+0000","last_clean":"2024-03-20T09:41:31.392922+0000","last_became_active":"2024-03-20T09:40:24.790001+0000","last_became_peered":"2024-03-20T09:40:24.790001+0000","last_unstale":"2024-03-20T09:41:31.392922+0000","last_undegraded":"2024-03-20T09:41:31.392922+0000","last_fullsized":"2024-03-20T09:41:31.392922+0000","mapping_epoch":1066,"log_start":"0'0","ondisk_log_start":"0'0","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:40:23.772372+0000","objects_scrubbed":0,"log_size":6391,"log_dups_size":0,"ondisk_log_size":6391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:41:32.720772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":418840576,"num_objects":6391,"num_object_clones":0,"num_object_copies":19173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6391,"num_write_kb":409024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,11],"acting":[6,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"20'137","reported_seq":924,"reported_epoch":1085,"state":"active+clean","last_fresh":"2024-03-20T09:41:28.050912+0000","last_change":"2024-03-20T09:39:49.837224+0000","last_active":"2024-03-20T09:41:28.050912+0000","last_peered":"2024-03-20T09:41:28.050912+0000","last_clean":"2024-03-20T09:41:28.050912+0000","last_became_active":"2024-03-20T09:35:28.000367+0000","last_became_peered":"2024-03-20T09:35:28.000367+0000","last_unstale":"2024-03-20T09:41:28.050912+0000","last_undegraded":"2024-03-20T09:41:28.050912+0000","last_fullsized":"2024-03-20T09:41:28.050912+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":966,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:39:49.837113+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:37:01.184206+0000","last_clean_scrub_stamp":"2024-03-20T09:39:49.837113+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:41:30.786993+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"1086'6033","reported_seq":6083,"reported_epoch":1086,"state":"active+clean","last_fresh":"2024-03-20T09:41:29.962112+0000","last_change":"2024-03-20T09:40:24.791266+0000","last_active":"2024-03-20T09:41:29.962112+0000","last_peered":"2024-03-20T09:41:29.962112+0000","last_clean":"2024-03-20T09:41:29.962112+0000","last_became_active":"2024-03-20T09:40:24.791005+0000","last_became_peered":"2024-03-20T09:40:24.791005+0000","last_unstale":"2024-03-20T09:41:29.962112+0000","last_undegraded":"2024-03-20T09:41:29.962112+0000","last_fullsized":"2024-03-20T09:41:29.962112+0000","mapping_epoch":1066,"log_start":"0'0","ondisk_log_start":"0'0","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:40:23.772372+0000","objects_scrubbed":0,"log_size":6033,"log_dups_size":0,"ondisk_log_size":6033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:41:45.575106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395378688,"num_objects":6033,"num_object_clones":0,"num_object_copies":18099,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6033,"num_write_kb":386112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"20'137","reported_seq":2383,"reported_epoch":1085,"state":"active+clean","last_fresh":"2024-03-20T09:41:27.947097+0000","last_change":"2024-03-20T09:40:07.387831+0000","last_active":"2024-03-20T09:41:27.947097+0000","last_peered":"2024-03-20T09:41:27.947097+0000","last_clean":"2024-03-20T09:41:27.947097+0000","last_became_active":"2024-03-20T09:20:25.341750+0000","last_became_peered":"2024-03-20T09:20:25.341750+0000","last_unstale":"2024-03-20T09:41:27.947097+0000","last_undegraded":"2024-03-20T09:41:27.947097+0000","last_fullsized":"2024-03-20T09:41:27.947097+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:40:07.387730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:40:07.387730+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:41:36.133062+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,2],"acting":[9,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.2","version":"1086'6156","reported_seq":6206,"reported_epoch":1086,"state":"active+clean","last_fresh":"2024-03-20T09:41:29.972331+0000","last_change":"2024-03-20T09:40:24.790582+0000","last_active":"2024-03-20T09:41:29.972331+0000","last_peered":"2024-03-20T09:41:29.972331+0000","last_clean":"2024-03-20T09:41:29.972331+0000","last_became_active":"2024-03-20T09:40:24.790265+0000","last_became_peered":"2024-03-20T09:40:24.790265+0000","last_unstale":"2024-03-20T09:41:29.972331+0000","last_undegraded":"2024-03-20T09:41:29.972331+0000","last_fullsized":"2024-03-20T09:41:29.972331+0000","mapping_epoch":1066,"log_start":"0'0","ondisk_log_start":"0'0","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:40:23.772372+0000","objects_scrubbed":0,"log_size":6156,"log_dups_size":0,"ondisk_log_size":6156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:41:40.467574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403439616,"num_objects":6156,"num_object_clones":0,"num_object_copies":18468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6156,"num_write_kb":393984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,9],"acting":[3,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"20'137","reported_seq":1064,"reported_epoch":1085,"state":"active+clean","last_fresh":"2024-03-20T09:41:27.947721+0000","last_change":"2024-03-20T09:39:58.145662+0000","last_active":"2024-03-20T09:41:27.947721+0000","last_peered":"2024-03-20T09:41:27.947721+0000","last_clean":"2024-03-20T09:41:27.947721+0000","last_became_active":"2024-03-20T09:24:16.444232+0000","last_became_peered":"2024-03-20T09:24:16.444232+0000","last_unstale":"2024-03-20T09:41:27.947721+0000","last_undegraded":"2024-03-20T09:41:27.947721+0000","last_fullsized":"2024-03-20T09:41:27.947721+0000","mapping_epoch":718,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:39:58.145548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:39:58.145548+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:41:29.413725+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.c","version":"1086'5974","reported_seq":6024,"reported_epoch":1086,"state":"active+clean","last_fresh":"2024-03-20T09:41:29.968980+0000","last_change":"2024-03-20T09:40:24.792008+0000","last_active":"2024-03-20T09:41:29.968980+0000","last_peered":"2024-03-20T09:41:29.968980+0000","last_clean":"2024-03-20T09:41:29.968980+0000","last_became_active":"2024-03-20T09:40:24.791734+0000","last_became_peered":"2024-03-20T09:40:24.791734+0000","last_unstale":"2024-03-20T09:41:29.968980+0000","last_undegraded":"2024-03-20T09:41:29.968980+0000","last_fullsized":"2024-03-20T09:41:29.968980+0000","mapping_epoch":1066,"log_start":"0'0","ondisk_log_start":"0'0","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:40:23.772372+0000","objects_scrubbed":0,"log_size":5974,"log_dups_size":0,"ondisk_log_size":5974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:41:36.515351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391512064,"num_objects":5974,"num_object_clones":0,"num_object_copies":17922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5974,"num_write_kb":382336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,9],"acting":[3,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"20'137","reported_seq":2313,"reported_epoch":1086,"state":"active+clean","last_fresh":"2024-03-20T09:41:29.957652+0000","last_change":"2024-03-20T09:40:32.325735+0000","last_active":"2024-03-20T09:41:29.957652+0000","last_peered":"2024-03-20T09:41:29.957652+0000","last_clean":"2024-03-20T09:41:29.957652+0000","last_became_active":"2024-03-20T09:12:14.306384+0000","last_became_peered":"2024-03-20T09:12:14.306384+0000","last_unstale":"2024-03-20T09:41:29.957652+0000","last_undegraded":"2024-03-20T09:41:29.957652+0000","last_fullsized":"2024-03-20T09:41:29.957652+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:40:32.325633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:40:32.325633+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:41:34.592829+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.d","version":"1086'5982","reported_seq":6032,"reported_epoch":1086,"state":"active+clean","last_fresh":"2024-03-20T09:41:30.137915+0000","last_change":"2024-03-20T09:40:24.813191+0000","last_active":"2024-03-20T09:41:30.137915+0000","last_peered":"2024-03-20T09:41:30.137915+0000","last_clean":"2024-03-20T09:41:30.137915+0000","last_became_active":"2024-03-20T09:40:24.812843+0000","last_became_peered":"2024-03-20T09:40:24.812843+0000","last_unstale":"2024-03-20T09:41:30.137915+0000","last_undegraded":"2024-03-20T09:41:30.137915+0000","last_fullsized":"2024-03-20T09:41:30.137915+0000","mapping_epoch":1066,"log_start":"0'0","ondisk_log_start":"0'0","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:40:23.772372+0000","objects_scrubbed":0,"log_size":5982,"log_dups_size":0,"ondisk_log_size":5982,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:41:52.664646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":392036352,"num_objects":5982,"num_object_clones":0,"num_object_copies":17946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5982,"num_write_kb":382848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"20'137","reported_seq":972,"reported_epoch":1085,"state":"active+clean","last_fresh":"2024-03-20T09:41:27.947107+0000","last_change":"2024-03-20T09:39:57.807397+0000","last_active":"2024-03-20T09:41:27.947107+0000","last_peered":"2024-03-20T09:41:27.947107+0000","last_clean":"2024-03-20T09:41:27.947107+0000","last_became_active":"2024-03-20T09:35:19.820895+0000","last_became_peered":"2024-03-20T09:35:19.820895+0000","last_unstale":"2024-03-20T09:41:27.947107+0000","last_undegraded":"2024-03-20T09:41:27.947107+0000","last_fullsized":"2024-03-20T09:41:27.947107+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:39:57.807320+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:19:03.805185+0000","last_clean_scrub_stamp":"2024-03-20T09:39:57.807320+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"1085'6038","reported_seq":6086,"reported_epoch":1085,"state":"active+clean","last_fresh":"2024-03-20T09:41:28.963575+0000","last_change":"2024-03-20T09:40:24.813515+0000","last_active":"2024-03-20T09:41:28.963575+0000","last_peered":"2024-03-20T09:41:28.963575+0000","last_clean":"2024-03-20T09:41:28.963575+0000","last_became_active":"2024-03-20T09:40:24.813215+0000","last_became_peered":"2024-03-20T09:40:24.813215+0000","last_unstale":"2024-03-20T09:41:28.963575+0000","last_undegraded":"2024-03-20T09:41:28.963575+0000","last_fullsized":"2024-03-20T09:41:28.963575+0000","mapping_epoch":1066,"log_start":"0'0","ondisk_log_start":"0'0","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:40:23.772372+0000","objects_scrubbed":0,"log_size":6038,"log_dups_size":0,"ondisk_log_size":6038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:41:38.796856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395706368,"num_objects":6038,"num_object_clones":0,"num_object_copies":18114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6038,"num_write_kb":386432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,1],"acting":[10,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"20'137","reported_seq":1842,"reported_epoch":1085,"state":"active+clean","last_fresh":"2024-03-20T09:41:27.947291+0000","last_change":"2024-03-20T09:39:46.970855+0000","last_active":"2024-03-20T09:41:27.947291+0000","last_peered":"2024-03-20T09:41:27.947291+0000","last_clean":"2024-03-20T09:41:27.947291+0000","last_became_active":"2024-03-20T09:35:19.822321+0000","last_became_peered":"2024-03-20T09:35:19.822321+0000","last_unstale":"2024-03-20T09:41:27.947291+0000","last_undegraded":"2024-03-20T09:41:27.947291+0000","last_fullsized":"2024-03-20T09:41:27.947291+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:39:46.970731+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:24:31.549603+0000","last_clean_scrub_stamp":"2024-03-20T09:39:46.970731+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.b","version":"1086'6171","reported_seq":6221,"reported_epoch":1086,"state":"active+clean","last_fresh":"2024-03-20T09:41:31.848893+0000","last_change":"2024-03-20T09:40:24.791414+0000","last_active":"2024-03-20T09:41:31.848893+0000","last_peered":"2024-03-20T09:41:31.848893+0000","last_clean":"2024-03-20T09:41:31.848893+0000","last_became_active":"2024-03-20T09:40:24.791110+0000","last_became_peered":"2024-03-20T09:40:24.791110+0000","last_unstale":"2024-03-20T09:41:31.848893+0000","last_undegraded":"2024-03-20T09:41:31.848893+0000","last_fullsized":"2024-03-20T09:41:31.848893+0000","mapping_epoch":1066,"log_start":"0'0","ondisk_log_start":"0'0","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:40:23.772372+0000","objects_scrubbed":0,"log_size":6171,"log_dups_size":0,"ondisk_log_size":6171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:41:47.354429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404422656,"num_objects":6171,"num_object_clones":0,"num_object_copies":18513,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6171,"num_write_kb":394944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"20'137","reported_seq":1379,"reported_epoch":1086,"state":"active+clean","last_fresh":"2024-03-20T09:41:29.971076+0000","last_change":"2024-03-20T09:40:00.596100+0000","last_active":"2024-03-20T09:41:29.971076+0000","last_peered":"2024-03-20T09:41:29.971076+0000","last_clean":"2024-03-20T09:41:29.971076+0000","last_became_active":"2024-03-20T09:31:07.424588+0000","last_became_peered":"2024-03-20T09:31:07.424588+0000","last_unstale":"2024-03-20T09:41:29.971076+0000","last_undegraded":"2024-03-20T09:41:29.971076+0000","last_fullsized":"2024-03-20T09:41:29.971076+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:40:00.595977+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:30:47.812125+0000","last_clean_scrub_stamp":"2024-03-20T09:40:00.595977+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:41:32.720737+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"1085'6022","reported_seq":6070,"reported_epoch":1085,"state":"active+clean","last_fresh":"2024-03-20T09:41:28.615871+0000","last_change":"2024-03-20T09:40:24.790494+0000","last_active":"2024-03-20T09:41:28.615871+0000","last_peered":"2024-03-20T09:41:28.615871+0000","last_clean":"2024-03-20T09:41:28.615871+0000","last_became_active":"2024-03-20T09:40:24.790197+0000","last_became_peered":"2024-03-20T09:40:24.790197+0000","last_unstale":"2024-03-20T09:41:28.615871+0000","last_undegraded":"2024-03-20T09:41:28.615871+0000","last_fullsized":"2024-03-20T09:41:28.615871+0000","mapping_epoch":1066,"log_start":"0'0","ondisk_log_start":"0'0","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:40:23.772372+0000","objects_scrubbed":0,"log_size":6022,"log_dups_size":0,"ondisk_log_size":6022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:41:51.357317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394657792,"num_objects":6022,"num_object_clones":0,"num_object_copies":18066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6022,"num_write_kb":385408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":2285,"reported_epoch":1086,"state":"active+clean","last_fresh":"2024-03-20T09:41:29.957571+0000","last_change":"2024-03-20T09:39:46.349239+0000","last_active":"2024-03-20T09:41:29.957571+0000","last_peered":"2024-03-20T09:41:29.957571+0000","last_clean":"2024-03-20T09:41:29.957571+0000","last_became_active":"2024-03-20T09:12:14.305989+0000","last_became_peered":"2024-03-20T09:12:14.305989+0000","last_unstale":"2024-03-20T09:41:29.957571+0000","last_undegraded":"2024-03-20T09:41:29.957571+0000","last_fullsized":"2024-03-20T09:41:29.957571+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:39:46.349109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:39:46.349109+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.a","version":"1086'6130","reported_seq":6180,"reported_epoch":1086,"state":"active+clean","last_fresh":"2024-03-20T09:41:30.128588+0000","last_change":"2024-03-20T09:40:24.813200+0000","last_active":"2024-03-20T09:41:30.128588+0000","last_peered":"2024-03-20T09:41:30.128588+0000","last_clean":"2024-03-20T09:41:30.128588+0000","last_became_active":"2024-03-20T09:40:24.812882+0000","last_became_peered":"2024-03-20T09:40:24.812882+0000","last_unstale":"2024-03-20T09:41:30.128588+0000","last_undegraded":"2024-03-20T09:41:30.128588+0000","last_fullsized":"2024-03-20T09:41:30.128588+0000","mapping_epoch":1066,"log_start":"0'0","ondisk_log_start":"0'0","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:40:23.772372+0000","objects_scrubbed":0,"log_size":6130,"log_dups_size":0,"ondisk_log_size":6130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:41:35.311757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401735680,"num_objects":6130,"num_object_clones":0,"num_object_copies":18390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6130,"num_write_kb":392320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"20'137","reported_seq":1679,"reported_epoch":1086,"state":"active+clean","last_fresh":"2024-03-20T09:41:29.965391+0000","last_change":"2024-03-20T09:39:49.175886+0000","last_active":"2024-03-20T09:41:29.965391+0000","last_peered":"2024-03-20T09:41:29.965391+0000","last_clean":"2024-03-20T09:41:29.965391+0000","last_became_active":"2024-03-20T09:39:17.944040+0000","last_became_peered":"2024-03-20T09:39:17.944040+0000","last_unstale":"2024-03-20T09:41:29.965391+0000","last_undegraded":"2024-03-20T09:41:29.965391+0000","last_fullsized":"2024-03-20T09:41:29.965391+0000","mapping_epoch":1041,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":1042,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:39:49.175806+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:10:19.513511+0000","last_clean_scrub_stamp":"2024-03-20T09:39:49.175806+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,8],"acting":[4,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.9","version":"1085'6019","reported_seq":6067,"reported_epoch":1085,"state":"active+clean","last_fresh":"2024-03-20T09:41:28.973508+0000","last_change":"2024-03-20T09:40:24.792998+0000","last_active":"2024-03-20T09:41:28.973508+0000","last_peered":"2024-03-20T09:41:28.973508+0000","last_clean":"2024-03-20T09:41:28.973508+0000","last_became_active":"2024-03-20T09:40:24.792650+0000","last_became_peered":"2024-03-20T09:40:24.792650+0000","last_unstale":"2024-03-20T09:41:28.973508+0000","last_undegraded":"2024-03-20T09:41:28.973508+0000","last_fullsized":"2024-03-20T09:41:28.973508+0000","mapping_epoch":1066,"log_start":"0'0","ondisk_log_start":"0'0","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:40:23.772372+0000","objects_scrubbed":0,"log_size":6019,"log_dups_size":0,"ondisk_log_size":6019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:41:45.822575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394461184,"num_objects":6019,"num_object_clones":0,"num_object_copies":18057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6019,"num_write_kb":385216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"20'137","reported_seq":1726,"reported_epoch":1086,"state":"active+clean","last_fresh":"2024-03-20T09:41:29.973619+0000","last_change":"2024-03-20T09:39:29.290472+0000","last_active":"2024-03-20T09:41:29.973619+0000","last_peered":"2024-03-20T09:41:29.973619+0000","last_clean":"2024-03-20T09:41:29.973619+0000","last_became_active":"2024-03-20T09:15:34.225022+0000","last_became_peered":"2024-03-20T09:15:34.225022+0000","last_unstale":"2024-03-20T09:41:29.973619+0000","last_undegraded":"2024-03-20T09:41:29.973619+0000","last_fullsized":"2024-03-20T09:41:29.973619+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:39:29.290359+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:20:12.169658+0000","last_clean_scrub_stamp":"2024-03-20T09:39:29.290359+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:41:32.720700+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,5],"acting":[6,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"1086'6158","reported_seq":6208,"reported_epoch":1086,"state":"active+clean","last_fresh":"2024-03-20T09:41:30.121975+0000","last_change":"2024-03-20T09:40:24.790523+0000","last_active":"2024-03-20T09:41:30.121975+0000","last_peered":"2024-03-20T09:41:30.121975+0000","last_clean":"2024-03-20T09:41:30.121975+0000","last_became_active":"2024-03-20T09:40:24.790189+0000","last_became_peered":"2024-03-20T09:40:24.790189+0000","last_unstale":"2024-03-20T09:41:30.121975+0000","last_undegraded":"2024-03-20T09:41:30.121975+0000","last_fullsized":"2024-03-20T09:41:30.121975+0000","mapping_epoch":1066,"log_start":"0'0","ondisk_log_start":"0'0","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:40:23.772372+0000","objects_scrubbed":0,"log_size":6158,"log_dups_size":0,"ondisk_log_size":6158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:41:49.502727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403570688,"num_objects":6158,"num_object_clones":0,"num_object_copies":18474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6158,"num_write_kb":394112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"20'137","reported_seq":2282,"reported_epoch":1086,"state":"active+clean","last_fresh":"2024-03-20T09:41:29.971344+0000","last_change":"2024-03-20T09:40:47.804234+0000","last_active":"2024-03-20T09:41:29.971344+0000","last_peered":"2024-03-20T09:41:29.971344+0000","last_clean":"2024-03-20T09:41:29.971344+0000","last_became_active":"2024-03-20T09:35:19.820396+0000","last_became_peered":"2024-03-20T09:35:19.820396+0000","last_unstale":"2024-03-20T09:41:29.971344+0000","last_undegraded":"2024-03-20T09:41:29.971344+0000","last_fullsized":"2024-03-20T09:41:29.971344+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:40:47.804101+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:14:10.560890+0000","last_clean_scrub_stamp":"2024-03-20T09:40:47.804101+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:41:50.189085+0000","scrub_duration":240,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"1085'6034","reported_seq":6082,"reported_epoch":1085,"state":"active+clean","last_fresh":"2024-03-20T09:41:28.615891+0000","last_change":"2024-03-20T09:40:24.791510+0000","last_active":"2024-03-20T09:41:28.615891+0000","last_peered":"2024-03-20T09:41:28.615891+0000","last_clean":"2024-03-20T09:41:28.615891+0000","last_became_active":"2024-03-20T09:40:24.791199+0000","last_became_peered":"2024-03-20T09:40:24.791199+0000","last_unstale":"2024-03-20T09:41:28.615891+0000","last_undegraded":"2024-03-20T09:41:28.615891+0000","last_fullsized":"2024-03-20T09:41:28.615891+0000","mapping_epoch":1066,"log_start":"0'0","ondisk_log_start":"0'0","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:40:23.772372+0000","objects_scrubbed":0,"log_size":6034,"log_dups_size":0,"ondisk_log_size":6034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:41:29.090824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395444224,"num_objects":6034,"num_object_clones":0,"num_object_copies":18102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6034,"num_write_kb":386176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,5],"acting":[9,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"20'137","reported_seq":2114,"reported_epoch":1086,"state":"active+clean","last_fresh":"2024-03-20T09:41:29.957415+0000","last_change":"2024-03-20T09:40:57.479400+0000","last_active":"2024-03-20T09:41:29.957415+0000","last_peered":"2024-03-20T09:41:29.957415+0000","last_clean":"2024-03-20T09:41:29.957415+0000","last_became_active":"2024-03-20T09:39:17.940497+0000","last_became_peered":"2024-03-20T09:39:17.940497+0000","last_unstale":"2024-03-20T09:41:29.957415+0000","last_undegraded":"2024-03-20T09:41:29.957415+0000","last_fullsized":"2024-03-20T09:41:29.957415+0000","mapping_epoch":1041,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":1042,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:40:57.479269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:40:57.479269+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:42:09.759764+0000","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,2],"acting":[11,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.6","version":"1086'6086","reported_seq":6136,"reported_epoch":1086,"state":"active+clean","last_fresh":"2024-03-20T09:41:30.137771+0000","last_change":"2024-03-20T09:40:24.792593+0000","last_active":"2024-03-20T09:41:30.137771+0000","last_peered":"2024-03-20T09:41:30.137771+0000","last_clean":"2024-03-20T09:41:30.137771+0000","last_became_active":"2024-03-20T09:40:24.792369+0000","last_became_peered":"2024-03-20T09:40:24.792369+0000","last_unstale":"2024-03-20T09:41:30.137771+0000","last_undegraded":"2024-03-20T09:41:30.137771+0000","last_fullsized":"2024-03-20T09:41:30.137771+0000","mapping_epoch":1066,"log_start":"0'0","ondisk_log_start":"0'0","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:40:23.772372+0000","objects_scrubbed":0,"log_size":6086,"log_dups_size":0,"ondisk_log_size":6086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:41:40.794840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398852096,"num_objects":6086,"num_object_clones":0,"num_object_copies":18258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6086,"num_write_kb":389504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"20'137","reported_seq":1538,"reported_epoch":1086,"state":"active+clean","last_fresh":"2024-03-20T09:41:29.965465+0000","last_change":"2024-03-20T09:40:07.235021+0000","last_active":"2024-03-20T09:41:29.965465+0000","last_peered":"2024-03-20T09:41:29.965465+0000","last_clean":"2024-03-20T09:41:29.965465+0000","last_became_active":"2024-03-20T09:19:54.506134+0000","last_became_peered":"2024-03-20T09:19:54.506134+0000","last_unstale":"2024-03-20T09:41:29.965465+0000","last_undegraded":"2024-03-20T09:41:29.965465+0000","last_fullsized":"2024-03-20T09:41:29.965465+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:40:07.234907+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:29:53.362224+0000","last_clean_scrub_stamp":"2024-03-20T09:40:07.234907+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"1085'6119","reported_seq":6167,"reported_epoch":1085,"state":"active+clean","last_fresh":"2024-03-20T09:41:28.979426+0000","last_change":"2024-03-20T09:40:24.813526+0000","last_active":"2024-03-20T09:41:28.979426+0000","last_peered":"2024-03-20T09:41:28.979426+0000","last_clean":"2024-03-20T09:41:28.979426+0000","last_became_active":"2024-03-20T09:40:24.813246+0000","last_became_peered":"2024-03-20T09:40:24.813246+0000","last_unstale":"2024-03-20T09:41:28.979426+0000","last_undegraded":"2024-03-20T09:41:28.979426+0000","last_fullsized":"2024-03-20T09:41:28.979426+0000","mapping_epoch":1066,"log_start":"0'0","ondisk_log_start":"0'0","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:40:23.772372+0000","objects_scrubbed":0,"log_size":6119,"log_dups_size":0,"ondisk_log_size":6119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401014784,"num_objects":6119,"num_object_clones":0,"num_object_copies":18357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6119,"num_write_kb":391616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,3],"acting":[10,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"20'137","reported_seq":2301,"reported_epoch":1086,"state":"active+clean","last_fresh":"2024-03-20T09:41:29.971236+0000","last_change":"2024-03-20T09:40:52.893763+0000","last_active":"2024-03-20T09:41:29.971236+0000","last_peered":"2024-03-20T09:41:29.971236+0000","last_clean":"2024-03-20T09:41:29.971236+0000","last_became_active":"2024-03-20T09:19:54.524397+0000","last_became_peered":"2024-03-20T09:19:54.524397+0000","last_unstale":"2024-03-20T09:41:29.971236+0000","last_undegraded":"2024-03-20T09:41:29.971236+0000","last_fullsized":"2024-03-20T09:41:29.971236+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:40:52.893621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:40:52.893621+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:42:08.831651+0000","scrub_duration":253,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"1086'6026","reported_seq":6076,"reported_epoch":1086,"state":"active+clean","last_fresh":"2024-03-20T09:41:30.687103+0000","last_change":"2024-03-20T09:40:24.789839+0000","last_active":"2024-03-20T09:41:30.687103+0000","last_peered":"2024-03-20T09:41:30.687103+0000","last_clean":"2024-03-20T09:41:30.687103+0000","last_became_active":"2024-03-20T09:40:24.789539+0000","last_became_peered":"2024-03-20T09:40:24.789539+0000","last_unstale":"2024-03-20T09:41:30.687103+0000","last_undegraded":"2024-03-20T09:41:30.687103+0000","last_fullsized":"2024-03-20T09:41:30.687103+0000","mapping_epoch":1066,"log_start":"0'0","ondisk_log_start":"0'0","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:40:23.772372+0000","objects_scrubbed":0,"log_size":6026,"log_dups_size":0,"ondisk_log_size":6026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:41:37.430359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394919936,"num_objects":6026,"num_object_clones":0,"num_object_copies":18078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6026,"num_write_kb":385664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.c","version":"570'138","reported_seq":2293,"reported_epoch":1086,"state":"active+clean","last_fresh":"2024-03-20T09:41:29.962178+0000","last_change":"2024-03-20T09:40:28.971174+0000","last_active":"2024-03-20T09:41:29.962178+0000","last_peered":"2024-03-20T09:41:29.962178+0000","last_clean":"2024-03-20T09:41:29.962178+0000","last_became_active":"2024-03-20T09:39:17.943381+0000","last_became_peered":"2024-03-20T09:39:17.943381+0000","last_unstale":"2024-03-20T09:41:29.962178+0000","last_undegraded":"2024-03-20T09:41:29.962178+0000","last_fullsized":"2024-03-20T09:41:29.962178+0000","mapping_epoch":1041,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":1042,"parent":"0.0","parent_split_bits":4,"last_scrub":"570'138","last_scrub_stamp":"2024-03-20T09:40:28.971056+0000","last_deep_scrub":"570'138","last_deep_scrub_stamp":"2024-03-20T09:16:13.725035+0000","last_clean_scrub_stamp":"2024-03-20T09:40:28.971056+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:41:44.725062+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,9],"acting":[3,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"1086'6235","reported_seq":6284,"reported_epoch":1086,"state":"active+clean","last_fresh":"2024-03-20T09:41:30.856808+0000","last_change":"2024-03-20T09:40:24.813656+0000","last_active":"2024-03-20T09:41:30.856808+0000","last_peered":"2024-03-20T09:41:30.856808+0000","last_clean":"2024-03-20T09:41:30.856808+0000","last_became_active":"2024-03-20T09:40:24.813276+0000","last_became_peered":"2024-03-20T09:40:24.813276+0000","last_unstale":"2024-03-20T09:41:30.856808+0000","last_undegraded":"2024-03-20T09:41:30.856808+0000","last_fullsized":"2024-03-20T09:41:30.856808+0000","mapping_epoch":1066,"log_start":"0'0","ondisk_log_start":"0'0","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:40:23.772372+0000","objects_scrubbed":0,"log_size":6235,"log_dups_size":0,"ondisk_log_size":6235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:41:52.209286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408616960,"num_objects":6235,"num_object_clones":0,"num_object_copies":18705,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6235,"num_write_kb":399040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"20'137","reported_seq":1852,"reported_epoch":1086,"state":"active+clean","last_fresh":"2024-03-20T09:41:29.965451+0000","last_change":"2024-03-20T09:40:49.244662+0000","last_active":"2024-03-20T09:41:29.965451+0000","last_peered":"2024-03-20T09:41:29.965451+0000","last_clean":"2024-03-20T09:41:29.965451+0000","last_became_active":"2024-03-20T09:19:54.502529+0000","last_became_peered":"2024-03-20T09:19:54.502529+0000","last_unstale":"2024-03-20T09:41:29.965451+0000","last_undegraded":"2024-03-20T09:41:29.965451+0000","last_fullsized":"2024-03-20T09:41:29.965451+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:40:49.244455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:40:49.244455+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:41:51.112042+0000","scrub_duration":165,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"20'137","reported_seq":2133,"reported_epoch":1086,"state":"active+clean","last_fresh":"2024-03-20T09:41:29.959007+0000","last_change":"2024-03-20T09:39:55.850462+0000","last_active":"2024-03-20T09:41:29.959007+0000","last_peered":"2024-03-20T09:41:29.959007+0000","last_clean":"2024-03-20T09:41:29.959007+0000","last_became_active":"2024-03-20T09:10:34.096758+0000","last_became_peered":"2024-03-20T09:10:34.096758+0000","last_unstale":"2024-03-20T09:41:29.959007+0000","last_undegraded":"2024-03-20T09:41:29.959007+0000","last_fullsized":"2024-03-20T09:41:29.959007+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:39:55.850369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:39:55.850369+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"20'137","reported_seq":893,"reported_epoch":1086,"state":"active+clean","last_fresh":"2024-03-20T09:41:29.957535+0000","last_change":"2024-03-20T09:40:34.510349+0000","last_active":"2024-03-20T09:41:29.957535+0000","last_peered":"2024-03-20T09:41:29.957535+0000","last_clean":"2024-03-20T09:41:29.957535+0000","last_became_active":"2024-03-20T09:35:19.822242+0000","last_became_peered":"2024-03-20T09:35:19.822242+0000","last_unstale":"2024-03-20T09:41:29.957535+0000","last_undegraded":"2024-03-20T09:41:29.957535+0000","last_fullsized":"2024-03-20T09:41:29.957535+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:40:34.510237+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:37:52.359932+0000","last_clean_scrub_stamp":"2024-03-20T09:40:34.510237+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:41:37.754593+0000","scrub_duration":209,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.10","version":"20'137","reported_seq":1696,"reported_epoch":1086,"state":"active+clean","last_fresh":"2024-03-20T09:41:29.958938+0000","last_change":"2024-03-20T09:39:56.799960+0000","last_active":"2024-03-20T09:41:29.958938+0000","last_peered":"2024-03-20T09:41:29.958938+0000","last_clean":"2024-03-20T09:41:29.958938+0000","last_became_active":"2024-03-20T09:35:19.820595+0000","last_became_peered":"2024-03-20T09:35:19.820595+0000","last_unstale":"2024-03-20T09:41:29.958938+0000","last_undegraded":"2024-03-20T09:41:29.958938+0000","last_fullsized":"2024-03-20T09:41:29.958938+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:39:56.799868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:39:56.799868+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"20'137","reported_seq":490,"reported_epoch":1086,"state":"active+clean","last_fresh":"2024-03-20T09:41:29.960128+0000","last_change":"2024-03-20T09:39:57.039180+0000","last_active":"2024-03-20T09:41:29.960128+0000","last_peered":"2024-03-20T09:41:29.960128+0000","last_clean":"2024-03-20T09:41:29.960128+0000","last_became_active":"2024-03-20T09:39:17.943385+0000","last_became_peered":"2024-03-20T09:39:17.943385+0000","last_unstale":"2024-03-20T09:41:29.960128+0000","last_undegraded":"2024-03-20T09:41:29.960128+0000","last_fullsized":"2024-03-20T09:41:29.960128+0000","mapping_epoch":1041,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":1042,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:39:57.039080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:39:57.039080+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":150,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"20'137","reported_seq":768,"reported_epoch":1086,"state":"active+clean","last_fresh":"2024-03-20T09:41:29.962273+0000","last_change":"2024-03-20T09:40:01.034430+0000","last_active":"2024-03-20T09:41:29.962273+0000","last_peered":"2024-03-20T09:41:29.962273+0000","last_clean":"2024-03-20T09:41:29.962273+0000","last_became_active":"2024-03-20T09:39:17.943337+0000","last_became_peered":"2024-03-20T09:39:17.943337+0000","last_unstale":"2024-03-20T09:41:29.962273+0000","last_undegraded":"2024-03-20T09:41:29.962273+0000","last_fullsized":"2024-03-20T09:41:29.962273+0000","mapping_epoch":1041,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":1042,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:40:01.034318+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:35:27.264203+0000","last_clean_scrub_stamp":"2024-03-20T09:40:01.034318+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:41:29.997473+0000","scrub_duration":164,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,11],"acting":[3,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"20'137","reported_seq":2274,"reported_epoch":1086,"state":"active+clean","last_fresh":"2024-03-20T09:41:29.960498+0000","last_change":"2024-03-20T09:40:32.177319+0000","last_active":"2024-03-20T09:41:29.960498+0000","last_peered":"2024-03-20T09:41:29.960498+0000","last_clean":"2024-03-20T09:41:29.960498+0000","last_became_active":"2024-03-20T09:35:19.822429+0000","last_became_peered":"2024-03-20T09:35:19.822429+0000","last_unstale":"2024-03-20T09:41:29.960498+0000","last_undegraded":"2024-03-20T09:41:29.960498+0000","last_fullsized":"2024-03-20T09:41:29.960498+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:40:32.177180+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:38:01.419884+0000","last_clean_scrub_stamp":"2024-03-20T09:40:32.177180+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:41:32.746519+0000","scrub_duration":166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"20'137","reported_seq":2302,"reported_epoch":1085,"state":"active+clean","last_fresh":"2024-03-20T09:41:28.051003+0000","last_change":"2024-03-20T09:40:34.722786+0000","last_active":"2024-03-20T09:41:28.051003+0000","last_peered":"2024-03-20T09:41:28.051003+0000","last_clean":"2024-03-20T09:41:28.051003+0000","last_became_active":"2024-03-20T09:20:25.349082+0000","last_became_peered":"2024-03-20T09:20:25.349082+0000","last_unstale":"2024-03-20T09:41:28.051003+0000","last_undegraded":"2024-03-20T09:41:28.051003+0000","last_fullsized":"2024-03-20T09:41:28.051003+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:40:34.722704+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:36:56.203907+0000","last_clean_scrub_stamp":"2024-03-20T09:40:34.722704+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:41:52.404640+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"20'137","reported_seq":1585,"reported_epoch":1086,"state":"active+clean","last_fresh":"2024-03-20T09:41:29.960894+0000","last_change":"2024-03-20T09:40:46.884196+0000","last_active":"2024-03-20T09:41:29.960894+0000","last_peered":"2024-03-20T09:41:29.960894+0000","last_clean":"2024-03-20T09:41:29.960894+0000","last_became_active":"2024-03-20T09:12:14.307581+0000","last_became_peered":"2024-03-20T09:12:14.307581+0000","last_unstale":"2024-03-20T09:41:29.960894+0000","last_undegraded":"2024-03-20T09:41:29.960894+0000","last_fullsized":"2024-03-20T09:41:29.960894+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:40:46.884076+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:34:48.079272+0000","last_clean_scrub_stamp":"2024-03-20T09:40:46.884076+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:41:55.116456+0000","scrub_duration":241,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"20'137","reported_seq":1851,"reported_epoch":1086,"state":"active+clean","last_fresh":"2024-03-20T09:41:29.965502+0000","last_change":"2024-03-20T09:41:28.263008+0000","last_active":"2024-03-20T09:41:29.965502+0000","last_peered":"2024-03-20T09:41:29.965502+0000","last_clean":"2024-03-20T09:41:29.965502+0000","last_became_active":"2024-03-20T09:19:54.502391+0000","last_became_peered":"2024-03-20T09:19:54.502391+0000","last_unstale":"2024-03-20T09:41:29.965502+0000","last_undegraded":"2024-03-20T09:41:29.965502+0000","last_fullsized":"2024-03-20T09:41:29.965502+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:41:28.262878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:41:28.262878+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:42:47.190386+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":6394609664,"num_objects":97574,"num_object_clones":0,"num_object_copies":292722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97574,"num_write_kb":6244736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19199164416,"data_stored":19199164416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97574,"ondisk_log_size":97574,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1925120,"data_stored":1914464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4248,"ondisk_log_size":4248,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860963,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2668156,"kb_used_data":1581872,"kb_used_omap":1,"kb_used_meta":1086270,"kb_avail":91076996,"statfs":{"total":95995035648,"available":93262843904,"internally_reserved":0,"allocated":1619836928,"data_stored":1619442076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":1112340932},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":861,"seq":3697966841987,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2180076,"kb_used_data":1926948,"kb_used_omap":103,"kb_used_meta":253016,"kb_avail":91565076,"statfs":{"total":95995035648,"available":93762637824,"internally_reserved":0,"allocated":1973194752,"data_stored":1972804365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":106169,"internal_metadata":259088711},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":80,"seq":343597384239,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2951668,"kb_used_data":2294372,"kb_used_omap":1,"kb_used_meta":657278,"kb_avail":90793484,"statfs":{"total":95995035648,"available":92972527616,"internally_reserved":0,"allocated":2349436928,"data_stored":2349046541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":673053120},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":70,"seq":300647711284,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1428280,"kb_used_data":779312,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":92316872,"statfs":{"total":95995035648,"available":94532476928,"internally_reserved":0,"allocated":798015488,"data_stored":797620636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":664533446},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":60,"seq":257698038328,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2947844,"kb_used_data":1969848,"kb_used_omap":1,"kb_used_meta":977982,"kb_avail":90797308,"statfs":{"total":95995035648,"available":92976443392,"internally_reserved":0,"allocated":2017124352,"data_stored":2016725948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":1001454026},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332669,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1498556,"kb_used_data":795440,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":92246596,"statfs":{"total":95995035648,"available":94460514304,"internally_reserved":0,"allocated":814530560,"data_stored":814135708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":719976902},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575447,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4213612,"kb_used_data":3105124,"kb_used_omap":1,"kb_used_meta":1108478,"kb_avail":89531540,"statfs":{"total":95995035648,"available":91680296960,"internally_reserved":0,"allocated":3179646976,"data_stored":3179256589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1135081926},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738966,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411920,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":852832,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":92892320,"statfs":{"total":95995035648,"available":95121735680,"internally_reserved":0,"allocated":1200128,"data_stored":815169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":872085957},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150285,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2965044,"kb_used_data":1927332,"kb_used_omap":1,"kb_used_meta":1037694,"kb_avail":90780108,"statfs":{"total":95995035648,"available":92958830592,"internally_reserved":0,"allocated":1973587968,"data_stored":1973197581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1062599111},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438954059,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1539196,"kb_used_data":768048,"kb_used_omap":1,"kb_used_meta":771134,"kb_avail":92205956,"statfs":{"total":95995035648,"available":94418898944,"internally_reserved":0,"allocated":786481152,"data_stored":786086300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":789641669},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":443,"seq":1902670512500,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4385716,"kb_used_data":3489452,"kb_used_omap":181,"kb_used_meta":896074,"kb_avail":89359436,"statfs":{"total":95995035648,"available":91504062464,"internally_reserved":0,"allocated":3573198848,"data_stored":3572804909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":186131,"internal_metadata":917580013},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3189178368,"data_stored":3189178368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1998323712,"data_stored":1998323712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":790167552,"data_stored":790167552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3601596416,"data_stored":3601596416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":821035008,"data_stored":821035008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2030764032,"data_stored":2030764032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":805306368,"data_stored":805306368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2355036160,"data_stored":2355036160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1973420032,"data_stored":1973420032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1634336768,"data_stored":1634336768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:41:34.852 INFO:tasks.thrashosds.thrasher:Setting 1.1a to [3, 7, 4] 2024-03-20T09:41:34.852 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '1.1a', '3', '7', '4'] 2024-03-20T09:41:34.852 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pg-upmap 1.1a 3 7 4 2024-03-20T09:41:35.030 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:41:35.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:35.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:41:35.076 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:35 smithi026 ceph-mon[68309]: 1.e scrub starts 2024-03-20T09:41:35.076 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:35 smithi026 ceph-mon[68309]: 1.11 scrub ok 2024-03-20T09:41:35.076 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:35 smithi026 ceph-mon[68309]: 1.e scrub ok 2024-03-20T09:41:35.076 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:35 smithi026 ceph-mon[68309]: 1.8 scrub starts 2024-03-20T09:41:35.076 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:35 smithi026 ceph-mon[68309]: 1.8 scrub ok 2024-03-20T09:41:35.076 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:35 smithi026 ceph-mon[68309]: 1.5 scrub starts 2024-03-20T09:41:35.076 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:35 smithi026 ceph-mon[68309]: 1.5 scrub ok 2024-03-20T09:41:35.076 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:35 smithi026 ceph-mon[68309]: osdmap e1088: 12 total, 12 up, 11 in 2024-03-20T09:41:35.076 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:35 smithi026 ceph-mon[68309]: 1.17 scrub starts 2024-03-20T09:41:35.077 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:35 smithi026 ceph-mon[68309]: 1.a scrub starts 2024-03-20T09:41:35.077 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:35 smithi026 ceph-mon[68309]: 1.a scrub ok 2024-03-20T09:41:35.077 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:35 smithi026 ceph-mon[68309]: pgmap v2637: 47 pgs: 47 active+clean; 6.1 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-03-20T09:41:35.142 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:41:35.196 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:35.222 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:41:35.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:35.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:41:35.330 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:41:35.383 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:35.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:35 smithi115 ceph-mon[72476]: 1.e scrub starts 2024-03-20T09:41:35.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:35 smithi115 ceph-mon[72476]: 1.11 scrub ok 2024-03-20T09:41:35.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:35 smithi115 ceph-mon[72476]: 1.e scrub ok 2024-03-20T09:41:35.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:35 smithi115 ceph-mon[72476]: 1.8 scrub starts 2024-03-20T09:41:35.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:35 smithi115 ceph-mon[72476]: 1.8 scrub ok 2024-03-20T09:41:35.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:35 smithi115 ceph-mon[72476]: 1.5 scrub starts 2024-03-20T09:41:35.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:35 smithi115 ceph-mon[72476]: 1.5 scrub ok 2024-03-20T09:41:35.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:35 smithi115 ceph-mon[72476]: osdmap e1088: 12 total, 12 up, 11 in 2024-03-20T09:41:35.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:35 smithi115 ceph-mon[72476]: 1.17 scrub starts 2024-03-20T09:41:35.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:35 smithi115 ceph-mon[72476]: 1.a scrub starts 2024-03-20T09:41:35.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:35 smithi115 ceph-mon[72476]: 1.a scrub ok 2024-03-20T09:41:35.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:35 smithi115 ceph-mon[72476]: pgmap v2637: 47 pgs: 47 active+clean; 6.1 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-03-20T09:41:35.397 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:41:35.423 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:41:35.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:35.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:41:35.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:35 smithi122 ceph-mon[73495]: 1.e scrub starts 2024-03-20T09:41:35.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:35 smithi122 ceph-mon[73495]: 1.11 scrub ok 2024-03-20T09:41:35.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:35 smithi122 ceph-mon[73495]: 1.e scrub ok 2024-03-20T09:41:35.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:35 smithi122 ceph-mon[73495]: 1.8 scrub starts 2024-03-20T09:41:35.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:35 smithi122 ceph-mon[73495]: 1.8 scrub ok 2024-03-20T09:41:35.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:35 smithi122 ceph-mon[73495]: 1.5 scrub starts 2024-03-20T09:41:35.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:35 smithi122 ceph-mon[73495]: 1.5 scrub ok 2024-03-20T09:41:35.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:35 smithi122 ceph-mon[73495]: osdmap e1088: 12 total, 12 up, 11 in 2024-03-20T09:41:35.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:35 smithi122 ceph-mon[73495]: 1.17 scrub starts 2024-03-20T09:41:35.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:35 smithi122 ceph-mon[73495]: 1.a scrub starts 2024-03-20T09:41:35.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:35 smithi122 ceph-mon[73495]: 1.a scrub ok 2024-03-20T09:41:35.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:35 smithi122 ceph-mon[73495]: pgmap v2637: 47 pgs: 47 active+clean; 6.1 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-03-20T09:41:35.469 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:41:35.480 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:41:35.531 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:41:35.571 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:35.602 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:41:35.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:35.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:41:35.709 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:41:35.760 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:41:35.760 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-03-20T09:41:35.760 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-03-20T09:41:35.760 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:41:35.760 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.760 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26605.0:102111 7.b e899/871 7:d38335cc:::benchmark_data_smithi122_187077_object102110:head v 899'6508, mlcod=899'6506)", 2024-03-20T09:41:35.760 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:32:21.688515+0000", 2024-03-20T09:41:35.760 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 554.07773689700002, 2024-03-20T09:41:35.760 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.12018569799999999, 2024-03-20T09:41:35.761 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:35.761 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:35.761 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:35.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.761 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:35.761 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.688515+0000", 2024-03-20T09:41:35.761 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.761 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:35.761 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.688515+0000", 2024-03-20T09:41:35.761 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.761 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.761 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.762 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:35.762 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.688516+0000", 2024-03-20T09:41:35.762 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0950000000000001e-06 2024-03-20T09:41:35.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.762 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:35.762 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.689061+0000", 2024-03-20T09:41:35.762 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00054485999999999998 2024-03-20T09:41:35.762 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.762 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.762 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:35.762 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.689076+0000", 2024-03-20T09:41:35.762 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5780000000000001e-05 2024-03-20T09:41:35.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.763 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:35.763 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.689084+0000", 2024-03-20T09:41:35.763 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8369999999999997e-06 2024-03-20T09:41:35.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.763 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:35.763 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.689176+0000", 2024-03-20T09:41:35.763 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1724999999999993e-05 2024-03-20T09:41:35.763 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.763 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.763 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:35.763 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.689196+0000", 2024-03-20T09:41:35.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9956e-05 2024-03-20T09:41:35.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.764 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:35.764 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.808666+0000", 2024-03-20T09:41:35.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.11946981600000001 2024-03-20T09:41:35.764 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.764 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.764 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:35.764 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.808700+0000", 2024-03-20T09:41:35.764 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4629000000000003e-05 2024-03-20T09:41:35.764 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.764 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:35.764 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.765 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26605.0:102120 7.b e899/871 7:d9b83c24:::benchmark_data_smithi122_187077_object102119:head v 899'6509, mlcod=899'6506)", 2024-03-20T09:41:35.765 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:32:21.697837+0000", 2024-03-20T09:41:35.765 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 554.06841409100002, 2024-03-20T09:41:35.765 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.11089062399999999, 2024-03-20T09:41:35.765 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:35.765 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:35.765 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:35.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.765 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:35.765 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.697837+0000", 2024-03-20T09:41:35.765 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.765 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.765 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.766 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:35.766 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.697837+0000", 2024-03-20T09:41:35.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.766 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:35.766 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.697839+0000", 2024-03-20T09:41:35.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4050000000000001e-06 2024-03-20T09:41:35.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.766 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.766 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:35.766 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.697878+0000", 2024-03-20T09:41:35.766 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9359e-05 2024-03-20T09:41:35.766 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.767 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:35.767 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.697892+0000", 2024-03-20T09:41:35.767 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4246999999999999e-05 2024-03-20T09:41:35.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.767 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:35.767 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.697901+0000", 2024-03-20T09:41:35.767 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5629999999999992e-06 2024-03-20T09:41:35.767 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.767 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.767 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:35.767 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.697974+0000", 2024-03-20T09:41:35.767 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3530999999999998e-05 2024-03-20T09:41:35.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.768 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:35.768 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.697995+0000", 2024-03-20T09:41:35.768 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0234e-05 2024-03-20T09:41:35.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.768 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:35.768 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.808704+0000", 2024-03-20T09:41:35.768 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.11070951900000001 2024-03-20T09:41:35.768 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.768 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.768 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:35.769 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.808728+0000", 2024-03-20T09:41:35.769 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3765999999999999e-05 2024-03-20T09:41:35.769 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.769 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:35.769 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.769 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.769 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.769 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26605.0:102123 7.b e899/871 7:da80447b:::benchmark_data_smithi122_187077_object102122:head v 899'6510, mlcod=899'6506)", 2024-03-20T09:41:35.769 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:32:21.702780+0000", 2024-03-20T09:41:35.769 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 554.06347174899997, 2024-03-20T09:41:35.769 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.105972337, 2024-03-20T09:41:35.769 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:35.769 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:35.769 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:35.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.770 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:35.770 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.702780+0000", 2024-03-20T09:41:35.770 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.770 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:35.770 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.702780+0000", 2024-03-20T09:41:35.770 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.770 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.770 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.770 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:35.770 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.702781+0000", 2024-03-20T09:41:35.770 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.173e-06 2024-03-20T09:41:35.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.771 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:35.771 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.702843+0000", 2024-03-20T09:41:35.771 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2426000000000002e-05 2024-03-20T09:41:35.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.771 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:35.771 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.702854+0000", 2024-03-20T09:41:35.771 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0667e-05 2024-03-20T09:41:35.771 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.771 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.771 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:35.771 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.702862+0000", 2024-03-20T09:41:35.771 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5320000000000001e-06 2024-03-20T09:41:35.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.772 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:35.772 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.702935+0000", 2024-03-20T09:41:35.772 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3360000000000002e-05 2024-03-20T09:41:35.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.772 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:35.772 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.702956+0000", 2024-03-20T09:41:35.772 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0988999999999998e-05 2024-03-20T09:41:35.772 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.772 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.772 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:35.772 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.808730+0000", 2024-03-20T09:41:35.772 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.105774294 2024-03-20T09:41:35.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.773 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.773 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:35.773 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.808752+0000", 2024-03-20T09:41:35.773 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1895999999999998e-05 2024-03-20T09:41:35.773 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.773 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:35.773 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.773 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.773 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.773 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26605.0:102129 7.b e899/871 7:d06d3109:::benchmark_data_smithi122_187077_object102128:head v 899'6511, mlcod=899'6506)", 2024-03-20T09:41:35.773 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:32:21.706346+0000", 2024-03-20T09:41:35.773 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 554.05990569400001, 2024-03-20T09:41:35.773 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.102430682, 2024-03-20T09:41:35.774 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:35.774 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:35.774 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:35.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.774 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:35.774 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.706346+0000", 2024-03-20T09:41:35.774 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.774 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.774 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:35.774 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.706346+0000", 2024-03-20T09:41:35.774 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.774 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.775 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:35.775 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.706347+0000", 2024-03-20T09:41:35.775 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6000000000000002e-07 2024-03-20T09:41:35.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.775 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:35.775 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.706409+0000", 2024-03-20T09:41:35.775 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1915999999999995e-05 2024-03-20T09:41:35.775 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.775 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.775 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:35.775 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.706420+0000", 2024-03-20T09:41:35.775 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1467e-05 2024-03-20T09:41:35.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.776 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:35.776 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.706428+0000", 2024-03-20T09:41:35.776 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4919999999999996e-06 2024-03-20T09:41:35.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.776 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:35.776 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.706489+0000", 2024-03-20T09:41:35.776 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1057999999999997e-05 2024-03-20T09:41:35.776 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.776 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.776 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:35.776 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.706507+0000", 2024-03-20T09:41:35.777 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8073000000000001e-05 2024-03-20T09:41:35.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.777 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:35.777 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.808754+0000", 2024-03-20T09:41:35.777 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.10224725799999999 2024-03-20T09:41:35.777 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.777 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.777 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:35.777 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:21.808776+0000", 2024-03-20T09:41:35.777 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2558e-05 2024-03-20T09:41:35.777 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.777 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:35.777 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.778 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.778 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26605.0:123288 7.d e907/871 7:b14c167e:::benchmark_data_smithi122_187077_object123287:head v 907'7696, mlcod=907'7694)", 2024-03-20T09:41:35.778 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:32:45.159595+0000", 2024-03-20T09:41:35.778 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 530.60665693500005, 2024-03-20T09:41:35.778 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.111427225, 2024-03-20T09:41:35.778 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:35.778 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:35.778 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:35.778 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.779 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:35.779 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.159595+0000", 2024-03-20T09:41:35.779 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.779 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:35.779 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.159595+0000", 2024-03-20T09:41:35.779 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.779 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.779 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:35.779 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.159596+0000", 2024-03-20T09:41:35.779 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4839999999999999e-06 2024-03-20T09:41:35.779 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.780 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.780 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:35.780 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.159907+0000", 2024-03-20T09:41:35.780 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00031138500000000001 2024-03-20T09:41:35.780 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.780 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.780 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:35.780 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.159929+0000", 2024-03-20T09:41:35.780 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1746000000000002e-05 2024-03-20T09:41:35.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.781 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:35.781 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.159939+0000", 2024-03-20T09:41:35.781 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9469999999999998e-06 2024-03-20T09:41:35.781 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.781 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.781 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:35.781 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.160053+0000", 2024-03-20T09:41:35.781 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000113445 2024-03-20T09:41:35.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.782 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:35.782 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.160075+0000", 2024-03-20T09:41:35.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2898e-05 2024-03-20T09:41:35.782 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.782 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.782 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:35.782 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.270998+0000", 2024-03-20T09:41:35.782 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.110922303 2024-03-20T09:41:35.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.783 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:35.783 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.271022+0000", 2024-03-20T09:41:35.783 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4017000000000001e-05 2024-03-20T09:41:35.783 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.783 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:35.783 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.783 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.783 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.783 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26605.0:123289 7.d e907/871 7:b88215d3:::benchmark_data_smithi122_187077_object123288:head v 907'7697, mlcod=907'7694)", 2024-03-20T09:41:35.783 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:32:45.163937+0000", 2024-03-20T09:41:35.783 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 530.60231485300005, 2024-03-20T09:41:35.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.107102487, 2024-03-20T09:41:35.784 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:35.784 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:35.784 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:35.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.784 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:35.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.163937+0000", 2024-03-20T09:41:35.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.784 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.784 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:35.784 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.163937+0000", 2024-03-20T09:41:35.784 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.784 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.785 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:35.785 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.163937+0000", 2024-03-20T09:41:35.785 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4200000000000005e-07 2024-03-20T09:41:35.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.785 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:35.785 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.163973+0000", 2024-03-20T09:41:35.785 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5136999999999999e-05 2024-03-20T09:41:35.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.785 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.785 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:35.785 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.163983+0000", 2024-03-20T09:41:35.785 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0553999999999999e-05 2024-03-20T09:41:35.785 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.786 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:35.786 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.163989+0000", 2024-03-20T09:41:35.786 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0460000000000002e-06 2024-03-20T09:41:35.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.786 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:35.786 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.164057+0000", 2024-03-20T09:41:35.786 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7362999999999999e-05 2024-03-20T09:41:35.786 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.786 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.786 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:35.786 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.164071+0000", 2024-03-20T09:41:35.786 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4460999999999999e-05 2024-03-20T09:41:35.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:35.787 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.271025+0000", 2024-03-20T09:41:35.787 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.106953985 2024-03-20T09:41:35.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.787 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:35.787 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.271039+0000", 2024-03-20T09:41:35.787 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4099e-05 2024-03-20T09:41:35.787 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.787 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:35.787 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.787 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.787 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.788 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26605.0:123293 7.d e907/871 7:b75d82f3:::benchmark_data_smithi122_187077_object123292:head v 907'7698, mlcod=907'7694)", 2024-03-20T09:41:35.788 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:32:45.167587+0000", 2024-03-20T09:41:35.788 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 530.59866464900006, 2024-03-20T09:41:35.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.10346756999999999, 2024-03-20T09:41:35.788 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:35.788 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:35.788 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:35.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.788 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:35.788 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.167587+0000", 2024-03-20T09:41:35.788 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.788 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.788 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.788 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:35.788 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.167587+0000", 2024-03-20T09:41:35.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.789 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:35.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.167588+0000", 2024-03-20T09:41:35.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.003e-06 2024-03-20T09:41:35.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.789 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:35.789 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.167617+0000", 2024-03-20T09:41:35.789 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9200000000000002e-05 2024-03-20T09:41:35.789 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.789 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.790 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:35.790 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.167630+0000", 2024-03-20T09:41:35.790 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2644e-05 2024-03-20T09:41:35.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.790 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:35.790 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.167636+0000", 2024-03-20T09:41:35.790 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1650000000000003e-06 2024-03-20T09:41:35.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.790 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.790 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:35.790 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.167689+0000", 2024-03-20T09:41:35.790 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2743e-05 2024-03-20T09:41:35.790 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.791 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:35.791 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.167714+0000", 2024-03-20T09:41:35.791 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5369999999999999e-05 2024-03-20T09:41:35.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.791 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:35.791 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.271042+0000", 2024-03-20T09:41:35.791 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.103327617 2024-03-20T09:41:35.791 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.791 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.791 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:35.791 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:32:45.271054+0000", 2024-03-20T09:41:35.791 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2828e-05 2024-03-20T09:41:35.792 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.792 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:35.792 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.792 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.792 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26605.0:146616 7.b e915/871 7:d8c7451a:::benchmark_data_smithi122_187077_object20005:head v 915'9296, mlcod=915'9289)", 2024-03-20T09:41:35.792 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:33:11.246953+0000", 2024-03-20T09:41:35.792 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 504.51929866500001, 2024-03-20T09:41:35.792 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.12578051700000001, 2024-03-20T09:41:35.792 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:35.792 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:35.792 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:35.792 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.793 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:35.793 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.246953+0000", 2024-03-20T09:41:35.793 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.793 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:35.793 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.246953+0000", 2024-03-20T09:41:35.793 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.793 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.793 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.793 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:35.793 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.246954+0000", 2024-03-20T09:41:35.793 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7800000000000001e-07 2024-03-20T09:41:35.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.794 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:35.794 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.246960+0000", 2024-03-20T09:41:35.794 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8669999999999996e-06 2024-03-20T09:41:35.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.794 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:35.794 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.246971+0000", 2024-03-20T09:41:35.794 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0763e-05 2024-03-20T09:41:35.794 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.794 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.794 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:35.794 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.246978+0000", 2024-03-20T09:41:35.795 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0369999999999997e-06 2024-03-20T09:41:35.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.795 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:35.795 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.247047+0000", 2024-03-20T09:41:35.795 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8619000000000001e-05 2024-03-20T09:41:35.795 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.795 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.795 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:35.795 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.247071+0000", 2024-03-20T09:41:35.796 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4000000000000001e-05 2024-03-20T09:41:35.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.796 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:35.796 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.372697+0000", 2024-03-20T09:41:35.796 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.125626549 2024-03-20T09:41:35.796 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.796 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.796 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:35.796 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.372733+0000", 2024-03-20T09:41:35.797 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5904e-05 2024-03-20T09:41:35.797 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.797 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:35.797 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.797 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.797 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26605.0:146623 7.b e915/871 7:d7b7e0c1:::benchmark_data_smithi122_187077_object20012:head v 915'9297, mlcod=915'9289)", 2024-03-20T09:41:35.797 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:33:11.257535+0000", 2024-03-20T09:41:35.797 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 504.50871603299998, 2024-03-20T09:41:35.797 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.11522500099999999, 2024-03-20T09:41:35.797 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:35.797 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:35.797 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:35.797 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.798 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:35.798 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.257535+0000", 2024-03-20T09:41:35.798 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.798 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:35.798 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.257535+0000", 2024-03-20T09:41:35.798 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.798 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.798 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:35.798 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.257536+0000", 2024-03-20T09:41:35.798 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5499999999999997e-07 2024-03-20T09:41:35.798 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:35.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.257543+0000", 2024-03-20T09:41:35.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7800000000000003e-06 2024-03-20T09:41:35.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:35.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.257559+0000", 2024-03-20T09:41:35.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5531000000000001e-05 2024-03-20T09:41:35.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.799 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.799 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:35.799 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.257566+0000", 2024-03-20T09:41:35.799 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1600000000000001e-06 2024-03-20T09:41:35.799 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.800 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:35.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.257613+0000", 2024-03-20T09:41:35.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7508e-05 2024-03-20T09:41:35.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.800 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:35.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.257632+0000", 2024-03-20T09:41:35.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.84e-05 2024-03-20T09:41:35.800 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.800 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.800 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:35.800 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.372736+0000", 2024-03-20T09:41:35.800 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.11510437599999999 2024-03-20T09:41:35.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.801 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:35.801 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:33:11.372760+0000", 2024-03-20T09:41:35.801 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4391000000000001e-05 2024-03-20T09:41:35.801 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.801 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:35.801 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.801 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.801 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.801 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26917.0:106346 8.1 e1007/974 8:8ecbb4fe:::benchmark_data_smithi122_197775_object106345:head v 1007'8838, mlcod=1007'8836)", 2024-03-20T09:41:35.801 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:37:11.898094+0000", 2024-03-20T09:41:35.801 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 263.86815697899999, 2024-03-20T09:41:35.801 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.11849124799999999, 2024-03-20T09:41:35.801 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:35.802 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:35.802 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:35.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.802 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:35.802 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.898094+0000", 2024-03-20T09:41:35.802 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.802 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:35.802 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.898094+0000", 2024-03-20T09:41:35.802 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.802 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.802 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.802 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:35.803 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.898095+0000", 2024-03-20T09:41:35.803 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4799999999999999e-07 2024-03-20T09:41:35.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.803 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:35.803 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.898134+0000", 2024-03-20T09:41:35.803 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8902000000000001e-05 2024-03-20T09:41:35.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.803 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:35.803 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.898150+0000", 2024-03-20T09:41:35.803 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6753e-05 2024-03-20T09:41:35.803 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.803 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.804 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:35.804 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.898159+0000", 2024-03-20T09:41:35.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3690000000000007e-06 2024-03-20T09:41:35.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.804 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:35.804 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.902438+0000", 2024-03-20T09:41:35.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042786630000000003 2024-03-20T09:41:35.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.804 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.804 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:35.804 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.902475+0000", 2024-03-20T09:41:35.804 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7265000000000003e-05 2024-03-20T09:41:35.804 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.805 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:35.805 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:12.016542+0000", 2024-03-20T09:41:35.805 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.114067627 2024-03-20T09:41:35.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.805 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:35.805 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:12.016586+0000", 2024-03-20T09:41:35.805 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3321000000000001e-05 2024-03-20T09:41:35.805 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.805 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:35.805 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.805 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.805 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.805 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.9.0:49921 8.1 e1007/974 8:889d441c:::benchmark_data_smithi122_197775_object68126:head v 1007'8839, mlcod=1007'8836)", 2024-03-20T09:41:35.806 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:37:11.908320+0000", 2024-03-20T09:41:35.806 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 263.85793118999999, 2024-03-20T09:41:35.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.108292439, 2024-03-20T09:41:35.806 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:35.806 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:35.806 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:35.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:35.806 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.908320+0000", 2024-03-20T09:41:35.806 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.806 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.806 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.806 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:35.807 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.908320+0000", 2024-03-20T09:41:35.807 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.807 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:35.807 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.908321+0000", 2024-03-20T09:41:35.807 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.6899999999999996e-07 2024-03-20T09:41:35.807 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.807 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.807 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:35.807 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.908329+0000", 2024-03-20T09:41:35.808 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.011e-06 2024-03-20T09:41:35.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.808 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:35.808 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.908348+0000", 2024-03-20T09:41:35.808 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8678999999999999e-05 2024-03-20T09:41:35.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.808 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:35.808 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.908357+0000", 2024-03-20T09:41:35.808 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2620000000000004e-06 2024-03-20T09:41:35.808 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.808 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.808 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:35.809 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.919273+0000", 2024-03-20T09:41:35.809 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010915822 2024-03-20T09:41:35.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.809 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:35.809 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.919296+0000", 2024-03-20T09:41:35.809 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3040999999999998e-05 2024-03-20T09:41:35.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.809 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:35.809 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:12.016589+0000", 2024-03-20T09:41:35.809 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.097292613 2024-03-20T09:41:35.809 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.809 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.809 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:35.810 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:12.016613+0000", 2024-03-20T09:41:35.810 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4042000000000001e-05 2024-03-20T09:41:35.810 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.810 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:35.810 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.810 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.810 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.810 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.9.0:49922 8.1 e1007/974 8:889d7ebf:::benchmark_data_smithi122_197775_object44478:head v 1007'8840, mlcod=1007'8836)", 2024-03-20T09:41:35.810 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:37:11.908378+0000", 2024-03-20T09:41:35.810 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 263.857873117, 2024-03-20T09:41:35.810 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.11271600499999999, 2024-03-20T09:41:35.810 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:35.810 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:35.811 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:35.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.811 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:35.811 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.908378+0000", 2024-03-20T09:41:35.811 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.811 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:35.811 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.908378+0000", 2024-03-20T09:41:35.811 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.811 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.811 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.811 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:35.811 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.908379+0000", 2024-03-20T09:41:35.811 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-07 2024-03-20T09:41:35.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.812 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:35.812 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.908392+0000", 2024-03-20T09:41:35.812 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3049e-05 2024-03-20T09:41:35.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.812 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.812 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:35.812 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.908399+0000", 2024-03-20T09:41:35.812 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4569999999999999e-06 2024-03-20T09:41:35.812 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.813 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:35.813 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.908405+0000", 2024-03-20T09:41:35.813 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3569999999999997e-06 2024-03-20T09:41:35.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.813 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:35.813 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.923493+0000", 2024-03-20T09:41:35.813 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015088552999999999 2024-03-20T09:41:35.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.813 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.813 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:35.813 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.923513+0000", 2024-03-20T09:41:35.813 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9621999999999999e-05 2024-03-20T09:41:35.813 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.814 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:35.814 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:12.021040+0000", 2024-03-20T09:41:35.814 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.097527480999999999 2024-03-20T09:41:35.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.814 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:35.814 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:12.021094+0000", 2024-03-20T09:41:35.814 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4069e-05 2024-03-20T09:41:35.814 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.814 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:35.814 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.814 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.814 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.815 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.9.0:49923 8.1 e1007/974 8:889e2baa:::benchmark_data_smithi122_197775_object65050:head v 1007'8841, mlcod=1007'8836)", 2024-03-20T09:41:35.815 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:37:11.910666+0000", 2024-03-20T09:41:35.815 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 263.85558582099998, 2024-03-20T09:41:35.815 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.110456586, 2024-03-20T09:41:35.815 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:35.815 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:35.815 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:35.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.815 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:35.815 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.910666+0000", 2024-03-20T09:41:35.815 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.815 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.815 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.815 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:35.815 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.910666+0000", 2024-03-20T09:41:35.816 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.816 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:35.816 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.910667+0000", 2024-03-20T09:41:35.816 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5600000000000004e-07 2024-03-20T09:41:35.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.816 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:35.816 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.910674+0000", 2024-03-20T09:41:35.816 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8350000000000004e-06 2024-03-20T09:41:35.816 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.816 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.816 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:35.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.910684+0000", 2024-03-20T09:41:35.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.6749999999999997e-06 2024-03-20T09:41:35.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:35.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.910692+0000", 2024-03-20T09:41:35.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7400000000000004e-06 2024-03-20T09:41:35.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.817 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:35.817 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.927680+0000", 2024-03-20T09:41:35.817 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016987957000000001 2024-03-20T09:41:35.817 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.817 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.927699+0000", 2024-03-20T09:41:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9567000000000001e-05 2024-03-20T09:41:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:12.021098+0000", 2024-03-20T09:41:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.093398315999999995 2024-03-20T09:41:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:12.021122+0000", 2024-03-20T09:41:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4539999999999999e-05 2024-03-20T09:41:35.818 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.9.0:49924 8.1 e1007/974 8:889e9671:::benchmark_data_smithi122_197775_object18223:head v 1007'8842, mlcod=1007'8836)", 2024-03-20T09:41:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:37:11.916138+0000", 2024-03-20T09:41:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 263.850113781, 2024-03-20T09:41:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.105007585, 2024-03-20T09:41:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:35.819 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.916138+0000", 2024-03-20T09:41:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.916138+0000", 2024-03-20T09:41:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.916139+0000", 2024-03-20T09:41:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4300000000000001e-06 2024-03-20T09:41:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.820 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.916151+0000", 2024-03-20T09:41:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1469e-05 2024-03-20T09:41:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.916172+0000", 2024-03-20T09:41:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1308999999999999e-05 2024-03-20T09:41:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.916185+0000", 2024-03-20T09:41:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2656999999999999e-05 2024-03-20T09:41:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.821 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.932286+0000", 2024-03-20T09:41:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016101481000000001 2024-03-20T09:41:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.932307+0000", 2024-03-20T09:41:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1002999999999999e-05 2024-03-20T09:41:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:12.021124+0000", 2024-03-20T09:41:35.822 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.088816955000000003 2024-03-20T09:41:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:12.021145+0000", 2024-03-20T09:41:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1280999999999998e-05 2024-03-20T09:41:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26917.0:142971 8.b e1020/974 8:d85fb872:::benchmark_data_smithi122_197775_object19667:head v 1020'9002, mlcod=1020'9000)", 2024-03-20T09:41:35.823 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:37:52.204355+0000", 2024-03-20T09:41:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 223.561896969, 2024-03-20T09:41:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.107852276, 2024-03-20T09:41:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:52.204355+0000", 2024-03-20T09:41:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:35.824 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:52.204355+0000", 2024-03-20T09:41:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:52.204355+0000", 2024-03-20T09:41:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8299999999999996e-07 2024-03-20T09:41:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:52.204362+0000", 2024-03-20T09:41:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4389999999999997e-06 2024-03-20T09:41:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.825 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:35.826 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:52.204373+0000", 2024-03-20T09:41:35.826 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1654999999999999e-05 2024-03-20T09:41:35.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.826 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:35.826 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:52.204381+0000", 2024-03-20T09:41:35.826 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8730000000000007e-06 2024-03-20T09:41:35.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.826 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.826 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:35.826 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:52.204448+0000", 2024-03-20T09:41:35.826 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6307000000000001e-05 2024-03-20T09:41:35.826 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.827 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:35.827 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:52.204470+0000", 2024-03-20T09:41:35.827 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2149000000000001e-05 2024-03-20T09:41:35.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.827 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:35.827 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:52.312172+0000", 2024-03-20T09:41:35.827 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.107702674 2024-03-20T09:41:35.827 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.827 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.827 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:35.827 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:52.312207+0000", 2024-03-20T09:41:35.828 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4396000000000001e-05 2024-03-20T09:41:35.828 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.828 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:35.828 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.828 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.828 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.828 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26917.0:182432 8.3 e1030/974 8:c2ab0e0e:::benchmark_data_smithi122_197775_object59128:head v 1030'11505, mlcod=1030'11502)", 2024-03-20T09:41:35.828 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:38:38.940258+0000", 2024-03-20T09:41:35.828 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 176.82599307800001, 2024-03-20T09:41:35.828 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.12006412599999999, 2024-03-20T09:41:35.828 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:35.828 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:35.828 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:35.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.829 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:35.829 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.940258+0000", 2024-03-20T09:41:35.829 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.829 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:35.829 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.940258+0000", 2024-03-20T09:41:35.829 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.829 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.829 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.829 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:35.829 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.940259+0000", 2024-03-20T09:41:35.829 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.15e-07 2024-03-20T09:41:35.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.830 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:35.830 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.940263+0000", 2024-03-20T09:41:35.830 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2259999999999999e-06 2024-03-20T09:41:35.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.830 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:35.830 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.940269+0000", 2024-03-20T09:41:35.830 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8130000000000001e-06 2024-03-20T09:41:35.830 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.830 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.830 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:35.830 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.940274+0000", 2024-03-20T09:41:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1610000000000002e-06 2024-03-20T09:41:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.943922+0000", 2024-03-20T09:41:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036481159999999999 2024-03-20T09:41:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.943943+0000", 2024-03-20T09:41:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1120000000000001e-05 2024-03-20T09:41:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.831 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:39.060295+0000", 2024-03-20T09:41:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.116351756 2024-03-20T09:41:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:39.060323+0000", 2024-03-20T09:41:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7619000000000001e-05 2024-03-20T09:41:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26917.0:182440 8.3 e1030/974 8:c9a37a50:::benchmark_data_smithi122_197775_object59136:head v 1030'11506, mlcod=1030'11503)", 2024-03-20T09:41:35.832 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:38:38.941044+0000", 2024-03-20T09:41:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 176.825207844, 2024-03-20T09:41:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.119320419, 2024-03-20T09:41:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.941044+0000", 2024-03-20T09:41:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:35.833 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.941044+0000", 2024-03-20T09:41:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.941044+0000", 2024-03-20T09:41:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8500000000000002e-07 2024-03-20T09:41:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.941048+0000", 2024-03-20T09:41:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.279e-06 2024-03-20T09:41:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.834 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.941055+0000", 2024-03-20T09:41:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0530000000000001e-06 2024-03-20T09:41:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.941060+0000", 2024-03-20T09:41:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0039999999999999e-06 2024-03-20T09:41:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.948506+0000", 2024-03-20T09:41:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074458959999999996 2024-03-20T09:41:35.835 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.948525+0000", 2024-03-20T09:41:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8739000000000001e-05 2024-03-20T09:41:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:39.060326+0000", 2024-03-20T09:41:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.111800881 2024-03-20T09:41:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:39.060364+0000", 2024-03-20T09:41:35.836 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8182000000000001e-05 2024-03-20T09:41:35.837 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.837 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:35.837 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.837 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.837 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26917.0:182445 8.3 e1030/974 8:c44e21cb:::benchmark_data_smithi122_197775_object59141:head v 1030'11507, mlcod=1030'11503)", 2024-03-20T09:41:35.837 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:38:38.951394+0000", 2024-03-20T09:41:35.837 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 176.81485788800001, 2024-03-20T09:41:35.837 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.10899990499999999, 2024-03-20T09:41:35.837 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:35.837 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:35.837 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:35.837 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.837 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:35.837 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.951394+0000", 2024-03-20T09:41:35.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:35.838 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.951394+0000", 2024-03-20T09:41:35.838 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.838 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.838 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.838 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:35.838 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.951394+0000", 2024-03-20T09:41:35.839 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8899999999999998e-07 2024-03-20T09:41:35.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.839 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:35.839 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.951401+0000", 2024-03-20T09:41:35.839 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9999999999999999e-06 2024-03-20T09:41:35.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.839 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:35.839 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.951412+0000", 2024-03-20T09:41:35.839 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0498e-05 2024-03-20T09:41:35.839 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.839 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.840 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:35.840 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.951419+0000", 2024-03-20T09:41:35.840 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1430000000000001e-06 2024-03-20T09:41:35.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.840 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:35.840 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.958710+0000", 2024-03-20T09:41:35.840 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.007290927 2024-03-20T09:41:35.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.840 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:35.840 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.958732+0000", 2024-03-20T09:41:35.840 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2487999999999999e-05 2024-03-20T09:41:35.840 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.840 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.841 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:35.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:39.060367+0000", 2024-03-20T09:41:35.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.10163462600000001 2024-03-20T09:41:35.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.841 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:35.841 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:39.060393+0000", 2024-03-20T09:41:35.841 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6434000000000001e-05 2024-03-20T09:41:35.841 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.841 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:35.841 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.841 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.841 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.841 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.27190.0:65717 9.0 e1080/1066 9:0049fc1c:::benchmark_data_smithi122_210164_object65716:head v 1080'4239, mlcod=1080'4235)", 2024-03-20T09:41:35.842 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:41:07.889791+0000", 2024-03-20T09:41:35.842 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 27.876460439999999, 2024-03-20T09:41:35.842 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.092319735999999999, 2024-03-20T09:41:35.842 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:35.842 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:35.842 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:35.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.842 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:35.842 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.889791+0000", 2024-03-20T09:41:35.842 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.842 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.842 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.842 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:35.842 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.889791+0000", 2024-03-20T09:41:35.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:35.843 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.889792+0000", 2024-03-20T09:41:35.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8100000000000002e-07 2024-03-20T09:41:35.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:35.843 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.889855+0000", 2024-03-20T09:41:35.843 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2867999999999995e-05 2024-03-20T09:41:35.843 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.843 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.843 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:35.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.889865+0000", 2024-03-20T09:41:35.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9380000000000004e-06 2024-03-20T09:41:35.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.844 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:35.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.889871+0000", 2024-03-20T09:41:35.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8009999999999998e-06 2024-03-20T09:41:35.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.844 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:35.844 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.889922+0000", 2024-03-20T09:41:35.844 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0414000000000002e-05 2024-03-20T09:41:35.844 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.844 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.845 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:35.845 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.889939+0000", 2024-03-20T09:41:35.845 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6966999999999999e-05 2024-03-20T09:41:35.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:35.846 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.982062+0000", 2024-03-20T09:41:35.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.092123249000000004 2024-03-20T09:41:35.846 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.846 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.846 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:35.846 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.982111+0000", 2024-03-20T09:41:35.846 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8817999999999999e-05 2024-03-20T09:41:35.847 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.847 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:35.847 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.847 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.847 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.847 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.27190.0:65723 9.4 e1080/1066 9:2e718743:::benchmark_data_smithi122_210164_object65722:head v 1080'4078, mlcod=1080'4074)", 2024-03-20T09:41:35.847 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:41:07.890566+0000", 2024-03-20T09:41:35.847 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 27.875685605000001, 2024-03-20T09:41:35.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.093069874999999996, 2024-03-20T09:41:35.848 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:41:35.848 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:41:35.848 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:41:35.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.848 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:41:35.848 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.890566+0000", 2024-03-20T09:41:35.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.848 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.848 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:41:35.848 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.890566+0000", 2024-03-20T09:41:35.848 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:41:35.848 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.849 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:41:35.849 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.890567+0000", 2024-03-20T09:41:35.849 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1699999999999997e-07 2024-03-20T09:41:35.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.849 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:41:35.849 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.890626+0000", 2024-03-20T09:41:35.849 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8927e-05 2024-03-20T09:41:35.849 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.849 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.849 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:41:35.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.890634+0000", 2024-03-20T09:41:35.850 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0090000000000007e-06 2024-03-20T09:41:35.850 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.850 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.850 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:41:35.850 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.890638+0000", 2024-03-20T09:41:35.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6829999999999999e-06 2024-03-20T09:41:35.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:41:35.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.890681+0000", 2024-03-20T09:41:35.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2808000000000001e-05 2024-03-20T09:41:35.851 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.851 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.851 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:41:35.851 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.890699+0000", 2024-03-20T09:41:35.851 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7689999999999998e-05 2024-03-20T09:41:35.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:41:35.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.983535+0000", 2024-03-20T09:41:35.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.092836034999999997 2024-03-20T09:41:35.852 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:41:35.852 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:41:35.852 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:41:35.852 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.983636+0000", 2024-03-20T09:41:35.852 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101006 2024-03-20T09:41:35.852 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.852 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:35.852 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.852 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:41:35.853 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:41:35.853 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:41:35.853 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 56 16 104734 104718 116.837 72.8125 0.0176754 0.00855488 2024-03-20T09:41:35.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:35.854 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:35.855 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:41:35.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:41:35.956 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:41:36.008 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:36.035 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:41:36.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:36.038 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:41:36.139 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:41:36.154 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:36 smithi026 ceph-mon[68309]: 1.17 scrub ok 2024-03-20T09:41:36.154 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:36 smithi026 ceph-mon[68309]: 1.10 scrub starts 2024-03-20T09:41:36.154 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:36 smithi026 ceph-mon[68309]: 1.10 scrub ok 2024-03-20T09:41:36.154 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:36 smithi026 ceph-mon[68309]: 9.3 scrub starts 2024-03-20T09:41:36.208 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:36.252 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:41:36.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:36.262 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:41:36.262 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:36 smithi122 ceph-mon[73495]: 1.17 scrub ok 2024-03-20T09:41:36.262 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:36 smithi122 ceph-mon[73495]: 1.10 scrub starts 2024-03-20T09:41:36.262 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:36 smithi122 ceph-mon[73495]: 1.10 scrub ok 2024-03-20T09:41:36.262 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:36 smithi122 ceph-mon[73495]: 9.3 scrub starts 2024-03-20T09:41:36.362 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:41:36.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:36 smithi115 ceph-mon[72476]: 1.17 scrub ok 2024-03-20T09:41:36.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:36 smithi115 ceph-mon[72476]: 1.10 scrub starts 2024-03-20T09:41:36.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:36 smithi115 ceph-mon[72476]: 1.10 scrub ok 2024-03-20T09:41:36.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:36 smithi115 ceph-mon[72476]: 9.3 scrub starts 2024-03-20T09:41:36.434 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:36.475 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:41:36.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:36.479 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:41:36.580 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:41:36.620 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:36.656 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:41:36.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:36.662 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:41:36.764 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:41:36.801 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:36.824 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 57 16 105749 105733 115.901 63.4375 0.00835794 0.0086235 2024-03-20T09:41:36.935 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:41:36.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:36.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:41:37.040 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:41:37.090 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:37.127 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:41:37.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:37.132 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:41:37.205 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:37 smithi026 ceph-mon[68309]: 9.7 scrub starts 2024-03-20T09:41:37.205 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:37 smithi026 ceph-mon[68309]: 1.4 scrub starts 2024-03-20T09:41:37.205 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:37 smithi026 ceph-mon[68309]: 1.4 scrub ok 2024-03-20T09:41:37.205 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:37 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:41:37.205 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:37 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:41:37.205 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:37 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:41:37.205 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:37 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:41:37.205 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:37 smithi026 ceph-mon[68309]: pgmap v2638: 47 pgs: 47 active+clean; 6.3 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-03-20T09:41:37.233 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:41:37.267 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:37.296 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:41:37.302 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:37 smithi122 ceph-mon[73495]: 9.7 scrub starts 2024-03-20T09:41:37.302 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:37 smithi122 ceph-mon[73495]: 1.4 scrub starts 2024-03-20T09:41:37.302 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:37 smithi122 ceph-mon[73495]: 1.4 scrub ok 2024-03-20T09:41:37.302 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:37 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:41:37.302 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:37 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:41:37.302 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:37 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:41:37.302 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:37 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:41:37.302 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:37 smithi122 ceph-mon[73495]: pgmap v2638: 47 pgs: 47 active+clean; 6.3 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-03-20T09:41:37.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:37.303 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:41:37.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:37 smithi115 ceph-mon[72476]: 9.7 scrub starts 2024-03-20T09:41:37.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:37 smithi115 ceph-mon[72476]: 1.4 scrub starts 2024-03-20T09:41:37.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:37 smithi115 ceph-mon[72476]: 1.4 scrub ok 2024-03-20T09:41:37.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:37 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:41:37.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:37 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:41:37.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:37 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:41:37.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:37 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:41:37.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:37 smithi115 ceph-mon[72476]: pgmap v2638: 47 pgs: 47 active+clean; 6.3 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-03-20T09:41:37.404 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:41:37.444 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:37.482 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:41:37.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:37.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:41:37.541 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-20T09:41:37.588 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:41:37.639 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:37.668 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:41:37.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:37.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:41:37.776 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:41:37.817 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:37.827 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 58 15 106771 106756 114.998 63.9375 0.0166143 0.00869176 2024-03-20T09:41:37.880 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:41:37.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:37.883 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:41:37.984 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:41:38.041 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:38.082 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:41:38.084 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:38 smithi122 ceph-mon[73495]: osdmap e1089: 12 total, 12 up, 11 in 2024-03-20T09:41:38.084 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:38 smithi122 ceph-mon[73495]: 9.3 scrub ok 2024-03-20T09:41:38.084 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:38 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:41:38.084 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:38 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:41:38.085 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:38 smithi122 ceph-mon[73495]: 1.9 scrub starts 2024-03-20T09:41:38.085 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:38 smithi122 ceph-mon[73495]: 9.7 scrub ok 2024-03-20T09:41:38.085 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:38 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:41:38.085 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:38 smithi122 ceph-mon[73495]: 1.9 scrub ok 2024-03-20T09:41:38.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:38.086 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:41:38.187 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:41:38.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:38 smithi026 ceph-mon[68309]: osdmap e1089: 12 total, 12 up, 11 in 2024-03-20T09:41:38.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:38 smithi026 ceph-mon[68309]: 9.3 scrub ok 2024-03-20T09:41:38.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:38 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:41:38.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:38 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:41:38.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:38 smithi026 ceph-mon[68309]: 1.9 scrub starts 2024-03-20T09:41:38.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:38 smithi026 ceph-mon[68309]: 9.7 scrub ok 2024-03-20T09:41:38.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:38 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:41:38.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:38 smithi026 ceph-mon[68309]: 1.9 scrub ok 2024-03-20T09:41:38.230 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:38.254 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:41:38.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:38.263 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:41:38.364 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:41:38.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:38 smithi115 ceph-mon[72476]: osdmap e1089: 12 total, 12 up, 11 in 2024-03-20T09:41:38.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:38 smithi115 ceph-mon[72476]: 9.3 scrub ok 2024-03-20T09:41:38.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:38 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:41:38.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:38 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:41:38.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:38 smithi115 ceph-mon[72476]: 1.9 scrub starts 2024-03-20T09:41:38.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:38 smithi115 ceph-mon[72476]: 9.7 scrub ok 2024-03-20T09:41:38.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:38 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:41:38.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:38 smithi115 ceph-mon[72476]: 1.9 scrub ok 2024-03-20T09:41:38.418 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:38.447 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:41:38.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:38.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:41:38.554 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:41:38.565 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:41:38.565 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:41:38.565 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:41:38.565 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:41:38.566 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:41:38.566 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:41:38.566 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:41:38.566 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:41:38.566 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:41:38.592 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:38.613 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:41:38.634 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:41:38.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:38.638 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:41:38.655 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:41:38.688 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:41:38.695 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:41:38.696 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:41:38.696 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:41:38.696 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:41:38.696 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:41:38.696 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:41:38.696 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:41:38.696 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:41:38.696 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:41:38.696 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:41:38.696 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:41:38.696 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:41:38.739 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:41:38.783 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:38.815 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:41:38.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:38.819 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:41:38.827 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 59 16 107850 107834 114.191 67.375 0.0189875 0.00875262 2024-03-20T09:41:38.920 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:41:38.961 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:39.000 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:41:39.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:39.014 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:41:39.115 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:41:39.165 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:39.189 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:41:39.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:39.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:41:39.294 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:41:39.342 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:39 smithi122 ceph-mon[73495]: osdmap e1090: 12 total, 12 up, 11 in 2024-03-20T09:41:39.342 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:39 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3626262063' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:41:39.342 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:39 smithi122 ceph-mon[73495]: pgmap v2641: 47 pgs: 47 active+clean; 6.4 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-03-20T09:41:39.342 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:39 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:41:39.343 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:39 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:41:39.343 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:39 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:41:39.345 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:39.376 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:41:39.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:39.382 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:41:39.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:39 smithi115 ceph-mon[72476]: osdmap e1090: 12 total, 12 up, 11 in 2024-03-20T09:41:39.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:39 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3626262063' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:41:39.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:39 smithi115 ceph-mon[72476]: pgmap v2641: 47 pgs: 47 active+clean; 6.4 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-03-20T09:41:39.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:39 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:41:39.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:39 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:41:39.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:39 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:41:39.456 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:41:39.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:39 smithi026 ceph-mon[68309]: osdmap e1090: 12 total, 12 up, 11 in 2024-03-20T09:41:39.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:39 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3626262063' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:41:39.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:39 smithi026 ceph-mon[68309]: pgmap v2641: 47 pgs: 47 active+clean; 6.4 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-03-20T09:41:39.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:39 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:41:39.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:39 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:41:39.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:39 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:41:39.483 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:41:39.529 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:39.828 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:2024-03-20T09:41:39.840661+0000 min lat: 0.00113956 max lat: 0.214394 avg lat: 0.00880629 2024-03-20T09:41:39.828 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T09:41:39.828 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 60 16 109014 108998 113.5 72.75 0.0179808 0.00880629 2024-03-20T09:41:40.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:40 smithi115 ceph-mon[72476]: 1.1 scrub starts 2024-03-20T09:41:40.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:40 smithi115 ceph-mon[72476]: 1.1 scrub ok 2024-03-20T09:41:40.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:40 smithi115 ceph-mon[72476]: 1.13 scrub starts 2024-03-20T09:41:40.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:40 smithi122 ceph-mon[73495]: 1.1 scrub starts 2024-03-20T09:41:40.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:40 smithi122 ceph-mon[73495]: 1.1 scrub ok 2024-03-20T09:41:40.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:40 smithi122 ceph-mon[73495]: 1.13 scrub starts 2024-03-20T09:41:40.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:40 smithi026 ceph-mon[68309]: 1.1 scrub starts 2024-03-20T09:41:40.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:40 smithi026 ceph-mon[68309]: 1.1 scrub ok 2024-03-20T09:41:40.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:40 smithi026 ceph-mon[68309]: 1.13 scrub starts 2024-03-20T09:41:40.829 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 61 16 110105 110089 112.755 68.1875 0.011414 0.00886491 2024-03-20T09:41:41.090 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:41:41.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:41.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:41:41.196 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:41:41.247 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:41.283 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:41:41.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:41.289 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:41:41.304 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:41 smithi026 ceph-mon[68309]: 1.13 scrub ok 2024-03-20T09:41:41.304 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:41 smithi026 ceph-mon[68309]: 1.3 scrub starts 2024-03-20T09:41:41.304 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:41 smithi026 ceph-mon[68309]: 1.3 scrub ok 2024-03-20T09:41:41.305 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:41 smithi026 ceph-mon[68309]: 1.12 scrub starts 2024-03-20T09:41:41.305 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:41 smithi026 ceph-mon[68309]: 1.12 scrub ok 2024-03-20T09:41:41.305 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:41 smithi026 ceph-mon[68309]: pgmap v2642: 47 pgs: 47 active+clean; 6.6 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-03-20T09:41:41.389 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:41:41.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:41 smithi115 ceph-mon[72476]: 1.13 scrub ok 2024-03-20T09:41:41.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:41 smithi115 ceph-mon[72476]: 1.3 scrub starts 2024-03-20T09:41:41.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:41 smithi115 ceph-mon[72476]: 1.3 scrub ok 2024-03-20T09:41:41.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:41 smithi115 ceph-mon[72476]: 1.12 scrub starts 2024-03-20T09:41:41.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:41 smithi115 ceph-mon[72476]: 1.12 scrub ok 2024-03-20T09:41:41.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:41 smithi115 ceph-mon[72476]: pgmap v2642: 47 pgs: 47 active+clean; 6.6 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-03-20T09:41:41.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:41 smithi122 ceph-mon[73495]: 1.13 scrub ok 2024-03-20T09:41:41.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:41 smithi122 ceph-mon[73495]: 1.3 scrub starts 2024-03-20T09:41:41.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:41 smithi122 ceph-mon[73495]: 1.3 scrub ok 2024-03-20T09:41:41.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:41 smithi122 ceph-mon[73495]: 1.12 scrub starts 2024-03-20T09:41:41.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:41 smithi122 ceph-mon[73495]: 1.12 scrub ok 2024-03-20T09:41:41.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:41 smithi122 ceph-mon[73495]: pgmap v2642: 47 pgs: 47 active+clean; 6.6 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-03-20T09:41:41.448 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:41.488 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:41:41.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:41.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:41:41.596 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:41:41.680 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:41.714 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:41:41.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:41.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:41:41.820 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:41:41.829 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 62 16 111098 111082 111.937 62.0625 0.0149975 0.00892923 2024-03-20T09:41:41.859 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:42.052 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:42 smithi115 ceph-mon[72476]: 9.e scrub starts 2024-03-20T09:41:42.052 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:42 smithi115 ceph-mon[72476]: 9.a scrub starts 2024-03-20T09:41:42.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:42 smithi026 ceph-mon[68309]: 9.e scrub starts 2024-03-20T09:41:42.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:42 smithi026 ceph-mon[68309]: 9.a scrub starts 2024-03-20T09:41:42.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:42 smithi122 ceph-mon[73495]: 9.e scrub starts 2024-03-20T09:41:42.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:42 smithi122 ceph-mon[73495]: 9.a scrub starts 2024-03-20T09:41:42.829 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 63 16 112108 112092 111.163 63.125 0.0192731 0.00899211 2024-03-20T09:41:43.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:43 smithi026 ceph-mon[68309]: 9.4 scrub ok 2024-03-20T09:41:43.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:43 smithi026 ceph-mon[68309]: 9.c scrub starts 2024-03-20T09:41:43.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:43 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3678678479' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.1a", "id": [3, 7, 4]}]: dispatch 2024-03-20T09:41:43.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:43 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.1a", "id": [3, 7, 4]}]: dispatch 2024-03-20T09:41:43.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:43 smithi026 ceph-mon[68309]: 9.e scrub ok 2024-03-20T09:41:43.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:43 smithi026 ceph-mon[68309]: pgmap v2643: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.7 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-03-20T09:41:43.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:43 smithi115 ceph-mon[72476]: 9.4 scrub ok 2024-03-20T09:41:43.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:43 smithi115 ceph-mon[72476]: 9.c scrub starts 2024-03-20T09:41:43.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:43 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3678678479' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.1a", "id": [3, 7, 4]}]: dispatch 2024-03-20T09:41:43.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:43 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.1a", "id": [3, 7, 4]}]: dispatch 2024-03-20T09:41:43.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:43 smithi115 ceph-mon[72476]: 9.e scrub ok 2024-03-20T09:41:43.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:43 smithi115 ceph-mon[72476]: pgmap v2643: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.7 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-03-20T09:41:43.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:43 smithi122 ceph-mon[73495]: 9.4 scrub ok 2024-03-20T09:41:43.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:43 smithi122 ceph-mon[73495]: 9.c scrub starts 2024-03-20T09:41:43.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:43 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3678678479' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.1a", "id": [3, 7, 4]}]: dispatch 2024-03-20T09:41:43.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:43 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.1a", "id": [3, 7, 4]}]: dispatch 2024-03-20T09:41:43.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:43 smithi122 ceph-mon[73495]: 9.e scrub ok 2024-03-20T09:41:43.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:43 smithi122 ceph-mon[73495]: pgmap v2643: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.7 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-03-20T09:41:43.552 INFO:teuthology.orchestra.run.smithi026.stderr:set 1.1a pg_upmap mapping to [3,7,4] 2024-03-20T09:41:43.830 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 64 16 113236 113220 110.527 70.5 0.015193 0.00904341 2024-03-20T09:41:44.171 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:44 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.1a", "id": [3, 7, 4]}]': finished 2024-03-20T09:41:44.171 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:44 smithi026 ceph-mon[68309]: osdmap e1091: 12 total, 12 up, 11 in 2024-03-20T09:41:44.171 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:44 smithi026 ceph-mon[68309]: 9.a scrub ok 2024-03-20T09:41:44.171 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:44 smithi026 ceph-mon[68309]: 9.c scrub ok 2024-03-20T09:41:44.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:44 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.1a", "id": [3, 7, 4]}]': finished 2024-03-20T09:41:44.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:44 smithi115 ceph-mon[72476]: osdmap e1091: 12 total, 12 up, 11 in 2024-03-20T09:41:44.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:44 smithi115 ceph-mon[72476]: 9.a scrub ok 2024-03-20T09:41:44.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:44 smithi115 ceph-mon[72476]: 9.c scrub ok 2024-03-20T09:41:44.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:44 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.1a", "id": [3, 7, 4]}]': finished 2024-03-20T09:41:44.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:44 smithi122 ceph-mon[73495]: osdmap e1091: 12 total, 12 up, 11 in 2024-03-20T09:41:44.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:44 smithi122 ceph-mon[73495]: 9.a scrub ok 2024-03-20T09:41:44.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:44 smithi122 ceph-mon[73495]: 9.c scrub ok 2024-03-20T09:41:44.832 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 65 16 114355 114339 109.899 69.9375 0.00834108 0.00909491 2024-03-20T09:41:44.918 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:41:44.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:44.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:41:45.028 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:41:45.065 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:45.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:45 smithi026 ceph-mon[68309]: 9.0 scrub starts 2024-03-20T09:41:45.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:45 smithi026 ceph-mon[68309]: osdmap e1092: 12 total, 12 up, 11 in 2024-03-20T09:41:45.209 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:45 smithi026 ceph-mon[68309]: pgmap v2646: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.8 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-03-20T09:41:45.332 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:41:45.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:45.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:41:45.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:45 smithi115 ceph-mon[72476]: 9.0 scrub starts 2024-03-20T09:41:45.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:45 smithi115 ceph-mon[72476]: osdmap e1092: 12 total, 12 up, 11 in 2024-03-20T09:41:45.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:45 smithi115 ceph-mon[72476]: pgmap v2646: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.8 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-03-20T09:41:45.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:45 smithi122 ceph-mon[73495]: 9.0 scrub starts 2024-03-20T09:41:45.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:45 smithi122 ceph-mon[73495]: osdmap e1092: 12 total, 12 up, 11 in 2024-03-20T09:41:45.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:45 smithi122 ceph-mon[73495]: pgmap v2646: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.8 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-03-20T09:41:45.437 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:41:45.479 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:41:45.489 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:45.514 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:41:45.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:45.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:41:45.621 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:41:45.677 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:45.832 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 66 16 115451 115435 109.272 68.5 0.0149483 0.00914754 2024-03-20T09:41:45.863 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:41:45.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:45.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:41:45.956 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:41:45.968 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:41:46.003 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:46.037 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:41:46.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:46.041 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:41:46.111 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:41:46.141 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:41:46.180 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:46.212 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:41:46.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:46.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:41:46.318 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:41:46.358 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:46 smithi122 ceph-mon[73495]: osdmap e1093: 12 total, 12 up, 11 in 2024-03-20T09:41:46.358 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:46 smithi122 ceph-mon[73495]: 9.0 scrub ok 2024-03-20T09:41:46.358 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:46 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2825307222' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:41:46.358 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:46 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:41:46.361 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:46.388 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:41:46.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:46.394 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:41:46.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:46 smithi115 ceph-mon[72476]: osdmap e1093: 12 total, 12 up, 11 in 2024-03-20T09:41:46.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:46 smithi115 ceph-mon[72476]: 9.0 scrub ok 2024-03-20T09:41:46.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:46 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2825307222' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:41:46.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:46 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:41:46.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:46 smithi026 ceph-mon[68309]: osdmap e1093: 12 total, 12 up, 11 in 2024-03-20T09:41:46.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:46 smithi026 ceph-mon[68309]: 9.0 scrub ok 2024-03-20T09:41:46.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:46 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2825307222' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:41:46.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:46 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:41:46.495 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:41:46.548 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:46.582 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:41:46.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:46.587 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:41:46.688 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:41:46.736 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:46.832 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 67 16 116577 116561 108.691 70.375 0.0101951 0.00919659 2024-03-20T09:41:46.991 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:41:46.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:46.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:41:47.003 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:41:47.003 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:41:47.100 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:41:47.143 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:47.176 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:41:47.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:47.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:41:47.280 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:41:47.337 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:47.337 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:47 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:41:47.338 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:47 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:41:47.338 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:47 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:41:47.338 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:47 smithi026 ceph-mon[68309]: Cluster is now healthy 2024-03-20T09:41:47.338 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:47 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:41:47.338 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:47 smithi026 ceph-mon[68309]: osdmap e1094: 12 total, 12 up, 11 in 2024-03-20T09:41:47.338 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:47 smithi026 ceph-mon[68309]: pgmap v2649: 47 pgs: 1 remapped+peering, 46 active+clean; 7.0 GiB data, 30 GiB used, 954 GiB / 983 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-03-20T09:41:47.372 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:41:47.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:47.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:41:47.381 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:47 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:41:47.381 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:47 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:41:47.381 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:47 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:41:47.382 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:47 smithi115 ceph-mon[72476]: Cluster is now healthy 2024-03-20T09:41:47.382 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:47 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:41:47.382 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:47 smithi115 ceph-mon[72476]: osdmap e1094: 12 total, 12 up, 11 in 2024-03-20T09:41:47.382 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:47 smithi115 ceph-mon[72476]: pgmap v2649: 47 pgs: 1 remapped+peering, 46 active+clean; 7.0 GiB data, 30 GiB used, 954 GiB / 983 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-03-20T09:41:47.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:47 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:41:47.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:47 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:41:47.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:47 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:41:47.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:47 smithi122 ceph-mon[73495]: Cluster is now healthy 2024-03-20T09:41:47.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:47 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:41:47.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:47 smithi122 ceph-mon[73495]: osdmap e1094: 12 total, 12 up, 11 in 2024-03-20T09:41:47.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:47 smithi122 ceph-mon[73495]: pgmap v2649: 47 pgs: 1 remapped+peering, 46 active+clean; 7.0 GiB data, 30 GiB used, 954 GiB / 983 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-03-20T09:41:47.481 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:41:47.524 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:47.560 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:41:47.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:47.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:41:47.668 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:41:47.705 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:47.738 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:41:47.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:47.745 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:41:47.832 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 68 16 117716 117700 108.14 71.1875 0.00838526 0.0092434 2024-03-20T09:41:47.846 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:41:47.902 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:48.163 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:41:48.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:48.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:41:48.268 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:41:48.315 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:48.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:48 smithi115 ceph-mon[72476]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-20T09:41:48.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:48 smithi115 ceph-mon[72476]: osdmap e1095: 12 total, 12 up, 11 in 2024-03-20T09:41:48.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:48 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:41:48.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:48 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:41:48.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:48 smithi122 ceph-mon[73495]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-20T09:41:48.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:48 smithi122 ceph-mon[73495]: osdmap e1095: 12 total, 12 up, 11 in 2024-03-20T09:41:48.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:48 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:41:48.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:48 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:41:48.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:48 smithi026 ceph-mon[68309]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-20T09:41:48.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:48 smithi026 ceph-mon[68309]: osdmap e1095: 12 total, 12 up, 11 in 2024-03-20T09:41:48.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:48 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:41:48.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:48 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:41:48.832 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 69 16 118748 118732 107.508 64.5 0.012446 0.0092972 2024-03-20T09:41:49.207 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 5, 6, 7, 8, 11, 3, 4, 9, 10] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 11, 5, 10] 2024-03-20T09:41:49.207 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:41:49.208 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-03-20T09:41:49.208 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-20T09:41:49.208 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-20T09:41:49.208 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-03-20T09:41:49.375 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:41:49.378 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:49 smithi026 ceph-mon[68309]: 9.2 scrub starts 2024-03-20T09:41:49.378 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:49 smithi026 ceph-mon[68309]: 9.b scrub starts 2024-03-20T09:41:49.378 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:49 smithi026 ceph-mon[68309]: osdmap e1096: 12 total, 12 up, 11 in 2024-03-20T09:41:49.378 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:49 smithi026 ceph-mon[68309]: pgmap v2652: 47 pgs: 1 remapped+peering, 46 active+clean; 7.0 GiB data, 30 GiB used, 954 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-03-20T09:41:49.378 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:49 smithi026 ceph-mon[68309]: 9.6 deep-scrub starts 2024-03-20T09:41:49.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:49.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:41:49.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:49 smithi115 ceph-mon[72476]: 9.2 scrub starts 2024-03-20T09:41:49.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:49 smithi115 ceph-mon[72476]: 9.b scrub starts 2024-03-20T09:41:49.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:49 smithi115 ceph-mon[72476]: osdmap e1096: 12 total, 12 up, 11 in 2024-03-20T09:41:49.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:49 smithi115 ceph-mon[72476]: pgmap v2652: 47 pgs: 1 remapped+peering, 46 active+clean; 7.0 GiB data, 30 GiB used, 954 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-03-20T09:41:49.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:49 smithi115 ceph-mon[72476]: 9.6 deep-scrub starts 2024-03-20T09:41:49.441 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:49 smithi122 ceph-mon[73495]: 9.2 scrub starts 2024-03-20T09:41:49.442 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:49 smithi122 ceph-mon[73495]: 9.b scrub starts 2024-03-20T09:41:49.442 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:49 smithi122 ceph-mon[73495]: osdmap e1096: 12 total, 12 up, 11 in 2024-03-20T09:41:49.442 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:49 smithi122 ceph-mon[73495]: pgmap v2652: 47 pgs: 1 remapped+peering, 46 active+clean; 7.0 GiB data, 30 GiB used, 954 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-03-20T09:41:49.442 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:49 smithi122 ceph-mon[73495]: 9.6 deep-scrub starts 2024-03-20T09:41:49.480 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:41:49.525 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:49.564 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:41:49.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:49.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:41:49.602 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:41:49.668 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:41:49.710 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:49.744 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:41:49.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:49.751 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:41:49.786 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:41:49.833 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 70 16 119492 119476 106.634 46.5 0.00428259 0.00937193 2024-03-20T09:41:49.853 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:41:49.889 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:49.918 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:41:49.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:49.922 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:41:50.023 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:41:50.093 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:50.127 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:41:50.131 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:50 smithi122 ceph-mon[73495]: 9.9 scrub starts 2024-03-20T09:41:50.131 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:50 smithi122 ceph-mon[73495]: 9.f scrub starts 2024-03-20T09:41:50.131 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:50 smithi122 ceph-mon[73495]: 9.2 scrub ok 2024-03-20T09:41:50.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:50.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:41:50.233 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:41:50.275 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:50 smithi115 ceph-mon[72476]: 9.9 scrub starts 2024-03-20T09:41:50.275 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:50 smithi115 ceph-mon[72476]: 9.f scrub starts 2024-03-20T09:41:50.275 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:50 smithi115 ceph-mon[72476]: 9.2 scrub ok 2024-03-20T09:41:50.282 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:50.442 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:50 smithi026 ceph-mon[68309]: 9.9 scrub starts 2024-03-20T09:41:50.442 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:50 smithi026 ceph-mon[68309]: 9.f scrub starts 2024-03-20T09:41:50.443 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:50 smithi026 ceph-mon[68309]: 9.2 scrub ok 2024-03-20T09:41:50.669 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:41:50.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:50.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:41:50.778 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:41:50.821 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:50.833 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 71 16 120316 120300 105.858 51.5 0.00864122 0.00944117 2024-03-20T09:41:50.856 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:41:50.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:50.865 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:41:50.966 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:41:51.008 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:51.041 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:41:51.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:51.044 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:41:51.078 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-20T09:41:51.131 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:51 smithi122 ceph-mon[73495]: 9.b scrub ok 2024-03-20T09:41:51.131 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:51 smithi122 ceph-mon[73495]: 1.1c scrub starts 2024-03-20T09:41:51.131 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:51 smithi122 ceph-mon[73495]: pgmap v2653: 47 pgs: 2 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 44 active+clean; 7.2 GiB data, 30 GiB used, 953 GiB / 983 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-03-20T09:41:51.145 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:41:51.171 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:51.191 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:41:51.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:51.194 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:41:51.295 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:41:51.296 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:51 smithi026 ceph-mon[68309]: 9.b scrub ok 2024-03-20T09:41:51.296 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:51 smithi026 ceph-mon[68309]: 1.1c scrub starts 2024-03-20T09:41:51.296 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:51 smithi026 ceph-mon[68309]: pgmap v2653: 47 pgs: 2 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 44 active+clean; 7.2 GiB data, 30 GiB used, 953 GiB / 983 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-03-20T09:41:51.361 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:51.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:51 smithi115 ceph-mon[72476]: 9.b scrub ok 2024-03-20T09:41:51.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:51 smithi115 ceph-mon[72476]: 1.1c scrub starts 2024-03-20T09:41:51.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:51 smithi115 ceph-mon[72476]: pgmap v2653: 47 pgs: 2 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 44 active+clean; 7.2 GiB data, 30 GiB used, 953 GiB / 983 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-03-20T09:41:51.397 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:41:51.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:51.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:41:51.504 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:41:51.541 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:51.705 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:41:51.748 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:41:51.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:51.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:41:51.766 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:41:51.833 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 72 16 121053 121037 105.028 46.0625 0.0175691 0.00951657 2024-03-20T09:41:51.858 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:41:51.923 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:51.943 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:41:51.943 INFO:teuthology.orchestra.run.smithi115.stdout: "success": "" 2024-03-20T09:41:51.943 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:41:51.959 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:41:51.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:51.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:41:52.068 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:41:52.103 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:52.131 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:41:52.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:52.135 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:41:52.177 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:41:52.213 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:41:52.235 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:41:52.240 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:41:52.271 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:41:52.277 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:52.301 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:41:52.332 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:52 smithi115 ceph-mon[72476]: 1.1c scrub ok 2024-03-20T09:41:52.332 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:52 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:41:52.332 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:52 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:41:52.332 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:52 smithi115 ceph-mon[72476]: 9.9 scrub ok 2024-03-20T09:41:52.332 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:52 smithi115 ceph-mon[72476]: 1.c scrub starts 2024-03-20T09:41:52.332 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:52 smithi115 ceph-mon[72476]: 1.c scrub ok 2024-03-20T09:41:52.332 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:52 smithi115 ceph-mon[72476]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-20T09:41:52.332 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:52 smithi115 ceph-mon[72476]: Cluster is now healthy 2024-03-20T09:41:52.332 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:52 smithi115 ceph-mon[72476]: osdmap e1097: 12 total, 12 up, 11 in 2024-03-20T09:41:52.332 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:52 smithi115 ceph-mon[72476]: 9.f scrub ok 2024-03-20T09:41:52.332 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:52 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/344882814' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:41:52.338 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:41:52.365 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:41:52.394 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:41:52.420 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:41:52.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:52 smithi122 ceph-mon[73495]: 1.1c scrub ok 2024-03-20T09:41:52.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:52 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:41:52.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:52 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:41:52.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:52 smithi122 ceph-mon[73495]: 9.9 scrub ok 2024-03-20T09:41:52.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:52 smithi122 ceph-mon[73495]: 1.c scrub starts 2024-03-20T09:41:52.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:52 smithi122 ceph-mon[73495]: 1.c scrub ok 2024-03-20T09:41:52.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:52 smithi122 ceph-mon[73495]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-20T09:41:52.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:52 smithi122 ceph-mon[73495]: Cluster is now healthy 2024-03-20T09:41:52.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:52 smithi122 ceph-mon[73495]: osdmap e1097: 12 total, 12 up, 11 in 2024-03-20T09:41:52.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:52 smithi122 ceph-mon[73495]: 9.f scrub ok 2024-03-20T09:41:52.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:52 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/344882814' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:41:52.455 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:41:52.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:52 smithi026 ceph-mon[68309]: 1.1c scrub ok 2024-03-20T09:41:52.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:52 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:41:52.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:52 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:41:52.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:52 smithi026 ceph-mon[68309]: 9.9 scrub ok 2024-03-20T09:41:52.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:52 smithi026 ceph-mon[68309]: 1.c scrub starts 2024-03-20T09:41:52.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:52 smithi026 ceph-mon[68309]: 1.c scrub ok 2024-03-20T09:41:52.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:52 smithi026 ceph-mon[68309]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-20T09:41:52.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:52 smithi026 ceph-mon[68309]: Cluster is now healthy 2024-03-20T09:41:52.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:52 smithi026 ceph-mon[68309]: osdmap e1097: 12 total, 12 up, 11 in 2024-03-20T09:41:52.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:52 smithi026 ceph-mon[68309]: 9.f scrub ok 2024-03-20T09:41:52.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:52 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/344882814' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:41:52.488 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:41:52.520 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:41:52.520 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:41:52.520 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:41:52.520 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:41:52.520 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:41:52.520 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:41:52.520 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:41:52.520 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:41:52.520 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:41:52.521 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:41:52.521 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:41:52.521 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:41:52.521 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:41:52.834 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 73 16 121958 121942 104.363 56.5625 0.024645 0.0095784 2024-03-20T09:41:53.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:53 smithi122 ceph-mon[73495]: pgmap v2655: 47 pgs: 2 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 44 active+clean; 7.3 GiB data, 31 GiB used, 953 GiB / 983 GiB avail; 57 MiB/s wr, 907 op/s 2024-03-20T09:41:53.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:53 smithi026 ceph-mon[68309]: pgmap v2655: 47 pgs: 2 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 44 active+clean; 7.3 GiB data, 31 GiB used, 953 GiB / 983 GiB avail; 57 MiB/s wr, 907 op/s 2024-03-20T09:41:53.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:53 smithi115 ceph-mon[72476]: pgmap v2655: 47 pgs: 2 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 44 active+clean; 7.3 GiB data, 31 GiB used, 953 GiB / 983 GiB avail; 57 MiB/s wr, 907 op/s 2024-03-20T09:41:53.834 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 74 16 122918 122902 103.764 60 0.0107794 0.00963184 2024-03-20T09:41:54.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:54 smithi122 ceph-mon[73495]: 1.d deep-scrub starts 2024-03-20T09:41:54.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:54 smithi122 ceph-mon[73495]: 1.d deep-scrub ok 2024-03-20T09:41:54.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:54 smithi122 ceph-mon[73495]: 1.7 scrub starts 2024-03-20T09:41:54.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:54 smithi122 ceph-mon[73495]: 1.7 scrub ok 2024-03-20T09:41:54.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:54 smithi026 ceph-mon[68309]: 1.d deep-scrub starts 2024-03-20T09:41:54.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:54 smithi026 ceph-mon[68309]: 1.d deep-scrub ok 2024-03-20T09:41:54.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:54 smithi026 ceph-mon[68309]: 1.7 scrub starts 2024-03-20T09:41:54.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:54 smithi026 ceph-mon[68309]: 1.7 scrub ok 2024-03-20T09:41:54.549 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:41:54.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:54.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:41:54.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:54 smithi115 ceph-mon[72476]: 1.d deep-scrub starts 2024-03-20T09:41:54.644 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:54 smithi115 ceph-mon[72476]: 1.d deep-scrub ok 2024-03-20T09:41:54.644 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:54 smithi115 ceph-mon[72476]: 1.7 scrub starts 2024-03-20T09:41:54.644 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:54 smithi115 ceph-mon[72476]: 1.7 scrub ok 2024-03-20T09:41:54.655 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:41:54.689 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:54.834 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 75 16 123585 123569 102.936 41.6875 0.00942062 0.00971025 2024-03-20T09:41:55.001 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:41:55.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:55.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:41:55.105 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:41:55.145 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:55.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:55 smithi122 ceph-mon[73495]: 9.5 scrub starts 2024-03-20T09:41:55.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:55 smithi122 ceph-mon[73495]: 9.1 deep-scrub starts 2024-03-20T09:41:55.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:55 smithi122 ceph-mon[73495]: 9.d scrub starts 2024-03-20T09:41:55.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:55 smithi122 ceph-mon[73495]: pgmap v2656: 47 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 45 active+clean; 7.4 GiB data, 31 GiB used, 953 GiB / 983 GiB avail; 53 MiB/s wr, 845 op/s 2024-03-20T09:41:55.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:55 smithi122 ceph-mon[73495]: 1.14 deep-scrub starts 2024-03-20T09:41:55.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:55 smithi122 ceph-mon[73495]: 1.14 deep-scrub ok 2024-03-20T09:41:55.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:55 smithi026 ceph-mon[68309]: 9.5 scrub starts 2024-03-20T09:41:55.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:55 smithi026 ceph-mon[68309]: 9.1 deep-scrub starts 2024-03-20T09:41:55.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:55 smithi026 ceph-mon[68309]: 9.d scrub starts 2024-03-20T09:41:55.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:55 smithi026 ceph-mon[68309]: pgmap v2656: 47 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 45 active+clean; 7.4 GiB data, 31 GiB used, 953 GiB / 983 GiB avail; 53 MiB/s wr, 845 op/s 2024-03-20T09:41:55.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:55 smithi026 ceph-mon[68309]: 1.14 deep-scrub starts 2024-03-20T09:41:55.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:55 smithi026 ceph-mon[68309]: 1.14 deep-scrub ok 2024-03-20T09:41:55.646 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:55 smithi115 ceph-mon[72476]: 9.5 scrub starts 2024-03-20T09:41:55.647 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:55 smithi115 ceph-mon[72476]: 9.1 deep-scrub starts 2024-03-20T09:41:55.647 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:55 smithi115 ceph-mon[72476]: 9.d scrub starts 2024-03-20T09:41:55.647 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:55 smithi115 ceph-mon[72476]: pgmap v2656: 47 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 45 active+clean; 7.4 GiB data, 31 GiB used, 953 GiB / 983 GiB avail; 53 MiB/s wr, 845 op/s 2024-03-20T09:41:55.647 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:55 smithi115 ceph-mon[72476]: 1.14 deep-scrub starts 2024-03-20T09:41:55.647 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:55 smithi115 ceph-mon[72476]: 1.14 deep-scrub ok 2024-03-20T09:41:55.834 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 76 16 124301 124285 102.171 44.75 0.00574417 0.00977786 2024-03-20T09:41:56.008 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:41:56.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:56.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:41:56.114 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:41:56.165 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:56.215 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:41:56.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:56.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:41:56.287 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:56 smithi026 ceph-mon[68309]: 9.5 scrub ok 2024-03-20T09:41:56.343 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:41:56.401 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:56 smithi115 ceph-mon[72476]: 9.5 scrub ok 2024-03-20T09:41:56.404 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:56.459 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:41:56.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:56.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:41:56.565 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:41:56.604 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:56.693 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:56 smithi122 ceph-mon[73495]: 9.5 scrub ok 2024-03-20T09:41:56.737 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:41:56.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:56.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:41:56.834 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 77 16 124975 124959 101.391 42.125 0.0250694 0.00985643 2024-03-20T09:41:56.841 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:41:56.894 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:56.934 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:41:56.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:56.940 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:41:57.034 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:41:57.041 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:41:57.085 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:57.112 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:41:57.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:57.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:41:57.216 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:41:57.288 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:57.316 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:41:57.316 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:41:57.316 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-03-20T09:41:57.316 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-03-20T09:41:57.316 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:41:57.326 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:41:57.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:57.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:41:57.332 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:57 smithi026 ceph-mon[68309]: 9.d scrub ok 2024-03-20T09:41:57.332 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:57 smithi026 ceph-mon[68309]: 9.8 deep-scrub starts 2024-03-20T09:41:57.332 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:57 smithi026 ceph-mon[68309]: pgmap v2657: 47 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 44 active+clean; 7.5 GiB data, 31 GiB used, 952 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-03-20T09:41:57.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:57 smithi115 ceph-mon[72476]: 9.d scrub ok 2024-03-20T09:41:57.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:57 smithi115 ceph-mon[72476]: 9.8 deep-scrub starts 2024-03-20T09:41:57.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:57 smithi115 ceph-mon[72476]: pgmap v2657: 47 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 44 active+clean; 7.5 GiB data, 31 GiB used, 952 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-03-20T09:41:57.432 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:41:57.432 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:41:57.472 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:57.684 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:57 smithi122 ceph-mon[73495]: 9.d scrub ok 2024-03-20T09:41:57.685 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:57 smithi122 ceph-mon[73495]: 9.8 deep-scrub starts 2024-03-20T09:41:57.685 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:57 smithi122 ceph-mon[73495]: pgmap v2657: 47 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 44 active+clean; 7.5 GiB data, 31 GiB used, 952 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-03-20T09:41:57.714 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 5, 6, 7, 8, 11, 3, 4, 9, 10] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 11, 5, 10] 2024-03-20T09:41:57.714 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:41:57.715 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-03-20T09:41:57.715 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-20T09:41:57.715 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-20T09:41:57.715 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-03-20T09:41:57.837 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 78 16 125497 125481 100.506 32.625 0.00661319 0.00993692 2024-03-20T09:41:58.337 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:41:58.644 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:58 smithi115 ceph-mon[72476]: from='client.27265 172.21.15.26:0/2751219495' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:41:58.644 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:58 smithi115 ceph-mon[72476]: from='client.27265 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:41:58.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:58 smithi122 ceph-mon[73495]: from='client.27265 172.21.15.26:0/2751219495' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:41:58.691 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:58 smithi122 ceph-mon[73495]: from='client.27265 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:41:58.708 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:58 smithi026 ceph-mon[68309]: from='client.27265 172.21.15.26:0/2751219495' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:41:58.708 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:58 smithi026 ceph-mon[68309]: from='client.27265 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:41:58.837 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 79 16 126068 126052 99.6862 35.6875 0.0174987 0.010026 2024-03-20T09:41:59.385 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:41:59.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:41:59.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:41:59.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:59 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:41:59.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:59 smithi026 ceph-mon[68309]: from='client.27265 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:41:59.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:59 smithi026 ceph-mon[68309]: osdmap e1098: 12 total, 12 up, 11 in 2024-03-20T09:41:59.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:41:59 smithi026 ceph-mon[68309]: pgmap v2659: 47 pgs: 1 active+clean+scrubbing, 3 active+clean+scrubbing+deep, 43 active+clean; 7.6 GiB data, 31 GiB used, 952 GiB / 983 GiB avail; 44 MiB/s wr, 707 op/s 2024-03-20T09:41:59.491 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:41:59.529 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:41:59.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:59 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:41:59.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:59 smithi115 ceph-mon[72476]: from='client.27265 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:41:59.644 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:59 smithi115 ceph-mon[72476]: osdmap e1098: 12 total, 12 up, 11 in 2024-03-20T09:41:59.644 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:41:59 smithi115 ceph-mon[72476]: pgmap v2659: 47 pgs: 1 active+clean+scrubbing, 3 active+clean+scrubbing+deep, 43 active+clean; 7.6 GiB data, 31 GiB used, 952 GiB / 983 GiB avail; 44 MiB/s wr, 707 op/s 2024-03-20T09:41:59.684 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:59 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:41:59.684 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:59 smithi122 ceph-mon[73495]: from='client.27265 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:41:59.684 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:59 smithi122 ceph-mon[73495]: osdmap e1098: 12 total, 12 up, 11 in 2024-03-20T09:41:59.684 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:41:59 smithi122 ceph-mon[73495]: pgmap v2659: 47 pgs: 1 active+clean+scrubbing, 3 active+clean+scrubbing+deep, 43 active+clean; 7.6 GiB data, 31 GiB used, 952 GiB / 983 GiB avail; 44 MiB/s wr, 707 op/s 2024-03-20T09:41:59.837 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:2024-03-20T09:41:59.850461+0000 min lat: 0.00113956 max lat: 0.324777 avg lat: 0.0101008 2024-03-20T09:41:59.837 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-20T09:41:59.837 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 80 16 126687 126671 98.9239 38.6875 0.0161503 0.0101008 2024-03-20T09:42:00.587 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:42:00.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:00.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:42:00.695 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:42:00.729 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:00.747 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:42:00.795 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-20T09:42:00.837 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 81 15 127276 127261 98.158 36.875 0.0107001 0.0101822 2024-03-20T09:42:01.009 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:42:01.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:01.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:42:01.120 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:42:01.168 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:01.195 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:42:01.198 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:42:01.207 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:01.207 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:42:01.308 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:42:01.345 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:01.380 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:42:01.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:01.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:42:01.484 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:42:01.531 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:01.579 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:42:01.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:01.585 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:42:01.601 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:42:01.631 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:42:01.655 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:42:01.681 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:42:01.686 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:42:01.707 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:42:01.722 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:01.746 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:42:01.780 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:42:01.837 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 82 16 127838 127822 97.3887 35.0625 0.0102018 0.0102626 2024-03-20T09:42:01.840 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:42:01.840 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:42:01.866 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:42:01.898 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:42:01.923 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:42:01.924 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:42:01.924 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:42:01.924 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:42:01.924 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:42:01.924 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:42:01.924 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:42:01.924 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:42:01.925 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:42:01.925 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:42:01.925 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:42:01.925 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:42:01.925 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:42:01.934 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:01 smithi122 ceph-mon[73495]: pgmap v2660: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 7.7 GiB data, 32 GiB used, 952 GiB / 983 GiB avail; 46 MiB/s wr, 734 op/s 2024-03-20T09:42:01.934 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:01 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/274258103' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:42:02.082 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:42:02.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:02.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:42:02.086 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:01 smithi026 ceph-mon[68309]: pgmap v2660: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 7.7 GiB data, 32 GiB used, 952 GiB / 983 GiB avail; 46 MiB/s wr, 734 op/s 2024-03-20T09:42:02.086 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:01 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/274258103' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:42:02.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:01 smithi115 ceph-mon[72476]: pgmap v2660: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 7.7 GiB data, 32 GiB used, 952 GiB / 983 GiB avail; 46 MiB/s wr, 734 op/s 2024-03-20T09:42:02.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:01 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/274258103' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:42:02.187 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:42:02.230 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:02.793 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:42:02.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:02.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:42:02.837 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 83 16 128407 128391 96.644 35.5625 0.0211222 0.0103393 2024-03-20T09:42:02.897 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:42:02.936 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:03.006 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:42:03.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:03.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:42:03.116 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:42:03.145 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:03.603 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:42:03.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:03.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:42:03.708 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:42:03.750 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:03.794 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:42:03.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:03.797 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:42:03.844 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 84 16 128949 128933 95.8895 33.875 0.00661697 0.0104235 2024-03-20T09:42:03.897 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:42:03.939 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:03.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:03 smithi026 ceph-mon[68309]: pgmap v2661: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 7.7 GiB data, 32 GiB used, 951 GiB / 983 GiB avail; 45 MiB/s wr, 712 op/s 2024-03-20T09:42:03.966 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:42:03.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:03.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:42:04.074 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:42:04.108 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:42:04.111 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:03 smithi115 ceph-mon[72476]: pgmap v2661: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 7.7 GiB data, 32 GiB used, 951 GiB / 983 GiB avail; 45 MiB/s wr, 712 op/s 2024-03-20T09:42:04.113 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:04.114 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:42:04.152 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:42:04.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:04.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:42:04.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:03 smithi122 ceph-mon[73495]: pgmap v2661: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 7.7 GiB data, 32 GiB used, 951 GiB / 983 GiB avail; 45 MiB/s wr, 712 op/s 2024-03-20T09:42:04.257 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:42:04.309 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:04.342 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:42:04.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:04.347 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:42:04.448 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:42:04.480 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:04.515 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:42:04.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:04.522 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:42:04.623 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:42:04.689 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:04.716 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:42:04.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:04.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:42:04.824 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:42:04.844 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 85 16 129515 129499 95.1778 35.375 0.0125118 0.0104988 2024-03-20T09:42:04.860 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:04.888 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:42:04.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:04.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:42:04.993 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:42:05.051 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:04 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:42:05.051 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:04 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:42:05.051 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:04 smithi115 ceph-mon[72476]: osdmap e1099: 12 total, 12 up, 11 in 2024-03-20T09:42:05.053 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:05.075 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:42:05.078 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:05.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:42:05.174 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:04 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:42:05.174 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:04 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:42:05.174 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:04 smithi026 ceph-mon[68309]: osdmap e1099: 12 total, 12 up, 11 in 2024-03-20T09:42:05.179 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:42:05.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:04 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:42:05.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:04 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:42:05.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:04 smithi122 ceph-mon[73495]: osdmap e1099: 12 total, 12 up, 11 in 2024-03-20T09:42:05.228 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:05.264 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:42:05.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:05.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:42:05.370 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:42:05.405 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:05.486 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:42:05.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:05.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:42:05.593 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:42:05.600 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:42:05.624 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:05.668 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:42:05.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:05.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:42:05.743 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:05 smithi026 ceph-mon[68309]: pgmap v2663: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 7.8 GiB data, 32 GiB used, 951 GiB / 983 GiB avail; 33 MiB/s wr, 532 op/s 2024-03-20T09:42:05.776 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:42:05.828 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:05.844 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 86 16 130082 130066 94.4834 35.4375 0.00766785 0.0105774 2024-03-20T09:42:05.863 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:42:05.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:05.868 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:42:05.893 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:05 smithi115 ceph-mon[72476]: pgmap v2663: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 7.8 GiB data, 32 GiB used, 951 GiB / 983 GiB avail; 33 MiB/s wr, 532 op/s 2024-03-20T09:42:05.969 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:42:06.010 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:06.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:05 smithi122 ceph-mon[73495]: pgmap v2663: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 7.8 GiB data, 32 GiB used, 951 GiB / 983 GiB avail; 33 MiB/s wr, 532 op/s 2024-03-20T09:42:06.672 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:42:06.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:06.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:42:06.776 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:42:06.814 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:06.845 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 87 16 130633 130617 93.7919 34.4375 0.0170531 0.0106546 2024-03-20T09:42:06.847 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:42:06.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:06.854 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:42:06.954 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:42:06.986 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:07.015 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:42:07.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:07.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:42:07.121 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:42:07.177 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:07.205 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:42:07.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:07.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:42:07.311 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:42:07.353 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:07.561 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:42:07.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:07.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:42:07.676 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:42:07.721 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:07.759 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:42:07.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:07.768 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:42:07.846 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 88 16 131174 131158 93.1105 33.8125 0.0124223 0.0107338 2024-03-20T09:42:07.869 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:42:07.906 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:07 smithi026 ceph-mon[68309]: pgmap v2664: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 7.9 GiB data, 32 GiB used, 951 GiB / 983 GiB avail; 43 MiB/s wr, 695 op/s 2024-03-20T09:42:07.910 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:08.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:07 smithi115 ceph-mon[72476]: pgmap v2664: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 7.9 GiB data, 32 GiB used, 951 GiB / 983 GiB avail; 43 MiB/s wr, 695 op/s 2024-03-20T09:42:08.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:07 smithi122 ceph-mon[73495]: pgmap v2664: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 7.9 GiB data, 32 GiB used, 951 GiB / 983 GiB avail; 43 MiB/s wr, 695 op/s 2024-03-20T09:42:08.742 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:42:08.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:08.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:42:08.846 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:42:08.847 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 89 16 131693 131677 92.429 32.4375 0.0161365 0.010811 2024-03-20T09:42:08.895 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:09.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:08 smithi026 ceph-mon[68309]: pgmap v2665: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 7.9 GiB data, 33 GiB used, 951 GiB / 983 GiB avail; 35 MiB/s wr, 554 op/s 2024-03-20T09:42:09.337 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:42:09.338 INFO:teuthology.orchestra.run.smithi026.stdout: "success": "" 2024-03-20T09:42:09.338 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:42:09.358 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:42:09.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:09.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:42:09.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:08 smithi115 ceph-mon[72476]: pgmap v2665: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 7.9 GiB data, 33 GiB used, 951 GiB / 983 GiB avail; 35 MiB/s wr, 554 op/s 2024-03-20T09:42:09.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:08 smithi122 ceph-mon[73495]: pgmap v2665: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 7.9 GiB data, 33 GiB used, 951 GiB / 983 GiB avail; 35 MiB/s wr, 554 op/s 2024-03-20T09:42:09.464 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:42:09.501 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:09.558 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:42:09.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:09.563 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:42:09.603 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:42:09.664 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:42:09.735 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:09.761 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:42:09.769 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:09.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:42:09.846 INFO:tasks.radosbench.radosbench.2.smithi122.stdout: 90 9 132227 132218 91.7779 33.8125 0.107945 0.0108876 2024-03-20T09:42:09.870 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:42:09.900 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:09.907 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:42:09.907 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:42:09.930 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:42:09.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:09.934 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:42:09.967 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:42:10.035 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:42:10.061 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:10.079 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:42:10.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:10.082 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:42:10.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:09 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1150339273' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:42:10.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:09 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:42:10.183 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:42:10.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:09 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1150339273' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:42:10.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:09 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:42:10.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:09 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1150339273' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:42:10.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:09 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:42:10.214 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:10.238 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:42:10.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:10.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:42:10.342 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:42:10.368 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:10.387 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:42:10.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:10.390 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:42:10.491 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:42:10.517 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:10.535 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:42:10.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:10.538 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:42:10.639 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:42:10.667 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:10.687 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:42:10.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:10.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:42:10.790 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:42:10.817 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:10.836 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:42:10.838 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:10.839 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:42:10.846 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Total time run: 90.1755 2024-03-20T09:42:10.846 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Total writes made: 132227 2024-03-20T09:42:10.846 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Write size: 65536 2024-03-20T09:42:10.846 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Object size: 65536 2024-03-20T09:42:10.847 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Bandwidth (MB/sec): 91.6456 2024-03-20T09:42:10.847 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Stddev Bandwidth: 46.3547 2024-03-20T09:42:10.847 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Max bandwidth (MB/sec): 267.438 2024-03-20T09:42:10.847 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Min bandwidth (MB/sec): 32.4375 2024-03-20T09:42:10.847 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Average IOPS: 1466 2024-03-20T09:42:10.847 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Stddev IOPS: 741.676 2024-03-20T09:42:10.847 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Max IOPS: 4279 2024-03-20T09:42:10.847 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Min IOPS: 519 2024-03-20T09:42:10.847 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Average Latency(s): 0.0108936 2024-03-20T09:42:10.847 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Stddev Latency(s): 0.014044 2024-03-20T09:42:10.847 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Max latency(s): 0.324777 2024-03-20T09:42:10.847 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Min latency(s): 0.00113956 2024-03-20T09:42:10.854 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Cleaning up (deleting benchmark objects) 2024-03-20T09:42:10.940 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:42:10.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:10 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:42:10.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:10 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:42:10.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:10 smithi115 ceph-mon[72476]: osdmap e1100: 12 total, 12 up, 11 in 2024-03-20T09:42:10.976 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:10 smithi115 ceph-mon[72476]: pgmap v2667: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 8.0 GiB data, 33 GiB used, 951 GiB / 983 GiB avail; 38 MiB/s wr, 601 op/s 2024-03-20T09:42:10.991 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:11.027 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:42:11.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:11.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:42:11.132 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:42:11.181 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:10 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:42:11.181 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:10 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:42:11.181 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:10 smithi122 ceph-mon[73495]: osdmap e1100: 12 total, 12 up, 11 in 2024-03-20T09:42:11.181 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:10 smithi122 ceph-mon[73495]: pgmap v2667: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 8.0 GiB data, 33 GiB used, 951 GiB / 983 GiB avail; 38 MiB/s wr, 601 op/s 2024-03-20T09:42:11.184 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:11.215 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:42:11.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:11.221 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:42:11.323 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:42:11.357 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:10 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:42:11.357 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:10 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:42:11.357 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:10 smithi026 ceph-mon[68309]: osdmap e1100: 12 total, 12 up, 11 in 2024-03-20T09:42:11.357 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:10 smithi026 ceph-mon[68309]: pgmap v2667: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 8.0 GiB data, 33 GiB used, 951 GiB / 983 GiB avail; 38 MiB/s wr, 601 op/s 2024-03-20T09:42:11.357 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:12.479 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-20T09:42:13.123 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:42:13.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:13.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:42:13.235 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:42:13.278 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:13.308 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:42:13.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:13.312 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:42:13.413 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:42:13.433 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:42:13.470 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:13.494 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:42:13.501 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:42:13.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:13.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:42:13.528 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:42:13.552 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:42:13.582 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:42:13.607 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:42:13.619 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:42:13.647 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:13.656 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:42:13.677 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:42:13.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:13.681 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:42:13.686 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:42:13.730 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:13 smithi026 ceph-mon[68309]: pgmap v2668: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 8.1 GiB data, 33 GiB used, 950 GiB / 983 GiB avail; 35 MiB/s wr, 564 op/s 2024-03-20T09:42:13.730 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:13 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/297533996' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:42:13.734 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:42:13.776 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:42:13.782 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:42:13.815 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:42:13.820 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:13.851 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:42:13.853 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:42:13.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:13.855 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:42:13.879 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:42:13.887 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:42:13.887 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:42:13.887 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:42:13.887 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:42:13.887 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:42:13.887 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:42:13.887 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:42:13.887 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:42:13.887 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:42:13.887 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:42:13.887 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:42:13.888 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:42:13.934 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:13 smithi122 ceph-mon[73495]: pgmap v2668: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 8.1 GiB data, 33 GiB used, 950 GiB / 983 GiB avail; 35 MiB/s wr, 564 op/s 2024-03-20T09:42:13.934 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:13 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/297533996' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:42:13.956 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:42:14.010 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:14.037 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:42:14.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:14.040 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:42:14.141 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:42:14.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:13 smithi115 ceph-mon[72476]: pgmap v2668: 47 pgs: 3 active+clean+scrubbing+deep, 44 active+clean; 8.1 GiB data, 33 GiB used, 950 GiB / 983 GiB avail; 35 MiB/s wr, 564 op/s 2024-03-20T09:42:14.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:13 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/297533996' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:42:14.195 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:14.227 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:42:14.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:14.236 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:42:14.337 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:42:14.375 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:14.405 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:42:14.407 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:14.408 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:42:14.509 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:42:14.559 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:14.589 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:42:14.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:14.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:42:14.693 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:42:14.744 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:14.776 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:42:14.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:14.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:42:14.879 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:42:14.945 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:15.011 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:42:15.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:15.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:42:15.123 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:42:15.144 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:42:15.159 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:15.524 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:42:15.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:15.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:42:15.567 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:42:15.632 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:42:15.679 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:15.706 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:42:15.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:15.714 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:42:15.717 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 5, 6, 7, 8, 11, 3, 4, 9, 10] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 11, 5, 10] 2024-03-20T09:42:15.717 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 1 1 2024-03-20T09:42:15.753 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:15 smithi026 ceph-mon[68309]: pgmap v2669: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 8.0 GiB data, 33 GiB used, 951 GiB / 983 GiB avail; 28 MiB/s wr, 567 op/s 2024-03-20T09:42:15.816 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:42:15.853 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:15.899 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:42:15.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:15.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:42:15.935 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:15 smithi122 ceph-mon[73495]: pgmap v2669: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 8.0 GiB data, 33 GiB used, 951 GiB / 983 GiB avail; 28 MiB/s wr, 567 op/s 2024-03-20T09:42:16.003 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:42:16.051 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:15 smithi115 ceph-mon[72476]: pgmap v2669: 47 pgs: 2 active+clean+scrubbing+deep, 45 active+clean; 8.0 GiB data, 33 GiB used, 951 GiB / 983 GiB avail; 28 MiB/s wr, 567 op/s 2024-03-20T09:42:16.052 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:16.097 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:42:16.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:16.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:42:16.186 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:42:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:42:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.27190.0:137810 9.1 9:86e30eeb:::benchmark_data_smithi122_210164_object5580:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1100)", 2024-03-20T09:42:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:42:16.189366+0000", 2024-03-20T09:42:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.0089855109999999998, 2024-03-20T09:42:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0090135890000000007, 2024-03-20T09:42:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:42:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:42:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:42:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.27190", 2024-03-20T09:42:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:42:16.187 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 137810 2024-03-20T09:42:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:42:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:42:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.189366+0000", 2024-03-20T09:42:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:42:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.189366+0000", 2024-03-20T09:42:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:42:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.189367+0000", 2024-03-20T09:42:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1489999999999999e-06 2024-03-20T09:42:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:42:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.189375+0000", 2024-03-20T09:42:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0490000000000004e-06 2024-03-20T09:42:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:42:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.189387+0000", 2024-03-20T09:42:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1421e-05 2024-03-20T09:42:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:42:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.189394+0000", 2024-03-20T09:42:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2749999999999998e-06 2024-03-20T09:42:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:42:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.189464+0000", 2024-03-20T09:42:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0328999999999997e-05 2024-03-20T09:42:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:42:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.189555+0000", 2024-03-20T09:42:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0989000000000006e-05 2024-03-20T09:42:16.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 6,9", 2024-03-20T09:42:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.189630+0000", 2024-03-20T09:42:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5001999999999998e-05 2024-03-20T09:42:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.197868+0000", 2024-03-20T09:42:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0082376629999999992 2024-03-20T09:42:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:42:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.197928+0000", 2024-03-20T09:42:16.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0111000000000001e-05 2024-03-20T09:42:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:42:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.27190.0:137816 9.1 9:8de3f8b8:::benchmark_data_smithi122_210164_object5586:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1100)", 2024-03-20T09:42:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:42:16.191244+0000", 2024-03-20T09:42:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.007107042, 2024-03-20T09:42:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0072111379999999997, 2024-03-20T09:42:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:42:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:42:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:42:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.27190", 2024-03-20T09:42:16.192 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:42:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 137816 2024-03-20T09:42:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:42:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:42:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.191244+0000", 2024-03-20T09:42:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:42:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.191244+0000", 2024-03-20T09:42:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:42:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.191245+0000", 2024-03-20T09:42:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2500000000000005e-07 2024-03-20T09:42:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:42:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.191249+0000", 2024-03-20T09:42:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9530000000000001e-06 2024-03-20T09:42:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:42:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.191258+0000", 2024-03-20T09:42:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2220000000000008e-06 2024-03-20T09:42:16.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:42:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.191264+0000", 2024-03-20T09:42:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0789999999999997e-06 2024-03-20T09:42:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:42:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.193323+0000", 2024-03-20T09:42:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0020584380000000001 2024-03-20T09:42:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:42:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.193406+0000", 2024-03-20T09:42:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3682000000000003e-05 2024-03-20T09:42:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 6,9", 2024-03-20T09:42:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.193495+0000", 2024-03-20T09:42:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8275000000000004e-05 2024-03-20T09:42:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.197915+0000", 2024-03-20T09:42:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.004420322 2024-03-20T09:42:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:42:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.197937+0000", 2024-03-20T09:42:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2413999999999999e-05 2024-03-20T09:42:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:42:16.196 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.27190.0:137818 9.7 e1100/1066 9:e0613c1b:::benchmark_data_smithi122_210164_object5588:head v 1100'8779, mlcod=1100'8777)", 2024-03-20T09:42:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:42:16.198037+0000", 2024-03-20T09:42:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 0.00031460599999999999, 2024-03-20T09:42:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000479944, 2024-03-20T09:42:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:42:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:42:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:42:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:42:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.198037+0000", 2024-03-20T09:42:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.197 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:42:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.198037+0000", 2024-03-20T09:42:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:42:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.198037+0000", 2024-03-20T09:42:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9299999999999998e-07 2024-03-20T09:42:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:42:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.198042+0000", 2024-03-20T09:42:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6979999999999997e-06 2024-03-20T09:42:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:42:16.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.198051+0000", 2024-03-20T09:42:16.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4619999999999996e-06 2024-03-20T09:42:16.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:42:16.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.198055+0000", 2024-03-20T09:42:16.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7260000000000002e-06 2024-03-20T09:42:16.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:42:16.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.198103+0000", 2024-03-20T09:42:16.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7771000000000001e-05 2024-03-20T09:42:16.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:42:16.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.198123+0000", 2024-03-20T09:42:16.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9482000000000001e-05 2024-03-20T09:42:16.200 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:16.200 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:42:16.200 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:16.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.200 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop_reply(client.27190.0:137810 9.1 e1100/1066 ondisk, result = 0)", 2024-03-20T09:42:16.200 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:42:16.198325+0000", 2024-03-20T09:42:16.200 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 2.6322999999999999e-05, 2024-03-20T09:42:16.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00024524600000000002, 2024-03-20T09:42:16.200 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:42:16.200 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:42:16.200 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:42:16.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:42:16.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.198325+0000", 2024-03-20T09:42:16.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:16.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:42:16.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.198325+0000", 2024-03-20T09:42:16.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:16.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:42:16.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.198326+0000", 2024-03-20T09:42:16.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2399999999999997e-07 2024-03-20T09:42:16.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:42:16.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.198335+0000", 2024-03-20T09:42:16.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0410000000000003e-06 2024-03-20T09:42:16.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:42:16.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.198347+0000", 2024-03-20T09:42:16.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2257e-05 2024-03-20T09:42:16.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:42:16.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.198354+0000", 2024-03-20T09:42:16.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8020000000000003e-06 2024-03-20T09:42:16.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:42:16.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.198408+0000", 2024-03-20T09:42:16.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4308000000000002e-05 2024-03-20T09:42:16.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:42:16.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.198418+0000", 2024-03-20T09:42:16.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.302e-06 2024-03-20T09:42:16.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:42:16.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.198571+0000", 2024-03-20T09:42:16.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015327599999999999 2024-03-20T09:42:16.204 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:16.204 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:42:16.204 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:16.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.204 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.27190.0:137822 9.1 9:84e564cf:::benchmark_data_smithi122_210164_object5592:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1100)", 2024-03-20T09:42:16.204 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:42:16.198339+0000", 2024-03-20T09:42:16.204 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 1.2850999999999999e-05, 2024-03-20T09:42:16.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00031326800000000001, 2024-03-20T09:42:16.204 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:42:16.204 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "reached pg", 2024-03-20T09:42:16.204 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:42:16.205 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.27190", 2024-03-20T09:42:16.205 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:42:16.205 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 137822 2024-03-20T09:42:16.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.205 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:42:16.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:42:16.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.198339+0000", 2024-03-20T09:42:16.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:16.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:42:16.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.198339+0000", 2024-03-20T09:42:16.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:16.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:42:16.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.198340+0000", 2024-03-20T09:42:16.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.252e-06 2024-03-20T09:42:16.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:42:16.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.198347+0000", 2024-03-20T09:42:16.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1760000000000004e-06 2024-03-20T09:42:16.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:42:16.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.198360+0000", 2024-03-20T09:42:16.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2975e-05 2024-03-20T09:42:16.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:42:16.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.198369+0000", 2024-03-20T09:42:16.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5269999999999998e-06 2024-03-20T09:42:16.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:42:16.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.198603+0000", 2024-03-20T09:42:16.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00023420599999999999 2024-03-20T09:42:16.207 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:16.207 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:42:16.207 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:16.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.208 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.27190.0:137824 9.1 9.ad1f8e61 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1100)", 2024-03-20T09:42:16.208 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:42:16.198383+0000", 2024-03-20T09:42:16.208 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 4294967295.999969, 2024-03-20T09:42:16.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000336046, 2024-03-20T09:42:16.208 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:42:16.208 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "queued for pg", 2024-03-20T09:42:16.208 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:42:16.208 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.27190", 2024-03-20T09:42:16.208 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:42:16.208 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 137824 2024-03-20T09:42:16.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.208 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:42:16.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:42:16.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.198383+0000", 2024-03-20T09:42:16.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:16.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:42:16.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.198383+0000", 2024-03-20T09:42:16.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:16.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:42:16.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.198383+0000", 2024-03-20T09:42:16.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5499999999999998e-07 2024-03-20T09:42:16.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:42:16.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.198386+0000", 2024-03-20T09:42:16.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.05e-06 2024-03-20T09:42:16.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:42:16.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.198407+0000", 2024-03-20T09:42:16.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1081e-05 2024-03-20T09:42:16.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:16.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:16.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:42:16.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:16.198415+0000", 2024-03-20T09:42:16.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4719999999999998e-06 2024-03-20T09:42:16.210 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:16.211 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:42:16.211 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:16.211 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:16.211 INFO:teuthology.orchestra.run.smithi026.stdout: ], 2024-03-20T09:42:16.211 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 6 2024-03-20T09:42:16.211 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:42:16.211 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:42:16.246 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:17.491 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:42:17.960 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:17 smithi026 ceph-mon[68309]: pgmap v2670: 47 pgs: 47 active+clean; 7.8 GiB data, 32 GiB used, 951 GiB / 983 GiB avail; 102 B/s rd, 17 MiB/s wr, 687 op/s 2024-03-20T09:42:18.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:17 smithi115 ceph-mon[72476]: pgmap v2670: 47 pgs: 47 active+clean; 7.8 GiB data, 32 GiB used, 951 GiB / 983 GiB avail; 102 B/s rd, 17 MiB/s wr, 687 op/s 2024-03-20T09:42:18.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:17 smithi122 ceph-mon[73495]: pgmap v2670: 47 pgs: 47 active+clean; 7.8 GiB data, 32 GiB used, 951 GiB / 983 GiB avail; 102 B/s rd, 17 MiB/s wr, 687 op/s 2024-03-20T09:42:18.524 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:42:18.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:18.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:42:18.631 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:42:18.681 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:18.718 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:42:18.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:18.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:42:18.806 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:18 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:42:18.807 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:18 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:42:18.807 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:18 smithi026 ceph-mon[68309]: osdmap e1101: 12 total, 12 up, 11 in 2024-03-20T09:42:18.807 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:18 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:42:18.807 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:18 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:42:18.807 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:18 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:42:18.807 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:18 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:42:18.807 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:18 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:42:18.807 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:18 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:42:18.807 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:18 smithi026 ceph-mon[68309]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:42:18.826 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:42:18.866 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:18.998 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:42:19.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:19.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:42:19.104 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:42:19.137 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:19.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:18 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:42:19.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:18 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:42:19.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:18 smithi115 ceph-mon[72476]: osdmap e1101: 12 total, 12 up, 11 in 2024-03-20T09:42:19.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:18 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:42:19.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:18 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:42:19.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:18 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:42:19.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:18 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:42:19.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:18 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:42:19.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:18 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:42:19.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:18 smithi115 ceph-mon[72476]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:42:19.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:18 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:42:19.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:18 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:42:19.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:18 smithi122 ceph-mon[73495]: osdmap e1101: 12 total, 12 up, 11 in 2024-03-20T09:42:19.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:18 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:42:19.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:18 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:42:19.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:18 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:42:19.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:18 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:42:19.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:18 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:42:19.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:18 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:42:19.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:18 smithi122 ceph-mon[73495]: from='mgr.14148 ' entity='mgr.y' 2024-03-20T09:42:19.583 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-20T09:42:19.700 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:42:19.705 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:42:19.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:19.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:42:19.712 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:42:19.811 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:42:19.861 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:19 smithi026 ceph-mon[68309]: pgmap v2672: 47 pgs: 47 active+clean; 7.8 GiB data, 32 GiB used, 951 GiB / 983 GiB avail; 117 B/s rd, 12 MiB/s wr, 711 op/s 2024-03-20T09:42:19.862 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:20.114 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:42:20.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:20.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:42:20.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:19 smithi115 ceph-mon[72476]: pgmap v2672: 47 pgs: 47 active+clean; 7.8 GiB data, 32 GiB used, 951 GiB / 983 GiB avail; 117 B/s rd, 12 MiB/s wr, 711 op/s 2024-03-20T09:42:20.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:19 smithi122 ceph-mon[73495]: pgmap v2672: 47 pgs: 47 active+clean; 7.8 GiB data, 32 GiB used, 951 GiB / 983 GiB avail; 117 B/s rd, 12 MiB/s wr, 711 op/s 2024-03-20T09:42:20.229 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:42:20.277 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:20.694 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:42:20.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:20.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:42:20.774 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:42:20.804 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:42:20.852 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:20.889 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:42:20.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:20.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:42:20.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:20 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3247327474' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:42:21.001 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:42:21.041 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:20 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3247327474' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:42:21.042 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:21.074 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:42:21.084 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:21.084 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:42:21.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:20 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3247327474' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:42:21.185 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:42:21.240 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:21.276 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:42:21.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:21.283 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:42:21.384 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:42:21.438 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:21.473 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:42:21.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:21.479 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:42:21.580 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:42:21.618 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:21.702 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:42:21.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:21.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:42:21.769 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:21 smithi122 ceph-mon[73495]: pgmap v2673: 47 pgs: 47 active+clean; 7.6 GiB data, 32 GiB used, 952 GiB / 983 GiB avail; 102 B/s rd, 4.1 MiB/s wr, 890 op/s 2024-03-20T09:42:21.769 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:21 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:42:21.769 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:21 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3247327474' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:42:21.769 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:21 smithi122 ceph-mon[73495]: osdmap e1102: 12 total, 12 up, 11 in 2024-03-20T09:42:21.812 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:42:21.845 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:21.870 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:42:21.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:21.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:42:21.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:21 smithi026 ceph-mon[68309]: pgmap v2673: 47 pgs: 47 active+clean; 7.6 GiB data, 32 GiB used, 952 GiB / 983 GiB avail; 102 B/s rd, 4.1 MiB/s wr, 890 op/s 2024-03-20T09:42:21.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:21 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:42:21.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:21 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3247327474' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:42:21.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:21 smithi026 ceph-mon[68309]: osdmap e1102: 12 total, 12 up, 11 in 2024-03-20T09:42:21.973 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:42:22.010 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:22.091 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:42:22.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:22.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:42:22.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:21 smithi115 ceph-mon[72476]: pgmap v2673: 47 pgs: 47 active+clean; 7.6 GiB data, 32 GiB used, 952 GiB / 983 GiB avail; 102 B/s rd, 4.1 MiB/s wr, 890 op/s 2024-03-20T09:42:22.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:21 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:42:22.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:21 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3247327474' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:42:22.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:21 smithi115 ceph-mon[72476]: osdmap e1102: 12 total, 12 up, 11 in 2024-03-20T09:42:22.196 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:42:22.254 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:22.290 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:42:22.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:22.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:42:22.399 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:42:22.441 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:22.470 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:42:22.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:22.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:42:22.578 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:42:22.640 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:22.672 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:42:22.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:22.677 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:42:22.778 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:42:22.834 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:22.862 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:42:22.866 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:22.866 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:42:22.967 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:42:23.031 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:23.193 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:42:23.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:23.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:42:23.297 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:42:23.346 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:23.385 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:42:23.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:23.388 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:42:23.489 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:42:23.537 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:23.606 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:42:23.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:23.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:42:23.711 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:42:23.766 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:23.776 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:42:23.805 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:42:23.817 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:42:23.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:23.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:42:23.829 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:42:23.859 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:42:23.893 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:42:23.925 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:42:23.928 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:42:23.962 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:42:23.975 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:42:23.999 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:24.016 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:42:24.055 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:42:24.097 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:42:24.121 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:42:24.135 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:42:24.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:23 smithi115 ceph-mon[72476]: pgmap v2675: 47 pgs: 47 active+clean; 7.5 GiB data, 31 GiB used, 952 GiB / 983 GiB avail; 127 B/s rd, 127 B/s wr, 1.05k op/s 2024-03-20T09:42:24.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:23 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:42:24.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:23 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:42:24.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:23 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2544271138' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:42:24.168 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:42:24.170 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:23 smithi026 ceph-mon[68309]: pgmap v2675: 47 pgs: 47 active+clean; 7.5 GiB data, 31 GiB used, 952 GiB / 983 GiB avail; 127 B/s rd, 127 B/s wr, 1.05k op/s 2024-03-20T09:42:24.170 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:23 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:42:24.170 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:23 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:42:24.170 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:23 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2544271138' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:42:24.176 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:42:24.176 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:42:24.177 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:42:24.177 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:42:24.177 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:42:24.177 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:42:24.177 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:42:24.177 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:42:24.177 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:42:24.177 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:42:24.177 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:42:24.177 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:42:24.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:23 smithi122 ceph-mon[73495]: pgmap v2675: 47 pgs: 47 active+clean; 7.5 GiB data, 31 GiB used, 952 GiB / 983 GiB avail; 127 B/s rd, 127 B/s wr, 1.05k op/s 2024-03-20T09:42:24.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:23 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:42:24.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:23 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:42:24.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:23 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2544271138' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:42:24.265 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:42:24.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:24.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:42:24.378 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:42:24.438 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:24.465 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:42:24.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:24.470 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:42:24.571 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:42:24.623 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:24.649 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:42:24.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:24.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:42:24.756 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:42:24.798 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:24.810 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.1 to 1 (10000) 2024-03-20T09:42:24.825 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:42:24.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:24.828 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:42:24.893 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:24 smithi115 ceph-mon[72476]: osdmap e1103: 12 total, 12 up, 11 in 2024-03-20T09:42:24.893 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:24 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1739041043' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-20T09:42:24.929 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:42:24.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:24 smithi026 ceph-mon[68309]: osdmap e1103: 12 total, 12 up, 11 in 2024-03-20T09:42:24.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:24 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1739041043' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-20T09:42:24.975 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:25.003 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:42:25.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:25.009 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:42:25.110 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:42:25.152 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:25.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:24 smithi122 ceph-mon[73495]: osdmap e1103: 12 total, 12 up, 11 in 2024-03-20T09:42:25.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:24 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1739041043' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-20T09:42:25.191 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:42:25.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:25.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:42:25.297 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:42:25.336 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:26.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:25 smithi115 ceph-mon[72476]: pgmap v2677: 47 pgs: 47 active+clean; 7.4 GiB data, 31 GiB used, 952 GiB / 983 GiB avail; 0 B/s wr, 963 op/s 2024-03-20T09:42:26.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:25 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1739041043' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-20T09:42:26.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:25 smithi115 ceph-mon[72476]: osdmap e1104: 12 total, 12 up, 11 in 2024-03-20T09:42:26.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:25 smithi115 ceph-mon[72476]: 9.6 scrub starts 2024-03-20T09:42:26.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:25 smithi122 ceph-mon[73495]: pgmap v2677: 47 pgs: 47 active+clean; 7.4 GiB data, 31 GiB used, 952 GiB / 983 GiB avail; 0 B/s wr, 963 op/s 2024-03-20T09:42:26.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:25 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1739041043' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-20T09:42:26.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:25 smithi122 ceph-mon[73495]: osdmap e1104: 12 total, 12 up, 11 in 2024-03-20T09:42:26.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:25 smithi122 ceph-mon[73495]: 9.6 scrub starts 2024-03-20T09:42:26.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:25 smithi026 ceph-mon[68309]: pgmap v2677: 47 pgs: 47 active+clean; 7.4 GiB data, 31 GiB used, 952 GiB / 983 GiB avail; 0 B/s wr, 963 op/s 2024-03-20T09:42:26.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:25 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1739041043' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-20T09:42:26.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:25 smithi026 ceph-mon[68309]: osdmap e1104: 12 total, 12 up, 11 in 2024-03-20T09:42:26.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:25 smithi026 ceph-mon[68309]: 9.6 scrub starts 2024-03-20T09:42:26.294 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:42:26.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:26.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:42:26.399 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:42:26.445 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:26.481 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:42:26.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:26.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:42:26.590 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:42:26.638 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:26.667 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:42:26.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:26.671 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:42:26.772 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:42:26.826 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:26.935 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:26 smithi122 ceph-mon[73495]: 1.1d scrub starts 2024-03-20T09:42:26.935 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:26 smithi122 ceph-mon[73495]: 1.1d scrub ok 2024-03-20T09:42:26.957 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:42:26.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:26.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:42:27.063 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:42:27.084 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:26 smithi026 ceph-mon[68309]: 1.1d scrub starts 2024-03-20T09:42:27.084 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:26 smithi026 ceph-mon[68309]: 1.1d scrub ok 2024-03-20T09:42:27.112 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:27.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:26 smithi115 ceph-mon[72476]: 1.1d scrub starts 2024-03-20T09:42:27.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:26 smithi115 ceph-mon[72476]: 1.1d scrub ok 2024-03-20T09:42:27.146 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:42:27.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:27.151 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:42:27.252 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:42:27.305 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:27.857 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 2 1 2024-03-20T09:42:28.101 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:27 smithi026 ceph-mon[68309]: pgmap v2679: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 7.2 GiB data, 31 GiB used, 953 GiB / 983 GiB avail; 0 B/s wr, 969 op/s 2024-03-20T09:42:28.101 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:27 smithi026 ceph-mon[68309]: 9.6 scrub ok 2024-03-20T09:42:28.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:27 smithi115 ceph-mon[72476]: pgmap v2679: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 7.2 GiB data, 31 GiB used, 953 GiB / 983 GiB avail; 0 B/s wr, 969 op/s 2024-03-20T09:42:28.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:27 smithi115 ceph-mon[72476]: 9.6 scrub ok 2024-03-20T09:42:28.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:27 smithi122 ceph-mon[73495]: pgmap v2679: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 7.2 GiB data, 31 GiB used, 953 GiB / 983 GiB avail; 0 B/s wr, 969 op/s 2024-03-20T09:42:28.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:27 smithi122 ceph-mon[73495]: 9.6 scrub ok 2024-03-20T09:42:28.275 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:42:28.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:28.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:42:28.381 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:42:28.419 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:28.442 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:42:28.455 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:42:28.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:28.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:42:28.566 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:42:28.603 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:28.645 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:42:28.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:28.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:42:28.705 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:42:28.705 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:42:28.754 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:42:28.804 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:28.839 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:42:28.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:28.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:42:28.945 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:42:28.979 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:29.010 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:42:29.014 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:28 smithi026 ceph-mon[68309]: 1.15 scrub starts 2024-03-20T09:42:29.014 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:28 smithi026 ceph-mon[68309]: pgmap v2680: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 7.2 GiB data, 30 GiB used, 953 GiB / 983 GiB avail; 0 B/s wr, 772 op/s 2024-03-20T09:42:29.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:29.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:42:29.115 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:42:29.116 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:42:29.160 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:29.198 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:42:29.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:29.210 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:42:29.310 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:42:29.346 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:28 smithi115 ceph-mon[72476]: 1.15 scrub starts 2024-03-20T09:42:29.346 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:28 smithi115 ceph-mon[72476]: pgmap v2680: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 7.2 GiB data, 30 GiB used, 953 GiB / 983 GiB avail; 0 B/s wr, 772 op/s 2024-03-20T09:42:29.348 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:29.374 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:42:29.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:29.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:42:29.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:28 smithi122 ceph-mon[73495]: 1.15 scrub starts 2024-03-20T09:42:29.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:28 smithi122 ceph-mon[73495]: pgmap v2680: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 7.2 GiB data, 30 GiB used, 953 GiB / 983 GiB avail; 0 B/s wr, 772 op/s 2024-03-20T09:42:29.478 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:42:29.534 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:29.557 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-20T09:42:29.572 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:42:29.575 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:29.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:42:29.677 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:42:29.749 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:29.783 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:42:29.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:29.791 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:42:29.893 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:42:29.929 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:29.967 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:42:29.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:29.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:42:30.071 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:42:30.107 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:29 smithi122 ceph-mon[73495]: 9.1 scrub starts 2024-03-20T09:42:30.108 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:29 smithi122 ceph-mon[73495]: 1.15 scrub ok 2024-03-20T09:42:30.108 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:29 smithi122 ceph-mon[73495]: 1.f scrub starts 2024-03-20T09:42:30.108 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:29 smithi122 ceph-mon[73495]: 1.f scrub ok 2024-03-20T09:42:30.108 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:29 smithi122 ceph-mon[73495]: 9.8 scrub starts 2024-03-20T09:42:30.108 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:29 smithi122 ceph-mon[73495]: 9.1 scrub ok 2024-03-20T09:42:30.110 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:30.148 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:42:30.153 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:30.153 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:42:30.254 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:42:30.300 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:30.339 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:42:30.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:30.343 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:42:30.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:29 smithi115 ceph-mon[72476]: 9.1 scrub starts 2024-03-20T09:42:30.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:29 smithi115 ceph-mon[72476]: 1.15 scrub ok 2024-03-20T09:42:30.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:29 smithi115 ceph-mon[72476]: 1.f scrub starts 2024-03-20T09:42:30.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:29 smithi115 ceph-mon[72476]: 1.f scrub ok 2024-03-20T09:42:30.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:29 smithi115 ceph-mon[72476]: 9.8 scrub starts 2024-03-20T09:42:30.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:29 smithi115 ceph-mon[72476]: 9.1 scrub ok 2024-03-20T09:42:30.411 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:29 smithi026 ceph-mon[68309]: 9.1 scrub starts 2024-03-20T09:42:30.411 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:29 smithi026 ceph-mon[68309]: 1.15 scrub ok 2024-03-20T09:42:30.411 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:29 smithi026 ceph-mon[68309]: 1.f scrub starts 2024-03-20T09:42:30.411 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:29 smithi026 ceph-mon[68309]: 1.f scrub ok 2024-03-20T09:42:30.411 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:29 smithi026 ceph-mon[68309]: 9.8 scrub starts 2024-03-20T09:42:30.411 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:29 smithi026 ceph-mon[68309]: 9.1 scrub ok 2024-03-20T09:42:30.444 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:42:30.488 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:42:30.492 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:30.520 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:42:30.521 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:42:30.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:30.528 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:42:30.629 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:42:30.665 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:30.872 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:42:30.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:30.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:42:30.977 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:42:31.013 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:31.052 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:42:31.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:31.059 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:42:31.160 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:42:31.204 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:31.253 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:42:31.264 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:30 smithi115 ceph-mon[72476]: 1.b scrub starts 2024-03-20T09:42:31.264 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:30 smithi115 ceph-mon[72476]: 1.b scrub ok 2024-03-20T09:42:31.264 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:30 smithi115 ceph-mon[72476]: pgmap v2681: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.0 GiB data, 30 GiB used, 954 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s 2024-03-20T09:42:31.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:31.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:42:31.366 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:42:31.416 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:31.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:31 smithi122 ceph-mon[73495]: 1.b scrub starts 2024-03-20T09:42:31.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:31 smithi122 ceph-mon[73495]: 1.b scrub ok 2024-03-20T09:42:31.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:31 smithi122 ceph-mon[73495]: pgmap v2681: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.0 GiB data, 30 GiB used, 954 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s 2024-03-20T09:42:31.445 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:42:31.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:31.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:42:31.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:30 smithi026 ceph-mon[68309]: 1.b scrub starts 2024-03-20T09:42:31.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:30 smithi026 ceph-mon[68309]: 1.b scrub ok 2024-03-20T09:42:31.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:30 smithi026 ceph-mon[68309]: pgmap v2681: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 7.0 GiB data, 30 GiB used, 954 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s 2024-03-20T09:42:31.549 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:42:31.604 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:31.684 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:42:31.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:31.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:42:31.799 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:42:31.845 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:31.886 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:42:31.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:31.891 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:42:31.991 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:42:32.031 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:32.109 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:42:32.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:32.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:42:32.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:31 smithi122 ceph-mon[73495]: 9.8 scrub ok 2024-03-20T09:42:32.216 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:42:32.235 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:31 smithi026 ceph-mon[68309]: 9.8 scrub ok 2024-03-20T09:42:32.257 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:32.283 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:42:32.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:32.289 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:42:32.390 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:42:32.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:31 smithi115 ceph-mon[72476]: 9.8 scrub ok 2024-03-20T09:42:32.443 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:32.470 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:42:32.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:32.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:42:32.583 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:42:32.630 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:32.656 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:42:32.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:32.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:42:32.770 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:42:32.820 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:33.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:32 smithi026 ceph-mon[68309]: 1.19 scrub starts 2024-03-20T09:42:33.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:32 smithi026 ceph-mon[68309]: 1.19 scrub ok 2024-03-20T09:42:33.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:32 smithi026 ceph-mon[68309]: pgmap v2682: 47 pgs: 47 active+clean; 6.9 GiB data, 30 GiB used, 954 GiB / 983 GiB avail; 0 B/s wr, 987 op/s 2024-03-20T09:42:33.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:32 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1551447625' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:42:33.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:33 smithi115 ceph-mon[72476]: 1.19 scrub starts 2024-03-20T09:42:33.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:33 smithi115 ceph-mon[72476]: 1.19 scrub ok 2024-03-20T09:42:33.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:33 smithi115 ceph-mon[72476]: pgmap v2682: 47 pgs: 47 active+clean; 6.9 GiB data, 30 GiB used, 954 GiB / 983 GiB avail; 0 B/s wr, 987 op/s 2024-03-20T09:42:33.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:33 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1551447625' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:42:33.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:33 smithi122 ceph-mon[73495]: 1.19 scrub starts 2024-03-20T09:42:33.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:33 smithi122 ceph-mon[73495]: 1.19 scrub ok 2024-03-20T09:42:33.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:33 smithi122 ceph-mon[73495]: pgmap v2682: 47 pgs: 47 active+clean; 6.9 GiB data, 30 GiB used, 954 GiB / 983 GiB avail; 0 B/s wr, 987 op/s 2024-03-20T09:42:33.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:33 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1551447625' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:42:34.290 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:42:34.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:33 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:42:34.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:33 smithi115 ceph-mon[72476]: Cluster is now healthy 2024-03-20T09:42:34.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:33 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1551447625' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:42:34.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:33 smithi115 ceph-mon[72476]: osdmap e1105: 12 total, 12 up, 11 in 2024-03-20T09:42:34.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:34 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:42:34.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:34 smithi122 ceph-mon[73495]: Cluster is now healthy 2024-03-20T09:42:34.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:34 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1551447625' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:42:34.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:34 smithi122 ceph-mon[73495]: osdmap e1105: 12 total, 12 up, 11 in 2024-03-20T09:42:34.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:33 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:42:34.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:33 smithi026 ceph-mon[68309]: Cluster is now healthy 2024-03-20T09:42:34.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:33 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1551447625' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:42:34.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:33 smithi026 ceph-mon[68309]: osdmap e1105: 12 total, 12 up, 11 in 2024-03-20T09:42:34.784 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:42:34.833 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:42:34.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:34.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:42:34.938 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:42:34.976 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:35.065 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:42:35.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:35.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:42:35.114 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:35 smithi026 ceph-mon[68309]: 1.18 scrub starts 2024-03-20T09:42:35.114 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:35 smithi026 ceph-mon[68309]: 1.18 scrub ok 2024-03-20T09:42:35.114 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:35 smithi026 ceph-mon[68309]: 1.1b scrub starts 2024-03-20T09:42:35.114 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:35 smithi026 ceph-mon[68309]: pgmap v2684: 47 pgs: 47 active+clean; 6.8 GiB data, 30 GiB used, 954 GiB / 983 GiB avail; 0 B/s wr, 938 op/s 2024-03-20T09:42:35.170 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:42:35.211 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:35.251 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:42:35.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:35.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:42:35.356 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:42:35.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:35 smithi115 ceph-mon[72476]: 1.18 scrub starts 2024-03-20T09:42:35.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:35 smithi115 ceph-mon[72476]: 1.18 scrub ok 2024-03-20T09:42:35.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:35 smithi115 ceph-mon[72476]: 1.1b scrub starts 2024-03-20T09:42:35.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:35 smithi115 ceph-mon[72476]: pgmap v2684: 47 pgs: 47 active+clean; 6.8 GiB data, 30 GiB used, 954 GiB / 983 GiB avail; 0 B/s wr, 938 op/s 2024-03-20T09:42:35.398 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:35.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:35 smithi122 ceph-mon[73495]: 1.18 scrub starts 2024-03-20T09:42:35.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:35 smithi122 ceph-mon[73495]: 1.18 scrub ok 2024-03-20T09:42:35.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:35 smithi122 ceph-mon[73495]: 1.1b scrub starts 2024-03-20T09:42:35.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:35 smithi122 ceph-mon[73495]: pgmap v2684: 47 pgs: 47 active+clean; 6.8 GiB data, 30 GiB used, 954 GiB / 983 GiB avail; 0 B/s wr, 938 op/s 2024-03-20T09:42:35.661 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:42:35.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:35.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:42:35.767 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:42:35.822 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:35.858 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:42:35.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:35.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:42:35.962 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:42:36.000 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:36.038 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:42:36.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:36.041 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:42:36.142 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:42:36.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:36 smithi115 ceph-mon[72476]: 1.1b scrub ok 2024-03-20T09:42:36.175 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:36.214 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:42:36.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:36.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:42:36.323 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:42:36.326 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:42:36.357 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:42:36.369 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:36 smithi026 ceph-mon[68309]: 1.1b scrub ok 2024-03-20T09:42:36.370 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:36.399 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:42:36.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:36 smithi122 ceph-mon[73495]: 1.1b scrub ok 2024-03-20T09:42:36.438 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:42:36.462 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:42:36.499 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:42:36.533 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:42:36.550 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:42:36.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:36.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:42:36.573 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:42:36.600 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:42:36.627 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:42:36.654 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:42:36.658 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:42:36.694 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:42:36.696 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:42:36.696 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:42:36.696 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:42:36.696 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:42:36.696 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:42:36.697 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:42:36.697 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:42:36.697 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:42:36.697 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:42:36.697 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:42:36.697 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:42:36.697 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:42:36.703 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:36.751 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:42:36.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:36.754 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:42:36.855 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:42:36.904 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:36.932 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:42:36.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:36.935 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:42:37.014 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:37 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2674356152' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:42:37.014 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:37 smithi026 ceph-mon[68309]: pgmap v2685: 47 pgs: 47 active+clean; 6.7 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 917 op/s 2024-03-20T09:42:37.016 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:37 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2674356152' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:42:37.016 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:37 smithi122 ceph-mon[73495]: pgmap v2685: 47 pgs: 47 active+clean; 6.7 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 917 op/s 2024-03-20T09:42:37.036 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:42:37.091 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:37.118 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:42:37.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:37.128 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:42:37.229 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:42:37.243 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:42:37.243 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-03-20T09:42:37.243 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-03-20T09:42:37.243 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:42:37.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.244 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:213819 7.11 7:8d9623d5:::benchmark_data_smithi122_187077_object87208:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e936)", 2024-03-20T09:42:37.244 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:36.133360+0000", 2024-03-20T09:42:37.244 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 481.11836570200001, 2024-03-20T09:42:37.244 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.225427131, 2024-03-20T09:42:37.244 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:42:37.244 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:42:37.244 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:42:37.244 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:42:37.244 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:42:37.244 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 213819 2024-03-20T09:42:37.244 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.244 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:42:37.244 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.244 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:42:37.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.133360+0000", 2024-03-20T09:42:37.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:42:37.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.133360+0000", 2024-03-20T09:42:37.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.245 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.245 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:42:37.245 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.133361+0000", 2024-03-20T09:42:37.245 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0070000000000001e-06 2024-03-20T09:42:37.245 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:42:37.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.133365+0000", 2024-03-20T09:42:37.246 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6049999999999998e-06 2024-03-20T09:42:37.246 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.246 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.246 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:42:37.246 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.133376+0000", 2024-03-20T09:42:37.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1289e-05 2024-03-20T09:42:37.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:42:37.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.133385+0000", 2024-03-20T09:42:37.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1800000000000002e-06 2024-03-20T09:42:37.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.247 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:42:37.247 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.135917+0000", 2024-03-20T09:42:37.247 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.002531789 2024-03-20T09:42:37.247 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.247 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:42:37.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:36.135928+0000", 2024-03-20T09:42:37.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0859000000000001e-05 2024-03-20T09:42:37.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:42:37.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.329225+0000", 2024-03-20T09:42:37.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.193297351 2024-03-20T09:42:37.248 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.248 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.248 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:42:37.248 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.329367+0000", 2024-03-20T09:42:37.248 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014130300000000001 2024-03-20T09:42:37.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4,6,7,9", 2024-03-20T09:42:37.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.329686+0000", 2024-03-20T09:42:37.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000319024 2024-03-20T09:42:37.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:42:37.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.331282+0000", 2024-03-20T09:42:37.249 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0015964869999999999 2024-03-20T09:42:37.249 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.249 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.249 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.249 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.335355+0000", 2024-03-20T09:42:37.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040729449999999997 2024-03-20T09:42:37.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.335396+0000", 2024-03-20T09:42:37.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0850999999999999e-05 2024-03-20T09:42:37.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.250 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.250 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.339952+0000", 2024-03-20T09:42:37.250 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0045558939999999996 2024-03-20T09:42:37.250 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.250 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.358609+0000", 2024-03-20T09:42:37.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018657183000000001 2024-03-20T09:42:37.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:42:37.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.358672+0000", 2024-03-20T09:42:37.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2490000000000006e-05 2024-03-20T09:42:37.251 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.251 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.251 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:42:37.251 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:37.358787+0000", 2024-03-20T09:42:37.251 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000115874 2024-03-20T09:42:37.251 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.251 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:42:37.252 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.252 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.252 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.252 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:220094 7.10 7:0de3b1d8:::benchmark_data_smithi122_187077_object93483:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e939)", 2024-03-20T09:42:37.252 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:44.621484+0000", 2024-03-20T09:42:37.252 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 472.63024240200002, 2024-03-20T09:42:37.252 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.044103123, 2024-03-20T09:42:37.252 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:42:37.252 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:42:37.252 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:42:37.252 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:42:37.252 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:42:37.252 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 220094 2024-03-20T09:42:37.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.253 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:42:37.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:42:37.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.621484+0000", 2024-03-20T09:42:37.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:42:37.253 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.621484+0000", 2024-03-20T09:42:37.253 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.253 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.253 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.253 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:42:37.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.621484+0000", 2024-03-20T09:42:37.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8400000000000003e-07 2024-03-20T09:42:37.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:42:37.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.621488+0000", 2024-03-20T09:42:37.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3409999999999999e-06 2024-03-20T09:42:37.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.254 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.254 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:42:37.254 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.621495+0000", 2024-03-20T09:42:37.254 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4769999999999997e-06 2024-03-20T09:42:37.254 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:42:37.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.621500+0000", 2024-03-20T09:42:37.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6430000000000003e-06 2024-03-20T09:42:37.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:42:37.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.624950+0000", 2024-03-20T09:42:37.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0034496700000000002 2024-03-20T09:42:37.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.255 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.255 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:42:37.255 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.624959+0000", 2024-03-20T09:42:37.255 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3999999999999998e-06 2024-03-20T09:42:37.255 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:42:37.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.660516+0000", 2024-03-20T09:42:37.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0355567640000001 2024-03-20T09:42:37.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.256 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:42:37.256 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.660602+0000", 2024-03-20T09:42:37.256 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6598e-05 2024-03-20T09:42:37.256 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.256 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:42:37.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.660906+0000", 2024-03-20T09:42:37.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00030355699999999998 2024-03-20T09:42:37.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.665420+0000", 2024-03-20T09:42:37.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0045136559999999996 2024-03-20T09:42:37.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.257 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:42:37.257 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.665427+0000", 2024-03-20T09:42:37.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9440000000000005e-06 2024-03-20T09:42:37.257 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.665506+0000", 2024-03-20T09:42:37.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8060999999999997e-05 2024-03-20T09:42:37.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:42:37.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.665541+0000", 2024-03-20T09:42:37.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5843999999999998e-05 2024-03-20T09:42:37.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:42:37.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.665587+0000", 2024-03-20T09:42:37.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5383999999999998e-05 2024-03-20T09:42:37.259 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.259 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:42:37.259 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.259 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:220162 7.10 7:0fd788c0:::benchmark_data_smithi122_187077_object93551:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e939)", 2024-03-20T09:42:37.259 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:44.694214+0000", 2024-03-20T09:42:37.259 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 472.557511734, 2024-03-20T09:42:37.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.972725795, 2024-03-20T09:42:37.259 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:42:37.260 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:42:37.260 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:42:37.260 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:42:37.260 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:42:37.260 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 220162 2024-03-20T09:42:37.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.260 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:42:37.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:42:37.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.694214+0000", 2024-03-20T09:42:37.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:42:37.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.694214+0000", 2024-03-20T09:42:37.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:42:37.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.694216+0000", 2024-03-20T09:42:37.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.24e-06 2024-03-20T09:42:37.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:42:37.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.694221+0000", 2024-03-20T09:42:37.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.994e-06 2024-03-20T09:42:37.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:42:37.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.694232+0000", 2024-03-20T09:42:37.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1725e-05 2024-03-20T09:42:37.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:42:37.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.694240+0000", 2024-03-20T09:42:37.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3030000000000002e-06 2024-03-20T09:42:37.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.262 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:42:37.262 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.696827+0000", 2024-03-20T09:42:37.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0025872740000000001 2024-03-20T09:42:37.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:42:37.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.696837+0000", 2024-03-20T09:42:37.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0171e-05 2024-03-20T09:42:37.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:42:37.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.662591+0000", 2024-03-20T09:42:37.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.96575427800000002 2024-03-20T09:42:37.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:42:37.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.662669+0000", 2024-03-20T09:42:37.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.784e-05 2024-03-20T09:42:37.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:42:37.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.662838+0000", 2024-03-20T09:42:37.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000169092 2024-03-20T09:42:37.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:42:37.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.665482+0000", 2024-03-20T09:42:37.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0026433369999999999 2024-03-20T09:42:37.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.665627+0000", 2024-03-20T09:42:37.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000145088 2024-03-20T09:42:37.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.666839+0000", 2024-03-20T09:42:37.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012126769999999999 2024-03-20T09:42:37.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:42:37.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.666895+0000", 2024-03-20T09:42:37.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5424999999999999e-05 2024-03-20T09:42:37.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:42:37.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.666940+0000", 2024-03-20T09:42:37.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5351e-05 2024-03-20T09:42:37.266 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.266 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:42:37.266 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.266 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:220174 7.10 7:0823f244:::benchmark_data_smithi122_187077_object93563:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e939)", 2024-03-20T09:42:37.266 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:44.705145+0000", 2024-03-20T09:42:37.266 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 472.546581016, 2024-03-20T09:42:37.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.96493656299999997, 2024-03-20T09:42:37.266 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:42:37.267 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:42:37.267 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:42:37.267 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:42:37.267 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:42:37.267 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 220174 2024-03-20T09:42:37.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.267 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:42:37.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:42:37.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.705145+0000", 2024-03-20T09:42:37.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:42:37.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.705145+0000", 2024-03-20T09:42:37.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:42:37.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.705146+0000", 2024-03-20T09:42:37.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.226e-06 2024-03-20T09:42:37.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:42:37.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.705154+0000", 2024-03-20T09:42:37.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0409999999999998e-06 2024-03-20T09:42:37.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:42:37.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.705165+0000", 2024-03-20T09:42:37.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1042e-05 2024-03-20T09:42:37.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:42:37.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.705172+0000", 2024-03-20T09:42:37.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4450000000000001e-06 2024-03-20T09:42:37.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:42:37.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.705224+0000", 2024-03-20T09:42:37.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2682000000000001e-05 2024-03-20T09:42:37.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:42:37.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.705234+0000", 2024-03-20T09:42:37.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7999999999999993e-06 2024-03-20T09:42:37.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:42:37.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.663989+0000", 2024-03-20T09:42:37.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.95875515899999997 2024-03-20T09:42:37.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:42:37.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.664080+0000", 2024-03-20T09:42:37.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0752000000000001e-05 2024-03-20T09:42:37.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:42:37.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.664253+0000", 2024-03-20T09:42:37.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00017302800000000001 2024-03-20T09:42:37.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.665670+0000", 2024-03-20T09:42:37.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0014167610000000001 2024-03-20T09:42:37.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.666991+0000", 2024-03-20T09:42:37.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001321196 2024-03-20T09:42:37.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:42:37.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.670018+0000", 2024-03-20T09:42:37.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0030268510000000001 2024-03-20T09:42:37.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:42:37.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.670052+0000", 2024-03-20T09:42:37.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3794999999999997e-05 2024-03-20T09:42:37.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:42:37.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.670082+0000", 2024-03-20T09:42:37.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9785e-05 2024-03-20T09:42:37.273 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.273 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:42:37.273 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.273 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:220176 7.10 7:0fc144d7:::benchmark_data_smithi122_187077_object93565:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e939)", 2024-03-20T09:42:37.273 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:44.705190+0000", 2024-03-20T09:42:37.273 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 472.54653574299999, 2024-03-20T09:42:37.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.97056713500000003, 2024-03-20T09:42:37.273 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:42:37.273 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:42:37.273 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:42:37.273 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:42:37.274 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:42:37.274 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 220176 2024-03-20T09:42:37.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.274 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:42:37.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:42:37.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.705190+0000", 2024-03-20T09:42:37.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:42:37.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.705190+0000", 2024-03-20T09:42:37.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:42:37.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.705191+0000", 2024-03-20T09:42:37.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1399999999999997e-07 2024-03-20T09:42:37.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:42:37.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.705194+0000", 2024-03-20T09:42:37.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.016e-06 2024-03-20T09:42:37.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.275 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:42:37.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.705201+0000", 2024-03-20T09:42:37.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9099999999999999e-06 2024-03-20T09:42:37.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:42:37.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.705207+0000", 2024-03-20T09:42:37.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0449999999999997e-06 2024-03-20T09:42:37.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:42:37.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.705259+0000", 2024-03-20T09:42:37.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2466000000000002e-05 2024-03-20T09:42:37.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:42:37.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.705265+0000", 2024-03-20T09:42:37.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-06 2024-03-20T09:42:37.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:42:37.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.665695+0000", 2024-03-20T09:42:37.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.96042964600000003 2024-03-20T09:42:37.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:42:37.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.665763+0000", 2024-03-20T09:42:37.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8093e-05 2024-03-20T09:42:37.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:42:37.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.665928+0000", 2024-03-20T09:42:37.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000164809 2024-03-20T09:42:37.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:42:37.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.670090+0000", 2024-03-20T09:42:37.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0041622650000000001 2024-03-20T09:42:37.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.670154+0000", 2024-03-20T09:42:37.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3595000000000004e-05 2024-03-20T09:42:37.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.675690+0000", 2024-03-20T09:42:37.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0055369069999999998 2024-03-20T09:42:37.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:42:37.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.675727+0000", 2024-03-20T09:42:37.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6189000000000002e-05 2024-03-20T09:42:37.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:42:37.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.675757+0000", 2024-03-20T09:42:37.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.0800000000000003e-05 2024-03-20T09:42:37.279 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.279 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:42:37.279 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.280 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:220274 7.10 7:0e598cf8:::benchmark_data_smithi122_187077_object93663:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e939)", 2024-03-20T09:42:37.280 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:44.817402+0000", 2024-03-20T09:42:37.280 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 472.434323619, 2024-03-20T09:42:37.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.858534405, 2024-03-20T09:42:37.280 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:42:37.280 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:42:37.280 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:42:37.280 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:42:37.280 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:42:37.280 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 220274 2024-03-20T09:42:37.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.280 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:42:37.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:42:37.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.817402+0000", 2024-03-20T09:42:37.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:42:37.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.817402+0000", 2024-03-20T09:42:37.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:42:37.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.817404+0000", 2024-03-20T09:42:37.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2529999999999999e-06 2024-03-20T09:42:37.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:42:37.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.817409+0000", 2024-03-20T09:42:37.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3380000000000004e-06 2024-03-20T09:42:37.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:42:37.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.817422+0000", 2024-03-20T09:42:37.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.309e-05 2024-03-20T09:42:37.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:42:37.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.817430+0000", 2024-03-20T09:42:37.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7489999999999998e-06 2024-03-20T09:42:37.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:42:37.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.817493+0000", 2024-03-20T09:42:37.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3461999999999996e-05 2024-03-20T09:42:37.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:42:37.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.817503+0000", 2024-03-20T09:42:37.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9580000000000002e-06 2024-03-20T09:42:37.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:42:37.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.667032+0000", 2024-03-20T09:42:37.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.849528426 2024-03-20T09:42:37.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:42:37.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.667153+0000", 2024-03-20T09:42:37.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000121532 2024-03-20T09:42:37.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:42:37.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.667368+0000", 2024-03-20T09:42:37.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000214305 2024-03-20T09:42:37.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.675643+0000", 2024-03-20T09:42:37.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0082752980000000004 2024-03-20T09:42:37.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.675819+0000", 2024-03-20T09:42:37.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000175921 2024-03-20T09:42:37.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:42:37.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.675835+0000", 2024-03-20T09:42:37.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6268000000000001e-05 2024-03-20T09:42:37.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:42:37.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.675890+0000", 2024-03-20T09:42:37.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4960999999999998e-05 2024-03-20T09:42:37.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:42:37.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.675937+0000", 2024-03-20T09:42:37.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6844000000000001e-05 2024-03-20T09:42:37.286 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.286 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:42:37.286 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.287 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:220312 7.10 7:0dcc3d73:::benchmark_data_smithi122_187077_object93701:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e939)", 2024-03-20T09:42:37.287 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:44.859829+0000", 2024-03-20T09:42:37.287 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 472.391896879, 2024-03-20T09:42:37.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.82423321599999999, 2024-03-20T09:42:37.287 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:42:37.287 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:42:37.287 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:42:37.287 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:42:37.287 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:42:37.287 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 220312 2024-03-20T09:42:37.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.288 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:42:37.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:42:37.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.859829+0000", 2024-03-20T09:42:37.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:42:37.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.859829+0000", 2024-03-20T09:42:37.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:42:37.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.859830+0000", 2024-03-20T09:42:37.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4099999999999999e-07 2024-03-20T09:42:37.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:42:37.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.859833+0000", 2024-03-20T09:42:37.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2559999999999998e-06 2024-03-20T09:42:37.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:42:37.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.859839+0000", 2024-03-20T09:42:37.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4829999999999996e-06 2024-03-20T09:42:37.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:42:37.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.859843+0000", 2024-03-20T09:42:37.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5499999999999999e-06 2024-03-20T09:42:37.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:42:37.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.859940+0000", 2024-03-20T09:42:37.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.6996000000000004e-05 2024-03-20T09:42:37.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:42:37.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.859947+0000", 2024-03-20T09:42:37.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6760000000000001e-06 2024-03-20T09:42:37.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:42:37.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.668600+0000", 2024-03-20T09:42:37.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.80865322500000003 2024-03-20T09:42:37.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:42:37.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.668701+0000", 2024-03-20T09:42:37.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101279 2024-03-20T09:42:37.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:42:37.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.668889+0000", 2024-03-20T09:42:37.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000188137 2024-03-20T09:42:37.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.683765+0000", 2024-03-20T09:42:37.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01487551 2024-03-20T09:42:37.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.683788+0000", 2024-03-20T09:42:37.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3748e-05 2024-03-20T09:42:37.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:42:37.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.683956+0000", 2024-03-20T09:42:37.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00016788799999999999 2024-03-20T09:42:37.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:42:37.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684013+0000", 2024-03-20T09:42:37.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6891000000000001e-05 2024-03-20T09:42:37.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:42:37.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684062+0000", 2024-03-20T09:42:37.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9135999999999998e-05 2024-03-20T09:42:37.294 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.294 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:42:37.294 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.294 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:220381 7.10 7:0b0ef641:::benchmark_data_smithi122_187077_object93770:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e939)", 2024-03-20T09:42:37.294 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:44.942756+0000", 2024-03-20T09:42:37.294 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 472.30897039899997, 2024-03-20T09:42:37.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.74139817399999997, 2024-03-20T09:42:37.294 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:42:37.294 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:42:37.294 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:42:37.294 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:42:37.295 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:42:37.295 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 220381 2024-03-20T09:42:37.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.295 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:42:37.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:42:37.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.942756+0000", 2024-03-20T09:42:37.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:42:37.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.942756+0000", 2024-03-20T09:42:37.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:42:37.296 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.942757+0000", 2024-03-20T09:42:37.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.672e-06 2024-03-20T09:42:37.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:42:37.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.942764+0000", 2024-03-20T09:42:37.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3099999999999997e-06 2024-03-20T09:42:37.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:42:37.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.942780+0000", 2024-03-20T09:42:37.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5928000000000001e-05 2024-03-20T09:42:37.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:42:37.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.942787+0000", 2024-03-20T09:42:37.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7319999999999998e-06 2024-03-20T09:42:37.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:42:37.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.942851+0000", 2024-03-20T09:42:37.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3829000000000002e-05 2024-03-20T09:42:37.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:42:37.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:44.942860+0000", 2024-03-20T09:42:37.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.5660000000000007e-06 2024-03-20T09:42:37.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:42:37.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.670195+0000", 2024-03-20T09:42:37.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.72733487699999999 2024-03-20T09:42:37.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:42:37.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.670305+0000", 2024-03-20T09:42:37.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011026099999999999 2024-03-20T09:42:37.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:42:37.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.670561+0000", 2024-03-20T09:42:37.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00025577500000000002 2024-03-20T09:42:37.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.683818+0000", 2024-03-20T09:42:37.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.013257193 2024-03-20T09:42:37.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.301 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.301 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.683838+0000", 2024-03-20T09:42:37.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0089e-05 2024-03-20T09:42:37.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.301 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:42:37.301 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684072+0000", 2024-03-20T09:42:37.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00023379999999999999 2024-03-20T09:42:37.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.301 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:42:37.301 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684116+0000", 2024-03-20T09:42:37.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4186e-05 2024-03-20T09:42:37.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.302 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:42:37.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684154+0000", 2024-03-20T09:42:37.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7956e-05 2024-03-20T09:42:37.302 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.302 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:42:37.302 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.302 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:220445 7.10 7:0f889c96:::benchmark_data_smithi122_187077_object93834:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e939)", 2024-03-20T09:42:37.302 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:45.016832+0000", 2024-03-20T09:42:37.302 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 472.23489352799999, 2024-03-20T09:42:37.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.66740921499999994, 2024-03-20T09:42:37.303 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:42:37.303 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:42:37.303 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:42:37.303 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:42:37.303 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:42:37.303 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 220445 2024-03-20T09:42:37.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.303 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:42:37.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:42:37.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.016832+0000", 2024-03-20T09:42:37.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:42:37.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.016832+0000", 2024-03-20T09:42:37.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:42:37.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.016833+0000", 2024-03-20T09:42:37.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.7600000000000006e-07 2024-03-20T09:42:37.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:42:37.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.016837+0000", 2024-03-20T09:42:37.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1939999999999998e-06 2024-03-20T09:42:37.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:42:37.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.016848+0000", 2024-03-20T09:42:37.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1442e-05 2024-03-20T09:42:37.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:42:37.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.016854+0000", 2024-03-20T09:42:37.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8000000000000004e-06 2024-03-20T09:42:37.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:42:37.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.016912+0000", 2024-03-20T09:42:37.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7976999999999997e-05 2024-03-20T09:42:37.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:42:37.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.016917+0000", 2024-03-20T09:42:37.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3820000000000003e-06 2024-03-20T09:42:37.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:42:37.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.671579+0000", 2024-03-20T09:42:37.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.65466185200000004 2024-03-20T09:42:37.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:42:37.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.671701+0000", 2024-03-20T09:42:37.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000121835 2024-03-20T09:42:37.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:42:37.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.671919+0000", 2024-03-20T09:42:37.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000217653 2024-03-20T09:42:37.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.683860+0000", 2024-03-20T09:42:37.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011941230000000001 2024-03-20T09:42:37.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.683885+0000", 2024-03-20T09:42:37.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4890999999999999e-05 2024-03-20T09:42:37.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:42:37.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684162+0000", 2024-03-20T09:42:37.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00027684200000000002 2024-03-20T09:42:37.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:42:37.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684204+0000", 2024-03-20T09:42:37.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2537999999999998e-05 2024-03-20T09:42:37.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.309 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:42:37.309 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684242+0000", 2024-03-20T09:42:37.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7602999999999999e-05 2024-03-20T09:42:37.309 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.309 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:42:37.309 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.309 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:220450 7.10 7:083cb24d:::benchmark_data_smithi122_187077_object93839:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e939)", 2024-03-20T09:42:37.309 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:45.022699+0000", 2024-03-20T09:42:37.310 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 472.22902736100002, 2024-03-20T09:42:37.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.66164454900000003, 2024-03-20T09:42:37.310 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:42:37.310 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:42:37.310 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:42:37.310 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:42:37.310 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:42:37.310 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 220450 2024-03-20T09:42:37.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.310 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:42:37.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:42:37.310 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.022699+0000", 2024-03-20T09:42:37.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:42:37.311 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.022699+0000", 2024-03-20T09:42:37.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:42:37.311 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.022700+0000", 2024-03-20T09:42:37.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1650000000000001e-06 2024-03-20T09:42:37.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:42:37.311 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.022708+0000", 2024-03-20T09:42:37.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3659999999999992e-06 2024-03-20T09:42:37.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:42:37.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.022719+0000", 2024-03-20T09:42:37.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1113e-05 2024-03-20T09:42:37.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:42:37.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.022725+0000", 2024-03-20T09:42:37.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.015e-06 2024-03-20T09:42:37.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:42:37.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.022771+0000", 2024-03-20T09:42:37.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5750000000000001e-05 2024-03-20T09:42:37.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:42:37.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.022779+0000", 2024-03-20T09:42:37.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4889999999999998e-06 2024-03-20T09:42:37.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:42:37.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.672870+0000", 2024-03-20T09:42:37.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.65009129499999996 2024-03-20T09:42:37.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:42:37.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.672971+0000", 2024-03-20T09:42:37.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00010152700000000001 2024-03-20T09:42:37.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:42:37.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.673129+0000", 2024-03-20T09:42:37.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015808 2024-03-20T09:42:37.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.683940+0000", 2024-03-20T09:42:37.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010810336 2024-03-20T09:42:37.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:42:37.315 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684250+0000", 2024-03-20T09:42:37.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00031030499999999998 2024-03-20T09:42:37.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.315 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684286+0000", 2024-03-20T09:42:37.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5509000000000002e-05 2024-03-20T09:42:37.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:42:37.315 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684315+0000", 2024-03-20T09:42:37.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9624999999999998e-05 2024-03-20T09:42:37.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.316 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:42:37.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684343+0000", 2024-03-20T09:42:37.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7974e-05 2024-03-20T09:42:37.316 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.316 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:42:37.316 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.316 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:220478 7.10 7:0f3ce08f:::benchmark_data_smithi122_187077_object93867:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e939)", 2024-03-20T09:42:37.316 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:45.066044+0000", 2024-03-20T09:42:37.316 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 472.18568243099998, 2024-03-20T09:42:37.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.61846596399999998, 2024-03-20T09:42:37.317 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:42:37.317 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:42:37.317 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:42:37.317 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:42:37.317 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:42:37.317 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 220478 2024-03-20T09:42:37.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.317 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:42:37.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:42:37.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.066044+0000", 2024-03-20T09:42:37.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:42:37.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.066044+0000", 2024-03-20T09:42:37.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:42:37.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.066045+0000", 2024-03-20T09:42:37.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.424e-06 2024-03-20T09:42:37.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:42:37.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.066051+0000", 2024-03-20T09:42:37.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.7119999999999997e-06 2024-03-20T09:42:37.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:42:37.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.066065+0000", 2024-03-20T09:42:37.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.38e-05 2024-03-20T09:42:37.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:42:37.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.066073+0000", 2024-03-20T09:42:37.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.2139999999999996e-06 2024-03-20T09:42:37.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:42:37.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.066137+0000", 2024-03-20T09:42:37.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3876999999999998e-05 2024-03-20T09:42:37.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:42:37.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.066148+0000", 2024-03-20T09:42:37.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1484999999999999e-05 2024-03-20T09:42:37.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:42:37.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.674079+0000", 2024-03-20T09:42:37.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.60793130500000003 2024-03-20T09:42:37.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:42:37.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.674195+0000", 2024-03-20T09:42:37.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000115457 2024-03-20T09:42:37.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.321 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:42:37.321 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.674398+0000", 2024-03-20T09:42:37.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00020298700000000001 2024-03-20T09:42:37.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:42:37.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684257+0000", 2024-03-20T09:42:37.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0098594200000000007 2024-03-20T09:42:37.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684371+0000", 2024-03-20T09:42:37.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000113529 2024-03-20T09:42:37.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684426+0000", 2024-03-20T09:42:37.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5336e-05 2024-03-20T09:42:37.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:42:37.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684480+0000", 2024-03-20T09:42:37.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4166000000000003e-05 2024-03-20T09:42:37.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:42:37.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684510+0000", 2024-03-20T09:42:37.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9252e-05 2024-03-20T09:42:37.323 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.323 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:42:37.323 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.324 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26605.0:220492 7.10 7:0816362b:::benchmark_data_smithi122_187077_object93881:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e939)", 2024-03-20T09:42:37.324 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:34:45.097130+0000", 2024-03-20T09:42:37.324 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 472.15459599100001, 2024-03-20T09:42:37.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.58752722199999996, 2024-03-20T09:42:37.324 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:42:37.324 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:42:37.324 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:42:37.324 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26605", 2024-03-20T09:42:37.324 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:42:37.324 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 220492 2024-03-20T09:42:37.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.324 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:42:37.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:42:37.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.097130+0000", 2024-03-20T09:42:37.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:42:37.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.097130+0000", 2024-03-20T09:42:37.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:42:37.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.097132+0000", 2024-03-20T09:42:37.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5269999999999999e-06 2024-03-20T09:42:37.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:42:37.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.097139+0000", 2024-03-20T09:42:37.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0910000000000004e-06 2024-03-20T09:42:37.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:42:37.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.097155+0000", 2024-03-20T09:42:37.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6062e-05 2024-03-20T09:42:37.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:42:37.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.097164+0000", 2024-03-20T09:42:37.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.38e-06 2024-03-20T09:42:37.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:42:37.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.097226+0000", 2024-03-20T09:42:37.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2038999999999993e-05 2024-03-20T09:42:37.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for peered", 2024-03-20T09:42:37.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.097236+0000", 2024-03-20T09:42:37.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3999999999999998e-06 2024-03-20T09:42:37.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.328 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:42:37.328 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.676020+0000", 2024-03-20T09:42:37.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.57878407700000001 2024-03-20T09:42:37.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.328 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:42:37.328 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.676095+0000", 2024-03-20T09:42:37.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5822999999999996e-05 2024-03-20T09:42:37.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 7,9", 2024-03-20T09:42:37.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.676258+0000", 2024-03-20T09:42:37.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00016275400000000001 2024-03-20T09:42:37.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:42:37.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684523+0000", 2024-03-20T09:42:37.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0082645029999999994 2024-03-20T09:42:37.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684573+0000", 2024-03-20T09:42:37.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0234e-05 2024-03-20T09:42:37.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684601+0000", 2024-03-20T09:42:37.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7659999999999999e-05 2024-03-20T09:42:37.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:42:37.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684629+0000", 2024-03-20T09:42:37.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8184e-05 2024-03-20T09:42:37.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:42:37.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:34:45.684657+0000", 2024-03-20T09:42:37.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8487999999999999e-05 2024-03-20T09:42:37.331 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.331 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:42:37.331 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.331 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.27190.0:101478 9.1 9:8a4a7c15:::benchmark_data_smithi122_210164_object101477:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1087)", 2024-03-20T09:42:37.331 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:41:33.116664+0000", 2024-03-20T09:42:37.331 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 64.135062082999994, 2024-03-20T09:42:37.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.145735948, 2024-03-20T09:42:37.331 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:42:37.332 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:42:37.332 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:42:37.332 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.27190", 2024-03-20T09:42:37.332 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:42:37.332 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 101478 2024-03-20T09:42:37.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.332 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:42:37.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:42:37.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:33.116664+0000", 2024-03-20T09:42:37.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:42:37.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:33.116664+0000", 2024-03-20T09:42:37.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:42:37.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:33.116665+0000", 2024-03-20T09:42:37.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1000000000000001e-06 2024-03-20T09:42:37.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:42:37.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:33.116863+0000", 2024-03-20T09:42:37.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00019824999999999999 2024-03-20T09:42:37.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:42:37.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:33.116877+0000", 2024-03-20T09:42:37.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3423e-05 2024-03-20T09:42:37.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:42:37.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:33.116885+0000", 2024-03-20T09:42:37.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6729999999999998e-06 2024-03-20T09:42:37.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:42:37.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:33.116950+0000", 2024-03-20T09:42:37.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4907999999999996e-05 2024-03-20T09:42:37.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:42:37.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:33.117117+0000", 2024-03-20T09:42:37.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00016672 2024-03-20T09:42:37.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 6,9", 2024-03-20T09:42:37.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:33.117264+0000", 2024-03-20T09:42:37.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014725099999999999 2024-03-20T09:42:37.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:33.123597+0000", 2024-03-20T09:42:37.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0063326179999999999 2024-03-20T09:42:37.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:33.174965+0000", 2024-03-20T09:42:37.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.051367821000000001 2024-03-20T09:42:37.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:42:37.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:33.262284+0000", 2024-03-20T09:42:37.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.087319035000000003 2024-03-20T09:42:37.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:42:37.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:33.262348+0000", 2024-03-20T09:42:37.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4090999999999993e-05 2024-03-20T09:42:37.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.337 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:42:37.337 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:33.262400+0000", 2024-03-20T09:42:37.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2058000000000002e-05 2024-03-20T09:42:37.337 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.337 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:42:37.337 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.337 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.27190.0:123975 9.1 9:882e8aeb:::benchmark_data_smithi122_210164_object123974:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1097)", 2024-03-20T09:42:37.337 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:41:55.385457+0000", 2024-03-20T09:42:37.338 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 41.866269430999999, 2024-03-20T09:42:37.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.14345419100000001, 2024-03-20T09:42:37.338 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:42:37.338 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:42:37.338 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:42:37.338 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.27190", 2024-03-20T09:42:37.338 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:42:37.338 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 123975 2024-03-20T09:42:37.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.338 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:42:37.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:42:37.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.385457+0000", 2024-03-20T09:42:37.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:42:37.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.385457+0000", 2024-03-20T09:42:37.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:42:37.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.385458+0000", 2024-03-20T09:42:37.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.144e-06 2024-03-20T09:42:37.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:42:37.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.385553+0000", 2024-03-20T09:42:37.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5617000000000003e-05 2024-03-20T09:42:37.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:42:37.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.385566+0000", 2024-03-20T09:42:37.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2857000000000001e-05 2024-03-20T09:42:37.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:42:37.340 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.385584+0000", 2024-03-20T09:42:37.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8085e-05 2024-03-20T09:42:37.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:42:37.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.438057+0000", 2024-03-20T09:42:37.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.052472692000000001 2024-03-20T09:42:37.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:42:37.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.438224+0000", 2024-03-20T09:42:37.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00016708599999999999 2024-03-20T09:42:37.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 6,9", 2024-03-20T09:42:37.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.438394+0000", 2024-03-20T09:42:37.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00017034999999999999 2024-03-20T09:42:37.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:42:37.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.447919+0000", 2024-03-20T09:42:37.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0095246519999999998 2024-03-20T09:42:37.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.466355+0000", 2024-03-20T09:42:37.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018436118000000001 2024-03-20T09:42:37.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.528825+0000", 2024-03-20T09:42:37.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.062470312 2024-03-20T09:42:37.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:42:37.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.528861+0000", 2024-03-20T09:42:37.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5573e-05 2024-03-20T09:42:37.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:42:37.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.528911+0000", 2024-03-20T09:42:37.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9704999999999999e-05 2024-03-20T09:42:37.343 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.343 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:42:37.344 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.344 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.27190.0:123984 9.1 9:832d5abd:::benchmark_data_smithi122_210164_object123983:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1097)", 2024-03-20T09:42:37.344 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:41:55.395746+0000", 2024-03-20T09:42:37.344 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 41.855979898000001, 2024-03-20T09:42:37.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.13336271599999999, 2024-03-20T09:42:37.344 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:42:37.344 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:42:37.344 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:42:37.344 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.27190", 2024-03-20T09:42:37.344 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:42:37.344 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 123984 2024-03-20T09:42:37.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.345 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:42:37.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:42:37.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.395746+0000", 2024-03-20T09:42:37.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:42:37.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.395746+0000", 2024-03-20T09:42:37.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:42:37.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.395747+0000", 2024-03-20T09:42:37.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4499999999999997e-07 2024-03-20T09:42:37.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:42:37.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.395786+0000", 2024-03-20T09:42:37.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9069000000000002e-05 2024-03-20T09:42:37.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.346 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:42:37.346 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.395799+0000", 2024-03-20T09:42:37.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3759e-05 2024-03-20T09:42:37.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:42:37.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.395806+0000", 2024-03-20T09:42:37.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4659999999999996e-06 2024-03-20T09:42:37.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:42:37.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.440786+0000", 2024-03-20T09:42:37.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.044980309000000003 2024-03-20T09:42:37.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.348 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:42:37.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.440950+0000", 2024-03-20T09:42:37.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00016368599999999999 2024-03-20T09:42:37.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.348 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 6,9", 2024-03-20T09:42:37.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.441104+0000", 2024-03-20T09:42:37.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000153836 2024-03-20T09:42:37.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.348 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:42:37.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.447929+0000", 2024-03-20T09:42:37.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0068248320000000003 2024-03-20T09:42:37.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.466419+0000", 2024-03-20T09:42:37.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018490587999999999 2024-03-20T09:42:37.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.529034+0000", 2024-03-20T09:42:37.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.062614388000000007 2024-03-20T09:42:37.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:42:37.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.529065+0000", 2024-03-20T09:42:37.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1760000000000001e-05 2024-03-20T09:42:37.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:42:37.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.529109+0000", 2024-03-20T09:42:37.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3477999999999999e-05 2024-03-20T09:42:37.350 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.350 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:42:37.350 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.351 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.27190.0:124209 9.1 9:880b00b9:::benchmark_data_smithi122_210164_object124208:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1097)", 2024-03-20T09:42:37.351 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:41:55.716518+0000", 2024-03-20T09:42:37.351 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 41.535207845999999, 2024-03-20T09:42:37.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.14587341100000001, 2024-03-20T09:42:37.351 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:42:37.351 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:42:37.351 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:42:37.351 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.27190", 2024-03-20T09:42:37.351 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:42:37.351 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 124209 2024-03-20T09:42:37.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.351 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:42:37.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:42:37.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.716518+0000", 2024-03-20T09:42:37.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:42:37.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.716518+0000", 2024-03-20T09:42:37.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:42:37.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.716519+0000", 2024-03-20T09:42:37.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.186e-06 2024-03-20T09:42:37.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:42:37.353 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.716606+0000", 2024-03-20T09:42:37.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6298999999999995e-05 2024-03-20T09:42:37.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:42:37.353 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.716617+0000", 2024-03-20T09:42:37.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1134999999999999e-05 2024-03-20T09:42:37.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:42:37.354 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.716625+0000", 2024-03-20T09:42:37.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.551e-06 2024-03-20T09:42:37.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.354 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:42:37.354 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.762821+0000", 2024-03-20T09:42:37.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.046195780999999998 2024-03-20T09:42:37.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.354 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:42:37.354 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.762985+0000", 2024-03-20T09:42:37.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00016389200000000001 2024-03-20T09:42:37.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 6,9", 2024-03-20T09:42:37.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.763166+0000", 2024-03-20T09:42:37.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018084500000000001 2024-03-20T09:42:37.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:42:37.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.775319+0000", 2024-03-20T09:42:37.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012152735 2024-03-20T09:42:37.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.786621+0000", 2024-03-20T09:42:37.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011302708999999999 2024-03-20T09:42:37.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.862246+0000", 2024-03-20T09:42:37.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.075624734999999998 2024-03-20T09:42:37.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:42:37.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.862292+0000", 2024-03-20T09:42:37.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.5974999999999999e-05 2024-03-20T09:42:37.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:42:37.357 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.862392+0000", 2024-03-20T09:42:37.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9568000000000006e-05 2024-03-20T09:42:37.357 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.357 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:42:37.357 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.357 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.27190.0:124215 9.1 9:832c0385:::benchmark_data_smithi122_210164_object124214:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1097)", 2024-03-20T09:42:37.357 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:41:55.724256+0000", 2024-03-20T09:42:37.357 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 41.527470385999997, 2024-03-20T09:42:37.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.138265049, 2024-03-20T09:42:37.357 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:42:37.357 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:42:37.358 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:42:37.358 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.27190", 2024-03-20T09:42:37.358 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:42:37.358 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 124215 2024-03-20T09:42:37.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.358 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:42:37.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.358 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:42:37.358 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.724256+0000", 2024-03-20T09:42:37.358 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.358 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:42:37.358 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.724256+0000", 2024-03-20T09:42:37.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:42:37.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.724257+0000", 2024-03-20T09:42:37.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.319e-06 2024-03-20T09:42:37.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:42:37.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.724286+0000", 2024-03-20T09:42:37.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9481000000000001e-05 2024-03-20T09:42:37.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.360 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:42:37.360 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.724302+0000", 2024-03-20T09:42:37.360 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5807e-05 2024-03-20T09:42:37.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.360 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:42:37.360 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.724310+0000", 2024-03-20T09:42:37.360 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0469999999999994e-06 2024-03-20T09:42:37.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.360 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:42:37.360 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.766903+0000", 2024-03-20T09:42:37.360 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.042593204000000003 2024-03-20T09:42:37.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.361 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:42:37.361 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.767068+0000", 2024-03-20T09:42:37.361 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000164303 2024-03-20T09:42:37.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.361 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 6,9", 2024-03-20T09:42:37.361 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.767230+0000", 2024-03-20T09:42:37.361 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000162409 2024-03-20T09:42:37.361 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.361 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.361 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:42:37.361 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.775324+0000", 2024-03-20T09:42:37.361 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0080938109999999994 2024-03-20T09:42:37.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.362 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.362 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.786656+0000", 2024-03-20T09:42:37.362 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.01133179 2024-03-20T09:42:37.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.362 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.362 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.862442+0000", 2024-03-20T09:42:37.362 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.075785805999999997 2024-03-20T09:42:37.362 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.362 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.362 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:42:37.363 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.862474+0000", 2024-03-20T09:42:37.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2552999999999998e-05 2024-03-20T09:42:37.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.363 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:42:37.363 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.862521+0000", 2024-03-20T09:42:37.363 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6519e-05 2024-03-20T09:42:37.363 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.363 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:42:37.363 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.363 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.363 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.363 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.27190.0:124301 9.1 9:8fa45a44:::benchmark_data_smithi122_210164_object124300:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1097)", 2024-03-20T09:42:37.363 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:41:55.845128+0000", 2024-03-20T09:42:37.364 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 41.406598162999998, 2024-03-20T09:42:37.364 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.13629161200000001, 2024-03-20T09:42:37.364 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:42:37.364 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:42:37.364 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:42:37.364 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.27190", 2024-03-20T09:42:37.364 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:42:37.364 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 124301 2024-03-20T09:42:37.364 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.364 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:42:37.364 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.364 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:42:37.364 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.845128+0000", 2024-03-20T09:42:37.365 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.365 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:42:37.365 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.845128+0000", 2024-03-20T09:42:37.365 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.365 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:42:37.365 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.845129+0000", 2024-03-20T09:42:37.365 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4300000000000001e-06 2024-03-20T09:42:37.365 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.365 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.366 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:42:37.366 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.845167+0000", 2024-03-20T09:42:37.366 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7630999999999999e-05 2024-03-20T09:42:37.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.366 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:42:37.366 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.845181+0000", 2024-03-20T09:42:37.366 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4297000000000001e-05 2024-03-20T09:42:37.366 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.366 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.366 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:42:37.366 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.845189+0000", 2024-03-20T09:42:37.367 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1899999999999995e-06 2024-03-20T09:42:37.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.367 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:42:37.367 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.864103+0000", 2024-03-20T09:42:37.367 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.018913345000000002 2024-03-20T09:42:37.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.367 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:42:37.367 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.864258+0000", 2024-03-20T09:42:37.367 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000155165 2024-03-20T09:42:37.367 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.367 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.367 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 6,9", 2024-03-20T09:42:37.368 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.864404+0000", 2024-03-20T09:42:37.368 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000145828 2024-03-20T09:42:37.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:42:37.368 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.871465+0000", 2024-03-20T09:42:37.368 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0070610389999999999 2024-03-20T09:42:37.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.368 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.886857+0000", 2024-03-20T09:42:37.368 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015392312 2024-03-20T09:42:37.368 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.368 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.368 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.369 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.981290+0000", 2024-03-20T09:42:37.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.094432686000000002 2024-03-20T09:42:37.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.369 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:42:37.369 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.981357+0000", 2024-03-20T09:42:37.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6810000000000006e-05 2024-03-20T09:42:37.369 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.369 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.369 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:42:37.369 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:55.981419+0000", 2024-03-20T09:42:37.369 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2879000000000006e-05 2024-03-20T09:42:37.369 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.369 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:42:37.369 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.370 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.370 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.27190.0:126205 9.1 9:821c8175:::benchmark_data_smithi122_210164_object126204:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1098)", 2024-03-20T09:42:37.370 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:41:59.076029+0000", 2024-03-20T09:42:37.370 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 38.175696897999998, 2024-03-20T09:42:37.370 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.17505138200000001, 2024-03-20T09:42:37.370 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:42:37.370 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:42:37.370 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:42:37.370 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.27190", 2024-03-20T09:42:37.370 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:42:37.370 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 126205 2024-03-20T09:42:37.370 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.371 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:42:37.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.371 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:42:37.371 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:59.076029+0000", 2024-03-20T09:42:37.371 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.371 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:42:37.371 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:59.076029+0000", 2024-03-20T09:42:37.371 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.371 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.371 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.371 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:42:37.371 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:59.076030+0000", 2024-03-20T09:42:37.372 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.288e-06 2024-03-20T09:42:37.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.372 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:42:37.372 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:59.076098+0000", 2024-03-20T09:42:37.372 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7371000000000003e-05 2024-03-20T09:42:37.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.372 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:42:37.372 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:59.076112+0000", 2024-03-20T09:42:37.372 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4219000000000001e-05 2024-03-20T09:42:37.372 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.372 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.373 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:42:37.373 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:59.076120+0000", 2024-03-20T09:42:37.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3299999999999999e-06 2024-03-20T09:42:37.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.373 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:42:37.373 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:59.130024+0000", 2024-03-20T09:42:37.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.053903949 2024-03-20T09:42:37.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.373 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:42:37.373 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:59.130177+0000", 2024-03-20T09:42:37.373 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000152569 2024-03-20T09:42:37.373 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.373 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.374 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 6,9", 2024-03-20T09:42:37.374 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:59.130341+0000", 2024-03-20T09:42:37.374 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000164403 2024-03-20T09:42:37.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.374 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:42:37.374 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:59.134426+0000", 2024-03-20T09:42:37.374 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040849119999999996 2024-03-20T09:42:37.374 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.374 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.374 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.374 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:59.145600+0000", 2024-03-20T09:42:37.374 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011173501000000001 2024-03-20T09:42:37.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.375 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.375 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:59.251001+0000", 2024-03-20T09:42:37.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.10540137099999999 2024-03-20T09:42:37.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.375 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:42:37.375 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:59.251037+0000", 2024-03-20T09:42:37.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5695000000000003e-05 2024-03-20T09:42:37.375 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.375 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.375 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:42:37.375 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:59.251081+0000", 2024-03-20T09:42:37.375 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3773999999999998e-05 2024-03-20T09:42:37.376 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.376 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:42:37.376 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.376 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.376 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.376 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.27190.0:127331 9.1 9:8291106b:::benchmark_data_smithi122_210164_object127330:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1098)", 2024-03-20T09:42:37.376 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:42:00.946898+0000", 2024-03-20T09:42:37.376 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 36.304827613, 2024-03-20T09:42:37.376 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.133071523, 2024-03-20T09:42:37.376 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:42:37.376 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:42:37.376 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:42:37.376 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.27190", 2024-03-20T09:42:37.376 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:42:37.376 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 127331 2024-03-20T09:42:37.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.377 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:42:37.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.377 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:42:37.377 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:00.946898+0000", 2024-03-20T09:42:37.377 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.377 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:42:37.377 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:00.946898+0000", 2024-03-20T09:42:37.377 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:42:37.377 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.377 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.377 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:42:37.378 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:00.946899+0000", 2024-03-20T09:42:37.378 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9699999999999994e-07 2024-03-20T09:42:37.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.378 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:42:37.378 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:00.946971+0000", 2024-03-20T09:42:37.378 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1802000000000001e-05 2024-03-20T09:42:37.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.378 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:42:37.378 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:00.946987+0000", 2024-03-20T09:42:37.378 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5668e-05 2024-03-20T09:42:37.378 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.378 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.379 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:42:37.379 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:00.946996+0000", 2024-03-20T09:42:37.379 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3659999999999998e-06 2024-03-20T09:42:37.379 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.379 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.379 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:42:37.379 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:00.947060+0000", 2024-03-20T09:42:37.379 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3313999999999997e-05 2024-03-20T09:42:37.379 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.379 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.379 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:42:37.379 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:00.947217+0000", 2024-03-20T09:42:37.379 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000157577 2024-03-20T09:42:37.379 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.380 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 6,9", 2024-03-20T09:42:37.380 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:00.947395+0000", 2024-03-20T09:42:37.380 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00017763300000000001 2024-03-20T09:42:37.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.380 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:42:37.380 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:00.951979+0000", 2024-03-20T09:42:37.380 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0045842440000000003 2024-03-20T09:42:37.380 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.380 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.380 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.380 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:00.964516+0000", 2024-03-20T09:42:37.380 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012536545 2024-03-20T09:42:37.381 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.381 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.381 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:42:37.381 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:01.079880+0000", 2024-03-20T09:42:37.381 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.115364694 2024-03-20T09:42:37.381 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.381 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.381 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:42:37.381 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:01.079918+0000", 2024-03-20T09:42:37.381 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7305000000000003e-05 2024-03-20T09:42:37.381 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:42:37.381 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:42:37.381 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:42:37.381 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:01.079970+0000", 2024-03-20T09:42:37.382 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2377999999999999e-05 2024-03-20T09:42:37.382 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.382 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:42:37.382 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.382 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:42:37.382 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:42:37.382 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:42:37.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:37.384 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:37.384 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:42:37.384 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:42:37.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:37 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2674356152' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:42:37.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:37 smithi115 ceph-mon[72476]: pgmap v2685: 47 pgs: 47 active+clean; 6.7 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 917 op/s 2024-03-20T09:42:37.485 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:42:37.535 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:37.575 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:42:37.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:37.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:42:37.681 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:42:37.725 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:37.767 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:42:37.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:37.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:42:37.873 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:42:37.913 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:37.960 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:42:37.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:37.967 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:42:38.068 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:42:38.113 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:38.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:38 smithi122 ceph-mon[73495]: 1.10 scrub starts 2024-03-20T09:42:38.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:38 smithi122 ceph-mon[73495]: 1.10 scrub ok 2024-03-20T09:42:38.370 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:38 smithi026 ceph-mon[68309]: 1.10 scrub starts 2024-03-20T09:42:38.370 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:38 smithi026 ceph-mon[68309]: 1.10 scrub ok 2024-03-20T09:42:38.371 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:42:38.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:38.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:42:38.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:38 smithi115 ceph-mon[72476]: 1.10 scrub starts 2024-03-20T09:42:38.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:38 smithi115 ceph-mon[72476]: 1.10 scrub ok 2024-03-20T09:42:38.477 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:42:38.524 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:38.554 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:42:38.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:38.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:42:38.584 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:42:38.664 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:42:38.731 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:38.777 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:42:38.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:38.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:42:38.885 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:42:38.930 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:38.933 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-20T09:42:38.960 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:42:38.963 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:38.964 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:42:39.044 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:39 smithi115 ceph-mon[72476]: pgmap v2686: 47 pgs: 47 active+clean; 6.6 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 920 op/s 2024-03-20T09:42:39.065 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:42:39.101 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:39.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:39 smithi122 ceph-mon[73495]: pgmap v2686: 47 pgs: 47 active+clean; 6.6 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 920 op/s 2024-03-20T09:42:39.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:39 smithi026 ceph-mon[68309]: pgmap v2686: 47 pgs: 47 active+clean; 6.6 GiB data, 29 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 920 op/s 2024-03-20T09:42:39.346 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:42:39.756 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:42:39.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:39.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:42:39.867 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:42:39.905 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:39.944 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:42:39.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:39.949 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:42:40.022 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:42:40.050 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:42:40.104 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:40.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:40 smithi122 ceph-mon[73495]: 1.1a scrub starts 2024-03-20T09:42:40.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:40 smithi122 ceph-mon[73495]: 1.1a scrub ok 2024-03-20T09:42:40.297 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:40 smithi026 ceph-mon[68309]: 1.1a scrub starts 2024-03-20T09:42:40.297 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:40 smithi026 ceph-mon[68309]: 1.1a scrub ok 2024-03-20T09:42:40.320 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:42:40.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:40.328 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:40 smithi115 ceph-mon[72476]: 1.1a scrub starts 2024-03-20T09:42:40.328 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:40 smithi115 ceph-mon[72476]: 1.1a scrub ok 2024-03-20T09:42:40.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:42:40.430 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:42:40.457 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:40.811 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:42:40.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:40.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:42:40.917 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:42:40.959 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:41.051 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.2 to 1 (10000) 2024-03-20T09:42:41.108 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:42:41.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:41.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:42:41.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:41 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2813066091' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:42:41.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:41 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:42:41.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:41 smithi026 ceph-mon[68309]: pgmap v2687: 47 pgs: 47 active+clean; 6.4 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 916 op/s 2024-03-20T09:42:41.215 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:42:41.278 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:41.308 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:42:41.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:41.314 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:42:41.317 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:41 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2813066091' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:42:41.317 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:41 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:42:41.317 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:41 smithi122 ceph-mon[73495]: pgmap v2687: 47 pgs: 47 active+clean; 6.4 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 916 op/s 2024-03-20T09:42:41.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:41 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2813066091' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:42:41.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:41 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:42:41.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:41 smithi115 ceph-mon[72476]: pgmap v2687: 47 pgs: 47 active+clean; 6.4 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 916 op/s 2024-03-20T09:42:41.415 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:42:41.434 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:42:41.455 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:41.572 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:42:41.572 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:42:41.572 INFO:teuthology.orchestra.run.smithi115.stdout: "num_ops": 0 2024-03-20T09:42:41.572 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:42:41.622 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:42:41.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:41.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:42:41.729 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:42:41.760 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:41.889 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:42:41.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:41.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:42:41.994 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:42:42.043 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:42.163 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:42 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-20T09:42:42.164 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:42 smithi026 ceph-mon[68309]: osdmap e1106: 12 total, 12 up, 11 in 2024-03-20T09:42:42.267 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:42 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-20T09:42:42.267 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:42 smithi115 ceph-mon[72476]: osdmap e1106: 12 total, 12 up, 11 in 2024-03-20T09:42:42.279 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:42:42.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:42.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:42:42.317 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-20T09:42:42.385 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:42:42.423 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:42.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:42 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-20T09:42:42.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:42 smithi122 ceph-mon[73495]: osdmap e1106: 12 total, 12 up, 11 in 2024-03-20T09:42:42.468 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:42:42.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:42.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:42:42.505 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 5 1 2024-03-20T09:42:42.574 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:42:42.617 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:42.657 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:42:42.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:42.660 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:42:42.663 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:42:42.761 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:42:42.783 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:42:42.832 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:43.094 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:42:43.134 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:43 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:42:43.134 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:43 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:42:43.134 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:43 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:42:43.134 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:43 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2858176516' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:42:43.134 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:43 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:42:43.134 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:43 smithi115 ceph-mon[72476]: pgmap v2689: 47 pgs: 47 active+clean; 6.4 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 942 op/s 2024-03-20T09:42:43.134 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:43 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:42:43.204 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:42:43.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:43 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:42:43.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:43 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:42:43.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:43 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:42:43.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:43 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2858176516' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:42:43.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:43 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:42:43.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:43 smithi026 ceph-mon[68309]: pgmap v2689: 47 pgs: 47 active+clean; 6.4 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 942 op/s 2024-03-20T09:42:43.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:43 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:42:43.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:43.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:42:43.311 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:42:43.355 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:43.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:43 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:42:43.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:43 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:42:43.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:43 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:42:43.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:43 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2858176516' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:42:43.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:43 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:42:43.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:43 smithi122 ceph-mon[73495]: pgmap v2689: 47 pgs: 47 active+clean; 6.4 GiB data, 28 GiB used, 955 GiB / 983 GiB avail; 0 B/s wr, 942 op/s 2024-03-20T09:42:43.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:43 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:42:43.562 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:42:43.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:43.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:42:43.670 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:42:43.726 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:43.753 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:42:43.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:43.761 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:42:43.862 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:42:43.908 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:43.936 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:42:43.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:43.941 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:42:44.041 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:42:44.078 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:44 smithi122 ceph-mon[73495]: 1.2 deep-scrub starts 2024-03-20T09:42:44.078 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:44 smithi122 ceph-mon[73495]: 1.2 deep-scrub ok 2024-03-20T09:42:44.078 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:44 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:42:44.078 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:44 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2858176516' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:42:44.078 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:44 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:42:44.078 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:44 smithi122 ceph-mon[73495]: osdmap e1107: 12 total, 12 up, 11 in 2024-03-20T09:42:44.078 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:44 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:42:44.078 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:44 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:42:44.094 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:44.127 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:42:44.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:44.133 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:42:44.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:44 smithi026 ceph-mon[68309]: 1.2 deep-scrub starts 2024-03-20T09:42:44.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:44 smithi026 ceph-mon[68309]: 1.2 deep-scrub ok 2024-03-20T09:42:44.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:44 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:42:44.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:44 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2858176516' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:42:44.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:44 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:42:44.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:44 smithi026 ceph-mon[68309]: osdmap e1107: 12 total, 12 up, 11 in 2024-03-20T09:42:44.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:44 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:42:44.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:44 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:42:44.234 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:42:44.288 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:44.314 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:42:44.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:44.322 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:42:44.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:44 smithi115 ceph-mon[72476]: 1.2 deep-scrub starts 2024-03-20T09:42:44.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:44 smithi115 ceph-mon[72476]: 1.2 deep-scrub ok 2024-03-20T09:42:44.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:44 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:42:44.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:44 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2858176516' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:42:44.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:44 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:42:44.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:44 smithi115 ceph-mon[72476]: osdmap e1107: 12 total, 12 up, 11 in 2024-03-20T09:42:44.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:44 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:42:44.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:44 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:42:44.423 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:42:44.502 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:44.695 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:42:44.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:44.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:42:44.799 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:42:44.857 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:45.225 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:42:45.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:45.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:42:45.281 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:42:45.332 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:42:45.351 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:42:45.372 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:45 smithi026 ceph-mon[68309]: pgmap v2691: 47 pgs: 47 active+clean; 6.3 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 772 op/s 2024-03-20T09:42:45.372 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:45 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:42:45.372 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:45 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:42:45.372 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:45 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:42:45.375 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:45.389 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:42:45.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:45 smithi115 ceph-mon[72476]: pgmap v2691: 47 pgs: 47 active+clean; 6.3 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 772 op/s 2024-03-20T09:42:45.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:45 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:42:45.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:45 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:42:45.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:45 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:42:45.427 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:42:45.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:45 smithi122 ceph-mon[73495]: pgmap v2691: 47 pgs: 47 active+clean; 6.3 GiB data, 28 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 772 op/s 2024-03-20T09:42:45.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:45 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:42:45.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:45 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:42:45.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:45 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:42:45.456 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:42:45.492 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:42:45.526 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:42:45.552 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:42:45.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:45.597 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:42:45.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:42:45.597 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:42:45.618 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:42:45.646 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:42:45.677 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:42:45.698 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:42:45.711 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:42:45.724 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:42:45.725 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:42:45.725 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:42:45.725 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:42:45.725 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:42:45.725 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:42:45.725 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:42:45.725 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:42:45.725 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:42:45.725 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:42:45.725 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:42:45.725 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:42:45.741 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:45.871 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:42:45.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:45.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:42:45.959 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:42:45.959 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:42:45.975 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:42:46.045 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:46.094 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:46 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2477613374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:42:46.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:46 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2477613374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:42:46.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:46 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2477613374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:42:46.773 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:42:47.112 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:47 smithi026 ceph-mon[68309]: pgmap v2692: 47 pgs: 47 active+clean; 6.1 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-03-20T09:42:47.327 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:42:47.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:47.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:42:47.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:47 smithi115 ceph-mon[72476]: pgmap v2692: 47 pgs: 47 active+clean; 6.1 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-03-20T09:42:47.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:47 smithi122 ceph-mon[73495]: pgmap v2692: 47 pgs: 47 active+clean; 6.1 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-03-20T09:42:47.434 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:42:47.486 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:42:47.490 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:47.520 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:42:47.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:47.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:42:47.628 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:42:47.692 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:47.723 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:42:47.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:47.731 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:42:47.832 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:42:47.869 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:48.421 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-20T09:42:48.539 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:42:48.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:48.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:42:48.647 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:42:48.730 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:48.781 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:42:48.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:48.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:42:48.858 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:42:48.888 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:42:48.938 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:48.982 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.5 to 1 (10000) 2024-03-20T09:42:48.992 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:42:49.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:48 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/151906277' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:42:49.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:48 smithi115 ceph-mon[72476]: pgmap v2693: 47 pgs: 47 active+clean; 6.1 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 660 op/s 2024-03-20T09:42:49.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:48 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/151906277' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:42:49.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:48 smithi122 ceph-mon[73495]: pgmap v2693: 47 pgs: 47 active+clean; 6.1 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 660 op/s 2024-03-20T09:42:49.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:48 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/151906277' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:42:49.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:48 smithi026 ceph-mon[68309]: pgmap v2693: 47 pgs: 47 active+clean; 6.1 GiB data, 27 GiB used, 956 GiB / 983 GiB avail; 0 B/s wr, 660 op/s 2024-03-20T09:42:49.713 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:42:49.720 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:49.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:42:49.822 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:42:49.869 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:50.032 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:42:50.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:50.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:42:50.146 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:42:50.181 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:50.206 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:42:50.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:50 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/151906277' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-20T09:42:50.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:50 smithi026 ceph-mon[68309]: osdmap e1108: 12 total, 12 up, 11 in 2024-03-20T09:42:50.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:50.210 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:42:50.311 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:42:50.347 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:50.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:50.385 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:42:50.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:42:50.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:50 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/151906277' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-20T09:42:50.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:50 smithi115 ceph-mon[72476]: osdmap e1108: 12 total, 12 up, 11 in 2024-03-20T09:42:50.402 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 6 1 2024-03-20T09:42:50.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:50 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/151906277' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-20T09:42:50.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:50 smithi122 ceph-mon[73495]: osdmap e1108: 12 total, 12 up, 11 in 2024-03-20T09:42:50.486 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:42:50.535 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:50.564 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:42:50.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:50.570 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:42:50.590 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:42:50.672 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:42:50.730 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:51.012 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:42:51.164 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:51 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:42:51.164 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:51 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:42:51.164 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:51 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3865503412' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:42:51.164 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:51 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:42:51.164 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:51 smithi026 ceph-mon[68309]: pgmap v2695: 47 pgs: 47 active+clean; 5.9 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 935 op/s 2024-03-20T09:42:51.289 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:51 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:42:51.289 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:51 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:42:51.289 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:51 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3865503412' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:42:51.289 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:51 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:42:51.289 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:51 smithi115 ceph-mon[72476]: pgmap v2695: 47 pgs: 47 active+clean; 5.9 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 935 op/s 2024-03-20T09:42:51.306 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:42:51.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:51.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:42:51.415 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:42:51.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:51 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:42:51.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:51 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:42:51.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:51 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3865503412' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:42:51.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:51 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:42:51.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:51 smithi122 ceph-mon[73495]: pgmap v2695: 47 pgs: 47 active+clean; 5.9 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 935 op/s 2024-03-20T09:42:51.455 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:51.499 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:42:51.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:51.503 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:42:51.604 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:42:51.685 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:51.962 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:42:51.972 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:51.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:42:52.011 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:42:52.011 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:42:52.074 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:42:52.114 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:52 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:42:52.114 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:52 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:42:52.114 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:52 smithi026 ceph-mon[68309]: osdmap e1109: 12 total, 12 up, 11 in 2024-03-20T09:42:52.120 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:52.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:52 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:42:52.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:52 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:42:52.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:52 smithi115 ceph-mon[72476]: osdmap e1109: 12 total, 12 up, 11 in 2024-03-20T09:42:52.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:52 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:42:52.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:52 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:42:52.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:52 smithi122 ceph-mon[73495]: osdmap e1109: 12 total, 12 up, 11 in 2024-03-20T09:42:52.490 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:42:52.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:52.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:42:52.595 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:42:52.632 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:52.661 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:42:52.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:52.669 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:42:52.743 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-20T09:42:52.769 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:42:52.824 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:52.849 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:42:52.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:52.856 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:42:52.957 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:42:52.993 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:53.211 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:42:53.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:53 smithi115 ceph-mon[72476]: pgmap v2697: 47 pgs: 47 active+clean; 5.8 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 913 op/s 2024-03-20T09:42:53.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:53 smithi122 ceph-mon[73495]: pgmap v2697: 47 pgs: 47 active+clean; 5.8 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 913 op/s 2024-03-20T09:42:53.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:53 smithi026 ceph-mon[68309]: pgmap v2697: 47 pgs: 47 active+clean; 5.8 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 913 op/s 2024-03-20T09:42:54.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:54 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:42:54.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:54 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:42:54.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:54 smithi115 ceph-mon[72476]: osdmap e1110: 12 total, 12 up, 11 in 2024-03-20T09:42:54.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:54 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:42:54.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:54 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:42:54.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:54 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:42:54.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:54 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:42:54.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:54 smithi122 ceph-mon[73495]: osdmap e1110: 12 total, 12 up, 11 in 2024-03-20T09:42:54.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:54 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:42:54.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:54 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:42:54.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:54 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:42:54.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:54 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:42:54.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:54 smithi026 ceph-mon[68309]: osdmap e1110: 12 total, 12 up, 11 in 2024-03-20T09:42:54.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:54 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:42:54.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:54 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:42:54.551 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:42:55.079 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:55 smithi026 ceph-mon[68309]: osdmap e1111: 12 total, 12 up, 11 in 2024-03-20T09:42:55.079 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:55 smithi026 ceph-mon[68309]: pgmap v2700: 47 pgs: 47 active+clean; 5.7 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 919 op/s 2024-03-20T09:42:55.296 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:42:55.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:55.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:42:55.304 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:42:55.304 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:42:55.333 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:55 smithi115 ceph-mon[72476]: osdmap e1111: 12 total, 12 up, 11 in 2024-03-20T09:42:55.333 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:55 smithi115 ceph-mon[72476]: pgmap v2700: 47 pgs: 47 active+clean; 5.7 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 919 op/s 2024-03-20T09:42:55.401 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:42:55.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:55 smithi122 ceph-mon[73495]: osdmap e1111: 12 total, 12 up, 11 in 2024-03-20T09:42:55.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:55 smithi122 ceph-mon[73495]: pgmap v2700: 47 pgs: 47 active+clean; 5.7 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 0 B/s wr, 919 op/s 2024-03-20T09:42:55.448 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:55.765 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:42:55.798 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:42:55.855 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:42:55.864 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:42:55.895 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:42:55.944 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:42:55.970 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:42:56.002 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:42:56.035 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:42:56.081 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:42:56.106 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:42:56.138 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:42:56.140 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:42:56.140 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:42:56.140 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:42:56.140 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:42:56.140 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:42:56.141 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:42:56.141 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:42:56.141 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:42:56.141 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:42:56.141 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:42:56.141 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:42:56.141 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:42:56.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:56 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:42:56.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:56 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:42:56.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:56 smithi115 ceph-mon[72476]: osdmap e1112: 12 total, 12 up, 11 in 2024-03-20T09:42:56.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:56 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1553536676' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:42:56.408 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:56 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:42:56.408 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:56 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:42:56.408 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:56 smithi026 ceph-mon[68309]: osdmap e1112: 12 total, 12 up, 11 in 2024-03-20T09:42:56.408 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:56 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1553536676' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:42:56.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:56 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:42:56.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:56 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:42:56.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:56 smithi122 ceph-mon[73495]: osdmap e1112: 12 total, 12 up, 11 in 2024-03-20T09:42:56.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:56 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1553536676' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:42:56.595 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:42:56.618 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:42:56.630 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:56.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:42:56.732 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:42:56.810 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:57.035 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-20T09:42:57.035 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:42:57.042 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:57.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:42:57.053 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:57 smithi026 ceph-mon[68309]: pgmap v2702: 47 pgs: 47 active+clean; 5.6 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 898 op/s 2024-03-20T09:42:57.142 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:42:57.199 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:57.257 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:42:57.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:57.264 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:57 smithi115 ceph-mon[72476]: pgmap v2702: 47 pgs: 47 active+clean; 5.6 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 898 op/s 2024-03-20T09:42:57.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:42:57.366 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:42:57.420 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:57.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:57 smithi122 ceph-mon[73495]: pgmap v2702: 47 pgs: 47 active+clean; 5.6 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 898 op/s 2024-03-20T09:42:57.457 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:42:57.457 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:42:57.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:57.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:42:57.563 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:42:57.635 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:57.645 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:42:57.678 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:42:57.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:57.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:42:57.797 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:42:57.859 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:57.926 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:42:57.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:57.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:42:58.030 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:42:58.077 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:58 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3666769386' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-20T09:42:58.077 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:58 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-20T09:42:58.082 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.6 to 1 (10000) 2024-03-20T09:42:58.093 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:58.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:58 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3666769386' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-20T09:42:58.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:58 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-20T09:42:58.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:58 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3666769386' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-20T09:42:58.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:58 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-20T09:42:58.965 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 7 1 2024-03-20T09:42:59.070 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:42:59.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:42:59.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:42:59.117 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:42:59.175 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:42:59.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:59 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-20T09:42:59.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:59 smithi026 ceph-mon[68309]: osdmap e1113: 12 total, 12 up, 11 in 2024-03-20T09:42:59.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:42:59 smithi026 ceph-mon[68309]: pgmap v2704: 47 pgs: 47 active+clean; 5.6 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 792 op/s 2024-03-20T09:42:59.241 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:42:59.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:59 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-20T09:42:59.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:59 smithi115 ceph-mon[72476]: osdmap e1113: 12 total, 12 up, 11 in 2024-03-20T09:42:59.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:42:59 smithi115 ceph-mon[72476]: pgmap v2704: 47 pgs: 47 active+clean; 5.6 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 792 op/s 2024-03-20T09:42:59.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:59 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-20T09:42:59.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:59 smithi122 ceph-mon[73495]: osdmap e1113: 12 total, 12 up, 11 in 2024-03-20T09:42:59.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:42:59 smithi122 ceph-mon[73495]: pgmap v2704: 47 pgs: 47 active+clean; 5.6 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 792 op/s 2024-03-20T09:43:00.395 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:43:00.396 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:43:00.396 INFO:teuthology.orchestra.run.smithi115.stdout: "complaint_time": 30, 2024-03-20T09:43:00.396 INFO:teuthology.orchestra.run.smithi115.stdout: "num_blocked_ops": 0 2024-03-20T09:43:00.396 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:43:00.611 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:43:00.619 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:00.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:43:00.675 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:43:00.720 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:43:00.726 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:43:00.761 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:00.844 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:43:00.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:00.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:43:00.954 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:43:01.024 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:01.114 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:01 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:43:01.114 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:01 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:43:01.114 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:01 smithi026 ceph-mon[68309]: osdmap e1114: 12 total, 12 up, 11 in 2024-03-20T09:43:01.114 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:01 smithi026 ceph-mon[68309]: pgmap v2706: 47 pgs: 47 active+clean; 5.4 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 1.07k op/s 2024-03-20T09:43:01.149 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-20T09:43:01.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:01 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:43:01.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:01 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:43:01.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:01 smithi115 ceph-mon[72476]: osdmap e1114: 12 total, 12 up, 11 in 2024-03-20T09:43:01.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:01 smithi115 ceph-mon[72476]: pgmap v2706: 47 pgs: 47 active+clean; 5.4 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 1.07k op/s 2024-03-20T09:43:01.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:01 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:43:01.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:01 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:43:01.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:01 smithi122 ceph-mon[73495]: osdmap e1114: 12 total, 12 up, 11 in 2024-03-20T09:43:01.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:01 smithi122 ceph-mon[73495]: pgmap v2706: 47 pgs: 47 active+clean; 5.4 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 0 B/s wr, 1.07k op/s 2024-03-20T09:43:01.487 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:43:02.124 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:02 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:43:02.124 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:02 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:43:02.124 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:02 smithi115 ceph-mon[72476]: osdmap e1115: 12 total, 12 up, 11 in 2024-03-20T09:43:02.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:02 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:43:02.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:02 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:43:02.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:02 smithi122 ceph-mon[73495]: osdmap e1115: 12 total, 12 up, 11 in 2024-03-20T09:43:02.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:02 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:43:02.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:02 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:43:02.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:02 smithi026 ceph-mon[68309]: osdmap e1115: 12 total, 12 up, 11 in 2024-03-20T09:43:02.691 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:43:02.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:02.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:43:02.799 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:43:02.860 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:02.887 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:43:02.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:02.892 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:43:02.992 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:43:03.027 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:03.055 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:43:03.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:03.058 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:43:03.143 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:03 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:43:03.144 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:03 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:43:03.144 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:03 smithi122 ceph-mon[73495]: osdmap e1116: 12 total, 12 up, 11 in 2024-03-20T09:43:03.144 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:03 smithi122 ceph-mon[73495]: pgmap v2709: 47 pgs: 47 active+clean; 5.3 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 968 op/s 2024-03-20T09:43:03.159 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:43:03.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:03 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:43:03.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:03 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:43:03.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:03 smithi026 ceph-mon[68309]: osdmap e1116: 12 total, 12 up, 11 in 2024-03-20T09:43:03.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:03 smithi026 ceph-mon[68309]: pgmap v2709: 47 pgs: 47 active+clean; 5.3 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 968 op/s 2024-03-20T09:43:03.210 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:03.231 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:43:03.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:03.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:43:03.336 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:43:03.381 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:03.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:03 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:43:03.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:03 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:43:03.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:03 smithi115 ceph-mon[72476]: osdmap e1116: 12 total, 12 up, 11 in 2024-03-20T09:43:03.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:03 smithi115 ceph-mon[72476]: pgmap v2709: 47 pgs: 47 active+clean; 5.3 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 968 op/s 2024-03-20T09:43:03.409 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:43:03.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:03.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:43:03.517 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:43:03.558 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:04.153 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:43:04.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:04 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4205339887' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:43:04.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:04 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4205339887' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:43:04.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:04 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4205339887' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:43:04.516 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:43:04.516 INFO:teuthology.orchestra.run.smithi115.stdout: "size": 20, 2024-03-20T09:43:04.516 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 600, 2024-03-20T09:43:04.517 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [ 2024-03-20T09:43:04.517 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.517 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221363 7.12 7:4d24fa2c:::benchmark_data_smithi122_187077_object94752:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e940)", 2024-03-20T09:43:04.517 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:46.500402+0000", 2024-03-20T09:43:04.517 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 498.01680419600001, 2024-03-20T09:43:04.517 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1879329379999999, 2024-03-20T09:43:04.517 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:04.517 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:43:04.517 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:43:04.517 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:43:04.517 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:43:04.517 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221363 2024-03-20T09:43:04.517 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.517 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:04.518 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.518 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:04.518 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.500402+0000", 2024-03-20T09:43:04.518 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.518 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.518 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.518 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:04.518 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.500402+0000", 2024-03-20T09:43:04.518 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.518 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.518 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.518 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:04.518 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.500403+0000", 2024-03-20T09:43:04.518 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.7100000000000011e-07 2024-03-20T09:43:04.519 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.519 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.519 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:04.519 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.500408+0000", 2024-03-20T09:43:04.519 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2129999999999999e-06 2024-03-20T09:43:04.519 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.519 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.519 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:04.519 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.500420+0000", 2024-03-20T09:43:04.519 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2519e-05 2024-03-20T09:43:04.519 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.519 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.519 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:04.519 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.500428+0000", 2024-03-20T09:43:04.519 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.6329999999999996e-06 2024-03-20T09:43:04.520 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.520 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.520 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.520 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.500484+0000", 2024-03-20T09:43:04.520 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6329000000000002e-05 2024-03-20T09:43:04.520 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.520 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.520 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:43:04.520 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.500606+0000", 2024-03-20T09:43:04.520 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000121401 2024-03-20T09:43:04.520 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.520 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.520 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 0,2,3", 2024-03-20T09:43:04.521 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.500957+0000", 2024-03-20T09:43:04.521 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000351274 2024-03-20T09:43:04.521 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.521 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.521 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:43:04.521 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.504667+0000", 2024-03-20T09:43:04.521 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0037103090000000002 2024-03-20T09:43:04.521 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.521 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.521 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:04.521 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.505148+0000", 2024-03-20T09:43:04.521 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00048108100000000001 2024-03-20T09:43:04.521 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.522 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.522 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:04.522 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.510137+0000", 2024-03-20T09:43:04.522 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0049886030000000003 2024-03-20T09:43:04.522 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.522 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.522 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.522 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.666198+0000", 2024-03-20T09:43:04.522 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.15606091599999999 2024-03-20T09:43:04.522 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.522 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.522 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:43:04.522 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.666208+0000", 2024-03-20T09:43:04.523 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0298e-05 2024-03-20T09:43:04.523 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.523 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.523 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.523 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.688325+0000", 2024-03-20T09:43:04.523 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.022116459 2024-03-20T09:43:04.523 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.523 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.523 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:04.523 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.688335+0000", 2024-03-20T09:43:04.523 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.9320000000000008e-06 2024-03-20T09:43:04.523 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.523 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:04.523 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.524 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.524 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.524 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221365 7.12 7:496d3fcf:::benchmark_data_smithi122_187077_object94754:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e940)", 2024-03-20T09:43:04.524 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:46.500453+0000", 2024-03-20T09:43:04.524 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 498.01675304999998, 2024-03-20T09:43:04.524 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1882682339999999, 2024-03-20T09:43:04.524 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:04.524 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:43:04.524 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:43:04.524 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:43:04.524 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:43:04.524 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221365 2024-03-20T09:43:04.524 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.524 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:04.524 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.525 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:04.525 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.500453+0000", 2024-03-20T09:43:04.525 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.525 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.525 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.525 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:04.525 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.500453+0000", 2024-03-20T09:43:04.525 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.525 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.525 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.525 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:04.525 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.500453+0000", 2024-03-20T09:43:04.525 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5400000000000002e-07 2024-03-20T09:43:04.525 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.526 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.526 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:04.526 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.500456+0000", 2024-03-20T09:43:04.526 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3360000000000002e-06 2024-03-20T09:43:04.526 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.526 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.526 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:04.526 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.500466+0000", 2024-03-20T09:43:04.526 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0223e-05 2024-03-20T09:43:04.526 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.526 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.526 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:04.526 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.500471+0000", 2024-03-20T09:43:04.527 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6130000000000003e-06 2024-03-20T09:43:04.527 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.527 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.527 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.527 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.502725+0000", 2024-03-20T09:43:04.527 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0022532020000000002 2024-03-20T09:43:04.527 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.527 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.527 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:43:04.527 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.502849+0000", 2024-03-20T09:43:04.527 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000124595 2024-03-20T09:43:04.527 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.527 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.527 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 0,2,3", 2024-03-20T09:43:04.528 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.503098+0000", 2024-03-20T09:43:04.528 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00024829499999999999 2024-03-20T09:43:04.528 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.528 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.528 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:43:04.528 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.504782+0000", 2024-03-20T09:43:04.528 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001684533 2024-03-20T09:43:04.528 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.528 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.528 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:04.528 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.507368+0000", 2024-03-20T09:43:04.528 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0025858510000000001 2024-03-20T09:43:04.528 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.528 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.529 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:04.529 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.510175+0000", 2024-03-20T09:43:04.529 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0028068479999999998 2024-03-20T09:43:04.529 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.529 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.529 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.529 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.666236+0000", 2024-03-20T09:43:04.529 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.15606133699999999 2024-03-20T09:43:04.529 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.529 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.529 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:43:04.529 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.666248+0000", 2024-03-20T09:43:04.529 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2027e-05 2024-03-20T09:43:04.529 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.530 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.530 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.530 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.688712+0000", 2024-03-20T09:43:04.530 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0224640629999999 2024-03-20T09:43:04.530 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.530 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.530 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:04.530 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.688721+0000", 2024-03-20T09:43:04.530 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.8570000000000001e-06 2024-03-20T09:43:04.530 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.530 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:04.531 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.531 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.531 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.531 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221389 7.12 7.24f0b3b2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e941)", 2024-03-20T09:43:04.531 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:46.694341+0000", 2024-03-20T09:43:04.531 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 497.82286478700001, 2024-03-20T09:43:04.531 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.99442238800000005, 2024-03-20T09:43:04.531 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:04.531 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:43:04.531 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:43:04.531 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:43:04.531 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:43:04.531 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221389 2024-03-20T09:43:04.531 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.532 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:04.532 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.532 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:04.532 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.694341+0000", 2024-03-20T09:43:04.532 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.532 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.532 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.532 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:04.532 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.694341+0000", 2024-03-20T09:43:04.532 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.532 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.532 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.532 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:04.532 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.694342+0000", 2024-03-20T09:43:04.532 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5899999999999996e-07 2024-03-20T09:43:04.533 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.533 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.533 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:04.533 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.694345+0000", 2024-03-20T09:43:04.533 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.4300000000000002e-06 2024-03-20T09:43:04.533 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.533 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.533 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:04.533 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.694356+0000", 2024-03-20T09:43:04.533 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0692e-05 2024-03-20T09:43:04.533 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.533 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.533 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:04.533 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.694362+0000", 2024-03-20T09:43:04.533 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8119999999999996e-06 2024-03-20T09:43:04.534 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.534 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.534 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.534 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.694400+0000", 2024-03-20T09:43:04.534 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.8597000000000003e-05 2024-03-20T09:43:04.534 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.534 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.534 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:43:04.534 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.694406+0000", 2024-03-20T09:43:04.534 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6400000000000002e-06 2024-03-20T09:43:04.534 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.534 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.534 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.534 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.688751+0000", 2024-03-20T09:43:04.534 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.99434531000000004 2024-03-20T09:43:04.535 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.535 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.535 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:04.535 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.688763+0000", 2024-03-20T09:43:04.535 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2347999999999999e-05 2024-03-20T09:43:04.535 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.535 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:04.535 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.535 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.535 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.535 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221410 7.12 7.4e88c252 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e941)", 2024-03-20T09:43:04.535 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:46.729117+0000", 2024-03-20T09:43:04.535 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 497.78808881200001, 2024-03-20T09:43:04.535 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.95966132800000004, 2024-03-20T09:43:04.535 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:04.535 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:43:04.536 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:43:04.536 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:43:04.536 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:43:04.536 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221410 2024-03-20T09:43:04.536 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.536 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:04.536 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.536 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:04.536 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.729117+0000", 2024-03-20T09:43:04.536 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.536 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.536 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.536 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:04.536 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.729117+0000", 2024-03-20T09:43:04.537 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.537 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.537 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.537 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:04.537 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.729118+0000", 2024-03-20T09:43:04.537 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9999999999999998e-07 2024-03-20T09:43:04.537 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.537 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.537 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:04.537 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.729120+0000", 2024-03-20T09:43:04.537 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9189999999999999e-06 2024-03-20T09:43:04.537 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.537 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.537 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:04.537 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.729133+0000", 2024-03-20T09:43:04.537 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2908000000000001e-05 2024-03-20T09:43:04.538 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.538 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.538 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:04.538 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.729139+0000", 2024-03-20T09:43:04.538 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2979999999999999e-06 2024-03-20T09:43:04.538 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.538 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.538 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.538 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.729203+0000", 2024-03-20T09:43:04.538 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4102000000000004e-05 2024-03-20T09:43:04.538 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.538 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.539 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:43:04.539 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.729209+0000", 2024-03-20T09:43:04.539 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9619999999999998e-06 2024-03-20T09:43:04.539 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.539 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.539 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.539 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.688774+0000", 2024-03-20T09:43:04.539 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.95956517799999996 2024-03-20T09:43:04.539 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.539 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.539 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:04.539 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.688778+0000", 2024-03-20T09:43:04.539 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4610000000000001e-06 2024-03-20T09:43:04.539 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.539 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:04.540 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.540 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.540 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.540 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221471 7.12 7.c01da832 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e941)", 2024-03-20T09:43:04.540 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:46.800088+0000", 2024-03-20T09:43:04.540 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 497.71711779899999, 2024-03-20T09:43:04.540 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.88872703399999997, 2024-03-20T09:43:04.540 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:04.540 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:43:04.540 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:43:04.540 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:43:04.540 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:43:04.540 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221471 2024-03-20T09:43:04.541 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.541 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:04.541 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.541 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:04.541 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.800088+0000", 2024-03-20T09:43:04.541 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.541 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.541 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.541 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:04.541 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.800088+0000", 2024-03-20T09:43:04.541 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.541 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.541 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.542 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:04.542 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.800089+0000", 2024-03-20T09:43:04.542 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.341e-06 2024-03-20T09:43:04.542 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.542 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.542 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:04.542 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.800098+0000", 2024-03-20T09:43:04.542 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.3639999999999999e-06 2024-03-20T09:43:04.542 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.542 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.542 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:04.542 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.800113+0000", 2024-03-20T09:43:04.542 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4805e-05 2024-03-20T09:43:04.542 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.542 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.543 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:04.543 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.800120+0000", 2024-03-20T09:43:04.543 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.5700000000000004e-06 2024-03-20T09:43:04.543 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.543 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.543 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.543 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.800196+0000", 2024-03-20T09:43:04.543 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.6099000000000004e-05 2024-03-20T09:43:04.543 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.543 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.543 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:43:04.543 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.800203+0000", 2024-03-20T09:43:04.543 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.5950000000000004e-06 2024-03-20T09:43:04.543 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.543 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.543 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.544 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.688806+0000", 2024-03-20T09:43:04.544 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.88860339099999996 2024-03-20T09:43:04.544 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.544 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.544 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:04.544 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.688815+0000", 2024-03-20T09:43:04.544 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.8689999999999993e-06 2024-03-20T09:43:04.544 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.544 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:04.544 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.544 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.544 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.544 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221511 7.12 7.31478952 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e941)", 2024-03-20T09:43:04.544 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:46.846911+0000", 2024-03-20T09:43:04.544 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 497.67029462699998, 2024-03-20T09:43:04.545 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.841917727, 2024-03-20T09:43:04.545 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:04.545 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:43:04.545 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:43:04.545 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:43:04.545 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:43:04.545 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221511 2024-03-20T09:43:04.545 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.545 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:04.545 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.545 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:04.545 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.846911+0000", 2024-03-20T09:43:04.545 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.545 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.545 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.545 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:04.546 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.846911+0000", 2024-03-20T09:43:04.546 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.546 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.546 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.546 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:04.546 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.846912+0000", 2024-03-20T09:43:04.546 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1319999999999999e-06 2024-03-20T09:43:04.546 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.546 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.546 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:04.546 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.846918+0000", 2024-03-20T09:43:04.546 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1989999999999997e-06 2024-03-20T09:43:04.546 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.546 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.546 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:04.547 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.846930+0000", 2024-03-20T09:43:04.547 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2761e-05 2024-03-20T09:43:04.547 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.547 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.547 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:04.547 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.846938+0000", 2024-03-20T09:43:04.547 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.123e-06 2024-03-20T09:43:04.547 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.547 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.547 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.547 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.846981+0000", 2024-03-20T09:43:04.547 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.2945999999999998e-05 2024-03-20T09:43:04.547 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.547 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.547 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:43:04.548 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.846989+0000", 2024-03-20T09:43:04.548 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7570000000000004e-06 2024-03-20T09:43:04.548 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.548 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.548 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.548 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.688825+0000", 2024-03-20T09:43:04.548 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.84183545000000004 2024-03-20T09:43:04.548 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.548 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.548 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:04.548 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.688829+0000", 2024-03-20T09:43:04.548 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.3590000000000001e-06 2024-03-20T09:43:04.548 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.549 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:04.549 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.549 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.549 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.549 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221573 7.12 7.a7b14f12 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e941)", 2024-03-20T09:43:04.549 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:46.914545+0000", 2024-03-20T09:43:04.549 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 497.602660442, 2024-03-20T09:43:04.549 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.77439364200000005, 2024-03-20T09:43:04.549 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:04.549 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:43:04.549 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:43:04.549 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:43:04.549 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:43:04.549 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221573 2024-03-20T09:43:04.550 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.550 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:04.550 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.550 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:04.550 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.914545+0000", 2024-03-20T09:43:04.550 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.550 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.550 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.550 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:04.550 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.914545+0000", 2024-03-20T09:43:04.550 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.550 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.550 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.550 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:04.551 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.914547+0000", 2024-03-20T09:43:04.551 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.186e-06 2024-03-20T09:43:04.551 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.551 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.551 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:04.551 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.914552+0000", 2024-03-20T09:43:04.551 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4750000000000001e-06 2024-03-20T09:43:04.551 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.551 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.551 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:04.551 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.914567+0000", 2024-03-20T09:43:04.551 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.519e-05 2024-03-20T09:43:04.551 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.551 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.551 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:04.552 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.914576+0000", 2024-03-20T09:43:04.552 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.7630000000000006e-06 2024-03-20T09:43:04.552 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.552 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.552 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.552 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.914639+0000", 2024-03-20T09:43:04.552 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.347e-05 2024-03-20T09:43:04.552 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.552 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.552 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:43:04.552 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.914650+0000", 2024-03-20T09:43:04.552 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0397999999999999e-05 2024-03-20T09:43:04.552 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.552 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.553 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.553 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.688843+0000", 2024-03-20T09:43:04.553 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.77419343399999996 2024-03-20T09:43:04.553 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.553 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.553 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:04.553 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.688939+0000", 2024-03-20T09:43:04.553 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.5725999999999998e-05 2024-03-20T09:43:04.553 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.553 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:04.553 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.553 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.553 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.553 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221579 7.12 7.e0ab76d2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e941)", 2024-03-20T09:43:04.554 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:46.920156+0000", 2024-03-20T09:43:04.554 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 497.59704945300001, 2024-03-20T09:43:04.554 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.76870872099999998, 2024-03-20T09:43:04.554 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:04.554 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:43:04.554 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:43:04.554 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:43:04.554 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:43:04.554 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221579 2024-03-20T09:43:04.554 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.554 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:04.554 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.554 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:04.554 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.920156+0000", 2024-03-20T09:43:04.555 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.555 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.555 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.555 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:04.555 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.920156+0000", 2024-03-20T09:43:04.555 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.555 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.555 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.555 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:04.555 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.920157+0000", 2024-03-20T09:43:04.555 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0869999999999999e-06 2024-03-20T09:43:04.555 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.555 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.555 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:04.555 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.920162+0000", 2024-03-20T09:43:04.555 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9300000000000002e-06 2024-03-20T09:43:04.556 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.556 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.556 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:04.556 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.920177+0000", 2024-03-20T09:43:04.556 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4321999999999999e-05 2024-03-20T09:43:04.556 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.556 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.556 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:04.556 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.920190+0000", 2024-03-20T09:43:04.556 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.329e-05 2024-03-20T09:43:04.556 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.556 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.556 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.557 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.920259+0000", 2024-03-20T09:43:04.557 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.9474999999999995e-05 2024-03-20T09:43:04.557 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.557 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.557 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:43:04.557 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.920272+0000", 2024-03-20T09:43:04.557 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2228e-05 2024-03-20T09:43:04.557 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.557 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.557 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.557 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.688861+0000", 2024-03-20T09:43:04.557 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.76858884900000002 2024-03-20T09:43:04.557 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.557 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.557 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:04.557 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.688865+0000", 2024-03-20T09:43:04.558 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5399999999999997e-06 2024-03-20T09:43:04.558 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.558 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:04.558 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.558 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.558 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.558 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221590 7.12 7.6fbad972 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e941)", 2024-03-20T09:43:04.558 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:46.931680+0000", 2024-03-20T09:43:04.558 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 497.585525727, 2024-03-20T09:43:04.558 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.75729486499999998, 2024-03-20T09:43:04.558 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:04.558 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:43:04.558 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:43:04.558 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:43:04.558 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:43:04.559 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221590 2024-03-20T09:43:04.559 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.559 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:04.559 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.559 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:04.559 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.931680+0000", 2024-03-20T09:43:04.559 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.559 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.559 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.559 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:04.559 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.931680+0000", 2024-03-20T09:43:04.559 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.559 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.559 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.559 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:04.559 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.931682+0000", 2024-03-20T09:43:04.560 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6700000000000001e-06 2024-03-20T09:43:04.560 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.560 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.560 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:04.560 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.931687+0000", 2024-03-20T09:43:04.560 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.626e-06 2024-03-20T09:43:04.560 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.560 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.560 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:04.560 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.931701+0000", 2024-03-20T09:43:04.560 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3550000000000001e-05 2024-03-20T09:43:04.560 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.560 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.560 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:04.560 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.931709+0000", 2024-03-20T09:43:04.560 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.644e-06 2024-03-20T09:43:04.561 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.561 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.561 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.561 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.931772+0000", 2024-03-20T09:43:04.561 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2957e-05 2024-03-20T09:43:04.561 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.561 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.561 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:43:04.561 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.931783+0000", 2024-03-20T09:43:04.561 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1347e-05 2024-03-20T09:43:04.561 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.561 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.561 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.561 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.688968+0000", 2024-03-20T09:43:04.561 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.75718535300000001 2024-03-20T09:43:04.562 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.562 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.562 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:04.562 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.688975+0000", 2024-03-20T09:43:04.562 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.7179999999999999e-06 2024-03-20T09:43:04.562 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.562 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:04.562 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.562 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.562 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.562 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221593 7.12 7.1ad4eaf2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e941)", 2024-03-20T09:43:04.562 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:46.933926+0000", 2024-03-20T09:43:04.562 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 497.58327947800001, 2024-03-20T09:43:04.562 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.75508512000000005, 2024-03-20T09:43:04.562 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:04.563 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:43:04.563 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:43:04.563 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:43:04.563 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:43:04.563 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221593 2024-03-20T09:43:04.563 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.563 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:04.563 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.563 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:04.563 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.933926+0000", 2024-03-20T09:43:04.563 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.563 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.563 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.563 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:04.563 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.933926+0000", 2024-03-20T09:43:04.564 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.564 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.564 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.564 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:04.564 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.933927+0000", 2024-03-20T09:43:04.564 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.7999999999999995e-07 2024-03-20T09:43:04.564 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.564 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.564 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:04.564 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.933930+0000", 2024-03-20T09:43:04.564 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.1599999999999998e-06 2024-03-20T09:43:04.564 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.564 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.564 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:04.564 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.933938+0000", 2024-03-20T09:43:04.564 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7300000000000005e-06 2024-03-20T09:43:04.565 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.565 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.565 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:04.565 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.933942+0000", 2024-03-20T09:43:04.565 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5340000000000001e-06 2024-03-20T09:43:04.565 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.565 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.565 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.565 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.933998+0000", 2024-03-20T09:43:04.565 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5813000000000002e-05 2024-03-20T09:43:04.565 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.565 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.565 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:43:04.565 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:46.934006+0000", 2024-03-20T09:43:04.565 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.8420000000000005e-06 2024-03-20T09:43:04.565 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.566 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.566 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.566 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689002+0000", 2024-03-20T09:43:04.566 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.75499634800000004 2024-03-20T09:43:04.566 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.566 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.566 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:04.566 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689011+0000", 2024-03-20T09:43:04.566 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.0129999999999999e-06 2024-03-20T09:43:04.566 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.566 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:04.566 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.566 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.566 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.566 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221706 7.12 7.6687b8b2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e941)", 2024-03-20T09:43:04.567 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:47.085552+0000", 2024-03-20T09:43:04.567 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 497.43165350499999, 2024-03-20T09:43:04.567 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.60347295400000001, 2024-03-20T09:43:04.567 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:04.567 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:43:04.567 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:43:04.567 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:43:04.567 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:43:04.567 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221706 2024-03-20T09:43:04.567 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.567 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:04.567 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.567 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:04.567 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.085552+0000", 2024-03-20T09:43:04.567 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.567 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.567 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.568 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:04.568 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.085552+0000", 2024-03-20T09:43:04.568 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.568 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.568 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.568 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:04.568 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.085554+0000", 2024-03-20T09:43:04.568 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.359e-06 2024-03-20T09:43:04.568 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.568 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.568 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:04.568 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.085560+0000", 2024-03-20T09:43:04.568 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0039999999999996e-06 2024-03-20T09:43:04.568 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.568 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.568 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:04.569 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.085574+0000", 2024-03-20T09:43:04.569 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4746000000000001e-05 2024-03-20T09:43:04.569 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.569 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.569 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:04.569 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.085583+0000", 2024-03-20T09:43:04.569 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.816e-06 2024-03-20T09:43:04.569 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.569 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.569 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.569 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.085645+0000", 2024-03-20T09:43:04.569 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2094999999999994e-05 2024-03-20T09:43:04.569 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.569 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.569 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:43:04.570 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.085655+0000", 2024-03-20T09:43:04.570 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.6309999999999998e-06 2024-03-20T09:43:04.570 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.570 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.570 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.570 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689021+0000", 2024-03-20T09:43:04.570 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.60336599300000004 2024-03-20T09:43:04.570 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.570 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.570 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:04.570 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689025+0000", 2024-03-20T09:43:04.570 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.3100000000000002e-06 2024-03-20T09:43:04.570 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.570 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:04.570 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.571 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.571 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.571 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221363 7.12 7.345f24b2 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e941)", 2024-03-20T09:43:04.571 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:47.146989+0000", 2024-03-20T09:43:04.571 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 497.370216513, 2024-03-20T09:43:04.571 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54205875999999997, 2024-03-20T09:43:04.571 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:04.571 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:43:04.571 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:43:04.571 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:43:04.571 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:43:04.571 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221363 2024-03-20T09:43:04.571 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.571 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:04.571 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.572 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:04.572 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.146989+0000", 2024-03-20T09:43:04.572 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.572 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.572 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.572 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:04.572 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.146989+0000", 2024-03-20T09:43:04.572 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.572 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.572 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.572 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:04.572 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.146990+0000", 2024-03-20T09:43:04.572 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.6000000000000003e-07 2024-03-20T09:43:04.572 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.572 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.573 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:04.573 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.146993+0000", 2024-03-20T09:43:04.573 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.326e-06 2024-03-20T09:43:04.573 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.573 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.573 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:04.573 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147000+0000", 2024-03-20T09:43:04.573 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.9519999999999997e-06 2024-03-20T09:43:04.573 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.573 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.573 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:04.573 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147006+0000", 2024-03-20T09:43:04.573 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1930000000000001e-06 2024-03-20T09:43:04.573 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.573 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.573 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.574 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147054+0000", 2024-03-20T09:43:04.574 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.7976999999999998e-05 2024-03-20T09:43:04.574 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.574 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.574 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:43:04.574 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147060+0000", 2024-03-20T09:43:04.574 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1630000000000001e-06 2024-03-20T09:43:04.574 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.574 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.574 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.574 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689039+0000", 2024-03-20T09:43:04.574 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54197916000000002 2024-03-20T09:43:04.574 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.574 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.574 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:04.575 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689048+0000", 2024-03-20T09:43:04.575 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.2289999999999992e-06 2024-03-20T09:43:04.575 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.575 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:04.575 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.575 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.575 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.575 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221365 7.12 7.f3fcb692 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e941)", 2024-03-20T09:43:04.575 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:47.147019+0000", 2024-03-20T09:43:04.575 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 497.37018719700001, 2024-03-20T09:43:04.575 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54204330700000003, 2024-03-20T09:43:04.575 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:04.575 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:43:04.575 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:43:04.575 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:43:04.575 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:43:04.576 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221365 2024-03-20T09:43:04.576 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.576 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:04.576 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.576 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:04.576 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147019+0000", 2024-03-20T09:43:04.576 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.576 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.576 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.576 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:04.576 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147019+0000", 2024-03-20T09:43:04.576 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.576 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.576 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.576 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:04.576 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147019+0000", 2024-03-20T09:43:04.577 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.7300000000000002e-07 2024-03-20T09:43:04.577 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.577 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.577 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:04.577 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147021+0000", 2024-03-20T09:43:04.577 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7230000000000001e-06 2024-03-20T09:43:04.577 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.577 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.577 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:04.577 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147025+0000", 2024-03-20T09:43:04.577 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5770000000000004e-06 2024-03-20T09:43:04.577 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.577 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.577 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:04.577 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147028+0000", 2024-03-20T09:43:04.577 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.0989999999999999e-06 2024-03-20T09:43:04.578 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.578 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.578 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.578 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147089+0000", 2024-03-20T09:43:04.578 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0770999999999998e-05 2024-03-20T09:43:04.578 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.578 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.578 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:43:04.578 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147095+0000", 2024-03-20T09:43:04.578 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1770000000000003e-06 2024-03-20T09:43:04.578 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.578 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.578 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.578 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689057+0000", 2024-03-20T09:43:04.578 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54196217199999996 2024-03-20T09:43:04.579 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.579 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.579 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:04.579 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689062+0000", 2024-03-20T09:43:04.579 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.515e-06 2024-03-20T09:43:04.579 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.579 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:04.579 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.579 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.579 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.579 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221471 7.12 7.c01da832 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e941)", 2024-03-20T09:43:04.579 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:47.147080+0000", 2024-03-20T09:43:04.579 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 497.37012576900003, 2024-03-20T09:43:04.579 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54204185800000004, 2024-03-20T09:43:04.579 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:04.579 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:43:04.580 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:43:04.580 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:43:04.580 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:43:04.580 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221471 2024-03-20T09:43:04.580 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.580 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:04.580 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.580 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:04.580 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147080+0000", 2024-03-20T09:43:04.580 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.580 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.580 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.580 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:04.580 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147080+0000", 2024-03-20T09:43:04.580 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.581 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.581 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.581 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:04.581 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147080+0000", 2024-03-20T09:43:04.581 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0100000000000001e-07 2024-03-20T09:43:04.581 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.581 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.581 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:04.581 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147084+0000", 2024-03-20T09:43:04.581 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.1729999999999997e-06 2024-03-20T09:43:04.581 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.581 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.581 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:04.581 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147089+0000", 2024-03-20T09:43:04.581 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.1030000000000004e-06 2024-03-20T09:43:04.581 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.582 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.582 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:04.582 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147092+0000", 2024-03-20T09:43:04.582 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.038e-06 2024-03-20T09:43:04.582 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.582 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.582 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.582 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147172+0000", 2024-03-20T09:43:04.582 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.0086000000000005e-05 2024-03-20T09:43:04.582 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.582 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.582 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:43:04.582 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147175+0000", 2024-03-20T09:43:04.583 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.4929999999999998e-06 2024-03-20T09:43:04.583 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.583 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.583 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.583 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689114+0000", 2024-03-20T09:43:04.583 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54193840100000001 2024-03-20T09:43:04.583 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.583 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.583 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:04.583 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689122+0000", 2024-03-20T09:43:04.583 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.3629999999999994e-06 2024-03-20T09:43:04.583 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.584 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:04.584 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.584 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.584 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.584 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221511 7.12 7.31478952 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e941)", 2024-03-20T09:43:04.584 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:47.147097+0000", 2024-03-20T09:43:04.584 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 497.37010929500002, 2024-03-20T09:43:04.584 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54203903899999994, 2024-03-20T09:43:04.584 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:04.584 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:43:04.584 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:43:04.584 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:43:04.584 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:43:04.584 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221511 2024-03-20T09:43:04.584 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.584 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:04.585 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.585 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:04.585 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147097+0000", 2024-03-20T09:43:04.585 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.585 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.585 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.585 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:04.585 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147097+0000", 2024-03-20T09:43:04.585 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.585 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.585 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.585 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:04.585 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147097+0000", 2024-03-20T09:43:04.585 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1799999999999999e-07 2024-03-20T09:43:04.585 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.586 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.586 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:04.586 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147098+0000", 2024-03-20T09:43:04.586 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2279999999999999e-06 2024-03-20T09:43:04.586 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.586 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.586 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:04.586 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147101+0000", 2024-03-20T09:43:04.586 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.0850000000000001e-06 2024-03-20T09:43:04.586 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.586 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.586 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:04.586 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147104+0000", 2024-03-20T09:43:04.586 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6570000000000001e-06 2024-03-20T09:43:04.586 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.587 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.587 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.587 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147206+0000", 2024-03-20T09:43:04.587 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000102191 2024-03-20T09:43:04.587 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.587 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.587 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:43:04.587 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147210+0000", 2024-03-20T09:43:04.587 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6320000000000001e-06 2024-03-20T09:43:04.587 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.587 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.587 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.587 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689131+0000", 2024-03-20T09:43:04.587 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54192172900000002 2024-03-20T09:43:04.587 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.587 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.588 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:04.588 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689136+0000", 2024-03-20T09:43:04.588 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.2989999999999998e-06 2024-03-20T09:43:04.588 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.588 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:04.588 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.588 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.588 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.588 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221573 7.12 7.a7b14f12 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e941)", 2024-03-20T09:43:04.588 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:47.147114+0000", 2024-03-20T09:43:04.588 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 497.37009158299998, 2024-03-20T09:43:04.588 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54204345600000003, 2024-03-20T09:43:04.588 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:04.588 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:43:04.588 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:43:04.588 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:43:04.589 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:43:04.589 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221573 2024-03-20T09:43:04.589 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.589 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:04.589 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.589 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:04.589 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147114+0000", 2024-03-20T09:43:04.589 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.589 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.589 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.589 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:04.589 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147114+0000", 2024-03-20T09:43:04.589 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.589 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.589 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.590 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:04.590 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147114+0000", 2024-03-20T09:43:04.590 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.23e-07 2024-03-20T09:43:04.590 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.590 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.590 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:04.590 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147116+0000", 2024-03-20T09:43:04.590 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.333e-06 2024-03-20T09:43:04.590 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.590 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.590 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:04.590 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147119+0000", 2024-03-20T09:43:04.590 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6509999999999999e-06 2024-03-20T09:43:04.590 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.590 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.590 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:04.591 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147122+0000", 2024-03-20T09:43:04.591 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8279999999999999e-06 2024-03-20T09:43:04.591 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.591 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.591 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.591 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147238+0000", 2024-03-20T09:43:04.591 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000115467 2024-03-20T09:43:04.591 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.591 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.591 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:43:04.591 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147244+0000", 2024-03-20T09:43:04.591 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.3400000000000003e-06 2024-03-20T09:43:04.591 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.591 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.592 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.592 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689149+0000", 2024-03-20T09:43:04.592 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54190539699999996 2024-03-20T09:43:04.592 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.592 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.592 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:04.592 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689158+0000", 2024-03-20T09:43:04.592 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.2169999999999994e-06 2024-03-20T09:43:04.592 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.592 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:04.592 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.592 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.592 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.592 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221579 7.12 7.e0ab76d2 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e941)", 2024-03-20T09:43:04.592 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:47.147132+0000", 2024-03-20T09:43:04.592 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 497.370073655, 2024-03-20T09:43:04.593 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54204047399999999, 2024-03-20T09:43:04.593 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:04.593 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:43:04.593 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:43:04.593 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:43:04.593 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:43:04.593 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221579 2024-03-20T09:43:04.593 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.593 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:04.593 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.593 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:04.593 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147132+0000", 2024-03-20T09:43:04.593 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.593 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.593 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.593 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:04.594 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147132+0000", 2024-03-20T09:43:04.594 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.594 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.594 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.594 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:04.594 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147132+0000", 2024-03-20T09:43:04.594 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.16e-07 2024-03-20T09:43:04.594 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.594 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.594 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:04.594 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147134+0000", 2024-03-20T09:43:04.594 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.291e-06 2024-03-20T09:43:04.594 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.594 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.594 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:04.595 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147138+0000", 2024-03-20T09:43:04.595 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.1189999999999999e-06 2024-03-20T09:43:04.595 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.595 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.595 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:04.595 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147141+0000", 2024-03-20T09:43:04.595 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.0319999999999999e-06 2024-03-20T09:43:04.595 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.595 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.595 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.595 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147275+0000", 2024-03-20T09:43:04.595 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013443600000000001 2024-03-20T09:43:04.595 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.595 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.595 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:43:04.596 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147282+0000", 2024-03-20T09:43:04.596 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4749999999999998e-06 2024-03-20T09:43:04.596 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.596 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.596 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.596 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689167+0000", 2024-03-20T09:43:04.596 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54188511299999997 2024-03-20T09:43:04.596 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.596 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.596 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:04.596 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689173+0000", 2024-03-20T09:43:04.596 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7919999999999998e-06 2024-03-20T09:43:04.596 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.596 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:04.597 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.597 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.597 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.597 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221590 7.12 7.6fbad972 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e941)", 2024-03-20T09:43:04.597 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:47.147144+0000", 2024-03-20T09:43:04.597 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 497.370061335, 2024-03-20T09:43:04.597 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54206811799999999, 2024-03-20T09:43:04.597 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:04.597 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:43:04.597 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:43:04.597 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:43:04.597 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:43:04.597 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221590 2024-03-20T09:43:04.597 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.597 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:04.597 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.598 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:04.598 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147144+0000", 2024-03-20T09:43:04.598 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.598 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.598 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.598 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:04.598 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147144+0000", 2024-03-20T09:43:04.598 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.598 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.598 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.598 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:04.598 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147145+0000", 2024-03-20T09:43:04.598 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.99e-07 2024-03-20T09:43:04.598 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.598 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.598 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:04.599 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147146+0000", 2024-03-20T09:43:04.599 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.629e-06 2024-03-20T09:43:04.599 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.599 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.599 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:04.599 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147149+0000", 2024-03-20T09:43:04.599 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.1020000000000001e-06 2024-03-20T09:43:04.599 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.599 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.599 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:04.599 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147152+0000", 2024-03-20T09:43:04.599 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.0060000000000001e-06 2024-03-20T09:43:04.599 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.599 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.599 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.600 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147315+0000", 2024-03-20T09:43:04.600 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000162547 2024-03-20T09:43:04.600 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.600 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.600 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:43:04.600 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147319+0000", 2024-03-20T09:43:04.600 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.3209999999999998e-06 2024-03-20T09:43:04.600 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.600 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.600 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.600 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689204+0000", 2024-03-20T09:43:04.600 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54188447900000003 2024-03-20T09:43:04.600 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.600 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.600 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:04.600 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689213+0000", 2024-03-20T09:43:04.601 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.8349999999999993e-06 2024-03-20T09:43:04.601 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.601 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:04.601 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.601 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.601 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.601 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221593 7.12 7.1ad4eaf2 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e941)", 2024-03-20T09:43:04.601 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:47.147166+0000", 2024-03-20T09:43:04.601 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 497.370039706, 2024-03-20T09:43:04.601 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54207133299999999, 2024-03-20T09:43:04.601 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:04.601 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:43:04.601 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:43:04.601 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:43:04.601 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:43:04.601 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221593 2024-03-20T09:43:04.602 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.602 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:04.602 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.602 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:04.602 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147166+0000", 2024-03-20T09:43:04.602 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.602 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.602 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.602 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:04.602 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147166+0000", 2024-03-20T09:43:04.602 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.602 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.602 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.602 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:04.602 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147166+0000", 2024-03-20T09:43:04.603 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0900000000000001e-07 2024-03-20T09:43:04.603 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.603 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.603 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:04.603 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147168+0000", 2024-03-20T09:43:04.603 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.308e-06 2024-03-20T09:43:04.603 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.603 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.603 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:04.603 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147173+0000", 2024-03-20T09:43:04.603 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2619999999999998e-06 2024-03-20T09:43:04.603 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.603 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.603 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:04.603 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147176+0000", 2024-03-20T09:43:04.603 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.7209999999999999e-06 2024-03-20T09:43:04.604 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.604 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.604 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.604 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147348+0000", 2024-03-20T09:43:04.604 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00017213800000000001 2024-03-20T09:43:04.604 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.604 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.604 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:43:04.604 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147351+0000", 2024-03-20T09:43:04.604 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6940000000000001e-06 2024-03-20T09:43:04.604 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.604 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.604 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.604 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689233+0000", 2024-03-20T09:43:04.604 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54188119899999998 2024-03-20T09:43:04.604 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.605 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.605 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:04.605 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689237+0000", 2024-03-20T09:43:04.605 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8019999999999999e-06 2024-03-20T09:43:04.605 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.605 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:04.605 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.605 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.605 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.605 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.26605.0:221706 7.12 7.6687b8b2 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e941)", 2024-03-20T09:43:04.605 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:47.147193+0000", 2024-03-20T09:43:04.605 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 497.37001241500002, 2024-03-20T09:43:04.605 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54207810999999995, 2024-03-20T09:43:04.605 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:04.605 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "reached pg", 2024-03-20T09:43:04.605 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:43:04.606 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.26605", 2024-03-20T09:43:04.606 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/891426581", 2024-03-20T09:43:04.606 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 221706 2024-03-20T09:43:04.606 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.606 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:04.606 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.606 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:04.606 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147193+0000", 2024-03-20T09:43:04.606 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.606 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.606 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.606 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:04.606 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147193+0000", 2024-03-20T09:43:04.606 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:04.606 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.607 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.607 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:04.607 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147194+0000", 2024-03-20T09:43:04.607 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.34e-07 2024-03-20T09:43:04.607 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.607 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.607 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:04.607 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147195+0000", 2024-03-20T09:43:04.607 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.643e-06 2024-03-20T09:43:04.607 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.607 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.607 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:04.607 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147199+0000", 2024-03-20T09:43:04.607 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7309999999999999e-06 2024-03-20T09:43:04.607 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.607 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.607 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:04.608 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147202+0000", 2024-03-20T09:43:04.608 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.96e-06 2024-03-20T09:43:04.608 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.608 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.608 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.608 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147364+0000", 2024-03-20T09:43:04.608 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000162306 2024-03-20T09:43:04.608 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.608 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.608 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for peered", 2024-03-20T09:43:04.608 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.147368+0000", 2024-03-20T09:43:04.608 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.3900000000000002e-06 2024-03-20T09:43:04.608 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.608 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.608 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:04.608 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689263+0000", 2024-03-20T09:43:04.609 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.54189511499999998 2024-03-20T09:43:04.609 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:04.609 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:04.609 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:04.609 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.689272+0000", 2024-03-20T09:43:04.609 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.7309999999999999e-06 2024-03-20T09:43:04.609 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.609 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:04.609 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.609 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:04.609 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:04.609 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:43:05.016 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:43:05.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:05.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:43:05.122 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:43:05.168 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:05.252 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:05 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:43:05.252 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:05 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4205339887' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:43:05.253 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:05 smithi026 ceph-mon[68309]: osdmap e1117: 12 total, 12 up, 11 in 2024-03-20T09:43:05.253 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:05 smithi026 ceph-mon[68309]: 1.e scrub starts 2024-03-20T09:43:05.253 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:05 smithi026 ceph-mon[68309]: pgmap v2711: 47 pgs: 47 active+clean; 5.2 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 975 op/s 2024-03-20T09:43:05.387 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:43:05.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:05.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:43:05.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:05 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:43:05.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:05 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4205339887' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:43:05.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:05 smithi115 ceph-mon[72476]: osdmap e1117: 12 total, 12 up, 11 in 2024-03-20T09:43:05.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:05 smithi115 ceph-mon[72476]: 1.e scrub starts 2024-03-20T09:43:05.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:05 smithi115 ceph-mon[72476]: pgmap v2711: 47 pgs: 47 active+clean; 5.2 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 975 op/s 2024-03-20T09:43:05.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:05 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:43:05.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:05 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4205339887' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:43:05.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:05 smithi122 ceph-mon[73495]: osdmap e1117: 12 total, 12 up, 11 in 2024-03-20T09:43:05.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:05 smithi122 ceph-mon[73495]: 1.e scrub starts 2024-03-20T09:43:05.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:05 smithi122 ceph-mon[73495]: pgmap v2711: 47 pgs: 47 active+clean; 5.2 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 975 op/s 2024-03-20T09:43:05.491 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:43:05.560 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:05.590 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:43:05.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:05.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:43:05.694 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:43:05.742 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:05.802 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:43:05.844 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:43:05.870 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:43:05.910 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:43:05.929 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-20T09:43:05.940 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:43:05.962 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:43:05.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:05.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:43:05.977 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:43:06.012 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:43:06.056 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:43:06.070 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:43:06.096 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:43:06.116 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:06.130 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:43:06.167 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:43:06.198 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:43:06.207 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:43:06.208 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:43:06.208 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:43:06.208 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:43:06.208 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:43:06.208 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:43:06.208 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:43:06.208 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:43:06.208 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:43:06.208 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:43:06.208 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:43:06.208 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:43:06.238 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:06 smithi026 ceph-mon[68309]: 1.e scrub ok 2024-03-20T09:43:06.238 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:06 smithi026 ceph-mon[68309]: 1.a scrub starts 2024-03-20T09:43:06.238 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:06 smithi026 ceph-mon[68309]: 1.a scrub ok 2024-03-20T09:43:06.238 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:06 smithi026 ceph-mon[68309]: 9.c scrub starts 2024-03-20T09:43:06.238 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:06 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3060959740' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:43:06.371 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:43:06.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:06 smithi122 ceph-mon[73495]: 1.e scrub ok 2024-03-20T09:43:06.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:06 smithi122 ceph-mon[73495]: 1.a scrub starts 2024-03-20T09:43:06.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:06 smithi122 ceph-mon[73495]: 1.a scrub ok 2024-03-20T09:43:06.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:06 smithi122 ceph-mon[73495]: 9.c scrub starts 2024-03-20T09:43:06.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:06 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3060959740' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:43:06.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:06 smithi115 ceph-mon[72476]: 1.e scrub ok 2024-03-20T09:43:06.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:06 smithi115 ceph-mon[72476]: 1.a scrub starts 2024-03-20T09:43:06.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:06 smithi115 ceph-mon[72476]: 1.a scrub ok 2024-03-20T09:43:06.646 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:06 smithi115 ceph-mon[72476]: 9.c scrub starts 2024-03-20T09:43:06.646 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:06 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3060959740' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:43:06.800 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:43:06.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:06.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:43:06.904 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:43:06.945 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:06.978 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:43:06.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:06.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:43:06.999 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:43:07.096 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:43:07.151 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:07.379 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:07 smithi026 ceph-mon[68309]: 1.16 scrub starts 2024-03-20T09:43:07.379 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:07 smithi026 ceph-mon[68309]: pgmap v2712: 47 pgs: 47 active+clean; 5.0 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 882 op/s 2024-03-20T09:43:07.379 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:07 smithi026 ceph-mon[68309]: 1.16 scrub ok 2024-03-20T09:43:07.379 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:07 smithi026 ceph-mon[68309]: 9.a scrub starts 2024-03-20T09:43:07.379 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:07 smithi026 ceph-mon[68309]: 9.c scrub ok 2024-03-20T09:43:07.406 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:43:07.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:07.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:43:07.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:07 smithi122 ceph-mon[73495]: 1.16 scrub starts 2024-03-20T09:43:07.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:07 smithi122 ceph-mon[73495]: pgmap v2712: 47 pgs: 47 active+clean; 5.0 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 882 op/s 2024-03-20T09:43:07.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:07 smithi122 ceph-mon[73495]: 1.16 scrub ok 2024-03-20T09:43:07.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:07 smithi122 ceph-mon[73495]: 9.a scrub starts 2024-03-20T09:43:07.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:07 smithi122 ceph-mon[73495]: 9.c scrub ok 2024-03-20T09:43:07.484 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:43:07.512 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:43:07.562 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:07.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:07 smithi115 ceph-mon[72476]: 1.16 scrub starts 2024-03-20T09:43:07.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:07 smithi115 ceph-mon[72476]: pgmap v2712: 47 pgs: 47 active+clean; 5.0 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 882 op/s 2024-03-20T09:43:07.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:07 smithi115 ceph-mon[72476]: 1.16 scrub ok 2024-03-20T09:43:07.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:07 smithi115 ceph-mon[72476]: 9.a scrub starts 2024-03-20T09:43:07.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:07 smithi115 ceph-mon[72476]: 9.c scrub ok 2024-03-20T09:43:07.767 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:43:07.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:07.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:43:07.874 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:43:07.922 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:08.180 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:08 smithi115 ceph-mon[72476]: 1.1 scrub starts 2024-03-20T09:43:08.180 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:08 smithi115 ceph-mon[72476]: 1.1 scrub ok 2024-03-20T09:43:08.180 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:08 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1126870436' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:43:08.180 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:08 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:43:08.180 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:08 smithi115 ceph-mon[72476]: 1.c scrub starts 2024-03-20T09:43:08.180 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:08 smithi115 ceph-mon[72476]: 1.c scrub ok 2024-03-20T09:43:08.181 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.7 to 1 (10000) 2024-03-20T09:43:08.199 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:43:08.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:08.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:43:08.303 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:43:08.347 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:08.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:08 smithi122 ceph-mon[73495]: 1.1 scrub starts 2024-03-20T09:43:08.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:08 smithi122 ceph-mon[73495]: 1.1 scrub ok 2024-03-20T09:43:08.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:08 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1126870436' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:43:08.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:08 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:43:08.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:08 smithi122 ceph-mon[73495]: 1.c scrub starts 2024-03-20T09:43:08.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:08 smithi122 ceph-mon[73495]: 1.c scrub ok 2024-03-20T09:43:08.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:08 smithi026 ceph-mon[68309]: 1.1 scrub starts 2024-03-20T09:43:08.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:08 smithi026 ceph-mon[68309]: 1.1 scrub ok 2024-03-20T09:43:08.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:08 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1126870436' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:43:08.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:08 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:43:08.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:08 smithi026 ceph-mon[68309]: 1.c scrub starts 2024-03-20T09:43:08.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:08 smithi026 ceph-mon[68309]: 1.c scrub ok 2024-03-20T09:43:08.732 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:43:08.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:08.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:43:08.836 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:43:08.898 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:08.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:08.961 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:43:08.961 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:43:09.062 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:43:09.132 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 8 1 2024-03-20T09:43:09.138 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:09.182 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:43:09.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:09.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:43:09.292 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:43:09.341 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:09.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:09 smithi115 ceph-mon[72476]: 9.3 scrub starts 2024-03-20T09:43:09.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:09 smithi115 ceph-mon[72476]: 1.0 scrub starts 2024-03-20T09:43:09.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:09 smithi115 ceph-mon[72476]: 1.0 scrub ok 2024-03-20T09:43:09.399 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:09 smithi115 ceph-mon[72476]: 9.a scrub ok 2024-03-20T09:43:09.400 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:09 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-20T09:43:09.400 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:09 smithi115 ceph-mon[72476]: osdmap e1118: 12 total, 12 up, 11 in 2024-03-20T09:43:09.400 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:09 smithi115 ceph-mon[72476]: 9.4 scrub starts 2024-03-20T09:43:09.400 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:09 smithi115 ceph-mon[72476]: 9.9 scrub starts 2024-03-20T09:43:09.400 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:09 smithi115 ceph-mon[72476]: 9.2 scrub starts 2024-03-20T09:43:09.400 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:09 smithi115 ceph-mon[72476]: pgmap v2714: 47 pgs: 47 active+clean; 5.0 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 676 op/s 2024-03-20T09:43:09.423 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:43:09.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:09 smithi026 ceph-mon[68309]: 9.3 scrub starts 2024-03-20T09:43:09.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:09 smithi026 ceph-mon[68309]: 1.0 scrub starts 2024-03-20T09:43:09.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:09 smithi026 ceph-mon[68309]: 1.0 scrub ok 2024-03-20T09:43:09.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:09 smithi026 ceph-mon[68309]: 9.a scrub ok 2024-03-20T09:43:09.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:09 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-20T09:43:09.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:09 smithi026 ceph-mon[68309]: osdmap e1118: 12 total, 12 up, 11 in 2024-03-20T09:43:09.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:09 smithi026 ceph-mon[68309]: 9.4 scrub starts 2024-03-20T09:43:09.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:09 smithi026 ceph-mon[68309]: 9.9 scrub starts 2024-03-20T09:43:09.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:09 smithi026 ceph-mon[68309]: 9.2 scrub starts 2024-03-20T09:43:09.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:09 smithi026 ceph-mon[68309]: pgmap v2714: 47 pgs: 47 active+clean; 5.0 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 676 op/s 2024-03-20T09:43:09.482 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:43:09.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:09.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:43:09.543 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:43:09.543 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:43:09.587 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:43:09.618 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:09 smithi122 ceph-mon[73495]: 9.3 scrub starts 2024-03-20T09:43:09.618 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:09 smithi122 ceph-mon[73495]: 1.0 scrub starts 2024-03-20T09:43:09.618 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:09 smithi122 ceph-mon[73495]: 1.0 scrub ok 2024-03-20T09:43:09.619 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:09 smithi122 ceph-mon[73495]: 9.a scrub ok 2024-03-20T09:43:09.619 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:09 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-20T09:43:09.619 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:09 smithi122 ceph-mon[73495]: osdmap e1118: 12 total, 12 up, 11 in 2024-03-20T09:43:09.619 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:09 smithi122 ceph-mon[73495]: 9.4 scrub starts 2024-03-20T09:43:09.619 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:09 smithi122 ceph-mon[73495]: 9.9 scrub starts 2024-03-20T09:43:09.619 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:09 smithi122 ceph-mon[73495]: 9.2 scrub starts 2024-03-20T09:43:09.619 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:09 smithi122 ceph-mon[73495]: pgmap v2714: 47 pgs: 47 active+clean; 5.0 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 676 op/s 2024-03-20T09:43:09.621 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:09.645 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:43:09.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:09.648 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:43:09.702 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:43:09.749 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:43:09.781 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:09.827 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:43:09.834 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:09.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:43:09.935 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:43:09.978 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:10.263 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:43:10.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:10.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:43:10.305 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-20T09:43:10.370 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:43:10.426 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:10.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:10 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:43:10.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:10 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:43:10.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:10 smithi026 ceph-mon[68309]: 1.11 scrub starts 2024-03-20T09:43:10.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:10 smithi026 ceph-mon[68309]: 1.1e scrub starts 2024-03-20T09:43:10.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:10 smithi026 ceph-mon[68309]: 1.1e scrub ok 2024-03-20T09:43:10.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:10 smithi026 ceph-mon[68309]: 1.11 scrub ok 2024-03-20T09:43:10.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:10 smithi026 ceph-mon[68309]: osdmap e1119: 12 total, 12 up, 11 in 2024-03-20T09:43:10.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:10 smithi026 ceph-mon[68309]: 9.3 scrub ok 2024-03-20T09:43:10.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:10 smithi026 ceph-mon[68309]: 9.2 scrub ok 2024-03-20T09:43:10.461 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:43:10.471 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:10 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:43:10.471 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:10 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:43:10.471 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:10 smithi115 ceph-mon[72476]: 1.11 scrub starts 2024-03-20T09:43:10.471 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:10 smithi115 ceph-mon[72476]: 1.1e scrub starts 2024-03-20T09:43:10.471 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:10 smithi115 ceph-mon[72476]: 1.1e scrub ok 2024-03-20T09:43:10.471 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:10 smithi115 ceph-mon[72476]: 1.11 scrub ok 2024-03-20T09:43:10.471 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:10 smithi115 ceph-mon[72476]: osdmap e1119: 12 total, 12 up, 11 in 2024-03-20T09:43:10.471 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:10 smithi115 ceph-mon[72476]: 9.3 scrub ok 2024-03-20T09:43:10.471 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:10 smithi115 ceph-mon[72476]: 9.2 scrub ok 2024-03-20T09:43:10.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:10.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:43:10.573 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:43:10.613 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:10.642 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:43:10.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:10.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:43:10.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:10 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:43:10.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:10 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:43:10.684 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:10 smithi122 ceph-mon[73495]: 1.11 scrub starts 2024-03-20T09:43:10.684 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:10 smithi122 ceph-mon[73495]: 1.1e scrub starts 2024-03-20T09:43:10.684 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:10 smithi122 ceph-mon[73495]: 1.1e scrub ok 2024-03-20T09:43:10.684 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:10 smithi122 ceph-mon[73495]: 1.11 scrub ok 2024-03-20T09:43:10.684 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:10 smithi122 ceph-mon[73495]: osdmap e1119: 12 total, 12 up, 11 in 2024-03-20T09:43:10.684 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:10 smithi122 ceph-mon[73495]: 9.3 scrub ok 2024-03-20T09:43:10.684 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:10 smithi122 ceph-mon[73495]: 9.2 scrub ok 2024-03-20T09:43:10.745 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:43:10.764 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:43:10.800 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:11.227 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:43:11.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:11 smithi115 ceph-mon[72476]: 9.4 scrub ok 2024-03-20T09:43:11.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:11 smithi115 ceph-mon[72476]: 9.9 scrub ok 2024-03-20T09:43:11.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:11 smithi115 ceph-mon[72476]: 1.3 scrub starts 2024-03-20T09:43:11.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:11 smithi115 ceph-mon[72476]: 1.3 scrub ok 2024-03-20T09:43:11.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:11 smithi115 ceph-mon[72476]: pgmap v2716: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 4.8 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 925 op/s 2024-03-20T09:43:11.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:11 smithi115 ceph-mon[72476]: 1.1c scrub starts 2024-03-20T09:43:11.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:11 smithi115 ceph-mon[72476]: 1.1c scrub ok 2024-03-20T09:43:11.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:11 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1234048893' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:43:11.688 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:11 smithi122 ceph-mon[73495]: 9.4 scrub ok 2024-03-20T09:43:11.688 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:11 smithi122 ceph-mon[73495]: 9.9 scrub ok 2024-03-20T09:43:11.688 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:11 smithi122 ceph-mon[73495]: 1.3 scrub starts 2024-03-20T09:43:11.688 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:11 smithi122 ceph-mon[73495]: 1.3 scrub ok 2024-03-20T09:43:11.688 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:11 smithi122 ceph-mon[73495]: pgmap v2716: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 4.8 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 925 op/s 2024-03-20T09:43:11.688 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:11 smithi122 ceph-mon[73495]: 1.1c scrub starts 2024-03-20T09:43:11.688 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:11 smithi122 ceph-mon[73495]: 1.1c scrub ok 2024-03-20T09:43:11.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:11 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1234048893' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:43:11.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:11 smithi026 ceph-mon[68309]: 9.4 scrub ok 2024-03-20T09:43:11.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:11 smithi026 ceph-mon[68309]: 9.9 scrub ok 2024-03-20T09:43:11.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:11 smithi026 ceph-mon[68309]: 1.3 scrub starts 2024-03-20T09:43:11.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:11 smithi026 ceph-mon[68309]: 1.3 scrub ok 2024-03-20T09:43:11.710 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:11 smithi026 ceph-mon[68309]: pgmap v2716: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 4.8 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 925 op/s 2024-03-20T09:43:11.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:11 smithi026 ceph-mon[68309]: 1.1c scrub starts 2024-03-20T09:43:11.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:11 smithi026 ceph-mon[68309]: 1.1c scrub ok 2024-03-20T09:43:11.711 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:11 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1234048893' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:43:12.644 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:12 smithi115 ceph-mon[72476]: 1.7 scrub starts 2024-03-20T09:43:12.644 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:12 smithi115 ceph-mon[72476]: 1.7 scrub ok 2024-03-20T09:43:12.644 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:12 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:43:12.644 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:12 smithi115 ceph-mon[72476]: Cluster is now healthy 2024-03-20T09:43:12.644 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:12 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1234048893' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:43:12.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:12 smithi115 ceph-mon[72476]: osdmap e1120: 12 total, 12 up, 11 in 2024-03-20T09:43:12.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:12 smithi115 ceph-mon[72476]: 9.e scrub starts 2024-03-20T09:43:12.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:12 smithi115 ceph-mon[72476]: 1.17 scrub starts 2024-03-20T09:43:12.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:12 smithi115 ceph-mon[72476]: 9.f scrub starts 2024-03-20T09:43:12.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:12 smithi122 ceph-mon[73495]: 1.7 scrub starts 2024-03-20T09:43:12.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:12 smithi122 ceph-mon[73495]: 1.7 scrub ok 2024-03-20T09:43:12.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:12 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:43:12.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:12 smithi122 ceph-mon[73495]: Cluster is now healthy 2024-03-20T09:43:12.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:12 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1234048893' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:43:12.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:12 smithi122 ceph-mon[73495]: osdmap e1120: 12 total, 12 up, 11 in 2024-03-20T09:43:12.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:12 smithi122 ceph-mon[73495]: 9.e scrub starts 2024-03-20T09:43:12.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:12 smithi122 ceph-mon[73495]: 1.17 scrub starts 2024-03-20T09:43:12.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:12 smithi122 ceph-mon[73495]: 9.f scrub starts 2024-03-20T09:43:12.708 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:12 smithi026 ceph-mon[68309]: 1.7 scrub starts 2024-03-20T09:43:12.708 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:12 smithi026 ceph-mon[68309]: 1.7 scrub ok 2024-03-20T09:43:12.708 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:12 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:43:12.708 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:12 smithi026 ceph-mon[68309]: Cluster is now healthy 2024-03-20T09:43:12.708 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:12 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1234048893' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:43:12.708 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:12 smithi026 ceph-mon[68309]: osdmap e1120: 12 total, 12 up, 11 in 2024-03-20T09:43:12.708 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:12 smithi026 ceph-mon[68309]: 9.e scrub starts 2024-03-20T09:43:12.708 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:12 smithi026 ceph-mon[68309]: 1.17 scrub starts 2024-03-20T09:43:12.708 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:12 smithi026 ceph-mon[68309]: 9.f scrub starts 2024-03-20T09:43:13.065 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:43:13.068 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:13.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:43:13.106 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:43:13.169 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:43:13.202 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:13.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:13 smithi026 ceph-mon[68309]: 9.0 scrub starts 2024-03-20T09:43:13.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:13 smithi026 ceph-mon[68309]: 1.17 scrub ok 2024-03-20T09:43:13.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:13 smithi026 ceph-mon[68309]: pgmap v2718: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 4.8 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 701 op/s 2024-03-20T09:43:13.569 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:13 smithi115 ceph-mon[72476]: 9.0 scrub starts 2024-03-20T09:43:13.570 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:13 smithi115 ceph-mon[72476]: 1.17 scrub ok 2024-03-20T09:43:13.570 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:13 smithi115 ceph-mon[72476]: pgmap v2718: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 4.8 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 701 op/s 2024-03-20T09:43:13.670 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:43:13.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:13.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:43:13.684 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:13 smithi122 ceph-mon[73495]: 9.0 scrub starts 2024-03-20T09:43:13.684 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:13 smithi122 ceph-mon[73495]: 1.17 scrub ok 2024-03-20T09:43:13.684 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:13 smithi122 ceph-mon[73495]: pgmap v2718: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 4.8 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 701 op/s 2024-03-20T09:43:13.780 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:43:13.826 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:13.877 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:43:13.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:13.885 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:43:13.986 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:43:14.045 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:43:14.045 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:43:14.066 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:14.109 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:43:14.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:14.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:43:14.220 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:43:14.267 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:14.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:14 smithi115 ceph-mon[72476]: 1.9 scrub starts 2024-03-20T09:43:14.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:14 smithi115 ceph-mon[72476]: 9.e scrub ok 2024-03-20T09:43:14.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:14 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:43:14.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:14 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:43:14.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:14 smithi115 ceph-mon[72476]: 9.0 scrub ok 2024-03-20T09:43:14.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:14 smithi115 ceph-mon[72476]: 1.9 scrub ok 2024-03-20T09:43:14.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:14 smithi115 ceph-mon[72476]: 9.f scrub ok 2024-03-20T09:43:14.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:14 smithi115 ceph-mon[72476]: osdmap e1121: 12 total, 12 up, 11 in 2024-03-20T09:43:14.473 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:43:14.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:14.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:43:14.581 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:43:14.618 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:14 smithi026 ceph-mon[68309]: 1.9 scrub starts 2024-03-20T09:43:14.618 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:14 smithi026 ceph-mon[68309]: 9.e scrub ok 2024-03-20T09:43:14.618 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:14 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:43:14.618 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:14 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:43:14.618 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:14 smithi026 ceph-mon[68309]: 9.0 scrub ok 2024-03-20T09:43:14.618 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:14 smithi026 ceph-mon[68309]: 1.9 scrub ok 2024-03-20T09:43:14.618 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:14 smithi026 ceph-mon[68309]: 9.f scrub ok 2024-03-20T09:43:14.618 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:14 smithi026 ceph-mon[68309]: osdmap e1121: 12 total, 12 up, 11 in 2024-03-20T09:43:14.623 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:14.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:14 smithi122 ceph-mon[73495]: 1.9 scrub starts 2024-03-20T09:43:14.684 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:14 smithi122 ceph-mon[73495]: 9.e scrub ok 2024-03-20T09:43:14.684 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:14 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:43:14.684 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:14 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:43:14.684 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:14 smithi122 ceph-mon[73495]: 9.0 scrub ok 2024-03-20T09:43:14.684 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:14 smithi122 ceph-mon[73495]: 1.9 scrub ok 2024-03-20T09:43:14.684 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:14 smithi122 ceph-mon[73495]: 9.f scrub ok 2024-03-20T09:43:14.684 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:14 smithi122 ceph-mon[73495]: osdmap e1121: 12 total, 12 up, 11 in 2024-03-20T09:43:14.823 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-20T09:43:14.986 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:43:14.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:14.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:43:15.090 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:43:15.125 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:15.247 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:43:15.549 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:15 smithi115 ceph-mon[72476]: 1.6 scrub starts 2024-03-20T09:43:15.549 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:15 smithi115 ceph-mon[72476]: 1.6 scrub ok 2024-03-20T09:43:15.549 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:15 smithi115 ceph-mon[72476]: pgmap v2720: 47 pgs: 47 active+clean; 4.7 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 866 op/s 2024-03-20T09:43:15.549 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:15 smithi115 ceph-mon[72476]: 1.12 scrub starts 2024-03-20T09:43:15.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:15 smithi122 ceph-mon[73495]: 1.6 scrub starts 2024-03-20T09:43:15.684 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:15 smithi122 ceph-mon[73495]: 1.6 scrub ok 2024-03-20T09:43:15.684 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:15 smithi122 ceph-mon[73495]: pgmap v2720: 47 pgs: 47 active+clean; 4.7 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 866 op/s 2024-03-20T09:43:15.684 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:15 smithi122 ceph-mon[73495]: 1.12 scrub starts 2024-03-20T09:43:15.703 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:43:15.707 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:15 smithi026 ceph-mon[68309]: 1.6 scrub starts 2024-03-20T09:43:15.707 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:15 smithi026 ceph-mon[68309]: 1.6 scrub ok 2024-03-20T09:43:15.707 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:15 smithi026 ceph-mon[68309]: pgmap v2720: 47 pgs: 47 active+clean; 4.7 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 866 op/s 2024-03-20T09:43:15.707 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:15 smithi026 ceph-mon[68309]: 1.12 scrub starts 2024-03-20T09:43:15.787 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:43:15.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:15.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:43:15.898 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:43:15.947 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:16.011 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:43:16.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:16.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:43:16.120 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:43:16.154 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:16.245 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:43:16.254 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:16.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:43:16.271 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:16 smithi026 ceph-mon[68309]: 1.13 scrub starts 2024-03-20T09:43:16.271 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:16 smithi026 ceph-mon[68309]: 1.12 scrub ok 2024-03-20T09:43:16.271 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:16 smithi026 ceph-mon[68309]: 1.5 scrub starts 2024-03-20T09:43:16.271 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:16 smithi026 ceph-mon[68309]: 1.5 scrub ok 2024-03-20T09:43:16.356 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:43:16.421 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:16.636 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:16 smithi115 ceph-mon[72476]: 1.13 scrub starts 2024-03-20T09:43:16.637 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:16 smithi115 ceph-mon[72476]: 1.12 scrub ok 2024-03-20T09:43:16.637 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:16 smithi115 ceph-mon[72476]: 1.5 scrub starts 2024-03-20T09:43:16.637 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:16 smithi115 ceph-mon[72476]: 1.5 scrub ok 2024-03-20T09:43:16.688 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:16 smithi122 ceph-mon[73495]: 1.13 scrub starts 2024-03-20T09:43:16.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:16 smithi122 ceph-mon[73495]: 1.12 scrub ok 2024-03-20T09:43:16.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:16 smithi122 ceph-mon[73495]: 1.5 scrub starts 2024-03-20T09:43:16.689 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:16 smithi122 ceph-mon[73495]: 1.5 scrub ok 2024-03-20T09:43:16.756 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:43:16.797 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:43:16.804 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:16.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:43:16.906 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:43:16.923 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:43:16.951 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:43:16.964 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:16.982 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:43:17.005 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:43:17.006 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:43:17.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:17.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:43:17.071 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:43:17.085 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:43:17.111 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:43:17.119 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:43:17.144 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:17.161 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:43:17.183 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:43:17.188 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:43:17.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:17.192 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:43:17.293 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:43:17.296 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.8 to 1 (10000) 2024-03-20T09:43:17.366 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:17.400 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:43:17.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:17.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:43:17.420 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:43:17.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:17 smithi122 ceph-mon[73495]: 1.13 scrub ok 2024-03-20T09:43:17.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:17 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/75759711' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:43:17.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:17 smithi122 ceph-mon[73495]: pgmap v2721: 47 pgs: 47 active+clean; 4.5 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 873 op/s 2024-03-20T09:43:17.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:17 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2513513891' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:43:17.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:17 smithi122 ceph-mon[73495]: 9.d scrub starts 2024-03-20T09:43:17.442 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:43:17.469 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:43:17.471 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:43:17.471 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:43:17.471 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:43:17.471 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:43:17.471 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:43:17.471 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:43:17.471 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:43:17.471 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:43:17.471 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:43:17.471 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:43:17.471 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:43:17.471 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:43:17.491 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:43:17.491 INFO:teuthology.orchestra.run.smithi115.stdout: "size": 20, 2024-03-20T09:43:17.491 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 600, 2024-03-20T09:43:17.491 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [ 2024-03-20T09:43:17.491 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.491 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26605.0:222035 7.15 e943/936 7:ad9477d6:::benchmark_data_smithi122_187077_object95424:head v 943'13498, mlcod=942'13496)", 2024-03-20T09:43:17.491 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:47.897707+0000", 2024-03-20T09:43:17.491 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 509.50082797599998, 2024-03-20T09:43:17.492 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.086363878000000005, 2024-03-20T09:43:17.492 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:17.492 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:43:17.492 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:17.492 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.492 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:17.492 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.897707+0000", 2024-03-20T09:43:17.492 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.492 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.492 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.492 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:17.492 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.897707+0000", 2024-03-20T09:43:17.493 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.493 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.493 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.493 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:17.493 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.897708+0000", 2024-03-20T09:43:17.493 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.85e-07 2024-03-20T09:43:17.493 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.493 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.493 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:17.493 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.897718+0000", 2024-03-20T09:43:17.493 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0064999999999999e-05 2024-03-20T09:43:17.493 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.493 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.493 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:17.493 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.897732+0000", 2024-03-20T09:43:17.494 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3771000000000001e-05 2024-03-20T09:43:17.494 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.494 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.494 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:17.494 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.897740+0000", 2024-03-20T09:43:17.494 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.9680000000000006e-06 2024-03-20T09:43:17.494 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.494 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.494 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:17.494 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.903105+0000", 2024-03-20T09:43:17.494 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0053646609999999997 2024-03-20T09:43:17.494 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.494 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.494 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:43:17.495 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.903126+0000", 2024-03-20T09:43:17.495 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1685e-05 2024-03-20T09:43:17.495 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.495 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.495 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:43:17.495 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.984025+0000", 2024-03-20T09:43:17.495 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.080898484000000007 2024-03-20T09:43:17.495 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.495 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.495 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:17.495 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:47.984071+0000", 2024-03-20T09:43:17.495 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6359000000000001e-05 2024-03-20T09:43:17.495 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.495 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:17.495 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.496 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.496 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.496 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "rep_scrubmap(1.15 e491 from shard 5)", 2024-03-20T09:43:17.496 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:48.016412+0000", 2024-03-20T09:43:17.496 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 509.38212369600001, 2024-03-20T09:43:17.496 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.060988498000000002, 2024-03-20T09:43:17.496 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:17.496 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:43:17.496 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:17.496 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.496 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:17.496 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.016412+0000", 2024-03-20T09:43:17.496 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.497 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.497 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.497 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:17.497 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.016412+0000", 2024-03-20T09:43:17.497 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.497 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.497 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.497 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:17.497 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.016413+0000", 2024-03-20T09:43:17.497 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.4099999999999997e-07 2024-03-20T09:43:17.497 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.497 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.497 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:17.497 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.016418+0000", 2024-03-20T09:43:17.497 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0939999999999998e-06 2024-03-20T09:43:17.498 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.498 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.498 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:17.498 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.016430+0000", 2024-03-20T09:43:17.498 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2795e-05 2024-03-20T09:43:17.498 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.498 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.498 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:17.498 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.016438+0000", 2024-03-20T09:43:17.498 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.1500000000000002e-06 2024-03-20T09:43:17.498 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.498 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.498 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:17.499 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.077379+0000", 2024-03-20T09:43:17.499 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.060941654999999997 2024-03-20T09:43:17.499 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.499 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.499 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:43:17.499 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.077387+0000", 2024-03-20T09:43:17.499 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.1019999999999997e-06 2024-03-20T09:43:17.499 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.499 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.499 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:17.500 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.077400+0000", 2024-03-20T09:43:17.500 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2761e-05 2024-03-20T09:43:17.500 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.500 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:17.500 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.500 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.500 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.500 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "rep_scrubmap(1.15 e491 from shard 8)", 2024-03-20T09:43:17.500 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:34:48.017154+0000", 2024-03-20T09:43:17.500 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 509.38138093800001, 2024-03-20T09:43:17.500 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.060297227000000002, 2024-03-20T09:43:17.500 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:17.501 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:43:17.501 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:17.501 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.501 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:17.501 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.017154+0000", 2024-03-20T09:43:17.501 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.501 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.501 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.501 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:17.501 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.017154+0000", 2024-03-20T09:43:17.501 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.501 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.501 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.501 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:17.501 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.017155+0000", 2024-03-20T09:43:17.502 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4199999999999995e-07 2024-03-20T09:43:17.502 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.502 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.502 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:17.502 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.017159+0000", 2024-03-20T09:43:17.502 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7100000000000001e-06 2024-03-20T09:43:17.502 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.502 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.502 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:17.502 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.017167+0000", 2024-03-20T09:43:17.503 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.4659999999999999e-06 2024-03-20T09:43:17.503 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.503 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.503 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:17.503 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.017172+0000", 2024-03-20T09:43:17.503 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0409999999999997e-06 2024-03-20T09:43:17.503 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.503 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.503 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:17.503 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.077425+0000", 2024-03-20T09:43:17.503 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.06025316 2024-03-20T09:43:17.503 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.503 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.503 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:43:17.504 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.077432+0000", 2024-03-20T09:43:17.504 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.7270000000000001e-06 2024-03-20T09:43:17.504 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.504 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.504 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:17.504 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:34:48.077452+0000", 2024-03-20T09:43:17.504 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9480999999999999e-05 2024-03-20T09:43:17.504 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.504 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:17.504 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.504 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.504 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.504 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:14523 8.d e980/974 8:b1c3d5d8:::benchmark_data_smithi122_197775_object14522:head v 980'870, mlcod=980'868)", 2024-03-20T09:43:17.504 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:05.332829+0000", 2024-03-20T09:43:17.505 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 432.06570685999998, 2024-03-20T09:43:17.505 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.080553347999999997, 2024-03-20T09:43:17.505 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:17.505 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:43:17.505 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:17.505 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.505 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:17.505 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.332829+0000", 2024-03-20T09:43:17.505 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.505 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.505 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.505 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:17.505 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.332829+0000", 2024-03-20T09:43:17.505 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.506 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.506 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.506 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:17.506 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.332829+0000", 2024-03-20T09:43:17.506 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5000000000000001e-07 2024-03-20T09:43:17.506 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.506 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.506 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:17.506 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.332921+0000", 2024-03-20T09:43:17.506 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.1848999999999994e-05 2024-03-20T09:43:17.506 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.506 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.507 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:17.507 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.332933+0000", 2024-03-20T09:43:17.507 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2071e-05 2024-03-20T09:43:17.507 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.507 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.507 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:17.507 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.332941+0000", 2024-03-20T09:43:17.507 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.8520000000000004e-06 2024-03-20T09:43:17.507 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.507 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.507 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:17.507 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.332994+0000", 2024-03-20T09:43:17.507 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2398000000000002e-05 2024-03-20T09:43:17.507 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.507 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.508 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:43:17.508 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.333015+0000", 2024-03-20T09:43:17.508 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1807999999999999e-05 2024-03-20T09:43:17.508 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.508 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.508 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:43:17.508 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.413352+0000", 2024-03-20T09:43:17.508 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.080337069999999997 2024-03-20T09:43:17.508 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.508 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.508 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:17.508 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.413382+0000", 2024-03-20T09:43:17.508 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9450000000000001e-05 2024-03-20T09:43:17.508 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.508 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:17.509 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.509 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.509 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.509 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:14525 8.d e980/974 8:bd7f84eb:::benchmark_data_smithi122_197775_object14524:head v 980'871, mlcod=980'868)", 2024-03-20T09:43:17.509 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:05.335283+0000", 2024-03-20T09:43:17.509 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 432.06325249399998, 2024-03-20T09:43:17.509 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.078128146999999995, 2024-03-20T09:43:17.509 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:17.509 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:43:17.509 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:17.509 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.509 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:17.509 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.335283+0000", 2024-03-20T09:43:17.509 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.509 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.509 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.510 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:17.510 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.335283+0000", 2024-03-20T09:43:17.510 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.510 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.510 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.510 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:17.510 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.335283+0000", 2024-03-20T09:43:17.510 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4700000000000001e-07 2024-03-20T09:43:17.510 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.510 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.510 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:17.510 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.335453+0000", 2024-03-20T09:43:17.511 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00016961600000000001 2024-03-20T09:43:17.511 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.511 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.511 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:17.511 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.335462+0000", 2024-03-20T09:43:17.511 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.2499999999999995e-06 2024-03-20T09:43:17.511 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.511 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.511 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:17.511 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.335468+0000", 2024-03-20T09:43:17.511 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8189999999999997e-06 2024-03-20T09:43:17.511 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.511 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.511 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:17.511 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.335511+0000", 2024-03-20T09:43:17.511 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.2478999999999998e-05 2024-03-20T09:43:17.512 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.512 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.512 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:43:17.512 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.335531+0000", 2024-03-20T09:43:17.512 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0040000000000001e-05 2024-03-20T09:43:17.512 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.512 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.512 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:43:17.512 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.413385+0000", 2024-03-20T09:43:17.512 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.077854520999999996 2024-03-20T09:43:17.512 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.512 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.512 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:17.512 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.413411+0000", 2024-03-20T09:43:17.513 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5874999999999999e-05 2024-03-20T09:43:17.513 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.513 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:17.513 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.513 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.513 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.513 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:14534 8.d e980/974 8:bd505d36:::benchmark_data_smithi122_197775_object14533:head v 980'872, mlcod=980'868)", 2024-03-20T09:43:17.513 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:05.337841+0000", 2024-03-20T09:43:17.513 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 432.06069461800001, 2024-03-20T09:43:17.513 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.075595009000000005, 2024-03-20T09:43:17.513 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:17.513 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:43:17.513 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:17.513 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.513 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:17.513 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.337841+0000", 2024-03-20T09:43:17.514 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.514 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.514 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.514 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:17.514 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.337841+0000", 2024-03-20T09:43:17.514 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.514 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.514 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.514 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:17.514 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.337841+0000", 2024-03-20T09:43:17.514 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.5199999999999998e-07 2024-03-20T09:43:17.514 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.514 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.514 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:17.514 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.337928+0000", 2024-03-20T09:43:17.515 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.6570000000000006e-05 2024-03-20T09:43:17.515 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.515 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.515 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:17.515 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.337935+0000", 2024-03-20T09:43:17.515 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.4059999999999999e-06 2024-03-20T09:43:17.515 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.515 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.515 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:17.515 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.337941+0000", 2024-03-20T09:43:17.515 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8699999999999997e-06 2024-03-20T09:43:17.515 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.515 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.515 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:17.515 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.337985+0000", 2024-03-20T09:43:17.515 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4030999999999999e-05 2024-03-20T09:43:17.516 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.516 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.516 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:43:17.516 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.338003+0000", 2024-03-20T09:43:17.516 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7759000000000001e-05 2024-03-20T09:43:17.516 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.516 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.516 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:43:17.516 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.413414+0000", 2024-03-20T09:43:17.516 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.075411117 2024-03-20T09:43:17.516 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.516 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.516 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:17.516 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.413436+0000", 2024-03-20T09:43:17.516 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1903999999999999e-05 2024-03-20T09:43:17.516 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.517 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:17.517 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.517 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.517 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.517 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:14545 8.d e980/974 8:b7f9247f:::benchmark_data_smithi122_197775_object14544:head v 980'873, mlcod=980'868)", 2024-03-20T09:43:17.517 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:05.340825+0000", 2024-03-20T09:43:17.517 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 432.05771057200002, 2024-03-20T09:43:17.517 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.072634562999999999, 2024-03-20T09:43:17.517 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:17.517 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:43:17.517 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:17.517 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.517 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:17.517 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.340825+0000", 2024-03-20T09:43:17.518 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.518 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.518 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.518 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:17.518 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.340825+0000", 2024-03-20T09:43:17.518 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.518 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.518 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.518 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:17.518 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.340825+0000", 2024-03-20T09:43:17.518 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.75e-07 2024-03-20T09:43:17.518 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.518 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.518 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:17.518 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.340902+0000", 2024-03-20T09:43:17.519 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7058999999999995e-05 2024-03-20T09:43:17.519 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.519 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.519 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:17.519 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.340911+0000", 2024-03-20T09:43:17.519 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.7399999999999993e-06 2024-03-20T09:43:17.519 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.519 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.519 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:17.519 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.340918+0000", 2024-03-20T09:43:17.519 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.5910000000000001e-06 2024-03-20T09:43:17.519 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.519 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.519 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:17.519 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.340957+0000", 2024-03-20T09:43:17.519 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.9146000000000001e-05 2024-03-20T09:43:17.520 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.520 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.520 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:43:17.520 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.340972+0000", 2024-03-20T09:43:17.520 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4972e-05 2024-03-20T09:43:17.520 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.520 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.520 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:43:17.520 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.413438+0000", 2024-03-20T09:43:17.520 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.072465871000000001 2024-03-20T09:43:17.520 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.520 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.520 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:17.520 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.413459+0000", 2024-03-20T09:43:17.520 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1608999999999999e-05 2024-03-20T09:43:17.521 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.521 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:17.521 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.521 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.521 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.521 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:14559 8.d e980/974 8:bfa51422:::benchmark_data_smithi122_197775_object14558:head v 980'874, mlcod=980'868)", 2024-03-20T09:43:17.521 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:05.348897+0000", 2024-03-20T09:43:17.521 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 432.04963833400001, 2024-03-20T09:43:17.521 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.064589677999999998, 2024-03-20T09:43:17.521 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:17.521 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:43:17.521 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:17.521 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.521 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:17.521 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.348897+0000", 2024-03-20T09:43:17.521 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.521 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.522 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.522 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:17.522 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.348897+0000", 2024-03-20T09:43:17.522 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.522 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.522 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.522 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:17.522 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.348898+0000", 2024-03-20T09:43:17.522 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.023e-06 2024-03-20T09:43:17.522 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.522 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.522 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:17.522 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.349026+0000", 2024-03-20T09:43:17.522 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012763799999999999 2024-03-20T09:43:17.523 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.523 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.523 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:17.523 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.349042+0000", 2024-03-20T09:43:17.523 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6256000000000002e-05 2024-03-20T09:43:17.523 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.523 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.523 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:17.523 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.349048+0000", 2024-03-20T09:43:17.523 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5709999999999997e-06 2024-03-20T09:43:17.523 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.523 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.523 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:17.523 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.349584+0000", 2024-03-20T09:43:17.523 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000536288 2024-03-20T09:43:17.524 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.524 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.524 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:43:17.524 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.349604+0000", 2024-03-20T09:43:17.524 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9874e-05 2024-03-20T09:43:17.524 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.524 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.524 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:43:17.524 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.413462+0000", 2024-03-20T09:43:17.524 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.063857842999999997 2024-03-20T09:43:17.524 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.524 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.524 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:17.524 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.413487+0000", 2024-03-20T09:43:17.524 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5185e-05 2024-03-20T09:43:17.525 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.525 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:17.525 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.525 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.525 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.525 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:14601 8.d e980/974 8:be1d5a33:::benchmark_data_smithi122_197775_object14600:head v 980'875, mlcod=980'868)", 2024-03-20T09:43:17.525 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:05.361367+0000", 2024-03-20T09:43:17.525 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 432.03716826300001, 2024-03-20T09:43:17.525 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.052143722000000003, 2024-03-20T09:43:17.525 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:17.525 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:43:17.525 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:17.525 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.525 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:17.525 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.361367+0000", 2024-03-20T09:43:17.526 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.526 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.526 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.526 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:17.526 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.361367+0000", 2024-03-20T09:43:17.526 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.526 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.526 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.526 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:17.526 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.361368+0000", 2024-03-20T09:43:17.526 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.048e-06 2024-03-20T09:43:17.526 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.526 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.526 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:17.527 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.361486+0000", 2024-03-20T09:43:17.527 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00011807399999999999 2024-03-20T09:43:17.527 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.527 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.527 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:17.527 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.361499+0000", 2024-03-20T09:43:17.527 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2991e-05 2024-03-20T09:43:17.527 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.527 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.527 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:17.527 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.361508+0000", 2024-03-20T09:43:17.527 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.6230000000000003e-06 2024-03-20T09:43:17.527 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.527 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.528 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:17.528 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.361565+0000", 2024-03-20T09:43:17.528 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7589e-05 2024-03-20T09:43:17.528 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.528 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.528 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:43:17.528 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.361590+0000", 2024-03-20T09:43:17.528 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.4677000000000001e-05 2024-03-20T09:43:17.528 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.528 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.528 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:43:17.528 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.413489+0000", 2024-03-20T09:43:17.528 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.051898442000000003 2024-03-20T09:43:17.529 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.529 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.529 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:17.529 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.413511+0000", 2024-03-20T09:43:17.529 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2277999999999999e-05 2024-03-20T09:43:17.529 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.529 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:17.529 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.529 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.529 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.529 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "rep_scrubmap(1.15 e491 from shard 5)", 2024-03-20T09:43:17.529 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:05.445441+0000", 2024-03-20T09:43:17.529 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 431.95309435000001, 2024-03-20T09:43:17.530 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.056021108, 2024-03-20T09:43:17.530 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:17.530 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:43:17.530 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:17.530 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.530 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:17.530 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.445441+0000", 2024-03-20T09:43:17.530 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.530 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.530 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.530 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:17.530 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.445441+0000", 2024-03-20T09:43:17.530 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.530 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.530 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.531 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:17.531 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.445442+0000", 2024-03-20T09:43:17.531 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0890000000000001e-06 2024-03-20T09:43:17.531 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.531 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.531 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:17.531 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.445449+0000", 2024-03-20T09:43:17.531 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.6839999999999999e-06 2024-03-20T09:43:17.531 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.531 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.531 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:17.531 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.445461+0000", 2024-03-20T09:43:17.531 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1946999999999999e-05 2024-03-20T09:43:17.531 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.531 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.531 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:17.532 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.445467+0000", 2024-03-20T09:43:17.532 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1539999999999999e-06 2024-03-20T09:43:17.532 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.532 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.532 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:17.532 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.501431+0000", 2024-03-20T09:43:17.532 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.055964106999999999 2024-03-20T09:43:17.532 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.532 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.532 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:43:17.532 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.501445+0000", 2024-03-20T09:43:17.532 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3645000000000001e-05 2024-03-20T09:43:17.532 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.532 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.532 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:17.533 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.501462+0000", 2024-03-20T09:43:17.533 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7482e-05 2024-03-20T09:43:17.533 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.533 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:17.533 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.533 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.533 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.533 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:85294 8.d e998/974 8:bf9f6105:::benchmark_data_smithi122_197775_object85293:head v 998'5362, mlcod=998'5360)", 2024-03-20T09:43:17.533 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:50.030225+0000", 2024-03-20T09:43:17.533 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 387.36831007299998, 2024-03-20T09:43:17.533 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.094034662000000005, 2024-03-20T09:43:17.533 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:17.533 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:43:17.533 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:17.533 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.533 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:17.534 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.030225+0000", 2024-03-20T09:43:17.534 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.534 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.534 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.534 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:17.534 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.030225+0000", 2024-03-20T09:43:17.534 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.534 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.534 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.534 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:17.534 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.030227+0000", 2024-03-20T09:43:17.534 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.33e-06 2024-03-20T09:43:17.534 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.534 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.534 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:17.535 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.030284+0000", 2024-03-20T09:43:17.535 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6991999999999999e-05 2024-03-20T09:43:17.535 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.535 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.535 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:17.535 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.030305+0000", 2024-03-20T09:43:17.535 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1276000000000001e-05 2024-03-20T09:43:17.535 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.535 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.535 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:17.535 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.030322+0000", 2024-03-20T09:43:17.535 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6739e-05 2024-03-20T09:43:17.535 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.535 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.536 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:17.536 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.030394+0000", 2024-03-20T09:43:17.536 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.2361999999999996e-05 2024-03-20T09:43:17.536 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.536 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.536 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:43:17.536 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.030416+0000", 2024-03-20T09:43:17.536 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1920999999999999e-05 2024-03-20T09:43:17.536 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.536 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.536 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:43:17.536 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124230+0000", 2024-03-20T09:43:17.536 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.093813721000000003 2024-03-20T09:43:17.536 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.536 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.536 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:17.537 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124260+0000", 2024-03-20T09:43:17.537 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.0321e-05 2024-03-20T09:43:17.537 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.537 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:17.537 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.537 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.537 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.537 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:85304 8.d e998/974 8:b9b9ee18:::benchmark_data_smithi122_197775_object85303:head v 998'5363, mlcod=998'5360)", 2024-03-20T09:43:17.537 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:50.035062+0000", 2024-03-20T09:43:17.537 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 387.363473296, 2024-03-20T09:43:17.537 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.089231913999999996, 2024-03-20T09:43:17.537 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:17.537 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:43:17.537 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:17.538 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.538 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:17.538 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.035062+0000", 2024-03-20T09:43:17.538 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.538 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.538 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.538 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:17.538 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.035062+0000", 2024-03-20T09:43:17.538 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.538 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.538 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.538 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:17.538 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.035063+0000", 2024-03-20T09:43:17.538 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.9999999999999996e-07 2024-03-20T09:43:17.538 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.539 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.539 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:17.539 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.035117+0000", 2024-03-20T09:43:17.539 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.363e-05 2024-03-20T09:43:17.539 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.539 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.539 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:17.539 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.035128+0000", 2024-03-20T09:43:17.539 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1561e-05 2024-03-20T09:43:17.539 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.539 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.539 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:17.539 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.035136+0000", 2024-03-20T09:43:17.539 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.4749999999999996e-06 2024-03-20T09:43:17.539 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.540 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.540 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:17.540 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.035232+0000", 2024-03-20T09:43:17.540 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.6311000000000007e-05 2024-03-20T09:43:17.540 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.540 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.540 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:43:17.540 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.035267+0000", 2024-03-20T09:43:17.540 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.4971000000000001e-05 2024-03-20T09:43:17.540 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.540 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.540 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:43:17.540 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124264+0000", 2024-03-20T09:43:17.540 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.088997007000000003 2024-03-20T09:43:17.540 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.541 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.541 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:17.541 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124294+0000", 2024-03-20T09:43:17.541 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.0059000000000001e-05 2024-03-20T09:43:17.541 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.541 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:17.541 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.541 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.541 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.541 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:85309 8.d e998/974 8:b696b706:::benchmark_data_smithi122_197775_object85308:head v 998'5364, mlcod=998'5360)", 2024-03-20T09:43:17.541 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:50.039124+0000", 2024-03-20T09:43:17.541 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 387.35941179700001, 2024-03-20T09:43:17.541 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.085202424999999998, 2024-03-20T09:43:17.541 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:17.541 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:43:17.542 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:17.542 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.542 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:17.542 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.039124+0000", 2024-03-20T09:43:17.542 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.542 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.542 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.542 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:17.542 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.039124+0000", 2024-03-20T09:43:17.542 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.542 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.542 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.543 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:17.543 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.039124+0000", 2024-03-20T09:43:17.543 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.6400000000000001e-07 2024-03-20T09:43:17.543 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.543 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.543 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:17.543 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.039228+0000", 2024-03-20T09:43:17.543 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000103757 2024-03-20T09:43:17.543 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.543 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.543 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:17.543 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.039240+0000", 2024-03-20T09:43:17.543 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1652e-05 2024-03-20T09:43:17.543 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.543 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.544 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:17.544 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.039247+0000", 2024-03-20T09:43:17.544 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3359999999999997e-06 2024-03-20T09:43:17.544 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.544 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.544 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:17.544 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.039311+0000", 2024-03-20T09:43:17.544 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.3662000000000001e-05 2024-03-20T09:43:17.544 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.544 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.544 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:43:17.544 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.039335+0000", 2024-03-20T09:43:17.544 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3725000000000001e-05 2024-03-20T09:43:17.544 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.544 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.544 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:43:17.545 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124296+0000", 2024-03-20T09:43:17.545 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.084961693000000005 2024-03-20T09:43:17.545 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.545 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.545 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:17.545 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124326+0000", 2024-03-20T09:43:17.545 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9736000000000001e-05 2024-03-20T09:43:17.545 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.545 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:17.545 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.545 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.545 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.545 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:85325 8.d e998/974 8:bc21c74a:::benchmark_data_smithi122_197775_object85324:head v 998'5365, mlcod=998'5360)", 2024-03-20T09:43:17.545 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:50.052544+0000", 2024-03-20T09:43:17.546 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 387.345991443, 2024-03-20T09:43:17.546 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.071814343000000003, 2024-03-20T09:43:17.546 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:17.546 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:43:17.546 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:17.546 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.546 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:17.546 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.052544+0000", 2024-03-20T09:43:17.546 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.546 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.546 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.546 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:17.546 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.052544+0000", 2024-03-20T09:43:17.546 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.547 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.547 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.547 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:17.547 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.052545+0000", 2024-03-20T09:43:17.547 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.415e-06 2024-03-20T09:43:17.547 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.547 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.547 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:17.547 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.052580+0000", 2024-03-20T09:43:17.547 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.4459999999999999e-05 2024-03-20T09:43:17.547 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.547 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.547 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:17.547 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.052600+0000", 2024-03-20T09:43:17.547 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9853000000000001e-05 2024-03-20T09:43:17.548 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.548 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.548 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:17.548 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.052610+0000", 2024-03-20T09:43:17.548 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0655999999999999e-05 2024-03-20T09:43:17.548 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.548 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.548 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:17.548 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.052671+0000", 2024-03-20T09:43:17.548 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0565e-05 2024-03-20T09:43:17.548 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.548 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.548 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:43:17.548 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.052693+0000", 2024-03-20T09:43:17.549 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2308000000000001e-05 2024-03-20T09:43:17.549 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.549 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.549 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:43:17.549 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124329+0000", 2024-03-20T09:43:17.549 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.071635628000000007 2024-03-20T09:43:17.549 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.549 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.549 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:17.549 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124358+0000", 2024-03-20T09:43:17.549 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9458000000000002e-05 2024-03-20T09:43:17.549 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.549 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:17.549 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.550 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.550 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.550 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:85327 8.d e998/974 8:b1150ede:::benchmark_data_smithi122_197775_object85326:head v 998'5366, mlcod=998'5360)", 2024-03-20T09:43:17.550 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:50.055053+0000", 2024-03-20T09:43:17.550 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 387.34348236099999, 2024-03-20T09:43:17.550 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.069332682000000007, 2024-03-20T09:43:17.550 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:17.550 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:43:17.550 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:17.550 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.550 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:17.550 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.055053+0000", 2024-03-20T09:43:17.550 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.550 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.551 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.551 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:17.551 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.055053+0000", 2024-03-20T09:43:17.551 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.551 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.551 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.551 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:17.551 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.055054+0000", 2024-03-20T09:43:17.551 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2470000000000001e-06 2024-03-20T09:43:17.551 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.551 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.551 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:17.551 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.055097+0000", 2024-03-20T09:43:17.551 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.2617999999999998e-05 2024-03-20T09:43:17.551 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.552 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.552 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:17.552 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.055110+0000", 2024-03-20T09:43:17.552 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3117e-05 2024-03-20T09:43:17.552 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.552 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.552 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:17.552 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.055116+0000", 2024-03-20T09:43:17.552 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4570000000000002e-06 2024-03-20T09:43:17.552 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.552 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.552 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:17.552 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.055321+0000", 2024-03-20T09:43:17.552 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000204231 2024-03-20T09:43:17.552 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.553 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.553 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:43:17.553 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.055340+0000", 2024-03-20T09:43:17.553 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8856999999999999e-05 2024-03-20T09:43:17.553 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.553 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.553 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:43:17.553 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124360+0000", 2024-03-20T09:43:17.553 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.069020916000000002 2024-03-20T09:43:17.553 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.553 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.553 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:17.553 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124386+0000", 2024-03-20T09:43:17.553 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5239e-05 2024-03-20T09:43:17.553 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.553 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:17.554 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.554 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.554 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.554 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:85328 8.d e998/974 8:b7d49900:::benchmark_data_smithi122_197775_object85327:head v 998'5367, mlcod=998'5360)", 2024-03-20T09:43:17.554 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:50.058952+0000", 2024-03-20T09:43:17.554 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 387.33958386199998, 2024-03-20T09:43:17.554 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.065465018, 2024-03-20T09:43:17.554 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:17.554 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:43:17.554 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:17.554 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.554 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:17.554 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.058952+0000", 2024-03-20T09:43:17.554 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.554 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.555 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.555 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:17.555 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.058952+0000", 2024-03-20T09:43:17.555 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.555 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.555 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.555 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:17.555 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.058953+0000", 2024-03-20T09:43:17.555 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0690000000000001e-06 2024-03-20T09:43:17.555 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.555 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.555 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:17.555 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.058989+0000", 2024-03-20T09:43:17.555 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6922000000000003e-05 2024-03-20T09:43:17.555 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.555 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.556 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:17.556 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.059003+0000", 2024-03-20T09:43:17.556 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3787e-05 2024-03-20T09:43:17.556 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.556 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.556 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:17.556 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.059011+0000", 2024-03-20T09:43:17.556 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.9330000000000001e-06 2024-03-20T09:43:17.556 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.556 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.556 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:17.556 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.059083+0000", 2024-03-20T09:43:17.556 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.2071999999999997e-05 2024-03-20T09:43:17.556 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.556 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.557 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:43:17.557 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.059105+0000", 2024-03-20T09:43:17.557 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1699999999999999e-05 2024-03-20T09:43:17.557 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.557 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.557 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:43:17.557 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124388+0000", 2024-03-20T09:43:17.557 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.065282824000000003 2024-03-20T09:43:17.557 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.557 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.557 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:17.557 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124417+0000", 2024-03-20T09:43:17.557 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8711e-05 2024-03-20T09:43:17.557 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.558 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:17.558 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.558 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.558 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.558 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:85334 8.d e998/974 8:b8549932:::benchmark_data_smithi122_197775_object85333:head v 998'5368, mlcod=998'5360)", 2024-03-20T09:43:17.558 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:50.062866+0000", 2024-03-20T09:43:17.558 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 387.33566911700001, 2024-03-20T09:43:17.558 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.061575585000000002, 2024-03-20T09:43:17.558 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:17.558 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:43:17.558 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:17.558 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.558 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:17.558 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.062866+0000", 2024-03-20T09:43:17.558 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.559 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.559 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.559 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:17.559 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.062866+0000", 2024-03-20T09:43:17.559 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.559 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.559 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.559 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:17.559 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.062867+0000", 2024-03-20T09:43:17.559 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0890000000000001e-06 2024-03-20T09:43:17.559 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.559 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.559 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:17.559 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.062972+0000", 2024-03-20T09:43:17.560 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000104344 2024-03-20T09:43:17.560 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.560 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.560 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:17.560 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.062987+0000", 2024-03-20T09:43:17.560 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5738000000000001e-05 2024-03-20T09:43:17.560 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.560 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.560 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:17.560 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.062997+0000", 2024-03-20T09:43:17.560 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.6129999999999993e-06 2024-03-20T09:43:17.560 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.560 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.560 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:17.561 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.063058+0000", 2024-03-20T09:43:17.561 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1370000000000004e-05 2024-03-20T09:43:17.561 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.561 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.561 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:43:17.561 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.063085+0000", 2024-03-20T09:43:17.561 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6676999999999999e-05 2024-03-20T09:43:17.561 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.561 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.561 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:43:17.561 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124419+0000", 2024-03-20T09:43:17.561 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.061333490999999997 2024-03-20T09:43:17.561 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.561 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.561 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:17.561 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124442+0000", 2024-03-20T09:43:17.562 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3263000000000001e-05 2024-03-20T09:43:17.562 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.562 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:17.562 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.562 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.562 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.562 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.27190.0:99030 9.e e1086/1066 9:74faa8ab:::benchmark_data_smithi122_210164_object99029:head v 1086'6211, mlcod=1086'6209)", 2024-03-20T09:43:17.562 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:41:31.147671+0000", 2024-03-20T09:43:17.562 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 106.25086402700001, 2024-03-20T09:43:17.562 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.055064747999999997, 2024-03-20T09:43:17.562 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:17.562 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:43:17.562 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:17.562 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.563 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:17.563 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:31.147671+0000", 2024-03-20T09:43:17.563 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.563 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.563 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.563 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:17.563 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:31.147671+0000", 2024-03-20T09:43:17.563 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.563 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.563 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.563 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:17.563 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:31.147672+0000", 2024-03-20T09:43:17.563 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.6599999999999994e-07 2024-03-20T09:43:17.563 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.564 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.564 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:17.564 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:31.147722+0000", 2024-03-20T09:43:17.564 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9852999999999999e-05 2024-03-20T09:43:17.564 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.564 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.564 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:17.564 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:31.147737+0000", 2024-03-20T09:43:17.564 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4606e-05 2024-03-20T09:43:17.564 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.564 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.564 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:17.564 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:31.147746+0000", 2024-03-20T09:43:17.564 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.2420000000000006e-06 2024-03-20T09:43:17.564 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.564 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.565 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:17.565 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:31.147829+0000", 2024-03-20T09:43:17.565 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.3426999999999999e-05 2024-03-20T09:43:17.565 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.565 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.565 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:43:17.565 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:31.147855+0000", 2024-03-20T09:43:17.565 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5851999999999999e-05 2024-03-20T09:43:17.565 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.565 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.565 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:43:17.565 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:31.202715+0000", 2024-03-20T09:43:17.565 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.054859992000000003 2024-03-20T09:43:17.565 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.565 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.565 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:17.566 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:31.202736+0000", 2024-03-20T09:43:17.566 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0809999999999999e-05 2024-03-20T09:43:17.566 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.566 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:17.566 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.566 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.566 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.566 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.27190.0:159485 9.d e1105/1066 9:bdbae123:::benchmark_data_smithi122_210164_object27255:head v 1105'9968, mlcod=1105'9966)", 2024-03-20T09:43:17.566 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:42:39.112796+0000", 2024-03-20T09:43:17.566 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 38.285739174, 2024-03-20T09:43:17.566 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.047367102000000001, 2024-03-20T09:43:17.566 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:17.566 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:43:17.567 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:17.567 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.567 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:17.567 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:42:39.112796+0000", 2024-03-20T09:43:17.567 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.567 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.567 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.567 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:17.567 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:42:39.112796+0000", 2024-03-20T09:43:17.567 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.567 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.567 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.567 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:17.567 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:42:39.112797+0000", 2024-03-20T09:43:17.568 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0640000000000001e-06 2024-03-20T09:43:17.568 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.568 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.568 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:17.568 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:42:39.112806+0000", 2024-03-20T09:43:17.568 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.4710000000000007e-06 2024-03-20T09:43:17.568 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.568 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.568 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:17.568 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:42:39.112819+0000", 2024-03-20T09:43:17.568 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2884000000000001e-05 2024-03-20T09:43:17.568 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.568 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.569 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:17.569 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:42:39.112827+0000", 2024-03-20T09:43:17.569 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5979999999999997e-06 2024-03-20T09:43:17.569 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.569 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.569 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:17.569 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:42:39.113138+0000", 2024-03-20T09:43:17.569 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00031114499999999998 2024-03-20T09:43:17.569 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.569 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.569 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:43:17.569 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:42:39.113158+0000", 2024-03-20T09:43:17.569 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9905999999999999e-05 2024-03-20T09:43:17.569 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.569 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.570 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:43:17.570 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:42:39.160110+0000", 2024-03-20T09:43:17.570 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.046951596999999998 2024-03-20T09:43:17.570 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.570 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.570 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:17.570 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:42:39.160163+0000", 2024-03-20T09:43:17.570 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3436999999999997e-05 2024-03-20T09:43:17.570 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.570 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:17.570 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.570 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.570 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.570 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.27190.0:159491 9.5 e1105/1066 9:a83affeb:::benchmark_data_smithi122_210164_object27261:head v 1105'10158, mlcod=1105'10154)", 2024-03-20T09:43:17.571 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:42:39.117201+0000", 2024-03-20T09:43:17.571 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 38.281334213000001, 2024-03-20T09:43:17.571 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.043007995, 2024-03-20T09:43:17.571 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:43:17.571 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:43:17.571 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:43:17.571 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.571 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:43:17.571 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:42:39.117201+0000", 2024-03-20T09:43:17.571 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.571 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.571 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.571 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:43:17.571 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:42:39.117201+0000", 2024-03-20T09:43:17.571 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:43:17.571 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.572 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.572 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:43:17.572 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:42:39.117202+0000", 2024-03-20T09:43:17.572 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.3399999999999998e-07 2024-03-20T09:43:17.572 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.572 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.572 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:43:17.572 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:42:39.117209+0000", 2024-03-20T09:43:17.572 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.9469999999999997e-06 2024-03-20T09:43:17.572 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.572 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.572 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:43:17.572 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:42:39.117219+0000", 2024-03-20T09:43:17.573 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.7890000000000006e-06 2024-03-20T09:43:17.573 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.573 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.573 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:43:17.573 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:42:39.117227+0000", 2024-03-20T09:43:17.573 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.4200000000000007e-06 2024-03-20T09:43:17.573 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.573 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.573 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:43:17.573 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:42:39.117970+0000", 2024-03-20T09:43:17.573 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00074273899999999996 2024-03-20T09:43:17.573 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.573 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.573 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:43:17.573 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:42:39.117994+0000", 2024-03-20T09:43:17.574 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.4372e-05 2024-03-20T09:43:17.574 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.574 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.574 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:43:17.574 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:42:39.160168+0000", 2024-03-20T09:43:17.574 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.042174121000000002 2024-03-20T09:43:17.574 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:43:17.574 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:43:17.574 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:43:17.574 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:42:39.160209+0000", 2024-03-20T09:43:17.574 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.0772999999999997e-05 2024-03-20T09:43:17.574 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.574 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:17.574 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.574 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:43:17.575 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:43:17.575 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:43:17.575 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:43:17.629 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:17 smithi115 ceph-mon[72476]: 1.13 scrub ok 2024-03-20T09:43:17.629 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:17 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/75759711' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:43:17.629 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:17 smithi115 ceph-mon[72476]: pgmap v2721: 47 pgs: 47 active+clean; 4.5 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 873 op/s 2024-03-20T09:43:17.630 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:17 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2513513891' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:43:17.630 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:17 smithi115 ceph-mon[72476]: 9.d scrub starts 2024-03-20T09:43:17.630 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:17.656 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:43:17.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:17.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:43:17.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:17 smithi026 ceph-mon[68309]: 1.13 scrub ok 2024-03-20T09:43:17.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:17 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/75759711' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:43:17.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:17 smithi026 ceph-mon[68309]: pgmap v2721: 47 pgs: 47 active+clean; 4.5 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 873 op/s 2024-03-20T09:43:17.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:17 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2513513891' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:43:17.712 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:17 smithi026 ceph-mon[68309]: 9.d scrub starts 2024-03-20T09:43:17.762 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:43:17.838 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:17.865 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:43:17.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:17.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:43:17.979 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:43:18.032 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:18.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:18 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/75759711' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-03-20T09:43:18.645 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:18 smithi115 ceph-mon[72476]: osdmap e1122: 12 total, 12 up, 11 in 2024-03-20T09:43:18.646 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:18 smithi115 ceph-mon[72476]: 1.8 scrub starts 2024-03-20T09:43:18.646 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:18 smithi115 ceph-mon[72476]: 1.8 scrub ok 2024-03-20T09:43:18.646 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:18 smithi115 ceph-mon[72476]: 1.4 scrub starts 2024-03-20T09:43:18.646 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:18 smithi115 ceph-mon[72476]: 1.4 scrub ok 2024-03-20T09:43:18.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:18 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/75759711' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-03-20T09:43:18.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:18 smithi122 ceph-mon[73495]: osdmap e1122: 12 total, 12 up, 11 in 2024-03-20T09:43:18.686 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:18 smithi122 ceph-mon[73495]: 1.8 scrub starts 2024-03-20T09:43:18.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:18 smithi122 ceph-mon[73495]: 1.8 scrub ok 2024-03-20T09:43:18.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:18 smithi122 ceph-mon[73495]: 1.4 scrub starts 2024-03-20T09:43:18.687 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:18 smithi122 ceph-mon[73495]: 1.4 scrub ok 2024-03-20T09:43:18.707 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:18 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/75759711' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-03-20T09:43:18.707 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:18 smithi026 ceph-mon[68309]: osdmap e1122: 12 total, 12 up, 11 in 2024-03-20T09:43:18.707 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:18 smithi026 ceph-mon[68309]: 1.8 scrub starts 2024-03-20T09:43:18.708 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:18 smithi026 ceph-mon[68309]: 1.8 scrub ok 2024-03-20T09:43:18.708 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:18 smithi026 ceph-mon[68309]: 1.4 scrub starts 2024-03-20T09:43:18.708 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:18 smithi026 ceph-mon[68309]: 1.4 scrub ok 2024-03-20T09:43:19.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:19 smithi026 ceph-mon[68309]: 9.d scrub ok 2024-03-20T09:43:19.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:19 smithi026 ceph-mon[68309]: pgmap v2723: 47 pgs: 47 active+clean; 4.5 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 625 op/s 2024-03-20T09:43:19.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:19 smithi026 ceph-mon[68309]: 9.5 scrub starts 2024-03-20T09:43:19.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:19 smithi115 ceph-mon[72476]: 9.d scrub ok 2024-03-20T09:43:19.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:19 smithi115 ceph-mon[72476]: pgmap v2723: 47 pgs: 47 active+clean; 4.5 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 625 op/s 2024-03-20T09:43:19.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:19 smithi115 ceph-mon[72476]: 9.5 scrub starts 2024-03-20T09:43:19.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:19 smithi122 ceph-mon[73495]: 9.d scrub ok 2024-03-20T09:43:19.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:19 smithi122 ceph-mon[73495]: pgmap v2723: 47 pgs: 47 active+clean; 4.5 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 625 op/s 2024-03-20T09:43:19.684 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:19 smithi122 ceph-mon[73495]: 9.5 scrub starts 2024-03-20T09:43:19.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:19.815 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:43:19.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:43:19.916 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:43:19.950 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:20.401 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_ops_in_flight 2024-03-20T09:43:20.646 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:20 smithi115 ceph-mon[72476]: 9.7 scrub starts 2024-03-20T09:43:20.646 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:20 smithi115 ceph-mon[72476]: 9.b scrub starts 2024-03-20T09:43:20.685 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:20 smithi122 ceph-mon[73495]: 9.7 scrub starts 2024-03-20T09:43:20.685 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:20 smithi122 ceph-mon[73495]: 9.b scrub starts 2024-03-20T09:43:20.707 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:20 smithi026 ceph-mon[68309]: 9.7 scrub starts 2024-03-20T09:43:20.707 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:20 smithi026 ceph-mon[68309]: 9.b scrub starts 2024-03-20T09:43:20.812 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 11 1 2024-03-20T09:43:20.844 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:43:21.071 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:43:21.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:21.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:43:21.176 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:43:21.229 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:21.269 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:43:21.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:21.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:43:21.377 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:43:21.417 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:21.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:21 smithi122 ceph-mon[73495]: 9.5 scrub ok 2024-03-20T09:43:21.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:21 smithi122 ceph-mon[73495]: pgmap v2724: 47 pgs: 47 active+clean; 4.3 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 925 op/s 2024-03-20T09:43:21.448 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:43:21.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:21.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:21 smithi026 ceph-mon[68309]: 9.5 scrub ok 2024-03-20T09:43:21.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:21 smithi026 ceph-mon[68309]: pgmap v2724: 47 pgs: 47 active+clean; 4.3 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 925 op/s 2024-03-20T09:43:21.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:43:21.560 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:43:21.611 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:43:21.626 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:21.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:21 smithi115 ceph-mon[72476]: 9.5 scrub ok 2024-03-20T09:43:21.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:21 smithi115 ceph-mon[72476]: pgmap v2724: 47 pgs: 47 active+clean; 4.3 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 925 op/s 2024-03-20T09:43:21.927 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:43:21.932 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:21.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:43:22.034 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:43:22.070 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:22.337 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:43:22.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:22 smithi026 ceph-mon[68309]: 9.7 scrub ok 2024-03-20T09:43:22.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:22 smithi026 ceph-mon[68309]: 9.b scrub ok 2024-03-20T09:43:22.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:22 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1980133734' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:43:22.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:22 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:43:22.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:22 smithi115 ceph-mon[72476]: 9.7 scrub ok 2024-03-20T09:43:22.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:22 smithi115 ceph-mon[72476]: 9.b scrub ok 2024-03-20T09:43:22.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:22 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1980133734' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:43:22.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:22 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:43:22.684 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:22 smithi122 ceph-mon[73495]: 9.7 scrub ok 2024-03-20T09:43:22.684 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:22 smithi122 ceph-mon[73495]: 9.b scrub ok 2024-03-20T09:43:22.684 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:22 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1980133734' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:43:22.684 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:22 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:43:23.152 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:43:23.152 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-03-20T09:43:23.152 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:23.152 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:197151 9.b 9:d28fe8a0:::benchmark_data_smithi122_210164_object64921:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1123)", 2024-03-20T09:43:23.152 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:43:23.157084+0000", 2024-03-20T09:43:23.152 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.0075768820000000001, 2024-03-20T09:43:23.152 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0076093489999999996, 2024-03-20T09:43:23.153 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:43:23.153 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:43:23.153 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:43:23.153 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:43:23.153 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:43:23.153 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 197151 2024-03-20T09:43:23.153 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:23.153 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:43:23.153 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:23.153 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:43:23.153 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:23.157084+0000", 2024-03-20T09:43:23.153 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:23.153 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:23.153 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:23.153 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:43:23.154 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:23.157084+0000", 2024-03-20T09:43:23.154 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:23.154 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:23.154 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:23.154 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:43:23.154 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:23.157086+0000", 2024-03-20T09:43:23.154 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9080000000000002e-06 2024-03-20T09:43:23.154 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:23.154 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:23.154 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:43:23.154 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:23.157095+0000", 2024-03-20T09:43:23.154 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7430000000000008e-06 2024-03-20T09:43:23.154 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:23.154 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:23.155 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:43:23.155 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:23.157115+0000", 2024-03-20T09:43:23.155 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0613999999999999e-05 2024-03-20T09:43:23.155 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:23.155 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:23.155 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:43:23.155 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:23.157125+0000", 2024-03-20T09:43:23.155 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.004e-05 2024-03-20T09:43:23.155 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:23.155 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:23.155 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:43:23.155 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:23.157187+0000", 2024-03-20T09:43:23.155 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1567000000000002e-05 2024-03-20T09:43:23.155 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:23.155 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:23.155 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:43:23.156 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:23.157324+0000", 2024-03-20T09:43:23.156 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000137002 2024-03-20T09:43:23.156 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:23.156 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:23.156 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,10", 2024-03-20T09:43:23.156 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:23.157463+0000", 2024-03-20T09:43:23.156 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013945299999999999 2024-03-20T09:43:23.156 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:23.156 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:23.156 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:23.156 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:23.163515+0000", 2024-03-20T09:43:23.156 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0060520390000000004 2024-03-20T09:43:23.156 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:23.156 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:23.156 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:43:23.157 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:23.163530+0000", 2024-03-20T09:43:23.157 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4178e-05 2024-03-20T09:43:23.157 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:23.157 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:43:23.157 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:23.157 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:23.157 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:23.157 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_repop(client.27190.0:197152 9.4 e1123/1066 9:230f3a16:::benchmark_data_smithi122_210164_object64922:head v 1123'12141, mlcod=1123'12139)", 2024-03-20T09:43:23.157 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:43:23.159396+0000", 2024-03-20T09:43:23.157 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.0052646510000000004, 2024-03-20T09:43:23.157 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0053905990000000003, 2024-03-20T09:43:23.157 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:43:23.157 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-03-20T09:43:23.157 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:43:23.157 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:23.157 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:43:23.158 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:23.159396+0000", 2024-03-20T09:43:23.158 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:23.158 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:23.158 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:23.158 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:43:23.158 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:23.159396+0000", 2024-03-20T09:43:23.158 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:23.158 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:23.158 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:23.158 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:43:23.158 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:23.159397+0000", 2024-03-20T09:43:23.158 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0189999999999999e-06 2024-03-20T09:43:23.158 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:23.158 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:23.159 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:43:23.159 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:23.159405+0000", 2024-03-20T09:43:23.159 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5900000000000002e-06 2024-03-20T09:43:23.159 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:23.159 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:23.159 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:43:23.159 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:23.159416+0000", 2024-03-20T09:43:23.159 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0807e-05 2024-03-20T09:43:23.159 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:23.159 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:23.159 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:43:23.159 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:23.159423+0000", 2024-03-20T09:43:23.159 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1500000000000002e-06 2024-03-20T09:43:23.159 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:23.160 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:23.160 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:43:23.160 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:23.161450+0000", 2024-03-20T09:43:23.160 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0020267929999999998 2024-03-20T09:43:23.160 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:23.160 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:23.160 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:43:23.160 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:23.161471+0000", 2024-03-20T09:43:23.160 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1212999999999998e-05 2024-03-20T09:43:23.160 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:23.160 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:43:23.160 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:23.160 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:23.160 INFO:teuthology.orchestra.run.smithi122.stdout: ], 2024-03-20T09:43:23.161 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 2 2024-03-20T09:43:23.161 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:43:23.197 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:43:23.230 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:43:23.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:23.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:43:23.347 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:43:23.387 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:23.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:23 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:43:23.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:23 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:43:23.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:23 smithi122 ceph-mon[73495]: osdmap e1123: 12 total, 12 up, 11 in 2024-03-20T09:43:23.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:23 smithi122 ceph-mon[73495]: pgmap v2726: 47 pgs: 47 active+clean; 4.3 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 900 op/s 2024-03-20T09:43:23.571 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:23 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:43:23.571 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:23 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:43:23.571 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:23 smithi026 ceph-mon[68309]: osdmap e1123: 12 total, 12 up, 11 in 2024-03-20T09:43:23.571 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:23 smithi026 ceph-mon[68309]: pgmap v2726: 47 pgs: 47 active+clean; 4.3 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 900 op/s 2024-03-20T09:43:23.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:23 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:43:23.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:23 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:43:23.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:23 smithi115 ceph-mon[72476]: osdmap e1123: 12 total, 12 up, 11 in 2024-03-20T09:43:23.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:23 smithi115 ceph-mon[72476]: pgmap v2726: 47 pgs: 47 active+clean; 4.3 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 900 op/s 2024-03-20T09:43:23.863 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:43:23.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:23.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:43:23.970 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:43:24.025 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:24.159 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:43:24.484 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.11 to 1 (10000) 2024-03-20T09:43:24.707 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:24 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2069648045' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-03-20T09:43:24.893 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:24 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2069648045' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-03-20T09:43:24.933 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:24 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2069648045' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-03-20T09:43:25.616 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:43:25.662 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:25 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2069648045' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-03-20T09:43:25.662 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:25 smithi026 ceph-mon[68309]: osdmap e1124: 12 total, 12 up, 11 in 2024-03-20T09:43:25.662 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:25 smithi026 ceph-mon[68309]: pgmap v2728: 47 pgs: 47 active+clean; 4.2 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 761 op/s 2024-03-20T09:43:25.893 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:25 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2069648045' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-03-20T09:43:25.893 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:25 smithi115 ceph-mon[72476]: osdmap e1124: 12 total, 12 up, 11 in 2024-03-20T09:43:25.893 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:25 smithi115 ceph-mon[72476]: pgmap v2728: 47 pgs: 47 active+clean; 4.2 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 761 op/s 2024-03-20T09:43:25.933 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:25 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2069648045' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-03-20T09:43:25.934 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:25 smithi122 ceph-mon[73495]: osdmap e1124: 12 total, 12 up, 11 in 2024-03-20T09:43:25.934 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:25 smithi122 ceph-mon[73495]: pgmap v2728: 47 pgs: 47 active+clean; 4.2 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 761 op/s 2024-03-20T09:43:26.070 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_blocked_ops 2024-03-20T09:43:26.097 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:43:26.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:26.103 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:43:26.204 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:43:26.228 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 3 1 2024-03-20T09:43:26.254 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:26.448 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:43:26.456 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:26.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:43:26.468 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:43:26.557 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:43:26.596 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:26.628 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:43:26.635 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:26.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:43:26.736 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:43:26.754 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:43:26.798 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:26.827 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:43:26.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:26.833 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:43:26.835 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:43:26.870 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:43:26.897 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:43:26.926 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:43:26.934 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:43:26.957 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:43:26.989 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:43:26.993 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:27.034 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:43:27.038 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:43:27.041 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:43:27.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:27.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:43:27.066 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:43:27.095 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:43:27.116 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:43:27.137 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:43:27.146 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:43:27.157 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:43:27.158 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:43:27.158 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:43:27.158 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:43:27.158 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:43:27.158 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:43:27.159 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:43:27.159 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:43:27.159 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:43:27.159 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:43:27.159 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:43:27.159 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:43:27.159 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:43:27.172 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:27.191 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:43:27.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:27.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:43:27.295 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:43:27.334 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:27.378 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:43:27.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:27.389 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:43:27.490 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:43:27.531 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:27.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:27 smithi026 ceph-mon[68309]: pgmap v2729: 47 pgs: 47 active+clean; 4.0 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s 2024-03-20T09:43:27.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:27 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/52639735' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:43:28.001 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:27 smithi122 ceph-mon[73495]: pgmap v2729: 47 pgs: 47 active+clean; 4.0 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s 2024-03-20T09:43:28.002 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:27 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/52639735' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:43:28.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:27 smithi115 ceph-mon[72476]: pgmap v2729: 47 pgs: 47 active+clean; 4.0 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s 2024-03-20T09:43:28.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:27 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/52639735' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:43:28.392 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:43:28.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:28.398 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:43:28.499 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:43:28.556 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:28.712 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:43:28.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:28.720 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:43:28.821 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:43:28.872 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:28.902 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:43:28.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:28.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:43:28.974 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:28 smithi122 ceph-mon[73495]: pgmap v2730: 47 pgs: 47 active+clean; 4.0 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 642 op/s 2024-03-20T09:43:29.008 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:43:29.048 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:28 smithi026 ceph-mon[68309]: pgmap v2730: 47 pgs: 47 active+clean; 4.0 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 642 op/s 2024-03-20T09:43:29.058 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:29.085 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:43:29.090 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:29.090 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:43:29.122 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:43:29.122 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:43:29.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:28 smithi115 ceph-mon[72476]: pgmap v2730: 47 pgs: 47 active+clean; 4.0 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 642 op/s 2024-03-20T09:43:29.191 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:43:29.242 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:29.392 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:43:29.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:29.396 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:43:29.496 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:43:29.551 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:29.588 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:43:29.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:29.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:43:29.692 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:43:29.730 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:29.859 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_historic_ops 2024-03-20T09:43:29.879 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:43:29.887 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:29.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:43:29.988 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:43:30.031 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:30.133 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:29 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:43:30.133 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:29 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:43:30.133 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:29 smithi026 ceph-mon[68309]: osdmap e1125: 12 total, 12 up, 11 in 2024-03-20T09:43:30.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:29 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:43:30.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:29 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:43:30.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:29 smithi122 ceph-mon[73495]: osdmap e1125: 12 total, 12 up, 11 in 2024-03-20T09:43:30.256 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:43:30.273 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:43:30.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:29 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:43:30.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:29 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:43:30.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:29 smithi115 ceph-mon[72476]: osdmap e1125: 12 total, 12 up, 11 in 2024-03-20T09:43:31.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:30 smithi115 ceph-mon[72476]: pgmap v2732: 47 pgs: 47 active+clean; 3.8 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 887 op/s 2024-03-20T09:43:31.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:30 smithi122 ceph-mon[73495]: pgmap v2732: 47 pgs: 47 active+clean; 3.8 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 887 op/s 2024-03-20T09:43:31.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:30 smithi026 ceph-mon[68309]: pgmap v2732: 47 pgs: 47 active+clean; 3.8 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 887 op/s 2024-03-20T09:43:31.992 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:43:32.006 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:32.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:43:32.094 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:43:32.097 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.3 to 1 (10000) 2024-03-20T09:43:32.108 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:43:32.119 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:43:32.157 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:32.350 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:32 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/440294262' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:43:32.350 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:32 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/662497398' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-20T09:43:32.350 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:32 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-20T09:43:32.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:32 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/440294262' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:43:32.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:32 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/662497398' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-20T09:43:32.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:32 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-20T09:43:32.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:32 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/440294262' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:43:32.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:32 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/662497398' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-20T09:43:32.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:32 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-20T09:43:32.633 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:43:32.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:32.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:43:32.738 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:43:32.771 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:32.798 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:43:32.805 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:32.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:43:32.907 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:43:32.942 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:32.975 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:43:32.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:32.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:43:33.083 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:43:33.103 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:33 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:43:33.103 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:33 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/440294262' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:43:33.103 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:33 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-20T09:43:33.103 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:33 smithi122 ceph-mon[73495]: osdmap e1126: 12 total, 12 up, 11 in 2024-03-20T09:43:33.103 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:33 smithi122 ceph-mon[73495]: pgmap v2734: 47 pgs: 47 active+clean; 3.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 835 op/s 2024-03-20T09:43:33.134 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:33.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:33 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:43:33.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:33 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/440294262' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:43:33.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:33 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-20T09:43:33.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:33 smithi026 ceph-mon[68309]: osdmap e1126: 12 total, 12 up, 11 in 2024-03-20T09:43:33.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:33 smithi026 ceph-mon[68309]: pgmap v2734: 47 pgs: 47 active+clean; 3.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 835 op/s 2024-03-20T09:43:33.217 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:43:33.217 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:43:33.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:33 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:43:33.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:33 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/440294262' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:43:33.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:33 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-20T09:43:33.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:33 smithi115 ceph-mon[72476]: osdmap e1126: 12 total, 12 up, 11 in 2024-03-20T09:43:33.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:33 smithi115 ceph-mon[72476]: pgmap v2734: 47 pgs: 47 active+clean; 3.8 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 835 op/s 2024-03-20T09:43:34.210 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_ops_in_flight 2024-03-20T09:43:34.361 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 4 1 2024-03-20T09:43:34.497 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:43:34.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:34.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:43:34.602 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:43:34.623 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:43:34.649 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:34.686 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:43:34.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:34.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:43:34.793 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:43:34.829 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:34.881 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:43:34.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:34.886 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:43:34.987 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:43:35.028 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:35.056 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:43:35.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:35.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:43:35.161 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:43:35.214 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:35.259 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:43:35.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:35.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:43:35.367 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:43:35.403 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:43:35.422 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:35.703 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:43:35.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:35.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:43:35.808 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:43:35.844 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:35.950 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:35 smithi026 ceph-mon[68309]: pgmap v2735: 47 pgs: 47 active+clean; 3.7 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 687 op/s 2024-03-20T09:43:35.951 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:43:36.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:35 smithi115 ceph-mon[72476]: pgmap v2735: 47 pgs: 47 active+clean; 3.7 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 687 op/s 2024-03-20T09:43:36.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:35 smithi122 ceph-mon[73495]: pgmap v2735: 47 pgs: 47 active+clean; 3.7 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 687 op/s 2024-03-20T09:43:36.415 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:43:36.422 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:36.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:43:36.523 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:43:36.564 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:36.686 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:43:36.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:36.695 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:43:36.796 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:43:36.837 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:36.899 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:43:36.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:36.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:43:36.969 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:43:37.006 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:43:37.028 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:43:37.035 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:43:37.072 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:37.073 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:43:37.123 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:43:37.157 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:43:37.201 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:43:37.240 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:43:37.267 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:43:37.296 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:43:37.329 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:43:37.353 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:43:37.354 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:43:37.354 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:43:37.354 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:43:37.354 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:43:37.354 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:43:37.354 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:43:37.354 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:43:37.355 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:43:37.355 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:43:37.355 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:43:37.355 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:43:37.355 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:43:37.445 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:43:37.445 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-03-20T09:43:37.446 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:37.446 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:209069 9.3 9:c5ee89a9:::benchmark_data_smithi122_210164_object76839:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1126)", 2024-03-20T09:43:37.446 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:43:37.445379+0000", 2024-03-20T09:43:37.446 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.0067122340000000001, 2024-03-20T09:43:37.446 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0067497900000000003, 2024-03-20T09:43:37.446 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:43:37.446 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:43:37.446 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:43:37.446 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:43:37.446 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:43:37.446 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 209069 2024-03-20T09:43:37.446 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:37.447 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:43:37.447 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:37.447 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:43:37.447 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:37.445379+0000", 2024-03-20T09:43:37.447 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:37.447 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:37.447 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:37.447 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:43:37.447 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:37.445379+0000", 2024-03-20T09:43:37.447 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:37.447 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:37.447 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:37.447 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:43:37.447 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:37.445380+0000", 2024-03-20T09:43:37.447 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.294e-06 2024-03-20T09:43:37.448 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:37.448 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:37.448 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:43:37.448 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:37.445387+0000", 2024-03-20T09:43:37.448 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6050000000000003e-06 2024-03-20T09:43:37.448 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:37.448 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:37.448 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:43:37.448 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:37.445405+0000", 2024-03-20T09:43:37.448 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7999000000000002e-05 2024-03-20T09:43:37.448 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:37.448 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:37.448 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:43:37.448 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:37.445415+0000", 2024-03-20T09:43:37.449 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0206e-05 2024-03-20T09:43:37.449 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:37.449 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:37.449 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:43:37.449 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:37.445478+0000", 2024-03-20T09:43:37.449 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3485999999999995e-05 2024-03-20T09:43:37.449 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:37.449 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:37.449 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:43:37.449 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:37.445626+0000", 2024-03-20T09:43:37.449 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014714599999999999 2024-03-20T09:43:37.449 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:37.449 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:37.449 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,5", 2024-03-20T09:43:37.449 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:37.445768+0000", 2024-03-20T09:43:37.450 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000142006 2024-03-20T09:43:37.450 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:37.450 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:37.450 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:43:37.450 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:37.451019+0000", 2024-03-20T09:43:37.450 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0052516560000000004 2024-03-20T09:43:37.450 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:37.450 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:43:37.450 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:37.450 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:37.450 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:37.450 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:209072 9.8 9:17360a3d:::benchmark_data_smithi122_210164_object76842:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1126)", 2024-03-20T09:43:37.450 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:43:37.449458+0000", 2024-03-20T09:43:37.450 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.0026333540000000001, 2024-03-20T09:43:37.451 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027573089999999999, 2024-03-20T09:43:37.451 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:43:37.451 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:43:37.451 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:43:37.451 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:43:37.451 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:43:37.451 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 209072 2024-03-20T09:43:37.451 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:37.451 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:43:37.451 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:37.451 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:43:37.451 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:37.449458+0000", 2024-03-20T09:43:37.451 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:37.451 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:37.452 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:37.452 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:43:37.452 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:37.449458+0000", 2024-03-20T09:43:37.452 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:37.452 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:37.452 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:37.452 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:43:37.452 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:37.449459+0000", 2024-03-20T09:43:37.452 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6500000000000008e-07 2024-03-20T09:43:37.452 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:37.452 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:37.452 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:43:37.452 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:37.449464+0000", 2024-03-20T09:43:37.452 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.045e-06 2024-03-20T09:43:37.452 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:37.453 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:37.453 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:43:37.453 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:37.449480+0000", 2024-03-20T09:43:37.453 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6552999999999999e-05 2024-03-20T09:43:37.453 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:37.453 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:37.453 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:43:37.453 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:37.449489+0000", 2024-03-20T09:43:37.453 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7790000000000001e-06 2024-03-20T09:43:37.453 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:37.453 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:37.453 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:43:37.453 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:37.449566+0000", 2024-03-20T09:43:37.454 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7228e-05 2024-03-20T09:43:37.454 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:37.454 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:37.454 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:43:37.454 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:37.449681+0000", 2024-03-20T09:43:37.454 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000115151 2024-03-20T09:43:37.454 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:37.454 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:37.454 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,11", 2024-03-20T09:43:37.454 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:37.449797+0000", 2024-03-20T09:43:37.454 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000115204 2024-03-20T09:43:37.454 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:37.454 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:43:37.454 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:37.455 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:37.455 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:37.455 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:209073 9.8 9.2d241f38 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1126)", 2024-03-20T09:43:37.455 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:43:37.449511+0000", 2024-03-20T09:43:37.455 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 0.0025804449999999998, 2024-03-20T09:43:37.455 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0027809380000000002, 2024-03-20T09:43:37.455 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:43:37.455 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "queued for pg", 2024-03-20T09:43:37.455 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:43:37.455 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:43:37.455 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:43:37.455 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 209073 2024-03-20T09:43:37.455 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:37.455 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:43:37.456 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:37.456 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:43:37.456 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:37.449511+0000", 2024-03-20T09:43:37.456 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:37.456 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:37.456 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:37.456 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:43:37.456 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:37.449511+0000", 2024-03-20T09:43:37.456 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:37.456 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:37.456 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:37.456 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:43:37.456 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:37.449511+0000", 2024-03-20T09:43:37.456 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4400000000000001e-07 2024-03-20T09:43:37.457 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:37.457 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:37.457 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:43:37.457 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:37.449514+0000", 2024-03-20T09:43:37.457 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1810000000000001e-06 2024-03-20T09:43:37.457 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:37.457 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:37.457 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:43:37.457 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:37.449522+0000", 2024-03-20T09:43:37.457 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0059999999999992e-06 2024-03-20T09:43:37.457 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:37.457 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:37.457 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:43:37.457 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:37.449527+0000", 2024-03-20T09:43:37.457 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5839999999999996e-06 2024-03-20T09:43:37.457 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:37.458 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:43:37.458 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:37.458 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:37.458 INFO:teuthology.orchestra.run.smithi122.stdout: ], 2024-03-20T09:43:37.458 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 3 2024-03-20T09:43:37.458 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:43:38.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:37 smithi115 ceph-mon[72476]: pgmap v2736: 47 pgs: 47 active+clean; 3.5 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 977 op/s 2024-03-20T09:43:38.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:37 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3823118406' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:43:38.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:37 smithi122 ceph-mon[73495]: pgmap v2736: 47 pgs: 47 active+clean; 3.5 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 977 op/s 2024-03-20T09:43:38.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:37 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3823118406' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:43:38.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:37 smithi026 ceph-mon[68309]: pgmap v2736: 47 pgs: 47 active+clean; 3.5 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 977 op/s 2024-03-20T09:43:38.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:37 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3823118406' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:43:38.259 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:43:38.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:38.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:43:38.271 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:43:38.362 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:43:38.394 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:38.522 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:43:38.526 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:38.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:43:38.627 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:43:38.689 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:38.742 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:43:38.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:38.747 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:43:38.787 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_blocked_ops 2024-03-20T09:43:38.848 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:43:38.926 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:38.955 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:43:38.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:38.964 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:43:39.065 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:43:39.095 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:39.129 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:43:39.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:39.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:43:39.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:39 smithi122 ceph-mon[73495]: pgmap v2737: 47 pgs: 47 active+clean; 3.5 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 783 op/s 2024-03-20T09:43:39.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:39 smithi026 ceph-mon[68309]: pgmap v2737: 47 pgs: 47 active+clean; 3.5 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 783 op/s 2024-03-20T09:43:39.236 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:43:39.248 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:43:39.275 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:39.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:39 smithi115 ceph-mon[72476]: pgmap v2737: 47 pgs: 47 active+clean; 3.5 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 783 op/s 2024-03-20T09:43:40.126 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.4 to 1 (10000) 2024-03-20T09:43:40.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:40 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/543508223' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-20T09:43:40.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:40 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/543508223' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-20T09:43:40.424 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:40 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/543508223' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-20T09:43:40.832 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:43:40.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:40.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:43:40.937 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:43:40.976 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:41.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:41 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/543508223' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-20T09:43:41.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:41 smithi026 ceph-mon[68309]: osdmap e1127: 12 total, 12 up, 11 in 2024-03-20T09:43:41.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:41 smithi026 ceph-mon[68309]: pgmap v2739: 47 pgs: 47 active+clean; 3.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 877 op/s 2024-03-20T09:43:41.339 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:41 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/543508223' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-20T09:43:41.339 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:41 smithi122 ceph-mon[73495]: osdmap e1127: 12 total, 12 up, 11 in 2024-03-20T09:43:41.339 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:41 smithi122 ceph-mon[73495]: pgmap v2739: 47 pgs: 47 active+clean; 3.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 877 op/s 2024-03-20T09:43:41.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:41 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/543508223' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-20T09:43:41.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:41 smithi115 ceph-mon[72476]: osdmap e1127: 12 total, 12 up, 11 in 2024-03-20T09:43:41.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:41 smithi115 ceph-mon[72476]: pgmap v2739: 47 pgs: 47 active+clean; 3.3 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 877 op/s 2024-03-20T09:43:41.754 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:43:41.754 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:43:41.755 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-03-20T09:43:41.755 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-03-20T09:43:41.755 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:43:42.227 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 9 1 2024-03-20T09:43:42.438 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:43:42.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:42.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:43:42.538 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_historic_ops 2024-03-20T09:43:42.545 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:43:42.598 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:42.644 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:43:42.648 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:43:42.658 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:42.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:43:42.760 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:43:42.804 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:42.910 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:43:42.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:42.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:43:42.942 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:43:43.018 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:43:43.059 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:43:43.071 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:43.100 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:43:43.105 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:43.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:43:43.206 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:43:43.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:43 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:43:43.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:43 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:43:43.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:43 smithi026 ceph-mon[68309]: osdmap e1128: 12 total, 12 up, 11 in 2024-03-20T09:43:43.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:43 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/952542600' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:43:43.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:43 smithi026 ceph-mon[68309]: pgmap v2741: 47 pgs: 47 active+clean; 3.2 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 911 op/s 2024-03-20T09:43:43.265 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:43.294 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:43:43.297 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:43.298 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:43:43.382 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:43:43.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:43 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:43:43.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:43 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:43:43.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:43 smithi115 ceph-mon[72476]: osdmap e1128: 12 total, 12 up, 11 in 2024-03-20T09:43:43.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:43 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/952542600' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:43:43.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:43 smithi115 ceph-mon[72476]: pgmap v2741: 47 pgs: 47 active+clean; 3.2 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 911 op/s 2024-03-20T09:43:43.399 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:43:43.400 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:43 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:43:43.400 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:43 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:43:43.400 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:43 smithi122 ceph-mon[73495]: osdmap e1128: 12 total, 12 up, 11 in 2024-03-20T09:43:43.400 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:43 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/952542600' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:43:43.400 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:43 smithi122 ceph-mon[73495]: pgmap v2741: 47 pgs: 47 active+clean; 3.2 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 911 op/s 2024-03-20T09:43:43.445 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:43.479 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:43:43.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:43.483 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:43:43.584 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:43:43.633 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:43.666 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:43:43.669 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:43.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:43:43.771 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:43:43.813 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:44.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:44 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:43:44.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:44 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/952542600' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:43:44.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:44 smithi115 ceph-mon[72476]: osdmap e1129: 12 total, 12 up, 11 in 2024-03-20T09:43:44.406 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:44 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:43:44.406 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:44 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/952542600' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:43:44.406 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:44 smithi026 ceph-mon[68309]: osdmap e1129: 12 total, 12 up, 11 in 2024-03-20T09:43:44.427 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:44 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:43:44.427 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:44 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/952542600' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:43:44.427 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:44 smithi122 ceph-mon[73495]: osdmap e1129: 12 total, 12 up, 11 in 2024-03-20T09:43:44.875 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:43:44.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:44.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:43:44.914 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:43:44.980 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:43:45.026 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:45.056 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:43:45.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:45 smithi122 ceph-mon[73495]: pgmap v2743: 47 pgs: 47 active+clean; 3.2 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 973 op/s 2024-03-20T09:43:45.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:45 smithi122 ceph-mon[73495]: 1.19 scrub starts 2024-03-20T09:43:45.060 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:45 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:43:45.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:45.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:43:45.162 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:43:45.202 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:45 smithi026 ceph-mon[68309]: pgmap v2743: 47 pgs: 47 active+clean; 3.2 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 973 op/s 2024-03-20T09:43:45.202 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:45 smithi026 ceph-mon[68309]: 1.19 scrub starts 2024-03-20T09:43:45.202 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:45 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:43:45.205 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:45.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:45 smithi115 ceph-mon[72476]: pgmap v2743: 47 pgs: 47 active+clean; 3.2 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 973 op/s 2024-03-20T09:43:45.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:45 smithi115 ceph-mon[72476]: 1.19 scrub starts 2024-03-20T09:43:45.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:45 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:43:45.899 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:43:45.899 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-03-20T09:43:45.899 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-03-20T09:43:45.899 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-03-20T09:43:45.899 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.899 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:11286 8.1 8:80a48722:::benchmark_data_smithi122_197775_object11285:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e980)", 2024-03-20T09:43:45.899 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:36:04.077724+0000", 2024-03-20T09:43:45.900 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 461.830232018, 2024-03-20T09:43:45.900 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20547268299999999, 2024-03-20T09:43:45.900 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:43:45.900 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:43:45.900 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:43:45.900 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:43:45.900 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:43:45.900 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 11286 2024-03-20T09:43:45.900 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.900 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:43:45.900 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.900 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:43:45.900 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.077724+0000", 2024-03-20T09:43:45.900 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:45.900 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.901 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.901 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:43:45.901 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.077724+0000", 2024-03-20T09:43:45.901 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:45.901 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.901 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.901 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:43:45.901 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.077726+0000", 2024-03-20T09:43:45.901 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4720000000000001e-06 2024-03-20T09:43:45.901 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.901 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.901 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:43:45.901 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.077753+0000", 2024-03-20T09:43:45.901 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7667000000000001e-05 2024-03-20T09:43:45.901 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.902 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.902 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:43:45.902 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.077769+0000", 2024-03-20T09:43:45.902 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5857000000000001e-05 2024-03-20T09:43:45.902 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.902 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.902 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:43:45.902 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.077778+0000", 2024-03-20T09:43:45.902 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9639999999999992e-06 2024-03-20T09:43:45.902 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.902 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.902 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:43:45.902 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.077839+0000", 2024-03-20T09:43:45.903 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1326000000000002e-05 2024-03-20T09:43:45.903 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.903 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.903 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:43:45.903 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.077952+0000", 2024-03-20T09:43:45.903 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000112459 2024-03-20T09:43:45.903 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.903 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.903 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,2", 2024-03-20T09:43:45.903 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.078075+0000", 2024-03-20T09:43:45.903 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012269200000000001 2024-03-20T09:43:45.903 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.903 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.903 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:43:45.903 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.079021+0000", 2024-03-20T09:43:45.904 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00094592199999999997 2024-03-20T09:43:45.904 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.904 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.904 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:45.904 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.080268+0000", 2024-03-20T09:43:45.904 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0012472799999999999 2024-03-20T09:43:45.904 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.904 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.904 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:45.904 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.283126+0000", 2024-03-20T09:43:45.904 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20285774600000001 2024-03-20T09:43:45.904 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.904 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.904 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:43:45.905 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.283159+0000", 2024-03-20T09:43:45.905 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3890999999999998e-05 2024-03-20T09:43:45.905 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.905 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.905 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:43:45.905 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.283197+0000", 2024-03-20T09:43:45.905 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7407000000000003e-05 2024-03-20T09:43:45.905 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:45.905 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:43:45.905 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:45.905 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.905 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.905 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:11290 8.1 8:80c13a40:::benchmark_data_smithi122_197775_object11289:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e980)", 2024-03-20T09:43:45.905 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:36:04.079238+0000", 2024-03-20T09:43:45.906 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 461.828717907, 2024-03-20T09:43:45.906 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20571076599999999, 2024-03-20T09:43:45.906 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:43:45.906 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:43:45.906 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:43:45.906 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:43:45.906 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:43:45.906 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 11290 2024-03-20T09:43:45.906 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.906 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:43:45.906 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.906 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:43:45.906 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.079238+0000", 2024-03-20T09:43:45.907 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:45.907 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.907 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.907 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:43:45.907 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.079238+0000", 2024-03-20T09:43:45.907 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:45.907 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.907 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.907 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:43:45.907 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.079240+0000", 2024-03-20T09:43:45.907 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.291e-06 2024-03-20T09:43:45.907 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.907 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.907 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:43:45.907 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.079262+0000", 2024-03-20T09:43:45.908 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2673000000000001e-05 2024-03-20T09:43:45.908 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.908 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.908 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:43:45.908 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.079275+0000", 2024-03-20T09:43:45.908 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3011e-05 2024-03-20T09:43:45.908 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.908 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.908 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:43:45.908 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.079294+0000", 2024-03-20T09:43:45.908 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8641e-05 2024-03-20T09:43:45.908 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.908 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.908 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:43:45.908 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.079356+0000", 2024-03-20T09:43:45.909 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2083999999999997e-05 2024-03-20T09:43:45.909 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.909 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.909 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:43:45.909 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.079516+0000", 2024-03-20T09:43:45.909 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016011000000000001 2024-03-20T09:43:45.909 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.909 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.909 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,2", 2024-03-20T09:43:45.909 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.079662+0000", 2024-03-20T09:43:45.909 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014622499999999999 2024-03-20T09:43:45.909 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.909 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.909 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:43:45.909 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.080579+0000", 2024-03-20T09:43:45.909 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00091626100000000003 2024-03-20T09:43:45.910 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.910 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.910 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:45.910 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.081707+0000", 2024-03-20T09:43:45.910 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0011281010000000001 2024-03-20T09:43:45.910 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.910 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.910 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:45.910 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.284880+0000", 2024-03-20T09:43:45.910 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.203173522 2024-03-20T09:43:45.910 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.910 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.910 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:43:45.910 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.284912+0000", 2024-03-20T09:43:45.910 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1931999999999999e-05 2024-03-20T09:43:45.911 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.911 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.911 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:43:45.911 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.284949+0000", 2024-03-20T09:43:45.911 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6915000000000001e-05 2024-03-20T09:43:45.911 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:45.911 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:43:45.911 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:45.911 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.911 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.911 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:11293 8.1 8:8391fa78:::benchmark_data_smithi122_197775_object11292:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e980)", 2024-03-20T09:43:45.911 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:36:04.080539+0000", 2024-03-20T09:43:45.911 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 461.82741730800001, 2024-03-20T09:43:45.911 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20506249200000001, 2024-03-20T09:43:45.911 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:43:45.911 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:43:45.912 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:43:45.912 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:43:45.912 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:43:45.912 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 11293 2024-03-20T09:43:45.912 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.912 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:43:45.912 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.912 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:43:45.912 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.080539+0000", 2024-03-20T09:43:45.912 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:45.912 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.912 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.912 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:43:45.912 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.080539+0000", 2024-03-20T09:43:45.913 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:45.913 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.913 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.913 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:43:45.913 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.080540+0000", 2024-03-20T09:43:45.913 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2359999999999999e-06 2024-03-20T09:43:45.913 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.913 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.913 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:43:45.913 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.080570+0000", 2024-03-20T09:43:45.913 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9910999999999999e-05 2024-03-20T09:43:45.913 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.913 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.913 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:43:45.914 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.080581+0000", 2024-03-20T09:43:45.914 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1286e-05 2024-03-20T09:43:45.914 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.914 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.914 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:43:45.914 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.080589+0000", 2024-03-20T09:43:45.914 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6240000000000002e-06 2024-03-20T09:43:45.914 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.914 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.914 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:43:45.914 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.080660+0000", 2024-03-20T09:43:45.914 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0764000000000002e-05 2024-03-20T09:43:45.914 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.914 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.914 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:43:45.915 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.080767+0000", 2024-03-20T09:43:45.915 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010772099999999999 2024-03-20T09:43:45.915 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.915 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.915 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,2", 2024-03-20T09:43:45.915 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.080882+0000", 2024-03-20T09:43:45.915 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000114647 2024-03-20T09:43:45.915 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.915 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.915 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:43:45.915 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.081465+0000", 2024-03-20T09:43:45.915 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00058243499999999996 2024-03-20T09:43:45.915 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.915 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.915 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:45.916 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.081836+0000", 2024-03-20T09:43:45.916 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00037194099999999998 2024-03-20T09:43:45.916 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.916 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.916 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:45.916 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.285534+0000", 2024-03-20T09:43:45.916 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.203697238 2024-03-20T09:43:45.916 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.916 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.916 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:43:45.916 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.285564+0000", 2024-03-20T09:43:45.916 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0026e-05 2024-03-20T09:43:45.916 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.917 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.917 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:43:45.917 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.285601+0000", 2024-03-20T09:43:45.917 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7663000000000002e-05 2024-03-20T09:43:45.917 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:45.917 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:43:45.917 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:45.917 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.917 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.917 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:11300 8.1 8:8fc41842:::benchmark_data_smithi122_197775_object11299:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e980)", 2024-03-20T09:43:45.917 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:36:04.082159+0000", 2024-03-20T09:43:45.917 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 461.82579684500001, 2024-03-20T09:43:45.917 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20485677899999999, 2024-03-20T09:43:45.917 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:43:45.918 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:43:45.918 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:43:45.918 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:43:45.918 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:43:45.918 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 11300 2024-03-20T09:43:45.918 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.918 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:43:45.918 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.918 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:43:45.918 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.082159+0000", 2024-03-20T09:43:45.918 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:45.918 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.918 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.918 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:43:45.919 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.082159+0000", 2024-03-20T09:43:45.919 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:45.919 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.919 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.919 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:43:45.919 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.082160+0000", 2024-03-20T09:43:45.919 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5899999999999996e-07 2024-03-20T09:43:45.919 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.919 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.919 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:43:45.919 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.082182+0000", 2024-03-20T09:43:45.919 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1566e-05 2024-03-20T09:43:45.919 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.919 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.920 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:43:45.920 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.082196+0000", 2024-03-20T09:43:45.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4599e-05 2024-03-20T09:43:45.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.920 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.920 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:43:45.920 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.082203+0000", 2024-03-20T09:43:45.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1110000000000002e-06 2024-03-20T09:43:45.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.920 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.920 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:43:45.920 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.083032+0000", 2024-03-20T09:43:45.920 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00082847699999999999 2024-03-20T09:43:45.920 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.920 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.921 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:43:45.921 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.083148+0000", 2024-03-20T09:43:45.921 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000116221 2024-03-20T09:43:45.921 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.921 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.921 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,2", 2024-03-20T09:43:45.921 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.083293+0000", 2024-03-20T09:43:45.921 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000144578 2024-03-20T09:43:45.921 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.921 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.921 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:43:45.921 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.084208+0000", 2024-03-20T09:43:45.921 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00091588100000000001 2024-03-20T09:43:45.921 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.922 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.922 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:45.922 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.084838+0000", 2024-03-20T09:43:45.922 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00062971700000000002 2024-03-20T09:43:45.922 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.922 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.922 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:45.922 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.286934+0000", 2024-03-20T09:43:45.922 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.202095357 2024-03-20T09:43:45.922 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.922 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.922 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:43:45.922 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.286972+0000", 2024-03-20T09:43:45.923 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8900999999999999e-05 2024-03-20T09:43:45.923 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.923 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.923 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:43:45.923 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:04.287016+0000", 2024-03-20T09:43:45.923 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3711999999999998e-05 2024-03-20T09:43:45.923 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:45.923 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:43:45.923 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:45.923 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.923 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.923 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:238411 8.1 8:86d760dd:::benchmark_data_smithi122_197775_object115107:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1056)", 2024-03-20T09:43:45.923 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:39:55.048854+0000", 2024-03-20T09:43:45.923 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 230.859101864, 2024-03-20T09:43:45.923 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.216121966, 2024-03-20T09:43:45.924 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:43:45.924 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:43:45.924 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:43:45.924 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:43:45.924 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:43:45.924 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 238411 2024-03-20T09:43:45.924 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.924 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:43:45.924 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.924 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:43:45.924 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.048854+0000", 2024-03-20T09:43:45.924 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:45.924 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.924 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.925 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:43:45.925 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.048854+0000", 2024-03-20T09:43:45.925 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:45.925 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.925 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.925 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:43:45.925 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.048855+0000", 2024-03-20T09:43:45.925 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7500000000000001e-07 2024-03-20T09:43:45.925 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.925 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:43:45.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.048858+0000", 2024-03-20T09:43:45.926 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2200000000000001e-06 2024-03-20T09:43:45.926 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.926 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:43:45.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.048864+0000", 2024-03-20T09:43:45.926 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.421e-06 2024-03-20T09:43:45.926 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.926 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.926 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:43:45.926 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.048870+0000", 2024-03-20T09:43:45.926 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4e-06 2024-03-20T09:43:45.926 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.926 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.927 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:43:45.927 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.053672+0000", 2024-03-20T09:43:45.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0048026240000000001 2024-03-20T09:43:45.927 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.927 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.927 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:43:45.927 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.053784+0000", 2024-03-20T09:43:45.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000111839 2024-03-20T09:43:45.927 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.927 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.927 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,2", 2024-03-20T09:43:45.927 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.053854+0000", 2024-03-20T09:43:45.927 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9915999999999999e-05 2024-03-20T09:43:45.927 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.928 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.928 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:43:45.928 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.057950+0000", 2024-03-20T09:43:45.928 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040960759999999997 2024-03-20T09:43:45.928 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.928 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.928 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:45.928 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.061752+0000", 2024-03-20T09:43:45.928 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0038016019999999998 2024-03-20T09:43:45.928 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.928 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.928 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:45.928 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.264887+0000", 2024-03-20T09:43:45.928 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.203135396 2024-03-20T09:43:45.928 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.928 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.929 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:43:45.929 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.264938+0000", 2024-03-20T09:43:45.929 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0494000000000001e-05 2024-03-20T09:43:45.929 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.929 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.929 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:43:45.929 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.264976+0000", 2024-03-20T09:43:45.929 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8603000000000003e-05 2024-03-20T09:43:45.929 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:45.929 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:43:45.929 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:45.929 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.929 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.929 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:238427 8.1 8:8f1c7409:::benchmark_data_smithi122_197775_object115123:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1056)", 2024-03-20T09:43:45.929 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:39:55.066135+0000", 2024-03-20T09:43:45.930 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 230.84182143000001, 2024-03-20T09:43:45.930 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.21237999900000001, 2024-03-20T09:43:45.930 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:43:45.930 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:43:45.930 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:43:45.930 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:43:45.930 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:43:45.930 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 238427 2024-03-20T09:43:45.930 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.930 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:43:45.930 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.930 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:43:45.930 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.066135+0000", 2024-03-20T09:43:45.930 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:45.930 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.931 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.931 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:43:45.931 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.066135+0000", 2024-03-20T09:43:45.931 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:45.931 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.931 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.931 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:43:45.931 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.066136+0000", 2024-03-20T09:43:45.931 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.519e-06 2024-03-20T09:43:45.931 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.931 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.931 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:43:45.931 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.066142+0000", 2024-03-20T09:43:45.931 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1140000000000002e-06 2024-03-20T09:43:45.931 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.932 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.932 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:43:45.932 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.066154+0000", 2024-03-20T09:43:45.932 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1418e-05 2024-03-20T09:43:45.932 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.932 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.932 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:43:45.932 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.066162+0000", 2024-03-20T09:43:45.932 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2570000000000007e-06 2024-03-20T09:43:45.932 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.932 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.932 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:43:45.932 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.066228+0000", 2024-03-20T09:43:45.932 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5634000000000002e-05 2024-03-20T09:43:45.932 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.933 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:43:45.933 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.066360+0000", 2024-03-20T09:43:45.933 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000132352 2024-03-20T09:43:45.933 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.933 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,2", 2024-03-20T09:43:45.933 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.066467+0000", 2024-03-20T09:43:45.933 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000106449 2024-03-20T09:43:45.933 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.933 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.933 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:43:45.933 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.071716+0000", 2024-03-20T09:43:45.933 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0052489870000000001 2024-03-20T09:43:45.934 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.934 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.934 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:45.934 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.080421+0000", 2024-03-20T09:43:45.934 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0087058599999999993 2024-03-20T09:43:45.934 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.934 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.934 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:45.934 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.278418+0000", 2024-03-20T09:43:45.934 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.197996124 2024-03-20T09:43:45.934 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.934 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.934 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:43:45.934 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.278471+0000", 2024-03-20T09:43:45.935 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3458000000000002e-05 2024-03-20T09:43:45.935 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.935 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.935 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:43:45.935 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.278515+0000", 2024-03-20T09:43:45.935 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3826999999999999e-05 2024-03-20T09:43:45.935 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:45.935 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:43:45.935 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:45.935 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.935 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.935 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:11182 9.3 9:c1e11bf1:::benchmark_data_smithi122_210164_object11181:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1077)", 2024-03-20T09:43:45.935 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:40:43.040902+0000", 2024-03-20T09:43:45.935 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 182.86705417600001, 2024-03-20T09:43:45.935 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20538585200000001, 2024-03-20T09:43:45.936 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:43:45.936 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:43:45.936 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:43:45.936 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:43:45.936 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:43:45.936 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 11182 2024-03-20T09:43:45.936 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.936 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:43:45.936 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.936 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:43:45.936 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.040902+0000", 2024-03-20T09:43:45.936 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:45.936 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.936 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.936 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:43:45.937 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.040902+0000", 2024-03-20T09:43:45.937 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:45.937 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.937 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.937 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:43:45.937 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.040903+0000", 2024-03-20T09:43:45.937 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.215e-06 2024-03-20T09:43:45.937 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.937 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.937 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:43:45.937 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.040923+0000", 2024-03-20T09:43:45.937 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9304999999999999e-05 2024-03-20T09:43:45.937 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.937 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.937 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:43:45.938 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.040936+0000", 2024-03-20T09:43:45.938 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3487e-05 2024-03-20T09:43:45.938 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.938 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.938 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:43:45.938 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.040942+0000", 2024-03-20T09:43:45.938 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2790000000000004e-06 2024-03-20T09:43:45.938 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.938 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.938 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:43:45.938 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.041002+0000", 2024-03-20T09:43:45.938 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9181000000000001e-05 2024-03-20T09:43:45.938 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.938 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.939 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:43:45.939 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.041139+0000", 2024-03-20T09:43:45.939 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000137646 2024-03-20T09:43:45.939 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.939 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.939 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,5", 2024-03-20T09:43:45.939 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.041263+0000", 2024-03-20T09:43:45.939 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012335800000000001 2024-03-20T09:43:45.939 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.939 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.939 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:43:45.939 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.042375+0000", 2024-03-20T09:43:45.939 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001112624 2024-03-20T09:43:45.939 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.939 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.940 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:45.940 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.044200+0000", 2024-03-20T09:43:45.940 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0018248100000000001 2024-03-20T09:43:45.940 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.940 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.940 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:45.940 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.246210+0000", 2024-03-20T09:43:45.940 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20201024100000001 2024-03-20T09:43:45.940 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.940 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.940 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:43:45.940 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.246250+0000", 2024-03-20T09:43:45.940 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9422999999999998e-05 2024-03-20T09:43:45.940 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.940 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.941 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:43:45.941 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.246288+0000", 2024-03-20T09:43:45.941 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8282999999999999e-05 2024-03-20T09:43:45.941 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:45.941 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:43:45.941 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:45.941 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.941 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.941 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:11185 9.3 9:cf95aa79:::benchmark_data_smithi122_210164_object11184:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1077)", 2024-03-20T09:43:45.941 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:40:43.042516+0000", 2024-03-20T09:43:45.941 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 182.86544033499999, 2024-03-20T09:43:45.941 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20459010499999999, 2024-03-20T09:43:45.941 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:43:45.941 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:43:45.941 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:43:45.941 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:43:45.942 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:43:45.942 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 11185 2024-03-20T09:43:45.942 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.942 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:43:45.942 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.942 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:43:45.942 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.042516+0000", 2024-03-20T09:43:45.942 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:45.942 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.942 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.942 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:43:45.942 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.042516+0000", 2024-03-20T09:43:45.942 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:45.943 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.943 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.943 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:43:45.943 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.042517+0000", 2024-03-20T09:43:45.943 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.19e-06 2024-03-20T09:43:45.943 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.943 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.943 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:43:45.943 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.042539+0000", 2024-03-20T09:43:45.943 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1953000000000001e-05 2024-03-20T09:43:45.943 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.943 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.943 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:43:45.943 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.042551+0000", 2024-03-20T09:43:45.943 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1605e-05 2024-03-20T09:43:45.944 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.944 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.944 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:43:45.944 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.042559+0000", 2024-03-20T09:43:45.944 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3410000000000003e-06 2024-03-20T09:43:45.944 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.944 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.944 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:43:45.944 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.042617+0000", 2024-03-20T09:43:45.944 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8263000000000001e-05 2024-03-20T09:43:45.944 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.944 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.944 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:43:45.944 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.042781+0000", 2024-03-20T09:43:45.944 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016405200000000001 2024-03-20T09:43:45.944 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.945 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.945 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,5", 2024-03-20T09:43:45.945 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.042934+0000", 2024-03-20T09:43:45.945 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015226899999999999 2024-03-20T09:43:45.945 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.945 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.945 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:45.945 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.044256+0000", 2024-03-20T09:43:45.945 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0013219480000000001 2024-03-20T09:43:45.945 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.945 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.945 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:43:45.945 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.044303+0000", 2024-03-20T09:43:45.945 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7048000000000001e-05 2024-03-20T09:43:45.945 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.946 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.946 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:45.946 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.247006+0000", 2024-03-20T09:43:45.946 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.202703042 2024-03-20T09:43:45.946 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.946 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.946 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:43:45.946 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.247057+0000", 2024-03-20T09:43:45.946 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1313999999999997e-05 2024-03-20T09:43:45.946 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.946 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.946 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:43:45.946 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.247106+0000", 2024-03-20T09:43:45.946 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9079999999999998e-05 2024-03-20T09:43:45.947 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:45.947 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:43:45.947 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:45.947 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.947 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.947 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:11188 9.3 9:ca80a92d:::benchmark_data_smithi122_210164_object11187:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1077)", 2024-03-20T09:43:45.947 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:40:43.042667+0000", 2024-03-20T09:43:45.947 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 182.86528969400001, 2024-03-20T09:43:45.947 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20458264000000001, 2024-03-20T09:43:45.947 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:43:45.947 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:43:45.947 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:43:45.947 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:43:45.948 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:43:45.948 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 11188 2024-03-20T09:43:45.948 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.948 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:43:45.948 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.948 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:43:45.948 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.042667+0000", 2024-03-20T09:43:45.948 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:45.948 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.948 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.948 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:43:45.948 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.042667+0000", 2024-03-20T09:43:45.948 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:45.948 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.948 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.948 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:43:45.949 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.042667+0000", 2024-03-20T09:43:45.949 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3399999999999997e-07 2024-03-20T09:43:45.949 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.949 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.949 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:43:45.949 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.042690+0000", 2024-03-20T09:43:45.949 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2909999999999999e-05 2024-03-20T09:43:45.949 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.949 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.949 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:43:45.949 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.042702+0000", 2024-03-20T09:43:45.949 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1654000000000001e-05 2024-03-20T09:43:45.949 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.949 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.949 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:43:45.949 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.042710+0000", 2024-03-20T09:43:45.950 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9370000000000004e-06 2024-03-20T09:43:45.950 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.950 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.950 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:43:45.950 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.043500+0000", 2024-03-20T09:43:45.950 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00078967100000000004 2024-03-20T09:43:45.950 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.950 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.950 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:43:45.950 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.043610+0000", 2024-03-20T09:43:45.950 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000110308 2024-03-20T09:43:45.950 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.950 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.950 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,5", 2024-03-20T09:43:45.950 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.043739+0000", 2024-03-20T09:43:45.951 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012889100000000001 2024-03-20T09:43:45.951 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.951 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.951 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:43:45.951 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.044451+0000", 2024-03-20T09:43:45.951 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00071210900000000005 2024-03-20T09:43:45.951 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.951 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.951 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:45.951 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.046356+0000", 2024-03-20T09:43:45.951 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0019048349999999999 2024-03-20T09:43:45.951 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.951 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.951 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:45.951 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.247148+0000", 2024-03-20T09:43:45.952 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.200792199 2024-03-20T09:43:45.952 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.952 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.952 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:43:45.952 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.247207+0000", 2024-03-20T09:43:45.952 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9242e-05 2024-03-20T09:43:45.952 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.952 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.952 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:43:45.952 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.247249+0000", 2024-03-20T09:43:45.952 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1950000000000003e-05 2024-03-20T09:43:45.952 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:45.952 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:43:45.952 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:45.952 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.953 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.953 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:123495 9.8 9:19bc1614:::benchmark_data_smithi122_210164_object123494:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1097)", 2024-03-20T09:43:45.953 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:41:54.581171+0000", 2024-03-20T09:43:45.953 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 111.326784848, 2024-03-20T09:43:45.953 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.21403297299999999, 2024-03-20T09:43:45.953 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:43:45.953 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:43:45.953 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:43:45.953 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:43:45.953 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:43:45.953 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 123495 2024-03-20T09:43:45.953 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.953 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:43:45.953 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.954 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:43:45.954 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581171+0000", 2024-03-20T09:43:45.954 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:45.954 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.954 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.954 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:43:45.954 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581171+0000", 2024-03-20T09:43:45.954 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:45.954 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.954 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.954 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:43:45.954 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581173+0000", 2024-03-20T09:43:45.954 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3179999999999999e-06 2024-03-20T09:43:45.954 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.955 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.955 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:43:45.955 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581197+0000", 2024-03-20T09:43:45.955 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4071e-05 2024-03-20T09:43:45.955 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.955 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.955 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:43:45.955 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581208+0000", 2024-03-20T09:43:45.955 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1379999999999999e-05 2024-03-20T09:43:45.955 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.955 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.955 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:43:45.955 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581214+0000", 2024-03-20T09:43:45.955 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1399999999999997e-06 2024-03-20T09:43:45.955 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.956 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.956 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:43:45.956 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581273+0000", 2024-03-20T09:43:45.956 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8227999999999999e-05 2024-03-20T09:43:45.956 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.956 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.956 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:43:45.956 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581408+0000", 2024-03-20T09:43:45.956 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013551899999999999 2024-03-20T09:43:45.956 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.956 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.956 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,11", 2024-03-20T09:43:45.956 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581500+0000", 2024-03-20T09:43:45.956 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2250999999999995e-05 2024-03-20T09:43:45.956 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.957 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.957 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:45.957 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.597971+0000", 2024-03-20T09:43:45.957 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016470496000000001 2024-03-20T09:43:45.957 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.957 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.957 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:43:45.957 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.598344+0000", 2024-03-20T09:43:45.957 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00037296299999999997 2024-03-20T09:43:45.957 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.957 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.957 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:45.957 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.795088+0000", 2024-03-20T09:43:45.957 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19674451900000001 2024-03-20T09:43:45.957 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.958 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.958 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:43:45.958 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.795154+0000", 2024-03-20T09:43:45.958 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5895000000000005e-05 2024-03-20T09:43:45.958 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.958 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.958 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:43:45.958 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.795204+0000", 2024-03-20T09:43:45.958 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0192999999999999e-05 2024-03-20T09:43:45.958 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:45.958 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:43:45.958 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:45.958 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.958 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.959 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:123498 9.8 9:15c518f2:::benchmark_data_smithi122_210164_object123497:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1097)", 2024-03-20T09:43:45.959 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:41:54.581234+0000", 2024-03-20T09:43:45.959 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 111.326722725, 2024-03-20T09:43:45.959 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.22821238999999999, 2024-03-20T09:43:45.959 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:43:45.959 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:43:45.959 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:43:45.959 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:43:45.959 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:43:45.959 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 123498 2024-03-20T09:43:45.959 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.959 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:43:45.959 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.959 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:43:45.959 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581234+0000", 2024-03-20T09:43:45.960 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:45.960 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.960 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.960 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:43:45.960 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581234+0000", 2024-03-20T09:43:45.960 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:45.960 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.960 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.960 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:43:45.960 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581234+0000", 2024-03-20T09:43:45.960 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7399999999999998e-07 2024-03-20T09:43:45.960 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.960 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.960 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:43:45.960 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581259+0000", 2024-03-20T09:43:45.961 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5176000000000001e-05 2024-03-20T09:43:45.961 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.961 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.961 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:43:45.961 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581266+0000", 2024-03-20T09:43:45.961 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8560000000000001e-06 2024-03-20T09:43:45.961 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.961 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.961 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:43:45.961 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581271+0000", 2024-03-20T09:43:45.961 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3720000000000004e-06 2024-03-20T09:43:45.961 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.961 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.961 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:43:45.961 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.585053+0000", 2024-03-20T09:43:45.962 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003781207 2024-03-20T09:43:45.962 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.962 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.962 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:43:45.962 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.585194+0000", 2024-03-20T09:43:45.962 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014127500000000001 2024-03-20T09:43:45.962 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.962 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.962 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,11", 2024-03-20T09:43:45.962 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.585342+0000", 2024-03-20T09:43:45.962 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014771399999999999 2024-03-20T09:43:45.962 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.962 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.962 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:45.963 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.598026+0000", 2024-03-20T09:43:45.963 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012684465000000001 2024-03-20T09:43:45.963 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.963 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.963 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:43:45.963 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.598351+0000", 2024-03-20T09:43:45.963 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00032446200000000003 2024-03-20T09:43:45.963 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.963 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.963 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:45.963 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.809328+0000", 2024-03-20T09:43:45.963 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.210977058 2024-03-20T09:43:45.963 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.963 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.964 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:43:45.964 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.809395+0000", 2024-03-20T09:43:45.964 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7188999999999997e-05 2024-03-20T09:43:45.964 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.964 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.964 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:43:45.964 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.809446+0000", 2024-03-20T09:43:45.964 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1141999999999999e-05 2024-03-20T09:43:45.964 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:45.964 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:43:45.964 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:45.964 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.964 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.964 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:123499 9.8 9:1ff497bd:::benchmark_data_smithi122_210164_object123498:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1097)", 2024-03-20T09:43:45.965 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:41:54.581280+0000", 2024-03-20T09:43:45.965 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 111.32667583200001, 2024-03-20T09:43:45.965 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.228306763, 2024-03-20T09:43:45.965 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:43:45.965 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:43:45.965 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:43:45.965 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:43:45.965 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:43:45.965 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 123499 2024-03-20T09:43:45.965 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.965 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:43:45.965 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.965 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:43:45.965 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581280+0000", 2024-03-20T09:43:45.966 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:45.966 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.966 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.966 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:43:45.966 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581280+0000", 2024-03-20T09:43:45.966 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:45.966 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.966 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.966 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:43:45.966 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581287+0000", 2024-03-20T09:43:45.966 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2430000000000002e-06 2024-03-20T09:43:45.966 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.966 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.966 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:43:45.967 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581310+0000", 2024-03-20T09:43:45.967 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.336e-05 2024-03-20T09:43:45.967 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.967 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.967 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:43:45.967 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581317+0000", 2024-03-20T09:43:45.967 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5189999999999998e-06 2024-03-20T09:43:45.967 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.967 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.967 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:43:45.967 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581322+0000", 2024-03-20T09:43:45.967 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1209999999999998e-06 2024-03-20T09:43:45.967 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.967 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.968 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:43:45.968 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.590394+0000", 2024-03-20T09:43:45.968 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0090721670000000008 2024-03-20T09:43:45.968 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.968 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.968 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:43:45.968 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.590524+0000", 2024-03-20T09:43:45.968 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000130278 2024-03-20T09:43:45.968 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.968 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.968 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,11", 2024-03-20T09:43:45.968 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.590617+0000", 2024-03-20T09:43:45.968 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3398000000000003e-05 2024-03-20T09:43:45.968 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.968 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.969 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:43:45.969 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.598356+0000", 2024-03-20T09:43:45.969 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0077385320000000002 2024-03-20T09:43:45.969 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.969 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.969 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:45.969 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.599682+0000", 2024-03-20T09:43:45.969 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0013261449999999999 2024-03-20T09:43:45.969 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.969 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.969 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:45.969 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.809492+0000", 2024-03-20T09:43:45.969 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.209810303 2024-03-20T09:43:45.969 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.969 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.970 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:43:45.970 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.809542+0000", 2024-03-20T09:43:45.970 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9919000000000001e-05 2024-03-20T09:43:45.970 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.970 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.970 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:43:45.970 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.809587+0000", 2024-03-20T09:43:45.970 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4777999999999997e-05 2024-03-20T09:43:45.970 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:45.970 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:43:45.970 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:45.970 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.970 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.970 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:123507 9.8 9:106df26e:::benchmark_data_smithi122_210164_object123506:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1097)", 2024-03-20T09:43:45.970 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:41:54.590309+0000", 2024-03-20T09:43:45.971 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 111.317647456, 2024-03-20T09:43:45.971 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.21939439699999999, 2024-03-20T09:43:45.971 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:43:45.971 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:43:45.971 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:43:45.971 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:43:45.971 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:43:45.971 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 123507 2024-03-20T09:43:45.971 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.971 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:43:45.971 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.971 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:43:45.971 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.590309+0000", 2024-03-20T09:43:45.971 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:45.971 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.972 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.972 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:43:45.972 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.590309+0000", 2024-03-20T09:43:45.972 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:45.972 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.972 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.972 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:43:45.972 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.590310+0000", 2024-03-20T09:43:45.972 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.576e-06 2024-03-20T09:43:45.972 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.972 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.972 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:43:45.972 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.590330+0000", 2024-03-20T09:43:45.972 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.946e-05 2024-03-20T09:43:45.973 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.973 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.973 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:43:45.973 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.590339+0000", 2024-03-20T09:43:45.973 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5810000000000003e-06 2024-03-20T09:43:45.973 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.973 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.973 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:43:45.973 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.590345+0000", 2024-03-20T09:43:45.973 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9739999999999999e-06 2024-03-20T09:43:45.973 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.973 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.973 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:43:45.973 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.593877+0000", 2024-03-20T09:43:45.974 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0035314790000000001 2024-03-20T09:43:45.974 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.974 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.974 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:43:45.974 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.594048+0000", 2024-03-20T09:43:45.974 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000171476 2024-03-20T09:43:45.974 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.974 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.974 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,11", 2024-03-20T09:43:45.974 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.594201+0000", 2024-03-20T09:43:45.974 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000152499 2024-03-20T09:43:45.974 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.974 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.974 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:43:45.974 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.598465+0000", 2024-03-20T09:43:45.975 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0042645019999999999 2024-03-20T09:43:45.975 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.975 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.975 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:45.975 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.599855+0000", 2024-03-20T09:43:45.975 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00138944 2024-03-20T09:43:45.975 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.975 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.975 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:45.975 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.809622+0000", 2024-03-20T09:43:45.975 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.209767017 2024-03-20T09:43:45.975 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.975 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.975 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:43:45.975 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.809669+0000", 2024-03-20T09:43:45.976 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7030000000000002e-05 2024-03-20T09:43:45.976 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.976 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.976 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:43:45.976 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.809703+0000", 2024-03-20T09:43:45.976 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4363000000000003e-05 2024-03-20T09:43:45.976 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:45.976 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:43:45.976 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:45.976 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.976 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.976 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:123514 9.3 9:cac7a75a:::benchmark_data_smithi122_210164_object123513:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1097)", 2024-03-20T09:43:45.976 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:41:54.607199+0000", 2024-03-20T09:43:45.976 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 111.30075751, 2024-03-20T09:43:45.977 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20607972799999999, 2024-03-20T09:43:45.977 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:43:45.977 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:43:45.977 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:43:45.977 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:43:45.977 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:43:45.977 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 123514 2024-03-20T09:43:45.977 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.977 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:43:45.977 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.977 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:43:45.977 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.607199+0000", 2024-03-20T09:43:45.977 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:45.977 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.978 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.978 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:43:45.978 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.607199+0000", 2024-03-20T09:43:45.978 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:45.978 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.978 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.978 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:43:45.978 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.607200+0000", 2024-03-20T09:43:45.978 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.409e-06 2024-03-20T09:43:45.978 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.978 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.978 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:43:45.978 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.607229+0000", 2024-03-20T09:43:45.978 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8921999999999999e-05 2024-03-20T09:43:45.979 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.979 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.979 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:43:45.979 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.607244+0000", 2024-03-20T09:43:45.979 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5339999999999999e-05 2024-03-20T09:43:45.979 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.979 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.979 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:43:45.979 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.607254+0000", 2024-03-20T09:43:45.979 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7520000000000008e-06 2024-03-20T09:43:45.979 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.979 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.979 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:43:45.979 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.607328+0000", 2024-03-20T09:43:45.979 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4337000000000003e-05 2024-03-20T09:43:45.979 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.980 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.980 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:43:45.980 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.607472+0000", 2024-03-20T09:43:45.980 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000143373 2024-03-20T09:43:45.980 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.980 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.980 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,5", 2024-03-20T09:43:45.980 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.607605+0000", 2024-03-20T09:43:45.980 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000132867 2024-03-20T09:43:45.980 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.980 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.980 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:45.980 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.611496+0000", 2024-03-20T09:43:45.980 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0038913799999999998 2024-03-20T09:43:45.981 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.981 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.981 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:43:45.981 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.614508+0000", 2024-03-20T09:43:45.981 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0030114759999999999 2024-03-20T09:43:45.981 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.981 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.981 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:45.981 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.813169+0000", 2024-03-20T09:43:45.981 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19866141600000001 2024-03-20T09:43:45.981 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.981 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.981 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:43:45.981 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.813232+0000", 2024-03-20T09:43:45.982 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2836999999999995e-05 2024-03-20T09:43:45.982 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.982 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.982 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:43:45.982 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.813278+0000", 2024-03-20T09:43:45.982 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6619000000000002e-05 2024-03-20T09:43:45.982 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:45.982 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:43:45.982 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:45.982 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.982 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.982 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:123515 9.8 9:1b95f987:::benchmark_data_smithi122_210164_object123514:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1097)", 2024-03-20T09:43:45.982 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:41:54.607280+0000", 2024-03-20T09:43:45.982 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 111.30067581199999, 2024-03-20T09:43:45.983 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.206131808, 2024-03-20T09:43:45.983 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:43:45.983 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:43:45.983 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:43:45.983 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:43:45.983 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:43:45.983 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 123515 2024-03-20T09:43:45.983 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.983 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:43:45.983 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.983 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:43:45.983 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.607280+0000", 2024-03-20T09:43:45.983 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:45.983 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.983 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.984 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:43:45.984 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.607280+0000", 2024-03-20T09:43:45.984 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:45.984 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.984 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.984 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:43:45.984 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.607289+0000", 2024-03-20T09:43:45.984 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1300000000000001e-06 2024-03-20T09:43:45.984 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.984 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.984 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:43:45.984 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.607307+0000", 2024-03-20T09:43:45.984 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8258999999999999e-05 2024-03-20T09:43:45.984 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.985 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.985 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:43:45.985 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.607315+0000", 2024-03-20T09:43:45.985 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6119999999999999e-06 2024-03-20T09:43:45.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.985 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.985 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:43:45.985 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.607322+0000", 2024-03-20T09:43:45.985 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0990000000000004e-06 2024-03-20T09:43:45.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.985 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.985 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:43:45.985 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.607363+0000", 2024-03-20T09:43:45.985 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1353000000000002e-05 2024-03-20T09:43:45.985 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.986 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.986 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:43:45.986 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.607458+0000", 2024-03-20T09:43:45.986 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4815999999999995e-05 2024-03-20T09:43:45.986 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.986 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.986 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,11", 2024-03-20T09:43:45.986 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.607593+0000", 2024-03-20T09:43:45.986 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013547400000000001 2024-03-20T09:43:45.986 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.986 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.986 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:43:45.986 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.614489+0000", 2024-03-20T09:43:45.986 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0068957300000000001 2024-03-20T09:43:45.987 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.987 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.987 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:45.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.615606+0000", 2024-03-20T09:43:45.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0011172339999999999 2024-03-20T09:43:45.987 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.987 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.987 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:45.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.813313+0000", 2024-03-20T09:43:45.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19770711499999999 2024-03-20T09:43:45.987 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.987 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.987 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:43:45.987 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.813369+0000", 2024-03-20T09:43:45.987 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5621000000000002e-05 2024-03-20T09:43:45.988 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.988 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:43:45.988 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.813412+0000", 2024-03-20T09:43:45.988 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3365000000000002e-05 2024-03-20T09:43:45.988 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:45.988 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:43:45.988 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:45.988 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.988 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.988 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:126241 9.8 9:18639737:::benchmark_data_smithi122_210164_object126240:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1098)", 2024-03-20T09:43:45.988 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:41:59.145423+0000", 2024-03-20T09:43:45.988 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 106.762532841, 2024-03-20T09:43:45.988 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.25194071499999998, 2024-03-20T09:43:45.988 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:43:45.989 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:43:45.989 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:43:45.989 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:43:45.989 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:43:45.989 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 126241 2024-03-20T09:43:45.989 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.989 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:43:45.989 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.989 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:43:45.989 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.145423+0000", 2024-03-20T09:43:45.989 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:45.989 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.989 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.989 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:43:45.989 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.145423+0000", 2024-03-20T09:43:45.990 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:45.990 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.990 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:43:45.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.145424+0000", 2024-03-20T09:43:45.990 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5100000000000001e-07 2024-03-20T09:43:45.990 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.990 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:43:45.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.145441+0000", 2024-03-20T09:43:45.990 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7757e-05 2024-03-20T09:43:45.990 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.990 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.990 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:43:45.990 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.145451+0000", 2024-03-20T09:43:45.991 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5540000000000003e-06 2024-03-20T09:43:45.991 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.991 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.991 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:43:45.991 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.145456+0000", 2024-03-20T09:43:45.991 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2089999999999996e-06 2024-03-20T09:43:45.991 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.991 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.991 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:43:45.991 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.169525+0000", 2024-03-20T09:43:45.991 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.024068790999999999 2024-03-20T09:43:45.991 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.991 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.991 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:43:45.991 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.169656+0000", 2024-03-20T09:43:45.992 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013127000000000001 2024-03-20T09:43:45.992 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.992 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.992 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,11", 2024-03-20T09:43:45.992 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.169797+0000", 2024-03-20T09:43:45.992 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014071099999999999 2024-03-20T09:43:45.992 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.992 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.992 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:43:45.992 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.178789+0000", 2024-03-20T09:43:45.992 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0089917069999999998 2024-03-20T09:43:45.992 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.992 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.992 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:45.993 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.195371+0000", 2024-03-20T09:43:45.993 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016581953999999999 2024-03-20T09:43:45.993 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.993 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.993 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:45.993 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.397219+0000", 2024-03-20T09:43:45.993 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20184849199999999 2024-03-20T09:43:45.993 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.993 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.993 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:43:45.993 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.397301+0000", 2024-03-20T09:43:45.993 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1580000000000002e-05 2024-03-20T09:43:45.993 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.993 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.994 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:43:45.994 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.397364+0000", 2024-03-20T09:43:45.994 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3338999999999998e-05 2024-03-20T09:43:45.994 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:45.995 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:43:45.995 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:45.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.995 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.995 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:126260 9.8 9:11562cb1:::benchmark_data_smithi122_210164_object126259:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1098)", 2024-03-20T09:43:45.995 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:41:59.171581+0000", 2024-03-20T09:43:45.995 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 106.736375331, 2024-03-20T09:43:45.995 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.225880312, 2024-03-20T09:43:45.995 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:43:45.995 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:43:45.995 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:43:45.995 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:43:45.995 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:43:45.995 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 126260 2024-03-20T09:43:45.995 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.996 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:43:45.996 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:43:45.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.171581+0000", 2024-03-20T09:43:45.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:45.996 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.996 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:43:45.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.171581+0000", 2024-03-20T09:43:45.996 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:45.996 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.996 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.996 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:43:45.996 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.171582+0000", 2024-03-20T09:43:45.997 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3969999999999999e-06 2024-03-20T09:43:45.997 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.997 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.997 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:43:45.997 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.171610+0000", 2024-03-20T09:43:45.997 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7991e-05 2024-03-20T09:43:45.997 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.997 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.997 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:43:45.997 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.171625+0000", 2024-03-20T09:43:45.997 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4790000000000001e-05 2024-03-20T09:43:45.997 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.997 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.997 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:43:45.998 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.171633+0000", 2024-03-20T09:43:45.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9489999999999996e-06 2024-03-20T09:43:45.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.998 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.998 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:43:45.998 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.174434+0000", 2024-03-20T09:43:45.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0028006789999999999 2024-03-20T09:43:45.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.998 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.998 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:43:45.998 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.174564+0000", 2024-03-20T09:43:45.998 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013007200000000001 2024-03-20T09:43:45.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.999 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,11", 2024-03-20T09:43:45.999 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.174699+0000", 2024-03-20T09:43:45.999 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013544699999999999 2024-03-20T09:43:45.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:45.999 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:43:45.999 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.195096+0000", 2024-03-20T09:43:45.999 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.020396705000000001 2024-03-20T09:43:45.999 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:45.999 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.000 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:46.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.199947+0000", 2024-03-20T09:43:46.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0048511170000000003 2024-03-20T09:43:46.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.000 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:46.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.397393+0000", 2024-03-20T09:43:46.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.197445803 2024-03-20T09:43:46.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.000 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:43:46.000 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.397424+0000", 2024-03-20T09:43:46.000 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1341999999999999e-05 2024-03-20T09:43:46.000 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.000 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.001 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:43:46.001 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.397461+0000", 2024-03-20T09:43:46.001 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7020000000000001e-05 2024-03-20T09:43:46.001 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:46.001 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:43:46.001 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:46.001 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.001 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.001 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:126266 9.8 9:10df459b:::benchmark_data_smithi122_210164_object126265:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1098)", 2024-03-20T09:43:46.001 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:41:59.180692+0000", 2024-03-20T09:43:46.001 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 106.727264336, 2024-03-20T09:43:46.001 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.237022861, 2024-03-20T09:43:46.001 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:43:46.001 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:43:46.002 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:43:46.002 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:43:46.002 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:43:46.002 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 126266 2024-03-20T09:43:46.002 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.002 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:43:46.002 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.002 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:43:46.002 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.180692+0000", 2024-03-20T09:43:46.002 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:46.002 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.002 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.002 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:43:46.002 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.180692+0000", 2024-03-20T09:43:46.003 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:46.003 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.003 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.003 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:43:46.003 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.180693+0000", 2024-03-20T09:43:46.003 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.592e-06 2024-03-20T09:43:46.003 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.003 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.003 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:43:46.003 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.180712+0000", 2024-03-20T09:43:46.003 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8862e-05 2024-03-20T09:43:46.003 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.003 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.003 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:43:46.004 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.180729+0000", 2024-03-20T09:43:46.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7102999999999999e-05 2024-03-20T09:43:46.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.004 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.004 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:43:46.004 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.180738+0000", 2024-03-20T09:43:46.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9360000000000005e-06 2024-03-20T09:43:46.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.004 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.004 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:43:46.004 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.195444+0000", 2024-03-20T09:43:46.004 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014706032000000001 2024-03-20T09:43:46.004 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.004 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.005 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:43:46.005 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.195558+0000", 2024-03-20T09:43:46.005 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000113338 2024-03-20T09:43:46.005 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.005 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,11", 2024-03-20T09:43:46.005 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.195661+0000", 2024-03-20T09:43:46.005 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000103167 2024-03-20T09:43:46.005 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.005 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:43:46.005 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.200493+0000", 2024-03-20T09:43:46.005 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0048325490000000002 2024-03-20T09:43:46.005 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.005 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.006 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:46.006 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.200844+0000", 2024-03-20T09:43:46.006 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00035064999999999999 2024-03-20T09:43:46.006 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.006 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.006 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:46.006 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.417602+0000", 2024-03-20T09:43:46.006 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.21675767200000001 2024-03-20T09:43:46.006 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.006 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.006 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:43:46.006 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.417656+0000", 2024-03-20T09:43:46.006 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4558000000000002e-05 2024-03-20T09:43:46.006 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.007 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.007 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:43:46.007 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.417715+0000", 2024-03-20T09:43:46.007 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8402000000000001e-05 2024-03-20T09:43:46.007 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:46.007 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:43:46.007 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:46.007 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.007 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.007 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:126282 9.8 9:1febbbcb:::benchmark_data_smithi122_210164_object126281:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1098)", 2024-03-20T09:43:46.007 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:41:59.210497+0000", 2024-03-20T09:43:46.007 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 106.69745945699999, 2024-03-20T09:43:46.007 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20739027800000001, 2024-03-20T09:43:46.007 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:43:46.007 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:43:46.008 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:43:46.008 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:43:46.008 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:43:46.008 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 126282 2024-03-20T09:43:46.008 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.008 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:43:46.008 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.008 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:43:46.008 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.210497+0000", 2024-03-20T09:43:46.008 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:46.008 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.008 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.008 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:43:46.008 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.210497+0000", 2024-03-20T09:43:46.009 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:46.009 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.009 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.009 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:43:46.009 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.210498+0000", 2024-03-20T09:43:46.009 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5880000000000001e-06 2024-03-20T09:43:46.009 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.009 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.009 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:43:46.009 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.210519+0000", 2024-03-20T09:43:46.009 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0211999999999999e-05 2024-03-20T09:43:46.009 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.009 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.009 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:43:46.010 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.210531+0000", 2024-03-20T09:43:46.010 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2884999999999999e-05 2024-03-20T09:43:46.010 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.010 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.010 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:43:46.010 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.210537+0000", 2024-03-20T09:43:46.010 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8510000000000004e-06 2024-03-20T09:43:46.010 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.010 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.010 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:43:46.010 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.210597+0000", 2024-03-20T09:43:46.010 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0133000000000002e-05 2024-03-20T09:43:46.010 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.010 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.011 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:43:46.011 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.210750+0000", 2024-03-20T09:43:46.011 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015294500000000001 2024-03-20T09:43:46.011 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.011 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.011 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,11", 2024-03-20T09:43:46.011 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.210906+0000", 2024-03-20T09:43:46.011 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015554499999999999 2024-03-20T09:43:46.011 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.011 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.011 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:43:46.011 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.214843+0000", 2024-03-20T09:43:46.011 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0039371179999999999 2024-03-20T09:43:46.011 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.011 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.012 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:46.012 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.215938+0000", 2024-03-20T09:43:46.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001095026 2024-03-20T09:43:46.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.012 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.012 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:46.012 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.417774+0000", 2024-03-20T09:43:46.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20183589099999999 2024-03-20T09:43:46.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.012 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.012 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:43:46.012 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.417830+0000", 2024-03-20T09:43:46.012 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6441999999999999e-05 2024-03-20T09:43:46.012 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.013 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.013 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:43:46.013 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.417887+0000", 2024-03-20T09:43:46.013 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6641999999999997e-05 2024-03-20T09:43:46.013 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:46.013 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:43:46.013 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:46.013 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.013 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.013 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:179020 9.3 9:c8b2e0b2:::benchmark_data_smithi122_210164_object46790:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1115)", 2024-03-20T09:43:46.013 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:43:01.328331+0000", 2024-03-20T09:43:46.013 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 44.579625057999998, 2024-03-20T09:43:46.013 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.204369245, 2024-03-20T09:43:46.013 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:43:46.013 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:43:46.014 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:43:46.014 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:43:46.014 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:43:46.014 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 179020 2024-03-20T09:43:46.014 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.014 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:43:46.014 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.014 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:43:46.014 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:01.328331+0000", 2024-03-20T09:43:46.014 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:46.014 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.014 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.014 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:43:46.015 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:01.328331+0000", 2024-03-20T09:43:46.015 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:43:46.015 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.017 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.017 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:43:46.017 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:01.328332+0000", 2024-03-20T09:43:46.017 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8700000000000004e-07 2024-03-20T09:43:46.017 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.017 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.018 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:43:46.018 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:01.328337+0000", 2024-03-20T09:43:46.018 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1989999999999997e-06 2024-03-20T09:43:46.018 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.018 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.018 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:43:46.018 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:01.328349+0000", 2024-03-20T09:43:46.018 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1742e-05 2024-03-20T09:43:46.018 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.018 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.018 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:43:46.018 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:01.328358+0000", 2024-03-20T09:43:46.018 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1519999999999998e-06 2024-03-20T09:43:46.018 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.019 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.019 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:43:46.019 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:01.332413+0000", 2024-03-20T09:43:46.019 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040543999999999997 2024-03-20T09:43:46.019 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.019 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.019 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:43:46.019 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:01.332535+0000", 2024-03-20T09:43:46.019 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000122881 2024-03-20T09:43:46.019 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.019 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.019 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,5", 2024-03-20T09:43:46.019 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:01.332650+0000", 2024-03-20T09:43:46.019 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000115028 2024-03-20T09:43:46.020 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.020 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.020 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:43:46.020 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:01.337721+0000", 2024-03-20T09:43:46.020 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0050703850000000002 2024-03-20T09:43:46.020 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.020 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.020 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:46.020 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:01.338807+0000", 2024-03-20T09:43:46.020 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010861359999999999 2024-03-20T09:43:46.020 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.020 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.020 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:43:46.020 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:01.532603+0000", 2024-03-20T09:43:46.020 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19379576500000001 2024-03-20T09:43:46.021 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.021 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.021 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:43:46.021 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:01.532656+0000", 2024-03-20T09:43:46.021 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3199999999999999e-05 2024-03-20T09:43:46.021 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:43:46.021 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:43:46.021 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:43:46.021 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:01.532700+0000", 2024-03-20T09:43:46.021 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4469999999999999e-05 2024-03-20T09:43:46.021 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:46.021 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:43:46.021 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:46.021 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:43:46.021 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:43:46.021 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:43:46.046 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:43:46.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:46 smithi122 ceph-mon[73495]: 1.19 scrub ok 2024-03-20T09:43:46.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:46 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:43:46.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:46 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:43:46.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:46 smithi122 ceph-mon[73495]: osdmap e1130: 12 total, 12 up, 11 in 2024-03-20T09:43:46.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:46 smithi122 ceph-mon[73495]: 9.6 scrub starts 2024-03-20T09:43:46.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:46 smithi122 ceph-mon[73495]: 1.1d scrub starts 2024-03-20T09:43:46.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:46 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1289810824' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-03-20T09:43:46.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:46 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-03-20T09:43:46.215 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.9 to 1 (10000) 2024-03-20T09:43:46.272 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:43:46.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:46.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:43:46.378 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:43:46.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:46 smithi115 ceph-mon[72476]: 1.19 scrub ok 2024-03-20T09:43:46.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:46 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:43:46.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:46 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:43:46.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:46 smithi115 ceph-mon[72476]: osdmap e1130: 12 total, 12 up, 11 in 2024-03-20T09:43:46.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:46 smithi115 ceph-mon[72476]: 9.6 scrub starts 2024-03-20T09:43:46.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:46 smithi115 ceph-mon[72476]: 1.1d scrub starts 2024-03-20T09:43:46.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:46 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1289810824' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-03-20T09:43:46.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:46 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-03-20T09:43:46.421 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:46.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:46 smithi026 ceph-mon[68309]: 1.19 scrub ok 2024-03-20T09:43:46.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:46 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:43:46.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:46 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:43:46.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:46 smithi026 ceph-mon[68309]: osdmap e1130: 12 total, 12 up, 11 in 2024-03-20T09:43:46.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:46 smithi026 ceph-mon[68309]: 9.6 scrub starts 2024-03-20T09:43:46.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:46 smithi026 ceph-mon[68309]: 1.1d scrub starts 2024-03-20T09:43:46.463 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:46 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1289810824' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-03-20T09:43:46.464 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:46 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-03-20T09:43:46.632 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:43:46.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:46.646 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:43:46.678 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_ops_in_flight 2024-03-20T09:43:46.747 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:43:46.784 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:47.135 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:43:47.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:47 smithi115 ceph-mon[72476]: 1.1d scrub ok 2024-03-20T09:43:47.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:47 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-03-20T09:43:47.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:47 smithi115 ceph-mon[72476]: osdmap e1131: 12 total, 12 up, 11 in 2024-03-20T09:43:47.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:47 smithi115 ceph-mon[72476]: 9.6 scrub ok 2024-03-20T09:43:47.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:47 smithi115 ceph-mon[72476]: 1.d scrub starts 2024-03-20T09:43:47.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:47 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:43:47.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:47 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:43:47.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:47 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:43:47.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:47 smithi115 ceph-mon[72476]: pgmap v2746: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 903 op/s 2024-03-20T09:43:47.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:47 smithi122 ceph-mon[73495]: 1.1d scrub ok 2024-03-20T09:43:47.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:47 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-03-20T09:43:47.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:47 smithi122 ceph-mon[73495]: osdmap e1131: 12 total, 12 up, 11 in 2024-03-20T09:43:47.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:47 smithi122 ceph-mon[73495]: 9.6 scrub ok 2024-03-20T09:43:47.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:47 smithi122 ceph-mon[73495]: 1.d scrub starts 2024-03-20T09:43:47.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:47 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:43:47.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:47 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:43:47.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:47 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:43:47.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:47 smithi122 ceph-mon[73495]: pgmap v2746: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 903 op/s 2024-03-20T09:43:47.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:47 smithi026 ceph-mon[68309]: 1.1d scrub ok 2024-03-20T09:43:47.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:47 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-03-20T09:43:47.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:47 smithi026 ceph-mon[68309]: osdmap e1131: 12 total, 12 up, 11 in 2024-03-20T09:43:47.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:47 smithi026 ceph-mon[68309]: 9.6 scrub ok 2024-03-20T09:43:47.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:47 smithi026 ceph-mon[68309]: 1.d scrub starts 2024-03-20T09:43:47.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:47 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:43:47.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:47 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:43:47.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:47 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:43:47.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:47 smithi026 ceph-mon[68309]: pgmap v2746: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 903 op/s 2024-03-20T09:43:48.076 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:48 smithi122 ceph-mon[73495]: 1.d scrub ok 2024-03-20T09:43:48.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:48 smithi115 ceph-mon[72476]: 1.d scrub ok 2024-03-20T09:43:48.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:48 smithi026 ceph-mon[68309]: 1.d scrub ok 2024-03-20T09:43:48.589 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:43:48.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:48.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:43:48.640 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 10 1 2024-03-20T09:43:48.657 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:43:48.697 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:43:48.750 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:48.784 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:43:48.788 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:48.788 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:43:48.889 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:43:48.922 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:48.963 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:43:48.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:48.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:43:49.073 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:43:49.113 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:49.143 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:49 smithi026 ceph-mon[68309]: 1.10 scrub starts 2024-03-20T09:43:49.144 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:49 smithi026 ceph-mon[68309]: 1.10 scrub ok 2024-03-20T09:43:49.144 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:49 smithi026 ceph-mon[68309]: pgmap v2747: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 742 op/s 2024-03-20T09:43:49.225 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:43:49.225 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:43:49.225 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-03-20T09:43:49.225 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:43:49.260 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:43:49.266 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:49.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:43:49.367 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:43:49.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:49 smithi115 ceph-mon[72476]: 1.10 scrub starts 2024-03-20T09:43:49.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:49 smithi115 ceph-mon[72476]: 1.10 scrub ok 2024-03-20T09:43:49.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:49 smithi115 ceph-mon[72476]: pgmap v2747: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 742 op/s 2024-03-20T09:43:49.403 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:49.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:49 smithi122 ceph-mon[73495]: 1.10 scrub starts 2024-03-20T09:43:49.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:49 smithi122 ceph-mon[73495]: 1.10 scrub ok 2024-03-20T09:43:49.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:49 smithi122 ceph-mon[73495]: pgmap v2747: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.0 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 742 op/s 2024-03-20T09:43:49.450 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:43:49.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:49.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:43:49.568 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:43:49.624 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:49.655 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:43:49.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:49.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:43:49.761 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:43:49.806 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:49.840 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:43:49.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:49.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:43:49.945 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:43:49.981 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:43:49.997 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:50.034 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:43:50.037 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:50.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:43:50.101 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:50 smithi026 ceph-mon[68309]: 1.18 scrub starts 2024-03-20T09:43:50.101 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:50 smithi026 ceph-mon[68309]: 1.18 scrub ok 2024-03-20T09:43:50.101 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:50 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1779535589' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:43:50.139 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:43:50.186 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:50.212 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:43:50.237 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:43:50.266 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:43:50.293 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:43:50.325 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:43:50.351 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:43:50.384 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:43:50.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:50 smithi115 ceph-mon[72476]: 1.18 scrub starts 2024-03-20T09:43:50.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:50 smithi115 ceph-mon[72476]: 1.18 scrub ok 2024-03-20T09:43:50.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:50 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1779535589' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:43:50.413 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:43:50.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:50 smithi122 ceph-mon[73495]: 1.18 scrub starts 2024-03-20T09:43:50.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:50 smithi122 ceph-mon[73495]: 1.18 scrub ok 2024-03-20T09:43:50.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:50 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1779535589' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:43:50.443 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:43:50.478 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:43:50.512 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:43:50.516 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:43:50.517 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:43:50.517 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:43:50.517 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:43:50.517 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:43:50.517 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:43:50.517 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:43:50.517 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:43:50.517 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:43:50.517 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:43:50.517 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:43:50.517 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:43:50.958 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_blocked_ops 2024-03-20T09:43:51.064 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:43:51.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:51.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:43:51.171 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:43:51.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:51 smithi122 ceph-mon[73495]: 1.b scrub starts 2024-03-20T09:43:51.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:51 smithi122 ceph-mon[73495]: pgmap v2748: 47 pgs: 47 active+clean; 2.8 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 998 op/s 2024-03-20T09:43:51.223 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:51.261 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:43:51.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:51.264 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:43:51.366 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:43:51.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:51 smithi115 ceph-mon[72476]: 1.b scrub starts 2024-03-20T09:43:51.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:51 smithi115 ceph-mon[72476]: pgmap v2748: 47 pgs: 47 active+clean; 2.8 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 998 op/s 2024-03-20T09:43:51.413 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:51 smithi026 ceph-mon[68309]: 1.b scrub starts 2024-03-20T09:43:51.413 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:51 smithi026 ceph-mon[68309]: pgmap v2748: 47 pgs: 47 active+clean; 2.8 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 998 op/s 2024-03-20T09:43:51.415 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:51.428 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:43:52.123 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:43:52.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:52 smithi115 ceph-mon[72476]: 1.b scrub ok 2024-03-20T09:43:52.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:52 smithi115 ceph-mon[72476]: 1.f scrub starts 2024-03-20T09:43:52.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:52 smithi115 ceph-mon[72476]: 1.f scrub ok 2024-03-20T09:43:52.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:52 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/915838178' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:43:52.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:52 smithi122 ceph-mon[73495]: 1.b scrub ok 2024-03-20T09:43:52.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:52 smithi122 ceph-mon[73495]: 1.f scrub starts 2024-03-20T09:43:52.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:52 smithi122 ceph-mon[73495]: 1.f scrub ok 2024-03-20T09:43:52.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:52 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/915838178' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:43:52.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:52 smithi026 ceph-mon[68309]: 1.b scrub ok 2024-03-20T09:43:52.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:52 smithi026 ceph-mon[68309]: 1.f scrub starts 2024-03-20T09:43:52.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:52 smithi026 ceph-mon[68309]: 1.f scrub ok 2024-03-20T09:43:52.459 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:52 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/915838178' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:43:52.832 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:43:52.886 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:43:52.888 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:52.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:43:52.990 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:43:53.030 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:53.082 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:43:53.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:53.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:43:53.191 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:43:53.243 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:53.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:53 smithi115 ceph-mon[72476]: 1.14 scrub starts 2024-03-20T09:43:53.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:53 smithi115 ceph-mon[72476]: 1.14 scrub ok 2024-03-20T09:43:53.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:53 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:43:53.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:53 smithi115 ceph-mon[72476]: Cluster is now healthy 2024-03-20T09:43:53.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:53 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/915838178' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:43:53.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:53 smithi115 ceph-mon[72476]: osdmap e1132: 12 total, 12 up, 11 in 2024-03-20T09:43:53.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:53 smithi115 ceph-mon[72476]: pgmap v2750: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 2.7 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 972 op/s 2024-03-20T09:43:53.401 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:53 smithi122 ceph-mon[73495]: 1.14 scrub starts 2024-03-20T09:43:53.401 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:53 smithi122 ceph-mon[73495]: 1.14 scrub ok 2024-03-20T09:43:53.401 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:53 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:43:53.401 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:53 smithi122 ceph-mon[73495]: Cluster is now healthy 2024-03-20T09:43:53.401 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:53 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/915838178' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:43:53.402 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:53 smithi122 ceph-mon[73495]: osdmap e1132: 12 total, 12 up, 11 in 2024-03-20T09:43:53.402 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:53 smithi122 ceph-mon[73495]: pgmap v2750: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 2.7 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 972 op/s 2024-03-20T09:43:53.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:53 smithi026 ceph-mon[68309]: 1.14 scrub starts 2024-03-20T09:43:53.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:53 smithi026 ceph-mon[68309]: 1.14 scrub ok 2024-03-20T09:43:53.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:53 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:43:53.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:53 smithi026 ceph-mon[68309]: Cluster is now healthy 2024-03-20T09:43:53.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:53 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/915838178' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:43:53.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:53 smithi026 ceph-mon[68309]: osdmap e1132: 12 total, 12 up, 11 in 2024-03-20T09:43:53.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:53 smithi026 ceph-mon[68309]: pgmap v2750: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 2.7 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 972 op/s 2024-03-20T09:43:54.308 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:43:54.308 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:43:54.695 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:43:54.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:54.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:43:54.807 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:43:54.854 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:54.864 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:43:54.884 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:43:54.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:54.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:43:54.987 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:43:55.016 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:55.042 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:43:55.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:55.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:43:55.148 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:43:55.199 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:55.285 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_historic_ops 2024-03-20T09:43:55.331 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:43:55.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:55.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:43:55.397 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:43:55.436 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:43:55.472 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:55.668 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:43:55.726 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:43:55.734 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:43:55.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:55.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:43:55.845 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:43:55.890 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:55.922 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:43:55.931 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:55.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:43:56.031 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:55 smithi122 ceph-mon[73495]: pgmap v2751: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 2.6 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 728 op/s 2024-03-20T09:43:56.031 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:55 smithi122 ceph-mon[73495]: 1.15 scrub starts 2024-03-20T09:43:56.032 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:55 smithi122 ceph-mon[73495]: 1.15 scrub ok 2024-03-20T09:43:56.033 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:43:56.049 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:55 smithi026 ceph-mon[68309]: pgmap v2751: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 2.6 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 728 op/s 2024-03-20T09:43:56.050 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:55 smithi026 ceph-mon[68309]: 1.15 scrub starts 2024-03-20T09:43:56.050 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:55 smithi026 ceph-mon[68309]: 1.15 scrub ok 2024-03-20T09:43:56.083 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:56.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:55 smithi115 ceph-mon[72476]: pgmap v2751: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 2.6 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 728 op/s 2024-03-20T09:43:56.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:55 smithi115 ceph-mon[72476]: 1.15 scrub starts 2024-03-20T09:43:56.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:55 smithi115 ceph-mon[72476]: 1.15 scrub ok 2024-03-20T09:43:56.769 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.10 to 1 (10000) 2024-03-20T09:43:57.052 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:43:57.054 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:56 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1742103262' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-03-20T09:43:57.054 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:56 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-03-20T09:43:57.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:57.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:43:57.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:56 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1742103262' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-03-20T09:43:57.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:56 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-03-20T09:43:57.156 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:43:57.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:56 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1742103262' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-03-20T09:43:57.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:56 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-03-20T09:43:57.184 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:57.206 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:43:57.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:57.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:43:57.310 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:43:57.379 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:57.753 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:43:57.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:57.760 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:43:57.861 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:43:57.915 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:57.959 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:43:57.966 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:57.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:43:58.056 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:57 smithi122 ceph-mon[73495]: 1.1a scrub starts 2024-03-20T09:43:58.056 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:57 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:43:58.056 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:57 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:43:58.056 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:57 smithi122 ceph-mon[73495]: pgmap v2752: 47 pgs: 47 active+clean; 2.5 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 871 op/s 2024-03-20T09:43:58.056 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:57 smithi122 ceph-mon[73495]: 1.1a scrub ok 2024-03-20T09:43:58.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:57 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-03-20T09:43:58.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:57 smithi122 ceph-mon[73495]: osdmap e1133: 12 total, 12 up, 11 in 2024-03-20T09:43:58.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:57 smithi122 ceph-mon[73495]: 1.2 scrub starts 2024-03-20T09:43:58.057 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:57 smithi122 ceph-mon[73495]: 1.2 scrub ok 2024-03-20T09:43:58.067 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:43:58.101 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:57 smithi026 ceph-mon[68309]: 1.1a scrub starts 2024-03-20T09:43:58.101 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:57 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:43:58.101 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:57 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:43:58.101 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:57 smithi026 ceph-mon[68309]: pgmap v2752: 47 pgs: 47 active+clean; 2.5 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 871 op/s 2024-03-20T09:43:58.101 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:57 smithi026 ceph-mon[68309]: 1.1a scrub ok 2024-03-20T09:43:58.101 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:57 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-03-20T09:43:58.102 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:57 smithi026 ceph-mon[68309]: osdmap e1133: 12 total, 12 up, 11 in 2024-03-20T09:43:58.102 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:57 smithi026 ceph-mon[68309]: 1.2 scrub starts 2024-03-20T09:43:58.102 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:57 smithi026 ceph-mon[68309]: 1.2 scrub ok 2024-03-20T09:43:58.104 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:58.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:57 smithi115 ceph-mon[72476]: 1.1a scrub starts 2024-03-20T09:43:58.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:57 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:43:58.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:57 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:43:58.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:57 smithi115 ceph-mon[72476]: pgmap v2752: 47 pgs: 47 active+clean; 2.5 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 871 op/s 2024-03-20T09:43:58.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:57 smithi115 ceph-mon[72476]: 1.1a scrub ok 2024-03-20T09:43:58.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:57 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-03-20T09:43:58.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:57 smithi115 ceph-mon[72476]: osdmap e1133: 12 total, 12 up, 11 in 2024-03-20T09:43:58.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:57 smithi115 ceph-mon[72476]: 1.2 scrub starts 2024-03-20T09:43:58.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:57 smithi115 ceph-mon[72476]: 1.2 scrub ok 2024-03-20T09:43:58.361 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:43:58.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:58.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:43:58.396 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-20T09:43:58.396 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:43:58.428 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:43:58.472 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:43:58.493 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:43:58.493 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:43:58.531 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:58.580 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:43:58.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:58.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:43:58.687 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:43:58.743 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:59.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:59 smithi122 ceph-mon[73495]: 9.8 scrub starts 2024-03-20T09:43:59.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:59 smithi122 ceph-mon[73495]: 9.8 scrub ok 2024-03-20T09:43:59.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:43:59 smithi122 ceph-mon[73495]: 9.1 scrub starts 2024-03-20T09:43:59.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:59 smithi026 ceph-mon[68309]: 9.8 scrub starts 2024-03-20T09:43:59.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:59 smithi026 ceph-mon[68309]: 9.8 scrub ok 2024-03-20T09:43:59.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:43:59 smithi026 ceph-mon[68309]: 9.1 scrub starts 2024-03-20T09:43:59.212 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:43:59.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:59.217 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:43:59.259 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-20T09:43:59.318 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:43:59.321 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:59 smithi115 ceph-mon[72476]: 9.8 scrub starts 2024-03-20T09:43:59.321 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:59 smithi115 ceph-mon[72476]: 9.8 scrub ok 2024-03-20T09:43:59.321 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:43:59 smithi115 ceph-mon[72476]: 9.1 scrub starts 2024-03-20T09:43:59.368 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:59.407 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:43:59.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:59.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:43:59.515 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:43:59.565 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:59.609 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:43:59.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:59.612 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:43:59.691 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:43:59.713 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:43:59.749 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:43:59.860 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:43:59.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:43:59.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:43:59.967 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:43:59.998 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:00 smithi026 ceph-mon[68309]: 9.1 scrub ok 2024-03-20T09:43:59.998 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:00 smithi026 ceph-mon[68309]: pgmap v2754: 47 pgs: 47 active+clean; 2.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 695 op/s 2024-03-20T09:44:00.025 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:00.055 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:44:00.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:00.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:44:00.162 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:44:00.240 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:00.281 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:44:00.284 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:00 smithi115 ceph-mon[72476]: 9.1 scrub ok 2024-03-20T09:44:00.284 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:00 smithi115 ceph-mon[72476]: pgmap v2754: 47 pgs: 47 active+clean; 2.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 695 op/s 2024-03-20T09:44:00.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:00.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:44:00.386 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:44:00.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:00 smithi122 ceph-mon[73495]: 9.1 scrub ok 2024-03-20T09:44:00.436 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:00 smithi122 ceph-mon[73495]: pgmap v2754: 47 pgs: 47 active+clean; 2.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 695 op/s 2024-03-20T09:44:00.438 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:00.902 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:44:00.902 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:44:00.902 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:44:00.902 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:44:00.902 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:44:00.902 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:44:00.902 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:44:00.903 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:44:00.903 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:44:00.903 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:44:00.905 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:44:00.905 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:44:00.906 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:44:00.906 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:44:00.906 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:44:00.906 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:44:00.906 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:44:00.906 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:44:00.906 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:44:00.906 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:44:00.906 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:44:00.906 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:44:00.906 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:44:00.906 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:44:01.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:01 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3633699873' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:44:01.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:01 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:44:01.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:01 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:44:01.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:01 smithi115 ceph-mon[72476]: pgmap v2755: 47 pgs: 47 active+clean; 2.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 888 op/s 2024-03-20T09:44:01.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:01 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3633699873' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:44:01.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:01 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:44:01.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:01 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:44:01.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:01 smithi026 ceph-mon[68309]: pgmap v2755: 47 pgs: 47 active+clean; 2.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 888 op/s 2024-03-20T09:44:01.221 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:44:01.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:01.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:44:01.331 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:44:01.368 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:01.437 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:01 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3633699873' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:44:01.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:01 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:44:01.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:01 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:44:01.438 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:01 smithi122 ceph-mon[73495]: pgmap v2755: 47 pgs: 47 active+clean; 2.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 888 op/s 2024-03-20T09:44:01.630 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:44:01.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:01.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:44:01.698 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:44:01.735 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:44:01.787 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:01.838 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:44:01.841 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:01.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:44:01.943 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:44:02.014 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:02.044 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:44:02.045 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [ 2024-03-20T09:44:02.045 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.045 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.27190.0:230580 9.3 e1134/1066 9:c69f5d6e:::benchmark_data_smithi122_210164_object98350:head v 1134'14500, mlcod=1134'14498)", 2024-03-20T09:44:02.045 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:44:02.046910+0000", 2024-03-20T09:44:02.045 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.0072643580000000003, 2024-03-20T09:44:02.045 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.007301966, 2024-03-20T09:44:02.045 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:44:02.045 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:44:02.045 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:44:02.045 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.045 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:44:02.045 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.046910+0000", 2024-03-20T09:44:02.045 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:44:02.046 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.046 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.046 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:44:02.046 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.046910+0000", 2024-03-20T09:44:02.046 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:44:02.046 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.046 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.046 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:44:02.046 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.046911+0000", 2024-03-20T09:44:02.046 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1340000000000001e-06 2024-03-20T09:44:02.046 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.046 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.046 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:44:02.047 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.046920+0000", 2024-03-20T09:44:02.047 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5599999999999994e-06 2024-03-20T09:44:02.047 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.047 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.047 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:44:02.047 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.046931+0000", 2024-03-20T09:44:02.047 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0998999999999999e-05 2024-03-20T09:44:02.047 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.047 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.047 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:44:02.047 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.046939+0000", 2024-03-20T09:44:02.047 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5089999999999993e-06 2024-03-20T09:44:02.047 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.047 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.047 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:44:02.048 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.048400+0000", 2024-03-20T09:44:02.048 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001460353 2024-03-20T09:44:02.048 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.048 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.048 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:44:02.048 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.048421+0000", 2024-03-20T09:44:02.048 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1447e-05 2024-03-20T09:44:02.048 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:44:02.048 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:44:02.048 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:44:02.048 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.048 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.048 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.27190.0:230582 9.8 e1134/1066 9:16f46537:::benchmark_data_smithi122_210164_object98352:head v 1134'16338, mlcod=1134'16336)", 2024-03-20T09:44:02.048 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:44:02.046960+0000", 2024-03-20T09:44:02.049 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.0072142489999999998, 2024-03-20T09:44:02.049 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0073211040000000002, 2024-03-20T09:44:02.049 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:44:02.049 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:44:02.049 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:44:02.049 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.049 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:44:02.049 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.046960+0000", 2024-03-20T09:44:02.049 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:44:02.049 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.049 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.049 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:44:02.049 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.046960+0000", 2024-03-20T09:44:02.049 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:44:02.050 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.050 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.050 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:44:02.050 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.046960+0000", 2024-03-20T09:44:02.050 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6100000000000002e-07 2024-03-20T09:44:02.050 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.050 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.050 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:44:02.050 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.046964+0000", 2024-03-20T09:44:02.050 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.596e-06 2024-03-20T09:44:02.050 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.050 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.050 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:44:02.050 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.046970+0000", 2024-03-20T09:44:02.051 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4339999999999997e-06 2024-03-20T09:44:02.051 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.051 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.051 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:44:02.051 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.046975+0000", 2024-03-20T09:44:02.051 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4530000000000004e-06 2024-03-20T09:44:02.051 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.051 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.051 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:44:02.051 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.047023+0000", 2024-03-20T09:44:02.051 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8378999999999999e-05 2024-03-20T09:44:02.051 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.051 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.051 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:44:02.051 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.047042+0000", 2024-03-20T09:44:02.052 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8899999999999999e-05 2024-03-20T09:44:02.052 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:44:02.052 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:44:02.052 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:44:02.052 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.052 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.052 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:230584 9.d 9:b5f340c1:::benchmark_data_smithi122_210164_object98354:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1134)", 2024-03-20T09:44:02.052 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:44:02.046994+0000", 2024-03-20T09:44:02.052 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.0071805979999999998, 2024-03-20T09:44:02.052 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0073533219999999998, 2024-03-20T09:44:02.052 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:44:02.052 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:44:02.052 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:44:02.052 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:44:02.053 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:44:02.053 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 230584 2024-03-20T09:44:02.053 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.053 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:44:02.053 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.053 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:44:02.053 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.046994+0000", 2024-03-20T09:44:02.053 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:44:02.053 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.053 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.053 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:44:02.053 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.046994+0000", 2024-03-20T09:44:02.053 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:44:02.053 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.053 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.053 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:44:02.054 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.046995+0000", 2024-03-20T09:44:02.054 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.9699999999999994e-07 2024-03-20T09:44:02.054 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.054 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.054 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:44:02.054 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.046998+0000", 2024-03-20T09:44:02.054 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.3799999999999998e-06 2024-03-20T09:44:02.054 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.054 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.054 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:44:02.054 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.047005+0000", 2024-03-20T09:44:02.054 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3880000000000002e-06 2024-03-20T09:44:02.054 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.054 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.055 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:44:02.055 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.047011+0000", 2024-03-20T09:44:02.055 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9939999999999997e-06 2024-03-20T09:44:02.055 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.055 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.055 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:44:02.055 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.050776+0000", 2024-03-20T09:44:02.055 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.003764238 2024-03-20T09:44:02.055 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.055 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.055 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:44:02.055 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.050892+0000", 2024-03-20T09:44:02.055 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000116768 2024-03-20T09:44:02.055 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.055 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.055 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,7", 2024-03-20T09:44:02.055 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.051006+0000", 2024-03-20T09:44:02.056 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00011356 2024-03-20T09:44:02.056 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:44:02.056 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:44:02.056 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:44:02.056 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.056 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.056 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:230585 9.f 9:fa0ae128:::benchmark_data_smithi122_210164_object98355:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1134)", 2024-03-20T09:44:02.056 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:44:02.047019+0000", 2024-03-20T09:44:02.056 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.0071548610000000002, 2024-03-20T09:44:02.056 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.007405175, 2024-03-20T09:44:02.056 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:44:02.056 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:44:02.056 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:44:02.056 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:44:02.056 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:44:02.057 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 230585 2024-03-20T09:44:02.057 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.057 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:44:02.057 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.057 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:44:02.057 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.047019+0000", 2024-03-20T09:44:02.057 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:44:02.057 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.057 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.057 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:44:02.057 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.047019+0000", 2024-03-20T09:44:02.057 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:44:02.057 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.057 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.057 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:44:02.057 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.047020+0000", 2024-03-20T09:44:02.058 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.5900000000000003e-07 2024-03-20T09:44:02.058 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.058 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.058 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:44:02.058 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.047022+0000", 2024-03-20T09:44:02.058 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2520000000000002e-06 2024-03-20T09:44:02.058 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.058 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.058 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:44:02.058 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.047027+0000", 2024-03-20T09:44:02.058 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1719999999999998e-06 2024-03-20T09:44:02.058 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.058 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.058 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:44:02.059 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.047032+0000", 2024-03-20T09:44:02.059 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5680000000000001e-06 2024-03-20T09:44:02.059 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.059 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.059 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:44:02.059 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.047149+0000", 2024-03-20T09:44:02.059 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000117291 2024-03-20T09:44:02.059 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.059 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.059 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:44:02.059 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.047263+0000", 2024-03-20T09:44:02.059 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00011439500000000001 2024-03-20T09:44:02.059 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.059 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.060 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,9", 2024-03-20T09:44:02.060 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.047378+0000", 2024-03-20T09:44:02.060 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000114486 2024-03-20T09:44:02.060 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:44:02.060 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:44:02.060 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:44:02.060 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.060 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.060 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.27190.0:230583 9.8 e1134/1066 9:17ae21a8:::benchmark_data_smithi122_210164_object98353:head v 1134'16339, mlcod=1134'16336)", 2024-03-20T09:44:02.060 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:44:02.047105+0000", 2024-03-20T09:44:02.060 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.0070695899999999997, 2024-03-20T09:44:02.060 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0073944650000000002, 2024-03-20T09:44:02.060 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:44:02.060 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:44:02.060 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:44:02.061 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.061 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:44:02.061 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.047105+0000", 2024-03-20T09:44:02.061 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:44:02.061 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.061 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.061 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:44:02.061 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.047105+0000", 2024-03-20T09:44:02.061 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:44:02.061 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.061 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.061 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:44:02.061 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.047105+0000", 2024-03-20T09:44:02.061 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.2899999999999999e-07 2024-03-20T09:44:02.061 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.062 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.062 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:44:02.062 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.047109+0000", 2024-03-20T09:44:02.062 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.839e-06 2024-03-20T09:44:02.062 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.062 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.062 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:44:02.062 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.052736+0000", 2024-03-20T09:44:02.062 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.005627239 2024-03-20T09:44:02.062 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.062 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.062 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:44:02.062 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.052747+0000", 2024-03-20T09:44:02.062 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1171e-05 2024-03-20T09:44:02.063 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.063 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.063 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:44:02.063 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.052801+0000", 2024-03-20T09:44:02.063 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3567999999999999e-05 2024-03-20T09:44:02.063 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.063 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.063 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:44:02.063 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.052822+0000", 2024-03-20T09:44:02.063 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0747e-05 2024-03-20T09:44:02.063 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:44:02.063 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:44:02.063 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:44:02.063 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.064 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.064 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop_reply(client.27190.0:230568 9.d e1134/1066)", 2024-03-20T09:44:02.064 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:44:02.052849+0000", 2024-03-20T09:44:02.064 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.001325269, 2024-03-20T09:44:02.064 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0017059029999999999, 2024-03-20T09:44:02.064 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:44:02.064 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "queued for pg", 2024-03-20T09:44:02.064 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:44:02.064 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.064 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:44:02.064 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.052849+0000", 2024-03-20T09:44:02.064 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:44:02.064 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.064 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.065 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:44:02.065 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.052849+0000", 2024-03-20T09:44:02.065 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:44:02.065 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.065 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.065 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:44:02.065 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.052850+0000", 2024-03-20T09:44:02.065 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.0999999999999997e-07 2024-03-20T09:44:02.065 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.065 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.065 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:44:02.065 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.052854+0000", 2024-03-20T09:44:02.065 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7129999999999999e-06 2024-03-20T09:44:02.065 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.065 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.065 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:44:02.066 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.052863+0000", 2024-03-20T09:44:02.066 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.6220000000000004e-06 2024-03-20T09:44:02.066 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.066 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.066 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:44:02.066 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.052869+0000", 2024-03-20T09:44:02.066 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7030000000000003e-06 2024-03-20T09:44:02.066 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:44:02.066 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:44:02.066 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:44:02.066 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.066 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.066 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop_reply(client.27190.0:230571 9.d e1134/1066)", 2024-03-20T09:44:02.066 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:44:02.052878+0000", 2024-03-20T09:44:02.066 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.001296724, 2024-03-20T09:44:02.067 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0017229039999999999, 2024-03-20T09:44:02.067 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:44:02.067 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "queued for pg", 2024-03-20T09:44:02.067 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:44:02.067 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.067 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:44:02.067 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.052878+0000", 2024-03-20T09:44:02.067 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:44:02.067 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.067 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.067 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:44:02.067 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.052878+0000", 2024-03-20T09:44:02.067 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:44:02.067 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.067 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.067 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:44:02.068 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.052878+0000", 2024-03-20T09:44:02.068 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.4400000000000001e-07 2024-03-20T09:44:02.068 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.068 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.068 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:44:02.068 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.052880+0000", 2024-03-20T09:44:02.068 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.894e-06 2024-03-20T09:44:02.068 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.068 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.068 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:44:02.068 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.052885+0000", 2024-03-20T09:44:02.068 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2800000000000003e-06 2024-03-20T09:44:02.068 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.068 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.068 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:44:02.068 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.052889+0000", 2024-03-20T09:44:02.069 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.929e-06 2024-03-20T09:44:02.069 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:44:02.069 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:44:02.069 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:44:02.069 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.069 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.069 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop_reply(client.27190.0:230581 9.6 e1134/1066)", 2024-03-20T09:44:02.069 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:44:02.054225+0000", 2024-03-20T09:44:02.069 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 4294967295.999949, 2024-03-20T09:44:02.069 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00042156000000000002, 2024-03-20T09:44:02.069 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:44:02.069 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "queued for pg", 2024-03-20T09:44:02.069 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:44:02.069 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.069 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:44:02.069 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.054225+0000", 2024-03-20T09:44:02.070 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:44:02.070 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.070 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.070 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:44:02.070 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.054225+0000", 2024-03-20T09:44:02.070 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:44:02.070 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.070 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.070 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:44:02.070 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.054226+0000", 2024-03-20T09:44:02.070 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.4900000000000005e-07 2024-03-20T09:44:02.070 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.070 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.070 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:44:02.071 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.054233+0000", 2024-03-20T09:44:02.071 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.3400000000000003e-06 2024-03-20T09:44:02.071 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.071 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.071 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:44:02.071 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.054245+0000", 2024-03-20T09:44:02.071 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2099000000000001e-05 2024-03-20T09:44:02.071 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.071 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.071 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:44:02.071 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.054252+0000", 2024-03-20T09:44:02.071 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.6190000000000002e-06 2024-03-20T09:44:02.071 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:44:02.071 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:44:02.072 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:44:02.072 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.072 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.072 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.27190.0:230565 9.5 e1134/1066 9:acdf7102:::benchmark_data_smithi122_210164_object98335:head v 1134'14644, mlcod=1134'14640)", 2024-03-20T09:44:02.072 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:44:02.016321+0000", 2024-03-20T09:44:02.072 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.037853462999999997, 2024-03-20T09:44:02.072 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.038378045999999999, 2024-03-20T09:44:02.072 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:44:02.072 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:44:02.072 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:44:02.072 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.072 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:44:02.072 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.016321+0000", 2024-03-20T09:44:02.072 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:44:02.073 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.073 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.073 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:44:02.073 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.016321+0000", 2024-03-20T09:44:02.073 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:44:02.073 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.073 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.073 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:44:02.073 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.016322+0000", 2024-03-20T09:44:02.073 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3090000000000001e-06 2024-03-20T09:44:02.073 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.073 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.073 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:44:02.073 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.016332+0000", 2024-03-20T09:44:02.073 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.4970000000000007e-06 2024-03-20T09:44:02.074 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.074 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.074 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:44:02.074 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.016340+0000", 2024-03-20T09:44:02.074 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.422e-06 2024-03-20T09:44:02.074 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.074 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.074 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:44:02.074 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.016351+0000", 2024-03-20T09:44:02.074 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0577000000000001e-05 2024-03-20T09:44:02.074 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.074 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.074 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:44:02.074 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.030274+0000", 2024-03-20T09:44:02.074 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.013923547 2024-03-20T09:44:02.075 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.075 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.075 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:44:02.075 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.030300+0000", 2024-03-20T09:44:02.075 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5952000000000002e-05 2024-03-20T09:44:02.075 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:44:02.075 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:44:02.075 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:44:02.075 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.075 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.075 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:230568 9.d 9:b3ea4c07:::benchmark_data_smithi122_210164_object98338:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1134)", 2024-03-20T09:44:02.075 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:44:02.017722+0000", 2024-03-20T09:44:02.075 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.036452054999999997, 2024-03-20T09:44:02.075 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.037038274000000003, 2024-03-20T09:44:02.075 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:44:02.076 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:44:02.076 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:44:02.076 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:44:02.076 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:44:02.076 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 230568 2024-03-20T09:44:02.076 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.076 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:44:02.076 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.076 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:44:02.076 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.017722+0000", 2024-03-20T09:44:02.076 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:44:02.076 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.076 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.076 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:44:02.076 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.017722+0000", 2024-03-20T09:44:02.077 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:44:02.077 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.077 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.077 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:44:02.077 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.017724+0000", 2024-03-20T09:44:02.077 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.235e-06 2024-03-20T09:44:02.077 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.077 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.077 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:44:02.077 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.017728+0000", 2024-03-20T09:44:02.077 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5589999999999999e-06 2024-03-20T09:44:02.077 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.077 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.077 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:44:02.077 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.017742+0000", 2024-03-20T09:44:02.077 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3956e-05 2024-03-20T09:44:02.078 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.078 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.078 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:44:02.078 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.017750+0000", 2024-03-20T09:44:02.078 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.5379999999999997e-06 2024-03-20T09:44:02.078 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.078 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.078 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:44:02.078 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.034725+0000", 2024-03-20T09:44:02.078 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.016975724000000001 2024-03-20T09:44:02.078 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.078 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.078 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:44:02.078 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.034821+0000", 2024-03-20T09:44:02.078 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.6040999999999997e-05 2024-03-20T09:44:02.079 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.079 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.079 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,7", 2024-03-20T09:44:02.079 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.034905+0000", 2024-03-20T09:44:02.079 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.3237000000000003e-05 2024-03-20T09:44:02.079 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:44:02.079 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:44:02.079 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:44:02.079 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.079 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.079 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:230571 9.d 9:b6966177:::benchmark_data_smithi122_210164_object98341:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1134)", 2024-03-20T09:44:02.079 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:44:02.021119+0000", 2024-03-20T09:44:02.079 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.033055515000000001, 2024-03-20T09:44:02.079 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.033717245999999999, 2024-03-20T09:44:02.079 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:44:02.080 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:44:02.080 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:44:02.080 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:44:02.080 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:44:02.080 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 230571 2024-03-20T09:44:02.080 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.080 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:44:02.080 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.080 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:44:02.080 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.021119+0000", 2024-03-20T09:44:02.080 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:44:02.080 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.080 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.080 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:44:02.081 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.021119+0000", 2024-03-20T09:44:02.081 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:44:02.081 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.081 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.081 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:44:02.081 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.021120+0000", 2024-03-20T09:44:02.081 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0139999999999999e-06 2024-03-20T09:44:02.081 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.081 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.081 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:44:02.081 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.021124+0000", 2024-03-20T09:44:02.081 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.454e-06 2024-03-20T09:44:02.081 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.081 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.081 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:44:02.081 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.021144+0000", 2024-03-20T09:44:02.082 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0199000000000001e-05 2024-03-20T09:44:02.082 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.082 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.082 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:44:02.082 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.021152+0000", 2024-03-20T09:44:02.082 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7100000000000007e-06 2024-03-20T09:44:02.082 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.082 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.082 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:44:02.082 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.039233+0000", 2024-03-20T09:44:02.082 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.018080888 2024-03-20T09:44:02.082 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.082 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.083 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:44:02.083 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.039345+0000", 2024-03-20T09:44:02.083 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000112143 2024-03-20T09:44:02.083 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.083 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.083 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,7", 2024-03-20T09:44:02.083 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.039444+0000", 2024-03-20T09:44:02.083 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.8311000000000001e-05 2024-03-20T09:44:02.083 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:44:02.083 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:44:02.083 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:44:02.083 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.083 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.083 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:230577 9.d 9:b89559bf:::benchmark_data_smithi122_210164_object98347:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1134)", 2024-03-20T09:44:02.084 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:44:02.033926+0000", 2024-03-20T09:44:02.084 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.020248439999999999, 2024-03-20T09:44:02.084 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.020984582000000002, 2024-03-20T09:44:02.084 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:44:02.084 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:44:02.084 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:44:02.084 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:44:02.084 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:44:02.084 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 230577 2024-03-20T09:44:02.084 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.084 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:44:02.084 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.084 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:44:02.084 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.033926+0000", 2024-03-20T09:44:02.084 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:44:02.085 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.085 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.085 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:44:02.085 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.033926+0000", 2024-03-20T09:44:02.085 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:44:02.085 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.085 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.085 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:44:02.085 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.033927+0000", 2024-03-20T09:44:02.085 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.468e-06 2024-03-20T09:44:02.085 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.085 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.085 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:44:02.085 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.033933+0000", 2024-03-20T09:44:02.085 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4589999999999997e-06 2024-03-20T09:44:02.086 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.086 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.086 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:44:02.086 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.033945+0000", 2024-03-20T09:44:02.086 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2611e-05 2024-03-20T09:44:02.086 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.086 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.086 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:44:02.086 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.033962+0000", 2024-03-20T09:44:02.086 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6750000000000001e-05 2024-03-20T09:44:02.086 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.086 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.086 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:44:02.087 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.045512+0000", 2024-03-20T09:44:02.087 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.011549524 2024-03-20T09:44:02.087 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.087 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.087 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:44:02.087 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.045636+0000", 2024-03-20T09:44:02.087 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012435899999999999 2024-03-20T09:44:02.087 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.087 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.087 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,7", 2024-03-20T09:44:02.087 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.045759+0000", 2024-03-20T09:44:02.087 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000122798 2024-03-20T09:44:02.087 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:44:02.087 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:44:02.087 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:44:02.087 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.088 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.088 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:230581 9.6 9:6724552b:::benchmark_data_smithi122_210164_object98351:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1134)", 2024-03-20T09:44:02.088 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:44:02.041729+0000", 2024-03-20T09:44:02.088 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 0.012445342, 2024-03-20T09:44:02.088 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.013255551000000001, 2024-03-20T09:44:02.088 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:44:02.088 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "waiting for sub ops", 2024-03-20T09:44:02.088 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:44:02.088 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:44:02.088 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:44:02.088 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 230581 2024-03-20T09:44:02.088 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.088 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:44:02.088 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.089 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:44:02.089 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.041729+0000", 2024-03-20T09:44:02.089 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:44:02.089 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.089 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.089 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:44:02.089 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.041729+0000", 2024-03-20T09:44:02.089 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:44:02.089 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.089 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.089 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:44:02.089 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.041730+0000", 2024-03-20T09:44:02.089 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.294e-06 2024-03-20T09:44:02.089 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.089 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.089 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:44:02.090 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.041735+0000", 2024-03-20T09:44:02.090 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.7600000000000002e-06 2024-03-20T09:44:02.090 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.090 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.090 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:44:02.090 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.041746+0000", 2024-03-20T09:44:02.090 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1205e-05 2024-03-20T09:44:02.090 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.090 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.090 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:44:02.090 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.041754+0000", 2024-03-20T09:44:02.090 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.9009999999999994e-06 2024-03-20T09:44:02.090 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.090 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.090 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:44:02.091 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.041809+0000", 2024-03-20T09:44:02.091 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4638000000000001e-05 2024-03-20T09:44:02.091 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.091 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.091 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:44:02.091 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.041931+0000", 2024-03-20T09:44:02.091 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012269700000000001 2024-03-20T09:44:02.091 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.091 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.091 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,10", 2024-03-20T09:44:02.091 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.042048+0000", 2024-03-20T09:44:02.091 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00011691100000000001 2024-03-20T09:44:02.091 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:44:02.091 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:44:02.092 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:44:02.092 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:02.053008+0000", 2024-03-20T09:44:02.092 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.010959986999999999 2024-03-20T09:44:02.092 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:44:02.092 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:44:02.092 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:44:02.092 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:44:02.092 INFO:teuthology.orchestra.run.smithi115.stdout: ], 2024-03-20T09:44:02.092 INFO:teuthology.orchestra.run.smithi115.stdout: "num_ops": 13 2024-03-20T09:44:02.092 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:44:02.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:02.093 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:44:02.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:44:02.194 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:44:02.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:02 smithi026 ceph-mon[68309]: 1.1b scrub starts 2024-03-20T09:44:02.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:02 smithi026 ceph-mon[68309]: 1.1b scrub ok 2024-03-20T09:44:02.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:02 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:44:02.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:02 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:44:02.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:02 smithi026 ceph-mon[68309]: osdmap e1134: 12 total, 12 up, 11 in 2024-03-20T09:44:02.244 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:02.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:02 smithi115 ceph-mon[72476]: 1.1b scrub starts 2024-03-20T09:44:02.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:02 smithi115 ceph-mon[72476]: 1.1b scrub ok 2024-03-20T09:44:02.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:02 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:44:02.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:02 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:44:02.395 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:02 smithi115 ceph-mon[72476]: osdmap e1134: 12 total, 12 up, 11 in 2024-03-20T09:44:02.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:02 smithi122 ceph-mon[73495]: 1.1b scrub starts 2024-03-20T09:44:02.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:02 smithi122 ceph-mon[73495]: 1.1b scrub ok 2024-03-20T09:44:02.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:02 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:44:02.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:02 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:44:02.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:02 smithi122 ceph-mon[73495]: osdmap e1134: 12 total, 12 up, 11 in 2024-03-20T09:44:02.851 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-20T09:44:03.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:03 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:44:03.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:03 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:44:03.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:03 smithi115 ceph-mon[72476]: pgmap v2757: 47 pgs: 47 active+clean; 2.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 918 op/s 2024-03-20T09:44:03.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:03 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3138333472' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:44:03.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:03 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:44:03.279 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:44:03.416 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:44:03.419 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:03 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:44:03.419 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:03 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:44:03.419 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:03 smithi026 ceph-mon[68309]: pgmap v2757: 47 pgs: 47 active+clean; 2.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 918 op/s 2024-03-20T09:44:03.419 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:03 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3138333472' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:44:03.419 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:03 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:44:03.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:03.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:44:03.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:03 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:44:03.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:03 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:44:03.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:03 smithi122 ceph-mon[73495]: pgmap v2757: 47 pgs: 47 active+clean; 2.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 918 op/s 2024-03-20T09:44:03.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:03 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3138333472' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:44:03.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:03 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:44:03.524 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:44:03.582 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:03.604 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:44:03.609 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:44:03.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:03.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:44:03.663 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:44:03.714 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:44:03.774 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:04.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:04 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:44:04.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:04 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:44:04.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:04 smithi115 ceph-mon[72476]: osdmap e1135: 12 total, 12 up, 11 in 2024-03-20T09:44:04.302 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:04 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:44:04.302 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:04 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:44:04.302 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:04 smithi026 ceph-mon[68309]: osdmap e1135: 12 total, 12 up, 11 in 2024-03-20T09:44:04.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:04 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:44:04.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:04 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:44:04.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:04 smithi122 ceph-mon[73495]: osdmap e1135: 12 total, 12 up, 11 in 2024-03-20T09:44:04.926 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:44:05.213 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:44:05.215 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:44:05.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:05 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:44:05.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:05 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:44:05.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:05 smithi115 ceph-mon[72476]: pgmap v2759: 47 pgs: 47 active+clean; 2.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 758 op/s 2024-03-20T09:44:05.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:05 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:44:05.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:05 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:44:05.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:05 smithi122 ceph-mon[73495]: pgmap v2759: 47 pgs: 47 active+clean; 2.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 758 op/s 2024-03-20T09:44:05.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:05 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:44:05.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:05 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:44:05.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:05 smithi026 ceph-mon[68309]: pgmap v2759: 47 pgs: 47 active+clean; 2.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 758 op/s 2024-03-20T09:44:05.660 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:44:05.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:05.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:44:05.769 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:44:05.809 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:06.294 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:06 smithi115 ceph-mon[72476]: osdmap e1136: 12 total, 12 up, 11 in 2024-03-20T09:44:06.294 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:06 smithi115 ceph-mon[72476]: from='client.17589 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:44:06.314 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:06 smithi026 ceph-mon[68309]: osdmap e1136: 12 total, 12 up, 11 in 2024-03-20T09:44:06.314 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:06 smithi026 ceph-mon[68309]: from='client.17589 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:44:06.344 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-20T09:44:06.386 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:44:06.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:06.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:44:06.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:06 smithi122 ceph-mon[73495]: osdmap e1136: 12 total, 12 up, 11 in 2024-03-20T09:44:06.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:06 smithi122 ceph-mon[73495]: from='client.17589 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:44:06.496 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:44:06.552 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:06.633 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:44:06.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:06.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:44:06.738 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:44:06.777 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:06.822 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:44:06.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:06.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:44:06.840 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":2760,"stamp":"2024-03-20T09:44:05.044246+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2238251576,"num_objects":34147,"num_object_clones":0,"num_object_copies":102441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34147,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":230311,"num_write_kb":8462529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":165303,"ondisk_log_size":165303,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":16112404,"kb_used_data":6558000,"kb_used_omap":299,"kb_used_meta":9553940,"kb_avail":1015084268,"statfs":{"total":1055945392128,"available":1039446290432,"internally_reserved":0,"allocated":6715392000,"data_stored":6712359138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":306651,"internal_metadata":9783235109},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-376307712,"num_objects":-5742,"num_object_clones":0,"num_object_copies":-17226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5742,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.341064"},"pg_stats":[{"pgid":"1.1e","version":"20'137","reported_seq":1759,"reported_epoch":1134,"state":"active+clean","last_fresh":"2024-03-20T09:44:01.023933+0000","last_change":"2024-03-20T09:43:09.164860+0000","last_active":"2024-03-20T09:44:01.023933+0000","last_peered":"2024-03-20T09:44:01.023933+0000","last_clean":"2024-03-20T09:44:01.023933+0000","last_became_active":"2024-03-20T09:13:20.071655+0000","last_became_peered":"2024-03-20T09:13:20.071655+0000","last_unstale":"2024-03-20T09:44:01.023933+0000","last_undegraded":"2024-03-20T09:44:01.023933+0000","last_fullsized":"2024-03-20T09:44:01.023933+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:09.164738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:09.164738+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:33.937151+0000","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'137","reported_seq":844,"reported_epoch":1134,"state":"active+clean","last_fresh":"2024-03-20T09:44:01.024051+0000","last_change":"2024-03-20T09:43:45.421863+0000","last_active":"2024-03-20T09:44:01.024051+0000","last_peered":"2024-03-20T09:44:01.024051+0000","last_clean":"2024-03-20T09:44:01.024051+0000","last_became_active":"2024-03-20T09:31:07.425574+0000","last_became_peered":"2024-03-20T09:31:07.425574+0000","last_unstale":"2024-03-20T09:44:01.024051+0000","last_undegraded":"2024-03-20T09:44:01.024051+0000","last_fullsized":"2024-03-20T09:44:01.024051+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:45.421742+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:29:02.585165+0000","last_clean_scrub_stamp":"2024-03-20T09:43:45.421742+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:55.396984+0000","scrub_duration":262,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2],"acting":[11,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"20'137","reported_seq":2448,"reported_epoch":1134,"state":"active+clean","last_fresh":"2024-03-20T09:44:01.025796+0000","last_change":"2024-03-20T09:43:11.053490+0000","last_active":"2024-03-20T09:44:01.025796+0000","last_peered":"2024-03-20T09:44:01.025796+0000","last_clean":"2024-03-20T09:44:01.025796+0000","last_became_active":"2024-03-20T09:35:03.815532+0000","last_became_peered":"2024-03-20T09:35:03.815532+0000","last_unstale":"2024-03-20T09:44:01.025796+0000","last_undegraded":"2024-03-20T09:44:01.025796+0000","last_fullsized":"2024-03-20T09:44:01.025796+0000","mapping_epoch":950,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":951,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:11.053365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:11.053365+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:21.391918+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.19","version":"20'137","reported_seq":1634,"reported_epoch":1135,"state":"active+clean","last_fresh":"2024-03-20T09:44:03.040936+0000","last_change":"2024-03-20T09:43:44.857190+0000","last_active":"2024-03-20T09:44:03.040936+0000","last_peered":"2024-03-20T09:44:03.040936+0000","last_clean":"2024-03-20T09:44:03.040936+0000","last_became_active":"2024-03-20T09:39:17.953184+0000","last_became_peered":"2024-03-20T09:39:17.953184+0000","last_unstale":"2024-03-20T09:44:03.040936+0000","last_undegraded":"2024-03-20T09:44:03.040936+0000","last_fullsized":"2024-03-20T09:44:03.040936+0000","mapping_epoch":1041,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":1042,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:44.857069+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:30:56.201554+0000","last_clean_scrub_stamp":"2024-03-20T09:43:44.857069+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:06.289387+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,7],"acting":[10,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"20'137","reported_seq":1911,"reported_epoch":1133,"state":"active+clean","last_fresh":"2024-03-20T09:43:56.783408+0000","last_change":"2024-03-20T09:43:49.190805+0000","last_active":"2024-03-20T09:43:56.783408+0000","last_peered":"2024-03-20T09:43:56.783408+0000","last_clean":"2024-03-20T09:43:56.783408+0000","last_became_active":"2024-03-20T09:10:35.277481+0000","last_became_peered":"2024-03-20T09:10:35.277481+0000","last_unstale":"2024-03-20T09:43:56.783408+0000","last_undegraded":"2024-03-20T09:43:56.783408+0000","last_fullsized":"2024-03-20T09:43:56.783408+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:49.190689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:49.190689+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:03.907117+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1b","version":"20'137","reported_seq":2285,"reported_epoch":1134,"state":"active+clean","last_fresh":"2024-03-20T09:44:01.026022+0000","last_change":"2024-03-20T09:44:00.886678+0000","last_active":"2024-03-20T09:44:01.026022+0000","last_peered":"2024-03-20T09:44:01.026022+0000","last_clean":"2024-03-20T09:44:01.026022+0000","last_became_active":"2024-03-20T09:39:17.951408+0000","last_became_peered":"2024-03-20T09:39:17.951408+0000","last_unstale":"2024-03-20T09:44:01.026022+0000","last_undegraded":"2024-03-20T09:44:01.026022+0000","last_fullsized":"2024-03-20T09:44:01.026022+0000","mapping_epoch":1041,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":1042,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:44:00.886543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:44:00.886543+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:05.967822+0000","scrub_duration":206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,2],"acting":[8,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.0","version":"1133'14696","reported_seq":14979,"reported_epoch":1133,"state":"active+clean","last_fresh":"2024-03-20T09:44:00.939450+0000","last_change":"2024-03-20T09:43:12.972823+0000","last_active":"2024-03-20T09:44:00.939450+0000","last_peered":"2024-03-20T09:44:00.939450+0000","last_clean":"2024-03-20T09:44:00.939450+0000","last_became_active":"2024-03-20T09:40:24.790001+0000","last_became_peered":"2024-03-20T09:40:24.790001+0000","last_unstale":"2024-03-20T09:44:00.939450+0000","last_undegraded":"2024-03-20T09:44:00.939450+0000","last_fullsized":"2024-03-20T09:44:00.939450+0000","mapping_epoch":1066,"log_start":"1081'4600","ondisk_log_start":"1081'4600","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1120'12079","last_scrub_stamp":"2024-03-20T09:43:12.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:12.972570+0000","objects_scrubbed":4832,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:35.233515+0000","scrub_duration":1243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143130624,"num_objects":2184,"num_object_clones":0,"num_object_copies":6552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14696,"num_write_kb":540160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,11],"acting":[6,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"20'137","reported_seq":1041,"reported_epoch":1135,"state":"active+clean","last_fresh":"2024-03-20T09:44:03.125692+0000","last_change":"2024-03-20T09:43:17.393179+0000","last_active":"2024-03-20T09:44:03.125692+0000","last_peered":"2024-03-20T09:44:03.125692+0000","last_clean":"2024-03-20T09:44:03.125692+0000","last_became_active":"2024-03-20T09:35:28.000367+0000","last_became_peered":"2024-03-20T09:35:28.000367+0000","last_unstale":"2024-03-20T09:44:03.125692+0000","last_undegraded":"2024-03-20T09:44:03.125692+0000","last_fullsized":"2024-03-20T09:44:03.125692+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":966,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:17.393079+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:37:01.184206+0000","last_clean_scrub_stamp":"2024-03-20T09:43:17.393079+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:36.547547+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"1133'16571","reported_seq":16955,"reported_epoch":1133,"state":"active+clean","last_fresh":"2024-03-20T09:44:00.153092+0000","last_change":"2024-03-20T09:43:58.187624+0000","last_active":"2024-03-20T09:44:00.153092+0000","last_peered":"2024-03-20T09:44:00.153092+0000","last_clean":"2024-03-20T09:44:00.153092+0000","last_became_active":"2024-03-20T09:40:24.791005+0000","last_became_peered":"2024-03-20T09:40:24.791005+0000","last_unstale":"2024-03-20T09:44:00.153092+0000","last_undegraded":"2024-03-20T09:44:00.153092+0000","last_fullsized":"2024-03-20T09:44:00.153092+0000","mapping_epoch":1066,"log_start":"1088'6500","ondisk_log_start":"1088'6500","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1133'16468","last_scrub_stamp":"2024-03-20T09:43:58.187354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:58.187354+0000","objects_scrubbed":2257,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:00.972878+0000","scrub_duration":557,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140378112,"num_objects":2142,"num_object_clones":0,"num_object_copies":6426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14112,"num_write_kb":520128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"20'137","reported_seq":2500,"reported_epoch":1135,"state":"active+clean","last_fresh":"2024-03-20T09:44:03.050377+0000","last_change":"2024-03-20T09:43:13.063995+0000","last_active":"2024-03-20T09:44:03.050377+0000","last_peered":"2024-03-20T09:44:03.050377+0000","last_clean":"2024-03-20T09:44:03.050377+0000","last_became_active":"2024-03-20T09:20:25.341750+0000","last_became_peered":"2024-03-20T09:20:25.341750+0000","last_unstale":"2024-03-20T09:44:03.050377+0000","last_undegraded":"2024-03-20T09:44:03.050377+0000","last_fullsized":"2024-03-20T09:44:03.050377+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:13.063918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:13.063918+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:22.184587+0000","scrub_duration":184,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,2],"acting":[9,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.2","version":"1133'14395","reported_seq":14681,"reported_epoch":1133,"state":"active+clean","last_fresh":"2024-03-20T09:44:00.167553+0000","last_change":"2024-03-20T09:43:09.666385+0000","last_active":"2024-03-20T09:44:00.167553+0000","last_peered":"2024-03-20T09:44:00.167553+0000","last_clean":"2024-03-20T09:44:00.167553+0000","last_became_active":"2024-03-20T09:40:24.790265+0000","last_became_peered":"2024-03-20T09:40:24.790265+0000","last_unstale":"2024-03-20T09:44:00.167553+0000","last_undegraded":"2024-03-20T09:44:00.167553+0000","last_fullsized":"2024-03-20T09:44:00.167553+0000","mapping_epoch":1066,"log_start":"1081'4300","ondisk_log_start":"1081'4300","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'11638","last_scrub_stamp":"2024-03-20T09:43:09.666260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:09.666260+0000","objects_scrubbed":4972,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:35.499687+0000","scrub_duration":1058,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143851520,"num_objects":2195,"num_object_clones":0,"num_object_copies":6585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14395,"num_write_kb":530880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,9],"acting":[3,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"20'137","reported_seq":1182,"reported_epoch":1135,"state":"active+clean","last_fresh":"2024-03-20T09:44:03.049065+0000","last_change":"2024-03-20T09:43:04.893745+0000","last_active":"2024-03-20T09:44:03.049065+0000","last_peered":"2024-03-20T09:44:03.049065+0000","last_clean":"2024-03-20T09:44:03.049065+0000","last_became_active":"2024-03-20T09:24:16.444232+0000","last_became_peered":"2024-03-20T09:24:16.444232+0000","last_unstale":"2024-03-20T09:44:03.049065+0000","last_undegraded":"2024-03-20T09:44:03.049065+0000","last_fullsized":"2024-03-20T09:44:03.049065+0000","mapping_epoch":718,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:04.893646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:04.893646+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:08.552904+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.c","version":"1133'14020","reported_seq":14301,"reported_epoch":1133,"state":"active+clean","last_fresh":"2024-03-20T09:44:00.148045+0000","last_change":"2024-03-20T09:43:06.941039+0000","last_active":"2024-03-20T09:44:00.148045+0000","last_peered":"2024-03-20T09:44:00.148045+0000","last_clean":"2024-03-20T09:44:00.148045+0000","last_became_active":"2024-03-20T09:40:24.791734+0000","last_became_peered":"2024-03-20T09:40:24.791734+0000","last_unstale":"2024-03-20T09:44:00.148045+0000","last_undegraded":"2024-03-20T09:44:00.148045+0000","last_fullsized":"2024-03-20T09:44:00.148045+0000","mapping_epoch":1066,"log_start":"1079'4000","ondisk_log_start":"1079'4000","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1117'11282","last_scrub_stamp":"2024-03-20T09:43:06.940719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:06.940719+0000","objects_scrubbed":4972,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:33.258717+0000","scrub_duration":1275,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144179200,"num_objects":2200,"num_object_clones":0,"num_object_copies":6600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14020,"num_write_kb":519040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,9],"acting":[3,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"20'137","reported_seq":2427,"reported_epoch":1134,"state":"active+clean","last_fresh":"2024-03-20T09:44:01.023620+0000","last_change":"2024-03-20T09:43:18.087277+0000","last_active":"2024-03-20T09:44:01.023620+0000","last_peered":"2024-03-20T09:44:01.023620+0000","last_clean":"2024-03-20T09:44:01.023620+0000","last_became_active":"2024-03-20T09:12:14.306384+0000","last_became_peered":"2024-03-20T09:12:14.306384+0000","last_unstale":"2024-03-20T09:44:01.023620+0000","last_undegraded":"2024-03-20T09:44:01.023620+0000","last_fullsized":"2024-03-20T09:44:01.023620+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:18.087165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:18.087165+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:46.885376+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.d","version":"1133'14241","reported_seq":14526,"reported_epoch":1133,"state":"active+clean","last_fresh":"2024-03-20T09:44:00.392479+0000","last_change":"2024-03-20T09:43:18.652174+0000","last_active":"2024-03-20T09:44:00.392479+0000","last_peered":"2024-03-20T09:44:00.392479+0000","last_clean":"2024-03-20T09:44:00.392479+0000","last_became_active":"2024-03-20T09:40:24.812843+0000","last_became_peered":"2024-03-20T09:40:24.812843+0000","last_unstale":"2024-03-20T09:44:00.392479+0000","last_undegraded":"2024-03-20T09:44:00.392479+0000","last_fullsized":"2024-03-20T09:44:00.392479+0000","mapping_epoch":1066,"log_start":"1081'4200","ondisk_log_start":"1081'4200","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1122'12042","last_scrub_stamp":"2024-03-20T09:43:18.651987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:18.651987+0000","objects_scrubbed":4624,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:48.073907+0000","scrub_duration":1636,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155844632,"num_objects":2379,"num_object_clones":0,"num_object_copies":7137,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2379,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14241,"num_write_kb":531777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"20'137","reported_seq":1090,"reported_epoch":1135,"state":"active+clean","last_fresh":"2024-03-20T09:44:03.040874+0000","last_change":"2024-03-20T09:43:15.623506+0000","last_active":"2024-03-20T09:44:03.040874+0000","last_peered":"2024-03-20T09:44:03.040874+0000","last_clean":"2024-03-20T09:44:03.040874+0000","last_became_active":"2024-03-20T09:35:19.820895+0000","last_became_peered":"2024-03-20T09:35:19.820895+0000","last_unstale":"2024-03-20T09:44:03.040874+0000","last_undegraded":"2024-03-20T09:44:03.040874+0000","last_fullsized":"2024-03-20T09:44:03.040874+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:15.623424+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:19:03.805185+0000","last_clean_scrub_stamp":"2024-03-20T09:43:15.623424+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:28.959946+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"1135'14606","reported_seq":14890,"reported_epoch":1135,"state":"active+clean","last_fresh":"2024-03-20T09:44:04.185472+0000","last_change":"2024-03-20T09:43:12.887942+0000","last_active":"2024-03-20T09:44:04.185472+0000","last_peered":"2024-03-20T09:44:04.185472+0000","last_clean":"2024-03-20T09:44:04.185472+0000","last_became_active":"2024-03-20T09:40:24.813215+0000","last_became_peered":"2024-03-20T09:40:24.813215+0000","last_unstale":"2024-03-20T09:44:04.185472+0000","last_undegraded":"2024-03-20T09:44:04.185472+0000","last_fullsized":"2024-03-20T09:44:04.185472+0000","mapping_epoch":1066,"log_start":"1081'4600","ondisk_log_start":"1081'4600","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1120'11793","last_scrub_stamp":"2024-03-20T09:43:12.887835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:12.887835+0000","objects_scrubbed":4853,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:14.822931+0000","scrub_duration":1311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131858432,"num_objects":2012,"num_object_clones":0,"num_object_copies":6036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14606,"num_write_kb":531776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,1],"acting":[10,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"20'137","reported_seq":1960,"reported_epoch":1135,"state":"active+clean","last_fresh":"2024-03-20T09:44:03.044470+0000","last_change":"2024-03-20T09:43:14.612155+0000","last_active":"2024-03-20T09:44:03.044470+0000","last_peered":"2024-03-20T09:44:03.044470+0000","last_clean":"2024-03-20T09:44:03.044470+0000","last_became_active":"2024-03-20T09:35:19.822321+0000","last_became_peered":"2024-03-20T09:35:19.822321+0000","last_unstale":"2024-03-20T09:44:03.044470+0000","last_undegraded":"2024-03-20T09:44:03.044470+0000","last_fullsized":"2024-03-20T09:44:03.044470+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:14.612037+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:24:31.549603+0000","last_clean_scrub_stamp":"2024-03-20T09:43:14.612037+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:21.810461+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.b","version":"1134'14271","reported_seq":14553,"reported_epoch":1134,"state":"active+clean","last_fresh":"2024-03-20T09:44:01.894721+0000","last_change":"2024-03-20T09:43:21.410182+0000","last_active":"2024-03-20T09:44:01.894721+0000","last_peered":"2024-03-20T09:44:01.894721+0000","last_clean":"2024-03-20T09:44:01.894721+0000","last_became_active":"2024-03-20T09:40:24.791110+0000","last_became_peered":"2024-03-20T09:40:24.791110+0000","last_unstale":"2024-03-20T09:44:01.894721+0000","last_undegraded":"2024-03-20T09:44:01.894721+0000","last_fullsized":"2024-03-20T09:44:01.894721+0000","mapping_epoch":1066,"log_start":"1081'4200","ondisk_log_start":"1081'4200","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1122'12128","last_scrub_stamp":"2024-03-20T09:43:21.409950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:21.409950+0000","objects_scrubbed":4326,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:49.041623+0000","scrub_duration":1397,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140705792,"num_objects":2147,"num_object_clones":0,"num_object_copies":6441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14271,"num_write_kb":525376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"20'137","reported_seq":1491,"reported_epoch":1133,"state":"active+clean","last_fresh":"2024-03-20T09:43:56.780288+0000","last_change":"2024-03-20T09:43:10.005346+0000","last_active":"2024-03-20T09:43:56.780288+0000","last_peered":"2024-03-20T09:43:56.780288+0000","last_clean":"2024-03-20T09:43:56.780288+0000","last_became_active":"2024-03-20T09:31:07.424588+0000","last_became_peered":"2024-03-20T09:31:07.424588+0000","last_unstale":"2024-03-20T09:43:56.780288+0000","last_undegraded":"2024-03-20T09:43:56.780288+0000","last_fullsized":"2024-03-20T09:43:56.780288+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:10.005211+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:30:47.812125+0000","last_clean_scrub_stamp":"2024-03-20T09:43:10.005211+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:10.107177+0000","scrub_duration":221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"1135'16438","reported_seq":16798,"reported_epoch":1135,"state":"active+clean","last_fresh":"2024-03-20T09:44:03.911877+0000","last_change":"2024-03-20T09:43:57.525315+0000","last_active":"2024-03-20T09:44:03.911877+0000","last_peered":"2024-03-20T09:44:03.911877+0000","last_clean":"2024-03-20T09:44:03.911877+0000","last_became_active":"2024-03-20T09:40:24.790197+0000","last_became_peered":"2024-03-20T09:40:24.790197+0000","last_unstale":"2024-03-20T09:44:03.911877+0000","last_undegraded":"2024-03-20T09:44:03.911877+0000","last_fullsized":"2024-03-20T09:44:03.911877+0000","mapping_epoch":1066,"log_start":"1087'6400","ondisk_log_start":"1087'6400","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1133'16101","last_scrub_stamp":"2024-03-20T09:43:57.524948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:57.524948+0000","objects_scrubbed":2292,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:15.662531+0000","scrub_duration":556,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127401984,"num_objects":1944,"num_object_clones":0,"num_object_copies":5832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14486,"num_write_kb":525760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":2398,"reported_epoch":1134,"state":"active+clean","last_fresh":"2024-03-20T09:44:01.023629+0000","last_change":"2024-03-20T09:43:08.050620+0000","last_active":"2024-03-20T09:44:01.023629+0000","last_peered":"2024-03-20T09:44:01.023629+0000","last_clean":"2024-03-20T09:44:01.023629+0000","last_became_active":"2024-03-20T09:12:14.305989+0000","last_became_peered":"2024-03-20T09:12:14.305989+0000","last_unstale":"2024-03-20T09:44:01.023629+0000","last_undegraded":"2024-03-20T09:44:01.023629+0000","last_fullsized":"2024-03-20T09:44:01.023629+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:08.050532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:08.050532+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:21.415537+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.a","version":"1133'14287","reported_seq":14567,"reported_epoch":1133,"state":"active+clean","last_fresh":"2024-03-20T09:44:00.415532+0000","last_change":"2024-03-20T09:43:08.178211+0000","last_active":"2024-03-20T09:44:00.415532+0000","last_peered":"2024-03-20T09:44:00.415532+0000","last_clean":"2024-03-20T09:44:00.415532+0000","last_became_active":"2024-03-20T09:40:24.812882+0000","last_became_peered":"2024-03-20T09:40:24.812882+0000","last_unstale":"2024-03-20T09:44:00.415532+0000","last_undegraded":"2024-03-20T09:44:00.415532+0000","last_fullsized":"2024-03-20T09:44:00.415532+0000","mapping_epoch":1066,"log_start":"1081'4200","ondisk_log_start":"1081'4200","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1117'11503","last_scrub_stamp":"2024-03-20T09:43:08.177809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:08.177809+0000","objects_scrubbed":4944,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:34.330102+0000","scrub_duration":1251,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139919360,"num_objects":2135,"num_object_clones":0,"num_object_copies":6405,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14287,"num_write_kb":525504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"20'137","reported_seq":1798,"reported_epoch":1133,"state":"active+clean","last_fresh":"2024-03-20T09:43:57.239236+0000","last_change":"2024-03-20T09:43:57.239285+0000","last_active":"2024-03-20T09:43:57.239236+0000","last_peered":"2024-03-20T09:43:57.239236+0000","last_clean":"2024-03-20T09:43:57.239236+0000","last_became_active":"2024-03-20T09:39:17.944040+0000","last_became_peered":"2024-03-20T09:39:17.944040+0000","last_unstale":"2024-03-20T09:43:57.239236+0000","last_undegraded":"2024-03-20T09:43:57.239236+0000","last_fullsized":"2024-03-20T09:43:57.239236+0000","mapping_epoch":1041,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":1042,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:57.239132+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:42:42.369236+0000","last_clean_scrub_stamp":"2024-03-20T09:43:57.239132+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:11.927324+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,8],"acting":[4,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.9","version":"1135'14559","reported_seq":14846,"reported_epoch":1135,"state":"active+clean","last_fresh":"2024-03-20T09:44:04.191874+0000","last_change":"2024-03-20T09:43:09.692400+0000","last_active":"2024-03-20T09:44:04.191874+0000","last_peered":"2024-03-20T09:44:04.191874+0000","last_clean":"2024-03-20T09:44:04.191874+0000","last_became_active":"2024-03-20T09:40:24.792650+0000","last_became_peered":"2024-03-20T09:40:24.792650+0000","last_unstale":"2024-03-20T09:44:04.191874+0000","last_undegraded":"2024-03-20T09:44:04.191874+0000","last_fullsized":"2024-03-20T09:44:04.191874+0000","mapping_epoch":1066,"log_start":"1081'4500","ondisk_log_start":"1081'4500","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'11635","last_scrub_stamp":"2024-03-20T09:43:09.692199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:09.692199+0000","objects_scrubbed":4930,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:31.593123+0000","scrub_duration":1148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130220032,"num_objects":1987,"num_object_clones":0,"num_object_copies":5961,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14559,"num_write_kb":529472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"20'137","reported_seq":1837,"reported_epoch":1133,"state":"active+clean","last_fresh":"2024-03-20T09:43:56.781080+0000","last_change":"2024-03-20T09:43:06.796634+0000","last_active":"2024-03-20T09:43:56.781080+0000","last_peered":"2024-03-20T09:43:56.781080+0000","last_clean":"2024-03-20T09:43:56.781080+0000","last_became_active":"2024-03-20T09:15:34.225022+0000","last_became_peered":"2024-03-20T09:15:34.225022+0000","last_unstale":"2024-03-20T09:43:56.781080+0000","last_undegraded":"2024-03-20T09:43:56.781080+0000","last_fullsized":"2024-03-20T09:43:56.781080+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:06.796525+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:20:12.169658+0000","last_clean_scrub_stamp":"2024-03-20T09:43:06.796525+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:36.395280+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,5],"acting":[6,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"1133'14397","reported_seq":14680,"reported_epoch":1133,"state":"active+clean","last_fresh":"2024-03-20T09:44:00.406543+0000","last_change":"2024-03-20T09:43:13.370701+0000","last_active":"2024-03-20T09:44:00.406543+0000","last_peered":"2024-03-20T09:44:00.406543+0000","last_clean":"2024-03-20T09:44:00.406543+0000","last_became_active":"2024-03-20T09:40:24.790189+0000","last_became_peered":"2024-03-20T09:40:24.790189+0000","last_unstale":"2024-03-20T09:44:00.406543+0000","last_undegraded":"2024-03-20T09:44:00.406543+0000","last_fullsized":"2024-03-20T09:44:00.406543+0000","mapping_epoch":1066,"log_start":"1081'4300","ondisk_log_start":"1081'4300","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1120'11823","last_scrub_stamp":"2024-03-20T09:43:13.370340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:13.370340+0000","objects_scrubbed":4782,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:20.684399+0000","scrub_duration":1344,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142278656,"num_objects":2171,"num_object_clones":0,"num_object_copies":6513,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14397,"num_write_kb":530176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"20'137","reported_seq":2393,"reported_epoch":1133,"state":"active+clean","last_fresh":"2024-03-20T09:43:56.780570+0000","last_change":"2024-03-20T09:43:11.044649+0000","last_active":"2024-03-20T09:43:56.780570+0000","last_peered":"2024-03-20T09:43:56.780570+0000","last_clean":"2024-03-20T09:43:56.780570+0000","last_became_active":"2024-03-20T09:35:19.820396+0000","last_became_peered":"2024-03-20T09:35:19.820396+0000","last_unstale":"2024-03-20T09:43:56.780570+0000","last_undegraded":"2024-03-20T09:43:56.780570+0000","last_fullsized":"2024-03-20T09:43:56.780570+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:11.044527+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:14:10.560890+0000","last_clean_scrub_stamp":"2024-03-20T09:43:11.044527+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:19.618048+0000","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"1135'14599","reported_seq":14880,"reported_epoch":1135,"state":"active+clean","last_fresh":"2024-03-20T09:44:03.942677+0000","last_change":"2024-03-20T09:43:09.360089+0000","last_active":"2024-03-20T09:44:03.942677+0000","last_peered":"2024-03-20T09:44:03.942677+0000","last_clean":"2024-03-20T09:44:03.942677+0000","last_became_active":"2024-03-20T09:40:24.791199+0000","last_became_peered":"2024-03-20T09:40:24.791199+0000","last_unstale":"2024-03-20T09:44:03.942677+0000","last_undegraded":"2024-03-20T09:44:03.942677+0000","last_fullsized":"2024-03-20T09:44:03.942677+0000","mapping_epoch":1066,"log_start":"1081'4500","ondisk_log_start":"1081'4500","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'11689","last_scrub_stamp":"2024-03-20T09:43:09.359969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:09.359969+0000","objects_scrubbed":4954,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:28.242282+0000","scrub_duration":1552,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132448256,"num_objects":2021,"num_object_clones":0,"num_object_copies":6063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14599,"num_write_kb":531840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,5],"acting":[9,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"20'137","reported_seq":2227,"reported_epoch":1134,"state":"active+clean","last_fresh":"2024-03-20T09:44:01.023735+0000","last_change":"2024-03-20T09:43:50.466520+0000","last_active":"2024-03-20T09:44:01.023735+0000","last_peered":"2024-03-20T09:44:01.023735+0000","last_clean":"2024-03-20T09:44:01.023735+0000","last_became_active":"2024-03-20T09:39:17.940497+0000","last_became_peered":"2024-03-20T09:39:17.940497+0000","last_unstale":"2024-03-20T09:44:01.023735+0000","last_undegraded":"2024-03-20T09:44:01.023735+0000","last_fullsized":"2024-03-20T09:44:01.023735+0000","mapping_epoch":1041,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":1042,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:50.466383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:50.466383+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:05.247322+0000","scrub_duration":307,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,2],"acting":[11,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.6","version":"1133'17279","reported_seq":17707,"reported_epoch":1133,"state":"active+clean","last_fresh":"2024-03-20T09:44:00.422736+0000","last_change":"2024-03-20T09:43:46.116088+0000","last_active":"2024-03-20T09:44:00.422736+0000","last_peered":"2024-03-20T09:44:00.422736+0000","last_clean":"2024-03-20T09:44:00.422736+0000","last_became_active":"2024-03-20T09:40:24.792369+0000","last_became_peered":"2024-03-20T09:40:24.792369+0000","last_unstale":"2024-03-20T09:44:00.422736+0000","last_undegraded":"2024-03-20T09:44:00.422736+0000","last_fullsized":"2024-03-20T09:44:00.422736+0000","mapping_epoch":1066,"log_start":"1093'7200","ondisk_log_start":"1093'7200","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1131'16486","last_scrub_stamp":"2024-03-20T09:43:46.115894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:46.115894+0000","objects_scrubbed":3011,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:57.198816+0000","scrub_duration":1002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143130624,"num_objects":2184,"num_object_clones":0,"num_object_copies":6552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14246,"num_write_kb":525760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"20'137","reported_seq":1649,"reported_epoch":1133,"state":"active+clean","last_fresh":"2024-03-20T09:43:56.792190+0000","last_change":"2024-03-20T09:43:04.334350+0000","last_active":"2024-03-20T09:43:56.792190+0000","last_peered":"2024-03-20T09:43:56.792190+0000","last_clean":"2024-03-20T09:43:56.792190+0000","last_became_active":"2024-03-20T09:19:54.506134+0000","last_became_peered":"2024-03-20T09:19:54.506134+0000","last_unstale":"2024-03-20T09:43:56.792190+0000","last_undegraded":"2024-03-20T09:43:56.792190+0000","last_fullsized":"2024-03-20T09:43:56.792190+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:04.334199+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:29:53.362224+0000","last_clean_scrub_stamp":"2024-03-20T09:43:04.334199+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:08.842466+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"1135'14743","reported_seq":15018,"reported_epoch":1135,"state":"active+clean","last_fresh":"2024-03-20T09:44:04.157328+0000","last_change":"2024-03-20T09:43:20.856797+0000","last_active":"2024-03-20T09:44:04.157328+0000","last_peered":"2024-03-20T09:44:04.157328+0000","last_clean":"2024-03-20T09:44:04.157328+0000","last_became_active":"2024-03-20T09:40:24.813246+0000","last_became_peered":"2024-03-20T09:40:24.813246+0000","last_unstale":"2024-03-20T09:44:04.157328+0000","last_undegraded":"2024-03-20T09:44:04.157328+0000","last_fullsized":"2024-03-20T09:44:04.157328+0000","mapping_epoch":1066,"log_start":"1083'4700","ondisk_log_start":"1083'4700","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1122'12369","last_scrub_stamp":"2024-03-20T09:43:20.856432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:20.856432+0000","objects_scrubbed":4430,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:43.625443+0000","scrub_duration":1246,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132448256,"num_objects":2021,"num_object_clones":0,"num_object_copies":6063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14743,"num_write_kb":536448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,3],"acting":[10,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"20'137","reported_seq":2413,"reported_epoch":1133,"state":"active+clean","last_fresh":"2024-03-20T09:43:56.780470+0000","last_change":"2024-03-20T09:43:50.972722+0000","last_active":"2024-03-20T09:43:56.780470+0000","last_peered":"2024-03-20T09:43:56.780470+0000","last_clean":"2024-03-20T09:43:56.780470+0000","last_became_active":"2024-03-20T09:19:54.524397+0000","last_became_peered":"2024-03-20T09:19:54.524397+0000","last_unstale":"2024-03-20T09:43:56.780470+0000","last_undegraded":"2024-03-20T09:43:56.780470+0000","last_fullsized":"2024-03-20T09:43:56.780470+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:50.972601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:50.972601+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:58.813078+0000","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"1133'14060","reported_seq":14337,"reported_epoch":1133,"state":"active+clean","last_fresh":"2024-03-20T09:44:00.767583+0000","last_change":"2024-03-20T09:43:09.459247+0000","last_active":"2024-03-20T09:44:00.767583+0000","last_peered":"2024-03-20T09:44:00.767583+0000","last_clean":"2024-03-20T09:44:00.767583+0000","last_became_active":"2024-03-20T09:40:24.789539+0000","last_became_peered":"2024-03-20T09:40:24.789539+0000","last_unstale":"2024-03-20T09:44:00.767583+0000","last_undegraded":"2024-03-20T09:44:00.767583+0000","last_fullsized":"2024-03-20T09:44:00.767583+0000","mapping_epoch":1066,"log_start":"1079'4000","ondisk_log_start":"1079'4000","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'11421","last_scrub_stamp":"2024-03-20T09:43:09.459059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:09.459059+0000","objects_scrubbed":4816,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:29.439861+0000","scrub_duration":1236,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141426688,"num_objects":2158,"num_object_clones":0,"num_object_copies":6474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14060,"num_write_kb":518976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.c","version":"570'138","reported_seq":2404,"reported_epoch":1133,"state":"active+clean","last_fresh":"2024-03-20T09:43:56.780090+0000","last_change":"2024-03-20T09:43:07.728946+0000","last_active":"2024-03-20T09:43:56.780090+0000","last_peered":"2024-03-20T09:43:56.780090+0000","last_clean":"2024-03-20T09:43:56.780090+0000","last_became_active":"2024-03-20T09:39:17.943381+0000","last_became_peered":"2024-03-20T09:39:17.943381+0000","last_unstale":"2024-03-20T09:43:56.780090+0000","last_undegraded":"2024-03-20T09:43:56.780090+0000","last_fullsized":"2024-03-20T09:43:56.780090+0000","mapping_epoch":1041,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":1042,"parent":"0.0","parent_split_bits":4,"last_scrub":"570'138","last_scrub_stamp":"2024-03-20T09:43:07.728813+0000","last_deep_scrub":"570'138","last_deep_scrub_stamp":"2024-03-20T09:16:13.725035+0000","last_clean_scrub_stamp":"2024-03-20T09:43:07.728813+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:26.795166+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,9],"acting":[3,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"1134'14593","reported_seq":14879,"reported_epoch":1134,"state":"active+clean","last_fresh":"2024-03-20T09:44:01.104585+0000","last_change":"2024-03-20T09:43:20.493432+0000","last_active":"2024-03-20T09:44:01.104585+0000","last_peered":"2024-03-20T09:44:01.104585+0000","last_clean":"2024-03-20T09:44:01.104585+0000","last_became_active":"2024-03-20T09:40:24.813276+0000","last_became_peered":"2024-03-20T09:40:24.813276+0000","last_unstale":"2024-03-20T09:44:01.104585+0000","last_undegraded":"2024-03-20T09:44:01.104585+0000","last_fullsized":"2024-03-20T09:44:01.104585+0000","mapping_epoch":1066,"log_start":"1081'4500","ondisk_log_start":"1081'4500","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1122'12403","last_scrub_stamp":"2024-03-20T09:43:20.493115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:20.493115+0000","objects_scrubbed":4492,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:21.692329+0000","scrub_duration":1417,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148439040,"num_objects":2265,"num_object_clones":0,"num_object_copies":6795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14593,"num_write_kb":539456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"20'137","reported_seq":1963,"reported_epoch":1133,"state":"active+clean","last_fresh":"2024-03-20T09:43:56.790744+0000","last_change":"2024-03-20T09:43:46.440102+0000","last_active":"2024-03-20T09:43:56.790744+0000","last_peered":"2024-03-20T09:43:56.790744+0000","last_clean":"2024-03-20T09:43:56.790744+0000","last_became_active":"2024-03-20T09:19:54.502529+0000","last_became_peered":"2024-03-20T09:19:54.502529+0000","last_unstale":"2024-03-20T09:43:56.790744+0000","last_undegraded":"2024-03-20T09:43:56.790744+0000","last_fullsized":"2024-03-20T09:43:56.790744+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:46.439952+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:41:52.574619+0000","last_clean_scrub_stamp":"2024-03-20T09:43:46.439952+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:05.231528+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"20'137","reported_seq":2244,"reported_epoch":1133,"state":"active+clean","last_fresh":"2024-03-20T09:43:56.783217+0000","last_change":"2024-03-20T09:43:15.202936+0000","last_active":"2024-03-20T09:43:56.783217+0000","last_peered":"2024-03-20T09:43:56.783217+0000","last_clean":"2024-03-20T09:43:56.783217+0000","last_became_active":"2024-03-20T09:10:34.096758+0000","last_became_peered":"2024-03-20T09:10:34.096758+0000","last_unstale":"2024-03-20T09:43:56.783217+0000","last_undegraded":"2024-03-20T09:43:56.783217+0000","last_fullsized":"2024-03-20T09:43:56.783217+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:15.202813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:15.202813+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:35.894669+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"20'137","reported_seq":1006,"reported_epoch":1134,"state":"active+clean","last_fresh":"2024-03-20T09:44:01.023823+0000","last_change":"2024-03-20T09:43:15.359792+0000","last_active":"2024-03-20T09:44:01.023823+0000","last_peered":"2024-03-20T09:44:01.023823+0000","last_clean":"2024-03-20T09:44:01.023823+0000","last_became_active":"2024-03-20T09:35:19.822242+0000","last_became_peered":"2024-03-20T09:35:19.822242+0000","last_unstale":"2024-03-20T09:44:01.023823+0000","last_undegraded":"2024-03-20T09:44:01.023823+0000","last_fullsized":"2024-03-20T09:44:01.023823+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:15.359586+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:37:52.359932+0000","last_clean_scrub_stamp":"2024-03-20T09:43:15.359586+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:29.699947+0000","scrub_duration":298,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.10","version":"20'137","reported_seq":1815,"reported_epoch":1133,"state":"active+clean","last_fresh":"2024-03-20T09:43:56.783555+0000","last_change":"2024-03-20T09:43:48.137633+0000","last_active":"2024-03-20T09:43:56.783555+0000","last_peered":"2024-03-20T09:43:56.783555+0000","last_clean":"2024-03-20T09:43:56.783555+0000","last_became_active":"2024-03-20T09:35:19.820595+0000","last_became_peered":"2024-03-20T09:35:19.820595+0000","last_unstale":"2024-03-20T09:43:56.783555+0000","last_undegraded":"2024-03-20T09:43:56.783555+0000","last_fullsized":"2024-03-20T09:43:56.783555+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:48.137502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:48.137502+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:17.764144+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"20'137","reported_seq":604,"reported_epoch":1134,"state":"active+clean","last_fresh":"2024-03-20T09:44:01.025578+0000","last_change":"2024-03-20T09:43:09.196572+0000","last_active":"2024-03-20T09:44:01.025578+0000","last_peered":"2024-03-20T09:44:01.025578+0000","last_clean":"2024-03-20T09:44:01.025578+0000","last_became_active":"2024-03-20T09:39:17.943385+0000","last_became_peered":"2024-03-20T09:39:17.943385+0000","last_unstale":"2024-03-20T09:44:01.025578+0000","last_undegraded":"2024-03-20T09:44:01.025578+0000","last_fullsized":"2024-03-20T09:44:01.025578+0000","mapping_epoch":1041,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":1042,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:09.196479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:09.196479+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:34.758699+0000","scrub_duration":184,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"20'137","reported_seq":879,"reported_epoch":1133,"state":"active+clean","last_fresh":"2024-03-20T09:43:56.779914+0000","last_change":"2024-03-20T09:43:06.840465+0000","last_active":"2024-03-20T09:43:56.779914+0000","last_peered":"2024-03-20T09:43:56.779914+0000","last_clean":"2024-03-20T09:43:56.779914+0000","last_became_active":"2024-03-20T09:39:17.943337+0000","last_became_peered":"2024-03-20T09:39:17.943337+0000","last_unstale":"2024-03-20T09:43:56.779914+0000","last_undegraded":"2024-03-20T09:43:56.779914+0000","last_fullsized":"2024-03-20T09:43:56.779914+0000","mapping_epoch":1041,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":1042,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:06.840344+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:35:27.264203+0000","last_clean_scrub_stamp":"2024-03-20T09:43:06.840344+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:31.325871+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,11],"acting":[3,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"20'137","reported_seq":2387,"reported_epoch":1134,"state":"active+clean","last_fresh":"2024-03-20T09:44:01.028406+0000","last_change":"2024-03-20T09:43:12.288008+0000","last_active":"2024-03-20T09:44:01.028406+0000","last_peered":"2024-03-20T09:44:01.028406+0000","last_clean":"2024-03-20T09:44:01.028406+0000","last_became_active":"2024-03-20T09:35:19.822429+0000","last_became_peered":"2024-03-20T09:35:19.822429+0000","last_unstale":"2024-03-20T09:44:01.028406+0000","last_undegraded":"2024-03-20T09:44:01.028406+0000","last_fullsized":"2024-03-20T09:44:01.028406+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:12.287881+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:38:01.419884+0000","last_clean_scrub_stamp":"2024-03-20T09:43:12.287881+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:27.962869+0000","scrub_duration":272,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"20'137","reported_seq":2420,"reported_epoch":1135,"state":"active+clean","last_fresh":"2024-03-20T09:44:03.125928+0000","last_change":"2024-03-20T09:43:51.249807+0000","last_active":"2024-03-20T09:44:03.125928+0000","last_peered":"2024-03-20T09:44:03.125928+0000","last_clean":"2024-03-20T09:44:03.125928+0000","last_became_active":"2024-03-20T09:20:25.349082+0000","last_became_peered":"2024-03-20T09:20:25.349082+0000","last_unstale":"2024-03-20T09:44:03.125928+0000","last_undegraded":"2024-03-20T09:44:03.125928+0000","last_fullsized":"2024-03-20T09:44:03.125928+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:51.249669+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:41:54.735667+0000","last_clean_scrub_stamp":"2024-03-20T09:43:51.249669+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:17.928220+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"20'137","reported_seq":1698,"reported_epoch":1134,"state":"active+clean","last_fresh":"2024-03-20T09:44:01.019285+0000","last_change":"2024-03-20T09:43:55.035201+0000","last_active":"2024-03-20T09:44:01.019285+0000","last_peered":"2024-03-20T09:44:01.019285+0000","last_clean":"2024-03-20T09:44:01.019285+0000","last_became_active":"2024-03-20T09:12:14.307581+0000","last_became_peered":"2024-03-20T09:12:14.307581+0000","last_unstale":"2024-03-20T09:44:01.019285+0000","last_undegraded":"2024-03-20T09:44:01.019285+0000","last_fullsized":"2024-03-20T09:44:01.019285+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:55.035080+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:34:48.079272+0000","last_clean_scrub_stamp":"2024-03-20T09:43:55.035080+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:02.488697+0000","scrub_duration":240,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"20'137","reported_seq":959,"reported_epoch":1133,"state":"active+clean","last_fresh":"2024-03-20T09:43:56.779821+0000","last_change":"2024-03-20T09:43:56.715773+0000","last_active":"2024-03-20T09:43:56.779821+0000","last_peered":"2024-03-20T09:43:56.779821+0000","last_clean":"2024-03-20T09:43:56.779821+0000","last_became_active":"2024-03-20T09:41:47.137425+0000","last_became_peered":"2024-03-20T09:41:47.137425+0000","last_unstale":"2024-03-20T09:43:56.779821+0000","last_undegraded":"2024-03-20T09:43:56.779821+0000","last_fullsized":"2024-03-20T09:43:56.779821+0000","mapping_epoch":1094,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":1095,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:56.715666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:56.715666+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:02.384244+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":2237661208,"num_objects":34145,"num_object_clones":0,"num_object_copies":102435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34145,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":230311,"num_write_kb":8462529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6627471360,"data_stored":6627459144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161055,"ondisk_log_size":161055,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1925120,"data_stored":1914464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4248,"ondisk_log_size":4248,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860993,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1676392,"kb_used_data":572064,"kb_used_omap":1,"kb_used_meta":1104318,"kb_avail":92068760,"statfs":{"total":95995035648,"available":94278410240,"internally_reserved":0,"allocated":585793536,"data_stored":585523504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":1130822084},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":861,"seq":3697966842018,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":978048,"kb_used_data":652728,"kb_used_omap":103,"kb_used_meta":325208,"kb_avail":92767104,"statfs":{"total":95995035648,"available":94993514496,"internally_reserved":0,"allocated":668393472,"data_stored":668114626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":106169,"internal_metadata":333013319},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":80,"seq":343597384270,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1494336,"kb_used_data":763448,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":92250816,"statfs":{"total":95995035648,"available":94464835584,"internally_reserved":0,"allocated":781770752,"data_stored":781491906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":748419520},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":70,"seq":300647711314,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":937208,"kb_used_data":274348,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":92807944,"statfs":{"total":95995035648,"available":95035334656,"internally_reserved":0,"allocated":280932352,"data_stored":280657938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":678754758},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":60,"seq":257698038358,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1726708,"kb_used_data":716780,"kb_used_omap":1,"kb_used_meta":1009918,"kb_avail":92018444,"statfs":{"total":95995035648,"available":94226886656,"internally_reserved":0,"allocated":733982720,"data_stored":733705064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":1034156490},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332699,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1056232,"kb_used_data":276768,"kb_used_omap":1,"kb_used_meta":779454,"kb_avail":92688920,"statfs":{"total":95995035648,"available":94913454080,"internally_reserved":0,"allocated":283410432,"data_stored":283140400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":798161350},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575478,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2201800,"kb_used_data":1019708,"kb_used_omap":1,"kb_used_meta":1182078,"kb_avail":91543352,"statfs":{"total":95995035648,"available":93740392448,"internally_reserved":0,"allocated":1044180992,"data_stored":1043898074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1210448326},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738996,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411951,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":852556,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":92892596,"statfs":{"total":95995035648,"available":95122018304,"internally_reserved":0,"allocated":917504,"data_stored":635106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":872085957},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150315,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1829160,"kb_used_data":715104,"kb_used_omap":1,"kb_used_meta":1114046,"kb_avail":91915992,"statfs":{"total":95995035648,"available":94121975808,"internally_reserved":0,"allocated":732266496,"data_stored":731996464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1140783559},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438954089,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1087912,"kb_used_data":280608,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":92657240,"statfs":{"total":95995035648,"available":94881013760,"internally_reserved":0,"allocated":287342592,"data_stored":287072560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":826603973},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":443,"seq":1902670512530,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2272052,"kb_used_data":1285548,"kb_used_omap":181,"kb_used_meta":986314,"kb_avail":91473100,"statfs":{"total":95995035648,"available":93668454400,"internally_reserved":0,"allocated":1316401152,"data_stored":1316123496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":186131,"internal_metadata":1009985773},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1024331776,"data_stored":1024327704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":721158144,"data_stored":721158144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":282460160,"data_stored":282460160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1309216768,"data_stored":1309212696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":281149440,"data_stored":281149440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":726077440,"data_stored":726073368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":278462464,"data_stored":278462464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":778305536,"data_stored":778305536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":655687680,"data_stored":655687680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":570621952,"data_stored":570621952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:44:06.843 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:44:06.928 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:44:06.973 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:07.046 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:07 smithi026 ceph-mon[68309]: pgmap v2761: 47 pgs: 47 active+clean; 1.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 882 op/s 2024-03-20T09:44:07.101 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:44:07.101 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:44:07.101 INFO:teuthology.orchestra.run.smithi115.stdout: "complaint_time": 30, 2024-03-20T09:44:07.101 INFO:teuthology.orchestra.run.smithi115.stdout: "num_blocked_ops": 0 2024-03-20T09:44:07.101 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:44:07.133 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:44:07.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:07.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:44:07.239 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:44:07.296 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:07.370 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:44:07.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:07.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:44:07.380 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:07 smithi115 ceph-mon[72476]: pgmap v2761: 47 pgs: 47 active+clean; 1.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 882 op/s 2024-03-20T09:44:07.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:07 smithi122 ceph-mon[73495]: pgmap v2761: 47 pgs: 47 active+clean; 1.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 882 op/s 2024-03-20T09:44:07.479 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:44:07.488 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:44:07.489 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:44:07.532 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:07.583 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:44:07.601 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:07.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:44:07.702 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:44:07.726 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:44:07.748 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:44:07.753 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:07.782 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:44:07.814 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:44:07.842 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:44:07.876 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:44:07.902 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:44:07.942 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:44:07.981 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:44:08.025 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:44:08.028 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:44:08.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:08.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:44:08.040 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-20T09:44:08.071 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:44:08.123 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:44:08.123 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:44:08.123 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:44:08.123 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:44:08.123 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:44:08.123 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:44:08.123 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:44:08.123 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:44:08.123 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:44:08.123 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:44:08.124 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:44:08.124 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:44:08.124 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:44:08.134 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:44:08.200 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:08.240 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:44:08.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:08.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:44:08.353 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:44:08.391 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:08 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1217610661' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:44:08.391 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:08.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:08 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1217610661' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:44:08.429 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:44:08.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:08.445 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:44:08.462 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:08 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1217610661' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:44:08.471 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:44:08.546 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:44:08.591 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:09.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:09 smithi115 ceph-mon[72476]: pgmap v2762: 47 pgs: 47 active+clean; 1.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 656 op/s 2024-03-20T09:44:09.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:09 smithi122 ceph-mon[73495]: pgmap v2762: 47 pgs: 47 active+clean; 1.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 656 op/s 2024-03-20T09:44:09.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:09 smithi026 ceph-mon[68309]: pgmap v2762: 47 pgs: 47 active+clean; 1.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 656 op/s 2024-03-20T09:44:10.433 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:44:10.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:10.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:44:10.541 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:44:10.580 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:10.615 INFO:teuthology.orchestra.run.smithi115.stdout:ERROR: (22) Invalid argument 2024-03-20T09:44:10.615 INFO:teuthology.orchestra.run.smithi115.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:44:10.738 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:44:10.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:10.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:44:10.843 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:44:10.879 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:11.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:11 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:44:11.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:11 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:44:11.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:11 smithi115 ceph-mon[72476]: osdmap e1137: 12 total, 12 up, 11 in 2024-03-20T09:44:11.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:11 smithi115 ceph-mon[72476]: pgmap v2764: 47 pgs: 47 active+clean; 1.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s 2024-03-20T09:44:11.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:11 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:44:11.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:11 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:44:11.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:11 smithi122 ceph-mon[73495]: osdmap e1137: 12 total, 12 up, 11 in 2024-03-20T09:44:11.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:11 smithi122 ceph-mon[73495]: pgmap v2764: 47 pgs: 47 active+clean; 1.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s 2024-03-20T09:44:11.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:11 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:44:11.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:11 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:44:11.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:11 smithi026 ceph-mon[68309]: osdmap e1137: 12 total, 12 up, 11 in 2024-03-20T09:44:11.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:11 smithi026 ceph-mon[68309]: pgmap v2764: 47 pgs: 47 active+clean; 1.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s 2024-03-20T09:44:12.040 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:44:12.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:12.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:44:12.144 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:44:12.145 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:44:12.196 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:12.683 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_ops_in_flight 2024-03-20T09:44:12.709 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:44:12.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:12.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:44:12.818 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:44:12.884 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:12.923 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:44:12.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:12.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:44:13.031 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:44:13.089 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:44:13.095 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:13.128 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:44:13.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:13.133 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:44:13.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:13 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:44:13.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:13 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:44:13.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:13 smithi026 ceph-mon[68309]: osdmap e1138: 12 total, 12 up, 11 in 2024-03-20T09:44:13.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:13 smithi026 ceph-mon[68309]: pgmap v2766: 47 pgs: 47 active+clean; 1.6 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 980 op/s 2024-03-20T09:44:13.234 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:44:13.280 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:13 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:44:13.280 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:13 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:44:13.280 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:13 smithi115 ceph-mon[72476]: osdmap e1138: 12 total, 12 up, 11 in 2024-03-20T09:44:13.280 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:13 smithi115 ceph-mon[72476]: pgmap v2766: 47 pgs: 47 active+clean; 1.6 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 980 op/s 2024-03-20T09:44:13.280 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:13.326 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:44:13.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:13.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:44:13.376 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:13 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:44:13.376 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:13 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:44:13.376 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:13 smithi122 ceph-mon[73495]: osdmap e1138: 12 total, 12 up, 11 in 2024-03-20T09:44:13.376 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:13 smithi122 ceph-mon[73495]: pgmap v2766: 47 pgs: 47 active+clean; 1.6 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 980 op/s 2024-03-20T09:44:13.433 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:44:13.477 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:13.838 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:44:13.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:13.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:44:13.947 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:44:13.979 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:44:14.010 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:14.041 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:44:14.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:14.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:44:14.132 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:14 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2328122687' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:44:14.132 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:14 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:44:14.134 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is set 2024-03-20T09:44:14.148 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:44:14.191 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:44:14.193 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:44:14.202 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:14.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:14 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2328122687' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:44:14.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:14 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:44:14.419 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:44:14.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:14.425 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:44:14.425 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:14 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2328122687' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:44:14.425 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:14 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:44:14.526 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:44:14.580 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:14.612 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:44:14.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:14.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:44:14.719 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:44:14.754 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:14.795 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:44:14.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:14.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:44:14.908 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:44:14.968 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:15.141 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:44:15.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:15.146 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:44:15.150 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:15 smithi026 ceph-mon[68309]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:44:15.150 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:15 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:44:15.150 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:15 smithi026 ceph-mon[68309]: osdmap e1139: 12 total, 12 up, 11 in 2024-03-20T09:44:15.150 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:15 smithi026 ceph-mon[68309]: from='client.17604 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:44:15.150 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:15 smithi026 ceph-mon[68309]: pgmap v2768: 47 pgs: 47 active+clean; 1.5 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 975 op/s 2024-03-20T09:44:15.247 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:44:15.257 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:44:15.257 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:44:15.310 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:15.350 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:44:15.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:15.356 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:44:15.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:15 smithi122 ceph-mon[73495]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:44:15.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:15 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:44:15.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:15 smithi122 ceph-mon[73495]: osdmap e1139: 12 total, 12 up, 11 in 2024-03-20T09:44:15.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:15 smithi122 ceph-mon[73495]: from='client.17604 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:44:15.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:15 smithi122 ceph-mon[73495]: pgmap v2768: 47 pgs: 47 active+clean; 1.5 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 975 op/s 2024-03-20T09:44:15.457 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:44:15.523 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:15 smithi115 ceph-mon[72476]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:44:15.524 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:15 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-20T09:44:15.524 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:15 smithi115 ceph-mon[72476]: osdmap e1139: 12 total, 12 up, 11 in 2024-03-20T09:44:15.524 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:15 smithi115 ceph-mon[72476]: from='client.17604 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:44:15.524 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:15 smithi115 ceph-mon[72476]: pgmap v2768: 47 pgs: 47 active+clean; 1.5 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 975 op/s 2024-03-20T09:44:15.527 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:15.553 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:44:15.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:15.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:44:15.663 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:44:15.713 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:16.005 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":2767,"stamp":"2024-03-20T09:44:14.136451+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1708720696,"num_objects":26067,"num_object_clones":0,"num_object_copies":78201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26067,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":238391,"num_write_kb":8462529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":164883,"ondisk_log_size":164883,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":14690408,"kb_used_data":5099644,"kb_used_omap":299,"kb_used_meta":9590036,"kb_avail":1016506264,"statfs":{"total":1055945392128,"available":1040902414336,"internally_reserved":0,"allocated":5222035456,"data_stored":5218856638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":306651,"internal_metadata":9820197413},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,1,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-345833472,"num_objects":-5277,"num_object_clones":0,"num_object_copies":-15831,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5277,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.431621"},"pg_stats":[{"pgid":"1.1e","version":"20'137","reported_seq":1765,"reported_epoch":1137,"state":"active+clean","last_fresh":"2024-03-20T09:44:10.095633+0000","last_change":"2024-03-20T09:43:09.164860+0000","last_active":"2024-03-20T09:44:10.095633+0000","last_peered":"2024-03-20T09:44:10.095633+0000","last_clean":"2024-03-20T09:44:10.095633+0000","last_became_active":"2024-03-20T09:13:20.071655+0000","last_became_peered":"2024-03-20T09:13:20.071655+0000","last_unstale":"2024-03-20T09:44:10.095633+0000","last_undegraded":"2024-03-20T09:44:10.095633+0000","last_fullsized":"2024-03-20T09:44:10.095633+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:09.164738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:09.164738+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:33.937151+0000","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'137","reported_seq":850,"reported_epoch":1137,"state":"active+clean","last_fresh":"2024-03-20T09:44:10.095971+0000","last_change":"2024-03-20T09:43:45.421863+0000","last_active":"2024-03-20T09:44:10.095971+0000","last_peered":"2024-03-20T09:44:10.095971+0000","last_clean":"2024-03-20T09:44:10.095971+0000","last_became_active":"2024-03-20T09:31:07.425574+0000","last_became_peered":"2024-03-20T09:31:07.425574+0000","last_unstale":"2024-03-20T09:44:10.095971+0000","last_undegraded":"2024-03-20T09:44:10.095971+0000","last_fullsized":"2024-03-20T09:44:10.095971+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:45.421742+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:29:02.585165+0000","last_clean_scrub_stamp":"2024-03-20T09:43:45.421742+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:55.396984+0000","scrub_duration":262,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2],"acting":[11,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"20'137","reported_seq":2454,"reported_epoch":1137,"state":"active+clean","last_fresh":"2024-03-20T09:44:10.092599+0000","last_change":"2024-03-20T09:43:11.053490+0000","last_active":"2024-03-20T09:44:10.092599+0000","last_peered":"2024-03-20T09:44:10.092599+0000","last_clean":"2024-03-20T09:44:10.092599+0000","last_became_active":"2024-03-20T09:35:03.815532+0000","last_became_peered":"2024-03-20T09:35:03.815532+0000","last_unstale":"2024-03-20T09:44:10.092599+0000","last_undegraded":"2024-03-20T09:44:10.092599+0000","last_fullsized":"2024-03-20T09:44:10.092599+0000","mapping_epoch":950,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":951,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:11.053365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:11.053365+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:21.391918+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.19","version":"20'137","reported_seq":1636,"reported_epoch":1136,"state":"active+clean","last_fresh":"2024-03-20T09:44:05.049952+0000","last_change":"2024-03-20T09:43:44.857190+0000","last_active":"2024-03-20T09:44:05.049952+0000","last_peered":"2024-03-20T09:44:05.049952+0000","last_clean":"2024-03-20T09:44:05.049952+0000","last_became_active":"2024-03-20T09:39:17.953184+0000","last_became_peered":"2024-03-20T09:39:17.953184+0000","last_unstale":"2024-03-20T09:44:05.049952+0000","last_undegraded":"2024-03-20T09:44:05.049952+0000","last_fullsized":"2024-03-20T09:44:05.049952+0000","mapping_epoch":1041,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":1042,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:44.857069+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:30:56.201554+0000","last_clean_scrub_stamp":"2024-03-20T09:43:44.857069+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:06.289387+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,7],"acting":[10,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"20'137","reported_seq":1919,"reported_epoch":1137,"state":"active+clean","last_fresh":"2024-03-20T09:44:10.096005+0000","last_change":"2024-03-20T09:43:49.190805+0000","last_active":"2024-03-20T09:44:10.096005+0000","last_peered":"2024-03-20T09:44:10.096005+0000","last_clean":"2024-03-20T09:44:10.096005+0000","last_became_active":"2024-03-20T09:10:35.277481+0000","last_became_peered":"2024-03-20T09:10:35.277481+0000","last_unstale":"2024-03-20T09:44:10.096005+0000","last_undegraded":"2024-03-20T09:44:10.096005+0000","last_fullsized":"2024-03-20T09:44:10.096005+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:49.190689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:49.190689+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:03.907117+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1b","version":"20'137","reported_seq":2291,"reported_epoch":1137,"state":"active+clean","last_fresh":"2024-03-20T09:44:10.092783+0000","last_change":"2024-03-20T09:44:00.886678+0000","last_active":"2024-03-20T09:44:10.092783+0000","last_peered":"2024-03-20T09:44:10.092783+0000","last_clean":"2024-03-20T09:44:10.092783+0000","last_became_active":"2024-03-20T09:39:17.951408+0000","last_became_peered":"2024-03-20T09:39:17.951408+0000","last_unstale":"2024-03-20T09:44:10.092783+0000","last_undegraded":"2024-03-20T09:44:10.092783+0000","last_fullsized":"2024-03-20T09:44:10.092783+0000","mapping_epoch":1041,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":1042,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:44:00.886543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:44:00.886543+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:05.967822+0000","scrub_duration":206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,2],"acting":[8,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.0","version":"1137'15251","reported_seq":15542,"reported_epoch":1137,"state":"active+clean","last_fresh":"2024-03-20T09:44:10.964455+0000","last_change":"2024-03-20T09:43:12.972823+0000","last_active":"2024-03-20T09:44:10.964455+0000","last_peered":"2024-03-20T09:44:10.964455+0000","last_clean":"2024-03-20T09:44:10.964455+0000","last_became_active":"2024-03-20T09:40:24.790001+0000","last_became_peered":"2024-03-20T09:40:24.790001+0000","last_unstale":"2024-03-20T09:44:10.964455+0000","last_undegraded":"2024-03-20T09:44:10.964455+0000","last_fullsized":"2024-03-20T09:44:10.964455+0000","mapping_epoch":1066,"log_start":"1083'5200","ondisk_log_start":"1083'5200","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1120'12079","last_scrub_stamp":"2024-03-20T09:43:12.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:12.972570+0000","objects_scrubbed":4832,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:35.233515+0000","scrub_duration":1243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106758144,"num_objects":1629,"num_object_clones":0,"num_object_copies":4887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15251,"num_write_kb":540160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,11],"acting":[6,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"20'137","reported_seq":1048,"reported_epoch":1138,"state":"active+clean","last_fresh":"2024-03-20T09:44:12.324939+0000","last_change":"2024-03-20T09:43:17.393179+0000","last_active":"2024-03-20T09:44:12.324939+0000","last_peered":"2024-03-20T09:44:12.324939+0000","last_clean":"2024-03-20T09:44:12.324939+0000","last_became_active":"2024-03-20T09:35:28.000367+0000","last_became_peered":"2024-03-20T09:35:28.000367+0000","last_unstale":"2024-03-20T09:44:12.324939+0000","last_undegraded":"2024-03-20T09:44:12.324939+0000","last_fullsized":"2024-03-20T09:44:12.324939+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":966,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:17.393079+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:37:01.184206+0000","last_clean_scrub_stamp":"2024-03-20T09:43:17.393079+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:36.547547+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"1137'17116","reported_seq":17508,"reported_epoch":1137,"state":"active+clean","last_fresh":"2024-03-20T09:44:10.163979+0000","last_change":"2024-03-20T09:43:58.187624+0000","last_active":"2024-03-20T09:44:10.163979+0000","last_peered":"2024-03-20T09:44:10.163979+0000","last_clean":"2024-03-20T09:44:10.163979+0000","last_became_active":"2024-03-20T09:40:24.791005+0000","last_became_peered":"2024-03-20T09:40:24.791005+0000","last_unstale":"2024-03-20T09:44:10.163979+0000","last_undegraded":"2024-03-20T09:44:10.163979+0000","last_fullsized":"2024-03-20T09:44:10.163979+0000","mapping_epoch":1066,"log_start":"1093'7100","ondisk_log_start":"1093'7100","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1133'16468","last_scrub_stamp":"2024-03-20T09:43:58.187354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:58.187354+0000","objects_scrubbed":2257,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:00.972878+0000","scrub_duration":557,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104660992,"num_objects":1597,"num_object_clones":0,"num_object_copies":4791,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14657,"num_write_kb":520128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"20'137","reported_seq":2506,"reported_epoch":1138,"state":"active+clean","last_fresh":"2024-03-20T09:44:12.127378+0000","last_change":"2024-03-20T09:43:13.063995+0000","last_active":"2024-03-20T09:44:12.127378+0000","last_peered":"2024-03-20T09:44:12.127378+0000","last_clean":"2024-03-20T09:44:12.127378+0000","last_became_active":"2024-03-20T09:20:25.341750+0000","last_became_peered":"2024-03-20T09:20:25.341750+0000","last_unstale":"2024-03-20T09:44:12.127378+0000","last_undegraded":"2024-03-20T09:44:12.127378+0000","last_fullsized":"2024-03-20T09:44:12.127378+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:13.063918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:13.063918+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:22.184587+0000","scrub_duration":184,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,2],"acting":[9,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.2","version":"1137'14944","reported_seq":15238,"reported_epoch":1137,"state":"active+clean","last_fresh":"2024-03-20T09:44:10.091149+0000","last_change":"2024-03-20T09:43:09.666385+0000","last_active":"2024-03-20T09:44:10.091149+0000","last_peered":"2024-03-20T09:44:10.091149+0000","last_clean":"2024-03-20T09:44:10.091149+0000","last_became_active":"2024-03-20T09:40:24.790265+0000","last_became_peered":"2024-03-20T09:40:24.790265+0000","last_unstale":"2024-03-20T09:44:10.091149+0000","last_undegraded":"2024-03-20T09:44:10.091149+0000","last_fullsized":"2024-03-20T09:44:10.091149+0000","mapping_epoch":1066,"log_start":"1083'4900","ondisk_log_start":"1083'4900","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'11638","last_scrub_stamp":"2024-03-20T09:43:09.666260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:09.666260+0000","objects_scrubbed":4972,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:35.499687+0000","scrub_duration":1058,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107872256,"num_objects":1646,"num_object_clones":0,"num_object_copies":4938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14944,"num_write_kb":530880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,9],"acting":[3,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"20'137","reported_seq":1188,"reported_epoch":1138,"state":"active+clean","last_fresh":"2024-03-20T09:44:12.128760+0000","last_change":"2024-03-20T09:43:04.893745+0000","last_active":"2024-03-20T09:44:12.128760+0000","last_peered":"2024-03-20T09:44:12.128760+0000","last_clean":"2024-03-20T09:44:12.128760+0000","last_became_active":"2024-03-20T09:24:16.444232+0000","last_became_peered":"2024-03-20T09:24:16.444232+0000","last_unstale":"2024-03-20T09:44:12.128760+0000","last_undegraded":"2024-03-20T09:44:12.128760+0000","last_fullsized":"2024-03-20T09:44:12.128760+0000","mapping_epoch":718,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:04.893646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:04.893646+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.c","version":"1137'14537","reported_seq":14826,"reported_epoch":1137,"state":"active+clean","last_fresh":"2024-03-20T09:44:10.153978+0000","last_change":"2024-03-20T09:43:06.941039+0000","last_active":"2024-03-20T09:44:10.153978+0000","last_peered":"2024-03-20T09:44:10.153978+0000","last_clean":"2024-03-20T09:44:10.153978+0000","last_became_active":"2024-03-20T09:40:24.791734+0000","last_became_peered":"2024-03-20T09:40:24.791734+0000","last_unstale":"2024-03-20T09:44:10.153978+0000","last_undegraded":"2024-03-20T09:44:10.153978+0000","last_fullsized":"2024-03-20T09:44:10.153978+0000","mapping_epoch":1066,"log_start":"1081'4500","ondisk_log_start":"1081'4500","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1117'11282","last_scrub_stamp":"2024-03-20T09:43:06.940719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:06.940719+0000","objects_scrubbed":4972,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:33.258717+0000","scrub_duration":1275,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110297088,"num_objects":1683,"num_object_clones":0,"num_object_copies":5049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14537,"num_write_kb":519040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,9],"acting":[3,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"20'137","reported_seq":2433,"reported_epoch":1137,"state":"active+clean","last_fresh":"2024-03-20T09:44:10.095458+0000","last_change":"2024-03-20T09:43:18.087277+0000","last_active":"2024-03-20T09:44:10.095458+0000","last_peered":"2024-03-20T09:44:10.095458+0000","last_clean":"2024-03-20T09:44:10.095458+0000","last_became_active":"2024-03-20T09:12:14.306384+0000","last_became_peered":"2024-03-20T09:12:14.306384+0000","last_unstale":"2024-03-20T09:44:10.095458+0000","last_undegraded":"2024-03-20T09:44:10.095458+0000","last_fullsized":"2024-03-20T09:44:10.095458+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:18.087165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:18.087165+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:46.885376+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.d","version":"1137'14842","reported_seq":15135,"reported_epoch":1137,"state":"active+clean","last_fresh":"2024-03-20T09:44:10.466897+0000","last_change":"2024-03-20T09:43:18.652174+0000","last_active":"2024-03-20T09:44:10.466897+0000","last_peered":"2024-03-20T09:44:10.466897+0000","last_clean":"2024-03-20T09:44:10.466897+0000","last_became_active":"2024-03-20T09:40:24.812843+0000","last_became_peered":"2024-03-20T09:40:24.812843+0000","last_unstale":"2024-03-20T09:44:10.466897+0000","last_undegraded":"2024-03-20T09:44:10.466897+0000","last_fullsized":"2024-03-20T09:44:10.466897+0000","mapping_epoch":1066,"log_start":"1083'4800","ondisk_log_start":"1083'4800","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1122'12042","last_scrub_stamp":"2024-03-20T09:43:18.651987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:18.651987+0000","objects_scrubbed":4624,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:48.073907+0000","scrub_duration":1636,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116457496,"num_objects":1778,"num_object_clones":0,"num_object_copies":5334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1778,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14842,"num_write_kb":531777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"20'137","reported_seq":1092,"reported_epoch":1136,"state":"active+clean","last_fresh":"2024-03-20T09:44:05.049532+0000","last_change":"2024-03-20T09:43:15.623506+0000","last_active":"2024-03-20T09:44:05.049532+0000","last_peered":"2024-03-20T09:44:05.049532+0000","last_clean":"2024-03-20T09:44:05.049532+0000","last_became_active":"2024-03-20T09:35:19.820895+0000","last_became_peered":"2024-03-20T09:35:19.820895+0000","last_unstale":"2024-03-20T09:44:05.049532+0000","last_undegraded":"2024-03-20T09:44:05.049532+0000","last_fullsized":"2024-03-20T09:44:05.049532+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:15.623424+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:19:03.805185+0000","last_clean_scrub_stamp":"2024-03-20T09:43:15.623424+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:28.959946+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"1136'14917","reported_seq":15204,"reported_epoch":1136,"state":"active+clean","last_fresh":"2024-03-20T09:44:09.155147+0000","last_change":"2024-03-20T09:43:12.887942+0000","last_active":"2024-03-20T09:44:09.155147+0000","last_peered":"2024-03-20T09:44:09.155147+0000","last_clean":"2024-03-20T09:44:09.155147+0000","last_became_active":"2024-03-20T09:40:24.813215+0000","last_became_peered":"2024-03-20T09:40:24.813215+0000","last_unstale":"2024-03-20T09:44:09.155147+0000","last_undegraded":"2024-03-20T09:44:09.155147+0000","last_fullsized":"2024-03-20T09:44:09.155147+0000","mapping_epoch":1066,"log_start":"1083'4900","ondisk_log_start":"1083'4900","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1120'11793","last_scrub_stamp":"2024-03-20T09:43:12.887835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:12.887835+0000","objects_scrubbed":4853,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:14.822931+0000","scrub_duration":1311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111476736,"num_objects":1701,"num_object_clones":0,"num_object_copies":5103,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14917,"num_write_kb":531776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,1],"acting":[10,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"20'137","reported_seq":1962,"reported_epoch":1136,"state":"active+clean","last_fresh":"2024-03-20T09:44:05.051343+0000","last_change":"2024-03-20T09:43:14.612155+0000","last_active":"2024-03-20T09:44:05.051343+0000","last_peered":"2024-03-20T09:44:05.051343+0000","last_clean":"2024-03-20T09:44:05.051343+0000","last_became_active":"2024-03-20T09:35:19.822321+0000","last_became_peered":"2024-03-20T09:35:19.822321+0000","last_unstale":"2024-03-20T09:44:05.051343+0000","last_undegraded":"2024-03-20T09:44:05.051343+0000","last_fullsized":"2024-03-20T09:44:05.051343+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:14.612037+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:24:31.549603+0000","last_clean_scrub_stamp":"2024-03-20T09:43:14.612037+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:21.810461+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.b","version":"1137'14844","reported_seq":15131,"reported_epoch":1137,"state":"active+clean","last_fresh":"2024-03-20T09:44:11.863522+0000","last_change":"2024-03-20T09:43:21.410182+0000","last_active":"2024-03-20T09:44:11.863522+0000","last_peered":"2024-03-20T09:44:11.863522+0000","last_clean":"2024-03-20T09:44:11.863522+0000","last_became_active":"2024-03-20T09:40:24.791110+0000","last_became_peered":"2024-03-20T09:40:24.791110+0000","last_unstale":"2024-03-20T09:44:11.863522+0000","last_undegraded":"2024-03-20T09:44:11.863522+0000","last_fullsized":"2024-03-20T09:44:11.863522+0000","mapping_epoch":1066,"log_start":"1083'4800","ondisk_log_start":"1083'4800","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1122'12128","last_scrub_stamp":"2024-03-20T09:43:21.409950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:21.409950+0000","objects_scrubbed":4326,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:49.041623+0000","scrub_duration":1397,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103153664,"num_objects":1574,"num_object_clones":0,"num_object_copies":4722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14844,"num_write_kb":525376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"20'137","reported_seq":1499,"reported_epoch":1137,"state":"active+clean","last_fresh":"2024-03-20T09:44:10.093993+0000","last_change":"2024-03-20T09:43:10.005346+0000","last_active":"2024-03-20T09:44:10.093993+0000","last_peered":"2024-03-20T09:44:10.093993+0000","last_clean":"2024-03-20T09:44:10.093993+0000","last_became_active":"2024-03-20T09:31:07.424588+0000","last_became_peered":"2024-03-20T09:31:07.424588+0000","last_unstale":"2024-03-20T09:44:10.093993+0000","last_undegraded":"2024-03-20T09:44:10.093993+0000","last_fullsized":"2024-03-20T09:44:10.093993+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:10.005211+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:30:47.812125+0000","last_clean_scrub_stamp":"2024-03-20T09:43:10.005211+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:10.107177+0000","scrub_duration":221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"1138'16988","reported_seq":17355,"reported_epoch":1138,"state":"active+clean","last_fresh":"2024-03-20T09:44:13.923465+0000","last_change":"2024-03-20T09:43:57.525315+0000","last_active":"2024-03-20T09:44:13.923465+0000","last_peered":"2024-03-20T09:44:13.923465+0000","last_clean":"2024-03-20T09:44:13.923465+0000","last_became_active":"2024-03-20T09:40:24.790197+0000","last_became_peered":"2024-03-20T09:40:24.790197+0000","last_unstale":"2024-03-20T09:44:13.923465+0000","last_undegraded":"2024-03-20T09:44:13.923465+0000","last_fullsized":"2024-03-20T09:44:13.923465+0000","mapping_epoch":1066,"log_start":"1090'6900","ondisk_log_start":"1090'6900","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1133'16101","last_scrub_stamp":"2024-03-20T09:43:57.524948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:57.524948+0000","objects_scrubbed":2292,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:15.662531+0000","scrub_duration":556,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91357184,"num_objects":1394,"num_object_clones":0,"num_object_copies":4182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15036,"num_write_kb":525760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":2404,"reported_epoch":1137,"state":"active+clean","last_fresh":"2024-03-20T09:44:10.099055+0000","last_change":"2024-03-20T09:43:08.050620+0000","last_active":"2024-03-20T09:44:10.099055+0000","last_peered":"2024-03-20T09:44:10.099055+0000","last_clean":"2024-03-20T09:44:10.099055+0000","last_became_active":"2024-03-20T09:12:14.305989+0000","last_became_peered":"2024-03-20T09:12:14.305989+0000","last_unstale":"2024-03-20T09:44:10.099055+0000","last_undegraded":"2024-03-20T09:44:10.099055+0000","last_fullsized":"2024-03-20T09:44:10.099055+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:08.050532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:08.050532+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:21.415537+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.a","version":"1137'14839","reported_seq":15127,"reported_epoch":1137,"state":"active+clean","last_fresh":"2024-03-20T09:44:10.434284+0000","last_change":"2024-03-20T09:43:08.178211+0000","last_active":"2024-03-20T09:44:10.434284+0000","last_peered":"2024-03-20T09:44:10.434284+0000","last_clean":"2024-03-20T09:44:10.434284+0000","last_became_active":"2024-03-20T09:40:24.812882+0000","last_became_peered":"2024-03-20T09:40:24.812882+0000","last_unstale":"2024-03-20T09:44:10.434284+0000","last_undegraded":"2024-03-20T09:44:10.434284+0000","last_fullsized":"2024-03-20T09:44:10.434284+0000","mapping_epoch":1066,"log_start":"1083'4800","ondisk_log_start":"1083'4800","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1117'11503","last_scrub_stamp":"2024-03-20T09:43:08.177809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:08.177809+0000","objects_scrubbed":4944,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:34.330102+0000","scrub_duration":1251,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103743488,"num_objects":1583,"num_object_clones":0,"num_object_copies":4749,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14839,"num_write_kb":525504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"20'137","reported_seq":1806,"reported_epoch":1137,"state":"active+clean","last_fresh":"2024-03-20T09:44:10.086262+0000","last_change":"2024-03-20T09:43:57.239285+0000","last_active":"2024-03-20T09:44:10.086262+0000","last_peered":"2024-03-20T09:44:10.086262+0000","last_clean":"2024-03-20T09:44:10.086262+0000","last_became_active":"2024-03-20T09:39:17.944040+0000","last_became_peered":"2024-03-20T09:39:17.944040+0000","last_unstale":"2024-03-20T09:44:10.086262+0000","last_undegraded":"2024-03-20T09:44:10.086262+0000","last_fullsized":"2024-03-20T09:44:10.086262+0000","mapping_epoch":1041,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":1042,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:57.239132+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:42:42.369236+0000","last_clean_scrub_stamp":"2024-03-20T09:43:57.239132+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:11.927324+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,8],"acting":[4,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.9","version":"1136'14825","reported_seq":15117,"reported_epoch":1136,"state":"active+clean","last_fresh":"2024-03-20T09:44:09.191787+0000","last_change":"2024-03-20T09:43:09.692400+0000","last_active":"2024-03-20T09:44:09.191787+0000","last_peered":"2024-03-20T09:44:09.191787+0000","last_clean":"2024-03-20T09:44:09.191787+0000","last_became_active":"2024-03-20T09:40:24.792650+0000","last_became_peered":"2024-03-20T09:40:24.792650+0000","last_unstale":"2024-03-20T09:44:09.191787+0000","last_undegraded":"2024-03-20T09:44:09.191787+0000","last_fullsized":"2024-03-20T09:44:09.191787+0000","mapping_epoch":1066,"log_start":"1083'4800","ondisk_log_start":"1083'4800","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'11635","last_scrub_stamp":"2024-03-20T09:43:09.692199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:09.692199+0000","objects_scrubbed":4930,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:31.593123+0000","scrub_duration":1148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112787456,"num_objects":1721,"num_object_clones":0,"num_object_copies":5163,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14825,"num_write_kb":529472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"20'137","reported_seq":1845,"reported_epoch":1137,"state":"active+clean","last_fresh":"2024-03-20T09:44:10.095543+0000","last_change":"2024-03-20T09:43:06.796634+0000","last_active":"2024-03-20T09:44:10.095543+0000","last_peered":"2024-03-20T09:44:10.095543+0000","last_clean":"2024-03-20T09:44:10.095543+0000","last_became_active":"2024-03-20T09:15:34.225022+0000","last_became_peered":"2024-03-20T09:15:34.225022+0000","last_unstale":"2024-03-20T09:44:10.095543+0000","last_undegraded":"2024-03-20T09:44:10.095543+0000","last_fullsized":"2024-03-20T09:44:10.095543+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:06.796525+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:20:12.169658+0000","last_clean_scrub_stamp":"2024-03-20T09:43:06.796525+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:36.395280+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,5],"acting":[6,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"1137'14948","reported_seq":15239,"reported_epoch":1137,"state":"active+clean","last_fresh":"2024-03-20T09:44:10.442827+0000","last_change":"2024-03-20T09:43:13.370701+0000","last_active":"2024-03-20T09:44:10.442827+0000","last_peered":"2024-03-20T09:44:10.442827+0000","last_clean":"2024-03-20T09:44:10.442827+0000","last_became_active":"2024-03-20T09:40:24.790189+0000","last_became_peered":"2024-03-20T09:40:24.790189+0000","last_unstale":"2024-03-20T09:44:10.442827+0000","last_undegraded":"2024-03-20T09:44:10.442827+0000","last_fullsized":"2024-03-20T09:44:10.442827+0000","mapping_epoch":1066,"log_start":"1083'4900","ondisk_log_start":"1083'4900","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1120'11823","last_scrub_stamp":"2024-03-20T09:43:13.370340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:13.370340+0000","objects_scrubbed":4782,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:20.684399+0000","scrub_duration":1344,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106168320,"num_objects":1620,"num_object_clones":0,"num_object_copies":4860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14948,"num_write_kb":530176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"20'137","reported_seq":2401,"reported_epoch":1137,"state":"active+clean","last_fresh":"2024-03-20T09:44:10.094386+0000","last_change":"2024-03-20T09:43:11.044649+0000","last_active":"2024-03-20T09:44:10.094386+0000","last_peered":"2024-03-20T09:44:10.094386+0000","last_clean":"2024-03-20T09:44:10.094386+0000","last_became_active":"2024-03-20T09:35:19.820396+0000","last_became_peered":"2024-03-20T09:35:19.820396+0000","last_unstale":"2024-03-20T09:44:10.094386+0000","last_undegraded":"2024-03-20T09:44:10.094386+0000","last_fullsized":"2024-03-20T09:44:10.094386+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:11.044527+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:14:10.560890+0000","last_clean_scrub_stamp":"2024-03-20T09:43:11.044527+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:19.618048+0000","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"1138'15148","reported_seq":15436,"reported_epoch":1138,"state":"active+clean","last_fresh":"2024-03-20T09:44:13.869286+0000","last_change":"2024-03-20T09:43:09.360089+0000","last_active":"2024-03-20T09:44:13.869286+0000","last_peered":"2024-03-20T09:44:13.869286+0000","last_clean":"2024-03-20T09:44:13.869286+0000","last_became_active":"2024-03-20T09:40:24.791199+0000","last_became_peered":"2024-03-20T09:40:24.791199+0000","last_unstale":"2024-03-20T09:44:13.869286+0000","last_undegraded":"2024-03-20T09:44:13.869286+0000","last_fullsized":"2024-03-20T09:44:13.869286+0000","mapping_epoch":1066,"log_start":"1083'5100","ondisk_log_start":"1083'5100","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'11689","last_scrub_stamp":"2024-03-20T09:43:09.359969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:09.359969+0000","objects_scrubbed":4954,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:28.242282+0000","scrub_duration":1552,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":1472,"num_object_clones":0,"num_object_copies":4416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15148,"num_write_kb":531840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,5],"acting":[9,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"20'137","reported_seq":2233,"reported_epoch":1137,"state":"active+clean","last_fresh":"2024-03-20T09:44:10.095494+0000","last_change":"2024-03-20T09:43:50.466520+0000","last_active":"2024-03-20T09:44:10.095494+0000","last_peered":"2024-03-20T09:44:10.095494+0000","last_clean":"2024-03-20T09:44:10.095494+0000","last_became_active":"2024-03-20T09:39:17.940497+0000","last_became_peered":"2024-03-20T09:39:17.940497+0000","last_unstale":"2024-03-20T09:44:10.095494+0000","last_undegraded":"2024-03-20T09:44:10.095494+0000","last_fullsized":"2024-03-20T09:44:10.095494+0000","mapping_epoch":1041,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":1042,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:50.466383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:50.466383+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:05.247322+0000","scrub_duration":307,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,2],"acting":[11,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.6","version":"1137'17851","reported_seq":18287,"reported_epoch":1137,"state":"active+clean","last_fresh":"2024-03-20T09:44:10.451901+0000","last_change":"2024-03-20T09:43:46.116088+0000","last_active":"2024-03-20T09:44:10.451901+0000","last_peered":"2024-03-20T09:44:10.451901+0000","last_clean":"2024-03-20T09:44:10.451901+0000","last_became_active":"2024-03-20T09:40:24.792369+0000","last_became_peered":"2024-03-20T09:40:24.792369+0000","last_unstale":"2024-03-20T09:44:10.451901+0000","last_undegraded":"2024-03-20T09:44:10.451901+0000","last_fullsized":"2024-03-20T09:44:10.451901+0000","mapping_epoch":1066,"log_start":"1096'7800","ondisk_log_start":"1096'7800","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1131'16486","last_scrub_stamp":"2024-03-20T09:43:46.115894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:46.115894+0000","objects_scrubbed":3011,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:57.198816+0000","scrub_duration":1002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105644032,"num_objects":1612,"num_object_clones":0,"num_object_copies":4836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14818,"num_write_kb":525760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"20'137","reported_seq":1657,"reported_epoch":1137,"state":"active+clean","last_fresh":"2024-03-20T09:44:10.086666+0000","last_change":"2024-03-20T09:43:04.334350+0000","last_active":"2024-03-20T09:44:10.086666+0000","last_peered":"2024-03-20T09:44:10.086666+0000","last_clean":"2024-03-20T09:44:10.086666+0000","last_became_active":"2024-03-20T09:19:54.506134+0000","last_became_peered":"2024-03-20T09:19:54.506134+0000","last_unstale":"2024-03-20T09:44:10.086666+0000","last_undegraded":"2024-03-20T09:44:10.086666+0000","last_fullsized":"2024-03-20T09:44:10.086666+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:04.334199+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:29:53.362224+0000","last_clean_scrub_stamp":"2024-03-20T09:43:04.334199+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:15.046031+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"1136'15016","reported_seq":15293,"reported_epoch":1136,"state":"active+clean","last_fresh":"2024-03-20T09:44:09.142294+0000","last_change":"2024-03-20T09:43:20.856797+0000","last_active":"2024-03-20T09:44:09.142294+0000","last_peered":"2024-03-20T09:44:09.142294+0000","last_clean":"2024-03-20T09:44:09.142294+0000","last_became_active":"2024-03-20T09:40:24.813246+0000","last_became_peered":"2024-03-20T09:40:24.813246+0000","last_unstale":"2024-03-20T09:44:09.142294+0000","last_undegraded":"2024-03-20T09:44:09.142294+0000","last_fullsized":"2024-03-20T09:44:09.142294+0000","mapping_epoch":1066,"log_start":"1083'5000","ondisk_log_start":"1083'5000","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1122'12369","last_scrub_stamp":"2024-03-20T09:43:20.856432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:20.856432+0000","objects_scrubbed":4430,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:43.625443+0000","scrub_duration":1246,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114556928,"num_objects":1748,"num_object_clones":0,"num_object_copies":5244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15016,"num_write_kb":536448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,3],"acting":[10,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"20'137","reported_seq":2421,"reported_epoch":1137,"state":"active+clean","last_fresh":"2024-03-20T09:44:10.094296+0000","last_change":"2024-03-20T09:43:50.972722+0000","last_active":"2024-03-20T09:44:10.094296+0000","last_peered":"2024-03-20T09:44:10.094296+0000","last_clean":"2024-03-20T09:44:10.094296+0000","last_became_active":"2024-03-20T09:19:54.524397+0000","last_became_peered":"2024-03-20T09:19:54.524397+0000","last_unstale":"2024-03-20T09:44:10.094296+0000","last_undegraded":"2024-03-20T09:44:10.094296+0000","last_fullsized":"2024-03-20T09:44:10.094296+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:50.972601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:50.972601+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:58.813078+0000","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"1137'14604","reported_seq":14889,"reported_epoch":1137,"state":"active+clean","last_fresh":"2024-03-20T09:44:10.809995+0000","last_change":"2024-03-20T09:43:09.459247+0000","last_active":"2024-03-20T09:44:10.809995+0000","last_peered":"2024-03-20T09:44:10.809995+0000","last_clean":"2024-03-20T09:44:10.809995+0000","last_became_active":"2024-03-20T09:40:24.789539+0000","last_became_peered":"2024-03-20T09:40:24.789539+0000","last_unstale":"2024-03-20T09:44:10.809995+0000","last_undegraded":"2024-03-20T09:44:10.809995+0000","last_fullsized":"2024-03-20T09:44:10.809995+0000","mapping_epoch":1066,"log_start":"1083'4600","ondisk_log_start":"1083'4600","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'11421","last_scrub_stamp":"2024-03-20T09:43:09.459059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:09.459059+0000","objects_scrubbed":4816,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:29.439861+0000","scrub_duration":1236,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105775104,"num_objects":1614,"num_object_clones":0,"num_object_copies":4842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14604,"num_write_kb":518976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.c","version":"570'138","reported_seq":2412,"reported_epoch":1137,"state":"active+clean","last_fresh":"2024-03-20T09:44:10.091153+0000","last_change":"2024-03-20T09:43:07.728946+0000","last_active":"2024-03-20T09:44:10.091153+0000","last_peered":"2024-03-20T09:44:10.091153+0000","last_clean":"2024-03-20T09:44:10.091153+0000","last_became_active":"2024-03-20T09:39:17.943381+0000","last_became_peered":"2024-03-20T09:39:17.943381+0000","last_unstale":"2024-03-20T09:44:10.091153+0000","last_undegraded":"2024-03-20T09:44:10.091153+0000","last_fullsized":"2024-03-20T09:44:10.091153+0000","mapping_epoch":1041,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":1042,"parent":"0.0","parent_split_bits":4,"last_scrub":"570'138","last_scrub_stamp":"2024-03-20T09:43:07.728813+0000","last_deep_scrub":"570'138","last_deep_scrub_stamp":"2024-03-20T09:16:13.725035+0000","last_clean_scrub_stamp":"2024-03-20T09:43:07.728813+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:26.795166+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,9],"acting":[3,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"1137'15165","reported_seq":15457,"reported_epoch":1137,"state":"active+clean","last_fresh":"2024-03-20T09:44:11.136365+0000","last_change":"2024-03-20T09:43:20.493432+0000","last_active":"2024-03-20T09:44:11.136365+0000","last_peered":"2024-03-20T09:44:11.136365+0000","last_clean":"2024-03-20T09:44:11.136365+0000","last_became_active":"2024-03-20T09:40:24.813276+0000","last_became_peered":"2024-03-20T09:40:24.813276+0000","last_unstale":"2024-03-20T09:44:11.136365+0000","last_undegraded":"2024-03-20T09:44:11.136365+0000","last_fullsized":"2024-03-20T09:44:11.136365+0000","mapping_epoch":1066,"log_start":"1083'5100","ondisk_log_start":"1083'5100","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1122'12403","last_scrub_stamp":"2024-03-20T09:43:20.493115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:20.493115+0000","objects_scrubbed":4492,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:21.692329+0000","scrub_duration":1417,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110952448,"num_objects":1693,"num_object_clones":0,"num_object_copies":5079,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15165,"num_write_kb":539456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"20'137","reported_seq":1971,"reported_epoch":1137,"state":"active+clean","last_fresh":"2024-03-20T09:44:10.086563+0000","last_change":"2024-03-20T09:43:46.440102+0000","last_active":"2024-03-20T09:44:10.086563+0000","last_peered":"2024-03-20T09:44:10.086563+0000","last_clean":"2024-03-20T09:44:10.086563+0000","last_became_active":"2024-03-20T09:19:54.502529+0000","last_became_peered":"2024-03-20T09:19:54.502529+0000","last_unstale":"2024-03-20T09:44:10.086563+0000","last_undegraded":"2024-03-20T09:44:10.086563+0000","last_fullsized":"2024-03-20T09:44:10.086563+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:46.439952+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:41:52.574619+0000","last_clean_scrub_stamp":"2024-03-20T09:43:46.439952+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:05.231528+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"20'137","reported_seq":2252,"reported_epoch":1137,"state":"active+clean","last_fresh":"2024-03-20T09:44:10.092862+0000","last_change":"2024-03-20T09:43:15.202936+0000","last_active":"2024-03-20T09:44:10.092862+0000","last_peered":"2024-03-20T09:44:10.092862+0000","last_clean":"2024-03-20T09:44:10.092862+0000","last_became_active":"2024-03-20T09:10:34.096758+0000","last_became_peered":"2024-03-20T09:10:34.096758+0000","last_unstale":"2024-03-20T09:44:10.092862+0000","last_undegraded":"2024-03-20T09:44:10.092862+0000","last_fullsized":"2024-03-20T09:44:10.092862+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:15.202813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:15.202813+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:35.894669+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"20'137","reported_seq":1012,"reported_epoch":1137,"state":"active+clean","last_fresh":"2024-03-20T09:44:10.095594+0000","last_change":"2024-03-20T09:43:15.359792+0000","last_active":"2024-03-20T09:44:10.095594+0000","last_peered":"2024-03-20T09:44:10.095594+0000","last_clean":"2024-03-20T09:44:10.095594+0000","last_became_active":"2024-03-20T09:35:19.822242+0000","last_became_peered":"2024-03-20T09:35:19.822242+0000","last_unstale":"2024-03-20T09:44:10.095594+0000","last_undegraded":"2024-03-20T09:44:10.095594+0000","last_fullsized":"2024-03-20T09:44:10.095594+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:15.359586+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:37:52.359932+0000","last_clean_scrub_stamp":"2024-03-20T09:43:15.359586+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:29.699947+0000","scrub_duration":298,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.10","version":"20'137","reported_seq":1823,"reported_epoch":1137,"state":"active+clean","last_fresh":"2024-03-20T09:44:10.096147+0000","last_change":"2024-03-20T09:43:48.137633+0000","last_active":"2024-03-20T09:44:10.096147+0000","last_peered":"2024-03-20T09:44:10.096147+0000","last_clean":"2024-03-20T09:44:10.096147+0000","last_became_active":"2024-03-20T09:35:19.820595+0000","last_became_peered":"2024-03-20T09:35:19.820595+0000","last_unstale":"2024-03-20T09:44:10.096147+0000","last_undegraded":"2024-03-20T09:44:10.096147+0000","last_fullsized":"2024-03-20T09:44:10.096147+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:48.137502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:48.137502+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:17.764144+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"20'137","reported_seq":610,"reported_epoch":1137,"state":"active+clean","last_fresh":"2024-03-20T09:44:10.092373+0000","last_change":"2024-03-20T09:43:09.196572+0000","last_active":"2024-03-20T09:44:10.092373+0000","last_peered":"2024-03-20T09:44:10.092373+0000","last_clean":"2024-03-20T09:44:10.092373+0000","last_became_active":"2024-03-20T09:39:17.943385+0000","last_became_peered":"2024-03-20T09:39:17.943385+0000","last_unstale":"2024-03-20T09:44:10.092373+0000","last_undegraded":"2024-03-20T09:44:10.092373+0000","last_fullsized":"2024-03-20T09:44:10.092373+0000","mapping_epoch":1041,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":1042,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:09.196479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:09.196479+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:34.758699+0000","scrub_duration":184,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"20'137","reported_seq":888,"reported_epoch":1137,"state":"active+clean","last_fresh":"2024-03-20T09:44:10.091195+0000","last_change":"2024-03-20T09:43:06.840465+0000","last_active":"2024-03-20T09:44:10.091195+0000","last_peered":"2024-03-20T09:44:10.091195+0000","last_clean":"2024-03-20T09:44:10.091195+0000","last_became_active":"2024-03-20T09:39:17.943337+0000","last_became_peered":"2024-03-20T09:39:17.943337+0000","last_unstale":"2024-03-20T09:44:10.091195+0000","last_undegraded":"2024-03-20T09:44:10.091195+0000","last_fullsized":"2024-03-20T09:44:10.091195+0000","mapping_epoch":1041,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":1042,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:06.840344+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:35:27.264203+0000","last_clean_scrub_stamp":"2024-03-20T09:43:06.840344+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:31.325871+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,11],"acting":[3,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"20'137","reported_seq":2393,"reported_epoch":1137,"state":"active+clean","last_fresh":"2024-03-20T09:44:10.092631+0000","last_change":"2024-03-20T09:43:12.288008+0000","last_active":"2024-03-20T09:44:10.092631+0000","last_peered":"2024-03-20T09:44:10.092631+0000","last_clean":"2024-03-20T09:44:10.092631+0000","last_became_active":"2024-03-20T09:35:19.822429+0000","last_became_peered":"2024-03-20T09:35:19.822429+0000","last_unstale":"2024-03-20T09:44:10.092631+0000","last_undegraded":"2024-03-20T09:44:10.092631+0000","last_fullsized":"2024-03-20T09:44:10.092631+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:12.287881+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:38:01.419884+0000","last_clean_scrub_stamp":"2024-03-20T09:43:12.287881+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:27.962869+0000","scrub_duration":272,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"20'137","reported_seq":2426,"reported_epoch":1138,"state":"active+clean","last_fresh":"2024-03-20T09:44:12.324865+0000","last_change":"2024-03-20T09:43:51.249807+0000","last_active":"2024-03-20T09:44:12.324865+0000","last_peered":"2024-03-20T09:44:12.324865+0000","last_clean":"2024-03-20T09:44:12.324865+0000","last_became_active":"2024-03-20T09:20:25.349082+0000","last_became_peered":"2024-03-20T09:20:25.349082+0000","last_unstale":"2024-03-20T09:44:12.324865+0000","last_undegraded":"2024-03-20T09:44:12.324865+0000","last_fullsized":"2024-03-20T09:44:12.324865+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:51.249669+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:41:54.735667+0000","last_clean_scrub_stamp":"2024-03-20T09:43:51.249669+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:17.928220+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"20'137","reported_seq":1704,"reported_epoch":1137,"state":"active+clean","last_fresh":"2024-03-20T09:44:10.089909+0000","last_change":"2024-03-20T09:43:55.035201+0000","last_active":"2024-03-20T09:44:10.089909+0000","last_peered":"2024-03-20T09:44:10.089909+0000","last_clean":"2024-03-20T09:44:10.089909+0000","last_became_active":"2024-03-20T09:12:14.307581+0000","last_became_peered":"2024-03-20T09:12:14.307581+0000","last_unstale":"2024-03-20T09:44:10.089909+0000","last_undegraded":"2024-03-20T09:44:10.089909+0000","last_fullsized":"2024-03-20T09:44:10.089909+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:55.035080+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:34:48.079272+0000","last_clean_scrub_stamp":"2024-03-20T09:43:55.035080+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:02.488697+0000","scrub_duration":240,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"20'137","reported_seq":967,"reported_epoch":1137,"state":"active+clean","last_fresh":"2024-03-20T09:44:10.091083+0000","last_change":"2024-03-20T09:43:56.715773+0000","last_active":"2024-03-20T09:44:10.091083+0000","last_peered":"2024-03-20T09:44:10.091083+0000","last_clean":"2024-03-20T09:44:10.091083+0000","last_became_active":"2024-03-20T09:41:47.137425+0000","last_became_peered":"2024-03-20T09:41:47.137425+0000","last_unstale":"2024-03-20T09:44:10.091083+0000","last_undegraded":"2024-03-20T09:44:10.091083+0000","last_fullsized":"2024-03-20T09:44:10.091083+0000","mapping_epoch":1094,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":1095,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:56.715666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:56.715666+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:02.384244+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":1708130328,"num_objects":26065,"num_object_clones":0,"num_object_copies":78195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26065,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":238391,"num_write_kb":8462529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5114900480,"data_stored":5114888264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160635,"ondisk_log_size":160635,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1925120,"data_stored":1914464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4248,"ondisk_log_size":4248,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860995,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1529052,"kb_used_data":428880,"kb_used_omap":1,"kb_used_meta":1100158,"kb_avail":92216100,"statfs":{"total":95995035648,"available":94429286400,"internally_reserved":0,"allocated":439173120,"data_stored":438885410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":1126562244},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":861,"seq":3697966842019,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":874060,"kb_used_data":561220,"kb_used_omap":103,"kb_used_meta":312728,"kb_avail":92871092,"statfs":{"total":95995035648,"available":95099998208,"internally_reserved":0,"allocated":574689280,"data_stored":574406002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":106169,"internal_metadata":320233799},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":80,"seq":343597384272,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1316196,"kb_used_data":572828,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":92428956,"statfs":{"total":95995035648,"available":94647250944,"internally_reserved":0,"allocated":586575872,"data_stored":586283730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":761199040},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":70,"seq":300647711316,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":888600,"kb_used_data":203472,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":92856552,"statfs":{"total":95995035648,"available":95085109248,"internally_reserved":0,"allocated":208355328,"data_stored":208067618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":701495750},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":60,"seq":257698038360,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1555816,"kb_used_data":531868,"kb_used_omap":1,"kb_used_meta":1023806,"kb_avail":92189336,"statfs":{"total":95995035648,"available":94401880064,"internally_reserved":0,"allocated":544632832,"data_stored":544337498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":1048377802},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332701,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":978588,"kb_used_data":207440,"kb_used_omap":1,"kb_used_meta":771134,"kb_avail":92766564,"statfs":{"total":95995035648,"available":94992961536,"internally_reserved":0,"allocated":212418560,"data_stored":212130850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":789641670},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575479,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2066192,"kb_used_data":875784,"kb_used_omap":1,"kb_used_meta":1190398,"kb_avail":91678960,"statfs":{"total":95995035648,"available":93879255040,"internally_reserved":0,"allocated":896802816,"data_stored":896515466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1218968006},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738998,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411953,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":852592,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":92892560,"statfs":{"total":95995035648,"available":95121981440,"internally_reserved":0,"allocated":954368,"data_stored":658674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":872085957},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150317,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1662156,"kb_used_data":539780,"kb_used_omap":1,"kb_used_meta":1122366,"kb_avail":92082996,"statfs":{"total":95995035648,"available":94292987904,"internally_reserved":0,"allocated":552734720,"data_stored":552451442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1149303239},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438954091,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1022668,"kb_used_data":215428,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":92722484,"statfs":{"total":95995035648,"available":94947823616,"internally_reserved":0,"allocated":220598272,"data_stored":220314994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":826603973},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":443,"seq":1902670512532,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1944488,"kb_used_data":962012,"kb_used_omap":181,"kb_used_meta":982154,"kb_avail":91800664,"statfs":{"total":95995035648,"available":94003879936,"internally_reserved":0,"allocated":985100288,"data_stored":984804954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":186131,"internal_metadata":1005725933},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":877989888,"data_stored":877985816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":542441472,"data_stored":542441472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":213843968,"data_stored":213843968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":975966208,"data_stored":975962136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":209256448,"data_stored":209256448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":536612864,"data_stored":536608792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":205520896,"data_stored":205520896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":564264960,"data_stored":564264960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":563544064,"data_stored":563544064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":425459712,"data_stored":425459712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:44:16.008 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-20T09:44:16.008 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:44:16.008 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-03-20T09:44:16.009 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:44:16.956 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:44:17.272 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:44:17.580 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:44:17.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:17.583 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:44:17.650 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:44:17.684 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:44:17.694 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:44:17.720 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:44:17.747 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:17.750 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:44:17.780 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:44:17.811 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:44:17.844 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:44:17.878 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:44:17.911 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:44:17.919 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:17 smithi026 ceph-mon[68309]: pgmap v2769: 47 pgs: 47 active+clean; 1.4 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1.09k op/s 2024-03-20T09:44:17.919 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:17 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3400285493' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:44:17.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:17 smithi122 ceph-mon[73495]: pgmap v2769: 47 pgs: 47 active+clean; 1.4 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1.09k op/s 2024-03-20T09:44:17.937 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:17 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3400285493' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:44:17.967 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:44:17.999 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:44:18.034 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:44:18.040 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:44:18.040 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:44:18.041 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:44:18.041 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:44:18.041 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:44:18.041 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:44:18.041 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:44:18.041 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:44:18.041 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:44:18.041 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:44:18.041 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:44:18.041 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:44:18.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:17 smithi115 ceph-mon[72476]: pgmap v2769: 47 pgs: 47 active+clean; 1.4 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 1.09k op/s 2024-03-20T09:44:18.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:17 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3400285493' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:44:18.158 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:44:18.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:18.162 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:44:18.206 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_blocked_ops 2024-03-20T09:44:18.264 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:44:18.298 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:18.454 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:44:18.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:18.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:44:18.560 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:44:18.607 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:44:18.615 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:18.640 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:44:18.645 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:18.646 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:44:18.747 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:44:18.792 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:18.899 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:44:18.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:18.903 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:44:19.004 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:44:19.050 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:19.052 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:44:19.089 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:44:19.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:19.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:44:19.202 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:44:19.234 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:19.263 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:44:19.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:19.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:44:19.373 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:44:19.421 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:19.451 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:44:19.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:19.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:44:19.556 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:44:19.600 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:19.632 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:44:19.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:19.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:44:19.742 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:44:19.763 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:19 smithi115 ceph-mon[72476]: pgmap v2770: 47 pgs: 47 active+clean; 1.4 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 699 op/s 2024-03-20T09:44:19.785 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:19.846 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:44:19.850 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:19 smithi026 ceph-mon[68309]: pgmap v2770: 47 pgs: 47 active+clean; 1.4 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 699 op/s 2024-03-20T09:44:19.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:19.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:44:19.933 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:19 smithi122 ceph-mon[73495]: pgmap v2770: 47 pgs: 47 active+clean; 1.4 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 699 op/s 2024-03-20T09:44:19.957 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:44:19.995 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:20.034 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:44:20.038 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:20.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:44:20.139 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:44:20.186 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:20.210 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:44:20.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:20.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:44:20.322 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:44:20.388 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:20.433 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:44:20.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:20.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:44:20.537 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:44:20.603 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:20.637 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:44:20.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:20.644 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:44:20.673 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:44:20.673 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:44:20.673 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-03-20T09:44:20.673 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-03-20T09:44:20.673 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:44:20.708 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:44:20.711 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:44:20.744 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:44:20.787 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:20.826 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:44:20.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:20.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:44:20.931 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:44:20.998 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:21.037 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:44:21.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:21.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:44:21.046 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:44:21.142 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:44:21.204 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:21.238 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:44:21.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:21.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:44:21.342 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:44:21.379 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:21.595 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:44:21.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:21.599 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:44:21.700 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:44:21.739 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:21.872 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:44:21.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:21.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:44:21.925 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":2771,"stamp":"2024-03-20T09:44:20.707692+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1242825272,"num_objects":18958,"num_object_clones":0,"num_object_copies":56874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18958,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":245500,"num_write_kb":8462529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":165192,"ondisk_log_size":165192,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":13367364,"kb_used_data":3759836,"kb_used_omap":299,"kb_used_meta":9606676,"kb_avail":1017829308,"statfs":{"total":1055945392128,"available":1042257211392,"internally_reserved":0,"allocated":3850072064,"data_stored":3846786328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":306651,"internal_metadata":9837236773},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-502857728,"num_objects":-7673,"num_object_clones":0,"num_object_copies":-23019,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7673,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.595455"},"pg_stats":[{"pgid":"1.1e","version":"20'137","reported_seq":1769,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:14.151945+0000","last_change":"2024-03-20T09:43:09.164860+0000","last_active":"2024-03-20T09:44:14.151945+0000","last_peered":"2024-03-20T09:44:14.151945+0000","last_clean":"2024-03-20T09:44:14.151945+0000","last_became_active":"2024-03-20T09:13:20.071655+0000","last_became_peered":"2024-03-20T09:13:20.071655+0000","last_unstale":"2024-03-20T09:44:14.151945+0000","last_undegraded":"2024-03-20T09:44:14.151945+0000","last_fullsized":"2024-03-20T09:44:14.151945+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:09.164738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:09.164738+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:33.937151+0000","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'137","reported_seq":854,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:14.149211+0000","last_change":"2024-03-20T09:43:45.421863+0000","last_active":"2024-03-20T09:44:14.149211+0000","last_peered":"2024-03-20T09:44:14.149211+0000","last_clean":"2024-03-20T09:44:14.149211+0000","last_became_active":"2024-03-20T09:31:07.425574+0000","last_became_peered":"2024-03-20T09:31:07.425574+0000","last_unstale":"2024-03-20T09:44:14.149211+0000","last_undegraded":"2024-03-20T09:44:14.149211+0000","last_fullsized":"2024-03-20T09:44:14.149211+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:45.421742+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:29:02.585165+0000","last_clean_scrub_stamp":"2024-03-20T09:43:45.421742+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:55.396984+0000","scrub_duration":262,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2],"acting":[11,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1c","version":"20'137","reported_seq":2458,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:14.137773+0000","last_change":"2024-03-20T09:43:11.053490+0000","last_active":"2024-03-20T09:44:14.137773+0000","last_peered":"2024-03-20T09:44:14.137773+0000","last_clean":"2024-03-20T09:44:14.137773+0000","last_became_active":"2024-03-20T09:35:03.815532+0000","last_became_peered":"2024-03-20T09:35:03.815532+0000","last_unstale":"2024-03-20T09:44:14.137773+0000","last_undegraded":"2024-03-20T09:44:14.137773+0000","last_fullsized":"2024-03-20T09:44:14.137773+0000","mapping_epoch":950,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":951,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:11.053365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:11.053365+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:21.391918+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.19","version":"20'137","reported_seq":1642,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:14.142733+0000","last_change":"2024-03-20T09:43:44.857190+0000","last_active":"2024-03-20T09:44:14.142733+0000","last_peered":"2024-03-20T09:44:14.142733+0000","last_clean":"2024-03-20T09:44:14.142733+0000","last_became_active":"2024-03-20T09:39:17.953184+0000","last_became_peered":"2024-03-20T09:39:17.953184+0000","last_unstale":"2024-03-20T09:44:14.142733+0000","last_undegraded":"2024-03-20T09:44:14.142733+0000","last_fullsized":"2024-03-20T09:44:14.142733+0000","mapping_epoch":1041,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":1042,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:44.857069+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:30:56.201554+0000","last_clean_scrub_stamp":"2024-03-20T09:43:44.857069+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:06.289387+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,7],"acting":[10,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"20'137","reported_seq":1923,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:14.147512+0000","last_change":"2024-03-20T09:43:49.190805+0000","last_active":"2024-03-20T09:44:14.147512+0000","last_peered":"2024-03-20T09:44:14.147512+0000","last_clean":"2024-03-20T09:44:14.147512+0000","last_became_active":"2024-03-20T09:10:35.277481+0000","last_became_peered":"2024-03-20T09:10:35.277481+0000","last_unstale":"2024-03-20T09:44:14.147512+0000","last_undegraded":"2024-03-20T09:44:14.147512+0000","last_fullsized":"2024-03-20T09:44:14.147512+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:49.190689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:49.190689+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:03.907117+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1b","version":"20'137","reported_seq":2295,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:14.141604+0000","last_change":"2024-03-20T09:44:00.886678+0000","last_active":"2024-03-20T09:44:14.141604+0000","last_peered":"2024-03-20T09:44:14.141604+0000","last_clean":"2024-03-20T09:44:14.141604+0000","last_became_active":"2024-03-20T09:39:17.951408+0000","last_became_peered":"2024-03-20T09:39:17.951408+0000","last_unstale":"2024-03-20T09:44:14.141604+0000","last_undegraded":"2024-03-20T09:44:14.141604+0000","last_fullsized":"2024-03-20T09:44:14.141604+0000","mapping_epoch":1041,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":1042,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:44:00.886543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:44:00.886543+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:05.967822+0000","scrub_duration":206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,2],"acting":[8,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.0","version":"1139'15514","reported_seq":15809,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:15.963090+0000","last_change":"2024-03-20T09:43:12.972823+0000","last_active":"2024-03-20T09:44:15.963090+0000","last_peered":"2024-03-20T09:44:15.963090+0000","last_clean":"2024-03-20T09:44:15.963090+0000","last_became_active":"2024-03-20T09:40:24.790001+0000","last_became_peered":"2024-03-20T09:40:24.790001+0000","last_unstale":"2024-03-20T09:44:15.963090+0000","last_undegraded":"2024-03-20T09:44:15.963090+0000","last_fullsized":"2024-03-20T09:44:15.963090+0000","mapping_epoch":1066,"log_start":"1083'5500","ondisk_log_start":"1083'5500","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1120'12079","last_scrub_stamp":"2024-03-20T09:43:12.972570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:12.972570+0000","objects_scrubbed":4832,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:35.233515+0000","scrub_duration":1243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89522176,"num_objects":1366,"num_object_clones":0,"num_object_copies":4098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15514,"num_write_kb":540160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,11],"acting":[6,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"20'137","reported_seq":1050,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:14.142730+0000","last_change":"2024-03-20T09:43:17.393179+0000","last_active":"2024-03-20T09:44:14.142730+0000","last_peered":"2024-03-20T09:44:14.142730+0000","last_clean":"2024-03-20T09:44:14.142730+0000","last_became_active":"2024-03-20T09:35:28.000367+0000","last_became_peered":"2024-03-20T09:35:28.000367+0000","last_unstale":"2024-03-20T09:44:14.142730+0000","last_undegraded":"2024-03-20T09:44:14.142730+0000","last_fullsized":"2024-03-20T09:44:14.142730+0000","mapping_epoch":965,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":966,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:17.393079+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:37:01.184206+0000","last_clean_scrub_stamp":"2024-03-20T09:43:17.393079+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:36.547547+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"1139'17629","reported_seq":18026,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:20.178564+0000","last_change":"2024-03-20T09:43:58.187624+0000","last_active":"2024-03-20T09:44:20.178564+0000","last_peered":"2024-03-20T09:44:20.178564+0000","last_clean":"2024-03-20T09:44:20.178564+0000","last_became_active":"2024-03-20T09:40:24.791005+0000","last_became_peered":"2024-03-20T09:40:24.791005+0000","last_unstale":"2024-03-20T09:44:20.178564+0000","last_undegraded":"2024-03-20T09:44:20.178564+0000","last_fullsized":"2024-03-20T09:44:20.178564+0000","mapping_epoch":1066,"log_start":"1097'7600","ondisk_log_start":"1097'7600","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1133'16468","last_scrub_stamp":"2024-03-20T09:43:58.187354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:58.187354+0000","objects_scrubbed":2257,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:00.972878+0000","scrub_duration":557,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71041024,"num_objects":1084,"num_object_clones":0,"num_object_copies":3252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15170,"num_write_kb":520128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"20'137","reported_seq":2509,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:14.147516+0000","last_change":"2024-03-20T09:43:13.063995+0000","last_active":"2024-03-20T09:44:14.147516+0000","last_peered":"2024-03-20T09:44:14.147516+0000","last_clean":"2024-03-20T09:44:14.147516+0000","last_became_active":"2024-03-20T09:20:25.341750+0000","last_became_peered":"2024-03-20T09:20:25.341750+0000","last_unstale":"2024-03-20T09:44:14.147516+0000","last_undegraded":"2024-03-20T09:44:14.147516+0000","last_fullsized":"2024-03-20T09:44:14.147516+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:13.063918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:13.063918+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:22.184587+0000","scrub_duration":184,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,2],"acting":[9,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.2","version":"1139'15488","reported_seq":15786,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:20.154933+0000","last_change":"2024-03-20T09:43:09.666385+0000","last_active":"2024-03-20T09:44:20.154933+0000","last_peered":"2024-03-20T09:44:20.154933+0000","last_clean":"2024-03-20T09:44:20.154933+0000","last_became_active":"2024-03-20T09:40:24.790265+0000","last_became_peered":"2024-03-20T09:40:24.790265+0000","last_unstale":"2024-03-20T09:44:20.154933+0000","last_undegraded":"2024-03-20T09:44:20.154933+0000","last_fullsized":"2024-03-20T09:44:20.154933+0000","mapping_epoch":1066,"log_start":"1083'5400","ondisk_log_start":"1083'5400","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'11638","last_scrub_stamp":"2024-03-20T09:43:09.666260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:09.666260+0000","objects_scrubbed":4972,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:35.499687+0000","scrub_duration":1058,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72220672,"num_objects":1102,"num_object_clones":0,"num_object_copies":3306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15488,"num_write_kb":530880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,9],"acting":[3,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"20'137","reported_seq":1190,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:14.147854+0000","last_change":"2024-03-20T09:43:04.893745+0000","last_active":"2024-03-20T09:44:14.147854+0000","last_peered":"2024-03-20T09:44:14.147854+0000","last_clean":"2024-03-20T09:44:14.147854+0000","last_became_active":"2024-03-20T09:24:16.444232+0000","last_became_peered":"2024-03-20T09:24:16.444232+0000","last_unstale":"2024-03-20T09:44:14.147854+0000","last_undegraded":"2024-03-20T09:44:14.147854+0000","last_fullsized":"2024-03-20T09:44:14.147854+0000","mapping_epoch":718,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:04.893646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:04.893646+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:17.914898+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.c","version":"1139'15081","reported_seq":15374,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:20.180190+0000","last_change":"2024-03-20T09:43:06.941039+0000","last_active":"2024-03-20T09:44:20.180190+0000","last_peered":"2024-03-20T09:44:20.180190+0000","last_clean":"2024-03-20T09:44:20.180190+0000","last_became_active":"2024-03-20T09:40:24.791734+0000","last_became_peered":"2024-03-20T09:40:24.791734+0000","last_unstale":"2024-03-20T09:44:20.180190+0000","last_undegraded":"2024-03-20T09:44:20.180190+0000","last_fullsized":"2024-03-20T09:44:20.180190+0000","mapping_epoch":1066,"log_start":"1083'5000","ondisk_log_start":"1083'5000","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1117'11282","last_scrub_stamp":"2024-03-20T09:43:06.940719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:06.940719+0000","objects_scrubbed":4972,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:33.258717+0000","scrub_duration":1275,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74645504,"num_objects":1139,"num_object_clones":0,"num_object_copies":3417,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15081,"num_write_kb":519040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,9],"acting":[3,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"20'137","reported_seq":2437,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:14.149288+0000","last_change":"2024-03-20T09:43:18.087277+0000","last_active":"2024-03-20T09:44:14.149288+0000","last_peered":"2024-03-20T09:44:14.149288+0000","last_clean":"2024-03-20T09:44:14.149288+0000","last_became_active":"2024-03-20T09:12:14.306384+0000","last_became_peered":"2024-03-20T09:12:14.306384+0000","last_unstale":"2024-03-20T09:44:14.149288+0000","last_undegraded":"2024-03-20T09:44:14.149288+0000","last_fullsized":"2024-03-20T09:44:14.149288+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:18.087165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:18.087165+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:46.885376+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.d","version":"1139'15442","reported_seq":15740,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:20.455414+0000","last_change":"2024-03-20T09:43:18.652174+0000","last_active":"2024-03-20T09:44:20.455414+0000","last_peered":"2024-03-20T09:44:20.455414+0000","last_clean":"2024-03-20T09:44:20.455414+0000","last_became_active":"2024-03-20T09:40:24.812843+0000","last_became_peered":"2024-03-20T09:40:24.812843+0000","last_unstale":"2024-03-20T09:44:20.455414+0000","last_undegraded":"2024-03-20T09:44:20.455414+0000","last_fullsized":"2024-03-20T09:44:20.455414+0000","mapping_epoch":1066,"log_start":"1083'5400","ondisk_log_start":"1083'5400","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1122'12042","last_scrub_stamp":"2024-03-20T09:43:18.651987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:18.651987+0000","objects_scrubbed":4624,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:48.073907+0000","scrub_duration":1636,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77135896,"num_objects":1178,"num_object_clones":0,"num_object_copies":3534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1178,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15442,"num_write_kb":531777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"20'137","reported_seq":1098,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:14.142629+0000","last_change":"2024-03-20T09:43:15.623506+0000","last_active":"2024-03-20T09:44:14.142629+0000","last_peered":"2024-03-20T09:44:14.142629+0000","last_clean":"2024-03-20T09:44:14.142629+0000","last_became_active":"2024-03-20T09:35:19.820895+0000","last_became_peered":"2024-03-20T09:35:19.820895+0000","last_unstale":"2024-03-20T09:44:14.142629+0000","last_undegraded":"2024-03-20T09:44:14.142629+0000","last_fullsized":"2024-03-20T09:44:14.142629+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:15.623424+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:19:03.805185+0000","last_clean_scrub_stamp":"2024-03-20T09:43:15.623424+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:28.959946+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"1139'15456","reported_seq":15749,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:19.189290+0000","last_change":"2024-03-20T09:43:12.887942+0000","last_active":"2024-03-20T09:44:19.189290+0000","last_peered":"2024-03-20T09:44:19.189290+0000","last_clean":"2024-03-20T09:44:19.189290+0000","last_became_active":"2024-03-20T09:40:24.813215+0000","last_became_peered":"2024-03-20T09:40:24.813215+0000","last_unstale":"2024-03-20T09:44:19.189290+0000","last_undegraded":"2024-03-20T09:44:19.189290+0000","last_fullsized":"2024-03-20T09:44:19.189290+0000","mapping_epoch":1066,"log_start":"1083'5400","ondisk_log_start":"1083'5400","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1120'11793","last_scrub_stamp":"2024-03-20T09:43:12.887835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:12.887835+0000","objects_scrubbed":4853,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:20.577024+0000","scrub_duration":1311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76152832,"num_objects":1162,"num_object_clones":0,"num_object_copies":3486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15456,"num_write_kb":531776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,1],"acting":[10,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"20'137","reported_seq":1968,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:14.143475+0000","last_change":"2024-03-20T09:43:14.612155+0000","last_active":"2024-03-20T09:44:14.143475+0000","last_peered":"2024-03-20T09:44:14.143475+0000","last_clean":"2024-03-20T09:44:14.143475+0000","last_became_active":"2024-03-20T09:35:19.822321+0000","last_became_peered":"2024-03-20T09:35:19.822321+0000","last_unstale":"2024-03-20T09:44:14.143475+0000","last_undegraded":"2024-03-20T09:44:14.143475+0000","last_fullsized":"2024-03-20T09:44:14.143475+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:14.612037+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:24:31.549603+0000","last_clean_scrub_stamp":"2024-03-20T09:43:14.612037+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:21.810461+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.b","version":"1139'15142","reported_seq":15434,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:16.858214+0000","last_change":"2024-03-20T09:43:21.410182+0000","last_active":"2024-03-20T09:44:16.858214+0000","last_peered":"2024-03-20T09:44:16.858214+0000","last_clean":"2024-03-20T09:44:16.858214+0000","last_became_active":"2024-03-20T09:40:24.791110+0000","last_became_peered":"2024-03-20T09:40:24.791110+0000","last_unstale":"2024-03-20T09:44:16.858214+0000","last_undegraded":"2024-03-20T09:44:16.858214+0000","last_fullsized":"2024-03-20T09:44:16.858214+0000","mapping_epoch":1066,"log_start":"1083'5100","ondisk_log_start":"1083'5100","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1122'12128","last_scrub_stamp":"2024-03-20T09:43:21.409950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:21.409950+0000","objects_scrubbed":4326,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:49.041623+0000","scrub_duration":1397,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83623936,"num_objects":1276,"num_object_clones":0,"num_object_copies":3828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15142,"num_write_kb":525376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"20'137","reported_seq":1503,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:14.149583+0000","last_change":"2024-03-20T09:43:10.005346+0000","last_active":"2024-03-20T09:44:14.149583+0000","last_peered":"2024-03-20T09:44:14.149583+0000","last_clean":"2024-03-20T09:44:14.149583+0000","last_became_active":"2024-03-20T09:31:07.424588+0000","last_became_peered":"2024-03-20T09:31:07.424588+0000","last_unstale":"2024-03-20T09:44:14.149583+0000","last_undegraded":"2024-03-20T09:44:14.149583+0000","last_fullsized":"2024-03-20T09:44:14.149583+0000","mapping_epoch":871,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":872,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:10.005211+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:30:47.812125+0000","last_clean_scrub_stamp":"2024-03-20T09:43:10.005211+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"1139'17233","reported_seq":17602,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:18.956659+0000","last_change":"2024-03-20T09:43:57.525315+0000","last_active":"2024-03-20T09:44:18.956659+0000","last_peered":"2024-03-20T09:44:18.956659+0000","last_clean":"2024-03-20T09:44:18.956659+0000","last_became_active":"2024-03-20T09:40:24.790197+0000","last_became_peered":"2024-03-20T09:40:24.790197+0000","last_unstale":"2024-03-20T09:44:18.956659+0000","last_undegraded":"2024-03-20T09:44:18.956659+0000","last_fullsized":"2024-03-20T09:44:18.956659+0000","mapping_epoch":1066,"log_start":"1093'7200","ondisk_log_start":"1093'7200","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1133'16101","last_scrub_stamp":"2024-03-20T09:43:57.524948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:57.524948+0000","objects_scrubbed":2292,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:15.662531+0000","scrub_duration":556,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75300864,"num_objects":1149,"num_object_clones":0,"num_object_copies":3447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15281,"num_write_kb":525760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,5],"acting":[9,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":2409,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:14.152003+0000","last_change":"2024-03-20T09:43:08.050620+0000","last_active":"2024-03-20T09:44:14.152003+0000","last_peered":"2024-03-20T09:44:14.152003+0000","last_clean":"2024-03-20T09:44:14.152003+0000","last_became_active":"2024-03-20T09:12:14.305989+0000","last_became_peered":"2024-03-20T09:12:14.305989+0000","last_unstale":"2024-03-20T09:44:14.152003+0000","last_undegraded":"2024-03-20T09:44:14.152003+0000","last_fullsized":"2024-03-20T09:44:14.152003+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:08.050532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:08.050532+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:21.415537+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.a","version":"1139'15346","reported_seq":15638,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:20.469139+0000","last_change":"2024-03-20T09:43:08.178211+0000","last_active":"2024-03-20T09:44:20.469139+0000","last_peered":"2024-03-20T09:44:20.469139+0000","last_clean":"2024-03-20T09:44:20.469139+0000","last_became_active":"2024-03-20T09:40:24.812882+0000","last_became_peered":"2024-03-20T09:40:24.812882+0000","last_unstale":"2024-03-20T09:44:20.469139+0000","last_undegraded":"2024-03-20T09:44:20.469139+0000","last_fullsized":"2024-03-20T09:44:20.469139+0000","mapping_epoch":1066,"log_start":"1083'5300","ondisk_log_start":"1083'5300","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1117'11503","last_scrub_stamp":"2024-03-20T09:43:08.177809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:08.177809+0000","objects_scrubbed":4944,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:34.330102+0000","scrub_duration":1251,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70516736,"num_objects":1076,"num_object_clones":0,"num_object_copies":3228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15346,"num_write_kb":525504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"20'137","reported_seq":1810,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:14.139586+0000","last_change":"2024-03-20T09:43:57.239285+0000","last_active":"2024-03-20T09:44:14.139586+0000","last_peered":"2024-03-20T09:44:14.139586+0000","last_clean":"2024-03-20T09:44:14.139586+0000","last_became_active":"2024-03-20T09:39:17.944040+0000","last_became_peered":"2024-03-20T09:39:17.944040+0000","last_unstale":"2024-03-20T09:44:14.139586+0000","last_undegraded":"2024-03-20T09:44:14.139586+0000","last_fullsized":"2024-03-20T09:44:14.139586+0000","mapping_epoch":1041,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":1042,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:57.239132+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:42:42.369236+0000","last_clean_scrub_stamp":"2024-03-20T09:43:57.239132+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:11.927324+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,8],"acting":[4,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.9","version":"1139'15369","reported_seq":15667,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:19.156932+0000","last_change":"2024-03-20T09:43:09.692400+0000","last_active":"2024-03-20T09:44:19.156932+0000","last_peered":"2024-03-20T09:44:19.156932+0000","last_clean":"2024-03-20T09:44:19.156932+0000","last_became_active":"2024-03-20T09:40:24.792650+0000","last_became_peered":"2024-03-20T09:40:24.792650+0000","last_unstale":"2024-03-20T09:44:19.156932+0000","last_undegraded":"2024-03-20T09:44:19.156932+0000","last_fullsized":"2024-03-20T09:44:19.156932+0000","mapping_epoch":1066,"log_start":"1083'5300","ondisk_log_start":"1083'5300","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'11635","last_scrub_stamp":"2024-03-20T09:43:09.692199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:09.692199+0000","objects_scrubbed":4930,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:31.593123+0000","scrub_duration":1148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77135872,"num_objects":1177,"num_object_clones":0,"num_object_copies":3531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15369,"num_write_kb":529472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"20'137","reported_seq":1849,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:14.149430+0000","last_change":"2024-03-20T09:43:06.796634+0000","last_active":"2024-03-20T09:44:14.149430+0000","last_peered":"2024-03-20T09:44:14.149430+0000","last_clean":"2024-03-20T09:44:14.149430+0000","last_became_active":"2024-03-20T09:15:34.225022+0000","last_became_peered":"2024-03-20T09:15:34.225022+0000","last_unstale":"2024-03-20T09:44:14.149430+0000","last_undegraded":"2024-03-20T09:44:14.149430+0000","last_fullsized":"2024-03-20T09:44:14.149430+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:06.796525+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:20:12.169658+0000","last_clean_scrub_stamp":"2024-03-20T09:43:06.796525+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:36.395280+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,5],"acting":[6,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"1139'15491","reported_seq":15785,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:20.473285+0000","last_change":"2024-03-20T09:43:13.370701+0000","last_active":"2024-03-20T09:44:20.473285+0000","last_peered":"2024-03-20T09:44:20.473285+0000","last_clean":"2024-03-20T09:44:20.473285+0000","last_became_active":"2024-03-20T09:40:24.790189+0000","last_became_peered":"2024-03-20T09:40:24.790189+0000","last_unstale":"2024-03-20T09:44:20.473285+0000","last_undegraded":"2024-03-20T09:44:20.473285+0000","last_fullsized":"2024-03-20T09:44:20.473285+0000","mapping_epoch":1066,"log_start":"1083'5400","ondisk_log_start":"1083'5400","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1120'11823","last_scrub_stamp":"2024-03-20T09:43:13.370340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:13.370340+0000","objects_scrubbed":4782,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:20.684399+0000","scrub_duration":1344,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70582272,"num_objects":1077,"num_object_clones":0,"num_object_copies":3231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15491,"num_write_kb":530176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"20'137","reported_seq":2405,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:14.149853+0000","last_change":"2024-03-20T09:43:11.044649+0000","last_active":"2024-03-20T09:44:14.149853+0000","last_peered":"2024-03-20T09:44:14.149853+0000","last_clean":"2024-03-20T09:44:14.149853+0000","last_became_active":"2024-03-20T09:35:19.820396+0000","last_became_peered":"2024-03-20T09:35:19.820396+0000","last_unstale":"2024-03-20T09:44:14.149853+0000","last_undegraded":"2024-03-20T09:44:14.149853+0000","last_fullsized":"2024-03-20T09:44:14.149853+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:11.044527+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:14:10.560890+0000","last_clean_scrub_stamp":"2024-03-20T09:43:11.044527+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:19.618048+0000","scrub_duration":279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"1139'15421","reported_seq":15711,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:18.908055+0000","last_change":"2024-03-20T09:43:09.360089+0000","last_active":"2024-03-20T09:44:18.908055+0000","last_peered":"2024-03-20T09:44:18.908055+0000","last_clean":"2024-03-20T09:44:18.908055+0000","last_became_active":"2024-03-20T09:40:24.791199+0000","last_became_peered":"2024-03-20T09:40:24.791199+0000","last_unstale":"2024-03-20T09:44:18.908055+0000","last_undegraded":"2024-03-20T09:44:18.908055+0000","last_fullsized":"2024-03-20T09:44:18.908055+0000","mapping_epoch":1066,"log_start":"1083'5400","ondisk_log_start":"1083'5400","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'11689","last_scrub_stamp":"2024-03-20T09:43:09.359969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:09.359969+0000","objects_scrubbed":4954,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:28.242282+0000","scrub_duration":1552,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78577664,"num_objects":1199,"num_object_clones":0,"num_object_copies":3597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15421,"num_write_kb":531840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,5],"acting":[9,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"20'137","reported_seq":2237,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:14.149072+0000","last_change":"2024-03-20T09:43:50.466520+0000","last_active":"2024-03-20T09:44:14.149072+0000","last_peered":"2024-03-20T09:44:14.149072+0000","last_clean":"2024-03-20T09:44:14.149072+0000","last_became_active":"2024-03-20T09:39:17.940497+0000","last_became_peered":"2024-03-20T09:39:17.940497+0000","last_unstale":"2024-03-20T09:44:14.149072+0000","last_undegraded":"2024-03-20T09:44:14.149072+0000","last_fullsized":"2024-03-20T09:44:14.149072+0000","mapping_epoch":1041,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":1042,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:50.466383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:50.466383+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:05.247322+0000","scrub_duration":307,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,2],"acting":[11,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.6","version":"1139'18395","reported_seq":18835,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:20.471913+0000","last_change":"2024-03-20T09:43:46.116088+0000","last_active":"2024-03-20T09:44:20.471913+0000","last_peered":"2024-03-20T09:44:20.471913+0000","last_clean":"2024-03-20T09:44:20.471913+0000","last_became_active":"2024-03-20T09:40:24.792369+0000","last_became_peered":"2024-03-20T09:40:24.792369+0000","last_unstale":"2024-03-20T09:44:20.471913+0000","last_undegraded":"2024-03-20T09:44:20.471913+0000","last_fullsized":"2024-03-20T09:44:20.471913+0000","mapping_epoch":1066,"log_start":"1097'8300","ondisk_log_start":"1097'8300","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1131'16486","last_scrub_stamp":"2024-03-20T09:43:46.115894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:46.115894+0000","objects_scrubbed":3011,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:57.198816+0000","scrub_duration":1002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69992448,"num_objects":1068,"num_object_clones":0,"num_object_copies":3204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15362,"num_write_kb":525760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"20'137","reported_seq":1661,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:14.139989+0000","last_change":"2024-03-20T09:43:04.334350+0000","last_active":"2024-03-20T09:44:14.139989+0000","last_peered":"2024-03-20T09:44:14.139989+0000","last_clean":"2024-03-20T09:44:14.139989+0000","last_became_active":"2024-03-20T09:19:54.506134+0000","last_became_peered":"2024-03-20T09:19:54.506134+0000","last_unstale":"2024-03-20T09:44:14.139989+0000","last_undegraded":"2024-03-20T09:44:14.139989+0000","last_fullsized":"2024-03-20T09:44:14.139989+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:04.334199+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:29:53.362224+0000","last_clean_scrub_stamp":"2024-03-20T09:43:04.334199+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:15.046031+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"1139'15587","reported_seq":15870,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:19.188514+0000","last_change":"2024-03-20T09:43:20.856797+0000","last_active":"2024-03-20T09:44:19.188514+0000","last_peered":"2024-03-20T09:44:19.188514+0000","last_clean":"2024-03-20T09:44:19.188514+0000","last_became_active":"2024-03-20T09:40:24.813246+0000","last_became_peered":"2024-03-20T09:40:24.813246+0000","last_unstale":"2024-03-20T09:44:19.188514+0000","last_undegraded":"2024-03-20T09:44:19.188514+0000","last_fullsized":"2024-03-20T09:44:19.188514+0000","mapping_epoch":1066,"log_start":"1083'5500","ondisk_log_start":"1083'5500","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1122'12369","last_scrub_stamp":"2024-03-20T09:43:20.856432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:20.856432+0000","objects_scrubbed":4430,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:43.625443+0000","scrub_duration":1246,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77135872,"num_objects":1177,"num_object_clones":0,"num_object_copies":3531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15587,"num_write_kb":536448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,3],"acting":[10,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"20'137","reported_seq":2425,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:14.149750+0000","last_change":"2024-03-20T09:43:50.972722+0000","last_active":"2024-03-20T09:44:14.149750+0000","last_peered":"2024-03-20T09:44:14.149750+0000","last_clean":"2024-03-20T09:44:14.149750+0000","last_became_active":"2024-03-20T09:19:54.524397+0000","last_became_peered":"2024-03-20T09:19:54.524397+0000","last_unstale":"2024-03-20T09:44:14.149750+0000","last_undegraded":"2024-03-20T09:44:14.149750+0000","last_fullsized":"2024-03-20T09:44:14.149750+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:50.972601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:50.972601+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:58.813078+0000","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"1139'14875","reported_seq":15164,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:15.819503+0000","last_change":"2024-03-20T09:43:09.459247+0000","last_active":"2024-03-20T09:44:15.819503+0000","last_peered":"2024-03-20T09:44:15.819503+0000","last_clean":"2024-03-20T09:44:15.819503+0000","last_became_active":"2024-03-20T09:40:24.789539+0000","last_became_peered":"2024-03-20T09:40:24.789539+0000","last_unstale":"2024-03-20T09:44:15.819503+0000","last_undegraded":"2024-03-20T09:44:15.819503+0000","last_fullsized":"2024-03-20T09:44:15.819503+0000","mapping_epoch":1066,"log_start":"1083'4800","ondisk_log_start":"1083'4800","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1119'11421","last_scrub_stamp":"2024-03-20T09:43:09.459059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:09.459059+0000","objects_scrubbed":4816,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:29.439861+0000","scrub_duration":1236,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88014848,"num_objects":1343,"num_object_clones":0,"num_object_copies":4029,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14875,"num_write_kb":518976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.c","version":"570'138","reported_seq":2416,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:14.142636+0000","last_change":"2024-03-20T09:43:07.728946+0000","last_active":"2024-03-20T09:44:14.142636+0000","last_peered":"2024-03-20T09:44:14.142636+0000","last_clean":"2024-03-20T09:44:14.142636+0000","last_became_active":"2024-03-20T09:39:17.943381+0000","last_became_peered":"2024-03-20T09:39:17.943381+0000","last_unstale":"2024-03-20T09:44:14.142636+0000","last_undegraded":"2024-03-20T09:44:14.142636+0000","last_fullsized":"2024-03-20T09:44:14.142636+0000","mapping_epoch":1041,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":1042,"parent":"0.0","parent_split_bits":4,"last_scrub":"570'138","last_scrub_stamp":"2024-03-20T09:43:07.728813+0000","last_deep_scrub":"570'138","last_deep_scrub_stamp":"2024-03-20T09:16:13.725035+0000","last_clean_scrub_stamp":"2024-03-20T09:43:07.728813+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:26.795166+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,9],"acting":[3,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"1139'15475","reported_seq":15771,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:16.084874+0000","last_change":"2024-03-20T09:43:20.493432+0000","last_active":"2024-03-20T09:44:16.084874+0000","last_peered":"2024-03-20T09:44:16.084874+0000","last_clean":"2024-03-20T09:44:16.084874+0000","last_became_active":"2024-03-20T09:40:24.813276+0000","last_became_peered":"2024-03-20T09:40:24.813276+0000","last_unstale":"2024-03-20T09:44:16.084874+0000","last_undegraded":"2024-03-20T09:44:16.084874+0000","last_fullsized":"2024-03-20T09:44:16.084874+0000","mapping_epoch":1066,"log_start":"1083'5400","ondisk_log_start":"1083'5400","created":1066,"last_epoch_clean":1067,"parent":"0.0","parent_split_bits":0,"last_scrub":"1122'12403","last_scrub_stamp":"2024-03-20T09:43:20.493115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T09:40:23.772372+0000","last_clean_scrub_stamp":"2024-03-20T09:43:20.493115+0000","objects_scrubbed":4492,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:21.692329+0000","scrub_duration":1417,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90636288,"num_objects":1383,"num_object_clones":0,"num_object_copies":4149,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15475,"num_write_kb":539456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"20'137","reported_seq":1975,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:14.139884+0000","last_change":"2024-03-20T09:43:46.440102+0000","last_active":"2024-03-20T09:44:14.139884+0000","last_peered":"2024-03-20T09:44:14.139884+0000","last_clean":"2024-03-20T09:44:14.139884+0000","last_became_active":"2024-03-20T09:19:54.502529+0000","last_became_peered":"2024-03-20T09:19:54.502529+0000","last_unstale":"2024-03-20T09:44:14.139884+0000","last_undegraded":"2024-03-20T09:44:14.139884+0000","last_fullsized":"2024-03-20T09:44:14.139884+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:46.439952+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:41:52.574619+0000","last_clean_scrub_stamp":"2024-03-20T09:43:46.439952+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:05.231528+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"20'137","reported_seq":2256,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:14.147808+0000","last_change":"2024-03-20T09:43:15.202936+0000","last_active":"2024-03-20T09:44:14.147808+0000","last_peered":"2024-03-20T09:44:14.147808+0000","last_clean":"2024-03-20T09:44:14.147808+0000","last_became_active":"2024-03-20T09:10:34.096758+0000","last_became_peered":"2024-03-20T09:10:34.096758+0000","last_unstale":"2024-03-20T09:44:14.147808+0000","last_undegraded":"2024-03-20T09:44:14.147808+0000","last_fullsized":"2024-03-20T09:44:14.147808+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:15.202813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:15.202813+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:35.894669+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"20'137","reported_seq":1016,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:14.149179+0000","last_change":"2024-03-20T09:43:15.359792+0000","last_active":"2024-03-20T09:44:14.149179+0000","last_peered":"2024-03-20T09:44:14.149179+0000","last_clean":"2024-03-20T09:44:14.149179+0000","last_became_active":"2024-03-20T09:35:19.822242+0000","last_became_peered":"2024-03-20T09:35:19.822242+0000","last_unstale":"2024-03-20T09:44:14.149179+0000","last_undegraded":"2024-03-20T09:44:14.149179+0000","last_fullsized":"2024-03-20T09:44:14.149179+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:15.359586+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:37:52.359932+0000","last_clean_scrub_stamp":"2024-03-20T09:43:15.359586+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:29.699947+0000","scrub_duration":298,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,10],"acting":[11,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.10","version":"20'137","reported_seq":1827,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:14.147678+0000","last_change":"2024-03-20T09:43:48.137633+0000","last_active":"2024-03-20T09:44:14.147678+0000","last_peered":"2024-03-20T09:44:14.147678+0000","last_clean":"2024-03-20T09:44:14.147678+0000","last_became_active":"2024-03-20T09:35:19.820595+0000","last_became_peered":"2024-03-20T09:35:19.820595+0000","last_unstale":"2024-03-20T09:44:14.147678+0000","last_undegraded":"2024-03-20T09:44:14.147678+0000","last_fullsized":"2024-03-20T09:44:14.147678+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:48.137502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:48.137502+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:17.764144+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"20'137","reported_seq":614,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:14.137786+0000","last_change":"2024-03-20T09:43:09.196572+0000","last_active":"2024-03-20T09:44:14.137786+0000","last_peered":"2024-03-20T09:44:14.137786+0000","last_clean":"2024-03-20T09:44:14.137786+0000","last_became_active":"2024-03-20T09:39:17.943385+0000","last_became_peered":"2024-03-20T09:39:17.943385+0000","last_unstale":"2024-03-20T09:44:14.137786+0000","last_undegraded":"2024-03-20T09:44:14.137786+0000","last_fullsized":"2024-03-20T09:44:14.137786+0000","mapping_epoch":1041,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":1042,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:09.196479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:09.196479+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:34.758699+0000","scrub_duration":184,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,3],"acting":[8,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"20'137","reported_seq":892,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:14.142609+0000","last_change":"2024-03-20T09:43:06.840465+0000","last_active":"2024-03-20T09:44:14.142609+0000","last_peered":"2024-03-20T09:44:14.142609+0000","last_clean":"2024-03-20T09:44:14.142609+0000","last_became_active":"2024-03-20T09:39:17.943337+0000","last_became_peered":"2024-03-20T09:39:17.943337+0000","last_unstale":"2024-03-20T09:44:14.142609+0000","last_undegraded":"2024-03-20T09:44:14.142609+0000","last_fullsized":"2024-03-20T09:44:14.142609+0000","mapping_epoch":1041,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":1042,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:06.840344+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:35:27.264203+0000","last_clean_scrub_stamp":"2024-03-20T09:43:06.840344+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:31.325871+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,11],"acting":[3,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.17","version":"20'137","reported_seq":2397,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:14.139501+0000","last_change":"2024-03-20T09:43:12.288008+0000","last_active":"2024-03-20T09:44:14.139501+0000","last_peered":"2024-03-20T09:44:14.139501+0000","last_clean":"2024-03-20T09:44:14.139501+0000","last_became_active":"2024-03-20T09:35:19.822429+0000","last_became_peered":"2024-03-20T09:35:19.822429+0000","last_unstale":"2024-03-20T09:44:14.139501+0000","last_undegraded":"2024-03-20T09:44:14.139501+0000","last_fullsized":"2024-03-20T09:44:14.139501+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:12.287881+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:38:01.419884+0000","last_clean_scrub_stamp":"2024-03-20T09:43:12.287881+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:44:27.962869+0000","scrub_duration":272,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"20'137","reported_seq":2428,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:14.142550+0000","last_change":"2024-03-20T09:43:51.249807+0000","last_active":"2024-03-20T09:44:14.142550+0000","last_peered":"2024-03-20T09:44:14.142550+0000","last_clean":"2024-03-20T09:44:14.142550+0000","last_became_active":"2024-03-20T09:20:25.349082+0000","last_became_peered":"2024-03-20T09:20:25.349082+0000","last_unstale":"2024-03-20T09:44:14.142550+0000","last_undegraded":"2024-03-20T09:44:14.142550+0000","last_fullsized":"2024-03-20T09:44:14.142550+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:51.249669+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:41:54.735667+0000","last_clean_scrub_stamp":"2024-03-20T09:43:51.249669+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:17.928220+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"20'137","reported_seq":1708,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:14.146674+0000","last_change":"2024-03-20T09:43:55.035201+0000","last_active":"2024-03-20T09:44:14.146674+0000","last_peered":"2024-03-20T09:44:14.146674+0000","last_clean":"2024-03-20T09:44:14.146674+0000","last_became_active":"2024-03-20T09:12:14.307581+0000","last_became_peered":"2024-03-20T09:12:14.307581+0000","last_unstale":"2024-03-20T09:44:14.146674+0000","last_undegraded":"2024-03-20T09:44:14.146674+0000","last_fullsized":"2024-03-20T09:44:14.146674+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:55.035080+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:34:48.079272+0000","last_clean_scrub_stamp":"2024-03-20T09:43:55.035080+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:02.488697+0000","scrub_duration":240,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"20'137","reported_seq":972,"reported_epoch":1139,"state":"active+clean","last_fresh":"2024-03-20T09:44:14.143773+0000","last_change":"2024-03-20T09:43:56.715773+0000","last_active":"2024-03-20T09:44:14.143773+0000","last_peered":"2024-03-20T09:44:14.143773+0000","last_clean":"2024-03-20T09:44:14.143773+0000","last_became_active":"2024-03-20T09:41:47.137425+0000","last_became_peered":"2024-03-20T09:41:47.137425+0000","last_unstale":"2024-03-20T09:44:14.143773+0000","last_undegraded":"2024-03-20T09:44:14.143773+0000","last_fullsized":"2024-03-20T09:44:14.143773+0000","mapping_epoch":1094,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":1095,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:43:56.715666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:43:56.715666+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:45:02.384244+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":1242234904,"num_objects":18956,"num_object_clones":0,"num_object_copies":56868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18956,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":245500,"num_write_kb":8462529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3766169600,"data_stored":3766157384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160944,"ondisk_log_size":160944,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1925120,"data_stored":1914464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4248,"ondisk_log_size":4248,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316860996,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1462260,"kb_used_data":357800,"kb_used_omap":1,"kb_used_meta":1104318,"kb_avail":92282892,"statfs":{"total":95995035648,"available":94497681408,"internally_reserved":0,"allocated":366387200,"data_stored":366090576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":1130822084},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":861,"seq":3697966842021,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":710448,"kb_used_data":385128,"kb_used_omap":103,"kb_used_meta":325208,"kb_avail":93034704,"statfs":{"total":95995035648,"available":95267536896,"internally_reserved":0,"allocated":394371072,"data_stored":394074448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":106169,"internal_metadata":333013319},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":80,"seq":343597384273,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1188848,"kb_used_data":453672,"kb_used_omap":1,"kb_used_meta":735038,"kb_avail":92556304,"statfs":{"total":95995035648,"available":94777655296,"internally_reserved":0,"allocated":464560128,"data_stored":464263504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":752679360},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":70,"seq":300647711317,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":855728,"kb_used_data":166504,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":92889424,"statfs":{"total":95995035648,"available":95118770176,"internally_reserved":0,"allocated":170500096,"data_stored":170203472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":705755590},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":60,"seq":257698038361,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1468540,"kb_used_data":440564,"kb_used_omap":1,"kb_used_meta":1027966,"kb_avail":92276612,"statfs":{"total":95995035648,"available":94491250688,"internally_reserved":0,"allocated":451137536,"data_stored":450833288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":1052637642},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332702,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":943984,"kb_used_data":172840,"kb_used_omap":1,"kb_used_meta":771134,"kb_avail":92801168,"statfs":{"total":95995035648,"available":95028396032,"internally_reserved":0,"allocated":176988160,"data_stored":176691536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":789641670},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575481,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1785972,"kb_used_data":595564,"kb_used_omap":1,"kb_used_meta":1190398,"kb_avail":91959180,"statfs":{"total":95995035648,"available":94166200320,"internally_reserved":0,"allocated":609857536,"data_stored":609556840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1218968006},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738999,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411954,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":852608,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":92892544,"statfs":{"total":95995035648,"available":95121965056,"internally_reserved":0,"allocated":966656,"data_stored":666480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":872085957},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150319,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1484464,"kb_used_data":366056,"kb_used_omap":1,"kb_used_meta":1118206,"kb_avail":92260688,"statfs":{"total":95995035648,"available":94474944512,"internally_reserved":0,"allocated":374841344,"data_stored":374544720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1145043399},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438954092,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":993140,"kb_used_data":181736,"kb_used_omap":1,"kb_used_meta":811390,"kb_avail":92752012,"statfs":{"total":95995035648,"available":94978060288,"internally_reserved":0,"allocated":186097664,"data_stored":185801040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":830863813},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":443,"seq":1902670512534,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1621372,"kb_used_data":639028,"kb_used_omap":181,"kb_used_meta":982154,"kb_avail":92123780,"statfs":{"total":95995035648,"available":94334750720,"internally_reserved":0,"allocated":654364672,"data_stored":654060424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":186131,"internal_metadata":1005725933},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":592515072,"data_stored":592511000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":365166592,"data_stored":365166592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":179044352,"data_stored":179044352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":650252288,"data_stored":650248216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":174456832,"data_stored":174456832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":442044416,"data_stored":442040344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":167247872,"data_stored":167247872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":462684160,"data_stored":462684160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":380305408,"data_stored":380305408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":352452608,"data_stored":352452608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:44:21.928 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-20T09:44:21.928 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pool set .mgr pg_num 41 2024-03-20T09:44:21.995 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:44:22.003 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:21 smithi026 ceph-mon[68309]: pgmap v2771: 47 pgs: 47 active+clean; 1.2 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 892 op/s 2024-03-20T09:44:22.003 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:21 smithi026 ceph-mon[68309]: from='client.27439 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:44:22.033 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:21 smithi122 ceph-mon[73495]: pgmap v2771: 47 pgs: 47 active+clean; 1.2 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 892 op/s 2024-03-20T09:44:22.033 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:21 smithi122 ceph-mon[73495]: from='client.27439 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:44:22.036 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:22.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:21 smithi115 ceph-mon[72476]: pgmap v2771: 47 pgs: 47 active+clean; 1.2 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 892 op/s 2024-03-20T09:44:22.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:21 smithi115 ceph-mon[72476]: from='client.27439 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:44:22.221 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:44:22.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:22.231 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:44:22.251 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_historic_ops 2024-03-20T09:44:22.332 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:44:22.379 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:22.402 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:44:22.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:22.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:44:22.432 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:44:22.507 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:44:22.548 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:22.689 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:44:22.756 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:44:22.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:22.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:44:22.860 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:44:22.897 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:22.944 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:44:22.948 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:22.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:44:23.050 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:44:23.083 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:23.267 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:44:23.285 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:44:23.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:23.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:44:23.392 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:44:23.427 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:23.471 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:44:23.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:23.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:44:23.576 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:44:23.631 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:23.798 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:44:23.934 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:23 smithi122 ceph-mon[73495]: pgmap v2772: 47 pgs: 47 active+clean; 1.1 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 875 op/s 2024-03-20T09:44:23.935 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:23 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/156527108' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:44:24.109 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:23 smithi026 ceph-mon[68309]: pgmap v2772: 47 pgs: 47 active+clean; 1.1 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 875 op/s 2024-03-20T09:44:24.110 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:23 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/156527108' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:44:24.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:23 smithi115 ceph-mon[72476]: pgmap v2772: 47 pgs: 47 active+clean; 1.1 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 875 op/s 2024-03-20T09:44:24.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:23 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/156527108' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:44:24.919 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:44:24.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:24.924 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:44:25.025 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:44:25.066 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:24 smithi122 ceph-mon[73495]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:44:25.066 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:24 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/156527108' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:44:25.067 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:24 smithi122 ceph-mon[73495]: osdmap e1140: 12 total, 12 up, 11 in 2024-03-20T09:44:25.067 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:24 smithi122 ceph-mon[73495]: 1.e scrub starts 2024-03-20T09:44:25.067 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:24 smithi122 ceph-mon[73495]: 1.e scrub ok 2024-03-20T09:44:25.067 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:24 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4209839028' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:44:25.070 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:25.123 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:24 smithi026 ceph-mon[68309]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:44:25.123 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:24 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/156527108' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:44:25.123 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:24 smithi026 ceph-mon[68309]: osdmap e1140: 12 total, 12 up, 11 in 2024-03-20T09:44:25.123 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:24 smithi026 ceph-mon[68309]: 1.e scrub starts 2024-03-20T09:44:25.123 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:24 smithi026 ceph-mon[68309]: 1.e scrub ok 2024-03-20T09:44:25.123 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:24 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4209839028' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:44:25.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:24 smithi115 ceph-mon[72476]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:44:25.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:24 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/156527108' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:44:25.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:24 smithi115 ceph-mon[72476]: osdmap e1140: 12 total, 12 up, 11 in 2024-03-20T09:44:25.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:24 smithi115 ceph-mon[72476]: 1.e scrub starts 2024-03-20T09:44:25.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:24 smithi115 ceph-mon[72476]: 1.e scrub ok 2024-03-20T09:44:25.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:24 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4209839028' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:44:25.253 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:44:25.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:25.258 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:44:25.359 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:44:25.412 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:25.448 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:44:25.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:25.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:44:25.503 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:44:25.503 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:44:25.503 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:44:25.503 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:44:25.503 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:44:25.503 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:44:25.503 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:44:25.503 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:44:25.503 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:44:25.503 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:44:25.503 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:44:25.503 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:44:25.503 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:44:25.504 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:44:25.504 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:44:25.504 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:44:25.504 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:44:25.504 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:44:25.504 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:44:25.504 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:44:25.504 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:44:25.504 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:44:25.504 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:44:25.504 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:44:25.552 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:44:25.600 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:25.651 INFO:teuthology.orchestra.run.smithi122.stdout:ERROR: (22) Invalid argument 2024-03-20T09:44:25.652 INFO:teuthology.orchestra.run.smithi122.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:44:25.708 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:44:25.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:25.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:44:25.815 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:44:25.863 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:25.942 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:44:25.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:25.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:44:25.982 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:25 smithi026 ceph-mon[68309]: 1.a scrub starts 2024-03-20T09:44:25.982 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:25 smithi026 ceph-mon[68309]: 1.a scrub ok 2024-03-20T09:44:25.982 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:25 smithi026 ceph-mon[68309]: pgmap v2774: 47 pgs: 47 active+clean; 1.0 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 870 op/s 2024-03-20T09:44:26.050 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:44:26.103 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:26.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:25 smithi115 ceph-mon[72476]: 1.a scrub starts 2024-03-20T09:44:26.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:25 smithi115 ceph-mon[72476]: 1.a scrub ok 2024-03-20T09:44:26.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:25 smithi115 ceph-mon[72476]: pgmap v2774: 47 pgs: 47 active+clean; 1.0 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 870 op/s 2024-03-20T09:44:26.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:25 smithi122 ceph-mon[73495]: 1.a scrub starts 2024-03-20T09:44:26.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:25 smithi122 ceph-mon[73495]: 1.a scrub ok 2024-03-20T09:44:26.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:25 smithi122 ceph-mon[73495]: pgmap v2774: 47 pgs: 47 active+clean; 1.0 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 870 op/s 2024-03-20T09:44:26.217 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:44:26.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:26.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:44:26.324 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:44:26.363 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:26.434 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:44:26.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:26.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:44:26.551 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:44:26.598 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:26.697 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: specified pg_num 41 < pg_num_max 32 2024-03-20T09:44:26.831 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:26 smithi122 ceph-mon[73495]: 1.9 scrub starts 2024-03-20T09:44:26.831 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:26 smithi122 ceph-mon[73495]: 1.9 scrub ok 2024-03-20T09:44:26.942 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-20T09:44:26.973 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:44:26.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:26.980 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:44:26.987 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:26 smithi026 ceph-mon[68309]: 1.9 scrub starts 2024-03-20T09:44:26.987 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:26 smithi026 ceph-mon[68309]: 1.9 scrub ok 2024-03-20T09:44:26.987 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:26 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2041181655' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"}]: dispatch 2024-03-20T09:44:26.988 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:26 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"}]: dispatch 2024-03-20T09:44:27.080 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:44:27.126 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:26 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2041181655' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"}]: dispatch 2024-03-20T09:44:27.127 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:26 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"}]: dispatch 2024-03-20T09:44:27.129 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:27.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:26 smithi115 ceph-mon[72476]: 1.9 scrub starts 2024-03-20T09:44:27.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:26 smithi115 ceph-mon[72476]: 1.9 scrub ok 2024-03-20T09:44:27.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:26 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2041181655' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"}]: dispatch 2024-03-20T09:44:27.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:26 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"}]: dispatch 2024-03-20T09:44:27.174 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:44:27.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:27.179 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:44:27.280 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:44:27.337 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:27.384 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:44:27.388 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:27.388 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:44:27.474 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T09:44:27.489 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:44:27.536 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:27.948 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:44:28.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:28.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:44:28.003 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:44:28.103 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:44:28.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:27 smithi115 ceph-mon[72476]: pgmap v2775: 47 pgs: 47 active+clean; 859 MiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 871 op/s 2024-03-20T09:44:28.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:27 smithi115 ceph-mon[72476]: 1.c scrub starts 2024-03-20T09:44:28.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:27 smithi115 ceph-mon[72476]: 1.c scrub ok 2024-03-20T09:44:28.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:27 smithi115 ceph-mon[72476]: 1.0 scrub starts 2024-03-20T09:44:28.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:27 smithi115 ceph-mon[72476]: 1.0 scrub ok 2024-03-20T09:44:28.159 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:27 smithi122 ceph-mon[73495]: pgmap v2775: 47 pgs: 47 active+clean; 859 MiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 871 op/s 2024-03-20T09:44:28.159 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:27 smithi122 ceph-mon[73495]: 1.c scrub starts 2024-03-20T09:44:28.159 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:27 smithi122 ceph-mon[73495]: 1.c scrub ok 2024-03-20T09:44:28.159 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:27 smithi122 ceph-mon[73495]: 1.0 scrub starts 2024-03-20T09:44:28.159 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:27 smithi122 ceph-mon[73495]: 1.0 scrub ok 2024-03-20T09:44:28.162 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:28.194 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:44:28.202 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:28.202 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:44:28.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:27 smithi026 ceph-mon[68309]: pgmap v2775: 47 pgs: 47 active+clean; 859 MiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 871 op/s 2024-03-20T09:44:28.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:27 smithi026 ceph-mon[68309]: 1.c scrub starts 2024-03-20T09:44:28.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:27 smithi026 ceph-mon[68309]: 1.c scrub ok 2024-03-20T09:44:28.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:27 smithi026 ceph-mon[68309]: 1.0 scrub starts 2024-03-20T09:44:28.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:27 smithi026 ceph-mon[68309]: 1.0 scrub ok 2024-03-20T09:44:28.303 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:44:28.332 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:28.352 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:44:28.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:28.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:44:28.457 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:44:28.493 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:28.516 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:44:28.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:28.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:44:28.621 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:44:28.668 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:28.693 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:44:28.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:28.697 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:44:28.798 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:44:28.813 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:44:28.855 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:28.897 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:44:28.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:28.905 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:44:29.006 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:44:29.053 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:29.085 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:44:29.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:29.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:44:29.193 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:44:29.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:29 smithi026 ceph-mon[68309]: 1.3 scrub starts 2024-03-20T09:44:29.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:29 smithi026 ceph-mon[68309]: 1.3 scrub ok 2024-03-20T09:44:29.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:29 smithi026 ceph-mon[68309]: pgmap v2776: 47 pgs: 47 active+clean; 842 MiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 868 op/s 2024-03-20T09:44:29.225 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:29.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:29 smithi115 ceph-mon[72476]: 1.3 scrub starts 2024-03-20T09:44:29.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:29 smithi115 ceph-mon[72476]: 1.3 scrub ok 2024-03-20T09:44:29.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:29 smithi115 ceph-mon[72476]: pgmap v2776: 47 pgs: 47 active+clean; 842 MiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 868 op/s 2024-03-20T09:44:29.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:29 smithi122 ceph-mon[73495]: 1.3 scrub starts 2024-03-20T09:44:29.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:29 smithi122 ceph-mon[73495]: 1.3 scrub ok 2024-03-20T09:44:29.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:29 smithi122 ceph-mon[73495]: pgmap v2776: 47 pgs: 47 active+clean; 842 MiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 868 op/s 2024-03-20T09:44:29.486 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:44:29.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:29.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:44:29.592 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:44:29.635 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:29.846 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:44:29.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:29.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:44:29.953 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:44:30.004 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:44:30.012 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:30.043 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:44:30.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:30.052 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:44:30.133 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:44:30.153 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:44:30.189 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:30 smithi026 ceph-mon[68309]: 1.7 scrub starts 2024-03-20T09:44:30.189 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:30 smithi026 ceph-mon[68309]: 1.7 scrub ok 2024-03-20T09:44:30.189 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:30 smithi026 ceph-mon[68309]: 9.3 scrub starts 2024-03-20T09:44:30.189 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:30 smithi026 ceph-mon[68309]: 9.3 scrub ok 2024-03-20T09:44:30.203 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:30.245 INFO:teuthology.orchestra.run.smithi026.stdout:ERROR: (22) Invalid argument 2024-03-20T09:44:30.245 INFO:teuthology.orchestra.run.smithi026.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-20T09:44:30.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:30 smithi115 ceph-mon[72476]: 1.7 scrub starts 2024-03-20T09:44:30.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:30 smithi115 ceph-mon[72476]: 1.7 scrub ok 2024-03-20T09:44:30.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:30 smithi115 ceph-mon[72476]: 9.3 scrub starts 2024-03-20T09:44:30.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:30 smithi115 ceph-mon[72476]: 9.3 scrub ok 2024-03-20T09:44:30.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:30 smithi122 ceph-mon[73495]: 1.7 scrub starts 2024-03-20T09:44:30.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:30 smithi122 ceph-mon[73495]: 1.7 scrub ok 2024-03-20T09:44:30.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:30 smithi122 ceph-mon[73495]: 9.3 scrub starts 2024-03-20T09:44:30.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:30 smithi122 ceph-mon[73495]: 9.3 scrub ok 2024-03-20T09:44:30.472 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:44:30.479 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:30.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:44:30.580 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:44:30.616 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:30.653 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:44:30.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:30.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:44:30.760 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:44:30.814 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:30.848 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:44:30.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:30.851 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:44:30.952 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:44:31.009 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:31.042 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:44:31.045 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:31.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:44:31.147 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:44:31.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:31 smithi122 ceph-mon[73495]: 9.f scrub starts 2024-03-20T09:44:31.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:31 smithi122 ceph-mon[73495]: 9.f scrub ok 2024-03-20T09:44:31.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:31 smithi122 ceph-mon[73495]: pgmap v2777: 47 pgs: 47 active+clean; 636 MiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 877 op/s 2024-03-20T09:44:31.192 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:31.288 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:31 smithi026 ceph-mon[68309]: 9.f scrub starts 2024-03-20T09:44:31.288 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:31 smithi026 ceph-mon[68309]: 9.f scrub ok 2024-03-20T09:44:31.288 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:31 smithi026 ceph-mon[68309]: pgmap v2777: 47 pgs: 47 active+clean; 636 MiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 877 op/s 2024-03-20T09:44:31.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:31 smithi115 ceph-mon[72476]: 9.f scrub starts 2024-03-20T09:44:31.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:31 smithi115 ceph-mon[72476]: 9.f scrub ok 2024-03-20T09:44:31.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:31 smithi115 ceph-mon[72476]: pgmap v2777: 47 pgs: 47 active+clean; 636 MiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 877 op/s 2024-03-20T09:44:31.724 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-20T09:44:32.154 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:44:32.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:32.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:44:32.184 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:44:32.266 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:44:32.299 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:32 smithi115 ceph-mon[72476]: 9.5 scrub starts 2024-03-20T09:44:32.299 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:32 smithi115 ceph-mon[72476]: 9.e scrub starts 2024-03-20T09:44:32.299 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:32 smithi115 ceph-mon[72476]: 9.e scrub ok 2024-03-20T09:44:32.299 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:32 smithi115 ceph-mon[72476]: 1.16 scrub starts 2024-03-20T09:44:32.306 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:32.330 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:44:32.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:32.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:44:32.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:32 smithi122 ceph-mon[73495]: 9.5 scrub starts 2024-03-20T09:44:32.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:32 smithi122 ceph-mon[73495]: 9.e scrub starts 2024-03-20T09:44:32.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:32 smithi122 ceph-mon[73495]: 9.e scrub ok 2024-03-20T09:44:32.435 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:32 smithi122 ceph-mon[73495]: 1.16 scrub starts 2024-03-20T09:44:32.436 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:44:32.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:32 smithi026 ceph-mon[68309]: 9.5 scrub starts 2024-03-20T09:44:32.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:32 smithi026 ceph-mon[68309]: 9.e scrub starts 2024-03-20T09:44:32.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:32 smithi026 ceph-mon[68309]: 9.e scrub ok 2024-03-20T09:44:32.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:32 smithi026 ceph-mon[68309]: 1.16 scrub starts 2024-03-20T09:44:32.471 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:44:32.476 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 5, 6, 7, 8, 11, 3, 4, 9, 10] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 11, 5, 10] 2024-03-20T09:44:32.476 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:44:32.476 INFO:tasks.thrashosds.thrasher:inject_pause on osd.11 2024-03-20T09:44:32.476 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-20T09:44:32.476 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-20T09:44:32.476 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 config set filestore_inject_stall 3 2024-03-20T09:44:32.503 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:32.783 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:44:32.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:32.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:44:32.888 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:44:32.905 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:44:32.950 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:32.980 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:44:32.985 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:32.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:44:33.040 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:33 smithi026 ceph-mon[68309]: 9.5 scrub ok 2024-03-20T09:44:33.040 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:33 smithi026 ceph-mon[68309]: 1.1c scrub starts 2024-03-20T09:44:33.040 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:33 smithi026 ceph-mon[68309]: 1.1c scrub ok 2024-03-20T09:44:33.040 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:33 smithi026 ceph-mon[68309]: 1.16 scrub ok 2024-03-20T09:44:33.040 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:33 smithi026 ceph-mon[68309]: pgmap v2778: 47 pgs: 47 active+clean; 564 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 885 op/s 2024-03-20T09:44:33.086 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:44:33.136 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:33.200 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:44:33.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:33.204 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:44:33.305 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:44:33.355 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:33 smithi122 ceph-mon[73495]: 9.5 scrub ok 2024-03-20T09:44:33.355 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:33 smithi122 ceph-mon[73495]: 1.1c scrub starts 2024-03-20T09:44:33.355 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:33 smithi122 ceph-mon[73495]: 1.1c scrub ok 2024-03-20T09:44:33.355 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:33 smithi122 ceph-mon[73495]: 1.16 scrub ok 2024-03-20T09:44:33.355 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:33 smithi122 ceph-mon[73495]: pgmap v2778: 47 pgs: 47 active+clean; 564 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 885 op/s 2024-03-20T09:44:33.358 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:33.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:33 smithi115 ceph-mon[72476]: 9.5 scrub ok 2024-03-20T09:44:33.398 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:33 smithi115 ceph-mon[72476]: 1.1c scrub starts 2024-03-20T09:44:33.398 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:33 smithi115 ceph-mon[72476]: 1.1c scrub ok 2024-03-20T09:44:33.398 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:33 smithi115 ceph-mon[72476]: 1.16 scrub ok 2024-03-20T09:44:33.398 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:33 smithi115 ceph-mon[72476]: pgmap v2778: 47 pgs: 47 active+clean; 564 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 885 op/s 2024-03-20T09:44:33.468 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:44:33.476 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:33.476 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:44:33.577 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:44:33.648 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:34.062 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:44:34.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:34 smithi026 ceph-mon[68309]: 1.17 scrub starts 2024-03-20T09:44:34.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:34 smithi026 ceph-mon[68309]: 1.17 scrub ok 2024-03-20T09:44:34.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:34 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4079962992' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:44:34.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:34 smithi026 ceph-mon[68309]: 9.c scrub starts 2024-03-20T09:44:34.212 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:34 smithi026 ceph-mon[68309]: 9.c scrub ok 2024-03-20T09:44:34.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:34 smithi115 ceph-mon[72476]: 1.17 scrub starts 2024-03-20T09:44:34.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:34 smithi115 ceph-mon[72476]: 1.17 scrub ok 2024-03-20T09:44:34.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:34 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4079962992' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:44:34.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:34 smithi115 ceph-mon[72476]: 9.c scrub starts 2024-03-20T09:44:34.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:34 smithi115 ceph-mon[72476]: 9.c scrub ok 2024-03-20T09:44:34.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:34 smithi122 ceph-mon[73495]: 1.17 scrub starts 2024-03-20T09:44:34.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:34 smithi122 ceph-mon[73495]: 1.17 scrub ok 2024-03-20T09:44:34.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:34 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4079962992' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:44:34.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:34 smithi122 ceph-mon[73495]: 9.c scrub starts 2024-03-20T09:44:34.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:34 smithi122 ceph-mon[73495]: 9.c scrub ok 2024-03-20T09:44:34.730 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:44:34.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:34.735 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:44:34.836 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:44:34.876 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:35.087 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:44:35.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:35.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:44:35.125 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:35 smithi026 ceph-mon[68309]: 1.13 scrub starts 2024-03-20T09:44:35.126 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:35 smithi026 ceph-mon[68309]: 1.13 scrub ok 2024-03-20T09:44:35.126 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:35 smithi026 ceph-mon[68309]: 9.4 scrub starts 2024-03-20T09:44:35.126 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:35 smithi026 ceph-mon[68309]: 9.4 scrub ok 2024-03-20T09:44:35.126 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:35 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:44:35.126 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:35 smithi026 ceph-mon[68309]: Cluster is now healthy 2024-03-20T09:44:35.126 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:35 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4079962992' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:44:35.126 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:35 smithi026 ceph-mon[68309]: osdmap e1141: 12 total, 12 up, 11 in 2024-03-20T09:44:35.126 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:35 smithi026 ceph-mon[68309]: 1.6 deep-scrub starts 2024-03-20T09:44:35.126 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:35 smithi026 ceph-mon[68309]: 1.6 deep-scrub ok 2024-03-20T09:44:35.126 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:35 smithi026 ceph-mon[68309]: pgmap v2780: 47 pgs: 47 active+clean; 478 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 885 op/s 2024-03-20T09:44:35.193 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:44:35.230 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:35.265 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:44:35.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:35.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:44:35.374 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:44:35.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:35 smithi115 ceph-mon[72476]: 1.13 scrub starts 2024-03-20T09:44:35.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:35 smithi115 ceph-mon[72476]: 1.13 scrub ok 2024-03-20T09:44:35.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:35 smithi115 ceph-mon[72476]: 9.4 scrub starts 2024-03-20T09:44:35.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:35 smithi115 ceph-mon[72476]: 9.4 scrub ok 2024-03-20T09:44:35.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:35 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:44:35.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:35 smithi115 ceph-mon[72476]: Cluster is now healthy 2024-03-20T09:44:35.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:35 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4079962992' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:44:35.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:35 smithi115 ceph-mon[72476]: osdmap e1141: 12 total, 12 up, 11 in 2024-03-20T09:44:35.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:35 smithi115 ceph-mon[72476]: 1.6 deep-scrub starts 2024-03-20T09:44:35.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:35 smithi115 ceph-mon[72476]: 1.6 deep-scrub ok 2024-03-20T09:44:35.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:35 smithi115 ceph-mon[72476]: pgmap v2780: 47 pgs: 47 active+clean; 478 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 885 op/s 2024-03-20T09:44:35.421 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:35.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:35 smithi122 ceph-mon[73495]: 1.13 scrub starts 2024-03-20T09:44:35.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:35 smithi122 ceph-mon[73495]: 1.13 scrub ok 2024-03-20T09:44:35.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:35 smithi122 ceph-mon[73495]: 9.4 scrub starts 2024-03-20T09:44:35.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:35 smithi122 ceph-mon[73495]: 9.4 scrub ok 2024-03-20T09:44:35.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:35 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-20T09:44:35.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:35 smithi122 ceph-mon[73495]: Cluster is now healthy 2024-03-20T09:44:35.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:35 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4079962992' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:44:35.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:35 smithi122 ceph-mon[73495]: osdmap e1141: 12 total, 12 up, 11 in 2024-03-20T09:44:35.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:35 smithi122 ceph-mon[73495]: 1.6 deep-scrub starts 2024-03-20T09:44:35.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:35 smithi122 ceph-mon[73495]: 1.6 deep-scrub ok 2024-03-20T09:44:35.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:35 smithi122 ceph-mon[73495]: pgmap v2780: 47 pgs: 47 active+clean; 478 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 885 op/s 2024-03-20T09:44:35.485 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:44:35.514 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:44:35.539 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:44:35.574 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:44:35.603 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:44:35.632 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:44:35.667 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:44:35.700 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:44:35.730 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:44:35.765 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:44:35.796 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:44:35.845 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:44:35.850 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:44:35.850 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:44:35.850 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:44:35.850 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:44:35.850 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:44:35.850 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:44:35.851 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:44:35.851 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:44:35.851 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:44:35.851 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:44:35.851 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:44:35.851 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:44:35.932 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:44:35.932 INFO:teuthology.orchestra.run.smithi122.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-20T09:44:35.932 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:44:36.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:36 smithi026 ceph-mon[68309]: 1.1e scrub starts 2024-03-20T09:44:36.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:36 smithi026 ceph-mon[68309]: 1.1e scrub ok 2024-03-20T09:44:36.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:36 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1214198757' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:44:36.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:36 smithi026 ceph-mon[68309]: 9.2 scrub starts 2024-03-20T09:44:36.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:36 smithi026 ceph-mon[68309]: 9.2 scrub ok 2024-03-20T09:44:36.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:36 smithi115 ceph-mon[72476]: 1.1e scrub starts 2024-03-20T09:44:36.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:36 smithi115 ceph-mon[72476]: 1.1e scrub ok 2024-03-20T09:44:36.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:36 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1214198757' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:44:36.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:36 smithi115 ceph-mon[72476]: 9.2 scrub starts 2024-03-20T09:44:36.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:36 smithi115 ceph-mon[72476]: 9.2 scrub ok 2024-03-20T09:44:36.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:36 smithi122 ceph-mon[73495]: 1.1e scrub starts 2024-03-20T09:44:36.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:36 smithi122 ceph-mon[73495]: 1.1e scrub ok 2024-03-20T09:44:36.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:36 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1214198757' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:44:36.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:36 smithi122 ceph-mon[73495]: 9.2 scrub starts 2024-03-20T09:44:36.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:36 smithi122 ceph-mon[73495]: 9.2 scrub ok 2024-03-20T09:44:36.693 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:44:36.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:36.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:44:36.803 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:44:36.837 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:37.005 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:44:37.009 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:37.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:44:37.110 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:44:37.153 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:37.239 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:44:37.243 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:37.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:44:37.247 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:37 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:44:37.247 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:37 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:44:37.247 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:37 smithi026 ceph-mon[68309]: osdmap e1142: 12 total, 12 up, 11 in 2024-03-20T09:44:37.247 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:37 smithi026 ceph-mon[68309]: 9.a scrub starts 2024-03-20T09:44:37.247 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:37 smithi026 ceph-mon[68309]: pgmap v2782: 47 pgs: 47 active+clean; 304 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.07k op/s 2024-03-20T09:44:37.345 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:44:37.375 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:37 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:44:37.376 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:37 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:44:37.376 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:37 smithi115 ceph-mon[72476]: osdmap e1142: 12 total, 12 up, 11 in 2024-03-20T09:44:37.376 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:37 smithi115 ceph-mon[72476]: 9.a scrub starts 2024-03-20T09:44:37.376 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:37 smithi115 ceph-mon[72476]: pgmap v2782: 47 pgs: 47 active+clean; 304 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.07k op/s 2024-03-20T09:44:37.379 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:37.410 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:44:37.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:37.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:44:37.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:37 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:44:37.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:37 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:44:37.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:37 smithi122 ceph-mon[73495]: osdmap e1142: 12 total, 12 up, 11 in 2024-03-20T09:44:37.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:37 smithi122 ceph-mon[73495]: 9.a scrub starts 2024-03-20T09:44:37.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:37 smithi122 ceph-mon[73495]: pgmap v2782: 47 pgs: 47 active+clean; 304 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.07k op/s 2024-03-20T09:44:37.524 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:44:37.566 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:37.596 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:44:37.596 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:44:37.597 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-03-20T09:44:37.597 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-03-20T09:44:37.597 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:44:37.610 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:44:37.613 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:37.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:44:37.715 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:44:37.772 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:37.825 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:44:37.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:37.828 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:44:37.928 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:44:37.976 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:38.005 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:44:38.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:38.009 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:44:38.087 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:38 smithi122 ceph-mon[73495]: 9.a scrub ok 2024-03-20T09:44:38.088 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:38 smithi122 ceph-mon[73495]: 1.12 scrub starts 2024-03-20T09:44:38.088 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:38 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:44:38.088 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:38 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:44:38.109 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:44:38.161 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:38.374 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:44:38.375 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:38 smithi026 ceph-mon[68309]: 9.a scrub ok 2024-03-20T09:44:38.375 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:38 smithi026 ceph-mon[68309]: 1.12 scrub starts 2024-03-20T09:44:38.376 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:38 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:44:38.376 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:38 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:44:38.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:38.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:44:38.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:38 smithi115 ceph-mon[72476]: 9.a scrub ok 2024-03-20T09:44:38.396 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:38 smithi115 ceph-mon[72476]: 1.12 scrub starts 2024-03-20T09:44:38.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:38 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:44:38.397 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:38 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:44:38.409 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-20T09:44:38.484 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:44:38.530 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:38.564 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:44:38.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:38.568 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:44:38.669 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:44:38.711 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-20T09:44:38.753 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:38.799 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:44:38.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:38.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:44:38.824 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:44:38.913 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:44:38.955 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:38.989 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:44:39.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:39.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:44:39.040 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd set noscrub 2024-03-20T09:44:39.102 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:44:39.133 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:39.164 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:44:39.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:39.170 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:44:39.271 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:44:39.302 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:39.332 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:44:39.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:39.338 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:44:39.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:39 smithi115 ceph-mon[72476]: 1.12 scrub ok 2024-03-20T09:44:39.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:39 smithi115 ceph-mon[72476]: 1.11 scrub starts 2024-03-20T09:44:39.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:39 smithi115 ceph-mon[72476]: 9.0 scrub starts 2024-03-20T09:44:39.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:39 smithi115 ceph-mon[72476]: 1.11 scrub ok 2024-03-20T09:44:39.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:39 smithi115 ceph-mon[72476]: 9.0 scrub ok 2024-03-20T09:44:39.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:39 smithi115 ceph-mon[72476]: osdmap e1143: 12 total, 12 up, 11 in 2024-03-20T09:44:39.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:39 smithi115 ceph-mon[72476]: pgmap v2784: 47 pgs: 47 active+clean; 289 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 734 op/s 2024-03-20T09:44:39.393 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:39 smithi026 ceph-mon[68309]: 1.12 scrub ok 2024-03-20T09:44:39.393 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:39 smithi026 ceph-mon[68309]: 1.11 scrub starts 2024-03-20T09:44:39.394 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:39 smithi026 ceph-mon[68309]: 9.0 scrub starts 2024-03-20T09:44:39.394 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:39 smithi026 ceph-mon[68309]: 1.11 scrub ok 2024-03-20T09:44:39.394 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:39 smithi026 ceph-mon[68309]: 9.0 scrub ok 2024-03-20T09:44:39.394 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:39 smithi026 ceph-mon[68309]: osdmap e1143: 12 total, 12 up, 11 in 2024-03-20T09:44:39.394 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:39 smithi026 ceph-mon[68309]: pgmap v2784: 47 pgs: 47 active+clean; 289 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 734 op/s 2024-03-20T09:44:39.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:39 smithi122 ceph-mon[73495]: 1.12 scrub ok 2024-03-20T09:44:39.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:39 smithi122 ceph-mon[73495]: 1.11 scrub starts 2024-03-20T09:44:39.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:39 smithi122 ceph-mon[73495]: 9.0 scrub starts 2024-03-20T09:44:39.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:39 smithi122 ceph-mon[73495]: 1.11 scrub ok 2024-03-20T09:44:39.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:39 smithi122 ceph-mon[73495]: 9.0 scrub ok 2024-03-20T09:44:39.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:39 smithi122 ceph-mon[73495]: osdmap e1143: 12 total, 12 up, 11 in 2024-03-20T09:44:39.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:39 smithi122 ceph-mon[73495]: pgmap v2784: 47 pgs: 47 active+clean; 289 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 734 op/s 2024-03-20T09:44:39.439 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:44:39.465 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:39.485 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:44:39.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:39.488 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:44:39.589 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:44:39.618 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:39.671 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:44:39.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:39.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:44:39.775 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:44:39.830 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:39.869 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:44:39.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:39.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:44:39.978 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:44:40.019 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:40.113 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:40 smithi115 ceph-mon[72476]: 1.1 scrub starts 2024-03-20T09:44:40.113 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:40 smithi115 ceph-mon[72476]: 1.1 scrub ok 2024-03-20T09:44:40.407 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:40 smithi026 ceph-mon[68309]: 1.1 scrub starts 2024-03-20T09:44:40.407 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:40 smithi026 ceph-mon[68309]: 1.1 scrub ok 2024-03-20T09:44:40.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:40 smithi122 ceph-mon[73495]: 1.1 scrub starts 2024-03-20T09:44:40.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:40 smithi122 ceph-mon[73495]: 1.1 scrub ok 2024-03-20T09:44:40.447 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:44:40.485 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:44:40.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:40.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:44:40.590 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:44:40.625 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:40.703 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:44:40.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:40.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:44:40.785 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Removed 132227 objects 2024-03-20T09:44:40.785 INFO:tasks.radosbench.radosbench.2.smithi122.stdout:Clean up completed and total clean up time :149.93 2024-03-20T09:44:40.808 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:44:40.836 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:40.960 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:44:40.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:40.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:44:41.064 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:44:41.090 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:41.109 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:44:41.112 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:41.112 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:44:41.213 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:44:41.240 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:41 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:44:41.240 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:41 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:44:41.240 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:41 smithi026 ceph-mon[68309]: osdmap e1144: 12 total, 12 up, 11 in 2024-03-20T09:44:41.240 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:41 smithi026 ceph-mon[68309]: 1.5 scrub starts 2024-03-20T09:44:41.240 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:41 smithi026 ceph-mon[68309]: 1.5 scrub ok 2024-03-20T09:44:41.240 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:41 smithi026 ceph-mon[68309]: pgmap v2786: 47 pgs: 47 active+clean; 93 MiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s 2024-03-20T09:44:41.243 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:41.392 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:44:41.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:41 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:44:41.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:41 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:44:41.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:41 smithi115 ceph-mon[72476]: osdmap e1144: 12 total, 12 up, 11 in 2024-03-20T09:44:41.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:41 smithi115 ceph-mon[72476]: 1.5 scrub starts 2024-03-20T09:44:41.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:41 smithi115 ceph-mon[72476]: 1.5 scrub ok 2024-03-20T09:44:41.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:41 smithi115 ceph-mon[72476]: pgmap v2786: 47 pgs: 47 active+clean; 93 MiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s 2024-03-20T09:44:41.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:41.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:44:41.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:41 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:44:41.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:41 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:44:41.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:41 smithi122 ceph-mon[73495]: osdmap e1144: 12 total, 12 up, 11 in 2024-03-20T09:44:41.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:41 smithi122 ceph-mon[73495]: 1.5 scrub starts 2024-03-20T09:44:41.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:41 smithi122 ceph-mon[73495]: 1.5 scrub ok 2024-03-20T09:44:41.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:41 smithi122 ceph-mon[73495]: pgmap v2786: 47 pgs: 47 active+clean; 93 MiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s 2024-03-20T09:44:41.496 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:44:41.529 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:41.550 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:44:41.552 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:41.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:44:41.613 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:44:41.613 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-03-20T09:44:41.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-03-20T09:44:41.613 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [] 2024-03-20T09:44:41.613 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:44:41.654 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:44:41.682 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:41.882 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:44:41.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:41.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:44:41.986 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:44:42.013 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:42.278 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 5, 6, 7, 8, 11, 3, 4, 9, 10] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 8, 9, 11, 5, 10] 2024-03-20T09:44:42.279 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-20T09:44:42.279 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-03-20T09:44:42.279 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd in 0 2024-03-20T09:44:42.306 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:42 smithi026 ceph-mon[68309]: 1.8 scrub starts 2024-03-20T09:44:42.306 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:42 smithi026 ceph-mon[68309]: 1.8 scrub ok 2024-03-20T09:44:42.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:42 smithi115 ceph-mon[72476]: 1.8 scrub starts 2024-03-20T09:44:42.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:42 smithi115 ceph-mon[72476]: 1.8 scrub ok 2024-03-20T09:44:42.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:42 smithi122 ceph-mon[73495]: 1.8 scrub starts 2024-03-20T09:44:42.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:42 smithi122 ceph-mon[73495]: 1.8 scrub ok 2024-03-20T09:44:43.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:43 smithi115 ceph-mon[72476]: pgmap v2787: 47 pgs: 47 active+clean; 33 MiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 697 op/s 2024-03-20T09:44:43.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:43 smithi122 ceph-mon[73495]: pgmap v2787: 47 pgs: 47 active+clean; 33 MiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 697 op/s 2024-03-20T09:44:43.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:43 smithi026 ceph-mon[68309]: pgmap v2787: 47 pgs: 47 active+clean; 33 MiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 697 op/s 2024-03-20T09:44:43.809 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_7 2024-03-20T09:44:43.809 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pool rm unique_pool_7 unique_pool_7 --yes-i-really-really-mean-it 2024-03-20T09:44:44.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:44 smithi115 ceph-mon[72476]: 9.9 scrub starts 2024-03-20T09:44:44.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:44 smithi115 ceph-mon[72476]: 9.9 scrub ok 2024-03-20T09:44:44.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:44 smithi122 ceph-mon[73495]: 9.9 scrub starts 2024-03-20T09:44:44.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:44 smithi122 ceph-mon[73495]: 9.9 scrub ok 2024-03-20T09:44:44.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:44 smithi026 ceph-mon[68309]: 9.9 scrub starts 2024-03-20T09:44:44.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:44 smithi026 ceph-mon[68309]: 9.9 scrub ok 2024-03-20T09:44:45.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:45 smithi115 ceph-mon[72476]: 9.7 scrub starts 2024-03-20T09:44:45.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:45 smithi115 ceph-mon[72476]: 9.7 scrub ok 2024-03-20T09:44:45.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:45 smithi115 ceph-mon[72476]: pgmap v2788: 47 pgs: 47 active+clean; 9.2 MiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 590 op/s 2024-03-20T09:44:45.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:45 smithi122 ceph-mon[73495]: 9.7 scrub starts 2024-03-20T09:44:45.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:45 smithi122 ceph-mon[73495]: 9.7 scrub ok 2024-03-20T09:44:45.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:45 smithi122 ceph-mon[73495]: pgmap v2788: 47 pgs: 47 active+clean; 9.2 MiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 590 op/s 2024-03-20T09:44:45.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:45 smithi026 ceph-mon[68309]: 9.7 scrub starts 2024-03-20T09:44:45.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:45 smithi026 ceph-mon[68309]: 9.7 scrub ok 2024-03-20T09:44:45.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:45 smithi026 ceph-mon[68309]: pgmap v2788: 47 pgs: 47 active+clean; 9.2 MiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 590 op/s 2024-03-20T09:44:45.496 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-20T09:44:45.545 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:44:45.566 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:44:45.800 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:44:45.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:45.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:44:45.904 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:44:45.937 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:45.959 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:44:45.961 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:45.962 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:44:46.063 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:44:46.096 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:46.117 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:44:46.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:46.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:44:46.221 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:44:46.252 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:46.802 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:44:46.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:46.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:44:46.911 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:44:46.939 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:46.960 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:44:46.962 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:46.963 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:44:47.064 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:44:47.090 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:46 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:44:47.092 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:47.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:46 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:44:47.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:46 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:44:47.231 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:44:47.251 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:44:47.270 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:44:47.290 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:44:47.310 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:44:47.330 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:44:47.350 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:44:47.370 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:44:47.391 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:44:47.411 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:44:47.431 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:44:47.442 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:44:47.445 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:47.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:44:47.453 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:44:47.455 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:44:47.455 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:44:47.455 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:44:47.455 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:44:47.455 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:44:47.455 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:44:47.455 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:44:47.455 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:44:47.455 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:44:47.455 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:44:47.456 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:44:47.456 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:44:47.467 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:44:47.547 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:44:47.576 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:47.596 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:44:47.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:47.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:44:47.700 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:44:47.730 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:47.749 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:44:47.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:47.752 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:44:47.853 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:44:47.878 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:47 smithi115 ceph-mon[72476]: pgmap v2789: 47 pgs: 47 active+clean; 577 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 534 op/s 2024-03-20T09:44:47.878 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:47 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3540526289' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:44:47.878 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:47 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:44:47.878 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:47 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:44:47.878 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:47 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:44:47.880 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:47.900 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:44:47.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:47.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:44:48.003 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:44:48.029 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:48.048 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:44:48.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:48.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:44:48.058 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:47 smithi026 ceph-mon[68309]: pgmap v2789: 47 pgs: 47 active+clean; 577 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 534 op/s 2024-03-20T09:44:48.058 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:47 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3540526289' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:44:48.058 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:47 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:44:48.058 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:47 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:44:48.058 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:47 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:44:48.151 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:44:48.177 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:48.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:47 smithi122 ceph-mon[73495]: pgmap v2789: 47 pgs: 47 active+clean; 577 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 534 op/s 2024-03-20T09:44:48.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:47 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3540526289' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:44:48.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:47 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:44:48.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:47 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:44:48.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:47 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:44:48.195 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:44:48.198 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:48.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:44:48.299 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:44:48.325 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:48.343 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:44:48.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:48.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:44:48.446 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2 2024-03-20T09:44:48.477 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:48.875 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2" found: no such container 2024-03-20T09:44:48.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:48.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.2' 2024-03-20T09:44:48.980 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:44:49.008 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:49.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:49 smithi026 ceph-mon[68309]: 1.4 scrub starts 2024-03-20T09:44:49.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:49 smithi026 ceph-mon[68309]: 1.4 scrub ok 2024-03-20T09:44:49.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:49 smithi026 ceph-mon[68309]: pgmap v2790: 47 pgs: 47 active+clean; 577 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 460 op/s 2024-03-20T09:44:49.257 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:44:49.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:49.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:44:49.315 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:44:49.362 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:44:49.390 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:49.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:49 smithi115 ceph-mon[72476]: 1.4 scrub starts 2024-03-20T09:44:49.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:49 smithi115 ceph-mon[72476]: 1.4 scrub ok 2024-03-20T09:44:49.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:49 smithi115 ceph-mon[72476]: pgmap v2790: 47 pgs: 47 active+clean; 577 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 460 op/s 2024-03-20T09:44:49.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:49 smithi122 ceph-mon[73495]: 1.4 scrub starts 2024-03-20T09:44:49.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:49 smithi122 ceph-mon[73495]: 1.4 scrub ok 2024-03-20T09:44:49.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:49 smithi122 ceph-mon[73495]: pgmap v2790: 47 pgs: 47 active+clean; 577 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 460 op/s 2024-03-20T09:44:49.539 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:44:49.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:49.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:44:49.643 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:44:49.671 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:49.819 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:44:49.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:49.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:44:49.924 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:44:49.957 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:49.979 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:44:49.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:49.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:44:50.083 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:44:50.112 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:50.132 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:44:50.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:50.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:44:50.236 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6 2024-03-20T09:44:50.264 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:50.284 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6" found: no such container 2024-03-20T09:44:50.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:50.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.6' 2024-03-20T09:44:50.388 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10 2024-03-20T09:44:50.420 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:50.441 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10" found: no such container 2024-03-20T09:44:50.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:50.444 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.10' 2024-03-20T09:44:50.544 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8 2024-03-20T09:44:50.574 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:50.594 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8" found: no such container 2024-03-20T09:44:50.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:50.597 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.8' 2024-03-20T09:44:50.698 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:44:50.727 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:50.748 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:44:50.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:50.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:44:50.799 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is set 2024-03-20T09:44:50.851 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11 2024-03-20T09:44:50.884 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:50.894 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-20T09:44:50.905 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11" found: no such container 2024-03-20T09:44:50.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:50.908 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.11' 2024-03-20T09:44:51.010 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:44:51.038 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:50 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4217306077' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:44:51.041 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:51.062 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:44:51.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:51.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:44:51.166 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:44:51.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:50 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4217306077' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:44:51.193 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:50 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4217306077' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-20T09:44:51.195 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:51.216 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:44:51.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:51.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:44:51.320 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1 2024-03-20T09:44:51.349 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:51.372 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1" found: no such container 2024-03-20T09:44:51.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:51.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.1' 2024-03-20T09:44:51.476 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:44:51.507 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:51.528 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:44:51.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:51.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:44:51.632 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:44:51.662 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:51.877 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:51 smithi026 ceph-mon[68309]: pgmap v2791: 47 pgs: 47 active+clean; 577 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 305 op/s 2024-03-20T09:44:51.877 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:51 smithi026 ceph-mon[68309]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:44:51.878 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:51 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4217306077' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:44:51.878 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:51 smithi026 ceph-mon[68309]: osdmap e1145: 12 total, 12 up, 11 in 2024-03-20T09:44:52.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:51 smithi115 ceph-mon[72476]: pgmap v2791: 47 pgs: 47 active+clean; 577 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 305 op/s 2024-03-20T09:44:52.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:51 smithi115 ceph-mon[72476]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:44:52.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:51 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4217306077' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:44:52.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:51 smithi115 ceph-mon[72476]: osdmap e1145: 12 total, 12 up, 11 in 2024-03-20T09:44:52.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:51 smithi122 ceph-mon[73495]: pgmap v2791: 47 pgs: 47 active+clean; 577 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 305 op/s 2024-03-20T09:44:52.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:51 smithi122 ceph-mon[73495]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-20T09:44:52.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:51 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4217306077' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-20T09:44:52.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:51 smithi122 ceph-mon[73495]: osdmap e1145: 12 total, 12 up, 11 in 2024-03-20T09:44:52.513 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:44:52.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:52.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:44:52.612 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:44:52.617 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:44:52.650 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:52.672 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:44:52.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:52.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:44:52.707 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:52 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mon-a[68286]: 2024-03-20T09:44:52.298+0000 7f683364b700 -1 mon.a@0(leader).osd e1145 definitely_dead 0 2024-03-20T09:44:52.776 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7 2024-03-20T09:44:52.808 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:52.814 INFO:teuthology.orchestra.run.smithi026.stderr:marked in osd.0. 2024-03-20T09:44:52.829 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7" found: no such container 2024-03-20T09:44:52.831 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:52.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.7' 2024-03-20T09:44:52.933 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:44:52.959 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:52 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2787187608' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-03-20T09:44:52.962 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:52.981 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:44:52.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:52.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:44:53.039 INFO:teuthology.orchestra.run.smithi026.stderr:pool 'unique_pool_7' removed 2024-03-20T09:44:53.071 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:52 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2787187608' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-03-20T09:44:53.085 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4 2024-03-20T09:44:53.124 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:53.148 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4" found: no such container 2024-03-20T09:44:53.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:53.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.4' 2024-03-20T09:44:53.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:52 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2787187608' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-03-20T09:44:53.253 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3 2024-03-20T09:44:53.284 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:53.384 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3" found: no such container 2024-03-20T09:44:53.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:53.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.3' 2024-03-20T09:44:53.488 DEBUG:teuthology.orchestra.run.smithi122:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9 2024-03-20T09:44:53.515 INFO:teuthology.orchestra.run.smithi122.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:53.534 INFO:teuthology.orchestra.run.smithi122.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9" found: no such container 2024-03-20T09:44:53.536 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:53.536 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi122 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.9' 2024-03-20T09:44:53.637 DEBUG:teuthology.orchestra.run.smithi115:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5 2024-03-20T09:44:53.664 INFO:teuthology.orchestra.run.smithi115.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:53.683 INFO:teuthology.orchestra.run.smithi115.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5" found: no such container 2024-03-20T09:44:53.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:53.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi115 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.5' 2024-03-20T09:44:53.786 DEBUG:teuthology.orchestra.run.smithi026:> sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0 2024-03-20T09:44:53.816 INFO:teuthology.orchestra.run.smithi026.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-03-20T09:44:53.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:53 smithi026 ceph-mon[68309]: pgmap v2793: 47 pgs: 47 active+clean; 577 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 52 op/s 2024-03-20T09:44:53.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:53 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2787187608' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-03-20T09:44:53.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:53 smithi026 ceph-mon[68309]: osdmap e1146: 12 total, 12 up, 12 in 2024-03-20T09:44:53.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:53 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2103275498' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-20T09:44:53.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:53 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2103275498' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-03-20T09:44:53.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:53 smithi026 ceph-mon[68309]: osdmap e1147: 12 total, 12 up, 12 in 2024-03-20T09:44:54.096 INFO:teuthology.orchestra.run.smithi026.stderr:Error: no container with name or ID "ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0" found: no such container 2024-03-20T09:44:54.099 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-03-20T09:44:54.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi026 with status 125: 'sudo docker kill -s 1 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd.0' 2024-03-20T09:44:54.113 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-03-20T09:44:54.123 INFO:tasks.thrashosds:joining thrashosds 2024-03-20T09:44:54.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:53 smithi115 ceph-mon[72476]: pgmap v2793: 47 pgs: 47 active+clean; 577 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 52 op/s 2024-03-20T09:44:54.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:53 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2787187608' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-03-20T09:44:54.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:53 smithi115 ceph-mon[72476]: osdmap e1146: 12 total, 12 up, 12 in 2024-03-20T09:44:54.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:53 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2103275498' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-20T09:44:54.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:53 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2103275498' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-03-20T09:44:54.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:53 smithi115 ceph-mon[72476]: osdmap e1147: 12 total, 12 up, 12 in 2024-03-20T09:44:54.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:53 smithi122 ceph-mon[73495]: pgmap v2793: 47 pgs: 47 active+clean; 577 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 52 op/s 2024-03-20T09:44:54.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:53 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2787187608' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-03-20T09:44:54.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:53 smithi122 ceph-mon[73495]: osdmap e1146: 12 total, 12 up, 12 in 2024-03-20T09:44:54.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:53 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2103275498' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-20T09:44:54.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:53 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2103275498' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-03-20T09:44:54.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:53 smithi122 ceph-mon[73495]: osdmap e1147: 12 total, 12 up, 12 in 2024-03-20T09:44:54.410 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset noscrub 2024-03-20T09:44:55.380 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:55 smithi026 ceph-mon[68309]: osdmap e1148: 12 total, 12 up, 12 in 2024-03-20T09:44:55.380 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:55 smithi026 ceph-mon[68309]: pgmap v2797: 31 pgs: 1 active+undersized+remapped, 30 active+clean; 577 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail 2024-03-20T09:44:55.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:55 smithi115 ceph-mon[72476]: osdmap e1148: 12 total, 12 up, 12 in 2024-03-20T09:44:55.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:55 smithi115 ceph-mon[72476]: pgmap v2797: 31 pgs: 1 active+undersized+remapped, 30 active+clean; 577 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail 2024-03-20T09:44:55.409 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-03-20T09:44:55.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:55 smithi122 ceph-mon[73495]: osdmap e1148: 12 total, 12 up, 12 in 2024-03-20T09:44:55.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:55 smithi122 ceph-mon[73495]: pgmap v2797: 31 pgs: 1 active+undersized+remapped, 30 active+clean; 577 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail 2024-03-20T09:44:56.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:56 smithi115 ceph-mon[72476]: osdmap e1149: 12 total, 12 up, 12 in 2024-03-20T09:44:56.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:56 smithi122 ceph-mon[73495]: osdmap e1149: 12 total, 12 up, 12 in 2024-03-20T09:44:56.435 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:56 smithi026 ceph-mon[68309]: osdmap e1149: 12 total, 12 up, 12 in 2024-03-20T09:44:56.471 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:44:57.380 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:57 smithi026 ceph-mon[68309]: osdmap e1150: 12 total, 12 up, 12 in 2024-03-20T09:44:57.380 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:57 smithi026 ceph-mon[68309]: pgmap v2800: 31 pgs: 1 peering, 2 active+undersized+remapped, 28 active+clean; 577 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-20T09:44:57.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:57 smithi115 ceph-mon[72476]: osdmap e1150: 12 total, 12 up, 12 in 2024-03-20T09:44:57.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:57 smithi115 ceph-mon[72476]: pgmap v2800: 31 pgs: 1 peering, 2 active+undersized+remapped, 28 active+clean; 577 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-20T09:44:57.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:57 smithi122 ceph-mon[73495]: osdmap e1150: 12 total, 12 up, 12 in 2024-03-20T09:44:57.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:57 smithi122 ceph-mon[73495]: pgmap v2800: 31 pgs: 1 peering, 2 active+undersized+remapped, 28 active+clean; 577 KiB data, 9.2 GiB used, 974 GiB / 983 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-20T09:44:57.731 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0: {} 2024-03-20T09:44:57.731 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1: {} 2024-03-20T09:44:57.731 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2: {} 2024-03-20T09:44:57.731 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3: {} 2024-03-20T09:44:57.731 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4: {} 2024-03-20T09:44:57.731 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5: {} 2024-03-20T09:44:57.731 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6: {} 2024-03-20T09:44:57.731 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7: {} 2024-03-20T09:44:57.731 INFO:teuthology.orchestra.run.smithi026.stdout:osd.8: {} 2024-03-20T09:44:57.731 INFO:teuthology.orchestra.run.smithi026.stdout:osd.9: {} 2024-03-20T09:44:57.731 INFO:teuthology.orchestra.run.smithi026.stdout:osd.10: {} 2024-03-20T09:44:57.732 INFO:teuthology.orchestra.run.smithi026.stdout:osd.11: {} 2024-03-20T09:44:57.732 INFO:teuthology.orchestra.run.smithi026.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-20T09:44:57.732 INFO:teuthology.orchestra.run.smithi026.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-20T09:44:57.732 INFO:teuthology.orchestra.run.smithi026.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-20T09:44:57.732 INFO:teuthology.orchestra.run.smithi026.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-20T09:44:57.732 INFO:teuthology.orchestra.run.smithi026.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-20T09:44:57.732 INFO:teuthology.orchestra.run.smithi026.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-20T09:44:57.732 INFO:teuthology.orchestra.run.smithi026.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-20T09:44:57.732 INFO:teuthology.orchestra.run.smithi026.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-20T09:44:57.732 INFO:teuthology.orchestra.run.smithi026.stderr:osd.8: osd_enable_op_tracker = '' 2024-03-20T09:44:57.732 INFO:teuthology.orchestra.run.smithi026.stderr:osd.9: osd_enable_op_tracker = '' 2024-03-20T09:44:57.732 INFO:teuthology.orchestra.run.smithi026.stderr:osd.10: osd_enable_op_tracker = '' 2024-03-20T09:44:57.733 INFO:teuthology.orchestra.run.smithi026.stderr:osd.11: osd_enable_op_tracker = '' 2024-03-20T09:44:58.136 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:44:58.136 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:44:58.136 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-03-20T09:44:58.136 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:44:58.276 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:58 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2884830127' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:44:58.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:58 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2884830127' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:44:58.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:58 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2884830127' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-20T09:44:59.284 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:59 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:44:59.284 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:59 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:44:59.284 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:59 smithi026 ceph-mon[68309]: osdmap e1151: 12 total, 12 up, 12 in 2024-03-20T09:44:59.284 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:59 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:44:59.284 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:59 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:44:59.284 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:59 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:44:59.284 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:59 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:44:59.284 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:59 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:44:59.285 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:59 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:44:59.285 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:44:59 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:44:59.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:59 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:44:59.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:59 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:44:59.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:59 smithi115 ceph-mon[72476]: osdmap e1151: 12 total, 12 up, 12 in 2024-03-20T09:44:59.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:59 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:44:59.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:59 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:44:59.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:59 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:44:59.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:59 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:44:59.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:59 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:44:59.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:59 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:44:59.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:44:59 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:44:59.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:59 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:44:59.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:59 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:44:59.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:59 smithi122 ceph-mon[73495]: osdmap e1151: 12 total, 12 up, 12 in 2024-03-20T09:44:59.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:59 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:44:59.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:59 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:44:59.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:59 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:44:59.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:59 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:44:59.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:59 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:44:59.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:59 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:44:59.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:44:59 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:45:00.036 INFO:teuthology.orchestra.run.smithi026.stderr:noscrub is unset 2024-03-20T09:45:00.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:00 smithi026 ceph-mon[68309]: from='client.17673 172.21.15.26:0/3202771271' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:45:00.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:00 smithi115 ceph-mon[72476]: from='client.17673 172.21.15.26:0/3202771271' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:45:00.410 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-20T09:45:00.410 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd pool set .mgr pgp_num 41 2024-03-20T09:45:00.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:00 smithi122 ceph-mon[73495]: from='client.17673 172.21.15.26:0/3202771271' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-20T09:45:00.949 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-20T09:45:01.125 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-20T09:45:01.136 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:45:01.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:01 smithi026 ceph-mon[68309]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-03-20T09:45:01.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:01 smithi026 ceph-mon[68309]: Cluster is now healthy 2024-03-20T09:45:01.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:01 smithi026 ceph-mon[68309]: from='client.17673 172.21.15.26:0/3202771271' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:45:01.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:01 smithi026 ceph-mon[68309]: osdmap e1152: 12 total, 12 up, 12 in 2024-03-20T09:45:01.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:01 smithi026 ceph-mon[68309]: pgmap v2804: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s, 0 objects/s recovering 2024-03-20T09:45:01.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:01 smithi115 ceph-mon[72476]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-03-20T09:45:01.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:01 smithi115 ceph-mon[72476]: Cluster is now healthy 2024-03-20T09:45:01.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:01 smithi115 ceph-mon[72476]: from='client.17673 172.21.15.26:0/3202771271' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:45:01.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:01 smithi115 ceph-mon[72476]: osdmap e1152: 12 total, 12 up, 12 in 2024-03-20T09:45:01.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:01 smithi115 ceph-mon[72476]: pgmap v2804: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s, 0 objects/s recovering 2024-03-20T09:45:01.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:01 smithi122 ceph-mon[73495]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-03-20T09:45:01.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:01 smithi122 ceph-mon[73495]: Cluster is now healthy 2024-03-20T09:45:01.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:01 smithi122 ceph-mon[73495]: from='client.17673 172.21.15.26:0/3202771271' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-20T09:45:01.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:01 smithi122 ceph-mon[73495]: osdmap e1152: 12 total, 12 up, 12 in 2024-03-20T09:45:01.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:01 smithi122 ceph-mon[73495]: pgmap v2804: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s, 0 objects/s recovering 2024-03-20T09:45:01.937 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:45:01.938 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:45:03.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:03 smithi026 ceph-mon[68309]: 1.f scrub starts 2024-03-20T09:45:03.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:03 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:45:03.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:03 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:45:03.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:03 smithi026 ceph-mon[68309]: 1.f scrub ok 2024-03-20T09:45:03.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:03 smithi026 ceph-mon[68309]: osdmap e1153: 12 total, 12 up, 12 in 2024-03-20T09:45:03.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:03 smithi026 ceph-mon[68309]: pgmap v2806: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:03.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:03 smithi115 ceph-mon[72476]: 1.f scrub starts 2024-03-20T09:45:03.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:03 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:45:03.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:03 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:45:03.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:03 smithi115 ceph-mon[72476]: 1.f scrub ok 2024-03-20T09:45:03.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:03 smithi115 ceph-mon[72476]: osdmap e1153: 12 total, 12 up, 12 in 2024-03-20T09:45:03.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:03 smithi115 ceph-mon[72476]: pgmap v2806: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:03.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:03 smithi122 ceph-mon[73495]: 1.f scrub starts 2024-03-20T09:45:03.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:03 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:45:03.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:03 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:45:03.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:03 smithi122 ceph-mon[73495]: 1.f scrub ok 2024-03-20T09:45:03.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:03 smithi122 ceph-mon[73495]: osdmap e1153: 12 total, 12 up, 12 in 2024-03-20T09:45:03.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:03 smithi122 ceph-mon[73495]: pgmap v2806: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:04.288 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:04 smithi026 ceph-mon[68309]: 1.15 scrub starts 2024-03-20T09:45:04.288 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:04 smithi026 ceph-mon[68309]: 1.15 scrub ok 2024-03-20T09:45:04.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:04 smithi115 ceph-mon[72476]: 1.15 scrub starts 2024-03-20T09:45:04.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:04 smithi115 ceph-mon[72476]: 1.15 scrub ok 2024-03-20T09:45:04.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:04 smithi122 ceph-mon[73495]: 1.15 scrub starts 2024-03-20T09:45:04.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:04 smithi122 ceph-mon[73495]: 1.15 scrub ok 2024-03-20T09:45:04.486 INFO:teuthology.orchestra.run.smithi026.stderr:Error EINVAL: specified pgp_num 41 > pg_num 31 2024-03-20T09:45:05.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:05 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2958750824' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "41"}]: dispatch 2024-03-20T09:45:05.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:05 smithi026 ceph-mon[68309]: pgmap v2807: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:05.298 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-20T09:45:05.299 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-03-20T09:45:05.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:05 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2958750824' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "41"}]: dispatch 2024-03-20T09:45:05.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:05 smithi115 ceph-mon[72476]: pgmap v2807: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:05.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:05 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2958750824' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "41"}]: dispatch 2024-03-20T09:45:05.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:05 smithi122 ceph-mon[73495]: pgmap v2807: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:06.777 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:45:06.802 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:45:06.802 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:45:06.802 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-03-20T09:45:06.802 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-03-20T09:45:06.802 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:45:07.058 INFO:teuthology.orchestra.run.smithi026.stderr:nodeep-scrub is unset 2024-03-20T09:45:07.079 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:07 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:45:07.079 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:07 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:45:07.079 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:07 smithi026 ceph-mon[68309]: osdmap e1154: 12 total, 12 up, 12 in 2024-03-20T09:45:07.079 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:07 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1043347814' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:45:07.080 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:07 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:45:07.080 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:07 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:45:07.080 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:07 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:45:07.080 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:07 smithi026 ceph-mon[68309]: pgmap v2809: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:07.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:07 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:45:07.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:07 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:45:07.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:07 smithi115 ceph-mon[72476]: osdmap e1154: 12 total, 12 up, 12 in 2024-03-20T09:45:07.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:07 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1043347814' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:45:07.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:07 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:45:07.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:07 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:45:07.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:07 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:45:07.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:07 smithi115 ceph-mon[72476]: pgmap v2809: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:07.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:07 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:45:07.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:07 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:45:07.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:07 smithi122 ceph-mon[73495]: osdmap e1154: 12 total, 12 up, 12 in 2024-03-20T09:45:07.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:07 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1043347814' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:45:07.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:07 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-20T09:45:07.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:07 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:45:07.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:07 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:45:07.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:07 smithi122 ceph-mon[73495]: pgmap v2809: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:08.362 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:08 smithi026 ceph-mon[68309]: 1.d scrub starts 2024-03-20T09:45:08.363 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:08 smithi026 ceph-mon[68309]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:45:08.363 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:08 smithi026 ceph-mon[68309]: osdmap e1155: 12 total, 12 up, 12 in 2024-03-20T09:45:08.363 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:08 smithi026 ceph-mon[68309]: 1.d scrub ok 2024-03-20T09:45:08.363 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:08 smithi026 ceph-mon[68309]: 1.18 scrub starts 2024-03-20T09:45:08.363 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:08 smithi026 ceph-mon[68309]: 1.18 scrub ok 2024-03-20T09:45:08.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:08 smithi115 ceph-mon[72476]: 1.d scrub starts 2024-03-20T09:45:08.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:08 smithi115 ceph-mon[72476]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:45:08.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:08 smithi115 ceph-mon[72476]: osdmap e1155: 12 total, 12 up, 12 in 2024-03-20T09:45:08.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:08 smithi115 ceph-mon[72476]: 1.d scrub ok 2024-03-20T09:45:08.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:08 smithi115 ceph-mon[72476]: 1.18 scrub starts 2024-03-20T09:45:08.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:08 smithi115 ceph-mon[72476]: 1.18 scrub ok 2024-03-20T09:45:08.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:08 smithi122 ceph-mon[73495]: 1.d scrub starts 2024-03-20T09:45:08.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:08 smithi122 ceph-mon[73495]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-20T09:45:08.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:08 smithi122 ceph-mon[73495]: osdmap e1155: 12 total, 12 up, 12 in 2024-03-20T09:45:08.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:08 smithi122 ceph-mon[73495]: 1.d scrub ok 2024-03-20T09:45:08.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:08 smithi122 ceph-mon[73495]: 1.18 scrub starts 2024-03-20T09:45:08.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:08 smithi122 ceph-mon[73495]: 1.18 scrub ok 2024-03-20T09:45:08.763 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-20T09:45:09.319 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:09 smithi026 ceph-mon[68309]: 1.1d scrub starts 2024-03-20T09:45:09.320 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:09 smithi026 ceph-mon[68309]: 1.1d scrub ok 2024-03-20T09:45:09.320 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:09 smithi026 ceph-mon[68309]: pgmap v2811: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:09.385 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:45:09.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:09 smithi115 ceph-mon[72476]: 1.1d scrub starts 2024-03-20T09:45:09.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:09 smithi115 ceph-mon[72476]: 1.1d scrub ok 2024-03-20T09:45:09.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:09 smithi115 ceph-mon[72476]: pgmap v2811: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:09.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:09 smithi122 ceph-mon[73495]: 1.1d scrub starts 2024-03-20T09:45:09.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:09 smithi122 ceph-mon[73495]: 1.1d scrub ok 2024-03-20T09:45:09.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:09 smithi122 ceph-mon[73495]: pgmap v2811: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:10.021 INFO:teuthology.orchestra.run.smithi026.stdout:mon.a: {} 2024-03-20T09:45:10.031 INFO:teuthology.orchestra.run.smithi026.stdout:mon.c: {} 2024-03-20T09:45:10.042 INFO:teuthology.orchestra.run.smithi026.stdout:mon.b: {} 2024-03-20T09:45:10.043 INFO:teuthology.orchestra.run.smithi026.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-20T09:45:10.043 INFO:teuthology.orchestra.run.smithi026.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-20T09:45:10.044 INFO:teuthology.orchestra.run.smithi026.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-20T09:45:10.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:10 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1862767785' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-20T09:45:10.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:10 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1862767785' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-20T09:45:10.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:10 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1862767785' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-20T09:45:11.106 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 0 1 2024-03-20T09:45:11.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:11 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:45:11.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:11 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:45:11.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:11 smithi115 ceph-mon[72476]: osdmap e1156: 12 total, 12 up, 12 in 2024-03-20T09:45:11.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:11 smithi115 ceph-mon[72476]: pgmap v2813: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:11.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:11 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:45:11.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:11 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:45:11.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:11 smithi122 ceph-mon[73495]: osdmap e1156: 12 total, 12 up, 12 in 2024-03-20T09:45:11.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:11 smithi122 ceph-mon[73495]: pgmap v2813: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:11.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:11 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:45:11.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:11 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:45:11.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:11 smithi026 ceph-mon[68309]: osdmap e1156: 12 total, 12 up, 12 in 2024-03-20T09:45:11.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:11 smithi026 ceph-mon[68309]: pgmap v2813: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:12.112 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:45:12.112 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-03-20T09:45:12.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-03-20T09:45:12.113 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:45:12.113 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.113 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26917.0:106346 8.1 e1007/974 8:8ecbb4fe:::benchmark_data_smithi122_197775_object106345:head v 1007'8838, mlcod=1007'8836)", 2024-03-20T09:45:12.113 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:37:11.898094+0000", 2024-03-20T09:45:12.113 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 480.22545159700002, 2024-03-20T09:45:12.113 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.11849124799999999, 2024-03-20T09:45:12.113 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:12.114 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:12.114 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:12.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:12.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.898094+0000", 2024-03-20T09:45:12.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.114 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.114 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:12.114 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.898094+0000", 2024-03-20T09:45:12.114 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.114 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:12.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.898095+0000", 2024-03-20T09:45:12.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4799999999999999e-07 2024-03-20T09:45:12.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.115 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.115 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:12.115 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.898134+0000", 2024-03-20T09:45:12.115 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8902000000000001e-05 2024-03-20T09:45:12.115 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:12.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.898150+0000", 2024-03-20T09:45:12.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6753e-05 2024-03-20T09:45:12.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:12.116 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.898159+0000", 2024-03-20T09:45:12.116 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3690000000000007e-06 2024-03-20T09:45:12.116 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.116 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.116 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:12.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.902438+0000", 2024-03-20T09:45:12.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0042786630000000003 2024-03-20T09:45:12.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:12.117 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.902475+0000", 2024-03-20T09:45:12.117 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7265000000000003e-05 2024-03-20T09:45:12.117 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.117 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.117 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:12.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:12.016542+0000", 2024-03-20T09:45:12.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.114067627 2024-03-20T09:45:12.118 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.118 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.118 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:12.118 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:12.016586+0000", 2024-03-20T09:45:12.118 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3321000000000001e-05 2024-03-20T09:45:12.118 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.118 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:12.118 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.119 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.119 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.9.0:49921 8.1 e1007/974 8:889d441c:::benchmark_data_smithi122_197775_object68126:head v 1007'8839, mlcod=1007'8836)", 2024-03-20T09:45:12.119 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:37:11.908320+0000", 2024-03-20T09:45:12.119 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 480.21522580800001, 2024-03-20T09:45:12.119 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.108292439, 2024-03-20T09:45:12.119 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:12.119 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:12.119 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:12.119 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:12.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.908320+0000", 2024-03-20T09:45:12.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.120 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.120 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:12.120 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.908320+0000", 2024-03-20T09:45:12.120 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.120 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:12.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.908321+0000", 2024-03-20T09:45:12.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.6899999999999996e-07 2024-03-20T09:45:12.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:12.121 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.908329+0000", 2024-03-20T09:45:12.121 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.011e-06 2024-03-20T09:45:12.121 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.121 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.121 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:12.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.908348+0000", 2024-03-20T09:45:12.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8678999999999999e-05 2024-03-20T09:45:12.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.122 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:12.122 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.908357+0000", 2024-03-20T09:45:12.122 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2620000000000004e-06 2024-03-20T09:45:12.122 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.122 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:12.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.919273+0000", 2024-03-20T09:45:12.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010915822 2024-03-20T09:45:12.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:12.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.919296+0000", 2024-03-20T09:45:12.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3040999999999998e-05 2024-03-20T09:45:12.123 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.123 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.123 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:12.123 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:12.016589+0000", 2024-03-20T09:45:12.123 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.097292613 2024-03-20T09:45:12.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.124 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:12.124 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:12.016613+0000", 2024-03-20T09:45:12.124 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4042000000000001e-05 2024-03-20T09:45:12.124 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.124 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:12.124 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.124 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.124 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.124 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.9.0:49922 8.1 e1007/974 8:889d7ebf:::benchmark_data_smithi122_197775_object44478:head v 1007'8840, mlcod=1007'8836)", 2024-03-20T09:45:12.124 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:37:11.908378+0000", 2024-03-20T09:45:12.124 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 480.21516773500002, 2024-03-20T09:45:12.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.11271600499999999, 2024-03-20T09:45:12.125 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:12.125 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:12.125 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:12.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:12.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.908378+0000", 2024-03-20T09:45:12.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.125 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:12.125 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.908378+0000", 2024-03-20T09:45:12.125 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.125 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.125 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:12.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.908379+0000", 2024-03-20T09:45:12.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1699999999999999e-07 2024-03-20T09:45:12.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:12.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.908392+0000", 2024-03-20T09:45:12.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3049e-05 2024-03-20T09:45:12.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.126 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.126 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:12.126 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.908399+0000", 2024-03-20T09:45:12.126 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4569999999999999e-06 2024-03-20T09:45:12.126 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:12.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.908405+0000", 2024-03-20T09:45:12.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3569999999999997e-06 2024-03-20T09:45:12.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:12.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.923493+0000", 2024-03-20T09:45:12.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015088552999999999 2024-03-20T09:45:12.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.127 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.127 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:12.127 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.923513+0000", 2024-03-20T09:45:12.127 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9621999999999999e-05 2024-03-20T09:45:12.127 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:12.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:12.021040+0000", 2024-03-20T09:45:12.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.097527480999999999 2024-03-20T09:45:12.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.128 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:12.128 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:12.021094+0000", 2024-03-20T09:45:12.128 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4069e-05 2024-03-20T09:45:12.128 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.128 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:12.128 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.128 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.128 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.128 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.9.0:49923 8.1 e1007/974 8:889e2baa:::benchmark_data_smithi122_197775_object65050:head v 1007'8841, mlcod=1007'8836)", 2024-03-20T09:45:12.129 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:37:11.910666+0000", 2024-03-20T09:45:12.129 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 480.212880439, 2024-03-20T09:45:12.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.110456586, 2024-03-20T09:45:12.129 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:12.129 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:12.129 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:12.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:12.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.910666+0000", 2024-03-20T09:45:12.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.129 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.129 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.129 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:12.129 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.910666+0000", 2024-03-20T09:45:12.129 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:12.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.910667+0000", 2024-03-20T09:45:12.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5600000000000004e-07 2024-03-20T09:45:12.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.130 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:12.130 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.910674+0000", 2024-03-20T09:45:12.130 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8350000000000004e-06 2024-03-20T09:45:12.130 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.130 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:12.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.910684+0000", 2024-03-20T09:45:12.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.6749999999999997e-06 2024-03-20T09:45:12.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:12.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.910692+0000", 2024-03-20T09:45:12.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7400000000000004e-06 2024-03-20T09:45:12.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.131 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.131 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:12.131 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.927680+0000", 2024-03-20T09:45:12.131 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016987957000000001 2024-03-20T09:45:12.131 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.132 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:12.132 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.927699+0000", 2024-03-20T09:45:12.132 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9567000000000001e-05 2024-03-20T09:45:12.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.132 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:12.132 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:12.021098+0000", 2024-03-20T09:45:12.132 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.093398315999999995 2024-03-20T09:45:12.132 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.132 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.132 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:12.132 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:12.021122+0000", 2024-03-20T09:45:12.133 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4539999999999999e-05 2024-03-20T09:45:12.133 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.133 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:12.133 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.133 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.133 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.9.0:49924 8.1 e1007/974 8:889e9671:::benchmark_data_smithi122_197775_object18223:head v 1007'8842, mlcod=1007'8836)", 2024-03-20T09:45:12.133 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:37:11.916138+0000", 2024-03-20T09:45:12.133 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 480.20740839899997, 2024-03-20T09:45:12.133 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.105007585, 2024-03-20T09:45:12.133 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:12.133 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:12.133 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:12.133 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.133 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:12.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.916138+0000", 2024-03-20T09:45:12.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.134 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:12.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.916138+0000", 2024-03-20T09:45:12.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.134 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.134 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.134 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:12.134 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.916139+0000", 2024-03-20T09:45:12.134 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4300000000000001e-06 2024-03-20T09:45:12.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:12.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.916151+0000", 2024-03-20T09:45:12.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1469e-05 2024-03-20T09:45:12.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.135 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:12.135 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.916172+0000", 2024-03-20T09:45:12.135 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1308999999999999e-05 2024-03-20T09:45:12.135 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.135 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:12.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.916185+0000", 2024-03-20T09:45:12.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2656999999999999e-05 2024-03-20T09:45:12.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:12.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.932286+0000", 2024-03-20T09:45:12.136 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016101481000000001 2024-03-20T09:45:12.136 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.136 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.136 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:12.136 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:11.932307+0000", 2024-03-20T09:45:12.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1002999999999999e-05 2024-03-20T09:45:12.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.137 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:12.137 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:12.021124+0000", 2024-03-20T09:45:12.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.088816955000000003 2024-03-20T09:45:12.137 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.137 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.137 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:12.137 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:12.021145+0000", 2024-03-20T09:45:12.137 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1280999999999998e-05 2024-03-20T09:45:12.137 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.138 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:12.138 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.138 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.138 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.138 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26917.0:142971 8.b e1020/974 8:d85fb872:::benchmark_data_smithi122_197775_object19667:head v 1020'9002, mlcod=1020'9000)", 2024-03-20T09:45:12.138 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:37:52.204355+0000", 2024-03-20T09:45:12.138 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 439.919191587, 2024-03-20T09:45:12.138 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.107852276, 2024-03-20T09:45:12.138 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:12.138 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:12.139 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:12.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:12.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:52.204355+0000", 2024-03-20T09:45:12.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.139 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.139 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:12.139 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:52.204355+0000", 2024-03-20T09:45:12.139 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.139 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:12.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:52.204355+0000", 2024-03-20T09:45:12.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8299999999999996e-07 2024-03-20T09:45:12.140 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.140 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.140 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:12.140 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:52.204362+0000", 2024-03-20T09:45:12.140 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4389999999999997e-06 2024-03-20T09:45:12.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:12.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:52.204373+0000", 2024-03-20T09:45:12.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1654999999999999e-05 2024-03-20T09:45:12.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:12.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:52.204381+0000", 2024-03-20T09:45:12.141 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8730000000000007e-06 2024-03-20T09:45:12.141 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.141 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.141 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:12.141 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:52.204448+0000", 2024-03-20T09:45:12.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6307000000000001e-05 2024-03-20T09:45:12.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.142 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:12.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:52.204470+0000", 2024-03-20T09:45:12.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2149000000000001e-05 2024-03-20T09:45:12.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.142 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.142 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:12.142 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:52.312172+0000", 2024-03-20T09:45:12.142 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.107702674 2024-03-20T09:45:12.142 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.143 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:12.143 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:52.312207+0000", 2024-03-20T09:45:12.143 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4396000000000001e-05 2024-03-20T09:45:12.143 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.143 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:12.143 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.143 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.143 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.143 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26917.0:182432 8.3 e1030/974 8:c2ab0e0e:::benchmark_data_smithi122_197775_object59128:head v 1030'11505, mlcod=1030'11502)", 2024-03-20T09:45:12.143 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:38:38.940258+0000", 2024-03-20T09:45:12.144 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 393.18328769599998, 2024-03-20T09:45:12.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.12006412599999999, 2024-03-20T09:45:12.144 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:12.144 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:12.144 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:12.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.144 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:12.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.940258+0000", 2024-03-20T09:45:12.144 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.144 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.144 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.144 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:12.144 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.940258+0000", 2024-03-20T09:45:12.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.145 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:12.145 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.940259+0000", 2024-03-20T09:45:12.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.15e-07 2024-03-20T09:45:12.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.145 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.145 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:12.145 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.940263+0000", 2024-03-20T09:45:12.145 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2259999999999999e-06 2024-03-20T09:45:12.145 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:12.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.940269+0000", 2024-03-20T09:45:12.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8130000000000001e-06 2024-03-20T09:45:12.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.146 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.146 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:12.146 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.940274+0000", 2024-03-20T09:45:12.146 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1610000000000002e-06 2024-03-20T09:45:12.146 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:12.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.943922+0000", 2024-03-20T09:45:12.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0036481159999999999 2024-03-20T09:45:12.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:12.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.943943+0000", 2024-03-20T09:45:12.147 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1120000000000001e-05 2024-03-20T09:45:12.147 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.147 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.147 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:12.147 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:39.060295+0000", 2024-03-20T09:45:12.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.116351756 2024-03-20T09:45:12.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.148 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:12.148 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:39.060323+0000", 2024-03-20T09:45:12.148 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7619000000000001e-05 2024-03-20T09:45:12.148 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.148 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:12.148 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.148 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.148 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.148 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26917.0:182440 8.3 e1030/974 8:c9a37a50:::benchmark_data_smithi122_197775_object59136:head v 1030'11506, mlcod=1030'11503)", 2024-03-20T09:45:12.149 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:38:38.941044+0000", 2024-03-20T09:45:12.149 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 393.182502462, 2024-03-20T09:45:12.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.119320419, 2024-03-20T09:45:12.149 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:12.149 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:12.149 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:12.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.149 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:12.149 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.941044+0000", 2024-03-20T09:45:12.149 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.149 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.149 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:12.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.941044+0000", 2024-03-20T09:45:12.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:12.150 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.941044+0000", 2024-03-20T09:45:12.150 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8500000000000002e-07 2024-03-20T09:45:12.150 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.150 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.150 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:12.151 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.941048+0000", 2024-03-20T09:45:12.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.279e-06 2024-03-20T09:45:12.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.151 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:12.151 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.941055+0000", 2024-03-20T09:45:12.151 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0530000000000001e-06 2024-03-20T09:45:12.151 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.151 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.151 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:12.151 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.941060+0000", 2024-03-20T09:45:12.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0039999999999999e-06 2024-03-20T09:45:12.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:12.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.948506+0000", 2024-03-20T09:45:12.152 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074458959999999996 2024-03-20T09:45:12.152 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.152 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.152 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:12.152 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.948525+0000", 2024-03-20T09:45:12.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8739000000000001e-05 2024-03-20T09:45:12.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:12.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:39.060326+0000", 2024-03-20T09:45:12.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.111800881 2024-03-20T09:45:12.153 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.153 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.153 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:12.153 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:39.060364+0000", 2024-03-20T09:45:12.153 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8182000000000001e-05 2024-03-20T09:45:12.154 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.154 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:12.154 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.154 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.154 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.154 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26917.0:182445 8.3 e1030/974 8:c44e21cb:::benchmark_data_smithi122_197775_object59141:head v 1030'11507, mlcod=1030'11503)", 2024-03-20T09:45:12.154 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:38:38.951394+0000", 2024-03-20T09:45:12.154 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 393.17215250599997, 2024-03-20T09:45:12.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.10899990499999999, 2024-03-20T09:45:12.155 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:12.155 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:12.155 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:12.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.155 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:12.155 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.951394+0000", 2024-03-20T09:45:12.155 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.155 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.155 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.156 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:12.156 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.951394+0000", 2024-03-20T09:45:12.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.156 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:12.156 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.951394+0000", 2024-03-20T09:45:12.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8899999999999998e-07 2024-03-20T09:45:12.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.156 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.156 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:12.156 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.951401+0000", 2024-03-20T09:45:12.156 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9999999999999999e-06 2024-03-20T09:45:12.156 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.157 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:12.157 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.951412+0000", 2024-03-20T09:45:12.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0498e-05 2024-03-20T09:45:12.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.157 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:12.157 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.951419+0000", 2024-03-20T09:45:12.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1430000000000001e-06 2024-03-20T09:45:12.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.157 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.157 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:12.157 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.958710+0000", 2024-03-20T09:45:12.157 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.007290927 2024-03-20T09:45:12.157 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:12.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.958732+0000", 2024-03-20T09:45:12.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2487999999999999e-05 2024-03-20T09:45:12.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:12.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:39.060367+0000", 2024-03-20T09:45:12.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.10163462600000001 2024-03-20T09:45:12.158 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.158 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.158 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:12.158 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:39.060393+0000", 2024-03-20T09:45:12.158 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6434000000000001e-05 2024-03-20T09:45:12.159 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.159 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:12.159 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.159 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.159 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.27190.0:65717 9.0 e1080/1066 9:0049fc1c:::benchmark_data_smithi122_210164_object65716:head v 1080'4239, mlcod=1080'4235)", 2024-03-20T09:45:12.159 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:41:07.889791+0000", 2024-03-20T09:45:12.159 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 244.23375505800001, 2024-03-20T09:45:12.159 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.092319735999999999, 2024-03-20T09:45:12.159 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:12.159 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:12.159 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:12.159 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.159 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:12.159 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.889791+0000", 2024-03-20T09:45:12.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:12.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.889791+0000", 2024-03-20T09:45:12.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:12.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.889792+0000", 2024-03-20T09:45:12.160 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8100000000000002e-07 2024-03-20T09:45:12.160 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.160 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.160 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:12.160 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.889855+0000", 2024-03-20T09:45:12.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2867999999999995e-05 2024-03-20T09:45:12.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:12.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.889865+0000", 2024-03-20T09:45:12.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9380000000000004e-06 2024-03-20T09:45:12.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:12.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.889871+0000", 2024-03-20T09:45:12.161 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8009999999999998e-06 2024-03-20T09:45:12.161 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.161 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.161 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:12.161 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.889922+0000", 2024-03-20T09:45:12.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0414000000000002e-05 2024-03-20T09:45:12.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.162 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:12.162 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.889939+0000", 2024-03-20T09:45:12.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6966999999999999e-05 2024-03-20T09:45:12.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.162 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:12.162 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.982062+0000", 2024-03-20T09:45:12.162 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.092123249000000004 2024-03-20T09:45:12.162 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.162 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.163 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:12.163 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.982111+0000", 2024-03-20T09:45:12.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8817999999999999e-05 2024-03-20T09:45:12.163 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.163 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:12.163 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.163 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.163 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.163 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.27190.0:65723 9.4 e1080/1066 9:2e718743:::benchmark_data_smithi122_210164_object65722:head v 1080'4078, mlcod=1080'4074)", 2024-03-20T09:45:12.163 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:41:07.890566+0000", 2024-03-20T09:45:12.163 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 244.232980223, 2024-03-20T09:45:12.163 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.093069874999999996, 2024-03-20T09:45:12.163 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:12.163 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:12.163 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:12.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:12.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.890566+0000", 2024-03-20T09:45:12.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:12.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.890566+0000", 2024-03-20T09:45:12.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.164 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.164 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.164 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:12.164 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.890567+0000", 2024-03-20T09:45:12.164 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1699999999999997e-07 2024-03-20T09:45:12.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:12.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.890626+0000", 2024-03-20T09:45:12.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8927e-05 2024-03-20T09:45:12.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:12.165 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.890634+0000", 2024-03-20T09:45:12.165 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0090000000000007e-06 2024-03-20T09:45:12.165 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.165 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.165 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:12.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.890638+0000", 2024-03-20T09:45:12.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6829999999999999e-06 2024-03-20T09:45:12.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:12.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.890681+0000", 2024-03-20T09:45:12.166 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2808000000000001e-05 2024-03-20T09:45:12.166 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.166 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.166 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:12.166 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.890699+0000", 2024-03-20T09:45:12.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7689999999999998e-05 2024-03-20T09:45:12.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:12.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.983535+0000", 2024-03-20T09:45:12.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.092836034999999997 2024-03-20T09:45:12.167 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.167 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.167 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:12.167 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:07.983636+0000", 2024-03-20T09:45:12.167 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101006 2024-03-20T09:45:12.167 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.168 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:12.168 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.168 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.168 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.27190.0:177096 9.a e1113/1066 9:5f52a355:::benchmark_data_smithi122_210164_object44866:head v 1113'11020, mlcod=1113'11018)", 2024-03-20T09:45:12.168 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:42:59.226227+0000", 2024-03-20T09:45:12.168 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 132.89731918699999, 2024-03-20T09:45:12.168 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.040449983000000002, 2024-03-20T09:45:12.168 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:12.168 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:12.168 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:12.168 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:12.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:59.226227+0000", 2024-03-20T09:45:12.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:12.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:59.226227+0000", 2024-03-20T09:45:12.169 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.169 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.169 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.169 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:12.169 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:59.226229+0000", 2024-03-20T09:45:12.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8080000000000001e-06 2024-03-20T09:45:12.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:12.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:59.226240+0000", 2024-03-20T09:45:12.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.168e-05 2024-03-20T09:45:12.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.170 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.170 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:12.170 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:59.226256+0000", 2024-03-20T09:45:12.170 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5777000000000002e-05 2024-03-20T09:45:12.170 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:12.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:59.226267+0000", 2024-03-20T09:45:12.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0566e-05 2024-03-20T09:45:12.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:12.171 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:59.227105+0000", 2024-03-20T09:45:12.171 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000838359 2024-03-20T09:45:12.171 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.171 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.171 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:12.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:59.227122+0000", 2024-03-20T09:45:12.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6806000000000001e-05 2024-03-20T09:45:12.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.172 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:12.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:59.266556+0000", 2024-03-20T09:45:12.172 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039433731 2024-03-20T09:45:12.172 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.172 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.172 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:12.172 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:59.266677+0000", 2024-03-20T09:45:12.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012125600000000001 2024-03-20T09:45:12.173 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.173 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:12.173 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.173 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.173 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.173 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.27190.0:177108 9.2 e1113/1066 9:49abb2c8:::benchmark_data_smithi122_210164_object44878:head v 1113'11069, mlcod=1113'11066)", 2024-03-20T09:45:12.173 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:42:59.241144+0000", 2024-03-20T09:45:12.173 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 132.88240187599999, 2024-03-20T09:45:12.173 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.041216295999999999, 2024-03-20T09:45:12.173 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:12.173 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:12.174 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:12.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:12.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:59.241144+0000", 2024-03-20T09:45:12.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.174 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.174 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:12.174 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:59.241144+0000", 2024-03-20T09:45:12.174 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.174 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:12.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:59.241145+0000", 2024-03-20T09:45:12.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7100000000000001e-07 2024-03-20T09:45:12.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:12.175 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:59.241153+0000", 2024-03-20T09:45:12.175 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7030000000000006e-06 2024-03-20T09:45:12.175 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.175 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.175 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:12.176 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:59.241168+0000", 2024-03-20T09:45:12.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5553000000000002e-05 2024-03-20T09:45:12.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.176 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:12.176 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:59.241176+0000", 2024-03-20T09:45:12.176 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8040000000000002e-06 2024-03-20T09:45:12.176 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.176 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.176 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:12.176 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:59.242892+0000", 2024-03-20T09:45:12.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001716373 2024-03-20T09:45:12.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:12.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:59.242912+0000", 2024-03-20T09:45:12.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9661999999999999e-05 2024-03-20T09:45:12.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.177 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.177 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:12.177 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:59.282318+0000", 2024-03-20T09:45:12.177 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039406238000000003 2024-03-20T09:45:12.177 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.178 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:12.178 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:42:59.282361+0000", 2024-03-20T09:45:12.178 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2191999999999999e-05 2024-03-20T09:45:12.178 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.178 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:12.178 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.178 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.178 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.178 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.27190.0:178442 9.2 e1114/1066 9:4273eaa8:::benchmark_data_smithi122_210164_object46212:head v 1114'11145, mlcod=1114'11142)", 2024-03-20T09:45:12.178 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:43:00.688323+0000", 2024-03-20T09:45:12.179 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 131.435222987, 2024-03-20T09:45:12.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.047207638000000003, 2024-03-20T09:45:12.179 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:12.179 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:12.179 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:12.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.179 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:12.179 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:00.688323+0000", 2024-03-20T09:45:12.179 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.179 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.179 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.179 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:12.179 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:00.688323+0000", 2024-03-20T09:45:12.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:12.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:00.688324+0000", 2024-03-20T09:45:12.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1030000000000001e-06 2024-03-20T09:45:12.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.180 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:12.180 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:00.688334+0000", 2024-03-20T09:45:12.180 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5570000000000001e-06 2024-03-20T09:45:12.180 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.180 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:12.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:00.688346+0000", 2024-03-20T09:45:12.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2432000000000001e-05 2024-03-20T09:45:12.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:12.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:00.688355+0000", 2024-03-20T09:45:12.181 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.4200000000000007e-06 2024-03-20T09:45:12.181 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.181 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.181 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:12.181 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:00.692804+0000", 2024-03-20T09:45:12.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0044491829999999998 2024-03-20T09:45:12.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:12.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:00.692831+0000", 2024-03-20T09:45:12.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.7486000000000001e-05 2024-03-20T09:45:12.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.182 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.182 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:12.182 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:00.735476+0000", 2024-03-20T09:45:12.182 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.042644960000000003 2024-03-20T09:45:12.182 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.183 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:12.183 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:00.735531+0000", 2024-03-20T09:45:12.183 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4496999999999997e-05 2024-03-20T09:45:12.183 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.183 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:12.183 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.183 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.183 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.183 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.27190.0:213534 9.6 e1128/1066 9:64d99b84:::benchmark_data_smithi122_210164_object81304:head v 1128'16252, mlcod=1128'16249)", 2024-03-20T09:45:12.183 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:43:42.484065+0000", 2024-03-20T09:45:12.183 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 89.639481188999994, 2024-03-20T09:45:12.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.041747594999999998, 2024-03-20T09:45:12.184 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:12.184 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:12.184 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:12.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.184 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:12.184 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:42.484065+0000", 2024-03-20T09:45:12.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.184 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.184 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:12.184 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:42.484065+0000", 2024-03-20T09:45:12.184 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.184 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:12.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:42.484066+0000", 2024-03-20T09:45:12.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5499999999999998e-07 2024-03-20T09:45:12.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:12.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:42.484073+0000", 2024-03-20T09:45:12.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9000000000000006e-06 2024-03-20T09:45:12.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.185 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.185 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:12.185 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:42.484084+0000", 2024-03-20T09:45:12.185 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0063e-05 2024-03-20T09:45:12.185 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:12.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:42.484090+0000", 2024-03-20T09:45:12.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8000000000000001e-06 2024-03-20T09:45:12.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:12.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:42.486845+0000", 2024-03-20T09:45:12.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0027544729999999999 2024-03-20T09:45:12.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.186 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.186 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:12.186 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:42.486867+0000", 2024-03-20T09:45:12.186 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1875999999999999e-05 2024-03-20T09:45:12.186 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:12.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:42.525764+0000", 2024-03-20T09:45:12.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.038896873999999998 2024-03-20T09:45:12.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.187 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:12.187 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:42.525812+0000", 2024-03-20T09:45:12.187 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8953999999999999e-05 2024-03-20T09:45:12.187 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.187 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:12.187 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.187 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.187 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.187 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.27190.0:213533 9.e e1128/1066 9:79f0c7ec:::benchmark_data_smithi122_210164_object81303:head v 1128'13438, mlcod=1128'13436)", 2024-03-20T09:45:12.188 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:43:42.484132+0000", 2024-03-20T09:45:12.188 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 89.639413709999999, 2024-03-20T09:45:12.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.041724051999999998, 2024-03-20T09:45:12.188 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:12.188 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:12.188 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:12.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:12.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:42.484132+0000", 2024-03-20T09:45:12.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.188 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.188 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.188 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:12.188 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:42.484132+0000", 2024-03-20T09:45:12.188 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:12.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:42.484133+0000", 2024-03-20T09:45:12.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8800000000000001e-07 2024-03-20T09:45:12.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:12.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:42.484141+0000", 2024-03-20T09:45:12.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7419999999999997e-06 2024-03-20T09:45:12.189 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.189 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.189 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:12.189 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:42.484153+0000", 2024-03-20T09:45:12.189 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1979e-05 2024-03-20T09:45:12.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:12.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:42.484161+0000", 2024-03-20T09:45:12.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8469999999999996e-06 2024-03-20T09:45:12.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:12.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:42.491691+0000", 2024-03-20T09:45:12.190 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075304200000000003 2024-03-20T09:45:12.190 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.190 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.190 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:12.190 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:42.491713+0000", 2024-03-20T09:45:12.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1756999999999999e-05 2024-03-20T09:45:12.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:12.191 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:42.525817+0000", 2024-03-20T09:45:12.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.034103623999999999 2024-03-20T09:45:12.191 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.191 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.191 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:12.191 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:42.525856+0000", 2024-03-20T09:45:12.191 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9795000000000001e-05 2024-03-20T09:45:12.191 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.191 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:12.191 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.192 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.192 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.192 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.27190.0:216819 9.d e1131/1066 9:bb175bef:::benchmark_data_smithi122_210164_object84589:head v 1131'13516, mlcod=1131'13513)", 2024-03-20T09:45:12.192 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:43:46.313085+0000", 2024-03-20T09:45:12.192 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 85.810461562, 2024-03-20T09:45:12.192 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.056608579999999999, 2024-03-20T09:45:12.192 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:12.192 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:12.192 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:12.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:12.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:46.313085+0000", 2024-03-20T09:45:12.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:12.193 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:46.313085+0000", 2024-03-20T09:45:12.193 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.193 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.193 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.193 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:12.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:46.313085+0000", 2024-03-20T09:45:12.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6800000000000001e-07 2024-03-20T09:45:12.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:12.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:46.313090+0000", 2024-03-20T09:45:12.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8629999999999999e-06 2024-03-20T09:45:12.194 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.194 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.194 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:12.194 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:46.313098+0000", 2024-03-20T09:45:12.194 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8229999999999995e-06 2024-03-20T09:45:12.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.195 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:12.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:46.313103+0000", 2024-03-20T09:45:12.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5060000000000003e-06 2024-03-20T09:45:12.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.195 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:12.195 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:46.319844+0000", 2024-03-20T09:45:12.195 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0067405679999999997 2024-03-20T09:45:12.195 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.195 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.195 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:12.196 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:46.319867+0000", 2024-03-20T09:45:12.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3095000000000002e-05 2024-03-20T09:45:12.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.196 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:12.196 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:46.369669+0000", 2024-03-20T09:45:12.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.04980217 2024-03-20T09:45:12.196 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.196 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.196 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:12.196 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:46.369693+0000", 2024-03-20T09:45:12.196 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4187000000000001e-05 2024-03-20T09:45:12.197 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.197 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:12.197 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.197 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.197 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.197 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.27190.0:235420 9.6 e1136/1066 9:65317bcc:::benchmark_data_smithi122_210164_object103190:head v 1136'17674, mlcod=1136'17671)", 2024-03-20T09:45:12.197 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:44:07.482983+0000", 2024-03-20T09:45:12.197 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 64.640562802999995, 2024-03-20T09:45:12.197 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039359309000000002, 2024-03-20T09:45:12.197 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:12.197 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:12.197 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:12.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:12.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:07.482983+0000", 2024-03-20T09:45:12.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.198 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:12.198 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:07.482983+0000", 2024-03-20T09:45:12.198 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.198 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.198 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:12.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:07.482984+0000", 2024-03-20T09:45:12.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5099999999999999e-07 2024-03-20T09:45:12.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:12.199 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:07.482989+0000", 2024-03-20T09:45:12.199 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0660000000000003e-06 2024-03-20T09:45:12.199 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.199 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.199 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:12.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:07.483000+0000", 2024-03-20T09:45:12.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0584999999999999e-05 2024-03-20T09:45:12.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:12.200 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:07.483007+0000", 2024-03-20T09:45:12.200 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0269999999999998e-06 2024-03-20T09:45:12.200 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.200 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.200 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:12.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:07.489061+0000", 2024-03-20T09:45:12.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0060542310000000002 2024-03-20T09:45:12.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:12.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:07.489087+0000", 2024-03-20T09:45:12.201 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6296999999999999e-05 2024-03-20T09:45:12.201 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.201 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.201 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:12.201 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:07.522290+0000", 2024-03-20T09:45:12.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.033202603999999997 2024-03-20T09:45:12.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.202 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:12.202 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:07.522343+0000", 2024-03-20T09:45:12.202 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2848e-05 2024-03-20T09:45:12.202 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.202 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:12.202 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.202 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.202 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.202 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.27190.0:235428 9.6 e1136/1066 9:653dcfcf:::benchmark_data_smithi122_210164_object103198:head v 1136'17675, mlcod=1136'17671)", 2024-03-20T09:45:12.202 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:44:07.489712+0000", 2024-03-20T09:45:12.202 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 64.633834538000002, 2024-03-20T09:45:12.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.042855115999999999, 2024-03-20T09:45:12.203 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:12.203 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:12.203 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:12.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:12.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:07.489712+0000", 2024-03-20T09:45:12.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.203 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.203 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:12.203 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:07.489712+0000", 2024-03-20T09:45:12.203 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.203 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:12.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:07.489713+0000", 2024-03-20T09:45:12.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.164e-06 2024-03-20T09:45:12.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:12.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:07.489720+0000", 2024-03-20T09:45:12.204 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.973e-06 2024-03-20T09:45:12.204 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.204 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.204 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:12.204 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:07.489728+0000", 2024-03-20T09:45:12.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6940000000000001e-06 2024-03-20T09:45:12.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:12.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:07.489735+0000", 2024-03-20T09:45:12.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1550000000000004e-06 2024-03-20T09:45:12.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.205 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.205 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:12.205 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:07.494648+0000", 2024-03-20T09:45:12.205 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049132300000000002 2024-03-20T09:45:12.205 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:12.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:07.494673+0000", 2024-03-20T09:45:12.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4743999999999999e-05 2024-03-20T09:45:12.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.206 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:12.206 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:07.532517+0000", 2024-03-20T09:45:12.206 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.037844093000000002 2024-03-20T09:45:12.206 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.206 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.207 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:12.207 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:07.532567+0000", 2024-03-20T09:45:12.207 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.0062999999999998e-05 2024-03-20T09:45:12.207 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.207 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:12.207 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.207 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.207 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.207 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.27190.0:235429 9.e e1136/1066 9:71a9c797:::benchmark_data_smithi122_210164_object103199:head v 1136'14817, mlcod=1136'14813)", 2024-03-20T09:45:12.207 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:44:07.489776+0000", 2024-03-20T09:45:12.208 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 64.633770326000004, 2024-03-20T09:45:12.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.042861098, 2024-03-20T09:45:12.208 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:12.208 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:12.208 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:12.208 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.208 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:12.208 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:07.489776+0000", 2024-03-20T09:45:12.208 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.208 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:12.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:07.489776+0000", 2024-03-20T09:45:12.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:12.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:12.209 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:07.489776+0000", 2024-03-20T09:45:12.209 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9800000000000003e-07 2024-03-20T09:45:12.209 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.209 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.209 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:12.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:07.489783+0000", 2024-03-20T09:45:12.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.3740000000000003e-06 2024-03-20T09:45:12.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:12.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:07.489792+0000", 2024-03-20T09:45:12.210 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4949999999999997e-06 2024-03-20T09:45:12.210 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.210 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.210 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:12.210 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:07.489813+0000", 2024-03-20T09:45:12.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0916000000000001e-05 2024-03-20T09:45:12.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:12.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:07.499685+0000", 2024-03-20T09:45:12.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0098714809999999997 2024-03-20T09:45:12.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.211 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.211 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:12.211 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:07.499707+0000", 2024-03-20T09:45:12.211 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2172e-05 2024-03-20T09:45:12.211 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:12.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:07.532598+0000", 2024-03-20T09:45:12.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.032891561 2024-03-20T09:45:12.212 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:12.212 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:12.212 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:12.212 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:07.532637+0000", 2024-03-20T09:45:12.212 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8501000000000003e-05 2024-03-20T09:45:12.212 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.212 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:12.212 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.212 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:12.212 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:12.213 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:45:12.213 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:45:13.538 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:13 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:45:13.538 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:13 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:45:13.538 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:13 smithi026 ceph-mon[68309]: osdmap e1157: 12 total, 12 up, 12 in 2024-03-20T09:45:13.539 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:13 smithi026 ceph-mon[68309]: pgmap v2815: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:13.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:13 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:45:13.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:13 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:45:13.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:13 smithi115 ceph-mon[72476]: osdmap e1157: 12 total, 12 up, 12 in 2024-03-20T09:45:13.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:13 smithi115 ceph-mon[72476]: pgmap v2815: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:13.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:13 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:45:13.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:13 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:45:13.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:13 smithi122 ceph-mon[73495]: osdmap e1157: 12 total, 12 up, 12 in 2024-03-20T09:45:13.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:13 smithi122 ceph-mon[73495]: pgmap v2815: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:13.887 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-20T09:45:14.451 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:45:14.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:14 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:45:14.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:14 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:45:14.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:14 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:45:14.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:14 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:45:14.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:14 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:45:14.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:14 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:45:15.235 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.0 to 1 (10000) 2024-03-20T09:45:15.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:15 smithi115 ceph-mon[72476]: osdmap e1158: 12 total, 12 up, 12 in 2024-03-20T09:45:15.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:15 smithi115 ceph-mon[72476]: 1.1a scrub starts 2024-03-20T09:45:15.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:15 smithi115 ceph-mon[72476]: 1.1a scrub ok 2024-03-20T09:45:15.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:15 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1169048540' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-20T09:45:15.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:15 smithi115 ceph-mon[72476]: pgmap v2817: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:15.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:15 smithi122 ceph-mon[73495]: osdmap e1158: 12 total, 12 up, 12 in 2024-03-20T09:45:15.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:15 smithi122 ceph-mon[73495]: 1.1a scrub starts 2024-03-20T09:45:15.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:15 smithi122 ceph-mon[73495]: 1.1a scrub ok 2024-03-20T09:45:15.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:15 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1169048540' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-20T09:45:15.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:15 smithi122 ceph-mon[73495]: pgmap v2817: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:15.699 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:15 smithi026 ceph-mon[68309]: osdmap e1158: 12 total, 12 up, 12 in 2024-03-20T09:45:15.699 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:15 smithi026 ceph-mon[68309]: 1.1a scrub starts 2024-03-20T09:45:15.700 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:15 smithi026 ceph-mon[68309]: 1.1a scrub ok 2024-03-20T09:45:15.700 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:15 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1169048540' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-20T09:45:15.700 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:15 smithi026 ceph-mon[68309]: pgmap v2817: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:16.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:16 smithi026 ceph-mon[68309]: 1.b scrub starts 2024-03-20T09:45:16.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:16 smithi026 ceph-mon[68309]: 1.b scrub ok 2024-03-20T09:45:16.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:16 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1169048540' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-20T09:45:16.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:16 smithi026 ceph-mon[68309]: osdmap e1159: 12 total, 12 up, 12 in 2024-03-20T09:45:16.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:16 smithi026 ceph-mon[68309]: 1.19 scrub starts 2024-03-20T09:45:16.458 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:16 smithi026 ceph-mon[68309]: 1.19 scrub ok 2024-03-20T09:45:16.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:16 smithi115 ceph-mon[72476]: 1.b scrub starts 2024-03-20T09:45:16.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:16 smithi115 ceph-mon[72476]: 1.b scrub ok 2024-03-20T09:45:16.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:16 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1169048540' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-20T09:45:16.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:16 smithi115 ceph-mon[72476]: osdmap e1159: 12 total, 12 up, 12 in 2024-03-20T09:45:16.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:16 smithi115 ceph-mon[72476]: 1.19 scrub starts 2024-03-20T09:45:16.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:16 smithi115 ceph-mon[72476]: 1.19 scrub ok 2024-03-20T09:45:16.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:16 smithi122 ceph-mon[73495]: 1.b scrub starts 2024-03-20T09:45:16.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:16 smithi122 ceph-mon[73495]: 1.b scrub ok 2024-03-20T09:45:16.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:16 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1169048540' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-20T09:45:16.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:16 smithi122 ceph-mon[73495]: osdmap e1159: 12 total, 12 up, 12 in 2024-03-20T09:45:16.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:16 smithi122 ceph-mon[73495]: 1.19 scrub starts 2024-03-20T09:45:16.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:16 smithi122 ceph-mon[73495]: 1.19 scrub ok 2024-03-20T09:45:16.887 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd primary-affinity 0 1 2024-03-20T09:45:17.037 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:45:17.037 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:45:17.037 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-03-20T09:45:17.038 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:45:17.615 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:17 smithi026 ceph-mon[68309]: pgmap v2819: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:17.642 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:17 smithi115 ceph-mon[72476]: pgmap v2819: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:17.645 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-20T09:45:17.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:17 smithi122 ceph-mon[73495]: pgmap v2819: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:17.702 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:45:19.281 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:45:19.519 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:19 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:45:19.519 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:19 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:45:19.519 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:19 smithi026 ceph-mon[68309]: osdmap e1160: 12 total, 12 up, 12 in 2024-03-20T09:45:19.519 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:19 smithi026 ceph-mon[68309]: 1.14 scrub starts 2024-03-20T09:45:19.519 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:19 smithi026 ceph-mon[68309]: 1.14 scrub ok 2024-03-20T09:45:19.520 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:19 smithi026 ceph-mon[68309]: pgmap v2821: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:19.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:19 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:45:19.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:19 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:45:19.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:19 smithi115 ceph-mon[72476]: osdmap e1160: 12 total, 12 up, 12 in 2024-03-20T09:45:19.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:19 smithi115 ceph-mon[72476]: 1.14 scrub starts 2024-03-20T09:45:19.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:19 smithi115 ceph-mon[72476]: 1.14 scrub ok 2024-03-20T09:45:19.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:19 smithi115 ceph-mon[72476]: pgmap v2821: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:19.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:19 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:45:19.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:19 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:45:19.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:19 smithi122 ceph-mon[73495]: osdmap e1160: 12 total, 12 up, 12 in 2024-03-20T09:45:19.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:19 smithi122 ceph-mon[73495]: 1.14 scrub starts 2024-03-20T09:45:19.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:19 smithi122 ceph-mon[73495]: 1.14 scrub ok 2024-03-20T09:45:19.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:19 smithi122 ceph-mon[73495]: pgmap v2821: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:20.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:20 smithi115 ceph-mon[72476]: 1.2 scrub starts 2024-03-20T09:45:20.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:20 smithi115 ceph-mon[72476]: 1.2 scrub ok 2024-03-20T09:45:20.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:20 smithi115 ceph-mon[72476]: 1.10 scrub starts 2024-03-20T09:45:20.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:20 smithi115 ceph-mon[72476]: 1.10 scrub ok 2024-03-20T09:45:20.666 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:20 smithi026 ceph-mon[68309]: 1.2 scrub starts 2024-03-20T09:45:20.666 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:20 smithi026 ceph-mon[68309]: 1.2 scrub ok 2024-03-20T09:45:20.667 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:20 smithi026 ceph-mon[68309]: 1.10 scrub starts 2024-03-20T09:45:20.667 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:20 smithi026 ceph-mon[68309]: 1.10 scrub ok 2024-03-20T09:45:20.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:20 smithi122 ceph-mon[73495]: 1.2 scrub starts 2024-03-20T09:45:20.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:20 smithi122 ceph-mon[73495]: 1.2 scrub ok 2024-03-20T09:45:20.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:20 smithi122 ceph-mon[73495]: 1.10 scrub starts 2024-03-20T09:45:20.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:20 smithi122 ceph-mon[73495]: 1.10 scrub ok 2024-03-20T09:45:21.187 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:45:21.187 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:45:21.187 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-03-20T09:45:21.187 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-03-20T09:45:21.187 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:45:21.259 INFO:teuthology.orchestra.run.smithi026.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-20T09:45:21.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:21 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:45:21.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:21 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:45:21.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:21 smithi115 ceph-mon[72476]: osdmap e1161: 12 total, 12 up, 12 in 2024-03-20T09:45:21.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:21 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4201308099' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-20T09:45:21.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:21 smithi115 ceph-mon[72476]: pgmap v2823: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:21.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:21 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:45:21.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:21 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:45:21.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:21 smithi122 ceph-mon[73495]: osdmap e1161: 12 total, 12 up, 12 in 2024-03-20T09:45:21.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:21 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4201308099' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-20T09:45:21.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:21 smithi122 ceph-mon[73495]: pgmap v2823: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:21.707 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:21 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:45:21.707 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:21 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:45:21.707 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:21 smithi026 ceph-mon[68309]: osdmap e1161: 12 total, 12 up, 12 in 2024-03-20T09:45:21.707 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:21 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4201308099' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-20T09:45:21.707 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:21 smithi026 ceph-mon[68309]: pgmap v2823: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:22.191 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 1 1 2024-03-20T09:45:22.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:22 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4201308099' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-20T09:45:22.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:22 smithi026 ceph-mon[68309]: osdmap e1162: 12 total, 12 up, 12 in 2024-03-20T09:45:22.642 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:22 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4201308099' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-20T09:45:22.643 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:22 smithi115 ceph-mon[72476]: osdmap e1162: 12 total, 12 up, 12 in 2024-03-20T09:45:22.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:22 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4201308099' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-20T09:45:22.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:22 smithi122 ceph-mon[73495]: osdmap e1162: 12 total, 12 up, 12 in 2024-03-20T09:45:22.706 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-20T09:45:22.807 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:45:23.289 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:23 smithi026 ceph-mon[68309]: pgmap v2825: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:23.642 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:23 smithi115 ceph-mon[72476]: pgmap v2825: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:23.683 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:23 smithi122 ceph-mon[73495]: pgmap v2825: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:24.038 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:45:25.768 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.1 to 1 (10000) 2024-03-20T09:45:26.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:25 smithi115 ceph-mon[72476]: pgmap v2826: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:26.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:25 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3029169015' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-20T09:45:26.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:25 smithi122 ceph-mon[73495]: pgmap v2826: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:26.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:25 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3029169015' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-20T09:45:26.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:25 smithi026 ceph-mon[68309]: pgmap v2826: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:26.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:25 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3029169015' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-20T09:45:26.257 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:45:26.257 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-03-20T09:45:26.257 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-03-20T09:45:26.257 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:45:26.257 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.258 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.9.0:50770 8.1 e1008/974 8:8a201a7d:::benchmark_data_smithi122_197775_object42703:head v 1008'9491, mlcod=1008'9475)", 2024-03-20T09:45:26.258 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:37:15.423167+0000", 2024-03-20T09:45:26.258 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 490.84529596599998, 2024-03-20T09:45:26.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.087135948000000005, 2024-03-20T09:45:26.258 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:26.258 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:26.258 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:26.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.258 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:26.258 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:15.423167+0000", 2024-03-20T09:45:26.258 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.258 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.258 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:26.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:15.423167+0000", 2024-03-20T09:45:26.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:26.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:15.423168+0000", 2024-03-20T09:45:26.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1500000000000004e-07 2024-03-20T09:45:26.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.259 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.259 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:26.259 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:15.423173+0000", 2024-03-20T09:45:26.259 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.524e-06 2024-03-20T09:45:26.259 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:26.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:15.423186+0000", 2024-03-20T09:45:26.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3155e-05 2024-03-20T09:45:26.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:26.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:15.423193+0000", 2024-03-20T09:45:26.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5300000000000002e-06 2024-03-20T09:45:26.260 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.260 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.260 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:26.260 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:15.451887+0000", 2024-03-20T09:45:26.260 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.028694148999999999 2024-03-20T09:45:26.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:26.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:15.451907+0000", 2024-03-20T09:45:26.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9701e-05 2024-03-20T09:45:26.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:26.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:15.510250+0000", 2024-03-20T09:45:26.261 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.058343632999999999 2024-03-20T09:45:26.261 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.261 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.261 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:26.261 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:15.510303+0000", 2024-03-20T09:45:26.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.2641e-05 2024-03-20T09:45:26.262 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.262 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:26.262 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.262 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.262 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.262 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(osd.9.0:50771 8.1 e1008/974 8:8a2141a7:::benchmark_data_smithi122_197775_object38153:head v 1008'9492, mlcod=1008'9475)", 2024-03-20T09:45:26.262 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:37:15.426369+0000", 2024-03-20T09:45:26.262 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 490.84209401599998, 2024-03-20T09:45:26.262 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.083970157000000004, 2024-03-20T09:45:26.262 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:26.262 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:26.262 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:26.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:26.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:15.426369+0000", 2024-03-20T09:45:26.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:26.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:15.426369+0000", 2024-03-20T09:45:26.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.263 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.263 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:26.263 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:15.426370+0000", 2024-03-20T09:45:26.263 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4499999999999995e-07 2024-03-20T09:45:26.263 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:26.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:15.426378+0000", 2024-03-20T09:45:26.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5979999999999999e-06 2024-03-20T09:45:26.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:26.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:15.426394+0000", 2024-03-20T09:45:26.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6072000000000001e-05 2024-03-20T09:45:26.264 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.264 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.264 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:26.264 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:15.426402+0000", 2024-03-20T09:45:26.264 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8449999999999992e-06 2024-03-20T09:45:26.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:26.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:15.457726+0000", 2024-03-20T09:45:26.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031323748999999998 2024-03-20T09:45:26.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:26.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:15.457747+0000", 2024-03-20T09:45:26.265 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1044999999999999e-05 2024-03-20T09:45:26.265 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.265 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.265 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:26.265 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:15.510306+0000", 2024-03-20T09:45:26.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.052558485000000002 2024-03-20T09:45:26.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.266 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:26.266 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:37:15.510339+0000", 2024-03-20T09:45:26.266 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3417999999999998e-05 2024-03-20T09:45:26.266 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.266 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:26.266 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.266 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.266 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.266 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26917.0:181741 8.1 e1030/974 8:89a6a754:::benchmark_data_smithi122_197775_object58437:head v 1030'14213, mlcod=1030'14211)", 2024-03-20T09:45:26.266 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:38:38.082087+0000", 2024-03-20T09:45:26.266 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 408.18637608099999, 2024-03-20T09:45:26.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.078371448999999996, 2024-03-20T09:45:26.267 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:26.267 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:26.267 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:26.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:26.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.082087+0000", 2024-03-20T09:45:26.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.267 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.267 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:26.267 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.082087+0000", 2024-03-20T09:45:26.267 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.267 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:26.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.082088+0000", 2024-03-20T09:45:26.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9500000000000001e-07 2024-03-20T09:45:26.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:26.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.082094+0000", 2024-03-20T09:45:26.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.28e-06 2024-03-20T09:45:26.268 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.268 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.268 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:26.268 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.082105+0000", 2024-03-20T09:45:26.268 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0668e-05 2024-03-20T09:45:26.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:26.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.082112+0000", 2024-03-20T09:45:26.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1080000000000004e-06 2024-03-20T09:45:26.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:26.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.082183+0000", 2024-03-20T09:45:26.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1111000000000004e-05 2024-03-20T09:45:26.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.269 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.269 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:26.269 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.082203+0000", 2024-03-20T09:45:26.269 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0519000000000001e-05 2024-03-20T09:45:26.269 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:26.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.160425+0000", 2024-03-20T09:45:26.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.078221999 2024-03-20T09:45:26.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.270 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:26.270 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:38.160458+0000", 2024-03-20T09:45:26.270 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.2969000000000002e-05 2024-03-20T09:45:26.270 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.270 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:26.270 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.270 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.270 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.271 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26917.0:183103 8.d e1031/974 8:b41dbca3:::benchmark_data_smithi122_197775_object59799:head v 1031'11464, mlcod=1030'11462)", 2024-03-20T09:45:26.271 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:38:39.781754+0000", 2024-03-20T09:45:26.271 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 406.48670944700001, 2024-03-20T09:45:26.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.037873153, 2024-03-20T09:45:26.271 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:26.271 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:26.271 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:26.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:26.271 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:39.781754+0000", 2024-03-20T09:45:26.271 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.271 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.271 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.271 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:26.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:39.781754+0000", 2024-03-20T09:45:26.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:26.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:39.781755+0000", 2024-03-20T09:45:26.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1999999999999999e-06 2024-03-20T09:45:26.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.272 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.272 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:26.272 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:39.781763+0000", 2024-03-20T09:45:26.272 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7050000000000005e-06 2024-03-20T09:45:26.272 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:26.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:39.781781+0000", 2024-03-20T09:45:26.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7416999999999999e-05 2024-03-20T09:45:26.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:26.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:39.781791+0000", 2024-03-20T09:45:26.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.9820000000000003e-06 2024-03-20T09:45:26.273 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.273 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.273 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:26.273 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:39.792897+0000", 2024-03-20T09:45:26.273 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011105703 2024-03-20T09:45:26.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:26.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:39.792916+0000", 2024-03-20T09:45:26.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9873000000000001e-05 2024-03-20T09:45:26.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:26.274 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:39.819401+0000", 2024-03-20T09:45:26.274 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.026484405999999999 2024-03-20T09:45:26.274 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.274 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.274 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:26.275 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:39.819627+0000", 2024-03-20T09:45:26.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00022586700000000001 2024-03-20T09:45:26.275 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.275 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:26.275 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.275 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.275 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26917.0:184477 8.f e1031/974 8:f6f73982:::benchmark_data_smithi122_197775_object61173:head v 1031'11420, mlcod=1031'11417)", 2024-03-20T09:45:26.275 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:38:41.469263+0000", 2024-03-20T09:45:26.275 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 404.79920018799999, 2024-03-20T09:45:26.275 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.033609901999999997, 2024-03-20T09:45:26.275 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:26.275 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:26.275 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:26.275 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:26.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:41.469263+0000", 2024-03-20T09:45:26.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:26.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:41.469263+0000", 2024-03-20T09:45:26.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.276 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:26.276 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:41.469263+0000", 2024-03-20T09:45:26.276 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6899999999999997e-07 2024-03-20T09:45:26.276 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.276 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:26.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:41.469270+0000", 2024-03-20T09:45:26.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1940000000000003e-06 2024-03-20T09:45:26.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:26.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:41.469277+0000", 2024-03-20T09:45:26.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.5410000000000003e-06 2024-03-20T09:45:26.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.277 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:26.277 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:41.469283+0000", 2024-03-20T09:45:26.277 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8969999999999996e-06 2024-03-20T09:45:26.277 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.277 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:26.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:41.469344+0000", 2024-03-20T09:45:26.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1292999999999997e-05 2024-03-20T09:45:26.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:26.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:41.469366+0000", 2024-03-20T09:45:26.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1988e-05 2024-03-20T09:45:26.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.278 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.278 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:26.278 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:41.502830+0000", 2024-03-20T09:45:26.278 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.033463402000000003 2024-03-20T09:45:26.278 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.279 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:26.279 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:41.502873+0000", 2024-03-20T09:45:26.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3018e-05 2024-03-20T09:45:26.279 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.279 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:26.279 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.279 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.279 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.279 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26917.0:188281 8.1 e1032/974 8:80e6a698:::benchmark_data_smithi122_197775_object64977:head v 1032'14619, mlcod=1032'14617)", 2024-03-20T09:45:26.279 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:38:46.258216+0000", 2024-03-20T09:45:26.279 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 400.01024712600002, 2024-03-20T09:45:26.279 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.034349853999999999, 2024-03-20T09:45:26.279 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:26.279 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:26.280 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:26.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:26.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:46.258216+0000", 2024-03-20T09:45:26.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:26.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:46.258216+0000", 2024-03-20T09:45:26.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.280 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.280 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.280 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:26.280 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:46.258217+0000", 2024-03-20T09:45:26.280 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.17e-06 2024-03-20T09:45:26.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:26.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:46.258227+0000", 2024-03-20T09:45:26.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0225e-05 2024-03-20T09:45:26.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:26.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:46.258249+0000", 2024-03-20T09:45:26.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1574999999999999e-05 2024-03-20T09:45:26.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.281 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.281 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:26.281 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:46.258258+0000", 2024-03-20T09:45:26.281 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.1849999999999993e-06 2024-03-20T09:45:26.281 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:26.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:46.258317+0000", 2024-03-20T09:45:26.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8495999999999997e-05 2024-03-20T09:45:26.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:26.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:46.258338+0000", 2024-03-20T09:45:26.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1129999999999999e-05 2024-03-20T09:45:26.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.282 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.282 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:26.282 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:46.292528+0000", 2024-03-20T09:45:26.282 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.034190335000000002 2024-03-20T09:45:26.282 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.283 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:26.283 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:46.292566+0000", 2024-03-20T09:45:26.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7737999999999997e-05 2024-03-20T09:45:26.283 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.283 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:26.283 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.283 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.283 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.283 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26917.0:191776 8.d e1032/974 8:bbbaf0a7:::benchmark_data_smithi122_197775_object68472:head v 1032'12053, mlcod=1032'12048)", 2024-03-20T09:45:26.283 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:38:50.701247+0000", 2024-03-20T09:45:26.283 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 395.56721600100002, 2024-03-20T09:45:26.283 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.030983947000000001, 2024-03-20T09:45:26.283 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:26.283 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:26.283 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:26.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:26.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:50.701247+0000", 2024-03-20T09:45:26.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:26.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:50.701247+0000", 2024-03-20T09:45:26.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.284 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.284 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:26.284 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:50.701248+0000", 2024-03-20T09:45:26.284 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1800000000000005e-07 2024-03-20T09:45:26.284 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:26.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:50.701254+0000", 2024-03-20T09:45:26.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6209999999999998e-06 2024-03-20T09:45:26.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:26.285 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:50.701268+0000", 2024-03-20T09:45:26.285 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3138e-05 2024-03-20T09:45:26.285 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.285 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.285 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:26.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:50.701275+0000", 2024-03-20T09:45:26.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4039999999999998e-06 2024-03-20T09:45:26.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:26.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:50.724646+0000", 2024-03-20T09:45:26.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.023370557 2024-03-20T09:45:26.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:26.286 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:50.724671+0000", 2024-03-20T09:45:26.286 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5490000000000002e-05 2024-03-20T09:45:26.286 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.286 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.286 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:26.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:50.732193+0000", 2024-03-20T09:45:26.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0075218899999999998 2024-03-20T09:45:26.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.287 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:26.287 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:38:50.732231+0000", 2024-03-20T09:45:26.287 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8028999999999998e-05 2024-03-20T09:45:26.287 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.287 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:26.287 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.287 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.287 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.287 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26917.0:204858 8.a e1036/974 8:5bc79f72:::benchmark_data_smithi122_197775_object81554:head v 1036'12850, mlcod=1036'12848)", 2024-03-20T09:45:26.287 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:39:06.701104+0000", 2024-03-20T09:45:26.287 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 379.56735933499999, 2024-03-20T09:45:26.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.032227503999999997, 2024-03-20T09:45:26.288 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:26.288 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:26.288 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:26.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:26.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:06.701104+0000", 2024-03-20T09:45:26.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.288 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:26.288 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:06.701104+0000", 2024-03-20T09:45:26.288 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.288 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.288 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:26.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:06.701104+0000", 2024-03-20T09:45:26.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9699999999999995e-07 2024-03-20T09:45:26.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:26.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:06.701111+0000", 2024-03-20T09:45:26.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1770000000000003e-06 2024-03-20T09:45:26.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.289 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:26.289 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:06.701121+0000", 2024-03-20T09:45:26.289 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0214000000000001e-05 2024-03-20T09:45:26.289 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.289 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:26.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:06.701127+0000", 2024-03-20T09:45:26.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5610000000000004e-06 2024-03-20T09:45:26.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:26.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:06.701184+0000", 2024-03-20T09:45:26.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6348000000000003e-05 2024-03-20T09:45:26.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.290 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.290 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:26.290 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:06.701206+0000", 2024-03-20T09:45:26.290 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2680999999999998e-05 2024-03-20T09:45:26.290 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:26.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:06.733283+0000", 2024-03-20T09:45:26.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.032076130000000001 2024-03-20T09:45:26.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.291 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:26.291 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:06.733331+0000", 2024-03-20T09:45:26.291 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8596e-05 2024-03-20T09:45:26.291 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.291 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:26.291 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.291 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.291 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.291 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26917.0:207679 8.f e1037/974 8:fe5431a2:::benchmark_data_smithi122_197775_object84375:head v 1037'12820, mlcod=1037'12817)", 2024-03-20T09:45:26.291 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:39:10.150210+0000", 2024-03-20T09:45:26.292 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 376.118253229, 2024-03-20T09:45:26.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.030963818000000001, 2024-03-20T09:45:26.292 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:26.292 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:26.292 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:26.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:26.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:10.150210+0000", 2024-03-20T09:45:26.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.292 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:26.292 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:10.150210+0000", 2024-03-20T09:45:26.292 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.292 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.292 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:26.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:10.150211+0000", 2024-03-20T09:45:26.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7000000000000004e-07 2024-03-20T09:45:26.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:26.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:10.150216+0000", 2024-03-20T09:45:26.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8050000000000003e-06 2024-03-20T09:45:26.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.293 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:26.293 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:10.150226+0000", 2024-03-20T09:45:26.293 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5449999999999992e-06 2024-03-20T09:45:26.293 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.293 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:26.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:10.150231+0000", 2024-03-20T09:45:26.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3550000000000004e-06 2024-03-20T09:45:26.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:26.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:10.155326+0000", 2024-03-20T09:45:26.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0050946560000000004 2024-03-20T09:45:26.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.294 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.294 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:26.294 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:10.155347+0000", 2024-03-20T09:45:26.294 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1498e-05 2024-03-20T09:45:26.294 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:26.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:10.181135+0000", 2024-03-20T09:45:26.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.025787138000000001 2024-03-20T09:45:26.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.295 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.295 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:26.295 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:10.181174+0000", 2024-03-20T09:45:26.295 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9051000000000003e-05 2024-03-20T09:45:26.295 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.295 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:26.295 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.295 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.296 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26917.0:212944 8.e 8:7027353b:::benchmark_data_smithi122_197775_object89640:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1042)", 2024-03-20T09:45:26.296 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:39:19.361264+0000", 2024-03-20T09:45:26.296 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 366.90719931199999, 2024-03-20T09:45:26.296 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.033146479, 2024-03-20T09:45:26.296 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:26.296 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:26.296 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:45:26.296 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26917", 2024-03-20T09:45:26.296 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:45:26.296 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 212944 2024-03-20T09:45:26.296 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.296 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:26.296 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.296 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:26.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:19.361264+0000", 2024-03-20T09:45:26.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:26.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:19.361264+0000", 2024-03-20T09:45:26.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.297 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:26.297 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:19.361264+0000", 2024-03-20T09:45:26.297 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.1800000000000001e-07 2024-03-20T09:45:26.297 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.297 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:26.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:19.361266+0000", 2024-03-20T09:45:26.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.2579999999999998e-06 2024-03-20T09:45:26.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:26.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:19.361286+0000", 2024-03-20T09:45:26.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0023000000000001e-05 2024-03-20T09:45:26.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.298 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.298 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:26.298 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:19.361292+0000", 2024-03-20T09:45:26.298 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9290000000000003e-06 2024-03-20T09:45:26.298 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:26.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:19.372269+0000", 2024-03-20T09:45:26.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010976859 2024-03-20T09:45:26.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:26.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:19.372415+0000", 2024-03-20T09:45:26.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014633699999999999 2024-03-20T09:45:26.299 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.299 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.299 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,4,7", 2024-03-20T09:45:26.299 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:19.372557+0000", 2024-03-20T09:45:26.299 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014167699999999999 2024-03-20T09:45:26.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:45:26.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:19.378284+0000", 2024-03-20T09:45:26.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0057272260000000002 2024-03-20T09:45:26.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:26.300 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:19.385687+0000", 2024-03-20T09:45:26.300 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0074026320000000001 2024-03-20T09:45:26.300 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.300 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.300 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:26.301 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:19.386996+0000", 2024-03-20T09:45:26.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001308795 2024-03-20T09:45:26.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.301 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:26.301 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:19.394271+0000", 2024-03-20T09:45:26.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0072753940000000001 2024-03-20T09:45:26.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.301 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:26.301 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:19.394313+0000", 2024-03-20T09:45:26.301 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.2153000000000001e-05 2024-03-20T09:45:26.301 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.301 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.301 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:26.302 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:19.394410+0000", 2024-03-20T09:45:26.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.6778000000000001e-05 2024-03-20T09:45:26.302 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.302 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:26.302 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.302 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.302 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.302 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26917.0:215212 8.e 8:77d37c56:::benchmark_data_smithi122_197775_object91908:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1043)", 2024-03-20T09:45:26.302 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:39:22.571209+0000", 2024-03-20T09:45:26.302 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 363.69725436200002, 2024-03-20T09:45:26.302 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.034535443999999998, 2024-03-20T09:45:26.302 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:26.302 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:26.302 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:45:26.303 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26917", 2024-03-20T09:45:26.303 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:45:26.303 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 215212 2024-03-20T09:45:26.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.303 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:26.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:26.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:22.571209+0000", 2024-03-20T09:45:26.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.303 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.303 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.303 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:26.303 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:22.571209+0000", 2024-03-20T09:45:26.303 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:26.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:22.571210+0000", 2024-03-20T09:45:26.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.618e-06 2024-03-20T09:45:26.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:26.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:22.571218+0000", 2024-03-20T09:45:26.304 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3030000000000002e-06 2024-03-20T09:45:26.304 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.304 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.304 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:26.304 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:22.571232+0000", 2024-03-20T09:45:26.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4773999999999999e-05 2024-03-20T09:45:26.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:26.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:22.571241+0000", 2024-03-20T09:45:26.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8130000000000002e-06 2024-03-20T09:45:26.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:26.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:22.579153+0000", 2024-03-20T09:45:26.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0079118399999999998 2024-03-20T09:45:26.305 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.305 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.305 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:26.305 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:22.579293+0000", 2024-03-20T09:45:26.305 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000140297 2024-03-20T09:45:26.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,4,7", 2024-03-20T09:45:26.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:22.579434+0000", 2024-03-20T09:45:26.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000140414 2024-03-20T09:45:26.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:26.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:22.583929+0000", 2024-03-20T09:45:26.306 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0044956420000000002 2024-03-20T09:45:26.306 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.306 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.306 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:26.306 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:22.594258+0000", 2024-03-20T09:45:26.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010328502 2024-03-20T09:45:26.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:26.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:22.594305+0000", 2024-03-20T09:45:26.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.7581999999999999e-05 2024-03-20T09:45:26.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:45:26.307 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:22.605559+0000", 2024-03-20T09:45:26.307 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011253249 2024-03-20T09:45:26.307 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.307 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.307 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:26.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:22.605617+0000", 2024-03-20T09:45:26.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8705999999999997e-05 2024-03-20T09:45:26.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.308 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:26.308 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:22.605744+0000", 2024-03-20T09:45:26.308 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000126704 2024-03-20T09:45:26.308 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.308 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:26.308 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.308 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.308 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.308 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26917.0:215230 8.e 8:746b97cf:::benchmark_data_smithi122_197775_object91926:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1043)", 2024-03-20T09:45:26.308 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:39:22.587982+0000", 2024-03-20T09:45:26.309 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 363.68048133100001, 2024-03-20T09:45:26.309 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.040619191999999998, 2024-03-20T09:45:26.309 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:26.309 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:26.309 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:45:26.309 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26917", 2024-03-20T09:45:26.309 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:45:26.309 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 215230 2024-03-20T09:45:26.309 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.309 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:26.309 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:26.310 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:22.587982+0000", 2024-03-20T09:45:26.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:26.310 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:22.587982+0000", 2024-03-20T09:45:26.310 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.310 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.310 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.310 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:26.310 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:22.587982+0000", 2024-03-20T09:45:26.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.1399999999999996e-07 2024-03-20T09:45:26.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:26.311 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:22.587988+0000", 2024-03-20T09:45:26.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.6049999999999997e-06 2024-03-20T09:45:26.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:26.311 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:22.588004+0000", 2024-03-20T09:45:26.311 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6096e-05 2024-03-20T09:45:26.311 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.311 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.311 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:26.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:22.588012+0000", 2024-03-20T09:45:26.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.303e-06 2024-03-20T09:45:26.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:26.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:22.600272+0000", 2024-03-20T09:45:26.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012259383 2024-03-20T09:45:26.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.312 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:26.312 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:22.600418+0000", 2024-03-20T09:45:26.312 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000145723 2024-03-20T09:45:26.312 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.312 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,4,7", 2024-03-20T09:45:26.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:22.600561+0000", 2024-03-20T09:45:26.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014296999999999999 2024-03-20T09:45:26.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:45:26.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:22.611488+0000", 2024-03-20T09:45:26.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010927549999999999 2024-03-20T09:45:26.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:26.313 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:22.612709+0000", 2024-03-20T09:45:26.313 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0012207850000000001 2024-03-20T09:45:26.313 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.313 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.313 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:26.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:22.614249+0000", 2024-03-20T09:45:26.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001540317 2024-03-20T09:45:26.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:26.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:22.628397+0000", 2024-03-20T09:45:26.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014148101 2024-03-20T09:45:26.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.314 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:26.314 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:22.628467+0000", 2024-03-20T09:45:26.314 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9575999999999999e-05 2024-03-20T09:45:26.314 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.314 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.315 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:26.315 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:22.628601+0000", 2024-03-20T09:45:26.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000133969 2024-03-20T09:45:26.315 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.315 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:26.315 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.315 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.315 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.315 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26917.0:215541 8.e 8:763809b7:::benchmark_data_smithi122_197775_object92237:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1043)", 2024-03-20T09:45:26.315 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:39:23.031626+0000", 2024-03-20T09:45:26.315 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 363.23683719899998, 2024-03-20T09:45:26.315 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.034101621999999998, 2024-03-20T09:45:26.315 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:26.315 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:26.315 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:45:26.315 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26917", 2024-03-20T09:45:26.316 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:45:26.316 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 215541 2024-03-20T09:45:26.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.316 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:26.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.316 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:26.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.031626+0000", 2024-03-20T09:45:26.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.316 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:26.316 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.031626+0000", 2024-03-20T09:45:26.316 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.316 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.316 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:26.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.031628+0000", 2024-03-20T09:45:26.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.182e-06 2024-03-20T09:45:26.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:26.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.031636+0000", 2024-03-20T09:45:26.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3159999999999997e-06 2024-03-20T09:45:26.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.317 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:26.317 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.031650+0000", 2024-03-20T09:45:26.317 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3682e-05 2024-03-20T09:45:26.317 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.317 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:26.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.031659+0000", 2024-03-20T09:45:26.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.4299999999999995e-06 2024-03-20T09:45:26.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:26.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.032647+0000", 2024-03-20T09:45:26.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0009873760000000001 2024-03-20T09:45:26.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.318 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.318 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:26.318 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.032747+0000", 2024-03-20T09:45:26.318 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000100636 2024-03-20T09:45:26.318 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,4,7", 2024-03-20T09:45:26.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.032855+0000", 2024-03-20T09:45:26.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000107884 2024-03-20T09:45:26.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:26.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.047200+0000", 2024-03-20T09:45:26.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014345060999999999 2024-03-20T09:45:26.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.319 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.319 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:45:26.319 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.054842+0000", 2024-03-20T09:45:26.319 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0076417739999999996 2024-03-20T09:45:26.319 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:26.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.065480+0000", 2024-03-20T09:45:26.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010638056999999999 2024-03-20T09:45:26.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:26.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.065526+0000", 2024-03-20T09:45:26.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.562e-05 2024-03-20T09:45:26.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:26.320 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.065596+0000", 2024-03-20T09:45:26.320 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0613999999999998e-05 2024-03-20T09:45:26.320 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.320 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.320 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:26.321 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.065727+0000", 2024-03-20T09:45:26.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013098999999999999 2024-03-20T09:45:26.321 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.321 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:26.321 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.321 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.321 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.321 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26917.0:215542 8.e 8:76372f19:::benchmark_data_smithi122_197775_object92238:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1043)", 2024-03-20T09:45:26.321 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:39:23.032326+0000", 2024-03-20T09:45:26.321 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 363.236137052, 2024-03-20T09:45:26.321 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.038992449999999998, 2024-03-20T09:45:26.321 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:26.321 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:26.321 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:45:26.322 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26917", 2024-03-20T09:45:26.322 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:45:26.322 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 215542 2024-03-20T09:45:26.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.322 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:26.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:26.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.032326+0000", 2024-03-20T09:45:26.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.322 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.322 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.322 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:26.322 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.032326+0000", 2024-03-20T09:45:26.322 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:26.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.032327+0000", 2024-03-20T09:45:26.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.0299999999999998e-07 2024-03-20T09:45:26.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:26.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.032332+0000", 2024-03-20T09:45:26.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9289999999999997e-06 2024-03-20T09:45:26.323 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.323 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.323 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:26.323 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.032344+0000", 2024-03-20T09:45:26.323 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.207e-05 2024-03-20T09:45:26.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:26.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.032354+0000", 2024-03-20T09:45:26.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0400999999999999e-05 2024-03-20T09:45:26.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:26.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.038012+0000", 2024-03-20T09:45:26.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0056583029999999999 2024-03-20T09:45:26.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.324 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.324 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:26.324 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.038114+0000", 2024-03-20T09:45:26.324 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000101427 2024-03-20T09:45:26.324 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,4,7", 2024-03-20T09:45:26.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.038209+0000", 2024-03-20T09:45:26.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5487000000000002e-05 2024-03-20T09:45:26.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:45:26.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.054849+0000", 2024-03-20T09:45:26.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.016639937 2024-03-20T09:45:26.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.325 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:26.325 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.065812+0000", 2024-03-20T09:45:26.325 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.010963125000000001 2024-03-20T09:45:26.325 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.325 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:26.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.065851+0000", 2024-03-20T09:45:26.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.8491999999999997e-05 2024-03-20T09:45:26.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:26.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.071181+0000", 2024-03-20T09:45:26.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0053303379999999996 2024-03-20T09:45:26.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.326 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.326 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:26.326 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.071240+0000", 2024-03-20T09:45:26.326 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8627e-05 2024-03-20T09:45:26.326 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.327 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:26.327 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.071318+0000", 2024-03-20T09:45:26.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8510999999999994e-05 2024-03-20T09:45:26.327 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.327 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:26.327 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.327 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.327 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.327 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26917.0:215544 8.e 8:74992e1d:::benchmark_data_smithi122_197775_object92240:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1043)", 2024-03-20T09:45:26.327 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:39:23.032368+0000", 2024-03-20T09:45:26.327 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 363.23609510900002, 2024-03-20T09:45:26.327 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.039141363999999998, 2024-03-20T09:45:26.327 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:26.327 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:26.328 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:45:26.328 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26917", 2024-03-20T09:45:26.328 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:45:26.328 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 215544 2024-03-20T09:45:26.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.328 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:26.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.328 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:26.328 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.032368+0000", 2024-03-20T09:45:26.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.328 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.328 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:26.328 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.032368+0000", 2024-03-20T09:45:26.328 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.328 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:26.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.032368+0000", 2024-03-20T09:45:26.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3799999999999997e-07 2024-03-20T09:45:26.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:26.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.032372+0000", 2024-03-20T09:45:26.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.101e-06 2024-03-20T09:45:26.329 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.329 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.329 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:26.329 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.032379+0000", 2024-03-20T09:45:26.329 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.9030000000000004e-06 2024-03-20T09:45:26.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:26.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.032386+0000", 2024-03-20T09:45:26.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4200000000000004e-06 2024-03-20T09:45:26.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:26.330 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.042159+0000", 2024-03-20T09:45:26.330 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0097728929999999995 2024-03-20T09:45:26.330 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.330 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.330 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:26.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.042322+0000", 2024-03-20T09:45:26.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00016326700000000001 2024-03-20T09:45:26.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,4,7", 2024-03-20T09:45:26.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.042490+0000", 2024-03-20T09:45:26.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000167925 2024-03-20T09:45:26.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.331 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:45:26.331 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.054856+0000", 2024-03-20T09:45:26.331 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012365920000000001 2024-03-20T09:45:26.331 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.331 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:26.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.065887+0000", 2024-03-20T09:45:26.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011030964000000001 2024-03-20T09:45:26.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:26.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.065921+0000", 2024-03-20T09:45:26.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4570999999999998e-05 2024-03-20T09:45:26.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.332 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.332 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:26.332 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.071373+0000", 2024-03-20T09:45:26.332 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054519989999999999 2024-03-20T09:45:26.332 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:26.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.071433+0000", 2024-03-20T09:45:26.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9561000000000001e-05 2024-03-20T09:45:26.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.333 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:26.333 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.071509+0000", 2024-03-20T09:45:26.333 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6302000000000002e-05 2024-03-20T09:45:26.333 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.333 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:26.333 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.333 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.333 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.333 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26917.0:215550 8.e 8:7c5f42a5:::benchmark_data_smithi122_197775_object92246:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1043)", 2024-03-20T09:45:26.334 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:39:23.040163+0000", 2024-03-20T09:45:26.334 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 363.228299488, 2024-03-20T09:45:26.334 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.031621020999999999, 2024-03-20T09:45:26.334 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:26.334 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:26.334 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:45:26.334 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26917", 2024-03-20T09:45:26.334 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:45:26.334 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 215550 2024-03-20T09:45:26.334 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.334 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:26.334 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.334 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:26.334 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.040163+0000", 2024-03-20T09:45:26.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:26.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.040163+0000", 2024-03-20T09:45:26.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:26.335 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.040165+0000", 2024-03-20T09:45:26.335 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7090000000000001e-06 2024-03-20T09:45:26.335 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.335 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.335 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:26.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.040174+0000", 2024-03-20T09:45:26.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8599999999999999e-06 2024-03-20T09:45:26.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:26.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.040194+0000", 2024-03-20T09:45:26.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9979999999999998e-05 2024-03-20T09:45:26.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:26.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.040205+0000", 2024-03-20T09:45:26.336 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0684e-05 2024-03-20T09:45:26.336 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.336 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.336 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:26.336 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.049851+0000", 2024-03-20T09:45:26.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0096461120000000001 2024-03-20T09:45:26.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.337 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:26.337 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.050008+0000", 2024-03-20T09:45:26.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015675300000000001 2024-03-20T09:45:26.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.337 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,4,7", 2024-03-20T09:45:26.337 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.050157+0000", 2024-03-20T09:45:26.337 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000149665 2024-03-20T09:45:26.337 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.337 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.337 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:26.337 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.065957+0000", 2024-03-20T09:45:26.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.015800216999999998 2024-03-20T09:45:26.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:26.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.065992+0000", 2024-03-20T09:45:26.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4866999999999997e-05 2024-03-20T09:45:26.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:45:26.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.071139+0000", 2024-03-20T09:45:26.338 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0051468360000000001 2024-03-20T09:45:26.338 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.338 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.338 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:26.338 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.071569+0000", 2024-03-20T09:45:26.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00042946500000000002 2024-03-20T09:45:26.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:26.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.071640+0000", 2024-03-20T09:45:26.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1600999999999994e-05 2024-03-20T09:45:26.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.339 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.339 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:26.339 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:23.071785+0000", 2024-03-20T09:45:26.339 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000144272 2024-03-20T09:45:26.339 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.339 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:26.339 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.339 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.340 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26917.0:223001 8.e 8:7e212c76:::benchmark_data_smithi122_197775_object99697:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1051)", 2024-03-20T09:45:26.340 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:39:34.332428+0000", 2024-03-20T09:45:26.340 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 351.93603481500003, 2024-03-20T09:45:26.340 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.037138468000000001, 2024-03-20T09:45:26.340 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:26.340 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:26.340 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:45:26.340 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26917", 2024-03-20T09:45:26.340 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:45:26.340 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 223001 2024-03-20T09:45:26.340 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.340 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:26.340 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.340 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:26.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:34.332428+0000", 2024-03-20T09:45:26.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:26.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:34.332428+0000", 2024-03-20T09:45:26.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.341 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:26.341 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:34.332429+0000", 2024-03-20T09:45:26.341 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6700000000000002e-07 2024-03-20T09:45:26.341 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.341 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:26.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:34.332433+0000", 2024-03-20T09:45:26.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3429999999999998e-06 2024-03-20T09:45:26.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:26.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:34.332444+0000", 2024-03-20T09:45:26.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0998e-05 2024-03-20T09:45:26.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.342 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:26.342 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:34.332450+0000", 2024-03-20T09:45:26.342 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.8989999999999998e-06 2024-03-20T09:45:26.342 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.342 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:26.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:34.334605+0000", 2024-03-20T09:45:26.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0021548579999999999 2024-03-20T09:45:26.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:26.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:34.334751+0000", 2024-03-20T09:45:26.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00014605 2024-03-20T09:45:26.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.343 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.343 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,4,7", 2024-03-20T09:45:26.343 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:34.334897+0000", 2024-03-20T09:45:26.343 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000145971 2024-03-20T09:45:26.343 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:45:26.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:34.339519+0000", 2024-03-20T09:45:26.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0046215229999999998 2024-03-20T09:45:26.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:26.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:34.340200+0000", 2024-03-20T09:45:26.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00068159600000000002 2024-03-20T09:45:26.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:26.344 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:34.344977+0000", 2024-03-20T09:45:26.344 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0047762999999999998 2024-03-20T09:45:26.344 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.344 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.344 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:26.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:34.369419+0000", 2024-03-20T09:45:26.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.024442684999999999 2024-03-20T09:45:26.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:26.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:34.369479+0000", 2024-03-20T09:45:26.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.9472999999999998e-05 2024-03-20T09:45:26.345 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.345 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.345 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:26.345 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:34.369567+0000", 2024-03-20T09:45:26.345 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7905000000000006e-05 2024-03-20T09:45:26.345 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.345 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:26.345 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.346 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.346 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.26917.0:224262 8.e 8:7b130222:::benchmark_data_smithi122_197775_object100958:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1051)", 2024-03-20T09:45:26.346 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:39:35.968552+0000", 2024-03-20T09:45:26.346 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 350.29991117100002, 2024-03-20T09:45:26.346 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.032430819, 2024-03-20T09:45:26.346 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:26.346 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:26.346 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:45:26.346 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.26917", 2024-03-20T09:45:26.346 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:45:26.346 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 224262 2024-03-20T09:45:26.346 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.346 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:26.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:26.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:35.968552+0000", 2024-03-20T09:45:26.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.347 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:26.347 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:35.968552+0000", 2024-03-20T09:45:26.347 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.347 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.347 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.348 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:26.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:35.968554+0000", 2024-03-20T09:45:26.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8050000000000001e-06 2024-03-20T09:45:26.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.348 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:26.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:35.968560+0000", 2024-03-20T09:45:26.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8430000000000004e-06 2024-03-20T09:45:26.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.348 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:26.348 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:35.968574+0000", 2024-03-20T09:45:26.348 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3312999999999999e-05 2024-03-20T09:45:26.348 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.348 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:26.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:35.968583+0000", 2024-03-20T09:45:26.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.8130000000000002e-06 2024-03-20T09:45:26.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:26.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:35.974077+0000", 2024-03-20T09:45:26.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0054943969999999998 2024-03-20T09:45:26.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.349 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:26.349 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:35.974227+0000", 2024-03-20T09:45:26.349 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00015022000000000001 2024-03-20T09:45:26.349 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.349 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 0,4,7", 2024-03-20T09:45:26.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:35.974364+0000", 2024-03-20T09:45:26.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00013725 2024-03-20T09:45:26.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:45:26.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:35.984185+0000", 2024-03-20T09:45:26.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0098209419999999992 2024-03-20T09:45:26.350 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.350 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.350 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:26.350 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:35.987992+0000", 2024-03-20T09:45:26.350 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0038062199999999999 2024-03-20T09:45:26.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:26.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:35.988204+0000", 2024-03-20T09:45:26.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000212196 2024-03-20T09:45:26.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:26.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:36.000754+0000", 2024-03-20T09:45:26.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.012550578 2024-03-20T09:45:26.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.351 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.351 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:26.351 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:36.000845+0000", 2024-03-20T09:45:26.351 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0655000000000005e-05 2024-03-20T09:45:26.351 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.352 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:26.352 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:36.000983+0000", 2024-03-20T09:45:26.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000137587 2024-03-20T09:45:26.352 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.352 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:26.352 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.352 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.352 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.352 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26917.0:227763 8.d e1052/974 8:bdec73b5:::benchmark_data_smithi122_197775_object104459:head v 1052'14260, mlcod=1052'14258)", 2024-03-20T09:45:26.352 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:39:40.383150+0000", 2024-03-20T09:45:26.352 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 345.88531269499998, 2024-03-20T09:45:26.352 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.033703192999999999, 2024-03-20T09:45:26.352 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:26.353 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:26.353 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:26.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:26.353 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:40.383150+0000", 2024-03-20T09:45:26.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:26.353 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:40.383150+0000", 2024-03-20T09:45:26.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.353 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.353 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.353 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:26.353 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:40.383151+0000", 2024-03-20T09:45:26.353 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.3900000000000003e-07 2024-03-20T09:45:26.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.354 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:26.354 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:40.383157+0000", 2024-03-20T09:45:26.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0979999999999999e-06 2024-03-20T09:45:26.354 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.354 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.354 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:26.354 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:40.383171+0000", 2024-03-20T09:45:26.354 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.329e-05 2024-03-20T09:45:26.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:26.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:40.383178+0000", 2024-03-20T09:45:26.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.0110000000000003e-06 2024-03-20T09:45:26.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:26.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:40.388137+0000", 2024-03-20T09:45:26.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0049595689999999996 2024-03-20T09:45:26.355 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.355 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.355 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:26.355 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:40.388162+0000", 2024-03-20T09:45:26.355 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4352999999999999e-05 2024-03-20T09:45:26.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:26.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:40.416822+0000", 2024-03-20T09:45:26.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.028660077999999999 2024-03-20T09:45:26.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.356 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:26.356 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:40.416853+0000", 2024-03-20T09:45:26.356 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1854999999999999e-05 2024-03-20T09:45:26.356 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.356 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:26.356 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.356 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.356 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.356 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.26917.0:227767 8.d e1052/974 8:b3eadf44:::benchmark_data_smithi122_197775_object104463:head v 1052'14261, mlcod=1052'14258)", 2024-03-20T09:45:26.357 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:39:40.383226+0000", 2024-03-20T09:45:26.357 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 345.88523689599998, 2024-03-20T09:45:26.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.033644822999999997, 2024-03-20T09:45:26.357 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:26.357 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:26.357 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:26.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.357 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:26.357 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:40.383226+0000", 2024-03-20T09:45:26.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.357 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.357 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:26.357 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:40.383226+0000", 2024-03-20T09:45:26.357 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:26.357 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.358 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:26.358 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:40.383226+0000", 2024-03-20T09:45:26.358 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.1899999999999999e-07 2024-03-20T09:45:26.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.358 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:26.358 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:40.383234+0000", 2024-03-20T09:45:26.358 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.7270000000000007e-06 2024-03-20T09:45:26.358 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.358 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.358 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:26.358 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:40.383241+0000", 2024-03-20T09:45:26.358 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.8220000000000001e-06 2024-03-20T09:45:26.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:26.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:40.383245+0000", 2024-03-20T09:45:26.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3680000000000004e-06 2024-03-20T09:45:26.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:26.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:40.392321+0000", 2024-03-20T09:45:26.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0090759329999999996 2024-03-20T09:45:26.359 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.359 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.359 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:26.359 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:40.392341+0000", 2024-03-20T09:45:26.359 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0245999999999999e-05 2024-03-20T09:45:26.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.360 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:26.360 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:40.416856+0000", 2024-03-20T09:45:26.360 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.024514264000000001 2024-03-20T09:45:26.360 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:26.360 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:26.360 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:26.360 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:39:40.416871+0000", 2024-03-20T09:45:26.360 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5244e-05 2024-03-20T09:45:26.360 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.360 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:26.360 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.360 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:26.361 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:26.361 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:45:26.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:26 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3029169015' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-20T09:45:26.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:26 smithi026 ceph-mon[68309]: osdmap e1163: 12 total, 12 up, 12 in 2024-03-20T09:45:27.017 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-20T09:45:27.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:26 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3029169015' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-20T09:45:27.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:26 smithi115 ceph-mon[72476]: osdmap e1163: 12 total, 12 up, 12 in 2024-03-20T09:45:27.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:26 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3029169015' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-20T09:45:27.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:26 smithi122 ceph-mon[73495]: osdmap e1163: 12 total, 12 up, 12 in 2024-03-20T09:45:27.511 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd primary-affinity 1 1 2024-03-20T09:45:27.625 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:45:28.118 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:27 smithi026 ceph-mon[68309]: 1.1b scrub starts 2024-03-20T09:45:28.118 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:27 smithi026 ceph-mon[68309]: pgmap v2828: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:28.118 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:27 smithi026 ceph-mon[68309]: 1.1b scrub ok 2024-03-20T09:45:28.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:27 smithi115 ceph-mon[72476]: 1.1b scrub starts 2024-03-20T09:45:28.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:27 smithi115 ceph-mon[72476]: pgmap v2828: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:28.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:27 smithi115 ceph-mon[72476]: 1.1b scrub ok 2024-03-20T09:45:28.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:27 smithi122 ceph-mon[73495]: 1.1b scrub starts 2024-03-20T09:45:28.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:27 smithi122 ceph-mon[73495]: pgmap v2828: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:28.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:27 smithi122 ceph-mon[73495]: 1.1b scrub ok 2024-03-20T09:45:28.420 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:45:29.247 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:29 smithi026 ceph-mon[68309]: pgmap v2829: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:29.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:29 smithi115 ceph-mon[72476]: pgmap v2829: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:29.625 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:29 smithi122 ceph-mon[73495]: pgmap v2829: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:30.725 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:45:30.725 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:45:30.725 INFO:teuthology.orchestra.run.smithi026.stdout: "num_ops": 0 2024-03-20T09:45:30.725 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:45:31.432 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-20T09:45:31.725 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:45:32.040 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:31 smithi026 ceph-mon[68309]: pgmap v2830: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:32.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:31 smithi115 ceph-mon[72476]: pgmap v2830: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:32.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:31 smithi122 ceph-mon[73495]: pgmap v2830: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:32.776 INFO:teuthology.orchestra.run.smithi026.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-20T09:45:32.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:32 smithi026 ceph-mon[68309]: 1.a scrub starts 2024-03-20T09:45:32.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:32 smithi026 ceph-mon[68309]: 1.a scrub ok 2024-03-20T09:45:32.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:32 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3859842796' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-20T09:45:33.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:32 smithi115 ceph-mon[72476]: 1.a scrub starts 2024-03-20T09:45:33.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:32 smithi115 ceph-mon[72476]: 1.a scrub ok 2024-03-20T09:45:33.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:32 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3859842796' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-20T09:45:33.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:32 smithi122 ceph-mon[73495]: 1.a scrub starts 2024-03-20T09:45:33.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:32 smithi122 ceph-mon[73495]: 1.a scrub ok 2024-03-20T09:45:33.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:32 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3859842796' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-20T09:45:33.691 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 2 1 2024-03-20T09:45:33.916 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:33 smithi026 ceph-mon[68309]: pgmap v2831: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:33.917 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:33 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3859842796' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-20T09:45:33.917 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:33 smithi026 ceph-mon[68309]: osdmap e1164: 12 total, 12 up, 12 in 2024-03-20T09:45:33.917 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:33 smithi026 ceph-mon[68309]: 1.7 scrub starts 2024-03-20T09:45:33.917 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:33 smithi026 ceph-mon[68309]: 1.7 scrub ok 2024-03-20T09:45:34.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:33 smithi115 ceph-mon[72476]: pgmap v2831: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:34.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:33 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3859842796' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-20T09:45:34.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:33 smithi115 ceph-mon[72476]: osdmap e1164: 12 total, 12 up, 12 in 2024-03-20T09:45:34.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:33 smithi115 ceph-mon[72476]: 1.7 scrub starts 2024-03-20T09:45:34.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:33 smithi115 ceph-mon[72476]: 1.7 scrub ok 2024-03-20T09:45:34.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:33 smithi122 ceph-mon[73495]: pgmap v2831: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:34.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:33 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3859842796' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-20T09:45:34.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:33 smithi122 ceph-mon[73495]: osdmap e1164: 12 total, 12 up, 12 in 2024-03-20T09:45:34.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:33 smithi122 ceph-mon[73495]: 1.7 scrub starts 2024-03-20T09:45:34.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:33 smithi122 ceph-mon[73495]: 1.7 scrub ok 2024-03-20T09:45:34.188 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:45:34.188 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [], 2024-03-20T09:45:34.188 INFO:teuthology.orchestra.run.smithi026.stdout: "complaint_time": 30, 2024-03-20T09:45:34.188 INFO:teuthology.orchestra.run.smithi026.stdout: "num_blocked_ops": 0 2024-03-20T09:45:34.188 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:45:34.212 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:45:34.834 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:34 smithi026 ceph-mon[68309]: osdmap e1165: 12 total, 12 up, 12 in 2024-03-20T09:45:35.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:34 smithi115 ceph-mon[72476]: osdmap e1165: 12 total, 12 up, 12 in 2024-03-20T09:45:35.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:34 smithi122 ceph-mon[73495]: osdmap e1165: 12 total, 12 up, 12 in 2024-03-20T09:45:35.243 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-20T09:45:35.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:35 smithi026 ceph-mon[68309]: pgmap v2834: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:36.051 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:45:36.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:35 smithi115 ceph-mon[72476]: pgmap v2834: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:36.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:35 smithi122 ceph-mon[73495]: pgmap v2834: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:37.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:36 smithi115 ceph-mon[72476]: 1.9 scrub starts 2024-03-20T09:45:37.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:36 smithi115 ceph-mon[72476]: 1.9 scrub ok 2024-03-20T09:45:37.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:36 smithi122 ceph-mon[73495]: 1.9 scrub starts 2024-03-20T09:45:37.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:36 smithi122 ceph-mon[73495]: 1.9 scrub ok 2024-03-20T09:45:37.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:36 smithi026 ceph-mon[68309]: 1.9 scrub starts 2024-03-20T09:45:37.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:36 smithi026 ceph-mon[68309]: 1.9 scrub ok 2024-03-20T09:45:37.795 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.2 to 1 (10000) 2024-03-20T09:45:37.888 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:37 smithi026 ceph-mon[68309]: pgmap v2835: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:37.888 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:37 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:45:37.888 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:37 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:45:37.888 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:37 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/308732972' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:45:37.888 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:37 smithi026 ceph-mon[68309]: 1.3 deep-scrub starts 2024-03-20T09:45:38.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:37 smithi115 ceph-mon[72476]: pgmap v2835: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:38.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:37 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:45:38.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:37 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:45:38.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:37 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/308732972' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:45:38.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:37 smithi115 ceph-mon[72476]: 1.3 deep-scrub starts 2024-03-20T09:45:38.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:37 smithi122 ceph-mon[73495]: pgmap v2835: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:38.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:37 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:45:38.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:37 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:45:38.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:37 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/308732972' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:45:38.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:37 smithi122 ceph-mon[73495]: 1.3 deep-scrub starts 2024-03-20T09:45:38.559 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-20T09:45:38.559 INFO:teuthology.orchestra.run.smithi026.stdout: "size": 20, 2024-03-20T09:45:38.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 600, 2024-03-20T09:45:38.560 INFO:teuthology.orchestra.run.smithi026.stdout: "ops": [ 2024-03-20T09:45:38.560 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.560 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.27190.0:126205 9.1 9:821c8175:::benchmark_data_smithi122_210164_object126204:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1098)", 2024-03-20T09:45:38.560 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:41:59.076029+0000", 2024-03-20T09:45:38.560 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 219.33203018099999, 2024-03-20T09:45:38.560 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.17505138200000001, 2024-03-20T09:45:38.560 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:38.560 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:38.560 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:45:38.560 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.27190", 2024-03-20T09:45:38.560 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:45:38.560 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 126205 2024-03-20T09:45:38.560 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.560 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:38.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:38.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:59.076029+0000", 2024-03-20T09:45:38.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:38.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:38.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:59.076029+0000", 2024-03-20T09:45:38.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:38.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.561 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:38.561 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:59.076030+0000", 2024-03-20T09:45:38.561 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.288e-06 2024-03-20T09:45:38.561 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.561 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:38.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:59.076098+0000", 2024-03-20T09:45:38.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.7371000000000003e-05 2024-03-20T09:45:38.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:38.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:59.076112+0000", 2024-03-20T09:45:38.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4219000000000001e-05 2024-03-20T09:45:38.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.562 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.562 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:38.562 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:59.076120+0000", 2024-03-20T09:45:38.562 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3299999999999999e-06 2024-03-20T09:45:38.562 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:38.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:59.130024+0000", 2024-03-20T09:45:38.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.053903949 2024-03-20T09:45:38.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:38.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:59.130177+0000", 2024-03-20T09:45:38.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000152569 2024-03-20T09:45:38.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.563 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.563 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 6,9", 2024-03-20T09:45:38.563 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:59.130341+0000", 2024-03-20T09:45:38.563 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000164403 2024-03-20T09:45:38.563 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:45:38.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:59.134426+0000", 2024-03-20T09:45:38.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040849119999999996 2024-03-20T09:45:38.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:38.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:59.145600+0000", 2024-03-20T09:45:38.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.011173501000000001 2024-03-20T09:45:38.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.564 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:38.564 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:59.251001+0000", 2024-03-20T09:45:38.564 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.10540137099999999 2024-03-20T09:45:38.564 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.564 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:38.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:59.251037+0000", 2024-03-20T09:45:38.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5695000000000003e-05 2024-03-20T09:45:38.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.565 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:38.565 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:41:59.251081+0000", 2024-03-20T09:45:38.565 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3773999999999998e-05 2024-03-20T09:45:38.565 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:38.565 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:38.565 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:38.565 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.565 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.565 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.27190.0:179010 9.c 9:386ccbb6:::benchmark_data_smithi122_210164_object46780:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1115)", 2024-03-20T09:45:38.565 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:43:01.315221+0000", 2024-03-20T09:45:38.565 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 157.092838105, 2024-03-20T09:45:38.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.190310331, 2024-03-20T09:45:38.566 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:38.566 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:38.566 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:45:38.566 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.27190", 2024-03-20T09:45:38.566 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:45:38.566 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 179010 2024-03-20T09:45:38.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.566 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:38.566 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.566 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:38.566 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.315221+0000", 2024-03-20T09:45:38.566 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:38.566 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:38.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.315221+0000", 2024-03-20T09:45:38.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:38.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:38.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.315223+0000", 2024-03-20T09:45:38.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.587e-06 2024-03-20T09:45:38.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.567 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:38.567 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.315229+0000", 2024-03-20T09:45:38.567 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4629999999999998e-06 2024-03-20T09:45:38.567 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.567 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:38.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.315243+0000", 2024-03-20T09:45:38.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3497e-05 2024-03-20T09:45:38.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:38.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.315252+0000", 2024-03-20T09:45:38.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.5799999999999998e-06 2024-03-20T09:45:38.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:38.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.315315+0000", 2024-03-20T09:45:38.568 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2540999999999996e-05 2024-03-20T09:45:38.568 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.568 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.568 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:38.568 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.315455+0000", 2024-03-20T09:45:38.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000140307 2024-03-20T09:45:38.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4,9", 2024-03-20T09:45:38.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.315578+0000", 2024-03-20T09:45:38.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012268899999999999 2024-03-20T09:45:38.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:45:38.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.319935+0000", 2024-03-20T09:45:38.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0043576079999999998 2024-03-20T09:45:38.569 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.569 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.569 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:38.569 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.321719+0000", 2024-03-20T09:45:38.569 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.001783122 2024-03-20T09:45:38.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:38.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.505416+0000", 2024-03-20T09:45:38.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.183697677 2024-03-20T09:45:38.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:38.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.505478+0000", 2024-03-20T09:45:38.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.1669999999999997e-05 2024-03-20T09:45:38.570 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.570 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.570 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:38.570 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.505532+0000", 2024-03-20T09:45:38.570 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.359e-05 2024-03-20T09:45:38.571 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:38.571 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:38.571 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:38.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.571 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.571 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.27190.0:179012 9.2 9:44ccea61:::benchmark_data_smithi122_210164_object46782:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1115)", 2024-03-20T09:45:38.571 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:43:01.322168+0000", 2024-03-20T09:45:38.571 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 157.08589082099999, 2024-03-20T09:45:38.571 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.18990584699999999, 2024-03-20T09:45:38.571 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:38.571 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:38.571 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:45:38.571 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.27190", 2024-03-20T09:45:38.571 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:45:38.571 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 179012 2024-03-20T09:45:38.571 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.571 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:38.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:38.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.322168+0000", 2024-03-20T09:45:38.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:38.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:38.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.322168+0000", 2024-03-20T09:45:38.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:38.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.572 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.572 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:38.572 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.322170+0000", 2024-03-20T09:45:38.572 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.469e-06 2024-03-20T09:45:38.572 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:38.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.322175+0000", 2024-03-20T09:45:38.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.023e-06 2024-03-20T09:45:38.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:38.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.322187+0000", 2024-03-20T09:45:38.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1973e-05 2024-03-20T09:45:38.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.573 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:38.573 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.322194+0000", 2024-03-20T09:45:38.573 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.9190000000000002e-06 2024-03-20T09:45:38.573 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.573 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:38.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.322239+0000", 2024-03-20T09:45:38.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.4826000000000001e-05 2024-03-20T09:45:38.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:38.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.505420+0000", 2024-03-20T09:45:38.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.183181228 2024-03-20T09:45:38.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.574 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 1,9", 2024-03-20T09:45:38.574 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.505555+0000", 2024-03-20T09:45:38.574 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000135048 2024-03-20T09:45:38.574 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.574 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:38.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.510817+0000", 2024-03-20T09:45:38.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0052623640000000003 2024-03-20T09:45:38.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:45:38.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.511427+0000", 2024-03-20T09:45:38.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00060980099999999998 2024-03-20T09:45:38.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:38.575 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.511982+0000", 2024-03-20T09:45:38.575 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00055471899999999996 2024-03-20T09:45:38.575 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.575 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.575 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:38.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.512025+0000", 2024-03-20T09:45:38.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3634000000000003e-05 2024-03-20T09:45:38.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.576 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:38.576 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.512074+0000", 2024-03-20T09:45:38.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.8843e-05 2024-03-20T09:45:38.576 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:38.576 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:38.576 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:38.576 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.576 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.576 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop_reply(client.27190.0:179010 9.c e1115/1066 ondisk, result = 0)", 2024-03-20T09:45:38.576 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:43:01.323951+0000", 2024-03-20T09:45:38.576 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 157.08410780099999, 2024-03-20T09:45:38.576 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.181586092, 2024-03-20T09:45:38.577 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:38.577 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:45:38.577 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:38.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:38.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.323951+0000", 2024-03-20T09:45:38.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:38.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:38.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.323951+0000", 2024-03-20T09:45:38.577 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:38.577 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.577 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.577 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:38.577 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.323952+0000", 2024-03-20T09:45:38.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.99e-07 2024-03-20T09:45:38.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:38.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.323956+0000", 2024-03-20T09:45:38.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0960000000000003e-06 2024-03-20T09:45:38.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:38.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.505276+0000", 2024-03-20T09:45:38.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.181319429 2024-03-20T09:45:38.578 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.578 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.578 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:38.578 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.505287+0000", 2024-03-20T09:45:38.578 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1654999999999999e-05 2024-03-20T09:45:38.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.579 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:38.579 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.505398+0000", 2024-03-20T09:45:38.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000110874 2024-03-20T09:45:38.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.579 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:38.579 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.505409+0000", 2024-03-20T09:45:38.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0954999999999999e-05 2024-03-20T09:45:38.579 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.579 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.579 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:38.579 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.505538+0000", 2024-03-20T09:45:38.579 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000128484 2024-03-20T09:45:38.580 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:38.580 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:38.580 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:38.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.580 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.580 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.27190.0:179021 9.1 9:884403c3:::benchmark_data_smithi122_210164_object46791:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1115)", 2024-03-20T09:45:38.580 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:43:01.328089+0000", 2024-03-20T09:45:38.580 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 157.07997020900001, 2024-03-20T09:45:38.580 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.19237621099999999, 2024-03-20T09:45:38.580 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:38.580 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:38.580 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:45:38.580 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.27190", 2024-03-20T09:45:38.580 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:45:38.580 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 179021 2024-03-20T09:45:38.580 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.581 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:38.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:38.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.328089+0000", 2024-03-20T09:45:38.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:38.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:38.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.328089+0000", 2024-03-20T09:45:38.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:38.581 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.581 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.581 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:38.581 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.328090+0000", 2024-03-20T09:45:38.581 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.09e-07 2024-03-20T09:45:38.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:38.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.328095+0000", 2024-03-20T09:45:38.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.729e-06 2024-03-20T09:45:38.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:38.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.505263+0000", 2024-03-20T09:45:38.582 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.177168669 2024-03-20T09:45:38.582 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.582 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.582 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:38.582 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.505281+0000", 2024-03-20T09:45:38.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.8176e-05 2024-03-20T09:45:38.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:38.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.505411+0000", 2024-03-20T09:45:38.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00012999999999999999 2024-03-20T09:45:38.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:38.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.505540+0000", 2024-03-20T09:45:38.583 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000128119 2024-03-20T09:45:38.583 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.583 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.583 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 6,9", 2024-03-20T09:45:38.583 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.505654+0000", 2024-03-20T09:45:38.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00011436 2024-03-20T09:45:38.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:38.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.520375+0000", 2024-03-20T09:45:38.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.014720963 2024-03-20T09:45:38.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:45:38.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.520384+0000", 2024-03-20T09:45:38.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.2089999999999994e-06 2024-03-20T09:45:38.584 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.584 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.584 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:38.584 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.520408+0000", 2024-03-20T09:45:38.584 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.3575999999999999e-05 2024-03-20T09:45:38.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.585 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:38.585 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.520439+0000", 2024-03-20T09:45:38.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.1233999999999999e-05 2024-03-20T09:45:38.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.585 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:38.585 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.520465+0000", 2024-03-20T09:45:38.585 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.6367e-05 2024-03-20T09:45:38.585 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:38.585 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:38.585 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:38.585 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.585 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.586 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_repop(client.27190.0:179027 9.7 e1115/1066 9:ec596744:::benchmark_data_smithi122_210164_object46797:head v 1115'11337, mlcod=1115'11335)", 2024-03-20T09:45:38.586 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:43:01.338879+0000", 2024-03-20T09:45:38.586 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 157.069180273, 2024-03-20T09:45:38.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.187056316, 2024-03-20T09:45:38.586 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:38.586 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:38.586 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:38.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:38.586 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.338879+0000", 2024-03-20T09:45:38.586 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:38.586 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.586 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.586 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:38.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.338879+0000", 2024-03-20T09:45:38.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:38.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:38.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.338880+0000", 2024-03-20T09:45:38.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.11e-06 2024-03-20T09:45:38.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:38.587 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.338888+0000", 2024-03-20T09:45:38.587 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.4019999999999996e-06 2024-03-20T09:45:38.587 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.587 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.587 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:38.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.505269+0000", 2024-03-20T09:45:38.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.16638134600000001 2024-03-20T09:45:38.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:38.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.505281+0000", 2024-03-20T09:45:38.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2371000000000001e-05 2024-03-20T09:45:38.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:38.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.505346+0000", 2024-03-20T09:45:38.588 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.4981999999999995e-05 2024-03-20T09:45:38.588 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.588 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.588 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:38.588 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.505376+0000", 2024-03-20T09:45:38.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.9399000000000001e-05 2024-03-20T09:45:38.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:38.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.525888+0000", 2024-03-20T09:45:38.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.020512760000000001 2024-03-20T09:45:38.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.589 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:38.589 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:43:01.525935+0000", 2024-03-20T09:45:38.589 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6946000000000001e-05 2024-03-20T09:45:38.589 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:38.589 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:38.589 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:38.589 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.589 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.590 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.27190.0:260843 9.1 9:8e89670e:::benchmark_data_smithi122_210164_object128613:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1142)", 2024-03-20T09:45:38.590 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:44:36.301030+0000", 2024-03-20T09:45:38.590 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.107029058000002, 2024-03-20T09:45:38.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.178654546, 2024-03-20T09:45:38.590 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:38.590 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:38.590 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:45:38.590 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.27190", 2024-03-20T09:45:38.590 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:45:38.590 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 260843 2024-03-20T09:45:38.590 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.590 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:38.590 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.590 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:38.590 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.301030+0000", 2024-03-20T09:45:38.590 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:38.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:38.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.301030+0000", 2024-03-20T09:45:38.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:38.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:38.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.301031+0000", 2024-03-20T09:45:38.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.023e-06 2024-03-20T09:45:38.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.591 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.591 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:38.591 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.301036+0000", 2024-03-20T09:45:38.591 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.3619999999999999e-06 2024-03-20T09:45:38.591 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:38.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.301046+0000", 2024-03-20T09:45:38.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.004e-05 2024-03-20T09:45:38.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:38.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.301052+0000", 2024-03-20T09:45:38.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5100000000000004e-06 2024-03-20T09:45:38.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.592 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.592 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:38.592 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.301114+0000", 2024-03-20T09:45:38.592 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.2087000000000004e-05 2024-03-20T09:45:38.592 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:38.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.301198+0000", 2024-03-20T09:45:38.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3570000000000001e-05 2024-03-20T09:45:38.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 6,9", 2024-03-20T09:45:38.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.301269+0000", 2024-03-20T09:45:38.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.1307e-05 2024-03-20T09:45:38.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.593 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:45:38.593 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.305281+0000", 2024-03-20T09:45:38.593 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0040123399999999997 2024-03-20T09:45:38.593 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.593 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:38.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.314604+0000", 2024-03-20T09:45:38.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0093226909999999993 2024-03-20T09:45:38.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:38.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.479505+0000", 2024-03-20T09:45:38.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.16490092100000001 2024-03-20T09:45:38.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.594 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:38.594 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.479608+0000", 2024-03-20T09:45:38.594 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000102929 2024-03-20T09:45:38.594 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.594 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.595 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:38.595 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.479685+0000", 2024-03-20T09:45:38.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.6766000000000003e-05 2024-03-20T09:45:38.595 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:38.595 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:38.595 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:38.595 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.595 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.595 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.27190.0:260848 9.2 9:47088d44:::benchmark_data_smithi122_210164_object128618:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1142)", 2024-03-20T09:45:38.595 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:44:36.307632+0000", 2024-03-20T09:45:38.595 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.100427416999999, 2024-03-20T09:45:38.595 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.18114635800000001, 2024-03-20T09:45:38.595 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:38.595 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:45:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.27190", 2024-03-20T09:45:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:45:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 260848 2024-03-20T09:45:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.307632+0000", 2024-03-20T09:45:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.307632+0000", 2024-03-20T09:45:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:38.596 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.307633+0000", 2024-03-20T09:45:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.16e-07 2024-03-20T09:45:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.307637+0000", 2024-03-20T09:45:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9260000000000002e-06 2024-03-20T09:45:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.307645+0000", 2024-03-20T09:45:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7010000000000002e-06 2024-03-20T09:45:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.597 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.307653+0000", 2024-03-20T09:45:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.2609999999999996e-06 2024-03-20T09:45:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.307702+0000", 2024-03-20T09:45:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.9505000000000001e-05 2024-03-20T09:45:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.307810+0000", 2024-03-20T09:45:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000108269 2024-03-20T09:45:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.598 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 1,9", 2024-03-20T09:45:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.307884+0000", 2024-03-20T09:45:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3236000000000002e-05 2024-03-20T09:45:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:45:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.312037+0000", 2024-03-20T09:45:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0041532239999999996 2024-03-20T09:45:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.313147+0000", 2024-03-20T09:45:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0011098150000000001 2024-03-20T09:45:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:38.599 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.488704+0000", 2024-03-20T09:45:38.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.175557782 2024-03-20T09:45:38.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:38.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.488741+0000", 2024-03-20T09:45:38.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6381000000000003e-05 2024-03-20T09:45:38.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.600 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:38.600 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.488778+0000", 2024-03-20T09:45:38.600 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7441999999999998e-05 2024-03-20T09:45:38.600 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:38.600 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:38.600 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:38.600 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.600 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.601 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "osd_op(client.27190.0:260851 9.c 9:3a0c59c0:::benchmark_data_smithi122_210164_object128621:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1142)", 2024-03-20T09:45:38.601 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:44:36.307731+0000", 2024-03-20T09:45:38.601 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 62.100328478000002, 2024-03-20T09:45:38.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.18260690800000001, 2024-03-20T09:45:38.601 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:38.601 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:38.601 INFO:teuthology.orchestra.run.smithi026.stdout: "client_info": { 2024-03-20T09:45:38.601 INFO:teuthology.orchestra.run.smithi026.stdout: "client": "client.27190", 2024-03-20T09:45:38.601 INFO:teuthology.orchestra.run.smithi026.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:45:38.601 INFO:teuthology.orchestra.run.smithi026.stdout: "tid": 260851 2024-03-20T09:45:38.601 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.601 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:38.601 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.601 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:38.601 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.307731+0000", 2024-03-20T09:45:38.601 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:38.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:38.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.307731+0000", 2024-03-20T09:45:38.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:38.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:38.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.307732+0000", 2024-03-20T09:45:38.602 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.3399999999999998e-07 2024-03-20T09:45:38.602 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.602 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.602 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:38.602 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.307736+0000", 2024-03-20T09:45:38.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.9620000000000004e-06 2024-03-20T09:45:38.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:38.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.307744+0000", 2024-03-20T09:45:38.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.7260000000000008e-06 2024-03-20T09:45:38.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:38.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.307751+0000", 2024-03-20T09:45:38.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.5490000000000003e-06 2024-03-20T09:45:38.603 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.603 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.603 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:38.603 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.307805+0000", 2024-03-20T09:45:38.603 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.4530999999999997e-05 2024-03-20T09:45:38.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:38.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.307918+0000", 2024-03-20T09:45:38.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000113026 2024-03-20T09:45:38.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "waiting for subops from 4,9", 2024-03-20T09:45:38.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.308028+0000", 2024-03-20T09:45:38.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000109183 2024-03-20T09:45:38.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.604 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.604 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "op_commit", 2024-03-20T09:45:38.604 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.312714+0000", 2024-03-20T09:45:38.604 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.0046862869999999999 2024-03-20T09:45:38.604 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:38.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.313509+0000", 2024-03-20T09:45:38.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00079554799999999998 2024-03-20T09:45:38.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:38.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.490273+0000", 2024-03-20T09:45:38.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.17676386799999999 2024-03-20T09:45:38.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "commit_sent", 2024-03-20T09:45:38.605 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.490309+0000", 2024-03-20T09:45:38.605 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.5701000000000003e-05 2024-03-20T09:45:38.605 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.605 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.605 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:38.606 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:44:36.490338+0000", 2024-03-20T09:45:38.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.8793e-05 2024-03-20T09:45:38.606 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:38.606 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:38.606 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:38.606 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.606 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDScrubReserve(1.1a,GRANT,e:1157,from:4,reservation_nonce:0)", 2024-03-20T09:45:38.606 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:45:14.555176+0000", 2024-03-20T09:45:38.606 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 23.852883683000002, 2024-03-20T09:45:38.606 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000140808, 2024-03-20T09:45:38.606 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:38.606 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:45:38.606 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:38.606 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:38.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:14.555176+0000", 2024-03-20T09:45:38.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:38.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:38.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:14.555176+0000", 2024-03-20T09:45:38.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:38.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:38.607 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:14.555176+0000", 2024-03-20T09:45:38.607 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.3300000000000002e-07 2024-03-20T09:45:38.607 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.607 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.607 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:38.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:14.555180+0000", 2024-03-20T09:45:38.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.7850000000000002e-06 2024-03-20T09:45:38.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:38.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:14.555191+0000", 2024-03-20T09:45:38.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1131999999999999e-05 2024-03-20T09:45:38.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:38.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:14.555202+0000", 2024-03-20T09:45:38.608 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1046999999999999e-05 2024-03-20T09:45:38.608 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.608 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.608 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:38.608 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:14.555258+0000", 2024-03-20T09:45:38.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.5399000000000003e-05 2024-03-20T09:45:38.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:38.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:14.555282+0000", 2024-03-20T09:45:38.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4283000000000001e-05 2024-03-20T09:45:38.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.609 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:38.609 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:14.555316+0000", 2024-03-20T09:45:38.609 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.4629000000000003e-05 2024-03-20T09:45:38.609 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:38.609 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:38.609 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:38.609 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.609 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.610 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDScrubReserve(1.1a,GRANT,e:1157,from:7,reservation_nonce:0)", 2024-03-20T09:45:38.610 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:45:14.555727+0000", 2024-03-20T09:45:38.610 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 23.852331852999999, 2024-03-20T09:45:38.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00032325700000000003, 2024-03-20T09:45:38.610 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:38.610 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:45:38.610 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:38.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:38.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:14.555727+0000", 2024-03-20T09:45:38.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:38.610 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.610 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.610 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:38.610 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:14.555727+0000", 2024-03-20T09:45:38.610 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:38.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:38.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:14.555728+0000", 2024-03-20T09:45:38.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.13e-07 2024-03-20T09:45:38.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:38.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:14.555732+0000", 2024-03-20T09:45:38.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6660000000000001e-06 2024-03-20T09:45:38.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.611 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.611 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:38.611 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:14.555741+0000", 2024-03-20T09:45:38.611 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.9120000000000004e-06 2024-03-20T09:45:38.611 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:38.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:14.555747+0000", 2024-03-20T09:45:38.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6229999999999999e-06 2024-03-20T09:45:38.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:38.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:14.555929+0000", 2024-03-20T09:45:38.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018178700000000001 2024-03-20T09:45:38.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.612 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:38.612 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:14.555944+0000", 2024-03-20T09:45:38.612 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5169e-05 2024-03-20T09:45:38.612 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.612 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.613 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:38.613 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:14.556051+0000", 2024-03-20T09:45:38.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000106587 2024-03-20T09:45:38.613 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:38.613 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:38.613 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:38.613 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.613 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.613 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "rep_scrubmap(1.1a e1094 from shard 4)", 2024-03-20T09:45:38.613 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:45:14.596474+0000", 2024-03-20T09:45:38.613 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 23.811584919000001, 2024-03-20T09:45:38.613 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.037492800999999999, 2024-03-20T09:45:38.613 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:38.613 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:45:38.613 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:38.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:38.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:14.596474+0000", 2024-03-20T09:45:38.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:38.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:38.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:14.596474+0000", 2024-03-20T09:45:38.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:38.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.614 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:38.614 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:14.596476+0000", 2024-03-20T09:45:38.614 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3799999999999999e-06 2024-03-20T09:45:38.614 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.614 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:38.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:14.596482+0000", 2024-03-20T09:45:38.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.0159999999999997e-06 2024-03-20T09:45:38.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:38.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:14.596496+0000", 2024-03-20T09:45:38.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.4328000000000001e-05 2024-03-20T09:45:38.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:38.615 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:14.596507+0000", 2024-03-20T09:45:38.615 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1141000000000001e-05 2024-03-20T09:45:38.615 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.615 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.615 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:38.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:14.633945+0000", 2024-03-20T09:45:38.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.037437656 2024-03-20T09:45:38.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:38.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:14.633955+0000", 2024-03-20T09:45:38.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0319000000000001e-05 2024-03-20T09:45:38.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.616 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:38.616 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:14.633967+0000", 2024-03-20T09:45:38.616 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.1960999999999999e-05 2024-03-20T09:45:38.616 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:38.616 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:38.616 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:38.616 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.616 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.617 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "rep_scrubmap(1.1a e1094 from shard 7)", 2024-03-20T09:45:38.617 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:45:14.596551+0000", 2024-03-20T09:45:38.617 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 23.811508727, 2024-03-20T09:45:38.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.037460393000000002, 2024-03-20T09:45:38.617 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:38.617 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:45:38.617 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:38.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:38.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:14.596551+0000", 2024-03-20T09:45:38.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:38.617 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.617 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.617 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:38.617 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:14.596551+0000", 2024-03-20T09:45:38.617 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:38.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:38.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:14.596551+0000", 2024-03-20T09:45:38.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.68e-07 2024-03-20T09:45:38.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:38.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:14.596555+0000", 2024-03-20T09:45:38.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.0749999999999999e-06 2024-03-20T09:45:38.618 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.618 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.618 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:38.618 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:14.596564+0000", 2024-03-20T09:45:38.618 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 9.0270000000000001e-06 2024-03-20T09:45:38.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:38.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:14.596570+0000", 2024-03-20T09:45:38.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 5.1769999999999997e-06 2024-03-20T09:45:38.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:38.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:14.633984+0000", 2024-03-20T09:45:38.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.037414559 2024-03-20T09:45:38.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.619 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.619 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:38.619 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:14.633991+0000", 2024-03-20T09:45:38.619 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 6.6919999999999996e-06 2024-03-20T09:45:38.619 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.620 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:38.620 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:14.634011+0000", 2024-03-20T09:45:38.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.0194999999999999e-05 2024-03-20T09:45:38.620 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:38.620 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:38.620 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:38.620 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.620 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.620 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDScrubReserve(1.7,QREQUEST,e:1164,from:6,reservation_nonce:16)", 2024-03-20T09:45:38.620 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:45:32.825569+0000", 2024-03-20T09:45:38.620 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 5.5824898740000002, 2024-03-20T09:45:38.620 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.000222897, 2024-03-20T09:45:38.620 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:38.620 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:45:38.620 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:38.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:38.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:32.825569+0000", 2024-03-20T09:45:38.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:38.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:38.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:32.825569+0000", 2024-03-20T09:45:38.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:38.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:38.621 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:32.825571+0000", 2024-03-20T09:45:38.621 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.866e-06 2024-03-20T09:45:38.621 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.621 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.621 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:38.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:32.825584+0000", 2024-03-20T09:45:38.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2227e-05 2024-03-20T09:45:38.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:38.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:32.825617+0000", 2024-03-20T09:45:38.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.3033e-05 2024-03-20T09:45:38.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:38.622 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:32.825641+0000", 2024-03-20T09:45:38.622 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4243000000000002e-05 2024-03-20T09:45:38.622 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.622 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.622 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:38.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:32.825727+0000", 2024-03-20T09:45:38.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6014999999999996e-05 2024-03-20T09:45:38.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:38.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:32.825746+0000", 2024-03-20T09:45:38.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.9049999999999999e-05 2024-03-20T09:45:38.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.623 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.623 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:38.623 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:32.825792+0000", 2024-03-20T09:45:38.623 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 4.6462999999999999e-05 2024-03-20T09:45:38.623 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:38.623 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:38.623 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:38.623 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.624 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "replica_scrub(pg: 1.7,from:0'0,to:0'0,epoch:1164/960,start:1:e0000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-03-20T09:45:38.624 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:45:32.916752+0000", 2024-03-20T09:45:38.624 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 5.4913077040000005, 2024-03-20T09:45:38.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00018497399999999999, 2024-03-20T09:45:38.624 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:38.624 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:45:38.624 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:38.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:38.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:32.916752+0000", 2024-03-20T09:45:38.624 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:38.624 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.624 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.624 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:38.624 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:32.916752+0000", 2024-03-20T09:45:38.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:38.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:38.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:32.916753+0000", 2024-03-20T09:45:38.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5099999999999999e-06 2024-03-20T09:45:38.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:38.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:32.916769+0000", 2024-03-20T09:45:38.625 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5441999999999999e-05 2024-03-20T09:45:38.625 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.625 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.625 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:38.625 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:32.916805+0000", 2024-03-20T09:45:38.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 3.6273999999999999e-05 2024-03-20T09:45:38.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:38.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:32.916821+0000", 2024-03-20T09:45:38.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.6158999999999999e-05 2024-03-20T09:45:38.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:38.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:32.916900+0000", 2024-03-20T09:45:38.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 7.8869000000000007e-05 2024-03-20T09:45:38.626 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.626 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.626 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:38.626 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:32.916912+0000", 2024-03-20T09:45:38.626 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.2325e-05 2024-03-20T09:45:38.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.627 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:38.627 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:32.916937+0000", 2024-03-20T09:45:38.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.4394999999999999e-05 2024-03-20T09:45:38.627 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:38.627 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:38.627 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:38.627 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.627 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.627 INFO:teuthology.orchestra.run.smithi026.stdout: "description": "MOSDScrubReserve(1.7,RELEASE,e:1164,from:6,reservation_nonce:0)", 2024-03-20T09:45:38.627 INFO:teuthology.orchestra.run.smithi026.stdout: "initiated_at": "2024-03-20T09:45:33.030611+0000", 2024-03-20T09:45:38.627 INFO:teuthology.orchestra.run.smithi026.stdout: "age": 5.3774482939999997, 2024-03-20T09:45:38.627 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0.00016619900000000001, 2024-03-20T09:45:38.627 INFO:teuthology.orchestra.run.smithi026.stdout: "type_data": { 2024-03-20T09:45:38.627 INFO:teuthology.orchestra.run.smithi026.stdout: "flag_point": "started", 2024-03-20T09:45:38.628 INFO:teuthology.orchestra.run.smithi026.stdout: "events": [ 2024-03-20T09:45:38.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "initiated", 2024-03-20T09:45:38.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:33.030611+0000", 2024-03-20T09:45:38.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:38.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "header_read", 2024-03-20T09:45:38.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:33.030611+0000", 2024-03-20T09:45:38.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 0 2024-03-20T09:45:38.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.628 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.628 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "throttled", 2024-03-20T09:45:38.628 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:33.030612+0000", 2024-03-20T09:45:38.628 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.6300000000000004e-07 2024-03-20T09:45:38.628 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "all_read", 2024-03-20T09:45:38.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:33.030637+0000", 2024-03-20T09:45:38.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 2.5204000000000001e-05 2024-03-20T09:45:38.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "dispatched", 2024-03-20T09:45:38.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:33.030651+0000", 2024-03-20T09:45:38.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.3774999999999999e-05 2024-03-20T09:45:38.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "queued_for_pg", 2024-03-20T09:45:38.629 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:33.030661+0000", 2024-03-20T09:45:38.629 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.0424999999999999e-05 2024-03-20T09:45:38.629 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.629 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.629 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "reached_pg", 2024-03-20T09:45:38.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:33.030744+0000", 2024-03-20T09:45:38.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 8.3183000000000006e-05 2024-03-20T09:45:38.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "started", 2024-03-20T09:45:38.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:33.030760+0000", 2024-03-20T09:45:38.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.5636000000000001e-05 2024-03-20T09:45:38.630 INFO:teuthology.orchestra.run.smithi026.stdout: }, 2024-03-20T09:45:38.630 INFO:teuthology.orchestra.run.smithi026.stdout: { 2024-03-20T09:45:38.630 INFO:teuthology.orchestra.run.smithi026.stdout: "event": "done", 2024-03-20T09:45:38.630 INFO:teuthology.orchestra.run.smithi026.stdout: "time": "2024-03-20T09:45:33.030777+0000", 2024-03-20T09:45:38.630 INFO:teuthology.orchestra.run.smithi026.stdout: "duration": 1.7113e-05 2024-03-20T09:45:38.630 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:38.630 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:38.630 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:38.630 INFO:teuthology.orchestra.run.smithi026.stdout: } 2024-03-20T09:45:38.631 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-20T09:45:38.631 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-20T09:45:38.909 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd primary-affinity 2 1 2024-03-20T09:45:39.046 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:38 smithi026 ceph-mon[68309]: 1.3 deep-scrub ok 2024-03-20T09:45:39.046 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:38 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/308732972' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-20T09:45:39.046 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:38 smithi026 ceph-mon[68309]: osdmap e1166: 12 total, 12 up, 12 in 2024-03-20T09:45:39.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:38 smithi115 ceph-mon[72476]: 1.3 deep-scrub ok 2024-03-20T09:45:39.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:38 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/308732972' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-20T09:45:39.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:38 smithi115 ceph-mon[72476]: osdmap e1166: 12 total, 12 up, 12 in 2024-03-20T09:45:39.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:38 smithi122 ceph-mon[73495]: 1.3 deep-scrub ok 2024-03-20T09:45:39.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:38 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/308732972' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-20T09:45:39.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:38 smithi122 ceph-mon[73495]: osdmap e1166: 12 total, 12 up, 12 in 2024-03-20T09:45:39.489 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-20T09:45:39.545 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:45:39.763 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:45:40.063 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:39 smithi115 ceph-mon[72476]: pgmap v2837: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:40.063 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:39 smithi115 ceph-mon[72476]: 1.1c scrub starts 2024-03-20T09:45:40.063 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:39 smithi115 ceph-mon[72476]: 1.1c scrub ok 2024-03-20T09:45:40.063 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:39 smithi115 ceph-mon[72476]: 1.16 scrub starts 2024-03-20T09:45:40.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:39 smithi122 ceph-mon[73495]: pgmap v2837: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:40.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:39 smithi122 ceph-mon[73495]: 1.1c scrub starts 2024-03-20T09:45:40.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:39 smithi122 ceph-mon[73495]: 1.1c scrub ok 2024-03-20T09:45:40.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:39 smithi122 ceph-mon[73495]: 1.16 scrub starts 2024-03-20T09:45:40.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:39 smithi026 ceph-mon[68309]: pgmap v2837: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:40.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:39 smithi026 ceph-mon[68309]: 1.1c scrub starts 2024-03-20T09:45:40.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:39 smithi026 ceph-mon[68309]: 1.1c scrub ok 2024-03-20T09:45:40.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:39 smithi026 ceph-mon[68309]: 1.16 scrub starts 2024-03-20T09:45:41.026 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:40 smithi026 ceph-mon[68309]: 1.16 scrub ok 2024-03-20T09:45:41.048 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:40 smithi115 ceph-mon[72476]: 1.16 scrub ok 2024-03-20T09:45:41.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:40 smithi122 ceph-mon[73495]: 1.16 scrub ok 2024-03-20T09:45:41.597 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:45:41.597 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:45:41.597 INFO:teuthology.orchestra.run.smithi115.stdout: "num_ops": 0 2024-03-20T09:45:41.597 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:45:41.807 INFO:teuthology.orchestra.run.smithi026.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-20T09:45:41.892 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:41 smithi115 ceph-mon[72476]: pgmap v2838: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:41.893 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:41 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1008393996' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:45:42.160 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:41 smithi026 ceph-mon[68309]: pgmap v2838: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:42.160 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:41 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1008393996' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:45:42.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:41 smithi122 ceph-mon[73495]: pgmap v2838: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:42.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:41 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1008393996' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-20T09:45:42.267 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-20T09:45:42.530 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:45:42.575 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 3 1 2024-03-20T09:45:42.814 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:42 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1008393996' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-20T09:45:42.814 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:42 smithi026 ceph-mon[68309]: osdmap e1167: 12 total, 12 up, 12 in 2024-03-20T09:45:42.860 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:45:43.049 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:42 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1008393996' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-20T09:45:43.049 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:42 smithi115 ceph-mon[72476]: osdmap e1167: 12 total, 12 up, 12 in 2024-03-20T09:45:43.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:42 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1008393996' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-20T09:45:43.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:42 smithi122 ceph-mon[73495]: osdmap e1167: 12 total, 12 up, 12 in 2024-03-20T09:45:43.992 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: pgmap v2840: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:43.992 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {auth=31,config=2,health=12,kv=55} crc {auth=468943359,config=2679774077,health=3897446751,kv=3787519498}) 2024-03-20T09:45:43.992 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {kv=6,logm=94} crc {kv=985904887,logm=3006310311}) 2024-03-20T09:45:43.992 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2307308668}) 2024-03-20T09:45:43.992 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3543012921}) 2024-03-20T09:45:43.992 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3489320101}) 2024-03-20T09:45:43.992 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2588804079}) 2024-03-20T09:45:43.992 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1252353007}) 2024-03-20T09:45:43.992 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=186895739}) 2024-03-20T09:45:43.993 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=395344853}) 2024-03-20T09:45:43.993 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2662394134}) 2024-03-20T09:45:43.993 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1221474424}) 2024-03-20T09:45:43.993 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3507683362}) 2024-03-20T09:45:43.993 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=99752390}) 2024-03-20T09:45:43.993 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1660321640}) 2024-03-20T09:45:43.993 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1110953866}) 2024-03-20T09:45:43.993 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1454141796}) 2024-03-20T09:45:43.993 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3674130086}) 2024-03-20T09:45:43.993 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173237947}) 2024-03-20T09:45:43.993 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4271256971}) 2024-03-20T09:45:43.993 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3786100698}) 2024-03-20T09:45:43.994 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3981235565}) 2024-03-20T09:45:43.994 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3671716925}) 2024-03-20T09:45:43.994 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=18344262}) 2024-03-20T09:45:43.994 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2712725659}) 2024-03-20T09:45:43.994 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1503099635}) 2024-03-20T09:45:43.994 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=683442135}) 2024-03-20T09:45:43.994 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=694718766}) 2024-03-20T09:45:43.994 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4237268612}) 2024-03-20T09:45:43.994 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2863063034}) 2024-03-20T09:45:43.994 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=145046285}) 2024-03-20T09:45:43.994 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1902644555}) 2024-03-20T09:45:43.994 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3729913511}) 2024-03-20T09:45:43.995 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=779469315}) 2024-03-20T09:45:43.995 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2586278531}) 2024-03-20T09:45:43.995 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3151146605}) 2024-03-20T09:45:43.995 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=656913120}) 2024-03-20T09:45:43.995 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2794893262}) 2024-03-20T09:45:43.995 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1333628599}) 2024-03-20T09:45:43.995 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=200782514}) 2024-03-20T09:45:43.995 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2836040915}) 2024-03-20T09:45:43.995 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4264839171}) 2024-03-20T09:45:43.995 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=977955017}) 2024-03-20T09:45:43.995 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2708039981}) 2024-03-20T09:45:43.995 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3165271155}) 2024-03-20T09:45:43.995 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1872309049}) 2024-03-20T09:45:43.996 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=884897093}) 2024-03-20T09:45:43.996 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1794563322}) 2024-03-20T09:45:43.996 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=816249318}) 2024-03-20T09:45:43.996 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2715516051}) 2024-03-20T09:45:43.996 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1350420747}) 2024-03-20T09:45:43.996 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3205600041}) 2024-03-20T09:45:43.996 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1457283699}) 2024-03-20T09:45:43.996 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1338535746}) 2024-03-20T09:45:43.996 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3873765646}) 2024-03-20T09:45:43.996 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3850763417}) 2024-03-20T09:45:43.996 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1721439851}) 2024-03-20T09:45:43.996 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2962553287}) 2024-03-20T09:45:43.996 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1272769225}) 2024-03-20T09:45:43.996 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=935444890}) 2024-03-20T09:45:43.996 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2745444659}) 2024-03-20T09:45:43.997 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2096880682}) 2024-03-20T09:45:43.997 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3044653563}) 2024-03-20T09:45:43.997 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2783729790}) 2024-03-20T09:45:43.997 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=494173609}) 2024-03-20T09:45:43.997 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2762304327}) 2024-03-20T09:45:43.997 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=371727687}) 2024-03-20T09:45:43.997 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1238811082}) 2024-03-20T09:45:43.997 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4154479773}) 2024-03-20T09:45:43.997 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=540010535}) 2024-03-20T09:45:43.997 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2669301424}) 2024-03-20T09:45:43.998 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4259599314}) 2024-03-20T09:45:43.998 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=218256349}) 2024-03-20T09:45:43.998 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3488461101}) 2024-03-20T09:45:43.998 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3834237020}) 2024-03-20T09:45:43.998 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=720618148}) 2024-03-20T09:45:43.998 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=132323715}) 2024-03-20T09:45:43.998 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=758175116}) 2024-03-20T09:45:43.998 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3955621885}) 2024-03-20T09:45:43.998 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=138168552}) 2024-03-20T09:45:43.998 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3679122715}) 2024-03-20T09:45:43.999 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2638334506}) 2024-03-20T09:45:43.999 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1692224037}) 2024-03-20T09:45:43.999 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=34127779}) 2024-03-20T09:45:43.999 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3481872439}) 2024-03-20T09:45:43.999 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2636773622}) 2024-03-20T09:45:43.999 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3427928824}) 2024-03-20T09:45:43.999 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=321911388}) 2024-03-20T09:45:43.999 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2920877539}) 2024-03-20T09:45:43.999 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1968970429}) 2024-03-20T09:45:43.999 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1781171442}) 2024-03-20T09:45:43.999 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1300201135}) 2024-03-20T09:45:43.999 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=492768974}) 2024-03-20T09:45:43.999 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=954485667}) 2024-03-20T09:45:43.999 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1503916025}) 2024-03-20T09:45:44.000 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1893245842}) 2024-03-20T09:45:44.000 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3885486018}) 2024-03-20T09:45:44.000 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3152819634}) 2024-03-20T09:45:44.000 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3031325831}) 2024-03-20T09:45:44.000 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1809932911}) 2024-03-20T09:45:44.000 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2364191123}) 2024-03-20T09:45:44.000 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=438142247}) 2024-03-20T09:45:44.000 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3921153728}) 2024-03-20T09:45:44.000 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {logm=50,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=16,mon_config_key=10} crc {logm=2906516983,mdsmap=3984425268,mgr=427508465,mgr_command_descs=3831600018,mgr_metadata=2702834764,mgrstat=2752800345,mon_config_key=444575730}) 2024-03-20T09:45:44.000 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1167437155}) 2024-03-20T09:45:44.000 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=521537992}) 2024-03-20T09:45:44.000 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=71,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=11} crc {mon_config_key=3921212036,monmap=3481182261,osd_metadata=222046520,osd_pg_creating=3062116144,osdmap=3570908654}) 2024-03-20T09:45:44.000 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2630124910}) 2024-03-20T09:45:44.000 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:43 smithi026 ceph-mon[68309]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=2} crc {osdmap=1601992281}) 2024-03-20T09:45:44.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: pgmap v2840: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:44.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {auth=31,config=2,health=12,kv=55} crc {auth=468943359,config=2679774077,health=3897446751,kv=3787519498}) 2024-03-20T09:45:44.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {kv=6,logm=94} crc {kv=985904887,logm=3006310311}) 2024-03-20T09:45:44.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2307308668}) 2024-03-20T09:45:44.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3543012921}) 2024-03-20T09:45:44.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3489320101}) 2024-03-20T09:45:44.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2588804079}) 2024-03-20T09:45:44.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1252353007}) 2024-03-20T09:45:44.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=186895739}) 2024-03-20T09:45:44.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=395344853}) 2024-03-20T09:45:44.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2662394134}) 2024-03-20T09:45:44.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1221474424}) 2024-03-20T09:45:44.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3507683362}) 2024-03-20T09:45:44.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=99752390}) 2024-03-20T09:45:44.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1660321640}) 2024-03-20T09:45:44.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1110953866}) 2024-03-20T09:45:44.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1454141796}) 2024-03-20T09:45:44.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3674130086}) 2024-03-20T09:45:44.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173237947}) 2024-03-20T09:45:44.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4271256971}) 2024-03-20T09:45:44.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3786100698}) 2024-03-20T09:45:44.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3981235565}) 2024-03-20T09:45:44.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3671716925}) 2024-03-20T09:45:44.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=18344262}) 2024-03-20T09:45:44.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2712725659}) 2024-03-20T09:45:44.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1503099635}) 2024-03-20T09:45:44.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=683442135}) 2024-03-20T09:45:44.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=694718766}) 2024-03-20T09:45:44.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4237268612}) 2024-03-20T09:45:44.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2863063034}) 2024-03-20T09:45:44.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=145046285}) 2024-03-20T09:45:44.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1902644555}) 2024-03-20T09:45:44.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3729913511}) 2024-03-20T09:45:44.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=779469315}) 2024-03-20T09:45:44.145 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2586278531}) 2024-03-20T09:45:44.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3151146605}) 2024-03-20T09:45:44.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=656913120}) 2024-03-20T09:45:44.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2794893262}) 2024-03-20T09:45:44.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1333628599}) 2024-03-20T09:45:44.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=200782514}) 2024-03-20T09:45:44.146 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2836040915}) 2024-03-20T09:45:44.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4264839171}) 2024-03-20T09:45:44.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=977955017}) 2024-03-20T09:45:44.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2708039981}) 2024-03-20T09:45:44.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3165271155}) 2024-03-20T09:45:44.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1872309049}) 2024-03-20T09:45:44.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=884897093}) 2024-03-20T09:45:44.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1794563322}) 2024-03-20T09:45:44.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=816249318}) 2024-03-20T09:45:44.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2715516051}) 2024-03-20T09:45:44.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1350420747}) 2024-03-20T09:45:44.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3205600041}) 2024-03-20T09:45:44.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1457283699}) 2024-03-20T09:45:44.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1338535746}) 2024-03-20T09:45:44.147 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3873765646}) 2024-03-20T09:45:44.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3850763417}) 2024-03-20T09:45:44.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1721439851}) 2024-03-20T09:45:44.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2962553287}) 2024-03-20T09:45:44.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1272769225}) 2024-03-20T09:45:44.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=935444890}) 2024-03-20T09:45:44.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2745444659}) 2024-03-20T09:45:44.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2096880682}) 2024-03-20T09:45:44.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3044653563}) 2024-03-20T09:45:44.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2783729790}) 2024-03-20T09:45:44.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=494173609}) 2024-03-20T09:45:44.148 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2762304327}) 2024-03-20T09:45:44.149 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=371727687}) 2024-03-20T09:45:44.149 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1238811082}) 2024-03-20T09:45:44.149 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4154479773}) 2024-03-20T09:45:44.149 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=540010535}) 2024-03-20T09:45:44.149 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2669301424}) 2024-03-20T09:45:44.149 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4259599314}) 2024-03-20T09:45:44.149 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=218256349}) 2024-03-20T09:45:44.149 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3488461101}) 2024-03-20T09:45:44.149 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3834237020}) 2024-03-20T09:45:44.149 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=720618148}) 2024-03-20T09:45:44.149 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=132323715}) 2024-03-20T09:45:44.149 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=758175116}) 2024-03-20T09:45:44.149 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3955621885}) 2024-03-20T09:45:44.149 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=138168552}) 2024-03-20T09:45:44.150 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3679122715}) 2024-03-20T09:45:44.150 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2638334506}) 2024-03-20T09:45:44.150 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1692224037}) 2024-03-20T09:45:44.150 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=34127779}) 2024-03-20T09:45:44.150 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3481872439}) 2024-03-20T09:45:44.150 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2636773622}) 2024-03-20T09:45:44.150 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3427928824}) 2024-03-20T09:45:44.150 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=321911388}) 2024-03-20T09:45:44.150 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2920877539}) 2024-03-20T09:45:44.150 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1968970429}) 2024-03-20T09:45:44.150 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1781171442}) 2024-03-20T09:45:44.150 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1300201135}) 2024-03-20T09:45:44.151 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=492768974}) 2024-03-20T09:45:44.151 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=954485667}) 2024-03-20T09:45:44.151 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1503916025}) 2024-03-20T09:45:44.151 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1893245842}) 2024-03-20T09:45:44.151 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3885486018}) 2024-03-20T09:45:44.151 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3152819634}) 2024-03-20T09:45:44.151 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3031325831}) 2024-03-20T09:45:44.151 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1809932911}) 2024-03-20T09:45:44.151 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2364191123}) 2024-03-20T09:45:44.151 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=438142247}) 2024-03-20T09:45:44.151 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3921153728}) 2024-03-20T09:45:44.151 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {logm=50,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=16,mon_config_key=10} crc {logm=2906516983,mdsmap=3984425268,mgr=427508465,mgr_command_descs=3831600018,mgr_metadata=2702834764,mgrstat=2752800345,mon_config_key=444575730}) 2024-03-20T09:45:44.152 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1167437155}) 2024-03-20T09:45:44.152 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=521537992}) 2024-03-20T09:45:44.152 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=71,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=11} crc {mon_config_key=3921212036,monmap=3481182261,osd_metadata=222046520,osd_pg_creating=3062116144,osdmap=3570908654}) 2024-03-20T09:45:44.152 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2630124910}) 2024-03-20T09:45:44.152 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:43 smithi115 ceph-mon[72476]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=2} crc {osdmap=1601992281}) 2024-03-20T09:45:44.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: pgmap v2840: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:44.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {auth=31,config=2,health=12,kv=55} crc {auth=468943359,config=2679774077,health=3897446751,kv=3787519498}) 2024-03-20T09:45:44.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {kv=6,logm=94} crc {kv=985904887,logm=3006310311}) 2024-03-20T09:45:44.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2307308668}) 2024-03-20T09:45:44.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3543012921}) 2024-03-20T09:45:44.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3489320101}) 2024-03-20T09:45:44.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2588804079}) 2024-03-20T09:45:44.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1252353007}) 2024-03-20T09:45:44.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=186895739}) 2024-03-20T09:45:44.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=395344853}) 2024-03-20T09:45:44.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2662394134}) 2024-03-20T09:45:44.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1221474424}) 2024-03-20T09:45:44.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3507683362}) 2024-03-20T09:45:44.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=99752390}) 2024-03-20T09:45:44.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1660321640}) 2024-03-20T09:45:44.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1110953866}) 2024-03-20T09:45:44.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1454141796}) 2024-03-20T09:45:44.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3674130086}) 2024-03-20T09:45:44.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1173237947}) 2024-03-20T09:45:44.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4271256971}) 2024-03-20T09:45:44.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3786100698}) 2024-03-20T09:45:44.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3981235565}) 2024-03-20T09:45:44.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3671716925}) 2024-03-20T09:45:44.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=18344262}) 2024-03-20T09:45:44.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2712725659}) 2024-03-20T09:45:44.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1503099635}) 2024-03-20T09:45:44.185 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=683442135}) 2024-03-20T09:45:44.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=694718766}) 2024-03-20T09:45:44.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4237268612}) 2024-03-20T09:45:44.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2863063034}) 2024-03-20T09:45:44.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=145046285}) 2024-03-20T09:45:44.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1902644555}) 2024-03-20T09:45:44.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3729913511}) 2024-03-20T09:45:44.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=779469315}) 2024-03-20T09:45:44.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2586278531}) 2024-03-20T09:45:44.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3151146605}) 2024-03-20T09:45:44.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=656913120}) 2024-03-20T09:45:44.186 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2794893262}) 2024-03-20T09:45:44.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1333628599}) 2024-03-20T09:45:44.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=200782514}) 2024-03-20T09:45:44.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2836040915}) 2024-03-20T09:45:44.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4264839171}) 2024-03-20T09:45:44.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=977955017}) 2024-03-20T09:45:44.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2708039981}) 2024-03-20T09:45:44.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3165271155}) 2024-03-20T09:45:44.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1872309049}) 2024-03-20T09:45:44.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=884897093}) 2024-03-20T09:45:44.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1794563322}) 2024-03-20T09:45:44.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=816249318}) 2024-03-20T09:45:44.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2715516051}) 2024-03-20T09:45:44.187 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1350420747}) 2024-03-20T09:45:44.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3205600041}) 2024-03-20T09:45:44.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1457283699}) 2024-03-20T09:45:44.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1338535746}) 2024-03-20T09:45:44.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3873765646}) 2024-03-20T09:45:44.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3850763417}) 2024-03-20T09:45:44.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1721439851}) 2024-03-20T09:45:44.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2962553287}) 2024-03-20T09:45:44.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1272769225}) 2024-03-20T09:45:44.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=935444890}) 2024-03-20T09:45:44.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2745444659}) 2024-03-20T09:45:44.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2096880682}) 2024-03-20T09:45:44.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3044653563}) 2024-03-20T09:45:44.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2783729790}) 2024-03-20T09:45:44.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=494173609}) 2024-03-20T09:45:44.188 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2762304327}) 2024-03-20T09:45:44.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=371727687}) 2024-03-20T09:45:44.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1238811082}) 2024-03-20T09:45:44.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4154479773}) 2024-03-20T09:45:44.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=540010535}) 2024-03-20T09:45:44.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2669301424}) 2024-03-20T09:45:44.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4259599314}) 2024-03-20T09:45:44.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=218256349}) 2024-03-20T09:45:44.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3488461101}) 2024-03-20T09:45:44.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3834237020}) 2024-03-20T09:45:44.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=720618148}) 2024-03-20T09:45:44.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=132323715}) 2024-03-20T09:45:44.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=758175116}) 2024-03-20T09:45:44.189 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3955621885}) 2024-03-20T09:45:44.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=138168552}) 2024-03-20T09:45:44.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3679122715}) 2024-03-20T09:45:44.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2638334506}) 2024-03-20T09:45:44.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1692224037}) 2024-03-20T09:45:44.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=34127779}) 2024-03-20T09:45:44.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3481872439}) 2024-03-20T09:45:44.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2636773622}) 2024-03-20T09:45:44.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3427928824}) 2024-03-20T09:45:44.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=321911388}) 2024-03-20T09:45:44.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2920877539}) 2024-03-20T09:45:44.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1968970429}) 2024-03-20T09:45:44.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1781171442}) 2024-03-20T09:45:44.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1300201135}) 2024-03-20T09:45:44.190 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=492768974}) 2024-03-20T09:45:44.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=954485667}) 2024-03-20T09:45:44.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1503916025}) 2024-03-20T09:45:44.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1893245842}) 2024-03-20T09:45:44.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3885486018}) 2024-03-20T09:45:44.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3152819634}) 2024-03-20T09:45:44.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3031325831}) 2024-03-20T09:45:44.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1809932911}) 2024-03-20T09:45:44.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2364191123}) 2024-03-20T09:45:44.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=438142247}) 2024-03-20T09:45:44.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3921153728}) 2024-03-20T09:45:44.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {logm=50,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=16,mon_config_key=10} crc {logm=2906516983,mdsmap=3984425268,mgr=427508465,mgr_command_descs=3831600018,mgr_metadata=2702834764,mgrstat=2752800345,mon_config_key=444575730}) 2024-03-20T09:45:44.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1167437155}) 2024-03-20T09:45:44.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=521537992}) 2024-03-20T09:45:44.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=71,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=11} crc {mon_config_key=3921212036,monmap=3481182261,osd_metadata=222046520,osd_pg_creating=3062116144,osdmap=3570908654}) 2024-03-20T09:45:44.191 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2630124910}) 2024-03-20T09:45:44.192 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:43 smithi122 ceph-mon[73495]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=2} crc {osdmap=1601992281}) 2024-03-20T09:45:44.231 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:45:44.231 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:45:44.231 INFO:teuthology.orchestra.run.smithi115.stdout: "complaint_time": 30, 2024-03-20T09:45:44.231 INFO:teuthology.orchestra.run.smithi115.stdout: "num_blocked_ops": 0 2024-03-20T09:45:44.231 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:45:44.826 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.3 to 1 (10000) 2024-03-20T09:45:44.871 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:44 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3087036155' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-20T09:45:44.897 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-20T09:45:45.158 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:44 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3087036155' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-20T09:45:45.160 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:45:45.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:44 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3087036155' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-20T09:45:45.601 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd primary-affinity 3 1 2024-03-20T09:45:45.886 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:45:46.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:45 smithi115 ceph-mon[72476]: pgmap v2841: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:46.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:45 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3087036155' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-20T09:45:46.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:45 smithi115 ceph-mon[72476]: osdmap e1168: 12 total, 12 up, 12 in 2024-03-20T09:45:46.146 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:45 smithi026 ceph-mon[68309]: pgmap v2841: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:46.146 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:45 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3087036155' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-20T09:45:46.146 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:45 smithi026 ceph-mon[68309]: osdmap e1168: 12 total, 12 up, 12 in 2024-03-20T09:45:46.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:45 smithi122 ceph-mon[73495]: pgmap v2841: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:46.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:45 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3087036155' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-20T09:45:46.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:45 smithi122 ceph-mon[73495]: osdmap e1168: 12 total, 12 up, 12 in 2024-03-20T09:45:47.028 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:45:47.028 INFO:teuthology.orchestra.run.smithi115.stdout: "size": 20, 2024-03-20T09:45:47.028 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 600, 2024-03-20T09:45:47.028 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [ 2024-03-20T09:45:47.028 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.028 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:125419 9.4 9:2fd45f54:::benchmark_data_smithi122_210164_object125418:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1097)", 2024-03-20T09:45:47.028 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:41:57.676703+0000", 2024-03-20T09:45:47.028 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 229.36173665600001, 2024-03-20T09:45:47.028 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.215804674, 2024-03-20T09:45:47.028 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:47.028 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:47.029 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:45:47.029 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:45:47.029 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:45:47.029 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 125419 2024-03-20T09:45:47.029 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.029 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:47.029 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.029 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:47.029 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.676703+0000", 2024-03-20T09:45:47.029 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.029 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.029 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.029 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:47.029 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.676703+0000", 2024-03-20T09:45:47.030 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.030 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.030 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.030 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:47.030 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.676704+0000", 2024-03-20T09:45:47.030 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.657e-06 2024-03-20T09:45:47.030 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.030 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.030 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:47.030 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.676737+0000", 2024-03-20T09:45:47.030 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.2814999999999997e-05 2024-03-20T09:45:47.030 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.030 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.030 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:47.031 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.676758+0000", 2024-03-20T09:45:47.031 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1197000000000001e-05 2024-03-20T09:45:47.031 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.031 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.031 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:47.031 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.676768+0000", 2024-03-20T09:45:47.031 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.9129999999999998e-06 2024-03-20T09:45:47.031 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.031 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.031 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:47.031 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.676833+0000", 2024-03-20T09:45:47.031 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4566999999999994e-05 2024-03-20T09:45:47.031 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.031 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.031 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:47.032 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.676965+0000", 2024-03-20T09:45:47.032 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000132247 2024-03-20T09:45:47.032 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.032 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.032 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,8", 2024-03-20T09:45:47.032 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.677090+0000", 2024-03-20T09:45:47.032 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012488700000000001 2024-03-20T09:45:47.032 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.032 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.032 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.032 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.685984+0000", 2024-03-20T09:45:47.032 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0088943990000000007 2024-03-20T09:45:47.032 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.032 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.032 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:45:47.033 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.690774+0000", 2024-03-20T09:45:47.033 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0047899090000000002 2024-03-20T09:45:47.033 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.033 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.033 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.033 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.892390+0000", 2024-03-20T09:45:47.033 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.20161539000000001 2024-03-20T09:45:47.033 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.033 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.033 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:45:47.033 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.892460+0000", 2024-03-20T09:45:47.033 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.0794e-05 2024-03-20T09:45:47.033 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.033 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.033 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:47.034 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.892507+0000", 2024-03-20T09:45:47.034 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6898999999999999e-05 2024-03-20T09:45:47.034 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.034 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:47.034 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.034 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.034 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.034 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:125421 9.4 9:2366ff65:::benchmark_data_smithi122_210164_object125420:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1097)", 2024-03-20T09:45:47.034 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:41:57.677805+0000", 2024-03-20T09:45:47.034 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 229.36063443099999, 2024-03-20T09:45:47.034 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.21777875999999999, 2024-03-20T09:45:47.034 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:47.034 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:47.034 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:45:47.035 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:45:47.035 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:45:47.035 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 125421 2024-03-20T09:45:47.035 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.035 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:47.035 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.035 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:47.035 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.677805+0000", 2024-03-20T09:45:47.035 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.035 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.035 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.035 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:47.035 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.677805+0000", 2024-03-20T09:45:47.035 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.035 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.036 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.036 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:47.036 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.677806+0000", 2024-03-20T09:45:47.036 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.16e-07 2024-03-20T09:45:47.036 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.036 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.036 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:47.036 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.677863+0000", 2024-03-20T09:45:47.036 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7602000000000002e-05 2024-03-20T09:45:47.036 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.036 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.036 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:47.036 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.677876+0000", 2024-03-20T09:45:47.036 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2359e-05 2024-03-20T09:45:47.037 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.037 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.037 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:47.037 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.677884+0000", 2024-03-20T09:45:47.037 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.0979999999999994e-06 2024-03-20T09:45:47.037 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.037 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.037 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:47.037 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.682250+0000", 2024-03-20T09:45:47.037 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0043660690000000002 2024-03-20T09:45:47.037 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.037 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.038 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:47.038 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.682372+0000", 2024-03-20T09:45:47.038 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000121729 2024-03-20T09:45:47.038 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.038 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.038 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,8", 2024-03-20T09:45:47.038 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.682517+0000", 2024-03-20T09:45:47.038 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00014497800000000001 2024-03-20T09:45:47.038 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.038 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.038 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.038 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.686267+0000", 2024-03-20T09:45:47.039 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.003750813 2024-03-20T09:45:47.039 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.039 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.039 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:45:47.039 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.690814+0000", 2024-03-20T09:45:47.039 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0045468440000000004 2024-03-20T09:45:47.039 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.039 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.039 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.039 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.895498+0000", 2024-03-20T09:45:47.039 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.20468401999999999 2024-03-20T09:45:47.039 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.039 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.039 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:45:47.040 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.895540+0000", 2024-03-20T09:45:47.040 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.1711000000000001e-05 2024-03-20T09:45:47.040 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.040 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.040 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:47.040 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.895584+0000", 2024-03-20T09:45:47.040 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.3621000000000001e-05 2024-03-20T09:45:47.040 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.040 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:47.040 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.040 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.040 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.040 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:125426 9.4 9:2c1ba576:::benchmark_data_smithi122_210164_object125425:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1097)", 2024-03-20T09:45:47.040 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:41:57.687146+0000", 2024-03-20T09:45:47.040 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 229.351293395, 2024-03-20T09:45:47.041 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.20856333399999999, 2024-03-20T09:45:47.041 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:47.041 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:47.041 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:45:47.041 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:45:47.041 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:45:47.041 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 125426 2024-03-20T09:45:47.041 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.041 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:47.041 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.041 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:47.041 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.687146+0000", 2024-03-20T09:45:47.041 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.041 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.041 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.042 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:47.042 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.687146+0000", 2024-03-20T09:45:47.042 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.042 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.042 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.042 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:47.042 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.687147+0000", 2024-03-20T09:45:47.042 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5430000000000001e-06 2024-03-20T09:45:47.042 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.042 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.042 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:47.042 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.687202+0000", 2024-03-20T09:45:47.042 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4953000000000001e-05 2024-03-20T09:45:47.042 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.043 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.043 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:47.043 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.687213+0000", 2024-03-20T09:45:47.043 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0558e-05 2024-03-20T09:45:47.043 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.043 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.043 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:47.043 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.687220+0000", 2024-03-20T09:45:47.043 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.4680000000000003e-06 2024-03-20T09:45:47.043 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.043 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.043 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:47.043 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.687290+0000", 2024-03-20T09:45:47.044 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.9754999999999999e-05 2024-03-20T09:45:47.044 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.044 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.044 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:47.044 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.687413+0000", 2024-03-20T09:45:47.044 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012257300000000001 2024-03-20T09:45:47.044 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.044 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.044 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,8", 2024-03-20T09:45:47.044 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.687562+0000", 2024-03-20T09:45:47.044 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00014886700000000001 2024-03-20T09:45:47.044 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.044 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.044 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:45:47.044 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.690919+0000", 2024-03-20T09:45:47.045 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0033579389999999999 2024-03-20T09:45:47.045 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.045 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.045 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.045 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.699260+0000", 2024-03-20T09:45:47.045 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0083410310000000005 2024-03-20T09:45:47.045 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.045 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.045 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.045 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.895622+0000", 2024-03-20T09:45:47.045 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.19636156299999999 2024-03-20T09:45:47.045 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.045 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.045 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:45:47.045 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.895675+0000", 2024-03-20T09:45:47.046 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3028000000000001e-05 2024-03-20T09:45:47.046 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.046 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.046 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:47.046 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.895709+0000", 2024-03-20T09:45:47.046 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.4056000000000001e-05 2024-03-20T09:45:47.046 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.046 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:47.046 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.046 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.046 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.046 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:125433 9.4 9:2be8f4a2:::benchmark_data_smithi122_210164_object125432:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1097)", 2024-03-20T09:45:47.046 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:41:57.692469+0000", 2024-03-20T09:45:47.046 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 229.34596985300001, 2024-03-20T09:45:47.047 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.21518273299999999, 2024-03-20T09:45:47.047 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:47.047 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:47.047 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:45:47.047 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:45:47.047 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:45:47.047 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 125433 2024-03-20T09:45:47.047 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.047 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:47.047 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.047 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:47.047 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.692469+0000", 2024-03-20T09:45:47.047 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.047 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.047 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.047 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:47.048 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.692469+0000", 2024-03-20T09:45:47.048 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.048 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.048 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.048 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:47.048 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.692471+0000", 2024-03-20T09:45:47.048 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4580000000000001e-06 2024-03-20T09:45:47.048 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.048 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.048 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:47.048 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.692534+0000", 2024-03-20T09:45:47.048 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2818999999999996e-05 2024-03-20T09:45:47.048 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.048 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.049 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:47.049 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.692546+0000", 2024-03-20T09:45:47.049 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2563e-05 2024-03-20T09:45:47.049 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.049 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.049 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:47.049 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.692554+0000", 2024-03-20T09:45:47.049 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.0730000000000005e-06 2024-03-20T09:45:47.049 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.049 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.049 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:47.049 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.692607+0000", 2024-03-20T09:45:47.049 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2757000000000003e-05 2024-03-20T09:45:47.049 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.050 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.050 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:47.050 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.692753+0000", 2024-03-20T09:45:47.050 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000146024 2024-03-20T09:45:47.050 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.050 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.050 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,8", 2024-03-20T09:45:47.050 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.692904+0000", 2024-03-20T09:45:47.050 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00015064000000000001 2024-03-20T09:45:47.050 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.050 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.050 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:45:47.050 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.696986+0000", 2024-03-20T09:45:47.050 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0040820190000000001 2024-03-20T09:45:47.050 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.051 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.051 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.051 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.699291+0000", 2024-03-20T09:45:47.051 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.002305531 2024-03-20T09:45:47.051 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.051 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.051 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.051 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.907535+0000", 2024-03-20T09:45:47.051 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.20824390100000001 2024-03-20T09:45:47.051 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.051 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.051 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:45:47.051 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.907602+0000", 2024-03-20T09:45:47.051 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.6600000000000006e-05 2024-03-20T09:45:47.051 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.051 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.052 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:47.052 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.907652+0000", 2024-03-20T09:45:47.052 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0348e-05 2024-03-20T09:45:47.052 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.052 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:47.052 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.052 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.052 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.052 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:125456 9.4 9:241584fb:::benchmark_data_smithi122_210164_object125455:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1097)", 2024-03-20T09:45:47.052 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:41:57.769260+0000", 2024-03-20T09:45:47.052 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 229.269178923, 2024-03-20T09:45:47.052 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.13851053099999999, 2024-03-20T09:45:47.052 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:47.052 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:47.052 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:45:47.053 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:45:47.053 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:45:47.053 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 125456 2024-03-20T09:45:47.053 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.053 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:47.053 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.053 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:47.053 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.769260+0000", 2024-03-20T09:45:47.053 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.053 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.053 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.053 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:47.053 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.769260+0000", 2024-03-20T09:45:47.053 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.053 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.054 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.054 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:47.054 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.769262+0000", 2024-03-20T09:45:47.054 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2000000000000001e-06 2024-03-20T09:45:47.054 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.054 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.054 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:47.054 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.769343+0000", 2024-03-20T09:45:47.054 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.0247000000000006e-05 2024-03-20T09:45:47.054 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.054 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.054 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:47.054 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.769366+0000", 2024-03-20T09:45:47.055 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3487999999999999e-05 2024-03-20T09:45:47.055 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.055 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.055 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:47.055 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.769377+0000", 2024-03-20T09:45:47.055 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0505e-05 2024-03-20T09:45:47.055 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.055 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.055 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:47.055 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.769455+0000", 2024-03-20T09:45:47.055 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7959000000000004e-05 2024-03-20T09:45:47.055 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.055 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.055 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:47.055 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.769616+0000", 2024-03-20T09:45:47.056 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00016164400000000001 2024-03-20T09:45:47.056 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.056 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.056 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,8", 2024-03-20T09:45:47.056 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.769762+0000", 2024-03-20T09:45:47.056 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00014571199999999999 2024-03-20T09:45:47.056 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.056 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.056 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:45:47.056 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.773105+0000", 2024-03-20T09:45:47.056 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0033432470000000001 2024-03-20T09:45:47.056 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.056 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.056 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.056 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.773778+0000", 2024-03-20T09:45:47.056 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000672505 2024-03-20T09:45:47.057 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.057 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.057 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.057 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.907689+0000", 2024-03-20T09:45:47.057 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.133910798 2024-03-20T09:45:47.057 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.057 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.057 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:45:47.057 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.907739+0000", 2024-03-20T09:45:47.057 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0183999999999999e-05 2024-03-20T09:45:47.057 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.057 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.057 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:47.057 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.907771+0000", 2024-03-20T09:45:47.057 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.2042000000000003e-05 2024-03-20T09:45:47.058 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.058 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:47.058 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.058 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.058 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.058 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:125471 9.4 9:29c17a80:::benchmark_data_smithi122_210164_object125470:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1097)", 2024-03-20T09:45:47.058 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:41:57.788717+0000", 2024-03-20T09:45:47.058 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 229.249722144, 2024-03-20T09:45:47.058 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.11917040399999999, 2024-03-20T09:45:47.058 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:47.058 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:47.058 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:45:47.058 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:45:47.059 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:45:47.059 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 125471 2024-03-20T09:45:47.059 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.059 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:47.059 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.059 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:47.059 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.788717+0000", 2024-03-20T09:45:47.059 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.059 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.059 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.059 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:47.059 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.788717+0000", 2024-03-20T09:45:47.059 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.060 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.060 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.060 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:47.060 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.788719+0000", 2024-03-20T09:45:47.060 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.544e-06 2024-03-20T09:45:47.060 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.060 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.060 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:47.060 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.788776+0000", 2024-03-20T09:45:47.060 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7049000000000002e-05 2024-03-20T09:45:47.060 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.060 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.060 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:47.060 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.788795+0000", 2024-03-20T09:45:47.060 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9755999999999999e-05 2024-03-20T09:45:47.060 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.061 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.061 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:47.061 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.788806+0000", 2024-03-20T09:45:47.061 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0813e-05 2024-03-20T09:45:47.061 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.061 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.061 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:47.061 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.788875+0000", 2024-03-20T09:45:47.061 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.8313999999999997e-05 2024-03-20T09:45:47.061 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.061 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.061 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:47.061 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.789035+0000", 2024-03-20T09:45:47.061 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00016029099999999999 2024-03-20T09:45:47.061 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.061 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.062 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,8", 2024-03-20T09:45:47.062 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.789221+0000", 2024-03-20T09:45:47.062 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000185966 2024-03-20T09:45:47.062 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.062 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.062 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.062 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.796441+0000", 2024-03-20T09:45:47.062 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0072207110000000003 2024-03-20T09:45:47.062 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.062 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.062 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:45:47.062 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.798264+0000", 2024-03-20T09:45:47.062 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001822054 2024-03-20T09:45:47.062 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.063 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.063 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.063 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.907804+0000", 2024-03-20T09:45:47.063 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.10954011399999999 2024-03-20T09:45:47.063 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.063 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.063 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:45:47.063 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.907853+0000", 2024-03-20T09:45:47.063 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9024999999999999e-05 2024-03-20T09:45:47.063 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.063 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.063 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:47.063 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.907887+0000", 2024-03-20T09:45:47.063 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.4767000000000001e-05 2024-03-20T09:45:47.063 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.063 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:47.064 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.064 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.064 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.064 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:125483 9.4 9:2840ba60:::benchmark_data_smithi122_210164_object125482:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1097)", 2024-03-20T09:45:47.064 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:41:57.801786+0000", 2024-03-20T09:45:47.064 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 229.23665289300001, 2024-03-20T09:45:47.064 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.10621507199999999, 2024-03-20T09:45:47.064 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:47.064 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:47.064 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:45:47.064 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:45:47.064 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:45:47.064 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 125483 2024-03-20T09:45:47.064 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.065 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:47.065 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.065 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:47.065 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.801786+0000", 2024-03-20T09:45:47.065 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.065 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.065 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.065 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:47.065 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.801786+0000", 2024-03-20T09:45:47.065 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.065 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.065 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.065 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:47.065 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.801788+0000", 2024-03-20T09:45:47.065 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.767e-06 2024-03-20T09:45:47.066 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.066 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.066 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:47.066 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.801862+0000", 2024-03-20T09:45:47.066 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.4263000000000004e-05 2024-03-20T09:45:47.066 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.066 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.066 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:47.066 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.801879+0000", 2024-03-20T09:45:47.066 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.666e-05 2024-03-20T09:45:47.066 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.066 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.066 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:47.066 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.801888+0000", 2024-03-20T09:45:47.066 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.3759999999999997e-06 2024-03-20T09:45:47.067 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.067 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.067 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:47.067 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.801956+0000", 2024-03-20T09:45:47.067 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.7379999999999995e-05 2024-03-20T09:45:47.067 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.067 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.067 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:47.067 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.802117+0000", 2024-03-20T09:45:47.067 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000161204 2024-03-20T09:45:47.067 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.067 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.067 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,8", 2024-03-20T09:45:47.067 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.802284+0000", 2024-03-20T09:45:47.068 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000167524 2024-03-20T09:45:47.068 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.068 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.068 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:45:47.068 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.809719+0000", 2024-03-20T09:45:47.068 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0074341549999999996 2024-03-20T09:45:47.068 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.068 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.068 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.068 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.810356+0000", 2024-03-20T09:45:47.068 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00063742599999999996 2024-03-20T09:45:47.068 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.068 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.068 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.069 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.907919+0000", 2024-03-20T09:45:47.069 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.097562466 2024-03-20T09:45:47.069 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.069 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.069 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:45:47.069 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.907967+0000", 2024-03-20T09:45:47.069 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8194e-05 2024-03-20T09:45:47.069 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.069 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.069 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:47.069 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:57.908001+0000", 2024-03-20T09:45:47.069 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.4656999999999997e-05 2024-03-20T09:45:47.069 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.070 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:47.070 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.070 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.070 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.070 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:163499 9.4 9:27d66fe0:::benchmark_data_smithi122_210164_object31269:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1107)", 2024-03-20T09:45:47.070 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:42:43.967670+0000", 2024-03-20T09:45:47.070 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 183.07076880299999, 2024-03-20T09:45:47.070 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.049994236999999997, 2024-03-20T09:45:47.070 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:47.070 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:47.070 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:45:47.070 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:45:47.070 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:45:47.070 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 163499 2024-03-20T09:45:47.071 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.071 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:47.071 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.071 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:47.071 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:42:43.967670+0000", 2024-03-20T09:45:47.071 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.071 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.071 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.071 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:47.071 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:42:43.967670+0000", 2024-03-20T09:45:47.071 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.071 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.071 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.071 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:47.071 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:42:43.967671+0000", 2024-03-20T09:45:47.072 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.0599999999999999e-07 2024-03-20T09:45:47.072 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.072 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.072 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:47.072 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:42:43.967675+0000", 2024-03-20T09:45:47.072 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.2880000000000003e-06 2024-03-20T09:45:47.072 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.072 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.072 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:47.072 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:42:43.980026+0000", 2024-03-20T09:45:47.072 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.01235032 2024-03-20T09:45:47.072 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.072 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.072 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:47.073 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:42:43.980043+0000", 2024-03-20T09:45:47.073 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7357999999999999e-05 2024-03-20T09:45:47.073 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.073 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.073 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:47.073 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:42:43.993011+0000", 2024-03-20T09:45:47.073 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.012967741 2024-03-20T09:45:47.073 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.073 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.073 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:47.073 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:42:43.993151+0000", 2024-03-20T09:45:47.073 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00014035 2024-03-20T09:45:47.073 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.073 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.073 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,8", 2024-03-20T09:45:47.074 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:42:43.993267+0000", 2024-03-20T09:45:47.074 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000115905 2024-03-20T09:45:47.074 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.074 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.074 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.074 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:42:44.006935+0000", 2024-03-20T09:45:47.074 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.013667540000000001 2024-03-20T09:45:47.074 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.074 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.074 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.074 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:42:44.007015+0000", 2024-03-20T09:45:47.074 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.0647000000000002e-05 2024-03-20T09:45:47.074 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.074 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.075 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:45:47.075 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:42:44.017570+0000", 2024-03-20T09:45:47.075 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.010554732000000001 2024-03-20T09:45:47.075 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.075 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.075 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:45:47.075 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:42:44.017621+0000", 2024-03-20T09:45:47.075 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1345999999999999e-05 2024-03-20T09:45:47.075 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.075 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.075 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:47.075 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:42:44.017665+0000", 2024-03-20T09:45:47.075 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.3204000000000002e-05 2024-03-20T09:45:47.075 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.075 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:47.076 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.076 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.076 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.076 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:231964 9.4 9:20fdf502:::benchmark_data_smithi122_210164_object99734:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1135)", 2024-03-20T09:45:47.076 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:44:03.591759+0000", 2024-03-20T09:45:47.076 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 103.446680019, 2024-03-20T09:45:47.076 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.039499236, 2024-03-20T09:45:47.076 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:47.076 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:47.076 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:45:47.076 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:45:47.076 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:45:47.076 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 231964 2024-03-20T09:45:47.076 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.076 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:47.076 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.077 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:47.077 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.591759+0000", 2024-03-20T09:45:47.077 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.077 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.077 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.077 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:47.077 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.591759+0000", 2024-03-20T09:45:47.077 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.077 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.077 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.077 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:47.077 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.591760+0000", 2024-03-20T09:45:47.077 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2810000000000001e-06 2024-03-20T09:45:47.077 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.077 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.077 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:47.078 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.591766+0000", 2024-03-20T09:45:47.078 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.468e-06 2024-03-20T09:45:47.078 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.078 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.078 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:47.078 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.591779+0000", 2024-03-20T09:45:47.078 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3414000000000001e-05 2024-03-20T09:45:47.078 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.078 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.078 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:47.078 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.591787+0000", 2024-03-20T09:45:47.078 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.5660000000000001e-06 2024-03-20T09:45:47.078 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.079 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.079 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:47.079 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.595363+0000", 2024-03-20T09:45:47.079 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0035764410000000001 2024-03-20T09:45:47.079 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.079 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.079 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:47.079 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.595501+0000", 2024-03-20T09:45:47.079 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013813599999999999 2024-03-20T09:45:47.079 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.079 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.079 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,8", 2024-03-20T09:45:47.079 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.595623+0000", 2024-03-20T09:45:47.080 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000121395 2024-03-20T09:45:47.080 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.080 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.080 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:45:47.080 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.606310+0000", 2024-03-20T09:45:47.080 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.010687448 2024-03-20T09:45:47.080 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.080 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.080 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.080 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.631118+0000", 2024-03-20T09:45:47.080 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.024807661000000002 2024-03-20T09:45:47.080 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.080 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.080 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.080 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.631188+0000", 2024-03-20T09:45:47.081 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.0258000000000004e-05 2024-03-20T09:45:47.081 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.081 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.081 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:45:47.081 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.631228+0000", 2024-03-20T09:45:47.081 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.9607000000000002e-05 2024-03-20T09:45:47.081 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.081 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.081 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:47.081 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.631258+0000", 2024-03-20T09:45:47.081 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.0561000000000001e-05 2024-03-20T09:45:47.081 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.081 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:47.081 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.081 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.082 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.082 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:231965 9.4 9:28d47dbe:::benchmark_data_smithi122_210164_object99735:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1135)", 2024-03-20T09:45:47.082 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:44:03.595223+0000", 2024-03-20T09:45:47.082 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 103.443215761, 2024-03-20T09:45:47.082 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.040842769000000001, 2024-03-20T09:45:47.082 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:47.082 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:47.082 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:45:47.082 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:45:47.082 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:45:47.082 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 231965 2024-03-20T09:45:47.082 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.082 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:47.082 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.083 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:47.083 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.595223+0000", 2024-03-20T09:45:47.083 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.083 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.083 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.083 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:47.083 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.595223+0000", 2024-03-20T09:45:47.083 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.083 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.083 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.083 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:47.083 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.595225+0000", 2024-03-20T09:45:47.083 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.285e-06 2024-03-20T09:45:47.083 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.084 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.084 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:47.084 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.595231+0000", 2024-03-20T09:45:47.084 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9309999999999996e-06 2024-03-20T09:45:47.084 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.084 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.084 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:47.084 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.595250+0000", 2024-03-20T09:45:47.084 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.944e-05 2024-03-20T09:45:47.084 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.084 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.084 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:47.084 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.595260+0000", 2024-03-20T09:45:47.084 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.696e-06 2024-03-20T09:45:47.084 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.084 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.085 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:47.085 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.600251+0000", 2024-03-20T09:45:47.085 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0049914479999999999 2024-03-20T09:45:47.085 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.085 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.085 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:47.085 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.600383+0000", 2024-03-20T09:45:47.085 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000131738 2024-03-20T09:45:47.085 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.085 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.085 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,8", 2024-03-20T09:45:47.085 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.600506+0000", 2024-03-20T09:45:47.085 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000123454 2024-03-20T09:45:47.085 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.085 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.086 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.086 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.631141+0000", 2024-03-20T09:45:47.086 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.030634847999999999 2024-03-20T09:45:47.086 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.086 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.086 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.086 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.631282+0000", 2024-03-20T09:45:47.086 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00014084699999999999 2024-03-20T09:45:47.086 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.086 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.086 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:45:47.086 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.635961+0000", 2024-03-20T09:45:47.086 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0046788309999999996 2024-03-20T09:45:47.086 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.087 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.087 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:45:47.087 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.636017+0000", 2024-03-20T09:45:47.087 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5946999999999998e-05 2024-03-20T09:45:47.087 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.087 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.087 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:47.087 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.636066+0000", 2024-03-20T09:45:47.087 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9304000000000001e-05 2024-03-20T09:45:47.087 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.087 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:47.087 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.087 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.087 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.087 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:231968 9.4 9:2ac0d0aa:::benchmark_data_smithi122_210164_object99738:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1135)", 2024-03-20T09:45:47.087 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:44:03.597502+0000", 2024-03-20T09:45:47.088 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 103.440937154, 2024-03-20T09:45:47.088 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.038649891999999998, 2024-03-20T09:45:47.088 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:47.088 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:47.088 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:45:47.088 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:45:47.088 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:45:47.088 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 231968 2024-03-20T09:45:47.088 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.088 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:47.088 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.088 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:47.088 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.597502+0000", 2024-03-20T09:45:47.088 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.089 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.089 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.089 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:47.089 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.597502+0000", 2024-03-20T09:45:47.089 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.089 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.089 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.089 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:47.089 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.597504+0000", 2024-03-20T09:45:47.089 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5489999999999999e-06 2024-03-20T09:45:47.089 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.089 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.089 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:47.090 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.597511+0000", 2024-03-20T09:45:47.090 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.0659999999999998e-06 2024-03-20T09:45:47.090 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.090 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.090 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:47.090 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.597525+0000", 2024-03-20T09:45:47.090 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.468e-05 2024-03-20T09:45:47.090 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.090 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.090 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:47.090 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.597535+0000", 2024-03-20T09:45:47.090 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.013e-05 2024-03-20T09:45:47.090 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.090 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.091 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:47.091 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.606341+0000", 2024-03-20T09:45:47.091 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0088055830000000005 2024-03-20T09:45:47.091 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.091 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.091 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:47.091 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.606452+0000", 2024-03-20T09:45:47.091 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00011058899999999999 2024-03-20T09:45:47.091 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.091 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.091 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,8", 2024-03-20T09:45:47.091 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.606562+0000", 2024-03-20T09:45:47.091 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000110049 2024-03-20T09:45:47.091 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.091 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.091 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.092 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.631162+0000", 2024-03-20T09:45:47.092 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.02460072 2024-03-20T09:45:47.092 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.092 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.092 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.092 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.631307+0000", 2024-03-20T09:45:47.092 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000144361 2024-03-20T09:45:47.092 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.092 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.092 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:45:47.092 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.636074+0000", 2024-03-20T09:45:47.092 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0047674960000000004 2024-03-20T09:45:47.092 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.092 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.092 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:45:47.093 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.636116+0000", 2024-03-20T09:45:47.093 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.1337000000000001e-05 2024-03-20T09:45:47.093 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.093 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.093 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:47.093 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.636152+0000", 2024-03-20T09:45:47.093 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6331999999999997e-05 2024-03-20T09:45:47.093 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.093 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:47.093 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.093 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.093 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.093 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:231975 9.4 9:27cf25eb:::benchmark_data_smithi122_210164_object99745:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1135)", 2024-03-20T09:45:47.093 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:44:03.610463+0000", 2024-03-20T09:45:47.094 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 103.42797590399999, 2024-03-20T09:45:47.094 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.034328714000000003, 2024-03-20T09:45:47.094 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:47.094 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:47.094 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:45:47.094 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:45:47.094 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:45:47.094 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 231975 2024-03-20T09:45:47.094 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.094 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:47.094 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.094 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:47.094 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.610463+0000", 2024-03-20T09:45:47.094 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.094 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.095 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.095 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:47.095 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.610463+0000", 2024-03-20T09:45:47.095 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.095 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.095 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.095 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:47.095 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.610465+0000", 2024-03-20T09:45:47.095 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4720000000000001e-06 2024-03-20T09:45:47.095 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.095 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.095 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:47.095 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.610472+0000", 2024-03-20T09:45:47.095 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.6869999999999994e-06 2024-03-20T09:45:47.095 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.096 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.096 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:47.096 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.610491+0000", 2024-03-20T09:45:47.096 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8539999999999999e-05 2024-03-20T09:45:47.096 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.096 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.096 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:47.096 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.610501+0000", 2024-03-20T09:45:47.096 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.9019999999999994e-06 2024-03-20T09:45:47.096 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.096 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.096 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:47.096 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.611487+0000", 2024-03-20T09:45:47.096 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0009864120000000001 2024-03-20T09:45:47.096 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.096 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.097 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:47.097 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.611606+0000", 2024-03-20T09:45:47.097 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000118418 2024-03-20T09:45:47.097 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.097 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.097 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,8", 2024-03-20T09:45:47.097 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.611721+0000", 2024-03-20T09:45:47.097 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000115158 2024-03-20T09:45:47.097 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.097 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.097 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:45:47.097 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.636159+0000", 2024-03-20T09:45:47.097 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.024437699 2024-03-20T09:45:47.097 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.098 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.098 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.098 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.641069+0000", 2024-03-20T09:45:47.098 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0049104159999999999 2024-03-20T09:45:47.098 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.098 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.098 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.098 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.644683+0000", 2024-03-20T09:45:47.098 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0036141799999999998 2024-03-20T09:45:47.098 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.098 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.098 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:45:47.098 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.644743+0000", 2024-03-20T09:45:47.099 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9568999999999998e-05 2024-03-20T09:45:47.099 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.099 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.099 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:47.099 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.644792+0000", 2024-03-20T09:45:47.099 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9261000000000002e-05 2024-03-20T09:45:47.099 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.099 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:47.099 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.099 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.099 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.099 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:231977 9.4 9:288de145:::benchmark_data_smithi122_210164_object99747:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1135)", 2024-03-20T09:45:47.099 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:44:03.610526+0000", 2024-03-20T09:45:47.100 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 103.427913377, 2024-03-20T09:45:47.100 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.035179963000000002, 2024-03-20T09:45:47.100 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:47.100 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:47.100 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:45:47.100 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:45:47.100 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:45:47.100 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 231977 2024-03-20T09:45:47.100 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.100 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:47.100 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.100 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:47.100 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.610526+0000", 2024-03-20T09:45:47.100 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.100 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.101 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.101 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:47.101 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.610526+0000", 2024-03-20T09:45:47.101 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.101 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.101 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.101 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:47.101 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.610526+0000", 2024-03-20T09:45:47.101 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9599999999999999e-07 2024-03-20T09:45:47.101 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.101 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.101 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:47.101 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.610529+0000", 2024-03-20T09:45:47.101 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.038e-06 2024-03-20T09:45:47.101 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.102 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.102 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:47.102 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.610536+0000", 2024-03-20T09:45:47.102 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4180000000000002e-06 2024-03-20T09:45:47.102 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.102 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.102 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:47.102 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.610541+0000", 2024-03-20T09:45:47.102 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1619999999999999e-06 2024-03-20T09:45:47.102 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.102 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.102 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:47.102 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.616675+0000", 2024-03-20T09:45:47.102 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0061337650000000002 2024-03-20T09:45:47.103 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.103 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.103 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:47.103 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.616804+0000", 2024-03-20T09:45:47.103 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000129782 2024-03-20T09:45:47.103 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.103 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.103 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,8", 2024-03-20T09:45:47.103 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.616939+0000", 2024-03-20T09:45:47.103 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013425700000000001 2024-03-20T09:45:47.103 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.103 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.103 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.103 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.641091+0000", 2024-03-20T09:45:47.103 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.024152742000000001 2024-03-20T09:45:47.104 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.104 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.104 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:45:47.104 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.644638+0000", 2024-03-20T09:45:47.104 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0035468309999999999 2024-03-20T09:45:47.104 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.104 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.104 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.104 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.645646+0000", 2024-03-20T09:45:47.104 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001008019 2024-03-20T09:45:47.104 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.104 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.104 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:45:47.105 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.645679+0000", 2024-03-20T09:45:47.105 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.2716000000000003e-05 2024-03-20T09:45:47.105 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.105 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.105 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:47.105 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:03.645706+0000", 2024-03-20T09:45:47.105 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6737000000000002e-05 2024-03-20T09:45:47.105 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.105 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:47.105 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.105 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.105 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.106 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:233182 9.4 9:20fecdae:::benchmark_data_smithi122_210164_object100952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1135)", 2024-03-20T09:45:47.106 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:44:04.926249+0000", 2024-03-20T09:45:47.106 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 102.11219065500001, 2024-03-20T09:45:47.106 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.033207607, 2024-03-20T09:45:47.106 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:47.106 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:47.106 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:45:47.106 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:45:47.106 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:45:47.106 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 233182 2024-03-20T09:45:47.106 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.106 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:47.106 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.106 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:47.107 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:04.926249+0000", 2024-03-20T09:45:47.107 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.107 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.107 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.107 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:47.107 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:04.926249+0000", 2024-03-20T09:45:47.107 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.107 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.107 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.107 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:47.107 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:04.926249+0000", 2024-03-20T09:45:47.107 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9799999999999999e-07 2024-03-20T09:45:47.107 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.107 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.108 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:47.108 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:04.926251+0000", 2024-03-20T09:45:47.108 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.063e-06 2024-03-20T09:45:47.108 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.108 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.108 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:47.108 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:04.926257+0000", 2024-03-20T09:45:47.108 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7389999999999996e-06 2024-03-20T09:45:47.108 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.108 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.108 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:47.108 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:04.926260+0000", 2024-03-20T09:45:47.108 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6789999999999998e-06 2024-03-20T09:45:47.108 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.108 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.108 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:47.109 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:04.938363+0000", 2024-03-20T09:45:47.109 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.012102913 2024-03-20T09:45:47.109 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.109 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.109 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:47.109 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:04.938467+0000", 2024-03-20T09:45:47.109 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000103742 2024-03-20T09:45:47.109 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.109 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.109 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,8", 2024-03-20T09:45:47.109 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:04.938572+0000", 2024-03-20T09:45:47.109 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000105165 2024-03-20T09:45:47.109 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.109 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.110 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:45:47.110 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:04.943209+0000", 2024-03-20T09:45:47.110 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0046367179999999997 2024-03-20T09:45:47.110 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.110 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.110 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.110 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:04.948031+0000", 2024-03-20T09:45:47.110 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0048221289999999997 2024-03-20T09:45:47.110 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.110 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.110 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.110 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:04.959387+0000", 2024-03-20T09:45:47.110 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.011355848 2024-03-20T09:45:47.110 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.111 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.111 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:45:47.111 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:04.959427+0000", 2024-03-20T09:45:47.111 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.0197000000000002e-05 2024-03-20T09:45:47.111 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.111 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.111 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:47.111 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:04.959456+0000", 2024-03-20T09:45:47.111 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9116e-05 2024-03-20T09:45:47.111 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.111 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:47.111 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.111 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.111 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.111 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:235153 9.4 9:2da0e0e4:::benchmark_data_smithi122_210164_object102923:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1136)", 2024-03-20T09:45:47.111 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:44:07.168026+0000", 2024-03-20T09:45:47.112 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 99.870412723000001, 2024-03-20T09:45:47.112 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.035344188999999998, 2024-03-20T09:45:47.112 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:47.112 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:47.112 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:45:47.112 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:45:47.112 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:45:47.112 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 235153 2024-03-20T09:45:47.112 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.112 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:47.112 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.112 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:47.112 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:07.168026+0000", 2024-03-20T09:45:47.112 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.113 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.113 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.113 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:47.113 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:07.168026+0000", 2024-03-20T09:45:47.113 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.113 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.113 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.113 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:47.113 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:07.168027+0000", 2024-03-20T09:45:47.113 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.1899999999999998e-07 2024-03-20T09:45:47.113 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.113 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.113 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:47.113 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:07.168029+0000", 2024-03-20T09:45:47.113 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5380000000000001e-06 2024-03-20T09:45:47.113 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.114 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.114 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:47.114 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:07.168036+0000", 2024-03-20T09:45:47.114 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2899999999999999e-06 2024-03-20T09:45:47.114 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.114 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.114 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:47.114 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:07.168041+0000", 2024-03-20T09:45:47.114 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.933e-06 2024-03-20T09:45:47.114 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.114 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.114 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:47.115 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:07.176260+0000", 2024-03-20T09:45:47.115 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0082193049999999997 2024-03-20T09:45:47.115 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.115 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.115 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:47.115 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:07.176406+0000", 2024-03-20T09:45:47.115 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00014584299999999999 2024-03-20T09:45:47.115 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.115 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.115 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,8", 2024-03-20T09:45:47.115 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:07.176542+0000", 2024-03-20T09:45:47.115 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013637999999999999 2024-03-20T09:45:47.115 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.115 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.115 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:45:47.116 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:07.186214+0000", 2024-03-20T09:45:47.116 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0096723060000000003 2024-03-20T09:45:47.116 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.116 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.116 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.116 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:07.186541+0000", 2024-03-20T09:45:47.116 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00032632599999999999 2024-03-20T09:45:47.116 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.116 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.116 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.116 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:07.203267+0000", 2024-03-20T09:45:47.116 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.016725838999999999 2024-03-20T09:45:47.116 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.116 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.116 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:45:47.116 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:07.203324+0000", 2024-03-20T09:45:47.117 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7179000000000003e-05 2024-03-20T09:45:47.117 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.117 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.117 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:47.117 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:07.203371+0000", 2024-03-20T09:45:47.117 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6830999999999999e-05 2024-03-20T09:45:47.117 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.117 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:47.117 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.117 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.117 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.117 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:245898 9.4 9:2c4775d5:::benchmark_data_smithi122_210164_object113668:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1139)", 2024-03-20T09:45:47.117 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:44:19.427080+0000", 2024-03-20T09:45:47.117 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 87.611359522000001, 2024-03-20T09:45:47.118 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.034324492999999998, 2024-03-20T09:45:47.118 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:47.118 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:47.118 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:45:47.118 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:45:47.118 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:45:47.118 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 245898 2024-03-20T09:45:47.118 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.118 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:47.118 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.118 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:47.118 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:19.427080+0000", 2024-03-20T09:45:47.118 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.118 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.119 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.119 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:47.119 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:19.427080+0000", 2024-03-20T09:45:47.119 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.119 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.119 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.119 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:47.119 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:19.427082+0000", 2024-03-20T09:45:47.119 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9199999999999998e-06 2024-03-20T09:45:47.119 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.119 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.119 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:47.119 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:19.427088+0000", 2024-03-20T09:45:47.119 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.7120000000000003e-06 2024-03-20T09:45:47.120 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.120 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.120 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:47.120 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:19.427102+0000", 2024-03-20T09:45:47.120 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.417e-05 2024-03-20T09:45:47.120 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.120 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.120 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:47.120 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:19.427111+0000", 2024-03-20T09:45:47.120 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.3599999999999996e-06 2024-03-20T09:45:47.120 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.120 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.120 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:47.120 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:19.436329+0000", 2024-03-20T09:45:47.121 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0092184339999999993 2024-03-20T09:45:47.121 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.121 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.121 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:47.121 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:19.436464+0000", 2024-03-20T09:45:47.121 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013464 2024-03-20T09:45:47.121 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.121 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.121 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,8", 2024-03-20T09:45:47.121 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:19.436590+0000", 2024-03-20T09:45:47.121 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012641599999999999 2024-03-20T09:45:47.121 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.121 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.121 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:45:47.122 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:19.458012+0000", 2024-03-20T09:45:47.122 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.021421997000000002 2024-03-20T09:45:47.122 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.122 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.122 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.122 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:19.461248+0000", 2024-03-20T09:45:47.122 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0032358999999999999 2024-03-20T09:45:47.122 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.122 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.122 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.122 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:19.461289+0000", 2024-03-20T09:45:47.122 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.0874999999999997e-05 2024-03-20T09:45:47.122 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.122 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.123 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:45:47.123 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:19.461352+0000", 2024-03-20T09:45:47.123 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2991999999999996e-05 2024-03-20T09:45:47.123 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.123 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.123 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:47.123 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:19.461404+0000", 2024-03-20T09:45:47.123 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2077000000000003e-05 2024-03-20T09:45:47.123 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.123 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:47.123 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.123 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.123 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.123 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:245900 9.4 9:2ce7d288:::benchmark_data_smithi122_210164_object113670:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1139)", 2024-03-20T09:45:47.123 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:44:19.434919+0000", 2024-03-20T09:45:47.124 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 87.603520662999998, 2024-03-20T09:45:47.124 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.032889950000000001, 2024-03-20T09:45:47.124 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:47.124 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:47.124 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:45:47.124 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:45:47.124 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:45:47.124 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 245900 2024-03-20T09:45:47.124 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.124 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:47.124 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.124 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:47.124 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:19.434919+0000", 2024-03-20T09:45:47.124 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.124 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.125 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.125 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:47.125 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:19.434919+0000", 2024-03-20T09:45:47.125 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.125 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.125 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.125 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:47.125 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:19.434921+0000", 2024-03-20T09:45:47.125 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0040000000000002e-06 2024-03-20T09:45:47.125 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.125 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.125 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:47.125 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:19.434928+0000", 2024-03-20T09:45:47.125 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.0990000000000001e-06 2024-03-20T09:45:47.125 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.125 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.126 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:47.126 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:19.434948+0000", 2024-03-20T09:45:47.126 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9995000000000001e-05 2024-03-20T09:45:47.126 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.126 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.126 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:47.126 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:19.434958+0000", 2024-03-20T09:45:47.126 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0229e-05 2024-03-20T09:45:47.126 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.126 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.126 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:47.126 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:19.442018+0000", 2024-03-20T09:45:47.127 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0070598680000000004 2024-03-20T09:45:47.127 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.127 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.127 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:47.127 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:19.442130+0000", 2024-03-20T09:45:47.127 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000112716 2024-03-20T09:45:47.127 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.127 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.127 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,8", 2024-03-20T09:45:47.127 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:19.442254+0000", 2024-03-20T09:45:47.127 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012316399999999999 2024-03-20T09:45:47.127 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.127 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.128 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:45:47.128 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:19.458018+0000", 2024-03-20T09:45:47.128 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.015764740999999999 2024-03-20T09:45:47.128 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.128 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.128 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.128 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:19.467686+0000", 2024-03-20T09:45:47.128 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0096679469999999997 2024-03-20T09:45:47.128 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.128 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.128 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.128 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:19.467730+0000", 2024-03-20T09:45:47.128 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.3284000000000001e-05 2024-03-20T09:45:47.129 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.129 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.129 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:45:47.129 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:19.467768+0000", 2024-03-20T09:45:47.129 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.8334999999999998e-05 2024-03-20T09:45:47.129 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.129 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.129 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:47.129 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:19.467808+0000", 2024-03-20T09:45:47.129 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.0568000000000002e-05 2024-03-20T09:45:47.129 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.129 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:47.129 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.129 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.129 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.129 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:248986 9.4 9:2a274423:::benchmark_data_smithi122_210164_object116756:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1139)", 2024-03-20T09:45:47.130 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:44:22.922197+0000", 2024-03-20T09:45:47.130 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 84.116242537999995, 2024-03-20T09:45:47.130 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.033519676999999998, 2024-03-20T09:45:47.130 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:47.130 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:47.130 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:45:47.130 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:45:47.130 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:45:47.130 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 248986 2024-03-20T09:45:47.130 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.130 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:47.130 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.130 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:47.130 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:22.922197+0000", 2024-03-20T09:45:47.131 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.131 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.131 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.131 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:47.131 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:22.922197+0000", 2024-03-20T09:45:47.131 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.131 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.131 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.131 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:47.131 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:22.922199+0000", 2024-03-20T09:45:47.131 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.857e-06 2024-03-20T09:45:47.131 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.131 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.131 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:47.131 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:22.922207+0000", 2024-03-20T09:45:47.132 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.1389999999999995e-06 2024-03-20T09:45:47.132 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.132 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.132 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:47.132 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:22.922228+0000", 2024-03-20T09:45:47.132 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0959e-05 2024-03-20T09:45:47.132 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.132 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.132 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:47.132 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:22.922238+0000", 2024-03-20T09:45:47.132 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0471e-05 2024-03-20T09:45:47.132 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.132 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.132 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:47.132 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:22.922305+0000", 2024-03-20T09:45:47.132 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.7041999999999999e-05 2024-03-20T09:45:47.133 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.133 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.133 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:47.133 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:22.922401+0000", 2024-03-20T09:45:47.133 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.5568000000000004e-05 2024-03-20T09:45:47.133 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.133 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.133 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,8", 2024-03-20T09:45:47.133 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:22.922480+0000", 2024-03-20T09:45:47.133 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.9219000000000002e-05 2024-03-20T09:45:47.133 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.133 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.133 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:45:47.134 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:22.926801+0000", 2024-03-20T09:45:47.134 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0043207649999999998 2024-03-20T09:45:47.134 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.134 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.134 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.134 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:22.930082+0000", 2024-03-20T09:45:47.134 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0032815370000000002 2024-03-20T09:45:47.134 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.134 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.134 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.134 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:22.955575+0000", 2024-03-20T09:45:47.134 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.025493078999999998 2024-03-20T09:45:47.134 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.134 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.135 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:45:47.135 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:22.955660+0000", 2024-03-20T09:45:47.135 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.4367e-05 2024-03-20T09:45:47.135 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.135 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.135 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:47.135 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:22.955716+0000", 2024-03-20T09:45:47.135 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6674e-05 2024-03-20T09:45:47.135 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.135 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:47.135 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.135 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.135 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.135 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:249291 9.4 9:2f5d22c8:::benchmark_data_smithi122_210164_object117061:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1139)", 2024-03-20T09:45:47.136 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:44:23.260690+0000", 2024-03-20T09:45:47.136 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 83.777749505000003, 2024-03-20T09:45:47.136 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.040136613000000002, 2024-03-20T09:45:47.136 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:47.136 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:47.136 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:45:47.136 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:45:47.136 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:45:47.136 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 249291 2024-03-20T09:45:47.136 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.136 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:47.136 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.136 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:47.136 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:23.260690+0000", 2024-03-20T09:45:47.137 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.137 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.137 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.137 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:47.137 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:23.260690+0000", 2024-03-20T09:45:47.137 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.137 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.137 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.137 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:47.137 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:23.260691+0000", 2024-03-20T09:45:47.137 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3430000000000001e-06 2024-03-20T09:45:47.137 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.138 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.138 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:47.138 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:23.260698+0000", 2024-03-20T09:45:47.138 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.9829999999999999e-06 2024-03-20T09:45:47.138 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.138 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.138 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:47.138 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:23.260714+0000", 2024-03-20T09:45:47.138 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6365999999999999e-05 2024-03-20T09:45:47.138 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.138 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.138 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:47.138 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:23.260724+0000", 2024-03-20T09:45:47.139 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.9320000000000008e-06 2024-03-20T09:45:47.139 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.139 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.139 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:47.139 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:23.260798+0000", 2024-03-20T09:45:47.139 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3755000000000001e-05 2024-03-20T09:45:47.139 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.139 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.139 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:47.139 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:23.260921+0000", 2024-03-20T09:45:47.139 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000122977 2024-03-20T09:45:47.139 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.139 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.139 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,8", 2024-03-20T09:45:47.140 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:23.261038+0000", 2024-03-20T09:45:47.140 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00011734 2024-03-20T09:45:47.140 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.140 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.140 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.140 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:23.265906+0000", 2024-03-20T09:45:47.140 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0048677479999999999 2024-03-20T09:45:47.140 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.140 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.140 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:45:47.140 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:23.266005+0000", 2024-03-20T09:45:47.140 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.8882000000000006e-05 2024-03-20T09:45:47.140 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.140 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.140 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:45:47.141 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:23.300741+0000", 2024-03-20T09:45:47.141 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.034735986000000003 2024-03-20T09:45:47.141 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.141 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.141 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:45:47.141 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:23.300778+0000", 2024-03-20T09:45:47.141 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6652000000000001e-05 2024-03-20T09:45:47.141 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.141 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.141 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:47.141 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:23.300826+0000", 2024-03-20T09:45:47.141 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8649000000000001e-05 2024-03-20T09:45:47.141 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.141 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:47.141 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.142 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.142 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.142 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.27190.0:249946 9.c e1140/1066 9:36426876:::benchmark_data_smithi122_210164_object117716:head v 1140'15289, mlcod=1140'15285)", 2024-03-20T09:45:47.142 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:44:24.012670+0000", 2024-03-20T09:45:47.142 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 83.025769642, 2024-03-20T09:45:47.142 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.039987058999999998, 2024-03-20T09:45:47.142 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:47.142 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:45:47.142 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:47.142 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.142 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:47.142 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:24.012670+0000", 2024-03-20T09:45:47.142 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.142 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.142 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.143 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:47.143 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:24.012670+0000", 2024-03-20T09:45:47.143 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:47.143 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.143 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.143 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:47.143 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:24.012670+0000", 2024-03-20T09:45:47.143 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.0199999999999998e-07 2024-03-20T09:45:47.143 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.143 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.143 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:47.143 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:24.012674+0000", 2024-03-20T09:45:47.143 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.1649999999999999e-06 2024-03-20T09:45:47.143 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.143 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.144 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:47.144 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:24.012679+0000", 2024-03-20T09:45:47.144 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.096e-06 2024-03-20T09:45:47.144 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.144 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.144 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:47.144 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:24.012683+0000", 2024-03-20T09:45:47.144 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6940000000000001e-06 2024-03-20T09:45:47.144 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.144 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.144 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:47.144 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:24.022036+0000", 2024-03-20T09:45:47.144 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0093530249999999992 2024-03-20T09:45:47.144 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.144 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.144 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:47.145 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:24.022052+0000", 2024-03-20T09:45:47.145 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6467e-05 2024-03-20T09:45:47.145 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.145 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.145 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:45:47.145 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:24.052618+0000", 2024-03-20T09:45:47.145 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.030565579999999998 2024-03-20T09:45:47.145 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:47.145 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:47.145 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:47.145 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:44:24.052657+0000", 2024-03-20T09:45:47.145 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.8729999999999997e-05 2024-03-20T09:45:47.145 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.146 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:47.146 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.146 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:47.146 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:47.146 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:45:47.712 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-20T09:45:47.979 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:45:48.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:47 smithi115 ceph-mon[72476]: pgmap v2843: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:48.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:47 smithi122 ceph-mon[73495]: pgmap v2843: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:48.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:47 smithi026 ceph-mon[68309]: pgmap v2843: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:48.835 INFO:teuthology.orchestra.run.smithi026.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-20T09:45:49.115 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:48 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2491157109' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-20T09:45:49.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:48 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2491157109' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-20T09:45:49.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:48 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2491157109' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-20T09:45:49.675 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:45:49.675 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:45:49.675 INFO:teuthology.orchestra.run.smithi115.stdout: "num_ops": 0 2024-03-20T09:45:49.676 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:45:49.985 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 4 1 2024-03-20T09:45:49.999 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:49 smithi115 ceph-mon[72476]: pgmap v2844: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:49.999 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:49 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2491157109' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-20T09:45:49.999 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:49 smithi115 ceph-mon[72476]: osdmap e1169: 12 total, 12 up, 12 in 2024-03-20T09:45:50.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:49 smithi122 ceph-mon[73495]: pgmap v2844: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:50.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:49 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2491157109' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-20T09:45:50.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:49 smithi122 ceph-mon[73495]: osdmap e1169: 12 total, 12 up, 12 in 2024-03-20T09:45:50.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:49 smithi026 ceph-mon[68309]: pgmap v2844: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:50.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:49 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2491157109' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-20T09:45:50.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:49 smithi026 ceph-mon[68309]: osdmap e1169: 12 total, 12 up, 12 in 2024-03-20T09:45:50.261 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:45:50.273 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-20T09:45:50.535 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:45:51.020 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:50 smithi115 ceph-mon[72476]: 1.e deep-scrub starts 2024-03-20T09:45:51.021 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:50 smithi115 ceph-mon[72476]: 1.11 scrub starts 2024-03-20T09:45:51.021 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:50 smithi115 ceph-mon[72476]: 1.e deep-scrub ok 2024-03-20T09:45:51.021 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:50 smithi115 ceph-mon[72476]: 1.11 scrub ok 2024-03-20T09:45:51.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:50 smithi122 ceph-mon[73495]: 1.e deep-scrub starts 2024-03-20T09:45:51.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:50 smithi122 ceph-mon[73495]: 1.11 scrub starts 2024-03-20T09:45:51.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:50 smithi122 ceph-mon[73495]: 1.e deep-scrub ok 2024-03-20T09:45:51.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:50 smithi122 ceph-mon[73495]: 1.11 scrub ok 2024-03-20T09:45:51.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:50 smithi026 ceph-mon[68309]: 1.e deep-scrub starts 2024-03-20T09:45:51.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:50 smithi026 ceph-mon[68309]: 1.11 scrub starts 2024-03-20T09:45:51.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:50 smithi026 ceph-mon[68309]: 1.e deep-scrub ok 2024-03-20T09:45:51.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:50 smithi026 ceph-mon[68309]: 1.11 scrub ok 2024-03-20T09:45:51.937 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:51 smithi115 ceph-mon[72476]: pgmap v2846: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:52.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:51 smithi122 ceph-mon[73495]: pgmap v2846: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:52.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:51 smithi026 ceph-mon[68309]: pgmap v2846: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:52.212 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:45:52.212 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:45:52.213 INFO:teuthology.orchestra.run.smithi115.stdout: "complaint_time": 30, 2024-03-20T09:45:52.213 INFO:teuthology.orchestra.run.smithi115.stdout: "num_blocked_ops": 0 2024-03-20T09:45:52.213 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:45:52.850 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.4 to 1 (10000) 2024-03-20T09:45:52.875 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-20T09:45:53.136 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:45:53.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:52 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:45:53.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:52 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:45:53.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:52 smithi115 ceph-mon[72476]: osdmap e1170: 12 total, 12 up, 12 in 2024-03-20T09:45:53.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:52 smithi115 ceph-mon[72476]: 1.12 scrub starts 2024-03-20T09:45:53.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:52 smithi115 ceph-mon[72476]: 1.12 scrub ok 2024-03-20T09:45:53.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:52 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2757124313' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-20T09:45:53.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:52 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:45:53.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:52 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:45:53.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:52 smithi122 ceph-mon[73495]: osdmap e1170: 12 total, 12 up, 12 in 2024-03-20T09:45:53.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:52 smithi122 ceph-mon[73495]: 1.12 scrub starts 2024-03-20T09:45:53.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:52 smithi122 ceph-mon[73495]: 1.12 scrub ok 2024-03-20T09:45:53.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:52 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2757124313' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-20T09:45:53.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:52 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:45:53.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:52 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:45:53.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:52 smithi026 ceph-mon[68309]: osdmap e1170: 12 total, 12 up, 12 in 2024-03-20T09:45:53.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:52 smithi026 ceph-mon[68309]: 1.12 scrub starts 2024-03-20T09:45:53.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:52 smithi026 ceph-mon[68309]: 1.12 scrub ok 2024-03-20T09:45:53.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:52 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2757124313' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-20T09:45:53.614 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd primary-affinity 4 1 2024-03-20T09:45:53.897 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:45:54.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:53 smithi115 ceph-mon[72476]: pgmap v2848: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:54.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:53 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2757124313' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-20T09:45:54.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:53 smithi115 ceph-mon[72476]: osdmap e1171: 12 total, 12 up, 12 in 2024-03-20T09:45:54.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:53 smithi115 ceph-mon[72476]: 1.13 scrub starts 2024-03-20T09:45:54.151 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:53 smithi026 ceph-mon[68309]: pgmap v2848: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:54.151 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:53 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2757124313' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-20T09:45:54.151 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:53 smithi026 ceph-mon[68309]: osdmap e1171: 12 total, 12 up, 12 in 2024-03-20T09:45:54.151 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:53 smithi026 ceph-mon[68309]: 1.13 scrub starts 2024-03-20T09:45:54.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:53 smithi122 ceph-mon[73495]: pgmap v2848: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:54.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:53 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2757124313' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-20T09:45:54.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:53 smithi122 ceph-mon[73495]: osdmap e1171: 12 total, 12 up, 12 in 2024-03-20T09:45:54.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:53 smithi122 ceph-mon[73495]: 1.13 scrub starts 2024-03-20T09:45:54.830 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:45:54.830 INFO:teuthology.orchestra.run.smithi115.stdout: "size": 20, 2024-03-20T09:45:54.830 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 600, 2024-03-20T09:45:54.830 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [ 2024-03-20T09:45:54.830 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.830 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "MOSDScrubReserve(1.f,GRANT,e:1152,from:4,reservation_nonce:0)", 2024-03-20T09:45:54.830 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:45:01.737106+0000", 2024-03-20T09:45:54.830 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 53.104411257999999, 2024-03-20T09:45:54.830 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000160512, 2024-03-20T09:45:54.830 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:54.830 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:45:54.830 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:54.830 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.831 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:54.831 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:01.737106+0000", 2024-03-20T09:45:54.831 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.831 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.831 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.831 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:54.831 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:01.737106+0000", 2024-03-20T09:45:54.831 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.831 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.831 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.831 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:54.831 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:01.737107+0000", 2024-03-20T09:45:54.831 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.4900000000000004e-07 2024-03-20T09:45:54.831 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.831 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.832 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:54.832 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:01.737111+0000", 2024-03-20T09:45:54.832 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.8149999999999999e-06 2024-03-20T09:45:54.832 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.832 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.832 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:54.832 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:01.737123+0000", 2024-03-20T09:45:54.832 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.216e-05 2024-03-20T09:45:54.832 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.832 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.832 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:54.832 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:01.737130+0000", 2024-03-20T09:45:54.832 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.0940000000000002e-06 2024-03-20T09:45:54.832 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.832 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.832 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:54.833 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:01.737193+0000", 2024-03-20T09:45:54.833 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2748000000000003e-05 2024-03-20T09:45:54.833 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.833 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.833 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:54.833 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:01.737219+0000", 2024-03-20T09:45:54.833 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6120000000000001e-05 2024-03-20T09:45:54.833 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.833 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.833 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:54.833 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:01.737266+0000", 2024-03-20T09:45:54.833 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.7626000000000001e-05 2024-03-20T09:45:54.833 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.833 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:54.833 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.833 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.834 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.834 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "MOSDScrubReserve(1.f,GRANT,e:1152,from:5,reservation_nonce:0)", 2024-03-20T09:45:54.834 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:45:01.737545+0000", 2024-03-20T09:45:54.834 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 53.103972446999997, 2024-03-20T09:45:54.834 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00025706600000000002, 2024-03-20T09:45:54.834 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:54.834 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:45:54.834 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:54.834 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.834 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:54.834 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:01.737545+0000", 2024-03-20T09:45:54.834 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.834 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.834 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.834 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:54.834 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:01.737545+0000", 2024-03-20T09:45:54.835 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.835 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.835 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.835 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:54.835 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:01.737545+0000", 2024-03-20T09:45:54.835 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.8599999999999997e-07 2024-03-20T09:45:54.835 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.835 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.835 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:54.835 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:01.737550+0000", 2024-03-20T09:45:54.835 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.7620000000000003e-06 2024-03-20T09:45:54.835 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.835 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.835 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:54.835 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:01.737561+0000", 2024-03-20T09:45:54.835 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0926e-05 2024-03-20T09:45:54.836 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.836 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.836 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:54.836 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:01.737567+0000", 2024-03-20T09:45:54.836 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0859999999999998e-06 2024-03-20T09:45:54.836 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.836 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.836 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:54.836 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:01.737609+0000", 2024-03-20T09:45:54.836 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.1567999999999999e-05 2024-03-20T09:45:54.836 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.836 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.836 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:54.836 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:01.737624+0000", 2024-03-20T09:45:54.836 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5471000000000001e-05 2024-03-20T09:45:54.836 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.837 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.837 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:54.837 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:01.737802+0000", 2024-03-20T09:45:54.837 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000177467 2024-03-20T09:45:54.837 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.837 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:54.837 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.837 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.837 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.837 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "rep_scrubmap(1.f e639 from shard 4)", 2024-03-20T09:45:54.837 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:45:01.905363+0000", 2024-03-20T09:45:54.837 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 52.936154541999997, 2024-03-20T09:45:54.837 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.011795317, 2024-03-20T09:45:54.837 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:54.837 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:45:54.838 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:54.838 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.838 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:54.838 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:01.905363+0000", 2024-03-20T09:45:54.838 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.838 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.838 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.838 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:54.838 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:01.905363+0000", 2024-03-20T09:45:54.838 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.838 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.838 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.838 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:54.838 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:01.905364+0000", 2024-03-20T09:45:54.838 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.466e-06 2024-03-20T09:45:54.839 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.839 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.839 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:54.839 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:01.905373+0000", 2024-03-20T09:45:54.839 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.4369999999999996e-06 2024-03-20T09:45:54.839 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.839 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.839 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:54.839 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:01.905391+0000", 2024-03-20T09:45:54.839 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8000999999999999e-05 2024-03-20T09:45:54.839 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.839 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.839 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:54.839 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:01.905403+0000", 2024-03-20T09:45:54.839 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1209999999999999e-05 2024-03-20T09:45:54.839 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.840 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.840 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:54.840 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:01.917127+0000", 2024-03-20T09:45:54.840 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.011723957 2024-03-20T09:45:54.840 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.840 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.840 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:54.840 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:01.917139+0000", 2024-03-20T09:45:54.840 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2649e-05 2024-03-20T09:45:54.840 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.840 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.840 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:54.840 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:01.917158+0000", 2024-03-20T09:45:54.840 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8597000000000002e-05 2024-03-20T09:45:54.840 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.840 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:54.841 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.841 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.841 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.841 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "MOSDScrubReserve(1.19,QREQUEST,e:1158,from:10,reservation_nonce:3)", 2024-03-20T09:45:54.841 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:45:15.496395+0000", 2024-03-20T09:45:54.841 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 39.345122553000003, 2024-03-20T09:45:54.841 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00016535200000000001, 2024-03-20T09:45:54.841 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:54.841 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:45:54.841 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:54.841 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.841 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:54.841 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:15.496395+0000", 2024-03-20T09:45:54.841 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.841 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.842 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.842 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:54.842 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:15.496395+0000", 2024-03-20T09:45:54.842 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.842 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.842 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.842 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:54.842 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:15.496396+0000", 2024-03-20T09:45:54.842 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0279999999999999e-06 2024-03-20T09:45:54.842 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.842 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.842 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:54.842 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:15.496401+0000", 2024-03-20T09:45:54.842 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0479999999999998e-06 2024-03-20T09:45:54.842 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.843 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.843 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:54.843 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:15.496414+0000", 2024-03-20T09:45:54.843 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3618000000000001e-05 2024-03-20T09:45:54.843 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.843 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.843 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:54.843 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:15.496425+0000", 2024-03-20T09:45:54.843 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0981e-05 2024-03-20T09:45:54.843 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.843 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.843 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:54.843 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:15.496496+0000", 2024-03-20T09:45:54.843 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.1272000000000005e-05 2024-03-20T09:45:54.843 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.843 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.844 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:54.844 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:15.496515+0000", 2024-03-20T09:45:54.844 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8646000000000001e-05 2024-03-20T09:45:54.844 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.844 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.844 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:54.844 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:15.496560+0000", 2024-03-20T09:45:54.844 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4759000000000003e-05 2024-03-20T09:45:54.844 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.844 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:54.844 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.844 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.844 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.844 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "replica_scrub(pg: 1.19,from:0'0,to:0'0,epoch:1158/1149,start:1:98000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-03-20T09:45:54.844 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:45:15.515052+0000", 2024-03-20T09:45:54.844 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 39.326465495999997, 2024-03-20T09:45:54.845 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00031392300000000002, 2024-03-20T09:45:54.845 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:54.845 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:45:54.845 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:54.845 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.845 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:54.845 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:15.515052+0000", 2024-03-20T09:45:54.845 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.845 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.845 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.845 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:54.845 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:15.515052+0000", 2024-03-20T09:45:54.845 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.845 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.845 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.845 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:54.846 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:15.515053+0000", 2024-03-20T09:45:54.846 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.624e-06 2024-03-20T09:45:54.846 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.846 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.846 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:54.846 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:15.515066+0000", 2024-03-20T09:45:54.846 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2513e-05 2024-03-20T09:45:54.846 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.846 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.846 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:54.846 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:15.515097+0000", 2024-03-20T09:45:54.846 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.1068999999999997e-05 2024-03-20T09:45:54.846 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.846 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.847 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:54.847 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:15.515121+0000", 2024-03-20T09:45:54.847 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.4686999999999999e-05 2024-03-20T09:45:54.847 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.847 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.847 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:54.847 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:15.515310+0000", 2024-03-20T09:45:54.847 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000188433 2024-03-20T09:45:54.847 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.847 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.847 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:54.847 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:15.515338+0000", 2024-03-20T09:45:54.847 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8563000000000001e-05 2024-03-20T09:45:54.847 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.847 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.847 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:54.848 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:15.515365+0000", 2024-03-20T09:45:54.848 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.7033999999999999e-05 2024-03-20T09:45:54.848 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.848 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:54.848 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.848 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.848 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.848 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "MOSDScrubReserve(1.19,RELEASE,e:1158,from:10,reservation_nonce:0)", 2024-03-20T09:45:54.848 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:45:15.582547+0000", 2024-03-20T09:45:54.848 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 39.258970441999999, 2024-03-20T09:45:54.848 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00020201600000000001, 2024-03-20T09:45:54.848 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:54.848 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:45:54.848 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:54.848 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.849 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:54.849 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:15.582547+0000", 2024-03-20T09:45:54.849 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.849 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.849 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.849 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:54.849 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:15.582547+0000", 2024-03-20T09:45:54.849 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.849 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.849 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.849 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:54.849 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:15.582547+0000", 2024-03-20T09:45:54.849 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4000000000000002e-07 2024-03-20T09:45:54.849 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.849 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.850 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:54.850 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:15.582550+0000", 2024-03-20T09:45:54.850 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.416e-06 2024-03-20T09:45:54.850 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.850 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.850 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:54.850 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:15.582559+0000", 2024-03-20T09:45:54.850 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.7539999999999995e-06 2024-03-20T09:45:54.850 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.850 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.850 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:54.850 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:15.582567+0000", 2024-03-20T09:45:54.850 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.8019999999999992e-06 2024-03-20T09:45:54.850 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.850 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.851 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:54.851 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:15.582711+0000", 2024-03-20T09:45:54.851 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00014401600000000001 2024-03-20T09:45:54.851 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.851 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.851 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:54.851 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:15.582728+0000", 2024-03-20T09:45:54.851 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6730000000000001e-05 2024-03-20T09:45:54.851 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.851 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.851 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:54.851 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:15.582749+0000", 2024-03-20T09:45:54.851 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0857999999999999e-05 2024-03-20T09:45:54.851 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.851 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:54.851 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.852 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.852 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.852 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "replica_scrub(pg: 1.a,from:0'0,to:0'0,epoch:1163/718,start:1:50000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-03-20T09:45:54.852 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:45:30.999658+0000", 2024-03-20T09:45:54.852 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 23.841859438, 2024-03-20T09:45:54.852 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00032017100000000001, 2024-03-20T09:45:54.852 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:54.852 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:45:54.852 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:54.852 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.852 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:54.852 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:30.999658+0000", 2024-03-20T09:45:54.852 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.852 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.852 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.852 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:54.853 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:30.999658+0000", 2024-03-20T09:45:54.853 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.853 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.853 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.853 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:54.853 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:30.999659+0000", 2024-03-20T09:45:54.853 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6929999999999999e-06 2024-03-20T09:45:54.853 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.853 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.853 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:54.853 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:30.999672+0000", 2024-03-20T09:45:54.853 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2765999999999999e-05 2024-03-20T09:45:54.853 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.853 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.853 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:54.853 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:30.999703+0000", 2024-03-20T09:45:54.853 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.0837000000000003e-05 2024-03-20T09:45:54.854 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.854 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.854 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:54.854 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:30.999720+0000", 2024-03-20T09:45:54.854 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7396000000000001e-05 2024-03-20T09:45:54.854 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.854 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.854 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:54.854 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:30.999882+0000", 2024-03-20T09:45:54.854 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000161628 2024-03-20T09:45:54.854 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.854 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.854 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:54.854 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:30.999923+0000", 2024-03-20T09:45:54.854 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.1229000000000001e-05 2024-03-20T09:45:54.854 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.855 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.855 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:54.855 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:30.999978+0000", 2024-03-20T09:45:54.855 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4622e-05 2024-03-20T09:45:54.855 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.855 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:54.855 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.855 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.855 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.855 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "MOSDScrubReserve(1.a,RELEASE,e:1163,from:9,reservation_nonce:0)", 2024-03-20T09:45:54.855 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:45:31.001282+0000", 2024-03-20T09:45:54.855 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 23.840235289999999, 2024-03-20T09:45:54.855 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000194124, 2024-03-20T09:45:54.855 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:54.855 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:45:54.855 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:54.856 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.856 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:54.856 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:31.001282+0000", 2024-03-20T09:45:54.856 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.856 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.856 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.856 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:54.856 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:31.001282+0000", 2024-03-20T09:45:54.856 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.856 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.856 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.856 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:54.856 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:31.001282+0000", 2024-03-20T09:45:54.856 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6100000000000001e-07 2024-03-20T09:45:54.856 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.857 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.857 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:54.857 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:31.001285+0000", 2024-03-20T09:45:54.857 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.2559999999999998e-06 2024-03-20T09:45:54.857 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.857 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.857 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:54.857 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:31.001293+0000", 2024-03-20T09:45:54.857 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.4259999999999997e-06 2024-03-20T09:45:54.857 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.857 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.857 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:54.857 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:31.001300+0000", 2024-03-20T09:45:54.857 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3370000000000002e-06 2024-03-20T09:45:54.857 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.858 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.858 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:54.858 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:31.001423+0000", 2024-03-20T09:45:54.858 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000122667 2024-03-20T09:45:54.858 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.858 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.858 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:54.858 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:31.001443+0000", 2024-03-20T09:45:54.858 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0197999999999999e-05 2024-03-20T09:45:54.858 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.858 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.858 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:54.858 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:31.001476+0000", 2024-03-20T09:45:54.858 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.2778999999999999e-05 2024-03-20T09:45:54.859 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.859 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:54.859 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.859 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.859 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.859 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "MOSDScrubReserve(1.7,GRANT,e:1164,from:3,reservation_nonce:0)", 2024-03-20T09:45:54.859 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:45:32.825838+0000", 2024-03-20T09:45:54.859 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 22.015679063, 2024-03-20T09:45:54.859 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00014691199999999999, 2024-03-20T09:45:54.859 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:54.859 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:45:54.859 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:54.859 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.859 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:54.859 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:32.825838+0000", 2024-03-20T09:45:54.859 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.860 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.860 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.860 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:54.860 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:32.825838+0000", 2024-03-20T09:45:54.860 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.860 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.860 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.860 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:54.860 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:32.825839+0000", 2024-03-20T09:45:54.860 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.6499999999999998e-07 2024-03-20T09:45:54.860 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.860 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.860 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:54.860 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:32.825843+0000", 2024-03-20T09:45:54.860 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.1080000000000003e-06 2024-03-20T09:45:54.860 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.861 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.861 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:54.861 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:32.825855+0000", 2024-03-20T09:45:54.861 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.19e-05 2024-03-20T09:45:54.861 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.861 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.861 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:54.861 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:32.825865+0000", 2024-03-20T09:45:54.861 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0465999999999999e-05 2024-03-20T09:45:54.861 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.861 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.861 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:54.861 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:32.825914+0000", 2024-03-20T09:45:54.861 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8724000000000003e-05 2024-03-20T09:45:54.861 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.861 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.862 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:54.862 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:32.825938+0000", 2024-03-20T09:45:54.862 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.4437e-05 2024-03-20T09:45:54.862 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.862 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.862 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:54.862 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:32.825985+0000", 2024-03-20T09:45:54.862 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6511999999999998e-05 2024-03-20T09:45:54.862 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.862 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:54.862 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.862 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.862 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.862 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "MOSDScrubReserve(1.7,GRANT,e:1164,from:10,reservation_nonce:0)", 2024-03-20T09:45:54.863 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:45:32.826644+0000", 2024-03-20T09:45:54.863 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 22.014872856, 2024-03-20T09:45:54.863 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00025888000000000002, 2024-03-20T09:45:54.863 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:54.863 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:45:54.863 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:54.863 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.863 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:54.863 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:32.826644+0000", 2024-03-20T09:45:54.863 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.863 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.863 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.863 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:54.863 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:32.826644+0000", 2024-03-20T09:45:54.863 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.864 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.864 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.864 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:54.864 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:32.826645+0000", 2024-03-20T09:45:54.864 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.6500000000000008e-07 2024-03-20T09:45:54.864 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.864 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.864 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:54.864 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:32.826650+0000", 2024-03-20T09:45:54.864 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5349999999999998e-06 2024-03-20T09:45:54.864 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.864 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.864 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:54.864 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:32.826663+0000", 2024-03-20T09:45:54.864 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3322000000000001e-05 2024-03-20T09:45:54.864 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.865 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.865 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:54.865 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:32.826670+0000", 2024-03-20T09:45:54.865 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.6379999999999998e-06 2024-03-20T09:45:54.865 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.865 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.865 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:54.865 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:32.826721+0000", 2024-03-20T09:45:54.865 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1777000000000002e-05 2024-03-20T09:45:54.865 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.865 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.865 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:54.865 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:32.826739+0000", 2024-03-20T09:45:54.865 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7473999999999999e-05 2024-03-20T09:45:54.866 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.866 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.866 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:54.866 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:32.826903+0000", 2024-03-20T09:45:54.866 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00016416900000000001 2024-03-20T09:45:54.866 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.866 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:54.866 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.866 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.866 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.866 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "rep_scrubmap(1.7 e960 from shard 10)", 2024-03-20T09:45:54.866 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:45:32.961523+0000", 2024-03-20T09:45:54.866 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 21.879993978000002, 2024-03-20T09:45:54.866 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.034719804, 2024-03-20T09:45:54.866 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:54.867 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:45:54.867 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:54.867 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.867 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:54.867 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:32.961523+0000", 2024-03-20T09:45:54.867 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.867 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.867 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.867 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:54.867 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:32.961523+0000", 2024-03-20T09:45:54.867 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.867 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.867 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.867 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:54.867 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:32.961525+0000", 2024-03-20T09:45:54.867 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7179999999999999e-06 2024-03-20T09:45:54.868 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.868 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.868 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:54.868 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:32.961536+0000", 2024-03-20T09:45:54.868 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1382e-05 2024-03-20T09:45:54.868 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.868 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.868 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:54.868 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:32.961572+0000", 2024-03-20T09:45:54.868 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6223999999999997e-05 2024-03-20T09:45:54.868 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.868 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.868 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:54.868 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:32.961601+0000", 2024-03-20T09:45:54.868 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8588000000000001e-05 2024-03-20T09:45:54.868 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.869 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.869 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:54.869 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:32.996213+0000", 2024-03-20T09:45:54.869 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.034611813999999998 2024-03-20T09:45:54.869 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.869 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.869 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:54.869 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:32.996226+0000", 2024-03-20T09:45:54.869 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.342e-05 2024-03-20T09:45:54.869 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.869 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.869 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:54.869 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:32.996243+0000", 2024-03-20T09:45:54.869 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6657999999999999e-05 2024-03-20T09:45:54.869 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.869 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:54.870 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.870 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.870 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.870 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "rep_scrubmap(1.7 e960 from shard 3)", 2024-03-20T09:45:54.870 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:45:33.029797+0000", 2024-03-20T09:45:54.870 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 21.811719932999999, 2024-03-20T09:45:54.870 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00014683599999999999, 2024-03-20T09:45:54.870 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:54.870 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:45:54.870 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:54.870 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.870 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:54.870 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:33.029797+0000", 2024-03-20T09:45:54.870 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.870 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.870 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.871 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:54.871 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:33.029797+0000", 2024-03-20T09:45:54.871 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.871 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.871 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.871 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:54.871 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:33.029798+0000", 2024-03-20T09:45:54.871 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1850000000000001e-06 2024-03-20T09:45:54.871 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.871 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.871 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:54.871 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:33.029806+0000", 2024-03-20T09:45:54.871 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.4189999999999996e-06 2024-03-20T09:45:54.871 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.872 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.872 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:54.872 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:33.029817+0000", 2024-03-20T09:45:54.872 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1398e-05 2024-03-20T09:45:54.872 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.872 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.872 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:54.872 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:33.029841+0000", 2024-03-20T09:45:54.872 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3864000000000001e-05 2024-03-20T09:45:54.872 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.872 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.872 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:54.872 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:33.029894+0000", 2024-03-20T09:45:54.872 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3321e-05 2024-03-20T09:45:54.872 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.873 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.873 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:54.873 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:33.029907+0000", 2024-03-20T09:45:54.873 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2568000000000001e-05 2024-03-20T09:45:54.873 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.873 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.873 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:54.873 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:33.029944+0000", 2024-03-20T09:45:54.873 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7081e-05 2024-03-20T09:45:54.873 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.873 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:54.873 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.873 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.874 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.874 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "MOSDScrubReserve(1.3,QREQUEST,e:1165,from:1,reservation_nonce:3)", 2024-03-20T09:45:54.874 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:45:37.498486+0000", 2024-03-20T09:45:54.874 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 17.343030765999998, 2024-03-20T09:45:54.874 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00014488599999999999, 2024-03-20T09:45:54.874 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:54.874 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:45:54.874 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:54.874 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.874 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:54.874 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:37.498486+0000", 2024-03-20T09:45:54.874 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.874 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.874 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.874 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:54.875 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:37.498486+0000", 2024-03-20T09:45:54.875 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.875 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.875 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.875 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:54.875 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:37.498487+0000", 2024-03-20T09:45:54.875 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1990000000000001e-06 2024-03-20T09:45:54.875 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.875 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.875 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:54.875 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:37.498493+0000", 2024-03-20T09:45:54.875 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0969999999999996e-06 2024-03-20T09:45:54.875 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.876 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.876 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:54.876 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:37.498507+0000", 2024-03-20T09:45:54.876 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3991999999999999e-05 2024-03-20T09:45:54.876 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.876 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.876 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:54.876 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:37.498515+0000", 2024-03-20T09:45:54.876 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.123e-06 2024-03-20T09:45:54.876 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.876 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.876 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:54.876 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:37.498579+0000", 2024-03-20T09:45:54.876 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4103999999999995e-05 2024-03-20T09:45:54.876 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.877 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.877 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:54.877 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:37.498594+0000", 2024-03-20T09:45:54.877 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5062000000000001e-05 2024-03-20T09:45:54.877 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.877 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.877 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:54.877 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:37.498631+0000", 2024-03-20T09:45:54.877 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7308999999999998e-05 2024-03-20T09:45:54.877 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.877 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:54.877 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.877 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.877 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.878 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "replica_scrub(pg: 1.3,from:0'0,to:0'0,epoch:1165/1164,start:1:c0000000::::0,end:MAX,chunky:1,deep:1,version:9,allow_preemption:1,priority=5)", 2024-03-20T09:45:54.878 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:45:37.633210+0000", 2024-03-20T09:45:54.878 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 17.208306575000002, 2024-03-20T09:45:54.878 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00029460499999999998, 2024-03-20T09:45:54.878 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:54.878 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:45:54.878 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:54.878 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.878 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:54.878 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:37.633210+0000", 2024-03-20T09:45:54.878 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.878 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.878 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.878 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:54.878 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:37.633210+0000", 2024-03-20T09:45:54.879 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.879 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.879 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.879 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:54.879 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:37.633212+0000", 2024-03-20T09:45:54.879 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.84e-06 2024-03-20T09:45:54.879 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.879 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.879 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:54.879 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:37.633225+0000", 2024-03-20T09:45:54.880 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2544000000000001e-05 2024-03-20T09:45:54.880 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.880 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.880 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:54.880 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:37.633257+0000", 2024-03-20T09:45:54.880 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.1795000000000003e-05 2024-03-20T09:45:54.880 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.880 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.880 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:54.880 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:37.633275+0000", 2024-03-20T09:45:54.880 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8366999999999999e-05 2024-03-20T09:45:54.880 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.880 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.880 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:54.881 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:37.633428+0000", 2024-03-20T09:45:54.881 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00015323999999999999 2024-03-20T09:45:54.881 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.881 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.881 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:54.881 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:37.633451+0000", 2024-03-20T09:45:54.881 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2657e-05 2024-03-20T09:45:54.881 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.881 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.881 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:54.881 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:37.633505+0000", 2024-03-20T09:45:54.881 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4162000000000001e-05 2024-03-20T09:45:54.881 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.881 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:54.881 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.881 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.882 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.882 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "MOSDScrubReserve(1.3,RELEASE,e:1165,from:1,reservation_nonce:0)", 2024-03-20T09:45:54.882 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:45:37.772345+0000", 2024-03-20T09:45:54.882 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 17.069172173999998, 2024-03-20T09:45:54.882 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00017331099999999999, 2024-03-20T09:45:54.882 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:54.882 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:45:54.882 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:54.882 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.882 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:54.882 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:37.772345+0000", 2024-03-20T09:45:54.882 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.882 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.882 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.882 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:54.882 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:37.772345+0000", 2024-03-20T09:45:54.883 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.883 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.883 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.883 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:54.883 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:37.772345+0000", 2024-03-20T09:45:54.883 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.7700000000000001e-07 2024-03-20T09:45:54.883 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.883 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.883 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:54.883 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:37.772348+0000", 2024-03-20T09:45:54.883 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9459999999999998e-06 2024-03-20T09:45:54.883 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.883 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.883 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:54.883 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:37.772356+0000", 2024-03-20T09:45:54.883 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.9570000000000002e-06 2024-03-20T09:45:54.884 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.884 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.884 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:54.884 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:37.772362+0000", 2024-03-20T09:45:54.884 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.3720000000000001e-06 2024-03-20T09:45:54.884 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.884 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.884 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:54.884 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:37.772470+0000", 2024-03-20T09:45:54.884 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00010715 2024-03-20T09:45:54.884 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.884 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.884 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:54.884 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:37.772489+0000", 2024-03-20T09:45:54.884 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9916e-05 2024-03-20T09:45:54.884 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.885 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.885 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:54.885 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:37.772518+0000", 2024-03-20T09:45:54.885 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8693000000000001e-05 2024-03-20T09:45:54.885 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.885 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:54.885 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.885 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.885 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.885 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "MOSDScrubReserve(1.1c,QREQUEST,e:1166,from:8,reservation_nonce:19)", 2024-03-20T09:45:54.885 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:45:38.745798+0000", 2024-03-20T09:45:54.885 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 16.095719022000001, 2024-03-20T09:45:54.885 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00017145800000000001, 2024-03-20T09:45:54.885 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:54.885 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:45:54.885 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:54.886 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.886 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:54.886 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:38.745798+0000", 2024-03-20T09:45:54.886 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.886 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.886 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.886 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:54.886 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:38.745798+0000", 2024-03-20T09:45:54.886 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.886 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.886 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.886 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:54.886 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:38.745800+0000", 2024-03-20T09:45:54.886 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.827e-06 2024-03-20T09:45:54.886 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.886 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.887 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:54.887 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:38.745821+0000", 2024-03-20T09:45:54.887 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0925e-05 2024-03-20T09:45:54.887 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.887 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.887 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:54.887 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:38.745835+0000", 2024-03-20T09:45:54.887 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4676e-05 2024-03-20T09:45:54.887 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.887 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.887 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:54.887 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:38.745845+0000", 2024-03-20T09:45:54.887 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.1749999999999994e-06 2024-03-20T09:45:54.887 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.887 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.887 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:54.887 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:38.745905+0000", 2024-03-20T09:45:54.888 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9976999999999999e-05 2024-03-20T09:45:54.888 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.888 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.888 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:54.888 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:38.745924+0000", 2024-03-20T09:45:54.888 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9738999999999999e-05 2024-03-20T09:45:54.888 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.888 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.888 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:54.888 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:38.745969+0000", 2024-03-20T09:45:54.888 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5139000000000003e-05 2024-03-20T09:45:54.888 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.888 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:54.888 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.888 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.888 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.889 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "replica_scrub(pg: 1.1c,from:0'0,to:0'0,epoch:1166/950,start:1:38000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-03-20T09:45:54.889 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:45:38.770201+0000", 2024-03-20T09:45:54.889 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 16.071316444000001, 2024-03-20T09:45:54.889 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000191416, 2024-03-20T09:45:54.889 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:54.889 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:45:54.889 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:54.889 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.889 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:54.889 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:38.770201+0000", 2024-03-20T09:45:54.889 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.889 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.889 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.889 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:54.889 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:38.770201+0000", 2024-03-20T09:45:54.890 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.890 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.890 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.890 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:54.890 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:38.770202+0000", 2024-03-20T09:45:54.890 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.863e-06 2024-03-20T09:45:54.890 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.890 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.890 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:54.890 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:38.770216+0000", 2024-03-20T09:45:54.890 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.345e-05 2024-03-20T09:45:54.890 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.890 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.890 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:54.890 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:38.770252+0000", 2024-03-20T09:45:54.891 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.5750000000000002e-05 2024-03-20T09:45:54.891 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.891 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.891 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:54.891 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:38.770278+0000", 2024-03-20T09:45:54.891 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6200999999999999e-05 2024-03-20T09:45:54.891 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.891 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.891 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:54.891 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:38.770351+0000", 2024-03-20T09:45:54.891 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3096000000000006e-05 2024-03-20T09:45:54.891 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.891 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.891 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:54.891 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:38.770362+0000", 2024-03-20T09:45:54.892 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1088e-05 2024-03-20T09:45:54.892 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.892 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.892 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:54.892 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:38.770392+0000", 2024-03-20T09:45:54.892 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9967999999999998e-05 2024-03-20T09:45:54.892 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.892 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:54.892 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.892 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.892 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.892 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "MOSDScrubReserve(1.1c,RELEASE,e:1166,from:8,reservation_nonce:0)", 2024-03-20T09:45:54.892 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:45:38.792678+0000", 2024-03-20T09:45:54.892 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 16.048839392000001, 2024-03-20T09:45:54.892 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000188524, 2024-03-20T09:45:54.892 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:54.893 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:45:54.893 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:54.893 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.893 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:54.893 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:38.792678+0000", 2024-03-20T09:45:54.893 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.893 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.893 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.893 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:54.893 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:38.792678+0000", 2024-03-20T09:45:54.893 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.893 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.893 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.893 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:54.893 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:38.792678+0000", 2024-03-20T09:45:54.893 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.37e-07 2024-03-20T09:45:54.894 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.894 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.894 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:54.894 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:38.792683+0000", 2024-03-20T09:45:54.894 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.843e-06 2024-03-20T09:45:54.894 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.894 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.894 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:54.894 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:38.792712+0000", 2024-03-20T09:45:54.894 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8829000000000001e-05 2024-03-20T09:45:54.894 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.894 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.894 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:54.894 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:38.792717+0000", 2024-03-20T09:45:54.894 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3249999999999998e-06 2024-03-20T09:45:54.894 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.895 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.895 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:54.895 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:38.792829+0000", 2024-03-20T09:45:54.895 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000111776 2024-03-20T09:45:54.895 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.895 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.895 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:54.895 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:38.792846+0000", 2024-03-20T09:45:54.895 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6623e-05 2024-03-20T09:45:54.895 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.895 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.895 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:54.895 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:38.792866+0000", 2024-03-20T09:45:54.895 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0491000000000001e-05 2024-03-20T09:45:54.895 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.895 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:54.896 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.896 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.896 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.896 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "MOSDScrubReserve(1.12,QREQUEST,e:1169,from:5,reservation_nonce:35)", 2024-03-20T09:45:54.896 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:45:52.008764+0000", 2024-03-20T09:45:54.896 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 2.83275341, 2024-03-20T09:45:54.896 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00018408199999999999, 2024-03-20T09:45:54.896 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:54.896 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:45:54.896 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:54.896 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.896 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:54.896 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:52.008764+0000", 2024-03-20T09:45:54.896 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.896 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.896 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.896 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:54.897 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:52.008764+0000", 2024-03-20T09:45:54.897 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.897 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.897 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.897 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:54.897 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:52.008765+0000", 2024-03-20T09:45:54.897 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1990000000000001e-06 2024-03-20T09:45:54.897 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.897 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.897 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:54.897 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:52.008770+0000", 2024-03-20T09:45:54.897 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1189999999999996e-06 2024-03-20T09:45:54.897 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.897 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.897 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:54.897 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:52.008785+0000", 2024-03-20T09:45:54.898 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4652e-05 2024-03-20T09:45:54.898 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.898 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.898 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:54.898 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:52.008792+0000", 2024-03-20T09:45:54.898 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3350000000000001e-06 2024-03-20T09:45:54.898 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.898 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.898 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:54.898 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:52.008883+0000", 2024-03-20T09:45:54.898 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.1015999999999997e-05 2024-03-20T09:45:54.898 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.898 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.898 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:54.898 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:52.008903+0000", 2024-03-20T09:45:54.898 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9976999999999999e-05 2024-03-20T09:45:54.899 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.899 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.899 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:54.899 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:52.008948+0000", 2024-03-20T09:45:54.899 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4783999999999997e-05 2024-03-20T09:45:54.899 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.899 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:54.899 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.899 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.899 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.899 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "MOSDScrubReserve(1.12,RELEASE,e:1170,from:5,reservation_nonce:0)", 2024-03-20T09:45:54.899 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:45:52.092503+0000", 2024-03-20T09:45:54.899 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 2.7490142720000001, 2024-03-20T09:45:54.899 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00014227200000000001, 2024-03-20T09:45:54.899 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:45:54.899 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:45:54.899 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:45:54.900 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.900 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:45:54.900 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:52.092503+0000", 2024-03-20T09:45:54.900 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.900 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.900 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.900 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:45:54.900 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:52.092503+0000", 2024-03-20T09:45:54.900 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:45:54.900 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.900 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.900 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:45:54.900 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:52.092503+0000", 2024-03-20T09:45:54.900 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.2899999999999999e-07 2024-03-20T09:45:54.900 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.900 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.901 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:45:54.901 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:52.092506+0000", 2024-03-20T09:45:54.901 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.1930000000000002e-06 2024-03-20T09:45:54.901 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.901 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.901 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:45:54.901 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:52.092514+0000", 2024-03-20T09:45:54.901 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3760000000000002e-06 2024-03-20T09:45:54.901 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.901 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.901 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:45:54.901 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:52.092521+0000", 2024-03-20T09:45:54.901 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.6550000000000004e-06 2024-03-20T09:45:54.901 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.901 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.901 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:45:54.901 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:52.092608+0000", 2024-03-20T09:45:54.902 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.7033000000000005e-05 2024-03-20T09:45:54.902 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.902 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.902 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:45:54.902 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:52.092623+0000", 2024-03-20T09:45:54.902 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4854999999999999e-05 2024-03-20T09:45:54.902 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:45:54.902 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:45:54.902 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:45:54.902 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:52.092645+0000", 2024-03-20T09:45:54.902 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1831000000000002e-05 2024-03-20T09:45:54.902 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.902 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:54.902 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.902 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:45:54.903 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:45:54.903 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:45:55.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:54 smithi115 ceph-mon[72476]: 1.13 scrub ok 2024-03-20T09:45:55.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:54 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:45:55.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:54 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:45:55.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:54 smithi122 ceph-mon[73495]: 1.13 scrub ok 2024-03-20T09:45:55.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:54 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:45:55.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:54 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:45:55.205 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:54 smithi026 ceph-mon[68309]: 1.13 scrub ok 2024-03-20T09:45:55.205 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:54 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:45:55.205 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:54 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:45:55.511 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-20T09:45:55.775 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:45:56.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:55 smithi115 ceph-mon[72476]: pgmap v2850: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:56.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:55 smithi115 ceph-mon[72476]: osdmap e1172: 12 total, 12 up, 12 in 2024-03-20T09:45:56.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:55 smithi122 ceph-mon[73495]: pgmap v2850: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:56.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:55 smithi122 ceph-mon[73495]: osdmap e1172: 12 total, 12 up, 12 in 2024-03-20T09:45:56.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:55 smithi026 ceph-mon[68309]: pgmap v2850: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:56.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:55 smithi026 ceph-mon[68309]: osdmap e1172: 12 total, 12 up, 12 in 2024-03-20T09:45:56.870 INFO:teuthology.orchestra.run.smithi026.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-20T09:45:56.915 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:56 smithi115 ceph-mon[72476]: 1.c scrub starts 2024-03-20T09:45:56.915 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:56 smithi115 ceph-mon[72476]: 1.c scrub ok 2024-03-20T09:45:56.915 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:56 smithi115 ceph-mon[72476]: 1.17 scrub starts 2024-03-20T09:45:56.915 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:56 smithi115 ceph-mon[72476]: 1.1 scrub starts 2024-03-20T09:45:56.916 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:56 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/409840240' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-20T09:45:56.916 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:56 smithi115 ceph-mon[72476]: 1.17 scrub ok 2024-03-20T09:45:56.916 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:56 smithi115 ceph-mon[72476]: 1.1 scrub ok 2024-03-20T09:45:57.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:56 smithi122 ceph-mon[73495]: 1.c scrub starts 2024-03-20T09:45:57.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:56 smithi122 ceph-mon[73495]: 1.c scrub ok 2024-03-20T09:45:57.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:56 smithi122 ceph-mon[73495]: 1.17 scrub starts 2024-03-20T09:45:57.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:56 smithi122 ceph-mon[73495]: 1.1 scrub starts 2024-03-20T09:45:57.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:56 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/409840240' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-20T09:45:57.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:56 smithi122 ceph-mon[73495]: 1.17 scrub ok 2024-03-20T09:45:57.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:56 smithi122 ceph-mon[73495]: 1.1 scrub ok 2024-03-20T09:45:57.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:56 smithi026 ceph-mon[68309]: 1.c scrub starts 2024-03-20T09:45:57.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:56 smithi026 ceph-mon[68309]: 1.c scrub ok 2024-03-20T09:45:57.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:56 smithi026 ceph-mon[68309]: 1.17 scrub starts 2024-03-20T09:45:57.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:56 smithi026 ceph-mon[68309]: 1.1 scrub starts 2024-03-20T09:45:57.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:56 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/409840240' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-20T09:45:57.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:56 smithi026 ceph-mon[68309]: 1.17 scrub ok 2024-03-20T09:45:57.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:56 smithi026 ceph-mon[68309]: 1.1 scrub ok 2024-03-20T09:45:57.404 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:45:57.404 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:45:57.405 INFO:teuthology.orchestra.run.smithi115.stdout: "num_ops": 0 2024-03-20T09:45:57.405 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:45:57.586 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 6 1 2024-03-20T09:45:57.870 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:45:58.065 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:57 smithi115 ceph-mon[72476]: 1.8 scrub starts 2024-03-20T09:45:58.065 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:57 smithi115 ceph-mon[72476]: 1.8 scrub ok 2024-03-20T09:45:58.065 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:57 smithi115 ceph-mon[72476]: pgmap v2852: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:58.065 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:57 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/409840240' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-20T09:45:58.065 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:57 smithi115 ceph-mon[72476]: osdmap e1173: 12 total, 12 up, 12 in 2024-03-20T09:45:58.090 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-20T09:45:58.124 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:57 smithi026 ceph-mon[68309]: 1.8 scrub starts 2024-03-20T09:45:58.124 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:57 smithi026 ceph-mon[68309]: 1.8 scrub ok 2024-03-20T09:45:58.124 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:57 smithi026 ceph-mon[68309]: pgmap v2852: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:58.124 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:57 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/409840240' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-20T09:45:58.124 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:57 smithi026 ceph-mon[68309]: osdmap e1173: 12 total, 12 up, 12 in 2024-03-20T09:45:58.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:57 smithi122 ceph-mon[73495]: 1.8 scrub starts 2024-03-20T09:45:58.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:57 smithi122 ceph-mon[73495]: 1.8 scrub ok 2024-03-20T09:45:58.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:57 smithi122 ceph-mon[73495]: pgmap v2852: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:58.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:57 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/409840240' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-20T09:45:58.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:57 smithi122 ceph-mon[73495]: osdmap e1173: 12 total, 12 up, 12 in 2024-03-20T09:45:58.392 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:45:59.161 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:59 smithi026 ceph-mon[68309]: 1.6 scrub starts 2024-03-20T09:45:59.162 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:59 smithi026 ceph-mon[68309]: 1.6 scrub ok 2024-03-20T09:45:59.162 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:45:59 smithi026 ceph-mon[68309]: pgmap v2854: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:59.282 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:59 smithi115 ceph-mon[72476]: 1.6 scrub starts 2024-03-20T09:45:59.282 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:59 smithi115 ceph-mon[72476]: 1.6 scrub ok 2024-03-20T09:45:59.282 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:45:59 smithi115 ceph-mon[72476]: pgmap v2854: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:45:59.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:59 smithi122 ceph-mon[73495]: 1.6 scrub starts 2024-03-20T09:45:59.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:59 smithi122 ceph-mon[73495]: 1.6 scrub ok 2024-03-20T09:45:59.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:45:59 smithi122 ceph-mon[73495]: pgmap v2854: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:00.020 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.6 to 1 (10000) 2024-03-20T09:46:00.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:00 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:46:00.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:00 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:46:00.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:00 smithi115 ceph-mon[72476]: 1.0 deep-scrub starts 2024-03-20T09:46:00.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:00 smithi115 ceph-mon[72476]: 1.0 deep-scrub ok 2024-03-20T09:46:00.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:00 smithi115 ceph-mon[72476]: osdmap e1174: 12 total, 12 up, 12 in 2024-03-20T09:46:00.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:00 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:46:00.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:00 smithi115 ceph-mon[72476]: from='client.17760 172.21.15.26:0/50779904' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-20T09:46:00.160 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:46:00.160 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:46:00.160 INFO:teuthology.orchestra.run.smithi115.stdout: "complaint_time": 30, 2024-03-20T09:46:00.160 INFO:teuthology.orchestra.run.smithi115.stdout: "num_blocked_ops": 0 2024-03-20T09:46:00.160 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:46:00.388 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:00 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:46:00.388 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:00 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:46:00.388 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:00 smithi026 ceph-mon[68309]: 1.0 deep-scrub starts 2024-03-20T09:46:00.388 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:00 smithi026 ceph-mon[68309]: 1.0 deep-scrub ok 2024-03-20T09:46:00.388 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:00 smithi026 ceph-mon[68309]: osdmap e1174: 12 total, 12 up, 12 in 2024-03-20T09:46:00.388 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:00 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:46:00.388 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:00 smithi026 ceph-mon[68309]: from='client.17760 172.21.15.26:0/50779904' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-20T09:46:00.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:00 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:46:00.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:00 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:46:00.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:00 smithi122 ceph-mon[73495]: 1.0 deep-scrub starts 2024-03-20T09:46:00.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:00 smithi122 ceph-mon[73495]: 1.0 deep-scrub ok 2024-03-20T09:46:00.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:00 smithi122 ceph-mon[73495]: osdmap e1174: 12 total, 12 up, 12 in 2024-03-20T09:46:00.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:00 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:46:00.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:00 smithi122 ceph-mon[73495]: from='client.17760 172.21.15.26:0/50779904' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-20T09:46:00.738 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd primary-affinity 6 1 2024-03-20T09:46:00.797 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-20T09:46:01.063 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:46:01.254 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:46:01.298 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:01 smithi115 ceph-mon[72476]: from='client.17760 172.21.15.26:0/50779904' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-20T09:46:01.298 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:01 smithi115 ceph-mon[72476]: osdmap e1175: 12 total, 12 up, 12 in 2024-03-20T09:46:01.298 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:01 smithi115 ceph-mon[72476]: pgmap v2857: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:01.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:01 smithi122 ceph-mon[73495]: from='client.17760 172.21.15.26:0/50779904' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-20T09:46:01.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:01 smithi122 ceph-mon[73495]: osdmap e1175: 12 total, 12 up, 12 in 2024-03-20T09:46:01.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:01 smithi122 ceph-mon[73495]: pgmap v2857: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:01.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:01 smithi026 ceph-mon[68309]: from='client.17760 172.21.15.26:0/50779904' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-20T09:46:01.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:01 smithi026 ceph-mon[68309]: osdmap e1175: 12 total, 12 up, 12 in 2024-03-20T09:46:01.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:01 smithi026 ceph-mon[68309]: pgmap v2857: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:02.755 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:46:02.755 INFO:teuthology.orchestra.run.smithi115.stdout: "size": 20, 2024-03-20T09:46:02.755 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 600, 2024-03-20T09:46:02.755 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [ 2024-03-20T09:46:02.755 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.755 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:14523 8.d e980/974 8:b1c3d5d8:::benchmark_data_smithi122_197775_object14522:head v 980'870, mlcod=980'868)", 2024-03-20T09:46:02.755 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:05.332829+0000", 2024-03-20T09:46:02.755 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 597.43343622800001, 2024-03-20T09:46:02.755 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.080553347999999997, 2024-03-20T09:46:02.756 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:02.756 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:02.756 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:02.756 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.756 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:02.756 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.332829+0000", 2024-03-20T09:46:02.756 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.756 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.756 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.756 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:02.757 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.332829+0000", 2024-03-20T09:46:02.757 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.757 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.757 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.757 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:02.757 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.332829+0000", 2024-03-20T09:46:02.757 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5000000000000001e-07 2024-03-20T09:46:02.757 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.757 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.757 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:02.757 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.332921+0000", 2024-03-20T09:46:02.758 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.1848999999999994e-05 2024-03-20T09:46:02.758 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.758 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.758 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:02.758 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.332933+0000", 2024-03-20T09:46:02.758 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2071e-05 2024-03-20T09:46:02.758 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.758 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.758 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:02.758 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.332941+0000", 2024-03-20T09:46:02.758 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.8520000000000004e-06 2024-03-20T09:46:02.759 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.759 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.759 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:02.759 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.332994+0000", 2024-03-20T09:46:02.759 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.2398000000000002e-05 2024-03-20T09:46:02.759 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.759 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.759 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:02.759 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.333015+0000", 2024-03-20T09:46:02.759 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1807999999999999e-05 2024-03-20T09:46:02.759 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.760 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.760 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:02.760 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.413352+0000", 2024-03-20T09:46:02.760 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.080337069999999997 2024-03-20T09:46:02.760 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.760 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.760 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:02.760 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.413382+0000", 2024-03-20T09:46:02.760 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9450000000000001e-05 2024-03-20T09:46:02.761 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.761 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:02.761 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.761 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.761 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.761 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:14525 8.d e980/974 8:bd7f84eb:::benchmark_data_smithi122_197775_object14524:head v 980'871, mlcod=980'868)", 2024-03-20T09:46:02.761 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:05.335283+0000", 2024-03-20T09:46:02.761 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 597.43098186199995, 2024-03-20T09:46:02.761 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.078128146999999995, 2024-03-20T09:46:02.761 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:02.761 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:02.761 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:02.761 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.762 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:02.762 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.335283+0000", 2024-03-20T09:46:02.762 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.762 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.762 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.762 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:02.762 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.335283+0000", 2024-03-20T09:46:02.762 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.762 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.762 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.762 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:02.763 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.335283+0000", 2024-03-20T09:46:02.763 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4700000000000001e-07 2024-03-20T09:46:02.763 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.763 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.763 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:02.763 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.335453+0000", 2024-03-20T09:46:02.763 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00016961600000000001 2024-03-20T09:46:02.763 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.763 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.763 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:02.763 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.335462+0000", 2024-03-20T09:46:02.763 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.2499999999999995e-06 2024-03-20T09:46:02.764 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.764 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.764 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:02.764 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.335468+0000", 2024-03-20T09:46:02.764 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8189999999999997e-06 2024-03-20T09:46:02.764 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.764 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.764 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:02.764 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.335511+0000", 2024-03-20T09:46:02.764 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.2478999999999998e-05 2024-03-20T09:46:02.764 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.765 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.765 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:02.765 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.335531+0000", 2024-03-20T09:46:02.765 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0040000000000001e-05 2024-03-20T09:46:02.765 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.765 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.765 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:02.765 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.413385+0000", 2024-03-20T09:46:02.765 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.077854520999999996 2024-03-20T09:46:02.765 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.765 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.766 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:02.766 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.413411+0000", 2024-03-20T09:46:02.766 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5874999999999999e-05 2024-03-20T09:46:02.766 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.766 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:02.766 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.766 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.766 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.766 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:14534 8.d e980/974 8:bd505d36:::benchmark_data_smithi122_197775_object14533:head v 980'872, mlcod=980'868)", 2024-03-20T09:46:02.766 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:05.337841+0000", 2024-03-20T09:46:02.766 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 597.42842398599998, 2024-03-20T09:46:02.767 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.075595009000000005, 2024-03-20T09:46:02.767 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:02.767 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:02.767 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:02.767 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.767 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:02.767 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.337841+0000", 2024-03-20T09:46:02.767 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.767 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.767 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.767 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:02.767 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.337841+0000", 2024-03-20T09:46:02.768 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.768 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.768 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.768 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:02.768 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.337841+0000", 2024-03-20T09:46:02.768 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.5199999999999998e-07 2024-03-20T09:46:02.768 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.768 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.768 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:02.768 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.337928+0000", 2024-03-20T09:46:02.768 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.6570000000000006e-05 2024-03-20T09:46:02.768 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.768 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.769 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:02.769 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.337935+0000", 2024-03-20T09:46:02.769 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.4059999999999999e-06 2024-03-20T09:46:02.769 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.769 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.769 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:02.769 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.337941+0000", 2024-03-20T09:46:02.769 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8699999999999997e-06 2024-03-20T09:46:02.769 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.769 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.769 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:02.770 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.337985+0000", 2024-03-20T09:46:02.770 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4030999999999999e-05 2024-03-20T09:46:02.770 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.770 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.770 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:02.770 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.338003+0000", 2024-03-20T09:46:02.770 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7759000000000001e-05 2024-03-20T09:46:02.770 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.770 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.770 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:02.770 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.413414+0000", 2024-03-20T09:46:02.770 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.075411117 2024-03-20T09:46:02.771 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.771 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.771 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:02.771 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.413436+0000", 2024-03-20T09:46:02.771 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1903999999999999e-05 2024-03-20T09:46:02.771 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.771 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:02.771 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.771 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.771 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.772 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:14545 8.d e980/974 8:b7f9247f:::benchmark_data_smithi122_197775_object14544:head v 980'873, mlcod=980'868)", 2024-03-20T09:46:02.772 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:05.340825+0000", 2024-03-20T09:46:02.772 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 597.42543994000005, 2024-03-20T09:46:02.772 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.072634562999999999, 2024-03-20T09:46:02.772 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:02.772 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:02.772 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:02.772 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.772 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:02.772 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.340825+0000", 2024-03-20T09:46:02.772 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.772 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.772 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.773 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:02.773 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.340825+0000", 2024-03-20T09:46:02.773 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.773 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.773 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.773 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:02.773 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.340825+0000", 2024-03-20T09:46:02.773 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.75e-07 2024-03-20T09:46:02.773 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.773 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.773 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:02.773 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.340902+0000", 2024-03-20T09:46:02.774 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7058999999999995e-05 2024-03-20T09:46:02.774 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.774 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.774 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:02.774 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.340911+0000", 2024-03-20T09:46:02.774 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.7399999999999993e-06 2024-03-20T09:46:02.774 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.774 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.774 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:02.774 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.340918+0000", 2024-03-20T09:46:02.774 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.5910000000000001e-06 2024-03-20T09:46:02.774 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.775 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.775 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:02.775 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.340957+0000", 2024-03-20T09:46:02.775 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.9146000000000001e-05 2024-03-20T09:46:02.775 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.775 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.775 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:02.775 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.340972+0000", 2024-03-20T09:46:02.775 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4972e-05 2024-03-20T09:46:02.775 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.775 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.775 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:02.775 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.413438+0000", 2024-03-20T09:46:02.776 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.072465871000000001 2024-03-20T09:46:02.776 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.776 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.776 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:02.776 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.413459+0000", 2024-03-20T09:46:02.776 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1608999999999999e-05 2024-03-20T09:46:02.776 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.776 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:02.776 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.776 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.776 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.776 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:14559 8.d e980/974 8:bfa51422:::benchmark_data_smithi122_197775_object14558:head v 980'874, mlcod=980'868)", 2024-03-20T09:46:02.776 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:05.348897+0000", 2024-03-20T09:46:02.777 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 597.41736770199998, 2024-03-20T09:46:02.777 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.064589677999999998, 2024-03-20T09:46:02.777 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:02.777 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:02.777 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:02.777 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.777 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:02.777 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.348897+0000", 2024-03-20T09:46:02.777 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.777 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.777 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.777 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:02.777 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.348897+0000", 2024-03-20T09:46:02.778 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.778 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.778 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.778 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:02.778 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.348898+0000", 2024-03-20T09:46:02.778 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.023e-06 2024-03-20T09:46:02.778 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.778 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.778 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:02.778 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.349026+0000", 2024-03-20T09:46:02.778 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012763799999999999 2024-03-20T09:46:02.779 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.779 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.779 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:02.779 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.349042+0000", 2024-03-20T09:46:02.779 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6256000000000002e-05 2024-03-20T09:46:02.779 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.779 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.779 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:02.779 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.349048+0000", 2024-03-20T09:46:02.779 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5709999999999997e-06 2024-03-20T09:46:02.779 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.779 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.780 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:02.780 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.349584+0000", 2024-03-20T09:46:02.780 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000536288 2024-03-20T09:46:02.780 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.780 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.780 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:02.780 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.349604+0000", 2024-03-20T09:46:02.780 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9874e-05 2024-03-20T09:46:02.780 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.780 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.780 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:02.780 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.413462+0000", 2024-03-20T09:46:02.781 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.063857842999999997 2024-03-20T09:46:02.781 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.781 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.781 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:02.781 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:05.413487+0000", 2024-03-20T09:46:02.781 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5185e-05 2024-03-20T09:46:02.781 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.781 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:02.781 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.781 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.781 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.781 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:85294 8.d e998/974 8:bf9f6105:::benchmark_data_smithi122_197775_object85293:head v 998'5362, mlcod=998'5360)", 2024-03-20T09:46:02.782 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:50.030225+0000", 2024-03-20T09:46:02.782 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 552.736039441, 2024-03-20T09:46:02.782 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.094034662000000005, 2024-03-20T09:46:02.782 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:02.782 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:02.782 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:02.782 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.782 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:02.782 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.030225+0000", 2024-03-20T09:46:02.782 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.782 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.782 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.783 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:02.783 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.030225+0000", 2024-03-20T09:46:02.783 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.783 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.783 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.783 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:02.783 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.030227+0000", 2024-03-20T09:46:02.783 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.33e-06 2024-03-20T09:46:02.783 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.783 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.783 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:02.784 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.030284+0000", 2024-03-20T09:46:02.784 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6991999999999999e-05 2024-03-20T09:46:02.784 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.784 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.784 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:02.784 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.030305+0000", 2024-03-20T09:46:02.784 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1276000000000001e-05 2024-03-20T09:46:02.784 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.784 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.784 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:02.784 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.030322+0000", 2024-03-20T09:46:02.785 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.6739e-05 2024-03-20T09:46:02.785 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.785 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.785 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:02.785 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.030394+0000", 2024-03-20T09:46:02.785 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.2361999999999996e-05 2024-03-20T09:46:02.785 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.785 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.785 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:02.785 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.030416+0000", 2024-03-20T09:46:02.785 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1920999999999999e-05 2024-03-20T09:46:02.785 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.786 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.786 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:02.786 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124230+0000", 2024-03-20T09:46:02.786 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.093813721000000003 2024-03-20T09:46:02.786 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.786 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.786 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:02.786 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124260+0000", 2024-03-20T09:46:02.786 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.0321e-05 2024-03-20T09:46:02.786 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.786 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:02.787 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.787 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.787 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.787 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:85304 8.d e998/974 8:b9b9ee18:::benchmark_data_smithi122_197775_object85303:head v 998'5363, mlcod=998'5360)", 2024-03-20T09:46:02.787 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:50.035062+0000", 2024-03-20T09:46:02.787 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 552.73120266399997, 2024-03-20T09:46:02.787 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.089231913999999996, 2024-03-20T09:46:02.787 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:02.787 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:02.787 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:02.787 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.787 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:02.789 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.035062+0000", 2024-03-20T09:46:02.789 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.789 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.789 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.789 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:02.789 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.035062+0000", 2024-03-20T09:46:02.789 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.789 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.790 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.790 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:02.790 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.035063+0000", 2024-03-20T09:46:02.790 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.9999999999999996e-07 2024-03-20T09:46:02.790 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.790 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.790 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:02.790 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.035117+0000", 2024-03-20T09:46:02.790 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.363e-05 2024-03-20T09:46:02.790 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.790 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.790 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:02.791 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.035128+0000", 2024-03-20T09:46:02.791 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1561e-05 2024-03-20T09:46:02.791 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.791 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.791 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:02.791 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.035136+0000", 2024-03-20T09:46:02.791 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.4749999999999996e-06 2024-03-20T09:46:02.791 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.791 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.791 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:02.791 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.035232+0000", 2024-03-20T09:46:02.792 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.6311000000000007e-05 2024-03-20T09:46:02.792 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.792 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.792 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:02.792 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.035267+0000", 2024-03-20T09:46:02.792 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.4971000000000001e-05 2024-03-20T09:46:02.792 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.792 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.792 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:02.792 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124264+0000", 2024-03-20T09:46:02.792 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.088997007000000003 2024-03-20T09:46:02.792 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.793 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.793 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:02.793 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124294+0000", 2024-03-20T09:46:02.793 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.0059000000000001e-05 2024-03-20T09:46:02.793 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.793 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:02.793 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.793 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.793 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.793 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:85309 8.d e998/974 8:b696b706:::benchmark_data_smithi122_197775_object85308:head v 998'5364, mlcod=998'5360)", 2024-03-20T09:46:02.793 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:50.039124+0000", 2024-03-20T09:46:02.793 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 552.72714116500003, 2024-03-20T09:46:02.794 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.085202424999999998, 2024-03-20T09:46:02.794 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:02.794 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:02.794 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:02.794 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.794 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:02.794 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.039124+0000", 2024-03-20T09:46:02.794 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.794 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.794 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.794 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:02.794 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.039124+0000", 2024-03-20T09:46:02.795 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.795 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.795 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.795 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:02.795 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.039124+0000", 2024-03-20T09:46:02.795 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.6400000000000001e-07 2024-03-20T09:46:02.795 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.795 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.795 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:02.795 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.039228+0000", 2024-03-20T09:46:02.795 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000103757 2024-03-20T09:46:02.796 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.796 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.796 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:02.796 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.039240+0000", 2024-03-20T09:46:02.796 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1652e-05 2024-03-20T09:46:02.796 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.796 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.796 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:02.796 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.039247+0000", 2024-03-20T09:46:02.796 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3359999999999997e-06 2024-03-20T09:46:02.796 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.796 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.797 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:02.797 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.039311+0000", 2024-03-20T09:46:02.797 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.3662000000000001e-05 2024-03-20T09:46:02.797 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.797 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.797 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:02.797 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.039335+0000", 2024-03-20T09:46:02.797 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3725000000000001e-05 2024-03-20T09:46:02.797 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.797 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.797 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:02.797 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124296+0000", 2024-03-20T09:46:02.797 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.084961693000000005 2024-03-20T09:46:02.798 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.798 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.798 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:02.798 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124326+0000", 2024-03-20T09:46:02.798 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9736000000000001e-05 2024-03-20T09:46:02.798 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.798 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:02.798 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.798 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.798 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.798 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:85325 8.d e998/974 8:bc21c74a:::benchmark_data_smithi122_197775_object85324:head v 998'5365, mlcod=998'5360)", 2024-03-20T09:46:02.799 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:50.052544+0000", 2024-03-20T09:46:02.799 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 552.71372081100003, 2024-03-20T09:46:02.799 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.071814343000000003, 2024-03-20T09:46:02.799 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:02.799 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:02.799 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:02.799 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.799 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:02.799 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.052544+0000", 2024-03-20T09:46:02.799 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.799 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.799 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.799 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:02.800 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.052544+0000", 2024-03-20T09:46:02.800 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.800 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.800 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.800 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:02.800 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.052545+0000", 2024-03-20T09:46:02.800 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.415e-06 2024-03-20T09:46:02.800 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.800 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.800 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:02.800 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.052580+0000", 2024-03-20T09:46:02.800 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.4459999999999999e-05 2024-03-20T09:46:02.801 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.801 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.801 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:02.801 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.052600+0000", 2024-03-20T09:46:02.801 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9853000000000001e-05 2024-03-20T09:46:02.801 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.801 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.801 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:02.801 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.052610+0000", 2024-03-20T09:46:02.801 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0655999999999999e-05 2024-03-20T09:46:02.801 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.802 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.802 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:02.802 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.052671+0000", 2024-03-20T09:46:02.802 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0565e-05 2024-03-20T09:46:02.802 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.802 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.802 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:02.802 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.052693+0000", 2024-03-20T09:46:02.802 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2308000000000001e-05 2024-03-20T09:46:02.802 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.802 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.802 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:02.803 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124329+0000", 2024-03-20T09:46:02.803 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.071635628000000007 2024-03-20T09:46:02.803 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.803 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.803 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:02.803 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124358+0000", 2024-03-20T09:46:02.803 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9458000000000002e-05 2024-03-20T09:46:02.803 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.803 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:02.803 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.803 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.803 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.803 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:85327 8.d e998/974 8:b1150ede:::benchmark_data_smithi122_197775_object85326:head v 998'5366, mlcod=998'5360)", 2024-03-20T09:46:02.804 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:50.055053+0000", 2024-03-20T09:46:02.804 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 552.71121172899996, 2024-03-20T09:46:02.804 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.069332682000000007, 2024-03-20T09:46:02.804 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:02.804 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:02.804 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:02.804 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.804 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:02.804 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.055053+0000", 2024-03-20T09:46:02.804 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.804 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.805 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.805 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:02.805 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.055053+0000", 2024-03-20T09:46:02.805 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.805 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.805 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.805 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:02.805 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.055054+0000", 2024-03-20T09:46:02.805 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2470000000000001e-06 2024-03-20T09:46:02.805 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.805 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.805 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:02.806 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.055097+0000", 2024-03-20T09:46:02.806 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.2617999999999998e-05 2024-03-20T09:46:02.806 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.806 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.806 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:02.806 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.055110+0000", 2024-03-20T09:46:02.806 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3117e-05 2024-03-20T09:46:02.806 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.806 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.806 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:02.806 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.055116+0000", 2024-03-20T09:46:02.807 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4570000000000002e-06 2024-03-20T09:46:02.807 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.807 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.807 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:02.807 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.055321+0000", 2024-03-20T09:46:02.807 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000204231 2024-03-20T09:46:02.807 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.807 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.807 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:02.807 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.055340+0000", 2024-03-20T09:46:02.807 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8856999999999999e-05 2024-03-20T09:46:02.807 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.807 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.808 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:02.808 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124360+0000", 2024-03-20T09:46:02.808 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.069020916000000002 2024-03-20T09:46:02.808 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.808 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.808 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:02.808 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124386+0000", 2024-03-20T09:46:02.808 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5239e-05 2024-03-20T09:46:02.808 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.808 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:02.808 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.808 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.809 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.809 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:85328 8.d e998/974 8:b7d49900:::benchmark_data_smithi122_197775_object85327:head v 998'5367, mlcod=998'5360)", 2024-03-20T09:46:02.809 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:50.058952+0000", 2024-03-20T09:46:02.809 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 552.70731322999995, 2024-03-20T09:46:02.809 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.065465018, 2024-03-20T09:46:02.809 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:02.809 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:02.809 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:02.809 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.809 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:02.809 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.058952+0000", 2024-03-20T09:46:02.809 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.809 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.810 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.810 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:02.810 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.058952+0000", 2024-03-20T09:46:02.810 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.810 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.810 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.810 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:02.810 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.058953+0000", 2024-03-20T09:46:02.810 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0690000000000001e-06 2024-03-20T09:46:02.810 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.810 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.811 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:02.811 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.058989+0000", 2024-03-20T09:46:02.811 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6922000000000003e-05 2024-03-20T09:46:02.811 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.811 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.811 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:02.811 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.059003+0000", 2024-03-20T09:46:02.811 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3787e-05 2024-03-20T09:46:02.811 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.811 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.811 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:02.811 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.059011+0000", 2024-03-20T09:46:02.812 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.9330000000000001e-06 2024-03-20T09:46:02.812 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.812 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.812 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:02.812 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.059083+0000", 2024-03-20T09:46:02.812 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.2071999999999997e-05 2024-03-20T09:46:02.812 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.812 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.812 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:02.812 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.059105+0000", 2024-03-20T09:46:02.812 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1699999999999999e-05 2024-03-20T09:46:02.812 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.812 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.813 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:02.813 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124388+0000", 2024-03-20T09:46:02.813 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.065282824000000003 2024-03-20T09:46:02.813 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.813 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.813 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:02.813 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124417+0000", 2024-03-20T09:46:02.813 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8711e-05 2024-03-20T09:46:02.813 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.813 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:02.813 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.814 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.814 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.814 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.26917.0:85334 8.d e998/974 8:b8549932:::benchmark_data_smithi122_197775_object85333:head v 998'5368, mlcod=998'5360)", 2024-03-20T09:46:02.814 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:36:50.062866+0000", 2024-03-20T09:46:02.814 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 552.70339848499998, 2024-03-20T09:46:02.814 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.061575585000000002, 2024-03-20T09:46:02.814 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:02.814 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:02.814 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:02.814 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.814 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:02.814 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.062866+0000", 2024-03-20T09:46:02.815 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.815 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.815 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.815 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:02.815 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.062866+0000", 2024-03-20T09:46:02.815 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.815 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.815 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.815 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:02.815 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.062867+0000", 2024-03-20T09:46:02.815 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0890000000000001e-06 2024-03-20T09:46:02.815 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.816 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.816 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:02.816 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.062972+0000", 2024-03-20T09:46:02.816 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000104344 2024-03-20T09:46:02.816 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.816 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.816 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:02.816 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.062987+0000", 2024-03-20T09:46:02.816 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5738000000000001e-05 2024-03-20T09:46:02.816 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.816 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.816 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:02.817 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.062997+0000", 2024-03-20T09:46:02.817 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.6129999999999993e-06 2024-03-20T09:46:02.817 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.817 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.817 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:02.817 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.063058+0000", 2024-03-20T09:46:02.817 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1370000000000004e-05 2024-03-20T09:46:02.817 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.817 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.817 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:02.817 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.063085+0000", 2024-03-20T09:46:02.817 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6676999999999999e-05 2024-03-20T09:46:02.818 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.818 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.818 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:02.818 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124419+0000", 2024-03-20T09:46:02.818 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.061333490999999997 2024-03-20T09:46:02.818 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.818 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.818 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:02.818 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:36:50.124442+0000", 2024-03-20T09:46:02.818 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3263000000000001e-05 2024-03-20T09:46:02.818 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.818 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:02.819 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.819 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.819 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.819 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.27190.0:216804 9.d e1131/1066 9:b9b9f334:::benchmark_data_smithi122_210164_object84574:head v 1131'13515, mlcod=1131'13513)", 2024-03-20T09:46:02.819 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:43:46.306690+0000", 2024-03-20T09:46:02.819 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 136.45957455300001, 2024-03-20T09:46:02.819 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.132920178, 2024-03-20T09:46:02.819 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:02.819 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:02.819 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:02.819 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.819 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:02.820 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.306690+0000", 2024-03-20T09:46:02.820 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.820 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.820 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.820 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:02.820 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.306690+0000", 2024-03-20T09:46:02.820 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.820 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.820 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.820 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:02.820 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.306691+0000", 2024-03-20T09:46:02.820 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0139999999999999e-06 2024-03-20T09:46:02.821 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.821 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.821 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:02.821 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.306700+0000", 2024-03-20T09:46:02.821 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.9700000000000005e-06 2024-03-20T09:46:02.821 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.821 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.821 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:02.821 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.306718+0000", 2024-03-20T09:46:02.821 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7425e-05 2024-03-20T09:46:02.821 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.821 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.821 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:02.822 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.306727+0000", 2024-03-20T09:46:02.822 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.1600000000000004e-06 2024-03-20T09:46:02.822 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.822 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.822 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:02.822 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.306783+0000", 2024-03-20T09:46:02.822 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6164e-05 2024-03-20T09:46:02.822 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.822 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.822 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:02.822 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.306806+0000", 2024-03-20T09:46:02.822 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.2901999999999999e-05 2024-03-20T09:46:02.823 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.823 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.823 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:02.823 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.439581+0000", 2024-03-20T09:46:02.823 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.13277550399999999 2024-03-20T09:46:02.823 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.823 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.823 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:02.823 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.439610+0000", 2024-03-20T09:46:02.823 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.9039000000000001e-05 2024-03-20T09:46:02.823 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.823 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:02.823 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.824 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.824 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.824 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.27190.0:216815 9.5 e1131/1066 9:a52a349a:::benchmark_data_smithi122_210164_object84585:head v 1131'13748, mlcod=1131'13746)", 2024-03-20T09:46:02.824 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:43:46.307579+0000", 2024-03-20T09:46:02.824 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 136.45868567299999, 2024-03-20T09:46:02.824 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.132058336, 2024-03-20T09:46:02.824 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:02.824 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:02.824 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:02.824 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.824 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:02.824 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.307579+0000", 2024-03-20T09:46:02.825 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.825 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.825 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.825 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:02.825 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.307579+0000", 2024-03-20T09:46:02.825 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.825 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.825 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.825 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:02.825 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.307580+0000", 2024-03-20T09:46:02.826 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.2099999999999996e-07 2024-03-20T09:46:02.826 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.826 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.826 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:02.826 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.307586+0000", 2024-03-20T09:46:02.826 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8390000000000003e-06 2024-03-20T09:46:02.826 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.826 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.826 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:02.826 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.307598+0000", 2024-03-20T09:46:02.826 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2401e-05 2024-03-20T09:46:02.826 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.827 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.827 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:02.827 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.307605+0000", 2024-03-20T09:46:02.827 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.1940000000000001e-06 2024-03-20T09:46:02.827 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.827 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.827 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:02.827 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.311887+0000", 2024-03-20T09:46:02.827 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.004281338 2024-03-20T09:46:02.827 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.827 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.827 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:02.828 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.311911+0000", 2024-03-20T09:46:02.828 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.4187999999999999e-05 2024-03-20T09:46:02.828 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.828 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.828 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:02.828 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.439613+0000", 2024-03-20T09:46:02.828 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.12770269200000001 2024-03-20T09:46:02.828 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.828 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.828 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:02.828 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.439637+0000", 2024-03-20T09:46:02.828 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3963000000000001e-05 2024-03-20T09:46:02.829 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.829 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:02.829 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.829 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.829 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.829 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.27190.0:216819 9.d e1131/1066 9:bb175bef:::benchmark_data_smithi122_210164_object84589:head v 1131'13516, mlcod=1131'13513)", 2024-03-20T09:46:02.829 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:43:46.312878+0000", 2024-03-20T09:46:02.829 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 136.45338631999999, 2024-03-20T09:46:02.829 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.12678308499999999, 2024-03-20T09:46:02.829 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:02.829 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:02.829 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:02.830 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.830 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:02.830 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.312878+0000", 2024-03-20T09:46:02.830 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.830 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.830 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.830 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:02.830 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.312878+0000", 2024-03-20T09:46:02.830 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.830 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.830 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.831 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:02.831 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.312879+0000", 2024-03-20T09:46:02.831 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.2999999999999999e-07 2024-03-20T09:46:02.831 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.831 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.831 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:02.831 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.312886+0000", 2024-03-20T09:46:02.831 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.1969999999999999e-06 2024-03-20T09:46:02.831 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.831 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.831 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:02.831 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.312897+0000", 2024-03-20T09:46:02.832 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0392e-05 2024-03-20T09:46:02.832 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.832 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.832 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:02.832 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.312904+0000", 2024-03-20T09:46:02.832 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.7789999999999998e-06 2024-03-20T09:46:02.832 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.832 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.832 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:02.832 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.316916+0000", 2024-03-20T09:46:02.832 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.004012407 2024-03-20T09:46:02.832 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.833 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.833 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:02.833 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.316935+0000", 2024-03-20T09:46:02.833 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8828999999999999e-05 2024-03-20T09:46:02.833 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.833 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.833 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:02.833 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.439640+0000", 2024-03-20T09:46:02.833 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.12270466100000001 2024-03-20T09:46:02.833 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.833 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.833 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:02.834 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.439661+0000", 2024-03-20T09:46:02.834 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1990000000000001e-05 2024-03-20T09:46:02.834 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.834 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:02.834 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.834 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.834 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.834 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.27190.0:216839 9.5 e1131/1066 9:a8a5bcdc:::benchmark_data_smithi122_210164_object84609:head v 1131'13749, mlcod=1131'13746)", 2024-03-20T09:46:02.834 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:43:46.343298+0000", 2024-03-20T09:46:02.834 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 136.422966414, 2024-03-20T09:46:02.834 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.096380757999999997, 2024-03-20T09:46:02.835 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:02.835 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:02.835 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:02.835 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.835 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:02.835 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.343298+0000", 2024-03-20T09:46:02.835 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.835 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.835 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.835 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:02.835 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.343298+0000", 2024-03-20T09:46:02.835 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.835 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.835 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.836 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:02.836 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.343299+0000", 2024-03-20T09:46:02.836 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.0699999999999998e-07 2024-03-20T09:46:02.836 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.836 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.836 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:02.836 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.343302+0000", 2024-03-20T09:46:02.836 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7479999999999999e-06 2024-03-20T09:46:02.836 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.836 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.836 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:02.836 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.343308+0000", 2024-03-20T09:46:02.836 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5389999999999998e-06 2024-03-20T09:46:02.836 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.837 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.837 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:02.837 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.343314+0000", 2024-03-20T09:46:02.837 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.711e-06 2024-03-20T09:46:02.837 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.837 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.837 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:02.837 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.343353+0000", 2024-03-20T09:46:02.837 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.9295999999999998e-05 2024-03-20T09:46:02.837 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.837 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.837 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:02.837 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.343372+0000", 2024-03-20T09:46:02.838 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.9079000000000001e-05 2024-03-20T09:46:02.838 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.838 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.838 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:02.838 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.439663+0000", 2024-03-20T09:46:02.838 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.096291363000000005 2024-03-20T09:46:02.838 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.838 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.838 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:02.838 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.439679+0000", 2024-03-20T09:46:02.838 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5715000000000001e-05 2024-03-20T09:46:02.839 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.839 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:02.839 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.839 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.839 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.839 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.27190.0:216858 9.5 e1131/1066 9:ab0882a1:::benchmark_data_smithi122_210164_object84628:head v 1131'13750, mlcod=1131'13746)", 2024-03-20T09:46:02.839 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:43:46.367236+0000", 2024-03-20T09:46:02.839 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 136.39902915499999, 2024-03-20T09:46:02.839 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.072464864000000004, 2024-03-20T09:46:02.839 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:02.839 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:02.839 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:02.839 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.840 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:02.840 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.367236+0000", 2024-03-20T09:46:02.840 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.840 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.840 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.840 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:02.840 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.367236+0000", 2024-03-20T09:46:02.840 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.840 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.840 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.840 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:02.840 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.367237+0000", 2024-03-20T09:46:02.841 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.054e-06 2024-03-20T09:46:02.841 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.841 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.841 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:02.841 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.367245+0000", 2024-03-20T09:46:02.841 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.6880000000000005e-06 2024-03-20T09:46:02.841 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.841 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.841 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:02.841 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.367259+0000", 2024-03-20T09:46:02.841 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3232e-05 2024-03-20T09:46:02.841 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.842 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.842 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:02.842 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.367267+0000", 2024-03-20T09:46:02.842 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.3869999999999995e-06 2024-03-20T09:46:02.842 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.842 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.842 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:02.842 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.367322+0000", 2024-03-20T09:46:02.842 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4868999999999999e-05 2024-03-20T09:46:02.842 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.842 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.843 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:02.843 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.367342+0000", 2024-03-20T09:46:02.843 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0169e-05 2024-03-20T09:46:02.843 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.843 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.843 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:02.843 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.439681+0000", 2024-03-20T09:46:02.843 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.072338685 2024-03-20T09:46:02.843 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.843 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.843 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:02.843 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.439700+0000", 2024-03-20T09:46:02.844 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.978e-05 2024-03-20T09:46:02.844 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.844 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:02.844 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.844 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.844 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.844 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.27190.0:216860 9.5 e1131/1066 9:a07aeb6a:::benchmark_data_smithi122_210164_object84630:head v 1131'13751, mlcod=1131'13746)", 2024-03-20T09:46:02.844 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:43:46.375500+0000", 2024-03-20T09:46:02.844 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 136.39076445800001, 2024-03-20T09:46:02.844 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.064216250000000002, 2024-03-20T09:46:02.844 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:02.844 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:02.845 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:02.845 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.845 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:02.845 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.375500+0000", 2024-03-20T09:46:02.845 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.845 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.845 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.845 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:02.845 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.375500+0000", 2024-03-20T09:46:02.845 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.845 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.845 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.845 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:02.846 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.375501+0000", 2024-03-20T09:46:02.846 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.0999999999999999e-07 2024-03-20T09:46:02.846 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.846 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.846 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:02.846 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.375508+0000", 2024-03-20T09:46:02.846 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.4259999999999997e-06 2024-03-20T09:46:02.846 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.846 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.846 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:02.846 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.375515+0000", 2024-03-20T09:46:02.847 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.9999999999999999e-06 2024-03-20T09:46:02.847 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.847 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.847 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:02.847 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.375521+0000", 2024-03-20T09:46:02.847 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7819999999999999e-06 2024-03-20T09:46:02.847 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.847 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.847 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:02.847 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.375570+0000", 2024-03-20T09:46:02.847 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8931000000000003e-05 2024-03-20T09:46:02.847 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.847 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.848 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:02.848 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.375591+0000", 2024-03-20T09:46:02.848 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.1087999999999999e-05 2024-03-20T09:46:02.848 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.848 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.848 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:02.848 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.439702+0000", 2024-03-20T09:46:02.848 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.064111380999999995 2024-03-20T09:46:02.848 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.848 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.848 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:02.848 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.439717+0000", 2024-03-20T09:46:02.848 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4232e-05 2024-03-20T09:46:02.849 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.849 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:02.849 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.849 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.849 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.849 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_repop(client.27190.0:216866 9.5 e1131/1066 9:ac7f8f44:::benchmark_data_smithi122_210164_object84636:head v 1131'13752, mlcod=1131'13746)", 2024-03-20T09:46:02.849 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:43:46.375533+0000", 2024-03-20T09:46:02.849 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 136.39073185999999, 2024-03-20T09:46:02.849 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.064199302, 2024-03-20T09:46:02.849 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:02.849 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:02.849 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:02.849 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.850 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:02.850 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.375533+0000", 2024-03-20T09:46:02.850 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.850 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.850 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.850 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:02.850 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.375533+0000", 2024-03-20T09:46:02.850 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.850 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.850 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.850 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:02.851 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.375533+0000", 2024-03-20T09:46:02.851 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.8799999999999998e-07 2024-03-20T09:46:02.851 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.851 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.851 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:02.851 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.375537+0000", 2024-03-20T09:46:02.851 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7809999999999999e-06 2024-03-20T09:46:02.851 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.851 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.851 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:02.851 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.375543+0000", 2024-03-20T09:46:02.851 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0329999999999997e-06 2024-03-20T09:46:02.851 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.851 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.852 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:02.852 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.375547+0000", 2024-03-20T09:46:02.852 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4150000000000001e-06 2024-03-20T09:46:02.852 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.852 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.852 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:02.852 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.380503+0000", 2024-03-20T09:46:02.852 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0049557899999999998 2024-03-20T09:46:02.852 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.852 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.852 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:02.852 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.380527+0000", 2024-03-20T09:46:02.853 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3998999999999998e-05 2024-03-20T09:46:02.853 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.853 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.853 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:02.853 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.439718+0000", 2024-03-20T09:46:02.853 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.059191039000000001 2024-03-20T09:46:02.853 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.853 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.853 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:02.853 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:46.439732+0000", 2024-03-20T09:46:02.853 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3957e-05 2024-03-20T09:46:02.853 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.854 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:02.854 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.854 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.854 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.854 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "rep_scrubmap(1.15 e491 from shard 5)", 2024-03-20T09:46:02.854 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:45:02.903307+0000", 2024-03-20T09:46:02.854 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 59.862958190000001, 2024-03-20T09:46:02.854 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.055610885999999998, 2024-03-20T09:46:02.854 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:02.854 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "started", 2024-03-20T09:46:02.854 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:02.854 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.855 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:02.855 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:02.903307+0000", 2024-03-20T09:46:02.855 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.855 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.855 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.855 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:02.855 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:02.903307+0000", 2024-03-20T09:46:02.855 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:02.855 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.855 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.855 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:02.855 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:02.903308+0000", 2024-03-20T09:46:02.856 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0300000000000001e-06 2024-03-20T09:46:02.856 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.856 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.856 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:02.856 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:02.903315+0000", 2024-03-20T09:46:02.856 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.2629999999999997e-06 2024-03-20T09:46:02.856 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.856 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.856 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:02.856 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:02.903327+0000", 2024-03-20T09:46:02.856 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2506e-05 2024-03-20T09:46:02.856 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.856 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.857 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:02.857 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:02.903336+0000", 2024-03-20T09:46:02.857 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5990000000000002e-06 2024-03-20T09:46:02.857 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.857 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.857 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:02.857 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:02.958885+0000", 2024-03-20T09:46:02.857 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.055549346999999999 2024-03-20T09:46:02.857 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.857 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.857 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:02.857 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:02.958899+0000", 2024-03-20T09:46:02.857 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3862e-05 2024-03-20T09:46:02.858 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:02.858 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:02.858 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:02.858 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:45:02.958917+0000", 2024-03-20T09:46:02.858 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8278999999999999e-05 2024-03-20T09:46:02.858 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.858 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:02.858 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.858 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:02.858 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:02.858 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:46:03.119 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:02 smithi115 ceph-mon[72476]: 1.1e scrub starts 2024-03-20T09:46:03.120 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:02 smithi115 ceph-mon[72476]: 1.1e scrub ok 2024-03-20T09:46:03.120 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:02 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:46:03.120 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:02 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:46:03.120 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:02 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:46:03.120 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:02 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:46:03.120 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:02 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:46:03.120 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:02 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:46:03.120 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:02 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:46:03.120 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:02 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:46:03.120 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:02 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:46:03.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:02 smithi122 ceph-mon[73495]: 1.1e scrub starts 2024-03-20T09:46:03.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:02 smithi122 ceph-mon[73495]: 1.1e scrub ok 2024-03-20T09:46:03.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:02 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:46:03.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:02 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:46:03.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:02 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:46:03.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:02 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:46:03.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:02 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:46:03.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:02 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:46:03.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:02 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:46:03.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:02 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:46:03.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:02 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:46:03.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:02 smithi026 ceph-mon[68309]: 1.1e scrub starts 2024-03-20T09:46:03.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:02 smithi026 ceph-mon[68309]: 1.1e scrub ok 2024-03-20T09:46:03.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:02 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:46:03.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:02 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:46:03.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:02 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:46:03.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:02 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:46:03.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:02 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:46:03.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:02 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:46:03.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:02 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:46:03.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:02 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:46:03.208 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:02 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:46:03.426 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_ops_in_flight 2024-03-20T09:46:03.694 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:46:04.010 INFO:teuthology.orchestra.run.smithi026.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-20T09:46:04.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:03 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:46:04.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:03 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:46:04.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:03 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:46:04.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:03 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:46:04.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:03 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/659962763' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-20T09:46:04.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:03 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:46:04.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:03 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:46:04.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:03 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:46:04.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:03 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:46:04.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:03 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/659962763' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-20T09:46:04.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:03 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:46:04.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:03 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:46:04.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:03 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:46:04.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:03 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:46:04.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:03 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/659962763' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-20T09:46:04.688 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 7 1 2024-03-20T09:46:04.969 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:46:05.319 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:05 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/659962763' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-20T09:46:05.319 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:05 smithi122 ceph-mon[73495]: osdmap e1176: 12 total, 12 up, 12 in 2024-03-20T09:46:05.319 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:05 smithi122 ceph-mon[73495]: pgmap v2860: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:05.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:05 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/659962763' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-20T09:46:05.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:05 smithi115 ceph-mon[72476]: osdmap e1176: 12 total, 12 up, 12 in 2024-03-20T09:46:05.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:05 smithi115 ceph-mon[72476]: pgmap v2860: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:05.454 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:05 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/659962763' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-20T09:46:05.454 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:05 smithi026 ceph-mon[68309]: osdmap e1176: 12 total, 12 up, 12 in 2024-03-20T09:46:05.454 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:05 smithi026 ceph-mon[68309]: pgmap v2860: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:05.642 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:46:05.642 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:46:05.642 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-03-20T09:46:05.642 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:46:06.312 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_blocked_ops 2024-03-20T09:46:06.575 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:46:07.781 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.7 to 1 (10000) 2024-03-20T09:46:07.878 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:07 smithi122 ceph-mon[73495]: pgmap v2861: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:07.878 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:07 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/26781369' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:46:08.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:07 smithi115 ceph-mon[72476]: pgmap v2861: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:08.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:07 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/26781369' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:46:08.147 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:07 smithi026 ceph-mon[68309]: pgmap v2861: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:08.147 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:07 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/26781369' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:46:08.446 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:46:08.446 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:46:08.446 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-03-20T09:46:08.446 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-03-20T09:46:08.446 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:46:08.564 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd primary-affinity 7 1 2024-03-20T09:46:08.849 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:46:08.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:08 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/26781369' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-20T09:46:08.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:08 smithi026 ceph-mon[68309]: osdmap e1177: 12 total, 12 up, 12 in 2024-03-20T09:46:09.111 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:08 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/26781369' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-20T09:46:09.111 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:08 smithi122 ceph-mon[73495]: osdmap e1177: 12 total, 12 up, 12 in 2024-03-20T09:46:09.137 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.8 dump_historic_ops 2024-03-20T09:46:09.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:08 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/26781369' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-20T09:46:09.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:08 smithi115 ceph-mon[72476]: osdmap e1177: 12 total, 12 up, 12 in 2024-03-20T09:46:09.399 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:46:09.933 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:09 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:46:09.933 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:09 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:46:09.933 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:09 smithi122 ceph-mon[73495]: 1.5 scrub starts 2024-03-20T09:46:09.933 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:09 smithi122 ceph-mon[73495]: 1.5 scrub ok 2024-03-20T09:46:09.933 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:09 smithi122 ceph-mon[73495]: pgmap v2863: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:09.933 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:09 smithi122 ceph-mon[73495]: osdmap e1178: 12 total, 12 up, 12 in 2024-03-20T09:46:10.044 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:09 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:46:10.045 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:09 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:46:10.045 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:09 smithi026 ceph-mon[68309]: 1.5 scrub starts 2024-03-20T09:46:10.045 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:09 smithi026 ceph-mon[68309]: 1.5 scrub ok 2024-03-20T09:46:10.045 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:09 smithi026 ceph-mon[68309]: pgmap v2863: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:10.045 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:09 smithi026 ceph-mon[68309]: osdmap e1178: 12 total, 12 up, 12 in 2024-03-20T09:46:10.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:09 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:46:10.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:09 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:46:10.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:09 smithi115 ceph-mon[72476]: 1.5 scrub starts 2024-03-20T09:46:10.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:09 smithi115 ceph-mon[72476]: 1.5 scrub ok 2024-03-20T09:46:10.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:09 smithi115 ceph-mon[72476]: pgmap v2863: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:10.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:09 smithi115 ceph-mon[72476]: osdmap e1178: 12 total, 12 up, 12 in 2024-03-20T09:46:10.900 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:10 smithi122 ceph-mon[73495]: 1.d scrub starts 2024-03-20T09:46:10.900 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:10 smithi122 ceph-mon[73495]: 1.d scrub ok 2024-03-20T09:46:11.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:10 smithi115 ceph-mon[72476]: 1.d scrub starts 2024-03-20T09:46:11.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:10 smithi115 ceph-mon[72476]: 1.d scrub ok 2024-03-20T09:46:11.181 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:46:11.181 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-03-20T09:46:11.181 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-03-20T09:46:11.181 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-03-20T09:46:11.181 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.181 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:61791 8.a 8:554604c2:::benchmark_data_smithi122_197775_object61790:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e991)", 2024-03-20T09:46:11.181 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:36:31.920764+0000", 2024-03-20T09:46:11.181 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 579.27125619900005, 2024-03-20T09:46:11.182 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20530047300000001, 2024-03-20T09:46:11.182 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:11.182 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:11.182 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:11.182 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:11.182 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:11.182 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 61791 2024-03-20T09:46:11.182 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.182 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:11.182 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.182 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:11.183 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.920764+0000", 2024-03-20T09:46:11.183 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.183 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.183 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.183 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:11.183 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.920764+0000", 2024-03-20T09:46:11.183 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.183 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.183 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.183 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:11.183 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.920764+0000", 2024-03-20T09:46:11.183 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0699999999999997e-07 2024-03-20T09:46:11.184 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.184 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.184 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:11.184 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.920788+0000", 2024-03-20T09:46:11.184 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3564999999999999e-05 2024-03-20T09:46:11.184 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.184 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.184 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:11.184 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.920795+0000", 2024-03-20T09:46:11.184 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6590000000000007e-06 2024-03-20T09:46:11.184 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.184 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.185 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:11.185 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.920802+0000", 2024-03-20T09:46:11.185 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.404e-06 2024-03-20T09:46:11.185 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.185 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.185 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:11.185 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.920946+0000", 2024-03-20T09:46:11.185 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014437 2024-03-20T09:46:11.185 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.185 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.185 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:11.186 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.921068+0000", 2024-03-20T09:46:11.186 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012204700000000001 2024-03-20T09:46:11.186 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.186 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.186 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2,9", 2024-03-20T09:46:11.186 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.921209+0000", 2024-03-20T09:46:11.186 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000141149 2024-03-20T09:46:11.186 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.186 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.186 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:11.186 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.925562+0000", 2024-03-20T09:46:11.186 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0043527749999999997 2024-03-20T09:46:11.186 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.187 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.187 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:11.187 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.925735+0000", 2024-03-20T09:46:11.187 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00017325500000000001 2024-03-20T09:46:11.187 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.187 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.187 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:11.187 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.125983+0000", 2024-03-20T09:46:11.187 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.200247807 2024-03-20T09:46:11.187 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.187 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.188 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:11.188 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.126022+0000", 2024-03-20T09:46:11.188 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9085999999999998e-05 2024-03-20T09:46:11.188 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.188 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.188 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:11.188 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.126064+0000", 2024-03-20T09:46:11.188 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1848999999999999e-05 2024-03-20T09:46:11.188 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.188 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:11.188 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.188 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.188 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.189 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:61803 8.a 8:5bfa8527:::benchmark_data_smithi122_197775_object61802:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e991)", 2024-03-20T09:46:11.189 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:36:31.928329+0000", 2024-03-20T09:46:11.189 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 579.26369097300005, 2024-03-20T09:46:11.189 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20050190900000001, 2024-03-20T09:46:11.189 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:11.189 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:11.189 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:11.189 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:11.189 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:11.189 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 61803 2024-03-20T09:46:11.189 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.190 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:11.190 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.190 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:11.190 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.928329+0000", 2024-03-20T09:46:11.190 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.190 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.190 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.190 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:11.190 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.928329+0000", 2024-03-20T09:46:11.190 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.191 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.191 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.191 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:11.191 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.928330+0000", 2024-03-20T09:46:11.191 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4339999999999999e-06 2024-03-20T09:46:11.191 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.191 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.191 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:11.191 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.928356+0000", 2024-03-20T09:46:11.191 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5955000000000001e-05 2024-03-20T09:46:11.191 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.191 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.192 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:11.192 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.928371+0000", 2024-03-20T09:46:11.192 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4470999999999999e-05 2024-03-20T09:46:11.192 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.192 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.192 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:11.192 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.928379+0000", 2024-03-20T09:46:11.192 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8830000000000006e-06 2024-03-20T09:46:11.192 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.192 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.192 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:11.192 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.928433+0000", 2024-03-20T09:46:11.193 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4060000000000001e-05 2024-03-20T09:46:11.193 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.193 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.193 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:11.193 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.928542+0000", 2024-03-20T09:46:11.193 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010909299999999999 2024-03-20T09:46:11.193 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.193 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.193 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2,9", 2024-03-20T09:46:11.193 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.928661+0000", 2024-03-20T09:46:11.193 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011899499999999999 2024-03-20T09:46:11.193 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.193 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.194 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:11.194 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.930652+0000", 2024-03-20T09:46:11.194 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0019911270000000001 2024-03-20T09:46:11.194 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.194 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.194 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:11.194 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.931446+0000", 2024-03-20T09:46:11.194 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00079412999999999999 2024-03-20T09:46:11.194 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.194 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.194 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:11.194 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.128753+0000", 2024-03-20T09:46:11.194 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19730726100000001 2024-03-20T09:46:11.195 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.195 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.195 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:11.195 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.128791+0000", 2024-03-20T09:46:11.195 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7886000000000003e-05 2024-03-20T09:46:11.195 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.195 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.195 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:11.195 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.128831+0000", 2024-03-20T09:46:11.195 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9613999999999997e-05 2024-03-20T09:46:11.195 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.195 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:11.195 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.196 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.196 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.196 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:61814 8.a 8:591b158a:::benchmark_data_smithi122_197775_object61813:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e991)", 2024-03-20T09:46:11.196 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:36:31.936182+0000", 2024-03-20T09:46:11.196 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 579.25583830999994, 2024-03-20T09:46:11.196 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.194264509, 2024-03-20T09:46:11.196 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:11.196 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:11.196 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:11.196 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:11.197 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:11.197 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 61814 2024-03-20T09:46:11.197 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.197 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:11.197 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.197 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:11.197 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.936182+0000", 2024-03-20T09:46:11.197 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.197 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.197 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.197 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:11.197 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.936182+0000", 2024-03-20T09:46:11.197 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.198 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.198 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.198 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:11.198 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.936183+0000", 2024-03-20T09:46:11.198 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.195e-06 2024-03-20T09:46:11.198 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.198 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.198 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:11.198 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.936206+0000", 2024-03-20T09:46:11.198 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2926999999999999e-05 2024-03-20T09:46:11.199 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.199 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.199 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:11.199 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.936222+0000", 2024-03-20T09:46:11.199 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5756999999999998e-05 2024-03-20T09:46:11.199 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.199 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.199 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:11.199 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.936230+0000", 2024-03-20T09:46:11.199 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1710000000000002e-06 2024-03-20T09:46:11.199 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.199 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.199 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:11.199 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.936293+0000", 2024-03-20T09:46:11.200 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3491000000000006e-05 2024-03-20T09:46:11.200 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.200 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.200 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:11.200 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.936434+0000", 2024-03-20T09:46:11.200 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014070400000000001 2024-03-20T09:46:11.200 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.200 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.200 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2,9", 2024-03-20T09:46:11.200 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.936574+0000", 2024-03-20T09:46:11.200 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014003099999999999 2024-03-20T09:46:11.200 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.200 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.201 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:11.201 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.938922+0000", 2024-03-20T09:46:11.201 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0023478980000000002 2024-03-20T09:46:11.201 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.201 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.201 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:11.201 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.939331+0000", 2024-03-20T09:46:11.201 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000409398 2024-03-20T09:46:11.201 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.201 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.201 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:11.201 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.130327+0000", 2024-03-20T09:46:11.201 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19099628199999999 2024-03-20T09:46:11.201 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.202 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.202 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:11.202 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.130391+0000", 2024-03-20T09:46:11.202 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3256000000000006e-05 2024-03-20T09:46:11.202 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.202 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.202 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:11.202 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.130446+0000", 2024-03-20T09:46:11.202 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5399000000000003e-05 2024-03-20T09:46:11.202 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.202 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:11.202 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.203 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.203 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.203 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:61826 8.a 8:5e6bf1b8:::benchmark_data_smithi122_197775_object61825:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e991)", 2024-03-20T09:46:11.203 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:36:31.944467+0000", 2024-03-20T09:46:11.203 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 579.24755257899994, 2024-03-20T09:46:11.203 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18614180999999999, 2024-03-20T09:46:11.203 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:11.203 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:11.203 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:11.203 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:11.203 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:11.203 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 61826 2024-03-20T09:46:11.204 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.204 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:11.204 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.204 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:11.204 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.944467+0000", 2024-03-20T09:46:11.204 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.204 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.204 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.204 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:11.204 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.944467+0000", 2024-03-20T09:46:11.204 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.204 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.204 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.205 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:11.205 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.944469+0000", 2024-03-20T09:46:11.205 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.198e-06 2024-03-20T09:46:11.205 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.205 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.205 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:11.205 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.944489+0000", 2024-03-20T09:46:11.205 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0421999999999999e-05 2024-03-20T09:46:11.205 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.205 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.205 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:11.205 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.944501+0000", 2024-03-20T09:46:11.205 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2228999999999999e-05 2024-03-20T09:46:11.206 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.206 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.206 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:11.206 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.944510+0000", 2024-03-20T09:46:11.206 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6279999999999994e-06 2024-03-20T09:46:11.206 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.206 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.206 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:11.206 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.944591+0000", 2024-03-20T09:46:11.206 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1007000000000006e-05 2024-03-20T09:46:11.206 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.206 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.206 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:11.206 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.944740+0000", 2024-03-20T09:46:11.207 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014940799999999999 2024-03-20T09:46:11.207 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.207 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.207 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2,9", 2024-03-20T09:46:11.207 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.944873+0000", 2024-03-20T09:46:11.207 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013238799999999999 2024-03-20T09:46:11.207 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.207 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.207 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:11.207 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.949407+0000", 2024-03-20T09:46:11.207 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0045346179999999998 2024-03-20T09:46:11.208 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.208 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.208 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:11.208 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.951576+0000", 2024-03-20T09:46:11.208 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0021686919999999998 2024-03-20T09:46:11.208 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.208 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.208 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:11.208 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.130503+0000", 2024-03-20T09:46:11.208 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.178927112 2024-03-20T09:46:11.208 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.208 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.208 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:11.208 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.130566+0000", 2024-03-20T09:46:11.209 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2624000000000002e-05 2024-03-20T09:46:11.209 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.209 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.209 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:11.209 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.130609+0000", 2024-03-20T09:46:11.209 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3483999999999999e-05 2024-03-20T09:46:11.209 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.209 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:11.209 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.209 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.209 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.209 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:61833 8.a 8:5a0e5f12:::benchmark_data_smithi122_197775_object61832:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e991)", 2024-03-20T09:46:11.209 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:36:31.946897+0000", 2024-03-20T09:46:11.210 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 579.24512264199996, 2024-03-20T09:46:11.210 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18616570900000001, 2024-03-20T09:46:11.210 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:11.210 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:11.210 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:11.210 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:11.210 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:11.210 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 61833 2024-03-20T09:46:11.210 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.210 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:11.210 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.210 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:11.210 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.946897+0000", 2024-03-20T09:46:11.211 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.211 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.211 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.211 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:11.211 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.946897+0000", 2024-03-20T09:46:11.211 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.211 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.211 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.211 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:11.211 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.946898+0000", 2024-03-20T09:46:11.211 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0160000000000001e-06 2024-03-20T09:46:11.211 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.212 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.212 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:11.212 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.946919+0000", 2024-03-20T09:46:11.212 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1132999999999999e-05 2024-03-20T09:46:11.212 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.212 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.212 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:11.212 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.946927+0000", 2024-03-20T09:46:11.212 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8019999999999992e-06 2024-03-20T09:46:11.212 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.212 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.212 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:11.212 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.946932+0000", 2024-03-20T09:46:11.213 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2340000000000002e-06 2024-03-20T09:46:11.213 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.213 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.213 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:11.213 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.947044+0000", 2024-03-20T09:46:11.213 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000111922 2024-03-20T09:46:11.213 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.213 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.213 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:11.213 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.947179+0000", 2024-03-20T09:46:11.213 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013415700000000001 2024-03-20T09:46:11.214 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.214 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.214 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2,9", 2024-03-20T09:46:11.214 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.947312+0000", 2024-03-20T09:46:11.214 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013350800000000001 2024-03-20T09:46:11.214 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.214 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.214 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:11.214 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.949526+0000", 2024-03-20T09:46:11.214 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0022135990000000001 2024-03-20T09:46:11.214 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.215 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.215 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:11.215 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.951677+0000", 2024-03-20T09:46:11.215 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0021511159999999998 2024-03-20T09:46:11.215 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.215 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.215 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:11.215 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.132971+0000", 2024-03-20T09:46:11.215 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.18129455899999999 2024-03-20T09:46:11.215 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.215 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.215 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:11.215 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.133025+0000", 2024-03-20T09:46:11.216 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3995000000000001e-05 2024-03-20T09:46:11.216 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.216 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.216 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:11.216 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.133063+0000", 2024-03-20T09:46:11.216 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7667999999999999e-05 2024-03-20T09:46:11.216 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.216 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:11.216 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.216 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.217 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.217 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:61846 8.a 8:53b4be94:::benchmark_data_smithi122_197775_object61845:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e991)", 2024-03-20T09:46:11.217 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:36:31.959308+0000", 2024-03-20T09:46:11.217 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 579.23271169300006, 2024-03-20T09:46:11.217 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.17387308800000001, 2024-03-20T09:46:11.217 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:11.217 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:11.217 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:11.217 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:11.217 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:11.217 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 61846 2024-03-20T09:46:11.217 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.217 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:11.218 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.218 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:11.218 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.959308+0000", 2024-03-20T09:46:11.218 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.218 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.218 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.218 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:11.218 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.959308+0000", 2024-03-20T09:46:11.218 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.218 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.218 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.219 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:11.219 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.959310+0000", 2024-03-20T09:46:11.219 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7290000000000001e-06 2024-03-20T09:46:11.219 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.219 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.219 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:11.219 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.959336+0000", 2024-03-20T09:46:11.219 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5704999999999999e-05 2024-03-20T09:46:11.219 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.219 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.219 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:11.219 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.959352+0000", 2024-03-20T09:46:11.219 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6205000000000002e-05 2024-03-20T09:46:11.220 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.220 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.220 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:11.220 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.959360+0000", 2024-03-20T09:46:11.220 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9049999999999997e-06 2024-03-20T09:46:11.220 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.220 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.220 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:11.220 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.959410+0000", 2024-03-20T09:46:11.220 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0235000000000003e-05 2024-03-20T09:46:11.220 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.220 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.221 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:11.221 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.959527+0000", 2024-03-20T09:46:11.221 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000117137 2024-03-20T09:46:11.221 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.221 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.221 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2,9", 2024-03-20T09:46:11.221 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.959636+0000", 2024-03-20T09:46:11.221 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010852400000000001 2024-03-20T09:46:11.221 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.221 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.221 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:11.221 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.961492+0000", 2024-03-20T09:46:11.221 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001856016 2024-03-20T09:46:11.221 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.222 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.222 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:11.222 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.961993+0000", 2024-03-20T09:46:11.222 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00050144899999999995 2024-03-20T09:46:11.222 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.222 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.222 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:11.222 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.133109+0000", 2024-03-20T09:46:11.222 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.17111535 2024-03-20T09:46:11.222 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.222 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.222 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:11.223 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.133142+0000", 2024-03-20T09:46:11.223 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3775000000000001e-05 2024-03-20T09:46:11.223 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.223 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.223 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:11.223 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.133181+0000", 2024-03-20T09:46:11.223 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9057999999999998e-05 2024-03-20T09:46:11.223 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.223 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:11.223 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.223 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.223 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.223 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:61860 8.a 8:5d2dce7b:::benchmark_data_smithi122_197775_object61859:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e991)", 2024-03-20T09:46:11.223 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:36:31.967164+0000", 2024-03-20T09:46:11.224 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 579.224855757, 2024-03-20T09:46:11.224 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.16871385699999999, 2024-03-20T09:46:11.224 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:11.224 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:11.224 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:11.224 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:11.224 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:11.224 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 61860 2024-03-20T09:46:11.224 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.224 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:11.224 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.224 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:11.225 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.967164+0000", 2024-03-20T09:46:11.225 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.225 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.225 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.225 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:11.225 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.967164+0000", 2024-03-20T09:46:11.225 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.225 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.225 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.225 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:11.225 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.967165+0000", 2024-03-20T09:46:11.225 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1930000000000001e-06 2024-03-20T09:46:11.226 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.226 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.226 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:11.226 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.967187+0000", 2024-03-20T09:46:11.226 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2014999999999998e-05 2024-03-20T09:46:11.226 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.226 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.226 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:11.226 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.967205+0000", 2024-03-20T09:46:11.226 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7368e-05 2024-03-20T09:46:11.226 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.226 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.226 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:11.227 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.967216+0000", 2024-03-20T09:46:11.227 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0899e-05 2024-03-20T09:46:11.227 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.227 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.227 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:11.227 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.967272+0000", 2024-03-20T09:46:11.227 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6285000000000001e-05 2024-03-20T09:46:11.227 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.227 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.227 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:11.227 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.967417+0000", 2024-03-20T09:46:11.227 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014539300000000001 2024-03-20T09:46:11.227 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.228 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.228 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2,9", 2024-03-20T09:46:11.228 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.967559+0000", 2024-03-20T09:46:11.228 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014150100000000001 2024-03-20T09:46:11.228 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.228 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.228 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:11.228 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.969574+0000", 2024-03-20T09:46:11.228 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0020152310000000001 2024-03-20T09:46:11.228 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.228 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.228 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:11.228 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.970596+0000", 2024-03-20T09:46:11.229 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001022261 2024-03-20T09:46:11.229 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.229 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.229 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:11.229 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.135771+0000", 2024-03-20T09:46:11.229 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.165174562 2024-03-20T09:46:11.229 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.229 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.229 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:11.229 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.135830+0000", 2024-03-20T09:46:11.229 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8739999999999997e-05 2024-03-20T09:46:11.229 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.229 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.230 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:11.230 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.135878+0000", 2024-03-20T09:46:11.230 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8408999999999997e-05 2024-03-20T09:46:11.230 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.230 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:11.230 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.230 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.230 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.230 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:61874 8.a 8:58788445:::benchmark_data_smithi122_197775_object61873:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e991)", 2024-03-20T09:46:11.230 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:36:31.981451+0000", 2024-03-20T09:46:11.230 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 579.21056941300003, 2024-03-20T09:46:11.231 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.15458785999999999, 2024-03-20T09:46:11.231 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:11.231 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:11.231 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:11.231 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:11.231 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:11.231 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 61874 2024-03-20T09:46:11.231 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.231 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:11.231 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.231 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:11.231 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.981451+0000", 2024-03-20T09:46:11.232 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.232 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.232 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.232 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:11.232 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.981451+0000", 2024-03-20T09:46:11.232 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.232 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.232 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.232 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:11.232 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.981452+0000", 2024-03-20T09:46:11.232 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3120000000000001e-06 2024-03-20T09:46:11.232 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.232 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.233 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:11.233 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.981474+0000", 2024-03-20T09:46:11.233 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1888000000000001e-05 2024-03-20T09:46:11.233 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.233 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.233 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:11.233 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.981488+0000", 2024-03-20T09:46:11.233 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4671000000000001e-05 2024-03-20T09:46:11.233 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.233 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.233 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:11.234 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.981497+0000", 2024-03-20T09:46:11.234 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8820000000000007e-06 2024-03-20T09:46:11.234 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.234 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.234 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:11.234 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.981553+0000", 2024-03-20T09:46:11.234 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6059e-05 2024-03-20T09:46:11.234 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.234 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.234 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:11.234 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.981670+0000", 2024-03-20T09:46:11.234 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000116821 2024-03-20T09:46:11.235 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.235 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.235 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2,9", 2024-03-20T09:46:11.235 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.981781+0000", 2024-03-20T09:46:11.235 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011034799999999999 2024-03-20T09:46:11.235 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.235 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.235 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:11.235 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.983844+0000", 2024-03-20T09:46:11.235 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0020633050000000001 2024-03-20T09:46:11.235 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.235 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.235 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:11.235 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.983891+0000", 2024-03-20T09:46:11.236 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6913999999999998e-05 2024-03-20T09:46:11.236 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.236 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.236 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:11.236 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.135954+0000", 2024-03-20T09:46:11.236 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.152063477 2024-03-20T09:46:11.236 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.236 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.236 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:11.236 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.136003+0000", 2024-03-20T09:46:11.236 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8559e-05 2024-03-20T09:46:11.236 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.236 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.237 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:11.237 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.136038+0000", 2024-03-20T09:46:11.237 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5624000000000003e-05 2024-03-20T09:46:11.237 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.237 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:11.237 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.237 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.237 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.237 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:61886 8.a 8:57f5331d:::benchmark_data_smithi122_197775_object61885:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e991)", 2024-03-20T09:46:11.237 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:36:31.991271+0000", 2024-03-20T09:46:11.237 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 579.20074933000001, 2024-03-20T09:46:11.237 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.14618183100000001, 2024-03-20T09:46:11.237 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:11.238 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:11.238 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:11.238 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:11.238 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:11.238 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 61886 2024-03-20T09:46:11.238 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.238 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:11.238 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.238 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:11.238 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.991271+0000", 2024-03-20T09:46:11.238 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.238 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.239 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.239 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:11.239 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.991271+0000", 2024-03-20T09:46:11.239 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.239 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.239 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.239 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:11.239 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.991272+0000", 2024-03-20T09:46:11.239 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.795e-06 2024-03-20T09:46:11.239 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.239 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.240 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:11.240 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.991307+0000", 2024-03-20T09:46:11.240 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4381000000000002e-05 2024-03-20T09:46:11.240 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.240 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.240 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:11.240 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.991326+0000", 2024-03-20T09:46:11.240 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9275000000000001e-05 2024-03-20T09:46:11.240 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.240 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.240 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:11.240 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.991335+0000", 2024-03-20T09:46:11.240 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.38e-06 2024-03-20T09:46:11.241 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.241 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.241 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:11.241 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.991396+0000", 2024-03-20T09:46:11.241 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0142000000000001e-05 2024-03-20T09:46:11.241 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.241 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.241 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:11.241 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.991573+0000", 2024-03-20T09:46:11.241 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000177591 2024-03-20T09:46:11.241 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.241 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.242 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2,9", 2024-03-20T09:46:11.242 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.991731+0000", 2024-03-20T09:46:11.242 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015784600000000001 2024-03-20T09:46:11.242 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.242 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.242 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:11.242 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.994112+0000", 2024-03-20T09:46:11.242 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00238098 2024-03-20T09:46:11.242 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.242 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.242 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:11.242 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.996517+0000", 2024-03-20T09:46:11.243 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0024049990000000001 2024-03-20T09:46:11.243 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.243 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.243 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:11.243 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.137346+0000", 2024-03-20T09:46:11.243 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.14082937100000001 2024-03-20T09:46:11.243 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.243 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.243 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:11.243 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.137408+0000", 2024-03-20T09:46:11.243 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1161999999999995e-05 2024-03-20T09:46:11.243 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.243 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.243 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:11.244 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.137452+0000", 2024-03-20T09:46:11.244 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4909e-05 2024-03-20T09:46:11.244 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.244 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:11.244 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.244 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.244 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.244 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:61888 8.a 8:535b7941:::benchmark_data_smithi122_197775_object61887:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e991)", 2024-03-20T09:46:11.244 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:36:31.991358+0000", 2024-03-20T09:46:11.244 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 579.20066222900005, 2024-03-20T09:46:11.244 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.14621425199999999, 2024-03-20T09:46:11.244 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:11.244 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:11.245 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:11.245 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:11.245 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:11.245 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 61888 2024-03-20T09:46:11.245 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.245 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:11.245 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.245 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:11.245 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.991358+0000", 2024-03-20T09:46:11.245 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.245 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.245 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.245 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:11.245 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.991358+0000", 2024-03-20T09:46:11.246 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.246 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.246 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.246 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:11.246 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.991358+0000", 2024-03-20T09:46:11.246 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.27e-07 2024-03-20T09:46:11.246 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.246 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.246 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:11.246 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.991375+0000", 2024-03-20T09:46:11.246 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7044000000000001e-05 2024-03-20T09:46:11.246 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.247 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.247 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:11.247 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.991381+0000", 2024-03-20T09:46:11.247 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3680000000000001e-06 2024-03-20T09:46:11.247 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.247 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.247 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:11.247 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.991384+0000", 2024-03-20T09:46:11.247 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7299999999999999e-06 2024-03-20T09:46:11.247 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.247 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.247 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:11.247 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.994305+0000", 2024-03-20T09:46:11.248 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0029210450000000002 2024-03-20T09:46:11.248 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.248 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.248 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:11.248 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.994425+0000", 2024-03-20T09:46:11.248 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011968699999999999 2024-03-20T09:46:11.248 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.248 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.248 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2,9", 2024-03-20T09:46:11.248 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.994546+0000", 2024-03-20T09:46:11.248 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012055100000000001 2024-03-20T09:46:11.248 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.248 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.248 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:11.249 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.997058+0000", 2024-03-20T09:46:11.249 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0025121840000000002 2024-03-20T09:46:11.249 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.249 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.249 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:11.249 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.997323+0000", 2024-03-20T09:46:11.249 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00026562999999999999 2024-03-20T09:46:11.249 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.249 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.249 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:11.249 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.137488+0000", 2024-03-20T09:46:11.249 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.14016420800000001 2024-03-20T09:46:11.249 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.249 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.250 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:11.250 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.137538+0000", 2024-03-20T09:46:11.250 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0164000000000003e-05 2024-03-20T09:46:11.250 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.250 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.250 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:11.250 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.137572+0000", 2024-03-20T09:46:11.250 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4214000000000001e-05 2024-03-20T09:46:11.250 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.250 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:11.250 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.250 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.250 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.251 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:61895 8.a 8:5a323f91:::benchmark_data_smithi122_197775_object61894:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e991)", 2024-03-20T09:46:11.251 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:36:31.997807+0000", 2024-03-20T09:46:11.251 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 579.19421248000003, 2024-03-20T09:46:11.251 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.14076506, 2024-03-20T09:46:11.251 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:11.251 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:11.251 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:11.251 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:11.251 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:11.251 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 61895 2024-03-20T09:46:11.251 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.251 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:11.251 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.251 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:11.252 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.997807+0000", 2024-03-20T09:46:11.252 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.252 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.252 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.252 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:11.252 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.997807+0000", 2024-03-20T09:46:11.252 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.252 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.252 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.252 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:11.252 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.997815+0000", 2024-03-20T09:46:11.252 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2799999999999998e-06 2024-03-20T09:46:11.252 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.252 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.253 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:11.253 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.997838+0000", 2024-03-20T09:46:11.253 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.304e-05 2024-03-20T09:46:11.253 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.253 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.253 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:11.253 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.997849+0000", 2024-03-20T09:46:11.253 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1678999999999999e-05 2024-03-20T09:46:11.253 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.253 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.253 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:11.253 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.997855+0000", 2024-03-20T09:46:11.253 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8000000000000004e-06 2024-03-20T09:46:11.253 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.254 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.254 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:11.254 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.997893+0000", 2024-03-20T09:46:11.254 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7375e-05 2024-03-20T09:46:11.254 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.254 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.254 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:11.254 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.997992+0000", 2024-03-20T09:46:11.254 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9053000000000002e-05 2024-03-20T09:46:11.254 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.254 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.254 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2,9", 2024-03-20T09:46:11.254 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:31.998096+0000", 2024-03-20T09:46:11.255 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104005 2024-03-20T09:46:11.255 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.255 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.255 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:11.255 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.000246+0000", 2024-03-20T09:46:11.255 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0021501509999999999 2024-03-20T09:46:11.255 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.255 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.255 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:11.255 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.000464+0000", 2024-03-20T09:46:11.255 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00021815499999999999 2024-03-20T09:46:11.255 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.255 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.255 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:11.256 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.138476+0000", 2024-03-20T09:46:11.256 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.138011574 2024-03-20T09:46:11.256 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.256 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.256 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:11.256 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.138530+0000", 2024-03-20T09:46:11.256 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4917999999999998e-05 2024-03-20T09:46:11.256 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.256 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.256 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:11.256 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.138573+0000", 2024-03-20T09:46:11.256 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2030000000000002e-05 2024-03-20T09:46:11.256 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.257 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:11.257 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.257 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.257 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.257 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:61913 8.a 8:57901749:::benchmark_data_smithi122_197775_object61912:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e991)", 2024-03-20T09:46:11.257 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:36:32.014311+0000", 2024-03-20T09:46:11.257 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 579.177708596, 2024-03-20T09:46:11.257 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.124403736, 2024-03-20T09:46:11.257 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:11.257 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:11.257 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:11.257 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:11.257 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:11.258 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 61913 2024-03-20T09:46:11.258 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.258 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:11.258 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.258 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:11.258 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.014311+0000", 2024-03-20T09:46:11.258 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.258 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.258 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.258 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:11.258 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.014311+0000", 2024-03-20T09:46:11.258 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.258 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.259 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.259 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:11.259 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.014313+0000", 2024-03-20T09:46:11.259 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.513e-06 2024-03-20T09:46:11.259 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.259 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.259 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:11.259 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.014338+0000", 2024-03-20T09:46:11.259 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5341999999999999e-05 2024-03-20T09:46:11.259 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.259 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.259 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:11.260 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.014356+0000", 2024-03-20T09:46:11.260 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8085e-05 2024-03-20T09:46:11.260 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.260 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.260 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:11.260 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.014365+0000", 2024-03-20T09:46:11.260 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8850000000000005e-06 2024-03-20T09:46:11.260 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.260 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.260 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:11.260 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.014427+0000", 2024-03-20T09:46:11.260 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1364000000000004e-05 2024-03-20T09:46:11.260 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.261 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.261 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:11.261 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.014585+0000", 2024-03-20T09:46:11.261 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000158756 2024-03-20T09:46:11.261 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.261 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.261 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2,9", 2024-03-20T09:46:11.261 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.014763+0000", 2024-03-20T09:46:11.261 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000177772 2024-03-20T09:46:11.261 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.261 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.261 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:11.261 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.016998+0000", 2024-03-20T09:46:11.261 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0022353949999999998 2024-03-20T09:46:11.262 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.262 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.262 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:11.262 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.017204+0000", 2024-03-20T09:46:11.262 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00020593900000000001 2024-03-20T09:46:11.262 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.262 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.262 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:11.262 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.138636+0000", 2024-03-20T09:46:11.262 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.12143140500000001 2024-03-20T09:46:11.262 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.263 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.263 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:11.263 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.138683+0000", 2024-03-20T09:46:11.263 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6829000000000002e-05 2024-03-20T09:46:11.263 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.263 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.263 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:11.263 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.138715+0000", 2024-03-20T09:46:11.263 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2450999999999998e-05 2024-03-20T09:46:11.263 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.263 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:11.264 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.264 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.264 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.264 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:61918 8.a 8:5935d738:::benchmark_data_smithi122_197775_object61917:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e991)", 2024-03-20T09:46:11.264 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:36:32.020523+0000", 2024-03-20T09:46:11.264 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 579.17149715699998, 2024-03-20T09:46:11.264 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.118315552, 2024-03-20T09:46:11.264 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:11.264 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:11.264 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:11.264 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:11.264 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:11.264 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 61918 2024-03-20T09:46:11.265 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.265 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:11.265 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.265 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:11.265 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.020523+0000", 2024-03-20T09:46:11.265 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.265 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.265 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.265 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:11.265 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.020523+0000", 2024-03-20T09:46:11.265 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.265 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.265 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.266 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:11.266 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.020524+0000", 2024-03-20T09:46:11.266 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4190000000000001e-06 2024-03-20T09:46:11.266 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.266 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.266 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:11.266 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.020543+0000", 2024-03-20T09:46:11.266 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9222e-05 2024-03-20T09:46:11.266 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.266 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.266 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:11.266 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.020554+0000", 2024-03-20T09:46:11.266 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0359e-05 2024-03-20T09:46:11.267 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.267 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.267 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:11.267 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.020559+0000", 2024-03-20T09:46:11.267 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4049999999999999e-06 2024-03-20T09:46:11.267 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.267 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.267 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:11.267 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.020604+0000", 2024-03-20T09:46:11.267 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4589000000000003e-05 2024-03-20T09:46:11.267 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.267 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.267 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:11.268 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.020703+0000", 2024-03-20T09:46:11.268 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.9117999999999995e-05 2024-03-20T09:46:11.268 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.268 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.268 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2,9", 2024-03-20T09:46:11.268 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.020789+0000", 2024-03-20T09:46:11.268 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6390999999999993e-05 2024-03-20T09:46:11.268 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.268 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.268 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:11.268 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.024685+0000", 2024-03-20T09:46:11.268 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0038958759999999999 2024-03-20T09:46:11.269 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.269 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.269 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:11.269 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.024693+0000", 2024-03-20T09:46:11.269 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8159999999999994e-06 2024-03-20T09:46:11.269 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.269 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.269 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:11.269 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.138753+0000", 2024-03-20T09:46:11.269 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.11405963399999999 2024-03-20T09:46:11.269 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.269 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.269 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:11.269 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.138806+0000", 2024-03-20T09:46:11.270 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3185e-05 2024-03-20T09:46:11.270 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.270 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.270 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:11.270 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.138838+0000", 2024-03-20T09:46:11.270 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2537999999999999e-05 2024-03-20T09:46:11.270 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.270 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:11.270 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.270 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.270 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.270 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:61921 8.a 8:5195be96:::benchmark_data_smithi122_197775_object61920:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e991)", 2024-03-20T09:46:11.270 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:36:32.022137+0000", 2024-03-20T09:46:11.271 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 579.16988331300001, 2024-03-20T09:46:11.271 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.116814765, 2024-03-20T09:46:11.271 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:11.271 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:11.271 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:11.271 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:11.271 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:11.271 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 61921 2024-03-20T09:46:11.271 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.271 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:11.271 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.271 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:11.271 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.022137+0000", 2024-03-20T09:46:11.272 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.272 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.272 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.272 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:11.272 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.022137+0000", 2024-03-20T09:46:11.272 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.272 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.272 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.272 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:11.272 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.022138+0000", 2024-03-20T09:46:11.272 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.26e-06 2024-03-20T09:46:11.272 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.272 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.272 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:11.273 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.022157+0000", 2024-03-20T09:46:11.273 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9224000000000001e-05 2024-03-20T09:46:11.273 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.273 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.273 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:11.273 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.022172+0000", 2024-03-20T09:46:11.273 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.52e-05 2024-03-20T09:46:11.273 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.273 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.273 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:11.273 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.022181+0000", 2024-03-20T09:46:11.273 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1309999999999995e-06 2024-03-20T09:46:11.273 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.274 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.274 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:11.274 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.022619+0000", 2024-03-20T09:46:11.274 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000437565 2024-03-20T09:46:11.274 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.274 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.274 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:11.274 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.022733+0000", 2024-03-20T09:46:11.274 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011391300000000001 2024-03-20T09:46:11.274 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.274 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.274 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 2,9", 2024-03-20T09:46:11.274 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.022874+0000", 2024-03-20T09:46:11.275 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000140617 2024-03-20T09:46:11.275 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.275 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.275 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:11.275 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.024847+0000", 2024-03-20T09:46:11.275 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0019738260000000001 2024-03-20T09:46:11.275 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.275 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.275 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:11.275 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.025073+0000", 2024-03-20T09:46:11.275 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00022584999999999999 2024-03-20T09:46:11.275 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.275 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.276 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:11.276 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.138870+0000", 2024-03-20T09:46:11.276 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.113796941 2024-03-20T09:46:11.276 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.276 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.276 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:11.276 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.138919+0000", 2024-03-20T09:46:11.276 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8730000000000003e-05 2024-03-20T09:46:11.276 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.276 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.276 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:11.276 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:36:32.138951+0000", 2024-03-20T09:46:11.276 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2508000000000001e-05 2024-03-20T09:46:11.277 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.277 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:11.277 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.277 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.277 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.277 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:142971 8.b 8:d85fb872:::benchmark_data_smithi122_197775_object19667:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1020)", 2024-03-20T09:46:11.277 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:37:52.199580+0000", 2024-03-20T09:46:11.277 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 498.99244022699997, 2024-03-20T09:46:11.277 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.116687492, 2024-03-20T09:46:11.277 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:11.277 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:11.277 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:11.278 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:11.278 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:11.278 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 142971 2024-03-20T09:46:11.278 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.278 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:11.278 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.278 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:11.278 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:37:52.199580+0000", 2024-03-20T09:46:11.278 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.278 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.278 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.278 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:11.279 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:37:52.199580+0000", 2024-03-20T09:46:11.279 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.279 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.279 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.279 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:11.279 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:37:52.199581+0000", 2024-03-20T09:46:11.279 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.432e-06 2024-03-20T09:46:11.279 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.279 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.279 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:11.279 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:37:52.199586+0000", 2024-03-20T09:46:11.279 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5240000000000002e-06 2024-03-20T09:46:11.279 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.280 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.280 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:11.280 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:37:52.199595+0000", 2024-03-20T09:46:11.280 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5650000000000007e-06 2024-03-20T09:46:11.280 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.280 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.280 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:11.280 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:37:52.199604+0000", 2024-03-20T09:46:11.280 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7490000000000004e-06 2024-03-20T09:46:11.280 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.280 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.280 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:11.280 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:37:52.199664+0000", 2024-03-20T09:46:11.281 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0299e-05 2024-03-20T09:46:11.281 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.281 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.281 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:11.281 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:37:52.199750+0000", 2024-03-20T09:46:11.281 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6151999999999998e-05 2024-03-20T09:46:11.281 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.281 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.281 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,4", 2024-03-20T09:46:11.281 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:37:52.199833+0000", 2024-03-20T09:46:11.281 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2478999999999994e-05 2024-03-20T09:46:11.281 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.281 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.281 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:11.282 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:37:52.204588+0000", 2024-03-20T09:46:11.282 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0047547800000000001 2024-03-20T09:46:11.282 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.282 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.282 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:11.282 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:37:52.209077+0000", 2024-03-20T09:46:11.282 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0044893010000000002 2024-03-20T09:46:11.282 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.282 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.282 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:11.282 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:37:52.316205+0000", 2024-03-20T09:46:11.282 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.107127721 2024-03-20T09:46:11.282 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.282 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.283 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:11.283 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:37:52.316238+0000", 2024-03-20T09:46:11.283 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3553000000000002e-05 2024-03-20T09:46:11.283 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.283 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.283 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:11.283 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:37:52.316267+0000", 2024-03-20T09:46:11.283 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8937000000000001e-05 2024-03-20T09:46:11.283 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.283 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:11.283 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.283 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.283 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.283 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:210006 9.b 9:d21f9aee:::benchmark_data_smithi122_210164_object77776:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1126)", 2024-03-20T09:46:11.284 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:43:38.501512+0000", 2024-03-20T09:46:11.284 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 152.69050836, 2024-03-20T09:46:11.284 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.10947626100000001, 2024-03-20T09:46:11.284 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:11.284 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:11.284 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:11.284 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:46:11.284 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:11.284 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 210006 2024-03-20T09:46:11.284 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.284 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:11.284 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.284 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:11.284 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:38.501512+0000", 2024-03-20T09:46:11.285 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.285 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.285 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.285 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:11.285 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:38.501512+0000", 2024-03-20T09:46:11.285 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.285 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.285 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.285 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:11.285 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:38.501513+0000", 2024-03-20T09:46:11.285 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.854e-06 2024-03-20T09:46:11.285 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.286 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.286 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:11.286 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:38.501521+0000", 2024-03-20T09:46:11.286 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3989999999999998e-06 2024-03-20T09:46:11.286 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.286 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.286 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:11.286 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:38.501538+0000", 2024-03-20T09:46:11.286 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6779e-05 2024-03-20T09:46:11.286 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.286 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.287 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:11.287 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:38.501547+0000", 2024-03-20T09:46:11.287 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7569999999999999e-06 2024-03-20T09:46:11.287 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.287 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.287 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:11.287 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:38.501613+0000", 2024-03-20T09:46:11.287 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5943000000000002e-05 2024-03-20T09:46:11.287 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.287 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.287 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:11.287 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:38.501742+0000", 2024-03-20T09:46:11.287 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012908599999999999 2024-03-20T09:46:11.288 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.288 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.288 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,10", 2024-03-20T09:46:11.288 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:38.501869+0000", 2024-03-20T09:46:11.288 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012666899999999999 2024-03-20T09:46:11.288 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.288 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.288 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:11.288 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:38.506723+0000", 2024-03-20T09:46:11.288 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0048539209999999998 2024-03-20T09:46:11.289 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.289 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.289 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:11.289 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:38.517382+0000", 2024-03-20T09:46:11.289 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.010658569 2024-03-20T09:46:11.289 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.289 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.289 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:11.289 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:38.610875+0000", 2024-03-20T09:46:11.289 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.093493166000000003 2024-03-20T09:46:11.289 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.289 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.290 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:11.290 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:38.610937+0000", 2024-03-20T09:46:11.290 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2226999999999999e-05 2024-03-20T09:46:11.290 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.290 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.290 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:11.290 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:38.610988+0000", 2024-03-20T09:46:11.290 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0890999999999998e-05 2024-03-20T09:46:11.290 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.290 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:11.290 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.290 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.291 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.291 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDScrubReserve(1.15,RELEASE,e:1153,from:7,reservation_nonce:0)", 2024-03-20T09:46:11.291 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:45:02.960029+0000", 2024-03-20T09:46:11.291 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 68.231990613999997, 2024-03-20T09:46:11.291 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00033223600000000002, 2024-03-20T09:46:11.291 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:11.291 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-03-20T09:46:11.291 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:11.291 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.291 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:11.291 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:45:02.960029+0000", 2024-03-20T09:46:11.291 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.291 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.292 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.292 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:11.292 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:45:02.960029+0000", 2024-03-20T09:46:11.292 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.292 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.292 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.292 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:11.292 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:45:02.960030+0000", 2024-03-20T09:46:11.292 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8999999999999997e-07 2024-03-20T09:46:11.292 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.292 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.292 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:11.292 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:45:02.960034+0000", 2024-03-20T09:46:11.292 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6e-06 2024-03-20T09:46:11.293 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.293 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.293 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:11.293 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:45:02.960048+0000", 2024-03-20T09:46:11.293 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.365e-05 2024-03-20T09:46:11.293 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.293 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.293 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:11.293 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:45:02.960064+0000", 2024-03-20T09:46:11.293 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6032000000000002e-05 2024-03-20T09:46:11.293 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.293 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.293 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:11.294 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:45:02.960308+0000", 2024-03-20T09:46:11.294 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00024399900000000001 2024-03-20T09:46:11.294 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.294 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.294 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:11.294 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:45:02.960342+0000", 2024-03-20T09:46:11.294 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3578000000000003e-05 2024-03-20T09:46:11.294 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.294 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.294 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:11.295 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:45:02.960362+0000", 2024-03-20T09:46:11.295 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9887000000000001e-05 2024-03-20T09:46:11.295 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.295 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:11.295 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.295 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.295 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.295 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "rep_scrubmap(1.1c e950 from shard 1)", 2024-03-20T09:46:11.295 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:45:38.770914+0000", 2024-03-20T09:46:11.295 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 32.421105931, 2024-03-20T09:46:11.295 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.02125896, 2024-03-20T09:46:11.295 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:11.295 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-03-20T09:46:11.296 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:11.296 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.296 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:11.296 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:45:38.770914+0000", 2024-03-20T09:46:11.296 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.296 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.296 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.296 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:11.296 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:45:38.770914+0000", 2024-03-20T09:46:11.296 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.296 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.296 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.296 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:11.296 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:45:38.770915+0000", 2024-03-20T09:46:11.297 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0279999999999999e-06 2024-03-20T09:46:11.297 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.297 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.297 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:11.297 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:45:38.770922+0000", 2024-03-20T09:46:11.297 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7819999999999996e-06 2024-03-20T09:46:11.297 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.297 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.297 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:11.297 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:45:38.770935+0000", 2024-03-20T09:46:11.297 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3118999999999999e-05 2024-03-20T09:46:11.297 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.297 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.298 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:11.298 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:45:38.770961+0000", 2024-03-20T09:46:11.298 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6554999999999999e-05 2024-03-20T09:46:11.298 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.298 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.298 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:11.298 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:45:38.792152+0000", 2024-03-20T09:46:11.298 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.021190476 2024-03-20T09:46:11.298 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.298 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.298 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:11.298 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:45:38.792161+0000", 2024-03-20T09:46:11.298 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3759999999999997e-06 2024-03-20T09:46:11.299 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.299 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.299 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:11.299 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:45:38.792173+0000", 2024-03-20T09:46:11.299 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1623999999999999e-05 2024-03-20T09:46:11.299 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.299 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:11.299 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.299 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.299 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.299 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "rep_scrubmap(1.1c e950 from shard 6)", 2024-03-20T09:46:11.299 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:45:38.771015+0000", 2024-03-20T09:46:11.300 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 32.42100525, 2024-03-20T09:46:11.300 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.021227804999999999, 2024-03-20T09:46:11.300 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:11.300 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-03-20T09:46:11.300 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:11.300 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.300 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:11.300 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:45:38.771015+0000", 2024-03-20T09:46:11.300 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.300 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.300 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.300 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:11.300 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:45:38.771015+0000", 2024-03-20T09:46:11.301 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.301 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.301 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.301 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:11.301 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:45:38.771015+0000", 2024-03-20T09:46:11.301 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1500000000000005e-07 2024-03-20T09:46:11.301 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.301 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.301 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:11.301 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:45:38.771018+0000", 2024-03-20T09:46:11.301 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7460000000000001e-06 2024-03-20T09:46:11.301 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.301 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.301 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:11.302 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:45:38.771025+0000", 2024-03-20T09:46:11.302 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6270000000000002e-06 2024-03-20T09:46:11.302 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.302 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.302 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:11.302 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:45:38.771029+0000", 2024-03-20T09:46:11.302 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9489999999999998e-06 2024-03-20T09:46:11.302 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.302 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.302 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:11.302 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:45:38.792205+0000", 2024-03-20T09:46:11.302 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.021176958999999999 2024-03-20T09:46:11.303 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.303 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.303 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:11.303 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:45:38.792214+0000", 2024-03-20T09:46:11.303 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2709999999999992e-06 2024-03-20T09:46:11.303 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.303 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.303 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:11.303 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:45:38.792242+0000", 2024-03-20T09:46:11.303 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8738000000000002e-05 2024-03-20T09:46:11.303 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.303 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:11.303 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.304 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.304 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.304 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "rep_scrubmap(1.17 e960 from shard 10)", 2024-03-20T09:46:11.304 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:45:55.925234+0000", 2024-03-20T09:46:11.304 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 15.266786238, 2024-03-20T09:46:11.304 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.021141035999999998, 2024-03-20T09:46:11.304 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:11.304 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-03-20T09:46:11.304 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:11.304 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.304 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:11.304 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:45:55.925234+0000", 2024-03-20T09:46:11.304 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.304 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.304 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.305 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:11.305 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:45:55.925234+0000", 2024-03-20T09:46:11.305 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:11.305 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.305 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.305 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:11.305 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:45:55.925235+0000", 2024-03-20T09:46:11.305 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2950000000000001e-06 2024-03-20T09:46:11.305 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.305 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.305 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:11.305 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:45:55.925243+0000", 2024-03-20T09:46:11.305 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4700000000000002e-06 2024-03-20T09:46:11.306 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.306 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.306 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:11.306 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:45:55.925259+0000", 2024-03-20T09:46:11.306 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5194000000000001e-05 2024-03-20T09:46:11.306 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.306 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.306 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:11.306 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:45:55.925268+0000", 2024-03-20T09:46:11.306 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.8079999999999999e-06 2024-03-20T09:46:11.306 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.306 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.306 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:11.306 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:45:55.946352+0000", 2024-03-20T09:46:11.307 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.021083504999999999 2024-03-20T09:46:11.307 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.307 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.307 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:11.307 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:45:55.946363+0000", 2024-03-20T09:46:11.307 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0909e-05 2024-03-20T09:46:11.307 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:11.307 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:11.307 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:11.307 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:45:55.946375+0000", 2024-03-20T09:46:11.307 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1855000000000001e-05 2024-03-20T09:46:11.307 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.307 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:11.307 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.308 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:11.308 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:11.308 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:46:11.308 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:10 smithi026 ceph-mon[68309]: 1.d scrub starts 2024-03-20T09:46:11.308 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:10 smithi026 ceph-mon[68309]: 1.d scrub ok 2024-03-20T09:46:11.794 INFO:teuthology.orchestra.run.smithi026.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-20T09:46:11.821 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_ops_in_flight 2024-03-20T09:46:12.084 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:46:12.087 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:11 smithi026 ceph-mon[68309]: pgmap v2865: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:12.087 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:11 smithi026 ceph-mon[68309]: from='client.17778 172.21.15.26:0/3454544408' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:46:12.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:11 smithi115 ceph-mon[72476]: pgmap v2865: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:12.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:11 smithi115 ceph-mon[72476]: from='client.17778 172.21.15.26:0/3454544408' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:46:12.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:11 smithi122 ceph-mon[73495]: pgmap v2865: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:12.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:11 smithi122 ceph-mon[73495]: from='client.17778 172.21.15.26:0/3454544408' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-20T09:46:12.426 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 8 1 2024-03-20T09:46:12.707 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:46:13.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:12 smithi115 ceph-mon[72476]: from='client.17778 172.21.15.26:0/3454544408' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-20T09:46:13.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:12 smithi115 ceph-mon[72476]: osdmap e1179: 12 total, 12 up, 12 in 2024-03-20T09:46:13.154 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:12 smithi026 ceph-mon[68309]: from='client.17778 172.21.15.26:0/3454544408' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-20T09:46:13.154 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:12 smithi026 ceph-mon[68309]: osdmap e1179: 12 total, 12 up, 12 in 2024-03-20T09:46:13.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:12 smithi122 ceph-mon[73495]: from='client.17778 172.21.15.26:0/3454544408' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-20T09:46:13.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:12 smithi122 ceph-mon[73495]: osdmap e1179: 12 total, 12 up, 12 in 2024-03-20T09:46:13.821 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:46:13.821 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:46:13.822 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-03-20T09:46:13.822 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:46:13.841 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:13 smithi122 ceph-mon[73495]: pgmap v2867: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:13.841 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:13 smithi122 ceph-mon[73495]: osdmap e1180: 12 total, 12 up, 12 in 2024-03-20T09:46:13.841 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:13 smithi122 ceph-mon[73495]: 1.9 scrub starts 2024-03-20T09:46:14.065 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:13 smithi026 ceph-mon[68309]: pgmap v2867: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:14.065 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:13 smithi026 ceph-mon[68309]: osdmap e1180: 12 total, 12 up, 12 in 2024-03-20T09:46:14.065 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:13 smithi026 ceph-mon[68309]: 1.9 scrub starts 2024-03-20T09:46:14.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:13 smithi115 ceph-mon[72476]: pgmap v2867: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:14.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:13 smithi115 ceph-mon[72476]: osdmap e1180: 12 total, 12 up, 12 in 2024-03-20T09:46:14.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:13 smithi115 ceph-mon[72476]: 1.9 scrub starts 2024-03-20T09:46:14.446 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_blocked_ops 2024-03-20T09:46:14.708 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:46:14.803 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.8 to 1 (10000) 2024-03-20T09:46:15.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:14 smithi115 ceph-mon[72476]: 1.9 scrub ok 2024-03-20T09:46:15.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:14 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/615670644' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:46:15.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:14 smithi122 ceph-mon[73495]: 1.9 scrub ok 2024-03-20T09:46:15.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:14 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/615670644' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:46:15.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:14 smithi026 ceph-mon[68309]: 1.9 scrub ok 2024-03-20T09:46:15.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:14 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/615670644' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:46:15.589 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd primary-affinity 8 1 2024-03-20T09:46:15.873 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:46:15.956 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:15 smithi026 ceph-mon[68309]: pgmap v2869: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:15.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:15 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/615670644' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-03-20T09:46:15.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:15 smithi026 ceph-mon[68309]: osdmap e1181: 12 total, 12 up, 12 in 2024-03-20T09:46:16.052 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:15 smithi122 ceph-mon[73495]: pgmap v2869: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:16.052 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:15 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/615670644' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-03-20T09:46:16.053 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:15 smithi122 ceph-mon[73495]: osdmap e1181: 12 total, 12 up, 12 in 2024-03-20T09:46:16.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:15 smithi115 ceph-mon[72476]: pgmap v2869: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:16.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:15 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/615670644' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-03-20T09:46:16.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:15 smithi115 ceph-mon[72476]: osdmap e1181: 12 total, 12 up, 12 in 2024-03-20T09:46:16.592 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:46:16.593 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:46:16.593 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-03-20T09:46:16.593 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-03-20T09:46:16.593 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:46:17.293 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.9 dump_historic_ops 2024-03-20T09:46:17.555 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:46:17.806 INFO:teuthology.orchestra.run.smithi026.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-03-20T09:46:18.091 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:17 smithi026 ceph-mon[68309]: pgmap v2871: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:18.092 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:17 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3141302157' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:46:18.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:17 smithi115 ceph-mon[72476]: pgmap v2871: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:18.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:17 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3141302157' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:46:18.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:17 smithi122 ceph-mon[73495]: pgmap v2871: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:18.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:17 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3141302157' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]: dispatch 2024-03-20T09:46:18.495 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 9 1 2024-03-20T09:46:18.774 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:46:18.933 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:18 smithi122 ceph-mon[73495]: 1.4 scrub starts 2024-03-20T09:46:18.933 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:18 smithi122 ceph-mon[73495]: 1.4 scrub ok 2024-03-20T09:46:18.933 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:18 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3141302157' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-03-20T09:46:18.933 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:18 smithi122 ceph-mon[73495]: osdmap e1182: 12 total, 12 up, 12 in 2024-03-20T09:46:19.070 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:18 smithi026 ceph-mon[68309]: 1.4 scrub starts 2024-03-20T09:46:19.070 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:18 smithi026 ceph-mon[68309]: 1.4 scrub ok 2024-03-20T09:46:19.070 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:18 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3141302157' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-03-20T09:46:19.070 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:18 smithi026 ceph-mon[68309]: osdmap e1182: 12 total, 12 up, 12 in 2024-03-20T09:46:19.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:18 smithi115 ceph-mon[72476]: 1.4 scrub starts 2024-03-20T09:46:19.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:18 smithi115 ceph-mon[72476]: 1.4 scrub ok 2024-03-20T09:46:19.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:18 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3141302157' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-03-20T09:46:19.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:18 smithi115 ceph-mon[72476]: osdmap e1182: 12 total, 12 up, 12 in 2024-03-20T09:46:19.245 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:46:19.245 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-03-20T09:46:19.246 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-03-20T09:46:19.246 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-03-20T09:46:19.246 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.246 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:238411 8.1 8:86d760dd:::benchmark_data_smithi122_197775_object115107:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1056)", 2024-03-20T09:46:19.246 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:39:55.048854+0000", 2024-03-20T09:46:19.246 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 384.20829756900002, 2024-03-20T09:46:19.246 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.216121966, 2024-03-20T09:46:19.246 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:19.246 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:19.246 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:19.246 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:19.246 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:19.246 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 238411 2024-03-20T09:46:19.246 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.247 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:19.247 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.247 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:19.247 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.048854+0000", 2024-03-20T09:46:19.247 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.247 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.247 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.247 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:19.247 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.048854+0000", 2024-03-20T09:46:19.247 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.247 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.247 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.247 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:19.247 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.048855+0000", 2024-03-20T09:46:19.248 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7500000000000001e-07 2024-03-20T09:46:19.248 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.248 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.248 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:19.248 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.048858+0000", 2024-03-20T09:46:19.248 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2200000000000001e-06 2024-03-20T09:46:19.248 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.248 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.248 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:19.248 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.048864+0000", 2024-03-20T09:46:19.248 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.421e-06 2024-03-20T09:46:19.248 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.248 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.248 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:19.249 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.048870+0000", 2024-03-20T09:46:19.249 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4e-06 2024-03-20T09:46:19.249 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.249 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.249 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:19.249 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.053672+0000", 2024-03-20T09:46:19.249 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0048026240000000001 2024-03-20T09:46:19.249 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.249 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.249 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:19.249 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.053784+0000", 2024-03-20T09:46:19.249 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000111839 2024-03-20T09:46:19.249 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.250 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.250 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,2", 2024-03-20T09:46:19.250 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.053854+0000", 2024-03-20T09:46:19.250 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9915999999999999e-05 2024-03-20T09:46:19.250 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.250 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.250 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:19.250 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.057950+0000", 2024-03-20T09:46:19.250 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040960759999999997 2024-03-20T09:46:19.250 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.250 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.251 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:19.251 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.061752+0000", 2024-03-20T09:46:19.251 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0038016019999999998 2024-03-20T09:46:19.251 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.251 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.251 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:19.251 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.264887+0000", 2024-03-20T09:46:19.251 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.203135396 2024-03-20T09:46:19.251 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.251 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.251 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:19.251 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.264938+0000", 2024-03-20T09:46:19.251 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0494000000000001e-05 2024-03-20T09:46:19.251 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.251 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.252 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:19.252 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.264976+0000", 2024-03-20T09:46:19.252 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8603000000000003e-05 2024-03-20T09:46:19.252 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.252 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:19.252 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.252 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.252 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.252 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:238427 8.1 8:8f1c7409:::benchmark_data_smithi122_197775_object115123:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1056)", 2024-03-20T09:46:19.252 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:39:55.066135+0000", 2024-03-20T09:46:19.252 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 384.19101713499998, 2024-03-20T09:46:19.252 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.21237999900000001, 2024-03-20T09:46:19.252 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:19.252 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:19.253 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:19.253 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:19.253 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:19.253 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 238427 2024-03-20T09:46:19.253 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.253 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:19.253 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.253 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:19.253 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.066135+0000", 2024-03-20T09:46:19.253 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.253 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.253 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.253 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:19.253 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.066135+0000", 2024-03-20T09:46:19.253 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.254 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.254 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.254 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:19.254 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.066136+0000", 2024-03-20T09:46:19.254 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.519e-06 2024-03-20T09:46:19.254 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.254 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.254 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:19.254 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.066142+0000", 2024-03-20T09:46:19.254 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1140000000000002e-06 2024-03-20T09:46:19.254 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.254 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.254 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:19.254 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.066154+0000", 2024-03-20T09:46:19.254 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1418e-05 2024-03-20T09:46:19.255 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.255 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.255 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:19.255 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.066162+0000", 2024-03-20T09:46:19.255 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2570000000000007e-06 2024-03-20T09:46:19.255 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.255 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.255 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:19.255 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.066228+0000", 2024-03-20T09:46:19.256 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5634000000000002e-05 2024-03-20T09:46:19.256 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.256 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.256 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:19.256 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.066360+0000", 2024-03-20T09:46:19.256 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000132352 2024-03-20T09:46:19.256 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.256 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.256 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,2", 2024-03-20T09:46:19.256 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.066467+0000", 2024-03-20T09:46:19.256 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000106449 2024-03-20T09:46:19.256 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.256 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.256 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:19.257 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.071716+0000", 2024-03-20T09:46:19.257 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0052489870000000001 2024-03-20T09:46:19.257 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.257 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.257 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:19.257 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.080421+0000", 2024-03-20T09:46:19.257 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0087058599999999993 2024-03-20T09:46:19.257 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.257 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.257 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:19.257 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.278418+0000", 2024-03-20T09:46:19.257 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.197996124 2024-03-20T09:46:19.257 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.257 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.257 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:19.258 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.278471+0000", 2024-03-20T09:46:19.258 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3458000000000002e-05 2024-03-20T09:46:19.258 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.258 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.258 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:19.258 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:55.278515+0000", 2024-03-20T09:46:19.258 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3826999999999999e-05 2024-03-20T09:46:19.258 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.258 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:19.258 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.258 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.258 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.258 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:11182 9.3 9:c1e11bf1:::benchmark_data_smithi122_210164_object11181:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1077)", 2024-03-20T09:46:19.258 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:40:43.040902+0000", 2024-03-20T09:46:19.258 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 336.21624988100001, 2024-03-20T09:46:19.259 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20538585200000001, 2024-03-20T09:46:19.259 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:19.259 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:19.259 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:19.259 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:46:19.259 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:19.259 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 11182 2024-03-20T09:46:19.259 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.259 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:19.259 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.259 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:19.259 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.040902+0000", 2024-03-20T09:46:19.259 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.260 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.260 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.260 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:19.260 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.040902+0000", 2024-03-20T09:46:19.260 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.260 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.260 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.260 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:19.260 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.040903+0000", 2024-03-20T09:46:19.260 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.215e-06 2024-03-20T09:46:19.260 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.261 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.261 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:19.261 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.040923+0000", 2024-03-20T09:46:19.261 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9304999999999999e-05 2024-03-20T09:46:19.261 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.261 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.261 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:19.261 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.040936+0000", 2024-03-20T09:46:19.261 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3487e-05 2024-03-20T09:46:19.261 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.261 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.261 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:19.261 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.040942+0000", 2024-03-20T09:46:19.261 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2790000000000004e-06 2024-03-20T09:46:19.262 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.262 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.262 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:19.262 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.041002+0000", 2024-03-20T09:46:19.262 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9181000000000001e-05 2024-03-20T09:46:19.262 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.262 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.262 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:19.262 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.041139+0000", 2024-03-20T09:46:19.262 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000137646 2024-03-20T09:46:19.262 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.262 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.262 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,5", 2024-03-20T09:46:19.263 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.041263+0000", 2024-03-20T09:46:19.263 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012335800000000001 2024-03-20T09:46:19.263 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.263 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.263 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:19.263 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.042375+0000", 2024-03-20T09:46:19.263 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001112624 2024-03-20T09:46:19.263 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.263 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.263 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:19.263 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.044200+0000", 2024-03-20T09:46:19.263 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0018248100000000001 2024-03-20T09:46:19.263 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.263 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.264 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:19.264 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.246210+0000", 2024-03-20T09:46:19.264 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20201024100000001 2024-03-20T09:46:19.264 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.264 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.264 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:19.264 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.246250+0000", 2024-03-20T09:46:19.264 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9422999999999998e-05 2024-03-20T09:46:19.264 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.264 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.264 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:19.264 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.246288+0000", 2024-03-20T09:46:19.264 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8282999999999999e-05 2024-03-20T09:46:19.264 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.265 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:19.265 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.265 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.265 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.265 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:11185 9.3 9:cf95aa79:::benchmark_data_smithi122_210164_object11184:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1077)", 2024-03-20T09:46:19.265 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:40:43.042516+0000", 2024-03-20T09:46:19.265 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 336.21463604000002, 2024-03-20T09:46:19.265 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20459010499999999, 2024-03-20T09:46:19.265 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:19.265 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:19.265 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:19.265 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:46:19.265 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:19.265 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 11185 2024-03-20T09:46:19.265 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.266 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:19.266 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.266 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:19.266 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.042516+0000", 2024-03-20T09:46:19.266 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.266 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.266 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.266 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:19.266 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.042516+0000", 2024-03-20T09:46:19.266 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.266 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.266 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.266 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:19.266 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.042517+0000", 2024-03-20T09:46:19.266 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.19e-06 2024-03-20T09:46:19.266 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.267 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.267 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:19.267 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.042539+0000", 2024-03-20T09:46:19.267 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1953000000000001e-05 2024-03-20T09:46:19.267 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.267 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.267 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:19.267 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.042551+0000", 2024-03-20T09:46:19.267 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1605e-05 2024-03-20T09:46:19.267 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.267 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.267 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:19.267 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.042559+0000", 2024-03-20T09:46:19.267 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3410000000000003e-06 2024-03-20T09:46:19.268 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.268 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.268 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:19.268 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.042617+0000", 2024-03-20T09:46:19.268 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8263000000000001e-05 2024-03-20T09:46:19.268 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.268 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.268 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:19.268 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.042781+0000", 2024-03-20T09:46:19.268 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016405200000000001 2024-03-20T09:46:19.268 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.268 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.268 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,5", 2024-03-20T09:46:19.268 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.042934+0000", 2024-03-20T09:46:19.268 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015226899999999999 2024-03-20T09:46:19.269 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.269 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.269 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:19.269 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.044256+0000", 2024-03-20T09:46:19.269 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0013219480000000001 2024-03-20T09:46:19.269 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.269 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.269 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:19.269 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.044303+0000", 2024-03-20T09:46:19.269 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7048000000000001e-05 2024-03-20T09:46:19.269 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.269 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.269 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:19.269 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.247006+0000", 2024-03-20T09:46:19.270 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.202703042 2024-03-20T09:46:19.270 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.270 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.270 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:19.270 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.247057+0000", 2024-03-20T09:46:19.270 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1313999999999997e-05 2024-03-20T09:46:19.270 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.270 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.270 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:19.270 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.247106+0000", 2024-03-20T09:46:19.270 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9079999999999998e-05 2024-03-20T09:46:19.270 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.271 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:19.271 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.271 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.271 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.271 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:11188 9.3 9:ca80a92d:::benchmark_data_smithi122_210164_object11187:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1077)", 2024-03-20T09:46:19.271 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:40:43.042667+0000", 2024-03-20T09:46:19.271 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 336.21448539900001, 2024-03-20T09:46:19.271 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20458264000000001, 2024-03-20T09:46:19.271 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:19.271 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:19.271 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:19.272 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:46:19.272 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:19.272 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 11188 2024-03-20T09:46:19.272 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.272 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:19.272 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.272 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:19.272 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.042667+0000", 2024-03-20T09:46:19.272 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.272 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.272 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.273 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:19.273 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.042667+0000", 2024-03-20T09:46:19.273 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.273 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.273 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.273 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:19.273 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.042667+0000", 2024-03-20T09:46:19.273 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3399999999999997e-07 2024-03-20T09:46:19.273 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.273 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.273 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:19.273 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.042690+0000", 2024-03-20T09:46:19.274 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.2909999999999999e-05 2024-03-20T09:46:19.274 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.274 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.274 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:19.274 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.042702+0000", 2024-03-20T09:46:19.274 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1654000000000001e-05 2024-03-20T09:46:19.274 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.274 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.274 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:19.274 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.042710+0000", 2024-03-20T09:46:19.274 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9370000000000004e-06 2024-03-20T09:46:19.274 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.274 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.275 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:19.275 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.043500+0000", 2024-03-20T09:46:19.275 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00078967100000000004 2024-03-20T09:46:19.275 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.275 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.275 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:19.275 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.043610+0000", 2024-03-20T09:46:19.275 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000110308 2024-03-20T09:46:19.275 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.275 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.275 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,5", 2024-03-20T09:46:19.275 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.043739+0000", 2024-03-20T09:46:19.275 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012889100000000001 2024-03-20T09:46:19.276 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.276 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.276 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:19.276 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.044451+0000", 2024-03-20T09:46:19.276 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00071210900000000005 2024-03-20T09:46:19.276 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.276 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.276 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:19.276 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.046356+0000", 2024-03-20T09:46:19.276 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0019048349999999999 2024-03-20T09:46:19.276 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.276 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.276 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:19.276 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.247148+0000", 2024-03-20T09:46:19.276 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.200792199 2024-03-20T09:46:19.277 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.277 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.277 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:19.277 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.247207+0000", 2024-03-20T09:46:19.277 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9242e-05 2024-03-20T09:46:19.277 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.277 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.277 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:19.277 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:40:43.247249+0000", 2024-03-20T09:46:19.277 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1950000000000003e-05 2024-03-20T09:46:19.277 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.277 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:19.277 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.277 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.277 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.278 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:123495 9.8 9:19bc1614:::benchmark_data_smithi122_210164_object123494:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1097)", 2024-03-20T09:46:19.278 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:41:54.581171+0000", 2024-03-20T09:46:19.278 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 264.67598055299999, 2024-03-20T09:46:19.278 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.21403297299999999, 2024-03-20T09:46:19.278 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:19.278 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:19.278 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:19.278 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:46:19.278 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:19.278 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 123495 2024-03-20T09:46:19.278 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.278 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:19.278 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.279 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:19.279 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581171+0000", 2024-03-20T09:46:19.279 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.279 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.279 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.279 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:19.279 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581171+0000", 2024-03-20T09:46:19.279 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.279 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.279 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.279 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:19.279 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581173+0000", 2024-03-20T09:46:19.279 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3179999999999999e-06 2024-03-20T09:46:19.279 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.280 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.280 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:19.280 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581197+0000", 2024-03-20T09:46:19.280 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.4071e-05 2024-03-20T09:46:19.280 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.280 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.280 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:19.280 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581208+0000", 2024-03-20T09:46:19.280 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1379999999999999e-05 2024-03-20T09:46:19.280 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.280 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.280 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:19.280 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581214+0000", 2024-03-20T09:46:19.280 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1399999999999997e-06 2024-03-20T09:46:19.281 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.281 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.281 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:19.281 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581273+0000", 2024-03-20T09:46:19.281 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8227999999999999e-05 2024-03-20T09:46:19.281 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.281 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.281 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:19.281 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581408+0000", 2024-03-20T09:46:19.281 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013551899999999999 2024-03-20T09:46:19.281 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.281 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.281 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,11", 2024-03-20T09:46:19.281 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581500+0000", 2024-03-20T09:46:19.282 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2250999999999995e-05 2024-03-20T09:46:19.282 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.282 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.282 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:19.282 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.597971+0000", 2024-03-20T09:46:19.282 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016470496000000001 2024-03-20T09:46:19.282 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.282 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.282 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:19.282 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.598344+0000", 2024-03-20T09:46:19.282 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00037296299999999997 2024-03-20T09:46:19.282 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.282 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.282 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:19.283 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.795088+0000", 2024-03-20T09:46:19.283 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19674451900000001 2024-03-20T09:46:19.283 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.283 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.283 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:19.283 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.795154+0000", 2024-03-20T09:46:19.283 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5895000000000005e-05 2024-03-20T09:46:19.283 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.283 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.283 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:19.283 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.795204+0000", 2024-03-20T09:46:19.283 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0192999999999999e-05 2024-03-20T09:46:19.283 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.283 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:19.283 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.283 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.284 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.284 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:123498 9.8 9:15c518f2:::benchmark_data_smithi122_210164_object123497:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1097)", 2024-03-20T09:46:19.284 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:41:54.581234+0000", 2024-03-20T09:46:19.284 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 264.67591843000002, 2024-03-20T09:46:19.284 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.22821238999999999, 2024-03-20T09:46:19.284 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:19.284 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:19.284 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:19.284 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:46:19.284 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:19.284 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 123498 2024-03-20T09:46:19.284 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.284 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:19.284 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.285 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:19.285 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581234+0000", 2024-03-20T09:46:19.285 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.285 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.285 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.285 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:19.285 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581234+0000", 2024-03-20T09:46:19.285 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.285 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.285 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.285 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:19.285 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581234+0000", 2024-03-20T09:46:19.285 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7399999999999998e-07 2024-03-20T09:46:19.285 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.285 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.286 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:19.286 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581259+0000", 2024-03-20T09:46:19.286 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5176000000000001e-05 2024-03-20T09:46:19.286 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.286 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.286 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:19.286 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581266+0000", 2024-03-20T09:46:19.286 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8560000000000001e-06 2024-03-20T09:46:19.286 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.286 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.286 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:19.286 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581271+0000", 2024-03-20T09:46:19.286 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3720000000000004e-06 2024-03-20T09:46:19.286 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.286 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.286 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:19.287 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.585053+0000", 2024-03-20T09:46:19.287 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003781207 2024-03-20T09:46:19.287 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.287 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.287 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:19.287 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.585194+0000", 2024-03-20T09:46:19.287 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014127500000000001 2024-03-20T09:46:19.287 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.287 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.287 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,11", 2024-03-20T09:46:19.287 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.585342+0000", 2024-03-20T09:46:19.287 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014771399999999999 2024-03-20T09:46:19.287 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.287 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.287 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:19.288 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.598026+0000", 2024-03-20T09:46:19.288 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012684465000000001 2024-03-20T09:46:19.288 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.288 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.288 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:19.288 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.598351+0000", 2024-03-20T09:46:19.288 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00032446200000000003 2024-03-20T09:46:19.288 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.288 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.288 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:19.288 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.809328+0000", 2024-03-20T09:46:19.288 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.210977058 2024-03-20T09:46:19.288 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.288 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.288 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:19.288 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.809395+0000", 2024-03-20T09:46:19.289 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7188999999999997e-05 2024-03-20T09:46:19.289 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.289 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.289 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:19.289 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.809446+0000", 2024-03-20T09:46:19.289 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1141999999999999e-05 2024-03-20T09:46:19.289 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.289 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:19.289 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.289 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.289 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.289 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:123499 9.8 9:1ff497bd:::benchmark_data_smithi122_210164_object123498:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1097)", 2024-03-20T09:46:19.289 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:41:54.581280+0000", 2024-03-20T09:46:19.289 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 264.67587153699998, 2024-03-20T09:46:19.290 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.228306763, 2024-03-20T09:46:19.290 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:19.290 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:19.290 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:19.290 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:46:19.290 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:19.290 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 123499 2024-03-20T09:46:19.290 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.290 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:19.290 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.290 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:19.290 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581280+0000", 2024-03-20T09:46:19.290 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.290 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.291 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.291 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:19.291 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581280+0000", 2024-03-20T09:46:19.291 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.291 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.291 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.291 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:19.291 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581287+0000", 2024-03-20T09:46:19.291 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2430000000000002e-06 2024-03-20T09:46:19.291 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.291 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.291 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:19.291 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581310+0000", 2024-03-20T09:46:19.291 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.336e-05 2024-03-20T09:46:19.291 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.291 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.292 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:19.292 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581317+0000", 2024-03-20T09:46:19.292 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5189999999999998e-06 2024-03-20T09:46:19.292 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.292 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.292 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:19.292 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.581322+0000", 2024-03-20T09:46:19.292 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1209999999999998e-06 2024-03-20T09:46:19.292 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.292 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.292 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:19.292 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.590394+0000", 2024-03-20T09:46:19.292 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0090721670000000008 2024-03-20T09:46:19.292 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.293 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.293 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:19.293 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.590524+0000", 2024-03-20T09:46:19.293 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000130278 2024-03-20T09:46:19.293 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.293 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.293 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,11", 2024-03-20T09:46:19.293 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.590617+0000", 2024-03-20T09:46:19.293 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.3398000000000003e-05 2024-03-20T09:46:19.293 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.293 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.293 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:19.293 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.598356+0000", 2024-03-20T09:46:19.293 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0077385320000000002 2024-03-20T09:46:19.293 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.294 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.294 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:19.294 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.599682+0000", 2024-03-20T09:46:19.294 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0013261449999999999 2024-03-20T09:46:19.294 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.294 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.294 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:19.294 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.809492+0000", 2024-03-20T09:46:19.294 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.209810303 2024-03-20T09:46:19.294 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.295 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.295 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:19.295 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.809542+0000", 2024-03-20T09:46:19.295 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9919000000000001e-05 2024-03-20T09:46:19.295 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.295 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.295 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:19.295 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.809587+0000", 2024-03-20T09:46:19.295 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4777999999999997e-05 2024-03-20T09:46:19.295 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.295 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:19.295 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.295 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.295 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.295 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:123507 9.8 9:106df26e:::benchmark_data_smithi122_210164_object123506:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1097)", 2024-03-20T09:46:19.296 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:41:54.590309+0000", 2024-03-20T09:46:19.296 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 264.66684316099997, 2024-03-20T09:46:19.296 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.21939439699999999, 2024-03-20T09:46:19.296 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:19.296 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:19.296 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:19.296 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:46:19.296 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:19.296 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 123507 2024-03-20T09:46:19.296 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.296 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:19.296 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.296 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:19.297 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.590309+0000", 2024-03-20T09:46:19.297 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.297 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.297 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.297 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:19.297 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.590309+0000", 2024-03-20T09:46:19.297 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.297 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.297 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.297 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:19.297 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.590310+0000", 2024-03-20T09:46:19.297 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.576e-06 2024-03-20T09:46:19.297 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.297 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.297 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:19.298 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.590330+0000", 2024-03-20T09:46:19.298 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.946e-05 2024-03-20T09:46:19.298 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.298 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.298 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:19.298 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.590339+0000", 2024-03-20T09:46:19.298 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5810000000000003e-06 2024-03-20T09:46:19.298 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.298 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.298 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:19.298 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.590345+0000", 2024-03-20T09:46:19.298 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9739999999999999e-06 2024-03-20T09:46:19.298 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.298 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.298 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:19.299 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.593877+0000", 2024-03-20T09:46:19.299 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0035314790000000001 2024-03-20T09:46:19.299 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.299 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.299 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:19.299 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.594048+0000", 2024-03-20T09:46:19.299 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000171476 2024-03-20T09:46:19.299 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.299 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.299 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,11", 2024-03-20T09:46:19.299 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.594201+0000", 2024-03-20T09:46:19.299 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000152499 2024-03-20T09:46:19.299 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.299 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.300 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:19.300 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.598465+0000", 2024-03-20T09:46:19.300 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0042645019999999999 2024-03-20T09:46:19.300 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.300 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.300 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:19.300 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.599855+0000", 2024-03-20T09:46:19.300 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00138944 2024-03-20T09:46:19.300 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.300 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.300 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:19.300 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.809622+0000", 2024-03-20T09:46:19.300 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.209767017 2024-03-20T09:46:19.300 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.301 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.301 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:19.301 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.809669+0000", 2024-03-20T09:46:19.301 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7030000000000002e-05 2024-03-20T09:46:19.301 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.301 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.301 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:19.301 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.809703+0000", 2024-03-20T09:46:19.301 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4363000000000003e-05 2024-03-20T09:46:19.301 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.301 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:19.301 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.301 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.301 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.301 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:123514 9.3 9:cac7a75a:::benchmark_data_smithi122_210164_object123513:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1097)", 2024-03-20T09:46:19.302 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:41:54.607199+0000", 2024-03-20T09:46:19.302 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 264.64995321499998, 2024-03-20T09:46:19.302 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20607972799999999, 2024-03-20T09:46:19.302 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:19.302 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:19.302 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:19.302 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:46:19.302 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:19.302 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 123514 2024-03-20T09:46:19.302 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.302 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:19.302 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.302 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:19.303 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.607199+0000", 2024-03-20T09:46:19.303 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.303 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.303 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.303 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:19.303 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.607199+0000", 2024-03-20T09:46:19.303 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.303 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.303 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.303 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:19.303 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.607200+0000", 2024-03-20T09:46:19.303 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.409e-06 2024-03-20T09:46:19.303 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.304 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.304 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:19.304 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.607229+0000", 2024-03-20T09:46:19.304 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8921999999999999e-05 2024-03-20T09:46:19.304 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.304 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.304 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:19.304 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.607244+0000", 2024-03-20T09:46:19.304 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5339999999999999e-05 2024-03-20T09:46:19.304 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.304 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.304 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:19.304 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.607254+0000", 2024-03-20T09:46:19.304 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7520000000000008e-06 2024-03-20T09:46:19.305 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.305 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.305 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:19.305 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.607328+0000", 2024-03-20T09:46:19.305 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4337000000000003e-05 2024-03-20T09:46:19.305 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.305 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.305 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:19.305 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.607472+0000", 2024-03-20T09:46:19.305 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000143373 2024-03-20T09:46:19.305 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.305 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.305 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,5", 2024-03-20T09:46:19.305 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.607605+0000", 2024-03-20T09:46:19.306 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000132867 2024-03-20T09:46:19.306 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.306 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.306 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:19.306 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.611496+0000", 2024-03-20T09:46:19.306 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0038913799999999998 2024-03-20T09:46:19.306 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.306 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.306 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:19.306 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.614508+0000", 2024-03-20T09:46:19.306 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0030114759999999999 2024-03-20T09:46:19.306 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.306 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.306 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:19.307 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.813169+0000", 2024-03-20T09:46:19.307 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19866141600000001 2024-03-20T09:46:19.307 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.307 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.307 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:19.307 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.813232+0000", 2024-03-20T09:46:19.307 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2836999999999995e-05 2024-03-20T09:46:19.307 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.307 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.307 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:19.307 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.813278+0000", 2024-03-20T09:46:19.307 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6619000000000002e-05 2024-03-20T09:46:19.307 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.307 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:19.307 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.307 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.308 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.308 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:123515 9.8 9:1b95f987:::benchmark_data_smithi122_210164_object123514:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1097)", 2024-03-20T09:46:19.308 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:41:54.607280+0000", 2024-03-20T09:46:19.308 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 264.64987151700001, 2024-03-20T09:46:19.308 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.206131808, 2024-03-20T09:46:19.308 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:19.308 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:19.308 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:19.308 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:46:19.308 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:19.308 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 123515 2024-03-20T09:46:19.308 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.308 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:19.308 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.309 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:19.309 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.607280+0000", 2024-03-20T09:46:19.309 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.309 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.309 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.309 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:19.309 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.607280+0000", 2024-03-20T09:46:19.309 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.309 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.309 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.309 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:19.309 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.607289+0000", 2024-03-20T09:46:19.309 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1300000000000001e-06 2024-03-20T09:46:19.309 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.309 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.310 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:19.310 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.607307+0000", 2024-03-20T09:46:19.310 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8258999999999999e-05 2024-03-20T09:46:19.310 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.310 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.310 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:19.310 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.607315+0000", 2024-03-20T09:46:19.310 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6119999999999999e-06 2024-03-20T09:46:19.310 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.310 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.310 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:19.310 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.607322+0000", 2024-03-20T09:46:19.310 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0990000000000004e-06 2024-03-20T09:46:19.310 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.310 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.311 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:19.311 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.607363+0000", 2024-03-20T09:46:19.311 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1353000000000002e-05 2024-03-20T09:46:19.311 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.311 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.311 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:19.311 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.607458+0000", 2024-03-20T09:46:19.311 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4815999999999995e-05 2024-03-20T09:46:19.311 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.311 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.311 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,11", 2024-03-20T09:46:19.311 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.607593+0000", 2024-03-20T09:46:19.311 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013547400000000001 2024-03-20T09:46:19.311 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.311 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.312 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:19.312 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.614489+0000", 2024-03-20T09:46:19.312 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0068957300000000001 2024-03-20T09:46:19.312 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.312 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.312 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:19.312 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.615606+0000", 2024-03-20T09:46:19.312 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0011172339999999999 2024-03-20T09:46:19.312 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.312 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.312 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:19.312 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.813313+0000", 2024-03-20T09:46:19.312 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19770711499999999 2024-03-20T09:46:19.312 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.313 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.313 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:19.313 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.813369+0000", 2024-03-20T09:46:19.313 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5621000000000002e-05 2024-03-20T09:46:19.313 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.313 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.313 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:19.313 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:54.813412+0000", 2024-03-20T09:46:19.313 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3365000000000002e-05 2024-03-20T09:46:19.313 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.313 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:19.313 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.313 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.313 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.313 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:126241 9.8 9:18639737:::benchmark_data_smithi122_210164_object126240:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1098)", 2024-03-20T09:46:19.314 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:41:59.145423+0000", 2024-03-20T09:46:19.314 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 260.11172854599999, 2024-03-20T09:46:19.314 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.25194071499999998, 2024-03-20T09:46:19.314 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:19.314 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:19.314 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:19.314 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:46:19.314 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:19.314 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 126241 2024-03-20T09:46:19.314 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.314 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:19.314 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.314 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:19.314 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.145423+0000", 2024-03-20T09:46:19.315 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.315 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.315 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.315 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:19.315 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.145423+0000", 2024-03-20T09:46:19.315 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.315 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.315 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.315 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:19.315 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.145424+0000", 2024-03-20T09:46:19.315 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5100000000000001e-07 2024-03-20T09:46:19.315 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.315 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.315 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:19.315 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.145441+0000", 2024-03-20T09:46:19.315 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7757e-05 2024-03-20T09:46:19.316 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.316 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.316 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:19.316 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.145451+0000", 2024-03-20T09:46:19.316 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5540000000000003e-06 2024-03-20T09:46:19.316 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.316 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.316 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:19.316 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.145456+0000", 2024-03-20T09:46:19.316 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2089999999999996e-06 2024-03-20T09:46:19.316 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.316 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.316 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:19.316 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.169525+0000", 2024-03-20T09:46:19.316 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.024068790999999999 2024-03-20T09:46:19.316 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.317 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.317 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:19.317 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.169656+0000", 2024-03-20T09:46:19.317 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013127000000000001 2024-03-20T09:46:19.317 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.317 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.317 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,11", 2024-03-20T09:46:19.317 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.169797+0000", 2024-03-20T09:46:19.317 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00014071099999999999 2024-03-20T09:46:19.317 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.317 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.317 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:19.317 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.178789+0000", 2024-03-20T09:46:19.317 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0089917069999999998 2024-03-20T09:46:19.317 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.318 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.318 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:19.318 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.195371+0000", 2024-03-20T09:46:19.318 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.016581953999999999 2024-03-20T09:46:19.318 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.318 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.318 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:19.318 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.397219+0000", 2024-03-20T09:46:19.318 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20184849199999999 2024-03-20T09:46:19.318 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.318 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.318 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:19.318 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.397301+0000", 2024-03-20T09:46:19.318 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.1580000000000002e-05 2024-03-20T09:46:19.318 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.319 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.319 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:19.319 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.397364+0000", 2024-03-20T09:46:19.319 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3338999999999998e-05 2024-03-20T09:46:19.319 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.319 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:19.319 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.319 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.319 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.319 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:126260 9.8 9:11562cb1:::benchmark_data_smithi122_210164_object126259:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1098)", 2024-03-20T09:46:19.319 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:41:59.171581+0000", 2024-03-20T09:46:19.319 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 260.08557103599998, 2024-03-20T09:46:19.319 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.225880312, 2024-03-20T09:46:19.320 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:19.320 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:19.320 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:19.320 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:46:19.320 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:19.320 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 126260 2024-03-20T09:46:19.320 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.320 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:19.320 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.320 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:19.320 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.171581+0000", 2024-03-20T09:46:19.320 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.320 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.320 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.321 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:19.321 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.171581+0000", 2024-03-20T09:46:19.321 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.321 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.321 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.321 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:19.321 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.171582+0000", 2024-03-20T09:46:19.321 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3969999999999999e-06 2024-03-20T09:46:19.321 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.321 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.321 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:19.321 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.171610+0000", 2024-03-20T09:46:19.321 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.7991e-05 2024-03-20T09:46:19.321 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.322 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.322 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:19.322 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.171625+0000", 2024-03-20T09:46:19.322 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4790000000000001e-05 2024-03-20T09:46:19.322 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.322 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.322 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:19.322 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.171633+0000", 2024-03-20T09:46:19.322 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9489999999999996e-06 2024-03-20T09:46:19.322 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.322 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.322 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:19.322 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.174434+0000", 2024-03-20T09:46:19.322 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0028006789999999999 2024-03-20T09:46:19.322 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.323 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.323 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:19.323 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.174564+0000", 2024-03-20T09:46:19.323 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013007200000000001 2024-03-20T09:46:19.323 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.323 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.323 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,11", 2024-03-20T09:46:19.323 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.174699+0000", 2024-03-20T09:46:19.323 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013544699999999999 2024-03-20T09:46:19.323 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.323 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.323 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:19.323 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.195096+0000", 2024-03-20T09:46:19.323 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.020396705000000001 2024-03-20T09:46:19.323 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.323 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.324 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:19.324 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.199947+0000", 2024-03-20T09:46:19.324 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0048511170000000003 2024-03-20T09:46:19.324 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.324 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.324 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:19.324 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.397393+0000", 2024-03-20T09:46:19.324 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.197445803 2024-03-20T09:46:19.324 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.324 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.324 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:19.324 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.397424+0000", 2024-03-20T09:46:19.324 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1341999999999999e-05 2024-03-20T09:46:19.324 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.324 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.324 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:19.325 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.397461+0000", 2024-03-20T09:46:19.325 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7020000000000001e-05 2024-03-20T09:46:19.325 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.325 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:19.325 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.325 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.325 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.325 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:126266 9.8 9:10df459b:::benchmark_data_smithi122_210164_object126265:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1098)", 2024-03-20T09:46:19.325 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:41:59.180692+0000", 2024-03-20T09:46:19.325 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 260.07646004100002, 2024-03-20T09:46:19.325 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.237022861, 2024-03-20T09:46:19.325 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:19.325 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:19.325 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:19.325 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:46:19.325 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:19.326 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 126266 2024-03-20T09:46:19.326 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.326 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:19.326 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.326 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:19.326 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.180692+0000", 2024-03-20T09:46:19.326 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.326 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.326 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.326 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:19.326 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.180692+0000", 2024-03-20T09:46:19.326 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.327 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.327 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.327 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:19.327 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.180693+0000", 2024-03-20T09:46:19.327 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.592e-06 2024-03-20T09:46:19.327 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.327 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.327 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:19.327 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.180712+0000", 2024-03-20T09:46:19.327 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8862e-05 2024-03-20T09:46:19.327 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.327 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.327 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:19.327 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.180729+0000", 2024-03-20T09:46:19.327 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7102999999999999e-05 2024-03-20T09:46:19.328 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.328 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.328 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:19.328 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.180738+0000", 2024-03-20T09:46:19.328 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9360000000000005e-06 2024-03-20T09:46:19.328 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.328 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.328 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:19.328 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.195444+0000", 2024-03-20T09:46:19.328 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.014706032000000001 2024-03-20T09:46:19.328 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.328 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.328 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:19.328 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.195558+0000", 2024-03-20T09:46:19.329 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000113338 2024-03-20T09:46:19.329 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.329 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.329 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,11", 2024-03-20T09:46:19.329 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.195661+0000", 2024-03-20T09:46:19.329 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000103167 2024-03-20T09:46:19.329 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.329 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.329 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:19.329 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.200493+0000", 2024-03-20T09:46:19.329 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0048325490000000002 2024-03-20T09:46:19.329 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.329 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.330 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:19.330 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.200844+0000", 2024-03-20T09:46:19.330 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00035064999999999999 2024-03-20T09:46:19.330 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.330 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.330 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:19.330 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.417602+0000", 2024-03-20T09:46:19.330 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.21675767200000001 2024-03-20T09:46:19.330 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.330 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.330 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:19.330 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.417656+0000", 2024-03-20T09:46:19.330 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4558000000000002e-05 2024-03-20T09:46:19.330 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.330 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.331 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:19.331 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.417715+0000", 2024-03-20T09:46:19.331 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8402000000000001e-05 2024-03-20T09:46:19.331 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.331 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:19.331 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.331 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.331 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.331 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:126282 9.8 9:1febbbcb:::benchmark_data_smithi122_210164_object126281:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1098)", 2024-03-20T09:46:19.331 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:41:59.210497+0000", 2024-03-20T09:46:19.331 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 260.04665516199998, 2024-03-20T09:46:19.331 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20739027800000001, 2024-03-20T09:46:19.331 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:19.331 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:19.331 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:19.332 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:46:19.332 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:19.332 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 126282 2024-03-20T09:46:19.332 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.332 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:19.332 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.332 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:19.332 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.210497+0000", 2024-03-20T09:46:19.332 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.332 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.332 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.332 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:19.332 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.210497+0000", 2024-03-20T09:46:19.332 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.333 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.333 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.333 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:19.333 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.210498+0000", 2024-03-20T09:46:19.333 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5880000000000001e-06 2024-03-20T09:46:19.333 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.333 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.333 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:19.333 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.210519+0000", 2024-03-20T09:46:19.333 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0211999999999999e-05 2024-03-20T09:46:19.333 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.333 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.333 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:19.333 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.210531+0000", 2024-03-20T09:46:19.333 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2884999999999999e-05 2024-03-20T09:46:19.333 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.334 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.334 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:19.334 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.210537+0000", 2024-03-20T09:46:19.334 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8510000000000004e-06 2024-03-20T09:46:19.334 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.334 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.334 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:19.334 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.210597+0000", 2024-03-20T09:46:19.334 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0133000000000002e-05 2024-03-20T09:46:19.334 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.334 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.334 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:19.334 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.210750+0000", 2024-03-20T09:46:19.334 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015294500000000001 2024-03-20T09:46:19.334 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.335 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.335 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,11", 2024-03-20T09:46:19.335 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.210906+0000", 2024-03-20T09:46:19.335 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00015554499999999999 2024-03-20T09:46:19.335 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.335 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.335 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:19.335 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.214843+0000", 2024-03-20T09:46:19.335 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0039371179999999999 2024-03-20T09:46:19.335 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.335 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.335 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:19.335 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.215938+0000", 2024-03-20T09:46:19.335 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001095026 2024-03-20T09:46:19.335 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.335 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.336 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:19.336 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.417774+0000", 2024-03-20T09:46:19.336 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20183589099999999 2024-03-20T09:46:19.336 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.336 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.336 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:19.336 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.417830+0000", 2024-03-20T09:46:19.336 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6441999999999999e-05 2024-03-20T09:46:19.336 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.336 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.336 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:19.336 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:41:59.417887+0000", 2024-03-20T09:46:19.336 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6641999999999997e-05 2024-03-20T09:46:19.336 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.336 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:19.337 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.337 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.337 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.337 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:179020 9.3 9:c8b2e0b2:::benchmark_data_smithi122_210164_object46790:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1115)", 2024-03-20T09:46:19.337 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:43:01.328331+0000", 2024-03-20T09:46:19.337 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 197.928820763, 2024-03-20T09:46:19.337 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.204369245, 2024-03-20T09:46:19.337 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:19.337 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:19.337 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:19.337 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:46:19.337 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:19.337 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 179020 2024-03-20T09:46:19.337 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.337 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:19.337 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.338 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:19.338 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:01.328331+0000", 2024-03-20T09:46:19.338 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.338 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.338 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.338 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:19.338 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:01.328331+0000", 2024-03-20T09:46:19.338 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.338 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.338 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.338 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:19.338 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:01.328332+0000", 2024-03-20T09:46:19.338 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8700000000000004e-07 2024-03-20T09:46:19.339 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.339 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.339 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:19.339 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:01.328337+0000", 2024-03-20T09:46:19.339 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1989999999999997e-06 2024-03-20T09:46:19.339 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.339 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.339 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:19.339 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:01.328349+0000", 2024-03-20T09:46:19.339 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1742e-05 2024-03-20T09:46:19.339 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.339 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.339 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:19.339 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:01.328358+0000", 2024-03-20T09:46:19.340 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1519999999999998e-06 2024-03-20T09:46:19.340 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.340 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.340 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:19.340 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:01.332413+0000", 2024-03-20T09:46:19.340 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040543999999999997 2024-03-20T09:46:19.340 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.340 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.340 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:19.340 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:01.332535+0000", 2024-03-20T09:46:19.340 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000122881 2024-03-20T09:46:19.340 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.340 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.340 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 3,5", 2024-03-20T09:46:19.340 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:01.332650+0000", 2024-03-20T09:46:19.340 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000115028 2024-03-20T09:46:19.341 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.341 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.341 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:19.341 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:01.337721+0000", 2024-03-20T09:46:19.341 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0050703850000000002 2024-03-20T09:46:19.341 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.341 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.341 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:19.341 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:01.338807+0000", 2024-03-20T09:46:19.341 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0010861359999999999 2024-03-20T09:46:19.341 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.341 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.341 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:19.341 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:01.532603+0000", 2024-03-20T09:46:19.341 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.19379576500000001 2024-03-20T09:46:19.342 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.342 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.342 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:19.342 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:01.532656+0000", 2024-03-20T09:46:19.342 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3199999999999999e-05 2024-03-20T09:46:19.342 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.342 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.342 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:19.342 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:43:01.532700+0000", 2024-03-20T09:46:19.342 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4469999999999999e-05 2024-03-20T09:46:19.342 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.342 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:19.342 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.342 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.342 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.343 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDScrubReserve(1.9,QREQUEST,e:1180,from:7,reservation_nonce:3)", 2024-03-20T09:46:19.343 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:46:13.594607+0000", 2024-03-20T09:46:19.343 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.6625446969999995, 2024-03-20T09:46:19.343 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000155033, 2024-03-20T09:46:19.343 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:19.343 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-03-20T09:46:19.343 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:19.343 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.343 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:19.343 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:46:13.594607+0000", 2024-03-20T09:46:19.343 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.343 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.343 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.343 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:19.343 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:46:13.594607+0000", 2024-03-20T09:46:19.344 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.344 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.344 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.344 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:19.344 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:46:13.594609+0000", 2024-03-20T09:46:19.344 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4720000000000001e-06 2024-03-20T09:46:19.344 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.344 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.344 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:19.344 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:46:13.594616+0000", 2024-03-20T09:46:19.344 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6380000000000004e-06 2024-03-20T09:46:19.344 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.344 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.344 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:19.344 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:46:13.594633+0000", 2024-03-20T09:46:19.344 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6480000000000001e-05 2024-03-20T09:46:19.345 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.345 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.345 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:19.345 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:46:13.594643+0000", 2024-03-20T09:46:19.345 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0062e-05 2024-03-20T09:46:19.345 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.345 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.345 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:19.345 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:46:13.594705+0000", 2024-03-20T09:46:19.345 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2384000000000004e-05 2024-03-20T09:46:19.345 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.345 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.345 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:19.345 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:46:13.594724+0000", 2024-03-20T09:46:19.346 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9012e-05 2024-03-20T09:46:19.346 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.346 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.346 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:19.346 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:46:13.594762+0000", 2024-03-20T09:46:19.346 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7985000000000003e-05 2024-03-20T09:46:19.346 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.346 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:19.346 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.346 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.346 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.346 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "replica_scrub(pg: 1.9,from:0'0,to:0'0,epoch:1180/1179,start:1:90000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-03-20T09:46:19.346 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:46:13.669400+0000", 2024-03-20T09:46:19.347 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.587751624, 2024-03-20T09:46:19.347 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000101507, 2024-03-20T09:46:19.347 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:19.347 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-03-20T09:46:19.347 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:19.347 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.347 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:19.347 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:46:13.669400+0000", 2024-03-20T09:46:19.347 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.347 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.347 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.349 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:19.349 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:46:13.669400+0000", 2024-03-20T09:46:19.349 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.349 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.349 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.349 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:19.349 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:46:13.669401+0000", 2024-03-20T09:46:19.349 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5600000000000005e-07 2024-03-20T09:46:19.349 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.349 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.349 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:19.349 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:46:13.669406+0000", 2024-03-20T09:46:19.350 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.293e-06 2024-03-20T09:46:19.350 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.350 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.350 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:19.350 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:46:13.669417+0000", 2024-03-20T09:46:19.350 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1059e-05 2024-03-20T09:46:19.350 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.350 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.350 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:19.350 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:46:13.669425+0000", 2024-03-20T09:46:19.350 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4259999999999997e-06 2024-03-20T09:46:19.350 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.350 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.351 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:19.351 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:46:13.669473+0000", 2024-03-20T09:46:19.351 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7794999999999999e-05 2024-03-20T09:46:19.351 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.351 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.351 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:19.351 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:46:13.669482+0000", 2024-03-20T09:46:19.351 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0799999999999995e-06 2024-03-20T09:46:19.351 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.351 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.351 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:19.351 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:46:13.669502+0000", 2024-03-20T09:46:19.351 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0197999999999999e-05 2024-03-20T09:46:19.351 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.352 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:19.352 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.352 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.352 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.352 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "MOSDScrubReserve(1.9,RELEASE,e:1180,from:7,reservation_nonce:0)", 2024-03-20T09:46:19.352 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:46:13.752803+0000", 2024-03-20T09:46:19.352 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 5.5043490159999999, 2024-03-20T09:46:19.352 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000145289, 2024-03-20T09:46:19.352 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:19.352 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "started", 2024-03-20T09:46:19.352 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:19.352 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.352 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:19.352 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:46:13.752803+0000", 2024-03-20T09:46:19.352 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.352 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.353 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.353 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:19.353 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:46:13.752803+0000", 2024-03-20T09:46:19.353 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:19.353 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.353 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.353 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:19.353 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:46:13.752803+0000", 2024-03-20T09:46:19.353 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6899999999999999e-07 2024-03-20T09:46:19.353 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.353 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.353 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:19.353 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:46:13.752806+0000", 2024-03-20T09:46:19.353 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.9040000000000001e-06 2024-03-20T09:46:19.353 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.354 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.354 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:19.354 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:46:13.752816+0000", 2024-03-20T09:46:19.354 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0178e-05 2024-03-20T09:46:19.354 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.354 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.354 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:19.354 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:46:13.752825+0000", 2024-03-20T09:46:19.354 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0880000000000001e-06 2024-03-20T09:46:19.354 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.354 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.354 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:19.354 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:46:13.752915+0000", 2024-03-20T09:46:19.354 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9883000000000007e-05 2024-03-20T09:46:19.355 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.355 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.355 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:19.355 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:46:13.752930+0000", 2024-03-20T09:46:19.355 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5156e-05 2024-03-20T09:46:19.355 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:19.355 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:19.355 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:19.355 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:46:13.752948+0000", 2024-03-20T09:46:19.355 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7810999999999999e-05 2024-03-20T09:46:19.355 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.355 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:19.355 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.355 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:19.355 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:19.355 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:46:19.920 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_ops_in_flight 2024-03-20T09:46:19.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:19 smithi026 ceph-mon[68309]: pgmap v2873: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:19.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:19 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:46:19.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:19 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:46:19.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:19 smithi026 ceph-mon[68309]: 1.18 scrub starts 2024-03-20T09:46:20.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:19 smithi115 ceph-mon[72476]: pgmap v2873: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:20.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:19 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:46:20.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:19 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:46:20.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:19 smithi115 ceph-mon[72476]: 1.18 scrub starts 2024-03-20T09:46:20.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:19 smithi122 ceph-mon[73495]: pgmap v2873: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:20.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:19 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:46:20.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:19 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:46:20.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:19 smithi122 ceph-mon[73495]: 1.18 scrub starts 2024-03-20T09:46:20.183 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:46:20.825 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.9 to 1 (10000) 2024-03-20T09:46:21.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:20 smithi115 ceph-mon[72476]: 1.18 scrub ok 2024-03-20T09:46:21.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:20 smithi115 ceph-mon[72476]: osdmap e1183: 12 total, 12 up, 12 in 2024-03-20T09:46:21.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:20 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/465129974' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-03-20T09:46:21.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:20 smithi122 ceph-mon[73495]: 1.18 scrub ok 2024-03-20T09:46:21.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:20 smithi122 ceph-mon[73495]: osdmap e1183: 12 total, 12 up, 12 in 2024-03-20T09:46:21.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:20 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/465129974' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-03-20T09:46:21.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:20 smithi026 ceph-mon[68309]: 1.18 scrub ok 2024-03-20T09:46:21.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:20 smithi026 ceph-mon[68309]: osdmap e1183: 12 total, 12 up, 12 in 2024-03-20T09:46:21.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:20 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/465129974' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]: dispatch 2024-03-20T09:46:21.699 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd primary-affinity 9 1 2024-03-20T09:46:21.917 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:46:21.917 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:46:21.917 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-03-20T09:46:21.917 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:46:21.932 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:21 smithi122 ceph-mon[73495]: pgmap v2875: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:21.933 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:21 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/465129974' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-03-20T09:46:21.933 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:21 smithi122 ceph-mon[73495]: osdmap e1184: 12 total, 12 up, 12 in 2024-03-20T09:46:21.933 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:21 smithi122 ceph-mon[73495]: 1.f scrub starts 2024-03-20T09:46:21.933 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:21 smithi122 ceph-mon[73495]: 1.f scrub ok 2024-03-20T09:46:21.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:21 smithi026 ceph-mon[68309]: pgmap v2875: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:21.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:21 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/465129974' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-03-20T09:46:21.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:21 smithi026 ceph-mon[68309]: osdmap e1184: 12 total, 12 up, 12 in 2024-03-20T09:46:21.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:21 smithi026 ceph-mon[68309]: 1.f scrub starts 2024-03-20T09:46:21.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:21 smithi026 ceph-mon[68309]: 1.f scrub ok 2024-03-20T09:46:21.984 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:46:22.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:21 smithi115 ceph-mon[72476]: pgmap v2875: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:22.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:21 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/465129974' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-03-20T09:46:22.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:21 smithi115 ceph-mon[72476]: osdmap e1184: 12 total, 12 up, 12 in 2024-03-20T09:46:22.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:21 smithi115 ceph-mon[72476]: 1.f scrub starts 2024-03-20T09:46:22.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:21 smithi115 ceph-mon[72476]: 1.f scrub ok 2024-03-20T09:46:22.607 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_blocked_ops 2024-03-20T09:46:22.871 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:46:23.102 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:22 smithi122 ceph-mon[73495]: 1.10 scrub starts 2024-03-20T09:46:23.103 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:22 smithi122 ceph-mon[73495]: 1.10 scrub ok 2024-03-20T09:46:23.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:22 smithi115 ceph-mon[72476]: 1.10 scrub starts 2024-03-20T09:46:23.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:22 smithi115 ceph-mon[72476]: 1.10 scrub ok 2024-03-20T09:46:23.147 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:22 smithi026 ceph-mon[68309]: 1.10 scrub starts 2024-03-20T09:46:23.147 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:22 smithi026 ceph-mon[68309]: 1.10 scrub ok 2024-03-20T09:46:24.123 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:23 smithi122 ceph-mon[73495]: pgmap v2877: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:24.123 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:23 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:46:24.123 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:23 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:46:24.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:23 smithi115 ceph-mon[72476]: pgmap v2877: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:24.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:23 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:46:24.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:23 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:46:24.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:23 smithi026 ceph-mon[68309]: pgmap v2877: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:24.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:23 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:46:24.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:23 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:46:24.692 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:46:24.692 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:46:24.692 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-03-20T09:46:24.692 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-03-20T09:46:24.692 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:46:24.847 INFO:teuthology.orchestra.run.smithi026.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-03-20T09:46:25.049 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:24 smithi122 ceph-mon[73495]: osdmap e1185: 12 total, 12 up, 12 in 2024-03-20T09:46:25.049 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:24 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2937517818' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-03-20T09:46:25.049 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:24 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:46:25.049 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:24 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:46:25.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:24 smithi115 ceph-mon[72476]: osdmap e1185: 12 total, 12 up, 12 in 2024-03-20T09:46:25.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:24 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2937517818' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-03-20T09:46:25.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:24 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:46:25.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:24 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:46:25.196 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:24 smithi026 ceph-mon[68309]: osdmap e1185: 12 total, 12 up, 12 in 2024-03-20T09:46:25.196 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:24 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2937517818' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]: dispatch 2024-03-20T09:46:25.197 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:24 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:46:25.197 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:24 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:46:25.383 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.11 dump_historic_ops 2024-03-20T09:46:25.616 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 11 1 2024-03-20T09:46:25.646 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:46:25.899 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:46:26.128 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:25 smithi122 ceph-mon[73495]: pgmap v2879: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:26.128 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:25 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2937517818' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-03-20T09:46:26.128 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:25 smithi122 ceph-mon[73495]: osdmap e1186: 12 total, 12 up, 12 in 2024-03-20T09:46:26.142 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:25 smithi026 ceph-mon[68309]: pgmap v2879: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:26.142 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:25 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2937517818' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-03-20T09:46:26.142 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:25 smithi026 ceph-mon[68309]: osdmap e1186: 12 total, 12 up, 12 in 2024-03-20T09:46:26.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:25 smithi115 ceph-mon[72476]: pgmap v2879: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:26.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:25 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2937517818' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-03-20T09:46:26.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:25 smithi115 ceph-mon[72476]: osdmap e1186: 12 total, 12 up, 12 in 2024-03-20T09:46:27.316 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:46:27.316 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-03-20T09:46:27.316 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-03-20T09:46:27.316 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-03-20T09:46:27.317 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.317 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:186750 8.f 8:f9ac61d7:::benchmark_data_smithi122_197775_object63446:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1031)", 2024-03-20T09:46:27.317 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:38:44.301265+0000", 2024-03-20T09:46:27.317 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 463.025473347, 2024-03-20T09:46:27.317 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.21922650900000001, 2024-03-20T09:46:27.317 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:27.317 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:27.317 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:27.317 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:27.318 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:27.318 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 186750 2024-03-20T09:46:27.318 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.318 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:27.318 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.318 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:27.318 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:38:44.301265+0000", 2024-03-20T09:46:27.319 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.319 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.319 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.319 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:27.319 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:38:44.301265+0000", 2024-03-20T09:46:27.319 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.319 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.319 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.319 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:27.320 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:38:44.301266+0000", 2024-03-20T09:46:27.320 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3300000000000003e-07 2024-03-20T09:46:27.320 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.320 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.320 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:27.320 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:38:44.301268+0000", 2024-03-20T09:46:27.320 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8090000000000001e-06 2024-03-20T09:46:27.320 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.321 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.321 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:27.321 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:38:44.301275+0000", 2024-03-20T09:46:27.321 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4230000000000002e-06 2024-03-20T09:46:27.321 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.321 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.321 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:27.321 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:38:44.301280+0000", 2024-03-20T09:46:27.321 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4739999999999996e-06 2024-03-20T09:46:27.322 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.322 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.322 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:27.322 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:38:44.304480+0000", 2024-03-20T09:46:27.322 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0031995700000000001 2024-03-20T09:46:27.322 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.322 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.322 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:27.322 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:38:44.304585+0000", 2024-03-20T09:46:27.323 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105568 2024-03-20T09:46:27.323 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.323 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.323 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,2", 2024-03-20T09:46:27.323 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:38:44.304690+0000", 2024-03-20T09:46:27.323 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104227 2024-03-20T09:46:27.323 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.323 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.323 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:27.323 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:38:44.314461+0000", 2024-03-20T09:46:27.324 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0097715920000000008 2024-03-20T09:46:27.324 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.324 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.324 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.324 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:38:44.315395+0000", 2024-03-20T09:46:27.324 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00093366800000000002 2024-03-20T09:46:27.324 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.324 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.324 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.324 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:38:44.520408+0000", 2024-03-20T09:46:27.324 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.20501339700000001 2024-03-20T09:46:27.325 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.325 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.325 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:27.325 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:38:44.520447+0000", 2024-03-20T09:46:27.325 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9131000000000002e-05 2024-03-20T09:46:27.325 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.325 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.325 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:27.325 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:38:44.520492+0000", 2024-03-20T09:46:27.325 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4217e-05 2024-03-20T09:46:27.325 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.326 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:27.326 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.326 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.326 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.326 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:201711 8.f 8:fdd3fd66:::benchmark_data_smithi122_197775_object78407:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1036)", 2024-03-20T09:46:27.326 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:39:02.845316+0000", 2024-03-20T09:46:27.326 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 444.48142277800002, 2024-03-20T09:46:27.326 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.227407364, 2024-03-20T09:46:27.327 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:27.327 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:27.327 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:27.327 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:27.327 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:27.327 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 201711 2024-03-20T09:46:27.327 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.327 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:27.327 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.327 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:27.328 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:02.845316+0000", 2024-03-20T09:46:27.328 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.328 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.328 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.328 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:27.328 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:02.845316+0000", 2024-03-20T09:46:27.328 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.328 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.328 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.328 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:27.328 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:02.845318+0000", 2024-03-20T09:46:27.329 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8479999999999999e-06 2024-03-20T09:46:27.329 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.329 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.329 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:27.329 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:02.845325+0000", 2024-03-20T09:46:27.329 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8050000000000007e-06 2024-03-20T09:46:27.329 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.329 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.329 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:27.329 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:02.845344+0000", 2024-03-20T09:46:27.330 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8813000000000001e-05 2024-03-20T09:46:27.330 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.330 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.330 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:27.330 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:02.845353+0000", 2024-03-20T09:46:27.330 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2420000000000006e-06 2024-03-20T09:46:27.330 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.330 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.330 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:27.330 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:02.846076+0000", 2024-03-20T09:46:27.331 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00072261000000000005 2024-03-20T09:46:27.331 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.331 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.331 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:27.331 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:02.846185+0000", 2024-03-20T09:46:27.331 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000108792 2024-03-20T09:46:27.331 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.331 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.331 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,2", 2024-03-20T09:46:27.331 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:02.846297+0000", 2024-03-20T09:46:27.331 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011248599999999999 2024-03-20T09:46:27.332 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.332 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.332 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:27.332 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:02.855690+0000", 2024-03-20T09:46:27.332 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0093927549999999992 2024-03-20T09:46:27.332 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.332 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.332 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.332 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:02.855901+0000", 2024-03-20T09:46:27.332 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000211072 2024-03-20T09:46:27.332 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.333 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.333 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.333 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:03.072657+0000", 2024-03-20T09:46:27.333 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.216755694 2024-03-20T09:46:27.333 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.333 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.333 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:27.333 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:03.072692+0000", 2024-03-20T09:46:27.333 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5269999999999999e-05 2024-03-20T09:46:27.333 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.333 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.334 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:27.334 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:03.072723+0000", 2024-03-20T09:46:27.334 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0976999999999998e-05 2024-03-20T09:46:27.334 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.334 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:27.334 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.334 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.334 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.334 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:201714 8.f 8:fc60ef14:::benchmark_data_smithi122_197775_object78410:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1036)", 2024-03-20T09:46:27.335 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:39:02.847255+0000", 2024-03-20T09:46:27.335 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 444.47948362, 2024-03-20T09:46:27.335 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.22559748399999999, 2024-03-20T09:46:27.335 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:27.335 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:27.335 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:27.335 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:27.335 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:27.335 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 201714 2024-03-20T09:46:27.335 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.335 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:27.335 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.336 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:27.336 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:02.847255+0000", 2024-03-20T09:46:27.336 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.336 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.336 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.336 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:27.336 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:02.847255+0000", 2024-03-20T09:46:27.336 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.336 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.336 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.337 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:27.337 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:02.847256+0000", 2024-03-20T09:46:27.337 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.327e-06 2024-03-20T09:46:27.337 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.337 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.337 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:27.337 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:02.847261+0000", 2024-03-20T09:46:27.337 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0560000000000004e-06 2024-03-20T09:46:27.337 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.338 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.338 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:27.338 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:02.847275+0000", 2024-03-20T09:46:27.338 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3210000000000001e-05 2024-03-20T09:46:27.338 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.338 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.338 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:27.338 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:02.847293+0000", 2024-03-20T09:46:27.338 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8963000000000001e-05 2024-03-20T09:46:27.338 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.339 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.339 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:27.339 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:02.851292+0000", 2024-03-20T09:46:27.339 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0039983689999999999 2024-03-20T09:46:27.339 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.339 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.339 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:27.340 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:02.851403+0000", 2024-03-20T09:46:27.340 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000111477 2024-03-20T09:46:27.340 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.340 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.340 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,2", 2024-03-20T09:46:27.340 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:02.851512+0000", 2024-03-20T09:46:27.340 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000109009 2024-03-20T09:46:27.340 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.340 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.340 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:27.340 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:02.855920+0000", 2024-03-20T09:46:27.340 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0044078370000000004 2024-03-20T09:46:27.341 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.341 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.341 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.341 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:02.863642+0000", 2024-03-20T09:46:27.341 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0077217919999999999 2024-03-20T09:46:27.341 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.341 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.341 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.341 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:03.072780+0000", 2024-03-20T09:46:27.341 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.209138518 2024-03-20T09:46:27.341 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.342 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.342 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:27.342 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:03.072814+0000", 2024-03-20T09:46:27.342 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3810000000000003e-05 2024-03-20T09:46:27.342 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.342 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.342 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:27.342 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:03.072852+0000", 2024-03-20T09:46:27.342 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8115999999999999e-05 2024-03-20T09:46:27.342 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.343 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:27.343 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.343 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.343 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.343 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:201728 8.f 8:fca71751:::benchmark_data_smithi122_197775_object78424:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1036)", 2024-03-20T09:46:27.343 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:39:02.858573+0000", 2024-03-20T09:46:27.343 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 444.46816510000002, 2024-03-20T09:46:27.343 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.21807142700000001, 2024-03-20T09:46:27.343 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:27.343 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:27.343 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:27.343 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:27.344 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:27.344 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 201728 2024-03-20T09:46:27.344 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.344 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:27.344 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.344 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:27.344 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:02.858573+0000", 2024-03-20T09:46:27.344 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.344 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.344 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.344 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:27.345 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:02.858573+0000", 2024-03-20T09:46:27.345 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.345 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.345 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.345 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:27.345 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:02.858574+0000", 2024-03-20T09:46:27.345 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8599999999999997e-07 2024-03-20T09:46:27.345 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.345 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.345 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:27.345 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:02.858579+0000", 2024-03-20T09:46:27.345 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0440000000000003e-06 2024-03-20T09:46:27.346 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.346 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.346 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:27.346 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:02.858592+0000", 2024-03-20T09:46:27.346 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2181000000000001e-05 2024-03-20T09:46:27.346 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.346 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.346 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:27.346 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:02.858599+0000", 2024-03-20T09:46:27.346 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6569999999999997e-06 2024-03-20T09:46:27.346 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.347 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.347 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:27.347 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:02.858655+0000", 2024-03-20T09:46:27.347 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5831000000000001e-05 2024-03-20T09:46:27.347 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.347 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.347 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:27.347 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:02.858763+0000", 2024-03-20T09:46:27.347 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010772600000000001 2024-03-20T09:46:27.347 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.347 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.347 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 1,2", 2024-03-20T09:46:27.348 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:02.858868+0000", 2024-03-20T09:46:27.348 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105369 2024-03-20T09:46:27.348 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.348 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.348 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:27.348 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:02.863768+0000", 2024-03-20T09:46:27.348 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0048998269999999998 2024-03-20T09:46:27.348 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.348 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.348 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.348 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:02.864993+0000", 2024-03-20T09:46:27.348 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001225432 2024-03-20T09:46:27.349 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.349 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.349 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.349 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:03.076549+0000", 2024-03-20T09:46:27.349 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.21155558199999999 2024-03-20T09:46:27.349 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.349 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.349 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:27.349 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:03.076603+0000", 2024-03-20T09:46:27.349 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4027000000000003e-05 2024-03-20T09:46:27.349 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.350 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.350 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:27.350 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:03.076645+0000", 2024-03-20T09:46:27.350 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1865e-05 2024-03-20T09:46:27.350 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.350 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:27.350 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.350 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.350 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.350 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:263230 9.5 9:ae9935be:::benchmark_data_smithi122_210164_object131000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1143)", 2024-03-20T09:46:27.350 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:44:39.116538+0000", 2024-03-20T09:46:27.350 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 108.210200683, 2024-03-20T09:46:27.351 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.57417216100000001, 2024-03-20T09:46:27.351 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:27.351 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:27.351 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:27.351 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:46:27.351 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:27.351 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 263230 2024-03-20T09:46:27.351 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.351 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:27.351 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.351 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:27.351 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.116538+0000", 2024-03-20T09:46:27.352 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.352 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.352 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.352 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:27.352 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.116538+0000", 2024-03-20T09:46:27.352 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.352 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.352 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.352 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:27.352 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.116540+0000", 2024-03-20T09:46:27.352 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7060000000000001e-06 2024-03-20T09:46:27.353 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.353 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.353 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:27.353 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.116547+0000", 2024-03-20T09:46:27.353 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4460000000000004e-06 2024-03-20T09:46:27.353 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.353 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.353 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:27.353 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.116565+0000", 2024-03-20T09:46:27.353 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7884999999999999e-05 2024-03-20T09:46:27.353 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.353 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.354 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:27.354 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.116575+0000", 2024-03-20T09:46:27.354 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.029e-05 2024-03-20T09:46:27.354 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.354 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.354 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:27.354 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.116637+0000", 2024-03-20T09:46:27.354 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1535e-05 2024-03-20T09:46:27.354 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.354 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.354 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:27.355 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.116770+0000", 2024-03-20T09:46:27.355 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013374599999999999 2024-03-20T09:46:27.355 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.355 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.355 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,7", 2024-03-20T09:46:27.355 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.116886+0000", 2024-03-20T09:46:27.355 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000115432 2024-03-20T09:46:27.355 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.355 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.355 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:27.355 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.121808+0000", 2024-03-20T09:46:27.355 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0049217599999999999 2024-03-20T09:46:27.356 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.356 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.356 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.356 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.123755+0000", 2024-03-20T09:46:27.356 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0019475079999999999 2024-03-20T09:46:27.356 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.356 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.356 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.356 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.690620+0000", 2024-03-20T09:46:27.356 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.56686519199999996 2024-03-20T09:46:27.356 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.356 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.357 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:27.357 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.690679+0000", 2024-03-20T09:46:27.357 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8267000000000003e-05 2024-03-20T09:46:27.357 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.357 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.357 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:27.357 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.690710+0000", 2024-03-20T09:46:27.357 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1393999999999998e-05 2024-03-20T09:46:27.357 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.357 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:27.357 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.357 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.358 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.358 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:263248 9.5 9:a3f2cb8c:::benchmark_data_smithi122_210164_object131018:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1143)", 2024-03-20T09:46:27.358 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:44:39.145647+0000", 2024-03-20T09:46:27.358 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 108.181091526, 2024-03-20T09:46:27.358 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.55290542099999995, 2024-03-20T09:46:27.358 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:27.358 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:27.358 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:27.359 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:46:27.359 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:27.359 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 263248 2024-03-20T09:46:27.359 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.359 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:27.359 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.359 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:27.359 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.145647+0000", 2024-03-20T09:46:27.359 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.359 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.359 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.360 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:27.360 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.145647+0000", 2024-03-20T09:46:27.360 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.360 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.360 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.360 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:27.360 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.145649+0000", 2024-03-20T09:46:27.360 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7e-06 2024-03-20T09:46:27.360 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.360 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.360 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:27.360 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.145657+0000", 2024-03-20T09:46:27.361 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4700000000000002e-06 2024-03-20T09:46:27.361 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.361 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.361 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:27.361 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.145673+0000", 2024-03-20T09:46:27.361 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6189999999999999e-05 2024-03-20T09:46:27.361 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.361 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.361 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:27.361 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.145684+0000", 2024-03-20T09:46:27.361 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.047e-05 2024-03-20T09:46:27.361 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.362 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.362 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:27.362 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.145761+0000", 2024-03-20T09:46:27.362 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7045999999999994e-05 2024-03-20T09:46:27.362 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.362 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.362 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:27.362 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.145877+0000", 2024-03-20T09:46:27.362 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011599600000000001 2024-03-20T09:46:27.362 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.362 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.362 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,7", 2024-03-20T09:46:27.363 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.145956+0000", 2024-03-20T09:46:27.363 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8619000000000001e-05 2024-03-20T09:46:27.363 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.363 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.363 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.363 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.151657+0000", 2024-03-20T09:46:27.363 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0057010699999999999 2024-03-20T09:46:27.363 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.363 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.363 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:27.363 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.151847+0000", 2024-03-20T09:46:27.363 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000190352 2024-03-20T09:46:27.364 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.364 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.364 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.364 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.698402+0000", 2024-03-20T09:46:27.364 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.54655473899999996 2024-03-20T09:46:27.364 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.364 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.364 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:27.364 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.698494+0000", 2024-03-20T09:46:27.364 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2658000000000006e-05 2024-03-20T09:46:27.364 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.364 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.365 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:27.365 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.698552+0000", 2024-03-20T09:46:27.365 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8111e-05 2024-03-20T09:46:27.365 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.365 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:27.365 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.365 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.365 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.365 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:263260 9.5 9:a469e368:::benchmark_data_smithi122_210164_object131030:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1143)", 2024-03-20T09:46:27.365 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:44:39.161311+0000", 2024-03-20T09:46:27.365 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 108.165427618, 2024-03-20T09:46:27.365 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.54362337699999996, 2024-03-20T09:46:27.365 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:27.366 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:27.366 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:27.366 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:46:27.366 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:27.366 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 263260 2024-03-20T09:46:27.366 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.366 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:27.366 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.366 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:27.366 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.161311+0000", 2024-03-20T09:46:27.366 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.367 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.367 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.367 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:27.367 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.161311+0000", 2024-03-20T09:46:27.367 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.367 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.367 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.367 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:27.367 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.161312+0000", 2024-03-20T09:46:27.367 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.578e-06 2024-03-20T09:46:27.367 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.367 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.368 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:27.368 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.161319+0000", 2024-03-20T09:46:27.368 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.3999999999999997e-06 2024-03-20T09:46:27.368 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.368 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.368 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:27.368 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.161334+0000", 2024-03-20T09:46:27.368 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5438e-05 2024-03-20T09:46:27.368 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.368 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.368 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:27.368 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.161342+0000", 2024-03-20T09:46:27.369 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0739999999999993e-06 2024-03-20T09:46:27.369 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.369 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.369 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:27.369 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.161412+0000", 2024-03-20T09:46:27.369 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9268000000000002e-05 2024-03-20T09:46:27.369 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.369 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.369 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:27.369 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.161536+0000", 2024-03-20T09:46:27.369 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000124738 2024-03-20T09:46:27.369 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.370 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.370 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,7", 2024-03-20T09:46:27.370 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.161668+0000", 2024-03-20T09:46:27.370 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013175400000000001 2024-03-20T09:46:27.370 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.370 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.370 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:27.370 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.166811+0000", 2024-03-20T09:46:27.370 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0051426450000000004 2024-03-20T09:46:27.370 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.370 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.370 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.371 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.169859+0000", 2024-03-20T09:46:27.371 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003047896 2024-03-20T09:46:27.371 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.371 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.371 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.371 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.704872+0000", 2024-03-20T09:46:27.371 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.53501338399999998 2024-03-20T09:46:27.371 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.371 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.371 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:27.371 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.704908+0000", 2024-03-20T09:46:27.371 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5574999999999997e-05 2024-03-20T09:46:27.371 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.372 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.372 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:27.372 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.704934+0000", 2024-03-20T09:46:27.372 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6627000000000001e-05 2024-03-20T09:46:27.372 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.372 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:27.372 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.372 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.372 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.372 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:263304 9.5 9:af0ddc27:::benchmark_data_smithi122_210164_object131074:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1143)", 2024-03-20T09:46:27.372 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:44:39.211902+0000", 2024-03-20T09:46:27.372 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 108.114836039, 2024-03-20T09:46:27.373 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.49313174199999998, 2024-03-20T09:46:27.373 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:27.373 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:27.373 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:27.373 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:46:27.373 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:27.373 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 263304 2024-03-20T09:46:27.373 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.373 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:27.373 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.373 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:27.373 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.211902+0000", 2024-03-20T09:46:27.374 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.374 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.374 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.374 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:27.374 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.211902+0000", 2024-03-20T09:46:27.374 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.374 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.374 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.374 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:27.374 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.211904+0000", 2024-03-20T09:46:27.374 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9379999999999999e-06 2024-03-20T09:46:27.374 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.375 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.375 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:27.375 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.211911+0000", 2024-03-20T09:46:27.375 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5039999999999999e-06 2024-03-20T09:46:27.375 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.375 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.375 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:27.375 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.211927+0000", 2024-03-20T09:46:27.375 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5970000000000001e-05 2024-03-20T09:46:27.375 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.375 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.375 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:27.375 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.211935+0000", 2024-03-20T09:46:27.376 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0269999999999995e-06 2024-03-20T09:46:27.376 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.376 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.376 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:27.376 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.211992+0000", 2024-03-20T09:46:27.376 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7240000000000001e-05 2024-03-20T09:46:27.376 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.376 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.376 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:27.376 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.212112+0000", 2024-03-20T09:46:27.376 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000120264 2024-03-20T09:46:27.376 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.376 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.377 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,7", 2024-03-20T09:46:27.377 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.212225+0000", 2024-03-20T09:46:27.377 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000112957 2024-03-20T09:46:27.377 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.377 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.377 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:27.377 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.221399+0000", 2024-03-20T09:46:27.377 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0091733530000000004 2024-03-20T09:46:27.377 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.377 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.377 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.377 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.221446+0000", 2024-03-20T09:46:27.377 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7085000000000001e-05 2024-03-20T09:46:27.378 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.378 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.378 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.378 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.704971+0000", 2024-03-20T09:46:27.378 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.48352506000000001 2024-03-20T09:46:27.378 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.378 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.378 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:27.378 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.705009+0000", 2024-03-20T09:46:27.379 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8279999999999999e-05 2024-03-20T09:46:27.379 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.379 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.379 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:27.379 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.705034+0000", 2024-03-20T09:46:27.379 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5063999999999999e-05 2024-03-20T09:46:27.379 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.379 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:27.379 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.379 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.379 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.379 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:263308 9.5 9:a79570f1:::benchmark_data_smithi122_210164_object131078:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1143)", 2024-03-20T09:46:27.380 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:44:39.214254+0000", 2024-03-20T09:46:27.380 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 108.11248419, 2024-03-20T09:46:27.380 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.49089385800000002, 2024-03-20T09:46:27.380 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:27.380 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:27.380 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:27.380 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:46:27.380 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:27.380 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 263308 2024-03-20T09:46:27.380 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.380 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:27.380 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.380 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:27.381 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.214254+0000", 2024-03-20T09:46:27.381 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.381 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.381 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.381 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:27.381 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.214254+0000", 2024-03-20T09:46:27.381 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.381 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.381 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.381 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:27.381 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.214255+0000", 2024-03-20T09:46:27.381 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.088e-06 2024-03-20T09:46:27.381 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.382 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.382 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:27.382 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.214261+0000", 2024-03-20T09:46:27.382 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1019999999999996e-06 2024-03-20T09:46:27.382 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.382 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.382 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:27.382 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.214275+0000", 2024-03-20T09:46:27.382 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4185999999999999e-05 2024-03-20T09:46:27.382 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.382 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.382 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:27.382 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.214292+0000", 2024-03-20T09:46:27.383 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7153999999999999e-05 2024-03-20T09:46:27.383 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.383 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.383 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:27.383 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.217095+0000", 2024-03-20T09:46:27.383 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002803327 2024-03-20T09:46:27.383 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.383 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.383 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:27.383 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.217224+0000", 2024-03-20T09:46:27.383 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012925400000000001 2024-03-20T09:46:27.384 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.384 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.384 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,7", 2024-03-20T09:46:27.384 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.217330+0000", 2024-03-20T09:46:27.384 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105832 2024-03-20T09:46:27.384 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.384 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.384 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:27.384 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.221487+0000", 2024-03-20T09:46:27.384 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0041571480000000003 2024-03-20T09:46:27.384 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.384 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.384 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.385 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.221871+0000", 2024-03-20T09:46:27.385 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000383624 2024-03-20T09:46:27.385 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.385 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.385 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.385 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.705070+0000", 2024-03-20T09:46:27.385 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.48319883699999999 2024-03-20T09:46:27.385 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.385 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.385 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:27.385 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.705111+0000", 2024-03-20T09:46:27.385 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0944e-05 2024-03-20T09:46:27.386 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.386 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.386 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:27.386 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.705148+0000", 2024-03-20T09:46:27.386 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7361999999999999e-05 2024-03-20T09:46:27.386 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.386 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:27.386 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.386 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.386 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.386 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:263316 9.5 9:aa208639:::benchmark_data_smithi122_210164_object131086:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1143)", 2024-03-20T09:46:27.386 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:44:39.225203+0000", 2024-03-20T09:46:27.386 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 108.101535743, 2024-03-20T09:46:27.387 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.48678930300000001, 2024-03-20T09:46:27.387 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:27.387 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:27.387 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:27.387 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:46:27.387 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:27.387 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 263316 2024-03-20T09:46:27.387 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.387 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:27.387 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.387 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:27.387 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.225203+0000", 2024-03-20T09:46:27.387 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.388 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.388 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.388 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:27.388 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.225203+0000", 2024-03-20T09:46:27.388 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.388 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.388 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.388 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:27.388 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.225204+0000", 2024-03-20T09:46:27.388 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.345e-06 2024-03-20T09:46:27.388 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.388 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.389 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:27.389 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.225210+0000", 2024-03-20T09:46:27.389 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6749999999999999e-06 2024-03-20T09:46:27.389 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.389 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.389 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:27.389 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.225224+0000", 2024-03-20T09:46:27.389 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3702e-05 2024-03-20T09:46:27.389 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.389 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.389 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:27.390 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.225231+0000", 2024-03-20T09:46:27.390 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7190000000000001e-06 2024-03-20T09:46:27.390 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.390 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.390 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:27.390 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.225301+0000", 2024-03-20T09:46:27.390 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9488999999999998e-05 2024-03-20T09:46:27.390 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.390 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.390 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:27.390 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.225389+0000", 2024-03-20T09:46:27.390 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.8494999999999999e-05 2024-03-20T09:46:27.391 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.391 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.391 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,7", 2024-03-20T09:46:27.391 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.225491+0000", 2024-03-20T09:46:27.391 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010148 2024-03-20T09:46:27.391 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.391 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.391 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.391 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.237541+0000", 2024-03-20T09:46:27.391 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.012050223000000001 2024-03-20T09:46:27.391 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.391 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.391 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:27.392 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.237678+0000", 2024-03-20T09:46:27.392 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013742800000000001 2024-03-20T09:46:27.392 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.392 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.392 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.392 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.711892+0000", 2024-03-20T09:46:27.392 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.474213775 2024-03-20T09:46:27.392 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.392 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.392 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:27.392 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.711949+0000", 2024-03-20T09:46:27.392 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6922999999999997e-05 2024-03-20T09:46:27.392 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.392 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.393 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:27.393 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.711992+0000", 2024-03-20T09:46:27.393 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3049000000000001e-05 2024-03-20T09:46:27.393 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.393 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:27.393 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.393 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.393 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.393 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:263318 9.5 9:aa1ca62c:::benchmark_data_smithi122_210164_object131088:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1143)", 2024-03-20T09:46:27.393 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:44:39.225297+0000", 2024-03-20T09:46:27.393 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 108.101441232, 2024-03-20T09:46:27.393 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.486842518, 2024-03-20T09:46:27.393 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:27.394 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:27.394 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:27.394 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:46:27.394 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:27.394 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 263318 2024-03-20T09:46:27.394 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.394 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:27.394 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.394 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:27.394 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.225297+0000", 2024-03-20T09:46:27.394 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.394 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.394 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.395 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:27.395 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.225297+0000", 2024-03-20T09:46:27.395 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.395 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.395 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.395 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:27.395 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.225298+0000", 2024-03-20T09:46:27.395 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2500000000000005e-07 2024-03-20T09:46:27.395 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.395 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.395 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:27.395 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.225302+0000", 2024-03-20T09:46:27.395 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.653e-06 2024-03-20T09:46:27.396 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.396 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.396 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:27.396 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.225310+0000", 2024-03-20T09:46:27.396 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5129999999999996e-06 2024-03-20T09:46:27.396 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.396 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.396 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:27.396 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.225317+0000", 2024-03-20T09:46:27.396 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.6889999999999998e-06 2024-03-20T09:46:27.396 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.396 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.396 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:27.396 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.229257+0000", 2024-03-20T09:46:27.397 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0039401920000000003 2024-03-20T09:46:27.397 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.397 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.397 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:27.397 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.229395+0000", 2024-03-20T09:46:27.397 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000137627 2024-03-20T09:46:27.397 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.397 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.397 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,7", 2024-03-20T09:46:27.397 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.229505+0000", 2024-03-20T09:46:27.397 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000110141 2024-03-20T09:46:27.397 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.397 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.398 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.398 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.237633+0000", 2024-03-20T09:46:27.398 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0081284900000000004 2024-03-20T09:46:27.398 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.398 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.398 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:27.398 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.237684+0000", 2024-03-20T09:46:27.398 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0909000000000003e-05 2024-03-20T09:46:27.398 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.398 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.398 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.398 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.712043+0000", 2024-03-20T09:46:27.398 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.47435855599999999 2024-03-20T09:46:27.399 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.399 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.399 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:27.399 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.712097+0000", 2024-03-20T09:46:27.399 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4500999999999999e-05 2024-03-20T09:46:27.399 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.399 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.399 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:27.399 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.712140+0000", 2024-03-20T09:46:27.399 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2521999999999997e-05 2024-03-20T09:46:27.399 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.399 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:27.399 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.400 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.400 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.400 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:263325 9.5 9:a553edef:::benchmark_data_smithi122_210164_object131095:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1143)", 2024-03-20T09:46:27.400 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:44:39.230446+0000", 2024-03-20T09:46:27.400 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 108.09629255, 2024-03-20T09:46:27.400 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.489886655, 2024-03-20T09:46:27.400 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:27.400 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:27.400 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:27.400 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:46:27.400 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:27.400 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 263325 2024-03-20T09:46:27.400 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.401 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:27.401 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.401 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:27.401 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.230446+0000", 2024-03-20T09:46:27.401 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.401 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.401 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.401 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:27.401 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.230446+0000", 2024-03-20T09:46:27.401 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.401 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.402 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.402 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:27.402 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.230447+0000", 2024-03-20T09:46:27.402 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2139999999999999e-06 2024-03-20T09:46:27.402 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.402 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.402 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:27.402 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.230452+0000", 2024-03-20T09:46:27.402 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.271e-06 2024-03-20T09:46:27.402 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.402 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.402 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:27.402 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.230468+0000", 2024-03-20T09:46:27.403 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5597e-05 2024-03-20T09:46:27.403 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.403 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.403 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:27.403 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.230476+0000", 2024-03-20T09:46:27.403 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7500000000000003e-06 2024-03-20T09:46:27.403 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.403 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.403 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:27.403 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.233343+0000", 2024-03-20T09:46:27.404 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0028674460000000001 2024-03-20T09:46:27.404 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.404 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.404 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:27.404 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.233445+0000", 2024-03-20T09:46:27.404 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00010144799999999999 2024-03-20T09:46:27.404 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.404 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.404 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,7", 2024-03-20T09:46:27.404 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.233520+0000", 2024-03-20T09:46:27.404 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5093999999999996e-05 2024-03-20T09:46:27.404 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.404 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.405 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:27.405 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.240632+0000", 2024-03-20T09:46:27.405 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0071119269999999997 2024-03-20T09:46:27.405 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.405 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.405 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.405 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.242638+0000", 2024-03-20T09:46:27.405 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.002005859 2024-03-20T09:46:27.405 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.405 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.405 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.405 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.720216+0000", 2024-03-20T09:46:27.405 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.47757854 2024-03-20T09:46:27.405 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.406 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.406 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:27.406 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.720276+0000", 2024-03-20T09:46:27.406 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0278000000000001e-05 2024-03-20T09:46:27.406 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.406 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.406 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:27.406 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.720333+0000", 2024-03-20T09:46:27.406 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6230999999999997e-05 2024-03-20T09:46:27.406 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.406 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:27.406 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.406 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.407 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.407 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:263339 9.5 9:a244f178:::benchmark_data_smithi122_210164_object131109:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1143)", 2024-03-20T09:46:27.407 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:44:39.250503+0000", 2024-03-20T09:46:27.407 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 108.07623529599999, 2024-03-20T09:46:27.407 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.46998140700000002, 2024-03-20T09:46:27.407 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:27.407 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:27.407 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:27.407 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:46:27.407 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:27.407 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 263339 2024-03-20T09:46:27.407 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.407 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:27.407 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.408 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:27.408 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.250503+0000", 2024-03-20T09:46:27.408 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.408 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.408 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.408 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:27.408 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.250503+0000", 2024-03-20T09:46:27.408 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.408 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.408 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.408 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:27.408 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.250505+0000", 2024-03-20T09:46:27.408 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.556e-06 2024-03-20T09:46:27.408 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.409 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.409 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:27.409 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.250510+0000", 2024-03-20T09:46:27.409 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4450000000000004e-06 2024-03-20T09:46:27.409 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.409 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.409 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:27.409 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.250524+0000", 2024-03-20T09:46:27.409 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3477e-05 2024-03-20T09:46:27.409 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.409 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.409 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:27.409 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.250532+0000", 2024-03-20T09:46:27.410 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4870000000000002e-06 2024-03-20T09:46:27.410 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.410 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.410 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:27.410 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.250594+0000", 2024-03-20T09:46:27.410 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1798999999999996e-05 2024-03-20T09:46:27.410 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.410 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.410 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:27.410 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.250713+0000", 2024-03-20T09:46:27.410 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000118547 2024-03-20T09:46:27.410 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.410 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.410 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,7", 2024-03-20T09:46:27.411 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.250824+0000", 2024-03-20T09:46:27.411 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000111619 2024-03-20T09:46:27.411 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.411 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.411 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:27.411 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.254774+0000", 2024-03-20T09:46:27.411 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0039499979999999997 2024-03-20T09:46:27.411 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.411 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.411 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.411 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.256647+0000", 2024-03-20T09:46:27.411 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0018731679999999999 2024-03-20T09:46:27.411 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.412 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.412 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.412 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.720392+0000", 2024-03-20T09:46:27.412 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.46374438400000001 2024-03-20T09:46:27.412 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.412 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.412 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:27.412 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.720446+0000", 2024-03-20T09:46:27.412 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4057000000000001e-05 2024-03-20T09:46:27.412 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.412 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.412 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:27.412 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.720485+0000", 2024-03-20T09:46:27.412 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.8869999999999999e-05 2024-03-20T09:46:27.413 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.413 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:27.413 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.413 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.413 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.413 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:263343 9.5 9:a175f92e:::benchmark_data_smithi122_210164_object131113:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1143)", 2024-03-20T09:46:27.413 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:44:39.256916+0000", 2024-03-20T09:46:27.413 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 108.069822876, 2024-03-20T09:46:27.413 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.46839756799999999, 2024-03-20T09:46:27.413 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:27.413 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:27.413 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:27.413 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:46:27.413 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:27.414 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 263343 2024-03-20T09:46:27.414 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.414 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:27.414 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.414 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:27.414 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.256916+0000", 2024-03-20T09:46:27.414 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.414 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.414 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.414 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:27.414 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.256916+0000", 2024-03-20T09:46:27.414 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.414 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.415 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.415 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:27.415 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.256917+0000", 2024-03-20T09:46:27.415 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4559999999999999e-06 2024-03-20T09:46:27.415 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.415 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.415 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:27.415 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.256922+0000", 2024-03-20T09:46:27.415 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.074e-06 2024-03-20T09:46:27.415 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.415 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.415 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:27.415 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.256937+0000", 2024-03-20T09:46:27.415 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4627000000000001e-05 2024-03-20T09:46:27.416 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.416 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.416 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:27.416 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.256945+0000", 2024-03-20T09:46:27.416 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.2940000000000006e-06 2024-03-20T09:46:27.416 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.416 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.416 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:27.416 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.257002+0000", 2024-03-20T09:46:27.416 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6638000000000002e-05 2024-03-20T09:46:27.416 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.416 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.416 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:27.416 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.257100+0000", 2024-03-20T09:46:27.417 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7858999999999999e-05 2024-03-20T09:46:27.417 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.417 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.417 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,7", 2024-03-20T09:46:27.417 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.257180+0000", 2024-03-20T09:46:27.417 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0615e-05 2024-03-20T09:46:27.417 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.417 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.417 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:27.417 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.261134+0000", 2024-03-20T09:46:27.417 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0039538660000000003 2024-03-20T09:46:27.417 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.417 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.417 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.418 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.261524+0000", 2024-03-20T09:46:27.418 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00038942099999999998 2024-03-20T09:46:27.418 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.418 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.418 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.418 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.725210+0000", 2024-03-20T09:46:27.418 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.46368641599999999 2024-03-20T09:46:27.418 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.418 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.418 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:27.418 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.725265+0000", 2024-03-20T09:46:27.418 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4595000000000002e-05 2024-03-20T09:46:27.418 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.418 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.419 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:27.419 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.725313+0000", 2024-03-20T09:46:27.419 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8707e-05 2024-03-20T09:46:27.419 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.419 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:27.419 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.419 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.419 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.419 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:263368 9.5 9:a7c7a537:::benchmark_data_smithi122_210164_object131138:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1143)", 2024-03-20T09:46:27.419 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:44:39.289491+0000", 2024-03-20T09:46:27.419 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 108.03724751199999, 2024-03-20T09:46:27.420 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.43598500299999998, 2024-03-20T09:46:27.420 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:27.420 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:27.420 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:27.420 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:46:27.420 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:27.420 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 263368 2024-03-20T09:46:27.420 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.420 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:27.420 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.420 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:27.420 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.289491+0000", 2024-03-20T09:46:27.421 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.421 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.421 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.421 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:27.421 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.289491+0000", 2024-03-20T09:46:27.421 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.421 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.422 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.422 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:27.422 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.289493+0000", 2024-03-20T09:46:27.422 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7710000000000001e-06 2024-03-20T09:46:27.422 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.422 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.422 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:27.422 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.289502+0000", 2024-03-20T09:46:27.422 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7110000000000001e-06 2024-03-20T09:46:27.423 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.423 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.423 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:27.423 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.289522+0000", 2024-03-20T09:46:27.423 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0313999999999999e-05 2024-03-20T09:46:27.423 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.423 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.423 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:27.423 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.289531+0000", 2024-03-20T09:46:27.423 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.285e-06 2024-03-20T09:46:27.423 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.423 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.423 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:27.423 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.289600+0000", 2024-03-20T09:46:27.424 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8986999999999995e-05 2024-03-20T09:46:27.424 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.424 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.424 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:27.424 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.289728+0000", 2024-03-20T09:46:27.424 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012816600000000001 2024-03-20T09:46:27.424 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.424 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.424 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,7", 2024-03-20T09:46:27.424 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.289845+0000", 2024-03-20T09:46:27.424 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00011686800000000001 2024-03-20T09:46:27.424 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.424 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.424 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:27.425 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.294450+0000", 2024-03-20T09:46:27.425 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0046049180000000004 2024-03-20T09:46:27.425 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.425 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.425 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.425 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.294785+0000", 2024-03-20T09:46:27.425 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00033452500000000001 2024-03-20T09:46:27.425 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.425 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.425 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.425 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.725387+0000", 2024-03-20T09:46:27.425 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.43060217299999998 2024-03-20T09:46:27.425 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.425 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.425 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:27.426 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.725435+0000", 2024-03-20T09:46:27.426 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8143000000000003e-05 2024-03-20T09:46:27.426 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.426 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.426 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:27.426 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.725476+0000", 2024-03-20T09:46:27.426 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1142e-05 2024-03-20T09:46:27.426 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.426 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:27.426 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.426 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.427 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.427 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:263399 9.5 9:a34c99a2:::benchmark_data_smithi122_210164_object131169:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1143)", 2024-03-20T09:46:27.427 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:44:39.341744+0000", 2024-03-20T09:46:27.427 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 107.98499456099999, 2024-03-20T09:46:27.427 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38777267399999998, 2024-03-20T09:46:27.427 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:27.427 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:27.427 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:27.427 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:46:27.427 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:27.427 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 263399 2024-03-20T09:46:27.427 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.427 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:27.427 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.427 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:27.428 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.341744+0000", 2024-03-20T09:46:27.428 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.428 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.428 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.428 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:27.428 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.341744+0000", 2024-03-20T09:46:27.428 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.428 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.428 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.428 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:27.428 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.341746+0000", 2024-03-20T09:46:27.428 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5629999999999999e-06 2024-03-20T09:46:27.428 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.428 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.429 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:27.429 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.341753+0000", 2024-03-20T09:46:27.429 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2169999999999997e-06 2024-03-20T09:46:27.429 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.429 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.429 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:27.429 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.341767+0000", 2024-03-20T09:46:27.429 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4242e-05 2024-03-20T09:46:27.429 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.429 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.429 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:27.429 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.341776+0000", 2024-03-20T09:46:27.429 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4860000000000003e-06 2024-03-20T09:46:27.430 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.430 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.430 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:27.430 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.341844+0000", 2024-03-20T09:46:27.430 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.7724000000000004e-05 2024-03-20T09:46:27.430 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.430 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.430 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:27.430 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.341998+0000", 2024-03-20T09:46:27.430 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000154093 2024-03-20T09:46:27.430 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.430 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.430 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,7", 2024-03-20T09:46:27.430 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.342142+0000", 2024-03-20T09:46:27.431 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000144089 2024-03-20T09:46:27.431 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.431 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.431 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:27.431 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.346722+0000", 2024-03-20T09:46:27.431 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0045800390000000002 2024-03-20T09:46:27.431 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.431 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.431 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.431 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.349068+0000", 2024-03-20T09:46:27.431 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0023455799999999999 2024-03-20T09:46:27.431 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.431 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.431 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.432 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.729422+0000", 2024-03-20T09:46:27.432 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.38035407100000002 2024-03-20T09:46:27.432 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.432 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.432 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:27.432 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.729474+0000", 2024-03-20T09:46:27.432 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2369e-05 2024-03-20T09:46:27.432 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.432 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.432 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:27.432 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.729517+0000", 2024-03-20T09:46:27.432 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2200999999999998e-05 2024-03-20T09:46:27.433 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.433 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:27.433 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.433 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.433 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.433 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:263425 9.5 9:a3ac42ff:::benchmark_data_smithi122_210164_object131195:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1143)", 2024-03-20T09:46:27.433 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:44:39.380620+0000", 2024-03-20T09:46:27.433 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 107.946118783, 2024-03-20T09:46:27.433 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.36384657100000001, 2024-03-20T09:46:27.433 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:27.433 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:27.433 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:27.433 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:46:27.433 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:27.433 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 263425 2024-03-20T09:46:27.434 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.434 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:27.434 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.434 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:27.434 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.380620+0000", 2024-03-20T09:46:27.434 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.434 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.434 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.434 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:27.434 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.380620+0000", 2024-03-20T09:46:27.434 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.434 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.434 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.435 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:27.435 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.380622+0000", 2024-03-20T09:46:27.435 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.9479999999999998e-06 2024-03-20T09:46:27.435 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.435 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.435 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:27.435 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.380630+0000", 2024-03-20T09:46:27.435 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6570000000000003e-06 2024-03-20T09:46:27.435 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.435 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.435 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:27.435 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.380647+0000", 2024-03-20T09:46:27.435 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6416999999999999e-05 2024-03-20T09:46:27.435 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.436 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.436 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:27.436 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.380656+0000", 2024-03-20T09:46:27.436 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.346e-06 2024-03-20T09:46:27.436 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.436 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.436 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:27.436 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.380719+0000", 2024-03-20T09:46:27.436 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2941999999999995e-05 2024-03-20T09:46:27.436 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.436 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.436 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:27.436 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.380805+0000", 2024-03-20T09:46:27.436 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6349999999999998e-05 2024-03-20T09:46:27.437 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.437 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.437 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,7", 2024-03-20T09:46:27.437 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.380880+0000", 2024-03-20T09:46:27.437 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.4255e-05 2024-03-20T09:46:27.437 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.437 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.437 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:27.437 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.384859+0000", 2024-03-20T09:46:27.437 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003979654 2024-03-20T09:46:27.437 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.437 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.438 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.438 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.386042+0000", 2024-03-20T09:46:27.438 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.001182413 2024-03-20T09:46:27.438 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.438 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.438 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.438 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.744343+0000", 2024-03-20T09:46:27.438 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.35830116099999998 2024-03-20T09:46:27.438 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.438 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.438 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:27.438 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.744408+0000", 2024-03-20T09:46:27.438 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.5078000000000003e-05 2024-03-20T09:46:27.438 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.439 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.439 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:27.439 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.744466+0000", 2024-03-20T09:46:27.439 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8350000000000002e-05 2024-03-20T09:46:27.439 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.439 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:27.439 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.439 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.439 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.439 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:263431 9.5 9:a203e6fc:::benchmark_data_smithi122_210164_object131201:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1143)", 2024-03-20T09:46:27.439 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:44:39.393730+0000", 2024-03-20T09:46:27.439 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 107.933008382, 2024-03-20T09:46:27.439 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.35084922699999999, 2024-03-20T09:46:27.439 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:27.440 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:27.440 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:27.440 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:46:27.440 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:27.440 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 263431 2024-03-20T09:46:27.440 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.440 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:27.440 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.440 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:27.440 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.393730+0000", 2024-03-20T09:46:27.440 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.440 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.440 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.440 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:27.441 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.393730+0000", 2024-03-20T09:46:27.441 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.441 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.441 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.441 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:27.441 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.393731+0000", 2024-03-20T09:46:27.441 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7499999999999998e-07 2024-03-20T09:46:27.441 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.441 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.441 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:27.441 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.393736+0000", 2024-03-20T09:46:27.441 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7010000000000004e-06 2024-03-20T09:46:27.442 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.442 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.442 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:27.442 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.393747+0000", 2024-03-20T09:46:27.442 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0774000000000001e-05 2024-03-20T09:46:27.442 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.442 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.442 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:27.442 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.393753+0000", 2024-03-20T09:46:27.442 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0650000000000004e-06 2024-03-20T09:46:27.442 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.442 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.442 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:27.442 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.393798+0000", 2024-03-20T09:46:27.443 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5493e-05 2024-03-20T09:46:27.443 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.443 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.443 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:27.443 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.393884+0000", 2024-03-20T09:46:27.443 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6273000000000006e-05 2024-03-20T09:46:27.443 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.443 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.443 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,7", 2024-03-20T09:46:27.443 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.393960+0000", 2024-03-20T09:46:27.443 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5469000000000005e-05 2024-03-20T09:46:27.443 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.443 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.443 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:27.444 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.397796+0000", 2024-03-20T09:46:27.444 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0038359800000000001 2024-03-20T09:46:27.444 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.444 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.444 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.444 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.398475+0000", 2024-03-20T09:46:27.444 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00067893700000000001 2024-03-20T09:46:27.444 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.444 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.444 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.444 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.744512+0000", 2024-03-20T09:46:27.444 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.34603756899999999 2024-03-20T09:46:27.444 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.444 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.444 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:27.445 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.744543+0000", 2024-03-20T09:46:27.445 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0388000000000001e-05 2024-03-20T09:46:27.445 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.445 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.445 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:27.445 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.744579+0000", 2024-03-20T09:46:27.445 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6603000000000002e-05 2024-03-20T09:46:27.445 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.445 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:27.445 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.445 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.445 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.445 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:263456 9.5 9:af5c1796:::benchmark_data_smithi122_210164_object131226:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1143)", 2024-03-20T09:46:27.445 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:44:39.460859+0000", 2024-03-20T09:46:27.446 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 107.865879663, 2024-03-20T09:46:27.446 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.28385473900000002, 2024-03-20T09:46:27.446 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:27.446 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:27.446 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:27.446 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:46:27.446 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:27.446 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 263456 2024-03-20T09:46:27.446 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.446 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:27.446 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.446 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:27.446 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.460859+0000", 2024-03-20T09:46:27.446 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.446 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.447 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.447 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:27.447 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.460859+0000", 2024-03-20T09:46:27.447 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.447 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.447 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.447 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:27.447 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.460861+0000", 2024-03-20T09:46:27.447 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.0040000000000002e-06 2024-03-20T09:46:27.447 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.447 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.447 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:27.447 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.460870+0000", 2024-03-20T09:46:27.448 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0729999999999994e-06 2024-03-20T09:46:27.448 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.448 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.448 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:27.448 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.460888+0000", 2024-03-20T09:46:27.448 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.8389999999999998e-05 2024-03-20T09:46:27.448 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.448 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.448 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:27.448 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.460899+0000", 2024-03-20T09:46:27.448 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1014e-05 2024-03-20T09:46:27.448 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.448 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.448 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:27.448 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.460979+0000", 2024-03-20T09:46:27.448 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9845999999999994e-05 2024-03-20T09:46:27.449 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.449 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.449 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:27.449 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.461087+0000", 2024-03-20T09:46:27.449 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000107575 2024-03-20T09:46:27.449 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.449 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.449 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,7", 2024-03-20T09:46:27.449 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.461173+0000", 2024-03-20T09:46:27.449 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6502999999999995e-05 2024-03-20T09:46:27.449 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.449 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.449 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:27.449 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.465037+0000", 2024-03-20T09:46:27.450 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0038639429999999999 2024-03-20T09:46:27.450 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.450 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.450 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.450 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.465633+0000", 2024-03-20T09:46:27.450 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00059541800000000001 2024-03-20T09:46:27.450 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.450 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.450 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.450 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.744625+0000", 2024-03-20T09:46:27.450 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.27899215999999999 2024-03-20T09:46:27.450 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.450 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.450 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:27.451 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.744680+0000", 2024-03-20T09:46:27.451 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4799999999999997e-05 2024-03-20T09:46:27.451 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.451 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.451 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:27.451 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.744714+0000", 2024-03-20T09:46:27.451 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4013000000000001e-05 2024-03-20T09:46:27.451 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.451 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:27.451 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.451 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.451 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.451 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.27190.0:263468 9.5 9:a54dea2c:::benchmark_data_smithi122_210164_object131238:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1143)", 2024-03-20T09:46:27.451 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:44:39.517504+0000", 2024-03-20T09:46:27.451 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 107.809234464, 2024-03-20T09:46:27.452 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.22733898999999999, 2024-03-20T09:46:27.452 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:27.452 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:27.452 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:27.452 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.27190", 2024-03-20T09:46:27.452 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:27.452 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 263468 2024-03-20T09:46:27.452 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.452 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:27.452 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.452 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:27.452 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.517504+0000", 2024-03-20T09:46:27.452 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.452 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.452 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.453 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:27.453 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.517504+0000", 2024-03-20T09:46:27.453 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:27.453 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.453 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.453 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:27.453 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.517505+0000", 2024-03-20T09:46:27.453 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2610000000000001e-06 2024-03-20T09:46:27.453 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.453 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.453 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:27.453 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.517511+0000", 2024-03-20T09:46:27.453 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6980000000000003e-06 2024-03-20T09:46:27.453 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.453 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.453 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:27.454 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.517524+0000", 2024-03-20T09:46:27.454 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2548000000000001e-05 2024-03-20T09:46:27.454 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.454 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.454 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:27.454 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.517531+0000", 2024-03-20T09:46:27.454 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2250000000000003e-06 2024-03-20T09:46:27.454 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.454 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.454 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:27.455 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.517588+0000", 2024-03-20T09:46:27.455 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6808000000000002e-05 2024-03-20T09:46:27.455 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.455 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.455 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:27.455 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.517683+0000", 2024-03-20T09:46:27.455 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5011000000000002e-05 2024-03-20T09:46:27.455 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.455 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.455 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 5,7", 2024-03-20T09:46:27.455 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.517763+0000", 2024-03-20T09:46:27.455 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.0233000000000002e-05 2024-03-20T09:46:27.455 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.455 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.455 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:27.456 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.521512+0000", 2024-03-20T09:46:27.456 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0037493230000000002 2024-03-20T09:46:27.456 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.456 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.456 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.456 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.522048+0000", 2024-03-20T09:46:27.456 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00053627500000000001 2024-03-20T09:46:27.456 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.456 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.456 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:27.456 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.744765+0000", 2024-03-20T09:46:27.456 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.22271617899999999 2024-03-20T09:46:27.456 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.456 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.456 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:27.457 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.744809+0000", 2024-03-20T09:46:27.457 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4221999999999998e-05 2024-03-20T09:46:27.457 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:27.457 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:27.457 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:27.457 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:44:39.744843+0000", 2024-03-20T09:46:27.457 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.4207e-05 2024-03-20T09:46:27.457 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.457 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:27.457 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.457 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:27.457 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:27.457 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:46:27.855 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.11 to 1 (10000) 2024-03-20T09:46:27.933 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:27 smithi122 ceph-mon[73495]: pgmap v2881: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:27.933 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:27 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2833822625' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-03-20T09:46:28.006 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-20T09:46:28.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:27 smithi115 ceph-mon[72476]: pgmap v2881: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:28.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:27 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2833822625' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-03-20T09:46:28.202 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:27 smithi026 ceph-mon[68309]: pgmap v2881: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:28.202 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:27 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2833822625' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]: dispatch 2024-03-20T09:46:28.275 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:46:28.611 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd primary-affinity 11 1 2024-03-20T09:46:28.859 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:28 smithi026 ceph-mon[68309]: 1.b scrub starts 2024-03-20T09:46:28.859 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:28 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2833822625' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-03-20T09:46:28.859 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:28 smithi026 ceph-mon[68309]: osdmap e1187: 12 total, 12 up, 12 in 2024-03-20T09:46:28.859 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:28 smithi026 ceph-mon[68309]: 1.b scrub ok 2024-03-20T09:46:28.895 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:46:29.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:28 smithi115 ceph-mon[72476]: 1.b scrub starts 2024-03-20T09:46:29.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:28 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2833822625' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-03-20T09:46:29.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:28 smithi115 ceph-mon[72476]: osdmap e1187: 12 total, 12 up, 12 in 2024-03-20T09:46:29.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:28 smithi115 ceph-mon[72476]: 1.b scrub ok 2024-03-20T09:46:29.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:28 smithi122 ceph-mon[73495]: 1.b scrub starts 2024-03-20T09:46:29.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:28 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2833822625' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-03-20T09:46:29.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:28 smithi122 ceph-mon[73495]: osdmap e1187: 12 total, 12 up, 12 in 2024-03-20T09:46:29.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:28 smithi122 ceph-mon[73495]: 1.b scrub ok 2024-03-20T09:46:30.002 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:46:30.003 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:46:30.003 INFO:teuthology.orchestra.run.smithi115.stdout: "num_ops": 0 2024-03-20T09:46:30.003 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:46:30.023 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:29 smithi115 ceph-mon[72476]: 1.1d scrub starts 2024-03-20T09:46:30.023 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:29 smithi115 ceph-mon[72476]: pgmap v2883: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:30.023 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:29 smithi115 ceph-mon[72476]: 1.1d scrub ok 2024-03-20T09:46:30.160 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:29 smithi026 ceph-mon[68309]: 1.1d scrub starts 2024-03-20T09:46:30.160 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:29 smithi026 ceph-mon[68309]: pgmap v2883: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:30.160 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:29 smithi026 ceph-mon[68309]: 1.1d scrub ok 2024-03-20T09:46:30.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:29 smithi122 ceph-mon[73495]: 1.1d scrub starts 2024-03-20T09:46:30.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:29 smithi122 ceph-mon[73495]: pgmap v2883: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:30.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:29 smithi122 ceph-mon[73495]: 1.1d scrub ok 2024-03-20T09:46:30.645 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-20T09:46:30.905 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:46:31.871 INFO:teuthology.orchestra.run.smithi026.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-03-20T09:46:32.075 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:31 smithi115 ceph-mon[72476]: pgmap v2884: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:32.076 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:31 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2287687253' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-03-20T09:46:32.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:31 smithi122 ceph-mon[73495]: pgmap v2884: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:32.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:31 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2287687253' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-03-20T09:46:32.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:31 smithi026 ceph-mon[68309]: pgmap v2884: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:32.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:31 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2287687253' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]: dispatch 2024-03-20T09:46:32.649 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 5 1 2024-03-20T09:46:32.660 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:46:32.660 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [], 2024-03-20T09:46:32.660 INFO:teuthology.orchestra.run.smithi115.stdout: "complaint_time": 30, 2024-03-20T09:46:32.660 INFO:teuthology.orchestra.run.smithi115.stdout: "num_blocked_ops": 0 2024-03-20T09:46:32.660 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:46:32.931 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:46:33.027 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:32 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:46:33.027 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:32 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:46:33.027 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:32 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2287687253' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-03-20T09:46:33.027 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:32 smithi115 ceph-mon[72476]: osdmap e1188: 12 total, 12 up, 12 in 2024-03-20T09:46:33.027 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:32 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:46:33.027 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:32 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:46:33.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:32 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:46:33.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:32 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:46:33.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:32 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2287687253' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-03-20T09:46:33.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:32 smithi122 ceph-mon[73495]: osdmap e1188: 12 total, 12 up, 12 in 2024-03-20T09:46:33.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:32 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:46:33.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:32 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:46:33.196 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:32 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:46:33.196 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:32 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:46:33.196 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:32 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2287687253' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-03-20T09:46:33.196 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:32 smithi026 ceph-mon[68309]: osdmap e1188: 12 total, 12 up, 12 in 2024-03-20T09:46:33.196 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:32 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:46:33.196 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:32 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:46:33.361 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-20T09:46:33.626 INFO:teuthology.orchestra.run.smithi115.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.b/config 2024-03-20T09:46:34.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:33 smithi115 ceph-mon[72476]: pgmap v2886: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:34.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:33 smithi115 ceph-mon[72476]: osdmap e1189: 12 total, 12 up, 12 in 2024-03-20T09:46:34.174 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:33 smithi026 ceph-mon[68309]: pgmap v2886: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:34.174 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:33 smithi026 ceph-mon[68309]: osdmap e1189: 12 total, 12 up, 12 in 2024-03-20T09:46:34.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:33 smithi122 ceph-mon[73495]: pgmap v2886: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:34.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:33 smithi122 ceph-mon[73495]: osdmap e1189: 12 total, 12 up, 12 in 2024-03-20T09:46:35.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:34 smithi115 ceph-mon[72476]: 1.15 scrub starts 2024-03-20T09:46:35.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:34 smithi115 ceph-mon[72476]: 1.15 scrub ok 2024-03-20T09:46:35.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:34 smithi122 ceph-mon[73495]: 1.15 scrub starts 2024-03-20T09:46:35.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:34 smithi122 ceph-mon[73495]: 1.15 scrub ok 2024-03-20T09:46:35.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:34 smithi026 ceph-mon[68309]: 1.15 scrub starts 2024-03-20T09:46:35.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:34 smithi026 ceph-mon[68309]: 1.15 scrub ok 2024-03-20T09:46:35.386 INFO:teuthology.orchestra.run.smithi115.stdout:{ 2024-03-20T09:46:35.386 INFO:teuthology.orchestra.run.smithi115.stdout: "size": 20, 2024-03-20T09:46:35.386 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 600, 2024-03-20T09:46:35.386 INFO:teuthology.orchestra.run.smithi115.stdout: "ops": [ 2024-03-20T09:46:35.386 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.386 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:10596 9.6 9:6abecf6b:::benchmark_data_smithi122_210164_object10595:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1077)", 2024-03-20T09:46:35.386 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:40:42.715591+0000", 2024-03-20T09:46:35.387 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 352.68136986299999, 2024-03-20T09:46:35.387 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.20462053799999999, 2024-03-20T09:46:35.387 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:35.387 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:35.387 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:46:35.387 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:46:35.387 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:35.387 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 10596 2024-03-20T09:46:35.387 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.387 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:35.387 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.387 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:35.387 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.715591+0000", 2024-03-20T09:46:35.387 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.387 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.387 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.388 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:35.388 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.715591+0000", 2024-03-20T09:46:35.388 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.388 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.388 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.388 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:35.388 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.715591+0000", 2024-03-20T09:46:35.388 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.6199999999999997e-07 2024-03-20T09:46:35.388 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.388 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.388 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:35.388 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.715618+0000", 2024-03-20T09:46:35.388 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6188000000000001e-05 2024-03-20T09:46:35.388 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.388 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.389 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:35.389 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.715627+0000", 2024-03-20T09:46:35.389 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.3279999999999995e-06 2024-03-20T09:46:35.389 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.389 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.389 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:35.389 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.715645+0000", 2024-03-20T09:46:35.389 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.8389999999999998e-05 2024-03-20T09:46:35.389 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.389 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.389 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:35.389 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.715708+0000", 2024-03-20T09:46:35.389 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.2692000000000002e-05 2024-03-20T09:46:35.389 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.390 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.390 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:35.390 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.715866+0000", 2024-03-20T09:46:35.390 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000157687 2024-03-20T09:46:35.390 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.390 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.390 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,10", 2024-03-20T09:46:35.390 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.716025+0000", 2024-03-20T09:46:35.390 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00015949599999999999 2024-03-20T09:46:35.390 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.390 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.390 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:46:35.390 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.717567+0000", 2024-03-20T09:46:35.391 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0015416690000000001 2024-03-20T09:46:35.391 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.391 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.391 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.391 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.717606+0000", 2024-03-20T09:46:35.391 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.9153000000000003e-05 2024-03-20T09:46:35.391 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.391 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.391 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.391 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.920107+0000", 2024-03-20T09:46:35.391 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.20250143300000001 2024-03-20T09:46:35.391 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.391 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.392 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:35.392 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.920161+0000", 2024-03-20T09:46:35.392 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3406999999999999e-05 2024-03-20T09:46:35.392 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.392 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.392 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:35.392 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.920211+0000", 2024-03-20T09:46:35.392 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0333000000000001e-05 2024-03-20T09:46:35.392 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.392 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:35.392 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.392 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.392 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.393 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:10599 9.6 9:65cc107b:::benchmark_data_smithi122_210164_object10598:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1077)", 2024-03-20T09:46:35.393 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:40:42.715666+0000", 2024-03-20T09:46:35.393 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 352.68129478499998, 2024-03-20T09:46:35.393 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.20620917699999999, 2024-03-20T09:46:35.393 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:35.393 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:35.393 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:46:35.393 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:46:35.393 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:35.393 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 10599 2024-03-20T09:46:35.393 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.393 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:35.393 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.393 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:35.393 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.715666+0000", 2024-03-20T09:46:35.394 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.394 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.394 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.394 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:35.394 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.715666+0000", 2024-03-20T09:46:35.394 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.394 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.394 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.394 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:35.394 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.715666+0000", 2024-03-20T09:46:35.394 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1799999999999995e-07 2024-03-20T09:46:35.394 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.394 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.395 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:35.395 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.716665+0000", 2024-03-20T09:46:35.395 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00099824399999999991 2024-03-20T09:46:35.395 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.395 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.395 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:35.395 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.716672+0000", 2024-03-20T09:46:35.395 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.8709999999999997e-06 2024-03-20T09:46:35.395 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.395 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.395 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:35.395 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.716677+0000", 2024-03-20T09:46:35.395 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9010000000000002e-06 2024-03-20T09:46:35.395 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.396 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.396 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:35.396 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.716732+0000", 2024-03-20T09:46:35.396 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4814000000000001e-05 2024-03-20T09:46:35.396 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.396 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.396 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:35.396 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.716874+0000", 2024-03-20T09:46:35.396 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00014188800000000001 2024-03-20T09:46:35.396 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.396 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.396 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,10", 2024-03-20T09:46:35.396 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.717035+0000", 2024-03-20T09:46:35.396 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000161359 2024-03-20T09:46:35.396 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.397 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.397 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.397 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.717943+0000", 2024-03-20T09:46:35.397 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00090768900000000004 2024-03-20T09:46:35.397 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.397 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.397 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:46:35.397 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.718077+0000", 2024-03-20T09:46:35.397 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013365499999999999 2024-03-20T09:46:35.397 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.397 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.397 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.397 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.921772+0000", 2024-03-20T09:46:35.397 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.20369572499999999 2024-03-20T09:46:35.397 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.398 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.398 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:35.398 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.921836+0000", 2024-03-20T09:46:35.398 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.3798000000000001e-05 2024-03-20T09:46:35.398 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.398 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.398 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:35.398 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.921875+0000", 2024-03-20T09:46:35.398 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.8615000000000002e-05 2024-03-20T09:46:35.398 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.398 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:35.398 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.398 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.398 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.398 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:10639 9.6 9:6832bb1a:::benchmark_data_smithi122_210164_object10638:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1077)", 2024-03-20T09:46:35.399 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:40:42.728167+0000", 2024-03-20T09:46:35.399 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 352.66879315599999, 2024-03-20T09:46:35.399 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.194306914, 2024-03-20T09:46:35.399 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:35.399 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:35.399 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:46:35.399 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:46:35.399 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:35.399 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 10639 2024-03-20T09:46:35.399 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.399 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:35.399 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.400 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:35.400 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.728167+0000", 2024-03-20T09:46:35.400 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.400 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.400 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.400 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:35.400 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.728167+0000", 2024-03-20T09:46:35.400 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.400 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.400 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.400 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:35.400 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.728168+0000", 2024-03-20T09:46:35.400 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.5000000000000001e-07 2024-03-20T09:46:35.400 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.401 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.401 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:35.401 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.728225+0000", 2024-03-20T09:46:35.401 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7204000000000003e-05 2024-03-20T09:46:35.401 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.401 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.401 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:35.401 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.728243+0000", 2024-03-20T09:46:35.401 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.7051e-05 2024-03-20T09:46:35.401 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.401 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.401 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:35.401 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.728249+0000", 2024-03-20T09:46:35.401 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.7279999999999998e-06 2024-03-20T09:46:35.401 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.402 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.402 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:35.402 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.728297+0000", 2024-03-20T09:46:35.402 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.7302999999999998e-05 2024-03-20T09:46:35.402 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.402 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.402 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:35.402 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.728431+0000", 2024-03-20T09:46:35.402 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00013412999999999999 2024-03-20T09:46:35.402 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.402 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.402 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,10", 2024-03-20T09:46:35.402 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.728624+0000", 2024-03-20T09:46:35.402 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00019335399999999999 2024-03-20T09:46:35.403 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.403 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.403 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:46:35.403 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.729475+0000", 2024-03-20T09:46:35.403 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00085104600000000005 2024-03-20T09:46:35.403 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.403 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.403 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.403 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.729784+0000", 2024-03-20T09:46:35.403 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000308647 2024-03-20T09:46:35.403 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.403 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.403 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.403 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.922417+0000", 2024-03-20T09:46:35.403 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.19263318099999999 2024-03-20T09:46:35.403 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.404 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.404 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:35.404 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.922451+0000", 2024-03-20T09:46:35.404 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.3633999999999997e-05 2024-03-20T09:46:35.404 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.404 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.404 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:35.404 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:42.922474+0000", 2024-03-20T09:46:35.404 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.3686e-05 2024-03-20T09:46:35.404 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.404 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:35.404 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.404 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.405 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.405 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:11175 9.f 9:fbd115b8:::benchmark_data_smithi122_210164_object11174:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1077)", 2024-03-20T09:46:35.405 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:40:43.038817+0000", 2024-03-20T09:46:35.405 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 352.35814363499998, 2024-03-20T09:46:35.405 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.20549536500000001, 2024-03-20T09:46:35.405 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:35.405 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:35.405 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:46:35.405 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:46:35.405 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:35.405 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 11175 2024-03-20T09:46:35.405 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.405 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:35.405 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.405 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:35.405 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.038817+0000", 2024-03-20T09:46:35.406 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.406 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.406 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.406 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:35.406 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.038817+0000", 2024-03-20T09:46:35.406 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.406 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.406 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.406 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:35.406 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.038818+0000", 2024-03-20T09:46:35.406 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.0599999999999999e-07 2024-03-20T09:46:35.406 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.406 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.406 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:35.406 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.038968+0000", 2024-03-20T09:46:35.406 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000149794 2024-03-20T09:46:35.407 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.407 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.407 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:35.407 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.038978+0000", 2024-03-20T09:46:35.407 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0465999999999999e-05 2024-03-20T09:46:35.407 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.407 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.407 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:35.407 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.038986+0000", 2024-03-20T09:46:35.407 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.1729999999999995e-06 2024-03-20T09:46:35.407 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.407 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.407 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:35.407 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.039031+0000", 2024-03-20T09:46:35.408 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4666000000000002e-05 2024-03-20T09:46:35.408 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.408 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.408 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:35.408 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.039135+0000", 2024-03-20T09:46:35.408 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0001037 2024-03-20T09:46:35.408 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.408 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.408 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,9", 2024-03-20T09:46:35.408 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.039257+0000", 2024-03-20T09:46:35.408 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000122481 2024-03-20T09:46:35.408 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.408 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.408 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:46:35.408 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.039812+0000", 2024-03-20T09:46:35.408 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00055496699999999996 2024-03-20T09:46:35.409 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.409 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.409 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.409 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.040554+0000", 2024-03-20T09:46:35.409 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00074191999999999997 2024-03-20T09:46:35.409 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.409 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.409 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.409 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.244208+0000", 2024-03-20T09:46:35.409 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.20365414100000001 2024-03-20T09:46:35.409 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.409 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.409 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:35.409 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.244268+0000", 2024-03-20T09:46:35.409 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9661999999999999e-05 2024-03-20T09:46:35.410 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.410 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.410 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:35.410 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.244312+0000", 2024-03-20T09:46:35.410 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4489e-05 2024-03-20T09:46:35.410 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.410 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:35.410 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.410 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.410 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.410 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:11184 9.f 9:f0bd2aa7:::benchmark_data_smithi122_210164_object11183:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1077)", 2024-03-20T09:46:35.410 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:40:43.042129+0000", 2024-03-20T09:46:35.410 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 352.35483195500001, 2024-03-20T09:46:35.410 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.20412398900000001, 2024-03-20T09:46:35.410 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:35.410 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:35.411 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:46:35.411 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:46:35.411 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:35.411 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 11184 2024-03-20T09:46:35.411 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.411 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:35.411 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.411 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:35.411 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.042129+0000", 2024-03-20T09:46:35.411 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.411 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.411 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.411 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:35.411 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.042129+0000", 2024-03-20T09:46:35.411 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.411 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.412 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.412 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:35.412 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.042129+0000", 2024-03-20T09:46:35.412 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.8999999999999995e-07 2024-03-20T09:46:35.412 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.412 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.412 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:35.412 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.042199+0000", 2024-03-20T09:46:35.412 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.9443999999999994e-05 2024-03-20T09:46:35.412 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.412 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.412 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:35.412 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.042209+0000", 2024-03-20T09:46:35.412 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0489000000000001e-05 2024-03-20T09:46:35.413 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.413 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.413 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:35.413 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.042215+0000", 2024-03-20T09:46:35.413 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.5860000000000004e-06 2024-03-20T09:46:35.413 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.413 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.413 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:35.413 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.042256+0000", 2024-03-20T09:46:35.413 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.1291000000000002e-05 2024-03-20T09:46:35.413 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.413 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.413 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:35.413 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.042346+0000", 2024-03-20T09:46:35.413 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.9839000000000005e-05 2024-03-20T09:46:35.413 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.414 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.414 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,9", 2024-03-20T09:46:35.414 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.042440+0000", 2024-03-20T09:46:35.414 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.3865999999999999e-05 2024-03-20T09:46:35.414 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.414 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.414 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:46:35.414 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.043623+0000", 2024-03-20T09:46:35.414 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.001182902 2024-03-20T09:46:35.414 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.414 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.414 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.414 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.043712+0000", 2024-03-20T09:46:35.414 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.8788000000000004e-05 2024-03-20T09:46:35.414 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.415 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.415 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.415 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.246152+0000", 2024-03-20T09:46:35.415 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.202440433 2024-03-20T09:46:35.415 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.415 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.415 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:35.415 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.246212+0000", 2024-03-20T09:46:35.415 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.0019000000000003e-05 2024-03-20T09:46:35.415 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.415 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.415 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:35.415 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.246253+0000", 2024-03-20T09:46:35.416 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.0441999999999997e-05 2024-03-20T09:46:35.416 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.416 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:35.416 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.416 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.416 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.416 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:11190 9.f 9:f76ec0dc:::benchmark_data_smithi122_210164_object11189:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1077)", 2024-03-20T09:46:35.416 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:40:43.042439+0000", 2024-03-20T09:46:35.416 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 352.354521788, 2024-03-20T09:46:35.416 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.20391409099999999, 2024-03-20T09:46:35.416 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:35.416 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:35.416 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:46:35.416 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:46:35.416 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:35.417 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 11190 2024-03-20T09:46:35.417 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.417 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:35.417 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.417 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:35.417 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.042439+0000", 2024-03-20T09:46:35.417 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.417 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.417 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.417 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:35.417 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.042439+0000", 2024-03-20T09:46:35.417 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.417 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.418 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.418 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:35.418 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.042439+0000", 2024-03-20T09:46:35.418 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.44e-07 2024-03-20T09:46:35.418 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.418 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.418 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:35.418 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.042496+0000", 2024-03-20T09:46:35.418 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7126000000000002e-05 2024-03-20T09:46:35.418 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.418 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.418 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:35.418 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.042502+0000", 2024-03-20T09:46:35.418 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8499999999999999e-06 2024-03-20T09:46:35.419 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.419 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.419 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:35.419 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.042506+0000", 2024-03-20T09:46:35.419 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.9910000000000004e-06 2024-03-20T09:46:35.419 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.419 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.419 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:35.419 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.043071+0000", 2024-03-20T09:46:35.419 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00056488199999999995 2024-03-20T09:46:35.419 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.419 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.419 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:35.419 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.043157+0000", 2024-03-20T09:46:35.420 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5842999999999998e-05 2024-03-20T09:46:35.420 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.420 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.420 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,9", 2024-03-20T09:46:35.420 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.043262+0000", 2024-03-20T09:46:35.420 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00010541 2024-03-20T09:46:35.420 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.420 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.420 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:46:35.420 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.043766+0000", 2024-03-20T09:46:35.420 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00050379299999999999 2024-03-20T09:46:35.420 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.420 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.420 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.421 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.044467+0000", 2024-03-20T09:46:35.421 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00070043899999999997 2024-03-20T09:46:35.421 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.421 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.421 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.421 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.246293+0000", 2024-03-20T09:46:35.421 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.20182663000000001 2024-03-20T09:46:35.421 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.421 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.421 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:35.421 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.246327+0000", 2024-03-20T09:46:35.421 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.3353999999999999e-05 2024-03-20T09:46:35.421 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.422 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.422 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:35.422 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.246353+0000", 2024-03-20T09:46:35.422 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.6228999999999999e-05 2024-03-20T09:46:35.422 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.422 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:35.422 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.422 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.422 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.422 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:11212 9.f 9:f6a0c3e4:::benchmark_data_smithi122_210164_object11211:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1077)", 2024-03-20T09:46:35.422 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:40:43.050001+0000", 2024-03-20T09:46:35.422 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 352.34695997199998, 2024-03-20T09:46:35.422 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.19650494900000001, 2024-03-20T09:46:35.423 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:35.423 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:35.423 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:46:35.423 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:46:35.423 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:35.423 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 11212 2024-03-20T09:46:35.423 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.423 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:35.423 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.423 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:35.423 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.050001+0000", 2024-03-20T09:46:35.423 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.423 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.423 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.423 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:35.424 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.050001+0000", 2024-03-20T09:46:35.424 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.424 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.424 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.424 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:35.424 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.050001+0000", 2024-03-20T09:46:35.424 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.1900000000000001e-07 2024-03-20T09:46:35.424 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.424 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.424 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:35.424 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.050086+0000", 2024-03-20T09:46:35.424 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5036000000000004e-05 2024-03-20T09:46:35.424 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.424 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.425 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:35.425 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.050094+0000", 2024-03-20T09:46:35.425 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.2080000000000003e-06 2024-03-20T09:46:35.425 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.425 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.425 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:35.425 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.050099+0000", 2024-03-20T09:46:35.425 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7529999999999998e-06 2024-03-20T09:46:35.425 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.425 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.425 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:35.425 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.050140+0000", 2024-03-20T09:46:35.425 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.0629e-05 2024-03-20T09:46:35.425 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.425 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.426 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:35.426 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.050258+0000", 2024-03-20T09:46:35.426 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0001182 2024-03-20T09:46:35.426 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.426 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.426 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,9", 2024-03-20T09:46:35.426 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.050385+0000", 2024-03-20T09:46:35.426 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012671700000000001 2024-03-20T09:46:35.426 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.426 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.426 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:46:35.426 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.050947+0000", 2024-03-20T09:46:35.426 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00056218100000000003 2024-03-20T09:46:35.427 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.427 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.427 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.427 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.051394+0000", 2024-03-20T09:46:35.427 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000447081 2024-03-20T09:46:35.427 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.427 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.427 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.427 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.246401+0000", 2024-03-20T09:46:35.427 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.195007029 2024-03-20T09:46:35.427 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.427 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.427 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:35.427 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.246459+0000", 2024-03-20T09:46:35.427 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7447000000000001e-05 2024-03-20T09:46:35.428 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.428 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.428 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:35.428 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:40:43.246505+0000", 2024-03-20T09:46:35.428 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6749000000000003e-05 2024-03-20T09:46:35.428 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.428 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:35.428 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.428 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.428 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.428 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:123501 9.f 9:fe940008:::benchmark_data_smithi122_210164_object123500:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1097)", 2024-03-20T09:46:35.428 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:41:54.577894+0000", 2024-03-20T09:46:35.428 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 280.81906649199999, 2024-03-20T09:46:35.429 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.21691714500000001, 2024-03-20T09:46:35.429 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:35.429 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:35.429 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:46:35.429 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:46:35.429 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:35.429 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 123501 2024-03-20T09:46:35.429 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.429 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:35.429 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.429 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:35.429 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.577894+0000", 2024-03-20T09:46:35.429 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.429 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.430 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.430 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:35.430 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.577894+0000", 2024-03-20T09:46:35.430 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.430 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.430 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.430 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:35.430 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.577895+0000", 2024-03-20T09:46:35.430 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.44e-07 2024-03-20T09:46:35.430 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.430 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.430 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:35.430 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.577925+0000", 2024-03-20T09:46:35.431 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.0159e-05 2024-03-20T09:46:35.431 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.431 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.431 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:35.431 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.577930+0000", 2024-03-20T09:46:35.431 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1410000000000004e-06 2024-03-20T09:46:35.431 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.431 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.431 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:35.431 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.577933+0000", 2024-03-20T09:46:35.431 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.529e-06 2024-03-20T09:46:35.431 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.431 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.431 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:35.431 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.577973+0000", 2024-03-20T09:46:35.431 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.9807e-05 2024-03-20T09:46:35.432 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.432 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.432 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:35.432 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.578140+0000", 2024-03-20T09:46:35.432 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00016659300000000001 2024-03-20T09:46:35.432 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.432 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.432 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,9", 2024-03-20T09:46:35.432 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.578253+0000", 2024-03-20T09:46:35.432 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00011340000000000001 2024-03-20T09:46:35.432 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.432 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.432 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:46:35.432 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.582380+0000", 2024-03-20T09:46:35.433 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0041272490000000004 2024-03-20T09:46:35.433 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.433 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.433 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.433 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.585317+0000", 2024-03-20T09:46:35.433 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.002936326 2024-03-20T09:46:35.433 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.433 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.433 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.433 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.794703+0000", 2024-03-20T09:46:35.433 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.20938625 2024-03-20T09:46:35.433 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.433 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.433 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:35.434 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.794764+0000", 2024-03-20T09:46:35.434 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1507999999999995e-05 2024-03-20T09:46:35.434 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.434 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.434 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:35.434 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.794811+0000", 2024-03-20T09:46:35.434 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6638999999999998e-05 2024-03-20T09:46:35.434 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.434 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:35.434 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.434 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.434 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.434 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:123516 9.f 9:f993fea8:::benchmark_data_smithi122_210164_object123515:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1097)", 2024-03-20T09:46:35.434 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:41:54.606990+0000", 2024-03-20T09:46:35.434 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 280.78997034600002, 2024-03-20T09:46:35.435 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.19976612599999999, 2024-03-20T09:46:35.435 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:35.435 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:35.435 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:46:35.435 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:46:35.435 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:35.435 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 123516 2024-03-20T09:46:35.435 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.435 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:35.435 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.435 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:35.435 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.606990+0000", 2024-03-20T09:46:35.435 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.435 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.435 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.436 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:35.436 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.606990+0000", 2024-03-20T09:46:35.436 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.436 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.436 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.436 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:35.436 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.606992+0000", 2024-03-20T09:46:35.436 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4050000000000001e-06 2024-03-20T09:46:35.436 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.436 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.436 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:35.436 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.607039+0000", 2024-03-20T09:46:35.436 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.7086999999999998e-05 2024-03-20T09:46:35.436 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.436 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.436 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:35.437 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.607050+0000", 2024-03-20T09:46:35.437 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1236000000000001e-05 2024-03-20T09:46:35.437 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.437 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.437 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:35.437 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.607058+0000", 2024-03-20T09:46:35.437 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.2949999999999994e-06 2024-03-20T09:46:35.437 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.437 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.437 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:35.437 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.607113+0000", 2024-03-20T09:46:35.437 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4874999999999999e-05 2024-03-20T09:46:35.437 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.437 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.437 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:35.438 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.607236+0000", 2024-03-20T09:46:35.438 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012267199999999999 2024-03-20T09:46:35.438 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.438 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.438 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,9", 2024-03-20T09:46:35.438 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.607385+0000", 2024-03-20T09:46:35.438 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00014965399999999999 2024-03-20T09:46:35.438 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.438 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.438 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:46:35.438 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.610362+0000", 2024-03-20T09:46:35.438 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.002976528 2024-03-20T09:46:35.438 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.438 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.439 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.439 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.611609+0000", 2024-03-20T09:46:35.439 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0012466790000000001 2024-03-20T09:46:35.439 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.439 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.439 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.439 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.806654+0000", 2024-03-20T09:46:35.439 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.19504556200000001 2024-03-20T09:46:35.439 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.439 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.439 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:35.439 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.806712+0000", 2024-03-20T09:46:35.439 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8103999999999998e-05 2024-03-20T09:46:35.440 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.440 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.440 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:35.440 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.806756+0000", 2024-03-20T09:46:35.440 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4029000000000001e-05 2024-03-20T09:46:35.440 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.440 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:35.440 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.440 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.440 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.440 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:123522 9.f 9:f5583def:::benchmark_data_smithi122_210164_object123521:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1097)", 2024-03-20T09:46:35.440 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:41:54.620353+0000", 2024-03-20T09:46:35.440 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 280.77660729799999, 2024-03-20T09:46:35.440 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.193410364, 2024-03-20T09:46:35.441 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:35.441 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:35.441 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:46:35.441 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:46:35.441 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:35.441 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 123522 2024-03-20T09:46:35.441 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.441 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:35.441 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.441 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:35.441 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.620353+0000", 2024-03-20T09:46:35.441 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.441 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.441 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.441 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:35.442 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.620353+0000", 2024-03-20T09:46:35.442 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.442 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.442 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.442 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:35.442 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.620355+0000", 2024-03-20T09:46:35.442 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5149999999999999e-06 2024-03-20T09:46:35.442 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.442 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.442 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:35.442 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.620399+0000", 2024-03-20T09:46:35.442 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.3949999999999998e-05 2024-03-20T09:46:35.442 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.442 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.443 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:35.443 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.620413+0000", 2024-03-20T09:46:35.443 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.4141000000000001e-05 2024-03-20T09:46:35.443 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.443 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.443 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:35.443 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.620422+0000", 2024-03-20T09:46:35.443 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.0569999999999999e-06 2024-03-20T09:46:35.443 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.443 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.443 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:35.443 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.620475+0000", 2024-03-20T09:46:35.443 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3084000000000002e-05 2024-03-20T09:46:35.443 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.443 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.444 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:35.444 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.620587+0000", 2024-03-20T09:46:35.444 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000112107 2024-03-20T09:46:35.444 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.444 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.444 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,9", 2024-03-20T09:46:35.444 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.620683+0000", 2024-03-20T09:46:35.444 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.5806999999999999e-05 2024-03-20T09:46:35.444 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.444 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.444 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:46:35.444 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.623619+0000", 2024-03-20T09:46:35.444 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.002936461 2024-03-20T09:46:35.444 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.444 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.444 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.445 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.627728+0000", 2024-03-20T09:46:35.445 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0041082430000000001 2024-03-20T09:46:35.445 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.445 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.445 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.445 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.813699+0000", 2024-03-20T09:46:35.445 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.185970999 2024-03-20T09:46:35.445 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.445 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.445 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:35.445 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.813736+0000", 2024-03-20T09:46:35.445 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.7691999999999998e-05 2024-03-20T09:46:35.445 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.445 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.445 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:35.445 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:41:54.813764+0000", 2024-03-20T09:46:35.445 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.7308e-05 2024-03-20T09:46:35.446 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.446 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:35.446 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.446 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.446 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.446 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:198155 9.a 9:5e224cc5:::benchmark_data_smithi122_210164_object65925:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1123)", 2024-03-20T09:46:35.446 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:43:24.267358+0000", 2024-03-20T09:46:35.446 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 191.12960268800001, 2024-03-20T09:46:35.446 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.223546984, 2024-03-20T09:46:35.446 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:35.446 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:35.446 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:46:35.446 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:46:35.446 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:35.446 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 198155 2024-03-20T09:46:35.447 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.447 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:35.447 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.447 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:35.447 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.267358+0000", 2024-03-20T09:46:35.447 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.447 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.447 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.447 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:35.447 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.267358+0000", 2024-03-20T09:46:35.447 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.447 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.447 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.447 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:35.447 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.267359+0000", 2024-03-20T09:46:35.448 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3659999999999999e-06 2024-03-20T09:46:35.448 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.448 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.448 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:35.448 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.267365+0000", 2024-03-20T09:46:35.448 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9850000000000003e-06 2024-03-20T09:46:35.448 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.448 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.448 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:35.448 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.267386+0000", 2024-03-20T09:46:35.448 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.0866e-05 2024-03-20T09:46:35.448 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.448 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.449 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:35.449 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.267393+0000", 2024-03-20T09:46:35.449 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3629999999999996e-06 2024-03-20T09:46:35.449 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.449 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.449 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:35.449 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.272457+0000", 2024-03-20T09:46:35.449 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0050632189999999999 2024-03-20T09:46:35.449 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.449 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.449 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:35.449 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.272588+0000", 2024-03-20T09:46:35.449 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000131497 2024-03-20T09:46:35.449 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.450 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.450 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,7", 2024-03-20T09:46:35.450 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.272698+0000", 2024-03-20T09:46:35.450 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000109546 2024-03-20T09:46:35.450 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.450 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.450 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.450 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.286324+0000", 2024-03-20T09:46:35.450 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.013625893 2024-03-20T09:46:35.450 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.450 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.450 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:46:35.450 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.291234+0000", 2024-03-20T09:46:35.451 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0049104079999999998 2024-03-20T09:46:35.451 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.451 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.451 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.451 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.490820+0000", 2024-03-20T09:46:35.451 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.19958569900000001 2024-03-20T09:46:35.451 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.451 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.451 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:35.451 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.490874+0000", 2024-03-20T09:46:35.451 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.4286000000000002e-05 2024-03-20T09:46:35.451 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.451 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.451 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:35.452 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.490905+0000", 2024-03-20T09:46:35.452 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.0855999999999997e-05 2024-03-20T09:46:35.452 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.452 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:35.452 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.452 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.452 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.452 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:198161 9.a 9:53c57d45:::benchmark_data_smithi122_210164_object65931:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1123)", 2024-03-20T09:46:35.452 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:43:24.273210+0000", 2024-03-20T09:46:35.452 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 191.12375097699999, 2024-03-20T09:46:35.452 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.217799785, 2024-03-20T09:46:35.452 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:35.452 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:35.452 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:46:35.453 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:46:35.453 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:35.453 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 198161 2024-03-20T09:46:35.453 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.453 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:35.453 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.453 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:35.453 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.273210+0000", 2024-03-20T09:46:35.453 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.453 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.453 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.453 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:35.453 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.273210+0000", 2024-03-20T09:46:35.453 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.453 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.454 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.454 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:35.454 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.273211+0000", 2024-03-20T09:46:35.454 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.641e-06 2024-03-20T09:46:35.454 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.454 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.454 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:35.454 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.273218+0000", 2024-03-20T09:46:35.454 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.2899999999999997e-06 2024-03-20T09:46:35.454 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.454 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.454 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:35.454 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.273234+0000", 2024-03-20T09:46:35.455 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.5882000000000001e-05 2024-03-20T09:46:35.455 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.455 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.455 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:35.455 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.273243+0000", 2024-03-20T09:46:35.455 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.0880000000000001e-06 2024-03-20T09:46:35.455 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.455 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.455 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:35.455 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.276721+0000", 2024-03-20T09:46:35.455 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0034780829999999999 2024-03-20T09:46:35.455 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.455 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.455 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:35.456 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.276821+0000", 2024-03-20T09:46:35.456 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.9438999999999994e-05 2024-03-20T09:46:35.456 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.456 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.456 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,7", 2024-03-20T09:46:35.456 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.276895+0000", 2024-03-20T09:46:35.456 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3887999999999995e-05 2024-03-20T09:46:35.456 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.456 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.456 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.456 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.296052+0000", 2024-03-20T09:46:35.456 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.019157072000000001 2024-03-20T09:46:35.456 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.456 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.456 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:46:35.457 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.300094+0000", 2024-03-20T09:46:35.457 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0040423780000000001 2024-03-20T09:46:35.457 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.457 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.457 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.457 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.490929+0000", 2024-03-20T09:46:35.457 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.19083514500000001 2024-03-20T09:46:35.457 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.457 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.457 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:35.457 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.490974+0000", 2024-03-20T09:46:35.457 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4499999999999997e-05 2024-03-20T09:46:35.457 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.457 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.457 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:35.458 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.491009+0000", 2024-03-20T09:46:35.458 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.5379000000000001e-05 2024-03-20T09:46:35.458 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.458 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:35.458 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.458 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.458 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.458 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:198162 9.d 9:b21f4cc2:::benchmark_data_smithi122_210164_object65932:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1123)", 2024-03-20T09:46:35.458 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:43:24.273261+0000", 2024-03-20T09:46:35.458 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 191.12369900100001, 2024-03-20T09:46:35.458 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.21768599199999999, 2024-03-20T09:46:35.458 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:35.458 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:35.459 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:46:35.459 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:46:35.459 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:35.459 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 198162 2024-03-20T09:46:35.459 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.459 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:35.459 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.459 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:35.459 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.273261+0000", 2024-03-20T09:46:35.459 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.459 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.459 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.459 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:35.459 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.273261+0000", 2024-03-20T09:46:35.460 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.460 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.460 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.460 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:35.460 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.273262+0000", 2024-03-20T09:46:35.460 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.2e-07 2024-03-20T09:46:35.460 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.460 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.460 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:35.460 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.273264+0000", 2024-03-20T09:46:35.460 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.5440000000000001e-06 2024-03-20T09:46:35.460 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.460 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.460 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:35.460 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.273271+0000", 2024-03-20T09:46:35.461 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.071e-06 2024-03-20T09:46:35.461 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.461 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.461 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:35.461 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.273275+0000", 2024-03-20T09:46:35.461 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.8189999999999999e-06 2024-03-20T09:46:35.461 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.461 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.461 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:35.461 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.273349+0000", 2024-03-20T09:46:35.461 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3830000000000003e-05 2024-03-20T09:46:35.461 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.461 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.461 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:35.461 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.273460+0000", 2024-03-20T09:46:35.462 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000110847 2024-03-20T09:46:35.462 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.462 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.462 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,7", 2024-03-20T09:46:35.462 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.273542+0000", 2024-03-20T09:46:35.462 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.2118000000000002e-05 2024-03-20T09:46:35.462 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.462 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.462 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.462 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.278461+0000", 2024-03-20T09:46:35.462 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0049186789999999996 2024-03-20T09:46:35.462 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.462 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.462 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:46:35.463 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.285936+0000", 2024-03-20T09:46:35.463 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0074756880000000003 2024-03-20T09:46:35.463 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.463 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.463 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.463 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.490848+0000", 2024-03-20T09:46:35.463 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.20491110700000001 2024-03-20T09:46:35.463 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.463 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.463 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:35.463 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.490904+0000", 2024-03-20T09:46:35.463 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6891000000000001e-05 2024-03-20T09:46:35.463 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.463 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.463 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:35.463 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.490947+0000", 2024-03-20T09:46:35.463 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.2978000000000001e-05 2024-03-20T09:46:35.464 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.464 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:35.464 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.464 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.464 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.464 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:198167 9.a 9:59eeb5bc:::benchmark_data_smithi122_210164_object65937:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1123)", 2024-03-20T09:46:35.464 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:43:24.279046+0000", 2024-03-20T09:46:35.464 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 191.11791420500001, 2024-03-20T09:46:35.464 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.21209787199999999, 2024-03-20T09:46:35.464 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:35.464 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:35.464 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:46:35.464 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:46:35.464 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:35.464 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 198167 2024-03-20T09:46:35.465 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.465 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:35.465 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.465 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:35.465 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.279046+0000", 2024-03-20T09:46:35.465 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.465 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.465 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.465 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:35.465 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.279046+0000", 2024-03-20T09:46:35.465 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.465 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.465 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.465 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:35.465 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.279048+0000", 2024-03-20T09:46:35.466 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.3540000000000001e-06 2024-03-20T09:46:35.466 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.466 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.466 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:35.466 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.279053+0000", 2024-03-20T09:46:35.466 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3349999999999997e-06 2024-03-20T09:46:35.466 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.466 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.466 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:35.466 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.279063+0000", 2024-03-20T09:46:35.466 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.0393e-05 2024-03-20T09:46:35.466 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.466 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.466 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:35.466 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.279070+0000", 2024-03-20T09:46:35.467 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.6919999999999996e-06 2024-03-20T09:46:35.467 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.467 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.467 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:35.467 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.281761+0000", 2024-03-20T09:46:35.467 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0026912339999999998 2024-03-20T09:46:35.467 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.467 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.467 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:35.467 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.281867+0000", 2024-03-20T09:46:35.467 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00010599699999999999 2024-03-20T09:46:35.467 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.467 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.467 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,7", 2024-03-20T09:46:35.467 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.281935+0000", 2024-03-20T09:46:35.468 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.7821e-05 2024-03-20T09:46:35.468 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.468 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.468 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.468 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.296091+0000", 2024-03-20T09:46:35.468 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.014155571 2024-03-20T09:46:35.468 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.468 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.468 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:46:35.468 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.300100+0000", 2024-03-20T09:46:35.468 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0040096869999999996 2024-03-20T09:46:35.468 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.468 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.468 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.468 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.491043+0000", 2024-03-20T09:46:35.469 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.190942266 2024-03-20T09:46:35.469 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.469 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.469 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:35.469 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.491091+0000", 2024-03-20T09:46:35.469 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.7883999999999998e-05 2024-03-20T09:46:35.469 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.469 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.469 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:35.469 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.491144+0000", 2024-03-20T09:46:35.469 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.3637999999999997e-05 2024-03-20T09:46:35.469 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.469 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:35.469 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.470 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.470 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.470 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:198170 9.d 9:b2d8f808:::benchmark_data_smithi122_210164_object65940:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1123)", 2024-03-20T09:46:35.470 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:43:24.281492+0000", 2024-03-20T09:46:35.470 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 191.115468443, 2024-03-20T09:46:35.470 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.20958100199999999, 2024-03-20T09:46:35.470 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:35.470 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:35.470 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:46:35.470 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:46:35.470 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:35.470 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 198170 2024-03-20T09:46:35.470 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.470 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:35.471 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.471 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:35.471 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.281492+0000", 2024-03-20T09:46:35.471 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.471 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.471 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.471 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:35.471 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.281492+0000", 2024-03-20T09:46:35.471 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.471 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.471 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.471 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:35.471 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.281493+0000", 2024-03-20T09:46:35.471 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.0599999999999999e-07 2024-03-20T09:46:35.471 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.472 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.472 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:35.472 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.281498+0000", 2024-03-20T09:46:35.472 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.215e-06 2024-03-20T09:46:35.472 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.472 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.472 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:35.472 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.281510+0000", 2024-03-20T09:46:35.472 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1576999999999999e-05 2024-03-20T09:46:35.472 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.472 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.472 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:35.472 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.281516+0000", 2024-03-20T09:46:35.472 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8690000000000001e-06 2024-03-20T09:46:35.473 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.473 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.473 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:35.473 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.281563+0000", 2024-03-20T09:46:35.473 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.7333000000000002e-05 2024-03-20T09:46:35.473 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.473 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.473 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:35.473 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.281659+0000", 2024-03-20T09:46:35.473 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.6274999999999996e-05 2024-03-20T09:46:35.473 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.473 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.473 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,7", 2024-03-20T09:46:35.473 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.281763+0000", 2024-03-20T09:46:35.473 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00010390700000000001 2024-03-20T09:46:35.474 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.474 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.474 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.474 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.286840+0000", 2024-03-20T09:46:35.474 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0050769960000000003 2024-03-20T09:46:35.474 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.474 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.474 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:46:35.474 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.289213+0000", 2024-03-20T09:46:35.474 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0023731889999999999 2024-03-20T09:46:35.474 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.474 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.474 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.474 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.490994+0000", 2024-03-20T09:46:35.475 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.201781194 2024-03-20T09:46:35.475 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.475 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.475 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:35.475 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.491046+0000", 2024-03-20T09:46:35.475 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.1489000000000001e-05 2024-03-20T09:46:35.475 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.475 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.475 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:35.475 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.491073+0000", 2024-03-20T09:46:35.475 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 2.7152e-05 2024-03-20T09:46:35.475 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.475 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:35.475 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.475 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.476 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.476 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:198171 9.a 9:5e79cf61:::benchmark_data_smithi122_210164_object65941:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1123)", 2024-03-20T09:46:35.476 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:43:24.281680+0000", 2024-03-20T09:46:35.476 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 191.11528052, 2024-03-20T09:46:35.476 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.23080156199999999, 2024-03-20T09:46:35.476 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:35.476 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:35.476 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:46:35.476 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:46:35.476 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:35.476 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 198171 2024-03-20T09:46:35.476 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.476 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:35.476 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.476 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:35.476 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.281680+0000", 2024-03-20T09:46:35.477 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.477 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.477 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.477 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:35.477 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.281680+0000", 2024-03-20T09:46:35.477 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.477 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.477 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.477 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:35.477 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.281681+0000", 2024-03-20T09:46:35.477 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.8400000000000004e-07 2024-03-20T09:46:35.477 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.477 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.477 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:35.478 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.281684+0000", 2024-03-20T09:46:35.478 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6459999999999999e-06 2024-03-20T09:46:35.478 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.478 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.478 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:35.478 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.281692+0000", 2024-03-20T09:46:35.478 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.2969999999999998e-06 2024-03-20T09:46:35.478 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.478 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.478 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:35.478 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.281700+0000", 2024-03-20T09:46:35.478 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.1450000000000008e-06 2024-03-20T09:46:35.478 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.478 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.479 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:35.479 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.286394+0000", 2024-03-20T09:46:35.479 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0046938209999999999 2024-03-20T09:46:35.479 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.479 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.479 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:35.479 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.286509+0000", 2024-03-20T09:46:35.479 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000115529 2024-03-20T09:46:35.479 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.479 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.479 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,7", 2024-03-20T09:46:35.479 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.286621+0000", 2024-03-20T09:46:35.479 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000112404 2024-03-20T09:46:35.479 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.480 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.480 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.480 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.296125+0000", 2024-03-20T09:46:35.480 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0095039500000000006 2024-03-20T09:46:35.480 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.480 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.480 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:46:35.480 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.300108+0000", 2024-03-20T09:46:35.480 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0039824730000000003 2024-03-20T09:46:35.480 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.480 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.480 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.480 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.512372+0000", 2024-03-20T09:46:35.480 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.21226425800000001 2024-03-20T09:46:35.480 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.480 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.481 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:35.481 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.512437+0000", 2024-03-20T09:46:35.481 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.4968000000000005e-05 2024-03-20T09:46:35.481 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.481 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.481 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:35.481 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.512482+0000", 2024-03-20T09:46:35.481 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.4387e-05 2024-03-20T09:46:35.481 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.481 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:35.481 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.481 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.481 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.481 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:198174 9.a 9:5af9ce18:::benchmark_data_smithi122_210164_object65944:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1123)", 2024-03-20T09:46:35.482 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:43:24.287339+0000", 2024-03-20T09:46:35.482 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 191.10962188299999, 2024-03-20T09:46:35.482 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.22527365199999999, 2024-03-20T09:46:35.482 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:35.482 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:35.482 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:46:35.482 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:46:35.482 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:35.482 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 198174 2024-03-20T09:46:35.482 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.482 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:35.482 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.482 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:35.482 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.287339+0000", 2024-03-20T09:46:35.483 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.483 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.483 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.483 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:35.483 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.287339+0000", 2024-03-20T09:46:35.483 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.483 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.483 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.483 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:35.483 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.287340+0000", 2024-03-20T09:46:35.483 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1909999999999999e-06 2024-03-20T09:46:35.483 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.483 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.483 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:35.483 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.287346+0000", 2024-03-20T09:46:35.484 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.9519999999999999e-06 2024-03-20T09:46:35.484 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.484 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.484 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:35.484 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.287357+0000", 2024-03-20T09:46:35.484 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1334e-05 2024-03-20T09:46:35.484 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.484 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.484 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:35.484 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.287365+0000", 2024-03-20T09:46:35.484 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7039999999999994e-06 2024-03-20T09:46:35.484 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.484 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.484 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:35.484 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.291279+0000", 2024-03-20T09:46:35.485 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.003913934 2024-03-20T09:46:35.485 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.485 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.485 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:35.485 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.291364+0000", 2024-03-20T09:46:35.485 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.5655999999999994e-05 2024-03-20T09:46:35.485 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.485 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.485 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,7", 2024-03-20T09:46:35.485 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.291439+0000", 2024-03-20T09:46:35.485 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.4852999999999996e-05 2024-03-20T09:46:35.485 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.485 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.485 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.485 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.296163+0000", 2024-03-20T09:46:35.486 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.004724079 2024-03-20T09:46:35.486 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.486 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.486 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:46:35.486 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.300113+0000", 2024-03-20T09:46:35.486 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0039498509999999999 2024-03-20T09:46:35.486 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.486 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.486 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.486 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.512526+0000", 2024-03-20T09:46:35.486 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.21241257199999999 2024-03-20T09:46:35.486 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.486 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.486 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:35.486 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.512576+0000", 2024-03-20T09:46:35.487 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.9879999999999997e-05 2024-03-20T09:46:35.487 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.487 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.487 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:35.487 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.512612+0000", 2024-03-20T09:46:35.487 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.6646000000000001e-05 2024-03-20T09:46:35.487 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.487 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:35.487 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.487 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.487 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.487 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:198176 9.a 9:50748565:::benchmark_data_smithi122_210164_object65946:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1123)", 2024-03-20T09:46:35.487 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:43:24.288159+0000", 2024-03-20T09:46:35.487 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 191.108801497, 2024-03-20T09:46:35.487 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.228858005, 2024-03-20T09:46:35.488 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:35.488 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:35.488 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:46:35.488 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:46:35.488 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:35.488 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 198176 2024-03-20T09:46:35.488 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.488 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:35.488 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.488 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:35.488 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.288159+0000", 2024-03-20T09:46:35.488 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.488 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.488 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.488 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:35.489 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.288159+0000", 2024-03-20T09:46:35.489 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.489 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.489 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.489 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:35.489 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.288160+0000", 2024-03-20T09:46:35.489 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.4099999999999998e-07 2024-03-20T09:46:35.489 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.489 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.489 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:35.489 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.288164+0000", 2024-03-20T09:46:35.489 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.1760000000000003e-06 2024-03-20T09:46:35.489 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.489 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.489 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:35.490 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.288172+0000", 2024-03-20T09:46:35.490 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.7870000000000002e-06 2024-03-20T09:46:35.490 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.490 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.490 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:35.490 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.288183+0000", 2024-03-20T09:46:35.490 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1807999999999999e-05 2024-03-20T09:46:35.490 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.490 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.490 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:35.490 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.296199+0000", 2024-03-20T09:46:35.490 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0080152950000000004 2024-03-20T09:46:35.490 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.490 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.491 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:35.491 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.296313+0000", 2024-03-20T09:46:35.491 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.000114462 2024-03-20T09:46:35.491 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.491 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.491 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,7", 2024-03-20T09:46:35.491 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.296439+0000", 2024-03-20T09:46:35.491 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00012587500000000001 2024-03-20T09:46:35.491 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.491 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.491 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:46:35.491 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.300248+0000", 2024-03-20T09:46:35.491 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0038085110000000001 2024-03-20T09:46:35.491 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.491 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.492 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.492 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.306775+0000", 2024-03-20T09:46:35.492 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0065277979999999996 2024-03-20T09:46:35.492 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.492 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.492 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.492 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.516925+0000", 2024-03-20T09:46:35.492 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.21014925600000001 2024-03-20T09:46:35.492 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.492 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.492 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:35.492 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.516975+0000", 2024-03-20T09:46:35.492 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.0633999999999997e-05 2024-03-20T09:46:35.492 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.493 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.493 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:35.493 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.517017+0000", 2024-03-20T09:46:35.493 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.1662000000000002e-05 2024-03-20T09:46:35.493 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.493 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:35.493 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.493 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.493 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.493 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:198189 9.a 9:539eff13:::benchmark_data_smithi122_210164_object65959:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1123)", 2024-03-20T09:46:35.493 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:43:24.301564+0000", 2024-03-20T09:46:35.493 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 191.09539618700001, 2024-03-20T09:46:35.493 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.21556658200000001, 2024-03-20T09:46:35.494 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:35.494 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:35.494 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:46:35.494 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:46:35.494 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:35.494 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 198189 2024-03-20T09:46:35.494 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.494 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:35.494 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.494 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:35.494 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.301564+0000", 2024-03-20T09:46:35.494 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.494 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.495 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.495 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:35.495 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.301564+0000", 2024-03-20T09:46:35.495 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.495 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.495 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.495 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:35.495 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.301566+0000", 2024-03-20T09:46:35.495 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.2130000000000001e-06 2024-03-20T09:46:35.495 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.495 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.495 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:35.495 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.301571+0000", 2024-03-20T09:46:35.495 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.7509999999999997e-06 2024-03-20T09:46:35.495 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.496 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.496 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:35.496 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.301583+0000", 2024-03-20T09:46:35.496 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1358e-05 2024-03-20T09:46:35.496 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.496 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.496 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:35.496 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.301589+0000", 2024-03-20T09:46:35.496 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 6.1589999999999998e-06 2024-03-20T09:46:35.496 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.496 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.496 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:35.496 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.301630+0000", 2024-03-20T09:46:35.496 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.1371000000000001e-05 2024-03-20T09:46:35.497 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.497 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.497 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:35.497 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.301721+0000", 2024-03-20T09:46:35.497 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 9.0359999999999995e-05 2024-03-20T09:46:35.497 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.497 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.497 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,7", 2024-03-20T09:46:35.497 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.301802+0000", 2024-03-20T09:46:35.497 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.1336999999999997e-05 2024-03-20T09:46:35.497 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.497 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.497 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:46:35.497 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.306562+0000", 2024-03-20T09:46:35.498 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.004759859 2024-03-20T09:46:35.498 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.498 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.498 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.498 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.306821+0000", 2024-03-20T09:46:35.498 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00025963400000000001 2024-03-20T09:46:35.498 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.498 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.498 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.498 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.517057+0000", 2024-03-20T09:46:35.498 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.210235801 2024-03-20T09:46:35.498 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.498 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.498 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:35.499 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.517098+0000", 2024-03-20T09:46:35.499 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.1032999999999998e-05 2024-03-20T09:46:35.499 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.499 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.499 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:35.499 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.517131+0000", 2024-03-20T09:46:35.499 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 3.2706000000000002e-05 2024-03-20T09:46:35.499 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.499 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:35.499 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.499 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.499 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.499 INFO:teuthology.orchestra.run.smithi115.stdout: "description": "osd_op(client.27190.0:198194 9.a 9:532dd5d5:::benchmark_data_smithi122_210164_object65964:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1123)", 2024-03-20T09:46:35.499 INFO:teuthology.orchestra.run.smithi115.stdout: "initiated_at": "2024-03-20T09:43:24.311230+0000", 2024-03-20T09:46:35.500 INFO:teuthology.orchestra.run.smithi115.stdout: "age": 191.085730485, 2024-03-20T09:46:35.500 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.21511122099999999, 2024-03-20T09:46:35.500 INFO:teuthology.orchestra.run.smithi115.stdout: "type_data": { 2024-03-20T09:46:35.500 INFO:teuthology.orchestra.run.smithi115.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:35.500 INFO:teuthology.orchestra.run.smithi115.stdout: "client_info": { 2024-03-20T09:46:35.500 INFO:teuthology.orchestra.run.smithi115.stdout: "client": "client.27190", 2024-03-20T09:46:35.500 INFO:teuthology.orchestra.run.smithi115.stdout: "client_addr": "172.21.15.122:0/2544420581", 2024-03-20T09:46:35.500 INFO:teuthology.orchestra.run.smithi115.stdout: "tid": 198194 2024-03-20T09:46:35.500 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.500 INFO:teuthology.orchestra.run.smithi115.stdout: "events": [ 2024-03-20T09:46:35.500 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.500 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "initiated", 2024-03-20T09:46:35.500 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.311230+0000", 2024-03-20T09:46:35.500 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.501 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.501 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.501 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "header_read", 2024-03-20T09:46:35.501 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.311230+0000", 2024-03-20T09:46:35.501 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0 2024-03-20T09:46:35.501 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.501 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.501 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "throttled", 2024-03-20T09:46:35.501 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.311231+0000", 2024-03-20T09:46:35.501 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.1680000000000001e-06 2024-03-20T09:46:35.501 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.501 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.501 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "all_read", 2024-03-20T09:46:35.502 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.311237+0000", 2024-03-20T09:46:35.502 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.8479999999999997e-06 2024-03-20T09:46:35.502 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.502 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.502 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "dispatched", 2024-03-20T09:46:35.502 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.311250+0000", 2024-03-20T09:46:35.502 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 1.323e-05 2024-03-20T09:46:35.502 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.502 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.502 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "queued_for_pg", 2024-03-20T09:46:35.502 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.311258+0000", 2024-03-20T09:46:35.502 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 7.3499999999999999e-06 2024-03-20T09:46:35.502 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.502 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.502 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "reached_pg", 2024-03-20T09:46:35.503 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.311304+0000", 2024-03-20T09:46:35.503 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.6560999999999997e-05 2024-03-20T09:46:35.503 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.503 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.503 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "started", 2024-03-20T09:46:35.503 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.311422+0000", 2024-03-20T09:46:35.503 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00011794 2024-03-20T09:46:35.503 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.503 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.503 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "waiting for subops from 1,7", 2024-03-20T09:46:35.503 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.311503+0000", 2024-03-20T09:46:35.503 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 8.0667000000000005e-05 2024-03-20T09:46:35.503 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.503 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.504 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.504 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.316434+0000", 2024-03-20T09:46:35.504 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.0049310860000000003 2024-03-20T09:46:35.504 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.504 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.504 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "op_commit", 2024-03-20T09:46:35.504 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.317302+0000", 2024-03-20T09:46:35.504 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.00086832100000000002 2024-03-20T09:46:35.504 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.504 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.504 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:35.504 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.526240+0000", 2024-03-20T09:46:35.504 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 0.208937662 2024-03-20T09:46:35.504 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.505 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.505 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "commit_sent", 2024-03-20T09:46:35.505 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.526296+0000", 2024-03-20T09:46:35.505 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 5.6135999999999999e-05 2024-03-20T09:46:35.505 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2024-03-20T09:46:35.505 INFO:teuthology.orchestra.run.smithi115.stdout: { 2024-03-20T09:46:35.505 INFO:teuthology.orchestra.run.smithi115.stdout: "event": "done", 2024-03-20T09:46:35.505 INFO:teuthology.orchestra.run.smithi115.stdout: "time": "2024-03-20T09:43:24.526341+0000", 2024-03-20T09:46:35.505 INFO:teuthology.orchestra.run.smithi115.stdout: "duration": 4.5252e-05 2024-03-20T09:46:35.505 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.505 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:35.505 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.505 INFO:teuthology.orchestra.run.smithi115.stdout: } 2024-03-20T09:46:35.505 INFO:teuthology.orchestra.run.smithi115.stdout: ] 2024-03-20T09:46:35.505 INFO:teuthology.orchestra.run.smithi115.stdout:} 2024-03-20T09:46:35.882 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.5 to 1 (10000) 2024-03-20T09:46:36.072 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:35 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:46:36.072 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:35 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:46:36.072 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:35 smithi115 ceph-mon[72476]: pgmap v2888: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:36.072 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:35 smithi115 ceph-mon[72476]: osdmap e1190: 12 total, 12 up, 12 in 2024-03-20T09:46:36.072 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:35 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/555114298' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:46:36.072 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:35 smithi115 ceph-mon[72476]: 1.19 deep-scrub starts 2024-03-20T09:46:36.072 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:35 smithi115 ceph-mon[72476]: 1.19 deep-scrub ok 2024-03-20T09:46:36.099 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_ops_in_flight 2024-03-20T09:46:36.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:35 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:46:36.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:35 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:46:36.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:35 smithi122 ceph-mon[73495]: pgmap v2888: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:36.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:35 smithi122 ceph-mon[73495]: osdmap e1190: 12 total, 12 up, 12 in 2024-03-20T09:46:36.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:35 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/555114298' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:46:36.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:35 smithi122 ceph-mon[73495]: 1.19 deep-scrub starts 2024-03-20T09:46:36.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:35 smithi122 ceph-mon[73495]: 1.19 deep-scrub ok 2024-03-20T09:46:36.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:35 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:46:36.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:35 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:46:36.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:35 smithi026 ceph-mon[68309]: pgmap v2888: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:36.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:35 smithi026 ceph-mon[68309]: osdmap e1190: 12 total, 12 up, 12 in 2024-03-20T09:46:36.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:35 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/555114298' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:46:36.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:35 smithi026 ceph-mon[68309]: 1.19 deep-scrub starts 2024-03-20T09:46:36.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:35 smithi026 ceph-mon[68309]: 1.19 deep-scrub ok 2024-03-20T09:46:36.378 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:46:36.679 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd primary-affinity 5 1 2024-03-20T09:46:36.956 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:36 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/555114298' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-20T09:46:36.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:36 smithi026 ceph-mon[68309]: osdmap e1191: 12 total, 12 up, 12 in 2024-03-20T09:46:36.967 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:46:37.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:36 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/555114298' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-20T09:46:37.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:36 smithi115 ceph-mon[72476]: osdmap e1191: 12 total, 12 up, 12 in 2024-03-20T09:46:37.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:36 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/555114298' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-20T09:46:37.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:36 smithi122 ceph-mon[73495]: osdmap e1191: 12 total, 12 up, 12 in 2024-03-20T09:46:38.102 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:46:38.102 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:46:38.103 INFO:teuthology.orchestra.run.smithi122.stdout: "num_ops": 0 2024-03-20T09:46:38.103 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:46:38.124 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:37 smithi122 ceph-mon[73495]: 1.1b scrub starts 2024-03-20T09:46:38.124 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:37 smithi122 ceph-mon[73495]: pgmap v2891: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:38.124 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:37 smithi122 ceph-mon[73495]: 1.1b scrub ok 2024-03-20T09:46:38.124 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:37 smithi026 ceph-mon[68309]: 1.1b scrub starts 2024-03-20T09:46:38.124 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:37 smithi026 ceph-mon[68309]: pgmap v2891: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:38.124 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:37 smithi026 ceph-mon[68309]: 1.1b scrub ok 2024-03-20T09:46:38.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:37 smithi115 ceph-mon[72476]: 1.1b scrub starts 2024-03-20T09:46:38.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:37 smithi115 ceph-mon[72476]: pgmap v2891: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:38.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:37 smithi115 ceph-mon[72476]: 1.1b scrub ok 2024-03-20T09:46:38.678 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_blocked_ops 2024-03-20T09:46:38.897 INFO:teuthology.orchestra.run.smithi026.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-20T09:46:38.932 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:38 smithi122 ceph-mon[73495]: 1.2 scrub starts 2024-03-20T09:46:38.933 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:38 smithi122 ceph-mon[73495]: 1.2 scrub ok 2024-03-20T09:46:38.933 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:38 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/45575977' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:46:38.943 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:46:39.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:38 smithi026 ceph-mon[68309]: 1.2 scrub starts 2024-03-20T09:46:39.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:38 smithi026 ceph-mon[68309]: 1.2 scrub ok 2024-03-20T09:46:39.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:38 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/45575977' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:46:39.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:38 smithi115 ceph-mon[72476]: 1.2 scrub starts 2024-03-20T09:46:39.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:38 smithi115 ceph-mon[72476]: 1.2 scrub ok 2024-03-20T09:46:39.394 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:38 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/45575977' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-20T09:46:39.599 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd reweight 10 1 2024-03-20T09:46:39.878 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:46:40.035 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:39 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:46:40.035 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:39 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:46:40.036 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:39 smithi122 ceph-mon[73495]: pgmap v2892: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:40.036 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:39 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/45575977' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-20T09:46:40.036 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:39 smithi122 ceph-mon[73495]: osdmap e1192: 12 total, 12 up, 12 in 2024-03-20T09:46:40.175 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:39 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:46:40.175 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:39 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:46:40.175 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:39 smithi026 ceph-mon[68309]: pgmap v2892: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:40.175 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:39 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/45575977' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-20T09:46:40.175 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:39 smithi026 ceph-mon[68309]: osdmap e1192: 12 total, 12 up, 12 in 2024-03-20T09:46:40.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:39 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:46:40.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:39 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:46:40.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:39 smithi115 ceph-mon[72476]: pgmap v2892: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:40.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:39 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/45575977' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-20T09:46:40.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:39 smithi115 ceph-mon[72476]: osdmap e1192: 12 total, 12 up, 12 in 2024-03-20T09:46:40.609 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:46:40.610 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [], 2024-03-20T09:46:40.610 INFO:teuthology.orchestra.run.smithi122.stdout: "complaint_time": 30, 2024-03-20T09:46:40.610 INFO:teuthology.orchestra.run.smithi122.stdout: "num_blocked_ops": 0 2024-03-20T09:46:40.610 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:46:41.021 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:40 smithi026 ceph-mon[68309]: 1.1a scrub starts 2024-03-20T09:46:41.021 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:40 smithi026 ceph-mon[68309]: 1.1a scrub ok 2024-03-20T09:46:41.021 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:40 smithi026 ceph-mon[68309]: osdmap e1193: 12 total, 12 up, 12 in 2024-03-20T09:46:41.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:40 smithi122 ceph-mon[73495]: 1.1a scrub starts 2024-03-20T09:46:41.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:40 smithi122 ceph-mon[73495]: 1.1a scrub ok 2024-03-20T09:46:41.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:40 smithi122 ceph-mon[73495]: osdmap e1193: 12 total, 12 up, 12 in 2024-03-20T09:46:41.364 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph daemon osd.10 dump_historic_ops 2024-03-20T09:46:41.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:40 smithi115 ceph-mon[72476]: 1.1a scrub starts 2024-03-20T09:46:41.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:40 smithi115 ceph-mon[72476]: 1.1a scrub ok 2024-03-20T09:46:41.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:40 smithi115 ceph-mon[72476]: osdmap e1193: 12 total, 12 up, 12 in 2024-03-20T09:46:41.626 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.c/config 2024-03-20T09:46:41.909 INFO:teuthology.orchestra.run.smithi026.stderr:reweighted osd.10 to 1 (10000) 2024-03-20T09:46:42.114 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:41 smithi122 ceph-mon[73495]: pgmap v2895: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:42.115 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:41 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2261485212' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-03-20T09:46:42.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:41 smithi026 ceph-mon[68309]: pgmap v2895: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:42.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:41 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2261485212' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-03-20T09:46:42.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:41 smithi115 ceph-mon[72476]: pgmap v2895: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:42.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:41 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2261485212' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]: dispatch 2024-03-20T09:46:42.593 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd primary-affinity 10 1 2024-03-20T09:46:42.875 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:46:43.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:42 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2261485212' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-03-20T09:46:43.059 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:42 smithi122 ceph-mon[73495]: osdmap e1194: 12 total, 12 up, 12 in 2024-03-20T09:46:43.172 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:42 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2261485212' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-03-20T09:46:43.172 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:42 smithi026 ceph-mon[68309]: osdmap e1194: 12 total, 12 up, 12 in 2024-03-20T09:46:43.327 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-03-20T09:46:43.327 INFO:teuthology.orchestra.run.smithi122.stdout: "size": 20, 2024-03-20T09:46:43.327 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 600, 2024-03-20T09:46:43.327 INFO:teuthology.orchestra.run.smithi122.stdout: "ops": [ 2024-03-20T09:46:43.327 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.327 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:222446 8.2 8.a98ff882 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1050)", 2024-03-20T09:46:43.327 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:39:33.007785+0000", 2024-03-20T09:46:43.327 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 430.33058456399999, 2024-03-20T09:46:43.327 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.92627273099999996, 2024-03-20T09:46:43.328 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:43.328 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:46:43.328 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:43.328 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:43.328 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:43.328 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 222446 2024-03-20T09:46:43.328 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.328 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:43.328 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.328 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:43.328 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.007785+0000", 2024-03-20T09:46:43.328 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.328 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.328 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.328 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:43.329 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.007785+0000", 2024-03-20T09:46:43.329 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.329 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.329 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.329 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:43.329 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.007786+0000", 2024-03-20T09:46:43.329 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.403e-06 2024-03-20T09:46:43.329 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.329 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.329 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:43.329 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.007792+0000", 2024-03-20T09:46:43.329 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.75e-06 2024-03-20T09:46:43.329 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.330 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.330 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:43.330 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.007804+0000", 2024-03-20T09:46:43.330 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2087e-05 2024-03-20T09:46:43.330 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.330 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.330 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:43.330 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.007812+0000", 2024-03-20T09:46:43.330 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8679999999999999e-06 2024-03-20T09:46:43.330 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.330 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.330 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.330 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.007870+0000", 2024-03-20T09:46:43.330 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8279999999999998e-05 2024-03-20T09:46:43.331 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.331 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.331 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:46:43.331 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.007879+0000", 2024-03-20T09:46:43.331 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6999999999999997e-06 2024-03-20T09:46:43.331 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.331 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.331 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.331 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.934048+0000", 2024-03-20T09:46:43.331 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.92616935 2024-03-20T09:46:43.331 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.331 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.331 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:43.331 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.934057+0000", 2024-03-20T09:46:43.331 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2930000000000006e-06 2024-03-20T09:46:43.332 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.332 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:43.332 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.332 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.332 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.332 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:222504 8.2 8.3f88e912 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1050)", 2024-03-20T09:46:43.332 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:39:33.074310+0000", 2024-03-20T09:46:43.332 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 430.264059143, 2024-03-20T09:46:43.332 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.86017001199999998, 2024-03-20T09:46:43.332 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:43.332 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:46:43.332 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:43.332 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:43.332 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:43.333 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 222504 2024-03-20T09:46:43.333 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.333 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:43.333 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.333 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:43.333 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.074310+0000", 2024-03-20T09:46:43.333 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.333 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.333 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.333 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:43.333 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.074310+0000", 2024-03-20T09:46:43.333 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.333 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.333 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.333 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:43.334 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.074312+0000", 2024-03-20T09:46:43.334 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6729999999999999e-06 2024-03-20T09:46:43.334 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.334 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.334 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:43.334 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.074318+0000", 2024-03-20T09:46:43.334 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.4779999999999996e-06 2024-03-20T09:46:43.334 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.334 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.334 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:43.334 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.074333+0000", 2024-03-20T09:46:43.334 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5140000000000001e-05 2024-03-20T09:46:43.334 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.335 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.335 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:43.335 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.074342+0000", 2024-03-20T09:46:43.335 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5560000000000008e-06 2024-03-20T09:46:43.335 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.335 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.335 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.335 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.074456+0000", 2024-03-20T09:46:43.335 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000114189 2024-03-20T09:46:43.335 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.335 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.335 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:46:43.335 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.074466+0000", 2024-03-20T09:46:43.335 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5589999999999994e-06 2024-03-20T09:46:43.335 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.336 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.336 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.336 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.934471+0000", 2024-03-20T09:46:43.336 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.86000494299999997 2024-03-20T09:46:43.336 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.336 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.336 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:43.336 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.934480+0000", 2024-03-20T09:46:43.336 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.4739999999999994e-06 2024-03-20T09:46:43.336 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.336 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:43.336 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.336 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.337 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.337 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:222505 8.2 8.89e21032 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1050)", 2024-03-20T09:46:43.337 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:39:33.074367+0000", 2024-03-20T09:46:43.337 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 430.26400263300002, 2024-03-20T09:46:43.337 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.86013762100000002, 2024-03-20T09:46:43.337 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:43.337 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:46:43.337 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:43.337 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:43.337 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:43.337 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 222505 2024-03-20T09:46:43.337 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.337 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:43.337 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.337 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:43.338 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.074367+0000", 2024-03-20T09:46:43.338 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.338 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.338 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.338 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:43.338 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.074367+0000", 2024-03-20T09:46:43.338 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.338 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.338 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.338 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:43.338 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.074367+0000", 2024-03-20T09:46:43.338 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1800000000000001e-07 2024-03-20T09:46:43.338 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.338 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.339 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:43.339 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.074370+0000", 2024-03-20T09:46:43.339 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.6230000000000001e-06 2024-03-20T09:46:43.339 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.339 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.339 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:43.339 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.074377+0000", 2024-03-20T09:46:43.339 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3370000000000002e-06 2024-03-20T09:46:43.339 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.339 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.339 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:43.339 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.074382+0000", 2024-03-20T09:46:43.339 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0069999999999997e-06 2024-03-20T09:46:43.339 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.339 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.340 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.340 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.074487+0000", 2024-03-20T09:46:43.340 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000105565 2024-03-20T09:46:43.340 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.340 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.340 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:46:43.340 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.074493+0000", 2024-03-20T09:46:43.340 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6030000000000004e-06 2024-03-20T09:46:43.340 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.340 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.340 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.340 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.934499+0000", 2024-03-20T09:46:43.340 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.86000615999999996 2024-03-20T09:46:43.340 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.340 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.341 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:43.341 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.934504+0000", 2024-03-20T09:46:43.341 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.9080000000000003e-06 2024-03-20T09:46:43.341 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.341 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:43.341 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.341 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.341 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.341 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:222536 8.2 8.93f55852 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1050)", 2024-03-20T09:46:43.341 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:39:33.105520+0000", 2024-03-20T09:46:43.341 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 430.23284894800003, 2024-03-20T09:46:43.341 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.82902991100000001, 2024-03-20T09:46:43.341 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:43.341 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:46:43.341 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:43.342 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:43.342 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:43.342 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 222536 2024-03-20T09:46:43.342 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.342 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:43.342 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.342 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:43.342 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.105520+0000", 2024-03-20T09:46:43.342 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.342 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.342 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.342 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:43.342 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.105520+0000", 2024-03-20T09:46:43.342 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.343 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.343 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.343 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:43.343 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.105521+0000", 2024-03-20T09:46:43.343 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.5999999999999991e-07 2024-03-20T09:46:43.343 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.343 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.343 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:43.343 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.105526+0000", 2024-03-20T09:46:43.343 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6439999999999999e-06 2024-03-20T09:46:43.343 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.343 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.343 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:43.343 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.105540+0000", 2024-03-20T09:46:43.343 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.4022999999999999e-05 2024-03-20T09:46:43.344 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.344 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.344 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:43.344 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.105557+0000", 2024-03-20T09:46:43.344 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.6840000000000001e-05 2024-03-20T09:46:43.344 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.344 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.344 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.344 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.109103+0000", 2024-03-20T09:46:43.344 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003546422 2024-03-20T09:46:43.344 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.344 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.344 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:46:43.344 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.109112+0000", 2024-03-20T09:46:43.344 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2220000000000008e-06 2024-03-20T09:46:43.345 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.345 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.345 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.345 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.934532+0000", 2024-03-20T09:46:43.345 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.825419876 2024-03-20T09:46:43.345 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.345 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.345 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:43.345 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.934550+0000", 2024-03-20T09:46:43.345 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.7924e-05 2024-03-20T09:46:43.345 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.345 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:43.345 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.346 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.346 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.346 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:222548 8.2 8.fe3e23c2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1050)", 2024-03-20T09:46:43.346 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:39:33.120490+0000", 2024-03-20T09:46:43.346 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 430.21787914399999, 2024-03-20T09:46:43.346 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.814110785, 2024-03-20T09:46:43.346 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:43.346 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:46:43.346 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:43.346 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:43.346 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:43.346 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 222548 2024-03-20T09:46:43.346 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.346 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:43.346 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.347 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:43.347 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.120490+0000", 2024-03-20T09:46:43.347 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.347 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.347 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.347 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:43.347 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.120490+0000", 2024-03-20T09:46:43.347 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.347 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.347 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.347 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:43.347 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.120491+0000", 2024-03-20T09:46:43.347 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.305e-06 2024-03-20T09:46:43.347 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.348 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.348 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:43.348 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.120497+0000", 2024-03-20T09:46:43.348 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5520000000000004e-06 2024-03-20T09:46:43.348 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.348 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.348 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:43.348 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.120512+0000", 2024-03-20T09:46:43.348 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5082000000000001e-05 2024-03-20T09:46:43.348 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.348 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.348 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:43.348 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.120521+0000", 2024-03-20T09:46:43.348 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.7970000000000006e-06 2024-03-20T09:46:43.349 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.349 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.349 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.349 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.120580+0000", 2024-03-20T09:46:43.349 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8810000000000001e-05 2024-03-20T09:46:43.349 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.349 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.349 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:46:43.349 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.120589+0000", 2024-03-20T09:46:43.349 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.0710000000000001e-06 2024-03-20T09:46:43.349 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.349 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.349 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.349 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.934585+0000", 2024-03-20T09:46:43.349 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.813996355 2024-03-20T09:46:43.350 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.350 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.350 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:43.350 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.934601+0000", 2024-03-20T09:46:43.350 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5812999999999999e-05 2024-03-20T09:46:43.350 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.350 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:43.350 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.350 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.350 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.350 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:222560 8.2 8.d8dcadf2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1050)", 2024-03-20T09:46:43.350 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:39:33.135247+0000", 2024-03-20T09:46:43.350 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 430.20312248800002, 2024-03-20T09:46:43.350 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.79938340399999996, 2024-03-20T09:46:43.351 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:43.351 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:46:43.351 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:43.351 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:43.351 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:43.351 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 222560 2024-03-20T09:46:43.351 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.351 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:43.351 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.351 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:43.351 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.135247+0000", 2024-03-20T09:46:43.351 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.351 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.351 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.352 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:43.352 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.135247+0000", 2024-03-20T09:46:43.352 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.352 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.352 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.352 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:43.352 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.135248+0000", 2024-03-20T09:46:43.352 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6600000000000005e-07 2024-03-20T09:46:43.352 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.352 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.352 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:43.352 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.135252+0000", 2024-03-20T09:46:43.352 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.6090000000000003e-06 2024-03-20T09:46:43.352 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.353 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.353 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:43.353 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.135263+0000", 2024-03-20T09:46:43.353 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0655999999999999e-05 2024-03-20T09:46:43.353 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.353 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.353 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:43.353 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.135270+0000", 2024-03-20T09:46:43.353 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.2779999999999996e-06 2024-03-20T09:46:43.353 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.353 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.353 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.353 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.135333+0000", 2024-03-20T09:46:43.353 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2978999999999995e-05 2024-03-20T09:46:43.353 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.354 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.354 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:46:43.354 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.135341+0000", 2024-03-20T09:46:43.354 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6750000000000002e-06 2024-03-20T09:46:43.354 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.354 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.354 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.354 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.934620+0000", 2024-03-20T09:46:43.354 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.79927906299999996 2024-03-20T09:46:43.354 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.354 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.354 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:43.354 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.934630+0000", 2024-03-20T09:46:43.355 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0278000000000001e-05 2024-03-20T09:46:43.355 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.355 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:43.355 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.355 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.355 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.355 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:222563 8.2 8.571f8522 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1050)", 2024-03-20T09:46:43.355 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:39:33.140582+0000", 2024-03-20T09:46:43.355 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 430.19778672899997, 2024-03-20T09:46:43.355 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.79412829100000004, 2024-03-20T09:46:43.355 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:43.355 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:46:43.355 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:43.355 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:43.355 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:43.356 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 222563 2024-03-20T09:46:43.356 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.356 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:43.356 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.356 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:43.356 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.140582+0000", 2024-03-20T09:46:43.356 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.356 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.356 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.356 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:43.356 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.140582+0000", 2024-03-20T09:46:43.356 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.356 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.356 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.356 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:43.357 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.140584+0000", 2024-03-20T09:46:43.357 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2389999999999999e-06 2024-03-20T09:46:43.357 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.357 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.357 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:43.357 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.140589+0000", 2024-03-20T09:46:43.357 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2220000000000001e-06 2024-03-20T09:46:43.357 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.357 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.357 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:43.357 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.140600+0000", 2024-03-20T09:46:43.357 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.1617000000000001e-05 2024-03-20T09:46:43.357 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.357 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.357 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:43.357 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.140608+0000", 2024-03-20T09:46:43.358 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3050000000000004e-06 2024-03-20T09:46:43.358 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.358 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.358 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.358 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.140670+0000", 2024-03-20T09:46:43.358 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.2204000000000003e-05 2024-03-20T09:46:43.358 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.358 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.358 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:46:43.358 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.140678+0000", 2024-03-20T09:46:43.358 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3359999999999995e-06 2024-03-20T09:46:43.358 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.358 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.359 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.359 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.934650+0000", 2024-03-20T09:46:43.359 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.79397132000000004 2024-03-20T09:46:43.359 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.359 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.359 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:43.359 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.934711+0000", 2024-03-20T09:46:43.359 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1048000000000002e-05 2024-03-20T09:46:43.359 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.359 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:43.359 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.359 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.359 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.359 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:222572 8.2 8.4c995a2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1050)", 2024-03-20T09:46:43.360 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:39:33.149772+0000", 2024-03-20T09:46:43.360 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 430.18859673700001, 2024-03-20T09:46:43.360 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.78497127200000005, 2024-03-20T09:46:43.360 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:43.360 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:46:43.360 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:43.360 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:43.360 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:43.360 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 222572 2024-03-20T09:46:43.360 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.360 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:43.360 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.360 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:43.360 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.149772+0000", 2024-03-20T09:46:43.360 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.361 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.361 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.361 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:43.361 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.149772+0000", 2024-03-20T09:46:43.361 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.361 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.361 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.361 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:43.361 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.149774+0000", 2024-03-20T09:46:43.361 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2240000000000001e-06 2024-03-20T09:46:43.361 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.361 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.361 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:43.361 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.149779+0000", 2024-03-20T09:46:43.361 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5570000000000003e-06 2024-03-20T09:46:43.361 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.362 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.362 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:43.362 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.149793+0000", 2024-03-20T09:46:43.362 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3949e-05 2024-03-20T09:46:43.362 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.362 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.362 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:43.362 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.149801+0000", 2024-03-20T09:46:43.362 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9859999999999994e-06 2024-03-20T09:46:43.362 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.362 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.362 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.363 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.149862+0000", 2024-03-20T09:46:43.363 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0931999999999999e-05 2024-03-20T09:46:43.363 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.363 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.363 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:46:43.363 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.149872+0000", 2024-03-20T09:46:43.363 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.7200000000000001e-06 2024-03-20T09:46:43.363 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.363 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.363 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.363 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.934738+0000", 2024-03-20T09:46:43.363 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.78486572899999996 2024-03-20T09:46:43.363 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.363 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.363 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:43.364 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.934744+0000", 2024-03-20T09:46:43.364 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.1750000000000002e-06 2024-03-20T09:46:43.364 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.364 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:43.364 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.364 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.364 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.364 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:222576 8.2 8.2a787592 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1050)", 2024-03-20T09:46:43.364 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:39:33.156435+0000", 2024-03-20T09:46:43.364 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 430.181934598, 2024-03-20T09:46:43.364 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.778334741, 2024-03-20T09:46:43.364 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:43.364 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:46:43.364 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:43.365 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:43.365 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:43.365 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 222576 2024-03-20T09:46:43.365 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.365 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:43.365 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.365 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:43.365 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.156435+0000", 2024-03-20T09:46:43.365 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.365 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.365 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.365 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:43.365 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.156435+0000", 2024-03-20T09:46:43.365 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.365 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.366 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.366 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:43.366 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.156436+0000", 2024-03-20T09:46:43.366 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.8100000000000001e-07 2024-03-20T09:46:43.366 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.366 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.366 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:43.366 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.156440+0000", 2024-03-20T09:46:43.366 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2339999999999996e-06 2024-03-20T09:46:43.366 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.366 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.366 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:43.366 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.156450+0000", 2024-03-20T09:46:43.366 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0530000000000001e-05 2024-03-20T09:46:43.367 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.367 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.367 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:43.367 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.156458+0000", 2024-03-20T09:46:43.367 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9740000000000002e-06 2024-03-20T09:46:43.367 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.367 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.367 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.367 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.156506+0000", 2024-03-20T09:46:43.367 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7738999999999998e-05 2024-03-20T09:46:43.367 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.367 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.367 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:46:43.367 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.156513+0000", 2024-03-20T09:46:43.367 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.0550000000000002e-06 2024-03-20T09:46:43.368 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.368 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.368 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.368 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.934764+0000", 2024-03-20T09:46:43.368 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.77825058300000005 2024-03-20T09:46:43.368 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.368 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.368 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:43.368 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.934769+0000", 2024-03-20T09:46:43.368 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6450000000000002e-06 2024-03-20T09:46:43.368 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.368 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:43.368 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.368 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.369 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.369 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:222588 8.2 8.a3270a62 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1050)", 2024-03-20T09:46:43.369 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:39:33.169444+0000", 2024-03-20T09:46:43.369 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 430.16892470900001, 2024-03-20T09:46:43.369 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.76535092000000005, 2024-03-20T09:46:43.369 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:43.369 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "reached pg", 2024-03-20T09:46:43.369 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:43.369 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:43.369 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:43.369 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 222588 2024-03-20T09:46:43.369 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.369 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:43.369 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.369 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:43.370 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.169444+0000", 2024-03-20T09:46:43.370 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.370 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.370 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.370 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:43.370 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.169444+0000", 2024-03-20T09:46:43.370 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.370 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.370 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.370 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:43.370 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.169445+0000", 2024-03-20T09:46:43.370 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.0410000000000001e-06 2024-03-20T09:46:43.370 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.371 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.371 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:43.371 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.169450+0000", 2024-03-20T09:46:43.371 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1679999999999997e-06 2024-03-20T09:46:43.371 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.371 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.371 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:43.371 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.169459+0000", 2024-03-20T09:46:43.371 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2820000000000002e-06 2024-03-20T09:46:43.371 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.371 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.371 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:43.371 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.169465+0000", 2024-03-20T09:46:43.371 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.8490000000000002e-06 2024-03-20T09:46:43.371 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.371 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.372 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.372 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.169536+0000", 2024-03-20T09:46:43.372 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.1508999999999996e-05 2024-03-20T09:46:43.372 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.372 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.372 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:46:43.372 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.169546+0000", 2024-03-20T09:46:43.372 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2599999999999994e-06 2024-03-20T09:46:43.372 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.372 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.372 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.372 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.934791+0000", 2024-03-20T09:46:43.372 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.76524510300000004 2024-03-20T09:46:43.372 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.372 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.373 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:43.373 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.934795+0000", 2024-03-20T09:46:43.373 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7079999999999996e-06 2024-03-20T09:46:43.373 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.373 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:43.373 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.373 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.373 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.373 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:222504 8.2 8:489711fc:::benchmark_data_smithi122_197775_object99200:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1050)", 2024-03-20T09:46:43.373 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:39:33.181368+0000", 2024-03-20T09:46:43.373 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 430.15700099399999, 2024-03-20T09:46:43.373 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.76508029799999999, 2024-03-20T09:46:43.373 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:43.373 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:43.373 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:43.374 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:43.374 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:43.374 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 222504 2024-03-20T09:46:43.374 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.374 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:43.374 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.374 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:43.374 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181368+0000", 2024-03-20T09:46:43.374 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.374 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.374 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.374 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:43.374 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181368+0000", 2024-03-20T09:46:43.374 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.375 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.375 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.375 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:43.375 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181369+0000", 2024-03-20T09:46:43.375 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3799999999999996e-07 2024-03-20T09:46:43.375 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.375 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.375 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:43.375 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181373+0000", 2024-03-20T09:46:43.375 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2370000000000003e-06 2024-03-20T09:46:43.375 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.375 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.375 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:43.375 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181385+0000", 2024-03-20T09:46:43.376 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.2174999999999999e-05 2024-03-20T09:46:43.376 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.376 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.376 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:43.376 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181393+0000", 2024-03-20T09:46:43.376 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9340000000000006e-06 2024-03-20T09:46:43.376 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.376 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.376 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.376 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181444+0000", 2024-03-20T09:46:43.376 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1255999999999999e-05 2024-03-20T09:46:43.376 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.376 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.376 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:46:43.377 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181449+0000", 2024-03-20T09:46:43.377 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0570000000000003e-06 2024-03-20T09:46:43.377 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.377 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.377 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.377 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.938642+0000", 2024-03-20T09:46:43.377 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.75719352600000001 2024-03-20T09:46:43.377 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.377 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.377 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:43.377 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.938772+0000", 2024-03-20T09:46:43.377 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012968300000000001 2024-03-20T09:46:43.377 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.377 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.378 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 4,9", 2024-03-20T09:46:43.378 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.938906+0000", 2024-03-20T09:46:43.378 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013441700000000001 2024-03-20T09:46:43.378 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.378 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.378 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:43.378 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.942418+0000", 2024-03-20T09:46:43.378 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.003512201 2024-03-20T09:46:43.378 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.378 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.378 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:43.378 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.942548+0000", 2024-03-20T09:46:43.378 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012920799999999999 2024-03-20T09:46:43.379 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.379 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.379 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:43.379 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.946343+0000", 2024-03-20T09:46:43.379 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0037952049999999998 2024-03-20T09:46:43.379 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.379 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.379 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:43.379 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.946400+0000", 2024-03-20T09:46:43.379 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.7025999999999999e-05 2024-03-20T09:46:43.379 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.379 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.379 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:43.379 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.946448+0000", 2024-03-20T09:46:43.379 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8634999999999998e-05 2024-03-20T09:46:43.380 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.380 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:43.380 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.380 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.380 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.380 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:222505 8.2 8:4c084791:::benchmark_data_smithi122_197775_object99201:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1050)", 2024-03-20T09:46:43.380 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:39:33.181406+0000", 2024-03-20T09:46:43.380 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 430.15696285199999, 2024-03-20T09:46:43.380 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.76970130199999998, 2024-03-20T09:46:43.380 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:43.380 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:43.380 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:43.380 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:43.380 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:43.380 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 222505 2024-03-20T09:46:43.380 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.381 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:43.381 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.381 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:43.381 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181406+0000", 2024-03-20T09:46:43.381 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.381 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.381 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.381 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:43.381 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181406+0000", 2024-03-20T09:46:43.381 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.381 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.381 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.381 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:43.381 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181407+0000", 2024-03-20T09:46:43.381 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4799999999999999e-07 2024-03-20T09:46:43.381 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.382 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.382 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:43.382 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181410+0000", 2024-03-20T09:46:43.382 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.0280000000000001e-06 2024-03-20T09:46:43.382 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.382 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.382 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:43.382 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181415+0000", 2024-03-20T09:46:43.382 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8239999999999999e-06 2024-03-20T09:46:43.382 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.382 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.382 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:43.382 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181418+0000", 2024-03-20T09:46:43.382 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6880000000000001e-06 2024-03-20T09:46:43.383 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.383 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.383 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.383 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181493+0000", 2024-03-20T09:46:43.383 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.5150999999999999e-05 2024-03-20T09:46:43.383 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.383 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.383 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:46:43.383 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181501+0000", 2024-03-20T09:46:43.383 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7170000000000008e-06 2024-03-20T09:46:43.383 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.383 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.383 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.383 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.942591+0000", 2024-03-20T09:46:43.383 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.76108989900000001 2024-03-20T09:46:43.384 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.384 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.384 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:43.384 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.942681+0000", 2024-03-20T09:46:43.384 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.9716000000000006e-05 2024-03-20T09:46:43.384 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.384 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.384 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 4,9", 2024-03-20T09:46:43.384 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.942760+0000", 2024-03-20T09:46:43.384 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.9263000000000003e-05 2024-03-20T09:46:43.384 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.384 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.384 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:43.384 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.951000+0000", 2024-03-20T09:46:43.384 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0082401699999999998 2024-03-20T09:46:43.385 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.385 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.385 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:43.385 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.951016+0000", 2024-03-20T09:46:43.385 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.5685e-05 2024-03-20T09:46:43.385 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.385 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.385 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:43.385 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.951044+0000", 2024-03-20T09:46:43.385 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8367000000000001e-05 2024-03-20T09:46:43.385 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.385 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.385 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:43.385 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.951079+0000", 2024-03-20T09:46:43.385 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5040000000000003e-05 2024-03-20T09:46:43.385 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.386 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.386 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:43.386 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.951108+0000", 2024-03-20T09:46:43.386 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.8306e-05 2024-03-20T09:46:43.386 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.386 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:43.386 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.386 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.386 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.386 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:222536 8.2 8:4a1aafc9:::benchmark_data_smithi122_197775_object99232:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1050)", 2024-03-20T09:46:43.386 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:39:33.181460+0000", 2024-03-20T09:46:43.386 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 430.15690868500002, 2024-03-20T09:46:43.386 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.77438609999999997, 2024-03-20T09:46:43.386 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:43.386 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:43.387 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:43.387 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:43.387 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:43.387 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 222536 2024-03-20T09:46:43.387 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.387 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:43.387 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.387 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:43.387 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181460+0000", 2024-03-20T09:46:43.387 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.387 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.387 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.387 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:43.387 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181460+0000", 2024-03-20T09:46:43.387 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.388 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.388 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.388 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:43.388 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181461+0000", 2024-03-20T09:46:43.388 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.4799999999999997e-07 2024-03-20T09:46:43.388 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.388 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.388 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:43.388 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181466+0000", 2024-03-20T09:46:43.388 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2549999999999999e-06 2024-03-20T09:46:43.388 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.388 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.388 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:43.388 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181473+0000", 2024-03-20T09:46:43.389 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.61e-06 2024-03-20T09:46:43.389 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.389 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.389 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:43.389 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181481+0000", 2024-03-20T09:46:43.389 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.3379999999999999e-06 2024-03-20T09:46:43.389 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.389 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.389 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.389 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181533+0000", 2024-03-20T09:46:43.389 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2210999999999999e-05 2024-03-20T09:46:43.389 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.389 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.389 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:46:43.389 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181540+0000", 2024-03-20T09:46:43.390 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.6369999999999999e-06 2024-03-20T09:46:43.390 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.390 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.390 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.390 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.946491+0000", 2024-03-20T09:46:43.390 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.76495049100000001 2024-03-20T09:46:43.390 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.390 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.390 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:43.390 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.946596+0000", 2024-03-20T09:46:43.390 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000104968 2024-03-20T09:46:43.390 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.390 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.391 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 4,9", 2024-03-20T09:46:43.391 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.946681+0000", 2024-03-20T09:46:43.391 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.5347999999999997e-05 2024-03-20T09:46:43.391 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.391 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.391 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:43.391 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.951150+0000", 2024-03-20T09:46:43.391 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0044691189999999997 2024-03-20T09:46:43.391 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.391 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.391 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:43.391 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.951198+0000", 2024-03-20T09:46:43.391 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7633999999999998e-05 2024-03-20T09:46:43.391 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.391 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.392 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:43.392 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.955765+0000", 2024-03-20T09:46:43.392 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.004566985 2024-03-20T09:46:43.392 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.392 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.392 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:43.392 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.955811+0000", 2024-03-20T09:46:43.392 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.5686999999999998e-05 2024-03-20T09:46:43.392 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.392 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.392 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:43.392 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.955847+0000", 2024-03-20T09:46:43.392 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5969000000000001e-05 2024-03-20T09:46:43.392 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.392 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:43.393 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.393 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.393 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.393 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:222548 8.2 8:43c47c7f:::benchmark_data_smithi122_197775_object99244:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1050)", 2024-03-20T09:46:43.393 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:39:33.181494+0000", 2024-03-20T09:46:43.393 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 430.15687531200001, 2024-03-20T09:46:43.393 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.78337511500000001, 2024-03-20T09:46:43.393 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:43.393 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:43.393 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:43.393 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:43.393 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:43.393 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 222548 2024-03-20T09:46:43.393 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.394 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:43.394 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.394 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:43.394 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181494+0000", 2024-03-20T09:46:43.394 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.394 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.394 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.394 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:43.394 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181494+0000", 2024-03-20T09:46:43.394 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.394 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.394 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.394 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:43.395 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181494+0000", 2024-03-20T09:46:43.395 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1699999999999999e-07 2024-03-20T09:46:43.395 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.395 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.395 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:43.395 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181497+0000", 2024-03-20T09:46:43.395 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.52e-06 2024-03-20T09:46:43.395 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.395 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.395 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:43.395 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181503+0000", 2024-03-20T09:46:43.395 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9619999999999998e-06 2024-03-20T09:46:43.395 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.395 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.395 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:43.396 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181508+0000", 2024-03-20T09:46:43.396 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.7890000000000002e-06 2024-03-20T09:46:43.396 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.396 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.396 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.396 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181564+0000", 2024-03-20T09:46:43.396 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.6314000000000003e-05 2024-03-20T09:46:43.396 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.396 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.396 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:46:43.396 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181569+0000", 2024-03-20T09:46:43.396 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5600000000000001e-06 2024-03-20T09:46:43.396 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.396 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.397 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.397 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.951230+0000", 2024-03-20T09:46:43.397 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.76966025299999996 2024-03-20T09:46:43.397 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.397 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.397 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:43.397 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.951322+0000", 2024-03-20T09:46:43.397 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.2294999999999996e-05 2024-03-20T09:46:43.397 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.397 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.397 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 4,9", 2024-03-20T09:46:43.397 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.951413+0000", 2024-03-20T09:46:43.397 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.1174000000000005e-05 2024-03-20T09:46:43.397 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.398 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.398 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:43.398 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.960170+0000", 2024-03-20T09:46:43.398 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0087566940000000006 2024-03-20T09:46:43.398 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.398 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.398 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:43.398 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.960207+0000", 2024-03-20T09:46:43.398 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.6829999999999998e-05 2024-03-20T09:46:43.398 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.398 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.398 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:43.398 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.964784+0000", 2024-03-20T09:46:43.398 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0045772340000000003 2024-03-20T09:46:43.399 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.399 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.399 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:43.399 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.964826+0000", 2024-03-20T09:46:43.399 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1993999999999998e-05 2024-03-20T09:46:43.399 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.399 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.399 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:43.399 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.964869+0000", 2024-03-20T09:46:43.399 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3078999999999999e-05 2024-03-20T09:46:43.399 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.399 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:43.399 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.399 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.400 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.400 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:222560 8.2 8:4fb53b1b:::benchmark_data_smithi122_197775_object99256:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1050)", 2024-03-20T09:46:43.400 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:39:33.181523+0000", 2024-03-20T09:46:43.400 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 430.15684570600001, 2024-03-20T09:46:43.400 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.78358289999999997, 2024-03-20T09:46:43.400 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:43.400 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:43.400 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:43.400 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:43.400 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:43.400 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 222560 2024-03-20T09:46:43.400 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.400 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:43.400 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.401 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:43.401 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181523+0000", 2024-03-20T09:46:43.401 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.401 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.401 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.401 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:43.401 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181523+0000", 2024-03-20T09:46:43.401 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.401 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.401 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.401 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:43.401 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181524+0000", 2024-03-20T09:46:43.401 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9999999999999998e-07 2024-03-20T09:46:43.401 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.402 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.402 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:43.402 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181526+0000", 2024-03-20T09:46:43.402 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.5380000000000001e-06 2024-03-20T09:46:43.402 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.402 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.402 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:43.402 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181533+0000", 2024-03-20T09:46:43.402 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8809999999999998e-06 2024-03-20T09:46:43.402 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.402 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.402 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:43.402 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181539+0000", 2024-03-20T09:46:43.402 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4240000000000001e-06 2024-03-20T09:46:43.403 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.403 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.403 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.403 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181590+0000", 2024-03-20T09:46:43.403 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1672999999999998e-05 2024-03-20T09:46:43.403 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.403 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.403 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:46:43.403 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181596+0000", 2024-03-20T09:46:43.403 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4689999999999996e-06 2024-03-20T09:46:43.403 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.403 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.403 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.403 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.955893+0000", 2024-03-20T09:46:43.403 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.77429719799999996 2024-03-20T09:46:43.404 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.404 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.404 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:43.404 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.956007+0000", 2024-03-20T09:46:43.404 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000113737 2024-03-20T09:46:43.404 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.404 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.404 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 4,9", 2024-03-20T09:46:43.404 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.956125+0000", 2024-03-20T09:46:43.404 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000118031 2024-03-20T09:46:43.404 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.404 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.404 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:43.404 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.960251+0000", 2024-03-20T09:46:43.404 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.004126368 2024-03-20T09:46:43.404 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.405 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.405 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:43.405 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.964937+0000", 2024-03-20T09:46:43.405 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0046853440000000001 2024-03-20T09:46:43.405 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.405 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.405 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:43.405 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.965014+0000", 2024-03-20T09:46:43.405 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.7167000000000002e-05 2024-03-20T09:46:43.405 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.405 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.405 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:43.405 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.965067+0000", 2024-03-20T09:46:43.405 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3661e-05 2024-03-20T09:46:43.406 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.406 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.406 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:43.406 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.965106+0000", 2024-03-20T09:46:43.406 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.9008999999999999e-05 2024-03-20T09:46:43.406 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.406 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:43.406 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.406 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.406 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.406 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:222563 8.2 8:44a1f8ea:::benchmark_data_smithi122_197775_object99259:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1050)", 2024-03-20T09:46:43.406 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:39:33.181560+0000", 2024-03-20T09:46:43.406 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 430.15680906900002, 2024-03-20T09:46:43.406 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.78764198399999996, 2024-03-20T09:46:43.406 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:43.407 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:43.407 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:43.407 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:43.407 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:43.407 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 222563 2024-03-20T09:46:43.407 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.407 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:43.407 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.407 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:43.407 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181560+0000", 2024-03-20T09:46:43.407 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.407 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.407 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.407 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:43.407 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181560+0000", 2024-03-20T09:46:43.408 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.408 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.408 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.408 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:43.408 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181560+0000", 2024-03-20T09:46:43.408 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.0999999999999999e-07 2024-03-20T09:46:43.408 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.408 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.408 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:43.408 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181563+0000", 2024-03-20T09:46:43.408 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.543e-06 2024-03-20T09:46:43.408 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.408 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.408 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:43.408 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181570+0000", 2024-03-20T09:46:43.409 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.781e-06 2024-03-20T09:46:43.409 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.409 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.409 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:43.409 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181575+0000", 2024-03-20T09:46:43.409 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4399999999999996e-06 2024-03-20T09:46:43.409 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.409 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.409 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.409 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181617+0000", 2024-03-20T09:46:43.409 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.1309e-05 2024-03-20T09:46:43.409 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.409 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.409 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:46:43.410 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181623+0000", 2024-03-20T09:46:43.410 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0589999999999999e-06 2024-03-20T09:46:43.410 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.410 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.410 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.410 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.960280+0000", 2024-03-20T09:46:43.410 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.77865716799999996 2024-03-20T09:46:43.410 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.410 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.410 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:43.410 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.960363+0000", 2024-03-20T09:46:43.410 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.3628999999999995e-05 2024-03-20T09:46:43.410 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.410 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.410 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 4,9", 2024-03-20T09:46:43.411 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.960450+0000", 2024-03-20T09:46:43.411 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 8.6394e-05 2024-03-20T09:46:43.411 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.411 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.411 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:43.411 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.964998+0000", 2024-03-20T09:46:43.411 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0045480160000000002 2024-03-20T09:46:43.411 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.411 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.411 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:43.411 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.969090+0000", 2024-03-20T09:46:43.411 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.004092559 2024-03-20T09:46:43.411 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.411 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.411 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:43.411 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.969104+0000", 2024-03-20T09:46:43.412 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 1.3944999999999999e-05 2024-03-20T09:46:43.412 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.412 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.412 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:43.412 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.969159+0000", 2024-03-20T09:46:43.412 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5055000000000001e-05 2024-03-20T09:46:43.412 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.412 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.412 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:43.412 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.969202+0000", 2024-03-20T09:46:43.412 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.2676000000000003e-05 2024-03-20T09:46:43.412 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.412 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:43.413 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.413 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.413 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.413 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:222572 8.2 8:45a99320:::benchmark_data_smithi122_197775_object99268:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1050)", 2024-03-20T09:46:43.413 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:39:33.181584+0000", 2024-03-20T09:46:43.413 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 430.15678507699999, 2024-03-20T09:46:43.413 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.79638384699999998, 2024-03-20T09:46:43.413 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:43.413 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:43.413 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:43.413 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:43.413 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:43.413 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 222572 2024-03-20T09:46:43.413 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.413 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:43.414 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.414 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:43.414 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181584+0000", 2024-03-20T09:46:43.414 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.414 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.414 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.414 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:43.414 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181584+0000", 2024-03-20T09:46:43.414 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.414 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.414 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.414 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:43.414 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181584+0000", 2024-03-20T09:46:43.414 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3000000000000002e-07 2024-03-20T09:46:43.414 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.415 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.415 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:43.415 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181587+0000", 2024-03-20T09:46:43.415 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.1509999999999998e-06 2024-03-20T09:46:43.415 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.415 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.415 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:43.415 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181593+0000", 2024-03-20T09:46:43.415 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.9649999999999996e-06 2024-03-20T09:46:43.415 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.415 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.415 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:43.415 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181598+0000", 2024-03-20T09:46:43.415 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2220000000000001e-06 2024-03-20T09:46:43.415 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.415 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.416 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.416 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181646+0000", 2024-03-20T09:46:43.416 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.8068000000000001e-05 2024-03-20T09:46:43.416 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.416 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.416 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:46:43.416 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181651+0000", 2024-03-20T09:46:43.416 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.4029999999999998e-06 2024-03-20T09:46:43.416 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.416 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.416 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.416 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.965148+0000", 2024-03-20T09:46:43.416 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.78349648299999997 2024-03-20T09:46:43.416 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.417 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.417 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:43.417 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.965244+0000", 2024-03-20T09:46:43.417 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.6477999999999993e-05 2024-03-20T09:46:43.417 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.417 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.417 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 4,9", 2024-03-20T09:46:43.417 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.965366+0000", 2024-03-20T09:46:43.417 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00012211599999999999 2024-03-20T09:46:43.417 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.417 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.417 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:43.417 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.969245+0000", 2024-03-20T09:46:43.417 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0038788149999999999 2024-03-20T09:46:43.417 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.418 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.418 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:43.418 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.973293+0000", 2024-03-20T09:46:43.418 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040475320000000004 2024-03-20T09:46:43.418 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.418 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.418 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:43.418 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.977892+0000", 2024-03-20T09:46:43.418 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0045994190000000004 2024-03-20T09:46:43.418 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.418 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.418 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:43.418 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.977925+0000", 2024-03-20T09:46:43.418 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2534999999999999e-05 2024-03-20T09:46:43.419 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.419 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.419 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:43.419 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.977968+0000", 2024-03-20T09:46:43.419 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.333e-05 2024-03-20T09:46:43.419 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.419 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:43.419 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.419 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.419 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.419 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:222576 8.2 8:49ae1e54:::benchmark_data_smithi122_197775_object99272:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1050)", 2024-03-20T09:46:43.419 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:39:33.181613+0000", 2024-03-20T09:46:43.419 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 430.15675590799998, 2024-03-20T09:46:43.419 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.80482790500000001, 2024-03-20T09:46:43.420 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:43.420 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:43.420 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:43.420 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:43.420 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:43.420 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 222576 2024-03-20T09:46:43.420 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.420 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:43.420 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.420 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:43.420 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181613+0000", 2024-03-20T09:46:43.420 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.420 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.420 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.421 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:43.421 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181613+0000", 2024-03-20T09:46:43.421 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.421 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.421 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.421 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:43.421 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181614+0000", 2024-03-20T09:46:43.421 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.3299999999999998e-07 2024-03-20T09:46:43.421 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.421 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.421 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:43.421 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181616+0000", 2024-03-20T09:46:43.421 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.514e-06 2024-03-20T09:46:43.421 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.421 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.422 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:43.422 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181623+0000", 2024-03-20T09:46:43.422 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.9539999999999998e-06 2024-03-20T09:46:43.422 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.422 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.422 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:43.422 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181629+0000", 2024-03-20T09:46:43.422 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0100000000000001e-06 2024-03-20T09:46:43.422 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.422 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.422 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.422 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181674+0000", 2024-03-20T09:46:43.422 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.4916999999999997e-05 2024-03-20T09:46:43.423 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.423 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.423 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:46:43.423 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181679+0000", 2024-03-20T09:46:43.423 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3340000000000001e-06 2024-03-20T09:46:43.423 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.423 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.423 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.423 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.969298+0000", 2024-03-20T09:46:43.423 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.78761846999999996 2024-03-20T09:46:43.423 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.423 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.423 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:43.425 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.969377+0000", 2024-03-20T09:46:43.425 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 7.8738000000000004e-05 2024-03-20T09:46:43.425 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.425 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.425 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 4,9", 2024-03-20T09:46:43.425 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.969445+0000", 2024-03-20T09:46:43.425 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.8245999999999997e-05 2024-03-20T09:46:43.426 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.426 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.426 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:43.426 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.977855+0000", 2024-03-20T09:46:43.426 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0084097939999999999 2024-03-20T09:46:43.426 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.426 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.426 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:43.426 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.978019+0000", 2024-03-20T09:46:43.426 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00016461199999999999 2024-03-20T09:46:43.426 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.426 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.426 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:43.426 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.986331+0000", 2024-03-20T09:46:43.427 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0083114450000000006 2024-03-20T09:46:43.427 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.427 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.427 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:43.427 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.986391+0000", 2024-03-20T09:46:43.427 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.0139000000000001e-05 2024-03-20T09:46:43.427 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.427 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.427 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:43.427 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.986441+0000", 2024-03-20T09:46:43.427 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.0399000000000003e-05 2024-03-20T09:46:43.427 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.427 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:43.427 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.427 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.428 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.428 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:222588 8.2 8:4650e4c5:::benchmark_data_smithi122_197775_object99284:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1050)", 2024-03-20T09:46:43.428 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:39:33.181642+0000", 2024-03-20T09:46:43.428 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 430.15672684499998, 2024-03-20T09:46:43.428 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.80494451700000003, 2024-03-20T09:46:43.428 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:43.428 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:43.428 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:43.428 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:43.428 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:43.428 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 222588 2024-03-20T09:46:43.428 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.428 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:43.428 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.429 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:43.429 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181642+0000", 2024-03-20T09:46:43.429 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.429 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.429 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.429 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:43.429 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181642+0000", 2024-03-20T09:46:43.429 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.429 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.429 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.429 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:43.429 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181643+0000", 2024-03-20T09:46:43.429 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.5600000000000001e-07 2024-03-20T09:46:43.429 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.429 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.430 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:43.430 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181645+0000", 2024-03-20T09:46:43.430 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.587e-06 2024-03-20T09:46:43.430 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.430 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.430 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:43.430 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181651+0000", 2024-03-20T09:46:43.430 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.207e-06 2024-03-20T09:46:43.430 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.430 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.430 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:43.430 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181656+0000", 2024-03-20T09:46:43.430 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.904e-06 2024-03-20T09:46:43.430 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.430 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.431 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.431 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181700+0000", 2024-03-20T09:46:43.431 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 4.3721999999999999e-05 2024-03-20T09:46:43.431 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.431 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.431 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:46:43.431 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181705+0000", 2024-03-20T09:46:43.431 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.3059999999999997e-06 2024-03-20T09:46:43.431 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.431 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.431 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.431 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.973341+0000", 2024-03-20T09:46:43.431 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.79163517699999997 2024-03-20T09:46:43.431 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.431 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.432 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:43.432 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.973782+0000", 2024-03-20T09:46:43.432 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00044108999999999999 2024-03-20T09:46:43.432 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.432 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.432 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 4,9", 2024-03-20T09:46:43.432 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.973913+0000", 2024-03-20T09:46:43.432 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00013104699999999999 2024-03-20T09:46:43.432 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.432 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.432 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:43.432 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.978007+0000", 2024-03-20T09:46:43.432 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0040942039999999997 2024-03-20T09:46:43.432 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.432 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.433 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:43.433 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.986206+0000", 2024-03-20T09:46:43.433 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0081995439999999996 2024-03-20T09:46:43.433 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.433 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.433 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:43.433 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.986494+0000", 2024-03-20T09:46:43.433 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00028800800000000002 2024-03-20T09:46:43.433 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.433 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.433 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:43.433 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.986550+0000", 2024-03-20T09:46:43.433 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.5247000000000001e-05 2024-03-20T09:46:43.433 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.433 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.434 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:43.434 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.986587+0000", 2024-03-20T09:46:43.434 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7117999999999999e-05 2024-03-20T09:46:43.434 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.434 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:43.434 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.434 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.434 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.434 INFO:teuthology.orchestra.run.smithi122.stdout: "description": "osd_op(client.26917.0:222596 8.2 8:45364881:::benchmark_data_smithi122_197775_object99292:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1050)", 2024-03-20T09:46:43.434 INFO:teuthology.orchestra.run.smithi122.stdout: "initiated_at": "2024-03-20T09:39:33.181673+0000", 2024-03-20T09:46:43.434 INFO:teuthology.orchestra.run.smithi122.stdout: "age": 430.15669655800002, 2024-03-20T09:46:43.434 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.81395974599999998, 2024-03-20T09:46:43.434 INFO:teuthology.orchestra.run.smithi122.stdout: "type_data": { 2024-03-20T09:46:43.434 INFO:teuthology.orchestra.run.smithi122.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-20T09:46:43.435 INFO:teuthology.orchestra.run.smithi122.stdout: "client_info": { 2024-03-20T09:46:43.435 INFO:teuthology.orchestra.run.smithi122.stdout: "client": "client.26917", 2024-03-20T09:46:43.435 INFO:teuthology.orchestra.run.smithi122.stdout: "client_addr": "172.21.15.122:0/1095574859", 2024-03-20T09:46:43.435 INFO:teuthology.orchestra.run.smithi122.stdout: "tid": 222596 2024-03-20T09:46:43.435 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.435 INFO:teuthology.orchestra.run.smithi122.stdout: "events": [ 2024-03-20T09:46:43.435 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.435 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "initiated", 2024-03-20T09:46:43.435 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181673+0000", 2024-03-20T09:46:43.435 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.435 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.435 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.435 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "header_read", 2024-03-20T09:46:43.435 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181673+0000", 2024-03-20T09:46:43.435 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0 2024-03-20T09:46:43.436 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.436 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.436 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "throttled", 2024-03-20T09:46:43.436 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181673+0000", 2024-03-20T09:46:43.436 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.58e-07 2024-03-20T09:46:43.436 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.436 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.436 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "all_read", 2024-03-20T09:46:43.436 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181675+0000", 2024-03-20T09:46:43.436 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 2.3709999999999998e-06 2024-03-20T09:46:43.436 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.436 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.436 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "dispatched", 2024-03-20T09:46:43.436 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181682+0000", 2024-03-20T09:46:43.436 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 6.922e-06 2024-03-20T09:46:43.436 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.437 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.437 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "queued_for_pg", 2024-03-20T09:46:43.437 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181687+0000", 2024-03-20T09:46:43.437 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.1749999999999996e-06 2024-03-20T09:46:43.437 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.437 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.437 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.437 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181725+0000", 2024-03-20T09:46:43.437 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.7768000000000002e-05 2024-03-20T09:46:43.437 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.437 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.437 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for peered", 2024-03-20T09:46:43.437 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.181730+0000", 2024-03-20T09:46:43.438 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 5.2650000000000004e-06 2024-03-20T09:46:43.438 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.438 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.438 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "reached_pg", 2024-03-20T09:46:43.438 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.978051+0000", 2024-03-20T09:46:43.438 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.79632081700000001 2024-03-20T09:46:43.438 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.438 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.438 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "started", 2024-03-20T09:46:43.438 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.978150+0000", 2024-03-20T09:46:43.438 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 9.8770999999999993e-05 2024-03-20T09:46:43.438 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.438 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.438 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "waiting for subops from 4,9", 2024-03-20T09:46:43.439 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.978254+0000", 2024-03-20T09:46:43.439 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.000103813 2024-03-20T09:46:43.439 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.439 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.439 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:43.439 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.986263+0000", 2024-03-20T09:46:43.439 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.0080094789999999999 2024-03-20T09:46:43.439 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.439 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.439 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "op_commit", 2024-03-20T09:46:43.439 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.986456+0000", 2024-03-20T09:46:43.439 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.00019298300000000001 2024-03-20T09:46:43.439 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.439 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.440 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "sub_op_commit_rec", 2024-03-20T09:46:43.440 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.995569+0000", 2024-03-20T09:46:43.440 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 0.009112551 2024-03-20T09:46:43.440 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.440 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.440 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "commit_sent", 2024-03-20T09:46:43.440 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.995600+0000", 2024-03-20T09:46:43.440 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.1381000000000003e-05 2024-03-20T09:46:43.440 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-03-20T09:46:43.440 INFO:teuthology.orchestra.run.smithi122.stdout: { 2024-03-20T09:46:43.440 INFO:teuthology.orchestra.run.smithi122.stdout: "event": "done", 2024-03-20T09:46:43.440 INFO:teuthology.orchestra.run.smithi122.stdout: "time": "2024-03-20T09:39:33.995632+0000", 2024-03-20T09:46:43.440 INFO:teuthology.orchestra.run.smithi122.stdout: "duration": 3.2091999999999997e-05 2024-03-20T09:46:43.440 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.440 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:43.441 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.441 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-03-20T09:46:43.441 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-03-20T09:46:43.441 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-03-20T09:46:43.441 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:42 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2261485212' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-03-20T09:46:43.441 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:42 smithi115 ceph-mon[72476]: osdmap e1194: 12 total, 12 up, 12 in 2024-03-20T09:46:43.958 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:43 smithi122 ceph-mon[73495]: pgmap v2897: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:44.127 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:43 smithi026 ceph-mon[68309]: pgmap v2897: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:44.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:43 smithi115 ceph-mon[72476]: pgmap v2897: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:44.921 INFO:teuthology.orchestra.run.smithi026.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-03-20T09:46:45.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:44 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1206410607' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-03-20T09:46:45.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:44 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1206410607' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-03-20T09:46:45.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:44 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1206410607' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]: dispatch 2024-03-20T09:46:45.745 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-03-20T09:46:45.745 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-03-20T09:46:45.745 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-03-20T09:46:45.745 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-03-20T09:46:45.745 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-20T09:46:45.745 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-20T09:46:46.023 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:46:46.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:45 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1206410607' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-03-20T09:46:46.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:45 smithi122 ceph-mon[73495]: osdmap e1195: 12 total, 12 up, 12 in 2024-03-20T09:46:46.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:45 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:46:46.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:45 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:46:46.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:45 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:46:46.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:45 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:46:46.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:45 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:46:46.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:45 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:46:46.184 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:45 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:46:46.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:45 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1206410607' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-03-20T09:46:46.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:45 smithi026 ceph-mon[68309]: osdmap e1195: 12 total, 12 up, 12 in 2024-03-20T09:46:46.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:45 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:46:46.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:45 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:46:46.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:45 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:46:46.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:45 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:46:46.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:45 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:46:46.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:45 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:46:46.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:45 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:46:46.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:45 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1206410607' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-03-20T09:46:46.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:45 smithi115 ceph-mon[72476]: osdmap e1195: 12 total, 12 up, 12 in 2024-03-20T09:46:46.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:45 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-20T09:46:46.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:45 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-20T09:46:46.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:45 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-20T09:46:46.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:45 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:46:46.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:45 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:46:46.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:45 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:46:46.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:45 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:46:47.175 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:46 smithi026 ceph-mon[68309]: 1.a scrub starts 2024-03-20T09:46:47.175 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:46 smithi026 ceph-mon[68309]: 1.a scrub ok 2024-03-20T09:46:47.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:46 smithi122 ceph-mon[73495]: 1.a scrub starts 2024-03-20T09:46:47.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:46 smithi122 ceph-mon[73495]: 1.a scrub ok 2024-03-20T09:46:47.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:46 smithi115 ceph-mon[72476]: 1.a scrub starts 2024-03-20T09:46:47.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:46 smithi115 ceph-mon[72476]: 1.a scrub ok 2024-03-20T09:46:47.969 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:46:47.970 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":1195,"fsid":"852fbb68-e696-11ee-95c9-87774f69a715","created":"2024-03-20T08:48:35.281088+0000","modified":"2024-03-20T09:46:44.929597+0000","last_up_change":"2024-03-20T09:30:44.068387+0000","last_in_change":"2024-03-20T09:44:52.814027+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-20T08:52:17.442324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"360","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"358","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.940000057220459,"score_stable":1.940000057220459,"optimal_score":1,"raw_score_acting":1.940000057220459,"raw_score_stable":1.940000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad3fd9c0-316f-4fe5-a63b-d480087c1729","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":1149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":2139805293},{"type":"v1","addr":"172.21.15.26:6803","nonce":2139805293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":2139805293},{"type":"v1","addr":"172.21.15.26:6805","nonce":2139805293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":2139805293},{"type":"v1","addr":"172.21.15.26:6809","nonce":2139805293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":2139805293},{"type":"v1","addr":"172.21.15.26:6807","nonce":2139805293}]},"public_addr":"172.21.15.26:6803/2139805293","cluster_addr":"172.21.15.26:6805/2139805293","heartbeat_back_addr":"172.21.15.26:6809/2139805293","heartbeat_front_addr":"172.21.15.26:6807/2139805293","state":["exists","up"]},{"osd":1,"uuid":"e85b94f5-2954-47a1-9c20-096f2606eb04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":1164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":3853355134},{"type":"v1","addr":"172.21.15.26:6811","nonce":3853355134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":3853355134},{"type":"v1","addr":"172.21.15.26:6813","nonce":3853355134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":3853355134},{"type":"v1","addr":"172.21.15.26:6817","nonce":3853355134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":3853355134},{"type":"v1","addr":"172.21.15.26:6815","nonce":3853355134}]},"public_addr":"172.21.15.26:6811/3853355134","cluster_addr":"172.21.15.26:6813/3853355134","heartbeat_back_addr":"172.21.15.26:6817/3853355134","heartbeat_front_addr":"172.21.15.26:6815/3853355134","state":["exists","up"]},{"osd":2,"uuid":"dd2b10f0-741c-47a4-b224-ac6bfe2c14ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":1054,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":3993357967},{"type":"v1","addr":"172.21.15.26:6819","nonce":3993357967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":3993357967},{"type":"v1","addr":"172.21.15.26:6821","nonce":3993357967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":3993357967},{"type":"v1","addr":"172.21.15.26:6825","nonce":3993357967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":3993357967},{"type":"v1","addr":"172.21.15.26:6823","nonce":3993357967}]},"public_addr":"172.21.15.26:6819/3993357967","cluster_addr":"172.21.15.26:6821/3993357967","heartbeat_back_addr":"172.21.15.26:6825/3993357967","heartbeat_front_addr":"172.21.15.26:6823/3993357967","state":["exists","up"]},{"osd":3,"uuid":"9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":1149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":4110225509},{"type":"v1","addr":"172.21.15.26:6827","nonce":4110225509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":4110225509},{"type":"v1","addr":"172.21.15.26:6829","nonce":4110225509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":4110225509},{"type":"v1","addr":"172.21.15.26:6833","nonce":4110225509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":4110225509},{"type":"v1","addr":"172.21.15.26:6831","nonce":4110225509}]},"public_addr":"172.21.15.26:6827/4110225509","cluster_addr":"172.21.15.26:6829/4110225509","heartbeat_back_addr":"172.21.15.26:6833/4110225509","heartbeat_front_addr":"172.21.15.26:6831/4110225509","state":["exists","up"]},{"osd":4,"uuid":"f14e34d5-9829-42a3-9b25-31a0976c5ef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":1148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":477923058},{"type":"v1","addr":"172.21.15.115:6801","nonce":477923058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":477923058},{"type":"v1","addr":"172.21.15.115:6803","nonce":477923058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":477923058},{"type":"v1","addr":"172.21.15.115:6807","nonce":477923058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":477923058},{"type":"v1","addr":"172.21.15.115:6805","nonce":477923058}]},"public_addr":"172.21.15.115:6801/477923058","cluster_addr":"172.21.15.115:6803/477923058","heartbeat_back_addr":"172.21.15.115:6807/477923058","heartbeat_front_addr":"172.21.15.115:6805/477923058","state":["exists","up"]},{"osd":5,"uuid":"cb431b6a-540d-4318-ae35-d379e1df0185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":41,"last_clean_end":273,"up_from":443,"up_thru":1192,"down_at":274,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":650416648},{"type":"v1","addr":"172.21.15.115:6809","nonce":650416648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":650416648},{"type":"v1","addr":"172.21.15.115:6811","nonce":650416648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":650416648},{"type":"v1","addr":"172.21.15.115:6815","nonce":650416648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":650416648},{"type":"v1","addr":"172.21.15.115:6813","nonce":650416648}]},"public_addr":"172.21.15.115:6809/650416648","cluster_addr":"172.21.15.115:6811/650416648","heartbeat_back_addr":"172.21.15.115:6815/650416648","heartbeat_front_addr":"172.21.15.115:6813/650416648","state":["exists","up"]},{"osd":6,"uuid":"753ed830-e764-4ff3-9d9b-87e7fa748b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":1066,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3278301843},{"type":"v1","addr":"172.21.15.115:6817","nonce":3278301843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":3278301843},{"type":"v1","addr":"172.21.15.115:6819","nonce":3278301843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":3278301843},{"type":"v1","addr":"172.21.15.115:6823","nonce":3278301843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":3278301843},{"type":"v1","addr":"172.21.15.115:6821","nonce":3278301843}]},"public_addr":"172.21.15.115:6817/3278301843","cluster_addr":"172.21.15.115:6819/3278301843","heartbeat_back_addr":"172.21.15.115:6823/3278301843","heartbeat_front_addr":"172.21.15.115:6821/3278301843","state":["exists","up"]},{"osd":7,"uuid":"55bcbd37-7a6f-4ab1-b5a9-91a951444d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":1179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":3152424001},{"type":"v1","addr":"172.21.15.115:6825","nonce":3152424001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3152424001},{"type":"v1","addr":"172.21.15.115:6827","nonce":3152424001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3152424001},{"type":"v1","addr":"172.21.15.115:6831","nonce":3152424001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3152424001},{"type":"v1","addr":"172.21.15.115:6829","nonce":3152424001}]},"public_addr":"172.21.15.115:6825/3152424001","cluster_addr":"172.21.15.115:6827/3152424001","heartbeat_back_addr":"172.21.15.115:6831/3152424001","heartbeat_front_addr":"172.21.15.115:6829/3152424001","state":["exists","up"]},{"osd":8,"uuid":"bc86248e-6bd0-4369-b8a2-e593bc2a610d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":1149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1309426740},{"type":"v1","addr":"172.21.15.122:6801","nonce":1309426740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1309426740},{"type":"v1","addr":"172.21.15.122:6803","nonce":1309426740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1309426740},{"type":"v1","addr":"172.21.15.122:6807","nonce":1309426740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1309426740},{"type":"v1","addr":"172.21.15.122:6805","nonce":1309426740}]},"public_addr":"172.21.15.122:6801/1309426740","cluster_addr":"172.21.15.122:6803/1309426740","heartbeat_back_addr":"172.21.15.122:6807/1309426740","heartbeat_front_addr":"172.21.15.122:6805/1309426740","state":["exists","up"]},{"osd":9,"uuid":"9b20a4aa-032a-4c09-ac75-153c554424e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":1066,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":2953210076},{"type":"v1","addr":"172.21.15.122:6809","nonce":2953210076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":2953210076},{"type":"v1","addr":"172.21.15.122:6811","nonce":2953210076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2953210076},{"type":"v1","addr":"172.21.15.122:6815","nonce":2953210076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2953210076},{"type":"v1","addr":"172.21.15.122:6813","nonce":2953210076}]},"public_addr":"172.21.15.122:6809/2953210076","cluster_addr":"172.21.15.122:6811/2953210076","heartbeat_back_addr":"172.21.15.122:6815/2953210076","heartbeat_front_addr":"172.21.15.122:6813/2953210076","state":["exists","up"]},{"osd":10,"uuid":"e62c1fb3-5dd1-4d0d-997c-d20633d19361","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":89,"last_clean_end":753,"up_from":861,"up_thru":1149,"down_at":754,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":2853088860},{"type":"v1","addr":"172.21.15.122:6817","nonce":2853088860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":2853088860},{"type":"v1","addr":"172.21.15.122:6819","nonce":2853088860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":2853088860},{"type":"v1","addr":"172.21.15.122:6823","nonce":2853088860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":2853088860},{"type":"v1","addr":"172.21.15.122:6821","nonce":2853088860}]},"public_addr":"172.21.15.122:6817/2853088860","cluster_addr":"172.21.15.122:6819/2853088860","heartbeat_back_addr":"172.21.15.122:6823/2853088860","heartbeat_front_addr":"172.21.15.122:6821/2853088860","state":["exists","up"]},{"osd":11,"uuid":"b83994d5-e5ef-48b0-9758-c26843cec3b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":96,"up_thru":1149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":2704069354},{"type":"v1","addr":"172.21.15.122:6825","nonce":2704069354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":2704069354},{"type":"v1","addr":"172.21.15.122:6827","nonce":2704069354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":2704069354},{"type":"v1","addr":"172.21.15.122:6831","nonce":2704069354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":2704069354},{"type":"v1","addr":"172.21.15.122:6829","nonce":2704069354}]},"public_addr":"172.21.15.122:6825/2704069354","cluster_addr":"172.21.15.122:6827/2704069354","heartbeat_back_addr":"172.21.15.122:6831/2704069354","heartbeat_front_addr":"172.21.15.122:6829/2704069354","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T09:46:32.345767+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T09:45:36.805567+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T09:46:08.354977+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T09:46:38.413884+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T09:46:22.859438+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T09:46:18.861167+0000","dead_epoch":273},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T09:46:24.484800+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T09:46:34.716681+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T09:45:58.302535+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T09:45:51.715212+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T09:46:31.623107+0000","dead_epoch":753},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T09:45:53.950336+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.1","osds":[6,11,5]},{"pgid":"1.1a","osds":[3,7,4]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:0/4291404191":"2024-03-21T08:49:35.336921+0000","172.21.15.26:0/642012566":"2024-03-21T08:49:35.336921+0000","172.21.15.26:6801/1181459999":"2024-03-21T08:49:35.336921+0000","172.21.15.26:0/3984513952":"2024-03-21T08:49:04.596754+0000","172.21.15.26:6800/1654945498":"2024-03-21T08:49:04.596754+0000","172.21.15.26:6800/1181459999":"2024-03-21T08:49:35.336921+0000","172.21.15.26:0/137770981":"2024-03-21T08:49:04.596754+0000","172.21.15.26:0/1451237762":"2024-03-21T08:49:35.336921+0000","172.21.15.26:0/3938386631":"2024-03-21T08:49:04.596754+0000","172.21.15.26:6801/1654945498":"2024-03-21T08:49:04.596754+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-20T09:46:48.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:47 smithi122 ceph-mon[73495]: pgmap v2900: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:48.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:47 smithi026 ceph-mon[68309]: pgmap v2900: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:48.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:47 smithi115 ceph-mon[72476]: pgmap v2900: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:48.713 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-20T09:46:48.713 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-20T09:46:48.956 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:48 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/891245187' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-20T09:46:48.992 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:46:49.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:48 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/891245187' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-20T09:46:49.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:48 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/891245187' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-20T09:46:50.039 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:49 smithi026 ceph-mon[68309]: pgmap v2901: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:50.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:49 smithi115 ceph-mon[72476]: pgmap v2901: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:50.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:49 smithi122 ceph-mon[73495]: pgmap v2901: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:50.778 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:46:50.778 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":1195,"fsid":"852fbb68-e696-11ee-95c9-87774f69a715","created":"2024-03-20T08:48:35.281088+0000","modified":"2024-03-20T09:46:44.929597+0000","last_up_change":"2024-03-20T09:30:44.068387+0000","last_in_change":"2024-03-20T09:44:52.814027+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-20T08:52:17.442324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"360","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"358","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.940000057220459,"score_stable":1.940000057220459,"optimal_score":1,"raw_score_acting":1.940000057220459,"raw_score_stable":1.940000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad3fd9c0-316f-4fe5-a63b-d480087c1729","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":1149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":2139805293},{"type":"v1","addr":"172.21.15.26:6803","nonce":2139805293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":2139805293},{"type":"v1","addr":"172.21.15.26:6805","nonce":2139805293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":2139805293},{"type":"v1","addr":"172.21.15.26:6809","nonce":2139805293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":2139805293},{"type":"v1","addr":"172.21.15.26:6807","nonce":2139805293}]},"public_addr":"172.21.15.26:6803/2139805293","cluster_addr":"172.21.15.26:6805/2139805293","heartbeat_back_addr":"172.21.15.26:6809/2139805293","heartbeat_front_addr":"172.21.15.26:6807/2139805293","state":["exists","up"]},{"osd":1,"uuid":"e85b94f5-2954-47a1-9c20-096f2606eb04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":1164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":3853355134},{"type":"v1","addr":"172.21.15.26:6811","nonce":3853355134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":3853355134},{"type":"v1","addr":"172.21.15.26:6813","nonce":3853355134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":3853355134},{"type":"v1","addr":"172.21.15.26:6817","nonce":3853355134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":3853355134},{"type":"v1","addr":"172.21.15.26:6815","nonce":3853355134}]},"public_addr":"172.21.15.26:6811/3853355134","cluster_addr":"172.21.15.26:6813/3853355134","heartbeat_back_addr":"172.21.15.26:6817/3853355134","heartbeat_front_addr":"172.21.15.26:6815/3853355134","state":["exists","up"]},{"osd":2,"uuid":"dd2b10f0-741c-47a4-b224-ac6bfe2c14ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":1054,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":3993357967},{"type":"v1","addr":"172.21.15.26:6819","nonce":3993357967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":3993357967},{"type":"v1","addr":"172.21.15.26:6821","nonce":3993357967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":3993357967},{"type":"v1","addr":"172.21.15.26:6825","nonce":3993357967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":3993357967},{"type":"v1","addr":"172.21.15.26:6823","nonce":3993357967}]},"public_addr":"172.21.15.26:6819/3993357967","cluster_addr":"172.21.15.26:6821/3993357967","heartbeat_back_addr":"172.21.15.26:6825/3993357967","heartbeat_front_addr":"172.21.15.26:6823/3993357967","state":["exists","up"]},{"osd":3,"uuid":"9b5bc0aa-08ef-43bb-aa2d-7e48f8da82f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":1149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":4110225509},{"type":"v1","addr":"172.21.15.26:6827","nonce":4110225509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":4110225509},{"type":"v1","addr":"172.21.15.26:6829","nonce":4110225509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":4110225509},{"type":"v1","addr":"172.21.15.26:6833","nonce":4110225509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":4110225509},{"type":"v1","addr":"172.21.15.26:6831","nonce":4110225509}]},"public_addr":"172.21.15.26:6827/4110225509","cluster_addr":"172.21.15.26:6829/4110225509","heartbeat_back_addr":"172.21.15.26:6833/4110225509","heartbeat_front_addr":"172.21.15.26:6831/4110225509","state":["exists","up"]},{"osd":4,"uuid":"f14e34d5-9829-42a3-9b25-31a0976c5ef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":1148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":477923058},{"type":"v1","addr":"172.21.15.115:6801","nonce":477923058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":477923058},{"type":"v1","addr":"172.21.15.115:6803","nonce":477923058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":477923058},{"type":"v1","addr":"172.21.15.115:6807","nonce":477923058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":477923058},{"type":"v1","addr":"172.21.15.115:6805","nonce":477923058}]},"public_addr":"172.21.15.115:6801/477923058","cluster_addr":"172.21.15.115:6803/477923058","heartbeat_back_addr":"172.21.15.115:6807/477923058","heartbeat_front_addr":"172.21.15.115:6805/477923058","state":["exists","up"]},{"osd":5,"uuid":"cb431b6a-540d-4318-ae35-d379e1df0185","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":41,"last_clean_end":273,"up_from":443,"up_thru":1192,"down_at":274,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":650416648},{"type":"v1","addr":"172.21.15.115:6809","nonce":650416648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":650416648},{"type":"v1","addr":"172.21.15.115:6811","nonce":650416648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":650416648},{"type":"v1","addr":"172.21.15.115:6815","nonce":650416648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":650416648},{"type":"v1","addr":"172.21.15.115:6813","nonce":650416648}]},"public_addr":"172.21.15.115:6809/650416648","cluster_addr":"172.21.15.115:6811/650416648","heartbeat_back_addr":"172.21.15.115:6815/650416648","heartbeat_front_addr":"172.21.15.115:6813/650416648","state":["exists","up"]},{"osd":6,"uuid":"753ed830-e764-4ff3-9d9b-87e7fa748b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":1066,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":3278301843},{"type":"v1","addr":"172.21.15.115:6817","nonce":3278301843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":3278301843},{"type":"v1","addr":"172.21.15.115:6819","nonce":3278301843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":3278301843},{"type":"v1","addr":"172.21.15.115:6823","nonce":3278301843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":3278301843},{"type":"v1","addr":"172.21.15.115:6821","nonce":3278301843}]},"public_addr":"172.21.15.115:6817/3278301843","cluster_addr":"172.21.15.115:6819/3278301843","heartbeat_back_addr":"172.21.15.115:6823/3278301843","heartbeat_front_addr":"172.21.15.115:6821/3278301843","state":["exists","up"]},{"osd":7,"uuid":"55bcbd37-7a6f-4ab1-b5a9-91a951444d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":1179,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":3152424001},{"type":"v1","addr":"172.21.15.115:6825","nonce":3152424001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":3152424001},{"type":"v1","addr":"172.21.15.115:6827","nonce":3152424001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":3152424001},{"type":"v1","addr":"172.21.15.115:6831","nonce":3152424001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":3152424001},{"type":"v1","addr":"172.21.15.115:6829","nonce":3152424001}]},"public_addr":"172.21.15.115:6825/3152424001","cluster_addr":"172.21.15.115:6827/3152424001","heartbeat_back_addr":"172.21.15.115:6831/3152424001","heartbeat_front_addr":"172.21.15.115:6829/3152424001","state":["exists","up"]},{"osd":8,"uuid":"bc86248e-6bd0-4369-b8a2-e593bc2a610d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":1149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":1309426740},{"type":"v1","addr":"172.21.15.122:6801","nonce":1309426740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":1309426740},{"type":"v1","addr":"172.21.15.122:6803","nonce":1309426740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":1309426740},{"type":"v1","addr":"172.21.15.122:6807","nonce":1309426740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":1309426740},{"type":"v1","addr":"172.21.15.122:6805","nonce":1309426740}]},"public_addr":"172.21.15.122:6801/1309426740","cluster_addr":"172.21.15.122:6803/1309426740","heartbeat_back_addr":"172.21.15.122:6807/1309426740","heartbeat_front_addr":"172.21.15.122:6805/1309426740","state":["exists","up"]},{"osd":9,"uuid":"9b20a4aa-032a-4c09-ac75-153c554424e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":1066,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":2953210076},{"type":"v1","addr":"172.21.15.122:6809","nonce":2953210076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":2953210076},{"type":"v1","addr":"172.21.15.122:6811","nonce":2953210076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2953210076},{"type":"v1","addr":"172.21.15.122:6815","nonce":2953210076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2953210076},{"type":"v1","addr":"172.21.15.122:6813","nonce":2953210076}]},"public_addr":"172.21.15.122:6809/2953210076","cluster_addr":"172.21.15.122:6811/2953210076","heartbeat_back_addr":"172.21.15.122:6815/2953210076","heartbeat_front_addr":"172.21.15.122:6813/2953210076","state":["exists","up"]},{"osd":10,"uuid":"e62c1fb3-5dd1-4d0d-997c-d20633d19361","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":89,"last_clean_end":753,"up_from":861,"up_thru":1149,"down_at":754,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":2853088860},{"type":"v1","addr":"172.21.15.122:6817","nonce":2853088860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":2853088860},{"type":"v1","addr":"172.21.15.122:6819","nonce":2853088860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":2853088860},{"type":"v1","addr":"172.21.15.122:6823","nonce":2853088860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":2853088860},{"type":"v1","addr":"172.21.15.122:6821","nonce":2853088860}]},"public_addr":"172.21.15.122:6817/2853088860","cluster_addr":"172.21.15.122:6819/2853088860","heartbeat_back_addr":"172.21.15.122:6823/2853088860","heartbeat_front_addr":"172.21.15.122:6821/2853088860","state":["exists","up"]},{"osd":11,"uuid":"b83994d5-e5ef-48b0-9758-c26843cec3b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":96,"up_thru":1149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":2704069354},{"type":"v1","addr":"172.21.15.122:6825","nonce":2704069354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":2704069354},{"type":"v1","addr":"172.21.15.122:6827","nonce":2704069354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":2704069354},{"type":"v1","addr":"172.21.15.122:6831","nonce":2704069354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":2704069354},{"type":"v1","addr":"172.21.15.122:6829","nonce":2704069354}]},"public_addr":"172.21.15.122:6825/2704069354","cluster_addr":"172.21.15.122:6827/2704069354","heartbeat_back_addr":"172.21.15.122:6831/2704069354","heartbeat_front_addr":"172.21.15.122:6829/2704069354","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T09:46:32.345767+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T09:45:36.805567+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T09:46:08.354977+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T09:46:38.413884+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T09:46:22.859438+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T09:46:18.861167+0000","dead_epoch":273},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T09:46:24.484800+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T09:46:34.716681+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T09:45:58.302535+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T09:45:51.715212+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T09:46:31.623107+0000","dead_epoch":753},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-20T09:45:53.950336+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.1","osds":[6,11,5]},{"pgid":"1.1a","osds":[3,7,4]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:0/4291404191":"2024-03-21T08:49:35.336921+0000","172.21.15.26:0/642012566":"2024-03-21T08:49:35.336921+0000","172.21.15.26:6801/1181459999":"2024-03-21T08:49:35.336921+0000","172.21.15.26:0/3984513952":"2024-03-21T08:49:04.596754+0000","172.21.15.26:6800/1654945498":"2024-03-21T08:49:04.596754+0000","172.21.15.26:6800/1181459999":"2024-03-21T08:49:35.336921+0000","172.21.15.26:0/137770981":"2024-03-21T08:49:04.596754+0000","172.21.15.26:0/1451237762":"2024-03-21T08:49:35.336921+0000","172.21.15.26:0/3938386631":"2024-03-21T08:49:04.596754+0000","172.21.15.26:6801/1654945498":"2024-03-21T08:49:04.596754+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-20T09:46:51.148 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:50 smithi026 ceph-mon[68309]: 1.14 scrub starts 2024-03-20T09:46:51.148 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:50 smithi026 ceph-mon[68309]: 1.14 scrub ok 2024-03-20T09:46:51.148 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:50 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/190927828' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-20T09:46:51.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:50 smithi115 ceph-mon[72476]: 1.14 scrub starts 2024-03-20T09:46:51.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:50 smithi115 ceph-mon[72476]: 1.14 scrub ok 2024-03-20T09:46:51.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:50 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/190927828' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-20T09:46:51.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:50 smithi122 ceph-mon[73495]: 1.14 scrub starts 2024-03-20T09:46:51.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:50 smithi122 ceph-mon[73495]: 1.14 scrub ok 2024-03-20T09:46:51.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:50 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/190927828' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-20T09:46:51.762 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-20T09:46:51.763 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-20T09:46:51.763 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-20T09:46:51.764 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-20T09:46:51.764 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-20T09:46:51.765 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-20T09:46:51.765 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-20T09:46:51.766 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-20T09:46:51.766 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-20T09:46:51.767 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-20T09:46:51.767 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-20T09:46:51.768 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-20T09:46:51.993 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:51 smithi026 ceph-mon[68309]: pgmap v2902: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:52.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:51 smithi115 ceph-mon[72476]: pgmap v2902: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:52.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:51 smithi122 ceph-mon[73495]: pgmap v2902: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:52.541 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:46:52.861 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:46:52.866 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:46:52.869 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:46:52.869 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:46:52.874 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:46:52.874 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:46:52.880 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:46:52.897 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:46:52.897 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:46:52.897 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:46:52.902 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:46:54.249 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:53 smithi026 ceph-mon[68309]: pgmap v2903: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:54.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:53 smithi115 ceph-mon[72476]: pgmap v2903: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:54.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:53 smithi122 ceph-mon[73495]: pgmap v2903: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:56.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:55 smithi026 ceph-mon[68309]: pgmap v2904: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:56.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:55 smithi115 ceph-mon[72476]: pgmap v2904: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:56.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:55 smithi122 ceph-mon[73495]: pgmap v2904: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:57.149 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:56 smithi026 ceph-mon[68309]: 1.1c scrub starts 2024-03-20T09:46:57.150 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:56 smithi026 ceph-mon[68309]: 1.1c scrub ok 2024-03-20T09:46:57.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:56 smithi115 ceph-mon[72476]: 1.1c scrub starts 2024-03-20T09:46:57.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:56 smithi115 ceph-mon[72476]: 1.1c scrub ok 2024-03-20T09:46:57.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:56 smithi122 ceph-mon[73495]: 1.1c scrub starts 2024-03-20T09:46:57.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:56 smithi122 ceph-mon[73495]: 1.1c scrub ok 2024-03-20T09:46:58.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:57 smithi115 ceph-mon[72476]: pgmap v2905: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:58.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:57 smithi122 ceph-mon[73495]: pgmap v2905: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:58.456 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:57 smithi026 ceph-mon[68309]: pgmap v2905: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:59.137 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:46:59 smithi026 ceph-mon[68309]: pgmap v2906: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:59.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:46:59 smithi115 ceph-mon[72476]: pgmap v2906: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:46:59.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:46:59 smithi122 ceph-mon[73495]: pgmap v2906: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:00.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:00 smithi026 ceph-mon[68309]: 1.c scrub starts 2024-03-20T09:47:00.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:00 smithi026 ceph-mon[68309]: 1.c scrub ok 2024-03-20T09:47:00.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:00 smithi115 ceph-mon[72476]: 1.c scrub starts 2024-03-20T09:47:00.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:00 smithi115 ceph-mon[72476]: 1.c scrub ok 2024-03-20T09:47:00.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:00 smithi122 ceph-mon[73495]: 1.c scrub starts 2024-03-20T09:47:00.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:00 smithi122 ceph-mon[73495]: 1.c scrub ok 2024-03-20T09:47:01.299 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:01 smithi026 ceph-mon[68309]: 1.16 scrub starts 2024-03-20T09:47:01.299 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:01 smithi026 ceph-mon[68309]: 1.16 scrub ok 2024-03-20T09:47:01.299 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:01 smithi026 ceph-mon[68309]: pgmap v2907: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:01.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:01 smithi115 ceph-mon[72476]: 1.16 scrub starts 2024-03-20T09:47:01.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:01 smithi115 ceph-mon[72476]: 1.16 scrub ok 2024-03-20T09:47:01.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:01 smithi115 ceph-mon[72476]: pgmap v2907: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:01.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:01 smithi122 ceph-mon[73495]: 1.16 scrub starts 2024-03-20T09:47:01.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:01 smithi122 ceph-mon[73495]: 1.16 scrub ok 2024-03-20T09:47:01.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:01 smithi122 ceph-mon[73495]: pgmap v2907: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:02.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:02 smithi115 ceph-mon[72476]: 1.3 scrub starts 2024-03-20T09:47:02.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:02 smithi115 ceph-mon[72476]: 1.12 scrub starts 2024-03-20T09:47:02.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:02 smithi115 ceph-mon[72476]: 1.3 scrub ok 2024-03-20T09:47:02.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:02 smithi115 ceph-mon[72476]: 1.12 scrub ok 2024-03-20T09:47:02.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:02 smithi115 ceph-mon[72476]: 1.7 scrub starts 2024-03-20T09:47:02.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:02 smithi115 ceph-mon[72476]: 1.7 scrub ok 2024-03-20T09:47:02.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:02 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:47:02.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:02 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:47:02.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:02 smithi122 ceph-mon[73495]: 1.3 scrub starts 2024-03-20T09:47:02.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:02 smithi122 ceph-mon[73495]: 1.12 scrub starts 2024-03-20T09:47:02.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:02 smithi122 ceph-mon[73495]: 1.3 scrub ok 2024-03-20T09:47:02.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:02 smithi122 ceph-mon[73495]: 1.12 scrub ok 2024-03-20T09:47:02.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:02 smithi122 ceph-mon[73495]: 1.7 scrub starts 2024-03-20T09:47:02.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:02 smithi122 ceph-mon[73495]: 1.7 scrub ok 2024-03-20T09:47:02.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:02 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:47:02.434 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:02 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:47:02.456 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:02 smithi026 ceph-mon[68309]: 1.3 scrub starts 2024-03-20T09:47:02.456 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:02 smithi026 ceph-mon[68309]: 1.12 scrub starts 2024-03-20T09:47:02.456 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:02 smithi026 ceph-mon[68309]: 1.3 scrub ok 2024-03-20T09:47:02.456 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:02 smithi026 ceph-mon[68309]: 1.12 scrub ok 2024-03-20T09:47:02.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:02 smithi026 ceph-mon[68309]: 1.7 scrub starts 2024-03-20T09:47:02.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:02 smithi026 ceph-mon[68309]: 1.7 scrub ok 2024-03-20T09:47:02.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:02 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:47:02.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:02 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:47:03.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:03 smithi026 ceph-mon[68309]: osdmap e1196: 12 total, 12 up, 12 in 2024-03-20T09:47:03.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:03 smithi026 ceph-mon[68309]: pgmap v2909: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:03.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:03 smithi115 ceph-mon[72476]: osdmap e1196: 12 total, 12 up, 12 in 2024-03-20T09:47:03.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:03 smithi115 ceph-mon[72476]: pgmap v2909: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:03.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:03 smithi122 ceph-mon[73495]: osdmap e1196: 12 total, 12 up, 12 in 2024-03-20T09:47:03.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:03 smithi122 ceph-mon[73495]: pgmap v2909: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:04.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:04 smithi115 ceph-mon[72476]: 1.13 scrub starts 2024-03-20T09:47:04.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:04 smithi115 ceph-mon[72476]: 1.13 scrub ok 2024-03-20T09:47:04.425 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:04 smithi026 ceph-mon[68309]: 1.13 scrub starts 2024-03-20T09:47:04.426 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:04 smithi026 ceph-mon[68309]: 1.13 scrub ok 2024-03-20T09:47:04.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:04 smithi122 ceph-mon[73495]: 1.13 scrub starts 2024-03-20T09:47:04.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:04 smithi122 ceph-mon[73495]: 1.13 scrub ok 2024-03-20T09:47:05.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:05 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:47:05.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:05 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:47:05.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:05 smithi115 ceph-mon[72476]: osdmap e1197: 12 total, 12 up, 12 in 2024-03-20T09:47:05.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:05 smithi115 ceph-mon[72476]: pgmap v2911: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:05.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:05 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:47:05.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:05 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:47:05.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:05 smithi122 ceph-mon[73495]: osdmap e1197: 12 total, 12 up, 12 in 2024-03-20T09:47:05.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:05 smithi122 ceph-mon[73495]: pgmap v2911: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:05.456 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:05 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:47:05.456 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:05 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:47:05.456 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:05 smithi026 ceph-mon[68309]: osdmap e1197: 12 total, 12 up, 12 in 2024-03-20T09:47:05.456 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:05 smithi026 ceph-mon[68309]: pgmap v2911: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:06.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:06 smithi026 ceph-mon[68309]: 1.8 scrub starts 2024-03-20T09:47:06.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:06 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:47:06.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:06 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:47:06.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:06 smithi026 ceph-mon[68309]: 1.8 scrub ok 2024-03-20T09:47:06.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:06 smithi026 ceph-mon[68309]: osdmap e1198: 12 total, 12 up, 12 in 2024-03-20T09:47:06.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:06 smithi115 ceph-mon[72476]: 1.8 scrub starts 2024-03-20T09:47:06.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:06 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:47:06.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:06 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:47:06.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:06 smithi115 ceph-mon[72476]: 1.8 scrub ok 2024-03-20T09:47:06.393 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:06 smithi115 ceph-mon[72476]: osdmap e1198: 12 total, 12 up, 12 in 2024-03-20T09:47:06.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:06 smithi122 ceph-mon[73495]: 1.8 scrub starts 2024-03-20T09:47:06.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:06 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:47:06.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:06 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:47:06.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:06 smithi122 ceph-mon[73495]: 1.8 scrub ok 2024-03-20T09:47:06.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:06 smithi122 ceph-mon[73495]: osdmap e1198: 12 total, 12 up, 12 in 2024-03-20T09:47:07.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:07 smithi115 ceph-mon[72476]: pgmap v2913: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:07.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:07 smithi122 ceph-mon[73495]: pgmap v2913: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:07.456 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:07 smithi026 ceph-mon[68309]: pgmap v2913: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:08.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:08 smithi115 ceph-mon[72476]: 1.6 scrub starts 2024-03-20T09:47:08.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:08 smithi115 ceph-mon[72476]: 1.6 scrub ok 2024-03-20T09:47:08.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:08 smithi122 ceph-mon[73495]: 1.6 scrub starts 2024-03-20T09:47:08.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:08 smithi122 ceph-mon[73495]: 1.6 scrub ok 2024-03-20T09:47:08.456 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:08 smithi026 ceph-mon[68309]: 1.6 scrub starts 2024-03-20T09:47:08.456 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:08 smithi026 ceph-mon[68309]: 1.6 scrub ok 2024-03-20T09:47:09.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:09 smithi115 ceph-mon[72476]: pgmap v2914: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:09.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:09 smithi122 ceph-mon[73495]: pgmap v2914: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:09.456 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:09 smithi026 ceph-mon[68309]: pgmap v2914: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:10.956 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:10 smithi026 ceph-mon[68309]: 1.0 scrub starts 2024-03-20T09:47:10.956 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:10 smithi026 ceph-mon[68309]: 1.0 scrub ok 2024-03-20T09:47:11.045 INFO:teuthology.orchestra.run.smithi026.stdout:77309411988 2024-03-20T09:47:11.046 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-20T09:47:11.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:10 smithi115 ceph-mon[72476]: 1.0 scrub starts 2024-03-20T09:47:11.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:10 smithi115 ceph-mon[72476]: 1.0 scrub ok 2024-03-20T09:47:11.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:10 smithi122 ceph-mon[73495]: 1.0 scrub starts 2024-03-20T09:47:11.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:10 smithi122 ceph-mon[73495]: 1.0 scrub ok 2024-03-20T09:47:11.956 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:11 smithi026 ceph-mon[68309]: pgmap v2915: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:12.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:11 smithi115 ceph-mon[72476]: pgmap v2915: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:12.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:11 smithi122 ceph-mon[73495]: pgmap v2915: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:13.322 INFO:teuthology.orchestra.run.smithi026.stdout:343597384306 2024-03-20T09:47:13.322 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-20T09:47:13.933 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:13 smithi026 ceph-mon[68309]: pgmap v2916: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:13.934 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:13 smithi026 ceph-mon[68309]: 1.11 deep-scrub starts 2024-03-20T09:47:13.934 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:13 smithi026 ceph-mon[68309]: 1.11 deep-scrub ok 2024-03-20T09:47:14.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:13 smithi115 ceph-mon[72476]: pgmap v2916: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:14.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:13 smithi115 ceph-mon[72476]: 1.11 deep-scrub starts 2024-03-20T09:47:14.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:13 smithi115 ceph-mon[72476]: 1.11 deep-scrub ok 2024-03-20T09:47:14.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:13 smithi122 ceph-mon[73495]: pgmap v2916: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:14.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:13 smithi122 ceph-mon[73495]: 1.11 deep-scrub starts 2024-03-20T09:47:14.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:13 smithi122 ceph-mon[73495]: 1.11 deep-scrub ok 2024-03-20T09:47:14.485 INFO:teuthology.orchestra.run.smithi026.stdout:300647711352 2024-03-20T09:47:14.485 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-20T09:47:15.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:14 smithi115 ceph-mon[72476]: 1.5 scrub starts 2024-03-20T09:47:15.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:14 smithi115 ceph-mon[72476]: 1.5 scrub ok 2024-03-20T09:47:15.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:14 smithi122 ceph-mon[73495]: 1.5 scrub starts 2024-03-20T09:47:15.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:14 smithi122 ceph-mon[73495]: 1.5 scrub ok 2024-03-20T09:47:15.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:14 smithi026 ceph-mon[68309]: 1.5 scrub starts 2024-03-20T09:47:15.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:14 smithi026 ceph-mon[68309]: 1.5 scrub ok 2024-03-20T09:47:15.661 INFO:teuthology.orchestra.run.smithi026.stdout:111669150354 2024-03-20T09:47:15.662 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-20T09:47:15.956 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:15 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:47:15.956 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:15 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:47:15.956 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:15 smithi026 ceph-mon[68309]: pgmap v2917: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:15.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:15 smithi026 ceph-mon[68309]: osdmap e1199: 12 total, 12 up, 12 in 2024-03-20T09:47:16.090 INFO:teuthology.orchestra.run.smithi026.stdout:55834575515 2024-03-20T09:47:16.090 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-20T09:47:16.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:15 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:47:16.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:15 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:47:16.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:15 smithi115 ceph-mon[72476]: pgmap v2917: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:16.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:15 smithi115 ceph-mon[72476]: osdmap e1199: 12 total, 12 up, 12 in 2024-03-20T09:47:16.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:15 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:47:16.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:15 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:47:16.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:15 smithi122 ceph-mon[73495]: pgmap v2917: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:16.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:15 smithi122 ceph-mon[73495]: osdmap e1199: 12 total, 12 up, 12 in 2024-03-20T09:47:16.829 INFO:teuthology.orchestra.run.smithi026.stdout:219043332738 2024-03-20T09:47:16.830 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-20T09:47:17.058 INFO:teuthology.orchestra.run.smithi026.stdout:3697966842057 2024-03-20T09:47:17.058 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-20T09:47:17.783 INFO:teuthology.orchestra.run.smithi026.stdout:412316861030 2024-03-20T09:47:17.783 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-20T09:47:17.956 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:17 smithi026 ceph-mon[68309]: pgmap v2919: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:18.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:17 smithi115 ceph-mon[72476]: pgmap v2919: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:18.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:17 smithi122 ceph-mon[73495]: pgmap v2919: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:18.741 INFO:teuthology.orchestra.run.smithi026.stdout:1902670512569 2024-03-20T09:47:18.741 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-20T09:47:18.857 INFO:teuthology.orchestra.run.smithi026.stdout:257698038396 2024-03-20T09:47:18.858 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-20T09:47:19.295 INFO:teuthology.orchestra.run.smithi026.stdout:34359739034 2024-03-20T09:47:19.295 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-20T09:47:19.723 INFO:teuthology.orchestra.run.smithi026.stdout:137438954129 2024-03-20T09:47:19.724 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-20T09:47:19.914 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:47:19.930 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:47:19.936 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:47:19.936 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:47:19.937 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:47:19.941 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:47:19.941 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:47:19.942 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:47:19.942 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:47:19.944 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:47:19.955 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:47:19.956 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:19 smithi026 ceph-mon[68309]: 1.e scrub starts 2024-03-20T09:47:19.956 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:19 smithi026 ceph-mon[68309]: pgmap v2920: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:19.956 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:19 smithi026 ceph-mon[68309]: 1.e scrub ok 2024-03-20T09:47:20.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:19 smithi115 ceph-mon[72476]: 1.e scrub starts 2024-03-20T09:47:20.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:19 smithi115 ceph-mon[72476]: pgmap v2920: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:20.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:19 smithi115 ceph-mon[72476]: 1.e scrub ok 2024-03-20T09:47:20.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:19 smithi122 ceph-mon[73495]: 1.e scrub starts 2024-03-20T09:47:20.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:19 smithi122 ceph-mon[73495]: pgmap v2920: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:20.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:19 smithi122 ceph-mon[73495]: 1.e scrub ok 2024-03-20T09:47:22.107 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:21 smithi026 ceph-mon[68309]: pgmap v2921: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:22.108 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:21 smithi026 ceph-mon[68309]: 1.1 scrub starts 2024-03-20T09:47:22.108 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:21 smithi026 ceph-mon[68309]: 1.1 scrub ok 2024-03-20T09:47:22.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:21 smithi115 ceph-mon[72476]: pgmap v2921: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:22.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:21 smithi115 ceph-mon[72476]: 1.1 scrub starts 2024-03-20T09:47:22.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:21 smithi115 ceph-mon[72476]: 1.1 scrub ok 2024-03-20T09:47:22.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:21 smithi122 ceph-mon[73495]: pgmap v2921: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:22.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:21 smithi122 ceph-mon[73495]: 1.1 scrub starts 2024-03-20T09:47:22.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:21 smithi122 ceph-mon[73495]: 1.1 scrub ok 2024-03-20T09:47:22.855 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:22 smithi026 ceph-mon[68309]: 1.17 scrub starts 2024-03-20T09:47:22.856 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:22 smithi026 ceph-mon[68309]: 1.17 scrub ok 2024-03-20T09:47:23.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:22 smithi115 ceph-mon[72476]: 1.17 scrub starts 2024-03-20T09:47:23.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:22 smithi115 ceph-mon[72476]: 1.17 scrub ok 2024-03-20T09:47:23.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:22 smithi122 ceph-mon[73495]: 1.17 scrub starts 2024-03-20T09:47:23.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:22 smithi122 ceph-mon[73495]: 1.17 scrub ok 2024-03-20T09:47:23.956 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:23 smithi026 ceph-mon[68309]: pgmap v2922: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:24.089 INFO:teuthology.orchestra.run.smithi026.stdout:77309411991 2024-03-20T09:47:24.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:23 smithi115 ceph-mon[72476]: pgmap v2922: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:24.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:23 smithi122 ceph-mon[73495]: pgmap v2922: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:25.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:24 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2664784858' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-20T09:47:25.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:24 smithi115 ceph-mon[72476]: 1.18 scrub starts 2024-03-20T09:47:25.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:24 smithi115 ceph-mon[72476]: 1.18 scrub ok 2024-03-20T09:47:25.166 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:24 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2664784858' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-20T09:47:25.166 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:24 smithi026 ceph-mon[68309]: 1.18 scrub starts 2024-03-20T09:47:25.166 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:24 smithi026 ceph-mon[68309]: 1.18 scrub ok 2024-03-20T09:47:25.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:24 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2664784858' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-20T09:47:25.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:24 smithi122 ceph-mon[73495]: 1.18 scrub starts 2024-03-20T09:47:25.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:24 smithi122 ceph-mon[73495]: 1.18 scrub ok 2024-03-20T09:47:26.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:25 smithi115 ceph-mon[72476]: pgmap v2923: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:26.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:25 smithi122 ceph-mon[73495]: pgmap v2923: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:26.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:25 smithi026 ceph-mon[68309]: pgmap v2923: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:28.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:27 smithi115 ceph-mon[72476]: pgmap v2924: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:28.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:27 smithi122 ceph-mon[73495]: pgmap v2924: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:28.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:27 smithi026 ceph-mon[68309]: pgmap v2924: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:29.268 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:29 smithi026 ceph-mon[68309]: 1.f scrub starts 2024-03-20T09:47:29.268 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:29 smithi026 ceph-mon[68309]: 1.f scrub ok 2024-03-20T09:47:29.268 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:29 smithi026 ceph-mon[68309]: pgmap v2925: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:29.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:29 smithi115 ceph-mon[72476]: 1.f scrub starts 2024-03-20T09:47:29.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:29 smithi115 ceph-mon[72476]: 1.f scrub ok 2024-03-20T09:47:29.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:29 smithi115 ceph-mon[72476]: pgmap v2925: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:29.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:29 smithi122 ceph-mon[73495]: 1.f scrub starts 2024-03-20T09:47:29.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:29 smithi122 ceph-mon[73495]: 1.f scrub ok 2024-03-20T09:47:29.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:29 smithi122 ceph-mon[73495]: pgmap v2925: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:30.530 INFO:teuthology.orchestra.run.smithi026.stdout:219043332741 2024-03-20T09:47:30.990 INFO:teuthology.orchestra.run.smithi026.stdout:55834575519 2024-03-20T09:47:31.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:31 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:47:31.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:31 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:47:31.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:31 smithi026 ceph-mon[68309]: osdmap e1200: 12 total, 12 up, 12 in 2024-03-20T09:47:31.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:31 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/158223697' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-20T09:47:31.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:31 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/901812663' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-20T09:47:31.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:31 smithi026 ceph-mon[68309]: pgmap v2927: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:31.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:31 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4127408405' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-20T09:47:31.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:31 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:47:31.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:31 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:47:31.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:31 smithi115 ceph-mon[72476]: osdmap e1200: 12 total, 12 up, 12 in 2024-03-20T09:47:31.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:31 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/158223697' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-20T09:47:31.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:31 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/901812663' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-20T09:47:31.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:31 smithi115 ceph-mon[72476]: pgmap v2927: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:31.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:31 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4127408405' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-20T09:47:31.424 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411988 got 77309411991 for osd.2 2024-03-20T09:47:31.424 DEBUG:teuthology.parallel:result is None 2024-03-20T09:47:31.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:31 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:47:31.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:31 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:47:31.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:31 smithi122 ceph-mon[73495]: osdmap e1200: 12 total, 12 up, 12 in 2024-03-20T09:47:31.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:31 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/158223697' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-20T09:47:31.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:31 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/901812663' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-20T09:47:31.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:31 smithi122 ceph-mon[73495]: pgmap v2927: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:31.433 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:31 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4127408405' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-20T09:47:32.376 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:32 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:47:32.376 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:32 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:47:32.376 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:32 smithi026 ceph-mon[68309]: osdmap e1201: 12 total, 12 up, 12 in 2024-03-20T09:47:32.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:32 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:47:32.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:32 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:47:32.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:32 smithi115 ceph-mon[72476]: osdmap e1201: 12 total, 12 up, 12 in 2024-03-20T09:47:32.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:32 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:47:32.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:32 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:47:32.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:32 smithi122 ceph-mon[73495]: osdmap e1201: 12 total, 12 up, 12 in 2024-03-20T09:47:32.641 INFO:teuthology.orchestra.run.smithi026.stdout:3697966842060 2024-03-20T09:47:32.885 INFO:teuthology.orchestra.run.smithi026.stdout:343597384311 2024-03-20T09:47:33.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:33 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2623969002' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-20T09:47:33.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:33 smithi115 ceph-mon[72476]: pgmap v2929: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:33.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:33 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2623969002' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-20T09:47:33.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:33 smithi122 ceph-mon[73495]: pgmap v2929: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:33.456 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:33 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2623969002' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-20T09:47:33.456 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:33 smithi026 ceph-mon[68309]: pgmap v2929: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:35.253 INFO:teuthology.orchestra.run.smithi026.stdout:412316861036 2024-03-20T09:47:36.069 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:35 smithi026 ceph-mon[68309]: pgmap v2930: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:36.069 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:35 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/3544258344' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-20T09:47:36.069 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:35 smithi026 ceph-mon[68309]: 1.1d scrub starts 2024-03-20T09:47:36.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:35 smithi115 ceph-mon[72476]: pgmap v2930: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:36.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:35 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/3544258344' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-20T09:47:36.144 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:35 smithi115 ceph-mon[72476]: 1.1d scrub starts 2024-03-20T09:47:36.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:35 smithi122 ceph-mon[73495]: pgmap v2930: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:36.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:35 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/3544258344' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-20T09:47:36.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:35 smithi122 ceph-mon[73495]: 1.1d scrub starts 2024-03-20T09:47:36.841 INFO:tasks.cephadm.ceph_manager.ceph:need seq 412316861030 got 412316861036 for osd.11 2024-03-20T09:47:36.842 DEBUG:teuthology.parallel:result is None 2024-03-20T09:47:37.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:36 smithi115 ceph-mon[72476]: 1.1d scrub ok 2024-03-20T09:47:37.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:36 smithi122 ceph-mon[73495]: 1.1d scrub ok 2024-03-20T09:47:37.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:36 smithi026 ceph-mon[68309]: 1.1d scrub ok 2024-03-20T09:47:37.307 INFO:teuthology.orchestra.run.smithi026.stdout:257698038402 2024-03-20T09:47:37.770 INFO:tasks.cephadm.ceph_manager.ceph:need seq 3697966842057 got 3697966842060 for osd.10 2024-03-20T09:47:37.770 DEBUG:teuthology.parallel:result is None 2024-03-20T09:47:38.139 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:37 smithi026 ceph-mon[68309]: pgmap v2931: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:38.139 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:37 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1247138011' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-20T09:47:38.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:37 smithi115 ceph-mon[72476]: pgmap v2931: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:38.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:37 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1247138011' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-20T09:47:38.165 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597384306 got 343597384311 for osd.9 2024-03-20T09:47:38.165 DEBUG:teuthology.parallel:result is None 2024-03-20T09:47:38.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:37 smithi122 ceph-mon[73495]: pgmap v2931: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:38.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:37 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1247138011' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-20T09:47:38.947 INFO:teuthology.orchestra.run.smithi026.stdout:1902670512574 2024-03-20T09:47:38.956 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:38 smithi026 ceph-mon[68309]: 1.10 scrub starts 2024-03-20T09:47:38.956 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:38 smithi026 ceph-mon[68309]: 1.1e scrub starts 2024-03-20T09:47:38.956 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:38 smithi026 ceph-mon[68309]: 1.10 scrub ok 2024-03-20T09:47:38.956 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:38 smithi026 ceph-mon[68309]: 1.1e scrub ok 2024-03-20T09:47:39.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:38 smithi115 ceph-mon[72476]: 1.10 scrub starts 2024-03-20T09:47:39.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:38 smithi115 ceph-mon[72476]: 1.1e scrub starts 2024-03-20T09:47:39.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:38 smithi115 ceph-mon[72476]: 1.10 scrub ok 2024-03-20T09:47:39.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:38 smithi115 ceph-mon[72476]: 1.1e scrub ok 2024-03-20T09:47:39.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:38 smithi122 ceph-mon[73495]: 1.10 scrub starts 2024-03-20T09:47:39.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:38 smithi122 ceph-mon[73495]: 1.1e scrub starts 2024-03-20T09:47:39.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:38 smithi122 ceph-mon[73495]: 1.10 scrub ok 2024-03-20T09:47:39.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:38 smithi122 ceph-mon[73495]: 1.1e scrub ok 2024-03-20T09:47:39.224 INFO:teuthology.orchestra.run.smithi026.stdout:111669150359 2024-03-20T09:47:39.956 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:39 smithi026 ceph-mon[68309]: pgmap v2932: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:39.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:39 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:47:39.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:39 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:47:39.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:39 smithi026 ceph-mon[68309]: osdmap e1202: 12 total, 12 up, 12 in 2024-03-20T09:47:39.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:39 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/774590770' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-20T09:47:39.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:39 smithi026 ceph-mon[68309]: 1.b scrub starts 2024-03-20T09:47:39.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:39 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/4252203640' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-20T09:47:39.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:39 smithi026 ceph-mon[68309]: 1.b scrub ok 2024-03-20T09:47:39.957 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:39 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/1090103881' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-20T09:47:40.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:39 smithi115 ceph-mon[72476]: pgmap v2932: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:40.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:39 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:47:40.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:39 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:47:40.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:39 smithi115 ceph-mon[72476]: osdmap e1202: 12 total, 12 up, 12 in 2024-03-20T09:47:40.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:39 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/774590770' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-20T09:47:40.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:39 smithi115 ceph-mon[72476]: 1.b scrub starts 2024-03-20T09:47:40.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:39 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/4252203640' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-20T09:47:40.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:39 smithi115 ceph-mon[72476]: 1.b scrub ok 2024-03-20T09:47:40.143 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:39 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/1090103881' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-20T09:47:40.161 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575515 got 55834575519 for osd.1 2024-03-20T09:47:40.162 DEBUG:teuthology.parallel:result is None 2024-03-20T09:47:40.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:39 smithi122 ceph-mon[73495]: pgmap v2932: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:40.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:39 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:47:40.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:39 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:47:40.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:39 smithi122 ceph-mon[73495]: osdmap e1202: 12 total, 12 up, 12 in 2024-03-20T09:47:40.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:39 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/774590770' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-20T09:47:40.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:39 smithi122 ceph-mon[73495]: 1.b scrub starts 2024-03-20T09:47:40.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:39 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/4252203640' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-20T09:47:40.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:39 smithi122 ceph-mon[73495]: 1.b scrub ok 2024-03-20T09:47:40.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:39 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/1090103881' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-20T09:47:40.575 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332738 got 219043332741 for osd.6 2024-03-20T09:47:40.575 DEBUG:teuthology.parallel:result is None 2024-03-20T09:47:40.997 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1902670512569 got 1902670512574 for osd.5 2024-03-20T09:47:40.997 DEBUG:teuthology.parallel:result is None 2024-03-20T09:47:41.468 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698038396 got 257698038402 for osd.7 2024-03-20T09:47:41.468 DEBUG:teuthology.parallel:result is None 2024-03-20T09:47:41.683 INFO:teuthology.orchestra.run.smithi026.stdout:300647711358 2024-03-20T09:47:42.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:41 smithi115 ceph-mon[72476]: pgmap v2934: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:42.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:41 smithi115 ceph-mon[72476]: 1.4 scrub starts 2024-03-20T09:47:42.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:41 smithi115 ceph-mon[72476]: 1.4 scrub ok 2024-03-20T09:47:42.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:41 smithi122 ceph-mon[73495]: pgmap v2934: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:42.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:41 smithi122 ceph-mon[73495]: 1.4 scrub starts 2024-03-20T09:47:42.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:41 smithi122 ceph-mon[73495]: 1.4 scrub ok 2024-03-20T09:47:42.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:41 smithi026 ceph-mon[68309]: pgmap v2934: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:42.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:41 smithi026 ceph-mon[68309]: 1.4 scrub starts 2024-03-20T09:47:42.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:41 smithi026 ceph-mon[68309]: 1.4 scrub ok 2024-03-20T09:47:42.332 INFO:tasks.cephadm.ceph_manager.ceph:need seq 300647711352 got 300647711358 for osd.8 2024-03-20T09:47:42.332 DEBUG:teuthology.parallel:result is None 2024-03-20T09:47:42.705 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:47:42.745 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669150354 got 111669150359 for osd.3 2024-03-20T09:47:42.745 DEBUG:teuthology.parallel:result is None 2024-03-20T09:47:42.956 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:42 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:47:42.956 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:42 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:47:43.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:42 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:47:43.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:42 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:47:43.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:42 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:47:43.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:42 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:47:43.482 INFO:teuthology.orchestra.run.smithi026.stdout:34359739041 2024-03-20T09:47:44.111 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:43 smithi026 ceph-mon[68309]: pgmap v2935: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:44.111 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:43 smithi026 ceph-mon[68309]: osdmap e1203: 12 total, 12 up, 12 in 2024-03-20T09:47:44.111 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:43 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/288186326' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-20T09:47:44.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:43 smithi115 ceph-mon[72476]: pgmap v2935: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:44.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:43 smithi115 ceph-mon[72476]: osdmap e1203: 12 total, 12 up, 12 in 2024-03-20T09:47:44.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:43 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/288186326' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-20T09:47:44.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:43 smithi122 ceph-mon[73495]: pgmap v2935: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:44.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:43 smithi122 ceph-mon[73495]: osdmap e1203: 12 total, 12 up, 12 in 2024-03-20T09:47:44.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:43 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/288186326' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-20T09:47:44.828 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359739034 got 34359739041 for osd.0 2024-03-20T09:47:44.828 DEBUG:teuthology.parallel:result is None 2024-03-20T09:47:45.512 INFO:teuthology.orchestra.run.smithi026.stdout:137438954134 2024-03-20T09:47:46.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:45 smithi115 ceph-mon[72476]: pgmap v2937: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:46.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:45 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:47:46.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:45 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2109005609' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-20T09:47:46.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:45 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:47:46.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:45 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:47:46.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:45 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:47:46.174 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:45 smithi026 ceph-mon[68309]: pgmap v2937: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:46.174 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:45 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:47:46.174 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:45 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2109005609' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-20T09:47:46.174 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:45 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:47:46.174 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:45 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:47:46.174 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:45 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:47:46.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:45 smithi122 ceph-mon[73495]: pgmap v2937: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:46.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:45 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:47:46.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:45 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2109005609' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-20T09:47:46.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:45 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-20T09:47:46.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:45 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-20T09:47:46.183 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:45 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' 2024-03-20T09:47:46.200 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438954129 got 137438954134 for osd.4 2024-03-20T09:47:46.200 DEBUG:teuthology.parallel:result is None 2024-03-20T09:47:46.201 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-20T09:47:46.201 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:47:46.477 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:47:47.996 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:47 smithi026 ceph-mon[68309]: pgmap v2938: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:48.142 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:47 smithi115 ceph-mon[72476]: pgmap v2938: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:48.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:47 smithi122 ceph-mon[73495]: pgmap v2938: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:48.455 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:47:48.457 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:47:49.174 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:49 smithi026 ceph-mon[68309]: 1.9 scrub starts 2024-03-20T09:47:49.174 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:49 smithi026 ceph-mon[68309]: 1.9 scrub ok 2024-03-20T09:47:49.174 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:49 smithi026 ceph-mon[68309]: from='client.18000 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:47:49.174 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:49 smithi026 ceph-mon[68309]: pgmap v2939: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:49.205 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":2938,"stamp":"2024-03-20T09:47:46.753020+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4248,"ondisk_log_size":4248,"up":93,"acting":93,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":93,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":10706852,"kb_used_data":10644,"kb_used_omap":301,"kb_used_meta":10695762,"kb_avail":1114234972,"statfs":{"total":1151940427776,"available":1140976611328,"internally_reserved":0,"allocated":10899456,"data_stored":7011447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":308246,"internal_metadata":10952461290},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001672"},"pg_stats":[{"pgid":"1.1e","version":"20'137","reported_seq":1923,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.830244+0000","last_change":"2024-03-20T09:47:38.106111+0000","last_active":"2024-03-20T09:47:42.830244+0000","last_peered":"2024-03-20T09:47:42.830244+0000","last_clean":"2024-03-20T09:47:42.830244+0000","last_became_active":"2024-03-20T09:13:20.071655+0000","last_became_peered":"2024-03-20T09:13:20.071655+0000","last_unstale":"2024-03-20T09:47:42.830244+0000","last_undegraded":"2024-03-20T09:47:42.830244+0000","last_fullsized":"2024-03-20T09:47:42.830244+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:38.105992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:47:38.105992+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:59.133211+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'137","reported_seq":1455,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.874108+0000","last_change":"2024-03-20T09:47:35.656296+0000","last_active":"2024-03-20T09:47:42.874108+0000","last_peered":"2024-03-20T09:47:42.874108+0000","last_clean":"2024-03-20T09:47:42.874108+0000","last_became_active":"2024-03-20T09:45:33.792524+0000","last_became_peered":"2024-03-20T09:45:33.792524+0000","last_unstale":"2024-03-20T09:47:42.874108+0000","last_undegraded":"2024-03-20T09:47:42.874108+0000","last_fullsized":"2024-03-20T09:47:42.874108+0000","mapping_epoch":1164,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:35.656192+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:29:02.585165+0000","last_clean_scrub_stamp":"2024-03-20T09:47:35.656192+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:40.691308+0000","scrub_duration":178,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"20'137","reported_seq":2610,"reported_epoch":1202,"state":"active+clean","last_fresh":"2024-03-20T09:47:39.713517+0000","last_change":"2024-03-20T09:46:55.841718+0000","last_active":"2024-03-20T09:47:39.713517+0000","last_peered":"2024-03-20T09:47:39.713517+0000","last_clean":"2024-03-20T09:47:39.713517+0000","last_became_active":"2024-03-20T09:35:03.815532+0000","last_became_peered":"2024-03-20T09:35:03.815532+0000","last_unstale":"2024-03-20T09:47:39.713517+0000","last_undegraded":"2024-03-20T09:47:39.713517+0000","last_fullsized":"2024-03-20T09:47:39.713517+0000","mapping_epoch":950,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":951,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:46:55.841552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:46:55.841552+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:14.261015+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.19","version":"20'137","reported_seq":1795,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.874402+0000","last_change":"2024-03-20T09:46:35.775936+0000","last_active":"2024-03-20T09:47:42.874402+0000","last_peered":"2024-03-20T09:47:42.874402+0000","last_clean":"2024-03-20T09:47:42.874402+0000","last_became_active":"2024-03-20T09:44:56.019901+0000","last_became_peered":"2024-03-20T09:44:56.019901+0000","last_unstale":"2024-03-20T09:47:42.874402+0000","last_undegraded":"2024-03-20T09:47:42.874402+0000","last_fullsized":"2024-03-20T09:47:42.874402+0000","mapping_epoch":1149,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":1150,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:46:35.775822+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:46:35.775822+0000","last_clean_scrub_stamp":"2024-03-20T09:46:35.775822+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:47:55.217014+0000","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"20'137","reported_seq":2076,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.874364+0000","last_change":"2024-03-20T09:47:24.230779+0000","last_active":"2024-03-20T09:47:42.874364+0000","last_peered":"2024-03-20T09:47:42.874364+0000","last_clean":"2024-03-20T09:47:42.874364+0000","last_became_active":"2024-03-20T09:10:35.277481+0000","last_became_peered":"2024-03-20T09:10:35.277481+0000","last_unstale":"2024-03-20T09:47:42.874364+0000","last_undegraded":"2024-03-20T09:47:42.874364+0000","last_fullsized":"2024-03-20T09:47:42.874364+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:24.230576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:47:24.230576+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:42.742383+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1b","version":"20'137","reported_seq":2448,"reported_epoch":1202,"state":"active+clean","last_fresh":"2024-03-20T09:47:39.713241+0000","last_change":"2024-03-20T09:46:36.826924+0000","last_active":"2024-03-20T09:47:39.713241+0000","last_peered":"2024-03-20T09:47:39.713241+0000","last_clean":"2024-03-20T09:47:39.713241+0000","last_became_active":"2024-03-20T09:44:56.021631+0000","last_became_peered":"2024-03-20T09:44:56.021631+0000","last_unstale":"2024-03-20T09:47:39.713241+0000","last_undegraded":"2024-03-20T09:47:39.713241+0000","last_fullsized":"2024-03-20T09:47:39.713241+0000","mapping_epoch":1149,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":1150,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:46:36.826748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:46:36.826748+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:04.251292+0000","scrub_duration":170,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.8","version":"20'137","reported_seq":1554,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.874507+0000","last_change":"2024-03-20T09:47:05.079602+0000","last_active":"2024-03-20T09:47:42.874507+0000","last_peered":"2024-03-20T09:47:42.874507+0000","last_clean":"2024-03-20T09:47:42.874507+0000","last_became_active":"2024-03-20T09:46:39.918170+0000","last_became_peered":"2024-03-20T09:46:39.918170+0000","last_unstale":"2024-03-20T09:47:42.874507+0000","last_undegraded":"2024-03-20T09:47:42.874507+0000","last_fullsized":"2024-03-20T09:47:42.874507+0000","mapping_epoch":1192,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":1193,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:05.079463+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:37:01.184206+0000","last_clean_scrub_stamp":"2024-03-20T09:47:05.079463+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:21.834998+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"20'137","reported_seq":628,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.826493+0000","last_change":"2024-03-20T09:46:13.752371+0000","last_active":"2024-03-20T09:47:42.826493+0000","last_peered":"2024-03-20T09:47:42.826493+0000","last_clean":"2024-03-20T09:47:42.826493+0000","last_became_active":"2024-03-20T09:46:12.814810+0000","last_became_peered":"2024-03-20T09:46:12.814810+0000","last_unstale":"2024-03-20T09:47:42.826493+0000","last_undegraded":"2024-03-20T09:47:42.826493+0000","last_fullsized":"2024-03-20T09:47:42.826493+0000","mapping_epoch":1179,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":1180,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:46:13.752273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:46:13.752273+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":158,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"20'137","reported_seq":1343,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.874137+0000","last_change":"2024-03-20T09:46:46.067485+0000","last_active":"2024-03-20T09:47:42.874137+0000","last_peered":"2024-03-20T09:47:42.874137+0000","last_clean":"2024-03-20T09:47:42.874137+0000","last_became_active":"2024-03-20T09:24:16.444232+0000","last_became_peered":"2024-03-20T09:24:16.444232+0000","last_unstale":"2024-03-20T09:47:42.874137+0000","last_undegraded":"2024-03-20T09:47:42.874137+0000","last_fullsized":"2024-03-20T09:47:42.874137+0000","mapping_epoch":718,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:46:46.067377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:46:46.067377+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:15.209062+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"20'137","reported_seq":2590,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.830242+0000","last_change":"2024-03-20T09:47:41.128034+0000","last_active":"2024-03-20T09:47:42.830242+0000","last_peered":"2024-03-20T09:47:42.830242+0000","last_clean":"2024-03-20T09:47:42.830242+0000","last_became_active":"2024-03-20T09:12:14.306384+0000","last_became_peered":"2024-03-20T09:12:14.306384+0000","last_unstale":"2024-03-20T09:47:42.830242+0000","last_undegraded":"2024-03-20T09:47:42.830242+0000","last_fullsized":"2024-03-20T09:47:42.830242+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:41.127897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:47:41.127897+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:49:10.331953+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"20'137","reported_seq":1249,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.874421+0000","last_change":"2024-03-20T09:47:13.488755+0000","last_active":"2024-03-20T09:47:42.874421+0000","last_peered":"2024-03-20T09:47:42.874421+0000","last_clean":"2024-03-20T09:47:42.874421+0000","last_became_active":"2024-03-20T09:35:19.820895+0000","last_became_peered":"2024-03-20T09:35:19.820895+0000","last_unstale":"2024-03-20T09:47:42.874421+0000","last_undegraded":"2024-03-20T09:47:42.874421+0000","last_fullsized":"2024-03-20T09:47:42.874421+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:13.488637+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:19:03.805185+0000","last_clean_scrub_stamp":"2024-03-20T09:47:13.488637+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:26.639558+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"20'137","reported_seq":2119,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.874522+0000","last_change":"2024-03-20T09:47:07.527703+0000","last_active":"2024-03-20T09:47:42.874522+0000","last_peered":"2024-03-20T09:47:42.874522+0000","last_clean":"2024-03-20T09:47:42.874522+0000","last_became_active":"2024-03-20T09:35:19.822321+0000","last_became_peered":"2024-03-20T09:35:19.822321+0000","last_unstale":"2024-03-20T09:47:42.874522+0000","last_undegraded":"2024-03-20T09:47:42.874522+0000","last_fullsized":"2024-03-20T09:47:42.874522+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:07.527585+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:44:34.512542+0000","last_clean_scrub_stamp":"2024-03-20T09:47:07.527585+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:23.296680+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.3","version":"20'137","reported_seq":1864,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.874058+0000","last_change":"2024-03-20T09:47:01.050823+0000","last_active":"2024-03-20T09:47:42.874058+0000","last_peered":"2024-03-20T09:47:42.874058+0000","last_clean":"2024-03-20T09:47:42.874058+0000","last_became_active":"2024-03-20T09:45:33.791162+0000","last_became_peered":"2024-03-20T09:45:33.791162+0000","last_unstale":"2024-03-20T09:47:42.874058+0000","last_undegraded":"2024-03-20T09:47:42.874058+0000","last_fullsized":"2024-03-20T09:47:42.874058+0000","mapping_epoch":1164,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:01.050701+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:45:37.771966+0000","last_clean_scrub_stamp":"2024-03-20T09:47:01.050701+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:15.644254+0000","scrub_duration":267,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":1030,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.826518+0000","last_change":"2024-03-20T09:47:10.015594+0000","last_active":"2024-03-20T09:47:42.826518+0000","last_peered":"2024-03-20T09:47:42.826518+0000","last_clean":"2024-03-20T09:47:42.826518+0000","last_became_active":"2024-03-20T09:46:12.816798+0000","last_became_peered":"2024-03-20T09:46:12.816798+0000","last_unstale":"2024-03-20T09:47:42.826518+0000","last_undegraded":"2024-03-20T09:47:42.826518+0000","last_fullsized":"2024-03-20T09:47:42.826518+0000","mapping_epoch":1179,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1180,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:10.015497+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:45:58.520390+0000","last_clean_scrub_stamp":"2024-03-20T09:47:10.015497+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:31.863180+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"20'137","reported_seq":1280,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.826493+0000","last_change":"2024-03-20T09:46:37.777695+0000","last_active":"2024-03-20T09:47:42.826493+0000","last_peered":"2024-03-20T09:47:42.826493+0000","last_clean":"2024-03-20T09:47:42.826493+0000","last_became_active":"2024-03-20T09:46:12.815891+0000","last_became_peered":"2024-03-20T09:46:12.815891+0000","last_unstale":"2024-03-20T09:47:42.826493+0000","last_undegraded":"2024-03-20T09:47:42.826493+0000","last_fullsized":"2024-03-20T09:47:42.826493+0000","mapping_epoch":1179,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":1180,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:46:37.777576+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:42:42.369236+0000","last_clean_scrub_stamp":"2024-03-20T09:46:37.777576+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:47:54.069393+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"20'137","reported_seq":2003,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.874193+0000","last_change":"2024-03-20T09:47:21.147759+0000","last_active":"2024-03-20T09:47:42.874193+0000","last_peered":"2024-03-20T09:47:42.874193+0000","last_clean":"2024-03-20T09:47:42.874193+0000","last_became_active":"2024-03-20T09:15:34.225022+0000","last_became_peered":"2024-03-20T09:15:34.225022+0000","last_unstale":"2024-03-20T09:47:42.874193+0000","last_undegraded":"2024-03-20T09:47:42.874193+0000","last_fullsized":"2024-03-20T09:47:42.874193+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:21.147640+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:20:12.169658+0000","last_clean_scrub_stamp":"2024-03-20T09:47:21.147640+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:23.014691+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,5],"acting":[6,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"20'137","reported_seq":2559,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.874829+0000","last_change":"2024-03-20T09:47:01.337794+0000","last_active":"2024-03-20T09:47:42.874829+0000","last_peered":"2024-03-20T09:47:42.874829+0000","last_clean":"2024-03-20T09:47:42.874829+0000","last_became_active":"2024-03-20T09:35:19.820396+0000","last_became_peered":"2024-03-20T09:35:19.820396+0000","last_unstale":"2024-03-20T09:47:42.874829+0000","last_undegraded":"2024-03-20T09:47:42.874829+0000","last_fullsized":"2024-03-20T09:47:42.874829+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:01.337692+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:14:10.560890+0000","last_clean_scrub_stamp":"2024-03-20T09:47:01.337692+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:27.673956+0000","scrub_duration":192,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"20'137","reported_seq":2400,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.830223+0000","last_change":"2024-03-20T09:47:39.279094+0000","last_active":"2024-03-20T09:47:42.830223+0000","last_peered":"2024-03-20T09:47:42.830223+0000","last_clean":"2024-03-20T09:47:42.830223+0000","last_became_active":"2024-03-20T09:44:56.018327+0000","last_became_peered":"2024-03-20T09:44:56.018327+0000","last_unstale":"2024-03-20T09:47:42.830223+0000","last_undegraded":"2024-03-20T09:47:42.830223+0000","last_fullsized":"2024-03-20T09:47:42.830223+0000","mapping_epoch":1149,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":1150,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:39.278991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:47:39.278991+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:50.220826+0000","scrub_duration":231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.e","version":"20'137","reported_seq":1815,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.874344+0000","last_change":"2024-03-20T09:47:18.794891+0000","last_active":"2024-03-20T09:47:42.874344+0000","last_peered":"2024-03-20T09:47:42.874344+0000","last_clean":"2024-03-20T09:47:42.874344+0000","last_became_active":"2024-03-20T09:19:54.506134+0000","last_became_peered":"2024-03-20T09:19:54.506134+0000","last_unstale":"2024-03-20T09:47:42.874344+0000","last_undegraded":"2024-03-20T09:47:42.874344+0000","last_fullsized":"2024-03-20T09:47:42.874344+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:18.794773+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:45:49.636483+0000","last_clean_scrub_stamp":"2024-03-20T09:47:18.794773+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:20.939771+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"20'137","reported_seq":2578,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.874741+0000","last_change":"2024-03-20T09:47:28.282024+0000","last_active":"2024-03-20T09:47:42.874741+0000","last_peered":"2024-03-20T09:47:42.874741+0000","last_clean":"2024-03-20T09:47:42.874741+0000","last_became_active":"2024-03-20T09:19:54.524397+0000","last_became_peered":"2024-03-20T09:19:54.524397+0000","last_unstale":"2024-03-20T09:47:42.874741+0000","last_undegraded":"2024-03-20T09:47:42.874741+0000","last_fullsized":"2024-03-20T09:47:42.874741+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:28.281922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:47:28.281922+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:43.709927+0000","scrub_duration":229,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"570'138","reported_seq":2583,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.873802+0000","last_change":"2024-03-20T09:46:59.159617+0000","last_active":"2024-03-20T09:47:42.873802+0000","last_peered":"2024-03-20T09:47:42.873802+0000","last_clean":"2024-03-20T09:47:42.873802+0000","last_became_active":"2024-03-20T09:44:56.021764+0000","last_became_peered":"2024-03-20T09:44:56.021764+0000","last_unstale":"2024-03-20T09:47:42.873802+0000","last_undegraded":"2024-03-20T09:47:42.873802+0000","last_fullsized":"2024-03-20T09:47:42.873802+0000","mapping_epoch":1149,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":1150,"parent":"0.0","parent_split_bits":4,"last_scrub":"570'138","last_scrub_stamp":"2024-03-20T09:46:59.159506+0000","last_deep_scrub":"570'138","last_deep_scrub_stamp":"2024-03-20T09:16:13.725035+0000","last_clean_scrub_stamp":"2024-03-20T09:46:59.159506+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:02.429882+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"20'137","reported_seq":555,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.826505+0000","last_change":"2024-03-20T09:46:12.815458+0000","last_active":"2024-03-20T09:47:42.826505+0000","last_peered":"2024-03-20T09:47:42.826505+0000","last_clean":"2024-03-20T09:47:42.826505+0000","last_became_active":"2024-03-20T09:46:12.815073+0000","last_became_peered":"2024-03-20T09:46:12.815073+0000","last_unstale":"2024-03-20T09:47:42.826505+0000","last_undegraded":"2024-03-20T09:47:42.826505+0000","last_fullsized":"2024-03-20T09:47:42.826505+0000","mapping_epoch":1179,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":1180,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:46:09.656036+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:41:52.574619+0000","last_clean_scrub_stamp":"2024-03-20T09:46:09.656036+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"20'137","reported_seq":2408,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.874758+0000","last_change":"2024-03-20T09:47:01.116616+0000","last_active":"2024-03-20T09:47:42.874758+0000","last_peered":"2024-03-20T09:47:42.874758+0000","last_clean":"2024-03-20T09:47:42.874758+0000","last_became_active":"2024-03-20T09:10:34.096758+0000","last_became_peered":"2024-03-20T09:10:34.096758+0000","last_unstale":"2024-03-20T09:47:42.874758+0000","last_undegraded":"2024-03-20T09:47:42.874758+0000","last_fullsized":"2024-03-20T09:47:42.874758+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:01.116516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:47:01.116516+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:12.143664+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"20'137","reported_seq":1845,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.873923+0000","last_change":"2024-03-20T09:47:03.072061+0000","last_active":"2024-03-20T09:47:42.873923+0000","last_peered":"2024-03-20T09:47:42.873923+0000","last_clean":"2024-03-20T09:47:42.873923+0000","last_became_active":"2024-03-20T09:45:33.792850+0000","last_became_peered":"2024-03-20T09:45:33.792850+0000","last_unstale":"2024-03-20T09:47:42.873923+0000","last_undegraded":"2024-03-20T09:47:42.873923+0000","last_fullsized":"2024-03-20T09:47:42.873923+0000","mapping_epoch":1164,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:03.071960+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:37:52.359932+0000","last_clean_scrub_stamp":"2024-03-20T09:47:03.071960+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:17.487704+0000","scrub_duration":275,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"20'137","reported_seq":1979,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.874629+0000","last_change":"2024-03-20T09:47:38.043879+0000","last_active":"2024-03-20T09:47:42.874629+0000","last_peered":"2024-03-20T09:47:42.874629+0000","last_clean":"2024-03-20T09:47:42.874629+0000","last_became_active":"2024-03-20T09:35:19.820595+0000","last_became_peered":"2024-03-20T09:35:19.820595+0000","last_unstale":"2024-03-20T09:47:42.874629+0000","last_undegraded":"2024-03-20T09:47:42.874629+0000","last_fullsized":"2024-03-20T09:47:42.874629+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:38.043732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:47:38.043732+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:49.450295+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"20'137","reported_seq":512,"reported_epoch":1202,"state":"active+clean","last_fresh":"2024-03-20T09:47:39.712976+0000","last_change":"2024-03-20T09:47:12.865818+0000","last_active":"2024-03-20T09:47:39.712976+0000","last_peered":"2024-03-20T09:47:39.712976+0000","last_clean":"2024-03-20T09:47:39.712976+0000","last_became_active":"2024-03-20T09:44:56.020550+0000","last_became_peered":"2024-03-20T09:44:56.020550+0000","last_unstale":"2024-03-20T09:47:39.712976+0000","last_undegraded":"2024-03-20T09:47:39.712976+0000","last_fullsized":"2024-03-20T09:47:39.712976+0000","mapping_epoch":1149,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":1150,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:12.865640+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:47:12.865640+0000","last_clean_scrub_stamp":"2024-03-20T09:47:12.865640+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:19.108754+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"20'137","reported_seq":1539,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.873981+0000","last_change":"2024-03-20T09:46:59.919713+0000","last_active":"2024-03-20T09:47:42.873981+0000","last_peered":"2024-03-20T09:47:42.873981+0000","last_clean":"2024-03-20T09:47:42.873981+0000","last_became_active":"2024-03-20T09:45:33.793676+0000","last_became_peered":"2024-03-20T09:45:33.793676+0000","last_unstale":"2024-03-20T09:47:42.873981+0000","last_undegraded":"2024-03-20T09:47:42.873981+0000","last_fullsized":"2024-03-20T09:47:42.873981+0000","mapping_epoch":1164,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:46:59.919605+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:35:27.264203+0000","last_clean_scrub_stamp":"2024-03-20T09:46:59.919605+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:21.314623+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"20'137","reported_seq":2548,"reported_epoch":1202,"state":"active+clean","last_fresh":"2024-03-20T09:47:39.713399+0000","last_change":"2024-03-20T09:47:21.949925+0000","last_active":"2024-03-20T09:47:39.713399+0000","last_peered":"2024-03-20T09:47:39.713399+0000","last_clean":"2024-03-20T09:47:39.713399+0000","last_became_active":"2024-03-20T09:35:19.822429+0000","last_became_peered":"2024-03-20T09:35:19.822429+0000","last_unstale":"2024-03-20T09:47:39.713399+0000","last_undegraded":"2024-03-20T09:47:39.713399+0000","last_fullsized":"2024-03-20T09:47:39.713399+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:21.949796+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:38:01.419884+0000","last_clean_scrub_stamp":"2024-03-20T09:47:21.949796+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:37.511012+0000","scrub_duration":197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"20'137","reported_seq":2571,"reported_epoch":1202,"state":"active+clean","last_fresh":"2024-03-20T09:47:39.712991+0000","last_change":"2024-03-20T09:46:49.337362+0000","last_active":"2024-03-20T09:47:39.712991+0000","last_peered":"2024-03-20T09:47:39.712991+0000","last_clean":"2024-03-20T09:47:39.712991+0000","last_became_active":"2024-03-20T09:20:25.349082+0000","last_became_peered":"2024-03-20T09:20:25.349082+0000","last_unstale":"2024-03-20T09:47:39.712991+0000","last_undegraded":"2024-03-20T09:47:39.712991+0000","last_fullsized":"2024-03-20T09:47:39.712991+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:46:49.337261+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:41:54.735667+0000","last_clean_scrub_stamp":"2024-03-20T09:46:49.337261+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:47:51.674586+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"20'137","reported_seq":1853,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.826681+0000","last_change":"2024-03-20T09:46:33.872550+0000","last_active":"2024-03-20T09:47:42.826681+0000","last_peered":"2024-03-20T09:47:42.826681+0000","last_clean":"2024-03-20T09:47:42.826681+0000","last_became_active":"2024-03-20T09:12:14.307581+0000","last_became_peered":"2024-03-20T09:12:14.307581+0000","last_unstale":"2024-03-20T09:47:42.826681+0000","last_undegraded":"2024-03-20T09:47:42.826681+0000","last_fullsized":"2024-03-20T09:47:42.826681+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:46:33.872418+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:34:48.079272+0000","last_clean_scrub_stamp":"2024-03-20T09:46:33.872418+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:47:46.392227+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"20'137","reported_seq":1117,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.873831+0000","last_change":"2024-03-20T09:46:39.156360+0000","last_active":"2024-03-20T09:47:42.873831+0000","last_peered":"2024-03-20T09:47:42.873831+0000","last_clean":"2024-03-20T09:47:42.873831+0000","last_became_active":"2024-03-20T09:41:47.137425+0000","last_became_peered":"2024-03-20T09:41:47.137425+0000","last_unstale":"2024-03-20T09:47:42.873831+0000","last_undegraded":"2024-03-20T09:47:42.873831+0000","last_fullsized":"2024-03-20T09:47:42.873831+0000","mapping_epoch":1094,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":1095,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:46:39.156256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:46:39.156256+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:06.149448+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4248,"ondisk_log_size":4248,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316861039,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1110660,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":1109886,"kb_avail":92634492,"statfs":{"total":95995035648,"available":94857719808,"internally_reserved":0,"allocated":778240,"data_stored":453928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":1136523716},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":861,"seq":3697966842063,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313604,"kb_used_data":760,"kb_used_omap":103,"kb_used_meta":312728,"kb_avail":93431548,"statfs":{"total":95995035648,"available":95673905152,"internally_reserved":0,"allocated":778240,"data_stored":453928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":106169,"internal_metadata":320233799},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":80,"seq":343597384316,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":762180,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":761406,"kb_avail":92982972,"statfs":{"total":95995035648,"available":95214563328,"internally_reserved":0,"allocated":778240,"data_stored":453928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":779680192},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":70,"seq":300647711359,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":685816,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93059336,"statfs":{"total":95995035648,"available":95292760064,"internally_reserved":0,"allocated":765952,"data_stored":446437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":701495750},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":60,"seq":257698038404,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1020620,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":1019646,"kb_avail":92724532,"statfs":{"total":95995035648,"available":94949920768,"internally_reserved":0,"allocated":983040,"data_stored":655176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":1044117962},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332745,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":768068,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":766974,"kb_avail":92977084,"statfs":{"total":95995035648,"available":95208534016,"internally_reserved":0,"allocated":1105920,"data_stored":781608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":785381830},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575523,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1191172,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":1190398,"kb_avail":92553980,"statfs":{"total":95995035648,"available":94775275520,"internally_reserved":0,"allocated":778240,"data_stored":453928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1218968006},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359739041,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1047096,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":1046014,"kb_avail":92698056,"statfs":{"total":95995035648,"available":94922809344,"internally_reserved":0,"allocated":1093632,"data_stored":774117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1071118789},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411996,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":852608,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":92892544,"statfs":{"total":95995035648,"available":95121965056,"internally_reserved":0,"allocated":970752,"data_stored":647685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":872085957},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150361,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1141508,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":1140414,"kb_avail":92603644,"statfs":{"total":95995035648,"available":94826131456,"internally_reserved":0,"allocated":1105920,"data_stored":781608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1167784391},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438954135,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":808004,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":92937148,"statfs":{"total":95995035648,"available":95167639552,"internally_reserved":0,"allocated":778240,"data_stored":453928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":826603973},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":443,"seq":1902670512576,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1005516,"kb_used_data":960,"kb_used_omap":181,"kb_used_meta":1004362,"kb_avail":92739636,"statfs":{"total":95995035648,"available":94965387264,"internally_reserved":0,"allocated":983040,"data_stored":655176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":186131,"internal_metadata":1028466925},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:47:49.207 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-20T09:47:49.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:49 smithi115 ceph-mon[72476]: 1.9 scrub starts 2024-03-20T09:47:49.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:49 smithi115 ceph-mon[72476]: 1.9 scrub ok 2024-03-20T09:47:49.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:49 smithi115 ceph-mon[72476]: from='client.18000 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:47:49.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:49 smithi115 ceph-mon[72476]: pgmap v2939: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:49.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:49 smithi122 ceph-mon[73495]: 1.9 scrub starts 2024-03-20T09:47:49.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:49 smithi122 ceph-mon[73495]: 1.9 scrub ok 2024-03-20T09:47:49.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:49 smithi122 ceph-mon[73495]: from='client.18000 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:47:49.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:49 smithi122 ceph-mon[73495]: pgmap v2939: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:49.489 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:47:50.314 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:50 smithi026 ceph-mon[68309]: purged_snaps scrub starts 2024-03-20T09:47:50.315 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:50 smithi026 ceph-mon[68309]: purged_snaps scrub ok 2024-03-20T09:47:50.315 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:50 smithi026 ceph-mon[68309]: osdmap e1204: 12 total, 12 up, 12 in 2024-03-20T09:47:50.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:50 smithi115 ceph-mon[72476]: purged_snaps scrub starts 2024-03-20T09:47:50.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:50 smithi115 ceph-mon[72476]: purged_snaps scrub ok 2024-03-20T09:47:50.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:50 smithi115 ceph-mon[72476]: osdmap e1204: 12 total, 12 up, 12 in 2024-03-20T09:47:50.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:50 smithi122 ceph-mon[73495]: purged_snaps scrub starts 2024-03-20T09:47:50.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:50 smithi122 ceph-mon[73495]: purged_snaps scrub ok 2024-03-20T09:47:50.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:50 smithi122 ceph-mon[73495]: osdmap e1204: 12 total, 12 up, 12 in 2024-03-20T09:47:51.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:51 smithi115 ceph-mon[72476]: 1.d scrub starts 2024-03-20T09:47:51.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:51 smithi115 ceph-mon[72476]: 1.d scrub ok 2024-03-20T09:47:51.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:51 smithi115 ceph-mon[72476]: pgmap v2941: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:51.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:51 smithi122 ceph-mon[73495]: 1.d scrub starts 2024-03-20T09:47:51.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:51 smithi122 ceph-mon[73495]: 1.d scrub ok 2024-03-20T09:47:51.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:51 smithi122 ceph-mon[73495]: pgmap v2941: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:51.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:51 smithi026 ceph-mon[68309]: 1.d scrub starts 2024-03-20T09:47:51.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:51 smithi026 ceph-mon[68309]: 1.d scrub ok 2024-03-20T09:47:51.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:51 smithi026 ceph-mon[68309]: pgmap v2941: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:51.492 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-20T09:47:51.494 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-20T09:47:52.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:52 smithi026 ceph-mon[68309]: 1.15 scrub starts 2024-03-20T09:47:52.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:52 smithi026 ceph-mon[68309]: 1.15 scrub ok 2024-03-20T09:47:52.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:52 smithi026 ceph-mon[68309]: from='client.18006 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:47:52.332 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":2941,"stamp":"2024-03-20T09:47:50.753954+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4248,"ondisk_log_size":4248,"up":93,"acting":93,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":93,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":10706900,"kb_used_data":10692,"kb_used_omap":301,"kb_used_meta":10695762,"kb_avail":1114234924,"statfs":{"total":1151940427776,"available":1140976562176,"internally_reserved":0,"allocated":10948608,"data_stored":7041411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":308246,"internal_metadata":10952461290},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001693"},"pg_stats":[{"pgid":"1.1e","version":"20'137","reported_seq":1923,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.830244+0000","last_change":"2024-03-20T09:47:38.106111+0000","last_active":"2024-03-20T09:47:42.830244+0000","last_peered":"2024-03-20T09:47:42.830244+0000","last_clean":"2024-03-20T09:47:42.830244+0000","last_became_active":"2024-03-20T09:13:20.071655+0000","last_became_peered":"2024-03-20T09:13:20.071655+0000","last_unstale":"2024-03-20T09:47:42.830244+0000","last_undegraded":"2024-03-20T09:47:42.830244+0000","last_fullsized":"2024-03-20T09:47:42.830244+0000","mapping_epoch":518,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":519,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:38.105992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:47:38.105992+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:59.133211+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"20'137","reported_seq":1457,"reported_epoch":1204,"state":"active+clean","last_fresh":"2024-03-20T09:47:49.614651+0000","last_change":"2024-03-20T09:47:35.656296+0000","last_active":"2024-03-20T09:47:49.614651+0000","last_peered":"2024-03-20T09:47:49.614651+0000","last_clean":"2024-03-20T09:47:49.614651+0000","last_became_active":"2024-03-20T09:45:33.792524+0000","last_became_peered":"2024-03-20T09:45:33.792524+0000","last_unstale":"2024-03-20T09:47:49.614651+0000","last_undegraded":"2024-03-20T09:47:49.614651+0000","last_fullsized":"2024-03-20T09:47:49.614651+0000","mapping_epoch":1164,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:35.656192+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:29:02.585165+0000","last_clean_scrub_stamp":"2024-03-20T09:47:35.656192+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:40.691308+0000","scrub_duration":178,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"20'137","reported_seq":2612,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.874504+0000","last_change":"2024-03-20T09:46:55.841718+0000","last_active":"2024-03-20T09:47:42.874504+0000","last_peered":"2024-03-20T09:47:42.874504+0000","last_clean":"2024-03-20T09:47:42.874504+0000","last_became_active":"2024-03-20T09:35:03.815532+0000","last_became_peered":"2024-03-20T09:35:03.815532+0000","last_unstale":"2024-03-20T09:47:42.874504+0000","last_undegraded":"2024-03-20T09:47:42.874504+0000","last_fullsized":"2024-03-20T09:47:42.874504+0000","mapping_epoch":950,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":951,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:46:55.841552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:46:55.841552+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:14.261015+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.19","version":"20'137","reported_seq":1795,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.874402+0000","last_change":"2024-03-20T09:46:35.775936+0000","last_active":"2024-03-20T09:47:42.874402+0000","last_peered":"2024-03-20T09:47:42.874402+0000","last_clean":"2024-03-20T09:47:42.874402+0000","last_became_active":"2024-03-20T09:44:56.019901+0000","last_became_peered":"2024-03-20T09:44:56.019901+0000","last_unstale":"2024-03-20T09:47:42.874402+0000","last_undegraded":"2024-03-20T09:47:42.874402+0000","last_fullsized":"2024-03-20T09:47:42.874402+0000","mapping_epoch":1149,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":1150,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:46:35.775822+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:46:35.775822+0000","last_clean_scrub_stamp":"2024-03-20T09:46:35.775822+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:47:55.217014+0000","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"20'137","reported_seq":2078,"reported_epoch":1204,"state":"active+clean","last_fresh":"2024-03-20T09:47:49.049883+0000","last_change":"2024-03-20T09:47:24.230779+0000","last_active":"2024-03-20T09:47:49.049883+0000","last_peered":"2024-03-20T09:47:49.049883+0000","last_clean":"2024-03-20T09:47:49.049883+0000","last_became_active":"2024-03-20T09:10:35.277481+0000","last_became_peered":"2024-03-20T09:10:35.277481+0000","last_unstale":"2024-03-20T09:47:49.049883+0000","last_undegraded":"2024-03-20T09:47:49.049883+0000","last_fullsized":"2024-03-20T09:47:49.049883+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:24.230576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:47:24.230576+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:42.742383+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1b","version":"20'137","reported_seq":2450,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.874688+0000","last_change":"2024-03-20T09:46:36.826924+0000","last_active":"2024-03-20T09:47:42.874688+0000","last_peered":"2024-03-20T09:47:42.874688+0000","last_clean":"2024-03-20T09:47:42.874688+0000","last_became_active":"2024-03-20T09:44:56.021631+0000","last_became_peered":"2024-03-20T09:44:56.021631+0000","last_unstale":"2024-03-20T09:47:42.874688+0000","last_undegraded":"2024-03-20T09:47:42.874688+0000","last_fullsized":"2024-03-20T09:47:42.874688+0000","mapping_epoch":1149,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":1150,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:46:36.826748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:46:36.826748+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:04.251292+0000","scrub_duration":170,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.8","version":"20'137","reported_seq":1556,"reported_epoch":1204,"state":"active+clean","last_fresh":"2024-03-20T09:47:49.049883+0000","last_change":"2024-03-20T09:47:05.079602+0000","last_active":"2024-03-20T09:47:49.049883+0000","last_peered":"2024-03-20T09:47:49.049883+0000","last_clean":"2024-03-20T09:47:49.049883+0000","last_became_active":"2024-03-20T09:46:39.918170+0000","last_became_peered":"2024-03-20T09:46:39.918170+0000","last_unstale":"2024-03-20T09:47:49.049883+0000","last_undegraded":"2024-03-20T09:47:49.049883+0000","last_fullsized":"2024-03-20T09:47:49.049883+0000","mapping_epoch":1192,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":1193,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:05.079463+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:37:01.184206+0000","last_clean_scrub_stamp":"2024-03-20T09:47:05.079463+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:21.834998+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"20'137","reported_seq":628,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.826493+0000","last_change":"2024-03-20T09:46:13.752371+0000","last_active":"2024-03-20T09:47:42.826493+0000","last_peered":"2024-03-20T09:47:42.826493+0000","last_clean":"2024-03-20T09:47:42.826493+0000","last_became_active":"2024-03-20T09:46:12.814810+0000","last_became_peered":"2024-03-20T09:46:12.814810+0000","last_unstale":"2024-03-20T09:47:42.826493+0000","last_undegraded":"2024-03-20T09:47:42.826493+0000","last_fullsized":"2024-03-20T09:47:42.826493+0000","mapping_epoch":1179,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":1180,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:46:13.752273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:46:13.752273+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":158,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"20'137","reported_seq":1343,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.874137+0000","last_change":"2024-03-20T09:46:46.067485+0000","last_active":"2024-03-20T09:47:42.874137+0000","last_peered":"2024-03-20T09:47:42.874137+0000","last_clean":"2024-03-20T09:47:42.874137+0000","last_became_active":"2024-03-20T09:24:16.444232+0000","last_became_peered":"2024-03-20T09:24:16.444232+0000","last_unstale":"2024-03-20T09:47:42.874137+0000","last_undegraded":"2024-03-20T09:47:42.874137+0000","last_fullsized":"2024-03-20T09:47:42.874137+0000","mapping_epoch":718,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":719,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:46:46.067377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:46:46.067377+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:15.209062+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.4","version":"20'137","reported_seq":2590,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.830242+0000","last_change":"2024-03-20T09:47:41.128034+0000","last_active":"2024-03-20T09:47:42.830242+0000","last_peered":"2024-03-20T09:47:42.830242+0000","last_clean":"2024-03-20T09:47:42.830242+0000","last_became_active":"2024-03-20T09:12:14.306384+0000","last_became_peered":"2024-03-20T09:12:14.306384+0000","last_unstale":"2024-03-20T09:47:42.830242+0000","last_undegraded":"2024-03-20T09:47:42.830242+0000","last_fullsized":"2024-03-20T09:47:42.830242+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:41.127897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:47:41.127897+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:49:10.331953+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"20'137","reported_seq":1249,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.874421+0000","last_change":"2024-03-20T09:47:13.488755+0000","last_active":"2024-03-20T09:47:42.874421+0000","last_peered":"2024-03-20T09:47:42.874421+0000","last_clean":"2024-03-20T09:47:42.874421+0000","last_became_active":"2024-03-20T09:35:19.820895+0000","last_became_peered":"2024-03-20T09:35:19.820895+0000","last_unstale":"2024-03-20T09:47:42.874421+0000","last_undegraded":"2024-03-20T09:47:42.874421+0000","last_fullsized":"2024-03-20T09:47:42.874421+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:13.488637+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:19:03.805185+0000","last_clean_scrub_stamp":"2024-03-20T09:47:13.488637+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:26.639558+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"20'137","reported_seq":2119,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.874522+0000","last_change":"2024-03-20T09:47:07.527703+0000","last_active":"2024-03-20T09:47:42.874522+0000","last_peered":"2024-03-20T09:47:42.874522+0000","last_clean":"2024-03-20T09:47:42.874522+0000","last_became_active":"2024-03-20T09:35:19.822321+0000","last_became_peered":"2024-03-20T09:35:19.822321+0000","last_unstale":"2024-03-20T09:47:42.874522+0000","last_undegraded":"2024-03-20T09:47:42.874522+0000","last_fullsized":"2024-03-20T09:47:42.874522+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:07.527585+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:44:34.512542+0000","last_clean_scrub_stamp":"2024-03-20T09:47:07.527585+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:23.296680+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.3","version":"20'137","reported_seq":1866,"reported_epoch":1204,"state":"active+clean","last_fresh":"2024-03-20T09:47:49.614620+0000","last_change":"2024-03-20T09:47:01.050823+0000","last_active":"2024-03-20T09:47:49.614620+0000","last_peered":"2024-03-20T09:47:49.614620+0000","last_clean":"2024-03-20T09:47:49.614620+0000","last_became_active":"2024-03-20T09:45:33.791162+0000","last_became_peered":"2024-03-20T09:45:33.791162+0000","last_unstale":"2024-03-20T09:47:49.614620+0000","last_undegraded":"2024-03-20T09:47:49.614620+0000","last_fullsized":"2024-03-20T09:47:49.614620+0000","mapping_epoch":1164,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:01.050701+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:45:37.771966+0000","last_clean_scrub_stamp":"2024-03-20T09:47:01.050701+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:15.644254+0000","scrub_duration":267,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":1030,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.826518+0000","last_change":"2024-03-20T09:47:10.015594+0000","last_active":"2024-03-20T09:47:42.826518+0000","last_peered":"2024-03-20T09:47:42.826518+0000","last_clean":"2024-03-20T09:47:42.826518+0000","last_became_active":"2024-03-20T09:46:12.816798+0000","last_became_peered":"2024-03-20T09:46:12.816798+0000","last_unstale":"2024-03-20T09:47:42.826518+0000","last_undegraded":"2024-03-20T09:47:42.826518+0000","last_fullsized":"2024-03-20T09:47:42.826518+0000","mapping_epoch":1179,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":1180,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:10.015497+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:45:58.520390+0000","last_clean_scrub_stamp":"2024-03-20T09:47:10.015497+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:31.863180+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"20'137","reported_seq":1280,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.826493+0000","last_change":"2024-03-20T09:46:37.777695+0000","last_active":"2024-03-20T09:47:42.826493+0000","last_peered":"2024-03-20T09:47:42.826493+0000","last_clean":"2024-03-20T09:47:42.826493+0000","last_became_active":"2024-03-20T09:46:12.815891+0000","last_became_peered":"2024-03-20T09:46:12.815891+0000","last_unstale":"2024-03-20T09:47:42.826493+0000","last_undegraded":"2024-03-20T09:47:42.826493+0000","last_fullsized":"2024-03-20T09:47:42.826493+0000","mapping_epoch":1179,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":1180,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:46:37.777576+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:42:42.369236+0000","last_clean_scrub_stamp":"2024-03-20T09:46:37.777576+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:47:54.069393+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"20'137","reported_seq":2003,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.874193+0000","last_change":"2024-03-20T09:47:21.147759+0000","last_active":"2024-03-20T09:47:42.874193+0000","last_peered":"2024-03-20T09:47:42.874193+0000","last_clean":"2024-03-20T09:47:42.874193+0000","last_became_active":"2024-03-20T09:15:34.225022+0000","last_became_peered":"2024-03-20T09:15:34.225022+0000","last_unstale":"2024-03-20T09:47:42.874193+0000","last_undegraded":"2024-03-20T09:47:42.874193+0000","last_fullsized":"2024-03-20T09:47:42.874193+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:21.147640+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:20:12.169658+0000","last_clean_scrub_stamp":"2024-03-20T09:47:21.147640+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:23.014691+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,5],"acting":[6,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"20'137","reported_seq":2559,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.874829+0000","last_change":"2024-03-20T09:47:01.337794+0000","last_active":"2024-03-20T09:47:42.874829+0000","last_peered":"2024-03-20T09:47:42.874829+0000","last_clean":"2024-03-20T09:47:42.874829+0000","last_became_active":"2024-03-20T09:35:19.820396+0000","last_became_peered":"2024-03-20T09:35:19.820396+0000","last_unstale":"2024-03-20T09:47:42.874829+0000","last_undegraded":"2024-03-20T09:47:42.874829+0000","last_fullsized":"2024-03-20T09:47:42.874829+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":3,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:01.337692+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:14:10.560890+0000","last_clean_scrub_stamp":"2024-03-20T09:47:01.337692+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:27.673956+0000","scrub_duration":192,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"20'137","reported_seq":2400,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.830223+0000","last_change":"2024-03-20T09:47:39.279094+0000","last_active":"2024-03-20T09:47:42.830223+0000","last_peered":"2024-03-20T09:47:42.830223+0000","last_clean":"2024-03-20T09:47:42.830223+0000","last_became_active":"2024-03-20T09:44:56.018327+0000","last_became_peered":"2024-03-20T09:44:56.018327+0000","last_unstale":"2024-03-20T09:47:42.830223+0000","last_undegraded":"2024-03-20T09:47:42.830223+0000","last_fullsized":"2024-03-20T09:47:42.830223+0000","mapping_epoch":1149,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":1150,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:39.278991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:47:39.278991+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:50.220826+0000","scrub_duration":231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.e","version":"20'137","reported_seq":1817,"reported_epoch":1204,"state":"active+clean","last_fresh":"2024-03-20T09:47:50.115382+0000","last_change":"2024-03-20T09:47:18.794891+0000","last_active":"2024-03-20T09:47:50.115382+0000","last_peered":"2024-03-20T09:47:50.115382+0000","last_clean":"2024-03-20T09:47:50.115382+0000","last_became_active":"2024-03-20T09:19:54.506134+0000","last_became_peered":"2024-03-20T09:19:54.506134+0000","last_unstale":"2024-03-20T09:47:50.115382+0000","last_undegraded":"2024-03-20T09:47:50.115382+0000","last_fullsized":"2024-03-20T09:47:50.115382+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:18.794773+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:45:49.636483+0000","last_clean_scrub_stamp":"2024-03-20T09:47:18.794773+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:20.939771+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"20'137","reported_seq":2578,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.874741+0000","last_change":"2024-03-20T09:47:28.282024+0000","last_active":"2024-03-20T09:47:42.874741+0000","last_peered":"2024-03-20T09:47:42.874741+0000","last_clean":"2024-03-20T09:47:42.874741+0000","last_became_active":"2024-03-20T09:19:54.524397+0000","last_became_peered":"2024-03-20T09:19:54.524397+0000","last_unstale":"2024-03-20T09:47:42.874741+0000","last_undegraded":"2024-03-20T09:47:42.874741+0000","last_fullsized":"2024-03-20T09:47:42.874741+0000","mapping_epoch":639,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:28.281922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:47:28.281922+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:43.709927+0000","scrub_duration":229,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"570'138","reported_seq":2585,"reported_epoch":1204,"state":"active+clean","last_fresh":"2024-03-20T09:47:50.115269+0000","last_change":"2024-03-20T09:46:59.159617+0000","last_active":"2024-03-20T09:47:50.115269+0000","last_peered":"2024-03-20T09:47:50.115269+0000","last_clean":"2024-03-20T09:47:50.115269+0000","last_became_active":"2024-03-20T09:44:56.021764+0000","last_became_peered":"2024-03-20T09:44:56.021764+0000","last_unstale":"2024-03-20T09:47:50.115269+0000","last_undegraded":"2024-03-20T09:47:50.115269+0000","last_fullsized":"2024-03-20T09:47:50.115269+0000","mapping_epoch":1149,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":1150,"parent":"0.0","parent_split_bits":4,"last_scrub":"570'138","last_scrub_stamp":"2024-03-20T09:46:59.159506+0000","last_deep_scrub":"570'138","last_deep_scrub_stamp":"2024-03-20T09:16:13.725035+0000","last_clean_scrub_stamp":"2024-03-20T09:46:59.159506+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:02.429882+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"20'137","reported_seq":555,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.826505+0000","last_change":"2024-03-20T09:46:12.815458+0000","last_active":"2024-03-20T09:47:42.826505+0000","last_peered":"2024-03-20T09:47:42.826505+0000","last_clean":"2024-03-20T09:47:42.826505+0000","last_became_active":"2024-03-20T09:46:12.815073+0000","last_became_peered":"2024-03-20T09:46:12.815073+0000","last_unstale":"2024-03-20T09:47:42.826505+0000","last_undegraded":"2024-03-20T09:47:42.826505+0000","last_fullsized":"2024-03-20T09:47:42.826505+0000","mapping_epoch":1179,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":1180,"parent":"0.0","parent_split_bits":4,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:46:09.656036+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:41:52.574619+0000","last_clean_scrub_stamp":"2024-03-20T09:46:09.656036+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"20'137","reported_seq":2410,"reported_epoch":1204,"state":"active+clean","last_fresh":"2024-03-20T09:47:49.049976+0000","last_change":"2024-03-20T09:47:01.116616+0000","last_active":"2024-03-20T09:47:49.049976+0000","last_peered":"2024-03-20T09:47:49.049976+0000","last_clean":"2024-03-20T09:47:49.049976+0000","last_became_active":"2024-03-20T09:10:34.096758+0000","last_became_peered":"2024-03-20T09:10:34.096758+0000","last_unstale":"2024-03-20T09:47:49.049976+0000","last_undegraded":"2024-03-20T09:47:49.049976+0000","last_fullsized":"2024-03-20T09:47:49.049976+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:01.116516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:47:01.116516+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:12.143664+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.13","version":"20'137","reported_seq":1847,"reported_epoch":1204,"state":"active+clean","last_fresh":"2024-03-20T09:47:49.614609+0000","last_change":"2024-03-20T09:47:03.072061+0000","last_active":"2024-03-20T09:47:49.614609+0000","last_peered":"2024-03-20T09:47:49.614609+0000","last_clean":"2024-03-20T09:47:49.614609+0000","last_became_active":"2024-03-20T09:45:33.792850+0000","last_became_peered":"2024-03-20T09:45:33.792850+0000","last_unstale":"2024-03-20T09:47:49.614609+0000","last_undegraded":"2024-03-20T09:47:49.614609+0000","last_fullsized":"2024-03-20T09:47:49.614609+0000","mapping_epoch":1164,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:03.071960+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:37:52.359932+0000","last_clean_scrub_stamp":"2024-03-20T09:47:03.071960+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:17.487704+0000","scrub_duration":275,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.10","version":"20'137","reported_seq":1982,"reported_epoch":1204,"state":"active+clean","last_fresh":"2024-03-20T09:47:49.050098+0000","last_change":"2024-03-20T09:47:38.043879+0000","last_active":"2024-03-20T09:47:49.050098+0000","last_peered":"2024-03-20T09:47:49.050098+0000","last_clean":"2024-03-20T09:47:49.050098+0000","last_became_active":"2024-03-20T09:35:19.820595+0000","last_became_peered":"2024-03-20T09:35:19.820595+0000","last_unstale":"2024-03-20T09:47:49.050098+0000","last_undegraded":"2024-03-20T09:47:49.050098+0000","last_fullsized":"2024-03-20T09:47:49.050098+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:38.043732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:47:38.043732+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:49.450295+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"20'137","reported_seq":514,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.825055+0000","last_change":"2024-03-20T09:47:12.865818+0000","last_active":"2024-03-20T09:47:42.825055+0000","last_peered":"2024-03-20T09:47:42.825055+0000","last_clean":"2024-03-20T09:47:42.825055+0000","last_became_active":"2024-03-20T09:44:56.020550+0000","last_became_peered":"2024-03-20T09:44:56.020550+0000","last_unstale":"2024-03-20T09:47:42.825055+0000","last_undegraded":"2024-03-20T09:47:42.825055+0000","last_fullsized":"2024-03-20T09:47:42.825055+0000","mapping_epoch":1149,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":1150,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:12.865640+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:47:12.865640+0000","last_clean_scrub_stamp":"2024-03-20T09:47:12.865640+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:19.108754+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"20'137","reported_seq":1541,"reported_epoch":1204,"state":"active+clean","last_fresh":"2024-03-20T09:47:49.614670+0000","last_change":"2024-03-20T09:46:59.919713+0000","last_active":"2024-03-20T09:47:49.614670+0000","last_peered":"2024-03-20T09:47:49.614670+0000","last_clean":"2024-03-20T09:47:49.614670+0000","last_became_active":"2024-03-20T09:45:33.793676+0000","last_became_peered":"2024-03-20T09:45:33.793676+0000","last_unstale":"2024-03-20T09:47:49.614670+0000","last_undegraded":"2024-03-20T09:47:49.614670+0000","last_fullsized":"2024-03-20T09:47:49.614670+0000","mapping_epoch":1164,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:46:59.919605+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:35:27.264203+0000","last_clean_scrub_stamp":"2024-03-20T09:46:59.919605+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:21.314623+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"20'137","reported_seq":2550,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.874692+0000","last_change":"2024-03-20T09:47:21.949925+0000","last_active":"2024-03-20T09:47:42.874692+0000","last_peered":"2024-03-20T09:47:42.874692+0000","last_clean":"2024-03-20T09:47:42.874692+0000","last_became_active":"2024-03-20T09:35:19.822429+0000","last_became_peered":"2024-03-20T09:35:19.822429+0000","last_unstale":"2024-03-20T09:47:42.874692+0000","last_undegraded":"2024-03-20T09:47:42.874692+0000","last_fullsized":"2024-03-20T09:47:42.874692+0000","mapping_epoch":960,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":961,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:47:21.949796+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:38:01.419884+0000","last_clean_scrub_stamp":"2024-03-20T09:47:21.949796+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:37.511012+0000","scrub_duration":197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.14","version":"20'137","reported_seq":2573,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:43.600996+0000","last_change":"2024-03-20T09:46:49.337362+0000","last_active":"2024-03-20T09:47:43.600996+0000","last_peered":"2024-03-20T09:47:43.600996+0000","last_clean":"2024-03-20T09:47:43.600996+0000","last_became_active":"2024-03-20T09:20:25.349082+0000","last_became_peered":"2024-03-20T09:20:25.349082+0000","last_unstale":"2024-03-20T09:47:43.600996+0000","last_undegraded":"2024-03-20T09:47:43.600996+0000","last_fullsized":"2024-03-20T09:47:43.600996+0000","mapping_epoch":649,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":650,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:46:49.337261+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:41:54.735667+0000","last_clean_scrub_stamp":"2024-03-20T09:46:49.337261+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:47:51.674586+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.15","version":"20'137","reported_seq":1853,"reported_epoch":1203,"state":"active+clean","last_fresh":"2024-03-20T09:47:42.826681+0000","last_change":"2024-03-20T09:46:33.872550+0000","last_active":"2024-03-20T09:47:42.826681+0000","last_peered":"2024-03-20T09:47:42.826681+0000","last_clean":"2024-03-20T09:47:42.826681+0000","last_became_active":"2024-03-20T09:12:14.307581+0000","last_became_peered":"2024-03-20T09:12:14.307581+0000","last_unstale":"2024-03-20T09:47:42.826681+0000","last_undegraded":"2024-03-20T09:47:42.826681+0000","last_fullsized":"2024-03-20T09:47:42.826681+0000","mapping_epoch":491,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:46:33.872418+0000","last_deep_scrub":"20'137","last_deep_scrub_stamp":"2024-03-20T09:34:48.079272+0000","last_clean_scrub_stamp":"2024-03-20T09:46:33.872418+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:47:46.392227+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1a","version":"20'137","reported_seq":1119,"reported_epoch":1204,"state":"active+clean","last_fresh":"2024-03-20T09:47:50.115144+0000","last_change":"2024-03-20T09:46:39.156360+0000","last_active":"2024-03-20T09:47:50.115144+0000","last_peered":"2024-03-20T09:47:50.115144+0000","last_clean":"2024-03-20T09:47:50.115144+0000","last_became_active":"2024-03-20T09:41:47.137425+0000","last_became_peered":"2024-03-20T09:41:47.137425+0000","last_unstale":"2024-03-20T09:47:50.115144+0000","last_undegraded":"2024-03-20T09:47:50.115144+0000","last_fullsized":"2024-03-20T09:47:50.115144+0000","mapping_epoch":1094,"log_start":"0'0","ondisk_log_start":"0'0","created":358,"last_epoch_clean":1095,"parent":"0.0","parent_split_bits":5,"last_scrub":"20'137","last_scrub_stamp":"2024-03-20T09:46:39.156256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-20T08:52:17.955489+0000","last_clean_scrub_stamp":"2024-03-20T09:46:39.156256+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:06.149448+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4248,"ondisk_log_size":4248,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":96,"seq":412316861039,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1110660,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":1109886,"kb_avail":92634492,"statfs":{"total":95995035648,"available":94857719808,"internally_reserved":0,"allocated":778240,"data_stored":453928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":1136523716},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":861,"seq":3697966842064,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313604,"kb_used_data":760,"kb_used_omap":103,"kb_used_meta":312728,"kb_avail":93431548,"statfs":{"total":95995035648,"available":95673905152,"internally_reserved":0,"allocated":778240,"data_stored":453928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":106169,"internal_metadata":320233799},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":80,"seq":343597384317,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":762180,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":761406,"kb_avail":92982972,"statfs":{"total":95995035648,"available":95214563328,"internally_reserved":0,"allocated":778240,"data_stored":453928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":779680192},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":70,"seq":300647711360,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":685828,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93059324,"statfs":{"total":95995035648,"available":95292747776,"internally_reserved":0,"allocated":778240,"data_stored":453928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":701495750},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":60,"seq":257698038404,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1020620,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":1019646,"kb_avail":92724532,"statfs":{"total":95995035648,"available":94949920768,"internally_reserved":0,"allocated":983040,"data_stored":655176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":1044117962},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332745,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":768068,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":766974,"kb_avail":92977084,"statfs":{"total":95995035648,"available":95208534016,"internally_reserved":0,"allocated":1105920,"data_stored":781608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":785381830},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575524,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1191172,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":1190398,"kb_avail":92553980,"statfs":{"total":95995035648,"available":94775275520,"internally_reserved":0,"allocated":778240,"data_stored":453928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1218968006},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359739042,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1047108,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":1046014,"kb_avail":92698044,"statfs":{"total":95995035648,"available":94922797056,"internally_reserved":0,"allocated":1105920,"data_stored":781608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1071118789},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411997,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":852620,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":92892532,"statfs":{"total":95995035648,"available":95121952768,"internally_reserved":0,"allocated":983040,"data_stored":655176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":872085957},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150362,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1141508,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":1140414,"kb_avail":92603644,"statfs":{"total":95995035648,"available":94826131456,"internally_reserved":0,"allocated":1105920,"data_stored":781608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":1167784391},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438954136,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":808004,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":92937148,"statfs":{"total":95995035648,"available":95167639552,"internally_reserved":0,"allocated":778240,"data_stored":453928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":826603973},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":443,"seq":1902670512577,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1005528,"kb_used_data":972,"kb_used_omap":181,"kb_used_meta":1004362,"kb_avail":92739624,"statfs":{"total":95995035648,"available":94965374976,"internally_reserved":0,"allocated":995328,"data_stored":662667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":186131,"internal_metadata":1028466925},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-20T09:47:52.333 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-20T09:47:52.333 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e7e80a6ac9849f0f23432d0d9832f096730308ae shell --fsid 852fbb68-e696-11ee-95c9-87774f69a715 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-03-20T09:47:52.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:52 smithi115 ceph-mon[72476]: 1.15 scrub starts 2024-03-20T09:47:52.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:52 smithi115 ceph-mon[72476]: 1.15 scrub ok 2024-03-20T09:47:52.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:52 smithi115 ceph-mon[72476]: from='client.18006 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:47:52.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:52 smithi122 ceph-mon[73495]: 1.15 scrub starts 2024-03-20T09:47:52.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:52 smithi122 ceph-mon[73495]: 1.15 scrub ok 2024-03-20T09:47:52.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:52 smithi122 ceph-mon[73495]: from='client.18006 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-20T09:47:52.609 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/mon.a/config 2024-03-20T09:47:53.207 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:53 smithi026 ceph-mon[68309]: pgmap v2942: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:53.391 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:53 smithi115 ceph-mon[72476]: pgmap v2942: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:53.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:53 smithi122 ceph-mon[73495]: pgmap v2942: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:54.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:54 smithi115 ceph-mon[72476]: 1.14 scrub starts 2024-03-20T09:47:54.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:54 smithi115 ceph-mon[72476]: 1.14 scrub ok 2024-03-20T09:47:54.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:54 smithi122 ceph-mon[73495]: 1.14 scrub starts 2024-03-20T09:47:54.432 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:54 smithi122 ceph-mon[73495]: 1.14 scrub ok 2024-03-20T09:47:54.456 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:54 smithi026 ceph-mon[68309]: 1.14 scrub starts 2024-03-20T09:47:54.457 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:54 smithi026 ceph-mon[68309]: 1.14 scrub ok 2024-03-20T09:47:55.093 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:55 smithi026 ceph-mon[68309]: 1.2 deep-scrub starts 2024-03-20T09:47:55.093 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:55 smithi026 ceph-mon[68309]: 1.2 deep-scrub ok 2024-03-20T09:47:55.093 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:55 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2622676679' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-20T09:47:55.093 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:55 smithi026 ceph-mon[68309]: from='client.? 172.21.15.26:0/2622676679' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-20T09:47:55.093 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:55 smithi026 ceph-mon[68309]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:47:55.094 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:55 smithi026 ceph-mon[68309]: pgmap v2943: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:55.123 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-20T09:47:55.133 INFO:tasks.cephadm:Teardown begin 2024-03-20T09:47:55.133 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-20T09:47:55.215 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-20T09:47:55.286 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-20T09:47:55.318 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:55 smithi122 ceph-mon[73495]: 1.2 deep-scrub starts 2024-03-20T09:47:55.318 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:55 smithi122 ceph-mon[73495]: 1.2 deep-scrub ok 2024-03-20T09:47:55.318 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:55 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2622676679' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-20T09:47:55.319 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:55 smithi122 ceph-mon[73495]: from='client.? 172.21.15.26:0/2622676679' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-20T09:47:55.319 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:55 smithi122 ceph-mon[73495]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:47:55.319 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:55 smithi122 ceph-mon[73495]: pgmap v2943: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:55.390 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-20T09:47:55.390 DEBUG:teuthology.orchestra.run.smithi026:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-20T09:47:55.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:55 smithi115 ceph-mon[72476]: 1.2 deep-scrub starts 2024-03-20T09:47:55.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:55 smithi115 ceph-mon[72476]: 1.2 deep-scrub ok 2024-03-20T09:47:55.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:55 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2622676679' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-20T09:47:55.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:55 smithi115 ceph-mon[72476]: from='client.? 172.21.15.26:0/2622676679' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-20T09:47:55.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:55 smithi115 ceph-mon[72476]: from='mgr.14148 172.21.15.26:0/3907335130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-20T09:47:55.392 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:55 smithi115 ceph-mon[72476]: pgmap v2943: 31 pgs: 31 active+clean; 577 KiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-20T09:47:55.419 DEBUG:teuthology.orchestra.run.smithi115:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-20T09:47:55.465 DEBUG:teuthology.orchestra.run.smithi122:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-20T09:47:55.487 INFO:tasks.cephadm:Stopping all daemons... 2024-03-20T09:47:55.487 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-20T09:47:55.487 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ceph-852fbb68-e696-11ee-95c9-87774f69a715@mon.a 2024-03-20T09:47:55.799 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:55 smithi026 systemd[1]: Stopping Ceph mon.a for 852fbb68-e696-11ee-95c9-87774f69a715... 2024-03-20T09:47:56.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:55 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mon-a[68286]: 2024-03-20T09:47:55.811+0000 7f6838e56700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-20T09:47:56.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:55 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mon-a[68286]: 2024-03-20T09:47:55.811+0000 7f6838e56700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-20T09:47:57.206 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 20 09:47:56 smithi026 bash[383541]: ceph-852fbb68-e696-11ee-95c9-87774f69a715-mon-a 2024-03-20T09:47:57.231 DEBUG:teuthology.orchestra.run.smithi026:> sudo pkill -f 'journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@mon.a.service' 2024-03-20T09:47:57.289 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-20T09:47:57.290 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-20T09:47:57.290 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-20T09:47:57.290 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-852fbb68-e696-11ee-95c9-87774f69a715@mon.b 2024-03-20T09:47:57.642 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:57 smithi115 systemd[1]: Stopping Ceph mon.b for 852fbb68-e696-11ee-95c9-87774f69a715... 2024-03-20T09:47:57.642 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:57 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mon-b[72452]: 2024-03-20T09:47:57.581+0000 7fe932423700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-20T09:47:57.642 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:57 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mon-b[72452]: 2024-03-20T09:47:57.581+0000 7fe932423700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-20T09:47:58.642 INFO:journalctl@ceph.mon.b.smithi115.stdout:Mar 20 09:47:58 smithi115 bash[224375]: ceph-852fbb68-e696-11ee-95c9-87774f69a715-mon-b 2024-03-20T09:47:58.737 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@mon.b.service' 2024-03-20T09:47:58.779 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-20T09:47:58.779 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-20T09:47:58.779 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-20T09:47:58.779 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-852fbb68-e696-11ee-95c9-87774f69a715@mon.c 2024-03-20T09:47:59.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:58 smithi122 systemd[1]: Stopping Ceph mon.c for 852fbb68-e696-11ee-95c9-87774f69a715... 2024-03-20T09:47:59.682 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:59 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mon-c[73463]: 2024-03-20T09:47:59.338+0000 7fb8319a0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-20T09:47:59.682 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:59 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mon-c[73463]: 2024-03-20T09:47:59.338+0000 7fb8319a0700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-20T09:48:00.182 INFO:journalctl@ceph.mon.c.smithi122.stdout:Mar 20 09:47:59 smithi122 bash[222051]: ceph-852fbb68-e696-11ee-95c9-87774f69a715-mon-c 2024-03-20T09:48:00.212 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@mon.c.service' 2024-03-20T09:48:00.251 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-20T09:48:00.252 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-20T09:48:00.252 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-20T09:48:00.252 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ceph-852fbb68-e696-11ee-95c9-87774f69a715@mgr.y 2024-03-20T09:48:00.535 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 09:48:00 smithi026 systemd[1]: Stopping Ceph mgr.y for 852fbb68-e696-11ee-95c9-87774f69a715... 2024-03-20T09:48:00.535 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 09:48:00 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y[68562]: 2024-03-20T09:48:00.523+0000 7fe62e3c0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-20T09:48:01.206 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 20 09:48:01 smithi026 bash[383686]: ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-y 2024-03-20T09:48:01.391 DEBUG:teuthology.orchestra.run.smithi026:> sudo pkill -f 'journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@mgr.y.service' 2024-03-20T09:48:01.429 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-20T09:48:01.429 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-20T09:48:01.429 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-20T09:48:01.430 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-852fbb68-e696-11ee-95c9-87774f69a715@mgr.x 2024-03-20T09:48:01.892 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 20 09:48:01 smithi115 systemd[1]: Stopping Ceph mgr.x for 852fbb68-e696-11ee-95c9-87774f69a715... 2024-03-20T09:48:02.392 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 20 09:48:02 smithi115 bash[224540]: ceph-852fbb68-e696-11ee-95c9-87774f69a715-mgr-x 2024-03-20T09:48:02.582 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 20 09:48:02 smithi115 systemd[1]: ceph-852fbb68-e696-11ee-95c9-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-03-20T09:48:02.582 INFO:journalctl@ceph.mgr.x.smithi115.stdout:Mar 20 09:48:02 smithi115 systemd[1]: ceph-852fbb68-e696-11ee-95c9-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2024-03-20T09:48:02.584 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@mgr.x.service' 2024-03-20T09:48:02.619 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-20T09:48:02.620 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-20T09:48:02.620 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-20T09:48:02.620 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.0 2024-03-20T09:48:02.956 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 20 09:48:02 smithi026 systemd[1]: Stopping Ceph osd.0 for 852fbb68-e696-11ee-95c9-87774f69a715... 2024-03-20T09:48:02.956 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 20 09:48:02 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-0[86340]: 2024-03-20T09:48:02.913+0000 7f97d8d5c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-20T09:48:02.957 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 20 09:48:02 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-0[86340]: 2024-03-20T09:48:02.913+0000 7f97d8d5c700 -1 osd.0 1204 *** Got signal Terminated *** 2024-03-20T09:48:02.957 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 20 09:48:02 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-0[86340]: 2024-03-20T09:48:02.913+0000 7f97d8d5c700 -1 osd.0 1204 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-20T09:48:09.206 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 20 09:48:08 smithi026 bash[383842]: ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-0 2024-03-20T09:48:11.077 DEBUG:teuthology.orchestra.run.smithi026:> sudo pkill -f 'journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.0.service' 2024-03-20T09:48:11.121 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-20T09:48:11.121 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-20T09:48:11.121 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-20T09:48:11.122 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.1 2024-03-20T09:48:11.436 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 20 09:48:11 smithi026 systemd[1]: Stopping Ceph osd.1 for 852fbb68-e696-11ee-95c9-87774f69a715... 2024-03-20T09:48:11.706 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 20 09:48:11 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-1[94087]: 2024-03-20T09:48:11.448+0000 7efe18792700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-20T09:48:11.706 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 20 09:48:11 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-1[94087]: 2024-03-20T09:48:11.448+0000 7efe18792700 -1 osd.1 1204 *** Got signal Terminated *** 2024-03-20T09:48:11.706 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 20 09:48:11 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-1[94087]: 2024-03-20T09:48:11.448+0000 7efe18792700 -1 osd.1 1204 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-20T09:48:18.206 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 20 09:48:17 smithi026 bash[384109]: ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-1 2024-03-20T09:48:19.649 DEBUG:teuthology.orchestra.run.smithi026:> sudo pkill -f 'journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.1.service' 2024-03-20T09:48:19.695 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-20T09:48:19.695 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-20T09:48:19.695 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-20T09:48:19.695 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.2 2024-03-20T09:48:19.956 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 20 09:48:19 smithi026 systemd[1]: Stopping Ceph osd.2 for 852fbb68-e696-11ee-95c9-87774f69a715... 2024-03-20T09:48:20.456 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 20 09:48:19 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-2[101308]: 2024-03-20T09:48:19.976+0000 7fb67466d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-20T09:48:20.456 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 20 09:48:19 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-2[101308]: 2024-03-20T09:48:19.976+0000 7fb67466d700 -1 osd.2 1204 *** Got signal Terminated *** 2024-03-20T09:48:20.456 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 20 09:48:19 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-2[101308]: 2024-03-20T09:48:19.976+0000 7fb67466d700 -1 osd.2 1204 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-20T09:48:26.206 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 20 09:48:26 smithi026 bash[384370]: ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-2 2024-03-20T09:48:27.786 DEBUG:teuthology.orchestra.run.smithi026:> sudo pkill -f 'journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.2.service' 2024-03-20T09:48:27.834 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-20T09:48:27.834 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-20T09:48:27.834 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-20T09:48:27.834 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.3 2024-03-20T09:48:28.091 INFO:journalctl@ceph.osd.3.smithi026.stdout:Mar 20 09:48:27 smithi026 systemd[1]: Stopping Ceph osd.3 for 852fbb68-e696-11ee-95c9-87774f69a715... 2024-03-20T09:48:28.456 INFO:journalctl@ceph.osd.3.smithi026.stdout:Mar 20 09:48:28 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-3[109291]: 2024-03-20T09:48:28.103+0000 7f4617475700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-20T09:48:28.456 INFO:journalctl@ceph.osd.3.smithi026.stdout:Mar 20 09:48:28 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-3[109291]: 2024-03-20T09:48:28.103+0000 7f4617475700 -1 osd.3 1204 *** Got signal Terminated *** 2024-03-20T09:48:28.456 INFO:journalctl@ceph.osd.3.smithi026.stdout:Mar 20 09:48:28 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-3[109291]: 2024-03-20T09:48:28.103+0000 7f4617475700 -1 osd.3 1204 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-20T09:48:30.456 INFO:journalctl@ceph.osd.3.smithi026.stdout:Mar 20 09:48:30 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-3[109291]: 2024-03-20T09:48:30.204+0000 7f4613a69700 -1 osd.3 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.704074+0000 front 2024-03-20T09:48:05.704151+0000 (oldest deadline 2024-03-20T09:48:29.204279+0000) 2024-03-20T09:48:30.933 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:30 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:30.629+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:48:30.933 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:30 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:30.818+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:48:31.456 INFO:journalctl@ceph.osd.3.smithi026.stdout:Mar 20 09:48:31 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-3[109291]: 2024-03-20T09:48:31.164+0000 7f4613a69700 -1 osd.3 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.704074+0000 front 2024-03-20T09:48:05.704151+0000 (oldest deadline 2024-03-20T09:48:29.204279+0000) 2024-03-20T09:48:31.657 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:31 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:31.308+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:48:31.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:31 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:31.670+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:48:31.932 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:31 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:31.822+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:48:32.142 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:31 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:31.829+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.499239+0000 front 2024-03-20T09:48:06.499385+0000 (oldest deadline 2024-03-20T09:48:31.099761+0000) 2024-03-20T09:48:32.456 INFO:journalctl@ceph.osd.3.smithi026.stdout:Mar 20 09:48:32 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-3[109291]: 2024-03-20T09:48:32.120+0000 7f4613a69700 -1 osd.3 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.704074+0000 front 2024-03-20T09:48:05.704151+0000 (oldest deadline 2024-03-20T09:48:29.204279+0000) 2024-03-20T09:48:32.639 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:32 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:32.275+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:48:32.833 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:32 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:32.520+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.210159+0000 front 2024-03-20T09:48:07.210270+0000 (oldest deadline 2024-03-20T09:48:32.411029+0000) 2024-03-20T09:48:32.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:32 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:32.653+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:48:32.932 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:32 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:32.855+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:48:33.141 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:32 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:32.846+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.499239+0000 front 2024-03-20T09:48:06.499385+0000 (oldest deadline 2024-03-20T09:48:31.099761+0000) 2024-03-20T09:48:33.142 INFO:journalctl@ceph.osd.4.smithi115.stdout:Mar 20 09:48:33 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-4[77898]: 2024-03-20T09:48:33.024+0000 7f4879c10700 -1 osd.4 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:03.125713+0000 front 2024-03-20T09:48:03.125774+0000 (oldest deadline 2024-03-20T09:48:32.526584+0000) 2024-03-20T09:48:33.456 INFO:journalctl@ceph.osd.3.smithi026.stdout:Mar 20 09:48:33 smithi026 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-3[109291]: 2024-03-20T09:48:33.072+0000 7f4613a69700 -1 osd.3 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.704074+0000 front 2024-03-20T09:48:05.704151+0000 (oldest deadline 2024-03-20T09:48:29.204279+0000) 2024-03-20T09:48:33.658 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:33 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:33.260+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:48:33.817 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:33 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:33.534+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.210159+0000 front 2024-03-20T09:48:07.210270+0000 (oldest deadline 2024-03-20T09:48:32.411029+0000) 2024-03-20T09:48:33.932 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:33 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:33.883+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:48:33.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:33 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:33.671+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:48:34.142 INFO:journalctl@ceph.osd.4.smithi115.stdout:Mar 20 09:48:34 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-4[77898]: 2024-03-20T09:48:34.065+0000 7f4879c10700 -1 osd.4 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:03.125713+0000 front 2024-03-20T09:48:03.125774+0000 (oldest deadline 2024-03-20T09:48:32.526584+0000) 2024-03-20T09:48:34.142 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:33 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:33.831+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.499239+0000 front 2024-03-20T09:48:06.499385+0000 (oldest deadline 2024-03-20T09:48:31.099761+0000) 2024-03-20T09:48:34.206 INFO:journalctl@ceph.osd.3.smithi026.stdout:Mar 20 09:48:34 smithi026 bash[384630]: ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-3 2024-03-20T09:48:34.639 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:34 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:34.285+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:48:34.774 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:34 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:34.504+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.210159+0000 front 2024-03-20T09:48:07.210270+0000 (oldest deadline 2024-03-20T09:48:32.411029+0000) 2024-03-20T09:48:34.893 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:34 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:34.653+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:48:35.099 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:34 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:34.784+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.499239+0000 front 2024-03-20T09:48:06.499385+0000 (oldest deadline 2024-03-20T09:48:31.099761+0000) 2024-03-20T09:48:35.182 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:34 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:34.907+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:48:35.392 INFO:journalctl@ceph.osd.4.smithi115.stdout:Mar 20 09:48:35 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-4[77898]: 2024-03-20T09:48:35.113+0000 7f4879c10700 -1 osd.4 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:03.125713+0000 front 2024-03-20T09:48:03.125774+0000 (oldest deadline 2024-03-20T09:48:32.526584+0000) 2024-03-20T09:48:35.622 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:35 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:35.237+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:48:35.738 DEBUG:teuthology.orchestra.run.smithi026:> sudo pkill -f 'journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.3.service' 2024-03-20T09:48:35.746 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:35 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:35.472+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.210159+0000 front 2024-03-20T09:48:07.210270+0000 (oldest deadline 2024-03-20T09:48:32.411029+0000) 2024-03-20T09:48:35.781 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-20T09:48:35.781 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-20T09:48:35.781 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-20T09:48:35.781 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.4 2024-03-20T09:48:35.915 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:35 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:35.636+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:48:36.079 INFO:journalctl@ceph.osd.4.smithi115.stdout:Mar 20 09:48:35 smithi115 systemd[1]: Stopping Ceph osd.4 for 852fbb68-e696-11ee-95c9-87774f69a715... 2024-03-20T09:48:36.079 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:35 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:35.760+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.499239+0000 front 2024-03-20T09:48:06.499385+0000 (oldest deadline 2024-03-20T09:48:31.099761+0000) 2024-03-20T09:48:36.182 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:35 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:35.928+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:48:36.391 INFO:journalctl@ceph.osd.4.smithi115.stdout:Mar 20 09:48:36 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-4[77898]: 2024-03-20T09:48:36.092+0000 7f4881624700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-20T09:48:36.391 INFO:journalctl@ceph.osd.4.smithi115.stdout:Mar 20 09:48:36 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-4[77898]: 2024-03-20T09:48:36.092+0000 7f4881624700 -1 osd.4 1204 *** Got signal Terminated *** 2024-03-20T09:48:36.392 INFO:journalctl@ceph.osd.4.smithi115.stdout:Mar 20 09:48:36 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-4[77898]: 2024-03-20T09:48:36.092+0000 7f4881624700 -1 osd.4 1204 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-20T09:48:36.392 INFO:journalctl@ceph.osd.4.smithi115.stdout:Mar 20 09:48:36 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-4[77898]: 2024-03-20T09:48:36.136+0000 7f4879c10700 -1 osd.4 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:03.125713+0000 front 2024-03-20T09:48:03.125774+0000 (oldest deadline 2024-03-20T09:48:32.526584+0000) 2024-03-20T09:48:36.642 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:36 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:36.241+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:48:36.758 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:36 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:36.431+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.210159+0000 front 2024-03-20T09:48:07.210270+0000 (oldest deadline 2024-03-20T09:48:32.411029+0000) 2024-03-20T09:48:36.902 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:36 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:36.656+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:48:37.141 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:36 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:36.772+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.499239+0000 front 2024-03-20T09:48:06.499385+0000 (oldest deadline 2024-03-20T09:48:31.099761+0000) 2024-03-20T09:48:37.182 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:36 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:36.916+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:48:37.463 INFO:journalctl@ceph.osd.4.smithi115.stdout:Mar 20 09:48:37 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-4[77898]: 2024-03-20T09:48:37.171+0000 7f4879c10700 -1 osd.4 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:03.125713+0000 front 2024-03-20T09:48:03.125774+0000 (oldest deadline 2024-03-20T09:48:32.526584+0000) 2024-03-20T09:48:37.649 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:37 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:37.230+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:48:37.714 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:37 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:37.477+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.210159+0000 front 2024-03-20T09:48:07.210270+0000 (oldest deadline 2024-03-20T09:48:32.411029+0000) 2024-03-20T09:48:37.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:37 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:37.663+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:48:37.932 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:37 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:37.910+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:48:37.932 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:37 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:37.910+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:48:38.121 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:37 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:37.728+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.499239+0000 front 2024-03-20T09:48:06.499385+0000 (oldest deadline 2024-03-20T09:48:31.099761+0000) 2024-03-20T09:48:38.392 INFO:journalctl@ceph.osd.4.smithi115.stdout:Mar 20 09:48:38 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-4[77898]: 2024-03-20T09:48:38.135+0000 7f4879c10700 -1 osd.4 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:03.125713+0000 front 2024-03-20T09:48:03.125774+0000 (oldest deadline 2024-03-20T09:48:32.526584+0000) 2024-03-20T09:48:38.392 INFO:journalctl@ceph.osd.4.smithi115.stdout:Mar 20 09:48:38 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-4[77898]: 2024-03-20T09:48:38.135+0000 7f4879c10700 -1 osd.4 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:12.527221+0000 front 2024-03-20T09:48:12.527236+0000 (oldest deadline 2024-03-20T09:48:37.227142+0000) 2024-03-20T09:48:38.651 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:38 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:38.238+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:48:38.891 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:38 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:38.719+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.499239+0000 front 2024-03-20T09:48:06.499385+0000 (oldest deadline 2024-03-20T09:48:31.099761+0000) 2024-03-20T09:48:38.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:38 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:38.481+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.210159+0000 front 2024-03-20T09:48:07.210270+0000 (oldest deadline 2024-03-20T09:48:32.411029+0000) 2024-03-20T09:48:38.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:38 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:38.481+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:12.411805+0000 front 2024-03-20T09:48:12.411874+0000 (oldest deadline 2024-03-20T09:48:38.311787+0000) 2024-03-20T09:48:38.904 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:38 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:38.665+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:48:39.182 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:38 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:38.919+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:48:39.182 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:38 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:38.919+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:48:39.392 INFO:journalctl@ceph.osd.4.smithi115.stdout:Mar 20 09:48:39 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-4[77898]: 2024-03-20T09:48:39.143+0000 7f4879c10700 -1 osd.4 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:03.125713+0000 front 2024-03-20T09:48:03.125774+0000 (oldest deadline 2024-03-20T09:48:32.526584+0000) 2024-03-20T09:48:39.392 INFO:journalctl@ceph.osd.4.smithi115.stdout:Mar 20 09:48:39 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-4[77898]: 2024-03-20T09:48:39.143+0000 7f4879c10700 -1 osd.4 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:12.527221+0000 front 2024-03-20T09:48:12.527236+0000 (oldest deadline 2024-03-20T09:48:37.227142+0000) 2024-03-20T09:48:39.623 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:39 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:39.195+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:48:39.623 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:48:39 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:48:39.425+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:48:39.892 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:39 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:39.723+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.499239+0000 front 2024-03-20T09:48:06.499385+0000 (oldest deadline 2024-03-20T09:48:31.099761+0000) 2024-03-20T09:48:39.892 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:39 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:39.723+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.800751+0000 front 2024-03-20T09:48:15.800663+0000 (oldest deadline 2024-03-20T09:48:39.300718+0000) 2024-03-20T09:48:39.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:39 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:39.509+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.210159+0000 front 2024-03-20T09:48:07.210270+0000 (oldest deadline 2024-03-20T09:48:32.411029+0000) 2024-03-20T09:48:39.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:39 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:39.509+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:12.411805+0000 front 2024-03-20T09:48:12.411874+0000 (oldest deadline 2024-03-20T09:48:38.311787+0000) 2024-03-20T09:48:39.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:39 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:39.636+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:48:40.391 INFO:journalctl@ceph.osd.4.smithi115.stdout:Mar 20 09:48:40 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-4[77898]: 2024-03-20T09:48:40.135+0000 7f4879c10700 -1 osd.4 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:03.125713+0000 front 2024-03-20T09:48:03.125774+0000 (oldest deadline 2024-03-20T09:48:32.526584+0000) 2024-03-20T09:48:40.392 INFO:journalctl@ceph.osd.4.smithi115.stdout:Mar 20 09:48:40 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-4[77898]: 2024-03-20T09:48:40.135+0000 7f4879c10700 -1 osd.4 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:12.527221+0000 front 2024-03-20T09:48:12.527236+0000 (oldest deadline 2024-03-20T09:48:37.227142+0000) 2024-03-20T09:48:40.392 INFO:journalctl@ceph.osd.5.smithi115.stdout:Mar 20 09:48:39 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-5[139249]: 2024-03-20T09:48:39.957+0000 7f37be85d700 -1 osd.5 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.729033+0000 front 2024-03-20T09:48:13.729135+0000 (oldest deadline 2024-03-20T09:48:39.029095+0000) 2024-03-20T09:48:40.410 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:40 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:40.162+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:48:40.410 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:40 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:40.162+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:16.365409+0000 front 2024-03-20T09:48:16.365395+0000 (oldest deadline 2024-03-20T09:48:39.265360+0000) 2024-03-20T09:48:40.410 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:39 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:39.952+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:48:40.411 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:39 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:39.952+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:48:40.671 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:48:40 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:48:40.423+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:48:40.892 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:40 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:40.742+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.499239+0000 front 2024-03-20T09:48:06.499385+0000 (oldest deadline 2024-03-20T09:48:31.099761+0000) 2024-03-20T09:48:40.892 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:40 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:40.742+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.800751+0000 front 2024-03-20T09:48:15.800663+0000 (oldest deadline 2024-03-20T09:48:39.300718+0000) 2024-03-20T09:48:40.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:40 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:40.557+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.210159+0000 front 2024-03-20T09:48:07.210270+0000 (oldest deadline 2024-03-20T09:48:32.411029+0000) 2024-03-20T09:48:40.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:40 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:40.557+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:12.411805+0000 front 2024-03-20T09:48:12.411874+0000 (oldest deadline 2024-03-20T09:48:38.311787+0000) 2024-03-20T09:48:40.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:40 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:40.685+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:48:40.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:40 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:40.685+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:48:41.392 INFO:journalctl@ceph.osd.5.smithi115.stdout:Mar 20 09:48:40 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-5[139249]: 2024-03-20T09:48:40.940+0000 7f37be85d700 -1 osd.5 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.729033+0000 front 2024-03-20T09:48:13.729135+0000 (oldest deadline 2024-03-20T09:48:39.029095+0000) 2024-03-20T09:48:41.392 INFO:journalctl@ceph.osd.4.smithi115.stdout:Mar 20 09:48:41 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-4[77898]: 2024-03-20T09:48:41.089+0000 7f4879c10700 -1 osd.4 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:03.125713+0000 front 2024-03-20T09:48:03.125774+0000 (oldest deadline 2024-03-20T09:48:32.526584+0000) 2024-03-20T09:48:41.392 INFO:journalctl@ceph.osd.4.smithi115.stdout:Mar 20 09:48:41 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-4[77898]: 2024-03-20T09:48:41.089+0000 7f4879c10700 -1 osd.4 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:12.527221+0000 front 2024-03-20T09:48:12.527236+0000 (oldest deadline 2024-03-20T09:48:37.227142+0000) 2024-03-20T09:48:41.412 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:40 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:40.948+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:48:41.413 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:40 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:40.948+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:48:41.413 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:41 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:41.142+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:48:41.413 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:41 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:41.142+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:16.365409+0000 front 2024-03-20T09:48:16.365395+0000 (oldest deadline 2024-03-20T09:48:39.265360+0000) 2024-03-20T09:48:41.686 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:48:41 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:48:41.426+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:48:41.686 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:41 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:41.644+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:48:41.686 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:41 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:41.644+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:48:41.892 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:41 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:41.731+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.499239+0000 front 2024-03-20T09:48:06.499385+0000 (oldest deadline 2024-03-20T09:48:31.099761+0000) 2024-03-20T09:48:41.892 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:41 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:41.731+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.800751+0000 front 2024-03-20T09:48:15.800663+0000 (oldest deadline 2024-03-20T09:48:39.300718+0000) 2024-03-20T09:48:41.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:41 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:41.600+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.210159+0000 front 2024-03-20T09:48:07.210270+0000 (oldest deadline 2024-03-20T09:48:32.411029+0000) 2024-03-20T09:48:41.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:41 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:41.600+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:12.411805+0000 front 2024-03-20T09:48:12.411874+0000 (oldest deadline 2024-03-20T09:48:38.311787+0000) 2024-03-20T09:48:42.222 INFO:journalctl@ceph.osd.5.smithi115.stdout:Mar 20 09:48:41 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-5[139249]: 2024-03-20T09:48:41.972+0000 7f37be85d700 -1 osd.5 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.729033+0000 front 2024-03-20T09:48:13.729135+0000 (oldest deadline 2024-03-20T09:48:39.029095+0000) 2024-03-20T09:48:42.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:41 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:41.970+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:48:42.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:41 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:41.970+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:48:42.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:42 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:42.161+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:48:42.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:42 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:42.161+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:16.365409+0000 front 2024-03-20T09:48:16.365395+0000 (oldest deadline 2024-03-20T09:48:39.265360+0000) 2024-03-20T09:48:42.583 INFO:journalctl@ceph.osd.4.smithi115.stdout:Mar 20 09:48:42 smithi115 bash[224692]: ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-4 2024-03-20T09:48:42.842 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:42 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:42.716+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.499239+0000 front 2024-03-20T09:48:06.499385+0000 (oldest deadline 2024-03-20T09:48:31.099761+0000) 2024-03-20T09:48:42.842 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:42 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:42.716+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.800751+0000 front 2024-03-20T09:48:15.800663+0000 (oldest deadline 2024-03-20T09:48:39.300718+0000) 2024-03-20T09:48:42.842 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:42 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:42.596+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.210159+0000 front 2024-03-20T09:48:07.210270+0000 (oldest deadline 2024-03-20T09:48:32.411029+0000) 2024-03-20T09:48:42.842 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:42 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:42.596+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:12.411805+0000 front 2024-03-20T09:48:12.411874+0000 (oldest deadline 2024-03-20T09:48:38.311787+0000) 2024-03-20T09:48:42.932 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:48:42 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:48:42.446+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:48:42.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:42 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:42.671+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:48:42.933 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:42 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:42.671+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:48:43.391 INFO:journalctl@ceph.osd.5.smithi115.stdout:Mar 20 09:48:42 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-5[139249]: 2024-03-20T09:48:42.940+0000 7f37be85d700 -1 osd.5 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.729033+0000 front 2024-03-20T09:48:13.729135+0000 (oldest deadline 2024-03-20T09:48:39.029095+0000) 2024-03-20T09:48:43.421 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:43 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:43.210+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:48:43.422 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:43 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:43.210+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:16.365409+0000 front 2024-03-20T09:48:16.365395+0000 (oldest deadline 2024-03-20T09:48:39.265360+0000) 2024-03-20T09:48:43.422 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:42 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:42.990+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:48:43.422 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:42 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:42.990+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:48:43.682 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:48:43 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:48:43.436+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:48:43.682 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:43 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:43.685+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:48:43.683 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:43 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:43.685+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:48:43.821 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:43 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:43.712+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.499239+0000 front 2024-03-20T09:48:06.499385+0000 (oldest deadline 2024-03-20T09:48:31.099761+0000) 2024-03-20T09:48:43.821 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:43 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:43.712+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.800751+0000 front 2024-03-20T09:48:15.800663+0000 (oldest deadline 2024-03-20T09:48:39.300718+0000) 2024-03-20T09:48:43.821 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:43 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:43.555+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.210159+0000 front 2024-03-20T09:48:07.210270+0000 (oldest deadline 2024-03-20T09:48:32.411029+0000) 2024-03-20T09:48:43.822 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:43 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:43.555+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:12.411805+0000 front 2024-03-20T09:48:12.411874+0000 (oldest deadline 2024-03-20T09:48:38.311787+0000) 2024-03-20T09:48:44.107 INFO:journalctl@ceph.osd.5.smithi115.stdout:Mar 20 09:48:43 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-5[139249]: 2024-03-20T09:48:43.902+0000 7f37be85d700 -1 osd.5 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.729033+0000 front 2024-03-20T09:48:13.729135+0000 (oldest deadline 2024-03-20T09:48:39.029095+0000) 2024-03-20T09:48:44.114 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.4.service' 2024-03-20T09:48:44.153 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-20T09:48:44.153 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-20T09:48:44.153 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-20T09:48:44.153 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.5 2024-03-20T09:48:44.199 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:43 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:43.951+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:48:44.199 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:43 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:43.951+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:48:44.391 INFO:journalctl@ceph.osd.5.smithi115.stdout:Mar 20 09:48:44 smithi115 systemd[1]: Stopping Ceph osd.5 for 852fbb68-e696-11ee-95c9-87774f69a715... 2024-03-20T09:48:44.682 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:48:44 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:48:44.420+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:48:44.682 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:44 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:44.213+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:48:44.683 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:44 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:44.213+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:16.365409+0000 front 2024-03-20T09:48:16.365395+0000 (oldest deadline 2024-03-20T09:48:39.265360+0000) 2024-03-20T09:48:44.891 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:44 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:44.683+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.499239+0000 front 2024-03-20T09:48:06.499385+0000 (oldest deadline 2024-03-20T09:48:31.099761+0000) 2024-03-20T09:48:44.891 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:44 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:44.683+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.800751+0000 front 2024-03-20T09:48:15.800663+0000 (oldest deadline 2024-03-20T09:48:39.300718+0000) 2024-03-20T09:48:44.892 INFO:journalctl@ceph.osd.5.smithi115.stdout:Mar 20 09:48:44 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-5[139249]: 2024-03-20T09:48:44.486+0000 7f37c2269700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-20T09:48:44.892 INFO:journalctl@ceph.osd.5.smithi115.stdout:Mar 20 09:48:44 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-5[139249]: 2024-03-20T09:48:44.486+0000 7f37c2269700 -1 osd.5 1204 *** Got signal Terminated *** 2024-03-20T09:48:44.892 INFO:journalctl@ceph.osd.5.smithi115.stdout:Mar 20 09:48:44 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-5[139249]: 2024-03-20T09:48:44.486+0000 7f37c2269700 -1 osd.5 1204 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-20T09:48:44.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:44 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:44.578+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.210159+0000 front 2024-03-20T09:48:07.210270+0000 (oldest deadline 2024-03-20T09:48:32.411029+0000) 2024-03-20T09:48:44.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:44 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:44.578+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:12.411805+0000 front 2024-03-20T09:48:12.411874+0000 (oldest deadline 2024-03-20T09:48:38.311787+0000) 2024-03-20T09:48:44.962 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:44 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:44.701+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:48:44.962 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:44 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:44.701+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:48:45.392 INFO:journalctl@ceph.osd.5.smithi115.stdout:Mar 20 09:48:44 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-5[139249]: 2024-03-20T09:48:44.914+0000 7f37be85d700 -1 osd.5 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.729033+0000 front 2024-03-20T09:48:13.729135+0000 (oldest deadline 2024-03-20T09:48:39.029095+0000) 2024-03-20T09:48:45.422 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:45 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:45.202+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:48:45.422 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:45 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:45.202+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:16.365409+0000 front 2024-03-20T09:48:16.365395+0000 (oldest deadline 2024-03-20T09:48:39.265360+0000) 2024-03-20T09:48:45.423 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:44 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:44.976+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:48:45.423 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:44 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:44.977+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:48:45.682 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:48:45 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:48:45.436+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:48:45.891 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:45 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:45.655+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.499239+0000 front 2024-03-20T09:48:06.499385+0000 (oldest deadline 2024-03-20T09:48:31.099761+0000) 2024-03-20T09:48:45.892 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:45 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:45.655+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.800751+0000 front 2024-03-20T09:48:15.800663+0000 (oldest deadline 2024-03-20T09:48:39.300718+0000) 2024-03-20T09:48:45.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:45 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:45.617+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.210159+0000 front 2024-03-20T09:48:07.210270+0000 (oldest deadline 2024-03-20T09:48:32.411029+0000) 2024-03-20T09:48:45.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:45 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:45.617+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:12.411805+0000 front 2024-03-20T09:48:12.411874+0000 (oldest deadline 2024-03-20T09:48:38.311787+0000) 2024-03-20T09:48:45.965 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:45 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:45.699+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:48:45.966 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:45 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:45.699+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:48:46.238 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:45 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:45.979+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:48:46.238 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:45 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:45.980+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:48:46.392 INFO:journalctl@ceph.osd.5.smithi115.stdout:Mar 20 09:48:45 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-5[139249]: 2024-03-20T09:48:45.922+0000 7f37be85d700 -1 osd.5 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.729033+0000 front 2024-03-20T09:48:13.729135+0000 (oldest deadline 2024-03-20T09:48:39.029095+0000) 2024-03-20T09:48:46.644 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:48:46 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:48:46.397+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:48:46.644 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:46 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:46.252+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:48:46.644 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:46 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:46.252+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:16.365409+0000 front 2024-03-20T09:48:16.365395+0000 (oldest deadline 2024-03-20T09:48:39.265360+0000) 2024-03-20T09:48:46.885 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:46 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:46.631+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.499239+0000 front 2024-03-20T09:48:06.499385+0000 (oldest deadline 2024-03-20T09:48:31.099761+0000) 2024-03-20T09:48:46.885 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:46 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:46.631+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.800751+0000 front 2024-03-20T09:48:15.800663+0000 (oldest deadline 2024-03-20T09:48:39.300718+0000) 2024-03-20T09:48:46.885 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:46 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:46.568+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.210159+0000 front 2024-03-20T09:48:07.210270+0000 (oldest deadline 2024-03-20T09:48:32.411029+0000) 2024-03-20T09:48:46.885 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:46 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:46.568+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:12.411805+0000 front 2024-03-20T09:48:12.411874+0000 (oldest deadline 2024-03-20T09:48:38.311787+0000) 2024-03-20T09:48:46.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:46 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:46.658+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:48:46.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:46 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:46.658+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:48:47.141 INFO:journalctl@ceph.osd.5.smithi115.stdout:Mar 20 09:48:46 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-5[139249]: 2024-03-20T09:48:46.898+0000 7f37be85d700 -1 osd.5 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.729033+0000 front 2024-03-20T09:48:13.729135+0000 (oldest deadline 2024-03-20T09:48:39.029095+0000) 2024-03-20T09:48:47.204 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:46 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:46.964+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:48:47.204 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:46 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:46.964+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:48:47.668 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:48:47 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:48:47.407+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:48:47.668 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:48:47 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:48:47.407+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.170023+0000 front 2024-03-20T09:48:24.170175+0000 (oldest deadline 2024-03-20T09:48:47.070022+0000) 2024-03-20T09:48:47.669 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:47 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:47.218+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:48:47.669 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:47 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:47.218+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:16.365409+0000 front 2024-03-20T09:48:16.365395+0000 (oldest deadline 2024-03-20T09:48:39.265360+0000) 2024-03-20T09:48:47.889 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:47 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:47.529+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.210159+0000 front 2024-03-20T09:48:07.210270+0000 (oldest deadline 2024-03-20T09:48:32.411029+0000) 2024-03-20T09:48:47.890 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:47 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:47.529+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:12.411805+0000 front 2024-03-20T09:48:12.411874+0000 (oldest deadline 2024-03-20T09:48:38.311787+0000) 2024-03-20T09:48:47.890 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:47 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:47.631+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.499239+0000 front 2024-03-20T09:48:06.499385+0000 (oldest deadline 2024-03-20T09:48:31.099761+0000) 2024-03-20T09:48:47.890 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:47 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:47.631+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.800751+0000 front 2024-03-20T09:48:15.800663+0000 (oldest deadline 2024-03-20T09:48:39.300718+0000) 2024-03-20T09:48:47.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:47 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:47.682+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:48:47.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:47 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:47.682+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:48:48.141 INFO:journalctl@ceph.osd.5.smithi115.stdout:Mar 20 09:48:47 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-5[139249]: 2024-03-20T09:48:47.903+0000 7f37be85d700 -1 osd.5 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.729033+0000 front 2024-03-20T09:48:13.729135+0000 (oldest deadline 2024-03-20T09:48:39.029095+0000) 2024-03-20T09:48:48.374 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:48 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:48.187+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:48:48.374 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:48 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:48.187+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:16.365409+0000 front 2024-03-20T09:48:16.365395+0000 (oldest deadline 2024-03-20T09:48:39.265360+0000) 2024-03-20T09:48:48.375 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:48 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:48.014+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:48:48.375 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:48 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:48.014+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:48:48.656 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:48:48 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:48:48.388+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:48:48.656 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:48:48 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:48:48.388+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.170023+0000 front 2024-03-20T09:48:24.170175+0000 (oldest deadline 2024-03-20T09:48:47.070022+0000) 2024-03-20T09:48:48.892 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:48 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:48.665+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.499239+0000 front 2024-03-20T09:48:06.499385+0000 (oldest deadline 2024-03-20T09:48:31.099761+0000) 2024-03-20T09:48:48.892 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:48 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:48.665+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.800751+0000 front 2024-03-20T09:48:15.800663+0000 (oldest deadline 2024-03-20T09:48:39.300718+0000) 2024-03-20T09:48:48.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:48 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:48.576+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.210159+0000 front 2024-03-20T09:48:07.210270+0000 (oldest deadline 2024-03-20T09:48:32.411029+0000) 2024-03-20T09:48:48.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:48 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:48.576+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:12.411805+0000 front 2024-03-20T09:48:12.411874+0000 (oldest deadline 2024-03-20T09:48:38.311787+0000) 2024-03-20T09:48:48.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:48 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:48.576+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:23.013164+0000 front 2024-03-20T09:48:23.013151+0000 (oldest deadline 2024-03-20T09:48:47.713097+0000) 2024-03-20T09:48:48.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:48 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:48.670+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:48:48.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:48 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:48.670+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:48:48.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:48 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:48.670+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:48:49.331 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:49 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:49.161+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:48:49.331 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:49 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:49.161+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:16.365409+0000 front 2024-03-20T09:48:16.365395+0000 (oldest deadline 2024-03-20T09:48:39.265360+0000) 2024-03-20T09:48:49.331 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:49 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:49.026+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:48:49.331 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:49 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:49.026+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:48:49.331 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:49 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:49.026+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:48:49.391 INFO:journalctl@ceph.osd.5.smithi115.stdout:Mar 20 09:48:48 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-5[139249]: 2024-03-20T09:48:48.935+0000 7f37be85d700 -1 osd.5 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.729033+0000 front 2024-03-20T09:48:13.729135+0000 (oldest deadline 2024-03-20T09:48:39.029095+0000) 2024-03-20T09:48:49.617 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:48:49 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:48:49.345+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:48:49.617 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:48:49 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:48:49.345+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.170023+0000 front 2024-03-20T09:48:24.170175+0000 (oldest deadline 2024-03-20T09:48:47.070022+0000) 2024-03-20T09:48:49.891 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:49 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:49.660+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.499239+0000 front 2024-03-20T09:48:06.499385+0000 (oldest deadline 2024-03-20T09:48:31.099761+0000) 2024-03-20T09:48:49.892 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:49 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:49.660+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.800751+0000 front 2024-03-20T09:48:15.800663+0000 (oldest deadline 2024-03-20T09:48:39.300718+0000) 2024-03-20T09:48:49.892 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:49 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:49.660+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:23.203257+0000 front 2024-03-20T09:48:23.203164+0000 (oldest deadline 2024-03-20T09:48:49.103302+0000) 2024-03-20T09:48:49.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:49 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:49.536+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.210159+0000 front 2024-03-20T09:48:07.210270+0000 (oldest deadline 2024-03-20T09:48:32.411029+0000) 2024-03-20T09:48:49.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:49 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:49.536+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:12.411805+0000 front 2024-03-20T09:48:12.411874+0000 (oldest deadline 2024-03-20T09:48:38.311787+0000) 2024-03-20T09:48:49.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:49 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:49.536+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:23.013164+0000 front 2024-03-20T09:48:23.013151+0000 (oldest deadline 2024-03-20T09:48:47.713097+0000) 2024-03-20T09:48:49.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:49 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:49.631+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:48:49.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:49 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:49.631+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:48:49.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:49 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:49.631+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:48:50.287 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:50 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:50.044+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:48:50.287 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:50 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:50.044+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:48:50.287 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:50 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:50.044+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:48:50.287 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:50 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:50.151+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:48:50.287 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:50 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:50.151+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:16.365409+0000 front 2024-03-20T09:48:16.365395+0000 (oldest deadline 2024-03-20T09:48:39.265360+0000) 2024-03-20T09:48:50.538 INFO:journalctl@ceph.osd.5.smithi115.stdout:Mar 20 09:48:50 smithi115 bash[224952]: ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-5 2024-03-20T09:48:50.583 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:48:50 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:48:50.301+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:48:50.583 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:48:50 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:48:50.301+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.170023+0000 front 2024-03-20T09:48:24.170175+0000 (oldest deadline 2024-03-20T09:48:47.070022+0000) 2024-03-20T09:48:50.891 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:50 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:50.673+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.499239+0000 front 2024-03-20T09:48:06.499385+0000 (oldest deadline 2024-03-20T09:48:31.099761+0000) 2024-03-20T09:48:50.892 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:50 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:50.673+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.800751+0000 front 2024-03-20T09:48:15.800663+0000 (oldest deadline 2024-03-20T09:48:39.300718+0000) 2024-03-20T09:48:50.892 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:50 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:50.673+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:23.203257+0000 front 2024-03-20T09:48:23.203164+0000 (oldest deadline 2024-03-20T09:48:49.103302+0000) 2024-03-20T09:48:50.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:50 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:50.552+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.210159+0000 front 2024-03-20T09:48:07.210270+0000 (oldest deadline 2024-03-20T09:48:32.411029+0000) 2024-03-20T09:48:50.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:50 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:50.552+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:12.411805+0000 front 2024-03-20T09:48:12.411874+0000 (oldest deadline 2024-03-20T09:48:38.311787+0000) 2024-03-20T09:48:50.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:50 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:50.552+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:23.013164+0000 front 2024-03-20T09:48:23.013151+0000 (oldest deadline 2024-03-20T09:48:47.713097+0000) 2024-03-20T09:48:50.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:50 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:50.598+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:48:50.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:50 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:50.598+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:48:50.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:50 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:50.598+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:48:51.291 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:50 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:50.996+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:48:51.291 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:50 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:50.996+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:48:51.291 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:50 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:50.996+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:48:51.292 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:51 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:51.129+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:48:51.292 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:51 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:51.129+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:16.365409+0000 front 2024-03-20T09:48:16.365395+0000 (oldest deadline 2024-03-20T09:48:39.265360+0000) 2024-03-20T09:48:51.292 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:51 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:51.129+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.367415+0000 front 2024-03-20T09:48:24.367214+0000 (oldest deadline 2024-03-20T09:48:50.167597+0000) 2024-03-20T09:48:51.563 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:48:51 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:48:51.305+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:48:51.564 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:48:51 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:48:51.305+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.170023+0000 front 2024-03-20T09:48:24.170175+0000 (oldest deadline 2024-03-20T09:48:47.070022+0000) 2024-03-20T09:48:51.832 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:51 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:51.719+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.499239+0000 front 2024-03-20T09:48:06.499385+0000 (oldest deadline 2024-03-20T09:48:31.099761+0000) 2024-03-20T09:48:51.832 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:51 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:51.719+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.800751+0000 front 2024-03-20T09:48:15.800663+0000 (oldest deadline 2024-03-20T09:48:39.300718+0000) 2024-03-20T09:48:51.832 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:51 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:51.719+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:23.203257+0000 front 2024-03-20T09:48:23.203164+0000 (oldest deadline 2024-03-20T09:48:49.103302+0000) 2024-03-20T09:48:51.832 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:51 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:51.545+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.210159+0000 front 2024-03-20T09:48:07.210270+0000 (oldest deadline 2024-03-20T09:48:32.411029+0000) 2024-03-20T09:48:51.832 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:51 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:51.545+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:12.411805+0000 front 2024-03-20T09:48:12.411874+0000 (oldest deadline 2024-03-20T09:48:38.311787+0000) 2024-03-20T09:48:51.832 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:51 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:51.545+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:23.013164+0000 front 2024-03-20T09:48:23.013151+0000 (oldest deadline 2024-03-20T09:48:47.713097+0000) 2024-03-20T09:48:51.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:51 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:51.578+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:48:51.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:51 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:51.578+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:48:51.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:51 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:51.578+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:48:52.100 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.5.service' 2024-03-20T09:48:52.140 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-20T09:48:52.140 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-20T09:48:52.140 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-20T09:48:52.140 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.6 2024-03-20T09:48:52.251 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:52 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:52.013+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:48:52.251 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:52 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:52.013+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:48:52.252 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:52 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:52.013+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:48:52.252 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:52 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:52.082+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:48:52.252 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:52 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:52.082+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:16.365409+0000 front 2024-03-20T09:48:16.365395+0000 (oldest deadline 2024-03-20T09:48:39.265360+0000) 2024-03-20T09:48:52.252 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:52 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:52.082+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.367415+0000 front 2024-03-20T09:48:24.367214+0000 (oldest deadline 2024-03-20T09:48:50.167597+0000) 2024-03-20T09:48:52.392 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:52 smithi115 systemd[1]: Stopping Ceph osd.6 for 852fbb68-e696-11ee-95c9-87774f69a715... 2024-03-20T09:48:52.588 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:48:52 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:48:52.265+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:48:52.588 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:48:52 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:48:52.265+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.170023+0000 front 2024-03-20T09:48:24.170175+0000 (oldest deadline 2024-03-20T09:48:47.070022+0000) 2024-03-20T09:48:52.737 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:52 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:52.445+0000 7ff2bf553700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-20T09:48:52.737 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:52 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:52.445+0000 7ff2bf553700 -1 osd.6 1204 *** Got signal Terminated *** 2024-03-20T09:48:52.737 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:52 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:52.445+0000 7ff2bf553700 -1 osd.6 1204 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-20T09:48:52.737 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:52 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:52.583+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.210159+0000 front 2024-03-20T09:48:07.210270+0000 (oldest deadline 2024-03-20T09:48:32.411029+0000) 2024-03-20T09:48:52.737 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:52 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:52.583+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:12.411805+0000 front 2024-03-20T09:48:12.411874+0000 (oldest deadline 2024-03-20T09:48:38.311787+0000) 2024-03-20T09:48:52.737 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:52 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:52.583+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:23.013164+0000 front 2024-03-20T09:48:23.013151+0000 (oldest deadline 2024-03-20T09:48:47.713097+0000) 2024-03-20T09:48:52.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:52 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:52.602+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:48:52.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:52 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:52.602+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:48:52.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:52 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:52.602+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:48:53.141 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:52 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:52.751+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.499239+0000 front 2024-03-20T09:48:06.499385+0000 (oldest deadline 2024-03-20T09:48:31.099761+0000) 2024-03-20T09:48:53.142 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:52 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:52.751+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.800751+0000 front 2024-03-20T09:48:15.800663+0000 (oldest deadline 2024-03-20T09:48:39.300718+0000) 2024-03-20T09:48:53.142 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:52 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:52.751+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:23.203257+0000 front 2024-03-20T09:48:23.203164+0000 (oldest deadline 2024-03-20T09:48:49.103302+0000) 2024-03-20T09:48:53.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:48:53 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:48:53.222+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:48:53.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:48:53 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:48:53.222+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.170023+0000 front 2024-03-20T09:48:24.170175+0000 (oldest deadline 2024-03-20T09:48:47.070022+0000) 2024-03-20T09:48:53.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:52 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:52.993+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:48:53.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:52 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:52.993+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:48:53.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:52 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:52.993+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:48:53.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:53 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:53.113+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:48:53.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:53 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:53.113+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:16.365409+0000 front 2024-03-20T09:48:16.365395+0000 (oldest deadline 2024-03-20T09:48:39.265360+0000) 2024-03-20T09:48:53.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:53 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:53.113+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.367415+0000 front 2024-03-20T09:48:24.367214+0000 (oldest deadline 2024-03-20T09:48:50.167597+0000) 2024-03-20T09:48:53.892 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:53 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:53.744+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.499239+0000 front 2024-03-20T09:48:06.499385+0000 (oldest deadline 2024-03-20T09:48:31.099761+0000) 2024-03-20T09:48:53.892 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:53 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:53.744+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.800751+0000 front 2024-03-20T09:48:15.800663+0000 (oldest deadline 2024-03-20T09:48:39.300718+0000) 2024-03-20T09:48:53.892 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:53 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:53.744+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:23.203257+0000 front 2024-03-20T09:48:23.203164+0000 (oldest deadline 2024-03-20T09:48:49.103302+0000) 2024-03-20T09:48:53.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:53 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:53.600+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.210159+0000 front 2024-03-20T09:48:07.210270+0000 (oldest deadline 2024-03-20T09:48:32.411029+0000) 2024-03-20T09:48:53.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:53 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:53.600+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:12.411805+0000 front 2024-03-20T09:48:12.411874+0000 (oldest deadline 2024-03-20T09:48:38.311787+0000) 2024-03-20T09:48:53.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:53 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:53.600+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:23.013164+0000 front 2024-03-20T09:48:23.013151+0000 (oldest deadline 2024-03-20T09:48:47.713097+0000) 2024-03-20T09:48:53.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:53 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:53.590+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:48:53.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:53 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:53.590+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:48:53.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:53 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:53.590+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:48:54.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:48:54 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:48:54.240+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:48:54.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:48:54 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:48:54.240+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.170023+0000 front 2024-03-20T09:48:24.170175+0000 (oldest deadline 2024-03-20T09:48:47.070022+0000) 2024-03-20T09:48:54.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:54 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:54.155+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:48:54.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:54 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:54.155+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:16.365409+0000 front 2024-03-20T09:48:16.365395+0000 (oldest deadline 2024-03-20T09:48:39.265360+0000) 2024-03-20T09:48:54.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:54 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:54.155+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.367415+0000 front 2024-03-20T09:48:24.367214+0000 (oldest deadline 2024-03-20T09:48:50.167597+0000) 2024-03-20T09:48:54.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:53 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:53.995+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:48:54.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:53 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:53.995+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:48:54.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:53 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:53.995+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:48:54.891 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:54 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:54.699+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.499239+0000 front 2024-03-20T09:48:06.499385+0000 (oldest deadline 2024-03-20T09:48:31.099761+0000) 2024-03-20T09:48:54.892 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:54 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:54.699+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.800751+0000 front 2024-03-20T09:48:15.800663+0000 (oldest deadline 2024-03-20T09:48:39.300718+0000) 2024-03-20T09:48:54.892 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:54 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:54.699+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:23.203257+0000 front 2024-03-20T09:48:23.203164+0000 (oldest deadline 2024-03-20T09:48:49.103302+0000) 2024-03-20T09:48:54.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:54 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:54.584+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.210159+0000 front 2024-03-20T09:48:07.210270+0000 (oldest deadline 2024-03-20T09:48:32.411029+0000) 2024-03-20T09:48:54.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:54 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:54.584+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:12.411805+0000 front 2024-03-20T09:48:12.411874+0000 (oldest deadline 2024-03-20T09:48:38.311787+0000) 2024-03-20T09:48:54.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:54 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:54.584+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:23.013164+0000 front 2024-03-20T09:48:23.013151+0000 (oldest deadline 2024-03-20T09:48:47.713097+0000) 2024-03-20T09:48:54.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:54 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:54.572+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:48:54.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:54 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:54.572+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:48:54.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:54 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:54.572+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:48:55.230 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:55 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:55.126+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:48:55.231 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:55 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:55.126+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:16.365409+0000 front 2024-03-20T09:48:16.365395+0000 (oldest deadline 2024-03-20T09:48:39.265360+0000) 2024-03-20T09:48:55.231 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:55 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:55.126+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.367415+0000 front 2024-03-20T09:48:24.367214+0000 (oldest deadline 2024-03-20T09:48:50.167597+0000) 2024-03-20T09:48:55.231 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:54 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:54.952+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:48:55.231 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:54 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:54.952+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:48:55.231 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:54 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:54.952+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:48:55.558 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:48:55 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:48:55.245+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:48:55.558 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:48:55 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:48:55.245+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.170023+0000 front 2024-03-20T09:48:24.170175+0000 (oldest deadline 2024-03-20T09:48:47.070022+0000) 2024-03-20T09:48:55.891 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:55 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:55.712+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.499239+0000 front 2024-03-20T09:48:06.499385+0000 (oldest deadline 2024-03-20T09:48:31.099761+0000) 2024-03-20T09:48:55.892 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:55 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:55.712+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.800751+0000 front 2024-03-20T09:48:15.800663+0000 (oldest deadline 2024-03-20T09:48:39.300718+0000) 2024-03-20T09:48:55.892 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:55 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:55.712+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:23.203257+0000 front 2024-03-20T09:48:23.203164+0000 (oldest deadline 2024-03-20T09:48:49.103302+0000) 2024-03-20T09:48:55.892 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:55 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:55.712+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:33.105154+0000 front 2024-03-20T09:48:33.105164+0000 (oldest deadline 2024-03-20T09:48:55.405038+0000) 2024-03-20T09:48:55.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:55 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:55.590+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.210159+0000 front 2024-03-20T09:48:07.210270+0000 (oldest deadline 2024-03-20T09:48:32.411029+0000) 2024-03-20T09:48:55.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:55 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:55.590+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:12.411805+0000 front 2024-03-20T09:48:12.411874+0000 (oldest deadline 2024-03-20T09:48:38.311787+0000) 2024-03-20T09:48:55.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:55 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:55.590+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:23.013164+0000 front 2024-03-20T09:48:23.013151+0000 (oldest deadline 2024-03-20T09:48:47.713097+0000) 2024-03-20T09:48:55.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:55 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:55.590+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.314749+0000 front 2024-03-20T09:48:32.314928+0000 (oldest deadline 2024-03-20T09:48:54.614729+0000) 2024-03-20T09:48:55.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:55 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:55.573+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:48:55.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:55 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:55.573+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:48:55.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:55 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:55.573+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:48:56.248 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:56 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:56.124+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:48:56.248 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:56 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:56.124+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:16.365409+0000 front 2024-03-20T09:48:16.365395+0000 (oldest deadline 2024-03-20T09:48:39.265360+0000) 2024-03-20T09:48:56.248 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:56 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:56.124+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.367415+0000 front 2024-03-20T09:48:24.367214+0000 (oldest deadline 2024-03-20T09:48:50.167597+0000) 2024-03-20T09:48:56.249 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:55 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:55.995+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:48:56.249 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:55 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:55.995+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:48:56.249 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:55 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:55.995+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:48:56.596 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:48:56 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:48:56.261+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:48:56.596 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:48:56 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:48:56.261+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.170023+0000 front 2024-03-20T09:48:24.170175+0000 (oldest deadline 2024-03-20T09:48:47.070022+0000) 2024-03-20T09:48:56.892 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:56 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:56.697+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.499239+0000 front 2024-03-20T09:48:06.499385+0000 (oldest deadline 2024-03-20T09:48:31.099761+0000) 2024-03-20T09:48:56.892 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:56 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:56.697+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.800751+0000 front 2024-03-20T09:48:15.800663+0000 (oldest deadline 2024-03-20T09:48:39.300718+0000) 2024-03-20T09:48:56.892 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:56 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:56.697+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:23.203257+0000 front 2024-03-20T09:48:23.203164+0000 (oldest deadline 2024-03-20T09:48:49.103302+0000) 2024-03-20T09:48:56.892 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:56 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6[92910]: 2024-03-20T09:48:56.697+0000 7ff2b7b3f700 -1 osd.6 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:33.105154+0000 front 2024-03-20T09:48:33.105164+0000 (oldest deadline 2024-03-20T09:48:55.405038+0000) 2024-03-20T09:48:56.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:56 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:56.599+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.210159+0000 front 2024-03-20T09:48:07.210270+0000 (oldest deadline 2024-03-20T09:48:32.411029+0000) 2024-03-20T09:48:56.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:56 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:56.599+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:12.411805+0000 front 2024-03-20T09:48:12.411874+0000 (oldest deadline 2024-03-20T09:48:38.311787+0000) 2024-03-20T09:48:56.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:56 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:56.599+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:23.013164+0000 front 2024-03-20T09:48:23.013151+0000 (oldest deadline 2024-03-20T09:48:47.713097+0000) 2024-03-20T09:48:56.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:56 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:56.599+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.314749+0000 front 2024-03-20T09:48:32.314928+0000 (oldest deadline 2024-03-20T09:48:54.614729+0000) 2024-03-20T09:48:56.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:56 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:56.609+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:48:56.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:56 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:56.609+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:48:56.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:56 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:56.609+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:48:57.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:48:57 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:48:57.234+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:48:57.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:48:57 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:48:57.234+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.170023+0000 front 2024-03-20T09:48:24.170175+0000 (oldest deadline 2024-03-20T09:48:47.070022+0000) 2024-03-20T09:48:57.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:57 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:57.035+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:48:57.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:57 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:57.035+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:48:57.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:57 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:57.035+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:48:57.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:57 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:57.156+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:48:57.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:57 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:57.156+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:16.365409+0000 front 2024-03-20T09:48:16.365395+0000 (oldest deadline 2024-03-20T09:48:39.265360+0000) 2024-03-20T09:48:57.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:57 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:57.156+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.367415+0000 front 2024-03-20T09:48:24.367214+0000 (oldest deadline 2024-03-20T09:48:50.167597+0000) 2024-03-20T09:48:57.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:57 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:57.156+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:31.868823+0000 front 2024-03-20T09:48:31.868814+0000 (oldest deadline 2024-03-20T09:48:56.469169+0000) 2024-03-20T09:48:57.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:57 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:57.625+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.210159+0000 front 2024-03-20T09:48:07.210270+0000 (oldest deadline 2024-03-20T09:48:32.411029+0000) 2024-03-20T09:48:57.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:57 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:57.625+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:12.411805+0000 front 2024-03-20T09:48:12.411874+0000 (oldest deadline 2024-03-20T09:48:38.311787+0000) 2024-03-20T09:48:57.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:57 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:57.625+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:23.013164+0000 front 2024-03-20T09:48:23.013151+0000 (oldest deadline 2024-03-20T09:48:47.713097+0000) 2024-03-20T09:48:57.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:57 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:57.625+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.314749+0000 front 2024-03-20T09:48:32.314928+0000 (oldest deadline 2024-03-20T09:48:54.614729+0000) 2024-03-20T09:48:57.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:57 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:57.626+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:48:57.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:57 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:57.626+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:48:57.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:57 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:57.626+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:48:58.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:48:58 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:48:58.257+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:48:58.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:48:58 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:48:58.257+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.170023+0000 front 2024-03-20T09:48:24.170175+0000 (oldest deadline 2024-03-20T09:48:47.070022+0000) 2024-03-20T09:48:58.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:58 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:58.158+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:48:58.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:58 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:58.158+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:16.365409+0000 front 2024-03-20T09:48:16.365395+0000 (oldest deadline 2024-03-20T09:48:39.265360+0000) 2024-03-20T09:48:58.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:58 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:58.158+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.367415+0000 front 2024-03-20T09:48:24.367214+0000 (oldest deadline 2024-03-20T09:48:50.167597+0000) 2024-03-20T09:48:58.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:58 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:58.158+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:31.868823+0000 front 2024-03-20T09:48:31.868814+0000 (oldest deadline 2024-03-20T09:48:56.469169+0000) 2024-03-20T09:48:58.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:58 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:58.073+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:48:58.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:58 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:58.073+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:48:58.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:58 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:58.073+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:48:58.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:58 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:58.073+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:48:58.641 INFO:journalctl@ceph.osd.6.smithi115.stdout:Mar 20 09:48:58 smithi115 bash[225219]: ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-6 2024-03-20T09:48:58.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:58 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:58.674+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:48:58.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:58 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:58.674+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:48:58.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:58 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:58.674+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:48:58.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:58 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:58.674+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.580362+0000 front 2024-03-20T09:48:28.580387+0000 (oldest deadline 2024-03-20T09:48:57.980977+0000) 2024-03-20T09:48:59.044 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:58 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:58.662+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.210159+0000 front 2024-03-20T09:48:07.210270+0000 (oldest deadline 2024-03-20T09:48:32.411029+0000) 2024-03-20T09:48:59.044 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:58 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:58.662+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:12.411805+0000 front 2024-03-20T09:48:12.411874+0000 (oldest deadline 2024-03-20T09:48:38.311787+0000) 2024-03-20T09:48:59.044 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:58 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:58.662+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:23.013164+0000 front 2024-03-20T09:48:23.013151+0000 (oldest deadline 2024-03-20T09:48:47.713097+0000) 2024-03-20T09:48:59.044 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:58 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:58.662+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.314749+0000 front 2024-03-20T09:48:32.314928+0000 (oldest deadline 2024-03-20T09:48:54.614729+0000) 2024-03-20T09:48:59.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:48:59 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:48:59.298+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:48:59.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:48:59 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:48:59.298+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.170023+0000 front 2024-03-20T09:48:24.170175+0000 (oldest deadline 2024-03-20T09:48:47.070022+0000) 2024-03-20T09:48:59.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:59 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:59.199+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:48:59.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:59 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:59.199+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:16.365409+0000 front 2024-03-20T09:48:16.365395+0000 (oldest deadline 2024-03-20T09:48:39.265360+0000) 2024-03-20T09:48:59.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:59 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:59.199+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.367415+0000 front 2024-03-20T09:48:24.367214+0000 (oldest deadline 2024-03-20T09:48:50.167597+0000) 2024-03-20T09:48:59.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:48:59 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:48:59.199+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:31.868823+0000 front 2024-03-20T09:48:31.868814+0000 (oldest deadline 2024-03-20T09:48:56.469169+0000) 2024-03-20T09:48:59.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:59 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:59.078+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:48:59.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:59 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:59.078+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:48:59.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:59 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:59.078+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:48:59.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:48:59 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:48:59.078+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:48:59.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:59 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:59.661+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.210159+0000 front 2024-03-20T09:48:07.210270+0000 (oldest deadline 2024-03-20T09:48:32.411029+0000) 2024-03-20T09:48:59.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:59 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:59.661+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:12.411805+0000 front 2024-03-20T09:48:12.411874+0000 (oldest deadline 2024-03-20T09:48:38.311787+0000) 2024-03-20T09:48:59.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:59 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:59.661+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:23.013164+0000 front 2024-03-20T09:48:23.013151+0000 (oldest deadline 2024-03-20T09:48:47.713097+0000) 2024-03-20T09:48:59.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:48:59 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:48:59.661+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.314749+0000 front 2024-03-20T09:48:32.314928+0000 (oldest deadline 2024-03-20T09:48:54.614729+0000) 2024-03-20T09:48:59.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:59 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:59.654+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:48:59.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:59 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:59.654+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:48:59.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:59 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:59.654+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:48:59.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:48:59 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:48:59.654+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.580362+0000 front 2024-03-20T09:48:28.580387+0000 (oldest deadline 2024-03-20T09:48:57.980977+0000) 2024-03-20T09:48:59.992 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.6.service' 2024-03-20T09:49:00.031 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-20T09:49:00.032 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-20T09:49:00.032 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-20T09:49:00.032 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.7 2024-03-20T09:49:00.334 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:49:00 smithi115 systemd[1]: Stopping Ceph osd.7 for 852fbb68-e696-11ee-95c9-87774f69a715... 2024-03-20T09:49:00.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:00 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:00.104+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:00.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:00 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:00.104+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:00.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:00 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:00.104+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:00.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:00 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:00.104+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:00.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:00 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:00.303+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:49:00.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:00 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:00.303+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.170023+0000 front 2024-03-20T09:48:24.170175+0000 (oldest deadline 2024-03-20T09:48:47.070022+0000) 2024-03-20T09:49:00.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:00 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:00.303+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.771177+0000 front 2024-03-20T09:48:28.771129+0000 (oldest deadline 2024-03-20T09:48:59.371678+0000) 2024-03-20T09:49:00.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:00 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:00.228+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:49:00.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:00 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:00.228+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:16.365409+0000 front 2024-03-20T09:48:16.365395+0000 (oldest deadline 2024-03-20T09:48:39.265360+0000) 2024-03-20T09:49:00.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:00 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:00.228+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.367415+0000 front 2024-03-20T09:48:24.367214+0000 (oldest deadline 2024-03-20T09:48:50.167597+0000) 2024-03-20T09:49:00.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:00 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:00.228+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:31.868823+0000 front 2024-03-20T09:48:31.868814+0000 (oldest deadline 2024-03-20T09:48:56.469169+0000) 2024-03-20T09:49:00.626 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:49:00 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:49:00.347+0000 7ff187ec8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-20T09:49:00.626 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:49:00 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:49:00.347+0000 7ff187ec8700 -1 osd.7 1204 *** Got signal Terminated *** 2024-03-20T09:49:00.627 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:49:00 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:49:00.347+0000 7ff187ec8700 -1 osd.7 1204 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-20T09:49:00.891 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:49:00 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:49:00.640+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.210159+0000 front 2024-03-20T09:48:07.210270+0000 (oldest deadline 2024-03-20T09:48:32.411029+0000) 2024-03-20T09:49:00.891 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:49:00 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:49:00.640+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:12.411805+0000 front 2024-03-20T09:48:12.411874+0000 (oldest deadline 2024-03-20T09:48:38.311787+0000) 2024-03-20T09:49:00.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:49:00 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:49:00.640+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:23.013164+0000 front 2024-03-20T09:48:23.013151+0000 (oldest deadline 2024-03-20T09:48:47.713097+0000) 2024-03-20T09:49:00.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:49:00 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:49:00.640+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.314749+0000 front 2024-03-20T09:48:32.314928+0000 (oldest deadline 2024-03-20T09:48:54.614729+0000) 2024-03-20T09:49:01.090 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:00 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:00.704+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:49:01.090 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:00 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:00.704+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:49:01.091 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:00 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:00.704+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:49:01.091 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:00 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:00.704+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.580362+0000 front 2024-03-20T09:48:28.580387+0000 (oldest deadline 2024-03-20T09:48:57.980977+0000) 2024-03-20T09:49:01.432 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:01 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:01.204+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:49:01.432 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:01 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:01.204+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:16.365409+0000 front 2024-03-20T09:48:16.365395+0000 (oldest deadline 2024-03-20T09:48:39.265360+0000) 2024-03-20T09:49:01.432 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:01 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:01.204+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.367415+0000 front 2024-03-20T09:48:24.367214+0000 (oldest deadline 2024-03-20T09:48:50.167597+0000) 2024-03-20T09:49:01.432 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:01 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:01.204+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:31.868823+0000 front 2024-03-20T09:48:31.868814+0000 (oldest deadline 2024-03-20T09:48:56.469169+0000) 2024-03-20T09:49:01.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:01 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:01.103+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:01.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:01 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:01.103+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:01.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:01 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:01.103+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:01.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:01 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:01.103+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:01.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:01 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:01.319+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:49:01.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:01 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:01.319+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.170023+0000 front 2024-03-20T09:48:24.170175+0000 (oldest deadline 2024-03-20T09:48:47.070022+0000) 2024-03-20T09:49:01.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:01 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:01.319+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.771177+0000 front 2024-03-20T09:48:28.771129+0000 (oldest deadline 2024-03-20T09:48:59.371678+0000) 2024-03-20T09:49:01.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:49:01 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:49:01.620+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.210159+0000 front 2024-03-20T09:48:07.210270+0000 (oldest deadline 2024-03-20T09:48:32.411029+0000) 2024-03-20T09:49:01.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:49:01 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:49:01.620+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:12.411805+0000 front 2024-03-20T09:48:12.411874+0000 (oldest deadline 2024-03-20T09:48:38.311787+0000) 2024-03-20T09:49:01.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:49:01 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:49:01.620+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:23.013164+0000 front 2024-03-20T09:48:23.013151+0000 (oldest deadline 2024-03-20T09:48:47.713097+0000) 2024-03-20T09:49:01.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:49:01 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:49:01.620+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.314749+0000 front 2024-03-20T09:48:32.314928+0000 (oldest deadline 2024-03-20T09:48:54.614729+0000) 2024-03-20T09:49:02.077 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:01 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:01.721+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:49:02.077 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:01 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:01.721+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:49:02.077 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:01 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:01.721+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:49:02.077 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:01 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:01.721+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.580362+0000 front 2024-03-20T09:48:28.580387+0000 (oldest deadline 2024-03-20T09:48:57.980977+0000) 2024-03-20T09:49:02.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:02 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:02.238+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:49:02.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:02 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:02.238+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:16.365409+0000 front 2024-03-20T09:48:16.365395+0000 (oldest deadline 2024-03-20T09:48:39.265360+0000) 2024-03-20T09:49:02.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:02 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:02.238+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.367415+0000 front 2024-03-20T09:48:24.367214+0000 (oldest deadline 2024-03-20T09:48:50.167597+0000) 2024-03-20T09:49:02.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:02 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:02.238+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:31.868823+0000 front 2024-03-20T09:48:31.868814+0000 (oldest deadline 2024-03-20T09:48:56.469169+0000) 2024-03-20T09:49:02.434 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:02 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:02.090+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:02.434 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:02 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:02.090+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:02.434 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:02 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:02.090+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:02.434 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:02 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:02.090+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:02.435 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:02 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:02.329+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:49:02.435 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:02 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:02.329+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.170023+0000 front 2024-03-20T09:48:24.170175+0000 (oldest deadline 2024-03-20T09:48:47.070022+0000) 2024-03-20T09:49:02.435 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:02 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:02.329+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.771177+0000 front 2024-03-20T09:48:28.771129+0000 (oldest deadline 2024-03-20T09:48:59.371678+0000) 2024-03-20T09:49:02.891 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:49:02 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:49:02.635+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.210159+0000 front 2024-03-20T09:48:07.210270+0000 (oldest deadline 2024-03-20T09:48:32.411029+0000) 2024-03-20T09:49:02.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:49:02 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:49:02.635+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:12.411805+0000 front 2024-03-20T09:48:12.411874+0000 (oldest deadline 2024-03-20T09:48:38.311787+0000) 2024-03-20T09:49:02.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:49:02 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:49:02.635+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:23.013164+0000 front 2024-03-20T09:48:23.013151+0000 (oldest deadline 2024-03-20T09:48:47.713097+0000) 2024-03-20T09:49:02.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:49:02 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:49:02.635+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.314749+0000 front 2024-03-20T09:48:32.314928+0000 (oldest deadline 2024-03-20T09:48:54.614729+0000) 2024-03-20T09:49:03.119 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:02 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:02.742+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:49:03.119 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:02 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:02.742+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:49:03.119 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:02 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:02.742+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:49:03.120 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:02 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:02.742+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.580362+0000 front 2024-03-20T09:48:28.580387+0000 (oldest deadline 2024-03-20T09:48:57.980977+0000) 2024-03-20T09:49:03.432 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:03 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:03.279+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:49:03.432 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:03 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:03.279+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:16.365409+0000 front 2024-03-20T09:48:16.365395+0000 (oldest deadline 2024-03-20T09:48:39.265360+0000) 2024-03-20T09:49:03.432 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:03 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:03.279+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.367415+0000 front 2024-03-20T09:48:24.367214+0000 (oldest deadline 2024-03-20T09:48:50.167597+0000) 2024-03-20T09:49:03.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:03 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:03.279+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:31.868823+0000 front 2024-03-20T09:48:31.868814+0000 (oldest deadline 2024-03-20T09:48:56.469169+0000) 2024-03-20T09:49:03.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:03 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:03.362+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:49:03.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:03 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:03.362+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.170023+0000 front 2024-03-20T09:48:24.170175+0000 (oldest deadline 2024-03-20T09:48:47.070022+0000) 2024-03-20T09:49:03.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:03 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:03.362+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.771177+0000 front 2024-03-20T09:48:28.771129+0000 (oldest deadline 2024-03-20T09:48:59.371678+0000) 2024-03-20T09:49:03.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:03 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:03.133+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:03.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:03 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:03.133+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:03.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:03 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:03.133+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:03.434 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:03 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:03.133+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:03.891 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:49:03 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:49:03.641+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.210159+0000 front 2024-03-20T09:48:07.210270+0000 (oldest deadline 2024-03-20T09:48:32.411029+0000) 2024-03-20T09:49:03.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:49:03 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:49:03.641+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:12.411805+0000 front 2024-03-20T09:48:12.411874+0000 (oldest deadline 2024-03-20T09:48:38.311787+0000) 2024-03-20T09:49:03.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:49:03 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:49:03.641+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:23.013164+0000 front 2024-03-20T09:48:23.013151+0000 (oldest deadline 2024-03-20T09:48:47.713097+0000) 2024-03-20T09:49:03.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:49:03 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:49:03.641+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.314749+0000 front 2024-03-20T09:48:32.314928+0000 (oldest deadline 2024-03-20T09:48:54.614729+0000) 2024-03-20T09:49:03.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:49:03 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:49:03.641+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.115:6804 osd.4 since back 2024-03-20T09:48:39.216365+0000 front 2024-03-20T09:48:39.216349+0000 (oldest deadline 2024-03-20T09:49:03.316322+0000) 2024-03-20T09:49:04.072 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:03 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:03.743+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:49:04.072 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:03 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:03.743+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:49:04.072 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:03 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:03.743+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:49:04.072 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:03 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:03.743+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.580362+0000 front 2024-03-20T09:48:28.580387+0000 (oldest deadline 2024-03-20T09:48:57.980977+0000) 2024-03-20T09:49:04.432 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:04 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:04.283+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:49:04.432 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:04 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:04.283+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:16.365409+0000 front 2024-03-20T09:48:16.365395+0000 (oldest deadline 2024-03-20T09:48:39.265360+0000) 2024-03-20T09:49:04.432 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:04 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:04.283+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.367415+0000 front 2024-03-20T09:48:24.367214+0000 (oldest deadline 2024-03-20T09:48:50.167597+0000) 2024-03-20T09:49:04.432 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:04 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:04.283+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:31.868823+0000 front 2024-03-20T09:48:31.868814+0000 (oldest deadline 2024-03-20T09:48:56.469169+0000) 2024-03-20T09:49:04.432 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:04 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:04.283+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.115:6804 osd.4 since back 2024-03-20T09:48:40.570221+0000 front 2024-03-20T09:48:40.570296+0000 (oldest deadline 2024-03-20T09:49:03.970680+0000) 2024-03-20T09:49:04.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:04 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:04.323+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:49:04.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:04 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:04.323+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.170023+0000 front 2024-03-20T09:48:24.170175+0000 (oldest deadline 2024-03-20T09:48:47.070022+0000) 2024-03-20T09:49:04.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:04 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:04.323+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.771177+0000 front 2024-03-20T09:48:28.771129+0000 (oldest deadline 2024-03-20T09:48:59.371678+0000) 2024-03-20T09:49:04.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:04 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:04.323+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.115:6804 osd.4 since back 2024-03-20T09:48:39.372390+0000 front 2024-03-20T09:48:39.372124+0000 (oldest deadline 2024-03-20T09:49:03.472082+0000) 2024-03-20T09:49:04.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:04 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:04.085+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:04.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:04 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:04.085+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:04.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:04 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:04.085+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:04.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:04 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:04.085+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:04.891 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:49:04 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:49:04.602+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.210159+0000 front 2024-03-20T09:48:07.210270+0000 (oldest deadline 2024-03-20T09:48:32.411029+0000) 2024-03-20T09:49:04.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:49:04 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:49:04.602+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:12.411805+0000 front 2024-03-20T09:48:12.411874+0000 (oldest deadline 2024-03-20T09:48:38.311787+0000) 2024-03-20T09:49:04.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:49:04 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:49:04.602+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:23.013164+0000 front 2024-03-20T09:48:23.013151+0000 (oldest deadline 2024-03-20T09:48:47.713097+0000) 2024-03-20T09:49:04.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:49:04 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:49:04.602+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.314749+0000 front 2024-03-20T09:48:32.314928+0000 (oldest deadline 2024-03-20T09:48:54.614729+0000) 2024-03-20T09:49:04.892 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:49:04 smithi115 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7[100830]: 2024-03-20T09:49:04.602+0000 7ff1804b4700 -1 osd.7 1204 heartbeat_check: no reply from 172.21.15.115:6804 osd.4 since back 2024-03-20T09:48:39.216365+0000 front 2024-03-20T09:48:39.216349+0000 (oldest deadline 2024-03-20T09:49:03.316322+0000) 2024-03-20T09:49:05.105 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:04 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:04.720+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:49:05.105 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:04 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:04.720+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:49:05.105 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:04 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:04.720+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:49:05.105 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:04 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:04.720+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.580362+0000 front 2024-03-20T09:48:28.580387+0000 (oldest deadline 2024-03-20T09:48:57.980977+0000) 2024-03-20T09:49:05.432 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:05 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:05.297+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:49:05.432 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:05 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:05.297+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:16.365409+0000 front 2024-03-20T09:48:16.365395+0000 (oldest deadline 2024-03-20T09:48:39.265360+0000) 2024-03-20T09:49:05.432 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:05 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:05.297+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.367415+0000 front 2024-03-20T09:48:24.367214+0000 (oldest deadline 2024-03-20T09:48:50.167597+0000) 2024-03-20T09:49:05.432 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:05 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:05.297+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:31.868823+0000 front 2024-03-20T09:48:31.868814+0000 (oldest deadline 2024-03-20T09:48:56.469169+0000) 2024-03-20T09:49:05.432 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:05 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:05.297+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.115:6804 osd.4 since back 2024-03-20T09:48:40.570221+0000 front 2024-03-20T09:48:40.570296+0000 (oldest deadline 2024-03-20T09:49:03.970680+0000) 2024-03-20T09:49:05.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:05 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:05.315+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:49:05.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:05 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:05.315+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.170023+0000 front 2024-03-20T09:48:24.170175+0000 (oldest deadline 2024-03-20T09:48:47.070022+0000) 2024-03-20T09:49:05.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:05 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:05.315+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.771177+0000 front 2024-03-20T09:48:28.771129+0000 (oldest deadline 2024-03-20T09:48:59.371678+0000) 2024-03-20T09:49:05.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:05 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:05.315+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.115:6804 osd.4 since back 2024-03-20T09:48:39.372390+0000 front 2024-03-20T09:48:39.372124+0000 (oldest deadline 2024-03-20T09:49:03.472082+0000) 2024-03-20T09:49:05.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:05 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:05.118+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:05.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:05 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:05.118+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:05.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:05 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:05.118+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:05.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:05 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:05.118+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:06.140 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:05 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:05.701+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:49:06.140 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:05 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:05.701+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:49:06.140 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:05 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:05.701+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:49:06.140 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:05 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:05.701+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.580362+0000 front 2024-03-20T09:48:28.580387+0000 (oldest deadline 2024-03-20T09:48:57.980977+0000) 2024-03-20T09:49:06.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:06 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:06.153+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:06.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:06 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:06.153+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:06.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:06 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:06.153+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:06.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:06 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:06.153+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:06.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:06 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:06.282+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:49:06.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:06 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:06.282+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:16.365409+0000 front 2024-03-20T09:48:16.365395+0000 (oldest deadline 2024-03-20T09:48:39.265360+0000) 2024-03-20T09:49:06.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:06 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:06.282+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.367415+0000 front 2024-03-20T09:48:24.367214+0000 (oldest deadline 2024-03-20T09:48:50.167597+0000) 2024-03-20T09:49:06.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:06 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:06.282+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:31.868823+0000 front 2024-03-20T09:48:31.868814+0000 (oldest deadline 2024-03-20T09:48:56.469169+0000) 2024-03-20T09:49:06.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:06 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:06.282+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.115:6804 osd.4 since back 2024-03-20T09:48:40.570221+0000 front 2024-03-20T09:48:40.570296+0000 (oldest deadline 2024-03-20T09:49:03.970680+0000) 2024-03-20T09:49:06.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:06 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:06.273+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:49:06.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:06 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:06.273+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.170023+0000 front 2024-03-20T09:48:24.170175+0000 (oldest deadline 2024-03-20T09:48:47.070022+0000) 2024-03-20T09:49:06.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:06 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:06.273+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.771177+0000 front 2024-03-20T09:48:28.771129+0000 (oldest deadline 2024-03-20T09:48:59.371678+0000) 2024-03-20T09:49:06.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:06 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:06.273+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.115:6804 osd.4 since back 2024-03-20T09:48:39.372390+0000 front 2024-03-20T09:48:39.372124+0000 (oldest deadline 2024-03-20T09:49:03.472082+0000) 2024-03-20T09:49:06.636 INFO:journalctl@ceph.osd.7.smithi115.stdout:Mar 20 09:49:06 smithi115 bash[225494]: ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-7 2024-03-20T09:49:07.130 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:06 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:06.714+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:49:07.131 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:06 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:06.714+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:49:07.131 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:06 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:06.714+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:49:07.131 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:06 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:06.714+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.580362+0000 front 2024-03-20T09:48:28.580387+0000 (oldest deadline 2024-03-20T09:48:57.980977+0000) 2024-03-20T09:49:07.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:07 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:07.314+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:49:07.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:07 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:07.314+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.170023+0000 front 2024-03-20T09:48:24.170175+0000 (oldest deadline 2024-03-20T09:48:47.070022+0000) 2024-03-20T09:49:07.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:07 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:07.314+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.771177+0000 front 2024-03-20T09:48:28.771129+0000 (oldest deadline 2024-03-20T09:48:59.371678+0000) 2024-03-20T09:49:07.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:07 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:07.314+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.115:6804 osd.4 since back 2024-03-20T09:48:39.372390+0000 front 2024-03-20T09:48:39.372124+0000 (oldest deadline 2024-03-20T09:49:03.472082+0000) 2024-03-20T09:49:07.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:07 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:07.238+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:49:07.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:07 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:07.238+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:16.365409+0000 front 2024-03-20T09:48:16.365395+0000 (oldest deadline 2024-03-20T09:48:39.265360+0000) 2024-03-20T09:49:07.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:07 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:07.238+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.367415+0000 front 2024-03-20T09:48:24.367214+0000 (oldest deadline 2024-03-20T09:48:50.167597+0000) 2024-03-20T09:49:07.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:07 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:07.238+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:31.868823+0000 front 2024-03-20T09:48:31.868814+0000 (oldest deadline 2024-03-20T09:48:56.469169+0000) 2024-03-20T09:49:07.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:07 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:07.238+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.115:6804 osd.4 since back 2024-03-20T09:48:40.570221+0000 front 2024-03-20T09:48:40.570296+0000 (oldest deadline 2024-03-20T09:49:03.970680+0000) 2024-03-20T09:49:07.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:07 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:07.144+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:07.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:07 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:07.144+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:07.434 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:07 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:07.144+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:07.434 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:07 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:07.144+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:08.136 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:07 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:07.715+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:49:08.136 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:07 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:07.715+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:49:08.136 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:07 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:07.715+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:49:08.136 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:07 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:07.715+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.580362+0000 front 2024-03-20T09:48:28.580387+0000 (oldest deadline 2024-03-20T09:48:57.980977+0000) 2024-03-20T09:49:08.171 DEBUG:teuthology.orchestra.run.smithi115:> sudo pkill -f 'journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.7.service' 2024-03-20T09:49:08.210 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-20T09:49:08.210 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-20T09:49:08.211 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-20T09:49:08.211 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.8 2024-03-20T09:49:08.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:08 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:08.326+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:49:08.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:08 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:08.326+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.170023+0000 front 2024-03-20T09:48:24.170175+0000 (oldest deadline 2024-03-20T09:48:47.070022+0000) 2024-03-20T09:49:08.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:08 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:08.326+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.771177+0000 front 2024-03-20T09:48:28.771129+0000 (oldest deadline 2024-03-20T09:48:59.371678+0000) 2024-03-20T09:49:08.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:08 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:08.326+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.115:6804 osd.4 since back 2024-03-20T09:48:39.372390+0000 front 2024-03-20T09:48:39.372124+0000 (oldest deadline 2024-03-20T09:49:03.472082+0000) 2024-03-20T09:49:08.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:08 smithi122 systemd[1]: Stopping Ceph osd.8 for 852fbb68-e696-11ee-95c9-87774f69a715... 2024-03-20T09:49:08.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:08 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:08.280+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:49:08.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:08 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:08.280+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:16.365409+0000 front 2024-03-20T09:48:16.365395+0000 (oldest deadline 2024-03-20T09:48:39.265360+0000) 2024-03-20T09:49:08.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:08 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:08.280+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.367415+0000 front 2024-03-20T09:48:24.367214+0000 (oldest deadline 2024-03-20T09:48:50.167597+0000) 2024-03-20T09:49:08.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:08 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:08.280+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:31.868823+0000 front 2024-03-20T09:48:31.868814+0000 (oldest deadline 2024-03-20T09:48:56.469169+0000) 2024-03-20T09:49:08.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:08 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:08.280+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.115:6804 osd.4 since back 2024-03-20T09:48:40.570221+0000 front 2024-03-20T09:48:40.570296+0000 (oldest deadline 2024-03-20T09:49:03.970680+0000) 2024-03-20T09:49:08.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:08 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:08.150+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:08.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:08 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:08.150+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:08.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:08 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:08.150+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:08.434 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:08 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:08.150+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:08.932 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:08 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:08.531+0000 7f96eabd6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-20T09:49:08.932 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:08 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:08.531+0000 7f96eabd6700 -1 osd.8 1204 *** Got signal Terminated *** 2024-03-20T09:49:08.932 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:08 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:08.531+0000 7f96eabd6700 -1 osd.8 1204 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-20T09:49:08.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:08 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:08.698+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:49:08.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:08 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:08.698+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:49:08.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:08 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:08.698+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:49:08.933 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:08 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:08.698+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.580362+0000 front 2024-03-20T09:48:28.580387+0000 (oldest deadline 2024-03-20T09:48:57.980977+0000) 2024-03-20T09:49:09.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:09 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:09.339+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:49:09.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:09 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:09.339+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.170023+0000 front 2024-03-20T09:48:24.170175+0000 (oldest deadline 2024-03-20T09:48:47.070022+0000) 2024-03-20T09:49:09.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:09 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:09.339+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.771177+0000 front 2024-03-20T09:48:28.771129+0000 (oldest deadline 2024-03-20T09:48:59.371678+0000) 2024-03-20T09:49:09.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:09 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:09.339+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.115:6804 osd.4 since back 2024-03-20T09:48:39.372390+0000 front 2024-03-20T09:48:39.372124+0000 (oldest deadline 2024-03-20T09:49:03.472082+0000) 2024-03-20T09:49:09.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:09 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:09.329+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:49:09.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:09 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:09.329+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:16.365409+0000 front 2024-03-20T09:48:16.365395+0000 (oldest deadline 2024-03-20T09:48:39.265360+0000) 2024-03-20T09:49:09.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:09 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:09.329+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.367415+0000 front 2024-03-20T09:48:24.367214+0000 (oldest deadline 2024-03-20T09:48:50.167597+0000) 2024-03-20T09:49:09.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:09 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:09.329+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:31.868823+0000 front 2024-03-20T09:48:31.868814+0000 (oldest deadline 2024-03-20T09:48:56.469169+0000) 2024-03-20T09:49:09.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:09 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:09.329+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.115:6804 osd.4 since back 2024-03-20T09:48:40.570221+0000 front 2024-03-20T09:48:40.570296+0000 (oldest deadline 2024-03-20T09:49:03.970680+0000) 2024-03-20T09:49:09.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:09 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:09.173+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:09.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:09 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:09.173+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:09.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:09 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:09.173+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:09.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:09 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:09.173+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:09.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:09 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:09.682+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:49:09.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:09 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:09.682+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:49:09.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:09 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:09.682+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:49:09.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:09 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:09.682+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.580362+0000 front 2024-03-20T09:48:28.580387+0000 (oldest deadline 2024-03-20T09:48:57.980977+0000) 2024-03-20T09:49:10.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:10 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:10.361+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:49:10.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:10 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:10.361+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.170023+0000 front 2024-03-20T09:48:24.170175+0000 (oldest deadline 2024-03-20T09:48:47.070022+0000) 2024-03-20T09:49:10.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:10 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:10.361+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.771177+0000 front 2024-03-20T09:48:28.771129+0000 (oldest deadline 2024-03-20T09:48:59.371678+0000) 2024-03-20T09:49:10.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:10 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:10.361+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.115:6804 osd.4 since back 2024-03-20T09:48:39.372390+0000 front 2024-03-20T09:48:39.372124+0000 (oldest deadline 2024-03-20T09:49:03.472082+0000) 2024-03-20T09:49:10.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:10 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:10.363+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:49:10.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:10 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:10.363+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:16.365409+0000 front 2024-03-20T09:48:16.365395+0000 (oldest deadline 2024-03-20T09:48:39.265360+0000) 2024-03-20T09:49:10.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:10 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:10.363+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.367415+0000 front 2024-03-20T09:48:24.367214+0000 (oldest deadline 2024-03-20T09:48:50.167597+0000) 2024-03-20T09:49:10.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:10 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:10.363+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:31.868823+0000 front 2024-03-20T09:48:31.868814+0000 (oldest deadline 2024-03-20T09:48:56.469169+0000) 2024-03-20T09:49:10.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:10 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:10.363+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.115:6804 osd.4 since back 2024-03-20T09:48:40.570221+0000 front 2024-03-20T09:48:40.570296+0000 (oldest deadline 2024-03-20T09:49:03.970680+0000) 2024-03-20T09:49:10.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:10 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:10.137+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:10.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:10 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:10.137+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:10.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:10 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:10.137+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:10.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:10 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:10.137+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:10.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:10 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:10.662+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:49:10.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:10 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:10.662+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:49:10.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:10 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:10.662+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:49:10.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:10 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:10.662+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.580362+0000 front 2024-03-20T09:48:28.580387+0000 (oldest deadline 2024-03-20T09:48:57.980977+0000) 2024-03-20T09:49:10.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:10 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:10.662+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:45.582911+0000 front 2024-03-20T09:48:45.582703+0000 (oldest deadline 2024-03-20T09:49:10.282841+0000) 2024-03-20T09:49:11.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:11 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:11.402+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:49:11.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:11 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:11.402+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.170023+0000 front 2024-03-20T09:48:24.170175+0000 (oldest deadline 2024-03-20T09:48:47.070022+0000) 2024-03-20T09:49:11.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:11 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:11.402+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.771177+0000 front 2024-03-20T09:48:28.771129+0000 (oldest deadline 2024-03-20T09:48:59.371678+0000) 2024-03-20T09:49:11.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:11 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:11.402+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.115:6804 osd.4 since back 2024-03-20T09:48:39.372390+0000 front 2024-03-20T09:48:39.372124+0000 (oldest deadline 2024-03-20T09:49:03.472082+0000) 2024-03-20T09:49:11.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:11 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:11.387+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:49:11.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:11 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:11.387+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:16.365409+0000 front 2024-03-20T09:48:16.365395+0000 (oldest deadline 2024-03-20T09:48:39.265360+0000) 2024-03-20T09:49:11.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:11 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:11.387+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.367415+0000 front 2024-03-20T09:48:24.367214+0000 (oldest deadline 2024-03-20T09:48:50.167597+0000) 2024-03-20T09:49:11.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:11 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:11.387+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:31.868823+0000 front 2024-03-20T09:48:31.868814+0000 (oldest deadline 2024-03-20T09:48:56.469169+0000) 2024-03-20T09:49:11.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:11 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:11.387+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.115:6804 osd.4 since back 2024-03-20T09:48:40.570221+0000 front 2024-03-20T09:48:40.570296+0000 (oldest deadline 2024-03-20T09:49:03.970680+0000) 2024-03-20T09:49:11.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:11 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:11.174+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:11.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:11 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:11.174+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:11.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:11 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:11.174+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:11.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:11 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:11.174+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:11.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:11 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:11.644+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:49:11.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:11 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:11.644+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:49:11.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:11 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:11.644+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:49:11.933 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:11 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:11.644+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.580362+0000 front 2024-03-20T09:48:28.580387+0000 (oldest deadline 2024-03-20T09:48:57.980977+0000) 2024-03-20T09:49:11.933 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:11 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:11.644+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:45.582911+0000 front 2024-03-20T09:48:45.582703+0000 (oldest deadline 2024-03-20T09:49:10.282841+0000) 2024-03-20T09:49:12.422 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:12 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:12.342+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:49:12.422 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:12 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:12.342+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:16.365409+0000 front 2024-03-20T09:48:16.365395+0000 (oldest deadline 2024-03-20T09:48:39.265360+0000) 2024-03-20T09:49:12.422 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:12 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:12.342+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.367415+0000 front 2024-03-20T09:48:24.367214+0000 (oldest deadline 2024-03-20T09:48:50.167597+0000) 2024-03-20T09:49:12.422 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:12 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:12.342+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:31.868823+0000 front 2024-03-20T09:48:31.868814+0000 (oldest deadline 2024-03-20T09:48:56.469169+0000) 2024-03-20T09:49:12.422 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:12 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:12.342+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.115:6804 osd.4 since back 2024-03-20T09:48:40.570221+0000 front 2024-03-20T09:48:40.570296+0000 (oldest deadline 2024-03-20T09:49:03.970680+0000) 2024-03-20T09:49:12.422 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:12 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:12.148+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:12.423 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:12 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:12.148+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:12.423 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:12 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:12.148+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:12.423 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:12 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:12.148+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:12.423 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:12 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:12.148+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:47.976942+0000 front 2024-03-20T09:48:47.976988+0000 (oldest deadline 2024-03-20T09:49:11.476934+0000) 2024-03-20T09:49:12.682 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:12 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:12.435+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:49:12.682 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:12 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:12.435+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.170023+0000 front 2024-03-20T09:48:24.170175+0000 (oldest deadline 2024-03-20T09:48:47.070022+0000) 2024-03-20T09:49:12.682 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:12 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:12.435+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.771177+0000 front 2024-03-20T09:48:28.771129+0000 (oldest deadline 2024-03-20T09:48:59.371678+0000) 2024-03-20T09:49:12.682 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:12 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:12.435+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.115:6804 osd.4 since back 2024-03-20T09:48:39.372390+0000 front 2024-03-20T09:48:39.372124+0000 (oldest deadline 2024-03-20T09:49:03.472082+0000) 2024-03-20T09:49:12.682 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:12 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:12.627+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:49:12.683 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:12 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:12.627+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:49:12.683 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:12 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:12.627+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:49:12.683 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:12 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:12.627+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.580362+0000 front 2024-03-20T09:48:28.580387+0000 (oldest deadline 2024-03-20T09:48:57.980977+0000) 2024-03-20T09:49:12.683 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:12 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:12.627+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:45.582911+0000 front 2024-03-20T09:48:45.582703+0000 (oldest deadline 2024-03-20T09:49:10.282841+0000) 2024-03-20T09:49:13.432 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:13 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:13.389+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:07.663597+0000 front 2024-03-20T09:48:07.663722+0000 (oldest deadline 2024-03-20T09:48:30.563748+0000) 2024-03-20T09:49:13.432 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:13 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:13.389+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:16.365409+0000 front 2024-03-20T09:48:16.365395+0000 (oldest deadline 2024-03-20T09:48:39.265360+0000) 2024-03-20T09:49:13.432 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:13 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:13.389+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.367415+0000 front 2024-03-20T09:48:24.367214+0000 (oldest deadline 2024-03-20T09:48:50.167597+0000) 2024-03-20T09:49:13.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:13 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:13.389+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:31.868823+0000 front 2024-03-20T09:48:31.868814+0000 (oldest deadline 2024-03-20T09:48:56.469169+0000) 2024-03-20T09:49:13.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:13 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:13.389+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.115:6804 osd.4 since back 2024-03-20T09:48:40.570221+0000 front 2024-03-20T09:48:40.570296+0000 (oldest deadline 2024-03-20T09:49:03.970680+0000) 2024-03-20T09:49:13.433 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:13 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8[79409]: 2024-03-20T09:49:13.389+0000 7f96e71ca700 -1 osd.8 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:47.972394+0000 front 2024-03-20T09:48:47.972415+0000 (oldest deadline 2024-03-20T09:49:13.272135+0000) 2024-03-20T09:49:13.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:13 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:13.142+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:13.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:13 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:13.142+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:13.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:13 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:13.142+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:13.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:13 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:13.142+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:13.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:13 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:13.142+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:47.976942+0000 front 2024-03-20T09:48:47.976988+0000 (oldest deadline 2024-03-20T09:49:11.476934+0000) 2024-03-20T09:49:13.932 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:13 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:13.462+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:49:13.932 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:13 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:13.462+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.170023+0000 front 2024-03-20T09:48:24.170175+0000 (oldest deadline 2024-03-20T09:48:47.070022+0000) 2024-03-20T09:49:13.932 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:13 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:13.462+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.771177+0000 front 2024-03-20T09:48:28.771129+0000 (oldest deadline 2024-03-20T09:48:59.371678+0000) 2024-03-20T09:49:13.932 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:13 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:13.462+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.115:6804 osd.4 since back 2024-03-20T09:48:39.372390+0000 front 2024-03-20T09:48:39.372124+0000 (oldest deadline 2024-03-20T09:49:03.472082+0000) 2024-03-20T09:49:13.933 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:13 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:13.609+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:49:13.933 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:13 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:13.609+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:49:13.933 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:13 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:13.609+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:49:13.933 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:13 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:13.609+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.580362+0000 front 2024-03-20T09:48:28.580387+0000 (oldest deadline 2024-03-20T09:48:57.980977+0000) 2024-03-20T09:49:13.933 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:13 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:13.609+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:45.582911+0000 front 2024-03-20T09:48:45.582703+0000 (oldest deadline 2024-03-20T09:49:10.282841+0000) 2024-03-20T09:49:14.432 INFO:journalctl@ceph.osd.8.smithi122.stdout:Mar 20 09:49:14 smithi122 bash[222221]: ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-8 2024-03-20T09:49:14.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:14 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:14.417+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:49:14.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:14 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:14.417+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.170023+0000 front 2024-03-20T09:48:24.170175+0000 (oldest deadline 2024-03-20T09:48:47.070022+0000) 2024-03-20T09:49:14.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:14 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:14.417+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.771177+0000 front 2024-03-20T09:48:28.771129+0000 (oldest deadline 2024-03-20T09:48:59.371678+0000) 2024-03-20T09:49:14.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:14 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:14.417+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.115:6804 osd.4 since back 2024-03-20T09:48:39.372390+0000 front 2024-03-20T09:48:39.372124+0000 (oldest deadline 2024-03-20T09:49:03.472082+0000) 2024-03-20T09:49:14.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:14 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:14.417+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:48.173484+0000 front 2024-03-20T09:48:48.173521+0000 (oldest deadline 2024-03-20T09:49:14.073277+0000) 2024-03-20T09:49:14.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:14 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:14.181+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:14.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:14 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:14.181+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:14.434 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:14 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:14.181+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:14.434 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:14 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:14.181+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:14.434 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:14 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:14.181+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:47.976942+0000 front 2024-03-20T09:48:47.976988+0000 (oldest deadline 2024-03-20T09:49:11.476934+0000) 2024-03-20T09:49:14.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:14 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:14.575+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:49:14.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:14 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:14.575+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:49:14.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:14 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:14.575+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:49:14.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:14 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:14.575+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.580362+0000 front 2024-03-20T09:48:28.580387+0000 (oldest deadline 2024-03-20T09:48:57.980977+0000) 2024-03-20T09:49:14.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:14 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:14.575+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:45.582911+0000 front 2024-03-20T09:48:45.582703+0000 (oldest deadline 2024-03-20T09:49:10.282841+0000) 2024-03-20T09:49:15.426 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:15 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:15.210+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:15.426 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:15 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:15.210+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:15.426 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:15 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:15.210+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:15.426 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:15 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:15.210+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:15.426 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:15 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:15.210+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:47.976942+0000 front 2024-03-20T09:48:47.976988+0000 (oldest deadline 2024-03-20T09:49:11.476934+0000) 2024-03-20T09:49:15.682 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:15 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:15.436+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:49:15.682 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:15 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:15.436+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.170023+0000 front 2024-03-20T09:48:24.170175+0000 (oldest deadline 2024-03-20T09:48:47.070022+0000) 2024-03-20T09:49:15.682 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:15 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:15.436+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.771177+0000 front 2024-03-20T09:48:28.771129+0000 (oldest deadline 2024-03-20T09:48:59.371678+0000) 2024-03-20T09:49:15.682 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:15 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:15.436+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.115:6804 osd.4 since back 2024-03-20T09:48:39.372390+0000 front 2024-03-20T09:48:39.372124+0000 (oldest deadline 2024-03-20T09:49:03.472082+0000) 2024-03-20T09:49:15.682 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:15 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:15.436+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:48.173484+0000 front 2024-03-20T09:48:48.173521+0000 (oldest deadline 2024-03-20T09:49:14.073277+0000) 2024-03-20T09:49:15.682 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:15 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:15.609+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:49:15.683 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:15 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:15.609+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:49:15.683 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:15 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:15.609+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:49:15.683 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:15 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:15.609+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.580362+0000 front 2024-03-20T09:48:28.580387+0000 (oldest deadline 2024-03-20T09:48:57.980977+0000) 2024-03-20T09:49:15.683 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:15 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:15.609+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:45.582911+0000 front 2024-03-20T09:48:45.582703+0000 (oldest deadline 2024-03-20T09:49:10.282841+0000) 2024-03-20T09:49:16.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:16 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:16.390+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:49:16.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:16 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:16.390+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.170023+0000 front 2024-03-20T09:48:24.170175+0000 (oldest deadline 2024-03-20T09:48:47.070022+0000) 2024-03-20T09:49:16.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:16 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:16.390+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.771177+0000 front 2024-03-20T09:48:28.771129+0000 (oldest deadline 2024-03-20T09:48:59.371678+0000) 2024-03-20T09:49:16.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:16 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:16.390+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.115:6804 osd.4 since back 2024-03-20T09:48:39.372390+0000 front 2024-03-20T09:48:39.372124+0000 (oldest deadline 2024-03-20T09:49:03.472082+0000) 2024-03-20T09:49:16.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:16 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:16.390+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:48.173484+0000 front 2024-03-20T09:48:48.173521+0000 (oldest deadline 2024-03-20T09:49:14.073277+0000) 2024-03-20T09:49:16.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:16 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:16.203+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:16.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:16 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:16.203+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:16.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:16 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:16.203+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:16.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:16 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:16.203+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:16.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:16 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:16.203+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:47.976942+0000 front 2024-03-20T09:48:47.976988+0000 (oldest deadline 2024-03-20T09:49:11.476934+0000) 2024-03-20T09:49:16.550 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.8.service' 2024-03-20T09:49:16.592 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-20T09:49:16.592 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-20T09:49:16.592 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-20T09:49:16.592 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.9 2024-03-20T09:49:16.899 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:16 smithi122 systemd[1]: Stopping Ceph osd.9 for 852fbb68-e696-11ee-95c9-87774f69a715... 2024-03-20T09:49:16.899 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:16 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:16.602+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:49:16.900 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:16 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:16.602+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:49:16.900 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:16 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:16.602+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:49:16.900 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:16 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:16.602+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.580362+0000 front 2024-03-20T09:48:28.580387+0000 (oldest deadline 2024-03-20T09:48:57.980977+0000) 2024-03-20T09:49:16.900 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:16 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:16.602+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:45.582911+0000 front 2024-03-20T09:48:45.582703+0000 (oldest deadline 2024-03-20T09:49:10.282841+0000) 2024-03-20T09:49:17.162 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:16 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:16.912+0000 7fc9641cd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-20T09:49:17.162 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:16 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:16.912+0000 7fc9641cd700 -1 osd.9 1204 *** Got signal Terminated *** 2024-03-20T09:49:17.162 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:16 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:16.912+0000 7fc9641cd700 -1 osd.9 1204 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-20T09:49:17.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:17 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:17.176+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:17.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:17 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:17.176+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:17.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:17 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:17.176+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:17.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:17 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:17.176+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:17.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:17 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:17.176+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:47.976942+0000 front 2024-03-20T09:48:47.976988+0000 (oldest deadline 2024-03-20T09:49:11.476934+0000) 2024-03-20T09:49:17.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:17 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:17.392+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:49:17.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:17 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:17.393+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.170023+0000 front 2024-03-20T09:48:24.170175+0000 (oldest deadline 2024-03-20T09:48:47.070022+0000) 2024-03-20T09:49:17.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:17 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:17.393+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.771177+0000 front 2024-03-20T09:48:28.771129+0000 (oldest deadline 2024-03-20T09:48:59.371678+0000) 2024-03-20T09:49:17.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:17 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:17.393+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.115:6804 osd.4 since back 2024-03-20T09:48:39.372390+0000 front 2024-03-20T09:48:39.372124+0000 (oldest deadline 2024-03-20T09:49:03.472082+0000) 2024-03-20T09:49:17.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:17 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:17.393+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:48.173484+0000 front 2024-03-20T09:48:48.173521+0000 (oldest deadline 2024-03-20T09:49:14.073277+0000) 2024-03-20T09:49:17.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:17 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:17.643+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:49:17.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:17 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:17.643+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:49:17.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:17 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:17.643+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:49:17.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:17 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:17.643+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.580362+0000 front 2024-03-20T09:48:28.580387+0000 (oldest deadline 2024-03-20T09:48:57.980977+0000) 2024-03-20T09:49:17.933 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:17 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:17.643+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:45.582911+0000 front 2024-03-20T09:48:45.582703+0000 (oldest deadline 2024-03-20T09:49:10.282841+0000) 2024-03-20T09:49:18.671 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:18 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:18.405+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:49:18.671 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:18 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:18.405+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.170023+0000 front 2024-03-20T09:48:24.170175+0000 (oldest deadline 2024-03-20T09:48:47.070022+0000) 2024-03-20T09:49:18.672 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:18 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:18.405+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.771177+0000 front 2024-03-20T09:48:28.771129+0000 (oldest deadline 2024-03-20T09:48:59.371678+0000) 2024-03-20T09:49:18.672 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:18 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:18.405+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.115:6804 osd.4 since back 2024-03-20T09:48:39.372390+0000 front 2024-03-20T09:48:39.372124+0000 (oldest deadline 2024-03-20T09:49:03.472082+0000) 2024-03-20T09:49:18.672 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:18 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:18.405+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:48.173484+0000 front 2024-03-20T09:48:48.173521+0000 (oldest deadline 2024-03-20T09:49:14.073277+0000) 2024-03-20T09:49:18.672 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:18 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:18.209+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:18.672 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:18 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:18.209+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:18.672 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:18 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:18.209+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:18.672 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:18 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:18.209+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:18.672 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:18 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:18.209+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:47.976942+0000 front 2024-03-20T09:48:47.976988+0000 (oldest deadline 2024-03-20T09:49:11.476934+0000) 2024-03-20T09:49:18.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:18 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:18.685+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:49:18.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:18 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:18.685+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:49:18.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:18 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:18.685+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:49:18.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:18 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:18.685+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.580362+0000 front 2024-03-20T09:48:28.580387+0000 (oldest deadline 2024-03-20T09:48:57.980977+0000) 2024-03-20T09:49:18.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:18 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:18.685+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:45.582911+0000 front 2024-03-20T09:48:45.582703+0000 (oldest deadline 2024-03-20T09:49:10.282841+0000) 2024-03-20T09:49:19.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:19 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:19.367+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:49:19.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:19 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:19.367+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.170023+0000 front 2024-03-20T09:48:24.170175+0000 (oldest deadline 2024-03-20T09:48:47.070022+0000) 2024-03-20T09:49:19.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:19 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:19.367+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.771177+0000 front 2024-03-20T09:48:28.771129+0000 (oldest deadline 2024-03-20T09:48:59.371678+0000) 2024-03-20T09:49:19.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:19 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:19.367+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.115:6804 osd.4 since back 2024-03-20T09:48:39.372390+0000 front 2024-03-20T09:48:39.372124+0000 (oldest deadline 2024-03-20T09:49:03.472082+0000) 2024-03-20T09:49:19.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:19 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:19.367+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:48.173484+0000 front 2024-03-20T09:48:48.173521+0000 (oldest deadline 2024-03-20T09:49:14.073277+0000) 2024-03-20T09:49:19.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:19 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:19.193+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:19.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:19 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:19.193+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:19.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:19 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:19.193+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:19.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:19 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:19.193+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:19.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:19 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:19.193+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:47.976942+0000 front 2024-03-20T09:48:47.976988+0000 (oldest deadline 2024-03-20T09:49:11.476934+0000) 2024-03-20T09:49:20.137 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:19 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:19.696+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:49:20.137 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:19 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:19.696+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:49:20.137 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:19 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:19.696+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:49:20.138 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:19 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:19.696+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.580362+0000 front 2024-03-20T09:48:28.580387+0000 (oldest deadline 2024-03-20T09:48:57.980977+0000) 2024-03-20T09:49:20.138 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:19 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:19.696+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:45.582911+0000 front 2024-03-20T09:48:45.582703+0000 (oldest deadline 2024-03-20T09:49:10.282841+0000) 2024-03-20T09:49:20.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:20 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:20.344+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:49:20.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:20 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:20.344+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.170023+0000 front 2024-03-20T09:48:24.170175+0000 (oldest deadline 2024-03-20T09:48:47.070022+0000) 2024-03-20T09:49:20.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:20 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:20.344+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.771177+0000 front 2024-03-20T09:48:28.771129+0000 (oldest deadline 2024-03-20T09:48:59.371678+0000) 2024-03-20T09:49:20.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:20 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:20.344+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.115:6804 osd.4 since back 2024-03-20T09:48:39.372390+0000 front 2024-03-20T09:48:39.372124+0000 (oldest deadline 2024-03-20T09:49:03.472082+0000) 2024-03-20T09:49:20.432 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:20 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:20.344+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:48.173484+0000 front 2024-03-20T09:48:48.173521+0000 (oldest deadline 2024-03-20T09:49:14.073277+0000) 2024-03-20T09:49:20.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:20 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:20.150+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:20.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:20 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:20.150+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:20.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:20 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:20.150+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:20.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:20 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:20.150+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:20.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:20 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:20.150+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:47.976942+0000 front 2024-03-20T09:48:47.976988+0000 (oldest deadline 2024-03-20T09:49:11.476934+0000) 2024-03-20T09:49:21.093 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:20 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:20.741+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:49:21.094 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:20 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:20.741+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:49:21.094 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:20 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:20.741+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:49:21.094 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:20 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:20.741+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.580362+0000 front 2024-03-20T09:48:28.580387+0000 (oldest deadline 2024-03-20T09:48:57.980977+0000) 2024-03-20T09:49:21.094 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:20 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:20.741+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:45.582911+0000 front 2024-03-20T09:48:45.582703+0000 (oldest deadline 2024-03-20T09:49:10.282841+0000) 2024-03-20T09:49:21.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:21 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:21.108+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:21.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:21 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:21.108+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:21.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:21 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:21.108+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:21.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:21 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:21.108+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:21.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:21 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:21.108+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:47.976942+0000 front 2024-03-20T09:48:47.976988+0000 (oldest deadline 2024-03-20T09:49:11.476934+0000) 2024-03-20T09:49:21.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:21 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:21.300+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:13.068109+0000 front 2024-03-20T09:48:13.068296+0000 (oldest deadline 2024-03-20T09:48:38.968204+0000) 2024-03-20T09:49:21.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:21 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:21.300+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:24.170023+0000 front 2024-03-20T09:48:24.170175+0000 (oldest deadline 2024-03-20T09:48:47.070022+0000) 2024-03-20T09:49:21.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:21 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:21.300+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.771177+0000 front 2024-03-20T09:48:28.771129+0000 (oldest deadline 2024-03-20T09:48:59.371678+0000) 2024-03-20T09:49:21.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:21 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:21.300+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.115:6804 osd.4 since back 2024-03-20T09:48:39.372390+0000 front 2024-03-20T09:48:39.372124+0000 (oldest deadline 2024-03-20T09:49:03.472082+0000) 2024-03-20T09:49:21.433 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:21 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9[87134]: 2024-03-20T09:49:21.300+0000 7fc95c7b9700 -1 osd.9 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:48.173484+0000 front 2024-03-20T09:48:48.173521+0000 (oldest deadline 2024-03-20T09:49:14.073277+0000) 2024-03-20T09:49:22.060 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:21 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:21.734+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:49:22.060 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:21 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:21.734+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:49:22.060 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:21 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:21.734+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:49:22.060 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:21 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:21.734+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.580362+0000 front 2024-03-20T09:48:28.580387+0000 (oldest deadline 2024-03-20T09:48:57.980977+0000) 2024-03-20T09:49:22.060 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:21 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:21.734+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:45.582911+0000 front 2024-03-20T09:48:45.582703+0000 (oldest deadline 2024-03-20T09:49:10.282841+0000) 2024-03-20T09:49:22.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:22 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:22.073+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:22.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:22 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:22.073+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:22.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:22 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:22.073+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:22.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:22 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:22.073+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:22.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:22 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:22.073+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:47.976942+0000 front 2024-03-20T09:48:47.976988+0000 (oldest deadline 2024-03-20T09:49:11.476934+0000) 2024-03-20T09:49:22.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:22 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:22.073+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6820 osd.6 since back 2024-03-20T09:48:57.278710+0000 front 2024-03-20T09:48:57.278650+0000 (oldest deadline 2024-03-20T09:49:21.978448+0000) 2024-03-20T09:49:22.932 INFO:journalctl@ceph.osd.9.smithi122.stdout:Mar 20 09:49:22 smithi122 bash[222486]: ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-9 2024-03-20T09:49:22.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:22 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:22.783+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:49:22.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:22 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:22.783+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:49:22.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:22 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:22.783+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:49:22.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:22 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:22.783+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.580362+0000 front 2024-03-20T09:48:28.580387+0000 (oldest deadline 2024-03-20T09:48:57.980977+0000) 2024-03-20T09:49:22.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:22 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:22.783+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:45.582911+0000 front 2024-03-20T09:48:45.582703+0000 (oldest deadline 2024-03-20T09:49:10.282841+0000) 2024-03-20T09:49:22.933 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:22 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:22.783+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.115:6820 osd.6 since back 2024-03-20T09:48:56.684450+0000 front 2024-03-20T09:48:56.684502+0000 (oldest deadline 2024-03-20T09:49:22.584261+0000) 2024-03-20T09:49:23.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:23 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:23.113+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:23.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:23 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:23.113+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:23.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:23 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:23.113+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:23.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:23 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:23.113+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:23.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:23 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:23.113+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:47.976942+0000 front 2024-03-20T09:48:47.976988+0000 (oldest deadline 2024-03-20T09:49:11.476934+0000) 2024-03-20T09:49:23.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:23 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:23.113+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6820 osd.6 since back 2024-03-20T09:48:57.278710+0000 front 2024-03-20T09:48:57.278650+0000 (oldest deadline 2024-03-20T09:49:21.978448+0000) 2024-03-20T09:49:23.933 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:23 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:23.822+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:49:23.933 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:23 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:23.822+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:49:23.933 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:23 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:23.822+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:49:23.933 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:23 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:23.822+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.580362+0000 front 2024-03-20T09:48:28.580387+0000 (oldest deadline 2024-03-20T09:48:57.980977+0000) 2024-03-20T09:49:23.933 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:23 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:23.822+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:45.582911+0000 front 2024-03-20T09:48:45.582703+0000 (oldest deadline 2024-03-20T09:49:10.282841+0000) 2024-03-20T09:49:23.933 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:23 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:23.822+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.115:6820 osd.6 since back 2024-03-20T09:48:56.684450+0000 front 2024-03-20T09:48:56.684502+0000 (oldest deadline 2024-03-20T09:49:22.584261+0000) 2024-03-20T09:49:24.276 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:24 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:24.127+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:24.277 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:24 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:24.127+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:24.277 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:24 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:24.127+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:24.277 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:24 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:24.127+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:24.277 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:24 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:24.127+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:47.976942+0000 front 2024-03-20T09:48:47.976988+0000 (oldest deadline 2024-03-20T09:49:11.476934+0000) 2024-03-20T09:49:24.277 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:24 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:24.127+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6820 osd.6 since back 2024-03-20T09:48:57.278710+0000 front 2024-03-20T09:48:57.278650+0000 (oldest deadline 2024-03-20T09:49:21.978448+0000) 2024-03-20T09:49:24.604 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.9.service' 2024-03-20T09:49:24.645 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-20T09:49:24.646 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-20T09:49:24.646 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-20T09:49:24.646 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.10 2024-03-20T09:49:24.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:24 smithi122 systemd[1]: Stopping Ceph osd.10 for 852fbb68-e696-11ee-95c9-87774f69a715... 2024-03-20T09:49:24.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:24 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:24.776+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:49:24.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:24 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:24.776+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:49:24.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:24 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:24.776+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:49:24.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:24 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:24.776+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.580362+0000 front 2024-03-20T09:48:28.580387+0000 (oldest deadline 2024-03-20T09:48:57.980977+0000) 2024-03-20T09:49:24.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:24 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:24.776+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:45.582911+0000 front 2024-03-20T09:48:45.582703+0000 (oldest deadline 2024-03-20T09:49:10.282841+0000) 2024-03-20T09:49:24.933 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:24 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:24.776+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.115:6820 osd.6 since back 2024-03-20T09:48:56.684450+0000 front 2024-03-20T09:48:56.684502+0000 (oldest deadline 2024-03-20T09:49:22.584261+0000) 2024-03-20T09:49:25.432 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:24 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:24.967+0000 7fa92f59a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-20T09:49:25.432 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:24 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:24.967+0000 7fa92f59a700 -1 osd.10 1204 *** Got signal Terminated *** 2024-03-20T09:49:25.432 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:24 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:24.967+0000 7fa92f59a700 -1 osd.10 1204 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-20T09:49:25.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:25 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:25.110+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:25.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:25 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:25.110+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:25.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:25 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:25.110+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:25.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:25 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:25.110+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:25.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:25 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:25.110+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:47.976942+0000 front 2024-03-20T09:48:47.976988+0000 (oldest deadline 2024-03-20T09:49:11.476934+0000) 2024-03-20T09:49:25.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:25 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:25.110+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6820 osd.6 since back 2024-03-20T09:48:57.278710+0000 front 2024-03-20T09:48:57.278650+0000 (oldest deadline 2024-03-20T09:49:21.978448+0000) 2024-03-20T09:49:26.117 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:25 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:25.797+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:49:26.117 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:25 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:25.797+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:49:26.117 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:25 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:25.797+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:49:26.117 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:25 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:25.797+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.580362+0000 front 2024-03-20T09:48:28.580387+0000 (oldest deadline 2024-03-20T09:48:57.980977+0000) 2024-03-20T09:49:26.117 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:25 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:25.797+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:45.582911+0000 front 2024-03-20T09:48:45.582703+0000 (oldest deadline 2024-03-20T09:49:10.282841+0000) 2024-03-20T09:49:26.117 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:25 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:25.797+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.115:6820 osd.6 since back 2024-03-20T09:48:56.684450+0000 front 2024-03-20T09:48:56.684502+0000 (oldest deadline 2024-03-20T09:49:22.584261+0000) 2024-03-20T09:49:26.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:26 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:26.131+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:26.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:26 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:26.131+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:26.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:26 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:26.131+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:26.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:26 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:26.131+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:26.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:26 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:26.131+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:47.976942+0000 front 2024-03-20T09:48:47.976988+0000 (oldest deadline 2024-03-20T09:49:11.476934+0000) 2024-03-20T09:49:26.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:26 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:26.131+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6820 osd.6 since back 2024-03-20T09:48:57.278710+0000 front 2024-03-20T09:48:57.278650+0000 (oldest deadline 2024-03-20T09:49:21.978448+0000) 2024-03-20T09:49:27.091 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:26 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:26.842+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:49:27.092 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:26 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:26.842+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:49:27.092 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:26 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:26.842+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:49:27.092 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:26 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:26.842+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.580362+0000 front 2024-03-20T09:48:28.580387+0000 (oldest deadline 2024-03-20T09:48:57.980977+0000) 2024-03-20T09:49:27.092 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:26 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:26.842+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:45.582911+0000 front 2024-03-20T09:48:45.582703+0000 (oldest deadline 2024-03-20T09:49:10.282841+0000) 2024-03-20T09:49:27.092 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:26 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:26.842+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.115:6820 osd.6 since back 2024-03-20T09:48:56.684450+0000 front 2024-03-20T09:48:56.684502+0000 (oldest deadline 2024-03-20T09:49:22.584261+0000) 2024-03-20T09:49:27.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:27 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:27.105+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:27.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:27 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:27.105+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:27.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:27 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:27.105+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:27.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:27 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:27.105+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:27.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:27 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:27.105+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:47.976942+0000 front 2024-03-20T09:48:47.976988+0000 (oldest deadline 2024-03-20T09:49:11.476934+0000) 2024-03-20T09:49:27.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:27 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:27.105+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6820 osd.6 since back 2024-03-20T09:48:57.278710+0000 front 2024-03-20T09:48:57.278650+0000 (oldest deadline 2024-03-20T09:49:21.978448+0000) 2024-03-20T09:49:28.136 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:27 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:27.863+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:49:28.137 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:27 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:27.863+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:49:28.137 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:27 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:27.863+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:49:28.137 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:27 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:27.863+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.580362+0000 front 2024-03-20T09:48:28.580387+0000 (oldest deadline 2024-03-20T09:48:57.980977+0000) 2024-03-20T09:49:28.137 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:27 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:27.863+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:45.582911+0000 front 2024-03-20T09:48:45.582703+0000 (oldest deadline 2024-03-20T09:49:10.282841+0000) 2024-03-20T09:49:28.137 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:27 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:27.863+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.115:6820 osd.6 since back 2024-03-20T09:48:56.684450+0000 front 2024-03-20T09:48:56.684502+0000 (oldest deadline 2024-03-20T09:49:22.584261+0000) 2024-03-20T09:49:28.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:28 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:28.150+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:28.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:28 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:28.150+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:28.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:28 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:28.150+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:28.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:28 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:28.150+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:28.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:28 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:28.150+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:47.976942+0000 front 2024-03-20T09:48:47.976988+0000 (oldest deadline 2024-03-20T09:49:11.476934+0000) 2024-03-20T09:49:28.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:28 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:28.150+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6820 osd.6 since back 2024-03-20T09:48:57.278710+0000 front 2024-03-20T09:48:57.278650+0000 (oldest deadline 2024-03-20T09:49:21.978448+0000) 2024-03-20T09:49:29.180 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:28 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:28.912+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:49:29.180 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:28 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:28.912+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:49:29.180 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:28 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:28.912+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:49:29.180 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:28 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:28.912+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.580362+0000 front 2024-03-20T09:48:28.580387+0000 (oldest deadline 2024-03-20T09:48:57.980977+0000) 2024-03-20T09:49:29.180 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:28 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:28.912+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:45.582911+0000 front 2024-03-20T09:48:45.582703+0000 (oldest deadline 2024-03-20T09:49:10.282841+0000) 2024-03-20T09:49:29.180 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:28 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:28.912+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.115:6820 osd.6 since back 2024-03-20T09:48:56.684450+0000 front 2024-03-20T09:48:56.684502+0000 (oldest deadline 2024-03-20T09:49:22.584261+0000) 2024-03-20T09:49:29.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:29 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:29.194+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:29.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:29 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:29.194+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:29.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:29 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:29.194+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:29.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:29 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:29.194+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:29.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:29 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:29.194+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:47.976942+0000 front 2024-03-20T09:48:47.976988+0000 (oldest deadline 2024-03-20T09:49:11.476934+0000) 2024-03-20T09:49:29.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:29 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:29.194+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6820 osd.6 since back 2024-03-20T09:48:57.278710+0000 front 2024-03-20T09:48:57.278650+0000 (oldest deadline 2024-03-20T09:49:21.978448+0000) 2024-03-20T09:49:29.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:29 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:29.194+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6828 osd.7 since back 2024-03-20T09:49:04.279645+0000 front 2024-03-20T09:49:04.279679+0000 (oldest deadline 2024-03-20T09:49:28.979650+0000) 2024-03-20T09:49:30.182 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:29 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:29.942+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:05.176877+0000 front 2024-03-20T09:48:05.177186+0000 (oldest deadline 2024-03-20T09:48:29.876940+0000) 2024-03-20T09:49:30.182 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:29 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:29.942+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.778315+0000 front 2024-03-20T09:48:15.778471+0000 (oldest deadline 2024-03-20T09:48:40.478334+0000) 2024-03-20T09:49:30.182 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:29 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:29.942+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.679988+0000 front 2024-03-20T09:48:22.680010+0000 (oldest deadline 2024-03-20T09:48:48.579850+0000) 2024-03-20T09:49:30.182 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:29 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:29.942+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:28.580362+0000 front 2024-03-20T09:48:28.580387+0000 (oldest deadline 2024-03-20T09:48:57.980977+0000) 2024-03-20T09:49:30.182 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:29 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:29.942+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:45.582911+0000 front 2024-03-20T09:48:45.582703+0000 (oldest deadline 2024-03-20T09:49:10.282841+0000) 2024-03-20T09:49:30.183 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:29 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:29.942+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.115:6820 osd.6 since back 2024-03-20T09:48:56.684450+0000 front 2024-03-20T09:48:56.684502+0000 (oldest deadline 2024-03-20T09:49:22.584261+0000) 2024-03-20T09:49:30.183 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:29 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10[184628]: 2024-03-20T09:49:29.942+0000 7fa927b86700 -1 osd.10 1204 heartbeat_check: no reply from 172.21.15.115:6828 osd.7 since back 2024-03-20T09:49:04.885411+0000 front 2024-03-20T09:49:04.885389+0000 (oldest deadline 2024-03-20T09:49:28.985254+0000) 2024-03-20T09:49:30.606 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:30 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:30.223+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:30.607 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:30 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:30.223+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:30.607 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:30 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:30.223+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:30.607 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:30 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:30.223+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:30.607 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:30 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:30.223+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:47.976942+0000 front 2024-03-20T09:48:47.976988+0000 (oldest deadline 2024-03-20T09:49:11.476934+0000) 2024-03-20T09:49:30.607 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:30 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:30.223+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6820 osd.6 since back 2024-03-20T09:48:57.278710+0000 front 2024-03-20T09:48:57.278650+0000 (oldest deadline 2024-03-20T09:49:21.978448+0000) 2024-03-20T09:49:30.607 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:30 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:30.223+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6828 osd.7 since back 2024-03-20T09:49:04.279645+0000 front 2024-03-20T09:49:04.279679+0000 (oldest deadline 2024-03-20T09:49:28.979650+0000) 2024-03-20T09:49:30.932 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 20 09:49:30 smithi122 bash[222748]: ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-10 2024-03-20T09:49:31.682 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:31 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:31.219+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:31.723 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:31 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:31.219+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:31.723 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:31 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:31.219+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:31.723 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:31 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:31.219+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:31.723 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:31 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:31.219+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:47.976942+0000 front 2024-03-20T09:48:47.976988+0000 (oldest deadline 2024-03-20T09:49:11.476934+0000) 2024-03-20T09:49:31.723 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:31 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:31.219+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6820 osd.6 since back 2024-03-20T09:48:57.278710+0000 front 2024-03-20T09:48:57.278650+0000 (oldest deadline 2024-03-20T09:49:21.978448+0000) 2024-03-20T09:49:31.724 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:31 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:31.219+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6828 osd.7 since back 2024-03-20T09:49:04.279645+0000 front 2024-03-20T09:49:04.279679+0000 (oldest deadline 2024-03-20T09:49:28.979650+0000) 2024-03-20T09:49:32.425 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:32 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:32.177+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:32.426 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:32 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:32.177+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:32.426 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:32 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:32.177+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:32.426 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:32 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:32.177+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:32.426 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:32 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:32.177+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:47.976942+0000 front 2024-03-20T09:48:47.976988+0000 (oldest deadline 2024-03-20T09:49:11.476934+0000) 2024-03-20T09:49:32.426 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:32 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:32.177+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6820 osd.6 since back 2024-03-20T09:48:57.278710+0000 front 2024-03-20T09:48:57.278650+0000 (oldest deadline 2024-03-20T09:49:21.978448+0000) 2024-03-20T09:49:32.426 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:32 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:32.177+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6828 osd.7 since back 2024-03-20T09:49:04.279645+0000 front 2024-03-20T09:49:04.279679+0000 (oldest deadline 2024-03-20T09:49:28.979650+0000) 2024-03-20T09:49:32.433 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.10.service' 2024-03-20T09:49:32.474 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-20T09:49:32.474 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-20T09:49:32.474 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-20T09:49:32.474 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.11 2024-03-20T09:49:32.682 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:32 smithi122 systemd[1]: Stopping Ceph osd.11 for 852fbb68-e696-11ee-95c9-87774f69a715... 2024-03-20T09:49:33.166 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:32 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:32.822+0000 7f2b29014700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-20T09:49:33.166 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:32 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:32.822+0000 7f2b29014700 -1 osd.11 1204 *** Got signal Terminated *** 2024-03-20T09:49:33.166 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:32 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:32.822+0000 7f2b29014700 -1 osd.11 1204 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-20T09:49:33.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:33 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:33.179+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:33.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:33 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:33.179+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:33.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:33 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:33.179+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:33.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:33 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:33.179+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:33.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:33 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:33.179+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:47.976942+0000 front 2024-03-20T09:48:47.976988+0000 (oldest deadline 2024-03-20T09:49:11.476934+0000) 2024-03-20T09:49:33.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:33 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:33.179+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6820 osd.6 since back 2024-03-20T09:48:57.278710+0000 front 2024-03-20T09:48:57.278650+0000 (oldest deadline 2024-03-20T09:49:21.978448+0000) 2024-03-20T09:49:33.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:33 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:33.179+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6828 osd.7 since back 2024-03-20T09:49:04.279645+0000 front 2024-03-20T09:49:04.279679+0000 (oldest deadline 2024-03-20T09:49:28.979650+0000) 2024-03-20T09:49:34.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:34 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:34.171+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:34.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:34 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:34.171+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:34.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:34 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:34.171+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:34.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:34 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:34.171+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:34.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:34 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:34.171+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:47.976942+0000 front 2024-03-20T09:48:47.976988+0000 (oldest deadline 2024-03-20T09:49:11.476934+0000) 2024-03-20T09:49:34.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:34 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:34.171+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6820 osd.6 since back 2024-03-20T09:48:57.278710+0000 front 2024-03-20T09:48:57.278650+0000 (oldest deadline 2024-03-20T09:49:21.978448+0000) 2024-03-20T09:49:34.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:34 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:34.171+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6828 osd.7 since back 2024-03-20T09:49:04.279645+0000 front 2024-03-20T09:49:04.279679+0000 (oldest deadline 2024-03-20T09:49:28.979650+0000) 2024-03-20T09:49:35.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:35 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:35.145+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:35.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:35 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:35.145+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:35.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:35 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:35.145+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:35.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:35 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:35.145+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:35.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:35 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:35.145+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:47.976942+0000 front 2024-03-20T09:48:47.976988+0000 (oldest deadline 2024-03-20T09:49:11.476934+0000) 2024-03-20T09:49:35.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:35 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:35.145+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6820 osd.6 since back 2024-03-20T09:48:57.278710+0000 front 2024-03-20T09:48:57.278650+0000 (oldest deadline 2024-03-20T09:49:21.978448+0000) 2024-03-20T09:49:35.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:35 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:35.145+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6828 osd.7 since back 2024-03-20T09:49:04.279645+0000 front 2024-03-20T09:49:04.279679+0000 (oldest deadline 2024-03-20T09:49:28.979650+0000) 2024-03-20T09:49:35.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:35 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:35.145+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.122:6804 osd.8 since back 2024-03-20T09:49:08.980141+0000 front 2024-03-20T09:49:08.980177+0000 (oldest deadline 2024-03-20T09:49:34.880092+0000) 2024-03-20T09:49:36.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:36 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:36.145+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:36.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:36 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:36.145+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:36.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:36 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:36.145+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:36.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:36 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:36.145+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:36.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:36 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:36.145+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:47.976942+0000 front 2024-03-20T09:48:47.976988+0000 (oldest deadline 2024-03-20T09:49:11.476934+0000) 2024-03-20T09:49:36.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:36 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:36.145+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6820 osd.6 since back 2024-03-20T09:48:57.278710+0000 front 2024-03-20T09:48:57.278650+0000 (oldest deadline 2024-03-20T09:49:21.978448+0000) 2024-03-20T09:49:36.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:36 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:36.145+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6828 osd.7 since back 2024-03-20T09:49:04.279645+0000 front 2024-03-20T09:49:04.279679+0000 (oldest deadline 2024-03-20T09:49:28.979650+0000) 2024-03-20T09:49:36.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:36 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:36.145+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.122:6804 osd.8 since back 2024-03-20T09:49:08.980141+0000 front 2024-03-20T09:49:08.980177+0000 (oldest deadline 2024-03-20T09:49:34.880092+0000) 2024-03-20T09:49:37.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:37 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:37.110+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-20T09:48:06.069723+0000 front 2024-03-20T09:48:06.069732+0000 (oldest deadline 2024-03-20T09:48:30.169570+0000) 2024-03-20T09:49:37.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:37 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:37.110+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-20T09:48:15.371597+0000 front 2024-03-20T09:48:15.371486+0000 (oldest deadline 2024-03-20T09:48:37.572012+0000) 2024-03-20T09:49:37.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:37 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:37.110+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-20T09:48:22.872947+0000 front 2024-03-20T09:48:22.872969+0000 (oldest deadline 2024-03-20T09:48:48.173181+0000) 2024-03-20T09:49:37.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:37 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:37.110+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-20T09:48:32.874555+0000 front 2024-03-20T09:48:32.874493+0000 (oldest deadline 2024-03-20T09:48:57.574352+0000) 2024-03-20T09:49:37.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:37 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:37.110+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6812 osd.5 since back 2024-03-20T09:48:47.976942+0000 front 2024-03-20T09:48:47.976988+0000 (oldest deadline 2024-03-20T09:49:11.476934+0000) 2024-03-20T09:49:37.432 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:37 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:37.110+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6820 osd.6 since back 2024-03-20T09:48:57.278710+0000 front 2024-03-20T09:48:57.278650+0000 (oldest deadline 2024-03-20T09:49:21.978448+0000) 2024-03-20T09:49:37.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:37 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:37.110+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.115:6828 osd.7 since back 2024-03-20T09:49:04.279645+0000 front 2024-03-20T09:49:04.279679+0000 (oldest deadline 2024-03-20T09:49:28.979650+0000) 2024-03-20T09:49:37.433 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:37 smithi122 ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11[101289]: 2024-03-20T09:49:37.110+0000 7f2b21600700 -1 osd.11 1204 heartbeat_check: no reply from 172.21.15.122:6804 osd.8 since back 2024-03-20T09:49:08.980141+0000 front 2024-03-20T09:49:08.980177+0000 (oldest deadline 2024-03-20T09:49:34.880092+0000) 2024-03-20T09:49:39.121 INFO:journalctl@ceph.osd.11.smithi122.stdout:Mar 20 09:49:38 smithi122 bash[223015]: ceph-852fbb68-e696-11ee-95c9-87774f69a715-osd-11 2024-03-20T09:49:40.641 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-852fbb68-e696-11ee-95c9-87774f69a715@osd.11.service' 2024-03-20T09:49:40.679 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-20T09:49:40.679 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-20T09:49:40.680 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 852fbb68-e696-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-20T09:49:40.932 INFO:teuthology.orchestra.run.smithi026.stdout:Deleting cluster with fsid: 852fbb68-e696-11ee-95c9-87774f69a715 2024-03-20T09:49:42.321 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 852fbb68-e696-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-20T09:49:42.550 INFO:teuthology.orchestra.run.smithi115.stdout:Deleting cluster with fsid: 852fbb68-e696-11ee-95c9-87774f69a715 2024-03-20T09:49:43.790 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 852fbb68-e696-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-20T09:49:44.023 INFO:teuthology.orchestra.run.smithi122.stdout:Deleting cluster with fsid: 852fbb68-e696-11ee-95c9-87774f69a715 2024-03-20T09:49:45.038 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-20T09:49:45.076 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-20T09:49:45.113 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-20T09:49:45.142 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-20T09:49:45.144 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-03-19_00:09:45-rados-wip-yuri5-testing-2024-03-18-1144-distro-default-smithi/7610089/remote/smithi026/crash 2024-03-20T09:49:45.145 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar c -f - -C /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/crash -- . 2024-03-20T09:49:45.217 INFO:teuthology.orchestra.run.smithi026.stderr:tar: /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-20T09:49:45.217 INFO:teuthology.orchestra.run.smithi026.stderr:tar: Error is not recoverable: exiting now 2024-03-20T09:49:45.219 DEBUG:teuthology.misc:Transferring archived files from smithi115:/var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-03-19_00:09:45-rados-wip-yuri5-testing-2024-03-18-1144-distro-default-smithi/7610089/remote/smithi115/crash 2024-03-20T09:49:45.220 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar c -f - -C /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/crash -- . 2024-03-20T09:49:45.417 INFO:teuthology.orchestra.run.smithi115.stderr:tar: /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-20T09:49:45.417 INFO:teuthology.orchestra.run.smithi115.stderr:tar: Error is not recoverable: exiting now 2024-03-20T09:49:45.419 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-03-19_00:09:45-rados-wip-yuri5-testing-2024-03-18-1144-distro-default-smithi/7610089/remote/smithi122/crash 2024-03-20T09:49:45.420 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/crash -- . 2024-03-20T09:49:45.481 INFO:teuthology.orchestra.run.smithi122.stderr:tar: /var/lib/ceph/852fbb68-e696-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-20T09:49:45.481 INFO:teuthology.orchestra.run.smithi122.stderr:tar: Error is not recoverable: exiting now 2024-03-20T09:49:45.482 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-20T09:49:45.482 DEBUG:teuthology.orchestra.run.smithi026:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | head -n 1 2024-03-20T09:49:45.609 INFO:teuthology.orchestra.run.smithi026.stdout:2024-03-20T08:50:40.165150+0000 mon.a (mon.0) 196 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-20T09:49:45.610 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-20T09:49:45.610 DEBUG:teuthology.orchestra.run.smithi026:> sudo egrep '\[SEC\]' /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | head -n 1 2024-03-20T09:49:45.652 DEBUG:teuthology.orchestra.run.smithi026:> sudo egrep '\[ERR\]' /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | head -n 1 2024-03-20T09:49:45.732 DEBUG:teuthology.orchestra.run.smithi026:> sudo egrep '\[WRN\]' /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | head -n 1 2024-03-20T09:49:45.812 INFO:teuthology.orchestra.run.smithi026.stdout:2024-03-20T08:50:40.165150+0000 mon.a (mon.0) 196 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-20T09:49:45.813 INFO:tasks.cephadm:Compressing logs... 2024-03-20T09:49:45.813 DEBUG:teuthology.orchestra.run.smithi026:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-20T09:49:45.856 DEBUG:teuthology.orchestra.run.smithi115:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-20T09:49:45.859 DEBUG:teuthology.orchestra.run.smithi122:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-20T09:49:45.891 INFO:teuthology.orchestra.run.smithi026.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-20T09:49:45.891 INFO:teuthology.orchestra.run.smithi026.stderr:: No such file or directory 2024-03-20T09:49:45.900 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-20T09:49:45.900 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-osd.2.log 2024-03-20T09:49:45.900 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-mgr.y.log 2024-03-20T09:49:45.900 INFO:teuthology.orchestra.run.smithi122.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-20T09:49:45.901 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-20T09:49:45.902 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-20T09:49:45.903 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-20T09:49:45.903 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-osd.3.log 2024-03-20T09:49:45.903 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph.audit.log: /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-osd.0.log 2024-03-20T09:49:45.903 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-mon.a.log 2024-03-20T09:49:45.903 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph.log 2024-03-20T09:49:45.903 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-osd.1.log 2024-03-20T09:49:45.904 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-osd.11.log 2024-03-20T09:49:45.904 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-20T09:49:45.906 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-20T09:49:45.906 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-20T09:49:45.906 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-osd.10.log 2024-03-20T09:49:45.906 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-osd.8.log 2024-03-20T09:49:45.906 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/objectstore_tool.7.log 2024-03-20T09:49:45.907 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-mon.c.log 2024-03-20T09:49:45.907 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/objectstore_tool.7.log: gzip -5 --verbose -- /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-osd.9.log 2024-03-20T09:49:45.907 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph.log 2024-03-20T09:49:45.914 INFO:teuthology.orchestra.run.smithi115.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-20T09:49:45.914 INFO:teuthology.orchestra.run.smithi115.stderr:: No such file or directory 2024-03-20T09:49:45.917 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-osd.9.log: /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph.log: 95.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-20T09:49:45.917 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph.log: /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-osd.1.log: 86.1% -- replaced with /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-20T09:49:45.920 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-osd.6.log 2024-03-20T09:49:45.922 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-20T09:49:45.922 INFO:teuthology.orchestra.run.smithi115.stderr:gzip -5 --verbose -- /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-20T09:49:45.922 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-osd.6.log: /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-20T09:49:45.922 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-osd.7.log 2024-03-20T09:49:45.922 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-mgr.x.log 2024-03-20T09:49:45.922 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/objectstore_tool.7.log 2024-03-20T09:49:45.924 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-mon.b.log 2024-03-20T09:49:45.924 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/objectstore_tool.7.log: gzip -5 --verbose -- /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph.log 2024-03-20T09:49:45.924 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-osd.4.log 2024-03-20T09:49:45.924 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-osd.5.log 2024-03-20T09:49:45.939 INFO:teuthology.orchestra.run.smithi026.stderr: 91.5% -- replaced with /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-20T09:49:45.950 INFO:teuthology.orchestra.run.smithi115.stderr:/var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-osd.4.log: /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-osd.5.log: 93.5% -- replaced with /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-mgr.x.log.gz 2024-03-20T09:49:45.952 INFO:teuthology.orchestra.run.smithi122.stderr: 91.3% -- replaced with /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-20T09:49:45.952 INFO:teuthology.orchestra.run.smithi122.stderr: 82.1% -- replaced with /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-20T09:49:46.118 INFO:teuthology.orchestra.run.smithi026.stderr: 89.9% -- replaced with /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-20T09:49:46.135 INFO:teuthology.orchestra.run.smithi115.stderr: 91.3% -- replaced with /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-20T09:49:46.201 INFO:teuthology.orchestra.run.smithi115.stderr: 95.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-20T09:49:46.292 INFO:teuthology.orchestra.run.smithi115.stderr: 94.7% -- replaced with /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/objectstore_tool.7.log.gz 2024-03-20T09:49:46.322 INFO:teuthology.orchestra.run.smithi122.stderr: 89.9% -- replaced with /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-20T09:49:46.566 INFO:teuthology.orchestra.run.smithi115.stderr: 82.1% -- replaced with /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-20T09:49:46.886 INFO:teuthology.orchestra.run.smithi026.stderr: 82.8% -- replaced with /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-20T09:49:47.130 INFO:teuthology.orchestra.run.smithi026.stderr: 96.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-20T09:49:47.291 INFO:teuthology.orchestra.run.smithi122.stderr: 86.1% -- replaced with /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-20T09:49:47.292 INFO:teuthology.orchestra.run.smithi115.stderr: 89.9% -- replaced with /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-20T09:49:47.971 INFO:teuthology.orchestra.run.smithi115.stderr: 86.1% -- replaced with /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-20T09:49:48.630 INFO:teuthology.orchestra.run.smithi122.stderr: 94.5% -- replaced with /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/objectstore_tool.7.log.gz 2024-03-20T09:49:49.914 INFO:teuthology.orchestra.run.smithi026.stderr: 91.3% -- replaced with /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-mgr.y.log.gz 2024-03-20T09:49:50.581 INFO:teuthology.orchestra.run.smithi122.stderr: 91.1% -- replaced with /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-mon.c.log.gz 2024-03-20T09:49:50.917 INFO:teuthology.orchestra.run.smithi115.stderr: 91.1% -- replaced with /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-mon.b.log.gz 2024-03-20T09:49:56.004 INFO:teuthology.orchestra.run.smithi026.stderr: 90.0% -- replaced with /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-mon.a.log.gz 2024-03-20T09:52:34.247 INFO:teuthology.orchestra.run.smithi115.stderr: 94.2% -- replaced with /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-osd.7.log.gz 2024-03-20T09:52:35.219 INFO:teuthology.orchestra.run.smithi122.stderr: 94.2% -- replaced with /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-osd.9.log.gz 2024-03-20T09:54:31.602 INFO:teuthology.orchestra.run.smithi122.stderr: 92.4% -- replaced with /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-osd.8.log.gz 2024-03-20T09:54:54.415 INFO:teuthology.orchestra.run.smithi122.stderr: 92.4% -- replaced with /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-osd.10.log.gz 2024-03-20T09:55:04.508 INFO:teuthology.orchestra.run.smithi115.stderr: 91.6% -- replaced with /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-osd.4.log.gz 2024-03-20T09:56:02.247 INFO:teuthology.orchestra.run.smithi026.stderr: 92.9% -- replaced with /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-osd.2.log.gz 2024-03-20T09:56:31.937 INFO:teuthology.orchestra.run.smithi115.stderr: 92.7% -- replaced with /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-osd.5.log.gz 2024-03-20T09:56:52.979 INFO:teuthology.orchestra.run.smithi115.stderr: 91.5% -- replaced with /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-osd.6.log.gz 2024-03-20T09:56:52.981 INFO:teuthology.orchestra.run.smithi115.stderr: 2024-03-20T09:56:52.981 INFO:teuthology.orchestra.run.smithi115.stderr:real 7m7.105s 2024-03-20T09:56:52.981 INFO:teuthology.orchestra.run.smithi115.stderr:user 9m4.592s 2024-03-20T09:56:52.981 INFO:teuthology.orchestra.run.smithi115.stderr:sys 0m22.211s 2024-03-20T09:56:53.890 INFO:teuthology.orchestra.run.smithi122.stderr: 92.1% -- replaced with /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-osd.11.log.gz 2024-03-20T09:56:53.892 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-03-20T09:56:53.892 INFO:teuthology.orchestra.run.smithi122.stderr:real 7m8.015s 2024-03-20T09:56:53.892 INFO:teuthology.orchestra.run.smithi122.stderr:user 8m31.816s 2024-03-20T09:56:53.892 INFO:teuthology.orchestra.run.smithi122.stderr:sys 0m21.260s 2024-03-20T09:59:13.261 INFO:teuthology.orchestra.run.smithi026.stderr: 92.0% -- replaced with /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-osd.3.log.gz 2024-03-20T09:59:17.661 INFO:teuthology.orchestra.run.smithi026.stderr: 92.1% -- replaced with /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-osd.1.log.gz 2024-03-20T10:01:25.807 INFO:teuthology.orchestra.run.smithi026.stderr: 91.1% -- replaced with /var/log/ceph/852fbb68-e696-11ee-95c9-87774f69a715/ceph-osd.0.log.gz 2024-03-20T10:01:25.809 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-20T10:01:25.810 INFO:teuthology.orchestra.run.smithi026.stderr:real 11m39.931s 2024-03-20T10:01:25.810 INFO:teuthology.orchestra.run.smithi026.stderr:user 14m28.703s 2024-03-20T10:01:25.810 INFO:teuthology.orchestra.run.smithi026.stderr:sys 0m31.904s 2024-03-20T10:01:25.810 INFO:tasks.cephadm:Archiving logs... 2024-03-20T10:01:25.810 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-03-19_00:09:45-rados-wip-yuri5-testing-2024-03-18-1144-distro-default-smithi/7610089/remote/smithi026/log 2024-03-20T10:01:25.812 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-20T10:03:56.018 DEBUG:teuthology.misc:Transferring archived files from smithi115:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-03-19_00:09:45-rados-wip-yuri5-testing-2024-03-18-1144-distro-default-smithi/7610089/remote/smithi115/log 2024-03-20T10:03:56.066 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-20T10:05:31.749 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-03-19_00:09:45-rados-wip-yuri5-testing-2024-03-18-1144-distro-default-smithi/7610089/remote/smithi122/log 2024-03-20T10:05:31.750 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-20T10:07:03.688 INFO:tasks.cephadm:Removing cluster... 2024-03-20T10:07:03.689 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 852fbb68-e696-11ee-95c9-87774f69a715 --force 2024-03-20T10:07:04.321 INFO:teuthology.orchestra.run.smithi026.stdout:Deleting cluster with fsid: 852fbb68-e696-11ee-95c9-87774f69a715 2024-03-20T10:07:05.463 DEBUG:teuthology.orchestra.run.smithi115:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 852fbb68-e696-11ee-95c9-87774f69a715 --force 2024-03-20T10:07:06.072 INFO:teuthology.orchestra.run.smithi115.stdout:Deleting cluster with fsid: 852fbb68-e696-11ee-95c9-87774f69a715 2024-03-20T10:07:06.961 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 852fbb68-e696-11ee-95c9-87774f69a715 --force 2024-03-20T10:07:07.437 INFO:teuthology.orchestra.run.smithi122.stdout:Deleting cluster with fsid: 852fbb68-e696-11ee-95c9-87774f69a715 2024-03-20T10:07:08.271 INFO:tasks.cephadm:Removing cephadm ... 2024-03-20T10:07:08.272 DEBUG:teuthology.orchestra.run.smithi026:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-20T10:07:08.304 DEBUG:teuthology.orchestra.run.smithi115:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-20T10:07:08.398 DEBUG:teuthology.orchestra.run.smithi122:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-20T10:07:08.421 INFO:tasks.cephadm:Teardown complete 2024-03-20T10:07:08.421 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-20T10:07:08.431 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-20T10:07:08.431 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-20T10:07:08.434 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-20T10:07:08.437 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-20T10:07:08.608 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-20T10:07:08.620 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-20T10:07:08.621 DEBUG:teuthology.orchestra.run.smithi026:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-20T10:07:08.624 DEBUG:teuthology.orchestra.run.smithi115:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-20T10:07:08.626 DEBUG:teuthology.orchestra.run.smithi122:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-20T10:07:08.643 INFO:teuthology.orchestra.run.smithi026.stderr:bash: ntpq: command not found 2024-03-20T10:07:08.644 INFO:teuthology.orchestra.run.smithi115.stderr:bash: ntpq: command not found 2024-03-20T10:07:08.652 INFO:teuthology.orchestra.run.smithi122.stderr:bash: ntpq: command not found 2024-03-20T10:07:08.681 INFO:teuthology.orchestra.run.smithi026.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-20T10:07:08.681 INFO:teuthology.orchestra.run.smithi026.stdout:=============================================================================== 2024-03-20T10:07:08.681 INFO:teuthology.orchestra.run.smithi026.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 52 -92us[ -66us] +/- 116ms 2024-03-20T10:07:08.681 INFO:teuthology.orchestra.run.smithi026.stdout:^* hv02.front.sepia.ceph.com 3 7 377 51 -1181us[-1155us] +/- 79ms 2024-03-20T10:07:08.681 INFO:teuthology.orchestra.run.smithi026.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 52 +1358us[+1385us] +/- 74ms 2024-03-20T10:07:08.682 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-20T10:07:08.683 INFO:teuthology.orchestra.run.smithi122.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-20T10:07:08.683 INFO:teuthology.orchestra.run.smithi122.stdout:=============================================================================== 2024-03-20T10:07:08.683 INFO:teuthology.orchestra.run.smithi122.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 50 -42us[ -42us] +/- 116ms 2024-03-20T10:07:08.683 INFO:teuthology.orchestra.run.smithi122.stdout:^* hv02.front.sepia.ceph.com 3 8 377 54 -1103us[-1006us] +/- 79ms 2024-03-20T10:07:08.683 INFO:teuthology.orchestra.run.smithi122.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 +1425us[+1425us] +/- 74ms 2024-03-20T10:07:08.683 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-20T10:07:08.703 INFO:teuthology.orchestra.run.smithi115.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-20T10:07:08.703 INFO:teuthology.orchestra.run.smithi115.stdout:=============================================================================== 2024-03-20T10:07:08.704 INFO:teuthology.orchestra.run.smithi115.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 54 -122us[ -122us] +/- 116ms 2024-03-20T10:07:08.704 INFO:teuthology.orchestra.run.smithi115.stdout:^* hv02.front.sepia.ceph.com 3 6 377 54 -1278us[-1271us] +/- 79ms 2024-03-20T10:07:08.704 INFO:teuthology.orchestra.run.smithi115.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 46 +1313us[+1313us] +/- 74ms 2024-03-20T10:07:08.704 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-20T10:07:08.705 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-20T10:07:08.721 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-20T10:07:08.722 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-20T10:07:08.741 DEBUG:teuthology.orchestra.run.smithi026:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-20T10:07:08.848 DEBUG:teuthology.orchestra.run.smithi115:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-20T10:07:08.968 DEBUG:teuthology.orchestra.run.smithi122:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-20T10:07:09.028 DEBUG:teuthology.orchestra.run.smithi026:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-20T10:07:09.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-20T10:07:09.065 DEBUG:teuthology.orchestra.run.smithi115:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-20T10:07:09.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-20T10:07:09.099 DEBUG:teuthology.orchestra.run.smithi122:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-20T10:07:09.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-20T10:07:09.128 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-20T10:07:09.148 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-20T10:07:09.231 INFO:teuthology.task.internal:Duration was 5352.491648 seconds 2024-03-20T10:07:09.232 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-20T10:07:09.264 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-20T10:07:09.265 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-20T10:07:09.268 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-20T10:07:09.270 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-20T10:07:09.346 INFO:teuthology.orchestra.run.smithi026.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-20T10:07:09.354 INFO:teuthology.orchestra.run.smithi122.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-20T10:07:09.355 INFO:teuthology.orchestra.run.smithi115.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-20T10:07:09.752 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-20T10:07:09.752 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi026.front.sepia.ceph.com 2024-03-20T10:07:09.752 DEBUG:teuthology.orchestra.run.smithi026:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-20T10:07:09.785 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi115.front.sepia.ceph.com 2024-03-20T10:07:09.786 DEBUG:teuthology.orchestra.run.smithi115:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-20T10:07:09.821 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi122.front.sepia.ceph.com 2024-03-20T10:07:09.821 DEBUG:teuthology.orchestra.run.smithi122:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-20T10:07:09.862 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-20T10:07:09.862 DEBUG:teuthology.orchestra.run.smithi026:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-20T10:07:09.865 DEBUG:teuthology.orchestra.run.smithi115:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-20T10:07:09.868 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-20T10:07:10.010 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-20T10:07:10.010 DEBUG:teuthology.orchestra.run.smithi026:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-20T10:07:10.012 DEBUG:teuthology.orchestra.run.smithi115:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-20T10:07:10.015 DEBUG:teuthology.orchestra.run.smithi122:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-20T10:07:10.303 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-20T10:07:10.313 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-20T10:07:10.314 DEBUG:teuthology.orchestra.run.smithi026:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-20T10:07:10.377 DEBUG:teuthology.orchestra.run.smithi115:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-20T10:07:10.430 DEBUG:teuthology.orchestra.run.smithi122:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-20T10:07:10.474 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-20T10:07:10.485 DEBUG:teuthology.orchestra.run.smithi026:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-20T10:07:10.487 DEBUG:teuthology.orchestra.run.smithi115:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-20T10:07:10.489 DEBUG:teuthology.orchestra.run.smithi122:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-20T10:07:10.518 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern = core 2024-03-20T10:07:10.523 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = core 2024-03-20T10:07:10.523 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern = core 2024-03-20T10:07:10.563 DEBUG:teuthology.orchestra.run.smithi026:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-20T10:07:10.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-20T10:07:10.595 DEBUG:teuthology.orchestra.run.smithi115:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-20T10:07:10.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-20T10:07:10.612 DEBUG:teuthology.orchestra.run.smithi122:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-20T10:07:10.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-20T10:07:10.629 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-20T10:07:10.639 INFO:teuthology.task.internal:Transferring archived files... 2024-03-20T10:07:10.641 DEBUG:teuthology.misc:Transferring archived files from smithi026:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-19_00:09:45-rados-wip-yuri5-testing-2024-03-18-1144-distro-default-smithi/7610089/remote/smithi026 2024-03-20T10:07:10.641 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-20T10:07:10.701 DEBUG:teuthology.misc:Transferring archived files from smithi115:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-19_00:09:45-rados-wip-yuri5-testing-2024-03-18-1144-distro-default-smithi/7610089/remote/smithi115 2024-03-20T10:07:10.701 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-20T10:07:10.747 DEBUG:teuthology.misc:Transferring archived files from smithi122:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-19_00:09:45-rados-wip-yuri5-testing-2024-03-18-1144-distro-default-smithi/7610089/remote/smithi122 2024-03-20T10:07:10.747 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-20T10:07:10.792 INFO:teuthology.task.internal:Removing archive directory... 2024-03-20T10:07:10.792 DEBUG:teuthology.orchestra.run.smithi026:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-20T10:07:10.795 DEBUG:teuthology.orchestra.run.smithi115:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-20T10:07:10.797 DEBUG:teuthology.orchestra.run.smithi122:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-20T10:07:10.841 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-20T10:07:10.854 INFO:teuthology.task.internal:Not uploading archives. 2024-03-20T10:07:10.854 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-20T10:07:10.882 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-20T10:07:10.882 DEBUG:teuthology.orchestra.run.smithi026:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-20T10:07:10.885 DEBUG:teuthology.orchestra.run.smithi115:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-20T10:07:10.887 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-20T10:07:10.901 INFO:teuthology.orchestra.run.smithi026.stdout: 525364 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 20 10:07 /home/ubuntu/cephtest 2024-03-20T10:07:10.902 INFO:teuthology.orchestra.run.smithi115.stdout: 525364 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 20 10:07 /home/ubuntu/cephtest 2024-03-20T10:07:10.904 INFO:teuthology.orchestra.run.smithi122.stdout: 525356 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 20 10:07 /home/ubuntu/cephtest 2024-03-20T10:07:10.905 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-20T10:07:10.915 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-20T10:07:10.983 INFO:teuthology.nuke:Checking targets against current locks 2024-03-20T10:07:11.009 DEBUG:teuthology.nuke:shortname: smithi026 2024-03-20T10:07:11.009 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-20T10:07:11.030 DEBUG:teuthology.nuke:shortname: smithi115 2024-03-20T10:07:11.030 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-20T10:07:11.036 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi026.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-19_00:09:45-rados-wip-yuri5-testing-2024-03-18-1144-distro-default-smithi/7610089', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-20 08:09:39.970341', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFbLfsyKtlfbM5hSsEszpvyZMDjMdWI9xNUVFfef50zLnCGyt0rpJ7JUTvAFjDcBw8134z1fXKU4dYKPnmdSf1M='} 2024-03-20T10:07:11.049 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi115.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-19_00:09:45-rados-wip-yuri5-testing-2024-03-18-1144-distro-default-smithi/7610089', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-20 08:09:39.972023', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLj6MPU4ULnRmUWzb51y6RVPXdfj2wc0JekcyMhoibK975aiTV3lLi0sYxM41j/jczfyoyeOitSz1Q5k9vstByY='} 2024-03-20T10:07:11.055 DEBUG:teuthology.nuke:shortname: smithi122 2024-03-20T10:07:11.055 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-20T10:07:11.077 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi122.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-19_00:09:45-rados-wip-yuri5-testing-2024-03-18-1144-distro-default-smithi/7610089', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-20 08:09:39.968332', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG6HnHBgjbRlOWERJX5IeyadJ4AUYQe3C1Oxb4ZYCrhO2nni/x68UjAN3rPgLS4Az4+DjCJ1tR+rHrWIrd5wwgg='} 2024-03-20T10:07:11.085 INFO:teuthology.orchestra.console.smithi026:Power off 2024-03-20T10:07:11.086 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-20T10:07:11.100 INFO:teuthology.orchestra.console.smithi115:Power off 2024-03-20T10:07:11.101 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-20T10:07:11.110 DEBUG:teuthology.orchestra.console.smithi026:power off output: Chassis Power Control: Down/Off 2024-03-20T10:07:11.111 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-20T10:07:11.124 DEBUG:teuthology.orchestra.console.smithi115:power off output: Chassis Power Control: Down/Off 2024-03-20T10:07:11.124 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-20T10:07:11.134 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is on 2024-03-20T10:07:11.150 INFO:teuthology.orchestra.console.smithi122:Power off 2024-03-20T10:07:11.150 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-20T10:07:11.159 DEBUG:teuthology.orchestra.console.smithi115:check power output: Chassis Power is on 2024-03-20T10:07:11.172 DEBUG:teuthology.orchestra.console.smithi122:power off output: Chassis Power Control: Down/Off 2024-03-20T10:07:11.172 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-20T10:07:11.195 DEBUG:teuthology.orchestra.console.smithi122:check power output: Chassis Power is on 2024-03-20T10:07:15.135 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-20T10:07:15.160 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-20T10:07:15.196 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-20T10:07:15.246 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is on 2024-03-20T10:07:15.270 DEBUG:teuthology.orchestra.console.smithi115:check power output: Chassis Power is on 2024-03-20T10:07:15.308 DEBUG:teuthology.orchestra.console.smithi122:check power output: Chassis Power is on 2024-03-20T10:07:19.251 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-20T10:07:19.270 DEBUG:teuthology.orchestra.console.smithi115:pexpect command: ipmitool -H smithi115.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-20T10:07:19.309 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-20T10:07:19.361 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is off 2024-03-20T10:07:19.381 DEBUG:teuthology.orchestra.console.smithi115:check power output: Chassis Power is off 2024-03-20T10:07:19.419 DEBUG:teuthology.orchestra.console.smithi122:check power output: Chassis Power is off 2024-03-20T10:07:19.463 INFO:teuthology.orchestra.console.smithi026:Power off completed 2024-03-20T10:07:19.482 INFO:teuthology.orchestra.console.smithi115:Power off completed 2024-03-20T10:07:19.521 INFO:teuthology.orchestra.console.smithi122:Power off completed 2024-03-20T10:07:19.625 INFO:teuthology.run:Summary data: description: rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/quincy backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/radosbench} duration: 5352.491647720337 failure_reason: '"2024-03-20T08:50:40.165150+0000 mon.a (mon.0) 196 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log' flavor: default owner: scheduled_yuriw@teuthology success: false 2024-03-20T10:07:19.625 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-20T10:07:19.711 INFO:teuthology.run:FAIL